I am new to this forum. I have been programming in C++ for some time now, but I have wondered how to compile the code into a custom file format. Most IDEs compile into a file format that the OS understands, but I want it to compile into my file extension, which will be something like *.tst (my way of saying test). Maybe there is a compiler that compiles into your own extension, but I have only used Visual Studio, so I don't know.

Do you want the file extension to be different, or do you want the file format to be different?

Changing the file extension is trivial; any compiler will let you specify the name of the generated file. On Visual Studio it's setting in the project properties.

Generating an executable file in a different format is immensely more complicated.

On unix I think executable is just a flag on a file, and extension is ignored.
on windows, specific extensions are executable, and nothing else can be (??) as far as I know: exe, scr, com*, bat**, ?? others? there are some more, many of them actually pieces/extensions to OS programs like ctrl panel.

com required specific compiler flags under dos, dunno what the state of the extension is now.
bat is an executable text file with command line calls and specific syntax.
scr is supposed to be a screen saver, but nothing checks how you use it.

there is a spoof using unicode that lets you flip 3 letters such that
fileexe.txt is taken as filetxt.exe and executes. Many virus scanners and such block this as it is abuse / hackery and not a legit thing to do. You can't do this easily; you need to inject hidden characters into the name (its more than just swapping them around). The web has details.