Making files more portable

[JRace]

Steve's $Embed thread (https://qb64phoenix.com/forum/showthread.php?tid=3142) got my curiosity going.


But first, an FYI:
For several years now I've been using the 7Zip self-extractor to compress software packages, such as editors or compiler suites, into single-file EXEs for easy transport on a thumbdrive.

7Zip's cleverly designed self-extractor stub (which I will call the .SFX) can extract these files into a temporary folder, and then execute a file of my choice.  Normally the executed file would be a script to copy files into a more permanent installation folder, but I instead run that now unpacked editor or a batch file to perform compilation steps.  The .SFX waits patiently and when all programs in the temp folder have ended, the .SFX conscientiously deletes that temp folder.  No muss, no fuss.

This saves a lot of space on a thumbdrive, compressing a 512 megabyte (after the deletion of some non-essential MinGW files) QB64PE suite down to just 53.48 megs.
It also saves a LOT! of time.  I don't know how long it would take to copy all 6350 (again, fat-trimmed) files of QB64PE to a thumbdrive, but copying MinGW alone can take several HOURS.  (At least, it took that long on the thumbdrives I was using.  I will never buy another SanDisk flash drive.  I own several and those turkeys are all SLOW.)

(If anyone is thinking of trying the .SFX stunt described above I have to warn you that it's not for the faint-of-heart.  It's a manual process that cannot be done from the 7Zip GUI.  It took much time, trial, and error to get it all working the way I envisioned.)

---

Anyway, I created a small executable ("Hello, World", written in C) and embedded it into a QB64PE test program which would extract HELLO.EXE to disk and run it.  The tester compiled and ran as it should.

Ohhhkay, now a slightly bigger test.  For this one I chose CharlieJV's Basic Anywhere Machine and modded the test program to fit:

Code: (Select All)

_Title "ExtractRun"

$Embed:'./BAM.html','embhandle'

o$ = _Embedded$("embhandle")
_WriteFile "./BAM_extracted.html", o$
If _FileExists("./BAM_extracted.html") Then Shell "start BAM_extracted.html"
End

(the filenames in ExtractRun can be changed as needed)

The compiled ExtractRun with a small executable is 1.95 megs.  Let's consider that the baseline, minimal ExtractRun size.
Uncompressed Basic Anywhere Machine weighs in at 4.54 megabytes.
The compiled ExtractRun with BAM is 2.81 megs (UPX can compress that down to about 1.5 megs.)
That means PE's $EMBED squeezes BAM down to just 0.86 megs.

Just posting this in case anyone wants to tinker or build with it.

[SMcNeill]

You can reduce your overall EXE size by adding a simple $CONSOLE:ONLY to the code above.  It's not printing or displaying anything to the screen, so there's no reason to add the graphic screen overhead to it.

[Dav]

Neat. I haven't tried this with 7zip yet.  I made a tutorial to do something like this using WinRar back when QB64 was still SDL and had a lot of files required with every EXE.  Winrar let me specify an icon for the generated EXE.  

Will have to give 7zip a try now.

- Dav

[JRace]

Thanks for the tip @SMcNeill!  I had considered $Console:Only, but not for file size reasons.

@Dav
Aha, I remember seeing that tutorial or one like it, but I could not remember who, where, or which.  Your pointy-clicky WinRar method would be more suitable for most folks' needs than the 7Zip method.

---

A few details for the 7Zip curious:
The required 7Zip installer is in "lzma????.7z from the 7Zip website, where "????" is the version number, currently 2408.  7zr.exe in that arcchive can do the archive creation work, while the "7z*.sfx" stubs handle the extraction & execution.

Any version of 7Zip can be used for simply launching a program, but for command-line use with compilers, 7Zip version 9.20 is needed.  That version has a bug which allows a batch file to access the folder which contains the self-extracting archive.  (I initially got my compiler archive process working using v9.20, then banged my head against the wall for a couple of days wondering why it wouldn't work on an up-to-date version of 7Zip.  Turned out I was making use of an "undocumented feature.")

Some config file editing is needed for the .SFX stubs, and the documentation is a bit sparse.  Hint: a small configuration text file must be embedded into the finished executable between the .SFX extractor and the .7z archive.  Not all of the available .SFX extractors will recognize that config file.

(((Notes if you experiment with 7z v9.20: the 7Zip files you need are in 7z920_extra.7z from SourceForge.  The v9.20 .SFX stubs get confused and run the wrong program if they see more than one executable or batch file, so ONLY one executable should be visible to them.  Hide everything else in subdirecctories.)))

---

I've never before mentioned this process online because it involves a fair amount of experimentation & configuration which is not for command-line virgins.  I was afraid that people might need a book's worth of explanation and a lot of hand-holding.

If the centerpiece of my SFX archive requires pre-run setup, or dynamic command-line parameters like input files, etc, then I configure the archive to run a batch file which does what's needed.

[SMcNeill]

I've always just found the easiest way to make a 7z self-extracting EXE is to use the 7z File Manager.  Select what files/folders I want, click the Add button, fill out the settings in the pop-up that makes the archive.  Just be certain to check the "Make SFX archive" button before clicking OK to finish.

   

[JRace]

I've always just found the easiest way to make a 7z self-extracting EXE is to use the 7z File Manager.  Select what files/folders I want, click the Add button, fill out the settings in the pop-up that makes the archive.  Just be certain to check the "Make SFX archive" button before clicking OK to finish.

Well, I don't wanna brag, but:

     
(Let's hear it for ludicrously long Send To menus!)
(Actually, "Quick7z exe" is the one that I would use.  The highlighted menu item turns a .ZIP file into a self-extracting .EXE that runs on Windows OR DOS.)

But anyway, the party trick I described actually runs the archived software from Windows TEMP folder then deletes it all without a trace when done.

• Need to compile FIZZBUZZ.BAS and nothing else?  Simple click&drag FIZZBUZZ.BAS onto qb64pe_win-x86-3.14.1__(+GCC-13.2.0)_(Vista+)_7zXu.exe and after a few seconds  FIZZBUZZ.EXE will magically appear.
  
• Want to install QB64PE?  Open a command prompt and type "qb64pe_win-x86-3.14.1__(+GCC-13.2.0)_(Vista+)_7zXu.exe UNPACK" and voila!  There it is!
  

(I append "_7zX" to the self-extractor file name to remind myself what it is.  "_7zXu" files have an UNPACK option.)

(Some anti-malware installations may complain about running executables from TEMP, but I haven't had any problems yet.  Knock on wood.  If worse comes to worst, 7Zip can unpack it's own installers, and many other types of installers as well.)

Depending on the batch script which drives the package, many other options may be available.  Here's an example of what  qb64pe_win-x86-3.14.1__(+GCC-13.2.0)_(Vista+)_7zXu.exe can do:

fullauto <name.ext> : Auto-select Basic IDE, or C/C++ console mode compile.
ide [name.ext]        : Start the QB64PE IDE.
cmdline                   : Open a Command Prompt in the QB64Pe temp directory, with PATH set appropriately.
bcomp <name.ext>      : Compile using QB64PE with the "-c" & "-s:exewithsource=true" flags.
cc <name.ext>         : Auto-select C or C++ console mode compile.
ccon <name.ext>       : Compile a Windows console-mode C program.
cppcon <name.ext>     : Compile a Windows console-mode C++ program.
cgui <name.ext>       : Compile a Windows GUI C program.
cppgui <name.ext>     : Compile a Windows GUI C++ program.
make [options]      : Run MinGW32 Make.
unpack [path]    : create a directory named "qb64pe" in the specified path, which must exist.

Default mode is fullauto if a filename is given and no mode is specified.
Default mode is ide if no filename is given and no mode is specified.

For a compiler suite like PE there is one big batch script driving everything.  If the UNPACK option is used then a whole set of batch files are made available (one for each option), allowing simple click&drag operation of the compiler(s) for simple uses.

(A note from the Distributor of TMI: After writing too many batch scripts for too many _7Zx archives I finally created a batch file generator that uses relatively simple .INI files to generate the needed batches.)

(protip for command-line noobs: don't type a stupidly long filename like " qb64pe_win-x86-3.14.1__(+GCC-13.2.0)_(Vista+)_7zXu.exe".  Use <TAB> command completion or right-click the .EXE and rename it first.)

The usefulness of this technique is subjective.  It started off with an "I wonder...." moment and turned into a pet project from there.

[grymmjack]

Well, I don't wanna brag, but:

 
(Let's hear it for ludicrously long Send To menus!)

haha, that's awesome. been there, but not to this extent! you win!

[bplus]

Not to distract but FIZZBUZZ.bas! I made a cool version of that extending it a bit, it might even be here

[euklides]

I use UPX (64bit) (see   https://github.com/upx/upx/releases/tag/v4.2.4)
(also for Linux and Win 32 available, free version 4.2.4)

Compresses EXE files directly for use  (50/60% reduced)

UPX for Windows: no installation. Use MS DOS command like "upx.exe myprogramme.exe"

Useful when your exe program works alone. Of course all additional files must be zipped.

[JRace]

UPX is a great tool that I use all the time on Windows.

Executable compression has its pros & cons, which you will find people talking about online if you do a bit of Googling.

Speaking practically, EXE compression is a space/speed trade-off.

I used to UPX-compress almost all of the EXEs & DLLs in compiler suites.  Not quite all, because some EXEs  & DLLs don't work when compressed.

I don't do that any more.  UPX can decompress and run files fairly quickly, but that decompression still takes time, and compilers like MinGW (the "backend" of QB64PE) & CLang are made up of many executables calling each other, loading DLLs, etc.  That adds up to a lot of decompression time during every compile.  Compressing every file in a compiler suite REALLY slows down the compiler.  You MAY (or may not) actually see some speed benefit by compressing EXEs on a slow thumbdrive, but not on a hard drive.  Do this on a hard drive or SSD only if you are desperately short on space.

(Fun fact: 7z.exe runs just fine when UPX-compressed, but if you use the UPX -decompress option to restore 7z.exe to its "original" form, that restored 7z.exe will no longer work.)

(Back in the Win98 daze I compressed just about every executable in my Win98 installation.  Ooopsie!  That's a surefire way to get Windows stuck in a bluescreen boot loop.  That quickly led to a Win98 REinstallation!)

Also, tightly compressed files don't archive well.  An archive of pre-compressed files is often larger than an archive of large, uncompressed files.

These days I just leave the hardworking compiler EXEs alone and have Windows automagically compress my desktop box's programming folder, which transparently compresses everything in the folder & subfolders right down to all those header and library files, which can add up to a lot of space.  This is all handled by the operating system and is invisible to the user.

This file-system level compression is an optional feature on all of the major operating systems.  It is not on by default.

It's still a space/speed trade-off, but it's one that I and my overstuffed hard drive can live with.