09-27-2023, 11:09 PM
Hi Dav,
nice idea with that Base85 encoding, so you can safly store it in strings without getting the IDE into trouble with binary data.
Guess you're lucky with it for your needs, but have to tell that your methods are deathly inefficient for larger files like digital photos or MP3 music.
Have tested a ~250KB picture, needs ~12sec. for encoding and ~3sec. for decoding/writeback.
However, a ~500KB picture already needs ~80sec. for encoding and ~18sec. for decoding/writeback.
Then tested with an ~1.5MB MP3 file, no idea how long, I aborted encoding after 10min.
I had the same problem with my MakeDATA and MakeCARR programs. It's basically the way how QB64 handles things like
A$ = A$ + "Hello", it gets a new string and then 1st copies the current A$ and then the added string (Hello) into the new string and then makes that the new A$. So obviously this copying with each and every string addition/concatenation gives us an exponential raising copy loop.
It's ok as long as A$ won't grow to more than a couple 10000 chars, but gets really nasty if it grows into millions of chars, which have to be copied with every addition you do.
So you should be able to precalculate the length needed for your output string and pre-define it as
A$ = SPACE$(length) and then using MID$ with an counting offset to replace the spaces in A$ with the actual content. That way the given literal strings are directly copied into the existing A$, hence you don't have that ever copying of the growing A$ itself. This alone shoud give your code a speed boost for such big files from several minutes down to seconds.
A second thing I see is your use of MID$(x$,x,1), when reading just one char ASC(x$,x) is about 5x faster, even if you in turn maybe need to do ASC(x$,x) = ASC("char") if you can't modify your entire logic to operate on ASCII codes rather than single chars.
nice idea with that Base85 encoding, so you can safly store it in strings without getting the IDE into trouble with binary data.
Guess you're lucky with it for your needs, but have to tell that your methods are deathly inefficient for larger files like digital photos or MP3 music.
Have tested a ~250KB picture, needs ~12sec. for encoding and ~3sec. for decoding/writeback.
However, a ~500KB picture already needs ~80sec. for encoding and ~18sec. for decoding/writeback.
Then tested with an ~1.5MB MP3 file, no idea how long, I aborted encoding after 10min.
I had the same problem with my MakeDATA and MakeCARR programs. It's basically the way how QB64 handles things like
A$ = A$ + "Hello", it gets a new string and then 1st copies the current A$ and then the added string (Hello) into the new string and then makes that the new A$. So obviously this copying with each and every string addition/concatenation gives us an exponential raising copy loop.
It's ok as long as A$ won't grow to more than a couple 10000 chars, but gets really nasty if it grows into millions of chars, which have to be copied with every addition you do.
So you should be able to precalculate the length needed for your output string and pre-define it as
A$ = SPACE$(length) and then using MID$ with an counting offset to replace the spaces in A$ with the actual content. That way the given literal strings are directly copied into the existing A$, hence you don't have that ever copying of the growing A$ itself. This alone shoud give your code a speed boost for such big files from several minutes down to seconds.
A second thing I see is your use of MID$(x$,x,1), when reading just one char ASC(x$,x) is about 5x faster, even if you in turn maybe need to do ASC(x$,x) = ASC("char") if you can't modify your entire logic to operate on ASCII codes rather than single chars.
GuiTools, Blankers & other Projects:
https://qb64phoenix.com/forum/forumdisplay.php?fid=32
Libraries & useful Functions:
https://qb64phoenix.com/forum/forumdisplay.php?fid=23
https://qb64phoenix.com/forum/forumdisplay.php?fid=32
Libraries & useful Functions:
https://qb64phoenix.com/forum/forumdisplay.php?fid=23