11-17-2023, 08:13 PM
Those results are more-or-less about what I expected. Let me break them down for you, and I'll explain what's going on with the values -- and why you might want to tweak your personal version of QB64 to suit your own specific needs.
1000 seconds for OUTPUT and PRINT. This is just a basic loop which uses PRINT to print the data line by line. It's our baseline time, and what QB64 uses currently.
400 seconds for PUT and BINARY. This uses PUT to place the data line by line, more or less just like the above -- but it's over twice as fast.
12,300 seconds for PUT with one single write. This is basically what I showed you above, that you said zoomed so speedily on your drive. As long as your programs are short, and there's little data to place into a single line, this is the method which will give you the best write speed. UNFORTUNATELY, as you can see from this test, when the files start getting longer and the program line count grows, the times here bloat exponentially. String manipulation is a convoluted and slow process, and even though this method has the fastest WRITE times, it has a very, very long BUILD time where it assembles all those various lines into one massive line to write all at once.
255 second for PUT with writing in batches. This is a middle ground method where we assemble a small number of lines together and then PUT them to disk in batches. It's probably the optimal method for your use on your network, if you have files which grow to any sort of size whatsoever.
As long as your programs are short, you'll probably find that the method I shared earlier is going to be the best for you. These lines are massive at about 67,000 characters per line, and in the end we end up writing over 60MB into the save file for the program -- much more than what most folks will ever have for a BAS file!!
But, there's that trade off that comes with the string concatenation with large strings. Once the program gets past X number of lines, any speed you save from that single write is going to be lost due to the massive overhead that comes from assembling all those lines into one.
For QB64PE itself, we're probably going to end up going with the simple PUT line-by-line method for everyone. It means that save times will be reduced by more than half for everyone -- including your network saves. Instead of a 9-minute save, I'd imagine the files would end up taking about 3 or 4.
You'll just have to decide for yourself what works best in your specific case. For us, we can't ever be certain how long an user's program might be, so the "add it all into one string and do it in a single write" method isn't viable as a general use method for the masses. The times involved with it are about a dozen times LONGER than what you saw originally, with massively long BAS files, and if you found a 9 minute save unbearable, think of how a 200 hour save would be!!
By delving deeper into this, we've shown we can improve save speeds by half, without having the issue of slowing things down inadvertently with much longer source files. That's not a bad accomplishment to tuck under the belt at the end of the day!
In your specific case however, you need to weigh the following factors:
1) Is file size an issue?
2) How great is the network delay.
3) Which is the way to get the best results given the size files you normally work with, the time it takes to assemble them into a single string, and then write them to the disk.
Small files? Write them all at once. The overhead is small in adding the lines together and the time saved in the write is worth it.
Very large files? Probably best to write them in batches of 10 to 25 lines in a single pass. Don't let the string you're writing grow overly long, but reduce the number of writes to the drive as much as possible.
For general use -- just PUT them line by line and improve speeds considerably.
For your specific network use? It all depends on the sizes involved and the optimal number of merges with their overhead attached, compared to the overhead for the file writing itself.
1000 seconds for OUTPUT and PRINT. This is just a basic loop which uses PRINT to print the data line by line. It's our baseline time, and what QB64 uses currently.
400 seconds for PUT and BINARY. This uses PUT to place the data line by line, more or less just like the above -- but it's over twice as fast.
12,300 seconds for PUT with one single write. This is basically what I showed you above, that you said zoomed so speedily on your drive. As long as your programs are short, and there's little data to place into a single line, this is the method which will give you the best write speed. UNFORTUNATELY, as you can see from this test, when the files start getting longer and the program line count grows, the times here bloat exponentially. String manipulation is a convoluted and slow process, and even though this method has the fastest WRITE times, it has a very, very long BUILD time where it assembles all those various lines into one massive line to write all at once.
255 second for PUT with writing in batches. This is a middle ground method where we assemble a small number of lines together and then PUT them to disk in batches. It's probably the optimal method for your use on your network, if you have files which grow to any sort of size whatsoever.
As long as your programs are short, you'll probably find that the method I shared earlier is going to be the best for you. These lines are massive at about 67,000 characters per line, and in the end we end up writing over 60MB into the save file for the program -- much more than what most folks will ever have for a BAS file!!
But, there's that trade off that comes with the string concatenation with large strings. Once the program gets past X number of lines, any speed you save from that single write is going to be lost due to the massive overhead that comes from assembling all those lines into one.
For QB64PE itself, we're probably going to end up going with the simple PUT line-by-line method for everyone. It means that save times will be reduced by more than half for everyone -- including your network saves. Instead of a 9-minute save, I'd imagine the files would end up taking about 3 or 4.
You'll just have to decide for yourself what works best in your specific case. For us, we can't ever be certain how long an user's program might be, so the "add it all into one string and do it in a single write" method isn't viable as a general use method for the masses. The times involved with it are about a dozen times LONGER than what you saw originally, with massively long BAS files, and if you found a 9 minute save unbearable, think of how a 200 hour save would be!!
By delving deeper into this, we've shown we can improve save speeds by half, without having the issue of slowing things down inadvertently with much longer source files. That's not a bad accomplishment to tuck under the belt at the end of the day!
In your specific case however, you need to weigh the following factors:
1) Is file size an issue?
2) How great is the network delay.
3) Which is the way to get the best results given the size files you normally work with, the time it takes to assemble them into a single string, and then write them to the disk.
Small files? Write them all at once. The overhead is small in adding the lines together and the time saved in the write is worth it.
Very large files? Probably best to write them in batches of 10 to 25 lines in a single pass. Don't let the string you're writing grow overly long, but reduce the number of writes to the drive as much as possible.
For general use -- just PUT them line by line and improve speeds considerably.
For your specific network use? It all depends on the sizes involved and the optimal number of merges with their overhead attached, compared to the overhead for the file writing itself.