Got a slow PC? Test these changes, kindly!

[SMcNeill]

@TerryRitchie You were asking about how the IDE slows down once you get to about 3000+ lines of code in it.   Here's a very simple fix, which seems to drastically cut down on that lag for my system.  Give it a try, if you will.  (And anyone else who wants to as well, as it'd be nice to make certain this doesn't break anything before trying to work it into the repo.)

To test this:

1) Have a version of the latest source from the repo (or at least from v3.13+)
2) Grab this file here:  
  ide_methods.bas (Size: 842.52 KB / Downloads: 82)
3) Go to your QB64PE folder, navigate to source/ide and rename ide_methods.bas to something like ide_methods.bak.
4) Move the file you downloaded to that QB64PE/source/ide folder and let it take the place of that old file.
5) Start QB64PE, open source/qb64pe.bas and build yourself a new version of qb64pe.  I'd imagine this should be named QB64PE(2).EXE or similar.
  --- note:  *DON'T have the "PUT EXE IN SOURCE FOLDER" option clicked.
6) If you didn't see the note above, move the QB64PE(2).exe from the source folder to the main folder. 
7) Test it with anything and everything that you want to test it with.

See if the IDE is more responsive.  If it pastes code faster than it did before (CTRL-V) into the IDE.  If your 20000 line program gets sluggish and lags behind your typing, or if it now keeps up with it better.

Personally, I think this is giving me quite an improvement as even 100,000 lines programs now keep up with my typing and such, without lagging behind like they used to sometimes -- but I've got a fairly high end PC and for the last year or so, that lag has been minimal at best for me.

Let me know if any of you guys notice any changes at all -- for better, or worse, or even if I broke something terribly (which I'm certainly more than capable of doing -- and have done to our poor source more than once.   ).  If things are an improvement, I'll see about pushing the changes into the repo to maybe help the IDE perform a little better for folks in the next release.

[TerryRitchie]

@TerryRitchie You were asking about how the IDE slows down once you get to about 3000+ lines of code in it.   Here's a very simple fix, which seems to drastically cut down on that lag for my system.  Give it a try, if you will.  (And anyone else who wants to as well, as it'd be nice to make certain this doesn't break anything before trying to work it into the repo.)

To test this:

1) Have a version of the latest source from the repo (or at least from v3.13+)
2) Grab this file here:  
3) Go to your QB64PE folder, navigate to source/ide and rename ide_methods.bas to something like ide_methods.bak.
4) Move the file you downloaded to that QB64PE/source/ide folder and let it take the place of that old file.
5) Start QB64PE, open source/qb64pe.bas and build yourself a new version of qb64pe.  I'd imagine this should be named QB64PE(2).EXE or similar.
  --- note:  *DON'T have the "PUT EXE IN SOURCE FOLDER" option clicked.
6) If you didn't see the note above, move the QB64PE(2).exe from the source folder to the main folder. 
7) Test it with anything and everything that you want to test it with.

See if the IDE is more responsive.  If it pastes code faster than it did before (CTRL-V) into the IDE.  If your 20000 line program gets sluggish and lags behind your typing, or if it now keeps up with it better.

Personally, I think this is giving me quite an improvement as even 100,000 lines programs now keep up with my typing and such, without lagging behind like they used to sometimes -- but I've got a fairly high end PC and for the last year or so, that lag has been minimal at best for me.

Let me know if any of you guys notice any changes at all -- for better, or worse, or even if I broke something terribly (which I'm certainly more than capable of doing -- and have done to our poor source more than once.   ).  If things are an improvement, I'll see about pushing the changes into the repo to maybe help the IDE perform a little better for folks in the next release.

I just saw this but heading to bed. I'll definitely try this in the morning and let you know what I find. Thank you for this.

[SMcNeill]

A second version of changes to also compare against:   
  ide_methods.bas (Size: 842.83 KB / Downloads: 43)

We're playing around with a few different concepts on how to refactor things to reduce that IDE lag as much as possible, and on a higher-end PC, it's hard to see much diffference.  There's little lag to begin with, so improvements from 0.01 second input lag to 0.009 seconds of input lag are really hard to tell apart.

I'm thinking the first version offered here will be faster than what's packed with the current IDE, and this version should be even a little faster than that one.  Only time and testing will really tell if there's much change honestly though. 

[TerryRitchie]

Ok, first thoughts. So far I have tested the first ide_methods.bas you posted and yes, it's much faster.

I'm testing it with widescreen asteroids. The source code is 3500 lines long and includes two library files that add an additional 1700 lines of code. Normally when I type into this code I have to wait a good 3 to 5 seconds for the ... in the status window to change to OK. With this modification the change happens in less than a second. I pasted the entire code three more times and still fast. Pasting is faster too compared to the original .EXE.

My computer specs for reference:
- Windows 7 Pro 64bit SP1
- Intel Core i7 6700K @ 4.00GHz
- 32GB dual channel 1069MHz RAM

Note: Even though the CPU is a K variant I have not overclocked it. It's running at its native 4GHz.

Something curious is happening though but I don't think it has anything to do with the changes.

I'm getting 6 warnings in my code (see image below). I loaded the code into the original v3.13.0 EXE and I get the same warnings. When I load the code into v3.4.1 there are no warnings. Not sure what the warnings mean. The Wiki does not show any delimiters being needed. The code compiles and executes just fine.

[SMcNeill]

It's telling you it prefers quotes there, to prevent any possible errors.

"Ritchcraft Creation"

It's much the same as when trying to SHELL a command -- without quotes, you may get unexpected results.

SHELL "my program.exe"   <-- this may look for a program called "my" (especially on linux/mac) and use "program.exe" as an parameter for it.  (Much like "type foo.txt" would.)

Putting things in quotes prevents those glitches, this the warning in the newer versions.  

[bplus]

do these time improvements have anything to do with string concatenation you were talking about earlier? when massive improvements seen with 100's 1000's of strings.

[SMcNeill]

do these time improvements have anything to do with string concatenation you were talking about earlier? when massive improvements seen with 100's 1000's of strings.

They do, in a round about away. 

For our IDE, you have to understand how our program is stored -- everything is all in one long string!  (idet$)

Now, when storing data like this, you also have to store the length of the data, or else you won't know where one starts and the other ends.  We do that by storing the length before AND after the string (for quick navigation back and forth between lines).

So any given line of data looks like:

MKL$(LEN(text$)) + text$ + MKL$(LEN(text$))

So a simple PRINT statement would be saved as basically "5PRINT5".

Now, let's add a second print statement -- idet$ is now:  "5PRINT55PRINT5"

Easy enough to understand, parse, and decipher.  Right?

But now think about how you'd add a new line, or command into the middle of that program.   Let's say we want to add a CLS statement:

idet$ = LEFT$(idet$, before_new_line) +  MKL$(LEN(text$)) + text$ + MKL$(LEN(text$)) + MID$(idet$, after_new_line)

We're adding 5 segments of information together to make one new contigious string to hold the whole program.

---

And with that in mind, do you remember the overhead that I was talking about with adding large strings?  <-- That's basically where the IDE slows down as your program gets longer and longer -- it's moving larger and larger strings about in memory with each new line, or even each new character you type!

So I had this basic thought on how to improve things:

To start with, let's look at what we're adding together:

LEFT$(idet$, before_new_line) -- generally a long string (say lines 0 to 1000 in your program)
MKL$(LEN(text$)) -- 4 byte short string (string representation of the length of the new line)
text$ -- probably a fairly short string.  (the new line, which is what? 100 characters or so max, in most cases?)
MKL$(LEN(text$)) -- 4 bytes once again.
MID$(idet$, after_new_line)-- generally a long string (say lines 1001 to the end of your program)

So just by writing the line where we add things together, we're doing a lot of string allocation, moving, and freeing.

long string + short string = long string  (lots of memory)    <first operation LEFT$ + MKL$)
long string + short string = long string  (lots of memory)    <that result + text$) 
long string + short string = long string  (lots of memory)    <that result + MKL$)
long string + long string = long string  (lots of memory)    <that result + MID$)

So, as you can see, that's lots of large chunks of memory being allocated, moved, added, and freed just to make our final result.

Sp let's make the simplest of changes in the world:

insert$ =  MKL$(LEN(text$)) + text$ + MKL$(LEN(text$))
idet$ = LEFT$ + insert$ + RIGHT$

The insert$ is now moving those small blocks of memory and adding them together to make one final small block of memory.
And we now only need to move large blocks of memory twice to get the same result.

---

And that's where my basic idea for how to improve the IDE performance started at.  Reduce/minimize/optimize the number of times we actually initialize/move/free those blocks of memory.

And once you start down this type of road for optimizing things, you just keep on tweaking one little thing at a time, until you just can't tweak it anymore. 

It's not changing the format of the way we hold our data (I'd love to swap over to using a pure array to store each line of data sometime).  We're not changing the index.  Or making any sweeping changes.

We're basically just changing the way we add those strings together to minimize how many times we move large chunks of data about in memory.

[grymmjack]

Is the IDE going to become natively faster in a future release? @SMcNeill that would be great.