Issue saving across VPN

[dano]

I have had this issue with QB64 for years, but have just finally decided to bring up the issue.

Here is where the issue happens:  I have a hardware point-to-point VPN between work and home.  I have a share set up on the server which we will call drive z:  I am running QB64 locally on my c: drive.  I open a .BAS file from drive z:, make some changes, and save it.  This is where the issue happens.  It takes FOREVER to save.  I just saved a program with 11,912 lines and it took just under 9 minutes for it to save.  Of course smaller programs take less time, but even they take a lot of time.

The VPN works great - there are no speed issues here.  I ran tests and have no issues.  Backups with large amounts of data run across the VPN late every night without issue.  If I copy the same .BAS file it completes this operation in less than 1 second regardless of direction (work>home vs. home>work) - there is something about when QB64's IDE writes the .BAS file that is causing this issue.

It is not the antivirus causing the issue either as I have tried this with the AV disabled and had the same result.

I am assuming that the system that the IDE uses to save is creating some kind of bottleneck here?  Does the save routine write and close repeatedly?  I am grabbing at straws here.

This is not a big issue since I know to avoid it so if I must work on a file locally, I will copy it, work on it, and  then send it back rather than try to save across the VPN.

Thanks,
dano

[TerryRitchie]

I have had this issue with QB64 for years, but have just finally decided to bring up the issue.

Here is where the issue happens:  I have a hardware point-to-point VPN between work and home.  I have a share set up on the server which we will call drive z:  I am running QB64 locally on my c: drive.  I open a .BAS file from drive z:, make some changes, and save it.  This is where the issue happens.  It takes FOREVER to save.  I just saved a program with 11,912 lines and it took just under 9 minutes for it to save.  Of course smaller programs take less time, but even they take a lot of time.

The VPN works great - there are no speed issues here.  I ran tests and have no issues.  Backups with large amounts of data run across the VPN late every night without issue.  If I copy the same .BAS file it completes this operation in less than 1 second regardless of direction (work>home vs. home>work) - there is something about when QB64's IDE writes the .BAS file that is causing this issue.

It is not the antivirus causing the issue either as I have tried this with the AV disabled and had the same result.

I am assuming that the system that the IDE uses to save is creating some kind of bottleneck here?  Does the save routine write and close repeatedly?  I am grabbing at straws here.

This is not a big issue since I know to avoid it so if I must work on a file locally, I will copy it, work on it, and  then send it back rather than try to save across the VPN.

Thanks,
dano

Yeah, this is odd. At first I thought this is a port forwarding issue but you said you can save the .BAS locally and then copy to the Z: drive with no issues, so that's not it.

Have you tried saving files with other programs, such as an office application, to the Z: drive? Have you noticed any slow saving with any other programs in other words?

I ran into something similar with Netware 3.12 back in the 90's and if I remember correctly it had something to do with file sharing. I had multiple people accessing a database I wrote and the performance across the drive shares was very, very slow. I seem to remember having to set something with file sharing to correct the issue but it's been so long I don't remember what I had to do.

[SMcNeill]

I run lots of stuff off a networked drive, similar to what you're describing, and one thing I've noticed is that these modern drives like to put themselves into sleep mode.  Whenever I go to access that networked drive the first time, it has to wake itself back up, get those platters spinning, get up to speed, and then make everything accessible to the OS -- which takes about half a minute or more on my system.  Once it's up and going, there's no problem with access times or transfering data, but there's that initial start-up time involved with the process.

With QB64, this would lead to it taking a minute to save a single file (as the drive had to awaken and come online), whereas, like you, late at night I could do large file transfers without any issue.  (After all, what's a 30 second wake up time when transfering 200GB of data over an hour?)

I'm wondering if something similar is happening for you.  Is the system taking an excessive amount of time to wake up that drive and get it transfer ready?   One easy test for such a thing would be to save the file, let it take X minutes to do its thing, and then try to resave the file and see if that transfer rate has went back down to something reasonable.  After all, the first save would be one to awaken the device, so the second attempt at saving should take that time out of the transfer equation, leaving you with a better idea of how fast you're actually moving the data across the network.

[dano]

Terry - No, I do not have any other issues with any other programs including Office.  Everything saves fast, not quite as fast as local but pretty close.

SMcNeil - The drives in the server are all SSD flash memory so there is no lag time, and the hardware based RAID controller in use does not allow sleep functions for online drives.  Good thinking though!  (yeah, our system at work is pretty drool worthy!  10GbE fiber to the desktop too!  It is so freakin fast!)


I ran 2 different programs to save the identical information 2 different ways using QB64 and there are astounding differences:

'Program 1
Dim raw$(4000)
For x = 1 To 4000 '      load array with data
    p% = Int(Rnd * (40)) + 20
    l% = Int(Rnd * (40)) + 20
    raw$(x) = Mid$("This is a sample sentence that we will be using to generate random length strings that have pseudo random characters in them by taking a random position with a random length", p%, l%)
Next x
s# = Timer
Open "o", #1, "z:\testfile.dat"
For x = 1 To 4000 '      write data from array
    Print #1, raw$(x)
    Print x
Next x
Close #1
Print "Time Elapsed: "; Timer - s#

This one took 5 minutes and 15 seconds to write all 4000 lines.  I then changed that program just a tad so that the output file is local to drive c and it saved the file in .05 seconds.



Then I copied the file that was generated in program 1 across the VPN to drive c: of the local machine and that took < 1 second.  I then used the following program to copy it back to z: but used binary mode instead:

'Program 2
s# = Timer
fA = FreeFile
Open "c:\temp\testfile.dat" For Binary As fA
fB = FreeFile
Open "z:\testfile.dat" For Binary As fB
fbc$ = Space$(LOF(fA))
Get fA, , fbc$
Put fB, , fbc$
Close fA
Close fB
Print "Time Elapsed: "; Timer - s#

This one finished in 1.15 seconds.


I hope this helps...
dano

[SMcNeill]

Sounds like the system isn't doing multiple transfers.  With the 4000 file output, it's connecting to the drive and having to write a line of data, do whatever confirmation it does to make certain the data transferred is valid, and then finally moves on to do the next line of data.   With the second method, it's transfering the whole file in one pass, then doing the validation, which saves a lot of time overall.

We see a similar type of behavior to this in OPEN FOR INPUT, which has to read a file one byte at a time, resulting in many many many unnecessary disk reads.  (Whereas reading AS BINARY reads data at the sector/clusters size of your drive -- let's say 4096 bytes for example -- which means you're doing in one disk read what INPUT takes 4096 disk reads and transfers to do.

IIRC, QB64 does a very similar method of drive writing such as your first example -- one line of a program at a time, dumped via a simple DO..LOOP.  It's never been an issue before, in the past, with saving to local drives, so nobody's ever bothered to optimize it.  Changing that save routine to using a single PUT statement as you've done in your second example should be easy enough to do.   It's late here tonight, but I'll take a look in our internals tomorrow and see about swapping us over to a single binary write of the whole file, and then you can test it out later and see if it makes much of a difference on your system. 

[SpriggsySpriggs]

Huh. Didn't know we weren't doing a binary output on the file.

[SMcNeill]

Huh. Didn't know we weren't doing a binary output on the file.

The current method:

Code: (Select All)

SUB idesave (f$)
    ideerror = 6
    OPEN f$ FOR OUTPUT AS #151
    ideerror = 1
    FOR i = 1 TO iden
        a$ = idegetline(i)
        PRINT #151, a$
    NEXT
    CLOSE #151
    IdeSaveBookmarks f$
    ideunsaved = 0
END SUB

[SMcNeill]

@dano Here's something to try:

1) GO into source/ide/ide_methods.bas and navigate down to SUB idesave.
2) Replace that sub with this one:

Code: (Select All)

SUB idesave (f$)
    ideerror = 6
    OPEN f$ FOR OUTPUT AS #151: close #151
    OPEN f$ FOR BINARY AS #151
    ideerror = 1
    FOR i = 1 TO iden
        a$ = idegetline(i)
        if INSTR(_OS$, "WIN") then
           outfile$ = outfile$ + a$ + chr$(13) + CHR$(10)
         else
           oufile$ = outfile$ + a$ +chr$(10)
        end if
    NEXT
    PUT #151, 1, outfile$
    CLOSE #151
    IdeSaveBookmarks f$
    ideunsaved = 0
END SUB

3) Save that and then navigate up to the /source folder and recompile QB64pe.BAS. (Make certain the "Export EXE to Source Folder" option is **DISABLED***.)
4) This should now make you a QB64pe(2).exe file, which you can run and see if it corrects the issue for you.

If the fix works, I'll push it into the repo later, and we'll start doing binary file writes with the next release. If it doesn't help, then you know the issue lies somewhere else and you'll just have to keep digging to see what might be affecting the performance for you.

[SpriggsySpriggs]

Would it not be safe to have the output file only use CHR$(10) for Windows as well?

EDIT: I know that Windows likes carriage return and line feed endings.

[SMcNeill]

Would it not be safe to have the output file only use CHR$(10) for Windows as well?

EDIT: I know that Windows likes carriage return and line feed endings.

It should be, but I doubt we could change that.  Who knows how many folks have old programs that manually look for 13+10 as they parse their stuff?  You wouldn't want to break that backwards compatibility.