BUG in 3.6.0 ?

[mdijkens]

I have an older program that used to run fine but now exits unexpectedly in 3.6.0 without any message.

Below a snippet of what I pinned down the error to: (r$ = Mid$(arr$(i&), 80, 20) in processArr&)

Code: (Select All)

Const RECS = 10000000
Dim Shared arr$(RECS)

Print fillArr
Print processArr

Function fillArr&
    For i& = 0 To RECS
        arr$(i&) = Space$(150)
    Next i&
    fillArr& = i&
End Function

Function processArr&
    For i& = 0 To RECS
        r$ = Mid$(arr$(i&), 80, 20)
    Next i&
    processArr& = i&
End Function

I expect it to be some kind of memory error.

(and Yes I know this is an array of 1.5GB, but it used to work up to 2GB)

[Jack]

the problem is with the Mid$ function and is also present in version 3.5.0
can you come up with a Mid$ example without using arrays that would show the problem?

[mnrvovrfc]

I expect it to be some kind of memory error.

(and Yes I know this is an array of 1.5GB, but it used to work up to 2GB)

How much RAM do you have on your computer? If less than 16GB then this shouldn't be done.

"It used to work up to 2GB" but remember that managing strings could take up to 4 times what it looks like strictly from the source code. Concatenation is the biggest offender. Function calls, even to in-built functions like MID$() could make copies of strings only to process.

The operating system needs memory for other things it has to do, including holding your memory-hogging program.

If possible you should break down the task into two or more parts.

[mnrvovrfc]

I was going to create a topic with about the same name because I struggle with _PUTIMAGE. I had a silly program going on Linux that re-saved 99% of the desktop screen in strips 20 pixels across. Each strip seemed allocated OK with _NEWIMAGE and saved with _PUTIMAGE, but when I tried to "reconstruct" the screen with the strips the program kept failing right at the first call of _PUTIMAGE. I ended up creating a triple-dimensional array and copying over colors with POINT().

I also had to deal with no support for _SCREENIMAGE on Linux, had to call a program that did a screenshot and saved to PNG, to then be turned into the visual screen by _LOADIMAGE, and _FULLSCREEN it afterwards.

Maybe I'll show off that program but it's Linux only and I'm not in the mood to make sure it works on Windows.

[mdijkens]

the problem is with the Mid$ function and is also present in version 3.5.0
can you come up with a Mid$ example without using arrays that would show the problem?

No, not really. I specifically run into this with huge string-arrays

[mdijkens]

I expect it to be some kind of memory error.

(and Yes I know this is an array of 1.5GB, but it used to work up to 2GB)

How much RAM do you have on your computer? If less than 16GB then this shouldn't be done.

"It used to work up to 2GB" but remember that managing strings could take up to 4 times what it looks like strictly from the source code. Concatenation is the biggest offender. Function calls, even to in-built functions like MID$() could make copies of strings only to process.

The operating system needs memory for other things it has to do, including holding your memory-hogging program.

If possible you should break down the task into two or more parts.

I have 128GB of RAM and use QB64 for years to process really big log-files (up to 1TB)

The code above is just a small snippet; the real program uses huge _MEM variables, but I use variable length strings to store ~10M temporary values between 50 and 200 bytes long. Making it fixed on 200 bytes is a lot of memory wasted.

The performance of this particular part is not a real issue (filling 10M strings in a few seconds); the whole process runs ~20mins ;-)

But the issue here is that it did work in older versions of QB64 and QB64pe!

[bplus]

Are you running compiler on optimized or different optimized settings?

That is a recent change that might effect memory?

Weird that old standard Mid$ would be effected by anything.

[mdijkens]

Are you running compiler on optimized or different optimized settings?

That is a recent change that might effect memory?

Weird that old standard Mid$ would be effected by anything.

Never changed any compiler options; so still default I guess?

[DSMan195276]

I think you're likely hitting an issue I noticed a few months ago, see here. Basically there's a single buffer backing all the variable length strings and it's size is a 32-bit number (even on 64-bit systems) so it can overflow, crashing your program when that happens. Additionally, because the overflow condition is not checked for there's no error given.

Personally I'd like to redo all that logic, but it's probably not all that hard to just fix the issue. Since nobody else had noticed it I didn't put it high on the list of things to do though Why it works in older versions is hard to say as we haven't touched any of that logic (yet), but the issue is a bit random since it depends on the order you create the strings in and what size they are and such. I think you're also right on the edge of the "supported" size.

Also note that only variable length strings suffer this problem, fixed length strings and _MEM work fine. For such small strings (50 to 200 bytes) if you're filling all of them in then you're probably not saving yourself a whole lot of memory by using variable length strings anyway. The internal `qbs` structure that holds the information on a variable length string is between 30 to 50 bytes in size, so that's 30 to 50 bytes of extra overhead for each string in your array in addition to the actual string data. Fixed length strings in comparison only consist of the backing data itself. You still might come out a bit ahead with the variable length strings, but not by as much, and if the majority of your strings are in the 150 to 200 byte range then it might actually be worse.

[mdijkens]

I think you're likely hitting an issue I noticed a few months ago, see here. Basically there's a single buffer backing all the variable length strings and it's size is a 32-bit number (even on 64-bit systems) so it can overflow, crashing your program when that happens. Additionally, because the overflow condition is not checked for there's no error given.

Personally I'd like to redo all that logic, but it's probably not all that hard to just fix the issue. Since nobody else had noticed it I didn't put it high on the list of things to do though Why it works in older versions is hard to say as we haven't touched any of that logic (yet), but the issue is a bit random since it depends on the order you create the strings in and what size they are and such. I think you're also right on the edge of the "supported" size.

Also note that only variable length strings suffer this problem, fixed length strings and _MEM work fine. For such small strings (50 to 200 bytes) if you're filling all of them in then you're probably not saving yourself a whole lot of memory by using variable length strings anyway. The internal `qbs` structure that holds the information on a variable length string is between 30 to 50 bytes in size, so that's 30 to 50 bytes of extra overhead for each string in your array in addition to the actual string data. Fixed length strings in comparison only consist of the backing data itself. You still might come out a bit ahead with the variable length strings, but not by as much, and if the majority of your strings are in the 150 to 200 byte range then it might actually be worse.

Thanks for this explanation.

For myself I have an easy work-around (fixed string array *200 bytes), so no worries there.

But it would be nice and improved quality if it gets fixed (or at least a gracefull error-exit) someday...