Question about _MEM blocks and arrays

[madscijr]

A practical use for mem?   

Anything that requires speed. 

Direct memory manipulation is much faster than referenced access to the same memory.   Array(index) has to first look up the index, then calculate the offset position of that index, then check the memory location for the data you want, then return it to you.  With mem, you specify the offset yourself, the amount of data you want, and presto, retrieve it

My question is, can't we update the compiler to be a little smarter, and generate code that uses these direct memory manipulation tricks, for commands like array(index)?

[Jack]

a simple example to copy an array

Code: (Select All)

Dim a(10), b(10), m As _MEM

For i = 0 To 10

    b(i) = i + 1

Next



m = _Mem(b()) '<-- to point to the array b()

_MemGet m, m.OFFSET, a() ' copy array b() into array a()

For i = 0 To 10

    Print a(i)

Next

[FCS_coder]

The motivation for me was the ability to do array processing on dynamicall sized blobs of data.  I wanted to use MEM blocks the same way Pascal lets you allocate arbitrary sized records by using either arrays or pointers to char/byte lists.

I want to have a record/struct  ocontaining a status byte, a size allocated byte, plus the dara.  Thinking about it I realized I can bump the data up by two bytes/words and just offset the data by that much.  That means I can just pass arrays to/from SUBs/PROCs as needed.  I was led to that by the info that you can't (shouldn't?) use arrays in UDTs.

How good is the heap tracking on REDIMed arrays?  Do I nee to worry about fragmentation right away?  Not sure how smart the back-end C compiler's memory anagementt library is.

[madscijr]

You know what would be very helpful, would be a mini-tutorial series showing all the different common tasks in QB64PE that can be done a lot faster with memory operations, with a simple example of the standard QB64PE method of doing the thing, alongside the memory manipulation method, and an explanation. Then we would have a nice and easy reference to work from all in one place...

[Jack]

it might also be helpful to have side-by-side comparison showing C pointers vs QB64 _Mem methods

[Kernelpanic]

@FCS_coder - I was led to that by the info that you can't (shouldn't?) use arrays in UDTs.

That is true, but you can use record arrays. Maybe this is a possibility.

PS: Bug fixed. But the loop isn't perfect yet.

Code: (Select All)

'Recordfeld, Hinweis S. 45 und S. 173 - 6. Juni 2025



Option _Explicit



Type Motorrad

  Modell As String * 30

  Kilowatt As Double

  Preis As Double

End Type



Dim MotorradMarken(2) As Motorrad



Dim As Integer anzahl, satzNummer, zeile, spalte



satzNummer = 1

zeile = 3: spalte = 3

Locate zeile, spalte

Do

  Input "Modell    : ", MotorradMarken(satzNummer).Modell

  Locate CsrLin, spalte

  Input "Kilowatt  : ", MotorradMarken(satzNummer).Kilowatt

  Locate CsrLin, spalte

  Input "Preis      : ", MotorradMarken(satzNummer).Preis

  satzNummer = satzNummer + 1

  

Loop Until satzNummer <= 2



Locate CsrLin + 2, spalte

For anzahl = 1 To satzNummer

  'Sa muss ein Semikolon hin, sonst langer Abstand

  Print Tab(4); "Modell    : "; MotorradMarken(anzahl).Modell

  Print Tab(4); Using "Kilowatt  : #####.##"; MotorradMarken(anzahl).Kilowatt

  Print Tab(4); Using "Preis      : #####.##"; MotorradMarken(anzahl).Preis

  Locate CsrLin + 1, spalte

Next



End

The loop doesn't work. 

[DSMan195276]

A practical use for mem?   

Anything that requires speed. 

Direct memory manipulation is much faster than referenced access to the same memory.   Array(index) has to first look up the index, then calculate the offset position of that index, then check the memory location for the data you want, then return it to you.  With mem, you specify the offset yourself, the amount of data you want, and presto, retrieve it

My question is, can't we update the compiler to be a little smarter, and generate code that uses these direct memory manipulation tricks, for commands like array(index)?

Steve is overstating the difference between the two  Functionally both do the exact same thing already and will have extremely similar performance. A slight difference is that arrays don't have to start from zero (Ex. `Dim a(20 to 40)`) so a subtraction has to be done to take an index like `a(21)` and make it zero-based (`21 - 20`), but overall that's pretty minor. Normal array access of a single element via `a(index)` vs `_MemGet()` are basically the same.

If you need to move large parts of an array though then `_Mem` is faster, `_MemCopy` uses faster techniques than just doing a loop. In theory the compiler could do optimizations like that, but determining whether a loop in the code can be replaced is complicated and in many cases not possible.

For example, a looping copy of an array to shift elements has pretty specific behavior when there's an error - it will copy every element before it hits the index that is too large. Optimizing the loop while also maintaining the same behavior on errors is pretty messy, `_MemCopy` for example does not have the same behavior (it errors before doing any copying). The alternative of determining at compile-time that the code cannot hit an error is also hard (and in many cases not possible).

[madscijr]

My question is, can't we update the compiler to be a little smarter, and generate code that uses these direct memory manipulation tricks, for commands like array(index)?

If you need to move large parts of an array though then `_Mem` is faster, `_MemCopy` uses faster techniques than just doing a loop. In theory the compiler could do optimizations like that, but determining whether a loop in the code can be replaced is complicated and in many cases not possible.

Thanks. That makes sense.
So how about they add a new native QB64PE command called MoveArrayElements(MyArray, SourceStartElementIndex, SourceEndElementIndex, DestIndex) that uses the memory method under the hood? Then the compiler does not need to try and determime what the source code is doing, it's now just a straightforward command.

(Or, I suppose we could create a user-defined function that does the same, with whatever checks to make sure the values are valid. Either way, in the spirt of BASIC, to keep life simple and save the programmer from having to worry about low level headaches like worrying about how many bytes a variable uses, etc. If we don't mind that low-level stuff, we might as well be coding in C!)

[DSMan195276]

Thanks. That makes sense.
So how about they add a new native QB64PE command called MoveArrayElements(MyArray, SourceStartElementIndex, SourceEndElementIndex, DestIndex) that uses the memory method under the hood? Then the compiler does not need to try and determime what the source code is doing, it's now just a straightforward command.

(Or, I suppose we could create a user-defined function that does the same, with whatever checks to make sure the values are valid. Either way, in the spirt of BASIC, to keep life simple and save the programmer from having to worry about low level headaches like worrying about how many bytes a variable uses, etc. If we don't mind that low-level stuff, we might as well be coding in C!)

That could be reasonable to add, it's actually fairly hard to implement yourself since you can't just take in a generic "array", it has to be of a specific type (of course, internal QB64 commands don't have this problem). That said, for most use-cases just using a regular loop is honestly fine.