Stack size limit for If/Then statements?

[JamesAlexander]

@SMcNeill: QB64 Phoenix Edition Version 3.8.0 on Ubuntu. OS: Ubuntu 20.04.6 LTS; RAM: 8031708 KiB (8GB); Intel Core i5-4310U @ 2ghz.

On my other system (16gb, Core i7), it works perfectly as you described, although I am on Windows 10 instead of linux here. As I mentioned, I am kinda restricted on what I can do with this code (using Select Case and structuring would be nice, but I am fairly limited on this, because the more changes I make, I will have to redo everything and it will be a monster to work woth, outweighing my time to even do it)

I am guessing that I'll have to switch to Windows (hesitantly) to do programming with? The issues that I'm having haven't been seen since 16 bit DOS as stated, but at least I know why this is occurring (a DPMI manager corrected this in DOS.)
 
It still doesn't make sense as to why this is happening on just the linux build and is fine on the Windows one.
Ah well...I tried. Thanks guys!

Food for thought: Is there any provision for adjusting the stack size in the future for linux or in general so that there would be no restriction on this? Such as: $STACK: NORMAL or $STACK:HUGE (65535) OR $STACKYNAMIC (go until we run out of memory!) ?

[SMcNeill]

 Ubuntu 20.04.6 LTS; RAM: 8031708 KiB (8GB);

On my other system (16gb, Core i7),

And there you have the issue. It's not OS. It's not a software limit. It's simply a case of one machine not having enough memory to compile such code. Go make the system swap file 64GB in size and the issue will probably disappear. Of course, compile times will increase drastically as you'll be using the hard drive as basically RAM, and it's much slower by nature.

As I mentioned, this was using 30+GB of RAM on my system to compile 65,536 lines of code written like you describe. That's more than BOTH your systems combined have available. If you expect to ever be able to do 16.7 million lines like this, you'll ***HAVE*** to find a different way. Restructure. Write programs in batch and have one shell out to another. Make this an external data file and look up find/replace entries via it. If 65,536 lines uses 30GB to compile, then I can only image that 16.7 millions of code would need *at least* 9+TB of memory to use, and would probably take years to compile.

What you have isn't expandable or substainable. You really need to go back to the drawing board and find a better way to automate this process.

---

And here's a question: Where are you getting the idea that STACK SIZE has *anything* at all to do with this program?? It uses no GOTO, no GOSUB. It has no RETURN statements. There's no RECURSIVE FUNCTION calls. There's *NO USE OF STACK* here at all.

You're not running into any sort of stack limit. This is a simple case of just running into MEMORY limits, and those are limited by your system. Your only real options are:

1) Go to the store and buy more memory and put it in your Linux machine.
2) Increase the swap file size to something outrageous and use the hard drive for "virtual memory".
3) Restructure and rewrite the program so that it's not something which is going to be trying to use 30+GB of RAM to compile and build.

[bplus]

There are no other nested loops or anything. Just "If X = [statement] THEN [command 1] : [command 2]: [command 3] on a single line.

@JamesAlexander

All that repetitious writing! Seems like a job to give to a program. Surely there is a formula that decides what to make command1: command2: command3 based on x, after all you calculate it for every line. Did you every write a program to write a .bas file? Makes the drudgery fun!

[JamesAlexander]

@BPlus: Yes I did lol. It was all fun and games after a few thousand lines, but about the 10,000 or 20,000 lines, things got serious. I'm glad that I didn't have to edit it after it was made.

@KernelPanic: Yes. It was a matter of running out of memory.

@SMcNeill: "What you have isn't expandable or substainable. You really need to go back to the drawing board and find a better way to automate this process."
--> I think I will have to, because if I hit the limit at 8GB, I'll start disk thrashing from there on out. I see no other choice but to do a virtual swap (because I have a max ram limit of 8GB; I have two 4GB chips on there now, and it is a good system, but it is showing it's age..that is why I have upgraded it with the newer laptop, maxing out at 32GB). With the 8GB system, even with the SSD to mitigate this, the virtual memory on the drive will be thrashing either way. I think I have an alternative way of doing this now that I'm laying this out, though...

@SMcNeill: "Where are you getting the idea that STACK SIZE has *anything* at all to do with this program?? It uses no GOTO, no GOSUB. It has no RETURN statements. There's no RECURSIVE FUNCTION calls. There's *NO USE OF STACK* here at all."
--> For reference: https://communities.actian.com/s/article...Stack-Size or rather, this: https://www.guru99.com/stack-vs-heap.html

[SMcNeill]

@BPlus: Yes I did lol. It was all fun and games after a few thousand lines, but about the 10,000 or 20,000 lines, things got serious. I'm glad that I didn't have to edit it after it was made.

@KernelPanic: Yes. It was a matter of running out of memory.

@SMcNeill: "What you have isn't expandable or substainable. You really need to go back to the drawing board and find a better way to automate this process."
--> I think I will have to, because if I hit the limit at 8GB, I'll start disk thrashing from there on out. I see no other choice but to do swapping and even with the SSD to mitigate this, with virtual memory the drive will be thrashing either way.

@SMcNeill: "Where are you getting the idea that STACK SIZE has *anything* at all to do with this program?? It uses no GOTO, no GOSUB. It has no RETURN statements. There's no RECURSIVE FUNCTION calls. There's *NO USE OF STACK* here at all."
--> For reference: https://communities.actian.com/s/article...Stack-Size

Look at your own article:

The two operations applicable to all stacks are:-

Push - Data items are placed at the location pointed to by the stack pointer, and the address in the stack pointer is adjusted by the size of the data item.
Pull or Pop - Data at the current location pointed to by the stack pointer is removed, and the stack pointer is adjusted by the size of the data item.

Now, where are you pushing any data onto a stack pointer? Where are you pulling or popping data from a stack pointer?


The way a stack works is like so:


GOSUB FOO >--- this now pushes the location of where this GOSUB is located in memory onto the stack.


RETURN >--- this now pops up back to that location and removes that entry off the stack.

---

IF x = 1 THEN whatever..... <-- Now, where the program making a leap that it has to return back from here? Where is that jump, that we need to put anything onto the stack??

IF statements don't add to the stack. Those are generally added to via GOSUB, CALL, or recursive routines.

Your program, as you've shared it, has *nothing* to do with stack space.


For reference to illustrate what the issue is for you, let's look at ONE line of code and translate it via QB64PE.

This is the QB64PE code:

Code: (Select All)

If x$ = Chr$(0) + Chr$(1) Then g$ = "foo": Put #1, , g$: g$ = ""

And this is how it gets translated:

Code: (Select All)

S_1:;
if ((qbs_cleanup(qbs_tmp_base,qbs_equal(__STRING_X,(qbs_add(func_chr( 0 ),func_chr( 1 ))))))||new_error){
if(qbevent){evnt(1);if(r)goto S_1;}
do{
qbs_set(__STRING_G,qbs_new_txt_len("foo",3));
qbs_cleanup(qbs_tmp_base,0);
if(!qbevent)break;evnt(1);}while(r);
do{
sub_put2( 1 ,NULL,byte_element((uint64)__STRING_G->chr,__STRING_G->len,byte_element_1),0);
qbs_cleanup(qbs_tmp_base,0);
if(!qbevent)break;evnt(1);}while(r);
do{
qbs_set(__STRING_G,qbs_new_txt_len("",0));
qbs_cleanup(qbs_tmp_base,0);
if(!qbevent)break;evnt(1);}while(r);
}

That one single line of BAS code just got translated into SIXTEEN lines of c code..

And you're wanting to work with 65,536 lines of BAS code. That's a program with over a MILLION lines of c-code which you're tossing into cpp and trying to compile.

A MILLION+ lines of c-code.... Let me stress that once again. OVER A MILLION LINES OF C CODE!!!

That takes memory to then translate those million c-lines into assembler. Then it takes more memory to link it all together. To build it. As I've mentioned, this process takes over 30 GB of RAM to run.

   

   

   

^There's some screenshots I took during the process. As you can see, it starts out with insanely high requirements and then just goes up and up and up from there.

That's not stack space that we're looking at. That's just pure RAM being used and allocated as it tries to translate a MILLION plus lines of c-code into machine code.

You've got to find a better way to do this, or else you're going to have to invest in multiple servers with petrabytes of memory to try and compile 16.7 million lines of BAS code (300 million lines of c-code....). I'm certainly not rich enough to do so, but I wish you the best of luck if you are. I'd love to see a picture of the machine which you build which can run and compile such a massive program!

[SMcNeill]

(To give you an idea to help you compare what you're talking about doing here, Windows itself is about 50 million lines of code.  The way you're writing this program, your EXE is going to end up being bigger than SIX whole versions of the Windows OS...    Does the scope of this help you understand why you're running into limits and running out of memory here?)

[Kernelpanic]

@James - 
IF Lookahead$=CHR$(0) THEN G$="(output)": PUT #2 , , G$: G$=""
IF Lookahead$=CHR$(1) THEN G$="(output)": PUT #2 , , G$: G$=""

I am wondering why this is happening without a loop.


Just a small example with a loop - but of course I do not know how the whole program is structured.

Code: (Select All)

Option _Explicit



Dim As Integer zaehler

Dim As String lookahead





For zaehler = 65 To 84

  lookahead = Chr$(zaehler)

  Print lookahead

Next



End

[Kernelpanic]

Hello, that's gets even better! 

Code: (Select All)

'Uebunmg, Zuweisung von Nummern an Stringvariable - 30. Jan. 2024

'https://qb64phoenix.com/forum/showthread.php?tid=2413



$Console:Only



Option _Explicit



Dim As Integer zaehler

Dim As String lookahead





For zaehler = 65 To 84

  lookahead = Chr$(zaehler)

  Print lookahead

  If Asc(Chr$(zaehler)) Mod 3 = 0 Then

    Print zaehler: Beep

  End If

Next



End

[mdijkens]

If I understand correctly your are bound to the generated rules one per each line in the same format?

I tried the following with 65,795 rules/lines and it maxed out at 4GB during compiling:

Code: (Select All)

Line Input "File to read in:", FILE$
Line Input "File to write out:", FILE2$

Open FILE$ For Binary As #1: 'For reading
Open FILE2$ For Output As #2: Close #2: Open FILE2$ For Binary As #2 ':For write

Do '(Main loop goes here)

  Convert Lookahead$, Chr$(0), "(output)"
  Convert Lookahead$, Chr$(1), "(output)"
  '... the other 254 + 65534 lines ...
  Convert Lookahead$, Chr$(255) + Chr$(254), "(output)"
  Convert Lookahead$, Chr$(255) + Chr$(255), "(output)"
Loop

Sub Convert (Lookahead$, test$, output$)
  If Lookahead$ = test$ Then Put #2, , output$
End Sub

[SMcNeill]

Honestly, this seems to me like you'd just make it a simple text file and then go to it and get this type of information

Now, since I wrote my data as STRING * 20 for those lookup values, all I do is GET the relevant record from the random file.

DIM returnvalue AS STRING * 20
OPEN "Lookup_file.txt" FOR RANDOM AS #1 LEN = 20
GET #1, x * 65536 + y * 256 + z +1, returnvalue
PUT #2, ,_TRIM$(returnvalue)


^That's all there is to it, since my text file looks like:

Code: (Select All)

I like cheese
Rootbeer is good
Whoopie doopie do
....and so on

I like cheese corresponds to record CHR$(0) + CHR$(0) + CHR$(0).
Rootbeer is good corresponds to record CHR$(0) + CHR$(0) + CHR$(1).
Whoopie doopie do  corresponds to record CHR$(0) + CHR$(0) + CHR$(2).
...and so on
            
As for x/y/z, they correspond to  ASC(Lookahead$,1), ASC(Lookahead$,2) , ASC(Lookahead$,3).

A simple random length text file with all the data we could ever want to put in it, instantly retrievable with one simple GET.   What could be any more efficient or simpler than that, with zero looping or millions of IF checks involved?