Tokenizer in QB64

[bplus]

OK I asked at Aurel's place I ask here I am debugging a QB64 version of tokenizer and need to know the tokens here:

@Aurel what are the 4 tokens in this string:

Code: (Select All)

Dim test As String: test = "func tokenizer in QB64"

[mnrvovrfc]

For me is main problem null terminated string or chr$(0) or this ""
And such a thing is solved or work in other Basic dialects
so it should be fixed in QB64 to
because produce ILLEGAL FUNCTION CALL runtime error

The ASC("") problem, sadly was created by Microsoft with QuickBASIC and QBasic. QB64 aims always to be compatible with those two products and therefore this won't be changed. QB64 doesn't care about "other" BASIC dialects.

The workaround was presented which was to create a wrapper function that catches the empty string and then the programmer is given the decision to do something about it. It's not any different from dividing by zero with integers, having to create a function for it only to catch zero as divisor.

[aurel]

. It's not any different from dividing by zero with integers,

..and this is just your speculation

of course it is different

[mnrvovrfc]

What in the hey are you talking about? "My" speculation?!

In Microsoft-influenced BASIC, the ASC() function was what it always was, crashing the program when it gets the empty string. This was a chosen behavior. Dividing by zero, however, is not, it comes from the rules of mathematics. Yet it also crashes the program for another operation which is not allowed. Regardless of where it comes from, there has to be a way to buffer those cases, because stopping a program run unexpectedly is not acceptable in many cases.

Do you really want to fix the ASC("") problem in QB64? Then fork the source code for the whole QB64 and program it in. Otherwise it will not be modified because many people expect that to still return an "Illegal Function Call", as one hint out of many that QB64 has to emulate QuickBASIC and QBasic most of all. Not Freebasic.

The "ASC2()" sucks indeed, having to use a wrapper, but otherwise a preprocessor will have to be built into QB64, or have a separate program that works with QB64, which allows macros, which allows original BASIC statement and function names redefined at the preprocessor level.

[bplus]

@mnrvovfrc Forgive Aurel, English is his 2nd language pretty good but he thinks he is perfectly understandable and likewise understanding of what you say. PLUS if it is different than Oxygen Basic then it is WRONG! LOL
You gotta laugh because if you take it serious then Aurel is the devil!

Also, he also forms strong bias for or against people and does not judge their arguments rationally.
That I find not so easy to laugh at.

@Aurel so in this:

Code: (Select All)

Dim test As String: test = "func tokenizer in QB64"

The tokenizer is supposed to count 4 words:
1. func
2. tokenizer
3. in
4. QB64

It doesn't save words it just counts them? IF THEY ARE NOT inside a Quote or Comment.
Correct?

It might not store the words but saves their start posiions? in line and column or p = position in the whole program = one string lines separated by chr$(10) + chr$(13)

Otherwise I need to know what 4 things it is supposed to be counting.

[aurel]

Mark
don't worry i understand mnrvovrfc perfectly
BUT he don't understand that ASC("") have  value and that value is null -zero

Do you really want to fix the ASC("") problem in QB64? Then fork the source code for the whole QB64 and program it in.

 Really ..that is your answer ...you must be very smart ..for sure

and where is written that QB64 must be clone of quickBasic ..nowhere
btw ..chr(0) is used in FileDialog api calls like separator :

check this WinApi code:

Code: (Select All)

string dir="",filter,title
string sep=chr(0)
'filter = "All Files"+sep+"*.*"+sep"Text files"+sep+"*.txt"+ sep
filter = "All files "+sep+"*.*"+sep+"micro(A) files "+sep+"*.bas"
title="Open File... "
fName = FileDialog(dir,filter,title,0,0,"bas")

[SMcNeill]

CHR$(0) =/= ""

[mnrvovrfc]

Really ..that is your answer ...you must be very smart ..for sure

and where is written that QB64 must be clone of quickBasic ..nowhere

Aurel I'm not one of your opponents. Not here to correct only you. But remember that QB64 was originally written by somebody who loved QBasic, the interpreter that came with MS-DOS v5, which was a cut-down version of another commercial Microsoft product, QuickBASIC v4.5. Both of them, QBasic and QuickBASIC, return an Illegal Function Call at ASC(""). Many people expect that. You are in the minority expecting it to behave like Visual Basic and Freebasic, or whatever was fit much more for Windows than for MS-DOS.

The people who expect QB64 to behave like QBasic and QuickBASIC do not care as well for compatibility with later-arriving BASIC dialects such as Visual Basic, which begins to deviate from what they have known and have become comfortable with. Many people, like me, also expect that QBasic and QuickBASIC were text-mode applications having to go along with the limits that MS-DOS imposed. They are not bothered by inability to deal with WinAPI, GDI and that other junk, in exchange for the old interrupts and I/O ports. If somebody wanted the ghost of a chance to access all RAM or something else about performance of Intel 80286 and later processors built, capable of running WindowsNT and later, back then when the World Series Championship was two years in Canada (before the stupid strike in MLB), we would have had to switch to BASIC PDS v7.1 or maybe VBDOS. Or just wait a few years until Visual Basic for Windows matured.

Another good example of it is what happened to "RETURN" keyword. M$ decided to change it in 2005 or so to match C, Python and the rest of the world of programming. Freebasic joined a bit later, throwing down a bunch of people into "lang qb" mode if they wanted to continue using "RETURN" to relate to "GOSUB". "Using GOSUB and RETURN requires three bytes of code generated by QuickBASIC compiler," Ethan Winer said in his book LOL I still remember.

Ehh... but I think this about "RETURN" causes more division.

Anyway Steve made "his own" version of QB64. Or like two or three others LOL, like the one still based on the old SDL for Windows. The code will have to be forked in order to change at least one behavior that is not perceived as correct, or that is considered convenient. Because not even the "official" QB64 will serve ASC("") without interrupting program flow and returning an error.

[aurel]

They are not bothered by inability to deal with WinAPI, GDI and that other junk

 how i miss "the rumbler" .."smart" moment  ...junk?
 it is not junk then base of most GUI programs on Windows
u use C or C++ or C# ...