Tokenizer in QB64 - Printable Version

Tokenizer in QB64 - Printable Version

+- QB64 Phoenix Edition (https://qb64phoenix.com/forum)
+-- Forum: QB64 Rising (https://qb64phoenix.com/forum/forumdisplay.php?fid=1)
+--- Forum: Code and Stuff (https://qb64phoenix.com/forum/forumdisplay.php?fid=3)
+---- Forum: Programs (https://qb64phoenix.com/forum/forumdisplay.php?fid=7)
+---- Thread: Tokenizer in QB64 (/showthread.php?tid=1502)

Pages: 1 2 3 4

Tokenizer in QB64 - aurel - 02-25-2023

I am trying to modify my tokenizer written in FB to QB64
and i am getting error ..what i am doing wrong ?

Code: (Select All)
'tokenizer in QB (fb) by Aurel

'INT startTime ,endTime: float procTime  ' GetTickCount -timer init

declare function tokenizer( src as string) as integer

declare function run_tokenizer(inputCode as string) as integer

const tkNULL=0, tkPLUS=1, tkMINUS=2, tkMULTI=3, tkDIVIDE=4

const tkCOLON=5, tkCOMMA=6, tkLPAREN=7, tkRPAREN=8, tkLBRACKET=9, tkRBRACKET=10

const tkIDENT = 11 , tkNUMBER = 12 , tkQSTRING = 13, tkCOMMAND =14 ,tkEOL = 15

const tkEQUAL = 16, tkMORE = 17, tkLESS = 18, tkAND = 19, tkOR = 20, tkNOT = 21

const tkHASH=22 , tkSSTR=23, tkMOD=24 , tkSEMI=25, tkDOT=26, tkLBRACE=27, tkRBRACE=28

const  tkQUEST=29, tkMONKEY=30 , tkBACKSLAH=31, tkPOWUP=32 ,tkAPOSTR=33 , tkTILDA=34

Dim shared tokList(1024)  As string                       'token array

Dim shared typList(1024)  As integer                      'token type array

Dim shared p              As Long : p=1

Dim shared start          as Long : start = 1

Dim shared tp             as long

Dim shared tn             as long

Dim shared n              as long 

Dim shared ltp            as long  : lpt = 1

Dim shared nTokens    As long                            'nTokens -> number of tokens

Dim shared lineCount As integer

Dim shared Lpar      as integer

Dim shared Rpar      as integer

Dim shared Lbrk      as integer

Dim shared Rbrk      as integer

Dim shared tokerr    as integer

Dim shared codeLen   as integer

Dim shared code      As String 

Dim shared chs       As String

Dim shared tch       As String

Dim shared tk        As String

Dim shared crlf      As String 

Dim shared bf        As String

Dim shared ntk       As String

crlf = chr$(13) + chr$(10)

'test string .......................................

Dim test as string  : test = "func tokenizer in QB64"

'...................................................

'call fn tokenizer()

call tokenizer(test)

' *** MAIN TOKENIZER FUNCTION ***

FUNCTION tokenizer& (src as string) 

print "tokenizer run:" + src

lineCount=0:ltp=start : nTokens = 0

tokenizer& = 0

END FUNCTION

do

loop until multikey(27)

RE: Tokenizer in QB64 - bplus - 02-25-2023

You can only "Call" Subs. Your Tokenizer is defined as a Function.

Plus all sub and function definitions follow the main code.

Skip the Declares unless using outside QB64 Language code.

RE: Tokenizer in QB64 - aurel - 02-25-2023

OK
so if i understand it

1.CALL only work with SUB.s --right?

2.skip declares you mean DECLARE FUNCTION ?

ok i will try again Wink

RE: Tokenizer in QB64 - bplus - 02-25-2023

Yes I mean Declare 's Sub and or Function, it's OK but just another place that can screw you up.

Code: (Select All)
'tokenizer in QB (fb) by Aurel

Const tkNULL = 0, tkPLUS = 1, tkMINUS = 2, tkMULTI = 3, tkDIVIDE = 4

Const tkCOLON = 5, tkCOMMA = 6, tkLPAREN = 7, tkRPAREN = 8, tkLBRACKET = 9, tkRBRACKET = 10

Const tkIDENT = 11, tkNUMBER = 12, tkQSTRING = 13, tkCOMMAND = 14, tkEOL = 15

Const tkEQUAL = 16, tkMORE = 17, tkLESS = 18, tkAND = 19, tkOR = 20, tkNOT = 21

Const tkHASH = 22, tkSSTR = 23, tkMOD = 24, tkSEMI = 25, tkDOT = 26, tkLBRACE = 27, tkRBRACE = 28

Const tkQUEST = 29, tkMONKEY = 30, tkBACKSLAH = 31, tkPOWUP = 32, tkAPOSTR = 33, tkTILDA = 34

Dim Shared tokList(1024) As String 'token array

Dim Shared typList(1024) As Integer 'token type array

Dim Shared p As Long: p = 1

Dim Shared start As Long: start = 1

Dim Shared tp As Long

Dim Shared tn As Long

Dim Shared n As Long

Dim Shared ltp As Long: lpt = 1

Dim Shared nTokens As Long 'nTokens -> number of tokens

Dim Shared lineCount As Integer

Dim Shared Lpar As Integer

Dim Shared Rpar As Integer

Dim Shared Lbrk As Integer

Dim Shared Rbrk As Integer

Dim Shared tokerr As Integer

Dim Shared codeLen As Integer

Dim Shared code As String

Dim Shared chs As String

Dim Shared tch As String

Dim Shared tk As String

Dim Shared crlf As String

Dim Shared bf As String

Dim Shared ntk As String

crlf = Chr$(13) + Chr$(10)

'test string .......................................

Dim test As String: test = "func tokenizer in QB64"

'...................................................

'call fn tokenizer()

temp& = tokenizer&(test)

Do

Loop Until _KeyDown(27)

' *** MAIN TOKENIZER FUNCTION ***

Function tokenizer& (src As String)

    Print "tokenizer run:" + src

    lineCount = 0: ltp = start: nTokens = 0

    tokenizer& = 0

End Function

Can't use As Type for Function definitions need to use Type suffixes on the Function name.

RE: Tokenizer in QB64 - aurel - 02-25-2023

ok Mark

but why i get syntax error with simple function call here:

'call function tokenizer()

tokenizer(test)

RE: Tokenizer in QB64 - aurel - 02-25-2023

and this is exactly what Ed do in his subsetQB

Code: (Select All)
'call function tokenizer()

tokenizer&(test)

and i again get syntax error...why?

RE: Tokenizer in QB64 - bplus - 02-25-2023

You defined Tokenizer as a Function.

Code: (Select All)
' *** MAIN TOKENIZER FUNCTION ***

Function tokenizer& (src As String)

    Print "tokenizer run:" + src

    lineCount = 0: ltp = start: nTokens = 0

    tokenizer& = 0

End Function

I added the & to make it return Long Type numbers.

Again Call is for Subs NOT Functions. BTW, Call is also not needed its old QB way, just say SubName and arguments

Best way to "call" a Sub:
MySub MyArgument1, MyArgument2,...

Throw out CALL just extra typing plus you have to put () around arguments If you use Call.

RE: Tokenizer in QB64 - bplus - 02-25-2023

(02-25-2023, 03:48 PM)aurel Wrote: and this is exactly what Ed do in his subsetQB

Code: (Select All)
'call function tokenizer() tokenizer&(test)
and i again get syntax error...why?

Ed did something else, I don't have his code handy. I am just telling you what you are setting up wrong.

RE: Tokenizer in QB64 - bplus - 02-25-2023

Reminder Functions are used on the Right side of = except eg Print MyFunction where MyFunction call is argument to a Sub call.

Test& = MyFunction&(myArg1, myArg2,...) ' <<< Basic Basic!

RE: Tokenizer in QB64 - aurel - 02-25-2023

I don't know how he did something else when both declare functions are same
look this :

his:
Declare function accept&(s as string)

my :
declare function tokenizer&( src as string)

so it is same ?

and i do as string variable which is of course declared with Dim

'call function tokenizer()

tokenizer&(test)

but he called function using quoted string
like :
accept&("text")

here is full code again:

Code: (Select All)
'tokenizer in QB64pe (fb) by Aurel

declare function tokenizer&( src as string) 

'declare function run_tokenizer(inputCode as string) as integer

const tkNULL=0, tkPLUS=1, tkMINUS=2, tkMULTI=3, tkDIVIDE=4

const tkCOLON=5, tkCOMMA=6, tkLPAREN=7, tkRPAREN=8, tkLBRACKET=9, tkRBRACKET=10

const tkIDENT = 11 , tkNUMBER = 12 , tkQSTRING = 13, tkCOMMAND =14 ,tkEOL = 15

const tkEQUAL = 16, tkMORE = 17, tkLESS = 18, tkAND = 19, tkOR = 20, tkNOT = 21

const tkHASH=22 , tkSSTR=23, tkMOD=24 , tkSEMI=25, tkDOT=26, tkLBRACE=27, tkRBRACE=28

const  tkQUEST=29, tkMONKEY=30 , tkBACKSLAH=31, tkPOWUP=32 ,tkAPOSTR=33 , tkTILDA=34

Dim shared tokList(1024)  As string                       'token array

Dim shared typList(1024)  As integer                      'token type array

Dim shared p              As Long : p=1

Dim shared start          as Long : start = 1

Dim shared tp             as long

Dim shared tn             as long

Dim shared n              as long 

Dim shared ltp            as long  : lpt = 1

Dim shared nTokens    As long                            'nTokens -> number of tokens

Dim shared lineCount As integer

Dim shared Lpar      as integer

Dim shared Rpar      as integer

Dim shared Lbrk      as integer

Dim shared Rbrk      as integer

Dim shared tokerr    as integer

Dim shared codeLen   as integer

Dim shared code      As String 

Dim shared chs       As String

Dim shared tch       As String

Dim shared tk        As String

Dim shared crlf      As String 

Dim shared bf        As String

Dim shared ntk       As String

crlf = chr$(13) + chr$(10)

'test string .......................................

Dim test as string  : test = "func tokenizer in QB64"

'...................................................

'call function tokenizer()

tokenizer&(test)

' *** MAIN TOKENIZER FUNCTION ***

FUNCTION tokenizer& (src as string) 

print "tokenizer run:" + src

lineCount=0:ltp=start : nTokens = 0

tokenizer& = 0

END FUNCTION

do

loop until multikey(27)