Tokenizer in QB64

[aurel]

I am trying to modify my tokenizer written in FB to QB64
and i am getting error ..what i am doing wrong ?

Code: (Select All)

'tokenizer in QB (fb) by Aurel

'INT startTime ,endTime: float procTime  ' GetTickCount -timer init
declare function tokenizer( src as string) as integer
declare function run_tokenizer(inputCode as string) as integer

const tkNULL=0, tkPLUS=1, tkMINUS=2, tkMULTI=3, tkDIVIDE=4
const tkCOLON=5, tkCOMMA=6, tkLPAREN=7, tkRPAREN=8, tkLBRACKET=9, tkRBRACKET=10
const tkIDENT = 11 , tkNUMBER = 12 , tkQSTRING = 13, tkCOMMAND =14 ,tkEOL = 15
const tkEQUAL = 16, tkMORE = 17, tkLESS = 18, tkAND = 19, tkOR = 20, tkNOT = 21
const tkHASH=22 , tkSSTR=23, tkMOD=24 , tkSEMI=25, tkDOT=26, tkLBRACE=27, tkRBRACE=28
const  tkQUEST=29, tkMONKEY=30 , tkBACKSLAH=31, tkPOWUP=32 ,tkAPOSTR=33 , tkTILDA=34

Dim shared tokList(1024)  As string                       'token array
Dim shared typList(1024)  As integer                      'token type array
Dim shared p              As Long : p=1
Dim shared start          as Long : start = 1
Dim shared tp             as long
Dim shared tn             as long
Dim shared n              as long
Dim shared ltp            as long  : lpt = 1
Dim shared nTokens    As long                            'nTokens -> number of tokens
Dim shared lineCount As integer
Dim shared Lpar      as integer
Dim shared Rpar      as integer
Dim shared Lbrk      as integer
Dim shared Rbrk      as integer
Dim shared tokerr    as integer
Dim shared codeLen   as integer
Dim shared code      As String
Dim shared chs       As String
Dim shared tch       As String
Dim shared tk        As String
Dim shared crlf      As String
Dim shared bf        As String
Dim shared ntk       As String
crlf = chr$(13) + chr$(10)
'test string .......................................
Dim test as string  : test = "func tokenizer in QB64"
'...................................................

'call fn tokenizer()
call tokenizer(test)



' *** MAIN TOKENIZER FUNCTION ***
FUNCTION tokenizer& (src as string)
print "tokenizer run:" + src
lineCount=0:ltp=start : nTokens = 0

tokenizer& = 0
END FUNCTION




do

loop until multikey(27)

[bplus]

You can only "Call" Subs. Your Tokenizer is defined as a Function.

Plus all sub and function definitions follow the main code.

Skip the Declares unless using outside QB64 Language code.

[aurel]

OK
so if i understand it

1.CALL only work with SUB.s --right?

2.skip declares you mean DECLARE FUNCTION ?

ok i will  try again

[bplus]

Yes I mean Declare 's Sub and or Function, it's OK but just another place that can screw you up.

Code: (Select All)

'tokenizer in QB (fb) by Aurel
Const tkNULL = 0, tkPLUS = 1, tkMINUS = 2, tkMULTI = 3, tkDIVIDE = 4
Const tkCOLON = 5, tkCOMMA = 6, tkLPAREN = 7, tkRPAREN = 8, tkLBRACKET = 9, tkRBRACKET = 10
Const tkIDENT = 11, tkNUMBER = 12, tkQSTRING = 13, tkCOMMAND = 14, tkEOL = 15
Const tkEQUAL = 16, tkMORE = 17, tkLESS = 18, tkAND = 19, tkOR = 20, tkNOT = 21
Const tkHASH = 22, tkSSTR = 23, tkMOD = 24, tkSEMI = 25, tkDOT = 26, tkLBRACE = 27, tkRBRACE = 28
Const tkQUEST = 29, tkMONKEY = 30, tkBACKSLAH = 31, tkPOWUP = 32, tkAPOSTR = 33, tkTILDA = 34

Dim Shared tokList(1024) As String 'token array
Dim Shared typList(1024) As Integer 'token type array
Dim Shared p As Long: p = 1
Dim Shared start As Long: start = 1
Dim Shared tp As Long
Dim Shared tn As Long
Dim Shared n As Long
Dim Shared ltp As Long: lpt = 1
Dim Shared nTokens As Long 'nTokens -> number of tokens
Dim Shared lineCount As Integer
Dim Shared Lpar As Integer
Dim Shared Rpar As Integer
Dim Shared Lbrk As Integer
Dim Shared Rbrk As Integer
Dim Shared tokerr As Integer
Dim Shared codeLen As Integer
Dim Shared code As String
Dim Shared chs As String
Dim Shared tch As String
Dim Shared tk As String
Dim Shared crlf As String
Dim Shared bf As String
Dim Shared ntk As String
crlf = Chr$(13) + Chr$(10)
'test string .......................................
Dim test As String: test = "func tokenizer in QB64"
'...................................................

'call fn tokenizer()
temp& = tokenizer&(test)


Do

Loop Until _KeyDown(27)

' *** MAIN TOKENIZER FUNCTION ***
Function tokenizer& (src As String)
    Print "tokenizer run:" + src
    lineCount = 0: ltp = start: nTokens = 0

    tokenizer& = 0
End Function

Can't use As Type for Function definitions need to use Type suffixes on the Function name.

[aurel]

ok Mark

but why i get syntax error with simple function call here:

'call function tokenizer()

tokenizer(test)

[aurel]

and this is exactly what Ed do in his subsetQB

Code: (Select All)

'call function tokenizer()

tokenizer&(test)

and i again get syntax error...why?

[bplus]

You defined Tokenizer as a Function.

Code: (Select All)

' *** MAIN TOKENIZER FUNCTION ***
Function tokenizer& (src As String)
    Print "tokenizer run:" + src
    lineCount = 0: ltp = start: nTokens = 0

    tokenizer& = 0
End Function

I added the & to make it return Long Type numbers.

Again Call is for Subs NOT Functions. BTW, Call is also not needed its old QB way, just say SubName and arguments

Best way to "call" a Sub:
MySub MyArgument1, MyArgument2,...

Throw out CALL just extra typing plus you have to put () around arguments If you use Call.

[bplus]

and this is exactly what Ed do in his subsetQB

Code: (Select All)

'call function tokenizer()

tokenizer&(test)

and i again get syntax error...why?

Ed did something else, I don't have his code handy. I am just telling you what you are setting up wrong.

[bplus]

Reminder Functions are used on the Right side of = except eg Print MyFunction where MyFunction call is argument to a Sub call.

Test& = MyFunction&(myArg1, myArg2,...)  ' <<< Basic Basic!

[aurel]

I don't know how he did something else when both declare functions are same
look this :

his:
Declare function accept&(s as string)

my :
declare function tokenizer&( src as string)


so it is same ?


and i do as string variable which is of course declared with Dim

'call function tokenizer()

tokenizer&(test)

but he called function using quoted string
like :
accept&("text")

here is full code again:

Code: (Select All)

'tokenizer in QB64pe (fb) by Aurel

declare function tokenizer&( src as string)
'declare function run_tokenizer(inputCode as string) as integer

const tkNULL=0, tkPLUS=1, tkMINUS=2, tkMULTI=3, tkDIVIDE=4
const tkCOLON=5, tkCOMMA=6, tkLPAREN=7, tkRPAREN=8, tkLBRACKET=9, tkRBRACKET=10
const tkIDENT = 11 , tkNUMBER = 12 , tkQSTRING = 13, tkCOMMAND =14 ,tkEOL = 15
const tkEQUAL = 16, tkMORE = 17, tkLESS = 18, tkAND = 19, tkOR = 20, tkNOT = 21
const tkHASH=22 , tkSSTR=23, tkMOD=24 , tkSEMI=25, tkDOT=26, tkLBRACE=27, tkRBRACE=28
const  tkQUEST=29, tkMONKEY=30 , tkBACKSLAH=31, tkPOWUP=32 ,tkAPOSTR=33 , tkTILDA=34

Dim shared tokList(1024)  As string                       'token array
Dim shared typList(1024)  As integer                      'token type array
Dim shared p              As Long : p=1
Dim shared start          as Long : start = 1
Dim shared tp             as long
Dim shared tn             as long
Dim shared n              as long
Dim shared ltp            as long  : lpt = 1
Dim shared nTokens    As long                            'nTokens -> number of tokens
Dim shared lineCount As integer
Dim shared Lpar      as integer
Dim shared Rpar      as integer
Dim shared Lbrk      as integer
Dim shared Rbrk      as integer
Dim shared tokerr    as integer
Dim shared codeLen   as integer
Dim shared code      As String
Dim shared chs       As String
Dim shared tch       As String
Dim shared tk        As String
Dim shared crlf      As String
Dim shared bf        As String
Dim shared ntk       As String
crlf = chr$(13) + chr$(10)
'test string .......................................
Dim test as string  : test = "func tokenizer in QB64"
'...................................................

'call function tokenizer()
tokenizer&(test)



' *** MAIN TOKENIZER FUNCTION ***
FUNCTION tokenizer& (src as string)
print "tokenizer run:" + src
lineCount=0:ltp=start : nTokens = 0

tokenizer& = 0
END FUNCTION




do

loop until multikey(27)