Tokenizer in QB64

[mdijkens]

I do think the code is very un-QB64 and could be structured way better which prevents issues and makes it much more readable.

- Get rid of old basic DECLARE's, WHILE/WEND's, etc.
- Don't make all variables shared unless you have a good reason; use local variables in functions/subs as much as possible
- Use sensible variablenames
- separate small blocks of code in their own functions
- don't use goto TokExit; it's a sign of bad structure in your function
- structure with FOR, SELECT CASE, etc.

A lot of things can go wrong here. For example in your WHILE p <= Len(src) it is very risky to do p=p+1:ch=mid$(src,p,1) since you don't check if p is now pointing outside ch. And there are more of these issues hidden in the structure of your function

something like this (didn't look at all the details):

Code: (Select All)

Const FALSE = 0, TRUE = Not FALSE
Function tokenizer& (src As String)
  srclen& = Len(src)
  srcpos& = 0
  Do While srcpos& < srclen&
    srcpos& = srcpos& + 1
    char% = Asc(src, srcpos&) ' get char
    Select Case char%
      Case 32 'skip space
      Case 9 'skip tab
      Case 13 'skip cr
        skipline% = FALSE: lineCount = lineCount + 1
      Case 39 'skip commentline
        skipline% = TRUE
      Case 10 ' lf
        skipline% = FALSE: lineCount = lineCount + 1
        tp& = addToken("eol")
      Case 34 'double quote
        If Not skipline% Then
          If dquote& = 0 Then
            dquote& = srcpos&
          Else
            tp& = addToken("from dquote& to srcpos&")
            dquote& = 0
          End If
        End If
      Case 48 To 57 '0 to 9
        If Not skipline% Then tp& = addToken("digit")
      Case tkPLUS 'Plus
        If Not skipline% Then tp& = addToken("tkPlus")
      Case tkMINUS 'Minus
        If Not skipline% Then tp& = addToken("tkMINUS")
      Case tkETCETERA 'all other const tk...
        If Not skipline% Then tp& = addToken("something")
      Case 92 To 125 'all other valid chars?
        If Not skipline% Then tp& = addToken("something")
      Case Else
    End Select
  Loop
  tokenizer& = tp&
End Function

Function addToken& (something$)
  If something$ = "allright" then
    tokens = tokens + 1
    ' add token to array
    addToken& = tokens
  Else
    addToken& = errorcode ' Negative
  End If
End Function

Function tokError (errcode%)
  ' show and/or handle error
End Function