qb64pe-json - A library for JSON parsing and creation

[DSMan195276]

qb64pe-json
https://github.com/mkilgore/qb64pe-json

qb64pe-json is a library for parsing and creating JSON strings. Currently it is hosted and developed on GitHub, and I just released the beta v0.1.0 version. Attached to this post is the v0.1.0 release, and also the examples currently present in the repository. Below is the README from the repository:

---

qb64pe-json

qb64pe-json is a JSON parsing and creation library for QB64-PE. Given a string containing JSON, it can convert it into a Json object which can then be queried to retrieve the values in the JSON. Additionally it contains a variety of `JsonTokenCreate*` functions which can be used to build up a Json object, which can then be rendered into a String version of that JSON.

Please see json.bi for more in-depth documentation around the API. Additionally see the examples/ for code samples of the API in use.

To use the API, download a release version and place the `json.bi` and `json.bm` files into your project. Then reference the two files via `'$include:`.

Overall Design

qb64pe-json works by turning a JSON structure into a collection of "tokens", which are kept internal to a `Json` object. Tokens are allocated as needed, and token IDs are returned from several Functions. You can then pass a token ID into many of the APIs to interact with the token, such as get its value, get its children, etc. Valid token IDs are always positive.

The main Type in qb64pe-json is the `Json` Type. After declaring one, you need to pass it to `JsonInit` to initialize it, and eventually pass it to `JsonClear` to release it. Not passing a `Json` object to `JsonClear` will result in memory leaks.

There are four types of tokens - Objects, Arrays, Keys, and Values. Values are then split up into several "primitive" types a value can be, which are strings, numbers, bools, and `null`. A typical token structure looks something like this:

This is the original JSON passed to `JsonParse()`:

Code: (Select All)

{
    "key1": {
        "key2": 20,
        "key3": [ true, "string", null ],
    },
    "key4": 50
}

This is the resulting token structure:

Code: (Select All)

Object (1)
  - Key (value = "key1") (2)
    - Object (3)
      - Key (value = "key2") (4)
        - Value (type = number, value = 20) (5)
      - Key (value = "key3") (6)
        - Array (7)
          - Value (type = bool, value = true) (8)
          - Value (type = string, value = "string") (9)
          - Value (type = null) (10)
  - Key (value = "key4") (11)
    - Value (type = number, value = 50) (12)

The numbers after each token signify its ID, which is what will be returned by the API when referring to that particular token. The typical way to interact with this structure is through `JsonQuery()`, which takes a query string and returns the token identified by it. For example, if you do `JsonQuery(json, "key1.key2")`, it will return 5, which is the token ID for the "20" Value token. You can then pass the token ID from that query to `JsonTokenGetValueInteger(token)` to retrieve the actual value 20 as an integer.

`JsonQuery(json, "key2.key3")` returns 7, the token ID for the Array. With this array you can make use of `JsonTokenTotalChildren(array)` and pass it the token ID to retrieve the number of children (entries) in that array. You can then additionally make use of `JsonTokenGetChild(array, index)` to get the token ID of each child of the array. Note the indexes into the array start at zero, so `JsonTokenGetChild(array, 0)` would return 8, the bool in the array since it is the first entry. `JsonTokenGetChild(array, 2)` would return 10, the last entry in the array. You can of course then pass those token IDs to the various `JsonTokenGetValue` functions to retrieve their values.

If you have a token and need to know what it is, you can use `JsonTokenGetType(token)` to retrieve a `JSONTOK_TYPE_*` value indicating its type. If its type is `JSONTOK_TYPE_VALUE`, then you can additionally use `JsonTokenGetPrimType(token)` to get its primitive type, in the form of a `JSONTOK_PRIM_*` value.

`Json` objects contain the concept of a "RootToken", which is simply the token of the base of the entire JSON structure. Several APIs start at the RootToken automatically, such as `JsonQuery()`, `JsonRender()`, etc. However all APIs offer an option to take a token directly to start with, ignoring the RootToken. This is powerful as it allows you to treat smaller subtrees of the entire structure as their own Json structure. For example in the above structure, you can use `JsonQueryFrom(3, "key2")` to do a query starting from the Object with index 3, completely ignoring the Object it's contained in.

Errors are reported from qb64pe-json via the global `JsonHadError` and `JsonError` variables. `JsonHadError` is zero (`JSON_ERR_Success`) when a function was successful, and a negative value when an error occurs. The negative values correspond to the `JSON_ERR_*` constants, and indicate the specific kind of error that occurred. `JsonError` will contain a human-readable string version of the error.

JSON Creation

In addition to parsing JSON, qb64pe-json allows you to create the Json structure yourself and then turn it into a JSON string (for storing or sending elsewhere). This is done by using the `JsonTokenCreate*()` functions. These functions create a new token and return its token ID. You can then make use of this token ID to add it to other tokens and build the Json structure. Objects and Arrays can have entries added to them via `JsonTokenArrayAdd` and `JsonTokenObjectAdd`.

Once you have built your Json structure, you can optionally use `JsonSetRootToken` to set the RootToken of the Json object to be the root of your created structure. Then, you can use `JsonRender$()` to produce a JSON string version of that structure.

`JsonRenderFormatted$()` gives you more control over the rendering. Currently, it allows you to include indentation in the result, which makes it easier to read.

[mnrvovrfc]

Thank you for this. It will be useful in the future.

[grymmjack]

So great! Will try it soon.

Question. When dealing with large JSON data it would be great if the lib could support splat operator as part of search function.

For example with this JSON:

Code: (Select All)

{
    "shapes": [
         "square": {
              "name": "Simple Square",
              "sides": 4
          },
         "triangle": {
              "name": "Just a Triangle",
              "sides": 3
          },
         "line": {
              "name": "Line is just 2 points",
              "sides": 0,
              "points": 2
          },
          "circle": {
              "name": "Here we have 1 point for origin of Circle",
              "sides": 0,
              "points": 1,
              "radius": 100
          }
    ]

}

Then you could use like:

Code: (Select All)

shapes[*].name

to get the names back as an array of strings (bad example but maybe you get my meaning?)


JSON is cool until you have to walk anonymous stuff 10 levels deep to get at what you really care about

Thanks for your lib!

[DSMan195276]

Then you could use like:

Code: (Select All)

shapes[*].name

to get the names back as an array of strings (bad example but maybe you get my meaning?)

Yes I know exactly what you're talking about. I definitely consider it something that would be nice to add somehow, the underlying issues are that the query logic gets a lot more complicated, and the resulting token will end up being something that does not exist in the original JSON structure.

For that second part, currently queries only return tokens that already exist, but your query would require making a new array token for the result. That's doable (just call `JsonTokenCreateArray()` and fill it in) but then what's the lifetime of that token and how does it get free'd? We could just leak it until `JsonClear` is called but that's a bit ugly, and also means `Json` objects would keep growing in size as you query them.

That said right now you can totally write your own helper functions to do this sort of transformation and manage the extra tokens yourself (this code is completely untested):

Code: (Select All)

resultNames& = splat(j, JsonQuery(j, "shapes"), "name")
resultSides& = splat(j, JsonQuery(j, "shapes"), "sides") 

Function splat&(j As Json, arr As Long, query As String)
    Dim newArr As Long, child As Long, count As Long, i As Long
    newArr = JsonTokenCreateArray(j)

    count = JsonTokenTotalChildren(j, arr)
    For i = 0 to count - 1
        JsonTokenArrayAdd j, newArr, JsonQueryFrom(j, JsonTokenGetChild(j, arr, i), query)
    Next

   splat& = newArr
End Function

That array then either sticks around until you do a `JsonClear`, or could be free'd manually before that with `JsonTokenFreeShallow`. And to be clear, that's definitely not as nice as just being able to pass it to a `JsonQuery()` and get a result, but writing some helper functions like that can be a pretty effective way to simplify your parsing, especially if there's some common patterns in your JSON.

Note that the above code still doesn't quite work for your example because the `"name"` is inside of other objects with different keys. Also, you can't have direct key-value pairs inside an array It would work for something like this though (keys removed):

Code: (Select All)

{
    "shapes": [
         {
              "name": "Simple Square",
              "sides": 4
         },
         {
              "name": "Just a Triangle",
              "sides": 3
         },
         {
              "name": "Line is just 2 points",
              "sides": 0,
              "points": 2
         },
         {
              "name": "Here we have 1 point for origin of Circle",
              "sides": 0,
              "points": 1,
              "radius": 100
         }
    ]
}

[mnrvovrfc]

The JSON is an ordinary text file, right? So just create a subprogram that returns the starting and end lines of braces for the largest entry. Then nest this particular search, narrowing down to a wanted field.

Something like:
SearchField json$, field$, value$, instart, inend, outstart, outend

Then use the "outstart" and "outend" as the new input range for a different field which is the child of the previous value of "field$". Like in the example above, first call this subprogram with "shapes" as "field$" and "value$" set to null string, then call it with "name" as "field$", and "value$" to something expected as value of "name" field.

It could just be the first call to this imaginary subprogram and capture whatever value is associated with "name" field, building a string array out of it. The important thing is to limit which big "record" to search into.

It would take more work out of that to find out which other fields exist which are cohorts of the sought one, and then report on all of them. Report it in a sensible way too. Difficult, but not impossible. BASIC doesn't have regular expressions but the string functions are quite good, have been workhorses for almost four centuries so far.

[DSMan195276]

@mnrvovrfc to clarify, my library can already do all that parsing, `JsonQuery()` can be used to find a single inner part of the Json structure. `JsonQueryFrom()` does what you're describing and allows you to start a query from an inner part of the Json structure, effectively doing the same as your `SearchField` but without having to deal with the strings and start/end locations.

grymmjack's example is special because it's result is not just an inner part of the JSON structure, but a new structure containing parts of the old structure.