JSON library

[Lartu]

We all love JSON files and JSON data. It would be nice to have a library able to parse those files into a useful data structure (or data structure + statements).

[Lartu]

The library should both be able to load and save JSON files.

[saper]

I wonder if we should have something nice to describe JSON in the data section to make it really easy to use, some kind of multicontainer.

Wild imagination based on https://cloud.google.com/translate/docs/advanced/discovering-supported-languages-v3

data:
     supported_languages has expected key "languages"
                is list containing "language" 
                      has expected key "languageCode" is text
                      has expected key "supportSource" is boolean 
                      has expected key "targetSource" is boolean

This way we could say what kind of keys we expect in the JSON file (otherwise there
should be an error). list containing could introduce some object that is given a name here (language). I'd prefer not to use confusing name 'object' at all, maybe just 'has [expected] key' as an object indicator.

Also, do we know how to map JSON data types? JSON numbers are weird...

[Lartu]

I do really like this idea, @saper! But at the moment I feel it could be hellish to implement. Maybe, however, we could settle for something in the middle between that and a map.

One could declare the statement

<json_name> HAS EXPECTED KEY <key name> IN KEY <parent key>

to add expected keys to a structure (a map of maps) that describes how a json should look. Then we could declare something like

LOAD <json_data> INTO <json_name>

And have that function check first if the json_data conforms to the structure specified using the previous statement.

This could be fun and not so hard to implement and would be entirely coded in LDPL, so there would've no need to modify the compiler.

[saper]

Can current maps handle different values? I was under impression (reading the docs...) that maps can have either text on number values... (if that is correct I am not sure map name is good...)

[Lartu]

Yes, maps can only handle either text or number values, as do maps in typed languages (like C++, at least as far as I know). But LDPL supports automatic type conversion between text and numbers. For example:

data:
result is number
foo is text

procedure:
store "128" in foo
in result solve 1 + "5" * foo
display result crlf

That actually works and displays 641. So if you wanted to access a numeric value in your JSON, you could use it just like that: myJson:"key":"otherkey":"textvalue that actually holds a number".

[Lartu]

Sorry, did you by any chance find any details about the automatic type conversion in the docs when reading them? I don't really know if that's documented or not, and if it's not that would be a major slip-up from my part; that feature has been available from over a year now haha. I will check that asap.

[xvxx]

I think there are two issues here: parsing JSON and validating JSON.

As far as a parser goes, I think it will be tough to create one that is satisfactory without changing the compiler. Both of these lists are valid JSON, according to https://jsonlint.com/:

• [1, true, "three"]
• [{"name": "Bilbo", "age": 111}, {"name": "Gandalf", "age": "Very Old"}, {"name": "Sauron", "age": null}]

I don't know how we'd express either with native LDPL data structures today. Maybe you just parse every value as Text and let the developer decide how to handle numbers? With a special case for true, false, and null? eg true becomes "JSONTrue" in LDPL or something? Even then, the compiler needs to know the type of multicontainers ahead of time and it just won't be there.

What most typed languages seem to do is return JSValue types, where the JSValue is capable of producing a valid string, number, or whatever in the host language. So in reality you'd get a List of JSValue structure from the first example above, and various statements/methods to work with it. The second example would maybe produce the List of JSObject type. Or they'd both just be JSObject.

To do that (and make it nice), we would probably need custom or "user-defined" types in LDPL. Then a C++ JSON parser could create new types and define LDPL statements to work with these types.

The bad thing about this approach is that it would take work and language changes, but the good thing is that a "user-defined" type would open a lot of doors and make LDPL more flexible. I'm sure a JSON parser isn't the only potential extension that needs its own data structure.

As for the validator, I like the idea, but I think it would be a shame to only support a subset of JSON.

[Lartu]

I do believe this could be simulated using just maps and prefixing special values to the keys. For example "list_" or "number_" etc. Maybe the first three characters of the key determine the type of what's contained in there, and the rest is the actual key name. And the "JSON" would be manipulated and accessed using special functions for that, as you would if you were, for example, using BASIC.

Of course it would be better to add native JSON support to the language or add user-defined types, but I'm positive JSON validation and parsing can be done with the tools the language provides right now without the need for anything else.

Of course I may also be wrong.

[xvxx]

I think I’m missing something, sorry - doesn’t the LDPL compiler need to know the depth and type of containers at compile time? How do you hold a value in an LDPL map that is sometimes a string, sometimes a list, and sometimes a map?

{
  “one”: [1,2,3],
  “two”: true,
  “three”: { “ok”: true },
}

[Lartu]

Oh, yes, that is a problem. But what you can do is also hold the depth as part of the key I believe, and simulate depth using a one-dimensional map.

[saper]

That's why I suggested a predeclared structure in the data section... it should work then?

[Lartu]

Yes, I really did overlook the fact that this is a compiled language 😓 Having predeclared structures would be something awesome, more if they even worked with containers.

[xvxx]

The issue is still that you can have a value in JSON that is sometimes a string, sometimes a number, sometimes a list, and sometimes a map, but you can’t express that idea in LDPL today. Specifically the changing container types. Even if you declare your LDPL data structure ahead of time, you can’t say “this is a list of sometimes lists of strings and sometimes maps of strings.” The variable type needs to be known at compile time.

So this code:

  STORE json-value:1 IN X

Can’t sometimes have X be a Map but sometimes be a List. It needs to only be one, concrete type when the code is compiled. I think this is the insurmountable issue with doing a JSON parser today.

[Lartu]

That is true :/ I do believe that some key management can provide some JSON functionality, but that's true. I will check if there's any JSON implementation for BASIC. If something happened to appear, that could probably be ported to LDPL.

[saper]

you can’t say “this is a list of sometimes lists of strings and sometimes maps of strings

If two different JSON objects need to parsed, they should have two different data declarations. Maybe we just need a record (struct) type?

PostgreSQL has lots of JSON functions to deal with this problem https://www.postgresql.org/docs/11/functions-json.html - for example taken from https://stackoverflow.com/a/37445088

select *
from
    json_to_recordset('[
      {"operation":"U","taxCode":1000},
      {"operation":"U","taxCode":10001}
    ]')
as x("operation" text, "taxCode" int);

will coerce JSON into a simple table (but no nesting is allowed).

[xvxx]

@saper Not two, but one single JSON object whose keys/fields each point to unique container types. Essentially unions whose members aren’t known at compile time. SQL doesn’t have this issue because it can return anything it wants at runtime.

@Lartu One BASIC-style way we might be able to do this is by either operating on an implicit JSON object, or by using a “file descriptor”-style system like socket libraries do.

So basically (ha ha...) we would treat the parsed JSON object the way BASIC treats the screen: all drawing commands are assumed to act on the screen, you don’t need to pass some object around.

SET COLOR 35
BOX CIRCLE 5,6,5,6

Those are operating implicitly on the screen. Maybe we could do:

JSON PARSE “some json string”
PUT JSON FIELDS IN my-list-var
PUT JSON VALUE AT “[0].gandalf” IN my-text-var

Or, like the socket libraries do it:

JSON PARSE “some json string” IN <num-var>
PUT JSON FIELDS FROM <num-var> IN my-list-var
PUT JSON VALUE FROM <num-var> AT “[0].gandalf” IN my-text-var
PUT JSON TYPE FROM <num-var> AT “[1].age” IN another-text-var

We wouldn’t need to change the compiler, just write a JSON extension in C++. I’m not totally sure it would work, we’d have to make sure all the use cases of JSON are there. Like being able to do a FOR LOOP over some json list and get the values out easily.

[saper]

unions whose members aren’t known at compile time

I was thinking about a simplification where we say what kind of types we expect in advance. If there is an union we could specify two different types that are being tried in a row (also useful for completely different JSON objects returned for errors).

[xvxx]

@saper If you have an example, please post it! I would be happy to help with this, as having JSON parsing in LDPL would be really useful. I just don't yet see the path forward.