MNX in JSON example documents and schema

[adrianholovaty]

As the next step in exploring the idea of moving to JSON, I've done the following:

1. Encoded three of our example documents into JSON.
2. Created a basic JSON Schema document.

My goals were:

1. Get a deeper sense of how MNX in JSON feels. Are there any unexpected pitfalls? Does it seem easy or difficult to work with?
2. Learn about JSON Schema and see whether it would work for describing the MNX JSON format. Can it handle what we need to encode?

In doing this, I took a first stab at porting our existing XML elements to a JSON structure. I put some thought into this but not a lot; I expect some of the decisions I made won't stand up to scrutiny and that there's a better way to organize the data.

I realize many people will be tempted to dive into the weeds and make suggestions on better ways to encode things. Please avoid that temptation — those sorts of decisions will come a bit later! I'm looking for higher-level feedback, specifically the following:

• Are there any fundamental problems with JSON identified here, and what are they?
• Are there any fundamental problems with JSON Schema here, and what are they?

My own take is that this feels good. Nice and compact, with just as much expressiveness as we've had in XML. The main ugliness is the need to encode a "type" for a few of the objects, in situations where we have an array of heterogeneous items (e.g., sequence content and measure content).

I'd never used JSON Schema before and was pleasantly surprised at how clear and easy it was to get started with it. In my estimation it supports all of the things we need to do. The trickiest bit I ran into so far is encoding the ability for an array to contain heterogeneous items; I'm not sure whether I've encoded this in the right way, but it seems some combination of "anyOf" or "allOf" will get the job done.

Without further ado, here are the experiments:

• Accidentals: Image and XML version, JSON version
• Styling elements (basic): Image and XML version, JSON version
• Multimeasure rests: Image and XML version, JSON version
• Experimental JSON Schema

If feedback here is positive, I think that'll seal the deal for using JSON, and we can commit to the change. The next step would be to put more thought into the details of encoding the various bits of the document, using my experiment as a starting point. I'll also need to change our docs system to support JSON, but I don't expect that'll be too much work.

[mscuthbert]

This looks amazing. It seems like everything from the XML version has been captured and nothing lost!

And huge improvement for parsing simplicity!

Smaller comments for future:

• Since JSON gets indented more frequently than XML maybe use 2-spaces for examples etc rather than 4 (I usually use 4 but make an exception for JSON)
• Consider removing "-" now from ID names unless we have a good reason to maintain that MusicXML compatibility. I can already see the support issues from people who don't understand why some parts are accessible from dot notation and some not. Similarly don't allow 0-9 or _ as the initial letter. Define "identifier" as "^[a-zA-Z][a-zA-Z0-9_]*$"
• next schema draft let's move all the definitions that have "type" to the top as types and then reference them where they can appear.
• for thought: now that we have real arrays is the "sequence" object necessary?

Fantastic work!

[adrianholovaty]

Thanks! I've updated all of the examples to use two spaces for indentation.

For the IDs, I agree with all of what you've written. The regex I used was quickly done, without much thought. :-)

For the "sequence" object: measure content currently can contain directions and sequences (see here) — hence the need to disambiguate between the types of objects within the measure via "type". I'd like to explore whether it'd be possible to make a measure contain only sequences, moving clefs elsewhere into the data structure. This would make inner-bar clef changes more elegant and would simplify parsing the meat of an MNX document (the notes and rests). It would also mean we wouldn't have to use "type" there, and in JSON the sequence content could indeed be an array.

[ahankinson]

I had a look at the examples, and while they do look interesting, I think I spotted a few "fundamental" problems with them as expressed. This is not to be a JSON-sceptic, but out of interest to see whether anything is actually gained with a JSON notation format.

Are there any fundamental problems with JSON identified here, and what are they?

This sort of thing will likely give you grief later on:

"parts": {
        "PartA": {
         ... 

Putting arbitrary values as keys works fine for parsing, (It's all just strings!) but for data understanding you will likely run into problems since:

a) parsers will have to iterate all keys in order to find the content, since they don't know the keys in advance, and
b) the order of keys in JSON objects is not guaranteed, and
c) (most importantly) It means that somebody can call their part something like "measures" and really screw things up!

I also see that you've converted the data structures in two different ways. One way is "direct" children with a list, e.g.,

"measure": [a b c d]

but also "indirect" children with the content key:, e.g.,

"measure": {
    "content": [e f g h]
}

Is there any pattern you were following for this? In the schema design when did you choose one pattern over the other? (See also my comment below on Myke's suggestion of "sequence"...)

I would also be careful with requiring constructs like this, where you specify empty placeholder objects for the number of measures:

"measures": [{
                "time": {"signature": "4/4"}
            }, {}, {}, {}, {}, {}, {}]

since empty objects tend to evaluate as "falsey" in most dynamic languages, so it would be really easy to miscount or discard.

for thought: now that we have real arrays is the "sequence" object necessary?

If you make "sequence" an array, will you need to find a place for the attributes you've already defined? In general, where will styling and other attributes go on the "parent" if you convert these things to "direct" children in arrays?

The main ugliness is the need to encode a "type" for a few of the objects, in situations where we have an array of heterogeneous items (e.g., sequence content and measure content).

I'm not sure I would call this ugly -- in fact, adding type to your objects would vastly aid in understanding the potential contents of data structures. Implementers will not be correlating an arbitrary MNX file with your JSON Schema at parsing time, so when they come across an object that has no type, they will be left trying to infer the type from the (possible) keys on that object, or the value of the parent. (This will be made more difficult if the parent is called "PartA" or some other arbitrary string!)

Also, consistency is nice. Knowing that there always is a type key when parsing would be useful, I think?

---

Out of interest, I took the Multimeasure Rests examples, as it was the largest, in both JSON and XML and saved them as files, in both "raw" and "minified" forms. I also reformatted the JSON example as 2-space indents, per Myke's comment. Here's what I found:

 12 KB mm_example.json
8.9 KB mm_example-2sp.json
4.5 KB mm_example.xml

2.5 KB mm_example.min.json
2.6 KB mm_example.min.xml

This would suggest that in fully expanded form, the JSON is 2.6x larger than the XML example, and with 2-space indents still ~2x larger than the XML. Minifying them by removing whitespace gives a slight edge to JSON. (The actual byte size is 2608 for the minified JSON, and 2653 for the minified XML, so a saving of 45 bytes for JSON).

[adrianholovaty]

Thanks for the thoughts — much appreciated.

Regarding parts as an object instead of an array: that's a great point, and I agree we should use an array here, in which each part has an "id" key. I'd been optimizing for easy part lookup by ID so that a "score" can easily find its parts, but the "score" object isn't required in an MNX document and the default behavior is to render all of the parts in order — so clearly the order matters and we should use an array.

It means that somebody can call their part something like "measures" and really screw things up!

I'm not seeing how this would be an issue...? The namespace for part IDs doesn't intersect with the namespace of JSON keys here. What am I missing?

Is there any pattern you were following for this? In the schema design when did you choose one pattern over the other?

I'd describe the philosophy as "Keep things using the tightest-possible data structure unless necessary." Keeping in mind that most objects will require some globally available keys such as "style".

I would also be careful with requiring constructs like this, where you specify empty placeholder objects for the number of measures:

I agree that part is ugly. It's also ugly in the XML version, in which <global> simply contains a bunch of empty <measure-global> elements.

Also, consistency is nice. Knowing that there always is a type key when parsing would be useful, I think?

I agree consistency is nice — to an extent. Using a "type" key in every single object feels very verbose and overengineered to me. Are you suggesting something like this?

{"time": {"type": "timesignature", "signature": "4/4"}}

I'd see that as redundant. You already know you're dealing with a time signature because the data lives in a certain context (within a `"time" key within a measure object).

It also introduces room for error, allowing the following:

{"time": {"type": "keysignature", "signature": "4/4"}

You might respond: "Yeah, it's obviously wrong to use "type": "keysignature" for a "time" object." Well, if the only valid value there is "type": "keysignature", then there's no point in requiring it!

Zooming out, I take the following philosophical stance:

• We can assume the parser knows its current context in the document and hence doesn't need to be told the type of each subobject.
• However, an explicit "type" is warranted in situations where a heterogeneous collection of subobjects is possible. We should aim to minimize these situations, but music notation is complex and it's not necessarily always possible.

[mscuthbert]

Just as a follow-up, I was curious about parsing times.

The four scenarios are:

1. Using orjson (~the fastest Python JSON library)
2. Using Python's built-in JSON library
3. Using lxml (probably the "most correct" C-based Python XML library, built on libxml2)
4. Using Python's built-in etree implementation

Hi Andrew -- I did a similar experiment a couple of weeks back (I did not try orjson though) using a sample score with 100 measures at 10 notes each, and got pretty similar results. About 25-40% faster.

Where I got about another speedup of about the same magnitude (this one I just compared Python json to etree) though was defining a custom JSON encoder that automatically transformed the notes into a simple Note class in the parser, as opposed to getting the attributes and subelements in etree in the fastest way I knew how (how music21 parses musicxml). But while significantly faster, it still did not give the order-of-magnitude speedup that I had hoped for.

[mscuthbert]

a) parsers will have to iterate all keys in order to find the content, since they don't know the keys in advance, and b) the order of keys in JSON objects is not guaranteed, and c) (most importantly) It means that somebody can call their part something like "measures" and really screw things up!

I agree w/ you and Adrian that anything where order is important should be an array, so that can change this. (Note though that only one place needs to have the parts in "score order" -- the actual measures/notes/etc. data can be in an object with keys as names). I'm with Adrian that I don't think that having a key named "measures" etc. will cause a problem except if doing a demonstration where particular part names are encoded explicitly. (But that can be done in musicxml/MEI/MNX-XML also -- you can always number your measures 10, 2, 8, 4, 3, in that order...)

re: direct encoding of array vs "content" array: Is there any pattern you were following for this? In the schema design when did you choose one pattern over the other?

If the caller is already a key within the parent object then it doesn't need a content, for instance if you had a part defined as (JS, not JSON encoding): {name: 'Flute 1', style: {...}, measures: [...]} then there's no need for a "content" object, but if the array was going to be the main content in a mixed data context, such as the content of a measure, where type needs to be specified, then we'd need an object for {type: "measured_tremolo", content: [note1, note2]} etc. (not proposing this as a syntax, for tremolos, parts, etc., just a meta-syntax).

I would also be careful with requiring constructs like this, where you specify empty placeholder objects for the number of measures ... [example removed], since empty objects tend to evaluate as "falsey" in most dynamic languages

I agree that it looks funny, but in addition to the drawback, there's a big advantage too of empty objects evaluating to Falsey -- it means I can do something like this:

measure_styles = {}
for i, mg in enumerate(measure_globals):
    if mg:
         measure_styles[i] = measure_global_to_native_object(mg)

and save a costly lookup/object creation for something with default style.

Python's etree considers empty XML tags with no attributes Truthy, which is inconsistent with the rest of the language. (Can't remember if lxml does the same).

If you make "sequence" an array, will you need to find a place for the attributes you've already defined?

Yes, that was a mistake on my part. Though I can imagine that sequences might become more generalized and subclasses could be used as grouping elements in more places (tuplets), etc.

The main ugliness is the need to encode a "type" for a few of the objects, in situations where we have an array of heterogeneous items (e.g., sequence content and measure content).

I'm not sure I would call this ugly -- in fact, adding type to your objects would vastly aid in understanding the potential contents of data structures. Implementers will not be correlating an arbitrary MNX file with your JSON Schema at parsing time, so when they come across an object that has no type, they will be left trying to infer the type from the (possible) keys on that object, or the value of the parent. (This will be made more difficult if the parent is called "PartA" or some other arbitrary string!)

As Adrian said, the type is often defined by the key of the object calling it as in "time_signature".

What hasn't been decided in this draft is where to put the type in the contexts where it is not defined. My preferred place would be as a two-element list of ["type", {object}]. Again in JS not JSON:

[['note', {pitch: 'C4'}], ['rest', {...}]]

or in the case where objects need to be contained in an array of event objects, that object could encode the type:

events: [
    {type: 'note', value: '/4', content: {pitch: 'C4'}}, 
    {type: 'rest', value: '/8d', content: {}},  // or null? 
]

I think it's may be worth breaking the inherited musicxml aversion to abbreviations and move "value" to "val" or "v", and content to "con" -- but probably there shouldn't ever be both a "value" and a "content" so "length" or "len" better for events. Though my personal opinion is that MEI went too far with some abbreviations.

I'm also on a war against text-microsyntaxes especially now that we have actual numbers available, so my world would probably be :

events: [
    {type: 'note', len: 4, val: {pitch: 'C', oct: 4}},
    {type: 'rest', len: 8, dots: 1, val: {}},
]

JSON with 2-space indents still ~2x larger than the XML. Minifying them by removing whitespace gives a slight edge to JSON.

I think it's too soon to see how much space savings/expansion the JSON format would be compared to XML, but because unquoted whitespace isn't significant, I would expect most files to be saved w/o spaces/returns in them, and examined/edited in an editor that can expand JSON and recompress it. (I once made the stupid mistake of storing JSON [and XML] in a DB with nice spaces so that the few times I needed to read it, it looked nice to me. When I compacted it, a 65GB database went down to 20GB).

[jacobtylerwalls]

Python's etree considers empty XML tags with no attributes Truthy, which is inconsistent with the rest of the language. (Can't remember if lxml does the same).

I think it's slightly different than that, but equally confusing (and so will start raising DeprecationWarning in Python 3.12). For the moment, the truth test for an element amounts to whether it has any child tags:

>>> from xml.etree import ElementTree as ET
>>> a = ET.fromstring('<a/>')
>>> bool(a)
False
>>> has_child = ET.fromstring('<a><b/></a>')
>>> bool(has_child)
True

[mscuthbert]

Python's etree considers empty XML tags with no attributes Truthy, which is inconsistent with the rest of the language. (Can't remember if lxml does the same).

I think it's slightly different than that, but equally confusing (and so will start raising DeprecationWarning in Python 3.12). For the moment, the truth test for an element amounts to whether it has any child tags:

>>> from xml.etree import ElementTree as ET
>>> a = ET.fromstring('<a/>')
>>> bool(a)
False
>>> has_child = ET.fromstring('<a><b/></a>')
>>> bool(has_child)
True

oh, right -- it was more complicated than I described. The way you've put it here, it seems totally consistent with Python. It's this (hypothetical) behavior that's confusing:

>>> empty_sharp = ET.fromstring('<accidental><sharp/></accidental>')
>>> 'yes' if empty_sharp.find('sharp') else 'no'
'no'
>>> fancy_sharp = ET.fromstring('<accidental><sharp><smufl>medievalSharp</smufl></sharp></accidental>')
>>> 'yes' if fancy_sharp.find('sharp') else 'no'
'yes'

the notion that find(tag) in a boolean context returns False if the thing found has no subattributes is the cause of many bugs. Since the Element does not support in or has this ends up making problems and resulting to unintuitive tests of is not None

[paul-bayleaf]

Thanks for providing the experimental schema. I've been experimenting generating MNX JSON using C++ with the nlohmann/json library and validating against this schema. But there a minor gremlin in the schema: syntax error line 173 spurious trailing comma.

[adrianholovaty]

Sorry about that — should be fixed now! Thanks for the pointer.

[joeberkovitz]

Very interesting developments here! I have been working on other non musical projects but it’s exciting to see this new (old?) initiative coming to life. I think I originally disliked the JSON idea back in the day but I think it’s very much worth considering now.

I will make just one comment concerning examples, because the proof of the relative advantages of this approach is going to come out of actually trying it. (I am sure that all the problems of migrating the music modeling to JSON are solvable — the real issues are around practicality, utility, community acceptance.)

For a real validation, my suggestion is: write some kind of toy renderer or even (dare I say it) toy editor for a subset of the proposed JSON version of MNX. No under the hood transformations to MusicXML allowed. How does it feel to write it?

Some wag better not suggest I do it (it will only validate my inability to code any more).

[webern]

Once again I would love to see the development include a spec document (I think you plan to use JSON Schema, Structural OpenAPI is also an option). With a spec document we can be doing code generation in other languages. We could even check-in code generating programs, a make file, and validation tests. All of this is unlocked by a machine-friendly specification language!