Removing Microsyntaxes in favor of explicit attributes

[mscuthbert]

In its original (XML-based) draft, MNX defines a set of microsyntaxes/microformats that embed multiple pieces of information in a single attribute (represented as a string as all attributes in XML are).

As we move to JSON and have the opportunity for (1) typed data as attributes and (2) object hierarchies at each level of data, it makes sense to take a closer look at microformats and ask if one of the major MNX goals--making data more machine readable and able to be used as internal representations--can be better served by replacing microformats with explicit data types.

Time, Event, Note

I looked at three commonly used microformats. In order from least to most complex (with prior XML implementations)

1. <time signature="3/4" /> the Time Signature data type
2. <event value="/2" /> the Note value data type
3. <note pitch="C#4" /> the Chromatic Pitch data type

Pros of Microformats

In these three examples (3/4 time, half note, C-sharp in octave 4), the microformats have several important pros:

1. Easy for humans to read and write
2. Small amount of disk space (and documentation space)
3. Familiar to people coming from abc/lilypond/other text-based notation formats.

Cons of Microformats

But here are other implementations of the same types that I believe are currently valid.

1. <time signature="7 +6/*2dd + 9 + 2 + + + /16" /> -- a two-part composite time signature where the first element is 13 double-dotted breves (divided 7+6) and the second element is 12 (sic, not 11) sixteenths divided as 9 + 2 + 1.
2. <event value="*4096dddddddddd"> a note equal to 4096 tied whole notes with ten dots.
3. <note pitch="G####0000123+-" /> -- a G quadruple sharp in octave 123 with no microtonal alterations. or <note pitch="Abbbb2+239123/714o"> An A quadruple flat in octave two adjusted upward by about 335 octaves.

Please take a minute to ask yourself -- are you sure your to-be-developed MNX parser can be added to your software without yielding a SIGFAULT? And even if you wanted to have a robust checker for your system before trying to create these objects, wouldn't you need an intermediate representation format where octaves and durations were an int or float first?

These are the extreme cases that show the negatives of microformats. Here are the more common cons:

1. Requires import and export parsers to be written for each implementing agent. And the algorithms are quite complex -- might make a Medium level leetcode challenge.
2. Requires knowledge of regular expressions to implement (not a given among newer programmers)
3. They do not reveal their underlying data types -- numerator in a basic time signature is a best represented as an int, not a string, etc.
4. Harder for specification authors to later add new attributes for new or overlooked situations without breaking existing parsers.
5. They make it hard for agents that only care to support simple cases to implement them.
6. Goes against one of the major use cases for MNX compared to MusicXML: that MNX can be used as an internal storage format for new agents or an easy intermediary format for translating to existing internal formats.

Proposal

The W3C Music-Notation Working Group co-chairs met today and strongly endorsed the replacement of microformats/microsyntaxes with more explicit objects that de-entangle the various parts of the syntax. (We propose beginning with these three common microsyntax, but with a general principle that applies throughout the specification).

Implementation

No replacement implementation is proposed at this moment. However, presumably attributes like "dots": 4 would replace the "dddd" part of the duration microsyntaxes, enumerations would replace some entries (such as note value). Precise implementations would be open to community discussion once discussion of the general principle concludes.

[SteveALee]

Possibly the proposal better allows something like https://json-schema.org/ to be used for validation of documents?

[mscuthbert]

It should at least make json schema validation easier. Their schemata allow for regular-expression filters, but some of these were quite complex.

[shoogle]

I agree that we should avoid inventing microsyntaxes just for the sake of it, but it may be worthwhile in some situations.

The particular examples you raise as being difficult to handle as microsyntax would also be difficult to handle in a native form. Maybe not quite as difficult, but difficult nonetheless.

Consider this simple example:

• Microsyntax: <time signature="3/4" />
• Possible native XML representation: <time-signature numerator="3" denominator="4" />

In this case the native form is perfectly readable, so the fact that the microsyntax form looks a bit nicer probably doesn't justify the increased programming complexity needed to parse it.

But the native form doesn't scale to cope with more complicated time signatures. Consider this example:

• Microsyntax: <time signature="1/2 + 2+3/4" />

We can't split this into a single numerator and denominator, so we end up requiring a native form that is almost as complicated to parse as the microsyntax. Maybe something like this in JSON:

// Possible native JSON representation
{
"time-signature": [
        { "numerator": 1, "denominator": 2 },
        { "numerator": [ 2, 3 ], "denominator": 4 }
    ]
}

Besides the increased visual complexity, we also see that:

• Here, the time-signature key refers to an array of objects when usually we would expect it to refer to just a single object.
• The numerator key can sometimes refer to values, sometimes to arrays of values.

These discrepancies could cause parsers to trip up just easily as unexpected microsyntax.

Admittedly, it is still probably a bit easier to parse the native version than the microsyntax, but not by a huge amount. In this case it might be worth suffering a small increase in programming complexity in return for a microsyntax that is cleaner and more flexible.

[SteveALee]

I wonder if "c" etc are best treated as an alias or mnemonic rather than valid value in fractional time signature.

It seems strong typing and an ability to handle complex expressions are always going to be in tension. In general, I think types are simple primatives, and expressions are, well, expressive :)

[ahankinson]

I wonder if "c" etc are best treated as an alias or mnemonic rather than valid value in fractional time signature.

I suspect people are going to be frustrated if they write a "c" in their score, save to MNX, and it gets converted to 4/4 when they re-open it. So some way of preserving the symbol (independent of the actual time) is needed. (In MEI we have numerator (@meter.count), denominator (@meter.unit), and a third (@meter.sym) that can capture the symbol used. Another, @meter.form can also capture what components of these can be rendered (only numerator, denominator as a note head, etc.).

(Maybe also worth pointing out here that time signatures are not fractions, so terms like "numerator" and "denominator" are misnomers)

It seems strong typing and an ability to handle complex expressions are always going to be in tension

Not necessarily, if you build your schema around custom types. If you're limiting your data only to the types that are available in JSON then yes, but if you have the ability to "strongly type" with custom types defined in your schema, then you can have your cake and eat it too!

[SteveALee]

Oh, I wasn't suggesting serialisation would be lossy. Just perhaps the types of the JSON fields might be primitives, perhaps an union of numeric and enumerated string values in this case. Sorry for confusion.

[SteveALee]

By "primatives" I meant simple domain specific types, not JSON types. :) but not complex expressions.

[mscuthbert]

But the native form doesn't scale to cope with more complicated time signatures. Consider this example:
1/2 + 2+3/4 We can't split this into a single numerator and denominator, so we end up requiring a native form that is almost as complicated to parse as the microsyntax.

What's different about this than the microsyntax is that a plausible native representation would be included with the schema. So yes, it may be that a time signature can take a list of time signatures each of which can have a list of numerators, symbolization possibilities, etc. (and perhaps the format should also require a most simplified form as [num, denom] common-denominator format also for simplicity?) and prepare to have a place to store them.

[dhoernel]

Honestly, I don't see any good reason for using microsyntaxes.

1. Easy for humans to read and write => The format should be easy to read and write for machines in the first place, not for humans. Everything that requires parsing makes implementations more expensive.

2. Small amount of disk space (and documentation space) => Is this really an important argument nowadays?

3. Familiar to people coming from abc/lilypond/other text-based notation formats. => See 1. In contrast to lilypond and other text-based notation software, MNX should be well-designed for machines and not for human reading or even writing.

I stongly favor removing microsyntaxes.

[samuelbradshaw]

I would argue against (2). In my day job I work with mobile apps, and in the mobile space, reducing how much data needs to be downloaded or stored is still an important concern, especially in developing countries.

That said, I agree with your other points and that avoiding microsyntaxes makes sense for MNX.

[mscuthbert]

I agree, @samuelbradshaw, but I think that there's also a reduction of processing power needed w/o having the microsyntaxes which is important. I imagine that MNX compressed (still need a file extension for it, MNXZ?) will be like MusicXML's MXL -- super compact. The whole Ring Cycle will likely take about the space than a single digital photograph today, whether microsyntaxes are used or not. They compress very well and may even compress better with each component of the microsyntax removed.

[joeberkovitz]

Given the reorientation of MNX towards parser/generator friendliness I think it would be consistent to remove microsyntaxes.

I originally proposed them with human compatibility top of mind but with the move to JSON that goal has changed.

[adrianholovaty]

An update on this: We've removed the time signature microformat, in favor of two separate keys in the JSON object ("count" and "unit"), in commit 41a932b. All example docs have been updated as part of that commit. I've updated issues #254 (compound time signatures) and #94 (cut/common time signatures) as well.

[adrianholovaty]

Closing this, as all of the microsyntaxes have now been converted to structured data.