Fallbacks within Union

[aschmahmann]

Feature request: it should be possible to describe unions with fallback cases.

Using IPLD Schema syntax to describe it, for example, all of the following should be valid:

type Foo String
type Bar Int

type Keyed union {
  | Foo "foo"
  | Bar "bar"
  | Any default # would match a map {"fancy" : 3.2 }
} representation keyed

type Kinded union {
  | Foo String
  | Bar Int
  | Any default # would match {"some-struct-field" : "data"}
} representation kinded

type Envelope union {
  | Foo "foo"
  | Bar "bar"
  | Any default # would match {"tag": "next",  "payload": { "some-field": 0}}, would not match if the discriminantKey or contentKey were missing
} representation envelope {
  discriminantKey "tag"
  contentKey "payload"
}

type Inline union {
  | Foo "foo"
  | Bar "bar"
  | Any default # would match {"tag": "next"}, or {"tag" : "next", "union-case" : "data"} but would not match if the discriminantKey was missing
} representation inline {
  discriminantKey "tag"
}

Motivation

Sometimes it is useful to describe an abstract format for data, such as a list of data that all has the form { "sometype" : some-nested-object} but with different type names. While we could write out every single type name into a keyed union we'd end up with errors if we missed one. Instead it'd be nice if we could allow passing through the data that fits the general format and let the next layer of the application figure out how to deal with it.

Here we have an example where we want some consumers of this data to be dumb processers of the data and not have to understand every type that could possibly be in the union even if that means they'll accept some data that's not 100% compliant with a global schema they're unaware of.

When this has been brought up in the past it's been noted that we could just try decoding that fragment of the schema and if it errors then just interpret the data with a different schema that looks like the "Any" schema. This both makes the schemas harder to use (and less descriptive in things like specs) and also this formula is so defined we could just support it.

Alternatives

If there was support for the probe union representation #177 this would be easy enough to support instead as (keyed union example below):

type JustFigureItOut union {
    | Keyed
    | {String : Any}
} representation probe

type Keyed union {
  | Foo "foo"
  | Bar "bar"
} representation keyed

[rvagg]

less risky and complex than probe, particularly if this is strictly for use with Any, i.e. Any default is the only valid string involving default.

[warpfork]

I'm broadly on board with this.

The main bit that needs some resolving here is how we want this to end up represented at the type level.

To set it up and review current behaviors: For example, currently, when we have something like the Envelope type above, and it matches on the representation-level data of {"tag":"bar","payload":0}, then the type-level view of that data is {"Bar":0} -- the type name as a key, and the payload as the value.

(The higher-level logic in force there is: "the type level representation should be purely a function of the type definition clause, and not regard the representation strategy".)

Now: if we roll with this concept of adding a default keyword: what happens when we match {"tag": "next", "payload": {"some-field": 0}} to the Envelope type? We get... {"Any":{"some-field": 0}}, at the type level view?

That would be defined, but we lost something. It's not clear how to access the "next" string, or anything semantically equivalent, in this example. I think we need to figure out where we expect that information to be accessible.

[aschmahmann]

Now: if we roll with this concept of adding a default keyword: what happens when we match {"tag": "next", "payload": {"some-field": 0}} to the Envelope type? We get... {"Any":{"some-field": 0}}, at the type level view?

Maybe I'm misunderstanding but why couldn't it be {"next" : {"some-field": 0}}} we properly matched on {"tag": "next", "payload": {"some-field": 0}} because we saw a tag and payload, we just didn't know how to validate any of the data in the payload and so we took it as is.

[warpfork]

What's the expected behaviour if we try to match {"Next":{"some-field":0}} onto

type Keyed union {
  | Next "otherkeyword"
  | Any default
} representation keyed

...?

That data would actually match the default, and so the type would be Any. And if we got data of {"otherkeyword":...}, that would match the type Next, so... the type level views would collide.

[rvagg]

@aschmahmann do you want to try and pursue this further or shall we close this for now?

[aschmahmann]

We've used this DSL description in the Reframe specification ipfs/specs#272. However, I do not currently have the bandwidth to pursue integrating this into the schema-schema.

[rvagg]

I was just thinking through alternatives here and I ended up in two places without realising I was retreading existing territory until late in the thought process:

1. Basically the same as probe in schemas: consider introducing a 'probe' union #177 - step this problem back up one level and solve it with a union of unions. This seems the most IPLDish way to solve this, we're performing matching after all. But we get into the depth problem as discussed over at schemas: consider introducing a 'probe' union #177 (although I'd say we could just spec it as a single level of matching and defer arguments about increasing that till later, although "one level" is even a bit hard to define when you get into complex structures that need matching ...).
2. Something similar to what @aschmahmann has presented in the "Alternatives" section above that integrates probe, with the {String:Any} matcher.

Sooo, that leads me to a possible in-between. Perhaps what | Foo default means is not that you get a datamodel.Node (an Any) but a {String:Any}. So it becomes a fallback for "this is a map, it doesn't have one of the predicted keys, so I'm giving you access to the decoded map. It's always going to be a map and we'll need to ensure that it's a map just to perform the matching.

It gets a bit verbose when you compare typed to data model forms of the same data, but maybe that's OK because at least it's not lossy:

type Bop struct {
  Nope Int
} representation tuple

type Keyed union {
  | String "foo"
  | Bop "bar"
  | Catchall default
} representation keyed

• { "foo": "Foo" } -> { "String": "Foo"}
• { "bar": [ 1 ] } -> { "Bop": { "Nope": 1 } }
• { "boop": "bing" } -> { "Catchall": {"boop": "bing" } }
• { "String": { "some-field": 0 } } -> { "Catchall": {"String": { "some-field": 0 } } }

I think the same thing might work for inline and envelope unions - you just treat the container as a map and give a {String:Any} as an "I give up, just have the whole lot". We'd have to decide what happens to additional fields - are we strict if we find more fields than we expect?

Kinded unions are different, an actual Any might make sense there and I don't think it'd be lossy because you're providing the underlying kind and there's no name for it anyway. In fact, kinded unions are probably the easiest to do this to.

type Kinded union {
  | Foo string
  | Bar int
  | Catchall default
} representation kinded

• "foo" -> { "Foo": "foo" }
• 100 -> { "Bar": 100 }
• { "some-field": 0 } -> { "Catchall": {"some-field": 0 } }
• 1.10101 -> { "Catchall": 1.10101 }