Support for generics in schema

[Gozala]

I often find myself resorting to use of Any type accompanied with a comment saying it’s actually a generic over ….

List, Map, Link are already generic through a special syntax support, but structs and unions aren’t :(

I don’t really know how hard it would be to actually add support for generics, yet I think having syntax for them even if they just desugar to Any would be useful way to communicate intent.

type Changeset <Op> = {
  origin optional &Changeset<Op>
  change Op
}

Could desugar to something like

type Changeset = {
  origin optional &Changeset
  change Changeset_change
}

type Changeset_change = Any

While improving schemas when used as a documentation tool

[Gozala]

I think there’s some overlap with #194 where you don’t want to fully specify details and keep type somewhat open

[RangerMauve]

Generics would be great to have. 👍

It seems like it'd require some major effort to update all the codebases to support this though, so we might want to find specific projects that needs this and find resources to implement it.

Is this something related to your UCAN work at web3.storage?

[Gozala]

Is this something related to your UCAN work at web3.storage?

Not necessarily UCANs related work but indeed while working on web3.storage. We've been specifying our protocols using IPLD schema, but found this limitation very painful. We end up resorting to using TS interfaces to disambiguate.

[RangerMauve]

Would folks from the web3.storage team have time to work on this and add changes to existing schema implementations? (go + js)

[rvagg]

There's two ways forward with this, both with downsides:

1. Treat this as DSL sugar, such that it doesn't show up at all in the DMT. Which means (a) we'd be having to invent names for types in the DMT for each parameterised form of a generic type and (b) round-trips through the DMT and back to the DSL aren't going to be a happy-time unless there's some funky hacky thing going on with type names. Neither of these are a no-go. Inventing names is something we do in some places, but never in the DSL->DMT transformation (for anonymous types we do it in Go codegen [using __] and there's some internal synthetic naming in the JS implementation too (using 'foo > bar' cause spaces are a nice cheat where it's not user-exposed). The round-trip isn't essential, it's just nice to be able to do and it also helps with testing; having DSL->DMT be one-way is unfortunate because there are many cases where you're shipping around a DMT and want to use a DSL to make it human-readable (we don't preserve comments, whitespace, or ordering anyway).
2. Build it into the full Schema spec, DSL, DMT and implementations. The problem with this is the amount of work, however, we don't guarantee that every feature works everywhere, in fact, we don't yet have an implementation or use of Schemas implements everything! So the important piece here would be coming up with how this is represented in the DMT which is achieved by figuring out how to represent it in the schema-schema.

Additional thoughts:

Can you do multiple parameters? Foo<X, Y, Z>

I can see generics being possible on maps (TypeDefnMap), lists (TypeDefnList) and structs (TypeDefnStruct), possibly with an additional optional field like templateParameters [String] (a list for multiple parameters?). What about unions? I wouldn't be surprised if there are some challenges there that need thinking through.

Do all of these recursive templates need to work?

type Foo<T> struct {
  bar Bar<T>
  baz {String:T}
  bing &Thing<T>
}

type Bar<T> [T]

TypeNameOrInlineDefn in the schema-schema would be where you'd probably work to do the recursion; but you'd need to introduce a new structure similar to InlineDefn to hold a combination of TypeName and the parameter(s).

[rvagg]

@Gozala if you want to start experimenting, you could poke around with https://github.com/ipld/js-ipld-schema/blob/master/ipld-schema.pegjs which I think you should be able to follow. If you've not looked at the schema-schema before then that may be an introduction. This might be even more helpful for you in grokking what the schema-schema is trying to do, my TypeScript version of it: https://github.com/ipld/js-ipld-schema/blob/master/schema-schema.ts - this validates the DMT, which is output from the fromDSL parser function in that package, e.g. import { fromDSL } from '@ipld/schema/from-dsl.js'; /** @type {Schema} */ const dmt = fromDSL('type Foo string');. In fact, adding a new feature on the TypeScript version first might not be such a bad thing, because as you can see in there from all my TODOs, there's a bunch of things that I think are currently missing from the schema-schema that should be there so there's some backporting already needing to be done.

[RangerMauve]

Talking to @expede, this is something she and Fission folks would also like to use. (In Rust in particular!)

[BigLep]

2023-01-03 IPLD triage notes: @Gozala are you able to respond here and do the followup work?