Custom schema definition language/IDL

[philpax]

Opening this up for discussion, but this will probably not happen for a while.

A common complaint I've seen amongst package users is that the TOML syntax for specifying package items (components, concepts, etc) can be confusing/counterintuitive, especially with error handling, and it's not always the most obvious how to naturally extend something. This is one of the bigger issues mentioned in #693.

The largest issue people have is TOML's poor support for nested tables - if they have something like

[messages.Input]
description = "Handles input"
fields = { shoot = "Bool" }

you cannot actually break the table onto multiple lines:

[messages.Input]
description = "Handles input"
fields = { 
  shoot = "Bool",
  direction = "Vec2"
}

You must either use an inline single-line table (which grows unwieldy with more fields)

[messages.Input]
description = "Handles input"
fields = { shoot = "Bool", direction = "Vec2" }

or use a separate table (which requires restructuring your TOML / looks ugly):

[messages.Input]
description = "Handles input"

[messages.Input.fields]
shoot = "Bool"
direction = "Vec2"

To fix this, we can either find a more relaxed format for the manifest (I'm partial to KDL), or we can break the schema out of the manifest and implement our own grammar for it. The latter would enable us to be much more lenient with parsing and to improve error handling.

This might also be necessary if we start supporting defining more complex types in our schema, like custom structs and such.

The syntax bikeshed is not open - especially as we don't know if we're going to do this - but a reasonable first pass would be something like

component player_model_ref {
  type: EntityId,
  description: "The player's body's entity ID",
  attributes: [Debuggable, Networked]
}

concept FpsPlayer {
  name: "FPS Player",
  description: "A player in a FPS gamemode."
  components: {
    is_fps_player: (),
    // can leave out default values if there's no meaningful default, which can't be done with TOML at present
    player_model_ref, 
  }
}

message Input {
  description: "Handles input",
  fields: {
    shoot: Bool,
    direction: Vec2,
  }
}

enum PlayerState {
  WaitingForSpawn,
  Alive, 
}

or a more attribute-based approach:

#[Description("The player's body's entity ID")]
#[Attributes(Debuggable, Networked)]
component player_model_ref: EntityId;

#[Name("FPS Player")]
#[Description("A player in a FPS gamemode.")]
concept FpsPlayer {
  is_fps_player: (),
  // can leave out default values if there's no meaningful default, which can't be done with TOML at present
  player_model_ref, 
}

#[Description("Handles input")]
message Input {
  shoot: Bool,
  direction: Vec2,
}

enum PlayerState {
  WaitingForSpawn,
  Alive,
}

(but please don't read too much into the details of these, these are just examples of what's possible!)

[FredrikNoren]

@philpax Hm so the schema might be part of the problem, but there's more that's missing that I think makes it harder:

• There's no syntax highlighting
• There's no auto-completion
• There are no helpful inline errors (and by extension squiggly red lines in VSCode etc.)

Moving to a custom syntax wouldn't (at least immediately) help with any of those. Other paths we could explore:

• Specify the schema in Rust
• Some other schema language with support for the above

[philpax]

Agreed. If we did this, we'd also implement some kind of LSP (hopefully using the same machinery) to enable those things, but that will take some engineering effort as well.

Specify the schema in Rust

If you're suggesting that the schema itself should be its own Rust file: that's too complicated and requires parsing/compiling Rust to extract information, and it's too powerful - you could express things that aren't actually possible in our system. Conversely, there are things that would be hard to define without using macros/a DSL, at which point you'd be back to square one.

If you're suggesting the schema is defined inline with the code: similar kind of problem with it being misaligned for the task, except now it would be very difficult, if not impossible, to use it with other languages. I know we're not really targeting other languages at this time, but we shouldn't lock ourselves out entirely.

Some other schema language with support for the above

Yeah, this is much more doable. There are lots of languages that support strong schema definitions that can be used to constrain what users type - JSON and XML are both good offerings here, but neither are super-pleasing for something to be readily edited by a human (at least for now).

There are other options out there, but nothing that comes to mind. YAML is too freeform/problematic. Other ecosystems either use their own language or something that's only machine-operable, I believe.

[droqen]

Auto-completion would really help me as a dev with #728, knowing/reminding me what options there are for that value

Or a machine tool of some sort but that's further out

[philpax]

On the subject of completion: apparently people are using JSON Schema with their TOML, and it seems to mostly work. That's something we could investigate - even more powerful would be to generate the JSON Schema definition directly from our definitions. Might make an issue for it if it appeals?

[dezren39]

kdl is a super cool language kthxbai

[FredrikNoren]

Oh that's new to me, interesting!

[philpax]

Another potential alternative: StrictYAML. There's a Rust crate from last year that we could potentially resurrect. This could benefit from all of the existing tooling around YAML validation.

[philpax]

A more experimental idea: we have a proc macro inside Rust that updates the ambient.toml for that package each time it's compiled, so that

ambient_api::schema! {
    #[description("The player's body's entity ID")]
    #[attributes(Debuggable, Networked)]
    component player_model_ref: EntityId;

    #[name("FPS Player")]
    #[description("A player in a FPS gamemode.")]
    concept FpsPlayer {
        is_fps_player = (),
        // can leave out default values if there's no meaningful default, which can't be done with TOML at present
        player_model_ref, 
    }

    #[description("Handles input")]
    message Input {
        shoot: Bool,
        direction: Vec2,
    }

    #[description("Describes the state of a player")]
    enum PlayerState {
        /// Waiting to spawn
        WaitingForSpawn,
        /// Alive
        Alive,
    }
}

automatically edits the ambient.toml with the following:

[component.player_model_ref]
type = "EntityId"
description = "The player's body's entity ID"
attributes = ["Debuggable", "Networked"]

[concept.FpsPlayer]
name = "FPS Player"
description = "A player in a FPS gamemode."
[concept.FpsPlayer.components]
is_fps_player = {}
player_model_ref = {}

[message.Input]
description = "Handles input"
[message.Input.fields]
shoot = { type = "Bool" }
direction = { type = "Vec2" }

[enum.PlayerState]
description = "Describes the state of a player"
[enum.PlayerState.members]
WaitingForSpawn = "Waiting to spawn"
Alive = "Alive"

Advantages:

• You can still use the TOML schema even if Rust isn't your guest language
• It's much easier to edit as it's in the same place in your code and similar-ish syntax
• There's some precedent with things like Turbosql automatically generating migrations based on structures

Disadvantages:

• Still a custom syntax embedded in Rust. It's unlikely you'll have functional autocomplete or anything of the sort.
  • We could experiment with trying to embed it more directly into Rust (as Turbosql does), but we have concepts that don't necessarily map well to Rust, and will end up in a different place to where they are in the schema (e.g. player_model_ref might "look like" a variable, but it'll end up in packages::this::components::player_model_ref)
• There's multiple sources of truth, and it's unclear what will happen if you make an edit to the schema toml and then save your Rust file with a different schema
• There's the potential of a loop where the schema macro updates the ambient.toml, which causes the Ambient main macro to regenerate its boilerplate, which causes the schema macro to run again, which updates the ambient.toml, which... - I don't know if this is a realistic issue, but I figured it was worth calling out.

[philpax]

The more time passes, the more I'm thinking that we should build a custom IDL with all the usual accoutrements (syntax highlighting, LSP). TOML will get better (the new version enables multiline inline tables! yay!) but there's still an obvious friction from trying to specify data and hierarchies in a format that's clearly not meant for it.

I think this would look a lot like WIT, as it's a very similar problem.