format: EBNF Syntax Highlighting, comments as whitespace, relax some reserved keywords

[huwaireb]

#154

Question:

1. Should doc-comments also be treated as white-space? As they can be accessed from the CST

Minor Note:
I replaced the EBNF grammar block's language from WIT -> EBNF,
as github supports EBNF for syntax highlighting

[lukewagner]

I see in #154 you're saying that the built-in types can be checked in the validator, but it seems useful just for documentation purposes to list out the scalar cases here. Is there some benefit to collapsing these cases into id?

[huwaireb]

They can be placed right above? Just not be a part of the grammar, unless of course you want to treat them differently in someway, not only as a way to reserve them / for documentation.

EBNF grammar is not meant to document builtin-types, and actually, cannot be used as such. This sets off in-correct assumptions in regards to what a grammar is. However what it is meant to be for is the specification of the format in itself, describing the syntax.

If you are however still not convinced, I'll revert this change.

[lukewagner]

That perhaps makes sense for the concrete grammar used in an implementation, but documentation and helping folks jump in quickly is indeed a key goal of this .md file.

[huwaireb]

If the markdown file's purpose is to document the WIT format, should we introduce a WIT.ebnf file?
One for the documentation of the format, one for the grammar of the format?

Should allow more freedom in how you express syntax in the documentation, as long as its equal to the EBNF form.

reverting this change.

[lukewagner]

Thanks! That's a good question regarding WIT.ebnf. So first, to lay out my rough thinking for how we should fully standardize Wit, I think we should mirror what we're doing with the component text/binary format by, in addition to the informal markdown files (whose primary goal is to introduce the proposal from scratch), we have a separate formal spec modeled after the core wasm spec designed for rigorous precision, completeness and being the final standardized document. For the component-model, the WIP is in #101. In the formal spec for Wit, I think what we should do is treat Wit packages as textual representations for component-model types, as Alex has brilliantly described at the end of Wit.md in #141. Thus the formal definition of Wit would be an attribute grammar that maps sequences of characters into the abstract syntax of componenttype and instancetype. Binary.md already has an attribute grammar mapping sequences of bytes into componenttype/instancetype here, so this lets us think of Wit as a third (after Wat and binary), much more human friendly, representation of component types. I expect this will end up producing a rather different grammar than WIT.ebnf; in particular, the attribute grammar would probably end up having separate cases for each built-in type because these are separate cases in the AST. Given that WIT.ebnf would otherwise be almost the same as WIT.md, I'm not sure it's worth maintaining both and keeping them in sync, in addition to an eventual formal spec.

[tschneidereit]

In the early days of the wit format, we talked about ensuring that doc comments could be properly translated into the right format for all languages we'd want to generate bindings for, including the right structure for parameter documentation, examples, etc. Doing that would require us to not treat comments as whitespace, and instead add their internal structure to the grammar, I think.

[huwaireb]

@tschneidereit I fully understand, I looked over how wit-bindgen and the wit-related projects by bytecodalliance function to get a better understanding

This is why I left only comments as white-space and kept doc-comments (there's two non-terminals, doc-comments being separated from comments).

However, as doc-comments can still be accessed from the CST, I'm leaving adding the doc-comments nonterminal to whitespace up to others.

[lukewagner]

This looks mergable to me. @tschneidereit, did @r-muhairi's reply make sense to you?

[huwaireb]

@lukewagner I've rebased the PR, as long as everything is okay this should be fine.

[lukewagner]

Oops, sorry for letting this drop. Thanks for the rebase and reminder!