I wrote a Tree-sitter grammar for Djot

[treeman]

Things happened, and I ended up creating a Tree-sitter grammar for Djot: https://github.com/treeman/tree-sitter-djot

It should be feature complete with Djot as described by the syntax description, but I'd appreciate any bug reports and feedback. This is the first Tree-sitter grammar I've created, so I'm sure I'm missing some tricks.

I also wrote a large-ish blog post on how to make a Tree-sitter grammar for a subset of Djot: https://www.jonashietala.se/blog/2024/03/19/lets_create_a_tree-sitter_grammar/

[NoahTheDuke]

This is cool as hell!

[NoahTheDuke]

Yeah, I'll open an issue here once I get it all figured it.

[rtbs-dev]

Ooh! Would you be willing to PR for the helix editor? They include a lot of TS grammar's by default, adding this one would be awesome! 🙏

[NoahTheDuke]

I don't use and don't know anything about the Helix editor so I'm the wrong person to ask, but adding this to nvim-treesitter has been pretty easy so I expect adding it to Helix would be equally easy.

[treeman]

I also know nothing about the Helix editor, but looking at their repo it indeed doeesn't seem difficult. A merge request needs to:

1. Add it to languages.toml: https://github.com/helix-editor/helix/blob/master/languages.toml
2. Insert the queries here: https://github.com/helix-editor/helix/tree/master/runtime/queries

Basically the same as for Neovim.

It's great to be enthusiastic, but the query files are still a bit immature, so I don't think we need to rush too much.

Still, if anyone wants to do it then feel free. I opened an issue for it on the grammar's repo to track it: treeman/tree-sitter-djot#10

[bdarcus]

I see the nvim-tree-sitter folks merged it. Looks good on quick testing!

[jgm]

Amazing! I'm starting an Emacs major mode that uses this.

[zetashift]

Hi, thank you got Djot! I was wondering if you ever ended up making the djot-ts-mode?

[jgm]

I have a work in progress. It's fairly basic.
https://gist.github.com/jgm/17e17d6730efc253078b23e12aefd3df

[vassudanagunta]

I'm curious how it performs against the manually coded implementations. Have you done any comparisons?

Also, does it make sense to either add your Tree-sitter grammar, or C and WASM parsers generated from it, to the above list?

[treeman]

I haven't done any performance comparisons against other implementations. I've only noted that it's fast enough for decently large files for my use-case.

Maybe it would make sense to add it to the implementations? But we should keep in mind that this is a Tree-sitter grammar, and should not be relied on to for instance generate html.

[vassudanagunta]

You mean to say it should only be used to implement syntax highlighting or to support syntax-aware editors?

If so, is it because renderers should be built against djot's standard AST, not the syntax tree produced by this Tree-sitter grammar? Does djot even have a standard AST, across the Lua, Javascript, Rust, Go, etc implementations?

Or is it because Tree-sitter parsers produce concrete syntax trees, not abstract syntax trees, more suitable for syntax highlighting and editors, not so suitable for compilers, interpreters or renderers?

[treeman]

Uh, I realize I'm out of my depth here and I should've said that I've heard that dedicated parsers are better suited, but I'm no expert on the trade-offs here. hides

Does djot even have a standard AST, across the Lua, Javascript, Rust, Go, etc implementations?

I don't think so no, at least with jotdown you don't have an AST, only a stream of iterators to work with.

[vassudanagunta]

Fair enough. I'm out of my depth too, wrt to Tree-sitter, hence the questions. I'm working on a metalanguage called Plain Text Style Sheets which focuses on human oriented languages like Markdown. Parsers based on it would output a standard abstract grammar tree or semantic tree (I'm not sure what the right terms are).