Roadmap: Nvim-treesitter 1.0

[clason]

Rationale

Nvim-treesitter was created to drive the user-space development of tree-sitter related features, including but not limited to highlighting. It basically offered three related but different features:

1. parser installation and management,
2. query aggregation and creation,
3. interface for other plugins to hook into as "modules".

This has been very successful, but the situation has changed significantly since it was started, with more and more parsers being added, while more actual features were included in Neovim core. At the same time, the current architecture is increasingly limiting further maintenance; in particular the module system is adding significant overhead and makes it hard to implement changes without breaking other plugins.

We should thus overhaul and rewrite nvim-treesitter to make it fit for the next few years. This will be a breaking change for users and plugins, but will hopefully allow us to mark the result as stable so the future experience will be much smoother. See below how this process will be handled.

Goal

Nvim-treesitter should be leaner and focus on its core goals (1. and 2. above).

Parser installation

This plugin should only handle installing, updating, and removing parsers and queries:

• parsers and matching queries should be installed in a standard directory (stdpath('data'), but configurable); ideally, you should not need to load the plugin if you only want to use parsers;

• in particular, only queries for installed parsers should be visible in runtimepath;

• parsers should be sorted into "tiers", e.g. (as a rough idea),

  • Tier 1 (stable) parsers are stable and versioned, queries kept updated, all tests must pass, changes are rare -- these parsers should be safe and useful to install for everyone (very small list, growing over time);
  • Tier 2 (core) parsers and queries are maintained by project team, all tests must pass -- these parsers should be safe to install but may be niche or have frequent breaking updates so should be installed individually and deliberately;
  • Tier 3 (community) parsers are stable, queries are actively(!) maintained by the community, tests muss pass -- these parsers have varying feature coverage and have to be updated manually, so caveat emptor;
  • Tier 4 (unstable) parsers are unstable, unmaintained, or marked experimental for other reasons, tests may be run but need not pass -- these parsers should only be installed by someone interested in helping develop them and move them to Tier 3.

  These tiers determine whether CI is run, whether parsers are automatically updated (e.g., only tier 1 and tier 2, one PR/commit per parser), and can be selected for installation (i.e., ensure_installed = 'core'). Parsers should be aggressively moved between tiers as quality or maintenance changes.

• only a limited set of reasonably popular platforms should be supported out-of-the-box for compilation; all others should be handled by making compilation commands easily configurable.

A big part of this will be pushing parser maintainers to use versioned releases (which may be a requirement for inclusion in Tier 1 or even Tier 2); one possible scheme would be to require

• patch releases for parser changes that do not affect queries at all (grammar.js or scanner.c bugfixes),
• minor releases which add nodes (meaning old queries will still work, but need changes to make full use of the updated grammar),
• major releases which remove or rename nodes (meaning old queries will no longer work).

Ideally, this would be automated via something like https://github.com/google-github-actions/release-please-action.

Queries

We should move towards closer alignment with upstream and other editors using tree-sitter; this will make it easier to share efforts (and quicker to add new parsers or pull improvements from, e.g., Helix), see neovim/neovim#22495

• injections: should use the upstream format which is now supported by Neovim, see chore(injections)!: Move to upstream syntax #4675
• highlights: where it makes sense, should use Helix' captures, which are derived from established SublimeText scopes, see Tracking issue: Moving towards Helix captures #4799
• locals: should move to upstream format (@local.definition etc., but keep the more specific nesting).

We should also investigate how to streamline or automate syncing with upstream queries (if parser repos contain them).

It might also make sense to include more queries here such as textobjects, to make sure they track parser version closely. (Rule of thumb: every feature used -- or planned to be used -- in Neovim should have queries provided by nvim-treesitter.)

Custom predicates and directives should be upstreamed to Neovim.

Modules

We should completely drop the module framework; plugins should instead handle setup and attaching themselves. (https://github.com/nvim-treesitter/nvim-treesitter-context demonstrates that this is quite straightforward.)

In particular, the bundled modules should be removed:

• indents is a candidate for inclusion in Neovim; it's not quite ready for primetime yet, so it can stay for a while, but in the form of a simple indentexpr function instead of a module;
• incremental-selection mostly served as a proof-of-concept for non-highlighting uses of tree-sitter; if people are actively using it, they should consider moving it to a separate plugin (or seeing if textobjects don't serve this purposes even better);
• statusline is a poor man's nvim-treesitter-context.

This also affects the external modules under the nvim-treesitter org:

• textobjects is also a candidate for inclusion; probably should be refactored into a plugin before that, though;
• refactor should be -- wait for it! -- refactored;
• playground is already mostly obsolete; the only missing feature (live query editor) is planned to be upstreamed.

Plan

As this is a breaking change, we will have to work on this in parallel:

1. create a new main branch (targeting Neovim nightly only) for the new slimmed down installation functionality;
2. once that has a working prototype, move queries and make the breaking changes for upstream alignment;
3. at some point, move development to the new branch (only bugfixes on master; parser addition and query updates target main and will be backported if desired);
4. after Neovim 0.10 is released, main will become the new default branch; the current master branch will remain (but frozen) for people requiring backwards compatibility;
5. make an 1.0 release targeting Neovim 0.10, and follow a proper release schedule (semver or otherwise) from then on
6. profit! 🥳

[theHamsta]

Having both main and master is probably confusing. Would suggest develop for the new thing until it becomes final. It could become main when master is dead.

[clason]

I don't think renaming branches is a great idea; there's some things that don't play nicely with it (e.g., Github Actions).