Are compile time kind and field ids available for the Rust bindings?

[VonTum]

Well, as the title states.

Is there some way to get compile time values for the kind and field ids for my custom language? This would allow the Rust optimizer to make much more efficient branch jump tables for compilers using tree-sitter as a compiler frontend.

Currently I am using the second-best option, namely simply requesting all ids ahead of time and storing them in a static data structure: https://github.com/pc2/sus-compiler/blob/3a3a5c9dfe5c5eeb8fde6ec7f49823af60d37f49/src/parser.rs#L84-L230

[WillLillis]

It doesn't appear to be possible with the current API. The rust binding id_for_node_kind() calls the C function ts_language_symbol_for_name(), which iterates through all of the given language's symbols and checks for a match via string comparison:

tree-sitter/lib/src/language.c

Lines 113 to 130 in cbcb51b

	TSSymbol ts_language_symbol_for_name(
	const TSLanguage *self,
	const char *string,
	uint32_t length,
	bool is_named
	) {
	if (!strncmp(string, "ERROR", length)) return ts_builtin_sym_error;
	uint16_t count = (uint16_t)ts_language_symbol_count(self);
	for (TSSymbol i = 0; i < count; i++) {
	TSSymbolMetadata metadata = ts_language_symbol_metadata(self, i);
	if ((!metadata.visible && !metadata.supertype) || metadata.named != is_named) continue;
	const char *symbol_name = self->symbol_names[i];
	if (!strncmp(symbol_name, string, length) && !symbol_name[length]) {
	return self->public_symbol_map[i];
	}
	}
	return 0;
	}

I'm sure there's plenty of hacky ways you could have these values available at compile time, but I imagine that would make changing to a different version of the grammar to be quite difficult and/or messy. In the past I tried looking at the generated code to have the node values known beforehand, but they didn't match up with the values returned at runtime. It was suggested to me in this issue to just use the Language::node_kind_for_id() and/or the Language::id_for_node_kind() methods.

[VonTum]

Hmm, I've been mulling it over, but I'm not finding a good place to fit the proc_macros into the generated crate. To start, it's not possible I believe to add them to the core library, because they require to be statically linked to the user-provided language, such that the compiler can access it.

The problem is that rust doesn't allow a crate to have both regular code, and proc_macros in the same crate.

I've thought to create a new folder in the generated project: bindings/rust/tree-sitter-PARSER_NAME-proc/Cargo.toml and bindings/rust/tree-sitter-PARSER_NAME-proc/lib.rs, but using it will then be rather ugly and brittle.

The user's Cargo.toml would then look something like this:

[dependencies]
tree-sitter-sus = {path = "path/to/sus"}
tree-sitter-sus-proc = {path = "path/to/sus/bindings/rust/tree-sitter-sus-proc"}

It seems nasty to have to reach down into the bindings folder. If you have any ideas?

[WillLillis]

I also tried adding it to the core and wasn't able to get anywhere. I haven't done much work with rust's macros though, so maybe I'm missing something?

I also tried your approach by adding another crate inside the bindings folder of a grammar and while that works I agree that it's a bit awkward/ brittle. Maybe another option would be to generate a tree-sitter-GRAMMAR_NAME-proc crate outside of the bindings directory? It could be published as a separate crate that then depends on tree-sitter-GRAMMAR_NAME. I was able to create a tree-sitter-c-proc crate with your macros, have it depend on the regular tree-sitter-c crate, and then add all of these dependencies to a sample program with the following Cargo.toml:

[package]
name = "ts_test"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tree-sitter = "0.22.2"
tree-sitter-c = "0.20.7"
tree-sitter-c-proc = { path = "/home/lillis/projects/tree-sitter-c-proc/" }

Of course once tree-sitter-c-proc (or any other grammar's proc variant) is published to crates.io we wouldn't have to specify a local path.

Maybe it would be good to open an issue and ask the maintainers for their thoughts? I've taken a look at the code used to generate the rust bindings and it doesn't look too bad to add to.

[VonTum]

So a major thing here too is that I believe generally subdirectories of Cargo projects should be dependencies of the main project, but in this case, the proc_macro library wants to depend on the main library. Maybe I just got it wrong, but in the proc macro's Cargo.toml, I had to make it depend on the parent like this:

[package]
name = "proc"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[lib]
proc-macro = true

[dependencies]

tree-sitter-testtree = {path = "../../.."}

tree-sitter = ">=0.22.2"

# proc_macro utils
syn = "1.0"
quote = "1.0"

You did it with an absolute path I see.

I don't quite know if the other maintainers would very much like an extra directory there either, like philosophically rust stuff should go in bindings/rust.

With a single subdirectory, it's more direct, and nicer to use but it'll still have to depend with path = ".."

Maybe we can just turn this discussion itself into an issue?

Also perhaps it is worth considering the alternative, of generating constants in a ids.rs file. To mirror parser.c:

enum ts_symbol_identifiers {
  sym_identifier = 1,
  anon_sym_COMMA = 2,
  anon_sym_COLON = 3,
  anon_sym_DASH_GT = 4,
  anon_sym_module = 5,
  sym_number = 6,
  anon_sym_COLON_COLON = 7,
  anon_sym_SQUOTE = 8,
  anon_sym_state = 9,
  anon_sym_gen = 10,
  anon_sym_PLUS = 11,
  anon_sym_DASH = 12,
  anon_sym_STAR = 13,
  anon_sym_BANG = 14,
  anon_sym_PIPE = 15,
  anon_sym_AMP = 16,
  anon_sym_CARET = 17,
  anon_sym_EQ_EQ = 18,
  .....
  sym_single_line_comment = 41,
  sym_multi_line_comment = 42,
  sym_source_file = 43,
  sym__comma = 44,
  sym_interface_ports = 45,
  sym_declaration_list = 46,
  sym_module = 47,
  sym_global_identifier = 48,
  sym_array_type = 49,
  sym__type = 50,
  sym_latency_specifier = 51,
  sym_declaration = 52,
  sym_unary_op = 53,
  sym_binary_op = 54,
  sym_array_op = 55,
  sym_func_call = 56,
  sym_parenthesis_expression_list = 57,
  sym_parenthesis_expression = 58,
  sym_array_bracket_expression = 59,
  sym__expression = 60,
  aux_sym__linebreak = 61,
  sym_block = 62,
  sym_write_modifiers = 63,
  sym_assign_to = 64,

Honestly I like the proc macro approach a bunch more

[VonTum]

I referenced this discussion in #3276

[WillLillis]

Maybe I just got it wrong, but in the proc macro's Cargo.toml, I had to make it depend on the parent like this:

Sorry for the confusion, the Cargo.toml in my last comment was for the example program I made using the new crate. The proc macro crate's Cargo.toml looks like this:

[package]
name = "tree-sitter-c-proc"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tree-sitter-c = "0.20.7"
quote = "1.0.36"
syn = "2.0.58"

[lib]
proc-macro = true

I don't quite know if the other maintainers would very much like an extra directory there either, like philosophically rust stuff should go in bindings/rust.

Yeah I could see that too. I haven't worked with too many large projects so I don't have a strong preference either way.

Honestly I like the proc macro approach a bunch more

Me too. I think it makes more sense to leverage the existing functionality since that's already been extensively tested.

I referenced this discussion in #3276

Awesome! Really hope this gets to move forward :)