Evaluation: moving from Xtext to ANTLR for integration in metafacture-core

[fsteeg]

Initial setup with ANTLR grammar:

• Copy generated grammar to src/main/antlr/
• Run ./gradlew generateGrammarSource
• See output in metafix/build/generated-src

While the basic idea of reusing the ANTLR grammar generated by Xtext seems to work, there are some issues:

• The generated grammar is very complex when compared to the original Xtext grammar
• The generated grammar only creates the parser and the lexer class, all domain classes are created by Xtext, with dependencies on Xtext and EMF classes
• Even the generated grammar itself contains references to Xtext/EMF classes

Overall, I see these different ways we could go, depending on our priorities:

1. Move only the build and repo structure to metafacture-core, retaining Xtext as a dependency, but integrated into the metafacture-core build (would reach our goal of having everything in metafacture-core, but we'd keep the two very different implementations for Fix and Flux, and the additional dependencies for the metafix module)
2. We switch to the generated grammar, but keep the class-level dependencies to Xtext/EMF; kind of an in-between approach where we'd have both the repo integration in metafacture-core and basically the same implementation approach for Fix and Flux, with an ANTLR grammar as the basis, even though it'd look very different (in principle, we could harmonize them over time and remove the Xtext dependencies bit by bit, however I have no idea if that's something we'd actually do)
3. We rewrite the Fix grammar from scratch in ANTLR, corresponding to the Flux implementation, and update our code to use the classes generated from that grammar; this would be a major effort but lead to full integration and unified implementation in metafacture-core

Both 2) and 3.) would also require considering how to implement the Language Server which is currently generated by Xtext, and which we use for the VSC extensions.

Perhaps 1.) would be a good first step, and we'll see how to proceed from there (to 2, to 3, or stay)?

[blackwinter]

Thanks for the initial evaluation! Agree with your assessment and conclusion. From a "purity" standpoint, I would prefer option 3), though. But I have no idea how time-consuming it would be.

There were also some practical reasons to move away from Xtext (#90, hbz/lobid-resources#1462), aside from the integration with metafacture-core. But if it turns out that switching to pure ANTLR is not feasible, then we'd probably just have to live with it.

[katauber]

According to my research there's no easy way for an ANTLR grammar to build an Language Server like there is for Xtext like we did for the VSC extension following this instructions. So if we switch to ANTLR we need to write a Language Server ourselves or do it like for the flux extension.

[fsteeg]

We discussed this in our meeting today and depending on our priorities we could either start with 1), the move into the core repo (if our priority is to unify the repo and build) or with 3), moving to plain ANTLR (if our priority is to get rid of the Xtext dependencies). So basically, these two are independent and would not block each other.

We also discussed that it might make sense to clean up and upgrade Gradle in the metafacture-core build before 1), see metafacture/metafacture-core#484.

And to assess what we'd actually lose initially when switching to plain ANTLR, it would be great to have an overview of the functionality that's currently coming from the language server (vs. highlighting implemented in the VSC extension itself), assigned @katauber for that.

[blackwinter]

start with 1), the move into the core repo (if our priority is to unify the repo and build)

This might actually be blocked by Xtext not being compatible with Gradle 8.x (see #283). Will check if this is still the case.

[katauber]

After my research I think we do not loose much when switching to ANTLR because the Xtext Language server generated by the gradle plugin is very simple. The server only evaluates if a fix file meets the requirements of the grammar file (e.g. if there is and end for every unless).
The current highlighting of the vsc extension is coded in the extension itself. Since LSP Version 3.16 there are semantic tokens which hopefully can replace the highlighting on client side by a highlighting on server side. (Highlighting on client side results in two different grammar files: one for Monaco Editor (MONARCH) and one for VSC extension (TextMate))
So no matter whether we use ANTRL or Xtext we will have to write a language server if we want more functionallity for the language server than evaluating the grammar (e.g. go to defintions, find references, hover, ...).
This language server can be used by the Monaco Editor (wen need to add Monaco languageclient) and by the vsc extension (vsc extension already uses the simple Xtext language server).