Support for Multiple Languages in a Single Notebook Document

[samir-abis-everest]

Hello Langium Community,

I'm working on a project that involves using Langium to define a language for metadata and validation rules within a notebook environment. We've encountered a challenge that I believe could benefit from your insights and potential solutions.

Problem Description:

In our setup, we have a notebook document that can contain cells with different languages (e.g., metadata, validations, permissions). Each cell within the notebook is expected to have its own service instance based on the language specified in the cell. However, we've found that the getServices method in our custom service registry implementation (CustomServiceRegistry) is not capable of selecting the correct services for individual cells within the notebook. This limitation arises because getServices is designed to work with the URI of the entire notebook file, rather than the specific cell.

Current Implementation:

Our current approach involves creating a new instance of CustomCompiler for each cell to ensure each cell has its own service. However, this approach does not dynamically select the appropriate services based on the cell's language. Instead, it relies on the file extension of the notebook file, which is not sufficient for our use case.

Proposed Solution:

We've considered modifying the getServices method to accept an additional parameter for the cell's language. This way, we could dynamically select the appropriate services based on the cell's language. However, this approach requires changes to how getServices is called throughout our codebase, which might not be straightforward.

Questions:

1. Is there a recommended way to handle multiple languages within a single notebook document in Langium?
2. Are there any built-in mechanisms or best practices in Langium for dynamically selecting services based on the language of individual cells within a notebook?
3. Has anyone faced a similar challenge, and if so, how did you address it?

Additional Context:

• Our project is a monorepo containing a language project designed to be user-friendly for non-programmers working with it.
• We're using Langium for defining the AST and syntax of our language, along with language server support.
• The notebook environment is integrated into VS Code using a custom VS Code extension.

We're looking forward to any suggestions, insights, or potential solutions that could help us overcome this limitation. Thank you for your time and support.

[msujew]

Hey @samir-abis-everest,

This limitation arises because getServices is designed to work with the URI of the entire notebook file, rather than the specific cell.

As far as I know, we don't manipulate the URI that comes from vscode in any way. Therefore, it should contain the full cell URI (which is effectively just vscode-notebook-cell://<notebook-path>#<cell-handle>).

We've considered modifying the getServices method to accept an additional parameter for the cell's language.

I'm a bit confused about this: Do you have multiple languages - that are registered as separate languages - in VSCode that use the same language server? If so, how? I'm not aware of such a functionality in the implementation of vscode-languageclient.

[msujew]

That should actually work as well, yeah. I initially added the map because I recommended a different (async) approach, see the history of that comment. But that probably isn't even necessary.

As the lookup is be O(1) there shouldn't be any performance impact. At this point the text documents are already filled, so it's also not too late.

[msujew]

@samir-abis-everest @cdietrich I've adapted this approach in #1492, as this makes a lot of sense as the default behavior of Langium.

[cdietrich]

@msujew i have a follow up question/problem.
with the artifical uris we cannot reason on the uri in other services
in Notebooks you can enrich NotebookCells with metadata.
so we could store additional information.

do you know if this information is somewhere funnelled to the server as the the cells are as text document?

if no:
when we implement notebooks.synchronization.onDidOpenNotebookDocumentand similar the text documents are no longer mainted. so i assume we need to do this ourself

what is the proper way to have both the textdocuments populated and maintained and the langiumdocuments in the documentbuilder filled?

manually downcast and fill the map, then call update?

for (const cell of p1.cellTextDocuments) {
    // eslint-disable-next-line @typescript-eslint/no-unsafe-call, @typescript-eslint/no-explicit-any, @typescript-eslint/no-unsafe-member-access
    ;((langiumSharedServices.workspace.TextDocuments as any)._syncedDocuments as Map<string, unknown>).set(
      cell.uri,
      TextDocument.create(cell.uri, cell.languageId, 1, cell.text)
    )
    uris.push(URI.parse(cell.uri))
  }
  langiumSharedServices.workspace.DocumentBuilder.update(uris, []).catch(x => {
    console.error(x)
  })

or is there a way to make this nicer / better adapted?

[msujew]

The vscode-languageserver package exports a NotebookDocuments class, that should be used similar to the currently used TextDocuments class. One would need to use that in the LanguageServer service implementation and also adapt the LangiumDocumentsFactory to handle that better.

It probably makes sense to implement support for that natively in Langium. I've created #1496 to track this.

[cdietrich]

i guess the interesting point is to make it work "hybrid".
some normal and notebook documents in the same ws.

[msujew]

Luckily, Langium is rarely using the TextDocuments, and so just using a simple TextDocuments.get(uri) ?? NotebookDocuments.get(uri) lookup should work in most circumstances.

[cdietrich]

i see do you think we can reuse the text documents like

const nds = new NotebookDocuments(langiumSharedServices.workspace.TextDocuments)
nds.onDidChange(change => console.log(change))
nds.listen(connection)

or would the NotebookDocuments need its own text documents?

[msujew]

Uhm, haven't tried it yet, but I can imagine that this should work out well.