Improve syntax highlighting using a Language Server Protocol implementation

[ggrossetie]

We currently rely on a TextMate grammar but it cannot properly parse AsciiDoc.

I think we should migrate to semantic highlighting: code.visualstudio.com/api/language-extensions/semantic-highlight-guide or LSP: code.visualstudio.com/api/language-extensions/language-server-extension-guide

If someone wants to give it a try, please feel free to do so!

Related issues:

• syntax highlighting bug with unix shell code sections such as [source,bash] #685
• Syntax highlighting spills out of source block #319
• Alternative code block breaks syntax hightlighting #439
• Inline Syntax Highlighting breakes with escaped ] #375
• Syntax Highlighting lost after using ListingBlock inside an OpenBlock #389
• Syntax highlighting broken by some JS source block #403
• Syntax highlighting broken for inline code and admonition blocks #342
• Unterminated #ifndef causes syntax highlighting to go wrong outside the source block #713
• syntax highlighter of source code block fails when using options #793
• Apostrophes (single quotes) triggering weird highlighting. #804
• Syntax highlighting broken if a variable is used with bash source #875

[ggrossetie]

It's worth noting that semantic tokens do not support embedded languages:

microsoft/vscode#163292

In other words, source blocks won't be properly highlighted in AsciiDoc.

I don't think we should adopt the semantic tokens feature until embedded languages are supported.

I guess our only solution right now is to use LSP.

[ViToni]

Is this basically the same issue as #625 ?

[ggrossetie]

Initially, I thought it would be possible to implement a syntax highlighter using semantic token provider, that's why I've created a new issue.

Since we need to use LSP, it's indeed a duplicate of #625. I think we should close #625 in favor of this one.

[ViToni]

Looking into LSP has been on my ToDo list for a long time. This issue might be somethIng to get this action item checked off. Wondering though if it might be better to have a dedicated project for that as a LSP implementation (e.g. asciidoctor-lsp) could be used independently of this extension.

[ggrossetie]

@ViToni If you want to work on it that would be awesome! Yes, we should create a dedicated project! In case you've missed it, the AsciiDoc specification is under way:

• Community chat: https://asciidoc.zulipchat.com/
• AsciiDoc Lang Repository: https://gitlab.eclipse.org/eclipse/asciidoc-lang/asciidoc-lang/

The current syntax highlighter is broken and we could do so much more with a proper LSP (suggestion, lint, reformat and so on)

[ViToni]

@ggrossetie I'd try to get my feets wet. Not sure how far I get, but at least I'd like to give it a try.
I'll join the chat in the next days.

[ViToni]

Related to: asciidoctor/asciidoctor#3630