LSP for slow-starting (remote) kernels

[Zsailer]

Description

I've been testing jupyterlab-lsp (4.3.x) with JupyterLab 3.6.x running on a system that

1. has slow (~1 min) kernel start up times.
2. runs kernels remotely from the server.

Due to (1), jupyterlab-lsp's Notebook adapter never connects on the first instance of the Notebook document. Instead, I wait for the kernel to fully start and connect, then close the document and re-open it for LSP to properly connect.

Due to (2), the LSP server is running in a different e.g. Python environment (i.e. on the server) than the (remote) kernel. This isn't a big deal, we can give some generic Python LSP power from language server running on Jupyter Server, and let the kernel give more useful help once it's warmed up.

Ideally, I would love to use jupyterlab-lsp to provide some basic (coming from the Jupyter server's python env) LSP hinting while the Notebook is starting (or even if no kernel is provided). Then, once the kernel is ready and connects, bring in the additional context from the kernel.

I don't think this is possible today, correct?

If so, I looked into writing a custom NotebookAdapter (from 4.x branch—I'm still stuck in JupyterLab 3.6.x land for awhile), but the interfaces and classes I need (e.g. NotebookAdapter and WidgetAdapter) are not exported.

So... (sorry for the long intro 😅) a few follow-up questions:

1. Would you be willing to export NotebookAdapter and WidgetAdapter in a 4.x release soon? I can submit the necessary PR and help with shipping, etc.
2. Does it make sense to fix for the slow, remote kernel case more generally?

(JupyterLab itself is pretty brittle when it comes to slow, remote kernels—so I think there is a broader discussion about how to improve the whole application for this usecase.)

Reproduce

Reproducing this is a bit of work. You need to mock a slow starting kernel (~1 min). I run this on a cluster, with kernels managed by a separate server from the Jupyter Server, then connect to the kernel remotely.

Expected behavior

Ideally, I would love to use jupyterlab-lsp to provide some basic (coming from the Jupyter server's python env) LSP hinting while the Notebook is starting (or even if no kernel is provided). Then, once the kernel is ready and connects, bring in the additional context from the kernel.

Context

• Operating System and version: all
• Browser and version: all
• JupyterLab version: 3.6.x
• Language server and version: 4.3.0

Required: installed server extensions

Paste the output from running `jupyter server extension list` (JupyterLab >= 3)
or `jupyter serverextension list` (JupyterLab < 3) from the command line here.
You may want to sanitize the paths in the output.

Required: installed lab extensions

Paste the output from running `jupyter labextension list` from the command line here.
You may want to sanitize the paths in the output.

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output (recommended for all interface issues)

Paste the output from your browser JavaScript console replacing the text in here.
To learn how to open the developer tools in your browser:

https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools#How_to_open_the_devtools_in_your_browser

If too many messages accumulated after many hours of working in JupyterLab,

consider refreshing the window and then reproducing the bug to reduce the noise in the logs.

[Zsailer]

If so, I looked into writing a custom NotebookAdapter (from 4.x branch—I'm still stuck in JupyterLab 3.6.x land for awhile), but the interfaces and classes I need (e.g. NotebookAdapter and WidgetAdapter) are not exported.

I see this was already requested:
#774 (comment)

[krassowski]

Would you be willing to export NotebookAdapter and WidgetAdapter in a 4.x release soon

If needs be, sure! But it would be perfect to have a more general solution so maybe it would not be needed?

Does it make sense to fix for the slow, remote kernel case more generally?

Certainly! Is there a way to get kernel_info earlier? If not we could use the one from notebook metadata, right?

[Zsailer]

If not we could use the one from notebook metadata, right?

Yeah, that's right.

In fact, this is what I was attempting to do when writing with my own NotebookAdapter. Determine the language from the Notebook's metadata and begin streaming LSP info right away... then try to ask the kernel for more context once its ready.

I'll work on a PR. 🙂

[krassowski]

Due to (2), the LSP server is running in a different e.g. Python environment (i.e. on the server) than the (remote) kernel.

Cross-linking with #1041 which discusses how some language servers can be configured to include packages from separate kernel environments in the inference process.