Consider separating the Jupyter environment

[jitseniesen]

Working with notebooks in Spyder involves three processes:

1. The Spyder process itself
2. The notebook server based on Jupyter Server, translating the notebook to HTML + JavaScript which is then displayed in Spyder
3. The kernel, which executes the commands inside the notebook

At the moment, Spyder and the notebook server both run in the Spyder environment, while the kernel can run in a different environment (as set in the Spyder preferences).

Should we allow for the notebook server to be run in a different environment?

That means that the three processes can run in three different environments:

1. The Spyder process runs in an environment with Spyder and all its plugins, including spyder-notebook.
2. The notebook server runs in an environment with Jupyter packages (Lab, Server, Notebook) and any Jupyter extensions that the user wants. The Jupyter environment also needs to have the Jupyter Server extension which is currently included in spyder-notebook.
3. The kernel runs in an environment with spyder-kernels and any libraries that the user requires.

To achieve this, we need to split off the Jupyter Server extension currently included in spyder-notebook, into a new package, which I will call spyder-notebook-server.

The advantage is a separation of Spyder and Jupyter environments: upgrading Jupyter packages and installing Jupiter extension will not interfere with Spyder. This is especially relevant if we distribute a stand-alone Spyder installer with Spyder plugins and we don't want the user to install packages (including Jupyter extensions) inside the Spyder environment.

The disadvantage is added complexity for the user who needs to understand which package goes where (and we need to explain this). There is some added complexity for us, in that we need to maintain two packages instead of one and we need to add an option to the plugin preferences for choosing the Jupyter environment, but that seems all quite manageable and it leads arguably to a cleaner design.

[ccordoba12]

I think this is a good idea, but only if we make it an optional feature. In other words, installing Spyder-notebook should install Spyder-notebook-server in the same env so that the plugin works out of the box. However, if advanced users want to use the server installed in a different environment (with possibly their own JupyterLab extensions), then they should go to Preferences and enable an option for that.

[jitseniesen]

installing Spyder-notebook should install Spyder-notebook-server in the same env so that the plugin works out of the box

Yes, that's also what I was thinking, just like spyder depends on spyder-kernels even though spyder-kernels (I think) only needs to be installed in the environment that Spyder uses to run Python code and not in the Spyder environment itself. It does negate some of the advantage of separating the Spyder and Jupyter environments but it makes installation easier, especially for people less familiar with environments.

[ccordoba12]

Yes, that's also what I was thinking, just like spyder depends on spyder-kernels even though spyder-kernels (I think) only needs to be installed in the environment that Spyder uses to run Python code and not in the Spyder environment itself

Yep, that's the idea.

It does negate some of the advantage of separating the Spyder and Jupyter environments but it makes installation easier, especially for people less familiar with environments.

And also because it'd require an extra, non-trivial and manual step to have Spyder-notebook up and running.