Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a topic on jupyter-lmod in documentation #640

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
130 changes: 130 additions & 0 deletions docs/source/370_jupyter_lmod.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
Jupyter and Lmod
================

It is possible to use Lmod inside of Jupyter with the terminal or jupyter-lmod.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a link to your repo from the start 🙂

Suggested change
It is possible to use Lmod inside of Jupyter with the terminal or jupyter-lmod.
It is possible to use Lmod inside of Jupyter with its terminal or the Jupyter extension `jupyter-lmod <https://github.com/cmd-ntrf/jupyter-lmod>`_.


jupyter-lmod is a frontend and server extension for Jupyter (notebook and jupyterlab).
The extension graphically provides most of Lmod's functions:
- Browser and filter available modules
- Load and unload modules
- Display a module content
- Create and load collections

The extension is located in the 'Softwares' tab of the main interface in Jupyter Notebook,
while in JupyterLab, it can be accessed as a sidepanel by clicking on the Lmod logo icon
located in the sidebar.

Installation
~~~~~~~~~~~~

To install jupyter-lmod extension: ::

pip install jupyterlmod
cmd-ntrf marked this conversation as resolved.
Show resolved Hide resolved

This enables the extension for both Jupyter Notebook and JupyterLab.

How does it work?
~~~~~~~~~~~~~~~~~

jupyter-lmod uses lmod python interface to generate Python code
that modifies the environment variables of the Python process running Jupyter. All child
processes of Jupyter (including kernels) created after Lmod calls inherit
the environment variables as defined by the set of loaded modules.
cmd-ntrf marked this conversation as resolved.
Show resolved Hide resolved
Comment on lines +29 to +32
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is a bit confusing, I suggest to simplify it:

Suggested change
jupyter-lmod uses lmod python interface to generate Python code
that modifies the environment variables of the Python process running Jupyter. All child
processes of Jupyter (including kernels) created after Lmod calls inherit
the environment variables as defined by the set of loaded modules.
jupyter-lmod uses lmod python interface to generate Python code
that modifies the environment variables of the Python process running Jupyter. All child
processes of Jupyter (*e.g.* newly spawned kernels) will inherit the environment
as currently defined by the set of loaded modules.


In concrete terms, kernels in use by active notebooks will not be aware of changes
made to the environment (e.g. **$PATH**, **$PYTHONPATH**, **$LD_LIBRARY_PATH**) by modules
loaded after they were spawned. Depending on the Jupyter version, a manual kernel restart
or a complete kernel shutdown/start cycle is needed to make an active notebook aware of changes
to the environment variables.

Lmod environment variable configuration for jupyter-lmod
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The plugin requires the environment variable **$LMOD_CMD** to be defined when Jupyter starts,
plus all environment variables required by Lmod to work properly (i.e.: **$MODULEPATH**).

Some JupyterHub spawners do not setup the complete user's environment, leaving **$LMOD_CMD**
and **$MODULEPATH** undefined. To address this issue, the environment variables have to be
defined in **jupyterhub_config.py** using
`c.Spawner.environment <https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.environment>`_.

If the environment variables are defined before JupyterHub launch, you can add their
names only to `c.Spawner.env_keep <https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.env_keep>`_
instead.
Comment on lines +51 to +53
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically these variable are only needed in the single-user server. JupyterHub itself does not need them.

Suggested change
If the environment variables are defined before JupyterHub launch, you can add their
names only to `c.Spawner.env_keep <https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.env_keep>`_
instead.
If the environment variables are already defined before JupyterHub is launched
or before the single-user server starts (Jupyter Notebook or JupyterLab),
you can propagate these variable by adding their names to
`c.Spawner.env_keep <https://jupyterhub.readthedocs.io/en/stable/api/spawner.html#jupyterhub.spawner.Spawner.env_keep>`_ instead.


Controlling launcher items with jupyter-lmod
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

jupyter-lmod can add a kernel or a server proxy to the Jupyter launcher
following module loading if the module matches some specifications.

kernels
+++++++

If a module adds one or more absolute paths to **$JUPYTER_PATH**, Jupyter's kernel spec manager
will be notified of the changes. If the added paths contain kernel spec files (e.g: **kernel.json**),
the kernels will be added to the available choices in Jupyter interface.

The following example demonstrates this feature with Julia. Here, we have an **ijulia-kernel/1.8** module
that prepends a path to **$JUPYTER_PATH**. ::

whatis("Description: IJulia kernelspec for Julia 1.8")
whatis("Homepage: https://julialang.org/")
whatis("URL: https://julialang.org/")
conflict("ijulia-kernel")
depends_on("julia/1.8.1")
depends_on("ijulia/.1.24.0")
prepend_path("JUPYTER_PATH","/software/Core/ijulia-kernel/1.8/share/jupyter")


In **/software/Core/ijulia-kernel/1.8/share/jupyter/kernels/julia-1.8**, there are four files ::

kernel.json logo-32x32.png logo-64x64.png logo-svg.svg

and **kernel.json** looks like this ::

{
"display_name": "Julia 1.8.1",
"argv": [
"/software/Core/julia/1.8.1/bin/julia",
"-i",
"--color=yes",
"/software/Core/ijulia/1.24.0/packages/IJulia/6TIq1/src/kernel.jl",
"{connection_file}"
],
"language": "julia",
"env": {},
"interrupt_mode": "signal"
}

The folders and files under **share/jupyter/kernels** are typically generated by a kernelspec installer.


server proxies
++++++++++++++

Jupyter Server Proxy is an extension for Jupyter and JupyterLab that let users run arbitrary
external processes (such as RStudio, Shiny Server, Syncthing, PostgreSQL, Code Server, etc)
alongside their notebook server and provide authenticated web access to them using a path
like /rstudio next to others like /lab. It also provides buttons in the launcher to launch
these external processes.

When jupyter-lmod and jupyter-server-proxy are both installed, it is possible for the launcher
button of an external process to appear only when its corresponding module is loaded.
In order to do so, the frontend extension of jupyter-server-proxy must be deactivated as
the control of the launcher will be entrusted to jupyter-lmod. ::

jupyter nbextension disable --py jupyter_server_proxy --sys-prefix
jupyter labextension disable @jupyterlab/server-proxy
jupyter labextension disable jupyterlab-server-proxy

Then, the name of each `server proxy <https://jupyter-server-proxy.readthedocs.io/en/latest/server-process.html#server-process-options>`
must match the name of module. When a module with a name corresponding to a server
proxy name is loaded, jupyter-lmod will add a button to the launcher to start that process.

References
~~~~~~~~~~

For more information, refer to jupyter-lmod github repo:
https://www.github.com/cmd-ntrf/jupyter-lmod

1 change: 1 addition & 0 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ Advanced Topics
340_inherit
350_community
360_check_syntax
370_jupyter_lmod

Internal Structure of Lmod
^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down