[pilatus] Update JupyterLab with CPE 23.12

[lucamar]

The extensions are installed correctly, but the sanity check fails with the current import error:

File "/capstor/scratch/cscs/lucamar/easybuild/software/jupyterlab/4.2.0-cpeGNU-23.12/lib/python3.11/site-packages/cryptography/exceptions.py", line 9, in <module>
    from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
ImportError: cannot import name 'exceptions' from 'cryptography.hazmat.bindings._rust' (unknown location)

A regressions test of the extensions might be needed to ensure that they work as expected and some variables in the recipe might need to be updated: e.g. cray-R/4.3.1 is available on the system, while 41 is referenced in the recipe.

[lucamar]

A similar issue has been reported in 9060: I tried to follow one of the workarounds proposed in the issue that I linked, namely pip install cryptography==3.4.8, but I get other ModuleNotFoundError: No module named ... errors, with patckages that are installed as extensions. Maybe the PYTHONPATH needs to be adjusted? Do you have any advice @omlins, @rsarm?

[lucamar]

The error reported by the CI/CD is due to the instability of the download of nodejs from http://nodejs.org:

== FAILED: Installation ended unsuccessfully (build directory: /tmp/jenscscs/f1b2152/nodejs/22.1.0/cpeGNU-23.12): build failed (first 300 chars): Unexpected error occurred when trying to download http://nodejs.org/dist/v22.1.0/node-v22.1.0.tar.gz to /capstor/scratch/cscs/jenscscs/TestingEB/pilatus/f1b2152-6551/sources/n/nodejs/node-v22.1.0.tar.gz: 'Failed to write to /capstor/scratch/cscs/jenscscs/TestingEB/pilatus/f1b2152-6551/sources/n/node (took 1 min 2 secs)

[lucamar]

The recipe is still work in progress, but the build succeeded locally and can be tested as follows on Pilatus:

ml cray
ml use /capstor/scratch/cscs/lucamar/easybuild/modules/all/Core
ml cpeGNU jupyterlab

Please note the following:

• I added the option 'pip_no_index': False to let pip download the dependencies of each extension: there is no control on the versions installed, but you don't need to list all dependencies individually (a total of 181 packages are downloaded)
• I skipped sanitycheck, otherwise the sanity check was failing for the extensions that downloaded additional packages and I could not find a workaround
• I commented out the creation of the "CSCS virtualenv" in the postinstallcmds, since the versions of the packages need to be cross-checked
• The versions of the extensions are retrieved from a local install of the same Python packages using pip, in order to avoid conflicts: this is why the versions installed are not the latest ones available on PyPI and dask_labextension is skipped for now

[lucamar]

I have added dask_extension and uncommented the commands to create the CSCS virtualenv, after cross-checking the versions of the packages manually installed in the virtual environment. Please review the recipe and test the modulefile (the Jupyterlab TDS server should likely be adjusted to use the version provided by this modulefile for JupyterHub) .

[omlins]

when trying to launch a server on jupyter-pilatus.cscs.ch, I get:
Spawn failed: sbatch: error: Batch job submission failed: Unable to contact slurm controller (connect failure)

[twrobinson]

Hi, thanks Luca. I'm still waiting for the JupyterHub server to be updated for testing on pilatus, see https://jira.cscs.ch/projects/SD/queues/custom/698/SD-61236.

[lucamar]

The last build failed due to a download error:

Unexpected error occurred when trying to download http://nodejs.org/dist/v22.1.0/node-v22.1.0.tar.gz
Failed to write to /capstor/scratch/cscs/jenscscs/TestingEB/pilatus/f5fb2ad-6566/sources/n/node

[lucamar]

I am going to merge this pull request in order to deploy and test JupyterLab and the required dependencies on Pilatus, since the issues related to SD-61236 have been addressed.