Jupyter Notebook CI fail

[thomasbtf]

For CI log see https://github.com/IKIM-Essen/uncovar/runs/4291269611?check_suite_focus=true#step:7:689

[Mon Nov 22 19:54:14 2021]
rule call_strains_kallisto:
    input: results/benchmarking/quant/non-cov2-NC_006213, results/benchmarking/nonhuman-reads/non-cov2-NC_006213.1.fastq.gz
    output: results/benchmarking/tables/strain-calls/non-cov2-NC_006213.strains.kallisto.tsv
    log: logs/benchmarking/call-strains/non-cov2-NC_006213.log
    jobid: 210
    wildcards: date=benchmarking, sample=non-cov2-NC_006213

Activating conda environment: /github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948
[NbConvertApp] ERROR | Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)

Failed validating 'additionalProperties' in code_cell:

On instance['cells'][0]:
{'cell_type': 'code',
 'execution_count': None,
 'id': '1b1e65a0',
 'metadata': {'tags': ['snakemake-job-properties']},
 'outputs': ['...0 outputs...'],
 'source': '\n'
           '######## snakemake preamble start (automatically inserted, do '
           'n...'}
[NbConvertApp] ERROR | Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)

Failed validating 'additionalProperties' in code_cell:

On instance['cells'][0]:
{'cell_type': 'code',
 'execution_count': None,
 'id': 'e05e13d4',
 'metadata': {'tags': ['snakemake-job-properties']},
 'outputs': ['...0 outputs...'],
 'source': '\n'
           '######## snakemake preamble start (automatically inserted, do '
           'n...'}
Traceback (most recent call last):
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/traitlets.py", line 537, in get
    value = obj._trait_values[self.name]
KeyError: 'kernel_dirs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_core/application.py", line 264, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 361, in start
    self.convert_notebooks()
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 533, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 498, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 427, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 186, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 204, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 323, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 80, in preprocess
    with self.setup_kernel():
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/client.py", line 460, in setup_kernel
    self.start_new_kernel(**kwargs)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/util.py", line 84, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/util.py", line 62, in just_run
    return loop.run_until_complete(coro)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/client.py", line 416, in async_start_new_kernel
    await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/nbclient/util.py", line 96, in ensure_async
    result = await obj
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/manager.py", line 657, in start_kernel
    kernel_cmd, kw = self.pre_start_kernel(**kw)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/manager.py", line 292, in pre_start_kernel
    kernel_cmd = self.format_kernel_cmd(extra_arguments=extra_arguments)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/manager.py", line 210, in format_kernel_cmd
    cmd = self.kernel_spec.argv + extra_arguments
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/manager.py", line 114, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/kernelspec.py", line 228, in get_kernel_spec
    resource_dir = self._find_spec_directory(kernel_name.lower())
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/kernelspec.py", line 204, in _find_spec_directory
    for kernel_dir in [kd for kd in self.kernel_dirs if os.path.isdir(kd)]:
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/traitlets.py", line 577, in __get__
    return self.get(obj, cls)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/traitlets.py", line 540, in get
    default = obj.trait_defaults(self.name)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/traitlets/traitlets.py", line 1580, in trait_defaults
    return self._get_trait_default_generator(names[0])(self)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/jupyter_client/kernelspec.py", line 158, in _kernel_dirs_default
    dirs.append(os.path.join(get_ipython_dir(), 'kernels'))
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/site-packages/IPython/paths.py", line 70, in get_ipython_dir
    os.makedirs(ipdir)
  File "/github/workspace/.tests/.snakemake/conda/ef3802cb519924eb937d04e949367948/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/github/home/.ipython'

[thomasbtf]

johkoester:
I think someone here has the same problem: jupyterlab/jupyterlab#9637

Probably an incompatibility of the used jupyter version to the python version.

I don't know exactly which constellation is causing the problem. Seems to happen with nbconvert...

[thomasbtf]

Will be fixed with snakemake/snakemake#1280 🎉

[thomasbtf]

cc: @AKBrueggemann

[thomasbtf]

Snakemake v6.11.1 is released. CI works again.