docs: singularity sometimes uses system /tmp explanation

[Maarten-vd-Sande]

Description

The singularity conda packages dir does not respect the system tmp. Our servers have very small /tmp, so everyone makes their own tmp folder on /scratch/{user}. But snakemake does not respect the TMPDIR variable 😄

QC

• The PR contains a test case for the changes or the changes are already covered by an existing test case.
• The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.8% Duplication

[nickp60]

Hi @Maarten-vd-Sande can I get some clarification? I'm working on an HPC that uses LSF. I'm trying to convert a workflow to using per-rule conda envs in a global container. The pipeline works if executed with just --use-conda, but if I try to add --use-singularity I too am getting the following error:

$ snakemake --use-conda --use-singularity --snakefile ./vdb_16S/Snakefile --directory test_results/  --conda-frontend conda --singularity-args '--bind ${PWD}:${PWD},/data/brinkvd/:/data/brinkvd/'
onfig file /lila/home/watersn/GitHub/vdblab-pipelines/vdb_16S/runconfig.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Creating conda environment ../vdb_16S/envs/minimap2.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /lila/home/watersn/GitHub/vdblab-pipelines/vdb_16S/rules/../envs/minimap2.yaml:
Command:
SINGULARITYENV_CONDA_PKGS_DIRS=/tmp/conda/9e5d92b4-e854-4e32-a81f-4360c633867a singularity  exec --home /lila/home/watersn/GitHub/vdblab-pipelines/test_results --bind ${PWD}:${PWD},/data/brinkvd/:/data/brinkvd/ /data/brinkvd/.singularity/487a72edc080f28be9d6fa74dcc90b96.simg sh -c 'conda config --set channel_priority strict && conda env create --quiet --file "/data/brinkvd/conda/0ce7b5045b512458b3f2c4cbac3d7203.yaml" --prefix "/data/brinkvd/conda/0ce7b5045b512458b3f2c4cbac3d7203"'
Output:
Collecting package metadata (repodata.json): ...working... failed

NoWritablePkgsDirError: No writeable pkgs directories configured.
  - /tmp/conda/9e5d92b4-e854-4e32-a81f-4360c633867a

When I saw this PR I tried adding --contain to singularity args, getting the following

Creating specified working directory test_results/.
Config file /lila/home/watersn/GitHub/vdblab-pipelines/vdb_16S/runconfig.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Creating conda environment ../vdb_16S/envs/minimap2.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /lila/home/watersn/GitHub/vdblab-pipelines/vdb_16S/rules/../envs/minimap2.yaml:
Command:
SINGULARITYENV_CONDA_PKGS_DIRS=/tmp/conda/4d6f5119-423b-4121-af91-c53997323199 singularity  exec --home /lila/home/watersn/GitHub/vdblab-pipelines/test_results --bind ${PWD}:${PWD},/data/brinkvd/:/data/brinkvd/ --contain /data/brinkvd/.singularity/487a72edc080f28be9d6fa74dcc90b96.simg sh -c 'conda config --set channel_priority strict && conda env create --quiet --file "/data/brinkvd/conda/0ce7b5045b512458b3f2c4cbac3d7203.yaml" --prefix "/data/brinkvd/conda/0ce7b5045b512458b3f2c4cbac3d7203"'
Output:
Collecting package metadata (repodata.json): ...working... failed

NoSpaceLeftError: No space left on devices.

I seem to have plenty of space on my home directory and all bound locations. Is this something you ran into as well? Sorry for the lack of minimal example; any insight would be appreciated!

[Maarten-vd-Sande]

I'm not very experienced with singularity so I'm not sure. You can check if the mount temp option is on in the configuration (probably: /etc/singularity/singularity.conf).

When you use contain your home isn't mounted anymore, so perhaps you need to add -B $HOME as well?

[nickp60]

@Maarten-vd-Sande I'm not exactly sure what the story is this this either. I think I'm having several issues at once, will link back when I get to the bottom of this. Binding HOME didn't resolve the issue