"module command not found" using Snakemake with Slurm

[kevinrue]

Snakemake version

8.10.6

Describe the bug

Recently Snakemake started to fail loading environment modules on my SLURM cluster, without any known change to the pipeline, the cluster, or my bashrc.

With a very simple Snakefile below:

rule my_cellranger:
    envmodules:
        "cellranger/7.2.0"
    shell:
        "cellranger --version"

And running the pipeline using:

snakemake --use-envmodule

I am getting the error:

/usr/bin/bash: line 1: module: command not found
/usr/bin/bash: line 1: cellranger: command not found

Logs

Full log

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided remote nodes: 1
Provided resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954
Select jobs to execute...
Execute 1 jobs...

[Thu Apr 11 09:54:12 2024]
rule my_cellranger:
    jobid: 0
    reason: Rules with neither input nor output files are always executed.
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_partition=short, runtime=10

cellranger --version
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Provided resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954
Select jobs to execute...
Execute 1 jobs...

[Thu Apr 11 09:54:18 2024]
localrule my_cellranger:
    jobid: 0
    reason: Rules with neither input nor output files are always executed.
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=/var/scratch/albrecht/1194465, slurm_partition=short, runtime=10

cellranger --version
Activating environment modules: cellranger/7.2.0
/usr/bin/bash: line 1: module: command not found
/usr/bin/bash: line 1: cellranger: command not found
[Thu Apr 11 09:54:18 2024]
Error in rule my_cellranger:
    jobid: 0
    shell:
        cellranger --version
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Storing output in storage.
WorkflowError:
At least one job did not complete successfully.

Minimal example

See above.

Additional context

Having pasted the module load command and the cellranger command in a Slurm script and submitted it using sbatch, that worked fine, so it doesn't seem to be a SLURM issue.

Might be unrelated but I also noticed a new issue with my VScode failing to launch login shells as it used to (see microsoft/vscode-remote-release#1671 (comment)), I had to switch bash -l to bash --login. Could Snakemake be affected and not launch a login shell (which is necessary to access the module command for me)?

Actually, if Snakemake indeed run Bash strict mode, how come I get two error messages?

/usr/bin/bash: line 1: module: command not found
/usr/bin/bash: line 1: cellranger: command not found

Shouldn't it terminate at the first and never make it to the second?

[kevinrue]

Update

snakemake --use-envmodules --executor local

works just fine.

Does that mean it's something goes sideways in snakemake-executor-plugin-slurm ?

[cmeesters]

solved