SNAKEMAKE_PROFILE environment variable does not play well with Slurm

[holtgrewe]

Snakemake version
v6.15.5

Describe the bug
Slurm is commonly configured to forward all environment variables. When defining env SNAKEMAKE_PROFILE then Slurm will by default forward this through sbatch. This causes the inner snakemake inside the job script to have args.profile set to a non-None value and subsequently snakemake existing with

Error: you need to specify the maximum number of jobs to be queued or executed at the same time with --jobs or -j.

Logs

Minimal example

• Have a Slurm cluster
• Use the following Snakefile

rule all:
  input: 'f.1', 'f.2', 'f.3'

rule one:
  output: 'f.{i}'
  shell: 'set -x; touch {output}'

• Use the current snakemake-profiles/slurm profile (e.g., to create slurm-bih profile)
• Start with export SNAKEMAKE_PROFILE=$HOME/work/Development/slurm-bih ; snakemake -j1 f.2.

Additional context
Specifying --profile=$HOME/work/Development/slurm-bih works.

The generated jobscript looks sane:

#!/bin/bash
# properties = {"type": "single", "rule": "one", "local": false, "input": [], "output": ["f.2"], "wildcards": {"i": "2"}, "params": {}, "log": [], "threads": 1, "resources": {"tmpdir": "/data/gpfs-1/users/holtgrem_c/scratch/tmp/hpc-cpu-104.cubi.bihealth.org"}, "jobid": 0, "cluster": {}}
 cd /data/gpfs-1/work/users/holtgrem_c/Development/snakemake-example && \
/data/gpfs-1/users/holtgrem_c/work/miniconda3/envs/snakemake-dev/bin/python3.8 \
-m snakemake f.2 --snakefile /data/gpfs-1/work/users/holtgrem_c/Development/snakemake-example/Snakefile \
--force --cores all --keep-target-files --keep-remote --max-inventory-time 0 \
--wait-for-files '/data/gpfs-1/work/users/holtgrem_c/Development/snakemake-example/.snakemake/tmp.oonr6sp8' --latency-wait 60 \
 --attempt 1 --force-use-threads --scheduler greedy \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
   --allowed-rules one --nocolor --notemp --no-hooks --nolock --scheduler-solver-path /data/gpfs-1/users/holtgrem_c/work/miniconda3/envs/snakemake-dev/bin \
--mode 2  --default-resources "tmpdir=system_tmpdir"  && exit 0 || exit 1

[holtgrewe]

I suggest to resolve this by removing the environment variable from calls to args.cluster.