Invalid threads definition: entries have to be defined as RULE=THREADS pairs (with THREADS being a positive integer). Unparseable value

[descostesn]

Hi,

Did you notice that set-threads do not work with a recent version of snakemake? It looks long but you just have to copy/paste. Here is a MRE:

mkdir snakemake-test && cd snakemake-test
touch snakeFile
mkdir profile && touch profile/config.yaml && touch profile/status-sacct.sh && chmod +x profile/status-sacct.sh
mkdir envs && touch envs/environment1.yaml && touch envs/environment2.yaml

In envs/environment1.yaml:

channels:
  - bioconda
  - conda-forge
dependencies:
  - snakemake-minimal=7.3.8
  - pandas=1.4.2
  - peppy=0.31.2
  - eido=0.1.4

In envs/environment2.yaml:

channels:
  - bioconda
  - conda-forge
dependencies:
  - snakemake-minimal=6.15.1
  - pandas=1.4.2
  - peppy=0.31.2
  - eido=0.1.4

In snakeFile:

onstart:
    print("\t Creating jobs output subfolders...\n")
    shell("mkdir -p jobs/downloadgenome")


GENOME = "mm39"
PREFIX = "Mus_musculus.GRCm39"

rule all:
  input:
    expand("data/fasta/{genome}/{prefix}.dna.chromosome.1.fa", genome=GENOME, prefix=PREFIX)

rule downloadgenome:
  output:
    "data/fasta/{genome}/{prefix}.dna.chromosome.1.fa"
  params:
    genomeLinks = "http://ftp.ensembl.org/pub/release-106/fasta/mus_musculus/dna/Mus_musculus.GRCm39.dna.chromosome.1.fa.gz"
  threads: 4
  shell:
    """
    wget {params.genomeLinks}
    gunzip {wildcards.prefix}.dna.chromosome.1.fa.gz
    
    mkdir -p data/fasta/{wildcards.genome}
    mv {wildcards.prefix}.dna.chromosome.1.fa data/fasta/{wildcards.genome}
    """

In profile/config.yaml:

snakefile: snakeFile
latency-wait: 60
printshellcmds: True

max-jobs-per-second: 1
max-status-checks-per-second: 10
jobs: 400
jobname: "{rule}.{jobid}"
cluster: "sbatch --output=\"jobs/{rule}/slurm_%x_%j.out\" --error=\"jobs/{rule}/slurm_%x_%j.log\" --cpus-per-task={threads} --ntasks=1 --parsable" # --parsable added for handling the timeout exception
cluster-status: "./profile/status-sacct.sh" #  Use to handle timeout exception, do not forget to chmod +x

set-threads:
  - downloadgenome=2

In profile/status-sacct.sh:

#!/usr/bin/env bash

# Check status of Slurm job

jobid="$1"

if [[ "$jobid" == Submitted ]]
then
  echo smk-simple-slurm: Invalid job ID: "$jobid" >&2
  echo smk-simple-slurm: Did you remember to add the flag --parsable to your sbatch call? >&2
  exit 1
fi

output=`sacct -j "$jobid" --format State --noheader | head -n 1 | awk '{print $1}'`

if [[ $output =~ ^(COMPLETED).* ]]
then
  echo success
elif [[ $output =~ ^(RUNNING|PENDING|COMPLETING|CONFIGURING|SUSPENDED).* ]]
then
  echo running
else
  echo failed
fi

Now build the conda environments:

cd envs
conda env create -p ./smake --file environment1.yaml
conda env create -p ./smake2 --file environment2.yaml
cd ..

If you run the whole thing with smake2 (snakemake snakemake-minimal=6.15.1) it indeeds run the job with 2 CPUs:

conda activate envs/smake2
snakemake --profile profile/
conda deactivate
rm -r data
rm -r jobs

If you do the same thing with smake (snakemake-minimal=7.3.8), it will crash with the error: Invalid threads definition: entries have to be defined as RULE=THREADS pairs (with THREADS being a positive integer). Unparseable value: '{downloadgenome :'.

conda activate envs/smake
snakemake --profile profile/
more jobs/downloadgenome/*log

I tried many things without success to solve the problem...

[johanneskoester]

Thanks for reporting! This has just been fixed in PR #1617.