Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using option --drmaa_logs crashes snakemake #2819

Open
sven0schuierer opened this issue Apr 18, 2024 · 0 comments
Open

Using option --drmaa_logs crashes snakemake #2819

sven0schuierer opened this issue Apr 18, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@sven0schuierer
Copy link

Snakemake version
8.10.7

Describe the bug
If option --drma_logs is used, snakemake crashes

Minimal example

$ vi snakemake-tutorial-v8+.yaml
>>>name: snakemake-tutorial-v8+
channels:
  - conda-forge
  - bioconda
dependencies:
  - snakemake
  - snakemake-executor-plugin-drmaa<<<
$ mamba env create -f snakemake-tutorial-v8+.yaml

$ curl -L https://api.github.com/repos/snakemake/snakemake-tutorial-data/tarball -o snakemake-tutorial-data.tar.gz
$ tar xfz snakemake-tutorial-data.tar.gz
$ cd snakemake-snakemake-tutorial-data-be6ae5a
$ conda activate snakemake-tutorial-v8+
(snakemake-tutorial-v8+) $ snakemake --version
8.10.7
(snakemake-tutorial-v8+) $ mkdir envs
(snakemake-tutorial-v8+) $ vi envs/mapping.yaml
>>>channels:
  - bioconda
  - conda-forge
dependencies:
  - bwa =0.7.17
  - samtools =1.9<<<
(snakemake-tutorial-v8+) $ mkdir drmaa
(snakemake-tutorial-v8+) $ vi drmaa/config.yaml
>>>jobs: 100
latency-wait: 120

default-resources:
    mem_mb:    1000
    h_vmem_mb: 1200
    h_rt:      14400
    
executor: drmaa
drmaa-args: "-V -l h_rt={resources.h_rt} -l m_mem_free={resources.mem_mb}M -l h_vmem={resources.h_vmem_mb}M -pe smp {threads}"
drmaa-log-dir: "drmaa_logs"<<<
(snakemake-tutorial-v8+) $ vi Snakefile
>>>rule map_reads:
    input:
        genome_file = "data/genome.fa",
        fastq_file  = "data/samples/A.fastq"
    output:
        "results/mapped/A.bam"
    conda:
        "envs/mapping.yaml"
    threads: 3
    resources:
        mem_mb    = 3000,
        h_vmem_mb = 4000
    shell:
        "bwa mem -t {threads} {input} | samtools view -b - > {output}"<<<
(snakemake-tutorial-v8+) $ snakemake --cores 1 --use-conda --profile drmaa -p
Using profile drmaa for setting default command line arguments.
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Creating conda environment envs/mapping.yaml...
Downloading and installing remote packages.
Environment for /<path to snakemake-tutorial>/snakemake-tutorial-v8+/snakemake-snakemake-tutorial-data-be6ae5a/envs/mapping.yaml created (location: .snakemake/conda/eb625a9af57137e3718451485da755e0_)
Using shell: /usr/bin/bash
Provided remote nodes: 100
Job stats:
job          count
---------  -------
map_reads        1
total            1

Select jobs to execute...
Execute 1 jobs...

[Wed Apr 17 13:47:43 2024]
rule map_reads:
    input: data/genome.fa, data/samples/A.fastq
    output: results/mapped/A.bam
    jobid: 0
    reason: Missing output files: results/mapped/A.bam
    resources: mem_mb=3000, mem_mib=2862, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, h_vmem_mb=4000, h_rt=14400

bwa mem -t 1 data/genome.fa data/samples/A.fastq | samtools view -b - > results/mapped/A.bam
Traceback (most recent call last):

bwa mem -t 1 data/genome.fa data/samples/A.fastq | samtools view -b - > results/mapped/A.bam
Traceback (most recent call last):

  File "<path to snakemake-tutorial-v8+ env>/snakemake-tutorial-v8+/lib/python3.12/site-packages/snakemake/cli.py", line 2068, in args_to_api
    dag_api.execute_workflow(

  File "<path to snakemake-tutorial-v8+ env>/snakemake-tutorial-v8+/lib/python3.12/site-packages/snakemake/api.py", line 589, in execute_workflow
    workflow.execute(

  File "<path to snakemake-tutorial-v8+ env>/snakemake-tutorial-v8+/lib/python3.12/site-packages/snakemake/workflow.py", line 1247, in execute
    raise e

  File "<path to snakemake-tutorial-v8+ env>/snakemake-tutorial-v8+/lib/python3.12/site-packages/snakemake/workflow.py", line 1243, in execute
    success = self.scheduler.schedule()
              ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "<path to snakemake-tutorial-v8+ env>/snakemake-tutorial-v8+/lib/python3.12/site-packages/snakemake/scheduler.py", line 306, in schedule
    self.run(runjobs)

  File "<path to snakemake-tutorial-v8+ env>/snakemake-tutorial-v8+/lib/python3.12/site-packages/snakemake/scheduler.py", line 394, in run
    executor.run_jobs(jobs)

  File "<path to snakemake-tutorial-v8+ env>/snakemake-tutorial-v8+/lib/python3.12/site-packages/snakemake_interface_executor_plugins/executors/base.py", line 72, in run_jobs
    self.run_job(job)

  File "<path to snakemake-tutorial-v8+ env>/snakemake-tutorial-v8+/lib/python3.12/site-packages/snakemake_executor_plugin_drmaa/__init__.py", line 118, in run_job
    jt.outputPath = ":" + self.drmaa_log_dir
                    ~~~~^~~~~~~~~~~~~~~~~~~~

TypeError: can only concatenate str (not "PosixPath") to str

Proposed solution
I would suggest to replace the lines 108 and 109 of init.py with the following code:

108          if self.drmaa_log_dir:
108-1            self.drmaa_log_dir = str(self.drmaa_log_dir)
108-2            if not os.path.isabs(self.drmaa_log_dir):
108-3                self.drmaa_log_dir = os.path.abspath(os.path.expanduser(self.drmaa_log_dir))
109              os.makedirs(self.drmaa_log_dir, exist_ok=True)

because even if the PosixPath self.drmaa_log_dir is cast to a string, one still would need to provide an absolute paths to --drmaa_logs although usually a convenient location for the drmaa_logs directory is inside the working directory and a relative path is preferable. This was already part of the implementation for SnakeMake 7.x.x. It is not clear why this was dropped.

Best regards,
Sven

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant