Snakemake fails at rule circularize_final

[brymerr921]

When providing Illumina and Nanopore reads, the pipeline fails at rule circularize_final because there is no directory called {sample}/3.circularization/3.circular_sequences/sh, which should have several files inside. Therefore, it fails with this error message:

519 of 524 steps (99%) done

[Thu Jun 27 23:23:58 2019]
localrule circularize_final:
    input: Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa, Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai
    output: Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta
    jobid: 528
    wildcards: sample=Prevotella-stercorea-2632


                cat Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/* | bash
                ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | cut -f1 -d '_' > circs.tmp || true
                (cat Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | grep -vf circs.tmp |
                cut -f1 | xargs samtools faidx Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa; ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | xargs cat) |
                sed 's/\([ACTG]\)\n//g' | fold -w 120 | cut -f1 -d ':' > Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta || true
                #rm circs.tmp

Activating singularity image /mnt/.snakemake/singularity/f075a3ce7bc1893be5302161e5e7807a.simg
Singularity: action-suid (U=0,P=30414)> Not mounting requested bind point (already mounted in container): /mnt/

WARNING: Not mounting requested bind point (already mounted in container): /mnt/
Singularity: action-suid (U=0,P=30414)> Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt

WARNING: Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt
Singularity: action-suid (U=0,P=30414)> Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt

WARNING: Trying to create directory snakemake outside of container in /var/lib/singularity/mnt/final/mnt
Singularity: action-suid (U=0,P=30414)> Skipping user bind, could not create bind point /mnt/snakemake: No such file or directory

WARNING: Skipping user bind, could not create bind point /mnt/snakemake: No such file or directory
Singularity: action-suid (U=0,P=30414)> USER=root, IMAGE='f075a3ce7bc1893be5302161e5e7807a.simg', COMMAND='exec'

cat: 'Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/*': No such file or directory
[Thu Jun 27 23:23:58 2019]
Error in rule circularize_final:
    jobid: 528
    output: Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta
    shell:

                cat Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/* | bash
                ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | cut -f1 -d '_' > circs.tmp || true
                (cat Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | grep -vf circs.tmp |
                cut -f1 | xargs samtools faidx Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa; ls Prevotella-stercorea-2632/3.circularization/3.circular_sequences/ | grep .fa$ | xargs cat) |
                sed 's/\([ACTG]\)\n//g' | fold -w 120 | cut -f1 -d ':' > Prevotella-stercorea-2632/3.circularization/4.Prevotella-stercorea-2632_circularized.fasta || true
                #rm circs.tmp

        (exited with non-zero exit code)

Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /mnt/.snakemake/log/2019-06-27T231855.973555.snakemake.log

This is the current directory structure:

Prevotella-stercorea-2632/
├── 0.basecall
│   ├── Prevotella-stercorea-2632.fq
│   ├── data_links
│   │   └── dummy
│   └── raw_calls
│       └── dummy
├── 1.assemble
│   ├── Prevotella-stercorea-2632_nomerge.fasta -> /mnt/Prevotella-stercorea-2632/1.assemble/assemble_4m/Prevotella-stercorea-2632_4m.contigs.corrected.fasta
│   ├── Prevotella-stercorea-2632_raw_assembly.fa
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.amb
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.ann
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.bwt
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.fai
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.pac
│   ├── Prevotella-stercorea-2632_raw_assembly.fa.sa
│   └── assemble_4m
│       ├── Prevotella-stercorea-2632_4m.contigs.corrected.fasta
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta.bam
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta.bam.bai
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta.fai
│       ├── Prevotella-stercorea-2632_4m.contigs.fasta.misassemblies.tsv
│       ├── Prevotella-stercorea-2632_4m.contigs.layout
│       ├── Prevotella-stercorea-2632_4m.contigs.layout.readToTig
│       ├── Prevotella-stercorea-2632_4m.contigs.layout.tigInfo
│       ├── Prevotella-stercorea-2632_4m.correctedReads.fasta.gz
│       ├── Prevotella-stercorea-2632_4m.report
│       ├── Prevotella-stercorea-2632_4m.seqStore
│       ├── Prevotella-stercorea-2632_4m.seqStore.err
│       ├── Prevotella-stercorea-2632_4m.seqStore.ssi
│       ├── Prevotella-stercorea-2632_4m.trimmedReads.fasta.gz
│       ├── Prevotella-stercorea-2632_4m.unassembled.fasta
│       ├── Prevotella-stercorea-2632_4m.unitigs.bed
│       ├── Prevotella-stercorea-2632_4m.unitigs.fasta
│       ├── Prevotella-stercorea-2632_4m.unitigs.gfa
│       ├── Prevotella-stercorea-2632_4m.unitigs.layout
│       ├── Prevotella-stercorea-2632_4m.unitigs.layout.readToTig
│       ├── Prevotella-stercorea-2632_4m.unitigs.layout.tigInfo
│       ├── canu-logs
│       ├── canu-scripts
│       ├── correction
│       ├── trimming
│       └── unitigging
├── 2.polish
│   ├── Prevotella-stercorea-2632_polished.fa
│   ├── Prevotella-stercorea-2632_polished.fa.fai
│   └── pilon
│       ├── Prevotella-stercorea-2632_pilon.fa
│       ├── corrections.vcf.gz
│       ├── corrections.vcf.gz.tbi
│       ├── ranges
│       ├── short_reads.bam
│       ├── short_reads.bam.bai
│       └── sub_runs
└── 3.circularization
    └── 1.candidate_genomes

Full log file attached:
190627__circularize_final__error.log

[elimoss]

I think I see what's happening. Would you mind showing me the result of sort -k2,2gr Prevotella-stercorea-2632/2.polish/Prevotella-stercorea-2632_polished.fa.fai | head please? I think you've found an edge case that I have failed to account for.

[brymerr921]

This is the output:

tig00000012	85799	743756	60	61
tig00000007	65682	616905	60	61
tig00000001_5	62620	28879	60	61
tig00000010	59063	683695	60	61
tig00000047	50189	1228806	60	61
tig00000023	50158	908211	60	61
tig00000413	46361	2760317	60	61
tig00000031	45324	1069679	60	61
tig00000003_5	45045	192121	60	61
tig00000029	43222	1025723	60	61

[brymerr921]

Running the following, and then restarting the snakemake pipeline allowed the pipeline to complete.

mkdir -p Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/
touch Prevotella-stercorea-2632/3.circularization/3.circular_sequences/sh/dummy

[saras224]

Hi @brymerr921
Did you get the solution to your problem? Cuz I am facing a similar issue where it says :
Activating singularity image /lustre/rsharma/.snakemake/singularity/f075a3ce7bc1893be5302161e5e7807a.simg
Waiting at most 30 seconds for missing files.
MissingOutputException in line 574 of /home/rsharma/lathe/Snakefile:
Job Missing files after 30 seconds:
W1/3.circularization/1.candidate_genomes
This might be due to filesystem latency. If that is the case, consider increasing the wait time with --latency-wait.
Job id: 25 completed successfully, but some output files are missing. 25
Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /lustre/rsharma/.snakemake/log/2021-08-12T150046.064606.snakemake.log
{'W1': ['/home/rsharma/WWK_forward_paired.fastq', '/home/rsharma/WWK_reverse_paired.fastq']}
Checking for updates or modifications to workflow
WARNING: Differences to latest version detected. Please reset changes and/or pull repo.

Saraswati

[saras224]

@brymerr921
Do I have to run the whole pipeline again or I can start from circularization?