[bug]Process BACTOPIA:ASSEMBLER:ASSEMBLER_MODULE (reads) terminated with an error exit status (1)

[happymanmohit]

Description
Process BACTOPIA:ASSEMBLER:ASSEMBLER_MODULE (reads) terminated with an error exit status (1)

Steps to Reproduce
bactopia --sample reads --se fastqs/raw_reads.fastq.gz -profile docker

bactopia ...
bactopia tools ...
!!ERROR!!

Expected Behavior
A description of what is expected to happen.

Execution Environment

• Bactopia Version: [bactopia 3.0.0]
• MacOS
• Environment: [e.g. conda, cluster, container, etc...]

Additional Information
Any other information you think might be helpful in squashing this bug.

[rpetit3]

Hi @happymanmohit

Can you share any more information? There should be a file called .nextflow.log in the directory you executed the command from

[happymanmohit]

nextflow run /Users/Sauer/miniforge3/envs/bactopia//share/bactopia-3.0.0/main.nf -w /Users/Sauer/work/ --sample my-sample --SE/Users/Sauer/bactopia/Sauer_C8H_1/reads/raw_reads.fastq.gz                 

2024-03-22 22:18:47 - mad_joliot - 0cd9f79ba7 ec645a1e-939b-4156-9038-f8718995049e nextflow run /Users/Sauer/miniforge3/envs/bactopia//share/bactopia-3.0.0/main.nf -w /Users/Sauer/work/ --sample my-sample --SE/Users/Sauer/bactopia/Sauer_C8H_1/reads/raw_reads.fastq.gz -profile docker
2024-03-22 22:21:23 - stoic_lagrange - 0cd9f79ba7 da11b87a-740c-49ae-8682-fa709e91b81a nextflow run /Users/Sauer/miniforge3/envs/bactopia//share/bactopia-3.0.0/main.nf -w /Users/Sauer/work/ --sample reads --SE/Users/Sauer/bactopia/Sauer_C8H_1/reads/raw_reads.fastq.gz -profile docker
2024-03-22 22:30:45 - amazing_crick - 0cd9f79ba7 6cd0679c-ec0f-4689-b1d7-9572b2f73c22 nextflow run /Users/Sauer/miniforge3/envs/bactopia//share/bactopia-3.0.0/main.nf -w /Users/Sauer/work/ --sample reads --SE/Users/Sauer/bactopia/Sauer_C8H_1/reads/raw_reads.fastq.gz -profile docker
2024-03-22 22:50:32 9m 4s fabulous_golick ERR 0cd9f79ba7 1a798122-1cdc-45bc-b351-9160f22bea00 nextflow run /Users/Sauer/miniforge3/envs/bactopia//share/bactopia-3.0.0/main.nf -w /Users/Sauer/work/ --sample reads --se fastqs/raw_reads.fastq.gz -profile docker
2024-03-22 23:28:46 10m 10s furious_aryabhata ERR 0cd9f79ba7 cfa3f6b7-5b4c-4a01-8637-750f60aed0a6 nextflow run /Users/Sauer/miniforge3/envs/bactopia//share/bactopia-3.0.0/main.nf -w /Users/Sauer/work/ --sample reads --se fastqs/raw_reads.fastq.gz -profile docker

[rpetit3]

Hi @happymanmohit

Can you share the error messages (the red text at the end of the run) or the .nextflow.log file?

[happymanmohit]

Hi@rpetit3
I tried looking for the .nextflow.log file but I was not able to locate the file. But I am attaching the error message here
Command error:
R1 null
R2 null
SE reads.fastq.gz
[shovill-se] Hello stranger
[shovill-se] You ran: /usr/local/bin/shovill-se --SE reads.fastq.gz --gsize 0 --outdir results --assembler skesa --minlen 500 --mincov 2 --force --keepfiles --depth 0 --noreadcorr --namefmt reads_%05d --cpus 4 --ram 7
[shovill-se] This is shovill-se 1.1.0
[shovill-se] Written by Torsten Seemann
[shovill-se] Homepage is https://github.com/tseemann/shovill
[shovill-se] Operating system is linux
[shovill-se] Perl version is v5.32.1
[shovill-se] Machine has 4 CPU cores and 3.83 GB RAM
[shovill-se] Set --ram to 7 but machine only has 3.83 GB
grep: results/shovill.log: No such file or directory

[rpetit3]

How much memory do you have of your system?

[happymanmohit]

8 GB 1867 MHz DDR3

[rpetit3]

Do mind trying the test?

bactopia -profile test,docker

I want to verify it works here.

Also, you might consider giving docker more resources: https://docs.docker.com/desktop/settings/mac/#resources

[happymanmohit]

Hi @rpetit3
I tested bactopia -profile test,docker and it shows this error
R1 SRR2838702_R1.fastq.gz
R2 SRR2838702_R2.fastq.gz
SE null
[shovill] Hello stranger
[shovill] You ran: /usr/local/bin/shovill --R1 SRR2838702_R1.fastq.gz --R2 SRR2838702_R2.fastq.gz --gsize 358242 --outdir results --assembler skesa --minlen 500 --mincov 2 --force --keepfiles --depth 0 --noreadcorr --namefmt SRR2838702_%05d --cpus 2 --ram 5
[shovill] This is shovill 1.1.0
[shovill] Written by Torsten Seemann
[shovill] Homepage is https://github.com/tseemann/shovill
[shovill] Operating system is linux
[shovill] Perl version is v5.32.1
[shovill] Machine has 4 CPU cores and 3.83 GB RAM
[shovill] Set --ram to 5 but machine only has 3.83 GB
grep: results/shovill.log: No such file or directory

[rpetit3]

[shovill] Set --ram to 5 but machine only has 3.83 GB

This is making me wonder if its a "not enough memory" error you are getting. Did you try giving Docker more memory? right now it looks like 3.83GB is allocated to Docker

[happymanmohit]

Thanks @rpetit3
I will try again by allocating more memory to the docker and let you know if that solves this issue

[happymanmohit]

Hii @rpetit3
As suggested I increased the allocated memory for the docker to 7
Command -bactopia -profile test, docker ran sucessfully, however when I ran my samples it gave me error

executor > local (13)
[skipped ] process > BACTOPIA:DATASETS [100%] 1 of 1, stored: 1 ✔
[7b/e97896] process > BACTOPIA:GATHER:GATHER_MODULE (SRR2838702) [100%] 1 of 1 ✔
[9f/845056] process > BACTOPIA:GATHER:CSVTK_CONCAT (meta) [100%] 1 of 1 ✔
[67/0300e8] process > BACTOPIA:QC:QC_MODULE (SRR2838702) [100%] 1 of 1 ✔
[fe/b9300b] process > BACTOPIA:ASSEMBLER:ASSEMBLER_MODULE (SRR2838702) [100%] 1 of 1 ✔
[5f/c7a524] process > BACTOPIA:ASSEMBLER:CSVTK_CONCAT (assembly-scan) [100%] 1 of 1 ✔
[3a/6e8dc7] process > BACTOPIA:SKETCHER:SKETCHER_MODULE (SRR2838702) [100%] 1 of 1 ✔
[29/2048ab] process > BACTOPIA:ANNOTATOR:PROKKA_MODULE (SRR2838702) [100%] 1 of 1 ✔
[56/27ce09] process > BACTOPIA:AMRFINDERPLUS:AMRFINDERPLUS_RUN (SRR2838702) [100%] 1 of 1 ✔
[cd/46dec7] process > BACTOPIA:AMRFINDERPLUS:GENES_CONCAT (amrfinderplus-genes) [100%] 1 of 1 ✔
[40/05432c] process > BACTOPIA:AMRFINDERPLUS:PROTEINS_CONCAT (amrfinderplus-proteins) [100%] 1 of 1 ✔
[4d/fb7121] process > BACTOPIA:MLST:MLST_MODULE (SRR2838702) [100%] 1 of 1 ✔
[72/15a95d] process > BACTOPIA:MLST:CSVTK_CONCAT (mlst) [100%] 1 of 1 ✔
[f5/091c05] process > BACTOPIA:DUMPSOFTWAREVERSIONS (1) [100%] 1 of 1 ✔

Bactopia Execution Summary
---------------------------
Bactopia Version : 3.0.0
Nextflow Version : 23.10.1
Command Line     : nextflow run /Users/Sauer/miniforge3/envs/bactopia//share/bactopia-3.0.0/main.nf -w /Users/Sauer/work/ -profile test,docker
Resumed          : false
Completed At     : 2024-03-24T22:31:14.765195-04:00
Duration         : 10m 51s
Success          : true
Exit Code        : 0
Error Report     : -
Launch Dir       : /Users/Sauer

WARN: Graphviz is required to render the execution DAG in the given format -- See http://www.graphviz.org for more info.
Completed at: 24-Mar-2024 22:31:16
Duration : 10m 53s
CPU hours : 0.3
Succeeded : 13

Command:bactopia --sample reads --se fastqs/raw_reads.fastq.gz -profile docker
Command error:
[shovill-se] Running: ln -sf /Users/Sauer/work/75/a074a6912af379ede0a51774b3a662/results/reads.fastq.gz SE.fq.gz 2>&1 | sed 's/^/[ln] /' | tee -a shovill-se.log
[shovill-se] Average read length looks like 4407 bp
[shovill-se] Setting k-mer range to (31 .. 127)
[shovill-se] Estimated K-mers: 31 55 79 103 127 [kn=5, ks=24, kmin=31, kmax=127]
[shovill-se] Using kmers: 31,55,79,103,127
[shovill-se] Enabled --noreadcorr, so no read correction will be performed
[shovill-se] Single-end reads, so no read stitching will be performed
[shovill-se] Assembling reads with 'skesa'
[shovill-se] Running: skesa --gz --fastq SE.fq.gz --contigs_out skesa.fasta --min_contig 1 --memory 7 --cores 4 --vector_percent 1 2>&1 | sed 's/^/[skesa] /' | tee -a shovill-se.log
[skesa] skesa --gz --fastq SE.fq.gz --contigs_out skesa.fasta --min_contig 1 --memory 7 --cores 4 --vector_percent 1
[skesa]
[skesa] WARNING: option --gz is deprecated - gzipped files are now recognized automatically
[skesa] Total reads: 165154
[skesa] Reads acquired in 49.642239s wall, 46.020000s user + 3.360000s system = 49.380000s CPU (99.5%)
[skesa] Adapters clip is disabled
[skesa]
[skesa] Kmer len: 21
[skesa] Raw kmers: 724471800 Memory needed (GB): 13.9099 Memory available (GB): 4.71962 3 cycle(s) will be performed
[skesa] Distinct kmers: 13230215
[skesa] Kmer count in 178.373070s wall, 580.490000s user + 89.420000s system = 669.910000s CPU (375.6%)
[skesa] Uniq kmers merging in 3.935948s wall, 7.630000s user + 3.300000s system = 10.930000s CPU (277.7%)
[skesa] WARNING: --min_count changed from 2 to 3 because of high coverage for genome size 5499724
[skesa] WARNING: --max_kmer_count 10 to 13 because of high coverage for genome size 5499724
[skesa] Kmers branching in 8.545117s wall, 29.920000s user + 0.000000s system = 29.920000s CPU (350.1%)
[skesa]
[skesa] Average read length: 4407
[skesa] Genome size estimate: 5609186
[skesa]
[skesa] Kmer: 21 Graph size: 8559463 Contigs in: 0
[skesa] Valley: 23
[skesa]
[skesa] Mark used kmers in 0.000022s wall, 0.000000s user + 0.000000s system = 0.000000s CPU (n/a%)
[skesa] Kmers in multiple/single contigs: 0 0
[skesa] Fragments before: 13423 6339241
[skesa] Fragments after: 13390 6339241
[skesa] New seeds: 11510
[skesa] New seeds in 9.110988s wall, 32.990000s user + 0.000000s system = 32.990000s CPU (362.1%)
[skesa] Fragments before: 23020 483420
[skesa] Fragments after: 23020 483420
[skesa] Connectors: 0 Extenders: 23020
[skesa] Connections and extensions in 2.860484s wall, 10.450000s user + 0.000000s system = 10.450000s CPU (365.3%)
[skesa] Contigs out: 11510 Genome: 5970934 N50: 818 L50: 2178
[skesa] Assembled in 12.008250s wall, 43.480000s user + 0.000000s system = 43.480000s CPU (362.1%)
[skesa]
[skesa]
[skesa] Kmer len: 511
[skesa]
[skesa] Memory provided is insufficient to do runs in 10 cycles for the read coverage. We find that 16 Gb for 20x coverage of a 5 Mb genome is usually sufficient
[shovill-se] Assembly failed - skesa.fasta has zero contigs!
grep: results/shovill.log: No such file or directory

[rpetit3]

I was afraid this might happen. The test uses a super small bacterium (genome size 300,000bp)

Your sample is mush larger (~5,000,000 bp), and is asking for more memory than your system has.

Do you have access to another system? Or if its not too many genomes I can help with processing for you.

[happymanmohit]

Hii @rpetit3
Thankyou for your offer to help processing our samples, we just want to try one more time with our hub system. If it doesn't work I will let you know.

[rpetit3]

No problem, if there are any questions related to setup on your hub system, please feel free to reach out to me.

[happymanmohit]

Thanks @rpetit3

[happymanmohit]

hii@rpetit3
can you please help me set up Bactopia on the hub? I am struggling to do it on the hub. Do I need to use conda on the hub for using Bactopia

[rpetit3]

Hi @happymanmohit

Is there any documentation about the hub, or can you give details about it? (Is it a single server or a cluster? Does it have a job scheduler (SLURM, SGE, etc..)? Does it support Docker or Singularity?)

Cheers,
Robert

[happymanmohit]

It is a cluster; yes, it has a job scheduler(SLURM). I don't know about Docker or Singularity.

[happymanmohit]

It shows me that it supports docker/18.09.9

[rpetit3]

Check on Singularity support

Do you have an example script that you would use to submit a job?

[happymanmohit]

No, I don't have any scripts right now. I am checking the documentation of the hub if they have an example

[happymanmohit]

yes they also have singularity support singularity/3.4.1

[rpetit3]

great, let's see if you can dig up an example script, then I think we can get something put together

[happymanmohit]

can you please have a look at them I tried looking fir the example script but it is not clear to me https://spiediedocs.binghamton.edu/docs/spiedie_modules.html

[rpetit3]

is there a module for conda? (miniforge, conda, miniconda, etc...)

[happymanmohit]

I see only python 37 and python 39 but no conda, miniconda and miniforge

[happymanmohit]

[rpetit3]

I see miniconda in there.

Here's what I'm thinking

# Start a screen  (or tmux if you use it)
screen -S bactopia

# Start an interactive session to get on a compute node
# https://spiediedocs.binghamton.edu/docs/submitting_jobs.html
srun -n4 --partition="Standard" --mem=32G --time=24:00:00  --pty bash

# This will put you on a compute node
module load singularity/3.4.1
module load miniconda/miniconda

# Install bactopia (only have to do this once)
conda create -n bactopia -c conda-forge -c bioconda bactopia
conda activate bactopia

# Run bbactopia test
bactopia -profile test,slurm --slurm_queue Standard

If this works, share the .nextflow.log file so we can confirm slurm was used

[happymanmohit]

[rpetit3]

This unfortunately might be an issue you will need to ask about. I assume your username is ksauer and you should be able to write to your home directory

[happymanmohit]

yes user name is ksauer. Do I need to ask the spiedie management about the permission to write.

[rpetit3]

Yeah, I think the screen shot should be helpful enough for them to see what is happening

[happymanmohit]

I will talk to them about this and contact you again.