Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Errors with spades #233

Open
drpturner opened this issue Sep 16, 2020 · 18 comments
Open

Errors with spades #233

drpturner opened this issue Sep 16, 2020 · 18 comments
Assignees
@cimendes
Labels
bug Something isn't working

Comments

@drpturner
Copy link

Apologies in advance if this is obvious..I am new!

I am using flowcraft via WSL2 (Ubuntu 20.04; singularity 2.5.2; i7 10th gen laptop with 16Gb RAM). Following the issues already noted, I downgraded nextflow to 18.10.1 and my pipeline comprising trimmomatic, fastqc, and skesa works fine. However, whatever I do, I cannot get spades to work (despite same version working fine outside of flowcraft): I get the following fail code every time.

Same issue when using a Windows workstation (same set-up, i7 7th gen with 64Gb RAM). Any suggestions on how to proceed...

(flowcraft) pault@POS-COMRU-NB04:/mnt/d/wgs/test2$ nextflow run pt2.nf --fastq "/mnt/d/wgs/fastq/_{1,2}."
N E X T F L O W ~ version 18.10.1
Launching pt2.nf [loving_woese] - revision: 44064f23ef

============================================================
F L O W C R A F T

Built using flowcraft v1.4.1

Input FastQ : 2
Input samples : 1
Reports are found in : ./reports
Results are found in : ./results
Profile : standard

Starting pipeline at Wed Sep 16 10:31:09 ICT 2020

[warm up] executor > local
WARN: Process configuration syntax $processName has been deprecated -- Replace process.$spades_1_2 = <value> with a process selector
[e6/ea457a] Submitted process > integrity_coverage_1_1 (29396_1#116)
[9f/746d95] Submitted process > status (29396_1#116)
[b5/3231e8] Submitted process > report_coverage_1_1
[97/6a649e] Submitted process > report (29396_1#116)
[a9/8acfda] Submitted process > spades_1_2 (29396_1#116)
[a9/8acfda] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (1)
[ba/e39263] Re-submitted process > spades_1_2 (29396_1#116)
[ba/e39263] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (2)
[77/68e9fe] Re-submitted process > spades_1_2 (29396_1#116)
[77/68e9fe] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (3)
[b7/5b334a] Re-submitted process > spades_1_2 (29396_1#116)
[b7/5b334a] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (4)
[6c/80b84f] Re-submitted process > spades_1_2 (29396_1#116)
[6c/80b84f] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (5)
[9b/e859a3] Re-submitted process > spades_1_2 (29396_1#116)
[9b/e859a3] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (6)
[6d/00578c] Re-submitted process > spades_1_2 (29396_1#116)
[6d/00578c] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Execution is retried (7)
[e3/029556] Re-submitted process > spades_1_2 (29396_1#116)
[e3/029556] NOTE: Process spades_1_2 (29396_1#116) terminated with an error exit status (1) -- Error is ignored
[64/f0eabb] Submitted process > compile_status_buffer (1)
[7a/21accc] Submitted process > compile_reports (1)
[b2/1f8c8f] Submitted process > compile_status
Completed at: Wed Sep 16 10:33:04 ICT 2020
Duration : 1m 55s
Success : true
Exit status : 0

.nextflow.log
.command.log
@cimendes cimendes self-assigned this Sep 18, 2020
@cimendes cimendes added the bug Something isn't working label Sep 18, 2020
@CreatorOfMoon
Copy link

I think i figured out why it does not work. at line 20 of your command log, when the file is running "spades.py" with the spades option, it does not run the actual Spades software, but it is running the Spades.py component of flowcraft again, with argument needed for spades software.

That's why Error log say there is not "SAMPLE_ID", of course cause the first argument is --careful and not sample id ^^'

The solution would be either changing the name of spades.py component of flowcraft for something else ( like fc_spades.py) and modify the pipeline accordingly,
Or changing this :
cli = [ "spades.py", "--careful", "--only-assembler", "--threads", "$task.cpus", "--cov-cutoff", min_coverage, "-o", "."

to this
cli = [ "$(which "spades.py") ", "--careful", "--only-assembler", "--threads", "$task.cpus", "--cov-cutoff", min_coverage, "-o", "."

in the spades.py part of flowcraft. Not sure about the second solution, but i can't figure out of an other way :P

@cimendes
Copy link
Member

Hi! Sorry for not getting back to you on this issue! Unfortunately I haven't found the time yet to look deeply into your issue, but you seem to be on the right track with your solution! :) I invite you to do a PR if you want with your fix. If not, i'll try to correct it sometime this week.

Best,
Inês

@drpturner
Copy link
Author

Many thanks for these replies...unfortunately no joy yet.

For the first option (changed template filename to fc_spades.py and updated the .nf file accordingly), spades still fails with error exist status (1). However the warnings / errors in the command.log are different:
== Warning == output dir is not empty! Please, clean output directory before run.
== Warning == No assembly mode was sepcified! If you intend to assemble high-coverage multi-cell/isolate data, use '--isolate' option.
== Error == system call for: "['/NGStools/SPAdes-3.14.0-Linux/bin/spades-core', '/tmp/nxf.Bx7cKswjiV/K21/configs/config.info', '/tmp/nxf.Bx7cKswjiV/K21/configs/careful_mode.info']" finished abnormally, OS return value: -11

I changed the fc_spades.py options to --isolate, but same result

If I try option (2), I get this error:
1 error
-- check template file: /mnt/d/wgs/p_spades/templates/fc_spades.py -- Execution is retried (1)
Unable to re-submit task spades_1_2 (29396_1#116)

[98/b63422] Submitted process > report (29396_1#116)
Workflow execution stopped with the following message:
Failed to parse template script (your template may contain an error or be trying to use expressions not currently supported): org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/groovy/script/Script556B61ECD10F86B7E17B67558EBD5F7C: 201: illegal string body character after dollar sign;
solution: either escape a literal dollar sign "$5" or bracket the value expression "${5}" @ line 201, column 12.
"$(which "spades.py") ",
^
I made the change to "$(which "spades.py") " , but get the same message (with added ).

Thanks for any further thoughts.
Paul

Logs from option (1) test here:
.command.log
.nextflow.log

@CreatorOfMoon
Copy link

CreatorOfMoon commented Sep 22, 2020 •
edited

Hi drpturner, sorry to hear that ^^

I wasn't sure about the second option, sadly i don't think we can fix this problem by changing this line then.

For the first option, the problem seems to be Spades Software since this time it launched. Can you provide the log from Spades please ?
( params.txt and spades.log files from the output directory.

SPAdes log can be found here: /tmp/nxf.Bx7cKswjiV/spades.log )

@drpturner
Copy link
Author

Thanks CreatorOfMoon - as far as I can tell that folder and log is not present in the tmp folder. Is it possible that this is part of the problem?

image

@cimendes
Copy link
Member

Sorry, I haven't been hable to reproduced your error yet. I'm running the master version of flowcraft with nextflow (v 20.01.0.5264) and docker (v 19.03.12-ce). The spades.pt template is calling the spades.py executable in the container correctly, and I'm able to obtain an assembly. Could you please tell me what container software are you using?

@drpturner
Copy link
Author

drpturner commented Sep 23, 2020 via email

@ramirma
Copy link

ramirma commented Sep 23, 2020

Could it be that you do not have write permissions to /temp with singularity?

@drpturner
Copy link
Author

Thanks for this - I removed singularity and reinstalled (carefully) to version 3.6.3 following the instructions here: https://sylabs.io/guides/3.6/admin-guide/installation.html#installation-on-linux. There is a comment "By default Singularity will be installed in the /usr/local directory hierarchy."...could this be an issue? From the text that followed it seemed to imply that this would be ok re permissions, but not totally sure.

Is it likely that this would be isolated to spades...everything else I am using with flowcraft works fine - mostly a pipeline "trimmomatic fastqc skesa assembly_mapping pilon (prokka | mlst | abricate)". Runs on both machines with no errors.

Further tests showed this singularity update didn't change the outcome. I'm not sure what else to try - I might have to be happy with skesa and use spades outside of flowcraft (which works fine).

@drpturner
Copy link
Author

An update - have just located the singularity.conf file and (I think) confirmed that /tmp is accessible:

MOUNT TMP: [BOOL]

DEFAULT: yes

Should we automatically bind mount /tmp and /var/tmp into the container? If

the --contain option is used, both tmp locations will be created in the

session directory or can be specified via the SINGULARITY_WORKDIR

environment variable (or the --workingdir command line option).

mount tmp = yes

@drpturner
Copy link
Author

OK - progress. I changed the pipeline params.config so that the scratch option was false (disableRR_1_2 = false)...this meant the spades files all ended up in the pipeline /work folder. Spades still fails - log file attached if anyone has any remaining energy to take a look!

spades.log

@cimendes
Copy link
Member

That is not a very informative error message :/
The spades executable is being called correctly in the container so we can exclude that. IDK if that error is related with memory. There's many people complaining about the say execution error code same error but from what I saw, no one explaining why it is happening.

@ramirma
Copy link

ramirma commented Sep 28, 2020

drpturner what is the OS you are running? We seem to have had problems in the past for reasons that remained unclarified while running some images built on different OSs than the one the user is running. This problem was restricted to singularity.

@drpturner
Copy link
Author

drpturner commented Sep 28, 2020 via email •
edited

@drpturner
Copy link
Author

drpturner commented Sep 28, 2020 via email •
edited

@ramirma
Copy link

ramirma commented Sep 28, 2020

drpturner I am not familiar with WSL2, but it was tricky to configure Docker with WSL1 (although in the end it actually worked pretty well). You may also consider trying Ubuntu 18, I do not have a lot of experience with the 20 release. Hope it works.

@drpturner
Copy link
Author

drpturner commented Sep 30, 2020 •
edited

Resolution :-) ...thanks for all of your help with this

I set up Ubuntu 18.04 using VirtualBox on my Windows machine, installed flowcraft using conda, and my spades pipeline worked as expected, using docker. I still had to downgrade nextflow to 18.10 to avoid this issue#217, but otherwise totally fine with no adjustments to templates etc.

If I have strength I will see if I can replicate this using WSL2 (but it didn't work with Ubuntu 20.04 and docker).

@ramirma
Copy link

ramirma commented Sep 30, 2020

Great! I suspect this may be a combination of issues (I would think Ubuntu 20.04 may be playing a role). The most important thing is that it is working for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cimendes cimendes

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ramirma @cimendes @CreatorOfMoon @drpturner