Mashtree species download #173

kusandeep · 2020-12-17T21:53:24Z

kusandeep
Dec 17, 2020

I am trying to do a mashtree using some of my Bacillus pumilus genomes and compere them with completed Bacillus pumilus assemblies from NCBI, but it seems like the assembly downloads stops after 50% completion. when I try --accessions option then it work but I have the limitation to use just one ref accession(or can I use multiple separated by ","). Any idea what is happing here? is this something to do with some funny accessions ID from NCBI?

bactopia tools mashtree --bactopia /bifo/itmp/SK_PMI/bactopia_out/maze_runner/run1/ --species Bacillus pumilus --outdir /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/ --include /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/bacillus_pumilus.txt --cpus 16 --force -resume
2020-12-18 10:03:34:root:INFO - Existing env (/dataset/SK_PMI/itmp/conda-envs/bactopia//share/bactopia-1.5.x/conda/envs/tools-mashtree-1.5.x) found, skipping unless --force_rebuild is used
N E X T F L O W ~ version 20.10.0
Launching /dataset/SK_PMI/itmp/conda-envs/bactopia/share/bactopia-1.5.x/tools/mashtree/main.nf [irreverent_kowalevski] - revision: 9e5859fa55
bactopia tools mashtree - 1.5.3
Including 15 samples for analysis
Found 15 samples to process
If this looks wrong, now's your chance to back out (CTRL+C 3 times).
Sleeping for 5 seconds...
executor > local (2)
[ea/18594c] process > collect_assemblies (1) [ 50%] 1 of 2, failed: 1, retries: 1
[- ] process > build_tree -
[1c/c7916b] NOTE: Process collect_assemblies (1) failed -- Execution is retried (1)

rpetit3 · 2020-12-17T22:06:39Z

rpetit3
Dec 17, 2020
Maintainer

Hi @kusandeep

With --accessions you can provide a file of accessions to be downloaded.

  --accessions STR        A file with Assembly accessions (e.g. GCF*.*) to download from RefSeq.

Either way, I think its weird that giving species does not work correctly, so I'm going to look into that.

Would you mind trying to add a limit to the number of genomes to download? It can be arbitrary: --limit 10000

I'm asking this because there's a difference in how genomes are downloaded when a limit is given:

if [ "!{params.limit}" != "null" ]; then
    ncbi-genome-download bacteria -l complete -o ./ -F fasta -p !{task.cpus} \
                                  -g "!{params.species}" -r 50 --dry-run > accession-list.txt
    shuf accession-list.txt | head -n !{params.limit} | cut -f 1,1 > accession-subset.txt
    ncbi-genome-download bacteria -l complete -o ./ -F fasta -p !{task.cpus} \
                                  -A accession-subset.txt -r 50
else
    ncbi-genome-download bacteria -l complete -o ./ -F fasta -p !{task.cpus} \
                                  -g "!{params.species}" -r 50
fi

I'm wondering if that's what is causing the problem. I've had issues downloading genomes in the past using the species name.

I'll dig it up where I had this issue.

EDIT: It was for bactopia datasets I had to change how genomes were downloaded.

It was download assemblies associated with a genera. (1 step)

Now it gets a list of accessions associated with a genera, then downloads assemblies from the accessions. (2 step)

1 reply

kusandeep Dec 17, 2020
Author

Yes, it s what happing that it downloading random species from Bacillus genus

kusandeep · 2020-12-17T22:20:59Z

kusandeep
Dec 17, 2020
Author

It worked with --limit 20 option.

I just tried with --limit 20 since Bacillus pumilus has only 13 completed genomes. however new issue is that it is downloading random species from Bacillus genus..! see the accession below, also the link.
https://www.ncbi.nlm.nih.gov/assembly?cmd=HistorySearch&QueryKey=7

GCF_000009825
GCF_000227485
GCF_000242855
GCF_000299555
GCF_000699465
GCF_000742855
GCF_000800825
GCF_001752685
GCF_001896025
GCF_002238395
GCF_002764075
GCF_003047025
GCF_003721475
GCF_004119655
GCF_005160425
GCF_006704205
GCF_006974185
GCF_009497655
GCF_009913555
GCF_013340745

bactopia tools mashtree --bactopia /bifo/itmp/SK_PMI/bactopia_out/maze_runner/run1/ --species Bacillus pumilus --limit 20 --outdir /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/ --include /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/bacillus_pumilus.txt --cpus 20 --force
2020-12-18 11:09:45:root:INFO - Existing env (/dataset/SK_PMI/itmp/conda-envs/bactopia//share/bactopia-1.5.x/conda/envs/tools-mashtree-1.5.x) found, skipping unless --force_rebuild is used
N E X T F L O W ~ version 20.10.0
Launching /dataset/SK_PMI/itmp/conda-envs/bactopia/share/bactopia-1.5.x/tools/mashtree/main.nf [golden_euclid] - revision: 9e5859fa55
bactopia tools mashtree - 1.5.3
Including 15 samples for analysis
Found 15 samples to process
If this looks wrong, now's your chance to back out (CTRL+C 3 times).
Sleeping for 5 seconds...
executor > local (2)
[ca/dbe8b7] process > collect_assemblies (1) [100%] 1 of 1 ✔
[f7/dc2b8f] process > build_tree [100%] 1 of 1 ✔

Bactopia Tool 'mashtree' - Execution Summary
---------------------------
Command Line    : nextflow run /dataset/SK_PMI/itmp/conda-envs/bactopia//share/bactopia-1.5.x/tools/mashtree/main.nf --condadir /dataset/SK_PMI/itmp/conda-envs/bactopia//share/bactopia-1.5.x/conda/envs/tools-mashtree-1.5.x -w /dataset/SK_PMI/itmp/bactopia_out/run6/work/mashtree --bactopia /bifo/itmp/SK_PMI/bactopia_out/maze_runner/run1/ --species Bacillus pumilus --limit 20 --outdir /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/ --include /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/bacillus_pumilus.txt --cpus 20 --force
Resumed         : false
Completed At    : 2020-12-18T11:11:50.318930+13:00
Duration        : 1m 57s
Success         : true
Exit Code       : 0
Error Report    : -
Launch Dir      : /bifo/itmp/SK_PMI/bactopia_out/run6

executor > local (2)
[ca/dbe8b7] process > collect_assemblies (1) [100%] 1 of 1 ✔
[f7/dc2b8f] process > build_tree [100%] 1 of 1 ✔
Completed at: 18-Dec-2020 11:11:52
Duration : 1m 59s
CPU hours : 0.1
Succeeded : 2

0 replies

rpetit3 · 2020-12-17T22:23:44Z

rpetit3
Dec 17, 2020
Maintainer

Oh! That's not good!

On the command line is Bacillus pumilusin quotes? --species "Bacillus pumilus"

2 replies

kusandeep Dec 17, 2020
Author

On the command line is Bacillus pumilusin quotes? --species "Bacillus pumilus"
not in the the above run but now I re-ran it and and the same issues, this time below are some random Bacillus assemblies!!
https://www.ncbi.nlm.nih.gov/assembly?cmd=HistorySearch&QueryKey=9
GCF_000494835
GCF_000512775
GCF_000978375
GCF_002072735
GCF_002173675
GCF_002214765
GCF_002250925
GCF_002251045
GCF_003096215
GCF_003571425
GCF_003665335
GCF_004119535
GCF_004119555
GCF_004119575
GCF_004119775
GCF_008274905
GCF_009759805
GCF_009789615
GCF_015602625
GCF_015643545

kusandeep Dec 17, 2020
Author

Full script
bactopia tools mashtree --bactopia /bifo/itmp/SK_PMI/bactopia_out/maze_runner/run1/ --species "Bacillus pumilus" --limit 20 --outdir /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/ --include /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/bacillus_pumilus.txt --cpus 20 --force

rpetit3 · 2020-12-17T22:41:06Z

rpetit3
Dec 17, 2020
Maintainer

You found a bug, looking into the .command.sh in wht work directory for some reason Bacillus pumilus is becoming Bacillus

ncbi-genome-download bacteria -l complete -o ./ -F fasta -p 1 -g "Bacillus" -r 50 --dry-run > accession-list.txt

I'm looking into this, and am going to elevate this to an Issue.

0 replies

rpetit3 · 2020-12-17T23:32:34Z

rpetit3
Dec 17, 2020
Maintainer

If you would like to keep going you can do the following:

ncbi-genome-download bacteria -l complete -o ./ -F fasta -p 1 -g "Bacillus pumilus" -r 50 --dry-run | cut -f 1  > bpumilus-accessions.txt
bactopia tools mashtree --bactopia /bifo/itmp/SK_PMI/bactopia_out/maze_runner/run1/ --outdir /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/ --include /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/bacillus_pumilus.txt --cpus 20 --force --accessions bpumilus-accessions.txt

I've opened an issue: #174 and am working on a fix

1 reply

kusandeep Dec 18, 2020
Author

Thanks I was trying something similar using --accessions option. I had a txt file with all the completed bpumilus accessions (just 13 of them exist atm) but had the similar problem as the one in the start of this thread(process > collect_assemblies (1) [ 50%] 1 of 2, failed: 1, retries: 1) .

But any way I ran the command that u gave me and again the error is same

(/dataset/SK_PMI/itmp/conda-envs/bactopia) kumars@invbfopp10:/dataset/SK_PMI/itmp/bactopia_out/run6$ ncbi-genome-download bacteria -l complete -o ./ -F fasta -p 1 -g "Bacillus pumilus" -r 50 --dry-run | cut -f 1 > bpumilus-accessions.txt
(/dataset/SK_PMI/itmp/conda-envs/bactopia) kumars@invbfopp10:/dataset/SK_PMI/itmp/bactopia_out/run6$ bactopia tools mashtree --bactopia /bifo/itmp/SK_PMI/bactopia_out/maze_runner/run1/ --outdir /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/ --include /bifo/itmp/SK_PMI//bactopia_out/maze_runner/run1/bacillus_pumilus.txt --cpus 20 --force --accessions bpumilus-accessions.txt
2020-12-18 13:11:25:root:INFO - Existing env (/dataset/SK_PMI/itmp/conda-envs/bactopia//share/bactopia-1.5.x/conda/envs/tools-mashtree-1.5.x) found, skipping unless --force_rebuild is used
N E X T F L O W ~ version 20.10.0
Launching /dataset/SK_PMI/itmp/conda-envs/bactopia/share/bactopia-1.5.x/tools/mashtree/main.nf [irreverent_heisenberg] - revision: 9e5859fa55
bactopia tools mashtree - 1.5.3
Including 15 samples for analysis
Found 15 samples to process
If this looks wrong, now's your chance to back out (CTRL+C 3 times).
Sleeping for 5 seconds...
executor > local (2)
[7b/2a288a] process > collect_assemblies (1) [ 50%] 1 of 2, failed: 1, retries: 1
[- ] process > build_tree -
[a9/0e2268] NOTE: Process collect_assemblies (1) terminated with an error exit status (1) -- Execution is retried (1)

rpetit3 · 2020-12-18T01:34:27Z

rpetit3
Dec 18, 2020
Maintainer

Hmmm, weird! It worked on my end. Is there anything in work/mashtree/7b/2a288a*???*/.command.log that might be helpful?

N E X T F L O W  ~  version 20.10.0
Launching `/local/home/rpetit/miniconda3/envs/bactopia/share/bactopia-1.5.x/tools/mashtree/main.nf` [tender_minsky] - revision: 9e5859fa55
bactopia tools mashtree - 1.5.3
Found 20 samples to process
If this looks wrong, now's your chance to back out (CTRL+C 3 times).
Sleeping for 5 seconds...
executor >  local (2)
[25/942c18] process > collect_assemblies (1) [100%] 1 of 1 ✔
[29/b0af06] process > build_tree             [100%] 1 of 1 ✔

    Bactopia Tool 'mashtree' - Execution Summary
    ---------------------------
    Command Line    : nextflow run /local/home/rpetit/miniconda3/envs/bactopia//share/bactopia-1.5.x/tools/mashtree/main.nf --condadir /local/home/rpetit/miniconda3/envs/bactopia//share/bactopia-1.5.x/conda/envs/tools-mashtree-1.5.x -w /local/home/rpetit/bactopia-dev/test-mashtree/work/mashtree --bactopia ../test-20-samples/ --cpus 10 --species 'Bacillus pumilus' --force
    Resumed         : false
    Completed At    : 2020-12-17T19:03:08.622042-05:00
    Duration        : 20.2s
    Success         : true
    Exit Code       : 0
    Error Report    : -
    Launch Dir      : /local/home/rpetit/bactopia-dev/test-mashtree

13 replies

rpetit3 Dec 18, 2020
Maintainer

It's totally copy and paste error. Apologies!

I'm going to get a version update submitted that fixes both of these bugs.

kusandeep Dec 18, 2020
Author

That's fine.
Sorry for coming up with all these issues.
You are doing a great job helping someone like me who is taking baby steps in bacterial genome world.

rpetit3 Dec 18, 2020
Maintainer

I honestly appreciate the issues, really helps out a lot!

On a side-note, how do you feel about "GitHub Discussions"? So far I'm thinking I like everything in the Issues section.

kusandeep Dec 18, 2020
Author

Indeed discussion section is interesting.
I personally think discussion would be more useful latter on when there will be more people sharing their experience and opinions on things. Also I think you might find it useful in filtering out what is a real queries and what can go in issues etc..?

rpetit3 Dec 18, 2020
Maintainer

I agree.

At the moment only Issues can be moved to Discussions. I'm going and move back to Issues, then do like you suggest filter to Discussions as necessary.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mashtree species download #173

{{title}}

Replies: 6 comments 17 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Mashtree species download #173

kusandeep Dec 17, 2020

Replies: 6 comments · 17 replies

rpetit3 Dec 17, 2020 Maintainer

kusandeep Dec 17, 2020 Author

kusandeep Dec 17, 2020 Author

rpetit3 Dec 17, 2020 Maintainer

kusandeep Dec 17, 2020 Author

kusandeep Dec 17, 2020 Author

rpetit3 Dec 17, 2020 Maintainer

rpetit3 Dec 17, 2020 Maintainer

kusandeep Dec 18, 2020 Author

rpetit3 Dec 18, 2020 Maintainer

rpetit3 Dec 18, 2020 Maintainer

kusandeep Dec 18, 2020 Author

rpetit3 Dec 18, 2020 Maintainer

kusandeep Dec 18, 2020 Author

rpetit3 Dec 18, 2020 Maintainer

kusandeep
Dec 17, 2020

Replies: 6 comments 17 replies

rpetit3
Dec 17, 2020
Maintainer

kusandeep Dec 17, 2020
Author

kusandeep
Dec 17, 2020
Author

rpetit3
Dec 17, 2020
Maintainer

kusandeep Dec 17, 2020
Author

kusandeep Dec 17, 2020
Author

rpetit3
Dec 17, 2020
Maintainer

rpetit3
Dec 17, 2020
Maintainer

kusandeep Dec 18, 2020
Author

rpetit3
Dec 18, 2020
Maintainer

rpetit3 Dec 18, 2020
Maintainer

kusandeep Dec 18, 2020
Author

rpetit3 Dec 18, 2020
Maintainer

kusandeep Dec 18, 2020
Author

rpetit3 Dec 18, 2020
Maintainer