Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing results and FANNGO error #21

Open
huh688 opened this issue Mar 17, 2021 · 12 comments
Open

Missing results and FANNGO error #21

huh688 opened this issue Mar 17, 2021 · 12 comments
Assignees
@wkpalan
Labels
bug Something isn't working

Comments

@huh688
Copy link

huh688 commented Mar 17, 2021

Describe the bug
I reran Sorghum bicolor genome peptides (v3.1.1, downloaded from Phytozome) with GOMAP, and found only 32,159 genes were successfully annotated (not 34,129 genes in your Cyverse dataset, DOI:10.25739/4ty0-ye98).

Then I compared outputs from b.raw_gaf folder, found number of records from TAIR, Argot, PANNZER and UniProt were exactly same as your Cyverse dataset. The differences from FANN-GO (331,081 vs 564,161) and InterProScan (179,129 vs 186,551).

The output/log from step 2 domain search didn't show any error, but step 3 FANN-GO had a lots errors below

Input File
Sbicolor_454_v3.1.1.protein_primaryTranscriptOnly.fa, .

Sobic.010G276200.1.p
MGGGGGGGGNTRELDQTPTWAVASVCGVIVLISILLEKGLHHVGEFFSHRKKKAMVEALEKVKAELMVLGFISLLLVFGQ
NYIIKICISKNAASTMLPCKLEAVEAAGAGHGKEPAEGKAAGGKDGHGKEGAAVPGKKKAAAEHLGGVLDWPPPYYAHNA
RMLMEASSKSKCPEGKVSLISINALHQLHIFIFFLAVFHVSYSAITMALGRAKIRAWKEWEKEAAGQDYEFSNDPTRFRF
THETSFVRNHMNVLNKFPASFYISNFFRQFFRSVRQADYSALRHSFVNVHLAPGSKFDFQKYIKRSLEDDFKVIVGISPT
LWASALIFLFLNVNGWHTMLWISIMPVVIILSVGTKLQGIICRMAIDITERHAVVQGIPLVQVSDSYFWFSKPTFVLFLI
HFTLFQNGFQIIYFLWILYEYGMDSCFNDSKEFVFARLCLGVVVQVLCSYVTLPLYALVSQMGSTMKQSIFDEQTSKALK
NWRAGAKKKHPTSSKHGGGGSPTAGGSPTKADGDA
Sobic.010G080700.2.p
MAVLAKGDNEGGKVVDTMPLPTYHSLAAPVRKPVDKFALLPAFLKVRGLVKEHIDSFNYFIDKGIKKIVEANNRIEARND
PSIYLRYLDVRVGMPSIYENCNVESITPHLCRLTDRTYSAPIEVDIQYTVGKQHVVKRRNKFIIGYMPIMLRSSACILNE
MDEDELAKYGECPLDPGGYFVVKGTEKVILIQEQLSKNRIIVDTDSKGRVIASVTSSTHAIKSKTVVLMDKGKIYLQLNQ
FTKPIPIIVVMKAMGMESDQEAVQMVGRDPRYGDLLFPSIQECASERIYTRQQALQYMDEKVMYPGATNNKEGRSNSILH
DVFIAHVPVKDGNLRPKCIYTAVMLRRMLDAILNSDTFDDKDYVGNKRFELSGQLISLLFEDLFKTMNSQAVERMNKNSD
KTRSSPLDFSQLIMQDTITFGLERAISTGNWDIKRFRMHRKGVSQVLSRLSYMATLGYMTRITPQFEKTRKTSGPRALQP
SQWGMLCPCDTPEGEACGLTKNLALLSHITTDQEEGPLIDLCHSLGVEDLLFLSSEEIHAPGSFLVMLNGLILGKHKDPQ
RLVIHLRNLRRSNVIGEFVSIFINEKQRCVHIASDGGRVCRPLIIADMGISRIDETHMTKLRDGTYDFDDFLSGGLIEYL
DVNEENNALIALYEHMDQDDVERSSITHIEIEPLSILGVVSGLIPYPHHNQSPRNTYQCAMGKQAMGNIAYNQLFRADSL
LYLLVYAQRPLLTTKTIELVGYDKLGAGQNATVAVMGYSGYDIEDAIVMNKSSLDRGFGRCIALKKYTVLKDNYVDGVSD
RIVKPQRDKDGVLMKQNMRALDDDGIVAPGQIIRNHDIYVNKQTPMNTSKGIGRPLTDSCYKDSPAIYKGVDGETTVVDR
VMLCSDTNDKLSIKCIIRHTRRPEVGDKFSSRHGQKGVCGTIIQQEDFPFSERGICPDLIMNPHGFPSRMTIGKMIELLG
GKAGVSCGRFHYGSAFGEPSGNADNVADISSTLIKHGFSYNGKDFLYSGILGQPLEAYIFMGPIYYQKLKHMVLDKMHAR
ASGPRVLLTRQPTEGRSRDGGLRLGEMERDCLIAYGASMLIFERLMLSSDPYQVQVCRKCGLLGYYNHKLKTSYCSMCKN
GENMAKMRLPYACKLLFQELHAMNVVPRLKLTEV

GOMAP step that crashed (if applicable)
FANN-GO and InterProScan

Attach the output files
Errors from 3-fanngo

Warning: [blastp] Query is Empty!
error: writing file '/home/hhu/.octave_hist': No space left on device
error: ignoring const octave::execution_exception& while preparing to exit
Warning: [blastp] Query is Empty!

No errors in step 2 domain search

System Details

  • OS: RedHat
  • Version: 4.8.5-16
  • Memory Size: 96GB
@huh688 huh688 added the bug Something isn't working label Mar 17, 2021
@wkpalan
Copy link
Collaborator

wkpalan commented Mar 17, 2021

Hi @huh688,

I am checking this out now. Give me some time as I am experiencing some technical issues with the cluster I am using

Thanks

Best
Kokul

@huh688
Copy link
Author

huh688 commented May 10, 2021

Hi @wkpalan,

Do you have any update for this issue?

I also tried several other species (setaria, pearl millet, tef), and no one got 100% annotation coverage.

@wkpalan
Copy link
Collaborator

wkpalan commented May 13, 2021

Hi @huh688 ,

I have not had time to finalize a solution, but I am looking into it.

Regret the delay

Thanks
Kokul

@wkpalan wkpalan mentioned this issue May 15, 2021
Merged
@wkpalan
Copy link
Collaborator

wkpalan commented May 16, 2021

Hi @huh688,

Thank you for bringing this to my attention. This was a weird bug that was difficult to troubleshoot. The issue was with the
update of singularity version and how octave runs the FANN-GO code.

The shell script to run the container uses a -c argument to prevent any local folders being mounted that end up in some conflicts. This option has triggered an issue with FANN-GO that end up with some steps failing but the output is written with wrong scores. This results in missing annotations and lower number. This also was affecting InterProScan/domain results.

Steps to fix and reproduce the annotations

  • Checkout the latest version
    • git checkout v1.3.5
  • Delete any output in the following directories
    • Sbicolor.BTx623.v3.1.1/GOMAP-Sbicolor.v3.1.1/tmp/domain/
    • Sbicolor.BTx623.v3.1.1/GOMAP-Sbicolor.v3.1.1/tmp/mixed-meth/fanngo
  • Run the domain, fanngo and aggregate steps

This should result in a complete dataset, but might produce a few more annotations (+/- 10) than the original dataset.

Thank you for your patience and please let me know if this fixes the issues from your side.

@huh688
Copy link
Author

huh688 commented May 17, 2021

Hi @wkpalan,

I reran the FANNGO step, still got the same error:

Warning: [blastp] Query is Empty!
error: writing file '/home/hhu/.octave_hist': No space left on device
error: ignoring const octave::execution_exception& while preparing to exit

I redownloaded new 1.3.5, and git-checkout showed "HEAD is now at ad548de... updating the documentation".

But when I ran ./setup.sh initially, it showed "INFO: Downloading shub image", then the setup ended.

When ran it again, I got error messages in the middle: "WARNING: destination /home/hhu/gomap-test already in mount list: destination is already in the mount point list
ERROR: environment_properties::capture: missing environment file. should be at [/home/hhu/.irods/irods_environment.json]", which I didn't see in 1.3.4.

See below for the complete output:

[hhu@pete gomap-test]$ ./setup.sh
INFO: Downloading shub image
109.61 MiB / 109.61 MiB [======================================================================================] 100.00% 74.62 MiB/s 1s
[hhu@pete gomap-test]$ pwd
/home/hhu/gomap-test
[hhu@pete gomap-test]$ ./setup.sh
WARNING: destination /home/hhu/gomap-test already in mount list: destination is already in the mount point list
ERROR: environment_properties::capture: missing environment file. should be at [/home/hhu/.irods/irods_environment.json]
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
0/1 - 0.00% of files done 0.000/39254.793 MB - 0.00% of file sizes done
Processing GOMAP.sif - 39254.793 MB 2021-05-17.15:14:06
From server: NumThreads=16, addr:r03c06u29-ds18.cyverse.org, port:20167, cookie=1281065451
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
GOMAP.sif - 120.000/39254.793 MB - 0.31% done 2021-05-17.15:14:11

@wkpalan
Copy link
Collaborator

wkpalan commented May 17, 2021 via email

@wkpalan
Copy link
Collaborator

wkpalan commented May 20, 2021

Hey @huh688,

I forgot to mention that the existing containers need to be deleted. Can you please do the following and try again..

  1. Delete the icommands.sif and the GOMAP.sif files and then run setup.sh
  2. git fetch -p && git checkout v1.3.5
  3. re-run the above steps after the new image has been downloaded

Thanks

Best
Kokul

@wkpalan
Copy link
Collaborator

wkpalan commented Jun 17, 2021

@huh688 can this be closed?

@huh688
Copy link
Author

huh688 commented Jun 17, 2021

@wkpalan, sorry for the delayed reply. Our HPC is experiencing technical issues, the scheduler is heavily backlogged, and I'm still waiting to start FANNGO and aggregating steps. I'll let you know asap when I have results.

@huh688
Copy link
Author

huh688 commented Jun 22, 2021

Hi @wkpalan,

I just got the results by rerunning the pipeline with Sorghum bicolor genome, but I got a pretty similar results as before: 177862 records from InterProScan, and 331079 from FANNGO.

The step 3 FANNGO didn't show any error (an empty .err output, no error in .out)

I followed your direction, installed new 1.3.5 version. But how can I confirm that I'm running 1.3.5? The README and RELEASE NOTE are still showing 1.3.4 info.

@wkpalan
Copy link
Collaborator

wkpalan commented Oct 17, 2021

Hey @huh688.

Can you please get the latest version which is 1.3.7 and check of the output improves. There were some bugs related to this I had to fix.

The README and CHANGELOG have been updated.

@huh688
Copy link
Author

huh688 commented Dec 2, 2021

Hi @wkpalan,

I reran the Sorghum dataset with v1.3.8 build, and got the 100% coverage of all 34,129 proteins.

I also reran several different species that I got lower coverage before, and found 100% sequences were successfully annotated with v1.3.8.

I think the problem is fixed now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wkpalan wkpalan

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wkpalan @huh688