error 'cannot get position' when running bam2gam.sh #72

olechnwin · 2021-03-30T19:56:16Z

Hi,
I'm trying to run bam2gam for my own dataset with the following command:

singularity exec --pwd /home/xxt050/opt/MoMI-G/scripts docker://momigteam/momig-tools:autobuild bash -x bam2gam.sh /home/xxt050/Data/A673_pacbio/HiFi/bam/A673.HiFi.hg19.sorted.md.bam /home/xxt050/Data/A673_pacbio/HiFi/vcf/momig_input.xg /vg/bin/vg

and encountered this error:

/vg/bin/vg annotate -p -x /home/xxt050/Data/A673_pacbio/HiFi/vcf/momig_input.xg -b .//.bed.2
terminate called after throwing an instance of 'std::runtime_error'
  what():  Cannot get position 159109948 along path chr7 of length 159101024

The momig_input.xg was generated using:

singularity run docker://momigteam/momig-tools:autobuild ~/Data/A673_pacbio/HiFi/vcf/A673.HiFi.hg19.sniffles.TRA.vcf momig_input ~/opt/vg ~/hg19/hg19.fa

Can you please help me figure out how to fix this?

The text was updated successfully, but these errors were encountered:

6br · 2021-03-31T02:10:48Z

chr7 of length in hg19 is 159138663, so I suspect that chr7 in momig_input.xg is truncated. Could you check this?

singularity -s exec docker://quay.io/vgteam/vg:v1.25.0 vg paths -L -E -x momig_input.xg | awk '/^chr7/' | grep -v v

Also, did you see any error messages when you run singularity run docker://momigteam/momig-tools:autobuild ~/Data/A673_pacbio/HiFi/vcf/A673.HiFi.hg19.sniffles.TRA.vcf momig_input ~/opt/vg ~/hg19/hg19.fa?

olechnwin · 2021-03-31T13:24:21Z

Thank you again for your continued support.
That's what I thought the error message was saying but had no idea what to do.

Here's the output:

singularity -s exec docker://quay.io/vgteam/vg:v1.25.0 vg paths -E -x ~/Data/A673_pacbio/HiFi/vcf/momig_input.xg | awk '/^chr7/' | grep -v v
chr7_gl000195_random    151024
chr7    159101024

It is indeed truncated.
Also, I did not see any error message when I ran the vg tool.

singularity run docker://momigteam/momig-tools:autobuild ~/Data/A673_pacbio/HiFi/vcf/A673.HiFi.hg19.sniffles.TRA.vcf momig_input ~/opt/vg ~/hg19/hg19.fa
+ singularity run docker://momigteam/momig-tools:autobuild /home/xxt050/Data/A673_pacbio/HiFi/vcf/A673.HiFi.hg19.sniffles.TRA.vcf momig_input /home/xxt050/opt/vg /home/xxt050/hg19/hg19.fa
INFO:    Using cached SIF image
+ input=.//momig_input.pcf
+ reference=/home/xxt050/hg19/hg19.fa
+ pref=.//momig_input.pcf
+ sort -t, -k 7n .//momig_input.pcf
+ sed -e 's/[ ]*//g'
+ sort -k 1,1 -k 2,2n
+ uniq
+ awk -F '[,:]' '{print $1,"\t",$2,"\n",$4,"\t",$5}' /dev/fd/63
++ sed 1d .//momig_input.pcf
+ '[' '!' -s /home/xxt050/hg19/hg19.fa.fai ']'
++ dirname /usr/src/app/vcf2gfa/pcf2gfa.sh
+ ruby /usr/src/app/vcf2gfa/json_to_breakpoint_list.rb .//momig_input.pcf.bp.tsv /home/xxt050/hg19/hg19.fa
+ sort -k 1,1 -k 2,2n
+ uniq
+ cat .//momig_input.pcf.bp.tsv breakpoint_list_tmp.tsv
++ dirname /usr/src/app/vcf2gfa/pcf2gfa.sh
+ ruby /usr/src/app/vcf2gfa/gfa_generator.rb .//momig_input.pcf.bp.merged.tsv .//momig_input.pcf.output.pcf /home/xxt050/hg19/hg19.fa
Job Statistics for 323672:
           JobID       User               Start                 End    Elapsed     MaxRSS   TotalCPU      State Exit        NodeList                                  ReqTRES 
---------------- ---------- ------------------- ------------------- ---------- ---------- ---------- ---------- ---- --------------- ---------------------------------------- 
          323672     xxt050 2021-03-30T11:08:27 2021-03-30T11:27:49   00:19:22             14:50.734  COMPLETED  0:0   n02         billing=1,cpu=1,mem=7812M,node=1

6br · 2021-04-01T04:29:29Z

When MoMI-G tools are executed in Singularity, Singularity seems to discard the original stderr. I assume that the error messages were lost due to this behavior. I modified the script to output messages to stdout, so is it possible to run singularity again?

Also, you can validate the gfa format via gfalint <momig_input.gfa 2> momig_input.log. gfalint is available at https://github.com/sjackman/gfalint.

olechnwin · 2021-04-01T14:47:02Z

I've re-run vg from docker, but it doesn't seem to have a more useful error messages other than "potentially truncated"

singularity run docker://momigteam/momig-tools:autobuild ~/Data/A673_pacbio/HiFi/vcf/A673.HiFi.hg19.sniffles.TRA.vcf momig_input /vg/bin/vg ~/hg19/hg19.fa
+ singularity run docker://momigteam/momig-tools:autobuild /home/xxt050/Data/A673_pacbio/HiFi/vcf/A673.HiFi.hg19.sniffles.TRA.vcf momig_input /vg/bin/vg /home/xxt050/hg19/hg19.fa
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:d7c3167c320d7a820935f54cf4290890ea19567da496ecf774e8773b35d5f065
Copying blob sha256:131f805ec7fd68d45a887e2ef82de61de0247b4eb934ab03b7c933650e854baa
Copying blob sha256:322ed380e680a77f30528ba013e3a802a7b44948a0609c7d1d732dd46a9a310d
Copying blob sha256:6ac240b130982ad1c3ba3188abbf18ba4e54bdd9e504ce2d5c2eff6d3e86b8dd
Copying blob sha256:856d98c9bfe8f4e7b52efa66486ea8190377f2de7240ea443eead8e59136d72f
Copying blob sha256:5560107aff26c0047ea863e633c8bbd693b3f8338ac7ed2b5f999973a63bf6a3
Copying blob sha256:930a42de8e687dc902163d8ab8363f73d7787b4c595c1f4148680ba66cc9999e
Copying blob sha256:b0cbb0a3ab3d9cd042b9ad60fef1c07980c4f4f237663c0a15999a4b3633fb26
Copying blob sha256:3d91deb1c564563078c9c8627361d8297a16a0cab5f5e4e5503944e751d15b10
Copying blob sha256:12668ae08e93a126123d2dd2d1d634248ea9fd74d1942824cb1f145d62f2e0cc
Copying blob sha256:b8939cdb0cd4955b12503c576e96a9fc473b9644d1cebe855784599ea54f7b96
Copying blob sha256:e4109c5ee115c7d79e0e9778048ce9b462ce457fca6f86cdca187bdcbced722d
Copying blob sha256:3a368b393ab7a45d5b102b9ed2bb0d7b45e2c47c5d15439730a03ef74d004b3b
Copying blob sha256:2e167ac4bee58176054f2735cb3d395b87114ffcfaa7e5e40e89c5e1b13ad0eb
Copying config sha256:380d434b70dee89d9be229e7fdfa7fdc57b7ea31351523ae2129f983de140f90
Writing manifest to image destination
Storing signatures
2021/04/01 09:47:23  info unpack layer: sha256:d7c3167c320d7a820935f54cf4290890ea19567da496ecf774e8773b35d5f065
2021/04/01 09:47:24  info unpack layer: sha256:131f805ec7fd68d45a887e2ef82de61de0247b4eb934ab03b7c933650e854baa
2021/04/01 09:47:24  info unpack layer: sha256:322ed380e680a77f30528ba013e3a802a7b44948a0609c7d1d732dd46a9a310d
2021/04/01 09:47:24  info unpack layer: sha256:6ac240b130982ad1c3ba3188abbf18ba4e54bdd9e504ce2d5c2eff6d3e86b8dd
2021/04/01 09:47:24  info unpack layer: sha256:856d98c9bfe8f4e7b52efa66486ea8190377f2de7240ea443eead8e59136d72f
2021/04/01 09:47:24  info unpack layer: sha256:5560107aff26c0047ea863e633c8bbd693b3f8338ac7ed2b5f999973a63bf6a3
2021/04/01 09:47:24  info unpack layer: sha256:930a42de8e687dc902163d8ab8363f73d7787b4c595c1f4148680ba66cc9999e
2021/04/01 09:47:24  info unpack layer: sha256:b0cbb0a3ab3d9cd042b9ad60fef1c07980c4f4f237663c0a15999a4b3633fb26
2021/04/01 09:47:24  info unpack layer: sha256:3d91deb1c564563078c9c8627361d8297a16a0cab5f5e4e5503944e751d15b10
2021/04/01 09:47:24  info unpack layer: sha256:12668ae08e93a126123d2dd2d1d634248ea9fd74d1942824cb1f145d62f2e0cc
2021/04/01 09:47:24  info unpack layer: sha256:b8939cdb0cd4955b12503c576e96a9fc473b9644d1cebe855784599ea54f7b96
2021/04/01 09:47:30  info unpack layer: sha256:e4109c5ee115c7d79e0e9778048ce9b462ce457fca6f86cdca187bdcbced722d
2021/04/01 09:47:30  info unpack layer: sha256:3a368b393ab7a45d5b102b9ed2bb0d7b45e2c47c5d15439730a03ef74d004b3b
2021/04/01 09:47:30  info unpack layer: sha256:2e167ac4bee58176054f2735cb3d395b87114ffcfaa7e5e40e89c5e1b13ad0eb
INFO:    Creating SIF file...
+ input=.//momig_input.pcf
+ reference=/home/xxt050/hg19/hg19.fa
+ pref=.//momig_input.pcf
+ sort -t, -k 7n .//momig_input.pcf
+ sed -e 's/[ ]*//g'
+ sort -k 1,1 -k 2,2n
+ uniq
+ awk -F '[,:]' '{print $1,"\t",$2,"\n",$4,"\t",$5}' /dev/fd/63
++ sed 1d .//momig_input.pcf
+ '[' '!' -s /home/xxt050/hg19/hg19.fa.fai ']'
++ dirname /usr/src/app/vcf2gfa/pcf2gfa.sh
+ ruby /usr/src/app/vcf2gfa/json_to_breakpoint_list.rb .//momig_input.pcf.bp.tsv /home/xxt050/hg19/hg19.fa
+ sort -k 1,1 -k 2,2n
+ uniq
+ cat .//momig_input.pcf.bp.tsv breakpoint_list_tmp.tsv
++ dirname /usr/src/app/vcf2gfa/pcf2gfa.sh
+ ruby /usr/src/app/vcf2gfa/gfa_generator.rb .//momig_input.pcf.bp.merged.tsv .//momig_input.pcf.output.pcf /home/xxt050/hg19/hg19.fa
Potentially truncated chromosomes:
Job Statistics for 324213:
           JobID       User               Start                 End    Elapsed     MaxRSS   TotalCPU      State Exit        NodeList                                  ReqTRES 
---------------- ---------- ------------------- ------------------- ---------- ---------- ---------- ---------- ---- --------------- ---------------------------------------- 
          324213     xxt050 2021-04-01T09:47:18 2021-04-01T10:10:17   00:22:59             17:27.064  COMPLETED  0:0   r1pl-hpcf-n02         billing=1,cpu=1,mem=7812M,node=1

I wasn't able to compile gfalint, so I can't validate the gfa format.

6br · 2021-04-02T03:20:22Z

Is it possible to check if hg19.fa.fai is correct?

$ grep chr7 ~/hg19/hg19.fa.fai
chr7	159138663	1258330213	50	51
chr7_gl000195_random	182896	3195389782	50	51

The output Potentially truncated chromosomes: (nothing) suggests there are no truncated chromosomes during the process. So I am wondering why paths are truncated...

olechnwin · 2021-04-02T14:03:22Z

I have the same thing:

[xxt050@log01 ~]$ grep chr7 ~/hg19/hg19.fa.fai
chr7    159138663       1258330213      50      51
chr7_gl000195_random    182896  3195389782      50      51

6br · 2021-04-02T23:36:51Z

Are other chromosomes truncated?
singularity -s exec docker://quay.io/vgteam/vg:v1.25.0 vg paths -E -x momig_input.xg | awk '/^c/' | grep -v v

olechnwin · 2021-04-05T13:48:17Z

This is what I got from the above command:
singularity -s exec docker://quay.io/vgteam/vg:v1.25.0 vg paths -E -x momig_input.xg | awk '/^c/' | grep -v v | sort -V

chr1    249250621
chr1_gl000191_random    106433
chr1_gl000192_random    547496
chr2    243199373
chr3    198022430
chr4    191154276
chr4_ctg9_hap1  590426
chr4_gl000193_random    189789
chr4_gl000194_random    191469
chr5    180915260
chr6    171115067
chr6_apd_hap1   4622290
chr6_cox_hap2   4795371
chr6_dbb_hap3   4610396
chr6_mann_hap4  4683263
chr6_mcf_hap5   4833398
chr6_qbl_hap6   4611984
chr6_ssto_hap7  4928567
chr7    159138663
chr7_gl000195_random    182896
chr8    146364022
chr9    141213431
chr9_gl000198_random    90085
chr9_gl000199_random    169874
chr9_gl000200_random    187035
chr10   135534747
chr11   135006516
chr12   133851895
chr13   115169878
chr14   107349540
chr15   102531392
chr16   90354753
chr17   81195210
chr17_ctg5_hap1 1680828
chr17_gl000203_random   37498
chr17_gl000204_random   81310
chr17_gl000205_random   174588
chr18   78077248
chr18_gl000207_random   4262
chr19   59128983
chr19_gl000208_random   92689
chr19_gl000209_random   159169
chr20   63025520
chr21   48129895
chr21_gl000210_random   27682
chr22   51304566
chrUn_gl000211  166566
chrUn_gl000212  186858
chrUn_gl000213  164239
chrUn_gl000214  137718
chrUn_gl000215  172545
chrUn_gl000216  172294
chrUn_gl000217  172149
chrUn_gl000218  161147
chrUn_gl000219  179198
chrUn_gl000220  161802
chrUn_gl000221  155397
chrUn_gl000222  186861
chrUn_gl000223  180455
chrUn_gl000224  179693
chrUn_gl000225  211173
chrUn_gl000226  15008
chrUn_gl000227  128374
chrUn_gl000228  129120
chrUn_gl000229  19913
chrUn_gl000231  27386
chrUn_gl000232  40652
chrUn_gl000234  40531
chrUn_gl000235  34474
chrUn_gl000237  45867
chrUn_gl000241  42152
chrX    155270560
chrY    59373566

I'm confused. Isn't this checking whether chromosomes are truncated in my momig_input.xg? But, the chr7 is no longer truncated?

6br · 2021-04-06T01:25:01Z

Strange, but anyway, it seems no longer truncated.
Is it possible to run bam2gam.sh again?

olechnwin · 2021-04-07T15:24:09Z

It is strange. Don't know what happened...

I was able to run bam2gam.sh to completion. Thank you! But the output files do not have any names.
Is there something I missed? Also, is there a way to specify the location of the outputs?

These are the output files:

.bed
.bed.1
.bed.2
.gam
.gam.1
.gam.index
.gam.json
.gam.json.2
.json
sankey.rb
.sorted.gam
.sorted.gam.gai

and these are the commands I ran:

cd ~/Data/A673_pacbio/HiFi/vcf/
+ cd /home/xxt050/Data/A673_pacbio/HiFi/vcf/
export TMPDIR=/tmp
+ export TMPDIR=/tmp
+ TMPDIR=/tmp
export SINGULARITY_LOCALCACHEDIR=/tmp
+ export SINGULARITY_LOCALCACHEDIR=/tmp
+ SINGULARITY_LOCALCACHEDIR=/tmp
export SINGULARITY_CACHEDIR=/tmp
+ export SINGULARITY_CACHEDIR=/tmp
+ SINGULARITY_CACHEDIR=/tmp
export SINGULARITY_TMPDIR=/tmp
+ export SINGULARITY_TMPDIR=/tmp
+ SINGULARITY_TMPDIR=/tmp

singularity exec --pwd ~/opt/MoMI-G/scripts docker://momigteam/momig-tools:autobuild bash -x bam2gam.sh ~/Data/A673_pacbio/HiFi/bam/A673.HiFi.hg19.sorted.md.bam ~/Data/A673_pacbio/HiFi/vcf/momig_input.xg /vg/bin/vg
+ singularity exec --pwd /home/xxt050/opt/MoMI-G/scripts docker://momigteam/momig-tools:autobuild bash -x bam2gam.sh /home/xxt050/Data/A673_pacbio/HiFi/bam/A673.HiFi.hg19.sorted.md.bam /home/xxt050/Data/A673_pacbio/HiFi/vcf/momig_input.xg /vg/bin/vg
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:d7c3167c320d7a820935f54cf4290890ea19567da496ecf774e8773b35d5f065
Copying blob sha256:131f805ec7fd68d45a887e2ef82de61de0247b4eb934ab03b7c933650e854baa
Copying blob sha256:322ed380e680a77f30528ba013e3a802a7b44948a0609c7d1d732dd46a9a310d
Copying blob sha256:6ac240b130982ad1c3ba3188abbf18ba4e54bdd9e504ce2d5c2eff6d3e86b8dd
Copying blob sha256:856d98c9bfe8f4e7b52efa66486ea8190377f2de7240ea443eead8e59136d72f
Copying blob sha256:5560107aff26c0047ea863e633c8bbd693b3f8338ac7ed2b5f999973a63bf6a3
Copying blob sha256:930a42de8e687dc902163d8ab8363f73d7787b4c595c1f4148680ba66cc9999e
Copying blob sha256:b0cbb0a3ab3d9cd042b9ad60fef1c07980c4f4f237663c0a15999a4b3633fb26
Copying blob sha256:3d91deb1c564563078c9c8627361d8297a16a0cab5f5e4e5503944e751d15b10
Copying blob sha256:12668ae08e93a126123d2dd2d1d634248ea9fd74d1942824cb1f145d62f2e0cc
Copying blob sha256:b8939cdb0cd4955b12503c576e96a9fc473b9644d1cebe855784599ea54f7b96
Copying blob sha256:e4109c5ee115c7d79e0e9778048ce9b462ce457fca6f86cdca187bdcbced722d
Copying blob sha256:3a368b393ab7a45d5b102b9ed2bb0d7b45e2c47c5d15439730a03ef74d004b3b
Copying blob sha256:668a8b7b7686c0c74661f9efb1ae8a7ce5a1644615fa7a20b83694304d4199cc
Copying config sha256:61a3dd33d6a44fb609b6e03ddbcad35bd4ca8dabd403b4c479394162d6a3f04c
Writing manifest to image destination
Storing signatures
2021/04/07 08:23:08  info unpack layer: sha256:d7c3167c320d7a820935f54cf4290890ea19567da496ecf774e8773b35d5f065
2021/04/07 08:23:09  info unpack layer: sha256:131f805ec7fd68d45a887e2ef82de61de0247b4eb934ab03b7c933650e854baa
2021/04/07 08:23:09  info unpack layer: sha256:322ed380e680a77f30528ba013e3a802a7b44948a0609c7d1d732dd46a9a310d
2021/04/07 08:23:09  info unpack layer: sha256:6ac240b130982ad1c3ba3188abbf18ba4e54bdd9e504ce2d5c2eff6d3e86b8dd
2021/04/07 08:23:09  info unpack layer: sha256:856d98c9bfe8f4e7b52efa66486ea8190377f2de7240ea443eead8e59136d72f
2021/04/07 08:23:09  info unpack layer: sha256:5560107aff26c0047ea863e633c8bbd693b3f8338ac7ed2b5f999973a63bf6a3
2021/04/07 08:23:09  info unpack layer: sha256:930a42de8e687dc902163d8ab8363f73d7787b4c595c1f4148680ba66cc9999e
2021/04/07 08:23:09  info unpack layer: sha256:b0cbb0a3ab3d9cd042b9ad60fef1c07980c4f4f237663c0a15999a4b3633fb26
2021/04/07 08:23:09  info unpack layer: sha256:3d91deb1c564563078c9c8627361d8297a16a0cab5f5e4e5503944e751d15b10
2021/04/07 08:23:09  info unpack layer: sha256:12668ae08e93a126123d2dd2d1d634248ea9fd74d1942824cb1f145d62f2e0cc
2021/04/07 08:23:10  info unpack layer: sha256:b8939cdb0cd4955b12503c576e96a9fc473b9644d1cebe855784599ea54f7b96
2021/04/07 08:23:15  info unpack layer: sha256:e4109c5ee115c7d79e0e9778048ce9b462ce457fca6f86cdca187bdcbced722d
2021/04/07 08:23:15  info unpack layer: sha256:3a368b393ab7a45d5b102b9ed2bb0d7b45e2c47c5d15439730a03ef74d004b3b
2021/04/07 08:23:15  info unpack layer: sha256:668a8b7b7686c0c74661f9efb1ae8a7ce5a1644615fa7a20b83694304d4199cc
INFO:    Creating SIF file...
+ '[' 3 -lt 3 ']'
+ bam_file=/home/xxt050/Data/A673_pacbio/HiFi/bam/A673.HiFi.hg19.sorted.md.bam
+ xg_file=/home/xxt050/Data/A673_pacbio/HiFi/vcf/momig_input.xg
+ vg_path=/vg/bin/vg
+ uuid=
+ tmp_dir=./
+ bed_file=.//.bed
+ gam_file=.//.gam
+ gam_sorted_file=.//.sorted.gam
+ gam_json_file=.//.gam.json
+ json_file=.//.json
+ '[' 3 -lt 4 ']'
+ uuid=/home/xxt050/Data/A673_pacbio/HiFi/bam/A673.HiFi.hg19.sorted.md.bam
+ : /vg/bin/vg version
+ echo '{"current": 0, "max": 5}'
+ bedtools bamtobed -cigar -i /home/xxt050/Data/A673_pacbio/HiFi/bam/A673.HiFi.hg19.sorted.md.bam
+ echo '{"current": 1, "max": 5}'
+ gawk '($6=="-"){$7=gensub(/(.*)[^0-9]([0-9]+)S$/, "\\2", "g", $7); gsub(/^.*\D.*$/, "0", $7)}1'
+ sort -k 4,4 -k 7,7n
+ gawk '($6=="+"){gsub(/S.*/, "", $7);gsub(/^.*\D.*$/, "0", $7)}1' .//.bed
+ grep -v chrM
+ awk '$1!~/_/{print $0}'
+ grep -v chrEBV
+ cat .//.bed.1
++ dirname bam2gam.sh
+ ruby ./bam2gam/uuid.rb
+ echo '{"current": 2, "max": 5}'
+ /vg/bin/vg annotate -p -x /home/xxt050/Data/A673_pacbio/HiFi/vcf/momig_input.xg -b .//.bed.2
+ /vg/bin/vg view -a .//.gam.1
+ echo '{"current": 3, "max": 5}'
+ cat .//.gam.json
++ dirname bam2gam.sh
+ ruby ./bam2gam/json.rb
+ echo '{"current": 4, "max": 5}'
+ /vg/bin/vg view -aJG .//.gam.json.2
+ /vg/bin/vg index -t 12 -N .//.gam -d .//.gam.index
+ /vg/bin/vg gamsort -d -t 12 .//.gam
+ /vg/bin/vg index -l .//.sorted.gam
+ echo '{"current": 5, "max": 5}'

6br · 2021-04-08T02:10:41Z

The fourth argument of bam2gam.sh would be a prefix for outputs of bam2gam.sh.

singularity exec --pwd ~/opt/MoMI-G/scripts docker://momigteam/momig-tools:autobuild bash -x bam2gam.sh ~/Data/A673_pacbio/HiFi/bam/A673.HiFi.hg19.sorted.md.bam ~/Data/A673_pacbio/HiFi/vcf/momig_input.xg /vg/bin/vg prefix

olechnwin · 2021-04-08T13:22:02Z

Great! Thank you!

olechnwin · 2021-04-09T12:51:06Z

@6br,

I was able to generate .gam and gam.index files/folder. But, the SequenceTubeMap is not displayed.
This is my config.yaml:

bin:
  vg: "vg"
  vg_tmp: "vg"
  vg_volume_prefix: ""
  graphviz: "dot"
  fa22bit: "faToTwoBit"
  bigbed: "bedToBigBed"
reference:
  chroms: "static/GRCh.json"
  data:
    - name: "hg19"
      features:
        - name: 'gene_annotation'
          url: "static/gencode.v37lift37.basic.annotation.gff3"
          chr_prefix: "chr"
data:
  - name: "a673_hifi"
    desc: "2021/03/29"
    chr_prefix: "chr"
    ref_id: "hg19"
    source:
      xg: "static/momig_input.xg" # To be rewritten
      csv: "static/momig_input.pcf" # To be rewritten
      gam: "static/A673.HiFi.hg19.gam"
      gamindex: "static/A673.HiFi.hg19.gam.index"
    features: []
    static_files: []

and this is the error from the backend-server:

singularity run --bind ~/Data/A673_pacbio/HiFi/vcf:/vg/static --pwd /vg docker://momigteam/momig-backend
INFO:    Using cached SIF image
[2021-04-09T12:40:58Z INFO  graph_genome_browser_backend::vg] VG Version: v1.6.0-213-gc0c19fe

[2021-04-09T12:40:58Z INFO  graph_genome_browser_backend::vg] VG minor version: 6
[2021-04-09T12:40:58Z INFO  graph_genome_browser_backend::features] Parsing:  "static/gencode.v37lift37.basic.annotation.gff3"
6
Start server on 0.0.0.0:8081
[2021-04-09T12:41:51Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=features 302 Found (0.095167 ms)
[2021-04-09T12:41:51Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=chromosomes 302 Found (0.017563 ms)
[2021-04-09T12:41:51Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=metadata 200 OK (0.016952 ms)
[2021-04-09T12:41:51Z INFO  graph_genome_browser_backend::handlers] chr12:80851974-80853202
[2021-04-09T12:41:51Z INFO  graph_genome_browser_backend::handlers] http://localhost:3000/api/v2/cache/raw_chr12:80851974-80853202.json, ./tmp/raw_chr12:80851974-80853202.json
[2021-04-09T12:41:51Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr12:80851974-80853202 500 Internal Server Error (0.483713 ms)
[2021-04-09T12:41:51Z INFO  logger] GET http://localhost:3000/api/v2/static/momig_input.pcf 200 OK (0.086842 ms)
[2021-04-09T12:41:51Z INFO  logger] GET http://localhost:3000/api/v2/static/GRCh.json 200 OK (0.103193 ms)
[2021-04-09T12:42:08Z INFO  graph_genome_browser_backend::handlers] chr11:128667505-128668505
[2021-04-09T12:42:08Z INFO  graph_genome_browser_backend::handlers] http://localhost:3000/api/v2/cache/raw_chr11:128667505-128668505.json, ./tmp/raw_chr11:128667505-128668505.json
[2021-04-09T12:42:08Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr11:128667505-128668505 500 Internal Server Error (0.426998 ms)
[2021-04-09T12:42:14Z INFO  graph_genome_browser_backend::handlers] chr11:128667505-128668505
[2021-04-09T12:42:14Z INFO  graph_genome_browser_backend::handlers] http://localhost:3000/api/v2/cache/raw_chr11:128667505-128668505.json, ./tmp/raw_chr11:128667505-128668505.json
[2021-04-09T12:42:14Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr11:128667505-128668505 500 Internal Server Error (0.25738 ms)
[2021-04-09T12:43:22Z INFO  graph_genome_browser_backend::handlers] chr11:49733421-49752882
[2021-04-09T12:43:22Z INFO  graph_genome_browser_backend::handlers] http://localhost:3000/api/v2/cache/raw_chr11:49733421-49752882.json, ./tmp/raw_chr11:49733421-49752882.json
[2021-04-09T12:43:22Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr11:49733421-49752882 500 Internal Server Error (0.295752 ms)
[2021-04-09T12:43:27Z INFO  graph_genome_browser_backend::handlers] chr11:49733421-49752882
[2021-04-09T12:43:27Z INFO  graph_genome_browser_backend::handlers] http://localhost:3000/api/v2/cache/raw_chr11:49733421-49752882.json, ./tmp/raw_chr11:49733421-49752882.json
[2021-04-09T12:43:27Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr11:49733421-49752882 500 Internal Server Error (0.419243 ms)

fyi, I just realized that running example CHM1 data on my own backend-server also doesn't display SequenceTubeMap.

Thank you again for your help.

6br · 2021-04-10T14:03:21Z

For the problem that running example CHM1 data on my own backend-server also doesn't display SequenceTubeMap, this might be related with #70 . I will check the behavior and update the codes if needed.
fyi, if vg v1.25 is used when running momi-g tools,docker://momigteam/momig-backend:prod will provide momi-g backend with vg v1.25. otherwise, momi-g tools can run with vg v1.6 docker://momigteam/momig-tools:6
singularity run or singularity exec command seems to lose stderr output, which makes it difficult to identify the problem. If you are using singularity >= 3, --vm-err option might help to inspect the error.

6br · 2021-04-10T14:32:47Z

For CHM1 data, my singularity environment works. Note that the CHM1 example contains only the region on chr21. e.g.http://localhost:3000/#annotations=true&layout=0&path=chr21%3A18607192-18608192-INS-3216&reference=hg38&uuid=
So is it possible to confirm if CHM1 data works?

olechnwin · 2021-04-12T13:12:49Z

I'm not sure how to navigate to the exact location that you referenced above for the CHM1 example data. Tried copy pasting that address in firefox but it doesn't look like it's doing anything.
So, I clicked on the first entry of the feature table and this is what it looks like:

edit: I tried singularity run --vm-err but only get the same internal server error.

olechnwin · 2021-04-12T14:07:50Z

Looks like I was using vg version 1.25:

singularity exec --pwd ~/opt/MoMI-G/scripts docker://momigteam/momig-tools:autobuild /vg/bin/vg
INFO:    Using cached SIF image
vg: variation graph tool, version v1.25.0 "Apice"

So, I tried your command to get vg 1.25 for backend-server, but I think it's still getting version 1.6 and I still have the same internal server error:

singularity run --pwd /vg docker://momigteam/momig-backend:prod
INFO:    Creating SIF file...
[2021-04-12T13:57:50Z INFO  graph_genome_browser_backend::vg] VG Version: v1.6.0-213-gc0c19fe
[2021-04-12T13:57:50Z INFO  graph_genome_browser_backend::vg] VG minor version: 6

I'm going to try to re-generate the files using vg 1.6

olechnwin · 2021-04-12T15:02:34Z

Tried to run vg 1.6 but got an error:

singularity run docker://momigteam/momig-tools:6 ~/Data/A673_pacbio/HiFi/vcf/A673.HiFi.hg19.sniffles.TRA.vcf momig_input_vg1_6 /vg/bin/vg ~/hg19/hg19.fa
...
ruby /usr/src/app/vcf2gfa/gfa_generator.rb .//momig_input_vg1_6.pcf.bp.merged.tsv .//momig_input_vg1_6.pcf.output.pcf /home/xxt050/hg19/hg19.fa
error:[vg] command convert not found
vg: variation graph tool, version v1.6.0-213-gc0c19fe

6br · 2021-04-13T02:14:01Z

For CHM1 demo, singularity run --pwd /vg docker://momigteam/momig-backend works in my environment. In CHM1 demo, no internal server error but not shown tubemap, is it true?

For docker://momigteam/momig-tools:6, now I fixed the bug error:[vg] command convert not found. But for bam2gam conversion, I don't recommend using vg v1.6.

For Internal Server Error, the old version of docker://momigteam/momig-backend:prod seemed to build on vg v1.6, so I modified to build on vg 1.25.

Also, is it possible to specify .sorted.gam instead of .gam in gam section on config.yaml?

data:
  - name: "a673_hifi"
    desc: "2021/03/29"
    chr_prefix: "chr"
    ref_id: "hg19"
    source:
      xg: "static/momig_input.xg" # To be rewritten
      csv: "static/momig_input.pcf" # To be rewritten
      gam: "static/A673.HiFi.hg19.sorted.gam"
      gamindex: "static/A673.HiFi.hg19.gam.index"
    features: []
    static_files: []

olechnwin · 2021-04-13T13:08:43Z

For CHM1 demo, singularity run --pwd /vg docker://momigteam/momig-backend doesn't have internal server error in the beginning and no tubemap. The internal server error appears after I clicked on the feature table and still no tubemap. I'm attaching the stdout and stderr in output.txt

Thanks for fixing the bug error:[vg] command convert not found. So, I should use the docker://momigteam/momig-tools:6 to regenerate only the xg and pcf files?

I'll try to use the docker://momigteam/momig-backend:prod and sorted.gam

olechnwin · 2021-04-13T17:41:51Z

I tried to run the following to display CHM1 demo running my own backend server using docker://momigteam/momig-backend:prod so I can use the vg 1.25 but got the following error:

singularity run --pwd /vg docker://momigteam/momig-backend:prod
INFO:    Using cached SIF image
thread 'main' panicked at 'couldn't open static/config.yaml: entity not found', src/utils.rs:12:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

6br · 2021-04-14T05:32:11Z

For CHM1 demo, that's strange... I run the same command on my environment.
I attached my log: output-1.txt

I recommend to use docker://momigteam/momig-tools to build xg, pcf and gam files and running it on docker://momigteam/momig-backend:prod. Although docker://momigteam/momig-backend:prod does not contain CHM1 demo, this is used to run data built on docker://momigteam/momig-tools.

olechnwin · 2021-04-16T13:19:48Z

I still cannot get both CHM1 demo and my own data to display SequenceTube.

I don't know if this helps, but I noticed in your CHM1 demo log, you have static/chm1_chr21.xg that never appeared in my log. I tried clicking many entries in the feature table, the line with static/chm1_chr21.xg was never there. Only the chm1_chr21.pcf is there.
output_chm1.txt

6br · 2021-04-17T01:34:00Z

There are possible causes of the behavior.

Backend needs to create a tmp file when backend retrieves data from vg, but it failed because the container is available as read-only. (--writable-tmpfs or --writable option of singularity might solve this.)
Parameters provided on requests for backend is not proper. (Because the parameters are the same as I tried, so I think parameters are good)
It failed to run vg on the container. (If so, I assume that we can see any error messages on log)

6br · 2021-04-17T02:57:53Z

Did you see any return message when you access to http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr21:15811802-15812802 via web browser?

Also, to help to identify the cause of this behavior, now I updated the debug messages on docker://momigteam/momig-backend.

[2021-04-17T05:02:07Z INFO  graph_genome_browser_backend::handlers] Range: chr21:15811802-15812802
[2021-04-17T05:02:07Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:8081/api/v2/cache/raw_chr21:15811802-15812802.json, ./tmp/raw_chr21:15811802-15812802.json
[2021-04-17T05:02:07Z DEBUG graph_genome_browser_backend::vg] Create a temporary file: "./tmp/raw_chr21:15811802-15812802.json"
[2021-04-17T05:02:07Z INFO  graph_genome_browser_backend::vg] static/chm1_chr21.xg, chr21:15811802-15812802
[2021-04-17T05:02:07Z INFO  graph_genome_browser_backend::vg] VG version: 6, cmd: ["vg"]
[2021-04-17T05:02:07Z DEBUG graph_genome_browser_backend::vg] ["chunk", "-t", "4", "-x", "static/chm1_chr21.xg", "-p", "chr21:15811802-15812802", "-c", "3", "-g", "-A", "-a", "static/t.gam.index", "-b", "./tmp/raw_chr21:15811802-15812802.json"]
[2021-04-17T05:02:07Z DEBUG graph_genome_browser_backend::vg] Status codes: exit code: 0 exit code: 0 exit code: 0
[2021-04-17T05:02:07Z INFO  logger] GET http://localhost:8081/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr21:15811802-15812802 302 Found (456.490551 ms)

olechnwin · 2021-04-19T14:01:20Z

I tried run singularity with --writable-tmpfs but the sequenceTube still doesn't appear and I don't have the status code. I made sure to clean up the cache before re-running:

singularity run --writable-tmpfs --pwd /vg docker://momigteam/momig-backend
INFO:    Using cached SIF image
[2021-04-19T13:51:25Z INFO  graph_genome_browser_backend::vg] VG Version: v1.6.0-213-gc0c19fe

[2021-04-19T13:51:25Z INFO  graph_genome_browser_backend::vg] VG minor version: 6
[2021-04-19T13:51:25Z INFO  graph_genome_browser_backend::features] Parsing:  "static/gencode.v27.basic.annotation.gff3"
[2021-04-19T13:51:58Z DEBUG graph_genome_browser_backend::features] 1641293 lines processed. end.
6
Start server on 0.0.0.0:8081
[2021-04-19T13:52:14Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=features 302 Found (0.092904 ms)
[2021-04-19T13:52:14Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=chromosomes 302 Found (0.093524 ms)
[2021-04-19T13:52:14Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=metadata 200 OK (0.063879 ms)
[2021-04-19T13:52:14Z INFO  graph_genome_browser_backend::handlers] Range: chr12:80851974-80853202
[2021-04-19T13:52:14Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr12:80851974-80853202.json, ./tmp/raw_chr12:80851974-80853202.json
[2021-04-19T13:52:14Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr12:80851974-80853202 500 Internal Server Error (0.501036 ms)
[2021-04-19T13:52:14Z INFO  logger] GET http://localhost:3000/api/v2/static/GRCh.json 200 OK (0.174897 ms)
[2021-04-19T13:52:14Z INFO  logger] GET http://localhost:3000/api/v2/static/chm1_chr21.pcf 200 OK (0.077906 ms)
[2021-04-19T13:52:18Z INFO  graph_genome_browser_backend::handlers] Range: chr21:15811802-15812802
[2021-04-19T13:52:18Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr21:15811802-15812802.json, ./tmp/raw_chr21:15811802-15812802.json
[2021-04-19T13:52:18Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr21:15811802-15812802 500 Internal Server Error (0.402993 ms)
[2021-04-19T13:52:33Z INFO  graph_genome_browser_backend::handlers] Range: chr21:15811802-15812802
[2021-04-19T13:52:33Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr21:15811802-15812802.json, ./tmp/raw_chr21:15811802-15812802.json
[2021-04-19T13:52:33Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr21:15811802-15812802 500 Internal Server Error (0.358529 ms)

This time, I was able to navigate to http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr21:15811802-15812802 and this is the error message:

It's graph.json but when I click to save, it only has a sentence permission denied.

olechnwin · 2021-04-19T15:09:30Z

I think I got further by binding /tmp to /vg/tmp but still no SequenceTube. I can now see it tries to load the index .xg file but failed. The status code you added is also there now. It does seem to be permission error when the backend server tries to create tmp file. I can't run singularity with --writable, I think it requires sudo which I don't have.

singularity shell --bind /tmp:/vg/tmp --pwd /vg docker://momigteam/momig-backend
INFO:    Using cached SIF image
Singularity> bash /.singularity.d/runscript
[2021-04-19T14:43:50Z INFO  graph_genome_browser_backend::vg] VG Version: v1.6.0-213-gc0c19fe

[2021-04-19T14:43:50Z INFO  graph_genome_browser_backend::vg] VG minor version: 6
[2021-04-19T14:43:50Z INFO  graph_genome_browser_backend::features] Parsing:  "static/gencode.v27.basic.annotation.gff3"
[2021-04-19T14:44:22Z DEBUG graph_genome_browser_backend::features] 1641293 lines processed. end.
6
Start server on 0.0.0.0:8081
[2021-04-19T14:44:29Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=features 302 Found (0.115034 ms)
[2021-04-19T14:44:29Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=chromosomes 302 Found (0.047559 ms)
[2021-04-19T14:44:29Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=metadata 200 OK (0.050635 ms)
[2021-04-19T14:44:29Z INFO  graph_genome_browser_backend::handlers] Range: chr12:80851974-80853202
[2021-04-19T14:44:29Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr12:80851974-80853202.json, ./tmp/raw_chr12:80851974-80853202.json
[2021-04-19T14:44:29Z DEBUG graph_genome_browser_backend::vg] Create a temporary file: "./tmp/raw_chr12:80851974-80853202.json"
[2021-04-19T14:44:29Z INFO  graph_genome_browser_backend::vg] static/chm1_chr21.xg, chr12:80851974-80853202
[2021-04-19T14:44:29Z INFO  graph_genome_browser_backend::vg] VG version: 6, cmd: ["vg"]
[2021-04-19T14:44:29Z DEBUG graph_genome_browser_backend::vg] ["chunk", "-t", "4", "-x", "static/chm1_chr21.xg", "-p", "chr12:80851974-80853202", "-c", "3", "-g", "-A", "-a", "static/t.gam.index", "-b", "./tmp/raw_chr12:80851974-80853202.json"]
error:[vg chunk] unable to load xg index file
[2021-04-19T14:44:29Z INFO  logger] GET http://localhost:3000/api/v2/static/GRCh.json 200 OK (0.295572 ms)
[2021-04-19T14:44:29Z INFO  logger] GET http://localhost:3000/api/v2/static/chm1_chr21.pcf 200 OK (0.330897 ms)
[2021-04-19T14:44:29Z DEBUG graph_genome_browser_backend::vg] Status codes: exit code: 1 exit code: 0 exit code: 0
[2021-04-19T14:44:29Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr12:80851974-80853202 500 Internal Server Error (194.345919 ms)
[2021-04-19T14:44:41Z INFO  graph_genome_browser_backend::handlers] Range: chr21:15811802-15812802
[2021-04-19T14:44:41Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr21:15811802-15812802.json, ./tmp/raw_chr21:15811802-15812802.json
[2021-04-19T14:44:41Z DEBUG graph_genome_browser_backend::vg] Create a temporary file: "./tmp/raw_chr21:15811802-15812802.json"
[2021-04-19T14:44:41Z INFO  graph_genome_browser_backend::vg] static/chm1_chr21.xg, chr21:15811802-15812802
[2021-04-19T14:44:41Z INFO  graph_genome_browser_backend::vg] VG version: 6, cmd: ["vg"]
[2021-04-19T14:44:41Z DEBUG graph_genome_browser_backend::vg] ["chunk", "-t", "4", "-x", "static/chm1_chr21.xg", "-p", "chr21:15811802-15812802", "-c", "3", "-g", "-A", "-a", "static/t.gam.index", "-b", "./tmp/raw_chr21:15811802-15812802.json"]
error:[vg chunk] unable to load xg index file

olechnwin · 2021-04-19T19:14:27Z

Sorry for the different scenarios back and forth with CHM1 demo and my own data. I hope it's not too confusing.
But, I was able to get the SequenceTube displayed by binding /vg/tmp to /tmp in the host:

singularity shell --bind ~/Data/A673_pacbio/HiFi/vcf:/vg/static --bind /tmp:/vg/tmp --pwd /vg docker://momigteam/momig-backend:prod

I'm very happy to make such a progress. Unfortunately, for some entries, the SequenceTube still is not displayed. It's no easy to see from the log, since the actual SV that I clicked was not visible in the log.

For example, these are the SVs that I clicked in the order shown below:

● chr11 128,667,505 - ● chr22 29,684,301 - 1 ● TRA [shows graph]
● chr5 138,560,805 + ● chr5 154,275,288 - 15714483 ● INV [**no graph**]
● chr5 17,517,208 + ● chr5 17,530,659 - 13451 ● INV [shows graph]
● chr10 38,560,670 - ● chr10 42,891,750 - 4331080 ● DUP [**no graph**]
● chr10 42,385,519 + ● chr10 42,597,744 + 212225 ● DEL [shows graph]

These are the messages shown on the backend server as I clicked on the SVs above:

singularity shell --bind ~/Data/A673_pacbio/HiFi/vcf:/vg/static --bind /tmp:/vg/tmp --pwd /vg docker://momigteam/momig-backend:prod
INFO:    Using cached SIF image
Singularity> bash /.singularity.d/runscript
[2021-04-19T18:53:20Z INFO  graph_genome_browser_backend::vg] VG Version: vg version v1.25.0 "Apice"
Compiled with g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 on Linux
Linked against libstd++ 20191114
Built by root@7032b439058e

[2021-04-19T18:53:20Z INFO  graph_genome_browser_backend::vg] VG minor version: 25
[2021-04-19T18:53:20Z INFO  graph_genome_browser_backend::features] Parsing:  "static/gencode.v37lift37.basic.annotation.gff3"
[2021-04-19T18:54:01Z DEBUG graph_genome_browser_backend::features] 1844760 lines processed. end.
25
Start server on 0.0.0.0:8081
[2021-04-19T18:54:13Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=features 302 Found (0.129783 ms)
[2021-04-19T18:54:13Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=chromosomes 302 Found (0.04285 ms)
[2021-04-19T18:54:13Z INFO  logger] GET http://localhost:3000/api/v2/overview?source=metadata 200 OK (0.060122 ms)
[2021-04-19T18:54:14Z INFO  graph_genome_browser_backend::handlers] Range: chr12:80851974-80853202
[2021-04-19T18:54:14Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr12:80851974-80853202.json, ./tmp/raw_chr12:80851974-80853202.json
[2021-04-19T18:54:14Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr12:80851974-80853202 302 Found (0.405196 ms)
[2021-04-19T18:54:14Z INFO  logger] GET http://localhost:3000/api/v2/static/momig_input.pcf 200 OK (0.136274 ms)
[2021-04-19T18:54:14Z INFO  logger] GET http://localhost:3000/api/v2/static/GRCh.json 200 OK (0.122088 ms)
[2021-04-19T18:54:14Z INFO  logger] GET http://localhost:3000/api/v2/cache/raw_chr12:80851974-80853202.json 200 OK (0.057838 ms)
[2021-04-19T18:55:07Z INFO  graph_genome_browser_backend::handlers] Range: chr11:128667505-128668505
[2021-04-19T18:55:07Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr11:128667505-128668505.json, ./tmp/raw_chr11:128667505-128668505.json
[2021-04-19T18:55:07Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr11:128667505-128668505 302 Found (0.26806 ms)
[2021-04-19T18:55:08Z INFO  logger] GET http://localhost:3000/api/v2/cache/raw_chr11:128667505-128668505.json 200 OK (0.051436 ms)
[2021-04-19T18:55:08Z INFO  logger] GET http://localhost:3000/api/v2/region?format=wig&multiple=true&path=chr11:128665551-128666523,chr11:128666523-128667495,chr11:128667495-128667504,chr11:128667504-128668519,chr11:128668519-128669534,chr11:128669534-128670549,chr11:128670549-128671564 200 OK (0.699446 ms)
[2021-04-19T18:55:08Z INFO  logger] GET http://localhost:3000/api/v2/static/pack.json 404 Not Found (0.035856 ms)
[2021-04-19T18:55:08Z ERROR iron::iron] Error handling:
Request {
    url: Url { generic_url: "http://localhost:3000/api/v2/static/pack.json" }
    method: Get
    remote_addr: V4(127.0.0.1:58084)
    local_addr: V4(0.0.0.0:8081)
}
Error was: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2021-04-19T18:55:08Z INFO  logger] GET http://localhost:3000/api/v2/region?format=bed&path=chr11:128665551-128671564 200 OK (0.271006 ms)
[2021-04-19T19:01:28Z INFO  graph_genome_browser_backend::handlers] Range: chr5:17517208-17530659
[2021-04-19T19:01:28Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr5:17517208-17530659.json, ./tmp/raw_chr5:17517208-17530659.json
[2021-04-19T19:01:28Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr5:17517208-17530659 302 Found (0.373768 ms)
[2021-04-19T19:01:28Z INFO  logger] GET http://localhost:3000/api/v2/cache/raw_chr5:17517208-17530659.json 200 OK (0.044954 ms)
[2021-04-19T19:01:28Z INFO  logger] GET http://localhost:3000/api/v2/static/pack.json 404 Not Found (0.125845 ms)
[2021-04-19T19:01:28Z ERROR iron::iron] Error handling:
Request {
    url: Url { generic_url: "http://localhost:3000/api/v2/static/pack.json" }
    method: Get
    remote_addr: V4(127.0.0.1:60476)
    local_addr: V4(0.0.0.0:8081)
}
Error was: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2021-04-19T19:01:28Z INFO  logger] GET http://localhost:3000/api/v2/region?format=bed&path=chr5:17515179-17534726 200 OK (0.358329 ms)
[2021-04-19T19:01:28Z INFO  logger] GET http://localhost:3000/api/v2/region?format=wig&multiple=true&path=chr5:17515179-17516191,chr5:17516191-17517203,chr5:17517203-17517207,chr5:17517207-17518167,chr5:17518167-17519127,chr5:17519127-17520087,chr5:17520087-17521047,chr5:17521047-17522007,chr5:17522007-17522967,chr5:17522967-17523927,chr5:17523927-17524887,chr5:17524887-17525847,chr5:17525847-17526807,chr5:17526807-17527767,chr5:17527767-17528727,chr5:17528727-17529687,chr5:17529687-17530647,chr5:17530647-17530658,chr5:17530658-17531675,chr5:17531675-17532692,chr5:17532692-17533709,chr5:17533709-17534726 200 OK (2.348294 ms)
[2021-04-19T19:02:48Z INFO  graph_genome_browser_backend::handlers] Range: chr10:42385519-42597744
[2021-04-19T19:02:48Z DEBUG graph_genome_browser_backend::handlers] Redirect URL: http://localhost:3000/api/v2/cache/raw_chr10:42385519-42597744.json, ./tmp/raw_chr10:42385519-42597744.json
[2021-04-19T19:02:48Z INFO  logger] GET http://localhost:3000/api/v2/graph?raw=true&cache=true&gam=true&steps=3&path=chr10:42385519-42597744 302 Found (0.268452 ms)
[2021-04-19T19:02:48Z INFO  logger] GET http://localhost:3000/api/v2/cache/raw_chr10:42385519-42597744.json 200 OK (0.047318 ms)
[2021-04-19T19:02:58Z INFO  logger] GET http://localhost:3000/api/v2/static/pack.json 404 Not Found (0.031228 ms)
[2021-04-19T19:02:58Z ERROR iron::iron] Error handling:
Request {
    url: Url { generic_url: "http://localhost:3000/api/v2/static/pack.json" }
    method: Get
    remote_addr: V4(127.0.0.1:32804)
    local_addr: V4(0.0.0.0:8081)
}
Error was: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2021-04-19T19:02:58Z INFO  logger] GET http://localhost:3000/api/v2/region?format=bed&path=chr10:42384907-42599262 200 OK (0.220541 ms)

For those that doesn't display a graph, the previously displayed graph was just on the web browser nothing else is changed.

6br · 2021-04-20T03:56:28Z

For CHM1 demo, I am wondering why vg binary cannot find xg file. Now I changed the path static/chm1_chr21.xg to the absolute path /vg/static/chm1_chr21.xg.

6br · 2021-04-20T05:08:17Z

Actually, when the interval to visualize is longer than 1 Mbp, retrieving chunks by vg becomes slow. Thus, MoMI-G backend restricts the maximum interval to visualize. If users wish to see the large variants, then push "left-align" or "right-align" button on the interval card deck. The variants can be visualized by retrieving subgraphs from the either of breakends.

The maximum length of the specified genomic range is limited by the interval option. The default option is /vg/graph-genome-browser-backend --config=/vg/static/config.yaml --interval=1500000 --http=0.0.0.0:8081", "--api=/api/v2/. For example, users can change the interval option like this.

singularity exec --bind ~/Data/A673_pacbio/HiFi/vcf:/vg/static --bind /tmp:/vg/tmp --pwd /vg docker://momigteam/momig-backend:prod /vg/graph-genome-browser-backend --config=/vg/static/config.yaml --interval=2000000 --http=0.0.0.0:8081", "--api=/api/v2/

Still, I wouldn't recommend to set much longer interval because it causes the performance problem.

olechnwin · 2021-04-20T18:09:30Z

Thanks for the screen shot. That is a great option that I didn't realize before.
Thanks for the settings in singularity. That'll be useful to know.

I tried the CHM1 demo again but still have the same "unable to load xg index file". I'm not sure why. I thought the absolute path will fix it.

[2021-04-20T18:06:13Z DEBUG graph_genome_browser_backend::vg] Create a temporary file: "./tmp/raw_chr21:18607192-18608192.json"
[2021-04-20T18:06:13Z INFO  graph_genome_browser_backend::vg] /vg/static/chm1_chr21.xg, chr21:18607192-18608192
[2021-04-20T18:06:13Z INFO  graph_genome_browser_backend::vg] VG version: 6, cmd: ["vg"]
[2021-04-20T18:06:13Z DEBUG graph_genome_browser_backend::vg] ["chunk", "-t", "4", "-x", "/vg/static/chm1_chr21.xg", "-p", "chr21:18607192-18608192", "-c", "3", "-g", "-A", "-a", "/vg/static/t.gam.index", "-b", "./tmp/raw_chr21:18607192-18608192.json"]
error:[vg chunk] unable to load xg index file
[2021-04-20T18:06:13Z DEBUG graph_genome_browser_backend::vg] Status codes: exit code: 1 exit code: 0 exit code: 0

Since I was able to display my own data in MoMI-G, I'm not going to worry about the CHM1 data anymore. Unless you want me to still trouble shoot it.

6br · 2021-04-21T02:30:27Z

I am glad to hear that you can visualize on your data in MoMI-G. Thank you for using MoMI-G and helping us debug MoMI-G. When you have any question, feel free to ask.

olechnwin · 2021-04-21T12:31:02Z

@6br
Thank you so much for all your help. MoMI-G is a great visualization tool. I'm very happy to get it to work!
Thank you for your patience and continued support.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error 'cannot get position' when running bam2gam.sh #72

error 'cannot get position' when running bam2gam.sh #72

olechnwin commented Mar 30, 2021

6br commented Mar 31, 2021 •

edited

olechnwin commented Mar 31, 2021

6br commented Apr 1, 2021

olechnwin commented Apr 1, 2021

6br commented Apr 2, 2021

olechnwin commented Apr 2, 2021

6br commented Apr 2, 2021

olechnwin commented Apr 5, 2021

6br commented Apr 6, 2021

olechnwin commented Apr 7, 2021

6br commented Apr 8, 2021 •

edited

olechnwin commented Apr 8, 2021

olechnwin commented Apr 9, 2021 •

edited

6br commented Apr 10, 2021 •

edited

6br commented Apr 10, 2021 •

edited

olechnwin commented Apr 12, 2021 •

edited

olechnwin commented Apr 12, 2021 •

edited

olechnwin commented Apr 12, 2021

6br commented Apr 13, 2021 •

edited

olechnwin commented Apr 13, 2021 •

edited

olechnwin commented Apr 13, 2021

6br commented Apr 14, 2021

olechnwin commented Apr 16, 2021

6br commented Apr 17, 2021

6br commented Apr 17, 2021 •

edited

olechnwin commented Apr 19, 2021 •

edited

olechnwin commented Apr 19, 2021 •

edited

olechnwin commented Apr 19, 2021 •

edited

6br commented Apr 20, 2021 •

edited

6br commented Apr 20, 2021 •

edited

olechnwin commented Apr 20, 2021 •

edited

6br commented Apr 21, 2021

olechnwin commented Apr 21, 2021

error 'cannot get position' when running bam2gam.sh #72

error 'cannot get position' when running bam2gam.sh #72

Comments

olechnwin commented Mar 30, 2021

6br commented Mar 31, 2021 • edited

olechnwin commented Mar 31, 2021

6br commented Apr 1, 2021

olechnwin commented Apr 1, 2021

6br commented Apr 2, 2021

olechnwin commented Apr 2, 2021

6br commented Apr 2, 2021

olechnwin commented Apr 5, 2021

6br commented Apr 6, 2021

olechnwin commented Apr 7, 2021

6br commented Apr 8, 2021 • edited

olechnwin commented Apr 8, 2021

olechnwin commented Apr 9, 2021 • edited

6br commented Apr 10, 2021 • edited

6br commented Apr 10, 2021 • edited

olechnwin commented Apr 12, 2021 • edited

olechnwin commented Apr 12, 2021 • edited

olechnwin commented Apr 12, 2021

6br commented Apr 13, 2021 • edited

olechnwin commented Apr 13, 2021 • edited

olechnwin commented Apr 13, 2021

6br commented Apr 14, 2021

olechnwin commented Apr 16, 2021

6br commented Apr 17, 2021

6br commented Apr 17, 2021 • edited

olechnwin commented Apr 19, 2021 • edited

olechnwin commented Apr 19, 2021 • edited

olechnwin commented Apr 19, 2021 • edited

6br commented Apr 20, 2021 • edited

6br commented Apr 20, 2021 • edited

olechnwin commented Apr 20, 2021 • edited

6br commented Apr 21, 2021

olechnwin commented Apr 21, 2021

6br commented Mar 31, 2021 •

edited

6br commented Apr 8, 2021 •

edited

olechnwin commented Apr 9, 2021 •

edited

6br commented Apr 10, 2021 •

edited

6br commented Apr 10, 2021 •

edited

olechnwin commented Apr 12, 2021 •

edited

olechnwin commented Apr 12, 2021 •

edited

6br commented Apr 13, 2021 •

edited

olechnwin commented Apr 13, 2021 •

edited

6br commented Apr 17, 2021 •

edited

olechnwin commented Apr 19, 2021 •

edited

olechnwin commented Apr 19, 2021 •

edited

olechnwin commented Apr 19, 2021 •

edited

6br commented Apr 20, 2021 •

edited

6br commented Apr 20, 2021 •

edited

olechnwin commented Apr 20, 2021 •

edited