Bump htsjdk dependency version to 2.20.3 #2195

heuermh · 2019-08-10T17:21:23Z

Expected to fail CI.

heuermh · 2019-08-10T17:24:54Z

ADAM version 0.28.0 with htsjdk 2.19.0

$ adam-shell

scala> import org.bdgenomics.adam.util.ADAMShell._
import org.bdgenomics.adam.util.ADAMShell._

scala> import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.rdd.ADAMContext._

scala> val reads = sc.loadAlignments("adam-core/src/test/resources/small.sam")
reads: org.bdgenomics.adam.rdd.read.AlignmentRecordDataset = RDDBoundAlignmentRecordDataset with 2 reference sequences, 0 read groups, and 2 processing steps

scala> printAlignmentAttributes(reads, Seq(), 200)

Alignment Attributes
+----------------+-----------+-----------+---------------------------+--------+------------+
| Reference Name |   Start   |    End    |         Read Name         | Sample | Read Group |
+----------------+-----------+-----------+---------------------------+--------+------------+
|              1 |  26472783 |  26472858 |  simread:1:26472783:false |        |            |
|              1 | 240997787 | 240997862 |  simread:1:240997787:true |        |            |
|              1 | 189606653 | 189606728 |  simread:1:189606653:true |        |            |
|              1 | 207027738 | 207027813 |  simread:1:207027738:true |        |            |
|              1 |  14397233 |  14397308 |  simread:1:14397233:false |        |            |
|              1 | 240344442 | 240344517 |  simread:1:240344442:true |        |            |
|              1 | 153978724 | 153978799 | simread:1:153978724:false |        |            |
|              1 | 237728409 | 237728484 |  simread:1:237728409:true |        |            |
|              1 | 231911906 | 231911981 | simread:1:231911906:false |        |            |
|              1 |  50683371 |  50683446 |  simread:1:50683371:false |        |            |
|              1 |  37577445 |  37577520 |  simread:1:37577445:false |        |            |
|              1 | 195211965 | 195212040 | simread:1:195211965:false |        |            |
|              1 | 163841413 | 163841488 | simread:1:163841413:false |        |            |
|              1 | 101556378 | 101556453 | simread:1:101556378:false |        |            |
|              1 |  20101800 |  20101875 |   simread:1:20101800:true |        |            |
|              1 | 186794283 | 186794358 |  simread:1:186794283:true |        |            |
|              1 | 165341382 | 165341457 |  simread:1:165341382:true |        |            |
|              1 |   5469106 |   5469181 |    simread:1:5469106:true |        |            |
|              1 |  89554252 |  89554327 |  simread:1:89554252:false |        |            |
|              1 | 169801933 | 169802008 |  simread:1:169801933:true |        |            |
+----------------+-----------+-----------+---------------------------+--------+------------+

This pull request with htsjdk 2.20.0

$ adam-shell

scala> import org.bdgenomics.adam.util.ADAMShell._
import org.bdgenomics.adam.util.ADAMShell._

scala> import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.rdd.ADAMContext._

scala> val reads = sc.loadAlignments("adam-core/src/test/resources/small.sam")
reads: org.bdgenomics.adam.rdd.read.AlignmentRecordDataset = RDDBoundAlignmentRecordDataset with 2 reference sequences, 0 read groups, and 2 processing steps

scala> printAlignmentAttributes(reads, Seq(), 200)

Alignment Attributes
+----------------+-----------+-----------+---------------------------+--------+------------+
| Reference Name |   Start   |    End    |         Read Name         | Sample | Read Group |
+----------------+-----------+-----------+---------------------------+--------+------------+
|              1 |  26472783 |  26472858 |             6472783:false |        |            |
|              1 | 240997787 | 240997862 |  simread:1:240997787:true |        |            |
|              1 | 189606653 | 189606728 |  simread:1:189606653:true |        |            |
|              1 | 207027738 | 207027813 |  simread:1:207027738:true |        |            |
|              1 |  14397233 |  14397308 |  simread:1:14397233:false |        |            |
|              1 | 240344442 | 240344517 |  simread:1:240344442:true |        |            |
|              1 | 153978724 | 153978799 | simread:1:153978724:false |        |            |
|              1 | 237728409 | 237728484 |  simread:1:237728409:true |        |            |
|              1 | 231911906 | 231911981 | simread:1:231911906:false |        |            |
|              1 |  50683371 |  50683446 |  simread:1:50683371:false |        |            |
|              1 |  37577445 |  37577520 |  simread:1:37577445:false |        |            |
|              1 | 195211965 | 195212040 | simread:1:195211965:false |        |            |
|              1 | 163841413 | 163841488 | simread:1:163841413:false |        |            |
|              1 | 101556378 | 101556453 | simread:1:101556378:false |        |            |
|              1 |  20101800 |  20101875 |   simread:1:20101800:true |        |            |
|              1 | 186794283 | 186794358 |  simread:1:186794283:true |        |            |
|              1 | 165341382 | 165341457 |  simread:1:165341382:true |        |            |
|              1 |   5469106 |   5469181 |    simread:1:5469106:true |        |            |
|              1 |  89554252 |  89554327 |  simread:1:89554252:false |        |            |
|              1 | 169801933 | 169802008 |  simread:1:169801933:true |        |            |
+----------------+-----------+-----------+---------------------------+--------+------------+

For some reason, the name of the first read has been clipped from simread:1:26472783:false to 6472783:false.

AmplabJenkins · 2019-08-10T17:34:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3039/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2195/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains f20a198 # timeout=10Checking out Revision f20a198 (origin/pr/2195/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f f20a19859fbfbd38e2ff8549054f83a999f5a401First time build. Skipping changelog.Triggering ADAM-prb ? 2.7.5,2.12,2.4.3,ubuntuTriggering ADAM-prb ? 2.7.5,2.11,2.4.3,ubuntuADAM-prb ? 2.7.5,2.12,2.4.3,ubuntu completed with result FAILUREADAM-prb ? 2.7.5,2.11,2.4.3,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2019-08-10T18:06:30Z

Same with htsjdk version 2.20.1.

AmplabJenkins · 2019-08-10T19:33:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3042/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2195/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 70fa095 # timeout=10Checking out Revision 70fa095 (origin/pr/2195/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 70fa095272d8c033cfbc4f9d55466c46baa60179First time build. Skipping changelog.Triggering ADAM-prb ? 2.7.5,2.12,2.4.3,ubuntuTriggering ADAM-prb ? 2.7.5,2.11,2.4.3,ubuntuADAM-prb ? 2.7.5,2.12,2.4.3,ubuntu completed with result FAILUREADAM-prb ? 2.7.5,2.11,2.4.3,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2019-08-30T23:09:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3045/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2195/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains f52fc1e # timeout=10Checking out Revision f52fc1e (origin/pr/2195/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f f52fc1e4de476c8d9a8ff314b8338f2045f5107cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.7.5,2.12,2.4.3,ubuntuTriggering ADAM-prb ? 2.7.5,2.11,2.4.3,ubuntuADAM-prb ? 2.7.5,2.12,2.4.3,ubuntu completed with result FAILUREADAM-prb ? 2.7.5,2.11,2.4.3,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2019-09-03T17:22:48Z

Another test failure, it appears htsjdk 2.20.3 cannot read this test file incorrectly

@SQ     SN:1    LN:249250621                                                                                                                                                                                                                                         
read:0  0       1       1       60      75M     *       0       0       GTATAAGAGCAGCCTTATTCCTATTTATAATCAGGGTGAAACACCTGTGCCAATGCCAAGACAGGGGTGCCAAGA     *       NM:i:0  AS:i:75 XS:i:0                                                                               
read:4  4       *       0       0       *       *       0       0       GTATAAGAGCAGCCTTATTCCTATTTATAATCAGGGTGAAACACCTGTGCCAATGCCAAGACAGGGGTGCCAAGA     *       NM:i:0  AS:i:75 XS:i:0 
...

htsjdk.samtools.SAMFormatException: Error parsing text SAM file. RNAME '75M' not found in any SQ record; Line 3
Line: 1	60	75M	*	0	0	GTATAAGAGCAGCCTTATTCCTATTTATAATCAGGGTGAAACACCTGTGCCAATGCCAAGACAGGGGTGCCAAGA	*	NM:i:0	AS:i:75	XS:i:0
	at htsjdk.samtools.SAMLineParser.reportErrorParsingLine(SAMLineParser.java:457)
	at htsjdk.samtools.SAMLineParser.validateReferenceName(SAMLineParser.java:199)
	at htsjdk.samtools.SAMLineParser.parseLine(SAMLineParser.java:255)
	at htsjdk.samtools.SAMTextReader$RecordIterator.parseLine(SAMTextReader.java:268)
	at htsjdk.samtools.SAMTextReader$RecordIterator.next(SAMTextReader.java:255)
	at htsjdk.samtools.SAMTextReader$RecordIterator.next(SAMTextReader.java:228)
	at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:574)
	at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:553)
	at org.seqdoop.hadoop_bam.SAMRecordReader.nextKeyValue(SAMRecordReader.java:175)
	at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:230)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
	at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
	at scala.collection.AbstractIterator.to(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
	at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

AmplabJenkins · 2019-09-17T15:48:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/3056/

Build result: FAILURE

[...truncated 3 lines...]Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prbWiping out workspace first.Cloning the remote Git repositoryCloning repository https://github.com/bigdatagenomics/adam.git > git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > git rev-parse origin/pr/2195/merge^{commit} # timeout=10 > git branch -a -v --no-abbrev --contains 9472dd0 # timeout=10Checking out Revision 9472dd0 (origin/pr/2195/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 9472dd08b735e42dd7e03ebd391021fcdb7bbbdaFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.7.5,2.11,2.4.4,ubuntuTriggering ADAM-prb ? 2.7.5,2.12,2.4.4,ubuntuADAM-prb ? 2.7.5,2.11,2.4.4,ubuntu completed with result FAILUREADAM-prb ? 2.7.5,2.12,2.4.4,ubuntu completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2021-02-23T17:59:38Z

I am afraid the only resolution to this would be #2111, which involves a lot of code change and possible performance implications. Hadoop-BAM is effectively no longer maintained.

heuermh mentioned this pull request Aug 10, 2019

Changes required to work with latest htsjdk disq-bio/disq#107

Merged

heuermh changed the title ~~Bump htsjdk dependency version to 2.20.0~~ Bump htsjdk dependency version to 2.20.1 Aug 10, 2019

heuermh force-pushed the htsjdk-2.20.0 branch from d9457ea to 500b338 Compare August 10, 2019 19:22

heuermh force-pushed the htsjdk-2.20.0 branch from 500b338 to 6747315 Compare August 30, 2019 22:59

heuermh changed the title ~~Bump htsjdk dependency version to 2.20.1~~ Bump htsjdk dependency version to 2.20.3 Aug 30, 2019

heuermh added this to the 0.29.0 milestone Sep 11, 2019

This was referenced Sep 11, 2019

add tests for SAM files that corrupt with adam + htsjdk 2.20.x disq-bio/disq#120

Merged

add tests for SAM files that corrupt with adam + htsjdk 2.20.x HadoopGenomics/Hadoop-BAM#203

Open

heuermh modified the milestones: 0.29.0, 0.30.0 Sep 18, 2019

heuermh modified the milestones: 0.30.0, 0.31.0 Dec 3, 2019

heuermh modified the milestones: 0.31.0, 0.32.0 Feb 25, 2020

heuermh modified the milestones: 0.32.0, 1.0.0 Jul 13, 2020

heuermh force-pushed the htsjdk-2.20.0 branch from 22bc8f9 to a895649 Compare February 23, 2021 17:58

Update htsjdk dependency version to 2.24.1.

18950f7

heuermh force-pushed the htsjdk-2.20.0 branch from a895649 to 18950f7 Compare October 14, 2021 15:10

heuermh mentioned this pull request May 4, 2023

Bump htsjdk from 2.19.0 to 3.0.1 #2381

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump htsjdk dependency version to 2.20.3 #2195

Bump htsjdk dependency version to 2.20.3 #2195

heuermh commented Aug 10, 2019 •

edited

heuermh commented Aug 10, 2019

AmplabJenkins commented Aug 10, 2019

heuermh commented Aug 10, 2019

AmplabJenkins commented Aug 10, 2019

AmplabJenkins commented Aug 30, 2019

heuermh commented Sep 3, 2019

AmplabJenkins commented Sep 17, 2019

heuermh commented Feb 23, 2021

Bump htsjdk dependency version to 2.20.3 #2195

Are you sure you want to change the base?

Bump htsjdk dependency version to 2.20.3 #2195

Conversation

heuermh commented Aug 10, 2019 • edited

heuermh commented Aug 10, 2019

AmplabJenkins commented Aug 10, 2019

Build result: FAILURE

heuermh commented Aug 10, 2019

AmplabJenkins commented Aug 10, 2019

Build result: FAILURE

AmplabJenkins commented Aug 30, 2019

Build result: FAILURE

heuermh commented Sep 3, 2019

AmplabJenkins commented Sep 17, 2019

Build result: FAILURE

heuermh commented Feb 23, 2021

heuermh commented Aug 10, 2019 •

edited