@mlinderm Would adding Scala 2.12 support be useful? Spark 2.4.3 supports Scala 2.12 but getting things running on the binary distribution is a nightmare.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/deca-prb/32/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/deca-prb/33/
Test PASSed.

@heuermh I am observing a substantial performance degradation with the new version. On a 16 core workstation, the time for calling CNVs in 2535 samples went from 9m17s with the old version, to 12m22s for the new version (with ADAM 0.27 and Spark 2.4.3). The difference seems to be in PCA/SVD step. Do you have any guesses as to what might have changed between the old spark/ADAM version the current version?

The difference seems to be in PCA/SVD step. Do you have any guesses as to what might have changed between the old spark/ADAM version the current version?

There are a lot of code changes between 0.24.0 and 0.27.0 in ADAM, but not much that would affect numerical method performance

bigdatagenomics/adam@maint_spark2_2.11-0.24.0...maint_spark2_2.11-0.27.0

I imagine differences between Spark 2.1.x and 2.4.3 might be more significant though. Is there a smaller benchmark we could use to reproduce what you are seeing, say with fewer samples?

There are datasets ranging from 500 samples to 2535 samples on the AMP BDG cluster at /user/mlinderman/deca/DATA.<samples>.RD.txt (if that is not easy to obtain, I can post input files for you). The former should only run for a few minutes or less. On the workstation the old version with Spark 2.1.0 took 1m34s for 500 samples, the new version took 2m17s. I can start working through different Spark versions to see if I observe a change.

There are datasets ranging from 500 samples to 2535 samples on the AMP BDG cluster

Great, thanks. I'll also take a look tomorrow.

I tried several spark distributions with the original code base (older ADAM). The performance degradation seems to occur between 2.2.3 and 2.3.3, that is 2.2.3 was 1m38s while 2.3.3 was 2m13s for 500 samples. One guess was that it was an issue with an upgrade to Breeze, but changing the Breeze dependency to 0.13.2 with Spark 2.4.3 did not improve performance.