Assert error during graph cleaning

[lucaptra]

I’m getting a puzzling “assert error” when using the “clean” function to first merge then clean multiple sample graphs and am looking for help with troubleshooting. For context, the inputs are 8 raw graphs (~60GB), run on a 1.5TB machine with 1.4TB allocated to the computation.

From the log file it doesn’t look to my naive eyes like McCortex doesn’t have sufficient memory. Can the issue be that the hash table is not large enough? Any tips are greatly appreciated!

Here is the log file:
[12 Jan 2019 00:36:15-jUF][cmd] /home/ltran20/mccortex/bin/mccortex63 clean -f -m 1400G -n 13G -t 16 --fallback 8 -T -U --covg-before /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.raw.cov.csv --covg-after /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.clean.cov.csv --len-before /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.raw.len.csv --len-after /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.clean.len.csv -o /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.clean.ctx /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR13_S4.raw.ctx /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR15_S5.raw.ctx /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR16_S6.raw.ctx /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR17_S7.raw.ctx /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR19_S8.raw.ctx /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR6_S3.raw.ctx /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/S_JAR50_S9.raw.ctx /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/S_JAR53_S13.raw.ctx
[12 Jan 2019 00:36:15-jUF][cwd] /home/ltran20/lizards
[12 Jan 2019 00:36:15-jUF][version] mccortex= zlib=1.2.3 htslib=1.8-17-g699ed53 ASSERTS=ON hash=Lookup3 CHECKS=ON k=33..63
[12 Jan 2019 00:36:15-jUF] Actions:
[12 Jan 2019 00:36:15-jUF] 0. Saving kmer coverage distribution to: /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.raw.cov.csv
[12 Jan 2019 00:36:15-jUF] 1. Saving unitig length distribution to: /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.raw.len.csv
[12 Jan 2019 00:36:15-jUF] 2. Cleaning tips shorter than 126 nodes
[12 Jan 2019 00:36:15-jUF] 3. Cleaning unitigs with auto-detected threshold
[12 Jan 2019 00:36:15-jUF] 4. Saving kmer coverage distribution to: /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.clean.cov.csv
[12 Jan 2019 00:36:15-jUF] 5. Saving unitig length distribution to: /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR.pop.clean.len.csv
[12 Jan 2019 00:36:15-jUF][memory] 448 bits per kmer
[12 Jan 2019 00:36:15-jUF] Note: Using less memory than requested (729GB < 1.4TB); allows for 13,958,643,712 kmers
[12 Jan 2019 00:36:15-jUF][memory] graph: 729GB
[12 Jan 2019 00:36:15-jUF][cleaning] 8 input graphs, max kmers: 3,140,896,287, using 8 colours
[12 Jan 2019 00:36:15-jUF][memory] total: 729GB of 1.5TB RAM
[12 Jan 2019 00:36:15-jUF][hasht] Allocating table with 13,958,643,712 entries, using 209GB
[12 Jan 2019 00:36:15-jUF][hasht] number of buckets: 536,870,912, bucket size: 26
[12 Jan 2019 00:36:15-jUF][graph] kmer-size: 63; colours: 8; capacity: 13,958,643,712
[12 Jan 2019 00:36:15-jUF][FileFilter] Reading file /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR13_S4.raw.ctx [1 src colour]
[12 Jan 2019 00:36:15-jUF][GReader] 2,958,587,869 kmers, 57.9GB filesize
[12 Jan 2019 01:47:14-jUF][GReader] Loaded 2,958,587,869 / 2,958,587,869 (100.00%) of kmers parsed
[12 Jan 2019 01:47:14-jUF][FileFilter] Reading file /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR15_S5.raw.ctx [1 src colour] with filter: 0->1
[12 Jan 2019 01:47:14-jUF][GReader] 2,952,133,198 kmers, 57.7GB filesize
[12 Jan 2019 02:51:21-jUF][GReader] Loaded 2,952,133,198 / 2,952,133,198 (100.00%) of kmers parsed
[12 Jan 2019 02:51:21-jUF][FileFilter] Reading file /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR16_S6.raw.ctx [1 src colour] with filter: 0->2
[12 Jan 2019 02:51:21-jUF][GReader] 2,919,980,595 kmers, 57.1GB filesize
[12 Jan 2019 03:56:29-jUF][GReader] Loaded 2,919,980,595 / 2,919,980,595 (100.00%) of kmers parsed
[12 Jan 2019 03:56:29-jUF][FileFilter] Reading file /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR17_S7.raw.ctx [1 src colour] with filter: 0->3
[12 Jan 2019 03:56:29-jUF][GReader] 2,886,256,646 kmers, 56.4GB filesize
[12 Jan 2019 05:07:03-jUF][GReader] Loaded 2,886,256,646 / 2,886,256,646 (100.00%) of kmers parsed
[12 Jan 2019 05:07:03-jUF][FileFilter] Reading file /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR19_S8.raw.ctx [1 src colour] with filter: 0->4
[12 Jan 2019 05:07:03-jUF][GReader] 2,958,893,598 kmers, 57.9GB filesize
[12 Jan 2019 06:23:01-jUF][GReader] Loaded 2,958,893,598 / 2,958,893,598 (100.00%) of kmers parsed
[12 Jan 2019 06:23:01-jUF][FileFilter] Reading file /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/JAR6_S3.raw.ctx [1 src colour] with filter: 0->5
[12 Jan 2019 06:23:01-jUF][GReader] 2,886,705,907 kmers, 56.5GB filesize
[12 Jan 2019 07:40:05-jUF][GReader] Loaded 2,886,705,907 / 2,886,705,907 (100.00%) of kmers parsed
[12 Jan 2019 07:40:05-jUF][FileFilter] Reading file /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/S_JAR50_S9.raw.ctx [1 src colour] with filter: 0->6
[12 Jan 2019 07:40:05-jUF][GReader] 3,140,896,287 kmers, 61.4GB filesize
[12 Jan 2019 09:05:35-jUF][GReader] Loaded 3,140,896,287 / 3,140,896,287 (100.00%) of kmers parsed
[12 Jan 2019 09:05:35-jUF][FileFilter] Reading file /scratch/ltran20/lizards/mccortex_full_uncorr_JAR/k63/graphs/S_JAR53_S13.raw.ctx [1 src colour] with filter: 0->7
[12 Jan 2019 09:05:35-jUF][GReader] 3,074,158,683 kmers, 60.1GB filesize
[12 Jan 2019 10:31:24-jUF][GReader] Loaded 3,074,158,683 / 3,074,158,683 (100.00%) of kmers parsed
[12 Jan 2019 10:31:24-jUF][cleaning] Total kmers loaded: 11,230,083,705
[12 Jan 2019 10:31:24-jUF][hasht] buckets: 536,870,912 [2^29]; bucket size: 26;
[12 Jan 2019 10:31:24-jUF][hasht] memory: 209GB; filled: 11,230,083,705 / 13,958,643,712 (80.45%)
[12 Jan 2019 10:31:24-jUF][hasht] collisions 0: 11031800044
[12 Jan 2019 10:31:24-jUF][hasht] collisions 1: 178522995
[12 Jan 2019 10:31:24-jUF][hasht] collisions 2: 17286272
[12 Jan 2019 10:31:24-jUF][hasht] collisions 3: 2132402
[12 Jan 2019 10:31:24-jUF][hasht] collisions 4: 291855
[12 Jan 2019 10:31:24-jUF][hasht] collisions 5: 42497
[12 Jan 2019 10:31:24-jUF][hasht] collisions 6: 6465
[12 Jan 2019 10:31:24-jUF][hasht] collisions 7: 994
[12 Jan 2019 10:31:24-jUF][hasht] collisions 8: 153
[12 Jan 2019 10:31:24-jUF][hasht] collisions 9: 22
[12 Jan 2019 10:31:24-jUF][hasht] collisions 10: 5
[12 Jan 2019 10:31:24-jUF][hasht] collisions 11: 1
[12 Jan 2019 10:31:24-jUF][cleaning] Calculating unitig stats with 16 threads...
[12 Jan 2019 10:31:24-jUF][cleaning] Using kmer gamma method
[src/graph/db_unitig.c:107] Assert Failed db_unitig_extend(): node.key != ((18446744073709551615UL)>>1)
[12 Jan 2019 10:31:24-jUF] Assert Error
[src/graph/db_unitig.c:107] Assert Failed db_unitig_extend(): node.key != ((18446744073709551615UL)>>1)
[12 Jan 2019 10:31:24-jUF] Assert Error