SQLite_TOOBIG error SAM2RMA

[MicroSeq]

Please leave the headers below and replace the relevant text. This will expedite addressing the ticket.

Name the workflow
Taxonomic-Profling-Diamond-Megan

Describe the bug

The workflow is crashing at the MakeRMAUnfiltered/Filtered step. It appears there is a SQLite error, I tried increasing the available memory to MEGAN (~400 Gb) and that did not resolve the issue. Possibly, I need to reduce the top hits at the diamond blastx step (currently at --top 10)? Perhaps I can tweak the SAM file to avoid having re-do the alignments to allow for this to be manageable?

Relevant section of the log file:

Input domination filter: MinPercentCoverToStronglyDominate=90.0 and TopPercentScoreToStronglyDominate=90.0
10% 20% 30% 40% Caught:
org.sqlite.SQLiteException: [SQLITE_TOOBIG] String or BLOB exceeds size limit (statement too long)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.core.DB.newSQLException(DB.java:1135)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.core.DB.newSQLException(DB.java:1146)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.core.DB.throwex(DB.java:1106)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.core.NativeDB.prepare_utf8(Native Method)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.core.NativeDB.prepare(NativeDB.java:122)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.core.DB.prepare(DB.java:264)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.jdbc3.JDBC3Statement.lambda$executeQuery$1(JDBC3Statement.java:75)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.jdbc3.JDBC3Statement.withConnectionTimeout(JDBC3Statement.java:429)
at org.xerial.sqlitejdbc@3.39.3.0/org.sqlite.jdbc3.JDBC3Statement.executeQuery(JDBC3Statement.java:73)
at megan/megan.accessiondb.AccessAccessionMappingDatabase.getValues(AccessAccessionMappingDatabase.java:222)
at megan/megan.rma6.RMA6FromBlastCreator.parseFiles(RMA6FromBlastCreator.java:257)
at megan/megan.tools.SAM2RMA6.createRMA6FileFromSAM(SAM2RMA6.java:340)
at megan/megan.tools.SAM2RMA6.run(SAM2RMA6.java:307)
at megan/megan.tools.SAM2RMA6.main(SAM2RMA6.java:69)

Expected behavior

Screenshots

Log files

NitrifyingCombined.MakeRMA.filtered.readCount.log

[MicroSeq]

I am wondering if it is a cache size issue, so I am trying to re-run it with sam2rma set to 5000 instead of 10000 as the default.

https://megan.cs.uni-tuebingen.de/t/sqlite-toobig-error-from-daa2rma/2212/5

Update: crashed at the same point with the decreased cache size.