use Record.filename to get match filename for full gather outputs
#303
Comments
4 tasks
|
shall we do a sourmash-rs release now that sourmash-bio/sourmash#3121 is released? |
|
hmm -- let's hold off for a bit. I think I actually needed |
bluegenes
added a commit
that referenced
this issue
May 10, 2024
…multigather` (#298) This PR adds utilities for building full gather results file for `fastgather` and non-rocksdb `fastmultigather`, and makes full output default. - Fixes #287 - Fixes #187 - Fixes #254 - includes a local fix for #318, which means that the `fastgather` and **non-rocksdb** `fastmultigather` full output here matches sourmash gather. Issues with rocksdb gather are being handled elsewhere. ## Benchmarking | software/version | command | details | time | max RAM | | -------- | -------- | -------- | -- | -- | | branchwater v0.9.3 | `fastgather` | minimal result | <span style="color:green">**1m 47s**</span> | <span style="color:black">**14 GB**</span> | | branchwater v0.9.3-dev | `fastgather` | full result | <span style="color:green">**1m 57s**</span> | <span style="color:black">**14 GB**</span> | | branchwater v0.9.3 | `fastmultigather` | minimal result | <span style="color:red">**8m 3s**</span> | <span style="color:black">**25 GB**</span> | | branchwater v0.9.3-dev | `fastmultigather` | full result | <span style="color:red">**8m 9s**</span> | <span style="color:black">**25 GB**</span> | | branchwater v0.9.3 | `fastmultigather` | rocksdb full result | <span style="color:green">**24s**</span> | <span style="color:green">**600 MB**</span> | progress/separate PRs: - [x] Fill out `match_filename` in full results (#303; requires new sourmash core release with sourmash-bio/sourmash#3121) - [x] switch to using `KmerMinHashBTree` for hash subtraction +benchmark. Per luiz, `KmerMinHashBTree` are better for any situation where we'll be subtracting/adding hashes to a sketch #310 - [x] sourmash: make getting `Record`.filename public in order to keep match_filename and write it to full results. (sourmash-bio/sourmash#3121) - [x] remove --full-results and make full results default #327 --------- Co-authored-by: C. Titus Brown <titus@idyll.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
a
match_filenamecolumn is being added to--full-resultsgather output in #298 so that the file matches standard sourmash gather output. However, we can't actually fill it in yet.After sourmash-bio/sourmash#3121 makes it into a release, we should be able to get the
match_filenamefrom theRecordduring prefetch. To use it in the result, I think it'll amatch_filenameparam may need to be added to the PrefetchResult.The full gather result is generated w/in
consume_query_by_gather, which usesprefetchinternally.The text was updated successfully, but these errors were encountered: