Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GGCAT keeps panicking about too many open files #25

Open
iosfwd opened this issue May 24, 2023 · 1 comment
Open

GGCAT keeps panicking about too many open files #25

iosfwd opened this issue May 24, 2023 · 1 comment

Comments

@iosfwd
Copy link

iosfwd commented May 24, 2023

I'm trying to run GGCAT on the following dataset: https://ftp.ebi.ac.uk/pub/databases/ENA2018-bacteria-661k/
I had 5T of free disk space and the following flags: -k 31 -m 32 -j 8 -s 1
However, my runs keep panicking on too many open files despite raising ulimit -n. My first two runs with ulimit -n of 1024 and 4096 crashed in the following phase:

Started phase: kmers merge prev stats:
Thread panicked at location: libs-crates/parallel-processor-rs/src/memory_fs/file/internal.rs:248:26
Error message: called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Uncategorized, message: "Too many open files" }
Backtrace:    0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
  18: <unknown>
  19: <unknown>
  20: <unknown>
  21: <unknown>
  22: <unknown>

Then I raised the ulimit -n to 10240, and it crashed in the following phase:

Finished phase: kmers merge. phase duration: 20675.80s gtime: 30013.67s
Started phase: hashes sorting prev stats: 
Finished phase: hashes sorting. phase duration: 646.23s gtime: 30659.90s
Started phase: links compaction prev stats: 
Thread panicked at location: libs-crates/parallel-processor-rs/src/memory_fs/file/internal.rs:248:26
Error message: called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Uncategorized, message: "Too many open files" }
Backtrace:    0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
  18: <unknown>
  19: <unknown>
  20: <unknown>
  21: <unknown>
  22: <unknown>
@tmaklin
Copy link
Contributor

tmaklin commented Apr 19, 2024

This seems like it may have been fixed in ec68afe? I'm not getting the error in a similar use case anymore after updating.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants