Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fast5 zip #135

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

Fast5 zip #135

wants to merge 18 commits into from

Conversation

robegan21
Copy link

This patch allows a .zip file (of fast5 files) to be processed efficiently and in the same way that the tarball of fast5 files are processed.

I have found the .zip to have several key advantages over a tarball of fast5 files:

  1. There is about 30% space saved in my data sets
  2. the .zip file is indexed, and efficient to extract a single file, so there is no need to extract the entire run or parse the whole archive
  3. the .zip file can be read in parallel (using multithread.Pool)
  4. the .zip has a checksums to protect against data corruption
  5. extracting zip files on a network drive (GPFS) is about 5-10x faster than working with each file individually, so long as the extraction is saved locally (/tmp) or to memory (python's ZipFile.read() or /dev/shm)

Please consider incorporating this into poretools, as I believe that zips are a workable way forward to managing the millions of fast5 files that are presently being generated per run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant