Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyVQ and HDF5 file reading scalability #170

Open
kwschultz opened this issue May 24, 2016 · 0 comments
Open

PyVQ and HDF5 file reading scalability #170

kwschultz opened this issue May 24, 2016 · 0 comments

Comments

@kwschultz
Copy link
Contributor

Currently, the QuakeLib library reads in the HDF5 event/simulation files completely into memory, regardless of size. Sometimes we have simulations that reach > 10GB and doing any simple PyVQ plots requires like 20GB of RAM and takes forever.

We need: An improved model that uses, say the h5py Python module to incrementally read in the event file. Maybe we should just be reading in 1000 events at a time, computing whatever we need, then keep the plot-able results and delete the events from memory ("del" in Python).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant