Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save and load hdf5 representation #632

Open
niekdejonge opened this issue Apr 24, 2024 · 0 comments
Open

Save and load hdf5 representation #632

niekdejonge opened this issue Apr 24, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@niekdejonge
Copy link
Collaborator

By saving and loading spectra in HDF5 it allows you to only access a few spectra at any position in a list of spectra. Storing spectra in this format is especially helpful for training models, since there is no need to have all spectra in memory.

It stores the spectra like same size vectors. This means the spectra need to have a predefined number of fragments and if they have less, they need to be padded with zeroes. But due to efficient compression this is not really a memory issue.

@roman-bushuiev implemented something similar already for https://github.com/roman-bushuiev/DreaMS and will try to implement it for matchms, so make the DreaMS package build on matchms.

@florian-huber florian-huber added the enhancement New feature or request label May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants