Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unpack cleaned subsamples directly into subsample table #7

Open
lgarrison opened this issue Feb 23, 2021 · 1 comment
Open

Unpack cleaned subsamples directly into subsample table #7

lgarrison opened this issue Feb 23, 2021 · 1 comment

Comments

@lgarrison
Copy link
Member

Currently, we build a concatenated table of all cleaned particles, reindex it, merge it with the original subsamples, then do the RVint unpacking on the whole table. We may be able to achieve better performance by never constructing the concatenated cleaned particle table and instead do the RVint unpacking directly into the final location in the combined particle table.

@lgarrison
Copy link
Member Author

In #16, I noticed that the peak RSS usage is way higher than the final usage. I think the ASDF objects are holding onto internal buffers with the uncompressed data, so garbage collection requires deleting the ASDF handles. So we will want to pay attention to that in this refactor. But it may be moot if we get an ASDF "read_into" functionality working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant