You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we build a concatenated table of all cleaned particles, reindex it, merge it with the original subsamples, then do the RVint unpacking on the whole table. We may be able to achieve better performance by never constructing the concatenated cleaned particle table and instead do the RVint unpacking directly into the final location in the combined particle table.
The text was updated successfully, but these errors were encountered:
In #16, I noticed that the peak RSS usage is way higher than the final usage. I think the ASDF objects are holding onto internal buffers with the uncompressed data, so garbage collection requires deleting the ASDF handles. So we will want to pay attention to that in this refactor. But it may be moot if we get an ASDF "read_into" functionality working.
Currently, we build a concatenated table of all cleaned particles, reindex it, merge it with the original subsamples, then do the RVint unpacking on the whole table. We may be able to achieve better performance by never constructing the concatenated cleaned particle table and instead do the RVint unpacking directly into the final location in the combined particle table.
The text was updated successfully, but these errors were encountered: