Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pickling a histogram creates an extra copy of the array in memory #828

Open
bendavid opened this issue Feb 13, 2023 · 1 comment
Open

Pickling a histogram creates an extra copy of the array in memory #828

bendavid opened this issue Feb 13, 2023 · 1 comment

Comments

@bendavid
Copy link

Pickling a large boost histogram results in a second copy being created in memory. Via the use of the PickleBuffer mechanism it should be possible to avoid this.

e.g.

import boost_histogram as bh

axis = bh.axis.Regular(1024*1024*1024, 0., 1)
htest = bh.Histogram(axis)

uses 8GB of memory as expected

import boost_histogram as bh
import pickle

axis = bh.axis.Regular(1024*1024*1024, 0., 1)
htest = bh.Histogram(axis)

with open("test.pkl", "wb") as f:
    pickle.dump(htest, f, protocol = pickle.HIGHEST_PROTOCOL)

uses 16GB

@henryiii
Copy link
Member

I think we'd need support in pybind11 for this (could be wrong, but I think so). That would mean it needs a CAPI interface, which I would guess it has?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants