Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pickle w/o Statics #1458

Open
ax3l opened this issue Jun 6, 2023 · 1 comment
Open

Pickle w/o Statics #1458

ax3l opened this issue Jun 6, 2023 · 1 comment
Labels
frontend: Python3 third party third party libraries that are shipped and/or linked

Comments

@ax3l
Copy link
Member

ax3l commented Jun 6, 2023

For the first implementation of multi-process (multi-node) Dask, we pickle objects like the Record and RecordComponents + their series.

The series is unpickled into a static function member, to avoid:

  • double open/parsing for each object
  • having to express lifetime of the series on the unpickled side with respect to the pickled object.

This works as a hack until you need to work with two series at a time.

https://github.com/openPMD/openPMD-api/blob/0.15.1/include/openPMD/binding/python/Pickle.hpp#L73-L77

@ax3l ax3l added third party third party libraries that are shipped and/or linked frontend: Python3 labels Jun 6, 2023
@pordyna
Copy link
Contributor

pordyna commented Jun 22, 2023

@ax3l I tried using dask delayed with openpmd-api for parallelization over iterations. I couldn't get it to work, I think, because dask wasn't able to pickle the series object. Would it make sense to add add_pickle to Series and Iteration?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
frontend: Python3 third party third party libraries that are shipped and/or linked
Projects
None yet
Development

No branches or pull requests

2 participants