Using pickle with parallel Meep #2801
Unanswered
SilvieLuisa
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi, I am running many simulations with a similar simulation setup. The aim is to compute transmission and reflection spectra, so I need to run a normalization simulation and a true simulation, but the normalization simulation could be run just once, as the geometries are really similar with equal volume size.
So I need to save the following objects from the normalization simulation and pass them to the true simulations:
I am using parallel Meep and each simulation is sent to a LSF queue as a separate job, so I cannot just use the variables as they are, I need to save them to file(s) and then read them by another job.
Unfortunately, I cannot use any built-in function based on HDF5, as there seems to be missing some library on the Linux servers and I am not a root. Moreover, I am afraid of difficulties since I am using parallel Meep.
My idea was to pickle and un-pickle the objects. For storing, I use the
mp.am_master()
check to make sure that the data is saved only once.However, even though the data "seems correct" in terms of dimensionality, the results for the reflection spectra are pure nonsense. When I do the same simulation with pickling, but without parallelism, it works well.
When I do the same simulation in just one file (without saving the data to pickle files at all), it also works fine.
There must be some issue with storing the data, even though only the master process should be doing it.
Please does anyone have a suggestion how to solve this? Thanks!
Beta Was this translation helpful? Give feedback.
All reactions