dill>0.3.5.1 unable to pickle CompiledFFI object

[Jmast]

fails starting with dill==0.3.6. Not sure what changed in dill; however, at the very least the maestro deps need to be dill<=0.3.5.1

pickler._batch_setitems(iter(source.items()))

File "/usr/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/home/mast2/spl_flux_test/local/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.8/pickle.py", line 717, in save_reduce
save(state)
File "/home/mast2/spl_flux_test/local/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/home/mast2/spl_flux_test/local/lib/python3.8/site-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/home/mast2/spl_flux_test/local/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.8/pickle.py", line 578, in save
rv = reduce(self.proto)
TypeError: cannot pickle 'CompiledFFI' object
2022-11-22 17:20:10,629:name : 453:ERRO: Maestro workflow finished with exit status: 1

[FrankD412]

Thanks for the issue @Jmast -- I'll fix this by the end of the weekend. Is this holding you up at all? Do you need something sooner?

[Jmast]

Thanks, no rush, @FrankD412 , but there are a few other things:

1. I have a issue outstanding regarding corona flux version and maestro being merged that I commented on in PR: "added flux adaptor v0.31.0 added flux adaptor v0.31.0 #395") comment is here
2. Until a more suitable approach is implemented, the version checks for flux probably need to be entirely removed as I have found that the "strict" version throws an exception for "unreleased" installs of flux. Ultimately I think that the flux plugin needs to support a range of versions and should just be backward compatible as much as possible with logic based on version numbers. Right now in my forked version I just commented out the entire try/except block in connect_to_flux in maestrowf/abstracts/interfaces/flux.py because the StrictVersion() call throws an exception for versions like 0.42.0-84-g1dcbcd97d

[FrankD412]

Oh -- sorry about the delayed fix. Slipped through the cracks on my end. I can take a look at that soon.