Pickling: jupyter notebooks

[zlobober]

Pickling is known not to be working when working in Jupyter/IPyNotebook.
It would be great to investigate the root cause of that and, ideally, to come up with a workaround.

Reproduce: yt.run_map(identity, ...) where identity is def identity(record): yield record.

[thenno]

Jupyterlab works on macbook m1. How to reproduce:

1. Install pyenv https://github.com/pyenv/pyenv
2. Install python 3.8: pyenv install 3.8 (this version works on our cluster by default)
3. Create and activate venv pyenv virtualenv 3.8 ytwrappertest && pyenv activate ytwrappertest
4. Install yt client and jupyter lab pip install jupyterlab==4.1.8 ytsaurus-client==0.13.14 ytsaurus-yson==0.4.5
5. Run jupyterlab YT_TOKEN="my token" YT_PROXY="my yt cluster" jupyter lab
6. Run notebook https://github.com/thenno/yt-on-jupyter-lab-example/blob/main/ytwrapperplusjupyter.ipynb

Maybe there are some differences between jupyter notebook / jupyter lab, but as far as I know the differences are only in the frontend.

[thenno]

There is a problem using class-based operations. For some reason yt client can't pickle classes.

How to reproduce: https://github.com/thenno/yt-on-jupyter-lab-example/blob/main/ytwrapperplusjupyter_broken.ipynb

Traceback (most recent call last):
  File "_py_runner.py", line 174, in <module>
    main()
  File "_py_runner.py", line 170, in main
    yt.wrapper.py_runner_helpers.process_rows(__operation_dump_filename, __config_dump_filename, start_time=start_time)
  File "/slot/sandbox/./tmpfs/modules/yt/wrapper/py_runner_helpers.py", line 309, in process_rows
    operation, params = unpickler.load(f_operation_dump)
  File "/slot/sandbox/./tmpfs/modules/yt/packages/dill/_dill.py", line 373, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/slot/sandbox/./tmpfs/modules/yt/packages/dill/_dill.py", line 647, in load
    obj = StockUnpickler.load(self)
  File "/slot/sandbox/./tmpfs/modules/yt/packages/dill/_dill.py", line 637, in find_class
    return StockUnpickler.find_class(self, module, name)
AttributeError: Can't get attribute 'Mapper' on <module '_main_module' from '/slot/sandbox/_main_module.py'>

[dmi-feo]

For some reason yt client can't pickle classes.

minor, but: it seems it can pickle, but can't unpickle, according to your traceback

[dmi-feo]

The issue is not reproduced if pickling/framework set to cloudpickle, which is designed to work with stuff like jupyter notebooks

[dmi-feo]

• likely it was broken by this - 87be97e#diff-eb29140ea94f56eb430edcf727c8d30ba898bedabd41283582fd371f5e067760R36 - commit
• difference between pickling by ref and by value is described in cloudpickle source code: https://github.com/cloudpipe/cloudpickle/blob/f111f7ab6d302e9b1e2a568d0e4c574895db6a6e/cloudpickle/cloudpickle.py#L232-L241. Cloudpickle detects whether an object should be pickled by ref, while dill allows only to set the setting before pickling
• since we can determine the case when wrapper is being used from interactive env (like jupyter or ipython), we can force dill to pickle by value only for such cases