Pickling: respawning

[zlobober]

Assume the user code is designed to work with some fixed docker image, e.g. ubuntu-22.04-py3. In order to use that, currently one needs to run smth like yt.run_map(..., spec={"mapper": {"docker_image": "ubuntu-22.04-py3"}}).

This does not ensure that the host that runs this run_map is actually using the same Python at the same version of Ubuntu; failure to use similar environment would result in failure during unpickling.

Let's introduce a helper, e.g. yt.use_docker_image("ubuntu-22.04-py3"), which is to be invoked immediately at the beginning of a python file. As a result, it must do following things:

• parent process spawns a child docker container (run on the local machine) with self as an entrypoint and exposing the full file system + host network + some env variable a-la YT_RESPAWNED_IN_CONTAINER=1
• the child process runs past the yt.use_docker_image because YT_RESPAWNED_IN_CONTAINER=1 and executes the rest of the script
• all operation invocations like yt.run_map automatically assume the specified docker image

NB: there is no pickling between the parent and the child processes, since the respawning should happen right at the beginning of a script. If user runs "use_docker_image" later, we do not guarantee the correct behavior.