Separate notebooks appear to share state

[ca-scribner]

Bug Description

Notebooks appear to share/clobber each other's state. If I create two notebook servers they'll share the same layout, try to write to the same notebooks, etc. It means its really easy for a user to overwrite their work or break one notebook server by using another.

To Reproduce

If I create a notebook server:

dss initialize --kubeconfig ~/.kube/config
dss create notebook1 --image=kubeflownotebookswg/jupyter-pytorch-cuda-full:latest --kubeconfig ~/.kube/config

and connect to the url, I see a fresh notebook server (starting on the splash screen).
I can create a few terminals and notebooks, and if I reconnect to the server I still see these terminals and notebooks remaining open.

If I then create a new notebook server:

dss create notebook2 --image=kubeflownotebookswg/jupyter-pytorch-cuda-full:latest --kubeconfig ~/.kube/config

and connect to the new url, I do not see the notebook splash screen but instead I start with the same configuration of open terminals and notebooks as the previous notebook server. This is really confusing as it seems like you connected to Notebook Server 1, when you actually connected to server 2. The new notebook server will try to continue using the notebooks of the previous server, but the kernel will be unavailable (since it is on a different machine). The terminals will be fresh terminals running on the new notebook server (we can see what notebook server the terminals are running on by looking at the machine name in their prompt).

I think what's happening here is that notebook servers store some local data about their state (in the .jupyter folder, I think?). When we run multiple dss notebook servers, these servers all save to the same state and clobber each others.

We might need to isolate these servers somehow, maybe keeping each of their state directories separate?

Environment

dss

Relevant Log Output

see above

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5608.

This message was autogenerated

[misohu]

After syncing with Maciej we conclude that the best best resolution will be to

• for each notebook create a separate PVC which will be mounted to home directory
• have one shared pvc which will be mounted on each notebook on $HOME/shared folder

In the first version we will just have the shared pvc mounted to $HOME/shared .

[misohu]

We just partialy resolved it for version 1 with shared folder.