run jupyterlab-pachyderm without needing root privileges

[erikerlandson]

What is the goal / desired outcome?
Run jupyterlab with pachyderm extension in OpenShift (which does not allow running containers as root by default)

If there is a way to accomplish this today via workaround, what does that require?
It is possible to configure containers to be allowed to run with root, but enabling this for user/external facing containers (jupyter notebooks) is bad security posture.

(Optional) What is your proposal for a feature to solve this?
If possible, alternative implementation not requiring fuse: some mounting alternative that can operate without a container having root privs.

Environment?:

• Kubernetes version (use kubectl version): OpenShift

[erikerlandson]

for reference, the container build file I'm hacking on

https://github.com/os-climate/pachyderm-notebook-image/blob/main/images/pachyderm-notebook/Containerfile

[lukemarsden]

Does https://github.com/pachyderm/jupyterlab-pachyderm/issues/108 sufficiently address this?

[erikerlandson]

@lukemarsden that link gives me a 404 - is that by any chance a private repo?

[lukemarsden]

Sorry! Here's a screenshot:

[erikerlandson]

@lukemarsden it might - that is, it should address the security issues by at least localizing the elevated privileges, to a container outside the user-exposed processes.

Since we are launching each user's jupyter pod from the JupyterHub launcher, I'm wondering what the implications of that are for configuring this new sidecar container. Can the JH launcher also be instructed to add these side cars?

cc @guimou

[guimou]

This could be an interesting workaround. Although a better solution would be to have a real CSI driver that kind of does the same job. It could work with definining a StorageClass for each connection, and you would mount directly as a volume in the pod at launch, similare to what JuiceFS does with S3 buckets.

[lukemarsden]

Yes, we believe the JH launcher can be configured appropriately, both with the sidecar and also the shared volume mount & propagation settings.

I agree that porting the mount-server to also function as a CSI driver would be cool, probably a pretty big project though. 😄

[erikerlandson]

FWIW, I have found that there is a very easy way to smoke-test if your container image is likely to work with OpenShift, and that is to set USER 9876:0 either in the last line of your docker- container-file itself, or in the pod yaml. If the uid is some id that does not correspond to any entry in passwd file, that pretty faithfully simulates OpenShift's behavior of assigning random and anonymous uids:

(app-root) sh-4.4$ id
uid=1000630000(1000630000) gid=0(root) groups=0(root),1000630000

If your container can run like this, it's probably going to run out-of-box in OpenShift too, or at least port with significantly less effort.

[claytonlemons]

This could be an interesting workaround. Although a better solution would be to have a real CSI driver that kind of does the same job.

@guimou, The CSI driver sounds like a more transparent (w.r.t. to the end user) solution than adding a privileged sidecar. In K8S, is it possible to dynamically modify a server pod's own volumes, especially without causing the pod to restart?

[RaananHadar]

Just bringing up from the past a discussion about Pachyderm CSI which can also be useful for many integrations.

[guimou]

@claytonlemons Sorry, forgot to answer your question. No, you cannot do this dynamically. Everything is allocated/mounted by Kubernetes at the creation of the pod.