Permission issues on mounted Volumes for RHDH container

[gazarenkov]

The issue shown up in the hosted Kubernetes environments, such as AKS and EKS with default RHDH (showcase) container using both Helm Chart and Operator.

On Creating Backstage CR on unchanged default configuration RHDH InitContainer failed with error like:

#Error: EACCES: permission denied, open '/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-github-dynamic-0.2.2.tgz'

The reason why it is not failed on local environment is that chown/chmod/fix-permissions stuffs seems to work for mounted directories there, but does not work on AKS/EKS (which is quite expected IMO), so, we have permissions like:

• drwxr-xr-x 1 root root 4096 Mar 1 18:56 dynamic-plugins - for "hosted" (fails, with permission error)
• drwxrwxr-x 1 root root 4096 Mar 1 18:56 dynamic-plugins - for "local" (works)
  To work around this problem, fsGroup= can be used in the Pod's securityContext, it makes directory permissions like:

drwxrwsr-x 14 root 1001 4096 Mar 11 14:59 dynamic-plugins-root

which works, but, it MAY lead to make Pod's startup slower. As stated here:
Kubernetes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted. For large volumes, checking and changing ownership and permissions can take a lot of time, slowing Pod startup.
Also, for Openshift it requires "Openshift friendly" UID.

So, it was decided to not to include it as a general solution and solve case-by-case, making proper documentation (for AKS/EKS etc), which is not ideal as well because it make the procedure more difficult while is not really "platform specific" strictly speaking.

[gazarenkov]

JIRA

[kadel]

drwxrwsr-x 14 root 1001 4096 Mar 11 14:59 dynamic-plugins-root

which works, but, it MAY lead to make Pod's startup slower. As stated here: Kubernetes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted. For large volumes, checking and changing ownership and permissions can take a lot of time, slowing Pod startup.

I don't think this will be a problem for the dynamic-plugins-root directory. This directory is mounted as emptyDir, so you can be sure there will be no files at startup.

[gazarenkov]

drwxrwsr-x 14 root 1001 4096 Mar 11 14:59 dynamic-plugins-root
which works, but, it MAY lead to make Pod's startup slower. As stated here: Kubernetes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted. For large volumes, checking and changing ownership and permissions can take a lot of time, slowing Pod startup.

I don't think this will be a problem for the dynamic-plugins-root directory. This directory is mounted as emptyDir, so you can be sure there will be no files at startup.

@kadel
Right, I do not think it will bring any practical problem, just the thing to mention.
The thing which I do not really like about it is that it is "Pod" and not "Container" security context.

[gazarenkov]

Another note:
Looks like fsGroup does not work with it either (tried with 1001), pretty much same error about userID
So, looks like we need at minimum 2 configurations:
for K8s (with fsGroup or with trick I proposed)
for OCP (w/o any user/group ID)

[rm3l]

PROPOSAL for Kubernets (hardly simply applicable for Openshift) which allow to solve this problem can be as simple as using root user in InitialContainer insead of default user = 1001 (see below).
This way InitialContainer has full control on dynamic plugins preparation and Backstage Container can safely use it as it is (read permissions is enough) .

deployment.yaml
 spec:
  ...
  template:
  ...
    spec:
      ...
      initContainers:
        - name: install-dynamic-plugins
        ...
         securityContext:
            runAsUser: 0

I would be reluctant to set runAsUser: 0 (as you suggested), as even on vanilla K8s, it may not work. I remember K8s introduced Pod Security Standards (enforced using labels at the namespace level), and a restricted policy could forbid running containers (and init containers) as root.