Storage Permissions on PVC

[TProhofsky]

My first few attempts of starting a cluster are running into some permissions issues on the PVC. The pods start and crashes right away with:
{"level":"info","ts":1652381597.1015086,"logger":"initdb","msg":"initdb: error: could not create directory "/var/lib/postgresql/data/pgdata": Permission denied\n","pipe":"stderr","logging_pod":"cluster-example-1"}.

My understanding is the CNPG operator is starting pods with a non-root user but the PVC is published with a mountpoint in the pod owned by root. I believe the operator should be specifying the fsGroup and fsUser in the securityContext of the pod. If so then this is a bug. If not, please let me know the design approach or if I have missed something using the quick start example.

[sxd]

Hi @TProhofsky

Are you deploying the operator in a vanilla Kubernetes or somewhere else? Because the containers are created with fsGroup and fsUser https://github.com/cloudnative-pg/cloudnative-pg/blob/main/pkg/specs/pods.go#L278

[TProhofsky]

Hi Jonathan,

I'm running Kubernetes V1.23 brought up with Kubespray on bare metal, no VMs. The PVCs are LVM2 volumes and published to a mount with root ownership and write permissions. To bring up the CNPG cluster I added an option to the CSI driver to chmod ugo+rwx the mount target to open the PVC to all. When I dump the pod's yaml I see the security context is set as expected by the operator:

 securityContext:
    fsGroup: 26
    runAsGroup: 26
    runAsNonRoot: true
    runAsUser: 26

When I look at the mounted PVC the mount is still owned by root:

[root@node1 ~]# kubectl exec cluster-example-1 -- ls -lta /var/lib/postgresql/data/pgdata
Defaulted container "postgres" out of: postgres, bootstrap-controller (init) total 72
drwx------.  2 postgres tape   101 May 14 16:09 pg_stat_tmp
drwx------.  3 postgres tape   268 May 13 17:46 pg_wal
 - - snip - -
drwx------.  2 postgres tape     6 May 13 17:39 pg_snapshots
drwx------.  2 postgres tape     6 May 13 17:39 pg_twophase
drwxrwxrwx.  3 root     root    20 May 13 17:39 ..

This appears that kubelet did not apply the fsGroup change. I suspect the fsGroupPolicy is missing or set to none in the CSIDriver spec. https://kubernetes-csi.github.io/docs/support-fsgroup.html.

CNPG is doing everything right from what I can gather. I'll get to the bottom of the CSI issue. Thanks for the reply.

[sxd]

Hi @TProhofsky

Can you tell us which CSI driver are you using? as I can see you already have the proper permissions to create everything even if .. is owned by root since, in fact, you have 777 on that directory, which it's not great at all but you should be able to work with.

Please let us know what you find, since this can go into a FAQ for CNPG and that will be a great contribution!

Regards!

[TProhofsky]

I'm using a LVM2 CSI driver developed by Mesosphere and forked by Seagate to add support for Shared Volume Groups. https://github.com/Seagate/csiclvm/tree/oVirtDev. I just checked in the fix that correctly handles the volume_mount_group CSI feature in a development branch. Now the PVC is restricted to UID/GID of postgres/tape after the cluster is started.

[root@node1 postgresql]# kubectl exec cluster-example-1 -- stat /var/lib/postgresql/data/pgdata
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
  File: /var/lib/postgresql/data/pgdata
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fd00h/64768d    Inode: 131         Links: 19
Access: (0700/drwx------)  Uid: (   26/postgres)   Gid: (   26/    tape)
Access: 2022-05-17 01:50:48.614105564 +0000
Modify: 2022-05-17 01:50:27.101667051 +0000
Change: 2022-05-17 01:50:27.101667051 +0000
 Birth: 2022-05-17 01:50:21.722557402 +0000

For a FAQ I would suggest folks see the pod state transition from PodInitializing to Error, then dump the pod log. If seeing:

{"level":"info","ts":1652750651.449766,"logger":"initdb","msg":"initdb: error: could not create directory \"/var/lib/postgresql/data/pgdata\": Permission denied\n","pipe":"stderr","logging_pod":"cluster-example-1"}

then PVC is not writeable by postgres uid and check if CSI driver supports fsGroup setting.

A 60 second sleep when initdb hits this would be nice. Kuberentes tears down the pod so fast, you can't exec in to see what is going on.