starting proxy as pod fails with wrong ownership of Squid cache PV

[jmozd]

Problem description

When starting the containerized version of the proxy in a Kubernetes environment, multiple PVs are created and attached to the single proxy pod.
One of the PVs is used by the Squid container, mounted as /var/cache/squid. Per the default configuration, the mounted file system is owned by user root, while the container (or rather the processes started in the container) runs as user "squid". Therefore, during container startup, changing ownership of the cache directory fails.

Steps to reproduce

1. create Uyuni proxy in Kubernetes cluster i.e. via
   helm install uyuni-proxy oci://registry.opensuse.org/uyuni/proxy-helm -f uyuni/config.yaml -f uyuni/httpd.yaml -f uyuni/ssh.yaml --set ingress=nginx
   2.Check the logs of the Proxy pod's "squid" container

chown: cannot read directory '/var/cache/squid/lost+found': Permission denied
2024-03-30T10:55:23.183795128Z chown: changing ownership of '/var/cache/squid': Operation not permitted

Uyuni version

Containers:
  [...]
  squid:
    Container ID:   docker://04a11ff0fb43c048f088ab7dcf6e3b38cd51b8302ca239d002c52f8668b338b4
    Image:          registry.opensuse.org/uyuni/proxy-squid:latest
    Image ID:       docker-pullable://registry.opensuse.org/uyuni/proxy-squid@sha256:08bff4863a1a2146684eb902370c4058a01ef53f573b20324217faa326277d01
    Port:           8088/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 30 Mar 2024 11:01:09 +0000
      Finished:     Sat, 30 Mar 2024 11:01:09 +0000
    Ready:          False
    Restart Count:  6
    Environment:    <none>
    Mounts:
      /etc/uyuni/config.yaml from config-volume (ro,path="config.yaml")
      /var/cache/squid from squid-cache (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dq7bf (ro)

Uyuni proxy version (if used)

Pulled: registry.opensuse.org/uyuni/proxy-helm:2024.2.0

Useful logs

chown: cannot read directory '/var/cache/squid/lost+found': Permission denied
2024-03-30T10:55:23.183795128Z chown: changing ownership of '/var/cache/squid': Operation not permitted
2024-03-30T10:55:23.199521363Z 2024/03/30 10:55:23| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2024-03-30T10:55:23.201225266Z 2024/03/30 10:55:23| WARNING: (B) '::/0' is a subnetwork of (A) '::/0'
2024-03-30T10:55:23.201264278Z 2024/03/30 10:55:23| WARNING: because of this '::/0' is ignored to keep splay tree searching predictable
2024-03-30T10:55:23.201269744Z 2024/03/30 10:55:23| WARNING: You should probably remove '::/0' from the ACL named 'all'
2024-03-30T10:55:23.231226340Z 2024/03/30 10:55:23 kid1| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2024-03-30T10:55:23.232607658Z 2024/03/30 10:55:23 kid1| WARNING: (B) '::/0' is a subnetwork of (A) '::/0'
2024/03/30 10:55:23 kid1| WARNING: because of this '::/0' is ignored to keep splay tree searching predictable
2024-03-30T10:55:23.232635814Z 2024/03/30 10:55:23 kid1| WARNING: You should probably remove '::/0' from the ACL named 'all'
2024-03-30T10:55:23.244943844Z 2024/03/30 10:55:23 kid1| Current Directory is /
2024-03-30T10:55:23.244963697Z 2024/03/30 10:55:23 kid1| Creating missing swap directories
2024-03-30T10:55:23.244968762Z 2024/03/30 10:55:23 kid1| /var/cache/squid exists
2024-03-30T10:55:23.245050087Z 2024/03/30 10:55:23 kid1| Not currently OK to rewrite swap log.
2024-03-30T10:55:23.245059258Z 2024/03/30 10:55:23 kid1| storeDirWriteCleanLogs: Operation aborted.
2024-03-30T10:55:23.245063506Z 2024/03/30 10:55:23 kid1| FATAL: Failed to make swap directory /var/cache/squid/00: (13) Permission denied
2024-03-30T10:55:23.245155770Z 2024/03/30 10:55:23 kid1| Squid Cache (Version 5.7): Terminated abnormally.
2024-03-30T10:55:23.255363550Z 2024/03/30 10:55:23 kid1| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback.
2024-03-30T10:55:23.256627130Z 2024/03/30 10:55:23 kid1| WARNING: (B) '::/0' is a subnetwork of (A) '::/0'
2024-03-30T10:55:23.256640501Z 2024/03/30 10:55:23 kid1| WARNING: because of this '::/0' is ignored to keep splay tree searching predictable
2024/03/30 10:55:23 kid1| WARNING: You should probably remove '::/0' from the ACL named 'all'
2024-03-30T10:55:23.267743537Z 2024/03/30 10:55:23 kid1| Current Directory is /
2024-03-30T10:55:23.267762758Z 2024/03/30 10:55:23 kid1| Creating missing swap directories
2024-03-30T10:55:23.267767774Z 2024/03/30 10:55:23 kid1| /var/cache/squid exists
2024/03/30 10:55:23 kid1| Not currently OK to rewrite swap log.
2024/03/30 10:55:23 kid1| storeDirWriteCleanLogs: Operation aborted.
2024-03-30T10:55:23.267896544Z 2024/03/30 10:55:23 kid1| FATAL: Failed to make swap directory /var/cache/squid/00: (13) Permission denied
2024-03-30T10:55:23.267915394Z 2024/03/30 10:55:23 kid1| Squid Cache (Version 5.7): Terminated abnormally.

Additional information

Adding a side-car container running as user root and then manually changing ownership of the (root directory of the) mounted FS let's the container start up correctly.

[avshiliaev]

@cbosdo what do you think? It looks like this operation is failing: https://github.com/avshiliaev/uyuni/blob/master/containers/proxy-squid-image/uyuni-configure.py#L22

[cbosdo]

@cbosdo what do you think? It looks like this operation is failing: https://github.com/avshiliaev/uyuni/blob/master/containers/proxy-squid-image/uyuni-configure.py#L22

That seems to be the problem. I wonder why there is a lost+found folder in a podman volume...

[jmozd]

I wonder why there is a lost+found folder in a podman volume...

because Kubernetes creates a new file system on the PV - and the FS type used happens to have "lost+found"...

[jmozd]

@cbosdo what do you think? It looks like this operation is failing:
https://github.com/avshiliaev/uyuni/blob/master/containers/proxy-squid-image/uyuni-configure.py#L22

That seems to be the problem.

The actual problem is that the container is running as user "squid", and thus has no permission to access/change the content of the (fresh) PV/file system allocated for the cache, as that is belonging to user root.