CatalogSource fails to create the Pod in user namespace with PSA 'restricted'

[pgodowski]

Bug Report

With the introduction of the Pod Security Adminssion, the recommended best practice is to enforce the restricted policy of admission.

However, if the user creates the CatalogSource in the namespace running with restricted policy, the CatalogSource Pod fails to be created with the error like:

status:
  message: >-
    couldn't ensure registry server - error ensuring pod: : error creating new
    pod: foobar-: pods "foobar-6ttkb" is forbidden:
    violates PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false
    (container "registry-server" must set
    securityContext.allowPrivilegeEscalation=false), unrestricted capabilities
    (container "registry-server" must set
    securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or
    container "registry-server" must set securityContext.runAsNonRoot=true),
    seccompProfile (pod or container "registry-server" must set
    securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
  reason: RegistryServerError

What did you do?

On the namespace I have the restricted policy defined:

  labels:
    kubernetes.io/metadata.name: test
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.24
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.24
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.24

Created CatalogSource in such namespace (aka private CatalogSource):

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: test
spec:
  displayName: IBM Operator Catalog
  image: 'icr.io/cpopen/ibm-operator-catalog:latest'
  publisher: IBM
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 45m

OLM attempts to create the CatalogSource Pod, but fails to create one due to the error above.
Problem is reproduced on both FBC-based catalog as well as SQLITE-based.

The work-around is to specify .spec.grpcPodConfig.securityContextConfig: restricted, so that the CatalogSource Pod is admitted with the restricted policy.

What did you expect to see?

Expected was that OLM would create the CatalogSource Pod correctly, without the end user having to specify the securityContextConfig explicitly.

What did you see instead? Under which circumstances?

With above reproduction steps, the CatalogSource was failed.

Environment

• operator-lifecycle-manager version: OCP 4.14.5

• Kubernetes version information: v1.27.6+d548052

• Kubernetes cluster kind: ROKS (Red Hat OpenShift on IBM Cloud) 4.14.5

Possible Solution

Have OLM discover whether the CatalogSource Pod should be created with restricted or not automatically - as it used to be doing initially after this PR: #2845 and then behavior changed with this PR #2906 , reverting the default to the legacy policy, vs initially proposed restricted policy, since the catalogs found in the field are were not compatible with restricted.

Perhaps it could do the "trial-and-error" aproach and if the default resulted with the PSA error, then add attributes allowing the restricted policy?

Additional context

Don't want to ask customers to create CatalogSources differently depending whether namespaces are configured with restricted policy of not, but without changing the CatalogSource to explicitly define .spec.grpcPodConfig.securityContextConfig: restricted as this require updating the CatalogSource definitions of the already released products in the field.

[cdjohnson]

We discussed this at today's OLM Dev call and it was mentioned that that operator-framework run bundle has a similar concept today.

@perdasilva Was there a reason why the default .spec.grpcPodConfig.securityContextConfig doesn't try to set the default based on the pod security policy assigned to the namespace as @pgodowski suggests above?

[cdjohnson]

I've added a related ticket to OpenShift Jira:
https://issues.redhat.com/browse/RHIBMCS-190