Kyverno installation on OpenShift 4.9 Seccomp/SCC

[aaronchar]

Kyverno Version

1.7.0

Description

During an upgrade and new cluster deployment, Kyverno 1.7 will not deploy correctly due to an SCC/Seccomp error.

Error creating: pods "kyverno-74b5c88f7f-kr4t6" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/kyverno-pre: Forbidden: seccomp may not be set pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/kyverno: Forbidden: seccomp may not be set provider "anyuid": Forbidden: not usable by user or serviceaccount provider "nonroot":

I believe the issue is related to this PR . With RuntimeDefault enabled it no longer matches any of the default SCC rules. Because none of them have seccomp enabled.

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[welcome]

Thanks for opening your first issue here! Be sure to follow the issue template!

[chipzoller]

This is an issue specific to OpenShift only? Is the recommendation to not define a seccomp profile?
cc @eddycharly

[aaronchar]

@chipzoller - I honestly don't have an answer for that, but one possibility would be to to create an SCC profile that has the required permissions as well as the ability to use the defined seccomp runtime.

I had to revert to the previous releases and I haven't had cycle's to work on determining the minimum privileges required.

[chipzoller]

An SCC is an OpenShift-only construct, though, so is the problem here that one would need to be created in advance in order to support the current change in the Helm chart?

[aaronchar]

Since it links to the service account it would ideally be an option during deployment so it can be generated and linked to the service account before the container is started.

[PendaGTP]

Related to openshift/cluster-kube-apiserver-operator#1325

A possible workaround is to remove the seccomp profile on the kyverno deployment, for example with kustomize :

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
 - kyverno.yaml

patches:
  - patch: |
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: kyverno
        namespace: kyverno
      spec:
        template:
          spec:
            initContainers:
              - name: kyverno-pre
                securityContext:
                  seccompProfile:
                    $patch: delete
            containers:
              - name: kyverno
                securityContext:
                  seccompProfile:
                    $patch: delete

Also possible with Helm chart :

securityContext:
  seccompProfile:

[eddycharly]

Hi, thanks for reporting the issue, I'll look into it tomorrow.

[eddycharly]

Thanks @PendaGTP for the proposed fix.

The default security context in the helm chart includes seccompProfile, the security context is run through a template to remove seccompProfile if the target cluster version is lower than 1.19.

If it is not supported on OpenShift, there's always the possibility to override the default value in the helm values.

TBH, I'm not sure we should change this except adding a note in the docs ? (cc @chipzoller)

[eddycharly]

I don't know much about OpenShift but reading the docs https://docs.openshift.com/container-platform/4.10/security/seccomp-profiles.html#configuring-default-seccomp-profile_configuring-seccomp-profiles it looks like openshift ships with a default seccomp profile.

[chipzoller]

Is the right thing to do here to enable this as a switchable value so seccompProfile can be active by default but deactivated if desired? The thing is we have to be able to pass the PSS restricted profile by default, which has a strict requirement captured here.

[chipzoller]

Kyverno 1.7.x also has a minimum requirement of Kubernetes 1.21 per the compatibility matrix and so the Helm chart should abide by this minimum and set defaults as they are relevant.

[eddycharly]

I can remove the 1.19 support.
seccompProfile is enabled by default but can be disabled by editing the values, not sure adding a switch brings any benefits over simply setting seccompProfile to null.

[PendaGTP]

Another way to solve this issue is to create the scc to allow the seccomp profile (as suggested by @chipzoller), the solution is presented here: https://bugzilla.redhat.com/show_bug.cgi?id=2010564

$ oc get scc restricted -o yaml > restricted-seccomp.yaml

$ vi restricted-seccomp.yaml
kind: SecurityContextConstraints
metadata:
  name: restricted  --> Change to restricted-seccomp
<..snip..>
seccompProfiles:    --> Add
- runtime/default   --> Add

$ oc create -f restricted-seccomp.yaml
securitycontextconstraints.security.openshift.io/restricted-seccomp created

$ oc adm policy add-scc-to-user restricted-seccomp -z kyverno
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:restricted-seccomp added: "kyverno"

However I don't think this should be directly supported by Kyverno, OpenShift users can either :

• override chart values to remove the seccomp profile (Kyverno installation on OpenShift 4.9 Seccomp/SCC #4118 (comment))
• remove the seccomp profile with patch Kustomize (Kyverno installation on OpenShift 4.9 Seccomp/SCC #4118 (comment))
• create a scc and add the role to the Kyverno sa (see above)

[ronmegini]

Related to openshift/cluster-kube-apiserver-operator#1325

A possible workaround is to remove the seccomp profile on the kyverno deployment, for example with kustomize :

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
 - kyverno.yaml

patches:
  - patch: |
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: kyverno
        namespace: kyverno
      spec:
        template:
          spec:
            initContainers:
              - name: kyverno-pre
                securityContext:
                  seccompProfile:
                    $patch: delete
            containers:
              - name: kyverno
                securityContext:
                  seccompProfile:
                    $patch: delete

Also possible with Helm chart :

securityContext:
  seccompProfile:

The issue happens in OpenShift version 4.8.23 too (Kubernetes version 1.21).
This workaround works for me, but I would like to know if removing the secomp won't impact the kyverno operation in some way?

[PendaGTP]

The issue happens in OpenShift version 4.8.23 too (Kubernetes version 1.21). This workaround works for me, but I would like to know if removing the secomp won't impact the kyverno operation in some way?

@ron34353435, as far as I know, this will not impact in any way the operation of Kyverno.

[sabre1041]

@eddycharly @chipzoller OpenShift contains a feature called Security Context Constraints which enforces certain security controls.

By default, end users deploying applications onto OpenShift will use the restricted SCC which not only ensures that pods do not run as root, but other features, such as limiting host level access.

The default properties of the securityContext of the Kyverno deployment violate the restrictions of the restricted SCC (which is part of the error shown in this issue). When running on OpenShift, the recommended configuration in this case is to omit the setting of the securityContext property so that the default security controls of the platform can take over.

When using the Helm chart, install kyverno using the following command:

helm upgrade -i --create-namespace -n kyverno kyverno kyverno/kyverno --set securityContext=null

Once the Helm release has been created, the kyverno pod can be inspected to see that the platform applied the appropriate securityContext controls (values are dependent on the specific environment):

  securityContext:
    fsGroup: 1000660000
    seLinuxOptions:
      level: s0:c26,c5

[chipzoller]

Thanks for weighing in, @sabre1041. It sounds to me that the best course of action here is to add a special note into the Kyverno documentation with guidance on how to install Kyverno in an OpenShift environment but to otherwise not modify the Helm chart. It is important that Kyverno be able to pass the restricted profile of the Pod Security Standards assuming no other mechanisms exist (i.e., like those in most all other distributions, including "vanilla" Kubernetes itself), and one of those controls is explicitly setting seccompProfile.type to either RuntimeDefault or Localhost, the latter of which the chart presently uses by default.

Would documentation suffice as a suitable resolution to this issue?

[dadav]

I'm having the same problem in OpenShift 4.10.20.

I use the following values (via umbrella chart):

---
kyverno:
  replicaCount: 3

  securityContext:
    seccompProfile:

But the deployment still has the

            seccompProfile:
              type: RuntimeDefault

entries, which blocks the deployment.

I had to create a custom SCC, because I couldn't deactivate the seccomp default via helm. I used this:

---
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: restricted-with-seccomp
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities: null
defaultAddCapabilities: null
fsGroup:
  type: MustRunAs
priority: null
readOnlyRootFilesystem: true
requiredDropCapabilities:
  - ALL
  - KILL
  - MKNOD
  - SETUID
  - SETGID
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
seccompProfiles:
  - runtime/default

[PendaGTP]

I'm having the same problem in OpenShift 4.10.20.

I use the following values (via umbrella chart):

---
kyverno:
  replicaCount: 3

  securityContext:
    seccompProfile:

But the deployment still has the

            seccompProfile:
              type: RuntimeDefault

entries, which blocks the deployment.

I had to create a custom SCC, because I couldn't deactivate the seccomp default via helm. I used this:

---
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: restricted-with-seccomp
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities: null
defaultAddCapabilities: null
fsGroup:
  type: MustRunAs
priority: null
readOnlyRootFilesystem: true
requiredDropCapabilities:
  - ALL
  - KILL
  - MKNOD
  - SETUID
  - SETGID
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
seccompProfiles:
  - runtime/default

Which version of Helm do you use? Tested and working in v3.9.0 (check helm/helm#5184).
Maybe you use a subchart? If so, probably related to this issue helm/helm#9136

[dadav]

Which version of Helm do you use? Tested and working in v3.9.0.
Maybe you use a subchart? If so, probably related to this issue helm/helm#9136

I'm using helm v3.8.0+g7ddadb4, which is included in the gitops operator from openshift.
Yes, I'm using a subchart. Thanks for the hint, I'll have a look at the other issue!!

[chipzoller]

Trying to bring this issue to a close, it looks like the best thing we can do as @sabre1041 suggested here, is to add a special instruction into the Kyverno Helm chart installation docs here that the securityContext value should be set to null. Will this be an adequate solution here?

[chipzoller]

Closing this issue and adding a note to the docs.

[joebowbeer]

According to the discussion below, OpenShift 4.11+ is compatible with the RuntimeDefault seccompProfile type.

redhat-openshift-ecosystem/community-operators-prod#1417

 securityContext:
   # Do not use SeccompProfile if your project must work on 
   # old k8s versions < 1.19 and Openshift < 4.11
   seccompProfile:
      type: RuntimeDefault

[chipzoller]

Thanks, @joebowbeer.

Is anyone still watching this issue with OpenShift 4.11+ who can test and confirm this? If so, will update docs.

[sabre1041]

@chipzoller I can test this today

[chipzoller]

Thanks, Andrew

[sabre1041]

@chipzoller @joebowbeer Can confirm that in OpenShift 4.11, the restricted-v2 SCC allows for the successful deployment without modification