It's possible to create TailoredProfiles that violates type mutual exclusion

[rhmdnd]

Tailored Profiles allow users to include or exclude rules and build their own custom profiles (to some extent). It's also possible to create a Tailored Profile without extending any existing profiles, allows users to curate profiles from scratch.

Currently, profiles are separated by scan type, where they can be either Platform or Node scans. Platform scans include checking things like kubernetes configuration. Node scan rules are responsible for things like checking file permissions of a file on the actual infrastructure, or nodes.

Tailored Profiles, just like regular profiles, must contain rules of the same type, and the compliance-operator validates this to some extent.

But, it's also possible do the following:

$ cat templates/tailored.yaml
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: ocp4-cis-modified
spec:
  extends: ocp4-cis # this is a profile dedicated to Platform checks
  description: CIS Benchmarks profile
  title: Modified CIS NodeBenchmarks  profile
  enabledRules:
  - name: ocp4-kubelet-enable-protect-kernel-defaults # this is a Node type rule
    rationale: RFE-2714 - This is set by default, no need to adjust kubelet configuration
$ oc apply -f templates/tailored.yaml
tailoredprofile.compliance.openshift.io/ocp4-cis-modified created
$ oc get tp
NAME                STATE
ocp4-cis-modified   READY
$ oc get tp -o yaml
apiVersion: v1
items:
- apiVersion: compliance.openshift.io/v1alpha1
  kind: TailoredProfile
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"ocp4-cis-modified","namespace":"openshift-compliance"},"spec":{"description":"CIS Benchmarks profile","enabledRules":[{"name":"ocp4-kubelet-enable-protect-kernel-defaults","rationale":"RFE-2714 - This is set by default, no need to adjust kubelet configuration"}],"extends":"ocp4-cis","title":"Modified CIS NodeBenchmarks  profile"}}
    creationTimestamp: "2022-07-11T20:25:27Z"
    generation: 1
    name: ocp4-cis-modified
    namespace: openshift-compliance
    ownerReferences:
    - apiVersion: compliance.openshift.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Profile
      name: ocp4-cis
      uid: 1f57bf8c-7045-4f2c-b5fd-6e57ce44b915
    resourceVersion: "219083"
    uid: e3fc0e69-c910-4058-b9ef-21d7c567a723
  spec:
    description: CIS Benchmarks profile
    extends: ocp4-cis
    title: Modified CIS NodeBenchmarks  profile
  status:
    id: xccdf_compliance.openshift.io_profile_ocp4-cis-modified
    outputRef:
      name: ocp4-cis-modified-tp
      namespace: openshift-compliance
    state: READY
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

In this example, I can create a tailored profile that extends a Platform profile, and have it include a Node rule, which should be excluded from all Platform profiles.

This can also lead to confusion if I create a tailored profile, similar to the following:

 $ oc delete tp ocp4-cis-modified
tailoredprofile.compliance.openshift.io "ocp4-cis-modified" deleted
$ cat templates/tailored.yaml
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: ocp4-cis-modified
spec:
  extends: ocp4-cis # this is a profile dedicated to Platform checks
  description: CIS Benchmarks profile
  title: Modified CIS NodeBenchmarks  profile
  disableRules:
    - name: ocp4-kubelet-enable-protect-kernel-defaults # this is a Node type rule
      rationale: RFE-2714 - This is set by default, no need to adjust kubelet configuration
    - name: ocp4-kubeadmin-removed # this is a Platform rule
      rationale: Disable the kubeadmin check
$ oc apply -f templates/tailored.yaml
tailoredprofile.compliance.openshift.io/ocp4-cis-modified created
$ oc get tp -o yaml
apiVersion: v1
items:
- apiVersion: compliance.openshift.io/v1alpha1
  kind: TailoredProfile
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"compliance.openshift.io/v1alpha1","kind":"TailoredProfile","metadata":{"annotations":{},"name":"ocp4-cis-modified","namespace":"openshift-compliance"},"spec":{"description":"CIS Benchmarks profile","disableRules":[{"name":"ocp4-kubelet-enable-protect-kernel-defaults","rationale":"RFE-2714 - This is set by default, no need to adjust kubelet configuration"},{"name":"ocp4-kubeadmin-removed","rationale":"Disable the kubeadmin check"}],"extends":"ocp4-cis","title":"Modified CIS NodeBenchmarks  profile"}}
    creationTimestamp: "2022-07-11T20:45:35Z"
    generation: 1
    name: ocp4-cis-modified
    namespace: openshift-compliance
    ownerReferences:
    - apiVersion: compliance.openshift.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Profile
      name: ocp4-cis
      uid: 1f57bf8c-7045-4f2c-b5fd-6e57ce44b915
    resourceVersion: "227920"
    uid: eb9cb2c4-63eb-46fc-9761-7b43c5e989c0
  spec:
    description: CIS Benchmarks profile
    disableRules:
    - name: ocp4-kubelet-enable-protect-kernel-defaults
      rationale: RFE-2714 - This is set by default, no need to adjust kubelet configuration
    - name: ocp4-kubeadmin-removed
      rationale: Disable the kubeadmin check
    extends: ocp4-cis
    title: Modified CIS NodeBenchmarks  profile
  status:
    errorMessage: 'Rule ''ocp4-kubeadmin-removed'' with type ''Platform'' didn''t
      match expected type: Node'
    outputRef:
      name: ""
      namespace: ""
    state: ERROR
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

In the case above, I get an error but it tells me the violation in question (ocp4-kubeadmin-removed) mismatches the Node scan type. This is because the first rule in the list is actually a Node rule, not a Platform rule. When I first experienced this, I was confused because I assumed (incorrectly) that the extends attribute of the Tailored Profile implied all rules should be of the same scan type.

[xiaojiey]

It seems that it is only the tp validation issue. When you execute scan with the tp, the disabled rules will not enabled:
##################disable node type when extending a platform type profile:
$ oc apply -f -<<EOF
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
name: ocp4-cis-modified
spec:
extends: ocp4-cis # this is a profile dedicated to Platform checks
description: CIS Benchmarks profile
title: Modified CIS NodeBenchmarks profile
disableRules:
- name: ocp4-kubelet-enable-protect-kernel-defaults # this is a Node type rule
rationale: RFE-2714 - This is set by default, no need to adjust kubelet configuration
EOF
tailoredprofile.compliance.openshift.io/ocp4-cis-modified created
$ oc get tp -w
NAME STATE
ocp4-cis-modified READY
^C
$ oc compliance bind -N test tailoredprofile/ocp4-cis-modified
Creating ScanSettingBinding test
$ oc get suite -w
NAME PHASE RESULT
test RUNNING NOT-AVAILABLE
test AGGREGATING NOT-AVAILABLE
test DONE NON-COMPLIANT
test DONE NON-COMPLIANT
^C$ oc get ccr | grep protect-kernel-defaults
$ oc get ccr | grep admin
ocp4-cis-modified-kubeadmin-removed FAIL medium
ocp4-cis-modified-rbac-limit-cluster-admin MANUAL medium

[rhmdnd]

If we implement strict validation on Tailored Profiles, we could break users in an incompatible way.

In the example where a user extends a Platform profile and excludes a Node rule, the profile is created, giving the impression that it works because it doesn't contain the disabled rule because it wasn't actually included in the original scan anyway.

$ oc get profile.compliance ocp4-cis -o json | jq .rules | grep enable-protect
$ oc get profile.compliance ocp4-cis-node -o json | jq .rules | grep enable-protect
  "ocp4-kubelet-enable-protect-kernel-defaults",
  "ocp4-kubelet-enable-protect-kernel-sysctl"

I'd argue the user experience would be better if we informed the user that they have a mismatch between the profile they're using and what they think they're excluding. But, doing that now could be considered a backwards incompatible change.