Automatic remediation updates fail if ComplianceScan is recreated

[montaguethomas]

Maybe this is due to incorrect usage of the operator and CRDs, but we are hitting an issue where automatic remediation updates won't work.

compliance-operator/pkg/apis/compliance/v1alpha1/compliancesuite_types.go

Lines 80 to 82 in 7f5d1b9

	// Defines whether or not the remediations should be updated automatically.
	// This is done by deleting the "outdated" object from the remediation.
	AutoUpdateRemediations bool `json:"autoUpdateRemediations,omitempty"`

compliance-operator/pkg/apis/compliance/v1alpha1/compliancesuite_types.go

Lines 29 to 33 in 7f5d1b9

	// RemoveOutdatedAnnotation is an annotation that, when set on a ComplianceSuite
	// will automatically remove outdated remediations so the operator will apply
	// only the up-to-date ones. It'll be removed once the outdated remediations have
	// been removed.
	const RemoveOutdatedAnnotation = "compliance.openshift.io/remove-outdated"

This is primarily due to the ownerReferences on the ComplianceRemediation objects created.

apiVersion: compliance.openshift.io/v1alpha1
kind: ComplianceRemediation
metadata:
  ownerReferences:
  - apiVersion: compliance.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ComplianceCheckResult

We've had several times we've needed to delete the ComplianceSuite or ScanSettingBinding objects in order to update settings or resolve an issue with scans failing to complete. By doing this, it triggers the deletion of the owned ComplianceRemediation objects. Because deletion of the ComplianceRemediation does not remove the applied remediation (MachineConfig, KubeletConfig, etc), new scans will not trigger findings and thus won't create new ComplianceRemediation. Additionally, even if the MachineConfig objects were removed, the created files on disk would still exist and are not removed/cleaned up.

Would it be possible to remove the ownerReferences on either the ComplianceRemediation or ComplianceCheckResult objects, thus leaving them behind when a ComplianceScan is deleted. By adding additional labels to these objects with details on which ssg rule and remediation each originate from, it could be possible to then relink the objects to a newly created ComplianceScan object.

[rhmdnd]

I recently hit this issue when writing end-to-end tests that clean up stale compliance check results when a scan is rerun.

I think we need to figure out if we can safely decouple the remediation ownership from compliance check results.