OCP4: use new assertion formate for OCP CI #11790
Conversation
|
Start a new ephemeral environment with changes proposed in this pull request: Fedora Environment Oracle Linux 8 Environment |
|
🤖 A k8s content image for this PR is available at: Click here to see how to deploy itIf you alread have Compliance Operator deployed: Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and: |
| @@ -0,0 +1,355 @@ | |||
| rule_results: | |||
| e2e-cis-accounts-restrict-service-account-tokens: | |||
There was a problem hiding this comment.
One potential way to simplify this would be to use:
e2e-cis-accounts-restrict-service-account-tokens: MANUAL
e2e-cis-api-server-admission-control-plugin-namespacelifecycle: PASS
e2e-cis-api-server-encryption-provider-cipher: FAIL
Then just have the file name be tests/assertions/ocp/ocp4-cis-4.14-default-results.yml
Then for the remediated results we could do the same thing:
e2e-cis-accounts-restrict-service-account-tokens: MANUAL
e2e-cis-api-server-admission-control-plugin-namespacelifecycle: PASS
e2e-cis-api-server-encryption-provider-cipher: PASS
But have them in a separate file called tests/assertions/ocp/ocp4-cis-4.14-remediated-results.yml
This is just an idea, and we can continue iterating on it in subsequent patches if we decide to do it, but it would simplify the overall file structure, and the look ups we need to perform in the suite.
There was a problem hiding this comment.
I don't have a strong opinion on the file structure, both seem good to me but I slightly lean on the format proposed by Vincent.
There was a problem hiding this comment.
the reason why I wasn't use tests/assertions/ocp/ocp4-cis-4.14-default-results.yml and tests/assertions/ocp/ocp4-cis-4.14-remediated-results.yml was because I wanted to keep some of existing logic handling, also it reduces the total assertion files we need to create.
There was a problem hiding this comment.
That makes sense - let's reuse what we have.
|
This PR is OpenShift-specific and will be leveraged once ComplianceAsCode/ocp4e2e#39 lands. |
Vincent056
force-pushed
the
assertion_new
branch
3 times, most recently
from
a2587e5
to
cd97d2d
Compare
|
/test |
|
@rhmdnd: The
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/test e2e-aws-ocp4-cis Let's kick off a couple tests to see if it picks up the new assertion files. Also, we should make sure we look for the coverage percentages to make sure they remain consistent and we didn't accidentally introduce any drift. |
|
/test e2e-aws-ocp4-cis |
|
/test 4.13-e2e-aws-ocp4-cis |
|
The CIS results look accurate with previous know values. Kicking off some more runs to test all 4.13 assertions. |
Vincent056
force-pushed
the
assertion_new
branch
from
a67e2ff
to
09bf909
Compare
|
/test 4.13-e2e-aws-ocp4-cis |
| @@ -0,0 +1,1447 @@ | |||
| rule_results: | |||
There was a problem hiding this comment.
Looks like the rhcos4-e8 CI is failing on 4.13 because we don't have an assertion file for that scenario, yet:
2024/04/22 16:13:16 E2E-INFO: No global test file or current version test file found, checking for other versioned files in /go/src/github.com/ComplianceAsCode/content/linux_os/guide/auditing/auditd_configure_rules/audit_login_events/audit_rules_login_events_faillock/tests/ocp4
helpers.go:836: E2E-FAILURE: the rule directory tests/ocp4 contains versioned files, but none for 4.13
There was a problem hiding this comment.
I proposed this in a separate patch so that we don't need to respin this one if we don't have to.
|
Looks like the rest of the failures in the 4.13 tests were due to OVN or CNI assertion issues, which I believe @yuumasato has a patch to fix. |
Vincent056
force-pushed
the
assertion_new
branch
2 times, most recently
from
db148f8
to
36b03c5
Compare
|
/test 4.14-e2e-aws-ocp4-cis |
|
@Vincent056: The specified target(s) for
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/test e2e-aws-ocp4-cis |
|
/retest |
|
We might want to consider rebasing this change to pickup @yuumasato's fix for the OVN rules, which should help by reducing the amount of false negatives in the tests. |
Vincent056
force-pushed
the
assertion_new
branch
from
040cbb8
to
d07ed09
Compare
|
/test e2e-aws-ocp4-cis |
Vincent056
force-pushed
the
assertion_new
branch
from
d07ed09
to
5fccbf1
Compare
| default_result: PASS | ||
| result_after_remediation: PASS | ||
| e2e-stig-node-master-file-permissions-ovn-cni-server-sock: | ||
| default_result: PASS |
There was a problem hiding this comment.
This didn't seem to get applied in the latest test run:
helpers.go:829: Result - Name: e2e-stig-node-master-file-permissions-ovn-cni-server-sock - Status: PASS - Severity: medium
helpers.go:836: E2E-FAILURE: The expected result for the e2e-stig-node-master-file-permissions-ovn-cni-server-sock rule didn't match. Expected 'NOT-APPLICABLE', Got 'PASS'
| e2e-stig-node-master-file-permissions-ovn-cni-server-sock: | ||
| default_result: PASS | ||
| result_after_remediation: NOT-APPLICABLE | ||
| e2e-stig-node-master-file-permissions-ovn-db-files: |
There was a problem hiding this comment.
This didn't seem to get applied in the latest test run:
helpers.go:829: Result - Name: e2e-stig-node-master-file-permissions-ovn-db-files - Status: PASS - Severity: medium
helpers.go:836: E2E-FAILURE: The expected result for the e2e-stig-node-master-file-permissions-ovn-db-files rule didn't match. Expected 'NOT-APPLICABLE', Got 'PASS'
| default_result: NOT-APPLICABLE | ||
| result_after_remediation: NOT-APPLICABLE | ||
| e2e-stig-node-worker-file-groupowner-ovn-cni-server-sock: | ||
| default_result: PASS |
There was a problem hiding this comment.
Similar comment here:
helpers.go:829: Result - Name: e2e-stig-node-worker-file-groupowner-ovn-cni-server-sock - Status: PASS - Severity: medium
helpers.go:836: E2E-FAILURE: The expected result for the e2e-stig-node-worker-file-groupowner-ovn-cni-server-sock rule didn't match. Expected 'NOT-APPLICABLE', Got 'PASS'
| default_result: NOT-APPLICABLE | ||
| result_after_remediation: NOT-APPLICABLE | ||
| e2e-stig-node-worker-file-owner-ovn-db-files: | ||
| default_result: PASS |
There was a problem hiding this comment.
Similar comment here:
helpers.go:829: Result - Name: e2e-stig-node-worker-file-owner-ovn-db-files - Status: PASS - Severity: medium
helpers.go:836: E2E-FAILURE: The expected result for the e2e-stig-node-worker-file-owner-ovn-db-files rule didn't match. Expected 'NOT-APPLICABLE', Got 'PASS'
| default_result: PASS | ||
| result_after_remediation: NOT-APPLICABLE | ||
| e2e-stig-node-worker-file-permissions-ovn-db-files: | ||
| default_result: PASS |
There was a problem hiding this comment.
Similar comment here:
helpers.go:829: Result - Name: e2e-stig-node-worker-file-permissions-ovn-db-files - Status: PASS - Severity: medium
helpers.go:836: E2E-FAILURE: The expected result for the e2e-stig-node-worker-file-permissions-ovn-db-files rule didn't match. Expected 'NOT-APPLICABLE', Got 'PASS'
5fccbf1
to
e90536b
Compare
|
/test e2e-aws-ocp4-high-node |
|
/test e2e-aws-ocp4-high-node |
Having a organized way to manage e2e assertion files, we will have all e2e assertion files located at tests/assertions/<platform>/<product-name>-<profile-name>-<ocp-version>.ymlfor example tests/ocp4/assertions/ocp4-cis-4.14.yml
Newlines are causing CI to fail, even for test related changes. This also clarifies the RHCOS assertion file name.
Adding ocp4-cis, ocp4-cis-node, ocp4-e8, ocp4-high, ocp4-high-node, ocp4-moderate, ocp4-moderate-node, ocp4-pci-dss, ocp4-pci-dss-node, ocp4-stig assertion files for OCP 4.13
Adding ocp4-cis, ocp4-cis-node, ocp4-e8, ocp4-high, ocp4-high-node, ocp4-moderate, ocp4-moderate-node, ocp4-pci-dss, ocp4-pci-dss-node, ocp4-stig assertion files for OCP 4.14
Added assertion files for profiles: ocp4-cis ocp4-cis-node ocp4-e8 ocp4-high ocp4-high-node ocp4-moderate ocp4-moderate-node ocp4-pci-dss ocp4-pci-dss-node ocp4-stig ocp4-stig-node rhcos4-e8 rhcos4-high rhcos4-moderate rhcos4-stig
Let's remove result_after_remediation for rules does not have remediation, also remove it for MANUAL result
Added rhcos4-high,rhcos4-moderate, rhcos4-stig assertion fiels
Vincent056
force-pushed
the
assertion_new
branch
from
e90536b
to
700f866
Compare
|
/test e2e-aws-ocp4-high-node |
Fix rule result for file-owner-ovn-db-files, file-permissions-ovn-cni-server-sock and file-permissions-ovn-db-files
Vincent056
force-pushed
the
assertion_new
branch
from
700f866
to
5e77fd4
Compare
|
/test e2e-aws-ocp4-cis |
|
Code Climate has analyzed commit eb04585 and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 100.0% (50% is the threshold). This pull request will bring the total coverage in the repository to 59.4% (0.0% change). View more on Code Climate. |
|
/test e2e-aws-ocp4-cis |
|
@Vincent056: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Having a organized way to manage e2e assertion files, we will have all e2e assertion files located at
tests/assertions/<platform>/<product-name>-<profile-name>-<ocp-version>.ymlfor example:
tests/assertions/ocp4/ocp4-cis-4.14.yml