OCPBUGS-19690: Enable host network to access host sysctls #497

yuumasato · 2024-03-15T18:59:30Z

With HostNetwork: true the sysctl net.core.bpf_jit_harden becomes visible to the scanner container.
Below is a pod that has access to the sysctls:

apiVersion: v1
kind: Pod
metadata:
  name: list-sysctls
spec:
  hostNetwork: true
  volumes:
    - name: host
      hostPath:
        path: /
        type: Directory
  containers:
  - name: list
    command:
      - cat
      - /host/proc/sys/net/ipv6/conf/all/accept_ra
      - /host/proc/sys/net/core/bpf_jit_harden
    image: registry.access.redhat.com/ubi8/ubi-minimal
    securityContext:
      runAsUser: 0
      privileged: true
    volumeMounts:
    - name: host
      mountPath: /host

$ oc create -f list-syctls-proc.yaml
$ oc logs list-sysctls

DNSPolicy: ClusterFirstWithHostNet allows the CO to upload to resultserver, otherwise we get the following error:
{"level":"info","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Trying to upload to resultserver","url":"https://upstream-rhcos4-high-worker-rs:8443/"} {"level":"error","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Failed to upload results to server","error":"Post \"https://upstream-rhcos4-high-worker-rs:8443/\": dial tcp: lookup upstream-rhcos4-high-worker-rs on 10.0.0.2:53: no such host","stacktrace":"github.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:316\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:18\ngithub.com/cenkalti/backoff/v4.doRetryNotify[...]\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:88\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:61\ngithub.com/cenkalti/backoff/v4.RetryNotify\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:49\ngithub.com/cenkalti/backoff/v4.Retry\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:38\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:299\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.handleCompleteSCAPResults.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:390"}
Use the content from Re-enable runtime check on network related sysctls content#11722, to check whether the scanner container can access the sysctls correctly.
oc compliance bind -S default-auto-apply -N test profile/upstream-rhcos4-moderate

EDIT: I have re-tested and DNSPolicy: ClusterFirstWithHostNet indeed solves the no such host error when trying to upload to resultserver.

'hostNetwork: true' grants access to the host's sysctl configurations. 'dnsPolicy: ClusterFirstWithHostnet' is required to access services.

openshift-ci-robot · 2024-03-15T18:59:36Z

@yuumasato: This pull request references Jira Issue OCPBUGS-19690, which is invalid:

expected the bug to target either version "4.16." or "openshift-4.16.", but it targets "4.15.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

With HostNetwork: true the sysctl net.core.bpf_jit_harden becomes visible to the scanner container.
Below is a pod that has access to the sysctls:
apiVersion: v1
kind: Pod
metadata:
 name: list-sysctls
spec:
 hostNetwork: true
 volumes:
   - name: host
     hostPath:
       path: /
       type: Directory
 containers:
 - name: list
   command:
     - cat
     - /host/proc/sys/net/ipv6/conf/all/accept_ra
     - /host/proc/sys/net/core/bpf_jit_harden
   image: registry.access.redhat.com/ubi8/ubi-minimal
   securityContext:
     runAsUser: 0
     privileged: true
   volumeMounts:
   - name: host
     mountPath: /host
$ oc create -f list-syctls-proc.yaml
$ oc logs list-sysctls

But with HostNetwork: true, the CO fails to upload to resultserver.
{"level":"info","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Trying to upload to resultserver","url":"https://upstream-rhcos4-high-worker-rs:8443/"} {"level":"error","ts":"2024-03-15T18:45:57Z","logger":"cmd","msg":"Failed to upload results to server","error":"Post \"https://upstream-rhcos4-high-worker-rs:8443/\": dial tcp: lookup upstream-rhcos4-high-worker-rs on 10.0.0.2:53: no such host","stacktrace":"github.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:316\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:18\ngithub.com/cenkalti/backoff/v4.doRetryNotify[...]\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:88\ngithub.com/cenkalti/backoff/v4.RetryNotifyWithTimer\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:61\ngithub.com/cenkalti/backoff/v4.RetryNotify\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:49\ngithub.com/cenkalti/backoff/v4.Retry\n\tgithub.com/cenkalti/backoff/v4@v4.2.1/retry.go:38\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.uploadToResultServer\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:299\ngithub.com/ComplianceAsCode/compliance-operator/cmd/manager.handleCompleteSCAPResults.func1\n\tgithub.com/ComplianceAsCode/compliance-operator/cmd/manager/resultcollector.go:390"}

DNSPolicy: ClusterFirstWithHostNet is my unsuccessful attempt to fix that.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Vincent056 · 2024-03-15T22:47:51Z

nice finding!

BhargaviGudi · 2024-03-19T05:55:29Z

/hold for test

BhargaviGudi · 2024-04-17T10:58:29Z

Verification passed with 4.16.0-0.nightly-2024-04-16-195622 + compliance-operator with PR #497 code + PR #11722 code

Install CO

$ oc get pb
NAME              CONTENTIMAGE                                 CONTENTFILE         STATUS
ocp4              ghcr.io/complianceascode/k8scontent:latest   ssg-ocp4-ds.xml     VALID
rhcos4            ghcr.io/complianceascode/k8scontent:latest   ssg-rhcos4-ds.xml   VALID
upstream-ocp4     ghcr.io/complianceascode/k8scontent:11722    ssg-ocp4-ds.xml     VALID
upstream-rhcos4   ghcr.io/complianceascode/k8scontent:11722    ssg-rhcos4-ds.xml   VALID

Create custom wrscan
create auto-rem-ss to scan wrscan mcp rule only

$ oc get ss auto-rem-ss -oyaml
apiVersion: compliance.openshift.io/v1alpha1
autoApplyRemediations: true
autoUpdateRemediations: true
kind: ScanSetting
maxRetryOnTimeout: 3
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"compliance.openshift.io/v1alpha1","autoApplyRemediations":true,"autoUpdateRemediations":true,"kind":"ScanSetting","maxRetryOnTimeout":3,"metadata":{"annotations":{},"creationTimestamp":"2023-09-25T02:05:43Z","generation":1,"name":"auto-rem-ss","namespace":"openshift-compliance","resourceVersion":"43973","uid":"29426481-7cd1-48f0-a3cf-934c96f651eb"},"rawResultStorage":{"pvAccessModes":["ReadWriteOnce"],"rotation":5,"size":"2Gi"},"roles":["wrscan"],"scanTolerations":[{"operator":"Exists"}],"schedule":"0 1 * * *","showNotApplicable":false,"strictNodeScan":false,"timeout":"30m"}
  creationTimestamp: "2024-04-17T10:14:11Z"
  generation: 1
  name: auto-rem-ss
  namespace: openshift-compliance
  resourceVersion: "108142"
  uid: b3a50385-baad-43cf-8ac3-2fb3f1c502a6
rawResultStorage:
  pvAccessModes:
  - ReadWriteOnce
  rotation: 5
  size: 2Gi
roles:
- wrscan
scanTolerations:
- operator: Exists
schedule: 0 1 * * *
showNotApplicable: false
strictNodeScan: false
suspend: false
timeout: 30m

Create ssb

$ oc compliance bind -N rhcos4-high-test -S auto-rem-ss profile/upstream-rhcos4-high
Creating ScanSettingBinding rhcos4-high-test
$ oc get scan
NAME                 PHASE   RESULT
upstream-rhcos4-high-wrscan   DONE    NON-COMPLIANT

All the rules with auto-remediations are applied after 3 rounds are rescan.

$ oc compliance rerun-now scansettingbinding rhcos4-high-test
Rerunning scans from 'rhcos4-high-test': upstream-rhcos4-high-wrscan
Re-running scan 'openshift-compliance/upstream-rhcos4-high-wrscan'
$ oc get ccr -l compliance.openshift.io/automated-remediation=,compliance.openshift.io/check-status=FAIL  
No resources found in openshift-compliance namespace.

BhargaviGudi · 2024-04-17T10:58:35Z

/unhold

BhargaviGudi · 2024-04-17T10:58:45Z

/label qe-approved

BhargaviGudi · 2024-04-17T10:59:02Z

/lgtm

openshift-ci · 2024-04-18T08:14:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BhargaviGudi, yuumasato

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [BhargaviGudi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

yuumasato · 2024-04-19T16:15:31Z

@BhargaviGudi Thank you for testing this.

I re-tested again and cannot reproduce the error I had mentioned in PR description.
I was probably doing something wrong before.

yuumasato · 2024-04-19T16:19:59Z

Below are some of the runtime objects collected, they match the static configuration now.

<unix-sys:sysctl_item id="100008715" status="exists">
  <unix-sys:name>net.core.bpf_jit_harden</unix-sys:name>
  <unix-sys:value>2</unix-sys:value>
</unix-sys:sysctl_item>
<ind-sys:textfilecontent_item id="100008714" status="exists">
  <ind-sys:filepath>/etc/sysctl.d/75-sysctl_net_core_bpf_jit_harden.conf</ind-sys:filepath>
  <ind-sys:path>/etc/sysctl.d</ind-sys:path>
  <ind-sys:filename>75-sysctl_net_core_bpf_jit_harden.conf</ind-sys:filename>
  <ind-sys:pattern>^[\s]*net.core.bpf_jit_harden[\s]*=[\s]*(.*)[\s]*$</ind-sys:pattern>
  <ind-sys:instance datatype="int">1</ind-sys:instance>
  <ind-sys:line>^[\s]*net.core.bpf_jit_harden[\s]*=[\s]*(.*)[\s]*$</ind-sys:line>
  <ind-sys:text>net.core.bpf_jit_harden=2</ind-sys:text>
  <ind-sys:subexpression>2</ind-sys:subexpression>
</ind-sys:textfilecontent_item>

<unix-sys:sysctl_item id="100008621" status="exists">
  <unix-sys:name>net.ipv6.conf.default.accept_ra</unix-sys:name>
  <unix-sys:value>0</unix-sys:value>
</unix-sys:sysctl_item>
<ind-sys:textfilecontent_item id="100008620" status="exists">
  <ind-sys:filepath>/etc/sysctl.d/75-sysctl_net_ipv6_conf_default_accept_ra.conf</ind-sys:filepath>
  <ind-sys:path>/etc/sysctl.d</ind-sys:path>
  <ind-sys:filename>75-sysctl_net_ipv6_conf_default_accept_ra.conf</ind-sys:filename>
  <ind-sys:pattern>^[\s]*net.ipv6.conf.default.accept_ra[\s]*=[\s]*(.*)[\s]*$</ind-sys:pattern>
  <ind-sys:instance datatype="int">1</ind-sys:instance>
  <ind-sys:line>^[\s]*net.ipv6.conf.default.accept_ra[\s]*=[\s]*(.*)[\s]*$</ind-sys:line>
  <ind-sys:text>net.ipv6.conf.default.accept_ra=0</ind-sys:text>
  <ind-sys:subexpression>0</ind-sys:subexpression>
</ind-sys:textfilecontent_item>

openshift-ci · 2024-05-10T06:31:37Z

@yuumasato: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-rosa	`a10228d`	link	true	`/test e2e-rosa`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Use the host's network namespace for scans

a10228d

'hostNetwork: true' grants access to the host's sysctl configurations. 'dnsPolicy: ClusterFirstWithHostnet' is required to access services.

openshift-ci-robot added the jira/valid-reference label Mar 15, 2024

openshift-ci bot requested review from BhargaviGudi and mrogers950 March 15, 2024 18:59

openshift-ci-robot added the jira/invalid-bug label Mar 15, 2024

yuumasato mentioned this pull request Mar 15, 2024

Re-enable runtime check on network related sysctls ComplianceAsCode/content#11722

Open

openshift-ci bot added the do-not-merge/hold label Mar 19, 2024

openshift-ci bot removed the do-not-merge/hold label Apr 17, 2024

openshift-ci bot added the qe-approved label Apr 17, 2024

openshift-ci bot assigned BhargaviGudi Apr 17, 2024

openshift-ci bot added the lgtm label Apr 17, 2024

BhargaviGudi approved these changes Apr 18, 2024

View reviewed changes

openshift-ci bot added the approved label Apr 18, 2024

yuumasato requested review from Vincent056 and rhmdnd and removed request for mrogers950 April 19, 2024 16:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-19690: Enable host network to access host sysctls #497

OCPBUGS-19690: Enable host network to access host sysctls #497

yuumasato commented Mar 15, 2024 •

edited

openshift-ci-robot commented Mar 15, 2024

Vincent056 commented Mar 15, 2024

BhargaviGudi commented Mar 19, 2024

BhargaviGudi commented Apr 17, 2024 •

edited by xiaojiey

BhargaviGudi commented Apr 17, 2024

BhargaviGudi commented Apr 17, 2024

BhargaviGudi commented Apr 17, 2024

openshift-ci bot commented Apr 18, 2024

yuumasato commented Apr 19, 2024

yuumasato commented Apr 19, 2024

openshift-ci bot commented May 10, 2024

OCPBUGS-19690: Enable host network to access host sysctls #497

Are you sure you want to change the base?

OCPBUGS-19690: Enable host network to access host sysctls #497

Conversation

yuumasato commented Mar 15, 2024 • edited

openshift-ci-robot commented Mar 15, 2024

Vincent056 commented Mar 15, 2024

BhargaviGudi commented Mar 19, 2024

BhargaviGudi commented Apr 17, 2024 • edited by xiaojiey

BhargaviGudi commented Apr 17, 2024

BhargaviGudi commented Apr 17, 2024

BhargaviGudi commented Apr 17, 2024

openshift-ci bot commented Apr 18, 2024

yuumasato commented Apr 19, 2024

yuumasato commented Apr 19, 2024

openshift-ci bot commented May 10, 2024

yuumasato commented Mar 15, 2024 •

edited

BhargaviGudi commented Apr 17, 2024 •

edited by xiaojiey