Large policies are not consistently applied in whitelisting mode

[pa-federici]

General Information

• Environment description eks 1.28
• Kernel version Linux bash 5.10.192-183.736.amzn2.x86_64 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Orchestration system version in use v1.28.1
• Target issue riproduced on different containers, including on standard images from docker hub like ubuntu:23.04
• Tested with kubearmor/kubearmor and kubearmor/kubearmor-init image versions v1.2.1

To Reproduce

1. Create a simple of a pod manifest with ubuntu latest image with a small script at the start that opens a file in the container. For example:

    args:
    - |
      while true; do
        if cat /etc/ca-certificates.conf >/dev/null 2>&1; then
          echo "The file /etc/ca-certificates.conf has been successfully opened."
        else
          echo "Could not open the file /etc/ca-certificates.conf."
        fi
        sleep 0.5
      done

3. Create a large kubearmor policy (my policy is about 500 lines) that match the pod created in the previous step and as a whitelist allows access to various files and folders in the pod including the one that is opened by the script
   test-policy.json

4. Apply pod manifest

Expected behavior

we expect the pod to be created without problems and that the script, can have access to the file without any problem

Current behavior

very often it happens that the script fails to reach the file because it is blocked by the policy, even if the file is present in the whitelist

[nyrahul]

Following are some of my observations:

1. the test-policy.json contains a lot of overlapping/duplicate rules ... For e.g. path /usr/ is recursively allowed and then there are individual/overlapping/dup rules to allow specific folders/file in /usr/.
2. I tried to remove some duplications and was able to reduce the lines from 525 to 338 ...
3. Just by adding /var/lib/ recursive file rule, I was further able to remove 256 lines ... so the final line count is 168.

PFA the updated policy test-policy-updated.yaml.txt

Hope this helps.

[pa-federici]

Hi @nyrahul, thank you for your response, however the policy I supplied was meant as a mere example for the purpose of reporting the issue. We're having similar problems with a different policy of about ~200 lines, without any overlap.

[bvenreply]

Hello @nyrahul, would you mind clarifying the status of this issue? Can the project spare any effort to fix this at the moment? Is this considered something to be fixed at all?

[nyrahul]

Hello @nyrahul, would you mind clarifying the status of this issue? Can the project spare any effort to fix this at the moment? Is this considered something to be fixed at all?

We pulled it up for triage today.
There needs to be some limitation that we need to put in place on the overall rules that can be applied on the given workload.
Would require some feedback from the community:

1. What are the kind of policies that you are looking at? If samples can be provided that would help.
2. Having too many inidivdual rules on individual files is going to be policy maintenace nightmare ... what is the sweet spot to work on?

It would be great to have some recommendations from folks who have ran into this problem.

[pa-federici]

hello @nyrahul The policy we use is this ide-policy.json which is much smaller than the policy that we have given in the issue. Potentially we could also try to further reduce this policy but we would need to understand under what limit we must go to be sure that everything works properly. Considering that the problem does not appear every time and is not easy to reproduce