Helm chart custom config (am I doing something wrong)?

[hangrymuppet]

I am trying to configure Tracee in Kubernetes using the custom config value in the helm chart here here

When I try to do this and apply the helm chart,

❯ git --no-pager diff && helm install tracee .
diff --git a/values.yaml b/values.yaml
index 9b50b0c..e5c86dd 100644
--- a/values.yaml
+++ b/values.yaml
@@ -56,7 +56,9 @@ affinity: {}

 webhook: ""

-traceeConfig: {}
+traceeConfig:
+  log:
+    level: debug

 config:
   blobPerfBufferSize: ""
Error: INSTALLATION FAILED: 1 error occurred:
	* ConfigMap in version "v1" cannot be handled as a ConfigMap: json: cannot unmarshal object into Go struct field ConfigMap.data of type string

I have not been able to apply custom configs via the helm chart without having to make this change to the original.

Is there a way to apply the custom config without the block scaler to the config.yaml? Am I doing something wrong?

Thanks!

[geyslan]

@hangrymuppet thanks for the report. I suppose you suggested a good fix. @josedonizetti could you test this?

[josedonizetti]

@hangrymuppet how are you trying to install, are you changing the values.yaml from the chart directly? I was envisioning two ways of installing, one you can pass a whole file:

eg:

helm upgrade --install tracee deploy/helm/tracee/ --create-namespace --namespace tracee-test1 --set-file traceeConfig=examples/config/global_config.yaml

or passing a specific config, eg:

 helm install tracee deploy/helm/tracee/ --create-namespace --namespace tracee-test2 --set config.log.level=debug

There are some others examples on the documentation https://aquasecurity.github.io/tracee/latest/docs/install/config/kubernetes/

Let me know if this answers you question.

[josedonizetti]

@hangrymuppet hey, does this answer your question?

[hangrymuppet]

@josedonizetti thanks for getting back to me.

Yes, I am changing the values.yaml file directly by patching the traceeConfig value in our CI pipeline. I can see that using the --set-file argument in helm upgrade --install will break with my proposed change. I can easily patch the tracee-config ConfigMap to support my use case but I was hoping to find an upstream fix if possible.

Can I suggest this instead?

diff --git a/deploy/helm/tracee/templates/tracee-config.yaml b/deploy/helm/tracee/templates/tracee-config.yaml
index f21edf3d9..bc214961c 100644
--- a/deploy/helm/tracee/templates/tracee-config.yaml
+++ b/deploy/helm/tracee/templates/tracee-config.yaml
@@ -7,6 +7,9 @@ metadata:
     {{- include "tracee.labels" . | nindent 4 }}
 data:
   {{- if .Values.traceeConfig }}
+  config.yaml: |-
+    {{- toYaml .Values.traceeConfig | nindent 4 }}
+  {{- if .Values.traceeConfigFile }}
   config.yaml:
     {{- toYaml .Values.traceeConfig | nindent 4 }}
   {{- else }}
diff --git a/deploy/helm/tracee/values.yaml b/deploy/helm/tracee/values.yaml
index 7aea107bf..81f44ed18 100644
--- a/deploy/helm/tracee/values.yaml
+++ b/deploy/helm/tracee/values.yaml
@@ -58,6 +58,8 @@ webhook: ""
 
 traceeConfig: {}
 
+traceeConfigFile: {}
+
 config:
   blobPerfBufferSize: ""
   perfBufferSize: 1024

[hangrymuppet]

Alternatively, I really like the way this is handled with the new policy chart here which could be applied to this configmap as well?

Currently I load in my policy and patch the .Values.defaultPolicy to false before installing Tracee.

[josedonizetti]

@hangrymuppet hmm, I like the idea of renaming the traceeConfig to traceeConfigFile, but about values, why not change the values in config?

diff --git a/deploy/helm/tracee/values.yaml b/deploy/helm/tracee/values.yaml
index a6b4049b1..0af2e42c4 100644
--- a/deploy/helm/tracee/values.yaml
+++ b/deploy/helm/tracee/values.yaml
@@ -72,7 +72,7 @@ config:
     type: mem
     size: 512
   log:
-    level: info
+    level: debug
   output:
     format: json
     options:

https://github.com/aquasecurity/tracee/blob/main/deploy/helm/tracee/values.yaml#L75

[hangrymuppet]

@josedonizetti, 👍 on renaming .Values.traceeConfig to .Values.traceeConfigFile.

There are two things I ran into with .Values.config as it is today which burned a bunch of time when I was troubleshooting issues... (but it could just be a me problem rather than a config problem 😄. That is not uncommon 😉 )

1. Not all tracee config is supported by config in values.yaml.

   In k8s for example, I need to add the below to get container enrichment info (which I think should probably be in the default config that is shipped considering that the volumeMounts exist in the pod by default). But regardless of whether it is in default or not, the point remains that not all tracee config is configurable via config in values.

   cri

   cri:
   - runtime:
       name: containerd
       socket: /var/run/containerd/containerd.sock
   - runtime:
       name: docker
       socket: /var/run/docker.sock
   - runtime:
       name: cri-o
       socket: /var/run/crio/crio.sock
   - runtime:
       name: podman
       socket: /var/run/podman/podman.sock

2. The .Values.config defaults are confusing because it is not the same as the documentation always. As an example, I intuitively thought by looking at this documentation page, that adding the below lines to .Values.config would just work and spent a bunch of time wondering what I was doing wrong. I also added the above cri config to the default .Values.confg block and it wouldn't work.

   log

   log:
     level: debug
     file: "/tmp/tracee.log"

   I would never do that in production but this was testing and debugging some issues on a local dev cluster. At that time I had not yet inspected the tracee-config.yaml to see that the .Values.config is not just copied verbatim but each key is templated individually.

   When I saw that the .Values.traceeConfig key however was just copied verbatim into the configmap, that's what I moved to. Then I discovered that it did not work as well, which is what prompted the original question in this discussion to begin with.

I hope this explains my thought process and is useful. I could have been a lot better at trying to do this in the initial post.

Thank you!