Helm should preform deep merge on multiple values files

[connormckelvey]

It looks like #1620 was supposed to provide this functionality but I am seeing that a list in the last specified values file completely override the list from the previous file

I have 2 additional values files foo.yaml, and bar.yaml as follows:

foo.yaml

cronjob:
  fileName: foo_job
  schedule: "0 11 * * *"
  env:
    - name: FOO
      value: bar

bar.yaml

cronjob:
  env:
  - name: BAR
    value: foo

cronjob.yaml template

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: {{ template "cronjob.fullname" . }}
  labels:
    app: {{ template "cronjob.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  schedule: {{ .Values.cronjob.schedule | quote }}
  startingDeadlineSeconds: {{ .Values.cronjob.startingDeadlineSeconds }}
  jobTemplate:
    spec:
      template:
        metadata:
          name: {{ template "cronjob.fullname" . }}
        spec:
          restartPolicy: Never
          imagePullSecrets:
          - name: quay-sts
          containers:
          - name: {{ template "cronjob.fullname" . }}
            imagePullPolicy: Always
            image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
            args:
            - npm
            - start
            - {{ .Values.cronjob.fileName }}
{{ if .Values.cronjob.env }}
            env:
{{ toYaml .Values.cronjob.env | indent 12 }}
{{ end }}

When running helm install --dry-run --debug ./cronjob -f values/foo.yaml,values/bar.yaml or when running helm install --dry-run --debug ./cronjob -f values/foo.yaml -f values/bar.yaml

It outputs the following:

actual output

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: intended-jaguar-cronjob
  labels:
    app: cronjob
    chart: cronjob-2.0.8
    release: intended-jaguar
    heritage: Tiller
spec:
  schedule: "0 11 * * *"
  startingDeadlineSeconds: 60
  jobTemplate:
    spec:
      template:
        metadata:
          name: intended-jaguar-cronjob
        spec:
          restartPolicy: Never
          imagePullSecrets:
          - name: quay-sts
          containers:
          - name: intended-jaguar-cronjob
            imagePullPolicy: Always
            image: XXXXXXXX
            args:
            - npm
            - start
            - foo_job

            env:
            - name: BAR
              value: foo

I expect the env lists from values.yaml, foo.yaml, and bar.yaml to have been merged resulting in the following:

expected output

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: intended-jaguar-cronjob
  labels:
    app: cronjob
    chart: cronjob-2.0.8
    release: intended-jaguar
    heritage: Tiller
spec:
  schedule: "0 11 * * *"
  startingDeadlineSeconds: 60
  jobTemplate:
    spec:
      template:
        metadata:
          name: intended-jaguar-cronjob
        spec:
          restartPolicy: Never
          imagePullSecrets:
          - name: quay-sts
          containers:
          - name: intended-jaguar-cronjob
            imagePullPolicy: Always
            image: XXXXXXXX
            args:
            - npm
            - start
            - foo_job

            env:
            # anything from the env list in values.yaml
            # ...
            - name: FOO
              value: bar
            - name: BAR
              value: foo

[connormckelvey]

I think i found the issue and ill try to open a PR

[unguiculus]

If this is implemented, it should be optional, i. e. the merge mode should be specified as a command-line flag. Anything else would be a breaking change and, I think, in most cases not desirable. The problem is that you would not be able to override defaults for lists.

Here's an example: https://github.com/kubernetes/charts/blob/5453fc144f6f9d722c6bb433790f90d7943a4fc3/stable/docker-registry/values.yaml#L19

For those coming from the Java world, here's how Maven solves this for plugin configurations: https://blog.sonatype.com/2011/01/maven-how-to-merging-plugin-configuration-in-complex-projects/

[jascott1]

Hi @connormckelvey

Sorry you did not get the expected results and I agree that its a frustrating limitation. If you are interested in contributing a feature please create a proposal spelling out the implementation which should include a flag as @unguiculus suggested.

[connormckelvey]

Thanks @unguiculus! I see now how this would be considered a breaking change.

The problem is that you would not be able to override defaults for lists.

Are you saying you wouldn't be able to override an entire list (for example overriding a list of 3 items with a list of 1 item)? What I'm proposing would allow you to override individual defaulted items in a list. Giving you more control.

I guess Im not seeing why merging maps is desirable and merging lists is not desirable, except that at the current time some projects expect a certain behavior while others expect another.

@jascott1 Thanks, I will definitely think about the proposal with my team, but what I really needed was a quick fix that I thought could be resolved with a quick PR. At this time we need to focus on our immediate problem.

[katbusch]

Was this functionality ever added?

[mlushpenko]

I also see this as a limitation, especially for projects like prometheus where the whole prometheus configuration is defined in values file. Not being able to merge on sub-keys, I have to copy almost the same config per environment with is a huge repetition of code.

[bacongobbler]

#4806 might be relevant, in which case this should exist for 2.12.0 (not yet released; try out the canary release to test).

[sfitts]

Just ran into this and I'm confused as to why this was closed. As @mlushpenko states this seems like a pretty major limitation for managing any reasonably sized deployment. In our case we would like to be able to combine affinity rules from multiple levels -- providing a default set of rules in the base chart that can be augmented when deployed. However, as it stands we have to repeat the default affinity rules in every values override file. Not even close to DRY.

This is compounded by the fact that it doesn't seem possible to specify default values that will be based on things like the release name which aren't known until the deployment is performed (granted that's unrelated, but it means that overriding affinity essentially has to be done on every deployment).

[bacongobbler]

@sfitts have you taken a look at #4806?

[sfitts]

@bacongobbler I have and I don't see how it applies. AFAICT, it only deals with updates made during upgrade of an already deployed chart and ensuring that existing values aren't clobbered by new ones. Certainly related, but I don't see how it helps avoid the need to repeat information when combining values files during the initial install.

[hwildwood]

We'd really like this feature as well. We have a hierarchy of values files (eg: base.yaml <- qa.yaml) which are passed to helm via -f values/base.yaml,values/qa.yaml. As others have mentioned it would be great if we could merge the qa values into the base values without having to repeat everything.

[bcg62]

I could also use this

[msvticket]

I know this issue is closed, but as a point of reference for those looking here and thinking of resolving the issue I'll supply a comparison with chef.

There this issue is resolved by a fixed set of priorities for values (which they call attributes). Arrays set with the same priority will be merged, while a higher priority value will replace lower priority values.

https://docs.chef.io/attributes.html#about-deep-merge

If implementing something similar in helm I think it would make sense to introduce new versions of --set and --values, say, --overrideSet and --overrideValues for high priority values. As for setting values in a chart another file called, say, overrideValues.yaml could be allowed which sets values at a higher priority than in the values.yaml file.

[svenbs]

We could also use this. +1

[janotav]

I'm a bit confused. The open end of this thread gave me the impression that this merging thing is still missing in helm. After trying it out (we do helm upgrade ... -f basic.yaml -f profile_extends_basic.yaml), it seems to work correctly. At least for me the #4806 seems to have nailed it.

[dubauski]

I'm a bit confused. The open end of this thread gave me the impression that this merging thing is still missing in helm. After trying it out (we do helm upgrade ... -f basic.yaml -f profile_extends_basic.yaml), it seems to work correctly. At least for me the #4806 seems to have nailed it.

@janotav The #4806 addressed deep merge for the upgrade scenario. This issue was about merging values during normal install.

[terrificsoysauce]

Yes. This is feature is needed. Another use case is to specify/merge/override a subset of environment variables across different deployment environments.

[whereisaaron]

This continues to be a problem with helm 2.x, does anyone know if helm 3.x have a solution for (optionally) merging arrays during install?

[rakesh-253]

I too would love to see this functionality!

[tarrall]

For the folks here who need to modify the list of environment variables you're passing into your deployment template, the solution is to define the vars and defaults as a dict rather than as a list!

values.yaml:

env:
  key1: default1
  key2: default2

values-production.yaml:

env:
  key1: prod-value

deployment.yaml:

{{- range $key, $value :=  .Values.env }}
- name: {{ $key }}
  value: {{ $value | quote }}
{{- end }}

Rendered, using values-production:

- name: key1
  value: "prod-value"
- name: key2
  value: "default2"

[jeremych1000]

@tarrall Can you supply the helm commands you used as well for the merge?

I'm trying to have my helm chart contain default values which are put into a configmap, then have a environment-specific file that I merge into the defaults, but it's overriding instead.

I'm using the range $key,$value in my configmap template.

Does it matter if you're using helm2 or helm3 btw?

[rmacian]

puppet with hiera also manages this, giving options of which kind of merge do you want. I am also repeating code because I have several pods that need to mount the exact same file from a configmap and another specific for each pod. The common should be declared in an upper hierarchy and then give the option to merge the values below

[galindro]

@bacongobbler isn't possible to reopen this issue taking in consideration that #4806 doesn't solves this problem because it does deep merge in maps and in an upgrade process. What @connormckelvey mentioned was the ability to do deep merge in arrays. This kind of deep merge should work for install and upgrade situations.

[holmesb]

I'm surprised helm has no solution for this. The workaround @tarrall suggested doesn't scale well into complex lists involving many keys & sub-lists. @connormckelvey please could you reopen?

[aknakshay]

I am using the following script as a workaround:

import yaml
from deepmerge import always_merger

fileA = “tmp.yaml"
fileB = “feature.yaml"

with open(fileA,'r+') as f:
fileAdictionary= yaml.load(f)

with open(fileB,'r+') as f:
fileBdictionary = yaml.load(f)

result = always_merger.merge(fileAdictionary, fileBdictionary)
with open(‘newFile.yaml’,'w+') as f:
yaml.dump(result,f)

[msvticket]

Merging could always be allowed via use of a command line flag just as comment 2 described it (#3486 (comment)). Normally user's values would override defaults but when a command line flag is provided (e.g. --deepMergeInputs) then a deep merge is done. That's how user would handle the need to null out existing values. Is there another complication to this approach?

An all or nothing approach like that is very dangerous when having big and/or many values files. Another problem with this is that you then can't control the behaviour in a chart. I prefer the solution I outlined in #3486 (comment)

[cccs-cat001]

I noticed recently that using toYaml will only use the yaml from the highest priority yaml file instead of doing the deep merge like it does for range.

Helm version:
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

[chf007]

In fact, only list does not support merge. We agree that all list is rewritten as map， (list in k8s generally has a key called name, and this name can be used as the key of the map)

value.yaml before

# value.yaml 

volumeMounts:
    - mountPath: /data/log/
      name: logs

volumes:
    - emptyDir: {}
      name: logs

initContainers: []

envFrom: []

env:
  - name: NODE_ENV
    value: production
  - name: PORT
    value: "3333"
  - name: SOME_SECRET
    valueFrom:
      secretKeyRef:
        key: SOME_SECRET
        name: some-secret
  

value.yaml after

# value.yaml

volumeMounts:
  logs:
    - mountPath: /data/log/
      name: logs

volumes:
  logs:
    - emptyDir: {}
      name: logs

initContainers: {}

envFrom: {}

env:
  NODE_ENV:
    - name: NODE_ENV
      value: production
  PORT:
    - name: PORT
      value: "3333"
  SOME_SECRET:
    - name: SOME_SECRET
      valueFrom:
        secretKeyRef:
          key: SOME_SECRET
          name: some-secret
  

deployment.yaml before

{{- with .Values.env }}
    env:
    {{- toYaml . | nindent 8 }}
{{- end }}

deployment.yaml after

{{- with .Values.env }}
    env:
    {{- range $key, $value := . }}
    {{- toYaml $value | nindent 8 }}
    {{- end }}
{{- end }}

This will support the merge of list.

If you want to delete a configuration

env:
  NODE_ENV: null

[alpozcan]

@tarrall 's solution didn't work for map-type values, such as valueFrom. The below template code also handles this case:

{{- range $ev_key, $ev_value :=  .Values.deployment.env }}
{{- if (typeIs "string" $ev_value) }}
          - name: {{ $ev_key }}
            value: {{ $ev_value | quote }}
{{- else }}
          - name: {{ $ev_key }}
{{ toYaml $ev_value | indent 12 }}
{{- end }}
{{- end }}

[n0n0x]

@alpozcan can you give an example with valueFrom ?

[leonids2005]

It is not clear why this item is closed!!!
I am trying to use @tarrall @alpozcan solution but it does not work for me. Obviously I am doing something wrong :(.

I have 2 files - values.yaml and values-dev.yaml in Helm command but only getting data from values-dev. Help is appreciated.

[Lucas-C]

Note for people from the future that stumble on this:
there is a nice StackOverflow answer detailing a recipe to have overridable env vars in charts
-> https://stackoverflow.com/a/58636928/636849

[jpaulodit]

Posting here in case it is useful for someone.

Iterating upon what others have shared here, I did the following to get it working for value: and valueFrom:.

# values.yaml

env:
  DB_USERNAME:
    name: DB_USERNAME
    value: db_user
  DB_PASSWORD:
    name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-secrets
        key: db-password

# deployment.yaml

    {{- range $env_key, $env_value :=  .Values.env }}
    {{- if (typeIs "string" $env_value.value ) }}
    - name: {{ $env_key }}
      value: {{ $env_value.value | quote }}
    {{- else }}
    - name: {{ $env_key }}
      valueFrom: {{ toYaml $env_value.valueFrom | nindent 16 }}
    {{- end }}
    {{- end }

[dastrobu]

Supporting deep merge is only possible with dictionaries. This breaks ordering and hence dependent environment variables. Since I was working on this issue for a while, I finally decided to write a bit more about it: An Advanced API for Environment Variables in Helm Charts and provide a library chart to solve this issue.

[arash-bizcover]

@connormckelvey could you please reopen this issue.
Deep merge 2 objects/map is such a basic functionality that should be added.

[elghali]

For the folks here who need to modify the list of environment variables you're passing into your deployment template, the solution is to define the vars and defaults as a dict rather than as a list!

values.yaml:

env:
  key1: default1
  key2: default2

values-production.yaml:

env:
  key1: prod-value

deployment.yaml:

{{- range $key, $value :=  .Values.env }}
- name: {{ $key }}
  value: {{ $value | quote }}
{{- end }}

Rendered, using values-production:

- name: key1
  value: "prod-value"
- name: key2
  value: "default2"

does this work in nested dicts:
I have in cronjob.yaml:

        {{- range $name, $val := .Values.initContainers }}
            - image: "{{ $image.repository }}:{{ $image.tag }}"
              imagePullPolicy: {{ $image.pullPolicy }}
              name: "{{ $val.stepName }}"
              env:
              {{- range $envname, $envvalue := $val.env }}
                - name: {{ $envname }}
                  value: {{ quote $envvalue }}
              {{- end }}
          {{- end }}

and

#values.yaml
initContainers:
  - loader-step-1:
    env:
      DESTTABLE: DEV_TABLE
      PARTITION: 0


#values-prd.yaml

initContainers:
  - loader-step-1:
    env:
      DESTTABLE: PRD_TABLE

I'm ending up with

- name: DESTTABLE
  value: "PRD_TABLE"

omitting the second env

[clear-ryan]

please reopen @monotek. I don't think a reasonable solution has been provided to figure this out.

For context, I found this via a search as I have 2 files the define the same pathway,

• default.yaml provides the majority of common annotations to an object

         service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
         service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
         service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
         service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
         service.beta.kubernetes.io/aws-load-balancer-type: nlb
         service.beta.kubernetes.io/aws-load-balancer-internal: 'true'
  

• dev.yaml has the remaining environment specific annotations

       service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <acm_cert_arn>
       service.beta.kubernetes.io/aws-load-balancer-subnets: <list of subnet ids>
  

What I expect is that I should be able to pass in these files and have the values merged for a proper output, but what I see for the Service is

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
  labels:
    ...

[monotek]

@clear-ryan
I'm not a helm maintainer.
I can't repopen this ticket.

[brandon-fryslie]

I am seeing the expected behavior on v3.9.0.

base.yaml:

topDict:
  subDict:
    subSubDict:
      isOverrideWorking: false
      abc: 123

override.yaml:

topDict:
  subDict:
    subSubDict:
      isOverrideWorking: false
      def: 123

override2.yaml:

topDict:
  subDict:
    subSubDict:
      isOverrideWorking: true
      xyz: 123

command: helm template --debug test my-chart/ -f ./base.yaml -f ./override.yaml -f ./override2.yaml

topDict value that is available in chart:

map[subDict:map[subSubDict:map[abc:123 def:123 isOverrideWorking:true xyz:123]]]

This is exactly the expected behavior. Appending lists to each other is not standard behavior for a "deep merge" on a dict (this functionality exists in many languages, not only helm). There are many reasons specified in this thread and elsewhere online for why this behavior is not desirable. It is inconvenient for some cases but causes intractable problems in many other cases. It's also inconsistent with the behavior of other data types. For example, would you expect string values to be concatenated rather than replaced? I don't think that would be logical either.

This issue should not be reopened because it already works as expected.