Helm install v2.14.0 "validation failed" error when using a template variable with value ""

[daniv-msft]

Hello,

After upgrading from v2.13.1 to v2.14.0, my chart now throws an error on helm install:

Error: validation failed: error validating "": error validating data: unknown object type "nil" in Deployment.spec.template.metadata.annotations.buildID

This seems to be due to the use in deployment.yaml file of a template variable "buildID" that is actually never declared in values.yaml.
Extract from deployment.yaml:

template:
  metadata:
    labels:
      app: {{ template "gateway.name" . }}
      draft: {{ default "draft-app" .Values.draft }}
      release: {{ .Release.Name }}
    annotations:
      buildID: {{ .Values.buildID }}

If I set the buildID variable in values.yaml file to "", I get the same error.
If I set the buildID variable in values.yaml file to any other string, such as "a", then my install works.
If I set "" to buildID in deployment.yaml (buildID: {{ "" }}), I get the same error.
If I set "" directly to buildID in deployment.yaml (buildID: ""), then my install works.

Could you please let me know if this is a known issue, or if I am missing anything here?

Thanks!

---

Output of helm version:
Client: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-25T02:41:57Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform:
AKS

[StephanX]

This bit me today as well.

[fooka03]

Looks like it might be due to this commit:
32d7f1a

TL;DR - Fix manifest validation

Unfortunately, if you have something already deployed with an empty string you can't deploy something that's "fixed" as your already deployed components will fail validation. Your only recourse is to helm delete --purge to get rid of the offending template(s) from history and go forward. Or roll back helm/tiller

[bacongobbler]

As discussed with a community member earlier today, #5576 was the change. Prior to 2.14, Helm silently accepted schema validation errors, but as of 2.14, all manifests are validated, including ones that were previously accepted. The end result being that upgrading to 2.14 causes Tiller to fail manifest validation on charts that were previously accepted, preventing upgrades. Sorry about that!

The mitigation for this is easy: downgrade to 2.13.1 to upgrade for now until a fix is released.

#5643 should fix this as the validation only occurs for new manifests being added to the release, and we'd love to hear if that solves the issues raised in here. If so, we may need to cut a 2.14.1 with the fix.

[bacongobbler]

EDIT: #5576 was the PR that made the change. #5643 is the PR that should fix this. :)

[fooka03]

I can try #5643 tomorrow unless someone beats me to it.

[StephanX]

Wouldn't the prudent course of action to surface validation errors that were 'silently' failing before, with a warning ,that in a future release becomes an error? Or at the least, honor a force flag or some such that allows the user to choose how to handle it?

[daniv-msft]

Thank you everyone for your replies!
I tried downloading the binaries from the latest Helm Canary build but the issue still reproduces. I'm not sure that these binaries correspond to the latest version of master though.
I'm having issues building Helm locally, so I'd be really interested in the outcome of your test @fooka03 !

[bacongobbler]

I'm not sure that these binaries correspond to the latest version of master though.

Check the output of helm version - that should tell you what commit your helm client and tiller are running as. Because the patch in #5643 was a server-side patch, you'll have to ensure that tiller was updated.

[pulpbill]

Same, using k8s 1.8.4 :

`Error: error validating "": error validating data: [ValidationError(Deployment.spec.template.spec.containers[1].ports[0]): unknown field "exec" in io.k8s.api.core.v1.ContainerPort, ValidationError(Deployment.spec.template.spec.containers[1].ports[0]): unknown field "initialDelaySeconds" in io.k8s.api.core.v1.ContainerPort]

Error: error validating "": error validating data: ValidationError(StatefulSet.spec): missing required field "serviceName" in io.k8s.api.apps.v1beta1.StatefulSetSpec Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(StatefulSet.spec): missing required field "serviceName" in io.k8s.api.apps.v1beta1.StatefulSetSpec`

[daniv-msft]

I'm not sure that these binaries correspond to the latest version of master though.

Check the output of helm version - that should tell you what commit your helm client and tiller are running as. Because the patch in #5643 was a server-side patch, you'll have to ensure that tiller was updated.

Thanks @bacongobbler ! Following your comment I upgraded my tiller:

Client: &version.Version{SemVer:"v2.14+unreleased", GitCommit:"9fb19967bab21ecb9748440a99487f2fb0560f63", GitTreeState:"clean"}
Server: &version.Version{SemVer:"canary+unreleased", GitCommit:"9fb19967bab21ecb9748440a99487f2fb0560f63", GitTreeState:"clean"}

However, I'm still getting the exact same error when running helm install. :(
Error: validation failed: error validating "": error validating data: unknown object type "nil" in Deployment.spec.template.metadata.annotations.buildID

The commit corresponds to 9fb1996, so it looks like the issue still reproduces in my case despite the fix.

[SeriousM]

Can we get an ETA for the hotfix? I really would like to avoid patching my server with a self-build helm/tiller. Thank you!

[karuppiah7890]

@daniv-msft Could you try doing this in your template yaml?

template:
  metadata:
    labels:
      app: {{ template "gateway.name" . }}
      draft: {{ default "draft-app" .Values.draft }}
      release: {{ .Release.Name }}
    annotations:
      buildID: {{ .Values.buildID | quote }}

[karuppiah7890]

@SeriousM for now you can rollback to 2.13.1 and wait for 2.14.1 release. I tried the commit 9fb19967 and it works for me

[SeriousM]

@SeriousM for now you can rollback to 2.13.1 and wait for 2.14.1 release. I tried the commit 9fb19967 and it works for me

we have a lot of azure devops release pipelines (30+) and each of them is trying to keep helm at the latest stable build version. I could downgrade for now but once the next build pipeline is started the version would be back on 2.14.0 and I really don't go over all the 30+ to disable the step and enable it later back again. Sorry, but I need to wait for the hotfix.

Is there any ETA on the hotfix?

@daniv-msft Could you try doing this in your template yaml?

template:
  metadata:
    labels:
      app: {{ template "gateway.name" . }}
      draft: {{ default "draft-app" .Values.draft }}
      release: {{ .Release.Name }}
    annotations:
      buildID: {{ .Values.buildID | quote }}

This is my content of the deployment.yaml file that matches the path Deployment.spec.template.metadata.annotations.buildID:

spec:
  template:
    metadata:
      annotations:
        buildID: {{ .Values.buildID }}
      labels:
        app: {{ template "fullname" . }}
        env: {{ .Values.labels.env }}

Do you think a | quote could fix the problem?

[karuppiah7890]

@SeriousM Oh. What error are you getting currently? And I tried the | quote with the master commit, not released version, and it will basically surround the value with double quotes, and it's useful when the value is empty, as the yaml is rendered as

  annotations:
    buildID: ""

If you don't use | quote, it will get rendered as

  annotations:
    buildID: 

which will lead to a validation error that is described in the issue. I verified this by using this dummy chart that I created:

issue-5750.tar.gz

[SeriousM]

@SeriousM Oh. What error are you getting currently?

My error Error: UPGRADE FAILED: error validating "": error validating data: unknown object type "nil" in Deployment.spec.template.metadata.annotations.buildID but I don't see why. First I thought that the buildID was not passed into the cli command from azure devops but since @daniv-msft got the same error I guess it's because of the server validation.

[SeriousM]

@SeriousM Oh. What error are you getting currently?

I tried to modify my deployment.yaml by adding | quote and even remove the metadata/annotations/buildID at all but this didn't helped.

This is the error I got when I removed the buildID annotation:

Error: failed decoding reader into objects: error validating "": error validating data: unknown object type "nil" in Deployment.spec.template.metadata.annotations.buildID

[bacongobbler]

In regards to a 2.14.1 release, we probably won't be able to cut a release until after KubeCon.

[SeriousM]

In regards to a 2.14.1 release, we probably won't be able to cut a release until after KubeCon.

To be sure to get this right, you mean this KubeCon?

[bacongobbler]

KubeCon EU, which is next week, not November. :)

[SeriousM]

KubeCon EU, which is next week, not November. :)

So we can expect a fix in form of v2.14.1 at the 24.5.19 ?
Can I compile it myself somehow?

[karuppiah7890]

@SeriousM The first error you got, which is
Error: UPGRADE FAILED: error validating "": error validating data: unknown object type "nil" in Deployment.spec.template.metadata.annotations.buildID
is due to the issue in the chart template yamls , which you seem to have fixed with the | quote

The next error
Error: failed decoding reader into objects: error validating "": error validating data: unknown object type "nil" in Deployment.spec.template.metadata.annotations.buildID
is due to the bad manifest in the existing release. To avoid this, the tiller should not validate the release manifests and only check the manifests that are given by the user, and that's what the PR #5643 (fix) does and has been merged into master. You can may be use the canary image of tiller to check if it all works for you. If the failing releases are fixed once (by upgrading with proper manifests), then if the pipeline uses 2.14 version, there won't be any validation problems

[SeriousM]

You can may be use the canary image of tiller to check if it all works for you.

How can I deploy this image?

[karuppiah7890]

@SeriousM With any helm client version, you can use

helm init --tiller-namespace <namespace> --upgrade --canary-image

To get the latest helm client (master), you can use this : https://helm.sh/docs/using_helm/#from-canary-builds

[fooka03]

So it looks like #5643 does fix the manifest validation issue:

helm version
Client: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"canary+unreleased", GitCommit:"9fb19967bab21ecb9748440a99487f2fb0560f63", GitTreeState:"clean"}

Release "belligerent-horse" has been upgraded.
LAST DEPLOYED: Fri May 17 10:32:12 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                             DATA  AGE
<redacted>-belligerent-horse  1     181d
<redacted>                      2     181d

==> v1/Pod(related)
NAME                               READY  STATUS       RESTARTS  AGE
<redacted>-belligerent-horse-0  1/1    Running      0         5m16s
<redacted>-belligerent-horse-1  1/1    Terminating  0         17h

==> v1/Service
NAME           TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)   AGE
<redacted>  ClusterIP  None        <none>       8080/TCP  181d

==> v1/StatefulSet
NAME                             READY  AGE
<redacted>-belligerent-horse  2/2    181d

Still have to set any missing validation fields in the "new" templates of course, but it will at least let you deploy over existing releases.

[SeriousM]

@SeriousM With any helm client version, you can use

helm init --tiller-namespace <namespace> --upgrade --canary-image

To get the latest helm client (master), you can use this : https://helm.sh/docs/using_helm/#from-canary-builds

Thank you very much, I will try it out asap

[daniv-msft]

Same thing here: I tested the fix from the latest canary build and it works for us too.

[RonZhang724]

@SeriousM With any helm client version, you can use

helm init --tiller-namespace <namespace> --upgrade --canary-image

To get the latest helm client (master), you can use this : https://helm.sh/docs/using_helm/#from-canary-builds

This worked for me, thanks very much

[pulpbill]

Does anyone knows how to prevent gitlab pipelines to use helm:latest? We are deploying everything via our laptops since gitlab uses 2.14. It's taking us lots of time.

[karuppiah7890]

@pulpbill How do you install or get the helm client in GitLab ? Do you download from releases? or use a docker image? And you want to install 2.13.1 right?

I tried to find a docker image for helm, but couldn't find any official ones. If you want, you could build a docker image by downloading and putting helm binary in it and then you can use that image in gitlab ci config. You can find the url for downloading binaries (all versions) from releases page - https://github.com/helm/helm/releases . And you can do the same (download and install in $PATH) inside your gitlab job too, if you don't want to use docker image and docker runner in gitlab.

[bacongobbler]

Let's try to keep the topic on subject. @pulpbill if you don't mind sending an email to the helm-users mailing list or by asking the gitlab team directly that'd be great; this seems like an issue with gitlab moreso than with Helm, and it doesn't seem related to the issue present here.

[mitchellmaler]

AutoDevops downloads helm when it runs the deploy job if you take a look here: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml#L472
I think if you set the env var HELM_VERSION in the cicd variables it might allow you to override it but am not sure.

[pulpbill]

Thank you @karuppiah7890 and @mitchellmaler Thanks for the tips! I remember I raised and issue to gitlab for autodevops. I will have to wait for the release of 2.4.1, don't have the time right now to build a new pipeline :(

@bacongobbler Sorry for the off-topic!

[bacongobbler]

Helm v2.14.1 has been released: https://github.com/helm/helm/releases/tag/v2.14.1

[RoyRimjhim]

Hello Team, We are facing this issue in the newly released v3 as well but not in the beta version v3.0.0-beta.4. Kindly help with the resolution.

[ahmadalli]

this is still happening on v3.2.4. v3.2.3 works fine though.

[lionelyoung]

this is still happening on v3.2.4. v3.2.3 works fine though.

Happening on 3.2.3 too on Mac

helm version
version.BuildInfo{Version:"v3.2.3", GitCommit:"8f832046e258e2cb800894579b1b3b50c2d83492", GitTreeState:"clean", GoVersion:"go1.13.12"}

[ahmadalli]

I don't have access to the old code, but I did have a real issue on my chart which resulted in and error on v3.3.0 and the error was gone when I fixed it