UPGRADE FAILED: No resource with the name "" found

[devth]

Repro

Create a simple Chart.yaml:

name: upgrade-repro
version: 0.1.0

With a single K8S resource in the templates/ dir:

kind: ConfigMap
apiVersion: v1
metadata:
  name: cm1
data:
  example.property.1: hello

Install the chart:

helm install .
exasperated-op
Last Deployed: Tue Sep 13 12:43:23 2016
Namespace: default
Status: DEPLOYED

Resources:
==> v1/ConfigMap
NAME      DATA      AGE
cm1       1         0s

Verify the release exists:

helm status exasperated-op
Last Deployed: Tue Sep 13 12:43:23 2016
Namespace: default
Status: DEPLOYED

Resources:
==> v1/ConfigMap
NAME      DATA      AGE
cm1       1         1m

Now add a 2nd K8S resource in templates/ dir:

kind: ConfigMap
apiVersion: v1
metadata:
  name: cm2
data:
  example.property.2: hello

Upgrade the chart:

helm upgrade exasperated-op .
Error: UPGRADE FAILED: Looks like there are no changes for cm1

That's weird. Bump the version in Chart.yaml:

name: upgrade-repro
version: 0.2.0

Try upgrade again:

helm upgrade exasperated-op .
Error: UPGRADE FAILED: No resource with the name cm2 found.

Expected

helm upgrade should create the cm2 resource instead of erroring that it doesn't exist.

Edit: to be clear: helm is creating the cm2 ConfigMap, but helm fails regardless.

Current state after performing steps

helm status exasperated-op
Last Deployed: Tue Sep 13 12:43:23 2016
Namespace: default
Status: DEPLOYED

Resources:
==> v1/ConfigMap
NAME      DATA      AGE
cm1       1         6m

kubectl get configmap --namespace default
NAME           DATA      AGE
cm1            1         6m
cm2            1         4m

[elementalvoid]

I'm running into a similar issue where I have a chart with bundled dependencies. If I add a new dependency and run a helm upgrade the result is the same as described. The resources are properly created however helm returns an error.

So, if this is installed: helm install -n my-release

my-thing/
  Chart.yml
  charts/
    depended-upon-thing/

And then a new chart is added as a dependency:

my-thing/
  Chart.yml
  charts/
    depended-upon-thing/
    new-dependency/

When the release is upgraded with: helm upgrade my-release my-thing helm produces the following error:

Error: UPGRADE FAILED: No resource with the name new-dependency found.

[michelleN]

@devth I'm not able to reproduce this issue on master. Are you still seeing this problem? What version of helm/tiller are you running?

Thanks!

[michelleN]

@elementalvoid I was also unable to reproduce the new dependency error on master. Are you still seeing this problem? What version of helm/tiller are you running?

Thank you.

[elementalvoid]

At the time I was on alpha 4. Using alpha 5 and @devth's example I was also unable to reproduce the issue.

[michelleN]

Alright. I'll close this for now. Feel free to file an issue if you see either of these problems again.

Thanks again.

[devth]

@michelleN thanks! Sorry I haven't had time this week to attempt a repro on master. Looking forward to upgrading soon!

[Emmenemoi]

Same for me when moving a hostPath Deployment/Volume spec to PVC.
Bug seems to be when an upgrade Manifest depends on a new one ("missing" in the old one?)
version: 2.7.2

[mmerrill3]

Strange, I am seeing the same behavior trying to upgrade a chart in version 2.7.2 with a new role. Tiller complains that it can't find the role and fails the deployments, even though it really created the role.

[mmerrill3]

My situation was that I had a new resource, and I deployed the new version of the helm chart with the new resource. That deployment failed b/c I fat fingered some yaml. Well, the new objects were created in kubernetes. I fixed the yaml, and ran the upgrade on my chart again, and voila, the error message that the resource is not found appears. I had to go into kubernetes and remove the new resources (in my case a role and rolebinding) that were created by the failed deployment. After that, the helm check to see if the current object exists fails (https://github.com/kubernetes/helm/blob/7432bdd716c4bc34ad95a85a761c7cee50a74ca3/pkg/kube/client.go#L257) will not succeed, and the resources are created again. Seems like a bug, where maybe new resources for a failed chart should be accounted for?

[Zhomart]

Getting similar error while upgrading:

$ helm upgrade --install bunny ./app --namespace=staging --reuse-values --debug
[debug] Created tunnel using local port: '53859'

[debug] SERVER: "127.0.0.1:53859"

Error: UPGRADE FAILED: no ConfigMap with the name "bunny-proxy-config" found

Configmap is created

$ k get configmap
NAME                 DATA      AGE
bunny-proxy-config   1         7m

My configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "proxy.fullname" . }}-config
  labels:
    app: {{ template "proxy.name" . }}
    chart: {{ template "proxy.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  asd: qwe

[benkeil]

We have the same issue.

[Zhomart]

I deleted the whole release and then installed again. Currently it seems to be working.

$ helm del --purge bunny

[sgandon]

I am also having this issue on

Client: &version.Version{SemVer:"v2.8.0", GitCommit:"14af25f1de6832228539259b821949d20069a222", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.0", GitCommit:"14af25f1de6832228539259b821949d20069a222", GitTreeState:"clean"}

[jaredallard]

This happens frequently with our usage of helm and requires a full --purge. That is not a solution.

[benkeil]

It is not applicable if you use CI/CD.
What happens if a upgrade fails and you use rolling update strategy. Must I delete my still working release?

[helgi]

I see the same issue as well when there is a Deployment issue or similar but the secret/cm got created but then Helm loses track of it, refusing to let you do much.

I've seen it, rarely tho, happen even on a non-broken release (i.e. it is seen as have gone through) but I have yet to figure out what could cause that.

[rich-zhang]

We're also able to reproduce this issue (server v2.8.2) when adding resources to existing helm deployments. Having to delete the deployment and redeploy each time a new resource has to be added will be a big problem in production.

[krishicks]

In our case we were adding a configmap to a chart and the chart fails to be upgraded with:

Error: UPGRADE FAILED: no resource with the name "" found

Note: We're using 2.7.2; on later versions this message has changed to include the type of the resource that can't be found.

I believe this happens because when helm is determining what has changed it looks for the new configmap resource in the old release, and fails to find it. See https://github.com/kubernetes/helm/blob/master/pkg/kube/client.go#L276-L280 for the code where this error comes from.

Tiller logs for the failing upgrade:

[tiller] 2018/05/03 19:09:14 preparing update for staging-collector
[storage] 2018/05/03 19:09:14 getting deployed release from "staging-collector" history
[tiller] 2018/05/03 19:10:39 getting history for release staging-collector
[storage] 2018/05/03 19:10:39 getting release history for "staging-collector"
[tiller] 2018/05/03 19:10:41 preparing update for staging-collector
[storage] 2018/05/03 19:10:41 getting deployed release from "staging-collector" history
[storage] 2018/05/03 19:10:42 getting last revision of "staging-collector"
[storage] 2018/05/03 19:10:42 getting release history for "staging-collector"
[tiller] 2018/05/03 19:10:44 rendering collector chart using values
[tiller] 2018/05/03 19:10:44 creating updated release for staging-collector
[storage] 2018/05/03 19:10:44 creating release "staging-collector.v858"
[tiller] 2018/05/03 19:10:44 performing update for staging-collector
[tiller] 2018/05/03 19:10:44 executing 0 pre-upgrade hooks for staging-collector
[tiller] 2018/05/03 19:10:44 hooks complete for pre-upgrade staging-collector
[kube] 2018/05/03 19:10:44 building resources from updated manifest
[kube] 2018/05/03 19:10:44 checking 3 resources for changes
[tiller] 2018/05/03 19:10:44 warning: Upgrade "staging-collector" failed: no resource with the name "collector-config" found 
[storage] 2018/05/03 19:10:44 updating release "staging-collector.v857"
[storage] 2018/05/03 19:10:44 updating release "staging-collector.v858" 

[krishicks]

This problem also arises when changing the name label of a deployed Service, perhaps other things as well.

I'm changing the name of a Service in a release and it fails to upgrade with:

Error: UPGRADE FAILED: no Service with the name "new-service-name" found

[jaredallard]

I'd be willing to create a PR to fix this behavior, but I'd like to know what the intended or suggested way of handling this. Even a CLI flag that allows --force to take precedence would be great.

[robermorales]

Agree on the importance.

This problem can be weird when you cannot simply delete a deployment.

[krishicks]

I found our issue was because of a failed deploy.

Helm doesn't attempt to clean up after a failed deploy, which means things like the new ConfigMap I added above get created but without a reference in the 'prior' deploy. That means when the next deploy occurs, helm finds the resource in k8s and expects it to be referenced in the latest deployed revision (or something; I'm not sure what exact logic it uses to find the 'prior' release) to check what changes there are. It's not in that release, so it cannot find the resource, and fails.

This is mainly an issue when developing a chart as a failed deploy puts k8s in a state helm does not properly track. When I figured out this is what was happening I knew I just needed to delete the ConfigMap from k8s and try the deploy again.

[jaredallard]

@krishicks Yes, this is one way to repro it. A failed deploy + a never created resource (i.e invalid configmap) can also cause this as well I've noticed, which then leads to a unrecoverable state

[distorhead]

Atomic will not resolve the issue

Example chart: https://github.com/distorhead/ex-helm-upgrade-failure

1. Check out master, run deploy.

git clone https://github.com/distorhead/ex-helm-upgrade-failure
cd ex-helm-upgrade-failure
helm upgrade --atomic --install --namespace myns myrelease .

Chart contains 2 deployments -- myserver1 and myserver2:

Release "myrelease" does not exist. Installing it now.
NAME:   myrelease
LAST DEPLOYED: Tue Feb  5 23:48:57 2019
NAMESPACE: myns
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Deployment
NAME       READY  UP-TO-DATE  AVAILABLE  AGE
myserver1  1/1    1           1          5s
myserver2  1/1    1           1          5s

2. Make breaking change. Delete deployment myserver1 from chart and modify deployment myserver2 with user-error (delete image field for example):

git checkout break-atomic
git diff master

diff --git a/templates/deploy.yaml b/templates/deploy.yaml
index 198516e..64be153 100644
--- a/templates/deploy.yaml
+++ b/templates/deploy.yaml
@@ -1,21 +1,5 @@
 apiVersion: apps/v1beta1
 kind: Deployment
-metadata:
-  name: myserver1
-spec:
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        service: myserver1
-    spec:
-      containers:
-      - name: main
-        command: ["/bin/bash", "-c", "while true ; do date ; sleep 1 ; done"]
-        image: ubuntu:16.04
----
-apiVersion: apps/v1beta1
-kind: Deployment
 metadata:
   name: myserver2
 spec:
@@ -28,4 +12,3 @@ spec:
       containers:
       - name: main
         command: ["/bin/bash", "-c", "while true ; do date ; sleep 1 ; done"]
-        image: ubuntu:16.04

3. Run deploy:

git checkout break-atomic
helm upgrade --atomic --install --namespace myns myrelease .

Say hello to our friend again:

UPGRADE FAILED
ROLLING BACK
Error: Deployment.apps "myserver2" is invalid: spec.template.spec.containers[0].image: Required value
Error: no Deployment with the name "myserver1" found

@bacongobbler @thomastaylor312 @jkroepke

[michelleN]

@distorhead what was your expected behavior for this scenario?

[distorhead]

Slightly offtop about rollbacks, but anyway.

For those people, who want to use rollback, but also do not want rollback to occur immediately after deploy as in --atomic for some reasons. Because, for example, there is no way for the user to manually inspect bad cluster state after failure and because --wait flag does not cause helm to log any info about failures in the resources being deployed. Then there is some way: rollback on the next run, before upgrade (more info #3149 (comment))

[ajcann]

To re-iterate the issue as well as the workaround:

When an upgrade that installs new resources fails, the release goes into a FAILED state and stops the upgrade process. The next time you call helm upgrade, Helm does a diff against the last DEPLOYED release. In the last DEPLOYED release, this object did not exist, so it tries to create the new resource, but fails because it already exists. The error message is completely misleading as @arturictus points out.

This can easily be worked around by manually intervening and deleting the resource that the error reports as "not found". Following the example I demonstrated in #4223 (comment):

><> helm fetch --untar https://github.com/helm/helm/files/2103643/foo-0.1.0.tar.gz
><> helm install ./foo/
...
><> vim foo/templates/service.yaml                    # change the service name from "foo" to "foo-bar"
><> kubectl create -f foo/templates/service.yaml      # create the service
service "foo-bar" created
><> helm upgrade $(helm last) ./foo/
Error: UPGRADE FAILED: no Service with the name "foo-bar" found
><> kubectl delete svc foo-bar
service "foo-bar" deleted
><> helm upgrade $(helm last) ./foo/
Release "riotous-echidna" has been upgraded. Happy Helming!
...
><> kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
foo-bar      ClusterIP   10.104.143.52   <none>        80/TCP    3s
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   1h

In other words, deleting resources created during the FAILED release works around the issue.

Thanks for putting this workaround together @bacongobbler - it's essentially what we came to as a process as well. One painful issue here is during complex upgrades many new resources - at times a few dependencies levels deep - may find themselves in this state. I haven't yet found a way to fully enumerate these states in an automatic way leading to situations where one needs to repeatedly fail an upgrade to "search" for all relevant resources. For example, recently a newly added dependency itself had a dependency on a postgresql chart. In order to resolve this issue it was necessary to delete a secret, configmap, service, deployment and pvc - each found the long way 'round.

[bacongobbler]

You could write a plugin similar to helm diff that would enumerate the templates created in the last release. You could even consume pkg/kube directly from Helm. client.Update has some business logic written for helm's resource tracking/deletion which could be reused by fetching the two releases from Tiller and reversing the comparison order. target.Difference(original) should give you a result of all the resources that were introduced since the previous release.

[simonkotwicz]

@bacongobbler What solution would you recommend to take an application that's deployed as part of release A (for example a larger release made up of several applications) and break it out of release A into its own release (or vice versa) without incurring any downtime (the workaround to delete resources would cause some downtime)... Trying to update a resource via a different release results in the error that's described by this Github issue.

[bfalese-navent]

it sound like this new chart its been installed and replaces the old charts even before a sucessfull deploy. Same thing with a failing upgrade --install. It sould not install if the chart is wrong.
Just rollback to previous state on error or update tiller charts on success.

[reneklacan]

This is a process I use to recover from this problem (so far it has worked every time without any incident... but be careful anyway):

1. Run helm list and find out latest revision for affected chart

   NAME        REVISION UPDATED                  STATUS  CHART              NAMESPACE
   fetlife-web 381      Thu Mar 15 19:46:00 2018 FAILED  fetlife-web-0.1.0  default
   

2. Go from there and find latest revision with DEPLOYED state

   kubectl -n kube-system edit cm fetlife-web.v381
   kubectl -n kube-system edit cm fetlife-web.v380
   kubectl -n kube-system edit cm fetlife-web.v379
   kubectl -n kube-system edit cm fetlife-web.v378
   

3. Once you find last DEPLOYED revision, change its state from DEPLOYED to SUPERSEDED and save the file

4. Try to do helm upgrade again, if it's successful then you are done!

5. If you encounter upgrade error like this:

   Error: UPGRADE FAILED: "fetlife-web" has no deployed releases
   

   then edit the status for very last revision from FAILED to DEPLOYED

   kubectl -n kube-system edit cm fetlife-web.v381
   

6. Try to do helm upgrade again, if it fails again just flip the table...

[selslack]

@bacongobbler @michelleN
Is there anything what makes it hard to improve error message for this issue?

I believe error message should state that "there is a conflict because resource wasn't created by helm and manual intervention is required" and not "not found". Only this small change to the error will improve user experience by a good margin.

[michelleN]

@selslack I would very much in favor of improving the error message 👍

[selslack]

@michelleN I've prepared a PR to change the error text: #5460.

[zacharyw]

I'm experiencing this issue and I'm in a situation where I'm not sure how to resolve it.

I tried all the steps listed by @reneklacan here: #1193 (comment)

Unfortunately that didn't work. The only thing that resolves the issue is to delete the resource generating the error message, then helm upgrade again, which will be successful.

However, the next helm upgrade will fail with the same error, and I have to delete the resource again and reupgrade... this isn't sustainable or good.

I have two environments I use helm to deploy to as part of our CI process: a QA and a production environment.

The QA environment had the same issue so I used helm delete --purge, and then helm upgrade - that resolved the issue permanently.

However I can't do this for the production environment - I can't just wipe it out and reupgrade, so currently I'm stuck deleting the resource before each deploy. I'm just lucky it's not an import resource.

[reneklacan]

@zacharyw what error are you facing at the moment? No resource with the name ... or "fetlife-web" has no deployed releases?

Can you share any additional info that would help with debugging this?

Maybe output of kubectl -n kube-system describe cm -l NAME=YOUR_RELEASE_NAME | grep -A1 STATUS= (replace YOUR_RELEASE_NAME)

Feel free to send me an email with more info if you don't want to spam this issue with potentially unrelated data (rene (at) klacan (dot) sk).

[bacongobbler]

Please see #1193 (comment) for a possible diagnosis and workaround, @zacharyw.

[zacharyw]

@reneklacan It's the no resource with the name ... error. In our case we added an ingress, it seemingly worked, but then subsequent upgrades started failing with this error... even though the ingress already existed.

The status of my most recent release (after deleting the offending ingress and allow helm upgrade to recreate it) is DEPLOYED:

STATUS=DEPLOYED
VERSION=197

However, if I were to try to upgrade again, it would fail.

@bacongobbler Unless I'm misunderstanding I think I already am doing the workaround in that comment: I delete the resource and let it get recreated... the issue is I have to do this every time.

[konrad-garus]

@reneklacan in #1193 (comment) saved my life.

It's a disappointment that Helm fails this way. Deleting things in pretty much any environment is far from ideal.

[bfalese-navent]

It would be great if helm updated it's own database when this kind of error appears, and then retry.

[bacongobbler]

I believe that with the --cleanup-on-fail flag enabled, this error case should go away. Closing as resolved via #4871 and #5143.

[bacongobbler]

If there are further issues arising without those flags, please re-open a new issue. Thanks!

[dimmel82]

The issue is closed, by I thought to add a comment about how to deal with the issue without having to delete the helm release or the running deployments.

So, I reproduced the issue with the following steps:

1. Install chart test-chart-failure with a service template.
2. Add a subchart with a service template that has a string (e.g. "test") in the service's port
3. Upgrade the chart. It will fail with error Service in version "v1" cannot be handled as a Service: v1.Service.Spec: v1.ServiceSpec.Ports: []v1.ServicePort: v1.ServicePort.Port: readUint32: unexpected character: ...

I was able to upgrade after correcting the port to a number, without running helm delete, by applying the suggestion at http://centosquestions.com/helm-how-to-delete-bad-deployment:

1. Found the failed revision with helm history test-chart-failure
2. Deleted the config map of the specific revision with kubectl delete cm -n kube-system test-chart-failure.v2
3. Executed helm upgrade with the corrected chart