Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: UPGRADE FAILED: no resource with the name "anything_goes" found #3275

Closed
zuzzas opened this issue Dec 19, 2017 · 72 comments
Closed

Error: UPGRADE FAILED: no resource with the name "anything_goes" found #3275

zuzzas opened this issue Dec 19, 2017 · 72 comments
Labels
bug Categorizes issue or PR as related to a bug.

Comments

@zuzzas
Copy link
Contributor

zuzzas commented Dec 19, 2017

Hi,

We are constantly hitting a problem that manifests itself with this Error: UPGRADE FAILED: no resource with the name "site-ssl" found, for example. They can appear after any innocuous update to a template.
Could you, please, help me with understanding the problem. What causes those messages to appear?

I've been unsuccessful in triaging the issue further, it may happen anytime, haven't really found a pattern yet.

Perhaps, there is a problem with how we deploy? helm upgrade hmmmmm /tmp/dapp-helm-chart-20171219-20899-1ppm74grrwrerq --set global.namespace=hmm --set global.env=test --set global.erlang_cookie=ODEzMTBlZjc5ZGY5NzQwYTM3ZDkwMzEx --set global.tests=no --set global.selenium_tests=no --namespace hmm --install --timeout 300

Helm: v2.7.2, v2.6.2, Kubernetes: v1.7.6, v1.8.5. I've tried every possible combination of these 4 versions, neither work.

@zuzzas
Copy link
Contributor Author

zuzzas commented Dec 20, 2017

Completely removing release from Helm via helm delete release works, but it is not a viable solution.

Why can't Helm just overwrite whatever is currently installed? Aren't we living in a declarative world with Kubernetes?

@jpoizat
Copy link

jpoizat commented Dec 21, 2017

Just got the same thing... quite new for me at it seems to be a new issue. delete the resource will fix it.
v2.7.2 with Kubernetes 1.7.7.
pretty it worked before...

@ajtrichards
Copy link

I had this problem - it was due to a PersistentVolume that i'd created. To resolve, I deleted the PV and PVC. Ran helm upgrade XXX XXX and it worked fine. Probably still something that should be investigated as the PV did exist.

@jpoizat
Copy link

jpoizat commented Dec 21, 2017

I got the feeling it might be related to bad pv... but then the error is quite misleading !
also some weirds logs from tiller... seems that it is working on 2 version at the same time...

just tried with 2.7.1 with no luck...

[main] 2017/12/21 15:30:48 Starting Tiller v2.7.1 (tls=false)
[main] 2017/12/21 15:30:48 GRPC listening on :44134
[main] 2017/12/21 15:30:48 Probes listening on :44135
[main] 2017/12/21 15:30:48 Storage driver is ConfigMap
[main] 2017/12/21 15:30:48 Max history per release is 0
[tiller] 2017/12/21 15:30:55 preparing update for xxx
[storage] 2017/12/21 15:30:55 getting deployed release from "xxx" history
[tiller] 2017/12/21 15:30:56 copying values from xxx (v65) to new release.
[storage] 2017/12/21 15:30:56 getting last revision of "xxx"
[storage] 2017/12/21 15:30:56 getting release history for "xxx"
[tiller] 2017/12/21 15:30:59 rendering helm-xxx chart using values
2017/12/21 15:30:59 info: manifest "helm-xxx/templates/scheduler-deploy.yaml" is empty. Skipping.
2017/12/21 15:30:59 info: manifest "helm-xxx/templates/recomposer-deploy.yaml" is empty. Skipping.
2017/12/21 15:31:00 info: manifest "helm-xxx/templates/recomposer-pvc.yaml" is empty. Skipping.
2017/12/21 15:31:00 info: manifest "helm-xxx/templates/scheduler-pvc.yaml" is empty. Skipping.
2017/12/21 15:31:00 info: manifest "helm-xxx/templates/scheduler-secret.yaml" is empty. Skipping.
2017/12/21 15:31:00 info: manifest "helm-xxx/templates/recomposer-secret.yaml" is empty. Skipping.
[tiller] 2017/12/21 15:31:09 creating updated release for xxx
[storage] 2017/12/21 15:31:09 creating release "xxx.v80"
[tiller] 2017/12/21 15:31:09 performing update for xxx
[tiller] 2017/12/21 15:31:09 executing 0 pre-upgrade hooks for xxx
[tiller] 2017/12/21 15:31:09 hooks complete for pre-upgrade xxx
[tiller] 2017/12/21 15:31:11 preparing update for xxx
[storage] 2017/12/21 15:31:11 getting deployed release from "xxx" history
[storage] 2017/12/21 15:31:11 getting last revision of "xxx"
[storage] 2017/12/21 15:31:11 getting release history for "xxx"
[tiller] 2017/12/21 15:31:18 rendering helm-xxx chart using values
2017/12/21 15:31:18 info: manifest "helm-xxx/templates/scheduler-secret.yaml" is empty. Skipping.
2017/12/21 15:31:18 info: manifest "helm-xxx/templates/scheduler-pvc.yaml" is empty. Skipping.
2017/12/21 15:31:19 info: manifest "helm-xxx/templates/scheduler-deploy.yaml" is empty. Skipping.
[kube] 2017/12/21 15:31:28 building resources from updated manifest
[tiller] 2017/12/21 15:31:46 creating updated release for xxx
[storage] 2017/12/21 15:31:46 creating release "xxx.v81"
[tiller] 2017/12/21 15:31:47 performing update for xxx
[tiller] 2017/12/21 15:31:47 executing 0 pre-upgrade hooks for xxx
[tiller] 2017/12/21 15:31:47 hooks complete for pre-upgrade xxx
[kube] 2017/12/21 15:31:49 checking 7 resources for changes
[kube] 2017/12/21 15:31:49 Looks like there are no changes for Secret "xxx-helm-xxx-nginx-secret"
[kube] 2017/12/21 15:31:50 Looks like there are no changes for Secret "xxx-application-secret"
[kube] 2017/12/21 15:31:50 Looks like there are no changes for Secret "azure-secret"
[kube] 2017/12/21 15:31:51 Looks like there are no changes for ConfigMap "xxx-helm-xxx-nginx-config"
[kube] 2017/12/21 15:31:51 Looks like there are no changes for ConfigMap "xxx-application-config"
[kube] 2017/12/21 15:31:51 Looks like there are no changes for Service "xxx-application-svc"
[kube] 2017/12/21 15:31:51 Looks like there are no changes for StatefulSet "xxx-application"
[tiller] 2017/12/21 15:31:51 executing 0 post-upgrade hooks for xxx
[tiller] 2017/12/21 15:31:51 hooks complete for post-upgrade xxx
[storage] 2017/12/21 15:31:51 updating release "xxx.v65"
[tiller] 2017/12/21 15:31:51 updating status for updated release for xxx
[storage] 2017/12/21 15:31:51 updating release "xxx.v80"
[kube] 2017/12/21 15:31:57 building resources from updated manifest
[kube] 2017/12/21 15:32:10 checking 11 resources for changes
[kube] 2017/12/21 15:32:10 Looks like there are no changes for Secret "xxx-helm-xxx-nginx-secret"
[tiller] 2017/12/21 15:32:10 warning: Upgrade "xxx" failed: no resource with the name "xxx-recomposer-secret" found
[storage] 2017/12/21 15:32:10 updating release "xxx.v65"
[storage] 2017/12/21 15:32:10 updating release "xxx.v81"

@jpoizat
Copy link

jpoizat commented Dec 21, 2017

seems it get confused at doing to release at the same time...

just reapplied the same config twice...

[tiller] 2017/12/21 15:50:46 preparing update for xxx
[storage] 2017/12/21 15:50:46 getting deployed release from "xxx" history
[storage] 2017/12/21 15:50:46 getting last revision of "xxx"
[storage] 2017/12/21 15:50:46 getting release history for "xxx"
[tiller] 2017/12/21 15:50:50 rendering helm-xxx chart using values
2017/12/21 15:50:50 info: manifest "helm-xxx/templates/scheduler-pvc.yaml" is empty. Skipping.
2017/12/21 15:50:50 info: manifest "helm-xxx/templates/recomposer-deploy.yaml" is empty. Skipping.
2017/12/21 15:50:50 info: manifest "helm-xxx/templates/scheduler-secret.yaml" is empty. Skipping.
2017/12/21 15:50:50 info: manifest "helm-xxx/templates/scheduler-deploy.yaml" is empty. Skipping.
2017/12/21 15:50:50 info: manifest "helm-xxx/templates/recomposer-secret.yaml" is empty. Skipping.
2017/12/21 15:50:50 info: manifest "helm-xxx/templates/recomposer-pvc.yaml" is empty. Skipping.
[tiller] 2017/12/21 15:50:58 creating updated release for xxx
[storage] 2017/12/21 15:50:58 creating release "xxx.v85"
[tiller] 2017/12/21 15:50:59 performing update for xxx
[tiller] 2017/12/21 15:50:59 executing 0 pre-upgrade hooks for xxx
[tiller] 2017/12/21 15:50:59 hooks complete for pre-upgrade xxx
[kube] 2017/12/21 15:51:13 building resources from updated manifest
[kube] 2017/12/21 15:51:22 checking 7 resources for changes
[kube] 2017/12/21 15:51:22 Looks like there are no changes for Secret "xxx-helm-xxx-nginx-secret"
[kube] 2017/12/21 15:51:23 Looks like there are no changes for Secret "xxx-application-secret"
[kube] 2017/12/21 15:51:23 Looks like there are no changes for Secret "azure-secret"
[kube] 2017/12/21 15:51:23 Looks like there are no changes for ConfigMap "xxx-helm-xxx-nginx-config"
[kube] 2017/12/21 15:51:23 Looks like there are no changes for ConfigMap "xxx-application-config"
[kube] 2017/12/21 15:51:24 Looks like there are no changes for Service "xxx-application-svc"
[kube] 2017/12/21 15:51:24 Deleting "xxx-recomposer-secret" in xxx...
[kube] 2017/12/21 15:51:24 Failed to delete "xxx-recomposer-secret", err: secrets "xxx-recomposer-secret" not found
[kube] 2017/12/21 15:51:24 Deleting "xxx-recomposer-config" in xxx...
[kube] 2017/12/21 15:51:24 Failed to delete "xxx-recomposer-config", err: configmaps "xxx-recomposer-config" not found
[kube] 2017/12/21 15:51:24 Deleting "xxx-recomposer-pv" in ...
[kube] 2017/12/21 15:51:24 Failed to delete "xxx-recomposer-pv", err: persistentvolumes "xxx-recomposer-pv" not found
[kube] 2017/12/21 15:51:24 Deleting "xxx-recomposer-pvc" in xxx...
[kube] 2017/12/21 15:51:24 Failed to delete "xxx-recomposer-pvc", err: persistentvolumeclaims "xxx-recomposer-pvc" not found
[kube] 2017/12/21 15:51:24 Deleting "xxx-recomposer" in xxx...
[kube] 2017/12/21 15:51:24 Using reaper for deleting "xxx-recomposer"
[kube] 2017/12/21 15:51:24 Failed to delete "xxx-recomposer", err: deployments.extensions "xxx-recomposer" not found
[tiller] 2017/12/21 15:51:24 executing 0 post-upgrade hooks for xxx
[tiller] 2017/12/21 15:51:24 hooks complete for post-upgrade xxx
[storage] 2017/12/21 15:51:24 updating release "xxx.v68"
[tiller] 2017/12/21 15:51:24 updating status for updated release for xxx
[storage] 2017/12/21 15:51:24 updating release "xxx.v85"
[storage] 2017/12/21 15:51:25 getting last revision of "xxx"
[storage] 2017/12/21 15:51:25 getting release history for "xxx"
[kube] 2017/12/21 15:51:38 Doing get for Secret: "xxx-helm-xxx-nginx-secret"
[kube] 2017/12/21 15:51:39 get relation pod of object: xxx/Secret/xxx-helm-xxx-nginx-secret
[kube] 2017/12/21 15:51:39 Doing get for Secret: "xxx-application-secret"
[kube] 2017/12/21 15:51:39 get relation pod of object: xxx/Secret/xxx-application-secret
[kube] 2017/12/21 15:51:39 Doing get for Secret: "azure-secret"
[kube] 2017/12/21 15:51:39 get relation pod of object: xxx/Secret/azure-secret
[kube] 2017/12/21 15:51:39 Doing get for ConfigMap: "xxx-helm-xxx-nginx-config"
[kube] 2017/12/21 15:51:39 get relation pod of object: xxx/ConfigMap/xxx-helm-xxx-nginx-config
[kube] 2017/12/21 15:51:39 Doing get for ConfigMap: "xxx-application-config"
[kube] 2017/12/21 15:51:39 get relation pod of object: xxx/ConfigMap/xxx-application-config
[kube] 2017/12/21 15:51:39 Doing get for Service: "xxx-application-svc"
[kube] 2017/12/21 15:51:39 get relation pod of object: xxx/Service/xxx-application-svc
[kube] 2017/12/21 15:51:39 Doing get for StatefulSet: "xxx-application"
[kube] 2017/12/21 15:51:39 get relation pod of object: xxx/StatefulSet/xxx-application

@jpoizat
Copy link

jpoizat commented Dec 21, 2017

might be related to #2941

@jpoizat
Copy link

jpoizat commented Dec 21, 2017

a said in the other thread, one of the way to fix the issue was to delete the buggy configmaps ... seems to do it for me currently...

@zuzzas
Copy link
Contributor Author

zuzzas commented Dec 21, 2017

That is all fine and dandy. Until that time, when you have to delete something critical from a production namespace. Which, coincidentally, happened to me just now. :c

@jubel-han
Copy link

I've faced the issue as well when we upgrade an release if there are multiple DEPLOYED status of this release, Have to fix it with delete those corresponding configmaps.

@amritb
Copy link

amritb commented Jan 16, 2018

Same problem. Everything was just fine yesterday and I did multiple upgrades. Today I just added a new yaml with service and deployment block separated with --- and the upgrade failed.

The interesting thing is, helm created the service and then complained about it (and didn't do the deployment).
I commented out the service and just ran upgrade with the deployment block - it worked. However, helm didn't delete the service - which it should have as it's removed from the yaml file.

Update: I manually deleted the service, uncommented it from the yaml and ran the upgrade - this time it worked like a charm!

@awwithro
Copy link

awwithro commented Jan 18, 2018

I was having this exact error. It looks like the issue is related to templates with multiple API objects similar to what @amritb saw. In my case, I had a template that had multiple API objects that could be toggled on and off similar to:

{{ if .Values.enabled }}
---
...

Breaking that into its own template file and cleaning up the orphaned objects that helm created and forgot about resolved the issue for me. It sounds like there is a bug in how helm gets previous config if the number of objects per template changes between releases.

@shz
Copy link

shz commented Jan 19, 2018

Adding another datapoint: I appear to be having the exact same issue as @awwithro. We're using a jinja loop to create multiple cronjobs via a template, and when a new upgrade caused this loop to fill in an additional cronjob, we ran into the bug. Seemed to trigger #2941 as well (or possibly one bug causes the other), and deleting the zombie configmaps fixes it.

@nick4fake
Copy link

Just trapped into this even without using any configmaps

@lsthornt
Copy link

Some extra color for anyone who may be stuck:
I was running into this when introducing new subcharts and objects to my release. I was able to solve by checking every object type that was being added, and deleting any existing objects that would cause a naming collision.

This seems to be in line with others' evidence that deletion is the only way to solve right now 😕

@quicksnap
Copy link

Also running across this =\

@quicksnap
Copy link

I also needed to delete affected resources. Not good for a production environment =_(

@jan-g
Copy link

jan-g commented Jan 31, 2018

I'm seeing something I think is similar. The problem appears to be that a helm upgrade does not --reuse-values from the previous deploy. If I specify the same set of values on the command-line as the initial installation did, then helm upgrade works. Dunno if this helps (or anyone else can confirm this).

@ryanlovett
Copy link

Like @amritb, after I manually deleted the object that helm initially failed on, it succeeded after the next upgrade. I did not experience #2941.

@Zhomart
Copy link

Zhomart commented Feb 1, 2018

The same problem using helm 2.8.0. Kubernetes versions client=v1.8.6 and server=v1.8.5-gke.0.

$ helm upgrade bunny ./app --debug
[debug] Created tunnel using local port: '54274'

[debug] SERVER: "127.0.0.1:54274"

Error: UPGRADE FAILED: no ConfigMap with the name "bunny-proxy-config" found

But the configmap exists in $ kubectl get configmap. If I manually delete the configmap, it works, but next time it fails again.

Here is the configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "proxy.fullname" . }}-config
  # namespace: {{ .Release.Namespace }} # I've tried adding and removing it
  labels: # labels are the same as labels from $ kubectl describe configmap bunny-proxy-config
    app: {{ template "proxy.name" . }}
    chart: {{ template "proxy.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  asd: qwe

@Zhomart
Copy link

Zhomart commented Feb 1, 2018

I deleted release and re-installed again. Also, I was using api version extensions/v1beta for deployment, I've changed to apiVersion: apps/v1beta2, I don't know if this helped or not.
Also currently I'm running tiller locally.

Right now seems like everything is working.

@nick4fake
Copy link

This is really easy to reproduce, happens if there is an error in manifest.

Like we have resource1 and resource2, resource2 depends on first. When we upgrade release, resource1 is created (eg PV & PVC), but resource2 fails. After this only deletion of resource1 helps, as helm always reports a problem on upgrade (PersistentVolume with name ... not found)

@bviolier
Copy link

bviolier commented Feb 5, 2018

We had the same issue (the resource that got us was Secrets). Removing the new secrets and re-deploying fixed it.

Do note that because of the failures, we now have 11 different releases when we do helm list, 10 FAILED ones and 1 DEPLOYED. That's not expected, right? Same issue as here it seems: #2941

yuvipanda added a commit to yuvipanda/zero-to-jupyterhub-k8s that referenced this issue Feb 6, 2018
helm/helm#3275 has caused
serious amounts of downtime in the last week or so for
various deployments at Berkeley, so let's recommend people use
the last known version that did not have these issues.
@yuvipanda
Copy link
Contributor

This has pretty much made helm unusable for regular production deploys for us :( We're currently investigating doing things like passing --dry-run to helm and piping it to kubectl apply... Since this seems to affect only a subset of users, am unsure what it is that we are doing wrong :(

@binoculars
Copy link
Contributor

After tailing the tiller logs, I found that tiller was trying to update an old release at the same time:

[storage] 2018/02/14 18:25:40 updating release "s2osf.v10"
[storage] 2018/02/14 18:25:40 updating release "s2osf.v44"

Deleting the old configmap for s2osf.v10 and then upgrading worked.

Client: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}

@rafagonc
Copy link

I am hitting this issue too. I tried adding subchart with a deployment in my chart, it succeeded when upgraded with helm upgrade chart chart-1.0.1.tgz just for the first time, after that when I tried helm upgrade chart chart-1.0.1.tgz it failed with the error Error: UPGRADE FAILED: no Deployment with name "subchart-deployment" found

Client: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}

The helm tiller logs just logs the same error. Anyone experiencing this too?

jianghang8421 pushed a commit to jianghang8421/helm that referenced this issue Feb 17, 2019
* add test for rolling back from a FAILED deployment

* Update naming of release variables

Use same naming as the rest of the file.

* Update rollback test

- Add logging
- Verify other release names not changed

* fix(tiller): Supersede multiple deployments

There are cases when multiple revisions of a release has been
marked with DEPLOYED status. This makes sure any previous deployment
will be set to SUPERSEDED when doing rollbacks.

Closes helm#2941 helm#3513 helm#3275
@pq-dong
Copy link

pq-dong commented Feb 20, 2019

Same problem. Everything was just fine yesterday and I did multiple upgrades. Today I just added a new yaml with service and deployment block separated with --- and the upgrade failed.

The interesting thing is, helm created the service and then complained about it (and didn't do the deployment).
I commented out the service and just ran upgrade with the deployment block - it worked. However, helm didn't delete the service - which it should have as it's removed from the yaml file.

Update: I manually deleted the service, uncommented it from the yaml and ran the upgrade - this time it worked like a charm!

hi,i also has this problem, and i can not solve it , can you would like give me some prompts.

@bacongobbler
Copy link
Member

see #1193 (comment).

@thedumbtechguy
Copy link

Just confirming I am witnessing the same issue and the cause also is as earlier indicated.

Added a new secret, referenced it in a volume (invalid syntax). Upgrade failed, subsequent upgrades failed with the error as above.

Listing secrets showed it had been created. Manually deleted secret and the upgrade went through successfully.

@tdavis
Copy link

tdavis commented Mar 15, 2019

Same, @thedumbtechguy. I run into this issue routinely. It's especially fun when Helm decides you need to delete all your secrets, configmaps, roles, etc. Upgrading becomes a game of wack-a-mole with an ever-increasing list of arguments to kubectl delete. I should have thrown in the towel on this sisyphean task months ago, but it's too late for that now. Sure hope this and the dozens of similar issues can be fixed!

@thedumbtechguy
Copy link

thedumbtechguy commented Mar 16, 2019 via email

@geosword
Copy link

I experienced the same with Helm v2.10. I already had a chart deployed, added another configMap to the chart. It reported that the deployment failed because it couldn't find configMap "blah". I did

helm upgrade <NAME> chart --debug --dryrun 

To verify the configMap was indeed being rendered, it was. Checked the configMaps in the cluster, and found it there. Deleted the blah configMap, re-ran the upgrade, it worked.

@bacongobbler
Copy link
Member

#5460 should better clarify the error message going forward.

@geosword
Copy link

Fair point.

$ helm upgrade linting-unicorn testrail                                                                                                                                                                                        
Error: UPGRADE FAILED: no ConfigMap with the name "linting-unicorn-testrail-php-config" found

Keep up the good work helm team.

@fernandrone
Copy link

In case this is a big deal to anyone else, thought I'd point out #4871 should fix these issues.

Note that it appears it still hasn't been approved by the Helm team. Plus there were some concerns about the automatic deletion resources. Just mentioning it in case anyone wants to build it from source and give it a try.

@dnetguru
Copy link

Having the same issue and only workaround seems to be helm delete --purge release and install again!

@kvangrae
Copy link

I ran into the same issue. @fbcbarbosa it looks like it was merged 2 weeks ago. It should hopefully be a part of the next release 2.14.0.

@majewsky
Copy link

Having the same issue and only workaround seems to be helm delete --purge release and install again!

A less destructive option is doing a helm rollback to the /current/ version (i.e. by 0 steps). I cannot guarantee success, but for us so far, it has always unwedged things successfully.

@CDingerdis
Copy link

Is there an idea if this is going to be in the next release, and if it does when it is coming?

@bacongobbler
Copy link
Member

#5460 was merged 2 months ago, which means it should be in helm 2.14.0.

@ishallbethat
Copy link

I fixed the issue by

  1. delete those resources that complained by "helm upgrade". (It says Not found but actually it can be found). Don't delete the whole release, otherwise if in production you will be screwed completely.
  2. redo helm upgrade. Now this time it should be "Happy Helming" shows up. :)

@Benbentwo
Copy link

We ran into this issue in PROD, when a requirement to our umbrella helm chart added a configmap based on a conditional. For us the work around fix was to

helm rollback <some revision that's acceptable>
helm upgrade <desired version>

@tobypeschel
Copy link

For us, a simple rollback to the current revision has always worked:

helm ls
helm rollback <NAME> <current REVISION>

@ypadlyak
Copy link

@tobypeschel do you have idea how your fix works?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests