helm upgrade --install no longer works

[kbrinnehl]

As of helm v2.7.1, after updating tiller, running helm upgrade --install flag no longer works. The following error is displayed: Error: UPGRADE FAILED: "${RELEASE}" has no deployed releases. Downgrading to v2.7.0 or v2.6.2 does not produce the error.

[tcolgate]

I thought I was experiencing the same problem, but it turned out I just had an old delete (but not purged), release hanging around. check helm list -a , and if your release is there, helm delete --purge releasename. helm upgrade -i is working successfully on 2.7.2 for me.

[bacongobbler]

This is a side-effect of fixing issues around upgrading releases that were in a bad state. #3097 was the PR that fixed this issue. Is there an edge case here that we failed to catch?

Check helm list -a as @tcolgate mentioned, perhaps also explaining how to reproduce it would also be helpful to determine if it's an uncaught edge case or a bug.

[TD-4242]

Also having the same problem, along with duplicate release names:

$>helm ls -a|grep ingress
nginx-ingress            	9       	Thu Nov 30 11:33:06 2017	FAILED  	nginx-ingress-0.8.2        	kube-ingress
nginx-ingress            	11      	Thu Nov 30 11:37:58 2017	FAILED  	nginx-ingress-0.8.2        	kube-ingress
nginx-ingress            	12      	Thu Nov 30 11:38:50 2017	FAILED  	nginx-ingress-0.8.2        	kube-ingress
nginx-ingress            	8       	Thu Nov 30 11:31:27 2017	FAILED  	nginx-ingress-0.8.2        	kube-ingress
nginx-ingress            	10      	Thu Nov 30 11:33:53 2017	FAILED  	nginx-ingress-0.8.2        	kube-ingress
$>helm diff nginx-ingress ./nginx-ingress
Error: "nginx-ingress" has no deployed releases

[bacongobbler]

When you were upgrading, what message was displayed?

[TD-4242]

same error as the diff above, but an install would say it was already installed.

[bacongobbler]

I mean in the previous upgrade attempts that ended up in a FAILED status. I want to know how we get into the situation where all releases are in a failed state.

[TD-4242]

Ohh, the duplicate release name deployments? That I'm not sure, I get it quite often. Sometimes they are all in a DEPLOYED state, sometimes a mix of FAILED and DEPLOYED. We use a CI/CD Jenkins server that constantly updates every PR merge so we do several helm upgrade's a day, typically only having a new container tag. Usually the duplicates are just annoying when looking at whats deployed, this was the first time we had a hard issue with them, and normally we don't upgrade the ingress controller as we were in this case.

[bcorijn]

I seem to have ended up with something similar, I see a few duplicates in my releases lists:

$ helm ls
NAME                      REVISION    UPDATED                     STATUS      CHART                           NAMESPACE
.....
front-prod                180         Tue Dec  5 17:28:22 2017    DEPLOYED    front-1                         prod
front-prod                90          Wed Sep 13 14:36:06 2017    DEPLOYED    front-1                         prod 
...

All of them seem to be in a DEPLOYED state, but it could well be that a previous upgrade failed at some point, as I have hit that bug several times. I am still on K8S 1.7, so have not updated to helm 2.7 yet.

[s4nch3z]

Same issue, can't upgrade over FAILED deploy

[aelbarkani]

Same here using 2.7.2. The first attempt of a release was failed. Then when I tried an upgrade --install I've got the error "Error: UPGRADE FAILED: "${RELEASE}" has no deployed releases".

[winjer]

Same problem here with 2.7.2, helm upgrade --install fails with:

Error: UPGRADE FAILED: "APPNAME" has no deployed releases

[winjer]

If the release is entirely purged with helm del --purge APPNAME then a subsequent upgrade --install works ok.

[prein]

I'm experiencing the same problem. Combined with #3134 that leaves no option for automated idempotent deployments without some scripting to workaround.

@winjer just tried deleting with --purge and for me it didn't work although the error changed
/ # helm upgrade foo /charts/foo/ -i --wait
Error: UPGRADE FAILED: "foo" has no deployed releases
/ # helm delete --purge foo
release "foo" deleted
/ # helm upgrade foo /charts/foo/ -i --wait
Release "foo" does not exist. Installing it now.
Error: release foo failed: deployments.extensions "foo-foo-some-service-name" already exists

[tcolgate]

@prein This is because you have a service that is not "owner" by helm, but already exists in the cluster. The behaviour you are experiencing seems correct to me. The deploy cannot succeed because helm would have to "take ownership" of an API object that it did not own before.

It does make sense to be able to upgrade a FAILED release, if the new manifest is actually correct and doesn't content with any other resources in the cluster.

[pierreozoux]

I'm also seeing this behavior on a release called content:

helm upgrade --install --wait --timeout 300 -f ./helm/env/staging.yaml --set image.tag=xxx --namespace=content content ./helm/content
Error: UPGRADE FAILED: no resource with the name "content-content" found
helm list | grep content
content                        	60      	Mon Dec 25 06:02:38 2017	DEPLOYED	content-0.1.0                	content           
content                        	12      	Tue Oct 10 00:02:24 2017	DEPLOYED	content-0.1.0                	content           
content                        	37      	Tue Dec 12 00:42:42 2017	DEPLOYED	content-0.1.0                	content           
content                        	4       	Sun Oct  8 05:58:44 2017	DEPLOYED	k8s-0.1.0                    	content           
content                        	1       	Sat Oct  7 22:29:24 2017	DEPLOYED	k8s-0.1.0                    	content           
content                        	61      	Mon Jan  1 06:39:21 2018	FAILED  	content-0.1.0                	content           
content                        	62      	Mon Jan  1 06:50:41 2018	FAILED  	content-0.1.0                	content           
content                        	63      	Tue Jan  2 17:05:22 2018	FAILED  	content-0.1.0                	content           

I will have to delete this to be able to continue to deploy, let me know if there is anything I can do to help debug this.
(I think we should rename the issue, as it is more about the duplicates?)
(we also run 2.7.2)

[pierreozoux]

I actually have another duplicate release on my cluster, if you have any command for me to run to help debug that? Let me know!

[rcorre]

just upgraded to tiller 2.7.2 and we're seeing the same thing. helm delete RELEASE_NAME followed by helm upgrade RELEASE_NAME . fails with Error: UPGRADE FAILED: "RELEASE_NAME" has no deployed releases. upgrade is the intended way to restore a deleted (but not purged) release, correct?

[rcorre]

Looks like you can restore the release by rolling back to the deleted version.

[ptagr]

seeing the same issue with v2.7.2 , fails when there are no previous successfully deployed releases

[stealthybox]

Also seeing two potential versions of this issue:

---

in CI:

+ helm upgrade --install --wait api-feature-persistent-data . --values -
+ cat
WARNING: Namespace doesn't match with previous. Release will be deployed to default
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
Error: UPGRADE FAILED: "api-feature-persistent-data" has no deployed releases

on my local machine:

( both at my OSX bash and in a gcloud/kubectl container )

+ helm upgrade --install --wait api-feature-persistent-data . --values -
+ cat
WARNING: Namespace doesn't match with previous. Release will be deployed to default
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
2018/01/25 00:19:07 warning: destination for annotations is a table. Ignoring non-table value <nil>
Error: UPGRADE FAILED: no PersistentVolumeClaim with the name "api-feature-persistent-data-db" found

---

The warnings are normal for our chart.
The errors are interesting because one of our subcharts has a pvc.yaml in it.

helm del --purge <release> does mitigate the problem.
This does make our CI pipeline difficult to upgrade.

[peay]

@adamreese what is the final resolution for this issue? We're on 2.8 and we still cannot upgrade a previously FAILED release with the change to helm list.

In particular, we're running into the following type of issues:

• deploy a release, OK
• upgrade --install --wait, but maybe there's a small bug and --wait doesn't succeed (e.g., liveness probes never make it up)
• after fixing the chart, upgrade --install --wait fails with xxx has no deployed releases

Deleting/purging is not desirable or acceptable here: the release may have multiple resources (pods, load balancers) that are not affected by the one resource that won't go up. In previous Helm versions, upgrade --install allowed us to only patch the change that broke the full release without having to remove all the resources.

Helm is the owner of all resources involved at all times here -- the resource is only marked FAILED because --wait didn't succeed to wait for all resources to be in a good state. I assume the same will happen if a pod is a bit too slow to start and in many similar cases.

[bacongobbler]

@peay see #3353 for follow-up discussion.

[peay]

Thanks -- that clears it up. Actually realized we were only hitting it when we had no successful release to begin with. In that case, purge is a fine workaround.

[stealthybox]

@MythicManiac FWIW:
I still have our teams pinned on v2.7.0 because of this behavior.
We don't seem to have any issues with resources upgrading and deleting when they are supposed to using helm upgrade --install with this version.

[KIVagant]

We also have this problem. It's very annoying that I need to delete K8s services and related AWS ELBs because helm has no deployed releases. The package manager is great but this issue leads to production downtime which is not good.

[tcolgate]

As a very hacky workaround, if the problem with the origin deploy is resolvable (e.g. preexisting service.), Doing a rollback to the original failed release can work.

…

On Fri, 5 Oct 2018, 18:13 Eugene Glotov, ***@***.***> wrote: We also have this problem. It's very annoying that I need to delete K8s services and related AWS ELBs because helm has no deployed releases. The package manager is great but this issue leads to production downtime which is not good. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3208 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEo8_pISzHFAuOz6Lsv5EjrFaEo_HkYks5uh5M7gaJpZM4Qtexz> .

[KIVagant]

@tcolgate, thank you! I just fixed another problem (#2426 (comment)) with your workaround and will try to test it for exist ELBs when I am deploying a new chart next week over existing resources.

[KIVagant]

Doing a rollback to the original failed release can work.

@tcolgate, I just tested and no, it doesn't work in the case of first deploy.

$ helm upgrade --wait --timeout 900 --install myproject charts/myproject/myproject-1.1.1.tgz
14:53:18 Release "myproject" does not exist. Installing it now.
14:53:18 Error: release myproject failed: deployments.apps "myproject" already exists

$ helm list
NAME        	REVISION	UPDATED                 	STATUS  	CHART           	APP VERSION	NAMESPACE
myproject    	1       	Mon Oct  8 11:53:18 2018	FAILED  	myproject-1.1.1  	           	default

$ helm rollback myproject 1
Error: "myproject" has no deployed releases

[KIVagant]

I am curious, if Helm can't deploy a chart over existing resources so why helm delete causes deleting exactly these resources?

[KIVagant]

@thomastaylor312, we faced this issue as well as #2426 (up: I found the real root cause for 2426) with helm 2.11.0. Do you think they should be reopened?

[krishofmans]

I found this thread after a Error: UPGRADE FAILED: "xxx-service" has no deployed releases
While it was visible from a helm ls -a.

I decided to see if it was an issue because of an incorrect --set value, and --debug --dry-run --force actually STILL deleted my running pod ... my expectation was that a dry run action would NEVER modify cluster resources.

It did work though, and the service could be redeployed afterwards, but we experienced downtime.

[stealthybox]

my expectation was that a dry run action would NEVER modify cluster resources.

This is a fair expectation -- we should make that... not happen

[bacongobbler]

I believe that was patched in #4402 but it'd be nice if someone were to check. Sorry about that!

[MohamedHedi]

same problem after upgrade to 2.11.0

[stealthybox]

Cross post:
FairwindsOps/reckoner#48 (comment)

[gerbsen]

Why is this closed? Do we have a proper way to handle this now?

[stealthybox]

@gerbsen, there isn't a way around this with current versions of Helm that is non-destructive.
We still use Helm 2.7.0 for everything in my org. It is a very old version that has other bugs, but it does not have this issue.

[notque]

Just had helm upgrade --install --force do a delete --purge and destroy my pvc/pv without telling me (on recycling). Had several failed releases, so it was in a state it was running in kubernetes, but helm thought there were no running releases. Not good times at all.

[alex88]

@notque after losing all grafana dashboard twice I've started doing backups before doing any kind of upgrade, having this kind of risk removes all the benefits of using helm

[yehee]

For those who are seeking for help, the issue was gone for me after upgrading helm to v.2.15.2

[ScubaDrew]

Still seeing this issue on 2.16.0

[nick4fake]

Why is it still closed? 2.16.1 is still affected

[alex88]

@nick4fake I think it's a duplicate of #5595