Detect failed deployments and notify or force

[crmejia]

There is a known helm issue: helm/helm#3208 where as sometimes a failed release causes the install to fail. An easy work around exist with helm delete --purge <FAILED RELEASE then helm upgrade --install or just helm upgrade --install --force. Reckoner should detect this issue and either notify the user or force the install.

[stealthybox]

It's hard to decide what to do here because helm has its hands tied.
There isn't really a non-destructive action you can take.

If you delete --purge, it causes downtime.

Maybe removing the release's storage (ConfigMap/Secret) from the tiller namespace could work in most cases as long as you're not messing with CRD's?
That changes the hook behavior though.

In general, the 2.7.1+ behavior is borked and really needs a proper fix in helm.
I'm still pinned on 2.7.0 because of this.

[mattdarwin]

at least we could modify the error message. Instead of
Error: UPGRADE FAILED: "my-app" has no deployed releases
It could say:
Error: UPGRADE FAILED: "my-app" was previously deleted, or deployment failed. Try helm delete --purge my-app

[shalomcbouncex]

I think a flag that allow helm to take over existing resources and override them(without deleting them or leading to any kind of down time) will be very useful in such cases(provide a descent bypass, other claim that force delete and create - hence potential downtime).
The real solution is to fix the bug. The bug is that on failed helm deployment/upgrade that was able to create one or more resources(but not all of them) - does not register those resources as created by helm(therefore leaving those resources as if they were created by other entity and that fails the next helm runs with the error).
The solution is that any resource that is successfully created by helm should get written to the state - even if the helm install/upgrade failed, regardless of the end result resources that got pushed should be registered accordingly.

[sudermanjr]

I think a flag that allow helm to take over existing resources and override them(without deleting them or leading to any kind of down time) will be very useful in such cases

Helm 3 actually has this capability by adding the correct labels to resources.

The real solution is to fix the bug. The bug is that on failed helm deployment/upgrade that was able to create one or more resources(but not all of them) - does not register those resources as created by helm(therefore leaving those resources as if they were created by other entity and that fails the next helm runs with the error).

Agreed, it would be nice if this bug were better handled by Helm. Helm 3 is getting better at it, but it's still not perfect.