New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
helm upgrade --install
doesn't perform an install/upgrade if the first ever install fails
#3353
Comments
This was intentional by design of #3097. Basically, diffing against a failed deployment caused undesirable behaviour, most notably this long list of bugs:
If your initial release ends up in a failed state, we recommend purging the release via Now that being said, it might be valuable to not perform a diff when no successful releases have been deployed. The experience would be the same as if the user ran |
The suggested fix seems completely untenable in an automated system. I definitely don't want everything invoking helm to have to know about "if first release fails, delete and retry". For one, most of my tooling isn't aware if it's an install or upgrade, or if it's the first time or 100th time, it's almost always just running |
I'd also like to call out that I commented on the original PR #3097 (comment) asking specifically about this case. |
The old behavior was better for this case. @bacongobbler What other edge-cases are we concerned about? |
My local development would go much smoother if I could make This might be analogous to the |
To be absolutely clear, yes this would be a good thing to fix. Anyone wanna take a crack at it while we've got our hands full with other work? |
If the first `upgrade --install` results in a state FAILED, you can not run the same command `upgrade --install` again without a failure. This happens becuase we are search only for releases with the status DEPLOYED. This change will if the search for DEPLOYED fails, then try to search for a release with the state FAILED, and if found upgrade that. This fixes issue helm#3353
Hi, |
I am not sure why we need the (I'm struggling with a new variant this problem behavior in 2.8.0. Since upgrading from 2.7.2 now if I have a failed install, and then |
I agree with @whereisaaron, I would be nice with a |
If the first `upgrade --install` results in a state FAILED, you can not run the same command `upgrade --install` again without a failure. This happens becuase we are search only for releases with the status DEPLOYED. This change will if the search for DEPLOYED fails, then try to search for a release with the state FAILED, and if found upgrade that. This fixes issue helm#3353
Perhaps the solution is to have helm automatically run
Perhaps this behavior could be triggered via the |
Good idea, but I don't think we should ever delete the release ledger without the user explicitly asking to remove that data. Operators of Helm will want to learn why the service failed to upgrade from previously failed releases, or deduce failures by collecting that data from the ledger. I provided a comment earlier in the thread that describes a solution to the issue. It's similar to your solution in execution, but without the need to delete the entire release ledger. I believe #3437 is attempting to apply that solution as a patch. |
@rchernobelskiy happens to me as well. Exactly as you describe. I run into this issue maybe once per day when deploying new apps. |
@gmanolache We're still on helm 2.7.0 for this reason. If you need to downgrade, here's a good way to do it: downgrade to 2.7.0 |
What is this useful sounding 'helm ledger' diagnostic info and how do we get to it? 😄 I'm worried the below might be read as moody, it is genuinely just an invitation to for pointers on how we can get diagnostic info when you have a failed deploy. Because it really sounds like I'm missing something. It sounds like the failed state is supposed to have some utility for operators? I trawled through the helm manual site again; will something like 'helm get manifest' work in a failed state to extract useful diagnostic info? My user experience when I get a failed deployment is you get no useful info. Helm disowns all the partially created/remaining resources such that 'helm status' doesn't show anything. All you can do is 'rollback' or 'delete --purge' (you can't just 'delete' or your CI 'upgrade --install' will keep failing). The failed state only seems to serve to break the idempotency of 'upgrade --install' that we all crave for our CI deployments. Would it be reasonable to have an '--auto-rollback' option for CI situations, e.g. 'upgrade --install --auto-rollback'. I'd usually rather a roll back that have to get out of bed to deal with a failed state 😆 😴 💤 |
|
Thanks @bacongobbler. Ok, I understand that list is what is meant by the ledger. And if you still have the ledger, that you can use
If we had I guess even without the '--auto-rollback' option we could use a wrapper automate to run That is, we could fashion a wrapper script to ensure the results of a CI 'helm upgrade --install' is always a state where the next CI 'helm upgrade --install' will always be possible. Whilst retaining the ledger information for any failed attempts (at least for releases whose initial install worked).
|
@whereisaaron that would be elegant 👍 |
Is there an easy way to get the latest working release other than something like |
Hello there, I'm using Helm 2.12.2 and still have the issue, that helm fails, when the first deployment is failed. Is this a regression maybe? |
I'm not sure it's a regression, but that it was never actually "fixed". |
@RickS-C137 I think this is supposed to be fixed by using |
Still trying to fix this issue in a Jenkins Pipeline I am trying to use. |
@bacongobbler What do you think about #3353 (comment)? I do not see how there would be downtime or data loss if we destroy and recreate the initial release if the initial release fails. |
I implemented this in our build:
|
With helm currently, you don't know which helm command+options combination to use without inspecting the current state. And for a given helm command you don't know what you are going to get, because it depends on what the current state is. That's not really the declarative desired state dream ☁️ 💤 😄 In helm 3 we can potentially deprecate The option of For right now we can use wrapper scripts or meta-tools like
|
In retrospect, this is a breathtakingly obvious design goal. |
Hi, Is there any workaround to deploy a chart when the initial release 'apparently failed' but it's actually ok? |
So is the conclusion that |
Is there a separate issue tracking the idea of merging |
Not that I know of @dcow. What is the use case over |
and
I generally agree that helm should work like |
@dcow Ok, do you want to create an issue then with your proposal? |
Using
helm upgrade --install
is a nice way to install or upgrade depending on if the release exists. But it looks like there's a bug in the logic; its not handling failed installs. In my case the first install failed; then a subsequent attempt wasn't even made as it crashes out immediately.Maybe if the last release failed then
helm upgrade --install
should delete it and install again?The text was updated successfully, but these errors were encountered: