Skip to content

Improved helmfile debugging with werf

Latest
Compare
Choose a tag to compare
@himanshu-garg-razorpay himanshu-garg-razorpay released this 04 Nov 13:08
· 1 commit to master since this release
e8a8408

This release adds a new script allowing us to (among other things) install all the tools required by devstack with a single command. This note will demo an upgrade from existing tools, showcasing the benefits of using werf over helm in handling and reporting error cases improving the debugging experience for helm releases.

Problem
We analysed all the issues that landed in our support channel in the last quarter and realised that a lot of these had one thing in common, developers didn't understand what went wrong. Even when our team started looking into a support issue, we had to jump through a few hoops to figure out the root cause, like events, logs and descriptions for all the deployments, jobs and hooks. And even before we did that, we had to list out all these resources from the helm chart. The underlying problems would usually be something like ImagePullBackoff (ErrImagePull), ContainerConfigError, CrashLoopBackoff etc. But helm doesn't exit the release in case of such an error, instead it keeps on waiting for success till the end of timeout. We only get very opaque errors from helm like timed out waiting for condition and that too after exhausting the entire timeout which is around 15-20 min of wait time.

Screenshot 2022-11-04 at 6 09 59 PM

Screenshot 2022-11-04 at 6 27 04 PM

Solution
We needed something that could track the status of all the resources being built in the release with full view of relevant events, logs etc, exit the release in case of errors and report it transparently.

Well werf has all that (and more)
Checkout how to install this at the end of this doc.

Demo
ErrImagePull - The docker image mentioned in the deployment doesn't exist

Screen.Recording.2022-11-04.at.6.21.27.PM.mov

CreateContainerConfigError

Screen.Recording.2022-11-09.at.11.16.11.AM.mov

Readiness Probe Failed

ReadinessProbeFailed.mov

Bad DNS Policy

BadDNSPolicy.mov

Easy Installation
Copy and run the following command in your terminal which installs werf. It also configures helmfile to work with it. And boom..! now every deployment you make, has nice logs.

Please, remember to open a new terminal after running this.

sh -euo pipefail -c "$(curl 'https://raw.githubusercontent.com/razorpay/devstack/master/setup-helper.sh'); setup_tools_only"
Screen.Recording.2022-11-04.at.6.31.20.PM.mov