Add 'helm install --app-version' command/versioning a chart against an app version

[munnerz]

As far as I'm aware, helm install currently only supports specifying the --version flag to specify a chart version to install.

I'm unsure how the 'appVersion' field in a Chart.yaml is supposed to be used, but it seems generally useful to add support for versioning your application against a specific version (or version set) of a Chart.

Am I mis-using the appVersion field here? Should I instead be constantly building my chart to be backward compatible with previous version, or otherwise how can I infer to my users which chart version to specify when running helm install if they want a particular version (this becomes even more complex when you consider a user can also change the version deployed with something like --set image.tag, which often results in a version change of an application).

[johngmyers]

I just ran into this as well. I normally want the image tag to be specified at time of packaging the chart, but for debugging the app wanted to do an install with a different tag.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[munnerz]

Bringing this up again as it was raised in helm/charts#5919

Copying over parts of my recent comment:

---

We specifically choose to keep the minor version of our chart aligned with the minor application version (although patch version numbers can, and do drift apart).

This is because we may add a new flag to a new release of cert-manager, and by adding support for it into the Helm chart would break compatibility with older releases of cert-manager, as they do not support the flag. This is a pretty fundamental question around Helm chart versioning in general IMO, and one we don't have a good story for.

I know it isn't recommended to try and align appVersion with chart version, but this way, a user knows they can use Helm chart version 0.3.x with any of cert-manager 0.3.x, and chart version 0.4.x with cert-manager 0.4.x. Compatibility is defined within the minor version.

[munnerz]

/remove-lifecycle stale

[asetty]

I'd like to bring this back up for discussion.

Overall we haven't seen much compelling in favor of versioning the charts for our internal apps when all that is changing is the image tag used by some components. When upgrading a release, the appVersion field seems like the right place for this information.

Copying over my original proposal referenced above:

---

Our current workflow for deploying helm charts involves ansible tasks that call the helm upgrade CLI command and it would be nice to be able to pass a flag to set the appVersion when revising a release for the same chart version.

It may be a little weird conceptually because an appVersion is associated with a chart rather than release, but in our case we are just updating the image tag used for some containers and our workflow hasn't come to incorporate chart version and/or chart repositories yet. This may change in the future, but for now I don't see any issue with adding a flag for --app-version on install and upgrade as the field is purely informational.

[TD-4242]

when developing our own internal applications the app itself changes a lot more than the chart that deploys it. Typically our Continuous Deployment command is a helm upgrade with nothing more than --set imageTag=<new_version> (obviously, used elsewhere in the chart to set the container version) If we replace that with --app-version it would give us another visual point in helm ls to see what versions of code are deployed along with the version of the chart that is deployed.

To make this more visible otherwise I have standardized on setting a metadata tag of imageTag that gets set to the imageTag passed on install/upgrade. This allows me to use the K8s dashboard or easily create Graphana dashboards with imageTag displayed, but requires me to leave the command line and go mousey clicky.

[FedeBev]

Any news about this?

Thanks

[omnipresent07]

Any update on this. It seems PR from @eraac does what is requested. As @TD-4242 mentioned, we also run helm upgrade --set imageTag=<imagetag>, however this doesn't update the APP VERSION listed in output of helm ls. Being able to --set app-version or --set version will allow us to run helm upgrade such that helm ls correctly shows the version that is deployed.

[thunder-spb]

Any updates?

[ChrisCooney]

Aaaaany time soon would be lovely!

[yanivpaz]

it will be very useful

[Stono]

Would also love the ability to set app version at install time as we use common charts to deploy applications

[jplonghi]

+1

[yadavnikhil]

+1

[dhavalc26]

It would be very helpful

[DE110283]

Requesting the same

[filipre]

Please stop the spam with the +1. There is already a PR (#4961) and people are discussing the proposal. The last reply was even 2 days ago.

[bgarcial]

Hi @filipre

There is already a PR (#4961) and people are discussing the proposal.

According to that PR, this was moved here:
#5492
This is the latest PR to #4961 and looks like if we're waiting for a review to be merged ...

[com30n]

@filipre Could you please say: what happens with the PR? Looks like PR has no move half of month

[alexellis]

This would be very useful. I ran into something where I need to pin to version 0.6.0 of an app, and the chart version has no relation to the app version.

[tomassundvall]

I agree, I also think this would be very useful. Any updates?

[timothyclarke]

I read through a bunch (not all) of the discussion above, so sorry if I'm repeating some points / views. I'm trying to put forward a more considered response.

I like seeing appVersion when I do a helm ls and conceptually moving from .Values.image.tag was a good one BUT not being able to set it at deployment time is a real show stopper and is something I've had to revert back to.

I'm firmly that (chart) version is the version of the chart and appVersion is the docker tag. In our CI process the docker tag is also a git tag.
We also have multiple microservices and have a desire to keep things as DRY as possible. We have generic charts in a local chart repo because the vast bulk of java-springboot apps are the same. The bulk of tomcat apps are the same (but different from the springboot ones). Rinse and repeat for other technologies. We then have environmental values as the deployment makes its way through various environments
Each of these microservices then makes use of the generic chart through CI/CD
eg helm upgrade release-name private-repo/generic-chart --values <environment>.yaml --set image.tag=<docker tag from build step> --namespace <environment> --install I would prefer to use .Chart.AppVersion rather than .Values.image.tag but I MUST be able to deploy in a way that is efficient for our Org.

If I do a helm ls I have both CHART and APP VERSION so the whole chart version must match app version falls flat on it's face right there. Continuing down that route will just alienate people and at some point the project will be forked because that mentality is too strict and not what many people are after. It's also starting to go down the route of "Lets remove image.*, nameOverride & fullnameOverride. image.* can be hard coded in deployment.yaml etc." for VERY similar reasons.

A final point is that many public charts will not exactly match the docker container version they make use of. Take a look at most well known docker containers eg alpine or nginx where the major and minor versions are rolling tags with only patch versions not rolling. Having a 1:1 mapping for every patch version introduces quite a significant overhead for little to no benefit.
It is not uncommon for production environments being unable to upgrade to latest version for a multitude of reasons. Don't even speak to most places about rolling versions in production.

The upshot of all of the above then begs the question "Why can helm use a chart repo at all?".
Not being able to overwrite appVersion at install / upgrade time means you already need to download and unpack the chart, edit appVersion per install/upgrade, or you might as well package the needed docker containers into the chart.
The exception is where a completely standard install is happening, and there's already plenty of debate surrounding auto generating passwords and the like.
I know the last paragraphs seemed like I was going down a rabbit hole and painting myself into a corner, but that IS where "appVersion is the docker tag AND cannot set appVersion via the command line or values" takes us.

[TBBle]

@timothyclarke: What you might want to do, for the helm upgrade use-case you described here, is helm package first, which lets you set --version and --app-version, and then you can helm install the tarball, and keep it around as a CI artifact, which increases your reproducibility for the install, as it won't need any --set parameters added. That's what we've moved to, although without the "generic-chart" aspect, as our charts are not generic.

It's also a good chance to add build metadata to the Version, with something like +g<shortCommitSHA>.

Per #7517, that let me remove a bunch of sed calls that were rewriting image.tag before installing onto our CI testing cluster, and then again when later packaging.

This approach might actually resolve the problems most people have hit here, if they're building their own charts, and particularly if they're installing from the chart source in their checkout. It doesn't really help if they need this functionality for a chart coming from a repo, but I think that's a different issue than most people here are hitting?

To me, the risk from overriding an app-version (or version) at install-time is that it's not as clearly visible to someone else trying to recreate the chart, that this was done. Unless it's somehow hacked into the values support, it won't be there when one extracts the current config of the chart using helm get values -o yaml, so it becomes one more thing that makes your live chart deployment different from what you get with helm install <some way to specify a particular package> --values <values from helm get values>, e.g., when trying to reproduce an issue seen in production on a testing setup.

[bokysan]

To me, the risk from overriding an app-version (or version) at install-time is that it's not as clearly visible to someone else trying to recreate the chart, that this was done. Unless it's somehow hacked into the values support, it won't be there when one extracts the current config of the chart using helm get values -o yaml 

You hit the nail on the head. This should have bee in values.yml from day one.

While I understand philosophical arguments against this feature, the field practice shows it would help people a lot -- including us.

There are many charts in the wild which let you set the version of the app through values.yml, specifically because it can't be set here.

[invidian]

I'm not sure if that has been discussed (did a quick CTRL+F and couldn't find any trace), but what about removing appVersion all together as an alternative solution? It seems to me, that it would avoid the whole confusion.

Right now appVersion is treated as kind of "special" value. I assume, that it's there to provide the visibility e.g. I can have chart version 123.5.6 of Prometheus, but it will have appVersion: 2.17.1, so I know what security patch version does it have and what Prometheus features to expect and I can look it up using helm ls.

I guess that could be provided in some different way. Maybe via release labels? Or maybe jsonPath query over all releases, similar to what is possible with kubectl e.g.:

kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'

Then, the support of that would be shifted to best practices, instead of being enforced by helm itself. It could be linted too.

On the other hand, it might be, that many people rely on existing implementation of appVersion, that also needs to be considered.

Maybe taking a step back and understanding why exactly appVersion has been added would help resolving this issue?

[timothyclarke]

@bokysan It was previously in values.yaml it was moved to Chart.yaml I'm guessing for the whole helm ls showing both the chart and docker tag rather than having to run a command such as
kubectl get deployment <release name> -o jsonpath='{.spec.template.spec.containers[0].image}'

@TBBle I'd address each of your points, but it would make this post as long as the previous one. I think this whole issue comes down to someone deciding that a generic chart is not a valid use case solely by looking at the charts in the public repo.

The entire premise of appVersion falls flat on it's face as soon as you need to start using initContainers and sidecars. To give a real world example one of the projects I currently need to manage has nginx with a php sidecar. The nginx and php tag's don't change often. The php container / version is very important to the developers writing the code. The container that changes most frequently is the initContainer which provides the content.
Do I set appVersion to the initContainer, to the php container or is it nginx, and by choosing only one of these what information has been lost ?

[TBBle]

If it's important to your users, then it should be the PHP version, surely? That's what you're advertising. You could also stick all three in your appVersion, it's a freetext field after all.

If you want to force appVersion == container image tag, then you're going to find that challenging with non-trivial charts, i.e. charts with more than one container, or with no containers. That's not really the point of it though, or it would be imageVersion. If you are packaging a single upstream app, use its version. If you are building a package of several upstream apps, pick one, e.g. prometheus-operator's chart, or ignore appVersion, since it's an optional field.

If your chart source is part of the application, e.g. Agones, just leave appVersion blank, or copy it from version if you have tooling that depends on it.

None of these things need to be helm install-time decisions. As I mentioned earlier, helm package is late enough for all the workflows except "switch out a different upstream version at install time for a third-party chart" and that should be in --values or --set like all the other "change X at install time" actions.

Honestly, the actual missing feature is probably "pass appVersion through tpl", so you can have it read from .Values.image.tag or whatever works for you.

[timothyclarke]

If you want to force appVersion == container image tag

Then we're probably in #7517 That maybe where all this stems from.

[ldath]

I do not understand this discussion at all. Why not giving people the option to use the app version in a way they think is the best fit for them is so much a big issue?

In a current form for me will be best to not have this APP VERSION at all. It is bringing only confusion to people in our project. We have >80 services which are using the same helm chart and because it is not possible to easily change this APP VERSION in the helm upgrade -i ... I see that all of our applications will forever stay with 1.0 here. And I do not plan to repackage the already packaged chart to just change the app version. Why I should complicate my CI to fit your design???

I also see that I just need to say to everyone to not use the helm list as it will be something not useful for them. To check which version of our applications they have they will need to use something else.

I was optimistic at the start of reading this conversation but after going to the end seeing how you discuss this and how you fight to force users to have your way of thinking I lost hope now :(.

[pniederlag]

Having two different outputs "CHART(version)" and "APP VERSION" in helm list, helm history and alike is very helpful and avoids the need to dig deeper into command line options and output parsing to get the most important facts.

If "CHART(version)" and "APP VERSION" are tied at build-time ("helm package"), the whole benefit of having two different values is somewhat lost. Building a chart and updating the APP VERSION without incrementing/updating the CHART VERSION will result in big trouble as the same CHART VERSION will give you very different results. :-<

So for now we are forced to build the package with every release and increment "CHART(version)" and "APP VERSION" in sync to not run into an insane/unclear situation.

As I just learned, we could drop "APP VERSION", as it is optional und use some custom value and use that instead of {{ Chart.appVersion }} for our image.... but then helm list would be way less informative. :-<

From Users(Developers) Perspective:

• ability to set some version property/flag/value at install time
• ability to see that version propert/flag/value in helm list/history output with label "XX version"

Any chance we can get that done?

[TBBle]

If "CHART(version)" and "APP VERSION" are tied at build-time ("helm package"), the whole benefit of having two different values is somewhat lost.

I think this is the crux of the disalignment. The benefit of App Version as I use it, and as appears to be intended by the current setup, is that you know the version of a wrapped Application for that chart version, because the Chart Version is the version of the whole chart, not the version of the templates in the chart. I'd hate to have every statement like "We require Version ~X.Y of the Helm chart" to require "Oh, and don't mess with the AppVersion" added to the end.

This benefit is lost of App Version (and the actual version of the app) is changed at install time, because now the Chart Version doesn't tell you what App you're using, and you lose the ability to use SemVer to ensure, e.g., you have the latest-but-API-compatible release.

For the use-case @pniederlag is describing, being able to have appVersion be a template that points at a Values entry would make helm list do what is desired, as long as the Chart supports having its application version (probably a container tag) changed at install time, via --set or --values like every other "changed at install time" configuration option.

[setevoy2]

Here where I got an issue with the AppVersion.

We are using both the Release version and AppVersions.

To set them now - I have to call the helm package explicitly beforehand of helm upgrade --install to create a local tar-archive with both version being set.

Now I'm adding the helm-secrets support, and...
And it's wrapper can't work with the helm package!

So - what now?
Drop all our versions' support and flow?
Drop using Secrets?
Any ideas?

Actually it's more an issue for the helm-secrets, but it's related to the --set app-version ability discussed here as well, because if I could use it in this way - I don't need to call the helm package at all.

UPD Oh, wait... I'm still able to use helm secrets upgrade chart.tgz -f secrets.yaml...
Okay.
But still, +1 to add the --set app-version.

[guice]

So - what now?
Drop all our versions' support and flow?
Drop using Secrets?
Any ideas?

We build two packages: a helm package with just the charts, sans the env values and secrets file. We rename this package to chart-{app-version}.tgz since chart-version means nothing for us, nor does chart-version support our app-version syntax. Our app updates include any potential chart updates (same repo, using git tagging).

Then we have a second tgz that's environment specific, chart-{app-version}-{env}.tgz that includes the chart tgz, values yaml, and encrypted secrets file. This file also includes a "package.yaml" that contains values such as tag, app, and environment name for our automated scripts to deploy using helm-secrets.

[bvis]

We are used to identify our applications versions, or most of them, with a semantic version number. Then use use this number in the APP VERSION to identify it easily in the helm history list for rollbacks or other operations.
As the system does not allow to inject it automatically in deploy time we execute this simple command automatically on our CI/CD pipeline before the deploy command:

sed -i -E "s/^appVersion: (.*)/appVersion: ${deploy.project.version}/" ${chartPath}/Chart.yaml

It's tricky, but it works as expected.

[timothyclarke]

@bvis
The issue with your workaround is that you have to edit the chart in your CI/CD pipeline.
If you use a centralized chart repo, then you're forced into helm pull repo/chart --untar

[jakovistuk]

Any further progress with programatically injecting Chart.yaml/appVersion ? Are there workarounds? It would give a tremendous boost in CI/CD Helm .

[davi5e]

@jakovistuk as far as I can tell, charts that use appVersion to show the container version do this directly via Chart.yaml, as seen in nginx-ingress/Chart.yaml for example...

I haven't given much thought to this issue for quite some time, so this may be a really dumb question, but is there a way to use Helm CLI to override appVersion?

[munnerz]

It seems like a lot of people here are asking for a way to override the ‘appVersion’ field. The original intent/request in this issue is to allow —app-version as a replacement for —version, so a user could run ‘helm fetch —app-version=v0.15.0’ and Helm would work out what the latest chart version last specified v0.15.0 as an appVersion and fetch that.

In our project/chart (cert-manager) we want to make it as clear to end users as possible which version they are installing, so allowing them to install by app version instead of chart version would be a far more natural installation experience.

That said, this issue was opened 2y ago now, and since then we have opted to just keep both of these version numbers in sync/lock-step. After a couple of years doing this, it’s been surprisingly easy and pain free, albeit users sometimes have to wait a couple of weeks for a new official release if there are changes made to our deployment manifests.

Given the age of this issue, it’s length, the huge variety of slightly different feature gates, and the changes in the Helm project since then (Helm 3, OCI charts etc), I don’t think this issue is in a good state to be driven forward as a feature request in its current form. I’m going to close this issue, but anyone else who has a similar feature request Is best to open a new issue and link to any other relevant comments in this issue to provide evidence. Hopefully that’ll work better for the Helm team’s triage process so your requests can get the visibility they need!

I also think this sort of functionality could be, and probably is best, implemented as an external tool or wrapper around
Helm, especially when taking account of the OCI changes which I think would make this trickier to implement.

[haimari]

Until this is solved (or not) Here is how i solved this in my CI/CD (GitLab):

package the chart with the app-version, then deploy it.
I know that the Chart version is not meant to be the same as the appVersion, but in our case it is fine as a workaround.

deploy:
  image: alpine/helm:3.2.4
  stage: deploy
  environment:
    name: ${ENV}
  script:
    - helm package --app-version=${CI_COMMIT_TAG} --version=${CI_COMMIT_TAG} ${NAMESPACE}
    -  | 
       helm upgrade -i --wait ${CI_PROJECT_NAME} ./${NAMESPACE}-${CI_COMMIT_TAG}.tgz \
       --set image.repository="${CI_REGISTRY_IMAGE}" \
       --set image.tag="${CI_COMMIT_TAG}" \
       --set-string ingress.enabled="${INGRESS}" \
       --set service.port="${CONTAINER_PORT}" \
       --set service.targetPort="${CONTAINER_PORT}" \
       --set dc="${CI_ENVIRONMENT_NAME}" \
       --set project="${CI_PROJECT_NAME}" \
       --namespace ${NAMESPACE}
    - helm history ${CI_PROJECT_NAME} -n ${NAMESPACE}
  tags:
    - kubernetes
  only:
    - tags

[TBBle]

If you default your image.tag to {{ .Chart.AppVersion }} then you won't need to --set it during the install, it'll already be correct. This works nicely for auto-builds as well, when your Docker images are tagged with SHA1 so the AppVersion matches the Docker image tag, and the Version is an auto-build SemVer.

There's no problem with Version being the same as AppVersion if your AppVersion happens to be SemVer.

For packages produced by my team, we're moving towards things that look for AppVersion, e.g. image.tag, defaulting to Version if AppVersion is unset. It's not a huge difference, just one less argument to helm package for tagged releases, but only makes sense if your chart is built from the same SCM as the thing you're packaging.

[woodcockjosh]

@TBBle that won't work if you are using a sub-chart to set your image tag

[TBBle]

Do you mean the image.tag is in a subchart, but you're trying to use the version of a parent chart? If so, yes, that's very awkward, and won't be easy to manage. I just bounced off exactly this layout in https://github.com/googleforgames/open-match/'s Helm charts. I suggest rolling the sub-charts in question back up into the main chart in this case.

Charts should be independently isolated/usable units, not relying on parent-chart behaviours to function. The subchart has its own version, that's the one that its images should be using, otherwise, why is it a subchart?

In Open Match's case, the subcharts appear to be used so that XXX.enable can be used as a shortcut in the values.yaml to disable a bunch of stuff at once, but then it introduces a bunch of structural issues like this. Open Match's subcharts all make heavy use of the parent chart named templates, and also have a local version of 0.0.0-dev, so there's already two code-smells that something is not structured well.

Or perhaps I've misunderstood the observation you're making.

[a0s]

@haimari Unfortunately, its not working (relate to #6921 ?):

> helm package $DIR/deployment/chart --app-version="1111e8" --version="3454e5" --namespace stage
Error: Invalid Semantic Version

But, this works:

> helm package $DIR/deployment/chart --app-version="0.0.0-1111e8" --version="0.0.0-3454e5" --namespace stage
Successfully packaged chart and saved it to: /Users/aws/service-0.0.0-3454e5.tgz

and even this (but seems dirty):

> helm package $DIR/deployment/chart --app-version="0-1111e8" --version="0-3454e5" --namespace stage
Successfully packaged chart and saved it to: /Users/aws/service-0-3454e5.tgz

helm version version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"dirty", GoVersion:"go1.15.3"}

[TBBle]

I think @haimari 's solution as-is only works because they use semver-compatible tags in their CI pipeline (i.e. this will be an example for a job run for tagged releases, not run on every-commit)

@a0s: I generally suggest:

helm package $DIR/deployment/chart --app-version="<container image tag>" --version="<semver version>"

And then have your container image tag value be something like {{ Values.image.tag | default .Chart.AppVersion | default .Chart.Version, so that you don't need to change it on-the-fly, as @haimari does.

In your examples you have what appear to be two different git versions, is that right? Is one for the container image, and one for the chart?

With SemVer, you can't really put a git commit SHA into the meaningful part of the semver, because semver implies ordering, and git commit SHAs are not sortable.

So you'll want to use a version something like 0.0.1-alpha.<build-id>+g<gitcommitsha> where <build-id> is somelike like the pipeline or job ID from your CI system, so it's always going up you commit to your project. That way you always get the latest build when you ask for it.

In SemVer, using a - means it's a pre-release for that version, so 0.0.1-<anything> falls between the 0.0.0 and 0.0.1 releases. The part after + is the build-info, and it's ignored for sorting, hence a good place to put git SHAs or branch names or other non-sortable tracking information.

So with what you have used here, 0-3454e5 will appear to be newer than the next commit, if its SHA happens to start with a 2, e.g., 0-2764e1.

[a0s]

In your examples you have what appear to be two different git versions, is that right? Is one for the container image, and one for the chart?

Yes, the app and the chart - there are two independent pieces of software.

That way you always get the latest build when you ask for it.

What if i don't want (and even can't imagine) ask for the latest.

0.0.1-alpha.<build-id>+g<gitcommitsha>

This string (after interpolation) seems too long to fit into one column of standard helm list output :)

I always know what version (sha hash) of the app i want to install (pass it with --set args). And i always know what version of the chart i use (as @haimari described, i will always use git checkout chart && helm pack && helm upgrade .tar.gz locally in my ci/cd)

What could go wrong?

1. An error during regular helm upgrade. Ok, i will fix the error and try again (with another sha commit of the app - 99%) (or using --atomic instead)
2. Manual rolling back: helm rollback <RELEASE_NAME> or deployment previous sha commit through CI/CD.

What am i missed?

PS To be honestly, i want to use short sha part in version and app-version for information purpose only (during helm list)

[TBBle]

If it's just for information purposes, then it goes after the + in a SemVer, not -. If you never care about ordering of releases, or distributing Helm charts to anyone, and your chart or app aren't already SemVer'd, then 0+g<commitsha> is a valid SemVer (equivalent to 0.0.0).

This is what Open Match's Helm auto-built charts do, for example; they are all currently 0.0.0-dev, and we've started looking at making that 0.0.0-dev+g<commitsha> so that if you're looking at what you have installed, you can at least tell which master build you have.