Seems reasonable, I guess. This should also optionally install the validating webhook?

Yes, I think so, this requirement is coming from that I am working on a Contour based Microservices API Gateway, we need to keep the k8s cluster with gateway-api installed, and I think it is a better way to make community to provide an official helm chart and maintained it by community.

I'm hesitant, but open to discussing. If there are some clear and strong motivators to do this I think it could be worth discussing at some length in our next community meeting.

I'm not sure a Helm chart is a good fit for Gateway API. Here's what Helm's docs have to say about CRDs:

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations

I understand and appreciate their hesitancy to support updating or deleting CRDs and I think more broadly that highlights how careful we need to be when installing and updating CRDs. I'm hoping that the role of CRD management will largely be handled by the cluster provider in the future. For example, on GKE, Gateway API CRDs can be installed and upgraded automatically for you: https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways#enable-gateway.

I'm not sure a Helm chart is a good fit for Gateway API. Here's what Helm's docs have to say about CRDs:

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations

I understand and appreciate their hesitancy to support updating or deleting CRDs and I think more broadly that highlights how careful we need to be when installing and updating CRDs. I'm hoping that the role of CRD management will largely be handled by the cluster provider in the future. For example, on GKE, Gateway API CRDs can be installed and upgraded automatically for you: https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways#enable-gateway.

Yes, this is helm chart design that can't upgrade and delete CRDs, In our practice, we manually manage CRDs in private K8s cluster, maybe we can request some feedback from end users for this.

I didn't even think about the CRD install issue, I think that makes this probably a -1 from me.

Eventhough Helm does not support it properly, in reality a lot of/majority of OSS tools do manage them through Helm though.

Helm's CRD management system means that you need to manage them via another tool after the initial install, so I don't see much that a chart would provide beyond a small reduction in install friction via a dependency (absent first party support for kustomize URL dependencies in Helm itself).

In practice, from what I've seen in the wild, I'd expect many clusters to have a CD system (Argo or similar) to ensure kustomize URL dependencies are deployed.

As per our comments, close it, when we have more requirement, let's reopen.

@wilsonwu, could this issue be reopened?

@wilsonwu, could this issue be reopened?

Yes, reopened.

I don't think anything has changed though - I don't see us creating a Helm chart for Gateway API CRDs any time soon unless something has changed about Helm and CRDs I haven't heard of?

@youngnick
As @tomkerkhove said:

Eventhough Helm does not support it properly, in reality a lot of/majority of OSS tools do manage them through Helm though.

I think this comment is more relevant than taken.
Prometheus CRDs, Grafana CRDs, Cert-manager CRDs, Cilium CRDs, Elastic CRDs, Istio CRDs, Velero CRDs, Argo CRDs, etc all use charts for deployment diffing and cleaning (it's hard to know when to remove something unneeded).

The additional problem that we have on top of the usual CRD management problems is that it's possible (maybe even likely) that multiple implementations that require the CRDs will be installed in the same cluster, and they would then need different versions of the Helm chart (if we provided one).

I worry that we could end up with implementations having their CRDs downgraded while they're running, because another implementation was installed in the cluster.

This is different to all the examples @lucasfcnunes mentioned in that all of those CRDs are owned by only a single implementation.

I'm not saying that we shouldn't do something to make this experience better, just that I don't think the solution can be "make a Helm chart" without addressing some of the concerns that arise from being a shared set of CRDs rather than a set of CRDs closely tied to a single implementation.

I worry that we could end up with implementations having their CRDs downgraded while they're running, because another implementation was installed in the cluster.

We just don't use /crds folder helm/helm#10585 and go with https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-2-separate-charts instead of https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-1-let-helm-do-it-for-you

Helm would actually protect us from downgrading (installing over current resources) as each resource is tied to a single release. Helm never takes over when the resources don't match the release's annotations and labels.
A simple kubectl apply doesn't do that.

I'm +1 for a community owned helm chart install for the CRDs and I want to pitch another item that would be nice to include:

Ability to generate Gateway API related objects (HTTPRoutes, TLSRoutes, etc)

Why?

• A lot of folks, myself included, are using GitOps for managing clusters and those GitOps engines typically support helm (of course Kustomize too)
• Many / most OSS helm charts do NOT include Gateway API objects (out of tree) but DO include Ingress Objects (part of core API)
• How do I gain the value of this out of tree spec + provider implementation ON TOP of whatever workload I'm using?

To me it seems like there's a couple options:

• Create a major initiative to update the common popular helm charts to include a Gateway API option (this means there would also have to be a standard to how Gateway API spec objects are written in helm). We've seen the issues with different Ingress versions:

{{- if .Values.ingress.enabled -}}
{{- $fullName := include "demo.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
  {{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
  {{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
  {{- end }}
{{- end }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
...

• Create a community owned helm chart which could be used to automatically install the CRDs and allow consumers to not worry about if their other helm charts support Gateway API.

Today, I'm working around this limitation with: https://github.com/jimangel/unofficial-gateway-api-helm

That does not include the CRDs, but in my opinion, I think the project could have some generic templating of Gateway API spec objects AND a /crd folder in the helm chart. Thus, allowing the end user to use the helm built-in: --skip-crds flag (https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) if they don't want to use the CRDs.

I think the trade-off would be documentation around upgrades / life-cycle. The alternative is more custom YAML and more custom solutions like I've already started. It would be great to have a centralized solution for the Gateway API spec to increase adoption.

The helm chart could be positioned as a secondary option, as in, "Gateway API project suggests installing via kustomize..." and then below have the helm options for advanced users and/or spec templating.

Just for my own personal clarity on where people are coming from here and what's actually being suggested, I wanna know what people's expected commitment level is here:

Is anyone in this thread specifically interested in owning and maintaining this long term? Or is this more of a request to have the current project as it stands take on the responsibility?

I still have the same concerns about CRD management directly via Helm, but will be the first to admit that Gateway API CRD management is nowhere near as easy as it could/should be. I've created #2678 to discuss an alternative installation method that could also be bundled in a Helm chart.

Is anyone in this thread specifically interested in owning and maintaining this long term?

@shaneutt I already have the burden of this with https://github.com/jimangel/unofficial-gateway-api-helm; and without a community chart, I would imagine we'll have more fragmented solutions as everyone makes their own "wedge" to solve this problem vs. a community standard. I would gladly help maintain and support a solution, assuming there's one we all agree on.

I kind of thread-jacked a bit with my rant on objects beyond CRDs, and honestly, I think that's going to be an increasingly bigger problem. I would love to see other OSS helm charts adopt support for Gateway API objects, similar to how we see with ingress today, but there's not even published guidance or standards around that.

I still have the same concerns about CRD management directly via Helm

These concerns are valid. As we chatted about, it's not an issue for external secrets operator or cert manager. But, to be fair, those are the more than only CRDs.

Given the "out-of-tree" spec for Gateway API, there needs to be a way to unify it with an implementation AND there needs to be a way to unify it with your workloads. Today, there is no standard and it's causing a lot of friction (especially for anyone trying to automate their clusters at scale using GitOps + OSS helm charts/kustomize).

I would be open to the discussion in #2678 for an alternative to install CRDs (defining the "official" way for other implementations/charts to include the Gateway API). Then my remaining gripe for "how can we make gateway api included in all the helm charts as a standard, like ingress" could be solved per workload.

If we define what that "official" way is, I'm happy to help write the docs and guidance so others can adopt this model.

It's less about helm and more about the fragmented state of things today and the friction caused by Gateway API being out of tree with no defined guidance on how other helm charts / OSS workloads can integrate seamlessly with BYOgateway API integration.

This has been stale for some time, if someone wants to champion this please let us know as that's what it will take to move it forward.

This has been stale for some time, if someone wants to champion this please let us know as that's what it will take to move it forward.

@shaneutt Hello. I just create a poc of crds-only helm chart and it works. See the draft pr #2951 for more detail. If the general direction is alright, I can finish the remaining todos and get the pr merged.

Edited:

The draft pr is on hold and awaiting further discussion. Sorry I didn't read through all the context, i.e. the complex requirements we want it to achieve.

See envoyproxy/gateway#1721 (comment) for an example to incorperate crds installation in iac/gitops.

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

👋 hey all, checking in because I've been MIA on paternity leave. I'll be back in office mid-June, happy to help see this to completion or help @networkhermit when I return.

/remove-lifecycle rotten