Helm chart: Allow the creation of extra manifests via values

[gplessis]

Pull Request Motivation

Sometimes, when installing the Helm chart, some extra manifests need to be deployed to have a fully-functional cert-manager: SecretProviderClass, ClusterIssuers, or Certificates...

This PR enables this use case by allowing the creation of extra manifests passed as Helm values. This is a similar approach to what Grafana and others do.

Implements #5900

When uncommenting the provided example extraObjects value - even with some template variables - it generates valid YAML:

$ tail -n 6 deploy/charts/cert-manager/values.yaml
extraObjects:
  - |
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: '{{ template "cert-manager.name" . }}-extra-configmap'


$ helm template cert-manager deploy/charts/cert-manager
(...)
---
# Source: cert-manager/templates/extras-objects.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: 'cert-manager-extra-configmap'

Kind

feature

Release Note

The Helm chart now allows you to supply `extraObjects`; a list of yaml manifests which will helm will install and uninstall with the cert-manager manifests.

[jetstack-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign wallrj for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jetstack-bot]

Hi @gplessis. Thanks for your PR.

I'm waiting for a cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wallrj]

Thanks @gplessis

I'm quite strongly opposed to this new Helm chart feature until you can give some specific examples of how it will be used and why other methods can not be used.
See #5900 (comment)

[wallrj]

Have you tested this? I think the toYaml function will choke on what is already yaml. Same as:

• [prometheus] extraManifests does not work after v21.0.0 prometheus-community/helm-charts#3328

I also do not think it will be possible / reliable to create cert-manager Custom Resources as part of the chart,
because the CRDs may not yet have been installed (possibly Helm orders those first?)
and because the cert-manager webhook takes some time to become Ready and it is not safe to create cert-manager custom resources without the webhook running.

Please supply some console output as evidence that this example works.

[gplessis]

Sorry for the late answer.

Yes, I do confirm that, when uncommenting the provided example value, it generates valid YAML:

$ helm template cert-manager deploy/charts/cert-manager
(...)
---
# Source: cert-manager/templates/extras-manifests.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: foo@bar.com
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx

You're right though: the usage of toYaml will become problematic when using some Helm templating instead of plain YAML as a value, as depicted in prometheus-community/helm-charts#3287

As a consequence, I would be happy to switch to {{ tpl . $ }} (from {{ tpl (toYaml .) $ }}), change the example value accordingly, and be more robust.

Also... I tested this with plain YAML in the context of an upgrade (when CRDs were already installed) and it worked as expected.
But, when installing from scratch, CRDs are missing and the Helm release fails. This would be solved by #6193. In the meantime, I'm ok to put this PR on the back burner.

[wallrj]

Don't use a cert-manager resource in the example, because we know that won't work.
Change the example to be ConfigMap and include a template variable in the example.
Perhaps the standard labels, like @katsew did in prometheus-community/helm-charts#2140

• [prometheus] Add extra manifest templates prometheus-community/helm-charts#2140

And add some example helm template commands and output to the PR description showing how you've tested it.

[wallrj]

Move the example up above the default value, so that the example is included in the documentation of the values in the chart README file.
For example:

cert-manager/deploy/charts/cert-manager/values.yaml

Lines 7 to 13 in b61de55

	# Reference to one or more secrets to be used when pulling images.
	# For more information, see [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
	#
	# For example:
	# imagePullSecrets:
	# - name: "image-pull-secret"
	imagePullSecrets: []

cert-manager/deploy/charts/cert-manager/README.template.md

Lines 76 to 89 in b61de55

	#### **global.imagePullSecrets** ~ `array`
	> Default value:
	> ```yaml
	> []
	> ```
	Reference to one or more secrets to be used when pulling images. For more information, see [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
	For example:
	```yaml
	imagePullSecrets:
	- name: "image-pull-secret"
	```

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale

[wallrj]

/remove-lifecycle stale

@gplessis I haven't forgotten about this. I half hoped that I might persuade Helm maintainers to add something to helm:

• Allow users to add extra objects and resources when installing a Helm chart helm/helm#12653

Please comment on that issue explaining your use-case; explain what extra manifests you plan to deploy with the chart.

Meanwhile, I'll review and test this.

[wallrj]

/ok-to-test

[wallrj]

@gplessis I rebased, signed-off and force pushed to your branch to fix the DCO problems.
Now I'll review and test the changes.

[wallrj]

/kind feature

[wallrj]

Don't use a cert-manager resource in the example, because we know that won't work.
Change the example to be ConfigMap and include a template variable in the example.
Perhaps the standard labels, like @katsew did in prometheus-community/helm-charts#2140

• [prometheus] Add extra manifest templates prometheus-community/helm-charts#2140

And add some example helm template commands and output to the PR description showing how you've tested it.

[wallrj]

Move the example up above the default value, so that the example is included in the documentation of the values in the chart README file.
For example:

cert-manager/deploy/charts/cert-manager/values.yaml

Lines 7 to 13 in b61de55

	# Reference to one or more secrets to be used when pulling images.
	# For more information, see [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
	#
	# For example:
	# imagePullSecrets:
	# - name: "image-pull-secret"
	imagePullSecrets: []

cert-manager/deploy/charts/cert-manager/README.template.md

Lines 76 to 89 in b61de55

	#### **global.imagePullSecrets** ~ `array`
	> Default value:
	> ```yaml
	> []
	> ```
	Reference to one or more secrets to be used when pulling images. For more information, see [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
	For example:
	```yaml
	imagePullSecrets:
	- name: "image-pull-secret"
	```

[pierluigilenoci]

We are so close!

[gplessis]

@wallrj Thanks for your comments and for the guidance. I addressed all of them. Please let me know what you think.

[wallrj]

Hey @gplessis

Thanks for your patience and for addressing all the code review feedback and for showing your testing in the PR description.

I was curious to see if I could use extraObjects for @pierluigilenoci 's use-case; to install a Grafana Dashboard alongside cert-manager.

Here's how I tested it

#!/usr/bin/env bash

set -o nounset
set -o errexit
set -o pipefail

script_dir=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd)
root_dir=$(cd -- "${script_dir}/../.." && pwd)

cd "${script_dir}"

# Build the Helm chart
make -C "${root_dir}" helm-chart VERSION=v0.0.0

# Create a Helm values file containing extraObjects with a Grafana dashboard ConfigMap
DASHBOARD_URL=https://raw.githubusercontent.com/monitoring-mixins/website/master/assets/cert-manager/dashboards/cert-manager.json
kubectl create configmap grafana-dashboard \
        --dry-run=client \
        -o yaml \
        --from-file=dashboard.json=<(curl -fsSL $DASHBOARD_URL) \
    | yq eval '. | .metadata.name="{{ template \"cert-manager.name\" . }}-grafana-dashboard"
                 | .metadata.labels={"grafana_dashboard": "1"}
                 | {"extraObjects": [. | @yaml]}' \
                     > values.extraobjects.yaml

# Create a cluster
kind create cluster || true

# Install kube-prometheus-stack
helm upgrade default kube-prometheus-stack \
     --repo https://prometheus-community.github.io/helm-charts \
     --install \
     --namespace monitoring \
     --create-namespace \
     --values "values.kube-prometheus-stack.yaml" \
     --wait

# Install cert-manager with the extraObjects values
helm upgrade cert-manager "${root_dir}/_bin/cert-manager-v0.0.0.tgz"  \
     --namespace cert-manager \
     --install  \
     --create-namespace \
     --values values.cert-manager.yaml \
     --values values.extraobjects.yaml \
     --wait

# values.cert-manager.yaml
prometheus:
  enabled: true
  servicemonitor:
    enabled: true

crds:
  enabled: true

acmesolver:
  image:
    tag: v1.14.4
cainjector:
  image:
    tag: v1.14.4
image:
  tag: v1.14.4
startupapicheck:
  image:
    tag: v1.14.4
webhook:
  image:
    tag: v1.14.4

# values.kube-prometheus-stack.yaml
grafana:
  enabled: true

# Enable discovery of all ServiceMonitor and PodMonitor resources
# https://github.com/prometheus-community/helm-charts/issues/1911#issuecomment-1106559031
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

I observed that the dashboard configmap got created with all the helm labels and with the helm release name prefix:

$ kubectl get cm -n cert-manager  cert-manager-grafana-dashboard -o json | jq  '.metadata'
{
  "annotations": {
    "meta.helm.sh/release-name": "cert-manager",
    "meta.helm.sh/release-namespace": "cert-manager"
  },
  "creationTimestamp": "2024-05-07T13:18:11Z",
  "labels": {
    "app.kubernetes.io/managed-by": "Helm",
    "grafana_dashboard": "1"
  },
  "name": "cert-manager-grafana-dashboard",
  "namespace": "cert-manager",
  "resourceVersion": "1626",
  "uid": "f749c2cd-5368-4215-8e2e-4b1b33468285"
}

And I saw the dashboard appear in Grafana:

kubectl port-forward -n monitoring deployments/default-grafana 3000

I then uninstalled cert-manager using Helm and observed that the grafana-dashboard configmap was deleted and that the dashboard was automatically removed from Grafana.

/lgtm
/approve

[cert-manager-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wallrj

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [wallrj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gplessis]

@wallrj thank you very much for your thorough review and your approval. I'm sure this feature will prove useful to a lot of people.

[pierluigilenoci]

Thank you! ❤️