Helm chart does not support tlsConfig for serviceMonitor/podMonitor

[volker-raschek]

Summary

As the title of the bug already describe, does the helm chart not support a custom tlsConfig for the serviceMonitor and podMonitor resource. I need to specify properties like ca, caFile, cert, certFile, insecureSkipVerify, key, keyFile, keySecret and serverName

More details

I've deployed an internal certificate authorithy via cert-manager. Each namespace get his own intermediate ca. Based on this intermediate ca, I issued a custom certificate to listen the web-server of the certificate-exporter on HTTPS. To archive this, I've created the following web-config.yaml and referenced the secret in the helm chart:

apiVersion: v1
kind: Secret
metadata:
  annotations: {}
  labels:
    shell-installer/name: prometheus-x509-certificate-exporter
  name: prometheus-x509-certificate-exporter-webconfig
stringData:
  webconfig.yaml: |
    tls_server_config:
      cert_file: /etc/ssl/server/tls.crt
      key_file: /etc/ssl/server/tls.key
      client_auth_type: NoClientCert
      client_ca_file: /etc/ssl/ca/ca.crt
      min_version: "TLS12"
      max_version: "TLS13"
type: Opaque

Based on the fact that I switched to HTTPS, the serviceMonitor created by the helm chart always listen on HTTP. I've inspect the helm chart and found the reason for listening on HTTP. The scheme will only be switched when turning rbacProxy to true, but my intention is not to turning on the rbacProxy option. I would like to use the foll tlsConfig spec which is supported by the api resource monitoring.coreos.com/v1. This includes more than predefined values.

Long term goal

Instead of using basic-auth or token auth, I would like to implement tls client authentication - mutual TLS. The spec of the api resource monitoring.coreos.com/v1 already contains the required values and the property client_auth_type of the webconfig.yaml also support it, but the helm chart does not provide tls configuration options for the serviceMonitor.

Proposal

It would be great, if the default values of the tlsConfig would be part of the helm chart. Otherwise is not possible to customize the configuration and to encrypt the communication between prometheus and the certificate exporter.

prometheusServiceMonitor:
  # -- Should a ServiceMonitor object be installed to scrape this exporter. For prometheus-operator (kube-prometheus) users.
  create: true
  # -- Target scrape interval set in the ServiceMonitor
  scrapeInterval: 60s
  # -- Target scrape timeout set in the ServiceMonitor
  scrapeTimeout: 30s
  # -- Additional labels to add to ServiceMonitor objects
  extraLabels: {}
  # -- Relabel config for the ServiceMonitor, see: https://coreos.com/operators/prometheus/docs/latest/api.html#relabelconfig
  relabelings: []
+ # -- Custom TLS config options
+ tlsConfig: 
+   ca: "" 
+   caFile: ""
+   cert: ""
+   certFile: "" 
+   key: ""
+   keyFile: ""
+   insecureSkipVerify: "" 
+   servername: "" 

prometheusPodMonitor:
  # -- Should a PodMonitor object be installed to scrape this exporter. For prometheus-operator (kube-prometheus) users.
  create: false
  # -- Target scrape interval set in the PodMonitor
  scrapeInterval: 60s
  # -- Target scrape timeout set in the PodMonitor
  scrapeTimeout: 30s
  # -- Additional labels to add to PodMonitor objects
  extraLabels: {}
  # -- Relabel config for the PodMonitor, see: https://coreos.com/operators/prometheus/docs/latest/api.html#relabelconfig
  relabelings: []
+ # -- Custom TLS config options
+ tlsConfig: 
+   ca: "" 
+   caFile: ""
+   cert: ""
+   certFile: "" 
+   key: ""
+   keyFile: ""
+   insecureSkipVerify: "" 
+   servername: "" 

Excerpt of the tlsConfig spec

The full spec can be found here.

tlsConfig:
  description: TLS configuration to use when scraping the target.
  properties:
    ca:
      description: Certificate authority used when verifying server
        certificates.
      properties:
        configMap:
          description: ConfigMap containing data to use for the
            targets.
          properties:
            key:
              description: The key to select.
              type: string
            name:
              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                TODO: Add other useful fields. apiVersion, kind,
                uid?'
              type: string
            optional:
              description: Specify whether the ConfigMap or its
                key must be defined
              type: boolean
          required:
          - key
          type: object
          x-kubernetes-map-type: atomic
        secret:
          description: Secret containing data to use for the targets.
          properties:
            key:
              description: The key of the secret to select from.  Must
                be a valid secret key.
              type: string
            name:
              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                TODO: Add other useful fields. apiVersion, kind,
                uid?'
              type: string
            optional:
              description: Specify whether the Secret or its key
                must be defined
              type: boolean
          required:
          - key
          type: object
          x-kubernetes-map-type: atomic
      type: object
    caFile:
      description: Path to the CA cert in the Prometheus container
        to use for the targets.
      type: string
    cert:
      description: Client certificate to present when doing client-authentication.
      properties:
        configMap:
          description: ConfigMap containing data to use for the
            targets.
          properties:
            key:
              description: The key to select.
              type: string
            name:
              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                TODO: Add other useful fields. apiVersion, kind,
                uid?'
              type: string
            optional:
              description: Specify whether the ConfigMap or its
                key must be defined
              type: boolean
          required:
          - key
          type: object
          x-kubernetes-map-type: atomic
        secret:
          description: Secret containing data to use for the targets.
          properties:
            key:
              description: The key of the secret to select from.  Must
                be a valid secret key.
              type: string
            name:
              description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                TODO: Add other useful fields. apiVersion, kind,
                uid?'
              type: string
            optional:
              description: Specify whether the Secret or its key
                must be defined
              type: boolean
          required:
          - key
          type: object
          x-kubernetes-map-type: atomic
      type: object
    certFile:
      description: Path to the client cert file in the Prometheus
        container for the targets.
      type: string
    insecureSkipVerify:
      description: Disable target certificate validation.
      type: boolean
    keyFile:
      description: Path to the client key file in the Prometheus
        container for the targets.
      type: string
    keySecret:
      description: Secret containing the client key file for the
        targets.
      properties:
        key:
          description: The key of the secret to select from.  Must
            be a valid secret key.
          type: string
        name:
          description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
            TODO: Add other useful fields. apiVersion, kind, uid?'
          type: string
        optional:
          description: Specify whether the Secret or its key must
            be defined
          type: boolean
      required:
      - key
      type: object
      x-kubernetes-map-type: atomic
    serverName:
      description: Used to verify the hostname for the targets.
      type: string
  type: object

Workaround

Create a patch file for the serviceMonitor resource and apply the patch after the deployment via helm.

spec:
  endpoints:
  - interval: 60s
    port: metrics
    scrapeTimeout: 30s
    scheme: https
    tlsConfig:
      caFile: /etc/ssl/ca/ca.crt
      certFile: /etc/ssl/server/tls.crt
      keyFile: /etc/ssl/server/tls.key
      serverName: prometheus-x509-certificate-exporter

Execute the following command to apply the patch:

kubectl patch servicemonitors.monitoring.coreos.com prometheus-x509-certificate-exporter --patch-file serviceMonitor-patch.yaml --type merge

Volker