Grafana does not reload its TLS server certificate data

[sysarch-repo]

What happened:
After the start, the Grafana pod reads the PKI information from the file(s) configured in grafana.ini. The running Grafana pod does not seem to monitor the update of these files and so the browser accessing the Grafana UI reports an expired certificate even when the information in the files (e.g. /etc/grafana/pki/tls.crt) contains a refreshed and valid certificate data.

What you expected to happen:
Grafana should regularly check the files storing the TLS cert and key for updates and utilize the updated data on the fly, i.e. without the need to restart the Grafana pod.

How to reproduce it (as minimally and precisely as possible):
Configure Grafana with a certificate that has a short validity of e.g. 2h. Start the Grafana pod and update the file storing the TLS cert (e.g. /etc/grafana/pki/tls.crt) with a refreshed certificate data (validity e.g. 10h) while keeping the Grafana pod running. Observe the PKI files in the Grafana pod's namespace being updated with the new certificate data (and the new validity) while the browser connecting to the Grafana UI will report an expired certificate 2h after the pod start, i.e. when the cert loaded at the start of the Grafana pod has expired.

Anything else we need to know?:
Jetstack cert-manager and K8S Secret are useful in automating the test case

Environment:

• Grafana version: 7.4.1-ubuntu
• OS Grafana is installed on: Ubuntu 20.04
• User OS & Browser: Microsoft Edge
• Others: TLS certificate and key supplied from cert-manager 1.4.0

[ying-jeanne]

The current design is we load config.ini just one time at the server start time, It is a good idea to make it updatable on the fly without restart the server, The ticket should be a feature request.

[frebib]

One small step towards this would be reloading all certificates and file references from the configuration on SIGHUP (or some other signal). Restarting Grafana every few hours/days isn't scalable and makes practical mTLS basically impossible as is currently implemented.

[sysarch-repo]

@frebib, @ying-jeanne - a simple approach IMO would be a task that regularly checks the PKI data (ca cert, tls cert, tls key) mounted in the Grafana pod namespace and - upon change - updates the internal TLS config of the Grafana server endpoint.

If running on Kubernetes, the Grafana pod - if supporting the liveness probe - could also simply send a negative response to the liveness probe upon the expiry of the server certificate. With this, Kubernetes re-starts the Grafana pod that picks up the recent PKI data.

[frebib]

This makes the assumption that Grafana is running in Kubernetes, which isn't always true. Any software should operate regardless of it's environment (i.e. on baremetal, in some other orchestration system, something else that doesn't yet exist). Moreover, polling for changes is unnecessary load on the server when the configuration hasn't changed. It may also poll too late for certificates that are renewed but the old one already expired.
Looking through the code, I don't think it's quite as simple as "update the internal .." because the configuration seems to be quite static. How configuration loading is handled and tied to the lifecycle of the datasource/other entity that loads external configuration needs to be refactored before any kind of dynamic reloaded can be implemented

[sysarch-repo]

All true. Fortunately, the Grafana services run very well behind a "deployment-specific prefix" so it can operate seamlessly behind a proxy that acts as the TLS server endpoint to the external clients. The proxy routes the "deployment" specific path (prefix) to the Grafana pod using HTTP, i.e. without TLS. This works nicely in my environment. If TLS would be required also between the proxy and the Grafana server endpoint, then a service mesh could / would deliver that. It's a bit of an architecture but in the end the operational experience and the security level are IMO good. Thoughts?

[gelicia]

Hello, as you may have heard, we are transitioning away from using discussions to discuss feature requests. Due to the age and number of responses to this discussion, we are deciding to close it. If this is something you would like to see in Grafana, feel free to open an issue so the discussion can continue. Thank you!