Persist grafana dashboards and make them editable

[alexbrand]

Currently, grafana dashboards are deployed via the dashboard provisioning feature . This is great because the dashboards come pre-loaded with the grafana deployment. The downside, however, is that grafana does not support modifying these dashboards, and persisting the changes (we are also using configmaps, which are read-only).

Ideally, the grafana dashboards can be modified, and the changes persisted to something like a PV+PVC.

[alexbrand]

One potential solution that just occurred to me is to continue using the provisioning feature to provide the out-of-the-box dashboards, but then document or change the grafana deployment to use a PV/PVC for the grafana data location.

There might be better solutions here though, but I am not super familiar with Grafana internals.

[stevesloka]

Should be able to mount the dashboards to the PVC, then use an initContainer to drop in the template files and boostrap.

[jeremyzhou]

Hi, I had used PV/PVC to store grafana data. But the dashboard and datasource will lost when restart the POD.
Any Solution is appreciated!!!

volumeMounts:
- mountPath: /var/lib/grafana
name: grafana-storage
- args:
- --watch-dir=/var/grafana-dashboards
- --grafana-url=http://127.0.0.1:3000
- --watch-dir=/var/additional-dashboards/redis-dashboard
env:
- name: GRAFANA_USER
valueFrom:
secretKeyRef:
key: user
name: kube-prometheus-grafana
- name: GRAFANA_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: kube-prometheus-grafana
- name: CONFVER
value: "3"
image: quay.io/coreos/grafana-watcher:v0.0.8
imagePullPolicy: IfNotPresent
name: grafana-watcher
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/grafana-dashboards
name: grafana-dashboards
- mountPath: /var/additional-dashboards/redis-dashboard
name: redis-dashboard
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: kube-prometheus-grafana
serviceAccountName: kube-prometheus-grafana
terminationGracePeriodSeconds: 30
volumes:
- name: grafana-storage
persistentVolumeClaim:
claimName: grafana-pvc2

ubuntu@ip-10-0-2-43:~$ k get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
grafana-pv2 10Gi RWO Delete Bound monitoring/grafana-pvc2 ssd 1d

ubuntu@ip-10-0-2-43:~$ k get pvc -n monitoring
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
alertmanager-kube-prometheus-db-alertmanager-kube-prometheus-0 Bound pvc-3e1846fb-0271-11e9-9298-0236c8eb6f64 50Gi RWO ssd 42d
grafana-pvc2 Bound grafana-pv2 10Gi RWO ssd 1d
prometheus-kube-prometheus-db-prometheus-kube-prometheus-0 Bound pvc-3e1ffc83-0271-11e9-9298-0236c8eb6f64 50Gi RWO ssd 42d