helm delete old PVC and create another one when run upgrade

[thuandt]

I run Grafana using stable/grafana chart, today when I upgrade grafana to new version. seem helm was deleted my PVC and create another one.

Output of helm version:

$ helm version        
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}

Output of kubectl version:

$ kubectl version         
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:23:29Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): kops in AWS

Output of helm history grafana:

REVISION        UPDATED                         STATUS          CHART           DESCRIPTION                                                 
1               Thu Sep 28 18:36:19 2017        SUPERSEDED      grafana-0.5.0   Install complete                                            
2               Fri Sep 29 10:28:28 2017        SUPERSEDED      grafana-0.5.0   Upgrade complete                                            
....                                         
26              Tue Mar 20 01:49:49 2018        SUPERSEDED      grafana-0.7.0   Upgrade complete                                            
27              Fri Mar 30 00:31:49 2018        SUPERSEDED      grafana-0.8.3   Upgrade complete                                            
28              Fri Mar 30 00:39:48 2018        DEPLOYED        grafana-0.8.3   Upgrade complete   

$ kubectl get pv
pvc-0fb0a98e-3377-11e8-8bcb-06770c064b20   10Gi       RWO            Delete           Bound    monitor/grafana                              gp2                      40m

I upgrade this release many times but it never happend before.

[bacongobbler]

This is working as intended. The only two resources that are not deleted on an upgrade are persistent volumes and namespaces. However, if you don't want a resource to be removed during an upgrade, you can apply an annotation with the value "helm.sh/resource-policy": keep.

[thuandt]

@bacongobbler I understand that, but my persistent volume has been deleted when run helm upgrade which I can't understand what is really happened.

[bacongobbler]

ok. Is there a particular action item here that we need to take action on in core?

[anhpham1509]

@thuandt Can you check the Reclaim Policy of that PV? If it's Delete, then the PV will be deleted when no PVC bind to that PV.

[thuandt]

@anhpham1509 I use stable grafana charts and as I said I upgrade this grafana release many times without problem.

Does helm upgrade will change PV reclaim policy?
Its deleted PV/PVC and create new one, NOT reuse i think.

[anhpham1509]

@thuandt Are you using stable/grafana when do helm upgrade? If yes then maybe there were changes in the chart that make the PV lost.

Also, it's recommended to use a database instead of sqlite while working with grafana.

I also worked with grafana and did not use PV. Instead, I used an AWS RDS instance.

[thuandt]

@anhpham1509 thanks for your comment.
I just want to understand what's cause PV lost during upgrade.
I'm already use Grafana with PostgreSQL since April.

[jascott1]

Hi @thuandt Have you confirmed the reclaim policy? From what I could tell that chart uses storage class and (correctly) lacks a PV chart so it would seem storage class related.

[bacongobbler]

Closing due to inactivity. Please open a new ticket against Helm 3 if this issue persists. Thanks!

[thuandt]

This maybe related to #3208