helm delete incomplete (helm 3)

[planetf1]

I have a helm3 chart deployed on IBM cloud: (from https://github.com/odpi/egeria )

➜  tutorials git:(master) kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vdc-atlas-c876d9468-lrvm4               2/2     Running   0          3d18h
vdc-cp-kafka-0                          1/1     Running   0          3d18h
vdc-cp-zookeeper-0                      1/1     Running   0          3d18h
vdc-egeria-74bbf4cf6b-gvh8d             1/1     Running   0          3d18h
vdc-gaian-deployment-56df6b846d-cbzkb   1/1     Running   0          3d18h
vdc-omrsmonitor-cf6c486c4-lzdls         1/1     Running   0          3d18h
vdc-openldap-7cf7dcdb55-zb8lm           1/1     Running   0          3d18h
vdc-postgresql-5f8898fd84-fc5pv         1/1     Running   0          3d18h
vdc-rangeradmin-558b5fc8d-8xrlh         1/1     Running   0          3d18h
vdc-ui-6d6868d894-szmjq                 1/1     Running   0          3d18h

➜  tutorials git:(master) helm list
NAME	NAMESPACE	REVISION	UPDATED                            	STATUS  	CHART
vdc 	egeria   	1       	2019-09-05 17:39:24.42986 +0100 BST	deployed	vdc-0.1.11

I want to delete the chart as I've made some code updates:

➜  tutorials git:(master) helm delete vdc
Error: uninstallation completed with 1 error(s): could not get apiVersions from Kubernetes: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Ok - I've see this before and am persuing. It may be istio related. That could be a separate github issue...

However

➜  tutorials git:(master) helm list
NAME	NAMESPACE	REVISION	UPDATED	STATUS	CHART

I thought delete failed? Ok so it worked then...

➜  tutorials git:(master) kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vdc-atlas-c876d9468-lrvm4               2/2     Running   0          3d18h
vdc-cp-kafka-0                          1/1     Running   0          3d18h
vdc-cp-zookeeper-0                      1/1     Running   0          3d18h
vdc-egeria-74bbf4cf6b-gvh8d             1/1     Running   0          3d18h
vdc-gaian-deployment-56df6b846d-cbzkb   1/1     Running   0          3d18h
vdc-omrsmonitor-cf6c486c4-lzdls         1/1     Running   0          3d18h
vdc-openldap-7cf7dcdb55-zb8lm           1/1     Running   0          3d18h
vdc-postgresql-5f8898fd84-fc5pv         1/1     Running   0          3d18h
vdc-rangeradmin-558b5fc8d-8xrlh         1/1     Running   0          3d18h
vdc-ui-6d6868d894-szmjq                 1/1     Running   0          3d18h

Ok so it didn't. I've now lost the helm chart install, yet the underlaying resources are still present - this applues to deployments and networks as well

➜  tutorials git:(master) helm version
version.BuildInfo{Version:"v3.0+unreleased", GitCommit:"180db556aaf45f34516f8ddb9ddac28d71736a3e", GitTreeState:"clean", GoVersion:"go1.13"}
➜  tutorials git:(master) kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3+IKS", GitCommit:"66a72e7aa8fd2dbf64af493f50f943d7f7067916", GitTreeState:"clean", BuildDate:"2019-08-23T08:07:38Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
➜  tutorials git:(master)

I believe the delete should fail or succeed but not end up in a half way house?

[planetf1]

As an aside the cause seems to be that on IKS at least, installation of knative causes this scenario (maybe a compat issue... but as above the main focus isn't that kubectl apiversions failed, but what happens when it does.. and if there are other error paths in helm that may result in this inconsistency)

[hickeyma]

@planetf1 Tog get a better idea of the current state, can you provide the output to the following commands:
helm list --all-namespaces

kubectl get secret --all-namespaces -l "owner=helm"

[planetf1]

Unfortunately I cleaned up manually, and got rid of the api error, so I can't get precise further info now. Basically it was relating to an error path, and I don't know how to inject that ...

I can update if I do manage to reproduce it? Other than that can you emulate/introduce the failure? else code inspection.

Otherwise as above we can close and I'll reopen if/when it reoccurs with that extra info.

[planetf1]

Ok - caught it

NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART
lab     egeria      1           2019-09-23 11:06:55.840506 +0100 BST    deployed    lab-0.1.0
~/src/egeria/open-metadata-resources/open-metadata-deployment/charts/lab
NAMESPACE   NAME     TYPE              DATA   AGE
egeria      lab.v1   helm.sh/release   1      85s
NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART
lab     egeria      1           2019-09-23 11:06:55.840506 +0100 BST    deployed    lab-0.1.0

^^ commands before the delete attempt
And after vvv

NAME    NAMESPACE   REVISION    UPDATED STATUS  CHART
NAME    NAMESPACE   REVISION    UPDATED STATUS  CHART

and yet:

➜  lab git:(fixdockermvn) ✗ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
lab-egeria-core-5555fbcb5d-k2hh7   1/1     Running   0          5m3s
lab-egeria-dev-8d57b9474-nkv5d     1/1     Running   0          5m3s
lab-egeria-lake-794877f46-kltzw    1/1     Running   0          5m3s
lab-egeria-ui-7c965ccd5d-vhsln     1/1     Running   0          5m3s
lab-jupyter-7f8957b7f6-2nvn6       1/1     Running   0          5m3s
lab-kafka-0                        1/1     Running   1          5m3s
lab-zookeeper-0                    1/1     Running   0          5m3s

ie all the pods etc are still there...

[planetf1]

It seems as if when that error occurs, the helm object is deleted, but the objects it's associated with are now. I would have expected perhaps the helm object is listed last, or if delete fails it's status goes to broken or similar?

[planetf1]

This is also related to #6361

[hickeyma]

@planetf1 Thanks for this. Can you show the output of the secrets after the delete: kubectl get secret --all-namespaces -l "owner=helm"?

[hickeyma]

@planetf1 Can you show the manifest(s) which use "custom.metrics.k8s.io/v1beta1"?

[planetf1]

➜  lab git:(fixdockermvn) ✗ kubectl get secret --all-namespaces -l "owner=helm"
No resources found
➜  lab git:(fixdockermvn) ✗

[planetf1]

I'm afraid my k8s skills aren;'t up to answering the second question - how would I query this?

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.