Trident snapshot objects not deleted with deletionPolicy Retain

[khankth]

FAILURE SCENARIO

1. Source PVC is created with reclaimPolicy: Retain
2. A PVC snapshot.storage.k8s.io/v1beta1 object is created with deletionPolicy: Retain
3. A snapshot pvc is successfully created with reclaimPolicy: Retain and attributes:

dataSource:
    name: volumesnapshot-test-pvc-nfs-retain
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

4. Delete snapshot PVC, deleted related k8s PV and Trident Volume
5. Delete k8s snapshot object and k8s volumeSnapshotContent
6. Trident Snapshot object cannot be deleted:
   a. Tridentctl get snapshot list shows the snapshot in the list
   b. Tridentctl get snapshot throws error:
   tctl get snapshot snapshot-abb03b32-432c-4c63-b550-a571050b787e

Error: could not get snapshot snapshot-abb03b32-432c-4c63-b550-a571050b787e: 404 Not Found
command terminated with exit code 1

c. Tridentctl delete snapshot throws error:

Error: could not delete snapshot snapshot-abb03b32-432c-4c63-b550-a571050b787e: 404 Not Found
command terminated with exit code 1

7. Delete trident PV, delete command runs successsfully but PV is still pending in Trident with "deleting" state

If this is a bug please look into it, otherwise, this is causing alot of leftover snapshot objects in the trident. Please provide instructions/guidance now on how to remove the trident snapshot objects with the deletePolicy is Retain?

SUCCESS SCENARIO

1. Source PVC is reated with reclaimPolicy: Delete
2. A PVC snapshot.storage.k8s.io/v1beta1 object is created with deletionPolicy: Delete
3. A snapshot pvc is successfully created with reclaimPolicy: Delete and attributes:

dataSource:
    name: volumesnapshot-test-pvc-nfs-delete
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

4. Delete snapshot PVC, related PV is deleted
5. Delete k8s source PVC, PV is pending in Trident with "deleting" state
6. Delete k8s snapshot object, k8s volumeSnapshotContent, Trident PV and Trident Snapshot objects are auto removed

Trident Snapshot objects are successfully deleted with deletionRetain: Delete
In this case they are auto-removed after deletion of volumesnapshots.snapshot.storage.k8s.io

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: [20.07.01]
• Trident installation flags used: [tctl install --k8s-timeout 30m --csi --use-custom-yaml --silence-autosupport]
• Container runtime: [ Docker 19.03.13-CE]
• Kubernetes version: [1.19.2]
• Kubernetes orchestrator: [k8s kubeadm]
• OS: [ RHEL 7.8]
• NetApp backend types: [ontap-san, ontap-nas]

[khankth]

Hi @gnarl, Any update on this?

[scaleoutsean]

This issue has a more pronounced effect on Element OS (probably E-Series too) because the maximum number of snapshots on these platforms is lower than ONTAP: while on ONTAP the "only" problem is leftover snapshots, on Element OS snapshot protection stops working because without automated expiration the maximum number of snapshots is reached within days.

[gnarl]

@khankth,

I think the problem here is the volumeSnapshotContent has the deletionPolicy of "Retain" and it should be updated to "Delete" prior to deleting the volumeSnapshot and volumeSnapshotConent if the goal is to actually delete the snapshot. This is different than how deleting a PV with the reclaimPolicy set to "Retain" works. In that case, K8S forces you to set edit the PV and change the reclaimPolicy to "Delete" before you can successfully delete the PV.

I'll talk to the team about how to handle this use case.

[balaramesh]

@khankth when deleting a snapshot with tridentctl, you will need to use it as follows:

 tridentctl delete snapshot -h
Delete one or more volume snapshots from Trident
 
Usage:
  tridentctl delete snapshot <volume/snapshot> [<volume/snapshot>...] [flags]
 
Aliases:
  snapshot, s, snap, snapshots
 
Flags:
      --all             Delete all snapshots
  -h, --help            help for snapshot
      --volume string   Delete all snapshots in volume
 
Global Flags:
  -d, --debug              Debug output
  -n, --namespace string   Namespace of Trident deployment
  -o, --output string      Output format. One of json|yaml|name|wide|ps (default)
  -s, --server string      Address/port of Trident REST interface


# tridentctl delete snapshot snapshot-9c214e0f-547d-41b5-923d-a075bcc97d8a pvc-fb6eabff-93b4-4df5-b5ab-943bcda44ad8 -n trident
Error: could not delete snapshot snapshot-9c214e0f-547d-41b5-923d-a075bcc97d8a: 404 Not Found
command terminated with exit code 1


# tridentctl delete snapshot pvc-fb6eabff-93b4-4df5-b5ab-943bcda44ad8/snapshot-9c214e0f-547d-41b5-923d-a075bcc97d8a -n trident


# tridentctl get snapshot -n trident
+------+--------+
| NAME | VOLUME |
+------+--------+
+------+--------+

In addition, the other error you have pointed out [tridentctl get snapshot doesn't return the details of the snapshot] has been fixed with v21.04 in this commit.