Cannot delete PVC in terminating state

[winkrs]

What happened?

After deletion of pod, it PVC is getting stuck in Terminating state. Tried to force delete but its not working, even tried to remove finaliser but getting error that that its not allowed as the field is immutable and I'm not allowed to change spec field when I did not even change the spec field.

What did you expect to happen?

I'm expecting that by removing the finaliser the PVC will get deleted.

How can we reproduce it (as minimally and precisely as possible)?

Not sure how to reproduce the steps.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
verison: v1.22.15

Cloud provider

bare metal

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

/sig storage

[carlory]

Can you paste the pvc manifests, events, and log files? @winkrs

[winkrs]

Below are the manifest and error log:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
  creationTimestamp: "2024-02-22T07:35:00Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-02-22T07:38:58Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/created-by: volsync
    volsync.backube/cleanup: df38a2c4-45d0-47f0-a3d0-ec05f259173f
  name: volsync-my-source-src
  namespace: source
  ownerReferences:
  - apiVersion: volsync.backube/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicationSource
    name: my-source
    uid: df38a2c4-45d0-47f0-a3d0-ec05f259173f
  resourceVersion: "40154511"
  uid: 5e22a621-d12c-4ae9-855a-c33b0355be15
spec:
  accessModes:
  - ReadWriteOnce
  dataSource:
    apiGroup: null
    kind: PersistentVolumeClaim
    name: busybox-pvc
  dataSourceRef:
    apiGroup: null
    kind: PersistentVolumeClaim
    name: busybox-pvc
  resources:
    requests:
      storage: 5Gi
  storageClassName: rook-ceph-block
  volumeMode: Filesystem
  volumeName: pvc-5e22a621-d12c-4ae9-855a-c33b0355be15
status:
  phase: Lost

when I try to remove the finalizers field from the pvc, this is what I get from the logs:

2024-04-24T04:28:57.237Z        DEBUG   controllers.ReplicationSource   temporary snapshot reconciled   {"replicationsource": {"name":"my-source","namespace":"source"}, "method": "RsyncTLS", "snapshot": {"name":"volsync-my-source-src","namespace":"source"}, "operation": "unchanged"}
2024-04-24T04:28:57.247Z        ERROR   controllers.ReplicationSource   reconcile failed        {"replicationsource": {"name":"my-source","namespace":"source"}, "method": "RsyncTLS", "pvc": {"name":"volsync-my-source-src","namespace":"source"}, "error": "PersistentVolumeClaim \"volsync-my-source-src\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \t... // 5 identical fields\n  \tVolumeMode:    &\"Filesystem\",\n  \tDataSource:    &{Kind: \"PersistentVolumeClaim\", Name: \"busybox-pvc\"},\n- \tDataSourceRef: &core.TypedLocalObjectReference{Kind: \"PersistentVolumeClaim\", Name: \"busybox-pvc\"},\n+ \tDataSourceRef: nil,\n  }\n"}

As you can see, getting error that I'm not allowed to change spec field when I'm not.

[kays002]

Have you tried deleting the pvc by specifying the the namespace it belongs to? @winkrs

[winkrs]

yes @kays002