Thin-Pool Snapshooting Clarification

[jakobmoellerdev]

I noticed that with Thin-Pool Snapshotting enabled, there is a limitation on bound PVCs that I would like to explore: https://github.com/topolvm/topolvm/blob/main/docs/limitations.md#snapshots-should-be-created-only-for-a-bound-pvc

I would like to understand the context of this.

In which case would Thin-Pool Snapshotting not work?

I am trying to reproduce a case where the "CSI Sanity" is skipped and I have a failing snapshot but I never managed to reproduce it. Does "WaitForFirstConsumer" on the storageclass skip this issue or is it irrelevant for scheduling in this case?

I am using Kubernetes Capacity Scheduling, not the scheduler extension of TopoLVM

Could you guide me on a setup on how to reproduce a Snapshot that would fail?

Thanks in advance for any example, tip or guidance!

[peng225]

Hi @jakobmoellerdev

In which case would Thin-Pool Snapshotting not work?

A volume created by TopoLVM is bound to a specific node, and a snapshot of the volume is also bound to the same node.

When you set the volume binding mode of TopoLVM's storage class to WaitForFirstConsumer and create a PVC by using the storage class, the PVC will be pending until it is used by any pod. If you create a snapshot for the PVC, I think the snapshot will not be created properly because the actual volume has not been created yet.

[jakobmoellerdev]

First of all, thanks for your answer!
AFAIK, If I create the snapshot for the PVC, The Snapshot will be unavailable as well until the PVC is bound. This means that the Snapshot cannot be created without the Bound PVC. I tried to replicate this scenario actually and I never got into a faulty state. First the PVC gets bound, after that the snapshot recovers.
Did you verify this manually once? If so, I can try again and show my results here.

[peng225]

@jakobmoellerdev

Did you verify this manually once?

I tried it. As a result, I found that the snapshot became false status as I expected.

0. Setup a topolvm example environment.
   • I followed this instruction.
1. Deploy necessarry CRDs and snapshot controller.
   I temporarily edited the example Makefile as follows and run make prepare-snap.

   EXTERNAL_SNAPSHOTTER_VERSION ?= v6.2.2
   .PHONY: prepare-snap
   prepare-snap: $(KUBECTL)
   	$(KUBECTL) apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(EXTERNAL_SNAPSHOTTER_VERSION)/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
   	$(KUBECTL) apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(EXTERNAL_SNAPSHOTTER_VERSION)/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
   	$(KUBECTL) apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(EXTERNAL_SNAPSHOTTER_VERSION)/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
   	$(KUBECTL) apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(EXTERNAL_SNAPSHOTTER_VERSION)/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
   	$(KUBECTL) apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(EXTERNAL_SNAPSHOTTER_VERSION)/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

2. Deploy VolumeSnapshotClass.

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: topolvm-snapclass
   driver: topolvm.io
   deletionPolicy: Delete

3. Create PVC, which will be pending.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: topolvm-pvc-thin-unbounded
     namespace: default
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi
     storageClassName: topolvm-provisioner-thin
     volumeMode: Filesystem

4. Create a snapshot for the pending PVC.

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: snap2
   spec:
     volumeSnapshotClassName: topolvm-snapclass
     source:
       persistentVolumeClaimName: topolvm-pvc-thin-unbounded

The final status of PVC and the snapshot is as follows.

❯ kubectl get pvc topolvm-pvc-thin-unbounded
NAME                         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS               AGE
topolvm-pvc-thin-unbounded   Pending                                      topolvm-provisioner-thin   10m

❯ kubectl get volumesnapshot snap2 
NAME    READYTOUSE   SOURCEPVC                    SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT   CREATIONTIME   AGE
snap2   false        topolvm-pvc-thin-unbounded                                         topolvm-snapclass                                    9m23s

[jakobmoellerdev]

Just one last question. Isnt it clear that its not ready to use since the PVC is not bound yet (and there is no data). As soon as you bind the PVC (e.g. with a Pod), shouldn't the Snapshot also come up? when the PVC is present?

[peng225]

Isnt it clear that its not ready to use since the PVC is not bound yet (and there is no data).

I agree with it. Do you mean the following part of the current limitation is only saying a trivial thing?

The LVM snapshots are required to be provisioned on the same node as the source logical volume. Therefore, the source PVC must be provisioned before the target so that scheduling decisions can be taken accordingly.

---

As soon as you bind the PVC (e.g. with a Pod), shouldn't the Snapshot also come up? when the PVC is present?

As for TopoLVM, when a unbound PVC is bounded, the snapshot of the PVC will become ready to use.
(Sorry, but I could not understand your point. My reply may be off the mark.)

[jakobmoellerdev]

I agree with it. Do you mean the following part of the current limitation is only saying a trivial thing?

Yes exactly that was point, I think it is obvious that a snapshot cannot be ready if the data used to create the snapshot is not ready yet.

I do think that you answered my question though. Im gonna mark it as an answer. However I think the limitation is not actually a TopoLVM thing, since all CSI implementations will not be able to create a ready to use snapshot if the PVC is not bound.