Unable to Resize FS after Online PVC Expansion

[frobobbo]

/kind bug

What happened:
Issued command from documentation to expand the PVC:
kubectl patch pvc -n ### config -p '{"spec": {"resources": {"requests": {"storage": "1Gi"}}}}'
PVC and PV both Expanded appropriately

Type Reason Age From Message

---

Warning ExternalExpanding 51m volume_expand waiting for an external controller to expand this PVC
Normal Resizing 51m external-resizer csi.vsphere.vmware.com External resizer is resizing volume pvc-23c3a61d-6205-4f70-aa18-cc8980b80212
Normal FileSystemResizeRequired 51m external-resizer csi.vsphere.vmware.com Require file system resize of volume on node

But when restarting the Pod, I get the following:
Warning FailedMount 79s (x11 over 7m33s) kubelet MountVolume.Setup failed while expanding volume for volume "pvc-23c3a61d-6205-4f70-aa18-cc8980b80212" : Expander.NodeExpand failed to expand the volume : rpc error: code = Internal desc = error when resizing filesystem on volume "66311253-5dbd-4961-9eae-de3304dfc60a" on node: error when resizing filesystem on devicePath /dev/sde and volumePath /var/lib/kubelet/pods/0149b574-815a-4edb-b3b4-d89e851f33f6/volumes/kubernetes.io~csi/pvc-23c3a61d-6205-4f70-aa18-cc8980b80212/mount, err: resize of device /dev/sde failed: exit status 1. resize2fs output: resize2fs 1.46.5 (30-Dec-2021)
resize2fs: Permission denied to resize filesystem
Filesystem at /dev/sde is mounted on /var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/82916293c9b57ae38c43dda00e08b9494cab75a0c9cdbb87c29b53430e391c0a/globalmount; on-line resizing required
old_desc_blocks = 4, new_desc_blocks = 8

What you expected to happen:
I expect the FS to be resized

How to reproduce it (as minimally and precisely as possible):
Stated Above

Anything else we need to know?:
I am running Kubernetes v1.28.5
vsphere-csi-driver: :v3.1.2

Environment:

• csi-vsphere version: :v3.1.2
• Kubernetes version: v1.28.5
• vSphere version: 8.0U2

[frobobbo]

I performed a resize of another volume and it worked as expected. So it doesn't seem to be anything specific to my environment, but perhaps specific to the PVC - what options do I have to correct/resize manually?

[frobobbo]

For anyone that is having this issue, I was able to fix the problem by logging into the worker node that was attempting to start the pod. Manually umount the filesystem and run e2fsck against. Then I killed the pod and let it recreate - this time, the resize completed successfully.
Perhaps that should be some check for problematic filesystems and run e2fsck against it before it attempt to auto-resize

[schmitch]

btw. I had the same issue and what @frobobbo said did help I unmounted the fileystem ran e2fsck against it and killed the pod a few times. the volume was full.