Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[csi-cinder-plugin] Pods stuck in "ContainerCreating" because a volume could not be formatted and mounted #2588

Open
NymanRobin opened this issue May 6, 2024 · 0 comments
Labels
component/cinder-csi-plugin Indicates that this PR or issue relates to the cinder-csi-plugin kind/bug Categorizes issue or PR as related to a bug.

Comments

@NymanRobin
Copy link

NymanRobin commented May 6, 2024

What happened:
When creating pods with a cinder volume, they may occasionally get stuck in the 'ContainerCreating' state. Upon closer examination, the disk appears with a size of 0B. This issue likely arises because occasionally hypervisors don't send a new volume size to the kernel when a volume is in use.

However, if we force a rescan of the block device on the node by executing echo "1" > /sys/class/block/[device]/device/rescan, the pod will start

What you expected to happen:

The pod to be started normally

How to reproduce it:

Issue it not consistently reproducible

Would it be possible to add the rescan bit during provisioning if it fails, not only on resize? I understand there might be a slight challenge since not all devices support this. However, we can check if the paths exists or if rescan-on-resize is set, could we also set the rescan bit?

Similar to the probeVolume case where "- - -" is set to /scan, we would also set "1" to /rescan as the "- - -" won't cover the case were the size is incorrectly reported

/kind bug

@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label May 6, 2024
@NymanRobin NymanRobin changed the title [Cinder CSI] Pods stuck in "ContainerCreating" because a volume could not be formatted and mounted [csi-cinder-plugin] Pods stuck in "ContainerCreating" because a volume could not be formatted and mounted May 13, 2024
@dulek dulek added the component/cinder-csi-plugin Indicates that this PR or issue relates to the cinder-csi-plugin label May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/cinder-csi-plugin Indicates that this PR or issue relates to the cinder-csi-plugin kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants