[cinder-csi-plugin] can not detach volume

[jmozd]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

When pods are stopped, the Cinder volumes remain in status "reserved", resulting in errors when restarting the pod

What you expected to happen:

It should be possible to start the pod again without error

How to reproduce it:

• pod running with PV attached (i.e. as part of StatefulSet)
• scale down StatefulSet by 1
  • pod is stopped
• scale up StatefulSet by 1
  • pod is started, but remains in "ContainerCreating" state until the Cinder volume's state is changed from "reserved" to "available"

Currently, the issue can be reproduced at will in the local environment.

Anything else we need to know?:

when pod is stopped, controllerplugin, container cinder-csi-plugin shows in its log:

I0216 15:11:38.327846 11 openstack_volumes.go:290] Successfully detached volume: 51f6ac5c-69e0-487a-bce7-f050f11a170c from compute: 0e5c3a64-3306-453d-bf94-6d4156d6dc2e

# openstack volume show 51f6ac5c-69e0-487a-bce7-f050f11a170c
...
status | reserved

After scaling up to have the pod started again, the log reports
E0216 15:17:14.403924 11 controllerserver.go:209] Failed to AttachVolume: failed to attach 51f6ac5c-69e0-487a-bce7-f050f11a170c volume to 0e5c3a64-3306-453d-bf94-6d4156d6dc2e compute: Bad request with: [POST http://**openstack**:8774/v2.1/servers/0e5c3a64-3306-453d-bf94-6d4156d6dc2e/os-volume_attachments], error message: {"badRequest": {"code": 400, "message": "Invalid input received: Invalid volume: Volume 51f6ac5c-69e0-487a-bce7-f050f11a170c status must be available or in-use or downloading to reserve, but the current status is reserved. (HTTP 400) (Request-ID: req-4da2e9ee-2851-43ac-b535-95437cf6d186)"}}

cinder-api.log reports no errors on detach (just "Volume info retrieved successfully", "Begin detaching volume completed successfully", "[...]/action returned with HTTP 202")

Environment:

• openstack-cloud-controller-manager(or other related binary) version:

I0216 14:38:51.167889 11 driver.go:81] Driver: cinder.csi.openstack.org
I0216 14:38:51.167963 11 driver.go:82] Driver version: 2.0.0@v1.29.0

• OpenStack version:
  Cinder 17.4.1

• Others:
  Kubernetes v1.26.11+k3s2
  runtime containerd://1.7.7-k3s1.26
  "image": "registry.k8s.io/provider-os/cinder-csi-plugin:v1.29.0"
  "imageRef": "registry.k8s.io/provider-os/cinder-csi-plugin@sha256:24d09525d215aad993b2319ec37a686d3470997c6f9ea73f30a82c2a53e7604d"

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale