Persistent Volumes created by a Storage Class on OpenStack ignore Availability Zones

[alexzose]

We've setup a new OKD 4.13 cluster on OpenStack and noticed that when creating Persistent Volumes via a Storage Class that allows volume creation in multiple availability zones , then the Persistent Volumes ignore the availability zone and have their nodeAffinity section populated with all the availability zones of the OpenStack instead of the one where they were created.

This has as a consequence, when creating Stateful Sets, the pods being scheduled on different availability zones than their volumes and the pod never manages to mount the volume.

It should be mentioned here that the cross-attach on the OpenStack is not enabled.

After searching the documentation of the CSI cinder driver for OpenStack, it states that the (Optional) option ignore-volume-az is disabled by default, which is not the case for the OKD as we have discovered.

In the ConfigMap "cloud-conf" in the "openshift-cluster-csi-drivers" there is the option "ignore-volume-az = yes" and this causes these issues.

After setting the mentioned option to "ignore-volume-az = no" in the ConfigMap "cloud-provider-config" in the "openshift-config" namespace, this setting is propagated to the ConfigMap "cloud-conf" mentioned above, and the issue with the volumes is resolved.

Then, the volume's nodeAffinity is set correctly to the availability zone it was created and the pods are scheduled in the correct availability zone.

The point is that this behavior is not mentioned in the OKD documentation. We think there should be a note or a warning that this options is set to true on OKD and that if cross-attach is not enabled on OpenStack then this pod / volume availability zone mismatch will occur.

Link to the cinder CSI plugin config: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md#block-storage