Can't restore backup of pvc/pv using eks/ebs

[corang]

Trying to test out backup/restore with velero using ebs snapshots for volume backups. Snapshots are taken by velero in AWS and show in the console, and the backups themselves don't have any errors. Stood up a second cluster to try a blue/green type of backup/restore and tried to restore a volume that had a snapshot taken, but got this error mentioning zone. I saw that this used to be an issue but was apparently fixed awhile ago? #4332

Does it have something to do with the cluster not having the volume snapshot crds installed? Any help would be appreciated

Restore Log

time="2024-04-02T19:03:54Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:535" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:54Z" level=info msg="Starting restore of backup velero/velero-uds-gitlab-backup-20240402030042" logSource="pkg/restore/restore.go:430" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=customresourcedefinitions.apiextensions.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=customresourcedefinitions.apiextensions.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=namespaces restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=storageclasses.storage.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2185" resource=volumesnapshotclass.snapshot.storage.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2185" resource=volumesnapshotcontents.snapshot.storage.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2185" resource=volumesnapshots.snapshot.storage.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=datauploads.velero.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Resource 'persistentvolumes' will be restored at cluster scope" logSource="pkg/restore/restore.go:2266" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Resource 'persistentvolumeclaims' will be restored into namespace 'gitlab'" logSource="pkg/restore/restore.go:2264" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=serviceaccounts restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=secrets restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=configmaps restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=limitranges restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=pods restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=replicasets.apps restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2185" resource=clusterclasses.cluster.x-k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=endpoints restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=services restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=apiservices.apiregistration.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=clusterrolebindings.rbac.authorization.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=clusterroles.rbac.authorization.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=cninodes.vpcresources.k8s.aws restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=controllerrevisions.apps restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=csidrivers.storage.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=csinodes.storage.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=customresourcedefinitions.apiextensions.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=deployments.apps restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=endpointslices.discovery.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=eniconfigs.crd.k8s.amazonaws.com restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=events restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=flowschemas.flowcontrol.apiserver.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=horizontalpodautoscalers.autoscaling restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=ingressclasses.networking.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=ingressclassparams.elbv2.k8s.aws restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=jobs.batch restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=mutatingwebhookconfigurations.admissionregistration.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=networkpolicies.networking.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=nodes restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=packages.uds.dev restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=poddisruptionbudgets.policy restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=policyendpoints.networking.k8s.aws restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=priorityclasses.scheduling.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=prioritylevelconfigurations.flowcontrol.apiserver.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=servicemonitors.monitoring.coreos.com restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=statefulsets.apps restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=storageclasses.storage.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=validatingwebhookconfigurations.admissionregistration.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=virtualservices.networking.istio.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because the restore spec excludes it" logSource="pkg/restore/restore.go:2202" resource=volumeattachments.storage.k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2185" resource=clusterbootstraps.run.tanzu.vmware.com restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2185" resource=clusters.cluster.x-k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:2185" resource=clusterresourcesets.addons.cluster.x-k8s.io restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="waiting informer cache sync ..." logSource="pkg/restore/restore.go:604" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:1050" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Find VolumeInfo for PV gitaly-volume." logSource="pkg/restore/restore.go:1228" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Restoring persistent volume from snapshot." logSource="pkg/restore/restore.go:2453" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Restored 1 items out of an estimated total of 4 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:807" name=gitaly-volume namespace= progress= resource=persistentvolumes restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Find VolumeInfo for PV pvc-2fc863a2-454f-4cb2-9809-fdda9a7218b4." logSource="pkg/restore/restore.go:1228" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="restore status includes excludes: <nil>" logSource="pkg/restore/restore.go:1342" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:1356" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:68" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:79" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Attempting to restore PersistentVolume: pvc-2fc863a2-454f-4cb2-9809-fdda9a7218b4" logSource="pkg/restore/restore.go:1513" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Restored 2 items out of an estimated total of 4 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:807" name=pvc-2fc863a2-454f-4cb2-9809-fdda9a7218b4 namespace= progress= resource=persistentvolumes restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Find VolumeInfo for PV pvc-d115ce88-a908-4441-9eb9-45d6fc237b27." logSource="pkg/restore/restore.go:1228" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="restore status includes excludes: <nil>" logSource="pkg/restore/restore.go:1342" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing item action for persistentvolumes" logSource="pkg/restore/restore.go:1356" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:68" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:79" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Attempting to restore PersistentVolume: pvc-d115ce88-a908-4441-9eb9-45d6fc237b27" logSource="pkg/restore/restore.go:1513" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Restored 3 items out of an estimated total of 4 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:807" name=pvc-d115ce88-a908-4441-9eb9-45d6fc237b27 namespace= progress= resource=persistentvolumes restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Getting client for /v1, Kind=PersistentVolumeClaim" logSource="pkg/restore/restore.go:1050" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="restore status includes excludes: <nil>" logSource="pkg/restore/restore.go:1342" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1356" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Adding PV gitaly-volume as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Skipping persistentvolumes/gitaly-volume because it's already been restored." logSource="pkg/restore/restore.go:1191" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1356" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing ChangePVCNodeSelectorAction" cmd=/velero logSource="pkg/restore/change_pvc_node_selector.go:66" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Done executing ChangePVCNodeSelectorAction" cmd=/velero logSource="pkg/restore/change_pvc_node_selector.go:94" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1356" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:68" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Done executing ChangeStorageClassAction" cmd=/velero logSource="pkg/restore/change_storageclass_action.go:79" pluginName=velero restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Attempting to restore PersistentVolumeClaim: repo-data-gitlab-gitaly-0" logSource="pkg/restore/restore.go:1513" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="the managed fields for gitlab/repo-data-gitlab-gitaly-0 is patched" logSource="pkg/restore/restore.go:1714" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Restored 4 items out of an estimated total of 4 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:807" name=repo-data-gitlab-gitaly-0 namespace=gitlab progress= resource=persistentvolumeclaims restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Waiting for all pod volume restores to complete" logSource="pkg/restore/restore.go:660" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Done waiting for all pod volume restores to complete" logSource="pkg/restore/restore.go:676" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Waiting for all post-restore-exec hooks to complete" logSource="pkg/restore/restore.go:680" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="Done waiting for all post-restore exec hooks to complete" logSource="pkg/restore/restore.go:688" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="hookTracker: map[], hookAttempted: 0, hookFailed: 0" logSource="pkg/restore/restore.go:695" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=error msg="Cluster resource restore error: error executing PVAction for persistentvolumes/gitaly-volume: rpc error: code = Unknown desc = operation error EC2: CreateVolume, https response error StatusCode: 400, RequestID: 0b022581-9ec0-46c5-8253-ea8a042b00e1, api error MissingParameter: The request must contain the parameter zone" logSource="pkg/controller/restore_controller.go:576" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=warning msg="Cluster resource restore warning: could not restore, PersistentVolume \"pvc-2fc863a2-454f-4cb2-9809-fdda9a7218b4\" already exists. Warning: the in-cluster version is different than the backed-up version" logSource="pkg/controller/restore_controller.go:587" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=warning msg="Cluster resource restore warning: could not restore, PersistentVolume \"pvc-d115ce88-a908-4441-9eb9-45d6fc237b27\" already exists. Warning: the in-cluster version is different than the backed-up version" logSource="pkg/controller/restore_controller.go:587" restore=velero/gitaly-pv-2
time="2024-04-02T19:03:55Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:594" restore=velero/gitaly-pv-2

[blackpiglet]

What is the version of the Velero and AWS plugin you are using?

[corang]

velero: v1.13.1
velero-plugin-for-aws: v1.9.1

[blackpiglet]

The issue you mentioned is not the reason.

Please check the backup including PV's information.
The AZ is read from the PV's label.

velero/pkg/backup/item_backupper.go

Lines 565 to 579 in ea5a89f

	pvFailureDomainZone, labelFound := pv.Labels[zoneLabel]
	if !labelFound {
	log.Infof("label %q is not present on PersistentVolume, checking deprecated label...", zoneLabel)
	pvFailureDomainZone, labelFound = pv.Labels[zoneLabelDeprecated]
	if !labelFound {
	var k string
	log.Infof("label %q is not present on PersistentVolume", zoneLabelDeprecated)
	k, pvFailureDomainZone = zoneFromPVNodeAffinity(pv, awsEbsCsiZoneKey, azureCsiZoneKey, gkeCsiZoneKey, cinderCsiZoneKey, manilaCsiZoneKey, zoneLabel, zoneLabelDeprecated)
	if pvFailureDomainZone != "" {
	log.Infof("zone info from nodeAffinity requirements: %s, key: %s", pvFailureDomainZone, k)
	} else {
	log.Infof("zone info not available in nodeAffinity requirements")
	}
	}
	}