Error restoring backup from minio deployed in vmware

[AndreCbrera]

I have been testing a velero + minio in kubernetes, this is my configuration I deployed in k8s:

Velero helm version 4.1.3
MInio helm version 14.1.7

USER-SUPPLIED VALUES:
affinity: {}
annotations: {}
backupsEnabled: true
cleanUpCRDs: false
configMaps: {}
configuration:
  backupStorageLocation:
  - accessMode: ReadWrite
    bucket: velero
    config:
      insecureSkipTLSVerify: true
      region: minio
      s3ForcePathStyle: true
      s3Url: http://minio-api.lab-k8s.map.com/
    credential:
      key: cloud
      name: cloud-credentials
    default: true
    name: velero
    prefix: /
    provider: aws
    validationFrequency: null
  backupSyncPeriod: null
  clientBurst: null
  clientPageSize: null
  clientQPS: null
  defaultBackupStorageLocation: null
  defaultBackupTTL: null
  defaultRepoMaintainFrequency: null
  defaultVolumeSnapshotLocations: null
  defaultVolumesToFsBackup: null
  disableControllers: null
  extraEnvVars: {}
  features: null
  fsBackupTimeout: null
  garbageCollectionFrequency: null
  logFormat: null
  logLevel: null
  metricsAddress: null
  namespace: velero
  pluginDir: null
  profilerAddress: null
  restoreOnlyMode: null
  restoreResourcePriorities: null
  storeValidationFrequency: null
  terminatingResourceTimeout: null
  uploaderType: null
  volumeSnapshotLocation: null
containerSecurityContext: {}
credentials:
  existingSecret: {}
  extraEnvVars: {}
  extraSecretRef: ""
  name: cloud-credentials
  secretContents:
    cloud: |
      [default]
      aws_access_key_id=xxx
      aws_secret_access_key=xxx
  useSecret: true
deployNodeAgent: true
dnsConfig: {}
dnsPolicy: ClusterFirst
extraObjects: []
extraVolumeMounts: []
extraVolumes: []
image:
  imagePullSecrets: []
  pullPolicy: IfNotPresent
  repository: velero/velero
  tag: v1.11.0
initContainers:
- image: vsphereveleroplugin/velero-plugin-for-vsphere:v1.5.1
  imagePullPolicy: IfNotPresent
  name: velero-plugin-for-vsphere
  volumeMounts:
  - mountPath: /target
    name: plugins
- image: velero/velero-plugin-for-aws:v1.7.0
  imagePullPolicy: IfNotPresent
  name: velero-plugin-for-aws
  volumeMounts:
  - mountPath: /target
    name: plugins
kubectl:
  annotations: {}
  containerSecurityContext: {}
  image:
    repository: docker.io/bitnami/kubectl
  labels: {}
  resources: {}
labels: {}
lifecycle: {}
livenessProbe:
  failureThreshold: 5
  httpGet:
    path: /metrics
    port: http-monitoring
    scheme: HTTP
  initialDelaySeconds: 10
  periodSeconds: 30
  successThreshold: 1
  timeoutSeconds: 5
metrics:
  enabled: true
  nodeAgentPodMonitor:
    additionalLabels: {}
    annotations: {}
    autodetect: true
    enabled: false
  podAnnotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "8085"
    prometheus.io/scrape: "true"
  prometheusRule:
    additionalLabels: {}
    enabled: false
    spec: []
  scrapeInterval: 30s
  scrapeTimeout: 10s
  service:
    annotations: {}
    labels: {}
  serviceMonitor:
    additionalLabels: {}
    annotations: {}
    autodetect: true
    enabled: false
nodeAgent:
  affinity: {}
  annotations: {}
  containerSecurityContext: {}
  dnsConfig: {}
  dnsPolicy: ClusterFirst
  extraEnvVars: {}
  extraVolumeMounts: []
  extraVolumes: []
  labels: {}
  lifecycle: {}
  nodeSelector: {}
  podSecurityContext:
    runAsUser: 0
  podVolumePath: /var/lib/kubelet/pods
  priorityClassName: ""
  privileged: true
  resources:
    limits:
      cpu: 1000m
      memory: 1024Mi
    requests:
      cpu: 500m
      memory: 512Mi
  tolerations: []
  useScratchEmptyDir: true
nodeSelector: {}
podAnnotations: {}
podLabels: {}
podSecurityContext: {}
priorityClassName: ""
rbac:
  clusterAdministrator: true
  clusterAdministratorName: cluster-admin
  create: true
readinessProbe:
  failureThreshold: 5
  httpGet:
    path: /metrics
    port: http-monitoring
    scheme: HTTP
  initialDelaySeconds: 10
  periodSeconds: 30
  successThreshold: 1
  timeoutSeconds: 5
resources:
  limits:
    cpu: 1000m
    memory: 512Mi
  requests:
    cpu: 500m
    memory: 128Mi
schedules: {}
serviceAccount:
  server:
    annotations: null
    create: true
    labels: null
    name: velero
snapshotsEnabled: true
terminationGracePeriodSeconds: 3600
tolerations: []
upgradeCRDs: false

The backups works perfectly:

In the first case postgresa01 it is saved correctly in minio:

and this is the restore:

The databases in postgres:

the origin:

the restore:

and this is my backup object in k8s:

# velero describe backup postgresa01 --details
Name:         postgresa01
Namespace:    velero
Labels:       velero.io/storage-location=velero
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.27.1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=27

Phase:  Completed


Namespaces:
  Included:  postgresa
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  velero

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          auto
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  1h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-04-17 12:19:46 +0200 CEST
Completed:  2024-04-17 12:20:01 +0200 CEST

Expiration:  2024-05-17 12:19:46 +0200 CEST

Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - ciliumendpoints.cilium.io
  apps/v1/ControllerRevision:
    - postgresa/my-postgresql-54cb67b546
  apps/v1/StatefulSet:
    - postgresa/my-postgresql
  cilium.io/v2/CiliumEndpoint:
    - postgresa/my-postgresql-0
  discovery.k8s.io/v1/EndpointSlice:
    - postgresa/my-postgresql-fjm7p
    - postgresa/my-postgresql-hl-bqpjn
  networking.k8s.io/v1/NetworkPolicy:
    - postgresa/my-postgresql
  v1/ConfigMap:
    - postgresa/kube-root-ca.crt
  v1/Endpoints:
    - postgresa/my-postgresql
    - postgresa/my-postgresql-hl
  v1/Namespace:
    - postgresa
  v1/PersistentVolume:
    - pvc-661c8fe4-7727-4e38-9033-896b02c4b660
  v1/PersistentVolumeClaim:
    - postgresa/data-my-postgresql-0
  v1/Pod:
    - postgresa/my-postgresql-0
  v1/Secret:
    - postgresa/my-postgresql
    - postgresa/sh.helm.release.v1.my-postgresql.v1
  v1/Service:
    - postgresa/my-postgresql
    - postgresa/my-postgresql-hl
  v1/ServiceAccount:
    - postgresa/default
    - postgresa/my-postgresql

Backup Volumes:
  <error getting backup volume info: DownloadRequest.velero.io "postgresa01-d4c1cbc2-cc7a-4fac-b475-90415ce5439d" is invalid: spec.target.kind: Unsupported value: "BackupVolumeInfos": supported values: "BackupLog", "BackupContents", "BackupVolumeSn
apshots", "BackupItemOperations", "BackupResourceList", "BackupResults", "RestoreLog", "RestoreResults", "RestoreResourceList", "RestoreItemOperations", "CSIBackupVolumeSnapshots", "CSIBackupVolumeSnapshotContents">

and my restore:

velero describe restores postgresa01
Name:         postgresa01
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       Completed
Total items to be restored:  20
Items restored:              20

Started:    2024-04-17 12:39:26 +0200 CEST
Completed:  2024-04-17 12:39:39 +0200 CEST

Warnings:
 Velero:     <none>
 Cluster:  could not restore, CustomResourceDefinition "ciliumendpoints.cilium.io" already exists. Warning: the in-cluster version is different than the backed-up version.
 Namespaces:
   postgresa-restore:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version.
                       could not restore, Endpoints "my-postgresql-hl" already exists. Warning: the in-cluster version is different than the backed-up version.
                       could not restore, Endpoints "my-postgresql" already exists. Warning: the in-cluster version is different than the backed-up version.

Backup:  postgresa01

Namespaces:
 Included:  all namespaces found in the backup
 Excluded:  <none>

Resources:
 Included:        *
 Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
 Cluster-scoped:  auto

Namespace mappings:  postgresa=postgresa-restore

Label selector:  <none>

Or label selector:  <none>

Restore PVs:  true

Existing Resource Policy:   <none>
ItemOperationTimeout:       1h0m0s

Preserve Service NodePorts:  auto

what I'm doing wrong?

thanks

[kaovilai]

Can you run velero describe restores postgresa01 --details?

[AndreCbrera]

This is other restore with the same problem, but it seems the pv is skipped

velero describe restores postgresa10 --details
Name:         postgresa10
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       Completed
Total items to be restored:  20
Items restored:              20

Started:    2024-04-17 17:03:59 +0200 CEST
Completed:  2024-04-17 17:04:11 +0200 CEST

Warnings:
  Velero:     <none>
  Cluster:  could not restore, CustomResourceDefinition "ciliumendpoints.cilium.io" already exists. Warning: the in-cluster version is differ
ent than the backed-up version.
  Namespaces:
    postgresa-restore:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the
 backed-up version.
                        could not restore, Endpoints "my-postgresql-hl" already exists. Warning: the in-cluster version is different than the
 backed-up version.
                        could not restore, Endpoints "my-postgresql" already exists. Warning: the in-cluster version is different than the ba
cked-up version.

Backup:  postgresa10

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage
.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  postgresa=postgresa-restore

Label selector:  <none>

Or label selector:  <none>

Restore PVs:  true

Existing Resource Policy:   <none>
ItemOperationTimeout:       1h0m0s

Preserve Service NodePorts:  auto


Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - ciliumendpoints.cilium.io(failed)
  apps/v1/ControllerRevision:
    - postgresa-restore/my-postgresql-54cb67b546(created)
  apps/v1/StatefulSet:
    - postgresa-restore/my-postgresql(created)
  cilium.io/v2/CiliumEndpoint:
    - postgresa-restore/my-postgresql-0(created)
  discovery.k8s.io/v1/EndpointSlice:
    - postgresa-restore/my-postgresql-fjm7p(created)
    - postgresa-restore/my-postgresql-hl-bqpjn(created)
  networking.k8s.io/v1/NetworkPolicy:
    - postgresa-restore/my-postgresql(created)
  v1/ConfigMap:
    - postgresa-restore/kube-root-ca.crt(failed)
  v1/Endpoints:
    - postgresa-restore/my-postgresql(failed)
    - postgresa-restore/my-postgresql-hl(failed)
  v1/Namespace:
    - postgresa-restore(created)
  v1/PersistentVolume:
    - pvc-661c8fe4-7727-4e38-9033-896b02c4b660(skipped)
  v1/PersistentVolumeClaim:
    - postgresa-restore/data-my-postgresql-0(created)
  v1/Pod:
    - postgresa-restore/my-postgresql-0(created)
  v1/Secret:
    - postgresa-restore/my-postgresql(created)
    - postgresa-restore/sh.helm.release.v1.my-postgresql.v1(created)
  v1/Service:
    - postgresa-restore/my-postgresql(created)
    - postgresa-restore/my-postgresql-hl(created)
  v1/ServiceAccount:
    - postgresa-restore/default(skipped)
    - postgresa-restore/my-postgresql(created
```)

[sseago]

Skipping the PV is normal behavior if there's a snapshot or fs-backup to restore. PVC is restored, and the provisioner will provision a new PV, data should come from snapshot or fs backup.

[AndreCbrera]

But velero do not save the data from one pv ? I do not configure the volumeSnapshotLocation: null May I need configure this section in my values?

[AndreCbrera]

looking the logs more close I saw this:

Snapshots and Error Handling

     time="2024-04-18T13:24:24Z" level=info msg="label \"topology.kubernetes.io/zone\" is not present on PersistentVolume, checking deprecated label..." backup=velero/postgres00 logSource="pkg/backup/item_backupper.go:500" name=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 namespace= persistentVolume=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 resource=persistentvolumes
     time="2024-04-18T13:24:24Z" level=info msg="label \"failure-domain.beta.kubernetes.io/zone\" is not present on PersistentVolume" backup=velero/postgres00 logSource="pkg/backup/item_backupper.go:504" name=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 namespace= persistentVolume=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 resource=persistentvolumes
     time="2024-04-18T13:24:24Z" level=info msg="zone info not available in nodeAffinity requirements" backup=velero/postgres00 logSource="pkg/backup/item_backupper.go:509" name=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 namespace= persistentVolume=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 resource=persistentvolumes

---

     time="2024-04-18T13:24:24Z" level=warning msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/postgres00 logSource="pkg/backup/item_backupper.go:533" name=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 namespace= persistentVolume=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 resource=persistentvolumes volumeSnapshotLocation=default

---

    time="2024-04-18T13:24:24Z" level=info msg="Persistent volume is not a supported volume type for snapshots, skipping." backup=velero/postgres00 logSource="pkg/backup/item_backupper.go:544" name=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 namespace= persistentVolume=pvc-a0dd79c7-8388-4039-ae10-733f9fe37fe9 resource=persistentvolumes

it seems something is going wrong

The pv is created by vsphere StorageClass in K8S, may exist some incompabilities with these technologies?

[AndreCbrera]

It works! I solved implementing a restic mover.

Thanks fellows