Get error when creating volume snapshot

[i5Js]

It's better explained here

kubernetes-csi/external-snapshotter#300

I hope longhorn will support latest version soon:

time="2023-12-04T14:31:24Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2: \"snapshot controller failed to update snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-0e1ad33d-708e-42d6-b091-95250cac6dc2\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi

[ricsanfre]

After fixing issue #273, that was causing different errors due to ArgoCD auto-sync policies, I am able to reproduce the problem.

How to reproduce the error

• Execute velero full backup

  velero backup create fullbackup --include-namespaces '*' --wait

• Backup ends with "Backup completed with status: PartiallyFailed" message

• Check logs

  velero backup logs fullbackup

VolumeSnapshot error messages are displayed at the end of the logs:

time="2024-02-03T10:22:07Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f: \"snapshot controller failed to update snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/full03022024 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T10:22:08Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-a339b4f4-dcac-49a3-8cb8-1a7ffbaefd5c: \"snapshot controller failed to update snapcontent-a339b4f4-dcac-49a3-8cb8-1a7ffbaefd5c on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-a339b4f4-dcac-49a3-8cb8-1a7ffbaefd5c\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/full03022024 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T10:22:08Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-98181105-aa85-4e05-8e46-c0670a1508ac: \"snapshot controller failed to update snapcontent-98181105-aa85-4e05-8e46-c0670a1508ac on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-98181105-aa85-4e05-8e46-c0670a1508ac\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/full03022024 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T10:22:08Z" level=error msg=0 backup=velero/full03022024 logSource="pkg/controller/backup_controller.go:675"
time="2024-02-03T10:22:08Z" level=error msg=1 backup=velero/full03022024 logSource="pkg/controller/backup_controller.go:675"
time="2024-02-03T10:22:08Z" level=error msg=2 backup=velero/full03022024 logSource="pkg/controller/backup_controller.go:675"

• Same error messages appears in longhorn's csi-snapshotter pod logs:

   I0203 10:20:51.894692       1 event.go:285] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f", UID:"1342dbbf-1767-4676-907b-2a85eee25a5d", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"272872", FieldPath:""}): type: 'Warning' reason: 'SnapshotCreationFailed' Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f: "snapshot controller failed to update snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-2a626df8-48a5-48e1-810c-22f489ab0a7f\": the object has been modified; please apply your changes to the latest version and try again
  

[ricsanfre]

After upgrading to latest longhorn version (1.6), which requires a newer version of csi-external-snapshotter (6.3.2), and latest velero version (1.13.0), the error seems not to be solved

• velero backup get command

Backup process seems not to end. Its status remains "WaitingForPluginOperationsPartiallyFailed"

velero backup get fullagain
NAME        STATUS                                      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
fullagain   WaitingForPluginOperationsPartiallyFailed   5        5          2024-02-03 13:02:27 +0100 CET   29d       default            <none>

• velero backup describe command

velero backup describe fullagain
Name:         fullagain
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.28.2+k3s1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=28

Phase:  WaitingForPluginOperationsPartiallyFailed


Warnings:
  Velero:    message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492: "snapshot controller failed to update snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
             message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: "snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
             message: /VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: "snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
             message: /VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: "snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
             message: /VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: "snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\": the object has been modified; please apply your changes to the latest version and try again". Snapshot controller will retry later.
  Cluster:    <none>
  Namespaces: <none>

Errors:
  Velero:    message: /0
             message: /1
             message: /2
             message: /3
             message: /4
  Cluster:    <none>
  Namespaces: <none>

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-02-03 13:02:27 +0100 CET
Completed:  <n/a>

Expiration:  2024-03-04 13:02:27 +0100 CET

Total items to be backed up:  11882
Items backed up:              11882

Backup Item Operations:  28 of 38 completed successfully, 5 failed (specify --details for more information)
Velero-Native Snapshots: <none included>

• velero backup log command

time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492: \"snapshot controller failed to update snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-2dc16475-3006-4eca-9fa6-4ca809b12492\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: \"snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:12Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: \"snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: \"snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: \"snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." backup=velero/fullagain cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T12:07:13Z" level=error msg=0 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=1 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=2 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=3 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"
time="2024-02-03T12:07:13Z" level=error msg=4 backup=velero/fullagain logSource="pkg/controller/backup_controller.go:678"

and velero logs shows that it keep on tryng the failed operation:

time="2024-02-03T13:43:31Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5: \"snapshot controller failed to update snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-f30bae8c-790d-461a-9be6-a53b63c974b5\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca: \"snapshot controller failed to update snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-17cded38-345b-4a7e-abbf-d1d657cb31ca\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362: \"snapshot controller failed to update snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-3717b965-1ca1-4853-ba75-1f82d8eaf362\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
time="2024-02-03T13:43:31Z" level=warning msg="VolumeSnapshot has a temporary error Failed to create snapshot: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18: \"snapshot controller failed to update snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-754910e0-a98d-4e56-a82c-f5b7b3b6da18\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi

[ricsanfre]

It seems there is a external-snapshotter's PR kubernetes-csi/external-snapshotter#876 that might solve this issue. This PR is merged into main branch but it is not going to be merged in 6.3 release branch (kubernetes-csi/external-snapshotter#876 (comment))

Another PR has been created that merges the changes into 6.3.3 branch: kubernetes-csi/external-snapshotter#974.

Latest version of Longhorn (1.6) does support csi-external-snapshotter 6.3.2. 6.3.3 version is not supported yet.