Slow restore from system backup

[comminutus]

Hi there, I've got a volume which has about 700Gi of pictures and videos. Whenever I recreate my k3s cluster and restore this volume from a system backup, the restore process takes about 60-70 hours. The backup target is a NAS and the machines are connected via 1G copper. Copying the same data set over SSH takes only a few hours. I'm using ext4 for the volume.

Why does it take this long to recover a volume?

[innobead]

cc @ChanYiLin

[derekbit]

Can you show the volume.cfg in the backstore and the result of kubectl -n longhorn-system get volume <name> -o yaml? If the compression method is gzip, the restore speed will be very slow.

[comminutus]

@derekbit thanks for replying, I don't use any special backstore, just the default, so I don't think I have a volume.cfg file (none that find reveals anyway. What does the default backstore use?

Here is the contents of the volume's yaml:

apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  annotations:
    longhorn.io/last-system-restored: nfs://<redacted>:/volume1/k3s/longhorn/backupstore/system-backups/v1.6.0/20240222
    longhorn.io/last-system-restored-at: "2024-02-29T16:57:58Z"
    longhorn.io/last-system-restored-backup: nfs://<redacted>:/volume1/k3s/longhorn?backup=backup-e57a3377db944479&volume=pvc-3b05391b-96e4-4f25-bcf8-54bf6ee6e4ea
  creationTimestamp: "2024-02-29T16:58:49Z"
  finalizers:
  - longhorn.io
  generation: 2
  labels:
    apps: <redacted>
    backup-volume: pvc-3b05391b-96e4-4f25-bcf8-54bf6ee6e4ea
    longhornvolume: pvc-3b05391b-96e4-4f25-bcf8-54bf6ee6e4ea
    recurring-job-group.longhorn.io/backups: enabled
    recurring-job-group.longhorn.io/snapshots: enabled
    setting.longhorn.io/remove-snapshots-during-filesystem-trim: ignored
    setting.longhorn.io/replica-auto-balance: ignored
    setting.longhorn.io/snapshot-data-integrity: ignored
  name: pvc-3b05391b-96e4-4f25-bcf8-54bf6ee6e4ea
  namespace: longhorn-system
  resourceVersion: "9629"
  uid: ec53bb41-70ed-4ce6-84bb-1c83f5dd6696
spec:
  Standby: false
  accessMode: rwo
  backendStoreDriver: v1
  backingImage: ""
  backupCompressionMethod: lz4
  dataEngine: v1
  dataLocality: disabled
  dataSource: ""
  disableFrontend: true
  diskSelector:
  - sata
  encrypted: false
  engineImage: ""
  fromBackup: nfs://<redacted>:/volume1/k3s/longhorn?backup=backup-e57a3377db944479&volume=pvc-3b05391b-96e4-4f25-bcf8-54bf6ee6e4ea
  frontend: blockdev
  image: longhornio/longhorn-engine:v1.6.0
  lastAttachedBy: ""
  migratable: false
  migrationNodeID: ""
  nodeID: <redacted>
  nodeSelector: []
  numberOfReplicas: 3
  offlineReplicaRebuilding: disabled
  replicaAutoBalance: ignored
  replicaDiskSoftAntiAffinity: ignored
  replicaSoftAntiAffinity: ignored
  replicaZoneSoftAntiAffinity: ignored
  restoreVolumeRecurringJob: ignored
  revisionCounterDisabled: false
  size: "1074815565824"
  snapshotDataIntegrity: ignored
  snapshotMaxCount: 250
  snapshotMaxSize: "0"
  staleReplicaTimeout: 20
  unmapMarkSnapChainRemoved: ignored
status:
  actualSize: 0
  cloneStatus:
    snapshot: ""
    sourceVolume: ""
    state: ""
  conditions:
  - lastProbeTime: ""
    lastTransitionTime: "2024-02-29T16:58:51Z"
    message: ""
    reason: ""
    status: "False"
    type: WaitForBackingImage
  - lastProbeTime: ""
    lastTransitionTime: "2024-02-29T16:58:51Z"
    message: ""
    reason: ""
    status: "False"
    type: TooManySnapshots
  - lastProbeTime: ""
    lastTransitionTime: "2024-02-29T16:58:51Z"
    message: ""
    reason: ""
    status: "True"
    type: Scheduled
  - lastProbeTime: ""
    lastTransitionTime: "2024-02-29T16:58:55Z"
    message: ""
    reason: RestoreInProgress
    status: "True"
    type: Restore
  currentImage: longhornio/longhorn-engine:v1.6.0
  currentMigrationNodeID: ""
  currentNodeID: <redacted>
  expansionRequired: false
  frontendDisabled: true
  isStandby: false
  kubernetesStatus:
    lastPVCRefAt: ""
    lastPodRefAt: ""
    namespace: default
    pvName: pvc-3b05391b-96e4-4f25-bcf8-54bf6ee6e4ea
    pvStatus: Bound
    pvcName: <redacted>
    workloadsStatus: null
  lastBackup: backup-76585d9b41264c6d
  lastBackupAt: "2024-02-29T03:00:16Z"
  lastDegradedAt: ""
  offlineReplicaRebuildingRequired: false
  ownerID: <redacted>
  pendingNodeID: ""
  remountRequestedAt: ""
  restoreInitiated: true
  restoreRequired: true
  robustness: healthy
  shareEndpoint: ""
  shareState: ""
  state: attached

[derekbit]

I don't think I have a volume.cfg file (none that find reveals anyway. What does the default backstore use?

You can find the volume.cfg in the backupstore directory of either s3 or nfs.
I want to check the backups is actually compressed by gzip or lz4.

[comminutus]

So in my NFS backup store directory, the system backup has a system-backup.cfg file but doesn't mention anything about compression. There is also a system-backup.zip file. This is all under <backup store dir>/system-backups/v1.6.0/<system backup name>/. There is another directory at <backup store dir>/volumes/ which contain the following directories: 0c 0d 1c 23 2a 36 42 44 4e 55 59 70 72 74 79 81 85 9e ad b2 b9 bb c6 cf d7 f8 but these don't seem to correspond with the GUID of the PVC (3b05391b-96e4-4f25-bcf8-54bf6ee6e4ea). None of the other GUIDS from the YAML posted above seem to match up either. How can I find the correct directory/backup of the backup used from the system backup I am restoring from?

[comminutus]

Hi @derekbit I was able to find the volume.cfg file with a sudo find . -type f -name 'volume.cfg' and filtering out the results. The compression method listed is lz4.

What else might cause such a slow recovery time?

[derekbit]

@comminutus
Thanks for the information. The performance of a restore process is determined by

• The decompression method (lz4 is powerful, so it might be not an issue)
• Concurrent restoration thread
• The disk performance of backupstore
• The disk performance of the k8s cluster
• The network performance between the backupstore and the cluster

[comminutus]

Thanks @derekbit , I found the concurrent restoration thread setting was set to 1, so that probably had something to do with it. Each node in my cluster has 4 cores. Would a setting of 3 or 4 be wise instead?

[derekbit]

Yes, they are both good.

[comminutus]

I will give 3 a shot, thanks again!