[FEATURE] Support periodic or on-demand full backups to enhance backup reliability

[derekbit]

Is your feature request related to a problem? Please describe (👍 if you like this request)

In the existing Longhorn backup system, the initial backup is a full backup, while subsequent backups are incremental. If any block becomes corrupted, all backup revisions relying on that block will also be corrupted as well. An approach to address the issue might perform a full backup after every N incremental backups. This method can decreases the likelihood of backup corruption, enhancing the overall reliability of the backup process.

Current implementation

A possible solution

Ref: https://www.architecting.it/blog/incrementals-forever-or-synthetic-fulls/

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[derekbit]

I would highlight the feature that can help improve the resilience to the silent corruption of a backup server.
cc @innobead

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• PRs:

  • feat(backup): add backup parameter types go-common-libs#26
  • feat: support full backup backupstore#166
    - feat(backup): support full backup longhorn-engine#1061
    - feat(backup): support full backup longhorn-instance-manager#459
    - feat(backup): support recurring and ondemand full backup longhorn-manager#2709
  • CRD: feat(CRD): update CRD for full backup #8290
  • LEP: feat(lep): add recurring and manual full backup support #8186
  • Website: TBD
  • UI: ticket: [FEATURE UI] Add parameters support to the Backup and RecurringJob #8291

Test

• Create a Volume with 10MB, and write 4MB data
• Create NFS backupstore

Test 1: Full Backup

• Take a Backup from UI
• Go into NFS backupstore to replace one of the block content with random data
• Restore the Volume should fail.
• Create a full Backup by
  • http://${LONGHORN_UI_ENDPOINT}/v1/volumes/${VOLUME}
  • Click snapshotBackup
    • name: the previous snapshot name
    • parameters: {"backup-mode": "full"}
• Restore the Volume should succeed.
• The first backup
  • .Status.NewlyUploadedDataSize: "4194304"
  • .Status.ReUploadedDataSize: "0"
• The second backup
  • .Status.NewlyUploadedDataSize: "0"
  • .Status.ReUploadedDataSize: "4194304"

Test 2: Recurring Full Backup - Alaways incremental

• Cleanup all the Backup/BackupVolume
• Prepare RecurringJob YAML (every 1 min)

  apiVersion: longhorn.io/v1beta1
  kind: RecurringJob
  metadata:
  name: backup-job
  namespace: longhorn-system
  spec:
  cron: "* * * * ?"
  task: "backup"
  groups:
  - default
  retain: 100
  concurrency: 1
  parameters:
      full-backup-interval: "2"
  

• Change the interval to 0, full-backup-interval: "0"
• Create the RecurringJob
• Wait for 2 Backup to be created
• k describe lhb -n longhorn-system | grep -A 2 "Parameters"
  • Both should be empty
• k describe lhb -n longhorn-system | grep -A 5 "Newly"
  • The first one should be "4194304"
  • The second one should be "0"
• k describe lhb -n longhorn-system | grep -A 5 "Re Uploaded Data Size"
  • The first one should be "0"
  • The second one should be "0"
• Cleanup all the Backup/BackupVolume

Test 3: Recurring Full Backup - Alaways Full

• Change the interval to 1, full-backup-interval: "1"
• Create the RecurringJob
• Wait for 2 Backup to be created
• k describe lhb -n longhorn-system | grep -A 2 "Parameters"
  • Both should be {"backup-mode": full}
• k describe lhb -n longhorn-system | grep -A 5 "Newly"
  • The first one should be "4194304"
  • The second one should be "0"
• k describe lhb -n longhorn-system | grep -A 5 "Re Uploaded Data Size"
  • The first one should be "0"
  • The second one should be "4194304"
• Cleanup all the Backup/BackupVolume

Test 4: Recurring Full Backup - Every N times

• Change the interval to 2, full-backup-interval: "2"
• Create the RecurringJob
• Wait for 4 Backup to be created
• k describe lhb -n longhorn-system | grep -A 2 "Parameters"
  • The 1st and 3th should be {"backup-mode": full}
  • The 2st and 4th should be empty
• k describe lhb -n longhorn-system | grep -A 5 "Newly"
  • The 1st should be "4194304"
  • The 2nd, 3rd and 4th should be "0"
• k describe lhb -n longhorn-system | grep -A 5 "Re Uploaded Data Size"
  • The 1st should be "0"
  • The 2nd, 3rd and 4th should be "4194304"

[derekbit]

The current implementation is adding a backup-mode label within a recurring job. User must establish two separate recurring jobs for incremental and full backups to regulate the frequency of full backups. This method complicates management.

However, following discussions with @ChanYiLin and @c3y1huang, we can record the frequency, period, or count within recurring job or backup volume, and the solution could simplify the configuration, requiring only a single recurring job.