-
Notifications
You must be signed in to change notification settings - Fork 567
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
doc(enhancement): add recurring and manual full backup support
ref: longhorn/longhorn 7070 Signed-off-by: Jack Lin <jack.lin@suse.com>
- Loading branch information
Showing
1 changed file
with
177 additions
and
0 deletions.
There are no files selected for viewing
177 changes: 177 additions & 0 deletions
177
enhancements/20240314-recurring-and-manual-full-backup-support.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,177 @@ | ||
# 20240314-recurring-and-manual-full-backup-support | ||
|
||
## Summary | ||
|
||
This feature enables Longhorn to create **recurring job** for full backup or **manually trigger** the full backup of the volume. | ||
|
||
### Related Issues | ||
|
||
- Community issue: https://github.com/longhorn/longhorn/issues/7069 | ||
- Improvement issue: https://github.com/longhorn/longhorn/issues/7070 | ||
|
||
## Motivation | ||
|
||
Longhorn always does incremental backup which only backup the newly updated blocks. | ||
There is a chance that the previous backup blocks on the backupstore are corrupted. In this case, users can not restore the volume anymore because Longhorn aborts the restoration when it finds those blocks have different checksum. | ||
|
||
### Goals | ||
|
||
- Add a new preserved label `longhorn.io/backup-mode: full` for the Backup CR to trigger the full backup | ||
- Add a new preserved label `longhorn.io/full-backup-interval` to execute full backup every N incremental backups. | ||
- With this label `longhorn.io/backup-mode: full`, user can create **recurring full backup job** or **manually trigger** the full backup of the volume | ||
- When doing full backup, Longhorn will backup **all the current blocks** of the volume and **overwrite them** on the backupstore even if those blocks already exists on the backupstore. | ||
|
||
## Proposal | ||
|
||
### User Stories | ||
|
||
### User Experience In Detail | ||
|
||
#### Recurring Full Backup - Always | ||
|
||
1. Create a `backup` task type RecurringJob with the label `longhorn.io/backup-mode: full` and assign it to the volume | ||
``` | ||
apiVersion: longhorn.io/v1beta2 | ||
kind: RecurringJob | ||
metadata: | ||
name: recurring-full-backup-per-min | ||
namespace: longhorn-system | ||
spec: | ||
concurrency: 1 | ||
cron: '* * * * *' | ||
groups: [] | ||
labels: | ||
longhorn.io/backup-mode: full | ||
name: recurring-full-backup-per-min | ||
retain: 0 | ||
task: backup | ||
``` | ||
2. The RecurringJob runs and fully backup the volume. | ||
|
||
#### Recurring Full Backup - Every N Incremental Backups | ||
|
||
1. Create a `backup` task type RecurringJob with the label `longhorn.io/full-backup-interval: 5` and assign it to the volume | ||
``` | ||
apiVersion: longhorn.io/v1beta2 | ||
kind: RecurringJob | ||
metadata: | ||
name: recurring-full-backup-per-min | ||
namespace: longhorn-system | ||
spec: | ||
concurrency: 1 | ||
cron: '* * * * *' | ||
groups: [] | ||
labels: | ||
longhorn.io/full-backup-interval: 5 | ||
name: recurring-full-backup-per-min | ||
retain: 0 | ||
task: backup | ||
``` | ||
2. The RecurringJob runs and fully backup the volume every 5 incremental backups. | ||
|
||
#### Manual Full Backup | ||
|
||
1. When creating backup, users can fill in the label `longhorn.io/backup-mode: full`. | ||
2. The backup will be full backup. | ||
3. Maybe adjust the UI to make the process more simple | ||
|
||
## Design | ||
|
||
### Implementation Overview | ||
|
||
#### CRD | ||
|
||
1. **BackupVolume**: add a new status `.Status.BackupCount` to record how many backups have been created. | ||
|
||
2. **Backup**: add a new reserved Longhorn label `longhorn.io/backup-mode: full` and `longhorn.io/full-backup-interval: N`. (NO need to do any change) | ||
- `longhorn.io/backup-mode: full` | ||
- When the Backup CR has such label, it will perform full backup. | ||
- Using label so we can filter Backup by the label to distinguish the full backup and normal backup easily. | ||
- We already store the label to the backupstore when doing backup. Thus, when we pull the Backup from the backupstore in a new cluster, the label will be pulled as well. | ||
- This label is only used when the backup happens and tell the engine/replica to do the full backup. | ||
- Since we already pass `Label` through the grpc call chain from `longhorn-manager`->`longhorn-instance-manager`->`longhorn-engine/replica`->`backupstore`. So we don't need to do much modification. | ||
- `longhorn.io/full-backup-interval: N` | ||
- This label only works when using RecurringJob. | ||
- RecurringJob runs and fully backup the volume every N incremental backups by checking `BackupVolume.Status.BackupCount`. | ||
|
||
Backup CR Example | ||
```yaml | ||
apiVersion: longhorn.io/v1beta2 | ||
kind: Backup | ||
metadata: | ||
name: backup-abcde1234 | ||
namespace: longhorn-system | ||
spec: | ||
snapshot: fake-snapshot | ||
labels: | ||
longhorn.io/backup-mode: full | ||
``` | ||
|
||
#### Backupstore | ||
1. In our implementation, if the Volume has `lastBackup`, we then always perform incremental Backup. | ||
2. Now, if `longhorn.io/backup-mode: full` exists in the label, | ||
- we then pretend the last Backup does not exist and force it to do the full Backup. | ||
- overwrites the block on the backupstore even it already exists. | ||
|
||
#### Webhook | ||
1. Check the options with the prefix `longhorn.io` to prevent from typo. | ||
|
||
### Test plan | ||
|
||
#### Manually Full Backup | ||
1. Create a Volume 4MB and fill in the content. | ||
2. Create a Backup of the Volume. | ||
3. Intentionally replace the content of the first block(2MB) on the backupstore | ||
4. Restore the Volume, and will get error logs like below | ||
``` | ||
[pvc-XXXXXX] time="XXXX" level=error msg="Backup data restore Error Found in Server[gzip: invalid checksum]" | ||
``` | ||
5. Create a full backup from the UI with the label `longhorn.io/backup-mode: full` | ||
6. Restore the backup, this time should work | ||
|
||
#### Recurring Job Full Backup | ||
1. Create a Volume 4MB and fill in the content. | ||
2. Create a Backup of the Volume. | ||
3. Intentionally replace the content of the first block(2MB) on the backupstore | ||
4. Restore the Volume, and will get error logs like below | ||
``` | ||
[pvc-XXXXXX] time="XXXX" level=error msg="Backup data restore Error Found in Server[gzip: invalid checksum]" | ||
``` | ||
5. Create a `backup` task type RecurringJob with the label `longhorn.io/backup-mode: full` and assign it to the volume | ||
``` | ||
apiVersion: longhorn.io/v1beta2 | ||
kind: RecurringJob | ||
metadata: | ||
name: recurring-full-backup-per-min | ||
namespace: longhorn-system | ||
spec: | ||
concurrency: 1 | ||
cron: '* * * * *' | ||
groups: [] | ||
labels: | ||
longhorn.io/longhorn-backup-mode: full | ||
name: recurring-full-backup-per-min | ||
retain: 0 | ||
task: backup | ||
``` | ||
6. Wait for the recurring job to be finished. | ||
7. Restore the backup, this time should work | ||
|
||
|
||
#### Concurrent Backup | ||
|
||
1. Create a Volume 4MB and fill in the content. | ||
2. Create 3 recurring job for every 1 min, two for normal incremental backup and the other one for full backup | ||
3. These 3 recurring job should be triggered at once. | ||
4. Wait for 3 backup to be finished. | ||
5. Restore the last backup of the BackupVolume. | ||
6. The content should be the same as original Volume. | ||
|
||
|
||
### Upgrade strategy | ||
|
||
No need. | ||
|
||
## Note [optional] | ||
|
||
None. |