Add support for Incremental Backups

[jaapmarcus]

Need it for my clients due to the large backup files and I really like the benefits of making 4 daily backups that are 10 times faster then 1 daily backups

Investigated both: Restic and Borg Backups for support for Incremental backups.

Finally went for Restic due to the advantages:

• Support for Rclone with that it support multiple backends including R2 and S3
• Easier to implement then Borg Backup

Backups are stored Encrypted on the backup server by an unique "Encryption key" unique per user.

Due to permissions issues if you delete the original user you are still able to restore the user how ever if the UID / GUID of the user changes an other restore from the same user is not possible.. Due to the permissions are different.

It is always smart to have always an "normal" backup available.

Need some UI rework as it doesn't have the UI changes implemented yet

[mdlaat]

Restic is great! Using it for > 2 years now on almost all my systems to different destinations. Great choice.

[jaapmarcus]

Restic is great! Using it for > 2 years now on almost all my systems to different destinations. Great choice.

When I wrote it few months a go I tested it on my servers for a while It is about 5% of the time to take the backup when you got it as well to a remote location...

[fra81]

I did not understand when a user is deleted, if is possible to restore it or not
When do you plan to release the feature?

[jaapmarcus]

I did not understand when a user is deleted, if is possible to restore it or not
When do you plan to release the feature?

Yes importing deleting users will work fine as long you have a the encryption key. How ever after the initial import you can't do it again due to the UID / GUID probably got changed...

[jakobbouchard]

Some feedback, from what I tested for now. Had a bit of trouble with getting my VM set up, so some things might be because of that.

• I needed to apt install restic && restic self-update first, which was a bit inconvenient.
• I think I needed to edit the hestiaweb cron using crontab -e -u hestiaweb?
• I didn't know the regular backup options had to be enabled for it to work. It wasn't super clear.
  • It's especially weird if I say local backups are "enabled" but use rclone as the repository for restic.
  • The backup directory option is useless once incremental backups are enabled?
  • Maybe the incremental backup should be a checkbox like the remote backups?
  • I think it should be a choice between compressed or incremental. Both cannot be enabled at once. Simpler to manage, and also clearer for users. It's how it works in WHM (they have Compressed, Uncompressed and Incremental). Removes need for a button in the Backup tab, we "only" need to load a different file/UI.
• I get an error message when adding config via UI, and it's very inconvenient/unclear. Error code: 127 due to $HESTIA not being defined when running with sudo? but only for some scripts I think? feels like something is missing.
  • this works: /usr/local/hestia/bin/v-add-backup-host-restic 'rclone:b2:/bucket/' '30' '8' '5' '3' '-1'
  • this doesn't: /usr/bin/sudo /usr/local/hestia/bin/v-add-backup-host-restic 'rclone:b2:/bucket/' '30' '8' '5' '3' '-1'
  • weird because other HESTIA_CMD exec calls work.
  • missing source /etc/hestiacp/hestia.conf?
• Are the v-backup-users and v-backup-users-restic scripts the same now?
• Some weird UI in the backups browsing. The "parent folder" row is bigger.

I still need to test all restore functions (Web, CLI, etc.), and check most shell scripts and some of the templates.

[jakobbouchard]

Suggested change

	$v_incremental_backups["SNAPSHOTS"] != $_POST["v_snapshots"] ||
	$v_incremental_backups["KEEP_DAILY"] != $_POST["v_keep_daily"] ||
	$v_incremental_backups["KEEP_WEEKLY"] != $_POST["v_keep_weekly"] ||
	$v_incremental_backups["KEEP_MONTHLY"] != $_POST["v_keep_montly"] ||
	$v_incremental_backups["KEEP_YEARLY"] != $_POST["v_keep_yearly"]
	$v_incremental_backups["restic"]["SNAPSHOTS"] != $_POST["v_snapshots"] ||
	$v_incremental_backups["restic"]["KEEP_DAILY"] != $_POST["v_keep_daily"] ||
	$v_incremental_backups["restic"]["KEEP_WEEKLY"] != $_POST["v_keep_weekly"] ||
	$v_incremental_backups["restic"]["KEEP_MONTHLY"] != $_POST["v_keep_montly"] ||
	$v_incremental_backups["restic"]["KEEP_YEARLY"] != $_POST["v_keep_yearly"]

[jaapmarcus]

Some feedback, from what I tested for now. Had a bit of trouble with getting my VM set up, so some things might be because of that.

• I needed to apt install restic && restic self-update first, which was a bit inconvenient.

We need to add it to the installer and probally depends list in control file. Should be no issue ...

• I think I needed to edit the hestiaweb cron using crontab -e -u hestiaweb?

We should add a cronjob on enabling .. Rustic

• I didn't know the regular backup options had to be enabled for it to work. It wasn't super clear.

No it is totally seperate that why I created 2 "boxes"

• It's especially weird if I say local backups are "enabled" but use rclone as the repository for restic.

Only reason to keep it enabled to create manual backups to migrate... (Or maybe you want not all users to backed up with Restic)

• The backup directory option is useless once incremental backups are enabled?
• Maybe the incremental backup should be a checkbox like the remote backups?

See 2 steps

• I think it should be a choice between compressed or incremental. Both cannot be enabled at once. Simpler to manage, and also clearer for users. It's how it works in WHM (they have Compressed, Uncompressed and Incremental). Removes need for a button in the Backup tab, we "only" need to load a different file/UI.

Migrating users between server is probally faster / easier with the "classic" backup method..

• I get an error message when adding config via UI, and it's very inconvenient/unclear. Error code: 127 due to $HESTIA not being defined when running with sudo? but only for some scripts I think? feels like something is missing.

  • this works: /usr/local/hestia/bin/v-add-backup-host-restic 'rclone:b2:/bucket/' '30' '8' '5' '3' '-1'
  • this doesn't: /usr/bin/sudo /usr/local/hestia/bin/v-add-backup-host-restic 'rclone:b2:/bucket/' '30' '8' '5' '3' '-1'
  • weird because other HESTIA_CMD exec calls work.
  • missing source /etc/hestiacp/hestia.conf?

• Are the v-backup-users and v-backup-users-restic scripts the same now?

There are a few minor Changes:

1. Restic version creates a "folder" with the "Hestia data" in /home/user/backups/ in stead a folder in the backup folder

This allows restore of the user / domain when needed ... Without creating them self first ...

3. Databases get dumped in the same folder

• Some weird UI in the backups browsing. The "parent folder" row is bigger.

@AlecRust made some major changes UI in the few months between I made it and now...

I still need to test all restore functions (Web, CLI, etc.), and check most shell scripts and some of the templates.

[jakobbouchard]

Hmm, I was trying to run the backup script and it didn't want to run because BACKUP_SYSTEM was unset. That's why I thought they were related. Maybe I was running the wrong one by accident honestly. It was late haha.

If the classic one is easier for migration, it might be better to keep them separate I guess. Maybe we could add a v-package-user command that would do basically the same as the regular backup, to make migrations easier. cPanel has that, quite useful. That way we could still simplify the UI. But I don't mind keeping both now that I understand the decision behind it.

[jaapmarcus]

Hmm, I was trying to run the backup script and it didn't want to run because BACKUP_SYSTEM was unset. That's why I thought they were related. Maybe I was running the wrong one by accident honestly. It was late haha.

That was a bug. Should be fixed now

[jaapmarcus]

Note to my self include documentation:

rclone:sftp or rclone:sftp/ doesn't work

Use rclone:sftp:/ instead

[jlguerrerosanz]

Hello! How is this going?

One question. I do not understand why I couldn't restore a deleted user and then get all the files.

If I keep my encryption key safe elsewhere or via cmd can I restore a backup or not? Can I restore a backup to a different server? If the encryption is the problem can I turn it off?

Can I deploy a backup to a different server? Maybe have a mirror server to decrease downtime as a failover mechanism.

[jaapmarcus]

Hello! How is this going?

One question. I do not understand why I couldn't restore a deleted user and then get all the files.

Yes with the encryption key you can restore a back up but the owner id are so changed that you can't restore a second time from the same "repo"

If I keep my encryption key safe elsewhere

Yes you still need to keep the encryption in a safe location offcourse

or via cmd can I restore a backup or not?

Yes you can ...

Can I restore a backup to a different server?

Yes you can how ever you need to add the "backup server" as repo to the new server...

If the encryption is the problem can I turn it off?

Restic refuses to backup without a encryption key...

Can I deploy a backup to a different server? Maybe have a mirror server to decrease downtime as a failover mechanism.

Yes you can deploy the backup to different server... Only be care full with duplicated user names..

[jlguerrerosanz]

Thank you @jaapmarcus for your time.

Yes with the encryption key you can restore a back up but the owner id are so changed that you can't restore a second time from the same "repo"

Ok, can I restore a backup of the web and the database but maintain the user intact?

Can I wipe a user but not deleting it? I mean:

rm -Rf /home/username/*

and then rebuild? This way I could start over but retaining the UserID

Or maybe only destroy the database and the website but keep the rest.

Can I change userIDs

The thing is that I may want to restore 3 or 4 times to find the version most up to date that has not been compromised or maybe we have made a mistake.

How should we proceed in such cases?

Maybe I am assuming things are a certain way and they are not.