Implement 'rewrite' command to exclude files from existing snapshots #2731

dionorgua · 2020-05-12T11:51:54Z

What is the purpose of this change? What does it change?

This adds 'rewrite' command that provides a way to remove data from existing snapshots.

This is some sort of preliminary pull requests. It's actually works here on test repo, but lacks tests, etc. Mostly to discuss things

Was the change discussed in an issue or in the forum before?

[ replaces #2720 because it was created from master branch ]
closes #14

Checklist

I have read the Contribution Guidelines
I have enabled maintainer edits for this PR
I have added tests for all changes in this PR
I have added documentation for the changes (in the manual)
There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
I have run gofmt on the code in all commits
All commit messages are formatted in the same style as the other commits in the repo
I'm done, this Pull Request is ready for review

aawsome

Looks mostly good to me 👍
I propose to call the existing code from cmd_forget to actually remove snapshots. That way also a pruning could be directly started from this command which IMO improves the usage for end users.

changelog/unreleased/issue-14

cmd/restic/cmd_rewrite.go

aawsome · 2020-07-10T11:06:23Z

cmd/restic/cmd_rewrite.go

+		return true, nil
+	}
+
+	err = repo.Flush(ctx)


You only need to flush the repo when the snapshot was modified.

I also wouldn't flush after finishing a single snapshot, but once after all snapshots are rewritten, i.e. in line 289.
The reason is that flush may finish small packs. If you modify a lot of snapshots generating only few new tree blobs, you may end up in having lots of new packs which each only contain few tree blobs.

However, the snapshot files should only be written after flush finishes with nil error. So also writing the snapshots should be postponed. I suggest returning the new snapshot object here and write all snapshots after Flush in runRewrite

cmd/restic/cmd_rewrite.go

dionorgua · 2020-08-21T07:42:41Z

Thanks a lot for review. I'll try to address everything soon

edmwagner · 2021-01-28T13:18:37Z

I Just mirrored my minio and used this branch to rewrite a few different scenarios. looks good so far. restoring files worked as expected and matched versions restored from original repo. :) The repo is several GB smaller though.

I would like to help move this feature forward.

dionorgua · 2021-02-10T17:10:13Z

Hi,
I was busy with personal things. Will try to catch up this weekend and at least update/rebase it.

ghost · 2021-04-07T01:09:53Z

Is supporting such a feature for a whole repository planned?

aawsome · 2021-04-07T04:45:05Z

Is supporting such a feature for a whole repository planned?

This PR already allows to give multiple snapshots or even multiple snapshot criteria. E.g. you can run it with all snapshots or give multiply -H options to cover all hosts.

We could add a possibility to specify all which resolves to all snapshots - but I would leave that to a future PR as it is pretty independent from this change.

ghost · 2021-04-07T13:52:52Z

Good stuff, thank you. 👍

bertbesser · 2021-08-16T16:06:19Z

What's the status of this PR? Any chance that it gets merge soon? I'd very much like to use the feature :-D

Good work 👍🏼

rawtaz · 2021-08-16T16:19:25Z

@bertbesser This PR should be considered work in progress. No decision has been made about whether this should be merged or not, the PR is not yet marked as done by the author, and there's a bunch of unaddressed comments. Clearly it's not ready for merging, and the status is simply what you see here in the GitHub GUI where you posted a comment. That said, there's a lot of other stuff going on if you look in this repository, so consider this WIP. Feel free to try it yourself though, just don't consider it production ready (at least not officially).

dionorgua · 2021-09-30T09:46:29Z

Hi,

@bertbesser

It was working pretty well on my machine in time it was created (but yes, it's very experimental). Good news is that it doesn't touch any data pack files at all. Basically It just adds new snapshot as copy of existing with a few files removed (plus optionally removes old snapshot). I don't think that it can corrupt repository (except new snapshots, that can be just deleted manually)

I think that 'rewrite + check + prune' scenario should be safe enough. I've already used it on my own 2.5TB repository without any problems. Removed most of garbage from old snapshots.

But I'm not sure that it'll work as is right now, I'll rebase it on top of latest git and try to address most of comments next week.

Gaibhne · 2021-10-21T19:25:59Z

This looks like a fantastic feature I sorely miss in Restic. Is there anything I, as a user, can do to help with this PR ?

edmwagner · 2021-10-21T19:38:04Z

Absolutely! I’ve also used it to clean up 30gb of junk from early backups. I didn’t know how to filter back then. My repo was 40 gb and now rest consistently under 10 with about 2years of backups. Really helped!!! But that was also some time ago.

…

On Thu, Oct 21, 2021 at 3:26 PM Gaibhne ***@***.***> wrote: This looks like a fantastic feature I sorely miss in Restic. Is there anything I, as a user, can do to help with this PR ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2731 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPBAZEXMQKIZCY5DGQ4YRTUIBSNNANCNFSM4M6YEN6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

rawtaz · 2021-10-21T22:10:40Z

It's good news that it seems to work fine for those who tried it, and the author's description of how it works suggests it might be not very intrusive and have limited ability to cause any harm if not working as intended. But it's still WIP, needs rebasing, need addressing comments and other polishing, not to mention tests and documentation. Let's conclude that this is the case and unless someone does those parts it's not even close to further review, sorry.

ghost · 2021-12-03T20:41:40Z

Hi @dionorgua, is there any chance you'd have an update regarding your work on this PR? I may or may have not oopsied a B2 backup again. 🤠

MichaelEischer · 2022-09-10T09:07:40Z

I've rebased the PR and made quite a few cleanups (see the commit messages for more details). The main tasks left are:

Writing some documentation
More tests. The current test coverage is just the absolute minimum
Decide what to do with the --inplace flag, I'll probably rename it to --forget
Make filter option handling and descriptions work as in Cleanup snapshot filter options #3912
~~[ ] Performance optimizations.~~ -> Can be done later on.
Finalize the CLI options

NovacomExperts · 2022-09-14T01:54:07Z

I'm so thrilled that this PR is getting some progress ! Many thanks to the team members for all the latest commits. I will do some test on my mostly "local" repos.

I support @MichaelEischer on the naming for --forget instead of --inplace. Of course, "are you sure" could follow before execution, just like prune. As discussed with @dionorgua before, a clear --tag-old-snapshots-with <tag> and --tag-new-snapshots-with <tag> would work great for managing 2 step forget process.

Keep on the great work !

bertbesser · 2022-09-14T16:54:51Z

It was working pretty well on my machine in time it was created (but yes, it's very experimental). Good news is that it doesn't touch any data pack files at all. Basically It just adds new snapshot as copy of existing with a few files removed (plus optionally removes old snapshot). I don't think that it can corrupt repository (except new snapshots, that can be just deleted manually)

I think that 'rewrite + check + prune' scenario should be safe enough. I've already used it on my own 2.5TB repository without any problems. Removed most of garbage from old snapshots.

@NovacomExperts The --tag-xxx-snapshots-with would just be a welcome addition to this, correct? Not a fundamentally other approch if I get you right. Cheers

MichaelEischer · 2022-10-14T22:34:05Z

I've added enough tests to achieve over 80% test coverage for the new code. There's now also a new documentation section. I'll probably split some of the preparatory refactoring commits into separate PRs to reduce the amount of code a bit.

As discussed with @dionorgua before, a clear --tag-old-snapshots-with <tag> and

--tag-old-snapshots is impossible to implement. We cannot change the tag of an existing snapshot without creating a new one and removing the old snapshot.

Instead of using tags, using the printed snapshot id should also work.

MichaelEischer · 2022-10-15T08:49:11Z

I've added a commit to let the rewrite command reject rewriting anything which would result in loosing data in a tree. This ensures that we can add new fields to the Tree struct without having to increase the repository version every time. Without the check, the rewrite command could just loose data.

The check is not completely optimal, but it should prevent rewriting for all problematic situations. If the need arises to handle this in a more targeted way, we can still adapt the implementation later on.

In principle, the JSON format of Tree objects is extensible without requiring a format change. In order to not loose information just play it safe and reject rewriting trees for which we could loose data.

The old check did not consider files containing case insensitive excludes. The check is now implemented as a function of the excludePatternOptions struct to improve cohesion.

MichaelEischer · 2022-11-12T19:02:15Z

LGTM. I've made one final change before merging this PR: when just passing--iexclude-file the rewrite command would complain about missing excludes.

Everything went very well and all checks are ok (including read-data).

Thanks for testing! Good to hear that the PR indeed worked as expected (it's been in a good shape for some time.

MichaelEischer

LGTM2

12nick12 · 2023-01-07T20:17:31Z

Any idea when this will be in the prebuilt binaries?

rawtaz · 2023-01-07T20:18:56Z

It will be in the next release, which is released when it's done.

aawsome · 2023-01-08T08:16:39Z

@12nick12 You might want to have a look at https://beta.restic.net/

rawtaz · 2023-01-08T14:42:43Z

FWIW, we're very close to a new release so stay tuned and you'll have it in your hands soon :) But indeed, nothing wrong with using the builds @aawsome pointed to (thanks for pointing that out)! I use them all the time.

rawtaz · 2023-01-12T21:07:32Z

@12nick12 It's out now! See: https://restic.net/blog/2023-01-12/restic-0.15.0-released/

MichaelEischer mentioned this pull request Jun 16, 2020

Fatal: number of used blobs is larger than number of available blobs! #2777

Closed

aawsome reviewed Jul 10, 2020

View reviewed changes

MichaelEischer mentioned this pull request Aug 8, 2020

Add repair command #2876

Merged

8 tasks

ghost mentioned this pull request Aug 21, 2020

Delete Files from Existing Snapshot #14

Closed

aawsome mentioned this pull request Aug 23, 2020

Add copy functionality. #2606

Merged

7 tasks

aawsome mentioned this pull request Nov 21, 2020

support for merged backup parents #3118

Open

gerardbosch mentioned this pull request Oct 28, 2021

Implement 'rewrite' command to exclude files from existing snapshots #2720

Closed

8 tasks

depau mentioned this pull request Jan 6, 2022

Implement 'rewrite' command to exclude files from existing snapshots #3621

Closed

8 tasks

MichaelEischer force-pushed the rewrite-snapshot branch from 202b8c0 to 3d7d495 Compare September 10, 2022 09:03

MichaelEischer force-pushed the rewrite-snapshot branch from 3d7d495 to 329fa17 Compare September 27, 2022 19:21

MichaelEischer force-pushed the rewrite-snapshot branch 2 times, most recently from 7f4d706 to 915c51d Compare October 14, 2022 21:45

MichaelEischer and others added 21 commits November 12, 2022 19:55

rewrite: simplify dryrun

c0f7ba2

rewrite: extract tree filtering

f6339b8

rewrite: filter all snapshots if none are specified

2b69a1c

unify exclude patterns with backup command

4cace1f

unify exclude pattern options

559acea

rewrite: start repository uploader goroutines

7ebaf6e

rewrite: use SelectByName like in the backup command

ad14d6e

rewrite: cleanup err handling and output

327f418

rewrite: non-exclusive lock if snapshots are only added

375a3db

rewrite: add minimal test

b044649

rewrite: use unified snapshot filter options

a47d9a1

rewrite: rename --inplace to --forget

73f54cc

walker: Add tests for FilterTree

0224e27

rewrite: Add tests for further ways to use the command

ec0c91e

rewrite: add documentation

11b8c3a

rewrite: Fail if a tree contains an unknown field

f88acd4

In principle, the JSON format of Tree objects is extensible without requiring a format change. In order to not loose information just play it safe and reject rewriting trees for which we could loose data.

rewrite: Revert unrelated documentation change

c15bedc

rewrite: Polish code and add missing messages

f86ef4d

rewrite: Polish documentation

f175da2

rewrite: Fix check that an exclude pattern was passed

537cfe2

The old check did not consider files containing case insensitive excludes. The check is now implemented as a function of the excludePatternOptions struct to improve cohesion.

Cleanup exclude pattern collection

bb0fa76

MichaelEischer force-pushed the rewrite-snapshot branch from 72196e5 to bb0fa76 Compare November 12, 2022 18:55

MichaelEischer approved these changes Nov 12, 2022

View reviewed changes

MichaelEischer merged commit 726a196 into restic:master Nov 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement 'rewrite' command to exclude files from existing snapshots #2731

Implement 'rewrite' command to exclude files from existing snapshots #2731

dionorgua commented May 12, 2020 •

edited by MichaelEischer

aawsome left a comment

aawsome Jul 10, 2020

dionorgua commented Aug 21, 2020

edmwagner commented Jan 28, 2021

dionorgua commented Feb 10, 2021

ghost commented Apr 7, 2021

aawsome commented Apr 7, 2021

ghost commented Apr 7, 2021

bertbesser commented Aug 16, 2021 •

edited

rawtaz commented Aug 16, 2021

dionorgua commented Sep 30, 2021

Gaibhne commented Oct 21, 2021

edmwagner commented Oct 21, 2021 via email

rawtaz commented Oct 21, 2021

ghost commented Dec 3, 2021

MichaelEischer commented Sep 10, 2022 •

edited

NovacomExperts commented Sep 14, 2022

bertbesser commented Sep 14, 2022 •

edited

MichaelEischer commented Oct 14, 2022

MichaelEischer commented Oct 15, 2022

MichaelEischer commented Nov 12, 2022

MichaelEischer left a comment

12nick12 commented Jan 7, 2023

rawtaz commented Jan 7, 2023

aawsome commented Jan 8, 2023

rawtaz commented Jan 8, 2023 •

edited

rawtaz commented Jan 12, 2023

Implement 'rewrite' command to exclude files from existing snapshots #2731

Implement 'rewrite' command to exclude files from existing snapshots #2731

Conversation

dionorgua commented May 12, 2020 • edited by MichaelEischer

What is the purpose of this change? What does it change?

Was the change discussed in an issue or in the forum before?

Checklist

aawsome left a comment

Choose a reason for hiding this comment

aawsome Jul 10, 2020

Choose a reason for hiding this comment

dionorgua commented Aug 21, 2020

edmwagner commented Jan 28, 2021

dionorgua commented Feb 10, 2021

ghost commented Apr 7, 2021

aawsome commented Apr 7, 2021

ghost commented Apr 7, 2021

bertbesser commented Aug 16, 2021 • edited

rawtaz commented Aug 16, 2021

dionorgua commented Sep 30, 2021

Gaibhne commented Oct 21, 2021

edmwagner commented Oct 21, 2021 via email

rawtaz commented Oct 21, 2021

ghost commented Dec 3, 2021

MichaelEischer commented Sep 10, 2022 • edited

NovacomExperts commented Sep 14, 2022

bertbesser commented Sep 14, 2022 • edited

MichaelEischer commented Oct 14, 2022

MichaelEischer commented Oct 15, 2022

MichaelEischer commented Nov 12, 2022

MichaelEischer left a comment

Choose a reason for hiding this comment

12nick12 commented Jan 7, 2023

rawtaz commented Jan 7, 2023

aawsome commented Jan 8, 2023

rawtaz commented Jan 8, 2023 • edited

rawtaz commented Jan 12, 2023

dionorgua commented May 12, 2020 •

edited by MichaelEischer

bertbesser commented Aug 16, 2021 •

edited

MichaelEischer commented Sep 10, 2022 •

edited

bertbesser commented Sep 14, 2022 •

edited

rawtaz commented Jan 8, 2023 •

edited