Allow mixing include and exclude patterns for backup and restore

[fd0]

This is a followup issue for #226. Up to now it is only possible to specify exclude patterns for backup and restore.

There are uses cases that aren't covered, for example: A user would like to backup her home directory at /home/user, excluding anything from the directory work except C source files.

Are there any other uses I haven't thought of?

Implementation/User Interface: Allow specifying both --include and --exclude patterns on the command line, which fill a common list (order does matter here). For each dir/file, check all patterns in the list. The action (exclude or include) of the last mattching pattern wins. The default action (which will also be used if neither include nor exclude patterns are specified) is "include".

This already contains a corner case: Should restic walk /home/user/work at all?

I think it should not. Instead, if this is a desirable behaviour, require the user to add a more specific pattern to signal that the excluded directory should also be walked, e.g. restic backup --exclude /home/user/work --include /home/user/work/**/*c. /home/user.

A better user interface would be to allow specifying a file to read the patterns from (--pattern-file or something like that). In this file, all lines starting with # are comments, empty lines ignored, all other lines must start with either + (include) or - (exclude) following a space character and a pattern. For the use case above a filter file would look like this:

# filter out everything from work, but include c source code files
- /home/user/work
+ /home/user/work/**/*.c

This issue can be closed once a definitive solution for having both include and exclude patterns is implemented.

[fw42]

Another possibility would be to read an exact list of files to backup from a file (or from stdin). Then people could just use whatever tools they want (find, grep, etc.) to build their list and "pipe" that list to restic.

[fw42]

I should read the whole text before posting a reply. Sorry.

[fd0]

This is a different issue, taking the list of files/dirs to backup from stdin instead of command line arguments. Do you think it's valuable to have that? If so, could you add an issue?

[fd0]

Just a reminder for myself: This issue is about include filters for backup, restore already has them.

[lathspell]

You should maybe just copy rsync's --exclude/--include/--exclude-from syntax, they have 20 years of experience :-)
(at least please add some examples of the current syntax to the user guide as it's unclear if "/foo" and "foo" is thesame or if "*.c" is supported.

[fd0]

Ah, thanks for the comment, I've added an issue for the missing examples in the manual: #396

To be honest, I don't like the filter syntax from rsync at all, because the rules are way too complex. But we'll see what we can come up with.

[ghost]

Any updates? It is definitely a must-have option. Currently it is not even possible to read the file list to back up from the stdin:

$ find -name '*.go' | restic backup --files-from -
open -: no such file or directory

whereas it could be written as

restic backup --exclude '*' --include '*.go'

[fd0]

Uhm, reading the list of files from stdin can be achieved by calling restic as follows:

$ find -name '*.go' | restic backup --files-from /dev/stdin

If you like, I'd accept a PR that adds handling - for --files-from. :)

[fd0]

@opennota would you mind describing your use case? It'd be interesting for us.

[fd0]

The dash (-) issue is tracked as #769.

[ghost]

@fd0
Not much of a use case. I just want to back up only the files with certain extensions without using a temporary file for the list.

If you like, I'd accept a PR that adds handling - for --files-from. :)

I'll see what I can do.

[runfalk]

You can kind of emulate this behavior using sed and named pipes:

restic --exclude-file <(sed -n 's/^- \(.*\)/\1/p' files.list) --files-from <(sed -n 's/^+ \(.*\)/\1/p' files.list)

Lines starting with - are excluded and those with + are included.

[dgasaway]

I think a good model is what borg recently implemented for --pattern and --patterns-from
https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
Not so much the different style selectors, but the options to specify root paths, include rules, exclude rules, and no-recurse exclude rules in a file.

[blacksails]

Why not just use the standard ignore format that .gitignore uses? eg.

# ignore everything
*
# include $HOME/.local
!$HOME/.local

[tnatanael]

The --include is scheduled to be implemented?

Sound like --include and --exclude cannot be implemented together, or at least would be a hierarchy of precedence with one on other...

[vincentbernat]

Providing a list of files with --files-from doesn't totally solve the problem as the snapshots subcommand will display gigantic list of files and forget subcommand doesn't work as expected.

My use case is backing up my home and I have a list of paths with some exclusion and some exceptions for the exclusion. The base list of paths contains already a hundred of items. As everything is under $HOME, I would expect to be able to say something like --exclude=** --include=~/path1 --include=~/path2 --exclude=~/path2/something --exclude=*~. So, to determine if a path should be included it should be matched against each --exclude and --include in the right order and the last matching wins.

[danielsharvey]

I think @vincentbernat's PR is an effective solution to this requirement.

Summary: Allow gitignore-style negative patterns to specify exclude rules for both backup and restore.

I've been using it effectively for several weeks now.

Notably, this enables me to simplify my snapshots listing, which was quite verbose previously:

b951f6a2  2019-06-15 11:30:18  elvandar    manual      /Users/daniel/Desktop
                                                       /Users/daniel/Documents
                                                       <lots more...>

to:

d0c0bed1  2019-06-18 08:20:57  elvandar                /Users/daniel

Also, I've implemented an effective (not perfect but working) solution providing continuous backup using this feature (will share details shortly).