Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

archiver: Shortcut for handling empty files #4267

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

greatroar
Copy link
Contributor

What does this PR change? What problem does it solve?

Regular files that are empty according to stat are now not opened for reading their contents. Such files are quite common (in my homedir, at least) and we can save multiple system calls this way. On a network filesystem, that can mean round trips. Also, we can back up empty files that we cannot open for reading. Finally, fixes #4257.

Existing tests cover this case. fs.Reader now no longer has a meaningful Size. Nothing depended on that.

I tested restic backup --stdin on the command line and it still works. It also still gives the same error for empty stdin.

Was the change previously discussed in an issue or on the forum?

Fixes #4257.

Checklist

  • I have read the contribution guidelines.
  • I have enabled maintainer edits.
  • I have added tests for all code changes.
  • I have added documentation for relevant changes (in the manual).
  • There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
  • I have run gofmt on the code in all commits.
  • All commit messages are formatted in the same style as the other commits in the repo.
  • I'm done! This pull request is ready for review.

Regular files that are empty according to stat are now not opened for
reading their contents. Such files are quite common (in my homedir, at
least) and we can save multiple system calls this way. On a network
filesystem, that can mean round trips. Also, we can back up empty files
that we cannot open for reading. Finally, fixes restic#4257.

Existing tests cover this case. fs.Reader now no longer has a meaningful
Size. Nothing depended on that.
@HeikoSchlittermann
Copy link
Contributor

Empty files may not be empty. What is about e.g. /proc/cpuinfo. Of course, if creating a backup of this directory is necessary, is another question. But anyway, I think, shortcuts like the one you suggest, are asking for trouble, sooner or later.

@pvgoran
Copy link

pvgoran commented Mar 24, 2023

But anyway, I think, shortcuts like the one you suggest, are asking for trouble, sooner or later.

I second this opinion.

@greatroar
Copy link
Contributor Author

I don't find the /proc example compelling at all, and we have at least one example were an empty file not being empty is exactly what is causing trouble.

@HeikoSchlittermann
Copy link
Contributor

Programs should not try to be more clever than me ;)

@HeikoSchlittermann
Copy link
Contributor

HeikoSchlittermann commented Mar 24, 2023

What about a broader approach like

restic backup --skip '{{ stat.Uid }} != 0'

or similiar. But, OTOH this duplicates things we can achive better using find and creating a file list then. Maybe this would help in your scenario too, using find ... -not -empty to create a list of files for the backup. (If I understand the options well, I'm new to restic, since about 3 days.)

@greatroar
Copy link
Contributor Author

I'm not sure I get you completely, but I certainly want the empty files in, not skipped. I just want don't want to actually open them if lstat says they're empty.

@HeikoSchlittermann
Copy link
Contributor

I'm not sure I fully understand you, but I definitely want the empty files to be there and not skipped. I just don't really want to open them when lstat says they are empty.

You are correct, I missed that point. Creating a list of files to back up would not help here. Just as the suggested filter would not help. So maybe you're suggesting a new option as --trust-file-size

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

juicefs - restic hangs
3 participants