Add support for CreationTime and File Attributes in Windows

[mchowdhury]

What does this PR change? What problem does it solve?

File created time (an NTFS specific attribute) and File Attributes don't get backed up or restored on Windows backups. See issue #3478 and #2075

Since we don't want to modify ctimes (because it's used for file change comparison #2212) we store creation time as an extended attribute and restore the creation time value when the file is restored.

Was the change previously discussed in an issue or on the forum?

This issue was filed in #3478 and #2075
A related issue is #1401 for backing up entire NTFS extended attributes. This uses microsoft provided go-winio library. In the future this would be a better approach but would be a significant re-write. See this comment: #1401 (comment)

Checklist

• I have read the contribution guidelines.
• I have enabled maintainer edits.
• I have added tests for all code changes.
• I have added documentation for relevant changes (in the manual).
• There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
• I have run gofmt on the code in all commits.
• All commit messages are formatted in the same style as the other commits in the repo.
• I'm done! This pull request is ready for review.

[DRON-666]

Can you add windows attributes as well? This would solve issue #2075.

[mchowdhury]

Can you add windows attributes as well? This would solve issue #2075.

Ideally it would be nice to have all the missing attributes included. This can be done what's described in #1401 (comment)

It's a much more significant effort but can be done. I'm not that strong with Go development. I would probably need some assistance with the implementation.

[DRON-666]

Ideally it would be nice to have all the missing attributes included.

I fully agree, but adding standard attributes to you PR is just the use of SetFileAttributes instead of SetFileTime.

This can be done what's described in #1401 (comment)

I don't think that BackupRead/Write API is compatible with restic architecture. It does not allow you to choose the type of requested information. Anyway BackupRead/Write is just a userland code with bunch of NT-function calls and you don't need to use it.

[mchowdhury]

I fully agree, but adding standard attributes to you PR is just the use of SetFileAttributes instead of SetFileTime.

This was a good idea. I updated and tested the PR with saving and restoring file attributes in addition to creation time

[mchowdhury]

@DRON-666 Thanks again for the feedback. I made the necessary changes and tested them.
Let me know if you have any other suggestions.

[mchowdhury]

@DRON-666 What else needs to be done to get this merged?

[hgraeber]

It would be nice to add security information the same. Would mean to use GetFileSecurity and SetFileSecurity the get and set the DACL. Converting the descriptor to/from SDDL string representation would be even better. ConvertSecurityDescriptorToStringSecurityDescriptor and ConvertSecurityDescriptorFromStringSecurityDescriptor are helpful for that.

Maybe you consider to use a readable representation for the other attributes, too.

[mchowdhury]

It would be nice to add security information the same. Would mean to use GetFileSecurity and SetFileSecurity the get and set the DACL. Converting the descriptor to/from SDDL string representation would be even better. ConvertSecurityDescriptorToStringSecurityDescriptor and ConvertSecurityDescriptorFromStringSecurityDescriptor are helpful for that.

Maybe you consider to use a readable representation for the other attributes, too.

@hgraeber GetFileSecurity and SetFileSecurity are not currently supported by the standard go syscall library. I'm open to other suggestions. Otherwise, should we make a separate issue for file security attributes and get this merged?

[MichaelEischer]

Mapping the creationTime / fileAttributes in to xattrs feels wrong, please just add new (optional) fields to the Node struct. I'd also prefer to store the creationTime as a regular timestamp and not some Windows specific timestamp format. Btw, the creationTime is known as birthTime on Linux, so it might be more appropriate to use that name (or short btime for the json attribute). Even though we can't write that timestamp on Linux.

The FileAttributes should also be accessible from the struct returned at

restic/internal/restic/node_windows.go

Line 56 in cccc17e

s, ok := i.(*syscall.Win32FileAttributeData)

. Adding two redundant syscalls (the two os.Stat) calls will likely hurt performance.

Judging from https://learn.microsoft.com/en-us/windows/win32/fileio/file-attribute-constants I'm not sure whether it's a good idea to blindly store and restore arbitrary file attributes. But I'm not familiar enough with the Windows API to accurately judge that.

Restoring access permissions etc. definitely doesn't belong into this PR.

[mchowdhury]

@MichaelEischer For Windows file attributes, these are essentially the same as Linux extended attributes and should be backed up and restored. I figured adding it here would be the cleanest way. creationTime also has the same challenge.

How do you propose we integrate and restore them if they are added as optional attributes to the node struct? This makes the node structure more messy for other platforms.

[hgraeber]

Judging from https://learn.microsoft.com/en-us/windows/win32/fileio/file-attribute-constants I'm not sure whether it's a good idea to blindly store and restore arbitrary file attributes.

I think the file attributes shall not be copied blindly, because they may have some side effect and cannot be simply restored. (https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-setfileattributesa). It is necessary to set flags on file creation or using special functions to reach the desired effect.

[hgraeber]

@mchowdhury

GetFileSecurity and SetFileSecurity are not currently supported by the standard go syscall library. I'm open to other suggestions. Otherwise, should we make a separate issue for file security attributes and get this merged?

Agreed. I will write an issue for this. There we can discuss how to implement backup of windows file security information.

[mchowdhury]

@hgraeber I don't quite understand the approach you are describing, and unsure how to implement it without impacting other platforms (node.go). Should creationTime follow the same approach? If not, we can separate file attributes to a separate PR and just focus on creationTime.

Would be happy to have your help on the implementation if you have bandwidth.

[ProactiveServices]

Have you considered the more unusual attributes and what happens if they cannot be restored, or what might unexpected behaviour might occur if they are restored? There's a host of weird and wonderful attributes other than the usual hidden/read-only/system. How is the restoration of a sparse file handled? What about attributes that only kernel-level code can restore? Will a restore apply to a file marked as temporary? How are you handling EFS-encrypted data? Files that have attributes that are not supported on the target filesystem?

[mchowdhury]

@MichaelEischer After more research, I like your suggestion with btime adding it to the node, also since darwin also supports btime. I will make a new PR to support btime on windows and os x.

Let's leave this PR open to discuss File Attributes on Windows. File attributes got pulled into this PR from @DRON-666 suggestion to fix issue #2075 using xattrs

[heapxor]

hello,
is that merge request stable and tested? thanks

[hanacchi]

any progress on getting this merged? i would love to use restic but lack of creation time support on windows is holding me back

[tszynalski]

As great as restic is, the fact that it does not preserve file creation dates on Windows is, for me, its biggest drawback compared to consumer backup solutions like Backblaze, Arq ad CrashPlan.

I sometimes use Windows file creation dates to answer questions like "How long ago did I write this HOWTO and can I expect it to still be relevant?", "Did I really write down this thought 10 years ago?", "Which of the two checklist.txt files is more recent?". File modification dates do not fulfill this purpose because they get updated even if you correct a single typo. Losing file creation dates in the event of a failure would be losing part of my personal "historical record".

It would be fantastic if file creation dates were backed up and restored correctly. I don't care about exotic file attributes.

[Dobatymo]

A file copy using the Windows Explorer doesn't preserve file creation dates (only modification dates). So I am not sure restic should restore them either.

[tszynalski]

A file copy using the Windows Explorer doesn't preserve file creation dates (only modification dates). So I am not sure restic should restore them either.

Copying a file is not the same as restoring a file from backup. When you copy a file, you create a second file with the same contents – but it's not the same file, so the creation date should be different. When you back up a file, you want to be able to "turn back time" and go back to the way things were in the event of failure. If something is lost (such as a piece of metadata), then the restore was not complete.

[MichaelEischer]

After taking a look at #4611 , I think we should follow that approach. The GenericAttributes suggested there just look like what I had in mind.

Sorry for not finding time to discuss this PR!