New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for syncing ownership of files #1329
Comments
Wouldn't that require running Syncthing as root? |
Yep that would require root. |
I am pretty sure this is unlikely to happen as uids:gids can make no sense on windows or on a device which doesn't support users (android) or even have them. |
You can solve the same problem by running multiple instances though. |
I am syncing the /Users directory between two mac, so I run syncthing as I run into the annoyance of having uid out of sync a few times. As a reference, I use crashplan for backups. They preserve the uid:gid So adding this values to the syncthing db and trying to apply if the OS is Unfortunately I am not a programmer, so I can't contribute a patch, but I
|
@AudriusButkevicius I don't think windows or android is a good reason not to support this. This is the same idea as mtime and permissions. It would not matter if this is the default or not but a configurable option to support this would go along way for use cases when using this as a deployment and configuration system for servers and not just desktop use. |
if you sync between 2 linux servers or more with SAMBA on top can UID and GID be synched then? |
@crosbymichael feel free to make a pull request. Saying Windows and Android is not a good reason not to support is not true (at least what I believe in). By the end of the day, we (or perhaps that is just my personal view) are trying to make a tool which works on multiple platforms, and we try to make sure that all features work on all platforms (more or less). I guess you could do uid:gid mapping based on the user account names across Unix and Windows, but then perhaps Unix people would expect it to work based on integers... Atleast that is the sort of behaviour I would expect syncing stuff from Unix to Windows... Not sure... gids don't even make sense on Windows. |
@AudriusButkevicius lets try to keep the discussion professional and not suggest that someone's contribution for a feature that they care about would be "Turning the code into an unmanageable mess just for the sake of it working on Unix" |
I am not saying that. What I am saying, is that the only solution which comes to my mind now, would make an unmanageable mess. I guess I am a bit of an advocate of Windows here, and I don't want Windows to be left out, hence why it would be a mess. As I said, given it's done sensibly, I am sure it could become a feature, even if an optional one. |
Ok thanks. I opened this issue first instead of a PR so that this could be discussed with the maintainers. I know what it's like to have feature requests implemented in a PR without any discussion. I'm more than happy to look for an elegant solution for this. If it turns out to add more complexity than what it would be worth, then I have no problem closing this issue. |
The There is some history of this available here (which might have some sensible ideas): Though I feel you cannot get a way with some sort of optional remapping config |
Yeah, the relevant discussion is linked above. Solving this as described above (just copy uid+gid) isn't rocket science. Unfortunately it's also a half baked solution, as it doesn't work on non-Unixy systems, and will anyway not solve the other aspects that will pop up if this is implemented (sync by username and not uid, map some uids to others, etc)... As such, it would need to be really neat and unobtrusive to be worth it, in my opinion. |
Solving this issue via copy uid+gid is good step and already will be usable via most wanted users on system with mutual user list (for example, ldap database or active directory). But in other cases uid for same user on different machines may be different, so when we sync numeric uid - on one computer user with uid 1002 may be Bob, on second - Peter. |
I appreciate this is a complex area. It's challenging even to work out what the best approach is, before you try and implement it. However- It's been done before, by Samba. In fact Samba goes way beyond the simple case to deal with ID mapping across LDAP domains etc., which is hopefully beyond what most syncthing users would need. It would seem sensible to look at some reduced subset of what Samba can do as an initial design goal, The benefits would be significant-
|
I think that in order for anything to happen in this area someone needs to step up with (first) a design proposal of what the relevant subset is, detailing which situations it handles and which situations it does not, and (second) an actual implementation of that proposal. The person taking on that challenge should certainly look at prior art in the area and count on this being a several months long project to drive home. I don't think it's helpful, in itself, to note that there are solutions that solve a different but similar problem after probably hundreds of person hours of development. That's clearly the case, but it doesn't move this issue forward. It's also self evident that there would be benefits to syncing ownership and advanced permissions. |
If this gets implemented, it should ideally provide some means of mapping ID's and names. People don't use the same user name everywhere (especially if they've got different operating systems involved), and there are cases where neither syncing based on numeric ID or based on name will work correctly (in fact, by numeric ID won't work most of the time when multiple users are involved, and is pretty much guaranteed not to work if syncing between multiple operating systems). |
It's not syncthing's task to do uid/gid name mapping, there is efficient robust dedicated tools to do that out there (like Providing uid/gid support is only making syncthing able to take care of two metadata numbers, no less no more. If someone have [ug]id↔name mapping problem it's a problem in his setup, it's not a syncthing problem. And I don't expect syncthing to be able to workaround user's misconfiguration problems (and please don't implement such things !). Tools like rsync or unison just take care of uid/gid number, it's up to the sysadmin to install an id mapping service, and syncthing does not have to do more (and please do not do more than just replicating the uid/gid numbers). I see this issue is open since almost three years now, it looks like the only real blocker is the fact the thread derivated from the very simple "syncthing needs uid/gid replication" topic (which is not much complex than permission or mtime replication) to the very complex "syncthing must be an id mapping service" topic. Let's not expect syncthing to do things it's not up to syncthing to do or uid/gid replication will never be there… |
In this issue - users want to sync files with different owners, so syncthing need to have solution for this needs - keep and sync file owners (no matter what way: uid/gid, or username, group, acl, other ways), not only file permissions. |
I think it's a sensible feature to have, but as I said earlier I think it's a can of worms in terms of implementation, because some people will want to sync based on account/group names, some will want based on uid/guid, I think windows only has names. So I think the first step of this would be to write an implementation proposal of how all of these pain points would be addressed. Also, just to note, this has moved on a bit since some of the last posts. There is now an advanced feature flag which copies the ownership from the directory the file is being created in, which might be sufficient. This obviously requires special capabilities/root. |
All right, I'll start preparing a proposal 👍 My thoughts about the pain points are as follows:
I hope that this should cover the pain points - I there something I missed? |
I think it's best if you start a forum thread. Because you'll hear tons of opinions here with objections, demands and what not here, so I'd rather keep that to the minimum. |
@aawsome there is two feature requests that are messing up in that whole thread:
This second feature request, which is legit but can be done as a second step, requires a far much amount of work. Sadly, this second feature request added so much noise to this thread it entirely took over this thread. So, five years later 1. is not implemented because of 2, but 2. requires 1. to be implemented first. That's part of why the thread is stalled. While I fully understand why feature request 2 would be very cool and I may use it myself (I know how hard it is to get a complete setup like the one using Winbind), it's possible to wait for 2. to be implemented after 1. is implemented, by relying on existing third-party id mapping tools. In any way, 1. is required for 2. I attempted to separate the issues again and created #6105 to track feature request 1. while this one would be for feature request 2, in an attempt to unblock the situation. I'm not sure the feature request 2. is going somewhere before 1. is fixed. It would probably better to reopen #6105 before spending time on this one if you worry about this one to be still open in the next five years. Edit: I forgot there is a third feature request messing with this thread:
|
I don't think just option 1 would ever be accepted, because it works on one platform only, which is a no-go for a cross platform tool. |
@AudriusButkevicius That's not correct. All my Linux boxes have Samba and Winbind installed...so even if I were syncing between a Windows box and a Linux box, all domain accounts are available in both places. But at least in my case I'm syncing between ~24 Linux boxes which all have the same account information available. I see two systems that can be implemented. The first is implementing ownership/group info. IIRC Windows doesn't have the concept of file ownership--just permissions. This would cover Linux-like systems.
The second would deal with permissions. Similar to the first system, permission options could be something along the lines of "don't sync", "sync by UID/GID/SID", "sync by name", and "override user/group" In my case I would choose to sync ownership and permissions either by UID/GID/SID or by lookup. A bit of pseudo code:
Permissions could follow a similar format. If Linux has extended ACL support enabled, have an option to sync the extended ACLs instead of the standard ACLs. Just a thought. |
This is "not correct" only in your local context. I don't think every Linux machine ever has winbind installed and pre-configured waiting for syncthing to do it's thing. Also, winbind means nothing on Windows. I don't think your comment disagrees with what needs doing, the actual implementation details may vary hence I am not going into details. I am mostly arguing that whatever implementation lands, it should land with ability to remap/lookup/whatever you want to call it stuff straight away, not to deprive/hinder clusters with mixed operating systems. |
Sure. What's the use case for "I want permissions synced" along with "...but the user accounts aren't the same on both (or all) machines"? If I want to sync with some internet rando with permissions (and/or ownership) to another internet rando...but they don't have accounts that are in sync? I just can't see that use-case. If I were syncing family photos to my mom's computer, I don't care about permissions or ownership on the files. If I'm distributing media files or a podcast, I don't care about permissions. But if I'm in a corporate environment where permissions matter, I already have the infrastructure set up to ensure usernames and passwords exist on the boxes....or why would I need syncing in the first place? In my case, my Linux boxes are DFS targets, as well as folder redirection targets for user profiles...so they already communicate with active directory. Maybe on a home network where one machine has 'owner' and the other machine has 'aaron'? A simple mapping would handle that...either a list of users to map to other users, or simply "map all permissions to "aaron"....but I think that's a rare case as opposed to just letting syncthing run as "aaron" and sync while ignoring permissions. Maybe I'm missing something? |
You are looking at it through a corporate prism, which is probably a minority of our users, hence I don't think we explicitly try to cater for them, but try to cater for the little guy, and I think the little guy will need this. You yourself pointed out a case in a home network. There are various appliances (synology comes to mind), that run an app under user X, but requires the files to be owned by user Y for some other app on the appliacne to be able to read/use those files. Stuff inside of docker containers falls under this as well I think. Nobody is saying that we need complicated mappings, it would simple mappings as you yourself pointed out. I am just saying that "just syncs uids, one to one with no remappings" as someone proposed above, is not good enough. |
I pointed out a home network case and showed that users probably wouldn't make use of the feature. What home would:
Probably a lot less than the 'minority' of corporate users you are talking about. Most home users just want files synced and working (like syncthing already does now). Permissions are almost exclusively a corporate or power-user thing.
Sure--so two ways to do that in this hypothetical situation:
But like I said, I doubt that's the majority. How many home users want to do weird permission mapping stuff? I'd bet a majority of corporate users want to handle permissions. In fact, that's why we still use Windows DFSR and other hacky solutions in our corporate environment instead of syncthing. Syncthing only manages stuff where I don't care about permissions--like globally accessible folders where everyone has full control or globally accessible folders where everyone gets read-only access.
I agree. Maybe boil it down to four 'permission options' for a folder:
|
The reason this is issue exists is not because there fundamental disagreement of what needs to be done here The reason this is open is because nobody needs this badly enough to work on it. There were a number of suggestions to reduce the scope "just to get something out the door", which as you say, would only be useful to people running ADs, which I am personally against, because it leaves other platforms (thinking about the specific proposal) and the little guys deprived. |
Agreed, however starting simple to "just get something out the door" to support the (probably) biggest use-case (permissions sync in a corp environment) wouldn't "leave other platforms deprived". You already don't have it. You're already deprived. ;)
I'd totally dive in and work on it....if Syncthing were written in Python or nodejs...Go has been on my bucket list for a while, but I just haven't had the time... |
If corporations need it, they can go and sponsor someone to work on this., yet I'd expect the full implementation. |
Who can I sponsor and how much do they need? |
kastelo.net I guess is the official "commercial" body behind syncthing, but I guess you can hire someone who knows a bit of Go and html to do this. |
Maybe I can tell about my story: I started implementing it (just minor change when syncing just uid and gid 1:1 and also with option to sync/not sync ownership). This however needed a DB schema update and also a rescan of all folders on all devices to update the DB. So I tested it and used it on my devices (all linux, all identical uid/gid) - works like a charm! Then I thought about the next steps to create a PR:
So, finally I left with just using my patched version which is still working like a charm.. 😉 Maybe if I find more time, I'll tackle the above mentioned points and prepare a PR. Or, if wished, I can simply share my modifications, maybe it is enough to opt in/opt out and someone can tackle the DB migration topic such that we'll only get an optional ownership syncing that only works for linux but doesn't harm anything on Windows. |
@aawsome I'd love to get a copy of the patch and buy you a beer or something from an Amazon wishlist... I only need 1:1 uid/gid mapping on Linux since all my NAS boxen have identical IDs through winbind. Or if you want to try getting a PR submitted or something, I might be able to do some sort of sponsorship through my company or one of my clients that needs it. |
@aawsome Could I get a copy of your changes please? I have 2 linux boxes I'd like to sync. I would value what you have even if the corner cases you outlined have not been addresses. Cheers! |
You can find the patch here : https://github.com/aawsome/syncthing/tree/sync-owner |
Sorry for not answering. This is the patch I'm still using, however it is based on syncthing 1.5.0 and there have been quite some changes in the master code since then. |
* main: all: Support syncing ownership (fixes syncthing#1329) (syncthing#8434) gui, man, authors: Update docs, translations, and contributors
* main: cmd/syncthing, lib/config: Remove restartOnWakeup option & functionality (fixes syncthing#8448) (syncthing#8449) gui: Remove blank meta tags (syncthing#8362) gui: Add device sync status (fixes syncthing#7981) (syncthing#8401) gui: Fix detailed staggered versioning information in folder info (ref syncthing#8348) (syncthing#8433) all: Support syncing ownership (fixes syncthing#1329) (syncthing#8434) gui, man, authors: Update docs, translations, and contributors lib/model, lib/config: Apply sensible defaults for auto-accepted encrypted folder (fixes syncthing#8296) (syncthing#8427) gui: Move filesystem watcher explanation from tooltip to help block (syncthing#8432) gui: Use discovered IDs from cache when adding a new remote device (syncthing#8382) build: Update goleveldb (syncthing#8440) gui, man, authors: Update docs, translations, and contributors cmd/syncthing/cli: Add show discovery command (fixes syncthing#8007) (syncthing#8378) gui, man, authors: Update docs, translations, and contributors lib/osutil: Only announce address of interfaces which are up (fixes syncthing#7458) (syncthing#8422) gui: Fix missing span end tag and missing nbsp semicolon in HTML (syncthing#8419)
This adds support for syncing ownership on Unixes and on Windows. The scanner always picks up ownership information, but it is not applied unless the new folder option "Sync Ownership" is set. Ownership data is stored in a new FileInfo field called "platform data". This is intended to hold further platform-specific data in the future (specifically, extended attributes), which is why the whole design is a bit overkill for just ownership.
v1.21.0 Bugfixes: - syncthing#8219: REST API: db/completion no output when one folder is paused - syncthing#8479: Panic in failure reporting Enhancements: - syncthing#1329: Add support for syncing ownership of files - syncthing#7981: Show likely status of disconnected devices - syncthing#8296: Auto-accepted receive-encrypted folders should have more sensible defaults - syncthing#8323: Show internally used paths in the GUI for debugging - syncthing#8448: Remove "restart on wakeup" functionality
It would be nice to add support for syncing file ownership with uid and gid. This already supports permissions.
2019 note: this issue was originally titled "...syncing uid:gid of files" and there's initial discussion of that, but that's too constricted hence I've retitled it. //@calmh
The text was updated successfully, but these errors were encountered: