Always syncing 95% for encrypted devices and removed files

[dmitryash]

Environment

Version: v1.23.4, Linux (64-bit Intel/AMD)

Configuration: three devices and one of devices is encrypted (i.e. untrusted).

Problem

Syncthing almost always shows state Syncing (95%, 0B) when one of peers is untrusted device (and encryption is enabled). I have never seen such problem between two trusted devices. Usually syncthing says that one item is out of sync and as I can see it is deleted file. Restarting/rescanning folders does not help at all, but sometimes probably after more changes are synchronized between devices, a set of unsynchronized files may be changed (there can be more than 1 item and they may be other files, i.e. state is changed periodically). I suppose that the third device forces resync sometimes.

The interesting thing I noticed latest time was that untrusted device showed that file had been modified by "Unknown" device (Mod. Device column) while trusted peer said that file had been modified by third device (trusted). I tend to think that to reproduce this issue three devices are needed. Let them be:

• T1: trusted device 1
• T2: trusted device 2
• U: untrusted device

All of devices share the same folder, all of devices are paired to each other and set up to synchronize in both directions:

• T1 ←→ T2 (sync in both directions, T1, T2 sends unencrypted files)
• T1 ←→ U (sync in both directions, T1 sends encrypted files)
• T2 ←→ U (sync in both directions, T2 sends encrypted files)

i.e. changes from T1 to T2 may be propagated directly or through U.

I believe that there may be some kind of race condition in propagation of deletions so that the problems happens not for all deleted files. Suppose I delete some file on device T1. T1 propagates deletion to both T2 and U device. After that T2 propagates deletion to U. At this point of time U receives two simultaneous deletions from T1 and T2 and probable handles it incorrectly. And bad state persists until more changes from T1 are received. While T1 is turned off, T2 and U always show Syncing 95% both.

[calmh]

For what it's worth, I'm unable to reproduce that while playing around with a triangle formation T1/T2/U and creating/deleting files. Some logs, some cli debug file for the affects file, might possibly clarify.

[dmitryash]

I see that there is strange device 7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4. Do you know who it may be? Will try to get some debug information.

[dmitryash]

Here is debug file output for two files for two hosts. I don't know how to match files to each other since one host is untrusted. Both files were modified by other device (actually android), I don't know how to get information from phone.

debug_file_output.zip

[calmh]

Thanks, interesting. The deleted files look like they were deleted "concurrently" on several (>1) devices, yielding several different "versions" of the deleted file. These are of course equivalent and this doesn't matter, except that all devices should ideally agree on which version is the latest, for completion purposes. The untrusted device doesn't get all the info the others have though, so its view of which deleted file is the latest can differ, and then it looks out of sync. I'll need to ponder this.

(77777 is the "local device" in the database)

[dmitryash]

@calmh Interesting. Well, I did the following:

1. Created three empty files F1, F2 and F3 and made sure that all three files are synchronized between all devices.
2. Disconnected (by using Pause) all devices from each other
3. Then removed files F1 and F2 from trusted device T1
4. Removed files F2 and F3 from trusted device T2
5. Enabled connection through untrusted device, i.e. T1 ←→ U and U ←→T2, but T1 and T2 left disconnected so that they could not speak to each other directly
6. Noticed that removed file F2 was added to the list of 'out of sync' items. F1 and F3 files are removed without problems.
7. Enabled direct connection between trusted devices T1 ←→ T2 but it did not help. F2 stayed in the list of 'out of sync' items. Both T1 and T2 show that F2 file was modified by T1.

It looks like deletion may not be correctly processed if state is propagated through untrusted device.

[imsodin]

Thanks, interesting. The deleted files look like they were deleted "concurrently" on several (>1) devices, yielding several different "versions" of the deleted file. These are of course equivalent and this doesn't matter, except that all devices should ideally agree on which version is the latest, for completion purposes. The untrusted device doesn't get all the info the others have though, so its view of which deleted file is the latest can differ, and then it looks out of sync. I'll need to ponder this.

Oh I think this might be an artifact of a fundamental issue with our "hiding" of the version vector on untrusted devices (we basically erase it). There are certain conditions where conflicts get stuck - which sounds a lot like what you describe. Last time I looked into it I didn't find any solution on the untrusted side - as far as I remember I pondered changing the general conflict resolution: Currently the losing side does that, the other side does nothing. The idea was to do conflict resolution everywhere. I do not remember if that idea was flawed or if I just didn't follow through. I'll see if I can find any record of that tomorrow.

[DanielBolef]

I'm also experiencing this issue. I have 2 untrusted devices. I've experienced this issue on version 1.23.7 & 1.24.0.

[jcotton42]

Any advice on how to fix this? Getting the same issue with an untrusted device?

[DanielBolef]

Still having this issue on version 1.27.4. Also, I think my previous comment has an error; my 2 devices are set up as trusted, not untrusted.

[digital-spinner]

I think I may have similar issue here: #9482