flat data structure with plain ics and vcf files

[eoli3n]

I can't understand the data structure, why didn't you use a plain data structure as khal and khard with vdirsyncer ?

I expected DecSync to be compatible with those. I wanted to be able to replace vdirsyncer static sync that uses DAV with decsync + syncthing. I like the way khal/khard let me manage plain files, to keep cross compatibility, it just sync ics and vcf files with a simple dir structure.

Evolution DecSync

➜ tree share/docs/decsync 
share/docs/decsync
└── calendars
    ├── colID14643
    │   ├── local
    │   │   ├── osz-Evolution
    │   │   │   └── info
    │   │   └── osz-Evolution-96506
    │   │       ├── info
    │   │       └── sequences
    │   └── v2
    │       ├── osz-Evolution
    │       │   ├── info
    │       │   └── sequences
    │       └── osz-Evolution-96506
    │           ├── 39
    │           ├── info
    │           └── sequences
    └── colID15653
        ├── local
        │   └── osz-Evolution-29453
        │       └── info
        └── v2
            └── osz-Evolution-29453
                ├── info
                └── sequences

13 directories, 11 files

➜ cat share/docs/decsync/calendars/colID14643/v2/osz-Evolution-96506/39
[["resources","4bf1df818e053d2587f14f08da81c18e9764dad2"],"2021-12-28T10:12:53",null,"BEGIN:VCALENDAR\r\nCALSCALE:GREGORIAN\r\nPRODID:-//Ximian//NONSGML Evolution Calendar//EN\r\nVERSION:2.0\r\nBEGIN:VEVENT\r\nUID:4bf1df818e053d2587f14f08da81c18e9764dad2\r\nDTSTAMP:20211228T100737Z\r\nDTSTART;TZID=/freeassociation.sourceforge.net/Europe/Paris:\r\n 20211228T120000\r\nDTEND;TZID=/freeassociation.sourceforge.net/Europe/Paris:20211228T123000\r\nSEQUENCE:2\r\nSUMMARY:Test\r\nTRANSP:OPAQUE\r\nCLASS:PUBLIC\r\nCREATED:20211228T101253Z\r\nLAST-MODIFIED:20211228T101253Z\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n"]

khal

➜ tree khal
khal
├── anniversaires
│   ├── a50d49fb-9812-401d-a8c8-584cb0531a08.ics
│   ├── aa6efc11-02a1-4d7c-960c-c62a071594a3.ics
│   └── ffdaf9d2-cc84-444c-bcbc-c274b9b5f646.ics
├── home
│   ├── 004a79fc-72c3-47b0-b21d-ed810457655e.ics
│   ├── fd2b182e-bf75-43a8-a455-4658257a11a1.ics
│   └── FDA6A018-28D1-4447-A026-E7C9031B0F1F.ics
└── tasks

khard

➜ tree khard
khard
└── contacts
    ├── 02591900-045D-45E3-B3A0-7EBA96565B82.vcf
    ├── 03A4088F-93F6-480E-8226-56EE138141BE.vcf
    └── 066de82a-281b-478b-ad35-510e78877d7f.vcf

[alensiljak]

Radicale seems to split the contacts into individual files in its working folder.
It is true that having the standard files (vcf, ics) would probably provide a better feeling for the end users.

[eoli3n]

If some data needs to be appended, it could be done in another dir, with a reference to the file.
Any way to see this improved in next versions ?

[39aldo39]

The main reasons for the structure are the following:

• DecSync prevents syncing conflicts from occurring by having separate directories for every device. Syncing the DecSync directory between devices should just work.
• It doesn't use plain ics/vcf files because DecSync is designed more general, for contacts/calendars it just happens to be mostly simple strings. Theoretically there could maybe be an exception for contacts/calendars, but that makes it more complicated and even then it wouldn't be directly compatible, as DecSync also uses timestamps that cannot directly be replaced by the modification time.
• As for compatibility, that is indeed not optimal, but the support for a vdir is also very limited. The main compatibility layer in my view is Radicale DecSync which adds compatibility to any CalDAD/CardDAV client, which is supported very broadly.
• The feeling for the files could be improved, but isn't a priority compared to the previous points. It is difficult to combine it with having a good syncing experience.

[ec1oud]

It seems needlessly ugly to me, with uuids and other made-up filenames. E.g. instead of contacts/7f2518db-e928-41e6-a3f6-6ace79b9cf6d/v2/Nexus%205X-DecSyncCC-82800/2f I want contacts/personal/john\ allan\ doe.vcard. And it should be a plain vcard 4.0 file, not escaped and embedded in json. (Not that vcard is so elegant, but it's old and it's an IETF standard, and many applications know how to import and export some bastardized variant, so it's better than inventing something new at this point.) And there should be a stb-style C header "library" for reading these things, and preferably for writing too.

The way I see it, the problem to solve is not "synchronization" exactly: it's representing contacts and calendar entries in such a standard form that I can use syncthing to sync directories full of them between all my actual computers and devices, and there should be a compliant application on each of them (on every operating system) that reads and writes those files. I'm one person, why would I want a separate calendar or separate set of contacts per device? The whole point is to always see the same up-to-date data on all of them.

You maybe will say you don't believe in replacing the Android contacts app? There are already lots of those in f-droid though, and I don't really see what is the point with all of those (wasting developers' time?), because they are using who-knows-what as the database and not exposing it: they don't tell you, you aren't supposed to know. (I know, that's the way on mobiles. So-called security, because the assumption is none of the apps are trustworthy. But I'd rather use free software, that way I will trust it and there is no need for the damn silos and isolation.) At the same time some of them don't use the built-in Android contacts database either. I want to rather use one-file-per-entry on the sdcard storage, available to all apps that want to share it (subject to permissions of course).

I tried them all, and DecSync is the first thing I see in F-droid that even tries to write one file per entry. It's just not really very portable.

I don't want to run CalDAV or some other manager daemon like Radicale: only syncthing. Even if I did want a "personal cloud" service, I'd want it to be a real standalone binary, not a Python script. The only reason for such a thing would be to work with proprietary OSes like iOS and such. I tried NextCloud; it's not even able to handle an upgrade from one version to the next without corrupting its own database. So much for that. And all this complicated cloud crap is the same or worse. The point with syncthing is simplicity: let's separate the the data representation problem from the sync problem, and use it universally. (An alternative would be the plan9 way: remote mounting instead of syncing. The application should not care which one you choose to use.) When it comes to version control: we have git for that. I should be able to store my calendar in a git repository, shouldn't I? just to make sure that if buggy software makes a mess, I can restore.

I didn't get my Librem 5 yet (waiting for years now); I wonder what sort of mess they have made of their contacts and calendars apps. I hope this is the year I can finally find out. If it's a mess, then I will write my own; that's one reason I'm not in a hurry to write my own Android apps.

I want to be able to edit these files with vi on any system where a nice front-end UI is missing. That's why the name should make sense, so I don't first have to do a bunch of grepping just to find the right one.

Reading and writing plain files is the only way that will stand the test of time. Any other way you write it, you are writing disposable software.

[39aldo39]

Let me elaborate on the problem of syncing conflicts. Suppose that all devices use the same file entries with the content {"foo": 1, "bar": 1}. Device A changes the value of foo to 2 and device B changes the value of bar to 2 while they are disconnected. This means that for A the file now contains {"foo": 2, "bar": 1} and for B it contains {"foo": 1, "bar": 2}. This is a file conflict and the file syncing software that is used cannot merge these files and will probably either select A or B, removing the update the other device made. It may let the user know there is a conflict in a way, but we cannot know how and either way it requires manual intervention by the user.

There are various ways to deal with this. The ones I can think of:

1. Accept the conflicts and let the user deal with it. If everything is always connected this can work, but otherwise it leads to a very bad user experience with lost data and/or manual resolving conflicts.
2. Write (almost) every value to a separate file. In the example this would be having two files, e.g. entries/foo and entries/bar with, in the end, both content 2. This will resolve most file conflicts and only the ones where the same value is concurrently written remain. This could be acceptable for contacts/calendars, although the file conflicts are still undesirable. However, for e.g. RSS this would require having all articles as separate files which would be a huge downside.
3. Let every device have its own subdirectory where only itself is allowed to write. This resolves all file conflicts and allows for more flexibility in storing the data for different tradeoffs. But this does mean that the data is duplicated.

With DecSync I have taken option (3) as I prioritize having a good user experience without having any file conflicts. Note that the structure stored by every device is still very flexible and could be plain ics/vcf files. However, the main drawbacks are:

• It is less general. I designed DecSync to be able to sync any data, hence making the contacts/calendars different requires more work, not less.
• Even if it was stored like a vdir for every device it wouldn't really add compatibility. As every device has its own subdirectory they won't communicate with each other without having extra software anyway.

It is possible to design a solution with different decisions, by for example basing it on (2). However, DecSync is not designed that way and if you really want that you have to use something else.

[ec1oud]

Sometimes it amazes me the degree to which git can avoid conflicts using plain old diff and patch. Of course it's a bit artificial that these old tools treat line feed as the universal record separator and don't look for finer-grained diffs; but vcard/ics are line-oriented too, so I suspect it would be mostly OK if one device changes one field and another device changes another. (especially if they don't happen to be adjacent lines... as silly as that is) If the same person changed foo and bar to 2 on two different devices, didn't he mean for both of them to end up being 2? And syncthing deals with conflicts in its own way anyhow; they are rare enough that I don't mind resolving them once in a while (on a computer, not a phone).

Another way would be log-based storage; but then it's a journal instead of a declarative format. Yeah, I was also thinking a long time ago that facts are fluid, each one is true only with a degree of probability and for a discrete period of time (like how long a person is alive, how long married, how long has the same phone numbers, the same job... or whether the person lied about some facts and you find out later) so declarative formats don't quite model the real world; but at least they are simple. But if the software was already designed to handle multiple conflicting facts being true for different periods of time and different probabilities, there is no such thing as conflict. If you resolve a conflict, you are actually destroying information that you used to believe was true, or that was true for some period of time. But at least if you are using git, you have a time machine. Whereas genealogy software has to deal with ambiguity in other ways.

DecSync probably can't turn into what I want; I need to fork some Android contact and calendar apps for their UIs and rewrite the storage layer, if I want it badly enough. I just figured it's not worth my time if I don't keep using Android for the long term. It's frustrating that this is at least a 25-year-old problem (if we go back to the PDA era) and still isn't solved, and so little software is designed for the long term that it's hardly even worth solving on today's mainstream platforms. Everything breaks every few years, and commercial developers never care about interoperability.

[ec1oud]

https://github.com/libvc/rolo has a nice terminal UI but needs all the vcards in one file, apparently. Maybe worth fixing.
https://github.com/DarkHobbit/doublecontact looks decent so far, but has a bit overcomplicated UI.
https://github.com/mbideau/vcardtools successfully split up my Android-exported big vcf file into nicely named vcards.

[alensiljak]

@ec1oud, DecSync's storage files contain the vcard and ics records in original format. They are one step away from having those in the separate files. A simple Python script that reads and parses the array of records is all it takes.

[eoli3n]

They are one step away from having those in the separate files.

The step is too much, that's import/export functions, not cross compatibility. I want to be able to read synced ics/vcf files with khal or khard, without having to export from decsync.

[alensiljak]

The step is too much, that's import/export functions, not cross compatibility. I want to be able to read synced ics/vcf files with khal or khard, without having to export from decsync.

That makes sense and I'm leaning towards this more lately. I've been using git for text content outside code, mostly for notes / knowledge base used on multiple devices, and the synchronization seems to be well handled in that scenario. I like lazygit.

With the sync side being taken care of, it would be optimal to have just the standardized plain-text files as the storage, I agree.
The approach has its pros and cons, of course, but perhaps the sync should be separate from data storage. There are other specialized sync solutions out there (i.e. syncthing, git, mercurial, or whatever else).
Just $0.02 extra.

[eoli3n]

You point on something.
@39aldo39 Why don't you rely on the sync tool ability to treat conflicts ? Syncthing manage conflicts its own way, as git does.
https://docs.syncthing.net/users/syncing.html?highlight=conflicts#conflicting-changes
Why would decsync does it on top of it, with its own way ? That should not be its job.

You should not care about conflicts. If there are some specific configurations for a specific sync tool, then you could ask the user to set a line in a config file.

sync=syncthing
# sync=git

[39aldo39]

Because relying on general purpose sync tools doesn't really solve the problem, as I already explained in a previous comment. They handle it in their own way and simply pass it on to the user. For example, for Syncthing the user gets a conflicting file and has to merge them together manually, while possibly not understanding their format, if they even notice that there is a conflict. This is in my view a terrible user experience, especially when data contains many entries, which is the case for RSS. You could maybe use specifics of the conflict treatment of the sync tools, but I like to keep it agnostic of the specific sync implementation.

[eoli3n]

The price is a terrible implementation, breaking all cross compatibilities, with the need to import/export to use with any other tool.
To me, the user experience is horrible as is, I would prefer a "harder" conflict management based on the sync tool than using something which lock me up in its own way.
As is, no way to sync with git properly, as you wrapped ics files in a oneline way, any change will diff the whole file/line.

With all the respect to your work, you should consider a midway solution: try to not wrap files, keep a non flat directory hierarchy, but clean the file part. Try to simplify as much as you can the hierarchy and clean it to be understandable by a human.

[39aldo39]

DecSync is designed to be more general and allow arbitrary key-value mappings. This is a major point. If it was designed for just contacts/calendars it could be significantly different.

To me, the user experience is horrible as is, I would prefer a "harder" conflict management based on the sync tool than using something which lock me up in its own way.

This is a trade-off and DecSync gives priority to an "easier" conflict management at the cost of having more complicated files.

As is, no way to sync with git properly, as you wrapped ics files in a oneline way, any change will diff the whole file/line.

I don't see how using git is not possible, as every device gets its own directory. How and between what devices are you trying to sync?

With all the respect to your work, you should consider a midway solution

I also don't see how having a midway solution actually improves the situation. If it is not exactly a vdir it still doesn't provide compatibility with anything and importing/exporting remains non-trivial. But the cost is not having arbitrary key-value mappings and making special cases.

[alensiljak]

@eoli3n have you tried Xandikos?
Unfortunately, DecSync with Radicale doesn't run in Termux due to some glibc dependencies so I'm looking into Xandikos as a potential solution.

[octvs]

@eoli3n and @ec1oud brought up really good points. Points which would make decsync stay test of time rather than to be stuck in its starting perspective. It would have let users use clients like khal/khard, or any other that will spawn and expect a reasonable file structure.

I guess one point why there wasn't no solution to the discussion is that, the users mentioned want a reliable tool for card/caldav management while @39aldo39 aimed a more generic tool. Which brings up the question, and this is lost on me, why we need decsync to be more general sync solution? Are we implementing a syncthing on top of syncthing that can manage conflicts by introducing an horrid file structure?

Nevertheless, thanks for the effort. A bit hit and miss for a larger user base.