Status quo of syncany project?

[Harvie]

Hello. News on syncany.org say this:

13 Jun 2016: The core team of Syncany is on hiatus for an indefinite amount of time.

Last commit to this repository is more than year old.

There was never any stable release of syncany, just alphas.

Does this mean that syncany is dead? I think it's the only file synchronisation software with the right ideas (use any arbitrary storage, automatic colision resolving, end to end encryption,...). Was there something that went wrong with syncany, so there's no motivation to continue the development? The demo videos seem quite cool and only feature i miss from syncany (except for android support) is that people will test some version throughly, so we can call it stable and be sure that it will not eat our data... Unless there are some hidden problems with syncany approach.

What is the future of syncany?

[binwiederhier]

Hey there,

all the Syncany devs (just me and @pimotte really) have been working on their own day jobs for a while, so the project is pretty much de-facto dead unfortunately. I think the idea of Syncany is awesome and its implementation is not half bad either -- and that maybe we'll come back to it eventually. Right now though I just don't have the time to do it.

There was never any stable release of syncany, just alphas.

Overall, the project core is well tested and functional for almost all cases, until it goes bad for some reason. The sync algorithm is really complex and we've been trying to find edge cases for a while back when we were working on it.

The reason why it never made it out of alpha is because of these edge cases: People expect beta software to work; and if there are bugs, they don't expect them to delete or lose their data. To release a beta of a piece of backup/sync software, you have to be absolutely sure that you won't destroy/delete data -- and we simple were not sure of that. When you dedup->compress->encrypt data like Syncany does and you mess that up, the outcome is as bad as it gets if you are unable to reassemble the files in the correct order.

To be fair, I think most engineers would have been fine releasing Syncany as a beta -- I was just not ...

Was there something that went wrong with syncany ...?

I think the design is solid for the most part, but has a few flaws:

• Using vector clocks for distributed decision making/branching was a good choice, but it makes things super complex. Syncany is effectively a distributed system, even though its storage backend is mostly one central system.
• Not having a server component (only dumb storage) definitely made things very difficult (and some things impossible): No tie breaking for conflicts, no web interface, no sensible access control and sharing features.
• The encryption/dedup concept is rock solid -- or at least that's what I am proudest. The dedup mechanisms and the on-disk format are pretty reasonable and a good trade-off between resource usage and efficiency.
• ...

I could go on, but if I would redo it from scratch, I would probably try to simplify it a little and add a central server component into the design.

What is the future of syncany?

Good question. I will likely spend some time on it whenever I get bored, or I feel like tackling it again. I think @pimotte probably feels the same. It was a fun project and to this day it still saddens me that we never managed to launch it properly.

It has all the potential in the universe, but my failure to bring it to a beta and to attract devs let it fail.

[Harvie]

I would probably try to simplify it a little and add a central server component into the design.

I like syncany design because it's only one that has none central server component :-) But i understand that it's super hard to resolve conflicts with such design.

[pimotte]

@binwiederhier FWIW, even now that I'm more experienced as an engineer, I still fully stand behind (y)our decision not to call the software that's there beta. Even at the end we still experienced occasional bugs involving botched data. Granted, most were due to resource exhaustion in some way and people were really testing Syncany to its limits.

If I were to start over/continue Syncany, I certainly would stick with the no-central server component. I feel it is what makes Syncany tick, and if I were in control of the server, I'd probably just run Nextcloud and be done with it. The first feature I would scrap is the multi-backend support, in the sense that I would start PoC'ing with one, and then leave supporting different backends to others. (We tried to do this at the end by having people who were interested be maintainer of a plugin). I might also just build it for ipfs. The whole content-addressed data part would be heavily leveraged in Syncany.

I agree the encryption/dedup is solid. I know relatively little about those parts and that's because there were never any problems with it to begin with:')

I think the biggest problem with the current codebase is its architecture and unit test coverage.
The current architecture is sort of onion-like, where the core stuff (encryption/dedup) is unit tested, then every layer is integration tested up to the commands. This makes it difficult to refactor the middle layers, where some of the resource exhaustion is happening. It would also help finding the edge cases that cause current failures. If I was going to reboot, I would probably be comfortable continuing from this codebase, but in case of a full rewrite I would be heavily eyeing go or rust.

[theAkito]

Fair.

[schmunk42]

Sad to see this thread. This project looks great!

Are there any alternatives?

[theAkito]

Sad to see this thread. This project looks great!

Are there any alternatives?

https://github.com/borgbackup/borg/

[schmunk42]

Thank you, an important feature for me would be a also a Desktop GUI for end-users.

[theAkito]

Thank you, an important feature for me would be a also a Desktop GUI for end-users.

I think this is the only feature missing in Borg, yet.

Though, if someone is doing regular backups of specific locations, it is easy to create an alias for the command, so the user has to type in a single word and the backup will succeed.

[Harvie]

Unfortunately borg backup is not an bidirectional file synchroniser... It's just unidirectional backup program... It's made for different use case. You can't use it to sync documents that you edit on multiple devices.

[theAkito]

Unfortunately borg backup is not an bidirectional file synchroniser... It's just unidirectional backup program... It's made for different use case. You can't use it to sync documents that you edit on multiple devices.

https://github.com/syncthing/syncthing

[Harvie]

@Akito13 But syncthing will not work on arbitrary network storage. Syncany works on plain (eg.) SFTP server without having to install some sync daemon on that server. That means, you can only use servers, that you can install software on...

[theAkito]

@Akito13 But syncthing will not work on arbitrary network storage. Syncany works on plain (eg.) SFTP server without having to install some sync daemon on that server. That means, you can only use servers, that you can install software on...

I understand. Maybe you could improve an existing project or create your own?

[Forger125]

Thank you, an important feature for me would be a also a Desktop GUI for end-users.

I think this is the only feature missing in Borg, yet.

Just for the record, there is now a Linux/MacOS GUI known as Vorta for Borg.

[cstroe]

If one were interested in helping keep this project alive, what would the path to that be?

[binwiederhier]

We never got it to the point of a stable release. Instead we kept adding features and gimmicks. The most worrisome issues are the edge cases in the sync algorithm in which it just bombs out and leaves you stranded.

If you are interested in resurrecting the project, I can walk you through some of the crazy parts if you want to. It's been a long time so I'd have to figure out what the heck we did for some of these.

Let me know.

[Harvie]

Can you please give some most common examples of such edge cases?

[binwiederhier]

Most of them are hard to reproduce and are only encountered when you sync between multiple parties all the time or have long gaps between syncs.

I cannot remember a specific bug, but if you look in the GitHub issue history there are some nasty ones in there.

Or try for yourself to stresstest Syncany with a few clients: move large folders around on multiple sites, take one client "offline" for a few sync cycles and then have them catch up while you have made offline changes. There are countless possible scenarios.

[binwiederhier]

There may also be some GitHub issue references in the code. Grep is your friend. Sorry I have very little time right now. I'm on mobile and my wife just had a baby a few weeks ago.

Always happy to help if you want to do a knowledge sharing session.

[pimotte]

I'm also happy to help onboard/answer questions:) Feel free to create issues. If you prefer instant messages, I just rejoined our gitter channel: https://gitter.im/syncany/syncany (Though I'm in Europe, so timezones :'))

Example of an issue that is quite critical: #533

Beyond this, a huge issue was that Up was more efficient than Down, so it was possible to upload a bunch of files, but the client would crash on the DownCommand due to resource issues. This has been mitigated, but I suspect its not to the point that there is work to be done here. I second the approach @binwiederhier suggested of stress testing. The issue is mostly due to a lot of different files, rather than large files, iirc. Artificially limiting the jvm heap size also helps pinpoint these issues more quickly.

Logic-wise, I'm most worried about the CleanupCommand. Up/Down have had a lot of attention, but cleanup deletes data and that's the most risky part.

Architecture-wise, it has been remarked that our testing strategy is largely integration based, and there are relatively few unit tests, especially when it comes to the logic of commands. Unit testing the Up/Down/Cleanup logic might be a good point to get familiar with the more hairy parts of the codebase.

[cstroe]

Thank you for the pointers, I will go in the direction of writing some unit tests for existing uncovered code. Bug #533 definitely sounds like a bad (and scary) failure mode, but I would have to spend more time with the code to understand it.

[JaneX8]

Oh this issue is a sad one. I just found out about this amazing project and what a good one! I'd love to see a stable release as well. It seems like an amazing product and honestly one of a kind in the category open-source.

Maybe some crowdfunding could help? I'd be happy to donate if it would result in a stable release and continuation of this amazing work.

[binwiederhier]

Sorry :-(

I wish I had time for the project too ... Maybe one of these days ...

[darkdragon-001]

I just found out about this project and it immediately made it to the first position on my list for sync solutions (based on the philosophy and features). Thanks a lot for all the effort which had been put into it! Sad to see it's quite dead 😢

I think one of the problems with onboarding new developers is Java (I feel exaclty like this guy on Reddit).

Another problem is probably focus. It is tempting to support all use-cases especially when it is easy to get started creating new plugins. From my experience (and probably yours now as well), it would be better to start with a single plugin (or the two or three most common ones), write a lot of tests and bring the project to beta and then stable state. Experimental plugins could co-exist but should not be focused on.

Every supported protocol/format has it's own problems (see Dropbox's problems with Linux file systems, rclone with the massive amount of cloud storage backends) and needs to be maintained for changes and regressions. Combing efforts reduces these drawbacks (see restic backup using rclone as backend). All these projects are written in the go (golang) programming language. Hopefully someone picks up development!

As for alternatives, I just stumbled over SparcleShare which is based on Git with the large files extension Git LFS. So basically any git server (like Github, Gitlab) can be used as storage backend. There is Rclone working with most cloud storage providers but only providing a cli which has to be executed manually and can only sync in one direction. There is also Cryptomator also providing smartphone apps and adding encryption on top of Dropbox/GDrive/OneDrive and the likes but having no own sync logic. Seafile and Nextcloud require their own server. Syncany is really unique with its GUI, automatic sync and exchangable backends! This is exactly what I am looking for quite some time. Hopefully someone picks up the work!