Firstly, to the creator and developers of Joplin: thank you! Using Joplin has been a wonderful experience so far. My one pain point has been the need for an intermediary syncing server such as Dropbox. A serverless sync feature would be great.
The CAP Theorem in computer science states that Consistency, Availability, and Partition tolerance (in other words, offline support) are a triple constraint for networked services -- you can only get two out of the three. This CAP Theorem spawned a generation of NoSQL DB's (typically tuned to sacrifice one CAP constraint and provide the other two) in the early aughts.
A relatively new group of data structures called CRDT's joyously violates the CAP Theorem. With CRDT's we can have all of the above: consistency, availability, and partition tolerance. By implementing something called CRDT's (Conflict-Free Replicable Datatypes), we could have reliable, provably correct peer-to-peer syncing across unlimited devices without requiring any intermediary server (Dropbox, Nextcloud, etc.). Given that Joplin is based on JavaScript, Yjs seems like a good choice for adding this serverless sync functionality. Yjs is a CRDT JS lib available on GitHub.
@laurent what are your thoughts regarding the feasibility of adding this serverless sync feature to Joplin?
Do you envision a gossip protocol to transport and propagate data?
And how does this work with a network with only my own devices as peers?
Or isnt that neccessary?
Hi @joeldebruijn , a lot of the use cases described for CRDTs are collaborative editors. So I think that we can repurpose that use case to Joplin, but instead of collaborative editors, we have collaborative devices for our personal Joplin notes (those of us using more than one device). So I'm thinking we can apply the same CRDT logic to how we sync among devices.
As far as gossip protocol or some other way to broadcast changes, I don't have much insight to offer, but I think each online client could periodically sync directly with the other online clients and as for offline clients, CRDTs can auto-merge everything without conflicts using something called a LWW (Last Writer Wins) strategy whenever an offline client comes back online and syncs up with the others.
I'm not well versed either, but collaborative editors assume a multi-device-network.
Even for one person the challenge remains the same: sync and merge from multiple sources with concurrently editing.
I only encounter 'direct syncing' without central coordination, between devices with bluetooth / usb / direct wifi connections.
Or we all have to run a bittorrent network in 'Limewire' style.
I think the requirement to have the clients online at the same time for sync to work is too much of a constraint. For example, I write a note on my laptop then switch it off. Then later I open my mobile to check the note - it's not there and syncing won't work. The only way is to go back home and turn on my laptop again.
If we had such a setup, for sure one of the first feature request would be to have an intermediate device that's always on and with which any device can sync at any time... and that's called a server, which of course is how the app already works.
