Request: Multiple sync targets to support versioned, encrypted, offsite git backups?

Apologies for the long topic, just trying to make sure I explain myself thoroughly. Thanks for reading!

Feature request

Can joplin add a feature to simultaneously(or serial) sync to multiple locations?

It seems to me that joplin is able to support multiple sync targets, just not simultaneously. One can switch back and forth between targets, it will sync fine. Can we make this always happen rather than needing to switch back and forth?

Details

I understand this looks like a workaround but if there was support for multiple sync targets I would be able to version, encrypt, and offsite my backup, in a joplin native way while still being able to sync my notes with my phone.

Setup:

  • Setup encryption configuration
  • Sync to cloud provider for mobile sync
  • Sync to file system to a git managed directory. This way I get encrypted versioned backups as opposed to raw export
  • Use fswatch or cron to trigger git add, commit, push for rollback. While files may not be totally atomic, in a disaster I should be able to recover the majority of my notes since they’ll eventually fsync to the filesystem and the disaster would have to happen during a sync. If a half write was committed, I can just keep rolling back until the files make sense. The versioning can also protect against silent corruption

Reasoning:

Even evernote has had a history of data loss and a backup strategy is key for any long term note strategy.

As a user of the mac gui app, I don’t have access to the cli export command, and I’d like to avoid using the cli with the --profile flag since then I have to worry about if the sqlite formats have changed between the cli and gui app because one version is out of date. Additionally even if I did have the cli tools, raw exports are unencrypted, and since I want offsite backups, that’s a non starter

Even if I had the export commands, and exported to jex or a raw export then ran my own encryption, I’d be only doing what joplin already knows how. I want to take advantage of each .md being encrypted so that each checkin only increases my repo size by the size of the new encrypted note rather than a large blob

Downsides:

The encryption code becomes even more critical path, if there’s bugs in encryption, both the backup and the cloud version will be corrupted. This is mitigated by git versioning that most likely not all your notes will be mangled, but it is a risk. Hopefully if this ever happens this would be caught while at least one plaintext version of the notes exists and then we have an escape hatch, but its not foolproof

(Sneaky alternate feature request)

Can the mac app come bundled with it’s own scheduled export command or a simple cli tool for export? That way, I could also work around creating my own backup strategy like laurent’s.

Syncing is not a backup !!!

Imagine you have 3 sync targets. For some reason some data is missing on one target. This means that the missing data is deleted from all targets and the local client. I would hardly call this a backup solution.

The only way to backup your notes is to do an export to JEX. So, if you really care about your data, you’ll install a terminal client on a server and run a cron job to export your notes to JEX.

2 Likes

Interesting, in your view fundamentally what's the difference between sync and export? Sounds like I'm misunderstanding some philosophy or intention here


For some reason some data is missing on one target. This means that the missing data is deleted from all targets and the local client

Isn't this true of export as well? If there's silent corruption where export is dropping files

Imagine you have 3 sync targets. For some reason some data is missing on one target...

Isn't this unrelated to the number of sync targets, but more related to the durability of your notes now being the minimum durability of syncTarget1, syncTarget2, syncTarget3 service and each implementation of syncing code? This doesn't sound like an increased risk scope since, if such a bug exists, joplin will happily silently delete your note, and then your export won't have said note either.

Fundamentally with today's sync implementation, you're right, but without any automated export functionality or CLI access, I'm looking for workarounds. At some point, though it might be easier for me to try to make the changes, but I'm not sure what kind of changes the joplin maintainers would welcome. I'm sure you guys(maintainers) have some longer term views on what sync means and what use cases it's to support that aren't immediately obvious to me. I can think a few workarounds to your concern here on missing files in targets, but they're probably not things that would make you guys happy :slight_smile:


Tangentially, this mistrust of sync more points that joplin is losing metadata in syncs. Joplin can't distinguish a delete from non existence, and with work, it could have better validation and robustness in this process rather than be afraid of syncing.

If more work was done around versioned sync or point in time syncs, joplin would be significantly more durable and able to better handle durability issues in backend sync services. For example, dropbox doesn't have any durability guarantees for an application developer to consider which puts the mitigations on the end application. I wouldn't be surprised if silent corruption occurs one day because the dropbox API doesn't return a file due to some secret eventual consistency. With a single syncer this isn't a problem but under multi-device there begins to be some nice race conditions that can probably make joplin unhappy

I agree with the multiple sync target idea.

I want to use Nextcloud and OneDrive, so when one down, I could use the other, and sync it again when the down one turn back on again.

Reconnecting, again and again, is tiring.

(´。• ω •。`)