Standalone Sync API

Features
1

Hi, I saw Standalone Sync API on GSoC 2024 ideas list. I see it shouldn't use dependencies from main app, but it needs to interact with local database (SQLite3) for syncing.

I want to ask how abstract this library should be? Should It take any arbitrary database interface, or works on joplin sqlite database schema only.

2

It's not that it shouldn't use the dependencies from the app, it just doesn't have to.

Up to you really. As long as it can sync with other Joplin clients.

3

I'm implementing a basic demo that has 2 sqlite databases: db1 write to onedrive, then db2 sync from that. The logic from original code pretty much the same, but different structure. Since there're so much details, if i change so much, im afraid it might break.

Currently, I will setup same Joplin Sync Architecture, but with minimal code for demo above to work , then I'll fill in the details/reimplement each part later.

Am I heading in the right direction?

4

Maybe. I think the database side is trivial, but the sync logic is much more complicated.
It may help to think of some use cases and work from there.

5

I'm thinking of some more control from users over the syncing, the idea is to provide some services/functions to run at different points of syncing process. Some use cases I can think of:

  1. Custom conflict handlers: at the handle conflict step, user can provide a service to handle conflicts as they want
  2. Data backup: if a delete action is performed on remote, user can provide a service to handle the deleted items, for e.g: save these items to another storage (this prevent data loss for accidental delete).
  3. Custom encryption: in encryption step, user can provide service to use their preferred encryption algorithm.
  4. Restricted items syncing: instead of syncing everything, user provide a service such that it filter out only a few items/folder to be synced
  5. Notification / logging: like Firebase database, when server data is changed, it emit event back to client.

These are just some ideas going on in my mind, and may not be practical yet, but I'd love to see how useful each of these might be. I hope to receive feedback from the community, thank you!

6

It might be better to think from the end user perspective. Come up with some ways in which the API may be used.

I'll start, just off the top of my head:

  • Email to note that does not require a plugin or Joplin Cloud. Note that you don't even need to have the full sync logic, all you need is to be able to push notes to the sync target.
  • Republishing notes to some other service, like a blog or something. Similar to the above, here you only need to be able to read from the server (and detect changes). And if you only read, there's no chance of data corruption.
  • Server-side OCR, AI processing, etc
7

I think listening to sync target changes & able to push without applying local syncs are 2 essential features. I'll be aware of these while implementing the sync library.