Sync upgrade draft

@tessus, this is planned as part of the sync upgrade. The more pressing issue is to make sync more scalable as several users are hiting the limits of some providers (someone reached the max number of files per dir in OneDrive, and someone else in pCloud), but as part of this change I’ll also include useful features like the ability to sync by item type.

The first step was to implement the lock mechanism so that sync targets can be locked and their structure changed.

That’s the draft spec I wrote some times ago:

  • [ ] Split into item type directories - folders, notes, tags, etc. to allow syncing certain items first
  • [ ] In each dir divide into 0, 1, … a, b, c, etc. - will allow 150000*26 items = 6 millions
  • [ ] Whenever adding file to it, call mkdir and cache that it’s been done
  • [ ] Check that errors are handled correctly if dir no longer exist after having been cached
  • [ ] Keep all lists methods the same - for webdav use PROPFIND with depth param to recursively get everything. For files, the same.
  • [ ] Means no change to delta either. Dropbox and OneDrive should return all latest items (use recurse for Dropbox)

  • [ ] Later - create Archive concept - items that are archived are read-only and moved to an archive folder in sync target. Doesn’t need to be synced as often.

@laurent Thanks for the detailed information. I moved your reply to a new topic in the dev category for reference.

The fact that enc keys were synced at a very late stage came up a few times and I always tried to wrap my head around it. After thinking about it for quite some time, I couldn't come up with a good reason why that was. So before my head exploded, I had to ask. :wink:

At the moment it syncs in random order. It depends on which order the backend sends the files, probably in alphabetical order.

Agree that making the encryption key higher priority for syncing would be a great improvement!