Performance with large collection

gufertum · 2018-08-22T06:41:29.982Z

I imported a large collection of website dumps (like ‘read it later’ archive) and have some issues. The import process was OK (using the web-clipper curl API), but the sync now takes like forever…

Are you aware of some size limitations?
Is everyone else having problems with a large amount of nodes?

What could be the bottleneck?
I use WebDAV sync and an OSX client.

Regards, Thomas

foxmask · 2018-08-22T09:17:05.478Z

in may i had that issue I confirm large sync takes age on Linux, not on Windows
The only size limitation is with Onedrive (file size cant not be very big) said @laurent

gufertum · 2018-08-22T18:24:28.318Z

I changed to local sync (Filesystem) for the initial sync and copied the files on the WebDAV folder - by hand. Completed in seconds...
Downloading takes some time but is way faster than WebDAV uploads.

tessus · 2018-08-22T19:37:55.707Z

Nice workaround. We should put that in an FAQ.

foxmask · 2018-08-22T20:07:17.899Z

cheater

francoisbruneau · 2018-10-08T10:32:09.348Z

How are subsequent syncs performing? Did you revert to Joplin’s WebDAV sync method? Post-initial syncs with no changes still take a very long time on my Mac (about 1h with ~6’000 notes & end-to-end encryption enabled).

gufertum · 2018-10-08T11:58:04.732Z

Yes, I only did the initial sync via local files and then switched to WebDAV sync. But before that i copied the locally synced folder to the WebDAV folder…
Now the WebDAV works, but it sure is slower (network and webdav overhead), but it’s doing ok.

laurent · 2018-10-09T18:57:20.887Z

tessus:

Nice workaround. We should put that in an FAQ.

I think there's something like this already in the FAQ, if not we should indeed add it or complete what's in there.

laurent · 2018-10-09T18:59:16.518Z

The next version should be a bit faster for initial sync as it will download resource data (which take a big chunck of the time) in parallel to notes, notebooks, etc. It also means the app will be ready to use earlier since notes will be there, and then the resources will keep downloading in the background over time.