I’m new to Joplin. I’ve installed the mobile version yesterday on my Android 10 device, deleted the default entries, created own entries and set up sync to my NextCloud. Anything is fine, initial sync took only a few seconds, than any sync needs less than 2 seconds. Perfect.
Later, I’ve installed the Linux client in the version 1.0.170 on three different Linux machines with Kubuntu 18.10 / 19.04…
…initial sync to my NextCloud needed more than 10 Minutes to write the default (“Welcome”) entries to NextCloud and than additional minutes to sync my via mobile manually created folders from NextCloud. After that (and after again deleting the default welcome entries) any sync takes nearly 4 minutes!
On each of my Linux machines each single sync needs nearly 4 minutes - with just having three empty folders (Notebooks) and no entries right now. Same sync to the same NextCloud URL takes less than 2 seconds.
I’m sorry, but I don’t have Linux with a DE to debug this.
I’d like to get a profile of the app during sync to see which functions take that long. Someone with a Linux box must be able to investigate this and know how to profile an app. It would be a great help to narrow down the issue.
@tux I’m running Joplin and syncing with Nextcloud on my Linux Mint laptop, my Mac mini, my iPad, and my iPhone and I have noticed that the sync seems to take forever on each of my devices, although it seems to be the fastest on my Mac. The sync can be counted in minutes whereas with other apps syncs are measured in seconds. To me that is too long and I have long been wondering if the problem was my internet speed, Joplin itself, Nextcloud, or maybe me having too big of a database (but I don’t think my database is that big).
How do I get a “profile of the app during sync”? I have the csv of the sync report, is that what you want?
No, I was talking about using oprofile on Linux. Or maybe perf, which is newer and has some nice new features. I mentioned the following: Someone with a Linux box must be able to investigate this and know how to profile an app. So I hope that someone with profiling experience can get this at one point...
Next to the profile a network trace with request/response times would also be helpful.
If you are not a system or performance engineer, this topic is not something I can explain in a few posts.
At least that is the only way I can think of to investigate this. But maybe somebody else has a better idea.
Sorry, I have no idea how to oprofile an application, so I can’t help with this. But I can see the difference in my webserver logfile. When sync’ing with my NextCloud 17 this happens:
Additional info: the same problem happens with NextCloud 16 and after update to NextCloud 17, using Kubuntu 18.09, 19.04 or - no completely fresh updated - 19.10. Tested resulting in the same problem on 3 notebooks, 2 Desktop-PCs and a VirtualBox-VM.
Doesn’t matter if I download the .AppImage-file and simply start it or if I install Joplin with the .sh-Installer. Same thing with version 1.0.170 or the newest 1.0.175.
This output only shows us that it takes a long time, but not why.
That's why we need an app profile (oprofile, perf) and and a network trace (Charles proxy, MITM proxy): The profile shows us which Electron calls/functions take a long time and/or system calls, and the req/resp info from the network trace will show us which request take a long time and where (client, server, connection).
Hi All!
First, thanks for the great job with Joplin.
I have the exact same problem: Nextcloud sync super slow on Ubuntu (19.10), when it fast from Android (Joplin version: 1.0.197).
I actually think it might not be specific to Nextcloud, but rather a WebDAV sync issue. Indeed, I tried setting up the sync using the "WebDAV target" against my Nextcloud server, and the issue is the same. (Unfortunately I don't have a non Nextcloud WebDAV server available to do the test).
So, I tried to dig a little bit. I first wanted to have a look a the network calls being performed, so I've setup a mitmproxy, and here comes the funny part: no more sync speed problem when using the proxy
And it is very consistent: the sync is always fast with the mitmproxy, and always slow without it.
So I had a look at the execution profiles, using perf. I did profile one fast sync (with mitmproxy) and one slow sync (wihout mitmproxy). The only obvious difference I can see is the overhead of the VizCompositor and Compositor processes when the sync is slow. And honestly I dont know how to understand it, as I believe these processes are related to rendering...
(I can share the complete perf data files if needed, but I don't want to put them here, as it is quite big)
Actually, I can share a tip that could be useful for others having the same problem. It is a workaround based on direct WebDAV access:
I’ve mounted my Nextcloud as a WebDAV volume on my desktop (with davfs2). In Joplin, I configured a “File System” sync, giving the Joplin folder on Nextcloud as the directory target.
And the sync is now blazing fast from my Ubuntu environment, and the data correctly synced on all my devices.
Add a crontab to that that runs at a reasonable interval and use one of the cli clients available, this solution could be extremely viable. My nextcloud instance syncs every 5 minutes since i have a home backup system setup on my nextcloud instance using fossil, but it definitely isn’t difficult to setup.
Since Joplin is not longer syncing correctly with NextCloud, I also have a mount point created via davfs2 pointing to the NextCloud folder, automatically mounted by systemd, whenever needed by Joplin.
So Joplin does not longer syncs to NextCloud/WebDAV directly, it syncs to this folder / mount point, systemd mounts it immediately when needed, wherever I am in the world with my notebook, perfect workaround.
So easy - I simply love Linux. More and more every day
Yes, no need of a crontab: using davfs2, you make your Nextcloud folder accessible via a local mount point. You can then fully rely on the “file system” sync mechanism provided by the Joplin app.
After installing the davfs2 package you can mount your WebDAV/NextCloud folder directly. Permanently with set up via /etc/fstab or as an auto mount via systemd. I perferred the systemd-Version, so it is only mounted automatically only when really needed.
I’m not using systemd, so I may need to keep using my current setup for time being. Thanks for answering. I’d have to research runit scripts for that part more
It is actually very simple to setup: just install davfs2 and configure it to store your credentials (you will easily find some more info on the web about that if needed) and add a entry in /etc/fstab so that your Nextcloud is mounted automatically into your file system on startup. Something like:
