Sync with NextCloud fast with mobile, extremely slow on Linux

@tux and @kara, would using davfs2 make using a crontab to sync up obsolete since it’s mounting directly to the server?

Yes, no need of a crontab: using davfs2, you make your Nextcloud folder accessible via a local mount point. You can then fully rely on the “file system” sync mechanism provided by the Joplin app.

After installing the davfs2 package you can mount your WebDAV/NextCloud folder directly. Permanently with set up via /etc/fstab or as an auto mount via systemd. I perferred the systemd-Version, so it is only mounted automatically only when really needed.

I’m not using systemd, so I may need to keep using my current setup for time being. Thanks for answering. I’d have to research runit scripts for that part more

@bedwardly-down but you can use the fstab option.

It is actually very simple to setup: just install davfs2 and configure it to store your credentials (you will easily find some more info on the web about that if needed) and add a entry in /etc/fstab so that your Nextcloud is mounted automatically into your file system on startup. Something like:

https://your.nextcloud.instance/remote.php/webdav/ /mnt/nextcloud davfs defaults,uid=YourUser,gid=YourUser,_netdev,auto 0 0

1 Like

Definitely know about the fstab. Just runit init does things a bit differently and I’d like to see how much control I have versus my current implementation too. Using the main CLI client does just that only when a specific user is logged in; if I ever make an extra account, this might be useful, too.

1 Like

Linux really is great for people that are up to figuring out crazy ways to get things done. Ha. :smiley:

Hello

I have been experiencing the same for quite some time. But I have observed that when using nextcloud dav sync, if I change the visible notes during the sync operation, it is not slow anymore, sync completes as the same amount of time as mobile. It seems like there is an event for sync to take the next action and changing the visible note triggering the event.

I have tried searching the issues but could not find a relevant one. I will open a new issue and try to explain it there. I will post the link here.

Hmm ok that shouldn't happen as the note doesn't need to render again in this case. Could you open an issue on GitHub to track this?

Opened #3004

1 Like

Getting the fstab davfs automount setup was a bit more challenging than I expected but once finished and working, I’m super happy about it.

Most resources require systemd to set it to mount at boot, but i found a workaround. Setting everything as needed in /etc/fstab, adding my user to the network group so it can only be mounted when connected to the internet, and mounting only when I’m logged in. No other user unless I explicitly add them, has the ability to mount.

Thanks for the tip, @kara

Hello,

To implement the workaround suggested by kara I used this tutorial:
https://blog.sleeplessbeastie.eu/2017/09/04/how-to-mount-webdav-share/

and it worked OK for me. Remember to reboot after applying changes to your user groups (if applicable). Permissions AFAIK require a reboot to be applied. The drive is not mounted automatically, so remember to mount it or adjust the /etc/fstab entry accordingly.

Performance seems OK. Also: I tested what happens when you’re offline and try to sync: Joplin handles this gracefully.

As for stability and data consistency I have some observations but this isn’t the place to share them. I’ll post this in my thread about issues with data consistency.

Good job @kara !

Yes and no. It requires a logout and login so that the groups for the user are refreshed, but there's a workaround:

$ newgrp GROUPNAME
1 Like

Don’t most of you have the Nextcloud client installed? Just point Joplin to the folder where it’s MD files are stored on the local disk. Instant sync.

1 Like

Thanks for that. I had all kind of weird synchronization issues with the desktop client (Win 10) but none with Android and this solved everything.

@Fmstrat could you elaborate a bit on this solution please?

Hello,

after a long time of not updating Joplin I finally upgraded. I went from 1.0.197 to 1.7.11 (the latest at the moment of writing).

I created a fresh backend location and performed a fresh initial sync with E2EE across 3 devices (android + 2x ubuntu). This time I switched from the dav2fs workaround to using webdav again because I had issues with putting my machines to sleep mode (the sync would prevent the OS from sleeping, to protect the consistency of the transfer, I assume).

Now on webdav I see that sadly the issue of super-slow transfers is still there. I loaded all my data from my JEX backup and its gonna take a "while" for this to sync. Days maybe.

:frowning:

EDIT

I added an issue:

All details I could think of are there.

This is not a WebDAV sync issue, but has everything to do with Nextcloud's flawed design.
Every single WebDAV request will trigger at least one database query, which will potentially go over millions of records.

I have experienced major performance issues when syncing with Nextcloud. This is just mind-boggling. I'm using Apache with event MPM, php-fpm (7.4) + OpCache, MySQL8 (via socket), Redis (via socket) (memcache.locking), APCu (memcache.local). The entire MySQL database fits into memory. All Nextcloud php files are compiled and cached in the OpCache.

For 910 items the sync with Nextcloud took about 1.5 hours. Running the same with a plain WebDAV implementation (without the Nextcluod BS), it took 17 minutes.

The issue

I closed the issue. I hope others will find it and draw their conclusions from this little conversation we had. I'm not gonna argue with tests performed by someone who manages both the client and server sides of the system.

@tessus thank you for the help! :blush: Please consider adding a note to the FAQ describing this issue.

https://joplinapp.org/faq/

I see that there are things about slow sync and Nextcloud, but those seem to be outdated.

The next steps :eyes:

This leaves me with the following options:

a) re-introduce the dav2fs workaround (no chance)
b) use the Nextcloud sync client (tried that, had what seemed to be race conditions, I also suffered from data getting reverted [Sometimes latest changes get reverted (during sync?) - #9 by tangodelta])
c) just use this slow as molasses WebDAV solution
d) try the S3 backend

I don't have the resources to go with d), so I'm gonna go with c) and when things stabilize I'll try d). Seeing how Nextcloud caused issues for me in the past, I'll try to migrate away from it.

I fixed this for me by using Apache httpd with the WebDAV module.

Maybe others have found a way to make Nextcloud faster, but I've been in performance for many years and I couldn't come up with anything I could have done better on my side. IMO the Nextcloud code needs a revamp.