Android sync not working (WebDAV target and 150k items)

[nonobio]

Operating system

Android 10

Smartphone

Motorola Moto G7 Plus
CPU Octa-core (4x1.8 GHz Kryo 260 Gold & 4x1.6 GHz Kryo 260 Silver)
Chipset Qualcomm SDM636 Snapdragon 636 (14 nm)
GPU Adreno 509
RAM 4GB

Joplin version

3.0.3 (pre-release APK)

Desktop version info

Joplin 3.0.6 (prod, win32)

Client ID: 773c93d7dee04e1289cb896f235df6a3
Sync Version: 3
Profile Version: 47
Keychain Supported: Yes

Revision: 18b9f5c

Backup: 1.4.1
Bidirectional Links: 0.1.2
Favorites: 1.3.2
Menu items, Shortcuts, Toolbar icons: 1.1.0
Note list and sidebar toggle buttons: 1.0.3
Note overview: 1.7.1
Note Tabs: 1.4.0
Outline: 1.5.13
Quick Links: 1.3.2
Rich Markdown: 0.15.0
Search & Replace: 2.2.0

Sync state on my first and main PC:
Attachments
Not downloaded: 0
Downloading: 0
Downloaded and decrypted: 2785
Downloaded and encrypted: 0
Created locally: 93621
Error: 0
Sync status (synced items / total items)
Note: 16467/16467
Folder: 210/210
Resource: 96406/96406
Tag: 4047/4047
NoteTag: 31053/31053
Revision: 1166/1166
Total: 149349/149349

Conflicted: 0
To delete: 0

Sync target

WebDAV on NAS Synology DS118 with Synology WebDAV Server

What issue do you have?

After synchronizing my first two devices (Windows PC), which was complicated but whose biggest slowness problems were resolved by version 3.0.6 of Joplin, I tried to synchronize my Android smartphone with version 3.0.3 from Joplin Android.

I indicated the WebDAV address accessible from outside my Synology WebDAV Server: https://*****.synology.me:5006/joplin/

The synchronization test indicates that the configuration is correct and synchronization starts.
I took care to leave Joplin open in the foreground and to leave the screen on permanently because I understood that there was no sync in the background (I set the brightness to 0).

The interface does not respond or very very slowly and several hours later the synchronization status indicates only 10 items.
I tried to use the local address http://192.168.1.26:5005/joplin/ (which I already use on my two PCs) but the problem is the same.

There may be a specific problem but I'm wondering if my large number of items (150k) is simply too large for Android sync?
Although Joplin Windows is now usable for me and sync works, it is slow:

@nonobio

Joplin 3.0.6 Laptop PC CPU Intel Core i5 4310U RAM 8 GB:
little sync (on changes): 17s 23s 28s 29s
big sync (sync interval or manual push on the button): 77s 91s 90s 63s

Joplin 3.0.6 Desktop PC CPU AMD Ryzen 5 5600G with Radeon Graphics RAM 16 GB:
little sync (on changes): 6s 9s 6s 6s
big sync (sync interval or manual push on the button): 53s 69s 47s 41s

So, sync is rather slow, but that doesn't stop me from using Joplin, thanks again to the 3.0.6 version and because I use it mainly on a single PC, I can space out the big sync intervals, or even deactivate it completely and do manual syncs.
If necessary, I can also completely deactivate all syncs (including small ones), by temporarily setting the target to None.

From what I've been told and what I've read, using Joplin Server would improve things, but I'm a little apprehensive about installing it because my NAS DS118 is not officially compatible, and in any case, even on a Windows PC, it seems complicated to install (Docker). In any case more complicated on the Synology WebDAV server.

So, maybe it's "ok" for Joplin Desktop Windows, but it's too much for my Joplin Android...

I also did tests listing my Joplin directory by connecting to WebDAV:
On my Windows PC with WinSCP: it takes 2 minutes on my laptop and 1min45 on my desktop PC to display the files!
On my Android mobile with FolderSync app: it takes 1min30. But with X-Plore file explorer it doesn't work: the application freezes.

If it is due to my too many items and there is nothing to do, I'll probably drop Joplin Android because I think I can live without it.
But if you think Joplin Android should work even with my large amount of data and it's possible to resolve the issue, I'd like to try to solve it because it would still be handy to have my notes on my smartphone, if only to consult them.
If it's due to the use of WebDAV and you think Joplin Server would solve the problem, I could consider installing it, although as I said, it seems complicated to me...

What do you think?

Thanks

Screenshots

After a few seconds when sync seems to be in progress (the two arrows rotate next to the Cancel button), the arrows no longer rotate and the application is frozen (sometimes it unfreezes on its own and sometimes it closes completely):

This is what the synchronization status indicates:

Log

20240502_mobile-log.7z (14.8 KB)

[nonobio]

On this other topic: Fails to sync

@Herve5 proposed this:

@Herve5

It may be wrong or too simple, but please check the status of locks files, for instance here

(it's about checking and deleting eventual lock files in .lock folder)

I checked: my .lock folder is empty.

I tried the Android sync again after deleting the data from the Android application to start from scratch: same problem, the arrows rotate and the message "Cancel" is displayed, which indicates that the sync is in progress, then the arrows stop running and the application is frozen.

Do you have an idea of ​​the problem @laurent and @personalizedrefriger?

I think I can live without the Android application but if there is just a small problem it would still be a plus to have it

[personalizedrefriger]

Sync issues with the mobile app

@nonobio

it takes 2 minutes on my laptop and 1min45 on my desktop PC to display the files!

WebDAV sync seems to involve listing all files in the sync directory. If this is slow, it could be leading to slow sync on Android and desktop. However, because the desktop sync completes successfully, I suspect that this isn't the issue.

Because this doesn't happen on desktop, one possibility is that the issue is related to how React Native's fetch is interacting with the WebDAV server. In the past, we've had to add workarounds (example) to support different WebDAV implementations.

Additional details

Based on the logs, it seems that Joplin is throwing a lockExpired error here because hasActiveLock returns false. Based on the implementation of hasActiveLock, it seems that this could also happen when no locks exist (and thus throw a lockExpired error). Because this doesn't happen on desktop/with other sync targets, I suspect that this isn't the issue.

Additional information that might help

• Does creating a new profile with 20-30 small notes, then syncing to a different subdirectory of the WebDAV server work? Does it work if a larger note is attached?
  • This could help determine whether the issue is with how React Native is interacting with that particular WebDAV implementation or with the mobile app itself.

I'm linking to two related GitHub issues:

• Disable sync lock support · Issue #10407 · laurent22/joplin · GitHub
• Support running synchronizer tests from "Synchronization Status" · Issue #10430 · laurent22/joplin · GitHub

Alternatives

@nonobio

If it's due to the use of WebDAV and you think Joplin Server would solve the problem, I could consider installing it, although as I said, it seems complicated to me...

If the issue is related to the number of files on the sync target, Joplin Server could help. Testing with Joplin Server would also help determine whether the issue is with 1) how the mobile app is interacting with WebDAV or 2) the data stored on the sync target.

[nonobio]

@personalizedrefrigerator

Additional information that might help

• Does creating a new profile with 20-30 small notes, then syncing to a different subdirectory of the WebDAV server work?

I already had a second profile (Joplin2) that I used for cleaning and testing when I moved from Evernote to Joplin. I no longer had any notes or notebooks in this profile and so I created a notebook with 25 small notes, created a new shared folder on my NAS and started the sync from my PC.

The first sync took a long time due to numerous "Deleted Items Objects":

but finally ended:

Joplin2 sync status

Attachments
Not downloaded: 0
Download in progress: 0
Downloaded and decrypted: 0
Downloaded and encrypted: 0
Created locally: 40
Error: 0
Synchronization status (sync objects / total)
Rating: 0/25
Folder: 0/1
Resources: 0/40
Tag: 0/1448
NoteTag: 0/19648
Review: 0/81
Total: 0/21243

Conflicts: 0
To delete: 20651
Notebooks
Notebook 1: 25 notes

If I understood correctly, these are traces of my old notes which were present during my previous tests which I nevertheless deleted.

To do the test in good conditions I recreated a new blank profile (Joplin3).
In passing, I would like to take this opportunity to ask you if there is a possibility of completely emptying a profile of its old “ghost” notes and/or of properly deleting a profile?

Back to my test, I recreated my joplintest shared folder on my NAS, here are its characteristics if it can be useful (they are the same as my main joplin sync folder):

I created a single notebook and 25 small notes on my Joplin3 profile (the set of these notes is 40 kb).
I configured the sync and launched it. It ended very quickly:

Joplin3 sync status

Attachments
Not downloaded: 0
Download in progress: 0
Downloaded and decrypted: 0
Downloaded and encrypted: 0
Created locally: 0
Error: 0
Synchronization status (sync objects / total)
Rating: 25/25
Folder: 1/1
Resources: 0/0
Tag: 0/0
RatingTag: 0/0
Review: 0/0
Total: 26/26

Conflicts: 0
To delete: 0
Notebooks
Notebook 1: 25 notes

I configure the sync on my Joplin Android with https ddns adress (I took care to delete the data from the Android application before) and launch it:
It works and finishes in 9s:

Android sync status

Attachments
Not downloaded: 0
Download in progress: 0
Downloaded and decrypted: 0
Downloaded and encrypted: C
Created locally: 3
Error: 0
Synchronization status (objects
sync. / total)
Rating: 30/30
Folder: 2/2
Resources: 3/3
RatingTag: 0/0
Review: 0/0
Overall: 35/35
Conflicts: 0
To delete: 0

@personalizedrefrigerator

Does it work if a larger note is attached?

I add a big note to my Joplin3 PC (the jex of this note is 1482 kb).
PC sync ends in 4s:

I launch the sync on Joplin Android, it works and it finishes in 29s:

In summary, testing the new profile with 25 small notes then with one more big note works.

@personalizedrefriger

If the issue is related to the number of files on the sync target, Joplin Server could help. Testing with Joplin Server would also help determine whether the issue is with 1) how the mobile app is interacting with WebDAV or 2) the data stored on the sync target.

Ok, the problem is that installing Joplin Server seems really complicated to me. Installation and configuration of the Synology WebDAV server was done in just a few clicks.

I do have a Synology NAS but the model I have (DS118: CPU 4-core 1.4 GHz Realtek RTD1296 and RAM 1 GB DDR4 non-ECC) is not officially compatible with Docker (or "Container Manager"), which is a priori necessary (by the way, I have never installed or used Docker). There is an unofficial solution here to make it compatible.

Otherwise I thought about my Desktop PC CPU AMD Ryzen 5 5600G with Radeon Graphics RAM 16 GB on which one of my 2 Joplin clients runs which is running 24/24. It is apparently possible to install Docker on Windows but again it seems complicated to me.

In both cases, it seems complicated and reserved for advanced users. I also read that Docker can be memory intensive...

In short, I don't really know if it's a good idea to start installing Joplin Server when I finally have my 2 Joplins PCs which work quite well with WebDAV.
Having my notes on my Android smartphone would be a plus but I don't know if it's worth it, and I can live without it.

Thanks