Extreme slow sync of "resources"

Support
1

Few months ago I have installed first instance of Joplin on Win7, syncing with webdav.
About 2GB of notes, above 14 thousands of files (almost images) in “resource” directory.
Currently using 1.0.161.

Then installed on Android, sync with Android client taken long time.
I was warned by this forum and left it to run whole night.
But the later usage and incremental syncs are reasonably quick. Version 1.0.281

After few months I have installed Linux version 1.0.165.

I have expected long running first synchronization. Surprisingly it finished in about 1 hour.
But the “resource” folder where all images should be saved is almost empty. After first successfully finished sync there was just about 60 files.

Left the client open for the following 20 hours, running sync every 5 minutes.
Still the folder “resources” has only 280 files (increasing just about 2-3 images by every sync).

Is it expected? Could I copy files from Windows PC to my linux to skip the file transfer? This way it will take weeks to copy all 14 thousands of images.

Is it bug? Will it help to delete everything and try some older version? Perhaps the same as on Win?

The sync on Linux client finishes every time with success.

Sync status (synced items / total items)
Note: 3783/3783
Folder: 21/21
Resource: 14329/14329
Tag: 100/100
NoteTag: 5528/5528
MasterKey: 0/0
Revision: 367/367
Total: 24128/24128
 
Conflicted: 0
To delete: 0

But definitely there are no 14329 files in “resources” as reported.

Many thanks for any advice.
//Rado1

2

The resources are now downloaded separately, independently of the sync process. I’m not sure why it’s so slow in your case though, although I suspect it’s related to the strange Linux bug that makes the app sleep when it’s not interacted with.

What does the log say? There should be “ResourceFetcher:” at regular interval. Can you find these messages?

3

Here is the excerpt from ~/.config/joplin-desktop/log.txt:

2019-08-16 19:58:07: "RevisionService::maintenance: Starting..."
2019-08-16 19:58:07: "RevisionService::maintenance: Service is enabled"
2019-08-16 19:58:07: "RevisionService::collectRevisions: Created revisions for 0 notes"
2019-08-16 19:58:07: "RevisionService::maintenance: Done in 105ms"
2019-08-16 19:58:44: "Running background sync on timer..."
2019-08-16 19:58:44: "Scheduling sync operation..."
2019-08-16 19:58:44: "Preparing scheduled sync"
2019-08-16 19:58:44: "Starting scheduled sync"
2019-08-16 19:58:46: "TaskQueue.stop: syncDownload: waiting for tasks to complete: 0"
2019-08-16 19:58:46: "TaskQueue.stop: syncDownload: Done, waited for 0"
2019-08-16 19:58:57: "Operations completed: "
2019-08-16 19:58:57: "fetchingTotal: -"
2019-08-16 19:58:57: "Total folders: 21"
2019-08-16 19:58:57: "Total notes: 3783"
2019-08-16 19:58:57: "Total resources: 14329"
2019-08-16 20:03:57: "Running background sync on timer..."
2019-08-16 20:03:57: "Scheduling sync operation..."
2019-08-16 20:03:57: "Preparing scheduled sync"
2019-08-16 20:03:57: "Starting scheduled sync"
2019-08-16 20:03:59: "TaskQueue.stop: syncDownload: waiting for tasks to complete: 0"
2019-08-16 20:03:59: "TaskQueue.stop: syncDownload: Done, waited for 0"
2019-08-16 20:04:11: "Operations completed: "
2019-08-16 20:04:11: "fetchingTotal: -"
2019-08-16 20:04:11: "Total folders: 21"
2019-08-16 20:04:11: "Total notes: 3783"
2019-08-16 20:04:11: "Total resources: 14329"
2019-08-16 20:06:41: "ResourceFetcher: Auto-add resources: Mode: always"
2019-08-16 20:06:41: "ResourceFetcher: Auto-added resources: 9"
2019-08-16 20:08:07: "Garbage collecting alarms..."
2019-08-16 20:08:07: "RevisionService::maintenance: Starting..."
2019-08-16 20:08:07: "RevisionService::maintenance: Service is enabled"
2019-08-16 20:08:07: "RevisionService::collectRevisions: Created revisions for 0 notes"
2019-08-16 20:08:07: "RevisionService::maintenance: Done in 110ms"

The line Auto-added resources: 9 is weird as there was zero updates of notes this time.

4

Deleted all data, installed version 1.0.161.
Sync started from the scratch.

By sync status I have (currently) fetched over 2700 resources but there are only 60 files in “resource” folder.
Will see tomorrow morning…

5

I see - among my processes - several processes writing to /tmp/.mount_JoplinSS3fAT.
It’s a fuse mounted filesystem but with very weird behavior.

Even as root I’m not able to see owner, group and permissions of the mounted folder. ls -l /tmp/ shown just question marks for them.
strace for the process handling this fuse mount is showing a lot of “Resource temporarily unavailable”.
Unfortunatelly, I don’t see what file-descriptor of socket is demanded.

[pid 16983] <... recvmsg resumed> {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 17582] epoll_wait(12,  <unfinished ...>
[pid 16983] read(18,  <unfinished ...>
[pid 17582] <... epoll_wait resumed> [], 32, 0) = 0
[pid 16983] <... read resumed> "!", 2)  = 1
[pid 17582] epoll_wait(12,  <unfinished ...>
[pid 16983] sendto(85, "x\0\0\0\20\0\1\0\0\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 120, MSG_NOSIGNAL, NULL, 0) = 120
[pid 16983] write(16, "\0", 1 <unfinished ...>
[pid 17582] <... epoll_wait resumed> [{EPOLLIN, {u32=15, u64=15}}], 32, 52386) = 1
[pid 16983] <... write resumed> )       = 1
[pid 17582] read(15, "\0", 1)           = 1
[pid 16983] recvmsg(27,  <unfinished ...>
[pid 17582] gettid( <unfinished ...>
[pid 16983] <... recvmsg resumed> {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 17582] <... gettid resumed> )      = 17582
[pid 16983] recvmsg(37,  <unfinished ...>
[pid 17582] sendto(65, "\310\0\0\0\20\0\1\0\0\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 200, MSG_NOSIGNAL, NULL, 0 <unfinished ...>
[pid 16983] <... recvmsg resumed> {msg_namelen=0}, 0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 17582] <... sendto resumed> )      = 200

What does it mean that nor my UID nor root has privileges to work with this fuse mounted filesystem?
Could not it be the reason of extremely slow ability to write data?

What’s the purpose of this fuse filesystem at all? Is it a mount accessing webdav?

6

Currently I have sync configuration:

webdav
\--Win client
\--Android client
\--Linux client

May I disconnect Win client from webdav, configure filesystem sync, use copy of this folder to initiate filesystem sync on Linux client and then reconfigure both Win and Lin clients to use webdav again?

Win client -> filesystem -> copy -> filesystem -> Linux client

Thanks.

7

It is used by AppImage to mount the application. Joplin is distributed as an AppImage on Linux. I have never used AppImages, so please don't bite my head off, if I'm not explaining this right.
The AppImage uses FUSE to mount its own image which holds the Joplin application directory structure. Basically a loopback mount of an image.

8

Many thanks.
Just found out it minute ago.
As ordinary user I can access this filesystem and I see the unpacked app and libs and s.o

Therefore the resources are fetched directly to ~/.config/joplin-desktop/resources.
But don’t understand why so slowly.
Is the initial sync through filesystem and then switching to webdav viable method?
I think (to be honest, don’t remember) I have used it for initial data transfer from Windows to webdav…

Thanks.

9

I do not know why it is that slow, but maybe Laurent can figure it out. He already replied, so I guess its on his radar.

The filesystem workaround has been used by several people on this forum. AFAIK it's a valid solution to speed up the initial sync time.

10

Syncing on Linux client 1.0.165 via filesystem.
80 resources per minute. Still very slow but could finish during night…

11

Initial sync via filesystem finished in several hours. Would expect minutes.
Hopefully will not repeat it in following months…

My current issue is resolved but generally there is something strange…

Many thanks for your support.

12

Oh, after switching to webdav, it tries to update all notes (probably it consider local copy newer than on webdav).

By speed about 50 items per minute. Over 24 thousands of items…

13

If not a bug, then it’s certainly an issue…

14

Hello,

I’m having the same issue, as mentioned here: Sometimes latest changes get reverted (during sync?)

I’ve switched from the file-system sync target to WebDav and the sync appears to be working, but painfully slow.

 16 2020-03-28 21:41:13: "RevisionService::maintenance: Starting..."
 15 2020-03-28 21:41:13: "RevisionService::maintenance: Service is enabled"
 14 2020-03-28 21:41:13: "RevisionService::collectRevisions: Created revisions for 0 notes"
 13 2020-03-28 21:41:13: "RevisionService::maintenance: Done in 103ms"
 12 2020-03-28 21:41:15: "Scheduling sync operation..."
 11 2020-03-28 21:41:15: "Preparing scheduled sync"
 10 2020-03-28 21:41:15: "Starting scheduled sync"
  9 2020-03-28 21:42:13: "Running background sync on timer..."
  8 2020-03-28 21:42:13: "Scheduling sync operation..."
  7 2020-03-28 21:42:13: "Preparing scheduled sync"
  6 2020-03-28 21:42:13: "Starting scheduled sync"
  5 2020-03-28 21:42:13: "Synchronisation is already in progress. State: in_progress"
  4 2020-03-28 21:47:13: "Running background sync on timer..."
  3 2020-03-28 21:47:13: "Scheduling sync operation..."
  2 2020-03-28 21:47:13: "Preparing scheduled sync"
  1 2020-03-28 21:47:13: "Starting scheduled sync"
193 2020-03-28 21:47:13: "Synchronisation is already in progress. State: in_progress"