Weird sync behaviour re. number of fetched items

Update here with Cloudamo -

I have given up and decided that there is either something about Cloudamo that is incompatible with Joplin, or there is a bug in the Joplin Windows application.

I reset everything and pared down my Joplin import (on a fresh install) to 1,000 notes. I set this to encrypt and sync to Webamo WebDav and it did perform the sync. However, after it finished, it resumed this "Fetching items" that just continues on forever. Joplin seems to be in some kind of failure loop. Log just shows the data below repeating indefinitely.

2020-12-18 14:06:08: "TaskQueue.stop: syncDownload: Done, waited for 0"
2020-12-18 14:06:14: "BasicDelta: Report: {"timestamp":1608306731000,"older":199,"newer":50,"equal":1}"
2020-12-18 14:06:53: "BasicDelta: Report: {"timestamp":1608306871000,"older":249,"newer":50,"equal":1}"
2020-12-18 14:06:54: "Saving settings..."
2020-12-18 14:06:54: "Settings have been saved."
2020-12-18 14:07:17: "BasicDelta: Report: {"timestamp":1608307010000,"older":299,"newer":50,"equal":1}"
2020-12-18 14:07:17: "Saving settings..."
2020-12-18 14:07:17: "Settings have been saved."
2020-12-18 14:07:22: "RevisionService::maintenance: Starting..."
2020-12-18 14:07:22: "RevisionService::maintenance: Service is enabled"
2020-12-18 14:07:23: "Saving settings..."
2020-12-18 14:07:24: "Settings have been saved."
2020-12-18 14:07:24: "RevisionService::collectRevisions: Created revisions for 1 notes"
2020-12-18 14:07:24: "RevisionService::maintenance: Done in 1967ms"
2020-12-18 14:07:35: "BasicDelta: Report: {"timestamp":1608307151000,"older":349,"newer":50,"equal":1}"
2020-12-18 14:07:35: "Saving settings..."
2020-12-18 14:07:36: "Settings have been saved."
2020-12-18 14:07:58: "BasicDelta: Report: {"timestamp":1608307311000,"older":399,"newer":50,"equal":1}"
2020-12-18 14:07:58: "Saving settings..."
2020-12-18 14:07:58: "Settings have been saved."

There's nothing obviously wrong in log. It's processing items 50 by 50 and that's working according to your log. Maybe just let it sync to completion on all devices?

Unfortunately it never finishes - it either starts over or Cloudamo locks the account thinking that something nefarious is going on (I assume). One other user on Reddit is reporting a similar issue with Cloudamo, so it may be something going on there.

Cloudamo (apparently) does not allow subfolders for WebDav, so the only way to get it to work is to let it use the root folder - not sure if that is causing an issue.

What I do not understand is why the desktop app needs to fetch anything after it synced all files to the cloud. No other apps are syncing yet with the folder.

1 Like

@laurent I have blown out everything and done some more testing. This is certainly looking like a bug in the Windows sync module:

  • Switched from Cloudamo to OneNote (eliminates Cloudamo as a problem)
  • Blew out Joplin on all devices, deleted all data
  • Using a RAW export of Joplin Windows that contains about 1,000 notes imported from Evernote
    [EDIT - Forgot to mention I am using the latest beta of the Windows app 1.5.7]
  • No MD files are over 2MB, no attachments over 7MB
  • Initiated new OneDrive sync from Windows app, encryption turned off
  • All items sync but app starts doing the never ending Fetch thing.
  • iPad and Android sync to OneDrive completely fine with no errors and notes created on one sync to/from the other, but the notes never sync to the Windows app
  • If I didn't need the Windows app I would not realize anything was wrong

Log file shows an error for a few MD files, but I am note sure what to make of them. These will be repeated forever:

2020-12-19 11:36:24: "FetchError: On file 424bf54e0c564bd2b1efc0ebbb69201f.md: request to https://public.bl.files.1drv.com/y4m-zqqVtwl2Jv-1JcySyRSPIwhIt2i5MzcgbM2zzXBHRJErvn4JrbwRazF9TpbR4a_aPKQDpmk_txPgunnCRYTFCA3h5XMu4unAN_L1eW3pzCIvr9s2LUee4ipWT9kJMIvt9EFTRxZU_zNcOUBAIO_VKoDrmJUrQZnDyslFsFwwhvuTbTodWCEJXzG3l-goVKa4wxQ-6mBMfJgr_NywRTlnC7ida_1j5BNRade0WWAvSo/424bf54e0c564bd2b1efc0ebbb69201f.md failed, reason: read ECONNRESET
Code: ECONNRESET
FetchError: request to https://public.bl.files.1drv.com/y4m-zqqVtwl2Jv-1JcySyRSPIwhIt2i5MzcgbM2zzXBHRJErvn4JrbwRazF9TpbR4a_aPKQDpmk_txPgunnCRYTFCA3h5XMu4unAN_L1eW3pzCIvr9s2LUee4ipWT9kJMIvt9EFTRxZU_zNcOUBAIO_VKoDrmJUrQZnDyslFsFwwhvuTbTodWCEJXzG3l-goVKa4wxQ-6mBMfJgr_NywRTlnC7ida_1j5BNRade0WWAvSo/424bf54e0c564bd2b1efc0ebbb69201f.md failed, reason: read ECONNRESET
at ClientRequest. (C:\Users\USER\AppData\Local\Programs\Joplin\resources\app.asar\node_modules@joplin\lib\node_modules\node-fetch\index.js:133:11)
at ClientRequest.emit (events.js:310:20)
at ClientRequest.EventEmitter.emit (domain.js:482:12)
at TLSSocket.socketErrorListener (_http_client.js:426:9)
at TLSSocket.emit (events.js:310:20)
at TLSSocket.EventEmitter.emit (domain.js:482:12)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21)"

1 Like

ECONNRESET errors are supposed to be temporary so the requests are always repeated, not necessarily immediately but on the next sync. So if for some reason it always fails, it will always repeat the request.

Is there anything special to this file 424bf54e0c564bd2b1efc0ebbb69201f.md ? Can you download it directly from OneDrive?

Thanks for the reply. Nothing special about either of those, I can download from OneNote. They are really short notes - about 500 characters.

I've also installed the Windows app on another Windows machine I have to see if there is anything machine specific. This new machine is a completely fresh install - only been a couple of hours but it appears to be doing the same thing - log files shows many of the ECONNRESET errors and in looking at the files, nothing special. This client also never syncs with new files created on the iPad, but the iPad is working perfectly fine and does pick up new files created with the Windows clients.

I am going to probably blow everything out again and just try with a few notes created in Joplin (no Evernote imports) to see there is something wrong in the Evernote files, unless there is anything else you suggest I try that might help with troubleshooting.

It would be surprising if it was related to Evernote as this is a network connection issue. It wouldn't be a firewall or proxy issue by any chance?

That is a good point. I will run some tests later by switching one of the Windows machines over to a cell hotspot (to eliminate my router) and I'll see if there are any errant firewall rules. I did get the notice on both machines during install to allow traffic through and I selected yes. I'll report back what I find.

This all appears to have been a OneDrive problem yesterday. I let things sit overnight and am now getting no errors and sync is completing as expected.

I think I was assuming this second issue was the same as my first issue, with Cloudamo. Apparently Cloudamo sees Joplin syncing many notes and thinks something is going on and it temporarily locks out that connection. It does eventually come back, but further syncing will cause it to lock again. Due to this, I believe that Cloudamo is not suitable for Joplin WebDAV syncing unless one is starting from scratch with a minimal number of notes to sync. This is the service that NextCloud is currently pushing, so I expect there will be more people who try this with a large number of Evernote notes as I did.

In any event, OneDrive is working really well for me and I will continue to use that.

2 Likes

That's pretty bad actually. If they do it on purpose, that's the wrong approach as it would randomly break any client that sync many files. Much better to do like Dropbox or OneDrive and throttle the connection as at least it means sync can be completed.

And if it's a bug, it looks bad because downloading and uploading files is the core feature of Nextcloud (or should be) and once again they don't seem to be getting this right.

FYI, I submitted a ticket with Cloudamo to ask them about these issues and I will report back with their answers if I get them. If they could resolve this, it would be great as overall it is a promising service. I paid $17 for a year's worth of 50GB storage, which is pretty reasonable.

For my situation, I had to switch over to DropBox as I got caught by the dreaded "4MB" note error on OneDrive and couldn't figure out how to resolve it. I am closely following this pull request as I believe OneDrive is a more economical and better performing option for Joplin syncing for many people.

I think I have the same issue. Coming from Evernote, imported 6k+ notes, encrypted to Onedrive. Long uploading but went OK. Now the desktop and Android version are always in fetching mode.
I.e. it just fetched 8k+ items on PC. Finished. I updated a couple of notes. Sync OK. Opened the Android app to try to finish the sync that never completed till now after a week and the PC version started a full fetching again!
BTW the desktop version overloads the CPU the whole time while fetching...

1 Like

I have the same issue. I am using the app both on phone and PC, haven't imported anything from other apps. Everything was written mostly from my phone. I am using OneDrive btw. On the desktop app it keeps synching indefinitely. At some point it reached around 65k fetched and I had to stop it but the changes made on the phone were not updated on the desktop app.

Note: I have around 150 notes written by hand, some have links, tables, embedded photos. on the phone it usually reaches close to 2k fetched but it is nowhere near 65k on the desktop.

Do you have any idea what might be the cause?

Joplin 1.5.14 on Win8 synching with OneDrive.

Roughly 30 notes only, 5- attachments.

Synching says:
Created 5 remote items.
Fetched: 6000 and counting.

Fetched items is ever increased by 12 but seems to run without end.

I am currently in the process of moving my notes from OneNote to Joplin. But such thing make me a little worried regarding reliability.

1 Like

Joplin 1.6.6 on Android syncing with OneDrive.
Joplin 1.6.6 on Windows syncing with OneDrive

I have only 11 text notes, one with a small .json file attached.
The Windows sync seems to work and the log file shows 61 items successfully synced.

The Android app is not getting updated when syncing, and like others mentioned shows a fetched item count that goes up and up. Currently at 6,000 items fetched and it keeps going.

In the meantime, I keep checking and the content of the note is not updated to match the desktop note.

I got exactly the same issue, it keeps fetching items. I got Windows 10, iPhone iOS 13, Ipad iOS13. I am using Joplin 1.6.6.

And what's really strange with every fetch, number of fetched Items is increasing by around 30-50.

I'm experiencing the same issue. I've just cancelled the fetching process when it reached around 5000 items and now it's starting to fetch again. I got Windows 10 and OneDrive.

OneDrive sync has issues. Have you tried Dropbox or webdav?

Have same problem with OneDrive. Everything sync-ed, click Sync, it starts fetching objects again. Both Windows and Android fetch objects every time.
Now I changed to Dropbox (desktop client not installed) and both Windows and Android will not fetch anything if I don't make any change.

Maybe this issue:

1 Like