Hi. I don't think this is a bug because I don't notice anything really wrong with the way that Joplin works. However, over the past few days I've noticed some odd behaviour every time I sync my phone and tablet. I've attached a screenshot showing the number of fetched items. Which seems high. And this seems to be happening once a day over the past week or so (I didn't make a note but it felt like it was that often). And this is after I've been using it for months, so why is it, all of sudden, fetching so many items?
1 Like
Item is not the only full note. If you are saving web pages as notes, every embedded image is a separate item. Is this not your case?
Sure, I get that. But I have more than 5000 items being fetched every couple of days (did it again this morning). And I also noticed that the Welcome folder is being recreated every time as well. So my situation is this:
- Every couple of days my sync process includes more than 5000 fetched items.
- I'm not saving that much content to Joplin, even if you were to include all the webpage items.
- It keeps recreating the Welcome folder, which I keep deleting from all my devices.
Any other thoughts?
I've just noticed that the Attachment download behaviour in Settings keeps switching from Manual (my preference) to Always. First of all, that doesn't seem right. And in case you're wondering, I do save the settings every time I switch it back to Manual.
Could this be a reason for why I regularly have to fetch 5000+ items?
1 Like
I'm also trying to understand something similar. I have started afresh and imported my Evernote notes, set up a new WebDav sync target and let the sync run for several days. The content now all seems to be on the WebDav, but I get never-ending "Fetched items" increases. The log shows lots of activity as if note content needs to be downloaded, but it should not - nothing has changed on the server side:
2020-11-09 11:48:16: "BasicDelta: Report: {"timestamp":1604692714000,"older":4188,"newer":50,"equal":1}"
2020-11-09 11:48:16: "Sync: fetchingTotal: Fetching delta items from sync target"
2020-11-09 11:48:16: "get dabd41564400485992762caf88332e10.md"
The only other odd thing I can see is that in the sync status the synced items/total items count is:
Total: 37988/37965
i.e. there are more sync'd items than the total. Not sure if this is triggering something that should not be needed?
and similar
@msbentley were you able to resolve this? I am having a similar issue:
Switching from EN to Joplin, I imported about 15k notes (HTML) into Windows 10 Joplin desktop in batches of about 1.5k per time.
I am syncing to NextCloud WebDav, cloud instance with Cloudamo (I've paid for 50GB) and using encryption.
I got everything imported and synced to Cloudamo from the desktop client, and syncing finished. I then installed app on my iPad and did a complete synch and de-encrypt. I then installed app to my Android phone and did a complete sync and decrypt.
At this point everything seemed to be going well. This morning, the desktop app has started doing an endless "fetch" and notes created on the mobile devices never sync to the desktop. I don't understand what would need to be "fetched" on the desktop since everything was imported and originated there. I see no syncing errors.
Should I uninstall the desktop and try re-installing and then sync everything back?
Update here with Cloudamo -
I have given up and decided that there is either something about Cloudamo that is incompatible with Joplin, or there is a bug in the Joplin Windows application.
I reset everything and pared down my Joplin import (on a fresh install) to 1,000 notes. I set this to encrypt and sync to Webamo WebDav and it did perform the sync. However, after it finished, it resumed this "Fetching items" that just continues on forever. Joplin seems to be in some kind of failure loop. Log just shows the data below repeating indefinitely.
2020-12-18 14:06:08: "TaskQueue.stop: syncDownload: Done, waited for 0"
2020-12-18 14:06:14: "BasicDelta: Report: {"timestamp":1608306731000,"older":199,"newer":50,"equal":1}"
2020-12-18 14:06:53: "BasicDelta: Report: {"timestamp":1608306871000,"older":249,"newer":50,"equal":1}"
2020-12-18 14:06:54: "Saving settings..."
2020-12-18 14:06:54: "Settings have been saved."
2020-12-18 14:07:17: "BasicDelta: Report: {"timestamp":1608307010000,"older":299,"newer":50,"equal":1}"
2020-12-18 14:07:17: "Saving settings..."
2020-12-18 14:07:17: "Settings have been saved."
2020-12-18 14:07:22: "RevisionService::maintenance: Starting..."
2020-12-18 14:07:22: "RevisionService::maintenance: Service is enabled"
2020-12-18 14:07:23: "Saving settings..."
2020-12-18 14:07:24: "Settings have been saved."
2020-12-18 14:07:24: "RevisionService::collectRevisions: Created revisions for 1 notes"
2020-12-18 14:07:24: "RevisionService::maintenance: Done in 1967ms"
2020-12-18 14:07:35: "BasicDelta: Report: {"timestamp":1608307151000,"older":349,"newer":50,"equal":1}"
2020-12-18 14:07:35: "Saving settings..."
2020-12-18 14:07:36: "Settings have been saved."
2020-12-18 14:07:58: "BasicDelta: Report: {"timestamp":1608307311000,"older":399,"newer":50,"equal":1}"
2020-12-18 14:07:58: "Saving settings..."
2020-12-18 14:07:58: "Settings have been saved."
There's nothing obviously wrong in log. It's processing items 50 by 50 and that's working according to your log. Maybe just let it sync to completion on all devices?
Unfortunately it never finishes - it either starts over or Cloudamo locks the account thinking that something nefarious is going on (I assume). One other user on Reddit is reporting a similar issue with Cloudamo, so it may be something going on there.
Cloudamo (apparently) does not allow subfolders for WebDav, so the only way to get it to work is to let it use the root folder - not sure if that is causing an issue.
What I do not understand is why the desktop app needs to fetch anything after it synced all files to the cloud. No other apps are syncing yet with the folder.
1 Like
@laurent I have blown out everything and done some more testing. This is certainly looking like a bug in the Windows sync module:
- Switched from Cloudamo to OneNote (eliminates Cloudamo as a problem)
- Blew out Joplin on all devices, deleted all data
- Using a RAW export of Joplin Windows that contains about 1,000 notes imported from Evernote
[EDIT - Forgot to mention I am using the latest beta of the Windows app 1.5.7] - No MD files are over 2MB, no attachments over 7MB
- Initiated new OneDrive sync from Windows app, encryption turned off
- All items sync but app starts doing the never ending Fetch thing.
- iPad and Android sync to OneDrive completely fine with no errors and notes created on one sync to/from the other, but the notes never sync to the Windows app
- If I didn't need the Windows app I would not realize anything was wrong
Log file shows an error for a few MD files, but I am note sure what to make of them. These will be repeated forever:
2020-12-19 11:36:24: "FetchError: On file 424bf54e0c564bd2b1efc0ebbb69201f.md: request to https://public.bl.files.1drv.com/y4m-zqqVtwl2Jv-1JcySyRSPIwhIt2i5MzcgbM2zzXBHRJErvn4JrbwRazF9TpbR4a_aPKQDpmk_txPgunnCRYTFCA3h5XMu4unAN_L1eW3pzCIvr9s2LUee4ipWT9kJMIvt9EFTRxZU_zNcOUBAIO_VKoDrmJUrQZnDyslFsFwwhvuTbTodWCEJXzG3l-goVKa4wxQ-6mBMfJgr_NywRTlnC7ida_1j5BNRade0WWAvSo/424bf54e0c564bd2b1efc0ebbb69201f.md failed, reason: read ECONNRESET
Code: ECONNRESET
FetchError: request to https://public.bl.files.1drv.com/y4m-zqqVtwl2Jv-1JcySyRSPIwhIt2i5MzcgbM2zzXBHRJErvn4JrbwRazF9TpbR4a_aPKQDpmk_txPgunnCRYTFCA3h5XMu4unAN_L1eW3pzCIvr9s2LUee4ipWT9kJMIvt9EFTRxZU_zNcOUBAIO_VKoDrmJUrQZnDyslFsFwwhvuTbTodWCEJXzG3l-goVKa4wxQ-6mBMfJgr_NywRTlnC7ida_1j5BNRade0WWAvSo/424bf54e0c564bd2b1efc0ebbb69201f.md failed, reason: read ECONNRESET
at ClientRequest. (C:\Users\USER\AppData\Local\Programs\Joplin\resources\app.asar\node_modules@joplin\lib\node_modules\node-fetch\index.js:133:11)
at ClientRequest.emit (events.js:310:20)
at ClientRequest.EventEmitter.emit (domain.js:482:12)
at TLSSocket.socketErrorListener (_http_client.js:426:9)
at TLSSocket.emit (events.js:310:20)
at TLSSocket.EventEmitter.emit (domain.js:482:12)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21)"
1 Like
ECONNRESET errors are supposed to be temporary so the requests are always repeated, not necessarily immediately but on the next sync. So if for some reason it always fails, it will always repeat the request.
Is there anything special to this file 424bf54e0c564bd2b1efc0ebbb69201f.md ? Can you download it directly from OneDrive?
Thanks for the reply. Nothing special about either of those, I can download from OneNote. They are really short notes - about 500 characters.
I've also installed the Windows app on another Windows machine I have to see if there is anything machine specific. This new machine is a completely fresh install - only been a couple of hours but it appears to be doing the same thing - log files shows many of the ECONNRESET errors and in looking at the files, nothing special. This client also never syncs with new files created on the iPad, but the iPad is working perfectly fine and does pick up new files created with the Windows clients.
I am going to probably blow everything out again and just try with a few notes created in Joplin (no Evernote imports) to see there is something wrong in the Evernote files, unless there is anything else you suggest I try that might help with troubleshooting.
It would be surprising if it was related to Evernote as this is a network connection issue. It wouldn't be a firewall or proxy issue by any chance?
That is a good point. I will run some tests later by switching one of the Windows machines over to a cell hotspot (to eliminate my router) and I'll see if there are any errant firewall rules. I did get the notice on both machines during install to allow traffic through and I selected yes. I'll report back what I find.
This all appears to have been a OneDrive problem yesterday. I let things sit overnight and am now getting no errors and sync is completing as expected.
I think I was assuming this second issue was the same as my first issue, with Cloudamo. Apparently Cloudamo sees Joplin syncing many notes and thinks something is going on and it temporarily locks out that connection. It does eventually come back, but further syncing will cause it to lock again. Due to this, I believe that Cloudamo is not suitable for Joplin WebDAV syncing unless one is starting from scratch with a minimal number of notes to sync. This is the service that NextCloud is currently pushing, so I expect there will be more people who try this with a large number of Evernote notes as I did.
In any event, OneDrive is working really well for me and I will continue to use that.
2 Likes
That's pretty bad actually. If they do it on purpose, that's the wrong approach as it would randomly break any client that sync many files. Much better to do like Dropbox or OneDrive and throttle the connection as at least it means sync can be completed.
And if it's a bug, it looks bad because downloading and uploading files is the core feature of Nextcloud (or should be) and once again they don't seem to be getting this right.
FYI, I submitted a ticket with Cloudamo to ask them about these issues and I will report back with their answers if I get them. If they could resolve this, it would be great as overall it is a promising service. I paid $17 for a year's worth of 50GB storage, which is pretty reasonable.
For my situation, I had to switch over to DropBox as I got caught by the dreaded "4MB" note error on OneDrive and couldn't figure out how to resolve it. I am closely following this pull request as I believe OneDrive is a more economical and better performing option for Joplin syncing for many people.
I think I have the same issue. Coming from Evernote, imported 6k+ notes, encrypted to Onedrive. Long uploading but went OK. Now the desktop and Android version are always in fetching mode.
I.e. it just fetched 8k+ items on PC. Finished. I updated a couple of notes. Sync OK. Opened the Android app to try to finish the sync that never completed till now after a week and the PC version started a full fetching again!
BTW the desktop version overloads the CPU the whole time while fetching...
1 Like
I have the same issue. I am using the app both on phone and PC, haven't imported anything from other apps. Everything was written mostly from my phone. I am using OneDrive btw. On the desktop app it keeps synching indefinitely. At some point it reached around 65k fetched and I had to stop it but the changes made on the phone were not updated on the desktop app.
Note: I have around 150 notes written by hand, some have links, tables, embedded photos. on the phone it usually reaches close to 2k fetched but it is nowhere near 65k on the desktop.
Do you have any idea what might be the cause?
Joplin 1.5.14 on Win8 synching with OneDrive.
Roughly 30 notes only, 5- attachments.
Synching says:
Created 5 remote items.
Fetched: 6000 and counting.
Fetched items is ever increased by 12 but seems to run without end.
I am currently in the process of moving my notes from OneNote to Joplin. But such thing make me a little worried regarding reliability.
1 Like