Questions about sync performance

bunk3m · 2023-05-16T14:18:09.537Z

Versions: Mac Joplin 2.10.18; Android Joplin 2.10.8; Macos 12.6.5 & Android 13 on tablet and Phone.

I have been syncing with Dropbox for some time. It worked quickly on the Mac Joplin but very slowly on Android. I've had to delete the Android app a few times in the past and allowed the sync to start from a new install. This usually fixed the sync problems. However recently it is so slow I've given up using Dropbox.

My file is now about 650MB in size with about 2800 notes and about 10,000 resources.

I recently set up a NextCloud server at home and thought this would provide a better sync. I have tried to move the files to NextCloud.

So far I've been unsuccessful. The sync from the Mac has been running for 2 days and only synced about 600 notes. Since I am attached to the Nextcloud via direct ethernet, I expected the sync to happen quicker.

When coping pictures to the Nextcloud instance, I moved 4.5GB of files in 20-30 min. This made me think that there is something strange going on with the NextCloud sync. ??

I have followed the various discussions and have wiped the NextCloud folder a few times and started fresh. I forced the upload from the desktop but nothing is improving the upload sync.

I thought I might move the joplin-desktop folder into the NextCloud folder on the Mac but can't find any way to do this. Is it possible?

I'm stuck and can't figure out what to do next. Ideally I'd like my sync to happen on my NextCloud. But I can't get the sync to complete there. What else should I be trying?

Thanks in advance.

bunk3m · 2023-05-16T14:52:03.357Z

Trying something new.

After reading how someone was using Syncthing to sync their Joplin notes, I decided to try a similar approach.

So I am doing a "File system" sync to get the notes to my local NextCloud folder. In a way this seems silly because now I have the same files twice on my laptop.

Then I can do the same "File system" sync to the Android Nextcloud. I suspect that I'll have two copies of my notes on the phone also.

So while the "local" sync has completed rather quickly and the NextCloud sync nearly instantaneously, I think I have four copies of the notes.

Is this correct?

Is there a better way?

laurent · 2023-05-16T15:07:36.271Z

Nextcloud is slow but should still be faster than Dropbox. Is it possible that your server hardware is not performant enough?

Otherwise, if you want to use WebDAV, Nginx or Apache will probably be faster. Although I don't think anyone ever done a benchmark but it's not hard to be faster than Nextcloud

bunk3m · 2023-05-16T15:38:14.802Z

Thank you @laurent

I think the server is fast enough. I have gigabit ethernet throughout and just purchased and updated my home server.

The local file sync on the Mac and the NextCloud sync went very quickly. The NextCloud sync between the Mac and NextCloud server was almost as fast as the Joplin local file sync.

Unfortunately, I can't get the Android Joplin permissions to work to read the NextCloud synced files. Here is the screenshot.

Screenshot_20230516_112232_Joplin617×1600 97.5 KB

I am not sure how to fix this error.

I'll create a backup and then try WebDAV and see how quickly it goes.

If I just change the sync method and point it to the folder on NextCloud will this sync properly or will it wipe out all the files?

bunk3m · 2023-05-16T17:59:41.280Z

Made the backup and connected via WebDAV.

The sync quit after uploading 2170 items and then decided to Cancel.

???

laurent · 2023-05-16T19:51:06.380Z

Any error in the log?

bunk3m · 2023-05-16T19:52:28.859Z

I didn't check before I deleted everything and started again. Sorry. This time I'll check for errors the log.

bunk3m · 2023-05-16T19:58:26.098Z

I'm trying another variant. This may be completely stupid but I'm running out of things to do.

I've imported my JEX backup into a completely clean Joplin Mac.

I mounted the NextCloud volume directly so I can copy files as if on the same network. I am waiting for a local sync to this directly connected volume. I've used Advanced -> push all local files to sync target.

After it has completed, I'll change the way the sync happens. I'll switch to WebDAV sync method. I hope then Joplin will check the files and see that they are identical and won't get stuck on the upload.

lyricism · 2023-05-16T20:42:15.940Z

It looks as though the error is occurring because ACTION_OPEN_DOCUMENT is not being used by the Joplin Android app. I am not very well-versed in Android APIs so maybe someone else has a solution for this issue. In the meantime, I can do some digging and see what I can figure out.

bunk3m · 2023-05-17T13:47:07.408Z

OK here is an update.

I completed the mount the NextCloud volume directly, forced the upload of / push the files from Joplin on Mac to mounted drive on NextCloud as "Local sync" target. That went quickly, about 20 min. to complete.

I changed the sync to "WebDAV" and ran the sync. It hung with some lock errors. I quit Joplin and came back 2 hours later and it completed it's sync.

For your info some interesting notes in the log.

I don't use encryption however there are many many errors about looking for encryption key. For example this:

2023-05-16 15:29:25: models/Setting: "Could not set setting on the keychain. Will be saved to database instead: plugin-io.github.jackgruber.backup.password:", "Error: Password is required.
Error: Password is required.
    at checkRequired (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/keytar/lib/keytar.js:5:11)
    at Object.setPassword (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/keytar/lib/keytar.js:20:5)
    at KeychainServiceDriver.<anonymous> (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/keychain/KeychainServiceDriver.node.js:19:43)
    at Generator.next (<anonymous>)
    at /Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/keychain/KeychainServiceDriver.node.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/keychain/KeychainServiceDriver.node.js:4:12)
    at KeychainServiceDriver.setPassword (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/keychain/KeychainServiceDriver.node.js:16:16)
    at KeychainService.<anonymous> (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/keychain/KeychainService.js:51:32)
    at Generator.next (<anonymous>)"

Here are the error messages when Joplin couldn't complete the sync on the WebDAV sync. Please note that there is no other Joplin instance syncing with the NextCloud so I'm not sure what is locking the file. The files are already on the NextCloud instance with no changes so there shouldn't be anything locked after it has completed it's scan:

2023-05-16 17:49:25: Synchronizer: "Error: PUT 213b63fbf6d74c5fac3647f0db423c05.md: "Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked (Exception OCA\DAV\Connector\Sabre\Exception\FileLocked) (423): <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked</s:message>
</d:error>

Code: 423
Error: PUT 213b63fbf6d74c5fac3647f0db423c05.md: "Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked (Exception OCA\DAV\Connector\Sabre\Exception\FileLocked) (423): <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked</s:message>
</d:error>

    at newError (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/WebDavApi.js:414:11)
    at WebDavApi.exec (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/WebDavApi.js:441:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async FileApiDriverWebDav.put (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/file-api-driver-webdav.js:203:10)"
2023-05-16 17:49:25: Synchronizer: "Req 17:49:01: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/88a98b728acb49949dfc347cee3a1dfb.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-45481","User-Agent":"Joplin/1.0","Content-Length":"21424"},"body":"some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:01: "
2023-05-16 17:49:25: Synchronizer: "Req 17:49:02: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/8923f7c2331c4660b901f5dc60c9357a.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-97483","User-Agent":"Joplin/1.0","Content-Length":"1989"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:02: "
2023-05-16 17:49:25: Synchronizer: "Req 17:49:02: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/46423ea9b2284df3ab98e6ac71e4b0bf.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-62419","User-Agent":"Joplin/1.0","Content-Length":"3905"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:02: "
2023-05-16 17:49:25: Synchronizer: "Req 17:49:02: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/49a801be215b40bca4ad83f726432f74.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-48602","User-Agent":"Joplin/1.0","Content-Length":"5565"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:02: "
2023-05-16 17:49:25: Synchronizer: "Req 17:49:03: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/2cd05a07657a4b7d80af19d740b3055a.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-15025","User-Agent":"Joplin/1.0","Content-Length":"7361"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:03: "
2023-05-16 17:49:25: Synchronizer: "Req 17:49:03: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/451369d3dc7e4ddb8b7d37755084ba7a.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-2924","User-Agent":"Joplin/1.0","Content-Length":"14984"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:03: "
2023-05-16 17:49:25: Synchronizer: "Req 17:49:03: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-36491","User-Agent":"Joplin/1.0","Content-Length":"3593"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:03: <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked</s:message>
</d:error>
"
2023-05-16 17:49:25: Synchronizer: "Req 17:49:08: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-26269","User-Agent":"Joplin/1.0","Content-Length":"3593"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:08: <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked</s:message>
</d:error>
"
2023-05-16 17:49:25: Synchronizer: "Req 17:49:15: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-19483","User-Agent":"Joplin/1.0","Content-Length":"3593"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:15: <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked</s:message>
</d:error>
"
2023-05-16 17:49:25: Synchronizer: "Req 17:49:25: PUT https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md {"headers":{"Authorization":"********","Content-Type":"text/plain","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-79805","User-Agent":"Joplin/1.0","Content-Length":"3593"},"body":"Some text","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:25: <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked</s:message>
</d:error>
"
2023-05-16 17:49:25: Synchronizer: "Req 17:49:25: PROPFIND https://cloud.nextcloud_site.tld/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/locks/ {"headers":{"Depth":1,"Authorization":"********","Content-Type":"text/xml","Cache-Control":"no-store","If-None-Match":"JoplinIgnore-33667","User-Agent":"Joplin/1.0","Content-Length":"190"},"body":"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\t\t\t<d:propfind xmlns:d=\"DAV:\">\n\t\t\t\t<d:prop xmlns:oc=\"http://owncloud.org/ns\">\n\t\t\t\t\t<d:getlastmodified/><d:resourcetype/>\n\t\t\t\t</d:prop>\n\t\t\t</d:propfind>","ignoreTlsErrors":false,"timeout":120000,"maxRetry":0}"
2023-05-16 17:49:25: Synchronizer: "Res 17:49:25: <?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns"><d:response><d:href>/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/locks/</d:href><d:propstat><d:prop><d:getlastmodified>Tue, 16 May 2023 21:48:52 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/files/8886AB4D-D4DE-4954-90D1-F7EDB276BD05/Home/Documents/joplin/locks/1_1_2c16a9a75c0f4607a75e70531a32404b.json</d:href><d:propstat><d:prop><d:getlastmodified>Tue, 16 May 2023 21:48:52 GMT</d:getlastmodified><d:resourcetype/></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>
"
2023-05-16 17:49:26: Synchronizer: "Sync: finished: Synchronisation finished [1684273546172]"
2023-05-16 17:49:26: Synchronizer: "Operations completed: "
2023-05-16 17:49:26: Synchronizer: "createRemote: 325"
2023-05-16 17:49:26: Synchronizer: "Total folders: 28"
2023-05-16 17:49:26: Synchronizer: "Total notes: 2056"
2023-05-16 17:49:26: Synchronizer: "Total resources: 5609"
2023-05-16 17:49:26: Synchronizer: "There was some errors:"
2023-05-16 17:49:26: Synchronizer: "Error: PUT 213b63fbf6d74c5fac3647f0db423c05.md: "Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked (Exception OCA\DAV\Connector\Sabre\Exception\FileLocked) (423): <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked</s:message>
</d:error>

Code: 423
Error: PUT 213b63fbf6d74c5fac3647f0db423c05.md: "Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked (Exception OCA\DAV\Connector\Sabre\Exception\FileLocked) (423): <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Home/Documents/joplin/213b63fbf6d74c5fac3647f0db423c05.md" is locked</s:message>
</d:error>

    at newError (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/WebDavApi.js:414:11)
    at WebDavApi.exec (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/WebDavApi.js:441:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async FileApiDriverWebDav.put (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/file-api-driver-webdav.js:203:10)"
2023-05-16 17:49:26: "Sync has finished and note has never been changed - reloading it"
2023-05-16 17:49:27: "DecryptionWorker: cannot start because no master key is currently loaded."
2023-05-16 17:50:23: RepositoryApi: "findWorkingMirror: Using: https://raw.githubusercontent.com/joplin/plugins/master"
2023-05-16 17:51:42: RevisionService: "maintenance: Starting..."
2023-05-16 17:51:42: RevisionService: "maintenance: Service is enabled"
2023-05-16 17:51:42: RevisionService: "collectRevisions: Created revisions for 0 notes"
2023-05-16 17:51:42: RevisionService: "maintenance: Done in 108ms"

Now I'll try to sync one of the Android devices to see if that works.

bunk3m · 2023-05-18T15:44:03.434Z

An update on Android sync.

I had to restart Joplin Android a few times but over the course of a day, it completed. Today the sync on Android took only 14 seconds for creating 46 local items, updating 4 and deleting 3 items. Not bad at all.

I suspect that the long initial sync was due to file sync locks conflicting between the two instances of Joplin. The Android was syncing and the Mac tried to check. Then the Android version locked up. I haven't figured out how to review the log files on Android efficiently. The scrolling in Joplin Config log record is very slow.

wh201906 · 2023-05-19T05:44:04.824Z

I guess Configuration->EXPORT DEBUG REPORT helps.

system · 2023-06-18T05:44:55.314Z

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.