2024-04-29 - Problems synchronising with the Android app and the Joplin Cloud

Coffee · 2024-06-12T11:55:59.073Z

Coffee:

And my most important Windows app sometimes feels like it doesn't respond at all, or at least extremely slowly, when I select a different note. And that's even though no sync process is running because the status is ‘Cannot acquire exclusive lock’. Nevertheless (or because of that?), the Windows app uses 50 or more percent of the CPU for minutes at a time, which then runs at a total of about 70 to 80 percent.

After a restart of my (virtual) Windows PC, the CPU usage of the Windows Joplin app is back to normal and the app is responding as usual:
CPU 0 to 10 per cent
Usual fast response to mouse clicks / switching to other notes

laurent · 2024-06-12T15:10:26.503Z

If it hasn't been suggested yet it would be worth looking at your plugins, as this is often the reason for extreme performance issues. This and OCR actually, but normally once OCR has processed all resources it should be fast for new ones.

laurent · 2024-06-12T15:11:30.188Z

Coffee:

A new error message has appeared on the Samsung S23, which reads as follows

This error indicates that your devices are not in sync, so you'd need to make sure synchronisation goes to completion on all of htem

Coffee · 2024-06-12T16:09:21.326Z

laurent:

OCR actually, but normally once OCR has processed all resources it should be fast for new ones

OCR could be the cause.
Hundreds of elements were reloaded from the Joplin cloud by the Windows app, although there was no reason for this in my opinion. If these were all analysed again by OCR, this could explain the high load.

The plugins don't seem suspicious to me.

Screenshot 2024-06-12 175608 - plugins704×434 32.6 KB

Coffee · 2024-06-12T16:21:07.171Z

laurent:

so you'd need to make sure synchronisation goes to completion on all of them

Coffee:

Can I afford to be patient, or should I take action?

Patience is therefore required

After yesterday's synchronisation on my S10 failed to take place without any recognisable action and even ‘Cancel’ had no effect, I rebooted the S10 and waited a few hours before restarting the Joplin app. After the next start of the app, the synchronisation started to run normally again, but seems to be synchronising 100% of the elements from the Joplin cloud again. This synchronisation process has been running for several hours and I don't want to interrupt it until it has been successfully completed.

Coffee · 2024-06-13T06:43:51.787Z

laurent:

so you'd need to make sure synchronisation goes to completion on all of htem

Patience alone will not solve the problem

After the Android app (Joplin 2.14.9) on my S10 still hadn't finished the very long sync process this morning, I discovered a message in the mobile-log.log that prompted me to quit and restart the app.
I suspect that the session / connection to the Joplin cloud expired without the Android app reacting accordingly. I can imagine that this behaviour of the Android app could / has led to inconsistent / incorrect data in the database on the Android device.

06-13T03:14:54: JoplinServerApi: "curl -v -X GET -H "X-API-AUTH: ******" -H "X-API-MIN-VERSION: 2.6.0" 'https://api.joplincloud.com/api/share_users'"
06-13T03:14:54: JoplinServerApi: "Code:", "403"
06-13T03:14:54: JoplinServerApi: "Error: Your session has expired. Please login again.
Code: 403
Error: Your session has expired. Please login again.
    at construct (native)
    at Wrapper (address at index.android.bundle:1:2619599)
    at construct (native)
    at _createSuperInternal (address at index.android.bundle:1:2618675)
    at call (native)
    at JoplinError (address at index.android.bundle:1:2619053)
    at newError (address at index.android.bundle:1:2819444)
    at ?anon_0_ (address at index.android.bundle:1:2819043)
    at next (native)
    at fulfilled (address at index.android.bundle:1:2815879)
    at tryCallOne (/root/react-native/ReactAndroid/hermes-engine/.cxx/MinSizeRel/5p1x2e66/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:53:16)
    at anonymous (/root/react-native/ReactAndroid/hermes-engine/.cxx/MinSizeRel/5p1x2e66/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:139:27)
    at apply (native)
    at anonymous (address at index.android.bundle:1:219452)
    at _callTimer (address at index.android.bundle:1:218401)
    at _callReactNativeMicrotasksPass (address at index.android.bundle:1:218565)
    at callReactNativeMicrotasks (address at index.android.bundle:1:220540)
    at __callReactNativeMicrotasks (address at index.android.bundle:1:121846)
    at anonymous (address at index.android.bundle:1:120939)
    at __guard (address at index.android.bundle:1:121687)
    at flushedQueue (address at index.android.bundle:1:120850)
    at invokeCallbackAndReturnFlushedQueue (address at index.android.bundle:1:120793)"
06-13T02:00:58: JoplinServerApi: "Code:", "403"
06-13T02:00:58: JoplinServerApi: "Error: Your session has expired. Please login again.
Code: 403
Error: Your session has expired. Please login again.
    at construct (native)
    at Wrapper (address at index.android.bundle:1:2619599)
    at construct (native)
    at _createSuperInternal (address at index.android.bundle:1:2618675)
    at call (native)
    at JoplinError (address at index.android.bundle:1:2619053)
    at newError (address at index.android.bundle:1:2819444)
    at ?anon_0_ (address at index.android.bundle:1:2819043)
    at next (native)
    at fulfilled (address at index.android.bundle:1:2815879)
    at tryCallOne (/root/react-native/ReactAndroid/hermes-engine/.cxx/MinSizeRel/5p1x2e66/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:53:16)
    at anonymous (/root/react-native/ReactAndroid/hermes-engine/.cxx/MinSizeRel/5p1x2e66/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:139:27)
    at apply (native)
    at anonymous (address at index.android.bundle:1:219452)
    at _callTimer (address at index.android.bundle:1:218401)
    at _callReactNativeMicrotasksPass (address at index.android.bundle:1:218565)
    at callReactNativeMicrotasks (address at index.android.bundle:1:220540)
    at __callReactNativeMicrotasks (address at index.android.bundle:1:121846)
    at anonymous (address at index.android.bundle:1:120939)
    at __guard (address at index.android.bundle:1:121687)
    at flushedQueue (address at index.android.bundle:1:120850)
    at invokeCallbackAndReturnFlushedQueue (address at index.android.bundle:1:120793)"
06-13T02:00:58: JoplinServerApi: "curl -v -X GET -H "X-API-AUTH: ******" -H "X-API-MIN-VERSION: 2.6.0" 'https://api.joplincloud.com/api/locks'"

Coffee · 2024-06-14T09:31:04.237Z

FYI:

2024-06-14 - Feature Request - Android - Number of items in a notebook Features

As in the Windows app, the number of items contained in a notebook should also be displayed in the Android app. This applies in particular to the notebooks Conflicts and Conflicts ( Attachments ) Android: [Screenshot_20240614_110222_Joplin] [Screenshot_20240614_110242_Joplin] Windows: [12dacf7723531e19d3a86f1d718c7f87] [e0d1450cc927b21081aaccedbf86fa7f]

Coffee · 2024-06-14T09:48:24.046Z

Coffee:

Patience alone will not solve the problem

Waiting for the synchronisation of all my Joplin apps to be completed did not solve the problem. The data quality and the amount of data in my main profile have deteriorated significantly overnight.

To limit the resulting risk to my data content, I only ran two apps this morning and deleted five apps completely. Since then, there seems to be no further uncontrollable increase in conflicts.

Screenshot 2024-06-14 1140341631×121 52.9 KB

Coffee · 2024-06-14T10:00:46.628Z

Coffee:

On the old Samsung S10:
5/5 -> 3/3 now 8/8
On the new Samsung S23
5/5 -> now 32/32

5/5 -> 3/3 -> 8/8 -> ... -> App deleted from S10 on 2024-06-14
On the new Samsung S23
5/5 -> 32/32 -> ... -> 100/100 on 2024-06-14

Coffee · 2024-06-14T10:23:56.548Z

personalizedrefriger:

I've created a plugin that might help debug this. Among other things, it lists information available through the data API about resources (e.g. whether Joplin still thinks the resources are encrypted) and allows deleting resources permanently.

It does, however, require a Joplin 3.x pre-release to work on mobile.

My Windows App is actually
Joplin 2.14.22 (prod, win32)
Backup: 1.4.1
Conflict Resolution: 1.2.3

Your plugin says

This plugin supports both Joplin Desktop >= 2.14 and Joplin Mobile >= 3.0.

Could this plugin already help me to clean up my data in my Windows app?

If so, what plugin name should I look for to set it up in my Windows app?

debug

?

Screenshot 2024-06-14 124439 - Debug Tool

personalizedrefriger · 2024-06-14T16:49:52.821Z

Yes — it should work in Joplin 2.14 (edit: the Joplin 3.0 requirement is just for mobile). Be sure that you have an up-to-date backup before deleting anything though!

"Joplin debug tool" is the plugin's name in Joplin's plugin repository.

Coffee · 2024-06-14T17:30:21.612Z

personalizedrefriger:

Yes — it should work in Joplin 2.14. Be sure that you have an up-to-date backup before deleting anything though!

Thank you – I will try to be very careful and only use the plugin for the advanced analysis of details for the time being and not make any changes with it yet.

Coffee · 2024-06-14T17:39:55.682Z

I forked 2024-06-14 - A closer look at a new conflict

personalizedrefriger · 2024-06-14T18:02:42.728Z

Coffee:

Thank you – I will try to be very careful and only use the plugin for the advanced analysis of details for the time being and not make any changes with it yet.

I've just added documentation on how to show information for a specific note/notebook/resource from an ID. (Previously, documentation on this feature was missing).

Coffee · 2024-06-17T13:40:11.378Z

Coffee:

To limit the resulting risk to my data content, I only ran two apps this morning and deleted five apps completely. Since then, there seems to be no further uncontrollable increase in conflicts.

Even after running just two apps and waiting a very long time to see if they were fully synced, I still couldn't get the notes to stabilise.
That's why I decided to take another drastic step: I deleted my main profile in my Android app and started from scratch with the synchronisation from the Joplin cloud.
After more than 24 hours, no further synchronisation progress was visible in the Android app. However, the synchronisation status has since shown a fairly constant number of around 5500 attachments that have not been downloaded.
To get another reference value, I installed a Joplin app on an iPad for the first time.
After about 24 hours, synchronisation was also complete on the iPad. In contrast to the Android app, all attachments are displayed as downloaded and decrypted on the iPad in synchronisation status.

Coffee · 2024-06-17T13:51:36.573Z

Coffee:

To get another reference value, I installed a Joplin app on an iPad for the first time.

After about 24 hours, synchronisation was also complete on the iPad. In contrast to the Android app, all attachments are displayed as downloaded and decrypted on the iPad in synchronisation status.
I spent about an hour noting certain parameters from the synchronisation status and manually counted elements in the conflict folders on Android and iOS after synchronising all three apps several times to the same state. Some values are strikingly identical, others strikingly different

Coffee · 2024-06-17T13:59:45.003Z

Coffee:

noting certain parameters

Number of attachments = 7445

Android: 5464 not downloaded + 1980 downloaded and decrypted +    1 created locally  
iOS:        0 not downloaded + 7445 downloaded and decrypted +    0 locally created  
Windows:    0 not downloaded + 1698 downloaded and decrypted + 5747 created locally

Coffee · 2024-06-17T14:13:56.761Z

Coffee:

noting certain parameters

Conflicts:

Android:   1  
iOS:      94  
Windows: 642

Coffee · 2024-06-17T14:16:32.649Z

Coffee:

noting certain parameters

Conflicts (attachments) = 413

Android: 413  
iOS:     413  
Windows: 413

Coffee · 2024-06-17T14:22:30.994Z

laurent:

so you'd need to make sure synchronisation goes to completion on all of them

Is there anything I can do to ensure that the newly created profile in the Android app downloads all the attachments? I don't think so. I think that's a problem you'll have to find a solution for.