Sync is blocked while Joplin on Android slowly deletes remote items

Support
1

Operating system

Android

Joplin version

3.2.7

Sync target

Nextcloud

What issue do you have?

On a new phone, Joplin's initial sync of around 5000 notes was relatively quick, only taking about 10 or 15 minutes to download the notes. The problem is that following the initial download all sync download activity is blocked while the status displays "Deleted remote items: n" as "n" slowly increments. This has yet to complete after nine hours. Is this expected behavior? Is there a way around this? Cancelling does not stop whatever process has taken over. On previous attempts to complete the sync I stopped Joplin and cleared all data and uninstalled. Nothing worked.

2

There should be a view "deletion log" option in the menu for both desktop and mobile. Have a look at that, and if you see lots of lines which say "Revision.deleteOldRevisions", then yes that is normal.

I'm guessing you are getting many revision deletions on the initial sync because you have set a lower value for the note history keep for setting, compared to on your other Joplin client(s)

3

Thanks for the advice. The logs show the entries you described. I increased the length of time history is kept, and also increased the number of simultaneous threads for sync in the hope it would speed things up.

UPDATE: This had little effect. I'm in day two of waiting for it to finish. This seems like a bug to me since it blocks all other synchronization while it's deleting the old revisions, making the app useless as a way to share notes.

4

On your old phone, go to settings, tools synchronization status. It should have a line something like "Revision: x/x". What is the value for the revision line? Is it a really big number or just something small? Normal behaviour would be to just remove revisions as they they expire which should be a small amount if you use Joplin regularly. But if you somehow have hundreds of thousands of revisions being continually deleted, then that would not be normal behaviour

5

Unfortunately, the old phone has been wiped already. I do have logs from the current phone, which are somewhat puzzling because Joplin claims to be deleting remote items right now, but the logs are all from two days ago on 30 April. I did hit refresh in the log view, but nothing changed. I've attached the logs FWIW.
mobile-log.log (499.6 KB)

6

I should also say, the first time this started happening last week when I initially transferred Joplin to the new phone, I stopped the app and deleted all the data, then reconfigured it for sync. It did the same thing as I described in the first post above. After a while, I de-installed and re-installed the Joplin app, and that's where I am now, still waiting for the finish of deleting remote items. Does this not sound like a bug?

7

Information is added to the deletion log when items are deleted locally. The above deletion log includes roughly 6,500 old deleted revisions. I suspect that:

  1. A few days ago, Joplin locally deleted all revisions older than the "Keep note history for" setting -- this is what's shown in the log.
  2. Joplin is now attempting to sync the ~6,000 local revision deletions.

I think it could make sense for Joplin to allow re-prioritizing different sync actions. For example, if an item is created during an ongoing sync, make creating that item a higher priority than any deletions scheduled for the sync.

For now, a workaround might be to:

  1. Make sure you have a recent backup.
  2. Reinstall Joplin.
  3. Set "Keep note history for" to a large number.
  4. Set up sync with Nextcloud.
  5. Sync.
1 Like
8

Please post the revision count on the new device then. Or if you have another device that is synced, check there.

You could also go on your Nextcloud and check the count of the number of items at the root of the Joplin folder for rough idea of how many objects exist for your profile

9

There's only 3809 matches on "Revision.deleteOldRevisions" in the log, are you sure you didn't search a term which doubled up the results?

10

I'm counting the matches with the following script:

function countMatches(text) {
  let count = 0;
  text.replace(/"[a-z0-9]{32}"/g, () => { count ++ });
  return count;
}

Some of the .deleteOldRevisions lines include multiple IDs.

11

Ah I see, some lines delete multiple revisions

12

I marked this as the solution, although it is really just a workaround as you point out. BTW, the max you can set "Keep note history" to is 730, but that got it through a sync. (I had already raised it to about 260, but that wasn't enough I guess.)

1 Like
13

Same problem on my side but on all other platforms too. On my side this is particularly bad because I did a bulk import of my Evernote notes a few months back and therefore the clean-up takes really long ( I mean hours on mobile devices).

What I do not understand is why does each device have to go through the deletion of the remote items? Why would each device cleans up the older versions? Why could not one device do the clean-up (like by example a desktop) and then the other ignores this?

Again this is particularly annoying because this really blocks the sync of my devices making Joplin useless for at least a day. Would there be a way to prioritise things so that the delete would not look the other sync functions? Or let only the desktop version to this? Or ...

Any idea?

Thanks

14

Unfortunately, there is nothing you can do on the user's side to overcome this. The problem with Android is that due to Google's restrictions, file system access is extremely slow in comparison to desktop operating systems, hence synchronisation also takes much longer :frowning_with_open_mouth:.

I believe he only workaround that actually works is to use a separate profile with essential notes, and only sync that with Android. The more notes (both in number and complexity) you've got, the slower synchronisation becomes.

15

Regarding slow sync on Android in general, what you have said is generally the case, although depending on the version of Android you have, some versions (mainly the newer versions) may sync faster than others.

Supposedly using Joplin server / Joplin cloud should remove any networking slowness, but wont help if the bottleneck is on the Android file access.

@jpservices I can't pinpoint the cause of so many note revision deletions happening, but if you do not care about note history, you can disable the cleanup by disabling note history on all your devices. This would also mean that no new note history is created, and any deletions which have been synced will need to be applied to all clients, before the note deletions will stop.

Alternatively, it is possible to set the note history expiry to a very high value on your desktop clients eg. 999999 days, so the cleanup will never happen in your lifetime - but on Android set note history to disabled (there is a hard limit of 720 days on Joplin mobile). This would mean that note history will get created when using the desktop client, but wont when using mobile. But it wont ever delete the note history either, unless you enable the note history expiry. This would have the disadvantage though that when you setup a new device, over time it will take longer and longer to do the initial sync, because note history will never get deleted.

16

Honestly, I'd say it is the opposite, e.g. on Android 15 sync is very slow, and this is true also when using filesystem sync (so no network impact here). For instance, I can barely use Joplin for grocery lists, as it may take several minutes to sync changes while I'm standing in a shop, waiting for them to load :sweat_smile:.

On the other hand, syncing should be much faster on Android 10 or older, because there, Google still allowed direct filesystem access.

1 Like
17

I havn't noticed slowness on Android 14, but it sounds like you're right about Android 11 or later having slowness. According to this you get slowness if you have lots of files in 1 folder, which is not the case for me with a pretty small number of objects in my profile:

EDIT: Just noticed that you were the author of that post, haha

It's just like how accessing files on my phone on Windows via MFT protocol (which is what you have to use since mass storage device mode was removed) has to load up all files in a folder before I can modify the contents of it. That makes it really slow to download folders from the Camera folder on my phone, so for this reason I resorted to using the device explorer in Android Data Studio to transfer data to and from my phone, which uses adb under the hood and does not need to list each directories contents before you can access it

18

@tomasz86 I read a bit more into the issue you mentioned with FUSE and found a related Google issue here:

Do you happen to have an Android 15 phone which also has a micro sd slot? The issue I linked (which also references issues with syncthing in post #33) seems to suggest that the slow performance issue is just with sd card access, and not with internal storage.

I recall a recent issue where a user with a recent Android phone needed to grant external sd card access to the Joplin app in order to get the internal storage access to work. So I wonder whether the file access performance issue with Joplin is actually a non issue on newer Android devices (bar Android 11 which I think has some additional issues related to that) if they don't have an external sd card slot (like for myself), but the presence of a slot causes the Joplin app to use the slower file access.

19

No SD card slot, and the slowness for me happens on all devices running new Android versions (e.g. Google Pixel 8 and Google Pixel 9 Pro Fold, both on Android 15, and previously on Android 14). I've done a lot of troubleshooting regarding this, but I'm pretty sure there is no workaround (other than modifying and compiling the Android OS yourself…).

For comparison, an old device running Android 7 is able to perform the same synchronisation blazingly fast.

For the record, as you've mentioned earlier, it seems to be about listing files located in the same directory when accessing a file, and with Joplin this is a problem, as it stores all its MD files under the same path. If the total number of those files is low, accessing them is fine, but as soon as the number exceeds 1,000 or so, things start to deteriorate.

1 Like
20

Thanks for looking into this.

I understand that mobile is slower than desktop and that the newer Android versions might bring slowness with the security features to funnel memory access ( I am on Android 11, 12 and 14 on 3 devices, all Samsung) but after initial syncing Joplin works for me. The problem is really this cleaning up that can happen at any time (here in my case was while on the road for a few weeks with little network bandwidth: it really looked me a few days!). But I also have a IOS with the same set of notes and got the same troubles and did not notice any significant difference.

Thanks for the suggestion here. My large set of deletion was finally more that 200K items. I suspect that this happens now because I imported thousands of notes and deleted them and reimported them. The deleted notes probably became obsolete in the history all at once and were then finally deleted 90 days later or something like this. With all the tests I did I assume I will still have such things in a few days.

So if I understand you correctly if I disable history on a mobile notes modified on that mobile will not be tracked in the history AND then the clean-up will never happen on that mobile.

So what happens when I sync on the desktop (where history would be set to 90 days)? Those notes modified on the above mobile would then be added to the history or would the version created on that mobile never show up on the history?

History is a cool feature and I would hate having to turn it off for performance reasons.

Thanks.