Filesystem sync not working well on Android

With the 2.9.8 on my Android 12 phone, I finally got the sync working by correctly specifying the path. So it appears the notes are backed up to the folder.

But the good experience stops there.

First I set up SyncThing to sync the Joplin folder with my two other phones (Android 9 running Joplin 2.8.1). I see the folder has over 3000 files with a total size of 3MB, but the sync took 30 minutes on the new phone. Once the files are there, I started Joplin and it took about 2 minutes to sync without allowing any operation from me. After syncing, the notes and notebooks are still what it had, nothing from the two other phones show up. And every time I start Joplin (after I threw the app window out), it does the 2-minute sync. This is so even I set the sync interval to "Disabled".

In comparison, the 2.8.1 on Android 9 doesn't seem to do the 2-minute sync. Things seem to work "nicely", but syncing of the files also don't always work. Some notes or some notebooks show up on the other phone, but many don't.

This is only to sync my 3 phones with about 100 small notes in total.

I wonder how this can be improved.

2 Likes

The slow sync (both in Syncthing and Joplin), especially when doing the initial synchronisation, is expected in Android 11 and newer. There is no direct filesystem access and everything has to go through FUSE (look up "android fuse" if you're interested in details) which can lead to extremely slow performance, in particular when dealing with large numbers of small files located in a single folder.

When it comes to the notes not showing issue, I've got a feeling that you might have been hit by https://github.com/laurent22/joplin/issues/6517, especially if you tried to perform the Joplin sync while Syncthing was still downloading the data. There's no easy fix for that one, unfortunately. I would suggest to make sure that your Syncthing synchronisation has fully finished first, then uninstall and reinstall Joplin (or just wipe the app data), and only then try to perform the first Joplin sync using the filesystem target. Make sure to keep the phone screen on until the sync has finished completely (which can be done by setting the screen timeout to a very high value in the OS settings or disabling it completely in Developer Options).

Just for reference, you may want to have a look at https://forum.syncthing.net/t/how-to-deal-with-abysmal-i-o-performance-on-android-10/18406 which describes my fight with syncing Joplin files onto Android with Syncthing. In the end, I had to basically reduce note history greatly in order to minimise the total file number, and in addition to that use the workaround described in https://github.com/laurent22/joplin/issues/6517 to finally achieve reliable synchronisation.

Thanks, @tomasz86 . Your analysis is right on.

I made a comment on Sync silently ignores new files with old timestamps · Issue #6517 · laurent22/joplin · GitHub.

As an extra question, if I disable note history, will the number of files be largely reduced? If so, that would be nice as I don't care much about the history.

Yep, it was reduced from nearly 12,000 to just 2,000 in my case, and I didn't even disable it completely but rather reduced the number of days to 7 (from the default 90). Of course, there may be differences depending on your use case and the type of notes you store in Joplin (and how often you edit them).

1 Like

I don't know how you get the number of files down. I set to disable history and the files in the synced folder don't change. In my case, 100 short notes in total on three phones generate over 3000 files, that sounds crazy.

I uninstalled/reinstalled Joplin and did sync again. After 1 hour, it appeared having finished, but many notes from other phones were not there. I pressed on "Sync" again, and it appeared to start the whole process from the beginning. So this appeared pretty unusable to me.

You talked about SyncThings being slow with many files. In my case, SyncThings took about 30 minutes to get all the files (over 3000 with 3MB in total size) from the other two Android 9 phones on to the new Android 12 phone, but Joplin sync in its own storage can't finish in 1 hour.

BTW, your notes regarding Android 12 using Fuse passthrough, is not entirely true. There is a statement here

Devices upgrading from Android 11 to Android 12 can't support FUSE passthrough as the kernels for these devices are frozen and they can't move to a kernel that's been officially upgraded with the FUSE passthrough changes.

My S21 now running Android 12 belongs to that category.

Do you, or anyone else reading this, believe that if Joplin implemented a "one file per note" mechanism then this situation would tremendously better? What are the pros and cons? I would like to look into the possibilities.

I think it may take up to 24 hours for Joplin to actually delete notes from history, so please re-check the situation the next day or so. The overall number of files should drop by then.

My comments weren't about the passthrough, just FUSE itself :upside_down_face:. The passthrough is something that supposedly improves FUSE performance in Android 12 and later. I'm sure it will still be slower than the direct filesystem access that older versions of Android provide.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.