Is Slow Synchronization in Joplin a Manipulation or a Necessity?

[KC_20CY]

@Medullitus

Thanks. I've read that too. But what our developer doesn't seem to understand or is ignoring is the problem caused by the first synchronization taking days.

My best guess is that @laurent knows about "taking days" part.

[KC_20CY]

@Medullitus

@KC_20CY

Who else or what else do you have on your mind?

I hope I am not going too far asking this question because you use the term "curse" in your first reply to me.

I think it's deliberate that a note-taking app that has been in operation since 2018 cannot find a solution for days-long synchronization in 2024. I also believe that the problems in other synchronization solutions are deliberately overlooked or even worsened to sell Joplin Cloud to people. I'm not presenting this as a claim. It's my own opinion based on the developments, and it can change.

Apart from a one-sentence accusation message that @laurent could respond to with a clear and detailed answer satisfying everyone and ending the discussions. His lack of contribution to the topic also justifies these suspicions.

What I am going to express might be considered relevant or irrelevant to this thread, but your earlier reply did inspire me and took one of my philosophical research subjects to a whole new different level, Just ignore this post if considered irrelevant, I am expressing my gratitude. I'm from Taiwan BTW.

[Medullitus]

Thank you. We're not here to argue anyway. We're all here to exchange ideas and keep the community spirit alive, striving to make Joplin a better application together. Greetings from Turkey

[KC_20CY]

@Medullitus

Thank you. We're not here to argue anyway. We're all here to exchange ideas and keep the community spirit alive, striving to make Joplin a better application together. Greetings from Turkey

Thanks, to you and your Turkish fellowman in below news:
Turkish rescue workers assist Taiwan's quake relief efforts - Focus Taiwan

[KC_20CY]

As in my Joplin profile goes, I am a self-titled AI philosopher and a conceptual artist, so except expressed my philosophical gratitude in my earlier post, please allow me to express my gratitude in a more artistic way. By doing so, my best guess is that any follower who is following my online activity can truly get inspired like I do from our idea exchange in this thread.

Below is my artistic gratitude with a few words and a video:

The aforementioned inspiration is a DOUBLE-COOL one.

How to write 酷 (kù) – cool (pleasant, fine) – stroke order, radical, examples and spoken audio

[roman_r_m]

Not many, 1260 notes, the total profile size is under 1gb I think.

[Medullitus]

Thanks. I had guessed that. Don't forget to drop by when you reach 11k notes. Of course, if nothing has changed by then.

[Medullitus]

@tomasz86

Not really possible even with root. You would need to modify the Android source code itself, and then compile your own version of the OS. Also, Syncthing can already run as root but it doesn't change anything. For the record, this slow file access through FUSE in Android is a rather new thing. If you use Syncthing in Android 10 or older, then the sync is going to be extremely quick.

@tomasz86

You're right, it's apparently Android 9 and older. I've got one Android 7 device still in use and the difference is not even funny, e.g. it may take 10 minutes to sync 10,000 files there, while on a new Android 14 device the same operation will take at least 10 hours, if not much longer.

FUSE has been used starting from Android 11. I'm using Android 10. Therefore, your claim about slowness due to the information provided and Android is incorrect in terms of FUSE! The proof link is below:

Source Android

Additionally, here you say that root access is required to overcome slow synchronization, but you also mention that the current Joplin app does not have a root mode. Now you're saying that even root access won't be enough to solve the problem. Which one is correct?

@tomasz86

This is a known issue if you look up "android fuse" in any search engine. The only way to work around this would be through root, but Joplin doesn't have any special root mode which would allow for direct filesystem access.

@tomasz86

Not really possible even with root. You would need to modify the Android source code itself, and then compile your own version of the OS. Also, Syncthing can already run as root but it doesn't change anything. For the record, this slow file access through FUSE in Android is a rather new thing. If you use Syncthing in Android 10 or older, then the sync is going to be extremely quick.

Android sync from FS still very slow

Yes, the problem affects filesystem sync only . How early of a version of Joplin? The older versions which targeted older SDKs were allowed to access the filesystem more directly (= faster). The current versions are forced to use FUSE which is more secure but also very slow (e.g. see https://www.reddit.com/r/androiddev/comments/kpn68k/android_11_very_slow_file_access_performance). This is a known issue if you look up "android fuse" in any search engine. The only way to work …

@roman_r_m

[KC_20CY]

@KC_20CY

@Medullitus

Thanks. I've read that too. But what our developer doesn't seem to understand or is ignoring is the problem caused by the first synchronization taking days.

My best guess is that @laurent knows about "taking days" part.

I am pondering on some phenomenon so I am asking this question.

Does anyone know that after the time of above message set, is there any update/forum news/forum announcement which relates "faster speed of the initial sync of a many-notes Joplin user"? Sometime the aforementioned updates hidden in somewhere of old but updated thread, so I ask this question.

[roman_r_m]

Not sure what you're expecting of me tbh. I think I'm done with explaining. If you prefer to think the slowness is introduced intentionally to nudge users towards Joplin Cloud, that's your right.

But then I'm going to think that you're working for one of the competitors like Obsidian or Standard Notes, and make these statements to push people away (just kidding, I don't actually think so, but since we're doing conspiracy theories..)

[Medullitus]

@roman_r_m

Not sure what you're expecting of me bth. I think I'm done with explaining. If you prefer to think the slowness is introduced intentionally to nudge users towards Joplin Cloud, that's your right.

But then I'm going to think that you're working for one of the competitors like Obsidian or Standard Notes, and make these statements to push people away (just kidding, I don't actually think so, but since we're doing conspiracy theories..)

Have you noticed your own words aside from what I've written? Didn't you emphasize @tomasz86 below?

Is Slow Synchronization in Joplin a Manipulation or a Necessity?

Also - Android's file system access is slow, as @tomasz86 has pointed out. Even if the existing code can be optimized, I am not convinced there can be huge performance gain.

Also, I had mentioned my thoughts about Obsidian and Standard Notes. Besides, I speak of links and sources that substantiate my claims. On the other hand, you're speaking based on your own thoughts, and you can even change your stance later on. My advice to you is to be honest and approach the topic impartially. Otherwise, you may harm the application you think you're protecting!

@Medullitus

However, let's not misunderstand here; there are note-taking applications such as Obsidian, Simplenote, etc... that do not allow for alternatives other than their own servers and charge monthly or yearly fees. I am not comparing Joplin to them, and I consider Joplin to be superior because it grants users such freedom.

[tomasz86]

@Medullitus

FUSE has been used starting from Android 11. I'm using Android 10. Therefore, your claim about slowness due to the information provided and Android is incorrect in terms of FUSE! The proof link is below:

Source Android

Additionally, here you say that root access is required to overcome slow synchronization, but you also mention that the current Joplin app does not have a root mode. Now you're saying that even root access won't be enough to solve the problem. Which one is correct?

The link is about FUSE passthrough, not FUSE itself. This is an additional mechanism that Google added later to supposedly speed up the slow I/O caused by FUSE. It hasn't really helped much though, at least in my use case.

When it comes to rooting, according to my current knowledge, root access alone cannot overcome this limitation.

[Medullitus]

@tomasz86

The link is about FUSE passthrough, not FUSE itself. This is an additional mechanism that Google added later to supposedly speed up the slow I/O caused by FUSE. It hasn't really helped much though, at least in my use case.

When it comes to rooting, according to my current knowledge, root access alone cannot overcome this limitation.

Really? No matter how much evidence I provide, you continue to make different explanations. Show me a source that Google brought FUSE as default to devices before Android 11, then I will apologize to you.

But I'm going a step further and sharing an image that shows my Android 10 device uses the SDCardFS infrastructure, not FUSE.

Please accept this, the problem is not Android. Joplin's synchronization feature is problematic and faulty. Please discuss this with @laurent .

[tomasz86]

Alright, so when it comes to different Android versions, I apparently got FUSE and scoped storage confused. Scoped storage has been used since Android 10, FUSE since Android 11 (see https://source.android.com/docs/core/storage/scoped#using-scoped-storage-with-fuse). In addition to that, FUSE and sdcardfs are not exclusive to each other (see https://source.android.com/docs/core/storage/scoped#fuse-and-sdcardfs). However, if you are still on Android 10, then you shouldn't be affected by the FUSE slowness regardless.

Following the above, this means that if it takes you a long time to sync Joplin files via Syncthing under Android 10, then the problem is almost certainly something else than FUSE. However, the slow file sync becomes a Syncthing problem at this point, not Joplin's (unless the sheer fact of Joplin creating large numbers of tiny files that need to be synced is considered the culprit).

With filesystem sync through Syncthing, there are two stages:

1. Device → Device: This stage utilises just Syncthing, i.e. Joplin is not involved at all.
2. Filesystem → Joplin: This stage syncs the Joplin's database with the files that have been synced with Syncthing. At this point Syncthing is no longer involved.

If 1) is slow, then it's a Syncthing problem, if 2) is slow, then the problem lies within Joplin (and possibly Android itself).

[Radiant]

I tried Dropbox, webdav, onedrive and dislike them alot.

Slow sync, device freeze/hang.

The best option is either using Joplin Server or Joplin Cloud.

Fast, reliable and light in terms of memory usage.

[alnetloc]

Sounds like a problem I had for a while. Turns out it was an issue with using Joplin on desktop and android at the "same time" - I was new to the app so kept comparing notes in both locations to check sync. All I had to do was remove some temp files - honestly can't remember which, but in one of the lock or sync folders. That totally solved the issue for me.

[Medullitus]

@tomasz86

Alright, so when it comes to different Android versions, I apparently got FUSE and scoped storage confused. Scoped storage has been used since Android 10, FUSE since Android 11 (see https://source.android.com/docs/core/storage/scoped#using-scoped-storage-with-fuse). In addition to that, FUSE and sdcardfs are not exclusive to each other (see https://source.android.com/docs/core/storage/scoped#fuse-and-sdcardfs). However, if you are still on Android 10, then you shouldn't be affected by the FUSE slowness regardless.

Following the above, this means that if it takes you a long time to sync Joplin files via Syncthing under Android 10, then the problem is almost certainly something else than FUSE. However, the slow file sync becomes a Syncthing problem at this point, not Joplin's (unless the sheer fact of Joplin creating large numbers of tiny files that need to be synced is considered the culprit).

With filesystem sync through Syncthing, there are two stages:

1. Device → Device: This stage utilises just Syncthing, i.e. Joplin is not involved at all.
2. Filesystem → Joplin: This stage syncs the Joplin's database with the files that have been synced with Syncthing. At this point Syncthing is no longer involved.

If 1) is slow, then it's a Syncthing problem, if 2) is slow, then the problem lies within Joplin (and possibly Android itself).

Thank you for your response. I had already shared my opinion on what the problem might be. But your acknowledgment of the possibility that the issue might be with Joplin is a small step for you but a big one for Joplin.

So let's brainstorm a bit more. When we use Syncthing with Obsidian, do we encounter the same days-long synchronization issue for 10k and above notes as we do with Joplin?

1-) The answer is yes, the problem is with Android.
2-) The answer is no, the problem is with Joplin.

Somehow, the possibility of the second option seems more likely. But dear @tomasz86 , do you know what the real issue is? It's @laurent deliberately not fixing the slow synchronization issue to sell Joplin Cloud. I would really like to be wrong about this! However, the silence from our developer confirms my suspicions.

[Medullitus]

@Radiant

The best option is either using Joplin Server or Joplin Cloud.

Yes, that's exactly what we were talking about. I recommend reading through the previous posts on the topic. But it's worth keeping in mind that the best option you suggest could turn into the worst option for those using Joplin for free.

"Are the free options too slow? Then switch to Joplin Cloud!" I meant exactly that with manipulation. Thank you for providing an example.

[Medullitus]

@alnetloc

Sounds like a problem I had for a while. Turns out it was an issue with using Joplin on desktop and android at the "same time" - I was new to the app so kept comparing notes in both locations to check sync. All I had to do was remove some temp files - honestly can't remember which, but in one of the lock or sync folders. That totally solved the issue for me.

Unfortunately, I'm quite skeptical that there's a magical file that will speed up synchronization when deleted.

[Holzk]

Well, finally it became completely clear what you have wanted to claim from the start:

@Medullitus

what the real issue is? It's @laurent deliberately not fixing the slow synchronization issue to sell Joplin Cloud.

@Medullitus

the silence from our developer confirms my suspicions.

This is one of the most terrible accusations I have ever seen against a free software developer. Why do you not understand yet that you've got no right to accuse someone for something provided AS IS basis on pure speculation? You should understand it is a horrible attitude.

You seem to try to pretend as a neutral user but do know that it is quite clear to anyone that you got an agenda here. But at least let's be based on a factual ground. Your opinion is pure speculation and really unproductive.