Is Slow Synchronization in Joplin a Manipulation or a Necessity?

roman_r_m · 2024-04-12T10:49:40.896Z

Not sure what you're expecting of me tbh. I think I'm done with explaining. If you prefer to think the slowness is introduced intentionally to nudge users towards Joplin Cloud, that's your right.

But then I'm going to think that you're working for one of the competitors like Obsidian or Standard Notes, and make these statements to push people away (just kidding, I don't actually think so, but since we're doing conspiracy theories..)

Medullitus · 2024-04-12T11:02:17.707Z

roman_r_m:

Not sure what you're expecting of me bth. I think I'm done with explaining. If you prefer to think the slowness is introduced intentionally to nudge users towards Joplin Cloud, that's your right.

But then I'm going to think that you're working for one of the competitors like Obsidian or Standard Notes, and make these statements to push people away (just kidding, I don't actually think so, but since we're doing conspiracy theories..)

Have you noticed your own words aside from what I've written? Didn't you emphasize @tomasz86 below?

Is Slow Synchronization in Joplin a Manipulation or a Necessity? Support

Also - Android's file system access is slow, as @tomasz86 has pointed out. Even if the existing code can be optimized, I am not convinced there can be huge performance gain.

Also, I had mentioned my thoughts about Obsidian and Standard Notes. Besides, I speak of links and sources that substantiate my claims. On the other hand, you're speaking based on your own thoughts, and you can even change your stance later on. My advice to you is to be honest and approach the topic impartially. Otherwise, you may harm the application you think you're protecting!

Medullitus:

However, let's not misunderstand here; there are note-taking applications such as Obsidian, Simplenote, etc... that do not allow for alternatives other than their own servers and charge monthly or yearly fees. I am not comparing Joplin to them, and I consider Joplin to be superior because it grants users such freedom.

tomasz86 · 2024-04-12T13:11:48.594Z

Medullitus:

FUSE has been used starting from Android 11. I'm using Android 10. Therefore, your claim about slowness due to the information provided and Android is incorrect in terms of FUSE! The proof link is below:

Source Android

Additionally, here you say that root access is required to overcome slow synchronization, but you also mention that the current Joplin app does not have a root mode. Now you're saying that even root access won't be enough to solve the problem. Which one is correct?

The link is about FUSE passthrough, not FUSE itself. This is an additional mechanism that Google added later to supposedly speed up the slow I/O caused by FUSE. It hasn't really helped much though, at least in my use case.

When it comes to rooting, according to my current knowledge, root access alone cannot overcome this limitation.

Medullitus · 2024-04-12T13:37:04.125Z

tomasz86:

The link is about FUSE passthrough, not FUSE itself. This is an additional mechanism that Google added later to supposedly speed up the slow I/O caused by FUSE. It hasn't really helped much though, at least in my use case.

When it comes to rooting, according to my current knowledge, root access alone cannot overcome this limitation.

Really? No matter how much evidence I provide, you continue to make different explanations. Show me a source that Google brought FUSE as default to devices before Android 11, then I will apologize to you.

But I'm going a step further and sharing an image that shows my Android 10 device uses the SDCardFS infrastructure, not FUSE.

WhatsApp Görsel 2024-04-12 saat 16.31.53_e88f2792738×1509 83.6 KB

Please accept this, the problem is not Android. Joplin's synchronization feature is problematic and faulty. Please discuss this with @laurent .

tomasz86 · 2024-04-12T16:32:17.343Z

Alright, so when it comes to different Android versions, I apparently got FUSE and scoped storage confused. Scoped storage has been used since Android 10, FUSE since Android 11 (see https://source.android.com/docs/core/storage/scoped#using-scoped-storage-with-fuse). In addition to that, FUSE and sdcardfs are not exclusive to each other (see https://source.android.com/docs/core/storage/scoped#fuse-and-sdcardfs). However, if you are still on Android 10, then you shouldn't be affected by the FUSE slowness regardless.

Following the above, this means that if it takes you a long time to sync Joplin files via Syncthing under Android 10, then the problem is almost certainly something else than FUSE. However, the slow file sync becomes a Syncthing problem at this point, not Joplin's (unless the sheer fact of Joplin creating large numbers of tiny files that need to be synced is considered the culprit).

With filesystem sync through Syncthing, there are two stages:

Device → Device: This stage utilises just Syncthing, i.e. Joplin is not involved at all.
Filesystem → Joplin: This stage syncs the Joplin's database with the files that have been synced with Syncthing. At this point Syncthing is no longer involved.

If 1) is slow, then it's a Syncthing problem, if 2) is slow, then the problem lies within Joplin (and possibly Android itself).

Radiant · 2024-04-13T02:56:22.347Z

I tried Dropbox, webdav, onedrive and dislike them alot.

Slow sync, device freeze/hang.

The best option is either using Joplin Server or Joplin Cloud.

Fast, reliable and light in terms of memory usage.

alnetloc · 2024-04-13T08:15:28.845Z

Sounds like a problem I had for a while. Turns out it was an issue with using Joplin on desktop and android at the "same time" - I was new to the app so kept comparing notes in both locations to check sync. All I had to do was remove some temp files - honestly can't remember which, but in one of the lock or sync folders. That totally solved the issue for me.

Medullitus · 2024-04-13T15:31:41.190Z

tomasz86:

Alright, so when it comes to different Android versions, I apparently got FUSE and scoped storage confused. Scoped storage has been used since Android 10, FUSE since Android 11 (see https://source.android.com/docs/core/storage/scoped#using-scoped-storage-with-fuse). In addition to that, FUSE and sdcardfs are not exclusive to each other (see https://source.android.com/docs/core/storage/scoped#fuse-and-sdcardfs). However, if you are still on Android 10, then you shouldn't be affected by the FUSE slowness regardless.

Following the above, this means that if it takes you a long time to sync Joplin files via Syncthing under Android 10, then the problem is almost certainly something else than FUSE. However, the slow file sync becomes a Syncthing problem at this point, not Joplin's (unless the sheer fact of Joplin creating large numbers of tiny files that need to be synced is considered the culprit).

With filesystem sync through Syncthing, there are two stages:

Device → Device: This stage utilises just Syncthing, i.e. Joplin is not involved at all.

Filesystem → Joplin: This stage syncs the Joplin's database with the files that have been synced with Syncthing. At this point Syncthing is no longer involved.

If 1) is slow, then it's a Syncthing problem, if 2) is slow, then the problem lies within Joplin (and possibly Android itself).

Thank you for your response. I had already shared my opinion on what the problem might be. But your acknowledgment of the possibility that the issue might be with Joplin is a small step for you but a big one for Joplin.

So let's brainstorm a bit more. When we use Syncthing with Obsidian, do we encounter the same days-long synchronization issue for 10k and above notes as we do with Joplin?

1-) The answer is yes, the problem is with Android.
2-) The answer is no, the problem is with Joplin.

Somehow, the possibility of the second option seems more likely. But dear @tomasz86 , do you know what the real issue is? It's @laurent deliberately not fixing the slow synchronization issue to sell Joplin Cloud. I would really like to be wrong about this! However, the silence from our developer confirms my suspicions.

Medullitus · 2024-04-13T15:35:43.435Z

Radiant:

The best option is either using Joplin Server or Joplin Cloud.

Yes, that's exactly what we were talking about. I recommend reading through the previous posts on the topic. But it's worth keeping in mind that the best option you suggest could turn into the worst option for those using Joplin for free.

"Are the free options too slow? Then switch to Joplin Cloud!" I meant exactly that with manipulation. Thank you for providing an example.

Medullitus · 2024-04-13T15:36:02.013Z

alnetloc:

Sounds like a problem I had for a while. Turns out it was an issue with using Joplin on desktop and android at the "same time" - I was new to the app so kept comparing notes in both locations to check sync. All I had to do was remove some temp files - honestly can't remember which, but in one of the lock or sync folders. That totally solved the issue for me.

Unfortunately, I'm quite skeptical that there's a magical file that will speed up synchronization when deleted.

Holzk · 2024-04-13T16:03:39.891Z

Well, finally it became completely clear what you have wanted to claim from the start:

Medullitus:

what the real issue is? It's @laurent deliberately not fixing the slow synchronization issue to sell Joplin Cloud.

Medullitus:

the silence from our developer confirms my suspicions.

This is one of the most terrible accusations I have ever seen against a free software developer. Why do you not understand yet that you've got no right to accuse someone for something provided AS IS basis on pure speculation? You should understand it is a horrible attitude.

You seem to try to pretend as a neutral user but do know that it is quite clear to anyone that you got an agenda here. But at least let's be based on a factual ground. Your opinion is pure speculation and really unproductive.

Medullitus · 2024-04-13T17:25:33.445Z

Holzk:

Well, finally it became completely clear what you have wanted to claim from the start:

Medullitus:

what the real issue is? It's @laurent deliberately not fixing the slow synchronization issue to sell Joplin Cloud.

Medullitus:

the silence from our developer confirms my suspicions.

This is one of the most terrible accusations I have ever seen against a free software developer. Why do you not understand yet that you've got no right to accuse someone for something provided AS IS basis on pure speculation? You should understand it is a horrible attitude.

You seem to try to pretend as a neutral user but do know that it is quite clear to anyone that you got an agenda here. But at least let's be based on a factual ground. Your opinion is pure speculation and really unproductive.

I've been saying this from the beginning. You can even check the topic title. If you're just grasping this now, either you have ill intentions or a more profound issue to ponder.

You have no authority or right to determine what I have the right to or not. If you can't tolerate me freely expressing my thoughts here, you're revealing that your concept of freedom contradicts the philosophy of free software.

In my opinion, your opening a new account and writing on this topic for the first time, and your writings not reflecting your own ideas but defending others, is very concerning on your behalf! It might be a good choice for you to either write from your real account or submit your resignation to the law firm (!) you belong to.

By the way, I'm substantiating my writings and claims with concrete sources and links. But some people, such as yourself, don't see them. Because breaking their prejudices is even harder than splitting an atom.

Radiant · 2024-04-13T23:33:59.992Z

Medullitus:

Radiant:

The best option is either using Joplin Server or Joplin Cloud.

Yes, that's exactly what we were talking about. I recommend reading through the previous posts on the topic. But it's worth keeping in mind that the best option you suggest could turn into the worst option for those using Joplin for free.

"Are the free options too slow? Then switch to Joplin Cloud!" I meant exactly that with manipulation. Thank you for providing an example.

You can always try with self hosted joplin server (with docker).

Me myself not using Joplin cloud but a self hosted one instead (docker version).

So far really good.

Why Joplin server is better, because it can sync your notes, tages, workbook in parallel, have chunked feature that split your files when uploading and use a very low memory print.

All notes, tag, workbook are synced perfectly accross all devices.

I'm using 5 devices that sync under 5-10 seconds.

Medullitus · 2024-04-14T07:03:00.264Z

Radiant:

Medullitus:

Radiant:

The best option is either using Joplin Server or Joplin Cloud.

Yes, that's exactly what we were talking about. I recommend reading through the previous posts on the topic. But it's worth keeping in mind that the best option you suggest could turn into the worst option for those using Joplin for free.

"Are the free options too slow? Then switch to Joplin Cloud!" I meant exactly that with manipulation. Thank you for providing an example.

You can always try with self hosted joplin server (with docker).

Me myself not using Joplin cloud but a self hosted one instead (docker version).

So far really good.

Why Joplin server is better, because it can sync your notes, tages, workbook in parallel, have chunked feature that split your files when uploading and use a very low memory print.

All notes, tag, workbook are synced perfectly accross all devices.

I'm using 5 devices that sync under 5-10 seconds.

The reason I've been asking for a detailed guide about WebDAV from the beginning is this. But Joplin server is not free. You need to have your own server or rent a server to set it up. These are paid solutions. I repeat, "paid".

In free solutions, synchronization takes three days or even longer, whereas in paid solutions, synchronization taking 5-10 seconds is not ethical and is manipulation. Those claiming otherwise need to technically explain the slowness in free options without leaving any doubt.

I also recommend those who complain about Android being slow in Joplin to read about this synchronization that takes 5-10 seconds. @tomasz86

Medullitus · 2024-04-14T07:07:40.553Z

@Holzk

By the way, the most dangerous ones for a project or application are those who lick the boots of the developer and lean towards authority figures like you. Despite people like you, I will continue to speak the truths I know. You can continue crying now. If you dare to disrespectfully write to me again and attack me without reason, you will hear much harsher words.

Holzk · 2024-04-15T03:55:22.224Z

Medullitus:

authority figures like you

This does not make sense (please look at my user rank and the number of the posts I have created on this forum so far) I am an ordinary user of Joplin who just knows how to behave in a FOSS community.

Medullitus:

the truths I know

Let's calm down... if you indeed know something which have not been elaborated yet, I think you are welcomed to create a Pull Request here so that anyone can have a look at your proposal which actually may improve the software (I am not excluding this possibility a priori).

Medullitus:

concrete sources and links

When it comes to FOSS, the direct and most concrete source of the information is its source code; let's be based on GitHub - laurent22/joplin: Joplin - the secure note taking and to-do app with synchronisation capabilities for Windows, macOS, Linux, Android and iOS., log the issue here concretely, discuss which issue to be fixed. Because the source code is available, you do not need to be based on hearsay.

To claim your victory to the world against the "evil" author and its community, please be based on the source code. Otherwise, nobody will be rationally persuaded.

Medullitus:

you will hear much harsher words.

It is up to moderators (and none of my business), but I think you are gradually risking yourself to be banned.

LeoW · 2024-04-15T04:50:21.464Z

It seems to me that if you choose the local file system sync method you should be able to zip the entire directory on your PC for example and move it to your Android phone easily either over Wi-Fi or cable. There are several ways to transfer files between Windows and Android. The fastest I think would be to put your PC data on a portable SSD drive with USB-C and then plug into phone. I also have a gigabit USB-C ethernet adapter. I might put both PC and phone on ethernet cables and transfer the data. A USB cable might be fast too. I have to run some tests.

Then, you unzip the file into the appropriate folder on your Android phone and tell Joplin to use that folder for file system sync and presumably, It should read the contents and be able to see all of your notes and notebooks. I'm thinking that this would be faster for doing your initial synchronization using any other method.

If you're interested in not using cloud storage and instead using file system sync and then directly synchronizing the files back and forth from an Android device to Windows or Linux there is an option that I am going to put to the test to confirm that it works only I can't do that in my schedule until Wednesday night.

You can share your windows folder on the network and then load an app on your Android to sync the Android folder with your PC.

I'm Planning to test four different apps to accomplish this.

SMBSync2
Roundsync
FolderSync
Rclone

Option 2 & 3 & 4 can also sync with a variety of cloud servers.

Option 4 good theoretically allow me to use file system sync on the PC and webdav sync on the Android devices. I want to compare the speed difference between local sync and webdav sync. Either way there won't be any server throttling.

I'll probably test Mega S3 as the server, and pcloud too. My bandwith to Mega is excellent.

Medullitus · 2024-04-15T16:51:17.116Z

Holzk:

Medullitus:

authority figures like you

This does not make sense (please look at my user rank and the number of the posts I have created on this forum so far) I am an ordinary user of Joplin who just knows how to behave in a FOSS community.

Medullitus:

the truths I know

Let's calm down... if you indeed know something which have not been elaborated yet, I think you are welcomed to create a Pull Request here so that anyone can have a look at your proposal which actually may improve the software (I am not excluding this possibility a priori).

Medullitus:

concrete sources and links

When it comes to FOSS, the direct and most concrete source of the information is its source code; let's be based on GitHub - laurent22/joplin: Joplin - the secure note taking and to-do app with synchronisation capabilities for Windows, macOS, Linux, Android and iOS., log the issue here concretely, discuss which issue to be fixed. Because the source code is available, you do not need to be based on hearsay.

To claim your victory to the world against the "evil" author and its community, please be based on the source code. Otherwise, nobody will be rationally persuaded.

Medullitus:

you will hear much harsher words.

It is up to moderators (and none of my business), but I think you are gradually risking yourself to be banned.

I didn't say you are an authority figure. I said you're someone who leans towards authority figures. Because your attitude really was like that. But since you're changing your attitude, then I will respond accordingly.

I'm not claiming that @laurent or the developers are bad people. If you had read my messages, you would have seen that. In fact, I mentioned that even if Laurent intentionally slows down synchronization to sell Joplin Cloud as I suggested, I can understand that, but I don't agree with it being done covertly.

Thank you for your suggestion. But I'm not a programmer. I'm just expressing my doubts about the application. And I explain the reasons for those doubts. If you're also curious why synchronization with Joplin Cloud takes a few minutes while it takes days in other options, and why the background synchronization feature hasn't been activated despite being requested five years ago, then as a user, you should voice your questions and doubts. I'm doing just that. If you don't have such concerns, you don't have to speak up. But I can ask if you're a Joplin Cloud user at this point.

Medullitus · 2024-04-15T16:53:26.679Z

LeoW:

It seems to me that if you choose the local file system sync method you should be able to zip the entire directory on your PC for example and move it to your Android phone easily either over Wi-Fi or cable. There are several ways to transfer files between Windows and Android. The fastest I think would be to put your PC data on a portable SSD drive with USB-C and then plug into phone. I also have a gigabit USB-C ethernet adapter. I might put both PC and phone on ethernet cables and transfer the data. A USB cable might be fast too. I have to run some tests.

Then, you unzip the file into the appropriate folder on your Android phone and tell Joplin to use that folder for file system sync and presumably, It should read the contents and be able to see all of your notes and notebooks. I'm thinking that this would be faster for doing your initial synchronization using any other method.

If you're interested in not using cloud storage and instead using file system sync and then directly synchronizing the files back and forth from an Android device to Windows or Linux there is an option that I am going to put to the test to confirm that it works only I can't do that in my schedule until Wednesday night.

You can share your windows folder on the network and then load an app on your Android to sync the Android folder with your PC.

I'm Planning to test four different apps to accomplish this.

SMBSync2
Roundsync
FolderSync
Rclone

Option 2 & 3 & 4 can also sync with a variety of cloud servers.

Option 4 good theoretically allow me to use file system sync on the PC and webdav sync on the Android devices. I want to compare the speed difference between local sync and webdav sync. Either way there won't be any server throttling.

I'll probably test Mega S3 as the server, and pcloud too. My bandwith to Mega is excellent.

That's a great idea. But I'm considering an even better solution. Have you heard of the Titanium Backup application?

LeoW · 2024-04-15T16:57:00.543Z

Medullitus:

Have you heard of the Titanium Backup application?

Nope. I'll have to check and read about it.