Is Slow Synchronization in Joplin a Manipulation or a Necessity?

Support
21

... even if it's just to provoke a reaction.

I would say you succeeded on that point.

... there is no clear explanation for the cause of this slowness

That is because there is no single reason and most of them are not under the control of Joplin.

  • The company providing the cloud storage may, for their own business reasons, have their systems configured in a way that is not ideal for the type of transfers Joplin has to make. Previously on this forum a user suggested adding a particular provider. It was found that the provider actually stated that their service was designed more for file archiving and so they apply throttling if too many transfers are requested in a particular period of their choosing.
  • Some companies providing could storage allow access to that storage using the WebDav protocol. However not all implement it properly and some will throttle connections as they do not want sync traffic.
  • With regards to limits, Joplin will not know of any bandwidth limit until it hits it. Then the connection to the third-party server slows down or, if there is a connection limit, it refuses any further connections and Joplin errors. This is all totally under the control of the third-party provider.
  • Some companies open up their API to allow users to access the storage (like Microsoft or Dropbox). Joplin can use that API to connect to the storage for syncing. If this is slower than one may expect the only people who can explain why their closed, proprietary systems are slow are those companies themselves. Why not contact Microsoft or Dropbox and ask for concrete proof they are not slowing down connections from Joplin?
  • IIRC the idea of creating Joplin Server / Cloud came about because of the variable quality of the third-party services available.
  • @roman_r_m has mentioned that there may be the possibility of improving sync a bit. However nothing they do can affect how the third-party servers are configured.

Therefore, the only thing we need to be sure that the developer is not acting manipulatively are concrete evidence.

If there was code in Joplin that limited all sync connections other than Joplin Cloud then someone could highlight it and point it out to you. How can someone highlight code that isn't actually there? It's not possible to give "concrete" proof of a negative.

Therefore, brushing off as a "conspiracy theory" is not a pleasant behavior for a developer.

You started by asking if the slowness of the sync was, "a manipulation by the developers to push us to buy Joplin Cloud?". So you put forward a hypothesis suggesting that the developers of Joplin were secretly working together to commit a wrongful action by sabotaging connections to anything other than their own Joplin Cloud product. That's a conspiracy theory. In response people have commented that that is not the case and that the quality and performance of the storage servers is likely the issue and that Joplin's code is open for inspection. In response to that you now call for "concrete" evidence that your theory is incorrect whilst providing no evidence to suggest that it is. A journalist called Christopher Hitchens devised an epistemological razor which states "what can be asserted without evidence can also be dismissed without evidence". Basically it means that if someone is going to throw around some wild theory they should have something to back it up before expecting those affected to have to use time and energy to prove them wrong.

11 Likes
22

@roman_r_m @dpoulton

Thank you for your responses. I especially appreciate dpoulton for taking the time and providing a detailed answer. Dpoulton if you look into the topic, you will also see that my only expectation was to receive a detailed response like yours. While I respect and appreciate the contributions of other respondents, I must emphasize that your response and approach are ideal.

As you can appreciate, aside from provocative aspects, through this topic and your post, people will develop a strong belief that the "days-long" synchronization process is generally due to factors outside of Joplin. I believe this is an important step towards better understanding the synchronization process. If I had seen such a response before, I would have been more careful in framing my topic to avoid misunderstandings. In fact, my suggestion to you is to issue a general warning considering people who are considering transitioning from applications like Evernote or Notion, which have their own servers, to Joplin.

However, let's not misunderstand here; there are note-taking applications such as Obsidian, Simplenote, etc... that do not allow for alternatives other than their own servers and charge monthly or yearly fees. I am not comparing Joplin to them, and I consider Joplin to be superior because it grants users such freedom.

Regarding your question about code, I am not knowledgeable about coding. In my response to Roman above, I mentioned that I don't know if a possible bandwidth limitation, being server-related, would be present in the application code. Therefore, I am not asking you to show me something you don't have. I am asking as developers if you are throttling bandwidth. If you, as far as I know as a developer, say that such throttling is not happening, then of course, I will believe you. However, as you can appreciate, companies, for example VPN companies, can market what they do as not doing certain things for consumers, such as keeping user logs or even selling them to advertising companies. Therefore, Joplin is also a company and deserves to be questioned.

English is not my native language. But as far as I know, the word "manipulation" does not imply insult. I have never had any intention of insulting in my questions. On the contrary, I opened a topic for explanations from you and other developers. But even if I was mistaken, you can appreciate that the way to correct a mistake is not by the other party making a mistake as well. So my question was never a hypothesis. I just asked a question. That's why I used the term "necessity" in my topic title. Nevertheless, I apologize if I was misunderstood or offended anyone, including you and @laurent . As I said, I opened this topic not to blame but to get information.

Besides, as a user who doesn't know code, it would be quite audacious even for me to propose a hypothesis to those who do know it!

One final suggestion, please consider creating a guide for Nextcloud and WebDAV for Joplin by yourself or knowledgeable people on the topic. This way, we can get rid of third-party software like Onedrive and Dropbox and improve our own solution. This would also strengthen the argument that Joplin Cloud is optional.

23

If we're talking about OneDrive or Dropbox then such hypothetical limitation can only be in the application code, because the server side is controlled by a 3rd party (Microsoft or Dropbox respectively in this case).

2 Likes
24

Unless you are going to set-up your own server and install WebDav or Nextcloud it will be to a third-party server as well.

If self-hosting an Internet connected server there is much more to the server than just installing WebDav or Nextcloud, and just installing them is not a simple thing. A lot of server configuration is also required to keep the server as safe as possible from attack and this is an ongoing task, not just when setting up the WebDav / Nextcloud server.

As @muzak already mentioned above, the Joplin web-site already has pages on connecting Joplin to Nextcloud or WebDav (although the WebDav document basically just says "See Nextcloud" as they both use the WebDav protocol).

Joplin - Nextcloud / Joplin - WebDav

If you want instructions on how to install WebDav or Nextcloud on your server then you will need to search the Internet for installing WebDav or visit Nextcloud's installation guide.

1 Like
25

Thank you for the response. In that case, let me share the latest update from my research.

I installed Docker on my computer and used Docker to set up Nextcloud. I connected a flash drive to the USB port of my modem. With Nextcloud's "external driver" feature, I can access this flash drive connected to my modem. Similarly, I can also access the flash drive from my computer via FTP. Now, what I aim to do is to create a Joplin folder on the flash drive and synchronize from there. But I haven't been able to start this yet because the synchronization process on my phone is still ongoing...

26

I like the spirit of starting this thread.

27

I'm afraid you might have cursed yourself by liking my post, and as a result, your sync speed could drop by 99% for three days. Good luck with that.

28

I never experience a 'fast' or 'smooth' Joplin synchronization from day 1 that I started using Joplin, so, three days of whatever makes no difference.

1 Like
29

I used Dropbox for a year without any issue. I didn't find the syncs after the first one, which was long, to be a problem. When I switched to Joplin Cloud, it was because then my annual contributions for the software I needed would be taken automatically, and I wouldn't have to remember it.

That said, I can't understand why it would seem odd that if you compare two systems. One a client server relationship between two SQL servers optimized for working together, to another normal file archiving system. Then find the SQL servers are much faster at transferring and processing their SQL data, and consider that odd.

Furthermore, OneDrive is notoriously slow. I hate to see Dropbox put in the same category.

As an English speaker, Manipulation does have a nefarious connotation.

1 Like
30

I have a feeling that Joplin users who have many-notes (I am not sure how many is that) and Joplin users who have relatively fewer notes might be dealing with a very different user-experience universe. I am wondering if this is something to be considered about.

1 Like
31

The issue is experienced on the Android app generally on the initial sync when you have a bunch of notes to be sync'd, and is not specific to OneDrive or any providers besides Joplin Cloud. Since you apparently experience your issue on Windows, it is not related. You can search the forum for "slow sync on Android" or something to check posts on that matter.

I'm not a native speaker but I would agree.

32

@SteveShank

We are already talking about the initial synchronization. There is not much of an issue with Onedrive in subsequent synchronizations. But let's say you've been using Joplin for 2 years and have around 30,000 notes. Have you thought about how you would do the initial setup and synchronization on a freshly formatted or newly acquired phone and how long (!) it would take? That's why this topic was brought up.

@KC_20CY

You're absolutely right. I wish the commenters would also mention how many notes they have. If I had 100 or 500 notes, I would also be a very satisfied user! So, using Joplin along with this issue is like using demo software: "Are you satisfied with the features? Then buy Joplin Cloud!"

@Holzk

I'm currently trying Syncthing. It's really slow on Android. The only positive point is that you can see the synchronization status and how much is left in the Syncthing app. But not being able to escape the "slow synchronization" nightmare except for Joplin Cloud and WebDAV is really frustrating. @dpoulton mentioned that I could contact Onedrive and Dropbox about this. But wouldn't it be more logical for @laurent , who is aware of the issue, to contact them and even Google about restrictions on Android? I think it would be much more effective for the developer of a famous application to do this.

A word can have both positive and negative connotations simultaneously, including in English and many other languages. Manipulation means intentional and covert direction. I meant this in my question. As you can see from the title, I'm not putting this forward as an accusation. I'm asking a question.

33

I am more concerned about how Joplin design people handle this during their default design environment. I have 25075 notes BTW.

1 Like
34

Yeah, Syncthing is going to be slow because of how Android deals with (or restricts) accessing thousands of files located in the same folder. If you sync just a few large files, they will download super fast. The slowness can be remedied a little by using beefy hardware, but still, I'd assume that the initial sync of 11,000+ notes will likely take a few days. It does get better after that, once the notes are already there and only new changes need to synced.

1 Like
35

Thanks. But to repeat, if Joplin Cloud can solve this problem, isn't it possible for other service providers to take steps forward despite Android's limitations? Additionally, we're talking about a much more flexible operating system than iOS, such as Android. Perhaps even a simple root permission would be enough to bypass Android's default restrictions.

36

Not really possible even with root. You would need to modify the Android source code itself, and then compile your own version of the OS. Also, Syncthing can already run as root but it doesn't change anything. For the record, this slow file access through FUSE in Android is a rather new thing. If you use Syncthing in Android 10 or older, then the sync is going to be extremely quick.

Please keep in mind that this applies to file system sync via Syncthing specifically. The slowness of other services like OneDrive, Dropbox, etc. is not the same issue.

37

These are very valuable pieces of information. I'm currently using Android 10. Unfortunately, the slowness issue persists. I believe you were referring to Android 9 and earlier.

We can also consider that cloud services like Syncthing and Onedrive may face similar issues. However, if Joplin Cloud can overcome this, it could be a solution for these mentioned applications as well. After all, Joplin Cloud is also an open-source project. If our options as users are limited, then we should see developers taking a more active approach to addressing this issue right?

unknown_2024.04.10-15.23
unknown_2024.04.10-15.23796×702 39.1 KB

Back to the 90s...

38

You're right, it's apparently Android 9 and older. I've got one Android 7 device still in use and the difference is not even funny, e.g. it may take 10 minutes to sync 10,000 files there, while on a new Android 14 device the same operation will take at least 10 hours, if not much longer.

1 Like
39

To be fair, last time I checked the native code that deals with the file system sync in Joplin was far from optimal. This was a while ago so it's possible things have changed since.

40

This is exactly what I'm talking about. I hope those who write "...but open-source bla bla bla" on the topic will read this. I bet with you, Roman, that nothing has changed with it. But I can't prove it...