How to survive with a large dataset on Joplin (10k notes, 20k resources, 10GB)

Lounge
1

Here is my advice based on my one year experience, if anyone interested.

Context: Android and MacOS devices, former Evernote fan (2009-2023).

Chances are, you only need a little fraction of your 10k notes most of the time. The main trick to fix Android poor performance is to store 95% of your notes in a big "archive" notebook and almost never access it. Accessing your small notebooks will work well. Accessing your big notebook or "All notes" will be extremely slow, avoid it as much as possible.

Next is synchronization. So many choices, let me save you some time.

On mobile, when setting up the initial sync, absolutely change "attachment download behavior" from always to auto.

Use Joplin Cloud with end to end encryption. Don't use Dropbox which is way too slow for many notes.

The initial sync will take several hours per device and you can only do one device at a time. I suggest to start one device everytime you go to bed and let it complete overnight.

Everytime you setup Joplin on a new android device, the full sync will block all your other devices for hours. Thus do it overnight.

To experiment with various datasets and synchronisation strategies, use Joplin profiles, available on both desktop and mobile, this is an awesome feature to experiment things without breaking your main dataset. You can test the total sync time of a different strategy with a copy of your full dataset this way, without commiting to actually doing it.

The only other option to Joplin Cloud is hosting your own Joplin Server. I am currently going this route but this is for advanced users only. It has advantages: faster sync, E2EE no longer needed, initial full sync no longer blocking thus you can full sync all your devices in parallel.

Syncthing is a great tool but unfortunately won't work for 10k notes because Android Joplin app is not able to offer a usable experience accessing a folder with 10k files, I tried very hard to make it work and failed.

7 Likes
2

There's an option now to turn off sync locks under Sync => Advanced settings. Sync locks will probably be removed for good from the next version. They were implemented for the sync migration mechanism but we don't do this anymore because it causes too much troubles, so the sync locks are not longer needed. That should also make sync faster to start since it doesn't need to do the lock checking dance.

9 Likes
3

I don't see this option yet on Joplin Android 3.0.8 but I'm very happy to hear it's coming! Also happy to hear that sync locks will soon be history.

4

Ask out of curiosity

  1. If multiple devices edit the same file at the same time, will it be damaged?
  2. If only half of the resource file is uploaded, will other devices download unusable files?

Thank's

5

With my own use case
I have a very low performance tablet and it has problems loading all the data.
So I built a JoplinServer on my Raspberry Pi
Then I synchronized all my data to the main account, and then created an empty clone account for the tablet. Then I shared some notebooks from the main account to the clone account from the computer.
In this way, the tablet computer will be very clean, with only the files and data needed on the tablet computer.

6

Very interesting discussion! I am also an ex-Evernote user (2014-2021) with earlier notes from other note-taking applications and services. I sync with Joplin Cloud. I've dealt with my large dataset by exporting most notes that haven't been updated for more than a couple of years to JEX files and importing them into a second profile that is sync'ed only on devices that are likely to need those old notes.

Some thoughts...

  • Sync'ing a new device is considerably faster, and importing the JEX files doesn't block other devices. I've still needed to install the Caffeine app to keep the phone from going to sleep during what's still a multi-hour initial sync.
  • It's inconvenient to switch profiles when I need one of those old notes. It would be nice to have both profiles accessible at the same time, but admittedly it's rare to need the old notes.
  • The ability to remove the locks makes keeping everything in one (sync'ed) profile more attractive. Enabling background sync on Android (so I wouldn't care much if the initial sync took a few days) would clinch things.
1 Like
7

It's actually in version 3.1 (available as a pre-release). I had sync locks turned off for a while on all my devices without any issue and at a theoretical level I think it's also safe to get rid of them.

8

No, conflicts will be generated as normal if the apps can't resolve them automatically.

Upload is an all or nothing - if it fails in the middle then nothing will be on the server so there can't be half resources on it. The app will simply try to upload it again at a later time.

1 Like
9

Be very, very careful about that. I've learned the hard way the painful lesson that a JEX regenerates all note IDs. Links between notes of the same JEX are preserved but links between any note of the JEX to another note outside of the JEX is broken forever.

I know they are technical reasons behind this controversial choice so I will not argue to change it. Just stating my experience and opinion, which is that by default note IDs should always be preserved, and only regenerated as an option. If ID conflict happens when importing a JEX, show a big warning and offer to regenerate IDs then and only then. Imho note IDs should be eternal by default.

3 Likes
10

Thanks. I had figured it was likely I'd mess up inter-note links, particularly since the old notes would be in a different profile, but I don't do too much linking and decided to go ahead.

The news about the sync locks is encouraging, though. I think I'll be merging everything back into a single profile.

11

Syncing a new Android device against Joplin cloud with ~15k notes and over 10gb total I found the only way to get the sync to complete is to prevent the Android device from locking with Caffeine.

I had to leave the device to sync overnight and plugged in to a charger, but the the sync completed without issue.

As you indicate; I find the best attachment setting is "auto".

12

I'm using joplin with 3000+ notes that require more than 10GB space.
On my android phone I set the "attachment download behavior" to ALWAYS, not auto.
This had a cost: I had to let the phone on overnight.
But now every attachment is ready to open at wish.
So, when I need a resource in a hurry, I have it at hands without waiting for infinite download time.
I use webdav sync with a really old/slow QNAP nas with slow connection to the web at home (I'm waiting for starlink or fiber connection). So, downloading some 5MBs is a pain for me.
Since Joplin is the most used app on my phone, I don't mind for the space it requires.
My 2c.
Tony

1 Like