How to survive with a large dataset on Joplin (10k notes, 20k resources, 10GB)

_vg · 2024-09-19T21:31:25.295Z

Here is my advice based on my one year experience, if anyone interested.

Context: Android and MacOS devices, former Evernote fan (2009-2023).

Chances are, you only need a little fraction of your 10k notes most of the time. The main trick to fix Android poor performance is to store 95% of your notes in a big "archive" notebook and almost never access it. Accessing your small notebooks will work well. Accessing your big notebook or "All notes" will be extremely slow, avoid it as much as possible.

Next is synchronization. So many choices, let me save you some time.

On mobile, when setting up the initial sync, absolutely change "attachment download behavior" from always to auto.

Use Joplin Cloud with end to end encryption. Don't use Dropbox which is way too slow for many notes.

The initial sync will take several hours per device and you can only do one device at a time. I suggest to start one device everytime you go to bed and let it complete overnight.

Everytime you setup Joplin on a new android device, the full sync will block all your other devices for hours. Thus do it overnight.

To experiment with various datasets and synchronisation strategies, use Joplin profiles, available on both desktop and mobile, this is an awesome feature to experiment things without breaking your main dataset. You can test the total sync time of a different strategy with a copy of your full dataset this way, without commiting to actually doing it.

The only other option to Joplin Cloud is hosting your own Joplin Server. I am currently going this route but this is for advanced users only. It has advantages: faster sync, E2EE no longer needed, initial full sync no longer blocking thus you can full sync all your devices in parallel.

Syncthing is a great tool but unfortunately won't work for 10k notes because Android Joplin app is not able to offer a usable experience accessing a folder with 10k files, I tried very hard to make it work and failed.

laurent · 2024-09-19T22:01:54.038Z

_vg:

the full sync will block all your other devices for hour

There's an option now to turn off sync locks under Sync => Advanced settings. Sync locks will probably be removed for good from the next version. They were implemented for the sync migration mechanism but we don't do this anymore because it causes too much troubles, so the sync locks are not longer needed. That should also make sync faster to start since it doesn't need to do the lock checking dance.

_vg · 2024-09-19T22:09:30.997Z

I don't see this option yet on Joplin Android 3.0.8 but I'm very happy to hear it's coming! Also happy to hear that sync locks will soon be history.

ymzk64 · 2024-09-21T04:44:44.785Z

Ask out of curiosity

If multiple devices edit the same file at the same time, will it be damaged?
If only half of the resource file is uploaded, will other devices download unusable files?

Thank's

ymzk64 · 2024-09-21T04:48:52.202Z

With my own use case
I have a very low performance tablet and it has problems loading all the data.
So I built a JoplinServer on my Raspberry Pi
Then I synchronized all my data to the main account, and then created an empty clone account for the tablet. Then I shared some notebooks from the main account to the clone account from the computer.
In this way, the tablet computer will be very clean, with only the files and data needed on the tablet computer.

dhwalker · 2024-09-21T06:23:45.712Z

Very interesting discussion! I am also an ex-Evernote user (2014-2021) with earlier notes from other note-taking applications and services. I sync with Joplin Cloud. I've dealt with my large dataset by exporting most notes that haven't been updated for more than a couple of years to JEX files and importing them into a second profile that is sync'ed only on devices that are likely to need those old notes.

Some thoughts...

Sync'ing a new device is considerably faster, and importing the JEX files doesn't block other devices. I've still needed to install the Caffeine app to keep the phone from going to sleep during what's still a multi-hour initial sync.
It's inconvenient to switch profiles when I need one of those old notes. It would be nice to have both profiles accessible at the same time, but admittedly it's rare to need the old notes.
The ability to remove the locks makes keeping everything in one (sync'ed) profile more attractive. Enabling background sync on Android (so I wouldn't care much if the initial sync took a few days) would clinch things.

laurent · 2024-09-22T11:54:54.140Z

It's actually in version 3.1 (available as a pre-release). I had sync locks turned off for a while on all my devices without any issue and at a theoretical level I think it's also safe to get rid of them.

laurent · 2024-09-22T11:56:50.946Z

ymzk64:

If multiple devices edit the same file at the same time, will it be damaged?

No, conflicts will be generated as normal if the apps can't resolve them automatically.

ymzk64:

If only half of the resource file is uploaded, will other devices download unusable files?

Upload is an all or nothing - if it fails in the middle then nothing will be on the server so there can't be half resources on it. The app will simply try to upload it again at a later time.

_vg · 2024-09-22T14:45:56.258Z

dhwalker:

exporting most notes that haven't been updated for more than a couple of years to JEX files

Be very, very careful about that. I've learned the hard way the painful lesson that a JEX regenerates all note IDs. Links between notes of the same JEX are preserved but links between any note of the JEX to another note outside of the JEX is broken forever.

I know they are technical reasons behind this controversial choice so I will not argue to change it. Just stating my experience and opinion, which is that by default note IDs should always be preserved, and only regenerated as an option. If ID conflict happens when importing a JEX, show a big warning and offer to regenerate IDs then and only then. Imho note IDs should be eternal by default.

dhwalker · 2024-09-22T17:47:29.633Z

Thanks. I had figured it was likely I'd mess up inter-note links, particularly since the old notes would be in a different profile, but I don't do too much linking and decided to go ahead.

The news about the sync locks is encouraging, though. I think I'll be merging everything back into a single profile.

AndyL · 2024-09-24T22:24:39.334Z

Syncing a new Android device against Joplin cloud with ~15k notes and over 10gb total I found the only way to get the sync to complete is to prevent the Android device from locking with Caffeine.

I had to leave the device to sync overnight and plugged in to a charger, but the the sync completed without issue.

As you indicate; I find the best attachment setting is "auto".

antonio-slog · 2024-10-01T20:54:56.502Z

I'm using joplin with 3000+ notes that require more than 10GB space.
On my android phone I set the "attachment download behavior" to ALWAYS, not auto.
This had a cost: I had to let the phone on overnight.
But now every attachment is ready to open at wish.
So, when I need a resource in a hurry, I have it at hands without waiting for infinite download time.
I use webdav sync with a really old/slow QNAP nas with slow connection to the web at home (I'm waiting for starlink or fiber connection). So, downloading some 5MBs is a pain for me.
Since Joplin is the most used app on my phone, I don't mind for the space it requires.
My 2c.
Tony

jpservices · 2024-11-11T18:07:53.537Z

Thanks for sharing. Good to put those tips and tricks in one place for those having a large set of notes.
I also migrated from Evernote in January when the situation was particularly bad at Evernote. I tried different tools and stumbled on Joplin which I link very much.
Because of the sync time and at the time much too slow end to end encryption I finally kept one private account on Joplin and Joplin cloud without encryption and continued with Evernote for more work related stuff. In the meantime Evernote got their act together and is working almost perfect but they will probably not do end to end encryption. The different changes at Evernote also raises a flag to how the data is used.
So I checked once more in the past week how things would run transferring my 4000+/10gb work related notes that have a fair amount of attachments and pictures.

I am using Joplin Could and E2EE. Got the ENEX on a MAC into Joplin, activated the E2EE on the Mac and then let it sync with an Android device (samsung galaxy tab 9 on Android 14). After more than 24hours all is fine and notes converted fine (except for the untitled notes bug that Evernote will probably never fix). All notes and attachments are also decrypted on the mobile.

Then I tried to create a few new notes of different size and timed it roughly:

for a text note no problem
for 1.1 MB note with pictures this is about 10s decryption. This is OK.
for a more typical note with a 12MB attachment this is about 2minutes. Marginally acceptable
with a note with 2 attachments 26MB + 13MB. 7 minutes (with the first attachment ready and decrypted on the Android after about 5 minutes. This is much too much

The typical use case at work is that I prepare a note on my PC, put the relevant documents as attachements (say for a review), read the attachments on the PC and add comments and then rush for a meeting room. During the "transport" I normally start my note taking (Evernote today) and sync on the way on my mobile. This is why 2 minutes is OK and 7 not.

So to me E2EE is still not working fine enough (it seems better than 9 months ago though).

So how to proceed?
a) could it be possible to have an encryption that is significantly faster at the cost of a lesser security?
b) could the decryption occur with priority on the note curently open and then the document to be opened?
c) could decrypt and sync happen in the background? I know: sync in the background on mobile has been painful but it would be so much more user friendly ...
d) could the encryption be activated at the notebook and not profile level? Having notes spread on multiple accounts and profiles is not quite what i want.?
e) can the decryption code be further optimised?

Thanks

personalizedrefriger · 2024-11-11T20:20:42.823Z

jpservices:

e) can the decryption code be further optimised?

The version 3.2 pre-release has an in-beta encryption method that should be faster. See this comment in the release notes for details.

jpservices · 2024-11-12T18:19:57.442Z

Sounds good. Thanks. Do you know what is the expected time reduction on Android?

Thanks

Wimvan · 2024-11-12T19:05:41.983Z

A lot faster if everything works as expected:

GSoC 2024: Native encryption support

Now the new methods have higher performance. The encrypted files and encrypted master key is smaller, and encryption speeds are significantly improved, with up to 8x faster performance on mobile platforms and up to 16x faster on desktop platforms. None of these improvements sacrifice security, and the new methods are actually more secure. Anyone can review the design by checking the codebase and my final report.

jpservices · 2024-11-13T17:40:14.517Z

Thanks. This sounds fantastic. Thanks again.