Joplin Server: Storage to filesystem instead of DB

Support
22

What I meant is that it’s kind of useful to have same understanding of terms, in order to avoid confusion.

To get an idea what are we even talking about, I’ve created simple bench tool, that writes certain number of files in a directory, then reads them, and finally deletes them.

Here are the results. I’ve tested FAT32&NTFS on my 8gen intel laptop and also ext4 on my qotom home server(Intel J4125/16GB RAM/mSata SSD)

First column is number of files used in test
Next 3 columns are times to create/read/delete all files in single directory in ms
Next 3 columns are times to create/read/delete per single file in ms
FAT32
10000 1225 6989 812 0.123 0.699 0.081
20000 4524 10909 1651 0.226 0.545 0.083
30000 7538 15628 2621 0.251 0.521 0.087
40000 14545 20458 3419 0.364 0.511 0.085
50000 19380 28139 4202 0.388 0.563 0.084
60000 28380 30734 5041 0.473 0.512 0.084

NTFS
30000 2577 1384 4736 0.086 0.046 0.158
60000 4926 2629 7720 0.082 0.044 0.129
90000 7081 3682 10715 0.079 0.041 0.119
120000 11027 4931 15767 0.092 0.041 0.131
150000 13905 7862 16614 0.093 0.052 0.111
180000 16164 9100 20642 0.09 0.051 0.115
210000 17193 10355 24114 0.082 0.049 0.115
240000 19276 11652 28936 0.08 0.049 0.121
270000 23165 11042 30183 0.086 0.041 0.112
300000 25662 12190 34959 0.086 0.041 0.117

ext4/qotom
30000 1375 749 745 0.046 0.025 0.025
60000 2518 1392 1540 0.042 0.023 0.026
90000 3805 2028 2243 0.042 0.023 0.025
120000 5142 2677 3274 0.043 0.022 0.027
150000 6483 3324 3679 0.043 0.022 0.025
180000 7932 3992 5239 0.044 0.022 0.029
210000 9627 4695 5968 0.046 0.022 0.028
240000 11147 5311 6719 0.046 0.022 0.028
270000 12504 5909 7540 0.046 0.022 0.028
300000 13686 7126 9403 0.046 0.024 0.031

With FAT32, results correspond with what you’ve mentioned, time to create a file increase with number of files already created, which is not good.

But as you can see, modern filesystems like NTFS and ext4 really do not have any issues with handling hundreds of thousands of files in single directory. Times to create, read or delete a file do not depend on number of files in directory.

It’s also interesting, that cheap lowend server has better IO performance than midrange laptop with much higher CPU performance.

23

Your benchmark actually highlights the problem, not the opposite. The reason FAT32 slows down is exactly why dumping everything into one folder is bad design. NTFS and ext4 mask it better with B-tree indexing, but the cost is still linear—you showed yourself that total wall-clock time grows steadily with directory size.

Real workloads like Joplin aren’t tight create/read/delete loops on hot caches. They do stat, glob *.md, scan timestamps, and hit the filesystem cold after a reboot. That’s when big single directories turn into multi-second stalls. The benchmarks you ran are the best case. Users experience the worst case.

24

Can you please stop with this off topic….

25

I apologize, but it seemed to me relevant to your issue. Now we know, that filesystems can easily 100k files in one directory or more.

I’ve generated and imported to joplin about 20k notes and tags. Sync to webdav on my qotom home server took about 12.5 minutes. I’ve observed that sync speed slows down somewhat with number of synced items, but it was not nearly as dramatic as you described it. Maybe getting better PC could help?

Joplin doesn’t seem to be very much optimized for large number of notes. Even when there are no changes in notes, repeated sync takes 7-8 seconds.

26

I think Core i9 with 32 GB RAM , 1 GBit uplink and PCI5 NVME should be fine:) I think thats good enough

27

If that’s the case, then you may want to switch from Joplin Server to plain webdav. I’m using Apache Tomcat as server and it can handle 20k files without breaking a sweat. Server load was in single digit percent all along, despite 10 other containers running on server as well.

28

no one said it can’t. that I said was that it’s inefficient for apps that access the data. Other software shard the files. One example is https://immich.app/

29

Well OK. If you’re convinced, that Joplin is going to benefit from file sharding, then give us some numbers. What operations are going to be faster and by how much?

30

I don’t need to convince you. And Laurent has already responded that he’s wanted to switch to this structure years ago, but it’s not been high priority, and it’s not an easy task due to data migration.

31

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.