Attachments are inflated on sync

I have enabled file system sync and had it point to my Proton Drive folder. I have also enabled encryption. And to my surprise, the size of the resources subfolder is around 1GB.

I looked into the Note Attachments table within Joplin, and found a suspicious inflation of my attachments. According to this table, my largest attachment has 35 MB being a 7z archive. I checked its id and found the file in my synced folder. There, it has 60 MB.

If I export everything into a .jex file, that file has only 300 MB.

Is this the side effect of encryption?

yes, it is

This answer is as true as it is concise, but I think this topic is worth discussing. I don't know if slant.co is a reliable source, but here it says that Joplin is known for its "gargantuan memory footprint" :rofl:

I'm not an expert at all in these things, but could it be that there are other implementations of end-to-end encryption which use considerably less resources in terms of memory?

I have looked further into this last night and found some interesting things. I closed all the tabs, which were mostly various comments here on the forum, but I will try to be as accurate as possible without them for anyone stumbling upon this in the future.

First off, the .jex file appears to be a .tar archive, according to my 7zip, which implies some level of compression, thus it will be smaller than whatever you have in your joplin-desktop folder. But, there is more.

Joplin doesn't seem to be handling attachments (files that you import, like images) very well. Two weeks ago, I dragged and dropped a .bmp image into joplin, only to find out that the md renderer doesn't render .bmp. I deleted the file by deleting the line in md editor, converted the image from .bmp to .jpg and uploaded again. But this .bmp image persisted and was visible in attachments list (found under Tools -> Note Attachments). The .bmp image was not used in any of my notes, but it was still an attachment, thus being orphaned.

When exporting to .jex, resources, aka attachments, get exported only if they are in one of the exported .md files. In other words, .jex export skips orphaned attachments. You could export your .jex file, delete everything in your joplin, including attachments, and re-import .jex file to get rid of orphaned attachments. Or use a script to do this for you, called joplin-vacuum, which is what I used.

That trimmed down my joplin-desktop folder from 1GB down to 500 MB, which is a lot closer to the 300 MB .jex file export.

This happens on a timer and only once your note history has expired. In other words, if you disable note history, then the old attachment should automatically be deleted in 24 hours or so. I may be wrong on the exact time frame as I haven't dug into the code, but this is the general mechanism that deals with deleted data.

Please find below a "brain-dump" of the information I have found on this forum about Joplin as well as what I have discovered over the years!

A JEX export is the same data as the "RAW" export but wrapped into a single TAR file. TAR files are not compressed. That is why you will often see files online that are filename.tar.gz, which means that the source files have been combined into a single TAR file and then GZipped to compress it.

A JEX or RAW export exports just the "live" notes and data. Note history and orphan resources / attachments are not included. The Resources folder in a JEX file is very likely to be smaller than the source folder on the system.

If you delete an attachment from a note and no other note has that attachment linked to it you have an "orphan" file. How long that file stays stored in Joplin depends on how long your note history is set for. If your history is set to 90 days it will take 90 days before that unused attachment is marked as being suitable for deletion. Then after that it should get deleted when Joplin next runs its routine to clear out files set for deletion.

There is a difference between memory (RAM) and storage (disk space):

RAM - The "gargantuan" memory footprint referred to is possibly due to Joplin being an an app developed using the Electron framework. Electron is known for being particularly resource heavy. I guess that Electron is used because it enables the app to be developed for several different platforms at the same time.

Storage - When a user is using End to End Encryption (E2EE) the notes and the files in the Resources folder are sent to the sync target encrypted. I have seen that an encrypted version of a file in the Resources folder will be almost twice the size of the source. You can see the size difference by looking in the Resources folder and comparing a file with its .crypted counterpart. However it appears that these are basically cache files and no longer needed after they are transferred. @laurent has stated that if Joplin is closed down they can be deleted.

Apparently, in the past there was a reason for keeping the .crypted files in place and not deleting them, but that reason is no longer all that valid. In fact there is an issue on GitHub to develop a method for Joplin to delete the .crypted files when it starts.

So if / when this is developed the joplin-desktop/resources folder will start cleaning itself of .crypted files automatically.

2 Likes