Why tens of thousands of resources for ~1000 notes?

Operating system

Windows

Joplin version

3.3.13

Desktop version info

Joplin 3.3.13 (prod, win32)

Device: win32, 11th Gen Intel(R) Core(TM) i7-11370H @ 3.30GHz
Client ID: b9515dd599f4448ebe92aa121ec1e9b0
Sync Version: 3
Profile Version: 47
Keychain Supported: Yes
Alternative instance ID: -

Revision: 144ed59

Backup: 1.4.3
Freehand Drawing: 3.0.1
Quick Links: 1.3.1
Quick Move: 1.1.0
Search & Replace: 2.2.0

Sync target

Joplin Cloud

What issue do you have?

Dear all,

I am truly very grateful for your work on all aspects of Joplin, the programming and constant updating of the main program, the plug-ins, your support and responsiveness here and elsewhere. A few days ago, I subscribed to Joplin Cloud.

Now I have a two-part question which bothers me for a while.
I do not understand why Joplin synchronization takes so long, involves so many "items", files and resources (what's the difference, anyways?), and why it fetches and updates constantly thousands resources even when I have not made any edits or other changes at all.

I have only around 1000 notes, and most of them do not have attachments, afaik.
But the number of resources involved in each sync, and also between syncs, even exceeds by far the number of files contained within a jex Backup-export, and that number (almost 8000 md files and more 5000+ additional files in the resources-folder) is already a multiple of the number of notes. Last time I synced to a new device, more than 30.000 resources where fetched.

What is going on here? Does Joplin keep track of older versions of my data before reorganizing things? Or is there another reason for so many files and resources? More importantly: If this is not normal (for some weird reason), how can I get Joplin back to a more normal behavior?

The other aspect of my question that bothers me is that Joplin even when NOT asked to synchronize is constantly up- or downloading resources, or "updating" resources locally or on the server, even when I have not made any changes in hours.

These two things may be related.

As far as I can tell, I have seen this behavior before, when I was still using WebDAV as my sync target.

Thanks in advance for any advice (or at least an explanation).

I attach two screenshots of my tree of notebooks which documents that I have just above 1000 notes, and the high number of "items", files and resources fetched, created and/or updated when synchronizing and between syncs (or is the moment of an automatic sync? Why is the "Sync now" button active, then?).

I am sorry if my question has been answered elsewhere, and am happy to be pointed to relevant threads, FAQ-items or wherever.

Screenshots


As you've already noticed, one note can correspond to multiple MD files, so naturally, the number of notes and the number of MD files will not match (and this is also one of the reasons why you should not modify MD files directly in the sync target).

However, I'd say that the main culprit in this case is likely note history. Especially when set to a longer time period, note history will generate a lot of MD files per note (and also resource files). If you want to reduce that number, you need to either decrease the history time, or disable it altogether.

Please also keep in mind that if you do change your note history settings, it will take Joplin some time (a day or so?) to actually delete the obsolete files from the disk.

1 Like

Hello,

You can see a more detailed breakdown under Help => Sync status

Usually it's either note revisions, or resources when you have a lot of notes clipped from websites

Great, thanks, this explains a lot.

There are "only" 1300+ "revisions" (this refers to earlier versions in the note history, I assume?), so that does not to be the main culprit, depending on what each revision entails.

I do have many tags, though, which adds some 6000+ to the number of "items".

I will experiment with switching the note history off for a while and see if things improve in terms of number of files and sync time.

Thanks again!

I would have given this also a "solution" tag, but I can only give one, and had given it already to Laurent's answer...

Great, this is what I needed to at least understand. It explains the difference between notes, files/resources and "items", and helps me t figure out where so much complexity comes in. Thanks again!

I reduced the days to keep revisions down. I find 7 sufficient. In fact, I'm sure 2 would be sufficient, since the problem is screwing up, seeing my mistake, and getting a good copy from history. I've never wanted to get something more than 3 days old, so I feel 7 instead of 90 is fine.

If space is not too much of an issue it can make sense to keep revisions for a few weeks or even months.

What can happen is that you may accidentally modify a note, then many days later you open it again and realise some content you expect to be there is gone. With the note history you can restore it, but if you kept a low interval that might be gone.

Personally I have it set to 9999 days since I don't mind about the extra sync items.

I suppose, but this isn't a problem I can remember facing. But to protect against it, without taking up huge amounts of space, it would be good to have a number of previous versions option, rather than time based. I'd also be happy with 3-5 previous versions of a note, rather than collect everything. That solves your problem.

One thing I do is keep a note called daily notes. Most days I add little notes for during the day, like marking the time I start working on something for a client, or notes I'm taking on a phone call. After a few months, I wipe out the old notes, more than a month old. Since I'm changing this note every day, often 5 times, I ended up with hundreds of useless revisions with the longer time period.

Personally, I'd also love to have this option, or perhaps a combination of both. The reason is that I've got some notes that are edited very often. If I set note history to a longer time period, I end up with a large number of additional files to sync that I don't really need (which is a big problem especially on Android with slow filesystem sync). On the other hand, I've got some notes that are rarely edited, yet it would still be nice to store at least some of their previous versions. A compromise between the two approaches isn't possible currently.

This would be solved for the most part by PR Desktop, Mobile: Fix historic issue whereby the first revision created for a note does not contain the original contents by mrjo118 · Pull Request #12674 · laurent22/joplin · GitHub , as long as the note has not been edited for over 7 days. But it might be worth decreasing the old note period, to be maybe 1 day, as part of that change, to offer more protection