SOLVED: Sync never ends, causes resources to balloon in size

Support
1

Operating system

Linux

Joplin version

3.5.12

Desktop version info

Joplin 3.5.12 (prod, linux)

Device: linux, AMD Ryzen 5 2400G with Radeon Vega Graphics
Client ID: 484adccee2c04b29939de021c183ef4d
Sync Version: 3
Profile Version: 49
Keychain Supported: No
Alternative instance ID: -

Backup: 1.5.1
Freehand Drawing: 4.2.0
Link Icon: 1.0.0
Make All Links: 1.0.3
MultiMarkdown Table Tools: 1.2.1
Rich Markdown: 0.17.1
Victor: 1.0.3

Sync target

Joplin Server

What issue do you have?

Since upgrading Joplin Server last night, the sync operation never ends. I have had a couple of hundred conflicts with attachments (all the same note, containing one single image, scanned via the "Smart notebook" function), which I've deleted, but the resources folder keeps growing in size. I've also emptied the Trash folder multiple times.

The sync status keeps reporting hundreds and hundreds of "Updated remote items", and eventually appears to stop, only for the sync operation to start again seconds later.

Joplin Server is v3.5.2, running via Docker.

I have just 63 notes, but the resources folder is taking up 2.1GB, up from 500MB when I looked earlier today. Even 500MB is just crazy for the small number of notes, with very few pictures.

The issue is not unique to Joplin Desktop, the same problem manifested itself last night on Joplin Android, never ending synchronisation. I've abandoned that, hoping to let the situation sort itself out in the desktop application, but it just never stops.

The log file contains tens of thousands of entries (over 71,000!!), and is, at 12MB, too big to upload here. Just in the time I've been writing this, roughly 1,000 new lines got written to the log file.

Has anyone else seen this before? What is going on here?

2

Fwiw I did not meet this issue after upgrading my Joplin Server to 3.5.2.

When you sync from a new profile (either Desktop or Mobile) is the data you get the correct one? If yes, starting new profiles on all devices could solve your issue.

If not, if I were you, I would trash my server instance and start over. See How to fix synchronisation issues and start over

If you don't want to trash your server for some reason, you could try the Victor plugin to empty its content before reimporting your latest JEX backup.

1 Like
3

Thanks for helping out!

It is possible that the issue is unrelated to the server upgrade itself, it’s just that it manifested itself after the server upgrade. Maybe mere coincidence?

I managed to sort the problem out yesterday (I think). Not sure which of the following actions was the actual solution, but I:

  • disabled note history
  • kept deleting the hundreds of attachment conflicts until no new ones appeared
  • kept emptying the trash
  • manually deleted the attachments themselves using the Notes attachments tool, as well as their corresponding notes
  • waited for all these changes to be synced to the server.

This sorted out my desktop. I would have thought that all that clearing out would be replicated to my laptop, but that turned out not to be the case. Today I had to repeat the same steps on my laptop.

My phone had an insane amount of sync operations, in particular resources, and I have no idea where to manage Joplin resources on Android.

I currently have 4 users on my server, including myself. I’d rather not trash my server and go through the hassle of inviting them and getting them to accept the invitations for the shared notebooks. It’s not complicated, but sadly my users are the typical users who find everything “technical” and “complicated”.

How does nuking everything with Victor, then restoring a backup affect the shared notebooks?

And how does starting a new profile affect the shared notebooks?

Note: I use E2EE.

Edit: I just read the link you posted (thanks!), and will just delete and reinstall the Android app, as per the article.

4

Unfortunately (afaik) it nukes all published notes and shared notebooks, you have to setup them all over again. In your case you indeed want to avoid this route.

And how does starting a new profile affect the shared notebooks?

I believe it is very safe and does not impact them. Playing with different profiles is very convenient for troobleshooting, you can always go back to your old profile. Name your profiles well to avoid mistakes.

1 Like
5

I I had a similar problem earlier this week. Synchronizing my desktop client (Ubuntu 22.04) with Nextcloud suddenly took a very long time. Then I noticed that the Joplin data in Nextcloud had grown from about 400MB to over 950MB, even though I had only added and deleted a few notes. The exported data (.jex) is only 320MB in size (about 800 notes, some with attachments). I then used Victor to delete the data in the client and also emptied the folder in Nextcloud. Afterwards, I imported the exported data into the empty notebook in the client and then re-uploaded it using "Upload local data to the synchronization target again." Unfortunately, this took a very long time, and in the end, there were 950MB in Nextcloud again. When I looked at the files there, sorted by size, I noticed that, for example... All attachments (PDF, JPG, etc.) were always saved two or three times under different names. The number of entries in my notebook hadn't changed, but over 2800 notes were counted during synchronization. When I tried to synchronize with the client on my smartphone (Android), all notes and folders were displayed multiple times. Since this was all too frustrating, I deleted both the desktop client and the one on my smartphone, as well as all data, to reinstall everything, hoping that when I re-import the exported data, a clean upload to Nextcloud would occur.

Axel

6

I’m marking my own ‘fix’ as the solution, as it did solve the problem. I still seem to have a rather large Postgres database of 2GB.

Upgrading the Postgresql container from v14 to v16, and importing the dumped database appears to have brought that down to 1.2GB, but that still seem very big for the amount of notes I’ve got, with not many attachments. Is there any way to clear that out?