Synchronization issues: sync interval timer is being ignored, and sync causes app unresponsiveness

Support
1

Operating system

Windows

Joplin version

3.7.10

Desktop version info

Joplin 3.7.10 (prod, win32)

Device: win32, 11th Gen Intel(R) Core(TM) i9-11900K @ 3.50GHz
Client ID: a504afc6d2a6496b99a0931c22649537
Sync Version: 3
Profile Version: 53
Keychain Supported: Yes
Alternative instance ID: -
Sync target: Dropbox
Editor: Rich Text

Revision: 0d5d4c4

Backup: 1.5.1
Combine notes: 1.2.3
Favorites: 1.3.2
Freehand Drawing: 4.3.0
Note list (Preview): 1.1.0
Test Plugin: 1.0.0

Sync target

Dropbox

Editor

Rich Text Editor

What issue do you have?

This is sort of a two point report, but they are semi-related.
My issue is with the triggering of the Sync interval. I have my Sync set to happen every 10 minutes. However it seems after I do anything (like move a note, tag it, or update a note and close it) the sync will kick off regardless.

Having it aggressively sync like that is nice, even though it's technically ignoring the interval setting. However the problem arises in that everytime Joplin kicks into sync, the application begins to hitch, lag, and become unresponsive in various brief moments.

Something that wouldn't be that big of a deal if it happened say, every 10 minutes. But it happening after every edit of a note, say when you're going through and editing/moving/cleaning-up a bunch of notes, it becomes pretty insufferable and makes the desktop app feel very clunky as a result.

Right now I'm primarily assuming it's the sync because I've noticed whenever Joplin begins becoming unresponsive, a sync is occurring/finishing.

I haven't attempted to disable sync yet to see if that improves anything (I'm still getting acquainted with Joplin so I don't know if setting Sync to None and back to Dropbox would somehow make it want to re-sync my entire database again, a full day process)

If the sync after every note edit is a desired behavior, it does feel like it should be an optional setting that could be disabled in Settings, as it is actively hindering my work and frustrating to fight with the responsiveness lag it causes.

2

If you disconnect your internet and then make the same kind of changes in Joplin, does it still lag? If it does not lag, then that would indeed point to the sync being the cause

3

Great idea. I disconnected from the internet and proceeded to continue tagging, moving, editing notes for about 5 minutes, and the random bouts of unresponsiveness definitely cleared up while being offline. That certainly seems to point to the sync containing the cause.

4

Does the issue occur during sync when changing the text on 1 or 2 notes, or only when making bulk actions?

Also, any chance you could open the performance tab of Windows task manager and observe the peak ram usage of Joplin while the lag occurs? And also specify the base ram usage when Joplin is idle.

5

I can cause it to happen simply by creating a new note and then selecting away from it. Or selecting that new note (or any note really), typing some quick text garbage into it, and selecting away.

Trying to observe from the Performance tab didn't tell me very much unfortunately, so I observed the memory in the Processes tab, since that will include all the various Joplin processes that are running.

Joplin takes up 546.5 MB when idle, and maybe jumps up to 561.5 MB when syncing (though this is muddied in that I have to navigate AWAY from the newly created note, and if I navigate to another note with images in it that clearly bumps the memory on its own, so I attempted to measure this by navigating away, waiting for the sync to start, and navigating back to the new note)

Anyways, nothing super outrageous in terms of memory use, that I can tell. With 64GB I have plenty of ram free almost always, at least 37GB available.

6

Hmm ok. I've never been able to reproduce these freezes, but always suspected the issue to be memory usage. It seems clear from your tests that the sync is the cause, but no memory spikes suggests that memory usage is not the problem. I'm not really sure what in particular about the sync could be causing the issue now.

7

Once the :clockwise_vertical_arrows: symbol in the Synchronize button begins spinning, that is when the general unresponsiveness begins. It's most noticeable when clicking around on various notes, and there can be a 1-2 second delay before the clicked note is selected.
Once the :check_mark: re-appears in the Synchronize button, Joplin goes back to being appropriately responsive.

8

I have some other questions and tests:

  1. Do you have end-to-end encryption enabled in Joplin?
  2. Are most of your notes still html notes, which you're looking to convert to markdown?
  3. Does the lag occur when changing or switching to large notes only, or it happens with small notes too?
  4. If you create 2 new notes with just 1 line of content and synchronize, if you make further changes to one of the notes and then switch to the other of those 2 notes, do you still get lag when the sync runs with internet connected?
  5. Above the Synchronize button, click the arrow so that it is pointing downwards. This means the details are expanded. After making a change to one note, for a scenario for which you are experiencing lag, please can you await synchronization to trigger, and record how long it takes for the following to occur: 1. How long the button spins for until the details show a counter of items being changed. 2. How long does that sync take to finish, i.e. for the button to stop spinning and show a tick?
9
  1. Just checked, apparently I do not!
  2. Most certainly yes. When I imported my Evernote db I did so as HTML, and there really was no way around this. I am converting some simple notes to Markdown on a case by case basis as I encounter them, but in general I expect the vast majority to stay as HTML.
  3. Lag happens regardless of any note size.
  4. Yes, this is actually exactly how I was testing to see if the Sync itself was causing the lag.
  5. In this single attempt it took about 7 seconds. I timed it myself separately to confirm as welll. In general I would say I usually see this number fluctuate between 7-9 seconds.
    image
    image540×174 7.26 KB
10

How long would you say it takes before it says 'Created remote items'? And does the lag appear before that message shows, or only after it shows?

11

I would say it's 1-2 seconds before Created remote items appears.

The lag appears to occurs even before the Created message appears. However the lag greatly intensifies towards the end of the sync when the :clockwise_vertical_arrows: stops spinning, just right before the checkmark appears.

I'll see if I can setup a situation to capture a GIF of it using ScreenToGif

12

GIF of it in action.
JoplinSyncLag

I should note I performed this after the Test Notes in this had already been sync'd, so in the GIF it is only syncing the 2 edits made

(Oh.. heh, sidenote, I also realize now in the GIF it apparently just discarded my edits, but sync'd anyways??? I'm using the 3.7.12 prelease. But that's a bug for another thread.)

13

I actually just fixed that bug. Will be fixed on 3.7.13 when it is released.

The gif was quite helpful, thanks. I assumed lag meant that the whole UI froze, but actually the sync indicator still spins as normal and it looks like its the IPC event handling which is freezing up (not allowing you to switch note), which continues for a bit even after the sync has completed.

14

Excellent, major thanks! :folded_hands:

Ah that's very good point. I hadn't thought to test other aspects of the interface for responsiveness.
If I perform the same sort of experiment with selecting over and over during sync open various parts of the File | Edit | View | etc. menu at the top, I do not see any noticeable lag with the menus responding.
If I perform the same experiment with selecting between different notebooks in the leftside navigator, I do see some of the lag present.

15

@skullthug I've observed something from your gif and your OP. You're using the Note list (Preview) plugin and have the preview list view enabled. Potentially this could be the cause of the lag in combination with your large dataset. Could you test whether the lag still occurs if you disable this plugin. If not, could you also try with all plugins disabled?

16

Good point.
I performed the same test with all plugins disabled and can confirm the lag still occurs.

17

@skullthug Thank you for your contributions assisting with this issue! I have found a root cause and raised an issue on GitHub: Long running SQL queries should not block execution of other SQL · Issue #16291 · laurent22/joplin · GitHub

Fixing this may not be so simple, however.

18

Great, thank you for diving further into that and for elaborating on the cause. This issue is one of the top 5 hinderances I'm encountering as a recent convert to Joplin.
Optimizing or multi-threading any form of SQL is unfortunately not in my skill set, but if there's anything further I can do to help out somehow, please do not hesitate to ask!

19

Also, since the issue might not be super easy to solve, I'll bring up this point again

If the auto-sync after the editing of a note could possibly be a Setting that I could disable, that would likely go a very long way to reducing how much this bug interferes with my regular Joplin use. It doesn't fix the problem, but it at least has the opportunity to reduce how much it occurs until a fully proper solution can be found.

20

I agree and this is exactly what I've been saying in another discussion about the same issue but caused in my case by long notes: give users a way to fully disable sync and/or until such time these tricky issues can be fully resolved.

Personally I believe that the freedom to able to fully disable sync (ie no auto sync) - ie sync only when I want to sync - is very much part of what I'd expect from Joplin and it's open ethos. See my posts here - Cannot Stop Synchronisation - #5 by mrjo118