Sync feature subtly broken? Some notes not syncing, sub-notebook structure differs between computers

[scribbler]

I'm running Joplin on two computers running Windows 10 - a laptop and a desktop. Both are running:

Joplin 1.0.201 (prod, win32)
Client ID: (different for each computer)
Sync Version: 1
Profile Version: 28
Revision: e65af8c1 (master)

They are synced through a Nextcloud account (which seems to be syncing all my other files just fine). I am not syncing to WebDAV as my host does not support it, I'm just using the File System as a target and Nextcloud is syncing that folder.

Unfortunately, I'm having some significant (but rather subtle and insidious) syncronisation issues.

For the first few months, things seemed fine. I inputted my data from my old evernote account (manually note by note, not with a file import) and started adding new notes in Joplin. There are probably one or two hundred notes at this point. Things seemed to be working great, apart from occasionally slow sync due to the 3-step sync process (Joplin syncs to local Nextcloud folder, which then syncs to nextcloud folder on other computer, and finally other Joplin client syncs with those files).

So far I have only had one file conflict, after I'd edited a file on both computers simultaneously, and it was nicely marked red in the notebook sidebar and I was able to manually correct the error.

However, in the last couple of weeks, I've started noticing some strange behaviour. Certain notes I wanted to add to were not present on the laptop, and my notebook structure seemed a little off. I thought the computers might just be out of sync or slow to sync, but things continued to seem off and I finally sat down with both computers, made 100% sure they were synced, and started comparing files.

The results were very worrying. While I have not found notes with conflicting content (yet), I have found that my notebook structure on laptop does not match what I have on desktop, and I have a non-trivial number of notes on desktop that do not exist at all on my laptop client. I do not know if the reverse is true, if certain parts of the laptop database do not exist on desktop, but I cannot rule it out.

I am quite shocked at this result. I cannot notice a pattern to the affected files. My only hunch (and it's pretty weak) is that it might have to do with moving sub-notebooks in and out of regular notebooks (with drag and drop in my case), and those changes perhaps not being reflected in the sync process and then causing an ongoing issue within that particular folder. But I have been editing notes inside some of the suspect folders yesterday that sync fine, while there are files created or edited a week or two ago that never synced, so I can't seem to pin down any definitive logic to the issue myself. Doing a software comparison of the databases is beyond me - I am not a programmer, but right now I'm kind of wishing that I were.

This has been a serious blow to my confidence in Joplin. The issue is not catastrophic (I don't seem to have mangled or conflicting data), but it's pretty bad, as I can't even be sure if my desktop database is definitive to start again from scratch with a fresh install.

I love open software, I love the idea of Joplin, I've been taking lots of notes on the issues I come across to file bugs and suggestions and try and contribute to its success, and I was very happy to finally be free of the closed system of Evernote. But this has the potential to subtly threaten the integrity of my personal database in the long term, which, needless to say, is not good. In some ways it's almost worse than a major corruption, because I can't tell when the issue started, can't tell exactly what is affected, and can't just restore from a backup.

I've drawn out a large diagram of the kind of differences I've come across between the two installations. Apologies for the massive image, but I thought it best to lay out the various parts of the problem visually.

If you can help explain the potential reasons behind the problem, or any ideas about how to run a (non-programmer friendly) content comparison and the most practical potential solutions (Reinitialize? Export and Re-import? Other fixes?), I would be enormously grateful.

I'm glad I was holding off on adding an Android client to the mix, or I would be in serious trouble with potentially 3 differing databases to try and reintegrate

Joplin is an amazing tool and I want to see it do well, but this has me seriously worried.

[eagle]

Hi @scribbler,

Your laptop not displaying the number of notes in within a notebook can be fixed with main menu View > Show note counts.

Also, have you compared the output of Tools > Syncronisation Status on both clients? Does the numbers match up?

Third, are you using end to end encryption when syncing?

[scribbler]

@eagle

Your laptop not displaying the number of notes in within a notebook can be fixed with main menu View > Show note counts.

Awesome, thanks, have now tweaked that.

@eagle

Also, have you compared the output of Tools > Syncronisation Status on both clients? Does the numbers match up?

This is really helpful, I have some actual concrete numbers now:

Looks like I'm at least 3 folders, 6 notes and 41 resources short. The revision numbers don't match either. Difference breakdown:

On the plus side, I now know definitively that there are also files on the laptop that are missing from the desktop.

I am not currently using any kind of encryption on the sync.

[scribbler]

Looking at these folder names, some of them will be folders I renamed or moved into or out of another folder, but I don’t think that’s the case for all of them, so not sure where that leaves my sub-folder theory.

[eagle]

I’m just a Joplin user - so hopefully someone else will step in after this.

But as an extra test, what happens if you try:

1. Create a new note called “From Desktop” on your desktop Joplin.
2. Create a new note called “From Laptop” on your laptop Joplin.
3. Sync both Joplins a couple of times with the “Synchronise” button at the bottom left.

Does both new notes appear on both computers after this?

[scribbler]

OK, running this test produced an unexpected piece of new information, although it doesn’t quite solve the puzzle:

1. Created a new notebook on Desktop called “From Desktop” containing a note “From Desktop”
2. Created a new notebook on Laptop called “From Laptop” containing a note called “From Laptop”
3. Ran sync.

The laptop file appeared on the desktop, but the desktop file did not appear on the laptop.

I checked my nextcloud status, which gave the following error:

I cleared a few gigabytes of space on the laptop, created a new pair of folders: “From Desktop 2” and “From Laptop 2”, and all 4 folders have now synced both ways, namely: From Desktop, From Desktop 2, From Laptop, From Laptop 2. However, none of the older problem folders have materialized and the laptop still contains no sign of:

• Art - Animation
• CV
• Japanese - Voice Actors
• Music - Software

While the desktop still does not have a copy of the “TO SEND OUT” folder.

It is indeed the case that over the last couple of weeks I’ve been doing some major shuffling around of files and have had very full disks on my computer, so that could have interrupted the sync. But as this little experiment seems to show, that should have been corrected after space was cleared, but has not happened with my missing folders (my HDD was not constantly packed to bursting, but has filled up and emptied out several times recently, so the Nextcloud software should have had a chance to catch up).

Thanks for your help so far @eagle.

[scribbler]

I saw a minor discrepancy in the file size and number of files in the Joplin sync target folder inside of Nextcloud on each system, but after doing a few forced syncs back and forth, they both aligned to exactly the same at 120 MB, 1379 files.

Checked both Joplin installations, each one is still missing the same folders as before. It looks like they both have the same source data to work with, but are somehow interpreting it differently.

[uxamanda]

This sort of sounds like an issue I was hitting for a while that turned out to be caused by a very large attachment that wasn’t syncing. I was able to see the problem file in the logs file: Uploading a large resource (resourceId: xxxxx, size:103089014 bytes) which may tie up the sync process. I am not sure that yours is the same issue, but sounds like it could be. More details on what I was seeing: Issue syncing tags, some notes, folders between Mac and iOS

[scribbler]

I do recall a few pop-us prompting me to let Joplin automatically downsize large images, and on at least one occasion I accidentally clicked no. I’ll have a dig around in the log files. I can already see in the attachment list that I have a fair number of multi-megabyte image files, and a lot of them are from that troublesome woodworking folder.

EDIT: There is no mention of “large resource” in the desktop log, and only a single instance in the laptop log:

2020-05-01 10:21:29: "Starting scheduled sync"
2020-05-01 10:21:29: "Uploading a large resource (resourceId: 110379885a994a6b8b2288e34d36fe46, size:11156583 bytes) which may tie up the sync process."
2020-05-01 10:21:30: "TaskQueue.stop: syncDownload: waiting for tasks to complete: 0"
2020-05-01 10:21:30: "TaskQueue.stop: syncDownload: Done, waited for 0"
2020-05-01 10:21:30: "Operations completed: "
2020-05-01 10:21:30: "updateRemote: 1"
2020-05-01 10:21:30: "createRemote: 4"
2020-05-01 10:21:30: "Total folders: 40"
2020-05-01 10:21:30: "Total notes: 179"
2020-05-01 10:21:30: "Total resources: 114"
2020-05-01 10:21:48: "Scheduling sync operation...", "30000"
2020-05-01 10:22:02: "SearchEngine: Updating FTS table..."

I’m afraid I have no idea what I’m looking at here though. Would it be helpful for me to upload the complete log files?

[uxamanda]

@scribbler

"Uploading a large resource (resourceId: 110379885a994a6b8b2288e34d36fe46, size:11156583 bytes)

I would try deleting this resource 110379885a994a6b8b2288e34d36fe46 to see if it changes anything. I found the offending resource in the Tools menu > Note attachments. Just deleting my one giant file was enough to fix my sync issues. As I mentioned, my problem might have been something else, I hope I am not sending you down the wrong rabbit hole!

[praxiq]

I was wondering about this. Joplin keeps its data in its internal database, which it syncs to the filesystem, which in turn is synced by Dropbox or Owncloud to another device, where another Joplin instance syncs with its own internal database. That's a very complex system with a lot of failure points, and a bug in any one of the steps, or in the interactions between the steps, can lead to data loss, which is scary for a notes app.

(Note that silent sync failures can always lead to data loss. "I don't need to back up my phone, because it syncs to my computer, and I back up my computer." Oops.)

Sync is hard, and I'd much rather my notes app wasn't trying to do it at all. I currently use QOwnNotes, which doesn't have a separate internal database - its "single source of truth" is simply a collection of Markdown files, with human-readable names, in folders on disk, which can be directly synced using Dropbox, Owncloud, or any other filesystem syncing service, since no special API is being used. This also means that even without a special app installed on a device, I still have read and write access to all my notes, which is very comforting.

I've been evaluating Joplin because I haven't found an Android app that lets me sync, browse, view, and edit a folder of markdown files in a way I'm happy with. (Also, because QOwnNotes is ugly as sin. ) But I've been concerned that Joplin's sync is way too complicated, and this only supports my fear!

[bedwardly-down]

@praxiq, my current fear with Joplin is becoming increasingly similar. Because of its dependency on Reactjs and Electron, an upstream Kernel commit broke Joplin usability back in January for multiple Linux distributions and only recently seemingly was fixed upstream. I fear that Joplin’s becoming volatile; its capabilities are commendable but will it be a long term, stable and safe solution for those that want to keep memories and important data for an indefinite amount of time?

[Rob]

@praxiq

Sync is hard, and I’d much rather my notes app wasn’t trying to do it at all. I currently use QOwnNotes, which doesn’t have a separate internal database - its “single source of truth” is simply a collection of Markdown files, with human-readable names, in folders on disk, which can be directly synced using Dropbox, Owncloud, or any other filesystem syncing service, since no special API is being used. This also means that even without a special app installed on a device, I still have read and write access to all my notes, which is very comforting.

That's my ideal, too.

A compromise position is one that I had for some years with Wikidpad — there is a database beyond the plain-text files, but it's purely for speed of search — if it's lost or corrupted, you can regenerate it from the text files.

The problem with this approach is that it's hard to maintain metadata beyond they tiny bit that the filesystem provides.

[NothingCtrl]

I think maybe the problem by full-disk issue that make bug on Joplin database on one of your devices. Another cause maybe files is not sync to NextCloud server, you should check the folder size and number of items on NextCloud web also.

One more problem is the way you sync data is waste of space and syncing will take very slow, because data store at least 5 location:

• Joplin database on Device 1
• NextCloud store on Device 1
• NextCloud server
• Joplin database on Device 2
• NextCloud store on Device 2

Try to correct the database on one device and sync directly to NextCloud.
Remember backup all notes before change sync address.

[DoubleK66]

I get your point and have same problem using Synology cloud, but even I use Wedav, dropbox, it will be the same scenario, it is the design of the product to do the sync like that, the problem is the way Joplin store message not the way how the user sync, I like this product and hope this can be fixed or add an option to reset the local storage, then I can use the export feature to copy latest note to the machine which missing the data.

[pavel]

Same problem. Syncing via Synology NAS. I periodically delete application data and start synchronization again, but it takes a lot of time, especially on a smartphone.