Sometimes latest changes get reverted (during sync?)

I’ve read the issue and sadly, I don’t think this would be applicable. I’m using Nextcloud WebDAV only on Android. On PC and laptop I’m using the Nextcloud Sync Client.

Moreover, I see that once again my endpoints lost sync:

PC
Screenshot_20200313_211747
laptop
Screenshot_20200313_213116
I really don’t want to use the touch * workaround again.

What can be done to prevent this from happening? Is anyone else having similar issues? I wanted to start moving more and more data to Joplin and make it my one and only note taking program. But now I’m having second thoughts.

Does this mean you’re using Filesystem sync in Joplin on the Desktop clients? I’m asking because, despite not being to assist much further with this, that piece of information could be the key to finding out what’s actually happening here. Any extra information is always better than being stuck guessing what’s happening. Thanks

Also, are both desktops running the same OS as posted in the first post?

I haven't read your post in detail yet, but some comments about sync how sync works:

Sync uses files and relies on accurate file timestamps, so ensuring the clocks on the machines at either end is important. If the machines don't have the same time of day, then sync will be given incorrect information to work with, and won't do what it is meant to do.

This can also be affected by the way different machines and filesystems handle file timestamps Eg

Could any of this explain what you're seeing?

This is particularly true when dual booting Linux and Windows on same machine. I keep forgetting about that, although, this situation doesn’t seem to be exactly that, but i digress.

I think you can edit the Windows registry to make Windows use UTC time, and so solve that problem.

1 Like

Hello,

yes, I forgot to mention I use the Filesystem sync on both ends. The ~/Nextcloud/Joplin folder is being synced automatically.

Both the laptop and desktop run the same config:

OS Family: Linux
System: KDE Neon (Ubuntu 18.04 LTS 64-bit)
Joplin: 1.0.193 appimage

Could things like not booting one of the computers for a few days or one of them being offline for some time affect the sync? If timestamps are in use, then I should be able to work for an extended time on an offline system and have no issues once it gets plugged again (I had a network outage for a few days and relied on tethering at times).

What’s the role of Joplin/.sync/version.txt? It gets modified every-time I sync. Is it to keep track of the last sync time?

@mic704b I don’t dual boot, I don’t run Joplin on Windows.

Should I start a separate thread about general sync issues, since the reverting of the notes may be a separate issue.

EDIT: And yes, the dates and times are in sync.

I’m not fully sure with the technical side of things, and would rather wait for Mic to get back to you on this one. I’m the one that asked about dual booting to make sure I understood what he was referring to since dual booting frequently causes timestamp issues.

It shouldn't. As long as the computers clock isn't drifting too much.

Correct.

The fact your updated note was found in the Conflict folder means it is most likely due to sync. If the note was edited on both this and the other computer since the last sync, then the note on this computer will be moved to Conflict, and the note from the other will replace it. Which seems pretty much like what you're seeing.

The first thing I suspect is the sync file timestamps.

I should say I have no experience with syncing with Nextcloud, so I don't know specific issues related to that.

I have an idea.

Previously, Joplin was slow to sync when using the Nextcloud back-end, but this has been resolved. What if I switch from the File-system back-end on both PC and laptop to the Nextcloud back-end?

That way I’d bypass the Nextcloud Desktop Client (it would still download changes, but Joplin would not care anymore). Since it would be only Joplin doing the modifications in this folder (~/Nextcloud/Joplin) this would allow me to troubleshoot further. Or maybe I should disable Nextcloud sync completely for Joplin’s folder, just to be safe?

I suspect, that before I continue, I should at the very least resolve the conflicts to make sure nothing gets mangled. A backup would be advisable.

ALSO: should I do touch ~/Nextcloud/Joplin/* before I switch to the Nextcloud back-end? I don’t want to accidentally “improve things too much” to the point where the whole experiment becomes no longer representative.

@mic704b - should I go forward with what I proposed above?

Unfortunately I have no experience with sync via NextCloud so I am not the best person to advise you and don’t know how it should be setup. I hope someone else will chime in.

However whatever you do, first make sure you have good backups to insure you against data loss. It’s a good idea anyway, but even more important here when playing with sync.

Is it possible that NextCloud is updating the file timestamps in the sync target, even when they are not changing? If this happens then your client will think they are newer than when last synced, and might do what you saw.

First I would look at the file timestamps, and see if they look correct. Perhaps see if you can replicate the problem with one note (and so one sync file), and follow it all the way through your setup, checking the timestamp along the way, if possible.

Apart from that, I encourage you to experiment. BUT BACKUP FIRST!

Backup ensures you don’t lose your data. It also can allow you to recreate everything so you can try different experiments from the same starting point.

Let me correct something I wrote earlier.

This is inaccurate, as the note won't be moved, just the content. In the case discussed, sync makes a new note in the conflict folder, with the content of the local note. The original note is then updated, depending on the type of conflict.

Alright. I’ll try reserving some time over the weekend and see what I can find.

I'm still not done with the testing. I actually had a setback: I borked the data and had to restore from backup (always back up, kids!). So the results won't be coming in soon.

But one question popped into my head in the meantime: is it normal, that after switching sync methods Joplin wants to sync everything from the ground up?
:thinking:
I went from file-system sync pointing to a folder managed by Nextcloud Desktop Sync to the Nextcloud back-end in Joplin and Joplin wanted to sync all. I allowed it but it was extremely slow (about 70 notes in 15 minutes) so I aborted.

Shouldn't Joplin just "get in touch" with the backend and sync only what's missing? If there is anything to sync at all?
:thinking:

1 Like

Yet another question popped up:

if I'm using NextCloud, does the server need to have a "joplin application" installed? Because I saw an error saying, that the Joplin application is not installed. In the logs I see this from that time:

 41 2020-03-28 21:37:13: "Total resources: 3007"
 40 2020-03-28 21:39:53: "Could not setup sync target:", "Error: Unsupported WebDAV URL format: https://REDACTED.TLD/remote.php/dav/files/REDACTED@REDACTED.TLD/Joplin
 39 Error: Unsupported WebDAV URL format: https://REDACTED.TLD/remote.php/dav/files/REDACTED@REDACTED.TLD/Joplin
 38     at Function.baseUrlFromNextcloudWebDavUrl (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:60:19)
 37     at Object.baseUrl (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/SyncTargetNextcloud.js:59:35)
 36     at JoplinServerApi.baseUrl (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:53:43)
 35     at JoplinServerApi.<anonymous> (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:117:33)
 34     at Generator.next (<anonymous>)
 33     at /tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:8:71
 32     at new Promise (<anonymous>)
 31     at __awaiter (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:4:12)
 30     at JoplinServerApi.exec (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:99:16)
 29     at JoplinServerApi.<anonymous> (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:75:25)"
 28 2020-03-28 21:39:59: "Could not setup sync target:", "Error: Unsupported WebDAV URL format: https://REDACTED.TLD/remote.php/dav/files/REDACTED@REDACTED.TLD/Joplin
 27 Error: Unsupported WebDAV URL format: https://REDACTED.TLD/remote.php/dav/files/REDACTED@REDACTED.TLD/Joplin
 26     at Function.baseUrlFromNextcloudWebDavUrl (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:60:19)
 25     at Object.baseUrl (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/SyncTargetNextcloud.js:59:35)
 24     at JoplinServerApi.baseUrl (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:53:43)
 23     at JoplinServerApi.<anonymous> (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:117:33)
 22     at Generator.next (<anonymous>)
 21     at /tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:8:71
 20     at new Promise (<anonymous>)
 19     at __awaiter (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:4:12)
 18     at JoplinServerApi.exec (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:99:16)
 17     at JoplinServerApi.<anonymous> (/tmp/.mount_JoplingmrqfN/resources/app.asar/lib/JoplinServerApi.js:75:25)"

So... does Nextcloud require some special configuration before I point to it with the Nextcloud sync backend in Joplin? :thinking: And otherwise I have to use WebDav? :thinking:

EDIT: I just got an error and I see lots of locking errors! A sample:

2020-03-28 22:08:27: "Error: PUT 182720c314184719a00c952fcbb6befe.md: "Joplin/182720c314184719a00c952fcbb6befe.md" is locked (Exception OCA\DAV\Connector\Sabre\Exception\FileLocked) (423): <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
  <s:message>"Joplin/182720c314184719a00c952fcbb6befe.md" is locked</s:message>
</d:error>

Does this look like a potential suspect for slow sync @laurent ?

The locking issue is a bug in Nextcloud. There are many threads about it in their forum (and a few here too).

Hello,

I'm seeing also errors like this:

2020-03-28 22:21:27: "Res 22:18:59: <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\MethodNotAllowed</s:exception>
  <s:message>The resource you tried to create already exists</s:message>
</d:error>

but after reading this

I assume that this is not an issue with Joplin.

I created a thread on the NC forums:

with the hope, that someone will pick this up, but so far only one response.

This is really frustrating. I was so happy, that I'd be able to migrate all of my notes, but now everything is on halt while "a breakthrough happens".

:frowning:

Any ideas what can we do with this to find more information?

EDIT: Maybe it's the preview generator causing trouble?

A small update:

Oh, I have an idea. Does Joplin require the folder it uses to be named Joplin? Can it be ex. Joplin-Debug? I want to create a fresh, small folder and not upset the main one :wink: @mic704b

I'm seeing sync conflicts, but most of this is because I caused conflicts by mixing webdav sync with nextcloud sync with the native client. I want to exclude those and make sure, that I'm not seeing some other conflicts.

Your actual database is usually stored in a “profile” directory called joplin-desktop, the location of it depends on your platform.

You can tell Joplin to use a different profile directory, using a --profile argument when you start it. I don’t know if it is officially supported, or exists on all platforms though.

That way you can experiment and not wreck your real data. Be careful though not to mix them up. (Always back up carefully.)

Hello everybody!

Troubleshooting

It's been a bit of a journey. I'm still not at the end, but it seems I'm making progress. :slight_smile:

TLDR;

A combination of a local issue and an obscure edge case will make your days an annoyance.

The whole process

After trying and doing observations I decided to reinstall my entire OS and migrate only configuration for software that I'm 100% sure I want to migrate. Joplin's data was backed twofold:

  1. the ~/.config/joplin-desktop folder
  2. *.jex archive of the entire Joplin installation

The first item was kept only for reference. I then proceeded to wipe my drive and install Ubuntu 18.04 on it (KDE Neon).

Nextcloud configuration

I created a folder ~/joplin/rel-test-2 for Joplin to use. This was to be synced with other nodes via Nextcloud.

I downloaded the Joplin appimage (1.0.197) and ran it, setting ~/joplin/rel-test-2 as a back-end (file-system back-end). I then proceeded to set encryption and prepare the second node (laptop) the same way. After testing that both nodes have the same data and that E2EE is working I loaded the *.jex archive onto one of the nodes (PC) and waited for the sync to complete (Joplin to disk, Nextcloud to cloud provider). I then powered on the laptop again and waited for the sync to complete here. I don't know why, but Joplin on the laptop couldn't detect the data to load. I recreated the .config/joplin-desktop folder and all went well.

Drift observation

I decided to keep track of potential file drift. I would every few hours create a debug report and also copy the contents of the Tools > Synchronisation Status page. I would put that on a separate drive for safekeeping.

After switching machines I'd use sort to make sure the entries are in alphabetical order (to help with using diff):

image

I would then use vimdiff file-a.sorted file-b.sorted and look for differences. Here is an example:


Apart from an occasional white-space and something incrementing by one (or a normal difference in downloaded and decrypted) all would be the same.

This meant that whatever was causing the drift, died when when I reinstalled the OS. Come to think of it, sometimes after waking the PC from sleep, I'd hear two BIOS beeps. I read that this may mean memory issues, but there was nothing mentioned about waking the PC up from sleep vs. a cold boot. It's been a week since I reinstalled and despite using sleep all the time, there were no beeps.

After becoming more optimistic I sadly noticed once again that notes are getting sometimes reverted to a previous state. But there was no drift.

This suggests that sync drift and sudden reverting of data are separate issues with separate causes.

So I decided to switch from file-system sync to WebDAV sync using a method outlined here:

I'm keeping track of the data I input into Joplin, by keeping a copy in a separate place so I can compare and look if anything is missing. So far I haven't noticed anything vanish. I'm giving myself some more days to observe the issue.

I remain optimistic.

1 Like