Photos sometimes not syncing (Syncthing, Android+Mac)

Support
1

Operating system

macOS

Joplin version

3.4.6

Desktop version info

Android Joplin version: 3.4.4

Sync target

File system

What issue do you have?

I have set up Joplin with "File system" target and Syncthing to sync between my macbook and my Android phone. Generally everything works great. However, notes with photos I create on my phone sometimes do not sync properly to my laptop. The note does appear on the laptop, but either all or some images do not show up there, being replaced by download buttons in the note preview. I have confirmed that the image files with the correct hash are present in my Joplin target folder (~/Joplin/.resources) and are not corrupted. They do not show up under "Tools -> Note attachments" though. Pressing the sync button on the macbook or the Android device does not solve the issue. What does work, however, is re-building the local database using "Delete local data and re-download from sync target", but this is obviously not an ideal solution.

Under "Help -> Synchronization status" there are no obvious problems, "Error" and "Conflicts" are both 0. When turning on debug mode and viewing the problematic note, I get errors like
"Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'file_extension')
at resourceFilename (resourceUtils.ts:9:55)
at Object.resolveImageSrc (Editor.tsx:117:39)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async renderBlockImages.ts:46:25
"
These errors do not appear for undamaged notes.
My issue does not happen reproducibly, when I tried just creating a note with a few photos while being at home, everything synced well. It happened twice now when being away from home for longer and creating a "daily journal" note during the day.

The photos Joplin's camera function takes are quite large (6-10 MB), I speculate this might have something to do with the issue, maybe Joplin tries to sync from the Syncthing share while Syncthing has downloaded the note, but not yet the resources. That would be fine, but what is not is Joplin stubbornly not incorporating the images to its local database even after they have been synced to the Joplin sync folder successfully, requiring a full rebuild.

Thanks for your help!

2

I think your guess is likely correct. Unfortunately, filesystem sync in its current state is unreliable unless you apply some workarounds (see Sync silently ignores new files with old timestamps · Issue #6517 · laurent22/joplin · GitHub).

1 Like
3

I was just reading this issue on GitHub as you replied! So this seems like an unresolved issue that affects many people and has been going on for quite a while now. I think this is quite a major caveat when using file-based sync, as I assume many users are using Syncthing or similar. Maybe Joplin should copy the modification detection logic used by Syncthing, this seems quite reliable to me?

4

The issue is quite difficult to tackle. The main difference between filesystem sync and other types of sync is that with the latter, you've got a single source that is used by all devices, and the whole synchronisation mechanism is built around it. In the case of filesystem sync, there is no single source, but rather the filesystem sync path on each device acts as its own source. When there is just a single source, Joplin can assume that its state is always complete. However, with filesystem sync, there is no such guarantee, and this eventually leads to the issue at hand.

I'm sure someone with enough coding skills could come up with a solution to the problem, but no-one has done it so far :frowning:. For the record, I've been using the PowerShell workaround mentioned on GitHub for a very long time now, and it does "fix" the problem, even though in a hacky way.

1 Like
5

@tomasz86 I’m a bit confused by your PowerShell script solution. Doesn’t that prevent any new incoming changes to all your existing notes from being synced until a year has passed?

If I take Syncthing out of the equation and run your PowerShell script on a filsystem sync directory, then set up a secondary instance of Joplin pointing to the same dir, then syncing incoming changes does not work on either Joplin client

Also you said:

When encountering the error myself, I've up to now relied on the "delete local data and re-download from sync target" button, but this requires a manual action, while I'd prefer to rely on an automated solution here.

How does the PowerShell workaround provide an automated solution?

6

No, I don't think so. This is because if those notes are modified again, then their corresponding MD files’ modified time also changes back to the present. After that, the whole cycle repeats. :wink:

I've got a script that runs indefinitely in the background. It processes MD files in the filesystem sync folder every minute, changing their modified time into the future (but only after 5 minutes has passed since the last modification). The script also ignores files whose modified time is already in the future.

The whole script goes like this:

Push-Location -LiteralPath $PSScriptRoot

if (-not $args[0]) {
	Exit
}

$NotesFolder = $args[0]
$NotesFolder

if (-not (Test-Path -LiteralPath $NotesFolder -PathType Container)) {
	New-Item -Path $NotesFolder -ItemType Directory
}

while ($true) {
	& {
		$Notes = Get-ChildItem -Path (Join-Path -Path $NotesFolder -ChildPath "*.md") -File | Where-Object {$_.LastWriteTime.AddMinutes(5) -le (Get-Date)}
		foreach ($Note in $Notes) {
			$CurrentTime = (Get-Date).ToUniversalTime().ToString("yyyy-MM-dd HH:mm:ss")
			$CurrentTime + " " + (Join-Path -Path $Note.Directory.Name -ChildPath $Note.Name)
			$Note.LastWriteTime = (Get-Date).AddYears(1)
		}
	}
	Start-Sleep -Seconds 60
}

Pop-Location

It can be used by pointing it at a path, e.g. JoplinDateFix.ps1 <filesystem-sync-dir>. I just want to stress to everyone who is reading this that the script has been created purely for personal use, so if you don't understand what it does exactly, please do not run it. :sweat_smile:

I also need to point out that I run the script on a single server-like computer that is on 24/7, and not on every device that is used for Joplin. This makes it possible to avoid potential issues like sync conflicts, etc.

7

I guess that only works if you do the initial sync for a new client before running the script, which would not be true in my test using a single source of truth.

Indeed the solution is quite a hack!

8

Are you sure? I can connect a new client and sync it from the target path (which has all those future-dated files) just fine.

9

There may have been an issue with the duplicated Welcome notebook from the initial notebooks when creating the new profile.

When repeating the test I was able to receive the objects on a new profile, with all the future dates. However if I make a change to an existing note on profile 2, sync it, then sync profile 1 it does not receive the change. So there does seem to be some issues if using this solution with a single source of truth

10

Is this even after running the script in between? It should change the date of that modified note again.

11

No I only ran it once after the initial sync of profile 1.

Are you saying that the PowerShell script must run continually otherwise it basically breaks your sync?

12

Yes, the script must run indefinitely. As I’ve mentioned before, I have it set to process the folder every minute. Of course, most of the time there is nothing to change, so it just goes through the file dates and that’s it.

I think the changes should still be detected in your case though, without running the script that is. Let me do some testing and I’ll come back with the results.

13

I previously put a lot of thought into how this issue can be fixed on the Joplin side which amounted to nothing. But after revisiting earlier posts of the ‘mega’ thread, I realised that the most important aspect to fix is detecting incoming changes for new notes. It’s actually a lot easier to cater for that in the existing algorithm, with a small caveat. So I’ve went ahead and created a PR to address this All: Prevent Joplin sync ignoring new objects when syncing with an external sync service concurrently by mrjo118 · Pull Request #13054 · laurent22/joplin · GitHub

I’m pretty sure though if this PR does get accepted and you stopped your script from running, then you are going to need to completely reset all your Joplin clients and the ‘server’ which you are running your script on in order to avoid sync issues. From what I can tell though, your script only handles the new note scenario, so this change in Joplin could be a direct replacement

14

Thanks you a lot for working on this! I will definitely be keeping eye on the progress. :smiley:

I think the script handles both cases:

  1. If a new note gets synced and ignored, my script bumps its modified time, which as a result forces Joplin to "see" it again. With your PR, this won't be necessary any longer, as Joplin will always process note files which haven't been added yet, regardless of their modified time.

  2. If a note gets modified, synced, and ignored, the script bumps its date, making Joplin process it during the next synchronisation. This leads to the note being updated properly.

If I understand correctly, what happens in point 2. will still be necessary even after merging your fix. Is this correct?

15

Yes, only point 1 is directly addressed by the PR.

But for anyone not using a script, if they know a note is missing a change, it is possible to get the latest version by forcing a conflict, without needing access to the device which the latest change was made. That is existing behaviour

16

I think I understand how your script is working with updates now. Whenever you sync a file which you bumped the timestamp for, that timestamp gets stored in the Joplin sqlite db so you’ll only be able to sync the file again on the same client if the timestamp is higher than the last one. The reason it doesn’t break sync is because the next time you bump the timestamp it will be higher than the last one. So if you ever stopped running the script, then all notes with a bumped timestamp wont sync again unless the timestamp time has passed, or you bump the timestamp on the sync target again for those notes

17

Hi, @toterpinguin

From what I know syncthing for Android is in maintenance mode and may have some issues synchronizing with the latest and greatest version that you have running on your Mac OS.

I have an idea or suggestion that might help that does not involve using syncthing.

Your Mac OS is capable of running open SSH server which provides SFTP access to your drive and data.

There is a program written for Android that is distributed on GitHub called RSAF. It is a document access tool and Cloud access tool and I'm not 100% sure if Joplin will recognize it or not so I will be testing this here so hang on. It would let you use the SFTP protocol to synchronize your Android with the server directly.

Your laptop would be the primary data store and use local file system access and the Android client would use SFTP to access the computer.

As I said I have not yet tested this but I will do so. If it works for me I can assist you in getting it to work for you. Let me find out if it works as I think that it should work before you spend any time on it.

I have an Android tablet that I use for testing purposes. I will try and get it to sync with SSH server.

18

Yeah, but the issue is that with larger or more complex notes, you may not know that there is a problem before it's too late. You then start making changes on another device, ending with a conflicted copy that is annoying to merge manually. I'd experienced this problem many times before I came up with the hacky PowerShell solution.

Thanks for a deep analysis of the script! What you've written seems correct, although to be honest, I didn't really analyse the possible outcomes so deeply when I was writing it. I just knew that I needed to come up with something quickly to make Joplin sync reliably, and that was the result. The script proved to do the job, so I left it like that. For the record, the current version, which I posted a few hours ago here as well, was last edited 9 months ago or so, and I haven't had any need to touch it since then.

This isn't exactly correct. The official Syncthing app for Android is no longer developed, however the recommended solution is the Syncthing-Fork app, which is fully up-to-date with Syncthing proper.

1 Like
19

Thanks for the update.

Joplin is still relying on a third party app to assist it with the synchronization.

If I can get my idea to work Joplin will do all of the synchronization itself.

20

I think this is a very sound analysis of the problem, particularly that the sync is built around the assumption of a single source of truth.

Looking at the basic delta algorithm, it uses a context timestamp to essentially mark that all files with a modified date below this time have been synced. This is ok when you have a single source of truth, because you can assume no new updates can be made on the server with an earlier timestamp to when you last synced. Note that in the sqlite db of the client, we currently store the a sync_time for every item, which is not used for determining the delta, but is used for determining conflicts.

If the sync_time on each item is changed to store the modified date on the server file at the point of synchronising an incoming change (instead of using the current timestamp generated by the client), the algorithm might be able to be modified to cater for multiple sources of truth. The basic delta function currently takes in the ids of all local objects for the client, which is used to compare against a list of all files on the sync target. The input could be modified to take in the sync_time alongside each local object id. Then to handle object updates, instead of comparing against the context timestamp, it could compare against the sync_time value specific for that object. The context timestamp would essentially become redundant, and creation and deletion of objects would be handled as of my open pr, where both are based on the presence of objects on either the client or the server.

All of this can be done without introducing any additional I/O calls to the algorithm, so it should not have any significant impact on performance. This would not affect Joplin server either, which uses a server side database instead of file objects, as it implements its own delta algorothm instead of using the basic delta. The problem then, is this would be a significant change to the syncing algorithm and may not be backwards compatible with existing client and server data. Also, for sync targets which do use basic delta, if the server is not the same device that the client is running on (any sync target which uses basic delta and is not connected to either file system sync or a local webdav server running on the device), then setting sync_time to the modified time of the server file would not be safe, because it could be prone to time sync issues. So this would need to be a behaviour that can be toggled, and potentially a separate column should be used instead of changing the value of sync_time, so that toggle on and off can be done safely.