Backlinks Navigator: 1.3.3
Backup: 1.5.1
Codeblock Autocomplete: 1.3.1
Context Utils: 1.8.1
Copy as HTML: 1.6.4
Heading Levels: 1.0.8
Heading Navigator: 0.7.1
Journal: 2.7.0
Markdown Alerts and Formatting Commands: 1.6.6
Markdown Formatter: 1.6.5
Note Tabs: 1.4.0
Paste HTML as Markdown: 1.1.13
Quick Links Plus: 0.7.5
Rich Tables: 1.8.18
Simple Image Resize: 1.6.0
Space Indenter: 0.2.6
Wrapped Line Indentation: 1.1.2
Sync target
S3
Editor
Markdown Editor
What issue do you have?
I've already restored the data so no data loss, but I'm just kind of baffled/concerned how this could have happened
I use Joplin for my work notes and sync with a cloudflare S3 bucket
I was using Joplin all day on my work laptop with no issues, had joplin in the background for a while, and when I switched back to the Joplin window found that all of my notes had been deleted (sync status showed ~2500 items deleted, and all notes were gone except a random 256 out of my ~1000 notes appearing in the conflicts folder, see screenshot).
I tried closing/relaunching joplin and clicking Sync and nothing changed.
I went to one of my other devices, which did still have joplin open to my work profile and none of these deletions had propagated there, I even hit Sync and all my notes were still there. So I copied the SQLite database from that machine and replaced the one on my work machine and all seems OK for now... but any idea what this was?? Local SQLite database corruption?
The deletion log is attached, the deletions are all the 8/7 dates.
I checked my backups folder and the backup from about an hour ago (12:12pm EST) appears to have everything, the deletions appeared to happen around 12:36.
Yikes, that's pretty bad. Are you absolutely sure that you don't have failsafe disabled on the profile? Also can you post the full main desktop log as well?
EDIT: Actually it might be better if you send the log to me in a PM, just in case it contains anything which should not be public
Is it possible that the s3 backend was faulty and temporarily reported an empty bucket?
Only S3-related change we had recently was reverted before 3.7.10 so that's not it either.
The only possible explanation for all these deleteLocal lines is that Joplin sees that the remote item has been deleted for some reason. Would you mind sharing the first lines from the log when sync starts?
Thanks. Based on the log it looks like the service reported an incorrect number of object for a short time:
Time
Listing saw
State
12:36:43
4325 items
healthy
12:36:52
1775 items
← the bad listing; wipe starts this second
12:37:27
1775 items
still on the bad cached context, finishing the delete
12:42:32
4326 items
recovered — the bucket had everything again
12:45+ onward
4326, 4330, 4331…
stable, fully back
And after 12:42:32, Joplin didn't restore the items from the service because they are locally flagged as deleted.
I'm not sure what is the solution here since we have to trust what the service tells us, but maybe we can adjust the failsafe, as it currently works when more than 90% of the notes are deleted. In your case, it's less than that but still a massive number of notes - so maybe it should be a mix of percentage and a cap on number of notes.
I've created the issue now. Maybe it would be worth submitting a bug report to Cloudflare as well? Seems like it could cause issues in any app, not just Joplin
I have to say, that makes me lose confidence in Cloudflare which I just switched to recently for sync...
I only have about 300-600 sync objects depending on how many revisions I've got at the time, so an arbitrary number for an absolute count circuit breaker might not help prevent deletions in a case like this. I guess at least the good thing is it did not propogate the deletions back to the server
I have another idea. I previously implemented an 'enhanced' basic delta algorithm for file system sync. I did some tests, and because local deletion via the delta step uses trackDeleted false (so it does not create deleted_item records) the only thing which stops files which 'temporarily' disappear from the sync target from being restored when they re-appear, is the logic of the delta. When using file system sync, because it uses my enhanced algorithm which considers files with both newer and older timestamps, it will restore an item in Joplin if you delete it from the file system target and later restore it.
For the non enhanced logic of the basic delta, all it would take to make a locally deleted file get restored when the item re-appears on the server is to not skip the file if the id doesn't match any of the existing local ids (which are already available via allItemIdsHandler). This can be done without any reduction to the performance of the basic delta. Making this change would then add the ability to self heal from this scenario for WebDAV and S3 bucket sync targets, any other target which uses basic delta.
It seems that WebDAV running on a NAS is the most reliable way to sync compared with other commercial cloud services. I am not sure why, but I've seen several posts on this forum from users who have experienced sync issues with commercial cloud services...
