What is the correct way to recover from the orphaned remote data bug?
After you delete a resource from a note Joplin does not delete this resource from the remote sync target.
I have deleted 11 resources from my local Joplin and verified that they still exist on the remote target.
In the past I have tried editing the database, deleting the resources manually, running scripts from some of the other users but this always resulted in corruption or sync errors. The only thing that has worked for me is deleting the entire data set form the remote and syncing again. Having to delete your entire dateset after each change like this is a huge waste of time and resources and horrible solution for a note taking app with synchronization capabilities.
Is there a "best practice" for dealing with this bug? What are other users doing to mitigate it?
I am having same issue since now and then, and every time have it I delete the entire data set from the remote (in my cases android and windows PC), and then syncing again. Hope there is a better way to deal with this bug.
I have tried running the scripts, deleting the resource manually and deleting the entry from the database, nothing works and it all ends in sync errors.
I tried opening a bug report on github and I carefully detailed the steps that I can go though to reproduce the bug but the bug tag was removed and I was told it is not a bug!
I also tried asking for help or advice or help on how to recover from the issue because the only way I know how to recover is to delete everything and start again as well. I have had to do this 5 or 6 times over the years. This bug is a very long standing issue but how can we communicate this to the developers?
If you regularly have to delete all your data everything to recover from a sync operation, this would seem to be a major issue not a minor one.
Thanks for replying and confirming the issue. If this is a well known issue that has been discussed many, many times then why would it not be appropriate to put it in the "bugs" section of the website along with a simple link to a work around or best practice for how to deal with it? I just proposed this on github and the issue was closed. Wouldn't transparency be the most productive policy? How many new users know that once you delete a resource from Joplin it appears to be deleted but it will actually forever be stored orphaned in the cloud?
"That's not the bug as far as I know. The bug only affects resources that have been attached, and then deleted before they were synced."
So perhaps there isn't that much clarity around the issue? I was not reporting a case where I attached a resource and then deleted before it was synced. I was trying to describe the issue where every resource that I delete is deleted locally but becomes orphaned remotely and is never deleted. This is a different issue than what laurent22 described as only being aware of.
The problem is there are other github issues where the bug was described as being fixed and asking users to report if it is still happening which I did.
I have also tested your script on two occasions in run only mode and it proposed to delete all of my notes. I still have the code on my computer. I would love it if this was a temporary solution as it would save me a lot of time.
The only way I currently know to fix the problem is to delete all my data and start again. I have to do this every time I delete a resource! I have tried deleting the notes manually, deleting the references in the database. Nothing works for me except deleting my entire data set and starting again. This usually takes half a day or more to upload and download the whole data set on all devices.
I am not the only user who has to delete everything, either. This is a major issue for an synchronization app. Once your data uploaded to the cloud it can never be deleted even though it appears to the user inside the app that it has been deleted?
Plugins and new interfaces are awesome improvements which bring small incremental and welcome quality of life improvements but at the heart Joplin is a note synchronization app, and this is what makes it special. How can we get attention for major core issues like this?
This is highly unlikely. My script does not delete any notes, but only resources (attachments). I also doubt that it would delete all of your attachments unless all of them are orphaned.
I've been using this script since the beginning and it does what it is supposed to.
I had to make changes, when note history was added to Joplin, to ensure that no attachments are deleted which are associated with old revisions. Maybe you are using an old script.
I am using the default 90 day notes history policy. If it is a note history issue then would the resources have been deleted from the local storage in this case? These resources all still exist on the sync target but not the local resource folder after I deleted them from the note. They do not exist in any other notes.
To test this I would like to delete the history for this note but I do not believe this is possible correct? As a work-around I thought of a potential solution for deleting the note history on a per note basis:
I copied the note content to a new note and deleted the original note. This should effectively act as a work-around for deleting the history of a single note.
Now there should be no note history containing deleted resource links. As of right now, all of those deleted resource do not exist in my local /resource folder. In my sync target folder they exist in two files:
I understand it will take 24 hours before the script deletes the resources? I will report back in 24 hours if the resources have been deleted from the sync target or if they still exist as orphaned resources.
Tessus: I will check your script again. How long do I need to wait until the script will recognize the orphaned resources? It was 6 months to 1 year since I last used it and I remember it suggested deleting everything which was scary.
Orphaned resources are resources that don't have any notes associated. After 24h Joplin usually sets that flag, but it is possible that those resources are not cleaned up.
My script checks for all resources that do not have notes associated. However, if you use a note history of 90 days, it could be possible that resources not currently associated had been associated within those 90 days. In that case if you were to delete that resource and look at a revision where the resource was still used, it would be gone, since you deleted it.
This is why resources will only be removed if not associated with a note and older than the note history time.
Okay so in this case, to clarify what is actually going on. I deleted the note with the history and created a new note. Now those resources are not associated with any note regardless of history. If Joplin is working as expected, they should be deleted in 24 hours is that right?
How would you delete the resource. Are you talking about deleting the reference from the note or deleting the file from the file system? Before I deleted the note I looked at the revision history. I did see there were still references to those resources. The resources however has been deleted from the local filesystem but they were not deleted from the sync target.
Okay, to rule our note history as the cause of the problem, it has been over 40 hours here is the result.
Recap of what I did:
Create a new note (with no note history)
Attach a resource and sync. Verify the sync was successful.
Waited 12 hours, deleted the resource from the note and sync.
Waited 30 hours and check to see if the resource was deleted.
The resource still exists on the sync target, it was not deleted.
Also, the resource still exist in the local resource folder as well!
The resource was not deleted from anywhere.
This was surprising. I thought the bug only affected sync target resources however it looks like Joplin doesn't remove deletes resources even from local storage. So the scope of the problem is not restricted to remote sync targets.
What do you think is happening here. In the previous thread you mentioned there is a script that runs every 24 hours. Is this script failing? I did not see any errors in the log.
Okay, I did some additional testing including using the script. Here is what I found.
Delete 11 resources from a note with history. Wait 24 hours, delete the parent note. Wait 1 week. Check the resources.
-In the notes table, the note id that these resources were attached to was deleted.
-In the resource table, the resources were all deleted.
-In the notes_resource table the deleted resources are still associated with the note id (but this note id doesn't exist in the notes table):
-is_associated=0, note_id= the deleted note id, last_seen_time is 15 days ago.
-In the deleted_items table, the resource are all here with the associated note in item_id (but this note id doesn't exist in the notes table).
These resources were delete from the local resource folder
These notes are all orphaned on the sync target.
These resources are not detected by Tessus script (version 1.2.1).
I have 228 of these orphaned resources going back to September 2020 (last clean Joplin install)
Is there any code that can deal with this situation? Or is this a new scenario?
Create a notes, attach a resource, sync, delete resource, wait 30 hours. Check the resources.
In the notes_resource table, is_associated=0 and last_scene_time=0.
In the resource table, the note still exists.
In the deleted_items table, the deleted resource does not exist.
Something should come along and delete the notes?
These resources are now orphaned on the sync target and the local resource folder.
Tessus script will detect and delete this orphaned resource.
I have 45 of these orphaned resources.
If an external script can take care of the problem then why can't Joplin be fixed to take care of the problem? It seems to affect all users, where they are aware of it or not.
The original question I was asking in this thread is how to recover from the bug. I ended up having to take a JEX backup, delete all my data from all devices, upload and then sync all devices again. This has to be the worse case scenario. If anyone knows of a better way to take care of these orphaned resources, I would love to hear it.
I'm not sure. In the case of orphaned resources on the sync target, that should no longer happen. But I guess the ones you have happened before the time that bug was fixed.
You can try my other script jnclnst -n. Attention: Only run it with the option -n
I can't recall if it takes note history into account. But it will list the number of resources. If the list has 228 entries you should be fine.
Other than that I think the only option is: export all to jex / delete sync target / import
That was explained several times, but I can't recall Laurent's explanation.
I can investigate this, do you recall which version this bug was fixed in?
Because the issue goes back so long, has so many posts, issues and users discussing it, it is very difficult for the end user to know what to do, how to handle the situation. I have been sorting though all of the posts for days and I am confused. This is why I suggest that a not be put on the Joplin homepage under the (https://joplinapp.org/#known-bugs) section along with a link to the resolution:
Something like this:
Due to [...reasons...], currently Joplin is not capable of removing deleted resources. After your data is removed from the application and you have synced all targets, it will still exist in possibly multiple locations. Currently, the only way to remedy the situation is to export your data, delete all sync targets, import your data, upload to sync target and sync to all devices."