Does anyone know how to force Joplin to run the unused resources cleanup routine?
I just used the web clipper to clip a webpage with almost 1000 icons. I thought they were unicode characters and I wanted a reference note so I could cut and paste but it turns out they are svg files :(.
I was able to quickly catch the mistake and deleted the note immediately but there are almost 1000 stranded attachment resources now that remain. They are easy to identify with timestamps.
I know that Joplin doesn’t delete the stranded resources right away but it waits a period of time then clears them out. Does anyone know how long Joplin waits before removing these and if I can somehow invoke this function manually?
Joplin want to sync all of the resources to dropbox then they will go to mobile which is not what I want at all. I just closed the app for now until I can find a safe way to do this.
Thank you very much tessus, I will have a look at the scripts now.
Will these scripts just remove the resources or will they also update the database as well. I have been monitoring my synchronization status page and Encryption status carefully since my sync got messed up recently.
So will I be able to see the correct number of resources on the synchronization status pages after running the scripts?
I can only speak for my script jnrmor but I’m pretty sure the perl script works similar.
The script searches for orphaned resources (files not used in a note) and removes them from the database and resource directory. After a sync they will be deleted from the sync target as well.
While I have never had any issues with the script, please make a backup - just in case.
cd ~/.config/joplin-desktop
for i in sqlite3 database.sqlite "select resource_id from note_resources where is_associated = 0"
do
rm resources/$i*
done
sqlite3 database.sqlite "delete from note_resources where is_associated = 0"
Also, can you confirm the resources should be automatically deleted after 48 hours? I know for sure mine aren’t. My resource folder and my dropbox folders are ballooning.
Long answer: The script is safer because it uses the joplin api instead of manipulating the database directly.
Also, I believe for the changes to be removed from the sync target records have to be inserted into another table, which is not done by the commands you quoted. Use the script.
Yes, they should be, but I think there's still a bug somewhere which results in orphaned resources not being deleted.
I was able to install a bash shell on windows, I installed the getop dependencies. I set the .conf and webclipper id and ran your command. I got as far as this:
$ ./jnrmor -n
./jnrmor: line 225: sqlite3: command not found
No orphaned resources found.
I assume that I needed to downloaded the sqlite tools so I did and installed the sqlite3.exe tools so I did (you should add this to your dependency list).
I got a list of records, around 1500. In the actual resource folder I have 815 records to delete. This output is roughly double, what exactly is bring reported here?
I haven't used Windows for 20 years, but I know that Windows now comes with a Linux subsystem. I believe a lot of users on this forum use it to run bash scripts and/or other linux code.
Hmm, on Windows executables use the .exe extension, but the script just calls sqlite3. Also, they should be added to the Windows PATH or fully referenced (with an absolute path) in the script.
Maybe the perl script is easier to run on Windows. I am very sorry, but I am not much of a help on Windows.
I actually solved this, I just had to do this: sudo apt install sqlite3 from withing the Windows linux subsystem ubuntu shell.
I do have the list of records now but it is roughly double what I know is orphaned. I have 815 records that were just created but I am getting 1500 reported from the tool. What exactly is it looking at when report the number of records?
It retrieves the list of ids from the database with the following command
select resource_id from note_resources where is_associated = 0
Are all notes and resources synced on that machine or is there still a sync operation in progress? Laurent also introduced an on demand download of attachments. My script was written before that was implemented. @laurent, if a note has an attachment, but is not downloaded, is there a record in the database that shows that resource as not associated? but in that case it can’t be on the local file system anyway, because it hasn’t been downloaded. so wouldn’t an attempt to delete such a resource via the joplin api fail anyway?
So I use the webclipper to clip a webpage with roughly 1000 svg files. Immediately I realized my error and deleted the note. The resources however are not deleted, they are all local on this machine yes. I can see them in the resources folder there are 815 exactly. The sync started, I quickly cancelled it and extended the sync time to 24 hours to give me time.
So a sync process has started yes. If I look at my sync screen here is what I see:
A few notes have already synced but if you subtract the first number from the second one it is 800 files.
I would like to know what to do next. I don’t want to let it sync to dropbox because it will be a disaster and files from my dropbox aren’t being deleted either so there won’t be a way to clean them up. I tried changing the setting to Manual however it still starts syncing them to dropbox as soon as I start the sync.
The script gave me roughly 1600 records which seems like almost double but it can’t be correct because it would leave me with just a few hundred resources.
This is exactly what I don’t understand. Where are these additional 800 resources coming from? How long have you been using Joplin? If you did something similar a while back or deleted notes with many resources, you could of course have 1600 orphaned resources laying around.
Theoretically you can still sync to the target, because after my script will have run, it will remove the files from the target as well at the next sync.
However, I would not run the script, if you do not know that these resources are really orphaned. You have to figure out what’s up with those 800 extra files.
Something else came to mind, which needs clarification from @laurent.
Let’s say you have 4 revisions of a note. In revision 2 a file was attached, but now in the latest revision the reference to that file is no longer there. So technically that resource is no longer associated with a note. But in the database that resource must not be set to is_associated = 0.
I think I may know what is going on. I changed the synchronization target to file system and it is encrypting the files. This makes sense- roughly double means one encrypted and one unencrypted. Do you know if the the database considers each file and its matching encrypted version to be two different resources?
Regarding the orphaned entries, I don’t think I have 1600 orphaned items. I just restored my database from a jex file last night and I am pretty sure that when you backup and restore from jex it doesn’t copy the orphaned resources. Maybe laurent can confirm this as well but I think from my testing this is how it works.
If this is right then maybe I just need to let the sync finish now and then try the script after.
Unfortunately these are questions I cannot answer.
I do not use encryption, nor note history, nor do I set anything other than always as the attachment download behavior. Thus I have never had the chance to play around with such a database.
Let's wait for Laurent's input. But whatever you do - do not run the script without making a backup first!
Yes I am looking for ways to recover. I was hoping I could recover from a jex file I made last night and then re import today’s work before the webclip problem by running a second instance of Joplin and but it looks like that option has been disabled or is not implemented yet.
I am letting it sync now and I will take a backup then run it when it is finished.
I have decided the cleanest way to recover is to export a jex backup then import into a new instance then sync again. Brute force but it is the safest method until I gain a better understanding of how Joplin works.
Thanks for your help tonight tessus. I learned a lot.
I would like to see what Laurent says about these issues as well.
