So, Joplin seems to be creating a lot of remote items. And I mean a lot more than actual notes I've got in total. I'm not sure what's going on here.
The dev console gives me a lot of these, but I'm not sure if that's related
Some other stuff from previously which may or may not be related:
This time it fetched a lot of them, but they don't seem to appear anywhere?
In general this is from revisions or resources. From the number of items you've synced I would guess that you use the webclipper and this is just syncing a lot of little resources that come along with the web page.
What does it mean? When Joplin Web Clipper cuts a webpage, will it add the resources to the Joplin attachment instead of using the original link?
I didn't use Joplin's official Web Clipper. I am not sure, but this will indeed generate a lot of small attachment resources. . .
I am using the data API to backup a bunch of posts from a website. However, I’m not downloading any resources, only text.
This is what I’m posting:
let toJoplin = {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
"id": postID,
"title": postTitle,
"parent_id": parentID,
"body": postWithCommentsAndNames,
"created_time": postCreated,
"user_created_time": postCreated
})
};
It does have html references to images online tho. As in <img src="link" width=""/> or .
But I do have to mention that I had deleted all of those notes when I screenshotted this.
Edit: link to full code
Yes that's how it should work, in some cases that can fail, but in general Joplin essentially makes the website available "offline".
In that case I don't think there will be any resources from the websites that you loaded in. But it's possible that revisions from those sites are getting synced around. How many websites did you import and then delete?
Under Help->Syncronisation Status you can see how many revisions are synced and other items.
I did a lot of testing back and forth, so I probably backed up either 50 or about 120-ish posts at a time, multiple times (50 per page which I usually tested with, 120 when I ran through everything)
I’ll check this when I’m at my PC again tomorrow. I can’t check that from my phone it seems.
It's available on the phone as well, under Configuration -> Sync Status
Hello,
generally speaking Joplin creates a .md file for everything, revisions (and I mean for each modification to any note), images inserted, attached files ... everything.
The trick is that the remote is never cleaned up (at least for me I'm using Nextcloud). Thus for instance even if the DB has been cleaned up of the old revisions after a while, the .md files corresponding to these old revisions remain on the remote (even though they aren't linked to anything in DB anymore).
This used to piss me off as there was no correlation between the number of files in the remote and the sync information in the app.
Long story short I created a bash script to check what's known to DB and remove everything on the remote of which there is no trace anymore in DB. This way DB and remote are in sync and I don't have anymore a crazy number of useless files on the remote.
I discussed this here : Orphaned revisions (at the end of the conversation)
But just be aware that the crazy number of files on the remote is, as far as I know, normal. Joplin works this way. And basically you don't really have to worry about it, you can certainly let it be as it is.
Cheers
@p6ril, I wasn't aware there was a bug with revisions not being deleted from the server, I'll take a look.
@laurent thanks I can't tell if it's a bug although reading the post here Orphaned revisions it appears we've had an exchange before on this topic and you seemed to suggest old revisions are indeed supposed to be removed from the remote on expiration.
In the end I created a bash script to remove the orphaned revisions on my remote and at some point I stopped using history altogether. I don't know if I'm the only one with this issue (maybe people with a huge number of notes and workbooks don't notice or don't care ... I mean you could live with it) but reading this post I may not be the only one. As far as I'm concerned I'm using Nextcloud (self hosted).
If I can help in any way let me know.
@laurent @p6ril hello! Is there any news on this issue?
Seems that I have the same bug. I use standard dropbox sync on desktop windows client. After creating several notes and then removing them, their revisions are still in the dropbox folder. According to @p6ril script I guess they have type 13.
If it could help, my current settings are:
Joplin 1.6.8
Dropbox folder contains 28 md files:
Example of one 13 type item:
id: 024a6c05566241dfb4f586dbcb354543
parent_id:
item_type: 1
item_id: 59ca09ea4544441a9b16e5a0be22c280
item_updated_time: 1611826368447
title_diff:
body_diff:
metadata_diff: {"new":{"id":"59ca09ea4544441a9b16e5a0be22c280","parent_id":"15b91f687684479587055c7ac244ff58","latitude":"0.00000000","longitude":"0.00000000","altitude":"0.0000","author":"","source_url":"","is_todo":0,"todo_due":0,"todo_completed":0,"source":"joplin-desktop","source_application":"net.cozic.joplin-desktop","application_data":"","order":0,"markup_language":1,"is_shared":0},"deleted":[]}
encryption_cipher_text:
encryption_applied: 0
updated_time: 2021-01-28T09:41:59.859Z
created_time: 2021-01-28T09:41:59.859Z
type_: 13
So if I have note history set for 1 day, and this item has update time on 28.01 (today is 30.01) I guess revision service is not going to remove it?
Hello @KetzerX,
I don't know the status with regards to cleaning up revisions on the remote (Laurent seemed to think it's a bug, but I don't know if it's foreseen or in the works for an upcoming release).
I indeed confirm that revision have the type 13 (I gathered the number from parsing the sqlite database but it's also indicated in Joplin's documentation).
If you want to use my script to clean up your remote folder, please only use the python script, not the bash script. Both work but the implementation logic isn't the same and the python script is much safer from this standpoint (i.e. it should cope better whatever your Joplin usage and current systematization status).
Obviously use at your own risk, but fyi I've been using these scripts for a while now and never had any issue myself. Then hopefully someday, Joplin will properly clean revisions on the remote.
@laurent has big projects going on (like the Joplin sync server) I guess this is an old low priority issue, but still I find it annoying too. For people having hundreds of even thousands of notes, I suppose it can be a problem as their remotes may be huge (actually it may even not be good for sync performance).
Have a nice week-end.
@p6ril thank you for the answer. Could you give a link where I can find the python script, please? I have seen only bash script.
it's probably easier to understand (if you know some python) and much more maintainable for most people compared to a bash script.
Just modify the main constants (mainly file and directory paths) at the beginning of the script and you should be fine. The script backups the local copy of your remote anyway and moves the unused revisions in a sub-directory. Thus it should be mostly harmless (nothing is actually deleted) and easy to reverse should you need to.
Thank you!
FYI this bug still exists.
Just did a fresh Joplin Server. Uploaded notes (no history, no attachments in any notes)
Same problem here. I have been struggling for 4-5 days against this. Joplin seemed to be stucked in a fail loop with fetched items on my Android devices (no synchronisation problem on my desktop). I ended up deleting all my data with the Victor pluggin, then importing one by one all my notebooks in markdown format. Quite time-consuming... but it seems to work fine now.