Is there a way to delete revisions from the sync target manually? Other than say deleting the whole Joplin folder from dropbox and resync with desktop. While I know that would work, all of my devices would have to download the whole dataset again.
While I do have history set to 2 days I seem to have a lot of revisions. How I know is recently I added a new Android device and it took a long time (not unexpected). But while I have 12,000 total items (mostly text notes only 500 resources and they are small) I noticed that during sync on the Android device it went way over 50,000 saying 20,000/5,000 when I stopped it at one point to check. And revision is 46/46 yet when the Android was syncing that was also way high 35,000/4 or something like that. Now though once it finished it has the name stats as the desktop 12,000/12,000 and 46/46.
So I am assuming there is a lot of orphan revisions on the sync target even though I only have my history set to a couple of days. Anyway I can clear them out? I also checked and the local database on desktop is about 100mb but on dropbox it is 700mb.
What version of Joplin are you actually using (I guess 3.1.24 is the default version in the template)? If you have history retention set to 2 days, then I would not expect there to be so many revisions, unless you’re on a really old version of Joplin.
Regarding being larger on Dropbox, the local database does not include the resources.
3.1.24 is the desktop (main) version I use yes. Other devices may have different versions. I originally had it set to 90 days but since dropped it back to 2 on all my devices. But as I understand it, the device with the lowest setting will take precedence regardless and should clear out anything older than 2 days. This particular sync target has been in use for about 3 years now (a lot of creation, deletion revised, etc during that time as data became changed/got old).
And yes I know the database does not include resources. But the resources is only 32mb, so it still don’t add up to over 700mb.
I’d suggest to set the history to 2 days on all devices, and then do an update all your devices to use the latest mainline release of Joplin (Joplin 3.4.x). Older versions had a bug where the revision cleaning does not actually sync to the server the deletions of the revisions. Be aware that I would expect you to have to wait for a very long sync on all your devices after upgrading, for all the revisions cleaned to be pushed and pulled from the server
Well some devices can’t use the latest version. And to do all that I might as well clear the sync target, upload, and resync all devices again. It would likely be faster anyway.
And I thought that the lowest history setting on any device took precedence? Therefore if you had it set to 2 on one device, whenever that device synced it would clear revisions older than 2 days? If that is true, the only one device would have to be updated to Joplin 3.4.x, set to 2 days and synced to clear out the old? Or has this changed?
Hmmm. I thought it was the reverse, that the device with the largest history setting determined the length of retention. Otherwise an error on one device could wipe out all the history everywhere.
I have history set to zero days yet syncing 5000 notes to a new device still involves 75,000 resources. I’ve never figured out how that works.
From what I have read zero days does not work. That is basically “off” or “never” try setting it to 1 or 2 and see if that clears it. I know in the past when I changed it from 90 to 1 to clear some history a while ago I saw it delete say 2000 notes at the time. But apparently some have been slipping through the cracks or didn’t always do it as per the bug mentioned.
@DBDigital is correct. The lowest value takes precedence. I only said to set the history to 2 days on all devices so that any device would pick up the cleaning job.
Also, it used to be possible to set the history to 0 days on desktop, but that has since been disallowed, though the existing setting wont change if it is already set to 0.
Well some devices can’t use the latest version. And to do all that I might as well clear the sync target, upload, and resync all devices again. It would likely be faster anyway.
If you want the note history cleaning to work properly, you’ll need to upgrade to Joplin 3.4.x on at least one of your clients (on older versions, even though you see the revision deletions when you sync, they will get redownloaded again). Otherwise the history will keep on increasing over time, even if you start from a blank slate. I can’t guarantee your other Joplin clients would be compatible though if you mix and match versions.
If upgrading is not an option and you’re ok to forfeit the use of note history at all, you could do a full reset ( How to fix synchronisation issues and start over ) to clear all revisions (this works because a jex export does not backup the revisions) and then disable the enable note history setting on all clients.
So I’m not so sure why you still have so many revisions, but the full reset instructions I linked will allow you to clear down the existing revisions and have a blank slate.
I believe it is because WebDAV uses the basicDelta algorithm for determining items to update. The sync loop uses pagination, but the basic delta fetches a stat (list of files) of the entire Joplin directory on the target every time it processes a page. So the fetch on the sync status can count up to a number many times more than the actual number of items on the server, because it adds to the fetch total as many items in the target directory, for every page, except for the last page. Indeed it is misleading, and I was confused when I noticed the same behaviour with my Joplin profile using Dropbox. From my understanding of the sync algorithm when reverse engineering the code, it won’t actually process items as many times as the fetched count indicates (so it wont repeat processing of items unnecessarily), but it is just the total that is wrong.
In other words what I’m saying is to ignore the count of fetched items on the status below the sync button. To see the real number of items in the profile and on the sync target, go to the actual sync status screen in Joplin.
It is finally making sense. To check this I repeated the clear and reload on machine “C”, a Linux box. I watched as the resource count climbed to over 80,000 and as soon as it started back down I quit. Restarting Joplin had it ignoring that earlier huge number and it was now downloading the real attachments. I could see this in the Sync status as you suggested. I’ve always waited while it counted back down, something that takes hours and hours, but will now just quit once the initial items are created and restart it to download the real attachments.
Thanks for clearing up this long standing confusion of mine. It will save me many hours as I tend to wipe things out and reload fairly often.
I’m surprised you’re able to make the initial sync quicker by restarting after the creations are done. But if you’ve verified the item counts all match correctly on the sync status screen, then I guess that is an inefficiency in the sync logic that it’s unnecessary to fetch all the items after the initial creations, at least for the initial sync. For the updated logic of the basicDelta which is used when using file system sync (or WebDAV, but only if the host is a localhost url), which I added in Joplin 3.5, fetching all the items which were just downloaded would however be necessary (and forced)
I’ve now done this on two Win 11 Pro boxes, a Win 10 Pro (Surface Pro) tablet and a Linux (Fedora 43) box and everything appears perfect. As soon as the “fetching resources” reached its maximum (around 75-80,000) and began to count backwards I quit. After restarting it then did the Fetching xx/yy over and over and I watched as the Sync Status was updating the attachments downloaded. It used to take about two hours to do one box but by skipping the countdown phase it drops to about a half-hour per box.
I had a hunch from what you said the fix should already be in this version. Thanks for confirming, I was going to look through the change log but I remembered I used to get deletions to the server before. Not sure what I did that stopped it so I thought I would do a test and I created a new profile on this machine Linux 3.1.24 and set it up to Dropbox then started a Sync. About halfway through the dataset download (5,000/12350 or so) it started doing deletions.
I let it do a deletion batch of 500, then I went to sync the rest of the devices thinking I would do it in batches instead of a one long massive one tying up everything. All the devices synced perfectly except one older Android. In this case it started, the wheel spun got up to fetch 433/500 then froze. I let it stay there for almost half a hour before I killed the task and started it again. I thought no big deal as I had seen this before when it got stuck due to a network glitch or such. Well it won’t go past 433. And now when the sync starts the wheel immediately freezes its spin and about 10 minutes later fetching 433/500 pops up immediately along with “canceling right below it. Then it just sits there with the wheel spinning and nothing ever finishes.
This Android is one of them that can’t be updated for several reasons. Lack of app storage space is the main one and the newer versions are larger. Not to mention possible sync incompatibilities are certainly is another factor.
Any suggestions other than just keep going with the sync target clean up, wipe this android’s profile and resync the whole thing? Which sadly is going to be a very long time indeed but at least less after the cleanup.
I’m using Dropbox as well and I’ve seen this happen before. I think it happens due to some kind of lock between Joplin and Dropbox. When that happens I make a change to the most recent modified note (at the top of the all notes notebook), then force close the Joplin app and open it again. Then after a few seconds I find the sync progresses again.
In terms of the revision cleaning issue, as you’re constrained from upgrading Joplin on all your devices, I can’t suggest anything else other than a full reset from jex backup and disabling revisions completely on all devices.
Thanks for the tip, but unfortunately it didn’t make a difference here. I kept trying different things such as clicking cancel or not and closing it via changing profile instead of through the task list etc. And while it started with 433/500 I eventually got it to 466/500 but couldn’t get it to go any further.
I can do a change in a note, delete a note on that device and that will always sync first but once it gets to fetching the wheel freezes and about 5-10 minutes later 436/500 appears with “canceling” under it. It appears to no longer be actually connected to the sync target as I tried with another device shortly after the 436/500 appears and the other device connects and syncs fine. No sync lock so I doubt it is the problematic device is connected at that point.
Now I am cleaning the sync target and once completed I will have to clear this profile on the device that hangs and start a fresh full dataset download. One question I have is what is the best way to clear the default profile so I can do this? I know if I delete or make changes that will change the sync target and I only want it to download as is. I know I can clear the storage but that would clear another profile which I don’t want to download that one again as well if there is anyway to avoid that. On desktop there is a “delete local data and download from sync target”, but is there a way do to that on Android? This version is before the plugins came to mobile.
There is no official way to clear data on the default profile without deleting all your other profiles on mobile unfortunately. I can think of one way to clear the data on the profile (by switching to file system sync, deleting everything in the sync directory and forcing it to delete the local data by disabling failsafe), but the default profile won’t be usable again to sync with the original target after that, and to be honest it might actually be quicker to clear app data and sync both your profiles again, because file system sync can be really slow on newer devices.
I was afraid of that. Wish I knew that before or I would never used the default. While apps can’t be put on as installed on SD card, I do have the SD card as storage and have made a essential profile and will zap the others and download to this device.
I have been doing the revision clean out in batches of say 4000, then wait everything else syncs, then do some more. It was working. But then I thought this could be done in 3-4 hours after doing the math. Well I was right for desktop. But not android.
I have a newer device and after the desktop revision clean out, the android stops every 10 minutes or so saying fetching 2500/2500 though it varies from 2000, 4000 to 7,000 or 11,000. It says complete then waits until I click sync again. And I have been working on it for 2 days. It should have been done by now, it would be faster to load the whole giant profile, revisions all!
Guess I will have to wipe the whole android Joplin storage and resync but I thought if you swapped sync target you could go back BUT you will resync entire profile etc and take forever. not that the sync target cant ever be used again for that device.
You can re-use the sync target on the same profile, but the problem is if you have replaced the sync target contents by some means, you could either get everything duplicated, everything get deleted, or you need wait for the sync to delete all the revisions (which as you mentioned can be very slow on mobile), depending on what steps your have taken. In your case where the purpose of a full reset is to clear down the revisions, this would result in either a long sync to delete the revisons, or there is a risk that not all revisions will be deleted, which defeats the purpose and would be a big waste of time if it doesn’t work.