Lost 90% of my notes after switching synchronization targets

I had my Joplin notes synchronized to Dropbox and was trying to switch the synchronization target to my file system. During this switch, most of my notes disappeared, but I thought I could just reinstall Joplin (2.6.10) and synchronize from my dropbox again. When doing so, I was getting these errors.


When downloading the raw joplin directory to import, it was also missing 90% of my notes.
I am lost on what to do to retrieve my notes again.

Okay, first of all, please tell that you did have a backup before pointing Joplin to an empty folder as your sync target, right?

Secondly, if I'm getting this right then your data supposed to be still on your Dropbox account. If you have other desktop device (or another user on current PC) you can try setting up the sync target to Dropbox and sync it. Does it download all your notes?

Thirdly, do you have any other synced device that still has 100% of your notes? If you do, please disable the sync and make an export of all the notes.

Lastly, if all previous steps failed, you can visit this post to see if you can retrieve your notes from Joplin database

Also, a few relevant links for the occasion:

P.S. out of curiousity, before changing the target have you seen by chance this message?

1 Like

I did not have a backup, but the backup was synced to my Dropbox so I wasn't initially worried. I've tried downloading the Raw Joplin Directory from my dropbox and attempted a sync, it but it only reads one notebook. The directory is a big file though (~500MB) so from your answer, I'm guessing I can retrieve notes that are not syncing?

Sync is not a backup. As you might have just learned, If your db gets corrupted locally, the error will be propagated to the sync target.

So, if you notes have some value to you, you must have a backup regardless of the sync target (even if it's filesystem).

Here's a link to the plugin making automated backups

You mean you synced a clean instance of Joplin to Dropbox and it only downloaded one notebook?

Or you downloaded the folder from Dropbox and tried import it manually after which the import shows up as one notebook?

Do you have any notes without a notebook? Search -notebook:* in Joplin and see if it returns anything

I cannot know if your database is corrupt or not. The best I can do is to give you a hand in navigating the situation. There's a possibility you're having completely fine database that just need an import. Or the database is corrupt and you can only extract useless bits and pieces of your notes. To see which is it you need to try restoring deleted notes (as described in the post linked below) after the attempt of sync is complete

However, this is the last resort method, so before that we gotta make sure there's no other device having noncorrupted data.

I'm assuming you don't have any other device with your notes, is that correct?

1 Like

Here's a link to the plugin making automated backups

Thank you for that as I'm learning a very harsh lesson here.

You mean you synced a clean instance of Joplin to Dropbox and it only downloaded one notebook? Or you downloaded the folder from Dropbox and tried import it manually after which the import shows up as one notebook?

I initially tried to sync, but there were many errors due to my local database being corrupt and missing a lot of imported images. So I then downloaded the folder from Dropbox and tried to import the raw Joplin directory, which led to only one notebook being imported (which was, for the most part, intact. It was just missing images)

Do you have any notes without a notebook? Search -notebook:* in Joplin and see if it returns anything

I tried entering this command in the notebook search
image
but it didn't bring up anything.

I'm assuming you don't have any other device with your notes, is that correct?

Yes I don't :frowning:

After trying to import the raw Joplin directory as well as the different md format imports, this is the result like you said: a corrupted database with bits and pieces of my notes. At least one of my notebooks are still intact though! I'm guessing at this point my only option is using the restoring deleted notes feature?

I think what you are calling the "raw Joplin directory" is not what you think it is. I suspect you are talking about the files in the sync target, in which case they are not a 1:1 representation of your notes. The sync target contains note data, meta data and deltas designed to keep your systems in sync (hence the emphasis that it is not a backup solution) and why it seems like it is "corrupt".
To "import" the data from a sync target you need to put it within a sync target and get Joplin to sync with it.

If you think your sync target data is good then you could just start afresh locally.
Take a backup of your entire sync target (the entire Joplin folder).
Take a JEX export (just in case), close Joplin and rename your profile dir (not sure what OS you are on so the directory in help > open profile directory).

If you start Joplin it should make a new profile which you can then set up sync with.

2 Likes

Take a JEX export (just in case), close Joplin and rename your profile dir (not sure what OS you are on so the directory in help > open profile directory ).

I'm on Windows 10. When going to help, 'open profile directory,' it just opens a folder. Not sure what to do from there.

I created a copy of the sync folder renamed it dir (which, I'm assuming, is renaming my profile?) and synced Joplin to that directory. It fully copied one notebook (with pictures intact!) but it doesn't read my other 3 notebooks. It fully synced my first notebook so this is progress though.

Ok, let's try it a different way, but please don't skip or mix steps

  1. copy your Joplin profile (that folder that's opens after clicking open profile directory) directory somewhere safe (~/Documents for example),
  2. Make complete .jex export of all notes, put it somewhere safe as well
  3. go to Joplin settings > Syncronization > setup your sync target as dropbox account as usual > Show advanced settings > Push the button "Delete local data and re-download from sync target"
  4. This will reload the app and delete all the local data. But since we exported everything in step 1 and 2, it's okay.
  5. App will launch and sync should start. If it doesn't, push the sync button and setup the sync target process again if prompted
  6. Await patiently while sync is happening, it might take hours
  7. Once the sync is complete, tell us what've you got in the end, are there notes missing?
2 Likes

Followed your steps, and am currently waiting patiently; however, I hope this question isn't my impatience getting the best of me.

It has synced the first notebook that has been continuously been successfully syncing, and is now working on the next one (which is significantly larger than the first, so I can safely assume the sync time will take much longer); however, the sync has been stuck on a certain fetched item:

image

It still shows that the sync is in progress though, just stuck on 6692 fetched. Is this nothing to worry about at this moment?

It's hard to say, maybe it's the corrupted file and it will time out, maybe it's a large attachment or maybe sync indeed stuck completely.

Regardless, I can't suggest to stop the process: one way or another it will either show an error or proceed as normal. If it will result in an error, attach logs.txt from Joplin profile so we could see what went wrong.

It has still been stuck and this is the current log when syncing from dropbox. I tried downloading the sync folder from dropbox and syncing from my desktop, and the sync occurs much faster; however, it only syncs one notebook.

On canceling this is the last error:
Last error: Error: On file 6401045ecb704562a7888aa76dae9aa0.md: POST files/list_folder: Error (429): {"error_summary": "too_many_requests/..", "error": {"reason": {".tag": "too_many_requests"}, "retry_after": 300}}

Oh well.

Since you've already downloaded the sync target folder you might try deleting this file causing the problem and do the sync to that folder again.

Be sure that you indeed downloaded the folder completely and the number of items inside the downloaded folder is the same as in the cloud folder.

Enable debugging logs just in case as described here.

Also try increasing the number in Max concurrent connections to 8 (located in settings > sync > show advanced)

Once it's all done, put the file 6401045ecb704562a7888aa76dae9aa0.md in recycle bin (in other words delete it locally) and try syncing via steps 3-7. But Instead of the Dropbox account, you choose filesystem sync and point to the folder with the downloaded Dropbox folder.

if you get an error attach the logs. When sending the logs please do so in either code block text or a .txt file, no screenshots.

1 Like

I just fixed the problem.

I checked my recently deleted files in dropbox (which they save for at least 30 days), and saw that a bulk of md files had been deleted 2 days ago in a span of a couple minutes. It must have been my dumbass syncing and resyncing stuff that I shouldn't have. I restored hundreds of md files back into my dropbox, downloaded the folder and synced locally and was able to restore all of my notebooks.

Thanks for all your help. My dumbass is not worthy for your help but it is greatly appreciated. I have downloaded the plugins you recommended so this will never happen again :smiley:

That's great to hear!

In order to help others in the future and while the memory is fresh, you may document your case in detail and the cause of this behaviour (syncing and resyncing stuff) as well as user mistakes made that led you to almost losing your entire database.

In other words, it might be helpful to know what new users see as unintuitive, what prerequisites/assumptions a new user follows and their thought process while they make mistakes.

On the forum we routinely encounter people who mistakenly did some bizarre things to their own databases: for example manually deleting the database files or wiping their entire database despite several warnings. However, it's hard to pinpoint exactly what causes the confusion and what crucial piece of docs we can to improve to avoid those situations in the future.

5 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.