That’s really helpful to know, thanks for spending the time looking into it.
So what you’re saying is that the issue with the particular problematic note did not actually get solved by jex export and import? To solve it you had to manually copy the contents into a new note, instead of using the built in duplicate note function in Joplin?
This problem only arose after the JEX import.
I consider it to be an indication of undetected corrupt data.
Background:
In addition to all the more Joplin-technical issues, I am in the process of consolidating the logical content structure of my data.
In this specific case, it concerns a note in a folder that originally belonged to Joplin user ‘B’ and which he had shared with me, Joplin user ‘A’. When I left the Joplin cloud as user ‘A’ a few months ago, I also exported this folder shared by “B” with a JEX export of all data from user ‘A’ and imported this export into a new database in my Nextcloud with new profiles.
As far as I can recall, the new error process proceeded as follows after the most recent JEX import:
This one note was a kind of master note with lots of Markdown links to other notes and with a very small screenshot, which, however, was displayed with a very large Markdown string, as it turned out during the manual repair. At least, that's what happened after the new error occurred. ( If you're interested, I can check my old profile with the data from before the last JEX export to see what the Markdown for that one page looks like there. )
I performed the following action with this note when the error occurred:
The note looked normal in my Windows Joplin app.
I moved this note from the folder of Joplin user ‘B’ to a new folder using the Windows Joplin app.
After this action, the note was completely empty in the Windows app and contained no information about previous versions of this note.
All information from this note had disappeared from the Windows app.
I could still see the content of the note in my Android app. Unfortunately, I can't remember whether I found it in the old folder or already in the new folder. In the Android app, I went into edit mode, selected the entire (very large) Markdown text and transferred it to my Nextcloud using the share function. I saved the rescued data in a TXT file and am using it to gradually restore the original content manually.
I think the content of the note is probably irrelevant. There has been various bug fixes made to Joplin Server in recent months relating to note shares, so this may be the result of a bug which was present while you were using Joplin Cloud, which put the metadata in an invalid state which is retained in the jex backup. Or this could be a bug regarding management of imported notebooks originating from note shares, even when the state of the metadata is valid.
Presumably, if you import the full jex to a new profile on the desktop app, export the notebook which originated from a share as just ‘markdown’, then delete the notebook and reimport the markdown export via the ‘markdown folder’ option, you would not get any issues with it then. That is because exporting as markdown will strip all the metadata from the notes and notebooks (possible the created and updated times as well).
You may also be able to successfully strip off any problematic metadata if you export to ‘markdown and frontmatter’ and reimport that instead. This will retain note metadata but probably wont retain any metadata about the notebooks themselves.
Thank you very much for all this information.
Especially about the difference between the actual content of the note and the metadata.
About the distinction between note metadata on the one hand and notebook metadata on the other.
And about the two export and import formats, which were new to me.
I also learned a lot about the possibility that errors in earlier versions of the Joplin Cloud Server could have caused invalid metadata, which is retained when exporting and importing in JEX format.
Since I have four different notebooks from four different share operations and have already moved some notes between notebooks, I still need to consider how I will actually use the information about the two new export and import formats.
Until then, it's better to refrain from moving any more notes between these folders.
Perhaps I should try out these new formats in my test profiles first and gain some experience.
I prefer the ‘Markdown and Frontmatter’ variant because it doesn't remove all metadata.
I have a few initial questions on this topic:
a) Are new object IDs also assigned during import with the two formats mentioned?
b) Are the Markdown links retained with these formats?
c) What would be the disadvantages of a radical approach in which the complete data is exported and re-imported via ‘Markdown and Frontmatter’?
a) Are new object IDs also assigned during import with the two formats mentioned?
Every import method assigns new note ids for all notes
b) Are the Markdown links retained with these formats?
I have tested this and the internal links are retained using both the Markdown and the Markdown + Frontmatter formats, as the links are re-pointed to the new note ids upon import
c) What would be the disadvantages of a radical approach in which the complete data is exported and re-imported via ‘Markdown and Frontmatter’?
To be honest I’m not sure there are any disadvantages. The Markdown and Frontmatter format will retain all note metadata available in the note properties screen, all note links, all tags and the full notebook hierarchy. Note history is not included in the export but that isn’t included in jex exports either. Of course it is best to hold onto the jex export just in case something does get lost.
But the only possible things I can think of which may be lost from the format is notebook metadata relating to shares or the origin / owner if that information is stored (which is not important or relevant outside of Joplin Cloud), or if your note is a html note which originated from another app such as Evernote, there could potentially be some loss during conversion in terms of formatting of a note. In that case I’m not sure if it will convert it to a markdown note or if it is kept in the original format
EDIT: Actually I do recall one particular disadvantage that you need to be weary of. Because the Markdown and Frontmatter format exports to a file structure rather than an internal flat structure, notebook and note names are truncated at 50 characters to mitigate the possibility of reaching the OS path limit. Also some characters may be stripped from notebook and note names which aren’t legal in filenames. So after doing the export and re-import process you would need to verify and correct any names which may have been changed or truncated
Thank you very much. I will try to import my production data into a test profile this way.
I made a first attempt.
Export with ‘MD - Markdown + Front Matter’ to C:/md
Copied the directory C:/md to an external SSD drive to F:/md and safely unmounted the SSD.
Connected the external SSD drive to another computer and imported it from there into my ‘Temp-2’ profile.
Import with ‘MD - Markdown + Front Matter (directory)’
After a few minutes, the following error message appeared
Invalid package F:\md\a -
Notebook\C:Users\User\AppData\Local\Programs\Joplin\resources\app.asar
I confirmed this with ’OK".
What bothers me about the result is that not a single screenshot made it into the notes.
Only symbols for broken images are displayed.
EDIT:
The Markdown links aren't working.
P.S.
The first attempt resulted in a subdirectory that cannot be copied or deleted.
The name of the Joplin folder and a note contained therein each contained over 50 characters and the special character &.
In my export directory C:/mdxxx, I now have a subdirectory that cannot be deleted or renamed.
Before the second export attempt, I gave the corresponding Joplin directory and the corresponding note a short name without special characters.
In my export directory C:/mdxxx, I now have a subdirectory that cannot be deleted or renamed.
The fact you cannot delete it suggests the OS path limit may have been reached, if you still can’t delete it after restarting your pc
The Markdown links aren't working.
Probably because the import resulted in a failure, it did not reach the markdown link substitution stage.
It’s interesting that there were no reported issues with the export, but the import resulted in issues
I think it’s probably not a good idea to export and re-import all your data from Markdown + FrontMatter given these issues. An alternative to strip share metadata would be to make manual modifications to the jex file before importing it, though this process would be more technical. A jex file is a tar achive. You could rename it to have the .tar extension and then extract the full contents into a directory. This can then be imported via the import RAW - Joplin Export Directory option, after making some changes to the metadata.
There are 2 attributes relating to shares in each note and notebook file. These are is_shared and share_id. To clear the share metadata, the is_shares value would need to be set to 0, and the share_id would have an empty value (a single space after the colon). The thing is though, you would need to find the correct files to change these values on, based on the uuid in the filename. I suspect in your case the issue is with the notebooks only, and not the notes, so would require not so many changes.
You can get the uuid for a note by going to the note properties dialog on Joplin desktop, then find the .md file matching this id. In the md file is an attribute called parent_id, which will correspond to the notebook the note is within. If the notebook is nested, the parent_id of the parent file would contain the id its parent, and so on. You would need to find the relevant parent id that way, unless you know sql and opened the sqlite db directly to find out the mappings of id to notebook names.
I’m not sure your level of technical knowledge so this may not be worth bothering to do. Do your previous shared notebooks actually show as being shared in the desktop UI like this?
If they don’t then they don’t actually have the share attributes set anyway. But I did try manually setting the metadata for a notebook to be shared and importing it, and I did not have any problem with data being lost when moving a note out of this notebook, so this may have just been an isolated issue for the 1 particular note, or a rare bug (in particular it may have been this bug or related to Fix handling of conflict when a folder title is changed · Issue #11902 · laurent22/joplin · GitHub)
I moved this note from the folder of Joplin user ‘B’ to a new folder using the Windows Joplin app.
After this action, the note was completely empty in the Windows app and contained no information about previous versions of this note.
I can’t comment exactly on the cause of the issue, but this PR Desktop, Mobile: Fix historic issue whereby the first revision created for a note does not contain the original contents by mrjo118 · Pull Request #12674 · laurent22/joplin · GitHub if it is accepted, should address the content being unrecoverable via the note history. Note you would need to either wait 10 minutes or restart the Joplin app for the note history to become available in that scenario
Yes, I agree with that now.
No. After importing the JEX file exported from Joplin Cloud into my NextCloud, there was no longer any reference to ‘shared’. This remains the case to this day.
Perhaps I will try your original suggestion again, exporting individual previously shared folders with “MD - Markdown + Front Matter” and importing them again. Before doing so, I would manually check the content for long names and special characters in the names.
I would first try the import again in a test profile.
If that doesn't work either, I can still manually rebuild the pages I actually want to move at the destination and delete the old note at the source location.
I can roughly understand the other approaches with the .tar file, but I don't have the necessary experience to really try it myself.
That could well have been the case. I restored my three-day-old backup of this virtual machine because neither restarting the virtual machine nor the VirtualBox host machine had solved the problem. Now everything is fine again: both Joplin and my virtual Cryptomator drives have their current contents.
I have now reset my test profile in NextCloud and all the apps involved and tested it with a single folder. I transferred the data to the other computer using an external drive, which was mounted as drive F: in both cases.
The good news is that the screenshots were transferred cleanly.
The bad news is that the Markdown links were not converted correctly.
I assume that the import function of “MD - Markdown + Front Matter (directory)” enters a file path that it expects under
/C:/Program%20Files
instead of the new IDs. This algorithm might work if the import were performed not directly from the F:/x folder but instead from the /C:/Program%20Files folder – but I'm definitely not going to try that.
The correct Markdown link should have looked like this:
[Best wishes](:/36e533daebe34cc9a15e88d7e009378f)
However, the import function entered the following Markdown link:
[Best wishes](/C:/Program%20Files///Best%20wishes.md ‘../../..///Best%20wishes.md’)
If you have note link references to notes which are not included in the data you are importing, then I suspect the link replacement logic might not work correctly for those links
I had already checked that before.
Now I have checked it again to be on the safe side.
The two notes involved in the link are located below the folder that I exported and re-imported.
The link works in the original folder that I exported.
The error described occurs in the imported folder.
I would like to describe their position as follows:
Below the exported folder are two folders, each with their own subfolders.
If I refer to the exported folder as ‘export’ and the subfolders as ‘A’ and ‘B’, then the note with the faulty Markdown link is located at the following position:
‘export’ -> folder ‘A’ -> folder ‘A1’ -> note ‘A123’.
‘export’ -> folder ‘B’ -> note ‘Best wishes’
is the location to which the Markdown link should point.
I’m not able to reproduce your issue. When I create a similar structure and export as markdown an frontmatter I get relative file paths in the markdown links rather than absolute paths, and they correctly get converted when imported
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
