Home / GitHub Page

How to emulate a "RAW - Joplin Export Directory" to import without error?

Hi @laurent et al.,

As I work towards finishing my applescript to export Apple notes into a Joplin-ready format with original note timestamps, I need to ask about and share an error I am constantly getting, this is an example which the first of 3 example notes gives upon import and fails (starts to show content of the second note):

Missing required property: type_: Example note exported from Apple.

Example note exported from Apple.

Etc.

Etc.

id: 1501e1663f284fcfa1bcccd4eec85384
parent_id: e94e88d3ff5f406da8d35f518daa85eb
created_time: 2020-10-16T16:35:14.000Z
updated_time: 2020-10-16T16:12:41.000Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 0
user_created_time: 2020-10-16T16:35:14.000Z
user_updated_time: 2020-10-16T16:12:41.000Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
type_: 1

It's eluding me. Sometimes I seem to solve it and it imports files I create, but usually not. I'm trying to pinpoint the cause.

I've just been 'reversing engineering' by guessing what's correct from looking at my own test "RAW - Joplin Export Directory" files from original Joplin notes.

I've narrowed it down to (but I'm not even sure the totality of exactly required ingredients, yet):

  • Making sure no empty lines at the end of any .md file
  • Having a repeat of the first note line at the top in non-markdown plain text (but I had a successful import that didn't need this)
  • Empty 'resources' folder
  • an .md file representing the folder with its own 32-character alphanumeric id string referenced in the individual note .md files (I just generate these in my applescript.)
  • Making sure the ids are hexademical. (post updated)

Is it a complicated timestamp issue? creation/modified timestamp metadata in the .md files themselves? UTF vs. ____ encoding or something?

I can provide as many sample files (ZIP?) as needed to get it right. :slight_smile:

Thank you :slight_smile:

PS I can share preview of the applescript if you wanted to take a look.
PPS I test on Linux and it brings up same error: Missing required property: type_:

To follow up, here is a ZIP of a RAW Export folder that my applescript will be able to produce:

Test emulation of a "RAW - Joplin Export Directory" to import into Joplin (not working).zip (7.6 KB)

Why isn't it able to import? Any word on this @laurent? I want to finish developing my applescript to make it convenient for the user, but if the files can't import the whole thing can't happen.

Here's a preview of my script:

#WARNING: this is a pre-release version. Please only use any published version at github.com/mindfulsource
#
#Some of my TODO:
#
#If mouse was already clicked in folder area in notes, select all won't work for the note. Make mouse click/focus in note area? (automator record to grab code?)
#
#Update script to make user choose folder then it chooses that as the path. Grab that from previous code I found and saved in another .scpt?
#
#Change from cmd-c/v to select menu items for copy and paste.
#
#Also todo: try on further notes to find more html code to filter out and to see how it translates to markdown. Need to find code to convert html bulleted (<ul>) and numbered (</ol>lists to markdown if possible.
#
#Then the other todos in my own notes.

set outfolder to POSIX path of (path to home folder from user domain) & "AppleNotes2Joplin/"
do shell script "mkdir -p " & outfolder
set parent_id to ""
set parent_id to do shell script "uuidgen | tr -d '-' | tr 'A-Z' 'a-z' "
tell application "Notes"
	activate
	repeat with aNote in notes in folder "0folder" #remove everything after 'notes' to export all notes from all folders
		show aNote
		delay 0.5
		tell application "System Events"
			keystroke "a" using command down
			keystroke "c" using command down
		end tell
		do shell script "osascript -e 'the clipboard as «class RTF »' | perl -ne 'print chr foreach unpack(\"C*\",pack(\"H*\",substr($_,11,-3)))' | textutil -stdin -stdout -convert html -format rtf | pbcopy"
		delay 1.5 #Give a little delay to help avoid hiccups.
		#Add Joplin data template to each note file, including its hexademical UUID format, and convert Apple note creation and last modified timestamps to fit into it.
		set joplinid to do shell script "uuidgen | tr -d '-' | tr 'A-Z' 'a-z' "
		
		set localcreateDate to creation date of aNote
		set UTCcreatedate to localcreateDate - (time to GMT)
		set cyy to year of UTCcreatedate as integer
		set cMM to month of UTCcreatedate as integer
		set cDD to day of UTCcreatedate as integer
		set cHH to hours of UTCcreatedate as integer
		set cmins to minutes of UTCcreatedate as integer
		set csecs to seconds of UTCcreatedate as integer
		set trzcMM to text -2 thru -1 of ("0" & cMM)
		set trzcDD to text -2 thru -1 of ("0" & cDD)
		set trzcHH to text -2 thru -1 of ("0" & cHH)
		set trzcmins to text -2 thru -1 of ("0" & cmins)
		set trzcsecs to text -2 thru -1 of ("0" & csecs)
		
		set localeditDate to modification date of aNote
		set UTCeditdate to localeditDate - (time to GMT)
		set eyy to year of UTCeditdate as integer
		set eMM to month of UTCeditdate as integer
		set eDD to day of UTCeditdate as integer
		set eHH to hours of UTCeditdate as integer
		set emins to minutes of UTCeditdate as integer
		set esecs to seconds of UTCeditdate as integer
		set trzeMM to text -2 thru -1 of ("0" & eMM)
		set trzeDD to text -2 thru -1 of ("0" & eDD)
		set trzeHH to text -2 thru -1 of ("0" & eHH)
		set trzemins to text -2 thru -1 of ("0" & emins)
		set trzesecs to text -2 thru -1 of ("0" & esecs)
		
		set myText to ((do shell script "pbpaste")) & ""
		set myText to myText & "id: " & joplinid & "
parent_id: " & parent_id & "
created_time: " & cyy & "-" & trzcMM & "-" & trzcHH & "T" & trzcHH & ":" & trzcmins & ":" & trzcsecs & ".000" & "Z" & "
updated_time: " & eyy & "-" & trzeMM & "-" & trzeHH & "T" & trzeHH & ":" & trzemins & ":" & trzesecs & ".000" & "Z" & "
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 0
user_created_time: " & cyy & "-" & trzcMM & "-" & trzcHH & "T" & trzcHH & ":" & trzcmins & ":" & trzcsecs & ".000" & "Z" & "
user_updated_time: " & eyy & "-" & trzeMM & "-" & trzeHH & "T" & trzeHH & ":" & trzemins & ":" & trzesecs & ".000" & "Z" & "
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
type_: 1"
		#Clean up HTML detritus and do some basic HTML -> MD conversion
		set deleteHTMLline1 to {"<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">"}
		set deleteHTMLline2 to {"<html>"}
		set deleteHTMLline3 to {"<head>"}
		set deleteHTMLline4 to {"<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">"}
		set deleteHTMLline5 to {"<meta http-equiv=\"Content-Style-Type\" content=\"text/css\">"}
		set deleteHTMLline6 to {"<title></title>"}
		set deleteHTMLline7 to {"<meta name=\"Generator\" content=\"Cocoa HTML Writer\">"}
		set deleteHTMLline8 to {"<meta name=\"CocoaVersion\" content=\"1894.6\">"}
		set deleteHTMLline9 to {"p.p1 {margin:"}
		set deleteHTMLline10 to {"p.p2 {margin"}
		set deleteHTMLline11 to {"p.p3 {margin:"}
		set deleteHTMLline12 to {"li.li1 {margin:"}
		set deleteHTMLline13 to {"span.s1 {color:"}
		set deleteHTMLline14 to {"ul.ul1 {list-style-type"}
		set deleteHTMLline15 to {"</style>"}
		set deleteHTMLline16 to {"</head>"}
		set deleteHTMLline17 to {"<body>"}
		set deleteHTMLline18 to {"<style type=\"text/css\">"}
		set deleteHTMLline19 to {"ol.ol1 {list-style-type: decimal}"}
		set deleteHTMLline20 to {"p.p4 {margin"}
		set deleteHTMLline21 to {"p.p5 {margin:"}
		
		
		set myText to deleteLinesFromText(myText, deleteHTMLline1) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline2) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline3) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline4) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline5) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline6) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline7) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline8) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline9) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline10) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline11) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline12) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline13) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline14) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline15) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline16) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline17) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline18) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline19) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline20) of me as text
		set myText to deleteLinesFromText(myText, deleteHTMLline21) of me as text
		
		set myText to replace_chars(myText, "<p class=\"p1\">", "") of me as text
		set myText to replace_chars(myText, "<p class=\"p2\">", "") of me as text
		set myText to replace_chars(myText, "<p class=\"p3\">", "") of me as text
		set myText to replace_chars(myText, "<p class=\"p4\">", "") of me as text
		set myText to replace_chars(myText, "<p class=\"p5\">", "") of me as text
		set myText to replace_chars(myText, "</p>", "") of me as text
		set myText to replace_chars(myText, "<span class=\"s1\">", "") of me as text
		set myText to replace_chars(myText, "</span>", "") of me as text
		set myText to replace_chars(myText, "<!DOCTYPE*<body>", "") of me as text
		set myText to replace_chars(myText, "<div>", "") of me as text
		set myText to replace_chars(myText, "</div>", "") of me as text
		set myText to replace_chars(myText, "<b>", "**") of me as text
		set myText to replace_chars(myText, "</b>", "**") of me as text
		set myText to replace_chars(myText, "<i>", "*") of me as text
		set myText to replace_chars(myText, "</i>", "*") of me as text
		set myText to replace_chars(myText, "<br>", "") of me as text
		set myText to replace_chars(myText, "</body>", "") of me as text
		set myText to replace_chars(myText, "</html>", "") of me as text
		
		#Write out the file for each note.
		do shell script "printf  " & quoted form of myText & " >  " & quoted form of (POSIX path of outfolder & joplinid & ".md")
		delay 1 #Give a little delay to help avoid hiccups.
	end repeat
end tell

#Code to perform Applescript-native text replacement
on replace_chars(this_text, search_string, replacement_string)
	set AppleScript's text item delimiters to the search_string
	set the item_list to every text item of this_text
	set AppleScript's text item delimiters to the replacement_string
	set this_text to the item_list as string
	set AppleScript's text item delimiters to ""
	return this_text
end replace_chars

#Code to perform Applescript-native deletion of lines containing X.
on deleteLinesFromText(theText, deletePhrase)
	set newText to ""
	try
		set textList to paragraphs of theText
		repeat with i from 1 to count of textList
			set thisLine to item i of textList
			if thisLine does not contain deletePhrase then
				set newText to newText & thisLine & return
			end if
		end repeat
		if newText is not "" then set newText to text 1 thru -2 of newText
	on error
		set newText to theText
	end try
	return newText
end deleteLinesFromText

#Reference for the Joplin notes format: https://joplinapp.org/api/references/rest_api/#item-type-id 

Right now it just exports notes in a folder you must have named "0folder".

Once importing can work, I will make it copy one's entire Apple Notes structure with hierarchical folders and create corresponding Joplin notebook .md files inside each notes folder, so that each folder is fully ready for import after just one click.

Thanks for your help.

I can't debug your script but if you say what error you got, someone might be able to help (not necessarily me, you shouldn't expect me to answer everything on this forum, there are other users who know as much as me or more on certain topics)

I found the culprit: my files had line breaks in the form of carriage returns (CR), instead of line feeds (LF).

Convert CRs to LFs and the files will import. :slight_smile:

I'll be able to continue on making the script and it'll be called AppleNotes2Joplin once published.

1 Like

FYI I finished the script: https://github.com/mindfulsource/AppleNotes2Joplin

@ staff, it's neatest if you delete this thread? Or if not, at least readers will now know to use this link here for the updated script and not the previous post.