Generating notes ID externally

Development
1

Is it possible to force joplin to accept externally generated IDs (respecting Joplin standard for note’s ids) when creating notes?
something like:

joplin_api.create_note(my_generated_id, note_title, note_content, notebook_id)

the above would allow me to import wiki pages from an external program and precalculate inter-wiki links by precalculating Joplin ids for each note and replacing the wiki markup links with the newly calculated ids.

My current import procedure works as follows:

I have a series of wiki pages in a database (or notes that’s just up to naming convention), let’s say I have page_1 with id pageid_1, page2 with id pageid_2 etc etc.
Each wiki page just like a Joplin note has an id, a title and a body.
The body of page_1 is made up of wiki markup possibly containing internal links to other pages.

My export/import script scans all wiki pages for links and creates a python dict of wiki links -> joplin links:

links_dict = {"page_1": "3a3445e6446cd2ba1f6b165533acab20", "pageid_2":"9964695d78136d6f0b42e93db0574f03"}

the joplin ids are calculated with the function:

def get_random_joplin_id(self)
    random_uuid = uuid.uuid4()
    uuid_dehyphenated = random_uuid.hex
    return hashlib.sha256(uuid_dehyphenated.encode('utf-8')).hexdigest()[0:32]

it then replaces all links from the body of all pages from the format accepted by the wiki to the format accepted by Joplin, that is

[[page2|pageid_2]]

is replaced with

[page2](:/9964695d78136d6f0b42e93db0574f03)

it then creates a Joplin note for each page in the wiki, calling:

joplin_api.create_note(page.title, page.content, notebook_id)

for the page1 I would for instance call something like:

joplin_api.create_note("page1", "Hello this is a link to [page2](:/9964695d78136d6f0b42e93db0574f03) hope it works", notebook_id)

from the example above I hope it’s clear that when I’m creating the note for page1 I have not created the note for page2 yet so I must know in advance note2 id in order to create the correct content for note1!

The solution for me would be to have some method like:

joplin_api.create_note("3a3445e6446cd2ba1f6b165533acab20", "page1", "Hello this is a link to [page2](:/9964695d78136d6f0b42e93db0574f03) which has not been created yet", notebook_id)

the above creates a note for page1 with id “3a3445e6446cd2ba1f6b165533acab20”, which creates a link to a page2 which has not been created yet.
then I would call:

joplin_api.create_note("9964695d78136d6f0b42e93db0574f03", "page2", "Finally this is page2 !!!!", notebook_id)

2

Probably more a question for @foxmask since it depends on the Python API is implemented.

If it’s based on the REST API then creating external ID is not currently possible but could be added relatively easily.

3

The API does not provide any "page" feature as we already exchange

So in the actual state of the API, I would say :

I read all the wiki pages, store them in an array, reverse that array to start by the end, and then I'll be able to pass the id of the page in the next loop

eg:

pages = []
wiki = {'page1': 'content1', 'page2': 'content2'}

for wiki_page in wiki:
    pages.append(('page': wiki_page[0], 'content': wiki_page[1], 'next_page_id': ''})
# reverse the result
pages2 = pages[::-1]
# will become [{'page': 'page2', 'content': 'content2', 'next_page_id': ''}, {'page': 'page1', 'content': 'content1', 'next_page_id': ''}]

next_page_id = ''
for page in pages2:
    # as i start by the last page I don't generate a link, but the next will ;)
    note_id = joplin.create_note(title=page['page'],content=page['content'], next_page_id )
    next_page_id = '[' + page['page'] + '](' + note_id + ')'      #link to put at the bottom of the next note

I just made this without testing :wink:

but if @laurent can improve the API to allow external id that would do the trick

4

@foxmask your solution wouldn't make it for me because inter links between wikis are non-linear so there's no "first" and "last" link of a sequence of pages but it's a network of links with possible cycles just like a real wikipedia.

@laurent ok I've been working on the issue and here is the current status of my experiments, I finally tried inserting rows straight in the .sqlite database and strangely enough the links still don't work!!!

I checked the links in sqlite and they are correct, the syntax for links is ok still when I click on a link it either doesn't do nothing or always returns me to another note!!

Here is my code for writing directly to the sqlite database:

class Page(object):
def __init__(self, tag, title, content):
    self.guid = None
    self.tag = tag
    self.title = title
    self.content = content

class Joplin(object):
def __init__(self):
    self.wacko_links_dict = {}
    self.type_dict = {
        "notebook": "2",
        "note": "1",
        "resource": ""
    }

def create_note(self, note_id, notebook_id, title, content):
    pass

def create_notebook(self, notebook_id, title):
    pass

def generate_id(self):
    random_uuid = uuid.uuid4()
    uuid_dehyphenated = random_uuid.hex
    return hashlib.sha256(uuid_dehyphenated.encode('utf-8')).hexdigest()[0:32]

def get_datetime(self):
    return datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%f')[:-3]+"Z"

def get_epoch(self):
    return time.time()

def persist(self):
    pass

def wacko_to_joplin(self, input):
    input = re.sub(r'(?is)====([\w\W\-]+?)====', r"#### \1 \n", input)
    input = re.sub(r'(?is)===([\w\W\-]+?)===', r"### \1 \n", input)
    input = re.sub(r'(?is)==([\w\W\-]+?)==', r"## \1 \n", input)
    input = re.sub(r'(?is)=([\w\W\-]+?)=', r"# \1 \n", input)

    # replace links
    links_result = re.findall('\[\[([\w\W]+?)\]\]', input)
    if links_result:
        for link_orig in links_result:
            link = self.normalize_link(link_orig)
            if link in self.wacko_links_dict and self.wacko_links_dict[link] is not None:
                input = re.sub('\[\[{}\]\]'.format(link_orig),
                               '[{}](:/{})'.format(link_orig, self.wacko_links_dict[link]),
                               input)
            else:
                log.error("Error: link {} not found!".format(link))
    return input

def normalize_link(self, link):
    new_link = str.lower(link)
    new_link = re.sub('[\s,:-]', '', new_link)
    new_link = new_link.replace(" ", "")
    return new_link

class JoplinDatabase(Joplin):
def __init__(self, db_path):
    super().__init__()
    self.db_path = db_path
    self.connect()

def connect(self):
    self.db = sqlite3.connect(self.db_path)
    self.cursor = self.db.cursor()

def disconnect(self):
    self.db.close()

def create_notebook(self, notebook_id, title):
    epoch_time = self.get_epoch()
    self.cursor.execute('''INSERT INTO folders(id,title,created_time,updated_time,user_created_time,user_updated_time,encryption_cipher_text,encryption_applied,parent_id)
              VALUES(?,?,?,?,?,?,?,?,?)''', (notebook_id, title, epoch_time, epoch_time, epoch_time, epoch_time, "", 0, ""))
    self.db.commit()

def create_note(self, note_id, notebook_id, title, content):
    epoch_time = self.get_epoch()
    self.cursor.execute('''INSERT INTO notes(id,parent_id,title,body,created_time,updated_time,is_conflict,latitude,longitude,altitude,author,source_url,is_todo,todo_due,todo_completed,source,source_application,application_data,`order`,user_created_time,user_updated_time,encryption_cipher_text,encryption_applied)
              VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)''', (note_id, notebook_id, title, content, epoch_time, epoch_time, 0, 0, 0, 0, "", "", 0, 0, 0, "joplin-desktop", "net.cozic.joplin-desktop", "", 0, epoch_time, epoch_time, "", 0))
    self.db.commit()


if __name__ == "__main__":
wacko = Wacko()
wacko.import_pages()
wacko.disconnect()

joplin_db = JoplinDatabase("/home/user/.config/joplin-desktop/database.sqlite")
notebook_id = joplin_db.generate_id()
joplin_db.create_notebook(notebook_id, "Wacko")

# generate guids for all pages
for page in wacko.page_arr:
    page.guid = joplin_db.generate_id()
    link = joplin_db.normalize_link(page.title)
    joplin_db.wacko_links_dict[link] = page.guid

    # create pages
for page in wacko.page_arr:
    page.content = joplin_db.wacko_to_joplin(page.content)
    joplin_db.create_note(page.guid, notebook_id, page.title, page.content)

of course ignore the wacko part which simply creates an array of page titles, content and a pre-cooked joplin id.
I tried editing a link from Joplin and re-saving the note and after that some links started working, I think there's something behind the lines about links which someone must explain me.
:frowning:

5

almost OT where do urls go for drinks :sunglasses:

6

In the next version it will be possible to set the note ID, simply providing an “id” property in the POST call. Please check the API doc for more info.

7

Hello Laurent, came back to this forum to see how Joplin is doing and planning on finally migrating to Joplin.
I just checked the API (Joplin Data API | Joplin) and I see it now contains the option of creating a note with a specific ID, hurrayyy.
Do you think it's now possible to create inter links between wiki pages through the API?
If yes do you think I could keep my code above which writes to the Joplin database directly and use the API just to create empty notes with a known ID that I can then change from the database?

I would change the generate_id() method to create a random guid and call the API method

curl --data '{ "id": "00a87474082744c1a8515da6aa5792d2", "title": "My note with custom ID"}' http://127.0.0.1:41184/notes

then the create note would actually update an already inserted note.
I'll give a try anyway and post updates here.

1 Like
8

No, this wouldn't follow the REST API specs, but I'm sure you can write a wrapper to do that.