Python script for importing notes from Synology's Note Station into Joplin

spurioustoad · 2020-06-17T23:06:36.577Z

So excited to see this, thank you!

Trying to get it to work, but am stymied.
I created a test notebook in Note Station - 4 notes, including 2 attachments.
When I run the script it creates the appropriate directories and places the attached files in the attachments folder, but no md (or any other) files are created.

Here is what the output of the script looks like.
Perhaps someone can point me in the right direction?

Found pandoc 1.19.2.4
Extracting notes from /home/raven/notestation-to-joplin/src/20200616_125920_2699_me.nsx
Reading notebook Joplin Import Test Notebook
Reading note 1/4: A Joplin Import note with an attachment
Reading note 2/4: A Joplin Import note with embedded links
Reading note 3/4: A Joplin Import note with an embedded image
Reading note 4/4: A Joplin Import test note
Writing note 1/4 in Joplin Import Test Notebook: A Joplin Import note with an attachment
Traceback (most recent call last):
File "nsx2joplin.py", line 463, in
nsx.export_to_joplin(token=joplin_token, nsx_content=nsx_content)
File "nsx2joplin.py", line 406, in export_to_joplin
note["content"] = asyncio.run(
File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 608, in run_until_complete
return future.result()
File "nsx2joplin.py", line 345, in create_resource
res = await joplin.create_resource(
File "/home/raven/.local/lib/python3.8/site-packages/joplin_api/core.py", line 474, in create_resource
return await self.query('post', '/resources/', **data)
File "/home/raven/.local/lib/python3.8/site-packages/joplin_api/core.py", line 106, in query
res = await self.client.post(self.JOPLIN_HOST + '/resources',
File "/home/raven/.local/lib/python3.8/site-packages/httpx/_client.py", line 1374, in post
return await self.request(
File "/home/raven/.local/lib/python3.8/site-packages/httpx/_client.py", line 1147, in request
response = await self.send(
File "/home/raven/.local/lib/python3.8/site-packages/httpx/_client.py", line 1168, in send
response = await self.send_handling_redirects(
File "/home/raven/.local/lib/python3.8/site-packages/httpx/_client.py", line 1195, in send_handling_redirects
response = await self.send_handling_auth(
File "/home/raven/.local/lib/python3.8/site-packages/httpx/_client.py", line 1232, in send_handling_auth
response = await self.send_single_request(request, timeout)
File "/home/raven/.local/lib/python3.8/site-packages/httpx/_client.py", line 1264, in send_single_request
) = await transport.request(
File "/home/raven/.local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 152, in request
response = await connection.request(
File "/home/raven/.local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 78, in request
return await self.connection.request(method, url, headers, stream, timeout)
File "/home/raven/.local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 62, in request
) = await self._receive_response(timeout)
File "/home/raven/.local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 115, in _receive_response
event = await self._receive_event(timeout)
File "/home/raven/.local/lib/python3.8/site-packages/httpcore/_async/http11.py", line 145, in _receive_event
data = await self.socket.read(self.READ_NUM_BYTES, timeout)
File "/home/raven/.local/lib/python3.8/site-packages/httpcore/_backends/asyncio.py", line 134, in read
return await asyncio.wait_for(
File "/usr/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
return fut.result()
File "/usr/lib/python3.8/asyncio/streams.py", line 701, in read
await self._wait_for_data('read')
File "/usr/lib/python3.8/asyncio/streams.py", line 534, in _wait_for_data
await self._waiter
RuntimeError: Task <Task pending name='Task-4255' coro=<StreamReader.read() running at /usr/lib/python3.8/asyncio/streams.py:701> cb=[_release_waiter(()]>)() at /usr/lib/python3.8/asyncio/tasks.py:429]> got Future attached to a different loop

relink · 2020-07-22T13:58:01.493Z

Im trying to run this script and it seams to read all my notes but begins to fail when trying to write them into Joplin. Im not sure how to fix this, but I know I cant switch if I cant get my notes out.

Traceback (most recent call last):
  File "nsx2joplin.py", line 463, in <module>
nsx.export_to_joplin(token=joplin_token, nsx_content=nsx_content)
  File "nsx2joplin.py", line 382, in export_to_joplin
joplin_id = asyncio.run(create_folder(notebook["title"]))
  File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
  File "nsx2joplin.py", line 332, in create_folder
res = await joplin.create_folder(folder=notebook_title)
  File "/home/relink/joplinimport/joplin-api/joplin_api/core.py", line 296, in create_folder
return await self.query('post', '/folders/', **data)
  File "/home/relink/joplinimport/joplin-api/joplin_api/core.py", line 116, in query
res = await self.client.post(full_path, json=payload, params=params)
  File "/home/relink/.local/lib/python3.8/site-packages/httpx/_client.py", line 1374, in post
return await self.request(
  File "/home/relink/.local/lib/python3.8/site-packages/httpx/_client.py", line 1147, in request
response = await self.send(
  File "/home/relink/.local/lib/python3.8/site-packages/httpx/_client.py", line 1168, in send
response = await self.send_handling_redirects(
  File "/home/relink/.local/lib/python3.8/site-packages/httpx/_client.py", line 1195, in send_handling_redirects
response = await self.send_handling_auth(
  File "/home/relink/.local/lib/python3.8/site-packages/httpx/_client.py", line 1232, in send_handling_auth
response = await self.send_single_request(request, timeout)
  File "/home/relink/.local/lib/python3.8/site-packages/httpx/_client.py", line 1264, in send_single_request
) = await transport.request(
  File "/home/relink/.local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 152, in request
response = await connection.request(
  File "/home/relink/.local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 65, in request
self.socket = await self._open_socket(timeout)
  File "/home/relink/.local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 85, in _open_socket
return await self.backend.open_tcp_stream(
  File "/home/relink/.local/lib/python3.8/site-packages/httpcore/_backends/auto.py", line 38, in open_tcp_stream
return await self.backend.open_tcp_stream(hostname, port, ssl_context, timeout)
  File "/home/relink/.local/lib/python3.8/site-packages/httpcore/_backends/asyncio.py", line 233, in open_tcp_stream
return SocketStream(
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
  File "/home/relink/.local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
raise to_exc(exc) from None
httpcore._exceptions.ConnectError: [Errno 111] Connect call failed ('127.0.0.1', 41184)

tessus · 2020-07-22T14:46:52.359Z

Your web clipper server is not running. Go to Joplin Preferences -> Web Clipper -> Step 1: Enable Web Clipper Service

relink · 2020-07-23T12:32:45.527Z

I didn’t notice that, thanks. Interestingly I’m still having issues though.

So in the folder where the script is located I get a folder created for each of my notebooks, and within each folder are any attachments my notes had. However in Joplin only 1 notebook transferred over, and none of the notes come with it. So I have 1 empty notebook that moved over…

tessus · 2020-07-23T13:44:40.090Z

I didn’t write the script, so we have to wait until @KraxelHuber can have a look.

relink · 2020-07-28T16:17:49.065Z

Well hopefully I can get it sorted out soon. I don’t mind the Synology apps, except in this case because I cant easily get my notes back out “intact”.

Im really wanting to give Joplin a shot, especially since it now has a wysiwyg editor.

relink · 2020-07-30T13:08:08.531Z

I am now stuck at the exact same error. Hopefully we can get it sorted soon.

zerosandones · 2020-11-12T17:50:11.964Z

Likewise. relink and spurioustoad - this is on FC33 with Python3.9
Exporting fine. about 300M's worth as an nsx file.
Reads in 1004/1004 notes, and then the Traceback ending in:

RuntimeError: Task <Task pending name='Task-12' coro=<StreamReader.read() running at /usr/lib64/python3.9/asyncio/streams.py:684> cb=[_release_waiter(<Future pendi...a339858e0>()]>)() at /usr/lib64/python3.9/asyncio/tasks.py:414]> got Future <Future pending> attached to a different loop
/home/user/.local/lib/python3.9/site-packages/httpx/_client.py:1781: UserWarning: Unclosed <httpx.AsyncClient object at 0x7f1a3396f580>. See https://www.python-httpx.org/async/#opening-and-closing-clients for details.

Looking at the tail log-clipper.txt I can see that it is generating:

2020-11-12 17:08:11: "Request: POST /folders/?token=2b9344de...33

But no errors to light the way as such. Straws to clutch welcome.

zerosandones · 2020-11-13T12:46:04.753Z

Regretfully same issue seen with Ubuntu Focul / Python 3.8.
Those people who did get this working - what specifically were you running on what?

foxmask · 2020-11-13T14:36:05.815Z

the error is simple.
on line notestation-to-joplin/src/nsx2joplin.py at master · andreas-vester/notestation-to-joplin · GitHub
it should be

from joplin_api import JoplinApiSync as JoplinApi

the author did not notice that I changed the JoplinApi class from "sync" to "async"
so the workarround is

from joplin_api import JoplinApiSync as JoplinApi

and nothing else has to be changed and that should work

zerosandones · 2020-11-13T16:49:51.650Z

foxmask:

from joplin_api import JoplinApiSync as JoplinApi

Line 16 - understood.

It's now throwing errors about 'await'. Removing await from the res= appears to resolve this. Which is lucky as it was a complete guess. So far the rest seem to be images that have not made the export properly or similar - so far so good.

Oh, and its only a simple error if Python is your thing Thank you so very much for your assistance foxmask

foxmask · 2020-11-13T17:11:30.561Z

ha, i didnt see the await later ; so no changed to be done. I check the first complet error message

as already pointed by ressus, the webclipper was not started, for the folder creation I dont know.

sorry for the noise.

zerosandones · 2020-11-13T17:18:12.449Z

If someone is re-working this - I am currently enjoying renaming .jpg files in the tmp directory to match what files suggests they are so it doesnt choke trying to import them:

2020-11-13 17:15:02: "Error: Image is invalid or does not exist: /tmp/Nrdmvg7e_6tyD5bab-EplW_l.jpg
Error: Image is invalid or does not exist: /tmp/Nrdmvg7e_6tyD5bab-EplW_l.jpg
    at handleResizeImage_ (/tmp/.mount_JoplinCHyX4H/resources/app.asar/lib/shim-init-node.js:121:31)
    at Object.shim.createResourceFromPath (/tmp/.mount_JoplinCHyX4H/resources/app.asar/lib/shim-init-node.js:225:21)"

It is nice to know someone is checking the attachments : )

[user@thinkpad src]$ file /tmp/Nrdmvg7e_6tyD5bab-EplW_l.jpg
/tmp/Nrdmvg7e_6tyD5bab-EplW_l.jpg: SVG Scalable Vector Graphics image

These mostly seem to be emoticons. What is wrong with ASCII I ask you?!
copy paste generic for now. Thanks again

bananasareyellow · 2021-03-13T17:45:35.139Z

Thank you @zerosandones for working your way through this. I'm now following your path. Looking at the GitHub repository, it seems the author has not worked on this since 02-Mar-2020, so none of your fixes have been applied. Anyway, I'm stuck at a new error.

I changed the import statement from:

from joplin_api import JoplinApi

to

from joplin_api import JoplinApiSync as JoplinApi

Then I changed the 3 occurrences of:

res = await some_function

to remove the await like this:

res = some_function

Now it gets through all my "Reading note" steps, and then it gets through 174/701 of my "Writing note" steps before failing:

Traceback (most recent call last):
  File "nsx2joplin.py", line 467, in <module>
    nsx.export_to_joplin(token=joplin_token, nsx_content=nsx_content)
  File "nsx2joplin.py", line 410, in export_to_joplin
    note["content"] = asyncio.run(
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "nsx2joplin.py", line 348, in create_resource
    res = joplin.create_resource(
  File "/home/david/.local/lib/python3.8/site-packages/joplin_api/core_sync.py", line 508, in create_resource
    return self.query('post', '/resources/', **data)
  File "/home/david/.local/lib/python3.8/site-packages/joplin_api/core_sync.py", line 123, in query
    res.raise_for_status()
  File "/home/david/.local/lib/python3.8/site-packages/httpx/_models.py", line 1108, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: 500 Server Error: Internal Server Error for url: http://127.0.0.1:41184/resources?token=5e9fa90fa40a75d5033b2dc4e8258e108b6a91a361c30a245cd587f5f8fe1456f03c2b08ddd49e50d85b6a6c8904b0f0c144b2b5b333174aa6c2b7405395cc61
For more information check: https://httpstatuses.com/500

I'm using Python 3.8.6.

Any idea what's causing this?

bananasareyellow · 2021-03-13T18:34:44.288Z

OK, so here is my workaround. Luckily each "Writing note" output line includes the title of the note, so I went back to Synology Note Station to find the offending note. It was not important, so I moved it to a new notebook called "Transfer Errors". Then I exported from Synology Note Station again but I excluded the "Transfer Errors" notebook.

I had to do this for 3 or 4 notes in the end, but now I'm done.

bananasareyellow · 2021-03-13T18:49:57.265Z

Well, my conversion is complete but all of my inline images are missing.

They all seem to have been converted to this:

![]
(webman/3rdparty/NoteStation/images/transparent.gif)

Examining a particular note and comparing it to the original in Synology Note Station, looking in the attachments subfolder for the converted notebook, I can see that the image (PNG file) is there, but correctly linking to it seems to have failed.

tessus · 2021-03-13T19:00:53.160Z

bananasareyellow:

it seems the author has not worked on this since 02-Mar-2020, so none of your fixes have been applied

Someone could fork the repo and apply the fixes. We can then create a new topic and mark this one as abandoned. As long as we mention the original author this should be fine. It's open source after all.

bananasareyellow · 2021-03-14T17:39:54.905Z

The error is in the function create_resource. The test that says:

if attachment_type == "image":

should be changed to:

if attachment_type == "image/png":

Now it works. Note that if the image was resized in Synology Note Station, the size information is lost. The image transferred to Joplin is displayed at full size but you can resize it again manually.

In summary, there are 3 changes to be made to nsx2joplin.py:

Change the import declaration.
Remove await in 3 places.
Change the attachment_type test.

tessus · 2021-03-14T20:04:35.556Z

You should test if the attachment_type starts with image, unless you really want to only target png.

JackGruber · 2021-03-14T20:38:19.927Z

You can use:

if attachment_type[:5] == "image":