Exporting all data should be way easier

I have been using Joplin for about two years and was very happy to have found an app that allows me to record my notes and thoughts without depending on a cloud provider. Then I found obsidian.md and played around with it. I figured, that I could export all my notes from Joplin to Markdown and have real data in obsidian to play with. Should be easy, right? The website of Joplinapp.org currently says

"100% your data
The app is open source and your notes are saved to an open format, so you'll always have access to them."

Unfortunately, that's not exactly true.
There are effectively two data formats I can export my notes to: Markdown and JEX/Raw.
When I export everything to Markdown, the following information are lost from all the notes:

  • tags
  • creation and modification time
  • geo coordinations
  • maybe more

If I export everything to RAW, then I have a bunch of files in a format that is woefully undocumented (for example: What "type_" are there, and what do types 1,2,4 and 5 stands for? Or what is the difference between "created_time" and "user_created_time").
I wrote myself a program that should produce Markdown for me from the Raw format and append my tags. I cursed quite often about the "raw" file format and didn't take tags, or other data hiding behind the "type_" stuff. I was ultimately lucky in that I used almost no tags from Joplin, and few other features that Joplin writes into the export, but makes difficult to access.

There was an option to export to JSON. However, that has been removed. This would have made parsing the Joplin data much easier. Had I been able to export to JSON, and had the JSON data been written so that tags would be in that JSON as text (and not as an id pointing to the file where the tag is the first line in the file), then you could claim that the data would be stored in an "open format".
As it is now, I can't recommend anyone to migrate to Joplin, because they simply lose too much information when exporting. I still want to like Joplin. But now I know, that my data is not easy to get out, and that feels terrible. So I use obsidian now just because of this experience. Joplin should be better than that.

By the way, if anyone finds this post because they are looking for ways to get their data out of Joplin, drop me a line. If enough demand arises, I'll finish writing my converter and be happy to help anyone who wants a free format for their data.

9 Likes

It's trivial to write a json export plugin.

Also 100% your data doesn't mean we give you a team of developers for free to export your data in your favourite format. It just means that the data is on your computer, it's in Markdown and we provide many ways to export it, including as lossless JEX. Try to export your notes from Apple Notes and see the difference.

I would write a spec for the JEX format if it was complex, but it's so trivial that I don't see the point. Someone recently wrote a JEX importer for another note taking app, and it took him like a day, with no spec.

5 Likes

That is not true. They lose nothing. Just because you don't understand it doesn't mean you lose any data. All the metadata is there (tags, notebooks, GPS, created/updated date time, ....).

This was not what I asked for. I was frustrated that the json exporter was removed from joplin without a good alternative. And that the raw format is a more internal data representation, than an export format. When I google for the RAW format, I find mostly posts from you, telling people that they should just trial-and-error themselves through the file format. Which is not very helpful.
And this: Export to other services? which is just a frustrating answer to a legit support request.

It would cost you like 20 minutes to do this. This would help people like me a lot. So why writing about the uselessness of doing it instead of just doing it?

So basically what you're saying is:

  1. The data is yours, but you have to be able to write software to get it in a usable format (or accept that you lose information on export)
  2. Joplin compares excellent to Apple Notes, which is written by a company, that is famous for building a highly secured walled garden

I'm writing software, and I was able to write my own converter. But not everyone is. Your response confirms what I was trying to point out in my first post: you are not interested in someone who has used Joplin having an easy way to export their data from joplin without loss.

I suggest giving them either a good documentation of the JEX/raw format (and, yes, this documentation will be short, this is a good thing), or even better give them a standard format as an export target, instead of the internal backup format of joplin. Like JSON with all tags and metadata written in every file instead of linking files by their ID.

5 Likes

Technically, it is true that no data is lost for the jex or raw export. Markdown export is definitely lossy.

Markdown export is the only way for people that are unable to write their own software, and for all of them, Joplin is effectively a walled garden.

The raw "export" format needs at least two passes to make sense out of the data: one to find all the special types with their ID and in the second putting this information back into the files when writing them to the target format. The raw format is just a dump of the sqlite database and a backup format for joplin. A better export format needs only one pass of parsing and converting.

Exactly. So why on earth there is none build right into the core of Joplin? Obviously, it doesn't need a "team of developers". I could do it. But I would take a day or two (it is not trivial to write code for someone else's software) and I couldn't be sure that my PR would get through because I don't feel like the core developer is open to the idea of having a very simple export format in the first place.

This is why I started this thread. Please support the idea of having another export format, my favorite would be json. Or at least give the option to put parts of the metadata at the end of the exported markdown files. There are a lot of options to let the user know that their data is really theirs.

6 Likes

If you want JSON export you can simply write a plugin, there's no need for it to be part of the core. And that way you're free to develop it without having to wait for pull request reviews and taking the risk that the PR won't be accepted. In fact it probably won't because it's not necessary - it would be a duplicate with JEX and, although I can agree that JSON is easier to parse, it's not a good enough reason to add a duplicate functionality.

By the way, there's a demo plugin for JSON export: joplin/index.ts at dev · laurent22/joplin · GitHub

2 Likes

You can also think of this the other way. Having a documented lossless format that is easily manageable (such as JSON) will allow users to import their notes and metadata to Joplin from other note-taking software, writing custom scripts.

Without the format being documented, it is not clear if it can change anytime soon.

2 Likes

Slightly off topic but if you need to write a script to import, it may be easier to bypass the import function altogether and just use the data API

1 Like

Having JSON as a format won't make things significantly simpler on the user side - they still need to write a script, and it's all the conversion logic from one app to another that takes time. Whether you serialise to a JSON, or to a key/value file (which is JEX), it doesn't make a big difference.

It's not going to change anytime soon. It's important that the JEX format is stable because people use this for backup, so it should always be backward compatible. After many years, it still is - I think you can import a file from 2017 into the latest Joplin version.

First, I am sorry that I started the thread with anger and for being not very constructive. I really loved joplin all the time that I used it, and I'm thankful for the hard work that was put into it.

Three days ago, I switched to obsidian.md. So I have no incentive anymore to contribute to joplin. I opened this thread to document the rocky way outside and yes, it was also a rant, and I'm sorry about being snarky.

I also wanted to push the idea to make the way out of a software as easy as the way in. Especially when it handles potentially the most important data one can have.

I would like to invite you looking at this from a new perspective: Doing an export is something different from doing a backup. Backups need to be stable, are in the best case resilient to corruption (or make it easy to detect tampered data with checksums), and are as easy as possible to write. And it is very important that the software that wrote the backup can read the backup. The JEX format has not all of these characteristics (like it can break when the files are edited with other text editors).

An export format has other characteristics and goals, like being very easy parsable. And also no linking of data between files in any way, every file should be self-sufficient. This would be json, yaml, whatever, with writing all metadata into the file. This denormalizes the data, and this is a good thing.
JEX is a format that tries to be both an export and a backup, and both functionalities could be done better. So this thread has not the goal of duplicating formats in the export function. It is about clearing things up, showing a way to make joplin even better than it is today and make life easier for the next to come.

So with adding an export method to something like json you could also harden the JEX(v2) format for being better backups.

7 Likes

Reading the responses to the humble and justified request in OP, is IMO a big turn-off for anyone considering even trying Joplin, let alone moving to it from another platform.

Given enough time, anyone can become an expert at, and do anything - but no one has time for everything. Pre-made software, like any good/service, is all about removing friction. Friction kills adoption. Importing and Exporting are great for reducing friction. Documentation, especially of file formats is the only way to reduce friction for third-party developers.

Being petty and dismissive is the opposite. It's like putting sand into the grease.

9 Likes

What point are you trying to make, @HVR88 ?

Laurent, tessus, roman_r_m and others who replied are LITERALLY the opposite of petty and dismissive. They were factual and helpful, and didn't even take offense when OP demanded they spend time to cater to their needs (which they actually apologized for later and acknowledged they were angry and not very constructive at first, so absolutely no hard feelings towards OP. We all get frustrated at times).

1 Like

I made my point in the first sentence. The rest backs it up.

If your post was meant to frustrate and anger, congratulations.

4 Likes

I didn't experience the responses as petty and dismissive. But I did find the OP to be a bit on the demanding side.

I don't think that developers need to be responsible for creating export to other formats, especially if there is plugin architecture that allows for people to create their own. There are going to be new applications all the time. If someone doesn't know how to code and it's valuable to them, then they can hire someone.

3 Likes

This has been discussed many times actually. The bottom line is that there is no standard export format for notes. Calendars and events have ICS, emails have EML, images have a ton of formats, etc.

But notes don't have a standard format, so we do the next best thing - offer various lossless and lossy export formats. You can export as HTML, Markdown, plain text and JEX, which is more than Evernote offers for example (and let's not mention the vendor locking horror that is Apple Notes). And as Roman mentioned there's also the data API, which makes it easy to access the complete data set from any script.

What I've been thinking is perhaps to add some additional metadata to the exported Markdown files. For example, there have been discussions to put a YML header with extra metadata. Perhaps tags could also appear as #my-tag.

Those are just ideas at this point. Perhaps we should discuss them further to reach some kind of spec. But even all this is just a workaround due to the fact that there's no standard note export format.

5 Likes

Well, that is very interesting! Look at that: https://joplinapp.org/api/references/rest_api/
There is a complete and helpful documentation of the raw format with all the types from 1 to 16, an explanation what's the difference between user_created_time and created_time together with data types and everything. Just what I asked for, but I used the wrong terms on the search.

Do not let perfect be the enemy of the good. Writing an exporter, that creates self-sufficient text files, in a format that a lot of people are able to parse (like json) will be a good enough solution. This is not a workaround, but a solution where people can work with.

3 Likes

Well, Laurent's responses to me are the same as to everyone in all the discussions, that I found about this topic:

  1. Jex is good enough, because writing a converter should be trivial
  2. There is no standard format for exporting data
  3. They dismissed every request to adding another export format (except for the json exporter years ago, but that got removed)

This isn't really encouraging and part of my frustration in my first post was, that I'm clearly not the first one that asked about a frictionless way of exporting my data and all the answers were going nowhere.

This is especially frustrating because joplin is communicated as being "100% your data" and this claim comes with a catch: you have to write your own software to make sense out of your data, and this is for most people a very big catch. So yes, I was hoping for a more supportive response from laurent, that goes roughly like this: "yes, the export function is trying to do two different things, and I'm very open to the idea that someone adds a better exporter to joplin. Maybe I will do it because I acknowledge that users will trust the software even more, if they know that they can leave anytime without much pain".

6 Likes

This is also my takeaway from this thread. Personally I feel like markdown with YAML front matter is the closest thing to a standardized note export format. Would you accept a PR that adds a new import/export format that basically just dumps the note properties into a yaml header? If so I can write up a spec, and possibly implement it if I can find the time. I think it makes sense to include this in the core app (rather than as a plugin) because of threads like this.

No offence @anhofmann but JSON is only marginally easier to parse than the Joplin RAW files and this doesn't do much for users that can't program. Having a JSON exporter is only a mild convenience for the subset of Joplin users who are capable of using the plugin API (or data API) anyway. It just feels like JSON export has a really low value prop and you're going to struggle trying to convince someone else to put the work in.
That said, I think it is always great to have more export options, and I'm sure the community would be appreciative if you (or someone else) creates a JSON exporter.

9 Likes

To me this topic seems like a matter of priorities. You could add features and fix bug which benefits existing users or you could add export functions only useful to people leaving Joplin (maybe be a turnoff for some people considering Joplin too).

Sure, it would be good to have it all but there's only 24 hours in a day.

Yes, you are right. Json is not the best solution for everyone, but more my personal favorite because it is so easy to parse. I actually converted joplin raw to markdown with yaml frontmatter, so this could be the cover the most use cases for most people. In this regard laurent is absolutly right, there is no standard format for notes, and it is likely that there never will be one. So there is no "one size fits all" solution for this problem.