General Usage Pattern Questions

[dsimmons]

Hello all! New to the Joplin product/community. Currently in the process of figuring out how to best incorporate Joplin into my existing "notes" workflow and/or determine if Joplin is the right fit. Accordingly, I'd very much appreciate any input more experienced users can provide in terms of what follows!

Context

As a technically-inclined user (software engineer, Linux), I already write all of my notes in Markdown on my desktop/laptop, organized into a folder/note hierarchy.

As a contrived/fictitious example:

notes/
  - note-a.md
  - note-b.md
  - project-X/
    - note-c.md
    - note-d.md
  - todos/
    - today.md
    - this-week.md
  - lists/
    - grocery-list.md

I edit these Markdown files using Vim/Neovim (again, on desktop/laptop). Strictly from a writing perspective, I have no desire to use another non-Vim tool, for productivity reasons. From a consumption (read) standpoint, although I have all of the live-preview goodies in my editor already, I'm open to incorporating other/additional tooling.

NOW, where this approach falls short for me is on mobile, where I find myself often either wanting to add things to my notes (spontaneous thoughts), mark items/todos completed (e.g. checklists), and/or consume from my notes (reference, learning, etc...).

So, that's to say, the latter is, first and foremost, the value that I'd hoped to get out of Joplin.

Finally, for what it's worth, I'm achieving syncing across all of the aforementioned devices using SyncThing (which I'm already using). In the case of the Joplin Android app, I'm using the local filesystem syncing, where that particular location on the Android filesystem syncs to my other devices.

Observations

My initial hope was that I could leave my existing Vim-based vanilla Markdown approach in place on desktop/laptop and make use of the Joplin Android app to view and interact with the rendered Markdown, kind of achieving "the best of both worlds" -- raw Markdown when I have Vim and a keyboard available to me, but a more user-friendly rendered version that supports interaction when that's not the case.

However, based on my initial experiments, that proved not to work well given the following observations:

• As far as I've been able to tell, at least in a mobile-only capacity via Joplin, I'm unable to import existing (named/structured) Markdown files, even provided the directory on the local filesystem where they exist already.

• The notes Joplin produces are named via what appears to be some digest/hash.

  • Per the FAQs, I understand this was a conscious design trade-off.
  • However, this effectively means my plain Markdown filesystem structure becomes obsoleted, assuming I buy into the "Joplin structure" wholeheartedly on desktop/laptop (I'm guessing).
  • Continuing from the last point, I observed that my filesystem then gets littered with all sorts of opaque "Markdown" files. Disabling "note history" seems to have reduced churn/noise as far as I can tell, but it still obviates what I had in place previously. As for why I just quoted "Markdown", see below.

• If I edit the Markdown files that Joplin mobile produces on desktop, the mobile app seems to barf.

  • Per some quick Googling, I saw something that suggested that, although these files have a .md extension, they're really more like "database files" that aren't meant to be edited.
  • Even if I leave all of the associated metadata at the bottom untouched, the above seems to be the case (the app refuses to load externally-edited files).

Questions

• Given what I've outlined above, does anyone have any concrete recommendations/prescriptions on an alternate workflow that'd achieve the goals I have?

• If I was to "commit" to the Joplin approach, I'm assuming I'd need to use the CLI tool to import all of my existing vanilla Markdown files/structure?

• It's my understanding that you can, at least in the Terminal version of Joplin, make use of "external" editors (Vim, Neovim). What does this look like concretely?

Thanks in advance for your time and consideration! I'm still pretty excited about the idea of this project on the whole, but I'm currently trying to understand how to best "fit" my existing/desired workflow into this tool's paradigms, or alternatively, arrive at the conclusion that this tool doesn't meet my needs.

[laurent]

You can import your notes either using the cli tool or desktop app, by importing a directory of Markdown files.

And yes you can use vim with the cli tool.

[Daeraxa]

@dsimmons

As far as I've been able to tell, at least in a mobile-only capacity via Joplin, I'm unable to import existing (named/structured) Markdown files, even provided the directory on the local filesystem where they exist already.

@dsimmons

If I was to "commit" to the Joplin approach, I'm assuming I'd need to use the CLI tool to import all of my existing vanilla Markdown files/structure?

No, CLI isn't required (but you can still use it), you can also do it with the desktop app. It is worth having the CLI or desktop app around simply because some features are unavailable on mobile and it can save a lot of problems if you have any data issues later on.
See readme for importing instructions - including markdown directory import to preserve the file structure - https://joplinapp.org/help/#importing-from-markdown-files

@dsimmons

The notes Joplin produces are named via what appears to be some digest/hash.

@dsimmons

If I edit the Markdown files that Joplin mobile produces on desktop, the mobile app seems to barf.

As you seem to have worked out these are files purely for syncing and should not be edited manually at risk of breaking the sync target. They are not a local copy of the notes, your actual note data you interact with via the clients is all stored within the Joplin directory in a sqlite database. Joplin does not work using plain markdown files, it requires notes to be imported and once imported you cannot simply interact with the note without using Joplin in one sense or another - the original file is no longer required, it is all now in the database.

@dsimmons

It's my understanding that you can, at least in the Terminal version of Joplin, make use of "external" editors (Vim, Neovim). What does this look like concretely?

Both CLI and desktop apps support external editing. e.g. putting nano as the CLI external editor will simply open nano when you attempt to edit the note, in the desktop app putting something else like Mark Text, Typora etc. will launch that application in a new window.
The note opens as a hashed temporary .md file that Joplin watches for changes and imports into the database once saved.
Example opening in Mark Text:

Basically yes, if you move to Joplin you will pretty much need to move everything into the Joplin ecosystem - you can't (easily) use Joplin as a "viewer" for markdown files you are creating and maintaining on a different machine.
There are certain things that can help, for example the Hotfolder plugin is somewhere you can drop files into for automatic import into Joplin.

Joplin does also have a lot of export options, including .md files so you aren't stuck in Joplin forever should you need to move your notes out.

If working and editing .md files directly is more important then Joplin might not be the right tool vs some other applications that use .md files as the notes database directly.

Otherwise feel free to ask any questions if anything hasn't been answered.

[dsimmons]

Thanks for the detailed response, I sincerely appreciate it!

@Daeraxa

See readme for importing instructions - including markdown directory import to preserve the file structure - What is Joplin? | Joplin

Doh! I don't know how I missed this section of the docs

@Daeraxa

As you seem to have worked out these are files purely for syncing and should not be edited manually at risk of breaking the sync target. They are not a local copy of the notes, your actual note data you interact with via the clients is all stored within the Joplin directory in a sqlite database. Joplin does not work using plain markdown files, it requires notes to be imported and once imported you cannot simply interact with the note without using Joplin in one sense or another - the original file is no longer required, it is all now in the database.

Thanks for clearing that up!

As a feature suggestion, and I think I saw this mentioned as something on the tentative roadmap, it'd be helpful to make this more explicit, whether with the file extension (something non-.md, I think I saw .bin proposed) and/or with some sort of "DO NOT EDIT" comment across the top, much like how the metadata section exists at the end.

@Daeraxa

The note opens as a hashed temporary .md file that Joplin watches for changes and imports into the database once saved.

Ah yes, makes sense, that's basically what I would have expected without yet having test-driven the pertinent desktop components on my Arch-based distro (partially because I was suspicious of the way in which it'd affect my existing filesystem structure).

@Daeraxa

There are certain things that can help, for example the Hotfolder plugin is somewhere you can drop files into for automatic import into Joplin.

Thanks for the tip, I'll check that out!

@Daeraxa

Joplin does also have a lot of export options, including .md files so you aren't stuck in Joplin forever should you need to move your notes out.

Yeah, this is what's sounding like the next logical "experiment" to increase my confidence around what that "escape hatch" looks like before "sending it" and committing fully.

@Daeraxa

If working and editing .md files directly is more important then Joplin might not be the right tool vs some other applications that use .md files as the notes database directly.

Thanks so much for your candid feedback, that's exactly what I was after! Out of curiosity, are there any such tools ("Markdown files as the database") that you know of off of the top of your head so I can do a side-by-side evaluation before arriving at a decision? No worries if not, I can do some Googling on my end

[dsimmons]

As a logical follow-up question(s):

• I assume the approach I outlined by syncing a filesystem directory across devices (using Syncthing in my case) is a valid "pattern" given that Dropbox is a supported option, right?

• Along the lines of the previous question, and maybe I can find this in the docs somewhere, does this SQLite database live within the specified directory (e.g. ~/notes) or outside of it (e.g. ~/.config/joplin or something)?

  • At the very least, I'd want to ensure that I have it covered in my backup processes.
  • At the point that Joplin "owns" the Markdown inside of a database, of what importance are those actual files / that specified directory? Is it just ephemeral "scratch space" for Joplin?

Thanks again for your patience!

[Daeraxa]

@dsimmons

Thanks so much for your candid feedback, that's exactly what I was after! Out of curiosity, are there any such tools ("Markdown files as the database") that you know of off of the top of your head so I can do a side-by-side evaluation before arriving at a decision?

The two biggest ones I came across when looking for an Evernote replacement (for which I eventually settled on Joplin) were Obsidian ((mostly) free but not FOSS) and QOwnNotes (I wasn't sold on the UI/UX + no mobile app). Both of those I believe work on .md "databases".

@dsimmons

I assume the approach I outlined by syncing a filesystem directory across devices (using Syncthing in my case) is a valid "pattern" given that Dropbox is a supported option, right

The file system sync target is a perfectly valid method of syncing, quite a lot of users use this mode in combination with syncthing etc.

The Dropbox sync (as well as OneDrive, NextCloud, WebDAV etc.) doesn't operate using the local file system, it works directly which is why it requires the username/password of those targets. You don't actually have to have that data on your local machine at all (i.e. the sync target data is purely online).

@dsimmons

Along the lines of the previous question, and maybe I can find this in the docs somewhere, does this SQLite database live within the specified directory (e.g. ~/notes ) or outside of it (e.g. ~/.config/joplin or something)

It lives in the profile directory which is, as default on linux as you found, ~/.config/joplin (CLI) and ~/.config/joplin-desktop (desktop app) which has the sqllite database along with other client specific things like your resources folder (images, attachments etc.), plugin installs and settings. This is indeed the area you would want to regularly back up - along performing a JEX export (joplin lossless data file) which maintains all the internal metadata as well as the notes themselves.
If you need to put this somewhere else you can either use the (unsupported) --profile flag to set a different directory or you can simply symlink.
It is also worth noting that the CLI and desktop clients are different clients in their own right and by default cannot access the same database at the same time - you have to actually sync between them in the same way as clients on different machines unless you configure something (at your own risk) to allow them access to the same sqlite database if you can ensure only one client is ever running at a time.

@dsimmons

At the point that Joplin "owns" the Markdown inside of a database, of what importance are those actual files / that specified directory? Is it just ephemeral "scratch space" for Joplin?

Not quite sure what you refer to here, you mean the sync target files? If so then these are used for all kinds of things, full notes, note revisions etc. as well as metadata so that each client can sync correctly based on changes made to any local notes. They are vital to how the sync works so that any new client you connect up can get the full set of notes, resources and note versions. It isn't simply a swap space or some kind of journalling sync between two clients, its a full notes database in its own right independent of any particular client (i.e. there is no 'master' client or anything).