Embed Joplin store structure in existing folder structure (local or remote)

Summary: I suggest a special Joplin Store format that embeds the Joplin MD notes into an existing folder structure of a user or group of users collaborating. The existing folder structure is shown instead or in addition to tagging. Creating new notes stays as simple as with other store formats, but optionally notes can be moved into folders. Reusing the folder structure enables user to reuse their existing grown personal information management practice in Joplin. By using existing folders from shared work contexts (i.e. family share, friends share, company share), Joplin notes can become accompanying notes for collaboration and can also be shared and collaboratively edited.

This requires Adding multiple sources / syncronisation directories as prerequisite.

Design principles/requirements:

  • The store is configured by type (webdav, dropbox, etc) and root folder to use and credentials how to connect. In addition, subfolders can be excluded from the store by pattern (i.e. “.svn”, “photos”, “2010/backup”). There is a “default folder for new notes” which is configured, for example “root/notes”. Part of this configuration (for example the exclusion patterns and default folder for new notes) are valid for all users connecting to this store and may therefore be stored in a subfolder or file within the root folder with a sensible name, for example “.joplin-config”.
  • Mobile and desktop clients connecting to remote stores may synchronize the folder structure of the remote store to the local device, but only including the folder names and the Joplin note files.
  • Joplin MD Files must be stored directly in an existing folder structure beneath the root folder.
  • File names must reflect the Joplin Note titles. Changing the title must change the file name.
  • A sensible file name pattern must be found, such as “.joplin.md” or “<title …>.joplinmd”
  • Links to other notes must be stored as relative hyperlinks pointing to the related file/note.
  • Renaming a note or moving a note must trigger an automatic search for notes linking to this note and updating the link.
  • When creating a new Note, the Note is placed into the “default folder for new notes”.
  • Notes should be movable into other folders.
  • The user should be able to create new folders.
  • When creating new folders in offline-capable clients connecting to the remote store, the client may create the folder on the client copy of the folder structure. During sync, the newly created folder should be synced to the store.
  • The user should be able to delete folders. If a folder contains other non-joplin files, the user must be warned about this fact, alternatively: deleting such folders must not be possible. Deletion of folders must be synced back to the store.
  • The user must be able to delete notes. If a note is deleted in the local client, it is also deleted on the store.
  • The folder selection should be visible to the left side of joplin. The gui may hide folders that do not contain joplin notes. The gui should offer a quick filter to find folders. When filtering, the gui should show parent folders of found folders, i.e. filter the hierarchy leaving all parents visible to contextualize a found folder.
  • When editing notes, it should be possible to embed pictures. When a picture is embedded, it should be included in the sync, given there is enough space on the client device. Client may offer a threshold which notes to sync. Client may load pictures on demand, when note is rendered, retrieving them directly from store into a tmp folder, to render note including embedded pictures.
  • When editing notes, it should be possible to link to other files in the same file root (non-joplin files, see next point).
  • Joplin should crawl the names of all files (but not their contents) to an index file and sync this to mobile clients to support the previous two features (embedding pictures, linking to files).

Positive Side-Effects

  • Notes can be used as additional information in existing data organization systems. Example: I have a project, I add a Joplin-Note as summary. See also: the README.MD we place in GitHub projects.
  • Backup? not needed anymore, as users and organizations who value their file and folder structure already do back up
  • Export? not needed anymore so often, as the MD files are “right there in my folders”
  • Sharing? in combination with 3546, this feature would realize support for sharing notes.
  • Reuse of existing thinking and enabling users to see Joplin as a life-long companion (see underlying facts below)

Cons/Problems

  • Synchronization for this store needs a new approach. At least connecting to Webdav/onedrive/dropbox/ and selecting the root folder stays the same, so some code can be reused.
  • The architecture for this approach must be simple, minimalistic, stable, to not harm any existing folder structure. It would be good to “get this right on the first try”
  • Other apps can rename folders/notes. Joplin should detect such operations and have a means to re-index its fulltext search-index as necessary. Also Joplin should embed unique identifiers in notes (i.e. GUIDs) and may include these GUIDs also as additional data in outgoing links as a way to make implementation more stable against other apps moving/renaming/deleting files.
  • Other apps copy-pasting notes can cause duplicates. Joplin should detect such copy-pasting and identify the copy (new file) as a new file, for example based on file date and index status, and change the internal GUId in the new file.

Underlying facts for this feature:

  • Folders and files have been around since the beginning of modern computing and will also stay around for millenia to come. All major existing and future file-exchange platforms will provide interfaces to have “folders” and “files”, so this approach is guaranteed to be valid today and for the lifetime of all current users of Joplin on all operating systems and online file management platforms.
  • The folder structure of a user or organization is a grown knowledge-management system and well understood by the user.
  • Users are known to know their folders by heart, organize them intuitively in a way that supports quick retrieval. For example, users intuitively go for an average of 21 files per folder, which is shown to be roughly the number of files per folder that leads to multi-level folder levels that are both quick to retrieve with fewest clicks (averge 14.76 seconds) and can at the same time store thousands if not hundreds of thousands of files The Effect of Folder Structure on Personal File Navigation
  • User’s prefer clicking through their folder structure compared to using fulltext search in “56–68% of le retrieval events but searched for only 4–15% of events” (http://doi.acm.org/10.1145/1402256.1402259)
  • When tags came up in addition to folders, they never replaced folders. It was clear from the beginning, that both folder structures and tags are going to be used in parallel: Better to organize personal information by folders or by tags?: The devil is in the details
  • “Placing information in folders may give users a sense of control that tagging does not” Providing for Paper, Place and People in Personal Projects
  • Folders are a thing when it comes to sharing information in groups.
  • In addition to these references to others, I also want to add firsthand personal observations: I built a personal information management system called “Semantic Desktop”, and I was also part of the team developing NEPOMUK-KDE, which became a core KDE project and was shipped to millions of KDE desktops. That allowed me to talk with a lot of other programmers working on PIM tools and interviewing a lot of users. From this personal experience, I know that users loved reusing their folder structures. Building on existing OS features makes software future-proof. Also I noticed in user experience tests that a “folder tree to the left that can be filtered with a search box above and still shows the parent folders when showing search result folders” in the context of a PIM tool like Joplin is something that users LOVED using. This was the single most loved feature of the PIM tool “gnowsis” I built. Even 15 years after the user interviews I did on that feature, I still remember it, so I believe it is something important I should share here.

I used the words MUST, CAN, SHOULD in the spirit of RFC 2119

Related ideas by other users:

I may note that I joined Joplin as patreon to one day be able to share my own experience of developing similar open source tools in the hope that now - in a decade where I am not in a position to actively contribute to open source projects - others may benefit from the lessons I learned (the hard way) and I benefit by having a better note-taking tool I can use for the rest of my life. Today is the day. I found some time to write up some key observations in this feature request. I believe the linked observations and facts stand for themselves and am looking forward to a constructive, evidence/fact-based discussion.

To put my money where my mouth is, I hereby promise to donate 100€ if this feature is implemented fulfilling the described specification or in an even better way.

9 Likes

Hi @leobard I like the idea of combining Joplin features and functionality with notes stored natively in files in folders.

However I suspect a big reason why apps like Joplin store in their own database is for control.

The simplicity of working with files and folders is countered by the complexity of trying to manage all the possible scenarios.

Even just considering that those files can be created, edited, moved, deleted, etc by other tools, and Joplin would then need to track it all and respond appropriately. It would take a massive amount of work. Not so much a Joplin feature, but a whole new tool.

Anyway, thanks for capturing all this information and thoughts on the topic. Whenever I create a note there is always a choice to be made - does it belong in Joplin, or in a file in the project tree. It’d be great if there was no decision to make.

I think what you really need is a standard directory + md file storage notes + excellent synchronization/search function…

1 Like

This is Core feature for a realy good PIM, DAM, or DM software, even though you can easily store documents in databases...

The problem earlier for Windows was the limited filepath length, but now you can enable 32-bit length on 64-bit systems, so if the software utilize it, its no longer an issue..

Keywords should be added to the files metadata where possible, but can also be added to a database of choice for faster search and more feature regarding grouping and sorting (i.e. creating different hierarchies of nested keywords keywords should as best as possible follow international standards ...
And then there should be TAG's, free words that can cross both keywords and folder structures.

All metadata should be added to a database, regardless of type of database.

Of course something like mongodb will be a lot faster regarding search of structured documents, but it limit the usage of the same documents since any editor or reader need to support a database connection to the type of database used.
With a folder structure you can easily edit the documents in another editor without the concern if you edit the latest version or not...

Third benefit are that you can use the OS's own version controls and the file systems restore point if something bad should happen... as mentioned you will also easily be able to utilize back-up systems that you may (should) already have going.

@mic704b exactly. There is always a choice to make between Joplin VS File-System. Let's see if more people have this need and comment here / upvote.

1 Like

Regarding “implementation”…

We need to be sure what we want because the implementation of this feature within Joplin is a very big project, taking hundreds if not thousands of hours. So the GUI/Usability/Story must be “useful & correct” on the first take.

(attention: this comment was heavily edited by me!)

Challenge

  • The code of Joplin is optimized for the current database model.
  • Example: joplin/ReactNativeClient/lib/models/Note.js. The class is currently 821 lines and contains multiple types of methods:
    • Note-specific: serial/deserial, preview, alarm, geolocationUrl
    • Model-specific: mustHandleConflict, batchDelete, save, duplicate, moveToFolder, who call this.modelSelectAll with SQL assembled within the method
    • viewing/filtering/sorting-specific: customOrderByColumns, dueNotes, filter,
  • This works, is tested, deployed, easy to read, elegant approach and should be kept. The approach of using modelSelectAll() and SQL contains much application logic of Joplin and must not be changed by this suggested “file&folder model”.
  • Changing the underlying database model into the suggested “existing folder structure model” would mean that specifically the Note.save() and the corresponding loading logic have to be rewritten/overwritten for this type of folder-backed model.
  • Changes to moveToFolder() and copyToFolder() have to be reviewed.

That said: if we try to add this “folder&file backed model”, I propose one of two models:

HybridModel: only store the notes differently

  • keeping the current model and modelSelectAll approach and SQL for metadata handling / application logic
  • changing the way note.save() / loading work to solely use flat files in other folders instead of sqlite3 (database.js)
  • adding sync code that
    • checks for modifications in the folder&file-system on startup (comparing the state when Joplin was previously closed to the current folder&file data)
    • … and afterwards …
    • registers file-system listeners for modifications while Joplin is running, to keep the metadata in the model in sync with the state that is in the folder&filesystem.
  • Pro: would make code changes minimal, as existing code is changes.
  • Con/Problem: this coding approach would change the core of Joplin and make this model “mandatory” for all developers/users to “accept as a viable option” and maintain for the years to come.

SyncModel with filesystem

  • Keep Note.js and database as they are today (all data only lives in sqlite, AFAIK)
  • Write synching code that
    • stores Joplin database changes in the folder/file system, using a trigger or listener
    • watches the folder/file system and updates the Joplin database
  • this is similar to what @rxliuli proposed above, I think
  • Pro: existing Joplin Codebase probably untouched
  • Pro: adding listeners/plugins to do the sync
  • Problem/Con: change to gui for folder management?!
  • Pro: if this was a bad idea, we didn’t change the Joplin core
  • Con: is this was a great idea and a majority of users love it, we have a messy listener system and data duplication “forever”. (we amass technical debt)
    • Solution: after 10 years of having the mess, someone may untangle all bits and write a migration to a newer “hybrid” model. Someone would have to pay back the technical debt.

I guess in both models the changes necessary to the existing code would be in the thousands of lines and the additions too, thousands of lines. This could take a year to do. All steps probably need checking/feedback from @laurent or whoever does overlook the core.

Which brings me back to the opening of this comment: we must be 100% sure what we want before we start working on this.

That said:

  • Paper mockups, Screen mockups!
  • User experience tests with at least 5-10 “typical” Joplin users are a must before starting designing the code

I think we should only start with those paper/screen mockups after Adding multiple sources / syncronisation directories is done.
*

1 Like

My background: Used Evernote, got tired of them pushing more and more for pro account, converted to Synology Notestation (have had a NAS for many years), it works decently but not well enough for rest of family (in my opinion). Now I converted to Joplin to try it out. Been testing for a couple weeks now.

I run Nextcloud on my NAS, so I initially assumed that would be a good way to sync. However to me it looks like in particular on mobile it is very slow.

Further, I was assuming that each notebook would be a separate folder, so that one could have a subfolder on Nextcloud actually being shared with another user, and thereby using Nextcloud to create a shared notebook. That is not the case. I don't know if this would work even if possible though, or if it would just generate too many conflicts if we both edit at the same time.

I also imagine that if each notebook was a separate folder, it may be somewhat faster for syncing on Nextcloud? Folder structures aren't that often intended for thousands of files on one level (or so I imagine).

What I now was thinking of, which is sort of in line with this proposal (again, I think), would be to run a Joplin server instead (which supposedly works better on mobile?), and have a "joplin backend client" running as an app on the Nextcloud server which presents all current notes as separate MD files, structured by folders (so not showing history then). These could be edited as a normal MD file (perhaps working with the Nextcloud Notes as well?) which would then go back to the server if I edited, or someone else edited in a shared folder upon synchronization. I absolutely do not have the skills or time to implement this unfortunately, so I throw it out as an idea. Or pipe dream is perhaps more accurate, I guess there is possibly some major blocker with my idea.

By the way, the github issue linked is auto closed by the stale bot.

Hi @yngveL . We are on the same path. My intention is described above in length. My current status:

  • Synology Disk Station as home NAS and Linux Server
  • I haven't looked at NoteStation yet, should do
  • I also installed NextCloud with the thinking "if someone or me makes a note-taking app embedded in the file system, NextCloud is probably the best architecture to start from"
    • I found that NextCloud now has the "text" app which makes editing Markdown files easy.
    • GitHub - nextcloud/text: 📑 Collaborative document editing using Markdown
    • I found NextCloud Text a bit weird to use (adding links is not convenient, some other issues) and am reporting them
    • So NextCloud Text is "close" to a note-taking tool embedded in the filesystem. Also, it is geared for multi-device architecture, sync, backups/versioning, fulltext search - the whole package of a modern note-taking app
  • I use Obsidian as desktop-based note-taking tool
    • https://obsidian.md/
    • Obsidian uses markdown as file format and stores the files in folders
    • Obsidian can "open" an existing file system as "vault", indexes everything found, and lets you add new files
    • No mobile version
    • Does not support links to files
    • Many other quirks and bugs
    • Very close to my vision of a note-taking too embedded in the local file system

So, no solution yet. Also, I didn't find any other tool that does what I want.

I also noticed that markdown is not a standard, but a mess going in all directions. It would be good to start a W3C markdown standardization group to let all markdown-based-note-editors agree on a uniform way to create hyperlinks and other important thingies.

1 Like

Hello. I think it would be nice for Joplin to at least keep the folder structure (note in the folder with the notepad name) when syncing with the file system. To preserve the structure, for example, as when exporting to .md
Now Joplin simply ignores nested directories and notes in them, and puts everything in one root directory without splitting into notebooks. What is inconvenient for working with the same notes outside of Joplin

Which you should not do anyway, or you might lose your changes.

Quick FYI, specifically aimed at "working with the same notes outside of Joplin": If your goal is to edit/view specific notes outside of Joplin, you might want to try [right-click on note in note list] > Edit in external editor. The desired editor can be set in the Joplin options.

This allows to easily view / edit one or multiple notes "externally", right from the Joplin UI, with all changes synced back to Joplin whenever the file is saved in the external editor.

I probably didn’t put it that way.
I know about setting and editing in an external editor, thanks. But when synchronizing notes in the cloud, it is almost impossible to view them with something other than Joplin, since all notes are mixed in one directory, without division into notebooks and with an incomprehensible file name instead of the note name.
I agree to edit notes only in Joplin, but storing (syncing) notes while maintaining the native structure of files (notes) and directories (notebooks) is really more convenient than now.

1 Like

This might help Web deployment - How I do it

Thank. But, your link describes a method with manual export to md. So now I'm doing it. But it is much more convenient for the process of exporting and importing from md to take place automatically, as during synchronization.
p.s. Thanks for Mkdocs, useful for other purposes

I am looking for a similar thing. I have been using joplin for some time now. All notes being saved in the same folder, and all file names being stored as hashes is making me go back to typing in to md files using Npp and saving the files in folders manually note saving.
If you find something, please let me know. I'll post here if I find something.

2024 need it!