Created a new version of Joplin that keeps local md files

[LeoK987]

I've created a modified version of Joplin based on 2.13.2. Among the main features added is having all notebooks and notes automatically saved as folders and markdown files onto the local file system, and the files and folders are kept up-to-date with the database of Joplin.

So far, this feature works only with the desktop application. The default directory where the notebooks and notes are saved to are under "JoplinFiles" under "Documents" of the user's home directory, in Linux notion: "/home/yourname/Documents/JoplinFiles/" (it can not be customized now). Under there, there are sub-directories based on Joplin's profile ID. If the user hasn't added additional profiles when using Joplin, all the notebooks and notes are saved under the "default" directory. Notebooks are organized as directory trees and note files are under the associated notebook folders. (Note, if you are running a dev version from source, the directory would be "/home/yourname/Documents/JoplinDevFiles/").

Upon the start of Joplin when the said directories don't exist, Joplin will create them and populate them with sub-directories and files. Directories are named with the titles of the notebooks. Files are named with the titles of the notes, as: "the note title.md". If the note is a to-do note, it is named as "X | note title.md" (if the to-do is not completed), or "V | note title.md" (if the to-do has been completed).

You can create, update, move, delete notes or notebooks in Joplin and the files and directories are promptly updated.

One efficiency improvement is on the frequency of saving notes during editing. Originally, Joplin saves the note to DB on every key stroke, which appears quite inefficient. Now, note is saved upon receiving the addition or extraction of a return character, or upon change of focus (like pointer changed to title from body, or vice versa), or starting a new note.

Rich text editor is currently not supported.

The Linux application appears to work well. And an AppImage file is included in the release. I don't have versions for Mac or Windows and welcome anyone willing to build and test them.

Currently, modifications of directories or files outside of Joplin are not updated back into Joplin, but this is being worked on.

This feature is also being ported to mobile apps (specifically the Android app).

I appreciate comments, critics and suggestions.

[graphit0]

Hello, since I'm interested in the discussion but don't have anything of value to add let me simply link here similar initiatives made in the past

Joplin Filesystem (JoplinFS)

What is it? JoplinFS is a (proof-of-concept) wrapper around the Joplin API that allows you to read your Joplin notes as if they were normal markdown files that live on your hard drive. To do this I use the FUSE interface to pass filesystem operations to the Joplin API. Using this enables you to view and interact with your notes as if they they lived on your hard drive instead of a database. If you're familiar with rxliuli's vscode plugin then this concept will be familiar to you. Still doesn't …

Embed Joplin store structure in existing folder structure (local or remote)

Summary: I suggest a special Joplin Store format that embeds the Joplin MD notes into an existing folder structure of a user or group of users collaborating. The existing folder structure is shown instead or in addition to tagging. Creating new notes stays as simple as with other store formats, but optionally notes can be moved into folders. Reusing the folder structure enables user to reuse their existing grown personal information management practice in Joplin. By using existing folders from s…

[dpoulton]

Linux Mint 21.2 Cinnamon
Forked Joplin 2.13.2

I have only had a quick look at this but would it be right to say that for this version Joplin it works exactly the same as Joplin, using the database, but just keeps an additional copy of the notes as plain text "md" files in ~/Documents/JoplinFiles/default?

On my brief test a few problems I encountered were:

• In the desktop app itself when entering a new note title it is truncated in the note list to a single first character until body text is added. However on some occasions the full title never appears.
• In the desktop app itself when creating a new note all of the body text of the note is truncated to a single first character of the note when moving away from the note and back to it again (see gif).
• The plain text folder cannot handle two notes with the same title in that "test note" appears as "test note.md" but if another "test note" is created it overwrites the first rather than use something like "test note(1).md"

I am not sure what benefit this adds at the moment, but it is early stages and if changes made in the plain text folder could actually be fed back into Joplin that could have some interesting uses.

@LeoK987

Currently, modifications of directories or files outside of Joplin are not updated back into Joplin, but this is being worked on.

[LeoK987]

@dpoulton

• In the desktop app itself when entering a new note title it is truncated in the note list to a single first character until body text is added. However on some occasions the full title never appears.
• In the desktop app itself when creating a new note all of the body text of the note is truncated to a single first character of the note when moving away from the note and back to it again (see gif).
• The plain text folder cannot handle two notes with the same title in that "test note" appears as "test note.md" but if another "test note" is created it overwrites the first rather than use something like "test note(1).md"

On the first two issues, it takes a return character in the body to save the note. (I just uploaded a new version that also takes ".", or a block cut/paste, to save the note). Then nothing is "truncated".

If during editing, you switch to a new note (create a new note or new to-do), then the current edited note gets saved.

I'm aware that if you switch to an existing note, the current edited note does not get properly saved. I haven't figured out the hook to do it yet.

The newly uploaded version also takes care of duplicate title issue. So if notes in the same notebook (folder) have the same title, the newer note will take a title like "test note(1).md", "test note(2).md", etc.

Additionally, I'm working on getting externally added/edited files synced back to Joplin. I need some reasoning about working conditions.

[LeoK987]

@dpoulton

I have only had a quick look at this but would it be right to say that for this version Joplin it works exactly the same as Joplin, using the database, but just keeps an additional copy of the notes as plain text "md" files in ~/Documents/JoplinFiles/default?

Generally, the new version works the same way as the original Joplin, particularly, it still uses the same DB. There are internal changes at the core to handle files and folders (and the less frequent manner to save notes to DB, etc). I am trying to have the changes not creating unintended changes to Joplin. So far, it works as described.

[LeoK987]

Uploaded a new revision.

Now, the edited note is saved every 60 seconds, or upon leaving the note editor.

And for naming to-do notes, it is named as "X - note title.md" (if the to-do is not completed), or "V - note title.md" (if the to-do has been completed). Changed from "|" to "-", as Android doesn't accept "|" in file names.

So I got the Android app working in the same ways. Even better, the Android app on Android 10 and newer allow customization of the file directory. Will upload an apk file soon.

Happy note taking!

[LeoK987]

First Android apk was uploaded for your testing. So as not to affect your normal use of Joplin, this app is named JoplinX.

Also added routines to remove from title any special character of ?:"*|/\<>

[graphit0]

Hey thanks for your work

So I tried android version and it does indeed keeps an extra copy of Joplin notes in the folder of my choosing

However, I couldn't make it to create the attached files in the folder

[LeoK987]

Thanks for the note.

Link or attachment can work easily on desktop version as the original resource directory is available for access. On Android, the resources are kept in Joplin's private space, accessing them from an outside MD file viewer would need additional transport.

I haven't looked into it yet, but will soon.

[PEP8]

So is this a possible use case? -

I have approx 14k notes in Joplin Linux.
It is too large for dropbox (free) etc. so i cant sync to Android in that way

My workaround before this post was -
I sync using file system.
Ie on Linux i export to my local disk
Then rsync the joplin file (incl *.md files, .resources folder etc) to my NAS.
Files on the NAS sync every day to my Androud sd card.
And Joplin in turn updates the Android db by importing the Joplin files on the sd card.
This approach fails because Joplin is horribly slow (maybe 3 notes per minute) at ingesting the detail into the Android Joplin db.

My new approach might be -
If i can have Android Joplin read the files but skip db update step, then my slow sync problem becomes a non issue.
Ie i can... install the new Joplin app, point it at the files on the sd card and can then access the sd card file via new Joplin, without the db update step.

Would the new approach solve my import problem?

[LeoK987]

The md files that are saved with this version are the whole individual md files with the file names associated to the titles. There are no step or history files as you get with Joplin's sync to file system. So, without considering resource files, you have about the same number of files as the number of notes (whereas in Joplin's sync to FS, you end up having many times more files).

As it stands, the md files are not sync'ed back to Joplin yet, but after more testing, I will enable the sync back to Joplin feature.

I'm not sure about your "3 notes per minute" situation, but I suspect it has to do with the step and history files. In my testing, getting the individual md files into Joplin is pretty fast.

[LeoK987]

A new revision now handles resources in markdown files. A Linux executable was uploaded for testing. Android app will be uploaded later.

The revision now has code base of Joplin 2.13.4.

Resources (images or attachments in notes) are now located in the ".resources" sub-directory in the directory of every notebook. Markdown file shown in external viewer now shows the related resources. Resources are saved on every note edit, or when the "JoplinFiles" directory first gets populated. If you already have the "JoplinFiles" directory and want all of your markdown files to have resources, you need to remove the "JoplinFiles" directory and restart Joplin. (Note, this feature now works only in the desktop version).

[LeoK987]

Another revision further on resources handling, and note editing efficiency.

Resources now follow the associated note, i.e., when you move/remove note, the related resource files will be similarly handled.

Resource files are currently copied from Joplin's config directory, so you have duplicate files on your system. Going forward, it appears more reasonable to have the resources close to the note files, so I'm looking into the possibilities of removing the resources directory under Joplin's config directory. But this will be at a later stage.

Further improved editor efficiency by minimizing resource lookups during editing.

And, Linux AppImage file is uploaded.

[LeoK987]

Android app capable of handling resources is uploaded.

[LeoK987]

Any comments on shrinking the size of resources?

Currently, I noticed if you take a photo, Joplin asks if you want to shrink it to 1920 pixels (what exactly it means I'm not too sure), so even if it's full HD dimension, it's quite big. My thinking is this is a note taking app (not intended as a photo album I guess), it's probably reasonable to keep very small images. So a photo can be highly compressed (not necessarily losing too much quality) down to a size below 100KB and stored with Joplin's file folders.

[dpoulton]

@LeoK987

My thinking is this is a note taking app (not intended as a photo album I guess), it's probably reasonable to keep very small images. So a photo can be highly compressed (not necessarily losing too much quality) down to a size below 100KB and stored with Joplin's file folders.

I'm not sure that that is a safe assumption to make. There has been a lot of conversation on this forum about Joplin prompting to resize images larger than 1920px down to 1920px. In fact the end result was the introduction of the Note > Resize large images option in the settings so that Joplin did not even prompt to resize large images. Seems that those attaching larger images don't even want to be asked to resize them never mind actually resize them.

Of course your version of the app may attract a different type of user compared Joplin itself.

Speaking of which, what with the above and your idea in another thread to delete attachments from the Resources folder when a note is deleted, it seems like your version is starting to drift away from how Joplin behaves. Would it not be beneficial to fork it into its own project?

[graphit0]

@LeoK987

so I'm looking into the possibilities of removing the resources directory under Joplin's config directory.

Could it be possible to use symbolic links to Joplin resources (config resources directory) instead of copying them?

That would certainly save some space and allow Joplin to function just as before

[dpoulton]

It's great that someone has seen something in Joplin that they like and want to develop it to suit their needs or just introduce their ideas. Purely as a Joplin user myself, my concern at the moment is that Joplin's forum is being used to ask for suggestions as to how an offshoot of Joplin should be developed. It's not a plugin that adds functionality to Joplin but what is becoming a fundamentally different version of Joplin, also called Joplin.

At the moment it is mainly contained in a thread that, if fully read through, shows it refers to an alternative version of Joplin. However as features are implemented in this alternative and mentioned on this forum it could be that readers believe that the changes are being made or have been made to Joplin itself. It could also cause problems if people search the forum for answers and have no idea that what they are reading does not relate to how Joplin itself works.

Not trying to be pessimistic but my "spidey sense" has just kicked in and it says having two functionally different versions of Joplin, both called Joplin, being discussed on the same forum could become increasingly confusing as time goes by.

Sorry for the "downer".

[graphit0]

I understand. If the author would want to redistribute new version there has to be new name and branding and much more

However, what I see is person experimenting with the codebase to get new functionality for his own needs. I've seen that he tried to open up PR with these changes, so intent to share his work back is there, I think.

Additionally the changes made are shared with the community and serve as a proof of concept. It might be not merged into the main but at least it shows what can be done. Based on it proper implementation can be made.

I'm not sure what to call it at this point -- a fork, a branch or totally new project. For now it's just a matter of proper use of language. I'm not native speaker myself so I can understand confusion over difference between "new version" and git branch. I guess we'll cross that bridge when we get to it.

Maybe there's better category for this topic -- such as Development or Apps? Or even Lounge?

[LeoK987]

@dpoulton Thank you for the polite notice. My hope is that someday, my revisions will get merged into Joplin, so I've not re-branded my revision but continued sharing my ideas and development progress here.

@graphit0 Regarding creating links to resources, since Android and iOS don't allow links across "file boundaries", so to make resources work for all platforms, I copied them now.

Regarding my thoughts in favor of storing resources close to the files and possibly removing the resources directory in Joplin's config dir, I don't believe it could affect how Joplin functions and effectively make my revision incompatible with Joplin. Here is why. (Please note by no means there is any hint on whether Joplin's arrangement is worse than my thoughts or whatsoever, it's just for open discussion.)

Let's call the part in Joplin that handles data storage "the data engine", so in the original Joplin, it manages database operations as well as managing resources files etc, and in my modified version, it also manages the md files and associated resources files on the file system.

So, first, this data engine doesn't care where the resources are stored, whether it's in the config directory or in the file system with the md files. As long as it's given the address and logic, it will correctly manage the resources. With that, Joplin continues to function normally regardless where the resources are. Agreed?

Then, at the choice of whether to store the resources in the config or with the md files (so we don't duplicate storage), there is no reason that the resources have to be in the config, but it makes more sense for them to be with the files. After all each resource data is not something on its own, it belongs to a note.