Notebook specific Resource location overwrite

Features
1

Hi there.

I have an interesting question that in one form or another has already been asked here, but as far as I know there is no definitive resolution that is accessible as a standard Joplin feature, that does not requires the user to rely on the OS-specific hacks or some Joplin-specific workaround, but feel free to correct me if I'm wrong.

In the interest of full disclosure, I am a Windows-OneNote user who is considering switching to Joplin and at least partially to Linux. I have not migrated yet, I'm only trying things out to see whether Joplin fits my needs and workflow.

My issue starts with something that many people are familiar with: I have a partition/SSD that I use for OS and applications, and then I have multiple other partitions, external disks and network drives where I store the data, so that at any point in time I could just drop the entire OS, format the disk and start with a fresh OS install, but with all of the data still being preserved. This particular concern can be somewhat addressed with backups or creating a Joplin profile that is not on a system partition, so it is not the most critical aspect that I want to focus on.

The other issue, and the primary reason I am asking this question, is the fact that I have few TBs worth of notes/data that simply cannot fit on a single SDD, and I have to split it apart. In my case, I have notebooks with lots of various media attachments (photos, videos, 3D models, diagrams, sketches, concept arts), so much so that a single notebook could be 10-50 GB in size, and 99% of that data are various attachments. And since I have few of such notebooks, my total notebook consumed space is about ~1.5 TB.

Since I am currently using OneNote, which is a file system based note-taking application, it is not that much of an issue for me, since I can store different notebooks and their attachments in different places.

  • I have some notebooks that are stored on the secondary drive of my PC.
  • I have some notes stored on an external drive that I carry with myself when I need to travel on work related trips.
  • I have some notes that are stored on my home network drive (yes, I know it makes things slower, but the performance so far was more than satisfactory for my use-case).

This workflow, of course, runs into a problem when considering the fact that Joplin stores everything, both database with notes/metadata and resources/attachments in the same profile folder, which by default is located on a system/OS partition.

I have done a bit of a research, and there were a few questions/feature requests related to this already, like the few links below:

The problem that I have and want to address though is not particularly answered in any of these threads.

The suggestion of using a backup works, well, for disaster recovery and OS re-install cases, but it does not address the fact that I cannot fit all of my data on a single partition, nor the fact that I do want to keep most of the data in different places for performance/access/convenience/organization reasons.

The suggestion of using profiles with different locations introduces an inconvenience of having to switch between profiles, something that I would like to avoid, since I often have to lookup things stored in one notebook while I am working on something in another notebook. Using different profiles just to workaround the space limitations of a single partition seems like a very iffy hack, which I imagine would quickly turn into an annoyance, if only because I will not be able to run search queries across multiple notebooks stored in different profiles at the same time.

The suggestion of using soft links/junctions is too OS-specific, and since I'm slowly trying to partially (but not fully) switch from Windows to Linux, I would like to avoid using something that depends on OS features, and would prefer to have something that is a Joplin application feature that works both on Windows and Linux without having to keep playing with the file system or OS flags and settings.

So here is my question to the Joplin community. I am a developer, and one of the reasons as to why I am looking into Joplin as a potential replacement for OneNote, is the fact that it is an open source project and I get the ability to add the features that are missing in Joplin, as well as features that I never even had in OneNote in the first place, but I always wanted to. So I was wondering whether this is something that I can start with, and perhaps even contribute back to the community as well.

My thought was that while a fully-fledged solution for my particular problem might be quite a sizeable amount of development work, I could start with a simple solution first, that addresses some of the use-cases, and then slowly, over time, improve it so as to have enough flexibility for people to be able to tweak it to their own needs.

So my initial idea was to implement a feature (plugin or what not), that allows the user to specify a secondary location for resource files on a per-notebook level.

The idea is that by default, if nothing is specified, then the application will use the current default behaviour and look for resources/attachment in the profile folder.

If a notebook has a resource location specified, then the application will first look for a resource in that location first (in essence acting like an overwrite for a profile path), and if the resource is not found in that location, then it will look for it in the default profile folder.

If the notebook is a child of another notebook itself, then a recursive lookup could be done:

  1. First, look for the resource in the folder of specified by the child notebook
  2. Then look in the folders of the respective parent notebooks, using bottom-up approach through the parent-child hierarchy
  3. Then look in the default profile resource folder

This way the feature will not change anything in the current behaviour of the application for those who are not interested in this specific feature, but those who want to use it can opt-in into it.

I have not contemplated yet how this would work with sync across multiple devices, since I haven't investigate yet how sync works in Joplin (hence one of the reasons for asking this question so that more knowledgeable members of the community could chip-in with their thoughts). But the most immediate options that come to my mind are:

  1. Only notebook Resource Path is synched, so that an external resource location, such as a network drive, would be a singular storage for resources, and then all of the devices can access it, provided they have access to the same network.
  2. Allow the user to specify a different Resource Path for each notebook per device, so that you could have a notebook that, for example, uses an external drive for storing resource on one device, while the same notebook could store its resources locally when on a mobile device.
  3. Provide support for all of the above, and let each user decide which one of these two options he would prefer.

To a certain degree, this feature could be extended with an abstraction level, so that the external resource location does not necessarily has to be a traditional file system, and could also be something like an AWS S3 bucket or similar cloud storage, provided the necessary API exists to support what Joplin needs for its operations. And since we are on the subject, it might be possible to implement support of multiple external resource paths per notebook, and let the user mix his local drive, network drive and an S3 bucket for reads, and one of them for writes.

There are certain limitations to this implementation and the problem I am trying to solve with it.

It only addresses the location/size of resources and not the notebooks/database, which I think is fine for the initial implementation, and might be more than enough for some people to find it useful, in particular for cases when the attachments are the primary reason for the bloat in the size of the notebook. Based on the experience of implementing this version of a feature, and the feedback that might result from it, a more refined version could be implemented.

The loading of resources could be slower when they are not stored locally and are located somewhere remotely, be it a network drive or some cloud-based storage/bucket. But for some users the performance hit might be very well within their limits of what is tolerable and usable.

I do intend it to be an opt-in feature, meaning that if you do not need it, then you do not have to use it nor pay the size/performance/complexity price for it, everything will work as it did up until this point without you having to do anything, and you will not be affected by its implementation.

On the API level, there would be a need to implement an abstraction layer for resource operations, that would allow the entire Joplin codebase to be indifferent to this change as much as possible.

So what would the Joplin community thinks about such a feature? Any comments, suggestions, issues that you see?

2

This is the official method to keep your Joplin data outside of your system drive. You should do it for the whole Joplin data directory, not just for resources though. Symbolic links are supported on all of Windows, Mac and Linux, so it's not OS specific.

Someone also mentioned there were issues with the Backup plugin not actually backing up to a different drive when changing the configurable directory, but if that's the case, that can also be solved using a soft symbolic link for the default backup directory as well.

Regarding keeping data in different places, that's not supported by Joplin and the plugin API won't allow you to do that.

3

I understand that different OS'es have support for some version of soft links and junctions. But I do not think that a typical user should be bothered to know how to setup a soft link/junction on his device, considering that this is often a terminal/console thing that your average user does not want to use, and also considering the fact that if you have both Windows and Linux/Mac/Android devices (something that quite a lot of people have these days) you would have to figure out how to do it for each OS specifically. As a developer, I can do it myself without much of a fuss, but the average person might have issues. And that is before we take into account that tweaking OS/file system by the user to make your application work is something that in my opinion is not a good user experience. This should be an application feature, and not console/terminal feature. Even if the soft links/junctions is what end up driving the under-the-hood implementation for it, it should still be an application setting. Having to look up how to create a soft link every time you want to access this feature is not as user-friendly as having a button in the application that allows you to set this up.

4

That's a fair criticism. Custom profile locations is a feature which has been requested a few times. Not my decision, but I suspect the feature has not been approved deliberately to discourage average users from doing it, because there's more risk of database corruption if you put it on network drives or external storage.

In terms of segmentation, I don't see a feature like that being approved, as it's quite a niche requirement and could add a lot of unnecessary complexity to the codebase. The sync is already very complicated, and when you have a very large data set like yours, you certainly don't want to risk data loss if you decided to have a go at implementing this yourself. You're welcome to do as you please, but I'm also a developer and very familiar with the Joplin codebase, and would urge you to be cautious because of the complexities of the sync (and how long the sync would take for a data set that large, if you have to start over).

I get that system drives can be limited in space, but buying a larger external drive doesn't have restrictions other than the up front cost. It's just application data you're storing, so an external harddrive is fine - no need to get an ssd

5

Also letting you know up front that if you enable end to end encryption, you'll need about twice the amount of storage to download the full data set on another device

6

That is a valid concern, and hence why I'm not suggesting doing for the database, and only for the resources.

I get that system drives can be limited in space, but buying a larger external drive doesn't have restrictions other than the up front cost. It's just application data you're storing, so an external harddrive is fine - no need to get an ssd

I knew that people will get too focused on the OS re-install part, which is not my primary concern. My primary need is to have the ability to store attachments on multiple different drives, depending on the notebook.

Yes, I can always buy a bigger NAS, and I can store everything on it. But when I need to travel to a customer, I either have to drag an entire NAS with me, or have to rely on a sync functionality to get the attachments on my laptop/smartphone, which is much slower, and it would be quite annoying when you are trying to show customer some large files.

Don't get me wrong, I have absolutely no desire to re-implemet the entire sync. The current sync implementation probably works just fine for notes/metadata, since this type of data is text-based and is usually small, so you can rely on sync to move the notes/metadata between devices. But it is all of the other attachments that are quite large in size that causing some problems for me, and I would rather have the option to drop them on a small external drive and bring them with me, while allowing the sync feature to move all of the text-based notes/metadata to my laptop/smartphone.