Home    |    GitHub Page    |    API    |    FAQ

Sharing notes or notebooks

Come to think of it, maybe all of that will be unnecessary, depending on how the rest of the sharing works.
If I could, for instance, add a notebook that is synced from a different sync target (I’ve seen requests for this, and similar things, around the forum), then that target’s master password could be shared between parties without exposing the rest of my (unshared) notes.

As far as I’m concenred, that would be enough and would require no changes to the encryption scheme.

It would be cool if you could make a “shared notebook” which will act differently than a notebook.

Notebook will be your personal notebook which is stored on your device or in the cloud of your choice’

Shared notebooks can be stored on your device and shared with syncthing or be stored in the cloud. You could have your personal notebook stored on your nextcloud server and then have a shared notebook stored on a different nextcloud server.

That is how I can see this being well implemented.

1 Like

Two thoughts. One is on backend-native sync. Don’t expect them to all do the same thing or to work well enough to support Joplin use case. I tried to sync my atom notes through dropbox and it kept duplicating files every time I had concurrent changes. No attempts to merge what-so-ever. May have improved since.

The other point is related to concurrent changes. To allow shared editing, the larger the number of people and the more off-line the editing happens, the harder it is to reconcile the different versions. It’s like doing merges in git without manual intervention or with a UI that’s accessible to non-tech people. From reading this thread (very long so I may have missed something, sorry) I am not sure people are completely aware of this challenge, since E2EE has dominated the discussion. One alternative would be shared reading, not editing, in which case this problem disappears. For editing, I would look into CRDTs and in particular http://swarmdb.net/about/ or https://github.com/yjs/ since they are in javascript. Particularly swarmdb docs talk about using any storage or transport as backend. Not used myself, not a JS developer.

I think whether there are 10 users editing the same note or just 2, conflict resolution will be the same.

Most likely conflicts will be handled like this:

  • first use diff-match-patch to merge the changes
  • if it can’t do it, build a “conflict note”, and ask the user to manually fix it.
    I think it’s not too complicated to implement and should work well in the general case.

I put my thoughts into a blog post, as they’re a bit long for a forum.

Realtime collaboration is hard :open_mouth: I’d definitely not rely on a shared backend to do conflict resolution for you.

1 Like

Thanks for sharing @naxxfish, your spec might be useful if someone decides to look into it.

Another approach is to leverage the existing sync functionality and conflict resolution of Joplin along with the file sharing of Nextcloud.

This can work if both users are on Nextcloud. You can share one of your file on your Nextcloud sync folder, and the person you share with would place this file in their own sync folder. Any change from any Joplin client to this file would be applied to the client of the other user, and any conflict would be handled by the current sync algorithm.

I’ve tested this and it works fine. The only thing, as you’ve correctly identified, is that you still need a central location (in this case, a Nextcloud app) that would generate the share, and place the file in the other person’s sync folder.

It’s just an idea at this point but maybe one way to get the feature working while keeping things relatively simple.

1 Like

I currently have this working via Syncthing.

  1. /sdcard/Notes is the fs sync target on my Joplin installs
  2. All devices have Syncthing installed and syncing that folder

It works quite well. Syncthing does version history and simple conflict resolution. Though I’ve never ran into conflicts with Joplin even when trying to force it. This is fine for me, I’m syncing shopping lists and stuff. In the worst case I can dive into Syncthing’s version history.

This would perfectly satisfy my use case (i.e., simple syncing between me and my partner) if the syncing was scoped to the notebook level. Currently Joplin syncs all notebooks. :disappointed_relieved:

I haven’t tried it with encryption, as that’s outside my use case. All my devices are already full disk encrypted, and if the data isn’t going to the cloud, I don’t need another layer of encryption at rest (Syncthing e2e encrypts on the wire of course).

This works today for those who want to sync all of Joplin’s contents. But then you could just use the built-in options :confused:

Any chance of per notebook syncing in the future?

1 Like

Probably not, but sharing a notebook is something I’d like to support.

1 Like

I think it is possible to have shared notebooks. Here is how it can work

First off, the file structure will have to be notebook->note. For example,

PersonalNotebook
—Note1
—Note2
—Note3
SharedNotebook
—Note1
—Note2

And the settings will have to be divided for each notebook, not for all notebooks. This way you can choose a specific notebook to sync to dropbox and different notebook to sync to nextcloud. Same for encryption. One notebook uses one password to encrypt/decrypt the notes, the other notebook will use a different password.

To share a notebook you will have to do that with the cloud storage provider by sharing the folder with others users.

When you drag a note to go from one notebook to another it will have to be recreated instead of moved due to encryption since each notebook will have a different password.

This is how I will see Joplin implementing sharing notes in a user friendly way.

4 Likes

Short of a few caveats that I think would meet my full-needs with Joplin, I have been impressed enough to permanently move to it as the main note-taking app. I also have been moving a couple of my tech staff to this solution as well and we’ve got a sync process worked out with some minor problems and a few workarounds.

I stood up NextCloud and have our organization’s external shares set up for our network (smb network shares to home/department/collaboration folders). One of the network shares is our department files. Creating a note folder there will allow the entire department to access and leverage the NextCloud/webdav sync capabilities.

When we have staff meetings, I now have one member of the team taking minutes versus multiple people interacting on the same note. This prevents the overwriting of data since person A’s edits may not have sync’d before person B’s edits started to sync, thus overwriting person A’s initial notes. This works as a viable alternative if there’s only a few people interacting with the notes. You start to have issues if a lot more people are connected to the same shares and interacting with any shared notebooks/notes because there is not a queue system in place between Joplin and NC to coordinate the order of synchronization.

As far as the structure of the notebooks/notes, we did something similar to what @trymeout suggested for a while now. We created a notebook for each position, e.g. IT Director, IT Administrator, IT Technician. Each role has their own section they edit/sync but all of us have capabilities to search in all the notebooks for things we need. We have a shared notebook consisting of checklists, projects, and a running-knowledgebase area where how-tos and documentation are revised and formatted before it is finalized and permanently stored in the Documize system. The neat part is that the knowledge base is ready to go since Documize supports markdown sections as well. Using the custom-stylesheet for joplin-wide and taking portions of the stylesheets from other members here (thanks to @tessus), I’ve been able to format the preview window so now that exported-to-PDF print-outs look structurally sound. It’s also easy to copy/paste the customizations to other staff members so we all have the same look, feel, and operation.

The only things I really could use and look forward to, if it ever comes, are:

  • In-app updates versus downloading and manually launching
  • Multiple notebooks (or a better process with password-protection or encryption) to cater to my own personal-work notes - accessible only to me, say performance reviews of my staff and such
  • Multiple sync connections (or a switching between sync accounts option) - since I have my own NC server at home that I have my personal notes for school, shopping lists and kid’s checklists (that my wife and I use in Joplin), and various other documentations

I look forward to every update and the new features everyone has been contributing. Thanks to many of you for dedicating the time/energy to this project, especially @laurent and @tessus for your active responses and contributions. I only wish my brain was wired to code/program to further the product some more.

3 Likes

This will make Joplin the best self hosted note sharing app out there. Joplin + Nextcloud = best open source note app!

2 Likes

Sharing notes works fine on my setup. But, is there any way to manage shares? I only can see a list of ID’s, but I cannot to find a way to recover shared notes URL or remove shares.

Are those features supported?

Thanx

At this point the note sharing capability is more or less only a proof of concept implementation and is lacking essential features. You’ll have to wait until Laurent or somebody else puts more time into this.

Hello

I’ve created a prototype for an encrypted real time (per line) collaboration feature for joplin desktop, based on websockets and sjcl. I didn’t pitch the idea here before i started as my main purpouse was to learn the basics on encryption, websockets and js in general. However i thought i might as well document my progress and bring the idea up for discussion here.

I’ll attach a demo to this post that show some of the features i’ve implemented so far in action. Feel free to comment on pros, cons and whatnot regarding design choices, assumptions, thoughts and the potential of this collaboration model as a future joplin feature.


The goal for this prototype was to see if i could get the following to work

  • Connect two joplin instances to a websocket server
  • Track each clients cursor moves (per line) and display it to the other clients
  • Track changes in the note from each client (per line) and push changes to the other clients without exposing the contents to the server.

Prototype demo
I recorded a short demo of how the prototype works in current state. Both clients and the server run on the same laptop from ca 2015 (i5 3GHz, 8gb ram on arch linux)

rt-collab-demo.mkv (3.8 MB)

Dependencies in current state
Client side

  • ace editor (already in codebase, though i remember reading in some post it’s planned to be replaced?)
  • i guess we are limited to standard keybindings
  • sjcl (already in codebase)
  • ws

Server side

  • node.js
  • ws

Prototype design choices

Node.js based websocket server
As i mentioned, mostly because i was aiming to learn about websockets. However i do think this feature would be quite easy to set up for anyone with access to a server running node. As the current share feature is based on nextcloud, i guess this collab model could be an additional alternative for self hosters.

Transfering note changes between clients (lineUpdate).
I chose transfering whole lines as i figured the encryption part might cause performance issues if i were to encrypt and send updates letter by letter or word by word. Not neccesarily true, just an assumption on my part.

Per line updates also made for an easy way to build a server side model of the note in ciphertext, and keeping it synced between clients.

Client side symmetric encryption.
The encryption is based on a key that will somehow have to be shared between collaborators (in prototype it’s just a set variable). This is of course a drawback from a security standpoint.

One reason for not going asym on lineUpdates is performance, each client would have to encrypt each sent line with a separate pubkey from each of the other connected clients. With symmetric encryption and a shared key this only has to be done once per lineUpdate. independent of the number of connected clients.

Server, current state

  • Stores the notes as line number and ciphertext in an array tied to a note object (stores nothing on restart right now).
  • Pushes complete note to client on connect
  • Recieves per line updates and cursor movements from clients
  • Pushes updates to connected clients
  • Pushes new clients to connected clients and vice versa

Client, current state

  • Encrypts each line before sending it to server
  • Decrypts each line recieved from server
  • Uses a 256 bit shared key stored in variable
  • Does not encrypt line numbers or cursor movements
  • Is tied to a bunch of Ace methods to locally execute remote note changes. That part may or may not be a problem to solve depending on what new editor is implemented.

API

I’ve defined a simple API for client-server communication with messages sent over websockets in json-strings. Some of the key functions are listed below.

Client-side

message ‘type’ direction contains description
newCollaborator in uid Creates a marker for new collaborator
getNote in Complete stored note from server Pushed from server on client connect, right now overwrites anything local.
cursorMove out uid, line number Sends all vertical cursor movement to server (letting the server know which line each client is at)
cursorMove in uid, line number Updates a marker object that highlights the line each collaborator is on.
updateLine out uid, line number, line ciphertext Sent when leaving a line. The client encrypts the contents of the line using sjcl.encrypt and a 256bit shared key, then sends line number and encrypted data to server
updateLine in uid, line number, line ciphertext Simply decrypts the received line and inserts it at line number, overwriting anything present on that line.

TODO

There’s a lot to be done still for this to become a usable feature. Below is a list of what my prototype currently lacks, in combination with hypothetical discussions on key sharing and gui implementations. There are many different ways a feature like could be implemented. Feel free to comment.

Editor

  • Passing on the following to server and connected clients
    • Deletion of lines (i’ve had a hard time catching del and backspace onkeydown, does anyone know if there’s a specific reason to that? Basically all other keys are catched and i have it working outside of joplin)
    • Selections, cut, paste, deleting or replacing text by seleting and typing
    • Images and other attachments

Thoughts on key sharing

The shared secret key should of course not be exposed to the server in plaintext. I guess it could be sent encrypted over the server together with salt to connecting clients, requirig only a password on the recieving end. Maybe it could also be done by a file shared by e.g email, that opens in joplin and contains an encrypted version of the shared key. A drawback for both of above alternatives is that they leave the shared key open to dictionary attacks if somehow intercepted.
Another alternative might be to encrypt the shared key asymmetrically. Each client would then need a public/private keypair in addition to the shared key.

Server

  • User management
  • Authentication/Authorization
  • Note management.
  • Note storage. Sqlite? Mariadb?
  • Look over security

Client and thoughts on GUI-integration

  • Settings, add a “Shares” tab for managing server credentials and keys
  • Note, show names for online collaborators somewhere
  • Note, give user markers different colors
  • Notebooklist, possibly a new section for shares, first level being either user or server.
  • on right click note -> share, an alternative could be added to share to a server

Mobile

  • I have not looked into this, but i believe that for this to become a usable feature it must also exist on mobile.
5 Likes

Looks cool!

A few thought on this.

  1. Maybe you can avoid the need for a sever and just connect 2 Joplin instances directly. I think this fits well with overall Joplin architecture.
  2. Crazy idea but what if you could extend this and use for synchronization. Say instead of sending these updates in realtime you serialize them and sync via Dropbox or any other way. Then all other instances can read them later and apply. I think it can help avoiding conflictsp plus you can synchronize undo history this way.
  3. Another interesing thing to explore is operational transformations or CRDTs for collaborative editing.

Thanks for your comments.

  1. I agree it would be nice not having to set up a server at all, but with a p2p setup both clients would need to be online at the same time to sync, which to me seems to make it less useable.

  2. That sounds like a good feature, but may be more related to the current sync implementation or the nextcloud share feature than to what this prototype does. It would be nice to be able to use the regular sync to the same node server though. Not sure how much work that would be.

  3. Maybe, i havent looked into that as my aim was to learn about the basics. On a side note i also saw there is a project for making real time collab apps with Ace editor which might make stuff easier. I’m not sure if that supports any type of encryption though.

I think Joplin should have shared notes and allow users to choose their storage provider for sharing notes. However if joplin can afford this, they could host a nextcloud server with like free 50mb of storage and even have paid plans for easy cloud deployment.

Collaboration and sharing, together with already existing privacy could be very interesting for business. I also believe, that there would be demand for paid services (e.g. hosting private instance), but the question is whether it is not against the Joplin’s philosophy. In different words, the users that won’t pay or won’t host (which is pretty close to paying) won’t have the access to this feature.

What about sharing a key through URL with pseudo protocol? For example: joplin://sync-my-joplin.com/178489dd-bc0b-4ebb-8053-e252bd4afca0. The link would contain the key or any other information. Joplin desktop app would be registered to handle the protocol. I am not aware how difficult is it to implement on different platform thought. The second option would be to simple drop the URL somewhere in the app.

I’m not familiar with pseudo protocol, but that sounds worth looking into if we get to that point.

I’ve kept working on the client and server in spare time for the past weeks, and realized it’s a quite complex project. However i’ve also grown more confident that it could turn out a nice feature.

I might put up a repo with the current state of things once i’ve cleaned up the code. Still long way to go though.

1 Like