Which library to be used for Real-Time Collaboration on a Note?

GSoC Projects Real-Time Collaboration
1

Real-Time Collaboration in Joplin

The base technologies used for real-time collaboration:

  • WebSockets

    In websockets server acts as a mediator for the communication between the clients. where changes by a client is sent to the server and the server emits/updates other clients in the session/room about the changes.

    This is very much server-dependent.

  • WebRTC

    In WebRTC only job of the server is to connect a new client to other clients in a session/room. While clients communicate with each other directly through a P2P connection.

    This has very less load on the server.

We could directly build on these base technologies, but that requires a lot of research, development, testing(Just to implement the communication/conflict resolution) and this is also very much error-prone.

Building on these base technologies (which are available as regular javascript APIs in all major browsers), there are libraries that provide easy integration, development, and additional features for implementing several kinds of real-time collaborative applications.

So, What are the libraries which we can use to create a Real-Time Collaborative Editor for a Joplin note?

Below are the options and pros/cons of the libraries that we can use:

Socket.IO

Socket.IO was the first technology that I looked into for developing a real-time application. This library is built on top of WebSockets. This is more suitable for chat applications / similar applications with new data coming in rather than the same data edited by multiple users.

Pros:

  1. Easy to implement
  2. Reliable and very active community.
  3. Full duplex communication over an HTTP connection(meaning reliable transmission of data).

Cons:

  1. We have to implement all the conflict resolution techniques.
  2. Needs a server in between. server-dependent communication between clients.

GUN

Gun is also implemented using WebSockets and provides conflict resolution using CRDTs. It is a very developer-friendly library (looking into the efforts they put into documenting and facilitating developers who are new to GUN). It needs a signaling server for P2P communication. the organization is currently working on establishing a direct connection between peers without the need of a signaling server through this project: AXE

Pros:

  1. P2P communication.(less load on the server)
  2. Provides user authentication (need to explore more to better understand how it works)
  3. Very active community and extremely visual tutorials for developers new to the technology.

Cons:

  1. Need to implement the data updations from and to the editor we use.(i.e. no support for editor libraries provided by the community).
  2. Signaling server is needed.

ShareDB (ShareJS)

ShareDB is also implemented using WebSockets and provides conflict resolution using OT. it is an open-source real-time backend. It basically maintains a copy of the data on the server-side and lets clients push changes to it. these changes are then synchronized using OT and all other clients are updated of it.

Pros:

  1. Offline support(meaning when the client is offline the changes made are stored at the client end and updated to the server on connection).
  2. Extensible via middleware with community-provided modules for authorization, etc.

Cons:

  1. Needs a server between clients for communication.
  2. Supporting other data types requires writing transforms from scratch, which is extremely time-intensive and error-prone (but supports basic data types like JSON, plain text, and rich-text)

TogetherJS

TogetherJS is just a plugin kind of thing that enables users on the website to collaborate. It needs a button to be added to the UI of the web app and when a user clicks it... it provides the user with a URL that is shared with other users. they can open that URL and collaborate with the same website. It is more of a Surfing-the-website-collaboratively than a library which we can use to edit a note collaboratively.

YJS (Y-WebRTC)

Y-WebRTC is implemented using WebRTC and provides conflict resolution using CRDTs.

Pros:

  1. Support for decentralized, peer-to-peer shared data editing
  2. Fairly rich set of supported types: Maps/arrays, text, and rich text (Quill only)
  3. Out of the box support for a variety of common UI libraries (Ace, Quill, CodeMirror, etc)
  4. Offline support

Cons:

  1. Needs a signaling server to start the P2P communication.

Conclusion:

  • No matter what technology we use, we need a server... but the thing we need to choose here is should we use a WebSocket-based library or a WebRTC based one.
  • As these servers are no different from an ExpressJS app... So yes, these can be dockerized.
  • All the above-mentioned repositories are very active.
  • Basic implementation into Joplin:
    • A signaling server.
    • A Collaborative web app (which loads the note from Joplin Server API).
1 Like
2

@laurent what is your preference?
Alternatively, we would create another server

3

have you checked automerge/automerge: A JSON-like data structure (a CRDT) that can be modified concurrently by different users, and merged again automatically. (github.com)?

4

Yes, I have now... I think it's just a conflict resolution/data merging library using CRDTs for a collaborative application. So, it doesn't make the collaboration serverless, It just handles the synchronization and version history of data on every client end application which is collaboratively edited, and also the git repository mentioned this:

Automerge includes general-purpose building blocks for network protocols, but you need to use a separate library to perform the actual communication

5

From those libraries which one do you think would be best for your project, and why?

6

yjs provides lots of supporting libraries like y-webrtc, y-codemirror and y-quill, that we might need to make development simpler.

Even after selecting a library, we might need to decide beforehand on:

  • How the web application syncs the changes to the server?
  • Can any user with the link join the collaborative session? or does the user needs to be a Joplin registered user?
  • Can all the users in the session sync changes from the web application to the server?
7

Yjs seems to make sense then. As for your other questions, you might want to investigate and find out what's best within the scope of GSoC. If you need some info on how to upload or download a note from Joplin Server I can explain later on.

1 Like
8

just a personal note.
From a user perspective, you share once a note or notebook with another user done.
In addition, it would be great if permission can expire or you create separate collaboration notebooks. That matches with use-cases like OneNote, Evernote, etc.

1 Like
9

for future reference

Yjs was primarily build for shared editing on text and rich-text. Yjs exposes state as mutable data types. State is mutable by design, because some computations can be done more efficiently on mutable objects.

I would argue that Yjs is the better choice if you plan to share huge (text / rich-text) documents, because 1. its algorithm is specifically designed for shared editing and 2. it encodes its structure very efficiently in a binary format. Automerge on the other hand might be easier to handle because state changes are just operations on the shared document.

Automerge is more general purpose collaboration of data while yjs focuses on text based applications.

Benchmarks

1 Like