Joplin Server pre-release is now available

Development
1

The first release of Joplin Server is now available as a pre-release:

https://github.com/laurent22/joplin/blob/dev/packages/server/README.md

What does it sync with?

You will need Joplin v1.6+ clients, which are available as pre-releases for desktop and Android.

What does it do?

At this point, this server allows you to sync any Joplin client with it, as you would do with Dropbox, OneDrive, etc. So in that way, it's not essential. Long term, the goal is to add collaboration features:

  • Sharing a note with anyone, using a URL. When the note is changed, the content at the URL is changed too.

  • Sharing a notebook with a user on the same Joplin Server instance. For example, if you share a notebook with another user, that user will see this notebook in their desktop or mobile app, and will be able to edit the notes, etc.

Any improvement over Nextcloud?

For now, one benefit of using Joplin Server, compared to Nextcloud or WebDAV in general, is that it is much faster and resource efficient. I've done a basic test with Nextcloud and Joplin Server running on the same server. Both have mostly default settings (except Nextcloud which uses Redis for file locks):

Type Nextcloud Joplin Server
Sync 744 items: Existing client with new sync target (mostly upload) 24 min 5 min
Sync 744 items: New client with existing sync target (mostly download) 7 minutes 0 min 51 sec
Change one note and sync (mostly upload - to test sync overhead) 18 seconds 6 seconds

Why is it so much faster? I assume it's in part because the WebDAV protocol is terribly inefficient. It sends unnecessarily large XML blobs for every request, which are time consuming to download and parse. It also doesn't support delta sync (unlike Joplin Server), which means the complete file list needs to be downloaded before syncing in order to compare the local and remote items.

Finally it's possible that Nextcloud file locking system means an overhead on each request. It shouldn't be much since it's handled by Redis but who knows. Joplin Server doesn't need locking as data consistency is handled by the clients.

So just by following common sense and transferring only the required data in a sane format (JSON), we can have something more performant. In my case, I appreciate that the mobile app no longer freezes when it starts synchronising - before it would do that due to the large XML WebDAV file that needs to be parsed.

Stability

I've been using it with the desktop and mobile app for a few weeks now and haven't had any issue so far. The server also passes all the existing sync-related test units (sync, e2ee and lock handling) so I'm reasonably confident it can already be used. As always though, keep making backups in case there's any issue.

Suggestions are welcome

This is still a pre-release and if you notice anything off please let me know. In particular, I believe it doesn't gzip responses, so that will need to be added. The process also doesn't restart when it crashes, which could be solved with pm2.

Also if you have any idea on how to make installation easier, your feedback is welcome.

44 Likes
3

Right, IMO this is the responsibility of the process manager (systemd, or in our case pm2 or docker).

Is this part of the module or does it have to be coded separately? gzip has pretty much been the standard for many, many years, but over the past 5 years, deflate and brotli got more and more common. e.g. if you just need to set a flag, I'd rather use brotli than gzip. If you have to code this part yoursef, gzip might be easier to add.

4

I would use Koa's compress library for this, which it seems enables brotli by default. Are there any known compatibility issues with this format?

5

Not that I'm aware of, but I'll do some additional research.

6

Hi Laurent -
just wanted to say thanks for all the work on this. I've just started using Joplin last week and loving it together with the server.
Migrated 876 notes from Evernote to Joplin and the server without any issue.

I've created a Docker Image to help getting it up and running easily as well as keeping it up to date. The only thing I'm still trying to figure out is how to get it to compile on arm64, given that both libvips and sharp need to be compiled from source for that platform.

4 Likes
7

Glad to hear it's working for you @florider, but how did you manage to run it you can't compile some of the packages?

both libvips and sharp need to be compiled

The truth is, sharp is not even needed at this point for Joplin Server, although it might be later on. What's the error message for libvips? I don't know what it's for.

8

So, the docker image builds fine on x64. When trying to build it on arm64, the build fails with:

#52 [linux/arm64 20/26] RUN npm run bootstrap
#52 10.31 lerna notice cli v3.22.1
#52 10.35 lerna info versioning independent
#52 11.10 lerna info Bootstrapping 8 packages
#52 11.51 lerna info Installing external dependencies
#52 1656. lerna ERR! npm install exited 1 in '@joplin/tools'
#52 1656. lerna ERR! npm install stdout:
#52 1656. 
#52 1656. > sharp@0.25.2 install /home/joplin/packages/tools/node_modules/sharp
#52 1656. > (node install/libvips && node install/dll-copy && prebuild-install --runtime=napi) || (node-gyp rebuild && node install/dll-copy)
#52 1656. 
#52 1656. make: Entering directory '/home/joplin/packages/tools/node_modules/sharp/build'
#52 1656.   CC(target) Release/obj.target/nothing/../node-addon-api/src/nothing.o
#52 1656.   AR(target) Release/obj.target/../node-addon-api/src/nothing.a
#52 1656.   COPY Release/nothing.a
#52 1656.   TOUCH Release/obj.target/libvips-cpp.stamp
#52 1656.   CXX(target) Release/obj.target/sharp/src/common.o
#52 1656. sharp.target.mk:139: recipe for target 'Release/obj.target/sharp/src/common.o' failed
#52 1656. make: Leaving directory '/home/joplin/packages/tools/node_modules/sharp/build'
#52 1656. 
#52 1656. lerna ERR! npm install stderr:
#52 1656. ERR! sharp Use with glibc 2.24 requires manual installation of libvips >= 8.9.1
#52 1656. info sharp Attempting to build from source via node-gyp but this may fail due to the above error
#52 1656. info sharp Please see https://sharp.pixelplumbing.com/install for required dependencies
#52 1656. ../src/common.cc:23:22: fatal error: vips/vips8: No such file or directory
#52 1656.  #include <vips/vips8>
#52 1656.                       ^
#52 1656. compilation terminated.
#52 1656. make: *** [Release/obj.target/sharp/src/common.o] Error 1
#52 1656. gyp ERR! build error 
#52 1656. gyp ERR! stack Error: `make` failed with exit code: 2
#52 1656. gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
#52 1656. gyp ERR! stack     at ChildProcess.emit (events.js:314:20)
#52 1656. gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:276:12)
#52 1656. gyp ERR! System Linux 5.4.0-1032-azure
#52 1656. gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
#52 1656. gyp ERR! cwd /home/joplin/packages/tools/node_modules/sharp
#52 1656. gyp ERR! node -v v12.20.0
#52 1656. gyp ERR! node-gyp -v v5.1.0
#52 1656. gyp ERR! not ok 
#52 1656. npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/fsevents):
#52 1656. npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"arm64"})
#52 1656. 
#52 1656. npm ERR! code ELIFECYCLE
#52 1656. npm ERR! errno 1
#52 1656. npm ERR! sharp@0.25.2 install: `(node install/libvips && node install/dll-copy && prebuild-install --runtime=napi) || (node-gyp rebuild && node install/dll-copy)`
#52 1656. npm ERR! Exit status 1
#52 1656. npm ERR! 
#52 1656. npm ERR! Failed at the sharp@0.25.2 install script.
#52 1656. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
#52 1656. 
#52 1656. npm ERR! A complete log of this run can be found in:
#52 1656. npm ERR!     /home/joplin/.npm/_logs/2021-01-04T11_38_04_880Z-debug.log
#52 1656. 
#52 1656. lerna ERR! npm install exited 1 in '@joplin/tools'
#52 1656. lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.
#52 1656. npm ERR! code ELIFECYCLE
#52 1656. npm ERR! errno 1
#52 1656. npm ERR! root@ bootstrap: `lerna bootstrap --no-ci`
#52 1656. npm ERR! Exit status 1
#52 1656. npm ERR! 
#52 1656. npm ERR! Failed at the root@ bootstrap script.
#52 1656. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
#52 1656. 
#52 1656. npm ERR! A complete log of this run can be found in:
#52 1656. npm ERR!     /home/joplin/.npm/_logs/2021-01-04T11_38_05_216Z-debug.log
#52 ERROR: executor failed running [/bin/sh -c npm run bootstrap]: exit code: 1

It seems two of the dependencies (sharp and libvips) aren't available as a prebuilt package for arm64 on Linux (as per https://sharp.pixelplumbing.com/install#prebuilt-binaries) resulting in the build failing on non-x64/x86.

You can find the Dockerfile and Github actions code here in the repo and the successfully built images for x64 here.

9

can the clients have an option that allows for self-signed certs and also is basic auth supported in the client?

10

Can this be installed without Docker? I'd rather it just be an application server on a VPS or baremetal box. Docker is another layer I don't want to have to deal with :slight_smile:

BTW, this is awesome. I love Nextcloud, but it's so frustrating for sync operations like this!

11

You could follow the instructions in Dockerfile and run them manually I guess, but currently it's not possible to install it without docker.

12

I'll make the certificate options of WebDAV available for Joplin Server, but I don't think basic auth will be supported.

1 Like
13

@laurent I looked at the Dockerfiles and I'm really puzzled by something. It seems that the storage for the db is not persistent. This will delete all data when the container is recycled, which happens rather often in the Docker world.

14

It is :wink: It's all stored in the Postgres DB, so you'll need to manage the pgdata directory either through a docker volume or a mount.

15

It's indeed persisted using a Docker volume, which as I understand is the recommended way to store persistent data. We could also mount a local directory, but then we have to make sure permissions, security, etc. is right.

So you can stop the container, and restart and the data will still be there. It will be gone however if you run the "down" command, but that's usually what you'd want.

16

All data is stored exclusively in the Postgres Database, right? There's nothing stored on the filesystem directly?

17

No, it's all in the database.

1 Like
18

I did not see either one of three ways to use persistent storage in any of the docker files:

  1. dir mount
  2. named volume
  3. volume plugin

Unless I'm missing something, there is no persistent storage. I'm sorry, but I really must be blind. IMO the data is gone as soon as you remove the container. (This is bad and should not be the case.)
Also, what happens, when you upgrade Postgres? (That's another story though.)

P.S.: Persistent storage does not mean that the storage won't be deleted when restarting the container - that is a given and in many cases even required when using stateless services. Persistent storage means the data persists in case the container is gone.

19

See an example docker-compose file here with the image and the volume definition for pgdata.

1 Like
20

Yep, this one makes more sense. I'm rather partial to named volumes, but this is more than enough (depending how postgres is actually setup in the container - e.g. transaction logs, are they also under data?).