Hi @laurent,
I'm interested to install/run joplin server without the need of docker. At the moment I just use the docker container mentioned in the readme but since the installation should not be that complicated I would prefer not to use docker at all. Would you mind to provide the basic steps/background/requirements how to install and configure the node part? I could create and share robust scripts doing the work.
The only "documentation" I found is the Dockerfile and I'm wondering which of these steps are really required / recommended since I'm not the node guy. Any hint / insight would be helpful and highly appreciated.
1 Like
The best documentation is indeed the Dockerfile (but you should use the official one), it couldn't be more precise since it tells you step by step what you need to do, and you can be sure it will work.
2 Likes
Just thought I'd chime in with a few observations. I've taken a brief look at the server today.
It's clear in the documentation that the SQLite backend is intended for development only. I'm wondering if there's concern about the quality of the SQLite backend or if it's more a practical problem involving the Docker persistent storage side of things?
Secondly, it seems as if there's several bits of the code that effectively mean the SQLite file can't be easily moved without causing various problems, compounded by the fact if the current working directory for the server isn't it's own folder, there seems to be various instances of relative paths that stop working (E.G, images in the admin interface). This means it unfortunately isn't as trivial as changing the CWD and calling the server from somewhere else in the system (There's a few other hardcoded paths beyond the DB itself, such as the log folder).
Allowing free movement of the SQLite file would trivialise making a snap version of the server. Obviously as is the case with the existing snaps I don't expect any real attention to third party distributions, but as far as proof of concepts go I think being able to change the SQLite DB location would be the only blocker of an naive snap based package for the server. I'll experiment some more at another time (and look into just using PostgreSQL with it too), but I thought you might appreciate the insights so far.
From a database perspective SQLite is not a real multiuser RDBMS, thus not suited to run on a server while multiple users try to access the db. Sqlite is used for applcation backends (one application/user accesses the db).
That makes sense, though I wonder how much of a workload needs to be involved before that bottleneck becomes a concern, say below 10 users I'd imagine the synchronisation to be sparoidic enough that it might not actually be that intense.
Regardless I appreciate the insight and I'll experiment with PostgreSQL at some point
Doesn't JS ecosystem have something like JDBC in Java where you have common API to most databases from Sqlite to Oracle, and you can swap database vendor by changing the connection string and placing the appropriate jar file?
I think you do not understand. The bottleneck is > 1 user. I don't know how much know about isolation levels, locking (tablepace, table, row, page), ...
Simply put: SQLite locks (exclusive lock) the entire database when writing.
Where do multiple users come from in this set up? Isn't the only user accessing the database is the Joplin server process?
It should be easy to move the SQLite database somewhere else if needed. Its path is a hard-coded variable at the moment but we could make take its value from somewhere else. What would be the use case though? When you mention a snap file, is it still based on the Docker image, or are you building the Node app directly?
What happens, if 2 users sync their client at the same time?
I don't think it's really that bad. That can be handled and SQLite is more than adequate for multi-user webpages.
(On the other hand, not wanting to waste time making sure this works when one can just use Postgres is completely reasonable; I'm not saying it should be used for more than just the dev setup.)
Can be users, or can be multiple threads or processes. For example, there's a service that processes recent changes to update the share objects, and that would run at the same time as users are accessing their notes. Except that when that service is running, the database will be locked all the time because it's being written to. Regular DBRMS don't have this issue.
I am not sure how the server is designed, I think nodejs is single threaded so I'd imagine actual reads/writes related to different users will interleaved but never concurrent.
Yes and no. There is an option to use WAL which would allow people to read while someone writes. Still, it is not a multiuser database engine.
I'd echo Zblesks sentiment above, if it's unreasonable to do then feel free to ignore my intrigue on using it for production, but I do think there's a decent chance that for casual user (E.G, people on Raspberry Pi's hosting for themselves and family/friends) the performance penalty of SQLite might not be that high, but the complexity of managing it would be reduced significantly.
The use case would be that the SQLite file in a snap needs to be stored seperate to the executables for the server itself, which are immutable. If the SQLIte file could be placed at an arbitrary location freely (say for example it reads an environment variable), then it could be put into a folder that's writable and persistent. If it can't be put into an arbitary location, the security policy won't allow for it at all. In theory this would benefit the Docker build too by letting it be placed in a persistent volume there as well.
The snap wouldn't be built from the Docker image but is conceptually similar to it. The process is essentially running npm install as usual and putting the results into a compressed archive, and then mounting that archive into what could be compared to a Docker image. Add on some metadata and snap will integrate it into a proper service managed by systemd. Obviously Docker still has better UI for users on other systems, but in theory a snap of Joplins Server would be a match made in heaven. The usual worries of the sandboxing impacting integration with the desktop edition doesn't apply when everything is done over the network anyway.
I see that the iOS-app has Joplin server as sync-option, but I can see that iOS is mentioned here to be supported.
Any words in this?
That makes sense, and indeed it would be nice if that could help setting up the server quickly on a Raspberry Pi. Currently it's only possible to set the SQLite database name but in the next version I'll add a way to set the path too, it will simply be an extra env variable.
1 Like
I've given a simple proof of concept a go. It works, it should be considered alpha quality, and there's a lot to consider whether it's possible to bundle PostgreSQL for the snaps own use or how to provide an interface for configuring the snap to use existing other PostgreSQL sessions, but all in all it provides a decent proof of concept and actually synchronises with the latest pre-releases.
If anyone fancies giving it a go. Please keep in mind that I'm not yet commited to publishing a snap properly to the Snap Store itself until the server matures some more and all the available options can be assessed.
Considering running unsquashfs on the snap file itself, you'll see that it's almost literally just a compressed archive of a build directory and some extra metadata. (Including shipping the desktop and mobile clients, I'll work on reducing the filesize later 
)
If you want to try with SQLite, the latest version allows setting the database path using the SQLITE_DATABASE env variable.
1 Like
The version above is using your SQLITE_DABATASE patches already, thanks for the work there Laurent.