It would be a good idea to have periodic automatic backups that get uploaded to your sync service. It could be in its own separate folder and maybe compressed to save space. This would be beneficial for many reasons, for example: there is a bug in syncing that deletes the notes from your service.
3 Likes
I agree, that would be cool. It would be awesome if once a day a JEX file was generated and place on your sync service/server. It would also be amazing if the JEX backup were to have a timed duplicate section like being able to choose how often it backs up or how many backups it stores until it deletes old versions to make room for new versions.
I think that would be a fantastic improvement to the project!
That can be done with Joplin terminal version in a scheluder
I see it is possible to do that with the export option in the terminal documentation but it would require a script to be written wouldnât it? Does the Joplin CLI support exporting and then syncing that file to the same place you put the main set of data?
Either way, I think it would still be something that people would benefit from heavily if it was a part of the main desktop app as an option. The CLI is a good option for now but ideally it would be connected to the sync options.
Iâm sure, but it would be nicer to have it in the desktop version for people who donât have the technical knowledge.
1 Like
I would like to add my vote in here as well. I understand the current syncing-is-not-a-backup idea, so that leaves a gap for folks that use the Desktop version and would like to automatically backup their Joplin notes.
Ideally the backup option would be password-protected/encrypted and also sync those backups to whatever sync endpoints are configured in Joplin as well.
Thank you for your time.
I scripted something in AutoHotKey which works pretty well if your interested. The only challenge is to get Joplin to always default to a specific backup directory.
I would like to bump this, this should be not too terribly hard to implement, since an âexport allâ feature already exists, it would mainly require these changes:
required features
- add option to enable/disable automatic backup
- add option to specify backup folder
- add option to specify schedule
ânice to haveâ features:
- add option to specify export format ? (or just force it to be a JEX format since that is the most âlosslessâ format
- add option to compress backup
- add option to do âpruningâ of backups, aka what backup software tries to do (aka 7 backups for the current week, then when the week advances you delete all but the last backup for that week, then when the month advances you keep 2 backups for that month, etc
questions:
should the mobile version of joplin be considered?
I too would like to add my vote for this feature.
Yes, it can be scripted, but having a well-tested built-in option available would be better than everyone having to build their own scripts. Furthermore, not everyone is similarly comfortable with scripting, particularly for something as important as a backup.
Ideally, backups could be set to happen to a different sync target, so that the 3-2-1 backup principle can be better applied.
I mainly left out the âdifferent sync targetâ thing since that seems like a much bigger ask at the moment, since it seems that only 1 sync target is available at one time, and joplin doesnât have support (even for normal note sync) to support different targets at the same time
but yes, having a different sync target would be a nice feature, and would enable the feature to work on mobile (for people who only use joplin on mobile devices)
For those who are seeking for a solution, there is an available plugin joplin-plugin-backup
1 Like
I use WebDav for sync and I wanted to automate it once a day to export the raw format to markdown. The joplin CLI is terrible to use for such automation because it isn't stateless. You have to use separate commands to first import note data and then use a different command to export the data.
You have to delete the local sqlite database to guarantee for it to be deterministic.
I managed to make this shell script which works for me:
rm ~/.config/joplin/database.sqlite
joplin import --format raw /path/to/input
rm -r /path/to/output/*
joplin export --format md /path/to/output
You could probably change the input and output formats for other use cases, like generating periodic JEX backups.
Can you please justify the commands you're running there, because as much as I don't want to come off as rude by accusing this of being AI generated content, a 3 hour old account bumping a 2 year old thread with code that looks like it would just delete all your notes certainly looks suspicious.
I could be entirely wrong and if so I'm sorry, but all I'm seeing that code do is delete all your data and then 3 lines of importing data from elsewhere and then exporting the data you just imported for no gain.
If a user runs that first line and doesn't have backups, they're absolutely toast.
4 Likes
Okay, so like I said, I use a webdav to backup my data, this is on the server that is hosting my webdav which is pointed to /path/to/input/ which is in the "raw" format on the filesystem. This is my main note sync location. My goal is to convert this data format into some other format every 24h as an extra backup and to also have my notes constantly in plain "md" format (/path/to/output/).
To accomplish this, on this Linux machine I installed the joplin CLI which I don't use for note taking, I just need it for performing the import and export. The database that it defaults to for me is useless and I delete it before every run in order to get a deterministic result, this sqlite database is just a mirror of my actual data source which is on /path/to/input/. After I delete it I perform an import so that it regenerates with the newest notes from my webdav save location. And again to clarify I DO NOT sync my sqlite with my /path/to/input. I just perform a one way import, just a read operation on my /path/to/input/.
After that I delete the files from my output directory which contains the "md" export so that I make sure I don't keep any deleted notes and notebooks, and immediately after that I export them there again.
You could change this part for example instead of overriding a single location to create a new directory depending on the date if you want to have historical daily exports or something.
I think it goes without saying that you shouldn't run a stranger on the internet's bash script blindly without knowing clearly what it is meant to do and what the paths represent.
I hope this explanation clears things up 
And hope I passed the touring test 
(obvious "human" joke)
That script doesn't run every 24 hours, for that you'd need something like cron or systemd-timerd. If that script does work as advertised, there's still nothing automatic about it and I'd expected to have seen something like a systemd unit that's instructed to run every 24 hours or every boot or etc, not just the literal script that's being run to do it.
The deterministic results also don't make sense. Either the import/export is perfect or it is not. Why would we care about the order in which the app generates the export, so long as the final result is correct? What does deterministic mean in this context?
And, you're not supposed to manage the sync target directly. The app does that itself. The files in the sync storage are not suitable for import directly, because they contain thousands other files containing metadata and revision history and lock files and etc that aren't purely note content like an export itself is. Messing with the sync folder directly is how people lose notes and we encourage people to never do it.
If the client is set to sync with WebDav, then why are you importing at all? The app should determine it's out of sync and restore state automatically. The objective is to export notes, not import them.
Sorry, but I'm still not entirely sure and still feel that leaving instructions that would delete most peoples files without any further explanation sums up, as the kids say, as "sus".
AI also has a problem of having to explain everything thoroughly but limit it to about the length of your posts. It'd be easier to convince me you weren't using AI if it was a single paragraph. (I believe ChatGPT and Bard and various others have some controls on verbosity if it helps).
Exactly, so why are the instructions so unclear? What's the bit that actually makes this automatic? Or the warnings that you're about to potentially lose all your data and should check backups first?
Ok, let me try again to explain this. I took the time to draw a diagram so that it is hopefully clearer.
- I am syncing my current devices (IOS Phone, Linux Desktop) to a Webdav server that is hosted on my Debian homelab server.
- This Webdav Server is a Docker container which is mounted to a
/path/to/inputdirectory on my NAS hard drive. - Most important thing about this whole setup is that the Webdav server is the ONLY thing that writes and modifies this folder, I do not manually edit anything here.
- I want to convert this "raw" storage format into other formats for other convenience use cases like:
- read-only markdown files that don't rely on Joplin clients
- hypothetical daily jex backups that I don't personally do, but the OP requested something similar and other users in this post suggested that could be theoretically done with the joplin CLI
- The Joplin CLI is a stateful program that stores it's current notes in a sqlite file. And has the ability to export those notes to any of the supported formats.
- The Joplin CLI does NOT have (as far as I know) a
converttype command which takes an input format and outputs in a different format while not writing any intermediate steps to disk. Although the import command does have an--output-formatargument which I didn't manage to find out what it does. - So I concluded that I needed to first run the
joplin importcommand which will load the "raw" notes into a intermediate sqlite format, and then I would need to run thejoplin exportcommand to convert this sqlite format into my desired format for my use case. - This whole concept of using an intermediate sqlite database is stateful, and, depending on how it is implemented might have a side effect that deleted notebooks and notes won't be deleted in my export for example, if it just adds new things.
- Because I don't want to bother finding out, and because I don't use the joplin CLI on my server for actual note taking, the sanest thing to do is to just delete the
database.sqlitefile before every import-export and just guarantee that the result will be deterministic and will depend only on the input structure.
This is the solution that I came to in order to accomplish what I wanted to accomplish. I didn't find any simpler way to do this from my research, and this was the closest post that requested something very similar to what I wanted.
Since I came up with a somewhat okay working solution for my problem I decided to post it here for others which might come across this posting searching for something similar.
From my visual illustration you can see that my setup is linear and nothing is written back to the sync directory. The devices and Webdav are not aware of and are not affected by this extra exporting behavior going on.
There may be a better way to do this that I just haven't found suggested anywhere, but I never phrased my solution as definitive.
Now to directly respond to some of the things you wrote.
Yes, I setup a cron job to run it every 24h, I never said that the script itself does that.
That is why I always make sure to only perform read operations on my sync directory. (apart from the Webdav)
I am not here trying to sell other people my scripts, or to try to mess up somebodies' system. I have nothing to gain from that. I am just sharing insights that I found while trying to accomplish a somewhat similar task to that requested from the original poster.
I work with different LLMs daily for my work and I can tell you that there are language models that are trained and fine tuned to vastly different styles. Having a feeling of how ChatGPT or other mainstream language models respond doesn't mean that you would be able to distinguish AI generated content by any LLM. As time progresses there won't be any real human checks on the internet. AI is able to understand and generate images and other types of content, and will eventually outnumber us on the internet.
I don't see your whole point of my writing being suggestive of an AI. Even if I was an AI, if I state opinions and give facts that can be checked and tried and are correct, it shouldn't really matter what I was.
But I assure you that everything that I typed till now was solely typed by me without any help from AI tools apart from my browser's spell checker. If writing in a way similar to how an AI writes is a negative comment on my writing style then I guess I shall take that critique.
I am not trying to write a tutorial with clear instructions. My goal isn't wasn't to explain what I did clearly step by step so that anyone can understand. I just wanted to take out 5 minutes of my day to quickly share how I accomplished a similar task. I am not suggesting that everyone should start using this hacky solution.
On the contrary, I would want only people that know what they're doing and that can do extra research on the commands to be thinking of attempting this.
Fine:
Use at your own discretion. I am not responsible for any loss of data or other unwanted side effects that might arise from incorrectly using my shared script. My script comes with NO warranty whatsoever.
1 Like
Thanks, I hope you accept my apology. With the context, this makes a lot more sense; unfortunately I've seen a plague of AI in other Discourse forums (mostly with the intention of leeching SEO by embedding links by editing posts on old threads so no one notices) and unfortunately on the probabilities this looked like similar to that. Hopefully you'll not take it personally since truly it's difficult to tell at times. (The 2 year old post and new account behaviour is often the giveaway most other times, and admittedly it's an awful heuristic to use but heuristics are sometimes all we've got).
Thanks for your detail and politeness, I'll put my pitchfork back where it belongs :).
1 Like
Haha we are cool.
I don't have such experience getting griefed by AI bots so that's why I didn't understand the immediate hop to accusing my comment for being AI generated. But since you have experience with that and the limited context in this case on which to judge I could see why your alertness is high.
You call me polite, but to be fair I was slightly annoyed by the situation, and it probably reflected in my writing as being a little bit passive aggressive. So I apologize for that.
As a positive to the discussion I hope someone in the future finds my more detailed explanation here more useful.
Have a good day/night.
Honestly, I was discussing this thread with my friends and came to the conclusion that had you thrown some expletives my way (not entirely undeservingly considering I was both wrong and very direct about the suspicions), it'd have actually have made me change opinion quicker as a more "human" response. The bog standard AI you'd get on online services is pretty strict these days on how much abuse it'll allow any user to generate and will attempts to outright tell you off for trying a lot of the time.
So no, considering my accusations you were actually incredibly polite and considerate :). And actually the way you're using Joplin is fairly unique so I'd imagine there'll certainly be someone who finds it valuable.
I'll take my leave from this thread now because I've essentially derailed it a bit, but hopefully I've not put you off the community, we're usually a lot friendlier than that, myself included, and you're welcome here, fellow human. 01010011 01101111 01110010 01110010 01111001 00100000 01100001 01100111 01100001 01101001 01101110 00100001
2 Likes