Hi Chen. Many thanks for your background sync version! I've been using it for a short time and it seems to work really well. A few quirks and bugs perhaps (sometimes background sync seems to stop) but overall a massive improvement and amazing work! Makes the Android version actually usable : )
I was wondering if you're going to continue development with this and keep it up to date with current/new features, eg Trash? Would be great if you could incorporate that and keep it current!
I don't know too much about the development of Joplin but I'm hoping your incredible work can make it into the main Android release at some point. It's such a massive improvement on the current version which is tricky of course to keep foreground on a phone.
Let's hope that something similar comes up for iOS, too! Background sync for mobile would be a huge gamechanger and close the gap to all other note-taking apps I know of (which don't have this problem, probably because they're native developments for Android or iOS).
Just coming from Evernote I am actually baffled at what Joplin can do. Background syncing is also for me a missing feature. Not because Evernote could do it (it does not) but because sync on Joplin is longer (in spite of the fact I use Joplin Cloud) and particularly so with encryption (which Evernote can also not do)
Any chance that this background sync makes the main distribution? Is the non iOS readiness a show stopper for a full deployment?
Still using it daily? Any downsides? I'm tempted to try it out. Will I need to rebuild the resources/database at the first sync? Is it better to uninstall the official Joplin apk first, with all its data/cache?
Also, does it drain the battery a lot when not in use?
Thanks for this great addition
It's good to see that my message was helpful for you. Unfortunately, @Chen 's application is not working. When the synchronization button is pressed, the application freezes.
My application was just a prototype, it contains some bug. As far as I observed, sometime, it will block phone to sleep. Still try to find time to debug it.
I can merge the latest change when I have time. Hope I can finish it next month.
You're doing an amazing job, truly. Thank you so much. It's wonderful to read your message again. Know that there are followers eagerly awaiting your work.
I am also eagerly following the topic of mobile background syncing as my wife does not âfollow the rulesâ to make mobile syncing work so I end up having to sync her phone manually all the time and resolve conflicts (as she expects it to âjust workâ - we transitioned from OneNote where syncing wasnât always perfect but at least she didnât have to open the app - sync it - make the changes - then sync it again before closing).
this would be a giant improvement in my quality of life to have this working!
@Chen I am hoping you will pick up this work again and save us all from our sync problems!
If nothing else - I wanted to bump this topic and keep it open in the hopes someone will pick it up and get at least Android syncing in the background.
I had another look into the topic. Itâs been a while since I followed discussions on background sync, but the PR which was rejected due to âreliability reasonsâ implemented a background service, for which the Android OS enforces some strict limitations.
A foreground service requires a persistant notification to remain active, but as far as I can tell they have less restrictions compared to a background service. It would require writing some native Android code, and Iâm guessing the only thing which may not be straightforward to implement would be to allow the user to opt out of the background sync (if they donât want to have the persistent notification and additional battery usage), which may require writing a UI to toggle this in native Android code, as Iâm not sure the node code would be able to have access to this configuration
EDIT: Looking at the code in the linked article, it seems you can start and stop the service from js code, so likely you can use the existing setting screens to toggle whether to use the service
Thanks for another look into the topic. I have now spend a bunch of time reading as many past threads on the topic as I could find and I see the issues involved are not easy to solve. Thanks for the link to the foreground services - interesting reading.
I am going to throw out some ideas here in case it provides some âfood for thoughtâ and perhaps gives some ideas to those who know the inner workings of Joplin.
Hybrid approach:
I am wondering if there is a hybrid approach which might be more reliable and solve some of the problems people have. What I would suggest (if it is possible) is to implement a âlazy syncâ which would allow the app to stay awake just long enough to finish an already started sync. See my use case below to understand why this would help for me and perhaps others.
I have no idea if this is possible or how hard it would be but how about this:
When a sync is initiated then a notification is created and a foreground service is used to continue the sync until completion or failure.
When the sync finishes (or fails) then the foreground service is stopped and the notification is removed.
For this to solve my use case a sync would need to be initiated as soon as a change is made to a note, and perhaps when the app loses focus to ensure any changes made at least attempt to sync.
For this to solve my use case a sync would also need to be initiated when the app is opened or gets focus - to ensure that any changes made elsewhere get synced before I make changes. To make this clear to the user it seems like the app would need some sort of visible icon on the UI which shows sync status clearly. Currently I use the plugin âExtra Markdown edit settingsâ to show the sync status while editing a note - this helps tremendously in trying to remember to wait for the sync to finish after making changes - but doesnât help with waiting for the initial sync before starting editing.
By doing this it would seem to avoid a battery draining mess by not syncing all the time, but would allow a sync to finish.
Problems with Hybrid Sync:
Right now I donât think there is a mechanism for a sync to âtime outâ and stop. We are often off-grid and have no internet access - it looks like on both mobile and desktop versions that Joplin keeps trying to sync with no hope of finishing and just burns CPU cycles forever. For Hybrid Sync to work there would need to be some mechanism for the sync to just give up at some point so the phone does not stay awake trying to sync forever. Perhaps this should just be itâs own feature request.
This does not solve the problem where you make a change while offline and the lazy sync times out or fails. If you go back into a location with data the app is not going to wake up and sync because this is not a true background sync solution.
Use Case:
Both me and my wife use Joplin to manage a shared shopping list (using a self hosted Joplin server). When we think of something to add to the list we open the phone app, add a new item to the end of the shared list and then immediately lock the phone and put it back in our pocket. We are also often off-line for large amounts of time - so any solution also has to have timeouts when sync will not work at all. I understand that when completely offline we have to be careful about conflicts and we do that by only using one of our phones for the shopping list when we are offline.
This creates two possible conflict scenarios (when we are online):
Upon opening the app - if we donât wait for a sync to complete on opening the app then we may add an item to the list which then creates a conflict if the other person had already changed the list.
Upon adding an item to the list and closing the app - we may not have waited long enough for the sync to finish so our changes would not be sent to the server.
It seems that the case 1 in the article âBackground App Refresh (iOS) and Background Processingâ would work for the hybrid approach above to attempt to complete a sync when the app goes into the background. It seems to me that this would work most of the time and if iOS or Android still stops the app occasionally because the sync takes to long then it would still be better than the current state of the mobile app.
@DoneWorkin The 30 second limit on iOS for a background service finish off task seems a bit short tbh, though it doesnât mention specifically the limit for Android. Regarding foreground services, it looks like in newer versions of Android there are more restrictions in place, most importantly a dataSync service can run only up to 6 hours in a 24 hour period (see https://developer.android.com/develop/background-work/services/fgs/timeout) so that means you canât really have it as a constantly running service.
Honestly though I think a 30 second finish off task on both iOS and Android is certainly better than nothing. It wont stop you needing to keep the screen on and Joplin open for the long initial sync when setting up a new client, but in most cases 30 seconds is plenty of time after you switch to another app or turn the phone screen off to complete the âsync as you typeâ in most cases (unless uploading large attachments or you happen to not have internet connection at the time).
Rather than specifying schedule intervals for a task to run, which other libraries seem to do, with this one you can start the service on demand (the app must be in the foreground when doing so) and stop it once the task has completed. So the Joplin code could be made to always execute the synchronization process as a background task, whenever the sync is triggered automatically or manually. Then if you switch app or put the phone to sleep, it will continue for a certain amount of time, or the service will stop itself if it finishes before then. It seems like this library will automatically switch between background and foreground services for Android, which means you can run the sync in the background for longer periods than on iOS, and youâd be able to have up to 6 hours of actual sync time every day, rather than 6 hours of a constantly running service.
EDIT 2: I have to say though, even if the above were to be implemented, Iâm not sure it would which actually stop you getting regular conflicts for your use case. Iâve noticed the automatic âsync as you typeâ mechanism that Joplin has, regularly does not sync changes made while the sync is in progress, at the point the sync completes. I have to manually press the sync button again to sync the latest change often enough. So letting sync finish in the background is not going to resolve that issue, if the sync finishes but has not actually registered the latest change made while sync was in progress.
Iâve only used the free sync option before, syncing with Google Tasks, but if itâs only shopping lists you need to share with your wife and you want a self hosted todo app with E2E encryption option, then Tasks.org may be a potential alternative to explore for your list sharing use case.
Interesting mrjo118! I think what you found may actually be one of our main pain points - Thanks!
Now that you pointed that out - it I can see it happening when we are editing our shopping list and is definitely causing many of our problems.
FYI - thanks for the other suggestions - we do actually use a lot of other shared notes, the shopping list was just the easiest to describe. So I think Joplin is still our best bet for our uses.
Also - after reading a bunch of the links you provided as well as the issue trackers for those libraries it appears that both iOS and Android regularly change what their requirements are and how a âbackgroundâ or âforegroundâ sync should work. So I can see why this has not been implemented so far - as it seems it would be a constant source of new work to keep up with what Google and Apple demand - and what they keep breaking.
I am hopeful your change makes it into the next release and will take care of a large portion of our problems - when we are on good internet we often get really quick sync behavior (using the self hosted server seems pretty quick).
Iâd also like to point to the Extra Markdown editor settings plugin if you are interested. It is a mobile compatible plugin and has a feature where you can show a sync indicator icon while editing a note, which will show you when the sync has completed. Currently where not all changes get synced, the save icon remains showing after stopping typing. But with the changes on my latest PR, the sync retriggers and always results in the save icon being cleared. You might consider asking your wife to install this plugin as a clear indicator when it is safe to exit the note and close the app, instead of her having to check the sidebar to do this after exiting the note.
