Workaround for slow initial bulk sync after evernote import?

Support
1

Appreciate the effort to support dropbox API sync integration and encryption features!

A key issue is some clients (Linux/Android) only seem to do syncing while the app is in focus and this makes the initial import and sync of bulk notes from evernote very cumbersome.

TL;DR: Joplin local “file system” sync to a Dropbox folder and letting the Dropbox agent do the sync is much faster! Use this as an interim hack to jumpstart bulk item sync.

Process:

  1. Dropbox app installed and running
  2. Setup devices to sync from “file system”
  3. Desktop client: Point the “file system” sync to the standard /Dropbox/App/Joplin subfolder of Dropbox (need to specify full path) - likely to work for Linux, OSX and Windows. This can possibly be left as is instead of needing to setup Dropbox sync.
  4. Setup encryption if so desired (Dropbox keeps history of files so you might not want unencrypted versions synced to dropbox).
  5. Import from evernote the Desktop client and wait for dropbox to complete syncing the folder
  6. Sync from the other Joplin desktop clients using the same “File system” sync method at the /Dropbox/Apps/Joplin location (not Dropbox sync!)
  7. On Android Dropbox, set the App/Joplin folder as “offline” and wait for it to be fully synced
  8. With an Android file manager, view the path of the offline folder, e.g. /sdcard/Android/data/com.dropbox.android/files//scratch/Apps/Joplin
  9. On Andorid Joplin, use the “File system” sync method with the offline scratch folder location of dropbox (Android app will need storage permission as per normal). Then sync
  10. On Android Joplin, after sync is 100% done, switch to the “Dropbox” sync method (uses the Dropbox API, not filesystem) in order to be able to edit notes. Unfortunately, it seems some oddity occurs where this sync method change will trigger an update tigger of sorts for “EVENT_NOTE_ALARM_FIELD_CHANGE”, but at least all items are now on the mobile device.
  11. [Optional] On the Desktop clients, you can switch from “File system” to “Dropbox” sync (But I leave the filesystem sync inside the Dropbox folder as is since faster sync helps avoid conflicts!)

Out the box, I tried to run Evernote import (~1000 objects) + Encryption + Dropbox sync, but it was just unreasonably slow and I couldn’t get 2x Ubuntu & 1x Android devices to play nice with sync + encryption. This is with the Linux desktop client and mobile client sync with ~500 notes, ~450 resources and ~50 tags imported from evernote. Spent a whole weekend fiddling with it in the officially documented way.

I experimented comparing the following setups:

  1. Encryption + Dropbox sync method
  2. No Encryption + file-system sync method into a Dropbox folder (let dropbox handle syncing files)
  3. No Encryption + Dropbox sync method

Encryption + Dropbox was so slow I was unable to sync clients over 2 days and multiple attempts

With encryption + Dropbox sync (Joplin authorised to use Dropbox API directly), I think in part the existing open bug with Linux/Android clients seeming to not sync when the UI is active and open (e.g. https://github.com/laurent22/joplin/issues/312) could be making the performance and sync speed issue more extenuated.

Device 1 - Linux Desktop: 1st Linux system took several prods to sync. Didn’t seem to sync unless app was in focus.
Device 2 - Android: Very cumbersome and slow to get all ~1000 objects synced after multiple prods. And, after finally catching up on synching the master key, it seemed to decrypt everything eventually. But then, when adding/changing a note on another, it got stuck decrypting changed notes…
Device 3 - Linux Laptop: Only synced notes and seemed to get stuck at pulling down tags, resources and master key. Had to delete config and try again.

Encryption + file-system sync was much faster

After the evernote import, Joplin took only a minute or two to sync from SQLlite database backend (I assume) to “local” file system. The Dropbox agent (not Joplin) synced the folders/files quite quickly on 2x linux systems (and can benefit from local LAN sync!)

On Android, it was a bit of trick to mark the Dropbox folder as offline and find the full path to it for config in Joplin with “File system” (not Dropbox) target. While the Android sync was fast from the file system, changing and item on the android side didn’t work out. Joplin would sync writing to the Dropbox scratch folder, but the Dropbox App on Android doesn’t seem to monitor or upload changes to the the scratch folder (probably assumes apps don’t use that directly). I’ve noticed apps that get file shares via dropbox “open with” use a content://com.dropbox.android.filecache.filecache/ folder and if they change, drobox notices that, but it seems to only use scratch to pull down files and expects a user to share from there.

And of course, iOS doesn’t allow apps to directly access any sort of shared storage as far as I know, so this approach can’t work for iOS.

No Encryption + Drobox sync was still slow and unreliable, so encryption isn’t the problem

While sparing the details, I also tried just Dropbox sync plaintext was still very slow and unreliable at bulk sync after an evernote import. It works well enough for a single note test case, but took really long to sync multiple notes. So the performance problems aren’t just related to encryption.

6 Likes
2

Thanks for sharing the info! I will add a link to your post in the FAQ as it’s indeed a recurring problem.

3

Maybe we should pin this topic in the Support category.

1 Like
4

@JPvRiel I found the same issue, it is a basic issue with very small file making Joplin not usable.
Do you happen to know how to get the local path to the dropbox folder on iPhone (iOS)?
(similar to the Android path you provided)

5

Just to point out another few issues.

If you do the local sync before the dropbox sync, Dropbox API creates a new folder named Joplin (1), rather than looking for a folder with the Joplin name. So if you go through the whole sync palaver, it won’t sync with the mobile app properly. What I think might be a workaround is running the Dropbox sync on the Android app with an empty database to create the Dropbox folder (before step 2), then run through the rest of the steps as normal, changing the Android sync to file system.

I’ve learnt this the hard way, Dropbox doesn’t like dealing with large amounts of files being moved around and have had to uninstall/delete cache a few times when the desktop client hangs.

Note, you need a pro account with Dropbox to download folders in 2020. I think they offer a 30 day try out, so if you haven’t taken that already, you could take advantage of it, as it’s a one time sync step.

1 Like
6

It looks like this guide is no longer possible, as Dropbox Android no longer allows offline files to be accessible to the Android system (steps 7 / 8).

I'm attempting to follow the rest of the process by circumventing these steps with a third-party tool to download from Dropbox.

7

Is this guide still relevant in 2024 and possible to implement?

8

I doubt it. It didn't work for me in 2020, and I haven't seen Dropbox notify me of any big changes like this in their Android implementation.