Unsyncable/Phantom Notebooks Between Linux and Android Devices

Operating system

Linux

Joplin version

2.13.6

Desktop version info

Joplin 2.13.6 (prod, linux)

Client ID: 3adf02dec2eb4428a30c38d233171131
Sync Version: 3
Profile Version: 44
Keychain Supported: No

Revision: e7dd981

Sync target

Joplin Cloud

What issue do you have?

Hello Joplin Staff and Community,

Could you please help me with a really nasty sync problem I am having?

I appreciate the below is quite lengthy, but the TL;DR is I can't sync a recent backup between Linux and Android over Joplin Cloud and the standard troubleshooting steps don't seem to work.


This all started after upgrading my desktop version (it was something like 2.12.15) to the latest release build on Nov 16th.

Previously I was syncing a Linux installation and an Android device (A) together over Joplin Cloud, but added another Android device (B) on the 16th (both Android devices on v2.12.3) .

I backed up my existing files in a .JEX and updated my Linux client to 2.12.19 before syncing Android Device B. I noticed that B only received a portion of my files, but left it. The next morning, it still wasn't syncing fully.

Since this usually helps, I wiped all clients, reinstalled all of them, and restored my backup on the Linux client (which worked just fine) and resynced, but afterwards both devices A and B wouldn't sync properly at all - the notebook structure from the backup wouldn't show up on them. New notebooks and notes created on any client would appear on any other device, but seemingly nothing from the backup would transfer - it stayed put on the Linux client.

I am confident this is linked with the recent issues regarding emojis (issue 9219 on the Github), as changing the name and emoji on one of the problematic notebooks on the Linux client would allow it to sync to the Android devices, however sub-notebooks and notes would be missing - in other words, the notebook was just a shell. I could only address this by changing the name and emoji of those sub-notebooks and moving the notes to a new dummy folder and back again.

I started using this method to manually restore my notebooks, but after perhaps 20 notebooks, the results became unreliable and some notebooks simply didn't sync across all devices, so I abandoned it.

Oddly, my Android devices would download some of my notes from the backup, but there would often be duplicates and they would only show up in 'All Notes'. Going into the individual notes on Android and looking at the dropdown menu used to swap each note's parent notebook, I found it would list a partial and unstructured list of my notebooks (perhaps 20% of the original total from the backup in a semi-jumbled order.) So clearly it was still downloading some 'phantom' directory structure from the Linux client, but a warped one that wasn't visible elsewhere on the UI.

Since then, I have spent a good amount of time trying all sorts of other methods to address this including:

  • Uninstalling and reinstalling (as mentioned above)
  • Upgrading my desktop and Android clients to the latest prereleases
  • Downgrading all clients to older versions that presumably didn't have this bug
  • Using the advanced options to clear the local clients and download what's in Joplin Cloud
  • Using the advanced options to clear out Joplin Cloud and re-upload the contents of my backup
  • Nuking everything with the Victor plugin and trying to start from scratch
  • Changing master encryption passwords
  • Waiting a night to see if something I don't understand in the cloud is straightening out
  • Restoring the backup from a separate Linux computer and syncing that to check my main computer isn't the problem
  • Signing up for a new Joplin Cloud account (yes, really) and using it to sync

None of this has worked.

I've been banging my head against this for the better part of two days now and I'm at my wit's end. I love this software, I've been using it for four years and I keep most of my life in there, so I'm desperate for a helping hand. Please help me if you know what's behind all of this.


One final semi-related question: when I signed up for a new Joplin Cloud account, I filled in my new syncing details before adding E2EE (I foolishly assumed it was already associated with the backup in some way.) So I only applied encryption midway through a sync once I saw it had been initiated and I had not been asked for my decryption password.

Is the connection with Joplin Cloud at least encrypted with TLS, even if the data is not E2EE? Or did I just send my entire database on the back of a postcard?

1 Like

Thank you for the detailed information!

I'm looking into the issue now. Hopefully it will be fixed soon (edit: see also this forum post)!

Is the connection with Joplin Cloud at least encrypted with TLS, even if the data is not E2EE? Or did I just send my entire database on the back of a postcard?

Joplin Cloud sync should take place over https! (Your data should be safe!)

This is some of the relevant code :slight_smile:

1 Like

It might work if you test again (see other post)!

Joplin Cloud had a bug where each time "Sync" was pressed, it would only synchronize 50-100 items. As such, for a large number of notes/notebooks, it could take many sync attempts for all to be downloaded.

Thanks for your swift reply and work on this, I appreciate it (especially on a weekend.)

I read the post from Laurent yesterday and tried again with mixed results (in short: still no sync to Android.)

First, I wiped my Joplin installation on each device and reimported my backup on desktop. Then I synced my Linux machine, which resulted in it pulling 6 copies of my backup from Joplin Cloud to my machine, so I had seven versions of each notebook at the same time. Usually it will fetch something like 3000 items, when I was trying to fix the issue on Friday and Saturday it was about 12,000, and yesterday it was 50,000 or so items.

I deleted everything, hoping it would purge that from the cloud and spare my Android devices from having to download so many redundant items. Afterwards, I let the Linux client sync the deletions, then reimported my backup and synced both my computer and one of my devices. No dice. The Android client ended up syncing 50k+ items.

Curiously, nothing is now synced to All Notes for Android. The visible part of the UI is totally empty of any notes and notebooks that should have been synced from the backup. If I create a dummy folder and a note inside it and try to move the note in the Android client, however, then this partial, jumbled, phantom folder structure is still visible in the dropdown menu.

Moreover, whenever I press the sync button on Android, it still fetches multiple thousands of items at a time, though what those are and where those are going are a mystery.

Thank you for checking!

I deleted everything, hoping it would purge that from the cloud and spare my Android devices from having to download so many redundant items.

If note history is enabled, old items will still be synced for some number of days after being deleted (defaults to 90 days);

Note history can be disabled by going to settings > note history and unchecking "enable note history".

Moreover, whenever I press the sync button on Android, it still fetches multiple thousands of items at a time, though what those are and where those are going are a mystery.

It's possible that it's still syncing deleted notes. Try decreasing the "note history" limit from 90 days to something like 1 day (or disabling it), then syncing again.

After a complete sync, pressing "Synchronize" shouldn't try to fetch more items.

Note that the "sync status" screen under the "Tools" section on Android may have more information. (It seems to contain less information if a sync is in progress.)

Thank you again for helping debug this!

If turning off note history and doing a full sync doesn't help, I can try to guide you through getting the following information:

  • Whether the missing notebooks are uploaded to Joplin Cloud.
  • Whether Joplin Cloud returns these notebooks when doing a diff (i.e. whether these notebooks are included in the list of things to be downloaded during synchronization).

Thanks for your replies!

I've been trying to keep note history off since I read it somewhere on here/the GitHub repo that doing so would help; when I checked just now it was off on both clients, so I'm unsure that's the issue.

However reinstalling obviously clears that value so I can't say it's been consistently off the whole time and what you said seems to match the number of items being fetched, so I will double check it's disabled on both clients, resync and let you know the result.

Whether the missing notebooks are uploaded to Joplin Cloud.
Whether Joplin Cloud returns these notebooks when doing a diff (i.e. whether these notebooks are included in the list of things to be downloaded during synchronization).

Could you please provide me with this information in the meantime?

Could you please provide me with this information in the meantime?

Yes! It's a bit complicated though :).

The only way I know how to do this is through the developer tools (with a script that requests information from the server).

If you have programming experience, we could try something similar. For context, this an incomplete draft of instructions for how to get this information using Joplin's development tools:

Draft

The end result of this will be similar to the script I posted here.

1. Open Joplin's development tools

To do this,

  1. Open the help menu, then click toggle development tools.
  2. In the window that appears, click on the console tab.

You should see a large amount of output and a textbox at the bottom left.

[[ screenshot ]]

We'll be running commands using that textbox.

:warning:

Beware! Commands run in the development tools have access to your notes and data.

Make sure you understand the commands (or at least trust their source) before running them.

2. Get access to the sync target

Now that the development tools are open, run

var reg = require('@joplin/lib/registry.js').reg;

The file registry.js provides an object (a collection of data and functions) named reg. This stores reg in a variable, which allows us to access it.

Explanation:
  • var: Creates one or more new variables. Writing var reg = means that we're giving the variable the name reg. Everything on the right side of = will be stored in reg.
  • require('@joplin/lib/registry.js'): Gives us all functions/data provided by @joplin/lib/registry.js. One of these is named reg.
  • require('@joplin/lib/registry.js').reg: Gives us the function/variable named reg that was defined in @joplin/lib/registry.js.
Explore similar lines in Joplin's source code

🛈 Tip

After pressing return, you can see a list of some of the things we can do with reg by typing

reg.

in the textbox at the bottom of the console window.

We want to get information about the sync target. Run

var syncTarget = reg.syncTarget();

This gets information about the current sync target and stores it in a new variable, syncTarget.

  • var syncTarget: Makes a new variable named syncTarget
  • =: The equals sign means we're storing a new value in the variable.
  • = reg.syncTarget(): Runs code provided by reg (in a function called syncTarget) and stores it in the variable that's on the left side of =.
Similar lines in Joplin's source code

3. Get the ID of a notebook that isn't syncing

We're going to search for one of the missing notebooks on the sync target. To do this, we first need the notebook's ID.

Right-click on a notebook in the notebook list and click copy external link. Paste the external link into a text editor.

You should see something similar to this,

joplin://x-callback-url/openFolder?id=1c6d2a65e53645cbb5c71b8f7965c4e0

We're interested in the part after id=. In the example above, the ID is 1c6d2a65e53645cbb5c71b8f7965c4e0.

Copy this ID. We're going to store it in a variable in the developer tools.

In the developer tools, type

var notebookIdThatWontSync = 'paste-the-id-here';

Replace 'paste-the-id-here' with the ID of the notebook that won't sync.

Press Return (Enter).

4. Check whether something with that ID is present on the server

Next, we'll use the syncTarget variable from earlier to check whether the notebook is on the server.

We want to look through the files on the sync target. Run

var fileApi = await syncTarget.fileApi();
Explanation

This

  1. var fileApi: Creates a new variable named fileApi
  2. = await syncTarget.fileApi(): Stores the result of syncTarget.fileApi() in fileApi after waiting for it to complete.

syncTarget.fileApi() is configured to run in the background (not return immediately), so we use await to wait for it to return a result.

Now that we have access to the fileApi, we need a list of all files on the server.

To do this, we'll use fileApi.list and a loop.

fileApi.list returns files on the server in groups of around 100. Each group is called a page.

We need to give fileApi.list:

  • The page number we want,
  • Where we want to list files from.

result = fileApi.list(pathHere, optionsHere) gives us:

  • A list of items (result.items)
  • Whether there are more pages (result.hasMore)
var context;

// Check at most 500 pages 
for (var page = 0; page < 500; page ++) {
	console.log('page', page);

	// Get the items for the current page (context is part of the output of the previous
	// call to `list`).
	var list = await fileApi.list('', { page, context });
	
	// .list wants the context from the previous call
	context = list.context;
	
	// Search for the missing notebook on the server
	for (var item of list.items) {
		if (item.path.includes(notebookIdThatWontSync)) {
			console.log("Found the notebook that won't sync!");
			console.log(item);
		}
	}

	// If no more pages, we can stop.
	if (!list.hasMore) {
		break;
	}
}

You can also test with the ID of a note or notebook that does sync, to verify that it can find such a note or notebook.

To-do: Include information about how to use fileApi.delta(''), which also uses pages.


If this looks too complicated, here are some alternative options:

  1. I can try to create a plugin that lists what has been uploaded to Joplin Cloud successfully and the notebooks returned by a diff. (If you're comfortable installing plugins).
  2. The log file on Android contains information about how many notes/notebooks the server marks as new/unsynced. For example,
    • 2023-11-13 20:37:20: Synchronizer: "BasicDelta: Report: {"timestamp":1699935079876,"older":4201,"newer":2,"equal":1}"


      A log (or a sync report) created from Android would contain this information. To export a sync report/log, open "Configuration", scroll down to the "Tools" section, then click "Export sync report".


      While this may be useful, it might still not provide enough information.
1 Like

Whether the missing notebooks are uploaded to Joplin Cloud.
Whether Joplin Cloud returns these notebooks when doing a diff (i.e. whether these notebooks are included in the list of things to be downloaded during synchronization).

Could you please provide me with this information in the meantime?

We're planning on adding a debugging feature (issue) to Joplin Server/Cloud that should make getting this information easier. (Hopefully it will be available some time in the next few days!)

Thank you again for helping us fix this!

It's possible that it's still syncing deleted notes. Try decreasing the "note history" limit from 90 days to something like 1 day (or disabling it), then syncing again.

After a complete sync, pressing "Synchronize" shouldn't try to fetch more items.

Note that the "sync status" screen under the "Tools" section on Android may have more information. (It seems to contain less information if a sync is in progress.)

Following this helped to some degree. I disabled note history on all devices and my notebooks are now syncing successfully from the Linux client to one of the Android devices. (Thank you for getting me this far!)

What I find odd is that I couldn't reproduce this on my second device. It is still stuck in the state I described in my second post here.

Moreover, all devices are still fetching a lot of items during the sync and reporting a lot of revisions in the sync status, despite disabling note history on all of them and notes reporting "This note has no history" when I inspect them individually.

I can try to create a plugin that lists what has been uploaded to Joplin Cloud successfully and the notebooks returned by a diff. (If you're comfortable installing plugins).

I don't have programming experience so I'm concerned I will break more than I will fix using the developer tools, but I'm perfectly comfortable installing plugins.

The log file on Android contains information about how many notes/notebooks the server marks as new/unsynced. For example,

I think maybe some of the below is relevant, but it would probably be best if you could please recommend somewhere I can attach the entire log file itself.

11-21T18:14:50,30,"Synchronizer: ""Sync: createLocal: remote exists but local does not: (Remote 7bb073.md)""
11-21T18:14:50,30,"Synchronizer: ""Sync: fetchingProcessed: Processing fetched item""

11-21T18:14:13,30,"Synchronizer: ""Sync: fetchingTotal: Fetching delta items from sync target""

11-21T18:14:40,30,"""Aborted deletion of old revisions for item "" (rev "485016") because one of the revisions is still encrypted", "Error: One of the revision to be deleted is encrypted
Code: revision_encrypted

I'm unsure why the revisions are still encrypted, since I've entered the master key for my database and it's not prompting me for any further keys.

1 Like

Following this helped to some degree. I disabled note history on all devices and my notebooks are now syncing successfully from the Linux client to one of the Android devices. (Thank you for getting me this far!)

Excellent! That means that the notes and notebooks are on the server (which is what much of the debugging information above would have been for :slight_smile: ).

(The debug report feature on Joplin server, as originally planned, seems similar to a feature that seems to have been removed for causing database lockups on the server).

Moreover, all devices are still fetching a lot of items during the sync and reporting a lot of revisions in the sync status, despite disabling note history on all of them and notes reporting "This note has no history" when I inspect them individually.

I'm unsure why the revisions are still encrypted, since I've entered the master key for my database and it's not prompting me for any further keys.

Here are a few possibilities:

  1. These statements are from before the revisions were decrypted.
  2. There's a bug in how revisions are being deleted!
  3. Many of the revisions are related to resources, which, reportedly, can take several days to be removed after all notes they were linked to have been deleted.

What I find odd is that I couldn't reproduce this on my second device. It is still stuck in the state I described in my second post here.

Another user had success with upgrading to the prerelease version of Joplin:

If you would like to try the prerelease, the Android prerelease can be downloaded from GitHub (or an unofficial version can be downloaded from F-Droid).

According to the Joplin Release Cycle, the current prerelease is in feature freeze and should become stable in a few weeks.

I updated my second Android device to the prerelease and resynced twice, but unfortunately that didn't address the issue. Could you please let me know what the next steps should be from here?

Fortunately the revisions are gone from the two devices that are now properly synced, though I still have a lot of notetags. Can you please help me understand what those are? Maybe it's an obvious question, but I couldn't find a mention of notetags in the documentation or the forums.

I presume they are different from my tags since they are listed separately. I have over 6000 of these notetags for a ~1300 note database when I average probably one tag per note.

I just noticed that when using the search anything function (CTRL+P) on desktop that a duplicate of one of the 'phantom' notebooks is actually still syncing to the Linux client.

For instance, usually my shopping queue is two folders down, but there's also a copy in root. I can navigate to it by clicking it in the search results and I can add notes to it in the Linux client, but it doesn't show up in the notebook overview on the leftmost panel and hence I can't interact with it beyond adding things into it.

This is only the case for notebooks that show up in the drop down list in the improperly synced Android client when I try to move a note (the ones I call 'phantom' notebooks).

Hopefully this information is helpful because it seems quite relevant.

1 Like

@personalizedrefriger Could you please provide an update on the situation? I'm happy to further attempt to debug things on my side.

Please could you clarify what issue you are having? As I understand there's one notebook that appears on one device but not on another?

Is this is the issue, please could you install the "Get Notebook ID" plugin, then right-click on that folder and copy the notebook ID, then paste it here?

As I understand there's one notebook that appears on one device but not on another?

It isn't one notebook, but the entire database - it only syncs correctly between a Linux device and one Android device. The third device, also Android, does not receive any of the database except a few entries in the in-note dropdown menu for changing parent folders of that note. I will PM you some screenshots.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.