I'd be kind of cautious with having Syncthing move binary files such as databases that are open and in process of getting updates with other computers. You are bound to hit corruption issues.
If they are closed and sync'd, that might work. But I wouldn't do that with my data.
In the sync options, there is a File System type. This type might have limitations of attachments or similar, I can't remember the specifics. When I migrated in to Joplin, I was using File System sync type and using Syncthing to move and keep those in sync. I had to ctrl-S at start and end of my sessions to ensure I wouldn't have sync conflicts between my different Joplins. I grew weary of sync conflicts, and I had to have 2 effective copies of my data on each machine/device.
I'm currently using webDAV from my email provider.
You should get this sync functionality all natively through Joplin Cloud though. Assuming your profiles are proper and linked.
What do you mean by this? There likely will be issues if you have added the files directly to the sync target folder (in this case where Synthing is syncing), even if the files were originally created by Joplin by another profile.
Ok, so you're saying with your 'JoplinTestSync' profile the sync works fine, but when using your actual imported data you still get sync issues?
And that happens even when you disable all plugins, disable ocr, set attachment download behaviour to auto and don't open any existing notes before doing your sync tests? It would be easier to set behaviour to manual, so nothing will download if a note is auto focused. Could you also try not uploading any attachment as part of your sync test?
I'd be kind of cautious with having Syncthing move binary files such as databases that are open and in process of getting updates with other computers. You are bound to hit corruption issues.
I don't. I am not using Syncthing on the folder where Joplin stores its database and attachments. I am having problems with items not syncing from one PC to another. Originally, I was using Joplin Cloud. The problem with that is that its a 'black box'. Data goes in and comes out, but I have no way to know what's inside. I tried Webdav on a cloud storage subscription I have (Koofr). I have thousands of notes, so syncing took a long time, especially trying to sync three different PCs in sequence.
I am not trying to keep the '.config' folders in sync to each. I inspected the folders and the SQLlite DB and found they are different on each PC so that would be disaster.
Since all instances of Joplin look at a single sync target, I guessed that if I have 3 sync targets that are all identical, that would be equivalent. So, I created a local sync target on each PC and connected each instance of Joplin to that local sync target folder. I use Syncthing to keep the three local sync targets synchronized.
Testing confirmed that this works without a problem. Plus, since I can see the contents of the sync target, when a sync issue occurs, I can see where the problem is occurring. If you read some of the earlier messages, the sync problems are sometimes when the non-originating instances of Joplin don't download the note from the sync target. Other times are when the originating instance of Joplin don't write the updates to the sync target.
Using the Joplin Cloud 'black box' would have prevented me from seeing where the problem occurred. Using Webdav on a cloud provider would have added days to my testing. Using Syncthing adds another layer to the sync targets so that may cause problems down the road, too. For now though, it allows lightning fast syncing in Joplin. Since all three PCs are currently on my local lan, Syncthing operates fast, too.
I'm not sure that's absolutely correct. To begin the test, I restore a '.jex' file to PC1. To make sure everything restored right, I open a couple of notes to make sure they look right and automatically download the attachments. I honestly don't remember how I got the data to the other PCs. With all plugins and OCR disabled is when I manually created a notebook, added a note with an attachment to it. That's when the sync problems I described above occurred.
Since I have my original '.jex.' file, I can test using any methodology we want.
Here's is the procedure that I think will accomplish what you are asking me to do:
Pause Syncthing.
Clear all three of the local sync targets.
Create new profiles on each PC:
De-install any plugins, other than the two default ones.
Turn off OCR on each instance.
Set the sync target on each PC to the local sync target folder.
Change the 'Attachment download behaviour' to manual for each PC.
Restore my existing '.jex'. file to PC1.
Un-pause Syncthing.
Let the other two PCs sync.
Create a new notebook in one of the PCs.
Create a new note in the new notebook.
See if the new items propagate to the other PCs.
If they don't, determine where the failure occurred, either on the originating instance or target instances.
If I restore the '.jex' file this way, I am assuming that the original attachments will not be in the sync targets. I can confirm that easily enough.
Before I start this test, I would like to hear from you to confirm this is way you want me to proceed. I have some errands today, so I'll be in and out. I appreciate the time you're spending with me on this.
Yes the procedure is correct for what I had in mind. The intention here is to determine whether what is causing the sync to get messed up relates to attachments.
I'd also like to add that if there are still sync issues when following that procedure, is it just notes created on PC 1 which do not get synced to other clients, or does the issue also occur from P2 > PC1 or PC2 > PC3
I'll perform the same test using PC2 and PC3 as the originating instances. For each test, I'll create notebooks and notes on each PC. I'll report back to see how we proceed after that.
I started the test and got to step 4 on PC1. Restoring the '.jex' file restores all of the attachments, regardless of the setting of 'Attachment download behavior'. That means that the attachments will be on PC1 and will also propagate to the local sync targets on PC2 and PC3. Since those PCs attachment setting is still set to manual, they won't download the attachment.
One solution to get all the attachments off Joplin on all three PCs would be to delete the profile on PC1 after it was synced to the local sync target and recreate the profile with the attachment setting on manual and allow the PC1 to get the data from the sync target.
Would that still answer your questions about whether or not an attachment was causing the problem?
That's the expected behaviour and is part of the test. The additional point I made about making a change on PC2 > PC3 will be a test where both clients do not have any attachments downloaded locally from PC1
I left the attachments on PC1. I created a new notebook on PC2. I checked the PC2 local sync target. The notebook record was missing and the note record was there. PC1 had the test record, but it wasn't in the notebook because the record wasn't in the sync target. PC3 had the note record in its sync target, but the note did not sync to Joplin.
I created a new notebook and note on PC3. Like PC2, the note record was in the sync target, but the notebook record wasn't there. PC1 had the note record, but not the notebook record. PC2 didn't have the note record or the notebook record.
I created a notebook and note on PC1. Both the notebook and note records appeared on the local sync target of all 3 PCs. Neither the notebook nor the note appeared in Joplin on PC2 and PC3.
That's correct. I exported the test notebooks containing the test notes. I think the next step would be to delete my notebooks, one at time, including emptying the trash. Then, import each test notebook on each PC and check for synchronization. I am hoping that I can narrow down which note/group of records is affecting the sync process.
Can you confirm that if I delete a record, including the trash, it is totally cleared from the database? I can test that, too. Is there a point to deleting all the attachments from PC1 first to test? I should be able to do that, by either creating a new profile with the attachments option set to manual or deleting and reloading from the sync target.
Can you confirm that if I delete a record, including the trash, it is totally cleared from the database
No it wont be. But what you can do is to do a jex export after you have permanently deleted some notebooks and their contents, then import that jex into a new profile
I am hoping that I can narrow down which note/group of records is affecting the sync process.
I'd recommend to do half slicing. Eg. Make a profile with the first half of notebooks, then another profile with the other half of notebooks. Whichever half continues to have the issue, you split that group of notebooks in half and repeat, until you find a specific notebook causing the issue. Then when you have 1 notebook, do half slicing for the notes within it
I like this method. It will be less time consuming than doing this for 79 notebooks and about 15000 notes.
While I was sitting around, I thought maybe I could export the notebook/note in a different format that might eliminate whatever is causing the problem. I exported the notes to 'MD Markdown + Frontmatter', created a new profile and imported the notes back.
PC2 and PC3 are currently syncing. I'll test and report back. If that doesn't work, I'll try the 'half a loaf' method you suggested.
So...Are the developers interested in what is causing the problem? Is it be something that they could overcome programmatically? That would make the sync process more robust and save someone else from this issue.
The developers are always interested in bug fixes. It just depends whether we can actually find a root cause and reproduction steps, otherwise it's pretty much impossible to fix. If we do, then I might fix it myself if it's just a small change required
So...Testing exporting as 'MD - Markdown + Frontmatter' was successful. I exported the records and re-imported the records (Directory option) into a new profile. While the import placed all notebooks under a notebook named after folder where the export was placed, it was trivial to move all of the notebooks out of it.
I synced the three sync targets using Syncthing.
Created new profiles on PC2 and PC3.
Synced PC2 and PC3 from local sync targets with attachment options set to 'Manual'.
Imported the test notebooks '.jex' file into each PC (actually, I couldn't find one of the '.jex' file so I manually created it again).
All three test notebooks and notes propagated to all three PCs.
I changed the attachment options from 'Manual' to 'Always' and synced the PCs again.
I created new notebooks/notes on each PC again.
All three test notebooks/notes propagated to all three PCs again.
I added a '.pdf' attachment to each test note. I used a different PC than the one that created the note. All three updates propagated.
Whatever issue was preventing syncing successfully carried into a '.jex' file, but did note make it into (or out of) the 'Markdown + Frontmatter' export/import. All three PCs are in sync.
So...Are there any properties of notes or notebooks that do not export or import to Markdown exports?
I plan to do the half export process with the '.jex'. exports to try to identify the issue with it.
Do you use markdown links to other notes? If you do, some or all of the links could potentially be broken from the Markdown + frontmatter export and reimport. Just raising that as a disclaimer, as that is something that was reported by someone else, though I could not reproduce this. Also note / notebook names over 50 characters will have been truncated.
But it's good that you're able to confirm it is definately a data issue, and have a way forward to actually use Joplin properly now. Would you mind continuing tests with the original jex data in test profiles using the half slice method, to continue investigation for the root cause? This would definately be something worth fixing, given that there doesn't seem to be an errors raised by the sync when the profile is in this state.
Also, which application was the your note data imported from originally?
I don't think I have any note titles or notebook names over 50 characters so that shouldn't be a problem. Not sure what markdown links in notes are, but if it means linking one note to another, I don't think I have any, but if I do, I should be able to recreate them.
I fully plan to continue to test using the half slice method. I'll need to get my actual database in order first. I'll post when I have something to report.
Thanks for all your help. I know a lot more about Joplin's data structure than I did before.