Contents discrepancy among machines

FernandoCabral · 2019-09-11T17:25:06.701Z

I keep Joplin in three different Linux machines and an Android. After some trepidation, it seems everything is working. Nevertheless, file sizes are different even thou they seem to have the same contents.

Those two databases should contain the same data and so it seems when visited, searched, etc. In spite of that, file sizes are quite different. (Yes, they are synchronized).

Maybe there is some leftover that could be eliminated?
Different note versions could be kept in different machines and not synchronized?

laurent · 2019-09-11T22:21:36.418Z

You can run the VACUUM command on both databases to see if it makes a difference.

FernandoCabral · 2019-09-11T23:09:07.993Z

I did. Both files had their sizes reduced a little. Nevertheless, they still have different sizes even thou I have repeated several times the compacting the synchronizing procedures:

22937600 set 11 19:56 database.sqlite
29569024 set 11 19:58 database.sqlite

Both shrunk, but still there is a 7 MB difference between them.

laurent · 2019-09-12T08:07:23.000Z

You’d need to look at the actual data to see where the difference comes from.

tessus · 2019-09-12T12:43:12.671Z

Differences are possible.

One example: you create a note and before it was synced you delete it. So there will be entries in the delete_items table in your local database, but not in the others (Afaik).

FernandoCabral · 2019-09-12T12:56:17.935Z

tessus:

One example: you create a note and before it was synced you delete it. So there will be entries in the delete_items table in your local database, but not in the others (Afaik).

I understand that. Nevertheless, I think it should not happen after you have run VACUUM on both copies, and resync'd. Especially if the sequence was repeated three or four times, as I did.

As to your example, for instance (delete_items table), I know it is empty because I have exported all the tables and I went directly to this one to see if it still retained any leftovers.

I did the same with some other tables, but not with all of them. I guess in the end I'll be able to figure out why the difference is there.

tessus · 2019-09-12T13:52:51.526Z

FernandoCabral:

Nevertheless, I think it should not happen after you have run VACUUM on both copies, and resync’d

Vacuum does not empty tables. So, if there are records in a table in one database and no records in the same table on another database, the size will still differ after a vacuum.

But in your case, if these tables are empty, the difference must be somewhere else.
My example was just that. There are other cases as well.

laurent · 2019-09-12T14:23:25.095Z

I’d expect the SQLite filesize is not directly related to its data. It might depend how and in which order the data has been added to the database. Perhaps after vacuum there are still holes in the file.

But anyway, if the data otherwise is valid it’s not a Joplin issue, but an SQLite one, and it’s most likely not a bug.

laurent · 2019-09-12T14:24:38.448Z

And you keep looking at the filesize but you still haven’t told us if the data is identical or not, which you can check for example in the Status screen of the apps.

FernandoCabral · 2019-09-12T15:16:48.746Z

laurent:

I’d expect the SQLite filesize is not directly related to its data. It might depend how and in which order the data has been added to the database. Perhaps after vacuum there are still holes in the file.

As far as I can tell, the data are the same. One machine is sync'ed with the other. Perusing notebooks and notes, they look the same.

Nevertheless, when I first started using Joplin I had some issues synchronizing among machines. This was especially true when I was synchronizing between Linux and Android. Lately I have had no issues in this front.

Now that you referred to the Status screen, is there a way for me to clear the log and status files and start anew? This would allow me to check if there are new error messages (Those I see are not time stamped, but seem old).

J-Metsis · 2019-09-18T07:34:20.401Z

Laurent is right here. Databases experience different physical sizes on different platforms (file system, disk allocation size etc.) and even on identical platforms because of various reasons (database growth settings, order in which data was stored etc.). Actually it is just by chance two databases with identical content would have exactly same physical sizes. By physical size is understood the sizes you see at the file system level.

A different thing is the logical size of a database. In order to function efficiently a database system reserves disk space in advance and formats it suitable for the database's use. A healthy database should always contain such unused space.

To repeat what Laurent said

But anyway, if the data otherwise is valid it’s not a Joplin issue, but an SQLite one, and it’s most likely not a bug.

It is a database issue and it's not a bug.

FernandoCabral · 2019-09-18T09:10:11.990Z

J-Metsis:

A different thing is the logical size of a database. In order to function efficiently a database system reserves disk space in advance and formats it suitable for the database’s use. A healthy database should always contain such unused space.

I can understand that. Totally. Nevertheless, it still seems to me that if I am using the same OS, same data, loading the same data in the same sequence, and doing the same operation (like compacting both databases with the same tools) I should get the same size.Those are not random operations that could lead to different organization.

Anyway, size differences alone would not worry me, but things like not synchronizing data, yes. This is the issue I am after. Looking for an explanation I noticed a persistent file size discrepancy. Understanding why this happens could help me to drop one more suspect of interference with data synchronization.

In this moment I still have three files on the resource directory that do not get synchronized no matter what I do. The log file says that those three files can not be found. Nevertheless, they are there, with the same name and path that the log says do not exist.

By the way, those files belong to a note I have deleted. The note does not exist anymore. Still, for some reason, Joplin tries do synchronize them (which it shouldn't) and complains they are not there (which it shouldn't either, because the files are there).

Those are the things that mystify me.

J-Metsis · 2019-09-18T09:58:49.147Z

By the way, those files belong to a note I have deleted. The note does not exist anymore. Still, for some reason, Joplin tries do synchronize them (which it shouldn’t) and complains they are not there (which it shouldn’t either, because the files are there).

Ok, this sounds suspicious, as if some status marking would be off.

laurent · 2019-09-18T14:18:04.402Z

I think we’re finally getting to the actual problem. However to be able to help we need much more precise info.

For example:

What client is in sync
Which one is not
What’s on the sync target (are the missing resources in there?)
What does the status screen of client 1 say
What does the status screen of client 2 say
Any error in any of the logs when you sync?

FernandoCabral · 2019-09-18T15:15:04.202Z

laurent:

I think we’re finally getting to the actual problem. However to be able to help we need much more precise info.

I will be able to provide those, but to be sure, I need some background information. Specifically, when I export and then import the whole database in a .jex file, does Joplin create a clean, brand new database, or does it export and import again things like deleted notes, or notebooks and link?

It is important to know this because I have been trying to export the whole thing, install Joplin again from scratch and then import the whole thing again. I still have two problems after doing this. One has to do with a particular file whose updated version is never copied after synching; the other has to do with a deleted note whose attached files do not get deleted. But, even thou they are still there (and they shouldn't), Joplin complains the files do not exist.

So I think if the start from a clean database I might be able to reproduce the problem.

laurent · 2019-09-18T16:27:23.850Z

We need either a JEX file that allows replicating the issue, or information like I mentioned above. Anything else is just a guessing game.

FernandoCabral · 2019-09-18T17:37:51.675Z

laurent:

either

I understand. That's why I am asking for that information. I wanna try to produce a file that has only the minimum amount of information needed to nail the problem down. If I send you what I have now, different issues might influence the result since I have already compressed the databases; I have copied files back and forth and I have made other attempts that may confuse things more.

FernandoCabral · 2019-09-19T22:18:05.163Z

Laurent, I’ll try to do this in stages. I have just isolated one machine and re-installed Joplin from scratch. In order to do that, I first deleted the the ~/.config/joplin-desktop directory and everything inside. Then a ran Joplin and it recreated the initial files. After that, I deleted the notebook “Welcome”.

Well, three files are still present in the directory ~/.config/joplin-desktop/resources. Those files are 2ec84055d2ea40c991643d5aeb62e6c1.png, 934f4e7f5241452a8e55cf023178e39b.png and d92de2ad3ba24a9c9645aa2f0e35156f.png.

There contents are as bellow.

2ec84055d2ea40c991643d5aeb62e6c1.png1075×641 63.1 KB

FernandoCabral · 2019-09-19T23:23:31.235Z

Ok: I've exported the database and installed Joplin in both machines from scratch. Then, I imported back the exported file in one of the machines. Next made the second machine to synchronize with the first. It did alright. After that I changed a file on the first machine and saved it. Alas, it did not synchronize.

Here is the file that was copied from the first to the second machine when this last one was first synchronized:

rw-rw-r-- 1 fernando fernando 773983 set 19 19:48 0d2ac266f98f416d9395f4a02b350c90.odt

And here is the file changed in the first machine, mas never brought to the second machine:

-rw-rw-r-- 1 fernando fernando 773715 set 19 20:12 0d2ac266f98f416d9395f4a02b350c90.odt

There are no errors registered on the log files on either machine.