OneDrive errors in Joplin "The maximum request length supported is 4MB" and "FetchError"

Support
1

I encountered a bizarre bug.
Retrospective:

  1. Synced. I did not have any issues with synchronization.
  2. I added one tag for 3000 notes.
  3. Synced. I did not have any issues with synchronization.
  4. I moved 1500 notes to another notebook.
  5. Synced. I did not have any issues with synchronization.
  6. I moved another 1500 notes to another notebook.
  7. Synced. I have this mistake "The maximum request length supported is 4MB". App not syncing anymore.

I have 480 files in .config\joplin-desktop\resources with size more than 4MB.
The same for the OneDrive folder, many files in .resources exceed 4MB size.

What have I done, and how to fix it?
How can I receive the name of the note from "e5bf60ca91354366a540f9147d466e89.md"?

Windows 10, Joplin 1.2.6

error.txt (2.2 KB)

So far I installed DB Browser for SQLite, opened c:\Users\xxx.config\joplin-desktop\database.sqlite and run
SELECT title FROM notes WHERE id="e5bf60ca91354366a540f9147d466e89"
in "Execute SQL" tab.

The result:
Execution finished without errors. Result: 0 rows returned in 13ms At line 1: SELECT title FROM notes WHERE id="e5bf60ca91354366a540f9147d466e89"

I couldn't locate "e5bf60ca91354366a540f9147d466e89" file in OneDrive\Apps\Joplin either.

Please, help!

2

Normally it should work with resources larger than 4 MB so I thought it would be a note, but you've already checked that. Maybe search for that ID in the other tables - resources, folders, tags, etc.

3

I'm not sure but for OneDrive upload bigger 4 MB you need a upload session.
Can this be the problem?

Use resumable file transfers for files larger than 10 MiB

The document says 10 MB, but it seems to be only 4 MB.
can this be the problem?

4

Thank you for your advice!

I found it in "revisions."
Revision entry contains "item_id," by which I found the note title.
It is a long note. Very long note! I assume that somehow my text.txt file from this note was transformed into plain text during the "import .enex as a Markdown" process.

What should I do now to fix the problem with syncing? If I do something careless with this note without asking @laurent (e.g. delete it, copy all the text back into .txt or .doc file), I can worsen the situation.

And an additional question, but not so important: why this mistake occurred only now, after the whole week of using the app without any issues on Windows and Android?

5

I doubt. I checked and I have 480 attached files bigger than 4MB. The biggest one is 50MB. All of them syncing flawlessly. I think it is something with a note size itself multiplied on history of this note. But it is only a guess of a new user.

6

Hmm, yes that's annoying as there's no easy way to get rid of the revisions. I guess your best option then is to run delete from revisions where item_id = ... and then delete from notes where id=.... Sync state might be a bit inconsistent but it should only concern this note.

7

I deleted revision and note. Now I am experiencing many sync errors of these types:

  • FetchError: request to XXX failed, reason: connect ETIMEDOUT XXX
  • Sync target lock error: syncLockGone. Original error was: network timeout
  • FetchError: On file XXX: request to XXX failed, reason: read ECONNRESET
  • FetchError: network timeout at: XXX

The app can not sync for 4 hours due to the stopping of these errors.

Is it worth to delete the Joplin folder in OneDrive and sync everything from the beginning?

8

I am in the process of fixing this issue for me. I had to delete and resync all data. I will explain and post my results at the end.

9

@laurent, sorry, can not continue without a little advice.

I am battling for the third day. After deleting revisions, I am getting "FetchError," many of them. The app stops syncing after these mistakes.
So now I am exporting .jex, deleting everything, importing, syncing... Long process.

Now, I have another "revision" in the "4MB error" report. As far as I need to delete the second revision, I would appreciate it if you explain to me a little what these revisions are and how they are connected with other items in the database. I need to know how deleting revisions can affect my database.

10

I think you should export everything as RAW. Then look in the generated folder and delete any .md file larger than 2MB (resources can stay). Then delete your Joplin profile, and re-import all this as RAW.

But given your note collection, perhaps you should use something other than OneDrive and its dumb 4MB limit. Dropbox doesn't have this and Nextcloud would work fine too.

11

I checked Dropbox and NextCloud and I prefer to stay with OneDrive as it is free and twice faster.

@laurent, what is your opinion, might it be a bug of Evernote Markdown Importer or Joplin Encryption?

Because it is a very strange, I have a note "Student needs when preparing for IELTS (Results, but only 10 answers)". It is not a very long note. In HTML export its size is 2.28MB, in Markdown 2.26MB.

But if we open this note in Joplin it contains a very long encrypted part. I do not know how this encrypted part formed, it was not my action. So I thought, maybe it is an importer or encryption? Because if so, it might destroy step by step other notes.

Can you take a glance? What is your opinion?

These two I imported from Joplin (nothing criminal inside):

Student needs when preparing for IELTS (Results, but only 10 answers).html (2.3 MB)

Student needs when preparing for IELTS (Results, b.md (2.3 MB)

This one I copy-pasted from Joplin (contains long encrypted block):

STRANGE Student needs when preparing for IELTS (Results, b.md (2.3 MB)

12

I'm afraid no-one can help with that. Encrypted data is larger than non-encrypted one so you'll keep having trouble with this, but it's up to you.

There's no plan for the OneDrive sync target to support notes or revisions larger than 4MB, although if someone wants to make a pull request I'll accept it.

13

I am not sure if I was clear in a previous message. It looks like the note was corrupted by encryption. We do not notice it with other sync targets, but the note is still corrupted. The size of this note should be 0.2MB, not 2.2MB.

It is not an OneDrive problem; it is something with encryption.

IMHO

14

Wow! Didn't know about this problem. As I'm also using Onedrive for syncing, I'm worried one day I'll have to rebuild my database. If I may, I would suggest adding a "Warning" in the Option->Synchronization page about using Onedrive as the Sync target.

1 Like
15

Yes, you are right. I am going to write here a little report about all errors I experienced to warn users of using OneDrive with Joplin. I hope tomorrow I will have enough spare time to do that.

2 Likes
16

I decided to put it here as I understood it might be helpful even though I could not fix the OneDrive synchronization. I think it is impossible to use Joplin with OneDrive in some cases.

So if you are experiencing:

The maximum request length supported is 4MB or

FetchError you have two possibilities to fix it:

  1. Short and reliable. Switch to Dropbox.
  2. Long and unpredictable. Try to find and delete problematic notes.

Switch to Dropbox

I believe that it is impossible to "repair" synchronization with OneDrive since it is not a problem with your local provider, your notes or OneDrive itself, but a problem with the Joplin connector.

Therefore the best solution would be to switch to Dropbox right now and not spend your time on useless attempts to fix OneDrive synchronization.

Try to restore OneDrive sync

In case you want to fix OneDrive synchronization, I want to warn you, nothing helped me, and I doubt it can help anyone, especially in a long-term period.

So the solution for fixing OneDrive is to find a "problematic" entity (e.g. note, tag, notebook), delete it, restore sync, copy the "problematic" entity back with simplification.

First of all, backup your database before you do any actions with it. The best solution is RAW export because it can be useful in other steps.

After creating RAW backup, you should repeat this cycle until you fix synchronization:

  1. Export database . Use JEX export. Close Joplin.
  2. Find errors. Find the error log in c:\Users\username\.config\joplin-desktop\log.txt file. Locate the "id" of conflicting item in it. It would be something like e5bf60ca91354366a540f9147d466e89.md . It might be several errors. Save all conflicting "id" without .md part somewhere.
  3. Find notes' names . Use DB Browser for SQLite to open c:\Users\username\.config\joplin-desktop\database.sqlite . Go to the Browse Data tab. Use "id" you found from step 2 to filtering conflicting items in notes, tags, resources, folders, revisions, etc. You need to find the item's type (e.g. note, tag, revision, etc.) and their actual names/titles. Find all connections with notes, save notes' "id" somewhere.
  4. Delete problematic items . Go to the Execute SQL tab. Use delete from SOURCE where id = "ITEM" command where SOURCE = Table where you found your problematic item and ITEM = problematic item id. Repeat this operation to delete not only all conflicting items but notes related to them. Close DB Browser for SQLite with saving database.
  5. Try to sync without deleting joplin-desktop . Open Joplin and try to sync it with OneDrive now. If everything is fine, you can stop the cycle. If you find another 4MB error, go to step 1. If you find a Fetch error, go to step 6.
  6. Try to sync with deleting joplin-desktop . Usually, after doing something with your database.sqlite in DB Browser for SQLite, you receive many FetchError. To fix it, you should recreate all your folders. Export database as JEX format. Close Joplin and delete c:\Users\username\.config\joplin-desktop\ folder. Go to OneDrive and delete the \Apps\Joplin folder. Open Joplin, do not set up synchronization, import JEX. Set up synchronization, try to sync. If you find any errors, go to step 1. If sync is finished successfully, stop the cycle.

When you fixed your synchronization, use the RAW folder from the beginning to open notes with names from step 3 in some Markdown editor. Create new notes in Joplin and copy the text there in a simplified format.

Warning! All exporting/importing steps can take a lot of time if you have many notes. Do not believe Joplin's "Sync complete" message, press "Syncing" again. And again, and again :slight_smile:

That's all. I think that the Joplin OneDrive connector is completely broken, and even if you are not experiencing problems now, you will find it in the future. But it is my IMHO.

I hope I will never repeat these steps, especially considering that the probability of fixing OneDrive synchronization is close to zero.

1 Like
17

Hi, @laurent .

I was looking at the MS Graph documentation, and it appears that session-based uploads have no real limit:

https://docs.microsoft.com/en-us/graph/api/driveitem-put-content?view=graph-rest-1.0&tabs=http

vs:

https://docs.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0

In fact, individual byte ranges can be up to 60MB.

What is your take on this?

18

Support for large files was added back in June: https://github.com/laurent22/joplin/pull/3195 but for resources not for notes. Pull requests are welcome.

19

@tnwn I just made a PR for this which fixes the problem. Let's hope that @laurent accepts the PR.

20

This is excellent news! I am using Dropbox now, but it would be such a relief for new users. I assume many of us have 1TB of OneDrive with an Office subscription, so many new users will use it for Joplin.

Also, OneDrive has twice bigger speed than Dropbox (at least in my case) and doesn't have restrictions on the number of requests.

You did a good job.