Constantly throttled by OneDrive, cannot sync at all on fresh device

davide · 27 April 2025 09:12

Operating system

Windows

Joplin version

3.2.13

Sync target

OneDrive

What issue do you have?

I am trying to sync on a new PC (from scratch) from OneDrive.
My archive is made up by about 13k items.
For the first time ever (did not happen in the past, I am user of Joplin for 4 years) I am constantly being throttled (as written in the logs).
Since two days I am trying to get the client synced but sync never reaches the end.

In the log I see Joplin sleeps for some minutes after receiving "throttled" message, but apparently it is no longer long enough.
After the pause, OneDrive lets download some more items but, in the end, it returns an HTML page "Sharepoint error".
I tried changing the concurrent connections from 5 to 1, to 20. Nothing helps.

Some thoughts:

Maybe sync should pause for longer when detecting a "throttled error"? Is there any way to set this pause time?
Maybe sync should make requests slower?
Maybe sync should continue since where it was left with the problem instead of restarting from the beginnning?

So now I am stuck in a endless loop, cannot sync any new device.

I know exporting/importing is not a solution because it leads to dupllcated notes.
Maybe I can copy the whole Joplin folder (containing the sqllite db) on the new device? Could it work or has some identifiers of the device and it would do a mess?

Thank you in advance for your help!

Steph · 27 April 2025 10:17

Is it helping ?

davide · 27 April 2025 10:21

Thank you for your answer.
I already started sync from a clean state by uninstalling Joplin completely and removing the data folder. Unfortunately it does not help.
Indeed, it looks like the problem is strictly on the sync process and its timings.

mrjo118 · 27 April 2025 17:51

Looking at the code, it suggests the amount of time to pause is taken from the amount of time instructed to wait by the api itself, so I wouldn't think there is anything that can be done better in that regard

github.com/laurent22/joplin

packages/lib/onedrive-api.ts

ffd775a8f


      
          
          					// Error: ETag does not match current item's value
          					// Code: resourceModified
          					// Header: {"_headers":{"cache-control":["private"],"transfer-encoding":["chunked"],"content-type":["application/json"],"request-id":["d...ea47"],"client-request-id":["d99...ea47"],"x-ms-ags-diagnostic":["{\"ServerInfo\":{\"DataCenter\":\"North Europe\",\"Slice\":\"SliceA\",\"Ring\":\"2\",\"ScaleUnit\":\"000\",\"Host\":\"AGSFE_IN_13\",\"ADSiteName\":\"DUB\"}}"],"duration":["96.9464"],"date":[],"connection":["close"]}}
          					// Request: PATCH https://graph.microsoft.com/v1.0/drive/root:/Apps/JoplinDev/f56c5601fee94b8085524513bf3e352f.md null "{\"fileSystemInfo\":{\"lastModifiedDateTime\":\"....\"}}" {"headers":{"Content-Type":"application/json","Authorization":"bearer ...
          
          					await handleRequestRepeat(error);
          					continue;
          				} else if (error?.code === 'activityLimitReached') {
          					// Wait for OneDrive throttling
          					// Relavent Microsoft Docs: https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#best-practices-to-handle-throttling
          					// Decrement retry count as multiple sync threads will cause repeated throttling errors - this will wait until throttling is resolved to continue, preventing a hard stop on the sync
          					i--;
          
          					const retryAfter = response.headers?.get?.('retry-after') ?? '1';
          					let sleepSeconds = parseFloat(retryAfter);
          
          					if (isNaN(sleepSeconds)) {
          						sleepSeconds = 5;
          					}

I'd suggest setting the value to 1 and then waiting a few hours before you attempt to sync again, because the throttling may take some time reset itself. Other than that I don't think there is anything you can do to avoid the throttling when syncing a new device from scratch, other than to use an alternate sync target to OneDrive, if your note collection is getting too big to work well with it.

If both old and new client are both Joplin desktop clients, there should be no issue to copy this to a new pc, so long as you don't merge the folder with an existing Joplin folder on the new pc, but delete any existing folder before you put the new one in its place.

LeoW · 27 April 2025 22:27

Hi,

I think what the software needs is the ability to use a specific client ID and Secret when connecting to the Google Drive. Using the Google Drive API through the secret and client ID will improve performance dramatically.

I use an open source app to mount my Google Drive with a client ID and Secret so the software can use the Google Drive API and it doubles the performance.

Joplin's OneDrive connector is not providing the option to enter the client ID and Secret.

mrjo118 · 27 April 2025 22:57

I think Joplin already does use client id and secret to call the OneDrive api. It will obtain and store them when you login using oauth:

github.com/laurent22/joplin

packages/lib/onedrive-api.ts

ffd775a8f


      
          import { _ } from './locale';
          
          const { stringify } = require('query-string');
          const urlUtils = require('./urlUtils.js');
          const Buffer = require('buffer').Buffer;
          
          const logger = Logger.create('OneDriveApi');
          
          export default class OneDriveApi {
          
          	private clientId_: string;
          	private clientSecret_: string;
          	// eslint-disable-next-line @typescript-eslint/no-explicit-any -- Old code before rule was applied
          	private auth_: any = null;
          	// eslint-disable-next-line @typescript-eslint/no-explicit-any -- Old code before rule was applied
          	private accountProperties_: any = null;
          	private isPublic_: boolean;
          	// eslint-disable-next-line @typescript-eslint/no-explicit-any -- Old code before rule was applied
          	private listeners_: Record<string, any>;
          
          	// `isPublic` is to tell OneDrive whether the application is a "public" one (Mobile and desktop

LeoW · 27 April 2025 23:10

Hi,

We would all have needed to launch the azure site and Link it to a new client ID and Secret that you put into the app only there is no place to enter this information when setting up OneDrive in the software. Perhaps there's a way to manually add it by editing a file I don't know. The software is using a shared client ID and Secret with perhaps thousands of other users which is why you want to have your own personal unique client ID and Secret which is not in the documentation and I don't think it's implemented.

Please, somebody correct me if I'm wrong.

The reason that I'm suspecting that this is a requirement is that I use another app called rclone which can either use the shared client ID or have its own client ID and the speed doubles when you set up your own client ID.

For example when you install Joplin and set up the OneDrive sync is that process offer you the opportunity to enter your own client ID and Secret and did you create an ID and secret?

The only way to create your own client ID in secret is to use Microsoft Azure Which requires that you log in with a Microsoft account And Give them your information to set up an account And then you can create your own ID and secret.

LeoW · 27 April 2025 23:21

UPDATE

I asked one of the AI search engines about using your own client ID and Secret and it said that you can when you set up your OneDrive synchronization there is an option to click on advanced settings and that's where you put in the client ID and Secret. If you've already created your OneDrive sync like I have there might be a way to add it in afterwards only I don't know where it would go. Probably need to back up data and recreate the synchronization for OneDrive and choose Advanced options.

Now you’re using your own OneDrive API credentials instead of Joplin’s default ones! This is useful for higher reliability and avoiding rate limits!!!

mrjo118 · 27 April 2025 23:30

There's definitely not an option to do that within the Joplin UI. Is that referring to the OneDrive oauth screen that you get a link to in order to authenticate? I just tried this and could not find any such option for advanced options throughout the login journey.

Are you sure this is possible with a free Microsoft account? It sounds like something which you might need a Microsoft Azure subscription for

LeoW · 27 April 2025 23:40

I'm not 100% sure where in Joplin you can add the information but I'm going to investigate this further.

You have me curious about boosting the performance even though personally I don't really need it because I don't upload and download amounts of data that would really make a difference.

Approximately how much data do you move up and down to the OneDrive on a regular basis? That would determine whether or not you're getting full results because of the shared default client ID.

YES. I needed to create a Microsoft Azure account and give it all my information including a credit card which was not charged anything because I'm not using anything in Azure that cost anything and they give you $200 in credit to start with anyway. It is totally free but you do have to create an account and set up a phone number and credit card and then you have the option to create your own client ID and Secret.

Like I said I got the instructions from AI so they may not be correct. I did find an option to edit the json file and manually add the client ID and Secret but I haven't tried it yet.

I'm going to post a question in the Joplin forum about setting your own client ID and Secret when configuring the OneDrive synchronization and see if one of the support Personnel or Developers sees it and has an answer.

I'm going to continue to research this with the internet and other search engines.

Now that I know it might be possible I want to do it because I do have the client ID and secret and azure setup for myself.

mrjo118 · 28 April 2025 00:05

I dug a bit further and found the client id and secret is hardcoded in the source code here:

github.com/laurent22/joplin

packages/lib/parameters.js

ffd775a8f


      
          		secret: 'qabchuPYL7931$ePDEQ3~_$',
          	},
          	dropbox: {
          		id: 'cx9li9ur8taq1z7',
          		secret: 'i8f9a1mvx3bijrt',
          	},
          };
          
          parameters_.prod = {
          	oneDrive: {
          		id: 'e09fc0de-c958-424f-83a2-e56a721d331b',
          		secret: 'JA3cwsqSGHFtjMwd5XoF5L5',
          	},
          	oneDriveDemo: {
          		id: '606fd4d7-4dfb-4310-b8b7-a47d96aa22b6',
          		secret: 'qabchuPYL7931$ePDEQ3~_$',
          	},
          	dropbox: {
          		id: 'm044w3cvmxhzvop',
          		secret: 'r298deqisz0od56',
          	},

There doesn't look to be any way to change them without modifying the source code.

LeoW · 28 April 2025 00:10

That is why I decided that I'm going to post this specific question to The Forum and see if I get an answer that we can both use.

Thank you for checking the source code and finding this information it's helpful as a start.

That has to be the default Joplin ID and secret because I have my own that I want to put in there from my Azure account.

bwat47 · 28 April 2025 15:00

Have you considered trying the s3 sync target? It might have better performance than onedrive, and there wouldn't be any kind of shared client ID or throttling

I've been using it with Cloudflare R2 (which has a very generous free tier, you only need to pay if you exceed 10gb storage/1 million class A and/or 10 million class B operations per month)

LeoW · 28 April 2025 15:20

Hi,

Personally, I don't do a lot of uploading and downloading and the throttling doesn't bother me at all or affect my work. I would just like to help someone who is having trouble with throttling. I would like to have it available for me as well if it's possible I already pay for my OneDrive storage so I don't need to open a new account with an S3 provider.

Topic		Replies	Views
Synchronization not working Support	2	614	1 October 2022
Sync problem Support	3	117	27 November 2024
Sync does not finish and cannot cancel Support	14	2691	6 July 2023
Syncing Issues Support	6	747	22 February 2024
Joplin is trying to fetch too many items while syncing (sync seems broken) Support	6	1222	8 February 2021

Constantly throttled by OneDrive, cannot sync at all on fresh device

Operating system

Joplin version

Sync target

What issue do you have?

Related topics