Make Synchronizer parallel

wh201906 · 11 November 2022 17:15

The synchronizer is pretty slow when uploading multiple files at the same time. It takes around 1 minute to upload a 10MB file then start the next one, but my network usage reaches the peak for only 2~3 seconds, then drops to nearly zero for the rest of one minute. I think the network bandwidth is not the bottleneck.

github.com

laurent22/joplin/blob/052d9f03d6b7687b1ba5102fef9b9bb79f343c41/packages/lib/Synchronizer.ts#L548


      
          			// last sync and apply the changes to remote.
          			// ========================================================================
          
          
			if (syncSteps.indexOf('update_remote') >= 0) {
          				const donePaths: string[] = [];
          
          
				const completeItemProcessing = (path: string) => {
          					donePaths.push(path);
          				};
          
          
				while (true) {
          					if (this.cancelling()) break;
          
          
					const result = await BaseItem.itemsThatNeedSync(syncTargetId);
          					const locals = result.items;
          
          
					await itemUploader.preUploadItems(result.items.filter((it: any) => result.neverSyncedItemIds.includes(it.id)));
          
          
					for (let i = 0; i < locals.length; i++) {
          						if (this.cancelling()) break;

Is it possible to make the huge loop parallel? let's say uploading 5 or 10 files at the same time.

wh201906 · 11 November 2022 17:31

According to my test, the code between Line 642 and Line 691 takes the most of the time.

github.com

laurent22/joplin/blob/052d9f03d6b7687b1ba5102fef9b9bb79f343c41/packages/lib/Synchronizer.ts#L642-L691


      
          						this.logSyncOperation(action, local, remote, reason);
          
          
						if (local.type_ === BaseModel.TYPE_RESOURCE && (action === 'createRemote' || action === 'updateRemote')) {
          							const localState = await Resource.localState(local.id);
          							if (localState.fetch_status !== Resource.FETCH_STATUS_DONE) {
          								// This condition normally shouldn't happen
          								// because the normal cases are as follow:
          								//
          								// - User creates a resource locally - in that
          								//   case the fetch status is DONE, so we cannot
          								//   end up here.
          								//
          								// - User fetches a new resource metadata, but
          								//   not the blob - in that case fetch status is
          								//   IDLE. However in that case, we cannot end
          								//   up in this place either, because the action
          								//   cannot be createRemote (because the
          								//   resource has not been created locally) or
          								//   updateRemote (because a resouce cannot be
          								//   modified locally unless the blob is present

This file has been truncated. show original

I'm using Joplin v2.9.8(prod.) on Android, syncing to Dropbox

laurent · 11 November 2022 17:48

The preUploadItems call you see there is what's used to upload multiple items in parallel, but currently it's only supported by Joplin Cloud/Server.

wh201906 · 11 November 2022 17:59

Will this be supported in other sync targets in the future?

laurent · 11 November 2022 23:08

If someone can make a pull request, why not, but I'm not sure it can be done with other sync targets, because there might be Joplin-specific behaviour happening on Joplin Server side (I forgot exactly how it's done)

system · 6 November 2023 23:09

This topic was automatically closed 360 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Faster synchronisation in upcoming Joplin version! News	8	893	27 December 2023
[solved ] Deploy joplin on multiple devices (again) Support	2	586	10 July 2021
Linux Sync extremely slow Support	0	654	28 March 2020
Interest in using the sync library for other projects Development	6	453	10 October 2021
Performance of synchronization with WebDAV or with Dropbox Support	12	677	17 May 2022

Make Synchronizer parallel

Related Topics