Final report of the Native Encryption project

GSoC Projects Native Encryption
1

Here is the final report of the new native encryption project of GSoC 2024. The code has not been merged yet but I guess it will be merged soon. I will keep improving the related code before and after the PR is merged.

Special thanks to @tessus and @roman_r_m for mentoring me throughout this project!
I would also like to thank @laurent and @personalizedrefriger for their support during the GSoC period.

GSoC Project Link: Google Summer of Code
GitHub PR Link: All: Add new encryption methods based on native crypto libraries by wh201906 · Pull Request #10696 · laurent22/joplin · GitHub

What was done

1. Evaluate the crypto libraries and the new encryption algorithms

Reports:

2. Migrate to better encryption algorithms and faster/native crypto libraries.

  • Utilized node:crypto for the desktop/CLI clients and react-native-quick-crypto for the mobile clients.

  • Developed new encryption methods in EncryptionService.ts based on the selected crypto libraries.

    • Key Derivation Algorithm: PBKDF2-HMAC-SHA512, 256 bits salt length, 220000 iterations for master key, 3 iterations for content encryption key
    • Cipher Algorithm: AES-256-GCM, 128 bits authentication tag length, 96 bits IV
    • The new encryption methods are expected to be more secure than the old ones.
    • Comparison of old and new encryption methods:
      • Cipher: AES-128-CCM -> AES-256-GCM
      • Authentication Tag Length: 64 bits -> 128 bits
      • Key Derivation Algorithm: PBKDF2-HMAC-SHA256 -> PBKDF2-HMAC-SHA512
      • Key Derivation Iteration Count for Master Key: 10000 -> 220000
      • Key Derivation Salt Length: 64 bits -> 256 bits

  • Designed a new nonce generation process

    • AES-GCM is known to be vulnerable if the key-IV pair is reused.
    • The content encryption key is derivated from master key and a 256 bits hashed salt with PBKDF2, so the equivalent nonce space is 256 + 96 = 352 bits, which decreases the possibility of reused key-IV pair.
    • The salt used in the content encryption key is based on random bytes (21 bytes), timestamp in milliseconds (7 bytes), and a counter (8 bytes), which decreases the possibility of collision.
    • The salt used in the content encryption key is hashed with SHA-256 before using, to make it unpredictable and prevent leakage of the metadata above.
    • The content encryption key is derivated with PBKDF2 rather than HKDF because HKDF is not available in react-native-quick-crypto, and the PBKDF2 with small iteration count also has a good performance.

  • (Performance test results)

3. Enhance tests

  • Applied some existing tests to different encryption methods.
  • Added tests for the ciphertext integrity check.
  • Added integration tests for the encryption compatibility across platform-specific implementations.
  • Added performance tests for the new/old encryption methods.

4. Reduce the size overhead of encryption

  • Extract the binary data from hex encoded string (for master key) or base64 encoded string (for files) before encryption.
  • Reduced encrypted master key size from ~860 bytes to ~460 bytes.
  • Reduced size overhead of encrypted files from ~77.7% to ~33.3%.

Performance test results

Based on commit edd2a9002a31b0fee4f11ded0a37951543e289ab, release build

Windows desktop client
Data Type Encryption Method Test Count Data Size Total Encryption Time (ms) Total Decryption Time (ms) Avg. Encryption Time per Test (ms) Encryption Speed Ratio (New/Old) Avg. Decryption Time per Test (ms) Decryption Speed Ratio (New/Old)
string StringV1 2400 100 1,943.00 10,865.40 0.81 42.95% 4.53 82.52%
string SJCL1a 5700 100 1,982.00 21,294.20 0.35 100.00% 3.74 100.00%
string StringV1 1700 10,000 1,875.30 8,330.60 1.10 118.45% 4.90 120.03%
string SJCL1a 1300 10,000 1,698.70 7,646.30 1.31 100.00% 5.88 100.00%
string StringV1 70 1,000,000 1,939.10 2,285.90 27.70 488.04% 32.66 287.71%
string SJCL1a 15 1,000,000 2,027.90 1,409.30 135.19 100.00% 93.95 100.00%
string StringV1 18 5,000,000 1,848.40 2,001.30 102.69 666.25% 111.18 381.15%
string SJCL1a 5 5,000,000 3,420.80 2,118.90 684.16 100.00% 423.78 100.00%
file FileV1 1000 100 2,110.30 1,323.90 2.11 79.72% 1.32 83.08%
file SJCL1a 1500 100 2,523.60 1,649.90 1.68 100.00% 1.10 100.00%
file FileV1 1000 10,000 3,242.10 1,428.70 3.24 119.25% 1.43 199.53%
file SJCL1a 400 10,000 1,546.50 1,140.30 3.87 100.00% 2.85 100.00%
file FileV1 80 1,000,000 1,566.50 1,282.50 19.58 1,522.07% 16.03 1,486.97%
file SJCL1a 5 1,000,000 1,490.20 1,191.90 298.04 100.00% 238.38 100.00%
file FileV1 20 5,000,000 2,003.40 1,500.50 100.17 1,688.33% 75.03 1,690.33%
file SJCL1a 3 5,000,000 5,073.60 3,804.50 1,691.20 100.00% 1,268.17 100.00%

Windows_desktop_client

Android emulator, API 26
Data Type Encryption Method Test Count Data Size Total Encryption Time (ms) Total Decryption Time (ms) Avg. Encryption Time per Test (ms) Encryption Speed Ratio (New/Old) Avg. Decryption Time per Test (ms) Decryption Speed Ratio (New/Old)
string StringV1 120 100 3,640.59 5,016.35 30.34 95.37% 41.80 103.77%
string SJCL1a 120 100 3,472.12 5,205.33 28.93 100.00% 43.38 100.00%
string StringV1 80 10,000 2,628.96 4,325.86 32.86 189.67% 54.07 156.10%
string SJCL1a 40 10,000 2,493.20 3,376.42 62.33 100.00% 84.41 100.00%
string StringV1 3 1,000,000 3,701.93 3,677.24 1,233.98 320.46% 1,225.75 351.71%
string SJCL1a 3 1,000,000 11,863.24 12,933.20 3,954.41 100.00% 4,311.07 100.00%
file FileV1 120 100 2,147.60 2,182.68 17.90 98.94% 18.19 99.55%
file SJCL1a 120 100 2,124.85 2,172.92 17.71 100.00% 18.11 100.00%
file FileV1 70 10,000 3,166.09 2,271.46 45.23 118.52% 32.45 207.49%
file SJCL1a 40 10,000 2,144.24 2,693.20 53.61 100.00% 67.33 100.00%
file FileV1 3 1,000,000 2,385.42 3,408.14 795.14 456.93% 1,136.05 332.45%
file SJCL1a 3 1,000,000 10,899.81 11,330.19 3,633.27 100.00% 3,776.73 100.00%

Android_emulator_API_26

Android device, API 29
Data Type Encryption Method Test Count Data Size Total Encryption Time (ms) Total Decryption Time (ms) Avg. Encryption Time per Test (ms) Encryption Speed Ratio (New/Old) Avg. Decryption Time per Test (ms) Decryption Speed Ratio (New/Old)
string StringV1 120 100 2,930.79 6,955.97 24.42 87.94% 57.97 111.58%
string SJCL1a 120 100 2,577.45 7,761.15 21.48 100.00% 64.68 100.00%
string StringV1 80 10,000 3,260.86 5,922.43 40.76 211.38% 74.03 172.31%
string SJCL1a 40 10,000 3,446.34 5,102.37 86.16 100.00% 127.56 100.00%
string StringV1 3 1,000,000 6,808.99 7,959.50 2,269.66 425.29% 2,653.17 515.58%
string SJCL1a 3 1,000,000 28,958.31 41,037.93 9,652.77 100.00% 13,679.31 100.00%
file FileV1 120 100 5,177.57 7,392.57 43.15 95.60% 61.60 70.36%
file SJCL1a 120 100 4,949.73 5,201.50 41.25 100.00% 43.35 100.00%
file FileV1 70 10,000 5,090.32 5,956.49 72.72 196.20% 85.09 211.49%
file SJCL1a 40 10,000 5,707.08 7,198.42 142.68 100.00% 179.96 100.00%
file FileV1 3 1,000,000 6,215.57 8,599.70 2,071.86 823.07% 2,866.57 486.61%
file SJCL1a 3 1,000,000 51,158.57 41,847.04 17,052.86 100.00% 13,949.01 100.00%

Android_device_API_29


Patch for the performance test 
diff --git a/packages/app-desktop/app.ts b/packages/app-desktop/app.ts
index 9a56280a1..f2fc786ae 100644
--- a/packages/app-desktop/app.ts
+++ b/packages/app-desktop/app.ts
@@ -46,6 +46,7 @@ import { homedir } from 'os';
 import getDefaultPluginsInfo from '@joplin/lib/services/plugins/defaultPlugins/desktopDefaultPluginsInfo';
 const electronContextMenu = require('./services/electron-context-menu');
 // import  populateDatabase from '@joplin/lib/services/debug/populateDatabase';
+import { runIntegrationTests as runCryptoIntegrationTests } from '@joplin/lib/services/e2ee/cryptoTestUtils';
 
 const commands = mainScreenCommands
 	.concat(noteEditorCommands)
@@ -750,7 +751,8 @@ class Application extends BaseApplication {
 		// 	});
 		// }, 2000);
 
-		// await runIntegrationTests();
+		await new Promise(resolve => setTimeout(resolve, 20000)); // run performance test after init
+		await runCryptoIntegrationTests();
 
 		return null;
 	}
diff --git a/packages/app-mobile/android/gradle.properties b/packages/app-mobile/android/gradle.properties
index d30028f17..de76881c6 100644
--- a/packages/app-mobile/android/gradle.properties
+++ b/packages/app-mobile/android/gradle.properties
@@ -46,3 +46,8 @@ hermesEnabled=true
 #
 # https://github.com/robolectric/robolectric/issues/6521
 android.jetifier.ignorelist=bcprov
+
+JOPLIN_RELEASE_STORE_FILE=my-upload-key.keystore
+JOPLIN_RELEASE_KEY_ALIAS=my-key-alias
+JOPLIN_RELEASE_STORE_PASSWORD=123456
+JOPLIN_RELEASE_KEY_PASSWORD=123456
\ No newline at end of file
diff --git a/packages/app-mobile/root.tsx b/packages/app-mobile/root.tsx
index 35f1d9903..cc445a8bd 100644
--- a/packages/app-mobile/root.tsx
+++ b/packages/app-mobile/root.tsx
@@ -856,7 +856,6 @@ async function initialize(dispatch: Dispatch) {
 		} else {
 			logger.info('Skipping encryption tests -- not supported on web.');
 		}
-		await runCryptoIntegrationTests();
 		await runOnDeviceFsDriverTests();
 	}
 
@@ -945,7 +944,9 @@ class AppComponent extends React.Component {
 	// https://discourse.joplinapp.org/t/webdav-config-encryption-config-randomly-lost-on-android/11364
 	// https://discourse.joplinapp.org/t/android-keeps-on-resetting-my-sync-and-theme/11443
 	public async componentDidMount() {
+		let runPerformanceTest = false;
 		if (this.props.appState === 'starting') {
+			runPerformanceTest = true;
 			this.props.dispatch({
 				type: 'APP_STATE_SET',
 				state: 'initializing',
@@ -1040,6 +1041,11 @@ class AppComponent extends React.Component {
 
 		// Setting.setValue('encryption.masterPassword', 'WRONG');
 		// setTimeout(() => NavService.go('EncryptionConfig'), 2000);
+
+		if (runPerformanceTest) {
+			await new Promise(resolve => setTimeout(resolve, 20000)); // run performance test after init
+			await runCryptoIntegrationTests();
+		}
 	}
 
 	public componentWillUnmount() {
diff --git a/packages/lib/models/Setting.ts b/packages/lib/models/Setting.ts
index 30849bb3d..4983bbcde 100644
--- a/packages/lib/models/Setting.ts
+++ b/packages/lib/models/Setting.ts
@@ -945,7 +945,7 @@ class Setting extends BaseModel {
 	public static async saveAll() {
 		if (Setting.autoSaveEnabled && !this.saveTimeoutId_) return Promise.resolve();
 
-		logger.debug('Saving settings...');
+		// logger.debug('Saving settings...');
 		shim.clearTimeout(this.saveTimeoutId_);
 		this.saveTimeoutId_ = null;
 
@@ -1016,7 +1016,7 @@ class Setting extends BaseModel {
 			}
 		}
 
-		logger.debug('Settings have been saved.');
+		// logger.debug('Settings have been saved.');
 	}
 
 	public static scheduleChangeEvent() {
diff --git a/packages/lib/registry.ts b/packages/lib/registry.ts
index 12b09ccf2..3442dd999 100644
--- a/packages/lib/registry.ts
+++ b/packages/lib/registry.ts
@@ -126,7 +126,7 @@ class Registry {
 				// return;
 			}
 
-			this.logger().debug('Scheduling sync operation...', delay);
+			// this.logger().debug('Scheduling sync operation...', delay);
 
 			const timeoutCallback = async () => {
 				this.timerCallbackCalls_.push(true);
diff --git a/packages/lib/services/e2ee/crypto.test.ts b/packages/lib/services/e2ee/crypto.test.ts
index ff39b3110..80e40ca7d 100644
--- a/packages/lib/services/e2ee/crypto.test.ts
+++ b/packages/lib/services/e2ee/crypto.test.ts
@@ -16,7 +16,7 @@ describe('e2ee/crypto', () => {
 
 	it('should encrypt and decrypt data from different devices', (async () => {
 		await expectNotThrow(async () => runIntegrationTests(true));
-	}));
+	}), 150000);
 
 	it('should not generate new nonce if counter does not overflow', (async () => {
 		jest.useFakeTimers();
diff --git a/packages/lib/services/e2ee/cryptoTestUtils.ts b/packages/lib/services/e2ee/cryptoTestUtils.ts
index 0c56f9311..2dfbbd66b 100644
--- a/packages/lib/services/e2ee/cryptoTestUtils.ts
+++ b/packages/lib/services/e2ee/cryptoTestUtils.ts
@@ -74,8 +74,12 @@ export async function checkDecryptTestData(data: DecryptTestData, options: Check
 		for (const msg of messages) {
 			if (hasError) {
 				logger.warn(msg);
+				console.warn(msg);
 			} else {
-				if (!options.silent) logger.info(msg);
+				if (!options.silent) {
+					logger.info(msg);
+					console.info(msg);
+				}
 			}
 		}
 	}
@@ -132,8 +136,12 @@ export async function testStringPerformance(method: EncryptionMethod, dataSize:
 		for (const msg of messages) {
 			if (hasError) {
 				logger.warn(msg);
+				console.warn(msg);
 			} else {
-				if (!options.silent) logger.info(msg);
+				if (!options.silent) {
+					logger.info(msg);
+					console.info(msg);
+				}
 			}
 		}
 	}
@@ -191,8 +199,12 @@ export async function testFilePerformance(method: EncryptionMethod, dataSize: nu
 		for (const msg of messages) {
 			if (hasError) {
 				logger.warn(msg);
+				console.warn(msg);
 			} else {
-				if (!options.silent) logger.info(msg);
+				if (!options.silent) {
+					logger.info(msg);
+					console.info(msg);
+				}
 			}
 		}
 	}
@@ -226,10 +238,11 @@ const decryptTestData: Record<string, DecryptTestData> = {
 
 // This can be used to run integration tests directly on device. It will throw
 // an error if something cannot be decrypted, or else print info messages.
-export const runIntegrationTests = async (silent = false, testPerformance = false) => {
+export const runIntegrationTests = async (silent = false, testPerformance = true) => {
 	const log = (s: string) => {
 		if (silent) return;
 		logger.info(s);
+		console.info(s);
 	};
 
 	log('Running integration tests...');
@@ -252,31 +265,35 @@ export const runIntegrationTests = async (silent = false, testPerformance = fals
 	if (testPerformance) {
 		log('Testing performance...');
 		if (shim.mobilePlatform() === '') {
-			await testStringPerformance(EncryptionMethod.StringV1, 100, 1000);
-			await testStringPerformance(EncryptionMethod.StringV1, 1000000, 10);
-			await testStringPerformance(EncryptionMethod.StringV1, 5000000, 10);
-			await testStringPerformance(EncryptionMethod.SJCL1a, 100, 1000);
-			await testStringPerformance(EncryptionMethod.SJCL1a, 1000000, 10);
-			await testStringPerformance(EncryptionMethod.SJCL1a, 5000000, 10);
+			await testStringPerformance(EncryptionMethod.StringV1, 100, 2400);
+			await testStringPerformance(EncryptionMethod.SJCL1a, 100, 5700);
+			await testStringPerformance(EncryptionMethod.StringV1, 10000, 1700);
+			await testStringPerformance(EncryptionMethod.SJCL1a, 10000, 1300);
+			await testStringPerformance(EncryptionMethod.StringV1, 1000000, 70);
+			await testStringPerformance(EncryptionMethod.SJCL1a, 1000000, 15);
+			await testStringPerformance(EncryptionMethod.StringV1, 5000000, 18);
+			await testStringPerformance(EncryptionMethod.SJCL1a, 5000000, 5);
 			await testFilePerformance(EncryptionMethod.FileV1, 100, 1000);
-			await testFilePerformance(EncryptionMethod.FileV1, 1000000, 3);
-			await testFilePerformance(EncryptionMethod.FileV1, 5000000, 3);
-			await testFilePerformance(EncryptionMethod.SJCL1a, 100, 1000);
-			await testFilePerformance(EncryptionMethod.SJCL1a, 1000000, 3);
+			await testFilePerformance(EncryptionMethod.SJCL1a, 100, 1500);
+			await testFilePerformance(EncryptionMethod.FileV1, 10000, 1000);
+			await testFilePerformance(EncryptionMethod.SJCL1a, 10000, 400);
+			await testFilePerformance(EncryptionMethod.FileV1, 1000000, 80);
+			await testFilePerformance(EncryptionMethod.SJCL1a, 1000000, 5);
+			await testFilePerformance(EncryptionMethod.FileV1, 5000000, 20);
 			await testFilePerformance(EncryptionMethod.SJCL1a, 5000000, 3);
 		} else {
-			await testStringPerformance(EncryptionMethod.StringV1, 100, 100);
-			await testStringPerformance(EncryptionMethod.StringV1, 500000, 3);
+			await testStringPerformance(EncryptionMethod.StringV1, 100, 120);
+			await testStringPerformance(EncryptionMethod.SJCL1a, 100, 120);
+			await testStringPerformance(EncryptionMethod.StringV1, 10000, 80);
+			await testStringPerformance(EncryptionMethod.SJCL1a, 10000, 40);
 			await testStringPerformance(EncryptionMethod.StringV1, 1000000, 3);
-			await testStringPerformance(EncryptionMethod.SJCL1a, 100, 100);
-			await testStringPerformance(EncryptionMethod.SJCL1a, 500000, 3);
 			await testStringPerformance(EncryptionMethod.SJCL1a, 1000000, 3);
-			await testFilePerformance(EncryptionMethod.FileV1, 100, 100);
-			await testFilePerformance(EncryptionMethod.FileV1, 100000, 3);
-			await testFilePerformance(EncryptionMethod.FileV1, 500000, 3);
-			await testFilePerformance(EncryptionMethod.SJCL1a, 100, 100);
-			await testFilePerformance(EncryptionMethod.SJCL1a, 100000, 3);
-			await testFilePerformance(EncryptionMethod.SJCL1a, 500000, 3);
+			await testFilePerformance(EncryptionMethod.FileV1, 100, 120);
+			await testFilePerformance(EncryptionMethod.SJCL1a, 100, 120);
+			await testFilePerformance(EncryptionMethod.FileV1, 10000, 70);
+			await testFilePerformance(EncryptionMethod.SJCL1a, 10000, 40);
+			await testFilePerformance(EncryptionMethod.FileV1, 1000000, 3);
+			await testFilePerformance(EncryptionMethod.SJCL1a, 1000000, 3);
 		}
 
 	}

Several miscellaneous tests provided by @personalizedrefriger

What's left to be done

  • Add specification of new encryption methods.
9 Likes