Weekly Update 12: Running Language Models, Improving UI/UX and Released (v.0.1.1) 🚀

GSoC Projects Summarize with AI
, ,
1

1. Progress

  • I have released a new version (v.0.1.1) of the plugin :rocket: It can be seen in npm, but it hasn't been updated in Joplin's Plugin Repository. Once it is there, I will create a topic about that. It has been more than 24 hours, though.
  • Fixing bugs:
    • The summary notes page was incorrectly displayed when starting the app (2 bugs). I believe @muzak encountered this, but it should be fixed now!
    • The summary title is not displaying in the panel
    • Pressing cancel still summarizes the note
    • etc. more will be discussed in the release topic
  • I got Transformers.js working in the plugin !!! Big thanks to @personalizedrefriger
    • I tested multiple ONNX quantized models from Xenova (Joshua). Based on the performance and speed, I chose the google/flan-t5-small model (roughly 60MB): google/flan-t5-small · Hugging Face
    • In the future, we can fine-tune this model to make better summaries with datasets I encountered while implementing NLP/unsupervised methods (machine learning without needing labeled data).
  • Tried to run Pyodide and word2vec but encountered a few issues -> low priority. This will be discussed in the "Future work" or "What's left to do" section for GSoC's final document.
  • Read Work Product and Submission Guidelines for GSoC: Work Product Submission Guidelines  |  Google Summer of Code  |  Google for Developers

2. Plan

  • I will do more testing to ensure the plugin works as expected.
  • I will finalize the final document for GSoC and mentors. I still need to discuss with the mentors whether the submission URL should be Discourse or the GitHub repository with the README.md
    In the proposal and also the weekly update in week 5 (Weekly Update 4-5: KMeans Clustering, Evaluation Research / Survey and Express.js in a plugin), I mentioned creating unit tests in Phase III, which was last week and this week. I will do some unit tests with Jest.
  • I will release a new version if we find critical bugs or improvements that need to be made.
  • Finally, clean up the code and add more logs if necessary.

3. Problem

  • (low-priority) I implemented a simple script using Pyodide in a web worker to see whether I can load Pyodide. I got this error:
    • SOLUTION 1: I tried to solve with using resolve.alias: Resolve | webpack. But that did not work
    • SOLUTION 2: I tried to solve using externals: Externals | webpack. It did not work either :confused:
joplin-plugin-ai-summarization@0.1.1 dist
> webpack --env joplin-plugin-config=buildMain && webpack --env joplin-plugin-config=buildExtraScripts && webpack --env joplin-plugin-config=createArchive

ERROR in node:child_process
Module build failed: UnhandledSchemeError: Reading from "node:child_process" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:918:25
    at Hook.eval [as callAsync] (eval at create (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:915:8)
    at processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:905:3)
    at NormalModule.build (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1081:15)
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/Compilation.js:1400:12
    at NormalModule.needBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1407:32)
 @ ./node_modules/pyodide/pyodide.js 3:1624-1671
 @ ./src/workers/pyodideWorker.ts 13:18-36

ERROR in node:crypto
Module build failed: UnhandledSchemeError: Reading from "node:crypto" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:918:25
    at Hook.eval [as callAsync] (eval at create (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:915:8)
    at processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:905:3)
    at NormalModule.build (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1081:15)
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/Compilation.js:1400:12
    at NormalModule.needBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1407:32)
 @ ./node_modules/pyodide/pyodide.js 3:1535-1575
 @ ./src/workers/pyodideWorker.ts 13:18-36

ERROR in node:fs/promises
Module build failed: UnhandledSchemeError: Reading from "node:fs/promises" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:918:25
    at Hook.eval [as callAsync] (eval at create (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:915:8)
    at processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:905:3)
    at NormalModule.build (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1081:15)
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/Compilation.js:1400:12
    at NormalModule.needBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1407:32)
 @ ./node_modules/pyodide/pyodide.js 3:1342-1387
 @ ./src/workers/pyodideWorker.ts 13:18-36

ERROR in node:fs
Module build failed: UnhandledSchemeError: Reading from "node:fs" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:918:25
    at Hook.eval [as callAsync] (eval at create (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:915:8)
    at processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:905:3)
    at NormalModule.build (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1081:15)
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/Compilation.js:1400:12
    at NormalModule.needBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1407:32)
 @ ./node_modules/pyodide/pyodide.js 3:1297-1333
 @ ./src/workers/pyodideWorker.ts 13:18-36

ERROR in node:path
Module build failed: UnhandledSchemeError: Reading from "node:path" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:918:25
    at Hook.eval [as callAsync] (eval at create (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:915:8)
    at processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:905:3)
    at NormalModule.build (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1081:15)
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/Compilation.js:1400:12
    at NormalModule.needBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1407:32)
 @ ./node_modules/pyodide/pyodide.js 3:3403-3441 3:1451-1489
 @ ./src/workers/pyodideWorker.ts 13:18-36

ERROR in node:url
Module build failed: UnhandledSchemeError: Reading from "node:url" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:918:25
    at Hook.eval [as callAsync] (eval at create (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/tapable/lib/Hook.js:18:14)
    at Object.processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:915:8)
    at processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:905:3)
    at NormalModule.build (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1081:15)
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/Compilation.js:1400:12
 @ ./node_modules/pyodide/pyodide.js 3:3455-3492 3:1242-1279
 @ ./src/workers/pyodideWorker.ts 13:18-36

ERROR in node:vm
Module build failed: UnhandledSchemeError: Reading from "node:vm" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:918:25
    at Hook.eval [as callAsync] (eval at create (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:915:8)
    at processResource (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:905:3)
    at NormalModule.build (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1081:15)
    at /Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/Compilation.js:1400:12
    at NormalModule.needBuild (/Users/billtonhoang/Documents/GitHub/joplin-hahabill/plugin-ai-summarisation/node_modules/webpack/lib/NormalModule.js:1407:32)
 @ ./node_modules/pyodide/pyodide.js 3:1397-1433
 @ ./src/workers/pyodideWorker.ts 13:18-36

webpack compiled with 7 errors"
  • I encountered some bugs but managed to solve them in the end.

Last Week of GSoC for Contributors

This is most likely my last Weekly Update for GSoC. Therefore, I would like to thank everyone (mentors, community, etc.) who embarked on this journey with me. It has been a truly amazing experience for me!! I am very grateful for that, and I hope you guys are enjoying the new plugin!!

I am not sure if you guys noticed, and it might not be obvious, but I have learned so much!! Plus, it was really fun to dive into the world of NLP!!
However, I really struggled with the AI ecosystem's limitations in Javascript. Still, it forced me to implement the algorithms from scratch, which was really cool! It was also cool to imagine and apply some linear algebra concepts! Usually, I would otherwise use sci-kit-learn or other machine-learning packages if they existed in Javascript.
However, having those packages would be very beneficial if we want to have the most optimal unsupervised algorithms. For example, I wanted to do co-reference resolution, which means we match nouns with pronouns in the following sentences (Spiderman is cool. He can fly! -> Spiderman is cool. Spiderman can fly!). That would strengthen the connection between sentences. One way is to do Hobb's algorithm. Still, the disadvantage is that it is heuristics-based, which lacks understanding of semantic connections between sentences (also, the distance between sentences plays is a problem). Using neural networks would be much better as they cover more cases and reduce false positives (GitHub - huggingface/neuralcoref: ✨Fast Coreference Resolution in spaCy with Neural Networks)!

Anyway, this is not the end!! I will still be here after the program since the plugin could be improved. I think it is nice that, in the end, we allow users to edit summaries and add their notes in the Tiptap editor to their actual notes!
Furthermore, I definitely want to implement a new plugin API with web workers (Tech Spec: Generic Web Worker in Joplin) that would be easy for future contributors to create workers!

Thank you once again for all the support and fun! I’m happy to be your GSoC contributor for 2024!

5 Likes