Starting the Joplin will at first make the Joplin AI Summarization panel appear. Users can hide/show panel by using keyboard shortcuts: command + shift + f (MacOS) and ctrl + shift + f (Windows)
Flowchart
I have created this diagram to show how you can get your summaries.
If you find something strange or you can see some improvements that need to be made, please do not hesitate to let me know. It is always great to hear other people's opinions and perspectives! Write your inputs by replying to this thread!
Thank you, and I hope you will have some fun with the plugin
I gave it a try and it seems to work - did you manage eventually to embed the model in the plugin?
Unless I'm missing something I feel the UX could be improved - at the moment, you replicate the notebook/note tree in that panel, and user needs to select a note in there, is that right? Instead, why not display the summary for the currently selected note, and get rid of the duplicate notebook tree?
In terms of summary quality, TextRank seems to be better simply because it formats the output with new lines. The others output a single line with all the text in there, so as a user I wouldn't use this option, but maybe that can be improved.
When I make it process an ordered list of items with item 1, 2, 3, 4, etc. it's going to pick certain items and display the summary as for example "5. this 17. that, 23. this" etc. All algorithms do this, except they all pick different numbers.
Hi, thank you for your feedback!! I am glad that the plugin works for you!!
UI/UX
The reason why I am displaying the whole notebook tree is that users would easily identify which notes are summarised and which are not. But I agree that if users click on the certain notes, it should display the corresponding note "crafting" in the panel. I will definitely implement that.
Also, maybe I should create a filter feature to see only the summarised notes.
AI
That's a good point actually. I can actually do hierarchical clustering on the final N selected sentence vectors and then create a line between those clusters. Formatting can be definitely improved with grouping sentences with some cohesion.
Oh yeah, the unsupervised algorithms are not good at dealing with ordered list of items. It basically only considers the textual value and disregards the numbered/alphabetic points. If it thinks that the sentence/keywords in the list items are worth including in the overall summary then it only takes that those points. That is why I think it displays ordered list of items randomly. I just need to see a way what is best to approach this problem.
The supervised methods (LLMs, Transformers.js, etc.) are better at those since they are generative. I am still working on creating a minimal web worker on the Joplin side and use the API for a plugin to use the Transformers.js.
Great work! Nice to test the plugin with my feedback added. I have other feedback alongside testing results. Please let me know if you have any questions.
Testing
Clicking OK or Cancel in the Summarize the notebook dialog throws a console error
Summarizing multiple selected notes by Ctrl or Shift click only summarizes the last selected note
Clicking on new line H1, H2, and Bullet list options don't do anything
These results can be compared with the video demonstration.
The notebook tree doesn't appear so I can't test the panel flow
Summary Title, Length, and Algorithms options are missing
Note: doesn't populate with note title
Default editor input decoration isn't visible
Feedback
GUI Preference
Make summarizing a 1 step process through dialog or panel based on a GUI preference in settings, as seen in the Search and Replace plugin screenshots.
With either mode, the editor and all options show at once, and OK then Save gets condensed to Save
With panel mode, comparison is permitted and Laurent's idea of displaying the current note's summary is met
With dialog mode, dialog appears on shortcut or context menu click and disappears on Save or Cancel button press
If the above is too confusing or too much work, can all options go into the panel?
Identifying summarized notes
This can be handled with plugin features and/or existing tooling:
Setting custom note titles and/or headers can allow search (plugins) to find or list summaries in one place
Sending summaries to designated notebooks (whether for all summaries or specific kinds) can serve the same function
Adding (inline) tags for summaries can also serve the same function
With these methods, you can skip duplicating the notebook tree, let alone filtering notes.
Visuals
Add instructions above summary editor, including instructions to show toolbar when highlighting text
Make editor fill remaining vertical space in panel, if any (editor height is fixed at 400px)
Scrollbar unexpectedly shows up right of editor whether editor is under or over fixed height
Remove excess spacing (padding, margin, etc.) and condense elements
For example, Search and Replace and VS Code Search plugins minimize whitespace and use space well
This minimizes panel resizing like in the video
This permits maximal editor space
Improve color contrast:
Make highlighted text toolbar background white
Use black or color with enough contrast for block quote, active editor input decoration
Add dark mode based on system default and/or dark theme selection
Show or hide the panel based on whether it was open or closed before quitting Joplin
Interactions
Change default keyboard shortcut to one without conflicts (VS Code Search plugin defaults to Ctrl+Shift+F)
Prevent note summarization after clicking Cancel or pressing Esc
Add simple interactive or text explanations for any summary options a user might not understand
Create radio button set for summary location including Above current note, Below current note, and Create new note
Manifest
If possible without changing plugin ID, change plugin name to "AI Summarisation" to match existing naming (plugin ID, Github URL, etc.)
Add "summary" to keywords and/or description so users searching for "summary" can find it
Thank you so much for your feedback!! I will get back to you in the following weeks since my focus now is AI development again. Your feedback and inputs are very clear and comprehensive though!!
The fact that you are not able to use the panel is a bit worrying and should be prioritised fixing it. It is strange to see that it does not work you.
Have you tried to restart your app or installing/uninstaling the plugin? On the startup, you should see the note/notebook tree. Have you looked in logs maybe? I am just trying to see where can I help.
@HahaBill, can I confirm again how this plugin is currently working without the worker API? I don't assume it's sending the note content to a web service, right? If it does that would need be mentioned very clearly somewhere
But how is the LLM model loaded, how does it produce the answer? I checked your code a bit and couldn't find so that would be great if you could clarify.
Not sure if my question is not clear or what, but it's the third time I ask you so it'd be nice to get an answer.
Firstly, to give you a context, up until now, I have only used NLP and implemented unsupervised methods. This is a type of machine learning where you don’t need labeled data, but you have to understand the fundamental theories and implement things more smartly.
With those, we can only do extractive summarization, which involves removing the most important sentences in the note and creating a summary.
If we want an abstractive summarization then we need to use LLMs or other generative systems. That’s where Transformers.js comes in and that’s why we want to create a worker and load the Transformers.js in the app to create such summaries because I and people in the community in the past were unable to use Transformers.js in plugins. I thought it would be super nice to do abstractive summarization too but it’s not mandatory.
How do algorithms produce answers/summaries?
We are using NLP to create summaries without using web workers or web services. The plugin works completely fine offline:
Getting the note body from plugin API
Tokenize sentences with natural (create array of sentences from the note)
Do vectorization -> convert sentences into vectors
Do various of algoritms. For example, in LSA, we create sentence vectors to form a matrix and then perform SVD to discover the most important dimensions. With those dimensions, we can determine which sentences are the most important.
You take n sentences from the result
You can find the implementations in utils/handlers folder.
Final Remarks
This is how the plugins work right now. I think getting Transforrmers.js working in Joplin is worth the effort and is not only good for this project but for future AI projects too. Plus it works offline, providing more privacy for users. So currently we are trying to accompish extra goals.
Oh I see, that makes perfect sense then. Indeed I hadn't realise it was done using NLP, which I wasn't even familiar with. Thanks for clarifying things, and I agree that supporting workers and libs such as Transformers.js would be very useful for all kind of plugins.
