This week, I started by working on automatic tag generation for our clusters and opened PR #23. I wrote a simple, dependency-free TF-IDF extractor in TypeScript that cleans the note text, filters out common stop words, and picks the most unique keywords for each cluster. These are now displayed as tags directly on the cluster cards in our React UI.
After that, since Joplin introduced native on-device AI search, I shifted focus to integrate it (PR #24). Since the new API doesn't expose raw embedding vectors directly, it only lets us query for similar notes, so I built a distance matrix by querying similar notes for each note and converting their similarity scores into distances (1 - score). I updated UMAP to support a custom distance function to project this matrix into a 10D space that our clustering algos can use.
I also set it up as a hybrid pipeline, so it automatically uses Joplin's native search when it's ready, or falls back to our local ONNX web worker when it isn't.
One problem that I faced during testing was that our local ONNX fallback was sometimes returning NaN vectors on longer notes, which corrupted the cache and broke the clustering.
For next week, I plan to address this issue by adding a dynamic fallback, along with a self-healing cache validator to automatically find and re-embed corrupted records. I also want to work on auto-naming the formed clusters.
The plugin API could expose more information. It seems in particular that giving access to the raw vectors would help here?
it automatically uses Joplin's native search when it's ready
How do you currently check if it's ready or not? Because I recently added a few internal functions to check the readiness of the AI APIs - would it make sense to expose them?
Yes, exposing the raw vectors would help a lot. It would let us fetch all embeddings in a single batch call instead of running O(N) individual search queries. it also avoids the top-20 search results limit, giving umap 100% accurate global distance data for projection.
I do it inside a try-catch block. This verifies if the ai namespace exists, if AI is enabled in settings, and if the native sqlite-vec extension is loaded. The problem is that it doesn't tell us if the database is fully indexed.
I checked your recent PR (#15785) - the embeddingAvailability() helper checks the conditions we need. Exposing it to the plugin API (e.g. joplin.ai.embeddingAvailability()) would be very helpful.
Also, if possible, having it indicate whether background note indexing is complete (so the database isn't empty on a fresh install) would be a great addition!
I tested both PRs locally and they work perfectly!
I was able to check the status, fetch all note embeddings paginated, group/average their chunks, and run clustering with no issues. The performance is super fast and the clustering quality is great (got a silhouette score of 0.93 on 50 notes).
These APIs cover everything we need for the hybrid pipeline, so they are good to go from my end. Thanks a lot!
This is a nice direction. A couple of things from having done embedding-based clustering and auto-labelling on personal notes:
On the cluster labels: TF-IDF is a reasonable start, but it optimises for distinctiveness, which isn't quite the same as representativeness. It tends to surface the rare, unusual terms in a cluster rather than the central one, so labels can come out technically-unique but not what a human would call the cluster. A cheap fix is to combine the two: take the TF-IDF candidates, then pick the one whose own embedding sits closest to the cluster centroid. Distinctive AND central usually reads much more like a real tag.
On the silhouette score: 0.93 on 50 notes is worth a quick sanity check before trusting it. At that size you probably have a handful of well-separated clusters, which silhouette loves. Real vaults are messier, since lots of notes sit genuinely between topics, so as N grows the clean separation tends to drop and silhouette with it. I would validate on a few thousand notes before tuning to that number. It is also worth remembering that personal notes are often legitimately multi-topic, so hard clustering (one note, one cluster) fights the data a little. A kNN similarity graph with community detection, or just allowing soft/overlapping membership, often matches how people actually think about their own notes better than forcing k-means partitions.
Last thing, more product than algorithm: auto-tags are a great candidate list, but the fastest way to make people distrust the feature is to apply them silently. Surfacing them as suggestions to accept or rename keeps the tag namespace clean and keeps the user in control, which matters a lot for something as personal as a tag vocabulary. Great to see this landing natively.
Thanks for the feedback, these are really good points!
For cluster naming, combining TF-IDF with embedding similarity to the centroid is a really good idea since we already calculate the centroids. I also want to give users the option to use Joplin's new built-in AI features to name the clusters (by sending note titles to the AI). I'll probably support both local TF-IDF naming and AI naming and let the user choose what they want to use.
On scores and scaling, I did test it with very different types of notes, but yeah, 50 notes is small. I want to test on much larger notebooks, but I don't have that many notes myself. The plan is to release a v1 in a few weeks, get feedback from users with larger vaults, and tune it based on that.
For the product side, that's exactly what I'm planning to do. We won't apply any tags silently; they will just show up as suggestions in the UI panel so the user has full control to review, rename, or accept them.
Makes sense, and giving users the local-vs-AI choice is the right call. One caution on the AI-naming path, since it is the most likely to surprise you: naming a cluster from note titles alone is about the weakest input you can hand the model. On personal notes the title is often the least representative part (lots of "Untitled", dates, one-word stubs, or a title that fits one note rather than the theme), so the model anchors on whatever the titles happen to share and will confidently return a plausible but wrong label. That is the hallucination risk with little of the signal.
The fix composes with the centroid idea you already have: use TF-IDF + centroid distance to pick the evidence (top terms plus the two or three notes nearest the centroid), and if you want AI phrasing, feed the model that rather than raw titles. The local method decides what the cluster is actually about; the LLM only makes it read like a human tag, and constraining its input to centroid-representative material is what keeps it from drifting to a nice-sounding label no note supports.
It also pairs with your suggestions-in-UI plan: keep the few source terms/notes attached to each suggested tag, and a user renaming one can see why it was proposed instead of guessing. Cheap to keep, and it makes the feature feel trustworthy rather than magic.