Hey everyone! Hope you all had a great week. Here is a quick update on what I worked on:
What I did this week:
- Refined the core UMAP & Clustering PR
- Refined the algorithms based on testing: I tested the clustering algorithms on our note vectors. Based on the results, I removed X-Means and K-Means-8 completely because they weren't performing well. I also thought about removing K-Means entirely, but as the number of notes increases, K-Means scales and performs better than K-Medoids. So, I decided to keep it. Currently, we have 3 main algorithms active: HDBSCAN, K-Means, and K-Medoids.
- Added interactive 3D visualizations! To compare the results better, we visualized the clustered notes in both 2D and 3D. You can check out the interactive files I have shared below.
Plan for this week:
I have two main directions I can go from here, and I'll be discussing with my mentor (Ton Hoang) which path to prioritize:
- Clustering Enhancements: Work on incorporating the notes' current notebooks (
parent_id) as weighted features to guide the clustering, and build a helper to automatically name the suggested clusters using title keywords and majority votes. - UI Integration: Start working on the user interface and Joplin panels to present these clusters visually to the user.
Visualization of the notes:
In 3d (using different algo):
hdbscan-3.html (15.0 KB)
hdbscan-3-ms2.html (15.0 KB)
hdbscan-5-ms2.html (14.7 KB)
kmeans-5.html (14.3 KB)
kmedoids-5.html (14.3 KB)
You can also view the previous 2D static plots here.