GSoC 2026: Opportunities for the AI projects

GSoC
21

That’s a fair point, especially with loosely titled notes.

In my approach, I’m not relying only on headings. While building the tree, each node also stores a short LLM-generated summary of its content. So even if a note is titled something like “Monday meeting”, the node would still capture what the meeting was actually about.

This makes the structure more semantic-aware rather than purely based on titles. The idea is to combine structure with lightweight semantic understanding, instead of depending entirely on embeddings.

That said, I agree it may not fully replace semantic search in all cases, but it could still work well as a complementary or lightweight alternative, especially for structured markdown notes.

22

If any GSoC contributor is reading this, as noted above that can be a good opportunity for a project, so don't hesitate creating a proposal if you have some ideas.

23

@adamoutler thanks for the detailed breakdown, lots of useful thinking in there. A few things I want to pick up on:

Your reranking observation is interesting: that it matters more for smaller on-device models than for larger cloud models. Good to keep in mind for the infrastructure project, since we'd want to support both.

The hybrid search idea (keyword ↔ vector slider) is a nice concrete way to think about the UI for that.

The "negative friction" idea for MCP design makes a lot of sense. Keeping round trips and context usage low matters. There are already a few Joplin MCP servers out there (including one I maintain) that work with the desktop client's Data API. Yours works with Joplin Server, which is cool, I don't think anyone else has explored that space. For the GSoC projects though, which run as plugins / core inside the app, I think we can keep things simpler by describing tools directly in LLM calls rather than going through MCP (as I described in my reply above). Whether Joplin should bundle an MCP server inside the app for external consumers is a different question.

On vector DB choices: the discussion here is mostly about plugin-level projects that need to work inside the app on desktop and mobile. That narrows things down quite a bit. Your overview is very helpful, and I agree that sqlite-vec looks like a natural starting point given its cross-platform support. Making sure whatever we pick actually works on mobile (where FS access is limited) should be part of any infrastructure project. Perhaps we may need to include PRs to the mobile app.

@Krishh interesting idea with the hierarchical summary tree. LLM-generated summaries at each node are more robust than headings alone, and a nice complement to embedding-based retrieval. As @adamoutler noted, pure structural approaches can miss semantic relationships, but combining a summary tree with Joplin's search tools (as I described above) could give you the best of both. Might be worth exploring as part of the search project.

24

This discussion aligns very closely with what I’ve been thinking while working on my proposal.

  • For the past 4 days I've been planning a proposal combining the ideas 1,3 and 4 because after thoroughly researching about these ideas, it came to my attention that they fall under the same single umbrella that is they share the same retrieval foundation.

  • I kept coming back to the fact that all 3 required building the embedding pipeline first then letting the three features consume it.

  • This approach aligns with the removal of the duplication issue i.e., multiple plugin usage in which each plugin builds its own pipeline and we keep repeating the same work multiple times which could have been minimized into a standalone pipeline used by all 3 features or more in future decreasing money wastage in terms of memory and architecture.

  • I’m currently thinking of framing the project around this shared infrastructure as the core deliverable, with a few minimal consumer features (search, chat, auto-tagging) implemented mainly to validate and demonstrate the system end-to-end rather than as fully independent products.

  • Would love to hear if this direction aligns with how you’d expect these ideas to be approached, or if there are constraints in Joplin’s current architecture that would push this in a different direction.

  • I’ll be creating a separate discussion post to explore this approach in more detail and clarify a few open questions, would really appreciate any guidance or feedback there.

My Discussion Post: Design Discussion: Shared Embedding & Retrieval Infrastructure for Joplin AI Features

25

As someone working on the categorisation proposal, I've already empirically validated the compressed similarity range issue with nomic-embed-text during POC development, notes with short bodies required thresholds below 0.60 regardless of topic distance. I'm planning to implement the put(note)/query(text) interface with a swappable backend so it can migrate to shared infrastructure later.

Quick question for @shikuz - for the categorisation use case specifically, would you recommend sqlite-vec embedded directly in the plugin, or designing around an external service interface from the start?"

26

I’m a paid user of Joplin for several years; I compared many note-taking apps and Joplin was clearly the optimal choice for me. I’m no technophobe; I made my living in software development, now retired. However, I’m a confirmed skeptic when it comes to AI; I’ve turned it off in all the desktop and Android apps where possible. So, I would implore anyone working on AI features to please make them optional, via an easily selectable on/off switch. Thanks.

27

Yeah we should definitely have a flag for each AI feature, since i have observed a desire among users to have an AI free experience.

28

Any such feature would definitely be optional. We have no intention to push these as some companies do - we hope however that whatever will be developed will be useful, and if it is then users can enable them themselves

31

When done correctly, AI is a completely transparent net-positive. Semantic search can provide enhanced contextual understanding using on-device, local-only models.

eg. Someone searches for “kitty” but they wrote the word cat. The semantic understanding provided by the vector model handles that automatically while the traditional search continues to provide the most recent exact match.

eg2. they search for “kitty” again, and find that note where they talked about “tom” and a picture of a cat.

As far as the user knows, it’s not AI. It’s just a really good search function. This sort of AI SHOULD be pushed. Joe Shmoe doesn’t know how it works, or why it works. They just know it is better… as long as it’s done right.

32

That’s a great insight. Initially I was thinking of adding a semantic search layer, but your approach feels simpler and more practical compared to introducing extra complexity.

I think we can start with Joplin search + structured tree, and optionally use semantic search as a fallback when keyword-based retrieval doesn’t work well. That way it stays simple while still handling harder cases.

I’ll explore this direction further.

33

From what I’ve explored while working on semantic search, a hybrid approach (lexical + semantic with fallback) seems to strike a good balance—lexical for precision and filters, and semantic primarily for recall when queries are more descriptive or ambiguous.

For the storage side, starting with something like sqlite-vec locally makes a lot of sense for simplicity and privacy, especially for a plugin-first approach. Designing the interface to be swappable (as you mentioned) feels important though, so it can evolve later without tight coupling to a single backend.

Curious to hear thoughts from @shikuz and others:
For Joplin specifically, would you lean toward keeping the first iteration strictly local (embedded vector store), or designing early for a pluggable backend even if it adds a bit more complexity upfront?

34

“Correctly” is carrying a lot of weight here.

I’m with guy-rouillier: I do not want generative AI anywhere near my Joplin experience or data.

35

Please note: I didn't say generative AI. But I'm certain you'd appreciate some level of semantic search where eg. The words code, programming, and similar are searched when you type the name of a programming language. The semantic search function, which all the main AI projects listed above depend upon, requires an AI Model of about the same level as the next word generation on your keyboard.

The generative AI gets things wrong and many times takes more action than people want. However keeping it to some level of hybrid, context aware, search is pretty much universally good.

36

Based on the feedback here and @shikuz suggestions, I've now created an initial spec for what I think we would need as AI infrastructure in Joplin. In a nutshell, the idea is to have a few core AI components that can be used by plugins or built-in features, with proper privacy controls that are on by default.

Please see below for the full spec - comments are welcome!

Note: Latest version of the spec is now in this document:

https://github.com/laurent22/joplin/blob/39848025afa6398b7b35ca918a79922636095628/readme/dev/spec/ai_primitives.md

AI features

This spec describes the core AI primitives that will be added to Joplin. The goal is not to ship a single AI feature, but to provide a platform on which features and plugins can be built. The primitives below are validated against four target use cases:

  • Chat with your note — a sidebar that can summarise, rewrite, or answer questions about the current note.
  • AI-based note categorisation and tagging — background analysis that suggests tags and notebook structures based on note content.
  • AI-generated note graphs — surfacing semantically related but unlinked notes.
  • Fuzzy semantic search — finding notes by meaning rather than exact terms (e.g. "the note about pet sitters for my dog").

Each primitive below should plausibly serve at least two of these use cases. Features should be built on top of these primitives, never alongside them.

Overview

The primitives are:

  1. Provider abstraction — pluggable layer for LLM and embedding models.
  2. Local embeddings index — background-indexed semantic store, local-only, not synced.
  3. Retrieval helpers — the shared query surface that plugins actually call.
  4. Privacy & cost guardrails — enforced at the provider layer so every feature inherits them.
  5. MCP server — exposes Joplin notes to external AI tools.

Primitives 1–3 are required for any of the four target use cases to work. Primitive 4 must be in place from day one. Primitive 5 is independently valuable and can ship in parallel.

1. Provider abstraction

A pluggable layer so users can pick their LLM and embedding model independently (cloud, self-hosted, or on-device). No provider is hardcoded.

Two models are configured independently:

  • Chat model — generates and transforms text (summaries, rewrites, answers).
  • Embedding model — turns text into vectors for retrieval.

Users may legitimately mix providers (e.g. a cloud chat model with a local embedding model) so the API treats them as two separate slots.

Configured providers and the active provider

Users configure a list of providers (each with its own settings — API key, base URL, model name) and select one as active for chat and one as active for embeddings.

Built-in providers

  • An OpenAI-compatible adapter (covers OpenAI, Ollama, LM Studio, vLLM, OpenRouter, and similar via base-URL override).
  • An Anthropic adapter.
  • A bundled local embedding model (see below)

2. Local embeddings index

Notes are chunked, embedded, and stored locally so retrieval can run without a network call.

Storage

  • A new local SQLite table holding (note_id, chunk_index, model_id, vector) and the source text of each chunk.
  • Implemented using the sqlite-vec extension for vector storage and similarity search.
  • Not synced. Embeddings are large, model-specific, and re-derivable.
  • Schema includes the model identifier so a model change triggers a re-index rather than silent corruption.

Indexing

  • Background task following the existing OCR Service pattern: timer-based, polls ItemChange, processes in chunks, persists progress in a settings key.
  • Chunking strategy: roughly 512–1024 tokens with overlap. Tunable internally.

Embedding model

  • Joplin ships a small embedding model (~100MB, e.g. from the nomic/mxbai/bge family) bundled with the desktop app, or downloaded after installation.
  • Runtime: ONNX Runtime (onnxruntime-node), loaded in-process. No external service, daemon, or Python required.
  • The bundled model is the default. Users may switch to a cloud embedding provider via the provider abstraction; doing so triggers a re-index.

Platform scope

  • Desktop and CLI: full support.
  • Mobile: deferred. sqlite-vec packaging for iOS/Android and on-device embedding cost on mobile are separate efforts. Mobile may eventually query an existing index produced on desktop, but that is out of scope for the initial work.

3. Retrieval helpers

The shared query surface. All four target features differ mainly in what they retrieve — same machinery, different scope.

API

A single primary call:

  • search({ query, scope, relevance }) — returns matching chunks with their source note ID, the chunk text, and a similarity score.
    • query: plain text. Embedded internally using the active embedding provider.
    • scope: where to search. One of 'note' (with a note ID), 'notebook' (with a folder ID), 'tag' (with a tag ID), or 'all'. Trashed and conflict notes are excluded by default.
    • relevance: 'strict' | 'normal' | 'loose'. A preset that maps internally to model-appropriate values for the number of results returned (k) and the minimum similarity threshold.

Raw thresholds (k, minScore) are a leaky abstraction: the right values depend on the embedding model, and silently break when the model changes. Plugins calibrated against one model would produce poor results against another with no signal that anything had changed.

The relevance preset is the contract plugins write against. Joplin owns the mapping from preset to numeric values per model. When the bundled model changes, the mapping is re-tuned and plugins continue working without modification.

Default mappings

Reference defaults (subject to per-model calibration):

relevance preset k minScore (cosine)
strict 5 ~0.55
normal 10 ~0.40
loose 20 ~0.25

These are internal values and are not part of the public API contract.

Hybrid search

Retrieval may be combined internally with the existing FTS-based keyword search. This is an implementation detail; plugins still call search() with the same shape.

Mapping to features

How each target use case composes the primitives:

Feature Retrieval scope Then
Chat with note note or notebook Pass chunks as context to chat model
Fuzzy search all Show chunks directly as results
Tag suggestions all, query = note content Inspect tags of returned chunks
Semantic graph all, per note Use scores as edge weights

Chat-based features additionally pass each chunk's source note ID into the prompt so the LLM can cite sources back to the user as clickable links.

4. Privacy & cost guardrails

Enforced at the provider layer so every feature (core or plugin) inherits these checks automatically.

Requirements

  • AI features off by default. A single top-level setting plus per-feature toggles.
  • Offline by default: User must explicitly grant permission to use online features.
  • Per-provider classification as local or remote. Surfaced in the provider picker and used by the indicator.
  • Token accounting per provider, queryable by plugins and shown to users.
  • No silent enablement. Switching from a local to a remote provider requires explicit user confirmation, with clear text about what data will be sent and where.

5. MCP server

Joplin runs an optional Model Context Protocol server that exposes notes to external AI applications (Claude Desktop, ChatGPT desktop, Cursor, Zed, etc.).

Scope

The server exposes a minimal tool surface:

  • Search notes
  • Read note by ID
  • Create note
  • Update note
  • List notebooks and tags

Implementation

  • Built as a thin protocol adapter on top of the existing Data API.
  • Auth uses the same token model as the Data API.
  • Disabled by default; enabled from the same settings page as the Web Clipper.

Why it belongs in this spec

The MCP server is not required for the four target use cases, but it is the cheapest way to make Joplin a first-class participant in the broader AI tool ecosystem without building any chat UI. It also exercises the same note-access surface that internal AI features will need, so the two efforts share infrastructure.

38

I'm absolutely third on this as well.

Not to say anything about you here, but it's not even 4 hours since I looked at "correctly implemented code" and we realised there were at least three unintended effects which would likely affect the end users in negative ways.

39

Thanks for the feedback. We understand the concerns (and have addressed them many times), but for now we'd like to focus on doing it well rather than not doing it at all. So please keep the discussion focussed on the spec

40

excited to see this Laurent!

list of primitives looks right. I'll add to the chat with note feature, that using these primitives one can ask their note collection (i.e., beyond the current note) a question, and get an answer (using retrieval helpers + provider abstraction). this is what a few plugins implement (here and in other note apps).

re. retrieval helpers, if anyone would like to experiment with ways to consume such an API (and perhaps iterate over the requirements / specs from experience), Jarvis already provides a search API that is available for other plugins. for example, this search API supports both free text query, and a noteID as input for finding related chunks (on the other hand, it doesn't support scope yet).

re. privacy (related to each of the primitives), it would be good to include / exclude notebooks and notes from these services via config.

re. MCP scope, this is a good start. from my experience users often request more tools and more control over Joplin via the MCP server (e.g., creating / updating notebooks and tags). additionally, having access to retrieval helpers via the MCP protocol would be great. I think that the two main things that an MCP adds on top Joplin's great data API are related to privacy (see above) and digesting the output for the LLM to keep its context clean: generate note previews / outlines, extract specific sections or paginate over note body, search within a note, provide note metadata / stats, edit specific lines (replace existing text). I also suggest enabling / disabling tools via settings.

41

Thanks for your feedback!

list of primitives looks right. I'll add to the chat with note feature

That's a good point - I've added this use case to the spec!

this search API supports both free text query, and a noteID as input for finding related chunks

Search by note would indeed be useful - I've added it to the spec as well as mentioned the Jarvis search API as prior art for reference, and to experiment.

re. privacy (related to each of the primitives), it would be good to include / exclude notebooks and notes from these services via config.

Hmm, yes that's a good point. But then it makes me think - should we add a way to exclude notebooks, or exclude everything by default and the user explicitely include them? The reason is that if we have such feature, we are making a commitment to privacy, but we don't really succeed if someeong creates a notebook about their medical records and it's automatically indexed because they forgot to exclude it, or didn't do it fast enough.

I think that requires reflection - including/excluding notebooks would indeed be useful, preferably excluded by default, but then it means the UI/UX needs to be simple enough that it doesn't get in the way.

re. MCP scope, this is a good start. from my experience users often request more tools and more control over Joplin via the MCP server (e.g., creating / updating notebooks and tags). additionally, having access to retrieval helpers via the MCP protocol would be great. I think that the two main things that an MCP adds on top Joplin's great data API are related to privacy (see above) and digesting the output for the LLM to keep its context clean: generate note previews / outlines, extract specific sections or paginate over note body, search within a note, provide note metadata / stats, edit specific lines (replace existing text). I also suggest enabling / disabling tools via settings.

Right, a way to enable and disable tools would be quite important, and probably something we should have on first release on the MCP server. I've added it to the spec.

As for support for other tools, we can indeed add more capabilities as we go based on user feedback.

I've now added the full spec in this pull request:

42

The AI chat feature is now nearly completed:

I think it covers all that was discussed but if you notice anything missing feel free to let me know! Documentation at:

https://github.com/laurent22/joplin/blob/e4f2e8bdf43532640629451aae9ec14b26f291b8/readme/dev/spec/ai_chat.md

and PR:

https://github.com/laurent22/joplin/pull/15643