Plugin: Jarvis (AI assistant) [v0.9.1, 2024-11-02]

JamesWriterNarry · 2024-03-30T08:08:50.501Z

@shikuz I was just watching some videos about AI agents. Have you thought about implementing agents in Jarvis? If so, what use cases did you have in mind?

davadev · 2024-03-30T08:28:44.906Z

Agents would be a cool feature! But I think it is too big for @shikuz, as he mentioned that his time is more or less limited to maintaining Jarvis. Perhaps the simplest implementation would be to have a limited number of prompts that run sequentially over the same text.

Something like this:
Prompt 1 - "Correct grammatical errors".
Prompt 2 - "Break the text into paragraphs and subheadings".
Prompt 3 - "Create a summary“

If I am not mistaken, something like this is almost possible, as you can save prompts in Jarvis that you want to use for text editing. It would just need some automation to allow you to select prompts to run in sequence over the same text...

JamesWriterNarry · 2024-03-30T09:09:07.626Z

Well, I was thinking about something more complex in the context of "Chat with your notes".
Agents could abstract the Jarvis "command block".

The user would ask a question and the first agent would decide what to parse into the "context". If you start "chat with your notes" in an empty note, you will get a very good selection of notes related to your first question. However, for each subsequent question, the results get worse as it does a semantic search on the whole note. You can partially fix this by always adding your last question to "Context"; that way you'll only get notes related to the last question, which is mostly fine if you don't need to take the context of the conversation into account.

I think the optimal solution would be for an agent to customise the user's last question to include some keywords or some kind of summary from the chat history that would give the semantic search relevant contextual information, but only relevant to that question. In this way, the semantic search query is improved to give the most relevant results.

The next level could then be an agent that reasoned step by step what information it needed to answer the user's question, then performed multiple semantic searches with a summary of results for each query, and finally combined the summaries into one answer.

davadev · 2024-03-30T09:23:46.007Z

This is indeed an intriguing idea. I think this could solve some of the problems I face when trying to build my support chatbot! I need the chatbot to have enough contextual information (for example, what troubleshooting steps have already been taken, or what device the user is talking about...).

Adding contextual information to the query could perhaps eliminate the need to include chat history, as the context would already be included in the user's query. In this way, by treating each question as a new chat query, we could save some valuable tokens for more note chunks! (Or make using offline models with smaller context window more useful!)

davadev · 2024-03-31T15:01:52.535Z

I have tried to install Xinference, but unfortunately I can't get it to work. I managed to get rid of all the python errors, but still the process gets killed while building the wheel. A quick search on the internet suggests that hardware resources (RAM) are to blame. It seems to be much easier to run LLMs locally than to run embedding models locally.

For my private projects I stick to the English language as it is currently the most feasible way to run something locally with few resources. I stumbled across this LLM " phi 2 Q5_K_M" which seems to be the smallest english RAG capable model and am curious if anyone can evaluate how well it works for their usecase.

The default Jarvis offline embedding model has its limitations, and for my latest project I tried it didn't manage to get the right related notes, while OpenAi text-embedding-ada-002 with the same context for semantic search gets exactly the notes I need. On the plus side, I now understand the critical role of the embedding model. Maybe I can get the same result as with OpenAi ada by reformatting my notes or changing the context .

@shikuz I am thinking about how to benchmark the different offline configurations. Maybe we could add some sample notes to the repo that everyone could use to test different models (LLM/embeddings) so that we can either find out which models work best or what formatting one has to have so that even not-so-good embeddings models manage to find the relevant information. But I am currently only assuming that my problem lies in the note formatting, as to my knowledge it could also be an embeddings model lacking data in the domain of my use...

JamesWriterNarry · 2024-03-31T15:20:59.764Z

Maybe you could use Ollama to run a different embedding model. Has anyone managed to make this work with Jarvis?

„nomic-embed-text“

(Apparently I can not insert links)

davadev · 2024-04-01T05:19:45.391Z

This could be an option. I just checked and there are some new text feature extraction models on huggingface in gguf. Practically they can be run in any of these LM Studio, Ollama, Jan. Optimal would be to have the LLM and Embedding model run with the same tool to save resources. I am only aware of ollama being able to run two models at once, but the question is whether this works with the provided workaround with Litellm.

shikuz · 2024-04-02T04:16:49.376Z

Great suggestions @JamesWriterNarry, I'm very interested in the directions that agents can take us. As I mentioned earlier (thanks @davadev), due to time constraints and to the fact the next release already includes a long list of small-to-medium fixes and updates (many of them still need to be implemented), I won't be able to work on large features for the coming release, but perhaps later this year. The same goes for cool / experimental chunking methods.

In the meantime I'd like to share a pro tip that I heard recently.

Pro Tip: Better note chunking

One thing that can improve the discoverability of your chunks is to set a low (128-256 tokens) setting of Notes: Max tokens. The idea is that smaller chunks are more homogeneous in content and are therefore represented in more distinct embeddings. This way, search may be able to pinpoint more accurately where relevant information exists in your notebooks.
In order to avoid a very short context due to smaller chunks, tell Jarvis to include a few (1-2) preceding and following chunks for every chunk that it finds, using the advanced settings Notes: Number of leading blocks to add and Notes: Number of trailing blocks to add.

davadev:

I have tried to install Xinference

I'm running Xinference on 8gb and 16gb laptops, and it seems to work well. The 16gb one is currently running both mistral-instruct-v0.2 (chat) and mxbai-embed-large-v1 (embedding) and is doing alright. Mistral is supported out of the box, and mxbai I configured manually (will be happy to share the JSON file). I used the pip install command to setup Xinference, but there seem to be dockers for it as well. In fact, Mistral is so lovely (albeit a little too chatty, perhaps I should tinker with its system message) that I'm considering going completely offline (I stuck with GPT until now).

EDIT: If you are having a hard time installing Xinference, and in addition do not have a Nvidia GPU, I recommend installing docker and doing the following: (Select a /path/on/your/pc to store downloaded models)

docker pull xprobe/xinference:latest-cpu

docker run --name xinfer -d -p 9997:9997 -e XINFERENCE_HOME=/data -v /path/on/your/pc:/data xprobe/xinference:latest-cpu xinference-local -H 0.0.0.0

(For GPUs see here)

davadev:

I am thinking about how to benchmark the different offline configurations

There are standard benchmarks for models such as this one. Naturally, it's best to have a benchmark that is closer to your use case (e.g., chatting over notes). If you have ideas for Jarvis benchmarks I'll be happy to collaborate, post on github or link to a webpage from the repo.

davadev:

while OpenAi text-embedding-ada-002 with the same context for semantic search gets exactly the notes I need

There are new recent embedding models by OpenAI, and they're supposed to be even better. I'll add them to next release, but you can start using them via the custom model setting. Still, I think that there are offline models that perform better (such as mxbai-embed-large-v1) on the benchmarks that I linked above.

davadev · 2024-04-04T06:59:10.648Z

@shikuz Thanks for the chunking tip. This actually makes sense. Is there a way to add not only nearby chunks, but perhaps all chunks in the same note?

Regarding chunking, I would like to share a few thoughts from my experiments.

Depending on your use case, you may face the problem that you need certain notes as a whole (e.g. a checklist or other content where integrity is very critical), for this use case it might actually be practical to have large chunks and then manually ensure that your notes are not longer than the chunk size. (That's why I'm grateful you're considering the approximate token count feature in notes)

EDIT: Ok I think this above is not accurate as I realised that even with a large chunk size I get small chunks due to frequent use of subheadings. To have everything in one chunk would require me to avoid further subheadings, I guess...

Another solution to the lack of context caused by chunking might be if Jarvis supports RAPTOR RAG in the future.
Another workaround I experimented with is providing metadata through subheadings. As I remember you once mentioned that the embedding model stores the subheadings that lead to the chunks, I use H1 subheading to include a keyword that is important for the context of each chunk and that the chunks themselves may not contain. I have yet to try how tags affect this behaviour, as I am a bit worried that they might not be as "noticeable" to the LLM as H1 subheadings. I think the use of metadata will be very important to further sort out notes that are semantically similar but do not fulfil a specific requirement. My question is whether I can achieve this selection of notes containing only some tags by using the "Search" command in the Jarvis block. (Once again, thanks for the decision to have a preview window, as this will help me with further refinement.)
If you are like me, your notes are not the same in all your notebooks. So it makes sense to have different chunk settings for each notebook. Would it be possible in the future to change the chunks settings on a notebook by notebook basis? By the way, this might be a bit similar to my earlier request to have different profiles in Jarvis. I just wanted to let you know that I have found a workable workaround for the profiles by simply using a different Joplin profile and importing the notes I want to experiment with. For testing this is great, but for day to day use it would be nice to be able to change the chunk settings on a notebook basis as I often change these depending on my use case and it means rebuilding the database each time.

Delfi_R · 2024-04-09T11:48:39.332Z

I've read joplin-plugin-jarvis/GUIDE.md at master · alondmnt/joplin-plugin-jarvis · GitHub
I run Ollama and OpenHui on my computer, so I have a ollama accessible at http://host.docker.internal:11434 or by another terminal app

I run Lm Studio too

But I don't know how to enter the needed values on Jarvis configuration

davadev · 2024-04-09T17:52:16.272Z

The settings in the guide are all available in the Jarvis plugin settings. I agree that sometimes it is a bit difficult to find them, but

"Model: OpenAI API Key"
"Chat: Model"
...
are all there, and only a few are hidden in the advanced options.

Can you be a bit more specific? Do you have problems with all the settings or just some?

PS: A tip for easier navigation the prefix like „chat“, „model“ … is really important as there are many settings that look quite similar

shikuz · 2024-05-04T06:02:01.213Z

v0.8.0

It's been a while.

download file

new features

revamped Edit with Jarvis command (@jakubjezek001) (screenshot below)
Auto-complete with Jarvis command to autocomplete any text at the current cursor position
scroll to line of a found note chunk from the panel
chat context preview dialog (screenshot below)
token counter command
display note similarity score in panel

new models

OpenAI
- replace text-davinci (deprecated) models with gpt-3.5-turbo-instruct (@jakubjezek001)
- 3rd generation embedding / notes models text-embedding-3-small and text-embedding-3-large
- chat model gpt-4-turbo: an efficient, strong model with a context window of 128K tokens
Google AI
- deprecated PaLM
- chat models gemini-1-pro and gemini-1.5-pro (a strong model with a context window of 1M tokens!)
- embedding / notes models embedding-001 and text-embedding-004

new settings

Notes: Context tokens: the number of context tokens to extract from notes in "Chat with your notes" (previously used Chat: Memory tokens)
Notes: Context history: the number of user prompts to base notes context on for "Chat with your notes"
Notes: Custom prompt: the prompt (or additional instructions) to use for generating "Chat with your notes" responses
Notes: Parallel jobs: the number of parallel jobs to use for calculating text embeddings

chat improvements

chat display format (screenshot below)
chat with notes default prompt
chat parsing

general improvements

CodeMirror 6 / beta editor support
load USE from cache instead of re-downloading every time
faster model test on startup / model switch
various fixes

ux

new standard dialog style

Thanks @davadev for the help testing this release.

Screenshot 1: New edit dialog

Screenshot 2: New chat context preview

Screenshot 3: New chat display format

runrunrirun · 2024-05-05T20:02:48.646Z

Jarvis is awesome, thanks!

Apparently PaLM keys work with Gemini, so I picked the 1.5 model even though I don't have a paid account, and it still works Don't know if they just silently fall back to the 1.0 model or maybe they decided on a grace period, the free preview access to 1.5 was supposed to end May 1st.

shikuz · 2024-05-06T09:44:41.700Z

As far as I know Google extended the period until billing starts to May 14.

erikprzekop · 2024-05-09T14:47:41.939Z

Thank you so much for pulling this together! I'm just exploring some of the basics right now to get a feel for how it works. I have a lot of notes (including thousands that I didn't import to Jopin from Evernote), and having something that can help me search them & use them for context should prove very useful.

Is there a limitation for the number it can process? The "related notes" tab shows this message, and seems to be stuck at 460. I'm using a paid Google Gemini account's API key. Maybe I need to compile from source & look into the logs?

image1150×254 12.2 KB

shikuz · 2024-05-09T15:39:44.527Z

Hi @erikprzekop, I suspect that there are notes that can break Jarvis (maybe some html notes, maybe imported ones from Evernote, I'm just guessing here because it never happened to me personally). It has happened once to a user once, and it seems that we were able to isolate and skip the notes that caused the issue. I never had an example that I could debug though. In any case, the process outlined in the link may help you identify and exclude the notes that get stuck. We can continue the discussion there.

shikuz · 2024-05-15T00:11:23.377Z

v0.8.3-v0.8.4

improve: exclude notes in trash from db
improve: OpenAI model updates
- added gpt-4o (latest model)
- deprecated legacy gpt-3.5-turbo-16k
  - gpt-3.5-turbo points to a newer version of this model
- deprecated legacy gpt-4
- all legacy models are still accessible via the openai-custom model setting
- improved model descriptions with tokens / price cateogry

Midnight · 2024-05-19T22:37:27.555Z

Thank you for the excellent plugin, I love it! I followed the guide and there is a more straight forward connection to Ollama, not requiring LiteLLM. Ollama servers openai compatible API already. Tested and working.

Offline chat model with ollama
Install ollama
Pick a LLM model to use from the ollama library and run ollama run MODELNAME (e.g., ollama run llama3:latest) in a terminal
Setting Advanced Value
Model: OpenAI API Key No Something, anything
Chat: Model No (online) OpenAI or compatible: custom model
Chat: Timeout (sec) Yes 600
Chat: OpenAI (or compatible) custom model ID Yes MODELNAME
Chat: Custom model is a conversation model Yes Yes
Chat: Custom model API endpoint Yes http:// ollama-IP :11434/chat/completions

shikuz · 2024-05-20T01:40:18.515Z

Thanks for the reminder @Midnight! I was vaguely aware that I had to update the guide and forgot to.

Updated it just now. Also improved Xinference's instructions, which now include a setup for both chat and notes / embedding models.

shikuz · 2024-06-04T02:02:32.118Z

v0.8.5

new: separate settings sections for chat, related notes, annotations and research
fix: set default values for API keys
- this is a workaround that ensures that keys are saved securely to your keychain (where available)
changed default settings
- context tokens: increased to 2048
- annotation: tags method changed to existing tags