Separate chat for interaction with an LLM using tools in the context of the Robrix app. Note that while this can appear as a chat history to the user, it should not be constructed as such, in order to avoid prompt injection pitfalls in the context of LLM tool use.
Features:
- connect with an LLM(local or remote).
- LLM tool use(skills)
We need a way to send requests and receve responses from an LLM.
For a local LLM, this is often done with an HTTP interface to something wrapping a lower-level inference engine. See Ollama(wrapping llama.cpp).
For remote, usually uses something OpenAI compatible
A solved problem.
This is the part that turns the LLM into an agent that can do stuff within the context of the app.
The basic idea is surprisingly simple, for example the below defines two skills:
- "Respond with a friendly greeting"
- "Ask current time"
You are a user assistant in the context of Robrix: a multi-platform Matrix chat client written in Rust,
using the Makepad UI toolkit and the Robius app dev framework.
Whenever the user sends you a request,
you can choose one or more of the below action(s) to perform in response:
- Respond with a friendly greeting.
- appropriate when the user doesn't seem to request anything specific.
- to perform this action, you need to know the current time first,
so as to give the appropriate greeting.
- format:
```
{
"action": "FRIENDLY_GREETING",
"value": "{insert response}"
}
```
- The `value` in your response will be sent to the user.
- Ask for the current time
- appropriate as a first step towards performing another actin requiring the current time.
- format:
```
{
"action": "CURRENT_TIME",
"value": "{insert name of the action for which this is required}"
}
```
- You will be provided with the current time, as well as the value you provided,
and you should use that information to perform whatever next step you had in mind.
The current user request is {user_request}.
Choose an action to perform, and respond with a JSON object in the appropriate format.
Below is a screenshot of a manual test run for the above setup; as you can see, it works(even with typos and inconsistent interpolation on my part).
This can be expanded to cover any kind of in-app workflow.
When providing data to the LLM to perform an action--like the current time in the example above--it is important to take prompt injection risk into account. This can be mitigated by designing a kind of routing system around the actions, and by managing conversation with the LLM around such routes.
This is not a solved problem.
As an example, for a "summarize" skills, where "unread messages" is the data to be summarized, when sending the unread messages to the LLM, this should be done in a new conversational context, where tool call is not available, so that the only thing the LLM can do with the data is to summarize and return a summary and show the resulting summary to the user only(so it can be shown as part of the chat history, but not actually be made available as context to the LLM in subsequent requests). This prevents an hostile prompt within unread messages from triggering tool use(but still allows it to corrupt the summary).
As you can imagine, the above setup can get somewhat complicated quickly, and it is probably best to end-up generilazing it by allowing the LLM to write a script to be executed in a sandbox with specific APIs made available. For example, the above workflow could be simplified by having the LLM just write a script that would use an api to request the current time, send the follow-up request to the LLM, and then output a friendly greeting.
Note: the "just write a script" approach comes with the downside of requiring a high-end model running in the cloud, whereas the "step by step" approach where the agent scaffolding simulates a scripting environement is good for less capable local models.
This brings us to the second idea, which is a general Web agent.
The Web is the sandboxed scripting environment by excellence, and so it seems uniquely positioned to run Agent workflows. So we can try to re-imagine the above workflow from the context of the Web.
Similar to a Web app manifest, a web app could use a link with rel=agent_skills, which would define a Js API that an agent could use to perform various actions in the contet of the app.
The UA could be responsible for running the agent and managing converstation context and history. The UA could also provide additional "native" skills to the agent, for example to use the local file system.
If the user is browsing an app which provided a set of agent skills, if the user makes a request to the agent, the description of the skills will be included in the context, and the agent can then write a JS script to use those skills.
The Js script written by the agent could be run in a kind of auxiliary context, where a bridge to the skills defined and implemented by the web app is available(using something like opener.postmessage).
The UA could also start its own such auxiliary context, which aren't linked to any running web app, that would allow the agent to run workflows requested by the user independently of any running web app.