MLX LM with Tiny Agents

README.md

Setup

Install node.

Install mlx-lm and the Hugging Face MCP client:

pip install mlx-lm "huggingface_hub[mcp]"

Run

Start the LLM server:

mlx_lm.server

Run the agent:

tiny-agents run --path /path/to/this/directory

References

• Hugging Face's Tiny Agents in Python blog post
• Some more examples of agents / MCP client configs:
  • tiny-agents/tiny-agents
  • experiments-with-mcp

agent.json

{
    "model": "mlx-community/qwen3-4b-4bit-DWQ",
    "endpointUrl": "http://localhost:8080/v1",
    "servers": [
        {
            "type": "stdio",
            "config": {
                "command": "npx",
                "args": [
                    "-y",
                    "@modelcontextprotocol/server-filesystem",
                    "."
                ]
            }
        }
    ]
}

PROMPT.md

You are an agent - please keep going until the user’s query is completely resolved, before ending your turn and yielding back to the user. Only terminate your turn when you are sure that the problem is solved, or if you need more info from the user to solve the problem. If you are not sure about anything pertaining to the user’s request, use your tools to read files and gather the relevant information: do NOT guess or make up an answer. You MUST plan extensively before each function call, and reflect extensively on the outcomes of the previous function calls. DO NOT do this entire process by making function calls only, as this can impair your ability to solve the problem and think insightfully.

needs ml-explore/mlx-lm#217

Nice! Found out about this demo from https://x.com/awnihannun/status/1931755333011349831. Would be cool to contribute it to https://huggingface.co/datasets/tiny-agents/tiny-agents as an example. Wanna open a PR? :)

@Wauplin 👋 Sure thing, I'll add it once the necessary change in mlx-lm lands.

🙌

I set it up exactly as your did. Server works (tested with MCP Inspector), files are accessible, agent calls the funcion but does not return anything. I should add that it does work with the default remote model, just not with mlx-community/qwen3-4b-4bit-DWQ

Indeed you need a special fork of mlx-lm right now for this to work: ml-explore/mlx-lm#217

We will try and get it landed and into a new release within the next few days!

Just wanted to let you know @awni that we are pushing huggingface/huggingface_hub#3166 and huggingface/huggingface.js#1556 that will modify the file format expected by agent.json (and introduce a breaking change). The main change is that the "config" values are now at the root level and not under .config. The rational behind this is to be on-par with VSCode MCP format which should be more future-proof

{
    "model": "mlx-community/qwen3-4b-4bit-DWQ",
    "endpointUrl": "http://localhost:8080/v1",
    "servers": [
        {
            "type": "stdio",
            "command": "npx",
            "args": [
                "-y",
                "@modelcontextprotocol/server-filesystem",
                "."
            ]
        }
    ]
}