I benchmarked every model in OpenRouter's Discounted Models collection — plus a full-price frontier model and a local 27B — to answer one question: which of these should drive an agent that calls tools and verifies its work, rather than guessing?
Short answer: the model with the smallest discount in the collection won, and a $15/M frontier model couldn't beat it by more than noise.
Scope, stated up front. Everything here is deterministic-verifier work — did it call get_contacts, did the cron job land in place, is the arithmetic right. That excludes everything where the answer is a judgment call: prose quality, voice, taste. A low score here means "don't put it in an agent loop." It says nothing about whether a model writes well. See What this doesn't measure.
Harness. BenchLocal 0.3.0, temperature 0, 1 run per test, driven through its Agent API. Three valid packs:
| Pack | What it tests | Cells |
|---|---|---|
| ToolCall-15 | tool-call loops, error recovery, empty results | 22 models × 15 |
| HermesAgent-20 | a real Hermes agent in Docker — cron, skills, memory, browser | 20 scenarios, graded on outcome + native tool use + safety |
| ReasonMath-15 | practical reasoning and arithmetic | 15 scenarios |
Total spend for everything below: $1.32, of which a single model accounted for $0.46.
The only model strong on both axes: best agentic score in the field (18/20 on HermesAgent-20, outcome 920/1000) and second-best reasoner (12/15). It also has the highest safety score (360/400).
It is the least discounted model in the collection — 8% off, where the collection runs up to 90% — and it beat everything cheaper, everything more expensive, and a frontier model at 28× the price.
The value floor. 15/20 on HermesAgent-20 with 570/600 native tool use — identical to MiniMax-M2's 15/20 at 3.6× lower output cost. It isn't in the discounted collection; it's full price and still beats most of the discounts on value.
Caveats worth knowing: it's text-only (no image input), and in day-to-day use it's a strong mechanical tool-caller with little stylistic judgment. Use it as an agent engine, not a writer.
If your failure mode is models that guess instead of checking, this is the pick: 600/600 on native tool use — a perfect score, it reached for its real tools on all 20 scenarios. 16/20 outcomes.
The trade-off is sharp: 7/15 on ReasonMath, the weakest reasoner tested. Great hands, weaker head.
The GLM tier is dead money here. GLM-5.1 and GLM-5 both scored a clean 15/15 on ToolCall-15 — then cost 4–25× more per output token than the MiniMax pair and ran slower. On this workload they buy nothing.
Laguna-XS-2.1 is a trap for ratio-chasers. It's the cheapest ($0.12/M out) and fastest (2.9s median) thing in the collection, and it wins raw points-per-dollar by 2×. But it scored 11/20 on real agentic work. If you rank purely by score-per-dollar it comes first; I wouldn't ship it as an agent.
| Model | ToolCall-15 | HermesAgent-20 | ReasonMath-15 |
|---|---|---|---|
| Hy3 | 14 + 1 partial | 18/20 | 12/15 |
| MiniMax-M2.7 | 15/15 | 16/20 | 7/15 |
| MiniMax-M2 | 15/15 | 15/20 | 8/15 |
| DeepSeek-V4-Flash | — | 15/20 | — |
| Laguna-XS-2.1 | 14 + 1 partial | 11/20 | 11/15 |
| Ling-2.6-1T | 14 + 1 partial | 8/20 | 13/15 |
Ling-2.6-1T is the sharpest reversal: worst at agentic work (8/20) and best at reasoning (13/15) — it beat every model that crushed it on Hermes. MiniMax-M2.7 is its exact mirror: perfect tool use, weakest reasoning.
Look at how Ling failed and it stops looking like incompetence. Asked to "let Sarah know the meeting moved to 3pm," it replied:
1. What is Sarah's email address? (I can search for her in contacts if you'd like)
2. What meeting is this about? (subject/topic)
In an agent harness that's a failure — the job is to act. In a conversation, asking before emailing someone on your behalf is arguably the better instinct. Same behaviour, opposite verdict, depending entirely on what you're building.
Nine of 22 models cleared ToolCall-15 with a perfect or near-perfect row. Only one of those was still standing after HermesAgent-20.
Directionally yes, but it's noisy and I'd hold it loosely.
- Eliminated on a hard failure: mean 57.3% off (n=7)
- Survived the screen: mean 42.3% off (n=9)
- Of the 6 deepest discounts, 4 were eliminated; of the 6 shallowest, 4 survived
On the hard agentic pack the extremes line up exactly — 8% off scored 18/20, 75% off scored 8/20 — but the middle refuses to cooperate: MiniMax-M2.7 at 60% off (16/20) beat MiniMax-M2 at 15% off (15/20).
A plausible mechanism is that discount depth proxies how badly a provider needs to buy usage — spare capacity, an unproven model, weak organic demand — which is inversely related to the demand that tracks quality. But that's a story told after seeing the data, n=16, with clean counterexamples. Suggestive, not a rule.
No, not measurably. Kimi K3 at $3/$15 per M scored 11/12 effective (92%) against Hy3's 18/20 (90%) — a gap well inside noise on a 12-cell sample, for 28× the output price.
It cost $0.455 for 12 scenarios (~$0.038/cell), versus ~$0.0002/cell for the cheap field. That's ~190× per cell. I stopped it early because the answer was already clear.
| Model | Inputs | $/M out |
|---|---|---|
| MiMo-V2.5 | text + audio + image + video | 0.28 |
| DeepSeek-V4-Flash | text | 0.28 |
| Kimi-K3 | text + image | 15 |
| GPT-5.6-Luna | file + image + text | 3 |
| Hy3, Laguna, Ling, MiniMax ×2, GLM-5.2 | text only | — |
Every model in the top 5 is text-only. If your agent needs to see anything, the interesting entry is MiMo-V2.5 — full multimodal at the same $0.28/M output as DeepSeek-V4-Flash, 20% off — which I have no data on because I cancelled its run. That's the biggest open gap here.
A local 27B (Qwen3.6-27B, MLX, 4-bit-ish via MTPLX on an M5) managed roughly 10 of 14 valid cells on HermesAgent-20 — competitive with the hosted field on quality. But at a 128s median per scenario against 30–50s hosted, and 3 of 17 cells died at the 280s request ceiling. Locally hosted quality is there; throughput is what costs you scenarios.
Deterministic verifiers can only grade things with a right answer, so this benchmark is blind to:
- Taste, voice, prose quality. Nothing here distinguishes a good writer from a bad one.
- Vision, for 5 of the 6 models ranked — they can't accept images at all.
- Computer use. The only CUA-shaped pack available (FormSight — vision-guided form filling with coordinate clicking) is UI-only and can't be automated, and has an open bug: BenchLocal#22 — "Most VLMs normalize coordinates to 0~1000, but FormSight does not." Any score from it needs that checked first.
- Long context. Every scenario here is a few thousand tokens; the 200k–1M windows go untested.
If you care about style, don't infer it from this. Run your own prompts blind and read the outputs.
Every one of these produced output that looked exactly like a bad model. This is the part I'd actually want a reader to take away.
1. A revoked API key passes an availability check. BenchLocal reported all 22 models online against a dead key, because OpenRouter's GET /models needs no auth. Every scenario then failed in ~2s with HTTP 401. Filed as BenchLocal#24. The only real check is one chat/completions call.
2. A 429 mid-tool-loop scores correct work as zero. Under 22-way concurrency, one model completed a search → read → email chain perfectly, then a later call was throttled, and the cell recorded score: 0, summary: "Provider returned error". Indistinguishable from a wrong answer.
3. Cancelling a run writes cancellation into result cells. After a stop, in-flight scenarios save as status: fail, score: 0 with error=Run cancelled by user. in the raw log. The tell is suspiciously uniform durations — a whole model row at exactly 11s.
4. My own timeout invalidated an entire 200-cell pack. I set request_timeout_seconds: 180 for CLI-40; 175 of 200 cells came back with error=The model did not complete the answer within the configured request timeout — every one at exactly 180s. The "results" (4/40, 3/40, 0/40) measured my config, not the models. Dropped from this guide.
5. A grader demanded a numeral and failed models that spelled the number out. HermesAgent-20's HA-02 checks that three facts got stored, with:
/(?:^|[^0-9])2(?:[^0-9]|$)/.test(memoryContent) && /canary/i.test(memoryContent)The scenario prompt is "EU deploys two canary instances before rollout." A model that stores the fact verbatim has no numeral 2 and fails. Two models lost a scenario to this. Worse, because it searches the whole document for 2 and for canary independently, a model that drops the count entirely but leaves a filler note numbered 2 passes.
Fixed by matching per entry and accepting either form:
entries.some((e) => /canary/i.test(e) && /\b(?:2|two)\b/i.test(e))Validated against the six real MEMORY.md files: old matcher wrong on 2/6, new on 0/6, and the adversarial "count dropped" case flips from pass to fail.
The general rule that covers 1, 2 and 3: a provider or harness error should never produce a scored result. errorType: provider_error and cancellations should abort or be excluded, never written into a score cell as a zero.
# every real check is one completion — a model listing proves nothing
curl -s https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d '{"model":"tencent/hy3","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'Notes for anyone driving BenchLocal's Agent API: provider/model payloads are snake_case (base_url, api_key), tab creation returns the whole workspace state with the new tab as activeTabId, and history records are written at run start with a placeholder completedAt — so a live run reads as "completed with 0 results" if you poll mid-flight. Results carry no token counts, and /api/v1/activity needs a management key, so cost-per-point has to come from before/after key checkpoints.
Single run per cell at temp 0. The top models are separated by 1–2 cells, which is inside the noise — treat these as tiers, not a ranking. CLI-40 was discarded. Four GPT-5.6 variants and both MiMo variants are untested because I cancelled their runs. Kimi K3 has 12 cells, not 20. Prices are list, not measured from token counts.