Hybrid AI agents: 2B local UI-grounding specialists (UI-TARS, ShowUI, browserground) replace frontier VLMs in browser-use loops
Hybrid AI agents: 2B local UI-grounding specialists (UI-TARS, ShowUI, browserground) replace frontier VLMs in browser-use loops
Hybrid SLM+LLM pattern for production AI agents. A 2B specialist handles click-grounding; the frontier LLM handles reasoning. Open-source reference implementation, Apache-2.0.
Last tested: May 2026. See Changelog at the bottom.
If this saves you setup time on hybrid agent architectures, follow @renezander030 — practical notes on production LLM ops, agent design, and self-hosted AI.
Companion repo with model + npm CLI: github.com/renezander030/browserground
| You want to... | Use |
|---|---|
| Cut browser-agent vision token cost | Split grounding into a local 2B specialist VLM |
Replace per-screenshot GPT-4V / Claude Vision / Gemini calls in browser-use |
Route through a 2B specialist (UI-TARS, ShowUI, or browserground) — strict JSON bbox, $0/call, local |
| Bench a UI-grounding model | ScreenSpot-v2 (standard click-grounding benchmark) |
| Train your own browser-grounding LoRA | LoRA rank 16, ~13k UI screenshots, ~$4 on a rented L40S for 3 hours |
Drop-in for browser-use, Skyvern, Claude Computer Use, OpenAI Operator |
Same JSON-bbox interface; route screenshots through the specialist first |
A real browser-agent run hits the vision endpoint 20-50 times per task. At $0.01-$0.05 per multimodal call, that's $0.20-$2.50 per task before any reasoning. Most of those calls are not reasoning. They're parser questions ("where is the Submit button?").
Rule of thumb: every time your agent loop calls a frontier VLM for a structured-output question on a screenshot, you are paying frontier rates for a parser. Replace it with a 2B specialist.
npm install -g browserground
browserground parse screenshot.png --target "Submit button"
# {"bbox_2d": [344, 612, 478, 658]}100% strict-JSON output. No markdown fences, no commentary. Feeds directly into your agent's click(x, y) call.
Drop-in adapters:
- Claude Code:
/install-plugin renezander030/browserground - OpenAI Codex CLI:
codex add-extension renezander030/browserground browser-use/ Skyvern: Python subprocess wrapper, 5 lines (below)
A general frontier LLM is the wrong tool for UI grounding. It's like asking a senior engineer to count rows in a CSV: they can, but you're burning expensive cycles on something a wc -l can do instantly.
The right shape for production AI agents in 2026 is hybrid:
- Specialist local models handle dedicated, narrow tasks (UI grounding, OCR, embedding generation, classification, structured-output extraction) where a 2B-parameter purpose-built model can match or beat the cloud giant.
- Cloud frontier LLMs handle what they're uniquely good at: planning, multi-step reasoning, open-ended dialogue, complex tool orchestration.
This is the compound AI pattern, articulated by Berkeley AI researchers in 2024 and now mainstream in production stacks. Different models for different modules. Routing layer between them.
The Apple-Silicon default for generic UI grounding is mlx-community/UI-TARS-1.5-7B-4bit — UI-TARS-1.5-7B (~94% on ScreenSpot-v2) converted to MLX 4-bit, ~4 GB on disk, ~5–6 GB RAM, drops into mlx-vlm directly. If you're on a ≥16 GB Mac and just want the best generic UI grounder running locally, install that and move on. The 2B-specialist tier (UI-TARS-2B-SFT 89.5%, ShowUI-2B 75.5%) is the alternate budget tier; the frontier (GPT-4o ~18% on v2, GPT-5.4 85.4% on the harder Pro) isn't competitive on this exact task. The interesting comparison isn't 2B-vs-frontier — it's which local specialist for which stack, with UI-TARS-1.5-7B-MLX as the obvious default and smaller specialists for niche slots.
| Model | Params | ScreenSpot-v2 overall | Apple Silicon native | RAM @ 4-bit |
|---|---|---|---|---|
| UI-TARS-1.5-7B-MLX (mlx-community) | 7 B | ~94% | ✅ MLX 4-bit (this build) | ~5–6 GB |
| UI-TARS-2B-SFT (ByteDance) | 2 B | 89.5% | ❌ server-class (no MLX build) | ~2 GB if converted |
| ShowUI-2B (showlab) | 2 B | 75.5% | ❌ server-class | ~2 GB if converted |
| browserground v0.3 (this repo) | 2 B | 60.0% | ✅ MLX 4-bit + Ollama + npm + pip + strict JSON | ~2 GB |
| SeeClick (Qwen-VL-Chat) | 9.6 B | 55.1% | ❌ | n/a |
| GPT-4o (cloud, general vision) | — | 18.3% | n/a | n/a |
| Qwen3-VL-2B (zero-shot, no UI fine-tune) | 2 B | 6.3% | ✅ | ~2 GB |
Cost per inference: $0 vs $0.01-$0.05. Latency: ~1-2s local vs 800ms-2s round-trip plus 200-500ms of compute. Tokens billed back to your cloud LLM: ~40 text tokens of bbox JSON vs 1500+ multimodal tokens for a raw screenshot. Data leaving your machine: none vs every screenshot.
The 2B specialist isn't smarter than GPT-4V at general reasoning. It was trained on a narrow question — "where is this UI element?" — and now answers that one question better, faster, and free. Use it for grounding; keep the frontier LLM for the actual reasoning step.
For generic UI grounding on Apple Silicon, you don't pick browserground. You pick mlx-community/UI-TARS-1.5-7B-4bit. ~94% on ScreenSpot-v2, MLX-native, ~6 GB RAM. ByteDance research-lab compute that you cannot reproduce on a budget. Honest answer first.
browserground earns the airtime in two narrow cases.
Case A: you want a custom UI grounder for your product, not generic UIs.
UI-TARS is a finished model — closed training pipeline, proprietary data, hard to fine-tune for your specific app. browserground is the recipe, open end-to-end:
- base model:
Qwen/Qwen3-VL-2B-Instruct(public, current-gen) - training scripts: public LoRA-rank-32 recipe (~$2.20 compute on a single L40S)
- data mix: 26k records from OS-Atlas + wave-ui, swappable for your screenshots
- output schema: strict JSON
{"bbox_2d": [...]}, 100% parseable
If your agent operates inside one specific app (your SaaS dashboard, your in-house tool, your customer portal), training a domain-specific grounder on your own screenshots is the right play. A 60-point generic baseline becomes 85–95% on a narrow domain because the test distribution finally matches what you trained on. browserground is the template for that workflow.
Case B: you're squeezing into a tight RAM budget.
| UI-TARS-1.5-7B-MLX | browserground 4-bit MLX | |
|---|---|---|
| Disk | ~4 GB | ~1 GB |
| RAM at inference | ~5–6 GB | ~2 GB |
That 4 GB delta matters on an 8 GB Mac mini, in a multi-model agent stack (where you're already running a 7B planner + OCR + embeddings in the same memory pool), or in batch workloads where you want to keep many models hot. browserground is the smallest viable click-grounder I know of.
A direct head-to-head benchmark of browserground vs UI-TARS-1.5-7B-MLX on the same Apple Silicon hardware is forthcoming.
import json, subprocess
def ground(screenshot_path: str, target: str) -> tuple[int, int, int, int]:
"""Return (x1, y1, x2, y2) bbox for the UI element matching `target`."""
out = subprocess.run(
["browserground", "parse", screenshot_path, "--target", target],
capture_output=True, text=True, check=True,
).stdout
return tuple(json.loads(out)["bbox_2d"])Hook this ahead of the frontier-model call in your agent loop. The frontier model now sees structured JSON, not pixels.
UI grounding is one example. The hybrid pattern applies to most narrow tasks in AI agent loops:
- OCR — Tesseract / docTR / TrOCR locally instead of GPT-4V OCR
- Embedding —
nomic-embed-text-v2orbge-largelocally instead of OpenAItext-embedding-3-large - Reranking —
mxbai-reranklocally instead of Cohere reranker - Classification — fine-tuned
distilbertlocally instead of GPT-4o classification calls - Structured extraction from PDFs — Marker / MinerU / Docling locally instead of GPT-4V on every page
- Browser navigation — a Qwen2.5-VL or Qwen3-VL fine-tune locally instead of GPT-4V per screenshot
Each one is a 2-7B-parameter model. Each costs $3-$50 of compute to train your own. Each cuts cloud-LLM billing by 50-95% for the task it owns. Together they make your cloud-LLM bill match what you actually use it for — reasoning, not commodity vision/text labor.
- Sending the whole screenshot to the frontier model on every step. Use a specialist for grounding first; pass JSON onwards.
- OCR-only without spatial grounding. Tesseract gives you text but not "where the Submit button is" with a bbox.
- Routing a structured-output question through a multimodal API endpoint. It's strapping a frontier model to a job that needs a 2B parser.
- Training your own UI-grounding model from scratch. LoRA on top of Qwen3-VL-2B or Qwen2-VL — the recipe takes 3 hours on $4 of compute, dataset and eval slice already published.
Part of Production AI Automation Notes — a running set of gists and repos on patterns that survive contact with users.
Other entries:
- CLAUDE.md — 10 rules, edit-time and runtime — Claude Code guardrails that hold up in production
- Production AI Automation Notes #1: Agent Approval Gates —
draft → validate → approve → dispatch → auditfor destructive agent actions - Context7 v2 — enterprise GraphQL MCP server pattern
- Claude Code with local LLMs — ANTHROPIC_BASE_URL, Ollama, LM Studio, vLLM, LiteLLM
- Minimal pipeline engine (Go) — YAML steps with budgets and HITL channels
If this gist saved you cloud-LLM cost or setup time, follow @renezander030.
- browserground (this post's reference implementation):
- Compound AI systems (Zaharia et al, Berkeley, 2024): bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/
- OS-Atlas paper (training data + benchmarks for UI grounding): arxiv.org/abs/2410.23218
- ScreenSpot-v2 benchmark — described in the UI-TARS and OS-Atlas papers
If you train or deploy a specialist for one of the narrow tasks above, drop a comment with:
- Machine + RAM (e.g. MacBook Air M3 16 GB, Linux box with RTX 4090)
- Backend (MLX / vLLM / llama.cpp / ONNX / other)
- Model + quant (e.g. Qwen3-VL-2B Q4_K_M)
- What worked / what failed (eval slice, JSON validity rate, latency, click accuracy)
ScreenSpot-v2 results across hardware are especially useful — let's build a community matrix.
- v0.3 — packaging release: three distribution channels for the same Tier-2 LoRA
- MLX 4-bit (1.8 GB) at huggingface.co/renezander030/browserground-mlx — Apple Silicon, ~1.5s/call
- GGUF Q4_K_M + f16 mmproj at huggingface.co/renezander030/browserground-gguf — Ollama, llama.cpp
- Ollama:
ollama run renezander030/browserground - PyPI:
pip install "browserground[mlx]"or[transformers]
- CLI additions: batch mode (
--targets), confidence (--confidence), diverse alternates (--alternatives N), HTTP REST daemon (serve --http :8401), eval CLI (browserground eval ...) - Drop-in adapters for
browser-use(Controller action) andSkyvern(local-first with cloud fallback)
- Tier-2 LoRA on Qwen3-VL-2B: 60.0% on ScreenSpot-v2 (300 items, 100/split)
- Beats SeeClick (9.6B) at 4.8x smaller; 100% strict-JSON format compliance
- 26k mixed-domain training set (macOS + Android + UIBert + web), 1 epoch, ~$2.20 compute
- Positioned vs UI-TARS-2B-SFT — newer Qwen3-VL base, strict JSON, browser-focus
- Initial publish
- Reference implementation
browsergroundv0.1: 45.3% on ScreenSpot-v2 (vs GPT-4o 18.3%) - Apache-2.0, MLX-native, 2 GB unified memory, ~1.8s/inference on MacBook Air
- Adapters: npm CLI shipping; Claude Code plugin + Codex CLI extension entry points
- Added section 3 "When to pick browserground over UI-TARS / ShowUI" — honest positioning at v0.1, the case for MLX / Qwen3-VL / reproducible LoRA / strict JSON / agent-stack adapters
Tags: hybrid-ai, compound-ai, ai-agent, browser-agent, ui-grounding, gpt-4v-alternative, local-llm, specialist-model, qwen3-vl, ScreenSpot-v2, browser-use, skyvern, claude-computer-use, openai-operator, mlx, apple-silicon