Skip to content

Instantly share code, notes, and snippets.

@elix22

elix22/local-ai-models-guide.md

Last active June 29, 2026 07:31
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save elix22/8604ec14d65c86fa2c1c468710963bc8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save elix22/8604ec14d65c86fa2c1c468710963bc8 to your computer and use it in GitHub Desktop.
Download ZIP
Local AI Models for C#/C/C++ Dev — MacBook Pro M5 Max (128GB)
Raw
local-ai-models-guide.md

Local AI Models for C#/C/C++ Development on a MacBook Pro M5 Max (128 GB)

A practical, opinionated setup guide tailored to:

  • Hardware: MacBook Pro M5 Max, 128 GB unified memory (~614 GB/s bandwidth on the 40‑core GPU part, a Neural Accelerator in every GPU core — this is genuinely a strong local‑LLM machine).
  • Languages: C# (primary) + C/C++ native libraries consumed by the C# framework.
  • Editors/agents you already have: VS Code + GitHub Copilot, Claude Code (Max ×20), opencode, Ollama, LM Studio, llama.cpp, MLX.
  • Extra requirements: some models must see images (UI failure screenshots); some must reach the internet (fetch packages, read docs).

Bottom line up front: No local model matches Claude (via your Claude Code Max plan) for the hardest reasoning/agentic work. The win from local models is privacy, zero marginal cost, offline capability, and instant inline completion. Use local models as your default daily drivers and keep Claude Code as the escalation tier. The sections below tell you exactly which model + which tool for each job.

🆕 2026 update (the headline change): The Qwen3.6 family (released April 2026, Apache 2.0) is now the recommended local coder. It's a generation ahead of Qwen3‑Coder, and — crucially — it's natively multimodal, so the same model that writes your C#/C/C++ can also read your UI failure screenshots. That collapses "coder" and "vision model" into one. Two variants matter for you: Qwen3.6‑35B‑A3B (fast MoE, your new daily driver) and Qwen3.6‑27B (dense, higher quality, slower). Details in §3.2–3.3 and §3.5.

0. TL;DR — the recommended stack at a glance

Use case Model Quant ~Size Run with Front‑end
Inline autocomplete (FIM) Qwen2.5‑Coder‑7B (or Codestral) Q5/Q6 ~6–9 GB llama.cpp (llama-server) or Ollama llama.vscode or Continue
Daily coding chat/edit (C#/C/C++) Qwen3.6‑35B‑A3B (multimodal) 8‑bit ~35 GB MLX / LM Studio Continue (VS Code) / opencode
Agentic coding (multi‑file, terminal, tools) Qwen3.6‑35B‑A3B (fast) or Qwen3.6‑27B (smarter) 4–8‑bit ~18–35 GB MLX / LM Studio opencode
Planning / architecture / reasoning gpt‑oss‑120b (great default) or Qwen3.6‑27B (thinking) MXFP4 / 8‑bit ~63 / ~27 GB MLX / LM Studio opencode / LM Studio chat
Max‑quality local "thinking" Qwen3‑235B‑A22B‑Thinking‑2507 3‑bit ~95 GB MLX / LM Studio LM Studio chat
Vision — UI screenshots Qwen3.6‑35B‑A3B doubles for this; Qwen3‑VL‑32B if you want a dedicated VLM 4–8‑bit ~9–35 GB mlx‑vlm / LM Studio LM Studio chat / Continue
Internet‑connected (any of the above) Any tool‑calling model above — — + MCP web‑search/fetch opencode / LM Studio / Claude Code
Hardest tasks / ceiling Claude (Opus/Sonnet) — cloud Claude Code terminal / VS Code

If you only set up three things, set up:

  1. llama.cpp FIM autocomplete (Qwen2.5‑Coder‑7B) → instant inline completions in VS Code.
  2. One server for Qwen3.6‑35B‑A3B (MLX or LM Studio) → chat, edits, agentic, and screenshot debugging from a single multimodal model.
  3. opencode pointed at that server → local agentic coding, with an MCP web‑search server for internet access.

Everything else is refinement.

About LM Studio: it's the friendliest GUI and a fine on‑ramp, but it's Electron and carries ~0.5–1.4 GB RAM + up to ~1.2 GB GPU‑VRAM overhead even idle. Since you already run llama.cpp + MLX, you don't actually need it for serving — a leaner stack (MLX for big/vision models, llama.cpp for FIM, Ollama for always‑on) is faster and lighter. Keep LM Studio for its conveniences (one‑click downloads, drag‑a‑screenshot‑in vision testing) and quit it when you want a tight setup. See §1.4 and §4.3.

1. What your 128 GB M5 Max can actually run

Unified memory is shared between macOS and the GPU. By default macOS lets the GPU "wire" ~75% of RAM (~96 GB on a 128 GB machine). You can raise that (see §1.3), but you must leave headroom for macOS, VS Code, the .NET toolchain, simulators, and the model's KV cache (context grows memory use as the conversation/file context grows).

1.1 Memory budget rule of thumb

  • Comfortable model ceiling: ~90–100 GB of weights + KV cache. Leave 16–24 GB for the system and your dev tools (you run Xcode/simulators/Android builds — keep this generous).
  • Quant sizing: 4‑bit ≈ 0.5–0.6 GB per 1B params, 8‑bit ≈ 1.0 GB per 1B. MoE models store all params but only compute the active ones — so they're large on disk but fast.

1.2 Concrete fits on 128 GB

Model Params (active) 4‑bit 8‑bit Verdict on 128 GB
Qwen2.5‑Coder‑7B 7B dense ~4 GB ~8 GB Trivial — run at 8‑bit
Qwen3.6‑27B (multimodal) 27B dense ~15 GB ~27 GB Easy — run 8‑bit; dense, so slower
Qwen3.6‑35B‑A3B (multimodal) 35B (3B) ~18 GB ~35 GB Easy — run 8‑bit, and it's fast
Qwen3‑Coder‑30B‑A3B (prev gen) 30B (3.3B) ~18 GB ~32 GB Easy — superseded by Qwen3.6
Qwen3‑VL‑32B (dedicated vision) 32B dense ~18 GB ~34 GB Easy — optional now
gpt‑oss‑120b 117B (5.1B) ~63 GB (MXFP4) — Comfortable — sweet spot for reasoning
GLM‑4.5‑Air 110B (12B) ~60–65 GB ~115 GB Comfortable at 4‑bit (heavier agentic option)
Devstral 2 123B dense ~62 GB — Fits, but slow (dense → all 123B active)
Qwen3‑235B‑A22B (Thinking) 235B (22B) ~132 GB ❌ — Does NOT fit at 4‑bit — use 3‑bit (~95 GB)
GLM‑4.6 / Kimi‑K2‑class 355B–1T 180 GB+ ❌ — Too big — use via API/Claude instead

Key takeaways for your machine:

  • Qwen3.6‑35B‑A3B is the new default — tiny for you (~35 GB at 8‑bit), fast (only 3B active), and multimodal. You can keep it and a 7B FIM model resident with room to spare.
  • The 120B‑class MoE (gpt‑oss‑120b) is your "big brain" for reasoning — fits with room and stays responsive because few params are active.
  • Qwen3‑235B is the smartest thing you can technically run, but only at 3‑bit and with modest context; it's slow (~15–25 tok/s). Treat it as an occasional "deep think," not a daily driver.
  • Don't waste your RAM: with 128 GB, run the coders at 8‑bit, not 4‑bit — the quality bump on code correctness is real and you have the memory.

1.3 Unlock more VRAM (only if you load >90 GB models)

Raise the GPU wired‑memory limit so big models + context fit:

# Allow ~118 GB for the GPU (leaves ~10 GB for macOS). Adjust down if you see memory pressure.
sudo sysctl iogpu.wired_limit_mb=120832

# Verify
sysctl iogpu.wired_limit_mb

This resets on reboot. To persist, create a LaunchDaemon:

sudo tee /Library/LaunchDaemons/com.local.gpu-wired-limit.plist >/dev/null <<'PLIST'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>com.local.gpu-wired-limit</string>
  <key>ProgramArguments</key>
  <array><string>/usr/sbin/sysctl</string><string>iogpu.wired_limit_mb=120832</string></array>
  <key>RunAtLoad</key><true/>
</dict></plist>
PLIST
sudo launchctl load /Library/LaunchDaemons/com.local.gpu-wired-limit.plist

You only need this for the 60–95 GB models (gpt‑oss‑120b, Qwen3‑235B). For Qwen3.6‑class daily work the default 96 GB is plenty — leave it alone and keep more RAM for builds/simulators.

1.4 MLX vs GGUF (llama.cpp/Ollama) vs LM Studio — which backend?

  • MLX (Apple's framework) is typically 10–30% faster than llama.cpp on Apple Silicon and uses memory efficiently. Prefer MLX for your coders and the big models (Qwen3.6, gpt‑oss‑120b, Qwen3‑235B). For vision, use mlx-vlm (the multimodal sibling of mlx-lm).
  • GGUF via llama.cpp gives the most control (FIM/infill server, speculative decoding, grammars, fine‑grained flags). Prefer it for autocomplete.
  • Ollama = simplest GGUF server, ~50–100 MB idle, runs headless. Great always‑on backend for Continue/opencode; built‑in web search.
  • LM Studio runs both MLX and GGUF, has the nicest GUI, an OpenAI‑compatible server, vision support, MCP, and one‑click downloads — but it's Electron: ~0.5–1.4 GB RAM and up to ~1.2 GB GPU‑VRAM overhead even idle. (LM Studio, Ollama, and GPT4All all wrap llama.cpp under the hood; LM Studio's one real edge on Mac is its built‑in MLX runtime.)

Practical rule (for you specifically): On 128 GB the LM Studio RAM overhead is noise, but the ~1.2 GB idle VRAM matters when you're maxing context on a 90 GB+ model. Since you're already comfortable with llama.cpp + MLX, run the lean stack — MLX/mlx-vlm for coders+vision+big models, llama.cpp for FIM, Ollama for an always‑on endpoint — and treat LM Studio as an optional GUI for downloads and quick screenshot tests.

2. The software layer — what each tool is for

You have six tools. They overlap; here's the clean division of labor.

Tool Best role in your setup Notes
llama.cpp FIM autocomplete server; power‑user inference (speculative decoding, grammars) Use llama-server + the official llama.vscode extension
MLX (mlx-lm / mlx-vlm) Fastest serving of coders, big MoE models, and vision mlx_lm.server / mlx_vlm.server expose an OpenAI‑compatible /v1 endpoint
Ollama Lightweight always‑on server (~50–100 MB idle, headless); easy pulls; built‑in web search Great backend for Continue/opencode; raise num_ctx for tools
LM Studio Optional GUI hub: model browser + OpenAI server + drag‑in vision + MCP Electron overhead (~1 GB RAM, ~1.2 GB VRAM idle) — quit it for lean serving
opencode Local agentic coding in the terminal (multi‑file edits, runs commands, tool use) Point it at MLX/Ollama/LM Studio; supports MCP
Claude Code (Max ×20) Escalation tier — the hardest reasoning/agentic tasks, large refactors, gnarly cross‑platform bugs Your quality ceiling; not local but already paid for
VS Code + Copilot Inline completion + chat you already know Copilot Chat can also point at Ollama models (Manage Models → Ollama); or use Continue for full local control

How they fit together (a single mental model):

            ┌─────────────────────────────────────────────┐
   VS Code  │  llama.vscode  → llama.cpp (FIM, port 8012)  │  inline completion
            │  Continue/Copilot → MLX (port 8080)         │  chat / edit / vision
            └─────────────────────────────────────────────┘
   Terminal │  opencode  → MLX or Ollama (port 8080/11434) │  agentic coding
            │  Claude Code → Anthropic (cloud)            │  hardest tasks
            └─────────────────────────────────────────────┘
   Servers  │  MLX (mlx-vlm) : Qwen3.6-35B-A3B (code + vision), gpt-oss-120b
            │  llama.cpp     : Qwen2.5-Coder-7B (FIM)
            │  + MCP web-search/fetch  → internet for any agent

3. Models by use case (with the why)

Model names move fast (it's mid‑2026). Everything below is verified as shipping. If a newer point release exists (e.g., a fresher Qwen3.6, GLM, or Devstral), the roles still apply — just pull the newer tag. See §9.

3.1 Inline autocomplete (Fill‑in‑the‑Middle) — the thing you feel every keystroke

Pick: Qwen2.5‑Coder‑7B (Q5_K_M/Q6), or Codestral (22B, Apache via Ollama) if you want stronger completions and can spare the latency.

  • Why not the big coder here? Autocomplete must return in tens of milliseconds. You want a small, FIM‑trained model. Qwen2.5‑Coder and Codestral were explicitly trained for Fill‑in‑the‑Middle, which is what inline completion needs (the big instruct/agentic coders like Qwen3.6 are tuned for chat/agent use, not raw FIM). On the M5 Max a 7B at Q5 will feel instant.
  • C# / C/C++ relevance: Qwen2.5‑Coder covers both well; Codestral advertises 80+ languages with strong C/C++/C#.

Run it (llama.cpp, recommended for FIM):

# llama.cpp ships FIM presets that auto-pick a good draft model for speculative decoding.
llama-server --fim-qwen-7b-default --port 8012
# (or explicit:)
# llama-server -m qwen2.5-coder-7b-instruct-q5_k_m.gguf -c 8192 --port 8012 --metrics

Then install the official llama.vscode extension and point it at http://127.0.0.1:8012. You get ghost‑text completions driven by the server's /infill endpoint.

Alternative (Ollama backend + Continue):

ollama pull qwen2.5-coder:7b      # or: ollama pull codestral

…and set it as Continue's autocomplete model (see §4.2).

3.2 Daily coding chat & edits — your everyday local workhorse

Pick: Qwen3.6‑35B‑A3B at 8‑bit.

  • Why: It's a 35B MoE with only ~3B active, released April 2026 — a clear generation ahead of Qwen3‑Coder. On your machine it's fast (~50–80 tok/s at 8‑bit) and strong (reported ~73% SWE‑bench Verified). It has a 256K context (extendable to ~1M), which matters for your multi‑file C#↔C interop, and it's natively multimodal — so the same model you chat code with can also look at a screenshot. At 8‑bit it's ~35 GB — trivial for 128 GB, and 8‑bit noticeably beats 4‑bit for code correctness.
  • C#/C/C++ fit: top‑tier among local coders across mainstream languages; handles P/Invoke‑style interop, CMake, and shader/GLSL questions well.
  • Previous‑gen fallback: Qwen3‑Coder‑30B‑A3B is still perfectly good if you already have it pulled — Qwen3.6 just does it better.

Run it (MLX, fastest):

# text-only serving:
mlx_lm.server --model mlx-community/Qwen3.6-35B-A3B-8bit --port 8080
# OR vision-capable serving (so it can read screenshots too):
mlx_vlm.server --model mlx-community/Qwen3.6-35B-A3B-8bit --port 8080

Or LM Studio: search qwen3.6-35b-a3b, choose the MLX 8‑bit build, Start Server (port 1234). Or Ollama: ollama pull qwen3.6:35b (tag may vary).

3.3 Agentic coding — multi‑file changes, runs the build, uses tools (opencode)

Pick (fast): Qwen3.6‑35B‑A3B — same model as §3.2, agentic‑tuned and quick (3B active). Pick (smarter, when you want more reasoning in the loop): Qwen3.6‑27B (dense 27B, reported ~77% SWE‑bench, matches frontier on some agentic benchmarks).

  • Why two options: Agentic loops issue many tool calls; speed compounds. Qwen3.6‑35B‑A3B is the responsive default. When a task needs deeper judgment inside the agent loop (tricky refactor, cross‑platform reasoning), Qwen3.6‑27B is higher quality and adds "Thinking Preservation" (it keeps reasoning across turns, which improves multi‑turn agent KV efficiency) — at the cost of dense‑model speed (~15–25 tok/s at 8‑bit, ~30–40 at 4‑bit). Both are multimodal and Apache 2.0.
  • Heavier alternative: GLM‑4.5‑Air (110B MoE / 12B active, ~4‑bit ≈ 60–65 GB) is purpose‑built for agentic + reasoning + coding if you want a bigger model in the loop. Devstral Small 2 (24B, Apache 2.0) is another excellent dense agentic coder; Devstral 2 (123B) is higher quality but dense → slow on Mac (skip for interactive use).
  • Text‑only agentic alternative: Qwen3‑Coder‑Next (Feb 2026; 80B total / 3B active, ~45 GB at 4‑bit) — a pure agentic coder built for long‑horizon tool use and failure recovery; same 3B‑active speed as Qwen3.6 but ~2× the footprint and not multimodal. Worth a head‑to‑head only if Qwen3.6 loses the thread on a very long agent run. (blog)
  • Brand‑new, not enough data yet — worth a head‑to‑head: Ornith‑1.0‑35B (DeepReinforce, released Jun 25 2026; 35B MoE / ~3B active, MIT, ~18 GB at 4‑bit) — an agentic coder whose scaffold self‑evolves during RL (auto‑plans, runs tools, rewrites failing steps); reports a strong ~64 on Terminal‑Bench 2.1. ⚠ Only days old, from a newer org, and benchmarks are vendor‑reported — there is not yet sufficient independent data to trust it. Likely text‑only (no vision) and MLX support may be nascent (GGUF available). Try it in opencode against Qwen3.6 on your own long agentic tasks before relying on it. (HF)
  • Tool‑calling matters most here. All of Qwen3.6, GLM‑4.5‑Air, Devstral, and gpt‑oss support tool/function calling — required for opencode to edit files and run commands.

Use with opencode (config in §4.1). For internet access inside the agent loop, add an MCP web‑search server (§5).

3.4 Planning / architecture / reasoning — "think before coding"

Pick (default): gpt‑oss‑120b. Pick (lean alternative): Qwen3.6‑27B in thinking mode. Pick (deepest, occasional): Qwen3‑235B‑A22B‑Thinking‑2507 at 3‑bit.

  • Why gpt‑oss‑120b: MoE (~117B / 5.1B active), ships in native MXFP4 (~63 GB), runs fast on your machine, and has adjustable reasoning effort (low/medium/high) with full chain‑of‑thought — ideal for design docs, architecture trade‑offs, and breaking work into steps (it's near o4‑mini‑class reasoning). Apache 2.0, first‑class MLX support.
  • Qwen3.6‑27B (thinking) is the lean planner: only ~27 GB, hybrid thinking/non‑thinking, and you may already have it loaded as your coder — one model for planning and execution.
  • Qwen3‑235B‑A22B‑Thinking is the single strongest local reasoner you can load (at 3‑bit, ~95 GB, slow). Reserve it for the occasional deep architectural problem where you'll wait for quality. Don't try the 4‑bit build — it's ~132 GB and won't fit.
  • Reality check: For your genuinely hard planning (e.g., a subtle WASM‑vs‑Metal divergence in Sokol.NET), Claude Code will still out‑think all of these. Use local planning for the 80% routine case and escalate the 20%. (This is also the basis of the hybrid workflow in §8.)

Run gpt‑oss‑120b (MLX):

mlx_lm.server --model lmstudio-community/gpt-oss-120b-MLX-8bit --port 8080
# set reasoning effort in the system/developer message: "Reasoning: high"

or LM Studio: search openai/gpt-oss-120b. or ollama pull gpt-oss:120b.

3.5 Vision — debugging from UI failure screenshots

Pick: your coder already does this. Because Qwen3.6‑35B‑A3B / ‑27B are natively multimodal, the model you're already running for code can read screenshots — drag in the broken panel and ask. That's the simplest setup and removes a whole separate model from your stack.

Want a dedicated, top‑quality VLM anyway? Qwen3‑VL‑32B (or Qwen3‑VL‑8B for a fast ~9 GB option).

  • Why Qwen3‑VL: it's specifically strong on structured visual content — screenshots, UI mockups, charts, tables, and reading on‑screen text (near‑perfect DocVQA), and processes images at native resolution, so small UI text and misaligned widgets stay legible. Use it when the Qwen3.6 coder's vision isn't quite resolving fine UI detail.
  • For your project specifically: when a Sokol.GUI panel overflows on a phone (CLAUDE.md §5), drop the screenshot in and ask it to identify the overflowing region and which widget lacks a ScrollView.

Runtime caveat (important): to actually use vision you need a vision‑capable runtime — mlx-vlm or LM Studio (drag‑and‑drop). A plain text‑mode llama-server won't see images even if the model is multimodal.

Easiest path — LM Studio: load a Qwen3.6 (or Qwen3‑VL) MLX build, open a chat, drag the screenshot in, ask. Lean path — mlx‑vlm: mlx_vlm.server --model mlx-community/Qwen3.6-35B-A3B-8bit --port 8080, then send an image via the OpenAI‑compatible API or from Continue.

To paste images inside VS Code, wire your multimodal model into Continue as a chat model with roles: [chat] (Continue supports image input to vision models) — see §4.2.

3.6 Internet‑connected — fetch packages, read docs, research

Two layers:

  1. Give a local agent web tools via MCP (works in opencode, LM Studio, and Claude Code). Add a web‑search and a fetch MCP server; then Qwen3.6 / GLM‑4.5‑Air / gpt‑oss can search and read pages during a task (e.g., "find the current SkiaSharp NuGet version and the breaking changes since 2.88"). Config in §5.
  2. Ollama's built‑in web search (REST API + MCP) if you standardize on Ollama.

For actually downloading and installing packages (NuGet/dotnet/Homebrew/npm), the model doesn't need raw internet — it needs shell access, which opencode and Claude Code already have. The web tools are for finding the right package/version and reading docs; the shell runs dotnet add package …. Keep that distinction: MCP web tools = read/research; shell tool = execute.

4. Concrete configurations

4.1 opencode → local server (agentic coding)

opencode talks to any OpenAI‑compatible endpoint. Create/merge ~/.config/opencode/opencode.json (or opencode.json in your project):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "mlx": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "MLX (local)",
      "options": { "baseURL": "http://127.0.0.1:8080/v1" },
      "models": {
        "qwen3.6-35b-a3b": { "name": "Qwen3.6-35B-A3B (fast code+vision)" },
        "qwen3.6-27b":     { "name": "Qwen3.6-27B (smarter)" },
        "gpt-oss-120b":    { "name": "gpt-oss-120b (planning)" }
      }
    },
    "lmstudio": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LM Studio (optional GUI)",
      "options": { "baseURL": "http://127.0.0.1:1234/v1" },
      "models": { "qwen3.6-35b-a3b": { "name": "Qwen3.6-35B-A3B" } }
    }
  }
}

Then in opencode: /models → pick the provider/model. Notes that bite people:

  • opencode needs a context window ≥ 64K. Make sure the served model is loaded with a large context (MLX: large by default for these models; LM Studio: set context length high).
  • If tool calls fail, increase context — for Ollama specifically, raise num_ctx to 32k+ (tool use breaks at small context).
  • Use Qwen3.6‑35B‑A3B for speed, switch to Qwen3.6‑27B for hard tasks, switch to gpt‑oss‑120b when you want it to plan first.

4.2 Continue (VS Code) → local chat/edit/autocomplete/vision

~/.continue/config.yaml:

name: Local Dev Stack
version: 1.0.0
models:
  # Daily chat + multi-file edits + screenshots (one multimodal model)
  - name: Qwen3.6-35B-A3B
    provider: openai
    model: qwen3.6-35b-a3b
    apiBase: http://127.0.0.1:8080/v1   # mlx_vlm.server (vision-capable)
    apiKey: none
    roles: [chat, edit, apply]

  # Inline completion (FIM) — small & fast, via Ollama or llama.cpp
  - name: Qwen2.5-Coder 7B (FIM)
    provider: ollama
    model: qwen2.5-coder:7b
    roles: [autocomplete]

  # Planning / reasoning
  - name: gpt-oss-120b (planning)
    provider: openai
    model: gpt-oss-120b
    apiBase: http://127.0.0.1:8081/v1
    apiKey: none
    roles: [chat]

Because Qwen3.6 is multimodal, the single chat model above also handles screenshot debugging — paste an image into Continue's chat. (If you prefer a dedicated VLM, add a Qwen3-VL-32B entry with roles: [chat].)

Copilot vs Continue: Keep Copilot if you like its completions; it can also use Ollama models via Manage Models → Ollama in recent VS Code. But Continue gives you full control over per‑role local models (separate autocomplete vs chat), which is what you want here. Running both is fine — just don't enable two inline‑completion providers at once (they'll fight).

4.3 LM Studio (optional GUI hub)

Optional for you — it's the easiest GUI, but Electron overhead (~1 GB RAM, ~1.2 GB VRAM idle) means the lean MLX/llama.cpp/Ollama stack is better for serving. Use LM Studio for downloads, browsing, and quick drag‑in vision tests; quit it when you want maximum VRAM/context.

  1. Download builds: prefer the MLX variants. Suggested set:
    • qwen3.6-35b-a3b → MLX 8‑bit (your code + vision daily driver)
    • qwen3.6-27b → MLX 4–8‑bit (smarter agentic / thinking)
    • openai/gpt-oss-120b → MLX (MXFP4/8‑bit) (planning)
    • qwen2.5-coder (7B) → GGUF Q5/Q6 (if you serve FIM from here instead of llama.cpp)
    • (optional) qwen3-vl (32B) → MLX 4–6‑bit (dedicated vision)
  2. Server: Developer tab → Start Server (defaults to http://127.0.0.1:1234/v1). Set Context Length high (≥ 64K) for opencode.
  3. Vision: open a chat with a multimodal model (Qwen3.6 or Qwen3‑VL) and drag the screenshot in.
  4. Speculative decoding: LM Studio supports a draft model for speed — pair a big coder with a tiny draft model to boost tok/s.
  5. MCP: LM Studio supports MCP servers (for web search/fetch) — see §5.

4.4 MLX direct (recommended — fastest, leanest)

pip install -U mlx-lm mlx-vlm        # or: uv pip install mlx-lm mlx-vlm

# Coder + vision (one multimodal server) — your daily driver
mlx_vlm.server --model mlx-community/Qwen3.6-35B-A3B-8bit --port 8080

# Smarter coder (dense)
mlx_vlm.server --model mlx-community/Qwen3.6-27B-8bit --port 8083

# Planning brain
mlx_lm.server --model lmstudio-community/gpt-oss-120b-MLX-8bit --port 8081

# Deep think (occasional; 3-bit so it fits 128 GB)
mlx_lm.server --model mlx-community/Qwen3-235B-A22B-Thinking-2507-3bit --port 8082

Each exposes OpenAI‑compatible /v1. Point opencode/Continue at the right port. (Run one large model at a time unless their combined size + context fits your wired limit. Qwen3.6‑class models are small enough to keep one resident alongside the 7B FIM server permanently.)

4.5 llama.cpp (FIM autocomplete + power features)

# FIM server with built-in speculative decoding preset
llama-server --fim-qwen-7b-default --port 8012

# Power-user explicit form:
# llama-server -m qwen2.5-coder-7b-q6_k.gguf -c 8192 --port 8012 \
#   -ngl 999 --metrics --flash-attn

Pair with the llama.vscode extension (set endpoint http://127.0.0.1:8012). This is the leanest, lowest‑latency inline‑completion path and runs comfortably alongside your MLX coder.

4.6 Tuning local agents — context length, output cap, and the harness effect (field notes)

A controlled test drove home a lesson worth banking: the agent harness moved results as much as the model did. Same prompt, four local models served by an MLX server (oMLX), two harnesses — opencode vs VS Code + Copilot (agent mode) — on a demanding whole‑repo task: traverse every source file, count files/components/lines exactly, and emit an Architecture.md with sequence + flow diagrams. The Copilot harness averaged ~10 points higher (0–100) than opencode across the four models; two models swung ~17 points from the harness alone, while one was completely harness‑invariant.

Why the harness matters so much:

  • Copilot agent mode "carries" the model — its own workspace index, semantic retrieval, and multi‑step orchestration surface the right files and manage the context budget for the model.
  • opencode is a thinner agent that leans on the model to drive the tool loop (read/grep/shell) and hold understanding in its own context window. It exposes the model's raw agentic ability and is acutely sensitive to served context length and tool‑calling quality.
  • So context‑hungry or weaker‑tool‑use models lose points in opencode; robust ones don't move.

The fix — it's almost always a server‑side config problem, not the model:

  1. Context length is the #1 culprit. If your MLX server (oMLX / LM Studio / mlx_lm.server) serves a small default (4K–8K), opencode's whole‑repo prompts get truncated → the model never sees enough → undercounts and omissions → low score. These models support 256K; on 128 GB you can afford a large KV cache for a 35B. Set served context to ≥64K (128K if allowed) — in oMLX that's the Server tab (max context / KV size).
  2. Raise max output tokens (≥8K, ideally 16K) so a long doc isn't cut off mid‑output.
  3. Enable opencode's shell tool and nudge it. Exact file/line counts must come from find … | wc -l, not the model eyeballing. And structural facts (component counts, route tables, registries) usually live in one manifest/registry file — a model that locates and parses it gets the exact number; one that infers from the directory tree miscounts. Tell the model to use shell and to find the registry.
  4. Confirm the editor harness actually used your local model (not a silent cloud fallback). If per‑model scores vary in the editor, it's using the local model — good.
  5. Quant recipe > bit count, and choose your backend deliberately. In a same‑harness test, moving one model from an MLX server to llama.cpp with an Unsloth Dynamic (UD) quant lifted its score +16 — a calibrated UD‑Q6 beat a naive higher‑bit quant. Rule of thumb: MLX = faster on Apple Silicon; llama.cpp = sometimes higher‑fidelity and the most battle‑tested. When a model underperforms, isolate backend vs. quant (run the same UD quant on both — Unsloth ships UD builds for GGUF and MLX) before blaming the model.

Model takeaways from that test (n = 1 per cell → directional, not a ranking; rerun before trusting):

  • The 35B‑A3B MoE class (3B active) beat a dense 27B — and beat a higher‑bit quant of it. Active‑param speed + tool use mattered more than raw size or quant fidelity.
  • Qwen‑AgentWorld‑35B‑A3B was the most consistent (flat across both harnesses) and led on the thin opencode harness — strong evidence of robust on‑its‑own agentic ability. Notably it's billed as a "language world model," not a coder; it would be easy to skip on the card alone, and the data corrected that. Test, don't assume from the model card.
  • Ornith‑1.0‑35B posted the highest single score but the widest variance — the classic "promising but unproven" signature (consistent with §3.3's caveat).
  • Qwen3.6‑35B‑A3B / ‑27B were solid and benefited most from the scaffolded harness — i.e., they're more sensitive to getting enough context, so they reward the tuning above.

Benchmark models trustworthily:

  • 3–5 runs per (model, harness), report mean ± range — single runs of open‑ended tasks mislead.
  • Split the score: objective (counts vs. a ground truth you compute once with shell) + subjective (diagram validity, prose, completeness). A blended 0–100 hides what you're measuring.
  • Equalize context length across harnesses and use a low temperature (~0.2) for "exact" tasks.

5. Giving local models internet access (MCP)

Add web search + fetch as MCP servers so any tool‑calling model can research and read pages. The same config style works for opencode, LM Studio, and Claude Code (each has an MCP config section).

opencode (opencode.json, merge into the file from §4.1):

{
  "mcp": {
    "fetch": {
      "type": "local",
      "command": ["uvx", "mcp-server-fetch"]
    },
    "websearch": {
      "type": "local",
      "command": ["uvx", "duckduckgo-mcp-server"]
    }
  }
}

Ollama's built‑in web search (if you standardize on Ollama): create a free Ollama key and expose its web‑search MCP:

{
  "mcpServers": {
    "ollama_web_search": {
      "command": "uvx",
      "args": ["ollama-web-search-mcp"],
      "env": { "OLLAMA_API_KEY": "your_key_here" }
    }
  }
}

Notes:

  • Use a DuckDuckGo or Brave/Exa‑backed search MCP (DuckDuckGo needs no key; Brave/Exa give better results with a free API key).
  • mcp-server-fetch lets the model read a specific URL (release notes, NuGet page, GitHub README) — pair it with search.
  • The model still executes installs via opencode/Claude Code's shell tool (dotnet add package, brew install, git submodule update). MCP web tools are for finding/reading, the shell is for doing.
  • Privacy: web tools send your queries out. Keep them on the agents where you want research; leave them off your pure‑offline coder if you care about a hermetic setup.

6. Two ready‑to‑use profiles

Profile A — "Lean & fast" (most days)

  • Inline: llama.cpp --fim-qwen-7b-default (port 8012) + llama.vscode.
  • Chat/edit/vision: MLX mlx_vlm.server serving Qwen3.6‑35B‑A3B (8‑bit) (port 8080) + Continue. One model covers code and screenshots.
  • Agentic: opencode → that same Qwen3.6 endpoint, with fetch+search MCP.
  • Memory: ~43 GB total resident — leaves tons for builds/simulators. No wired‑limit tweak, no Electron, no LM Studio needed.

Profile B — "Big brain" (hard problem, plugged in)

  • Raise wired limit (§1.3).
  • Planning: MLX gpt-oss-120b (port 8081), reasoning effort high.
  • Agentic: opencode → Qwen3.6‑27B (8‑bit) (port 8083) — or GLM‑4.5‑Air (4‑bit) for a bigger model in the loop — with MCP web tools.
  • Vision: the Qwen3.6 model already handles it; add Qwen3‑VL‑32B only if you need finer UI detail.
  • Occasional deep think: Qwen3‑235B‑A22B‑Thinking 3‑bit (load alone).
  • Run one 60–95 GB model at a time; keep the 7B FIM server up alongside (it's tiny).

7. When to use local vs Claude Code (your escalation ladder)

  1. Inline completion / boilerplate / repetitive edits → local FIM (free, instant, private).
  2. Single‑file or small multi‑file changes, explain‑this‑code, quick refactors → Qwen3.6‑35B‑A3B local.
  3. Screenshot triage ("why does this panel overflow?") → Qwen3.6 (multimodal) or Qwen3‑VL local.
  4. Routine planning / "outline the steps" → gpt‑oss‑120b or Qwen3.6‑27B local.
  5. Hard cross‑platform bugs (Metal vs WebGL2 vs GLES3), large refactors, subtle interop/NativeAOT issues, "get this actually right" → Claude Code (Max ×20). Your CLAUDE.md is explicit that correctness across 6 platforms is the bar — that's Claude's job.

Rule of thumb: draft locally, finish with Claude when it matters. You're paying for the Max plan; spend it on the 20% of tasks where model quality changes the outcome, and keep the cheap/private local models busy on the other 80%.

8. Hybrid workflow: Claude Code plans, local models build

This is the highest‑leverage way to use everything you have at once: Claude Code writes the plan, a local agent (opencode) executes it. The economics are the point — planning is a short, high‑judgment task (a few thousand tokens where quality decides everything), while agentic execution is a long, low‑judgment grind (dozens of tool calls, file edits, build runs — exactly what burns tokens/time). So spend Claude's quality on the plan and spend your free, private, local compute on the grind.

Why not let the local model plan too? Because the two failure modes that wreck a coding session are cross‑platform blind spots (a change that works on macOS Metal but breaks WASM/WebGL2 or Android GLES3) and API hallucination (inventing a method that doesn't exist). Claude is markedly better at both. A good Claude plan pre‑answers those questions so the local executor never has to make the judgment call it's bad at — it just follows precise, verifiable steps.

  ┌──────────────┐   plan file    ┌───────────────────┐  verify   ┌──────────┐
  │  Claude Code │ ─────────────▶ │ opencode + local  │ ────────▶ │  build / │
  │  (Plan Mode) │  docs/plans/   │ model (executor)  │ ◀──────── │  tests   │
  └──────────────┘   task.md      └───────────────────┘  fail→fix └──────────┘
        ▲                                   │ stuck (escalation trigger)
        └───────────────────────────────────┘

8.1 Step 1 — Plan with Claude Code

Use Claude Code's Plan Mode (it proposes a plan and won't touch files until you approve). Iterate on the plan interactively, then have Claude persist the approved plan to a file the local agent will read:

"Write the approved plan to docs/plans/<task>.md using the executable‑plan template: acceptance criteria, cross‑platform constraints, exact files + symbols, numbered steps each with a verify command, and escalation triggers. Assume the executor is a mid‑size local model with no judgment — be explicit about API signatures and never leave a step ambiguous."

The plan format is what makes or breaks this. A weaker executor needs a plan that removes ambiguity. Have Claude emit exactly this shape:

# Task: <one line>

## Acceptance criteria (definition of done)
- [ ] Builds on macOS:  dotnet build examples/<proj>/<Proj>.csproj
- [ ] Shaders compile:  dotnet build examples/<proj>/<Proj>.csproj -t:CompileShaders
- [ ] <visual/behavioral check, e.g. "panel scrolls fully on a 360px-wide viewport">

## Constraints / gotchas (cross-platform — DO NOT violate)
- Editor UI uses Sokol.GUI, NOT ImGui (no ig* calls).
- NEVER hand-edit src/sokol/generated/ — run ./scripts/generate-bindings.sh.
- Change verified for: macOS Metal. Re-check WASM/WebGL2 + Android GLES3 before "done".

## Files to touch (and nothing else)
- examples/<proj>/Source/Foo.cs   — add the widget + sync
- ext/<lib>/foo.c                  — new native helper (then regen bindings)

## Steps (each independently verifiable)
1. [Foo.cs] Add `NumberInput` field "Scale" to Rebuild().
   Use signature: `new NumberInput { Min = 0f, Max = 10f, Step = 0.1f }`.
   verify: dotnet build … → expect 0 errors.
2. [Foo.cs] Wire it in SyncValues(): `_scale.Value = comp.Scale;`
   verify: build → expect 0 errors.
3. [ext/<lib>/foo.c] Add `SOKOL_API_DECL void foo_fast(float* p, int n);`
   then run ./scripts/generate-bindings.sh
   verify: git status shows regenerated src/sokol/generated/*.cs; build → 0 errors.

## Escalation triggers (STOP and hand back to Claude Code)
- A referenced symbol/method does not exist in the codebase.
- Any verify step fails twice in a row.
- A change forces edits to files not listed above.

The discipline that matters: acceptance criteria are commands, steps name exact symbols/signatures, and escalation triggers tell the local agent when to stop digging itself into a hole instead of hallucinating its way forward.

8.2 Step 2 — Execute with a local agent (opencode)

First, give the local agent the same guardrails Claude has. opencode reads an AGENTS.md from the repo root, just like Claude reads CLAUDE.md. Symlink them so the weaker model inherits every project rule (don't‑edit‑generated, Sokol.GUI‑not‑ImGui, UI‑must‑fit‑phones, local player's pieces at the bottom, etc.):

ln -s CLAUDE.md AGENTS.md     # opencode now follows your project's rules too

Then run the executor against the plan. Use Qwen3.6‑35B‑A3B as the default (fast), and switch to Qwen3.6‑27B (or GLM‑4.5‑Air) for steps involving native interop or tricky logic:

# non-interactive: execute the whole plan
opencode run "Execute docs/plans/<task>.md one step at a time. After each step, run its
verify command and paste the result. Do NOT skip ahead, do NOT touch files outside the
'Files to touch' list, and STOP immediately if any escalation trigger fires."

# or interactive TUI, then /models to pick qwen3.6-35b-a3b or qwen3.6-27b
opencode

Tips that keep a mid‑size model on the rails:

  • One step at a time. The plan's per‑step verify command is the leash. Don't let it batch.
  • Make the build the judge, not the model. The local agent's opinion that code is correct is worth little; the compiler/test exit code is worth everything. Every step ends in a command.
  • Keep context large. opencode needs ≥ 64K context; load the model accordingly (§4.1) so it can hold the plan + the files it's editing.
  • Annotate model‑per‑step in the plan if you like (# step 3: use qwen3.6-27b) and switch with /models when you reach it.

8.3 Step 3 — Verify and escalate

The loop per step is: local edit → run verify command → green? next step : local fix (≤2 tries) → still red? escalate. When an escalation trigger fires, hand that single stuck step back to Claude Code (with the error output), let Claude diagnose/patch or amend the plan, then return control to the local agent for the rest. You're not abandoning the local run — you're spending one expensive Claude turn only on the 1–2 steps that actually needed it.

This maps cleanly onto your project's reality (CLAUDE.md §4): "done" here is builds + the stated visual/behavioral check on real targets, not the model's say‑so. For anything UI, the acceptance check must include "renders and is fully scrollable on a phone‑sized screen," verified on an actual Android/iOS target before you call it finished — keep that in the plan's acceptance criteria so the local agent treats it as a gate, not an afterthought. (Bonus: since your local coder is multimodal, the agent can even look at a screenshot it captured and self‑check the UI before escalating.)

8.4 C# / C / C++ specifics for the executor

The split shines on your mixed codebase, but each language layer has a trap the local model must be steered around — so Claude's plan should pre‑empt these:

  • C# + NativeAOT / P/Invoke: local models guess marshalling and struct layout. The plan must give the exact [DllImport]/[LibraryImport] signature, [StructLayout], and which struct‑by‑value calls need the WASM pointer‑return wrapper (the auto‑generated ext/sokol_csharp_internal_wrappers.h path). Don't let the executor improvise interop.
  • Generated bindings: the executor must never hand‑edit src/sokol/generated/. The correct step is edit the C header → run ./scripts/generate-bindings.sh → rebuild. Put that as an explicit verify step; the guardrail in AGENTS.md reinforces it.
  • C/C++ in ext/: native changes mean a toolchain rebuild (scripts/build-*.sh) and possibly a bindings regen — both slow. Mark them as their own steps so the agent doesn't assume a C# rebuild picked up a C change.
  • Shaders: any .glsl change needs -t:CompileShaders before the app build, and must compile for every backend dialect (glsl430/hlsl5/metal_*/glsl300es). Make shader compilation its own verify gate.
  • "Verified on which platform?" A local agent that builds on macOS will happily declare victory. The plan's acceptance criteria must list the other targets (WASM/WebGL2, Android GLES3) as explicit, separate checks — this is the single most common way a change silently breaks a platform.

8.5 When this hybrid is worth it (and when it isn't)

  • Worth it: well‑bounded features and refactors with a clear definition of done — add a widget/field, expose a native helper, port a pattern across examples, mechanical multi‑file edits. Claude plans in one cheap pass; the local agent grinds for free.
  • Skip the split, go straight to Claude Code: genuinely hard diagnosis (a z‑fighting/timing bug that differs across Metal vs WebGL2), or anything where the plan itself is the unknown. If you can't write crisp acceptance criteria, the local executor has nothing to verify against — keep it with Claude until the problem is understood.
  • Skip the split, go straight to local: trivial one‑liners and inline completion — no plan needed; just use the local coder/FIM directly (§3.1–3.2).

9. Keeping current (the field moves monthly)

  • The roles in this doc are stable; the specific tags churn. Periodically check for newer point releases of: Qwen3.6 (and successors), gpt‑oss, GLM‑4.5/4.6‑Air‑class, Devstral, Qwen3‑VL. Pull the newer tag, keep the same wiring.
  • Watch the mlx-community and lmstudio-community Hugging Face orgs — they publish MLX conversions within days of a release.
  • Re‑run a quick C# + C/C++ interop sanity prompt and a screenshot‑debug prompt whenever you swap a model, so you trust the change before relying on it.
  • Sanity‑check any brand‑new model name you see on blogs against its official Hugging Face/GitHub repo — the SEO content farms invent plausible‑sounding versions that don't exist. (Qwen3.6‑35B‑A3B and Qwen3.6‑27B do exist — verified against the official Qwen blog and Hugging Face.)

Sources / further reading

Written 2026‑06‑29, updated same day for the Qwen3.6 release. Tailored to an M5 Max / 128 GB. Model availability verified as of this date; benchmark figures are vendor/community‑reported — pull newer point releases as they ship.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment