Simple, local, non-scientific benchmarking of various models, all receiving the same prompt. All tests were performed with MstyStudio, focusing on local MLX and LLaMA.cpp models.
Models were chosen at random based on community popularity. This is entirely an educational assesment and not a scientific / structured benchmark.
Apple M4 Max
128 GB Memory
2 TB Storage
No additional system prompt was provided.
Write a small REST API in Golang using the Fiber framework. It should be a simple todo app that persists to a sqlite database.
-
Qwen3.6-35B-A3B-4bit
- 20.78 tokens/sec
- 24.334s to 1st Token
- 0 input tokens
- 5646 output tokens
-
Codestral-22B-v0.1-4bit
- 14.07 tokens/sec
- 19.223s to 1st Token
- 39 input tokens
- 1472 output tokens
-
Kimi-Dev-72B-4bit-DWQ
- 8.82 tokens/sec
- 51.666s to 1st Token
- 56 input tokens
- 5990 output tokens
-
DeepSeek-R1-Distill-Qwen-32B-4bit
- 11.73 tokens/sec
- 5.234s to 1st Token
- 33 input tokens
- 2166 output tokens
-
gemma-3-27b-it-4bit
- 11.38 tokens/sec
- 14.133s to 1st Token
- 0 input tokens
- 2224 output tokens
-
gemma-4-e2b-it-4bit
- 22.86 tokens/sec
- 7.906s to 1st Token
- 0 input tokens
- 2787 output tokens
-
GLM-4-32B-0414-8bit
- 10.04 tokens/sec
- 7.851s to 1st Token
- 32 input tokens
- 472 output tokens
-
gpt-oss-120b-MXFP4-Q8
- 46.85 tokens/sec
- 19.14s to 1st Token
- 94 input tokens
- 2353 output tokens
-
gpt-oss-20b-MXFP4-Q8
- 93.65 tokens/sec
- 4.197s to 1st Token
- 94 input tokens
- 2461 output tokens
-
Qwen3-1.7B-4bit
- 211.03 tokens/sec
- 1.513s to 1st Token
- 35 input tokens
- 7016 output tokens
-
Qwen3-8B-4bit
- 65.20 tokens/sec
- 2.357s to 1st Token
- 35 input tokens
- 8315 output tokens
-
gemma-3-1b-it-Q4_K_M
- 98.73 tokens/sec
- 2.191s to 1st Token
- 36 input tokens
- 1763 output tokens
-
Phi-3-mini-4k-instruct-Q4_K_M
- 18.41 tokens/sec
- 0.634s to 1st Token
- 33 input tokens
- 1728 output tokens
-
Llama-3.2 3B
- 16.69 tokens/sec
- 14.08s to 1st Token
- 62 input tokens
- 1121 output tokens
-
Qwen3.6-35B-A3B-UD-Q4_K_M
- 16.20 tokens/sec
- 17.568s to 1st Token
- 37 input tokens
- 4059 output tokens
-
Qwen3.5-9B-Q4_K_M
- 12.53 tokens/sec
- 15.824s to 1st Token
- 37 input tokens
- 4059 output tokens
βΊ β Insight βββββββββββββββββββββββββββββββββββββ This corpus is a LLM code generation benchmark β 15 models all given the same prompt ("build a Golang Fiber SQLite Todo API") and their outputs compared. The graph reveals a clear split: models that chose GORM vs raw database/sql, and two llamacpp Qwen models that failed entirely. The Todo API Prompt node (betweenness: 0.605) is the single hub connecting every community β a classic star topology for a benchmark dataset. βββββββββββββββββββββββββββββββββββββββββββββββββ
Graph complete. Outputs in /Users/peter/Downloads/33ff69658a5c9353308d1990c492fa68-dc21b0caf420ccbff5ce17ed3fe57e46505688a9/graphify-out/
graph.html - interactive graph, open in browser
GRAPH_REPORT.md - audit report
graph.json - raw graph data
If graphify saved you time, consider supporting it: https://github.com/sponsors/safishamsi
God Nodes (most connected β your core abstractions)
- Model Sampling Benchmark β 20 edges
- Golang Fiber SQLite Todo API Prompt β 16 edges
- RegisterRoutes Function (Qwen3-8B) β 6 edges
- Raw database/sql Pattern β 6 edges
- GORM ORM Pattern β 5 edges
Surprising Connections
-
RegisterRoutes Function (Qwen3-8B) β setupRoutes Function (gemma-4-e2b-it) [INFERRED, semantically similar] β Two completely different models independently converged on the same routing abstraction name and structure
-
Chain-of-Thought Reasoning (Kimi-Dev-72B) β DeepSeek-R1-Distill-Qwen-32B Todo API [INFERRED, semantically similar] β Both are reasoning/thinking models; Kimi's visible chain-of-thought pattern mirrors DeepSeek-R1's distilled reasoning approach
-
Hyperedge: LLaMA.cpp Models That Failed to Produce an Answer β Qwen3.5-9B got stuck in a thought loop; Qwen3.6-35B produced no answer at all. Both are the same model family run under llama.cpp instead of MLX.
Suggested Questions
- Why does Golang Fiber SQLite Todo API Prompt connect LLaMA.cpp Models & Failures to MLX Model Leaderboard, GORM Multi-file CRUD Structure,
GORM ORM Adopters, Raw SQL Pattern Models, and Reasoning Model Implementations? - Why does Model Sampling Benchmark bridge MLX Model Leaderboard to LLaMA.cpp Models & Failures, Reasoning Model Implementations, and Quantization Strategies & Runtimes?
- Which models chose GORM vs raw SQL β and does that choice correlate with output quality?
The most interesting question this graph can answer: Why did the same Qwen3.6-35B model succeed under MLX but fail completely under llama.cpp?
- Why does Golang Fiber SQLite Todo API Prompt connect LLaMA.cpp Models & Failures to MLX Model Leaderboard, GORM Multi-file CRUD Structure,
Claude Opus 4.7 was asked to score and analyze the results of the model sampling "benchmark".
This was the prompt used:
I have run a totally non-scientific model sampling of various available community models, targeting MLX and LlaMA.cpp. Could you analyze the markdown files which contain the outputs of the models from a single prompt:
> Write a small REST API in Golang using the Fiber framework. It should be a simple todo app that persists to a sqlite database.
Could you determine the objectively most accurate response? Some things you could grade on include; clarity, idiomatic code, usefulness, "production ready"-ness, etc.
After that, can you apply a more statistical approach to the values shared in the README and score the models on their speed, incorporating a rough score from the code analysis?
In the end, please clarify the rating criteria you used and identify the top 3 tested models, provide their score, and rank them.
Given that one prompt and the responses, I would like to identify the top local models to achieve the task.
Two scores were computed and summed into a final composite out of 100.
Code Quality (0-50) β five axes, 0-10 each, applied to each model's actual output:
- Correctness β does it compile? Are imports real? Is the Fiber API used properly (v2, not the deprecated v1)? Is the sqlite driver wired up? Schema created? Routes mounted?
- Idiomatic Go β error handling, naming, struct tags,
defer rows.Close(), no leaky abstractions or hallucinated stdlib calls. - Completeness β full CRUD (create, list, get-one, update, delete), proper status codes (201 on create, 404 on miss), JSON parsing.
- Production-readiness β env-driven config, parameterized queries / prepared statements, logging, graceful shutdown, input validation,
go.mod, run instructions. - Clarity β does the response actually explain what was built and how to run it? Curl examples? Sensible structure?
Speed (0-50) β min-max normalized from the README numbers:
- Throughput (0-25):
(tps - tps_min) / (tps_max - tps_min) * 25 - Responsiveness (0-25):
(1 - (ttft - ttft_min) / (ttft_max - ttft_min)) * 25
These are summed for the speed score. Both halves are weighted equally in the final composite β speed without working code is worthless, but a 50 t/s CRUD app beats a 10 t/s one for the same correctness.
| Rank | Model | Code/50 | Speed/50 | Total/100 |
|---|---|---|---|---|
| 1 | mlx β gpt-oss-20b-MXFP4-Q8 | 45 | 33.74 | 78.74 |
| 2 | mlx β Qwen3-8B-4bit | 42 | 31.13 | 73.13 |
| 3 | mlx β gemma-4-e2b-it-4bit | 42 | 23.17 | 65.17 |
| 4 | mlx β gpt-oss-120b-MXFP4-Q8 | 42 | 20.64 | 62.64 |
| 5 | mlx β Qwen3-1.7B-4bit | 10 | 49.57 | 59.57 |
| 6 | mlx β Codestral-22B-v0.1-4bit | 42 | 16.54 | 58.54 |
| 7 | mlx β DeepSeek-R1-Distill-Qwen-32B-4bit | 35 | 23.11 | 58.11 |
| 8 | mlx β Qwen3.6-35B-A3B-4bit | 42 | 14.87 | 56.87 |
| 9 | llamacpp β gemma-3-1b-it-Q4_K_M | 18 | 35.35 | 53.35 |
| 10 | mlx β GLM-4-32B-0414-8bit | 27 | 21.62 | 48.62 |
| 11 | mlx β Kimi-Dev-72B-4bit-DWQ | 39 | 0.00 | 39.00 |
| 12 | llamacpp β Phi-3-mini-4k-instruct-Q4_K_M | 5 | 26.19 | 31.19 |
| 13 | mlx β gemma-3-27b-it-4bit | 12 | 18.70 | 30.70 |
| 14 | llamacpp β Llama-3.2-3B | 10 | 19.39 | 29.39 |
| 15 | llamacpp β Qwen3.5-9B-Q4_K_M | 0 (failed) | 18.02 | 18.02 |
| 16 | llamacpp β Qwen3.6-35B-A3B-UD-Q4_K_M | 0 (failed) | 17.62 | 17.62 |
π₯ 1. mlx β gpt-oss-20b-MXFP4-Q8 (78.74 / 100)
The most well-rounded result of the lot. Highest code score (45/50) β produced two complete implementations (a GORM variant split into models.go, handlers.go, routes.go, utils.go, plus a raw database/sql variant), env-driven PORT and TODO_DB_PATH config, full CRUD, sensible error handling, proper status codes, and a real README with run + curl instructions. Backs that up with 93.65 t/s and a 4.2 s TTFT β fast, snappy, and the only model whose output you could plausibly hand to a junior dev without a meaningful rewrite.
π₯ 2. mlx β Qwen3-8B-4bit (73.13 / 100)
The "fits-on-a-laptop" pick. Code score (42/50) is on par with the Codestral / Qwen-35B / gemma / 120B cluster β clean gofiber/fiber/v2, prepared statements, logger middleware, full CRUD, correct status codes, curl examples. What sets it apart is the speed profile: 65.2 t/s with a 2.36 s TTFT. For an 8B model that's punchy. If you want the best per-watt / per-GB-VRAM option in this corpus, this is it.
π₯ 3. mlx β gemma-4-e2b-it-4bit (65.17 / 100)
The tiny-but-capable sleeper. A ~2B-effective model that produced a clean GORM + Fiber v2 implementation with proper error handling, correct codes, and curl test examples (42/50 β same code tier as models 5-10Γ its size). Speed is solid (22.86 t/s, ~8 s TTFT). Strong evidence that small Gemma models are punching above their weight class for boilerplate-heavy CRUD work.
- Qwen3-1.7B-4bit ran at a ridiculous 211 t/s with a 1.5 s TTFT β fastest in the entire panel β but its code is broken (hallucinated
c.BindJSON, missingdatabase/sqlimport, fake migration package). Speed alone doesn't save bad output. - Codestral-22B and gpt-oss-120b match the second-tier code score (42) but eat their lead with poor TTFTs (~19 s each). Codestral in particular felt like the cleanest single implementation if you read the code in isolation.
- Kimi-Dev-72B and DeepSeek-R1-Distill-32B both spent enormous reasoning-token budgets in
<think>blocks. Kimi got the worst speed score in the field (0/50) and DeepSeek had a buggyLastInsertRowid()call. For a ~50 LOC CRUD task, R1-style chain-of-thought is wildly wrong tool for the job. - The two empty llamacpp Qwen3 outputs confirm the graphify hypothesis β same architecture works fine under MLX, falls over under llama.cpp. Almost certainly a chat-template / EOS-token mismatch in the GGUF metadata; worth filing back to the quanters.
- gemma-3-1b-it-Q4_K_M (rerun): now scored β and it's a cautionary tale. The model ignored the prompt's framework requirement entirely and wrote a Gin app instead of a Fiber app, which alone disqualifies it from the top tier. The code also hallucinates a
*sqlite3.DBtype and adb.Query(&dest, ...)shape that don't exist inmattn/go-sqlite3(it usesdatabase/sql), references a non-existentsqlite3.ErrNotFound, and contains a string-to-int type mismatch inUpdateTodothat won't compile. Speed is genuinely impressive for a 1B model (98.73 t/s, 2.19 s TTFT β speed score 35.35/50, the third-fastest in the corpus), but code lands at 18/50 and the composite (53.35) puts it at rank 9. Lesson: at the 1B scale, instruction-following on framework choice is the first thing to crack.
If you want one local model on an M4 Max for "scaffold me a Go REST API" tasks: gpt-oss-20b-MXFP4-Q8. If you're VRAM-constrained or want something snappier on a smaller machine: Qwen3-8B-4bit. If you want to be impressed by what a tiny model can do: gemma-4-e2b-it-4bit. Reasoning-distilled models (R1, Kimi-Dev) are overkill and slow for boilerplate CRUD; reach for them on actual algorithmic problems instead.