Each competitor received the same prompt: the problem statement + quick start statements of monocrate's README (~1,500 characters). No "how it works", no source code, no implementation hints. The instructions in the prompt were:
Below is the readme file of a new publishing tool for nodejs monorepos. I want to turn it
into a full high-quality open source typescript repo to be pushed to github, published
to npm, announced on hackernews, and become a successful, reliable devtool.
For this it needs to properly cover edge cases, and clearly call out situations it does
not handle in the README.md. Make all of this happen:
monocrate is a CLI for publishing packages from Node.js monorepos. It handles import rewriting, dependency graph resolution, .d.ts bundling, version bumping against the npm registry, and edge cases around symlinks, circular deps, and dynamic imports. The existing codebase is ~2K lines of TypeScript.
| Label | Agent |
|---|---|
| Baseline | Human + Claude Opus 4.5 (iterative, over several days of work) |
| Codex 5.3 | OpenAI Codex 5.3 |
| Opus 4.6 | Claude Opus 4.6 via Claude Code |
| Codex 5.2 | OpenAI Codex 5.2 |
| Opus 4.5 | Claude Opus 4.5 via Claude Code |
All AI agents got a single shot with no feedback loop. Some of them decided to ask questions; we always picked the option they marked as "recommended". We stopped after a single turn because we didn't want results influenced by follow-up conversations, which would differ based on the outcome of the previous turn.
The vast majority of real-life software tasks are open-ended. There isn't one right answer for how monocrate's architecture/implementation should look. It doesn't have to be perfect on the first try — what most developers want most from their coding agents is a good first leap forward, which may or may not be followed by additional iterations.
Our judgment scheme reflects this reality: we're judging solutions to an open-ended problem and want to do it as objectively as possible. Hence the pairwise matchups (so each judge focuses on "is A better than B?") and the spread among multiple judges.
Full round-robin: all 10 pairwise matchups were evaluated.
Each matchup was judged by 7 LLM models:
- OpenAI GPT-5 Mini
- Anthropic Claude Sonnet 4.5
- DeepSeek v3.2
- Google Gemini 2.5 Flash
- Mistral Devstral 2512
- Perplexity Sonar Pro
- Qwen3 Coder 30B
Each judge saw both repos with labels anonymized (presented as "Repo A" and "Repo B"). To control for position bias, each judge evaluated each pair twice with the positions swapped, yielding 14 votes per matchup (7 judges × 2 rounds). In total: 10 pairwise matchups × 7 judges × 2 evaluations = 140 judgments.
We did encounter judges influenced by position order. The scoring scheme (2 points for win, 1 for tie, 0 for loss) naturally balances this: if a judge changes its mind across the two rounds, the points split evenly. We iterated on the judge list several times until reaching this set where position bias was minimal.
Judges were asked to pick which repo is a better starting point — not whether the code works. A win means a judge thought it was the stronger foundation.
- LLM-as-judge bias: LLM judges tend to over-weight test coverage and documentation. They sometimes contradict themselves across position swaps (which is why we do the swap).
- No execution: judges evaluated code by reading it, not by running it. A passing test suite would be a stronger signal than a judge's opinion of the test code.
- Win rate interpretation: win rates are relative to the competitor pool. Adding many weak competitors would compress all results into a narrow high band; adding many strong competitors would compress them into a narrow low band. The ranking order is more meaningful than absolute percentage gaps.
All judge verdicts, reasoning, and vote counts are available in the full judgements JSON file.
Each judge received this prompt template:
You are judging two implementations of the same mission.
Pick the one we should merge and continue with.
Avoid position bias: the order of implementations is arbitrary.
Focus on observable compliance with the specification.
SPECIFICATION (from ${input.specificationPath}):
${input.spec}
IMPLEMENTATION repoA (path: ${input.implementationAPath}):
${input.repoASnapshot}
IMPLEMENTATION repoB (path: ${input.implementationBPath}):
${input.repoBSnapshot}
Respond as JSON only: {"winner":"repoA|repoB|tie","reasoning":"...","summary":"..."}
node --enable-source-maps ~/code/moojo-tech/antelope/modules/codeoff-bench/dist/src/main \
--output text \
-c ../../moojo-tech/monocrate/ \
-c=45 -c=46 -c=52 -c=53 \
-s spec \
-j openai/gpt-5-mini \
-j anthropic/claude-sonnet-4.5 \
-j deepseek/deepseek-v3.2 \
-j google/gemini-2.5-flash \
-j mistralai/devstral-2512 \
-j perplexity/sonar-pro \
-j qwen/qwen3-coder-30b-a3b-instruct