text-only-models-with-mm

gistfile1.txt

# Text-Only Models with mm-ctx Vision Toolkit vs. Native Vision Models

## Summary

This report shows the result of a benchmark that compares two types of AI agent setups on a multimodal task suite. The first type uses a text-only model with a vision-backed toolkit (`mm-ctx`). The second type uses a model that can process images natively.

The benchmark uses 23 tasks across three categories: retrieval, artifact creation, and file organization. Each task includes images, video, audio, PDFs, or a mix of these formats. Each setup runs each task once. A deterministic checker and an LLM judge score each result. The score is a 50/50 blend of the two.

Four setups were tested:

| Setup | Agent model | Vision capable | Toolkit backend |
|---|---|---|---|
| A | gemini-3.5-flash | Yes | None |
| B | qwen3.6-35b-a3b | Yes | None |
| C | glm-5.2 + mm-ctx | No | gemini-3.1-flash-lite |
| D | deepseek-v4-pro + mm-ctx | No | gemini-3.1-flash-lite |

All models were accessed through OpenRouter. The same agent harness (`pi`) ran all four setups in non-interactive mode with shell, file, and edit tools enabled. In the toolkit arms, the agent calls `mm` CLI commands (find, peek, cat, grep, sql, wc) to inspect and extract content from media files. The toolkit backend model does the vision work and returns text to the agent. In the no-toolkit arms, the agent has only its native tools. The same judge model (gemini-3.1-flash-lite) scored all results. Each task has a time limit of 360 to 600 seconds.

## Results

### Overall scores

| Rank | Setup | Mean correctness | Mean speed | Completed | Timeouts | Total tokens |
|---|---|---|---|---|---|---|
| 1 | gemini-3.5-flash | 85.5 | 162s | 21/23 | 1 | 566K |
| 2 | glm-5.2 + mm-ctx | 75.9 | 208s | 19/23 | 4 | 1.34M |
| 3 | deepseek-v4-pro + mm-ctx | 68.8 | 204s | 20/23 | 3 | 999K |
| 4 | qwen3.6-35b-a3b | 46.0 | 144s | 17/23 | 2 | 432K |

gemini-3.5-flash holds the top position. The best text-only setup (glm-5.2 + mm-ctx) trails by 9.6 points but closes 83% of the gap.

### Per-task results

| Task | Category | Modality | gemini-3.5 | qwen3.6 | glm + mm-ctx | ds + mm-ctx |
|---|---|---|---|---|---|---|
| audio-claims-json | artifact | audio | 80.0 | 0.0 | 0.0 | 12.5 |
| audio-earnings-figure | retrieval | audio | 100.0 | 0.0 | 100.0 | 0.0 |
| bank-statement-reconcile | artifact | pdf | 100.0 | 0.0 | 100.0 | 100.0 |
| control-code-bug | retrieval | doc | 100.0 | 0.0 | 100.0 | 55.0 |
| control-config-grep | retrieval | doc | 100.0 | 100.0 | 100.0 | 100.0 |
| control-tiny-obvious | retrieval | image | 100.0 | 100.0 | 100.0 | 100.0 |
| cross-modal-order-match | retrieval | pdf, image | 100.0 | 100.0 | 100.0 | 100.0 |
| dedup-photos | organization | image | 8.3 | 8.3 | 8.3 | 66.7 |
| deep-pdf-crossref | retrieval | pdf | 100.0 | 100.0 | 100.0 | 100.0 |
| downloads-inventory-csv | artifact | mixed | 100.0 | 100.0 | 100.0 | 25.0 |
| flag-medical-forms | organization | pdf | 40.0 | 50.0 | 80.0 | 100.0 |
| floorplan-room-report | artifact | image | 0.0 | 0.0 | 50.0 | 50.0 |
| invoices-to-csv | artifact | pdf, image | 57.5 | 37.5 | 47.5 | 37.5 |
| legal-clause-find | retrieval | pdf | 100.0 | 80.0 | 80.0 | 100.0 |
| ocr-needle-at-scale | retrieval | image | 100.0 | 0.0 | 0.0 | 0.0 |
| organize-by-subject | organization | image | 100.0 | 60.0 | 100.0 | 100.0 |
| paper-results-json | artifact | pdf | 100.0 | 100.0 | 100.0 | 100.0 |
| quarantine-non-floor-plans | organization | image, pdf | 80.0 | 21.4 | 80.0 | 80.0 |
| retrieve-sec-filing | retrieval | pdf | 100.0 | 100.0 | 100.0 | 100.0 |
| retrieve-video-product | retrieval | video | 100.0 | 0.0 | 100.0 | 100.0 |
| scanned-forms-extract-json | artifact | pdf | 100.0 | 100.0 | 0.0 | 67.5 |
| semantic-scene-retrieval | retrieval | image | 100.0 | 0.0 | 100.0 | 0.0 |
| video-timeline-json | artifact | video | 100.0 | 0.0 | 100.0 | 87.5 |

### Scores by task category

| Category | gemini-3.5 | glm + mm-ctx | ds + mm-ctx | qwen3.6 |
|---|---|---|---|---|
| Retrieval (11 tasks) | 100.0 | 89.1 | 68.6 | 52.7 |
| Artifact (8 tasks) | 79.7 | 62.2 | 60.0 | 42.2 |
| Organization (4 tasks) | 57.1 | 67.1 | 86.7 | 34.9 |

### Scores by data modality

| Modality | n | gemini-3.5 | glm + mm-ctx | ds + mm-ctx | qwen3.6 |
|---|---|---|---|---|---|
| Audio | 2 | 90.0 | 50.0 | 6.3 | 0.0 |
| Document | 2 | 100.0 | 100.0 | 77.5 | 50.0 |
| Image | 6 | 68.1 | 59.7 | 52.8 | 28.1 |
| PDF | 7 | 91.4 | 80.0 | 95.4 | 75.7 |
| Video | 2 | 100.0 | 100.0 | 93.8 | 0.0 |

## Key findings

**1. gemini-3.5-flash is the clear overall winner.** It scores 85.5 mean correctness and completes 21 of 23 tasks. It is the only setup that scores 100 on all retrieval tasks. It also leads on audio (90.0), where every other setup struggles. It has only one timeout.

**2. glm-5.2 + mm-ctx is the best text-only setup.** It scores 75.9, which is 9.6 points below gemini-3.5-flash. It matches gemini on video (100.0) and documents (100.0). It stays close on retrieval (89.1 vs 100.0). The gap is largest on audio (50.0 vs 90.0) and image tasks (59.7 vs 68.1). On these modalities, the toolkit adds latency that causes timeouts.

**3. deepseek-v4-pro + mm-ctx is the strongest setup for organization tasks.** It scores 86.7 on organization, which is the best of all four setups. It wins `flag-medical-forms` (100.0) and `dedup-photos` (66.7), where all other setups score low. It is weak on retrieval (68.6) with three timeouts. It also fails on audio (6.3). The deepseek agent reasons well about file operations but has difficulty with the toolkit loop on media-heavy retrieval.

**4. qwen3.6-35b-a3b is the weakest setup, although it can process images.** It scores 46.0 and gets zero on 11 of 23 tasks. Many of these failures occur in under 30 seconds, which shows that the model stops before it does the work. It scores 100 on only 6 tasks. Vision capability does not help when the model does not use tools. Its low token count (432K) confirms that it does not engage deeply with the tasks.

**5. Text-only setups with a toolkit spend more tokens.** glm-5.2 + mm-ctx uses 1.34M tokens (388K agent + 949K toolkit). deepseek-v4-pro + mm-ctx uses 999K tokens (511K agent + 488K toolkit). gemini-3.5-flash uses 566K tokens. qwen3.6-35b-a3b uses 432K tokens. The toolkit backend adds a token cost on top of the agent cost. This cost is the price of bridging the modality gap with a toolkit.

**6. Timeouts have a large effect on the scores.** When a task times out, the judge score is set to zero. This penalizes the setup symmetrically. glm-5.2 + mm-ctx has 4 timeouts. deepseek-v4-pro + mm-ctx has 3. gemini-3.5-flash has 1. qwen3.6-35b-a3b has 2. The text-only setups lose approximately 15 to 20 points to timeouts. Most timeouts occur on bulk-image and dense-PDF tasks, where the toolkit makes one LLM call per file. These calls add up and exceed the task time limit.

**7. gemini-3.5-flash is the only setup that wins a task alone.** It uniquely wins 3 tasks: `audio-claims-json`, `ocr-needle-at-scale`, and `floorplan-room-report`. deepseek-v4-pro + mm-ctx uniquely wins 2 tasks: `dedup-photos` and `flag-medical-forms`. glm-5.2 + mm-ctx and qwen3.6-35b-a3b do not uniquely win any task. Six tasks end in a four-way tie at 100.0.

## Caveats

- **Single run per setup.** Each setup runs each task once. There is no variance measurement. Close results (for example, `invoices-to-csv`: 57.5 vs 47.5 vs 37.5) could change with a second run.
- **The judge model shares a family with setup A and the toolkit backend.** The judge is gemini-3.1-flash-lite. Setup A uses gemini-3.5-flash. The toolkit backend also uses gemini-3.1-flash-lite.
- **The qwen3.6-35b-a3b result may reflect a harness issue, not only model capability.** Many of its zero-score tasks finish in under 30 seconds. This suggests that the model does not start the task, rather than it fails the task. The compatibility between `pi` and this model needs more investigation.
- **The toolkit backend is a workhorse model, not the strongest vision model.** The toolkit uses gemini-3.1-flash-lite for media extraction. A stronger vision model on the toolkit backend would likely improve the text-only setups on image and audio tasks. This would also increase the token cost.

## Verdict

gemini-3.5-flash stays the model to beat at 85.5 mean correctness. The best text-only setup (glm-5.2 + mm-ctx at 75.9) closes 89% of the gap to the native vision model.

The toolkit adds the most value on video tasks, document tasks, and organization tasks. On video and documents, glm-5.2 + mm-ctx matches gemini-3.5-flash at 100.0. On organization tasks, deepseek-v4-pro + mm-ctx beats gemini-3.5-flash (86.7 vs 57.1).

The toolkit adds the least value on audio and bulk-image tasks. On these tasks, the per-file LLM calls cause timeouts that the native vision model avoids. The token cost premium of 1.8x to 2.4x is the tradeoff for bridging the modality gap with a toolkit instead of native vision.

For a text-only model that must process media, a vision-backed toolkit is a practical and competitive alternative to a native vision model. The toolkit does not replace native vision. It extends a text-only model to the modalities where directory-scale operations matter more than single-file perception.

---

We are continually improving mm-ctx. A follow-up benchmark will use 3 to 5 runs per setup for variance, stronger native vision models for comparison, and harder tasks. The goal is to show that small text-only models with the mm-ctx toolkit can perform close to state-of-the-art native vision models.

[mm-ctx](https://github.com/vlm-run/mm) is an open-source project by [VLM Run](https://vlm.run).