Benchmarking multi-turn LLM workloads with AIPerf — 100 conversations × 12 turns × 32K ISL / 1K OSL with shared system prompt + KV reuse

aiperf-multiturn-gist.md

Benchmarking multi-turn LLM workloads with AIPerf

Goal: Simulate 100 multi-turn conversations averaging 12 turns each, with 60K avg ISL / 1K avg OSL, hitting 95% KV cache reuse — a large shared system prompt across all conversations plus accumulating turn-over-turn history.

This is a useful pattern for stress-testing KV-aware routing in disaggregated inference, prefix-cache hit rates, and goodput under realistic agentic-conversation load. It's also the rough shape of real coding-agent traffic (Claude Code, Cursor, Cline, etc.) — long shared system prompt + turn-over-turn growth.

The command

aiperf profile \
  --model Qwen/Qwen3-235B-A22B-Instruct \
  --url https://your-endpoint/v1 \
  --endpoint-type chat --streaming \
  --api-key "$API_KEY" \
  --concurrency 8 \
  --conversation-num 100 \
  --conversation-turn-mean 12 \
  --conversation-turn-stddev 3 \
  --num-prefix-prompts 1 \
  --prefix-prompt-length 35000 \
  --isl 3000 \
  --osl 1000

Replace --url and --api-key with whatever OpenAI-compatible endpoint you're testing — vLLM, SGLang, TRT-LLM, NIM, or any cloud provider's chat-completions API.

How each flag maps to the spec

Spec	Flag	Notes
100 conversations	--conversation-num 100	exact count
Avg 12 turns	--conversation-turn-mean 12 --conversation-turn-stddev 3	normal-ish distribution; ~95% of conversations land between 6 and 18 turns
Shared system prompt	--num-prefix-prompts 1 --prefix-prompt-length 35000	pool of 1 prefix → every conversation samples the same 35K-token system prompt. That's the "shared across all conversations" L1 layer — realistic for tool-heavy agentic configs (Claude Code, Cursor, Cline).
Avg OSL 1K	--osl 1000	per-turn assistant response
Avg ISL 60K, 95% cache hit	--isl 3000	this is the per-turn user-message size. The 3K of new tokens per turn is exactly 5% of the 60K avg input — the other 95% is L1 prefix + accumulated turn history.

The ISL + cache-hit math

--isl controls the per-turn user message length, not the full per-request input. The per-request input grows as the conversation accumulates context:

Turn N input  =  35,000 (shared system prompt)
              +  N × 3,000     (cumulative user messages)
              +  (N–1) × 1,000 (cumulative assistant responses)

Summing across turns 1–12 gives 720,000 tokens total / 12 = 60,000 avg ISL per turn.

Turn	Per-turn input	Cached	New
1	38K	35K	3K
6	58K	55K	3K
12	82K	79K	3K
Average	60K	57K	3K

Across the full benchmark: 720K total input tokens, 36K of which are net-new = 95% cache reuse.

Why this shape?

The split between large shared system prompt and accumulating turn history is what real agentic workloads look like. A coding agent's full prompt is typically:

• Big fixed system prompt with instructions, tool definitions, schemas, persona — shared across every session, often 20–40K tokens
• Growing context — codebase snippets, tool outputs, prior turns — unique per session, accumulates with the conversation

KV-aware routers should hit the system prompt on every request (cross-session L1 cache) and within-session prefix on every turn after the first (per-session L2 cache). At 95% cache reuse, a well-implemented disaggregated stack is doing 20× less prefill than a stateless one — the spread between a router that nails it and one that doesn't is enormous, which is why this workload is a useful stress test.

Goodput, automatically

--endpoint-type chat --streaming populates TTFT, ITL, decode tok/s, and goodput vs SLO automatically. If you want to define explicit SLO gates:

  --goodput-ttft 500 \
  --goodput-itl 30

(values in milliseconds)

Variants

Per-turn length variance — add stddev to the user-message and assistant-response lengths:

  --isl-stddev 800 --osl-stddev 200

Single-stream KV-reuse signal — drop concurrency to 1 to remove router contention and isolate cache behavior:

  --concurrency 1

Smaller system prompt, longer conversations — keep the same 60K avg ISL and 95% cache reuse, but with a more modest 6K shared prefix and ~27 turns of accumulation instead of a 35K prefix and 12 turns:

  --conversation-turn-mean 27 \
  --prefix-prompt-length 6000 \
  --isl 3000

Multiple shared prefix pools (model "tenants" or "projects" sharing context across cohorts of sessions):

  --num-prefix-prompts 10 \
  --prefix-prompt-length 35000

Each conversation now samples one of 10 distinct 35K-token prefixes — modeling a workload with multiple tenants/projects sharing context across cohorts of sessions.

Total request count + wall-clock

100 conversations × ~12 turns ≈ 1,200 requests. At --concurrency 8 against a healthy endpoint, expect 15–25 minutes wall-clock. Drop concurrency for clean single-stream signal; raise for full load.

What you get back

A standard AIPerf run produces:

• Live TUI dashboard with TTFT, ITL, decode tok/s, goodput vs SLO
• CSV / JSON / Parquet exports of per-request raw timings
• Server-side metrics scrape if your endpoint exposes Prometheus (vllm:*, sglang:*, including vllm:spec_decode_draft_acceptance_rate and friends if speculative decoding is on)
• Multi-run statistical aggregation when you re-run

See also

• github.com/ai-dynamo/aiperf — full repo, Apache-2.0
• pip install aiperf
• aiperf profile --help — every flag documented in the CLI

---

Generated 2026-05-06 as a reference snippet for the NVIDIA Inference Co-design Days talk on disciplined inference benchmarking.