AI Simulate Sweeper for DGDR v1beta2 — Where It Helps and Where It Doesn't

ai-simulate-sweeper-for-dgdr-v1beta2.md

AI Simulate Sweeper for DGDR v1beta2 — Where It Helps and Where It Doesn't

Experimental: Sweeper and the DGDR v1beta2 integration are still evolving.

Management Summary

Finding	Decision or number
Decision	Adopt Sweeper as DGDR v1beta2's search and offline simulation engine.
Current repository status	11 fixes merged; 1 duplicate with its underlying issue fixed; 1 partial fix; 1 path requires revalidation.
Historical sub-issue ownership: handled by Sweeper	2/14 findings = 14.3%
Historical sub-issue ownership: risk reduced by the Sweeper path	2/14 findings = 14.3%
Historical sub-issue ownership: other integration layers	10/14 findings = 71.4%
Largest owner among those 14 findings	Candidate-to-DGD/runtime materialization: 6/14 findings = 42.9%
What these percentages do not measure	Current model coverage, usable recipe combinations, or the share of DGDR v1 failures fixed by Sweeper.
Consequence	Materializer, version binding, run lifecycle, and real deployment validation remain first-class work.

This is an ownership analysis, not a defect count or a quality comparison. In this validation sample, Sweeper handles two paths directly and reduces risk for two more. The remaining ten identify work in materialization, operator lifecycle, GPU accounting, and documentation. Together, the first two classes cover 4/14 findings (28.6%); this combined number is not another result class. The denominator contains only the 14 child issues. It excludes the parent issue's model-coverage analysis and therefore must not be used to estimate how many models DGDR v1beta2 can support.

Responsibility Boundary

flowchart LR
    DGDR["DGDR"] --> RUN["Immutable DGDRRun"]
    RUN --> MAP["SmartSearchConfig mapper"]
    MAP --> SWEEP["AI Simulate Sweeper"]
    SWEEP <--> REPLAY["Dynamo offline Replay"]
    SWEEP --> EVENTS["Progress and Candidate events"]
    EVENTS --> PUB["Publisher sidecar"]
    PUB --> STATUS["DGDRRun status"]
    PUB --> DGDC["Bounded DGDC set"]
    DGDC --> DGD["Explicit DGD creation"]
    OVERRIDES["DGD overrides"] --> PUB
    VERSION["Runtime/backend versions"] --> PUB

Loading

Sweeper owns	DGDR integration owns
Search-space validation and sampling	DGDR-to-SmartSearchConfig mapping
Parallel-shape enumeration	Run and Job lifecycle
Offline Replay execution	Progress-event transport and status
Scoring, scalar ranking, Pareto front	Bounded and diverse DGDC publication
Trial failure and timeout isolation	Candidate-to-DGD materialization
Run-local result cache	Promotion and deployment validation

Model and Performance-Data Coverage Gate

Sweeper improves search and system simulation only after the requested model, hardware, backend, and performance-model version can be resolved. The current path still uses AI Configurator for model facts, system facts, KV-memory feasibility, and forward-pass timing. Its KV estimator disables the naive fallback deliberately. An unsupported combination therefore cannot produce a scored Candidate or a DGDC.

The parent DGDR v1 validation issue reported a coverage problem that is not represented by one of its 14 child issues:

State reported by the v1 validation	Combinations	What it means for Sweeper
Not in AIC	DeepSeek-R1, Llama-3.3-70B FP8, Nemotron-3-Super 120B FP8, Qwen3-VL-30B FP8, Kimi-K2.5-NVFP4	Still unsupported unless current AIC model and performance data now cover the requested tuple.
In AIC, but all reported configurations failed	DeepSeek-V3.2-NVFP4, GLM-5-NVFP4	Requires current revalidation. DeepSeek-V3.2 had a known TRT-LLM MoE conversion failure; the parent issue did not identify a root cause for GLM-5.
SKU/backend mismatch	Qwen3-32B-FP8 with vLLM on A100	Search cannot make an unsupported tuple valid.
Backend-version regression	Qwen models with SGLang 0.5.9	Coverage must be bound to the backend version, not only model and hardware names.

This table records the state at the time of that issue; it is not a current support matrix. The MVP must re-run the listed recipe combinations against today's Sweeper and report missing model facts, performance data, hardware, backend, and backend-version coverage as explicit unsupported run conditions. It must not generate a naive deployment whose objective was never evaluated.

Where Sweeper Helps

Capability	Concrete benefit	Limit
KV-capacity estimation	Rejects more OOM-prone shapes before Replay	Model-based estimate, not real engine startup
Hard gpu_budget	Rejects over-budget Candidates	Publisher must preserve worker counts and resources
System Replay	Evaluates workload, fleet, Router, and Planner interactions	Uses simulated engines, not real GPUs
Isolated trials	Failed, unsupported, or timed-out points do not abort the search	Does not manage Kubernetes Job retries
Scalar and Pareto optimization	Returns ranked results or a non-dominated front	No diverse top-N projection for Kubernetes objects
on_round callback	Enables periodic best-known Candidate updates	Does not expose full evaluation progress or failures

Current implementation limits:

• Dynamo Replay supports vLLM and SGLang in aggregated and disaggregated mode.
• TensorRT-LLM is supported only in aggregated mode.
• Automatic enumeration pins pipeline parallelism to one.
• Model, hardware, KV, and performance coverage still depend on AI Configurator data.
• Sweeper currently resolves the latest performance-database version independently of the target Dynamo runtime.

Where Sweeper Does Not Help

Concern	Owner
DGD components, images, commands, arguments, and environment	Version-aware DGD materializer
Model cache PVCs, paths, revisions, and remote-code policy	DGDR API and materializer
Node count, placement, RDMA, EFA, and cloud resources	DGD overrides and operator
Exact vLLM/SGLang/TensorRT-LLM CLI compatibility	Backend-specific materializer
Real engine memory use and startup	Deployment/conformance tests
Job retry, cancellation, resume, and garbage collection	DGDRRun reconciler
Durable reports, logs, and UI data	Publisher and UI service
Promotion and traffic cutover	User or separate orchestrator

Sweeper also does not make unsupported models or hardware simulatable. Missing performance data must produce an explicit unsupported result, never a naive single-GPU fallback.

DGDR v1 Validation Findings

Method

• Denominator: the 14 Dynamo sub-issues in the DGDR v1 validation tracking issue.
• Issue closure is not treated as proof of a fix. The table checks merged PRs and current code.
• Handled by Sweeper: Sweeper owns the relevant behavior directly.
• Risk reduced: the v1 path is removed, but DGDR integration must preserve the relevant data or behavior.
• Other integration layer: materialization, operator lifecycle, semantics, or documentation owns the behavior.

Issue and failure	Current v1 status	Sweeper effect	DGDR v1beta2 responsibility
#8273: MoE fields lost before Planner interpolation	Fixed: #8335	Risk reduced: no separate interpolation path	Event and materializer must preserve TP/DP/MoE dimensions.
#8480: TRT-LLM DGD/service names too long	Fixed: #8563; #8807 made violations terminal	Other integration layer	Materializer and operator must generate and validate safe names.
#8566: same TRT-LLM/Grove naming failure	Duplicate: underlying issue fixed by #8563	Other integration layer	Same responsibility as #8480; not a separate v1beta2 work item.
#8567: wrong multinode.nodeCount	Fixed: merged with #8617	Other integration layer	Materializer must distinguish per-worker GPUs from replicas and placement.
#8568: model-cache path doubled	Fixed: #9177	Other integration layer	Materializer owns model identity, PVC mounts, and container paths.
#8571: OOM/over-budget points and slow failure recovery	Partial: enumeration fixed by #8479; Job retry fixed by #9166; early CrashLoopBackOff detection was deferred in #9213	Risk reduced: KV checks, budget gating, and offline Replay avoid the reported per-trial deployment path	Real promoted DGDs can still OOM; validation and run failure reporting remain required.
#8576: Attention DP counted as physical GPUs	Fixed: #8610	Other integration layer: current Sweeper computes tp * pp * attention_dp	Define and test physical GPU accounting across Sweeper, DGDC, and DGD.
#8579: TRT-LLM MoE converter rejected all points	Requires revalidation: the current v1beta1 TrtllmConfigModifier still rejects this configuration	Other integration layer: TRT-LLM disaggregated Replay is unsupported and Sweeper emits no DGD	Backend support and a version-aware TRT-LLM materializer are required.
#8583: 50-GPU result for a 32-GPU budget	Fixed: #8617	Handled by Sweeper: enumeration enforces gpu_budget	Publisher must preserve the selected worker counts and GPU resources.
#8638: decode Pod logs missing	Fixed: #8733 and #9171	Handled by Sweeper: offline Replay returns metrics in-process	No per-Candidate Pod-log scraping in the Sweeper path.
#8648: failed sidecar retried the whole Job	Fixed: #9166	Other integration layer	DGDRRun reconciler and publisher termination policy own Job behavior.
#8659: conflicting TRT-LLM CLI forms	Fixed: #9107	Other integration layer: Candidate output contains no TRT-LLM command line	Version-aware backend materializer must produce one valid CLI form.
#8658: Kubernetes and DGDR documentation gaps	Docs fix merged: #8777; product gaps listed as out of scope were not fixes claimed by this issue	Other integration layer	v1beta2 user documentation and explicit unsupported conditions remain DGDR work.
#9229: hard-coded NATS/etcd endpoints	Fixed: #9271	Other integration layer	Run Job construction must use operator configuration and least-privilege credentials.

Historical Sub-Issue Ownership Result

Category	Issues	Count	Share
Handled by Sweeper	#8583, #8638	2	14.3%
Risk reduced; integration validation required	#8273, #8571	2	14.3%
Owned by other integration layers	all others	10	71.4%

The first two classes together cover 4/14 findings (28.6%). This is a combined observation, not another result class. These percentages classify responsibility for the 14 child findings. They do not measure model coverage, successful recipe combinations, remaining unfixed defects, or overall Sweeper effectiveness.

Ownership of the Other 10 Findings

Remaining owner	Issues	Count	Share of all 14
Candidate-to-DGD/runtime/Kubernetes materialization	#8480, #8566, #8567, #8568, #8579, #8659	6	42.9%
DGDR operator and Job lifecycle	#8648, #9229	2	14.3%
Parallelism/GPU-accounting semantics	#8576	1	7.1%
Mixed documentation and support coverage	#8658	1	7.1%

These are ownership classes, not claims that the findings remain unfixed in v1. Candidate-to-DGD materialization is the primary v1beta2 integration workstream.

Candidate-to-DGD Contract

DGDR v1: AIC picker output

The v1 rapid path does not receive a DGD directly from the AIC optimizer. AIC first returns a pandas.Series selected from its result table. This is an illustrative real result from the installed AIC version for Qwen3-32B on H200 with vLLM; values are not a stable API contract:

model: Qwen/Qwen3-32B
system: h200_sxm
backend: vllm
version: 0.24.0

parallel: tp2pp1dp1etp1ep1
tp: 2
pp: 1
dp: 1
moe_tp: 1
moe_ep: 1
num_total_gpus: 2
bs: 96

gemm: bfloat16
moe: bfloat16
kvcache: bfloat16
fmha: bfloat16
comm: half
memory: 93.345

concurrency: 96
ttft: 522.631
tpot: 45.586
request_latency: 46063.429
request_rate: 2.084
tokens/s: 2081.999
tokens/s/gpu: 1040.999

# AIC also returned operation-level empirical/silicon provenance in _per_ops_source.

DGDR v1 then combines that row with the original AIC Task through task_config_to_generator_config. The same example produced this intermediate generator input:

ServiceConfig:
  model_path: Qwen/Qwen3-32B
  served_model_path: Qwen/Qwen3-32B
  include_frontend: true
K8sConfig:
  k8s_image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.0
DynConfig:
  mode: agg
  enable_router: false
WorkerConfig:
  agg_workers: 8
  agg_gpus_per_worker: 2
NodeConfig:
  num_gpus_per_node: 8
  system_name: h200_sxm
params:
  agg:
    tensor_parallel_size: 2
    pipeline_parallel_size: 1
    data_parallel_size: 1
    gpus_per_worker: 2
    moe_tensor_parallel_size: 1
    moe_expert_parallel_size: 1
    max_batch_size: 96
    gemm_quant_mode: bfloat16
    moe_quant_mode: bfloat16
    kvcache_quant_mode: bfloat16
    fmha_quant_mode: bfloat16
    comm_quant_mode: half
    memory: 93.345

The difference between num_total_gpus: 2 in the selected row and eight two-GPU workers in the generator input is intentional current behavior of this example: the bridge filled the original 16-GPU task budget because the row had no total_gpus_needed field. It demonstrates why the Candidate-to-generator boundary must define worker-count semantics explicitly.

Finally, v1 calls generate_backend_artifacts(..., use_dynamo_generator=True). AIC resolves backend CLI rules and versioned templates; its dynamo-python target then invokes Dynamo's CONFIG_MODIFIERS[backend].build_dgd_config to assemble the DGD. The DGD is therefore a combined AIC-generator and Dynamo-materializer output, not raw optimizer output.

Source map for the v1 template and rendering path

DGDR v1 has concrete DGD base templates under components/src/dynamo/profiler/templates/dgd: aggregate and disaggregated templates for vLLM, SGLang, and TensorRT-LLM, plus a disaggregated Mocker template. These are the actual Kubernetes DGD templates used by the v1 materializer. They are one layer of the complete rendering path: AIC first selects and renders backend-specific values, then Dynamo injects those values into the base templates and applies DGDR-specific transforms. The Dynamo links below are pinned to the source revision inspected for this document. The AIC links show the v0.11 generator layout; the output example above ran with aiconfigurator==0.11.0.dev20260728.

1. AIC selection and bridge: Dynamo's rapid.py receives the picker row. AIC's module_bridge.py combines that row with the original Task and lowers both into generator parameters.
2. AIC backend rendering: The rendering engine combines the deployment schema, backend parameter mapping, backend-version matrix, versioned backend CLI templates, rule plugins, and model/hardware/runtime facts.
3. Concrete Dynamo DGD base templates and assembly: For DGDR v1's dynamo-python target, AIC calls the backend config modifiers. They load the vLLM, SGLang, and TensorRT-LLM aggregate/disaggregated base YAMLs and the Mocker disaggregated base YAML through dgd_template.py, then inject model identity, generated CLI tokens, replicas, GPU resources, PVCs, and component shape.
4. DGDR-specific transforms: dgd_materialization.py applies DGD overrides, runtime constraints, tolerations, and remote-code handling. dgd_generation.py then layers Planner, Mocker, profile data, ConfigMaps, and runtimeVersionOverride onto the picked base DGD.
5. Thorough-mode candidate templates: Dynamo's thorough.py asks AIC's enumerate.py to build temporary per-role benchmark DGDs through the same Dynamo modifiers before deploying them. The final combined pick returns to the rapid generator path.
6. Newer typed reuse boundary: AIC now exposes a typed GeneratorRequest and a typed Kubernetes DGD builder. These are better prototype targets than preserving the legacy pandas/Task bridge.

Sweeper Candidate output

Sweeper returns a Candidate with a concrete evaluated search point in config, used_gpus, metrics, a scalar score, and optional Pareto objectives. The core backend configuration is flat; resolved adapter configuration remains nested by provider. A real one-round projection for the same model/hardware/backend input had this shape; Replay metrics are omitted because that probe used a deterministic test runner:

config:
  deployment_mode: agg
  backend: vllm
  backend_version: 0.24.0

  model_name: Qwen/Qwen3-32B
  hardware_sku: h200_sxm
  gpu_budget: 16
  context_length: null

  tp: 4
  pp: 1
  attention_dp: 1
  moe_tp: 1
  moe_ep: 1
  strategy: tp
  replicas: 4
  used_gpus: 16

  agg_max_num_batched_tokens: 16384
  agg_max_num_seqs: 512
  agg_block_size: 64
  agg_gpu_memory_utilization: 0.9
  agg_enable_prefix_caching: true

  concurrency: 96
used_gpus: 16
score: <normalized objective score>
metrics: <ReplayReport metrics>
objectives: null

The ReplaySpec passed to the runner is richer than the public Candidate. For this projection its backend portion also carries explicit num_workers and per-role engine arguments, including the AIC model/system/version identity, batching limits, block size, memory fraction, and prefix caching. The current Candidate does not retain that complete object after evaluation.

Output comparison

Information	v1 AIC picker row	Sweeper Candidate and ReplaySpec
Model, system, backend, performance-data version	Present	Present
Concrete parallel shape and workers	Shape present; worker count may be derived or rewritten by Task/bridge behavior	Present explicitly in the projected config and BackendDeploymentSpec
Engine knobs	Batch size, quantization, memory estimate; generator fills further defaults	Explicit batching, block-size, memory-utilization, prefix-cache, and per-role Replay engine arguments
Performance result	Predicted TTFT/TPOT, throughput, request rate, memory, and per-operation provenance	Replay metrics, normalized score, optional Pareto objectives, and system-level adapter effects
Router and Planner selection	Added later by DGDR-specific code	Concrete adapter selections and runtime hooks in the Replay specification
Kubernetes/DGD data	Neither the picker row nor Task contains a DGD	Neither Candidate nor ReplaySpec contains a DGD

Sweeper output is therefore not simply a strict superset of the old AIC row. It is richer about the exact evaluated system configuration and adapter behavior. The old result contains useful AIC micro-model diagnostics and provenance that Sweeper does not currently publish.

Generator reuse

The existing AIC generator is a strong reuse candidate. It already owns backend facts, versioned CLI templates, engine-argument rendering, typed GeneratorRequest, and DGD builders. Reuse should not mean reconstructing a legacy Task and fake pandas row. The prototype should instead:

1. map the resolved Sweeper Candidate plus the selected ReplaySpec and DGDR fields into AIC's typed GeneratorRequest;
2. render the backend CLI and base DGD with the existing generator;
3. apply DGDR model-cache, security, runtime-version, placement, and DGD overrides deliberately; and
4. verify that worker counts, GPU use, backend version, and every searched engine knob survive byte-for-byte or semantically equivalent into the DGD.

This also affects process ownership. A Go publisher cannot directly reuse the Python AIC generator. If the publisher remains Go, the Python Job must emit either a resolved GeneratorRequest or an already materialized DGD-spec payload with each Candidate event. Reimplementing AIC's backend and version logic in the Go sidecar would create a second templating engine.

Regardless of the reuse choice, the optimizer result is not a DynamoGraphDeployment, and it does not contain:

• DGD apiVersion, kind, metadata, or component graph;
• Frontend and worker container images;
• vLLM, SGLang, or TensorRT-LLM commands and command-line arguments;
• container environment, resources, volumes, model-cache mounts, or security context;
• Kubernetes placement, multinode, Grove, RDMA, EFA, or cloud configuration; or
• the complete ReplaySpec, BackendDeploymentSpec, or adapter runtime hooks used by the simulation.

The last point matters for the Job protocol: the current Candidate alone does not carry enough information for a separate Go publisher to reproduce the evaluated deployment without loss. The Sweeper event must include a deliberately defined resolved Candidate contract, potentially including the selected ReplaySpec or generator result, rather than serializing Candidate blindly.

It does not return a DGD. The materialization and publisher path must create a DGDC containing:

• flat DGD fields;
• backend workers, Frontend, optional Planner, and Router configuration;
• exact images, resources, commands, arguments, and environment;
• DGDR DGD overrides; and
• unstructured parameters for evaluated values that are not DGD fields.

A typed Candidate prevents information loss. It does not perform the translation.

Candidate publication policy:

• Scalar: retain the best-known N.
• Pareto: retain at most recommendation.maxCandidates non-dominated points.
• Pareto N must use an explicit diversity policy; current Sweeper does not provide one.

Runtime and Simulation Version Binding

flowchart LR
    IMAGE["Image tag or runtimeVersionOverride"] --> DYNAMO["Dynamo runtime version"]
    DYNAMO --> MATRIX["Dynamo-to-backend version matrix"]
    MATRIX --> ENGINE["Target engine version"]
    ENGINE --> PERF["Matching performance data"]
    ENGINE --> RENDER["Version-compatible DGD renderer"]
    PERF --> CANDIDATE["Simulated Candidate"]
    RENDER --> DGD["Materialized DGD"]

Loading

Current gap: Sweeper selects the latest AI Configurator performance database for a hardware/backend pair. That version may differ from the engine in the target runtime. The integration must bind both or report the mismatch explicitly.

The DGDR v1 validation campaign did not establish a version mismatch as a root cause. Its findings were primarily about coverage, feasibility, materialization, and operator behavior.

Core Backend Problems the DGDR API Cannot Solve

API and Kubernetes work is tracked in #13545. The problems below must already be solved by a standalone sweeper -> dgd.yaml path; running that path in an operator changes none of them.

Coverage is still a hard input gate

• Problem: An engine may support a model for which Replay has no model facts or performance data.
• Examples from #8469: DeepSeek-R1 was absent from AIC; Qwen3-32B-FP8 had no matching vLLM/A100 data; GLM-5-NVFP4 was known to AIC but produced no viable configuration.
• Required preflight: Check the complete (model, hardware, backend, backend version, topology) tuple before searching.
• Result: Either confirm usable memory/performance data or return a specific unsupported reason. Never turn an unscorable tuple into a naive, unscored deployment.

The returned Candidate is not the evaluated configuration boundary

• Evaluated: ReplaySpec contains concrete worker counts, engine arguments, runtime hooks, and performance-data identity.
• Returned: Candidate contains only config, GPU use, score, metrics, and objectives.
• Missing contract: Return or emit Candidate + exact ReplaySpec; do not reconstruct the latter from the smaller Candidate.config.
• Test: Serialize that pair in a Python fixture and verify that no evaluated value is lost.

DGD generation needs one canonical rendering path

• Existing templates: Concrete vLLM, SGLang, TensorRT-LLM, and Mocker DGD base templates live in components/src/dynamo/profiler/templates/dgd.
• Current v1 path: picker row + Task -> AIC bridge -> backend CLI renderer -> Dynamo modifier -> DGD base template -> DGDR post-processing.
• Observed failure mode: A selected two-GPU AIC result became eight two-GPU workers because the bridge refilled the original 16-GPU Task budget.
• Target path: Use a typed, direct translation:

Candidate + selected ReplaySpec + deployment context
                         |
                         v
                  GeneratorRequest
                         |
                         v
                       DGD

• Deployment context: Runtime/image, model path/cache, remote-code policy, placement, security, and user DGD overrides.
• Decision: Compare AIC's dynamo-python output with its typed builder for identical input, then select one canonical renderer. Do not reimplement backend rendering in the Go publisher.

Simulation, runtime, and renderer versions must be one identity

• Possible mismatch: Sweeper performance database version, engine version in the runtime image, and CLI-renderer version.
• Concrete example: For requested vLLM 0.24.0, the local generator found no exact CLI template and selected 0.20.1.
• Required chain: runtimeVersionOverride -> contained backend version -> matching performance data -> exact compatible renderer.
• Failure policy: Reject an unprovable chain. Do not silently choose the nearest renderer.

Every simulated choice must survive materialization

The materializer must account for each evaluated value:

Sweeper value	Required DGD representation
aggregate/disaggregated mode and worker counts	components and replicas
TP, PP, attention-DP, MoE-TP, MoE-EP	backend arguments or generated engine config
physical GPU count	Pod GPU resources and placement
batching and cache knobs	backend arguments or environment
Router and Planner provider/config	concrete DGD components and configuration

Alongside dgd.yaml, produce a mapping from every searched value to its DGD destination. An unmapped value is an error; silently using an engine default changes the evaluated Candidate.

YAML validity is not backend validity

• v1 examples: Conflicting TensorRT-LLM arguments, incorrect nodeCount, a duplicated model-cache path, excessive resource-name length, and wrong GPU accounting.
• Before Kubernetes:
  • compare normalized output with a hand-written recipe;
  • validate the command line against the pinned backend version/container;
  • round-trip roles, replicas, GPU resources, parallelism, model paths, and adapters back to the selected Candidate.
• Afterwards: Start the DGD on real hardware as a conformance test.

Only then does the DGDR-specific path become mechanical: map API input, run Sweeper, transport the resolved result, and persist DGDRRun/DGDC objects. It must not add another DGD renderer.

References

• DGDR v1 validation campaign
• AI Simulate Sweeper overview
• Sweeper architecture
• Sweeper configuration
• Sweeper results

dgdr-v1-validation-matrix-vs-ai-simulate-sweeper.md

DGDR v1 validation matrix vs. AI Simulate.Sweeper

Assessment date: 2026-08-24
Dynamo revision: 5985e2b59b08ebf1e8430dcb94e240fd28381eae
Source matrix: DGDR v1 validation issue #8469

Summary

This report compares two different evidence sets:

• DGDR v1: the original AIC coverage assessment plus recorded Rapid and Thorough results from issue #8469 and its linked bug reports. The 29 recipe rows were proposed as a target validation matrix; they were not all executed through DGDR v1.
• AI Simulate.Sweeper: the current checks performed for this report against all 29 target rows.

The original issue marked 8/29 rows as AIC-covered and 21/29 as not covered by AIC simulation. DGDR v1 could still generate a naive aggregated fallback for those 21 rows, but without performance estimates or SLA validation. The original issue used ❌ for this state; this report uses ⚠️ to distinguish fallback-only coverage from an observed execution failure.

Current Sweeper coverage is:

• 15/29 rows (52%) pass the base model, hardware, KV-capacity, forward-performance-model, and Replay-runner checks on every listed GPU SKU.
• Of those 15, 5 rows have important recipe semantics that the simulation does not currently represent: KVBM, EAGLE3 speculative decoding, EPP/GAIE, or explicit multi-node networking.
• 1/29 rows (3%) passes only on a subset of its listed GPU SKUs.
• 13/29 rows (45%) are blocked as specified.
• 9/11 model families have at least one usable model/backend/hardware path from the matrix. Nemotron has none; Qwen3-VL requires at least two GB200 GPUs instead of the listed budget of one.

Of the 11 rows that were originally fallback-only but now pass the base Sweeper checks, five use model coverage added explicitly to AIC. The other six pass through AIC's architecture-level forward model without requiring an exact support-matrix entry. A green base result therefore does not by itself mean that the complete recipe, or even the exact model/backend/SKU tuple, has become a published AIC support-matrix entry.

In the comparison table, ✅ AIC simulation means that the target was covered by AIC; ⚠️ Naive fallback only means that v1 could emit an unprofiled aggregated fallback DGD. In an execution column, ✅ means that the named scope succeeded and ❌ means that a recorded execution failed. Neither success mark means that a production-valid DGD was generated and deployed unless the cell explicitly says so.

Recorded DGDR v1 executions

These are the distinct Rapid and Thorough outcomes documented in #8469 and its linked issues. They were run with different branches, images, options, and hardware. They are historical observations, not a controlled 29-row comparison.

Strategy	Model / backend	What completed	Recorded outcome	Evidence
Rapid	Qwen3-0.6B: vLLM, SGLang, TensorRT-LLM	Candidate generation and DGD output with autoApply=false	✅ 3/3 reached DGDR Ready	Rapid validation
Rapid	Qwen3-32B: vLLM, SGLang, TensorRT-LLM	Candidate generation and DGD output with autoApply=false	✅ 3/3 reached DGDR Ready	Rapid validation
Rapid	Qwen3-30B-A3B: SGLang, TensorRT-LLM	Candidate generation and DGD output with autoApply=false	✅ 2/2 reached DGDR Ready	Rapid validation
Rapid	Qwen3-235B-A22B-FP8: TensorRT-LLM, vLLM, SGLang	Candidate generation and DGD output with autoApply=false	✅ 3/3 reached DGDR Ready	Rapid validation
Rapid	Qwen3-235B-A22B-FP8 / TensorRT-LLM, disaggregated, H100	Full auto-apply deployment after fixes	✅ All six pods Ready; DGDR Deployed and healthy	#8579
Rapid + Planner	Qwen3-235B-A22B-FP8 / backend not fixed	Sweep and config selection	❌ Interpolation crashed because MoE parallelism fields were missing	#8273
Rapid	Qwen3-235B-A22B-FP8 / TensorRT-LLM, disaggregated, H100	Profiling and DGD generation	❌ Admission rejected the generated DGD because resource names were too long	#8480
Not stated	Qwen3-235B-A22B-FP8 / TensorRT-LLM, H100	Profiling and DGD generation	❌ Generated multi-node request over-provisioned the cluster; pods remained Pending	#8567
Rapid + Planner	Qwen3-235B-A22B-FP8	Config selection	⚠️ Selected config reported an inflated GPU count; no terminal run result was documented	#8576
Rapid	GPT-OSS-120B / TensorRT-LLM, disaggregated, H100	Profiling and DGD generation	❌ A 32-GPU budget produced a 50-GPU DGD; pods remained Pending	#8583
Rapid	Qwen3-32B / vLLM	Profiling and DGD generation	❌ With modelCache but no pvcModelPath, generated workers crashed on a doubled model path	#8568
Thorough	DeepSeek-R1 / SGLang	Profiling	❌ Failed	Thorough results
Thorough	DeepSeek-V3.2 / SGLang	Profiling	❌ Failed; this backend is not in the 29-row target matrix for that model	Thorough results
Thorough	DeepSeek-V3.1-NVFP4 / TensorRT-LLM	Profiling	❌ Failed; the target matrix contains V3.2-NVFP4	Thorough results
Thorough	Qwen3-235B-A22B-FP8 / TensorRT-LLM	Profiling with Planner disabled	✅ Profiling passed; mode, SKU, and GPU count were not recorded	Thorough results
Thorough	Qwen3-32B / vLLM	Profiling with Planner disabled	✅ Profiling passed; mode, SKU, and GPU count were not recorded	Thorough results
Thorough	Qwen3-235B-A22B-FP8 / TensorRT-LLM, disaggregated, H100	Candidate enumeration	❌ Produced zero candidates because the Thorough conversion path rejected MoE	#8579
Thorough	DeepSeek-R1 / backend not stated, H100	First candidate deployment	❌ Generated TP=1 candidates that OOMed and stalled instead of trying the next candidate	#8571
Thorough	Llama-3.3-70B FP8 / TensorRT-LLM, H100	Candidate deployment	❌ Worker crashed because generated TensorRT-LLM arguments were mutually exclusive	#8659
Thorough	Qwen3-0.6B / vLLM	Prefill benchmarking	❌ Decode benchmarking could not read pod logs and produced no final DGD	#6962
Thorough	Model/backend not recorded	Prefill benchmarking	❌ Decode benchmarking produced no logs or final DGD	#8638

The same validation work also exposed retry/status behavior after failed Thorough runs (#7038, #7262, #8648). Those reports describe repeated execution of an already failed run, not additional successful or failed model/backend cases, so they are not counted again above.

Row-by-row comparison

#	Target recipe	Backend / mode	Hardware; GPU budget	Original AIC coverage in #8469	Recorded DGDR v1 execution	Current AI Simulate.Sweeper result	Why previously fallback-only rows now run	Remaining limitation or unmodeled recipe scope
1	DeepSeek-R1	SGLang / disaggregated	H200; 16	⚠️ Naive fallback only	❌ Thorough profiling failed*	❌ Blocked	— Still blocked	Missing SGLang context-MLA performance data for FP8 attention/KV.
2	DeepSeek-R1	SGLang / disaggregated	H200; 32	⚠️ Naive fallback only	❌ Thorough profiling failed*	❌ Blocked	— Still blocked	Same context-MLA performance-data gap as row 1.
3	DeepSeek-R1	TensorRT-LLM / disaggregated	GB200; 36	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	❌ Blocked	— Still blocked	TensorRT-LLM disaggregated is unsupported by the Replay runner; tested shapes also lack required MoE performance data.
4	DeepSeek-R1	vLLM / disaggregated	H200; 32	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Exact AIC support added: #851	—
5	DeepSeek-V3.2-NVFP4	TensorRT-LLM / aggregated	GB200; 32	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Architecture path: existing DSA family/data via #662; no exact matrix row required	—
6	DeepSeek-V3.2-NVFP4	TensorRT-LLM / disaggregated	GB200; 32	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	❌ Blocked	— Still blocked	TensorRT-LLM disaggregated is unsupported by the Replay runner.
7	GLM-5-NVFP4	SGLang / disaggregated	GB200; 20	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Exact model/operation support added: #991, #1059, #1223, #1342	—
8	GPT-OSS-120B	TensorRT-LLM / aggregated	GB200; 4	✅ AIC simulation	— No matching DGDR v1 execution recorded	✅ Base simulation ran	— Already AIC-covered	—
9	GPT-OSS-120B	TensorRT-LLM / disaggregated	GB200, B200; 5	✅ AIC simulation	❌ Rapid generated an over-budget DGD on H100/32 GPUs*	❌ Blocked	— Already AIC-covered	TensorRT-LLM disaggregated is unsupported. Feasible shapes use up to four GPUs under the five-GPU budget.
10	Kimi-K2.5-NVFP4	TensorRT-LLM / aggregated	B200; 8	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Exact AIC support added: #757; KV estimate fixed in #912	—
11	Kimi-K2.5-NVFP4 + KVBM	TensorRT-LLM / aggregated	B200; 8	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Exact Kimi base support added: #757, #912	Historical KVBM recipe semantics are not modeled by the current Sweeper path.
12	Kimi-K2.5-NVFP4 + EAGLE3	TensorRT-LLM / aggregated	B200; 32	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Exact Kimi base support added: #757, #912	Current aic_nextn does not represent the complete EAGLE3 draft-model configuration.
13	Llama-3.3-70B FP8	vLLM / aggregated	H100, H200; 4	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Architecture path: HF config resolves to LlamaForCausalLM; no exact matrix row	—
14	Llama-3.3-70B FP8	vLLM / disaggregated	H100, H200; 8	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Architecture path: HF config resolves to LlamaForCausalLM; no exact matrix row	—
15	Llama-3.3-70B FP8, multi-node	vLLM / disaggregated	H100, H200; 16	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Architecture path: HF config resolves to LlamaForCausalLM; no exact matrix row	Does not validate concrete node placement, RDMA, EFA, or other inter-node behavior.
16	Llama-3.3-70B FP8 + GAIE/EPP	vLLM / aggregated	H100, H200; 4	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Architecture path: HF config resolves to LlamaForCausalLM; no exact matrix row	GAIE/EPP behavior is outside the simulated topology.
17	Llama-3.3-70B FP8 + GAIE/EPP	vLLM / disaggregated	H100, H200; 8	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	✅ Base simulation ran	Architecture path: HF config resolves to LlamaForCausalLM; no exact matrix row	GAIE/EPP behavior is outside the simulated topology.
18	Nemotron-3-Super	SGLang / aggregated	H100, H200; 4	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	❌ Blocked	— Still blocked	Missing MoE performance data for the model's 512-expert, top-k-22 configuration.
19	Nemotron-3-Super	SGLang / disaggregated	H100, H200; 4	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	❌ Blocked	— Still blocked	Same MoE performance-data gap as row 18.
20	Nemotron-3-Super	TensorRT-LLM / disaggregated	H100, H200; 4	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	❌ Blocked	— Still blocked	Missing MoE performance data and unsupported TensorRT-LLM disaggregated topology.
21	Nemotron-3-Super	vLLM / aggregated	H100, H200; 4	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	❌ Blocked	— Still blocked	Missing MoE performance data for the model configuration.
22	Qwen3-235B FP8	TensorRT-LLM / aggregated	H100, H200; 16	✅ AIC simulation	✅ Rapid output and Thorough profiling passed, but mode/SKU were not recorded*	✅ Base simulation ran	— Already AIC-covered	—
23	Qwen3-235B FP8	TensorRT-LLM / disaggregated	H100, H200; 16	✅ AIC simulation	✅ Rapid fully deployed on H100/32 GPUs; ❌ Thorough failed on that configuration*	❌ Blocked	— Already AIC-covered	TensorRT-LLM disaggregated is unsupported by the Replay runner.
24	Qwen3-32B-FP8	TensorRT-LLM / aggregated	H100, H200, A100; 2	✅ AIC simulation	— No matching DGDR v1 execution recorded	⚠️ H100/H200 only	— Already AIC-covered	A100 lacks the required FP8 block-GEMM performance data.
25	Qwen3-32B-FP8	TensorRT-LLM / disaggregated	H100, H200, A100; 8	✅ AIC simulation	— No matching DGDR v1 execution recorded	❌ Blocked	— Already AIC-covered	TensorRT-LLM disaggregated is unsupported; A100 also lacks FP8 block-GEMM performance data.
26	Qwen3-32B-FP8	vLLM / disaggregated	A100; 8	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	❌ Blocked	— Still blocked	The A100/vLLM database lacks the required FP8 block-GEMM data.
27	Qwen3-32B	vLLM / aggregated, round-robin	H200; 16	✅ AIC simulation	✅ Rapid output and Thorough profiling passed; mode/SKU were not recorded. A separate Rapid auto-apply cache-path case failed*	✅ Base simulation ran	— Already AIC-covered	—
28	Qwen3-32B	vLLM / disaggregated, KV router	H200; 16	✅ AIC simulation	✅ Rapid output and Thorough profiling passed; mode/SKU were not recorded. A separate Rapid auto-apply cache-path case failed*	✅ Base simulation ran	— Already AIC-covered	—
29	Qwen3-VL-30B-A3B FP8 + embedding cache	vLLM / aggregated	GB200; 1	⚠️ Naive fallback only	— No matching DGDR v1 execution recorded	❌ Blocked	— Still blocked	No KV-feasible configuration fits within one GPU. The base model succeeds with a two-GPU budget; embedding-cache behavior is not modeled.

* At least one material dimension differs from, or was not recorded for, the target row: deployment mode, GPU SKU, GPU budget, feature set, or completion scope. The detailed execution table above states what was actually exercised.

Why previously fallback-only rows now run

The current result has three distinct causes. They should not be collapsed into "AIC support was added":

Rows	Classification	What changed
4: DeepSeek-R1 / vLLM	Exact AIC model coverage added	AIC #851 added the cached DeepSeek-R1 configuration and registered the model for support-matrix generation. R1 reuses the existing DeepseekV3ForCausalLM model family and performance data. The tested AIC package contains exact H200/vLLM PASS rows for this model.
5: DeepSeek-V3.2-NVFP4 / TensorRT-LLM	Architecture-level path; no exact matrix row in the tested package	The DeepSeek-V3.2/DSA model family was added in AIC #662. GB200 TensorRT-LLM 1.3.0rc10 operation data had already landed in commits ac31bf1d and 6c62e221. The tested Sweeper loaded the NVIDIA checkpoint's Hugging Face configuration directly and successfully exercised that family/data combination. Its pinned 2026-07-28 AIC package has no exact support-matrix row for this checkpoint. The cached exact checkpoint configuration was added subsequently in AIC #1423.
7: GLM-5-NVFP4 / SGLang	Exact AIC model and operation coverage added	The support chain consists of the base DeepSeek-V3.2/GLM-5 family in #662, explicit FP8/NVFP4 variants in #991 and #1059, SGLang DSA/sparse modeling in #1223, and GB200 correctness work in #1249. #1342 added the SGLang 0.5.14 data used by this test; #1366 regenerated the exact PASS matrix row.
10-12: Kimi-K2.5-NVFP4 / TensorRT-LLM	Exact AIC model coverage added; recipe features remain outside the test	AIC #757 added the Kimi-K2.5 family, NVIDIA NVFP4 checkpoint, quantization handling, and silicon data. #912 then replaced an approximately 6.2x KV-cache overestimate with model-derived MLA dimensions. This explains the base topology result. It does not establish KVBM or EAGLE3 support for rows 11 and 12.
13-17: Llama-3.3-70B FP8 / vLLM	Architecture-level path; no exact matrix row	No exact RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic support entry was added. AIC has supported loading an arbitrary Hugging Face configuration and dispatching by architecture since AIC #86. Sweeper's model/hardware resolution, introduced with the initial AI Simulate integration in Dynamo #11923, calls that path directly instead of gating the search on an exact support-matrix row. The checkpoint resolves to LlamaForCausalLM, allowing the existing Llama forward model to run. Multi-node networking and GAIE/EPP remain outside the simulated scope of rows 15-17.

In simplified form, the relevant control-flow difference is:

DGDR v1: matrix lookup -> exact/architecture fallback -> fail -> naive fallback
Sweeper: HF config -> supported architecture -> model/KV/FPM evaluation

This architecture-level path expands executable coverage, but it is not a claim that every checkpoint has dedicated measurements or that the generated deployment recipe is production-valid.

Main coverage gaps

Replay topology support

The current runner advertises vLLM aggregated/disaggregated, SGLang aggregated/disaggregated, and TensorRT-LLM aggregated. It does not advertise TensorRT-LLM disaggregated support. See components/src/dynamo/replay/simulation.py.

Performance-model coverage

The concrete gaps observed in this matrix are:

• SGLang context-MLA data for DeepSeek-R1 on H200;
• MoE data for Nemotron-3-Super's 512 experts and top-k 22;
• FP8 block-GEMM data for Qwen3-32B-FP8 on A100;
• some TensorRT-LLM MoE shapes for DeepSeek-R1 on GB200.

Recipe features outside the base simulation

The current checks do not establish parity for:

• historical KVBM configuration;
• full EAGLE3 speculative-decoding configuration;
• GAIE/EPP behavior;
• concrete multi-node placement and interconnect behavior;
• multimodal embedding-cache behavior.

These are distinct from model-performance coverage: a valid performance prediction for the engine topology does not prove that the complete recipe is represented or deployable.

Method

For each of the 29 rows, the check used the matrix's model, backend, deployment mode, GPU SKU, and GPU budget. Rows listing multiple SKUs expanded to 45 concrete probes.

The assessment performed:

1. model and system resolution through current AI Simulate.Sweeper code;
2. KV-capacity enumeration with a 3,300-token maximum sequence length, corresponding to the 3,000-token ISL plus 300-token OSL used in the historical validation context;
3. real AIC forward-model smoke predictions for prefill and decode;
4. comparison against the topology capabilities advertised by the current Replay simulation runner;
5. exhaustive forward-model checks across all KV-feasible parallel-role shapes for combinations that initially failed.

The environment used the repository's current aisimulate package and its pinned aiconfigurator==0.11.0.dev20260728 / aiconfigurator-core==0.11.0.dev20260728 dependencies.

What this assessment does not prove

• It is not a full multi-hour smart sweep of every row.
• It does not compare predicted metrics with measurements from live deployments.
• It does not generate, validate, or deploy a DGD.
• It does not validate runtime images, engine CLI arguments, environment variables, networking, storage, scheduling, or cloud-provider integration.
• Some AIC datasets emit low-fidelity fallback warnings. A successful smoke prediction is therefore evidence of an executable model path, not a production-accuracy claim.

The result should be used as a current coverage map and a starting point for integration work, not as a replacement for hardware validation.