RTen panic fuzzing

rten-panics.md

rten ONNX Panic Fuzzing — Findings

Goal: find ONNX models that make cargo run -p rten-cli -r -- <model> panic or crash (not return a clean error). A panic where ONNX Runtime returns a clean error still counts as a robustness bug. Clean Err returns from rten do not count.

Harness: /tmp/rten-fuzz/ — harness.py builds models, runs ORT (validity check) then rten, and flags crashes. Binary: target/release/rten.

Severity scale

• High — segfault / memory unsafety / undefined behavior (e.g. a crash inside unsafe, out-of-bounds unchecked access, heap corruption). None found yet.
• Medium — a Rust panic (clean abort, no UB). All findings below are Medium. A panic on a valid model that ORT runs (a clean divergence) is the more serious kind of Medium; a panic where ORT also rejects is a robustness gap (should return Err, not panic).

Status

• Batch 1 (empty/zero-size tensors): no panics — rten handles 0-size gracefully.

• Batch 2 (attribute edge cases: group=0, stride=0, axis OOR, blocksize): no panics — all validated.

• Batches 3-5 (RNN/STFT/reduce/squeeze/norm/einsum/broadcast): findings #1-#4.

• Batches 6-8 (Mod/Div by zero, Pad reflect, quant per-axis, ArgMax/TopK/Compress, control-flow If/Loop, RoiAlign/LRN/MaxUnpool, Gemm, Trilu, Resize): findings #5-#6. No panics from: Pad reflect-overflow, DequantizeLinear/QuantizeLinear per-axis mismatch, ArgMax/TopK/Compress edge cases, If/Loop, MatMulInteger/ConvInteger, Trilu huge k, Resize size=0/huge, NonZero, GatherND batch_dims — all clean errors or correct results.

• Batch 9 (Slice step0/dup-axes/oor, Tile bad reps, Det, Unique, window fns, GridSample, Pow, BitShift, ScatterND, Shape start/end, ConvTranspose output_shape): no panics.

• Batch 10 (input-pair shape mismatches + huge-size allocations): findings #7-#10. No panics from: LSTM seq_lens/peephole, LayerNorm/GroupNorm/PRelu scale mismatch, Gemm bias mismatch, Reshape huge product, ConstantOfShape negative, Resize bad roi.

• Batch 11 (conv-bias path variants + Conv+Add fusion + huge shape producers): findings #7 breadth (grouped/1D/depthwise), #11 (Range OOM), #12 (Upsample hang). No panics from: Conv+Add bias fusion with mismatched operand, QLinearConv bias mismatch, Pad huge, ScatterElements update mismatch, conv bias longer than channels.

• Harness hardened to run ORT in a memory-capped subprocess + 30s timeouts. NOTE: macOS does not enforce RLIMIT_AS, so genuinely-huge allocations (count fits in usize) can still OOM-kill — avoid >~1e8-element Range/alloc cases when iterating.

• Batch 12 (Sequence/Optional, loss ops, CenterCropPad, MaxRoiPool): no panics — Sequence position ops use resolve_index → clean errors; many ops are unsupported → clean errors.

• Batch 13 (RNN weight/state inconsistencies, MatMul+Add fusion, conv weight mismatch): finding #13 (LSTM W/R, direction, initial_h). No panics from: GRU W/R mismatch (validated), MatMul+Add fusion bad bias, InstanceNorm mismatch, conv in-channel/kernel mismatch. Graph optimizer/fusions (src/optimize/fusions.rs) look defensive (maybe_fuse → Result).

• Batch 14 (basic RNN, einsum mismatch, Pad/Resize axes input, OneHot): no panics. RNN and GRU validate W/R/bias (LSTM is the outlier); Pad/Resize axes validated; OneHot OOR indices handled. Concat validates non-axis dims (the expect("should have capacity") is unreachable).

• Batch 15 (DFT/STFT edge cases): finding #14 (DFT huge dft_length OOM). DFT otherwise robust.

Coverage note

Search is approaching saturation for the "broad panic" goal — recent batches (12, 14) found nothing; rten is genuinely robust across most operators. The crashes found fall into three root-cause families, each a single class of fix:

1. Duplicate axes not de-duplicated before computing output shape (#1 reduce, #1b squeeze).
2. Unvalidated input-pair shape assumption — an op derives a dim from input A and slices/indexes/matmuls input B by it without checking, then panics (often .unwrap()): LSTM bias/W/R/h0 (#2,#13), BatchNorm scale (#4), Conv bias (#7), ConvTranspose bias (#8).
3. Unguarded data-derived allocation — a size/count from attributes or data flows into a Vec/FFT allocation with no bound: capacity-overflow when the product overflows usize (#3 STFT, #9 ConstantOfShape, #10 Expand, Tile), or OOM/SIGKILL when it fits in usize (#11 Range, #14 DFT) — plus unbounded compute (#12 Upsample). Plus standalone: integer Div/Mod by zero (#5,#6).

Pattern that keeps paying off: an operator assumes input B's length matches a dimension derived from input A, then slices/indexes B by that dimension without validating (LSTM bias #2, BatchNorm scale #4, Conv bias #7, ConvTranspose bias #8). And: a size/count derived from data flows into a Vec allocation without an overflow guard (STFT #3, ConstantOfShape #9, Expand #10).

Summary (as of batch 17 — broad search saturated)

~16 distinct panics across 14 finding groups, all reproduced by models in fuzz-repros/ (26 files; all re-verified to still crash). Every crash is a Rust panic or resource-exhaustion kill — no segfaults / memory-unsafety (no High-severity) found. The crashes reduce to three fixable root-cause families (+ Div/Mod-by-zero); see "Coverage note" near the bottom. Batches 12, 14, 16, 17 found nothing new → the broad sweep has reached saturation; rten is robust outside these patterns.

Confirmed findings

#1 — ReduceSum/ReduceMean/etc. with duplicate axes → panic (rten crashes, ORT runs fine) ★ MEDIUM (clean divergence)

• Severity: Medium — panic (no UB). Notable because it's a clean divergence: ORT accepts the model and runs; rten panics. The most impactful Medium since the model is valid.
• Op: Any reduce op (ReduceSum, ReduceMean, ReduceMax, …) with a repeated entry in axes.
• Trigger: input shape [2,3,4], axes = [0,1,0] (or [0,0]). Works via the axes input (opset 18) and the axes attribute (opset 13).
• Panic: src/ops/reduce.rs:490 — Tensor::from_data(&reduced_shape, reduced_data) → data length 1 does not match shape [1, 1, 4]. reduced_shape is computed treating each axis occurrence independently, but the reduction over resolved_axes.len() inner dims produces fewer elements, so the length/shape mismatch panics in from_data.
• Root cause: axes are never de-duplicated/validated for uniqueness before computing reduced_shape. ONNX Runtime tolerates duplicate axes (treats them as the set of axes).
• Repro: /tmp/rten-fuzz/reduce_dup_axes.onnx, /tmp/rten-fuzz/reduce_dup_axes_attr.onnx
• Fix direction: de-duplicate resolved_axes (and reject or ignore repeats) before computing reduced_shape, matching ORT semantics.

#1b — Squeeze with duplicate axes → panic (rten crashes, ORT runs fine) ★ MEDIUM (clean divergence)

• Same duplicate-axes root cause as #1, different crash site.
• Op: Squeeze, input [1,1,4], axes=[0,0] (or axes=[0,-3], which alias to axis 0).
• Panic: rten-tensor/src/layout.rs:596 — index out of bounds: the len is 4 but the index is 18446744073709551615 (= usize::MAX). squeeze_in_place removes axis 0, then tries to remove it again with axis - n_removed = 0 - 1 underflowing to usize::MAX.
• ORT runs fine. Repro: /tmp/rten-fuzz/squeeze_dup.onnx, /tmp/rten-fuzz/squeeze_dup_neg.onnx
• Confirmed breadth of #1/#1b (all panic, ORT ok): ReduceSum/ReduceMean/ReduceL2/ ReduceMin/ReduceMax with duplicate axes (input or attribute form, opset 13 & 18), duplicate-via-negative ([0,-3]), and Squeeze. Unsqueeze and Transpose are validated (clean errors) — so the gap is specifically squeeze_in_place / reduce-axes not de-duplicating. Single fix point: de-dupe axes in the shared helper.

#2 — LSTM with undersized bias → panic instead of error ★ MEDIUM

• Severity: Medium — panic on an invalid model. ORT returns a clean InvalidArgument ("Input B must have shape {1,16}. Actual:{1,8}"); rten panics. GRU validates this and returns an error, so LSTM is inconsistent with both ORT and rten's own GRU.
• Op: LSTM, hidden_size=2, inputs X[1,1,3], W[1,8,3], R[1,8,2], B[1,8] (bias has 8 cols; needs 8*hidden_size = 16; 8 % 8 == 0 passes the only check).
• Panic: rten-tensor/src/tensor.rs:1592 via the bias slice((dir, ..(n_gates*hidden))) in src/ops/rnn.rs (~line 499) — slice range exceeds bias size(1).
• Root cause: bias validation only checks size(1) % 8 == 0, not size(1) >= 8*hidden.
• Repro: /tmp/rten-fuzz/lstm_bad_bias.onnx

#3 — STFT with window/n_fft larger than signal length → capacity overflow panic ★ MEDIUM

• Severity: Medium — panic on an invalid model. ORT returns a clean FAIL; rten panics with an allocation capacity overflow.
• Op: STFT, onesided=1, signal[1,4,1], frame_step=1, window[6] (window length 6 > signal_len 4).
• Panic: alloc capacity overflow — n_frames computed from signal_len - n_fft underflows (unsigned wrap in release) → astronomically large frame count → huge allocation. Origin src/ops/fft.rs (~line 98 frame-count calc; ~115-118 indexing).
• Root cause: no validation that signal_len >= n_fft before computing n_frames.
• Repro: /tmp/rten-fuzz/stft_big_window.onnx

#4 — BatchNormalization with mismatched scale/bias/mean/var length → panic instead of error ★ MEDIUM

• Severity: Medium — panic on an invalid model. ORT rejects at load ("Dimension mismatch in unification between 2 and 4"); rten panics.
• Op: BatchNormalization, X[1,4,2,2] (4 channels), but scale/bias/mean/var all length 2 instead of 4.
• Panic: rten-tensor/src/layout.rs:246 (broadcast/shape assertion). The per-channel validation that should compare scale.size(0) to the channel count is missing/bypassed for the 4D path.
• Repro: fuzz-repros/bn_bad_scale.onnx

#5 — Integer Mod by zero → panic instead of error ★ MEDIUM

• Severity: Medium — panic (no UB). ORT rejects (constant-folds the zero divisor at init and raises); rten panics in the kernel.
• Op: Mod (integer, fmod=0), A=[5,7,9] (int32), B=[0,0,0] (int32).
• Panic: src/ops/binary_elementwise.rs:729 — integer remainder by zero. Panics in the Mod kernel itself (also under --no-optimize); not just constant folding.
• Note: rten only divides by zero when the divisor is a constant here, because rten-cli feeds random non-zero values to runtime int inputs. The kernel has no zero-divisor guard.
• Repro: fuzz-repros/mod_zero.onnx

#6 — Integer Div by zero → panic instead of error ★ MEDIUM

• Severity: Medium — panic (no UB). ORT rejects; rten panics.
• Op: Div (integer), A=[5,7,9] (int32), B=[0,0,0] (int32).
• Panic: src/ops/binary_elementwise.rs:586 — integer divide by zero (also under --no-optimize).
• Repro: fuzz-repros/div_zero.onnx

#7 — Conv with mismatched bias length → panic (rten crashes, ORT runs fine) ★ MEDIUM (clean divergence)

• Severity: Medium — panic (no UB). Clean divergence: ORT accepts and runs (output (1,4,6,6)); rten panics. Like #1, a model ORT runs that rten crashes on.
• Op: Conv, X[1,3,8,8], W[4,3,3,3] (4 output channels), bias B[2] (should be 4).
• Panic: src/ops/conv.rs:335 — &b.data()[out_chans.clone()] slices bias by the group's output-channel range [0..4] but bias has length 2: range end index 4 out of range for slice of length 2.
• Root cause: no validation that bias length == output channels before the per-group slice.
• Breadth (all clean divergences — ORT runs, rten panics): standard, grouped (group=2), and 1D Conv all panic at conv.rs:335; depthwise Conv (group=channels) panics at a different site rten-tensor/src/layout.rs:246. Bias longer than channels is fine (slices a prefix). Repros: conv_bad_bias.onnx, grpconv_bad_bias.onnx, conv1d_bad_bias.onnx, dwconv_bad_bias.onnx.
• Repro: fuzz-repros/conv_bad_bias.onnx

#8 — ConvTranspose with mismatched bias length → panic instead of error ★ MEDIUM

• Severity: Medium — panic on invalid model. ORT rejects ("Bias shape is not compatible with number of output channels"); rten panics.
• Op: ConvTranspose, X[1,3,4,4], W[3,4,3,3] (out_c=4), bias B[2].
• Panic: rten-tensor/src/layout.rs:246 (broadcast/shape assertion).
• Repro: fuzz-repros/convt_bad_bias.onnx

#9 / #10 — Huge output dims → capacity overflow panic instead of error ★ MEDIUM

• Severity: Medium — panic on invalid model. ORT rejects via SafeInt overflow check; rten panics with an allocation capacity overflow (same class as STFT #3).
• #9 Op: ConstantOfShape, shape [2^40, 2^40]. Repro: fuzz-repros/cos_huge.onnx
• #10 Op: Expand, input [1], shape [2^32, 2^32]. Repro: fuzz-repros/expand_huge.onnx
• Root cause: the output element count (product of dims) overflows / is astronomically large and is passed straight to a Vec allocation without a sanity/overflow check. Note Reshape to a huge product is validated (clean error) — gap is shape-producing ops.

#13 — LSTM with inconsistent weight/state shapes → panic (multiple sites) ★ MEDIUM

Extends #2 (LSTM bias): LSTM has several unvalidated shape relationships between its inputs. All inputs are constants here so rten reaches the kernel.

• #13a — W and R imply different hidden sizes → panic, ORT runs fine (clean divergence). X[1,1,3], W[1,8,3] (⇒hidden=2), R[1,20,5] (⇒hidden=5), hidden_size=2. ORT runs (output (1,1,1,2)); rten panics at src/ops/rnn.rs:548 — .unwrap() on the gemm of the hidden state with R (dimension mismatch). Repro: fuzz-repros/lstm_wr_mismatch.onnx
• #13b — direction=bidirectional but weights have num_directions=1 → panic, ORT runs fine. W[1,8,3]/R[1,8,2] (num_dir=1) with direction=bidirectional. ORT runs (output (1,2,1,2)); rten panics at rten-tensor/src/tensor.rs:1817 (indexes a direction dim that doesn't exist). Repro: fuzz-repros/lstm_dir_mismatch.onnx
• #13c — initial_h wrong hidden size → panic instead of error. initial_h[1,1,99] with hidden=2. ORT rejects ("Input initial_h must have shape {1,1,2}"); rten panics at rnn.rs:548. Repro: fuzz-repros/lstm_bad_h0.onnx
• Root cause: the kernel derives hidden_size from W (and the attribute) but does not validate R, initial_h/initial_c, B, or the direction dim against it before slicing / matmul; the failing op result is .unwrap()ed.

#11 — Range with huge element count → unbounded allocation → OOM / SIGKILL ★ MEDIUM (resource exhaustion)

• Severity: Medium — not UB, but worse operationally than a clean panic: the process is killed (SIGKILL) or thrashes the machine. ORT rejects (bad_alloc is caught and surfaced as an error); rten attempts the allocation.
• Op: Range, start=0, limit=1e18, delta=1 (≈1e18 elements ≈ 4 EB). The element count fits in usize so the Vec capacity-overflow guard does NOT fire — rten just tries to allocate.
• Root cause: no upper bound / available-memory sanity check on the computed output length before allocating. Repro: fuzz-repros/range_huge.onnx ⚠️ Running this can OOM the machine — macOS does not enforce RLIMIT_AS. Run under a real memory cgroup/limit if reproducing.

#12 — Upsample/Resize with huge scales → unbounded compute → hang (timeout) ★ MEDIUM (resource exhaustion)

• Severity: Medium — rten hangs (>30s, no completion) attempting to materialize a 1e6 × 1e6 output. ORT rejects. Op: Upsample (opset 9), X[1,1,2,2], scales=[1,1,1e6,1e6]. Repro: fuzz-repros/upsample_huge.onnx

#14 — DFT with huge dft_length → unbounded allocation → OOM / SIGKILL ★ MEDIUM (resource exhaustion)

• Same class as #11 (Range). DFT, signal [1,4,1], dft_length=2^34. dft_length is taken as usize (line ~254 in src/ops/fft.rs) and used to plan the FFT and Vec::with_capacity(n_fft) (line ~304) with no upper bound → ~128 GB allocation → process killed. ORT rejects at load.
• Repro: fuzz-repros/dft_len_huge.onnx ⚠️ can OOM the machine (macOS ignores RLIMIT_AS).
• Note: DFT is otherwise robust — zero-pad/truncate/IRFFT/low-rank/axis-OOR/bad-components all return clean errors or correct results; only the unbounded dft_length is unguarded.

Capacity-overflow class (additional instances of #3/#9/#10)

Tile with reps=[2^40,2^40] also panics with capacity overflow (fuzz-repros/tile_huge.onnx). Same root pattern as STFT/ConstantOfShape/Expand: a data-derived size whose product overflows usize hits Rust's Vec capacity-overflow panic. (Contrast #11: when the count fits in usize the guard doesn't fire and it OOM-kills instead — the more dangerous variant.)

Repro files

All reproducing models are saved under fuzz-repros/. Run any with: cargo run -p rten-cli -r -- fuzz-repros/<name>.onnx

File	Finding
reduce_dup_axes.onnx, reduce_dup_axes_attr.onnx, reducel2_dup.onnx, reduce_dup_neg.onnx, reducemin_dup_attr.onnx	#1 reduce duplicate axes
squeeze_dup.onnx, squeeze_dup_neg.onnx	#1b squeeze duplicate axes
lstm_bad_bias.onnx	#2 LSTM undersized bias
stft_big_window.onnx	#3 STFT window > signal
bn_bad_scale.onnx	#4 BatchNorm scale length mismatch
mod_zero.onnx	#5 integer Mod by zero
div_zero.onnx	#6 integer Div by zero
conv_bad_bias.onnx	#7 Conv bias length mismatch (ORT runs!)
convt_bad_bias.onnx	#8 ConvTranspose bias length mismatch
cos_huge.onnx	#9 ConstantOfShape huge dims → capacity overflow
expand_huge.onnx	#10 Expand huge dims → capacity overflow
grpconv_bad_bias.onnx, conv1d_bad_bias.onnx, dwconv_bad_bias.onnx	#7 Conv bias (grouped/1D/depthwise paths)
range_huge.onnx	#11 Range huge count → OOM/SIGKILL (⚠️ can OOM machine)
upsample_huge.onnx	#12 Upsample huge scales → hang
tile_huge.onnx	capacity-overflow class
lstm_wr_mismatch.onnx, lstm_dir_mismatch.onnx, lstm_bad_h0.onnx	#13 LSTM shape inconsistencies
dft_len_huge.onnx	#14 DFT huge dft_length → OOM (⚠️ can OOM machine)

Notes

• rten is well-defended on operator attribute validation and empty tensors (batches 1 & 2 found nothing).
• Negative/out-of-range Gather/GatherND/GatherElements/ScatterElements indices are all validated → clean errors (good).
• Next: confirm exact lines for #2/#3, probe more reduce-family + axes-dedup ops (e.g. Unsqueeze/Squeeze duplicate axes), Transpose duplicate perm, einsum, more RNN shape mismatches.