DEEP_AUDIT_RESULTS.json

gistfile1.txt

 ═══════════════════════════════════════════════════════════════╗
║     DEEP AUDIT: NVIDIA NEMOTRON-3-NANO CLAIMS VALIDATION     ║
╚═══════════════════════════════════════════════════════════════╝

📊 TEST MATRIX:
   Questions: 9
   Configurations: 5
   Runs per config: 3
   Total tests: 135


════════════════════════════════════════════════════════════════════════════════
CONFIGURATION: BASELINE
Description: Default reasoning mode
Thinking: ON | Max Tokens: 2000
════════════════════════════════════════════════════════════════════════════════

[Q1/9] "What is 2 + 2?"
--------------------------------------------------------------------------------
  Run 1: PD=0.051 | CDI=0.643 | Traces=262 | Len=320ch | Latency=2244ms
  Run 2: PD=0.051 | CDI=0.652 | Traces=25 | Len=11ch | Latency=921ms
  Run 3: PD=0.010 | CDI=0.628 | Traces=404 | Len=324ch | Latency=2377ms

[Q2/9] "What is 12 × 5?"
--------------------------------------------------------------------------------
  Run 1: PD=0.111 | CDI=0.644 | Traces=29 | Len=21ch | Latency=1119ms
  Run 2: PD=0.022 | CDI=0.408 | Traces=622 | Len=237ch | Latency=3135ms
  Run 3: PD=0.051 | CDI=0.652 | Traces=25 | Len=13ch | Latency=434ms

[Q3/9] "What is 127 × 83?"
--------------------------------------------------------------------------------
  Run 1: PD=0.043 | CDI=0.344 | Traces=141 | Len=29ch | Latency=946ms
  Run 2: PD=0.023 | CDI=0.560 | Traces=64 | Len=23ch | Latency=554ms
  Run 3: PD=0.000 | CDI=0.651 | Traces=108 | Len=19ch | Latency=773ms

[Q4/9] "If x² - 5x + 6 = 0, what are the solutions?"
--------------------------------------------------------------------------------
  Run 1: PD=0.000 | CDI=0.640 | Traces=44 | Len=188ch | Latency=814ms
  Run 2: PD=0.033 | CDI=0.371 | Traces=47 | Len=801ch | Latency=1903ms
  Run 3: PD=0.066 | CDI=0.000 | Traces=49 | Len=251ch | Latency=645ms

[Q5/9] "Calculate 12,345 × 6,789 step by step"
--------------------------------------------------------------------------------
  Run 1: PD=0.066 | CDI=0.000 | Traces=48 | Len=2671ch | Latency=4292ms
  Run 2: PD=0.050 | CDI=0.000 | Traces=703 | Len=1968ch | Latency=8243ms
  Run 3: PD=0.050 | CDI=0.464 | Traces=777 | Len=1728ch | Latency=7923ms

[Q6/9] "Solve x³ - 6x² + 11x - 6 = 0 for all real solutions"
--------------------------------------------------------------------------------
  Run 1: PD=0.051 | CDI=0.643 | Traces=89 | Len=687ch | Latency=1211ms
  Run 2: PD=0.059 | CDI=0.000 | Traces=150 | Len=775ch | Latency=2014ms
  Run 3: PD=0.059 | CDI=0.000 | Traces=126 | Len=721ch | Latency=2059ms

[Q7/9] "What is 100 + 200?"
--------------------------------------------------------------------------------
  Run 1: PD=0.015 | CDI=0.630 | Traces=25 | Len=17ch | Latency=423ms
  Run 2: PD=0.022 | CDI=0.457 | Traces=24 | Len=17ch | Latency=416ms
  Run 3: PD=0.051 | CDI=0.643 | Traces=31 | Len=17ch | Latency=430ms

[Q8/9] "Now multiply your previous answer by 3"
--------------------------------------------------------------------------------
  Run 1: PD=0.028 | CDI=0.000 | Traces=450 | Len=242ch | Latency=2542ms
  Run 2: PD=0.040 | CDI=0.000 | Traces=338 | Len=181ch | Latency=2012ms
  Run 3: PD=0.040 | CDI=0.000 | Traces=203 | Len=285ch | Latency=1231ms

[Q9/9] "Now subtract 150 from that result"
--------------------------------------------------------------------------------
  Run 1: PD=0.022 | CDI=0.636 | Traces=101 | Len=107ch | Latency=829ms
  Run 2: PD=0.042 | CDI=0.614 | Traces=77 | Len=134ch | Latency=762ms
  Run 3: PD=0.028 | CDI=0.000 | Traces=172 | Len=331ch | Latency=1418ms

════════════════════════════════════════════════════════════════════════════════
CONFIGURATION: BUDGET_LOW
Description: Low budget (500 tokens) - should constrain reasoning
Thinking: ON | Max Tokens: 500
════════════════════════════════════════════════════════════════════════════════

[Q1/9] "What is 2 + 2?"
--------------------------------------------------------------------------------
  Run 1: PD=0.059 | CDI=0.562 | Traces=485 | Len=30ch | Latency=1995ms
  Run 2: PD=0.015 | CDI=0.543 | Traces=17 | Len=11ch | Latency=419ms
  Run 3: PD=0.022 | CDI=0.408 | Traces=363 | Len=233ch | Latency=2116ms

[Q2/9] "What is 12 × 5?"
--------------------------------------------------------------------------------
  Run 1: PD=0.051 | CDI=0.646 | Traces=28 | Len=13ch | Latency=418ms
  Run 2: PD=0.051 | CDI=0.659 | Traces=22 | Len=13ch | Latency=418ms
  Run 3: PD=0.051 | CDI=0.648 | Traces=27 | Len=22ch | Latency=401ms

[Q3/9] "What is 127 × 83?"
--------------------------------------------------------------------------------
  Run 1: PD=0.033 | CDI=0.533 | Traces=108 | Len=29ch | Latency=799ms
  Run 2: PD=0.052 | CDI=0.458 | Traces=100 | Len=19ch | Latency=763ms
  Run 3: PD=0.061 | CDI=0.642 | Traces=500 | Len=0ch | Latency=1986ms

[Q4/9] "If x² - 5x + 6 = 0, what are the solutions?"
--------------------------------------------------------------------------------
  Run 1: PD=0.061 | CDI=0.628 | Traces=32 | Len=754ch | Latency=1495ms
  Run 2: PD=0.050 | CDI=0.000 | Traces=54 | Len=472ch | Latency=1260ms
  Run 3: PD=0.050 | CDI=0.000 | Traces=35 | Len=524ch | Latency=1218ms

[Q5/9] "Calculate 12,345 × 6,789 step by step"
--------------------------------------------------------------------------------
  Run 1: PD=0.010 | CDI=0.628 | Traces=499 | Len=0ch | Latency=2248ms
  Run 2: PD=0.033 | CDI=0.628 | Traces=500 | Len=0ch | Latency=2260ms
  Run 3: PD=0.061 | CDI=0.628 | Traces=500 | Len=0ch | Latency=2122ms

[Q6/9] "Solve x³ - 6x² + 11x - 6 = 0 for all real solutions"
--------------------------------------------------------------------------------
  Run 1: PD=0.022 | CDI=0.576 | Traces=137 | Len=878ch | Latency=2438ms
  Run 2: PD=0.040 | CDI=0.000 | Traces=66 | Len=649ch | Latency=1569ms
  Run 3: PD=0.051 | CDI=0.629 | Traces=500 | Len=0ch | Latency=1804ms

[Q7/9] "What is 100 + 200?"
--------------------------------------------------------------------------------
  Run 1: PD=0.012 | CDI=0.637 | Traces=500 | Len=0ch | Latency=2451ms
  Run 2: PD=0.018 | CDI=0.594 | Traces=31 | Len=17ch | Latency=419ms
  Run 3: PD=0.039 | CDI=0.445 | Traces=18 | Len=17ch | Latency=406ms

[Q8/9] "Now multiply your previous answer by 3"
--------------------------------------------------------------------------------
  Run 1: PD=0.028 | CDI=0.000 | Traces=331 | Len=400ch | Latency=2195ms
  Run 2: PD=0.010 | CDI=0.558 | Traces=487 | Len=52ch | Latency=2448ms
  Run 3: PD=0.028 | CDI=0.000 | Traces=184 | Len=219ch | Latency=1061ms

[Q9/9] "Now subtract 150 from that result"
--------------------------------------------------------------------------------
  Run 1: PD=0.028 | CDI=0.614 | Traces=150 | Len=257ch | Latency=1913ms
  Run 2: PD=0.040 | CDI=0.000 | Traces=220 | Len=175ch | Latency=1398ms
  Run 3: PD=0.042 | CDI=0.614 | Traces=307 | Len=194ch | Latency=1830ms

════════════════════════════════════════════════════════════════════════════════
CONFIGURATION: BUDGET_MEDIUM
Description: Medium budget (1000 tokens)
Thinking: ON | Max Tokens: 1000
════════════════════════════════════════════════════════════════════════════════

[Q1/9] "What is 2 + 2?"
--------------------------------------------------------------------------------
  Run 1: PD=0.040 | CDI=0.000 | Traces=335 | Len=185ch | Latency=1813ms
  Run 2: PD=0.111 | CDI=0.656 | Traces=23 | Len=11ch | Latency=403ms
  Run 3: PD=0.111 | CDI=0.647 | Traces=27 | Len=23ch | Latency=434ms

[Q2/9] "What is 12 × 5?"
--------------------------------------------------------------------------------
  Run 1: PD=0.059 | CDI=0.000 | Traces=262 | Len=307ch | Latency=1749ms
  Run 2: PD=0.051 | CDI=0.648 | Traces=27 | Len=13ch | Latency=417ms
  Run 3: PD=0.051 | CDI=0.646 | Traces=28 | Len=13ch | Latency=406ms

[Q3/9] "What is 127 × 83?"
--------------------------------------------------------------------------------
  Run 1: PD=0.022 | CDI=0.576 | Traces=56 | Len=19ch | Latency=584ms
  Run 2: PD=0.052 | CDI=0.560 | Traces=68 | Len=19ch | Latency=628ms
  Run 3: PD=0.023 | CDI=0.560 | Traces=110 | Len=19ch | Latency=610ms

[Q4/9] "If x² - 5x + 6 = 0, what are the solutions?"
--------------------------------------------------------------------------------
  Run 1: PD=0.008 | CDI=0.657 | Traces=998 | Len=0ch | Latency=4333ms
  Run 2: PD=0.050 | CDI=0.000 | Traces=67 | Len=138ch | Latency=818ms
  Run 3: PD=0.050 | CDI=0.000 | Traces=39 | Len=247ch | Latency=867ms

[Q5/9] "Calculate 12,345 × 6,789 step by step"
--------------------------------------------------------------------------------
  Run 1: PD=0.061 | CDI=0.642 | Traces=1000 | Len=0ch | Latency=3784ms
  Run 2: PD=0.022 | CDI=0.576 | Traces=1000 | Len=0ch | Latency=3735ms
  Run 3: PD=0.059 | CDI=0.000 | Traces=852 | Len=427ch | Latency=4623ms

[Q6/9] "Solve x³ - 6x² + 11x - 6 = 0 for all real solutions"
--------------------------------------------------------------------------------
  Run 1: PD=0.066 | CDI=0.000 | Traces=176 | Len=835ch | Latency=2519ms
  Run 2: PD=0.042 | CDI=0.614 | Traces=75 | Len=733ch | Latency=1867ms
  Run 3: PD=0.000 | CDI=0.000 | Traces=57 | Len=1034ch | Latency=2210ms

[Q7/9] "What is 100 + 200?"
--------------------------------------------------------------------------------
  Run 1: PD=1.000 | CDI=0.641 | Traces=31 | Len=17ch | Latency=456ms
  Run 2: PD=1.000 | CDI=0.641 | Traces=31 | Len=35ch | Latency=430ms
  Run 3: PD=1.000 | CDI=0.641 | Traces=31 | Len=17ch | Latency=435ms

[Q8/9] "Now multiply your previous answer by 3"
--------------------------------------------------------------------------------
  Run 1: PD=0.028 | CDI=0.000 | Traces=263 | Len=219ch | Latency=1336ms
  Run 2: PD=0.040 | CDI=0.551 | Traces=907 | Len=116ch | Latency=4890ms
  Run 3: PD=0.040 | CDI=0.000 | Traces=145 | Len=116ch | Latency=888ms

[Q9/9] "Now subtract 150 from that result"
--------------------------------------------------------------------------------
  Run 1: PD=0.042 | CDI=0.614 | Traces=197 | Len=118ch | Latency=1024ms
  Run 2: PD=0.028 | CDI=0.614 | Traces=472 | Len=368ch | Latency=3199ms
  Run 3: PD=0.050 | CDI=0.000 | Traces=514 | Len=606ch | Latency=2392ms

════════════════════════════════════════════════════════════════════════════════
CONFIGURATION: BUDGET_HIGH
Description: High budget (3000 tokens)
Thinking: ON | Max Tokens: 3000
════════════════════════════════════════════════════════════════════════════════

[Q1/9] "What is 2 + 2?"
--------------------------------------------------------------------------------
  Run 1: PD=0.033 | CDI=0.643 | Traces=38 | Len=2ch | Latency=398ms
  Run 2: PD=0.111 | CDI=0.628 | Traces=277 | Len=359ch | Latency=1635ms
  Run 3: PD=0.051 | CDI=0.643 | Traces=326 | Len=337ch | Latency=1679ms

[Q2/9] "What is 12 × 5?"
--------------------------------------------------------------------------------
  Run 1: PD=0.051 | CDI=0.648 | Traces=27 | Len=13ch | Latency=465ms
  Run 2: PD=0.051 | CDI=0.662 | Traces=21 | Len=13ch | Latency=397ms
  Run 3: PD=0.051 | CDI=0.648 | Traces=27 | Len=13ch | Latency=418ms

[Q3/9] "What is 127 × 83?"
--------------------------------------------------------------------------------
  Run 1: PD=0.041 | CDI=0.589 | Traces=78 | Len=19ch | Latency=661ms
  Run 2: PD=0.023 | CDI=0.560 | Traces=63 | Len=19ch | Latency=585ms
  Run 3: PD=0.023 | CDI=0.560 | Traces=64 | Len=23ch | Latency=531ms

[Q4/9] "If x² - 5x + 6 = 0, what are the solutions?"
--------------------------------------------------------------------------------
  Run 1: PD=0.080 | CDI=0.000 | Traces=38 | Len=387ch | Latency=1106ms
  Run 2: PD=0.052 | CDI=0.323 | Traces=40 | Len=115ch | Latency=633ms
  Run 3: PD=0.050 | CDI=0.000 | Traces=51 | Len=851ch | Latency=1843ms

[Q5/9] "Calculate 12,345 × 6,789 step by step"
--------------------------------------------------------------------------------
  Run 1: PD=0.050 | CDI=0.621 | Traces=950 | Len=2188ch | Latency=7748ms
  Run 2: PD=0.059 | CDI=0.000 | Traces=968 | Len=817ch | Latency=6095ms
  Run 3: PD=0.058 | CDI=0.000 | Traces=854 | Len=1637ch | Latency=6587ms

[Q6/9] "Solve x³ - 6x² + 11x - 6 = 0 for all real solutions"
--------------------------------------------------------------------------------
  Run 1: PD=0.000 | CDI=0.000 | Traces=156 | Len=820ch | Latency=2210ms
  Run 2: PD=0.000 | CDI=0.000 | Traces=138 | Len=527ch | Latency=1237ms
  Run 3: PD=1.000 | CDI=0.614 | Traces=138 | Len=937ch | Latency=2714ms

[Q7/9] "What is 100 + 200?"
--------------------------------------------------------------------------------
  Run 1: PD=1.000 | CDI=0.641 | Traces=31 | Len=21ch | Latency=437ms
  Run 2: PD=1.000 | CDI=0.641 | Traces=31 | Len=21ch | Latency=467ms
  Run 3: PD=1.000 | CDI=0.641 | Traces=33 | Len=17ch | Latency=415ms

[Q8/9] "Now multiply your previous answer by 3"
--------------------------------------------------------------------------------
  Run 1: PD=0.040 | CDI=0.000 | Traces=568 | Len=135ch | Latency=3311ms
  Run 2: PD=0.040 | CDI=0.000 | Traces=1254 | Len=167ch | Latency=4885ms
  Run 3: PD=0.040 | CDI=0.000 | Traces=373 | Len=593ch | Latency=2920ms

[Q9/9] "Now subtract 150 from that result"
--------------------------------------------------------------------------------
  Run 1: PD=0.010 | CDI=0.558 | Traces=225 | Len=199ch | Latency=1480ms
  Run 2: PD=0.042 | CDI=0.614 | Traces=272 | Len=164ch | Latency=1827ms
  Run 3: PD=0.010 | CDI=0.558 | Traces=223 | Len=212ch | Latency=1097ms

════════════════════════════════════════════════════════════════════════════════
CONFIGURATION: NO_REASONING
Description: Reasoning disabled - should be faster, more concise
Thinking: OFF | Max Tokens: 2000
════════════════════════════════════════════════════════════════════════════════

[Q1/9] "What is 2 + 2?"
--------------------------------------------------------------------------------
  Run 1: PD=0.111 | CDI=0.651 | Traces=25 | Len=2ch | Latency=414ms
  Run 2: PD=0.088 | CDI=0.665 | Traces=299 | Len=278ch | Latency=1934ms
  Run 3: PD=0.022 | CDI=0.643 | Traces=35 | Len=2ch | Latency=434ms

[Q2/9] "What is 12 × 5?"
--------------------------------------------------------------------------------
  Run 1: PD=0.079 | CDI=0.683 | Traces=16 | Len=13ch | Latency=508ms
  Run 2: PD=0.022 | CDI=0.643 | Traces=34 | Len=13ch | Latency=501ms
  Run 3: PD=0.051 | CDI=0.654 | Traces=24 | Len=13ch | Latency=425ms

[Q3/9] "What is 127 × 83?"
--------------------------------------------------------------------------------
  Run 1: PD=0.023 | CDI=0.560 | Traces=103 | Len=19ch | Latency=857ms
  Run 2: PD=0.023 | CDI=0.636 | Traces=111 | Len=30ch | Latency=724ms
  Run 3: PD=0.023 | CDI=0.560 | Traces=65 | Len=29ch | Latency=763ms

[Q4/9] "If x² - 5x + 6 = 0, what are the solutions?"
--------------------------------------------------------------------------------
  Run 1: PD=0.052 | CDI=0.323 | Traces=40 | Len=246ch | Latency=799ms
  Run 2: PD=0.050 | CDI=0.000 | Traces=50 | Len=263ch | Latency=963ms
  Run 3: PD=0.033 | CDI=0.614 | Traces=1997 | Len=0ch | Latency=7876ms

[Q5/9] "Calculate 12,345 × 6,789 step by step"
--------------------------------------------------------------------------------
  Run 1: PD=0.050 | CDI=0.464 | Traces=680 | Len=1793ch | Latency=7608ms
  Run 2: PD=0.008 | CDI=0.464 | Traces=1008 | Len=1972ch | Latency=9960ms
  Run 3: PD=0.040 | CDI=0.000 | Traces=57 | Len=4722ch | Latency=10672ms

[Q6/9] "Solve x³ - 6x² + 11x - 6 = 0 for all real solutions"
--------------------------------------------------------------------------------
  Run 1: PD=0.051 | CDI=0.679 | Traces=629 | Len=1259ch | Latency=3415ms
  Run 2: PD=1.000 | CDI=0.614 | Traces=75 | Len=609ch | Latency=1719ms
  Run 3: PD=0.050 | CDI=0.000 | Traces=77 | Len=1174ch | Latency=2869ms

[Q7/9] "What is 100 + 200?"
--------------------------------------------------------------------------------
  Run 1: PD=0.059 | CDI=0.562 | Traces=329 | Len=282ch | Latency=2594ms
  Run 2: PD=0.051 | CDI=0.643 | Traces=343 | Len=173ch | Latency=2027ms
  Run 3: PD=0.051 | CDI=0.643 | Traces=30 | Len=20ch | Latency=449ms

[Q8/9] "Now multiply your previous answer by 3"
--------------------------------------------------------------------------------
  Run 1: PD=0.111 | CDI=0.663 | Traces=90 | Len=2ch | Latency=617ms
  Run 2: PD=0.042 | CDI=0.614 | Traces=397 | Len=210ch | Latency=3023ms
  Run 3: PD=0.040 | CDI=0.000 | Traces=566 | Len=431ch | Latency=2690ms

[Q9/9] "Now subtract 150 from that result"
--------------------------------------------------------------------------------
  Run 1: PD=0.042 | CDI=0.614 | Traces=208 | Len=142ch | Latency=1371ms
  Run 2: PD=0.042 | CDI=0.614 | Traces=219 | Len=219ch | Latency=1346ms
  Run 3: PD=0.040 | CDI=0.558 | Traces=235 | Len=183ch | Latency=1719ms


════════════════════════════════════════════════════════════════════════════════
📊 DEEP AUDIT ANALYSIS
════════════════════════════════════════════════════════════════════════════════

================================================================================
CLAIM 1 VALIDATION: "Budget control keeps costs predictable"
================================================================================

📈 TRACE COUNT BY BUDGET:
Budget    | Avg Traces | Min  | Max  | CV    | Variance
----------------------------------------------------------------------
BUDGET_LOW |        230 |   17 |  500 | 85.5% | ❌ High
BUDGET_MEDIUM |        289 |   23 | 1000 | 118.6% | ❌ High
BUDGET_HIGH |        269 |   21 | 1254 | 125.8% | ❌ High
BASELINE   |        190 |   24 |  777 | 112.7% | ❌ High

🔍 INTERPRETATION:
   ❌ CLAIM REJECTED: Low budget (230) ≈ High budget (269)
   Budget control has minimal effect on trace count

================================================================================
CLAIM 2 VALIDATION: "Reasoning mode can be toggled on/off"
================================================================================

📊 REASONING ON vs OFF:
   Traces:  ON=190 | OFF=287 | Reduction: -51%
   CDI:     ON=0.381 | OFF=0.510 | Change: 34%
   Latency: ON=1914ms | OFF=2529ms | Speedup: 0.8x

   ❌ CLAIM REJECTED: Reasoning toggle has minimal effect

================================================================================
CROSS-CORRELATION ANALYSIS
================================================================================

📊 CORRELATION MATRIX:
Metric Pair                  | Correlation | Strength
------------------------------------------------------------
Trace Count vs PD            |      -0.171 | WEAK
Trace Count vs CDI           |      -0.086 | WEAK
Response Length vs PD        |      -0.055 | WEAK
Response Length vs CDI       |      -0.373 | WEAK
Latency vs PD                |      -0.137 | WEAK
Latency vs CDI               |      -0.253 | WEAK
PD vs CDI                    |       0.191 | WEAK

================================================================================
VARIANCE ANALYSIS: PD Reproducibility by Configuration
================================================================================

📊 PD VARIANCE:
Config         | Avg PD | CV     | Reproducible?
------------------------------------------------------------
BASELINE        | 0.040  | 57.2% | ❌ No
BUDGET_LOW      | 0.038  | 43.0% | ❌ No
BUDGET_MEDIUM   | 0.152  | 197.7% | ❌ No
BUDGET_HIGH     | 0.184  | 185.3% | ❌ No
NO_REASONING    | 0.084  | 217.0% | ❌ No


╔═══════════════════════════════════════════════════════════════╗
║              DEEP AUDIT COMPLETE                             ║
╚═══════════════════════════════════════════════════════════════╝