dotnet-roslyn-roslyn-CI-health.md

Health Report — dotnet/roslyn / roslyn-CI

Overall Health: 🔴 RED (Critical — Infrastructure Failure + Masked Regression)

Current Status: Helix Infrastructure Instability + Masked Razor Regression

The pipeline remains critically impaired. 7 consecutive main branch failures since the last success (1425328, May 18 2:57 PM PDT). Only 3 out of 25 main builds succeeded in the last 3 days (12% success rate).

Active Problems

P1 — Helix Work Item Infrastructure Crashes (Days Active: 3+)

• Impact: Primary cause of all CI failures; every main build since 1425328 fails due to this
• Root Cause: Helix test harness crashes on two queues:
  • windows.amd64.vs2026.pre.scout.open (32-bit legs) — exit code 1
  • Windows.10.Amd64.Server2022.ES.Open (Spanish_64 leg) — exit code -3
• Severity: Builds 1425605 and 1425492 produce zero test results — harness crashes before publishing. Other builds show single workitem crashes (workitem_2, workitem_21, workitem_23).
• Affected Jobs: Test_Windows_Desktop_Debug_32, Test_Windows_Desktop_Release_32, Test_Windows_Desktop_Spanish_Release_64, Test_Windows_CoreClr_RuntimeAsync_Debug
• Status: 🔴 BLOCKING — no improvement since last report
• Last Successful Main Build: 1425328 (May 18, 2:57 PM PDT)
• Action Required: 🚨 Escalate to Helix/infrastructure team

P2 — Razor AddImport Regression (Days Active: ~1, MASKED)

• Impact: Medium — confirmed failure on main in build 1425519 (Test_Linux_Debug leg)
• Tests: AddUsing, AddUsing_Multiple, AddUsing_Typo, AddUsing_WithExisting, FullyQualify, FullyQualify_LegacyWithPageDirective, and others (10 total)
• Error: Code action AddImport no longer available; only GenerateType offered
• Known Issue: #83775 filed
• Status: 🟡 MASKED — Test_Linux_Debug has not reported results in any build since 1425519. Cannot verify if fixed.
• Action Required: Razor/IDE team should investigate proactively. Will resurface once Helix stabilizes.

P3 — Pool Leak: ArrayBuilder<PENamedTypeSymbol> (Days Active: 3+)

• Impact: Low — sporadic, 2 CI occurrences in window
• Test: TestNoSuppression_CompilerSemanticError
• Known Issue: #83749
• Status: 🟡 Tracked, low priority

Recently Resolved ✅

• IDE0005 Razor Build Errors — Last seen May 18, 11:04 AM (build 1424439). Not recurring. Fixed.
• Compound Assignment Initializer Tests — Confirmed PR-only (feature development on features/closed-class), not CI concern.

Metrics

Metric	Value	Status
CI Success Rate (Last 24h)	~14% (1/7 main builds)	🔴 CRITICAL
CI Success Rate (3-day)	12% (3/25 main builds)	🔴 CRITICAL
Consecutive Main Failures	7	🔴
Last Successful Main Build	May 18, 2:57 PM PDT
Trend	➡️ UNCHANGED — same infra issues persist
Days in RED	3+

Risk Assessment

• PR Merge Confidence: 🔴 VERY LOW — Builds fail regardless of code quality
• Regression Detection: 🔴 COMPROMISED — Real failures (P2) masked by infra noise (P1)
• Developer Productivity: 🔴 SEVERELY IMPACTED — Multiple retries needed per merge

Immediate Actions Required

1. 🚨 URGENT — Escalate Helix Stability: Both Windows.10.Amd64.Server2022.ES.Open and windows.amd64.vs2026.pre.scout.open queues are non-functional. Infrastructure team must investigate work item crash pattern (exit codes 1 and -3).

2. 🟡 Investigate Razor AddImport regression: Known issue #83775 filed but fix unverified. Once Helix stabilizes, these tests will likely fail again on Test_Linux_Debug.

3. Consider disabling Spanish_64 and 32-bit legs temporarily to improve success rate while queues are repaired.

4. Monitor Pool Leak: #83749 — fix when bandwidth allows.

---

Last Updated: May 18, 2026 11:11 PM PDT Days in RED: 3+ Change from Previous: No improvement. Consecutive failure count increased from 6 to 7. Razor regression remains masked. No new issues emerged. Next Review: After infrastructure team acknowledges Helix issue OR next successful main build Status: 🔴 CRITICAL — Infrastructure Failure + Masked Product Regression