Debugging a RequestContext Memory Leak with AI Assistance - A case study in human-AI collaboration

debugging-requestcontext-leak.md

Debugging a RequestContext Memory Leak with AI Assistance

TL;DR

A senior developer and Claude AI collaborated to diagnose and fix a memory leak in an ASP.NET application where RequestContext objects were being created but not torn down. The fix took ~2 hours of iterative debugging, not "ask AI and done." This document shows how AI assistance requires domain expertise to be effective.

The Problem

ActiveContextCount kept climbing from 1 to 139+ without returning to baseline. Requests were being set up but not torn down, causing a memory/connection leak.

The Debugging Journey

Phase 1: Initial Hypothesis (Wrong Direction)

AI's first theory: The authentication pipeline was making unnecessary DB calls for static assets.

We added static asset detection to skip GetUserId() calls for CSS/JS/fonts. This was a valid optimization but didn't address the leak.

Phase 2: Adding Instrumentation

AI suggestion: Add logging to track context creation/teardown.

// In RequestContext.Setup()
var count = Interlocked.Increment(ref _activeContextCount);
Loggers.Info(LogType.FRAMEWORK, $"RequestContext.Setup: ActiveContextCount={count}, ContextID={contextID}");

// In RequestContext.Teardown()
var count = Interlocked.Decrement(ref _activeContextCount);
Loggers.Info(LogType.FRAMEWORK, $"RequestContext.Teardown: ActiveContextCount={count}, ContextID={ContextID}");

Phase 3: Analyzing Logs (AI Misinterpretation)

The developer shared logs showing:

Teardown: ContextID=698575cce76c56350432ba10  (count 137)
Setup:    ContextID=698575cce76c56350432ba10  (count 138)
Setup:    ContextID=698575cde76c56350432ba2c  (count 138)

AI's interpretation: "The logs are out of order. Two requests are happening concurrently."

Developer's correction: "The logs are out of order, but what I'm showing you is a clear case of the file being requested twice but teardown only once."

This was the critical insight - 2 Setups, 1 Teardown = Leak.

Phase 4: Searching for the Cause

AI explored several theories:

1. Multiple SetupRequest() calls in Global.asax.cs (developer confirmed guards existed)
2. IIS static file handler bypassing ASP.NET (checked web.config - runAllManagedModulesForAllRequests="true")
3. Client aborting requests before EndRequest fires

Phase 5: Finding the Dual Teardown Hack

AI discovered two teardown paths:

1. ExecutingContext.Complete() in Global_EndRequest
2. RequestContextTeardown.Dispose() via DisposeOnPipelineCompleted

Developer's key context: "This dual teardown was a hack to try to fix the problem you are investigating. I suspected somehow Complete wasn't being called."

This told us the problem existed before the hack was added, and the hack wasn't working.

Phase 6: The Breakthrough

AI examined WebThreadStorageFactory:

public T Value {
    get {
        HttpContext httpContext = HttpContext.Current;
        if (httpContext != null && httpContext.Items.Contains(name))
            return (T)httpContext.Items[name];
        return default(T);  // Returns null if HttpContext.Current is null!
    }
}

The insight: If HttpContext.Current is null when DisposeOnPipelineCompleted runs (thread switching, async cleanup), the lookup fails and RequestContext.Current returns null.

Then in RequestContextTeardown.Dispose():

RequestContext.Current?.Teardown();  // null?.Teardown() = nothing happens!

Phase 7: First Fix Attempt (Incomplete)

AI's solution: Capture HttpContext directly in RequestContextTeardown.

Developer's correction: "I still think we have an issue because Teardown accesses Current which is thread storage which goes to HttpContext... I think we need to capture RequestContext.Current, not HttpContext."

The developer recognized that even with a captured HttpContext, the Teardown() method itself was calling var current = Current; internally, which would fail the same way.

Phase 8: The Final Fix

Two changes:

1. RequestContext.cs - Use this instead of Current, add teardown gate:

private bool _tornDown = false;

public void Teardown()
{
    if (_tornDown)
        return;
    _tornDown = true;

    // Changed from: var current = Current; current?._executionTimer.Stop();
    // To:
    _executionTimer.Stop();
    OnRequestCompleted?.Invoke(this);  // Use 'this', not 'current'
    // ... rest uses 'this' throughout
}

2. Global.asax.cs - Capture the instance directly:

private static URL SetupRequest()
{
    if (!ExecutingContext.IsInitialized)
    {
        RequestContext.Setup(context, url, nestedContainer);
        // Capture AFTER Setup creates it - direct reference, no HttpContext dependency
        System.Web.HttpContext.Current.DisposeOnPipelineCompleted(
            new RequestContextTeardown(RequestContext.Current));
    }
    return url;
}

Key Lessons

1. AI Needs Domain Context

The AI couldn't know that:

• The dual teardown was already a hack attempting to fix this
• HttpContext.Current can be null during pipeline cleanup
• The codebase's specific threading model and async patterns

2. Iterative Feedback is Essential

Each correction from the developer refined the search:

• "Keep logging" → Instrumentation was valuable
• "2 Setups, 1 Teardown" → Identified the exact symptom
• "Capture RequestContext, not HttpContext" → Pointed to the real fix

3. AI Accelerates, Doesn't Replace

The AI could:

• Search the codebase quickly
• Trace code paths
• Generate hypotheses
• Write the fix once identified

The AI couldn't:

• Know which hypotheses were already tried
• Understand the historical context of "hack" code
• Recognize subtle ASP.NET threading behaviors without hints

4. The Expert Guides, AI Executes

The developer's role:

• Provided critical context ("this was a hack")
• Corrected wrong directions quickly
• Recognized the real issue from AI's analysis
• Validated the final approach

The AI's role:

• Explored the codebase systematically
• Generated multiple hypotheses
• Wrote instrumentation and fixes
• Documented the journey

The Root Cause (Summary)

HttpContext.Current can be null when DisposeOnPipelineCompleted runs due to ASP.NET's thread management. The thread storage lookup failed, RequestContext.Current returned null (cast to DummyContext), and Teardown() was never called on the actual context instance.

Fix: Capture the RequestContext instance directly when it's created, eliminating the dependency on HttpContext.Current during teardown.

---

Debugging session: ~2 hours Lines of code changed: ~30 Hypotheses explored: 6+ Developer corrections that redirected the investigation: 4