.NET Debugging.md

Comprehensive Learning Examples from DotNetBuggyApp

Based on the documentation analysis, here are all the practical configuration examples and learning points you should understand:

1. Container Configuration Examples

Dockerfile/Containerfile for .NET Diagnostic Tools

FROM mcr.microsoft.com/dotnet/sdk:8.0

WORKDIR /app

# User/group setup for security
RUN chown 1001:0 /app

# Switch to root to install tools
USER root

# Install .NET diagnostic tools
RUN dotnet tool install --global dotnet-dump --version 8.*
RUN dotnet tool install --global dotnet-trace --version 8.*
RUN dotnet tool install --global dotnet-counters --version 8.*
RUN dotnet tool install --global dotnet-gcdump --version 8.*

# Add tools to PATH
ENV PATH="${PATH}:/root/.dotnet/tools"

# For sidecar debugging - install to specific path
RUN mkdir -p /app/tools && chown 1001:0 /app/tools && \
    dotnet tool install --tool-path /app/tools dotnet-dump --version "8.*" && \
    dotnet tool install --tool-path /app/tools dotnet-gcdump --version "8.*" && \
    dotnet tool install --tool-path /app/tools dotnet-trace --version "8.*" && \
    dotnet tool install --tool-path /app/tools dotnet-counters --version "8.*"

# Switch back to non-root user
USER 1001

Learning Points:

• How to install .NET diagnostic tools in containers
• User permission management in containers
• PATH configuration for global tools

2. Kubernetes Deployment Configuration

Basic Deployment with Crash Dump Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dotnet-memory-leak-app
  namespace: dotnet-memory-leak-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dotnet-memory-leak-app
  template:
    metadata:
      labels:
        app: dotnet-memory-leak-app
    spec:
      serviceAccountName: dotnet-app-sa
      containers:
      - name: dotnet-app
        image: quay.io/your-namespace/dotnet-memory-leak-app:v1
        ports:
        - containerPort: 8881
        env:
        # Critical .NET crash dump configuration
        - name: COMPlus_DbgEnableElfDumpOnCrash
          value: "1"
        - name: COMPlus_DbgCrashDumpType
          value: "3"  # Full dump
        - name: COMPlus_DbgMiniDumpName
          value: "/app/dumps/dump.dmp"
        - name: TMPDIR
          value: "/app/dumps/tmp"
        volumeMounts:
        - name: dump-storage
          mountPath: /app/dumps
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          runAsUser: 1001
          capabilities:
            drop:
            - ALL
      volumes:
      - name: dump-storage
        persistentVolumeClaim:
          claimName: dotnet-memory-leak-dumps

Learning Points:

• .NET crash dump environment variables
• Resource requests vs limits
• Volume mounting for persistent storage
• Security context configuration

LimitRange Configuration

apiVersion: v1
kind: LimitRange
metadata:
  name: dotnet-limitrange
  namespace: dotnet-memory-leak-app
spec:
  limits:
  - type: Container
    # Default values if not specified
    defaultRequest:
      cpu: "100m"
      memory: "64Mi"
    default:
      cpu: "500m"
      memory: "256Mi"
    # Maximum allowed values
    max:
      cpu: "1"
      memory: "1Gi"
    # Minimum required values
    min:
      cpu: "50m"
      memory: "32Mi"

Learning Points:

• Resource governance in Kubernetes
• Default resource allocation
• Preventing resource starvation

Security Context Constraints (OpenShift)

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: dotnet-scc
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegedContainer: false
allowedCapabilities:
- SYS_PTRACE  # Required for debugging
defaultAddCapabilities: []
fsGroup:
  type: RunAsAny
readOnlyRootFilesystem: false
requiredDropCapabilities:
- KILL
- MKNOD
- SETUID
- SETGID
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
users:
- system:serviceaccount:dotnet-memory-leak-app:dotnet-app-sa

Learning Points:

• OpenShift security model
• Required capabilities for debugging
• Service account binding

3. Building and Deployment Commands

Container Build Process

# Navigate to project directory
cd DotNetBuggyApp-main

# Build container image
podman build -t quay.io/your-namespace/dotnet-memory-leak-app:v1 -f Containerfile .

# Push to registry
podman push quay.io/your-namespace/dotnet-memory-leak-app:v1

# For air-gapped environments
podman save -o dotnet-memory-leak-app.tar quay.io/your-namespace/dotnet-memory-leak-app:v1

# Transfer and load in air-gapped environment
sudo podman load -i dotnet-memory-leak-app.tar
sudo podman images  # Verify image is loaded

Learning Points:

• Container build workflows
• Registry operations
• Air-gapped deployment strategies

Kubernetes Deployment Commands

# Deploy using Kustomize
cd DotNetBuggyApp-main/kubernetes
oc apply -k .

# Check deployment status
oc get pods -n dotnet-memory-leak-app --watch

# Get application route
export ROUTE_HOST=$(oc get route dotnet-memory-leak-route -n dotnet-memory-leak-app -o jsonpath='{.spec.host}')

# Trigger memory leak
curl http://$ROUTE_HOST/triggerMemoryLeak

Learning Points:

• Kustomize usage
• Route/Ingress configuration
• Application endpoint testing

4. Debugging Techniques

Method 1: Embedded Tools (Educational - Not Recommended)

# Get pod name
export POD_NAME=$(oc get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')

# Access container shell
oc rsh -c dotnet-app "$POD_NAME"

# Inside container - find process
ps -ef

# Collect dump
dotnet-dump collect --process-id <PID> -o /app/dumps/app_collected_dump.dmp

# Exit container
exit

Learning Points:

• Container shell access
• Process identification
• Manual dump collection
• Why this approach is problematic

Method 2: Automatic OOM Dumps (Recommended)

# Monitor logs during memory leak
export POD_NAME=$(oc get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')
oc logs "$POD_NAME" -n dotnet-memory-leak-app -f

# After crash, verify dump exists
oc rsh "$POD_NAME" -c dotnet-app -- ls -l /app/dumps/

# Copy dump for analysis
oc cp "$POD_NAME":/app/dumps/dump.dmp ./crash_dump.dmp -n dotnet-memory-leak-app

# Analyze dump locally
dotnet-dump analyze ./crash_dump.dmp

Learning Points:

• Automatic crash dump generation
• Log monitoring techniques
• File extraction from containers
• Local dump analysis

Method 3: Sidecar Container Pattern

// add-sidecar.json
[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/-",
    "value": {
      "name": "debugger",
      "image": "quay.io/your-namespace/dotnet-debug:v1",
      "command": ["sleep", "infinity"],
      "env": [
        {
          "name": "TMPDIR",
          "value": "/app/dumps/tmp"
        }
      ],
      "volumeMounts": [
        {
          "mountPath": "/app/dumps",
          "name": "dump-storage"
        }
      ]
    }
  }
]

// remove-sidecar.json
[
  {
    "op": "remove",
    "path": "/spec/template/spec/containers/1"
  }
]

# Add sidecar for debugging
oc patch deployment dotnet-memory-leak-app -n dotnet-memory-leak-app --type=json --patch-file add-sidecar.json

# Wait for rollout
oc get pods -n dotnet-memory-leak-app --watch

# Access debugger sidecar
export POD_NAME=$(oc get pods -l app=dotnet-memory-leak-app -n dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')
oc exec -it "$POD_NAME" -n dotnet-memory-leak-app -c debugger -- /bin/bash

# Collect dump from sidecar
ps -ef  # Find main app process
/app/tools/dotnet-dump collect -p <PID> -o /app/dumps/dump_SUCCESS.dmp

# Remove sidecar when done
oc patch deployment dotnet-memory-leak-app -n dotnet-memory-leak-app --type=json --patch-file remove-sidecar.json

Learning Points:

• JSON patch operations
• Sidecar container patterns
• Dynamic deployment modification
• Shared volume usage between containers

Method 4: Ephemeral Debug Containers

# Get pod name
export POD_NAME=$(kubectl get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')

# Launch ephemeral debug container
kubectl debug -it "$POD_NAME" --image=quay.io/your-namespace/dotnet-memory-leak-app:v1 --target=dotnet-app -- /bin/bash

# Alternative with SDK image
kubectl debug -it "$POD_NAME" --image=registry.redhat.io/rhel8/dotnet-80:8.0 --target=dotnet-app -- /bin/bash

# Inside ephemeral container
ps -ef  # Find main app process
dotnet-dump collect --process-id <PID> -o /app/dumps/ephemeral_collected_dump.dmp

Learning Points:

• Modern Kubernetes debugging features
• Ephemeral containers concept
• Namespace sharing between containers
• kubectl vs oc differences

5. Monitoring and Logging

Application Monitoring Commands

# Stream application logs
oc logs "$POD_NAME" -n dotnet-memory-leak-app -f

# Expected log output during memory leak:
# info: Program[0]
#       Allocated 1516.00 MB this round, Total: 1516.00 MB
# fail: Program[0]
#       OutOfMemoryException caught! Application is likely to crash soon.
#       System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

# Check pod status
oc get pods -n dotnet-memory-leak-app
oc describe pod "$POD_NAME" -n dotnet-memory-leak-app

# Check events
oc get events -n dotnet-memory-leak-app --sort-by='.lastTimestamp'

Learning Points:

• Real-time log monitoring
• Pod lifecycle events
• OutOfMemory behavior patterns

6. Security Configuration Examples

Pod Security Policy Violations

# Common warning messages you'll encounter:
# Warning: would violate PodSecurity "restricted:latest": 
# unrestricted capabilities (container "diagnostic-tools-sidecar" must set securityContext.capabilities.drop=["ALL"]; 
# container "diagnostic-tools-sidecar" must not include "SYS_PTRACE" in securityContext.capabilities.add)

# Error: creating: pods "..." is forbidden: violates PodSecurity "restricted"

Troubleshooting Security Issues

# Check current security context constraints
oc get scc

# Describe specific SCC
oc describe scc dotnet-scc

# Check service account permissions
oc describe sa dotnet-app-sa -n dotnet-memory-leak-app

# Verify RBAC
oc describe rolebinding dotnet-app-rolebinding -n dotnet-memory-leak-app

Learning Points:

• Security policy debugging
• RBAC troubleshooting
• Permission escalation patterns

7. Resource Management Examples

Resource Monitoring

# Check resource usage
oc top pods -n dotnet-memory-leak-app
oc top nodes

# Describe resource limits
oc describe limitrange dotnet-limitrange -n dotnet-memory-leak-app

# Check PVC status
oc get pvc -n dotnet-memory-leak-app
oc describe pvc dotnet-memory-leak-dumps -n dotnet-memory-leak-app

Learning Points:

• Resource consumption monitoring
• Storage management
• Capacity planning

8. Environment Variable Configuration

Critical .NET Diagnostic Variables

env:
- name: COMPlus_DbgEnableElfDumpOnCrash
  value: "1"                    # Enable crash dumps
- name: COMPlus_DbgCrashDumpType  
  value: "3"                    # Full dump (vs mini dump)
- name: COMPlus_DbgMiniDumpName
  value: "/app/dumps/dump.dmp"  # Output path
- name: TMPDIR
  value: "/app/dumps/tmp"       # Temp directory for IPC

Learning Points:

• .NET runtime configuration
• Crash dump types and their uses
• Inter-process communication setup

9. Air-Gapped Deployment Workflow

Complete Air-Gapped Process

# 1. Build and save image (connected environment)
podman build -t app:v1 .
podman save -o app.tar app:v1

# 2. Transfer to air-gapped environment
scp app.tar user@airgapped-host:~/

# 3. Load in air-gapped environment
sudo podman load -i app.tar

# 4. Deploy using local registry or direct image reference
oc apply -k kubernetes/

Learning Points:

• Offline deployment strategies
• Image transfer techniques
• Registry-less deployments

Key Learning Outcomes

After working through these examples, you should understand:

1. Container Security: How security policies affect debugging capabilities
2. Resource Management: Critical importance in single-node clusters
3. Debugging Strategies: Multiple approaches for different scenarios
4. Production Readiness: Trade-offs between security and observability
5. Kubernetes Operations: Real-world deployment and troubleshooting
6. .NET Diagnostics: Proper configuration for containerized applications
7. Air-Gapped Operations: Deployment in restricted environments

This project provides a comprehensive hands-on learning experience for container orchestration, application debugging, and production operations in Kubernetes/OpenShift environments.