Lab: Java Production Debugging 101 (summary)

java_debugging_lab.md

Lab: Java Production Debugging 101 (summary)

This lab will focus on common Java server production issues (including deadlocks, memory leaks and thread starvation), and introduce some of these tools by demonstrating how they can be used to analyze and resolve each scenario.

Forensics process

• Gather evidence
• Restore production
• Analyze findings
• Implement solution
• Post Mortem

What to collect

• Thread dumps
• Heap dump
• VM \ GC metrics
• System metrics
• Logs Will be omitted from discussion, because logs are application-specific.

Always use JDK, not JRE. JDK offers great analysis tools with the same performance as JRE.

Tools

jStack

• Lightweight, minimalistic
• Non-blocking, works against a running process
• Simple stdout output
• Identifies deadlocks

jstat

• Tracks JVM metrics over time
• Console-based
• Especially useful for GC monitoring

jmap

• Heap dump
• Lengthy process
• Freezes VM
• Not recommended for most scenarios

The JVM GC

From empirical experience, most object live for a very short time. Hence, the heap is divided into age buckets, and surviving objects are promoted from younger age buckets to older age buckets.

When the Tenured GC is filled, a Full GC cleanup is performed. This is often experienced as a long freeze.