Skip to content

Instantly share code, notes, and snippets.

@aperiodic
Created January 25, 2012 01:23
Show Gist options
  • Save aperiodic/1673985 to your computer and use it in GitHub Desktop.
Save aperiodic/1673985 to your computer and use it in GitHub Desktop.
TT JStack (when reduce tasks are hanging)
Deadlock Detection:
No deadlocks found.
Thread 23966: (state = IN_NATIVE)
- java.lang.UNIXProcess.waitForProcessExit(int) @bci=0 (Interpreted frame)
- java.lang.UNIXProcess.access$900(java.lang.UNIXProcess, int) @bci=2, line=20 (Interpreted frame)
- java.lang.UNIXProcess$1$1.run() @bci=165, line=132 (Interpreted frame)
Thread 23965: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
- java.lang.UNIXProcess.waitFor() @bci=8, line=165 (Compiled frame)
- org.apache.hadoop.util.Shell.runCommand() @bci=233, line=244 (Interpreted frame)
- org.apache.hadoop.util.Shell.run() @bci=23, line=182 (Interpreted frame)
- org.apache.hadoop.util.Shell$ShellCommandExecutor.execute() @bci=1, line=375 (Interpreted frame)
- org.apache.hadoop.mapred.DefaultTaskController.launchTask(java.lang.String, java.lang.String, java.lang.String, java.util.List, java.util.List, java.io.File, java.lang.String, java.lang.String) @bci=291, line=133 (Interpreted frame)
- org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(org.apache.hadoop.mapred.JvmManager$JvmEnv) @bci=157, line=497 (Interpreted frame)
- org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run() @bci=5, line=471 (Interpreted frame)
Thread 23962: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.hadoop.mapred.TaskRunner.launchJvmAndWait(java.util.List, java.util.Vector, java.io.File, java.io.File, long, java.io.File) @bci=47, line=267 (Interpreted frame)
- org.apache.hadoop.mapred.TaskRunner.run() @bci=274, line=222 (Interpreted frame)
Thread 23184: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=198 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=424 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=323 (Compiled frame)
- java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=874 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=62, line=945 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23349: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- org.apache.hadoop.mapred.TaskTracker$1.run() @bci=7, line=417 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23348: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.monitor() @bci=4, line=131 (Interpreted frame)
- org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager$1.run() @bci=4, line=66 (Interpreted frame)
Thread 23347: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- org.apache.hadoop.mapred.UserLogCleaner.run() @bci=4, line=93 (Interpreted frame)
Thread 23346: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Compiled frame)
- java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame)
Thread 23345: (state = IN_NATIVE)
- sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
- sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
- sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
- org.mortbay.io.nio.SelectorManager$SelectSet.doSelect() @bci=723, line=511 (Compiled frame)
- org.mortbay.io.nio.SelectorManager.doSelect(int) @bci=24, line=193 (Compiled frame)
- org.mortbay.jetty.nio.SelectChannelConnector.accept(int) @bci=5, line=124 (Compiled frame)
- org.mortbay.jetty.AbstractConnector$Acceptor.run() @bci=153, line=708 (Compiled frame)
- org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25, line=582 (Interpreted frame)
Thread 23344: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run() @bci=29, line=2180 (Interpreted frame)
Thread 23343: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run() @bci=29, line=2180 (Interpreted frame)
Thread 23339: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.mapred.TaskTracker$MapEventsFetcherThread.run() @bci=265, line=982 (Compiled frame)
Thread 23337: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59, line=680 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=723 (Compiled frame)
Thread 23336: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- org.apache.hadoop.ipc.Server$Handler.run() @bci=68, line=1410 (Compiled frame)
Thread 23335: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- org.apache.hadoop.ipc.Server$Handler.run() @bci=68, line=1410 (Compiled frame)
Thread 23334: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- org.apache.hadoop.ipc.Server$Handler.run() @bci=68, line=1410 (Compiled frame)
Thread 23333: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- org.apache.hadoop.ipc.Server$Handler.run() @bci=68, line=1410 (Compiled frame)
Thread 23332: (state = IN_NATIVE)
- sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
- sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
- sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
- sun.nio.ch.SelectorImpl.select() @bci=2, line=84 (Interpreted frame)
- org.apache.hadoop.ipc.Server$Listener.run() @bci=56, line=468 (Interpreted frame)
Thread 23331: (state = IN_NATIVE)
- sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
- sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
- sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
- org.apache.hadoop.ipc.Server$Responder.run() @bci=63, line=638 (Interpreted frame)
Thread 23330: (state = IN_NATIVE)
- sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
- sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
- sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
- sun.nio.ch.SelectorImpl.select() @bci=2, line=84 (Compiled frame)
- org.apache.hadoop.ipc.Server$Listener$Reader.run() @bci=53, line=352 (Compiled frame)
Thread 23329: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Compiled frame)
- java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame)
Thread 23298: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- org.apache.hadoop.mapred.CleanupQueue$PathCleanupThread.run() @bci=47, line=130 (Compiled frame)
Thread 23297: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Compiled frame)
- java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame)
Thread 23287: (state = BLOCKED)
Thread 23286: (state = BLOCKED)
Thread 23285: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame)
Thread 23284: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
Thread 23277: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.mapred.TaskTracker.offerService() @bci=62, line=1546 (Compiled frame)
- org.apache.hadoop.mapred.TaskTracker.run() @bci=56, line=2344 (Interpreted frame)
- org.apache.hadoop.mapred.TaskTracker.main(java.lang.String[]) @bci=68, line=3614 (Interpreted frame)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment