How to get a stack trace from a stuck/hanging python script

pyrasite-stacktrace-how-to.md

How to get a stack trace for each thread in a running Python script

Sometimes a Python script will simply hang forever with no indication of what is going wrong. Perhaps it's polling a service that will never return a value that allows the program to move forward.

Here's a way to see where the program is currently stuck, using pyrasite a tool for injecting code into running Python processes.

Install gdb and pyrasite

Install gdb.

# Redhat, CentOS, etc
$ yum install gdb

# Ubuntu, Debian, etc
$ apt-get update && apt-get install gdb

Install pyrasite, for example with pip (in a virtual environment):

$ pip install pyrasite

Inspect running process with pyrasite

Find the process ID for the stuck Python process and run pyrasite-shell with it.

# Assuming process ID is 12345
$ pyrasite-shell 12345

You should now see a Python REPL. Run the following in the REPL to see stack traces for all threads.

import sys, traceback

def show_thread_stacks():
    for thread_id, frame in sys._current_frames().items():
        print('\n--- Stack for thread {t} ---'.format(t=thread_id))
        traceback.print_stack(frame, file=sys.stdout)

show_thread_stacks()

Just press up and enter to run it again.

Non-interactive

Alternatively, save that snippet in a file (e.g. show-stacktraces.py) and inject that into the running process:

# Assuming process ID is 12345
$ pyrasite 12345 show-stacktraces.py

It's important to note that the stack traces will now show up in the standard output of the running process and not in your current shell like with pyrasite-shell.