How to get a stack trace from a stuck/hanging python script

pyrasite-stacktrace-how-to.md

How to get a stack trace for each thread in a running Python script

Sometimes a Python script will simply hang forever with no indication of what is going wrong. Perhaps it's polling a service that will never return a value that allows the program to move forward.

Here's a way to see where the program is currently stuck, using pyrasite a tool for injecting code into running Python processes.

Install gdb and pyrasite

Install gdb.

# Redhat, CentOS, etc
$ yum install gdb

# Ubuntu, Debian, etc
$ apt-get update && apt-get install gdb

Install pyrasite, for example with pip (in a virtual environment):

$ pip install pyrasite

Inspect running process with pyrasite

Find the process ID for the stuck Python process and run pyrasite-shell with it.

# Assuming process ID is 12345
$ pyrasite-shell 12345

You should now see a Python REPL. Run the following in the REPL to see stack traces for all threads.

import sys, traceback

def show_thread_stacks():
    for thread_id, frame in sys._current_frames().items():
        print('\n--- Stack for thread {t} ---'.format(t=thread_id))
        traceback.print_stack(frame, file=sys.stdout)

show_thread_stacks()

Just press up and enter to run it again.

Non-interactive

Alternatively, save that snippet in a file (e.g. show-stacktraces.py) and inject that into the running process:

# Assuming process ID is 12345
$ pyrasite 12345 show-stacktraces.py

It's important to note that the stack traces will now show up in the standard output of the running process and not in your current shell like with pyrasite-shell.

pyrasite-shell hangs on Arch Linux and openSUSE. This change would make it work: lmacken/pyrasite#75 (comment)

On CentOS 7.6 the interactive version fails for me with:

Traceback (most recent call last):
  File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/bin/pyrasite-shell", line 8, in <module>
    sys.exit(shell())
  File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pyrasite/tools/shell.py", line 61, in shell
    payload = ipc.recv()
  File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pyrasite/ipc.py", line 174, in recv
    header_data = self.recv_bytes(4)
  File "/opt/pytroll/pytroll_inst/miniconda3/envs/pytroll-py38/lib/python3.8/site-packages/pyrasite/ipc.py", line 187, in recv_bytes
    chunk = self.sock.recv(n - len(data))
socket.timeout: timed out

the non-interactive version shows no traceback, but nothing is written to the stdout of the running process (which is running as a supervisord subprocess).

Seems like the pyrasite[dot]com link is broken and redirects to a scam site.

@DiegoSanchezE thanks for letting me know!
I update the link to github repo which is probably more stable