Attach to a running python process and explore its state - useful for troubleshooting deadlocks, memory leaks, etc.

python_attach_howto.md

Attach to PID with GDB

Prerequisites

• If running inside docker, make sure to execute docker exec with --privileged
• Install remote_pdb

pip install remote_pdb

gdb -p <pid>

Interactively inspect the running process

To interactively debug a running python process, you can use gdb to execute pdb:

# via stdin/stdout
call (int)PyGILState_Ensure()
call (int)PyRun_SimpleString("import pdb; pdb.run('where'); pdb.set_trace()")

# via nc
call (int)PyGILState_Ensure()
call (int)PyRun_SimpleString("import remote_pdb; remote_pdb.set_trace('localhost', 4444)")

rlwrap -a -c nc localhost 4444

Once in pdb you can traverse up and down in the call stack and inspect variables

# show current stack trace
where

up
down
locals()

If in async, one of these frames should have the main coroutine. You can traverse into the coroutine and inspect its variables:

# get all async tasks
(Pdb) import asyncio
(Pdb) tasks = list(asyncio.all_tasks())

# inspect task
(Pdb) task = tasks[0]
(Pdb) coro = task.get_coro()
(Pdb) frame = coro.cr_frame
(Pdb) frame
<frame at 0x7444843b9300, file '/root/bbot/bbot/cli.py', line 246, code _main>
# print its local variables
(Pdb) frame.f_locals
{'scan': <bbot.scanner.scanner.Scanner object at 0x74448430ff50>}

You can do the same to threads:

(Pdb) import threading
(Pdb) threads = threading.enumerate()
(Pdb) desired_thread = threads[1]
# and traverse into them
(Pdb) import sys
(Pdb) frames = sys._current_frames()
(Pdb) frame = frames[desired_thread.ident]
# you can traverse the stack and print variables
(Pdb) frame = frame.f_back
(Pdb) frame.f_locals
(Pdb) frame = frame.f_back
(Pdb) frame.f_locals
# show as string
(Pdb) traceback.format_stack(frame)
# print to stdout of process
(Pdb) traceback.print_stack(frame)

Print the traceback

(gdb) call (int)PyGILState_Ensure()
(gdb) call (int)PyRun_SimpleString("import traceback; traceback.print_stack()")
(gdb) call (int)PyGILState_Release($1)

Dump detailed traceback to file

code.py:

import sys
import linecache

def get_stack_info():
    stack_info = []
    frame = sys._getframe(1)  # Start from the caller's frame
    while frame:
        filename = frame.f_code.co_filename
        lineno = frame.f_lineno
        function = frame.f_code.co_name
        locals_dict = dict(frame.f_locals)
        line = linecache.getline(filename, lineno).strip()
        stack_info.append((filename, lineno, function, locals_dict, line))
        frame = frame.f_back
    return stack_info

output_file = '/tmp/debug_output.txt'

with open(output_file, 'w') as f:
    f.write('Call stack with local variables:\n')
    for filename, lineno, function, locals_dict, line in reversed(get_stack_info()):
        f.write(f'\n{filename}:{lineno} in {function}\n')
        f.write(f'  {line}\n')
        for key, value in list(locals_dict.items()):
            try:
                f.write(f'    {key} = {str(value)}\n')
            except Exception as e:
                f.write(f'    {key} = ERROR on type:{type(value)}: {e}\n')

print(f'Debug information written to {output_file}')

(gdb) call (int)PyGILState_Ensure()
(gdb) call (int)PyRun_AnyFile(fopen("/code.py", "r"), "/code.py")
(gdb) call (int)PyGILState_Release($1)