readme.md

I recently came across a situation where I wanted to capture the output of a subprocess started by a Python script, but also let it print to the terminal normally. An example of where this may be useful is with something like curl, where progress is output to stderr (with the -o option). In an interactive program, you may want to show the user that progress information, but also capture it for parsing in your script. By default, subprocess.run does not capture any output, but the subprocess does print to the terminal. Passing stdout=subprocess.PIPE, stderr=subprocess.STDOUT to subprocess.run captures the output but does not let the subprocess print. So you don't see any output until the subprocess has completed. Redirecting sys.stdout or sys.stderr doesn't work because it only replaces the Python script's stdout or stderr, it doesn't have an effect on the subprocess'.

The only way to accomplish this seems to be to start the subprocess with the non-blocking subprocess.Popen, poll for available output, and both print it and accumulate it in a variable. The code shown here requires the selectors module, which is only available in Python 3.4+.

capture-and-print-subprocess-output.py

import io
import selectors
import subprocess
import sys

def capture_subprocess_output(subprocess_args):
    # Start subprocess
    # bufsize = 1 means output is line buffered
    # universal_newlines = True is required for line buffering
    process = subprocess.Popen(subprocess_args,
                               bufsize=1,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.STDOUT,
                               universal_newlines=True)

    # Create callback function for process output
    buf = io.StringIO()
    def handle_output(stream, mask):
        # Because the process' output is line buffered, there's only ever one
        # line to read when this function is called
        line = stream.readline()
        buf.write(line)
        sys.stdout.write(line)

    # Register callback for an "available for read" event from subprocess' stdout stream
    selector = selectors.DefaultSelector()
    selector.register(process.stdout, selectors.EVENT_READ, handle_output)

    # Loop until subprocess is terminated
    while process.poll() is None:
        # Wait for events and handle them with their registered callbacks
        events = selector.select()
        for key, mask in events:
            callback = key.data
            callback(key.fileobj, mask)

    # Get process return code
    return_code = process.wait()
    selector.close()

    success = (return_code == 0)

    # Store buffered output
    output = buf.getvalue()
    buf.close()

    return (success, output)

Thanks for this code snippet.

I'm using this to call bash commands from python. It works great when you don't need to keep the bash context by passing it ["bash","-c", statement] where statement is the bash command.

However, I'd like to keep a bash subprocess running and feeding it commands when I wanna. I've tried calling your snipped with only ["bash"] so it stays open, adding stdin as PIPE and then writing commands to stdin and it locks.

I don't know why that happens

This was very helpful! Thank you!

This worked very well for me, but I noticed that at times it would not get the last line of stdout. I added this code snippet

    # Ensure all remaining output is processed
    while True:
        line = process.stdout.readline()
        if not line:
            break
        buf.write(line)
        sys.stdout.write(line)

at line 36, and I believe it catches all of the dangling lines.