python get wget progress bar output.md

###trick

if you create a new wget child process "wget http://foo.bar", but redirect child stdin and stdout to the same pipe, wget will give you something like this

  0K .......... .......... .......... .......... ..........  0%  105K 2h1m
 50K .......... .......... .......... .......... ..........  0%  158K 1h41m
100K .......... .......... .......... .......... ..........  0%  302K 81m58s

if you want get a progress bar like output, you should redirect child stdin and stdout to different pipe, or not use redirection, just use fork default behavior, inherit parent's stdout and stderr

7% [=====>                                                                              ] 58,231,600  2.63M/s  eta 5m 7s

###use subprocess to redirect wget like process bar output

http://stackoverflow.com/questions/11179321/reading-stdout-of-subprocess

import logging
import subprocess
import sys
import inspect

def run_and_printchar(args):
    try:
        pipe = subprocess.Popen(args, bufsize = 0,
            shell = False,
            stdout = None, # no redirection, child use parent's stdout
            stderr = subprocess.PIPE) # redirection stderr, create a new pipe, from which later we will read
    except Exception as e:
        #inspect.stack()[1][3] will get caller function name
        logging.error(inspect.stack()[1][3] + ' error: ' + str(e))
        return False
    while 1:
        #use read(1), can get wget progress bar like output
        s = pipe.stderr.read(1)
        if s:
            sys.stdout.write(s)
        if pipe.returncode is None:
            code = pipe.poll()
        else:
            break
    if not 0 == pipe.returncode:
        return False
    return True
    
run_and_printchar(['wget','http://mirrors.neusoft.edu.cn/ubuntu-releases//quantal/ubuntu-12.10-desktop-i386.iso'])

###use twisted

http://twistedmatrix.com/documents/11.1.0/core/howto/process.html

http://jineshkj.wordpress.com/2006/12/22/how-to-capture-stdin-stdout-and-stderr-of-child-program/

http://unix.stackexchange.com/questions/58252/what-sets-a-childs-stderr-stdout-and-stdin

from twisted.internet import protocol
from twisted.internet import reactor
import sys

class MyPP(protocol.ProcessProtocol):
    def connectionMade(self):
        pass
    def outReceived(self, data):
        sys.stdout.write(data)
        sys.stdout.flush()
    def errReceived(self, data):
        sys.stderr.write(data)
        sys.stderr.flush()
    def inConnectionLost(self):
        pass
    def outConnectionLost(self):
        pass
    def errConnectionLost(self):
        pass
    def processExited(self, reason):
        pass
    def processEnded(self, reason):
        pass
        reactor.stop()

pp = MyPP()
reactor.spawnProcess(pp, "wget", 
                     ['wget','http://mirrors.neusoft.edu.cn/ubuntu-releases//quantal/ubuntu-12.10-desktop-i386.iso'],
                     usePTY = True)
reactor.run()

# after dig twisted source code, found using the default childFDs ({ 0: "w", 1: "r", 2: "r" }),
# the child stdout and stderr will be redirect to the same new pipe,
# wget will detect this, won't give progress bar behavior

# using { 0: 0, 1: 1, 2: 2}), child process inherits copies of the parent's set of open file descriptors.
# but outReceived/errReceived callback function will not get data

# setting usePTY to True, will redirect stdout and stderr to different pipe

wynemo/python get wget progress bar output.md

genzj commented Sep 12, 2014