Using pyCLI to create some commands and subcommands.

README.markdown

Hello world python subcommand pattern

Using pyCLI vs commando to create some commands and subcommands.

https://gist.github.com/1202975.git

git clone [email protected]:1202975.git python-hello-subcommand

How to write a well behaved console application? The ones with the best UIs allow configuration of core commands in a flexible config file, can override important defaults from the environment, and use the subcommand pattern to logically organize commands.

It should be tab-completable, with nice help output.

Circa 2013, there's still not a good way to do this. Here are a few attempts.

• https://gist.github.com/von/949337/

pycli

In terms of frameworks, this one is the only one with the needed pre-setup hook needed to do an initial pass over the arguments to sniff any parameters which might be used to then initialize the rest of the arguments. This happens, for example, when determining which configuration file to use. If we know there is a configuration file, why not make the configured values default?

Goal is to get a combination of tab completion + subcommands in such a way that will allow me to implement independent python modules as a main command, and then re-use the implementation as a subcommand in my main app.

simple

This attempt just uses argparse to iterate over a bunch of example subcommands. The behavior is pretty close to what we want; it's clear where to add reading from environment variables, and the set_parser method offers a nice way to set arguments, regardless of where the parser came form. argparse does not allow any way of re-using parser logic for subcommands.

At one point I tried separating the parsing into two phases, one for the core to find the subcommand, and then another to pass the remaining args to the subcommand as a main parser, but autocompletion failed to cope with this setup.

These examples on the other hand are tab completable subcommands with default configuration read from a config file.

$ ./simple
usage: simple [-h] [-d] [-c CONFIG] {qux,fux,bux,lux} ...

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug
  -c CONFIG, --config CONFIG

Title Group A:
  This is a long description on a simple command.

  {qux,fux,bux,lux}     some help on group A

$ ./simple 
bux       --config  --debug   -h        lux       
-c        -d        fux       --help    qux       
$ ./simple lux 
final pass Namespace(command='lux', config='example.conf', debug=False, foo='XXX', run=<bound method F.main of <__main__.F object at 0x2417d50>>) None
Hello, I'm lux.
['lux',
 'params',
 <__main__.F object at 0x2417d50>,
 Namespace(command='lux', config='example.conf', debug=False, foo='XXX', run=<bound method F.main of <__main__.F object at 0x2417d50>>)]

$ ./simple qux
final pass Namespace(command='qux', config='example.conf', debug=False, foo='baz', run=<bound method F.main of <__main__.F object at 0x1c632d0>>) None
Hello, I'm qux.
['qux',
 'params',
 <__main__.F object at 0x1c632d0>,
 Namespace(command='qux', config='example.conf', debug=False, foo='baz', run=<bound method F.main of <__main__.F object at 0x1c632d0>>)]

$ ./simple qux --help
usage: simple qux [-h] [--foo FOO]

long desc of qux

optional arguments:
  -h, --help  show this help message and exit
  --foo FOO

So the only thing missing is the default values from the help output.

add_config.py

import argparse
import ConfigParser
import os

"""
add_config.py - http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll
"""

def _identity(x):
    return x

_SENTINEL = object()

class AddConfigFile(argparse.Action):
    def __call__(self,parser,namespace,values,option_string=None):
        #I can never remember if `values` is a list all the time or if it
        #can be a scalar string ... -- This takes care of both.
        if isinstance(values,basestring):
            parser.config_files.append(values)
        else:
            parser.config_files.extend(values)


class ArgumentConfigEnvParser(argparse.ArgumentParser):
    def __init__(self,*args,**kwargs):
        """
        Added 2 new keyword arguments to the ArgumentParser constructor:

           config --> List of filenames to parse for config goodness
           default_section --> name of the default section in the config file
        """
        self.config_files = kwargs.pop('config',[])  #Must be a list
        self.default_section = kwargs.pop('default_section','MAIN')
        self._action_defaults = {}
        argparse.ArgumentParser.__init__(self,*args,**kwargs)


    def add_argument(self,*args,**kwargs):
        """
        works like ArgumentParser.add_argument except that we've added an action:

           config: add a config file to the parser

        and we've added the ability to specify the section to pull the data from in the
        config file via the `section` keyword.  Relies on the undocumented fact that
        `ArgumentParser.add_argument` actually returns the `Action` object that it creates.
        We need this to reliably get `dest`, although we could probably write a simple
        function to do this for us.
        """

        if 'action' in kwargs and kwargs['action'] == 'config':
            kwargs['action'] = AddConfigFile
            kwargs['default'] = argparse.SUPPRESS

        #argparse won't know what to do with the section, so we'll pop it out and add it back in
        #later.
        #Also, we have to prevent any type conversion by argparse.  We'll do it explicitly
        #in parse_known_args.  This way, we can reliably check whether argparse has replaced the default.
        section = kwargs.pop('section',self.default_section)
        type = kwargs.pop('type',_identity)
        default = kwargs.pop('default',_SENTINEL)

        kwargs.update(default=_SENTINEL if default is not argparse.SUPPRESS else argparse.SUPPRESS)

        action = argparse.ArgumentParser.add_argument(self,*args,**kwargs)

        kwargs.update(section=section,type=type,default=default)

        self._action_defaults[action.dest] = (args,kwargs)
        return action

    def parse_known_args(self,args=None, namespace=None):
        #parse_args calls parse_known_args, so we should be OK with this...
        ns, argv = argparse.ArgumentParser.parse_known_args(self,args=args,namespace=namespace)
        config_parser = ConfigParser.SafeConfigParser()
        config_files = [os.path.expanduser(os.path.expandvars(x)) for x in self.config_files]
        config_parser.read(config_files)


        for dest,(args,init_dict) in self._action_defaults.items():
            type_converter = init_dict['type']
            default = init_dict['default']

            obj = default
            if getattr(ns,dest,_SENTINEL) is not _SENTINEL: #found on command line
                obj = getattr(ns,dest)
            else: #not found on commandline
                try:  #get from config file
                    obj = config_parser.get(init_dict['section'],dest)
                except (ConfigParser.NoSectionError,ConfigParser.NoOptionError): #Nope, not in config file
                    try: #get from environment
                        obj = os.environ[dest.upper()]
                    except KeyError:
                        pass

            if obj is _SENTINEL:
                setattr(ns,dest,None)
            elif obj is argparse.SUPPRESS:
                pass
            else:
                setattr(ns,dest,type_converter(obj))

        return ns, argv


if __name__ == '__main__':
    fake_config = """
[MAIN]
foo:bar
bar:1
"""
    with open('_config.file','w') as fout:
        fout.write(fake_config)

    parser = ArgumentConfigEnvParser()
    parser.add_argument('--config-file',action='config',help="location of config file")
    parser.add_argument('--foo',type=str,action='store',default="grape",help="don't know what foo does ...")
    parser.add_argument('--bar',type=int,default=7,action='store',help="This is an integer (I hope)")
    parser.add_argument('--baz',type=float,action='store',help="This is an float(I hope)")
    parser.add_argument('--qux',type=int,default='6',action='store',help="this is another int")
    ns = parser.parse_args([])

    parser_defaults = {'foo':"grape",'bar':7,'baz':None,'qux':6}
    config_defaults = {'foo':'bar','bar':1}
    env_defaults = {"baz":3.14159}

    #This should be the defaults we gave the parser
    print ns
    assert ns.__dict__ == parser_defaults

    #This should be the defaults we gave the parser + config defaults
    d = parser_defaults.copy()
    d.update(config_defaults)
    ns = parser.parse_args(['--config-file','_config.file'])
    print ns
    assert ns.__dict__ == d

    os.environ['BAZ'] = "3.14159"

    #This should be the parser defaults + config defaults + env_defaults
    d = parser_defaults.copy()
    d.update(config_defaults)
    d.update(env_defaults)
    ns = parser.parse_args(['--config-file','_config.file'])
    print ns
    assert ns.__dict__ == d

    #This should be the parser defaults + config defaults + env_defaults + commandline
    commandline = {'foo':'3','qux':4} 
    d = parser_defaults.copy()
    d.update(config_defaults)
    d.update(env_defaults)
    d.update(commandline)
    ns = parser.parse_args(['--config-file','_config.file','--foo=3','--qux=4'])
    print ns
    assert ns.__dict__ == d


    os.remove('_config.file')

auto.sh

complete -F _optcomplete b.py

auto_c.sh

# Copyright 2012-2013, Andrey Kislyuk and argcomplete contributors.
# Licensed under the Apache License. See https://github.com/kislyuk/argcomplete for more info.

_python_argcomplete_global() {
    local ARGCOMPLETE=0
    if [[ "$1" == python* ]] || [[ "$1" == pypy* ]]; then
        if [[ -f "${COMP_WORDS[1]}" ]] && (head -c 1024 "${COMP_WORDS[1]}" | grep --quiet "PYTHON_ARGCOMPLETE_OK") >/dev/null 2>&1; then
            local ARGCOMPLETE=2
            set -- "${COMP_WORDS[1]}"
        fi
    elif (which "$1" && head -c 1024 $(which "$1") | grep --quiet "PYTHON_ARGCOMPLETE_OK") >/dev/null 2>&1; then
        local ARGCOMPLETE=1
    elif (which "$1" && head -c 1024 $(which "$1") | egrep --quiet "(EASY-INSTALL-SCRIPT|EASY-INSTALL-ENTRY-SCRIPT)" \
        && python-argcomplete-check-easy-install-script $(which "$1")) >/dev/null 2>&1; then
        local ARGCOMPLETE=1
    fi

    if [[ $ARGCOMPLETE == 1 ]] || [[ $ARGCOMPLETE == 2 ]]; then
        local IFS=$(echo -e '\v')
        COMPREPLY=( $(_ARGCOMPLETE_IFS="$IFS" \
            COMP_LINE="$COMP_LINE" \
            COMP_POINT="$COMP_POINT" \
            _ARGCOMPLETE=$ARGCOMPLETE \
            "$1" 8>&1 9>&2 1>/dev/null 2>&1) )
        if [[ $? != 0 ]]; then
            unset COMPREPLY
        fi
    fi
}
complete -o nospace -o default -D -F _python_argcomplete_global

b.py

#!/usr/bin/python
import sys
import commando
from pprint import pprint
import argcomplete # PYTHON_ARGCOMPLETE_OK

__version__ = '0.0.1-dev'

"""
class ParamGroup(commando.param):
  __params__ = ( ('-d', '--debug'), dict(action='store_true' ) )
  def __init__(self):
    params      = self.params( )
    args, kwds  = params[0], params[1]
    self.values = commando.application.values._make( (args, kwds) )

  def params(self):
    return self.__params__
"""

## base_val = commando.values
class TaggedMemo(object):
  # subcommand = base_val._make(((), dict()))
  commando   = True
  params     = [ ]
  def __init__(self, name=None):
    # default = base_val._make(((name, ), dict( )))
    self.subcommand = getattr(self.main, 'subcommand', default)
    self.params     = [ ]

  def __call__(self, app, params):
    self.main(params)

  def main(self, params):
    print "I'm an unsubclassed Command"
    pprint([self, params])

class DemoRecurse(TaggedMemo):
  @commando.subcommand('far', help='demo far recurse')
  def main(self, params):
    print "used decorator on a subclass"
    
class FooBar(commando.Application):
  @commando.subcommand('one', help="one")
  def one(self, params):
    print "one"

  @commando.subcommand('two', help="two")
  def two(self, params):
    print "two"

  #@commando.command(description="my command description")
  @commando.command('far', description='demo far recurse')
  def main(self,  params):
    print "used decorator on a simple class"

class Root(commando.Application):

  def print_usage(self):
    self.__parser__.print_help( )
  @commando.command(description="b - my command description")
  @commando.param('-v', '--version', action='version',
                  version='%(prog)s ' + __version__)
  @commando.param('-foo', dest='foo', action='store', type=str)
  def main(self, app, params):
    pprint([self, params])

  @commando.subcommand('init', help="main command")
  def init(self, params):
    self.main(self, params)

  @commando.subcommand('help', help="This help")
  def help(self, params):
    self.print_usage( )


  @commando.subcommand('foo', help="foo command")
  @commando.param('extra',  action='store', type=str, nargs="*")
  def foo(self, params):
    pprint([self, params])
    print "HAHA", params
    f = FooBar( )
    f.run(params.extra)


  def __init__(self):
    super(type(self), self).__init__( )
    # print self.__parser__
    argcomplete.autocomplete(self.__parser__)
    print "foo.parser"
    f = FooBar( )
    pprint(self.foo.parser)
    print "foo.parser.subcommand"
    #pprint(self.foo.subcommand)

# class Application(Root, commando.WeirdApp):
class Application(Root):
  # __subcommands__ = [ FooBar( ), DemoRecurse('demo') ]
  def __init__(self):
    super(Application, self).__init__()
  pass



def main( ):
  app = Root( )
  app.run( )
  
  #tree = commando.TreeApp(root=Root( ))
  # tree.run( )
if __name__ == '__main__':
  main( )

#####
# EOF

example.conf

[qux]
foo=baz

pycli

#!/usr/bin/python

from cli.log import LoggingApp
import argcomplete # PYTHON_ARGCOMPLETE_OK
from pprint import pprint

"""
Very rough.
"""

class Subcommand(object):
  name = ''
  def __init__(self, handler):
    self.session = handler 

  def options(self):
    return [ ]
  def setup(self, parser):
    self.parser = parser
    for args, kwds in self.options( ):
      parser.add_argument(*args, **kwds)
  def help(self):
    return self.__doc__

  def main(self, app):
    pprint([self, app])
    print self.name

class QuxApp(Subcommand):
  """Qux Does several special things"""
  name = "qux"
class FuxApp(Subcommand):
  """Fux is accidently different."""
  name = "fux"
class BuxApp(Subcommand):
  """Bux seems special, but it's a trick."""
  name = "bux"
class BaxApp(Subcommand):
  """Bax seems special, bat it's a trick."""
  name = "bax"
class FaxApp(Subcommand):
  """Fax is accidently different."""
  name = "fax"
class QaxApp(Subcommand):
  """Qax Does several special things"""
  name = "qax"

class Command(object):
  "Fake help"
  subcommands = { }
  def __init__(self, name, subcommands=None):
    self.subcommands = { }
    self.name  = name
    if subcommands is not None:
      for Flow in subcommands:
        self.addFlow(Flow)

  def __repr__(self):
    return "%s:%s" % (self.name, type(self))
  def subcommand_factory(self, Flow):
    return Flow(self)

  def addFlow(self, Flow):
    flow = self.subcommand_factory(Flow)
    self.subcommands[flow.name] = flow

  def base(self, params):
    pprint(self, params)
    print "Without a subcommand!!"

  def setup(self, parser):
    n = self.name
    self.parser   = parser
    pprint(['before', parser._positionals])
    self.commands = parser.add_subparsers(dest='subcommand',
                                          title="my commands title",
                                          help=self.help( ),
                                          prog="command")
    pprint(['after', parser._positionals])
    parser.set_defaults(run=self.base)
    #p = self.commands.add_parser(None, help='basic help')
    for flow in self.subcommands.values( ):
      #flow = Flow( )
      p = self.commands.add_parser(flow.name, help=flow.help())
      flow.setup(p)
 
  def help(self):
    return self.__doc__

  def main(self, app):
    subcommand = self.subcommands[app.params.subcommand]
    subcommand.main(self)
    
def get_commands():
  devices = [ ]
  a = Command('AAA', [ FuxApp, BuxApp ] )
  b = Command('BBB', [ FaxApp, QaxApp, ] )
  c = Command('CCC', [ FuxApp, BaxApp, QuxApp, BuxApp ] )
  return [ a, b, c ]

class GlobalOptions(object):
  def setup(self):
    print "setting up global options"
class Application(LoggingApp, GlobalOptions):
  """Test Hello World
  """
  name = "dispatcher"
  commands = { }

  def setup(self):
    # just after wrapping argument during __call__
    # !? or during __init__
    super(Application, self).setup( )
    GlobalOptions.setup(self)
    argcomplete.autocomplete(self.argparser)
    self.add_param("bar", help="fake option", action='store_true')
    self.subparsers = self.argparser.add_subparsers(dest='command', help='fake help on this command')

    for dev in get_commands():
      self.add_command( dev )

  def pre_run(self):
    # called just before main, updates params, parses args
    super(Application, self).pre_run()

  def add_command(self, command):
    self.commands[command.name] = command
    parser = self.subparsers.add_parser(command.name, help=command.help())
    #parser.set_defaults(run=self.main)
    command.setup(parser)

  def main(self):
    self.log.warn("hello world warn")
    self.log.debug("hello world debug")
    self.log.info("hello world info")
    self.log.error("hello world error")
    self.log.critical("hello world critical")
    self.log.fatal("hello world fatal")

    command  = self.commands[self.params.command]
    pprint(self.params)
    command.main(self)
    

if __name__ == '__main__':
  app = Application()
  app.run( )

#####
# EOF

simple

#!/usr/bin/python

"""
Simple app using argparse.

How can I provide a bunch of subcommands, but make one of them a default?

"""


from pprint import pprint, pformat
import argparse
import argcomplete # PYTHON_ARGCOMPLETE_OK
import logging
import ConfigParser

log = logging.getLogger(__name__)


class F(object):
  def __init__(self, name):
    self.name = name
    self.parser = argparse.ArgumentParser(prog=name, description=self.descr( ), add_help=True)
  def descr(self):
    desc = "A very long description of %s" % self.name
    return desc

  def set_parser(self, parser, options):
    self.parser = parser
    parser.add_argument('--foo',
                        type=str,
                        default=options.get('foo', 'XXX'))
    parser.set_defaults(**options)

  def parse_args(self, argv):
    return self.parser.parse_args(argv)
  def main(self, app, params):
    print "Hello, I'm %s." % self.name
    pprint([ self.name, 'params', self, params ])

class Simple(object):
  """

  This is a very simple description.
  We'll simulate it very long here.
  Very long indeed.
  """
  def global_parser(self):
    desc = type(self).__doc__
    parser = argparse.ArgumentParser(description=desc, add_help=False,
                      formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    parser.add_argument('-d', '--debug', action='store_true')
    parser.add_argument('-c', '--config', default="example.conf", type=str)
    return parser

  def __init__(self):
    desc = type(self).__doc__
    conf_parser = self.global_parser( )
    self.pre_parser = argparse.ArgumentParser(add_help=False, parents=[conf_parser])

    # first sniff for basic debuggery, help fill in defaults
    self.known, args = self.pre_parser.parse_known_args( )
    # then set up the real parser, cloning the initial one
    self.parser = argparse.ArgumentParser(parents=[conf_parser], add_help=True,
                           formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    self.read_config(self.known.config)
    self.commands = { }
    self.setup_a(self.parser)
    if not args:
      self.parser.print_help( )

    argcomplete.autocomplete(self.parser)

  def read_config(self, path):
    self.config = config = getattr(self, 'config', ConfigParser.ConfigParser( ))
    config.read(path)
    return config

  def setup_a(self, parser, name='lalala'):
    desc = """This is a long description on a simple command."""
    subparsers = parser.add_subparsers(title='Title Group A',
                                       dest='command',
                                       # default='qux',
                                       description = desc,
                                       help='some help on group A')
    config = self.config
    choices = [ ]
    for x in [ 'qux', 'fux', 'bux', 'lux' ]:
      f = F(x)
      kwds = dict( description = "long desc of %s" % x,
                 )
                 #  default= 'qux')
      p = subparsers.add_parser(f.name, **kwds)
      cfg = config.has_section(f.name) and dict(config.items(f.name)) or { }
      f.set_parser(p, cfg)
      self.commands[f.name] = f
      p.set_defaults(run=f.main)
      choices.append(f.name)
    self.pre_parser.add_argument('command', type=str, choices=choices, default='help')


  def parse(self, argv):
    return self.parser.parse_args(argv)

  def run(self, args=None):
    remain = None
    if args is None:
      import sys
      args = self.parse(sys.argv[1:])
      print 'final pass', pformat(args), pformat(remain)
    if hasattr(args, 'run'):
      args.run(self, args)
    else:
      pass
      # self.main(self.tool, args)

  def main(self, app, params):
    pass
    # self.tool.main(self, params)

if __name__ == '__main__':
  app = Simple()
  app.run( )

#####
# EOF

test_opt_subcommand.py

#!/usr/bin/python

import unittest
import argparse 
from pprint import pprint

class F(object):
  def __init__(self, name):
    self.name = name
  def main(self, app, params):
    pprint([ self.name, self, params ])
    print "Hello, I'm %s." % self.name
    
def get_simple_parser( ):
  parser = argparse.ArgumentParser()
  # flesh it out a bit with a no-op option
  parser.add_argument('-d', '--debug', action='store_true')
  return parser

class TestSubparserDefaultFun(unittest.TestCase):
  def setUp(self):
    self.parser = get_simple_parser( )
    setup_a(self.parser)

  def parse_missing_subcommand_args(self):
    pprint(parser.parse_args(prep('')))

  def donttest_required_subcommand_behavior(self):
    self.parser.parse_args( )

  def test_optional_subcommand_behavior(self):
    self.parser.parse_args(prep('  '))


def setup_a(parser, name='lalala'):
  desc = """This is a long description on a simple command."""
  # goal, make qux the default
  # default = 'qux'
  kwds = dict( title='Title Group A',
               dest='command',
               description = desc,
               prog= "my program",
               # XXX: PATCH
               default = 'qux',
               help='some help on group A')
  subparsers = parser.add_subparsers(**kwds)
  for x in [ 'qux', 'fux', 'bux', 'lux' ]:
    f = F(x)
    kwds = dict( description = "long desc of %s" % x,
               )
               #  default= 'qux')
    p = subparsers.add_parser(x, **kwds)
    p.set_defaults(run=f.main)

def prep(args):
  return args.split( )

def main( ):
  parser = get_simple_parser( )
  #sub = parser.add_subparsers()
  #sub.add_parser("info")
  #parser.add_argument("paths", "+")
  #pprint(parser.parse_args(["foo", "bar"]))

  # add some subcommands
  setup_a(parser)

  #
  pprint(parser.parse_args(prep('')))
  pprint(parser.parse_args(prep('')))
  

if __name__ == '__main__':
  unittest.main( )

#####
# EOF

@n-west on completion:

• http://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html
• http://tldp.org/LDP/abs/html/tabexpansion.html
• http://fahdshariff.blogspot.com/2011/04/writing-your-own-bash-completion.html