Skip to content

Instantly share code, notes, and snippets.

@Jawabiscuit

Jawabiscuit/Python 3(->2) Metaprogramming

Last active August 29, 2015 14:04
Show Gist options
  • Save Jawabiscuit/8ee9c748384af4ade5c7 to your computer and use it in GitHub Desktop.
Save Jawabiscuit/8ee9c748384af4ade5c7 to your computer and use it in GitHub Desktop.
Download ZIP
PyCon 2013 discussion by David Beazley techniques converted to work with Python 2. There are only a few idioms that Python 3 uses that David goes over. Nothing that you can't do in 2 really.
Raw
Python 3(->2) Metaprogramming
{
"metadata": {
"name": "python3-metaprogramming"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#Python 3(->2) Metaprogramming"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[PyCon 2013](http://pyvideo.org/video/1716/python-3-metaprogramming) discussion by [David Beazley](http://pyvideo.org/speaker/125/david-beazley) techniques **converted** to work with Python 2. There are only a few idioms that Python 3 uses that David goes over. Nothing that you can't do in 2 really.\n",
"\n",
"##Summary\n",
"\n",
"Some of the most significant changes in Python 3 are related to metaprogramming. In this tutorial, I'll cover decorators, class decorators, descriptors, and metaclasses. However, the focus will be on idioms and examples that are only made possible using features that are unique to Python 3. For instance, making free use of function annotations, signatures, new metaclass features and more."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Debugging"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Print statement is the best method. Problem: get repetitive very quickly. Solution: decorator"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from __future__ import print_function\n",
"\n",
"def add(c, y):\n",
" return x + y\n",
"\n",
"def sub(x, y):\n",
" return x - y\n",
"\n",
"def mul(x, y):\n",
" return x * y\n",
"\n",
"def div(x, y):\n",
" return x / y\n"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 3
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"decorator: func that takes a func as input"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def debug(func):\n",
" # func is function to be wrapped\n",
" def wrapper(*args, **kwargs):\n",
" print(func.__name__)\n",
" return func(*args, **kwargs)\n",
" return wrapper\n",
"\n",
"@debug\n",
"def add(x, y):\n",
" return x + y"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 8
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"add(2, 3)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"add\n"
]
},
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 12,
"text": [
"5"
]
}
],
"prompt_number": 12
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Technicalities: they lose a lot of their information, like help."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"help(add)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Help on function wrapper in module __main__:\n",
"\n",
"wrapper(*args, **kwargs)\n",
" # func is function to be wrapped\n",
"\n"
]
}
],
"prompt_number": 14
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"what is this \"wrapper\" all about?"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from functools import wraps\n",
"\n",
"def debug(func):\n",
" # func is function to be wrapped\n",
" @wraps(func)\n",
" def wrapper(*args, **kwargs):\n",
" print(func.__name__)\n",
" return func(*args, **kwargs)\n",
" return wrapper\n",
" "
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 15
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What does functools.wraps do? It copies metadata from one func to another\n",
"\n",
"##Big Picture\n",
"\n",
"- Debugging code is isolated to single location\n",
"- This makes it easy to change (or to disable)\n",
"- No need to worry about the decorator implementation, you just sprinkle it where needed."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from functools import wraps\n",
"import logging\n",
"\n",
"logging.basicConfig(format='%(message)s')\n",
"\n",
"\n",
"def debug(func):\n",
" log = logging.getLogger(func.__module__)\n",
" log.setLevel(logging.DEBUG)\n",
" msg = func.__name__\n",
" # func is function to be wrapped\n",
" @wraps(func)\n",
" def wrapper(*args, **kwargs):\n",
" log.debug(msg)\n",
" return func(*args, **kwargs)\n",
" return wrapper\n",
"\n",
"@debug\n",
"def mul(x, y):\n",
" return x * y"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 42
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"mul(3, 2)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stderr",
"text": [
"DEBUG:__main__:mul\n"
]
},
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 44,
"text": [
"6"
]
}
],
"prompt_number": 44
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Key idea\n",
"\n",
"Can change decorator independently of code that uses it, maybe something like using environment variables."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from functools import wraps\n",
"import os\n",
"\n",
"\n",
"def debug(func):\n",
" if 'DEBUG' not in os.environ:\n",
" return func\n",
" msg = func.__name__\n",
" # func is function to be wrapped\n",
" @wraps(func)\n",
" def wrapper(*args, **kwargs):\n",
" print(func.__name__)\n",
" return func(*args, **kwargs)\n",
" return wrapper\n",
"\n",
"@debug\n",
"def div(x, y):\n",
" return x / y"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 60
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"os.getenv('DEBUG')"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 46
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"div(12, 4)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 49,
"text": [
"3"
]
}
],
"prompt_number": 49
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"os.environ['DEBUG'] = 'True'\n",
"print(os.getenv('DEBUG'))"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"True\n"
]
}
],
"prompt_number": 56
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"div(12, 4)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"div\n"
]
},
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 62,
"text": [
"3"
]
}
],
"prompt_number": 62
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Key idea: You're not doing it right if you aren't using a prefix! You know, for grepping!?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Calling convention\n",
"\n",
" @decorator(args)\n",
" def func():\n",
" pass\n",
"\n",
"Gets evaluated as\n",
"\n",
" func = decorator(args)(func)\n",
"\n",
"Essentially 2 levels of nested functions\n",
"\n",
" def debug(prefix=''):\n",
" def decorate(func):\n",
" msg = prefix + func.__name__\n",
" @wraps(func)\n",
" def wrapper(*args, **kwargs):\n",
" print(msg)\n",
" return func(*args, **kwargs)\n",
" return wrapper\n",
" return decorate\n",
"\n",
"Usage\n",
"\n",
" @debug(prefix='***')\n",
" def add(x, y):\n",
" return x + y\n",
"\n",
"Outer function provides \"environment\" for the inner function. In this case the prefix variable is just available to all inside the decorator.\n",
"\n",
"Interesting reformulation\n",
"\n",
" from functools import wraps, partial\n",
" \n",
" def debug(func=None, *, prefix=''):\n",
" if func is None:\n",
" # wasn't passed, call debug func again\n",
" # return a callable to invoke again\n",
" return partial(debug, prefix=prefix)\n",
" \n",
" msg = prefix + func.__name__\n",
" @wraps(func)\n",
" def wrapper(*args, **kwargs):\n",
" print(msg)\n",
" return func(*args, **kwargs)\n",
" return wrapper"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from functools import wraps, partial\n",
" \n",
"def debug(func=None, prefix=''):\n",
" if func is None:\n",
" # wasn't passed, call debug func again\n",
" # return a callable to invoke\n",
" return partial(debug, prefix=prefix)\n",
" \n",
" msg = prefix + func.__name__\n",
" @wraps(func)\n",
" def wrapper(*args, **kwargs):\n",
" print(msg)\n",
" return func(*args, **kwargs)\n",
" return wrapper\n",
"\n",
"@debug\n",
"def mod(x, y):\n",
" return x % y\n",
"\n",
"mod(1024, 8)\n",
"\n",
"@debug(prefix='***')\n",
"def sub(x, y):\n",
" return x - y\n",
"\n",
"sub(7, 3)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"mod\n",
"***sub\n"
]
},
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 70,
"text": [
"4"
]
}
],
"prompt_number": 70
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Class Decorator\n",
"\n",
"Example use\n",
" \n",
" @debugmethods\n",
" class Spam:\n",
" def grok(self):\n",
" pass\n",
" def bar(self):\n",
" pass\n",
" def foo(self):\n",
" pass\n",
"\n",
"- One decorator application\n",
"- Covers all defs in the class\n",
"- mostly works...\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def debugmethods(cls):\n",
" for k, v in vars(cls).items():\n",
" if callable(v):\n",
" setattr(cls, k, debug(v))\n",
" return cls\n",
"\n",
"@debugmethods\n",
"class Spam:\n",
" def a(self):\n",
" pass\n",
" def b(self):\n",
" pass\n",
" def c(self):\n",
" pass\n"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 71
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"s = Spam()\n",
"s.a()\n",
"s.b()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"a\n",
"b\n"
]
}
],
"prompt_number": 77
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"help(vars)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Help on built-in function vars in module __builtin__:\n",
"\n",
"vars(...)\n",
" vars([object]) -> dictionary\n",
" \n",
" Without arguments, equivalent to locals().\n",
" With an argument, equivalent to object.__dict__.\n",
"\n"
]
}
],
"prompt_number": 76
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Limitations\n",
"\n",
" @debugmethods\n",
" class BrokenSpam:\n",
" @classmethod\n",
" def grok(cls): # NOT wrapped\n",
" pass\n",
" @staticmethod\n",
" def bar(): # NOT wrapped\n",
" pass\n",
"\n",
"- Only instance methods get wrapped\n",
"- Why? An exercise for the reader...\n",
"\n",
"##Variation: Debug Access\n",
"\n",
" def debugattr(cls):\n",
" orig_getattribute = cls.__getattribute__\n",
" \n",
" def __getattribute__(self, name):\n",
" print('Get:', name)\n",
" return orig_getattribute(self, name)\n",
" \n",
" cls.__getattribute__ = __getattribute__\n",
" return cls\n",
"\n",
"- Rewriting part of the class itself\n",
"- Taking the orig `__getattribute__` spec method and adding something on it and returning it back"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def debugattr(cls):\n",
" orig_getattribute = cls.__getattribute__\n",
"\n",
" def __getattribute__(self, name):\n",
" print('Get:', name)\n",
" return orig_getattribute(self, name)\n",
"\n",
" cls.__getattribute__ = __getattribute__\n",
" return cls\n",
"\n",
"\n",
"@debugattr\n",
"class Point(object):\n",
" def __init__(self, x, y):\n",
" self.x = x\n",
" self.y = y\n",
"\n",
"p = Point(2, 3)\n",
"p.x\n",
"p.y"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Get: x\n",
"Get: y\n"
]
},
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 81,
"text": [
"3"
]
}
],
"prompt_number": 81
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Debug all the classes\n",
"\n",
" @debugmethods\n",
" class Base:\n",
" pass\n",
" \n",
" @debugmethods\n",
" class Spam(Base):\n",
" pass\n",
" \n",
" @debugmethods\n",
" class Grok(Spam):\n",
" pass\n",
" \n",
" @debugmethods\n",
" class Monkey(Grok):\n",
" pass\n",
"\n",
"- Many classes with debugging\n",
"- Once again repitition!\n",
"\n",
"##Solution: A Metaclass\n",
"\n",
" class debugmeta(type):\n",
" def __new__(cls, clsname, bases, clsdict):\n",
" clsobj = super(debugmeta, cls).__new__(cls, clsname,\n",
" bases, clsdict)\n",
" clsobj = debugmethods(clsobj)\n",
" return clsobj\n",
"\n",
"##Usage\n",
"\n",
" class Base:\n",
" __metaclass__ = debugmeta\n",
" \n",
" class Spam(Base):\n",
" pass"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class debugmeta(type):\n",
" def __new__(cls, clsname, bases, clsdict):\n",
" clsobj = super(debugmeta, cls).__new__(cls, clsname,\n",
" bases, clsdict)\n",
" clsobj = debugmethods(clsobj)\n",
" return clsobj\n",
"\n",
"class Base:\n",
" __metaclass__ = debugmeta\n",
" def a(self):\n",
" pass\n",
" def b(self):\n",
" pass\n",
" def c(self):\n",
" pass\n",
"\n",
"class Spam(Base):\n",
" def a(self):\n",
" pass\n",
" def b(self):\n",
" pass\n",
" def c(self):\n",
" pass"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 83
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"b = Base()\n",
"b.a()\n",
"s = Spam()\n",
"s.c()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"a\n",
"c\n"
]
}
],
"prompt_number": 84
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Types and Classes\n",
"Classes define new types\n",
"\n",
" class Spam:\n",
" pass\n",
"\n",
" s = Spam()\n",
" type(s)\n",
" '''<class '__main__.Spam'>'''\n",
" \n",
" type(int)\n",
" '''<class 'type'>'''\n",
" \n",
" type(str)\n",
" '''<class 'type'>'''\n",
" \n",
" type(Spam)\n",
" '''<class 'type'>'''\n",
" \n",
"- The class is the type of instance created\n",
"- The class is a callable that creates instances\n",
"- Types are their own class (builtin)\n",
"- This class creates new instances of \"type\" objects\n",
"- Used when defining a class\n",
"\n",
"##Class Definition Process\n",
"What happens during class definition (execution model)?\n",
"\n",
" class Spam(Base):\n",
" def __init__(self, name):\n",
" self.name = name\n",
" def bar(self):\n",
" print \"I'm Spam.bar\"\n",
"\n",
"Step 1: Body of class is isolated:\n",
" \n",
" body = '''\n",
" def __init__(self, name):\n",
" self.name = name\n",
" def bar(self):\n",
" print \"I'm Spam.bar\"\n",
" '''\n",
"Step 2: The class dictionary is created (`__prepare__` is Python3)\n",
"\n",
" clsdict = type.__prepare__('Spam', (Base,))\n",
"\n",
"This dictionary serves as the local namespace for statements in the class body.\n",
"By default, it's a simple dictionary.\n",
"\n",
"Step 3: Body is executed in returned dict\n",
"\n",
" exec(body, globals(), clsdict)\n",
" \n",
"What exec actually does is populate the clsdict\n",
"\n",
"`>>> clsdict`\n",
"`{'__init__': <function __init__ at 0x4da10>, 'bar': <function bar at 0x4dd70>}`\n",
"\n",
"Step 4: Class is constructed from its name, base classes, and the dictionary\n",
"\n",
" Spam = type('Spam', (Base,), clsdict)\n",
" \n",
"##Using a Metaclass\n",
"\n",
"- Metaclasses get information about class definitions at the time of definition\n",
" - can inspect\n",
" - can modify\n",
"- Similar to a class decorator\n",
"- Q: Why wouldn't you just use a class decorator?\n",
"\n",
"##Inheritance\n",
"\n",
"- Metaclassed **propagate** down heirarchies\n",
"- Think of it as genetic mutation\n",
"\n",
"Example\n",
"\n",
" class Base:\n",
" __metaclass__ = mytype\n",
" \n",
" class Spam(Base): # metaclass=mytype\n",
" ...\n",
" \n",
" class Grok(Spam): # metaclass=mytype\n",
" ...\n",
"\n",
"##Big Picture\n",
"\n",
"- It's mostly about wrapping/rewriting\n",
" - Decorators: functions\n",
" - Class Decorators: Classes\n",
" - Metaclasses: Class hierarchies\n",
"- You have the power to change things\n",
"- Class decorators: do something to a class after it has been fully formed\n",
"- Metaclasses: do something to the class before it has been created"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Problem: Structures"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Stock:\n",
" def __init__(self, name, shares, price):\n",
" self.name = name\n",
" self.shares = shares\n",
" self.price = price\n",
" \n",
"class Point:\n",
" def __init__(self, x, y):\n",
" self.x = x\n",
" self.y = y\n",
"\n",
"class Host:\n",
" def __init__(self, address, port):\n",
" self.address = address\n",
" self.port = port"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 2
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Structure:\n",
" _fields = []\n",
" def __init__(self, *args):\n",
" for attr, val in zip(self.__class__._fields, args):\n",
" setattr(self, attr, val)\n",
"\n",
"class Stock(Structure):\n",
" _fields = ['name', 'shares', 'price']\n",
" \n",
"class Point(Structure):\n",
" _fields = ['x', 'y']\n",
"\n",
"class Host(Structure):\n",
" _fields = ['hostname', 'port']"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 3
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"s = Stock('GOOG', 100, 490.1)\n",
"print(s.name)\n",
"print(s.price)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"GOOG\n",
"490.1\n"
]
}
],
"prompt_number": 100
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Problems Introduced\n",
"- No support for keyword args\n",
"- Missing calling signatures\n",
"- Generalized `__init__`\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Bit about inspect, signatures, parameters, and binding here..."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from __future__ import print_function\n",
"import inspect\n",
"#inspect.signature # Python3\n",
"#inspect.Parameter # Python3\n",
"\n",
"\n",
"def add(c, y):\n",
" return x + y\n",
"\n",
"def sub(x, y):\n",
" return x - y\n",
"\n",
"def mul(x, y):\n",
" return x * y\n",
"\n",
"def div(x, y):\n",
" return x / y\n",
"\n",
"print('add is function?', inspect.isfunction(add))\n",
"print('Stock.__init__ is a function?', inspect.isfunction(Stock.__init__))\n",
"print('Stock.__init__ is a method?', inspect.ismethod(Stock.__init__))\n",
"print('signature of Stock.__init__:', inspect.getargspec(Stock.__init__))\n",
"print('Structure.__init__ is a method?', inspect.ismethod(Structure.__init__))\n",
"print('signature of Structure.__init__:', inspect.getargspec(Structure.__init__))\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"add is function? True\n",
"Stock.__init__ is a function? False\n",
"Stock.__init__ is a method? True\n",
"signature of Stock.__init__: ArgSpec(args=['self'], varargs='args', keywords=None, defaults=None)\n",
"Structure.__init__ is a method? True\n",
"signature of Structure.__init__: ArgSpec(args=['self'], varargs='args', keywords=None, defaults=None)\n"
]
}
],
"prompt_number": 7
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"###Advice\n",
"- Use a class decorator if the goal is to tweak classes that might be unrelated\n",
"- Use a metaclass if you're trying to perform some modification in combination with inheritance\n",
"- Don't be so quick to dismiss techniques (e.g., 'metaclasses suck ... blah blah')\n",
"- All of the tools are meant to work together"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#Owning the dot"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Start with a simple descriptor and build some inheritance structure."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from __future__ import print_function\n",
"\n",
"\n",
"from weakref import WeakKeyDictionary\n",
"\n",
"\n",
"class Descriptor(object):\n",
" '''\n",
" Descriptor that checks with user-supplied check function if an\n",
" attribute is valid.\n",
" '''\n",
" def __init__(self, default):\n",
" # inst var\n",
" self.default = default\n",
" self.data = WeakKeyDictionary()\n",
" \n",
" def __get__(self, instance, owner):\n",
" ''' spec method '''\n",
" return self.data.get(instance, self.default)\n",
" \n",
" def __set__(self, instance, value):\n",
" self.data[instance] = value\n",
"\n",
"\n",
"class Typed(Descriptor):\n",
" # class var\n",
" typ = object # expected type\n",
" def __set__(self, instance, value):\n",
" if not isinstance(value, self.__class__.typ):\n",
" raise TypeError('Unexpected type: %s' % type(value))\n",
" super(Typed, self).__set__(instance, value)\n",
"\n",
"\n",
"class Integer(Typed):\n",
" typ = int\n",
"\n",
"\n",
"class String(Typed):\n",
" typ = str\n",
"\n",
"\n",
"class Float(Typed):\n",
" typ = float\n",
"\n",
"\n",
"class Positive(Descriptor):\n",
" def __set__(self, instance, value):\n",
" if value < 0:\n",
" raise ValueError('Value must be > 0')\n",
" super(Positive, self).__set__(instance, value)\n",
"\n",
"\n",
"class PositiveInteger(Integer, Positive):\n",
" pass\n",
"\n",
"\n",
"class PositiveFloat(Float, Positive):\n",
" pass\n",
"\n",
"\n",
"print(PositiveFloat.__mro__)\n",
"\n",
"\n",
"if __name__ == '__main__':\n",
" pass\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"(<class '__main__.PositiveFloat'>, <class '__main__.Float'>, <class '__main__.Typed'>, <class '__main__.Positive'>, <class '__main__.Descriptor'>, <type 'object'>)\n"
]
}
],
"prompt_number": 26
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice the way multiple resolution order works. It the method lookup order isn't purely from child to parent in a linear fashion. It's much more in a tree-like fashion. Order of inherited objects *matters* in the class signature.\n",
"\n",
"Below illustrates type and value checking is operational."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Restricted(object):\n",
" '''Use checked attributes'''\n",
" attr1 = PositiveInteger(10)\n",
" attr2 = PositiveFloat(12.5)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 33
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"r = Restricted()\n",
"print('attr1', r.attr1)\n",
"r.attr1 = 100\n",
"print('attr1', r.attr1)\n",
"\n",
"try:\n",
" val = 200.12\n",
" r.attr1 = val\n",
"except TypeError:\n",
" print(val, 'Value must be int')\n",
"\n",
"r.attr1 = 200\n",
"print('attr1', r.attr1)\n",
"\n",
"try:\n",
" val = -200\n",
" r.attr1 = val\n",
"except ValueError:\n",
" print(val, 'Value must be positive')\n",
"\n",
"r.attr1 = 300\n",
"print('attr1', r.attr1)\n",
"\n",
"try:\n",
" r.attr2 = -200.1\n",
"except ValueError, e:\n",
" print(e)\n",
"except TypeError, e:\n",
" print(e)\n",
"\n",
"r.attr2 = 200.1\n",
"print('attr2', r.attr2)\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"attr1 10\n",
"attr1 100\n",
"200.12 Value must be int\n",
"attr1 200\n",
"-200 Value must be positive\n",
"attr1 300\n",
"Value must be > 0\n",
"attr2 200.1\n"
]
}
],
"prompt_number": 34
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In Python 2 there isn't such thing as an optional kwarg. [This stackoverflow question](http://stackoverflow.com/questions/13687043/python-optional-positional-and-keyword-arguments) illustrates more. The option will then need to be manually looked up in kwargs dict and pulled out. Below illustrates the idea."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"d = dict(key1='foo', key2='bar', key3='spam', maxlen=10)\n",
"print('before remove:', d)\n",
"if 'maxlen' in d:\n",
" d.pop('maxlen')\n",
"print('after move', d)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"before remove: {'key3': 'spam', 'maxlen': 10, 'key1': 'foo', 'key2': 'bar'}\n",
"after move {'key3': 'spam', 'key1': 'foo', 'key2': 'bar'}\n"
]
}
],
"prompt_number": 5
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using inheritance to create more value checking descriptors that take options using the `__init__` function. The keyword option this Descriptor cares about is extracted and the options dict passed along."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Sized(Descriptor):\n",
" def __init__(self, *args, **kwargs):\n",
" self.maxlen = kwargs.get('maxlen', 5)\n",
" if 'maxlen' in kwargs:\n",
" kwargs.pop('maxlen')\n",
" super(Sized, self).__init__(*args, **kwargs)\n",
" \n",
" def __set__(self, instance, value):\n",
" if len(value) > self.maxlen:\n",
" raise ValueError('Too big')\n",
" super(Sized, self).__set__(instance, value)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 27
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The Sized descriptor being passed a default name and optional parameter."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Restricted(object):\n",
" '''Use checked attributes'''\n",
" attr1 = PositiveInteger(10)\n",
" attr2 = PositiveFloat(12.5)\n",
" attr3 = Sized('', maxlen=10) "
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 28
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"r = Restricted()\n",
"\n",
"print(r.attr3)\n",
"try:\n",
" val = 'elephantitis'\n",
" r.attr3 = val\n",
"except ValueError, e:\n",
" print(val, str(e))\n",
" r.attr3 = 'unicorn'\n",
"\n",
"print(r.attr3, 'nice')"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\n",
"elephantitis Too big\n",
"unicorn nice\n"
]
}
],
"prompt_number": 29
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A SizedString descriptor using multiple inheritance being can be used to check type and value. The optional parameter and mro enable descriptor classes to all work in concert with one another."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class SizedString(String, Sized):\n",
" pass\n",
"\n",
"class Restricted(object):\n",
" '''Use checked attributes'''\n",
" attr1 = PositiveInteger(10)\n",
" attr2 = PositiveFloat(12.5)\n",
" attr3 = SizedString('', maxlen=10) "
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 30
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"r = Restricted()\n",
"\n",
"print(r.attr3)\n",
"try:\n",
" val = 'elephantitis'\n",
" r.attr3 = val\n",
"except ValueError, e:\n",
" print(val, str(e))\n",
"\n",
"try:\n",
" val = 10000\n",
" r.attr3 = val\n",
"except TypeError, e:\n",
" print(val, str(e))\n",
"\n",
"r.attr3 = 'unicorn'\n",
"print(r.attr3, 'Is a good value')"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\n",
"elephantitis Too big\n",
"10000 Unexpected type: <type 'int'>\n",
"unicorn nice\n"
]
}
],
"prompt_number": 32
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Value checking wouldn't be complete without regex!"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import re\n",
"\n",
"# Pattern matching\n",
"class Regex(Descriptor):\n",
" def __init__(self, *args, **kwargs):\n",
" self.pat = re.compile(kwargs.get('pat', ''))\n",
" if 'pat' in kwargs:\n",
" kwargs.pop('pat')\n",
" super(Regex, self).__init__(*args, **kwargs)\n",
" \n",
" def __set__(self, instance, value):\n",
" if not self.pat.match(value):\n",
" raise ValueError('Invalid string')\n",
" super(Regex, self).__set__(instance, value)\n",
"\n",
"\n",
"class SizedRegex(SizedString, Regex):\n",
" pass"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 62
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Restricted(object):\n",
" '''Use checked attributes'''\n",
" attr1 = PositiveInteger(10)\n",
" attr2 = PositiveFloat(12.5)\n",
" attr3 = SizedRegex('DEFAULT', pat='^[A-Za-z]+$', maxlen=10)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 63
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"r = Restricted()\n",
"\n",
"print(r.attr3, 'is the default')\n",
"\n",
"try:\n",
" val = 'unicorn'\n",
" r.attr3 = val\n",
"except ValueError, e:\n",
" print(val, str(e))\n",
"else:\n",
" print(r.attr3, 'is valid')\n",
"\n",
"try:\n",
" val = 'UNICORN'\n",
" r.attr3 = val\n",
"except ValueError, e:\n",
" print(val, str(e))\n",
"else:\n",
" print(val, 'is valid')\n",
"\n",
"# The default regex doesn't allow numeric characters\n",
"try:\n",
" val = 'un1c0rn'\n",
" r.attr3 = val\n",
"except ValueError, e:\n",
" print(val, str(e))\n",
"\n",
"try:\n",
" val = 'elephantitis'\n",
" r.attr3 = val\n",
"except ValueError, e:\n",
" print(val, str(e))\n",
"else:\n",
" print(r.attr3, 'is valid')\n",
"\n",
"# Again, the attr must be a string type\n",
"try:\n",
" val = 100\n",
" r.attr3 = val\n",
"except TypeError, e:\n",
" print(val, str(e))\n",
"else:\n",
" print(r.attr3, 'is valid')"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"DEFAULT is the default\n",
"unicorn is valid\n",
"UNICORN is valid\n",
"un1c0rn Invalid string\n",
"elephantitis Too big\n",
"100 Unexpected type: <type 'int'>\n"
]
}
],
"prompt_number": 68
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"###Still, there's a problem with repetition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Consider:\n",
" \n",
" class Regex(Descriptor):\n",
" def __init__(self, *args, **kwargs):\n",
" self.pat = re.compile(kwargs.get('pat', ''))\n",
" if 'pat' in kwargs:\n",
" kwargs.pop('pat')"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 2
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ordered dicts\n",
"\n",
"Todo: Look for alternate method in Python 2.6"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"###Problem: performance hit!\n",
"\n",
"Solution? It gets hairy: on-the-fly source code creation, eval, and in the end xml and hacking import. Fun science project but not for production!"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"#This makes everything pretty\n",
"\n",
"from IPython.core.display import HTML\n",
"from urllib import urlopen\n",
"def css_styling():\n",
" styles = open('styles/custom.css', 'r').read()\n",
" return HTML(styles)\n",
"css_styling()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"html": [
"<style>\n",
" @font-face {\n",
" font-family: \"Computer Modern\";\n",
" src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');\n",
" }\n",
" @font-face {\n",
" font-family: \"Computer Modern\";\n",
" font-weight: bold;\n",
" src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');\n",
" }\n",
" @font-face {\n",
" font-family: \"Computer Modern\";\n",
" font-style: oblique;\n",
" src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');\n",
" }\n",
" @font-face {\n",
" font-family: \"Computer Modern\";\n",
" font-weight: bold;\n",
" font-style: oblique;\n",
" src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');\n",
" }\n",
" div.cell{\n",
" width:800px;\n",
" margin-left:16% !important;\n",
" margin-right:auto;\n",
" }\n",
" h1 {\n",
" font-family: Helvetica, serif;\n",
" }\n",
" h4{\n",
" margin-top:12px;\n",
" margin-bottom: 3px;\n",
" }\n",
" div.text_cell_render{\n",
" font-family: Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
" line-height: 145%;\n",
" font-size: 130%;\n",
" width:800px;\n",
" margin-left:auto;\n",
" margin-right:auto;\n",
" }\n",
" .CodeMirror{\n",
" font-family: \"Source Code Pro\", source-code-pro,Consolas, monospace;\n",
" }\n",
" .prompt{\n",
" display: None;\n",
" }\n",
" .text_cell_render h5 {\n",
" font-weight: 300;\n",
" font-size: 22pt;\n",
" color: #4057A1;\n",
" font-style: italic;\n",
" margin-bottom: .5em;\n",
" margin-top: 0.5em;\n",
" display: block;\n",
" }\n",
" \n",
" .warning{\n",
" color: rgb( 240, 20, 20 )\n",
" } \n",
"</style>\n",
"<script>\n",
" MathJax.Hub.Config({\n",
" TeX: {\n",
" extensions: [\"AMSmath.js\"]\n",
" },\n",
" tex2jax: {\n",
" inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
" displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
" },\n",
" displayAlign: 'center', // Change this to 'center' to center equations.\n",
" \"HTML-CSS\": {\n",
" styles: {'.MathJax_Display': {\"margin\": 4}}\n",
" }\n",
" });\n",
"</script>\n"
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 69,
"text": [
"<IPython.core.display.HTML at 0x4f582b0>"
]
}
],
"prompt_number": 69
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment