Bitey Magic, an IPython extension

.gitignore

*.py[oc]

Bitey Magic.ipynb

{
 "metadata": {
  "name": "Bitey Magic"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Bitey Magic\n",
      "===========\n",
      "\n",
      "[Bitey] is a tool to import LLVM bitcode directly into Python.  Bitey magic is a small [IPython] extension which adds a `%%bitey` cell magic for automatically compiling C (or C++) code into LLVM bitcode and loading the bitcode with Bitey.\n",
      "\n",
      "Requirements\n",
      "------------\n",
      "\n",
      "* [Bitey], importable from Python as `import bitey`\n",
      "* [clang], executable as `clang` in the shell\n",
      "\n",
      "[Bitey]: https://github.com/dabeaz/bitey\n",
      "[IPython]: http://www.ipython.org/\n",
      "[clang]: http://clang.llvm.org/"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load_ext biteymagic"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Demos\n",
      "=====\n",
      "\n",
      "The following demos are taken, with slight modification, from the examples in the Bitey source."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The Basics\n",
      "----------\n",
      "\n",
      "Calculate the Fibonacci numbers using a simple recursive formula."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%bitey\n",
      "int fib(int n) {\n",
      "  if (n < 3) {\n",
      "    return 1;\n",
      "  } else {\n",
      "    return fib(n-2) + fib(n-1);\n",
      "  }\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "map(fib, xrange(1,10))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "[1, 1, 2, 3, 5, 8, 13, 21, 34]"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Bitey understands most basic C datatypes including integers, floats,\n",
      "void, pointers, arrays, and structures.  Because it builds a ctypes\n",
      "based interface, you would access the code using the same\n",
      "techniques. Here is an example that mutates a value through a\n",
      "pointer."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%bitey\n",
      "void mutate_int(int *x) {\n",
      "  *x *= 2;\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import ctypes\n",
      "x = ctypes.c_int(2)\n",
      "mutate_int(x)\n",
      "x.value"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "4"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Structs\n",
      "-------\n",
      "\n",
      "An example involing a structure.  One subtle issue with structure wrapping is that LLVM bitcode doesn't encode the names of structure fields. So, Bitey simply assigns them to an indexed element variable like this:\n",
      "\n",
      "    >>> p1.e0         # (Returns the .x component)\n",
      "    3\n",
      "    >>> p1.e1         # (Returns the .y component)\n",
      "    4\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%bitey\n",
      "#include <math.h>\n",
      "\n",
      "struct Point {\n",
      "  double x;\n",
      "  double y;\n",
      "};\n",
      "\n",
      "double distance(struct Point *p1, struct Point *p2) {\n",
      "  return sqrt((p1->x - p2->x)*(p1->x - p2->x) + \n",
      "\t      (p1->y - p2->y)*(p1->y - p2->y));\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "p1 = Point(3,4)\n",
      "p2 = Point(6,8)\n",
      "print distance(p1,p2)\n",
      "print p1.e0, p1.e1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "5.0\n",
        "3.0 4.0\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Performance\n",
      "-----------\n",
      "\n",
      "The performance profile of Bitey is going to be virtually identical that of using ``ctypes``.  LLVM bitcode is translated to native machine code and Bitey builds a ``ctypes``-based interface to it in exactly the same manner as a normal C library."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%bitey -03\n",
      "/* A function that determines if an integer is a prime number or not.\n",
      "   This is just a naive implementation--there are faster ways to do it */\n",
      "\n",
      "int isprime(int n) {\n",
      "  int factor = 3;\n",
      "  /* Special case for 2 */\n",
      "  if (n == 2) {\n",
      "    return 1;\n",
      "  }\n",
      "  /* Check for even numbers */\n",
      "  if ((n % 2) == 0) {\n",
      "    return 0;\n",
      "  }\n",
      "  /* Check for everything else */\n",
      "  while (factor*factor < n) {\n",
      "    if ((n % factor) == 0) {\n",
      "      return 0;\n",
      "    }\n",
      "    factor += 2;\n",
      "  }\n",
      "  return 1;\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%timeit\n",
      "isprime(10143937)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "10000 loops, best of 3: 36.8 us per loop\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Additional optimization flags given after `%%bitey` are passed directly to `clang`.  For example, consider a simple function which sums a vector."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%bitey\n",
      "double sum1d(double* x, int n)\n",
      "{\n",
      "  int i;\n",
      "  double ret;\n",
      "  ret = 0.0;\n",
      "  for(i=0; i<n; i++)\n",
      "    ret = ret + x[i];\n",
      "  return ret;\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%bitey -O2\n",
      "double sum1d_O2(double* x, int n)\n",
      "{\n",
      "  int i;\n",
      "  double ret;\n",
      "  ret = 0.0;\n",
      "  for(i=0; i<n; i++)\n",
      "    ret = ret + x[i];\n",
      "  return ret;\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "n = 1000\n",
      "A = np.random.rand(n)\n",
      "x = np.ctypeslib.as_ctypes(A)\n",
      "print('sum1d:')\n",
      "%timeit sum1d(x, n)\n",
      "print('sum1d_O2:')\n",
      "%timeit sum1d_O2(x, n)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sum1d:\n",
        "100000 loops, best of 3: 7.39 us per loop"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "sum1d_O2:\n",
        "100000 loops, best of 3: 3.29 us per loop"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "C++ Support\n",
      "-----------\n",
      "\n",
      "Bitey does not support C++ functions, but C++ code can be compiled if the function is decorated with `extern \"C\"`. In addition, you will need to pass `-x c++` to `clang` to specify that the language is C++."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%%bitey -O2 -x c++\n",
      "#include <numeric>\n",
      "extern \"C\" double sum1d_cpp(double* x, int n)\n",
      "{\n",
      "  return std::accumulate(x, x+n, 0.0);\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print('sum1d_cpp')\n",
      "%timeit sum1d_cpp(x, n)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sum1d_cpp\n",
        "100000 loops, best of 3: 3.33 us per loop"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 16
    }
   ],
   "metadata": {}
  }
 ]
}

biteymagic.py

# -*- coding: utf-8 -*-
"""Bitey related magics.

Bitey and a list of its requirements may be found at
https://github.com/dabeaz/bitey.
"""
#-----------------------------------------------------------------------------
# Copyright (C) 2012, IPython Development Team.
#
# Distributed under the terms of the Modified BSD License.
#
# The full license is in the file COPYING.txt, distributed with this software.
#-----------------------------------------------------------------------------

from __future__ import print_function

import bitey
import imp
import io
import os
import pipes
import subprocess
import sys

try:
    import hashlib
except ImportError:
    import md5 as hashlib

from IPython.core.magic import Magics, magics_class, cell_magic

@magics_class
class BiteyMagics(Magics):
    """Magics for Bitey, a Bitcode Import Tool"""

    def _import_all(self, module):
        for k,v in module.__dict__.items():
            if not k.startswith('__'):
                self.shell.push({k:v})

    @cell_magic
    def bitey(self, line, cell):
        """Compile C code into LLVM bitcode and import using bitey.

        Usage, in cell mode::

            %%bitey <compiler flags>
            <C code>

        The compiler flags are passed verbatim to `clang` so they may be
        used to control warnings (`-Wall`), add optimizations (`-O2`), and
        modify features (`-fno-builtin`).

        Bitey may also be used to compile C++ code if the functions are
        defined with C linkage (`extern "C"`) and `-x c++` is added to the
        compiler flags.
        """
        code = cell if cell.endswith('\n') else cell+'\n'
        lib_dir = os.path.join(self.shell.ipython_dir, 'bitey')
        key = line, code, sys.version_info, sys.executable
        if not os.path.exists(lib_dir):
            os.makedirs(lib_dir)

        module_name = "_bitey_magic_" + \
                      hashlib.md5(str(key).encode('utf-8')).hexdigest()
        c_name = module_name+'.c'
        o_name = module_name+'.o'

        c_path = os.path.join(lib_dir, c_name)
        o_path = os.path.join(lib_dir, o_name)

        if not os.path.exists(c_path):
            with io.open(c_path, 'w', encoding='utf-8') as f:
                f.write(code)

        if not os.path.exists(o_path):
            try:
                startupinfo = None
                if os.name == 'nt':
                    # Avoid a console window in Microsoft Windows.
                    startupinfo = subprocess.STARTUPINFO()
                    startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
                subprocess.check_output(['clang', '-c', '-emit-llvm'] +
                                        line.split() + [c_name],
                                        stderr=subprocess.STDOUT,
                                        cwd=lib_dir,
                                        startupinfo=startupinfo)
            except subprocess.CalledProcessError as e:
                print(e.output, file=sys.stderr)
                print("ERROR: command `%s` failed." %
                      ' '.join(map(pipes.quote, e.cmd)),
                      file=sys.stderr)
                return

        with io.open(o_path, 'rb') as f:
            bitcode = f.read()

        module = bitey.loader.build_module(module_name, bitcode)
        sys.modules[module_name] = module
        self._import_all(module)

_loaded = False

def load_ipython_extension(ip):
    """Load the extension in IPython."""
    global _loaded
    if not _loaded:
        ip.register_magics(BiteyMagics)
        _loaded = True