Pickle Examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiple References to Objects and Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import numba\n",
    "import cPickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python example\n",
    "\n",
    "When dealing with regular Python objects, the `pickle` module is smart enough to ensure that multiple references to an object are preserved during unpickling.\n",
    "\n",
    "As an example, let's consider a dictionary which contains the same array twice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': array([1, 2, 3]), 'b': array([1, 2, 3])}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "an_array = np.array([1,2,3])\n",
    "a_dict = { 'a': an_array, 'b': an_array }\n",
    "a_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since it is the same array object for both dictionary values, if we modify an element in one array, we see it both places:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': array([1, 2, 4]), 'b': array([1, 2, 4])}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_dict['a'][-1] = 4\n",
    "a_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can confirm the objects have the same Python id (memory address on most platforms):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id(a_dict['a']) == id(a_dict['b'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moreover, if we pickle and unpickle the dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': array([1, 2, 4]), 'b': array([1, 2, 4])}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pickled = cPickle.dumps(a_dict)\n",
    "new_dict = cPickle.loads(pickled)\n",
    "new_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can verify the same array object is present for both values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id(new_dict['a']) == id(new_dict['b'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we modify the contents of one array, we see it in both places as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': array([1, 2, 5]), 'b': array([1, 2, 5])}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_dict['a'][-1] = 5\n",
    "new_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if you test this out with views on NumPy arrays, you'll find the link is broken by pickling and unpickling, and you'll end up with distinct objects when you unpickle.  (Good to keep in mind if you are sending NumPy views to a multiprocessing worker...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numba example\n",
    "\n",
    "The analog for JIT classes would be to have a class with two attributes, `a` and `b`, which are initialized with the same array object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "spec = [\n",
    "    ('a', numba.int32[:]),\n",
    "    ('b', numba.int32[:]),\n",
    "]\n",
    "\n",
    "@numba.jitclass(spec)\n",
    "class MyClass(object):\n",
    "    def __init__(self):\n",
    "        self.a = np.zeros(3, dtype=np.int32)\n",
    "        self.b = self.a\n",
    "        self.a[:] = 1, 2, 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we create one of these objects, and print the contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1, 2, 3], dtype=int32), array([1, 2, 3], dtype=int32))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = MyClass()\n",
    "m.a, m.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then verify that modifying `a` also modifies `b`, because they point to the same array data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1, 2, 4], dtype=int32), array([1, 2, 4], dtype=int32))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.a[-1] = 4\n",
    "m.a, m.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we access the attributes from Python, Numba generates a temporary Python object to wrap the data.  To check the ids we need to first need to save them to local variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = m.a\n",
    "b = m.b\n",
    "id(a) == id(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite sharing the same underlying data pointer, `a` and `b` look like different objects to the Python interpreter.  (In effect, both are view objects on the underlying data.)  Without some care implementing pickle for JIT classes, this would result in broken links during unpickling.\n",
    "\n",
    "Even worse, the nature of reference counting means that you can easily see the same id in an expression for two different, short lived objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id(m.a) == id(m.b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This happens because the temporary wrapper around `m.a` is released as soon as `id()` returns, and that same memory location is used to create the temporary wrapper around `m.b`.  Any pickle solution for JIT classes has to deal with both of these issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}