baseline_a2c.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agentnet intro\n",
    "\n",
    "In this tutorial we'll\n",
    "* еrain on Atari MsPacman game with deterministic framerate using OpenAI gym\n",
    "* train a simple lasagne neural network for n-step q-learning\n",
    "* no experience replay, 10 synchronous agents, soft target networks\n",
    "\n",
    "_Note that this agent is not state of the art on this particular game_, it's just simple and it works.\n",
    "\n",
    "\n",
    "### About OpenAI Gym\n",
    "\n",
    "* Its a recently published platform that basicly allows you to train agents in a wide variety of environments with near-identical interface.\n",
    "* This is twice as awesome since now we don't need to write a new wrapper for every game\n",
    "* Go check it out!\n",
    "  * Blog post - https://openai.com/blog/openai-gym-beta/\n",
    "  * Github - https://github.com/openai/gym\n",
    "\n",
    "\n",
    "### New to Lasagne and AgentNet?\n",
    "* We only require surface level knowledge of theano and lasagne, so you can just learn them as you go.\n",
    "* Alternatively, you can find Lasagne tutorials here:\n",
    " * Official mnist example: http://lasagne.readthedocs.io/en/latest/user/tutorial.html\n",
    " * From scratch: https://github.com/ddtm/dl-course/tree/master/Seminar4\n",
    " * From theano: https://github.com/craffel/Lasagne-tutorial/blob/master/examples/tutorial.ipynb\n",
    "* This is pretty much the basic tutorial for AgentNet, so it's okay not to know it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: THEANO_FLAGS=device=gpu0,floatX=float32\n"
     ]
    }
   ],
   "source": [
    "#setup theano/lasagne. Prefer GPU\n",
    "%env THEANO_FLAGS=device=gpu0,floatX=float32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#If you are running on a server, launch xvfb to record game videos\n",
    "#Please make sure you have xvfb installed\n",
    "import os\n",
    "if os.environ.get(\"DISPLAY\") is str and len(os.environ.get(\"DISPLAY\"))!=0:\n",
    "    !bash ../xvfb start\n",
    "    %env DISPLAY=:1\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment setup\n",
    "* Here we basically just load the game and check that it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "#number of parallel agents and batch sequence length (frames)\n",
    "N_AGENTS = 10\n",
    "SEQ_LENGTH = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using gpu device 0: GeForce GTX 1080 (CNMeM is enabled with initial size: 45.0% of memory, cuDNN 5105)\n",
      "[2017-02-22 03:16:51,268] Making new env: MsPacmanDeterministic-v0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fa983a717d0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASAAAAFjCAYAAACdT9ZCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3X2QXVW55/Hvk5D3t86LdIy5kiCYAN5KSIAUMshIBAOO\nKBNfCBeYkcr4RizMTJVoBQuEyZXiehUEIsplFIQLwwW8KBVFQIdXAUOHBkJAc0nAENLaeU86byRr\n/jjn7Kxuzuk+u3uvs/Y5/ftUpWpln+esvdbaJytr7b3X3uacQ0QkhgGxCyAi/Zc6IBGJRh2QiESj\nDkhEolEHJCLRqAMSkWjUAYlINOqARCQadUAiEo06IBGJJmoHZGaXmNlaM9ttZs+Y2YkxyyMitRWt\nAzKzzwP/DFwBHA+0Ag+Z2YRYZRKR2rJYi1HN7BngWefcpcW/G/AX4IfOuWujFEpEauqwGDs1s0HA\nbOAfS9ucc87MHgFOLhM/Hvg4sA7YU6NiikjvDAWmAA855zZ1FxilAwImAAOBti7b24BpZeI/DtwZ\nulAikql/AP61u4BYHVBa6wDOOeccVq5cydy5c5MP3nnnnSR98ODBIDt/7LHHOO2004LkXTJw4MAk\nfdhhhcPyyCOPNGRd87LfLPdZ7vgB+Kc4Ssfv97//fV3XFQ7V16/rgAGFU8rt7e088MADUPx3251Y\nHVA7cABo7rK9GdhYJn4PwMqVK9m8eTNPPvlk8sEHP/hBjjnmGCDcP8ohQ4bQ3Ny1qNnyf8CDBg1K\n9jtx4sRk+/79+5N0Pdc1L/vNcp/ljh907oBKx6/e6wqH6vunP/2JV155BYDCaVzYu3dvKazH0yVR\nOiDn3H4zex6YC/wSkpPQc4EfVvreL37xC77zne9w++23J9tuvfXWJL1169Yg5R04cCAjRozIPN/S\nAQNoampK0hdffDFQ6HAvu+yyZHs91zWP+81in6VjWO74dVU6fkOHDq3rusKh+i5ZsiTZNmbMGABa\nWlo44YQTqsoz5hTs+8DPih3Rc8BiYDjws4hlEpEaitYBOefuKd7zcxWFqdcLwMedc3+LVSYRqa2o\nJ6Gdc8uAZWm+c95553U6GfvWW28l6fb29szK5mtubuaNN94IknfJhAmH7r88cOAAAOeee25D1jUv\n+81yn+WOH3Q+B1Q6fkcccURd1xUO1deva2mK5k/VelJ3a8EWLFhQ831OnTq15vsEmD9/fs33Gauu\nMfYbq67Tp0+v+T5j1bUnddcBiUjjqJf7gCryh7j+pVD/qoRvy5YtSbrSVags4quJTXslK6919eOr\nia10C0Fe69RdfJpjWA/Hr7v4EFdeNQISkWjUAYlINHU/BfONHj06SVe6GeyWW25J0v6wM+v4tHmn\nlae6+vHVxFYayue1TtXGp9Gf6todjYBEJBp1QCISTUNNwbZv356k77yz/NM7duzYUXZ7yPhqYseO\nHVs2ppJ6rmslea1TNfGNcvyqjU9b30o0AhKRaNQBiUg0DTUF89elbNiwIdV3Q8anzbsajVjXvNap\nN/E96U917Y5GQCISjTogEYmmoaZgafnPs/XP6vvrdKrhD6c3b978rm15kFVdG1G549d1e2whf6vQ\n+UbESle4/LVjWdEISESiUQckItH06ymYvx7nkksuSdLjxo1LlY8/lL3++uuBMMPVvsiqro2o3PGD\nfB3DkL9V6DwF8/P3+fFZ0QhIRKJRByQi0fTrKVjpTY5w6J1GXdPV8B8c7+eZJ1nVtRH1p+NXqa7+\nFMyf7lUqQ1by2doi0i+oAxKRaPr1FKySat5r5D9gvBz/xrHJkyeXjVm/fn26ggXIP807nBpJT8cP\nwh7DWh6/SnWt9F1/u//dEL8VjYBEJBp1QCISjaZggYwaNSpJf/azny0b05eHfIfOX8K2cR6Onz+9\n8t/V9vbbb5eN8deRZbWGUCMgEYlGHZCIRKMpWCDbtm1L0suWLSsbs3v37iSddk1P6PwlXRvX+/Hz\ny3PTTTfVrDwaAYlINOqARCQaTcEC8a8q7Nq1q+7yl7BtnLfjF6s8GgGJSDTqgEQkmn49BfMfTbBm\nzZok/be//S1VPlu3bi2bZxZl8W8E623e3eWftq6NqD8dvyzqmiWNgEQkGnVAIhKNVfNYgtjMbBbw\n/IoVK5g1axabNm1KPlu6dGmS9reLSDgTJkwAYMmSJcm28ePHA9DS0sLs2bMBZjvnWrrLRyMgEYlG\nHZCIRKMOSESiUQckItGoAxKRaBrqRsShQ4cm6RkzZpSNeeGFF5K0/5DtrOND5p3n+Gpi9+7dWzYm\nT8cvbXyeyhIqvrW1tez2vtAISESiUQckItE01BRsyJAhSXrmzJllY1avXp2k/WFn1vEh885zfDWx\nlaZgeTp+aePzVJZaxGcl8xGQmX3LzJ4zs+1m1mZmvzCzD5aJu8rMNphZh5k9bGZHZV0WEcm3EFOw\nU4EbgDnAx4BBwG/NbFgpwMwuAxYBXwROAnYBD5nZ4ADlEZGcCr4WzMwmAH8FPuKce7K4bQPwT865\nHxT/PhpoA/6bc+6eMnloLZhIjtTTWrAmwAGbAcxsKjAReLQU4JzbDjwLnFyD8ohITgTtgKxwZus6\n4Enn3CvFzRMpdEhtXcLbip+JSD8R+irYMuBY4JTA+xGROhSsAzKzG4GzgVOdc297H20EDGim8yio\nGVjZXZ6LFy+mqamJffv2+fth+vTpmZVbRKp33333cf/99wMweHDhGpL/ksOeBDkJXex8PgWc5px7\nvcznlU5CX+Sc+7cy8Z1OQu/cuTP57OGHH07SO3bsKFuePXv2JOkVK1Yk6Ur3o1TiLxUonmR7l1L+\nWeXtb69G6LqmLU8MeWqDGL+9LPOvVNfRo0cDcMYZZyTbRo4cCaQ7CZ35CMjMlgELgHOAXWbWXPxo\nm3OudDSuAy43szXAOuBqYD3wQNblEZH8CjEF+zKFk8z/r8v2LwC3AzjnrjWz4cCPKVwlewI4yzm3\nDxHpNzLvgJxzVV1Zc85dCVyZJm8zw8ySoR7Aueee2+P3/PuDXn311SSddpjq73f+/PllY0r5Z5V3\n6d6KaoWua9ryxJCnNojx28sy/9DHW4tRRSQadUAiEk1drob3V+5WUoMlJsH25eedt7pWU54Y8tQG\n9fzb65p/6OOtEZCIRKMOSESiqcspWFb8M/9nnXVW2Zjly5fnLu/eyFt5YshTG4QuS57q2h2NgEQk\nGnVAIhJNv56C+VcQ/Bu3sriyEDLv3shbeWLIUxuELkue6todjYBEJBp1QCISTb+egu3atStJ//KX\nv+wxftiwYT3G1CLv3shbeWLIUxuELkue6todjYBEJBp1QCISjTogEYlGHZCIRKMOSESiqaurYPfe\ney/PPfdc6u/5N2L5D7TPE/+B+nfccUeSHjJkSKp8sqprVuUpGThwYJIeMWJE2Rj/ys2BAwd6tR/I\nVxvUw28Psj3eb775ZtWxGgGJSDTqgEQkmrqagrW2trJ+/frYxQjCH6o///zzEUtSkHV5pkyZkqRn\nzpyZpAcMOPR/4Msvv5yk83Cc83ZMQsqyrv6D+HuiEZCIRKMOSESiqaspmNQX/2rXtGnTkvRf/vKX\nJO1fZZk4cWKS3rx5c5Lu6OgIVUSJTCMgEYlGHZCIRNNQU7CmpqYkfeGFF5aNuf3225O0/86jauLz\nVJa8xvux/oPR33rrrSTtXyXxr4I1Nzcn6TPOOCNJ+zfw5bUNQpclrdC/v23btvW6bD6NgEQkGnVA\nIhJNQ03Bdu/enaQff/zxsjF79uwpu72a+EGDBuWmLHmN92NHjRqVpI8++ugkffzxx5fNb//+/Un6\niSeeSNL+OqV6a4Os8k7z24Pa/v76QiMgEYlGHZCIRNNQUzB/PUtra2uq71YT708pYpclr/F+rH8V\nbNKkSWW3+++p8q92vfDCC2W3Z1HG0PGxf3tQ299fX2gEJCLRqAMSkWjqago2YMCATjeuVcsf5od+\nNW3pyX+9KWcWQtTVv0nNT/cU6+//pZdeStKV2sa/yuKn07Zl7DbIir8f/4mStdxvb+qa5nhpBCQi\n0agDEpFo6moKtnDhQo477rjU39u6dWuSvu2225J0VutZ/HU3ixYtAuCdd97JJO+0sqqrX6eLLrqo\n7Pa8asQ2OOywQ/9Ux4wZk6T9uvZFlnVdtWoVDz74YFWxGgGJSDTqgEQkmrqagk2ePJkPfOADqb/n\nP/7BH8pmxb9CccQRR2SefxpZ1dX/rv9A+fHjx/c6z1pRG6SXZV3TTHU1AhKRaNQBiUg0dTUFM7Oq\nb4yq5gY0/4apSmtt/EdBdFeumELX1a/fwYMHk7S/RsvfXit+nfz1ZdXE96UN+tPx7k1d03xHIyAR\niUYdkIhEU1dTsKz5N3QtXLiwbMwtt9xSq+IElVVd/SscN998c5LesmVLH0rXO2PHjk3SX/nKV3qM\n1/HuLA91DT4CMrNvmtlBM/t+l+1XmdkGM+sws4fN7KjQZRGRfAnaAZnZicAXgdYu2y8DFhU/OwnY\nBTxkZoNDlkdE8iXYFMzMRgJ3AAuBb3f5+FLgaufcg8XYi4A24NPAPaHK1JV/FeCuu+4qG+Nf6amH\ndVCVZFVX/2pXe3t72XStHDhwIElXcxVOx7uzPNQ15AjoJuBXzrnf+RvNbCowEXi0tM05tx14Fjg5\nYHlEJGeCjIDM7DxgJnBCmY8nAo7CiMfXVvxMRPqJzDsgM5sMXAd8zDm3v6f4mPxHZqxfvz5iScIL\nUdfQT5fMmo53/oQYAc0G3gO02KFbIgcCHzGzRcB0wIBmOo+CmoGV3WW8ePHiTpcXARYsWMCCBQsy\nKrqIpHHXXXe96xxTmsWoITqgR4C/77LtZ8Bq4Brn3OtmthGYC7wIYGajgTkUzhtVdMUVVzBjxox3\nbS+tfvZvPx89enRvy1+Rf9Jz8+bNmeefBb9cfnklPR3vznlu3749SZdO+p955pmceeaZnb7T2trK\n3Llzq8o/8w7IObcLeMXfZma7gE3OudXFTdcBl5vZGmAdcDWwHngg6/KISH7V6k7oTicLnHPXmtlw\n4MdAE/AEcJZzbl+NyiMiOVCTDsg5d3qZbVcCV6bJ59Zbb6W5ubni5+PGjUvSl1xySZqsq+IPQW+8\n8cYkXctXpvTEHzJXs5JfKtPx7nw+Z9myZUm6uylpW1vXC9yVaTGqiESjDkhEoqmr1fBbt26tevgb\n4gFZ/nA3xupvqS0d787/jvxnbfvprtK8KkgjIBGJRh2QiERTV1OwrPmvIql0dW3jxo1J2n/WbU/x\naWLzEp/m6kU90vHuLA/HWyMgEYlGHZCIRNOvp2D+60rOP//8sjH+c3P9YW1P8Wli8xifNX+d3rBh\nw8rGdHR0JOkQK+11vCvHx6IRkIhEow5IRKLp11Mwf51LpeGovx7IlyY+ZN6h4v11dVl4//vfn6Rn\nzZqVpP2p2R/+8Ick/dZbb2W6f9Dx7i4+6+NdLY2ARCQadUAiEk2/noL561zSrF9JGx8y71rE99aI\nESOS9LHHHpuk/WcUDx586FVwkyZNStL+2iv/6lhf6Hjnj0ZAIhKNOiARiaaupmCTJ0/mve99b8XP\n/TP5/mM7/FeU9IW/lsifLvjbY/Pr6l9JivGAev9muCFDhvQY78f4341Fx7tzXadMmZKku76dxlfp\nRtNyNAISkWjUAYlINPkZS1Zh4cKFnW5i645/g1t3T29Lo6mpKUl/7WtfS9ITJkzIJP8stLe3J+lr\nrrkmSWfVBr01cuTIJH388ceXjdm5c2eSXrmy23dU1oSOd+ep1qJFi6r6TktLCz/5yU+qitUISESi\nUQckItHU1RRswIABnaZWMflXB/JSJsjXO6t8/lUZ/4ZA/7EbWV2tDKG/Hm//amS19U7TPvlpSRHp\nd9QBiUg0dTUFM7Oqb1Cr5ol6/o1v06ZNKxvz2muvVVWumGpZ1zT8adcLL7yQpCtNG3bv3l02nRUd\n787SHu9q652mfTQCEpFo1AGJSDR1NQXLmr9m5bTTTisb88Ybb9SqOEFlVVf/atDUqVOT9NixY7v9\n3v79+8umKznyyCN7jPH3Wc3aPx3vzvJQV42ARCQadUAiEk2/noL5T4q74YYbeozP0xqgtLKqq782\nyF8flSeV1kHpeFcWq64aAYlINOqARCSafj0Fk/Ri34RXSYhXOUt4GgGJSDTqgEQkmrqagrW2trJn\nz57U3/NfQdub7/fEz3PVqlUA7N27N/P9VCOruvrfXbFiRZIeNWpUr/MsqfS4Bn8a1ZcpVT20QVr+\n2q7jjjsu8/yzrGuaNWYaAYlINOqARCSaupqC3XvvvYwfPz52Md7Ff5j6nXfeCXR+WHg9KlenvvBf\n0/yZz3wmSftT1TVr1iRp//EdsZ6UmHUb9IV/o+CSJUsyzz/LuqZ5IL5GQCISjTogEYmmrqZg9UA3\nxJW3a9euJO1PtU499dQk7V+JaW1trU3BJCqNgEQkGnVAIhJNQ03B/Cstp59+etmYRx99NEn765qq\nic9TWfIaX02sf4VrxowZSXr48OFJet68ecHKGCI+dFnSCv376+jo6HXZfEFGQGY2ycx+bmbtZtZh\nZq1mNqtLzFVmtqH4+cNmdlSIsohIfmXeAZlZE/AUsBf4OHAM8L+ALV7MZcAi4IvAScAu4CEzG5x1\neUQkv0JMwb4JvOmcW+ht6/r060uBq51zDwKY2UVAG/Bp4J7e7tgfRg4dOrTHmGrie/sa3hBlqYf4\namL9VzP7ab+t81SnauLz9NsLVZ4Qj2IJMQX7JLDCzO4xszYzazGzpDMys6nARCCZUDrntgPPAicH\nKI+I5FSIDuhI4CvAa8CZwI+AH5rZhcXPJwKOwojH11b8TET6Ccv6xjkz2ws855w71dt2PXCCc+4U\nMzsZeBKY5Jxr82L+L3DQObegTJ6zgOfPPvvsqGvBKq3H8dtw6dKlQLr1MP3N3Llzk7S/Luzuu+9O\n0o899lhNy5R3aX57EPf3t2nTJpYvXw4w2znX0l1siHNAbwOru2xbDfzXYnojYEAznUdBzcDK7jJe\nsWIFgwd3Pk89ZcqUTi/IE5HaWbt2LevWreu0bd++fVV/P0QH9BQwrcu2aRRPRDvn1prZRmAu8CKA\nmY0G5gA3dZfxCSeckMvV8CL91dSpU981APBGQD0K0QH9AHjKzL5F4YrWHGAh8D+8mOuAy81sDbAO\nuBpYDzwQoDySA/5NhpMnT07Sv/3tb5P0U089VdMySXyZd0DOuRVmdi5wDfBtYC1wqXPubi/mWjMb\nDvwYaAKeAM5yzlU/dhORuhdkKYZzbjnQ7RjMOXclcGWafD/xiU9w9NFHpy6P/7Al/39cf4V2bCNH\njkzSZ5xxRtmYvpS9Vvn7efv79O8hOeywQz+7pqamJL1gwbuuPwDZHb+QbRC6fUOrVH5/e7X+/Oc/\nVz0F02JUEYlGHZCIRFNXq+HnzJnDrFmzeg7swr8n4vHHH0/SeRoG+7e/+w/p8vWl7LXK3887qyuW\nWR2/kG0Qun1Dq1T+3hzDcePGVR2rEZCIRKMOSESiqaspmJlVvSK3np/N7NcxRD1C5l9phXVaoY9f\nrdqgHn+HfT2Gab6jEZCIRKMOSESiqaspWNbGjBmTpM8777yyMXfddVfu8m6E/LNQz21Qz2XPkkZA\nIhKNOiARiaZfT8H27t2bpCu9idN/tknXZxHFyrsR8s9Cntogb+1bD8cPNAISkYjUAYlINP16CrZn\nz54k/cwzz/QYn+bRBCHzboT8s5CnNshb+9bD8QONgEQkInVAIhKNOiARiUYdkIhEow5IRKKpq6tg\nzjmcc2zfvj3Zdueddybpbdu2lf3e/v37e4ypxtatW5P0TTeVf4VZX/KvZ6W28dtl0KBBmeSd1fGr\nZ6F/e5Xyr3QMS2vNLrjggmTb6NGjU+9XIyARiUYdkIhEU1dTsBJ/SL5mzZok3d7eHnS/77zzTtn9\nyqG2UbuEEfq3lzb/CRMmAJ3/LfaGRkAiEo06IBGJpi6nYD7/od8DBhzqT/33HPl2796dpP2HZ2cd\nX02sv14nrWrqGjr/cm1TTWylB7WHOH55beM8/Varje9LW1aiEZCIRKMOSESiqfspmM9/EPfChQvL\nxtxyyy1J2h92Zh2fNu+00tY1dP6l+lYT69/0lsU+q41PK2Qb5+m32pv4rGgEJCLRqAMSkWgaagq2\nY8eOJH3vvfeWjdm5c2fZ7SHjq4ltamoqG1NJ2rqGzr83sVnts9r4kG1Qq/bNS3za+laiEZCIRKMO\nSESiaagpmL+eZe3atam+GzI+bd7V6EtdQ+bfl7LU8vhVI2Qb5/W32pv4vtAISESiUQckItE01BQs\nLX89zrBhw5L0wIEDU+Vz4MCBJN3R0QFUXu8Uy8GDB5O0/0RJv5x+TFqltuxLO1ZSrn0hfRuHboOQ\nQv5WId7vVSMgEYlGHZCIRNOvp2D+Q7S/9KUvJemxY8emymfLli1J+uabbwby9/B0f8pR6aHm/s1x\naZXasi/tWEm59oX0bRy6DUIK+VuFeL9XjYBEJBp1QCISTb+egh122KHqNzc3J+nx48enymfw4MFl\n88wT/8a3v/71r5nnX6p3X9qxkqzaN3QbhNSov1WNgEQkGnVAIhJN/DFYhvybsg4//PCyMZWG3v4T\n4fx0JT3duNWXslSjmvzb2tqStF+nrOLLlT9tO1ZSzY1xeW2DauTpt9rX8vRF5iMgMxtgZleb2etm\n1mFma8zs8jJxV5nZhmLMw2Z2VNZlEZF8CzEF+ybwJeCrwHTgG8A3zGxRKcDMLgMWAV8ETgJ2AQ+Z\n2eB3ZycijSrEFOxk4AHn3G+Kf3/TzM6n0NGUXApc7Zx7EMDMLgLagE8D9/R2x/7NWhdccEHZmBAP\n1o5RlrT5+0P1EPEx1HMb5Om3CvHKE2IE9DQw18yOBjCzGcApwPLi36cCE4FHS19wzm0HnqXQeYlI\nPxFiBHQNMBp41cwOUOjkljjn7i5+PhFwFEY8vrbiZyLST4TogD4PnA+cB7wCzASuN7MNzrmfB9hf\nwl/r89Of/rRsjL/WJ6u1SjHKkjZ/X4j4kG1ZSZ7aIPTxC92+scoTogO6Fviuc+7fin9fZWZTgG8B\nPwc2AgY003kU1Ays7C7jxYsX09TUxL59+5JtZsb06dMzK7yIVO++++7j/vvvBw7dZZ1mYWuIDmg4\ncKDLtoMUzzc559aa2UZgLvAigJmNBuYA5ZcoF/3gBz9g1qxZbNq0Kdm2dOnS7EouIqnMnz+f+fPn\nA4eWhbS0tDB79uyqvh+iA/oVcLmZrQdWAbOAxcC/eDHXFWPWAOuAq4H1wAN92bH/tLf29va+ZNVn\nocvSl/xDx9dKPbdBnn6rEK88ITqgRRQ6lJuAw4ENwI+K2wBwzl1rZsOBHwNNwBPAWc65fe/OTkQa\nVeYdkHNuF/A/i3+6i7sSuDLr/YtI/WiotWCNyH9kwuTJk5P0yJEjM8nff0TFxo0bk7Q/JG8UlR5p\nkcVjKfyrQv66Kr9986xS2/j8dXJZ0Wp4EYlGI6CcGzNmTJJetChZTpfZa1T8K4rf+973ym5vFE1N\nTUnab8ssHpzmL9vwH/qV53b0y1ypbXz+7yMrGgGJSDTqgEQkGk3Bcq7S0D4rQ4YMyTzPejB06NAk\n3V/boJJK7dGXB8xVohGQiESjDkhEotEULBD/voqjjir/tNk1a9akyjOrIXA1V9BClD9rWZUx66lF\nvbev3x5ZXW2tRCMgEYlGHZCIRKMpWCD+Uol58+aVjan04Kc8qIfy10MZK8lD2f3plb/05s033ywb\n4y8ryeqtqhoBiUg06oBEJBpNwQLZunVrkr7xxhvLxvhD2gkTJgQvUxr1UP56KGMleSu7X54bbrih\nZuXRCEhEolEHJCLRaApWA/XyUKpK6qH89VDGSvJW9lqWRyMgEYlGHZCIRNOvp2D+Cw5ffvnlJD1q\n1KhU+fhvlfTzjFGWtLIou//dEGXPuoxQuzbOW9mzKk9WNAISkWjUAYlINP16CuYPR2+99daIJclX\nWXqjVP48l72e27iey94djYBEJBp1QCISjTogEYlGHZCIRKMOSESiaairYMOHD0/SH/7wh8vGPPXU\nU0naf/h21vEh885zfDWxu3fvLhuTp+OXNj5PZQkV//TTT5fd3hcaAYlINOqARCSahpqCDRhwqD+d\nNGlSjzH+sDPr+JB55zm+mthK8nT80sbnqSy1iM+KRkAiEo06IBGJxkK/ejULZjYLeH7FihXMmjWL\nTZs2JZ8tXbo0SfvbRSSc0kPplyxZkmwbP348AC0tLcyePRtgtnOupbt8NAISkWjUAYlINOqARCQa\ndUAiEo06IBGJpi5vRBw2bFiSPvvss5N0R0dHjOKI9DuldXv+v8Xe0AhIRKJRByQi0dTVFOzXv/41\nq1evziy/gQMHJml/LUwWr6YNmbc0Lv934ztw4ECNS9K9HTt2AHD//fe/67N169ZVnY9GQCISjTog\nEYmmrtaC9fb7Q4YMSdLz5s1L0ieddFKSnj59epL+zne+k6RffPHFXuWfVd7S+AYNGpSkTzzxxCR9\n2GGHzpCsWrUqSdfRmsfs14KZ2alm9ksze8vMDprZOWVirjKzDWbWYWYPm9lRXT4fYmY3mVm7me0w\ns3vN7PC0ZRGR+tabKdgI4AXgq8C7hk9mdhmwCPgicBKwC3jIzAZ7YdcBnwDmAx8BJgH39aIsIlLH\nUl8Fc879BvgNgPmXdw65FLjaOfdgMeYioA34NHCPmY0GLgbOc849Voz5ArDazE5yzj3Xq5p0w3+S\n27HHHpuk/eHuuHHjym6vZppULv+s8pbG5/8zevXVV5P08ccfn6T9382KFSuSdHt7e+DShZXpSWgz\nmwpMBB4tbXPObQeeBU4ubjqBQsfnx7wGvOnFiEg/kPVVsIkUpmVtXba3FT8DaAb2FTumSjEi0g/U\n1Y2IvVXpPVRbt25N0ldddVWS/uMf/9jn/LPKWxrfvn37krR/w6H/u/KviM2ZMydJL1++PEnXwxXt\nrrIeAW0EjMIox9dc/KwUM7h4LqhSjIj0A5l2QM65tRQ6kbmlbcWOZg5Qeq3i88A7XWKmAe8H/pBl\neUQk31IrOtMQAAAGaklEQVRPwcxsBHAUhZEOwJFmNgPY7Jz7C4VL7Jeb2RpgHXA1sB54AAonpc3s\nVuD7ZrYF2AH8EHgqxBWw7vjDXf9RHlms1wqZtzSumTNnJmn/BsVTTjmlbPyTTz6ZpLdt2xauYIH0\n5hzQCcDvKZxsdsA/F7ffBlzsnLvWzIYDPwaagCeAs5xz+7w8FgMHgHuBIRQu61/SqxqISN3qzX1A\nj9HD1M05dyVwZTef7wW+VvwjIv1Uv7gK5tu48dB57q9//etJ2r/p67vf/W6Svu2223qVf4i8pTGN\nGTMmSX/0ox9N0v4Nrv5asC1btiTpnTt3Bi5dWFoNLyLRqAMSkWj63RTs7rvvTtLve9/7krT/+Iy+\nrNEq5R8ib2lM/g2HL730UpL2byx88MEHk7R/42K90whIRKLpFw8kq2Tw4ENPCPHvufDv2+lt+4TM\nWxpXvTwTukrZP5BMRCQr9XIOaGiITA8ePJik/f9hshiZhMxbGleD/T56/HdbLx3QlBCZ+ssisl4i\nETJvaVz+f1wNYAqH1oCWVS/ngMYDH6ewtmxP3NKISA+GUuh8HnLOdfsE/brogESkMekktIhEow5I\nRKJRByQi0agDEpFo6qYDMrNLzGytme02s2fM7MSev5Uq/z6/8bUX+/yWmT1nZtvNrM3MfmFmHwy5\nXzP7spm1mtm24p+nzWxel5hM61mhHN8stvP3Q+3bzK4o7sP/80qo/XXJd5KZ/bz49t+OYpvPCrnv\n4r+PrvU9aGY3hNpnnznncv8H+DyFy+8XAdMpPG1xMzAhw33MA64CPkXhaY3ndPn8suI+/wvwIeDf\ngf8ABvdhn8uBC4FjgL8HHqRwq8GwUPul8EbaecAHKDxa938De4FjQtWzTBlOBF4HVgLfD1jXK4AX\ngfcAhxf/jAt5TIv5NgFrgX8BZgNHAB8Dpgb+PY336nk4heeuHwBOrdWxTV3mWDtO2bDPANd7fzcK\nz5n+RqD9HSzTAW0AFnt/Hw3sBj6X4X4nFPf9n2q8303AF2qxP2Ak8BpwOoVH+/odUKb7LnZALd18\nHqSuwDXAYz3E1OK4Xgf8qZb7TPsn91MwMxtE4X8R/02qDniEGr1Jtco3vmahicJztjfXYr9mNsDM\nzgOGA0/XqJ43Ab9yzv2uS1lC7fvo4rT6P8zsDjP7u8D7A/gksMLM7ilOrVvMbGHpw1q0c/HfzT8A\nt9Zqn72R+w6IwqhgIN2/bTW0at742idWeEH4dcCTzrnSeYog+zWzD5nZDgpTr2XAua7weuyg9Sx2\ndjOBb5X5OMS+nwH+O4W76L8MTAUeL77ZJWRdjwS+QmGkdybwI+CHZnZh8fPgvyfgXGAMhZdF1Gqf\nqdXLWrD+YBlwLFD+/SvZehWYQeEH+hngdjP7SMgdmtlkCh3sx5xz+0Puq8Q595D315fN7DngDeBz\nFNoglAHAc865bxf/3mpmH6LQCf484H59FwO/ds7l+mWf9TACaqdwIq27t62GVs0bX3vNzG4Ezgb+\ns3Pu7dD7dc6945x73Tm30jm3BGgFLg21v6LZFE4Gt5jZfjPbD5wGXGpm+yj8TxysjQGcc9uAP1E4\n+R6yrm8Dq7tsW03h5ZsE3jdm9n4KJ71v8TYH3Wdv5b4DKv5v+Tyd36Rqxb93u9I2wzJU88bXXil2\nPp8CPuqce7NW++1iADAk8P4eoXClbyaF0dcMYAVwBzDDOfd6wH2X8htJofPZELiuTwHTumybRmH0\nVYvjejGFDj15cXwNf0vpxDr7nfJs/ueADjpfht8EvCfDfYyg8I9iJoUrUV8v/v3vip9/o7jPT1L4\nh/TvwJ/p22XTZcAW4FQK/xOV/gz1YjLdL/CPxf0dQeFS7HcpvCr79FD17KYsXa+CZV3XfwI+Uqzr\nh4GHKfzDHB+yrhRe3rmXwrmuDwDnU3gD8Hmh6urlaxRu5Vha5rOaHduqyxtrx71o2K8WG3Y3hXfI\nn5Bx/qcVO54DXf78Hy/mSgqXMjuAh4Cj+rjPcvs7AFzUJS6z/VK4N+X1YjtuBH5b6nxC1bObsvzO\n74AC1PUuCrdr7AbeBP4V716ckHWlMKV+sZjvKgpvDe4ak/m+gTOKv6GyedXq2Fb7R4/jEJFocn8O\nSEQalzogEYlGHZCIRKMOSESiUQckItGoAxKRaNQBiUg06oBEJBp1QCISjTogEYlGHZCIRKMOSESi\n+f/b/lkfP+adIQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa986083190>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "from agentnet.experiments.openai_gym.wrappers import PreprocessImage\n",
    "#game maker consider https://gym.openai.com/envs\n",
    "def make_env():\n",
    "    env = gym.make(\"MsPacmanDeterministic-v0\")\n",
    "    return PreprocessImage(env,height=105,width=80,\n",
    "                           grayscale=True,\n",
    "                           crop=lambda img:img[:-25])\n",
    "\n",
    "#spawn game instance\n",
    "env = make_env()\n",
    "observation_shape = env.observation_space.shape\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "obs = env.step(0)[0]\n",
    "\n",
    "plt.imshow(obs[0],interpolation='none',cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic agent setup\n",
    "Here we define a simple agent that maps game images into Qvalues using simple convolutional neural network.\n",
    "\n",
    "![scheme](https://s18.postimg.org/gbmsq6gmx/dqn_scheme.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import theano, lasagne\n",
    "from lasagne.layers import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#observation\n",
    "observation_layer = InputLayer((None,)+observation_shape,)\n",
    "\n",
    "#4-tick window over images\n",
    "from agentnet.memory import WindowAugmentation, LSTMCell\n",
    "\n",
    "prev_wnd = InputLayer((None,4)+observation_shape)\n",
    "new_wnd = WindowAugmentation(observation_layer,prev_wnd)\n",
    "        \n",
    "#reshape to (frame, h,w). If you don't use grayscale, 4 should become 12.\n",
    "wnd_reshape = reshape(new_wnd, (-1,4*observation_shape[0])+observation_shape[1:])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((None, 4, 1, 105, 80), (None, 4, 1, 105, 80))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prev_wnd.output_shape,new_wnd.output_shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from lasagne.nonlinearities import elu,tanh,softmax\n",
    "\n",
    "#network body\n",
    "conv0 = Conv2DLayer(wnd_reshape,32,3,stride=2,nonlinearity=elu)\n",
    "conv1 = Conv2DLayer(conv0,32,3,stride=2,nonlinearity=elu)\n",
    "conv2 = Conv2DLayer(conv1,64,3,stride=2,nonlinearity=elu)\n",
    "conv3 = Conv2DLayer(conv2,128,3,stride=2,nonlinearity=elu)\n",
    "        \n",
    "dense = DenseLayer(dropout(conv3,0.1),512,nonlinearity=tanh,name='dense')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "#baseline for all qvalues\n",
    "qvalues_layer = DenseLayer(dense,n_actions,nonlinearity=None,name='qval')\n",
    "        \n",
    "#sample actions proportionally to policy_layer\n",
    "from agentnet.resolver import EpsilonGreedyResolver\n",
    "action_layer = EpsilonGreedyResolver(qvalues_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.target_network import TargetNetwork\n",
    "targetnet = TargetNetwork(qvalues_layer,conv2)\n",
    "qvalues_old = targetnet.output_layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Finally, agent\n",
    "We declare that this network is and MDP agent with such and such inputs, states and outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from agentnet.agent import Agent\n",
    "#all together\n",
    "agent = Agent(observation_layers=observation_layer,\n",
    "              policy_estimators=(qvalues_layer,qvalues_old),\n",
    "              agent_states={new_wnd:prev_wnd},\n",
    "              action_layers=action_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[W, b, W, b, W, b, W, b, dense.W, dense.b, qval.W, qval.b]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Since it's a single lasagne network, one can get it's weights, output, etc\n",
    "weights = lasagne.layers.get_all_params(action_layer,trainable=True)\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and manage a pool of atari sessions to play with\n",
    "\n",
    "* To make training more stable, we shall have an entire batch of game sessions each happening independent of others\n",
    "* Why several parallel agents help training: http://arxiv.org/pdf/1602.01783v1.pdf\n",
    "* Alternative approach: store more sessions: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-02-22 03:16:52,267] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,299] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,326] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,350] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,374] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,398] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,424] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,448] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,472] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:16:52,496] Making new env: MsPacmanDeterministic-v0\n"
     ]
    }
   ],
   "source": [
    "from agentnet.experiments.openai_gym.pool import EnvPool\n",
    "\n",
    "pool = EnvPool(agent,make_env, N_AGENTS) #may need to adjust\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "actions:\n",
      "[5 2 2 2 8 2 2 2 2 5]\n",
      "rewards\n",
      "[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      "CPU times: user 228 ms, sys: 16 ms, total: 244 ms\n",
      "Wall time: 241 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#interact for 7 ticks\n",
    "_,action_log,reward_log,_,_,_  = pool.interact(10)\n",
    "\n",
    "print('actions:')\n",
    "print(action_log[0])\n",
    "print(\"rewards\")\n",
    "print(reward_log[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#load first sessions (this function calls interact and remembers sessions)\n",
    "pool.update(SEQ_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-learning\n",
    "* An agent has a method that produces symbolic environment interaction sessions\n",
    "* Such sessions are in sequences of observations, agent memory, actions, q-values,etc\n",
    "  * one has to pre-define maximum session length.\n",
    "\n",
    "* SessionPool also stores rewards (Q-learning objective)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/py27/lib/python2.7/site-packages/agentnet/utils/logging.py:14: UserWarning: [Verbose>=1] optimize_experience_replay is deprecated and will be removed in 1.0.2. Use experience_replay parameter.\n",
      "  default_warn(\"[Verbose>=%s] %s\"%(verbosity_level,message),**kwargs)\n"
     ]
    }
   ],
   "source": [
    "#get agent's Qvalues obtained via experience replay\n",
    "replay = pool.experience_replay\n",
    "\n",
    "_,_,_,_,(qvalues_seq,old_qvalues_seq) = agent.get_sessions(\n",
    "    replay,\n",
    "    session_length=SEQ_LENGTH,\n",
    "    optimize_experience_replay=True,\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#get reference Qvalues according to Qlearning algorithm\n",
    "from agentnet.learning import qlearning\n",
    "\n",
    "#crop rewards to [-1,+1] to avoid explosion.\n",
    "rewards = replay.rewards/10.\n",
    "\n",
    "#loss for Qlearning = \n",
    "#(Q(s,a) - (r+ gamma*r' + gamma^2*r'' + ...  +gamma^10*Q(s_{t+10},a_max)))^2\n",
    "elwise_mse_loss = qlearning.get_elementwise_objective(qvalues_seq,\n",
    "                                                      replay.actions[0],\n",
    "                                                      rewards,\n",
    "                                                      replay.is_alive,\n",
    "                                                      qvalues_target=old_qvalues_seq,\n",
    "                                                      gamma_or_gammas=0.99,\n",
    "                                                      n_steps=10)\n",
    "\n",
    "#mean over all batches and time ticks\n",
    "loss = elwise_mse_loss.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Compute weight updates\n",
    "updates = lasagne.updates.adam(loss,weights,learning_rate=1e-4)\n",
    "\n",
    "#compile train function\n",
    "train_step = theano.function([],loss,updates=updates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-02-22 03:17:32,373] Making new env: MsPacmanDeterministic-v0\n",
      "[2017-02-22 03:17:32,402] DEPRECATION WARNING: env.spec.timestep_limit has been deprecated. Replace your call to `env.spec.timestep_limit` with `env.spec.tags.get('wrapper_config.TimeLimit.max_episode_steps')`. This change was made 12/28/2016 and is included in version 0.7.0\n",
      "[2017-02-22 03:17:32,404] Clearing 2 monitor files from previous run (because force=True was provided)\n",
      "[2017-02-22 03:17:32,419] Starting new video recorder writing to /home/hedgedir/rl_projects/records/openaigym.video.0.34712.video000000.mp4\n",
      "[2017-02-22 03:17:34,054] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/hedgedir/rl_projects/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 333 timesteps with reward=190.0\n"
     ]
    }
   ],
   "source": [
    "action_layer.epsilon.set_value(0)\n",
    "\n",
    "untrained_reward = np.mean(pool.evaluate(save_path=\"./records\",\n",
    "                                         record_video=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"./records/openaigym.video.0.34712.video000000.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./records/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./records/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#starting epoch\n",
    "epoch_counter = 1\n",
    "\n",
    "#full game rewards\n",
    "rewards = {}\n",
    "loss,reward_per_tick,reward =0,0,0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAFkCAYAAAAQQyCBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xl8VOX1x/HPAQREZbOyuKMgggpKFFQWFVTctWKRoKJg\n625tWrdarQjW1hU3bN3FLWpBBTcQN36ICkoQEEFUNhVBkFVRtjy/P86kmQxJyDKTO8l836/XvCZz\n7zN3zlwjHJ7lPBZCQERERCRKtaIOQEREREQJiYiIiEROCYmIiIhETgmJiIiIRE4JiYiIiEROCYmI\niIhETgmJiIiIRE4JiYiIiEROCYmIiIhETgmJiIiIRK7cCYmZdTezMWb2nZnlm9kpxbRpZ2ajzWyV\nmf1kZpPNbNe48/XMbLiZLTeztWY20syaJVyjiZk9Y2arzWylmT1iZttV7GuKiIhIOqtID8l2wKfA\nJcAWG+GY2d7AROBzoAdwADAU+DWu2d3AiUCfWJudgVEJl3oWaAf0irXtATxYgXhFREQkzVllNtcz\ns3zgtBDCmLhjucCGEMK5JbynIbAM6BdCeCl2rC0wGzg0hDDFzNoBs4CsEMK0WJvewGvAriGEJRUO\nWkRERNJOUueQmJnhvRlfmtlYM1tqZh+Z2alxzbKAOsDbBQdCCF8Ai4DDYocOBVYWJCMxb+E9Ml2S\nGbOIiIhEr06Sr9cM2B64BvgbcDVwPPCimR0ZQpgItMB7UNYkvHdp7Byx5x/iT4YQNpvZirg2RZjZ\njkBvYAFFh4dERESkdPWBPYFxIYQfowgg2QlJQY/LyyGEe2M/zzCzw4GL8LklqdIbeCaF1xcREanp\nzsLncFa5ZCcky4FN+HyQeLOBrrGflwB1zaxhQi9J89i5gjaJq25qA03j2iRaAPD000/Trl27isYv\n5ZSTk8OwYcOiDiOj6J5XPd3zqqd7XrVmz57N2WefDbG/S6OQ1IQkhLDRzD4G2iac2gdYGPt5Kp60\n9ALiJ7XuDnwYa/Mh0NjMDoqbR9ILMGByCR//K0C7du3o1KlTEr6NlEWjRo10v6uY7nnV0z2verrn\nkYlsykO5E5JYLZDWeHIAsJeZdQRWhBC+AW4HnjOzicC7+BySk4AjAEIIa8zsUeAuM1sJrAXuBSaF\nEKbE2swxs3HAw2Z2MVAXuA/I1QobERGRmqciPSQH44lGiD3ujB0fAQwKIbxsZhcB1wH3AF8Ap4cQ\nPoy7Rg6wGRgJ1APGApcmfE5/4H58dU1+rO0VFYhXRERE0ly5E5IQwgS2slw4hPAE8EQp59cDl8ce\nJbVZBZxd3vhERESk+tFeNlIp2dnZUYeQcXTPq57uedXTPc88larUmk7MrBMwderUqZoIJSIiUg55\neXlkZWWBV0jPiyIG9ZCIiIhI5JSQiIiISOSUkIiIiEjklJCIiIhI5JSQiIiISOSUkIiIiEjklJCI\niIhI5JSQiIiISOSUkIiIiEjklJCIiIhI5Cqy26+IiIjEufVW+PRTWLcO/vMfaNky6oiqH/WQiIiI\nVMLixXDttTB3LowbB08/HXVE1ZMSEhERkUr4+GN/fvllOP54eOmlaOOprpSQiIiIVMKUKdCiBey6\nK/z2t/Dhh/D991FHVf0oIREREamEKVOgc2cwg5NOgtq1vbdEykcJiYiISAXl5/uQTefO/rppUzjq\nKA3bVIQSEhERkXIIAW66Ce6/H776ClavLkxIwIdt3n0XVq2KLsbqSAmJiIhIOfzznzB4MOTkwGOP\n+bGDDy48f8IJsGkTvPdeFNFVX0pIREREymjUKPjb3+C662Dvvb3+yD77QJMmhW323BP22gveeSey\nMKslJSQiIiJl9Oij0KMH3HwzPPCAHzvkkC3b9eoFb79dtbFVd0pIREREyiA/Hz76CI4+2lfU9OwJ\n99wDl1++ZduePeHzz7X8tzyUkIiIiJTBF1/AypVw2GGFx/74R+jSZcu2PXv6s4Ztyk4JiYiISBl8\n8AHUqlV0RU1JmjWDAw7QsE15KCEREREpgw8+8CSjYcOytS+YRxJCauOqKZSQiIiIlMEHH8Dhh5e9\nfc+esGgRzJuXuphqEiUkIiIiW7FiBcyZU76E5IgjvIy8hm3KptwJiZl1N7MxZvadmeWb2SmltP1P\nrM0fE47XM7PhZrbczNaa2Ugza5bQpomZPWNmq81spZk9YmbblTdeERGRyvroI3+On9C6NQ0b+pJg\nJSRlU5Eeku2AT4FLgBJHxszst0AX4LtiTt8NnAj0AXoAOwOjEto8C7QDesXa9gAerEC8IiIilTJ2\nLOy8sxc8K49evXylTX5+auKqScqdkIQQxoYQ/h5CGA1YcW3MbBfgHqA/sCnhXENgEJATQpgQQpgG\nDAS6mlnnWJt2QG/g/BDCJyGED4DLgX5m1qK8MYuIiFTU5s3wwgvQt6/XHymPnj1h+XL47LPUxFaT\nJH0OiZkZ8CRwWwhhdjFNsoA6wP86sUIIXwCLgILOsEOBlbFkpcBbeI9MMSu+RUREUmPCBFi6FPr1\nK/97Dz8c6tfXsE1ZpGJS67XAhhDC/SWcbxE7vybh+NLYuYI2P8SfDCFsBlbEtREREUm5556DVq3K\nVn8kUf360LWrEpKyqJPMi5lZFvBH4KBkXrc8cnJyaNSoUZFj2dnZZGdnRxSRiIhUB9dc43M+jj22\n8NiGDTByJFx0UfmHawp07w733+/1SCp6jWTKzc0lNze3yLHVq1dHFE2hpCYkQDdgJ+AbK7zrtYG7\nzOxPIYS9gCVAXTNrmNBL0jx2jthz4qqb2kDTuDbFGjZsGJ06dar0FxERkcwRAtx3H8yfXzQhmTDB\ny8WfeWbFr92xo88jWbwYdtml8rFWVnH/SM/LyyMrKyuiiFyyh2yeBDoAHeMei4Hb8EmqAFPxia69\nCt5kZm2B3YEPY4c+BBqbWXxPSy98Eu3kJMcsIiIZbulS+OUXmDSpaGXVTz+F7bf3Cq0VdeCB/jx9\neuVirOnK3UMSqwXSmsIVNnuZWUdgRQjhG2BlQvuNwJIQwpcAIYQ1ZvYo3muyElgL3AtMCiFMibWZ\nY2bjgIfN7GKgLnAfkBtCKLWHREREpLwKqqkuXgwLF8Kee/rrWbOgfXvfw6ai9tgDGjXyhOSEEyod\nao1VkVt8MDAN7+kIwJ1AHnBTCe2Lq1WSA7wKjATew3tR+iS06Q/MwVfXvAr8H3BhBeIVEREp1fz5\nhT9/8EHhz599BvvvX7lrm0GHDuoh2Zpy95CEECZQjkQmNm8k8dh6vK7I5aW8bxVwdnnjExERKa95\n82CnnaBpUx+26d/fi5nNnu0/V1bHjvDWW5W/Tk2mvWxERCTjzZ/vVVi7dvWEBGDBAli3Dvbbr/LX\nP/BAmDvXr5fo++9h0CDYcUf45pvKf1Z1pYREREQy3rx5Xmvk8MNh5kxYs6awumplh2zAe0jy8wuv\nmZ8P//439O7tidCYMZ6sPPts5T+rulJCIiIiGS++hyQ/3+eRzJrlk1F33rny199vP58YO306/PQT\nnH46XHqpzy+55Rb46is49dTMTkiSXYdERESkWtmwwYdKWrWCtm1h3329JkmTJp5IJKOY2bbb+rVv\nuQVuuAF+/tl7RU46qbBN//6elCRjIm11pB4SERHJaIsWee2Rvfby5GPwYHj9dXjtteQmBgMH+hLg\nAQPg44+LJiMAxx3nSVBCEdWtys+HvLzkxRkVJSQiIlLtbNwIX3+dnGsV1CDZK7Ym9He/80Joq1Yl\nZ0Jrgauugvfeg9tu816YRHXrwhln+LBNKK5gRjG++AKOOMKHmn74Yevt05kSEhERqVbWrYNTTvEh\nkIULK3+9efOgdm3YdVd/XasWDBniP3fsWPnrl8cZZ/jqnpkzt9527lxfvbNkCYwdC82abf096UwJ\niYiIVBvr13u10//7P08iXnyx8tecP9+HUurEzao89VT48EPo0aPy1y+PI46ABg18yGhrXnvNn6dN\n8/dVd0pIRESk2pgwwR9jxviS2VGjKn/NgiW/8czg0EOrfnfeevXg6KMLk43STJwIXbr4Xjs1gRIS\nERGpNqZP97+AjzoK+vTxImbffVe5a86dC61bJye+ZDjxRF92vHJlyW1CgPffh+7dqy6uVFNCIiIi\n1cb06T7htFYtn0dSpw689FLFr7dpk08MTebk1co64QRfOTNunL9evx4uvhief76wzRdfwLJlSkhE\nREQiMX164UTTJk18eOO//6349ebN87/w27dPTnzJsOuuvhnfs88WFkz7z3/gL3/xmingwzW1asFh\nh0UbazIpIRERkWph/XqYM6foypezzvIJrmVZlVKczz/353TqIQHf2+aVV6BNG08+7r/fh6YKekkm\nToSDDoIddog2zmRSQiIiItXC55/7EEt8QnLmmbD77nDrrRW75qxZ3tPSvHlyYkyWK67w6rGvvOJF\nzy69FI4/Hu64w+ePTJxY9SuAUk0JiYiIVAuffuqrXg44oPDYNtvAlVfCc8/58t3ymjUreeXhk23X\nXb2aa9u2/vrKK2HGDNhpJ69VcswxkYaXdEpIRESkWpg+Hfbee8tlruefD02beu9BeX3+eXrNHynN\nUUd5tdcLLoC33/Yek5pEm+uJiEi1ED+hNV6DBnDuuT4J9P77y97bsXmzz0kZNCi5caaKmZedr6nU\nQyIiImkvhJITEvBKpYsXl6+UfDqusMlkSkhERCTtzZzphcIOOaT484cf7s8TJ5b9mrNm+XO6rbDJ\nVEpIREQk7T3zjM8T6dmz+PNNm3pi8f77Zb/mrFnQuDG0aJGcGKVylJCIiEhay8+H3Fzo2xfq1i25\nXbdu5UtIPv3UV+yk4wqbTKSERERE0trEiV6T46yzSm/XrZuvmvnxx61fMwTfB6dr1+TEKJWnhERE\nRNLaM8/AHnsUzhMpSbdu/vzBB1u/5vz58P33SkjSiRISERFJW+vX+141/fv73i2l2WMP2GUXePBB\n71EpsHkzLF1atO2kSf68tSRHqo4SEhERSSt5efDOO/7z2LGwatXWh2vA54IMGeLJRqtWsM8+cOCB\n0KiRT1x9663Ctu+/78t9mzZNzXeQ8lNCIiIiaeXGG+H0032Z7zPPeO2Rsi7NHTTIe0eGD4fTTvPd\ncG+8EbKy4G9/87kj4ElLwRCPpAdVahURkbQydy6sXg033OCbyw0ZUr73b789XHhh0WOdOsHRR8Or\nr3oiMmsWXH118mKWylNCIiIiaWPjRq+guttu3sthBtnZlb9uz55ezfWqq3xPGFAPSbop95CNmXU3\nszFm9p2Z5ZvZKXHn6pjZrWY2w8x+irUZYWYtE65Rz8yGm9lyM1trZiPNrFlCmyZm9oyZrTazlWb2\niJltV/GvKiIi6W7BAti0Ce680+d+HHGE73pbWWZ+TbPCYaBWrSp/XUmeivSQbAd8CjwKvJhwrgFw\nIHATMANoAtwLjAY6x7W7Gzge6AOsAYYDo4DucW2eBZoDvYC6wBPAg8DZFYhZRESqgblz/fnQQ+H1\n15M76TQrC2bPTt71JLnKnZCEEMYCYwHMita3CyGsAXrHHzOzy4DJZrZrCOFbM2sIDAL6hRAmxNoM\nBGabWecQwhQzaxe7TlYIYVqszeXAa2Z2ZQhhSbm/qYiIpL25c2HbbX357m67RR2NVKWqWGXTGAjA\nqtjrLDwRerugQQjhC2ARcFjs0KHAyoJkJOat2HW6pDpgERGJxty50KbN1muOSM2T0v/kZlYP+Bfw\nbAjhp9jhFsCGWG9KvKWxcwVtfog/GULYDKyIayMiIjXMl196/RDJPClbZWNmdYD/4r0al6TqcxLl\n5OTQqFGjIseys7PJTsY0bRERSam5c+Gcc6KOombLzc0lNze3yLHVq1dHFE2hlCQkccnIbkDPuN4R\ngCVAXTNrmNBL0jx2rqBN4qqb2kDTuDbFGjZsGJ06darkNxARkaq2bp0XNVMPSWoV94/0vLw8srKy\nIorIJX3IJi4Z2QvoFUJYmdBkKrAJXz1T8J62wO7Ah7FDHwKNzeyguPf1AgyYnOyYRUQkel995c9t\n2kQbh0Sj3D0ksVogrfHkAGAvM+uIz+/4Hl++eyBwErCNmTWPtVsRQtgYQlhjZo8Cd5nZSmAtvjR4\nUghhCkAIYY6ZjQMeNrOL8WW/9wG5WmEjIlIzFSz5VQ9JZqrIkM3BwLv43JAA3Bk7PgKvP3Jy7Pin\nseMWe30U8H+xYznAZmAkUA9fRnxpwuf0B+7HV9fkx9peUYF4RUTSxs8/w08/QfPmW2+baWbMgCZN\nYMcdo45EolCROiQTKH2oZ6vDQCGE9cDlsUdJbVahImgiUkOsWuUbvc2Z46/z8uCgg0p/T00wcyZ8\n/z0cc4xXSS3J5s0wYgScemrp7aTm0kpvEZEUeeghKFjMcM89XhZ9xAho0ADeeivS0KrMOedA796+\nl0zBkExxXn0VFi2CSxP7yiVjKCEREUmBDRt8I7cBAzz5uPtu34F2wADvKZk4MeoIU2/aNJg+Ha69\nFhYu9O8eQuH5NWvgssu8t+j++6FLFzj44OjilWgpIRERSYEJE/wv3N12g+OPh19+Kdzuvnt3eP99\nyM+PNsZUe/xxaNEChg6Ff/8bJk8u2jP09tu+o+8hh/hx9Y5kNiUkIiIp8PLLsOeeMH68D9FceCHs\nvLOf694dVq6EWbMiDTGl1q/3XXXPOQfq1IFjj/XEY8iQwl6SWbOgcWO49VY48UT43e+ijVmipYRE\nRCTJQoDRo32C5t57w7x5cNddhecPPdT/kq7JwzajR8OKFXDeef7aDG64wXuG/i+23vLzz2G//eDK\nK30OSf36kYUraUAJiYhIkk2dCt99B6ed5q933BFq1y4836CBz5WoSQlJfr4Pwaxf7/vRXHKJ94q0\nb1/Y5qSTvNdo1Ch/PWuWJyQioIRERCTpRo+Gpk2hW7eS23Tv7j0F8ZM8q7Nnn4Wjj/YeoWOOgd/8\npnCFUQEz/96TJsGmTb4EWgmJFFBCIiKSZJ9+WjgsU5KePWHxYl+FAp6YlDU5WbAAHnww+mTmhRfg\n22/95wcfhM6d4aijfOjljTc8KUvUtat/5xkzfCWSEhIpoIRERCTJ5s+HVq1Kb9Orl1drHTHCX19+\nObRt64XEvv0Wzj/fJ8QW5/HH4aKL4IEHkht3efz8M2Rnw5lneszvvw9/+Qs89ZT3fJT0/bt29SJo\njz/ur5WQSAElJCIiSRRC2RKSbbaBs8+Gp5/2uRT//jcsX+41Stq3h8ceg+uuK/69s2dDrVqQkwMf\nfZT871AWs2b5vJEPPoCTT4ZmzQrnzJSmfXtfWfPUU96DohL6UkAJiYhIKULwf9GX1bJlsG7d1hMS\n8BUoy5f7X+gtW/put/37+1LZp56CTz7xR6I5c2DgQJ8YO3BgNEM306d7UnTRRV70bOBAqFt36++r\nVcuTrtWrPTlRmXgpoIRERKQUf/kL9OlT9vbz5/tzWRKS/ff3pGL+fLj+eu8xeOghLxaWne1F1R58\nsOh7Nm/2EuwdOnjBsTlzvJeiqs2Y4bvy3nmnx/7nP5f9vV27+rOGaySeEhIRkVLMnQvjxvkEzLIo\nT0ICPuxy6KEwaFDR47VrwwUX+OqVRx7xv/R/+sl7I9avh3339Qmke+xROB+jKk2fDh07+hLmoUN9\nyKaslJBIcZSQiIiUYsUK+PXX4odOijN/vs+RaNy4bO3794cPPyx+uOP8871H5IIL4B//8OqvBbsF\n77uvD3+cdx48/7xPMq0qIXgPSYcOFXv/oYf6MNVxxyU3LqnelJCIiJRi5Up/LmsRs7JMaC2rli29\nyNiqVd4bMX68JyTbbQe77uptzj3Xe05GjkzOZ5bFokU+B6Rjx4q9v359GDMG2rRJblxSvSkhEREp\nRUFCUlDufGuSmZCAzyNp2NCLjY0f7yts2rb13hHwzzriCK8JUlUKaqdUtIdEpDhKSEREShCCJyQt\nW3p10bKstkl2QlLgmGPg++/hlVd8uCbe0UeXPb5kmDEDmjQp7KURSQYlJCIiJVi3zieznnKKD1HM\nnFl6+82bfTgjFQlJ9+5Qrx4sXbplQtKtm8f32WfJ/9wCS5d6ddnWrX1JcocOWrIryaWERESkBAXD\nNb17+6TTrc0jWbwYNm5MTUKy7baelMCWCUmXLl5oLVWb9c2eDYcc4s+HHw7ffFP6Pj0iFaGERESk\nBAUJyc47+1/IW5tHUt4lv+V1zDH+nJiQbLtt6nYPDgH+8Adf3vvxx/Dkk77y6Kabkv9ZktlK2fpJ\nRCSzrVjhz02aQI8eXs49hJKHKgoSkj33TE0855wD330H7dptea57dy9DX1p8FfHWWz4/5bXXCueM\n1K+fvOuLFFAPiYhICQp6SJo08b/wly71Zbgl+eQT2H1377FIhZYt4Z57it9FuHt3HzIqSIoqY+lS\nL9Q2fjzceKMPCR1/fOWvK1Ia9ZCIiJSgICFp3NjnTtSq5cMi++yzZdv1672q6u9/X7UxFjj8cH+e\nOBH22qty13rqKa/+WlABduxYTWCV1FMPiYhICVasgB128AmjjRrBgQeWPI/klVe8/cCBVRtjgaZN\nvQLqPffApk2Vu9bLL/vKotdfh9tvh2OPTU6MIqVRQiIiUoKVK324pkD37iVPHH38cU8IEiecVqV7\n7/WiZXfdVfFrLF3qm/X99rc+THPlleodkaqhhEREpASJCUmPHj5H49tvi7abNcuHNaLqHSlwyCHw\npz/5vI+vv67YNV55xROQk05KbmwiW6OERESkBCtX+lBIgYLaGyec4CtpDjgAsrJg//2heXPo2zeS\nMIsYMsT3unnssYq9/+WXvSfoN79JblwiW1PuhMTMupvZGDP7zszyzeyUYtoMMbPFZrbOzMabWeuE\n8/XMbLiZLTeztWY20syaJbRpYmbPmNlqM1tpZo+Y2Xbl/4oiIhWzYkXRHpJmzeDii31TuH79fA+Z\n9u3hmWfgq6/KvsNvKm23nRdye+MNf71pk/febNzoS4LvuQeGDi3+vT/95Mt8Tz216uIVKVCRVTbb\nAZ8CjwIvJp40s2uAy4ABwALgZmCcmbULIWyINbsbOB7oA6wBhgOjgO5xl3oWaA70AuoCTwAPAmdX\nIGYRkXJbuXLL/VoeeCCaWMrjuON8xc+SJTBqFFx2mdcuadPGd9mtUwcuvbRo7w9478j69XD66dHE\nLZmt3D0kIYSxIYS/hxBGA8VNdboCGBpCeDWE8BmemOwMnAZgZg2BQUBOCGFCCGEaMBDoamadY23a\nAb2B80MIn4QQPgAuB/qZWYvyf00RkfJLHLKpLnr39uc33oD77oMjj/QhmHfegeHDfc+dV17xHpO+\nff0YeE9Pt26wxx6RhS4ZLKl1SMysFdACeLvgWAhhjZlNBg4DXgAOjn1ufJsvzGxRrM0U4FBgZSxZ\nKfAWEIAuwOhkxi0iUpzEIZvqolkzLyV/002wcCE89JDPC/n1Vy/a9uyz3nOyyy7w3//68t4ePbwQ\n2v33Rx29ZKpkT2ptgScNSxOOL42dAx+G2RBCWFNKmxbAD/EnQwibgRVxbUREUiYEWLWqeiYk4Et2\nFy6Ejh09GTErrCDbpw+8+SbccINPzN1mG29vBmecEW3ckrm0ykZEpBhr1/rQRnVNSE480Z8vv3zL\nOiKnn+5zRT76yJcIX3ed75Fz3HFaXSPRSXbp+CX4vJLmFO0laQ5Mi2tT18waJvSSNI+dK2iTuOqm\nNtA0rk2xcnJyaNSoUZFj2dnZZGdnl++biEhGK9hYrzrOIQHo3BnefttXAiXaYw8f0lmzBk47zVfg\nvPGGJy9S8+Xm5pKbm1vk2OrVqyOKppCFECr+ZrN84LQQwpi4Y4uB20MIw2KvG+LJyYAQwn9jr5cB\n/UIIL8XatAVmA4eGEKaY2b7ALODggnkkZnYs8Dqwawhhi6TEzDoBU6dOnUqnTp0q/J1ERACmTYNO\nneDjj/0v75rmyy+956R16623lZovLy+PrKwsgKwQQl4UMZS7hyRWC6Q1hSts9jKzjsCKEMI3+JLe\n683sK3zZ71DgW2ITUWOTXB8F7jKzlcBa4F5gUghhSqzNHDMbBzxsZhfjy37vA3KLS0ZERJItfqff\nmqhNm6gjECmqIkM2BwPv4pNXA3Bn7PgIYFAI4TYza4DXDGkMTASOj6tBApADbAZGAvWAscClCZ/T\nH7gfX12TH2t7RQXiFREpt+o+ZCNS3ZQ7IQkhTGArk2FDCIOBwaWcX4/XFSlxxDKEsAoVQRORiKxc\n6UMaCVPSRCRFtMpGRKQYK1d6MlJLf0qKVAn9ryYiUoxPPoG99446CpHMoYRERCTBTz/Bq6/C734X\ndSQimUMJiYhIgldegV9+gTPPjDoSkcyhhEREJMFzz0GXLrDnnlFHIpI5lJCIiMRZtQrGjoV+/aKO\nRCSzKCEREYlZvBj69/c9bDR/RKRqJXsvGxGRamnxYth/f6hXD15+GXbZJeqIRDKLEhIREWDKFK89\nMn++5o6IREFDNiIiwMKFUL++74QrIlVPCYmICLBggScjZlttKiIpoIRERATvIVHviEh0lJCIiKCE\nRCRqSkhERPAhG01mFYmOEhIRyXg//QQrVqiHRCRKSkhEJOMtXOjP6iERiY4SEhHJeAsW+LN6SESi\no4RERDLewoVQpw60bBl1JCKZSwmJiGS8hQth992hdu2oIxHJXEpIRCTjFRRFE5HoKCERkYynGiQi\n0VNCIiIZZ+FCmDu36GutsBGJlnb7FZGM89vfwrRp0Lq194wsWaIeEpGoKSERkYyyaRPMmgXnnQcN\nGsDy5dCvHxx9dNSRiWQ2JSQiklEWLIANG+Css5SEiKQTzSERkYwye7Y/t2sXbRwiUpQSEhHJKHPm\nwA47wM47Rx2JiMRTQiIiGWX2bNh3XzCLOhIRiZf0hMTMapnZUDObZ2brzOwrM7u+mHZDzGxxrM14\nM2udcL6emQ03s+VmttbMRppZs2THKyKZpSAhEZH0kooekmuBC4FLgH2Bq4GrzeyyggZmdg1wGXAB\n0Bn4GRhnZnXjrnM3cCLQB+gB7AyMSkG8IpIhQvAhG80fEUk/qUhIDgNGhxDGhhAWhRBeBN7EE48C\nVwBDQwiajS8wAAAgAElEQVSvhhA+AwbgCcdpAGbWEBgE5IQQJoQQpgEDga5mFn8dEZEtfP019O4N\nP/5Y9PjSpbBqlXpIRNJRKhKSD4BeZtYGwMw6Al2B12OvWwEtgLcL3hBCWANMxpMZgIPxJcnxbb4A\nFsW1EREp1i23wJtvQm5u0eNz5vizEhKR9JOKhORfwPPAHDPbAEwF7g4hPBc73wIIwNKE9y2NnQNo\nDmyIJSoltRER2cK338JTT3nRs6efLnpu9myoU8crtIpIeklFYbQzgf5AP+Bz4EDgHjNbHEJ4KgWf\nV0ROTg6NGjUqciw7O5vs7OxUf7SIpIE774Ttt4c77oDzz4cvv4Q2bfzcnDmw996wzTbRxigSpdzc\nXHITug9Xr14dUTSFLISQ3AuaLQL+GUL4d9yxvwFnhRDax4ZsvgYODCHMiGvzHjAthJBjZkcBbwFN\n4ntJzGwBMCyEcE8xn9sJmDp16lQ6deqU1O8kItXD6tVeX+Qvf4G//hVatICcHBg82Ce07r8/dOrk\nPSgiUigvL4+srCyArBBCXhQxpGLIpgGwOeFYfsFnhRDmA0uAXgUnY5NYu+DzT8CHeTYltGkL7A58\nmIKYRaQGePNNWLcOfv972HZbOOMMTz42b/bN9D7/HPr3jzpKESlOKoZsXgGuN7NvgVlAJyAHeCSu\nzd2xNl8BC4ChwLfAaPBJrmb2KHCXma0E1gL3ApNCCFNSELOIVDMLF8I330DnzlA3VjDg9de9F2T3\n3f31hRfCY4/BiBHw2WfQrBkcc0x0MYtIyVKRkFyGJxjDgWbAYuDfsWMAhBBuM7MGwINAY2AicHwI\nYUPcdXLwnpaRQD1gLHBpCuIVkWomPx9OOsmTjAYN4K674A9/8ITkvPMK23XuDNnZcN11/jo72ye1\nikj6Sfr/miGEn4E/xx6ltRsMDC7l/Hrg8thDROR/XnzRk5ERIzwJufJK2GUX+OEHOPHEom3/9S9o\n2xZ+/RXOOSeaeEVk67SXjYhUC9OnQ9++nogMHQpHHw0DBsADD3ivxznnQKNGcFhCpaLdd4ebboIj\njvAJrSKSnpSQiEi18Npr8N//QseOMGMG3HijH2/aFG64wSuw9u5d/JLeq6+G997Thnoi6UyjqSJS\nLSxaBPvtB6ed5slHt26F5y69FN54AwYOjC4+EakcJSQiUi0sXOgFzm6+ectz9erB+PFVH5OIJI+G\nbESkWli0qHA5r4jUPEpIRCTthaCERKSmU0IiImlv1Sr46SclJCI1mRISEUl7Cxf68x57RBuHiKSO\nEhIRSXuLFvmzekhEai4lJCKS9hYt8v1qmjWLOhIRSRUlJCKS9hYtgt12g1r6E0ukxtL/3iKSltav\n94Jn33/vc0g0f0SkZlNhNBFJS5Mn+z41jRp5D8m++0YdkYikknpIRCQtffyxPz/2GMybpwmtIjWd\nekhEJC19/DHssgt8952/VkIiUrOph0RE0tLHH8PvfgeHHeavNYdEpGZTQiIiaWf5ch+mOeQQuOgi\nP7bXXtHGJCKppSEbEUk7n3ziz4ccAnvvDW3bKiERqenUQyIiKfXJJ76Etyy++QZWr/bhmsaNoXVr\nrz3SpUtqYxSR6CkhEZGUeeQR7+W46qrCYyHArbfCccfBhg2Fx9etg4MPhgMPhNGj/Wezqo9ZRKKh\nhEREUuKll+DCC3245d//hq++grVr4Ywz4NprYdw4ePzxwvYjRvjckQYNYOpUT2REJHMoIRGRpAsB\nLr4YTjnFh2yaN4dLLvGhl/HjPVnJzoabb/bhnM2b4c474fTT4aOP4K9/hfPPj/pbiEhV0qRWEUm6\n2bNh6VIv/b799jB0KAwa5NVWp0zx5333hf32g1tugR13hK+/hmefhR128GMiklmUkIhI0r33Hmyz\nTWENkQEDYLvtfN5Iw4Z+bN99YeBAGDLEXx91FHTuHEm4IpIGlJCISNK9954nF9tt569r14a+fbds\n99BDkJMD+fnQqlWVhigiaUZzSEQkKf75Tzj5ZJ8P8t57cOSRW39PrVo+bHPAAT60IyKZSz0kIlJp\n334LN93kE1SvvBKWLStbQiIiUkAJiYhU2j/+4T0cJ50Ed99ddP6IiEhZpGTIxsx2NrOnzGy5ma0z\ns+lm1imhzRAzWxw7P97MWiecr2dmw2PXWGtmI82sWSriFZGK+e47L2L2yCNwzTUwfDg0auQ1RArm\nj4iIlEXSe0jMrDEwCXgb6A0sB9oAK+PaXANcBgwAFgA3A+PMrF0IoaB2493A8UAfYA0wHBgFdE92\nzCJSPpMnw+DBMHasv27b1pf4NmgAY8b4s4hIeaRiyOZaYFEI4fdxxxYmtLkCGBpCeBXAzAYAS4HT\ngBfMrCEwCOgXQpgQazMQmG1mnUMIU1IQt4iUYN062HZbL+X+7rvQqxe0awdPPgldu8Iee/hKGoAe\nPaKNVUSqp1QM2ZwMfGJmL5jZUjPLM7P/JSdm1gpogfegABBCWANMBgpGnQ/Gk6X4Nl8Ai+LaiEgV\nWL/eV8Kceir8/LNXYO3WDWbMgHPO8V14C5IREZGKSkUPyV7AxcCdwD+AzsC9ZrY+hPAUnowEvEck\n3tLYOYDmwIZYolJSGxGpAs88AwsW+EqaDh1g0SIYNUpJiIgkVyoSklrAlBDCDbHX081sf+Ai4KkU\nfF4ROTk5NGrUqMix7OxssrOzU/3RIjVOfj7ccYfvSZOdDf37w9VXe4+JiFRPubm55ObmFjm2evXq\niKIplIqE5HtgdsKx2cDpsZ+XAIb3gsT3kjQHpsW1qWtmDRN6SZrHzpVo2LBhdOrUqbQmIlJGY8f6\nvjQPPeTDNJ07w557Rh2ViFRGcf9Iz8vLIysrK6KIXCrmkEwC2iYca0tsYmsIYT6eVPQqOBmbxNoF\n+CB2aCqwKaFNW2B34MMUxCwiCfLzfVO8Ll184ir4fJFaqu8sIimQih6SYcAkM/sr8AKeaPwe+ENc\nm7uB683sK3zZ71DgW2A0+CRXM3sUuMvMVgJrgXuBSVphI1I1HnkEPvoIJkzw1TUiIqmU9IQkhPCJ\nmf0W+BdwAzAfuCKE8Fxcm9vMrAHwINAYmAgcH1eDBCAH2AyMBOoBY4FLkx2viGxpyRIvdDZokJbx\nikjVSEnp+BDC68DrW2kzGBhcyvn1wOWxh4ikUAgwbhx8+il8/LHXGqlTB267LerIRCRTaC8bEWH4\ncLj8cthhB+jYEf74R19Vs+OOUUcmIplCCYlIhtuwAW69Fc46C556SvNFRCQami8vkuGeftqLnl13\nnZIREYmOEhKRDLZsGfzrX3D66dC+fdTRiEgmU0IikoE2bYIzz4SWLeGbb+D666OOSEQynRISkQz0\n3nvwwgvwz3/63jQHHRR1RCKS6TSpVSQDPf+8V1298krNGxGR9KAeEpE08+GHsHFj6q6/YQO8+KIP\n2SgZEZF0oR4SkTSyaBEcfjj85S++y26B/HxYtw62377i1x4xwq/TvDmsWOEJiYhIulBCIpJGpk71\n57vu8pUvhx/uSUrfvr40d+5caNCg/Nf94Qe46CL49Vdo0gTatoUOHZIbu4hIZWjIRiSNTJsGO+3k\nO+z27++9GAcdBIsX+/4yDz1Usevef7/v0vvoo95LMmiQhmtEJL0oIRFJI9OmeQIyYgS0agXLl0O/\nfn58wACvqPrLL+W75s8/e2n4P/zBE5EffoCrrkpN/CIiFaWERKQEn33mCUFVKkhI9tnHN7h7+21P\nJnbc0Sup/vADPPJIYfvNm0u+1iefwDnnwKmnwurVkJPjx+vWVe+IiKQfJSQiJTjtNB82SYW1a2Hg\nQPjTnwqPLVsG331Xck2Q1q29l+Rvf4MZM2DWLNhtN+jTB9as2bL9tdfC+PGwfj38/e+wxx6p+S4i\nIsmgSa0ixfj1V5g3D77+Gt55B3r2TM511671no9rroGvvvKKqWecAd26ee8IlF6k7L77YPp0OOEE\nX767447w1ltwyCGefOy+u7f7/HPvXXnmmdQlVSIiyaQeEpFifP01hOArUq691n8GnxA6dWrh6/LI\nzfUE4tRToX597+XIyoI//9mvO22aL+tt3brka2y/Pbz6qk9Q3XVXeP99H5r55RfvcSmI6777oEUL\nT3ZERKoDJSQixfjyS39+4AH4+GN47jl/PWwYHHywJxHlSUrmzvVJpaed5tfOy4N27fx6H3/sk1U/\n+QQ6dvRkozQ77+w9IB995AlOmzY+r+Sdd/z5m2/gySfhwgt9voiISHWgIRuRYsydCzvs4MtuR42C\nyy7zv/gHD/aE5O67fSlu9+6+iuWDD7zg2A03+LyOeOvX+7DJzjvDY48VLW7WvbvPI7nuOn99+eVl\niy+xQNqxx/oKmksu8WGghg09IRERqS7UQyJSjLlzfaWLGfznP1CvHnTtCttu63M1Hn/cS7z/+c8w\ndKjPDXn5ZX/PtdfCqlV+nRA8MZg503tZiqu0OmyY946cdZY/Kuquu3wlzVNPeS9My5YVv5aISFVT\nD4lIMb780pML8GGRJ57wiaR33AGNG8N55/kjBH/UquVJye23w513wsMP+5LbELymyLPPQqdOJX9e\nVhY8/XTlYm7UCG67rXLXEBGJihISkWLMnQtHHln4+thjvSZJ48ZF25kV1vTYYQcYMgQuvhj+9S8f\n6vn2Wx/myc6uqshFRKonJSQiCdas8TLtBT0kBRKTkZK0bAn33OPzTBYtKlyKKyIiJdMcEslYEyf6\nRNRzz4XJkwuPf/WVPycmJOVl5sXIVBVVRGTrlJBIRgrBi5PtsANMmuSFyQp22p0715/btIkuPhGR\nTKOERDLS2LG+Smb4cK/p0aGDT0L95RdPSHbaqexDNCIiUnmaQyIZJwTf26VrV5+sauaFxLKy4Ljj\nfCJqZYdrRESkfNRDIkm1cqUXBnvttagjKdlnn3ndj+uuK5zfsd9+XuV040avoPrHP0Ybo4hIplEP\niSTV8897D8M113hvQ+3aUUe0pZkz/fnww4seP/tsf4iISNVLeQ+JmV1rZvlmdlfC8SFmttjM1pnZ\neDNrnXC+npkNN7PlZrbWzEaaWbNUxyuV8+STPtwxa5YnJ+lo5kzfmE5zRERE0kdKExIzOwS4AJie\ncPwa4LLYuc7Az8A4M4vfCuxu4ESgD9AD2BkYlcp4pXK+/NInig4ZAief7Pu6DBvmSUp+ftTRFZo5\nEw44IOooREQkXsoSEjPbHnga+D2wKuH0FcDQEMKrIYTPgAF4wnFa7L0NgUFATghhQghhGjAQ6Gpm\nnVMVs1TOU0/5pm6nnAI33ww//uiTR88916uXpktS8tlnSkhERNJNKntIhgOvhBDeiT9oZq2AFsDb\nBcdCCGuAycBhsUMH4/Nb4tt8ASyKayNp5Ndffb+X3/3ON6Dr0ME3mFu71jeie/hhnygaQrRxrlkD\nCxfC/vtHG4eIiBSVkkmtZtYPOBBPLBK1AAKwNOH40tg5gObAhliiUlIbSSN33AHff++73yY67zxf\nvXLBBZ4IXHRRlYf3P7Nm+bN6SERE0kvSExIz2xWf/3F0CGFjsq+/NTk5OTRq1KjIsezsbLK1u1nK\nLFgAt9wCOTnQvn3xbf7wB5g+3XtJGjWCpk3hkEP8OZnWrYMZM+Dgg6FO3G/38uX+uTNn+sqfdu2S\n+7kiItVFbm4uubm5RY6tXr06omgKWUhyH7qZnQq8CGwGCnbxqI33imwG9gW+Ag4MIcyIe997wLQQ\nQo6ZHQW8BTSJ7yUxswXAsBDCPcV8bidg6tSpU+lU2j7vknR9+3r59TlzvBR7STZsgF694P33/fU+\n+8CUKZ4oAKxY4a+POabiy4UHD4abboIdd/Tk5+9/h2XLPFHq3BlatYJ33vHqrCIi4vLy8sjKygLI\nCiHkRRFDKoZs3gISO8SfAGYD/wohzDOzJUAvYAb8bxJrF3zeCcBUYFOszUuxNm2B3YEPUxCzVNDC\nhTBqlJdgLy0ZAahbF959F+bN8/klxx4LAwZ478mDD3o5902b4JJL4P77K7Yp3csv+3X32QduvNGX\n9777rs9xef11qF/fVwCJiEh6SXpCEkL4GSjy708z+xn4MYQwO3bobuB6M/sKWAAMBb4FRseuscbM\nHgXuMrOVwFrgXmBSCGFKsmOWinvoIdh++7IXFKtTp7As+9NPe3IwZowPsQwb5nNN/vxn2H13L65W\nHgsX+rDQ8897r82vv/p8lY0bfWLtrFk+10XzR0RE0k9VVWotMi4UQrjNzBoADwKNgYnA8SGEDXHN\ncvAhnpFAPWAscGnVhCtlsX69l1s/91xPSsrrpJPgjTd8HknnuMXcK1bAtdf6XjPdupX9emPGwDbb\nQO/e/vq++3w+SdOmHuPGjR7zGWeUP1YREUmtpM8hiYrmkFS93Fzo39/nYyRzkujmzZ6MrF0L06b5\nUE9ZHHOMD/O8+WbRa0F6lrAXEUkX6TCHRJvrSYU9+ST06JH8FSu1a/tQ0BdfwO23l9522TK4+mq4\n80547z049dQtr6VkREQk/WlzPamQX37xBOAf/0jN9Tt0gCuv9Impe+0Fxa3aXrIEjj4aFi3yFTxm\nXiVWRESqHyUkUiETJvik0eOOS91n3HyzJx1nnQUvvQRffw1NmsDpp8PSpV4ZdtMmXyrcqpXPPWnZ\nMnXxiIhI6ighkQp54w1fCZPKAmN16sBjj3mSMX48HHggfPON1xfZYQc48UTfyG+vvby9khERkepL\nCYlUyNix3jtSkVoh5VGrFvzzn/4osGaN75ezzTap/WwREak6mtQq5TZvHsydC8cfH83nN2yoZERE\npKZRQiLlNnq0D6f07Bl1JCIiUlMoIZFy+fFHX1nTr5/3VIiIiCSDEhIp1ZdfevGz7bbzjequuspX\nttxxR9SRiYhITaJJrVKiRYugY0cvvX7WWXDLLV759IEHoHnzqKMTEZGaRAmJlOjee6FePS8N37Ah\n/P73Xpb9gguijkxERGoaJSRSrLVr4eGH4eKLC+eKdO5cdBM8ERGRZNEcEinWY4/BunVw2WVRRyIi\nIplACYlsYeNGuOce6NsXdt016mhERCQTaMgmQ4Xg80FGjPCN6Z5+GurX93NPPQXz5/v+MSIiIlVB\nPSQZ6r//9dLvn34Kr70G550H+fneO3LzzdCnj6+wERERqQrqIclQEyZA27Ywa5b3hJxxhicju+zi\nvSOjR0cdoYiIZBL1kGSoadOgUyffHO/00+HRR2H2bLjvPjjzTDjggKgjFBGRTKKEJANt3gzTp8NB\nBxUeGzjQ640sWQJPPBFZaCIikqE0ZJOBvvzSl/R26rTlOVVgFRGRKKiHJANNm+bP8T0kIiIiUVJC\nUsPk53sNkeHDS24zbRrsvrvvUSMiIpIONGRTgyxbBmef7fVFABYv9iW8ZkXbTZum3hEREUkv6iGp\nQa66Cj75xBOSO+7w3XlPPRUmTfJCaODPeXlKSEREJL2oh6SG+OknGDkSrr0WjjnGH7vsAoMHQ7du\nvkFeu3aw116wYoUSEhERSS9KSGqIUaPg55/hnHMKj/Xr5/vRvPOO95zMng1z5kCbNnD44dHFKiIi\nkkgJSQ0xYgQcdRTssUfR47VqwdFH+0NERCRdJX0OiZn91cymmNkaM1tqZi+Z2T7FtBtiZovNbJ2Z\njTez1gnn65nZcDNbbmZrzWykmTVLdrw1wcKF8O67cO65UUciIiJSMamY1NoduA/oAhwNbAO8aWbb\nFjQws2uAy4ALgM7Az8A4M6sbd527gROBPkAPYGdgVArirfZGjvSdevv0iToSERGRikn6kE0I4YT4\n12Z2HvADkAW8Hzt8BTA0hPBqrM0AYClwGvCCmTUEBgH9QggTYm0GArPNrHMIYUqy467O3nwTjjgC\ntt8+6khEREQqpiqW/TYGArACwMxaAS2AtwsahBDWAJOBw2KHDsaTpfg2XwCL4toI8OuvMHGir6oR\nERGprlKakJiZ4UMv74cQPo8dboEnKEsTmi+NnQNoDmyIJSoltRHggw/gl180aVVERKq3VK+yeQBo\nD3RN8edkrPHjoVkzOOCAqCMRERGpuJQlJGZ2P3AC0D2E8H3cqSWA4b0g8b0kzYFpcW3qmlnDhF6S\n5rFzJcrJyaFRo0ZFjmVnZ5OdnV2h75Huxo/33pFaqrkrIiJlkJubS25ubpFjq1evjiiaQhYKaoon\n86KejJwKHBFCmFfM+cXA7SGEYbHXDfHkZEAI4b+x18vwSa0vxdq0BWYDhxY3qdXMOgFTp06dSqdO\nnZL+ndLRjz/CTjvBY4/BeedFHY2IiFRXeXl5ZGVlAWSFEPKiiCEVdUgeAM4C+gM/m1nz2KN+XLO7\ngevN7GQzOwB4EvgWGA3/m+T6KHCXmR1pZlnAY8Ckmr7C5uWX4cILYd4WaZxbscIrsq5a5ct869eH\n3r2rNkYREZFkS0VH/0VAQ+A9YHHco29BgxDCbXitkgfx1TXbAseHEDbEXScHeBUYGXetGltpY+VK\nGDQIfvtbePZZ33fm2mthTdyA1TvvQMuW0KQJ7LsvzJjhQzYtW0YXt4iISDKkog5JmZKcEMJgYHAp\n59cDl8ceNVYI8J//wPXXw8aNPvzSty/cfjvcdhs88QRcdplvinfRRXDkkXDyyTBzJlxxBbRvH/U3\nEBERqTztZROxV1+FSy6BgQPhllugRWxR8+DBcP758Le/eWKydi106QIvvgjbbRdpyCIiIkmnhCRC\nIcDf/+5VVh99FMyKnt9tN3jySdi8Gb7+GvbcE+rWLfZSIiIi1ZoSkirw4IPwxhtw+eWwyy4+76N1\na1i3Dj79FN57b8tkJF7t2rDPFtsTioiI1BxKSFLss8/gj3+ERo1g9Gg/VqcObNrktUN69fIeEhER\nkUymhCSFNm70+iCtW8PUqfDRR94rcuSRkJcHjz8Of/pT1FGKiIhETwlJEi1bBhdfDIcfDh07ws03\n+5DMhx96vZAjjyxs262bP0REREQJSVJddx28/jq88gps2AD77ec/H3JI1JGJiIikNyUkSfLxx75S\n5r77IDvb54507eoTUkVERKR0SkiSIARfQdOhg5d9r1MHevSIOioREZHqQwlJEowdC5Mnw9tvezIi\nIiIi5aNN65Pg1luhc2c46qioIxEREame9O/5Spo8GSZMgFGjSi9uJiIiIiVTQlJOGzfCc8/B3XfD\nqlV+bJ994NRTo41LRESkOlNCUowPPvAN7PbZx1fLfP017L8/fPONFzKbOxeOP94rrM6c6bvxajWN\niIhIxSkhibNpE+TkwP33l9zmyCPh+efhwAOrLCwREZEaTwlJzNq10K8fjBvnCUnHjvDFF9C+PbRp\nA7Nm+XBNr16aKyIiIpJsGZ2QLF4Mc+ZA3bo+7DJ/vldaPfZYPx9f2l0b4ImIiKRORiUkq1Z5NdW6\ndWHGDBgxwns9APbYAyZN8rkiIiIiUrUyKiF5+GG45hqoVw+aNIF//MNXx6xd6xNYd9gh6ghFREQy\nU0YlJGPGwMknw+jRUUciIiIi8TKmUuvy5b6c95RToo5EREREEmVMQvL665CfDyeeGHUkIiIikihj\nEpIxY6BLF2jRIupIREREJFFGJCTr13t9kZNPjjoSERERKU6NT0h++cULnv3yC/TpE3U0IiIiUpwa\nvcpm0ybo3Rs++cRX1uy7b9QRiYiISHFqdEIydixMnAhvveUl30VERCQ91eghm8cfhw4doGfPqCOp\nuXJzc6MOIePonlc93fOqp3ueedI+ITGzS81svpn9YmYfmdkhZXnf8uXwyiswcKA2w0sl/aFR9XTP\nq57uedXTPc88aZ2QmNmZwJ3AjcBBwHRgnJn9ZmvvffZZCAHOOivFQYqIiEilpXVCAuQAD4YQngwh\nzAEuAtYBg0p6w7Jl8J//wJ13+jLfnXaqqlBFRESkotJ2UquZbQNkAbcUHAshBDN7CzispPddfjnM\nmwc9esDQoVUQqIiIiFRa2iYkwG+A2sDShONLgbbFtK8PMGjQbLp0gUaNvCBaXl6Ko8xwq1evJk83\nuUrpnlc93fOqp3tetWbPnl3wY/2oYrAQQlSfXSozawl8BxwWQpgcd/xWoEcI4bCE9v2BZ6o2ShER\nkRrlrBDCs1F8cDr3kCwHNgPNE443B5YU034ccBawAPg1pZGJiIjULPWBPfG/SyORtj0kAGb2ETA5\nhHBF7LUBi4B7Qwi3RxqciIiIJE0695AA3AU8YWZTgSn4qpsGwBNRBiUiIiLJldYJSQjhhVjNkSH4\nUM2nQO8QwrJoIxMREZFkSushGxEREckM6V4YTURERDKAEhIRERGJXI1ISCq6AV+mM7MbzSw/4fF5\nQpshZrbYzNaZ2Xgza51wvp6ZDTez5Wa21sxGmlmzhDZNzOwZM1ttZivN7BEz264qvmPUzKy7mY0x\ns+9i9/eUYtpUyT02s93M7DUz+9nMlpjZbWZWI/4MiLe1e25mjxfze/96Qhvd83Iws7+a2RQzW2Nm\nS83sJTPbp5h2+l1PkrLc8+r2u17t/wNZJTbgEwA+wycMt4g9uhWcMLNrgMuAC4DOwM/4va0b9/67\ngROBPkAPYGdgVMJnPAu0A3rF2vYAHkzBd0lH2+GTsS8BtpiwVVX3OPYHw+v4RPZDgXOB8/AJ4zVN\nqfc85g2K/t5nJ5zXPS+f7sB9QBfgaGAb4E0z27aggX7Xk26r9zym+vyuhxCq9QP4CLgn7rUB3wJX\nRx1buj/wJC6vlPOLgZy41w2BX4C+ca/XA7+Na9MWyAc6x163i70+KK5Nb2AT0CLqe1DF9zsfOCWK\newwcD2wEfhPX5kJgJVAn6ntTxff8ceDFUt6je175+/6b2P3pFndMv+tVf8+r1e96te4hscIN+N4u\nOBb8TpS6AZ8U0SbWtf21mT1tZrsBmFkrPJuOv7drgMkU3tuD8Yw4vs0XePG6gjaHAitDCNPiPvMt\n/F+uXVLzlaqHKr7HhwIzQwjL49qMAxoB+yXpK1UnR8a6ueeY2QNm1jTuXBa655XVGL8XK0C/61Wk\nyB1pECgAAANGSURBVD2PU21+16t1QkLpG/C1qPpwqp2P8G613sBFQCvg/2Jjgy3wX7jS7m1zYEPs\nD5aS2rQAfog/GULYjP9Pk+n/jaryHrco4XMg8/47vAEMAHoCVwNHAK+bmcXOt0D3vMJi9/Fu4P0Q\nQsGcNP2up1AJ9xyq2e96WhdGk9QKIcTvWfCZmU0BFgJ9gTnRRCWSWiGEF+JezjKzmcDXwJHAu5EE\nVbM8ALQHukYdSAYp9p5Xt9/16t5DUt4N+KQUIYTVwFygNX7/jNLv7RKgrpk13EqbxBnbtYGm6L9R\nVd7jJSV8DmT4f4cQwnz8z5L/b+/uXaMIwjiOf0dQg4oEglbRIAjaxBeMjSAqAQtB7Cz9A6ysUgtW\naiuxEGwULWxSiAiWEg8hKaxEBBULX0ARURJBwlo8e7JZNeCR27nV7we2yM2Q2/1l2H12d4Z0V3yY\neY9SSleAE8DRoijeVpoc632yQua/GPSx3uqCpCiK78A8MfMX+PnoahJ4lGu/2iqltIkYqG/KgfuO\n5dluJt4ZdrOdJyY2VfvsArYDnfKjDjCcUtpf+apJ4uT0uD9H0g4NZ9wBxmurz44Dn4FlS73/Nyml\nUWAE6J7MzbwH5YXxFHCsKIrX1TbHen+slPkf+g/2WM89M3gVZhafBhaI92S7iaVIH4Etufdt0Dfg\nMrF8aww4BDwg3vuNlO1TZZYngXFgBngOrKv8jmngJfEI8AAwCzysfc89YA44SDxSfAbcyH38DWW8\nEdgL7CNmqp8rf97WZMbEzccT4p3yHmLe0HvgQu6Mmsy8bLtEXAjHiBPrHPAUWGvmPWc+TayoOEzc\nGXe3oUofx3qDmbdxrGcPdZX+MGeBV8QSsg4wkXuf2rABt4kl0ovErOpbwI5an/PEcr0FYtb0zlr7\nemIt/AfgC3AH2FrrMwzcJKrlT8A1YEPu428o4yPERXGptl1vOmPignwX+FqeLC4Ca3Jn1GTmwBBw\nn7hb/wa8AK5Su4Ex87/O/Hd5LwFnav0c6w1l3sax7j/XkyRJ2bV6DokkSfo3WJBIkqTsLEgkSVJ2\nFiSSJCk7CxJJkpSdBYkkScrOgkSSJGVnQSJJkrKzIJEkSdlZkEiSpOwsSCRJUnY/ABKjF+31XFT2\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa8fc1ae2d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24110/150000 [1:56:09<10:01:29,  3.49it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24110\tepsilon=0.145\tloss=8.775\treward/tick=1.601\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24120/150000 [1:56:11<7:17:38,  4.79it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24120\tepsilon=0.145\tloss=10.569\treward/tick=1.596\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24130/150000 [1:56:13<7:14:37,  4.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24130\tepsilon=0.145\tloss=17.946\treward/tick=1.841\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24140/150000 [1:56:15<7:19:56,  4.77it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24140\tepsilon=0.145\tloss=11.187\treward/tick=1.759\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24150/150000 [1:56:17<8:16:35,  4.22it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24150\tepsilon=0.145\tloss=6.942\treward/tick=1.721\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24160/150000 [1:56:19<7:14:59,  4.82it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24160\tepsilon=0.144\tloss=5.296\treward/tick=1.905\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24170/150000 [1:56:21<7:11:43,  4.86it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=24170\tepsilon=0.144\tloss=3.790\treward/tick=1.860\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|█▌        | 24180/150000 [1:56:23<7:23:14,  4.73it/s]"
     ]
    }
   ],
   "source": [
    "from tqdm import trange\n",
    "from IPython.display import clear_output\n",
    "\n",
    "\n",
    "for i in trange(150000):    \n",
    "    \n",
    "    ##update agent's epsilon (in e-greedy policy)\n",
    "    current_epsilon = 0.01 + 0.45*np.exp(-epoch_counter/20000.)\n",
    "    action_layer.epsilon.set_value(np.float32(current_epsilon))\n",
    "\n",
    "    #play\n",
    "    pool.update(SEQ_LENGTH)\n",
    "\n",
    "    #train\n",
    "    loss = 0.95*loss + 0.05*train_step()\n",
    "    targetnet.load_weights(0.01)\n",
    "    \n",
    "    \n",
    "    if epoch_counter%10==0:\n",
    "        #average reward per game tick in current experience replay pool\n",
    "        reward_per_tick = 0.95*reward_per_tick + 0.05*pool.experience_replay.rewards.get_value().mean()\n",
    "        print(\"iter=%i\\tepsilon=%.3f\\tloss=%.3f\\treward/tick=%.3f\"%(epoch_counter,\n",
    "                                                               current_epsilon,\n",
    "                                                               loss,\n",
    "                                                               reward_per_tick))\n",
    "        \n",
    "    ##record current learning progress and show learning curves\n",
    "    if epoch_counter%100 ==0:\n",
    "        action_layer.epsilon.set_value(0)\n",
    "        reward = 0.95*reward + 0.05*np.mean(pool.evaluate(record_video=False))\n",
    "        action_layer.epsilon.set_value(np.float32(current_epsilon))\n",
    "        \n",
    "        rewards[epoch_counter] = reward\n",
    "        \n",
    "        clear_output(True)\n",
    "        plt.plot(*zip(*sorted(rewards.items(),key=lambda (t,r):t)))\n",
    "        plt.show()\n",
    "        \n",
    "\n",
    "    \n",
    "    epoch_counter  +=1\n",
    "\n",
    "    \n",
    "# Time to drink some coffee!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating results\n",
    " * Here we plot learning curves and sample testimonials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "plt.plot(*zip(*sorted(rewards.items(),key=lambda k:k[0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.utils.persistence import save\n",
    "save(action_layer,\"pacman.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "###LOAD FROM HERE\n",
    "from agentnet.utils.persistence import load\n",
    "load(action_layer,\"pacman.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "action_layer.epsilon.set_value(0.01)\n",
    "rw = pool.evaluate(n_games=20,save_path=\"./records\",record_video=False)\n",
    "print(\"mean session score=%f.5\"%np.mean(rw))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./records/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./videos/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Once you got it working,\n",
    "Try building a network that maximizes the final score\n",
    "\n",
    "* Moar lasagne stuff: convolutional layers, batch normalization, nonlinearities and so on\n",
    "* Recurrent agent memory layers, GRUMemoryLayer, etc\n",
    "* Different reinforcement learning algorithm (p.e. qlearning_n_step), other parameters\n",
    "* Experience replay pool\n",
    "\n",
    "\n",
    "Look for info?\n",
    "* [lasagne doc](http://lasagne.readthedocs.io/en/latest/)\n",
    "* [agentnet doc](http://agentnet.readthedocs.io/en/latest/)\n",
    "* [gym homepage](http://gym.openai.com/)\n",
    "\n",
    "\n",
    "You can also try to expand to a different game: \n",
    " * all OpenAI Atari games are already compatible, you only need to change GAME_TITLE\n",
    " * Other discrete action space environments are also accessible this way\n",
    " * For continuous action spaces, either discretize actions or use continuous RL algorithms (e.g. .learning.dpg_n_step)\n",
    " * Adapting to a custom non-OpenAI environment can be done with a simple wrapper\n",
    " \n",
    " \n",
    "__Good luck!__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:py27]",
   "language": "python",
   "name": "conda-env-py27-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}