Deep Learning for NLP- An Application-based Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-07T21:11:09.810311Z",
     "start_time": "2017-11-07T21:11:09.790928Z"
    }
   },
   "outputs": [],
   "source": [
    "x = []\n",
    "y = []\n",
    "for sg in sg_list:\n",
    "    num_ex = len(sg[0])\n",
    "    for i in range(num_ex):\n",
    "        x.append(sg[0][i][0])\n",
    "        y.append(sg[1][i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `Keras` Setup\n",
    "- For visualizing networks:\n",
    "  - `brew install graphviz`\n",
    "- `pip install keras tensorflow pydot pydot-ng `"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-07T21:56:31.828681Z",
     "start_time": "2017-11-07T21:56:31.728059Z"
    },
    "init_cell": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/IPython/core/magics/pylab.py:160: UserWarning: pylab import has clobbered these variables: ['random']\n",
      "`%matplotlib` prevents importing * from pylab and numpy\n",
      "  \"\\n`%matplotlib` prevents importing * from pylab and numpy\"\n"
     ]
    }
   ],
   "source": [
    "# Imports\n",
    "# Basics\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "# gensim\n",
    "from gensim import corpora, models, similarities, matutils\n",
    "from gensim.models import Word2Vec\n",
    "\n",
    "# nltk\n",
    "from nltk.corpus import gutenberg\n",
    "\n",
    "# sklearn\n",
    "\n",
    "# keras\n",
    "np.random.seed(13)\n",
    "import keras\n",
    "from IPython.display import SVG\n",
    "\n",
    "# logging for gensim (set to INFO)\n",
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "\n",
    "import seaborn\n",
    "seaborn.set_style(\"whitegrid\")\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- What is `Keras`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  - Around what?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Deep Learning Reminder\n",
    "- Deep Learning is going to take over the world\n",
    "<img  src=\"img/landscape-1431110160-terminator.jpg\" style='height:200px'/>\n",
    "- Allegedly!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Intro to `Keras` with Artificial Neural Networks (ANNs)\n",
    "\n",
    "**What is Keras?**\n",
    "\n",
    "Keras is a wrapper around a few of the most-used Deep Learning packages. Keras allows us to rapidly build and modify Deep Learning networks while abstracting away some of the more tedious details. \n",
    "\n",
    "Read the docs!:\n",
    "- [Keras](https://keras.io/)\n",
    "- [Tensorflow](https://www.tensorflow.org/)\n",
    "- [CNTK](https://www.microsoft.com/en-us/cognitive-toolkit/)\n",
    "- [Theano](http://deeplearning.net/software/theano/)\n",
    "\n",
    "We'll be using Tensorflow as the backend engine but feel free to experiment with others."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- What is `Keras`?\n",
    "  - A fantastic **wrapper**!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Deep Learning Reminder \n",
    "- Deep Learning = Fancy ANNs (Artificial Neural Networks)\n",
    "- \"**Deep**\" $\\rightarrow$ Many Hidden Layers\n",
    "- Fancy $\\rightarrow$ Complex Network structures that capture more information from data\n",
    "<div>\n",
    "<table align=\"center\"><tr><td><img  src=\"img/neural_net.jpeg\" style=\"float: left; width: 100%; margin-right: 1%; margin-bottom: 0.5em;\"/></td><td><img  src=\"img/rnn.png\" style=\"float: left; width: 100%; height: 300px; margin-right: 1%; margin-bottom: 0.5em;\"/></td></tr><tr><td style=\"text-align: center\">Simple 2-layer ANN</td><td style=\"text-align: center\">Deep Recurrent Neural Network (RNN)</td></tr></table>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Deep Learning Don't Stress\n",
    "- You got this!\n",
    "- The concepts are all the same:\n",
    "  - Nodes feedforward to their successors and backpropagate to update weights to their predecessors\n",
    "  - Weights matrices represent transitions between nodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  - **TensorFlow** and/or **Theano**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Deep Learning Don't Stress\n",
    "1. Take the input\n",
    "2. Run it thru the network\n",
    "3. Compute the Error/Cost\n",
    "4. Backpropagate to update weights via gradient of cost function\n",
    "\n",
    "We could do a whole course on this!  But today, we'll focus on applications :) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  - And what do those do?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `Keras`: Using the GPU\n",
    "- Deep Learning requires tons of matrix computations\n",
    "- GPUs can do this really fast in parallel (many cores!)\n",
    "- Optional: Set up with your GPU (if you can!)\n",
    "  - Install [CUDA](http://docs.nvidia.com/cuda/index.html#axzz4Pa5zY8Qi) \n",
    "  - In `.bashrc`/`.bash_profile`: `export THEANO_FLAGS=device=gpu,floatX=float32`\n",
    "- Don't worry, you can just use AWS's big toys!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  - **Deep Learning**!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `Keras` Models\n",
    "- The core objects in `Keras` are `Models` and `Layers`\n",
    "- `Models` set up the container for your network\n",
    "- `Layers` fill in the architecture (connections, unit types, activation functions, etc)\n",
    "- The 2 options for `Models`:\n",
    "  - `Sequential`: The basic one we'll focus on\n",
    "  - Function API: Specify complex uncommon models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: build a feedforward model and train\n",
    "\n",
    "### The Sequential Model\n",
    "- Guide is [here](https://keras.io/models/sequential/) (read it!)\n",
    "- Allow you to stack all sorts of layers in your network\n",
    "- Canvas on which you paint your beautiful network!\n",
    "- Tons of possibilities\n",
    "- We'll start with these today\n",
    "- Let's take a look!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 293,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:43:39.053440Z",
     "start_time": "2017-11-08T12:43:39.034174Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Make some fake data\n",
    "import sklearn.datasets\n",
    "import sklearn.model_selection\n",
    "\n",
    "X, y = sklearn.datasets.make_classification(\n",
    "    n_classes=10,\n",
    "    n_clusters_per_class=1,\n",
    "    n_samples=1000,\n",
    "    n_features=100,\n",
    "    n_informative=5,\n",
    "    random_state=27)\n",
    "\n",
    "X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(\n",
    "    X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 294,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:43:39.640681Z",
     "start_time": "2017-11-08T12:43:39.546841Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Create a Keras Sequential model \n",
    "\n",
    "# Specify the network architecture\n",
    "# Here we add layers to our model, we'll come back to that in a bit\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(65, input_shape=(100,), activation=\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\"),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Viewing the Network\n",
    "- `Keras` let's you view your network architecture!\n",
    "- Here's how:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:43:40.802022Z",
     "start_time": "2017-11-08T12:43:40.684511Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"215pt\" viewBox=\"0.00 0.00 321.56 215.00\" width=\"322pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 211)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-211 317.5557,-211 317.5557,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 5110288736 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>5110288736</title>\n",
       "<polygon fill=\"none\" points=\"0,-162.5 0,-206.5 313.5557,-206.5 313.5557,-162.5 0,-162.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"87.1191\" y=\"-180.3\">dense_58_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"174.2383,-162.5 174.2383,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"202.0728\" y=\"-191.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"174.2383,-184.5 229.9072,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"202.0728\" y=\"-169.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"229.9072,-162.5 229.9072,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"271.7314\" y=\"-191.3\">(None, 100)</text>\n",
       "<polyline fill=\"none\" points=\"229.9072,-184.5 313.5557,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"271.7314\" y=\"-169.3\">(None, 100)</text>\n",
       "</g>\n",
       "<!-- 5110748776 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>5110748776</title>\n",
       "<polygon fill=\"none\" points=\"31.4932,-81.5 31.4932,-125.5 282.0625,-125.5 282.0625,-81.5 31.4932,-81.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"87.1191\" y=\"-99.3\">dense_58: Dense</text>\n",
       "<polyline fill=\"none\" points=\"142.7451,-81.5 142.7451,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"170.5796\" y=\"-110.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"142.7451,-103.5 198.4141,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"170.5796\" y=\"-88.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"198.4141,-81.5 198.4141,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"240.2383\" y=\"-110.3\">(None, 100)</text>\n",
       "<polyline fill=\"none\" points=\"198.4141,-103.5 282.0625,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"240.2383\" y=\"-88.3\">(None, 65)</text>\n",
       "</g>\n",
       "<!-- 5110288736&#45;&gt;5110748776 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>5110288736-&gt;5110748776</title>\n",
       "<path d=\"M156.7778,-162.3664C156.7778,-154.1516 156.7778,-144.6579 156.7778,-135.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"160.2779,-135.6068 156.7778,-125.6068 153.2779,-135.6069 160.2779,-135.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 5106369592 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>5106369592</title>\n",
       "<polygon fill=\"none\" points=\"34.9932,-.5 34.9932,-44.5 278.5625,-44.5 278.5625,-.5 34.9932,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"90.6191\" y=\"-18.3\">dense_59: Dense</text>\n",
       "<polyline fill=\"none\" points=\"146.2451,-.5 146.2451,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"174.0796\" y=\"-29.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"146.2451,-22.5 201.9141,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"174.0796\" y=\"-7.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"201.9141,-.5 201.9141,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"240.2383\" y=\"-29.3\">(None, 65)</text>\n",
       "<polyline fill=\"none\" points=\"201.9141,-22.5 278.5625,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"240.2383\" y=\"-7.3\">(None, 10)</text>\n",
       "</g>\n",
       "<!-- 5110748776&#45;&gt;5106369592 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>5110748776-&gt;5106369592</title>\n",
       "<path d=\"M156.7778,-81.3664C156.7778,-73.1516 156.7778,-63.6579 156.7778,-54.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"160.2779,-54.6068 156.7778,-44.6068 153.2779,-54.6069 160.2779,-54.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 295,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View the network graphically\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "from IPython.display import SVG\n",
    "SVG(model_to_dot(model, show_shapes=True).create(prog=\"dot\", format=\"svg\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So pretty! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Compiling the Network\n",
    "- Sets up training parameters before training\n",
    "- e.g.: Cost (Loss) function, Optimization method, Scoring metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 296,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:43:42.718575Z",
     "start_time": "2017-11-08T12:43:42.604749Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Compile the network\n",
    "def top_3(y_true, y_pred):\n",
    "    return keras.metrics.sparse_top_k_categorical_accuracy(y_true, y_pred, k=3)\n",
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='sgd', metrics=['accuracy', top_3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Training the Network\n",
    "- Like `sklearn`, call `fit` (usually) on `numpy` arrays!\n",
    "- ***Epoch***: 1 Full pass through the training set\n",
    "  - Remember, training with Gradient Descent (or similar) we likely take many passes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 297,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:43:43.655031Z",
     "start_time": "2017-11-08T12:43:43.650631Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# y_train = keras.utils.to_categorical(y_train) # same as sklearn.preprocessing.OneHotEncoder\n",
    "# y_test = keras.utils.to_categorical(y_test)\n",
    "# y_train[:5,:]\n",
    "\n",
    "### we'lll use sparse sparse_categorical_crossentropy so we don't have to one-hot ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 298,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:43:54.042237Z",
     "start_time": "2017-11-08T12:43:44.673128Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 562 samples, validate on 188 samples\n",
      "Epoch 1/200\n",
      "562/562 [==============================] - 1s 2ms/step - loss: 2.7063 - acc: 0.0996 - top_3: 0.2936 - val_loss: 2.6115 - val_acc: 0.1596 - val_top_3: 0.3617\n",
      "Epoch 2/200\n",
      "562/562 [==============================] - 0s 88us/step - loss: 2.6132 - acc: 0.1121 - top_3: 0.3327 - val_loss: 2.5528 - val_acc: 0.1702 - val_top_3: 0.3617\n",
      "Epoch 3/200\n",
      "562/562 [==============================] - 0s 72us/step - loss: 2.5325 - acc: 0.1335 - top_3: 0.3594 - val_loss: 2.5028 - val_acc: 0.1702 - val_top_3: 0.3777\n",
      "Epoch 4/200\n",
      "562/562 [==============================] - 0s 98us/step - loss: 2.4614 - acc: 0.1423 - top_3: 0.3915 - val_loss: 2.4593 - val_acc: 0.1755 - val_top_3: 0.3723\n",
      "Epoch 5/200\n",
      "562/562 [==============================] - 0s 85us/step - loss: 2.3974 - acc: 0.1530 - top_3: 0.4199 - val_loss: 2.4209 - val_acc: 0.1702 - val_top_3: 0.3777\n",
      "Epoch 6/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 2.3385 - acc: 0.1726 - top_3: 0.4413 - val_loss: 2.3860 - val_acc: 0.1862 - val_top_3: 0.3883\n",
      "Epoch 7/200\n",
      "562/562 [==============================] - 0s 70us/step - loss: 2.2836 - acc: 0.1886 - top_3: 0.4591 - val_loss: 2.3545 - val_acc: 0.1968 - val_top_3: 0.3989\n",
      "Epoch 8/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 2.2334 - acc: 0.2117 - top_3: 0.4893 - val_loss: 2.3257 - val_acc: 0.2021 - val_top_3: 0.4149\n",
      "Epoch 9/200\n",
      "562/562 [==============================] - 0s 51us/step - loss: 2.1864 - acc: 0.2295 - top_3: 0.5125 - val_loss: 2.2993 - val_acc: 0.2021 - val_top_3: 0.4362\n",
      "Epoch 10/200\n",
      "562/562 [==============================] - 0s 58us/step - loss: 2.1415 - acc: 0.2456 - top_3: 0.5391 - val_loss: 2.2741 - val_acc: 0.2021 - val_top_3: 0.4468\n",
      "Epoch 11/200\n",
      "562/562 [==============================] - 0s 66us/step - loss: 2.1000 - acc: 0.2598 - top_3: 0.5623 - val_loss: 2.2508 - val_acc: 0.2128 - val_top_3: 0.4574\n",
      "Epoch 12/200\n",
      "562/562 [==============================] - 0s 52us/step - loss: 2.0597 - acc: 0.2722 - top_3: 0.5783 - val_loss: 2.2284 - val_acc: 0.2128 - val_top_3: 0.4628\n",
      "Epoch 13/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 2.0225 - acc: 0.2811 - top_3: 0.5925 - val_loss: 2.2076 - val_acc: 0.2287 - val_top_3: 0.4840\n",
      "Epoch 14/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 1.9862 - acc: 0.2972 - top_3: 0.5979 - val_loss: 2.1877 - val_acc: 0.2287 - val_top_3: 0.4840\n",
      "Epoch 15/200\n",
      "562/562 [==============================] - 0s 71us/step - loss: 1.9515 - acc: 0.3132 - top_3: 0.6174 - val_loss: 2.1689 - val_acc: 0.2394 - val_top_3: 0.4894\n",
      "Epoch 16/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 1.9183 - acc: 0.3292 - top_3: 0.6423 - val_loss: 2.1507 - val_acc: 0.2447 - val_top_3: 0.4947\n",
      "Epoch 17/200\n",
      "562/562 [==============================] - 0s 70us/step - loss: 1.8863 - acc: 0.3470 - top_3: 0.6619 - val_loss: 2.1334 - val_acc: 0.2447 - val_top_3: 0.5000\n",
      "Epoch 18/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 1.8557 - acc: 0.3470 - top_3: 0.6690 - val_loss: 2.1170 - val_acc: 0.2553 - val_top_3: 0.5106\n",
      "Epoch 19/200\n",
      "562/562 [==============================] - 0s 71us/step - loss: 1.8260 - acc: 0.3577 - top_3: 0.6922 - val_loss: 2.1011 - val_acc: 0.2606 - val_top_3: 0.5319\n",
      "Epoch 20/200\n",
      "562/562 [==============================] - 0s 48us/step - loss: 1.7977 - acc: 0.3701 - top_3: 0.7046 - val_loss: 2.0860 - val_acc: 0.2660 - val_top_3: 0.5319\n",
      "Epoch 21/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 1.7696 - acc: 0.3826 - top_3: 0.7171 - val_loss: 2.0716 - val_acc: 0.2713 - val_top_3: 0.5319\n",
      "Epoch 22/200\n",
      "562/562 [==============================] - 0s 56us/step - loss: 1.7432 - acc: 0.4039 - top_3: 0.7331 - val_loss: 2.0574 - val_acc: 0.2766 - val_top_3: 0.5426\n",
      "Epoch 23/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 1.7175 - acc: 0.4181 - top_3: 0.7384 - val_loss: 2.0443 - val_acc: 0.2819 - val_top_3: 0.5479\n",
      "Epoch 24/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 1.6931 - acc: 0.4270 - top_3: 0.7456 - val_loss: 2.0310 - val_acc: 0.2819 - val_top_3: 0.5532\n",
      "Epoch 25/200\n",
      "562/562 [==============================] - 0s 76us/step - loss: 1.6686 - acc: 0.4448 - top_3: 0.7544 - val_loss: 2.0190 - val_acc: 0.2872 - val_top_3: 0.5532\n",
      "Epoch 26/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 1.6450 - acc: 0.4502 - top_3: 0.7616 - val_loss: 2.0075 - val_acc: 0.2872 - val_top_3: 0.5691\n",
      "Epoch 27/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 1.6226 - acc: 0.4555 - top_3: 0.7633 - val_loss: 1.9960 - val_acc: 0.2926 - val_top_3: 0.5691\n",
      "Epoch 28/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 1.6006 - acc: 0.4662 - top_3: 0.7722 - val_loss: 1.9854 - val_acc: 0.2926 - val_top_3: 0.5691\n",
      "Epoch 29/200\n",
      "562/562 [==============================] - 0s 65us/step - loss: 1.5791 - acc: 0.4751 - top_3: 0.7829 - val_loss: 1.9748 - val_acc: 0.3032 - val_top_3: 0.5851\n",
      "Epoch 30/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 1.5587 - acc: 0.4840 - top_3: 0.7954 - val_loss: 1.9646 - val_acc: 0.3191 - val_top_3: 0.5851\n",
      "Epoch 31/200\n",
      "562/562 [==============================] - 0s 77us/step - loss: 1.5380 - acc: 0.4929 - top_3: 0.7989 - val_loss: 1.9548 - val_acc: 0.3245 - val_top_3: 0.5851\n",
      "Epoch 32/200\n",
      "562/562 [==============================] - 0s 128us/step - loss: 1.5183 - acc: 0.5053 - top_3: 0.8060 - val_loss: 1.9449 - val_acc: 0.3298 - val_top_3: 0.5904\n",
      "Epoch 33/200\n",
      "562/562 [==============================] - 0s 68us/step - loss: 1.4989 - acc: 0.5142 - top_3: 0.8149 - val_loss: 1.9357 - val_acc: 0.3298 - val_top_3: 0.6117\n",
      "Epoch 34/200\n",
      "562/562 [==============================] - 0s 83us/step - loss: 1.4801 - acc: 0.5160 - top_3: 0.8185 - val_loss: 1.9274 - val_acc: 0.3351 - val_top_3: 0.6223\n",
      "Epoch 35/200\n",
      "562/562 [==============================] - 0s 79us/step - loss: 1.4617 - acc: 0.5231 - top_3: 0.8310 - val_loss: 1.9187 - val_acc: 0.3351 - val_top_3: 0.6223\n",
      "Epoch 36/200\n",
      "562/562 [==============================] - 0s 62us/step - loss: 1.4430 - acc: 0.5356 - top_3: 0.8345 - val_loss: 1.9105 - val_acc: 0.3351 - val_top_3: 0.6223\n",
      "Epoch 37/200\n",
      "562/562 [==============================] - 0s 80us/step - loss: 1.4256 - acc: 0.5427 - top_3: 0.8363 - val_loss: 1.9028 - val_acc: 0.3351 - val_top_3: 0.6330\n",
      "Epoch 38/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 1.4082 - acc: 0.5480 - top_3: 0.8434 - val_loss: 1.8953 - val_acc: 0.3351 - val_top_3: 0.6330\n",
      "Epoch 39/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 1.3914 - acc: 0.5587 - top_3: 0.8488 - val_loss: 1.8879 - val_acc: 0.3404 - val_top_3: 0.6383\n",
      "Epoch 40/200\n",
      "562/562 [==============================] - 0s 84us/step - loss: 1.3752 - acc: 0.5605 - top_3: 0.8559 - val_loss: 1.8807 - val_acc: 0.3404 - val_top_3: 0.6383\n",
      "Epoch 41/200\n",
      "562/562 [==============================] - 0s 68us/step - loss: 1.3591 - acc: 0.5712 - top_3: 0.8577 - val_loss: 1.8735 - val_acc: 0.3457 - val_top_3: 0.6543\n",
      "Epoch 42/200\n",
      "562/562 [==============================] - 0s 63us/step - loss: 1.3431 - acc: 0.5747 - top_3: 0.8577 - val_loss: 1.8667 - val_acc: 0.3511 - val_top_3: 0.6543\n",
      "Epoch 43/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 1.3279 - acc: 0.5836 - top_3: 0.8630 - val_loss: 1.8602 - val_acc: 0.3511 - val_top_3: 0.6543\n",
      "Epoch 44/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 1.3124 - acc: 0.5907 - top_3: 0.8630 - val_loss: 1.8534 - val_acc: 0.3564 - val_top_3: 0.6596\n",
      "Epoch 45/200\n",
      "562/562 [==============================] - 0s 101us/step - loss: 1.2975 - acc: 0.5961 - top_3: 0.8665 - val_loss: 1.8477 - val_acc: 0.3564 - val_top_3: 0.6649\n",
      "Epoch 46/200\n",
      "562/562 [==============================] - 0s 94us/step - loss: 1.2834 - acc: 0.5979 - top_3: 0.8737 - val_loss: 1.8413 - val_acc: 0.3564 - val_top_3: 0.6755\n",
      "Epoch 47/200\n",
      "562/562 [==============================] - 0s 83us/step - loss: 1.2687 - acc: 0.6014 - top_3: 0.8790 - val_loss: 1.8351 - val_acc: 0.3670 - val_top_3: 0.6755\n",
      "Epoch 48/200\n",
      "562/562 [==============================] - 0s 62us/step - loss: 1.2547 - acc: 0.6157 - top_3: 0.8826 - val_loss: 1.8294 - val_acc: 0.3830 - val_top_3: 0.6809\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 49/200\n",
      "562/562 [==============================] - 0s 70us/step - loss: 1.2407 - acc: 0.6281 - top_3: 0.8843 - val_loss: 1.8239 - val_acc: 0.3830 - val_top_3: 0.6862\n",
      "Epoch 50/200\n",
      "562/562 [==============================] - 0s 58us/step - loss: 1.2271 - acc: 0.6281 - top_3: 0.8861 - val_loss: 1.8183 - val_acc: 0.3883 - val_top_3: 0.6862\n",
      "Epoch 51/200\n",
      "562/562 [==============================] - 0s 51us/step - loss: 1.2141 - acc: 0.6317 - top_3: 0.8879 - val_loss: 1.8135 - val_acc: 0.3883 - val_top_3: 0.6915\n",
      "Epoch 52/200\n",
      "562/562 [==============================] - 0s 83us/step - loss: 1.2013 - acc: 0.6352 - top_3: 0.8915 - val_loss: 1.8088 - val_acc: 0.3883 - val_top_3: 0.6968\n",
      "Epoch 53/200\n",
      "562/562 [==============================] - 0s 57us/step - loss: 1.1886 - acc: 0.6388 - top_3: 0.8932 - val_loss: 1.8044 - val_acc: 0.3936 - val_top_3: 0.6968\n",
      "Epoch 54/200\n",
      "562/562 [==============================] - 0s 72us/step - loss: 1.1758 - acc: 0.6441 - top_3: 0.9004 - val_loss: 1.8000 - val_acc: 0.3936 - val_top_3: 0.6968\n",
      "Epoch 55/200\n",
      "562/562 [==============================] - 0s 66us/step - loss: 1.1637 - acc: 0.6530 - top_3: 0.9021 - val_loss: 1.7956 - val_acc: 0.3936 - val_top_3: 0.6968\n",
      "Epoch 56/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 1.1516 - acc: 0.6601 - top_3: 0.9039 - val_loss: 1.7916 - val_acc: 0.4043 - val_top_3: 0.6915\n",
      "Epoch 57/200\n",
      "562/562 [==============================] - 0s 52us/step - loss: 1.1396 - acc: 0.6673 - top_3: 0.9057 - val_loss: 1.7873 - val_acc: 0.4043 - val_top_3: 0.6915\n",
      "Epoch 58/200\n",
      "562/562 [==============================] - 0s 57us/step - loss: 1.1281 - acc: 0.6708 - top_3: 0.9093 - val_loss: 1.7829 - val_acc: 0.4043 - val_top_3: 0.6915\n",
      "Epoch 59/200\n",
      "562/562 [==============================] - 0s 51us/step - loss: 1.1165 - acc: 0.6762 - top_3: 0.9146 - val_loss: 1.7791 - val_acc: 0.4043 - val_top_3: 0.6915\n",
      "Epoch 60/200\n",
      "562/562 [==============================] - 0s 52us/step - loss: 1.1052 - acc: 0.6833 - top_3: 0.9164 - val_loss: 1.7753 - val_acc: 0.4096 - val_top_3: 0.6915\n",
      "Epoch 61/200\n",
      "562/562 [==============================] - 0s 54us/step - loss: 1.0945 - acc: 0.6815 - top_3: 0.9181 - val_loss: 1.7713 - val_acc: 0.4043 - val_top_3: 0.6915\n",
      "Epoch 62/200\n",
      "562/562 [==============================] - 0s 51us/step - loss: 1.0832 - acc: 0.6851 - top_3: 0.9199 - val_loss: 1.7678 - val_acc: 0.4043 - val_top_3: 0.6915\n",
      "Epoch 63/200\n",
      "562/562 [==============================] - 0s 55us/step - loss: 1.0727 - acc: 0.6922 - top_3: 0.9235 - val_loss: 1.7643 - val_acc: 0.4043 - val_top_3: 0.6968\n",
      "Epoch 64/200\n",
      "562/562 [==============================] - 0s 49us/step - loss: 1.0619 - acc: 0.6957 - top_3: 0.9217 - val_loss: 1.7604 - val_acc: 0.4043 - val_top_3: 0.6968\n",
      "Epoch 65/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 1.0514 - acc: 0.6975 - top_3: 0.9288 - val_loss: 1.7570 - val_acc: 0.4043 - val_top_3: 0.6968\n",
      "Epoch 66/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 1.0411 - acc: 0.7046 - top_3: 0.9270 - val_loss: 1.7536 - val_acc: 0.4096 - val_top_3: 0.7021\n",
      "Epoch 67/200\n",
      "562/562 [==============================] - 0s 50us/step - loss: 1.0309 - acc: 0.7100 - top_3: 0.9288 - val_loss: 1.7509 - val_acc: 0.4096 - val_top_3: 0.7021\n",
      "Epoch 68/200\n",
      "562/562 [==============================] - 0s 55us/step - loss: 1.0206 - acc: 0.7100 - top_3: 0.9324 - val_loss: 1.7480 - val_acc: 0.4149 - val_top_3: 0.7021\n",
      "Epoch 69/200\n",
      "562/562 [==============================] - 0s 58us/step - loss: 1.0113 - acc: 0.7135 - top_3: 0.9359 - val_loss: 1.7453 - val_acc: 0.4096 - val_top_3: 0.7074\n",
      "Epoch 70/200\n",
      "562/562 [==============================] - 0s 55us/step - loss: 1.0015 - acc: 0.7153 - top_3: 0.9359 - val_loss: 1.7426 - val_acc: 0.4096 - val_top_3: 0.7128\n",
      "Epoch 71/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.9917 - acc: 0.7206 - top_3: 0.9395 - val_loss: 1.7397 - val_acc: 0.4096 - val_top_3: 0.7128\n",
      "Epoch 72/200\n",
      "562/562 [==============================] - 0s 52us/step - loss: 0.9826 - acc: 0.7206 - top_3: 0.9395 - val_loss: 1.7369 - val_acc: 0.4096 - val_top_3: 0.7181\n",
      "Epoch 73/200\n",
      "562/562 [==============================] - 0s 58us/step - loss: 0.9730 - acc: 0.7242 - top_3: 0.9395 - val_loss: 1.7342 - val_acc: 0.4149 - val_top_3: 0.7181\n",
      "Epoch 74/200\n",
      "562/562 [==============================] - 0s 56us/step - loss: 0.9641 - acc: 0.7242 - top_3: 0.9431 - val_loss: 1.7317 - val_acc: 0.4096 - val_top_3: 0.7181\n",
      "Epoch 75/200\n",
      "562/562 [==============================] - 0s 55us/step - loss: 0.9547 - acc: 0.7242 - top_3: 0.9466 - val_loss: 1.7288 - val_acc: 0.4149 - val_top_3: 0.7234\n",
      "Epoch 76/200\n",
      "562/562 [==============================] - 0s 51us/step - loss: 0.9464 - acc: 0.7313 - top_3: 0.9466 - val_loss: 1.7263 - val_acc: 0.4149 - val_top_3: 0.7234\n",
      "Epoch 77/200\n",
      "562/562 [==============================] - 0s 54us/step - loss: 0.9372 - acc: 0.7331 - top_3: 0.9502 - val_loss: 1.7240 - val_acc: 0.4096 - val_top_3: 0.7234\n",
      "Epoch 78/200\n",
      "562/562 [==============================] - 0s 48us/step - loss: 0.9284 - acc: 0.7313 - top_3: 0.9502 - val_loss: 1.7217 - val_acc: 0.4096 - val_top_3: 0.7287\n",
      "Epoch 79/200\n",
      "562/562 [==============================] - 0s 58us/step - loss: 0.9198 - acc: 0.7384 - top_3: 0.9502 - val_loss: 1.7195 - val_acc: 0.4149 - val_top_3: 0.7287\n",
      "Epoch 80/200\n",
      "562/562 [==============================] - 0s 56us/step - loss: 0.9115 - acc: 0.7384 - top_3: 0.9502 - val_loss: 1.7178 - val_acc: 0.4149 - val_top_3: 0.7287\n",
      "Epoch 81/200\n",
      "562/562 [==============================] - 0s 49us/step - loss: 0.9032 - acc: 0.7456 - top_3: 0.9502 - val_loss: 1.7156 - val_acc: 0.4149 - val_top_3: 0.7287\n",
      "Epoch 82/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.8951 - acc: 0.7438 - top_3: 0.9502 - val_loss: 1.7137 - val_acc: 0.4149 - val_top_3: 0.7287\n",
      "Epoch 83/200\n",
      "562/562 [==============================] - 0s 55us/step - loss: 0.8868 - acc: 0.7456 - top_3: 0.9520 - val_loss: 1.7115 - val_acc: 0.4202 - val_top_3: 0.7287\n",
      "Epoch 84/200\n",
      "562/562 [==============================] - 0s 48us/step - loss: 0.8786 - acc: 0.7491 - top_3: 0.9537 - val_loss: 1.7101 - val_acc: 0.4202 - val_top_3: 0.7287\n",
      "Epoch 85/200\n",
      "562/562 [==============================] - 0s 64us/step - loss: 0.8708 - acc: 0.7527 - top_3: 0.9537 - val_loss: 1.7085 - val_acc: 0.4202 - val_top_3: 0.7287\n",
      "Epoch 86/200\n",
      "562/562 [==============================] - 0s 56us/step - loss: 0.8629 - acc: 0.7580 - top_3: 0.9555 - val_loss: 1.7065 - val_acc: 0.4202 - val_top_3: 0.7340\n",
      "Epoch 87/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.8553 - acc: 0.7616 - top_3: 0.9555 - val_loss: 1.7050 - val_acc: 0.4255 - val_top_3: 0.7340\n",
      "Epoch 88/200\n",
      "562/562 [==============================] - 0s 58us/step - loss: 0.8474 - acc: 0.7616 - top_3: 0.9555 - val_loss: 1.7032 - val_acc: 0.4255 - val_top_3: 0.7340\n",
      "Epoch 89/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.8403 - acc: 0.7651 - top_3: 0.9573 - val_loss: 1.7018 - val_acc: 0.4255 - val_top_3: 0.7447\n",
      "Epoch 90/200\n",
      "562/562 [==============================] - 0s 65us/step - loss: 0.8326 - acc: 0.7669 - top_3: 0.9573 - val_loss: 1.7005 - val_acc: 0.4255 - val_top_3: 0.7447\n",
      "Epoch 91/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.8252 - acc: 0.7687 - top_3: 0.9573 - val_loss: 1.6990 - val_acc: 0.4255 - val_top_3: 0.7447\n",
      "Epoch 92/200\n",
      "562/562 [==============================] - 0s 66us/step - loss: 0.8183 - acc: 0.7758 - top_3: 0.9573 - val_loss: 1.6977 - val_acc: 0.4255 - val_top_3: 0.7447\n",
      "Epoch 93/200\n",
      "562/562 [==============================] - 0s 64us/step - loss: 0.8110 - acc: 0.7811 - top_3: 0.9591 - val_loss: 1.6965 - val_acc: 0.4255 - val_top_3: 0.7447\n",
      "Epoch 94/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 0.8042 - acc: 0.7829 - top_3: 0.9591 - val_loss: 1.6952 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 95/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.7969 - acc: 0.7865 - top_3: 0.9591 - val_loss: 1.6940 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 96/200\n",
      "562/562 [==============================] - 0s 68us/step - loss: 0.7903 - acc: 0.7847 - top_3: 0.9626 - val_loss: 1.6927 - val_acc: 0.4309 - val_top_3: 0.7447\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 97/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.7832 - acc: 0.7900 - top_3: 0.9626 - val_loss: 1.6919 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 98/200\n",
      "562/562 [==============================] - 0s 80us/step - loss: 0.7766 - acc: 0.7972 - top_3: 0.9626 - val_loss: 1.6903 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 99/200\n",
      "562/562 [==============================] - 0s 54us/step - loss: 0.7699 - acc: 0.8043 - top_3: 0.9626 - val_loss: 1.6897 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 100/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 0.7633 - acc: 0.8043 - top_3: 0.9644 - val_loss: 1.6891 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 101/200\n",
      "562/562 [==============================] - 0s 64us/step - loss: 0.7569 - acc: 0.8114 - top_3: 0.9644 - val_loss: 1.6881 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 102/200\n",
      "562/562 [==============================] - 0s 55us/step - loss: 0.7505 - acc: 0.8132 - top_3: 0.9644 - val_loss: 1.6874 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 103/200\n",
      "562/562 [==============================] - 0s 64us/step - loss: 0.7444 - acc: 0.8203 - top_3: 0.9644 - val_loss: 1.6865 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 104/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 0.7382 - acc: 0.8149 - top_3: 0.9644 - val_loss: 1.6852 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 105/200\n",
      "562/562 [==============================] - 0s 62us/step - loss: 0.7320 - acc: 0.8203 - top_3: 0.9662 - val_loss: 1.6851 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 106/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.7257 - acc: 0.8256 - top_3: 0.9680 - val_loss: 1.6845 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 107/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 0.7195 - acc: 0.8292 - top_3: 0.9662 - val_loss: 1.6837 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 108/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.7133 - acc: 0.8327 - top_3: 0.9680 - val_loss: 1.6829 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 109/200\n",
      "562/562 [==============================] - 0s 57us/step - loss: 0.7076 - acc: 0.8327 - top_3: 0.9698 - val_loss: 1.6825 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 110/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.7020 - acc: 0.8345 - top_3: 0.9715 - val_loss: 1.6820 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 111/200\n",
      "562/562 [==============================] - 0s 57us/step - loss: 0.6961 - acc: 0.8381 - top_3: 0.9698 - val_loss: 1.6818 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 112/200\n",
      "562/562 [==============================] - 0s 56us/step - loss: 0.6904 - acc: 0.8363 - top_3: 0.9715 - val_loss: 1.6816 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 113/200\n",
      "562/562 [==============================] - 0s 62us/step - loss: 0.6844 - acc: 0.8381 - top_3: 0.9733 - val_loss: 1.6809 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 114/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.6790 - acc: 0.8399 - top_3: 0.9715 - val_loss: 1.6807 - val_acc: 0.4309 - val_top_3: 0.7553\n",
      "Epoch 115/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 0.6732 - acc: 0.8381 - top_3: 0.9733 - val_loss: 1.6802 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 116/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.6683 - acc: 0.8416 - top_3: 0.9733 - val_loss: 1.6800 - val_acc: 0.4362 - val_top_3: 0.7447\n",
      "Epoch 117/200\n",
      "562/562 [==============================] - 0s 58us/step - loss: 0.6628 - acc: 0.8434 - top_3: 0.9733 - val_loss: 1.6800 - val_acc: 0.4362 - val_top_3: 0.7447\n",
      "Epoch 118/200\n",
      "562/562 [==============================] - 0s 56us/step - loss: 0.6573 - acc: 0.8452 - top_3: 0.9733 - val_loss: 1.6799 - val_acc: 0.4362 - val_top_3: 0.7447\n",
      "Epoch 119/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.6518 - acc: 0.8470 - top_3: 0.9769 - val_loss: 1.6793 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 120/200\n",
      "562/562 [==============================] - 0s 72us/step - loss: 0.6467 - acc: 0.8470 - top_3: 0.9751 - val_loss: 1.6792 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 121/200\n",
      "562/562 [==============================] - 0s 81us/step - loss: 0.6414 - acc: 0.8470 - top_3: 0.9769 - val_loss: 1.6786 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 122/200\n",
      "562/562 [==============================] - 0s 86us/step - loss: 0.6365 - acc: 0.8505 - top_3: 0.9751 - val_loss: 1.6784 - val_acc: 0.4255 - val_top_3: 0.7447\n",
      "Epoch 123/200\n",
      "562/562 [==============================] - 0s 70us/step - loss: 0.6316 - acc: 0.8523 - top_3: 0.9769 - val_loss: 1.6783 - val_acc: 0.4309 - val_top_3: 0.7447\n",
      "Epoch 124/200\n",
      "562/562 [==============================] - 0s 73us/step - loss: 0.6262 - acc: 0.8541 - top_3: 0.9786 - val_loss: 1.6781 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 125/200\n",
      "562/562 [==============================] - 0s 55us/step - loss: 0.6211 - acc: 0.8505 - top_3: 0.9769 - val_loss: 1.6783 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 126/200\n",
      "562/562 [==============================] - 0s 77us/step - loss: 0.6162 - acc: 0.8523 - top_3: 0.9786 - val_loss: 1.6785 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 127/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 0.6111 - acc: 0.8541 - top_3: 0.9786 - val_loss: 1.6783 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 128/200\n",
      "562/562 [==============================] - 0s 82us/step - loss: 0.6064 - acc: 0.8594 - top_3: 0.9786 - val_loss: 1.6782 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 129/200\n",
      "562/562 [==============================] - 0s 56us/step - loss: 0.6015 - acc: 0.8577 - top_3: 0.9804 - val_loss: 1.6784 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 130/200\n",
      "562/562 [==============================] - 0s 65us/step - loss: 0.5966 - acc: 0.8612 - top_3: 0.9804 - val_loss: 1.6784 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 131/200\n",
      "562/562 [==============================] - ETA: 0s - loss: 0.5907 - acc: 0.9219 - top_3: 0.984 - 0s 65us/step - loss: 0.5918 - acc: 0.8612 - top_3: 0.9804 - val_loss: 1.6786 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 132/200\n",
      "562/562 [==============================] - 0s 75us/step - loss: 0.5871 - acc: 0.8665 - top_3: 0.9822 - val_loss: 1.6784 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 133/200\n",
      "562/562 [==============================] - 0s 56us/step - loss: 0.5825 - acc: 0.8665 - top_3: 0.9840 - val_loss: 1.6788 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 134/200\n",
      "562/562 [==============================] - 0s 55us/step - loss: 0.5779 - acc: 0.8665 - top_3: 0.9858 - val_loss: 1.6790 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 135/200\n",
      "562/562 [==============================] - 0s 62us/step - loss: 0.5736 - acc: 0.8701 - top_3: 0.9858 - val_loss: 1.6792 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 136/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 0.5689 - acc: 0.8701 - top_3: 0.9858 - val_loss: 1.6794 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 137/200\n",
      "562/562 [==============================] - 0s 139us/step - loss: 0.5643 - acc: 0.8737 - top_3: 0.9875 - val_loss: 1.6797 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 138/200\n",
      "562/562 [==============================] - 0s 87us/step - loss: 0.5600 - acc: 0.8737 - top_3: 0.9875 - val_loss: 1.6798 - val_acc: 0.4309 - val_top_3: 0.7500\n",
      "Epoch 139/200\n",
      "562/562 [==============================] - 0s 90us/step - loss: 0.5555 - acc: 0.8754 - top_3: 0.9875 - val_loss: 1.6799 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 140/200\n",
      "562/562 [==============================] - 0s 78us/step - loss: 0.5513 - acc: 0.8772 - top_3: 0.9875 - val_loss: 1.6799 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 141/200\n",
      "562/562 [==============================] - 0s 79us/step - loss: 0.5468 - acc: 0.8826 - top_3: 0.9875 - val_loss: 1.6804 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 142/200\n",
      "562/562 [==============================] - 0s 71us/step - loss: 0.5427 - acc: 0.8843 - top_3: 0.9875 - val_loss: 1.6808 - val_acc: 0.4362 - val_top_3: 0.7500\n",
      "Epoch 143/200\n",
      "562/562 [==============================] - 0s 87us/step - loss: 0.5385 - acc: 0.8861 - top_3: 0.9875 - val_loss: 1.6808 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 144/200\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "562/562 [==============================] - 0s 72us/step - loss: 0.5341 - acc: 0.8861 - top_3: 0.9875 - val_loss: 1.6810 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 145/200\n",
      "562/562 [==============================] - 0s 74us/step - loss: 0.5302 - acc: 0.8879 - top_3: 0.9893 - val_loss: 1.6810 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 146/200\n",
      "562/562 [==============================] - 0s 95us/step - loss: 0.5260 - acc: 0.8879 - top_3: 0.9893 - val_loss: 1.6809 - val_acc: 0.4362 - val_top_3: 0.7553\n",
      "Epoch 147/200\n",
      "562/562 [==============================] - 0s 84us/step - loss: 0.5218 - acc: 0.8915 - top_3: 0.9893 - val_loss: 1.6813 - val_acc: 0.4415 - val_top_3: 0.7553\n",
      "Epoch 148/200\n",
      "562/562 [==============================] - 0s 68us/step - loss: 0.5180 - acc: 0.8932 - top_3: 0.9893 - val_loss: 1.6819 - val_acc: 0.4415 - val_top_3: 0.7553\n",
      "Epoch 149/200\n",
      "562/562 [==============================] - 0s 115us/step - loss: 0.5139 - acc: 0.8986 - top_3: 0.9893 - val_loss: 1.6820 - val_acc: 0.4415 - val_top_3: 0.7553\n",
      "Epoch 150/200\n",
      "562/562 [==============================] - 0s 65us/step - loss: 0.5099 - acc: 0.9021 - top_3: 0.9893 - val_loss: 1.6823 - val_acc: 0.4415 - val_top_3: 0.7553\n",
      "Epoch 151/200\n",
      "562/562 [==============================] - 0s 92us/step - loss: 0.5060 - acc: 0.9021 - top_3: 0.9893 - val_loss: 1.6829 - val_acc: 0.4415 - val_top_3: 0.7553\n",
      "Epoch 152/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 0.5020 - acc: 0.9021 - top_3: 0.9893 - val_loss: 1.6833 - val_acc: 0.4468 - val_top_3: 0.7553\n",
      "Epoch 153/200\n",
      "562/562 [==============================] - 0s 66us/step - loss: 0.4983 - acc: 0.9021 - top_3: 0.9893 - val_loss: 1.6836 - val_acc: 0.4468 - val_top_3: 0.7606\n",
      "Epoch 154/200\n",
      "562/562 [==============================] - 0s 91us/step - loss: 0.4944 - acc: 0.9039 - top_3: 0.9893 - val_loss: 1.6839 - val_acc: 0.4468 - val_top_3: 0.7606\n",
      "Epoch 155/200\n",
      "562/562 [==============================] - 0s 71us/step - loss: 0.4907 - acc: 0.9075 - top_3: 0.9893 - val_loss: 1.6845 - val_acc: 0.4468 - val_top_3: 0.7606\n",
      "Epoch 156/200\n",
      "562/562 [==============================] - 0s 63us/step - loss: 0.4869 - acc: 0.9057 - top_3: 0.9893 - val_loss: 1.6851 - val_acc: 0.4521 - val_top_3: 0.7606\n",
      "Epoch 157/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 0.4834 - acc: 0.9075 - top_3: 0.9893 - val_loss: 1.6862 - val_acc: 0.4521 - val_top_3: 0.7606\n",
      "Epoch 158/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 0.4796 - acc: 0.9093 - top_3: 0.9893 - val_loss: 1.6865 - val_acc: 0.4521 - val_top_3: 0.7606\n",
      "Epoch 159/200\n",
      "562/562 [==============================] - 0s 62us/step - loss: 0.4759 - acc: 0.9075 - top_3: 0.9893 - val_loss: 1.6871 - val_acc: 0.4521 - val_top_3: 0.7606\n",
      "Epoch 160/200\n",
      "562/562 [==============================] - 0s 57us/step - loss: 0.4725 - acc: 0.9093 - top_3: 0.9893 - val_loss: 1.6876 - val_acc: 0.4521 - val_top_3: 0.7606\n",
      "Epoch 161/200\n",
      "562/562 [==============================] - 0s 57us/step - loss: 0.4687 - acc: 0.9110 - top_3: 0.9893 - val_loss: 1.6876 - val_acc: 0.4521 - val_top_3: 0.7606\n",
      "Epoch 162/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 0.4651 - acc: 0.9146 - top_3: 0.9893 - val_loss: 1.6883 - val_acc: 0.4521 - val_top_3: 0.7606\n",
      "Epoch 163/200\n",
      "562/562 [==============================] - 0s 63us/step - loss: 0.4616 - acc: 0.9110 - top_3: 0.9893 - val_loss: 1.6889 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 164/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 0.4583 - acc: 0.9164 - top_3: 0.9893 - val_loss: 1.6894 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 165/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 0.4546 - acc: 0.9217 - top_3: 0.9911 - val_loss: 1.6898 - val_acc: 0.4521 - val_top_3: 0.7660\n",
      "Epoch 166/200\n",
      "562/562 [==============================] - 0s 66us/step - loss: 0.4512 - acc: 0.9235 - top_3: 0.9911 - val_loss: 1.6902 - val_acc: 0.4521 - val_top_3: 0.7660\n",
      "Epoch 167/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 0.4480 - acc: 0.9235 - top_3: 0.9911 - val_loss: 1.6902 - val_acc: 0.4521 - val_top_3: 0.7660\n",
      "Epoch 168/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.4446 - acc: 0.9253 - top_3: 0.9911 - val_loss: 1.6907 - val_acc: 0.4521 - val_top_3: 0.7660\n",
      "Epoch 169/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 0.4414 - acc: 0.9288 - top_3: 0.9911 - val_loss: 1.6912 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 170/200\n",
      "562/562 [==============================] - 0s 72us/step - loss: 0.4379 - acc: 0.9288 - top_3: 0.9911 - val_loss: 1.6919 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 171/200\n",
      "562/562 [==============================] - 0s 52us/step - loss: 0.4346 - acc: 0.9306 - top_3: 0.9911 - val_loss: 1.6923 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 172/200\n",
      "562/562 [==============================] - 0s 54us/step - loss: 0.4314 - acc: 0.9324 - top_3: 0.9911 - val_loss: 1.6929 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 173/200\n",
      "562/562 [==============================] - 0s 74us/step - loss: 0.4281 - acc: 0.9324 - top_3: 0.9911 - val_loss: 1.6932 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 174/200\n",
      "562/562 [==============================] - 0s 47us/step - loss: 0.4250 - acc: 0.9324 - top_3: 0.9911 - val_loss: 1.6943 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 175/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 0.4219 - acc: 0.9359 - top_3: 0.9911 - val_loss: 1.6946 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 176/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.4187 - acc: 0.9359 - top_3: 0.9911 - val_loss: 1.6956 - val_acc: 0.4362 - val_top_3: 0.7660\n",
      "Epoch 177/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 0.4157 - acc: 0.9359 - top_3: 0.9911 - val_loss: 1.6962 - val_acc: 0.4362 - val_top_3: 0.7660\n",
      "Epoch 178/200\n",
      "562/562 [==============================] - 0s 68us/step - loss: 0.4126 - acc: 0.9413 - top_3: 0.9911 - val_loss: 1.6967 - val_acc: 0.4362 - val_top_3: 0.7660\n",
      "Epoch 179/200\n",
      "562/562 [==============================] - 0s 68us/step - loss: 0.4095 - acc: 0.9448 - top_3: 0.9911 - val_loss: 1.6971 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 180/200\n",
      "562/562 [==============================] - 0s 53us/step - loss: 0.4064 - acc: 0.9431 - top_3: 0.9911 - val_loss: 1.6977 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 181/200\n",
      "562/562 [==============================] - 0s 49us/step - loss: 0.4034 - acc: 0.9466 - top_3: 0.9911 - val_loss: 1.6983 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 182/200\n",
      "562/562 [==============================] - 0s 54us/step - loss: 0.4006 - acc: 0.9448 - top_3: 0.9911 - val_loss: 1.6991 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 183/200\n",
      "562/562 [==============================] - 0s 60us/step - loss: 0.3976 - acc: 0.9502 - top_3: 0.9929 - val_loss: 1.6996 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 184/200\n",
      "562/562 [==============================] - 0s 74us/step - loss: 0.3947 - acc: 0.9484 - top_3: 0.9929 - val_loss: 1.7004 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 185/200\n",
      "562/562 [==============================] - 0s 78us/step - loss: 0.3917 - acc: 0.9520 - top_3: 0.9929 - val_loss: 1.7009 - val_acc: 0.4415 - val_top_3: 0.7606\n",
      "Epoch 186/200\n",
      "562/562 [==============================] - 0s 70us/step - loss: 0.3890 - acc: 0.9537 - top_3: 0.9929 - val_loss: 1.7016 - val_acc: 0.4415 - val_top_3: 0.7606\n",
      "Epoch 187/200\n",
      "562/562 [==============================] - 0s 81us/step - loss: 0.3862 - acc: 0.9537 - top_3: 0.9929 - val_loss: 1.7023 - val_acc: 0.4468 - val_top_3: 0.7606\n",
      "Epoch 188/200\n",
      "562/562 [==============================] - 0s 83us/step - loss: 0.3833 - acc: 0.9537 - top_3: 0.9929 - val_loss: 1.7029 - val_acc: 0.4468 - val_top_3: 0.7606\n",
      "Epoch 189/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.3806 - acc: 0.9537 - top_3: 0.9929 - val_loss: 1.7036 - val_acc: 0.4468 - val_top_3: 0.7606\n",
      "Epoch 190/200\n",
      "562/562 [==============================] - 0s 61us/step - loss: 0.3777 - acc: 0.9555 - top_3: 0.9929 - val_loss: 1.7046 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 191/200\n",
      "562/562 [==============================] - 0s 68us/step - loss: 0.3750 - acc: 0.9555 - top_3: 0.9929 - val_loss: 1.7053 - val_acc: 0.4415 - val_top_3: 0.7660\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 192/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 0.3724 - acc: 0.9555 - top_3: 0.9929 - val_loss: 1.7058 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 193/200\n",
      "562/562 [==============================] - 0s 79us/step - loss: 0.3697 - acc: 0.9555 - top_3: 0.9929 - val_loss: 1.7064 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 194/200\n",
      "562/562 [==============================] - 0s 59us/step - loss: 0.3671 - acc: 0.9555 - top_3: 0.9929 - val_loss: 1.7072 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 195/200\n",
      "562/562 [==============================] - 0s 75us/step - loss: 0.3642 - acc: 0.9555 - top_3: 0.9947 - val_loss: 1.7081 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 196/200\n",
      "562/562 [==============================] - 0s 63us/step - loss: 0.3617 - acc: 0.9555 - top_3: 0.9929 - val_loss: 1.7090 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 197/200\n",
      "562/562 [==============================] - 0s 72us/step - loss: 0.3591 - acc: 0.9555 - top_3: 0.9947 - val_loss: 1.7100 - val_acc: 0.4415 - val_top_3: 0.7660\n",
      "Epoch 198/200\n",
      "562/562 [==============================] - 0s 70us/step - loss: 0.3566 - acc: 0.9573 - top_3: 0.9947 - val_loss: 1.7107 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 199/200\n",
      "562/562 [==============================] - 0s 69us/step - loss: 0.3540 - acc: 0.9573 - top_3: 0.9947 - val_loss: 1.7116 - val_acc: 0.4468 - val_top_3: 0.7660\n",
      "Epoch 200/200\n",
      "562/562 [==============================] - 0s 67us/step - loss: 0.3515 - acc: 0.9573 - top_3: 0.9947 - val_loss: 1.7123 - val_acc: 0.4468 - val_top_3: 0.7660\n"
     ]
    }
   ],
   "source": [
    "# Fit the model\n",
    "history = model.fit(X_train, y_train, epochs=200, batch_size=64, validation_split=.25);\n",
    "# Fit in batches\n",
    "# model.train_on_batch(X_batch, Y_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 309,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:52:44.991239Z",
     "start_time": "2017-11-08T12:52:44.962496Z"
    }
   },
   "outputs": [],
   "source": [
    "def plot_history(history):\n",
    "\n",
    "    measures = np.unique([m.replace('val_', '') for m in history.history.keys()])\n",
    "    num_meas = len(measures)\n",
    "    x = arange(len(history.history[measures[0]]))\n",
    "    fix, axes = subplots(nrows=num_meas,ncols=1,squeeze=True, sharex=True,figsize=(6,2*num_meas), tight_layout=True)\n",
    "    for i,meas in enumerate(measures):\n",
    "        axes[i].plot(x, history.history[meas], label=meas)\n",
    "        if \"val_\"+meas in history.history.keys():\n",
    "            axes[i].plot(x, history.history[\"val_\"+meas], label=\"val_\"+meas)\n",
    "        axes[i].legend()\n",
    "        if meas in [\"acc\", \"top_3\"]:\n",
    "            axes[i].set_ylim((-0.01,1.01))\n",
    "    axes[-1].set_xlabel(\"epoch\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 300,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:43:55.645942Z",
     "start_time": "2017-11-08T12:43:54.112588Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/matplotlib/figure.py:1999: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.\n",
      "  warnings.warn(\"This figure includes Axes that are not compatible \"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa4AAAGoCAYAAAAerAGHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xl4VOXd//H3bMnMZJnsOyQEyCEE\nwhI2FVxRQbRSRCsoqNVan1br0sdabbX2qbVVXH62VVukuLVoKYWqqIgLiqIIBFmEcICwJiH7nskk\nk5nz++MkISGJJJBtku/ruuaambPNdwYynznn3Pd9DJqmIYQQQvgKY18XIIQQQnSFBJcQQgifIsEl\nhBDCp0hwCSGE8CkSXEIIIXyKBJcQQgifYu7MQoqiTAWeUFX1wlOmXwU8AjQAy1VVfUlRFBvwDyAK\nqAJuUlW1qFurFkIIMWiddo9LUZRfAMsA6ynTLcCzwGXABcDtiqJEA/8D7FZVdQbwGvDr7i5aCCHE\n4NWZPa5sYB7w+inTU4GDqqqWASiK8gVwPjAdeLJxmfeBh9vbaGZmpvR8FkII0aGMjAxDe9NPG1yq\nqv5HUZSkdmYFAxUtnlcBjlOmN01rl91uP93Ln5bL5cJqtZ5+wX5Eau4dvlazr9ULUnNvGYw1O53O\nDud16hxXByqBoBbPg4DyU6Y3TWtXamrqWby8Lisrq1u205uk5t7hazX7Wr0gNfeWwVhzZmZmh/PO\nJriygJGKooQB1eiHCZ8CEoErgC3AbODzs3gNIYQQopUuB5eiKAuBQFVVlyqKch/wAXojj+WqquYq\nivIi8GrjOa96YGG3ViyEEGJQ61Rwqap6BJjW+HhFi+nvAO+csqwTuLb7ShRCCHGmPF6NoyU15JW7\nqHK528x3ezUKKlyU1NR3absaGmU19RRU1tHg9TZPT08I4YFZo8667u9yNocKhRBC9IHCKhdqfhUh\nNj8sZgM7j5dTUlNPTLCVrBOVfJRViMlowGGzsO9EJTX1ntNu02IyYDC024ivQyE2CzEOK36mkz2r\n6hu837FG95DgEkKIfuhERS0HCqrxahrHc53kGwrZdrSU93fnc6i4psP1LCYD542IwN9spKzGzbyJ\nCaQnOEgItRNit3BqNpkMBqKCrQRbzV0Orr4iwSWEEL2gtt7D7twKdudW4PZ4qW/wcqLCxYmKWvLK\na3G22Ctyub0UV9edsoV8jAY4d3gEC6YMJS0umKq6BmrrPYxNcBATbCW/0kVEoD8Om6V331wvk+AS\nQohuUuF0s27PCTYeKCavvJYKpxsNqKx1t3sOKTzAj9gQK4nhAQRZT34dm40GUmODGR0bjMVs5MiR\nIyQlJTE0zE5EoH+Hrz88MrAn3la/I8ElhBCNNE1rPlymaRoVtW5OVLiwWkxEBflT5qznRIWLvPJa\nsgur2X6sHA2NcQkhqPlVbDxQhNujEeewMiwygPgQGwaDgUB/M3EOK6mxwYwbEkKgvxmT0YCfuXPj\nnNtq8kkdGtqTb92n9Nvg+k9mDiu3HT/tck6nE/vGDvs4t3LdpCFck5FwtqUJIXxIg8dL1okqsvIr\n8Xg1TpyoZEfVMQA0Dcqc9RwvdbLjeDkHC6sJD/QjwM/MiQoXte6OGzUYDZAaGwzAXz/LJibYyi3n\nDeOq9DjGxAf7zPkiX9Rvg6svVFdX86tf/YqqqioKCwtZuHAhaWlpPP7443i9XqKjo3nqqadQVbXN\nNF8bjkWIgarC6eZgURU5ZbV8phbx4d4CquoaTlmquNWzsAA/0uKCOT8lkrKaepz1Hi5UoogLsRLr\nsOFyeyisqiPEbiHWYSUuxEZCqA27n/4V6nJ78DMZMRolrHpDvw2uazISOrV31J1DoRw9epQ5c+Zw\n2WWXUVBQwKJFi7DZbDzzzDMMHz6cf//732RnZ/PII4+0mZaWltYtNQgh2vJ4NQ4WVpNXXktueS35\njXtDXk2jqKqOoqo6vJpGmdPNwcLq5vWCrWZmj41hxshIxsY7sFpMHDhwgJEjRzYv47BZsPmZzqo+\nq+Xs1hdd02+Dqy9ERETw6quvsn79egIDA2loaKC4uJjhw4cDcO21er/q9qYJIc6Oy+2hqEpvSVfu\ndJNXUcuJ8loOFFbzwZ6CVq3sTEYDNosJAxAe6EdUsBWLyUhSuJ254+NIi3MQF2JjWERAm/NIZQFm\nYhxyhMSXSXC1sHz5csaPH8/ChQvZvHkzn332GVFRUc0tepYuXcqwYcPanXbppZf2dflC9FvHSpyo\nBVXNzz1e/bzTnrxK3B4v5bVu9uZV4Pa0vdqRzWLiolGRzEyNJjE8gLgQK1FBVkxyWG7QkuBq4aKL\nLuKxxx7jvffeIygoCJPJxKOPPspDDz2E0WgkMjKSm2++mejo6DbThBjMKmrdfLS3gIpaN9HBVo6U\n1LAnr4L6Bo2CShe7cyvarGM0wIioQGx+ZmwWIz+cPozhEYEYDBBssxDnsBEbYiU8wE8aOohWJLha\nmDZtGmvXrm0zfcWKFa2ep6ent5kmhK9zuT3Ue04O11PlamDHsXLU/Epyy/WOsvmVLuIcNsbEO6ip\na2B/ThHVHxZzoKC61boASeF27H5mAv3NPDh7FFOTwzG32EtKDLcTZB3YHWVFz5DgEmKA8Xi15sNo\nXq9GcXUdJypc2PxMhAX4sT+/it25FeSV15LXOHLDifKOB1k1GiA62Eqsw4oSHcTREidLN2YT6G8m\nzGZkWFQA542IYPaYGBLDA8ivcBEXYiXE7tebb1sMIhJcQvigoqo6Mo+WUVDpQtM0Sp1ucsr0vkiH\nimoItVuw+5kprHK1e94IIMhqJs5hIy7ESnpCCLHB1lat6/wtJtLjHaTGBrdp4NAUju216g0LkMAS\nPUuCSwgf4HJ7+FQtZP3eArYdKeNYaevLmhsMEB1kZUx8MFeMiaXMqfdFig62Eh9iJcZho9btobiq\njuTIAMYPCTmrPSJpGCH6kgSXEP2EV9MorHLxVXYJf//iMKU19VyeFkNpTT0f7i2guq6BsAA/piSF\nsWhaIhMTQ0kKt2MwGAiymrGYOjd8kBC+ToJLiD5SUeumpLqO6roG3thynNXbj1PXcBiA5MgAUqKD\neO2rI9j9zFyZHsuV6XFMSw7DLAElBjkJLiF62eHiGl76/BD/ycyhrvGie35mIxckBTB9dCIjogI5\nJzkco9GAs74Bi8koe1NCtCDBdQYWLVrEo48+2jx6hhAdqXS58TcbMRoMfHGwmDe3HGP93gIsJiPX\nTIxn6jA9oM5JDqc45xCpqUmt1m8aC08IcVL//avY8QZ884/TLjbUWQObAzq3zQk3wvgFZ1mYEO1z\nuT18sq+QHcfLyS2vZW9eJYcbr1TrbzZS1+AlxG7hzotGsPicJCKDWl9Xqbi9jQoh2ui/wdUH7rzz\nThYvXsyUKVPYvXs3Tz75JGFhYa1Gi1+4cOFpt7Nu3Tr++c9/0tDQgMFg4C9/+QuhoaH87ne/Y9eu\nXVRVVXH//fdzySWXNE9zu93cddddzJw5sxfeqegumqZxpMTJ8i8Os3p7DjX1HvzMRuIcVkZEBTE/\nIwGPV6Pc6ebc4eHMSInA3ywDsgpxNk4bXIqiGIEXgHFAHXCbqqoHG+eNB/5fi8WnAXOBLcB+4NvG\n6WtUVX2uS5WNX9CpvaNj3Tg6/LXXXsuaNWuYMmUKq1evZurUqaSkpLQaLb4zwXXkyBGWLl2KzWbj\nkUce4YsvvsBms1FWVsaqVavYunUrmzZtQtO05mkVFRW8/PLLElz9UFlNPQeLqvF6NQqq6th+tIzs\nIn2k8hMVLpz1HiwmA1eNi+OaiQlMHSYNKIToSZ3Z45oLWFVVPUdRlGnA08DVAKqq7gAuBFAU5Vog\nV1XVdYqizATeUFX1rp4pu2fMmDGDJUuWUF5ezrZt21i2bBlPP/10q9HiOyM8PJwHHniAgIAADh06\nxPjx4zl8+DDjx48HIDAwkHvuuYelS5c2T3M4HNxzzz099t5E53m8GlknKnn/2xN8sKeg1WUyAOx+\nJkZEBTIyKojzUyIZEmrnyvRYooJlxHEhekNngms6sA5AVdXNiqJMOnUBRVECgN8C5zdOygAyFEX5\nDCgEfqaq6onuKbnnGI1GZs2axaOPPsrMmTPbHS3+dKqqqvjTn/7Ep59+CsAtt9yCpmkkJyezbt06\nAGpqarj11ltZsGBB87Sqqiruuece/v73v/fY+xMnVbncfLKvELufGYfNQmGViwMF1Ww/VsY3x8qp\nrmvA1NhoYt7EeEbHBuNnMhJsszAqJkj2qIToQ50JrmCg5dDOHkVRzKqqttz9uBX4t6qqTeeX9wGZ\nqqp+pCjKDcCfgfmnbjgrK+sMyz7J5XJ1y3aapKen88ILL/DCCy9QWFjISy+9xKpVqwgICMDj8bBr\n1y5qamo4dOgQ9fVtx3bTNI0RI0Zw9dVXYzQaCQwMZM+ePcyfPx+Px8PcuXNxu90sWLCAuLi45mke\nj4cf/OAH3fpeulN3f869oWXNTreXDw9WcbzCTX2Dl03HanC6Ww+FZDRAUogfFybZSY30Z2KcnRCb\nCXCDtwS8+sMDlT1fr6+QmnuH1NyaQdPaH8esiaIozwCbVVVd2fg8R1XVhFOW+RqYr6rq8cbnQYBT\nVVWPoih2YLeqqq3ajmdmZmoZGRln/Qa68wrIvUVq7jkNHi/vfZvPv7Ye48CJCmrcGlHBVoqr66hy\nNRBqt2AyGpiaHM7N5ybhZzJSXusmKsifIWF2Av37rr2Sr3zGLUnNvWMw1pyZmUlGRka7Y4t15q90\nE3AVsLLxHNfuljMVRXEA/k2h1WgZ8B9gJXAJkHkmhfdnu3btYsmSJW2mz549u1MNOETXaZrG7twK\nPtlXSG29h6o6/bIbR0tqmpdp8GrUNXgZFhHA2GgrQ2MjKKqqw99sYtE5iYwfEtKH70AI0R06E1xr\ngEsVRfkSMAC3KIpyH3BQVdW3gRTgyCnr/BJYrijKT4Aa4LbuK7l/SE9P5/XXX+/rMgakCqebbUdL\nafBqlFTXs/1YGQcKq8ktc1JcXY/RoI804Wcykp4QwtTkMEyNFxo0GGByUhgzU6NR1X0+9ytVCHF6\npw0uVVW9wB2nTN7XYv5W9JaHLdc5DFzUHQWKwUHTNL7KLuHlL4/wqVrY6lIcYQF+jI4N5uJRUUxK\nDOPytBgcdrkAoRCDlXRAFn2i0uVmw75Cth8t42BRNUdLnOSU1RIR6MdN5yRx6ehoAq361XOHhtnl\n0u1CiGYSXKLHHStxUl3XQESQHwcLq/kkq5A3tx6nuq4Bu5+JlOggxiWE8NOLRvD9CfFYLTKyhBCi\nYxJcoke4PV6+zC7htS+P8PG+wlbzTEYDc8bGctO5SYxLcEifKCFEl0hwiW5TWlPPOzvz2HKklC8P\nFlPmdBMW4Mfdl4wkJTqIwioXSREBTBwSKueohBBnTIJLnLWjJTUs+/ww/848jsvtJT7ExgUpkVwx\nNpbzUyLl0J8QoltJcIkzdrzUyZIPVNbuysNsNDJ3Qhy3Tk9GiQnq69KEEAOYBJfoErfHy7YjZbyz\nK49VmTmYDAZuP384PzwvSQaZFUL0Cgku8Z3KnXoH4A+2l3Jk41fszCnH5fZis5i4elwc912WQqzD\n1tdlCiEGEQku0YrL7eG37+zhw70FaBqU1OgDCRsNMCbewfWThzJlWBgXKpFyWXkhRJ+Qbx7R7ERF\nLbe/lsnu3Aq+Ny6OIKuZWIeVjMQwrDX5TEhP6+sShRBCgkvoDhZWsfjvW6h0NbBs8SRmjo5uNT8r\nq7CDNYUQondJcA1SOWVO9p2oIre8ll05FXy4Nx8/s4l//XgaaXGOvi5PCCE6JME1iLjcHlZvz2Xl\ntuPsOF7ePD0i0I/zRkTwy9mjSAwP6MMKhRDi9CS4BgG3x8s/Nh/l+Q0HKa6uZ1RMEA/MGsW05DDi\nQ2xEBvnLILZCCJ8hwTVAaZrGxgPFbNxfxMdZBRwpcXLu8HD+vGAk05LDJKiEED5LgmuA8Xo18ipq\n+f27Wbz/bT7+ZiPjhoTw6zmjuSQ1SgJLCOHzJLgGiAqnm9+u3cPanSeo93ixmAz8YpbCbdOT8TPL\n6OtCiIFDgsvH5ZbXsnZnHi9vOkJxdR3XTxlCSnQQ5w6PYERUYF+XJ4QQ3U6Cy0ftOF7OCxsO8mGW\nPsLFhKEhvLR4EmMTpCm7EGJgk+DyMZqm8eJn2Sz5QCXYauEnFw7nuklDpBm7EGLQkODyEZqm8fXh\nUp7fcJDPDxRz1bg4/jBvLIH+8k8ohBhcTvutpyiKEXgBGAfUAbepqnqwxfzngOlAVeOkqwELsAKw\nAXnALaqqOru39IGvqf/Viq+PkVNWS63bQ3iAH7+5ajQ3n5skLQSFEINSZ36uzwWsqqqeoyjKNOBp\n9HBqkgFcrqpqcdMERVH+BKxQVfUVRVF+CfwYeLYb6x7wsk5UcueK7WQX1TApMZQZIyMZFRvE98bF\nyRWFhRCDWmeCazqwDkBV1c2KokxqmtG4NzYSWKooSjTwd1VVlzeu83jjYu83Ppbg6gRN03hrRx4P\nrt5NsM3MssWTpP+VEEK00JngCgYqWjz3KIpiVlW1AQgA/gw8A5iADYqibDtlnSqg3aZuWVlZZ1p3\nM5fL1S3b6U0d1XyotI6lW0vYme9idJQ/v7owmjBDKfv2lfZBla0NpM+5v/K1ekFq7i1Sc2udCa5K\nIKjFc2NjaAE4geeazl8pivIJ+rmwpnVqG+/LaUdqauoZln1SVlZWt2ynN51ac1FVHc98qPLm1lxC\nbBZ+d3UaC6YMxWzqPx2HB8Ln3N/5Wr0gNfeWwVhzZmZmh/M6E1ybgKuAlY3nuHa3mJcC/EtRlAmA\nEf0Q4auN61wBvALMBj4/k8IHuvoGL3//4jDPbziIy+3h1vOGcdfFI3HYLX1dmhBC9FudCa41wKWK\nonwJGIBbFEW5DzioqurbiqK8DmwG3MBrqqruURTlMeBVRVF+BBQDC3uofp+1N6+S+1buYF9+FTNT\no3noilEkR8pIF0IIcTqnDS5VVb3AHadM3tdi/hJgySnrFACzuqPAgaSspp73v83nX1/lsbvgEGEB\n/ry0eBKXnnK1YSFED/N6wFXRdnpNMWS9BXk7QNM6t63QREj7PoSPgLoq2L8OjnwOXm/bZeMmwOjv\nQUBk23n+wWDqhX6ZDfVQX912utkKfvbGZeqgvqbjbfgHganxyFB9jb58E79AMPt1X73tkN6rvUDT\nNH63NovXvjpCg1cjPtjCnReN4ObzhhEW0LP/wEIMKm4XZH8Mxfs7XCTm8E54ZyM4SzreToQCps78\nbWpw8CPY/ELryaHD9C/wljz1oL4LGx5rf1O2MD3UQpNOTvMLhJTL2y5btF9/nw2uTtTYVKoGBXtA\nfR/c7YSS0QLDLwa/AD183d/R9dY/GJQroDofDm8ErUVIx6TDHT17dkiCqxc89/EBlm86zLUZCdx0\nbhLGilxGj1b6uizRk0oPQ2Vel1YJzNkNWf8Pcrd1/td+Z4QMgdSr9HoOrIf6dr6QQhNh9Fx9r6GJ\nxQax48BogrKjsGcN7FsLzpOtXJPr6+HDFl/wYcn63kfLL9+u0rxwbLP+Je+q7Nq61YVQX/WdizhM\nVhh1BQyZAoZTGkCZrTDiEnAkdP41XZX65+os0T+rpBkQ2cHfd0WuHnSnBo7mhZxtsOvfbUPlvf9l\neEAcfGjTn3vqoeJ45+tryRYKY+dD1Gg4tYtN+THY+5YeWOk/gKgOGlZoGuTv0v8v2MPhvLshKPbk\n/MhRZ1ZbF0hw9SBnfQPPbzjI8xuymZ+RwJPz0zEYDGR18QtN9BKPu+00V6X+BXpsc+tflR3RNCjc\nq/9hd9EQAKtD/+IzW7u8fgcF6Ye93v25/iWdeJ6+N3HqMrnb4Z2ftV09MAaCYyHvG/153ET9cFcj\nV2Ul/sHBjZvx6qH71k+6p/QhU1u9VqdYg2HUlTB0WttQaqTuP0hq2thuKLDFa46d37llHfGQcVPH\n8z0N4G3x/7DqBOxZQ+2Bzfg1fc4GA8Tfqf8YsYd1rVaTPxi/o7Xy5b/vwsZeOP0iPUSCq4dsO1LK\nnSu+Ib/SxbwJ8fxh3ljpRNwVmqZ/WVYch+SL9C+H9pbJ3Q6VuTD8In3awY+bf/n6eWKBxl+NDfX6\nIY3yo223U10Ae/4LxWrH9QRE6nsgnREYA5c/DtFp6O2ZOudIzgmSzp3X/ecHNA2KVP3XcWA751aa\nlinY0/rwWXUh7P2v/vnMfLTdPam8rCwcLZs8N/0ar223B0znhY/Qv+R7grEff+2ZzK3Pc4Ulw4yf\nkxdxyuc8yPXjf0Hf9dHeAn66YjtxITZW3XEOk5K6+KvIF1XkQu0pHaU1DfJ364cUrCGQNheC49pf\n39sAR74AdZ1+4thZcvJwiMlfP/Ry6i/ommKozNEfN+2htDgEMxzg65H6MfuyI+Dq6MvUoO+JjJmn\nH+ppyWiBYefrv/x7+IdHbV1Wz5zUNhgg6jSHbwwGiBnTdnr6tV1/rdhxXVtHiC6S4Opm246U8uN/\nZJIWF8zLN08mPNC/r0vqWW4XfPo4fPnnjg+lOYbqLbh2rjj99mLS9ePlIUPhgl/ovziz1kLpobbL\nhibCyAchJFEPR02D0Vfrv9brqyn4/FWiaw/oLcii0/RDK3ETaLMXZLHqh+iEED5BgqsbVTjd3P3m\nDuJDbPzjtqkEW324I7HXA8e+gsoTAATn5ULVF5D1DuRsPdl4wNt4TH7iYhh5WdvtBMfrYeGph6Nf\ntt8Mt0l0mh5Up0qafvp6h804dWOUKguIlsMrQgw4ElzdxOX2cN/KHRRUuvjP/5zrm6F1+HP44lk9\nZIr36+c2GjWfbQhLhgmLwNxiT3L4xSfPMXXE7H/6ZYQQohMkuLpBbnktd7yeye7cCn53dRrjhoT0\ndUldd/hz+Oe1eiulkKGQeK5+2C16DGAg+1A2w0eO0udJIxMhRB+S4DpLX2YXc+eKb3A3eFm2eBIz\nfW0UDE2DnW/qzaVDE+Gmte22PKsvcuvzhRCij0lwnaEGj5e/bTzEMx/uJyncztLFkxjeX8carKvS\n+yhV5unNmwtbXGqgpgiOfw1DpsF1r3XcXFoIIfoJCa4uqGvw8MKGbAqrXOzJq2RXTgVz0mP547yx\nBPW3c1pVBXov+D1r9EYWNDamMBghIuVkXxaDAS57DKb9pG1TcCGE6IckuDpJ0zR+sWoXb+3IIzLI\nn0B/M39aMIHvjeugX1J3c5ZC0b6O51fl63tT+d/qzdLLj+r3UaP1ZuX2cL0/08jLIDCqd2oWQoge\nIMHVSU9+oPLWjjz+97IU7rx4ZM++WEMdZH+ij3/mrtVb9x3eqDc9/y4BkXpHWqMZ0q/TRzroaLwx\nIYTwURJcnfCXTw7w4qfZLJgylJ9eNOL0K5ypvB2wZane4bauQh+B2RYCFrt+KG/YBR1f9sASoPeX\n6o3LIgghRB+Sb7nT+Ntn2Ty1fj/zJsTz2NwxPTfe4JFN8M/5+t7SqDkw5hpIvvDkNW+EEEIAElzf\nafkXh/nD+/u4Mj2WJ+enYzL2UGgd+hTevEG/lMLN78o5KCGE+A4SXB14ffNR/m/tXmalxfDsD8Zj\nNn3HpQC6yOiugX3v6de9Of61fngwQoGb3pbQEkKI05Dgase/th7j4f9+y8zUKP60YAKWMwktTdP7\nTXnd+kUF96yBssPg9TAyZxt4WlzqeuodcMlvTl42WwghRIckuFoorq7jmQ/388aWY1yQEsnzN0zE\nz9zF0CrYC3tWw7eroTT75HS/QH0QWYOR8mFXETb9FgiM1qcHx3a8PSGEEK2cNrgURTGiX+pyHFAH\n3Kaq6sEW8+8Frm98+p6qqr9VFMUA5AAHGqd/parqg91aeTfbsK+Qu9/8Bme9h5vPTeKBWaPwN3ey\nQ27xAT2o9qzW+1oZjPpVbKf+GPyD9MtlJ1/YfCHCgqwswpKkmboQQpyJzuxxzQWsqqqeoyjKNOBp\n4GoARVGSgRuAqYAX+EJRlDWAE9iuqupVPVN299E0jRc+zeap9SqpMcH8acEERkR1YuimsiMnwyp/\nN/rFCM+FK57SB6eVc1VCCNEjOhNc04F1AKqqblYUZVKLeceBWaqqegAURbEALiADiFcUZQNQC9yr\nqup3XBe9b3i8Go++vYfXNx/le+PieOKadGx+p9nLqq+BD38DW1/SnydMhll/1MOqo6v7CiGE6DYG\nremCgB1QFGUZ8B9VVd9vfH4MSFZVtaHFMgZgCRCkquqPFUU5H4hWVfXfiqJMB55VVXVyy+1mZmZq\ndvvZN0ZwuVxYrdYur+fxajz1RRGfHq5mfpqDH2aEtdtHy1RXQfDRDwjK2YDZVYKpvhJzXTmlI6+j\nRFlAQ0DXz0+dac19SWrueb5WL0jNvWUw1ux0OsnIyGi3D1Jn9rgqgaAWz42nhJYVWA5UAT9pnLwN\naABQVfULRVHiFEUxqKraKiVTu+HqtFlZWV3ejtercf+qXXx6uJr7L1fajobh9eitAtV3Yf294CyB\nqDRInKx3EJ6wiLBhMwjrxZr7mtTc83ytXpCae8tgrDkzM7PDeZ0Jrk3AVcDKxnNcu5tmNO5pvQV8\noqrqEy3W+Q1QAjypKMo44PipodVXNE3j/9bu5T/bc7jv0pS2obV7Fbx1JzTU6s9jx8OiNRA7rveL\nFUII0UZngmsNcKmiKF8CBuAWRVHuAw4CJuACwF9RlNmNyz8I/BH4h6Ioc9D3vG7u7sLP1LLPD/PK\nl0e4bfowfnZJ42C5zlK9z1XeN/DO3ZAwCUZcqjewGL9Qhl0SQoh+5LTBpaqqF7jjlMktr6/R0UHM\nOWdaVE/5bH8Rv38viznpsTx0Rap+OHD7q/DBr6C+Wl9o6Dlwwyrw76cXhRRCiEFucHRA1jRqvn0X\n95rn2W3dSeBhDcMf0K9X1eCCYefDpFvB7K+PwC4jWAghRL818IOrught7d0E7HuXCVow9cocDBHR\nJ+dHpsK4BWDsvrEIhRBC9JyBHVxlR+DlOXiqCnjCvRDb+Xdy3+VpfV2VEEKIszAwgitvh36VYFcF\n7FsLRSoMvwSy3sHlrOQa16OkTpzBg5eO7utKhRBCnCXfDi5XJbFbfg+H3zk5LSBSv1z9lqW4TVbm\n1zzAsLHn8sQ16Rh76npaQggrF1t6AAAgAElEQVQheo1vB9fae3AceRem3wupV4HRAlGjwWTm0PEc\nbl72JSHxcTx17bieuwikEEKIXuXbwTXj5xyOn0vyOd9rnqRpGv/JzOG3b+/BzxzKmzdmYLV0cpR3\nIYQQ/Z5vB1d0GnWlJ1sDZp2o5LF397LpYAlTksJ46tpxxIXY+rBAIYQQ3c23g6uFjfuLuOWVrQRZ\nzfz2e2ncOC1RDg8KIcQANCCCq7DKxX0rdzA8MoCVPz6HELtfX5ckhBCih/h8cLkavNzz5g6q6xpY\n8aNpElpCCDHA+XRwHS918vP38zhcVs+T16STEh10+pWEEEL4NJ8Orsffy6KgqoHlN03molFRfV2O\nEEKIXuDTwfXIVaM5PNqPcyW0hBBi0PDp4Ip12Ci3+fRbEEII0UUyJLoQQgifIsElhBDCp0hwCSGE\n8CkSXEIIIXyKBJcQQgifIsElhBDCp5y2LbmiKEbgBWAcUAfcpqrqwRbzfwT8GGgAHlNVda2iKBHA\nCsAG5AG3qKrq7IH6hRBCDDKd2eOaC1hVVT0H+CXwdNMMRVFigJ8B5wGXA39QFMUfeARYoarqDOAb\n9GATQgghzlpngms6sA5AVdXNwKQW86YAm1RVrVNVtQI4CKS3XAd4H5jZbRULIYQY1Doz7EQwUNHi\nuUdRFLOqqg3tzKsCHKdMb5rWRlZWVpcLPpXL5eqW7fQmqbl3+FrNvlYvSM29RWpurTPBVQm0HHbd\n2Bha7c0LAspbTK9tMa0Np7N7Tnt113Z6k9TcO3ytZl+rF6Tm3iI1n9SZ4NoEXAWsVBRlGrC7xbwt\nwO8VRbEC/kAq8G3jOlcArwCzgc9P3WhGRoZcnlgIIUSXGTRN+84FWrQqTAcMwC3ooXRQVdW3G1sV\n3o5+vuxxVVX/oyhKNPAq+t5WMbBQVdWannsbQgghBovTBpcQQgjRn0gHZCGEED5FgksIIYRPkeAS\nQgjhUyS4hBBC+BQJLiGEED5FgksIIYRPkeASQgjhUyS4hBBC+BQJLiGEED5FgksIIYRPkeASQgjh\nUyS4hBBC+JTOXNakR2RmZsrovkIIITrU0eWv+iy4ADIyMs56G1lZWaSmpnZDNb1Hau4dvlazr9UL\nUnNvGYw1Z2ZmdjhPDhUKIYTwKRJcQgghfIpPB9fT61VW7i7v6zKEEEL0Ip8OruLqelbsLKPcWd/X\npQghhOglPh1ci89JpM6j8a+tx/u6FCGEEL3Ep4MrNTaYsdFWXt98FI9XWtcLIcRg4NPBRdZa7og7\nSE5ZLR9lFfR1NUIIIXqBbwfXvrXM2v8bpjgqePHTbDRN9rqEEOJUq1ev5qmnnurrMrqNbwfXxb8G\no5E/25ex83gpnx8o7uuKhBBC9LA+HTnjrDkSKJhwL3FbHuP+gHU893E4M0ZGYDC0O0qIEEL0qf9k\n5rByW9cbkzmdTuwb2+/6c92kIVyTkdCp7Sxfvpx3330Xs9nMpEmTuP/++8nMzOSJJ57AbDZjs9l4\n7rnnKCoq4sEHH8RsNuP1enn66aeJjY3tct09xbeDC6hImkNc9bfcsXcFm44P4fMDIzk/JbKvyxJC\niH7l6NGjfP3117z55puYzWbuuusuNmzYwJYtW5g9ezY33XQTn3zyCZWVlXz55Zekp6dz//33s23b\nNqqqqiS4upXBAFc/D0X7eL7oL9zzTjLn3XstJqPsdQkh+pdrMhI6vXfUUneMVZiVlcWFF16IxWIB\nYNKkSRw4cIA77riDv/71r9x0001ER0eTnp7O/Pnzeemll7jtttsICgri3nvvPavX7m6+fY6riX8g\nxutXYPcz8mjFr1n75Y6+rkgIIfqV1NRUdu3aRUNDA5qmsXXrVoYNG8bbb7/N97//fV5//XVGjhzJ\nypUr+fjjj8nIyODVV19l1qxZLFu2rK/Lb8X397iahA/HsmgVMX+/EtdHt1A95kMCQ+SQoRBCACQm\nJjJx4kQWLFiA1+slIyODmTNnsmvXLn79619js9kwGo383//9H5qm8cADD/Diiy/i9Xp58MEH+7r8\nVgZOcAGGIVPIuWwpwz64lbK/zSbwznUQENHXZQkhRJ+aN29e8+Nbbrml1bxx48axcuXKNuu88cYb\nPV7XmRoYhwpbGHHuXFYMfxKH8yiuly6H8mN9XZIQQohuNOCCC2DetYv5mfnXNFScQFt2KeTJOS8h\nhBgoBmRwOWwWvj/3Or7v+g1V9RosvxwyXwUZWUMIIXzegAwugCvGxjJ2/FRmVv+WyqjJ8M7PYM0d\nUF/T16UJIYQ4CwM2uAAevToNS3A0c8vvoX7GL2HXv+Cli+XQoRBC+LABHVzBVgtPXzeOw2V1PFp5\nJSz+L9SW6+H10aPgru3rEoUQQnTRgA4ugGnJ4dw+I5kVXx9jfe0o+OlmGL8AvngW/jodjn7Z1yUK\nIYToggEfXAD3XZbC2HgH963cyYFKsz5E1KL/gqceXp4Nq34I5XIVZSHE4LZo0SKys7M7nH/xxRdT\nV1fXixW1b0B1QO6Iv9nE0sUZXPXnTfzotW289dPpOIZfBD/ZDJue02/73oPp98C5d4FfQF+XLIQY\niHa8Ad/8o8urDXXWwOYOvpcm3KgfRRpEBsUeF0Csw8Zfb5xIbnktd76xnQaPVw+oix6CO7eCMhs+\n/QM8Nw6+egHcrr4uWQghusWdd97Jli1bANi9ezeLFi3i7rvv5oc//CFXXnklK1as6NL2cnJyWLx4\nMTfccAM33ngj+/btA+DBBx9k4cKFzJs3jw0bNgDw7LPPcv311zN//nyWLl3aLe9nUOxxNZmUFMZj\nc8fwwH928/h7+3jkqtH6jJChcO3LMO1/4JPH4IMH4cs/wYyf679mLLa+LVwIMTCMX3BGe0fHznJ0\n+GuvvZY1a9YwZcoUVq9ezdSpU0lJSeGyyy6joKCARYsWsXDhwk5v78knn2Tx4sXMnDmTrKwsHnro\nIV577TW2bt3aPHxU0/0777zDa6+9RlRUFKtXrz7j99DSoAougB9MHkrWiSqWbzpMfKiNW6cPOzlz\nyBS46W04/Dls+D2897/6XtiU22HyjyAgvO8KF0KIMzRjxgyWLFlCeXk527ZtY9myZTz99NOsX7+e\nwMBAGhoaurS97OxsJk+eDOijzufn5xMYGMhDDz3Eww8/THV1dfP8JUuW8PTTT1NcXMyMGTO65f0M\nmkOFLf16TiqXp0Xzu7V7Wb09p+0Cw2bALe/Dze9CwmQ9vJ5Ng3d/DqWHer9gIYQ4C0ajkVmzZvHo\no48yc+ZMli9fzvjx43nqqaeYNWsWWhdHFRo+fDjbtm0D9Ot8RUREUFhYyJ49e3j++edZunQpr776\nKvX19axbt45nnnmG1157jTVr1pCbm3vW72fQ7XEBmE1Gnrt+Are8vJX7V+0ixG7h4lHRrRcyGCBp\nun4r3Adf/Rm2vwbblsOIS2HSLfq9aVB+hEIIH3PNNdcwc+ZMPvjgA3Jycnjsscd47733CAoKwmQy\nUV9f3+lt/eIXv+Dhhx9m+fLlNDQ08Pvf/57IyEiKioq4/vrrMRqNzJ07Fz8/PxwOB9dddx1Wq5Xz\nzjuPuLi4s34vhq4mbXfJzMzUMjIyzno7Z3Nl0CqXmwUvbeZAQTWv3zqVKcPCTrNCPmxdpgdYdQEE\nxennwCYu0s+T9ULNfUVq7nm+Vi9Izb1lMNacmZlJRkZGu5eyH9S7C0FWC6/cMoXr/voVt7y8heU3\nT2Zq8necxwqKgYt/DRc8APs/gMxXYOMS/Tb8IhgzH1KvBKuj196DEEJ0p127drFkyZI202fPnt2l\nBhw9aVAHF0BEoD9v3D6NhS9t5uaXt/L3myZx7ojTXHzSZNEDKvVK/Xpf21/Xx0F86yew9l4YeSmM\nnQ8jLwc/e++8ESGE6Abp6em8/vrrfV3Gd+qR4FIUxQIsB5IAf+AxVVXf7onX6g7RwVbevP0cbli2\nmVte2cpLiydxfkpk51YOGQoX/0rvD5abCbtXwZ7VsG8t+AWCcgWMmQfDzpeOzUII0Q16qlXhjUCJ\nqqozgFnAX3rodbpNZJA/b/xoGsMiArjttW2s+za/axswGCBhEsz+I9yXBYvfhjHXwIH18Mb18MQw\neP37sPlFLFVyVWYhhDhTPRVc/wYebnxsALrWSaCPhAfq4TU6Npif/DOT1zcfPbMNGU2QfAF870/w\nvwdg0RqYfBtU5MC6XzLivevgufHw3v2wfz3UVXfvGxFCiAGsR1sVKooSBLwNvKSqaqsxRTIzMzW7\n/ezP/7hcLqxW61lvp9U2G7z84bNCtuQ4uX5sCIsnhGIwtNu4pcss1Xn4H99ISNFWAgq3YfTUoRlM\n1IaNxhmdQU1UBrXhY9HM3fuezlZPfM49zddq9rV6QWruLYOxZqfT2WGrwh4LLkVRhgBrgBdUVV1+\n6vz+0Bz+uzR4vPz6v9/y5tbjXDMxgcfnjcHfbOqWbTfX7K6FY1/pI3Uc3gh534DmAZOf3vE5aQYM\nnQrxGX3eUnEwNsftbb5WL0jNvWUw1tzrzeEVRYkG1gN3qqr6cU+8Rk8zm4z8Yd5YYh02nv1oP0dK\navjrjRlEBvl334tYbDD8Yv0G4KqEY5vhyEY9yD57Amj8YRGh6GGWkKHfR6ZK52chxKDUU998DwGh\nwMOKojSd65qtqqpPXXLYYDBw98yRjIgK5Of/3sH3/vIFLy2exJj4Htr7sQZDymX6DcBVAbnbIWcb\n5G6D/e/DjsZLIlgCIGYMRI9pvB8L0aOl5aIQYsDrkeBSVfVu4O6e2HZfmJMeS2K4ndtf28b8v37J\nH+elM3dCfM+/sNWhd2wefpH+XNOg7MjJIDuxC3b/G7b9vXEFA4QlnwyymDEQMxaC4/VWj0IIMQDI\nsaZOGhPv4K07p/PTFdu551872HKklEeuHI3V0j3nvTrFYICwYfot/Vp9mqbpnaALvoX8byF/F5zY\nCXvfOrmeNQQiFQgfCRFNtxQITdI7UwshhA+R4OqCyCB/Vtw2lafW7+evn2Wz83g5L9wwkcTwPjw8\nZzBAaKJ+GzXn5HRXJRTuhfzdULAHig/ofcp2tLj6qtGsh1dECoSP0O+bQs1+mnEbhRCij0hwdZHZ\nZOSXs0cxOSmU+1bu5Mo/fcFvr07j+xPiu63JfLewBsPQafqtpdpyKDmoB1nxfig5oD8++BF4WowO\nbQ3RwzBkKIQkElrnD8YpEByrDy5sDwfjoLwqjhCij0lwnaFLUqNZe9d07lu5g/tW7uSjrAJ+P3cs\noQF+fV3ad7OF6CN8JExqPd3TAOVHT4ZaabZ+CLJIhQMfEtPggu0tljdaICi2MchiITiuxX3Mycdy\n9WghRDeT4DoLQ8LsvHn7OSzdeIhnPlTZdqSMJ+enc6ES1deldZ3JDOHD9VvK5a3naRr7v9lESqQ/\nVOVB5Qn9viofKvP082sHPgR3TdvtWkNahFosBESCPQICIhrvw/V7e7gMSCyE6BQJrrNkMhr4nwuH\nc35KBPf9ayc3v7yVG6cN5aErUrH7DZCP12DAYwuHId/RmVDToK6yMdQab5V5jfeNzwv3Qk0ReDsY\nAcxibx1mAY2BFhABtjC9laXVoe81Wh16KPoHS382IQYZ+YvvJmlxDt668zyeXq+y7IvDbNhXxO/m\nprW9svJAZTCcDJaoUR0vp2l6/zRnCdQUg7O48b7klGlFULRPf95wmu5/foEng6ypBquD6FoPnBjW\nIuha3PyD9L5wfgF6YMr5OiF8hgRXN7JaTPxqzmguT4vhwdW7+eEr27hibAy/uSqN6GDfGmesxxgM\nepDYQvTDkp1RXwO1ZXrguSr0BiZNj5tvLaZV5kDhHhw1pXCgkwMYN4WYX4AehH721s8tdv18ndkK\nFiuYbae/N/vrw3eZ/fVuB6bG5xKSg4PXqw/h5vWcct/16baibLCV6kcrznAb+n0DeNwt7t2nPG/4\njnlu/Vx4h9Mb58VPgoVv9uhHK8HVAyYlhfHuz2bw0ueHeO7jA3y+v5hfzFK4YWoiRmM/annoK5oC\nxJHQpdX2Z2WRqqRAXVXbgKur1s/J1Xd0q9bXq8rXH9fXgNul7/1p3rN7PwZT2zAzWUj2AJ8FtpqG\n2V/vtmA06fcGU4vnptbPDcbGm6HF4/ZuBn299uY1azGGaavxTFtPDyssgLLo1tNPs84ZT9e0bggC\nLwmV5bDdrj/3Nuj/nm3W9XQwvbOv1b0XxEjq1q21YLTo/8+MFv3/UNNjk7nFPHOL6ZaT/3dbLmc0\nn3wcN76nqm0mwdVD/MxGfnrRCOaMjeVX/93Nw2/tYVVmDo9clYY0QehFRtPJPbzuoGn6r8yG2pNB\n1tF9Q53excDjbvG4xa2h9fO6shL87f4t5rv0kG3+gm28957yBdn05atpjfcd3HpAnx0IN7QM7qZ7\nYzvTjW2Ws9S7QbO3CPymeZZObrMzr2Xucl3fNf1YTi5DE5Na/Hg5022bWwdVf+rC0wUSXD0sKSKA\nf9w6lbd25PH4e1lc8+KXXDgsgMdjk4gLkabiPsdgALOffuvmEftzs7II7skRwJuCrWlv4nTh1upL\nzdDudHX/AZSUlHaW73idM57etKd4lodaD/vgSOs1DVmQ7Fs19yQJrl5gMBiYOyGeS0dH89fPsvnb\nZ9lc/PSn3H7+cO64IHngtD4U/VvzF3/3DVPmtQTond2F6EXyjdmLAvzN/PwyhYzQOlYdaOBPHx9g\n5dbj3D1zJNdmJGA2yUl7IYQ4Hfmm7APRgRb+snAiq+44h9gQKw+u3s2lz27k7Z15eL09d0VqIYQY\nCCS4+tCkpDBW/8+5vLR4En4mIz974xvm/PkLPtlXQE9dmVoIIXydBFcfMxgMXDo6mvfunsH/+8F4\nauoa+OEr25j7wpd8tFcCTAghTiXB1U+YjHoDjo9/fgGPf38sJdV13PbaNmY/9znv7MzDI4cQhRAC\nkODqdywmIwunDmXD/17I09eOw+3xctcb33DpM5+xcttx3J6e6Y8jhBC+QoKrn7KYjFyTkcD6ey/g\nhRsmYrWY+MWqXVy45FNe2XSYmrru7ZkvhBC+QoKrnzMZDVwxNpZ3fzadl2+eTIzDyqPv7OXcP37C\nE+v2UVDp6usShRCiV0k/Lh9hMBi4aFQUF42KIvNoKcs+P8zfPstm2eeHuGpcHLdNT2Z0nHQEFUIM\nfBJcPigjMYyMxDCOltTw8qYjrNx2nNXbc5mcFMqN0xKZNSYGf3P3jY4ghBD9iQSXD0sMD+DR76Vx\n78wU/rXtGP/8+hh3v7mD8AA/rp00hBumDmVImAzpK4QYWCS4BgCH3cLt5w/ntunJfHGwmH9sPsrS\njdn8bWM2F6REcuPURC5UImVIKSHEgCDBNYAYjQbOT4nk/JRITlTU8saW47y55Ri3vbaNyCB/5k2I\nZ35GAiOjg/q6VCGEOGMSXANUrMPGfZemcNfFI/hkXyGrMnP4+xeH+dvGQ4wbEsL8jAS+lx6Hw27p\n61KFEKJLJLgGOIvJyOVpMVyeFkNxdR3//SaXVZk5PPzfb/nd2r1cOjqa+RkJzBgRIYcShRA+QYJr\nEIkI9Oe2GcncOn0Ye/IqWZWZw1s7cnl31wmigvy5Mj2OOemxTBwagsFHr4wqhBj4JLgGIYPBwJh4\nB2PiHTx0RSqf7Ctk9fYc/vH1UZZvOkx8iI056bFcmR7L2HiHhJgQol+R4Brk/MxGZo2JYdaYGCpd\nbj7aW8DaXSd4edNhlm48RGK4nTljY0kLqmOUpkmICSH6nASXaBZstTBvYgLzJiZQ4XTzwZ583tmV\nx982HsLj1UjeXM7laTFcNjqacQkhGI0SYkKI3ifBJdrlsFu4bvIQrps8hJLqOl75eAfbizRe2niI\nFz/NJirIn5mjo7lsdDTnDA+XkTqEEL1GgkucVnigP1ekBPPzq1OpcLrZoBayfm8+b32Ty4qvjxHo\nb+YCJZLLRkdz0agogq3SxF4I0XMkuESXOOwW5k6IZ+6EeFxuD19ll7B+bz4f7i3k3V0nMBsNZCSG\ncn5KJBekRDI6NlgOKQohupUElzhjVoupecT638/V+OZ4OR9lFfCZWsSSD1SWfKASEejH+SMjuUCJ\nZPqICMID/fu6bCGEj5PgEt3C2LinlZEYygOzRlFY5eLz/cVsPFDEBrWQ1d/kYjDA2HhHc5BNGBIi\nnZ6FEF0mwSV6RFSQlWsyErgmIwGPV+Pb3Ao27i/is/1FvPhZNn/ZcJAgfzPnjYhgRkoE5ySHMywi\nQJrbCyFOq0eDS1GUqcATqqpe2JOvI/o3k9HAuCEhjBsSwl2XjKSi1s2XB4v5bH8RG/cXsW5PPgAx\nwVbOGR7OOcnhnDM8XC7JIoRoV48Fl6IovwAWATU99RrCNzlsFmaPjWX22Fg0TeNwcQ1fHSrhq+wS\nPj9QxJpvcgGID7FxzvBwzh0ezrTkcOJCbH1cuRCiP+jJPa5sYB7weg++hvBxBoOB5MhAkiMDuWFq\nIpqmcaCwmq+yS/gyu5iPsgpYlZkD6EE2OSmUycPCmJIUxoioQDm0KMQgZNA0rcc2rihKEvCmqqrT\nTp2XmZmp2e1nfyjI5XJhtVrPeju9SWruPK+mcbisnm8LXHxb4GJPgYsylweAYH8jaVFW0qKtpEVZ\nGRHuj7lF03tf+5x9rV6QmnvLYKzZ6XSSkZHR7i/TPm2ckZqaetbbyMrK6pbt9CapuWvSgCsbH2ua\nxpESJ1sPl7LlSCnbjpTy1bZSAGwWExOGhjBxaCjpCQ7s3iIm+NDnLP8veofU3DvOtubMzMwO50mr\nQuFTDAYDwyICGBYRwHWThwBQWOli65Eyth4pZeuRUl78LBuPVz+SEPNBIekJDsYNCSE9wUF6fIhc\nPFMIHyfBJXxeVLCVOemxzEmPBaC23sPeExWs37affLc/u3IqWL+3oHn5YREBepglhDBuiIO0OAdW\ni4y1KISv6NHgUlX1CNDm/JYQPcnmZyIjMQy709F8qKLC6WZXbjm7cirYebycrw+V8taOPEBvrq9E\nBzFuiIP0hBDGJYSQEh0onaOF6Kdkj0sMCg67hRkjI5kxMrJ5WkGli53HG8Msp5z3dufzxpbjAFgt\nRtLiHKQnOBg/JIT0hBCSwu3SilGIfkCCSwxa0cFWLkuL4bK0GEBv+HG0xMnOnHJ2Hq9gV045b2w5\nxsubjgAQbDUzNsGBEh3MqJggUmKCSIkOxO4nf0ZC9Cb5ixOikcFgICkigKSIAK4eHw9Ag8fLgcJq\ndh4vZ2dOBXvzKnhjyzFq3Z7GdWBomB0lOohRMUEoMcEoMUEkhdvlUKMQPUSCS4jvYDYZSY0NJjU2\nmOun6NO8Xo1jpU725Veh5lehFlSyL7+Kj7IKaGzMiJ/ZyMioQJToIJQY/TYqJpjoYH853CjEWZLg\nEqKLjMaTe2azxsQ0T3e5PRwsrGZffhX7C6rYl1/FpuxiVjcOYQX6cFd6iAU134+MDpKLbwrRBRJc\nQnQTq8XEmHgHY+IdraaX1dSjFuh7Z/peWiWrt+dSXdfQvEx8iI34QAPjD0NyRADDowIZHhlIWIBf\nb78NIfo9CS4helhogB/TkvWBgptomkZOWW3znpmaX8WuY8W88uUR6hu8J9e1W0iODGR4ZADDI/Uw\nS44MYGiYnEMTg5cElxB9wGAwMCTMzpAwO5ekRgP6EDkpyihyy2rJLq4mu7Ca7KIaDhVV88m+IlZu\ny2le32IykBgewPDIAJIjAxkWHsDQcDuJ4Xaig6wYjXIeTQxcElxC9CMmo4Gh4XaGhtu5SIlqNa+i\n1s2hIj3MsotOBtvHWYU0eE8Olu1vNjIkzE5SuJ2hYQEkNm4vMcxOQqgdP7PsqQnfJsElhI9w2CxM\nGBrKhKGhraa7PV7yyms5WuLkaKmTYyU1HC1xcqzUyaaDJc1N9wGMBoh12Ehs3DtrDrYw/XmQNBIR\nPkCCSwgfZzEZSQwPIDE8oM08TdMoqq7jWImzdbCVOlm/p4CSmvpWy4cF+DWHWGKYvXG7+h5bZKA0\n5Rf9gwSXEAOYwWAgKshKVJCVSUlhbeZXudwcK3XqwVbqbNxTqyHzaBnv7MyjxRFIrBYjCaF2hoTa\nSAi1kxBqw+Csxh1UTkKonVC7RYJN9AoJLiEGsSCrhbQ4fYT8U9U3eMktr+VoSU1zuOWU1ZJT7mT7\nsXIqat36gp8VAhDgZ2oOtPhQG3EhNmIdVuJD9MdRQf7SElJ0CwkuIUS7/MzG5muftafS5eaLzD2Y\nQ2LIKavleJkebMdLnWw7WnYy2BqZjAZigq3EOqzENYZZfIiVWEfTYxvBNrPstYnTkuASQpyRYKuF\nYWH+pKbGtDu/uq6BE+W15JbXcqLCRV7j47zyWnYcL2fdt/nUe7yt1gnwMxEXYiPGYW0OuejGx03T\nwgL8JNwGOQkuIUSPCPQ3MzJaH9KqPV6vRnFNHXnleqjpN/3xiUoX+wuKKKqqa3WeDcDPZCTa4d8Y\nZjZigv2JDtb33GIc+uOoIKs0+x/AJLiEEH3CaDzZcGT8kJB2l2nweCmurudERS0FlS5OVLjIr3RR\nUKE/3p1TzvoKF3UN3jbrhtgtRAb6Exmk3yIaH7urqig0FTXPCwvwwyQdtn2KBJcQot8ym4z6IUKH\ntcNlNE2jotZNfmOwFVS4KKiso7i6jqKqOoqq69hxvJzCyrqTfdq+KGpe32iAsICWAeenP24KvRbh\n57BJy8n+QIJLCOHTDAYDIXY/Qux+jIoJ/s5la+oa2LxjL47ohOZQK26815/Xk11YTVFVXZvzb6AP\ntRXRItAiWoRay726yCB/AvxMEnI9RIJLCDFoBPibiQu2kNpOn7aWNE2j0tWgh1k7AVdcXUd+pYvd\nuRWU1NTjOfVEHHq/t1MDLjzQnzC7hdAAP8IC/Ai16/dhAX5YLaaeetsDjgSXEEKcwmAw4LBZcNgs\njIgK/M5lvV6NMmd9q1BrCrzi6nqKquo4WqJ3EShz1qO1zTgAbBaTHmYBFsICWgSc3Y+6qkoON5xo\nDrrQAAuhdj8sg7RfnPv0a/kAACAASURBVASXEEKcBaPRQHigvjc1qv2eAc08Xv18XGlNPWXOev2+\npp5S5/9v787jo6ruxo9/Zkky2QMJEELYl0NA2VcVS12KoBQ3LOJuBWlFffRnq621Vh/0qQtaylOf\n1gX3XcQNF2ppVUSKBCmI4bCvYcu+zEwy2++POwlDFjJJJpmZ5Pt+vfJi7rnLfOcyM985555zrv/f\nSldt+d6CSoorqymvuW/buoJ6x0u2WUlPjK1NcCfX5IzkZjSjxpAWH0NqQgxx1uiv2UniEkKIdmIx\nm2qbBoNV7fby7X+2kt6rrz/RuSiyV1NUEZD87NUcKXOSd7iMwsrqBntZ1oiPsZCWEFNbozSSmpHc\nUgMep8XHkFKzPiE2oq7ZSeISQogIFms10zXB2mTHk0COak9tcit1uChxVFNidxmP7cbjEoexvLfA\nTomjhGK766SbmNZlMZtIsVlrE16K/6922Wb8OyK7/l3AQ00SlxBCdDDxsRZ6xRrTaDWH0+XxJ7UT\nia7UbjRfljmN5TKH2yh3uDhU7Kgtd3mMi3cDMhJZfdfUNnhVJ0jiEkIIAYAtxkJmquWU4+Ya4vP5\ncLg8lDncJNnaPq1I4hJCCNEqJpOJhFgrCbHtk1I6Z19KIYQQUUsSlxBCiKgiiUsIIURUMfkaG8bd\nxnJzc8PzxEIIIaLC2LFjGxw4FrbEJYQQQrSENBUKIYSIKpK4hBBCRBVJXEIIIaKKJC4hhBBRRRKX\nEEKIqCKJSwghRFSRxCWEECKqSOISQggRVSRxCSGEiCqSuIQQQkQVSVxCCCGiSlB3/VJKTQQe0VpP\nrVM+E/g94AaWaa2fUUrFA68A3YFy4Dqt9fGQRi2EEKLTarLGpZT6NfAsYKtTHgM8CfwE+BEwXynV\nA/gFsEVrPQV4CfhdqIMWQgjReQVT49oFXAq8XKc8B9iptS4GUEqtAc4GzgIe9W/zCXBfQweV25oI\nIYQ4lcZua9Jk4tJaL1dK9WtgVQpQGrBcDqTWKa8pa1BCQkJTT98kp9OJzWZresMIIjG3j2iLOdri\nhdDG7Pb6qKz2Nmsfnw+KHW6KHB6C/SVcXV1NbGxs8wMMgtcHRXY3xU4PQQcUBLfbjdXadD3DB5RX\neSi0e3B5QxeA2+uj0O7B4fKSnmDFbIICu5tqd/3nGNrdxu+m9mj1e8Nutze6LqhrXI0oA5IDlpOB\nkjrlNWUNysnJacXTG/Ly8kJynPYkMbePaIs5GuKtcnuIMZsxm034fD42bvmB7P6DAONLs9Tu4miZ\nk2q3N2AfL4dLnRwudZBfYvx7pNRJtefkbQoqqpDbA7ZeYqyFzFQbcUEkumBZzCYGZsaRGGflSKkT\nr8/HuMx4EmMt9bYdnpVCTk6/Vr+fc3NzG13XmleWBwxWSnUFKjCaCR8H+gIzgPXAdOCrVjyHEKIJ\nbo+XY+VVHCuvIjU+hq4JsRyvcJJf4iS/xIEPyEy14az2cLTMibuBX+IHiuxsOlCC2WyiW1IcJXYX\nJY5quiXHER9j8SceJ0WV1cRYTGQkxVFsr8bp8gJ7g4rTbIIeKTZ6ptrI6ZlCXMyJS+yxFjOZqTbS\n4mMwmRpsHWqQyQRdE2PpkWLDYg5uv71799KvX7+gn6M5TEC35Di6JcdhacbraMq2bdsYOnRoUNta\nzKZmncNo1OzEpZSaCyRprZ9WSt0JfIbRyWOZ1vqQUur/gBf917yqgbkhjViIKON0edh5rAKX5+Rm\nMB9QYq/mSGkVbq+XI0dK+bZk78nb+KCosprDpY7axJGeFEdSnIUj/mRytMxJa1uFEmItjMxOw2yG\nPQWVdEmIpW96IsfLqyisqCYz1cbI3mlkptiwV3s4Xl5Fl4QYfI5Semb2qD1OUpyVzFQb8TEnfonH\nWM1kptjonhyH1RL+ETjxlUfI6dMl3GE0i8VsiohzFymCSlxa673AJP/j1wLKPwQ+rLOtHZgduhCF\naB9Ol4f8Egf2ag8+HxRUVnG8rApPQPtVZZWb/BInDpe73v5VLqNJrMzpqi1ze3zsLqjA5Qk2sxQ2\nWNo9OY6eafF0T46jsLKag8VuMlNsnDEwg6w0Gz1TjXWlDhfFdqOm1DM1nqw0GyaTiSOlDuKsRhNS\nrLX+F2BCjKVFX4xGc9CAZu8nRGuErhFUiDA7VuZk4/4Syp0uqtxe1mw9zrF/FjVZG6l2ezlSZtRm\ngpEQayEprv5HJ8bf3JWZYuNES42JHw/tzojsVOIbuB6QYouhZ6qNOKuZ7Tt2MGTw4HrbJNtiGkw2\nzdErLb5V+wsRSSRxiYhX5nRRUF6F1+djx9EKthwqpSrg4n9BRRW5+4o5WOw4ab+kWDOnZacRa62f\nMALFmE2M6pNGVqpRc0m2GR+LmmsnMQE1kfgYCynx1ja5hpBms5CeFBfy4wrR0UjiEmHn8/nYdKAE\ne7Wntrlrb6GdjfuL2bivGH20/KTeZlazCVvANZRkm5XRfdK4/ox+jO3bhYykOCxmE8X5exg+bFgY\nXpEQoi1J4hLtwuv1UVhZTX6Jg31FdjbuK+ZQiYMeKXFsPljK5oOl9fZJjrMyum8XZpzekz5dEzCZ\noE/XBIZnpQbVdFZ6uGP3rBKis5LEJULO5/Ox5VApKzcfZtOBEg6XOuuN27HFmOndJYH1e4rISIrl\noUtOY2C3JI6WOUmNjyG7SwL9MxKD7uIshOg8JHGJFvF6fRwpO3lQaX6Jk/1Fdr7bX0yx3YXVbGJk\n7zRG9U6j5+k2slLjyUqLp1daPIN7JJ107UgIIYIliUsEzV5tdAX/Znchz321m72FJ0/JkhRnpVda\nPOcP68H4fl05f1gP0hLaZmodIUTnJYlLnKQmOR0udXC4xEl+qYODxQ6+3XWU/aW7aztJjMxO5b9n\nDSe7awJZqfH0TLORYosJb/BCiE5BElcnV+328tWO43y29Qgb9hazu6DypPUmE2QkxdE3xcql4/rT\nNz2Bgd2SOK1XSoefVkYIEZkkcXVSR0qdvLZ+P6+s20dRZTUpNisT+qdzyehe9O6aQM9UG1lp8fRI\nMWZaMGZIqD84Vggh2pskrg6q2u01Zun2eCl3uvn7D0f4dk8xXp+Po+VODhQ5MJngvJweXDmhN2cN\n6tbq2RmEEKI9SOLqQKrdXlZuyee5NXvYml920qBdi9nESP+0Q6f3SuW6yf04Z2h3BnRLCl/AQgjR\nApK4OoDvD5Xyyrp9fPL9EUodLgZ1T+LWcwbTK82GLcZCjMXMhP5dyZDphIQQHYAkrihT7fay5VAp\n3+0vZl+hnbzDZWzYV0xCrIWfDOvBrNG9+NHgbphl4K4QooOSxBVFdhwt5+aXc2t7/hkzTMTz2xlD\nmTOhj3RHF0J0CpK4Ipzb4+WrHQWs31vES2v3Eh9rZcmcUUwemE73ZFu4wxNCiHYniStC+Xw+/rX9\nOA+tzGPnsQosZhNnDEznsctHkpkqCUsI0XlJ4oogPp+Pg8UOvtldyMvf7GPLoVL6pSfw1FVjmKq6\nkRAr/11CCCHfhBHiSKmTha9tZMO+YgAGdEvk4UtO5/Kx2TK+SgghAkjiigDf7Crk1te/w17t5ncX\n5jB5YDo5mSnSM1AIIRogiSuM7NVuFq/aznNr9tA/I5HX5k1kSI/kcIclhBARrcnEpZQyA08BI4Eq\n4Cat9U7/ulHAnwI2nwRcDKwHtgPf+8tXaK2XhDDuqFVR5WZnYRWb1u9nyec7OFLm5JpJffnNjKFy\nDUuI9hY4vUxL9z+UC9s+BHthaGJqQM+SUtie2vwdLbEwYCoMOh9i4hvexlEMeR/AwQ1AK88HQOZI\nmDi/9cc5hWC+KS8GbFrryUqpScBiYBaA1noTMBVAKTUbOKS1/lQpdR7wutb61rYJO/ocKLLz9Je7\neTv3AE6XFzjEiOxUls4dzfh+XcMdnog2Xg/s/wYKdzW+TXwXGHgOmMyw50tI6gZZY4wp/6OZuwp2\nfwHlh1u0e9rhw2D/Nxz+D/zwAdgLWh+TJRYSu7X+OI1IdLmgsAXjNKvKYcOy4LZNyABrCGbXqaqI\niMR1FvApgNZ6nVJqXN0NlFKJwAPA2f6iscBYpdQXwDHgNq11y95lUa7U4eKpf+3k+TV7AbhkdC8G\nJ1Uz6fTBDOsp17GikrMMyg61/jheD+xbC/pjqCqnn8MBaxr5VVxX6QGoONr0dtZ4I3G5/Ler6dIP\nhl8CagbENaNZ2ueDg9/Cto/AXlRb3KyYQ8IHBTuhqrTFR+hZ8yAmAYZcAN1U60JK6wtqOsSnte44\np7AzL4+cnJzm7+hxw96vjP87n7fhbSyxxg+cniOj5keNyddEVVkp9SywXGv9iX95PzBAa+0O2OY2\nIF1rfb9/eRZQqbX+XCl1FXCJ1vrywOPm5ub6EhISWv0CnE4nNltkjWvy+XzsLXHx953lfLq9DIfb\nx/kDk7h2dFcyEq0RGXNTJGbA5yNl/9/J3Pg4luqykB22KqU/roQeeL0ezGZLUPt4YpMp73U2jvTT\njcTUgNiKgyQfWI3J56E8eypW+3FSDnxO4tFvMfk8LYq1Oimb6qTs2uXmxBwq7vgMynqfQ1XqQKD5\nX7RVVU7i4mx4YlPwWaPjPd0ZP392u52xY8c2+B8cTI2rDAj8aWYOTFp+VwGBiWk1UHNf9xXAgw0d\nuEW/IOrIa+kvkTZyoMjOzS/n8sPhMqxmEzNHZjFvygCGZaXUbhNpMQcjqmMu2AlHNjdvZ48Ldq2G\nnZ8bTVMA+KC6ArLHw8QFEIov7G45xHUfShzNP8fBXfG4CoAutcu/gspC2L8WvHU/xk3oOoDYzBHE\nBvwqD9f7ojV1m7y8PAZH63s5irQ25tzc3EbXBZO4vgZmAm/5r3FtCVyplEoF4rTWBwKKnwWWA28B\n5wKNR9CB5Jc4uPKZdZQ73fz3rOFMP72nzMjeVnw+o6ms5su39iL5R1BpXLPoXVkJ/6yEY1tb9hy2\nVBgyHRLST5SlD4Sx14cmaYVLYjrkzAx3FEK0WDCJawVwvlJqLUa9/Aal1J3ATq31B8AQYG+dfe4B\nlimlfglUAjeFLuTIlHe4jHkvbaDU7uLVeRMZkd127d2dUlk+bFsJlcehuhK2fwqFO+tvl9gNuvQH\nwOx2QEoGTHsY+p8N5mZc3DaZjONYY0P0AoQQodJk4tJae4EFdYq3Baz/FqPnYeA+e4AfhyLAaPDB\nf/K5+53NpMRbeeUmSVot5nEZXXKrK0+UFe+B7981etDVdNU1WaDvGTDu5xAXcCPMtL7Q90ywGG/r\nfVHYvCKEaJoMHGoFt8fLo59pnv5yN+P6duGpq8d0zBnbnaWk7lkJPywGT3X99RnK6KmW2iu449kL\nIe9DOLSR2mRU0727obEw3XLgx7+F4ZdCxqAWvwwhRMcgiasVFq3M44W1e7lmUl/uu2hYx5hT0OOC\nPV/AsW1G99n938DOz8nyVENSZv0uv16PkYS+fLT5z9Wl/8njRgZMhWEXQ3LPE2XxXSRZCSFOIomr\nhdbuLOCFtXu5/ox+/OGnw8MdTssV7TGuGzlLjY4NP3wAjhPjdEjuCeNvYk/SGPqfeXnD4zzKj8KO\nz4zBjsGwxhnjRroOCM1rEEJ0KpK4WmB/oZ1fvbOZ/hmJ3H3B0HCH0zIuJ6z+b/jmL9Q218UkGgMp\nT7vUuIZkskBsEpjNOPPyGh+cmNwDxlzbbqELITo3SVzN4HR5uGf5Zj74Tz5Wi5nX500kPjaKukXX\ndBn/fjlsXWFMmTP2Bsi5yBg932scxLZ+ULgQQrQlSVxB8vl83Lvie97blM/8swdw45n9I/tOxM4y\nWHknFO89UVZ+xJgqyBJrTLo5YR4M7DSdP4UQHYQkriC9vG4fyzce5PZzB3PH+UPCHc6pVVXAq7Ph\n0AboN+VEE19yT/jxvTB0hjG4VgghopAkriDsPl7Boo/yOGdod24/d3C4wznB6zWmJaoqM2aR2Poe\nHN0KXpfRO/DyZTD84qaPI4QQUUQSVxNqmgjjYsz88bLTwzebu9frv42DvyOFoxg+uRv2fX1im+7D\nYNRcMFth8HlGzz0hhOhgJHE14Z3cg3yzu5CHLjktfIOLj2+H9xYYHSsCxaXAT5dC9gSjU0Van/DE\nJ4QQ7UgS1ynoI+Xc/8FWJvTrypXjw5QUdq2G16807h30k0UB16ZMRo0q2NkqhBCig5DE1YijZU7m\nv7yBxDgrS+eODk8TYfkRWD7PGKh7zXvGeCkhhOjkJHHVsaegkt+//z1f7yzAYjbxxvxJ9EgJQxNh\nyX5475fGhLOXPy9JSwgh/CRxBdhysJTrn1+P1+dj4Y8HcfHoXgzoltT0jq21YRmsf+bErbU9Lija\nZTye9RR0j9LZOYQQog1I4vLbuL+Ya59bT2p8DC//fEL7JCwwEtbHd0HWGEjrfaJ89FXGjOsyn58Q\nQpxEEhew+WAJ1z23noykWN6YP7ntZsQoPQivXEbO8W0nl6sZMPtFuWmhEEIEodMnrqLKam584VvS\nEmN4bd6k0Cctd5Ux7VJ1Bbzzc7AXUTDsBjK6+a9ZxXeBcTdI0hJCiCB1+sR133vfU+pw8cpNE8lK\niw/NQT0u2P0FbH0X8j6CqlKjPDYZrn2P4+WJZMideYUQokU6deL6aHM+K7cc5lfTFEMzU0Jz0K3v\nwSe/NqZgikuBoRfCgB+DJQayRhnXrPLyQvNcQgjRCXXaxFVR5eaBD39gRHYqN5/dwg4Q+9bCmifB\n6zaWqyrg4HroOQouehIGngsxETyDvBBCRKFOm7iWrt7B8fIqnrl2HFaLufkHKMuHN6825gUMnGrp\nnN/Bmf9l1LCEEEKEXJOJSyllBp4CRgJVwE1a650B65cAZwE1922fBcQArwHxQD5wg9baHtrQW273\n8QqWrdnD7LHZjOqd1vwDeNzw7nzjLsI3fwEZETRjvBBCdHDBVDUuBmxa68nAPcDiOuvHAtO01lP9\nf6XA74HXtNZTgO+Am0MZdGt4vT7ueXcLthgLv7pANf8Ahbvg+emw9yuY8ZgkLSGEaGfBJK6zgE8B\ntNbrgHE1K/y1scHA00qpr5VSN9bdB/gEOC9kEbfS82v3sn5PEffPHB78bO/OMvjPG/DqFfCXiVCg\n4dJnjUHCQggh2lUw17hSgNKAZY9Syqq1dgOJwFLgCcAC/FMptaHOPuVAg7fbzQtB7zqn0xn0cfLL\nXDzyyUEmZicwLL6s0f1iS/dgdlUSaz9M8v7PSTr8DWZvNa6EHpQNnk3RkDm4Y7q3uHdgc2KOFBJz\n24u2eEFibi8S88mCSVxlQHLAstmftADswJKa61dKqdUY18Jq9nH4/y1p6MA5IRjLlJeXF/Rxnn5z\nE2aziT9fO7nxiXP//TR8+qsTy0mZMP5GGH4pMdnjSTebSW/HmCOFxNz2oi1ekJjbS2eMOTc3t9F1\nwSSur4GZwFtKqUnAloB1Q4A3lVKjMZodzwJe9O8zA3gBmA581ZLAQ2nH0XLe23SI+WcPaDxpHf4P\nrLoXBp0PExcY977qNQbMlvYNVgghRKOCSVwrgPOVUmsBE3CDUupOYKfW+gOl1MvAOsAFvKS13qqU\nWgS8qJSaBxQAc9so/qA9+fl2EmOtLDh7YP2VlQWwdQV8/WdIyIBL/gaJra1XCSGEaAtNJi6ttRdY\nUKd4W8D6x4DH6uxzFLggFAGGwlvfHuDjLUe4/dzBdEkMmBPQ54NNr8Knv4GqMuiWAz9dKklLCCEi\nWIcfgPzF9uP8ZsUWpgzOYOE5g06sKD8KH94O2z+BvmfC9Ech87TwBSqEECIoHTpxHSiys/C1jQzu\nnsRTV40hpmaGjP3r4PUrjbsLT3sYJv4CzC2YPUMIIUS767CJy+Xxcvsb34EPnr5mHMk2/xRMlQXw\n1nUQnwY3fgrdWjAIWQghRNh02MT1l3/uZOP+EpZeOZo+6QlGodcL7/0CHMVw1duStIQQIgp1yMR1\nuNTB//1rFzNHZjFzZJZR6PPBp/fAjlUw/THoOSK8QQohhGiRDnlh54lV2/H54NfT/DUqrwc+uxfW\n/w0m3QIT5oU3QCGEEC3W4Wpc246U8c7Gg9x0Vn96d00wJsV975dwYB1MuBmmPQQmU7jDFEII0UId\nLnEt+XwHSbFWbpk6EL59FlbdB+YYuORpGHGFJC0hhIhyHSdx6U/xvHszV9j7cn2f0aS98Hs4vs24\nC/FPl0Jqr3BHKIQQIgSiO3G5HMSW7YGt2+Dd+Ry3ZDLEnE/Woc3QZzJc/FcYOUdqWUII0YFEd+J6\ndz4D8z4AwJlxGhccuo2fTTmd35zbG+KSm9hZCCFENIruxDXtIQ6mjadXdh9uXZuCJ87F/LMHQlxc\nuCMTQgjRRqK7O3xaH8r7nM/npsn8fZeDO88fQnqSJC0hhOjIorvGBXi8PhZ9/ANDeiRx9aS+4Q5H\nCCFEG4vuGhew/qCdfYV27jxfnZhEVwghRIcV9d/0n2wvo0dKHOfldA93KEIIIdpBVCeuQyUONhxy\n8LNxvbFKbUsIITqFqP62f3P9fgCuGN87zJEIIYRoL1GduLYdKeeMvolkd0kIdyhCCCHaSVT3Klw6\ndzR6mw53GEIIIdpRVNe44qwWYiwynZMQQnQmTda4lFJm4ClgJFAF3KS13hmw/g5gjn/xY631A0op\nE3AQ2OEv/0Zr/ZuQRi6EEKJTCqap8GLAprWerJSaBCwGZgEopQYAVwETAS+wRim1ArADG7XWM9sm\nbCGEEIGqqqr44IMPmD17dquPtXPnTu677z58Ph/9+vVj0aJFWK2Rc2XJ5PP5TrmBUuoJYL3W+g3/\n8iGtdS//4xggVWtd4F9eD1wNjAbuBkoBB3CH1vqki1G5ubm+hITWd6pwOp3YbLZWH6c9ScztI9pi\njrZ4QWJuL8HEfPToURYvXsyjjz7a6ud7+OGHmTVrFsOHD2fJkiVMnDiRSZMmNesYrT3PdrudsWPH\nNngtKJgUmoKRgGp4lFJWrbVba+0CCvxNg48B32mttyulMoH/0Vq/rZQ6C3gFGF/3wDk5Oc1+MXXl\n5eWF5DjtSWJuH9EWc7TFCxJzQ5bnHuStDQdCesyzelm49aLRp9zm1Vdf5dChQ/zjH/9gy5YtVFRU\n4PF4uP3225k8eTIzZsxg3Lhx7Nixg9TUVJ544gkaqzy88MILWCwWqqurcblcDB06tNnnrLXnOTc3\nt9F1wXTOKAMC7xFi1lq7axaUUjbgVf82v/QXbwDeB9BarwGy/MlNCCFEG1iwYAGDBg2isrKSM844\ng1dffZUlS5Zw77334vP5cDqdzJw5k9dff50BAwbw5ptvNnosi8XCoUOHuOiiiyguLmbo0KHt+Eqa\nFkyN62tgJvCW/xrXlpoV/mT0PrBaa/1IwD73A4XAo0qpkcABrfWp2ySFEKIDuGxsNpeNzQ7pMfPy\n8oLedteuXcycaXQv6NGjB0lJSRQWFmK1Whk/3mj4GjNmDF9++eUpj9OrVy9WrVrF22+/zR//+Ece\neeSRU27fnoJJXCuA85VSawETcINS6k5gJ2ABfgTEKaWm+7f/DfBH4BWl1IWAG7g+1IELIYQ4wWw2\n4/V6GThwIBs2bGDYsGEcPXqUsrIy0tLScLvdbNu2jaFDh5Kbm8ugQYMaPdaCBQu455576NevH4mJ\niZjNkTVyqsnEpbX2AgvqFG8LeNzY1bcLWxqUEEKI5klPT8flclFeXs6+ffv47LPPcDqdPPjgg7U9\nAp955hny8/PJysrijjvuaPRY8+fP55577iEmJob4+HgWLVrUXi8jKJHTv1EIIUSLxcXF8f77759y\nm4cffpi4IO4QP2bMGN54441QhRZykriEEKITys/P5+67765XPn78eG677bYwRBQ8SVxCCNEJrF69\n+qTlrKwsXn755TBF0zqRdcVNCCGEaIIkLiGEEFFFEpcQQoioIolLCCFEVJHEJYQQncQ111zDrl27\nGl3/7bffsm3btkbXN9eTTz7J7NmzueKKK/j3v/8dsuNK4hJCCAHA8uXLOXbsWEiOtXv3bjZt2sRb\nb73FE088wUMPPRSS44J0hxdCiNDa9Dp890pID5na4xw4xUzrCxcu5Nprr2XChAls2bKFRx99lK5d\nu1JeXs6xY8eYO3cuc+fOPeVzfP/993z11Vds3bqVQYMGsWHDBl588UViY2Pp168fDz74IB9++CGf\nf/45lZWVFBcXc8sttzBt2rQGjzdgwACee+45TCYT+fn5pKSktOocBJLEJYQQUW727NmsWLGCCRMm\n8O677zJx4kSGDBnCT37yE44ePco111zTZOI67bTTmDJlCjNmzCA+Pp6lS5eyYsUKkpKSePjhh3nz\nzTdJSEjA4XDw/PPPU1RUxOzZszn33HMbvcmk1WrlySef5KWXXuK+++4L2euVxCWEEKE06krjL4RK\n8/LIOsX6KVOm8Nhjj1FSUsKGDRt49tlnWbx4MatWrSIpKQm3232Kves7cOAAgwYNIikpCTBm01iz\nZg0jR45k/PjxmM1mMjIySElJoaioiO7duzd6rDvuuIN58+bxs5/9jHHjxtGnT59mxdIQucYlhBBR\nzmw2c8EFF/CHP/yB8847j2XLljFq1Cgef/xxLrjgApq6030Nk8mEz+cjOzubXbt2YbfbAVi/fj39\n+/cHYOvWrQAUFBRQUVFBenp6g8favHkzDzzwAGDMo2i1WjGZQnNbRqlxCSFEB3DZZZdx3nnn8dln\nn3Hw4EEWLVrExx9/THJycu3djJsycuRIHn/8cf70pz9x6623cu2112I2m+nTpw933XUXK1eupKCg\ngOuuu47y8nLuv/9+LBZLg8caPnw4P/zwA3PmzMHr9XLVVVfRu3fvkLxWSVxCCNEB9OzZs7Y2lJ2d\nzUcffVRvm6bmJpwzZw5z5swBYODAgbU3pAw0fvx47rrrribjsVgstTWuUJPEJYQQncjmzZt57LHH\n6pVPnz69yQ4cDfnf//3fBsdo/fznPyfnFD0hW0MSlxBCdCIjRoxo8azwl156ab2yhQsXsnDhwnrl\neXl5LXqOYEjn1qyyvQAAB+1JREFUDCGEEFFFEpcQQoioIolLCCFEVGnyGpdSygw8BYwEqoCbtNY7\nA9bPA24G3MAirfVHSqkM4DUgHsgHbtBa29sgfiGEEJ1MMDWuiwGb1noycA+wuGaFUioTuA04E5gG\n/I9SKg74PfCa1noK8B1GYhNCCCFaLZjEdRbwKYDWeh0wLmDdBOBrrXWV1roU2AmMCNwH+AQ4L2QR\nCyGE6NSC6Q6fApQGLHuUUlattbuBdeVAap3ymrJ6QtFd0ul0tmm3y7YgMbePaIs52uIFibm9SMwn\nCyZxlQHJActmf9JqaF0yUBJQ7ggoq6dmHqzWCtVx2pPE3D6iLeZoixck5vYiMZ8QTOL6GpgJvKWU\nmgRsCVi3HnhIKWUD4oAc4Hv/PjOAF4DpwFd1Dzp27NjQzLYohBCiUzE1NWtwQK/CEYAJuAEjKe3U\nWn/g71U4H+N62cNa6+VKqR7Aixi1rQJgrta6su1ehhBCiM6iycQlhBBCRJKonauwqfFlkUIpFQMs\nA/phNKcuAg4AHwE7/Jv9n9b6zbAE2Ail1EaMa5UAe4C/AUswxuut0lq3zbTPLaSUuh643r9oA0YB\nVwKPY5xvgPu11l+0e3B1KKUmAo9oracqpQZhNKn7MJrZb9Fae5VS9wMXYpzv/9Jarw9bwNSLeRSw\nFPBgfPau1VofVUotwehRXO7fbZa/t3FY1Il5NA185iL8PL8BZPpX9QPWaa3nKKXeBzIAF+DQWk8P\nU6wNfbf9QDu8n6M2cREwvsx/7W0xMCvMMTXkaqBQa32NUqorsAl4EHhCa7341LuGh/+apUlrPTWg\nbBNwGbAbWKmUGq21/i5MIdajtX4B4wODUuovGB+oscCvtdbLwxfZyZRSvwauAWqazp8Afqe1/pdS\n6q/ALKXUPuBHwESgN7AcGB+OeKHBmJcAt2qtNymlbgbuBu7EON/TtNYF4Yn0hAZiHkudz5xSagwR\nfJ611nP85V2AfwJ3+DcdDAzXWoe7uayh77ZNtMP7OZqnfDrV+LJI8jZwn/+xCeMXx1jgQqXUl0qp\n55RSyY3uHR4jgQSl1Cql1Gql1NlAnNZ6l//D8hkROjZPKTUO40P9NMZ5vlEp9ZVSarFSKhJ+qO0C\nAqfYHgvU1AJrxjyehVGr9Wmt9wNWpVS39g3zJHVjnqO13uR/bAWc/haQwcDTSqmvlVI3tneQdTR0\nnut+5iL9PNd4AFiqtT7s7z+QBnyolFqjlLqoXSM8WWPfbW3+fo7mxNXg+LJwBdMYrXWF1rrc/0F5\nB/gdRm/MX2mtz8aowdwfzhgbYMdoYpsGLACe95fVaHRsXgT4LcYHHeDvwK3A2UASxmsJK3/tzxVQ\nZAr45dzQOMjA8rCoG7PW+jCAUuoMYCHwJJCI0Xx4NXAB8Eul1Ij2j7Y2xrrnuaHPXESfZwClVHfg\nXPytCUAsRuvSxRhJ7kn/Nu2uke+2dnk/R3PiOtX4soiilOqNUdV/WWv9GrBCa53rX70CGB224Bq2\nHXjF/wtpO8abrmvA+kbH5oWTUioNUFrrf/qLlmmtd/s/SO8TeecZwBvwuO44yLrlEUMp9TPgr8CF\nWuvjGD9slmit7VrrcmA1Rs09UjT0mYv48wxcjjF9nse/fAT4q9barbU+hjGlngpXcA18t7XL+zma\nE1fNWDEaGF8WMfxV+1XA3VrrZf7iz5RSE/yPzwVyG9w5fG7EPyelUioLSAAqlVIDlVImjJpYvbF5\nEeBs4B8A/jg3K6Wy/esi8TwDfKeUmup/XDPm8WtgmlLKrJTqg/GjLOzXjWoopa7GqGlN1Vrv9hcP\nAb5WSln8F+3PAjaGK8YGNPSZi+jz7HceRpNb4PLbAEqpJOA0ICxTajTy3dYu7+eIa1prhhXA+Uqp\ntZwYXxaJfgt0Ae5TStW0B9+JUcV3YfyCmh+u4BrxHPCCUmoNRu+gGzF+Sb0KWDDaq+vfqzv8FEYz\nEFprn1LqJuBdpZQDo7fTM+EMrhH/D3hGKRWL8QX0jtbao5T6CvgG48flLeEMMJBSygL8GdiPcW4B\nvtBa36+UehlYh9Hc9ZLWemv4Iq3nF8DSwM+c1rosUs9zgNr3NIDW+hOl1DSl1DqMz+Rvw5hsG/pu\nux34c1u/n2UclxBCiKgSzU2FQgghOiFJXEIIIaKKJC4hhBBRRRKXEEKIqCKJSwghRFSRxCVEFFBK\n/UspNTTccQgRCSRxCSGEiCoyjkuIEPPPHPFXjElnzRhzuD2FMYvAcKAI45Yr1RjzQA7AGNj9hP9W\nGxOBP/n3PQRchTF7wmGgB8a8gFcGzFohRKciNS4hQu8moMA/oess4C8Y02a9qrU+C9gG3Oz/O661\nPgNjKp9FSqkMjHuf3ai1ngisBHL8x12ptT4HI4ld3p4vSIhIEs1TPgkRqU4HpvhrTmB8znxa6y/9\ny2sx5nFzA58D+GfZ/gEYCGRqrfP85c8B+KdWqplr8QgnbjAoRKcjNS4hQm8b8Lr/RpzTMSZFtSil\namZLPxPYijGX2xQA/60hTse423S+Umqwv/xupdQl/v2kXV8IJHEJ0Rb+BgxVSn2BUbvahzEh6t3+\niYt7+bd5Gkj3l/0LeMB/q4qbgWX+/UcDH7f/SxAicknnDCHagVJqLzBUa+0McyhCRD2pcQkhhIgq\nUuMSQggRVaTGJYQQIqpI4hJCCBFVJHEJIYSIKpK4hBBCRBVJXEIIIaKKJC4hhBBR5f8DHGcnoGHq\ny/cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x130f36588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I see some overfitting but overall, not bad!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check:\n",
    "\n",
    "What happens if you re-run the fit code? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Evaluating and Making Predictions\n",
    "- Like `sklearn`, we have nice predict and evaluation functions!\n",
    "- Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-07T19:52:37.099187Z",
     "start_time": "2017-11-07T19:52:36.969906Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2500/2500 [==============================] - 0s 48us/step\n",
      "Test loss: 0.98, Test Acc: 0.6416, Test top 2: 0.8572\n"
     ]
    }
   ],
   "source": [
    "loss, acc, top_2 = model.evaluate(X_test, y_test, batch_size=32)\n",
    "print(f'Test loss: {loss:.2f}, Test Acc: {acc:.4f}, Test top 2: {top_2:.4f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-07T19:52:45.301854Z",
     "start_time": "2017-11-07T19:52:45.113931Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2500/2500 [==============================] - 0s 27us/step\n",
      "[3 9 7 6 0]\n",
      "2500/2500 [==============================] - 0s 40us/step\n",
      "[[  9.77193704e-04   3.90130989e-02   1.06703872e-02   5.67591250e-01\n",
      "    1.26476225e-04   3.68625224e-01   7.59360194e-03   5.07171033e-04\n",
      "    4.89472738e-03   8.57792543e-07]\n",
      " [  2.12158412e-07   1.38904620e-03   4.39259484e-02   1.33146368e-05\n",
      "    3.06499751e-05   1.07749924e-02   1.49287109e-04   3.60558271e-01\n",
      "    2.20529721e-04   5.82937717e-01]\n",
      " [  8.71434249e-03   3.11673619e-03   1.30473524e-02   2.33596205e-04\n",
      "    2.91365059e-03   2.83507794e-01   1.75659448e-01   4.87488419e-01\n",
      "    6.19693019e-05   2.52567008e-02]\n",
      " [  3.56755555e-02   5.51575702e-03   3.96524556e-02   5.69694713e-02\n",
      "    6.05951138e-02   2.41894983e-02   6.86174512e-01   7.38689303e-02\n",
      "    1.01686064e-02   7.19021028e-03]\n",
      " [  4.53367978e-01   7.74765015e-03   1.54660534e-04   3.20681006e-01\n",
      "    1.24942549e-01   8.94623771e-02   3.88024346e-05   1.86787292e-04\n",
      "    3.33523517e-03   8.27765616e-05]]\n"
     ]
    }
   ],
   "source": [
    "# Make Predictions\n",
    "classes = model.predict_classes(X_test, batch_size=32)\n",
    "print(classes[:5])\n",
    "proba = model.predict_proba(X_test, batch_size=32)\n",
    "print(proba[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### The Functional API\n",
    "- Let's you specify more complicated models\n",
    "- Not our focus today, `Sequential` is plenty!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `Keras` Layers\n",
    "- Layers are where the real action's at\n",
    "- This is where you \"architect\" your network\n",
    "- Different architectures solve different problems well!\n",
    "- Layers define:\n",
    "  - Nodes (Network Units, can be complex themselves)\n",
    "  - Connections\n",
    "  - Properties on Units and Connections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Dense Layers\n",
    "- Simplest kind of layer, `Dense`\n",
    "- Familiar from standard ANN\n",
    "- Fully connected between inputs and outputs\n",
    "- This is the usual gal!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-07T19:53:27.520780Z",
     "start_time": "2017-11-07T19:53:27.399063Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# 2 ways to specify layers:\n",
    "# Model Constructor or add() method\n",
    "# These are the same model: \n",
    "model1 = keras.models.Sequential([\n",
    "    keras.layers.Dense(32, input_dim=784),\n",
    "    keras.layers.Activation('relu'),\n",
    "    keras.layers.Dense(10),\n",
    "    keras.layers.Activation('softmax'),\n",
    "])\n",
    "model2 = keras.models.Sequential()\n",
    "model2.add(keras.layers.Dense(32, input_dim=784))\n",
    "model2.add(keras.layers.Activation('relu'))\n",
    "model2.add(keras.layers.Dense(10))\n",
    "model2.add(keras.layers.Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-07T19:53:30.524254Z",
     "start_time": "2017-11-07T19:53:30.416395Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"377pt\" viewBox=\"0.00 0.00 314.56 377.00\" width=\"315pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 373)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-373 310.5557,-373 310.5557,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 4850051112 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>4850051112</title>\n",
       "<polygon fill=\"none\" points=\"0,-324.5 0,-368.5 306.5557,-368.5 306.5557,-324.5 0,-324.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"83.6191\" y=\"-342.3\">dense_5_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"167.2383,-324.5 167.2383,-368.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.0728\" y=\"-353.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"167.2383,-346.5 222.9072,-346.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.0728\" y=\"-331.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"222.9072,-324.5 222.9072,-368.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"264.7314\" y=\"-353.3\">(None, 784)</text>\n",
       "<polyline fill=\"none\" points=\"222.9072,-346.5 306.5557,-346.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"264.7314\" y=\"-331.3\">(None, 784)</text>\n",
       "</g>\n",
       "<!-- 4850050104 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>4850050104</title>\n",
       "<polygon fill=\"none\" points=\"31.4932,-243.5 31.4932,-287.5 275.0625,-287.5 275.0625,-243.5 31.4932,-243.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"83.6191\" y=\"-261.3\">dense_5: Dense</text>\n",
       "<polyline fill=\"none\" points=\"135.7451,-243.5 135.7451,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"163.5796\" y=\"-272.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"135.7451,-265.5 191.4141,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"163.5796\" y=\"-250.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"191.4141,-243.5 191.4141,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233.2383\" y=\"-272.3\">(None, 784)</text>\n",
       "<polyline fill=\"none\" points=\"191.4141,-265.5 275.0625,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233.2383\" y=\"-250.3\">(None, 32)</text>\n",
       "</g>\n",
       "<!-- 4850051112&#45;&gt;4850050104 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>4850051112-&gt;4850050104</title>\n",
       "<path d=\"M153.2778,-324.3664C153.2778,-316.1516 153.2778,-306.6579 153.2778,-297.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"156.7779,-297.6068 153.2778,-287.6068 149.7779,-297.6069 156.7779,-297.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4850050496 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>4850050496</title>\n",
       "<polygon fill=\"none\" points=\"11.6587,-162.5 11.6587,-206.5 294.897,-206.5 294.897,-162.5 11.6587,-162.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"87.1191\" y=\"-180.3\">activation_5: Activation</text>\n",
       "<polyline fill=\"none\" points=\"162.5796,-162.5 162.5796,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.4141\" y=\"-191.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"162.5796,-184.5 218.2485,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.4141\" y=\"-169.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"218.2485,-162.5 218.2485,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"256.5728\" y=\"-191.3\">(None, 32)</text>\n",
       "<polyline fill=\"none\" points=\"218.2485,-184.5 294.897,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"256.5728\" y=\"-169.3\">(None, 32)</text>\n",
       "</g>\n",
       "<!-- 4850050104&#45;&gt;4850050496 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>4850050104-&gt;4850050496</title>\n",
       "<path d=\"M153.2778,-243.3664C153.2778,-235.1516 153.2778,-225.6579 153.2778,-216.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"156.7779,-216.6068 153.2778,-206.6068 149.7779,-216.6069 156.7779,-216.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4850050552 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>4850050552</title>\n",
       "<polygon fill=\"none\" points=\"34.9932,-81.5 34.9932,-125.5 271.5625,-125.5 271.5625,-81.5 34.9932,-81.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"87.1191\" y=\"-99.3\">dense_6: Dense</text>\n",
       "<polyline fill=\"none\" points=\"139.2451,-81.5 139.2451,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"167.0796\" y=\"-110.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"139.2451,-103.5 194.9141,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"167.0796\" y=\"-88.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"194.9141,-81.5 194.9141,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233.2383\" y=\"-110.3\">(None, 32)</text>\n",
       "<polyline fill=\"none\" points=\"194.9141,-103.5 271.5625,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233.2383\" y=\"-88.3\">(None, 10)</text>\n",
       "</g>\n",
       "<!-- 4850050496&#45;&gt;4850050552 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>4850050496-&gt;4850050552</title>\n",
       "<path d=\"M153.2778,-162.3664C153.2778,-154.1516 153.2778,-144.6579 153.2778,-135.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"156.7779,-135.6068 153.2778,-125.6068 149.7779,-135.6069 156.7779,-135.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4850050944 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>4850050944</title>\n",
       "<polygon fill=\"none\" points=\"11.6587,-.5 11.6587,-44.5 294.897,-44.5 294.897,-.5 11.6587,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"87.1191\" y=\"-18.3\">activation_6: Activation</text>\n",
       "<polyline fill=\"none\" points=\"162.5796,-.5 162.5796,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.4141\" y=\"-29.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"162.5796,-22.5 218.2485,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.4141\" y=\"-7.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"218.2485,-.5 218.2485,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"256.5728\" y=\"-29.3\">(None, 10)</text>\n",
       "<polyline fill=\"none\" points=\"218.2485,-22.5 294.897,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"256.5728\" y=\"-7.3\">(None, 10)</text>\n",
       "</g>\n",
       "<!-- 4850050552&#45;&gt;4850050944 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>4850050552-&gt;4850050944</title>\n",
       "<path d=\"M153.2778,-81.3664C153.2778,-73.1516 153.2778,-63.6579 153.2778,-54.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"156.7779,-54.6068 153.2778,-44.6068 149.7779,-54.6069 156.7779,-54.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Same model \n",
    "SVG(model_to_dot(model1, show_shapes=True).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-07T19:53:32.207948Z",
     "start_time": "2017-11-07T19:53:32.126869Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"377pt\" viewBox=\"0.00 0.00 314.56 377.00\" width=\"315pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 373)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-373 310.5557,-373 310.5557,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 4849714400 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>4849714400</title>\n",
       "<polygon fill=\"none\" points=\"0,-324.5 0,-368.5 306.5557,-368.5 306.5557,-324.5 0,-324.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"83.6191\" y=\"-342.3\">dense_7_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"167.2383,-324.5 167.2383,-368.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.0728\" y=\"-353.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"167.2383,-346.5 222.9072,-346.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.0728\" y=\"-331.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"222.9072,-324.5 222.9072,-368.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"264.7314\" y=\"-353.3\">(None, 784)</text>\n",
       "<polyline fill=\"none\" points=\"222.9072,-346.5 306.5557,-346.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"264.7314\" y=\"-331.3\">(None, 784)</text>\n",
       "</g>\n",
       "<!-- 4848561904 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>4848561904</title>\n",
       "<polygon fill=\"none\" points=\"31.4932,-243.5 31.4932,-287.5 275.0625,-287.5 275.0625,-243.5 31.4932,-243.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"83.6191\" y=\"-261.3\">dense_7: Dense</text>\n",
       "<polyline fill=\"none\" points=\"135.7451,-243.5 135.7451,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"163.5796\" y=\"-272.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"135.7451,-265.5 191.4141,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"163.5796\" y=\"-250.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"191.4141,-243.5 191.4141,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233.2383\" y=\"-272.3\">(None, 784)</text>\n",
       "<polyline fill=\"none\" points=\"191.4141,-265.5 275.0625,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233.2383\" y=\"-250.3\">(None, 32)</text>\n",
       "</g>\n",
       "<!-- 4849714400&#45;&gt;4848561904 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>4849714400-&gt;4848561904</title>\n",
       "<path d=\"M153.2778,-324.3664C153.2778,-316.1516 153.2778,-306.6579 153.2778,-297.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"156.7779,-297.6068 153.2778,-287.6068 149.7779,-297.6069 156.7779,-297.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4848578800 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>4848578800</title>\n",
       "<polygon fill=\"none\" points=\"11.6587,-162.5 11.6587,-206.5 294.897,-206.5 294.897,-162.5 11.6587,-162.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"87.1191\" y=\"-180.3\">activation_7: Activation</text>\n",
       "<polyline fill=\"none\" points=\"162.5796,-162.5 162.5796,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.4141\" y=\"-191.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"162.5796,-184.5 218.2485,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.4141\" y=\"-169.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"218.2485,-162.5 218.2485,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"256.5728\" y=\"-191.3\">(None, 32)</text>\n",
       "<polyline fill=\"none\" points=\"218.2485,-184.5 294.897,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"256.5728\" y=\"-169.3\">(None, 32)</text>\n",
       "</g>\n",
       "<!-- 4848561904&#45;&gt;4848578800 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>4848561904-&gt;4848578800</title>\n",
       "<path d=\"M153.2778,-243.3664C153.2778,-235.1516 153.2778,-225.6579 153.2778,-216.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"156.7779,-216.6068 153.2778,-206.6068 149.7779,-216.6069 156.7779,-216.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4850051056 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>4850051056</title>\n",
       "<polygon fill=\"none\" points=\"34.9932,-81.5 34.9932,-125.5 271.5625,-125.5 271.5625,-81.5 34.9932,-81.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"87.1191\" y=\"-99.3\">dense_8: Dense</text>\n",
       "<polyline fill=\"none\" points=\"139.2451,-81.5 139.2451,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"167.0796\" y=\"-110.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"139.2451,-103.5 194.9141,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"167.0796\" y=\"-88.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"194.9141,-81.5 194.9141,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233.2383\" y=\"-110.3\">(None, 32)</text>\n",
       "<polyline fill=\"none\" points=\"194.9141,-103.5 271.5625,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233.2383\" y=\"-88.3\">(None, 10)</text>\n",
       "</g>\n",
       "<!-- 4848578800&#45;&gt;4850051056 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>4848578800-&gt;4850051056</title>\n",
       "<path d=\"M153.2778,-162.3664C153.2778,-154.1516 153.2778,-144.6579 153.2778,-135.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"156.7779,-135.6068 153.2778,-125.6068 149.7779,-135.6069 156.7779,-135.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4849434408 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>4849434408</title>\n",
       "<polygon fill=\"none\" points=\"11.6587,-.5 11.6587,-44.5 294.897,-44.5 294.897,-.5 11.6587,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"87.1191\" y=\"-18.3\">activation_8: Activation</text>\n",
       "<polyline fill=\"none\" points=\"162.5796,-.5 162.5796,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.4141\" y=\"-29.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"162.5796,-22.5 218.2485,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.4141\" y=\"-7.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"218.2485,-.5 218.2485,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"256.5728\" y=\"-29.3\">(None, 10)</text>\n",
       "<polyline fill=\"none\" points=\"218.2485,-22.5 294.897,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"256.5728\" y=\"-7.3\">(None, 10)</text>\n",
       "</g>\n",
       "<!-- 4850051056&#45;&gt;4849434408 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>4850051056-&gt;4849434408</title>\n",
       "<path d=\"M153.2778,-81.3664C153.2778,-73.1516 153.2778,-63.6579 153.2778,-54.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"156.7779,-54.6068 153.2778,-44.6068 149.7779,-54.6069 156.7779,-54.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVG(model_to_dot(model2, show_shapes=True).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Activation Layers\n",
    "- Specify the **activation function** for a layer\n",
    "- How the inputs are combined with weights in the layer\n",
    "- Options: `softmax`, `sigmoid`, `relu`, `tanh`, more...\n",
    "- Use `softmax` for multiclass outputs!  For K classes:\n",
    "$P(y=j|x) = frac{e^{x^Tw_j}}{\\sum\\limits_{k=1}^K e^{x^Tw_k}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Merge Layers\n",
    "- Merge multiple `Sequential` models into a single layer\n",
    "- A number of options for merging outputs: `concat`, `sum`, `ave`, etc\n",
    "- Like so:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Build a branched model (skip-grams!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-07T19:58:09.424826Z",
     "start_time": "2017-11-07T19:58:09.114628Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"296pt\" viewBox=\"0.00 0.00 561.68 296.00\" width=\"562pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 292)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-292 557.6797,-292 557.6797,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 4850456000 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>4850456000</title>\n",
       "<polygon fill=\"none\" points=\"0,-243.5 0,-287.5 267.6797,-287.5 267.6797,-243.5 0,-243.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"64.1812\" y=\"-261.3\">input_1: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"128.3623,-243.5 128.3623,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"156.1968\" y=\"-272.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"128.3623,-265.5 184.0313,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"156.1968\" y=\"-250.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"184.0313,-243.5 184.0313,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"225.8555\" y=\"-272.3\">(None, 784)</text>\n",
       "<polyline fill=\"none\" points=\"184.0313,-265.5 267.6797,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"225.8555\" y=\"-250.3\">(None, 784)</text>\n",
       "</g>\n",
       "<!-- 4850160304 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>4850160304</title>\n",
       "<polygon fill=\"none\" points=\"22.0552,-162.5 22.0552,-206.5 265.6245,-206.5 265.6245,-162.5 22.0552,-162.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"74.1812\" y=\"-180.3\">dense_9: Dense</text>\n",
       "<polyline fill=\"none\" points=\"126.3071,-162.5 126.3071,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"154.1416\" y=\"-191.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"126.3071,-184.5 181.9761,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"154.1416\" y=\"-169.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"181.9761,-162.5 181.9761,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"223.8003\" y=\"-191.3\">(None, 784)</text>\n",
       "<polyline fill=\"none\" points=\"181.9761,-184.5 265.6245,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"223.8003\" y=\"-169.3\">(None, 32)</text>\n",
       "</g>\n",
       "<!-- 4850456000&#45;&gt;4850160304 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>4850456000-&gt;4850160304</title>\n",
       "<path d=\"M136.5724,-243.3664C137.5866,-235.1516 138.7586,-225.6579 139.8614,-216.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"143.3589,-216.9604 141.1106,-206.6068 136.4116,-216.1026 143.3589,-216.9604\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4850161144 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>4850161144</title>\n",
       "<polygon fill=\"none\" points=\"286,-243.5 286,-287.5 553.6797,-287.5 553.6797,-243.5 286,-243.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"350.1812\" y=\"-261.3\">input_2: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"414.3623,-243.5 414.3623,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"442.1968\" y=\"-272.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"414.3623,-265.5 470.0313,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"442.1968\" y=\"-250.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"470.0313,-243.5 470.0313,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"511.8555\" y=\"-272.3\">(None, 784)</text>\n",
       "<polyline fill=\"none\" points=\"470.0313,-265.5 553.6797,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"511.8555\" y=\"-250.3\">(None, 784)</text>\n",
       "</g>\n",
       "<!-- 4850483040 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>4850483040</title>\n",
       "<polygon fill=\"none\" points=\"288.5552,-162.5 288.5552,-206.5 539.1245,-206.5 539.1245,-162.5 288.5552,-162.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"344.1812\" y=\"-180.3\">dense_10: Dense</text>\n",
       "<polyline fill=\"none\" points=\"399.8071,-162.5 399.8071,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"427.6416\" y=\"-191.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"399.8071,-184.5 455.4761,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"427.6416\" y=\"-169.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"455.4761,-162.5 455.4761,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"497.3003\" y=\"-191.3\">(None, 784)</text>\n",
       "<polyline fill=\"none\" points=\"455.4761,-184.5 539.1245,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"497.3003\" y=\"-169.3\">(None, 32)</text>\n",
       "</g>\n",
       "<!-- 4850161144&#45;&gt;4850483040 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>4850161144-&gt;4850483040</title>\n",
       "<path d=\"M418.2003,-243.3664C417.5918,-235.1516 416.8886,-225.6579 416.2269,-216.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"419.7066,-216.3209 415.4774,-206.6068 412.7258,-216.8381 419.7066,-216.3209\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4850456896 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>4850456896</title>\n",
       "<polygon fill=\"none\" points=\"84.8755,-81.5 84.8755,-125.5 466.8042,-125.5 466.8042,-81.5 84.8755,-81.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"171.1948\" y=\"-99.3\">concatenate_1: Concatenate</text>\n",
       "<polyline fill=\"none\" points=\"257.5142,-81.5 257.5142,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"285.3486\" y=\"-110.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"257.5142,-103.5 313.1831,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"285.3486\" y=\"-88.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"313.1831,-81.5 313.1831,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389.9937\" y=\"-110.3\">[(None, 32), (None, 32)]</text>\n",
       "<polyline fill=\"none\" points=\"313.1831,-103.5 466.8042,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"389.5073\" y=\"-88.3\">(None, 64)</text>\n",
       "</g>\n",
       "<!-- 4850160304&#45;&gt;4850456896 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>4850160304-&gt;4850456896</title>\n",
       "<path d=\"M179.9094,-162.3664C195.7701,-152.6337 214.5565,-141.1057 231.2819,-130.8424\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"233.1212,-133.8202 239.8139,-125.6068 229.4601,-127.8539 233.1212,-133.8202\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4850483040&#45;&gt;4850456896 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>4850483040-&gt;4850456896</title>\n",
       "<path d=\"M376.1307,-162.3664C359.5491,-152.6337 339.9088,-141.1057 322.4232,-130.8424\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"323.8992,-127.6504 313.5034,-125.6068 320.3558,-133.6874 323.8992,-127.6504\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4850456672 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "<title>4850456672</title>\n",
       "<polygon fill=\"none\" points=\"154.3115,-.5 154.3115,-44.5 397.3682,-44.5 397.3682,-.5 154.3115,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"209.6812\" y=\"-18.3\">dense_11: Dense</text>\n",
       "<polyline fill=\"none\" points=\"265.0508,-.5 265.0508,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"292.8853\" y=\"-29.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"265.0508,-22.5 320.7197,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"292.8853\" y=\"-7.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"320.7197,-.5 320.7197,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"359.0439\" y=\"-29.3\">(None, 64)</text>\n",
       "<polyline fill=\"none\" points=\"320.7197,-22.5 397.3682,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"359.0439\" y=\"-7.3\">(None, 10)</text>\n",
       "</g>\n",
       "<!-- 4850456896&#45;&gt;4850456672 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "<title>4850456896-&gt;4850456672</title>\n",
       "<path d=\"M275.8398,-81.3664C275.8398,-73.1516 275.8398,-63.6579 275.8398,-54.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"279.3399,-54.6068 275.8398,-44.6068 272.3399,-54.6069 279.3399,-54.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the 2 models\n",
    "left_input = keras.layers.Input(shape=(784, ))\n",
    "left_dense = keras.layers.Dense(32, activation=\"relu\")(left_input)\n",
    "\n",
    "right_input = keras.layers.Input(shape=(784, ))\n",
    "right_dense = keras.layers.Dense(32, activation=\"relu\")(right_input)\n",
    "\n",
    "# Merge\n",
    "merged = keras.layers.Concatenate(axis=-1)([left_dense, right_dense])\n",
    "\n",
    "out = keras.layers.Dense(10, activation='softmax')(merged)\n",
    "final_model = keras.models.Model(inputs=[left_input, right_input], outputs=out)\n",
    "\n",
    "# Take a look\n",
    "SVG(\n",
    "    model_to_dot(final_model, show_shapes=True).create(\n",
    "        prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Advanced Layers\n",
    "- Bunch of additional layer options\n",
    "- For state of the art performance on NLP, Image Proc, etc\n",
    "- e.g.:\n",
    "  - Convolutional Layers\n",
    "  - Pooling Layers\n",
    "  - Recurrent Layers\n",
    "  - Embedding Layers\n",
    "  - and more!\n",
    "- We'll briefly touch on a few as we see them in examples\n",
    "- Remember, today is about applications!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Step 3 our first NLP model (warning: this one won't work well)\n",
    "- Let's try a simple example with document classification\n",
    "- We'll use only Dense and Activation Layers\n",
    "- Plus 1 more!\n",
    "- **Dropout**:\n",
    "  - Controls for **Overfitting** $\\rightarrow$ **Regularization**\n",
    "  - Randomly zeros out certain inputs in the layer\n",
    "- We'll use the famous Reuters Document Classification set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T04:24:05.089803Z",
     "start_time": "2017-11-08T04:23:26.876226Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "8982/8982 [==============================] - 2s 249us/step - loss: 4.6568 - acc: 0.2144\n",
      "Epoch 2/20\n",
      "8982/8982 [==============================] - 2s 168us/step - loss: 4.8070 - acc: 0.2142\n",
      "Epoch 3/20\n",
      "8982/8982 [==============================] - 2s 188us/step - loss: 4.8040 - acc: 0.2086\n",
      "Epoch 4/20\n",
      "8982/8982 [==============================] - 1s 166us/step - loss: 4.8585 - acc: 0.2095\n",
      "Epoch 5/20\n",
      "8982/8982 [==============================] - 2s 214us/step - loss: 4.7593 - acc: 0.2138\n",
      "Epoch 6/20\n",
      "8982/8982 [==============================] - 2s 178us/step - loss: 4.7957 - acc: 0.2126\n",
      "Epoch 7/20\n",
      "8982/8982 [==============================] - 2s 187us/step - loss: 4.7776 - acc: 0.2103\n",
      "Epoch 8/20\n",
      "8982/8982 [==============================] - 2s 174us/step - loss: 4.7395 - acc: 0.2147\n",
      "Epoch 9/20\n",
      "8982/8982 [==============================] - 2s 167us/step - loss: 4.8642 - acc: 0.2095\n",
      "Epoch 10/20\n",
      "8982/8982 [==============================] - 2s 193us/step - loss: 4.7918 - acc: 0.2139\n",
      "Epoch 11/20\n",
      "8982/8982 [==============================] - 2s 206us/step - loss: 4.8475 - acc: 0.2182\n",
      "Epoch 12/20\n",
      "8982/8982 [==============================] - 2s 223us/step - loss: 4.8556 - acc: 0.2148\n",
      "Epoch 13/20\n",
      "8982/8982 [==============================] - 2s 204us/step - loss: 4.7109 - acc: 0.2202\n",
      "Epoch 14/20\n",
      "8982/8982 [==============================] - 2s 224us/step - loss: 4.7890 - acc: 0.2080\n",
      "Epoch 15/20\n",
      "8982/8982 [==============================] - 2s 198us/step - loss: 4.7648 - acc: 0.2132\n",
      "Epoch 16/20\n",
      "8982/8982 [==============================] - 2s 214us/step - loss: 4.8778 - acc: 0.2070\n",
      "Epoch 17/20\n",
      "8982/8982 [==============================] - 2s 209us/step - loss: 4.8008 - acc: 0.2141\n",
      "Epoch 18/20\n",
      "8982/8982 [==============================] - 2s 212us/step - loss: 4.7959 - acc: 0.2141\n",
      "Epoch 19/20\n",
      "8982/8982 [==============================] - 2s 202us/step - loss: 4.6940 - acc: 0.2159\n",
      "Epoch 20/20\n",
      "8982/8982 [==============================] - 2s 210us/step - loss: 4.6868 - acc: 0.2141\n",
      "2246/2246 [==============================] - 1s 253us/step\n"
     ]
    }
   ],
   "source": [
    "# Set the max number of words to keep, 2000 most frequent \n",
    "max_features = 2000\n",
    "(X_train, y_train), (X_test, y_test) = keras.datasets.reuters.load_data(num_words=max_features)\n",
    "maxlen = 10\n",
    "# Data is stored in sentences, pad any that are shorter than 10 words with zeros\n",
    "X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=maxlen)\n",
    "X_test = keras.preprocessing.sequence.pad_sequences(X_test, maxlen=maxlen)\n",
    "\n",
    "# Dense(64) is a fully-connected layer with 64 hidden units.\n",
    "# in the first layer, you must specify the expected input data shape:\n",
    "# here, 20-dimensional vectors.\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(64, input_dim=10, activation=\"tanh\"),\n",
    "    keras.layers.Dropout(.5),\n",
    "    keras.layers.Dense(64, activation=\"tanh\"),\n",
    "    keras.layers.Dropout(.5),\n",
    "    keras.layers.Dense(46, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "\n",
    "# Momentum: when gradient is noisy, it can be helpful to give the gradient a sense of weight so that it doesn't rapidly change from step to step\n",
    "model.compile(loss='sparse_categorical_crossentropy',\n",
    "              optimizer=keras.optimizers.SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, y_train,\n",
    "          epochs=20,\n",
    "          batch_size=16)\n",
    "score = model.evaluate(X_test, y_test, batch_size=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Mmm...not a great job!  Although with 46 classes it's okay, but I think we can do better.  We'll return!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A (slightly) more complex example: Skipgrams!\n",
    "- Back to your old favorite\n",
    "- Let's see if we can implement Skipgrams in `Keras`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading in Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T04:33:44.554617Z",
     "start_time": "2017-11-08T04:33:44.539541Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Load in Corpus using Keras utility\n",
    "# We'll use Alice in Wonderland :)\n",
    "path = keras.utils.get_file('carrol-alice.txt', origin=\"http://www.gutenberg.org/files/11/11-0.txt\")\n",
    "corpus = open(path).readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Keras: Preprocessing Data\n",
    "- `Keras` has some nice text preprocessing features too!\n",
    "- We need to convert into a special format, like below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T04:36:38.039787Z",
     "start_time": "2017-11-08T04:36:37.825937Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1823, 1295, 247, 339, 10, 479, 58, 1004, 1005],\n",
       " [22, 439, 31, 24, 1, 149, 6, 705, 1006, 19, 46, 818, 3, 18],\n",
       " [480, 46, 1296, 1297, 11, 174, 340, 8, 311, 8, 167, 27],\n",
       " [1298, 149, 8, 204, 1, 205, 6, 1, 56, 47, 294, 1007],\n",
       " [18, 22, 439, 27, 819, 19, 621, 47, 341]]"
      ]
     },
     "execution_count": 239,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras.preprocessing.text\n",
    "# For simplicity, one \"sentence\" per line \n",
    "corpus = [sentence for sentence in corpus if sentence.count(\" \") >= 2]\n",
    "\n",
    "# Tokenize using Keras\n",
    "tokenizer = keras.preprocessing.text.Tokenizer(filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n‘'+\"'\", num_words=3000)\n",
    "tokenizer.fit_on_texts(corpus)\n",
    "\n",
    "# Convert tokenized sentences to sequence format\n",
    "sequences = tokenizer.texts_to_sequences(corpus)\n",
    "nb_samples = sum(len(s) for s in corpus)\n",
    "\n",
    "# Vocab size\n",
    "V = len(tokenizer.word_index) + 1\n",
    "# Dimension to reduce to\n",
    "dim = 100\n",
    "window_size = 2\n",
    "sequences[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Skipgrams: Generating Input and Output Labels\n",
    "- Now we need to generate our `X_train` and `y_train`\n",
    "- So we can train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T04:36:39.632849Z",
     "start_time": "2017-11-08T04:36:39.466463Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Function to generate the inputs and outputs for all windows\n",
    "def generate_data(sequences, window_size):\n",
    "    x = []\n",
    "    y = []\n",
    "    maxlen = window_size*2\n",
    "    # For each line (sentence)\n",
    "    for words in sequences:\n",
    "        L = len(words)\n",
    "        # Choose the target word\n",
    "        for target_i, word in enumerate(words):\n",
    "            # Create the window\n",
    "            start_i = target_i-window_size\n",
    "            end_i = target_i+window_size+1\n",
    "                    \n",
    "\n",
    "            # Create the input/outputs for skipgrams\n",
    "            for i in range(start_i, end_i):\n",
    "                if i != target_i and 0 <= i < L:\n",
    "                    x.append(word)\n",
    "                    y.append(words[i])\n",
    "\n",
    "    return np.array(x), np.array(y)\n",
    "\n",
    "x, y = generate_data(sequences, window_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T04:36:40.102862Z",
     "start_time": "2017-11-08T04:36:40.092372Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(105600,)"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Skipgrams: Creating the Model\n",
    "- Lastly, we create the (shallow) network!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T04:40:40.837025Z",
     "start_time": "2017-11-08T04:40:40.549317Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"296pt\" viewBox=\"0.00 0.00 337.89 296.00\" width=\"338pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 292)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-292 333.8867,-292 333.8867,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 4977263168 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>4977263168</title>\n",
       "<polygon fill=\"none\" points=\"0,-243.5 0,-287.5 329.8867,-287.5 329.8867,-243.5 0,-243.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"102.2847\" y=\"-261.3\">embedding_12_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"204.5693,-243.5 204.5693,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4038\" y=\"-272.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"204.5693,-265.5 260.2383,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4038\" y=\"-250.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"260.2383,-243.5 260.2383,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"295.0625\" y=\"-272.3\">(None, 1)</text>\n",
       "<polyline fill=\"none\" points=\"260.2383,-265.5 329.8867,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"295.0625\" y=\"-250.3\">(None, 1)</text>\n",
       "</g>\n",
       "<!-- 4977265632 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>4977265632</title>\n",
       "<polygon fill=\"none\" points=\"2.7139,-162.5 2.7139,-206.5 327.1729,-206.5 327.1729,-162.5 2.7139,-162.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"88.2847\" y=\"-180.3\">embedding_12: Embedding</text>\n",
       "<polyline fill=\"none\" points=\"173.8555,-162.5 173.8555,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"201.6899\" y=\"-191.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"173.8555,-184.5 229.5244,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"201.6899\" y=\"-169.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"229.5244,-162.5 229.5244,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"278.3486\" y=\"-191.3\">(None, 1)</text>\n",
       "<polyline fill=\"none\" points=\"229.5244,-184.5 327.1729,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"278.3486\" y=\"-169.3\">(None, 1, 100)</text>\n",
       "</g>\n",
       "<!-- 4977263168&#45;&gt;4977265632 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>4977263168-&gt;4977265632</title>\n",
       "<path d=\"M164.9434,-243.3664C164.9434,-235.1516 164.9434,-225.6579 164.9434,-216.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"168.4435,-216.6068 164.9434,-206.6068 161.4435,-216.6069 168.4435,-216.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4977265744 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>4977265744</title>\n",
       "<polygon fill=\"none\" points=\"21,-81.5 21,-125.5 308.8867,-125.5 308.8867,-81.5 21,-81.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"88.2847\" y=\"-99.3\">reshape_12: Reshape</text>\n",
       "<polyline fill=\"none\" points=\"155.5693,-81.5 155.5693,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"183.4038\" y=\"-110.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"155.5693,-103.5 211.2383,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"183.4038\" y=\"-88.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"211.2383,-81.5 211.2383,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"260.0625\" y=\"-110.3\">(None, 1, 100)</text>\n",
       "<polyline fill=\"none\" points=\"211.2383,-103.5 308.8867,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"260.0625\" y=\"-88.3\">(None, 100)</text>\n",
       "</g>\n",
       "<!-- 4977265632&#45;&gt;4977265744 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>4977265632-&gt;4977265744</title>\n",
       "<path d=\"M164.9434,-162.3664C164.9434,-154.1516 164.9434,-144.6579 164.9434,-135.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"168.4435,-135.6068 164.9434,-125.6068 161.4435,-135.6069 168.4435,-135.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4828388880 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>4828388880</title>\n",
       "<polygon fill=\"none\" points=\"36.1587,-.5 36.1587,-44.5 293.728,-44.5 293.728,-.5 36.1587,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"91.7847\" y=\"-18.3\">dense_51: Dense</text>\n",
       "<polyline fill=\"none\" points=\"147.4106,-.5 147.4106,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"175.2451\" y=\"-29.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"147.4106,-22.5 203.0796,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"175.2451\" y=\"-7.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"203.0796,-.5 203.0796,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"248.4038\" y=\"-29.3\">(None, 100)</text>\n",
       "<polyline fill=\"none\" points=\"203.0796,-22.5 293.728,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"248.4038\" y=\"-7.3\">(None, 3388)</text>\n",
       "</g>\n",
       "<!-- 4977265744&#45;&gt;4828388880 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>4977265744-&gt;4828388880</title>\n",
       "<path d=\"M164.9434,-81.3664C164.9434,-73.1516 164.9434,-63.6579 164.9434,-54.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"168.4435,-54.6068 164.9434,-44.6068 161.4435,-54.6069 168.4435,-54.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 245,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the Keras model and view it \n",
    "skipgram = keras.models.Sequential([\n",
    "    keras.layers.Embedding(input_dim=V, output_dim=100, input_length=1, embeddings_initializer=\"glorot_uniform\"),\n",
    "    keras.layers.Reshape((dim,)),\n",
    "    keras.layers.Dense(input_dim=dim, units=V, activation='softmax')\n",
    "])\n",
    "skipgram.compile(loss='sparse_categorical_crossentropy', optimizer=\"adadelta\")\n",
    "SVG(model_to_dot(skipgram, show_shapes=True).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Skipgrams: Compiling and Training\n",
    "- Time to compile and train\n",
    "- We use crossentropy, common loss for classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T04:49:36.778320Z",
     "start_time": "2017-11-08T04:40:42.399953Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 79200 samples, validate on 26400 samples\n",
      "Epoch 1/10\n",
      "79200/79200 [==============================] - 47s 595us/step - loss: 8.0128 - val_loss: 7.8817\n",
      "Epoch 2/10\n",
      "79200/79200 [==============================] - 46s 577us/step - loss: 7.5495 - val_loss: 7.4449\n",
      "Epoch 3/10\n",
      "79200/79200 [==============================] - 52s 655us/step - loss: 7.1274 - val_loss: 7.2062\n",
      "Epoch 4/10\n",
      "79200/79200 [==============================] - 52s 661us/step - loss: 6.8406 - val_loss: 7.0404\n",
      "Epoch 5/10\n",
      "79200/79200 [==============================] - 53s 665us/step - loss: 6.6353 - val_loss: 6.9285\n",
      "Epoch 6/10\n",
      "79200/79200 [==============================] - 52s 660us/step - loss: 6.4918 - val_loss: 6.8487\n",
      "Epoch 7/10\n",
      "79200/79200 [==============================] - 54s 676us/step - loss: 6.3880 - val_loss: 6.7904\n",
      "Epoch 8/10\n",
      "79200/79200 [==============================] - 64s 805us/step - loss: 6.3091 - val_loss: 6.7464\n",
      "Epoch 9/10\n",
      "79200/79200 [==============================] - 58s 727us/step - loss: 6.2462 - val_loss: 6.7122\n",
      "Epoch 10/10\n",
      "79200/79200 [==============================] - 57s 721us/step - loss: 6.1942 - val_loss: 6.6810\n"
     ]
    }
   ],
   "source": [
    "history = skipgram.fit(x,y,batch_size=128, epochs=10, validation_split=.25);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 301,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:44:31.565667Z",
     "start_time": "2017-11-08T12:44:30.915261Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/matplotlib/figure.py:1999: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.\n",
      "  warnings.warn(\"This figure includes Axes that are not compatible \"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa4AAAGoCAYAAAAerAGHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xl4VOXd//H3bMnMZJnsOyQEyCEE\nwhI2FVxRQbRSRCsoqNVan1br0sdabbX2qbVVXH62VVukuLVoKYWqqIgLiqIIBFmEcICwJiH7nskk\nk5nz++MkISGJJJBtku/ruuaambPNdwYynznn3Pd9DJqmIYQQQvgKY18XIIQQQnSFBJcQQgifIsEl\nhBDCp0hwCSGE8CkSXEIIIXyKBJcQQgifYu7MQoqiTAWeUFX1wlOmXwU8AjQAy1VVfUlRFBvwDyAK\nqAJuUlW1qFurFkIIMWiddo9LUZRfAMsA6ynTLcCzwGXABcDtiqJEA/8D7FZVdQbwGvDr7i5aCCHE\n4NWZPa5sYB7w+inTU4GDqqqWASiK8gVwPjAdeLJxmfeBh9vbaGZmpvR8FkII0aGMjAxDe9NPG1yq\nqv5HUZSkdmYFAxUtnlcBjlOmN01rl91uP93Ln5bL5cJqtZ5+wX5Eau4dvlazr9ULUnNvGYw1O53O\nDud16hxXByqBoBbPg4DyU6Y3TWtXamrqWby8Lisrq1u205uk5t7hazX7Wr0gNfeWwVhzZmZmh/PO\nJriygJGKooQB1eiHCZ8CEoErgC3AbODzs3gNIYQQopUuB5eiKAuBQFVVlyqKch/wAXojj+WqquYq\nivIi8GrjOa96YGG3ViyEEGJQ61Rwqap6BJjW+HhFi+nvAO+csqwTuLb7ShRCCHGmPF6NoyU15JW7\nqHK528x3ezUKKlyU1NR3absaGmU19RRU1tHg9TZPT08I4YFZo8667u9yNocKhRBC9IHCKhdqfhUh\nNj8sZgM7j5dTUlNPTLCVrBOVfJRViMlowGGzsO9EJTX1ntNu02IyYDC024ivQyE2CzEOK36mkz2r\n6hu837FG95DgEkKIfuhERS0HCqrxahrHc53kGwrZdrSU93fnc6i4psP1LCYD542IwN9spKzGzbyJ\nCaQnOEgItRNit3BqNpkMBqKCrQRbzV0Orr4iwSWEEL2gtt7D7twKdudW4PZ4qW/wcqLCxYmKWvLK\na3G22Ctyub0UV9edsoV8jAY4d3gEC6YMJS0umKq6BmrrPYxNcBATbCW/0kVEoD8Om6V331wvk+AS\nQohuUuF0s27PCTYeKCavvJYKpxsNqKx1t3sOKTzAj9gQK4nhAQRZT34dm40GUmODGR0bjMVs5MiR\nIyQlJTE0zE5EoH+Hrz88MrAn3la/I8ElhBCNNE1rPlymaRoVtW5OVLiwWkxEBflT5qznRIWLvPJa\nsgur2X6sHA2NcQkhqPlVbDxQhNujEeewMiwygPgQGwaDgUB/M3EOK6mxwYwbEkKgvxmT0YCfuXPj\nnNtq8kkdGtqTb92n9Nvg+k9mDiu3HT/tck6nE/vGDvs4t3LdpCFck5FwtqUJIXxIg8dL1okqsvIr\n8Xg1TpyoZEfVMQA0Dcqc9RwvdbLjeDkHC6sJD/QjwM/MiQoXte6OGzUYDZAaGwzAXz/LJibYyi3n\nDeOq9DjGxAf7zPkiX9Rvg6svVFdX86tf/YqqqioKCwtZuHAhaWlpPP7443i9XqKjo3nqqadQVbXN\nNF8bjkWIgarC6eZgURU5ZbV8phbx4d4CquoaTlmquNWzsAA/0uKCOT8lkrKaepz1Hi5UoogLsRLr\nsOFyeyisqiPEbiHWYSUuxEZCqA27n/4V6nJ78DMZMRolrHpDvw2uazISOrV31J1DoRw9epQ5c+Zw\n2WWXUVBQwKJFi7DZbDzzzDMMHz6cf//732RnZ/PII4+0mZaWltYtNQgh2vJ4NQ4WVpNXXktueS35\njXtDXk2jqKqOoqo6vJpGmdPNwcLq5vWCrWZmj41hxshIxsY7sFpMHDhwgJEjRzYv47BZsPmZzqo+\nq+Xs1hdd02+Dqy9ERETw6quvsn79egIDA2loaKC4uJjhw4cDcO21er/q9qYJIc6Oy+2hqEpvSVfu\ndJNXUcuJ8loOFFbzwZ6CVq3sTEYDNosJAxAe6EdUsBWLyUhSuJ254+NIi3MQF2JjWERAm/NIZQFm\nYhxyhMSXSXC1sHz5csaPH8/ChQvZvHkzn332GVFRUc0tepYuXcqwYcPanXbppZf2dflC9FvHSpyo\nBVXNzz1e/bzTnrxK3B4v5bVu9uZV4Pa0vdqRzWLiolGRzEyNJjE8gLgQK1FBVkxyWG7QkuBq4aKL\nLuKxxx7jvffeIygoCJPJxKOPPspDDz2E0WgkMjKSm2++mejo6DbThBjMKmrdfLS3gIpaN9HBVo6U\n1LAnr4L6Bo2CShe7cyvarGM0wIioQGx+ZmwWIz+cPozhEYEYDBBssxDnsBEbYiU8wE8aOohWJLha\nmDZtGmvXrm0zfcWKFa2ep6ent5kmhK9zuT3Ue04O11PlamDHsXLU/Epyy/WOsvmVLuIcNsbEO6ip\na2B/ThHVHxZzoKC61boASeF27H5mAv3NPDh7FFOTwzG32EtKDLcTZB3YHWVFz5DgEmKA8Xi15sNo\nXq9GcXUdJypc2PxMhAX4sT+/it25FeSV15LXOHLDifKOB1k1GiA62Eqsw4oSHcTREidLN2YT6G8m\nzGZkWFQA542IYPaYGBLDA8ivcBEXYiXE7tebb1sMIhJcQvigoqo6Mo+WUVDpQtM0Sp1ucsr0vkiH\nimoItVuw+5kprHK1e94IIMhqJs5hIy7ESnpCCLHB1lat6/wtJtLjHaTGBrdp4NAUju216g0LkMAS\nPUuCSwgf4HJ7+FQtZP3eArYdKeNYaevLmhsMEB1kZUx8MFeMiaXMqfdFig62Eh9iJcZho9btobiq\njuTIAMYPCTmrPSJpGCH6kgSXEP2EV9MorHLxVXYJf//iMKU19VyeFkNpTT0f7i2guq6BsAA/piSF\nsWhaIhMTQ0kKt2MwGAiymrGYOjd8kBC+ToJLiD5SUeumpLqO6roG3thynNXbj1PXcBiA5MgAUqKD\neO2rI9j9zFyZHsuV6XFMSw7DLAElBjkJLiF62eHiGl76/BD/ycyhrvGie35mIxckBTB9dCIjogI5\nJzkco9GAs74Bi8koe1NCtCDBdQYWLVrEo48+2jx6hhAdqXS58TcbMRoMfHGwmDe3HGP93gIsJiPX\nTIxn6jA9oM5JDqc45xCpqUmt1m8aC08IcVL//avY8QZ884/TLjbUWQObAzq3zQk3wvgFZ1mYEO1z\nuT18sq+QHcfLyS2vZW9eJYcbr1TrbzZS1+AlxG7hzotGsPicJCKDWl9Xqbi9jQoh2ui/wdUH7rzz\nThYvXsyUKVPYvXs3Tz75JGFhYa1Gi1+4cOFpt7Nu3Tr++c9/0tDQgMFg4C9/+QuhoaH87ne/Y9eu\nXVRVVXH//fdzySWXNE9zu93cddddzJw5sxfeqegumqZxpMTJ8i8Os3p7DjX1HvzMRuIcVkZEBTE/\nIwGPV6Pc6ebc4eHMSInA3ywDsgpxNk4bXIqiGIEXgHFAHXCbqqoHG+eNB/5fi8WnAXOBLcB+4NvG\n6WtUVX2uS5WNX9CpvaNj3Tg6/LXXXsuaNWuYMmUKq1evZurUqaSkpLQaLb4zwXXkyBGWLl2KzWbj\nkUce4YsvvsBms1FWVsaqVavYunUrmzZtQtO05mkVFRW8/PLLElz9UFlNPQeLqvF6NQqq6th+tIzs\nIn2k8hMVLpz1HiwmA1eNi+OaiQlMHSYNKIToSZ3Z45oLWFVVPUdRlGnA08DVAKqq7gAuBFAU5Vog\nV1XVdYqizATeUFX1rp4pu2fMmDGDJUuWUF5ezrZt21i2bBlPP/10q9HiOyM8PJwHHniAgIAADh06\nxPjx4zl8+DDjx48HIDAwkHvuuYelS5c2T3M4HNxzzz099t5E53m8GlknKnn/2xN8sKeg1WUyAOx+\nJkZEBTIyKojzUyIZEmrnyvRYooJlxHEhekNngms6sA5AVdXNiqJMOnUBRVECgN8C5zdOygAyFEX5\nDCgEfqaq6onuKbnnGI1GZs2axaOPPsrMmTPbHS3+dKqqqvjTn/7Ep59+CsAtt9yCpmkkJyezbt06\nAGpqarj11ltZsGBB87Sqqiruuece/v73v/fY+xMnVbncfLKvELufGYfNQmGViwMF1Ww/VsY3x8qp\nrmvA1NhoYt7EeEbHBuNnMhJsszAqJkj2qIToQ50JrmCg5dDOHkVRzKqqttz9uBX4t6qqTeeX9wGZ\nqqp+pCjKDcCfgfmnbjgrK+sMyz7J5XJ1y3aapKen88ILL/DCCy9QWFjISy+9xKpVqwgICMDj8bBr\n1y5qamo4dOgQ9fVtx3bTNI0RI0Zw9dVXYzQaCQwMZM+ePcyfPx+Px8PcuXNxu90sWLCAuLi45mke\nj4cf/OAH3fpeulN3f869oWXNTreXDw9WcbzCTX2Dl03HanC6Ww+FZDRAUogfFybZSY30Z2KcnRCb\nCXCDtwS8+sMDlT1fr6+QmnuH1NyaQdPaH8esiaIozwCbVVVd2fg8R1XVhFOW+RqYr6rq8cbnQYBT\nVVWPoih2YLeqqq3ajmdmZmoZGRln/Qa68wrIvUVq7jkNHi/vfZvPv7Ye48CJCmrcGlHBVoqr66hy\nNRBqt2AyGpiaHM7N5ybhZzJSXusmKsifIWF2Av37rr2Sr3zGLUnNvWMw1pyZmUlGRka7Y4t15q90\nE3AVsLLxHNfuljMVRXEA/k2h1WgZ8B9gJXAJkHkmhfdnu3btYsmSJW2mz549u1MNOETXaZrG7twK\nPtlXSG29h6o6/bIbR0tqmpdp8GrUNXgZFhHA2GgrQ2MjKKqqw99sYtE5iYwfEtKH70AI0R06E1xr\ngEsVRfkSMAC3KIpyH3BQVdW3gRTgyCnr/BJYrijKT4Aa4LbuK7l/SE9P5/XXX+/rMgakCqebbUdL\nafBqlFTXs/1YGQcKq8ktc1JcXY/RoI804Wcykp4QwtTkMEyNFxo0GGByUhgzU6NR1X0+9ytVCHF6\npw0uVVW9wB2nTN7XYv5W9JaHLdc5DFzUHQWKwUHTNL7KLuHlL4/wqVrY6lIcYQF+jI4N5uJRUUxK\nDOPytBgcdrkAoRCDlXRAFn2i0uVmw75Cth8t42BRNUdLnOSU1RIR6MdN5yRx6ehoAq361XOHhtnl\n0u1CiGYSXKLHHStxUl3XQESQHwcLq/kkq5A3tx6nuq4Bu5+JlOggxiWE8NOLRvD9CfFYLTKyhBCi\nYxJcoke4PV6+zC7htS+P8PG+wlbzTEYDc8bGctO5SYxLcEifKCFEl0hwiW5TWlPPOzvz2HKklC8P\nFlPmdBMW4Mfdl4wkJTqIwioXSREBTBwSKueohBBnTIJLnLWjJTUs+/ww/848jsvtJT7ExgUpkVwx\nNpbzUyLl0J8QoltJcIkzdrzUyZIPVNbuysNsNDJ3Qhy3Tk9GiQnq69KEEAOYBJfoErfHy7YjZbyz\nK49VmTmYDAZuP384PzwvSQaZFUL0Cgku8Z3KnXoH4A+2l3Jk41fszCnH5fZis5i4elwc912WQqzD\n1tdlCiEGEQku0YrL7eG37+zhw70FaBqU1OgDCRsNMCbewfWThzJlWBgXKpFyWXkhRJ+Qbx7R7ERF\nLbe/lsnu3Aq+Ny6OIKuZWIeVjMQwrDX5TEhP6+sShRBCgkvoDhZWsfjvW6h0NbBs8SRmjo5uNT8r\nq7CDNYUQondJcA1SOWVO9p2oIre8ll05FXy4Nx8/s4l//XgaaXGOvi5PCCE6JME1iLjcHlZvz2Xl\ntuPsOF7ePD0i0I/zRkTwy9mjSAwP6MMKhRDi9CS4BgG3x8s/Nh/l+Q0HKa6uZ1RMEA/MGsW05DDi\nQ2xEBvnLILZCCJ8hwTVAaZrGxgPFbNxfxMdZBRwpcXLu8HD+vGAk05LDJKiEED5LgmuA8Xo18ipq\n+f27Wbz/bT7+ZiPjhoTw6zmjuSQ1SgJLCOHzJLgGiAqnm9+u3cPanSeo93ixmAz8YpbCbdOT8TPL\n6OtCiIFDgsvH5ZbXsnZnHi9vOkJxdR3XTxlCSnQQ5w6PYERUYF+XJ4QQ3U6Cy0ftOF7OCxsO8mGW\nPsLFhKEhvLR4EmMTpCm7EGJgk+DyMZqm8eJn2Sz5QCXYauEnFw7nuklDpBm7EGLQkODyEZqm8fXh\nUp7fcJDPDxRz1bg4/jBvLIH+8k8ohBhcTvutpyiKEXgBGAfUAbepqnqwxfzngOlAVeOkqwELsAKw\nAXnALaqqOru39IGvqf/Viq+PkVNWS63bQ3iAH7+5ajQ3n5skLQSFEINSZ36uzwWsqqqeoyjKNOBp\n9HBqkgFcrqpqcdMERVH+BKxQVfUVRVF+CfwYeLYb6x7wsk5UcueK7WQX1TApMZQZIyMZFRvE98bF\nyRWFhRCDWmeCazqwDkBV1c2KokxqmtG4NzYSWKooSjTwd1VVlzeu83jjYu83Ppbg6gRN03hrRx4P\nrt5NsM3MssWTpP+VEEK00JngCgYqWjz3KIpiVlW1AQgA/gw8A5iADYqibDtlnSqg3aZuWVlZZ1p3\nM5fL1S3b6U0d1XyotI6lW0vYme9idJQ/v7owmjBDKfv2lfZBla0NpM+5v/K1ekFq7i1Sc2udCa5K\nIKjFc2NjaAE4geeazl8pivIJ+rmwpnVqG+/LaUdqauoZln1SVlZWt2ynN51ac1FVHc98qPLm1lxC\nbBZ+d3UaC6YMxWzqPx2HB8Ln3N/5Wr0gNfeWwVhzZmZmh/M6E1ybgKuAlY3nuHa3mJcC/EtRlAmA\nEf0Q4auN61wBvALMBj4/k8IHuvoGL3//4jDPbziIy+3h1vOGcdfFI3HYLX1dmhBC9FudCa41wKWK\nonwJGIBbFEW5DzioqurbiqK8DmwG3MBrqqruURTlMeBVRVF+BBQDC3uofp+1N6+S+1buYF9+FTNT\no3noilEkR8pIF0IIcTqnDS5VVb3AHadM3tdi/hJgySnrFACzuqPAgaSspp73v83nX1/lsbvgEGEB\n/ry0eBKXnnK1YSFED/N6wFXRdnpNMWS9BXk7QNM6t63QREj7PoSPgLoq2L8OjnwOXm/bZeMmwOjv\nQUBk23n+wWDqhX6ZDfVQX912utkKfvbGZeqgvqbjbfgHganxyFB9jb58E79AMPt1X73tkN6rvUDT\nNH63NovXvjpCg1cjPtjCnReN4ObzhhEW0LP/wEIMKm4XZH8Mxfs7XCTm8E54ZyM4SzreToQCps78\nbWpw8CPY/ELryaHD9C/wljz1oL4LGx5rf1O2MD3UQpNOTvMLhJTL2y5btF9/nw2uTtTYVKoGBXtA\nfR/c7YSS0QLDLwa/AD183d/R9dY/GJQroDofDm8ErUVIx6TDHT17dkiCqxc89/EBlm86zLUZCdx0\nbhLGilxGj1b6uizRk0oPQ2Vel1YJzNkNWf8Pcrd1/td+Z4QMgdSr9HoOrIf6dr6QQhNh9Fx9r6GJ\nxQax48BogrKjsGcN7FsLzpOtXJPr6+HDFl/wYcn63kfLL9+u0rxwbLP+Je+q7Nq61YVQX/WdizhM\nVhh1BQyZAoZTGkCZrTDiEnAkdP41XZX65+os0T+rpBkQ2cHfd0WuHnSnBo7mhZxtsOvfbUPlvf9l\neEAcfGjTn3vqoeJ45+tryRYKY+dD1Gg4tYtN+THY+5YeWOk/gKgOGlZoGuTv0v8v2MPhvLshKPbk\n/MhRZ1ZbF0hw9SBnfQPPbzjI8xuymZ+RwJPz0zEYDGR18QtN9BKPu+00V6X+BXpsc+tflR3RNCjc\nq/9hd9EQAKtD/+IzW7u8fgcF6Ye93v25/iWdeJ6+N3HqMrnb4Z2ftV09MAaCYyHvG/153ET9cFcj\nV2Ul/sHBjZvx6qH71k+6p/QhU1u9VqdYg2HUlTB0WttQaqTuP0hq2thuKLDFa46d37llHfGQcVPH\n8z0N4G3x/7DqBOxZQ+2Bzfg1fc4GA8Tfqf8YsYd1rVaTPxi/o7Xy5b/vwsZeOP0iPUSCq4dsO1LK\nnSu+Ib/SxbwJ8fxh3ljpRNwVmqZ/WVYch+SL9C+H9pbJ3Q6VuTD8In3awY+bf/n6eWKBxl+NDfX6\nIY3yo223U10Ae/4LxWrH9QRE6nsgnREYA5c/DtFp6O2ZOudIzgmSzp3X/ecHNA2KVP3XcWA751aa\nlinY0/rwWXUh7P2v/vnMfLTdPam8rCwcLZs8N/0ar223B0znhY/Qv+R7grEff+2ZzK3Pc4Ulw4yf\nkxdxyuc8yPXjf0Hf9dHeAn66YjtxITZW3XEOk5K6+KvIF1XkQu0pHaU1DfJ364cUrCGQNheC49pf\n39sAR74AdZ1+4thZcvJwiMlfP/Ry6i/ommKozNEfN+2htDgEMxzg65H6MfuyI+Dq6MvUoO+JjJmn\nH+ppyWiBYefrv/x7+IdHbV1Wz5zUNhgg6jSHbwwGiBnTdnr6tV1/rdhxXVtHiC6S4Opm246U8uN/\nZJIWF8zLN08mPNC/r0vqWW4XfPo4fPnnjg+lOYbqLbh2rjj99mLS9ePlIUPhgl/ovziz1kLpobbL\nhibCyAchJFEPR02D0Vfrv9brqyn4/FWiaw/oLcii0/RDK3ETaLMXZLHqh+iEED5BgqsbVTjd3P3m\nDuJDbPzjtqkEW324I7HXA8e+gsoTAATn5ULVF5D1DuRsPdl4wNt4TH7iYhh5WdvtBMfrYeGph6Nf\ntt8Mt0l0mh5Up0qafvp6h804dWOUKguIlsMrQgw4ElzdxOX2cN/KHRRUuvjP/5zrm6F1+HP44lk9\nZIr36+c2GjWfbQhLhgmLwNxiT3L4xSfPMXXE7H/6ZYQQohMkuLpBbnktd7yeye7cCn53dRrjhoT0\ndUldd/hz+Oe1eiulkKGQeK5+2C16DGAg+1A2w0eO0udJIxMhRB+S4DpLX2YXc+eKb3A3eFm2eBIz\nfW0UDE2DnW/qzaVDE+Gmte22PKsvcuvzhRCij0lwnaEGj5e/bTzEMx/uJyncztLFkxjeX8carKvS\n+yhV5unNmwtbXGqgpgiOfw1DpsF1r3XcXFoIIfoJCa4uqGvw8MKGbAqrXOzJq2RXTgVz0mP547yx\nBPW3c1pVBXov+D1r9EYWNDamMBghIuVkXxaDAS57DKb9pG1TcCGE6IckuDpJ0zR+sWoXb+3IIzLI\nn0B/M39aMIHvjeugX1J3c5ZC0b6O51fl63tT+d/qzdLLj+r3UaP1ZuX2cL0/08jLIDCqd2oWQoge\nIMHVSU9+oPLWjjz+97IU7rx4ZM++WEMdZH+ij3/mrtVb9x3eqDc9/y4BkXpHWqMZ0q/TRzroaLwx\nIYTwURJcnfCXTw7w4qfZLJgylJ9eNOL0K5ypvB2wZane4bauQh+B2RYCFrt+KG/YBR1f9sASoPeX\n6o3LIgghRB+Sb7nT+Ntn2Ty1fj/zJsTz2NwxPTfe4JFN8M/5+t7SqDkw5hpIvvDkNW+EEEIAElzf\nafkXh/nD+/u4Mj2WJ+enYzL2UGgd+hTevEG/lMLN78o5KCGE+A4SXB14ffNR/m/tXmalxfDsD8Zj\nNn3HpQC6yOiugX3v6de9Of61fngwQoGb3pbQEkKI05Dgase/th7j4f9+y8zUKP60YAKWMwktTdP7\nTXnd+kUF96yBssPg9TAyZxt4WlzqeuodcMlvTl42WwghRIckuFoorq7jmQ/388aWY1yQEsnzN0zE\nz9zF0CrYC3tWw7eroTT75HS/QH0QWYOR8mFXETb9FgiM1qcHx3a8PSGEEK2cNrgURTGiX+pyHFAH\n3Kaq6sEW8+8Frm98+p6qqr9VFMUA5AAHGqd/parqg91aeTfbsK+Qu9/8Bme9h5vPTeKBWaPwN3ey\nQ27xAT2o9qzW+1oZjPpVbKf+GPyD9MtlJ1/YfCHCgqwswpKkmboQQpyJzuxxzQWsqqqeoyjKNOBp\n4GoARVGSgRuAqYAX+EJRlDWAE9iuqupVPVN299E0jRc+zeap9SqpMcH8acEERkR1YuimsiMnwyp/\nN/rFCM+FK57SB6eVc1VCCNEjOhNc04F1AKqqblYUZVKLeceBWaqqegAURbEALiADiFcUZQNQC9yr\nqup3XBe9b3i8Go++vYfXNx/le+PieOKadGx+p9nLqq+BD38DW1/SnydMhll/1MOqo6v7CiGE6DYG\nremCgB1QFGUZ8B9VVd9vfH4MSFZVtaHFMgZgCRCkquqPFUU5H4hWVfXfiqJMB55VVXVyy+1mZmZq\ndvvZN0ZwuVxYrdYur+fxajz1RRGfHq5mfpqDH2aEtdtHy1RXQfDRDwjK2YDZVYKpvhJzXTmlI6+j\nRFlAQ0DXz0+dac19SWrueb5WL0jNvWUw1ux0OsnIyGi3D1Jn9rgqgaAWz42nhJYVWA5UAT9pnLwN\naABQVfULRVHiFEUxqKraKiVTu+HqtFlZWV3ejtercf+qXXx6uJr7L1fajobh9eitAtV3Yf294CyB\nqDRInKx3EJ6wiLBhMwjrxZr7mtTc83ytXpCae8tgrDkzM7PDeZ0Jrk3AVcDKxnNcu5tmNO5pvQV8\noqrqEy3W+Q1QAjypKMo44PipodVXNE3j/9bu5T/bc7jv0pS2obV7Fbx1JzTU6s9jx8OiNRA7rveL\nFUII0UZngmsNcKmiKF8CBuAWRVHuAw4CJuACwF9RlNmNyz8I/BH4h6Ioc9D3vG7u7sLP1LLPD/PK\nl0e4bfowfnZJ42C5zlK9z1XeN/DO3ZAwCUZcqjewGL9Qhl0SQoh+5LTBpaqqF7jjlMktr6/R0UHM\nOWdaVE/5bH8Rv38viznpsTx0Rap+OHD7q/DBr6C+Wl9o6Dlwwyrw76cXhRRCiEFucHRA1jRqvn0X\n95rn2W3dSeBhDcMf0K9X1eCCYefDpFvB7K+PwC4jWAghRL818IOrught7d0E7HuXCVow9cocDBHR\nJ+dHpsK4BWDsvrEIhRBC9JyBHVxlR+DlOXiqCnjCvRDb+Xdy3+VpfV2VEEKIszAwgitvh36VYFcF\n7FsLRSoMvwSy3sHlrOQa16OkTpzBg5eO7utKhRBCnCXfDi5XJbFbfg+H3zk5LSBSv1z9lqW4TVbm\n1zzAsLHn8sQ16Rh76npaQggrF1t6AAAgAElEQVQheo1vB9fae3AceRem3wupV4HRAlGjwWTm0PEc\nbl72JSHxcTx17bieuwikEEKIXuXbwTXj5xyOn0vyOd9rnqRpGv/JzOG3b+/BzxzKmzdmYLV0cpR3\nIYQQ/Z5vB1d0GnWlJ1sDZp2o5LF397LpYAlTksJ46tpxxIXY+rBAIYQQ3c23g6uFjfuLuOWVrQRZ\nzfz2e2ncOC1RDg8KIcQANCCCq7DKxX0rdzA8MoCVPz6HELtfX5ckhBCih/h8cLkavNzz5g6q6xpY\n8aNpElpCCDHA+XRwHS918vP38zhcVs+T16STEh10+pWEEEL4NJ8Orsffy6KgqoHlN03molFRfV2O\nEEKIXuDTwfXIVaM5PNqPcyW0hBBi0PDp4Ip12Ci3+fRbEEII0UUyJLoQQgifIsElhBDCp0hwCSGE\n8CkSXEIIIXyKBJcQQgifIsElhBDCp5y2LbmiKEbgBWAcUAfcpqrqwRbzfwT8GGgAHlNVda2iKBHA\nCsAG5AG3qKrq7IH6hRBCDDKd2eOaC1hVVT0H+CXwdNMMRVFigJ8B5wGXA39QFMUfeARYoarqDOAb\n9GATQgghzlpngms6sA5AVdXNwKQW86YAm1RVrVNVtQI4CKS3XAd4H5jZbRULIYQY1Doz7EQwUNHi\nuUdRFLOqqg3tzKsCHKdMb5rWRlZWVpcLPpXL5eqW7fQmqbl3+FrNvlYvSM29RWpurTPBVQm0HHbd\n2Bha7c0LAspbTK9tMa0Np7N7Tnt113Z6k9TcO3ytZl+rF6Tm3iI1n9SZ4NoEXAWsVBRlGrC7xbwt\nwO8VRbEC/kAq8G3jOlcArwCzgc9P3WhGRoZcnlgIIUSXGTRN+84FWrQqTAcMwC3ooXRQVdW3G1sV\n3o5+vuxxVVX/oyhKNPAq+t5WMbBQVdWannsbQgghBovTBpcQQgjRn0gHZCGEED5FgksIIYRPkeAS\nQgjhUyS4hBBC+BQJLiGEED5FgksIIYRPkeASQgjhUyS4hBBC+BQJLiGEED5FgksIIYRPkeASQgjh\nUyS4hBBC+JTOXNakR2RmZsrovkIIITrU0eWv+iy4ADIyMs56G1lZWaSmpnZDNb1Hau4dvlazr9UL\nUnNvGYw1Z2ZmdjhPDhUKIYTwKRJcQgghfIpPB9fT61VW7i7v6zKEEEL0Ip8OruLqelbsLKPcWd/X\npQghhOglPh1ci89JpM6j8a+tx/u6FCGEEL3Ep4MrNTaYsdFWXt98FI9XWtcLIcRg4NPBRdZa7og7\nSE5ZLR9lFfR1NUIIIXqBbwfXvrXM2v8bpjgqePHTbDRN9rqEEOJUq1ev5qmnnurrMrqNbwfXxb8G\no5E/25ex83gpnx8o7uuKhBBC9LA+HTnjrDkSKJhwL3FbHuP+gHU893E4M0ZGYDC0O0qIEEL0qf9k\n5rByW9cbkzmdTuwb2+/6c92kIVyTkdCp7Sxfvpx3330Xs9nMpEmTuP/++8nMzOSJJ57AbDZjs9l4\n7rnnKCoq4sEHH8RsNuP1enn66aeJjY3tct09xbeDC6hImkNc9bfcsXcFm44P4fMDIzk/JbKvyxJC\niH7l6NGjfP3117z55puYzWbuuusuNmzYwJYtW5g9ezY33XQTn3zyCZWVlXz55Zekp6dz//33s23b\nNqqqqiS4upXBAFc/D0X7eL7oL9zzTjLn3XstJqPsdQkh+pdrMhI6vXfUUneMVZiVlcWFF16IxWIB\nYNKkSRw4cIA77riDv/71r9x0001ER0eTnp7O/Pnzeemll7jtttsICgri3nvvPavX7m6+fY6riX8g\nxutXYPcz8mjFr1n75Y6+rkgIIfqV1NRUdu3aRUNDA5qmsXXrVoYNG8bbb7/N97//fV5//XVGjhzJ\nypUr+fjjj8nIyODVV19l1qxZLFu2rK/Lb8X397iahA/HsmgVMX+/EtdHt1A95kMCQ+SQoRBCACQm\nJjJx4kQWLFiA1+slIyODmTNnsmvXLn79619js9kwGo383//9H5qm8cADD/Diiy/i9Xp58MEH+7r8\nVgZOcAGGIVPIuWwpwz64lbK/zSbwznUQENHXZQkhRJ+aN29e8+Nbbrml1bxx48axcuXKNuu88cYb\nPV7XmRoYhwpbGHHuXFYMfxKH8yiuly6H8mN9XZIQQohuNOCCC2DetYv5mfnXNFScQFt2KeTJOS8h\nhBgoBmRwOWwWvj/3Or7v+g1V9RosvxwyXwUZWUMIIXzegAwugCvGxjJ2/FRmVv+WyqjJ8M7PYM0d\nUF/T16UJIYQ4CwM2uAAevToNS3A0c8vvoX7GL2HXv+Cli+XQoRBC+LABHVzBVgtPXzeOw2V1PFp5\nJSz+L9SW6+H10aPgru3rEoUQQnTRgA4ugGnJ4dw+I5kVXx9jfe0o+OlmGL8AvngW/jodjn7Z1yUK\nIYToggEfXAD3XZbC2HgH963cyYFKsz5E1KL/gqceXp4Nq34I5XIVZSHE4LZo0SKys7M7nH/xxRdT\nV1fXixW1b0B1QO6Iv9nE0sUZXPXnTfzotW289dPpOIZfBD/ZDJue02/73oPp98C5d4FfQF+XLIQY\niHa8Ad/8o8urDXXWwOYOvpcm3KgfRRpEBsUeF0Csw8Zfb5xIbnktd76xnQaPVw+oix6CO7eCMhs+\n/QM8Nw6+egHcrr4uWQghusWdd97Jli1bANi9ezeLFi3i7rvv5oc//CFXXnklK1as6NL2cnJyWLx4\nMTfccAM33ngj+/btA+DBBx9k4cKFzJs3jw0bNgDw7LPPcv311zN//nyWLl3aLe9nUOxxNZmUFMZj\nc8fwwH928/h7+3jkqtH6jJChcO3LMO1/4JPH4IMH4cs/wYyf679mLLa+LVwIMTCMX3BGe0fHznJ0\n+GuvvZY1a9YwZcoUVq9ezdSpU0lJSeGyyy6joKCARYsWsXDhwk5v78knn2Tx4sXMnDmTrKwsHnro\nIV577TW2bt3aPHxU0/0777zDa6+9RlRUFKtXrz7j99DSoAougB9MHkrWiSqWbzpMfKiNW6cPOzlz\nyBS46W04/Dls+D2897/6XtiU22HyjyAgvO8KF0KIMzRjxgyWLFlCeXk527ZtY9myZTz99NOsX7+e\nwMBAGhoaurS97OxsJk+eDOijzufn5xMYGMhDDz3Eww8/THV1dfP8JUuW8PTTT1NcXMyMGTO65f0M\nmkOFLf16TiqXp0Xzu7V7Wb09p+0Cw2bALe/Dze9CwmQ9vJ5Ng3d/DqWHer9gIYQ4C0ajkVmzZvHo\no48yc+ZMli9fzvjx43nqqaeYNWsWWhdHFRo+fDjbtm0D9Ot8RUREUFhYyJ49e3j++edZunQpr776\nKvX19axbt45nnnmG1157jTVr1pCbm3vW72fQ7XEBmE1Gnrt+Are8vJX7V+0ixG7h4lHRrRcyGCBp\nun4r3Adf/Rm2vwbblsOIS2HSLfq9aVB+hEIIH3PNNdcwc+ZMPvjgA3Jycnjsscd47733CAoKwmQy\nUV9f3+lt/eIXv+Dhhx9m+fLlNDQ08Pvf/57IyEiKioq4/vrrMRqNzJ07Fz8/PxwOB9dddx1Wq5Xz\nzjuPuLi4s34vhq4mbXfJzMzUMjIyzno7Z3Nl0CqXmwUvbeZAQTWv3zqVKcPCTrNCPmxdpgdYdQEE\nxennwCYu0s+T9ULNfUVq7nm+Vi9Izb1lMNacmZlJRkZGu5eyH9S7C0FWC6/cMoXr/voVt7y8heU3\nT2Zq8necxwqKgYt/DRc8APs/gMxXYOMS/Tb8IhgzH1KvBKuj196DEEJ0p127drFkyZI202fPnt2l\nBhw9aVAHF0BEoD9v3D6NhS9t5uaXt/L3myZx7ojTXHzSZNEDKvVK/Xpf21/Xx0F86yew9l4YeSmM\nnQ8jLwc/e++8ESGE6Abp6em8/vrrfV3Gd+qR4FIUxQIsB5IAf+AxVVXf7onX6g7RwVbevP0cbli2\nmVte2cpLiydxfkpk51YOGQoX/0rvD5abCbtXwZ7VsG8t+AWCcgWMmQfDzpeOzUII0Q16qlXhjUCJ\nqqozgFnAX3rodbpNZJA/b/xoGsMiArjttW2s+za/axswGCBhEsz+I9yXBYvfhjHXwIH18Mb18MQw\neP37sPlFLFVyVWYhhDhTPRVc/wYebnxsALrWSaCPhAfq4TU6Npif/DOT1zcfPbMNGU2QfAF870/w\nvwdg0RqYfBtU5MC6XzLivevgufHw3v2wfz3UVXfvGxFCiAGsR1sVKooSBLwNvKSqaqsxRTIzMzW7\n/ezP/7hcLqxW61lvp9U2G7z84bNCtuQ4uX5sCIsnhGIwtNu4pcss1Xn4H99ISNFWAgq3YfTUoRlM\n1IaNxhmdQU1UBrXhY9HM3fuezlZPfM49zddq9rV6QWruLYOxZqfT2WGrwh4LLkVRhgBrgBdUVV1+\n6vz+0Bz+uzR4vPz6v9/y5tbjXDMxgcfnjcHfbOqWbTfX7K6FY1/pI3Uc3gh534DmAZOf3vE5aQYM\nnQrxGX3eUnEwNsftbb5WL0jNvWUw1tzrzeEVRYkG1gN3qqr6cU+8Rk8zm4z8Yd5YYh02nv1oP0dK\navjrjRlEBvl334tYbDD8Yv0G4KqEY5vhyEY9yD57Amj8YRGh6GGWkKHfR6ZK52chxKDUU998DwGh\nwMOKojSd65qtqqpPXXLYYDBw98yRjIgK5Of/3sH3/vIFLy2exJj4Htr7sQZDymX6DcBVAbnbIWcb\n5G6D/e/DjsZLIlgCIGYMRI9pvB8L0aOl5aIQYsDrkeBSVfVu4O6e2HZfmJMeS2K4ndtf28b8v37J\nH+elM3dCfM+/sNWhd2wefpH+XNOg7MjJIDuxC3b/G7b9vXEFA4QlnwyymDEQMxaC4/VWj0IIMQDI\nsaZOGhPv4K07p/PTFdu551872HKklEeuHI3V0j3nvTrFYICwYfot/Vp9mqbpnaALvoX8byF/F5zY\nCXvfOrmeNQQiFQgfCRFNtxQITdI7UwshhA+R4OqCyCB/Vtw2lafW7+evn2Wz83g5L9wwkcTwPjw8\nZzBAaKJ+GzXn5HRXJRTuhfzdULAHig/ofcp2tLj6qtGsh1dECoSP0O+bQs1+mnEbhRCij0hwdZHZ\nZOSXs0cxOSmU+1bu5Mo/fcFvr07j+xPiu63JfLewBsPQafqtpdpyKDmoB1nxfig5oD8++BF4WowO\nbQ3RwzBkKIQkElrnD8YpEByrDy5sDwfjoLwqjhCij0lwnaFLUqNZe9d07lu5g/tW7uSjrAJ+P3cs\noQF+fV3ad7OF6CN8JExqPd3TAOVHT4ZaabZ+CLJIhQMfEtPggu0tljdaICi2MchiITiuxX3Mycdy\n9WghRDeT4DoLQ8LsvHn7OSzdeIhnPlTZdqSMJ+enc6ES1deldZ3JDOHD9VvK5a3naRr7v9lESqQ/\nVOVB5Qn9viofKvP082sHPgR3TdvtWkNahFosBESCPQICIhrvw/V7e7gMSCyE6BQJrrNkMhr4nwuH\nc35KBPf9ayc3v7yVG6cN5aErUrH7DZCP12DAYwuHId/RmVDToK6yMdQab5V5jfeNzwv3Qk0ReDsY\nAcxibx1mAY2BFhABtjC9laXVoe81Wh16KPoHS382IQYZ+YvvJmlxDt668zyeXq+y7IvDbNhXxO/m\nprW9svJAZTCcDJaoUR0vp2l6/zRnCdQUg7O48b7klGlFULRPf95wmu5/foEng6ypBquD6FoPnBjW\nIuha3PyD9L5wfgF6YMr5OiF8hgRXN7JaTPxqzmguT4vhwdW7+eEr27hibAy/uSqN6GDfGmesxxgM\nepDYQvTDkp1RXwO1ZXrguSr0BiZNj5tvLaZV5kDhHhw1pXCgkwMYN4WYX4AehH721s8tdv18ndkK\nFiuYbae/N/vrw3eZ/fVuB6bG5xKSg4PXqw/h5vWcct/16baibLCV6kcrznAb+n0DeNwt7t2nPG/4\njnlu/Vx4h9Mb58VPgoVv9uhHK8HVAyYlhfHuz2bw0ueHeO7jA3y+v5hfzFK4YWoiRmM/annoK5oC\nxJHQpdX2Z2WRqqRAXVXbgKur1s/J1Xd0q9bXq8rXH9fXgNul7/1p3rN7PwZT2zAzWUj2AJ8FtpqG\n2V/vtmA06fcGU4vnptbPDcbGm6HF4/ZuBn299uY1azGGaavxTFtPDyssgLLo1tNPs84ZT9e0bggC\nLwmV5bDdrj/3Nuj/nm3W9XQwvbOv1b0XxEjq1q21YLTo/8+MFv3/UNNjk7nFPHOL6ZaT/3dbLmc0\nn3wcN76nqm0mwdVD/MxGfnrRCOaMjeVX/93Nw2/tYVVmDo9clYY0QehFRtPJPbzuoGn6r8yG2pNB\n1tF9Q53excDjbvG4xa2h9fO6shL87f4t5rv0kG3+gm28957yBdn05atpjfcd3HpAnx0IN7QM7qZ7\nYzvTjW2Ws9S7QbO3CPymeZZObrMzr2Xucl3fNf1YTi5DE5Na/Hg5022bWwdVf+rC0wUSXD0sKSKA\nf9w6lbd25PH4e1lc8+KXXDgsgMdjk4gLkabiPsdgALOffuvmEftzs7II7skRwJuCrWlv4nTh1upL\nzdDudHX/AZSUlHaW73idM57etKd4lodaD/vgSOs1DVmQ7Fs19yQJrl5gMBiYOyGeS0dH89fPsvnb\nZ9lc/PSn3H7+cO64IHngtD4U/VvzF3/3DVPmtQTond2F6EXyjdmLAvzN/PwyhYzQOlYdaOBPHx9g\n5dbj3D1zJNdmJGA2yUl7IYQ4Hfmm7APRgRb+snAiq+44h9gQKw+u3s2lz27k7Z15eL09d0VqIYQY\nCCS4+tCkpDBW/8+5vLR4En4mIz974xvm/PkLPtlXQE9dmVoIIXydBFcfMxgMXDo6mvfunsH/+8F4\nauoa+OEr25j7wpd8tFcCTAghTiXB1U+YjHoDjo9/fgGPf38sJdV13PbaNmY/9znv7MzDI4cQhRAC\nkODqdywmIwunDmXD/17I09eOw+3xctcb33DpM5+xcttx3J6e6Y8jhBC+QoKrn7KYjFyTkcD6ey/g\nhRsmYrWY+MWqXVy45FNe2XSYmrru7ZkvhBC+QoKrnzMZDVwxNpZ3fzadl2+eTIzDyqPv7OXcP37C\nE+v2UVDp6usShRCiV0k/Lh9hMBi4aFQUF42KIvNoKcs+P8zfPstm2eeHuGpcHLdNT2Z0nHQEFUIM\nfBJcPigjMYyMxDCOltTw8qYjrNx2nNXbc5mcFMqN0xKZNSYGf3P3jY4ghBD9iQSXD0sMD+DR76Vx\n78wU/rXtGP/8+hh3v7mD8AA/rp00hBumDmVImAzpK4QYWCS4BgCH3cLt5w/ntunJfHGwmH9sPsrS\njdn8bWM2F6REcuPURC5UImVIKSHEgCDBNYAYjQbOT4nk/JRITlTU8saW47y55Ri3vbaNyCB/5k2I\nZ35GAiOjg/q6VCGEOGMSXANUrMPGfZemcNfFI/hkXyGrMnP4+xeH+dvGQ4wbEsL8jAS+lx6Hw27p\n61KFEKJLJLgGOIvJyOVpMVyeFkNxdR3//SaXVZk5PPzfb/nd2r1cOjqa+RkJzBgRIYcShRA+QYJr\nEIkI9Oe2GcncOn0Ye/IqWZWZw1s7cnl31wmigvy5Mj2OOemxTBwagsFHr4wqhBj4JLgGIYPBwJh4\nB2PiHTx0RSqf7Ctk9fYc/vH1UZZvOkx8iI056bFcmR7L2HiHhJgQol+R4Brk/MxGZo2JYdaYGCpd\nbj7aW8DaXSd4edNhlm48RGK4nTljY0kLqmOUpkmICSH6nASXaBZstTBvYgLzJiZQ4XTzwZ583tmV\nx982HsLj1UjeXM7laTFcNjqacQkhGI0SYkKI3ifBJdrlsFu4bvIQrps8hJLqOl75eAfbizRe2niI\nFz/NJirIn5mjo7lsdDTnDA+XkTqEEL1GgkucVnigP1ekBPPzq1OpcLrZoBayfm8+b32Ty4qvjxHo\nb+YCJZLLRkdz0agogq3SxF4I0XMkuESXOOwW5k6IZ+6EeFxuD19ll7B+bz4f7i3k3V0nMBsNZCSG\ncn5KJBekRDI6NlgOKQohupUElzhjVoupecT638/V+OZ4OR9lFfCZWsSSD1SWfKASEejH+SMjuUCJ\nZPqICMID/fu6bCGEj5PgEt3C2LinlZEYygOzRlFY5eLz/cVsPFDEBrWQ1d/kYjDA2HhHc5BNGBIi\nnZ6FEF0mwSV6RFSQlWsyErgmIwGPV+Pb3Ao27i/is/1FvPhZNn/ZcJAgfzPnjYhgRkoE5ySHMywi\nQJrbCyFOq0eDS1GUqcATqqpe2JOvI/o3k9HAuCEhjBsSwl2XjKSi1s2XB4v5bH8RG/cXsW5PPgAx\nwVbOGR7OOcnhnDM8XC7JIoRoV48Fl6IovwAWATU99RrCNzlsFmaPjWX22Fg0TeNwcQ1fHSrhq+wS\nPj9QxJpvcgGID7FxzvBwzh0ezrTkcOJCbH1cuRCiP+jJPa5sYB7weg++hvBxBoOB5MhAkiMDuWFq\nIpqmcaCwmq+yS/gyu5iPsgpYlZkD6EE2OSmUycPCmJIUxoioQDm0KMQgZNA0rcc2rihKEvCmqqrT\nTp2XmZmp2e1nfyjI5XJhtVrPeju9SWruPK+mcbisnm8LXHxb4GJPgYsylweAYH8jaVFW0qKtpEVZ\nGRHuj7lF03tf+5x9rV6QmnvLYKzZ6XSSkZHR7i/TPm2ckZqaetbbyMrK6pbt9CapuWvSgCsbH2ua\nxpESJ1sPl7LlSCnbjpTy1bZSAGwWExOGhjBxaCjpCQ7s3iIm+NDnLP8veofU3DvOtubMzMwO50mr\nQuFTDAYDwyICGBYRwHWThwBQWOli65Eyth4pZeuRUl78LBuPVz+SEPNBIekJDsYNCSE9wUF6fIhc\nPFMIHyfBJXxeVLCVOemxzEmPBaC23sPeExWs37affLc/u3IqWL+3oHn5YREBepglhDBuiIO0OAdW\ni4y1KISv6NHgUlX1CNDm/JYQPcnmZyIjMQy709F8qKLC6WZXbjm7cirYebycrw+V8taOPEBvrq9E\nBzFuiIP0hBDGJYSQEh0onaOF6Kdkj0sMCg67hRkjI5kxMrJ5WkGli53HG8Msp5z3dufzxpbjAFgt\nRtLiHKQnOBg/JIT0hBCSwu3SilGIfkCCSwxa0cFWLkuL4bK0GEBv+HG0xMnOnHJ2Hq9gV045b2w5\nxsubjgAQbDUzNsGBEh3MqJggUmKCSIkOxO4nf0ZC9Cb5ixOikcFgICkigKSIAK4eHw9Ag8fLgcJq\ndh4vZ2dOBXvzKnhjyzFq3Z7GdWBomB0lOohRMUEoMcEoMUEkhdvlUKMQPUSCS4jvYDYZSY0NJjU2\nmOun6NO8Xo1jpU725Veh5lehFlSyL7+Kj7IKaGzMiJ/ZyMioQJToIJQY/TYqJpjoYH853CjEWZLg\nEqKLjMaTe2azxsQ0T3e5PRwsrGZffhX7C6rYl1/FpuxiVjcOYQX6cFd6iAU134+MDpKLbwrRBRJc\nQnQTq8XEmHgHY+IdraaX1dSjFuh7Z/peWiWrt+dSXdfQvEx8iI34QAPjD0NyRADDowIZHhlIWIBf\nb78NIfo9CS4helhogB/TkvWBgptomkZOWW3znpmaX8WuY8W88uUR6hu8J9e1W0iODGR4ZADDI/Uw\nS44MYGiYnEMTg5cElxB9wGAwMCTMzpAwO5ekRgP6EDkpyihyy2rJLq4mu7Ca7KIaDhVV88m+IlZu\ny2le32IykBgewPDIAJIjAxkWHsDQcDuJ4Xaig6wYjXIeTQxcElxC9CMmo4Gh4XaGhtu5SIlqNa+i\n1s2hIj3MsotOBtvHWYU0eE8Olu1vNjIkzE5SuJ2hYQEkNm4vMcxOQqgdP7PsqQnfJsElhI9w2CxM\nGBrKhKGhraa7PV7yyms5WuLkaKmTYyU1HC1xcqzUyaaDJc1N9wGMBoh12Ehs3DtrDrYw/XmQNBIR\nPkCCSwgfZzEZSQwPIDE8oM08TdMoqq7jWImzdbCVOlm/p4CSmvpWy4cF+DWHWGKYvXG7+h5bZKA0\n5Rf9gwSXEAOYwWAgKshKVJCVSUlhbeZXudwcK3XqwVbqbNxTqyHzaBnv7MyjxRFIrBYjCaF2hoTa\nSAi1kxBqw+Csxh1UTkKonVC7RYJN9AoJLiEGsSCrhbQ4fYT8U9U3eMktr+VoSU1zuOWU1ZJT7mT7\nsXIqat36gp8VAhDgZ2oOtPhQG3EhNmIdVuJD9MdRQf7SElJ0CwkuIUS7/MzG5muftafS5eaLzD2Y\nQ2LIKavleJkebMdLnWw7WnYy2BqZjAZigq3EOqzENYZZfIiVWEfTYxvBNrPstYnTkuASQpyRYKuF\nYWH+pKbGtDu/uq6BE+W15JbXcqLCRV7j47zyWnYcL2fdt/nUe7yt1gnwMxEXYiPGYW0OuejGx03T\nwgL8JNwGOQkuIUSPCPQ3MzJaH9KqPV6vRnFNHXnleqjpN/3xiUoX+wuKKKqqa3WeDcDPZCTa4d8Y\nZjZigv2JDtb33GIc+uOoIKs0+x/AJLiEEH3CaDzZcGT8kJB2l2nweCmurudERS0FlS5OVLjIr3RR\nUKE/3p1TzvoKF3UN3jbrhtgtRAb6Exmk3yIaH7urqig0FTXPCwvwwyQdtn2KBJcQot8ym4z6IUKH\ntcNlNE2jotZNfmOwFVS4KKiso7i6jqKqOoqq69hxvJzCyrqTfdq+KGpe32iAsICWAeenP24KvRbh\n57BJy8n+QIJLCOHTDAYDIXY/Qux+jIoJ/s5la+oa2LxjL47ohOZQK26815/Xk11YTVFVXZvzb6AP\ntRXRItAiWoRay726yCB/AvxMEnI9RIJLCDFoBPibiQu2kNpOn7aWNE2j0tWgh1k7AVdcXUd+pYvd\nuRWU1NTjOfVEHHq/t1MDLjzQnzC7hdAAP8IC/Ai16/dhAX5YLaaeetsDjgSXEEKcwmAw4LBZcNgs\njIgK/M5lvV6NMmd9q1BrCrzi6nqKquo4WqJ3EShz1qO1zTgAbBaTHmYBFsICWgSc3Y+6qkoON5xo\nDrrQAAuhdj8sg7RfnPv0a/kAACAASURBVASXEEKcBaPRQHigvjc1qv2eAc08Xv18XGlNPWXOev2+\npp5S5/9v787jo6ruxo9/Zkky2QMJEELYl0NA2VcVS12KoBQ3LOJuBWlFffRnq621Vh/0qQtaylOf\n1gX3XcQNF2ppVUSKBCmI4bCvYcu+zEwy2++POwlDFjJJJpmZ5Pt+vfJi7rnLfOcyM985555zrv/f\nSldt+d6CSoorqymvuW/buoJ6x0u2WUlPjK1NcCfX5IzkZjSjxpAWH0NqQgxx1uiv2UniEkKIdmIx\nm2qbBoNV7fby7X+2kt6rrz/RuSiyV1NUEZD87NUcKXOSd7iMwsrqBntZ1oiPsZCWEFNbozSSmpHc\nUgMep8XHkFKzPiE2oq7ZSeISQogIFms10zXB2mTHk0COak9tcit1uChxVFNidxmP7cbjEoexvLfA\nTomjhGK766SbmNZlMZtIsVlrE16K/6922Wb8OyK7/l3AQ00SlxBCdDDxsRZ6xRrTaDWH0+XxJ7UT\nia7UbjRfljmN5TKH2yh3uDhU7Kgtd3mMi3cDMhJZfdfUNnhVJ0jiEkIIAYAtxkJmquWU4+Ya4vP5\ncLg8lDncJNnaPq1I4hJCCNEqJpOJhFgrCbHtk1I6Z19KIYQQUUsSlxBCiKgiiUsIIURUMfkaG8bd\nxnJzc8PzxEIIIaLC2LFjGxw4FrbEJYQQQrSENBUKIYSIKpK4hBBCRBVJXEIIIaKKJC4hhBBRRRKX\nEEKIqCKJSwghRFSRxCWEECKqSOISQggRVSRxCSGEiCqSuIQQQkQVSVxCCCGiSlB3/VJKTQQe0VpP\nrVM+E/g94AaWaa2fUUrFA68A3YFy4Dqt9fGQRi2EEKLTarLGpZT6NfAsYKtTHgM8CfwE+BEwXynV\nA/gFsEVrPQV4CfhdqIMWQgjReQVT49oFXAq8XKc8B9iptS4GUEqtAc4GzgIe9W/zCXBfQweV25oI\nIYQ4lcZua9Jk4tJaL1dK9WtgVQpQGrBcDqTWKa8pa1BCQkJTT98kp9OJzWZresMIIjG3j2iLOdri\nhdDG7Pb6qKz2Nmsfnw+KHW6KHB6C/SVcXV1NbGxs8wMMgtcHRXY3xU4PQQcUBLfbjdXadD3DB5RX\neSi0e3B5QxeA2+uj0O7B4fKSnmDFbIICu5tqd/3nGNrdxu+m9mj1e8Nutze6LqhrXI0oA5IDlpOB\nkjrlNWUNysnJacXTG/Ly8kJynPYkMbePaIs5GuKtcnuIMZsxm034fD42bvmB7P6DAONLs9Tu4miZ\nk2q3N2AfL4dLnRwudZBfYvx7pNRJtefkbQoqqpDbA7ZeYqyFzFQbcUEkumBZzCYGZsaRGGflSKkT\nr8/HuMx4EmMt9bYdnpVCTk6/Vr+fc3NzG13XmleWBwxWSnUFKjCaCR8H+gIzgPXAdOCrVjyHEKIJ\nbo+XY+VVHCuvIjU+hq4JsRyvcJJf4iS/xIEPyEy14az2cLTMibuBX+IHiuxsOlCC2WyiW1IcJXYX\nJY5quiXHER9j8SceJ0WV1cRYTGQkxVFsr8bp8gJ7g4rTbIIeKTZ6ptrI6ZlCXMyJS+yxFjOZqTbS\n4mMwmRpsHWqQyQRdE2PpkWLDYg5uv71799KvX7+gn6M5TEC35Di6JcdhacbraMq2bdsYOnRoUNta\nzKZmncNo1OzEpZSaCyRprZ9WSt0JfIbRyWOZ1vqQUur/gBf917yqgbkhjViIKON0edh5rAKX5+Rm\nMB9QYq/mSGkVbq+XI0dK+bZk78nb+KCosprDpY7axJGeFEdSnIUj/mRytMxJa1uFEmItjMxOw2yG\nPQWVdEmIpW96IsfLqyisqCYz1cbI3mlkptiwV3s4Xl5Fl4QYfI5Semb2qD1OUpyVzFQb8TEnfonH\nWM1kptjonhyH1RL+ETjxlUfI6dMl3GE0i8VsiohzFymCSlxa673AJP/j1wLKPwQ+rLOtHZgduhCF\naB9Ol4f8Egf2ag8+HxRUVnG8rApPQPtVZZWb/BInDpe73v5VLqNJrMzpqi1ze3zsLqjA5Qk2sxQ2\nWNo9OY6eafF0T46jsLKag8VuMlNsnDEwg6w0Gz1TjXWlDhfFdqOm1DM1nqw0GyaTiSOlDuKsRhNS\nrLX+F2BCjKVFX4xGc9CAZu8nRGuErhFUiDA7VuZk4/4Syp0uqtxe1mw9zrF/FjVZG6l2ezlSZtRm\ngpEQayEprv5HJ8bf3JWZYuNES42JHw/tzojsVOIbuB6QYouhZ6qNOKuZ7Tt2MGTw4HrbJNtiGkw2\nzdErLb5V+wsRSSRxiYhX5nRRUF6F1+djx9EKthwqpSrg4n9BRRW5+4o5WOw4ab+kWDOnZacRa62f\nMALFmE2M6pNGVqpRc0m2GR+LmmsnMQE1kfgYCynx1ja5hpBms5CeFBfy4wrR0UjiEmHn8/nYdKAE\ne7Wntrlrb6GdjfuL2bivGH20/KTeZlazCVvANZRkm5XRfdK4/ox+jO3bhYykOCxmE8X5exg+bFgY\nXpEQoi1J4hLtwuv1UVhZTX6Jg31FdjbuK+ZQiYMeKXFsPljK5oOl9fZJjrMyum8XZpzekz5dEzCZ\noE/XBIZnpQbVdFZ6uGP3rBKis5LEJULO5/Ox5VApKzcfZtOBEg6XOuuN27HFmOndJYH1e4rISIrl\noUtOY2C3JI6WOUmNjyG7SwL9MxKD7uIshOg8JHGJFvF6fRwpO3lQaX6Jk/1Fdr7bX0yx3YXVbGJk\n7zRG9U6j5+k2slLjyUqLp1daPIN7JJ107UgIIYIliUsEzV5tdAX/Znchz321m72FJ0/JkhRnpVda\nPOcP68H4fl05f1gP0hLaZmodIUTnJYlLnKQmOR0udXC4xEl+qYODxQ6+3XWU/aW7aztJjMxO5b9n\nDSe7awJZqfH0TLORYosJb/BCiE5BElcnV+328tWO43y29Qgb9hazu6DypPUmE2QkxdE3xcql4/rT\nNz2Bgd2SOK1XSoefVkYIEZkkcXVSR0qdvLZ+P6+s20dRZTUpNisT+qdzyehe9O6aQM9UG1lp8fRI\nMWZaMGZIqD84Vggh2pskrg6q2u01Zun2eCl3uvn7D0f4dk8xXp+Po+VODhQ5MJngvJweXDmhN2cN\n6tbq2RmEEKI9SOLqQKrdXlZuyee5NXvYml920qBdi9nESP+0Q6f3SuW6yf04Z2h3BnRLCl/AQgjR\nApK4OoDvD5Xyyrp9fPL9EUodLgZ1T+LWcwbTK82GLcZCjMXMhP5dyZDphIQQHYAkrihT7fay5VAp\n3+0vZl+hnbzDZWzYV0xCrIWfDOvBrNG9+NHgbphl4K4QooOSxBVFdhwt5+aXc2t7/hkzTMTz2xlD\nmTOhj3RHF0J0CpK4Ipzb4+WrHQWs31vES2v3Eh9rZcmcUUwemE73ZFu4wxNCiHYniStC+Xw+/rX9\nOA+tzGPnsQosZhNnDEznsctHkpkqCUsI0XlJ4oogPp+Pg8UOvtldyMvf7GPLoVL6pSfw1FVjmKq6\nkRAr/11CCCHfhBHiSKmTha9tZMO+YgAGdEvk4UtO5/Kx2TK+SgghAkjiigDf7Crk1te/w17t5ncX\n5jB5YDo5mSnSM1AIIRogiSuM7NVuFq/aznNr9tA/I5HX5k1kSI/kcIclhBARrcnEpZQyA08BI4Eq\n4Cat9U7/ulHAnwI2nwRcDKwHtgPf+8tXaK2XhDDuqFVR5WZnYRWb1u9nyec7OFLm5JpJffnNjKFy\nDUuI9hY4vUxL9z+UC9s+BHthaGJqQM+SUtie2vwdLbEwYCoMOh9i4hvexlEMeR/AwQ1AK88HQOZI\nmDi/9cc5hWC+KS8GbFrryUqpScBiYBaA1noTMBVAKTUbOKS1/lQpdR7wutb61rYJO/ocKLLz9Je7\neTv3AE6XFzjEiOxUls4dzfh+XcMdnog2Xg/s/wYKdzW+TXwXGHgOmMyw50tI6gZZY4wp/6OZuwp2\nfwHlh1u0e9rhw2D/Nxz+D/zwAdgLWh+TJRYSu7X+OI1IdLmgsAXjNKvKYcOy4LZNyABrCGbXqaqI\niMR1FvApgNZ6nVJqXN0NlFKJwAPA2f6iscBYpdQXwDHgNq11y95lUa7U4eKpf+3k+TV7AbhkdC8G\nJ1Uz6fTBDOsp17GikrMMyg61/jheD+xbC/pjqCqnn8MBaxr5VVxX6QGoONr0dtZ4I3G5/Ler6dIP\nhl8CagbENaNZ2ueDg9/Cto/AXlRb3KyYQ8IHBTuhqrTFR+hZ8yAmAYZcAN1U60JK6wtqOsSnte44\np7AzL4+cnJzm7+hxw96vjP87n7fhbSyxxg+cniOj5keNyddEVVkp9SywXGv9iX95PzBAa+0O2OY2\nIF1rfb9/eRZQqbX+XCl1FXCJ1vrywOPm5ub6EhISWv0CnE4nNltkjWvy+XzsLXHx953lfLq9DIfb\nx/kDk7h2dFcyEq0RGXNTJGbA5yNl/9/J3Pg4luqykB22KqU/roQeeL0ezGZLUPt4YpMp73U2jvTT\njcTUgNiKgyQfWI3J56E8eypW+3FSDnxO4tFvMfk8LYq1Oimb6qTs2uXmxBwq7vgMynqfQ1XqQKD5\nX7RVVU7i4mx4YlPwWaPjPd0ZP392u52xY8c2+B8cTI2rDAj8aWYOTFp+VwGBiWk1UHNf9xXAgw0d\nuEW/IOrIa+kvkTZyoMjOzS/n8sPhMqxmEzNHZjFvygCGZaXUbhNpMQcjqmMu2AlHNjdvZ48Ldq2G\nnZ8bTVMA+KC6ArLHw8QFEIov7G45xHUfShzNP8fBXfG4CoAutcu/gspC2L8WvHU/xk3oOoDYzBHE\nBvwqD9f7ojV1m7y8PAZH63s5irQ25tzc3EbXBZO4vgZmAm/5r3FtCVyplEoF4rTWBwKKnwWWA28B\n5wKNR9CB5Jc4uPKZdZQ73fz3rOFMP72nzMjeVnw+o6ms5su39iL5R1BpXLPoXVkJ/6yEY1tb9hy2\nVBgyHRLST5SlD4Sx14cmaYVLYjrkzAx3FEK0WDCJawVwvlJqLUa9/Aal1J3ATq31B8AQYG+dfe4B\nlimlfglUAjeFLuTIlHe4jHkvbaDU7uLVeRMZkd127d2dUlk+bFsJlcehuhK2fwqFO+tvl9gNuvQH\nwOx2QEoGTHsY+p8N5mZc3DaZjONYY0P0AoQQodJk4tJae4EFdYq3Baz/FqPnYeA+e4AfhyLAaPDB\nf/K5+53NpMRbeeUmSVot5nEZXXKrK0+UFe+B7981etDVdNU1WaDvGTDu5xAXcCPMtL7Q90ywGG/r\nfVHYvCKEaJoMHGoFt8fLo59pnv5yN+P6duGpq8d0zBnbnaWk7lkJPywGT3X99RnK6KmW2iu449kL\nIe9DOLSR2mRU0727obEw3XLgx7+F4ZdCxqAWvwwhRMcgiasVFq3M44W1e7lmUl/uu2hYx5hT0OOC\nPV/AsW1G99n938DOz8nyVENSZv0uv16PkYS+fLT5z9Wl/8njRgZMhWEXQ3LPE2XxXSRZCSFOIomr\nhdbuLOCFtXu5/ox+/OGnw8MdTssV7TGuGzlLjY4NP3wAjhPjdEjuCeNvYk/SGPqfeXnD4zzKj8KO\nz4zBjsGwxhnjRroOCM1rEEJ0KpK4WmB/oZ1fvbOZ/hmJ3H3B0HCH0zIuJ6z+b/jmL9Q218UkGgMp\nT7vUuIZkskBsEpjNOPPyGh+cmNwDxlzbbqELITo3SVzN4HR5uGf5Zj74Tz5Wi5nX500kPjaKukXX\ndBn/fjlsXWFMmTP2Bsi5yBg932scxLZ+ULgQQrQlSVxB8vl83Lvie97blM/8swdw45n9I/tOxM4y\nWHknFO89UVZ+xJgqyBJrTLo5YR4M7DSdP4UQHYQkriC9vG4fyzce5PZzB3PH+UPCHc6pVVXAq7Ph\n0AboN+VEE19yT/jxvTB0hjG4VgghopAkriDsPl7Boo/yOGdod24/d3C4wznB6zWmJaoqM2aR2Poe\nHN0KXpfRO/DyZTD84qaPI4QQUUQSVxNqmgjjYsz88bLTwzebu9frv42DvyOFoxg+uRv2fX1im+7D\nYNRcMFth8HlGzz0hhOhgJHE14Z3cg3yzu5CHLjktfIOLj2+H9xYYHSsCxaXAT5dC9gSjU0Van/DE\nJ4QQ7UgS1ynoI+Xc/8FWJvTrypXjw5QUdq2G16807h30k0UB16ZMRo0q2NkqhBCig5DE1YijZU7m\nv7yBxDgrS+eODk8TYfkRWD7PGKh7zXvGeCkhhOjkJHHVsaegkt+//z1f7yzAYjbxxvxJ9EgJQxNh\nyX5475fGhLOXPy9JSwgh/CRxBdhysJTrn1+P1+dj4Y8HcfHoXgzoltT0jq21YRmsf+bErbU9Lija\nZTye9RR0j9LZOYQQog1I4vLbuL+Ya59bT2p8DC//fEL7JCwwEtbHd0HWGEjrfaJ89FXGjOsyn58Q\nQpxEEhew+WAJ1z23noykWN6YP7ntZsQoPQivXEbO8W0nl6sZMPtFuWmhEEIEodMnrqLKam584VvS\nEmN4bd6k0Cctd5Ux7VJ1Bbzzc7AXUTDsBjK6+a9ZxXeBcTdI0hJCiCB1+sR133vfU+pw8cpNE8lK\niw/NQT0u2P0FbH0X8j6CqlKjPDYZrn2P4+WJZMideYUQokU6deL6aHM+K7cc5lfTFEMzU0Jz0K3v\nwSe/NqZgikuBoRfCgB+DJQayRhnXrPLyQvNcQgjRCXXaxFVR5eaBD39gRHYqN5/dwg4Q+9bCmifB\n6zaWqyrg4HroOQouehIGngsxETyDvBBCRKFOm7iWrt7B8fIqnrl2HFaLufkHKMuHN6825gUMnGrp\nnN/Bmf9l1LCEEEKEXJOJSyllBp4CRgJVwE1a650B65cAZwE1922fBcQArwHxQD5wg9baHtrQW273\n8QqWrdnD7LHZjOqd1vwDeNzw7nzjLsI3fwEZETRjvBBCdHDBVDUuBmxa68nAPcDiOuvHAtO01lP9\nf6XA74HXtNZTgO+Am0MZdGt4vT7ueXcLthgLv7pANf8Ahbvg+emw9yuY8ZgkLSGEaGfBJK6zgE8B\ntNbrgHE1K/y1scHA00qpr5VSN9bdB/gEOC9kEbfS82v3sn5PEffPHB78bO/OMvjPG/DqFfCXiVCg\n4dJnjUHCQggh2lUw17hSgNKAZY9Syqq1dgOJwFLgCcAC/FMptaHOPuVAg7fbzQtB7zqn0xn0cfLL\nXDzyyUEmZicwLL6s0f1iS/dgdlUSaz9M8v7PSTr8DWZvNa6EHpQNnk3RkDm4Y7q3uHdgc2KOFBJz\n24u2eEFibi8S88mCSVxlQHLAstmftADswJKa61dKqdUY18Jq9nH4/y1p6MA5IRjLlJeXF/Rxnn5z\nE2aziT9fO7nxiXP//TR8+qsTy0mZMP5GGH4pMdnjSTebSW/HmCOFxNz2oi1ekJjbS2eMOTc3t9F1\nwSSur4GZwFtKqUnAloB1Q4A3lVKjMZodzwJe9O8zA3gBmA581ZLAQ2nH0XLe23SI+WcPaDxpHf4P\nrLoXBp0PExcY977qNQbMlvYNVgghRKOCSVwrgPOVUmsBE3CDUupOYKfW+gOl1MvAOsAFvKS13qqU\nWgS8qJSaBxQAc9so/qA9+fl2EmOtLDh7YP2VlQWwdQV8/WdIyIBL/gaJra1XCSGEaAtNJi6ttRdY\nUKd4W8D6x4DH6uxzFLggFAGGwlvfHuDjLUe4/dzBdEkMmBPQ54NNr8Knv4GqMuiWAz9dKklLCCEi\nWIcfgPzF9uP8ZsUWpgzOYOE5g06sKD8KH94O2z+BvmfC9Ech87TwBSqEECIoHTpxHSiys/C1jQzu\nnsRTV40hpmaGjP3r4PUrjbsLT3sYJv4CzC2YPUMIIUS767CJy+Xxcvsb34EPnr5mHMk2/xRMlQXw\n1nUQnwY3fgrdWjAIWQghRNh02MT1l3/uZOP+EpZeOZo+6QlGodcL7/0CHMVw1duStIQQIgp1yMR1\nuNTB//1rFzNHZjFzZJZR6PPBp/fAjlUw/THoOSK8QQohhGiRDnlh54lV2/H54NfT/DUqrwc+uxfW\n/w0m3QIT5oU3QCGEEC3W4Wpc246U8c7Gg9x0Vn96d00wJsV975dwYB1MuBmmPQQmU7jDFEII0UId\nLnEt+XwHSbFWbpk6EL59FlbdB+YYuORpGHGFJC0hhIhyHSdx6U/xvHszV9j7cn2f0aS98Hs4vs24\nC/FPl0Jqr3BHKIQQIgSiO3G5HMSW7YGt2+Dd+Ry3ZDLEnE/Woc3QZzJc/FcYOUdqWUII0YFEd+J6\ndz4D8z4AwJlxGhccuo2fTTmd35zbG+KSm9hZCCFENIruxDXtIQ6mjadXdh9uXZuCJ87F/LMHQlxc\nuCMTQgjRRqK7O3xaH8r7nM/npsn8fZeDO88fQnqSJC0hhOjIorvGBXi8PhZ9/ANDeiRx9aS+4Q5H\nCCFEG4vuGhew/qCdfYV27jxfnZhEVwghRIcV9d/0n2wvo0dKHOfldA93KEIIIdpBVCeuQyUONhxy\n8LNxvbFKbUsIITqFqP62f3P9fgCuGN87zJEIIYRoL1GduLYdKeeMvolkd0kIdyhCCCHaSVT3Klw6\ndzR6mw53GEIIIdpRVNe44qwWYiwynZMQQnQmTda4lFJm4ClgJFAF3KS13hmw/g5gjn/xY631A0op\nE3AQ2OEv/0Zr/ZuQRi6EEKJTCqap8GLAprWerJSaBCwGZgEopQYAVwETAS+wRim1ArADG7XWM9sm\nbCGEEIGqqqr44IMPmD17dquPtXPnTu677z58Ph/9+vVj0aJFWK2Rc2XJ5PP5TrmBUuoJYL3W+g3/\n8iGtdS//4xggVWtd4F9eD1wNjAbuBkoBB3CH1vqki1G5ubm+hITWd6pwOp3YbLZWH6c9ScztI9pi\njrZ4QWJuL8HEfPToURYvXsyjjz7a6ud7+OGHmTVrFsOHD2fJkiVMnDiRSZMmNesYrT3PdrudsWPH\nNngtKJgUmoKRgGp4lFJWrbVba+0CCvxNg48B32mttyulMoH/0Vq/rZQ6C3gFGF/3wDk5Oc1+MXXl\n5eWF5DjtSWJuH9EWc7TFCxJzQ5bnHuStDQdCesyzelm49aLRp9zm1Vdf5dChQ/zjH/9gy5YtVFRU\n4PF4uP3225k8eTIzZsxg3Lhx7Nixg9TUVJ544gkaqzy88MILWCwWqqurcblcDB06tNnnrLXnOTc3\nt9F1wXTOKAMC7xFi1lq7axaUUjbgVf82v/QXbwDeB9BarwGy/MlNCCFEG1iwYAGDBg2isrKSM844\ng1dffZUlS5Zw77334vP5cDqdzJw5k9dff50BAwbw5ptvNnosi8XCoUOHuOiiiyguLmbo0KHt+Eqa\nFkyN62tgJvCW/xrXlpoV/mT0PrBaa/1IwD73A4XAo0qpkcABrfWp2ySFEKIDuGxsNpeNzQ7pMfPy\n8oLedteuXcycaXQv6NGjB0lJSRQWFmK1Whk/3mj4GjNmDF9++eUpj9OrVy9WrVrF22+/zR//+Ece\neeSRU27fnoJJXCuA85VSawETcINS6k5gJ2ABfgTEKaWm+7f/DfBH4BWl1IWAG7g+1IELIYQ4wWw2\n4/V6GThwIBs2bGDYsGEcPXqUsrIy0tLScLvdbNu2jaFDh5Kbm8ugQYMaPdaCBQu455576NevH4mJ\niZjNkTVyqsnEpbX2AgvqFG8LeNzY1bcLWxqUEEKI5klPT8flclFeXs6+ffv47LPPcDqdPPjgg7U9\nAp955hny8/PJysrijjvuaPRY8+fP55577iEmJob4+HgWLVrUXi8jKJHTv1EIIUSLxcXF8f77759y\nm4cffpi4IO4QP2bMGN54441QhRZykriEEKITys/P5+67765XPn78eG677bYwRBQ8SVxCCNEJrF69\n+qTlrKwsXn755TBF0zqRdcVNCCGEaIIkLiGEEFFFEpcQQoioIolLCCFEVJHEJYQQncQ111zDrl27\nGl3/7bffsm3btkbXN9eTTz7J7NmzueKKK/j3v/8dsuNK4hJCCAHA8uXLOXbsWEiOtXv3bjZt2sRb\nb73FE088wUMPPRSS44J0hxdCiNDa9Dp890pID5na4xw4xUzrCxcu5Nprr2XChAls2bKFRx99lK5d\nu1JeXs6xY8eYO3cuc+fOPeVzfP/993z11Vds3bqVQYMGsWHDBl588UViY2Pp168fDz74IB9++CGf\nf/45lZWVFBcXc8sttzBt2rQGjzdgwACee+45TCYT+fn5pKSktOocBJLEJYQQUW727NmsWLGCCRMm\n8O677zJx4kSGDBnCT37yE44ePco111zTZOI67bTTmDJlCjNmzCA+Pp6lS5eyYsUKkpKSePjhh3nz\nzTdJSEjA4XDw/PPPU1RUxOzZszn33HMbvcmk1WrlySef5KWXXuK+++4L2euVxCWEEKE06krjL4RK\n8/LIOsX6KVOm8Nhjj1FSUsKGDRt49tlnWbx4MatWrSIpKQm3232Kves7cOAAgwYNIikpCTBm01iz\nZg0jR45k/PjxmM1mMjIySElJoaioiO7duzd6rDvuuIN58+bxs5/9jHHjxtGnT59mxdIQucYlhBBR\nzmw2c8EFF/CHP/yB8847j2XLljFq1Cgef/xxLrjgApq6030Nk8mEz+cjOzubXbt2YbfbAVi/fj39\n+/cHYOvWrQAUFBRQUVFBenp6g8favHkzDzzwAGDMo2i1WjGZQnNbRqlxCSFEB3DZZZdx3nnn8dln\nn3Hw4EEWLVrExx9/THJycu3djJsycuRIHn/8cf70pz9x6623cu2112I2m+nTpw933XUXK1eupKCg\ngOuuu47y8nLuv/9+LBZLg8caPnw4P/zwA3PmzMHr9XLVVVfRu3fvkLxWSVxCCNEB9OzZs7Y2lJ2d\nzUcffVRvm6bmJpwzZw5z5swBYODAgbU3pAw0fvx47rrrribjsVgstTWuUJPEJYQQncjmzZt57LHH\n6pVPnz69yQ4cDfnf//3fBsdo/fznPyfnFD0hW0MSlxBCdCIjRoxo8azwl156ab2yhQsXsnDhwnrl\neXl5LXqOYEjn1qyyvQAAB+1JREFUDCGEEFFFEpcQQoioIolLCCFEVGnyGpdSygw8BYwEqoCbtNY7\nA9bPA24G3MAirfVHSqkM4DUgHsgHbtBa29sgfiGEEJ1MMDWuiwGb1noycA+wuGaFUioTuA04E5gG\n/I9SKg74PfCa1noK8B1GYhNCCCFaLZjEdRbwKYDWeh0wLmDdBOBrrXWV1roU2AmMCNwH+AQ4L2QR\nCyGE6NSC6Q6fApQGLHuUUlattbuBdeVAap3ymrJ6QtFd0ul0tmm3y7YgMbePaIs52uIFibm9SMwn\nCyZxlQHJActmf9JqaF0yUBJQ7ggoq6dmHqzWCtVx2pPE3D6iLeZoixck5vYiMZ8QTOL6GpgJvKWU\nmgRsCVi3HnhIKWUD4oAc4Hv/PjOAF4DpwFd1Dzp27NjQzLYohBCiUzE1NWtwQK/CEYAJuAEjKe3U\nWn/g71U4H+N62cNa6+VKqR7Aixi1rQJgrta6su1ehhBCiM6iycQlhBBCRJKonauwqfFlkUIpFQMs\nA/phNKcuAg4AHwE7/Jv9n9b6zbAE2Ail1EaMa5UAe4C/AUswxuut0lq3zbTPLaSUuh643r9oA0YB\nVwKPY5xvgPu11l+0e3B1KKUmAo9oracqpQZhNKn7MJrZb9Fae5VS9wMXYpzv/9Jarw9bwNSLeRSw\nFPBgfPau1VofVUotwehRXO7fbZa/t3FY1Il5NA185iL8PL8BZPpX9QPWaa3nKKXeBzIAF+DQWk8P\nU6wNfbf9QDu8n6M2cREwvsx/7W0xMCvMMTXkaqBQa32NUqorsAl4EHhCa7341LuGh/+apUlrPTWg\nbBNwGbAbWKmUGq21/i5MIdajtX4B4wODUuovGB+oscCvtdbLwxfZyZRSvwauAWqazp8Afqe1/pdS\n6q/ALKXUPuBHwESgN7AcGB+OeKHBmJcAt2qtNymlbgbuBu7EON/TtNYF4Yn0hAZiHkudz5xSagwR\nfJ611nP85V2AfwJ3+DcdDAzXWoe7uayh77ZNtMP7OZqnfDrV+LJI8jZwn/+xCeMXx1jgQqXUl0qp\n55RSyY3uHR4jgQSl1Cql1Gql1NlAnNZ6l//D8hkROjZPKTUO40P9NMZ5vlEp9ZVSarFSKhJ+qO0C\nAqfYHgvU1AJrxjyehVGr9Wmt9wNWpVS39g3zJHVjnqO13uR/bAWc/haQwcDTSqmvlVI3tneQdTR0\nnut+5iL9PNd4AFiqtT7s7z+QBnyolFqjlLqoXSM8WWPfbW3+fo7mxNXg+LJwBdMYrXWF1rrc/0F5\nB/gdRm/MX2mtz8aowdwfzhgbYMdoYpsGLACe95fVaHRsXgT4LcYHHeDvwK3A2UASxmsJK3/tzxVQ\nZAr45dzQOMjA8rCoG7PW+jCAUuoMYCHwJJCI0Xx4NXAB8Eul1Ij2j7Y2xrrnuaHPXESfZwClVHfg\nXPytCUAsRuvSxRhJ7kn/Nu2uke+2dnk/R3PiOtX4soiilOqNUdV/WWv9GrBCa53rX70CGB224Bq2\nHXjF/wtpO8abrmvA+kbH5oWTUioNUFrrf/qLlmmtd/s/SO8TeecZwBvwuO44yLrlEUMp9TPgr8CF\nWuvjGD9slmit7VrrcmA1Rs09UjT0mYv48wxcjjF9nse/fAT4q9barbU+hjGlngpXcA18t7XL+zma\nE1fNWDEaGF8WMfxV+1XA3VrrZf7iz5RSE/yPzwVyG9w5fG7EPyelUioLSAAqlVIDlVImjJpYvbF5\nEeBs4B8A/jg3K6Wy/esi8TwDfKeUmup/XDPm8WtgmlLKrJTqg/GjLOzXjWoopa7GqGlN1Vrv9hcP\nAb5WSln8F+3PAjaGK8YGNPSZi+jz7HceRpNb4PLbAEqpJOA0ICxTajTy3dYu7+eIa1prhhXA+Uqp\ntZwYXxaJfgt0Ae5TStW0B9+JUcV3YfyCmh+u4BrxHPCCUmoNRu+gGzF+Sb0KWDDaq+vfqzv8FEYz\nEFprn1LqJuBdpZQDo7fTM+EMrhH/D3hGKRWL8QX0jtbao5T6CvgG48flLeEMMJBSygL8GdiPcW4B\nvtBa36+UehlYh9Hc9ZLWemv4Iq3nF8DSwM+c1rosUs9zgNr3NIDW+hOl1DSl1DqMz+Rvw5hsG/pu\nux34c1u/n2UclxBCiKgSzU2FQgghOiFJXEIIIaKKJC4hhBBRRRKXEEKIqCKJSwghRFSRxCVEFFBK\n/UspNTTccQgRCSRxCSGEiCoyjkuIEPPPHPFXjElnzRhzuD2FMYvAcKAI45Yr1RjzQA7AGNj9hP9W\nGxOBP/n3PQRchTF7wmGgB8a8gFcGzFohRKciNS4hQu8moMA/oess4C8Y02a9qrU+C9gG3Oz/O661\nPgNjKp9FSqkMjHuf3ai1ngisBHL8x12ptT4HI4ld3p4vSIhIEs1TPgkRqU4HpvhrTmB8znxa6y/9\ny2sx5nFzA58D+GfZ/gEYCGRqrfP85c8B+KdWqplr8QgnbjAoRKcjNS4hQm8b8Lr/RpzTMSZFtSil\namZLPxPYijGX2xQA/60hTse423S+Umqwv/xupdQl/v2kXV8IJHEJ0Rb+BgxVSn2BUbvahzEh6t3+\niYt7+bd5Gkj3l/0LeMB/q4qbgWX+/UcDH7f/SxAicknnDCHagVJqLzBUa+0McyhCRD2pcQkhhIgq\nUuMSQggRVaTGJYQQIqpI4hJCCBFVJHEJIYSIKpK4hBBCRBVJXEIIIaKKJC4hhBBR5f8DHGcnoGHq\ny/cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x130f51588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Skipgrams: Saving the Vectors\n",
    "- Let's save the vectors to a file\n",
    "- So we can load them into word2vec and test them out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 302,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:44:50.795454Z",
     "start_time": "2017-11-08T12:44:50.324260Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Write the resulting vectors to a text file\n",
    "f = open('vectors.txt' ,'w')\n",
    "f.write(\" \".join([str(V-1),str(dim)]))\n",
    "f.write(\"\\n\")\n",
    "vectors = skipgram.get_weights()[0]\n",
    "for word, i in tokenizer.word_index.items():\n",
    "    f.write(word)\n",
    "    f.write(\" \")\n",
    "    f.write(\" \".join(map(str, list(vectors[i,:]))))\n",
    "    f.write(\"\\n\")\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Skipgrams: Examining the Vectors\n",
    "- Let's load the vectors in to query them with `gensim`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 303,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:44:52.591645Z",
     "start_time": "2017-11-08T12:44:51.969023Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-11-08 07:44:51,971 : INFO : loading projection weights from ./vectors.txt\n",
      "2017-11-08 07:44:52,588 : INFO : loaded (3387, 100) matrix from ./vectors.txt\n"
     ]
    }
   ],
   "source": [
    "# Load the vectors into word2vec and see how we did!\n",
    "import gensim\n",
    "w2v = gensim.models.KeyedVectors.load_word2vec_format('./vectors.txt', binary=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 304,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:44:53.664721Z",
     "start_time": "2017-11-08T12:44:53.651831Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-11-08 07:44:53,653 : INFO : precomputing L2-norms of word weight vectors\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('look', 0.9606662392616272),\n",
       " ('about', 0.9577377438545227),\n",
       " ('off', 0.9559640288352966),\n",
       " ('more', 0.9557409286499023),\n",
       " ('before', 0.9552998542785645),\n",
       " ('after', 0.954471230506897),\n",
       " ('well', 0.95406574010849),\n",
       " ('made', 0.9537373781204224),\n",
       " ('time', 0.9530203342437744),\n",
       " ('quite', 0.952734112739563)]"
      ]
     },
     "execution_count": 304,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.most_similar(positive=['head'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Recurrent Neural Networks (RNNs)\n",
    "- So far, stuck to simple ANNs (fully connected)\n",
    "- RNNs change the game\n",
    "- How do they do it?\n",
    "\n",
    "- Connections between units form a **directed cycle**\n",
    "- Allows network to retain internal state from past units $\\rightarrow$ **memory**\n",
    "- Allows dynamic temporal behavior\n",
    "- Can use memory to process arbitrary input sequences!\n",
    "- Terrific success in various NLP tasks\n",
    "\n",
    "- Hidden Units at a time step t are dependent on:\n",
    "  - The previous hidden unit\n",
    "  - The input at time step t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "<img  src=\"img/rnn3.jpg\" style=\"width:50%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Recurrent Neural Networks (RNNs)\n",
    "- Drawback of ANNs: Fixed # of inputs and outputs\n",
    "- RNNs can map arbitrary length sequences of each!\n",
    "<img  src=\"img/rnn2.jpeg\" style=\"width:50%\"/>\n",
    "- Don't stress!  Just a different architecture with some nice features!  Same solving concepts apply!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Recurrent Neural Networks (RNNs)\n",
    "- One more view!\n",
    "<img  src=\"img/rnn4.png\" style=\"width:50%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RNN Example: Text Classification\n",
    "- Let's try an RNN for the same Reuters classification task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 318,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:57:29.417579Z",
     "start_time": "2017-11-08T12:57:28.346117Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Same data loading as before\n",
    "max_features = 4000\n",
    "(X_train, y_train), (X_test, y_test) = keras.datasets.reuters.load_data(\n",
    "    num_words=max_features)\n",
    "maxlen = 20\n",
    "X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=maxlen)\n",
    "X_test = keras.preprocessing.sequence.pad_sequences(X_test, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 319,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:57:30.486543Z",
     "start_time": "2017-11-08T12:57:30.019039Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"296pt\" viewBox=\"0.00 0.00 344.89 296.00\" width=\"345pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 292)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-292 340.8867,-292 340.8867,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 5125272128 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>5125272128</title>\n",
       "<polygon fill=\"none\" points=\"0,-243.5 0,-287.5 336.8867,-287.5 336.8867,-243.5 0,-243.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"102.2847\" y=\"-261.3\">embedding_20_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"204.5693,-243.5 204.5693,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4038\" y=\"-272.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"204.5693,-265.5 260.2383,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4038\" y=\"-250.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"260.2383,-243.5 260.2383,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"298.5625\" y=\"-272.3\">(None, 20)</text>\n",
       "<polyline fill=\"none\" points=\"260.2383,-265.5 336.8867,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"298.5625\" y=\"-250.3\">(None, 20)</text>\n",
       "</g>\n",
       "<!-- 5125328568 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>5125328568</title>\n",
       "<polygon fill=\"none\" points=\"2.7139,-162.5 2.7139,-206.5 334.1729,-206.5 334.1729,-162.5 2.7139,-162.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"88.2847\" y=\"-180.3\">embedding_20: Embedding</text>\n",
       "<polyline fill=\"none\" points=\"173.8555,-162.5 173.8555,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"201.6899\" y=\"-191.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"173.8555,-184.5 229.5244,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"201.6899\" y=\"-169.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"229.5244,-162.5 229.5244,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"281.8486\" y=\"-191.3\">(None, 20)</text>\n",
       "<polyline fill=\"none\" points=\"229.5244,-184.5 334.1729,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"281.8486\" y=\"-169.3\">(None, 20, 100)</text>\n",
       "</g>\n",
       "<!-- 5125272128&#45;&gt;5125328568 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>5125272128-&gt;5125328568</title>\n",
       "<path d=\"M168.4434,-243.3664C168.4434,-235.1516 168.4434,-225.6579 168.4434,-216.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"171.9435,-216.6068 168.4434,-206.6068 164.9435,-216.6069 171.9435,-216.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 5125325264 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>5125325264</title>\n",
       "<polygon fill=\"none\" points=\"3.4795,-81.5 3.4795,-125.5 333.4072,-125.5 333.4072,-81.5 3.4795,-81.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"88.2847\" y=\"-99.3\">simple_rnn_8: SimpleRNN</text>\n",
       "<polyline fill=\"none\" points=\"173.0898,-81.5 173.0898,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"200.9243\" y=\"-110.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"173.0898,-103.5 228.7588,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"200.9243\" y=\"-88.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"228.7588,-81.5 228.7588,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"281.083\" y=\"-110.3\">(None, 20, 100)</text>\n",
       "<polyline fill=\"none\" points=\"228.7588,-103.5 333.4072,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"281.083\" y=\"-88.3\">(None, 40)</text>\n",
       "</g>\n",
       "<!-- 5125328568&#45;&gt;5125325264 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>5125328568-&gt;5125325264</title>\n",
       "<path d=\"M168.4434,-162.3664C168.4434,-154.1516 168.4434,-144.6579 168.4434,-135.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"171.9435,-135.6068 168.4434,-125.6068 164.9435,-135.6069 171.9435,-135.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 5125328008 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>5125328008</title>\n",
       "<polygon fill=\"none\" points=\"46.6587,-.5 46.6587,-44.5 290.228,-44.5 290.228,-.5 46.6587,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"102.2847\" y=\"-18.3\">dense_63: Dense</text>\n",
       "<polyline fill=\"none\" points=\"157.9106,-.5 157.9106,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"185.7451\" y=\"-29.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"157.9106,-22.5 213.5796,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"185.7451\" y=\"-7.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"213.5796,-.5 213.5796,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"251.9038\" y=\"-29.3\">(None, 40)</text>\n",
       "<polyline fill=\"none\" points=\"213.5796,-22.5 290.228,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"251.9038\" y=\"-7.3\">(None, 46)</text>\n",
       "</g>\n",
       "<!-- 5125325264&#45;&gt;5125328008 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>5125325264-&gt;5125328008</title>\n",
       "<path d=\"M168.4434,-81.3664C168.4434,-73.1516 168.4434,-63.6579 168.4434,-54.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"171.9435,-54.6068 168.4434,-44.6068 164.9435,-54.6069 171.9435,-54.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 319,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the model\n",
    "model = keras.Sequential([\n",
    "    keras.layers.Embedding(\n",
    "        input_dim=max_features,\n",
    "        output_dim=100,\n",
    "        embeddings_initializer='glorot_uniform',\n",
    "        input_length=maxlen),\n",
    "    keras.layers.SimpleRNN(40, return_sequences=False, activation=\"relu\"),\n",
    "    # This is the most basic kind of RNN!  We're using 20 units,\n",
    "    # which somewhat reflects our \"memory\" of past events in a sequence\n",
    "    # For the purposes of keras, it's just another type of \"unit\" you can try!\n",
    "    keras.layers.Dense(46, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model.compile(\n",
    "    loss='sparse_categorical_crossentropy',\n",
    "    optimizer='adadelta',\n",
    "    metrics=['accuracy', top_3])\n",
    "SVG(model_to_dot(model, show_shapes=True).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 320,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:58:08.289824Z",
     "start_time": "2017-11-08T12:57:30.934105Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 8982 samples, validate on 2246 samples\n",
      "Epoch 1/10\n",
      "8982/8982 [==============================] - 5s 587us/step - loss: 2.5485 - acc: 0.3264 - top_3: 0.5730 - val_loss: 2.0660 - val_acc: 0.4225 - val_top_3: 0.6296\n",
      "Epoch 2/10\n",
      "8982/8982 [==============================] - 3s 377us/step - loss: 2.0401 - acc: 0.4272 - top_3: 0.6357 - val_loss: 2.0373 - val_acc: 0.4519 - val_top_3: 0.6371\n",
      "Epoch 3/10\n",
      "8982/8982 [==============================] - 3s 372us/step - loss: 1.9619 - acc: 0.4500 - top_3: 0.6539 - val_loss: 1.9603 - val_acc: 0.4684 - val_top_3: 0.6567\n",
      "Epoch 4/10\n",
      "8982/8982 [==============================] - 3s 368us/step - loss: 1.8883 - acc: 0.4771 - top_3: 0.6676 - val_loss: 2.0537 - val_acc: 0.4265 - val_top_3: 0.6380\n",
      "Epoch 5/10\n",
      "8982/8982 [==============================] - 3s 382us/step - loss: 1.8167 - acc: 0.5080 - top_3: 0.6765 - val_loss: 1.8674 - val_acc: 0.5102 - val_top_3: 0.6714: 1.8106 - acc: 0.\n",
      "Epoch 6/10\n",
      "8982/8982 [==============================] - 3s 375us/step - loss: 1.7567 - acc: 0.5323 - top_3: 0.6907 - val_loss: 1.9659 - val_acc: 0.5013 - val_top_3: 0.6745\n",
      "Epoch 7/10\n",
      "8982/8982 [==============================] - 4s 395us/step - loss: 1.7033 - acc: 0.5444 - top_3: 0.6977 - val_loss: 1.8320 - val_acc: 0.5156 - val_top_3: 0.6736\n",
      "Epoch 8/10\n",
      "8982/8982 [==============================] - 3s 384us/step - loss: 1.6579 - acc: 0.5592 - top_3: 0.7051 - val_loss: 1.9184 - val_acc: 0.5085 - val_top_3: 0.6821\n",
      "Epoch 9/10\n",
      "8982/8982 [==============================] - 4s 395us/step - loss: 1.6082 - acc: 0.5694 - top_3: 0.7173 - val_loss: 1.8430 - val_acc: 0.5223 - val_top_3: 0.6723\n",
      "Epoch 10/10\n",
      "8982/8982 [==============================] - 4s 415us/step - loss: 1.5551 - acc: 0.5794 - top_3: 0.7310 - val_loss: 1.9746 - val_acc: 0.5187 - val_top_3: 0.6915\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "history = model.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    batch_size=64,\n",
    "    epochs=10,\n",
    "    validation_data=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**WHOA!** Over 100% improvement on ANN!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 321,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T12:58:08.968082Z",
     "start_time": "2017-11-08T12:58:08.293780Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/matplotlib/figure.py:1999: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.\n",
      "  warnings.warn(\"This figure includes Axes that are not compatible \"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa4AAAGoCAYAAAAerAGHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd4XNWd//H39KbebMm9Hgsb2yBX\nsMEQh2ASEhLKghMgJCxhA4HALxtKIMsSliSYsqQvsM5SQjPghBBaCKEZjLEMWDbyccO25CrJVh2N\npv7+uCNp1GxJM7I0mu/reeaZO+feuTpzLM9H595zzzVFIhGEEEKIZGEe7AoIIYQQfSHBJYQQIqlI\ncAkhhEgqElxCCCGSigSXEEKIpCLBJYQQIqlYe7ORUmo+8Eut9ZJO5ecCPwWCwEqt9cNKKRfwBFAA\nNACXa62rElprIYQQKeuYPS6l1I+BRwBnp3Ib8ABwFnA6cJVSagTwb0CZ1nox8BhwW6IrLYQQInX1\npse1A/gG8Hin8mJgu9b6CIBS6j3gNGARcE90m1eA27vbaWlpqVz5LIQQokclJSWm7sqPGVxa6+eV\nUuO7WZUB1MW8bgAyO5W3lnXL7XYf68cfk8/nw+l0HntD0YW0XXyk/fpP2i4+qdB+Xq+3x3W9OsfV\ng3ogPeZ1OlDbqby1rFvFxcVx/HhDeXl5QvaTiqTt4iPt13/SdvFJhfYrLS3tcV08wVUOTFFK5QCN\nGIcJ7wXGAecA64BlwLtx/AwhhBCigz4Hl1JqOZCmtX5IKXUj8BrGII+VWuu9SqnfA49Gz3n5geUJ\nrbEQQoiU1qvg0lrvAhZEl5+MKf8r8NdO23qBCxNXRSGEEKKdXIAshBAiqUhwCSGESCoSXEIIIZKK\nBJcQQoikIsElhBAiqcRzHZcQQoh+er60kmfXV/TrvV6vF/c7Xed2uGjOGM4vGR1v1YY8CS4hhEgR\njY2N/OQnP6GhoYFDhw6xfPlypk+fzt133004HGbEiBHce++9aK27lA2lKaYkuIQQYhCcXzK6372j\n/k75tHv3br785S9z1llncfDgQS699FJcLhf3338/kyZNYtWqVezYsYOf/vSnXcqmT5/er7oOBAku\nIYRIEXl5eTz66KO8/vrrpKWlEQwGqa6uZtKkSQBceKExd0R3ZUOJDM4QQogUsXLlSmbPns29997L\n2WefTSQSoaCggF27dgHw0EMP8fe//73bsqFEelxCCJEizjjjDO666y5efvll0tPTsVgs3HHHHdx6\n662YzWby8/P59re/zYgRI7qUDSUSXEIIkSIWLFjASy+91KX8ySef7PB65syZXcqGEjlUKIQQIqlI\ncAkhhEgqElxCCCGSigSXEEKIpCLBJYQQIqlIcAkhhEgqElxCCCE6uPTSS9mxY8dgV6NHch2XEEIM\nhk+ego+f6Ndbx3qbYK2n64qTvgWzL4mzYkPfMYNLKWUGfgfMAlqAK7XW26PrZgP/HbP5AuA8YB2w\nFdgULV+ttX4wgfUWQgjRR9deey2XXXYZ8+bNo6ysjHvuuYecnJwOs8UvX778mPt59dVX+dOf/kQw\nGMRkMvGb3/yG7Oxsfvazn7Fx40YCgQA/+MEP+MIXvtClbOnSpXF/jt70uM4DnFrrhUqpBcB9wNcA\ntNafAEsAlFIXAnu11q8qpZYCT2mtfxB3DYUQYjiafUm/e0d7+jk7/IUXXsjq1auZN28eL7zwAvPn\nz2fq1KkdZovvTXDt2rWLhx56CJfLxU9/+lPee+89XC4XR44c4bnnnqOuro4//vGPRCKRLmXHK7gW\nAa8CaK3XKqXmdN5AKeUB/hM4LVpUApQopd4GDgHXaa33x11bIYQQ/bZ48WJWrFhBbW0t69ev55FH\nHuG+++7rMFt8b+Tm5nLTTTfh8XjYuXMns2fP5vPPP2f27NkAZGZm8sMf/pCHHnqoS1ki9Ca4MoC6\nmNchpZRVax37Cb8LrNJaV0dfbwFKtdZvKKW+CfwauKDzjsvLy/tZ7XY+ny8h+0lF0nbxkfbrP2m7\n+MTTfiUlJdx4441ts8QXFhaybNkyysrKeOONNygvL6epqYmdO3fi9/u7vL+pqYn777+fhx9+GIA7\n7riDffv2kZaWxpo1a5g3bx5NTU2sWLGCZcuWdSm744474vnoQO+Cqx5Ij3lt7hRaAN+kYzC9CXij\ny6uBO7vbcX+6up3194ZqQtouXtJ+/SdtF5942u/qq69m6dKlvPbaa1RWVnLXXXexYcMG0tPTcTqd\nTJo0CY/Hw8SJE9vuyRUrEokwb9487rjjDqxWKzk5OVgsFi6//HIqKiq48847CYVCXHPNNZx22mld\nynpb79LS0h7X9Sa41gDnAs9Gz3GVxa5USmUCDq11RUzxI8DzwLPAF4CeayCEEOK4KSwsZPPmzQCM\nHj2629niH3/88R7fbzKZePDB7sfa3X777b0qi1dvgms18EWl1PuACbhCKXUjsF1r/SIwFdjV6T03\nAyuVUt8HmoArE1dlIYQQA23jxo2sWLGiS/myZct6NYBjIB0zuLTWYeDqTsVbYtZ/hDHyMPY9nwNn\nJKKCQgghjr+ZM2cetec1mGTmDCGEEElFgksIIURSkeASQgiRVCS4hBBCJBUJLiGEEElFgksIIURS\nkeASQgiRVCS4hBBCJBUJLiGEEElFgksIIURSkeASQgiRVCS4hBBCJBUJLiGEEEmlN7c1EUIIkULC\n4Qj+UJiWQJiWUAh/MExLMIy/9RFd549Z17q+IN3BWdNHDmj9JLiEEGIICIaMQIgNifbnUIfA2Lm7\nEe3b21be0iFM2gMm9n2dA6ZtfYftjedgONLvz5HltrG0eARmsymBrdORBJcQIuX5g2Ga/SGaAyG8\n/iC+QGsQhLqESesXfndh0RYUsdu39VyOvk2oz2FxqNtSm8WE3WLGbjXjsFqwW41lu8WMw2Y8pzut\nOGLXt21vbt8+Zr2ju/UWMw6bpcN7HVYzGS7bgIYWSHAJIZJAIBSmORCi2R/C6w9FQyaIN/raFwi1\nLTf7g9EACsWEUet7gzQHwjT7gx3Wx9PDgPaw6PxF3h4AZjLttrbw6C4IYkOkNWDaA6FjAO2t2M20\nKZO6/jyLecBDYyiQ4BJC9EskEjEOPQXC+IJGeLQEw/gCIXyB1ueYsqDRg2kJhqnYdxjPjs/agsYb\nDZC2YArEBk2IQKhvwWIxm3DbLDjtFtx2Cy5b9NluIcfjwB0td0bL25et0eVjhYllUMPC3rififlp\nx/VnDiUSXEIMI6FwhAZfgPpm4wu/NTx80fBoDZGWmHBpD5auZa2hFBtOsdv0lwlw2xtwRYPCZTNC\nxW23kOW2xQSKtUO4uDqEUNf3ti7bLWZMpuHf80hVElxCDDGRSARfIExts5+65gC13gB1zQHqvIGu\nZdFHrTdArddPQ0uQSB+PeplN4LQZPQ6n1YzTZsFhs+CwmnHazOR47DitFhw2M06r0RtxRtc7Wt8X\nXde+jVHmiN0+5vWOrZoTTjhhYBpwuAuHIdL/PxqGg2MGl1LKDPwOmAW0AFdqrbfHrH8QWAQ0RIu+\nBtiAJwEXsA+4QmvtTWzVhRjagqEw9b4gtd5o2DQHqG8LmWjgNPvbylq3qfMG8Id6/mKymE1kuWxk\numxkum3keuxMzPOQ5baT4bKR5bKR4bK1HfJqDRRHD4FiswzQ5ZyRCARbIOCFQB20NENDEwSa8Rza\nDp5aMNvAYo0+28BsBYs9utzNuqHci+rweZujD28Pz3GUhVoobv2ZJguYzGCOPre9NvewztTxdduy\nueu2x1xn6v5nZI+D028a0H+r3vS4zgOcWuuFSqkFwH0Y4dSqBPiS1rq6tUAp9SvgSa31/ymlbga+\nBzyQwHoLcVz5AiGq6n00HNxJZPf7OPavJ1h3iI3/8NActuANWfCGzDQGLTQGzTQETDQEzQSw4sdG\nIGLFT+vDRgArFpsDh8OJy+Ek2+XCleXCXeTG7UrH4/aQ7nGR7vGQ6XaS4baT5baR5bbjsVviPwwW\niUAoAP76jl+Mfm+nL8rosr+pa1nb9rHlrdtHl+m++zcO4K1+1LstxDqFWltZa/jZulnX0/ax6+zG\ncjjcy9DptL6Hz3tUVhfYXGBzR5+jy3YPePI7ltlcYHVSVV1Nfl4uhEMQCRk9sHDI+Hft8Lp1OdpL\n69O6SHQ5cJR1nd8XhsM74LQfD3pwLQJeBdBar1VKzWldEe2NTQEeUkqNAP5Xa70y+p67o5u9El2W\n4BJDRjgcoa45QE1TC1UNfmqaWqhp9FPd2EJ19PlwQzMZjduZ2LSRWZFy5pg1J5gOA1AfcVMdycBh\nCuIwBbETxGYKYosEsBIyTuLYelERf/TRcLSNTMYXqtUR/dKNPlsd0S/a6MMas2y2QsgfEzjNEIhZ\n9jcZXzh9Ymr/8rS7Y75o3eDOBdvo6Bdup3Vt7/G0fQnvrtzPuDGjIBQ06hkOGEEaDkafo6/bloMx\nZf5O27Wu83fcLhw0ekAtDV33EY7+3M4/M/YQnNnWNUxal9NGdB823T4fZZ3VafSO+qi6vJz84uJj\nbzhM9Sa4MoC6mNchpZRVax0EPMCvgfsBC/BPpdT6Tu9pADK723F5eXl/693G5/MlZD+paLi1nT8U\noc4XotYXorY5+tx5Ofq6zhei80A1OwFmmXeyyL6Vr5o1J0Y0aZEmMEO9NZf96TNZm3Ui3vzZmPMm\nYydEVpqra+8nEsYUDmAKB43nkD+67I++DkTXd3q0bdddmfFsDvm7f68/gKm5OaYsSNhiJ2JxEra6\nCNvyiLhGE7a4CFudRCwOo9ziJGJ1dng21hvPYavLWLY4iFgc8f8VHQH84MvIxtvibC83MTTOuIdD\nmCJBIiaLEf6JEIg+AAgBjdFH/w23/7t91Zt/mXogPea1ORpaAF7gwdbzV0qpNzHOhbW+pzn6XNvd\njosT8BdDeXl5QvaTioZ620UiERpagm09oZrGFqoa/dQ0tkRf+2N6SS3U+4Ld7sdpM5OX5iA3zcGE\nAnt02U6h08/kls2MbviU3JoNuA59ginUYrwpR8HYC2DcKTB2IRlZY8no9KU91NtvKJO2i08qtF9p\naWmP63oTXGuAc4Fno+e4ymLWTQWeUUqdhDFh7yLg0eh7zgH+D1gGvNufiovhIxKJ4PWHONzk54jX\nzxFvgCOty01+DncoC7SV+XsYcp3lthkB5LFTXJRBnsfeFk65acZyXvTZ3XpOqOEA7H4f9nwAuz6A\ng5uNQ0NmKxTOgnn/CmMXwtgF4Mk7zi0khOit3gTXauCLSqn3MTr0VyilbgS2a61fVEo9DqzF6Aw/\nprXerJS6C3hUKfWvQDWwfIDqnzoiEQj6jHMT/kbj5HfbctMxlrt5HWhiqr8ZMosgczRkjok+RkNW\n9DljlHEepUtVIjT5Q0a4tAWRn8NNxpDsw01+ar2BDuuONPU8Us5kgiyXjWy3nWyPnVFZLmYUZZDj\nae8dtT7npznI9tiPPRIuEoGa7bDpfdizFva8D0d2GetsHhgz1xj5NHYhjJ5jnH8RQiSFYwaX1joM\nXN2peEvM+hXAik7vOQicnYgKJq1QELw1xonhngIlcJRg6e51X67dsHmML2O7B+xpxrMzAzIK217X\n1TaQbWshVFuB6cBrWLwd5z6LYKLBmkONtYAD5FEZzmN3MIft/iz2hHLZG8mjDg/G3zMGswmy3Hay\n3UYQjclxM3N0JtkeOzlue1s4ZbttbWUZLhuWeGceCAXhwEajN7U7Glbe6EBXd57Ri5p3lfE8cqYx\nuEEIkZSGwunQ5BAKGEHUVA1NVe3L3ujrpuqO633dntbrymQBR1p7uLQGTXph1+CxubtuZ/cQtrlp\nwkl9yEFdyE5twEqdL0S9z7g2qL45aDz7jGuE6muM8sMNPhr87TNB2wkw0nSYUabq6KOGCabDjInU\nMI6dlIQ/xB7xG7810d+coNVDIK2ISMZoLNljsOWMw9zaY8sabXyOgQgJfxNUrjeCas8HUPGR8YcA\nQPZ4mHKWEVLjToHcyUP7+h8hRJ+kbnCFAjHB0xo6VR3LYpd7CiKT2RgK7M4zzouMnNG+7M4FZ1ZM\n0HTTC7LYwWTCFwhR3xou3YVNc4C6w53KmwPUN9fT0HL4qLMlmE2QEb1gNcNpPI/MdBLOMjNxVEG3\nvaBst510p7XjHGyRiNEWdRVQVwl1FVjrKrHWVUBtBWwra+/lxLZPemH7Yci2Q5Fj2g9ROjOO/e/V\nVAMVa9vPUe3/1BjGjAlGzICTvhk9P7XQ6FUKIYat4RNcQX97+HirjS+6Lr2hmGDy1XW/n9Yg8uQb\nzyNPjAZRPnhyY5bzjGVXdrfXYTT7Q6zbdZgDdc3U18aGjY/65obo62DblD09DUJo5bJZyHBZ28Jn\nZIaTqSPSjdcuGxlOa9tyW0C5jfI0h7XbC1aNkUnTet/GJhOk5RuPUSd3v43fC/V728Ottj3k2Lse\nPvuLcb1MLEdmx3NrrYEWDrYf9qvWxrYWB4wqgVOvN0JqzDxwdnu1hRBimEru4Pr0GSb+/Wfw53po\nOVoQxfSACme1h44nr9NyvtFD6scFgZFIhJ3VTbylq3hLH+LDzw93CKPWXk9rjyfDZWVkprMtZDI6\nhE7HEDLunWPpbysdX3Y35E0xHt0Jh6HpUDTQYkKt9XnP2o69W0cmjJ0Psy42DvsVndTtgBEhROpI\n7uBy5+LLmYZj5IT2HlJrALWGUT+DqDe8/iAf7KgxwmrrISoONwMwMd/Dt+aPY4nKZ2K+h0yXDY/d\nmhL3yTkmsxnSRxqPMXO736alAer2GoNR8qcN2L+fECI5JXdwTVnKvuAoMo/ThXiRSIQdVU28pQ/x\n9taqtl6Vy2bh1Mm5XHXaJJZMzWdMjvu41GfYcqRDQR8OYQohUkpyB9dx4PUHeX97DW9tPcRbuorK\nI0avanJBGpctGMcSVcDcCdnJcyhPCCGSnARXJ0avqjF6rqqKdZ8fxh8K47ZbOGVSHlefPonTpVcl\nhBCDRoILaGoJsmZ7NW9treJtXcXeWqNXNaUgjctPMXpVc8ZLr0oIIYaClAyuSCTCtkONvKWNw38f\n7TpMIBTBY7dwyuQ8vn+G0asanS29KiGEGGpSJrgaW3tVuop3trb3qqaOSOOKUyewZGo+c8bnYLfK\nCDYhhBjKhm1wRSIRth5s71Wt393eqzp1ch7XnDGZ01U+o7Jcg11VIYQQfTCsgqvBF2DN9hre3nqI\nt3UV++p8AEwbmc53Fk1gydQCSsZlS69KCCGSWFIHVyQS4fMjft5+ewdv6UOs33WEYDhCmsPKosl5\nXPeFfE5X+RRmSq9KCCGGi6QOrl+/uZ37/14JGL2qKxdPZInKp2Rc9rHv1ySEECIpJXVwnXPiSEzN\nR7hw8UxGZjoHuzpCCCGOg6TulkwuSGfppHQJLSGESCFJHVxCCCFSjwSXEEKIpCLBJYQQIqkcc3CG\nUsoM/A6YBbQAV2qtt8esvwG4OPryZa31fyqlTEAlsC1a/oHW+paE1lwIIURK6s2owvMAp9Z6oVJq\nAXAf8DUApdRE4JvAfCAMvKeUWg14gQ1a63MHptpCCCFSVW8OFS4CXgXQWq8F5sSsqwDO1lqHtNYR\nwAb4gBJglFLqn0qpl5VSKsH1FkIIkaJ60+PKAOpiXoeUUlatdVBrHQCqo4cGVwAfa623KqVGAj/X\nWq9SSi0CngC63Ke9vLw87g/g8/kSsp9UJG0XH2m//pO2i0+qt19vgqseSI95bdZaB1tfKKWcwEqg\nAfh+tHg9EATQWr+nlCpSSpmivbI2xcXF8dQdMMIvEftJRdJ28ZH26z9pu/ikQvuVlpb2uK43hwrX\nAOcARM9xlbWuiPa0/gJ8qrX+ntY6FF31H8APo9vMAio6h5YQQgjRH73pca0GvqiUeh8wAVcopW4E\ntgMW4HTAoZRaFt3+FuAXwBNKqS9j9Ly+neiKCyGESE3HDC6tdRi4ulPxlpjlnuZb+nJ/KyWEEEL0\nRC5AFkIIkVQkuIQQQiQVCS4hhBBJRYJLCCFEUpHgEkIIkVQkuIQQQiQVCS4hhBBJRYJLCCFEUpHg\nEkIIkVQkuIQQQiQVCS4hhBBJRYJLCCFEUpHgEkIIkVQkuIQQQiQVCS4hhBBJRYJLCCFEUpHgEkII\nkVQkuIQQQiQVCS4hhBBJRYJLCCFEUpHgEkIIkVSsx9pAKWUGfgfMAlqAK7XW22PW/yvwPSAI3KW1\nfkkplQc8CbiAfcAVWmvvANRfCCFEiulNj+s8wKm1XgjcDNzXukIpNRK4DjgV+BLwc6WUA/gp8KTW\nejHwMUawCSGEEHE7Zo8LWAS8CqC1XquUmhOzbh6wRmvdArQopbYDM6PvuTu6zSvR5Qc677i8vDyO\nqht8Pl9C9pOKpO3iI+3Xf9J28Un19utNcGUAdTGvQ0opq9Y62M26BiCzU3lrWRfFxcV9rnBn5eXl\nCdlPKpK2i4+0X/9J28UnFdqvtLS0x3W9Ca56ID3mtTkaWt2tSwdqY8qbY8r6VLG+SNR+UpG0XXyk\n/fpP2i4+qdx+vQmuNcC5wLNKqQVAWcy6dcB/KaWcgAMoBjZF33MO8H/AMuDdzjstKSkxxVVzIYQQ\nKckUiUSOukHMqMKZgAm4AiOUtmutX4yOKrwKY6DH3Vrr55VSI4BHMXpb1cByrXXTwH0MIYQQqeKY\nwSWEEEIMJXIBshBCiKQiwSWEECKpSHAJIYRIKhJcQgghkooElxBCiKQiwSWEECKpSHAJIYRIKhJc\nQgghkooElxBCiKQiwSWEECKpSHAJIYRIKhJcQgghkooElxBCiKQiwSWEECKpSHAJIYRIKr25A/KA\nKC0tlRuBCSGE6FFJSYmpu/JBCy6AkpKSuPdRXl5OcXFxAmqTeqTt4iPt13/SdvFJhfYrLS3tcZ0c\nKhRCCJFUJLiEEEIklaQOrje3HOSlLXWDXQ0hhBDHUVIHlz7QyG8/rOGljfsGuypCCCGOk6QOrisX\nT0DlObjl+TJ21zQNdnWEEEIcB8ccVaiUsgErgfGAA7hLa/1izPobgCuBqmjR94A9wBNAAdAAXK61\nriLBbBYzt5xewHV/28+1T37Mc/+2EIfVkugfI4QQYgjpTY/rW0CN1noxcDbwm07rS4DLtNZLog8N\n/BtQFn3PY8Btiax0rBFpNlZcOIuyvXX8/OUtA/VjhBBCDBG9Ca5VwO3RZRMQ7LS+BLhFKfWeUuqW\naNki4NXo8ivA0ngrejRfmj6Sb58ynv97fxevbT4wkD9KCCGSzgsvvMC999472NVImGMeKtRaNwIo\npdKB5+jae3oa+C1QD6xWSn0FyABah/s1AJnd7bu8vLx/tY7h8/koLy/n6xNNrNF2/t8zH+M4dxQj\n0mxx73u4a2070T/Sfv0nbRefvrbfvn37qKmpGTZt3quZM5RSY4DVwO+01k/GlJuA/9Za10Vf/w04\nCSPE0qObpQO13e03EVd+x15B/kjhOL7yq/f473UNPPu9hditST32ZMClwtX3A0nar/9Sue2eL63k\n2fUVce3D6/XidrvbXl80Zwznl4zucfvy8nJ8Ph8ffPABf/vb37BarcyZM4d///d/p7S0lF/+8pdY\nrVZcLhcPPvggVVVV3HLLLVitVsLhMPfddx+FhYVx1bmv4po5Qyk1AngduElrvbLT6gxgk1IqLRpi\nZwKlwBrgnOg2y4B3+1HvPhuX6+EX58/kk4pa7n1dH48fKYQQSWH37t288sorPP300zz99NPs3r2b\nf/7zn7zxxhssW7aMJ554gksuuYT6+nref/99Zs6cyR//+Ed+8IMf0NDQMNjV76A3Pa5bgWzgdqVU\n67muhwGP1vohpdStwD+BFuAfWuuXlVJvAY8qpd4D/MDyxFe9e1+eWcgHO8fy0Ds7WTAxhzOnjThe\nP1oIIY7p/JLRR+0d9UZ/eqzl5eUsWbIEm804jTJnzhy2bdvG1VdfzR/+8Acuv/xyRowYwcyZM7ng\nggt4+OGHufLKK0lPT+eGG26Iq76J1ptzXNcD1x9l/ePA453KvMCFcdeun2778gmU7q7lxmc/5eXr\nFlOU5RqsqgghxJBQXFzMxo0bCQaDWCwWPvroI8477zxefPFFvv71r3PTTTfxP//zPzz77LNMnDiR\nkpISrr32Wl566SUeeeQRfv7znw/2R2gzqLPDDxSnzcJvl5/Eub9+j+ue+pinr1qA1SLnu4QQqWvc\nuHGcfPLJXHLJJYTDYUpKSli6dCkbN27ktttuw+VyYTabufPOO4lEItx00038/ve/JxwOc8sttxz7\nBxxHwzK4ACbmp3H3N07k+qc/4f6/b+XHZ08b7CoJIcSg+MY3vtG2fMUVV3RYN2vWLJ599tku73nq\nqacGvF79Nay7IV+bPYqL547hd2/t4O2tCZ+4QwghxCAY1sEF8B/nTkeNSOfGZz7hYL1vsKsjhBAi\nTsM+uFx2C7/95kl4/SGuf/pjQuHIYFdJCCFEHIZ9cAFMLkjnZ+fNYO3Ow/zqH9sGuzpCCCHikBLB\nBXBByWi+cfIofvXmNt7fXj3Y1RFCCNFPKRNcAD/72gwm5nm4/plPqGpoGezqCCGE6IeUCi6Pw8pv\nv3ky9c0BbnjmE8JyvksIIZJOSgUXwLSRGdzx1em8t72a3721fbCrI4QQQ8all17Kjh07elx/5pln\n0tIy+Eerhu0FyEdz8dwxfLCjhvv/vpW543OYPzF3sKskhEgVnzwFHz8R1y7Geptgrae94KRvwexL\n4qxY8kjJ4DKZTNz9jRPZWFnLdU9/zMvXLSY3zTHY1RJCiAFx7bXXctlllzFv3jzKysq45557yMnJ\noaGhgUOHDrF8+XKWL+/9XOiVlZXceuuthEIhTCYTt912G9OmTeOWW25h9+7d+Hw+LrvsMs477zwe\neOABPvzwQ4LBIGeddRZXXXVV3J8nJYMLIM1h5TfLT+Ybv3uf/7fqU1ZePhez2TTY1RJCDHezL4m7\nd7Snj7PDX3jhhaxevZp58+bxwgsvMH/+fKZOncpZZ53FwYMHufTSS/sUXPfccw+XXXYZS5cupby8\nnFtvvZXHHnuMjz76qG36qDVr1gDw17/+lccee4yCggJeeOGFvn3QHqTcOa5YM0ZlcvtXinlLV/HQ\nuzsHuzpCCDEgFi9eTFlZGbUqlFyWAAAgAElEQVS1taxfv54LL7yQN954gx/96Ef8/ve/JxgM9ml/\nO3bsYO7cuYAx6/yBAwdIS0vj1ltv5fbbb+eGG27A7/cDsGLFCu677z6++93vUl9fn5DPk9LBBfCt\nBeM458SRrHhNU7r78GBXRwghEs5sNnP22Wdzxx13sHTpUlauXMns2bO59957Ofvss4lE+jbCetKk\nSaxfvx4w7vOVl5fHoUOH2Lx5M7/97W956KGHWLFiBX6/n1dffZX777+fxx57jNWrV7N37964P0/K\nHipsZTKZ+MX5Mynb+y7XPfUJf7tuEVlu+2BXSwghEur8889n6dKlvPbaa1RWVnLXXXfx8ssvk56e\njsViaesh9caPf/xjbr/9dlauXEkwGOS//uu/yM/Pp6qqiosvvhiz2cx3vvMd7HY7mZmZXHTRRTid\nTk499VSKiori/iymviZtopSWlkZKSkri3k9/7gTanY2VtZz/+/c5fWoBD19Wgsk0/M93JartUpW0\nX/9J28UnFdqvtLSUkpKSbr+IU77H1Wrm6CxuWVbMnS99xso1u/juogmDXSUhhDjuNm7cyIoVK7qU\nL1u2rE8DOAbSUYNLKWUDVgLjAQdwl9b6xZj1lwA/BIJAGfB9rXVYKbUBaD0L97nWuuOdy4aoK04d\nzwc7a/jFK+XMGZfNrDFZg10lIYQ4rmbOnMnjjz8+2NU4qmMNzvgWUKO1XgycDfymdYVSygXcBZyh\ntT4VyAS+opRyAiat9ZLoIylCC4zzXSsumElBupNrntxAXXNgsKskhBCik2MF1yrg9uiyCaNn1aoF\nOEVr7Y2+tgI+YBbgVkq9rpR6Uym1IJEVHmhZbju/Xn4SB+p83Pz8xj6PthFCCDGwejU4QymVDrwI\nPKy1frKb9T8Azok+ZgALgEeAKcArgNJad7hQoLS0NOJ2u+P+AD6fD6fTGfd+OntuUy3/W3qY78/P\n5dxpmQnf/1AwUG2XKqT9+k/aLj6p0H5er7f/gzOUUmOA1cDvOoeWUsoM3ANMBc7XWkeUUluB7Vrr\nCLBVKVUDFAIVnfediFExAzW65icqws7Gj3hkfQ3nzCtmxqjhF16pMDJpIEn79Z+0XXxSof1KS0t7\nXHfUQ4VKqRHA68BNWuuV3WzyP4ATOC/mkOF3gPui7y8CMoD9fa/24DKbTdx30WxyPHaufXIDDT45\n3yWEEEPBsXpctwLZwO1KqdZzXQ8DHmA98F3gXeBNpRTAg8D/Av+nlHoPiADf6XyYMFnkeIzzXRc/\ntJZbV2/iVxfPTonru4QQYig7anBpra8Hrj/KJj312IbGYP8EmDs+hxu/OJUVr2kWTsxl+fyxg10l\nIYRIaSk/V2Fv/Nvpk1g8JY///OtmyvcnZpJIIYQQ/SPB1Qtms4kH/mU2GS4b1zy5gaaWpDzyKYQQ\nw4IEVy/lpTl48OLZ7Kpu4vY/b5Lru4QQYpBIcPXBKZPyuO4LU3jh4708V1o52NURQoiUJMHVRz84\ncwoLJ+by079sZtvBhsGujhBCpBwJrj6ymE08ePFsPA4L1zy5gWZ/aLCrJIQQKUWCqx8KMpw88C+z\n2XaokTte3DzY1RFCiJQiwdVPi6fkc82SyTyzvoI/fxz/raiFEEL0jgRXHH64dArzxudw6+oydlQ1\nDnZ1hBAiJUhwxcFqMfPgJbNxWM1c86cN+AJyvksIIQaaBFecCjNd3H/RbLYcaOBnL3022NURQohh\nT4IrAc6YVsD3TpvInz7cw0sb9w12dYQQYliT4EqQH31JcfLYLG5+vozdNU2DXR0hhBi2kju4IhEs\nLbUQGvy5A20WM7+65CQsZhPXPLmBlqCc7xJCiIFwzDsgD2lv3MHUNf8NfwacmeDOBVcOuHM6LedE\nl3NjlnPA6khodUZnu1lxwUyueryUn7+8hTu+Oj2h+xdCCJHswTXvKg54zYzMsEPzYfAeBm8NNB6E\nQ+XG68BRDtvZ0zqGW5fgy+5Ungt291GrdNb0kXzn1AmsXPM5CybmcvaMkQn+0MNYOAQHN8GuNbB7\nDYSDsPAaGL8Y5AaeQoio5A6uzFEcmXoRI4uLe94m4GsPteZosHljX0fLmg/D4c+N1y11Pe/P6ozp\nvWV37MlFA+6WKVnUbj/Ig8/9HZd1KSeNzyPDaUv85092oSAc+LQ9qHZ/0N722eMh0AyPngtjF8Lp\nP4aJZ0iACSGSPLh6w+YEWxFkFPX+PaEANB/pGnZty0falw9uNp59tRAJGz8SuD+6q+an7GyOjGe3\nYyq+/Jm4xs1h3NSZTB+dg9NmSfjHHdJCAdj3Cex+zwirPWvBH52oOGcSTP8ajFsE40+FzNFGcG14\nHN57AB7/OoyeC6ffBJOXSoAJkcKGf3D1h8UGaQXGo7fCYSO8YnpyzbUHqNn5CYX7P2FGwxs4970E\n+6DxfSefRsZT6ZqGv2AWaRPmMmHqiajCDGyW5B4v00GwBfZuaA+qinXth27zFMy8EMadajwyCru+\n3+aC+VdByeXw8RNGgP3pAig6yQiwqWdLgAmRgo4ZXEopG7ASGA84gLu01i/GrD8X+CkQBFZqrR9W\nSrmAJ4ACoAG4XGtdlfjqDyFmc/u5sigXMHp+9EU4BNVbqduxjrod6xhz4BNOanoZe8WfoQLq33bz\nUWQi+z3FBEfMImPSPKZMLWZifjpmc5J8OQd8sHd99NDfe1DxEQSbjXUFJ8Ds5UZvatypffujwOqA\nud+Fky6FjU/DO/fCUxfDyBONAFNfNtpfCJESetPj+hZQo7W+VCmVA3wCvAhtofYAMBdoAtYopV4E\nvgmUaa3vUEpdDNwGXD8QHyBpmC1QUExmQTGZCy83ykIBIofKObxtHQ2fr2P8wY3M867GumsV7ILD\nb6TxPpM4lF5MuPAksibNQ01RjM5xYxoKPQ2/FyrXtZ+jqlwPoRbABCNnQMm3jaAaewp4cuP/eVY7\nnHwZzLoEylbBOyvgmW9BwXQ47UdwwteMdhZCDGu9Ca5VwHPRZRNGz6pVMbBda30EQCn1HnAasAi4\nJ7rNK8DtCantcGOxYSqcSW7hTHJPu9IoC7YQOvAZ1VvX0rTrIyZVfcrCxmexbHsatkHVK5m8a5pE\nTcZ0KJpNzpQFFE+ZTEGGc+Dr29IIFR8aIbVrDewthXAATGYYORPm/SuMXwRjFxgjMgeKxWb03k68\nCDa/YATYc1cYhx9P+3eY8Q0JMCGGMVMkEunVhkqpdIye1sNa6yejZYuAH2it/yX6+k5gD3BxtLxc\nKWUG9mitR8fur7S0NOJ2H31oeW/4fD6czuPwpT2ITEEf1sPbaN63CdOhz8hu0IwMVmLBGAyyL5KD\nNk3ikHsqLbnFuIqmM66wgHTH0b+8j9V25kATrupPcR/agOfQxziPbMEUCRExWfBlT6Op4CS8+SfT\nnDeTsD0toZ+5T8Ih0ivfJG/zH3HW76QlfSw1xd+mbtxZYB6407ip8Ls3UKTt4pMK7ef1eikpKen2\n0FKv/lcrpcYAq4HftYZWVD2QHvM6HajtVN5a1kXx0Yax91J5eXlC9jP0nQRc1P6ypZGWyk85pD+g\nZU8pJxwu4wzvR+AFKqAinM8O2xTqsmdgHX0yI6YtoHjCGDyO9n/yLm3XXAt7PoBd7xm9qv2fGiMl\nzTYYdTKccD2MX4RpzHxcjjRcx+2z98L0GfDFa2HLX3G8fQ9F6+6kaNtjsPhHMOtio5eWYKnzu5d4\n0nbxGZLtFwrC529B/X44+dK4d1daWtrjut4MzhgBvA5cq7X+R6fV5cCU6LmvRozDhPcC44BzgHXA\nMuDdftVc9MyRhmPSqYyZdGp7ma+Oxl2lVOu1BCs3MP3IJvKr34dq4BPYFR7BOsdUGnNOxD62BGzp\nTAptw175Aex6Fw5sAiJgsRtDzxf/yDhHNXreMS+8HhLMZuM817RzYesr8PYv4cVr4Z17YNGNMPub\nxnmy4ayuEna+DQc2wph5MHVZcvzbieQUDhunDzY9B5tXG5cGZY4dsD8WW/Wmx3UrkA3crpRqPVf1\nMODRWj+klLoReA1j3sOVWuu9SqnfA49Gz3n5geUDUHfRmTOTtGlnkjbtzPYy72Hqdn5E9da1hPd+\nzIm1m8k7+C4cbN+kBTt73NNpmHg17qmnMfrE00jzDOKhv3iZzTDty6DOgW2vGwH20g+N0YiLfmiM\nTrQNk8MsTdXGHx0734bP34bDO41ysw0+/IMxO0zxuXDiBTBhCVjkChgRp0jEOBqz6TnYtBrqK8Hq\nAnU2zLjAuM5yAEML+nCOK9FKS0sjJSUlce9nSHaZh7hI4yFqtq1j66cfsNszkzcbRvHxPh/VjS2A\ncWnUhFwP00dlMqMogxmjMplelEGWO0l7K5EI7HjTCLCKDyG9EE79oXF9mK3/BzwH5XevpQF2vx8N\nqnfgYJlRbk83escTToMJp0P+NONwb9kq+OxFY0YST4ExcOXEC2FUyaBeAyf/b+MzKO1XvQ3KnoNN\nz0PNNuP88aQvGH8UqWXgSD/2PvqgtLQ0vnNcYngxpRWQd9JXqHJO4pTiYi6Jlh+q97FpXx2b9taz\neV8dG3Yf4a+ftt9fbFSWixmjMphRlNkWZsdlNGO8TCaY/AWYdKbxZf/2PfDqTfDufXDqdTDnO2D3\nDHYtuxfwGZccfP6OEVZ7SyESAovDOBR45m1GUBWd1PWv3ImnG49z7jV6nmWrYP0fjZ5Y9gQjwGZe\nBHlTBueziaGvrtIIqrLnjMPPmIyRwwuvMQ7Lx1y3ejxJcIk2BRlOzsxwcua0EW1lR5r8bN5nBNmm\nffVs3lvHa5vbjzPmpztiemVGmI3Odg2N68w6M5nav8x3vWcE2Ou3GTNynPIDmHtlwv9q7LNQ0DgM\n8/lbRljtWQtBn3HJQdHJcOr1Rv3HzO99b9HmhBO+ajyaa2HLS7DxWeMygnfugcJZxqUFM87vfgYT\nkVqaqo3zVZueNwZrgfG796W7YfrX+zZ93gCR4BJHle2xs2hKHoum5LWVNfgClO9vMMIs2jt7Z1s1\nobBx2DnTZWvrmU2P9swm5HqG1gwg4xcZjz1rjQB74w5Y86Dxl+S8q4zb5BwPkYhxJ4PP3zHOUe1a\n0z7RcMEJUHKFEVTjTklMnVxZcNK3jEf9fuM6uLJV8PpPjBCfsNgIseJzjW1TScNB41By5TrAZAR6\n0UlG73S4z8ziqzf+oCl7Dna+ZfTq86fBGbcZh5dzJw12DTuQ4BJ9lu60MW9CDvMmtB8m8AVCbDnQ\nMcz+uGYX/pBxrZnHbuGEooy2XtmMUZlMLkgb/LkZxy6AS18wZv14+x548y54/9ew4Psw/3sDcyH1\nkV3tgyk+fweaorOhZY+H6edFz1Od1rdpsfojo9AI6oXXtJ+/KHvWGIn5t/8HU88yDidO+dLwGczS\nKhwy/mCoWGvMoblnLdTuNtZZovfpCxnnfHFkGCHWGmSFsyFnYvKHWaAZtr5mDLLY+rrxebPGGofP\nZ1wAI6YP2blAJbhEQjhtFmaPyWL2mPa/0gOhMNsONrJ5Xx2b99WzaW8dz66vwOs37g5tt5opHpne\n1iubUZSJGpk+OLPmj54D33wW9n0Mb6+At34OH/zWCK8F34/vWH7DwfYe1edvQ+0eozxtBExcYpyj\nmnAaZI9LxCfpn7wpcMYtsORmY2LkslXGoaLyvxpf3MVfNSZFHr84OWclaWmAyo/aQ6pyffudCdJG\nGIde511lPBfOMr6wD5XD/k+MOxrs/wTWPdwxzEbOhKLZRpAVzTbucDDUwywUMHpUZc/Blr8ZbeAp\nMKZnO/EC4zKYIRpWsWRUYQobjLYLhSPsqmli0972MNu0t456nzGTmMVsYkpBGtOLMpkxKoMTCjMo\nLso4/vczO1BmnAP67C/GkPJ5/woLrwVP+yHTHtuvudYY0dc68q+q3Ch3Zhpf/K1Bla+G9pdEKAi7\n3oGNq4wA8zdA2kjjXNjMC40v7H7Wf0B/9yIRo/dUsc449LfnQzi0OXrbIROMmGEMbBkzH8bOh6xx\nvfscoQBUbWkPsn2fGDc+DfqM9fZ0KJwZ7Z1Fwyx38oAEfZ/aLxw2zlVtes74ffbWgCMTTjjX6FmN\nXzwkL5M42qhCCa4UNlTaLhKJUHmkue0wY+vIxtbh+QBjc9ycUJjB9KKMtkOOIzIcAz8I5OBn8O69\nsOkFYzDE3O/CKddBWkF7+/m9xiGn1pF/+z8xviStLhi3sH2IeuGs5OytQPSw0qvGX+pbXzPmqMyd\nYhxKPPGCPp8DSejvXtBvjHir+NDoTVWsg8YDxjp7utGbbg2pUXPAmZGYnwtGuFdt6dgzO7Cp/a4I\n9jTjLgatQVY42+jdxvl7cMz2i0SMupRFLwyu3xu91mqZ8e81ealx14UhTIJLdGuot92heh+b99fz\n2b7oY389n1c3ta3P8dg7hVkGE/LSsAzEIJCqrUaAla0yZhYpuYKqphD5jeXGF2bIb1zXMmqOMZhi\nwmnGYZch/uXQL81HjL/cy54zRmcSMa4LO/EiY9RZ+ohj7iKu372mGmMARWtI7dvQ3uvJGtceUmPm\nGwNcjvcfC6EgVOuOPbMDZe1hZvMYYRZ7mDFvap/q2WP7VW2NXhj8PNRsN34nJy81elZqGTiSZ2IB\nCS7RrWRsu8aWIFv217M5Jsz0gYa2QSBOmxk1Mhpm0VCbNjIDlz1BX141O4zrvz59mkgkjGnkiUZI\nTVwCYxcm1RdDQtTtjV7n86zx5WwyG21x4oUw7Ss99m56/bsXDhsXu7aGVMVa4wsZjNlBCme1B9Xo\neUN3OH8oCNVbO/XMyiDgNdbb3N30zKb2eAivQ/vVVhj/BpueM/bZeq3ViRcY5yYH6VqreElwiW4N\nl7YLhMLsqGpk814jyD6LXnfWet7MbIIJeR6mF2W29cxOKMwgNy2O3lDjIbZu38nU2QsS9CmGgUNb\njB5p2SrjHJPVadyleuZFMPmLHeaJ7PF3z+81LrKu+DD6WGfcWRzAldOxN1V0Ulwznwy66M1l24Js\n/6ewf2P7XcKtrm56ZgosVrZ+vIapgc+MXm/FWmP7USVGz2r614dugPeBzJwhhjWbxcy0kUbP6vxo\nWSQSYW9tc4eeWenuI7wYMxPIiAyHEWYxhxvHZLt7d71ZWgEhR83AfKBkVTANvnC7MZtH5UfRkYkv\nwGd/BmeWMdPCzIuMG4u2qtsbE1IfGj2GcPSWf3nKuGh6zHwYs8A4jzaUB7P0VfTmshQUw+zo/DXh\nkNGjjD3M+MmTsO4hY73VBTkTmVK1JXqtVbHR3jPON4bopwgJLjEsmUwmRme7GZ3t5kvTR7aV13r9\nbUFm9MzqeXtrVdvF0+kOK8WFRoidEO2ZTRmRhsOapIMqBoPJFB21N8+YbWHn28ahxLLnYMOjkDGK\nUemT4dWdUFdhvMfqMgZRnHq9EVSj5ybtIa64mC3GaNN8BbP+xSgLh4xD1K1BVlVOTe4c8pZ8z7jW\nKgVJcImUkuW2c8rkPE6Z3D6s3RcIsfVgQ1ugbd5X3+F6M5vFxOSC9A49s+LCBI5MG84sNpiy1Hj4\nm0C/AmXP4dy7EcbPMy5+HjPfOCQ2wDOKJy2zBfKnGo+Zxj35qsrLyRuR/If5+0uCS6Q8p83CzNFZ\nzBzdfvF0OHq92WcxA0He2VbF8xsq27bJ91iYVlTPpPw0JhWkMTk/jUkFHvLTjsMw/WRk9xgDBk68\ngB3D5PyqGBwSXEJ0w2w2MTE/jYn5aXxlZvukoocafG2HGEu37aW6OcCq9RU0RXtnABlOa1uQTS5I\nY1L0eXS2C+tgT3ElxDAgwSVEHxSkOylQTpaoAspHBiguLiYSiXCg3sf2Q41sP9TIjirj+a2tVawq\nbe+h2S1mJuR5mFTgifbOjFCbmO/BbZf/ikL0lvxvESJOJpOJwkwXhZkuFk/J77CuzhtgR3V7oO04\n1Ej5/gZe3XSAcMyVKKOyXB0ON7YGW67HLocdhehEgkuIAZTptnHy2GxOHttxlvmWYIjdNd4uvbSP\nPj9Mc6D9sGOW22aEWOthxwIPk/PTGZXtGpgZQoRIAr0KLqXUfOCXWuslMWUjgadjNpsN3Az8D1AJ\nbIuWf6C1viUhtRVimHBYLUwdkc7UER1vXBkOR9hX18yOqqYOgfaPLQd5Zn1FzPtbDzumtfXOJkcP\nOw7K7PpCHEfHDC6l1I+BS4Gm2HKt9QFgSXSbhcB/AQ8Dk4ANWutzE11ZIYY7s7n9+rPTp3Y87Fjr\n9bcFWWuwbdpbx8tl+2mdAMdkgtHZLiblpzE+18P4XDfj8zyMz/UwKts1+Pc/EyIBetPj2gF8A3i8\nu5VKKRPwa+CbWuuQUqoEGKWU+ifQDNygtdaJqrAQqSrLbadkXA4l4zpemOsLhPi8uqlDqO041Mi6\nzw+3XYsGxi1jRme7GBcNtHExwTY62yUXWYuk0au5CpVS44GntdZdJmZTSn0VOF9rfXn09WnACK31\nKqXUIuABrfXczu8rLS2NuN3ueOuPz+fD6Rxmd2c9TqTt4jPU2y8SiXDEF2JffZD9DQH21QfY1xBg\nf0OAvfUBvIH2//tmE+R7rBSl2yhMt1KUYaMo3XiMTLfisCa2pzbU226oS4X283q9AzpX4beAB2Ne\nrweCAFrr95RSRUopk9a6S0Im4gLE4TJR7GCQtotPMrdfJBLhiDfArpomdlU3savGy+4a4/n9yiZq\nvQ0dti/MdBqHHvPcHXps43Ld/RrKn8xtNxSkQvuVlpb2uC4RwTUHeD/m9X8ANcA9SqlZQEV3oSWE\nGDwmk4kcj50cj73LiEcwzqftrvGyq6ap7XlXdROvbz5ITZO/w7YF6Q7GR0Os9XzauFw343LdpB/v\nO1eLlNDn4FJKLQfStNYPKaXygfpOwfQL4Aml1Jcxel7fTkhNhRDHTZbbTpbbzqwxWV3W1fsC7IkN\ntWrj+a2tVVTFXHANkJdmb+uZjY95bmkJEYlE5Bo10S+9Ci6t9S5gQXT5yZjyKoxh8LHbHgG+nLgq\nCiGGkgynjRmjMpkxKrPLuqaWILtjDjsaz018sKOGFzbs7bCt54VKRmW7GJXloijL1bY8KrpckO6U\na9VEt+QCZCFEwngc1rZbwnTmC4TaDjuuL/+cgD2DvbXN7Ktt5uOKWmq9gQ7bW80mCrOcFGUaQTa6\nU8AVZbnkmrUUJcElhDgunDYLamQ6amQ6Y81HugwuaGoJsq+2mcraZvYeaW4Ltb1HmvlgRw0H630d\npskC41Bkaw+tNeBGxYRbpssmhyOHIQkuIcSQ4HFYmTIinSmdZhNpFQiFOVDn6xBoe2uNx5YDDfyj\n/BAtwXDHfdotRqhldQw0ORyZ3CS4hBBJwWYxMybHzZic7q//jEQi1DT5O4Ra5ZFoyNU280kPhyNH\nZjq7hNrobDdjc9wUZjlltpEhSIJLCDEsmEwm8tIc5KU5OtwUNFbs4cgOvbYjzazdUcOBTocjLWYT\nhZlOxua4GZPtZmyuEZxjsl2MzXGTI7P3DwoJLiFEyujt4cjKI81UHPZSccTLnsNeKg57+ceWQ1Q3\ntnTY3m23MDbH3dZDG5vjYkyOu63MZZfBIwNBgksIIaJiD0cunJTbZb3XH6TySDN7ajqGWsVhL2u2\nV3e4JQ1Afroj2lszemijc1oDzs2IDDm/1l8SXEII0Utuu7Xb29GAcY6tutFPxZH2MNsTfXy06wgv\nfrqvw2FIm6X1TgCutjBr7a2NyXaT6ZZZR3oiwSWEEAlgMpnIT3eQn+7odhotfzDM/rrmtjCrONx+\nOPJvZfu7DBzJcFqNc2rRw5CxvbVAKLVn0ZPgEkKI48BuNUenv/J0u77eF2jrqVUcbg84fdAY6u8P\ntQ/1N5tgVPaBDtNojc0x5oocm+Me9hdmS3AJIcQQkOG0Mb0ok+lFXafSCocjHGzwUXG4md01TWzY\nuocmk5vdNU28+Mk+6n3BDtsXZjpj5ofsOJu/x5H8X/vJ/wmEEGKYM5tNFGa6KMx0MW9CDjM8jR1m\nHqn1+tvnhqxunyPyjfKDVDd2nM0/L83R4Uai4/Lagy3TlRzn1SS4hBAiyWW57cx225ndzWz+Db5A\ndOLj1hn9jQmQ39texfMbOg7vz3bb2npmne+UPZSuWZPgEkKIYSz9KLP5N/tD7DncMdB21zSxPjoK\nMhIzBiTdYWVc7I1Ec9rvwVaQ7jiuoSbBJYQQKcplb5/4uLOWYKjtnFrsrWo2763j1U0HCMWM7XfZ\nLG03D50zLofvLpqAeQCvUZPgEkII0YXDamFyQRqTC9K6rAuEwuyrbe5wXm3P4SZ2VBnL3100YUDr\nJsElhBCiT2yW2KH9+cf958u0x0IIIZKKBJcQQoikIsElhBAiqZgikcGZ86q0tDS1J9sSQghxVCUl\nJd0OTRy04BJCCCH6Qw4VCiGESCoSXEIIIZKKBJcQQoikIsElhBAiqUhwCSGESCoSXEIIIZKKBJcQ\nQoikIsElhBAiqUhwCSGESCoSXEIIIZKKBJcQQoikIsElhBAiqUhwCSGESCoSXEIIIZKKBJcQQoik\nIsElhBAiqVh7s5FSaj7wS631kk7l5wI/BYLASq31w0opF/AEUAA0AJdrrasSWmshhBAp65g9LqXU\nj4FHAGenchvwAHAWcDpwlVJqBPBvQJnWejHwGHBboisthBAidfWmx7UD+AbweKfyYmC71voIgFLq\nPeA0YBFwT3SbV4Dbu9tpaWlppD8VFkIIkRpKSkpM3ZUfM7i01s8rpcZ3syoDqIt53QBkdipvLeuW\n2+0+1o8/Jp/Ph9PpPPaGogtpu/hI+/WftF18UqH9vF5vj+t6dY6rB/VAeszrdKC2U3lrWbeKi4vj\n+PGG8vLyhOwnFUnbxUfar/+k7eKTCu1XWlra47p4gqscmKKUygEaMQ4T3guMA84B1gHLgHfj+BlC\nCCFEB30OLqXUciBNa/2QUupG4DWMQR4rtdZ7lVK/Bx6NnvPyA8sTWmMhhBAprVfBpbXeBSyILj8Z\nU/5X4K+dtvUCFyauiivKHZoAABnVSURBVEIIIUQ7uQBZCCFEUpHgEkIIkVQkuIQQQiQVCa4YLS0t\nrFq1KiH72r59O5dccgkXX3wxN998M8FgMCH7FUKIVCfBFaOqqiphwXX//fdz44038vTTTwPwz3/+\nMyH7FUKIVBfPdVwD6vnSSp5dX3HM7bxeL+53erzGuYOL5ozh/JLRPa7/wx/+wPbt2/nNb35DWVkZ\njY2NhEIhrr/+ehYuXMg555zDnDlz2LZtG5mZmdx///09zv7x61//GovFgt/vp6qqirS0tF7VUQgh\nxNFJjyvG1VdfzeTJk2lqauKUU07hT3/6Ew8++CA/+clPiEQi+Hw+zj33XJ566ikmTpzIM8880+O+\nLBYLe/fu5Stf+QpHjhxh2rRpx/GTCCHE8DVke1znl4w+au+o1UBMfbJjxw7OPfdcAEaMGEFaWho1\nNTVYrVbmzp0LwMknn8w777xz1P2MGjWK119/nVWrVvGLX/yCX/7ylwmtpxBCpCLpccUwm82Ew2Em\nTZrE+vXrATh48CD19fVkZWURDAbZsmULYMyjNXny5B73dfXVV7Nr1y4APB4PZrM0tRBCJMKQ7XEN\nhtzcXAKBAA0NDezevZvXXnsNn8/HnXfeidVqNNXDDz/Mvn37KCoq4oYbbuhxX1dddRU333wzNpsN\nl8vFXXfddbw+hhBCDGsSXDEcDgd/+ctfjrrN3XffjcPhOOa+Tj755LYRhUIIIRJHgisO+/bt46ab\nbupSPnfuXK677rpBqJEQQgx/Elx98Oabb3Z4XVRUxOOPd74xtBBCiIEkIwaEEEIkFQkuIYQQSUWC\nSwghRFKR4BJCCJFUJLj64dJLL2XHjh09rv/oo4/aLlROhP/f3r0HyVUddh7/3ke/pntGgF4jLEAY\noSM5GCkRzxg53iCMROyFpRBe49gVWIipWHngoip2yiTGBThlGYgXryqJDV6Ig8tgUK3t2MTFrp2I\nl1kUEx4eHSJjjLCQhBCjmZ6Zfnf+uLdnrnpamtY8NLrq36eq6957zr13Th9E//rcV999991s2LCB\nq6++mp/+9KfTtl8RkThScM2ARx55hL17907Lvn7+85/z/PPP89BDD3HXXXdx++23T8t+RUTi6ti9\nHP75b8HPvjnhaqcOD8Ez2fb2+Zu/D6s+esjqjRs38olPfILzzjuPF198kS996UucdNJJDA4Osnfv\nXq655hquueaaw/6Jl156ia1bt/Lyyy+zdOlSnnvuOe6//36SySRLlizhC1/4At/73vd4/PHHGRoa\n4p133uFTn/oUl156acv9vec97+Hee+/FcRx27dpFT09Pe+9VROQ4NWFwGWNcYDOwEigC11trd4R1\nq4C/iax+AXAF8CzwCvBSWL7FWvuVaWz3jNiwYQNbtmzhvPPO49FHH+X8889n2bJlfPCDH2TPnj18\n/OMfnzC4zjrrLNasWcNll11GJpPhnnvuYcuWLeRyOe644w6+/e1v09XVxcjICN/4xjfYv38/GzZs\n4OKLLx59rFQz3/e5++67eeCBB7jllltm4q2LiMRGOyOuK4C0tfZCY8wFwJ3A5QDW2ueBDwAYYzYA\nv7bWPmaMWQt8y1r7x5Nu2aqPHnZ01PD6ND4dfs2aNWzatIn+/n6ee+45vv71r3PnnXfyox/9iFwu\nd8S/Yrxz506WLl06+ltc5557Lk888QQrV67k3HPPxXVd5s2bR09PD/v372fBggWH3NdNN93EDTfc\nwEc+8hHOOeccTj311Cm9VxGRuGrnHNdFwGMA1tpngHOaVzDGZIFbgT8Ni1YDq40x/2KMedgYs2ia\n2jujXNdl3bp1fP7zn2ft2rXcd999rFq1ii9/+cusW7eOer3e1n4cx6Fer7N48WJ+8YtfMDw8DMCz\nzz7L6aefDsDLL78MwL59+8jn88ydO7flvp5++mluvfVWIHiWou/7OI4z1bcqIhJb7Yy4eoADkeWq\nMca31kaHH/8DeNhauy9c3g5ss9Y+boz5GHAPcFXzjvv6+ibZ7DGFQmFa9tNw9tlns3nzZjZv3sze\nvXv52te+xne+8x2y2SzVapUXXniBoaEhXn31VUqlUst9LFy4kNtvv52bb76ZK6+8kquvvhrHcVi0\naBEf+tCH2Lp1Kzt37uSqq65ieHiY6667jldeeaXlvrq7u9m/fz+XX345tVqNtWvXks/nj8m+6zTq\nv8lT301Np/efM9EowhhzF/CMtfahcPkNa+3ipnV+Clxlrd0ZLncDw9baqjGmC3jRWntGdJtt27bV\nV69ePeU3MBM/JDnTHn30UV599VVuvvnmWW1HHPvuWKL+mzz13dTMRP/V63VK1RqFco1CucpIqcpI\nOXgVIvMjpWpQX64yUqoF9eWxsrnZFLd8aMWUjwxt27aN1atXt9xJOyOuJ4EPAw+F57hejFYaY+YA\nqUZohb4OPAI8BFwMbJtMw49lL7zwAps2bRpXvn79+gkv4Gjlq1/9ast7tO644w5OOeWUSbVRROKv\nXq9TrNQolmsUKkFAvPZOieLO/qYQqY6GyEipSqFycLAcFEQt1h8pV6m1dzbkIEnPJZ1wySQ9MgmP\nJfOy1Oswk2c02gmuLcAlxpinAAe41hjzaWCHtfa7wDLgtaZtPgPcZ4z5I2AIuH76mnxsOPvssyf9\nZPgrr7xyXNnGjRvZuHHjVJslIjMoOioplqvBtBJMG6HSGLEUylUKlWC9YiVS1qgP6wphXWN/0f0U\nK8G2rQ+MvXHYtnquQ1fCIx0GSibhkU64pBMeJ2WTZE4IyyL1maRHenTeJZPwSCUOrg/2E67ru/je\n0b8deMLgstbWgBubirdH6v8/wZWH0W1+CfyX6WigiBzfarU65VqNcrVOpRpMy9UalWqjPJyvRtap\n1SlXalRq49cft264/rh9H1Qerh/ur9gIlRbh1OY1WuM4DqT9IDxS/liIpBJBAJzQlRwr84NpOqxL\nNebDbffvfZOlp58aCZlIOIXziVkIlKPl2L0BWUTa1hgJjB4OKlXHPqQjH9qV5g/22sFhUGkKgIND\nYnxARLc/qL75743+3aCsWK5Qrb9GpVqb1OGpI5XwHHzXJeE5JDwXP5wmPBffbcw7+OHynEyCdHdq\nNFTSkdFKI1gOrovUN4VTKqxLeu60XRHc1zfIiuULp2VfU1YpQjEPxQEoDkJ6Dpx42oz+SQWXyFFQ\nrtbGzkOE4TJcGju/MBw5AT4cOfcwXKowUqqNzZerjJRrjDTmS2PzMxUAjQ9933NIhh/6jRDwIx/8\nvueQcF2SvktXyicZ2a4REH4YEAP977BwwTwSbjREwv36LonIugeHS2P9RvkhwihsS8Jz8FxnfGDU\nalArQ7UE1XBaKY7NOw54SfBT4KXAT4bT1MyevDlaajUoDwVBM/oaCAOouSycL+XHlxUHg/6KchPw\nub3gztyIT8Elx71qrU6pUqMUHhIqVcampdHl+lh50zrlao1ii3Xe3LuP9Ev/TqFUpliuMBJOi5Fp\nqVKhVCpTq9VwqeMyNnWcOl50mehynYwPmYRDxndI+w4LfEiH8+kuh4wPKR/SnhNOIeU5JD1IeuC7\nLp4LnufiOS5uGAau4+C5Lp7n4Lkevuvghsu+4+J6Dr7r4bnBvY2O0/gAcoKz3ADUw/lqUE41/EAP\nV5hg/rVfvcGSxdXWodEIk3Jkvtqivnn+oH20WjdSXzuyhwkcxE2An46EWfM0FQm9aPgdru5Q+0iP\nL/OSJPK7YHflECGTbx08B4XPYPDfcCJ+GlLdwSuZg1QP9CweK0t1QyosbyzPXTqjoQUKLjkKGqON\n5ktqx65oarqkdrSuKTyqNcpNAVSKhMn4oAmWj2Qk4lFlLgPMd/qZ7/SzwOlnPgeY7/SzyOlnvnMg\nLOsnTQnPaWPnycn3HTWgFL6OI0uOdAM3EX6QJ0c/vPESTdMwCFLd48u9w23fYr5ej4RhY1qESqlp\nWmhRVgoCYvjtFvsoBds0j1KO0NLD1jqRIMkF03QPzHlXWNYThlA0fJrWb6zjT+Uf78xRcHWCWg1K\ngwd/8yoMkH39FUbYScHJUPSyDDsZhukiT5qRqjMaNIXyIS6pPdS9HY2ro8L6yiSOYSU8h5TvkfSD\nwz3BNDhPEJ3PJBtlwWGsRLTeD+oSrkOOYXqqbzOnup9ceT+58j66Sm+TKb1NpriPVGEfycJb+IX9\nOC2+idZTc6jnFuDkFuJ0vxdyC9g3MMK8+QvAccHxgtGE4wYv1xubb/U6qH6ibSP1Levc8fU4QVm9\nzug36yOep831w/WOYP5Xr7/Oae8+s0VoHCJIjofDc1FtB2Mk7Bpl1RK73urn5NNNOBLqPjiEktnj\nr7+aKLiOZbVq8M2tEAz368UBSkMHKA4doDTUT3n4ANWRA9QKg9QLAzilQdziIF45j1/Jk6wMkawN\nkakNt9z9qQBPQ6ZF3Ug9SZ4M+XqaPBmGyDBYzzBMmoKbpeh2UfSzlL0sFT9L2c9RS2ap5YJvbfVk\nN07qBLxMjnQyOXr57Nhlud74svDy23S7V0RVipDfG772QH53ZD6cDu4JptXi+O29JOQWQm4BLDwD\ncheOLecWQndvMJ9dgJNI0/xR8FZfH/N0E+2kDJf64LQO7jvHGTt0OAkH+vo4uYP/7Sm4pkO9Hhwz\nHz2eXh498Vsp5Cnk+ynm+ykOHxgLm5EB6mEgOaU8XnmQRDlPopInWR0iXRsmUx856M84QCp8RQ3W\nM+TDYAnCJsOIu5Cim6Xo5yinclQSWerJ7uCV6sFL9zAwUuSUuVmyFMgyQld9hEw9+Nup6jDJ6hAL\nqkOcXM7jVYZwS3mc0js4jWPl5QKU2+ifRFfTcfLuFsu5pkMYwYOJyb/VFEhhKA3uhkJ/67/XNRdy\nYeicdsZYEOUWQvfCsXBKn3DcfzMVOR7FO7gKA+R2PQnsGBcYVMMgqZWb5huvVsutwudw+43UHYIP\n5MJXVK3ukCc9GjSDdDHsdlFwF1P2s0HY+DmqyRy1ZHDM2U1346Z78LvmkMjOIdl1AplsD9l0kmzK\npyflc3Lapyvh4boTfyBP+bEx1XLrk76HKhtdzkP/zuDEcaNsomP+ia6x8Jm3DJasGQugxsgotxCy\n84NDSyJy3Ip3cP3ki5zyzOb21/eS4Une8BWd95Lg+mPH1P0kuNlw2R/dtuYmGCjB/kKdfSM13hqu\nsSdfY3+hThmfCh5uIsm8nizz5+RIZ+fgZnrwM0HgJLNzSGVPoCs3h1wYOHNTHinfm7l+mileArpO\nCl5T1bgXZPRcXB6oj4VTMqfRkYgAcQ+ui/+KX+bO4fQzzmwdPqPhlAxPWrf/wVev19kzUGT77gHs\n7kG27x5k+68H+cXePKVqDQDfdThjfg6ztBvT281vLerG9PZw8py0fnrkSDWO92db/7yLiEhDvIMr\nkaYw9z2waGonKYeKFeyeQezu4NX35gB2zyD9w2OHAHt70pjebt6/bB7Le7tZ3tvDu+dn4zlSEhGJ\nsXgH1xGq1uq89vYQ298cxO4eCEZRuwd5ff/YVXfZpMey3m7Wn9XL8t4eTG83y3u7OaHr2LyfQUSk\n0xy3wfXWYDE8xNcIqAH+Y0+eYiU4zOc6sGRelve+aw5XrV48OopafGKmrQsbRERkdsQ+uAqVGi+8\n0R+E05uD2D0DbH9zkLeHxq5Sm5dLsby3m49fcBqmt5sVi3pYuiBHOqHDfCIicRPr4Lr3iV9y2/df\nox7+HFjKdzG93fzu8gUsX9TD8t7gool5ucnd5CciIseeWAfX6tNO5GMrT+R9Z52O6e3mtLlZPB3m\nExE5rsU6uFadcgKpVSeyYsWi2W6KiIgcJcfvT2SKiMhxScElIiKxouASEZFYmfAclzHGBTYDK4Ei\ncL21dkek/ivARcBgWHQ5kAAeJPjFjF3Atdba1r+tISIicgTaGXFdAaSttRcCnwHubKpfDVxqrf1A\n+DoA/CXwoLV2DfAz4JPT2WgREelc7QTXRcBjANbaZ4BzGhXhaOxM4O+NMU8aY65r3gb4IbB22los\nIiIdrZ3L4XuAA5HlqjHGt9ZWgCxwD3AX4AE/NsY817TNIDCn1Y77+vom2+5RhUJhWvbTidR3U6P+\nmzz13dR0ev+1E1wDQHdk2Q1DC2AY+Erj/JUx5v8RnAtrbDMSTlv+VO2UfsQwNOUfQ+xg6rupUf9N\nnvpuajqh/7Zt23bIunYOFT4JXAZgjLkAeDFStwx40hjjGWMSBIcI/y26DbAe2HrkzRYRERmvnRHX\nFuASY8xTgANca4z5NLDDWvtdY8w/AM8AZeABa+3LxpjbgPuNMTcA+4BrZqj9IiLSYSYMLmttDbix\nqXh7pH4TsKlpmz3AuulooIiISJRuQBYRkVhRcImISKwouEREJFYUXCIiEisKLhERiRUFl4iIxIqC\nS0REYkXBJSIisaLgEhGRWFFwiYhIrCi4REQkVhRcIiISKwouERGJFQWXiIjEioJLRERiRcElIiKx\nouASEZFYUXCJiEisKLhERCRWFFwiIhIr/kQrGGNcYDOwEigC11trd0TqbwL+e7j4A2vtrcYYB3gD\n+I+w/Glr7WenteUiItKRJgwu4Aogba290BhzAXAncDmAMebdwMeA84Ea8IQxZgswDPybtfbDM9Ns\nERHpVO0cKrwIeAzAWvsMcE6kbiewzlpbtdbWgQRQAFYD7zLG/NgY8wNjjJnmdouISIdqZ8TVAxyI\nLFeNMb61tmKtLQP7wkODm4CfWWtfMcb0Al+01j5sjLkI+CZwbvOO+/r6pvwGCoXCtOynE6nvpkb9\nN3nqu6np9P5rJ7gGgO7IsmutrTQWjDFp4D5gEPijsPg5oAJgrX3CGHOyMcYJR2WjVqxYMZW2A0H4\nTcd+OpH6bmrUf5OnvpuaTui/bdu2HbKunUOFTwKXAYTnuF5sVIQjrf8D/Lu19pPW2mpY9VfAn4Xr\nrAR2NoeWiIjIZLQz4toCXGKMeQpwgGuNMZ8GdgAe8DtAyhizPlz/s8BfA980xvwewcjrD6a74SIi\n0pkmDC5rbQ24sal4e2Q+fYhNf2+yjRIRETkU3YAsIiKxouASEZFYUXCJiEisKLhERCRWFFwiIhIr\nCi4REYkVBZeIiMSKgktERGJFwSUiIrGi4BIRkVhRcImISKwouEREJFYUXCIiEisKLhERiRUFl4iI\nxIqCS0REYkXBJSIisaLgEhGRWFFwiYhIrCi4REQkVvyJVjDGuMBmYCVQBK631u6I1N8AfBKoALdZ\na79vjJkHPAhkgF3Atdba4Rlov4iIdJh2RlxXAGlr7YXAZ4A7GxXGmF7gT4D3AZcCXzTGpIC/BB60\n1q4BfkYQbCIiIlPWTnBdBDwGYK19BjgnUnce8KS1tmitPQDsAM6ObgP8EFg7bS0WEZGONuGhQqAH\nOBBZrhpjfGttpUXdIDCnqbxRNk5fX98RN7hZoVCYlv10IvXd1Kj/Jk99NzWd3n/tBNcA0B1ZdsPQ\nalXXDfRHykciZeMMD0/Paa/p2k8nUt9Njfpv8tR3U9PJ/ddOcD0JfBh4yBhzAfBipO5Z4HZjTBpI\nASuAl8JtLgP+N7Ae2Nq809WrVztTarmIiHQkp16vH3aFyFWFZwMOcC1BKO2w1n43vKrwDwnOl91h\nrX3EGLMQuJ9gtLUPuMZaOzRzb0NERDrFhMElIiJyLGnnUOExaaL7y+TQjDEJ4D5gCcEh3tustd+d\n1UbFjDFmAbANuMRau3222xMnxpjPAv8VSAKbrbX3znKTYiH8//Z+gv9vq8ANnfpvL85Pzjjk/WUy\nod8H3g7vs1sHfHWW2xMr4QfI3xFcfCRHwBjzAeC3Ce79/B3glFltULxcBvjW2t8GvgDcPsvtmTVx\nDq7D3V8mh/cwcEs47xA89UTa92XgbwmeCiNH5lKCC7y2AN8Dvj+7zYmVVwA/PNrUA5RnuT2zJs7B\n1fL+stlqTJxYa/PW2kFjTDfwHeBzs92muDDG/AHwlrX2n2e7LTE1j+BL5gbgRuAfjTG6wrg9eYLD\nhNuBrwH/c1ZbM4viHFyHu79MJmCMOQX4MfAP1toHZ7s9MXIdcIkx5ifAKuCB8NFn0p63gX+21pas\ntRYoAPNnuU1xcRNB3y0jOLd/f3grUseJ8wjlcPeXyWGEtyv8CNhorf2/s92eOLHWvr8xH4bXjdba\n3bPXoth5AvhTY8xdwCIgSxBmMrF3GDs8uB9IAN7sNWf2xDm4thB8832KsfvLpD1/AZwI3GKMaZzr\nWm+t1cUGMqPCX494P8HDC1zgU9ba6iw3Ky7uBu4zxmwluCLzLzr1/ljdxyUiIrES53NcIiLSgRRc\nIiISKwouERGJFQWXiIjEioJLRERiRcElEhPGmJ8YY5bPdjtEZpuCS0REYkX3cYnMgPAJ8n8LnEnw\nBfFzBD/DsxX4DYInH3wUKAHfAN5N8BSEu6y13zbGnA/8Tbjtr4GPAT8E3gQWEjxx4qPW2leP4tsS\nOSZoxCUyM64H9oWPiLoc+F9AF/CP1tqLCB6U+snw9Vb4UxVrgduMMfMIfjblOmvt+cA/ASvC/f6T\ntfZ3CULsqqP5hkSOFXF+5JPIsey9wJpw5ATB/2t1a+2/hstPAesJflLmcYDwif0/B84Aeq21fWH5\nvQDGGAh+vBJgN6CH+0pH0ohLZGZsB75lrf0AQUA9DHjGmJVh/fuAl4E+YA1A+DMz7wV+CewyxpwZ\nlv+5Mea/hdvp2L50PAWXyMz4O2C5MeZfCEZXvwJqwJ8bY54A3hWu8/fA3LDsJ8Ct1tq9BIcQ7wu3\n/03gB0f/LYgcm3RxhshRYox5DVhurS3MclNEYk0jLhERiRWNuEREJFY04hIRkVhRcImISKwouERE\nJFYUXCIiEisKLhERiRUFl4iIxMp/Amw1DA5HxVLOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x131781080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Long Short-Term Memory (LSTM) Networks\n",
    "- LSTM are a special kind of RNN (invented in 1997)\n",
    "- State of the art for many sequence to sequence mapping and text generation tasks\n",
    "- Adds an explicit \"memory\" unit\n",
    "- Augment RNNs with a few additional **Gate Units**\n",
    "  - Gate Units control how long/if events will stay in memory\n",
    "  - **Input Gate**: If its value is such, it causes items to be stored in memory\n",
    "  - **Forget Gate**: If its value is such, it causes items to be removed from memory\n",
    "  - **Output Gate**: If its value is such, it causes the hidden unit to feed forward (output) in the network\n",
    "- Here's what it looks like:\n",
    "<img  src=\"img/LSTM.png\" style=\"height:300px\"/>\n",
    "- We won't go much further in theory, just know this is state of the art (ish)\n",
    "  - And you can do it!  Watch!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## LSTM Example: Sentiment Analysis\n",
    "- Here is some code to train sentiment analysis on IMDB reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 326,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-11-08T13:30:16.061127Z",
     "start_time": "2017-11-08T13:22:25.026554Z"
    },
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n",
      "25000 train sequences\n",
      "25000 test sequences\n",
      "Pad sequences (samples x time)\n",
      "X_train shape: (25000, 40)\n",
      "X_test shape: (25000, 40)\n",
      "Build model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/ipykernel_launcher.py:23: UserWarning: The `dropout` argument is no longer support in `Embedding`. You can apply a `keras.layers.SpatialDropout1D` layer right after the `Embedding` layer to get the same behavior.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train...\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/10\n",
      "25000/25000 [==============================] - 68s 3ms/step - loss: 0.5990 - acc: 0.6782 - val_loss: 0.4787 - val_acc: 0.7779\n",
      "Epoch 2/10\n",
      "25000/25000 [==============================] - 65s 3ms/step - loss: 0.4289 - acc: 0.8046 - val_loss: 0.4229 - val_acc: 0.8037\n",
      "Epoch 3/10\n",
      "25000/25000 [==============================] - 64s 3ms/step - loss: 0.3838 - acc: 0.8296 - val_loss: 0.4231 - val_acc: 0.8047\n",
      "Epoch 4/10\n",
      "25000/25000 [==============================] - 236s 9ms/step - loss: 0.3638 - acc: 0.8408 - val_loss: 0.4323 - val_acc: 0.8023\n",
      "25000/25000 [==============================] - 26s 1ms/step\n",
      "Test score: 0.432257632713\n",
      "Test accuracy: 0.80232\n"
     ]
    }
   ],
   "source": [
    "# Set parameters\n",
    "max_features = 4000\n",
    "maxlen = 40  # cut texts after this number of words (among top max_features most common words)\n",
    "\n",
    "# Load data (Keras utility)\n",
    "print('Loading data...')\n",
    "(X_train,\n",
    " y_train), (X_test,\n",
    "            y_test) = keras.datasets.imdb.load_data(num_words=max_features)\n",
    "print(len(X_train), 'train sequences')\n",
    "print(len(X_test), 'test sequences')\n",
    "\n",
    "# Pad Short sentences\n",
    "print('Pad sequences (samples x time)')\n",
    "X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=maxlen)\n",
    "X_test = keras.preprocessing.sequence.pad_sequences(X_test, maxlen=maxlen)\n",
    "print('X_train shape:', X_train.shape)\n",
    "print('X_test shape:', X_test.shape)\n",
    "\n",
    "# Build our model!\n",
    "print('Build model...')\n",
    "model = keras.Sequential([\n",
    "    keras.layers.Embedding(max_features, 128, dropout=0.2),\n",
    "    keras.layers.Dropout(.2),\n",
    "    # Here's the LSTM magic!\n",
    "    keras.layers.LSTM(128, dropout=0.2, recurrent_dropout=0.2\n",
    "                      ),  # try using a GRU instead, for fun\n",
    "    keras.layers.Dense(\n",
    "        1, activation=\"sigmoid\")  # Sigmoid for binary classification\n",
    "])\n",
    "\n",
    "# try using different optimizers and different optimizer configs\n",
    "model.compile(\n",
    "    loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "print('Train...')\n",
    "history = model.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    batch_size=512,\n",
    "    epochs=10,\n",
    "    validation_data=(X_test, y_test),\n",
    "    callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=2)])\n",
    "# Early stoping monitors one of the metrics and stops the process early if it's no longer increasing\n",
    "score, acc = model.evaluate(X_test, y_test, batch_size=batch_size)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolutional Neural Networks (CNN)\n",
    "- You've seen CNNs a little\n",
    "- They'll be discussed more next week with images\n",
    "- Had great success with images\n",
    "- But, they do some nice things with NLP too!..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## CNN Example: Sentiment Analysis Revisited\n",
    "- Here's the same Sentiment task with a CNN + LSTM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Embedding\n",
    "max_features = 20000\n",
    "maxlen = 100\n",
    "embedding_size = 128\n",
    "\n",
    "# Convolution\n",
    "filter_length = 5\n",
    "nb_filter = 64\n",
    "pool_length = 4\n",
    "\n",
    "# LSTM\n",
    "lstm_output_size = 70\n",
    "\n",
    "# Training\n",
    "batch_size = 30\n",
    "nb_epoch = 2\n",
    "\n",
    "'''\n",
    "Note:\n",
    "batch_size is highly sensitive.\n",
    "Only 2 epochs are needed as the dataset is very small.\n",
    "'''\n",
    "\n",
    "# Load data\n",
    "print('Loading data...')\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)\n",
    "print(len(X_train), 'train sequences')\n",
    "print(len(X_test), 'test sequences')\n",
    "\n",
    "# Pad sentences\n",
    "print('Pad sequences (samples x time)')\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=maxlen)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=maxlen)\n",
    "print('X_train shape:', X_train.shape)\n",
    "print('X_test shape:', X_test.shape)\n",
    "\n",
    "print('Build model...')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, embedding_size, input_length=maxlen))\n",
    "model.add(Dropout(0.25))\n",
    "# Convolution!\n",
    "model.add(Convolution1D(nb_filter=nb_filter,\n",
    "                        filter_length=filter_length,\n",
    "                        border_mode='valid',\n",
    "                        activation='relu',\n",
    "                        subsample_length=1))\n",
    "model.add(MaxPooling1D(pool_length=pool_length))\n",
    "model.add(LSTM(lstm_output_size))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print('Train...')\n",
    "model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch,\n",
    "          validation_data=(X_test, y_test))\n",
    "score, acc = model.evaluate(X_test, y_test, batch_size=batch_size)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bidirectional Recurrent Neural Networks (RNN)\n",
    "- Bidirectionall RNNs simply connect in both directions\n",
    "- Thus, output can be dependent on both future and past inputs\n",
    "- Good for context around a word, for instance\n",
    "  - e.g. Named Entity Recognition, is this a \"person\" token?\n",
    "<img  src=\"brnn.jpg\" style=\"width:70%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How do you feel about all this sentiment?\n",
    "- One more time on the sentiment, now with a BRNN!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "max_features = 20000\n",
    "maxlen = 100  # cut texts after this number of words (among top max_features most common words)\n",
    "batch_size = 32\n",
    "\n",
    "print('Loading data...')\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)\n",
    "print(len(X_train), 'train sequences')\n",
    "print(len(X_test), 'test sequences')\n",
    "\n",
    "print(\"Pad sequences (samples x time)\")\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=maxlen)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=maxlen)\n",
    "print('X_train shape:', X_train.shape)\n",
    "print('X_test shape:', X_test.shape)\n",
    "y_train = np.array(y_train)\n",
    "y_test = np.array(y_test)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 128, input_length=maxlen))\n",
    "# Bidirectional LSTM!!!\n",
    "model.add(Bidirectional(LSTM(64)))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# try using different optimizers and different optimizer configs\n",
    "model.compile('adam', 'binary_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "print('Train...')\n",
    "model.fit(X_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          nb_epoch=4,\n",
    "          validation_data=[X_test, y_test])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Onward to AWS!\n",
    "- Now that you saw how long those CPUs can take...\n",
    "- Let's go to GPUs!\n",
    "- That means AWS..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Setting up a Deep Learning AWS GPU Instance\n",
    "- We're going to use the **fantastic** EC2 Image that a former student of mine set up\n",
    "- It has all sorts of goodies preconfigured for your Deep Learning joy\n",
    "- You can find his repo [here](https://github.com/Miej/GoDeeper)\n",
    "- Now follow along with me as we venture Beyond the Wall!\n",
    "- **Caution**: This will cost $.  If you don't have your AWS credits, be wary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Launching the Instance\n",
    "1. Proceed to the AWS EC2 Console\n",
    "2. Click \"Launch Instance\"\n",
    "3. Click Community AMIs\n",
    "4. Search for \"3e22685e\" (make sure you're in the US West (N California) zone or it won't show up)\n",
    "5. Click Select\n",
    "6. Select the g2.2xlarge instance and click \"Next: Configure Instance Details\"\n",
    "7. Click thru the defaults until Step 5: Tag Instance, give it a name \"GPU\"\n",
    "8. Click thru to Step 6: Configure Security Group.  In the rules, select All TCP and Address \"My IP\", \"Review and Launch\"!\n",
    "9. Choose the keypair you should already have and let 'er rip!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Configuration\n",
    "1. Log onto your server via:\n",
    "  - `ssh -i <your_keypair.pem> icarus@<your_public_ip>`\n",
    "  - Password: `changetheworld`\n",
    "2. Log out\n",
    "3. In EC2 Console, reboot your instance (for init scripts to run)\n",
    "4. Log back in again (#1 above) and let's go!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Start a Jupyter Notebook (on your server)\n",
    "- You know how to do this!:\n",
    "  - `jupyter notebook`\n",
    "- If you want it to be open to the internet, you'll need to follow these instructions:\n",
    "  - [Running Jupyter Notebook Server in AWS](http://jupyter-notebook.readthedocs.io/en/latest/public_server.html)\n",
    "- In your local browser, browse to:\n",
    "  - `https://<your_public_ip>:8888`\n",
    "  - Note, you'll probably have to `pip install gensim` on the server\n",
    "- And let's begin!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## AWS LSTM + CNN Example: Sentiment 1 more time!\n",
    "- Let's marvel at how blazing fast the same example runs on your GPU instance!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Embedding\n",
    "max_features = 20000\n",
    "maxlen = 100\n",
    "embedding_size = 128\n",
    "\n",
    "# Convolution\n",
    "filter_length = 5\n",
    "nb_filter = 64\n",
    "pool_length = 4\n",
    "\n",
    "# LSTM\n",
    "lstm_output_size = 70\n",
    "\n",
    "# Training\n",
    "batch_size = 30\n",
    "nb_epoch = 2\n",
    "\n",
    "'''\n",
    "Note:\n",
    "batch_size is highly sensitive.\n",
    "Only 2 epochs are needed as the dataset is very small.\n",
    "'''\n",
    "\n",
    "print('Loading data...')\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)\n",
    "print(len(X_train), 'train sequences')\n",
    "print(len(X_test), 'test sequences')\n",
    "\n",
    "print('Pad sequences (samples x time)')\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=maxlen)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=maxlen)\n",
    "print('X_train shape:', X_train.shape)\n",
    "print('X_test shape:', X_test.shape)\n",
    "\n",
    "print('Build model...')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, embedding_size, input_length=maxlen))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Convolution1D(nb_filter=nb_filter,\n",
    "                        filter_length=filter_length,\n",
    "                        border_mode='valid',\n",
    "                        activation='relu',\n",
    "                        subsample_length=1))\n",
    "model.add(MaxPooling1D(pool_length=pool_length))\n",
    "model.add(LSTM(lstm_output_size))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print('Train...')\n",
    "model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch,\n",
    "          validation_data=(X_test, y_test))\n",
    "score, acc = model.evaluate(X_test, y_test, batch_size=batch_size)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## LSTM Example: Text Generation\n",
    "- Finally, a different example!\n",
    "- We're going to do **Text Generation** with **LSTM**\n",
    "- We'll watch our model start spitting out words of Nils's favorite philosopher in real time!\n",
    "  - ps, it's Friedrich Nietzsche\n",
    "- As we go through epoch's, the ability to generate Nietzcheian sentences will get better and better!\n",
    "- Let's go!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "path = get_file('nietzsche.txt', origin=\"https://s3.amazonaws.com/text-datasets/nietzsche.txt\")\n",
    "text = open(path).read().lower()\n",
    "print('corpus length:', len(text))\n",
    "\n",
    "chars = sorted(list(set(text)))\n",
    "print('total chars:', len(chars))\n",
    "char_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "indices_char = dict((i, c) for i, c in enumerate(chars))\n",
    "\n",
    "# cut the text in semi-redundant sequences of maxlen characters\n",
    "maxlen = 40\n",
    "step = 3\n",
    "sentences = []\n",
    "next_chars = []\n",
    "for i in range(0, len(text) - maxlen, step):\n",
    "    sentences.append(text[i: i + maxlen])\n",
    "    next_chars.append(text[i + maxlen])\n",
    "print('nb sequences:', len(sentences))\n",
    "\n",
    "print('Vectorization...')\n",
    "X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)\n",
    "y = np.zeros((len(sentences), len(chars)), dtype=np.bool)\n",
    "for i, sentence in enumerate(sentences):\n",
    "    for t, char in enumerate(sentence):\n",
    "        X[i, t, char_indices[char]] = 1\n",
    "    y[i, char_indices[next_chars[i]]] = 1\n",
    "\n",
    "\n",
    "# build the model: a single LSTM\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(LSTM(128, input_shape=(maxlen, len(chars))))\n",
    "model.add(Dense(len(chars)))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "optimizer = RMSprop(lr=0.01)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer)\n",
    "\n",
    "\n",
    "def sample(preds, temperature=1.0):\n",
    "    # helper function to sample an index from a probability array\n",
    "    preds = np.asarray(preds).astype('float64')\n",
    "    preds = np.log(preds) / temperature\n",
    "    exp_preds = np.exp(preds)\n",
    "    preds = exp_preds / np.sum(exp_preds)\n",
    "    probas = np.random.multinomial(1, preds, 1)\n",
    "    return np.argmax(probas)\n",
    "\n",
    "# train the model, output generated text after each iteration\n",
    "for iteration in range(1, 60):\n",
    "    print()\n",
    "    print('-' * 50)\n",
    "    print('Iteration', iteration)\n",
    "    model.fit(X, y, batch_size=128, nb_epoch=1)\n",
    "\n",
    "    start_index = random.randint(0, len(text) - maxlen - 1)\n",
    "\n",
    "    for diversity in [0.2, 0.5, 1.0, 1.2]:\n",
    "        print()\n",
    "        print('----- diversity:', diversity)\n",
    "\n",
    "        generated = ''\n",
    "        sentence = text[start_index: start_index + maxlen]\n",
    "        generated += sentence\n",
    "        print('----- Generating with seed: \"' + sentence + '\"')\n",
    "        sys.stdout.write(generated)\n",
    "\n",
    "        for i in range(400):\n",
    "            x = np.zeros((1, maxlen, len(chars)))\n",
    "            for t, char in enumerate(sentence):\n",
    "                x[0, t, char_indices[char]] = 1.\n",
    "\n",
    "            preds = model.predict(x, verbose=0)[0]\n",
    "            next_index = sample(preds, diversity)\n",
    "            next_char = indices_char[next_index]\n",
    "\n",
    "            generated += next_char\n",
    "            sentence = sentence[1:] + next_char\n",
    "\n",
    "            sys.stdout.write(next_char)\n",
    "            sys.stdout.flush()\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Additional Excitement: `seq2seq`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `seq2seq` Example: Question Answering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `seq2seq` Use Case: Machine Translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Other Successful Deep Learning Applications\n",
    "- Machine Translation\n",
    "- Named Entity Recognition\n",
    "- Image Processing (next week!)\n",
    "- Image + Video Captioning\n",
    "- Chatbots\n",
    "- Text Generation\n",
    "- Question Answering"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "latex_envs": {
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0
  },
  "livereveal": {
   "height": "100%",
   "margin": 0,
   "maxScale": 1,
   "minScale": 1,
   "scroll": true,
   "start_slideshow_at": "selected",
   "theme": "sky",
   "transition": "zoom",
   "width": "100%"
  },
  "toc": {
   "nav_menu": {
    "height": "369px",
    "width": "252px"
   },
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {
    "height": "457px",
    "left": "0px",
    "right": "968px",
    "top": "130px",
    "width": "214px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}