Cifar10 - Classifying with a Deep CNN

Cifar10 - Classifying with a Deep CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifing Cifar10 images with a Deep CNN\n",
    "\n",
    "CIFAR10 is a subset of IMAGENET in which each image can be classified in one of 10 classes:\n",
    "- airplane\n",
    "- automobile\n",
    "- bird\n",
    "- cat\n",
    "- deer\n",
    "- dog\n",
    "- frog\n",
    "- horse\n",
    "- ship\n",
    "- truck\n",
    "\n",
    "I will try to build a DeepCNN using Tensorflow in order to solve the problem, taking inspiration from AlexNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert Data to TFRecord\n",
    "\n",
    "From the website is possible to download a \"pickled\" python dictionary with the data or the binary version. I downloaded the latter in order to convert it to a standard Tensorflow format: **TFRecord**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filenames = [os.path.join(\"cifar_tensorflow/cifar10_data/cifar-10-batches-bin/\", 'data_batch_%d.bin' % i)\n",
    "             for i in range(1, 6)]\n",
    "label_bytes = 1  # 2 for CIFAR-100\n",
    "height = 32\n",
    "width = 32\n",
    "depth = 3\n",
    "image_bytes = height * width * depth\n",
    "\n",
    "record_bytes = label_bytes + image_bytes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "classes = [\"airplane\",\n",
    "\"automobile\",\n",
    "\"bird\",\n",
    "\"cat\",\n",
    "\"deer\",\n",
    "\"dog\",\n",
    "\"frog\",\n",
    "\"horse\",\n",
    "\"ship\",\n",
    "\"truck\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting Training Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "labels = []\n",
    "images = []\n",
    "for fname in filenames:\n",
    "    with open(fname, \"br\") as f:\n",
    "        string = f.read(record_bytes)\n",
    "        while string:\n",
    "            labels.append(string[:label_bytes])\n",
    "            images.append(string[label_bytes:])\n",
    "            string = f.read(record_bytes)\n",
    "length = len(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "an example image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHfhJREFUeJztnVmMpNd13/+n1t5npqdn6Vk0CzWmQS0c0Q2aliiaNC2D\nFhRIDBDCehD4IHj8YAsR4jwQdBApeVKcSLJgJwpGFmM6UGQpkQiNEyUxRRggbMk0m9twyKHEbYaz\ndE/P0j291/adPFQxGLbu/3ZNL9VD3f8PGEz1PXW/e+t+deqruv/vnGPuDiFEeuQ2egJCiI1Bzi9E\nosj5hUgUOb8QiSLnFyJR5PxCJIqcX4hEkfMLkShyfiESpbCazmZ2H4CvAcgD+HN3/1Ls+QObNvu2\n7cPEyu80NAt/RuVyRvt45HMtdk+jgR/TSEfeY5nRLDb/FR0RRu/YjIwVOWD0/s/4C7/+wdaBtR4t\nPv2VjcZ6xYcKWy9PnMfM9GRbZ2bFzm9meQD/EcDHAJwF8LSZHXP3l1mfbduH8aU/eSRoy7KMjtVd\nLgfbS11dtE+WD/cBgLrzD4YC8tSWb4Tbi3zq0XeLF/g8auyTBvE3Ra5BrF6kfeo1fsRGjrxoYEXO\nH7udPHqreWSsLIvMn3SMfrhG5hF7nzYakbWKjUfa69G1Cs/j3/6LB9oedzVf+28H8Jq7v+HuVQB/\nBeCTqzieEKKDrMb5dwM4c83fZ1ttQoh3Aeu+4WdmR8xs1MxGp69OrvdwQog2WY3znwOw95q/97Ta\n3oG7H3X3EXcfGdi0ZRXDCSHWktU4/9MADpnZATMrAfgdAMfWZlpCiPVmxbv97l43sz8A8H/RlPoe\ncfeXluuXkV3bQpnvRlez8C7q3NUZ2qfYy7eH88VuaoPzfhnZOa5HduYbizVqW7y6QG2lLq5WNMB3\nnGcXZoPtOePH6+vdRG0eGSuL7G4bkTFXusseWeLobj87ZzFhIbajH5tjbLefrQcAZGRVshWqDu2y\nKp3f3X8I4IernoUQouPoDj8hEkXOL0SiyPmFSBQ5vxCJIucXIlFWtdt/vTSyBqbnwlJUrcYlsUsX\nLwfbz56boH3yXb3U1tfPbzYq57gkxlTAap3PPavVqW1+JrwWANBd5PNAjss8M9Ww/Fmtcqnp4IFD\n1Pbem/ZRW3cssIpIUVGJKhK84xFjFtMBWZzTSgOMVkhM6suR15ZFZNa1QFd+IRJFzi9Eosj5hUgU\nOb8QiSLnFyJROrrbPzs3hx//w0+Ije985xAO+lmo8F3ZxUZYIQCAYonb8hn/PGyQDdtF5zv6jchO\ndG+J75Z3Gz81XWWeaqyRqwbb5+a4IjF6/Dlqm7h0ntoOHjhAbUNDQ8H27p4e2sdj6bgiQTMZSWkF\nAMbOZ6dzCcaChVgQ1AoCe65HqdCVX4hEkfMLkShyfiESRc4vRKLI+YVIFDm/EInS2cCeRoap2XDe\nOo/kzjMSnVEo8bx/PRGpLJ/jthJK1LaIsNxUj3yGzszPUdvCHLeVjct5fc6DfvLkpRXLPG/h4uwi\ntb1+5ucSMv9/To+NU9vmgXBewL179tA+24a28uNt4cFYhVykyhKRAVcavMMKIgE8X+By47HqO/Ec\nfquXKnXlFyJR5PxCJIqcX4hEkfMLkShyfiESRc4vRKKsSuozs1MAZgA0ANTdfST2/MwdC9WwrFEs\nxqZCop4aPFLNwW2Wj5RViigo1VpYEqtFpt7f00dtM9Pz1DZd5aW8KpEIsVIpLFX2l/gLy+e5vDlX\nr/B+kQjIyqWrwfapKR692dvH5cjh4V3UdtOBg9TWVwrLomWyTkA8n2QtklbPwSXHWOQhkwFjaiST\nHGO5DpeyFjr/Pe5+aQ2OI4ToIPraL0SirNb5HcCPzOwZMzuyFhMSQnSG1X7tv9Pdz5nZdgCPm9kr\n7v7ktU9ofSgcAYCu3oFVDieEWCtWdeV393Ot/ycAPAbg9sBzjrr7iLuPlLr4ho4QorOs2PnNrNfM\n+t9+DOC3AJxYq4kJIdaX1Xzt3wHgsVYZogKA/+bu/yfWIXPHQiUsl1Vq/HOIlTrqipSLisU8RQII\no6WfmG0ukny0q5sPVi5GEnHWeL/FCpcB60ai2CKvqxSJiotfHvgxC4XwMWPzmJnn63j11ZPUduky\nF5v6u8LRhXt28+jCLZEIwlIkOjJWbyyr8ySvdaICxqJFGx6Wqzsi9bn7GwBuXWl/IcTGIqlPiESR\n8wuRKHJ+IRJFzi9Eosj5hUiUjibwdHdUSXSTNXjUE6tLluXalzXeQTmSaDHPPw+zXFiuKURWsRaJ\nzisVuFTZ182jzuarPOFmHeE5RsoaolLnxnIk2Wk+EsXm5LpSyyKSF0mQCgC5HD8v41cmqO18JVyX\n8bXTb9E+27aF6wwCwK5de6mtr6+f2rrKEVmaSK01j0h9pHZh4zoSe+rKL0SiyPmFSBQ5vxCJIucX\nIlHk/EIkSmd3+wHUI7nMGA2yQ7w4O0P7FCJb8I2ISFDIVamNBQQVi/yAhdgSR3LxxZIJ9kXKlNXJ\nx3kk3R5qkXnUG3w9csYP6iRapRHZ0W/kY0nruCmW684svFb1SDK+6fOT1HZ67BS1lUt8R7+np4fa\nWIBaLM9gsRh+XdUKzwu5FF35hUgUOb8QiSLnFyJR5PxCJIqcX4hEkfMLkSgdD+yp1MLSEcvTBwAZ\nCVZgZY4AoB7Jc7cQkUOKERktT6StcoH3cZJTDwDMI+WdIvKbZ1z3YnEd8w0eUFMFHysXye9XjZyz\nItFFPcfHquX464rJebl8JAehhYOgInFC0fyPWUQzrS7wHITTcxGtksmpFX485i8L89N8nCXoyi9E\nosj5hUgUOb8QiSLnFyJR5PxCJIqcX4hEWVbqM7NHAHwCwIS7v7/VNgjgOwD2AzgF4AF356FQLbIs\nw/xiWHopxLSXjEwzIoctzF2gtlKJizmDO3gZp26i1uQiMlo+kovPczVquzoZzj0HAAuzXM7Zd+Dm\nYPtMrZf2mZy8Sm3lMo9GqxHZFgCMhOFlMc2OL2O0XyNyyBLCa5zLR3IJRkqlNWLhkbEox8octWVT\nZ4Ltl8+9wcci+f1qEblxKe1c+f8CwH1L2h4C8IS7HwLwROtvIcS7iGWd392fBHBlSfMnATzaevwo\ngE+t8byEEOvMSn/z73D3sdbjcTQr9goh3kWsesPPm/fY0l9dZnbEzEbNbLRRrax2OCHEGrFS579g\nZsMA0PqfVk1w96PuPuLuI/lSeYXDCSHWmpU6/zEAD7YePwjgB2szHSFEp2hH6vs2gLsBDJnZWQBf\nAPAlAN81s88COA3ggXYGczgadSKxROSaLeXuYPtAL5ehFnoiL824RFWc5dGAXSQ75vbt22mfxW6e\n1LFa51Jfdxd/bfme8HoAQM/AQLB9c+8w7bNziP8ci0UXLkbkt3nSb/wil2Brc1PUVnS+VoU6L1+W\nz8LnulaLJH/N87XPwM9nFilthgU+3vT5U8H2yiRfq9nZ8Dmrk8SpIZZ1fnf/NDHd2/YoQogbDt3h\nJ0SiyPmFSBQ5vxCJIucXIlHk/EIkSkcTeMIdqIell009/bTbZiLbnRt7i/ZZiNxQVIlE4dn4aWo7\nsDUs6W3fu5v2eeX8eWrzjEeP9cxxyXFTL5ebXjzzQrC9byePKusr8wSkb/7sZWpr9G6hts2HPhge\na9d7aZ+50yepLR+JZBxwHsk2PxuWD+dn6H1pKBX7qG16kScL7d68jdq2dvNzPUsiDxGpKWksCjaS\nMHYpuvILkShyfiESRc4vRKLI+YVIFDm/EIki5xciUTou9eUaYVljZx+XVy5MhmWZWj/XQgr9XDrM\nGZdr6jWeh3Tfbe8Ltk9Gat1Vt0Si84wvf26Ay3lT0zxCbGYxLBFm8zxirrLIpc9NkXmcmeUS29zF\ncALSfZs30z67bg7LgwAw9TKP3Js7x+XZyQth2/QcT5DaINGbAHB1gb/nurdwqa9/L7fVSX29xQUe\nbclqKFpMH1x6jLafKYT4hULOL0SiyPmFSBQ5vxCJIucXIlE6uttfyOcxOBDehR/q47vzU1fCucwG\nu3hASrnIdz3rNb67vf2mcLkrADg4vDfY/tJbvKzS5jIv11WPlLvavpPviueGuDIyVwh/nuf6+Twm\nL45T277tvHzZfInPf7IRDiS6MnmR9skNv4fa9txyB7WdO/sKtS0uzAfbi3n+/vBI/a98xnMJVqZ4\nsNBFcIWmPh+eYy7Pr80NUjruetCVX4hEkfMLkShyfiESRc4vRKLI+YVIFDm/EInSTrmuRwB8AsCE\nu7+/1fZFAL8L4G3d5mF3/+FyxyoV89i3czBo+6e//Ru03+k39gfbZxZ5YEllkctQ9QqX+vbv4nKT\nZ2EJyId20j5XI3Le3Dyf/54hXgKs7jyQaHYuHADjXTynYZ/zXHz5jGtKOzbxsmFzE2FJb/ZcWNYC\ngFqFv67eHVxy3PW+j1JbVrsabJ84/zrtMz/LZTlE1mOglweMFcBzMjrxwto8H8tJAI9HSqgtpZ0r\n/18AuC/Q/lV3P9z6t6zjCyFuLJZ1fnd/EsCVDsxFCNFBVvOb/3NmdtzMHjEz/r1RCHFDslLn/zqA\ngwAOAxgD8GX2RDM7YmajZjZaIYkmhBCdZ0XO7+4X3L3h7hmAbwC4PfLco+4+4u4j5S6+QSSE6Cwr\ncn4zG77mz/sBnFib6QghOkU7Ut+3AdwNYMjMzgL4AoC7zewwAAdwCsDvtTNY3hwD+bAU9Wu3cYnt\n9veFy2HNzPMcZzXnn2u1OpdD6vP8p8nCYni8A1Vermu+wuWa2UhJrmKRn5rJaV66qutAOHpvocLX\nyjcPUdu58TFqe/VNXi7tli1hqfKti5G944xLZY0uHvXZt+82avvoTfuD7VfOcKnvp88+Q20T4z+l\ntl7j+R9R4eXSFhskH1/Gpc9CMdynSnJkBo+x3BPc/dOB5m+2PYIQ4oZEd/gJkShyfiESRc4vRKLI\n+YVIFDm/EInS0QSeWb2O2SthOeTsm/xWgT27DwTbdw/voH0KPVwayiJlsqYvXaK2qanw3LcObqV9\n5ha49DK/EIn4m+XS0MzsJmq7+aaD4ePNRaSmBS45buvm0YDFCn9tv/KrHw62X5nnfU6NhyPwAKCa\n42XDGgu8lBdICa1dHwy/pwBg2wc/Rm31yXAyWQC4cvIpanvzxNPUdun1nwXbcyV+znKFsAxokeS0\nP3eMtp8phPiFQs4vRKLI+YVIFDm/EIki5xciUeT8QiRKR6W+fC6Pzd29QdvMZV4vboxENw3t5PXW\nNuX5S+vt53XwsIlLhHkLy1T9kTQFmyI1CD23sjp+J1/mtem2bQtLWz09PGpyPiIr3rqfRyz++giP\nplsgkZPzESXq0F4eAXnhMpcjz4/zSMHxN88E29+K1ONbjMjE3Zt5ItHN7w+lumxy+OZfo7bdbx4P\nth//MU+NeXH8zWC7G0+QuhRd+YVIFDm/EIki5xciUeT8QiSKnF+IROnobn8xn8fwYDgoxao84OPK\nhYlg+wvHX6N9njvBc63t2L2X2j7663dR2+5t4bkvTvId1nwhIgVEdvsLBX5q3rOLl0no7ioG28sl\n/jk/UOqhNvTzOdYafB4zJKBpocEVmpOvnqK2yUq4/BcA3HYwrHAAwOz28Dq+OcbVpZOnuZrywhv8\nPTdT5irS0ABf41t2hBWVkbt4gNFzP3k82H76Na7cLEVXfiESRc4vRKLI+YVIFDm/EIki5xciUeT8\nQiSKufMABwAws70A/hLADjTLcx1196+Z2SCA7wDYj2bJrgfcPVKvCNjS3+d3j3wgaPvAe8LlnQBg\n09awlPPMS1ySeSUiG33knnuprQ6+Hv/k3juD7Vu6eJ+ubh4kUihy+WdhkcuH27byteophwOnqpFy\nXTEsHyl7Frl2WDGcc+/V02dpnz/+91+ltksTPHjnV+8InxcA+MQ/+0yw3Ss879+Jp/+R2s7XuVT5\n0hQvr5XleS5EX5gKth+K+MS5V58Ntv/4iWO4euUSn+Q1tHPlrwP4Q3e/BcAdAH7fzG4B8BCAJ9z9\nEIAnWn8LId4lLOv87j7m7s+2Hs8AOAlgN4BPAni09bRHAXxqvSYphFh7rus3v5ntB/AhAE8B2OHu\nb5dwHUfzZ4EQ4l1C285vZn0Avgfg8+7+jhrR3tw4CP7wNbMjZjZqZqOVWvvlg4UQ60tbzm9mRTQd\n/1vu/v1W8wUzG27ZhwEEb8B396PuPuLuI+Vi+L5zIUTnWdb5zcwAfBPASXf/yjWmYwAebD1+EMAP\n1n56Qoj1op2ovo8A+AyAF83s+VbbwwC+BOC7ZvZZAKcBPLDcgWqNDBenwhLWK0UetZWfuBxsf2ts\nLNgOAHfdeze1Pfyv/oja/vTP/hO1/a+/PhZs/+XdvFxXsZSntt7+AWprNHg+u8FNg9S2bTC89RKL\nEiyVeOReLlLabLbBE/JVC+Hrytf/83+hfV5+5UVqKxf5HB879t+pbc/NRFo+9Eu0T3eZlwYbcP6a\nd/VRE+pkPQBgjkQ6epXLs/t2h3MyjkbWaSnLOr+7/x0AphtywVwIcUOjO/yESBQ5vxCJIucXIlHk\n/EIkipxfiETpaALPUrmM3fvfG7Q1MEP71WrhCKxSL9dWhvfyMlNuPApv7y5ejulHP/hesH1mnCey\n7Onm0Vzl7khyTyqwAOUCv1mqrye8Jj3dPIKwFJGHukp8jt7FX9vFhfD5fOnky7TPb/4mF49uPXwr\ntX3jz7l8+JMn/3ew/eBOnmyz1MPl2UvjPPHnC6/+jNqKvXwddwyE59JY4HJvN0nI2lY4Xwtd+YVI\nFDm/EIki5xciUeT8QiSKnF+IRJHzC5EoHZX6HI46wvJFI+PyW6kclql6eVAcpmd5AswLEzyC8NIV\nnoP07Hg4utDrPElJV5lLPLUal3JiaVXLRX7aesthGTBf4PJVdxePYuvq4hJhlufC0lsXL4QNzvt8\n6v77qe3DH/4wtZ05w5OCPnbsr4Ptz72wj/ZpLFapbfLCVWqrXj5HbYUGT+Q6X58Ntr8xeYb26SmH\n5dlKZYH2WYqu/EIkipxfiESR8wuRKHJ+IRJFzi9EonR0t79eb+DSVHjHvFbn5ZMKufBnlNf5bvlz\nx09Q2wdu/ZVIP55HjpWnqhb4jn61xnfZx8YuUdtipJxUKZKPr0iGiwV8FEs8UKgYURYazstTzS6G\nd50Hh3h5h6GtPBfizPQ0te0c3kltVybDys7f/M0PaZ/F2Tlqu3w5vDMPAHPGr6WFSIBXniggW3aE\ny9QBwPYd4ddcj+R+XIqu/EIkipxfiESR8wuRKHJ+IRJFzi9Eosj5hUiUZaU+M9sL4C/RLMHtAI66\n+9fM7IsAfhfA21rKw+7O9RM0c+c1LCwPWZ7nkZudDwfpLMxy2WX8YlhSBIA/+dM/o7bTr53m86iG\nZZTXzvFAIY8ELMVKctUaXEazBi/jlCef5xYR+yySK86Nl6eK5ovz8Ovu7uVzv3yZn7NypKTY9FUu\nA1Yq4fmfOsWDgSwiIdf4aYFHgqBigVosh2JvmeeonJ8LzzGLvN+W0o7OXwfwh+7+rJn1A3jGzB5v\n2b7q7v+h7dGEEDcM7dTqGwMw1no8Y2YnAfDUuEKIdwXX9ZvfzPYD+BCAp1pNnzOz42b2iJnx/NVC\niBuOtp3fzPoAfA/A5919GsDXARwEcBjNbwZfJv2OmNmomY3WqzzphRCis7Tl/GZWRNPxv+Xu3wcA\nd7/g7g13zwB8A8Dtob7uftTdR9x9pBC5h1wI0VmWdX4zMwDfBHDS3b9yTfvwNU+7HwCPpBFC3HC0\ns9v/EQCfAfCimT3fansYwKfN7DCaKsYpAL+37GCFAga3DhIrj35bIFFWlUi5rlwkwmpqcoratm7b\nTm2bBsNRVvWIvJI5zwdXr3HZq1HnElss919WC88lJitWKnyOGZHsAACRqL4cua5MRaLz/v7Hf09t\n99xzD7W99PJJamMvuxo5Z/nIezGLvK9i8myjEvnJWw3P5cxpnsMvXw7nBKxdx0/rdnb7/w5hSTeq\n6Qshbmx0h58QiSLnFyJR5PxCJIqcX4hEkfMLkSjmMSlnjdk0uMnvvPfOoC2LREuRCl/IR8SKQiTJ\npcVeciSii0VM5fJcGqpXedmwrMEltkZENsoii8VOZ73GpcPZOR4dWalwObJWi8yfrGPseD3dPBHq\n/gMHqG30mWepbWo6nAg1FuUY84lGxBapRAZYNAYySC7H31ddPeEIwsXZKTQa9bYG05VfiESR8wuR\nKHJ+IRJFzi9Eosj5hUgUOb8QidLRWn0Gg1lYvigW+eeQ5Yly0eCKRrEYyR0QC1SLSDJlJulF+pQi\nK2zooraYNNeI6aJEiorJkVuHWKQlUIvMwyNRfUyqzDIupc7NcVl0/MIFatu/n8uAM3PhKLf5hXAt\nwSb8DVKPyoARCTZyzti5yZEalU1b+D03sThD+/zcMdp+phDiFwo5vxCJIucXIlHk/EIkipxfiESR\n8wuRKB2V+hwG97Cs4VmklhyJwIoFSsUi36IyYIFLYkYGzMUmEjlePiLlFCMJJms1nqSRJuqMTDFW\nTzBvfK3qDS4DMmWxGHnN3f2bqW33e3itvlh9ugVSXzEmYcbeO5bn849FA8aOmSeLFU+6Go6OvHrl\nEu2zFF35hUgUOb8QiSLnFyJR5PxCJIqcX4hEWXa338y6ADwJoNx6/v9w9y+Y2SCA7wDYj2a5rgfc\nfTJ2LM8c1cXwDibbSQcAtsEa2zmO7q7G8vtFduedBHxkkUAQi5R3ykV20ovd3OZ5vttfjuxGc1aW\nz64eKylWDef3yyLBL7HjzVdjQUR8V3yxHl6r2PsNLJAMgEfGigXvlEpcrYjlm2T0kBx+sWCgn3tu\nG8+pAPgNd78VzXLc95nZHQAeAvCEux8C8ETrbyHEu4Rlnd+bvJ3etdj65wA+CeDRVvujAD61LjMU\nQqwLbX1HMLN8q0LvBIDH3f0pADvcfaz1lHEAO9ZpjkKIdaAt53f3hrsfBrAHwO1m9v4ldgfJgGBm\nR8xs1MxG2e9AIUTnua7dIXefAvC3AO4DcMHMhgGg9f8E6XPU3UfcfaQY2fQQQnSWZZ3fzLaZ2ebW\n424AHwPwCoBjAB5sPe1BAD9Yr0kKIdaedjSGYQCPWjP5Xg7Ad939f5rZTwB818w+C+A0gAfaGdBp\nTSMur7DSTzAuu5TLZWqLB8ZwW7EUlt9ismIBXLJrRIJL6rE8g7EAEiI7spxvQFz2sljwUTkStFQM\nf8uLjRWT7GJrXCNyHgDksvAaZ5Gx6hFbPlKTK4tIlbFztpKSeVzSa78s2LLO7+7HAXwo0H4ZwL1t\njySEuKHQHX5CJIqcX4hEkfMLkShyfiESRc4vRKLYSmSGFQ9mdhFNWRAAhgC0n3Bs/dA83onm8U7e\nbfPY5+7b2jlgR53/HQObjbr7yIYMrnloHpqHvvYLkSpyfiESZSOd/+gGjn0tmsc70TzeyS/sPDbs\nN78QYmPR134hEmVDnN/M7jOzn5rZa2a2Ybn/zOyUmb1oZs+b2WgHx33EzCbM7MQ1bYNm9riZvdr6\nf8sGzeOLZnautSbPm9nHOzCPvWb2t2b2spm9ZGb/vNXe0TWJzKOja2JmXWb2j2b2Qmse/6bVvrbr\n4e4d/QcgD+B1AAcBlAC8AOCWTs+jNZdTAIY2YNy7ANwG4MQ1bX8M4KHW44cA/LsNmscXAfzLDq/H\nMIDbWo/7AfwMwC2dXpPIPDq6JmjG5fa1HhcBPAXgjrVej4248t8O4DV3f8PdqwD+Cs1koMng7k8C\nuLKkueMJUck8Oo67j7n7s63HMwBOAtiNDq9JZB4dxZuse9LcjXD+3QDOXPP3WWzAArdwAD8ys2fM\n7MgGzeFtbqSEqJ8zs+OtnwXr/vPjWsxsP5r5IzY0SeySeQAdXpNOJM1NfcPvTm8mJv1tAL9vZndt\n9ISAeELUDvB1NH+SHQYwBuDLnRrYzPoAfA/A5919+lpbJ9ckMI+Or4mvImluu2yE858DsPeav/e0\n2jqOu59r/T8B4DE0f5JsFG0lRF1v3P1C642XAfgGOrQmZlZE0+G+5e7fbzV3fE1C89ioNWmNfd1J\nc9tlI5z/aQCHzOyAmZUA/A6ayUA7ipn1mln/248B/BaAE/Fe68oNkRD17TdXi/vRgTWxZmK/bwI4\n6e5fucbU0TVh8+j0mnQsaW6ndjCX7GZ+HM2d1NcB/NEGzeEgmkrDCwBe6uQ8AHwbza+PNTT3PD4L\nYCuaZc9eBfAjAIMbNI//CuBFAMdbb7bhDszjTjS/wh4H8Hzr38c7vSaReXR0TQB8EMBzrfFOAPjX\nrfY1XQ/d4SdEoqS+4SdEssj5hUgUOb8QiSLnFyJR5PxCJIqcX4hEkfMLkShyfiES5f8BnyzbOCIm\ng5EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe7cc714da0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "automobile\n"
     ]
    }
   ],
   "source": [
    "j=4   # set the index of the image you want to show\n",
    "imm_list = [i for i in images[j]]\n",
    "imagine = np.array(imm_list).reshape([3,32,32]).transpose([1, 2, 0])\n",
    "imagine = imagine.astype(np.uint8)\n",
    "plt.imshow(imagine)\n",
    "plt.show()\n",
    "print(classes[labels[j][0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = \"train.tfrecords\"\n",
    "writer = tf.python_io.TFRecordWriter(filename)\n",
    "for i in range(length):\n",
    "    example = tf.train.Example(\n",
    "        features=tf.train.Features(\n",
    "            feature={\n",
    "                #'label': tf.train.Feature(bytes_list=tf.train.BytesList(value=[labels[i]])),\n",
    "                'label': tf.train.Feature(int64_list=tf.train.Int64List(value=[labels[i][0]])), \n",
    "                'x': tf.train.Feature(bytes_list=tf.train.BytesList(value=[images[i]]))\n",
    "            })\n",
    "        )\n",
    "    writer.write(example.SerializeToString())\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting Testing Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "labels = []\n",
    "images = []\n",
    "fname=\"cifar_tensorflow/cifar10_data/cifar-10-batches-bin/test_batch.bin\"\n",
    "with open(fname, \"br\") as f:\n",
    "    string = f.read(record_bytes)\n",
    "    while string:\n",
    "        labels.append(string[:label_bytes])\n",
    "        images.append(string[label_bytes:])\n",
    "        string = f.read(record_bytes)\n",
    "length = len(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = \"test.tfrecords\"\n",
    "writer = tf.python_io.TFRecordWriter(filename)\n",
    "for i in range(length):\n",
    "    example = tf.train.Example(\n",
    "        features=tf.train.Features(\n",
    "            feature={\n",
    "                #'label': tf.train.Feature(bytes_list=tf.train.BytesList(value=[labels[i]])),\n",
    "               'label': tf.train.Feature(int64_list=tf.train.Int64List(value=[labels[i][0]])), \n",
    "               'x': tf.train.Feature(bytes_list=tf.train.BytesList(value=[images[i]]))\n",
    "            })\n",
    "        )\n",
    "    writer.write(example.SerializeToString())\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verifying encoding\n",
    "\n",
    "In order to very that the operation was done correctly, we try to represent one the returned images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    dataset = tf.contrib.data.TFRecordDataset([\"train.tfrecords\"]) \n",
    "\n",
    "    def parser(example_proto):\n",
    "        features={\n",
    "          'label': tf.FixedLenFeature([], tf.int64),\n",
    "          'x': tf.FixedLenFeature([], tf.string),\n",
    "        }\n",
    "        parsed_features = tf.parse_single_example(example_proto, features)\n",
    "        image = tf.decode_raw(parsed_features['x'], tf.uint8)\n",
    "        label = tf.cast(parsed_features[\"label\"], tf.int32)\n",
    "        #image.set_shape([mnist.IMAGE_PIXELS])\n",
    "\n",
    "        depth_major = tf.reshape(image, [3, 32, 32])\n",
    "        uint8image = tf.transpose(depth_major, [1, 2, 0])\n",
    "        reshaped_image = tf.cast(uint8image, tf.float32)\n",
    "        resized_image = tf.image.resize_image_with_crop_or_pad(reshaped_image, 24, 24)\n",
    "        float_image = tf.image.per_image_standardization(resized_image)\n",
    "        float_image.set_shape([24, 24, 3])   \n",
    "\n",
    "        #label.set_shape([1])\n",
    "        # Convert label from a scalar uint8 tensor to an int32 scalar.\n",
    "\n",
    "\n",
    "        #return float_image, label\n",
    "        return uint8image, label\n",
    "    # Use `Dataset.map()` to build a pair of a feature dictionary and a label \n",
    "    # tensor for each example.\n",
    "    dataset = dataset.map(parser)\n",
    "    dataset = dataset.shuffle(buffer_size=10000)\n",
    "    #dataset = dataset.batch(128)\n",
    "    #dataset = dataset.repeat(None)\n",
    "    iterator = dataset.make_one_shot_iterator()\n",
    "\n",
    "    # `features` is a dictionary in which each value is a batch of values for\n",
    "    # that feature; `labels` is a batch of labels.\n",
    "    features, labels = iterator.get_next()\n",
    "    for i in range(1):\n",
    "        imm, lab = sess.run([features, labels])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH0RJREFUeJztnWuMXdd13//rnPucOzN8DR8jihJFh7SiSNHDjKLARuDG\nSKAaBmwDiRADNfTBiFIgNWIg/SC4QO22X5y2dmAUhQG6EqIUrmMntmHVNdLYqgE1biqLkvWgXrYe\npEWKryHncWfu85y7+mEuUWq0/3uG5PAO5fP/AQTv7HX3Ofvsc9Y99+7/WWuZu0MIUTySjR6AEGJj\nkPMLUVDk/EIUFDm/EAVFzi9EQZHzC1FQ5PxCFBQ5vxAFRc4vREEpXUlnM7sXwJcBpAD+i7t/Ifb+\nyU2bfcfOXUFblmW0X5qm4T79Pu3jg5zaSuUytWXZgNpAnoYslcLjA4As52PMI7ZyiY+xVq1Rm+e9\nYHulWqV9DEZtWS+8PQDIc37OzMLbjD1RmuX8nPUzPle12hi1JWn4Em+1O7TPYMDHGJurJHIdlCv8\nfC4tNsPbS/i9uVSuBNvPnTuHxWaTD/LibazlTSHMLAXwnwH8LoDjAJ40s0fd/UXWZ8fOXfgP/+mh\noG1m5jTd1+TktmD7+TMnaJ9eOzyhALBjxzS1nT+3QG0DcrFvnZqkfc7N8eNqzp6htl3bd1Dbgf2/\nSm392aPB9ht/5QDtUzF+GZw5eozaFhbO821WwtvsdLiDzzb59k7O8Lk6cPOd1NYYnwq2P3vkFdpn\nqc0/8NiNCADq2zZT264brqO2w//7fwXba2P8Q21q+sZg+5//u39L+6zkSr723w3gVXd/3d17AP4a\nwEevYHtCiBFyJc6/G8CbF/19fNgmhHgXcNUX/MzsATM7bGaHF+bnrvbuhBBr5Eqc/wSAPRf9ff2w\n7W24+yF3P+juByc38d9EQojRciXO/ySA/WZ2k5lVAPwhgEfXZ1hCiKvNZa/2u3tmZv8CwP/EstT3\nsLu/EOuT97tonnwtaKsmXHpZOBVe1Z8c20T7dJOwFAIA8/Mz1DaxmX87GatPBNvN+GdoMj9Pbf02\nX/neOrGF2tBvU9MiWYHvdvn8lqrj1JaW+LHV6nyOsyy8Yt7t8XGcOvMWtZ09x1f7r79uL7WNV8Pn\nbN80V1OeOvwUtb36evj6BYA/uP+T1Hb7be+ltsG58HHvveU22ufsQvgaiEm6K7kind/dvw/g+1ey\nDSHExqAn/IQoKHJ+IQqKnF+IgiLnF6KgyPmFKChXtNp/qWRZhvPnwzJbI6zIAADSUjjSrtNt0T71\nRjigAwASdKmtl/JttvtLwfZ+j0e3zfdm+faIHAYAgwEPzOo0+ZOSiZOxRKLp8gGPZMwj0ZGDQSSq\nLwnvj0mAAJB3Ittr8zGeeeMNajv72tFge3NxkfZ56fDT1Hb6OA8m+79j/F76k+/9DbUtkPm//bc+\nQPu8cTos6cYiEleiO78QBUXOL0RBkfMLUVDk/EIUFDm/EAVlpKv9/XyANxfCq+n1AV8FTkrhVeCF\n+VO0T7lyktp6zoNL8oyvAvsgnEeu34/kHzSe9mmsxHPxdTysLADAhPN8cD2S17Db5tvrtfiK/sIS\nT2uWGD9uJyvYrRYfR97kSkvvLE/x9epb/FyPVcPBR6WEH3O6yLe3OY1cp4s8YGx+hitMH/nnfxps\n333dnmA7ADz1bDhbnnskB+UKdOcXoqDI+YUoKHJ+IQqKnF+IgiLnF6KgyPmFKCgjlfosAWqN8OfN\n5FaeRy4nOfKSOv/sSlOeXy5vcbmm2ubjqA/Cck0754E2rTbP4ZemkZJiXS45liZ5/rlOEg4IWupG\nJLaIVLnY4fkCKwkvoYV++Nhm3noz2A4Ap19/mdrKkXJuYzUufU426sH2pSYPuNq8iQfH3Pwb91Db\n73zk96ltZp7LmNtv2h9sf/bIM7RPnhEJ1vk1tRLd+YUoKHJ+IQqKnF+IgiLnF6KgyPmFKChyfiEK\nyhVJfWZ2FEATQA4gc/eDsffXasD+XwlLNjPz52i/114JR3SlfR4xN07y/gFA6ySPvtpWishG9XAp\npHKfyzi+wI+r3ebRhb84zctTze/ZR20JySM3VwtLXgDQjcxjuculvm6vSW1zZ8ORcfOnXqV96hUu\nHVZrfIyNCV6iarF5Nth+7ATP+7eQ8fyJ7/u1u6jtNz/2z6htZpZLt//9e38bbP/xsz+mfdKJsBzZ\nbPNzspL10Pn/ibtzbxJCXJPoa78QBeVKnd8B/NDMnjKzB9ZjQEKI0XClX/s/4O4nzGwHgB+Y2cvu\n/vjFbxh+KDwAANu285LaQojRckV3fnc/Mfz/DIDvALg78J5D7n7Q3Q9OTDauZHdCiHXksp3fzBpm\nNnHhNYDfA3BkvQYmhLi6XMnX/p0AvmNmF7bz39z972Iduvkcjs3/j6AtSfhQfvXADcH28y/z6Lzy\nIk+YuHOMR4htrnJJaUDKdS10eIRYucTlqx038ui8xXkusS288lNqs0p4Hhdf+xnt4yUulSWRKLFe\niyf3bDXD8uxYmd9vxhtj1NaJRPXNzXGx6dz5sNSXRhJd1iMJUs8c48k958/w+ajX+Lfe1mw48rM+\nxs/L+TwsIQ+w9qi+y3Z+d38dwO2X218IsbFI6hOioMj5hSgocn4hCoqcX4iCIucXoqCMvFbfyWZY\nDqkZl16sHU7G2T/LJZmtNZ6Is0YSgi7D5RVPw5FU1TrfV2OcJxIdG+P9koRHCvYrYfkKANrtcPTY\nIFIjD8aj2KzEpc8a+DnL87DEmQ749roZlzezyBiTJBLd2QjPcTfl1wDJ0woAOHmSRwO++NwT1HbT\n/lv4/kj9yt2bd/GB5OFjThPuEyvRnV+IgiLnF6KgyPmFKChyfiEKipxfiIIy0tX+fADMd8K77Ax4\nkE52KpyX7Iax62iftH2a2volvgKflnhwSZlMV72+NbI9HmjRISWtAMAqNWpDO7Kim4SXqis1vr0k\n4QpHZJEd3udBS7XxsBJQGvBSWN0lrkj0IgErjc0T1OYIr4ovLHKl4nyX7+u6Cd5v4eQr1DY7vZPb\nBuHru93l+3I2jXx634Hu/EIUFDm/EAVFzi9EQZHzC1FQ5PxCFBQ5vxAFZaRSn1mKcmkyaBuv8aGU\nmmFpruY8Fbh3ePBLPSKjjTW4bJSQPHJJEglwiQQspSX+2VtyPh9pLTyHAJANwjJVqcLLdZXKPMAo\nj+S6Qzkiz+Zhzanb5gFLpQqXMJOMj6Ob8UicVi9cEu18j8uUW95zgNrakWCm1gIvlVWqcDl1BuE5\n6WZ8jPTasbVrfbrzC1FQ5PxCFBQ5vxAFRc4vREGR8wtRUOT8QhSUVaU+M3sYwEcAnHH3W4dtWwF8\nA8BeAEcB3OfuvGbVkMTKqJVIdFOkLFQyHpbmrLSZ9hnMRqStlEf1lcDD2DKSl67di5QNm+BlmiYm\nuWQ3d4ZLlYNIpF1aCx93qcrnIy1x6bMUC+sb8HnMiSzaj0QCwvm+Jsf5PPbA5/+tVngcOw/cSvtM\n3XYHtVmf72ty+x5q2zHFo/qmb9wbbD928lXap1IJRyta7HytYC13/r8EcO+KtgcBPObu+wE8Nvxb\nCPEuYlXnd/fHAaysuvhRAI8MXz8C4GPrPC4hxFXmcn/z73T3C+VKT2G5Yq8Q4l3EFS/4ubsjkj/E\nzB4ws8NmdrgTKZsthBgtl+v8p81sGgCG/59hb3T3Q+5+0N0P1sb5881CiNFyuc7/KID7h6/vB/Dd\n9RmOEGJUrEXq+zqADwKYMrPjAD4H4AsAvmlmnwJwDMB9a9mZIUE5CUeQ9TOevLGLsFwzH4mYq0fK\nFuW9SBRel0tRvX54HJ2U78tqXI7MIyXFsoQrp4OInNMmY8xL/JjHx/g3sk43HBUHAJWUXz4DYkur\nEVmxwu9F9TovybU4e4raKuPh+T9w62/QPv2tfAmrOTdPbcdPzVHbbS1+Xe3ZFi7LdXaGlwZLS+Fr\n4BKUvtWd390/QUwfWvtuhBDXGnrCT4iCIucXoqDI+YUoKHJ+IQqKnF+IgjLSBJ5wwElQVL/HP4ey\nLKxf5HU+/EqDy2jd2QVqy1tcYuuQOnP5xBTts9TnElVivC5gMs7r/41VeYRbfTKc1DSW1jGWXLLV\naVNbPuDyYTYIS46tHk/gWa/zyMNeJOrz7PmVoSf/n+rW3cH22SU+jvmMPrOGnCQmBYBsaYba/v5H\nf09tJxbeIhvk++oNwtKhR2ohrkR3fiEKipxfiIIi5xeioMj5hSgocn4hCoqcX4iCMlKpL0UJkx6O\nsmqUttN+mYU/o/oZl3+ySKTdwLl8tTTPI7NQDctvrRZP6vjUM4ep7TfvPkhtO8d49FulzhN/bmmE\nj3thnstQC0u8xlwSkQGR84i/xMJS39wCH0ezyy/HscgxZwmvr7h9x43B9k6Zh7+dafPkqZUal2d7\nJPoUAM6/+jS1tSx8/QzAIwEr1fBcrV3o051fiMIi5xeioMj5hSgocn4hCoqcX4iCMtLV/nJSwfT4\nvqCtm/O03k22Gp3yFdu0zoNtcj9HbVbin4ebpsNBIpU6Vyr2ZDywJ42s6DedB810e3xVueXheVxa\n5IEsrX6kTNbmyCr7LM+72FoMn7NNE+HAIwAoV/iqfW5cddi6O5wDDwCq26eD7aVJHhzVfZNfHxn4\neelH3KlEFCuA51f0yNp9Tmq2+SUs9+vOL0RBkfMLUVDk/EIUFDm/EAVFzi9EQZHzC1FQ1lKu62EA\nHwFwxt1vHbZ9HsAfAbgQAfFZd//+qntLEpovrtPlwTGehvWLmCyXRAJ7qg0uX6UTXFKaJbWQWhmX\n0aZ281x8i5F+pyI55izjUp/PhvPBded53sJde3+N2l7/BS+FVW/xXHdbe+HzuWvrDtrn3AIPuDrT\n5vJbaQffZqcVDiTatpXnCyyXuTzbiQQzeYm7k5X4NquN8Fj6kfOcs5yGtr45/P4SwL2B9r9w9zuG\n/1Z3fCHENcWqzu/ujwPg6VGFEO9KruQ3/6fN7Dkze9jMtqzbiIQQI+Fynf8rAPYBuAPASQBfZG80\nswfM7LCZHV5sLl7m7oQQ681lOb+7n3b33N0HAL4K4O7Iew+5+0F3Pzg+wQtpCCFGy2U5v5ldHC3x\ncQBH1mc4QohRsRap7+sAPghgysyOA/gcgA+a2R1YThl2FMAfr2VnnX4HL59+OWhrdbm0xUo/VSIl\nnDbnPFJtrMplwFnjkswM2V9zkUteg0j5pEFkX802lz5JEBgAIFkIj2U8qdA+k1M8KvG50/zYthA5\nDwCmkvB9JYtEJI5vu47atk1z2/mc57p74eevBduTSE7AckSyY2WyACAnufgAoBtR4PJe+IRawq8P\nIGy7lBx+qzq/u38i0PzQJexDCHENoif8hCgocn4hCoqcX4iCIucXoqDI+YUoKCNN4JkPcswvhZ/y\na7d5RNdYOSxTTVV5AsxaziUlK/OIrqU+19HmyDbTEpfR8kjJpcVWJKlmOzL+Fpc404Xw/O7YEU4+\nCgCZ8cvg1HlevqzU4MJSloVtZ2d4ua733Pw+arvr/R+itr978h+pbSk/Gmw/foxLmFaJyMSbeLmu\nUiRyzyPRdl2iAyax7TGZWwk8hRCrIecXoqDI+YUoKHJ+IQqKnF+IgiLnF6KgjFTqG6+O4Z733Bm0\ndTo8MeJYEh5mqXk22A4Agzlu2xRJIjnf4zJapRRO7lmJJAs9NztLbdbhSSl3bOe5D0oJl6JalXBk\nWWnAZaNYlOBt772Z98v5+Mc74Tp+g4xHvqHOpdtGnc/H5CaeSGrnDWGJM+lxTaxEpGUAmNjC6wm2\nI5Gp3chxm4dtAyKXAkApDZ9PfmW8E935hSgocn4hCoqcX4iCIucXoqDI+YUoKCNd7bckQZms6A5I\nKSwAyLrhIJeFGb6iv6XKV7f7kYCgZpMHspyaJbVLEq4Q9CIll84u8fH3505SW60UKUXWCQdITY1v\no30s61Jb8/xpaltq8yCdu3/99mD7WGRFvx0JSqlFglzyLleKZhfCaksSufQr4GOcneHp51muSQDw\nyDVSJrfgfsaDwkokz2A+iEg3K9CdX4iCIucXoqDI+YUoKHJ+IQqKnF+IgiLnF6KgrKVc1x4AfwVg\nJ5YzhB1y9y+b2VYA3wCwF8slu+5zdx7FAmCp28ZPXnsuaOtHSj/lS2Ep54ZIEE7ZeDmmo6e5xHa8\nxQ9hsR8OVvGEyyuZc1szX6C2WJmvivEgl6wSlsTGdk3RPhiLBAoNuPSZTHD5rbE9vL+xSCms428e\no7ZuZ57aUvBrp5eHz1mt3qB9Wl1+zJ7Eyq9FcitW+H3WySbbEQmzlIWDjwbrLPVlAP7M3W8BcA+A\nPzGzWwA8COAxd98P4LHh30KIdwmrOr+7n3T3p4evmwBeArAbwEcBPDJ82yMAPna1BimEWH8u6Te/\nme0FcCeAJwDsdPcLj6GdwvLPAiHEu4Q1O7+ZjQP4FoDPuPvbfqy6u4NkDDezB8zssJkd7izx3PxC\niNGyJuc3szKWHf9r7v7tYfNpM5se2qcBBKsguPshdz/o7gdrDV4sQwgxWlZ1fjMzAA8BeMndv3SR\n6VEA9w9f3w/gu+s/PCHE1WItUX3vB/BJAM+b2TPDts8C+AKAb5rZpwAcA3DfahtyH6CfhyPI+pHI\nMiclrypjm2ifXfv2U9urx16htoU+j9oqJSz8KpzbDwC6kZJckwmXI8c3cylqbIyfNvOwDHiiyWVF\n9Hn02MQOLit6yqWtJSKLZh1+vzlx+i1qe+nFn1Jbv8Pl2cZkOAKyUovMYcbH2O7yn65plUdbIuVy\narcdliobDX4NDAbh7Rm7RgOs6vzu/g/geQF5ATUhxDWNnvAToqDI+YUoKHJ+IQqKnF+IgiLnF6Kg\njDaBpyWopuEHfWJRffVSWPK4fj8vJXXHb91DbT9+k0t9YyVeqqnZCss8i20uD26KlJnaVBujttin\ncjfj0V55Hg4Rs0ghp0okEiyLlLVCyhNWtlphqW8x55fcyQ6Xe9948xfU9lYz+HwZAKBEErla+IFU\nAMBYnUu3HknImkVKoqUV/oBblZichfsBGJCEoEkkEe473rvmdwohfqmQ8wtRUOT8QhQUOb8QBUXO\nL0RBkfMLUVBGKvX5wJERSa9W4/LKIA/LGl7lUU+lsS18HJHD7kXqreVJ2FaqcnmlVI/UIEy4vFmJ\nJLosG5eUkBHZLqIAlUnSTwCYqm2mtpjk2JgMJ/Cci8h5s8bnvraLJ4rK2zxisd1uBtvHKzwCLy3x\ne+LmSCRps8XnwzMu2yUkKWgsIagRm1lEml253zW/UwjxS4WcX4iCIucXoqDI+YUoKHJ+IQrKSFf7\ny2kJOxrbgrb5SI65vodzzGXOV6lz54dWL/OAmgFfgEfWDa9Gb2rw4J1IdSeUynzFuZ/zle8BmQ8A\nsCS8rJ8TxQQAssi+JmoT1JZH5r/eCK/2z3RP0T7dyK1ox+4bqW22xcf/s9efD7bnJX5iFiOr9uXI\ndVVJeVBYFim/lrHrO6I8eRpWdSKxQO9Ad34hCoqcX4iCIucXoqDI+YUoKHJ+IQqKnF+IgrKq1Gdm\newD8FZZLcDuAQ+7+ZTP7PIA/AnB2+NbPuvv3Y9uqpBXs3XJD0HYumeP9SE616Z3TkX1xGerGqeup\nbebsOWpjme4mSY5BAMgiOd/SfiRAh6fVgw+4bFcmASuTDS5vZpFyXefmuQSbOZcqGySwaiyS77Bu\nke2VuZy6fTO/DurVN4LtnT6X82D8njgRkWdLkX79iNTaz4lsF83HF97XpUh9a9H5MwB/5u5Pm9kE\ngKfM7AdD21+4+39c++6EENcKa6nVdxLAyeHrppm9BGD31R6YEOLqckm/+c1sL4A7ATwxbPq0mT1n\nZg+bGQ+gF0Jcc6zZ+c1sHMC3AHzG3RcAfAXAPgB3YPmbwRdJvwfM7LCZHV5shhMrCCFGz5qc38zK\nWHb8r7n7twHA3U+7e+7uAwBfBXB3qK+7H3L3g+5+cHyCPycuhBgtqzq/mRmAhwC85O5fuqj94iXW\njwM4sv7DE0JcLday2v9+AJ8E8LyZPTNs+yyAT5jZHViW/44C+OPVNpSkKWoTk0HbdTUul1VJCS3r\nc/mks8B/YuyY2E5tt95wK7WVa+FxNOo8mqsTicxCJNJr0Ofhha2MH1tKcv/FRKPTZyLlruo8514s\nLx0LPGwYP8+7K1yyq6b8W+PUOJ+rbWk45146sYv2yVkeRADjdS6ZTm0NRzICQDcSOXlq/mywPVbC\nLiWaXinhUuQ73rvaG9z9HxC+dqKavhDi2kZP+AlRUOT8QhQUOb8QBUXOL0RBkfMLUVBGmsBzsb2I\n//PiE0Hb9kn+dHCdJEZMT3Kpqd/h0WMOLhG2Wrzf1OSO8PbqPDrPnItsg4yPI015+bJJv/QoQo+E\ne+3czuUrpHyM9YQfd2upHWzft+c9fF/Oj/nszHlqm6hyeevA9E3B9iSSmLSa1qmtFCmjNhl5iC2t\n8n7+xsvB9vnzs7TP9FRYrq6W+ByuRHd+IQqKnF+IgiLnF6KgyPmFKChyfiEKipxfiIIyUqkv9xxz\n3bB80Z7liSLbRH6rR4a/0Jqhtpv2hOUfAGgN+DhePBGOfmsZTwYZS+A5cB491qhyuWky5VJfTvfH\nJcfYGKvlGrVtHg9HaALA3Fw4EeoN+/bQPlun+HEdOfIstb3v9ruo7b0HDgTbT5ziiVrHazxZaCxB\nZr/DIyD73UjkIakduWWKz32NJBJNI0lEV6I7vxAFRc4vREGR8wtRUOT8QhQUOb8QBUXOL0RBGanU\nBwCehj9v8hLXUGwsHD1WqfDhz3bCSREBID8RkWQiMzK/FJYpkxr/DK3X+Aa7kQSksy0+/qzC6xom\npL5buxOOsgOAUsrH2ChvprbWIk8k2muH5c/0WX7MjQkeXXi2zSPcnnzpBWq7ef/NwfZTM8dpn04W\nru8HAGNjfIyDAZdua1UebVchbtjt8eu0uRSWZ7Oc111cie78QhQUOb8QBUXOL0RBkfMLUVDk/EIU\nlFVX+82sBuBxANXh+//W3T9nZlsBfAPAXiyX67rP3fmSLIAkSdEgZbm6vUgJqiS8gr3Y5X0mIgEp\ns0t8tdwT/nlYroZVh0HOV7BLkY9XjwRh5GVeAgwpz1mX5+EV5yThq82xY+4lfPU4c74abfWwevPW\nwlu0T6nFL8dOic/xa6f5yn0P4fkYJDzQ5lTzJLXl8zwIKomoJrVyLFArnPsvieRIzAfh+cicz9M7\ntr+G93QB/I67347lctz3mtk9AB4E8Ji77wfw2PBvIcS7hFWd35e5EFNbHv5zAB8F8Miw/REAH7sq\nIxRCXBXW9JvfzNJhhd4zAH7g7k8A2OnuF74fnQKw8yqNUQhxFViT87t77u53ALgewN1mdusKu2P5\n28A7MLMHzOywmR3ukFzuQojRc0mr/e4+B+BHAO4FcNrMpgFg+H8wzY27H3L3g+5+sNbgix5CiNGy\nqvOb2XYz2zx8XQfwuwBeBvAogPuHb7sfwHev1iCFEOvPWgJ7pgE8YmYplj8svunu3zOzfwTwTTP7\nFIBjAO5bfVMDAGGJpVbmn0PVSvgbw0KbyzVZTCbp8wCMJOU2FnpkJFgJABYzLpUNiCy3vM3I+DPe\nr0w+zxvjm2ifXkQeynnqP5RIEBEAVJLwpZWW+HF5LM8glqitOsGlz9nefLB9YTHcDgDlSPm1SoWf\n61YkT58TyREAxsokcC0iwQ4GRHKMnK+VrOr87v4cgDsD7ecAfGjtuxJCXEvoCT8hCoqcX4iCIucX\noqDI+YUoKHJ+IQqKeaz+0HrvzOwslmVBAJgCwGtqjQ6N4+1oHG/n3TaOG919+1o2OFLnf9uOzQ67\n+8EN2bnGoXFoHPraL0RRkfMLUVA20vkPbeC+L0bjeDsax9v5pR3Hhv3mF0JsLPraL0RB2RDnN7N7\nzewVM3vVzDYs95+ZHTWz583sGTM7PML9PmxmZ8zsyEVtW83sB2b28+H/WzZoHJ83sxPDOXnGzD48\ngnHsMbMfmdmLZvaCmf3psH2kcxIZx0jnxMxqZvYTM3t2OI5/M2xf3/lw95H+A5ACeA3APgAVAM8C\nuGXU4xiO5SiAqQ3Y728DuAvAkYva/j2AB4evHwTw5xs0js8D+Jcjno9pAHcNX08A+BmAW0Y9J5Fx\njHROsByYOz58XQbwBIB71ns+NuLOfzeAV939dXfvAfhrLCcDLQzu/jiA8yuaR54QlYxj5Lj7SXd/\nevi6CeAlALsx4jmJjGOk+DJXPWnuRjj/bgBvXvT3cWzABA9xAD80s6fM7IENGsMFrqWEqJ82s+eG\nPwuu+s+PizGzvVjOH7GhSWJXjAMY8ZyMImlu0Rf8PuDLiUn/KYA/MbPf3ugBAfGEqCPgK1j+SXYH\ngJMAvjiqHZvZOIBvAfiMuy9cbBvlnATGMfI58StImrtWNsL5TwDYc9Hf1w/bRo67nxj+fwbAd7D8\nk2SjWFNC1KuNu58eXngDAF/FiObEzMpYdrivufu3h80jn5PQODZqTob7vuSkuWtlI5z/SQD7zewm\nM6sA+EMsJwMdKWbWMLOJC68B/B6AI/FeV5VrIiHqhYtryMcxgjkxMwPwEICX3P1LF5lGOidsHKOe\nk5ElzR3VCuaK1cwPY3kl9TUA/2qDxrAPy0rDswBeGOU4AHwdy18f+1he8/gUgG1YLnv2cwA/BLB1\ng8bxXwE8D+C54cU2PYJxfADLX2GfA/DM8N+HRz0nkXGMdE4A/DqAnw73dwTAvx62r+t86Ak/IQpK\n0Rf8hCgscn4hCoqcX4iCIucXoqDI+YUoKHJ+IQqKnF+IgiLnF6Kg/D/olNK1HVxB5wAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fee4e965e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "horse\n"
     ]
    }
   ],
   "source": [
    "plt.imshow(imm)\n",
    "plt.show()\n",
    "print(classes[lab])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# High-Level API\n",
    "\n",
    "The high-level API make us able to define a CNN model easily, using the **Layer** class. We will feed this model with some input functions (input_fn) which we define using the **Dataset API**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_input_fn():\n",
    "    dataset = tf.contrib.data.TFRecordDataset([\"train.tfrecords\"]) \n",
    "    def parser(example_proto):\n",
    "        features={\n",
    "          'label': tf.FixedLenFeature([], tf.int64),\n",
    "          'x': tf.FixedLenFeature([], tf.string),\n",
    "        }\n",
    "        parsed_features = tf.parse_single_example(example_proto, features)\n",
    "        image = tf.decode_raw(parsed_features['x'], tf.uint8)\n",
    "        label = tf.cast(parsed_features[\"label\"], tf.int32)\n",
    "        #image.set_shape([mnist.IMAGE_PIXELS])\n",
    "\n",
    "        depth_major = tf.reshape(image, [3, 32, 32])\n",
    "        uint8image = tf.transpose(depth_major, [1, 2, 0])\n",
    "        reshaped_image = tf.cast(uint8image, tf.float32)\n",
    "        resized_image = tf.image.resize_image_with_crop_or_pad(reshaped_image, 24, 24)\n",
    "        float_image = tf.image.per_image_standardization(resized_image)\n",
    "        float_image.set_shape([24, 24, 3])   \n",
    "\n",
    "        #label.set_shape([1])\n",
    "        # Convert label from a scalar uint8 tensor to an int32 scalar.\n",
    "        return float_image, label\n",
    "        #return uint8image, label\n",
    "    # Use `Dataset.map()` to build a pair of a feature dictionary and a label \n",
    "    # tensor for each example.\n",
    "    dataset = dataset.map(parser)\n",
    "    dataset = dataset.shuffle(buffer_size=50000)\n",
    "    dataset = dataset.batch(128)\n",
    "    dataset = dataset.repeat()\n",
    "    iterator = dataset.make_one_shot_iterator()\n",
    "\n",
    "    # `features` is a dictionary in which each value is a batch of values for\n",
    "    # that feature; `labels` is a batch of labels.\n",
    "    features, labels = iterator.get_next()\n",
    "    return features, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_input_fn():\n",
    "    dataset = tf.contrib.data.TFRecordDataset([\"test.tfrecords\"]) \n",
    "\n",
    "    def parser(example_proto):\n",
    "        features={\n",
    "          'label': tf.FixedLenFeature([], tf.int64),\n",
    "          'x': tf.FixedLenFeature([], tf.string),\n",
    "        }\n",
    "        parsed_features = tf.parse_single_example(example_proto, features)\n",
    "        image = tf.decode_raw(parsed_features['x'], tf.uint8)\n",
    "        label = tf.cast(parsed_features[\"label\"], tf.int32)\n",
    "        #image.set_shape([mnist.IMAGE_PIXELS])\n",
    "\n",
    "        depth_major = tf.reshape(image, [3, 32, 32])\n",
    "        uint8image = tf.transpose(depth_major, [1, 2, 0])\n",
    "        reshaped_image = tf.cast(uint8image, tf.float32)\n",
    "        resized_image = tf.image.resize_image_with_crop_or_pad(reshaped_image, 24, 24)\n",
    "        float_image = tf.image.per_image_standardization(resized_image)\n",
    "        float_image.set_shape([24, 24, 3])   \n",
    "\n",
    "        #label.set_shape([1])\n",
    "        # Convert label from a scalar uint8 tensor to an int32 scalar.\n",
    "        return float_image, label\n",
    "        #return uint8image, label\n",
    "    # Use `Dataset.map()` to build a pair of a feature dictionary and a label \n",
    "    # tensor for each example.\n",
    "    dataset = dataset.map(parser)\n",
    "    #dataset = dataset.shuffle(buffer_size=50000)\n",
    "    dataset = dataset.batch(128)\n",
    "    dataset = dataset.repeat()\n",
    "    iterator = dataset.make_one_shot_iterator()\n",
    "\n",
    "    # `features` is a dictionary in which each value is a batch of values for\n",
    "    # that feature; `labels` is a batch of labels.\n",
    "    features, labels = iterator.get_next()\n",
    "    return features, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the model\n",
    "\n",
    "As it has been said before, I wanted to make a model similar to AlexNet, therefore the structure is the following:\n",
    "- 2 convolutional layers (kernel = 5x5, stride = 1), with an activation layer (ReLu) and a Max-Pooling layer (kernel = 3x3, stride = 2) each;\n",
    "- 3 convolutional layers (kernel = 3x3, stride = 1) without any Max-Pooling layer;\n",
    "- 2 dense layers\n",
    "- The final layer in our neural network is the logits layer, which will return the raw values for our prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def cnn_model_fn(features, labels, mode):\n",
    "    \"\"\"Model function for CNN.\"\"\"\n",
    "\n",
    "    input_layer = tf.reshape(features, [-1, 24, 24, 3])\n",
    "\n",
    "    conv1 = tf.layers.conv2d(\n",
    "      inputs=input_layer,\n",
    "      filters=64,\n",
    "      kernel_size=[5, 5],\n",
    "      padding=\"same\",\n",
    "      activation=tf.nn.relu)\n",
    "\n",
    "    pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[3, 3], strides=2, padding=\"same\")\n",
    "\n",
    "    conv2 = tf.layers.conv2d(\n",
    "      inputs=pool1,\n",
    "      filters=64,\n",
    "      kernel_size=[5, 5],\n",
    "      padding=\"same\",\n",
    "      activation=tf.nn.relu)\n",
    "\n",
    "    pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[3, 3], strides=2, padding=\"same\")\n",
    "\n",
    "    conv3 = tf.layers.conv2d(\n",
    "      inputs=pool2,\n",
    "      filters=64,\n",
    "      kernel_size=[3, 3],\n",
    "      padding=\"same\",\n",
    "      activation=tf.nn.relu)\n",
    "    \n",
    "    conv4 = tf.layers.conv2d(\n",
    "      inputs=conv3,\n",
    "      filters=64,\n",
    "      kernel_size=[3, 3],\n",
    "      padding=\"same\",\n",
    "      activation=tf.nn.relu)\n",
    "    \n",
    "    conv5 = tf.layers.conv2d(\n",
    "      inputs=conv4,\n",
    "      filters=64,\n",
    "      kernel_size=[3, 3],\n",
    "      padding=\"same\",\n",
    "      activation=tf.nn.relu)\n",
    "    \n",
    "    \n",
    "    # Flatten tensor into a batch of vectors\n",
    "    # Input Tensor Shape: [batch_size, 7, 7, 64]\n",
    "    # Output Tensor Shape: [batch_size, 7 * 7 * 64]\n",
    "    \n",
    "    \n",
    "    conv5_flat = tf.reshape(conv5, [-1, 6 * 6 * 64])\n",
    "\n",
    "    \n",
    "    # Dense Layer\n",
    "    # Densely connected layer with 1024 neurons\n",
    "    # Input Tensor Shape: [batch_size, 7 * 7 * 64]\n",
    "    # Output Tensor Shape: [batch_size, 1024]\n",
    "    \n",
    "    dense1 = tf.layers.dense(inputs=conv5_flat, units=384, activation=tf.nn.relu)\n",
    "    dense2 = tf.layers.dense(inputs=dense1, units=192, activation=tf.nn.relu)\n",
    "    \n",
    "    # Add dropout operation; 0.6 probability that element will be kept\n",
    "    dropout = tf.layers.dropout(inputs=dense2,\n",
    "                                training=mode == tf.estimator.ModeKeys.TRAIN)\n",
    "\n",
    "    # Logits layer\n",
    "    # Input Tensor Shape: [batch_size, 1024]\n",
    "    # Output Tensor Shape: [batch_size, 10]\n",
    "    logits = tf.layers.dense(inputs=dropout, units=10)\n",
    "\n",
    "    predictions = {\n",
    "      # Generate predictions (for PREDICT and EVAL mode)\n",
    "      \"classes\": tf.argmax(input=logits, axis=1),\n",
    "      # Add `softmax_tensor` to the graph. It is used for PREDICT and by the\n",
    "      # `logging_hook`.\n",
    "      \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\")\n",
    "    }\n",
    "    if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
    "\n",
    "    # Calculate Loss (for both TRAIN and EVAL modes)\n",
    "    onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=10)\n",
    "    loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels,\n",
    "                                           logits=logits)\n",
    "\n",
    "    # Configure the Training Op (for TRAIN mode)\n",
    "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n",
    "        train_op = optimizer.minimize(loss=loss,\n",
    "                                      global_step=tf.train.get_global_step())\n",
    "        return tf.estimator.EstimatorSpec(mode=mode,\n",
    "                                          loss=loss, \n",
    "                                          train_op=train_op)\n",
    "\n",
    "    # Add evaluation metrics (for EVAL mode)\n",
    "    eval_metric_ops = {\n",
    "      \"accuracy\": tf.metrics.accuracy(\n",
    "          labels=labels, predictions=predictions[\"classes\"])}\n",
    "    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_save_summary_steps': 100, '_keep_checkpoint_every_n_hours': 10000, '_tf_random_seed': 1, '_log_step_count_steps': 100, '_keep_checkpoint_max': 5, '_save_checkpoints_secs': 600, '_model_dir': 'cnn2', '_save_checkpoints_steps': None, '_session_config': None}\n"
     ]
    }
   ],
   "source": [
    "# Create the Estimator\n",
    "classifier = tf.estimator.Estimator(model_fn=cnn_model_fn,\n",
    "                                    model_dir=\"cnn2\")\n",
    "\n",
    "# Set up logging for predictions\n",
    "tensors_to_log = {\"probabilities\": \"softmax_tensor\"}\n",
    "#logging_hook = tf.train.LoggingTensorHook(tensors=tensors_to_log,\n",
    "#                                          every_n_iter=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "accuracy_list = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    classifier.train(input_fn=train_input_fn,  steps=1000)\n",
    "    eval_results = classifier.evaluate(input_fn=test_input_fn, steps=1000)\n",
    "    accuracy_list.append(eval_results[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Low-level API\n",
    "\n",
    "The same model can be obtained using a low-level API which allows for more complex operations. An implementation for the same dataset already exist:\n",
    "https://www.tensorflow.org/tutorials/deep_cnn.\n",
    "I have modified the *inference* function adding 3 convolutional layer as in AlexNet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope('conv3') as scope:\n",
    "    kernel = _variable_with_weight_decay('weights',\n",
    "                                         shape=[3, 3, 64, 64],\n",
    "                                         stddev=5e-2,\n",
    "                                         wd=0.0)\n",
    "    conv = tf.nn.conv2d(pool2, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = _variable_on_cpu('biases', [64], tf.constant_initializer(0.1))\n",
    "    pre_activation = tf.nn.bias_add(conv, biases)\n",
    "    conv3 = tf.nn.relu(pre_activation, name=scope.name)\n",
    "    _activation_summary(conv3)\n",
    "\n",
    "with tf.variable_scope('conv4') as scope:\n",
    "    kernel = _variable_with_weight_decay('weights',\n",
    "                                       shape=[3, 3, 64, 64],\n",
    "                                       stddev=5e-2,\n",
    "                                       wd=0.0)\n",
    "    conv = tf.nn.conv2d(conv3, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = _variable_on_cpu('biases', [64], tf.constant_initializer(0.1))\n",
    "    pre_activation = tf.nn.bias_add(conv, biases)\n",
    "    conv4 = tf.nn.relu(pre_activation, name=scope.name)\n",
    "    _activation_summary(conv4)\n",
    "\n",
    "with tf.variable_scope('conv5') as scope:\n",
    "    kernel = _variable_with_weight_decay('weights',\n",
    "                                         shape=[3, 3, 64, 64],\n",
    "                                         stddev=5e-2,\n",
    "                                         wd=0.0)\n",
    "    conv = tf.nn.conv2d(conv4, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = _variable_on_cpu('biases', [64], tf.constant_initializer(0.1))\n",
    "    pre_activation = tf.nn.bias_add(conv, biases)\n",
    "    conv5 = tf.nn.relu(pre_activation, name=scope.name)\n",
    "    _activation_summary(conv5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing models\n",
    "Here we compare the accuracy of three models:\n",
    "- CNN1: obtained with the High-Level API\n",
    "- CNN2: obtained with the High-Level API with Dropout\n",
    "- CNN3: obtained with the Low-Level API (with normalization after the first max-pooling and before the second one)\n",
    "I have stopped the models after 10k steps, because training resulted to be very long on CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cnn1 = [0.23030098,0.30115432,0.36020401,0.39436743,0.42069858,0.43565246,0.45099324,0.4718371,0.48533824,0.49514434]\n",
    "cnn2 = [0.23935699, 0.34801352, 0.38385865, 0.41350588, 0.43493399, 0.45812279, 0.48355389, 0.50912708,0.52534425, 0.54138768]\n",
    "cnn3 = [0.099980, 0.538766, 0.696301, 0.732100, 0.754945, 0.770965, 0.781843,0.794897, 0.801325, 0.809335]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8W9Wd///XsS3v+77IjrM5+26yQhLWQlqgUKBhmbaU\nIYUBugDzg7YzLUz76JS9TKEsk1I6TQttaWeG0gyBXwsJWIHEhJA9wVktb/G+yZuk8/3jyrZsnFh2\nZF9J/jwfDz0kXR1LH4vkzcm5556jtNYIIYQILWFmFyCEEML/JNyFECIESbgLIUQIknAXQogQJOEu\nhBAhSMJdCCFCkIS7EEKEIAl3IYQIQRLuQggRgiLM+uD09HRdWFho1scLIURQ+uijj+q01hnDtTMt\n3AsLCyktLTXr44UQIigppU760k6GZYQQIgRJuAshRAiScBdCiBBk2pj7UHp6erDb7XR2dppdyriI\njo7GarVisVjMLkUIEWICKtztdjsJCQkUFhailDK7nDGltaa+vh673c7kyZPNLkcIEWICalims7OT\ntLS0kA92AKUUaWlpE+ZfKUKI8RVQ4Q5MiGDvNZF+VyHE+PJpWEYpdTnwNBAObNRa/3TQ60nAJqDA\n856Pa61/5edahRAiKHS5umjpaqGl23PzftzdwoKMBazMXTmmNQwb7kqpcOBZ4FLADuxUSr2utT7g\n1ewu4IDW+kqlVAZwWCn1W61195hUHQSeeeYZfvazn3H06FFqa2tJT083uyQhhI+01nQ4O2jpbqG1\nu3XIgB4c3n3tulvocnWd9f1vm3ub+eEOLAXKtNbHAJRSrwJXA97hroEEZYwzxAMNgNPPtQaVVatW\n8YUvfIG1a9eaXYoQE05vODucDtq622h3ttPe3T4ggIcLbKf77BGWYEkgMSqRxEjjlpGc0fe493hC\nZMKQxyLCxn4uiy+fkAeUez23A8sGtXkGeB2oBBKAL2ut3X6p0AT/9V//xeOPP45Sivnz5xMeHk5i\nYiKlpaVUV1fz6KOPct111/Huu+/y0EMPkZ6ezr59+1iyZAmbNm1CKcWiRYvM/jWECCpaa7pcXbT1\ntOHocdDW00Z7T/uA21DH+o53t/eFeLuzHfcwERSmwgaGb2QiOXE5Qwby4MfxlnjCw8LH6ZsZHX/9\n7+NzwG7gImAq8LZS6j2tdYt3I6XUBmADQEFBwVnf8OG/7OdAZctZ24zU7NxEfnjlnLO22b9/Pz/+\n8Y+x2Wykp6fT0NDAvffeS1VVFe+//z6HDh3iqquu4rrrrgPg448/Zv/+/eTm5rJq1SpKSko4//zz\n/Vq3EMHE5XbR0NlAjaPGuLXX0NjVSFt3W39P2iuUe4Pc0ePAqYf/B79CEW+JJ9YSS7wlnrjIOOIi\n4siKzSI2Ipb4yHjiLHHEWeIGtvM87w3pOEscYSrg5pT4jS/hXgHkez23eo55uxX4qdZaA2VKqePA\nTGCHdyOt9YvAiwDFxcV6tEWPpb///e9cf/31fWPkqampAHzxi18kLCyM2bNnU1NT09d+6dKlWK1W\nABYuXMiJEyck3EXI6nZ1c9pxmhpHjXHfXtMX4r3H6xx1Q4Z0nMUI4bjI/tBNi0k7YxB7H/d+HhMR\nIzPNfOBLuO8EpiulJmOE+nrgpkFtTgEXA+8ppbKAGcCxcylsuB72eIuKiup7bPw/7LPHw8PDcTon\n9KkGEcTauts47ThNtaO6L7gHBLmjhobOhs/8XExEDFmxWWTFZbE0eylZsVlkxmYa93HGfWp0akj3\nkgPRsOGutXYqpe4GtmBMhXxJa71fKXWH5/XngR8BLyul9gIKeEBrXTeGdY+Ziy66iGuuuYZ7772X\ntLQ0Gho++4dZiGDi1m4aOxsHBHV1e3Xf895j7T3tn/nZlKgUI6jjspiTPscIcc+t93i8JV560gHI\npzF3rfVmYPOgY897Pa4ELvNvaeaYM2cO3//+91mzZg3h4eGjPjH6H//xHzz66KNUV1czf/581q1b\nx8aNG/1crZiotNa09bRR31FPfWf9We9PO07T4+4Z8PPhKpz0mHSyYrOYljyNlbkrPxPambGZRIVH\nnaECEeiU9xDDeCouLtaDN+s4ePAgs2bNMqUes0zE31kMTWtNc1fzGUO6obNhwPNu92cvI1EoUqJT\nSI1OJS0mrS/AM2MzyY7N7gvutOi0gJ/tIYamlPpIa108XLuAWjhMiFDjcrto7GrsD+chgruho6Hv\n9aFORIar8L6wTotOY0ryFNKi00iLSRtwPC0mjZSoFAltAUi4C3FOOp2dVLZVYm+zU95ajr3VTkVb\nBRVtFdR11NHU1TTkfGtLmKUvlDNiM5iZOnNASHvfJ0YlyslIMWIS7kKchVu7qXXUYm8zQtveajdu\nbcZ9bUftgPYxETFYE6zkxeUxP2P+kGGdGpNKgiVBTkKKMSXhLia89p72AYHd2/u2t9mpaK0YMLat\nUGTHZWNNsHJ+3vlGkMfnYU2wYo23khqdKqEtAoKEuwh5TreTGkdNf2gP6n03djUOaJ9gScCaYGVa\n8jTWWtf2Bbc1wUpOXA6WcNk5SwQ+CXcREtzajb3VzuHGw5xqOdXX67a32alqqxpwojJCRZATn4M1\n3solky75TO87KSrJxN9ECP+QcB8jN998M6WlpVgsFpYuXcoLL7wge6X6SZeri7KmMg43HOZQwyEO\nNRzicMNhHE5HX5vU6FTy4vOYmzaXywsv7wvuvIQ8smKzxmVVPiHMJH/Cx8jNN9/Mpk2bALjpppvY\nuHEjd955p8lVBZ/mruYBAX6o8RDHm4739cTjLHHMSJnB1dOuZmbqTGakzqAwsZA4S5zJlQthLgn3\nIfhjyd9169b1vd/SpUux2+0m/kaBT2tNRVtFX4D3Bnp1e3Vfm8zYTGamzmStdS0zU2cyK3UWeQl5\nMk1QiCEEbrj/34NQvde/75k9D6746Vmb+HvJ356eHn7zm9/w9NNP+/d3CWI9rh6ONh/t74177lt7\nWgFjne3JiZNZnLmYmakz+3rkqdGpJlcuRPAI3HA3ib+X/P2nf/onVq9ezQUXXDCOv0XgaOlu4XDD\n4f4QbzxMWVNZ3y43MRExFKUUsW7KOmakzmBW6iymJU8jOiLa5MqFCG6BG+7D9LDH22iW/H344Yep\nra3lhRdeGJ8iTaS1prq92hhOaezvkVe09S/9nx6TzozUGazKXdXXI89PyJfL5YUYA4Eb7ibx15K/\nGzduZMuWLfztb38jLCw0x4Tbe9rZXrmdbfZtvFfxHnUdxirPCsWkxEnMS5/HdUXX9QV5eoxsEi7E\neJFwH8RfS/7ecccdTJo0iRUrVgBw7bXX8oMf/MCfpZriVMspttm3sdW+ldKaUpxuJwmWBFblrWJJ\n1hJmps6kKKWIWEus2aUKMaHJkr8mC/Tfucfdw8c1H7PVvpVt9m2caDkBwJSkKayxruEC6wUszFyI\nJUzm8AsxHmTJXzFq9R31vF/xPtvs27BV2mjracMSZuG87PNYP3M9q62ryU/IH/6NhBCmkXAXaK05\n3HiYreVb2Vaxjb21e9FoMmIy+Fzh57jAegErclbIUIsQQUTCfYJy9Dj4sOpDtlVsY5t9G6cdpwGY\nlz6POxfeyRrrGmamzpQLhIQIUj6Fu1LqcuBpjA2yN2qtfzro9X8GbvZ6z1lAhtZadpcOIBVtFX0n\nQ3dW7aTb3U2cJY6VuStZbV3N+Xnny4wWIULEsOGulAoHngUuBezATqXU61rrA71ttNaPAY952l8J\nfEeC3XxOt5NPaj9hq30r79nfo6ypDICChAJumHEDa/LXsCRziSxhK0QI8qXnvhQo01ofA1BKvQpc\nDRw4Q/sbgVf8U54YqeauZt6veJ+t9q2UVJTQ0t1ChIpgSdYSrim+htXW1RQmFZpdphBijPkS7nlA\nuddzO7BsqIZKqVjgcuDucy8tuN12222UlpaitaaoqIiXX36Z+Ph4v3+O1pqyprK+3vnu2t24tZvU\n6FQuzL+Q1dbVrMhdQUJkgt8/WwgRuPx9QvVKoORMQzJKqQ3ABoCCggI/f3Rgeeqpp0hMTATg3nvv\n5ZlnnuHBBx/062e4tZv7t97P2yffBmBW6ixun3c7a6xrmJM+R06GCjGB+RLuFYD3pGar59hQ1nOW\nIRmt9YvAi2BcxORjjePOH0v+9ga71pqOjo4x2Vfzl3t/ydsn3+b2ebfz5RlfJisuy++fIYQITr6E\n+05gulJqMkaorwduGtxIKZUErAFu8Udhj+x4hEMNh/zxVn1mps7kgaUPnLWNP5f8vfXWW9m8eTOz\nZ8/miSee8Ovv8kHVBzyz+xmumHwF9yy6RzZlFkIMMOy/27XWTowx9C3AQeAPWuv9Sqk7lFJ3eDW9\nBnhLa90+NqWOj9Eu+RsWFta35G+vX/3qV1RWVjJr1ix+//vf+63GmvYaHtj2AIWJhTy04iEJdiHE\nZ/g05q613gxsHnTs+UHPXwZe9ldhw/Wwx9tolvztPbZ+/XoeffRRbr311nOuo8fdw/1b76fT2clT\nlz8lV40KIYYkZ9wGueiii/jjH/9IfX09wKiW/NVaU1ZW1vf49ddfZ+bMmX6p78nSJ9ldu5uHVz3M\nlKQpfnlPIUTokeUHBvHHkr9aa7761a/S0tKC1poFCxbw3HPPnXNtb554k00HN3HzrJu5vPDyc34/\nIUTokiV/Tebr73ys+Rg3vnEj01Om86vP/UquKhVigvJ1yV8ZlgkCjh4H9717H1HhUTy+5nEJdiHE\nsGRYJsBprXl4+8McbTrKC5e+QHZcttklCSGCQMD13M0aJjKDL7/r7w//ns3HN3PXwrtYkbtiHKoS\nQoSCgAr36Oho6uvrJ0TAa62pr68nOjr6jG321u7lkZ2PcEHeBdw+//ZxrE4IEewCaljGarVit9up\nra01u5RxER0djdVqHfK1ps4m7tt6H5kxmfz7Bf8u68QIIUYkoMLdYrEwefJks8swnVu7efD9B6nr\nqOM3V/yGpKgks0sSQgQZ6Q4GoBf2vEBJRQkPLn2QOelzzC5HCBGEJNwDTElFCc/tfo4rp1zJ9UXX\nm12OECJISbgHkKq2Kh5870GmJk/lX1f8qywIJoQYNQn3ANHjMhYE63H38NTap4iJiDG7JCFEEAuo\nE6oT2WOlj7Gnbg9PrHlC9jgVQpwz6bkHgL8e+yuvHHqFr8z+CpcVXmZ2OUKIECDhbrKjTUd5ePvD\nLM5czLeXfNvscoQQIULC3UTtPe18593vEBMRw2NrHsMSJguCCSH8Q8bcTaK15iHbQ5xsOcl/Xvqf\nZMZmml2SECKESM/dJL879DvePPEm9yy6h6U5S80uRwgRYnwKd6XU5Uqpw0qpMqXUg2dos1YptVsp\ntV8ptdW/ZYaW3ad38/jOx1lrXcvX537d7HKEECFo2GEZpVQ48CxwKWAHdiqlXtdaH/Bqkwz8Arhc\na31KKSVjDGfQ0NnA/VvvJysuix+f/2NZEEwIMSZ8SZalQJnW+pjWuht4Fbh6UJubgD9rrU8BaK1P\n+7fM0OByu3hg2wM0djby1NqnZEEwIcSY8SXc84Byr+d2zzFvRUCKUupdpdRHSqmv+KvAUPLcJ8/x\nQdUHfG/Z95iVNrH2ihVCjC9/zZaJAJYAFwMxwHal1Ada6yPejZRSG4ANAAUFBX766OCwzb6NF/a8\nwNVTr+ba6deaXY4QIsT50nOvAPK9nls9x7zZgS1a63atdR2wDVgw+I201i9qrYu11sUZGRmjrTno\nVLRV8N33vsuMlBl8f/n3ZUEwIcSY8yXcdwLTlVKTlVKRwHrg9UFt/hc4XykVoZSKBZYBB/1banDq\ndnVz37v34dZunlz7pCwIJoQYF8MOy2itnUqpu4EtQDjwktZ6v1LqDs/rz2utDyql3gT2AG5go9Z6\n31gWHiwe2fEI++v387MLf0ZB4sQaihJCmMenMXet9WZg86Bjzw96/hjwmP9KC35/OfoX/nDkD9w6\n51YuLrjY7HKEEBOITLIeI0caj/Bv2/+NJVlL+Obib5pdjhBigpFwHwNt3W3c++69xEfG89jqx4gI\nkyV8hBDjS1LHz7TW/MD2A+ytdjZetpGM2IkzK0gIETik5+5nvznwG94++TbfWvwtirOLzS5HCDFB\nSbj70a6aXTz50ZNclH8RX5vzNbPLEUJMYBLuflLXUcf9W+8nNz6XH53/I7lQSQhhKhlz9wOn28kD\n2x6gpbuF5y55jsTIRLNLEkJMcNJz94Nndz/Ljuod/Mvyf2FG6gyzyxFCCAn3c/XOqXfYuHcjX5r+\nJb447YtmlyOEEICE+zkpby3n++9/n1mps/jusu+aXY4QQvSRMfdR6nJ1cd+794GCJ9Y+QVR4lNkl\nCSECibMLmsqh8QQ0Hjfum04a9wtuhBV3jenHS7iP0q/3/5qDDQf5+UU/Jz8hf/gfEEKEFq2h7fTA\n0Pa+tVQCur99RDSkFELyJIgb+51IJdxH6d3yd5mfMZ+1+WvNLkUIMVa6HYOC2+tx00nocQxsn5Br\nBPjk1ca99y0uE8LGbyRcwn0Umjqb2Fe3j28s+IbZpQghzoXbDa1Vn+119wZ6W83A9pHxRlCnTYVp\nFxu98N7wTi4AS/T41n8WEu6j8EH1B2g0q3JXmV2KEGI4Xa2fDe++ED8Fru7+tioMEq2QMgmmX/bZ\n3ndsGgTJBYoS7qNgq7CRYElgbvpcs0sRQrhdRu+74fgQAX4cHPUD20clQWohZM2BmZ/vHwdPKYSk\nfIiIHOdfYGxIuI+Q1pqSyhKW5y6XpXyFGC9dbUMH95C973BIshphPevKgT3v5EkQmzru5ZtB0mmE\njjYd5bTjNCtzV5pdihCh4zNj34N64e21A9sP1ftOmezpfVsh3DK+9QcgCfcRKqksAZBwF2KkutuH\nHvduOO7pfXf1t1Vh/b3vGVf0B3fvLSYlaMa+zeJTuCulLgeextgge6PW+qeDXl8L/C9w3HPoz1rr\nf/NjnQFje+V2ChMLyY3PNbsUIQJXSxVUfNR/O30Q2k8PbBOZYPS+M2d5ArzQuKVONsa+pfd9ToYN\nd6VUOPAscClgB3YqpV7XWh8Y1PQ9rfUXxqDGgNHp7KS0ppTriq4zuxQhAkdnC1R+7BXmu6C10ngt\nLMIYOim6zOh9p07uH0KR3veY8qXnvhQo01ofA1BKvQpcDQwO95C3q2YXXa4uGZIRE5ezG2r29Yd4\nxUdQd4S+KzFTp0DhKshbYtyy54ElxtSSJypfwj0PKPd6bgeWDdFupVJqD1AB3K+13j+4gVJqA7AB\noKCgYOTVmqyksgRLmIXiLNk+T0wAbjc0HBs4vFK9p39mSlyGEeDzroO8xZC7eMLMRDlXWusx39DH\nXydUdwEFWus2pdQ64H+A6YMbaa1fBF4EKC4u1oNfD3S2ShuLMxcTa4k1uxQh/K+1emCQV3wMXc3G\na5Y4yF0Ey+7w9MoXG+PiMqyC1hpHt4uG9m7q27tp9Nw3tHfR0N7jue/uu9W3d3PrykLuvWxs937w\nJdwrAO+VsayeY3201i1ejzcrpX6hlErXWtf5p0zz1bTXUNZUxpVLrjS7FCHOXWcLVO0eOE7e4vlr\nrcKNcfK51/YPr2TMgLBwc2seJ263pqmjxyuQ+0O63iukvW9dTveQ72UJV6TGRZIaF0VqnIV5Kcmk\nxUWysCB5zH8PX8J9JzBdKTUZI9TXAzd5N1BKZQM1WmutlFqKsU58/WfeKYjZKm0AsuSACD7Obji9\nf+A4ee1h+sbJUyZDwYqB4+SRofOvU601LZ1Oqpo7qGvtpn5QT9q7x93Q3k2joxv3GcYV4qMiSI2L\nJCUukqzEaGblJHrC23OLjSQ1PpI0z/P4qAjT9lMeNty11k6l1N3AFoypkC9prfcrpe7wvP48cB1w\np1LKCXQA67XWQTfscja2ShvpMekUpRSZXYoQZzZ4nLxyF1Tt6Z9DHptuBPica/uHV4J4nLw3uKub\nO6ls7qC6uZOqpg6qmjs9N+Oxo9v1mZ8NU5ASawR1alwk0zLjSYnrD2bvW1pcFMmxFqItwfOvF5/G\n3LXWm4HNg4497/X4GeAZ/5YWOFxuF9urtrPGusa0/wsLMSTv+eSVuwaNk8dCzkJYtsE42Zm3xFi5\nMEj+DJ8tuKtbOqlsMo61DwruMAWZCdFkJ0UzIzuBNUWZ5CYbzzPio0iLN4ZJkmIshIcFx3cxGnKF\nqg8O1B+guatZpkAKc3U0GfPJK3f1D6+0VhmvhUVA5myvcfLFkD4DwgPzr7jWmtYuJ1VN/b3rKk94\nny24lYLMhChykmIoyjKCOycpmpzkaOM+KYaMhCgs4bKDaGD+lw8wvePty3OWm1yJmDB6Or3mk3vG\nyus/7X89dSoUXmCEeIDOJ+/scXG0to0jNa0cr3NQPSjERxrc2UkxZEpw+0zC3Qe2ShuzUmeRFpNm\ndikiFLldxoVA3ic8a/aDu8d4PT7LCPAFXzbucxcZV3cGCKfLzYl6B0dqWjlc3Wrc17Ryoq6978Rk\nb3BnJ8UwPTOeC6ank5sUQ3ZStGfIRILb3yTch9Ha3contZ9w69xbzS5FhAKtobm8P8QrdhlTErvb\njNcjEyBvEay8u3+cPDE3IMbJ3W5NRVNHX3gfqW7lcE0bR0+30e0ypgKGKShMi6MoK4EvzM9lRlYC\nM7LjmZQWJ8E9ziTch7Gjagcu7ZLxdjE6job+IK/03PcuXxseaQynLLixfxpi2rRx3WdzKFpralu7\nOOzVEz9S08anNa0DhlLykmMoyopndVE6M7ISKMpKYFpmfFDNKAllEu7DKKksITYiloUZC80uRQS6\n9nqo/sSYelj1iRHmjSc8LypIL4Jpl3rGyRdD1lyIiDKzYpodPUaI9/XEjTBvcvT0tUmPj6QoK4Hr\ni/OZkW2E+PSseBKjZdXGQCbhfhZaa2yVNpZmL8Uiy4+KXr1DK1V7jLVWeu9bvC7cTrQaAb7ka0aP\nPGchRCeaVrKj28mnNW2fCfGalv411BOiIijKTuCKuTnMyIqnyBPk6fHm/g9IjI6E+1mcaj1FRVsF\nX53zVbNLEWZxOY1ZKn1B/glU74XOJuN1FQZp02HSSsieDznzjXsTLgxq73JS3ujA3tCBvdFBeWMH\nJ+vbOVLTxqkGR1+7qIgwpmfFs2qaZzglO4EZWQnkJEXLdRwhRML9LEoqjF2XZMmBCaKnA2oO9A+t\nVO8xZq04O43Xw6OMNVdmX+0J8QXG83G6VL+j24W90YG9sT+8e5+XNzho9BpKAYi2hJGfEss8axLX\nLbFSlJXAjOwEClJjQ/riHWGQcD8LW6UNa7yVgsTgW55YDKOj0RPge/uHVuqOgPacMIxOMnrgxbf1\n98bTi8b0oqDOHhcVTR394d3QMSDM69q6B7SPjAjDmhKDNSWWuXlJ5KfEep7HkJ8aS1pcpPTEJzAJ\n9zPocfWwo3oHV029yuxSxLnQGloqB46NV+2B5lP9bRJyjQCfdaUxeyVnPiRP8vv0w26nm0rv8O4L\nbqPnfbq1a0B7S7giL9kI70tmZZGf6hXeKbGkx0cRJj1wcQYS7mewu3Y3Hc4OVuSuMLsU4SuX01g0\nq3rQiU5H7wKlCtKmgrUYzvu60RvPng/xGX75+G6nm+rmTqO33dSBvcErvBsdVLd04r2cXniYIjc5\nGmtyLGuKMvrCu/c+MyFahk/EqEm4n0FJRQkRKoJl2UNtOiVMpTW0nTYuzz99wBgnP73fWMa2b3w8\n0rPx8jrIWWD0yLPmQlT8qD+2s8fV1/M2hk8cVDT2Px8c3mEKcpJiyEuJYeXU9AFDJtaUGLITo4mQ\nC3vEGJFwPwNbpY35GfOJjxx9GAg/6G6H04e8gny/ce/w2i4gPhuyZsN5/2ic4MyeZyyaFRE5oo9y\ndDv7wto+RHjXDho2CQ9T5CRFk5dshHeeJ7ytKTFYk2PJToomMkLCW5hDwn0IdR11HGw4yD2L7jG7\nlInD7TKGVGr2eXriniBvPEHfphKWOKM3PvPzkDnHCPTMORDn25o/LZ09VDR2eALb0Xfysve+oX3g\nCUtLuCI32QjrC2dkYPWcsMxLjsGaGktWQpT0vEXAknAfwvbK7YBMgRwTvUMqp/cb4T3UkIoKMy7D\nz1kAC28ylrLNmg3JhWe9NN/pcvPp6TZO1jsGhrcnzFs6nQPaR0WEeXrbsczJTervdafEkJccS2aC\nnLAUwUvCfQjbK7eTHJXMzNSZZpcS3HqHVPqCfKghlSwjvHuHVDJnG/t1+rB8bU1LJx+fauTj8iY+\nPtXEXnszHT39a5/ERob39bSXTErpGzbpnYGSHi9TBUXoknAfxK3d2CptrMhZQfgE2RD4nGkNjceN\n2Sm9wymfGVKJ7T/BmTV3xEMqHd0u9lU28/GpRnZ7wryq2ejpW8IVs3OT+PJ5+SzMT2ZqRjx5KTGk\nxFokvMWEJeE+yJHGI9R31rMyT1aBPCOtjQt+TrwPJ23GrbXSeE2FGRtJ5Mw3VjvMmuPTkMrAt9cc\nr2vn41NNRpCXN3KwqhWXZ3Fwa0oMxYWpLMxPZlFBMrNzEmUlQiEG8SnclVKXA09jbJC9UWv90zO0\nOw/YjrFB9mt+q3Ic9S45IEv8enG7jaGVk7b+QHfUGa/FZ0PhKmNtlbwlkDFzxDsCNTm6+3rju8uN\nW3OHcSl9XGQ4C/KT+cbqKSwqSGFhfjIZCbKQlRDDGTbclVLhwLPApYAd2KmUel1rfWCIdo8Ab41F\noePFVmljesp0MmMzzS7FPC6nsb7KSRucKIFTNuj0bLqcVADTLzXCfNIqSJ0yois5e1xuDle39o2V\n7z7VxLG6dsB4m6LMBK6Ym+3placwLTNeLuQRYhR86bkvBcq01scAlFKvAlcDBwa1uwf4E3CeXysc\nR44eB7tO7+LmmTebXcr4cnYbGy+f9PTKT30I3a3Ga6lTYdZVUHi+EejJI1tnp6q5g49PNfWNle+x\nN9PlNHbtSY+PZGF+Cl9aYmVRfjLzrEkkyBrhQviFL+GeB5R7PbcDAy7bVErlAdcAF3KWcFdKbQA2\nABQUBN5iXKU1pTjdztAfb+/pAHspnCwxbuU7wdlhvJYxE+bfYAy1FKyExByf39bR7WSvvbmvR/5x\neWPfeuFXA8fcAAAUUUlEQVSR4WHMyUvk5mWTWFiQzKL8ZKwpMXLCU4gx4q8Tqj8DHtBau8/2l1Vr\n/SLwIkBxcbE+Y0OTlFSUEB0ezZKsJWaX4l9dbVD+oefkZ4mx1ZurG1CQPdfYUGLSSuMWl+7z2zpd\nbj4ub2Lr4Vq2fVrL/sqWvpOek9JiWT4ljUX5ySwsSGFWTgJREXLSU4jx4ku4VwD5Xs+tnmPeioFX\nPcGeDqxTSjm11v/jlyrHia3SxpLsJUSFB/kJu44mOPVBf8+8crexlK0Kh9yFsOwOY7y8YDnEJI/o\nrauaO9h6uJatR2p5v6yO1k4n4WGKxQXJ3LlmKosnJbPAmkya7N4jhKl8CfedwHSl1GSMUF8P3OTd\nQGs9ufexUupl4I1gC/aKtgpOtJzghhk3mF3KyLXXGyc9T5QY4+bV+wBtLJ6VtwTO/7YR5vnLRrxw\nVpfTRemJRrYeqWXr4VoO1xhj8dmJ0aybm8PaGRmsnJZOUoyMlQsRSIYNd621Uyl1N7AFYyrkS1rr\n/UqpOzyvPz/GNY4LW6UNCJIpkG4XVOyCI2/CkS1Qs9c4HhED+efB2geNMLcWj3haIsDJ+va+MLcd\nraejx0VkeBjnTU7hS0tmsqYok6KseBkvFyKA+TTmrrXeDGwedGzIUNdaf+3cyxp/tgobWbFZTEma\nYnYpQ+tshrK/wadvGTdHvTHMUrAcLvpXYzZL7uIRr4QIxonQD47V9w23nKg39tssSI3l+mIra4oy\nWD4ljbgoueZNiGAhf1sBp9vJh1UfcmnhpYHTG9Ua6sv6e+entoPbCTEpMO1SKPocTL1oVBsxa60p\nO91m9M6P1PLh8Qa6nW6iLWGsnJrOrasms6Yog8L0uDH4xYQQ40HCHdhbt5fWnlbzh2ScXcYJ0CNv\nGaHeeNw4njkbVt4DRZdDXvGo9vFs6ezBVlbXN9xS6VmXZXpmPF9ZPok1MzI4rzBVLuMXIkRIuGNM\ngQxTYSzPWT7+H95a4xlq2QJH34HuNgiPgilrYMVdRg99hBcOAbjdmgNVLX1h/tGpRlxuTUJUBKum\npXPPxRmsLsogL3nkY/JCiMAn4Y6xxO/c9LkkRSWN/Ye53cal/Ue2GLfKXcbxhFyYd73RO5+8GiJj\nR/zWDe3dvPdpbd+887o2Y/OJuXmJ3LFmCmuKMllUkIxFNpgQIuRN+HBv7mpmX/0+NszfMHYf0tUG\nx941hlo+fQvaagAF1vPgon8xAj1r7ojWaOl1pKaVN/ZUsfVILXvsTWgNKbEWVhdlsKYogwumZ8hC\nW0JMQBM+3LdXbcet3f7fdanhuNEz/3SLsZKiqxuiEmHaxTD9c8biWyO4GnSwfRXN/Pzvn7Jlfw1h\nChbmJ/Pti4tYMyODeXlJstiWEBPchA93W4WNBEsCc9PnntsbuXqMS/yPvGmcEK07bBxPmw5LNxi9\n84LlEH5uF/t8dLKRZ/7+Ke8criUhOoJvXjydr66YJFeECiEGmNDhrrWmpLKE5bnLiQgbxVfRXg9l\nbxs99LK/QVczhFmMOefFt8L0yyBtql/q/OBYA8+88yklZfWkxFr458/N4B9WTCJRVlEUQgxhQof7\nseZjnHacHvkUSLcb/v8fwPZnQbshLhNmX2n0zqeshagEv9Sntea9T+v4+d8/ZeeJRtLjo/jeupnc\nvGySXFAkhDirCZ0Qo9p1ydkF/3Mn7PsTLLoFim+DnIU+byHnC601fzt4mp+/U8Yn5U3kJEXz8FVz\n+PJ5+TIPXQjhkwkd7rZKG4WJheTG5/r2A53N8OrNcOI9uORhWPWtUc1wORO3W/Pm/mp+/vcyDla1\nYE2J4SfXzONLS/JkuVwhxIhM2HDvdHZSWlPKdUXX+fYDLVXw2+ug9hBc8wIsWO+3WpwuN2/sqeKZ\nd8ooO93GlPQ4Hr9+AVcvzJU56UKIUZmw4b6rZhddri7fhmRqj8CmLxmLdd30B2M6ox/0uNz8964K\nfvFuGSfqHRRlxfMfNy7i8/NyZCqjEOKcTNhwt1XasIRZKM4qPnvD8h3wuxsgLAJu/SvkLjrnz+5y\nuvhjqZ3n3j1KRVMHc3ITef6WJVw2O4swCXUhhB9M2HAvqSxhceZiYi1nucz/0GZ47evGPqK3/AlS\nz2054I5uF6/sOMUL245S09LFooJkfvzFuaydkRE4q1EKIULChAz3mvYayprKuHLJlWdu9NHL8MZ3\njJkwN/0B4jNG/XltXU42fXCSje8do66tm2WTU3nyhoWsnJomoS6EGBMTMtx7d10acskBrWHrI/Du\nvxvrpl//8oi3puvV3NHDr20neKnkOE2OHi6Yns49F01n6eSRr8EuhBAjMWHDPT0mnaKUooEvuJzw\n13th169h4c1w5dOjWi6gob2bl94/zq9tJ2jtcnLJrEzuunAaiwpS/PQbCCHE2U24cHe5XWyv2s4a\n65qBQyLdDvjTbXB4M1xwv7Fa4wiHTE63drLxveNs+uAkHT0urpibzV0XTmNO7jgsJSyEEF58Cnel\n1OXA0xgbZG/UWv900OtXAz8C3IAT+LbW+n0/1+oXBxsO0tzVzIrcFf0HHQ3wuy+DfSesexyW3j6i\n96xq7uCFrcd4ZccpelxurlqQy10XTmN6ln+WIRBCiJEaNtyVUuHAs8ClgB3YqZR6XWt9wKvZ34DX\ntdZaKTUf+AMwcywKPle9Sw6syPGEe+NJYw570ym44dcw+2qf36u8wcEv3j3Kax+VozVcuziPO9dO\nY7LsPSqEMJkvPfelQJnW+hiAUupV4GqgL9y11m1e7eMA7c8i/clWaWNW6izSYtKgei9sug6cHfCV\n/4FJvq0x09jezRNvH+aVHeWEK8UNxfncsWYq+akj3z1JCCHGgi/hngeUez23A8sGN1JKXQP8O5AJ\nfH6oN1JKbQA2ABQUjHxf0HPV2t3KJ7WfcOvcW+HYVvj9LcYKjl/fApmzhv15p8vN73ac4om3jtDW\n5eSmpQXcdeE0spOix6F6IYTwnd9OqGqt/xv4b6XUaozx90uGaPMi8CJAcXHxuPfud1TvwKVdrOzs\nMYZi0qbBLa9BknXYn91+tJ6H/7KfQ9WtrJyaxg+vnMOMbBlTF0IEJl/CvQLI93pu9RwbktZ6m1Jq\nilIqXWtdd64F+pOtwkassrDwrR9BwUq48XcQc/bpifZGBz/ZfJDNe6vJS47h+VsW87k52XLxkRAi\noPkS7juB6UqpyRihvh64ybuBUmoacNRzQnUxEAXU+7vYc6FdLkqOvsHStmYss66Ca/8TLGceTuno\ndvH81qM8v/UoSsG9lxaxYfUUWU9dCBEUhg13rbVTKXU3sAVjKuRLWuv9Sqk7PK8/D3wJ+IpSqgfo\nAL6stQ6ck6rObk7999epcDn4auYSuPZlCBs6pLXWbN5bzU82H6SiqYMvzM/hu+tmkZccM741CyHE\nOfBpzF1rvRnYPOjY816PHwEe8W9pftLVCr+/hZLaUkhPZdWlj50x2A9Vt/DQ6/v54FgDs3ISeeKG\nBSyfkjbOBQshxLkL7StUW2uMDTZq9rN9/hqs7k4KkiZ9plmTo5sn3z7Cpg9Okhhj4cdfnMuNSwtk\nTXUhRNAK3XCvK4NN10J7LT3rf8uO0oe4curAVSBdbu2Z2niYlo4e/mH5JL5zaRHJsZEmFS2EEP4R\nmuFuLzU22EDB195gd7gbh9MxYMmBD47V89DrxtTGFVPS+OFVs5mZnWhezUII4UehF+5HtsAfvwbx\nmXDLnyFtKiUf/YwIFcGy7GVUNHXwk80H+eueKvKSY3ju5sVcPlemNgohQktohfuu38BfvgXZc+Hm\n14yAx1hyYF76fH65rYrntpYB8J1LivjGGpnaKIQITaER7lrDtsfhnR/D1Ivghv8ylhUA6hx1HGw4\nSFTr59lmP8Ln5+fwPZnaKIQIccEf7m4XbP5nKP0lzF8PV/0cIowTooeqW/jOG5vAAol6Dj+/fTkr\npsrURiFE6AvucO/pgD/9Ixx6A1Z9Gy55CJSiydHNU28fYdOHp4jL201MdCL/d8d6oiwj31VJCCGC\nUfCGu6MBXrkRyj+EKx6FZd/A5da88uFJnnjrMM0dPdy0NJ/3uk6wLGeVBLsQYkIJznBvKjdWdWw8\nDtf/CuZcw4fH6nnoLwc4WNXC8imp/PDKOaioSv73L/WszPNtnXYhhAgVwRfuNfuNDTa62+GWP1OZ\nUsxPfreLNzxTG5+9aTHr5hlTG1/a9xoAK3Ml3IUQE0vwhXtHE0RE0rX+D7xwKJpfvPsuWsO3Lp7O\nHWumEhPZP7XRVmFjesp0MmMzTSxYCCHGX9CFu560krcu/As/2lSGvbGDdfOy+d66WVhTBm5x5+hx\nsOv0Lm6aedMZ3kkIIUJX0IX7H0rLeeBPe5mRlcDvbl/GyqnpQ7YrrSmlx90j4+1CiAkp6ML9ygW5\nuDVcv8RKRHjYGduVVJQQHR7Nkqwl41idEEIEhqAL99jICG5cOvzm2rZKG0uylxAVHjUOVQkhRGA5\nc9c3iFW2VXKi5QSrcleZXYoQQpgiJMO9pLIEkCmQQoiJy6dwV0pdrpQ6rJQqU0o9OMTrNyul9iil\n9iqlbEqpBf4v1Xe2ChtZsVlMSZpiZhlCCGGaYcNdKRUOPAtcAcwGblRKzR7U7DiwRms9D/gR8KK/\nC/WV0+3kw6oPWZW3StZoF0JMWL703JcCZVrrY1rrbuBV4GrvBlprm9a60fP0A8Dq3zJ9t69uH609\nrTIkI4SY0HwJ9zyg3Ou53XPsTG4D/u9cijoXJZUlhKkwlucsN6sEIYQwnV+nQiqlLsQI9/PP8PoG\nYANAQcHw0xlHw1ZhY27aXJKiksbk/YUQIhj40nOvAPK9nls9xwZQSs0HNgJXa63rh3ojrfWLWuti\nrXVxRkbGaOo9q+auZvbV75OrUoUQE54v4b4TmK6UmqyUigTWA697N1BKFQB/Bv5Ba33E/2X6ZnvV\ndtzaLfPbhRAT3rDDMlprp1LqbmALEA68pLXer5S6w/P688APgDTgF54ZKk6tdfHYlT207ZXbSbAk\nMDd97nh/tBBCBBSfxty11puBzYOOPe/1+B+Bf/RvaSOjtaakooTlucuJCAu6VRWEEMKvQuYK1WPN\nx6hx1LAid4XZpQghhOlCJtxLKowlB2S8XQghQijcbZU2ChMLyY3PNbsUIYQwXUiEe5eri9KaUlbl\nSa9dCCEgRML9o5qP6HJ1yZIDQgjhERLhbquwYQmzUJw17rMvhRAiIIVEuJdUlrA4czGxltjhGwsh\nxAQQ9OFe015DWVOZLDkghBBegj7ct1dtB2QKpBBCeAv6cLdV2EiPSacopcjsUoQQImAEdbi73C62\nV21nZe5K2XVJCCG8BHW4H2w4SFNXkyw5IIQQgwR1uPcuObAiR8JdCCG8BXW42yptzEqdRVpMmtml\nCCFEQAnacG/rbmNP7R5ZckAIIYYQtOH+YfWHOLVTlhwQQoghBG242ypsxEbEsjBjodmlCCFEwAnK\ncNdaU1JZwtLspVjCLWaXI4QQAScow728tZyKtgpZckAIIc7Ap3BXSl2ulDqslCpTSj04xOszlVLb\nlVJdSqn7/V/mQCWVsuuSEEKczbA7SSulwoFngUsBO7BTKfW61vqAV7MG4JvAF8ekykFsFTas8VYK\nEgvG4+OEECLo+NJzXwqUaa2Paa27gVeBq70baK1Pa613Aj1jUOMAPa4edlTvkFkyQghxFr6Eex5Q\n7vXc7jlmit21u3E4HTLeLoQQZzGuJ1SVUhuUUqVKqdLa2tpRvUdEWATn553Psuxlfq5OCCFChy/h\nXgHkez23eo6NmNb6Ra11sda6OCMjYzRvwaLMRTx3yXPER8aP6ueFEGIi8CXcdwLTlVKTlVKRwHrg\n9bEtSwghxLkYdraM1tqplLob2AKEAy9prfcrpe7wvP68UiobKAUSAbdS6tvAbK11yxjWLoQQ4gyG\nDXcArfVmYPOgY897Pa7GGK4RQggRAILyClUhhBBnJ+EuhBAhSMJdCCFCkIS7EEKEIAl3IYQIQUpr\nbc4HK1ULnBzlj6cDdX4sJ9jJ9zGQfB/95LsYKBS+j0la62GvAjUt3M+FUqpUa11sdh2BQr6PgeT7\n6CffxUAT6fuQYRkhhAhBEu5CCBGCgjXcXzS7gAAj38dA8n30k+9ioAnzfQTlmLsQQoizC9aeuxBC\niLMIunAfbrPuiUQpla+UekcpdUAptV8p9S2zazKbUipcKfWxUuoNs2sxm1IqWSn1mlLqkFLqoFJq\nhdk1mUUp9R3P35F9SqlXlFLRZtc01oIq3L02674CmA3cqJSabW5VpnIC92mtZwPLgbsm+PcB8C3g\noNlFBIingTe11jOBBUzQ70UplQd8EyjWWs/FWLp8vblVjb2gCnd82Kx7ItFaV2mtd3ket2L85TVt\nf1uzKaWswOeBjWbXYjalVBKwGvglgNa6W2vdZG5VpooAYpRSEUAsUGlyPWMu2MI9oDbrDiRKqUJg\nEfChuZWY6mfA/we4zS4kAEwGaoFfeYapNiql4swuygxa6wrgceAUUAU0a63fMreqsRds4S6GoJSK\nB/4EfHui7n6llPoCcFpr/ZHZtQSICGAx8JzWehHQDkzIc1RKqRSMf+FPBnKBOKXULeZWNfaCLdz9\ntll3qFBKWTCC/bda6z+bXY+JVgFXKaVOYAzXXaSU2mRuSaayA3atde+/5F7DCPuJ6BLguNa6Vmvd\nA/wZWGlyTWMu2MJdNuv2opRSGGOqB7XWT5pdj5m01t/VWlu11oUYfy7+rrUO+d7ZmXi2vixXSs3w\nHLoYOGBiSWY6BSxXSsV6/s5czAQ4uezTHqqB4kybdZtclplWAf8A7FVK7fYc+55nz1sh7gF+6+kI\nHQNuNbkeU2itP1RKvQbswphh9jET4EpVuUJVCCFCULANywghhPCBhLsQQoQgCXchhAhBEu5CCBGC\nJNyFECIESbgLIUQIknAXQogQJOEuhBAh6P8B6AMVOAhUO4AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fee7f8cd6a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cnn1, label=\"cnn1\")\n",
    "plt.plot(cnn2, label=\"cnn2\")\n",
    "plt.plot(cnn3, label=\"cnn3\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as we can see the first two networks do not achieve satisfying performances compared to the state-of-the-art, while cnn3 has already reached a competitive accuracy after only 10k steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cnn1_2k = [0.52739704, 0.54876202, 0.56877685, 0.58505714, 0.59643441]\n",
    "cnn2_2k = [0.56489229, 0.58396763, 0.59968734, 0.61042511, 0.62248135]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VfX9x/HXN5uRMBKSAJlswpYQkD0EERGstYqzRa1V\na5e12tYO189StFao+lMLuLA/2zrRiqAMZYkEUCELQsJIIIMEkpCQeT+/P84V0ohwA0nOHZ/n48Gj\nufeeJJ97TN49+ZzP+R4jIiillPIdfnYXoJRSqm1p8CullI/R4FdKKR+jwa+UUj5Gg18ppXyMBr9S\nSvkYDX6llPIxGvxKKeVjNPiVUsrHBNhdwJlERERIQkKC3WUopZTH2L59+1ER6ebKtm4Z/AkJCaSm\nptpdhlJKeQxjzAFXt9VWj1JK+RgNfqWU8jEa/Eop5WPcssd/JnV1deTl5VFdXW13Ka0uJCSEmJgY\nAgMD7S5FKeWFPCb48/LyCA0NJSEhAWOM3eW0GhGhpKSEvLw8EhMT7S5HKeWFPKbVU11dTXh4uFeH\nPoAxhvDwcJ/4y0YpZQ+PCX7A60P/a77yPpVS9vCo4FdKKa9UWwlpb8PGp9rk22nwt7Gnn36aPn36\nYIzh6NGjdpejlLJLTQXsegP+eSMs7A3//gF8/ndoqGv1b+0xJ3e9xbhx45g9ezaTJ0+2uxSlVFur\nLoM9qyDtHcj+GBpqoGMUjLgRkuZC/Fjw82/1MlwKfmPMTGAR4A8sEZEFZ9hmMvAUEAgcFZFJxphY\n4BUgChDgBRFZ1EK1t7lXXnmFJ554AmMMQ4cOxd/fn7CwMFJTUykoKGDhwoVcffXVrF+/ngcffJCI\niAh2797NyJEjWb58OcYYRowYYffbUEq1pZPHIWslpL8D+9ZCQy2E9oDk+VbYx45uk7Bv7JzBb4zx\nB54BpgN5wDZjzAoRSW+0TWfgWWCmiBw0xkQ6X6oHfikiO4wxocB2Y8xHjT/3fDz0Xhrph8sv5Et8\nQ1KPMP54xaBvfT0tLY1HH32UzZs3ExERQWlpKffccw9Hjhxh48aNZGZmMmfOHK6++moAdu7cSVpa\nGj169GDcuHFs2rSJ8ePHt2jNSik3VVUKmf+B9HchZz046iAsBkb90Ar7mFHgZ1+n3ZUj/hQgW0Ry\nAIwxrwNzgcbhfT3wlogcBBCRIuf/HgGOOD+uMMZkAD2bfK5HWLt2Ld/73veIiIgAoGvXrgBceeWV\n+Pn5kZSURGFh4antU1JSiImJAWD48OHs379fg18pb1ZZApnvWWGf+yk46qFzHIy5A5KuhJ4jwU0m\n9lwJ/p7AoUaP84DRTbbpBwQaY9YDocAiEXml8QbGmARgBLD1TN/EGHM7cDtAXFzcWQs625F5WwsO\nDj71sYic8Xl/f3/q6+vbtC6lVBs4UWyFfdo7sH8jSAN0SYCL77aO7HuMcJuwb6ylTu4GACOBaUA7\nYIsx5jMR2QNgjOkIvAn8XETO2KMRkReAFwCSk5PlTNvYaerUqXznO9/hnnvuITw8nNLSUrtLUkrZ\noaIAMpxH9gc2gTiga28Y/3Mr7KOHumXYN+ZK8OcDsY0exzifaywPKBGRSqDSGPMpMAzYY4wJxAr9\n10TkrRao2RaDBg3igQceYNKkSfj7+5/3SdrFixezcOFCCgoKGDp0KLNmzWLJkiUtXK1SqkWVH4b0\nFVbYH9wCCET0gwn3wqArITLJ7cO+MdO4PXHGDYwJAPZgHc3nA9uA60UkrdE2A4GngUuBIOBzYB6Q\nBrwMlIrIz10tKjk5WZreiCUjI4OBAwe6+iU8nq+9X6XcTlmeFfTp78IhZ4c6Msk6qk+aC5Hu9ftp\njNkuIsmubHvOI34RqTfG3A2swhrnXCYiacaYO5yvPyciGcaYD4GvAAfWyOduY8x44CZglzHmC+eX\n/K2IfHAe70sppVrXsQOnwz7fefAZNQSm/M4K+2797K2vhbjU43cG9QdNnnuuyePHgcebPLcR8Jy/\nf5RSvqc053TYH95pPdd9GEz7gzWNE97b3vpagV65q5TyPUezrQuq0t+Fgq+s53pcBJc8ZB3Zd/Xu\nJdE1+JVSvqE46/SRfeFu67mYUTDjURg4B7rE21tfG9LgV0p5JxEoyjgd9sUZ1vOxY+DSP0HSHOgU\nY2+NNtHgV0p5DxEoTDvdxjm6BzDW4meXLYSBV0BYD7urtJ0Gfxu74YYbSE1NJTAwkJSUFJ5//nm9\nt65SF0LE6tOnOcO+dB8YP4gfB6N/BAOugNAou6t0Kxr8beyGG25g+fLlAFx//fUsWbKEO++80+aq\nlPIwInB4x+k2zrH9YPwhcQKM/QkMmA0du9ldpdvS4G+GlliWedasWae+XkpKCnl5eTa+I6U8iAjk\npTrbOCug7CD4BUDiJBh/jxX2HcLtrtIjeGbwr/w1FOxq2a8ZPQQu+8ZtBk5p6WWZ6+rqePXVV1m0\nyGNvT6BU63M4IO9z55H9CijPA79A6D0FJv8a+l8G7bvaXaXH8czgt0FLL8t81113MXHiRCZMmNCG\n70IpD+BogIOfWWGfsQIqjoB/EPSeBtN+D/1mQrvOdlfp0Twz+M9yZN7WzmdZ5oceeoji4mKef/75\ntilSKXfnaLBWukx/11r58kQh+AdD3+nW1bP9LoWQMLur9BqeGfw2aKllmZcsWcKqVatYs2YNfjbe\ngUcp2zXUw/4NVthnvg+VxRDQzgr7QVdC3xkQHGp3lV5Jg99FLbUs8x133EF8fDwXX3wxAFdddRV/\n+MMfWrJUpdxXQx3kfuI8sn8fTpZCYAfriD5prhX6QR3srtLrnXNZZjvossy+936VF6uvte47+/WR\nffVxCOponZhNmmv17oPa212lx2vRZZmVUqrZak5YYZ/5PmR+ADVlEBwG/Wc5w34qBIbYXaXP0uBX\nSrWMkn2wdzXsWWWdqG2ohZBOMHC2Ffa9JkNA8Lm+imoDHhX8IoLxoNubnS93bL8p9Q31tXBwM+xZ\nDXtXQUm29XxEP0i53erbx10M/rokibvxmOAPCQmhpKSE8PBwrw5/EaGkpISQEP0zWLmhikLI/sg6\nqt+3DmorrBn7hPFW2Ped4fVr2XsDjwn+mJgY8vLyKC4utruUVhcSEnLq4i+lbOVwwJGdp4/qv75D\nVWh3GHyVdVSfOAmCO9pbp2oWjwn+wMBAEhP1SEKpVlddDjnrrKP6vR9BZRFgrJuWTP0d9L3UWuLE\ni//y9nYeE/xKqVYiYvXn96yyjuoPbAFHnXVitvc0a4mEPpfoAmheRINfKV9UXwP7N56ewjmWaz3f\nbSBcfJd1VB87Gvw1IryR/ldVyleUH7GCfu9q68RsXSUEhEDiRLj4x9aJWR+676wv0+BXyls5GiB/\nh9W+2bPKuksVQFgMDLvWauEkTNCrZn2QBr9S3uTkcdi3xprCyf4Iqkqs2xDGjoZpf7SmcCKT9MSs\nj9PgV8qTiUBxlvOofjUc3ALSAO26QJ/pVtD3nqo3K1H/RYNfKU9TV20tZ/z1FM7xg9bzUYNh3M+s\nFk5MMvj521unclsa/Ep5grI8Z9CvhpxPoP4kBLZ33m/2F9aJ2U560Z9yjQa/Uu7I0QB5206HfeFu\n6/nO8XDRTda4ZcJ4XeFSnRcNfqXcRVUpZK+x2jfZH8PJY2D8rYXOpj9shX23/npiVl0wDX6l7CIC\nRenWUf2eVZD3OYgD2kdYffq+M6wTs3pjcdXCNPiVaku1VZD76ekpnPI86/nuw2DCvdYUTo+LQO/H\nrFqRBr9Sre3YgdNLI+zfAPXV1n1me0+BSfdZR/Zh3e2uUvkQl4LfGDMTWAT4A0tEZMEZtpkMPAUE\nAkdFZJKrn6uUV2moh0NbT18xW5xpPd+1F4ycD/1mQPw4vRuVss05g98Y4w88A0wH8oBtxpgVIpLe\naJvOwLPATBE5aIyJdPVzlfIKlSWNblCyBqrLwC/ACviLbrZOzEb0sbtKpQDXjvhTgGwRyQEwxrwO\nzAUah/f1wFsichBARIqa8blKeR4RKNh1+iKqvFRAoEMkDLjCOqrvNQVCwuyuVKlvcCX4ewKHGj3O\nA0Y32aYfEGiMWQ+EAotE5BUXP1cpz1BbCTnrT9+gpOKw9XyPi2Dyr61efffhemJWub2WOrkbAIwE\npgHtgC3GmM+a8wWMMbcDtwPExcW1UFlKtYDSHPh8CexcDjVlEBQKfaZa7Zu+06FjpN0VKtUsrgR/\nPhDb6HGM87nG8oASEakEKo0xnwLDnM+f63MBEJEXgBcAkpOTxaXqlWotItbR/dbnYc+H1ro3SVda\nV83GjYWAILsrVOq8uRL824C+xphErNCeh9XTb+xd4GljTAAQhNXO+SuQ6cLnKuU+aivhy9etwD+a\nZV1MNfFXkHyLjlwqr3HO4BeRemPM3cAqrJHMZSKSZoy5w/n6cyKSYYz5EPgKcGCNbe4GONPnttJ7\nUer8lebCtiWw41WrndN9OFz5HAy+SsculdcxIu7XVUlOTpbU1FS7y1DeTgRyP7GO7rNWOts5cyHl\nRxCbomviKI9ijNkuIsmubKtX7irfU1sJX/0Ttr4AxRnOds69znZOD7urU6rVafAr33HsAGz7O+x4\nxbrAKnooXPm/MOgqXd5Y2UpESDtcTu7RSq4Y1voHHxr8yruJWOvjbH0esj4ADCTNgdF3WPeh1XaO\nskldg4PPc0v5KL2Qj9ILyT9+ktCQAC4bHE2Af+teC6LBr7xTbRXs+pcV+EXp0K6rdaeq5FuhU0+7\nq1M+6kRNPZ/uKWZ1WgFrM4sor64nOMCPCX278bNL+jJtQGSrhz5o8Ctvc/ygNZ2z/WWoPg7RQ2Du\nMzD4uxDYzu7qlA8qqqhmTUYRq9MK2LSvhNp6B13aBzI9KZoZg6KY0DeC9kFtG8Ua/MrzicCBTbD1\nOcj8D2Bg4BUw+kfW3au0naPa2L7iE3yUXsjqtAJ2HjqOCMR2bcdNY+KZnhRFcnyXNjmy/zYa/Mpz\n1Z2EXf+22jmFu612zrifw6hb9cbjqk05HMIXecdZnVbIR+kF7CuuBGBwzzB+cUk/pidFMSA6FOMm\nByEa/MrzHD8EqUth+0vWfWmjBsOcp2HI1drOUW2mpr6BzftKWJ1WyMcZhRRX1BDgZxjdqys3X5zA\nJUlR9Ozsnj+PGvzKM4jAgc3w+fOQ8T4gMGC2NZ0TP1bbOapNlFXVsS6riI/SC1mfVURlbQMdgvyZ\n3D+S6UlRTOkfSaf2gXaXeU4a/Mq91Z2EXW842zm7oF0XGPsTq53TWVdxVa3v8PGTVr8+vYCtOaXU\nO4SIjsHMGd6TGYOiuLhXOCGB/naX2Swa/Mo9leXBtq/bOaUQOQiuWAxDvgdB7e2uTnkxESGzoOJU\n2O/OLwegd7cO3DahFzMGRTE8pjN+fp77V6YGv3IfInDwM2s6J+M9QKD/LKudkzBe2zmq1dQ3OEg9\ncMw6OZtRwKHSkxgDI2I7c//MAUxPiqJPZEe7y2wxGvzKfnXVsPtNK/ALvoKQTnDxj2HUbdAl3u7q\nlJc6WdvAp3uLWZ1WyNrMQo5V1RHk78e4PuHcNbkP0wZGEhnqnUt5aPAr+5QfdrZzXoSqEug2EGY/\nBUOvgaAOdlenvFDJiRrWZBaxOq2QDXuLqal3EBYSwNQBkcwYFM3Eft3oGOz9sej971C5FxE4tNU6\nWZv+LojD2c75ESRO1HaOanH7j1aeWg8n9UApDoEenUK4LiWO6UlRpCR2JdDGi6nsoMGv2kZdNaS9\nZbVzjnwJwZ1gzJ2Q8kPokmB3dcqLiAi78stYnWadnN1TeAKAAdGh3D21LzOSohjUI8xtLqaygwa/\nal3lR6yLrVJfhKqj0G0AzP4rDL1W2zmqxdTWO/gsp+TUkX1BeTV+BlISu/L72UnMSIoitqtOg31N\ng1+1PBHI22Yd3ae/C44G6H+Zs50zSds5qkVUVNexPquY1emFrM8soqKmnpBAPyb168a9Sf2ZOiCS\nrh2C7C7TLWnwq5ZTXwNpb1uBf3in1c4ZfYc1ndM10e7qlBcoLK92ztcXsmXfUeoahPAOQVw2JJoZ\nSdGM7xvhcRdT2UGDX124igJIXWb9qyyGiH5w+V9g6DwI9p7ZZ9X2RITsohOsdob9l4eOAxAf3p4f\njE1gxqBoLorrgr8HX0xlBw1+df7yUq2j+7S3rXZOv0utdk6vKdrOUeetwSHsPHiM1c5+fe5Ra6XL\nYTGduHdGP2YMiqZvZEefPjl7oTT4VfPU10L6O1bg52+H4DBIud1q54T3trs65aGq6xrYuPcoH6Vb\nK12WVNYS6G8Y0yucW8YnMn1gFNGdvPNiKjto8CvXVBRaF1qlLoMThRDeF2Y9AcPmQXCo3dUpD3Ss\nspa1mdZKl5/sKeZkXQMdgwOY3L8bMwZFM7l/N8JC3H+lS0+kwa/OLn+7dbHV7rfAUQd9ZzjbOVPB\nz7cuelEX7lBp1anFz7btP0aDQ4gKC+a7I3syPSmaMb26EhygJ2dbmwa/+qb6WshYYbVz8rZBUKi1\nDHLK7drOUc0iIqQdLj81iZNxxFrpsm9kR+6Y1IsZSdEM6dnJo1e69EQa/Oq0E0XWhVapy+BEAXTt\nDZcthGHXQUiY3dUpD1JQVs0rW/bz7heHyT9urXSZHN+F384awPSkaBIj9OI9O2nwK8jfYbVz0t6C\nhlrocwmMfhp6T9N2jmqW3fllLN2Yy3tfHsYhwqR+3fjptD5MGxhFRMdgu8tTThr8vsrhsNo5W56B\nvM8hqCOMnG+tnRPR1+7qlAdxOIQ1mUUs2ZDD1txSOgT5c9PF8cwfm0hcuC6T4I40+H2NCOxZBWsf\ntW5l2LUXzPwzDL9e2zmqWapq63lzex7LNu0n92glPTqF8NtZA7h2VByd2uk0jjvT4PcluZ/Cmkes\nI/wuiXDV32Hwd8FPpyiU6wrLq3l5835e23qQspN1DIvpxN+uG8HMwdE+t7yxp9Lg9wV522Htw5Cz\nHkJ7WDc7GXEj+OtRmXLd7vwylm3M5b2vDlPvEC5Niua2CYmMjO+iV9F6GA1+b1awG9b9D2R9AO0j\n4NI/QfItEKhXQCrXOBzC2swilmzM4bOcUtoH+XPD6Hjmj0sgPlwnczyVBr83KtkH6x6z7mMbHAZT\nfwej79QF05TLqmrreXNHPi9uzCXnaCXdO4Xwm8sGMC9F+/fewKXgN8bMBBYB/sASEVnQ5PXJwLtA\nrvOpt0TkYedrvwBuAwTYBcwXkeoWqV79t7I8+OTPsPM1CAiG8b+AsT+B9l3trkx5iMJya/7+ta0H\nOV5Vx9CYTiy+bgSXaf/eq5wz+I0x/sAzwHQgD9hmjFkhIulNNt0gIrObfG5P4KdAkoicNMb8C5gH\nvNQSxSunE0Ww4UnrTldgjWSOvwdCo+ytS3mMtMOn5+/rHcKMpChum9CLZO3feyVXjvhTgGwRyQEw\nxrwOzAWaBv/Zvkc7Y0wd0B44fD6FqjM4eQw2LbaWVqivsUYyJ90PnWPtrkx5AIdDWJdVxJINuWzJ\nKdH+vQ9xJfh7AocaPc4DRp9hu7HGmK+AfOBeEUkTkXxjzBPAQeAksFpEVl9o0T6v5gRs/V/Y9Deo\nKbNGMif/FiL62F2Z8gAnaxt4c0cey7R/77Na6uTuDiBORE4YY2YB7wB9jTFdsP46SASOA/82xtwo\nIsubfgFjzO3A7QBxcXEtVJaXqau21tHZ8BfrxuX9LoOpD0D0ELsrUx7gTP37RfOGM2tId+3f+xhX\ngj8faNw7iHE+d4qIlDf6+ANjzLPGmAhgCpArIsUAxpi3gLHAN4JfRF4AXgBITk6WZr4P79ZQB1+8\nBp8shPJ8SJwIU/8AsaPsrkx5gDP1728d34tRCdq/91WuBP82rKP3RKzAnwdc33gDY0w0UCgiYoxJ\nAfyAEqwWzxhjTHusVs80ILUF6/duDoc1krn+MSjNgZhRcOX/Qq9Jdlem3JzDIazfY/XvN+873b//\nwdgEEnRlTJ93zuAXkXpjzN3AKqxxzmUikmaMucP5+nPA1cCdxph6rICfJyICbDXGvIHVCqoHduI8\nqldnIWJddLX2UShKh6jBcN0/rXva6hGaOouTtQ28tTOPpRtzySmuJDoshF9fNoDrRsXRqb3275XF\nWPnsXpKTkyU11Qf/MBCBnHVW4Odvt9bDn/oAJH1Hl0dWZ1VUXs0rWw7w2tYDHKuqY0jPTtw2IVH7\n9z7EGLNdRJJd2Vav3HUXB7fC2kdg/wYIi4E5T1s3QPHX/0Tq26UfLmfpxlxWfJlPvUOYPtCav9f+\nvTobTRW7HfnKOsLfuwo6RFp3vBr5A+vKW6XOwOEQPtlTzJKNOWzKLqFdoD/Xp8Qxf1yi9u+VSzT4\n7XJ0r7WAWtrbENIZpv3Ruol5kP7iqjOrrmvgrR35LN2Ywz5n//7+mQO4PkX796p5NPjb2rED1no6\nX/4fBLSDib+Ci++Gdp3trky5qaKKal7dcoDln1n9+8E9w3T+Xl0QDf62UlEAnz4B218C42etljn+\nF9Cxm92VKTeVccTZv//iMHUOB5cMjOK28YmkJHbV/r26IBr8ra2qFDY9BVtfAEeddQOUifdBp552\nV6bckMMhfLK3mKUbctmYfZR2gf7MS4ll/rhEErV/r1qIBn9rqamALc/Clqetj4deA5N/bd3jVqkm\nmvbvo8KCuX/mAK5LiaVz+yC7y1NeRoO/pdWdhG1LYONfoaoEBsyGKQ9AVJLdlSk3VFRRzfItB1i+\n9SCllbUM7hnGU9da/fugAO3fq9ahwd9S6mth56vw6eNQcQR6T7XufNVzpN2VKTeUWVDO0g25vOvs\n308bEMVtExIZrf171QY0+C+UowG++hes/xMcPwCxY+C7SyBhvN2VKTfzdf9+2cZcNuy1+vfXjopl\n/rgEenXT22KqtqPBf75EIGOFdW/b4kyIHgo3vAF9LtH1dNR/qa5r4O2d+SzdmEt20QmiwoK5b2Z/\nrk+J0/69soUGf3OJQPYaa3mFI19ARD/43sswcI6up6P+S3FFDa9+Zs3fl1bWMqhHGH+9dhiXD+mh\n/XtlKw3+5jiwGdY8Agc3Q+c4a4nkodeCn7/dlSk3klVQwdKNObyz83T//tbxiYzppf175R40+F1x\neKcV+PvWQMdomPUEXPR9CNA/05VFxFo/Z6mzfx8S6Kf9e+W2NPjPpigT1j0KGe9Buy4w/WEY9UMI\nam93ZcpNVNc18I6zf7+36ASRocH86lKrf9+lgx4YKPekwX8mpbmwfgF89U8I6giTfwNj7oKQMLsr\nU26iuKKG5c7+fUllLUndw3jymmHMHqr9e+X+NPgbKz9szeHveAX8AmDsT6z1dNp3tbsy5SZO9e+/\nOExtvYNLBkZy6/he2r9XHkWDH6CyBDY+aV1x62iw1sOfcC+Edbe7MuUmsgoq+POHmazNLCIk0I9r\nkmOYPy6R3tq/Vx7It4O/ugw2Pw2fPQt1VTB0Hky+H7ok2F2ZchMFZdU8+VEWb2zPo0NwAL+c3o8b\nx8Rr/155NN8M/toq+Px52PgUVB+HpLnWejrd+ttdmXIT5dV1PP/JPpZuzMXhgFvGJfLjKX008JVX\n8K3gr6+B7S/DhifgRCH0nWEFfo/hdlem3ERtvYN/bD3A4rXZlFbWcuXwHvxyRn9iu+okl/IevhH8\nDfXw1euw/s9QdhDix8E1r0DcGLsrU25CRPhgVwELV2VyoKSKsb3D+c1lAxkS08nu0pRqcd4d/A4H\npL9jradTshd6jIArnrJWztQJDOW0NaeEx1Zm8uWh4wyIDuWl+aOY1K+bTukor+WdwS8Ce1dbV9sW\n7oJuA+Ha5dba+PrLrJyyiypYsDKLjzMKiQ4L4fGrh3LVRTH4++nPiPJu3hf8uRtgzcOQ97k1nXPV\n32Hwd3U9HXVKUXk1f/14L//cdpAOQQHcN7M/88cm0i5If0aUb/Ce4K8ug3/dDDnrIbQHzH7Kur+t\nf6DdlSk3caKmnhc+zeHvn+ZQ73Dw/bEJ/GRqX7rqpI7yMd4T/MFhENgeLn0Mkm+FwBC7K1Juoq7B\nwevbDrHo4z0cPVHL5UO7c9+l/YkP15uXK9/kPcFvDFz3f3ZXodyIiLAqrZCFH2aSc7SSlMSuLPn+\nQIbHdra7NKVs5T3Br1Qj2w+U8tgHmWw/cIw+kR1Z+v1kpg6I1EkdpdDgV14mp/gECz/M4sO0AiJD\ng1lw1RCuHhlDgL+umKnU1zT4lVcorqhh8Zq9/OPzg4QE+PHL6f24dUIi7YP0R1yppvS3Qnm0qtp6\nlmzI5flP9lFT7+CG0XH8dFpfIjoG212aUm7LpeA3xswEFgH+wBIRWdDk9cnAu0Cu86m3RORh52ud\ngSXAYECAW0RkS4tUr3xWfYODf2/P48mP9lBcUcNlg6P51aX99TaHSrngnMFvjPEHngGmA3nANmPM\nChFJb7LpBhGZfYYvsQj4UESuNsYEAbralTpvIsKajCIWfJhJdtEJRsZ34bkbL2JkvN4sRylXuXLE\nnwJki0gOgDHmdWAu0DT4v8EY0wmYCPwAQERqgdrzLVb5tp0Hj/GnlZl8nltKr4gOPH/TSGYkRemk\njlLN5Erw9wQONXqcB4w+w3ZjjTFfAfnAvSKSBiQCxcCLxphhwHbgZyJSeWFlK1+y/2glj6/K4j+7\njhDRMYhHrxzMtaNiCdRJHaXOS0ud3N0BxInICWPMLOAdoK/z618E/EREthpjFgG/Bn7f9AsYY24H\nbgeIi4trobKUJys5UcPf1mbz2tYDBPj58bNpffnhxF50DNaZBKUuhCu/QflAbKPHMc7nThGR8kYf\nf2CMedYYE4H110GeiGx1vvwGVvB/g4i8ALwAkJycLC6/A+V1TtY2sGxTLs+t30dVXQPXjorl59P6\nEhmmy3Ao1RJcCf5tQF9jTCJW4M8Drm+8gTEmGigUETHGpAB+QInz8SFjTH8RyQKm4cK5AeWbGhzC\nmzvyeHL1HgrKq5meFMX9M/vTJzLU7tKU8irnDH4RqTfG3A2swhrnXCYiacaYO5yvPwdcDdxpjKkH\nTgLzROTro/afAK85J3pygPmt8D6UBxMR1u8pZsEHmWQVVjA8tjOLrxtBSqJO6ijVGszpfHYfycnJ\nkpqaancBifwQAAANDElEQVQZqg3syivjTysz2LyvhPjw9tw/cwCXDY7WSR2lmskYs11Ekl3ZVs+S\nKVscKq3iidVZvPvFYbp2COKhOYO4LiWOoACd1FGqtWnwqzZ1rLKWp9dl8+qWA/j5wd1T+vCjSb0I\nDdEb5ijVVjT4VZuormvgpc37eWZdNpU19XxvZCy/mN6P6E46qaNUW9PgV63K4RDe3pnPX1Zncbis\nmqkDIrl/5gD6R+ukjlJ20eBXrWbD3mIe+yCTjCPlDOnZiSeuGcbY3hF2l6WUz9PgVy0u7XAZC1Zm\nsmHvUWK7tmPxdSOYPaQ7fn46qaOUO9DgVy0m//hJ/rI6i7d35tOpXSC/n53EjWPiCA7wt7s0pVQj\nGvzqgpWdrOPZ9dm8uGk/AD+a2Js7J/emUzud1FHKHWnwq/NWU9/Aq1sO8PS6bMpO1nHViBjumdGP\nnp3b2V2aUuosNPhVszkcwntfHebxVVnkHTvJxH7d+PXMAST1CLO7NKWUCzT4VbNszj7KYysz2J1f\nTlL3MF69dQgT+nazuyylVDNo8CuXZBaUs2BlJuuziunZuR1/vXYYc4f11EkdpTyQBr86qyNlJ3ly\n9R7e2JFHaHAAv501gJsvTiAkUCd1lPJUGvzqjMqr63j+k30s3ZiLwwG3jU/kx1P60Ll9kN2lKaUu\nkAa/+i+19Q7+sfUAi9dmU1pZy5XDe/DLGf2J7dre7tKUUi1Eg18B1s1Q/rPrCI+vyuJASRVje4fz\n21kDGdyzk92lKaVamAa/YmtOCY+tzOTLQ8cZEB3KS/NHMalfN70ZilJeSoPfh2UXVbBgZSYfZxQR\nHRbC41cP5aqLYvDXSR2lvJoGvw8qO1nHUx/v4ZUtB2gf6M99M/tzy7hEndRRykdo8PuQBofwr9RD\nPL4qi2NVtVyfEsc90/sR3jHY7tKUUm1Ig99HpO4v5Y8r0kg7XE5KQlf+OCeJQT30xK1SvkiD38sd\nKTvJgpWZvPvFYbp3CuFv141g9tDueuJWKR+mwe+lqusaWLoxl6fXZtMgwk+n9uGOyb1pH6T/yZXy\ndZoCXkZEWJ1eyP/8J4ODpVXMHBTNA5cP1AuwlFKnaPB7keyiCh56L50Ne4/SL6ojr902mnF99B63\nSqn/psHvBcpO1rHo4728smU/7YP8efCKJG4cE0+Av5/dpSml3JAGvwdrcAj/do5nllbVcl1KHL/U\n8Uyl1Dlo8Huo1P2lPPheGrvzyxmV0IWXr0jRdXWUUi7R4PcwBWXVLFiZwTtfHCY6LIRF84YzZ1gP\nHc9USrlMg99DfD2e+cy6bOodwt1T+nDXFB3PVEo1n6aGmxMRPs4o4pH30zlYWsWlg6J4YFYSceE6\nnqmUOj8a/G6s8Xhm38iOLL91NOP76nimUurCaPC7ofJqazzz5c37aRfkzx9mJ3HTxfEE6nimUqoF\nuBT8xpiZwCLAH1giIguavD4ZeBfIdT71log83Oh1fyAVyBeR2S1Qt1dyOIR/bz/Ewg+t8cx5o+K4\nd4aOZyqlWtY5g98Z2s8A04E8YJsxZoWIpDfZdMNZQv1nQAYQdiHFerPtB0p5cEU6u/LLSI7vwstz\ndDxTKdU6XDniTwGyRSQHwBjzOjAXaBr8Z2SMiQEuB/4HuOc86/RaheXVLFiZyds783U8UynVJlwJ\n/p7AoUaP84DRZ9hurDHmKyAfuFdE0pzPPwXcB4ReSKHepqb+9OqZ9Q3Cj6f05q7JfegQrKddlFKt\nq6VSZgcQJyInjDGzgHeAvsaY2UCRiGx3ngf4VsaY24HbAeLi4lqoLPcjIqzJKOKR/6RzoKSK6UlR\n/O7ygcSHd7C7NKWUj3Al+POB2EaPY5zPnSIi5Y0+/sAY86wxJgIYB8xx/p9BCBBmjFkuIjc2/SYi\n8gLwAkBycrI0+514gOyiEzz8fjqf7immT2RHXrklhYn9utldllLKx7gS/Nuwjt4TsQJ/HnB94w2M\nMdFAoYiIMSYF8ANKROQ3wG+c20zGagF9I/S9XXl1HYs/3stLzvHM389O4mYdz1RK2eScwS8i9caY\nu4FVWOOcy0QkzRhzh/P154CrgTuNMfXASWCeiHjlUXtzOBzCG9vzWLgqk5LKWq5NjuXeS/sToeOZ\nSikbGXfM5+TkZElNTbW7jAuy4+AxHlqRxpd5ZYyM78KDVwxiSIyOZyqlWocxZruIJLuyrY6QtLCi\n8moWfJjJWzvyiQoL5qlrhzN3uI5nKqXchwZ/C6mpb2DZxv08vXYvdQ3CXZN78+MpOp6plHI/mkoX\nSERYm2mtnrm/pIpLBkbx+9k6nqmUcl8a/BdgX/EJHnk/nfVZxfTu1oGXb0lhko5nKqXcnAb/eaio\nrmPxmr28uGk/7QL9+d3lA/n+2AQdz1RKeQQN/mZwOIQ3duSx8MMsSipruGZkLL+aqeOZSinPosHv\nop0Hj/Hge+l8eeg4F8V1ZtkPkhka09nuspRSqtk0+M+hqLyaP3+YxZs78ogMDeav1w7jyuE9dTxT\nKeWxNPi/RW29gxc35bJ4jTWeeadzPLOjjmcqpTycptgZrMss4uH308k9WsklAyP53eVJJEToeKZS\nyjto8DeS4xzPXJdVTK9uHXhp/igm94+0uyyllGpRGvxY45lPr81m2aZcQgKs8cybL04gKEDHM5VS\n3seng9/hEN7amc+ClZkcPVHDNckx/OrSAXQL1fFMpZT38tng/+LQcf64Io0vDx1nRFxnln4/mWGx\nOp6plPJ+Phf8RRXVLPwwize259EtNJgnr7HGM/38dDxTKeUbfCb4a+sdvLQ5l8Vrsqmpb+COSb25\ne6qOZyqlfI9PpN66rCIeeS+dnKOVTBsQye9mJ5Go45lKKR/l1cGfe7SSR95PZ21mEb0iOvDi/FFM\n0fFMpZSP88rgP1FTz9/W7mXZxlyCA/x5YJa1eqaOZyqllJcFv8MhvL0znwUfZlJcUcP3Rsbwq5n9\niQwNsbs0pZRyG14T/GVVdXz/xc/54tBxhsd25u83JzNcxzOVUuobvCb4w9oFEB/enpvGxPOdETqe\nqZRS38Zrgt8Yw6J5I+wuQyml3J6e7VRKKR+jwa+UUj5Gg18ppXyMBr9SSvkYDX6llPIxGvxKKeVj\nNPiVUsrHaPArpZSPMSJidw3fYIwpBg6c56dHAEdbsJyWonU1j9bVPFpX83hjXfEi0s2VDd0y+C+E\nMSZVRJLtrqMprat5tK7m0bqax9fr0laPUkr5GA1+pZTyMd4Y/C/YXcC30LqaR+tqHq2reXy6Lq/r\n8SullDo7bzziV0opdRYeGfzGmJnGmCxjTLYx5tdneN0YYxY7X//KGHORm9Q12RhTZoz5wvnvD21U\n1zJjTJExZve3vG7X/jpXXXbtr1hjzDpjTLoxJs0Y87MzbNPm+8zFutp8nxljQowxnxtjvnTW9dAZ\ntrFjf7lSly0/Y87v7W+M2WmMef8Mr7Xu/hIRj/oH+AP7gF5AEPAlkNRkm1nASsAAY4CtblLXZOB9\nG/bZROAiYPe3vN7m+8vFuuzaX92Bi5wfhwJ73ORnzJW62nyfOfdBR+fHgcBWYIwb7C9X6rLlZ8z5\nve8B/nGm79/a+8sTj/hTgGwRyRGRWuB1YG6TbeYCr4jlM6CzMaa7G9RlCxH5FCg9yyZ27C9X6rKF\niBwRkR3OjyuADKBnk83afJ+5WFebc+6DE86Hgc5/TU8e2rG/XKnLFsaYGOByYMm3bNKq+8sTg78n\ncKjR4zy++cPvyjZ21AUw1vmn20pjzKBWrslVduwvV9m6v4wxCcAIrKPFxmzdZ2epC2zYZ862xRdA\nEfCRiLjF/nKhLrDnZ+wp4D7A8S2vt+r+8sTg92Q7gDgRGQr8DXjH5nrcna37yxjTEXgT+LmIlLfl\n9z6bc9Rlyz4TkQYRGQ7EACnGmMFt8X3PxYW62nx/GWNmA0Uisr21v9e38cTgzwdiGz2OcT7X3G3a\nvC4RKf/6T08R+QAINMZEtHJdrrBjf52TnfvLGBOIFa6vichbZ9jEln12rrrs/hkTkePAOmBmk5ds\n/Rn7trps2l/jgDnGmP1YLeGpxpjlTbZp1f3licG/DehrjEk0xgQB84AVTbZZAdzsPDM+BigTkSN2\n12WMiTbGGOfHKVj7v6SV63KFHfvrnOzaX87vuRTIEJEnv2WzNt9nrtRlxz4zxnQzxnR2ftwOmA5k\nNtnMjv11zrrs2F8i8hsRiRGRBKycWCsiNzbZrFX3V0BLfaG2IiL1xpi7gVVYkzTLRCTNGHOH8/Xn\ngA+wzopnA1XAfDep62rgTmNMPXASmCfOU/ityRjzf1jTCxHGmDzgj1gnumzbXy7WZcv+wjoiuwnY\n5ewPA/wWiGtUmx37zJW67Nhn3YGXjTH+WMH5LxF53+7fSRfrsutn7Bvacn/plbtKKeVjPLHVo5RS\n6gJo8CullI/R4FdKKR+jwa+UUj5Gg18ppXyMBr9SSvkYDX6llPIxGvxKKeVj/h9Ctr6K7wmdTwAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fee7f86d0b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cnn1_2k, label=\"cnn1\")\n",
    "plt.plot(cnn2_2k, label=\"cnn2\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have tried to train the first two models for 20k steps more (here plotted at inverval of 2k), but they still don't reach the performance of cnn3. \n",
    "However they could simply need more training time to reach convergence"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}