tianqig · July 19, 2017 12:00
diff --git a/Neural+Network(1).ipynb b/Neural+Network(1).ipynb
 {
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Deep Learning in Python\n",
    "Williams Wei | March 2017\n",
    "\n",
    "<b>Deep learning</b> refers to neural networks with multiple hidden layers that can learn increasingly abstract representations of the input data. A good example of Deep Learning is perception, recognizing what's in an image, what people are saying when they are talking, helping robots explore the world and interact with it.\n",
    "\n",
    "\n",
    "This notebook will guide you through the applications of Neural Networks, the core of deep learning. Details like the training process of Neural Networks will not be discussed in order to let beginners to get started on deep learning without frustration.In this step-by-step tutorial, you’ll learn how to build a convolutional neural network in Python.\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "* [what is a Neural Network?](#first-bullet)\n",
    "* [What is a Convolutional Neural Network?](#second-bullet)\n",
    "* [Handwritten Digit Recognition](#third-bullet)\n",
    "  * [Loading the MNIST dataset in Keras](#fourth-bullet)\n",
    "  * [A baseline Neural Network Model](#fifth-bullet)\n",
    "  * [A simple Convolutional Neural Network for MNIST](#sixth-bullet)\n",
    "* [Summary](#seventh-bullet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a Neural Network?<a class=\"anchor\" id=\"first-bullet\"></a>\n",
    "\n",
    "Neural Networks (NN), also called as Artificial Neural Network is named after its artificial representation of working of a human being’s nervous system. Each neuron takes input from numerous other neurons through the dendrites. It then performs the required processing on the input and sends another electrical pulse through the axon into the terminal nodes from where it is transmitted to numerous other neurons.\n",
    "\n",
    "A NN works in a very similar fashion. The general structure of a neural network looks like:\n",
    "\n",
    "<img src=\"https://www.analyticsvidhya.com/wp-content/uploads/2016/03/2.-ann-structure.jpg\" width=\"800\" height=\"500\" align=\"left\"/>\n",
    "\n",
    "\n",
    "Neural Network is divided into layer of 3 types:\n",
    "\n",
    "* <b>Input Layer</b>: The training observations are fed through these neurons \n",
    "\n",
    "* <b>Hidden Layers</b>: These are the intermediate layers between input and output which help the Neural Network learn the complicated relationships involved in data.\n",
    "\n",
    "* <b>Output Layer</b>: The final output is extracted from previous two layers. For Example: In case of a classification problem with 5 classes, the output later will have 5 neurons.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a Convolutional Neural Network?<a class=\"anchor\" id=\"second-bullet\"></a>\n",
    "\n",
    "Convolutional Neural Networks (CNN's) are multi-layer neural networks that assume the input data to be images.By making this assumption, CNN's can drastically reduce the number of parameters that need to be tuned. The **Convolutional layer** is the building block of a Convolutional Network.Basically, it is a 3D volume which is composed of neurons.\n",
    "\n",
    "\n",
    "<img src=\"https://irenelizihui.files.wordpress.com/2016/02/cnn2.png\" width=\"800\" height=\"500\" align=\"left\"/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keras\n",
    "\n",
    "Keras is a high-level neural network library that, among many other things, wraps an API similar to scikit-learn's. It allows you to define and train your neural network models in a few short lines of code.Its minimalistic and modular approach makes it very convenient to get deep neural networks up and running. More detailed documents can be found [here](https://keras.io/layers/convolutional/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handwritten Digit Recognition<a class=\"anchor\" id=\"third-bullet\"></a>\n",
    "A popular showcase of the ability of deep learning is object recognition in images. The \"hello world\" of object recognition for deep learning is the MNIST dataset for handwritten digit recognition. In this tutorial, we will discover how to develop a CNN to achieve state of art performance on MNIST handwritten digit recognition task using the Keras library. Images in the MNIST data set were taken from a variety of scanned documents, normalized in size and centered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the MNIST dataset in Keras<a class=\"anchor\" id=\"fourth-bullet\"></a>\n",
    "The Keras library provides a convenient method for loading the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATsAAAD+CAYAAABFjqJ0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvVmIrdl53/1fe57neVfVrjO1uq223QRiCDLEIcYRIdDG\nF0I4BHlA+MKKDRFEsm6aJL6wctGQz+ALK4pomRgnESjdDsRuCSGCAk7kWK2W3a0+R+ecGvc8z/P6\nLmo/66z3PVWnqvY8rB+8VJ2q2u9ZVfXU/33Ws56Bcc6hUCgU245h1QtQKBSKZaDETqFQ7ARK7BQK\nxU6gxE6hUOwESuwUCsVOoMROoVDsBDOJHWPsk4yxHzPGHjLGvjCvRSkUq0bZ9vbBps2zY4wZADwE\n8I8BpAF8H8CnOec/nt/yFIrlo2x7O5nFs/s5AI8458ec8wGAPwPw+nyWpVCsFGXbW4hphtcmAZxK\n/z7DhZFoYIypEo01hHPOVr2GNUbZ9gZzlW2rAwqFQrETzCJ25wAOpH/vTT6mUGw6yra3kFnE7vsA\n7jPGUowxC4BPA3hnPstSKFaKsu0tZOqYHed8xBj7HIB3cSGaX+Wcfzi3lSkUK0LZ9nYyderJjf8D\nFcRdS9QBxewo215P1AGFQqHYaZTYKRSKnUCJnUKh2AmU2CkUip1AiZ1CodgJlNgpFIqdYJbaWIVC\nsWUwxmA0GsVlsVg0l9lshtlshsViwWg0wmAwQL/fF2/pGgwGGI/HGI1GGI1Gq/62ACixUygUEkaj\nEVarFRaLBVarFR6PB16vFx6PB263G263Gy6XCy6XC/1+H41GA81mE81mE7VaDfV6HfV6Ha1WC/1+\nH71eD+PxGOswslWJnUKhEJA353Q64XQ6EYlExBUOhxEKhRAMBhEKhdBut1EsFlEqlVAsFpHP55HL\n5WAymcAYQ7vdxng8Rr/fX/W3BWBGsWOMHQGoARgDGHDOn2uDswswxsAYE+9f9nmCcy4u+bXX3UP/\nWv19FPNlF22bMQaTyQS73Q632w2fz4d4PI79/X3s7+8jmUwiHo+Lq9lsIp1OI5PJIJPJwO12w2w2\na+xyMBiAMbYWdjqrZzcG8Auc88o8FrNOGAyG50ToKux2u7gsFotGtOT4h8Fg0Lj9nHM4nU64XC44\nnU7Y7XZYrVaxjSAD4Zyj3++j3W6j0+mg3W6j1WqJt71eb+E/jx1ka21bhgSOrkAggEQigXg8jkQi\ngXA4jGg0inA4jHA4DJ/PB7vdDoPBALPZDJfLhWAwKO7jcDgQCASQyWRwenqK09NTNBoNjMfjVX+r\nM4sdw5ad6JJQGQwGzfWir/d4PPD7/fD7/XA4HOI1jDERzLVarTAYDMjlcshms8jlchiPx5ptgtfr\nFfERp9MpYh2cc7RaLZRKJXEVi0UUi0UMh0Mldoth62z7MshGrVYrbDYbwuEwUqkU7t+/j3v37omY\nndfrhdvthtPphM1mE69zu90wGAyw2WxwOp1CLEOhEBhjaDQaOD8/x3A4XPW3OrPYcQDfYoyNAPwx\n5/wrc1jTypA9Mr1X9qLXuN1uhMNhxONx+Hw+4cWREZDXZzKZ8PjxYwBAu93GaDRCNBrF4eEh7ty5\nI+Ii9AQlsRuPx6hWqzg7OxOX1WrFaDRCo9FYys9mB9kq274KeevqdDoRCoWQSqXwyiuv4NVXX4XN\nZhOXxWIRHqDs2dlsNng8HgQCAXS7XXS7Xfh8PtTrdZyfn7/QWVgms4rdJzjnGcZYGBeG8SHn/Hvz\nWNi0vCj+pffW9Jf8WvlEymS6+sfEGEMikRCXLHYU7LXb7bDZbDAajRgMBuh2u0Ls9vb2cHh4iHv3\n7iEWi2nEbjQaYTgcYjQaoVgsot/vo9VqoVqtwmazwWw2v1CIFTOxdrY9LxhjwuatViu8Xi8CgQD8\nfj8ODg7ElUqlxMOeDh2AZ7Fj8u5MJhNsNhscDodINRkOhwiHw/B6vbDZbBgOh5qH9yqYSew455nJ\n2wJj7Ju46NO/UoMwGAwwmUxCcIBnAijnC8liRk8s2SOTY2k2m+3K/48xhkAgIIzF5XJpjEnvHUYi\nEXQ6HWEwh4eH2N/fF14hbYOHw6E4uu/3+6jVaqjVaqhWq6hUKmg0Guh2u2uxPdhG1tG254XJZBJ/\nB263G8lkEslkEolEAoeHh9jb24PP5xMenOyZkVjpD8jk9+WHvNPphNfrBQCRizccDldyuDa12DHG\nHAAMnPMmY8wJ4JcA/Ju5rWy6NcFoNIrER6PRKITOYDCIH77T6YTD4RBi5nQ6YbFYNE8xn88Hv98P\nn88Hp9P5wv/TbrfD4XDA4XCIAwq6ZOMYDAYIh8MYj8fi65LJJPb29hCLxcTrjUajiMXRoUS9XhdC\nJ4vduiRsbhPraNvzRPbEAoEAkskk7t+/j/v37yOZTCISiQix0++UxuOxSBbWZwXIuySLxQKbzQaX\nywWv14vxeIx2uw3OucZmlyl4s3h2UQDfnDQwNAH4z5zzd+ezrOkhsbNarTCbzULo6CCBgq1erxc+\nnw8+nw9er1fE1OgKh8Oag4MXIXty+pPbwWAgvLNutwvOuXiiGgwGxGIxRKNRRKNRGAwGIY6y2FHC\npvLslsZa2va8ILFzuVzw+/1IJpN48OABfvqnfxrhcFg8uPXhG3po01aVhI/EjhwM+hu02+1C7AaD\nATjnYsdCW9mNEDvO+VMAr81xLVMhbxUtFgv8fj+CwSACgYA4Iqevoa0pZYFTRrjb7YbVatV4dn6/\nH4FAAD6fDy6X68br0f/yKGWExIrSTprNpvBEh8Mh2u02AGA4HGI4HGIwGKDVaokrn8/j9PQU2WwW\npVIJjUYDnU5HeXYLYF1se1bowSuHUoxGo4gLRyIRJJNJpFIpxGIxBAIBuN1uEdbRI4sVxZ47nQ66\n3S76/b6m2sJqtSIUCuHw8BDtdhv5fF5kElSrVXQ6HXEty4Y3voJCLm9xOp3Y29sTAVav16v5JdPx\nOp2QyidNZrP5OWG87Ol2W7rdLiqVCtLpNHK5nOaXzBhDtVoVW+nxeIzBYCAu+WsrlYpIWymXy2i3\n28qzU1yJvKOhnQ7FqGOxmPgb2d/fx97eHiKRCBwOh/DOLjtBlcWu0+mgVquhUqmgWq2i0WggkUhg\nPB7DZrPBarUiEolgMBjA4XCIxONMJoNsNivCMYPBQIndTSGPzuFwwOv1IplM4uWXX8Yrr7yCcDgs\ntqXksV31vv40lvLjphU78vC63S7K5TLOz89xdHSkKZQm158uMiT56vV66PV6aLVawkCq1SoGg4Hw\nAhUKPXKs2mw2izw4h8OBeDyOO3fu4KWXXkIqldLkiFoslitzS0nser0eOp0OqtWqELByuYzxeAy7\n3Y5gMAiPx4NwOAy73Y5YLIazszP4/X4RHzeZTBgMBqjX60vLE914sZN/mS6XC5FIBIeHh/j4xz+O\nRCIBk8kkjsevq4S4istOnOSP64O48tf3ej1UKhWcn5/jyZMnmnSSy7a88gksbWeHw6FIV6FrHcpv\nFOuJ/qCOHAGPxwOPx4NkMonDw0M8ePAAqVRKeGIUyrkKOZ7c7/fRbDZRKpVwfn6ObDYLl8uFcDiM\nXq8Ho9EoQkEULychHY1G6Pf7qNfrM++cbsPGi50czO92u8ITIrGgX/ws0EkqXXK+EB04kCdoNBo1\notjtdlGv11EsFpFOp0VQ97JOECRuJHByzpL+2F6huAqj0ag5gCPPjcSHtq8+n0/kkVLmgoy+ppUc\nC7vdDgAioZjsfjQaicwBKp2kS05HodfRAeKy2Bqxo9NOCpaSaNAJ5yyQ2FH8jOIMJFoOhwNOp1Ns\nheXj+F6vJ8Quk8k8l6ek/17ovvL96d8kgArFizAajfB4PEgkEtjb20M0GkUoFBJXIBBAMBgUYkcx\nOr3w6O2TclWBixNdqueWPTaK5dG2GYCIA1qtVpHyZbfbRZx8WWyd2JFXR16QyWSaOWOb2tS02200\nGg30er3nPC/KLaL/j0SNPLtSqSTEDrj6yP2yjib0/rr0BVOsN+TZJRIJPHjwAPv7+4jFYohEIohG\no+Kwguz1qmYXV3l2JFyyaJFnR/ZOifjUBUWOrcuenRK7W0BBU+Biy9hsNlGpVFAoFMTTxel0ip5a\n5CkB0HRf1W91Za+KTlQLhQKKxaLw7kjw5G2C0+nUnAA3Gg1xNZvN5f5wFDuDnOtpt9vh9XoRiUSw\nv7+Pg4MDTU86fcsxemjLuwq65O7EtNWVD/Po9VTKWKlUNIn1VDcrxxFJZJddM7sVYke/ICqrymQy\nePr0KXq9ngjKer1eTY4QAE08Qy92cpyMCpqPj49xfHyMZrMpTkJHo5GIj1AQmDq5Op1OlEolNJvN\ntWlgqNhOqATMbDbD4/HA5/MhEAiIbavL5RJbUBlqH0axbsqd63Q66PV6zyXey2JIh2bNZhP1eh1m\nsxkAxN8FdUWhOlpZZJcZqyO2QuzoZLPX66FarSKbzcJms6HdbmsCs3Rk3m63RamWwWAQScUy9Muk\nI/bz83M8evQIH3zwAWq1mmYbS4Lqdrvh9/sRCoVEV1cldoplIMfEZLELBoOatBK9yFCIptVqodls\notFoiPbqlDvHORd5qTR3gmLYstgBEB8fj8eiG9Blsb9VcK3YMca+CuCfAchxzn9m8jE/gP8CIAXg\nCMCnOOe1Ba7zSuR4FgCR+zMej9FoNBAOh0W1AcXcGo2GeNK43W7EYrHn7iuXa1HqyMOHD/Hee++h\nXC5rXH25KkM+7er3+6LagbxJxfqw7rZ9G0wmE6xWqyi8pwd8KBSC3+8XW1E9cmPYWq2GUqkkwjXl\nchmcczgcDoRCIQDPx8ipgWy9XsdwOBSnsQAQDodFmtRNG+Eukpt4dl8D8IcAvi597IsAvs05//eM\nsS8A+L3Jx1YK5xzdbhe1Wg0Gg0E8ZRqNBiqVCrrdrii/MhgMcDgccLvd8Hg8GAwGmi4o3W4X1WoV\nhUIBZ2dnyOfzqFQqaLVa4slFV6fTAQDRxsZkMomTKWq0SeVgirViY2xbD8XAKG0kFAohGo0iFosh\nmUyK/ohOp1MTbwO0KU6tVgvZbFZUNlSrVTSbTdEJu1Qqwe12w2azoV6va8odnz59iqdPnyKXy6HR\naAjRpJw9ShjmnIueedT3jrbGbrcb7XZbs1NaFNeKHef8e4yxlO7DrwP4h5P33wLwXayBQYzHY/R6\nPTQaDbENpQMLmoZEtXwmk0lTHytvR0nsqMzr9PQUhUJBeGj6PDnyAuUTVCrxajabKBQKSuzWkE2y\nbT10+k/CEolEcOfOHdy9exeHh4eiewn1UZRTS0iU6EDh+PgYR0dHODo6QqvV0th2uVwWIR6bzYZK\npYJyuYxKpYJMJoPz83Mhdp1ORyTxOxwO0ayCHADqdzccDkXqi9frRbPZFDHDlYrdFUQ45zkA4Jxn\nGWOROa5paihuR95Wo9EQ2eE2m008zfr9PqxWq/DqqAMJ51z8YkmsqJc+iV2/3xe/EDIIOhyR3Xs6\nlZLz8xQbwVrath4SO2ovFo1GcefOHXz84x/HgwcPhG3b7XZNqzMAovqhVqshl8vh+PgYH330ET76\n6CP0ej04HA7RXbtcLoMxhn6/D8YYcrmcuCqVimhwQaEhuR8kpWlRWaTD4RAHgdT/0ePxoFargTEm\n/o4WlV41rwOKtUn+IkEDtMNEjEajiDeMRiPYbDYxEi4YDIoESWrnRJ1IqK2S3D/usmRgihlSHati\na1gb2waelSZSJYPb7YbX60U0GsXBwQHu37+PBw8eCJuXt69ko91uF41GA6VSCblcTtRtP3r0CKPR\nSJzA+nw+4UDUajWMRiMxTSydTqPdbmsqe2RqtRra7bYQL6PRqKmakA8OabTAYDBY6A5oWrHLMcai\nnPMcYywGID/PRc0TfaNB2UXX5xTJn6N21dFoFM1mU5zKqjboW8/a2rbcuMLtdiMYDIpT/3g8jkAg\noOnUI1fqyBU5dBAhd9GhjIHhcCge1oPBAM1mU3QGGo/HKJfLqNVq6PV6opLoJp6Y3JiAQkihUAh7\ne3ui0olEddWeHZtcxDsAfg3AlwF8BsDb813WfJB/0Xqxk/9NOXOXiV0kEkG73Ua1WkUul1Nit31s\nhG3TgQQdoLndbiFy1FKdUkxoF0P2TPZNV7VaRbFYRDabFWLXarXErqjZbIrwC53imkwm8MmUO7m9\nmLyruW79V4kdVTzRdnZR3CT15E8B/AKAIGPsBMAbAP4AwH9jjP0GgGMAn1rYCmdkPB6DMXZpZ1R9\nzan8lLLZbKLDaq/XQy6XE/Mh9GU0is1k02xbLqb3eDwIBoNIJBJIpVKIx+NC7Kh9ktzAQm4bRmKn\n9+zI1knoyNblww39Tgi4ebdhEjzqFxkKhcTBRK1WQzabXa3Ycc5/9YpP/eKc17IwLvtlyHWrxWJR\nBHT9fr84kbJarfD5fOh0OggEAuKonA4pVGH+ZrNJtm00GuFwOMRclHg8jr29PTHDJBQKwe12w2w2\nYzQaiWTfZrMpPDG6zs7OxClquVzWxKNv6qlNAwkZTTVzOp3iRJaSnhfJxldQTMtoNEKz2UQ+n9fU\n8VFffhog4nQ6RSCVLkqeVANvFItG9oY8Hg+i0aiYBkZdhmOxGNxut9h59Pt9VCoVMYy9Wq1qeiGW\ny2Vx1et1cdCwrN2KXCdLqTPLaAqws2I3Ho/FHAhKC6FyL+qwSuU3nHMheMFgEL1eTxyVd7vdFX8n\nim1FrjqgQ4lYLIY7d+7g4OAA8Xgc8XgcsVhM9KWjQH+5XMbp6SmePHmCQqEgvDzy4uiitCh5CM4y\noA4qsthd1lNvnuys2JFnR23TgYvcn3A4jEQiIUYkUhtp2bOjjG8ldIpFIwf1PR4PYrGYSBymwTlU\nykWxtH6/L8Tuxz/+Mc7Pz1Gv18Ul58NR1dGqPbtldEHZWbEDtGkprVYLhUIBx8fHsNls2Nvbw3A4\nFKdf1Ayx2WzC7XaLSUnlclnE7qgMh56alOCsUEyD3BLJ7XaL4v5wOIxgMCgaWBgMBjQaDVSrVVSr\nVeTzeTx69AjHx8diGh1VTHS73efE7jYpJPNmmfWyOy12cipKp9NBoVCA1WoVwkWjGanagjy+QCAg\nhK5cLmvErdVqCaOj0jKFYhoo1YROX1/UyaTZbCKTyeDk5ASnp6c4Pj7GycmJiNlRGydKuJezChZd\nk7ouKLGbiFG73UahUMBoNEK9XhdCt7e3J+ZpJhIJ0bZGDvJSexyqNaQmBJQZrlBMA4md0+kUPRlp\nLrLf7xfVQQDQbDaRTqfx8OFDPHr0SJR05fN5kV1AokaelDxCYBe6YO+02AHP0lIo14d64lEOU6lU\ngt/vFxOS6AlLRkelZHQVi0UwxkRyJg0B1k9PVyiuw2w2i2wA2rpS6onL5RL2RA/WXC6Hp0+f4uHD\nh2J3UavVVtpLcZUtnfTsvNgRNKCagqQU4KUOx9QdhdrdeDwe0cmh3W6LU61gMKhp9U41gvR5eZiO\nEj3Fi6AZrAcHB0ilUkgmk/D5fCKXTk4UppnCNFe41Wot/YRVz2WdiVdp80rsJsizLIbDIUqlEs7O\nzmC1WjEYDJBIJMQ21mazif5cVGVBWerValXUMAJAsVgUBkilafT/KbFTvAi73Y5QKISDgwM8ePAA\niURCiB21M6P+jHqxoxjdOsSMV92OnVBiN4E8LYpplEolcVjR7XbBOYfb7QZwUUoml9LI21NqHErx\nEBpVR8086f/ahYCwYjZsNptG7KiKRxY7atWkFzt5wt0q0ccH5Y8vW/imbcv+BoDP4llHiC9xzv9i\nYatcErK3RZnmBPXFM5vNCAQCcDqdou+X3I2CMYZIJCKaFuqTJim5s9VqiSev8vJWw7rZttySzGQy\naYbmhMNhMbrQaDSKRrRU7lipVEQDzU0YAUDbcMpkWEbmwrRt2QHgTc75m/Nf0npADQ4Jk8mE4XCI\nRqOBSCQiBg4Hg0ExjISEj5I/qecYfd5ms4n8PODZsCBVY7sy1sq2jUajsBeHwyEOJLxer2ZGK2NM\nlCzWajUUi0VREqbvK7dq9HOS6cFO+ajyg384HC70oT9tW3ZA2xZn6yCxo9gHCV0+n0cikcDBwYFI\n0PR4PAAg2uHQVsPr9YqmoCR21AmZnmbU3FCJ3fJZN9umBpeUZiK3LqcifxI7CovUajUUCoW1FTvg\ncsGjCiSKOVLLqEUyS8zuc4yxfwHgrwF8fhMmMN0GeSAJjZXL5/NwuVwoFouiPpZqZ+nAghKQKb7n\ncrnEVpa6OlCaS6fTEXEV1TZqrViJbRsMBiF2tGuQPTu58kH27AqFAiqViijoXwf08bjLPLtOpyPC\nOssoWZtW7P4IwL/lnHPG2O8DeBPAb85vWesFPYlI4OikljGGbreLYDAoLp/PJ8Yq0mBiSkQm6Ame\nyWRQLBZRKpU0sy0UK2Vlti03uNRf+n5y1J6sXC4/15Nu2dCaafvtcDhweHiISCQiRFoeOt9ut5HN\nZkWjgqOjIxQKBbRarfUTO855QfrnVwD8+XyWs57IbvdoNEK5XIbBYECn00G5XBbxOwokUycKu90u\n6mrpqU3zPWmyGQ3loVm0itWyatsm4aD5ESR01ICWxI4OJ6i9Oo35XLbYyS2oqHtyMBgUYud0OjWe\naKvVQq1WE4Osnjx5gtPTUxSLxbURO03rasZYjHOenfzzVwD87bwXtk7IxdLkgVG3lHQ6LUSOOqaM\nx2PY7XaEw2GYzWa43W5R3yj3zHM4HCIhdNEdHxRXsla2/SLvjoSOgvuNRgPlchm5XA61Wg2tVmup\nMTt5rSR2kUgE+/v7Gs+Oeuy1220h0NlsFicnJ3j8+DHS6bQYYr9SsbuidfU/Yoy9BmCMi6npv7Ww\nFa4JcgdX8vJqtRrMZrMoFatWq+j1eqLJIk0xo/pF+sVbLBY4nU5wzsUQbhpoop9Jq1gc62jbsoDo\nW6JTfHc0GqHX6wnxoPnEl035WiTUJp6yDkKhEJLJJA4PD5FKpRAKhURMW55oRrMv6CoWi0tpRjBt\nW/avLWAtG4NsdMBFTl6lUhEtoarVqigNo68ng7VYLHA4HBiPx2KcHF2UPyV3p1Asjk20bdn2KA52\n20lfsyAnCdNuherFqc/enTt3kEgkRFZCv98XQkdD5/P5POr1uhiMvYwHvKqgmBISsvF4jHa7LeIo\nstiR8ckGIg8L1rd7By66V9BWRaGQkTuUUOmh/HBc5PwIGfI2bTYb/H4/otGo6KB8584d0ViUvD4S\nu2KxiHQ6jZOTE+TzedF4Q57+t0iU2E2BbHSMMbRaLXQ6HTDGYLFYRM7TZYZHvfatVutznh09nXu9\n3gq+K8W6I7diusyzW0boQ95m2+12BAIBJBIJ4dGR2Pl8PgwGAyHIerErFAoaz24ZKLG7IXQyRmVh\nNEuTysDoisfjCAaD4hRKj34bQk/mXq8nYi6rrmdUrB9yblqj0UC9Xtd0NlmU0Mknw5RLSgnyJHIU\no4tEIiJGRzHsWq2GarWKk5MTcfJKXh3tiJaFErsbQDWLlBhMsynooqRhq9WKSCSCVCqFQCAgst1l\nZIEjo6VCbhqGso5Z8IrVwjkXNkNNY5vN5sJ3ASaTCRaLRQyfol6OPp8PiUQC+/v72N/fRzKZhNVq\nBWMM9XodnU4H6XQa5+fnSKfTopFoPp9HuVwW4x2V2K0ZNAmJ6l+pY2wgEIDP59MIXzAYFGJHjQEI\n2n7QSRpNe5LFjlx/hUJGFjsaCUAnsAtN1zCZYLPZ4HA44PV6NRPNEokEkskkEomEaH5Byc6FQgEP\nHz4UnZMpNYbGOZKdL3MXo8TuEuQDBTpUsNvtoioiFAohGo0iEokgEonA6/VqTqWokoJSTmSoEoNK\nZWTvbtF5RorNhbaxlJRL9rKIgyx5hCO1haeE4WQyiVQqhVQqhXg8Lv4O/H4/8vk82u02qtUqTk9P\n8fDhQ7z//vt4//331yIOrcRuAm1VKT5B3Sdoy+r1euHz+YQbL5+iUomMw+GAy+USW9vL6gPlUzSK\n0a1qspNivZEfunSo5fF4EAwGUalURAXOPKEwDbUvk7v7hMNhxGIxxGIxRKNRUf9N4ZezszOcnp7i\n9PQUJycnODs7Q61WW5sYtBK7CTSbk+Jy1EuMTkppqhM1UPR4PKLgn15DsQ2bzQaLxaI5oKATXPlg\not/vi5QB1dNOIYvbZZ+jJhPD4RCVSkU0mZhnE0yLxaIJ0yQSCbF1jUQi4oHv8/lEzSsV8h8dHeHx\n48d4/Pgxzs/PUSgUlNitI/KEcur9L8cnqBwsHA6L8i+6qJxHPrGVp5vLHR8oh07v2SkUlyELIImd\nwWAQnh21DJsXZrNZVADJW1Y6baW4tc1mQ6vVQrlcFmklx8fHePjwIT788ENkMhmRZbAxYscY28NF\nc8MoLkpovsI5//8YY34A/wVAChdlNZ/ahDZPcjxCThlxOByaoTqJRAJ7e3tIJpOIxWIad57mddIl\nIyd9kpjJMyqoeScVcFerVdHVWLFcNsm25dix0WgUYRXywOgUXy67ot2CvtZWTiXRd1YJhULY29sT\nF5220khR+WvJqysWizg/Pxcnr+l0GoVC4ZrvaPncxLMbAvhXnPP3GGMuAP+PMfYugF8H8G3O+b9n\njH0BwO8B+OIC1zozsvdFhkOxNq/Xq4nDRSIR4ckFg0GxbaVOJmQoemhKGXluzWYT9XpdXMViUVzU\n5oZmBiiWztrbthzaoFAL5xwulwvhcBgHBwcol8ua004qVaSLaljpIpt3Op2w2WyiKSiFb+jgLRwO\ni356VMpIgjocDlEoFER6yfn5OYrFIprN5tpmE9ykNjYLIDt5v8kY+xDAHoDXAfzDyZe9BeC7WHOx\nI2+OjtMpNuHz+RAOh0XZSzQaFXEJ6hJLsTir1SqE7rJYCedcdHjodDooFouagcWFQgGFQgHFYhHl\nclkMSFFb2eWzbrat7+irh8SOMQan0ynErtPpoFAoiPy7er2uESWaP0siR96g3++Hx+PRjBVwu93C\n7r1er2YsKM29oCubzQqxozZNjUZjc8VOhjF2COA1AH8FIMo5zwEXRsMYi8x9dXOGPDuLxSLEjkQu\nmUwKd317ZV5wAAAgAElEQVR/f19zwkpBYFngrgoKk2dHme6FQgEnJyc4OjrC8fGxELtCoYBGo6EZ\noK1YHeto23rxI/szGo3Cs6OJdTTikzpeU+hkMBiIU1w6WIvFYiIWHQqFREqVy+USp7B0ybNpKV2K\ndimZTAbpdBpnZ2c4OzsT3VfWta77xmI3cfO/AeB3J09B/eNnLY4S9e1x5EMDt9utyYcLh8PCXY9G\no+Ly+/2aqgiz2az5P+iJSU9PCsTSHE+axl6tVjVPv0wmg2q1qqmWUKyeTbJtgprCRiIXOkx9Er1e\nLyqVyqViR5fcbNbn82ke7OTFMcbEDoW2x/owDO1YstmsELp1OpDQcyOxY4yZcGEMf8I5f3vy4Rxj\nLMo5zzHGYng2em6lyAFYikPQ20gkIraplAhJMTrZdacpTlfF5Ujgut0uer2eRtwqlYrYTtA2Vd6u\nUtMAtW1dDzbJtoFngkdNYWk2MTWEjUajYhtLVQo0GoC6Y8sX7VwobYq6ctNFvRqpTI28uUwmI2KF\n1WoV9Xpd/D1stNgB+E8APuCc/wfpY+8A+DUAXwbwGQBvX/K6pUNxDTmNhK5EIiE6M+zt7WmedjQs\nhy59738ZeRp7s9lENptFJpNBJpNBLpfTPP1ocpI8H5O6myjWgo2xbYJOZimp2OPxwO/3i0ldnU7n\nObGjckaai0KXnEgv91OUW6jTg7xQKIhwzPHxsfDk6NKfBq8bN0k9+QSAfw7gR4yxH+DCpf8SLgzh\nvzLGfgPAMYBPLXKhV6E/UqcnFAkd/YKdTif29/dx7949fOxjH0MqlRInUuS+y9CxPT3p5PY6cslO\ntVrF2dmZMAA6laK43LJa7yhuzzrattwVR35L7cRkz44OHgBoYmskciQ+JpNJbFPtdvtz/598jUYj\n0YyWdip0ZbNZHB0d4enTp3jy5MnGhWFuchr7vwEYr/j0L853OTdDjsk5nU5NNYOc9EhiRlc8Hsfe\n3p7IlaOW6ZdtVanKQZ6IJMcuZGOgk9ZcLodSqSRc+nV15xUXrJttU25mu90WD1J6qNbrdSFw5I3J\n0MOeOu2YTCbhZVHnErJz+cFNIwZoC0r5n9lsFvl8Ho1GQ4w7rFQqKBQKosHsprFxFRT66UtyJwbq\npyUfs8vvy+1pyJvTdyYhaKgJdSehRGA5HkexODLGWq2GZrOJTqcjxE55dYqbIodHjEajELparYZG\noyEe5HqhA57VdtPfhpzcTqEdeh15cDRAik5XaerXyckJTk5OcHp6qtmmkgg3Go2NfJBvtNiZTCZ4\nPB4kEgncv38fh4eHIghLW1c5cVg+sKAn4FVxORr9RoHZ8/NznJ2diZo/ypOjma/UfJPicbT9UChu\niuzZcc41Xl29XhcJwpeViMlpKZR4DDyftkL/DzWkkFsyFQoFHB8f49GjR3j06BEeP34svlaebEb1\n3JvGWoqdPm1E7gpMBw8UYKV20DTsg0SOPDrqqko5SITscelPoGiOK4lZoVDQiF2xWBRxDKp+UFPB\nFLNC+XG0K6hUKsjn82L6HDWl6PV64uEtP7ivyv0kkSL7pqRgmvhFIZh8Po+TkxMcHx/j7OwMmUxm\nyT+BxbKWYkdPLzpsoPw48tjkE9ZYLCZqWKlQmSodKEfuMrcf0AZnqTEixSgobkFGUCqVxAkrtcSm\nY3bVtUQxD+RGEZxzcfhlMBjQbrc1+XFUxkXZBC/qfDIYDDQpJPV6XdMyXQ7LUJVPu91e4ne+HNZa\n7Kg/XCQS0eTGUba30+mE3+/XtF4ymUyai05qL0OOa3S7XXG8TrNc6crlciLNhNrZ0MmX6kWnmBck\ndvSWhqfTQHbqJVer1RCLxTAYDGAymURfuasYDAaarWqhUBCiViqVNFtlEsRWq7Wk73p5rLXYuVwu\n+Hw+xONx0WYmGo2KJxolANNFsYzrSroIeUoYiR0V5z99+lRcuVxOs81VnpxiEcgHB4PBAJVKBd1u\nF6VSSYRP6BCMhM7j8Vxrh+TZUemi/CCn7sKUaSDb97axlmJH+UPUgYEK9OnEVY7L0VaVRhRSKRcF\nYCl1hOr16JdIc16pJ365XNZMKc9kMmKQL3WRUIcOimVClTq0+7DZbDAYDM/VXp+dnV0ZqgGAWq0m\nSrvkwTcknlQNRJPKtpW1FTvqzkDtZuS2M3Klg7xdpYMGEjfKAqenFv0iKRAsl3bJyZN08KAfdr2N\nTzvF+kIPWIIOwyjpN5fLiXSqq0I1ANDpdDRpLHLLMaq22IVwzFqKnclkEvE4Ejkq0g8EApryFn1Z\nF+UOdTodEYiVT03lAwmq8Uun02KwNXl7cn6R2rYqVgE9vOVdBe1CqHMPPfRfFLKhIU9yipR+LMC2\nbl1lpulU/Mec8z9kjL0B4LN4ViT9Jc75X8xjUZQESb9Qi8UiOjHIaSJ6xuOx8OSopKtYLIqTVFm0\nOp2OGAxyenoqiqevurdi+1iFbd8GOaYMXJSEKaZn2k7F35p87k3O+ZvzXhSdPlmtVjFcJJ1O4/Hj\nx3C5XFe+jjLQqfyl1WqJEyZKyqSnV7/fR6FQEG3RyZXf5piF4jmWbtuK1TFtp+Lk5NPzG2skQWI3\nHA5Rr9eRTqdFZcR1o+MoeZIOJuSOI3LMbjQaaVJJSOi23ZVXPGMVtq1YHew2f9yTbq7fBfAqgM/j\nog1ODcBfA/j8ZUNJLmmEeC1U8qK/KEb3IuQiZ7nURfbY9GMN5c/vithxztUfs8SybFuxeK6y7RuL\n3cTN/y6Af8c5f5sxFgZQ5JxzxtjvA4hzzn/zktcpg1hDlNg9Q9n2djGT2E26uf4PAP9T1+SQPp8C\n8Oec85+55HPKINYQJXYXKNvePq6y7RfvCZ/xXDfXSbtq4lcA/O30y1MoVoay7R3hWs9u0s31fwH4\nES46uVI311/FxTSmMS4GCf8WTWTSvV49/dYQ5dkp295WZo7ZTYsyiPVEid3sKNteT2bdxioUCsVG\no8ROoVDsBErsFArFTqDETqFQ7ARK7BQKxU6w8NNYhUKhWAeUZ6dQKHYCJXYKhWInUGKnUCh2goWL\nHWPsk4yxHzPGHjLGvjDjvY4YYz9kjP2AMfZ/b/G6rzLGcoyx96WP+Rlj7zLGPmKM/SVjzDvlfd5g\njJ0xxv5mcn3yBvfZY4x9hzH2d4yxHzHGfmeaNV1yn3857ZoUt2detq3s+tr7zMeu5UHR875wIaY/\nAZACYAbwHoCXZ7jfEwD+KV7387iodXxf+tiXAfzryftfAPAHU97nDVx0u73NemIAXpu87wLwEYCX\nb7umF9zn1mtS161tam62rez6xveZya4X7dn9HIBHnPNjzvkAwJ8BeH2G+zFM4Y1yzr8HoKL78OsA\n3pq8/xaAX57yPrSu26wnyzl/b/J+E8CHAPZuu6Yr7qM67S6Hedq2suvr7zOzXS9a7JIATqV/n+HZ\noqeBA/gWY+z7jLHPzrQyIMInnSz4RXvuyAz3+hxj7D3G2H+8ybZBZtIh9zUAfwUgOu2apPv8n1nX\npLgR87RtZdfX32dmu960A4pPcM7/HoB/CuC3GWM/P8d7T5tw+EcA7nLOX8PFPIMbD2mZdMj9BoDf\nnTzB9Gu40Zouuc/Ua1KsBGXXN7vPTHa9aLE7B3Ag/Xtv8rGp4JxnJm8LAL6Ji63EtOQYY1FANGvM\nX/P1V62pwCfBBQBfAfD3b/K6SYfcbwD4E87529Ou6bL7TLsmxa2Ym20ru77ZfWa160WL3fcB3GeM\npRhjFgCfBvDONDdijDkmSg/GmBPAL+F2HWQZtPv9d3AxVAUAPgPgbf0LbnIfNn1X2+c65E65JtVp\ndzXMxbaVXd/8PjPb9bQnG7c4ofkkLk5THgH44gz3uYOLE68f4KKz7I3vBeBPAaQB9ACcAPh1AH4A\n356s7V0Avinv83UA70/W9t9xEZ+47j6fADCSvp+/mfycArdZ0wvuc+s1qWs1tq3senl2rWpjFQrF\nTrBpBxQKhUIxFUrsFArFTqDETqFQ7ARK7BQKxU4wk9jNqxBaoVg3lG1vH1OfxjLGDAAeAvjHuDi2\n/j6AT3POfzy/5SkUy0fZ9nZimuG1ohAaABhjVAitMQimBgmvJVwNyX4RyrY3mKtse5Zt7LyL/BWK\ndUHZ9haiDigUCsVOMIvYzbXIX6FYI5RtbyGziN3civwVijVD2fYWMvUBBed8xBj7HC4Kew0Avso5\n/3BuK1MoVoSy7e1k4Y0A1InVeqJOY2dH2fZ6sojTWIVCodgYlNgpFIqdQImdQqHYCZTYKRSKnUCJ\nnUKh2AmU2CkUip1AiZ1CodgJlNgpFIqdQImdQqHYCWbpZwfG2BGAGoAxgAHnfJZJ5juHwWCAyWSC\n2WyG2WyGw+GAy+WC0+mE1WpFo9FAvV5Ho9FAt9vFaDTCaDTCeDxe9dK3HmXb12OxWGC322Gz2WC3\n2zWX2WzWfG2r1RL23Gq1MBgMMBwOMRgMsKxxrjOJHS4M4Rc455V5LGbXMBgMwlAcDgfC4TBisRhi\nsRg8Hg/Ozs5wdnaG8/NzVKtV9Pt9DAYD9Pv9VS99F1C2fQ1WqxU+nw+BQEBcwWAQgUAADocDAMDY\nReVWLpcTtpzNZtHpdNDpdDAejzEcDpey3lnFjkFthafGYDDAarXC7XbD4/Fgf38f9+/fx/379xGJ\nRPDBBx/AZDKh3W6j1+sBgPLqloey7WsgsYvH49jb29NcXq9XCB1jDI8fP8YHH3wAg8GAbrcLg8GA\n8Xgs7HoZzCp2HMC3GGMjAH/MOf/KHNYE4EIIGGOaHxgAcM7BOcd4PF6a+7sojEYjrFYrXC4X/H4/\notEoUqkUXnrpJcTjcbRaLRQKBZycnMBqtWI4HKLf74MxtvHf+wawMNveZAwGg7hcLhdCoRCSySTu\n3r2LO3fuiCsQCAB49ndrs9nQbrdRKBSQyWQwGAzQ7XbF55fBrGL3Cc55hjEWxoVhfMg5/97MizKZ\nYLFYYDabYbFYhPAZDAaMRiP0ej10u130er2N/qNnjIm4h8fjgc1mAwB0u100Gg10Oh30+33lza2G\nhdj2JmM0GjVxuWQyif39faRSKaRSKUQiEbjdbphMJnDOxUOZ3ur/Vpf9tzuT2HHOM5O3BcbYN3Ex\nqGRmgzAajbDZbHA4HHA4HDAajeIaDoeo1+vgnKPf72+02BkMBlgsFjgcDrjdbthsNnDO0e12Ua/X\n0W63N/573FQWZdubjNFohMPhgM/ng8/nw97enkbsvF6vEDsAGqFbB6YWO8aYA4CBc95kjDkB/BKA\nfzOXRZlMsNlsIpYln1iSN9fr9dBsNufx360MWew8Hg+sViuAC88OANrtNgaDgfLslswibXuTIbHz\n+/2IxWLPeXYWi0XsyK7y7ORr2czi2UUBfHPSwNAE4D9zzt+dx6LIs3O73fD7/eKHaLFY0Ol0hNAt\nc7+/CAwGA8xmM2w2G5xOpziy1xuHYukszLY3GXo4O51OeL1e4eF5vV54vV5NPA/YIs+Oc/4UwGtz\nXIvAZDLBbrfD7XYjEAjAarXCZrPBarWi1Wqh2WyiXC5vhdhRfFKfq0TxSpPJtPHf56axSNvedGR7\ntVqtMJvNMJlMzx0oriOzHlAsBHkb6/f7hQDYbDbYbDaUSiVYrda1/sFeBxmG0WgUhxTy90liZzQa\nN/r7VGwPZK9msxlWqxVWq1XYKIndOrOWYkenPh6PRyQo0mW1WpHJZDZa7Gjd5NlZrVaN0NntdvEE\nlZ+a8msVimUjP5wv8+zWnbUUO7PZDJfLhUAggFgsBpPJBJPJJLycdXeXX4Qc17DZbPB4PAgGg4jF\nYnC73bBYLCK9ptVqod1uo91uo9PpYDAYYDQarU0MRLH90OGgyWRCIBBAOBxGNBpFMplEJBLRHKyN\nx2NR0kj222w20Wq18JOf/ARPnz5FPp9Hs9lEt9tdaqkYsKZiZ7FYRMJiPB7HcDgUP8RNh7w5k8kk\njvHD4TCSySRMJpMoB2s0Gmg2m8JYSOyWVVqjUAAQNdsOhwOhUAjRaBSJRAL7+/uIxWLw+XxC7Cjp\nfTAYoFarIZPJIJ1OI5PJ4OzsDKenp8hkMqjX6yvJIV1LsSPPLhgMikoC8nA2HflQwm63w+v1IhwO\nI5FIYDgcolqtot1ua8Su2Wyi0+moJgCKpUOnrx6P5zmxCwQComkFAIxGI/T7fXS7XVQqFRwdHeHh\nw4f46KOPUCwWUa1WUa1WUa/XxUN958WOvB6v14tgMAij0bj0OrpFwBiD2WwW8Tmfzwe/349gMIhw\nOCzc/sFgIDw62sJu+veu2BwoTGQwGMQDmYSOGlVQ2IV2KZxzDAYDdDodNBoNFItFnJ2d4eHDh/jh\nD3+IRqOBfr8vPL/xeCyuZbGWYkfIQflNjdHJMMbgdrsRDocRiURw9+5dxONxuN1uURhNNYMkcMPh\nUMXoFEuDDiDoxDUWi4ni/v39fezt7SEQCDx3eMY5R6fTQblcRj6fx+npKXK5nNip9Ho9EYYhkduo\ncrFFso2nj4wxuFwuxONx3LlzB/fu3UM8HofH49GIXa/XEwaixE6xTKg5hcPhgNPpRCwWQyqVwr17\n93BwcIBoNAq/3w+r1ao5MByPx2i32yiVSjg/P8fZ2Rny+Tyq1ap4cFMYhoROiZ3ENgkdcBGvc7vd\niMfjopWTLHaj0Uh5doqVQtVLLpcLPp8PsVgMh4eH+NjHPoaDgwNNc1k5B5Rzjna7jXK5jPPzc5ye\nngqxo7LHVVcFXZscwxj7KmMsxxh7X/qYnzH2LmPsI8bYXzLGvDMvRArcy/k7lHKyCUmLN4FaOgWD\nQQSDQZFuwhjDaDRCt9tFs9lErVYTjQDUocRiWJZtbxL6LIFIJCJidZFIBF6vVzTnAJ6dwLbbbdRq\nNRQKBaTTaaTTaZRKJRGDJq9ulQ/um2QCfg3AP9F97IsAvs05/xiA7wD4vZkWoasRdTqdcDgcsNls\nmtjANkCJmXTJtYTD4RCdTgf1eh2VSgWNRgO9Xk+J3eJYuG1vGnImRCKRQDgcFgJHsTzy6Ojh3Gg0\nUKlUUCqVkM/nkc1mkc/nUavV0O1212Zncq2CTHp46VtTvw7grcn7bwH45VkWQaeU5D5TUTyJnfwD\n3mTkU67LEqUHg4F4QpbLZZF8uQ35hevIMmx709CLXSgU0oidfCihF7tisYh8Po9MJrOWYjdtzC7C\nOc8BAOc8yxiLzLIIOSWDxI5Kwy4rmdpkyJOTPTv6vmTPrlwuK89uNczVtjcNfY6r7NlRPh1BlRLN\nZlN4doVCQXh2q47R6ZnXAcVM3w3NYqASMZ/PpwmCkkBsotjJ3hwV98tdXMhrBSCGj1An5n6/rw4o\nVs/W//DJPhljzyW6B4NBuFwuMS1MFjA6kMhmszg/PxepJnTyum5MGwjLMcaiAMAYiwHIz7IIuX+d\nXuw2WeiAZzE6+fCFiv2puwltZTnnIuBLeUmqFnbpzNW2NwHaadDuSha7QCAAl8slEodHo5EYgdhq\ntcTp69HREbLZrKiOWEduKnZschHvAPi1yfufAfD2TIvQFcX7/X64XK7ncnk2kcva4lCrKlnsgGee\nnRK7pbJQ2153yKu7TOySySSCwSCcTqcQOyr2J7ErlUpIp9N4+vQpcrncWovdtdtYxtifAvgFAEHG\n2AmANwD8AYD/xhj7DQDHAD41yyLkbazf79cEROUM7VW3dZ4GuSWOHI+02+1iG0vfn/zUpBy7VR/X\nbzPLsO11Rt+yyWaziTmwoVAI4XAYwLPBOGSfVPZVr9dRLBaRTqdxcnIiMgg2Vuw45796xad+cV6L\nYIyJvm5yyonBYBAiQLk8rVZro5JtbTabGB4ciUSwt7eHUCikeVpSGQ0lEpNHpwr/F8sybHsdkQ/I\nbDabyPkMBoN46aWXkEwm4Xa7n2upPhqN0Gq1UK/XUa/XxcDrQqGASqWCZrO51gdqa1FBoRc7aksu\nJy5SCRWVUa1jAPQyrFYrgsEg9vf3cXBwgP39fc3WgLYEo9HoObFbVVmNYruhLSudvFJJ2MHBAQ4P\nD4XY6e1uNBqh2WyKFBMSu2KxiEqlgl6vt9Z/m2spdnrP7jKx2zTPbn9/Hw8ePNB4dmazWWxbu92u\nmIUrF0xvwveo2CwoM8BqtcLtdiMajeLevXt45ZVXkEwmEQqF4Ha7n3vdcDgUcbrz8/PnPDuK5ynP\n7gXog6RyiRhw8UPudrtotVpotVrodrtr0cRSTiuRqyLo9NVoNCKRSIiOEQcHBwiHw6K7K42ErNfr\nqNVqKJVKaDQaIpF4HUpsFNsHORZU/xoOhxGPx0WhPyX1AxB2SFvYSqWCXC6H09NTURJWr9fF+M91\nZi3E7jpI7BqNhhCDZbd0lpFFjk6S6bLb7aJjhMPhwN7eHu7du4f9/f3nagtp4Hcul0Mmk8H5+TlK\npRJarZam55dCMU/MZjOcTid8Pp/IfqAB15Q8TCEkeddRLBaRy+XEgUQ2mxV5dZvAWosdxatI7OSu\nvasUO0Ab5KXhQG63G16vF36/X1zxeBx7e3uiZz+JoclkQq/XQ6PRQC6Xw9HRkUbsaJuuYnaKeUOt\n1unkVS92crycShgbjQZKpZIQu9PTUxSLRdTrdSV284Q6oDabTeHZrXIbK3t1NOOWcgTD4bCmm2sk\nEhHH+IFAQGzPjUYjBoOB8OwuEzuFYhFc5dl5PB44HA5NXivl01WrVeHZUQunWq0m0lA2gbUWO32n\n4ttOFtN3GNF/Tp6cJE/9kjus6Eu95JgivaWJ6HRRu3WqAtHP1qT1j8dj4bFWKhUxiGRd85QUmwnZ\nL12RSATJZBIHBwdIpVJiShilQtGAq+FwiEKhgEwmg0wmg5OTE5yfn6NcLosUsHU+kNCz1mJHyCVX\n5GLfpOWTfOpEPeMI2n7SRQ0H6JJFiY7o3W43XC6XOCmmS74PJQqTwAHPtuPj8ViTIC3nDzYaDY0B\nKRTzgnrUUfu0RCKBg4MD3Lt3T4gdzZOg4n5qIJvNZnF0dISnT59qxG4Tk943Quxo26dvi3ST19EU\nL3LPCZPJBI/HI9x3Ejy6ZC+McuVCoRCCwaAY0C1vZ8l75JxrBovIcTcSO9mz6/f7YkjJpuUQKjYD\nuSGn3+8XYnf37l2kUilR2UOhFdptNBoNZDIZHB0d4aOPPsLx8TFqtRqq1Sr6/b4oZdwasWOMfRXA\nPwOQ45z/zORjbwD4LJ4VSX+Jc/4X81yY/EOkkhYSLfKibDbbC4WB8ojIK5O9QbPZjEAgIC6n0yk6\nkZCYkeDZbDbRrTUajcJms2nWKR/P93o91Go10WmYcuZkw5DFTw4At1otYUSKxbMq214m1D5NHjqf\nTCaxv7+PVCqF/f19TaiHcj0pTlcoFHB2dobHjx/j5OREPMj7/f7GiBxxE8/uawD+EMDXdR9/k3P+\n5rwWIm/tZNeYhtTEYjF0Oh3Y7XbEYjGUy2WUSqUbiR1dcgzQaDRqhFDfCp7SPkajkfDYms2meP1w\nOBQXuf29Xk9zkNJsNhEOh5FKpWA2m+HxeDQeoTxgh/IHVRv2pbIU214Fcr9Er9eLRCKBu3fv4u7d\nu5p8On1PRTmJv9PpoNPpCLuUk903kZvUxn6PMZa65FNza0Miezr66UM0fjAajcJgMMDn86FWq4lE\n3Bf94F/k2VHzAfLm9H3zSIRkd528Lzn3iJKd5Tmv8nVwcACLxYJAICC+N6oMkUcn0nxY8gIVi2cZ\ntr0K5E47ZrMZXq9XDHl69dVXRR2sy+V67uBML3b0EJf/FjYpTiczS8zuc4yxfwHgrwF8nnNem2Uh\nerGjHyh5dowxeDwexONxISqtVuuFP3SLxSK8OpfLJcSOnmL6Xnn6xoSdTue5t+S5yRfFMWq1miiG\nlsUwEAjg4OBAI+Ky2FEpXLfb3VhD2jLmaturgEI/VqsVPp8PiUQC9+/fx8/+7M+Kckw6lAPwnGdH\nsWS5jJFSTDbVPqcVuz8C8G8555wx9vsA3gTwm9MuglIwaJI4latQNjcF9inhkdol9Xq9a8WOTqCc\nTqdmayq/JW+K3PThcKjxzlqtFhqNBur1utieym8pRkdxOlmwKW6n9+joaF9umaNy69aCudr2KpBb\nprndboRCITElLB6Pv/Bwj2yfJoVVq1URS97U7SsxldhxzgvSP78C4M9nWcRgMEC1WsX5+TnG47Em\nVtButzUnsdQSia4XQSJG6R30tJK7i1Asglx1+UkmX7J3R8fy9L68fR0Oh5qDjkgkIjovm81mjdjR\nGjbdiLaJedv2KjAYDHA6nULkkskkAoEA7Ha72C1dBR2wUcrJujfkvA03FTtNN1fGWIxznp3881cA\n/O0sixgOhyL+Vq/XxVQtEhU5j41ygW4SOzAajULgTCaT8MLIQ5O3muTB0SUfQFBnEv1FHyfB7Pf7\nIlZCZWN6saP2TSSwqhvxylmoba8CWez29/c1Yge8eBva7/dRq9VEykmhUEC9Xt+YKokXMW2n4n/E\nGHsNwBjAEYDfmmUR5NnV63UYjUYR96L25FSZ4PF4YLPZxC/rOoHQV2CUSiUUi0UUCgWRBS5vU+WD\nDzn15bIuyVd9jvL1vF6vKBfz+/1iC05CLbdeV57daliGba8Co9EoxI7qsv1+vyZl6ip6vR6q1arw\n7ChcsxOe3RXdXL8274WQCFC9aLFYFP3e5BNVi8Uy9f9Rq9VQqVRQLpfFTEvajtKJKl3TQknI+klp\nVMFBtYY0CJuqJpRnt3yWZdvLQE5sdzqd8Hg8CAQCoiabZrrooUMy2qVUKhVUq1Vx4EYhn214IK9l\nBQU9XRhj6HQ6mmlcdHo0DeTFUecUOUGS4nWzHhJcNk+DOi9T/zrqICF3jVBip5gFo9EoyiLdbjd8\nPp8o9JfDKHrocJD+NkqlkpglQWlW27L7WGux63a7qFQqmjKxm9TEXgWdfMq/QLn6geJzs6AXOypF\no8MVvdjRIGwldopZILGjDjwUMw4EAqKH4lViRw1kq9UqSqUSqtWqKF+k2LgSuwVBYtRoNFa9lFtD\n9aVr7LcAAAqSSURBVLhOp1P0CCOxAyDErlgsii6vSuwUs0JjED0ej+ilSKWQXq9XjO0EoIk3U6pJ\ntVpFPp9HPp/XdODZJtZS7LYV2bOjcjfl2SnmgcvlEiMAqKNJLBYTsTr9ACuK01Gx//HxMU5OTsT8\n11arteLvaP4osVsi1BFFv43tdrtK7BQz4Xa7kUwm8corr+DBgweIRqOIRqOi7vuymS6dTgflchnp\ndBpPnjzBo0ePcH5+jlwuN9Mh3bqixG7J9Ho9NJtNlEol5dkp5gZ5di+//DJeffVV0bZJzgSgVCx5\nzEG5XEYmk8HTp0/x4YcfolAoiJSsbUOJ3RKR52lQ8vImjYVUrC82mw0+n09MCZMb0eoP9fr9vuiO\nXSgUkMvlkM1mkU6nUalUxIHdtqHETqHYAvTdvPXjBWTkkrDz83MRTtn0ribXocROodgCZLEjb+6q\neS3dbhfValVMCiuVSqJKYluFDgCuTVpjjO0xxr7DGPs7xtiPGGO/M/m4nzH2LmPsI8bYXzLGvItf\nrkIxP7bJtqkhrH5Oy4vETvbsms3m1nt2N8nQHQL4V5zzjwP4BwB+mzH2MoAvAvg25/xjAL4D4PcW\nt8ztQG6qSF1RKFlasRI21rZNJhPcbjcikQhSqRTi8biof32RVwdo2zhVKhXReGMbEodfxLVixznP\ncs7fm7zfBPAhgD0ArwN4a/JlbwH45UUtcpuQ52nQNLKbDhBSzJdNtm0aAnVwcICf+qmfwp07dxAO\nh0VnE7kJhh46JKPmF9RzcRu9OZlbxewYY4cAXgPwVwCinPMccGE0jLHI3Fe3ZZBnJ4ud3C1WsTo2\nzbap1X8qlcLh4aEQO5qiR33rLutfJ4tdvV7fmuac13HjvzLGmAvANwD8Lue8yRjTPwa2+7EwI2Rw\nJHY2m02U8CjPbrVsom1brVYEAgHs7+/j5Zdfxt7eHoLBoGjjpBc8uURsMBiI8Z31el0M1FGeHQDG\nmAkXxvAnnPO3Jx/OMcainPMcYyyGZ6PnFJdwmSG9KK6iWA6bZNty7pzH4xFzYIPBoKbhBKDts9jv\n90XFRLfbRSaTQblc1oz63HavDrjZAQUA/CcAH3DO/4P0sXcA/Nrk/c8AeFv/IsXVKJFbGzbCthlj\nMJlMsNlsol+d3NmEGtvS2E+92NVqNeTzeRwfHyObzYpeijT7ZFtPYGVu0qn4EwD+OYAfMcZ+gAuX\n/ksAvgzgvzLGfgPAMYBPLXKh24gSvNWySbZNw64vEzvZs9OLndzCiSolstms6NQtp5tsOzfpVPy/\nAVyVG/GL813O9kO5UPIou1l69CmmZ9Nsm9KWbDYbHA4HXC6X6ODtcDhgsVhEGpM8SY8K/qkGNp1O\ni20seXXKs1PMHeo7RvNsqYux8vIU16EvCTObzWI2rHzQJc8kHg6HaDabKBaLOD09xaNHj5DL5VAu\nlzVzirdd6AAldkuFtiIkdh6PR3SQVWKnuA59SRiJncViee5UnzpvU5edfD6Pk5MTPHz4UEzX63Q6\nSuwUi8NkMl3q2SkUL4LKwWiwjix05NnR6T5tS2m8J3XGPj09xcOHD8U85V2J1RFK7JYIGay+YPuq\nGkaFQg/ZEAmb/iLkAwoa3UlT9HbBi7sMFRlXKBQ7gRI7hUKxEyixWyC7ul1QLJbL7GpXDhlmQYnd\nnKGhwzRBrFqtimHDCsW0cM4xGo3Q6/XQ6XTQbDZF1xKysV6vt5Xt1OeFErs5Q2JXr9fFwGGqQVQo\nZoGGvLfbbbRaLVHIX6vVRE86JXZXo05j58xoNNJ4dsFgUImdYmZkz85gMAjPrl6vo1qtilNa1TLs\nam5SG7sH4OsAogDGAP6Yc/6HjLE3AHwWzzpCfIlz/hcLW+mGIHt2hUJB1C/SVPZ8Pi88Pkrs3IXG\nievIptn2aDQSD00ag5jL5XB6eopWqwWv1wuv1wu73a7pclIul0Xb9V3mJo8Aal393qTv1/9jjH1r\n8rk3OedvLm55m8doNEK73Ua5XBYC1u12USwW8ejRI1QqFZTLZVQqFeTzeWSzWdTr9Z1K7lwjNsa2\nybMbDocAgHq9jnQ6DaPRiHa7LebEUo1sv98XScWZTAZPnjxBpVJZ8XexWm7SCCALIDt5v8kY+xBA\ncvJplQmrg8SOc45eryeE7/j4GG63G+12W1wUYG40GkrsVsCm2fZ4PBaF+41GA+l0Gp1OB/l8Hlar\nVdNcQq6SoPZO1Wp1p3cQ7Dbf/KR19XcBvArg87jo+VUD8NcAPs85r13ymp376dJkJ+pwQhUTRqNR\nZLTTWzJKemIvC8752v0xr5JNsW2547XczFNfWSGnogyHQwwGA3FtO1fZ9o3FbuLmfxfAv+Ocv80Y\nCwMocs45Y+z3AcQ55795yet2Tuw2ASV2z1C2vV3MJHaT1tX/A8D/1HV0pc+nAPw55/xnLvmcMog1\nRIndBcq2t4+rbHvqtuyT3vzErwD42+mXp1CsDGXbO8K1nt2kdfX/AvAjXLStptbVv4qL0XNjAEcA\nfovGz+ler55+a4jy7JRtbyszx+ymRRnEeqLEbnaUba8ns25jFQqFYqNRYqdQKHYCJXYKhWInUGKn\nUCh2AiV2CoViJ1j4aaxCoVCsA8qzUygUO4ESO4VCsRMosVMoFDvBwsWOMfZJxtiPGWMPGWNfmPFe\nR4yxHzLGfsAY+7+3eN1XGWM5xtj70sf8jLF3GWMfMcb+kjHmnfI+bzDGzhhjfzO5PnmD++wxxr7D\nGPs7xtiPGGO/M82aLrnPv5x2TYrbMy/bVnZ97X3mY9fU92oRFy7E9CcAUgDMAN4D8PIM93sCwD/F\n634eF7WO70sf+zKAfz15/wsA/mDK+7yBi263t1lPDMBrk/ddAD4C8PJt1/SC+9x6Teq6tU3NzbaV\nXd/4PjPZ9aI9u58D8Ihzfsw5HwD4MwCvz3A/him8Uc759wDoe1K/DuCtyftvAfjlKe9D67rNerKc\n8/cm7zcBfAhg77ZruuI+a9tpd8uYp20ru77+PjPb9aLFLgngVPr3GZ4teho4gG8xxr7PGPvsTCsD\nInzSyYJftOeOzHCvzzHG3mOM/cebbBtkJh1yXwPwVwCi065Jus//mXVNihsxT9tWdn39fWa26007\noPgE5/zvAfinAH6bMfbzc7z3tAmHfwTgLuf8NVzMM7jxkJZJh9xvAPjdyRNMv4YbremS+0y9JsVK\nUHZ9s/vMZNeLFrtzAAfSv/cmH5sKznlm8rYA4Ju42EpMS44xFgVEs8b8NV9/1ZoKfBJcAPAVAH//\nJq+bdMj9BoA/4Zy/Pe2aLrvPtGtS3Iq52bay65vdZ1a7XrTYfR/AfcZYijFmAfBpAO9McyPGmGOi\n9GCMOQH8Em7XQZZBu99/BxdDVQDgMwDe1r/gJvdh03e1fa5D7pRrUp12V8NcbFvZ9c3vM7NdT3uy\ncYsTmk/i4jTlEYAvznCfO7g48foBLjrL3vheAP4UQBpAD8AJgF8H4Afw7cna3gXgm/I+Xwfw/mRt\n/x0X8Ynr7vMJACPp+/mbyc8pcJs1veA+t16TulZj28qul2fXqjZWoVDsBJt2QKFQKBRTocROoVDs\nBErsFArFTqDETqFQ7ARK7BQKxU6gxE6hUOwESuwUCsVO8P8Dv9o9yOe5wrgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f60d8153b70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot ad hoc mnist instances\n",
    "from keras.datasets import mnist\n",
    "import matplotlib.pyplot as plt\n",
    "# load (downloaded if needed) the MNIST dataset\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "# plot 4 images as gray scale\n",
    "plt.subplot(221)\n",
    "plt.imshow(X_train[0], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(222)\n",
    "plt.imshow(X_train[1], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(223)\n",
    "plt.imshow(X_train[2], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(224)\n",
    "plt.imshow(X_train[3], cmap=plt.get_cmap('gray'))\n",
    "# show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the classes and functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy \n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Dropout\n",
    "from keras.utils import np_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# fix random seed for reproducibility\n",
    "seed = 2017\n",
    "numpy.random.seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A baseline Neural Network Model<a class=\"anchor\" id=\"fifth-bullet\"></a>\n",
    "Do we really need a complex model like a convolutional neural network to get the best result with object recognition?\n",
    "\n",
    "Actually, we can get very good results using a very simple neural network model with a single hidden layer. In this section we will create a simple multi-layer neural network model. We will use this as a baseline for comparing more complex convolutional neural network models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training dataset is structured as a 3-dimensional array of instance, image width and image height. For a multi-layer neural network model we must reduce the images down into a vector of pixels. In this case the 28×28 sized images will be 784 pixel input values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# flatten 28*28 images to a 784 vector for each image\n",
    "num_pixels = X_train.shape[1] * X_train.shape[2]\n",
    "X_train = X_train.reshape(X_train.shape[0], num_pixels).astype('float32')\n",
    "X_test = X_test.reshape(X_test.shape[0], num_pixels).astype('float32')\n",
    "# The pixel values are gray scale between 0 and 255. \n",
    "# It is almost always a good idea to perform some scaling of input values when using neural network models\n",
    "X_train = X_train / 255\n",
    "X_test = X_test / 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One Hot Encoding of Labels\n",
    "Our labels are integers from 0 to 9. This is a multi-class classification problem. As such, it is good practice to use a one hot encoding of the class values, transforming the vector of class integers into a binary matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 10)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# one hot encoding \n",
    "y_train = np_utils.to_categorical(y_train)\n",
    "y_test = np_utils.to_categorical(y_test)\n",
    "num_classes = y_test.shape[1]\n",
    "y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Designing Neural Network Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#create our baseline neural network model\n",
    "def baseline_model():\n",
    "    # create model\n",
    "    model = Sequential()\n",
    "    # insert a hidden layer with 784 neurons\n",
    "    model.add(Dense(num_pixels, input_dim=num_pixels, init='normal', activation='relu'))\n",
    "    # insert a output layer with 10 neurons\n",
    "    model.add(Dense(num_classes, init='normal', activation='softmax'))\n",
    "    # Compile model\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    return model\n",
    "# build the model\n",
    "model = baseline_model()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is a simple neural network with one hidden layer having the same number of neurons as the input (784). \n",
    "Before we proceed to train our model, here are some important things to know:\n",
    "\n",
    "**activation function**: each neuron consumes a sum of weighted inputs from the previous layer and processes it through a activation function\n",
    "\n",
    "**RELU/relu**: the RELU activation function has the form f(x) = max{ 0 , x }. In other words, the activation is simply thresholded at zero.\n",
    "\n",
    "**softmax**: an activation function used on the output layer to turn the inputs into probability-like values\n",
    "\n",
    "**loss function** used to measure the gap between actual observations and predicted values\n",
    "\n",
    "**optimizer** an algorithm used to update parameters in a NN as batches of data coming in and reduce loss.\n",
    "\n",
    "**accuracy** the proportion of observations which are correctly classified in the validation set \n",
    "\n",
    "**epoch** one epoch means one pass of the full training to update parameters in a NN\n",
    "\n",
    "**batch_size**  the number of randomly selected samples used to update parameters each time\n",
    "\n",
    "Now, we can fit and evaluate the model. The model is fit over 10 epochs with a parameter update every 200 images. The test data is used as the validation dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train our Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "7s - loss: 0.2788 - acc: 0.9203 - val_loss: 0.1399 - val_acc: 0.9588\n",
      "Epoch 2/10\n",
      "6s - loss: 0.1085 - acc: 0.9690 - val_loss: 0.0907 - val_acc: 0.9729\n",
      "Epoch 3/10\n",
      "6s - loss: 0.0704 - acc: 0.9794 - val_loss: 0.0750 - val_acc: 0.9771\n",
      "Epoch 4/10\n",
      "7s - loss: 0.0484 - acc: 0.9863 - val_loss: 0.0674 - val_acc: 0.9788\n",
      "Epoch 5/10\n",
      "6s - loss: 0.0359 - acc: 0.9899 - val_loss: 0.0605 - val_acc: 0.9816\n",
      "Epoch 6/10\n",
      "6s - loss: 0.0249 - acc: 0.9937 - val_loss: 0.0606 - val_acc: 0.9806\n",
      "Epoch 7/10\n",
      "6s - loss: 0.0188 - acc: 0.9956 - val_loss: 0.0651 - val_acc: 0.9803\n",
      "Epoch 8/10\n",
      "6s - loss: 0.0145 - acc: 0.9966 - val_loss: 0.0616 - val_acc: 0.9808\n",
      "Epoch 9/10\n",
      "6s - loss: 0.0105 - acc: 0.9978 - val_loss: 0.0568 - val_acc: 0.9819\n",
      "Epoch 10/10\n",
      "6s - loss: 0.0082 - acc: 0.9982 - val_loss: 0.0619 - val_acc: 0.9820\n",
      "Baseline accuracy: 0.98\n"
     ]
    }
   ],
   "source": [
    "# Fit the model\n",
    "model.fit(X_train, y_train, validation_data=(X_test, y_test), nb_epoch=10, batch_size=200, verbose=2)\n",
    "# Final evaluation of the model\n",
    "loss, accuracy = model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"Baseline accuracy: {:.2f}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The log message traced how the Neural Network learned by printing out the loss and accuracy it got on the validation set once a epoch was done.\n",
    "This simple Neural Network model achieves an accuracy of 98%.It is a quite impressive result since even a human will misrecognize some digits.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Simple Convolutional Neural Network for MNIST<a class=\"anchor\" id=\"sixth-bullet\"></a>\n",
    "We have seen how to train and evaluate a simple neural network.Now, it's time to develop a more sophisticated Convolutional Neural Network.\n",
    "A Convolutional Neural Network usually consists of three types of layers:Convolutional Layers, Pooling Layers and Fully-Connected Layers. \n",
    "\n",
    "**Convolutional Layers**: neurons in a Convolutional layer form a 3D volume which imitates the data structure of images (a image has dimensions width, height and colors). Each neuron only accepts inputs from a local region of the input, and such region is determined by a hyperparameter called **receptive field**\n",
    "\n",
    "**Pooling Layers** a 3D layer performs down-sampling.It partitions each slice of the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum value.\n",
    "\n",
    "**Fully-Connected Layers** Fully-Connected layers are 1-dimensional arrays of neurons which take all inputs from the previous layer.\n",
    "\n",
    "A simple Neural Network only has fully-connected layers, which means each neuron is connected to all inputs from the previous layer. This dramatically increases the number of parameters and the risk of **overfitting**.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers import Dropout\n",
    "from keras.layers import Flatten\n",
    "from keras.layers.convolutional import Convolution2D\n",
    "from keras.layers.convolutional import MaxPooling2D\n",
    "from keras.utils import np_utils\n",
    "from keras import backend as K\n",
    "K.set_image_dim_ordering('th')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# load data\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "# reshape to be [samples][colors][width][height]\n",
    "X_train = X_train.reshape(X_train.shape[0], 1, 28, 28).astype('float32')\n",
    "X_test = X_test.reshape(X_test.shape[0], 1, 28, 28).astype('float32')\n",
    "# normalize inputs from 0-255 to 0-1\n",
    "X_train = X_train / 255\n",
    "X_test = X_test / 255\n",
    "# one hot encode outputs\n",
    "y_train = np_utils.to_categorical(y_train)\n",
    "y_test = np_utils.to_categorical(y_test)\n",
    "num_classes = y_test.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convolutional neural networks are more complex than simple neural networks in the sense that there are different types of layers. To effectively use CNN's, we need to stack these layers in a wise way. During the development of deep learning, researchers have invented plenty of CNN architectures. We will start with a simple one which has the form:\n",
    "\n",
    "**INPUT -> CONVOLUTION LAYER ->POOLING LAYER -> OUTPUT**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define the structure of our CNN model\n",
    "def CNN_model():\n",
    "    # initialize the model\n",
    "    model = Sequential()\n",
    "    # insert the first Convolutional layer with receptive field 5*5 and depth 32 \n",
    "    model.add(Convolution2D(32, 5, 5, border_mode='valid', input_shape=(1, 28, 28), activation='relu'))\n",
    "    # insert the first Pooling layer which takes the maximum of each 2*2 sub-region\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    # then, randomly drop 20% of the neurons in the previous layer (by setting their values to zeros)\n",
    "    model.add(Dropout(0.2))\n",
    "    # insert two fully-connected layers \n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dense(num_classes, activation='softmax'))\n",
    "    # Compile model\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    return model\n",
    "# build CNN model\n",
    "model = CNN_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We evaluate the model in the same way as before. The CNN is fit over 10 epochs with a batch size of 200."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "37s - loss: 0.0114 - acc: 0.9964 - val_loss: 0.0344 - val_acc: 0.9898\n",
      "Epoch 2/10\n",
      "37s - loss: 0.0118 - acc: 0.9963 - val_loss: 0.0345 - val_acc: 0.9900\n",
      "Epoch 3/10\n",
      "37s - loss: 0.0096 - acc: 0.9970 - val_loss: 0.0408 - val_acc: 0.9884\n",
      "Epoch 4/10\n",
      "37s - loss: 0.0082 - acc: 0.9975 - val_loss: 0.0410 - val_acc: 0.9882\n",
      "Epoch 5/10\n",
      "37s - loss: 0.0074 - acc: 0.9978 - val_loss: 0.0434 - val_acc: 0.9885\n",
      "Epoch 6/10\n",
      "37s - loss: 0.0068 - acc: 0.9980 - val_loss: 0.0386 - val_acc: 0.9890\n",
      "Epoch 7/10\n",
      "37s - loss: 0.0074 - acc: 0.9977 - val_loss: 0.0482 - val_acc: 0.9878\n",
      "Epoch 8/10\n",
      "37s - loss: 0.0056 - acc: 0.9983 - val_loss: 0.0405 - val_acc: 0.9886\n",
      "Epoch 9/10\n",
      "37s - loss: 0.0054 - acc: 0.9980 - val_loss: 0.0382 - val_acc: 0.9904\n",
      "Epoch 10/10\n",
      "37s - loss: 0.0058 - acc: 0.9982 - val_loss: 0.0490 - val_acc: 0.9883\n",
      "CNN Accuracy: 0.99\n"
     ]
    }
   ],
   "source": [
    "# Fit the model\n",
    "model.fit(X_train, y_train, validation_data=(X_test, y_test), nb_epoch=10, batch_size=200, verbose=2)\n",
    "# Final evaluation of the model\n",
    "loss,accuracy = model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"CNN Accuracy: {:.2f}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We increased the accuracy by 0.01 at the cost of longer training time. The MNIST data set does not give our CNN model a chance to show its advantages because the images were preprocessed very well. However, if you are dealing with raw and obscure images, CNN should be your first choice. Nowadays, most image classification competitions are won by a carefully designed Convolutional Neural Network.\n",
    "\n",
    "Let's look at some tricky cases where the CNN model beats most of humans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The digit looks like a 4 is actually a 6\n",
      "Our CNN model recognizes it as 6 with 100.0% confidence\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfVuIc+t53vNJmtGc55+9bW+Dd7PTkmtjWuobF+KQEEwJ\nuOTCNS7FSUrIRd0G2gs7vtlQehH7wuAGclHXMXaIycHg2im0dUIJxYU0bhK3dmPXgXY7ceL9e/+H\n0egwOoz09eKfZ/3PevWttaQZjUYz633gYy2tmZE+afSs9/y+IcYIh8NRLzRuewMOh2P9cOI7HDWE\nE9/hqCGc+A5HDeHEdzhqCCe+w1FDXIv4IYT3hBC+HUL4Tgjhw6valMPhuFmEq8bxQwgNAN8B8OMA\n/hrA1wC8P8b4bfN7nijgcNwSYowhdf06Ev+dAP48xvjdGOMEwG8CeO81ns/hcKwJ1yH+2wD8pTz+\n3uU1h8Ox4XDnnsNRQ1yH+H8F4Ifk8cuX1xwOx4bjOsT/GoAfCSG8EkLYBvB+AF9ezbYcDsdNonXV\nP4wxTkMIHwLwFTy7gXw6xvitle3M4XDcGK4czlv4BTyc53DcGm4inOdwOO4onPgORw3hxHc4aggn\nvsNRQzjxHY4awonvcNQQTnyHo4Zw4jscNYQT3+GoIZz4DkcN4cR3OGqIKxfpOK6OEAJCCLnzRRcA\nzGYzxBiTRwBg/YXWYfioNIfCib9mhBDQaDTQbDbRbDazc73G1Wq15s5jjJhMJri4uMgdeT6dThFj\nzN0Q9NzhAJz4a0cIAc1mE1tbW2i1Wmi1WnPn29vbhcfZbIbhcIjRaIThcDh3TvLPZrPcAuDkd2Rw\n4q8ZJH6r1cL29na22u12dmy329jZ2cHOzs7c+Ww2w2AwQL/fz61Wq4UQAkajEabTaW4Bruo78nDi\nrxmW+CQ11+7uLvb29nJLr81mM5ydnWWr3W5npCfJLy4ucHFxkfkE1P53OAAn/tpBG39rayuT5krw\n/f19HBwc4ODgAIeHh9k513Q6xdOnT7G3t5cj/Ww2w2QywWw2yzkCSfpGo4EQgkt+BwAn/tqRkvh7\ne3s5wh8fH+Po6Cg76vnFxcUc6afTKcbjMc7Pz5OSfjqdZtccDsCJv3aoc4/E393dzST88fExHjx4\ngJOTE5ycnGTnPE4mkyTph8Mh+v0+xuMxAGTe/Ol0imazmbshOBxO/FtAo9FISvzDw8NMunM9ePAg\ndyOgZB8MBuj1etjb28POzg62t7ezyADJ3mg0MhXfSe9QeObemqESX218SvzDw0Ps7+/PEbrR8H+V\nY3Xwb9OaUWbjHx0dZQ693d3dHPGbzaZLbcfK4MRfM1I2vkr8o6OjOYm/tbWVqewOxyrgNv6awXCe\nSnyG8ajqM7zHpB6X+I5Vw4m/ZpRJfKr6mtBDie/Ed6wSTvw1Q218de6pxKekZwovnXtOfMeq4MRf\nMzRzL2XjHx4eYmtrKyvc4bHZbN721h33CE78NUAlNctvtRJPbf2dnZ1cDJ5JOAAwnU6zarzRaITx\neIzxeJwryeXSyjwm8zgchBP/BmCbbOg5k2xIfvuYNfcksCVst9vF6ekpOp0Out0u+v0+BoMBzs/P\ns5uCvRk4+R0WTvwbAomumXNq39smG3zcaDTmymp1dbtddDodnJ2dodvtotfrod/vZ8Qn6a0W4KR3\nKJz4NwAlPYnPY5m0pxOPhCWJVaU/OzvD6elpjviU+DQBWJZrm3I4+R3EtYgfQngNQAfADMAkxvjO\nVWzqLsOS3q4yaU+JDyBXfDMcDjOJTuIXqfrj8Thp7zvpHYrrSvwZgHfHGJ+uYjP3CfYGoH31iqQ9\nJT5t/MlkklXdcXU6nVJVfzQaZRLepb2jCNclfoCn/c4hRfqiBpqphprAsy46rMTr9Xrodrs4OztD\np9MpVfXH4/Fc911vtumwuC7xI4DfCyFMAfzbGOOnVrCnOw915lnyl0n7IlW/3+9nKj7V/DKvPjDf\nYttJ71Bcl/jvijF+P4TwZjy7AXwrxvjVVWzsrkIJb4mtDTY1SUc1ArbHms1mmdQfjUY4Pz+fI3lR\n+M7hqMK11PQY4/cvj28A+CKA2jv3gOeNNpig0263cw0zNQ9fbXuC6jlvAGqzq+POE3QcV8WViR9C\n2AshHFye7wP4SQDfXNXG7ipU2ivxbVNN5uOzAMfm4qdIn8rMc9I7roLrqPovAfhiCCFePs9vxBi/\nsppt3V1YNd+20KbEtyW3lPgq7bVZZhHxlfx+A3AsiisTP8b4/wC8Y4V7uRfQevuUxN/d3c0kPu18\nK/GtJ96SX0N1TnrHVeChuBuAqvrsrWdt/JTEt6q+VfdT9r2H6RxXgRN/xShS9ZX41sZnq2zCkr5M\n1XeJ77gKnPgrxiLOvVR3HZ10k/Lqk/CpijsnvWNZOPFvAFXhPJX41rlHWLLrWGwtwPGyW8dV4MRf\nMVKqvnXuqY2fcu7ZuL1W6I1Go7myWx2F7XAsAif+ipHy6pfZ+JrAo6q+SnsSXzvvpBptOByLwol/\nA0jZ+EWZeyrxCSvxNXW3qN2Wq/qOZeDEXzGKVP0yr37KuWdV/clkkpP4qUYbDseicOKvGEWqflHK\nrlX1ASSJX2Xju8R3LANvvbViFIXzihJ4rKpvM/WsjZ/qqec2vmNZOPFvAFWqvq3OS6XsMlMv5dxL\n9dRzie9YBk78FaCqb3673c5CepT2WotvpX1RDJ8rVZ7rpHcsAyf+FVDVN585+jowg+RX0qdCeCrJ\nrWQvq8V34juWgRP/iijqm09nHUlvya8OPR2EmZLyqccXFxfJRppOfMcycOJfAVV981XqpyS+ttu2\nxC+T9jZ8Z0nv5HcsCif+klikb74OvCTxVeKrhqAOvSppz8e2Vt9tfMeycOJfEUV9863Etw6+7e3t\n3PNYG71M2vOxrd5zVd+xLJz4V0BV33yr6pPwJH9KWvNYJe1teq6T3nEVOPGvgLK++Up+q+bzBkAi\nEyTzItL+4uIitxcnveMqcOIvCTv1Vu35VquVLLvlz3QMNiW8JbcW4aTm3t8Vomtug4Y8i36n7Noy\nz58aJFI0XEQ/y7vyua4KTvwloaRXNZ7r8PAQ+/v7WYaeZucRmpWnE3EnkwkGg0E2LEOLce5SEY4N\nc9qQZyr/geeLPHfqeTUnwkY99HFqrFgdR4w58ZcEic8mmpTuzMo7PDzEwcHBHPHtF5NDMW2dvZ2D\nx0y9u/LFLIp4aFpy0Y1AbwKLPD+fU5ftZaDaUioHQisb78pnvAo48ZeEVt9R4jMHf29vL5P4rMBj\npp4dj2WJTylvia+x+7sCLVRKraIcCNtpuOi5i56XZpSmOGvLspTpRF9LXQhPOPGXhFX1WW67v7+P\ng4ODQlXfSnzbYOP8/ByDwSBT9e+yxC+bHZiS0irBq4hf9LzqP6GfxK5ms5nrW2iLouoEJ/6SUFVf\n22qR9FT16eRTG99m6VHiD4fDHPE5FFM77dwV4gPISXub08DrqqrrtSrip55TH89ms8x04g10NBrl\nnt8mTs1ms9yNuQ5w4l8BquqrxCfxrcRfVNVPSfy7puqnJL7mNNjpwKlpwWVoNpu5Ggh7nM1m2Y1T\nbzZF2ZL8XywaTbgvcOIviZTE39vby0l82vhW1Qfma+1Tqv5ddu4ByKUv2/JkEpGEt0fbZtyi2Wzm\nIimaHEXit9ttDAaDOU0rlfg0nU7RbDZzqn8d4MRfEilpps02NI5va+6txKfDSVXTopn3m0b6ojg6\nPxPbg4BLcxqU9FxV5LNh1BTxNbxql2oDw+Fw7v8C1CPO78S/Asqy9tReLbIpLfk1cUcTeDZ1YEZZ\nnH57ezvrK6hH7TVoya6PF5H4qZJnVfVt1yOS/fz8HOfn5+j3+5l21e/3c687Ho9rEed34l8RNl6d\nsldt6S2RGpihkn+TB2ZUxelJuIODA+zv72fRDh7b7fYc2ZchfqPRSDr1+DjGmO3BOvjoR+n1euh2\nu7m9AM/+LyGEWsT5nfhLwjqJUnn6lhT6+0Xts63E3+SeemVx+pSz8+joKDvf2dlJEl4/uzKo7yBl\nKsQYsbu7WxjS6/f76HQ6mZbA19OS5zrE+SuJH0L4NICfAvAwxvj2y2snAH4LwCsAXgPwvhhj5wb3\nuVGwpC9T91MZabaTrs3g2+QuulVxenV2Hh0d4fj4GMfHx9n57u7utYlflcBjk3f0vN/v55yMwPMU\n6vF4nHO83uc4/yIS/zMAfgXA5+TaRwD8fozx4yGEDwP4pctrtUFK4hclqaRIb218K/VtltkmffHK\n4vTMa6DEPz4+xoMHD3BycoKTkxPs7e0VqvmLePXtDdcuAMlSZh57vV7uBsMbLyMrJPx9j/NXEj/G\n+NUQwivm8nsB/Ojl+WcB/AFqRPyUnVsm8VPkv0rf/E340lXF6W148/j4GCcnJ3jxxRfxwgsvYH9/\nPxnKW4b4ZUU6QN6HYm31Xq+Xk/T87On0m0wmtYjzX9XGf0uM8SEAxBhfDyG8ZYV72mjYL56Vftax\np19OomhElpJ/k9tnl8XprY1P4r/wwgt485vfPEd8e75Irr49t0fb1UjPu90ugOc2PSV9v99Ht9vF\naDTK/c19jfOvyrm3Od/KNSLl6CsK4/H3bdssq/arU2kT22fzPdnSZB0Tpt58Lk1uSiXu8Nx+RqkF\nzNfb83FKI9D/x2w2y/bU6/Wy836/j4ODA0ynUwyHw7n06ipN5K7hqsR/GEJ4Kcb4MITwVgA/WOWm\nHJuLosxFxsyPjo6y8J2OC9Nkpqo8B3sTrJonoI+ZRJQq4Gm1WpmvhKm/Ozs7mXZCwjPrj2Snw6+O\nEj9cLuLLAH4GwMcAfBDAl1a7LcemQqW9Ve1JIBLfdiPSWH2K/AByZbW2Scl4PJ67AVh7noROZfbx\nJsH3sL29nTkiOZqMJgxr++kHWKSO4C5hkXDe5wG8G8CLIYS/APAqgF8G8DshhJ8D8F0A77vJTTo2\nB1XViUUS384JTJGeyTPT6TSrWkylMts4u65ms5nLGuSKMaLRaCQl/sHBQVYBSdMAyHv8a0f8GOMH\nCn70Eyvei+OOoKo6UQuVmKar2XXW/laPPCV+qlx5MBjMDQ21a2trK5cpyKQcZvyR+Nz/7u5uJenP\nz8/rR3yHQ6ESX1V9JuwsIvGLQnLAcxufpBsMBuh2u+j1euj1ehgOh7kBojZBZ2trC8fHxxiNRllx\nE29U7XY755xk3oGq/9Q6lPTcuxPfUVtYj35K4qdsfHXuWfU+leNAid/v99Hr9XB2doZOp4Pz8/O5\nRCd93G63s3AonX28SU0mE7Raz77yVPX5O3zcaDTmwnzM8rtPnn0nvmMpWBuf9rTa+HT0pSQ+iafP\np1DnnhbVdDodnJ6eotfrzTUo1fN2uz2n3rNoZzKZZBoGia7n/Dk1Dab3bm9vu6rvqAeKEmU0YYex\ne3rGGROnY61I2tsQnD4m4VhB1+12M2mfIr4l/+7ubjKZiL/XbDZzDj7gecUfHYRaQmwbqdwXOPEd\ncyhKhw0hzDXY0Hr7Kk8+kJ8KnFqDwQBPnz7F6elp7tjpdNDpdDAYDJK9C3hOiV20WMijcX8ttNJB\nKDYK4RLfcW+RqkPQuLt206HXXtuL7+3t5caFqX1MaU8bWqU1j/1+H6enp5mE1+PZ2RkGg0HSuUcH\nXxXx1UlXdGNLEf8+kR5w4jsSsFWHuijtqcor+SnxtdUVCUTyMCGGzjO7VLUn2XlO515ZOK+K+KzB\nV0muEt/u25ZX3xc48R05pKrvdKmqn5L4u7u7c11xrKrPOP35+XkWpuv1elmhzNnZWW7pteFwmKxr\n0ASeMuLT/ACQc/Tp+0vt3W18x71HWb29te9T5LeFN0WqPolPqa5EL1qj0aiw7HZR4vM90sbXOP94\nPM5FIPg+7pO0B5z4DoNF6u2tnW+deykfgar6JL6G6p4+fZo58VQLsGs0GpU2w2y1WqXEb7fb2XtT\n55427nRV31FLlNXbLyLxqxJ0rKpP4j969AidTicL5/X7/Wzx8WQySZbm8lhF/PF4nFXpKfFt225X\n9R21wiL19jo/QEN3JIxKYHusitMzZKer3+9nLbIvLi5K96+OPhs2tJV9qfdrl+3oS/9AUaOPuwIn\nviOHqnr71BhwHVNVFafv9Xp4/Pgxnjx5gtPTU5yenmZ2Pfvd29mB6+w5qDa/HZayt7eHyWQyVxJs\n23DfBTjxHTlU1dszJbdsGnBZnL7X6+HJkyd48uRJZtOT+L1eL6vAu80RYvQB2NJjEj/VIIRpwndF\n8jvxHTlU1dvbacA6lILEL4vTn52d5bLySHza8DYzb90txlXiq6nDtGStBGRBD/C8wacT33FnUVVv\nXzYNuCpOb+15lfgkvi21Zb38OqARjZSqr6nBti/fXYIT35FDVb19ysbXIpZF4vR6rrF71tun+uHf\ntqrPz4GFPvp+72JPPie+I4eqenva+DoGXCW+tq0uitOT5JqYQ41AE3R0rct+rnLuUdoDz9V7pgo7\n8R13FlX19kUSX238qjg9SW6P/X4/N60WWH+ozCYvaXoyiQ/kqwzvYpMOJ/41YPu925i1/dLqubaf\nSlWJ2Xr4m5Qm+txMUdUZ97bentJee+ipyqv98rSDTlGcnuE79tO7bdgiJZu9yE4+dzmzz4m/JFID\nMbRIxI62tmpqiuxlI7huQoVMZdXxXFXcVLMNbZ7JRha2QSY746o3P0X024jTO57BiX8FWOluJ+Fo\nAUlK8lsJr+RPNX+4CWlSpGWkRmJpWi473KQ8+TqHXomvGXibEKd3OPGvjGUkfkqa2WKYIml/UxJf\nbzh6AyjKzddcfN2n2vUxxpy0T7XHvu04veMZnPhXwHVn31nipaS9tR1XRf7Ua+tSe7ZI4tv98H1y\n7pyV+LT1NyFO73gGJ/6SSDn0Ug0hUjY+q8EWsfFT02ZWKflTr2/DWFbdp8QvKsCZzWalqj7Xbcfp\nHU78K6NI4hcNeLRIka7IsXfTqr6+bpGNz1j+7u7uXBEOVX1r51vSU+LfZpze8QxO/CtAJXjRDLeU\ntCeqJL7tCXdT0j71+kU2vkr88Xic2fZ8r+rVtza+vQHc9ZLW+wAn/pJQ9Z5SjpVn/NKzhRPtV1Vl\nbYKI7QHPv2NjyFV3erW56FprvrW1VVhvr+2yp9NpjviW9LrUg8/3tg7YMKV1YvLnegOyGpz+j61P\nougGf1fgxF8SRYkqmsFmq7rUecXMOE2HZY23voadH7eqvm/6+kponp+cnOD4+HguO09fn8RgP3sr\n2c/PzzOy34YNX+S/UI1KyW+JzvfFm5Y6KO9LHoIT/wpQUg6Hw1zveM16293dzX35dYAjBzbqTYFf\nSD4/R0itcoSTpuTa9lm7u7t48OBBNviSPfI1Zq83PqvtaIJOKk6/LhSZMintyTppSXz9/+p7uy95\nCE78JZFSb/XLxN7zzHTjl8P2eKPEV9JvbW0BQE7S29TY64Kxekp5bbKxv7+Pk5OTjPhaiGOJrxKf\nUpHEoIqvYbt11tPzaMOl1nFKaLGNlfgpP8V9yEOoJH4I4dMAfgrAwxjj2y+vvQrg5wH84PLXPhpj\n/E83tssNAr/4quZqB9lms5mRnuqgSnwlnpKe1wDkKttSqvZ1oIMjtOSWlXdW4vPGY1V9JYj14Ksq\nfJuqflGClKr6RRKfxNab2n3KQ1hE4n8GwK8A+Jy5/okY4ydWv6XNhpX4dlBEq9XKyKRffn7x1RwA\nnpOeWkIIISN9EfGuA83OU+IfHx/jwYMHOD4+ztn46tRTh55+BlbVp8S/bVW/LEHKJiAV2fgapbhP\neQiVxI8xfjWE8EriR3enFGmFUMebkp7XWq0WDg4Okg4gIO9c09JPfoEAZNVsSrybcO7ZJhsPHjyo\nlPjq9S5S9a1zb52qvr7PsgKoKhvfqvpq49+HPITr2PgfCiH8YwD/A8C/jDF2VrSnjYZKfBKBJCBB\njo6OMslXpOo3m8/GNW9vb8/F+s/OzuaIt0ob3zogDw8PcXx8nPPoFxFfnXtlqv5tOb/KCqCKqh7L\niK/kp6p/H/IQrkr8XwXwr2KMMYTwrwF8AsA/Wd22NheauGOdXRcXF9jZ2cmpuhrP5w2AXziWtQLP\nnVLqYdeJNRp6s2nDuq8q2Di+pubq65ZNjC0jCt9vKodhldBwnJ7ruK/Ush2BeUPmzYyamuYg3EYe\nwk3jSsSPMb4hDz8F4HdXs527AVUP+aWjmq7hIOsk4ginVEmsnpe1vtKSX00iWadUTfUkSBUq3YQa\nnJLmunR4p0705Q2UN04g3yST19gGzDYHuWvNNKuwKPEDxKYPIbw1xvj65cOfBvDNVW9sk2GJr9dt\nSChFfptYQl+BfqGL2lurmq2vpfta12eQynRbR0abTTFWNZ4dgC35qTFtbW0VpllPp9Nct1/10dw1\n510VFgnnfR7AuwG8GEL4CwCvAvixEMI7AMwAvAbgF25wjxsJJb9e0+IVJb0Sn1/WGGOm7qvaWjTQ\n4ujoCLPZLPec2gNune/7qtWJ18UiKcc6+Sc16ms8Hmd7s5oZiZ8KSdaK+DHGDyQuf+YG9nInYL/0\n9pqSPiXt+aVrtVrZF4lfZp6nRlhR4k+n08zhZEOJq67iq/oMtBzXkr6sH8F1wM/Klg9zFUl8VfVJ\neB41ZKcTfWpNfMc8+GVvNBqZum+93WV2vqqNauNrEU+ZjT8YDOaSaVYV7lvmM7Dqfsq+vwkVOVXk\nlOoUlHKQbm1tYTKZZBEZEp+NQs7OznItwmqr6jvSsDa+OvnKJD7nu/NvaOPrtaIRVufn57nXpIbB\nFN/bkPhVqv46JL5tCKrS3tr59AUAyP5XmilpnXsu8R0ZNIRmyWZtfGvn0ya3MWaSo8zGp7bA19bM\nuXVJ/JSNbzPfbkrNB6pHXPGzSjn3dnZ2ss8XQE7iDwaDbLiHde7dxZTcKjjxrwn7pa7K/tra2sqV\n6FJ62bJdHWjBYh+t8LPRBJJPbyL8mS7eVFJz4PWY6varKKtZT3UiWvYGUBanT3UG4tLZfmwFzpwE\n+kTULGMCEiU+7ftUabFLfEchNLPPfqm63S4ajUYud53E18y+snp9hvrskAfaucPhsDDGP51Oc/aw\n7ayjwzA1R99OirFxe2vSpBJ4FpWYVXH6nZ2dLLMwtY6OjrIxX5p1qPsuy8VnJd5t9hNYB5z4Kwal\niU1npeOIDkEAOVu13W7nUnqL6vUt4ZX0JL6tHNNzdXZp91xdmiWovQa0mq2scUVR96FFURan15wG\nHetlF4m/vb09F/0oSsktKjJyVd9RCZX4WrXW7/cz5xKQL8Vtt9uFEt9K+iLCk8zn5+c56Wv9C6om\nW/JT4mt4zKa5AvMSX30NNk2ZN4irphRb7YbEV+nOkuLUNF+V+CkNJVVvX6Tmu8R3FILEV4k/GAxy\nnWws6W3Zblm9vi5L+p2dnUxi2cXXtQ00rcTf3d2dy21XVd8m7tiONbYq76qqflGc3vYPODo6wvHx\ncXZkAxEurWy04daiDjtFZcVOfEcpUo4j25NPq+O0bLeqXl/TT1P2uu1hb+fX299PkV/V62WKdMps\n/GUkZlmcnsM7SXz2D2Avgf39/dzfWBt/EVWf1+17uE9w4q8YKYmv5KvqyVdVr09PdUpy7+7uZtNp\n+aXXct7ZbJbTDoqceymnmtr4ZY0rrludVxWnp5pPW177CDx48CC7celaRtVXU+k2WoetC078FSPl\n1VeJqz359vb2suovfrGq6vVZ4ZeS2lzavMM6thaR+EWhNH2PfL6UBLUhvWUkflWcvkzin5ycZF2M\nbDtt/Qyq6u2tU9QlvqMSNkZsbeRWq5WTtOpQOj8/z+Xs25JdALksP/6emg+qptsKQNr4dIDZeDdv\nGMu+Xwvb+kr7DthUZXvOfHvuTWP0e3t7OZuezjx9P4yO2HRiPu71ehnB9XOnQ280GiXLi9255yiF\nqsKULJb4aqvbhhfs45cib7PZnDMJWGZqJVLKrKDH/k1velPWTZdk2draWjjzT6Uy1Wm1q6fTaZaU\nZFN56cAsWkp8S/69vb2cpGfYjjcuJulo+FKPk8kE3W4XT58+xenpaS49Vx15N5lyvClw4q8YVg22\nDTmbzWaS9Ep8m0GnIS01CdQBBuTr+a0DkaRptVo4OTnJEX9nZ2fh1l4p0muYcWdnJ5k4xCP3XRSn\n1xTllNSnqs9liU8/xMXFRead1446Z2dnePLkCU5PT7OCHEp77Q940z0FbhtO/BuASny1L5lSm5L0\nJO14PM6F67a3tzGdTjNy84utDTuA+aQXJT1bfTOsRzXZZrhpdl4ZLPnVFtda99QKIcylB2vokJ77\nIvLbZCM1VVjqTOKPRqPceG4mUT19+hSdTidXe08vflEvASBt1txVOPFXDKvq8xrtRqqzKXs8hIDJ\nZJI533iu8Xwt37XJQCSgkt4m8TQajTlVmqRZVOKnkmyU+LZWX88ZsUjF6KkxFNn3lO5adKN19ltb\nW5nE1qq7breLs7OzbHU6HXQ6nTmJT7OgaP/3CU78FUNVfT6mzcsbQYr0xMXFRS62b4t3tIEH/1bt\nUObjp2xc3nhSxFlG4qvTziba7OzszDXqsEVCGoq0OQnUUIqkvpLcHtUU0jp7SvmnT5/i7Owsq8Jj\nUY6q+pqlZ9/DfYIT/wagpbPT6RSNRiPrwx9jzJJKUokx9AuoN5kk07xz9frr0jx661xLqdq6lpH4\n1qzQLES+l1SVoM7ss40wbU5BivxMe04lGTWbTUwmk5yqT4l/enqKR48e5Rpt6BBMVfVT+75vcOKv\nGKoepkJWs9ksi+urhLXZcDaFl51jdCCE9fxT9S8rywXyTkB7vghSNj7Ni1T5qj5O9cVLrSLib29v\nJ0OdXGx0ojY+Jf7jx49xenpaOM6bWlHR3u8TnPg3gDJnUKPRyCbspvLgU44lJS2n9ajXv9V69m+0\nZE7deK7brMNGEzh4kyPDNGafAvPty0ivjTJtNx2GCdW00cfMXKQ6z0XbvtvtZn4PW1twH235Ijjx\n14xUZp/NsNPUUjuXrszGtfXzVhtINdRYFswsVMJrrcFwOCz9e81jsBWCtjeeajD8vGyzD9rlPO92\nu3jjjTfyIg8UAAAM8ElEQVTwxhtvZBKe6n2q3Pa+JuhUwYm/ZqRy+atIr9NaUw4xTZ4h+W24DMC1\nSQ/MFw0dHBxkPo1ms4nRaFT69/zbIgeflgPbqjqq8VoXYI/dbhdPnjzB48eP5+L12ic/VTZcJzjx\n1wyb0ltEetvEg2qwdYjZpQSy8X+aBNcBn4dtwbSqkH6IMtBJWbRsQo8m5PCmaYtqdHW7XXQ6HZye\nnuL09DQXr9fJOKmy4TqR34m/ZqjEtwU0Wto6HA6zMlv1ejOzrcwOTsX/NQx4HajEV0nPUJ51jqX+\nPtVgg+fqnyBIyul0ivF4nNnxuopse22gydRclfT3sdZ+ETjx1wy18YF8xZhWi7Xb7ayBh41zW0/3\ncDjE/v5+zv5V0jPRZhVfbrXx+fzaH5AaQBGK2mrxnKq9TaLhcTgcZmPE6bDToxbhaDGOqvraQaiO\n0h5w4q8d/AKzgSbj/EoAm9mm51qTfn5+joODgzmnV6pH36q+3GoyUHqr6lxVvlrURFOzEtVZRxWf\n59q/kEk5unq9Xi5EZ8/Ve69OvbqR34m/ZmgIikUrNi6dKtDhOefZ2xZRtlOMLdRZlQOLz0v1PkWi\nMqRKcfX9U53XZCTb0YjEPz09xePHj/Ho0SM8evQIjx8/Rq/Xm5tkpIsaiQ251on0gBP/VlBGkBAC\nLi4uClXhsiIYW1KaWu12O5m4Y+v+uZfU/spi9UUZe6liFz3yhkXvvZo9ek7CqwOP6+nTp+j3+7l6\nerXnqT04nPgbCVU/VRICyHn8mQGokQL6Cuj17na7ODg4QKfTmct116UhNKC4UcYie0+V41p72iYo\naY69quhWVWc9PUN13W4365NXp3r668KJv4GwxNfrjJVrgQ7JxYgASd/r9eay5FIz5XjOXPpU5x9g\nsTwA9WHYnv6pslfraGNxDWfX6aJ9r9V1JL7OE6hDPf114cTfUCj59Rrz9W3832b5FcX52ZBDm1lQ\nm9DsPt5UNId/UeJrZmKq374tHtI1Ho+zmLw2wuTjolCeVtfVoZ7+uqgkfgjhZQCfA/ASgBmAT8UY\n/00I4QTAbwF4BcBrAN4XY+zc4F5rAav+2mts1gHk4/+U9LaJhz3f39/POtKyvxzwvAmoahK2EnDR\n/dsbkarq2nM/lXLL96Erlaij16zEtyHAunnsF8EiEv8CwL+IMX49hHAA4I9DCF8B8LMAfj/G+PEQ\nwocB/BKAj9zgXmsDfmEbjUam7qfCTza1NzX80h4PDg4ym5i5BCT9/v7+XLyfcXXt/LPI3tULb3vW\n2z54dsgFSa0NMXlUZ58dP27LapeJNtQNlcSPMb4O4PXL814I4VsAXgbwXgA/evlrnwXwB3DirwzW\nxteyXqrKau+nCnI09ZWPDw8Pc6RnMg4n8jIxh9A6/0X3nZokRJX8/Pw8+5k9ksiaeGOXTbm1K6Xe\nO+nnsZSNH0L4YQDvAPCHAF6KMT4Ent0cQghvWfnuagr90loJSw+/jf/zZ1Xr6OgoU++1yu7o6Giu\nHl1t/WX2bqsPtd+dzqVT25/nWpCUOqZae5VJdid9GgsT/1LN/wKAX7yU/PYT9U/4BrDqL/JsNssN\nnbRdaJgnoDX+1DSqau11j+rAU1XeDvEsqrSzI6x5Q2CHHftZOMGXw0LEDyG08Iz0vx5j/NLl5Ych\nhJdijA9DCG8F8IOb2qRjdVBS6kQZEkvj+upQXEbyazhQ+/MxEUkjFbabD/0aNimJNxCaHXVPub0u\nFpX4vwbgz2KMn5RrXwbwMwA+BuCDAL6U+DvHBkJLgLUakMTnNBogT8xFiGXNDkt6rRK0P2cOQYr4\n1Bpo6tg8AL4vJ/9iWCSc9y4A/wjAN0IIf4pnKv1H8Yzwvx1C+DkA3wXwvpvcqGM1sBEB63m3QztI\nzKriG0WVxOfv8GeMOlCiK6Gt8w7AXOyf78uxOBbx6v83AEXG3U+sdjuOm8Yiqj7wPCefkngZaapZ\nfza6wP79qt4rianqq6RXH0GUSj1qFzbRyVENz9yrIapUfRKTkti2nF4EKYlPkqsWYdX2RqORuxHY\n6jqtXwDyOQ/LhB3rDid+zbCIqm8n8yxSZ6+wqr6V+GXOORJfpbxGBDS3QbWXVfQTrBOc+DXDIqq+\nJf0y7alS/QVU4msasu6JR512a0OA7CvA32XOQLPZzKn+jmo48WuGosw6HTNtbXB2BdLGnTymMgvV\nLEjdCLgP3ROPqh3YZbv1LFND4MjDiV8zKPFJesbP9edFJa07OzvJRh5cqWYg9jmKavFjjLk8fG2F\nrbX2tkmmV98tDyd+zUCvOFX8wWAwV+JrC10U4/E4Odeejy0xi3LnU7X4s9mskPRayVd1Y3FUw4lf\nM1iJb1t8l02WYVuw1Gz72WyWRQCqmmCkUnp5npL4lvTeZOP6cOLXDCrxlfQsiWXIzIL29MXFRW4A\nht4oaONXNcPQcKKtyV9E4nuHnevDiV8zqEefj7VPnxbBEOqYm06n2cQenSfP36NGYTWHKvJrPb4W\n8Cwq8Z34y8GJXzOQmOPxOCOe2uhVxNd+eirptftuUS49X98S39bj62BLK/GtOeL5+VeDE79moKrP\nDDr2sOfi0MuiJBz+vRJOp/WwiCYl8Ql16inxrbRPTcctk/h+A1gcTvwaQqVuCozpc+kAS0r8osWb\nh5XeRdfsY+2wa6V/mY3vWA5OfEcOVXH+yWSCwWCQnG2/u7ubNQO1TTeqHtuBodpZV/v0uY2/Gjjx\nHTlUxflHo9HcXHs9DyHMNdBMraKfK/mtRqB99dyrfz048R05VMX5h8Nh6Xx7AMm22WXLDt9IaQVW\n1S8KFToWgxPfkUNVnF/Hb6Xm29O5Z8NvqZX6HTt5x54zfKhefbfzl4cT35FDVZyfzr7UQE91AKZC\nevZa6nHZzcG2z7Z5/o7F4cR35FAV509Vyenic6Rq7YvO7bWylUoGctIvj3DTH1qiDbdjw2GHZhYt\n/V2eE9buLirDLTra8yI73klfjhhjsm7Zie9w3GMUEb+x7o04HI7bhxPf4aghnPgORw3hxHc4aggn\nvsNRQzjxHY4awonvcNQQTnyHo4Zw4jscNYQT3+GoISqJH0J4OYTwX0II/zuE8I0Qwj+7vP5qCOF7\nIYQ/uVzvufntOhyOVaAyVz+E8FYAb40xfj2EcADgjwG8F8A/BNCNMX6i4u89V9/huCUU5epXluXG\nGF8H8PrleS+E8C0Ab7v8sU8sdDjuIJay8UMIPwzgHQD+++WlD4UQvh5C+HchhOMV783hcNwQFib+\npZr/BQC/GGPsAfhVAH8rxvgOPNMISlV+h8OxOVioHj+E0ALwHwD8xxjjJxM/fwXA78YY3574mdv4\nDsct4br1+L8G4M+U9JdOP+KnAXzz6ttzOBzrxCJe/XcB+K8AvgEgXq6PAvgAntn7MwCvAfiFGOPD\nxN+7xHc4bgneesvhqCG89ZbD4cjgxHc4aggnvsNRQzjxHY4awonvcNQQTnyHo4Zw4jscNYQT3+Go\nIZz4DkcN4cR3OGoIJ77DUUM48R2OGsKJ73DUEE58h6OGcOI7HDWEE9/hqCGc+A5HDXHjHXgcDsfm\nwSW+w1FDOPEdjhpibcQPIbwnhPDtEMJ3QggfXtfrLooQwmshhP8ZQvjTEMIfbcB+Ph1CeBhC+F9y\n7SSE8JUQwv8JIfzn25xeVLC/jRmkmhj2+s8vr2/EZ3jbw2jXYuOHEBoAvgPgxwH8NYCvAXh/jPHb\nN/7iCyKE8H8B/J0Y49Pb3gsAhBD+HoAegM9xUEkI4WMAHscYP3558zyJMX5kg/b3KhYYpLoOlAx7\n/VlswGd43WG018W6JP47Afx5jPG7McYJgN/Esze5SQjYINMnxvhVAPYm9F4An708/yyAf7DWTQkK\n9gdsyCDVGOPrMcavX573AHwLwMvYkM+wYH9rG0a7ri/62wD8pTz+Hp6/yU1BBPB7IYSvhRB+/rY3\nU4C3cGjJ5RTjt9zyflLYuEGqMuz1DwG8tGmf4W0Mo90YCbcBeFeM8W8D+PsA/umlKrvp2LRY7MYN\nUk0Me7Wf2a1+hrc1jHZdxP8rAD8kj1++vLYxiDF+//L4BoAv4pl5sml4GEJ4CchsxB/c8n5yiDG+\nEZ87jT4F4O/e5n4uh71+AcCvxxi/dHl5Yz7D1P7W9Rmui/hfA/AjIYRXQgjbAN4P4Mtreu1KhBD2\nLu+8CCHsA/hJbMYQ0IC8vfdlAD9zef5BAF+yf7Bm5Pa3gYNU54a9YrM+w1sbRru2zL3LsMQn8exm\n8+kY4y+v5YUXQAjhb+KZlI8AWgB+47b3F0L4PIB3A3gRwEMArwL49wB+B8DfAPBdAO+LMZ5u0P5+\nDAsMUl3T/oqGvf4RgN/GLX+G1x1Ge+3X95Rdh6N+cOeew1FDOPEdjhrCie9w1BBOfIejhnDiOxw1\nhBPf4aghnPgORw3hxHc4aoj/DxY6cETvLFs/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f60c893feb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(X_test[22][0], cmap=plt.get_cmap('gray'))\n",
    "plt.show\n",
    "print(\"The digit looks like a 4 is actually a {}\".format(numpy.argmax(y_test[22])))\n",
    "y_res = model.predict(X_test)\n",
    "print(\"Our CNN model recognizes it as {} with {}% confidence\".format(numpy.argmax(y_res[22]),numpy.max(y_res[22])*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary<a class=\"anchor\" id=\"seventh-bullet\"></a>\n",
    "\n",
    "In this tutorial, we have discovered the MNIST handwritten digit recognition problem and seen deep learning models built in Python using the Keras library. Both the NN model and CNN model achieved an excellent result (an accuracy > 95%).\n",
    "\n",
    "Working through this tutorial, I hope you can get some ideas on:\n",
    "\n",
    "* How to use Keras to create a simple Neural Network for image recognition problems.\n",
    "* How to use Keras to develop and evaluate a Convolutional Neural Network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "<sup>[1]</sup>  Jason Brownlee (June 27, 2016) [Handwritten Digit Recognition using Convolutional Neural Networks in Python with Keras](machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.5 (Experimental) with Spark 2.0",
   "language": "python",
   "name": "python3-spark20"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
 }