functional-micrograd.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPfBlYytwdEV/FZlucDwRt/",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/cghawthorne/f5892a148c08897155af27db07e556b6/functional-micrograd.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Functional MicroGrad\n",
        "\n",
        "A tiny, scalar-only autograd engine using only Python built-in functions.\n",
        "\n",
        "I've really enjoyed Karpathy's [Neural Networks: Zero to Hero](https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ) series, especially [building micrograd](https://www.youtube.com/watch?v=VMj-3S1tku0). I've used JAX more than PyTorch, so just for fun I wanted to try making a version with a more \"functional\" flavor, where `Value` objects cannot be modified after instantiation. So, no worrying about zeroing out gradients or reusing objects.\n"
      ],
      "metadata": {
        "id": "vQbFCAQQK4-o"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Autograd Engine"
      ],
      "metadata": {
        "id": "g4rt8o78nkJV"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "XYGctNE0sbrt"
      },
      "outputs": [],
      "source": [
        "import dataclasses\n",
        "import math\n",
        "\n",
        "from typing import Optional, Sequence"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "@dataclasses.dataclass(eq=False, frozen=True, repr=False)\n",
        "class Value:\n",
        "  data: float\n",
        "  parents: Optional['Value'] = None\n",
        "  backwards: Optional[Sequence[float]] = None\n",
        "\n",
        "  def __add__(self, other):\n",
        "    other = other if isinstance(other, Value) else Value(other)\n",
        "    return Value(self.data + other.data, parents=(self, other), backwards=(1, 1))\n",
        "\n",
        "  def __mul__(self, other):\n",
        "    other = other if isinstance(other, Value) else Value(other)\n",
        "    return Value(self.data * other.data, parents=(self, other),\n",
        "                 backwards=(other.data, self.data))\n",
        "\n",
        "  def __pow__(self, other):\n",
        "    other = other if isinstance(other, Value) else Value(other)\n",
        "    return Value(\n",
        "        self.data**other.data, parents=(self, other),\n",
        "        backwards=(other.data*self.data**(other.data-1),\n",
        "                   math.nan if self.data <= 0\n",
        "                   else math.log(self.data)*(self.data**other.data)))\n",
        "\n",
        "  def relu(self):\n",
        "    return Value(max(0, self.data), parents=(self,),\n",
        "                 backwards=(1 if self.data > 0 else 0,))\n",
        "\n",
        "  def __neg__(self): # -self\n",
        "      return self * Value(-1)\n",
        "\n",
        "  def __radd__(self, other): # other + self\n",
        "      return self + other\n",
        "\n",
        "  def __sub__(self, other): # self - other\n",
        "      return self + (-other)\n",
        "\n",
        "  def __rsub__(self, other): # other - self\n",
        "      return other + (-self)\n",
        "\n",
        "  def __rmul__(self, other): # other * self\n",
        "      return self * other\n",
        "\n",
        "  def __truediv__(self, other): # self / other\n",
        "      other = other if isinstance(other, Value) else Value(other)\n",
        "      return self * other**Value(-1)\n",
        "\n",
        "  def __rtruediv__(self, other): # other / self\n",
        "      other = other if isinstance(other, Value) else Value(other)\n",
        "      return other * self**Value(-1)\n",
        "\n",
        "  def _grad(self, cur_grad=1):\n",
        "    if self.parents is None:\n",
        "      # We're at the base case, return current node with current gradient.\n",
        "      yield self, cur_grad\n",
        "    else:\n",
        "      # We need to go deeper.\n",
        "      for parent, backwards in zip(self.parents, self.backwards):\n",
        "        yield from parent._grad(cur_grad * backwards)\n",
        "\n",
        "  def grad(self, vals):\n",
        "    # Accumulate gradients for the Value objects passed in.\n",
        "    grads = {val: 0 for val in vals}\n",
        "    for val, grad in self._grad():\n",
        "      if val in grads:\n",
        "        grads[val] += grad\n",
        "    return [grads.get(val, 0) for val in vals]\n",
        "\n",
        "@dataclasses.dataclass\n",
        "class ValueHolder:\n",
        "  value: Value"
      ],
      "metadata": {
        "id": "ewBFFZZNwWYk"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Simple Test\n",
        "Do calculations from https://github.com/karpathy/micrograd in both this engine and JAX."
      ],
      "metadata": {
        "id": "7ftyciG5LRFW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "a = Value(-4.0)\n",
        "b = Value(2.0)\n",
        "c = a + b\n",
        "d = a * b + b**3\n",
        "c += c + 1\n",
        "c += 1 + c + (-a)\n",
        "d += d * 2 + (b + a).relu()\n",
        "d += 3 * d + (b - a).relu()\n",
        "e = c - d\n",
        "f = e**2\n",
        "g = f / 2.0\n",
        "g += 10.0 / f\n",
        "\n",
        "print(g.data)\n",
        "print(g.grad([a, b]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "McZspx2J1rA1",
        "outputId": "063dae19-1b2b-4ee0-c049-e12dd4762078"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24.70408163265306\n",
            "[138.8338192419825, 645.5772594752187]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import jax.numpy as jnp\n",
        "import jax\n",
        "\n",
        "def operations(inp):\n",
        "  a = inp[0]\n",
        "  b = inp[1]\n",
        "  c = a + b\n",
        "  d = a * b + b**3\n",
        "  c += c + 1\n",
        "  c += 1 + c + (-a)\n",
        "  d += d * 2 + jax.nn.relu(b + a)\n",
        "  d += 3 * d + jax.nn.relu(b - a)\n",
        "  e = c - d\n",
        "  f = e**2\n",
        "  g = f / 2.0\n",
        "  g += 10.0 / f\n",
        "  return g\n",
        "\n",
        "jax.value_and_grad(operations, reduce_axes=())(jnp.array([-4, 2], jnp.float32))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZPiXitn26MHl",
        "outputId": "52b0589a-a586-4562-8c05-cf0c8d59c005"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(DeviceArray(24.704082, dtype=float32),\n",
              " DeviceArray([138.83382, 645.5773 ], dtype=float32))"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Neural Network Library\n",
        "\n",
        "Based on [MicroGrad's nn.py](https://github.com/karpathy/micrograd/blob/master/micrograd/nn.py), but uses `ValueHolder` to store `Value`. `Value` objects cannot be updated after instantiation, but `ValueHolder` objects can be updated with new `Value` objects. This approach was easier than passing around a Flax-style parameter tree.\n"
      ],
      "metadata": {
        "id": "A2QnyAhoLi-N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "metadata": {
        "id": "CTLMDVKba5tG"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "np.random.seed(1337)\n",
        "random.seed(1337)"
      ],
      "metadata": {
        "id": "bfkgY7aEa709"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class Module:\n",
        "\n",
        "    def parameters(self):\n",
        "        return []\n",
        "\n",
        "class Neuron(Module):\n",
        "\n",
        "    def __init__(self, nin, nonlin=True):\n",
        "        self.w = [ValueHolder(Value(random.uniform(-1,1))) for _ in range(nin)]\n",
        "        self.b = ValueHolder(Value(0))\n",
        "        self.nonlin = nonlin\n",
        "\n",
        "    def __call__(self, x):\n",
        "        act = sum((wi.value*xi for wi,xi in zip(self.w, x)), self.b.value)\n",
        "        return act.relu() if self.nonlin else act\n",
        "\n",
        "    def parameters(self):\n",
        "        return self.w + [self.b]\n",
        "\n",
        "    def __repr__(self):\n",
        "        return f\"{'ReLU' if self.nonlin else 'Linear'}Neuron({len(self.w)})\"\n",
        "\n",
        "class Layer(Module):\n",
        "\n",
        "    def __init__(self, nin, nout, **kwargs):\n",
        "        self.neurons = [Neuron(nin, **kwargs) for _ in range(nout)]\n",
        "\n",
        "    def __call__(self, x):\n",
        "        out = [n(x) for n in self.neurons]\n",
        "        return out[0] if len(out) == 1 else out\n",
        "\n",
        "    def parameters(self):\n",
        "        return [p for n in self.neurons for p in n.parameters()]\n",
        "\n",
        "    def __repr__(self):\n",
        "        return f\"Layer of [{', '.join(str(n) for n in self.neurons)}]\"\n",
        "\n",
        "class MLP(Module):\n",
        "\n",
        "    def __init__(self, nin, nouts):\n",
        "        sz = [nin] + nouts\n",
        "        self.layers = [Layer(sz[i], sz[i+1], nonlin=i!=len(nouts)-1) for i in range(len(nouts))]\n",
        "\n",
        "    def __call__(self, x):\n",
        "        for layer in self.layers:\n",
        "            x = layer(x)\n",
        "        return x\n",
        "\n",
        "    def parameters(self):\n",
        "        return [p for layer in self.layers for p in layer.parameters()]\n",
        "\n",
        "    def __repr__(self):\n",
        "        return f\"MLP of [{', '.join(str(layer) for layer in self.layers)}]\""
      ],
      "metadata": {
        "id": "HGmtuYAzZLoX"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Train a Network on Moons Dataset\n",
        "\n",
        "Based on [MicroGrad Demo](https://github.com/karpathy/micrograd/blob/master/demo.ipynb), but modified to use `ValueHolder` for updates."
      ],
      "metadata": {
        "id": "Av85bo4UL9co"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# make up a dataset\n",
        "\n",
        "from sklearn.datasets import make_moons, make_blobs\n",
        "X, y = make_moons(n_samples=100, noise=0.1)\n",
        "\n",
        "y = y*2 - 1 # make y be -1 or 1\n",
        "# visualize in 2D\n",
        "plt.figure(figsize=(5,5))\n",
        "plt.scatter(X[:,0], X[:,1], c=y, s=20, cmap='jet')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 337
        },
        "id": "Kj7QBky7a-Uv",
        "outputId": "52b053c5-4706-409b-bcc3-d15f70eaff06"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.collections.PathCollection at 0x7f053a347c10>"
            ]
          },
          "metadata": {},
          "execution_count": 8
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 360x360 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAEvCAYAAADM0uPSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3iUVdrA4d+ZlmRSIEBCh1CVJi1SVJpIERGwIbgqIoh11bVg2f3ctaMua1dEBVFRQVFBBSkqiiJIUXpHqgESAmkzybTz/ZEYM6Qnk7yZzHNfVy7y9icDPDnvqUprjRBCiJKZjA5ACCGCgSRLIYQoA0mWQghRBpIshRCiDCRZCiFEGUiyFEKIMrAYHUBFNGjQQCckJBgdhhCiltmwYUOK1jquqGNBmSwTEhJYv3690WEIIWoZpdTB4o7Ja7gQQpSBJEshhCgDSZZCCFEGkiyFEKIMJFkKIUQZSLIUQogykGQphBBlEJT9LEVgbNlynM2bj9OiRR2aNYshPj6SyEib0WEJUSNJsgxRr7++jvvuWwaAw+HBYlFYLGbefHMk117b1eDohKh55DU8BKWn5/CPfyzF4fDgcHgA8Hg02dkepkz5kv37TxkcoRA1jyTLEJScnIXFUvRfvdVqZtu2E9UckRA1nyTLENS8eR3Cw4uugfF4vLRqFVvNEQlR80myDEE2m5lly64jPj4Ss1kBYLdbiYiwcN9959G5c7zBEQpR80gDT4jq0aMxx47dS1paDqdPZ7NjRzItW9alY8ciZ6cSIuRJyTKEKaWoWzechIS6DByYwMaNSbz22jp27z5pdGhC1DhSsgwRSUkZjBkzjw0b/iA2NoI5c8YwYkQ7ALKyXCQmvsnhw2n4fBqlFF9+OZ5Bg1qV6xkbNyaxadMxWreOZcCAhCr4KYQwjpQsQ8SIER+wYcMfeL2alBQHV101P78E+dZbGzlw4DRZWW6cTg8Oh5spU74s1/1ffHEt/frN5o47ljBixAfcfvtXVfFjCGEYSZYhIDvbw+bNx/F6df4+pRQ//XQIgOPHs8jO9vhdc/Kko8z3T0vLZurU5Tgc7vyvmTM3smnTscD8AELUAJIsQ0BYmBmbzey3TylFgwZ2AAYPboXdbvU7/8ILy/4KfvKkE6/X57fP4/Hx/vubKxE1/P77KT75ZDs//ngIrXXpFwhRhSRZhgClFK+9NgK73UpYmJnISCuJiU3y6ywHD27Nf/87lMhIK2azYtCgBGbNGl3m+zdvHoPPVziZpaY6KxzzV1/tpnPn15k0aSHDh7/Ptdd+KglTGEoaeELExInd6dw5ntWrD9OoURRXXNERs/mv35W33prIrbcmonVuA095WK1munRpyObNx/P3hYWZ6dmzSYVi1VozfvwCHA53/r5Fi3azYsV+hgxpU6F7ClFZkixDyLnnNuXcc5uWeE55E+WfPv74Ks4/fxY5OR68Xh/nndecm27qUaF75eR4ycpy++3TWnP4cHqF7idEIEiyFAHRvn199u27k40bk4iMtNKzZxNMJsXp09ksXrwHrTXDh7elfn17qfcKD7fQqlVd9u8/xZ9v3lpDz56Nq/inEKJ4kixFwMTEhDFwYEL+dlJSBj16zCQz0wVowsOtrFt3EwkJdUu91+LFf2PIkHc5fjwLgJdfvpiuXRtVUeRClC4gDTxKqVlKqRNKqa3FHFdKqZeUUnuVUpuVUj0KHJuglNqT9zUhEPGImuGf//yWlJQsMjNdZGa6OXXKyb33LivTte3b1+fAgbs5fPgfpKc/xI03dq/iaIUoWaBaw98Bhpdw/GKgXd7XFOB1AKVUPeDfQG+gF/BvpZRMeVNLHDqUhsfzVwu216s5fDityHMzMnIKtagrpYiLiyzU7UkIIwQkWWqtfwBSSzhlNPCuzrUGqKuUagwMA5ZrrVO11qeA5ZScdEUQGTasrV//zYgIC8OG+bdm79yZQsuWL1Cv3rNERz/NZ5/tqO4whSiT6upn2RQ4XGD7SN6+4vaLWuCee/pw7bVdsFhMmM2KK67owCOPDMg/rrVmyJB3OXw4DY/Hh8Ph5tprP2XfvpJ+7wphjKBp4FFKTSH3FZ4WLVoYHI0oC7PZxBtvXMqrr14CUGh29pMnnSQnOyjY19xiMbNxYxJt2tSrzlCFKFV1lSyPAs0LbDfL21fc/kK01jO11ola68S4OJlzMZhYLKYil7GoUyes0D6v10eTJtHVEZYQ5VJdyXIRcH1eq3gfIE1rnQQsBYYqpWLzGnaG5u0TIcBqNTNz5kgiIixERdmIjLRyxRUdOe+85qVfLEQ1C8hruFLqQ2Ag0EApdYTcFm4rgNZ6BrAYGAHsBRzAxLxjqUqpx4F1ebd6TGstFVYl2Lr1BCkpDrp2bUhsbITR4VTa9dd3IzGxKRs3JtGsWQwDBrQs8yii3btPsmHDHzRtGkO/fi0qPPpIiLJQwTg5QWJiol6/fr3RYVQrrTU33riQ+fO3Y7XmvhAsX35dqcMXa6v587cxceLnmM0mfD7N5Zd3YM6cMZIwRaUopTZorROLOhY0DTyh7quv9vDxx9v9Jpe46qqPOXDgbgOjMobX6+OGGz7H6fxrDs5PP91BixYxvP/+FlwuLxMmdOWJJy70myxEiMqQZBkk9uw5idvtP2fkkSOhObFEZqYLj8f/s/D5NM899zMulxeAl176hbAwC//5z0ADIhS1kfzaDRLnnNMw//UbQClo27b83Wuyslw8+uhKxo9fwCuv/FLkPJQ1XUxMGI0bR1Pwjdvl8uYnSgCHw13pyYeFKEiSZZAYPLg1d93Vm7AwM1FRNuLjI/n883Hluofb7aVfv9lMm/YTH320lQceWMGECZ9XUcRVRynFsmXX0rx5DFarifBwC0OGtMlfA/1P0dGFuyYJUVHSwBNkTpzIIjXVSatWdQkLK18tysqVBxg16kMyMlz5+2w2M0lJ91KvXvC1rGutSUvLITraxtGjGXTtOoOMjBy8Xo3dbmXhwnFcdFFro8MUQUQaeCrh0KE0nnxyFSdOZHL55R259touhra4xsdHEh8fWaFrc3I8hWI3mXL3B6M/1z0HaNGiDps338Jbb23E6fQwdmwnEhMrNlO7EEWRZFmCY8cy6d79DdLSsvF6NcuW7eePP9J54IELjA6tQvr2bU5EhIWsLBder8ZmM3POOQ1p1CjK6NAConnzOjz66CCjwxC1lNRZlmDevK35iQVyGw2mTfupWp79669JzJnzG6tWHQRgzZojfPTRVnbsSK7wPWNiwlizZjKDB7emTZtYrryyI8uWXSd9E4UoAylZlsDt9hVqLT5zydeq8NJLa3nooRWYTCa01rRpE8u+facwmRQej49XXx3BxIkVmww3IaEuS5deG+CIhaj9pGRZgssuO9uvEcVut3LDDd2q9JlpadlMnboch8NDZqaLrCw3mzefICvLTUaGC6fTw623foXT6S79ZjXYs8/+RFTUU9hsj3PNNQuCtt5UhA5JliVo06YeP/xwAwMHJtClSzxTp57H888Pq9JnpqQ4ipyhpyCTSXHyZMXX5DbaggXbefTR78nKcuN2+/jss53cc4/MnyJqNnkNL0X37o357rvqWxqoefM6RERYCy0FW1BUlC2oG2W+/HK337DN7GwPixfvMTAiIUonJcsaxmYzs2LFdTRtGo3ZrKhTJ4xHHx2A3W7FZjPTsGEky5ZdV2rpsyZr1CjKbzQSUOHuUEJUF+mUHkA5OR7Wrj2K1prevZsRHl65gntWlgu73YpSCq/XR1paDrGx4UHfep2S4qBbtxmcOpWNz+fDbDbx3XcTQnYGJVFzSKf0anDqlJM+fd4mKSkDgIYNo1izZhL169srfM/ISFv+92azKShH2RSlQQM727bdxiefbMfp9DBiRDtat5ZFPUXNJskyQB566BsOHDidP5lDdvZp7r9/ObNmjTY4spqpTp1wJk3qUfqJQtQQwVvxVcPs2JHsN+uN2+1jx44UAyMSQgSSJMsA+XMo4Z/Cwy307duMo0fTmTLlCy655ANmzFhPMNYRCyHkNTxg/vOfgWzcmJQ3PFHRt28z7rmnDz16vEFqqhOPR7Ny5QEOHDjNtGkXGR1uUHn77Y3cd98yHA4PI0a05b33Licqylb6hUIEkLSGB5DWmj/+yG3gadIkmrff/pW77vrar09hWJgZp/OfQd+iXV1WrjzAJZd8kP8ZhoWZGTPmbD766EqDIxO1kbSGVxOlFE2bxuRve72+Qq/dwTgzuZGWLdvn98smJ8fLsmX7DIxIhCqps6xCI0e2x2o15y9/YLdbuP76rlKqLIe4OHuh/qq1YQlgEXwkWVahpk1jWLNmEsOGtaVbt4b84x99mTFjpNFhBZXJk3vQrFk0druVsDAzdruVGTMuqZZnb958nPHjP2HkyA/49NMd1fJMUXNJnaWo8TIzXcybt5WMDBdDhrSmU6f4Kn/mjh3JnHvumzgcbrTOnXHq1VdHVPmsU8JYUmcZRE6fzmb//lM0bx5DXJyMl4bciUOquwP7G29s8JvMxOFw89RTqyRZhjB5Da9BlizZQ7Nm/2PQoDm0aPECb721weiQQtaZ65ID+TPmi9AkybKGcDjcXHXVx2RluUlPzyE728Odd37NgQOnjQ4tJN14Y3fsdmv+dkSEhbvu6m1gRMJokixriKNH0wu1kttsZvbsOWlQRKGtR4/GvPHGSMLCzEBul682bWSyj1AWkGSplBqulNqllNqrlHqwiOPPK6V+y/varZQ6XeCYt8CxRYGIJxg1aRJdqA+my+Wlbdt6BkUU2nw+zQMPrCAnJ3e8f06Ol7FjP+HgQSnph6pKJ0ullBl4FbgY6AiMV0p1LHiO1vofWutuWutuwMvApwUOO/88prUeVdl4glVkpI0PP7wcu91KnTphRERY+O9/h9KqlZRmjHD8eCapqf5Ld1gsJn799ZhBEQmjBaI1vBewV2u9H0Ap9REwGthezPnjgX8H4Lm1zqhRZ3Pw4N3s3ZtKixZ1aNIk2uiQQlZsbESh0Vder4/GjYN3OQ9ROYF4DW8KHC6wfSRvXyFKqZZAK+DbArvDlVLrlVJrlFJjAhBPUGvQwE6fPs0kURosPNzCyy+PwG63EhVlIzLSytixnejVS2ZzD1XV3c9yHPCJ1tpbYF9LrfVRpVRr4Ful1BatdaHBv0qpKcAUgBYtWlRPtCKk3XRTD3r3bsqvvybRokUdBg5MkKGqISwQyfIo0LzAdrO8fUUZB9xecIfW+mjen/uVUiuB7kChZKm1ngnMhNwRPJWOWogyOOechpxzTkOjwxA1QCBew9cB7ZRSrZRSNnITYqFWbaXU2UAs8HOBfbFKqbC87xsA51N8XacQQhim0slSa+0B7gCWAjuA+VrrbUqpx5RSBVu3xwEfaf9a8w7AeqXUJuA7YJrWOqiS5Vdf7aZRo/8SHv4EF144h5MnHUaHJISoAjKRRgVprVmz5ggXXjiH7OzcKlir1UTv3s1YtWqiobEJISpGJtIIsMOH0xg69D327En1Gy/sdvtYvfowPp/GZJKGACFqExnuWAGXXTavUKL8k91ulUQZRLKyXBw/nikLyYlSSbKsgE2bjhdKlCaTIiLCwiuvjDAoKlFe//rXt8TGPkPLli/QqdNrJCVlGB2SqMEkWVZAXJzdbzsszMz48Z359tsJTJjQ1aCoRHl8+eVuXnhhDW63j5wcL3v2nGTcuE+MDkvUYJIsK+D99y8nMtJKdLSNqCgbF17YinffvYw+fZoZHZooo19+Oeo3ua/Ho2XctyiRNPBUwIUXtmLbtttYu/Yo9etHMGhQK6mnDDKtWtXFbrf6rRzZtKkMMRXFk2RZQS1b1qVly7pGhyEq6LrruvLee5tZt+6P/F90779/ucFRiZpMkqUISRaLiRUrrmfVqoOkpeXQp08z4uNlzSNRPKmzLIc339xAixbP07jxdB59dGWhyXpF9XM43GzY8Ae//36q3NeaTIoBAxIYNeosSZSiVFKyLKNPP93B3Xcvza/jevbZ1djtVu6//3yDIwtdO3YkM2DAO+TkeHG5vFx//TnMmDFSZgYSVUJKlmU0d+4Wv8YAh8PN3LlbDIxIXHXVx6SkOPIXeJs7dwsLF+4yOixRS0myLKO6dcMKtXjHxIQZFI0A2Ls3lYIDb7KzPezYkVype/p8mvvuW0adOtOoV+8Zpk37UUb3CECSZZk9+OAFREXZMJsVSuUOa3z66cFGhxXS2rSJpeAbd3i4hQ4d4ip1z6efXsXrr68nPT2HU6eyefzxH3jvvc2VjFTUBpIsi+H1+jh8OI3MTBcA7drVZ9OmW3j44X488MD5rFkzifPPlxnbjfTxx2OpX99OTEzuAm/XXNOF0aPPqtQ9P/lke6Hqlo8/DqpZA0UVkQaeIuzZc5LBg98lJcWB1+vjyScHc99955GQUJfHHhtkdHgiT8eOcRw4cBc7dqRQr14ErVtXfiXM+vX9h7KaTKrQ8FYRmqRkWYRRoz7iyJF0nE4PLpePf/97JT/9dMjosEQRIiNtJCY2CUiiBHjuuSFERdmwWk3YbGZiYsJ45JEBAbm3CG5SsjyDz6fZtSvFr+HA59Ns3Jgkr90hoHv3xvz2280sWLADi8XEuHGdZaVNAUiyLMRkUsTHR3L8eFb+PovFRKtWgSm5iJqvTZt6TJ0q/WeFP3kNL8K8eVcSFWWjTp0wIiOtXHppey65pJ3RYQkhDCQlyyIMGJDA7t13sHFjEnFxkZx7bhMZFSJEiJNkWYzGjaO55BKpqwp1mZku5s3bSmami6FD21S6H6cIXpIshShGWlo2PXq8wbFjWXi9Ph5++Fu+/HI8gwa1Mjo0YQCpsxSiGG+8sYGjRzNwONzk5HhxONzceutXRoclDCLJUohiJCdnkZPj9duXmuo0KBphNEmWQhRj6NA22O3W/O3wcAvDhrUxMCJhJEmWQhRjyJA2TJ8+lJiYMGw2MyNGtGXGjJFGhyUMooJx+qnExES9fv16o8MQQtQySqkNWuvEoo5JyVIIIcogIMlSKTVcKbVLKbVXKfVgEcdvUEolK6V+y/uaXODYBKXUnryvCYGIRwghAq3S/SyVUmbgVWAIcARYp5RapLU+cxLAeVrrO864th7wbyAR0MCGvGvLv/qUEEJUoUCULHsBe7XW+7XWLuAjYHQZrx0GLNdap+YlyOXA8ADEJIQIUYdXr2bOhRfyZq9erHvttYAtCxKIETxNgcMFto8AvYs47wqlVH9gN/APrfXhYq5tGoCYhBAh6NimTbw3ZAhuhwOA5G3bcDscnHfffZW+d3U18HwBJGitzyG39DinvDdQSk1RSq1XSq1PTq7colQisBwnT/LhpZfyXFwcb/TsyfHNsmaNMMamd9/NT5QAboeDX15+OSD3DkSyPAo0L7DdLG9fPq31Sa11Tt7mW0DPsl5b4B4ztdaJWuvEuDiZzKCm0Frz/tCh7F26FEdKCsc2bmR2//5kyS80YQCTyQRnzBCmzObA3DsA91gHtFNKtVJK2YBxwKKCJyilGhfYHAXsyPt+KTBUKRWrlIoFhubtCzoul5d33vmNZ575MaSWoHCmpnJi61Z87r8W+UJrDv/0k3FBiZDV46absEVG5idMq91Ov4cfDsi9K11nqbX2KKXuIDfJmYFZWuttSqnHgPVa60XAnUqpUYAHSAVuyLs2VSn1OLkJF+AxrXVqZWOqbm63l/79Z7N16wlycrzYbGZefHE4kyf3MDq0KmeNiED7fH77tM+HLSrKoIgqZ+PGJN5/fzM2m5mbbupBmzb1Sjx/1aqD3HjjIpKTs7jggha8995lxMZGVFO0tZfP42Hlf/7DjgULiKhfn2H/+x9Ne/Uq9br67dsz6eefWfX007jS0+k6YQIdr7wyIDHJCJ4A+PTTHUyY8Hn+srmQO47Y4Xg4JCYNXn7//ax7/XXcWVlYIiJo2KULN/70EyZLcM0A+MMPB7n44rk4HG5MptzF0Natu4mzzmpQ5Pm//36KLl1eJysrt1Rts5no06cZ338/sTrDrpWW3Hknv779dn79ozUykikbNtDgrMotdVwaGcFTxU6dcuLz+f/Scbm8eDy+Yq6oXS569llGz5pF77vu4qJp07jhhx+CLlECPPTQN/lrhvt8uRP/PvNM8dUJK1ce8Nt2uXz8+ONh3G5v0ReIMts0Z45fQ403J4edn39uYEQy+W9A9O/f0m/bajXRs2cTrNbAVCzXdEopOo0dS6exY40OpVKyslx+21pDWlpOMWdDTExYoTcHq9WExSJlkMo685etMpsx22wGRZNL/lYDoF27+nz22dU0bRpNeLiF889vwaJF44wOS5TT9dd39ZuSzW63MmFC12LPv/TSs2jbth4REZb8859++qKQqHqpav3/7/+w2u1AbqK0RUXR5ZprDI1J6ixFubgyMzmydi2WsDCa9ekTlK/bxdFaM23aj7z++nqsVhP/938DuOGGbiVek53tYdasX0lKyqB//5YMGSLzXQbKtvnz2bFgAfa4OC548EFimjWr8meWVGcpyVKUWdrhw7zdpw+uzEy0z0f99u2ZuGpVfglAiGAnDTwiIL685RYyjx8nJz0dV2Ymydu38+MzzxgdVsBorXn00ZU0bjyd5s2fZ8YM+YUs/lJ73qFElUvdvRvt/aul15OdTcqOHSVcEVz+97+fefbZ1fkt4vfeu4z69SO46qpOBkcmaoJaX7L89tvfefXVX/jmm/1GhxL0mvbq5dciabXbad63r985KTt3svq//2Xda6/hPBVcM+3NnbslP1ECOBxuPvhgi4ERiZqkVpcsp05dzmuvrcPr1ZjNiptv7sn06cOMDitojXj1VU7u3s2JbdvQXi/tR46k15135h8/uGoVc4cPx+t2YzKb+eGJJ7h182bsDYru1F3TxMSE+W2bTIq6dcMNikbUNLW2gefQoTTOOutlsrP/em0MD7ewY8ftJCTUreoQay2tNRlHj2K22YiMj/c7NqNbN45v2pS/bbJaOX/qVC584onqDrNC1qw5wuDB7+J0ujGZFJGRVtatm0L79vWNDk1Uk5IaeGptyTI5OQubzeKXLG02MydOZEmyrASlVLFdOJyp/sP6fW43WSdOVEdYAdGnTzPWrp3MBx9swWo1MXFid/m3IvLV2mR59tkNsFj8OwebzYoOHYLjlTAYnTVqFL/OmoXH6QRy6zTPGl3WSfNrhs6d43nqqcFGhyFqoFrbwBMZaWPFiutp2bIOSkHz5jGsWHE90dF/1UtlZOTw4IMruOyyj5g+fTVeb2iM5a4qQ6dPp9PYsVjtdsJjYxny7LO0v+QSo8MSIiBqbZ1lQT6fxmTyL2W6XF569pzJnj0nycnxYrdbGTWqPR9+GJjpnIQQwSfkO6WfmSgBfvzxEAcPniYnJ7dO0+Fw8+mnO0lNdVZ3eEKIIBASybIobre30IQHSiHTa4liLVy4k7FjP2bKlC/Yty/o5qiuMbJOnODQjz+Sdii4VhSotQ08pTnvvOZERlrJynLh9WrCwswkJjYhPj7S6NBEDeP1+rj77q+ZOXMjLpcXkwnmzdvGpk23SGt5Oe1cuJBPr7kGk9WK1+Vi8NNP0+euu4wOq0xCtmQZHR3G2rWTGTGiHR06NOBvf+vCkiV/k+m1hB+v18fQoe/z6qvrcLly3zp8vty5L2fP/tXg6IKL2+Hg02uuwe1wkJOWhsfp5JuHHiJ1716jQyuTkCtZbt16gsmTF3HkSHr+mil16sgoDVG0zz7bydq1RzizHdTr1fn13aJsMpKSCq28aLbZSN23j3pt2xoUVdmFVLI8fjyTCy6YRXp6DlrD55/v5I8/MvjhB1kzRRTt2LFMvN7CPUYiIiyMH9/ZgIiCV3STJoXe3LwuV5WvqxMoIfUa/v33B/H5dH4pISfHy88/HyEjo/ilA0RhwdjdrKL69m12ZmGIyEgry5ZdR9eujYwJKkhZIyK4+rPPsEVFYYuOxhIezoiXX6ZuQoLRoZVJSCVLu91a6HUKcodBitJtmTuXaXXq8LjVyuwBA3CcPGl0SFWuZ88mvP76SCIiLJhMim7dGrF799+54IIWRocWlFpfdBH3/PEHN/74I/ccPUr3SZOMDqnMQqJT+p9ycjwkJr7J3r2pZGd7sNut3HZbIs89N7QKoqxd/li/nncGDMhfcc9ktdLigguY8O23BkdWPbTWuFxewsJCquaqyniys1ly113s/uILwmNjGfHKK7QaNMjosEJzIo2ihIVZWLNmEi++uJYDB04zcGCC1DuV0YHvv8fn8eRv+9xuDv9U/DKxtY1SShJlAC266SZ2LFiAx+kkMymJD0eOZPIvvxDfqeZOtBxyf/uRkTYefrif0WEEnci4uPy+cX8Kq1PHwIhEMNv56af5E64AeN1u9ixeXGKy3L5gAetefRWzzUa/hx+mZf/+1RFqvpCqsxQV13ncOBqcfTbWyEgs4eFYIiIY9dZbRoclgpQl3L+7nsliwRZZ/ICQrfPm8fn113Pgu+/Yt3Qpcy++mMOrV1d1mH5CrmQpKsZsszFp9Wq2L1iA8+RJEgYOJL6zVGGIihk8bRpL774bt8OB2WbDXr9+ieuCr3722fz6csjt4L725Zdpft551REuIMlSlIPZZqPL+PFGhyFqgZ433USdFi3Y89VXRMbHc+5ttxFet4Sho0WMrKvu0XYBSZZKqeHAi4AZeEtrPe2M4/cAkwEPkAzcqLU+mHfMC/y5KtQhrfWoQMRU1VwuLx6PD7vdanQoQgSltsOG0XZY2dbEOn/qVD6fOBFPXunSEhFB7wLrP1WHSidLpZQZeBUYAhwB1imlFmmttxc47VcgUWvtUErdCjwLXJ13zKm17lbZOKqL1pr771/Oiy+uRWvNgAEJLFw4jqgoW+kXCyEqpNPYsZhtNn75s4HnoYdo1qdPtcYQiJJlL2Cv1no/gFLqI2A0kJ8stdbfFTh/DXBtAJ5riLlzt/D66+vxeHJnVf/pp0PcfvtXzJlzmcGRCVG7nT1mDGePGWPY8wPRGt4UOFxg+0jevuJMApYU2A5XSq1XSq1RShn3SZTRN9/s91tbOifHy/ffHzQwIiFEdajWBh6l1LVAIjCgwO6WWuujSqnWwLdKqS1a631FXNrLNF8AACAASURBVDsFmALQooVxQ80SEuoSFmbOn3FGKWjaNMaweIQQ1SMQJcujQPMC283y9vlRSl0E/BMYpbXOn7lCa30078/9wEqge1EP0VrP1Fonaq0T4+LiAhB2xdxzT18SEuoSHW0jOtpGTEwYM2eONCweUTvkTvASfEOPQ0kgSpbrgHZKqVbkJslxgF+HKaVUd+ANYLjW+kSB/bGAQ2udo5RqAJxPbuNPjRUdHcZvv93C0qV7cTo9DBqUQMOGUUaHJWoorTWzZ//Gp5/uID4+kn//ewAtW/7VRcbj8XHzzV/w7rubUQpuu+1c/ve/YUWuGyWMVelkqbX2KKXuAJaS23VoltZ6m1LqMWC91noR8BwQBXyc1zfqzy5CHYA3lFI+cku5085oRa+RwsMtjB59ttFhiCDw5JOrePrpH3E43JjNioULd7Ft2200apT7C/aJJ37go4+25TcYvvnmRlq3juXOO3sbGbYoQkjNOlQZXq+PvXtTCQ+30KJFHVl+QpRJ3brTSEv7a77UsDAzzzxzEXfdldvt5dxzZ7J+fZLfNUOHtmbp0uuqNU6RK+SXwq2slBQH55wzg549Z9Khw6uMGTMvvyQgirdpzhxmdOvGG927s+3jj40OxxBnzrKutf++Jk2i/V65zWYlDYY1lCTLMrj55i/Zs+ckWVlunE4PK1bs55VXfjE6rBpty9y5fHXbbRzftIljv/3GwhtuYPeXXxodVrWbNKl7/igvpXInmr7ssr+qcKZPH0ZMTBh2u5XISCv160fw+OPGz+soCpOx4WXw669JuN1/lSQdDje//FKowV8UsO611wpNfLD+9ddpPzK0eg5Mnz6UBg3sfPrpDuLi7Dz33FBatYrNP962bT127rydL7/cjcmkGD36bOrVizAw4uDj83g4vHo1nuxsmvXpQ1hM1ZTMJVmWQYcODTh0KC3/9SkiwkKXLvEGR1WzmW2Fh3+eOS1XKDCbTfzrX/3517+Kn3uxYcMoJk3qUY1R1R5up5PZ/ftzcudOlMmEJTycST//TGzr1gF/Vsi+hh85ks7MmRuYPftXTp/OLvHcmTMvpWnTaGJiwoiMtNKzZxPuuadvNUVqDLfDwcJJk3i+RQve7NWLpI0by3V9/0cewRLxVwnJardz/gMPBDpMEeLWPP88yVu34srMJCc9HUdKCl9MmVIlzwrJ1vCtW09w3nlv4/H4MJkUMTG5fSfj44uffNTpdPPbb8cIC7PQrVujWt8P7sPRo9m3bBne7NxfJLaoKG7bto065Rg9deinn1j/+uuYzGZ6/f3vNEksspFR1CKp+/axYPx4Tu7aRb127bjiww+p365dlT3vs+uvZ/N77/ntq5OQwN2//16h+0lr+Bn+/vclZGa6cDo9ZGW5SU528OSTq0q8JiLCSt++zenRo3GtT5Q+r5c9X32VnygBtM/HvuXLy3WfFuefz+Xvv8+YOXMkUYYAT3Y2s/v3J2nDBnLS00nauJHZF1zgV3cdaM379sVqt+dvm202mvbqVSXPCslkeexYpt+SuB6Pjz/+SDcuoBpGmUyYzGcsD2wyYY2QhgdRvJSdO3FlZKB9eY2hWuPJzubEtm1V9syeN99MxyuvxGyzYQkPJ75LF0bOmFElzwrJZDliRDsiIv5q27LbrYwY0d7AiGoWpRT9/vnP/N/YZpuNyPh4zhoVFPMyC4OE1amDz+322+d1uwmvwoXtlMnEmDlzuDcpiTv37eOmdeuIiI0t/cIKCMnW8KefHszx45nMn78Nk0lxzz19uOGGrkaHVaMMeOQRGpx9NvuWLSOmWTP63H03tigZAy+KF9uqFZ3GjmX7ggW4s7KwRkZy1qhR1KvCOss/RdSrV+XPCMkGnj/9+bPL0EUhAkNrzbZ58zixdStxHTvSedw4lCl4XmBLauAJyZLlnyRJFqZ9PpI2bsTtdNKkZ0+/ynMhSqOUovO4cUaHUSVCOlkKf16Xi/eGDOGPDRswmc3YoqK4cfVq6rZsWa77aK358emn+Xn6dLTW9Lz5ZgY/+WRQlTCEOJP86xX51r70EkfXrcOdlUVOejqZx4/zxeTJ5b7Pr7NmserJJ3GmppJ96hS/vPQSPz//fBVELET1kWQp8p3YuhWP05m/rb1eUnbtKvd9ts2bV2hc+Lb58wMSoxBGkWQp8jXt1cuvjtJktdKoe5GrfJTIHhfn/8qtFPb69QMRohCGkWQp8vW8+WbaXXIJlvBwrJGR1GvbllFvvVXu+wx69FFs0dGYbDZMViu2yEgueuaZKohYiOoT0l2HRNHSjxzB7XQS27p14ZE8ZZR2+DDb5s1D+3x0vPLKKpkFpjY5dcrJc8+t5tChNC6+uC3XXNMlv7fGkiV7WLJkL40aRXHbbedSt27ozd5UXUrqOiTJUgiDZWa66NLldf74IwOXy4vdbuWee/rw+OMX8sorv/DAAytwONzYbGaaNIlm8+ZbiI4OMzrsWkkm0hCiBvvii12kpGThcuWuRe9wuHn22Z/QWvPww9/gcOQOIXS5vJw4kcX8+VU31loUT5JlKU6dcrJq1UF27UoxOhRRS2VnezjzBc/r1Xi9muxszxn7fWRl+Y+/FtVDkmUJ1qw5QkLCi1x66Yd07/4Gt9/+FcFYbSFqtqFD22A2//VfMTzcwsiR7bFYTIwadRbh4X+NHTGbTQwf3taIMEOeJMsSXH75PNLTc0hLy8Hp9DBnzia++aZik4qGEk92NulHjuDzeEo/WdC0aQyrVk3k/POb06pVXSZM6MqHH14BwLvvXsa4cZ1o1CiKTp3i+Prrv9G+vXTDMoIMdyyG1+vj2LFMv30+n2b37pNcdJG07BZny9y5LJo8GZTCFhnJtUuX0riHrC9TmnPOaciPP95YaL/dbmX27DEGRCTOJCXLYpjNJlq29J+HTylF586yUFlxUvfuZdGUKXiys/E4nThSUnh/+PC/JoMVIcntcPDNww/z4aWX8v3jj+N1uYwOqUKkZFmCRYvGc+GF75KT48Hl8nLffX3p3798k0qEkuObN2O2WCj48u3KyCArOZmohg0Ni0tUn5SdO1l+//1knjhBhzFj6HvffbwzcCAntmzBk53N/m++4fCPP/K3r78Oulm/JFmWoEuXhhw+/A/27UslLi6yxAXNBNRp2bJwPaVS1TIxqzBe2uHDvNW7NzkZGaA1yVu3cnzLFlJ27MCTt56Tx+nk4KpVnD5wgNhWrQyOuHzkNbwU4eEWOnWKl0RZBk169qTnzTdjtdsJi4nBYrdz2bvvYrZajQ5NVIOdn3+Ox+Xiz35QboeDHZ99BmeUIJVSQdn4F5CSpVJqOPAiYAbe0lpPO+N4GPAu0BM4CVyttT6Qd+whYBLgBe7UWi8NREzCGMP+9z/Oue460g4douE55wRd6UFUnFKKM1+sTSYTkfHxeLKz8bndmG026rdvT702bQyJsTIqXbJUSpmBV4GLgY7AeKVUxzNOmwSc0lq3BZ4Hnsm7tiMwDugEDAdey7ufCGKNu3fn7NGjiWnalI1vvcX3jz/O7999Z3RYoop1vPJKLBER+TNOWe12et91F5NWr6bD5ZcT17kzncePZ8J33wXlRNCBKFn2AvZqrfcDKKU+AkYD2wucMxr4T973nwCvqNza3dHAR1rrHOB3pdTevPv9HIC4hIG8bjez+/XjxNatuLOzsYaHM/jpp+l9551GhyaqSFSjRkzZsIHvHnmEzGPH6HDZZSTeeitKKa786COjw6u0QCTLpsDhAttHgN7FnaO19iil0oD6efvXnHFt0wDEJAy2Z/Fikrdvz58E2O1wsPz+++l1xx1BWaoQZRPbqhWXv/ee0WFUiaD5V6uUmqKUWq+UWp+cnGx0OKIU2adPF9rn83qDto+dEIFIlkeB5gW2m+XtK/IcpZQFqENuQ09ZrgVAaz1Ta52otU6Mi4sLQNiiKrXs399vHL3JaqVJYiKWcJmLsSwcDjdff72XJUv2kJUlv2BqgkAky3VAO6VUK6WUjdwGm0VnnLMImJD3/ZXAtzr3f9IiYJxSKkwp1QpoB/wSgJiEwWJbteKar76iTkIC1shIWvbvzzVffml0WEEhOTmLTp1eY+zYj7n66k/o2PE1jh/PLP1CUaUqXWeZVwd5B7CU3K5Ds7TW25RSjwHrtdaLgLeB9/IacFLJTajknTef3MYgD3C71tpb2ZhEzZAwYAB3/y4Tj5TXQw99w9Gj6bjducNEnU4PU6euYM4cGSNupID0s9RaLwYWn7HvkQLfZwNXFXPtk8CTgYhDVD+vy8UPTzzB4dWraXD22Vz4xBOE161rdFhBbffuk/mJEsDj8bFnz0kDIxIgwx1FJWitmXfZZfz+3Xd4nE4OrVrF/hUruGXTJixhsuxBRQ0Y0JL16//A6cwd5RIRYaFfvxYGRyWCpjVc1DxZx4+z/5tv8tca97pcZPzxB0fWrCnlSlGS//u/AQwd2gar1YTNZmLQoFY89tggo8MKeVKyFBWmtS407heQKdkqyWYz8/nn4zh1yonWUK9ehNEhCaRkKSohqlEjmvXund8dyGS1ElG/Ps369DE4stohNjZCEmUNIslSVJhSir8tXkyPm26icWIincaO5aa1a7FGyH9wUfvIa7ioFKvdzsUvvWR0GEJUOSlZAmlp2ezalYLTKUuMGuHAypXMHTGC94YNY8/ixaVfIGqt9CNH2DBzJr/NmUNOerrR4fgJ+ZLl7Nm/cttti7FYTJhMiq++uoYLLpBuGtXlwPffM3fEiPwW9UOrVnHlvHmcdemlBkcmqtuxTZuY3a8f2usFpfjuX//i5l9/xd6ggdGhASFesty3L5Xbb19MdraHzEwX6ek5jBz5AS6XDCKqLmuefz4/UULusgM/Pftsqdc5U1N596KLeDwsjOfi4nJn5BZBbfEdd+DKyMDtcODOyiLz+HF+euYZo8PKF9LJcvv2ZKxW/7mG3W5voSVwRRUqMNlGvjJ0PZp/5ZUc/OEHfC4XjpQUPr32Wo5t2lQFAYrqknXsmN+2z+0m/WiR8+oYIqSTZevWsbjd/qVIrZH1dqpR77vvxlKg9dwSEcF5U6eWet2hVavwuf+qY9ZeLwdWrqyKEAW5fWedqal+M0kFWtvhw/3+LVjtdtqNGFFlzyuvkE6WnTrF89BD/YiIsFCnThh2u5W5cy8nPDzkq3KrTatBgxj/xRe0HjKEhEGDuHLePM4ePRpHSgofXnop/23UiLd69yZ5xw6/62zR0X7bJosFe/361Rl6yNj/zTdMq1uX6Y0b81yDBlU2QmvIf//LWZdeisliwRwWRt9776XL3/5WJc+qCFWVvymqSmJiol6/fn3A7rdnz0kOHkyjY8c4mjSJLv0CUaW01rzRvTvJ27fnlh6VIiI2lr/v2ZO/rO7WefNYOHEi2uvFZLVSv317Jv38s4xJDzBHSgovJCTgzsrK3xdety73HD2K1W6vkmdqnw+UMmRdcaXUBq11YlHHpAgFtGtXn3btpFRSU2QeO0bKrl1/vWZrjc/j4ciaNfmvZZ2vvpp6bdty8Pvviahfn87jxkmirALJ27djsvinCZ/Xy6nffye+U6dy3Wvb/PlsmzePiPr16ffww9RNSCjyvJq67IgkS1HjWCMicruPFKB9PqyR/nXJTXr2pEnPntUZWsiJbtq00FIgXpeLqIYNy3WftS+9xDcPPYTb4UCZTGz/+GNu3bqVmKbBs+RWzUzhIqSF161LzylT8pOjJSKC+C5daHH++QZHFnrqtWnDeffdh9VuxxYdjSUigsHTppW77+MPTzyRv3id9vlwZWWxZe7cqgi5ykjJUtRIF7/8Ms369uXIzz9Tr107Em+5pdDroKgegx57jLNGj+bk7t3Ed+pEw3POKfc9fB6P37b2+fC6g2vEnDTwCCGq3PL772fda6/lly6tkZFMWb+eBmefbXBk/qSBRwhhqIueeYawOnXYNn8+4bGxDH3uuVITZfrRo2SfPk29tm1rROOdlCyFEJV2Yts2krdto167djTu3r1S99Jas+TOO9n45puYbTZsUVFM/OEH6rVtG6Boi1dSyVIaeIQQlbL2pZd489xzWTR5MrMvuICV//lPpe63a9Eifps9G29ODq6MDDKPHWP+lVcGJthKkGQphKgwR0oKy6dOxeN05k+C8dMzz5C6b1+F73li61a/yVXQmpO7dwcg2sqRZCmEqLDMY8cw22x++8xhYWRUYgKM+u3b+40RB4rtwF6dJFkKISostnXrQiNufB4PDTp0qPA9O155JWeNHo3VbiesTh0i6tXjqvnzKxtqpUlruDCcJyenRrR2ivKz2u38bckSPrjkEtwOB2abjbGffEJkXFyF76mU4vL33ydlxw6yT58mvksXwqKNn7NBkqUwzJG1a/lo1CgcKSlExsczbtEimp57rtFhiXJq3rcvU1NScJw8SUS9epjM5tIvKoVSiriOHQMQXeDIa7gwRE56Ou8PG0bWiRNon4/MY8d4f+hQcjIyjA5NVIAymYiMiwtIoqypJFkKQ6Ts3FlolnTt89WIVs9glnboEHuXLpXPsQpUKlkqpeoppZYrpfbk/RlbxDndlFI/K6W2KaU2K6WuLnDsHaXU70qp3/K+ulUmHhE80o8cwVVgjkTIrbss72w24i9bPviAVzp04JOrr2ZGt278OG2a0SHVKpUtWT4IfKO1bgd8k7d9Jgdwvda6EzAceEEpVbfA8fu11t3yvn6rZDwiCBxYuZJPr7vObxo2S3g450+dSkyzZgZGFrxcmZksmjQJj8NBTloaHqeT7x97jJN79hgdWq1R2WQ5GpiT9/0cYMyZJ2itd2ut9+R9/wdwAqh4U5kIet8/9hievAkV/tS0d28GPfaYQREFv8xjx1Bn1BeabTZO//67QRHVPpVNlg211kl53x8DSnyHUkr1AmxAwe79T+a9nj+vlJL+IyHAm5NTaJ8tUhaJq4yYZs0KNa54Xa5K9XcU/kpNlkqpFUqprUV8jS54ns6dkaPYWTmUUo2B94CJWus/1zp9CDgbOBeoBzxQwvVTlFLrlVLrk5OTS//JRI2VeOutfuu3WO12et5yi4ERBT9LeDjjFi0iLCYGa2QklogIRs+aRZ3mzY0Ordao1KxDSqldwECtdVJeMlyptT6riPNigJXAU1rrT4q510DgPq31yNKeK7MOBb/fZs/m5+efR5lMXPDQQ3S++urSLxKl8mRnk370KFGNGklpvQKqcj7LRcAEYFrenwuLeLgN+Ax498xEqZRqnJdoFbn1nVsrGY8IEt0mTqTbxIlGh1HrWMLDqdemjdFh1EqVrbOcBgxRSu0BLsrbRimVqJR6K++csUB/4IYiugjNVUptAbYADYAnKhmPEEJUCZn8Vwgh8sjkv0IIUUmSLIUQogwkWQohRBlIshS1VsquXbzWpQtPhIfzaocOHN+yxeiQgpYrM5MDK1dy9Jdf0D5f6RfUQjKfpaiVPNnZvDNgAFknToDWpOzcyZyBA7nrwIEaMZFsMDm1fz9vn3ceHqcTn9dLk549uW758kLLSdR2UrIUtdLJ3btxOxx+08D5vF5ObJWuvOW18MYbcSQnk5Oejjsri6Pr1rHutdeMDqvaSbIUtVJEvXp4XS6/fV6Xi4h69QyKKHil7tnj9+rtcTpJ3r7dwIiMIclS1EoxzZrRY/JkrJGRKLMZa2QkncaOpcFZhUbjilI07tEDk+WvGjtlMuHJySEY+2hXhnRKF7WW1prdX37Jia1baXD22Zw9Zgy5I2tFeWSdOMGsCy4gtcDcmJaICAY88ggXPFjUFLbBq6RO6ZIshQhyJ/fsIev4ceI6dcJqt+NITiayYUPMVmvAnrHqqaf47pFH/CZstjdowP21bAawqpxIQ4iQ5HW7OfjDD3iys2l+3nlExBZaUaVaLLnzTja+9RZmmy23jlZrUAqzzcb4L76gZb9++eem7NpF0oYNxDRrRot+/cpdylZK+c3BGIwFrcqQZClEObkdjtzX0r17UUphstmYtHo19du1q9Y4fv/2W36dNQuP04nH6fQ75nE6+XDkSO5NSsJqt7P1ww9ZOGkSJosF7fPR8YorGP3OO2VOmJ2uvpofn346d90krbHa7Zx7++1V8WPVWNLAI0Q5rXnhBZJ37MCVkUFOejrZqal8MWVKtceRsmtXiR3EtdacPngQn8fDwhtvxON04srIwJ2VxfYFCzj8009lfla9Nm24cfVq2o0YQbO+fRn81FMM/M9/AvBTBA8pWQpRTid37cKbnZ2/rX0+Q9a6ie/cGWUqvrzjc7uJatSInPT0QklVmUykHzlSruc17NKFa778skKx1gZSshQij9aaAytXsv2TT0g7fLjY81r06+e3LIbZZqNZnz4l3jv79GmW3nMPH156KaunT8dXoKGkolr260ffe+/FHBaGLToaS0QE5rAwwmJisEREMPyll4iIjSU8NpbIhg2hwCu39npp3LNnpWMIJdIaLgS5pcMPR4/m4MqVKJMJn9fL+C++oNWgQUWe++Utt/DbO++gTCYadu3KdUuXEl63bhF3BrfTyYyuXUk7eBCvy4XVbqfDFVdw2bvvBiT2zGPHyEpOpl7btpzat4/UffuI69jRrw41ZedO3hs2jMykJEwWC2PmzKHTVVcF5Pm1iXQdEqIU2xcs4PMJE3BnZeXvi2rUiHuTkoq9Jic9HU92Nva4uBIbSvYsWcInV1+NKyMjf5/JYmFqaqrfOHWtdZX2A9Vak5OeTlh0dImv76FMJv8VohRphw7hc7v99mWV0ocwLCaGyPj4UhPcmfcFQKn8Potuh4N5l1/OEzYbT0VFsebFF8sXfBkppQivU0cSZQXJpyZqnfVvvMELCQm80LIlP0+fXqb+gE179So0pC++Sxe01vw8fTovJCTwYuvWbJg5E4AD33/Pi23a8HRMDO9ffDGZJ06QvH07J/fsKfS8lgMGYI2IQOWt620JD6flgAH5r+1f3XYbe5cswefx4M7K4tuHH2bP4sWB+jhEgMhruKhVtn74IYsmT86dcYjcNcmHTp9OYhnWJV/zwgssnzoVZTJRp3lzrluxgr1LlrDs3nv97jf46af55uGH81/ZlcWCxWbLbUDRmqZ9+vC3xYuxhIXl3zvt0CEW33EHp/bvp0W/fgybPj2/kei/jRqRdfy4Xyy9/v53Ln7ppYB8JqLsZASPCBm/vfNOfmKD3FfcTXPmlClZ9rn7bs697TZy0tOJqF8fpVSR99v41lt+12mPB7fHk7995Oef+emZZxjwyCP5++q0aMH4RYuKfK49Ls4vWZptNqIaNy79hxXVSl7DRa0SFhPj10UGwFaOyX7NNhv2Bg3y6yFtUVH+JyiFLTq6xHpKj9PJH+vWlfmZI19/HWtkJJbwcKyRkUQ3bcq5t91W5utF9ZBkKWqV/o88kvt6m5fMrHY7gx5/vML3u/CJJ/7qU6kUVrudES+/TP327fOfo8zm/PpIyK2TbNi1a5mf0eKCC7j5118Z8txzXPzyy9y6eTPhdepUOGZRNaTOUtQ6Kbt28evbb6N9PrpOmEDDLl0qdb/jmzfz25w5mEwmuk+eTIOzzsKTnc1vc+aQmZREw3POYcWDD5J57Bja56Nhly5c/+23WCMiAvQTieoi/SyFqGJel4vjmzdjsliI79IFU4GSZlnlZGTw6TXXsG/ZMqx2O8Oef55uN9wQ+GBFsaSBR4gqZrbZaJJY5P+xMls4cSL7li/H63LhdblYfPvtxLZp4zfNWmWk7t3L/hUrsEVH0+Hyy6XkW06SLIWoIfYvX443Jyd/2+10sn/FioAky4M//MDcESPy57tc9cQT3LR+PbbIyErfO1RIA48QNcSZY8stYWFExsUF5N5f3HQT7qws3A4H7qwsTh84wMY33wzIvUNFpZKlUqqeUmq5UmpP3p9FThetlPIqpX7L+1pUYH8rpdRapdRepdQ8pVRoLUQsRAEj33gDq92O2WbDardTp0ULuk2cGJB7O1JS/LY92dlklDDuXRRW2ZLlg8A3Wut2wDd520Vxaq275X2NKrD/GeB5rXVb4BQwqZLxCBG02g4fzqQ1axg8bRoXv/wyUzZuDNhrcsKFF2IuMKLIarfTevDggNw7VFSqNVwptQsYqLVOUko1BlZqrQutNaqUytRaR52xTwHJQCOttUcp1Rf4j9Z6WGnPldZwIconJz2dj8eOZf+KFZhtNi6aNo3ed95pdFg1TlW2hjfUWv9Zlj8GNCzmvHCl1HrAA0zTWn8O1AdOa63/HCd2BGhayXiEEEUIi4nh2q+/zp0xXSlZErgCSk2WSqkVQKMiDv2z4IbWWiuliiumttRaH1VKtQa+VUptAdLKE6hSagowBaBFixbluVQIkUemZ6u4UpOl1vqi4o4ppY4rpRoXeA0/Ucw9jub9uV8ptRLoDiwA6iqlLHmly2bA0RLimAnMhNzX8NLiFqKqaa1Ba0lAIaKyf8uLgAl5308AFp55glIqVikVlvd9A+B8YLvOrSz9DriypOuFqIlWPfkkT0ZE8LjNxkdjxvjNTCRqp8omy2nAEKXUHuCivG2UUolKqT/nseoArFdKbSI3OU7TWm/PO/YAcI9Sai+5dZhvVzIeIarc9k8+YdVTT+HNyUF7vexbupTFf/+70WGJKlapBh6t9UmgUP8DrfV6YHLe96uBImcy0FrvB3pVJgYhqtver7/2K0l6srPZv3x5qdelHz3Kzs8+Y+1LL+E8dYqW/foxetasYhc6EzWLDHcUopximjXDbLPhdbny90U2LK4jSK5t8+bx+Q034Cmw3vier77iozFjuGHlyqoKVQSQ1EwLUU597r6b6KZNsUZGYrXbsUVFMXLGjGLPd2Vm8vnEiX6JEnJnKjr04494CowHFzWXlCyFKKfwunW5dfNmdn7+OW6nkzZDh1K3Zctiz89ISip2yjZlMmG2WqsqVBFAkiyFqABbVBTnXHttmc6Nadas0FIXCn28vgAABPNJREFUkDujev//+z/pehQk5G9JiCpmjYhg3MKF2KKjsUVFYbJYaHvxxVw5fz79Hn7Y6PBEGUnJUohq0GrQIO47doy0w4eJbtKEsHIsoiZqBkmWQlQTq91Og7MKzTMjgoS8hgshRBlIshRCiDKQZCmEEGUgyVIIIcpAkqUQQpSBJEshhCgDSZZCCFEGkiyFEKIMKrW6o1GUUsnAwQpc2gBIKfUs40h8lSPxVVxNjg2qL76WWuu4og4EZbKsKKXU+uKWuawJJL7KkfgqribHBjUjPnkNF0KIMpBkKYQQZRBqyXKm0QGUQuKrHImv4mpybFAD4gupOkshhKioUCtZCiFEhdTqZKmUukoptU0p5VNKFduSppQarpTapZTaq5R6sBrjq6eUWq6U2pP3Z2wx53mVUr/lfS2qhrhK/DyUUmFKqXl5x9cqpRKqOqZyxHaDUiq5wOc1ubpiy3v+LKXUCaXU1mKOK6XUS3nxb1ZK9ahh8Q1USqUV+PweqcbYmiulvlNKbc/7f3tXEecY9/lprWvtF9ABOAtYCSQWc44Z2Ae0BmzAJqBjNcX3LPBg3vcPAs8Uc15mNX5mpX4ewG3AjLzvxwHzalBsNwCvGPhvrj/QA9hazPERwBJAAX2AtTUsvoHAlwZ9do2BHnnfRwO7i/j7Nezzq9UlS631Dq31rlJO6wXs1Vrv11q7gI+A0VUfHeQ9Z07e93OAMdX03JKU5fMoGPcnwGCliliRy5jYDKW1/gFILeGU0cC7OtcaoK5SqnH1RFem+AyjtU7SWm/M+z4D2AE0PeM0wz6/Wp0sy6gpcLjA9hEK/wVVlYZa66S8748BDYs5L1wptV4ptUYpVdUJtSyfR/45WmsPkAbUr+K4yhobwBV5r2ifKKWaV0Nc5WHkv7ey6quU2qSUWqKU6mREAHlVO92BtWccMuzzC/o1eJRSK4BGRRz6p9Z6YXXHc6aS4iu4obXWSqniuia01FofVUq1Br5VSm3RWu8LdKy1xBfAh1rrHKXUzeSWgC80OKZgspHcf2+ZSqkRwOdAu+oMQCkVBSwA7tZap1fns0sS9MlSa31RJW9xFChY+miWty8gSopPKXVcKdVYa52U9ypxoph7HM37c79SaiW5v3GrKlmW5fP485wjSikLUAc4WUXxlCs2rXXBON76//btnqWBIAjj+H8KX1o1hdgpCH4AEYl+ghQBwdoUaSz8FDZ2dlppKRaCRYqA4Esr2ohBLdRaLC3FYi12hMMYsiDZS/H84GC5LGQYlkl29o7YFx4mA11v/1UsTiGEtpntm1klhJDlvXEzGyEWyqMQwukfU0rLn7bhcAvMm9msmY0SDywGfuLsWkDDxw2g65+wmU2Y2ZiPK8AK8DjAmFLyUYx7HbgM3n0fsL6x/epf1Yl9r2HSAjb8VHcZ+Ci0YkpnZtM//WczWyLWiBw/hPj3HgJPIYTdHtPKy18Zp165LmCN2NP4BN6BM78/A7QL82rEk7dX4vY9V3xTwAXwDJwDk35/ETjwcRXoEE9+O0AzQ1xd+QC2gbqPx4ET4AW4AeYy5qxfbDvAg+frCljIvOaOgTfgy9deE9gENv1zA/Y8/g49ntIoMb6tQv6ugWrG2FaBANwDd37VhiV/eoNHRCSBtuEiIglULEVEEqhYiogkULEUEUmgYikikkDFUkQkgYqliEgCFUsRkQTfgZ70LBHGhiUAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model = MLP(2, [16, 16, 1]) # 2-layer neural network\n",
        "print(model)\n",
        "print(\"number of parameters\", len(model.parameters()))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1D1dzoIvalKj",
        "outputId": "188d33aa-0eda-4bee-a31c-0bf392051609"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MLP of [Layer of [ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2)], Layer of [ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16)], Layer of [LinearNeuron(16)]]\n",
            "number of parameters 337\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# loss function\n",
        "def loss(batch_size=None):\n",
        "    \n",
        "    # inline DataLoader :)\n",
        "    if batch_size is None:\n",
        "        Xb, yb = X, y\n",
        "    else:\n",
        "        ri = np.random.permutation(X.shape[0])[:batch_size]\n",
        "        Xb, yb = X[ri], y[ri]\n",
        "    inputs = [list(map(Value, xrow)) for xrow in Xb]\n",
        "    \n",
        "    # forward the model to get scores\n",
        "    scores = list(map(model, inputs))\n",
        "    \n",
        "    # svm \"max-margin\" loss\n",
        "    losses = [(1 + -yi*scorei).relu() for yi, scorei in zip(yb, scores)]\n",
        "    data_loss = sum(losses) * (1.0 / len(losses))\n",
        "    # L2 regularization\n",
        "    alpha = 1e-4\n",
        "    reg_loss = alpha * sum((p.value*p.value for p in model.parameters()))\n",
        "    total_loss = data_loss + reg_loss\n",
        "    \n",
        "    # also get accuracy\n",
        "    accuracy = [(yi > 0) == (scorei.data > 0) for yi, scorei in zip(yb, scores)]\n",
        "    return total_loss, sum(accuracy) / len(accuracy)\n",
        "\n",
        "total_loss, acc = loss()\n",
        "print(total_loss.data, acc)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WfyUT8HTbDAa",
        "outputId": "25cf63c8-a961-49e9-e67d-ef86693735d7"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.8958441028683222 0.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# optimization\n",
        "for k in range(100):\n",
        "    \n",
        "    # forward\n",
        "    total_loss, acc = loss()\n",
        "    \n",
        "    # backward\n",
        "    grads = total_loss.grad([p.value for p in model.parameters()])\n",
        "    \n",
        "    # update (sgd)\n",
        "    learning_rate = 1.0 - 0.9*k/100\n",
        "    for p, grad in zip(model.parameters(), grads):\n",
        "        p.value = Value(p.value.data - learning_rate * grad)\n",
        "    \n",
        "    if k % 1 == 0:\n",
        "        print(f\"step {k} loss {total_loss.data}, accuracy {acc*100}%\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AIqMAA9WcHD-",
        "outputId": "e2112cb7-b66e-4e4a-d274-adfdbcb33463"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "step 0 loss 0.8958441028683222, accuracy 50.0%\n",
            "step 1 loss 1.7235905336972024, accuracy 81.0%\n",
            "step 2 loss 0.7429006313851129, accuracy 77.0%\n",
            "step 3 loss 0.7705641260584203, accuracy 82.0%\n",
            "step 4 loss 0.3692793385976538, accuracy 84.0%\n",
            "step 5 loss 0.31354548191852194, accuracy 86.0%\n",
            "step 6 loss 0.28142343497724337, accuracy 89.0%\n",
            "step 7 loss 0.2688873331398391, accuracy 91.0%\n",
            "step 8 loss 0.25671472860574157, accuracy 91.0%\n",
            "step 9 loss 0.27048625516379227, accuracy 91.0%\n",
            "step 10 loss 0.2450702385365803, accuracy 91.0%\n",
            "step 11 loss 0.25099055297915035, accuracy 92.0%\n",
            "step 12 loss 0.21560951851922944, accuracy 91.0%\n",
            "step 13 loss 0.23090378446402732, accuracy 93.0%\n",
            "step 14 loss 0.20152151227899445, accuracy 92.0%\n",
            "step 15 loss 0.22574506279282233, accuracy 93.0%\n",
            "step 16 loss 0.19447987596204097, accuracy 92.0%\n",
            "step 17 loss 0.21089496199246363, accuracy 93.0%\n",
            "step 18 loss 0.159830773563036, accuracy 94.0%\n",
            "step 19 loss 0.18453748746883916, accuracy 93.0%\n",
            "step 20 loss 0.18977522856087634, accuracy 91.0%\n",
            "step 21 loss 0.19072704042579644, accuracy 93.0%\n",
            "step 22 loss 0.11733695088756478, accuracy 97.0%\n",
            "step 23 loss 0.12173524408232447, accuracy 95.0%\n",
            "step 24 loss 0.12615712612770447, accuracy 95.0%\n",
            "step 25 loss 0.1604909778080167, accuracy 95.0%\n",
            "step 26 loss 0.18747197705245802, accuracy 92.0%\n",
            "step 27 loss 0.16741837891059405, accuracy 95.0%\n",
            "step 28 loss 0.09586583491455394, accuracy 97.0%\n",
            "step 29 loss 0.08778783707420912, accuracy 96.0%\n",
            "step 30 loss 0.11731297569011848, accuracy 95.0%\n",
            "step 31 loss 0.09340146460619832, accuracy 97.0%\n",
            "step 32 loss 0.1245445490310345, accuracy 95.0%\n",
            "step 33 loss 0.0798400265277727, accuracy 97.0%\n",
            "step 34 loss 0.0772751923292168, accuracy 97.0%\n",
            "step 35 loss 0.07661250143094468, accuracy 98.0%\n",
            "step 36 loss 0.10610492379198369, accuracy 96.0%\n",
            "step 37 loss 0.0906280842926597, accuracy 99.0%\n",
            "step 38 loss 0.10671887043036933, accuracy 95.0%\n",
            "step 39 loss 0.052256599219758504, accuracy 98.0%\n",
            "step 40 loss 0.0601600989523446, accuracy 100.0%\n",
            "step 41 loss 0.08596724533333948, accuracy 96.0%\n",
            "step 42 loss 0.051121079431795995, accuracy 99.0%\n",
            "step 43 loss 0.05240142401642835, accuracy 97.0%\n",
            "step 44 loss 0.04530684179001564, accuracy 100.0%\n",
            "step 45 loss 0.07211073370655108, accuracy 97.0%\n",
            "step 46 loss 0.03334238651310238, accuracy 99.0%\n",
            "step 47 loss 0.0314322279575112, accuracy 100.0%\n",
            "step 48 loss 0.03658536747111518, accuracy 99.0%\n",
            "step 49 loss 0.04829139382390286, accuracy 99.0%\n",
            "step 50 loss 0.09875114765619633, accuracy 96.0%\n",
            "step 51 loss 0.05449063965875457, accuracy 99.0%\n",
            "step 52 loss 0.03392679435708289, accuracy 100.0%\n",
            "step 53 loss 0.05261517263568447, accuracy 97.0%\n",
            "step 54 loss 0.032502952514249284, accuracy 99.0%\n",
            "step 55 loss 0.028883273872078122, accuracy 100.0%\n",
            "step 56 loss 0.041391511040272465, accuracy 98.0%\n",
            "step 57 loss 0.018987407426128457, accuracy 100.0%\n",
            "step 58 loss 0.025238335238837444, accuracy 100.0%\n",
            "step 59 loss 0.020796565213418834, accuracy 100.0%\n",
            "step 60 loss 0.03259711157810238, accuracy 99.0%\n",
            "step 61 loss 0.017863351693480314, accuracy 100.0%\n",
            "step 62 loss 0.023008717832211773, accuracy 100.0%\n",
            "step 63 loss 0.022079325463581358, accuracy 100.0%\n",
            "step 64 loss 0.029432917853529764, accuracy 99.0%\n",
            "step 65 loss 0.01625151464409186, accuracy 100.0%\n",
            "step 66 loss 0.02846853448326455, accuracy 99.0%\n",
            "step 67 loss 0.013994365546208736, accuracy 100.0%\n",
            "step 68 loss 0.015552344843651273, accuracy 100.0%\n",
            "step 69 loss 0.03389119946160181, accuracy 99.0%\n",
            "step 70 loss 0.014229870065926934, accuracy 100.0%\n",
            "step 71 loss 0.013255281583285518, accuracy 100.0%\n",
            "step 72 loss 0.012300277590022075, accuracy 100.0%\n",
            "step 73 loss 0.012676052498355966, accuracy 100.0%\n",
            "step 74 loss 0.020593811955954805, accuracy 100.0%\n",
            "step 75 loss 0.011845398205364469, accuracy 100.0%\n",
            "step 76 loss 0.01601269747288298, accuracy 100.0%\n",
            "step 77 loss 0.025458360239222176, accuracy 100.0%\n",
            "step 78 loss 0.01438293028966192, accuracy 100.0%\n",
            "step 79 loss 0.011698962425817985, accuracy 100.0%\n",
            "step 80 loss 0.012318500800515756, accuracy 100.0%\n",
            "step 81 loss 0.014121117031464238, accuracy 100.0%\n",
            "step 82 loss 0.011664591962446225, accuracy 100.0%\n",
            "step 83 loss 0.011589314549188722, accuracy 100.0%\n",
            "step 84 loss 0.010990299347735225, accuracy 100.0%\n",
            "step 85 loss 0.01098922672069161, accuracy 100.0%\n",
            "step 86 loss 0.010988193757655071, accuracy 100.0%\n",
            "step 87 loss 0.010987200447388703, accuracy 100.0%\n",
            "step 88 loss 0.010986246779084923, accuracy 100.0%\n",
            "step 89 loss 0.010985332742365272, accuracy 100.0%\n",
            "step 90 loss 0.010984458327280174, accuracy 100.0%\n",
            "step 91 loss 0.010983623524308863, accuracy 100.0%\n",
            "step 92 loss 0.010982828324359074, accuracy 100.0%\n",
            "step 93 loss 0.010982072718767005, accuracy 100.0%\n",
            "step 94 loss 0.010981356699297042, accuracy 100.0%\n",
            "step 95 loss 0.010980680258141725, accuracy 100.0%\n",
            "step 96 loss 0.010980043387921508, accuracy 100.0%\n",
            "step 97 loss 0.010979446081684675, accuracy 100.0%\n",
            "step 98 loss 0.010978888332907225, accuracy 100.0%\n",
            "step 99 loss 0.010978370135492719, accuracy 100.0%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# visualize decision boundary\n",
        "\n",
        "h = 0.25\n",
        "x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
        "y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
        "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
        "                     np.arange(y_min, y_max, h))\n",
        "Xmesh = np.c_[xx.ravel(), yy.ravel()]\n",
        "inputs = [list(map(Value, xrow)) for xrow in Xmesh]\n",
        "scores = list(map(model, inputs))\n",
        "Z = np.array([s.data > 0 for s in scores])\n",
        "Z = Z.reshape(xx.shape)\n",
        "\n",
        "fig = plt.figure()\n",
        "plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral, alpha=0.8)\n",
        "plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)\n",
        "plt.xlim(xx.min(), xx.max())\n",
        "plt.ylim(yy.min(), yy.max())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "5UDvz4MIcqqZ",
        "outputId": "4dce5ab5-e5ae-4da6-8328-2f8198f4557a"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(-1.548639298268643, 1.951360701731357)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deYyk933n9/f3eZ6q6vu+z5meGQ45pEiKl0jJkiWLtihFkNYbGZASBHayCwXBCpsLSLww4EUWWMDJBgmctRGHkB17gbW9gjaytVh5JetYHZYlc0RRFMmZ4dwz3dM9fZ9VXcfzfPPHU31U19HVXdVnfV8Awe66nqdrqj711O/5/r4/UVWMMcacfs5R74AxxpjDYYFvjDE1wgLfGGNqhAW+McbUCAt8Y4ypEd5R70Ap7bF6HWhsOerdMMaYE+OdhelZVe0udN2xDvyBxhb+9Jc/d9S7YYwxJ8bTX/rdu8WusyEdY4ypERb4xhhTIyzwjTGmRljgG2NMjbDAN8aYGmGBb4wxNcIC3xhjaoQFvjHG1AgLfGOMqREW+MYYUyMs8I0xpkZY4BtjTI2wwDfGmBphgW+MMTXCAt8YY2qEBb4xxtQIC3xjjKkRFvjGGFMjLPCNMaZGWOAbY0yNsMA3xpgaYYFvjDE1wgLfGGNqRFUCX0T+SESmReStItd/WESWROSN7H+/XY3tGmOMKZ9Xpcf5Y+D3gH9V4jbfV9VPVml7xhhj9qgqR/iq+j1gvhqPZYwx5mAc5hj+SyLyMxH5KxF5vNiNROTzInJZRC4vJBOHuHvGGHO6HVbgvw6MqupTwL8E/qLYDVX1VVV9TlWfa4/VH9LuGWPM6Xcoga+qy6q6mv35a0BERLoOY9vGGGNChxL4ItInIpL9+YXsducOY9vGGGNCVanSEZE/Az4MdInIOPBPgQiAqv4B8BngvxGRDJAAPquqWo1tG2OMKU9VAl9VP7fL9b9HWLZpjDHmiNhMW2OMOSWG++6UvN4C3xhjToHhvjs0jA2XvE21ZtoaY4w5AhtH9Q1jw0SebCh5WzvCN8aYE2ovYQ8W+MYYcyIVCnvnxV8seR8LfGOMOWH2E/ZggW+MMSdKqbCX+p6S97WTtsYYc0LsDHunZwTGzgJh2F9dvF7y/naEb4wxJ0B5YV/6xK0d4RtjzDG3Efatf+8iQE7YX0suQXIJaODLt0q3lLcjfGOMOcZ2DXugnLAHC3xjjDm29hr2r32zpeTjWeAbY8wxVO2wBwt8Y4w5dvYb9r31pUPfTtqaY2N9Jcnc7XkSy0nEgYbWetpH2qhrjh31rhlzaDaaoG3W2GfDPrfsMgz7OreN7389AHYPe7DAN8fE3N0F7v1kAvW31sVZmlhh8uo0fY/2MPB47xHunTG72601cbmKTajaCPsv3xJg72EPFvjmGPDTfl7Ybwpg+toMrX3NNHbu3hzKmKOwvUa+UruHPfsKe7DAN0cssbzO5NsP0aD4ipeBr0zfnONsGYGvqsTnE2igNHTU47h2msocrJ1DMNVwEGEPFvjmCE28Ocn0jbnCR/Y7LN5fIvl4L7HGaNHbrM6ucfMHd/AzASiICANP9dF7oauau20MUKSnTc9I5Q9coFVCNcIeLPDNEVmdWWOmzLAH0ECZ+NkkY+8fLXh9Opnh+ndv53xTUFUm3pgkWufRPtxW1nZS8RT335hkeXIFgNaBZoaeHiBaHynr/qY27LdbZbkOIuzBAt8ckdnb8wRlhv2GpWwIF3y8m3NFh4Xu/fQBiBCp82jsbEBECt4uk/K5+s0bZJL+5mWL48uszsZ5/JVHcCPunvbXnE6VdKss10ZfnI2yy2qEPVjgmyPip4M936dITgOwOhsvvq2kz93XxgHwYh4XPnSWWFP+0NDs7flwOChvX33m7izQY0NDNe8ow76SoN9QlcAXkT8CPglMq+oTBa4X4HeBTwBx4DdU9fVqbNucTG1DLaw8XMk7yhcHQPKP1gXahlqLPl5dS4yVh6tFrw+yQZ7KpLj+vds8/vFHSMXTPPj5FEuTK4gjuJ5TcIhJfWVles0Cv8btFva7tSYu384JVdUJe6jeTNs/Bl4pcf3HgQvZ/z4P/N9V2q45odqHW6lrqUOcrcN2cYT61npGXxhEXIHsVY7rEK2PMPhUf9HH679U/tFVJplhaXKFq9+8wcL9JYJMgJ/yScXThe8gEG20MfxaVqg1ceGwb6j4v73Ont2Lqhzhq+r3RORMiZt8GvhXqqrAj0SkTUT6VXWyGts3x0NyNcnixDIaKK0DLdS31hW9reM4PPKRMWZuzjF3ZxGAztE2us934rgOjR2NzN6aJx1P09zTSPtIW8kSSy/qMfrCEHf/brysfZ27Pb951L8bcYTuc51ooCxNhmP6kTqPjtF2InU2KnralduHPgzq3TtWluMgwh4Obwx/ELi/7ffx7GV5gS8inyf8FkB/Q/Oh7Jyp3NTVaSbfngYUVZi8Mk3X2Q6Gnu4vepLUcR16H+mm95HuvOtijVEG39O3p33oHG2npaeJ2TvzpOJplidXSSfyj9o1UNZXUyVr/x0v++GiMPL8IF7M48pfXycVTxNkAsQRHvx8Cq/Ow08H1DVHGXiij5Y+e82eJnvtQ1/nllcNVko1Ts4Wc+wOT1T1VeBVgMc7evdWxmGORHwxweQ707klkb4yd3uelv5mWkuEoGp4u8krM6QTaWKNUfof76VjZH9vnEh9hP7HwjYMyw9Xufk3d3LG5cUV2odaySR9ksvJvPuLI/Q92k1DRz0gNPc04rgO934yQXIlRfgllc2/NZ3IhM/Bwjo3f3iXM88PlV0Cao63/TUw23sxQiEHEfZweIE/AWyfczyUvcwcgORaiul3Z1mbjxNritL7SDcN7fUHtr252wsFT3YGvjJ7a75k4E9dneHhlenNk7fJ1RT3Lo8TZAK6xjoq2q+W3ibOfeAMEz+bJLG8jhd16T7fSd9jPaxMr7Iys5q/3wKdZzuINuSO2c/fW9wM+2LUV8Z/NknbUGvRbzXmZDiobpVH7bAC/6vAF0Tkz4H3AUs2fn8w4osJ3v3OLQI/nG0an0+wOLHM6PNDdFTpyFNVWZpcYfrdWTLJTMmxcD/lF70uyAQ5Yb95ua9M/HyKzrPtFQdnS28TLb9yIe/ypu5GOs+2M3drYevEsSpnXhzJC3ug5PDPdpmkTyblE4kduy/PpkynNeyhemWZfwZ8GOgSkXHgnwIRAFX9A+BrhCWZNwjLMv/LamzX5Lt3eSIvgNVX7v9kgrbBFhyn8sKsB29NMXN9bteJU44rtA0VfxOsryazxfWFvh0EpNczBzLDdfb2PBM/m0Q1/PCKRD16L3bTcaYd1yv8/DT3Nm3Ovt1Nsccwx99pDnuoXpXO53a5XoF/VI1tmeKCTEB8sXCVgCrEFxI0dTZuXuanfdZXkkTqPKINxXvUbJeKp5h+t/is1u28+gidZ4oPy3gxr/jjKAcys3VpaoXxnz7I+bBKxdNMXZ0pOYQ09FQ/12bXNvv0FCKOhB+q1rDtRDrIPvTHhX3vPE12Gf3YOLpXVR78fIrp63OIE05yauxsYOylEbxdhiJWHq4iUjTzcvbl3PtHSx7tRusjNHY1sDqzlvOA4oTfDA7iSHnqnfwhJAg//JamVmgbKPzmrWuO8divXODhtVlWZlbxoi7JlVT2AyB8vLrWOkaeHaz6PpuDVf6EqpMd9mCBf6o4rkNzTxMr06t5iex6DvVtYV38zPW5sHFZoJtH2Ksza1z99k3qmmOoH9A+0kZHgdp38ZyiwzA5txPZ9cMDYOzFEa5//3ZYMSOCqtLY0XBgwZlcTRW8XIMg77pMymdpchn1lZa+JqINUYbfO7B1H1VWZ9ZIrqWob62job3eTtaeMOWGfSWLjhwnFvinzOhzg1z91k2CtE/gK+IKIsLZl0Y2w2jq6kzBo9zUaopUNvTW5uLM3pznkY+M5YR+a1/zrtUqAPVtdWVNSvJiHo+9fIH4QoLkaoq6lljJCVuVqmuJsTqTybtcHIf6lq2lFOfvL3L3tfHwOdNwbkHPI105cwNEhOaeJqzy/mTaW9hXr4HZUbLAP2WiDVEe//hF5u8tEJ9fJ9YcpXPbjFBVJZPMD7ydAl9JLK8zd2eB7nOdm5e7EZezL45w+0f3QPOrVxzXwfGEM+/b28o/De31B1o6uqH/8V5ufP92bjmmQKTOo7m3CQjPU9x9bRz1Fd32TWbm+ixNXY209lcv4jfW8U0l0rT0NtM+3GrnAA5BLYY9WOCfSq7n0D3WCWP514kI0YZI8b4x26ivzN9dzAl8PxOQXk+Hk5dSGWKNUZq6m8gkMyRXktS11tE+3HZsK1Wauxs588Iw4z99QCZbMtrc08jo88Ob34Dm7i5ujstvF/jK9PXZqgX+5jq+gYLC0oMVpq5Mc/Gj5/Gi1or5oNRq2IMFfk0aeE9vWL5ZRj/67c3Nkmsprn3rBkEmCIeLHGHVidMx2k57iU6Wx037UCttgy2k1zO4npNXDZRZz6BFphaU8+2oHJlU/jq+QSY8jzD51hTDz9jJ34NQXl+c0xn2UL1umeYE6RhpZ+i9A3gxNwx0oWCFj+MKnWfaN3+/+9o4maS/+UGhgRJkAm7/7b2yxvWPExEhWh8pWPrZ3Nu01Utn+30cqVqvnOWpEou53FmoyjZMrvKboOWGfW99y6kIe7Aj/JrVdbaDzjPtZFI+rufw4K2HzN7cmkzleA5NXQ2bPW38jM/a7FrBx8okM6wvJw/0ZOthau1vpq45RmJpfeschYAbcarWE19Vi85BUF/xM8GxHRY7ifbaBK3afeiPCwv8GpRYXiedSFPfWkekLpzJOvRUP+1DrczdDfvitA210tLXtDmuXWyIAwjLKctsPXASiAgXPjzG1JVp5u8sEARK20AL/U/0Vq0dcktfc/EJXAKrM6u09p+usDkqp3327F5Y4NeITDLD7O15Zq7PkU5lsuWG0DHSxsizg4gjNHY20NjZUPD+XtSlrqWOxNJ63nUicmqO7je4nsPge/r23KK5XJGYFw6jFQh9sSqdqrGwz2WBXwOWHixz60f3ck4QbpQbzt9fJNIQYeDx3l0fZ+S5Qa5/9/ZmYzYI2w2PPj+Yc3LX5MokM8zemmd1Nk6sOUr3uU7qmmO0D7ayML5U8D7NPU2HvJenj4V9Pgv8U85P+9zeEfbbqa/MXJ+l/1LPrrNEGzsaeOyXL/Dw3Rni8wlizVF6L3bT0Hbw9fMn1fpKkmvfvkngB+G/wUOYvTXP2EsjDD7dz8rsGn7aD6+T8MTwyHODVotfIQv7wizwT7nFB8u79r3x09kj9jIO0mNNUUasZLBs916fyG0RreGH7J0f3+fJT13i8VceYe7OAisza0QbIptH/2b/aqEJ2n5Z4J9i68vrTL09XfTofkO0IWJDMmVKriaZujbL2uwa0cbwG05zd2PB2wZBEDaGK0AV1ubjNHU10nOhq2rVP7Ws1vri7IcF/imVXk9z7ds3w6P3EsQVBg7oxORps3NxmfXlJKvTqww91U/XttnI5VBVxt+YJJPKUN9WT/9jPYfSWuK0quXZs3thgX+KpBNpZm7Nk1hax982QaoYr85j6Kn+fa8fu1cbb8rj4P7Umb3f5/UHeYvLBNllDdtH8xdPcRyH5u5GVqbzj/LVV+IL4fhxai3N8tQK5z5whpZeO1m7Vxb25bPAPyXW5uNc/+7tnJbHxUQbI5z/4NlDHSveGFc9Loa5s6fQ10BZm4sXvlKEtbl4wbAefnaQa9/adtK2SCmm+sr91ye49Moj1mJ5Dyzs98YC/xRQVW7/6H7JtWU3iAPtw22HFvaF3pDHQQPDDHMHKPNoX8IJUcU6SDhu4ZCua4rx+McfYfbWAmtzaziey+LEUsHzKql4Gj/ll7WOgLGw3w97ZZ0CqbUU6fXdu19COEmq1FJ+1VTwDdkzcijbLiWYvkfkyQYaGCZ+6z7Dfbsf7YsIrYOtLE4s5R2hO64UnbAG4EU9+h7tBrpZX0mGj1GABsqDt6bof7x3cwa0Kaz8vjjhsJmFfciKfU+B3couHc/B8RzcqMvYB0aJNZa3fm0ljmvYw9Z+RJ5s2BxmKuf8wvAzA0QbopuN1cQVHM9h7P2jZQ/DxJqiJdcPnr29wJVvXCe9Xp2unKfR9hr7vYT9aWqCtl92hH8KxBqjRGJewR730aYoZ54fQkRo6DicJfhKfdU+LpyxswQ/+u6ejvQjMY9Lr1xgaWKZtfkE0YYIHaNteNHy30Yiwtj7R7j+nVv4G+P622k4WW762gyDT/Xv8687vawJWmUs8E8BkXCFqRvfu0MQhCWD4oTL9p193zCNHYc3dr7buOpxoYlpnBd/cc+h7zgO7cNttA/vv7KpvqWOJz75KLd/dI+lB/ltkjWAxcllC/wdbPZs5aoS+CLyCvC7gAt8UVV/Z8f1vwH8C2Aie9HvqeoXq7FtE2rqauSxj11g5sYciaV1Gtrr6T7XSbTh8MaCy11c4jh4tO3CvkO/GhzXoam7ieWp1YJVVYX69NeyaoW9JjNk7i2h8wmIuLjDLThdDTVTGVVx4IuIC/w+8MvAOPCaiHxVVd/ZcdN/o6pfqHR7prhYY5ShIzoqLPck2kHxM8oPvprk9W+lSCagd8Tho5+tY+TRwi/xq4vXc0KfW7eJPHnvUEO/fbiVBz+fyrvccYXu83ubyHWaVS3s1zOk/m4CMhvN/9JklpI4Qy1ELhxOIcNRq8YR/gvADVW9BSAifw58GtgZ+OaYyCQzjP9sksXxJVShpa+JoacGiDXt72TuXsdVD8KDL2WI31A0e65z6k7An/7vcQb/c5f60fzahM+MNWx+47gYa4WxszhA5Ml7tD55kaW/uHZgoe+nfebuLhCfT9A60BwO60hYpSOO0DbUSsdIG0EQMH9nkbm74QpYnWfa6Rxtr6k2GMX64kD42oqvKj/9tstrr68QaYUJbYGmwsM4mZsLsHPmeaAE48vocAtSpbUOjrNq/IWDwP1tv48D7ytwu/9URD4EvAv896p6v8BtzAEL/ICr37oRnuDNjiQsPVhhdeYGl165sOdywL0efdW51Z/Vuz6TIX5jYTPsN2gG5r4tnP+Hudtc9xf58q0EnxlrAOJcSy7lhH4wfY/Wv3cwob+xLrCfCU/YboR397lOInUeLX3N1LfWoYFy/bu3SSwkNmdMxxcSzN9d5MKHztZE6O/WBG1pVvl//5eA9YSPZiAONDlLeBe7oECXimC2yMQ5IJhP4A5UZ/nK4+ywPtL+HfBnqpoUkf8a+BPglwrdUEQ+D3weoL/h9P8DHLaF+0tkkn5eLWfgB0xfn9vTgh/7CfuNWuiKqeKuZHDXfWTdJxYUrjGOT2TytvnBj7UdWejfvRyuC7z5Z2TH7+fvLfKeTz66OZa8ML5EYmE9pz2G+kp8Ps7ig+UTtWj8XpU7oepPvhiQWNWttRkAAvCvzeH2NCI7l4gs9iEpJa47ZaoR+BPA9jnzQ2ydnAVAVee2/fpF4H8r9mCq+irwKsDjHb2nZ928Y2J1Zq3gjFwNlNWZ1bIfZ3/jqtUpj9OUT/r1SXQ9k32zKxT5HJGIm7fN7399GWjh+ZeXDzX0A79498wgE5BYXN9soLZwfyls0pb3GMrC/cVTG/h7mT0bv66FJ6EIBAvruN2554yc/iaC+0v5rxUFp+v4zAI/SNWYePUacEFEzopIFPgs8NXtNxCR7WcSPwVcqcJ2T73E8jp3/u4+V75xnds/urfZbKsSpVohR0pMCNpu+5sSyDtBG8ofxqlWeVz6rWk0ngZfISge9uFueOj2fvTb9mPj5F64n+EbfvPDauzs5gStjQ+1Spu/qVJ0llzgBznVOsVaNUBYGnoa7bVVQsn1Gwpc551tQ5qi4G67vyN4j3flfxs4pSr+K1U1A3wB+DphkH9JVd8WkX8mIp/K3uwfi8jbIvIz4B8Dv1Hpdk+7lelVrn3zBvP3FkksrbNwf4lr37nJ4oPlih6382x7wTeDuELPhfIrQwo1QitUZ1/tMXtN+ejievHpxTv/toUkqR9PlAz9OrctL/Slvicv9BvGhisKfddzkGJBrlDXstXfqPNMe8HQd1wn/Dc8ZfbTFyfZFi36MnDa89dYFtch8twA3hM9OCOtuOfaib5/CLeGlpOsyseaqn5NVR9R1XOq+s+zl/22qn41+/M/UdXHVfUpVf2Iql6txnZPK1Xl7uWJcPx22ytafeXe5XGCIGBhfIl3v3uLK399ncl3HpLZEWjFRBuijL00stluwcmG0NCT/TR1Fl7I4zjRtL/38da0T+bOYt7FG6H//a8HeaF/dfF6XuhvtGIY7ruzr+D3M0HRTqbiSs4C8c29TbSPtOWEvuMKHaNtNBVZcOWk2m8TtMSZRiTmbr0eNo7YL3UVXQheRHC7Gohc6MAbbUNqrFFdbf21J0RmPUM6UbgZWuArt398n5XJlc0TeuvLSWZvLfDYL58vq9Nia38LT37qMVam19BAae5pPDETfaS+RBWRU6SdpUIwE4dH8r/B9Na38DCxzPe/HvDBj7Xx5VuLgPCZMd2q1R/bXyuGvH0v8Tm1cbJ27s4CqXiK+rZ6hp8ZoOtsBwvj4YdV+1BbySZtJ9Fu8zfemnmXe1djfG88Sf0oNDa2b56E72ltQ19qwZ9cRRfWoc7FG2xBDnGy4UljgX8MFf3aT3hydWliOffIP1Ay6xkeXpth8MnyJl45rkNr/8mrghJHcM934F+fD8fvNziCdDegM/GCoV+qjHFn6IcVPDtCvwqzcjee86XJlbwhKVXl3e/c2tpfV4g2RLn4kTGGnhoo6/FPmlIn/qW+h7/6wTW+8vsB6SAePl0+rA2t0Xuud/MxxHXwhlpgyNonlKM2zlScMF7UK7rcnQaFKxNUlYXxwm13TxtvqAX3UhfUe2Gf+sYI3hPdRIqdg3AEZ7D0h9vO4R3YGkLYHN5ha6hhr502N4w8O0i0PpLTcRPIa6KmvpJcSXL/pw/KfuyTZLcqr8t33uXf/l8ByQQESdBkOK+i4V6CoArFC7XKAv+YOvPCMF7MxdkYiyxj2Pq0Vm/sFCyu499cgI16dkeQhggSc3Ef7QyHdjaeL1eQthhuGUeAhxH6kboIj3/8IqPPD9H3WHe4vGSJf7bF8WW02KorJ1Q5Jb1v/dAjU+i0VKBkbiycuufksNRGQpxAsaYoT3ziUYbf20/3+c5d815cofOQFjY5ShpPk/7pFCQy4ZCOgq6kSF+eRNM+Xn8z0ZeGcM+145xpJfJUL5Gn+8qemXoYoS+O0D7UysATfeHAfomyUtXdl6w8Scqdv/H67RRapA5Bl5Nkrszmhb6q4j9cI/X6JKm/myBzeyE8yW82WeAfY47n0Hm2g8En+0ouciKO0NjRQPf50x/4mXtLuWP3GwLFnwwnjkmdhzfaRuRcB0773tcA2FgoIz/0G/JC3+kZ2ffwDkC0vvi8CID6trqtb3kn3F4m69WPOGiJPzt4uIYuJnMuy1ydJfPODLqwjq6k8O8shSW5FvqbTscr6ZQTR4gWq04ROPO+IS784tmiQzrJtRR3L0/w9l9d493v3GRxorJa/qOkq6nCVwRa/Lp92gj9nRO0ri5ez5ugFXmyYV8TtDrPthet3hEHRp4ZrOAvOD42+uKUOzP76kQrQbRU4iv+w62Z4cFqimBqLfdgIFBI+WTu1sa5rXJY4J8AIsLgU3151TuOK3Sd66R9qK3oUez68jpXvnGduTvzJFdTrM7GufPjezx4++Fh7HrVSWORDz5Hil9XgYOelRutj3A2Oy9C3K1zD/VtdTz68oVTUYZZqgnazrCvc9vC59oR6l8YglKhv00wV7g6C4VgunA7i1pkZZknRPtQG+I4TLw5SXI1hRf16L3YRc8jXSXvN/7mVF7vnMBXHl6d2ezQeJK4I635R3IAwoF1O9wo23ztmwfTf2drXsRqdl5EU1nzItZXkizcWyQIlNaBZho7woU8VJXF8WXm7y6AQMdoO22DLYe+yMfeJlQlCi407p3vIHNtLmyjsZ0juL2NOb8XUwudRct1st7tNa5toIW2gfLrjddXkixP5S+hB+GbYGVmlY4Kluo7Ck5jlMiTvaTfmQkXsiA7Zv9EN1IgJDWZIVhKIp6DtNftO/QOOvTDGv3y/22nrk4z+fZ0eOJSYeb6LK0DLYy+MMStv7kbNsnLhuTKw1Xme5oY+0D5i61Xar+zZyG355LT24RMrqLLya3QdwWnpxFp22qf4HY34t9YyD/KdwSnBtoel8uGdE6pibemuPKN68V7zhD2djmJnM56or8wTOSFAaLvGyTy4iBOcyznNqpK+vocqR/eJ/PODOk3H5L6/j2C5WSRR93dUTVd2ymxvM7kO9M5czICX1l6sMLEm1OszsRz2ioHvrIysxZO+DoE1Qp7CA9MIk/34V3qxuluwOltJPKeHrzHunI+vKTOw73QkV+S2xqW5AarKYK5OJrcsWhCjbEj/BMovphgeXJlc3WkWGNul8vV2TWm350tXc4n0HxCmkapKjqfwH+wAr7i9Dbi9DbhNBbv7hlMrRGMr2RLHrPPg6+kfzpF9BeGi/Za2c32I/0Pfszhy7cWc470N1sxsHWkn34zzjDV66k/f2+x4L9t4AfM310s3FY5EzB/b3FP3xD3Y19h/x98nLhPp9Shjh/2x9lGHMHtacTtKd1DyBtqwe2ox59aRTMBbmcDNHqkLz9A19LhB4GC09uI92hXTQ71WOCfIKrK/dcnmLsbvuFF4MFbDxl4opfei92bt5u9NZ83c3M7xxXOvX/0xJT7Za7NEUyubo7bBwvryPgykWf6iwa3f2+xcPmmKsFsHLd3/x92hfrvbIR+qf47w9wBqDj4dXNN1kJXFr/fQY/m7Cfsf/DVNM3vrOAmA9KyDKo4/c14Fzv3NfwkDRG8sa1uoqkfT+RVbwUP1/BjLt6501/GvNPJeMcbAJYmV5i/uxSGuYIGYauFB28/zOm06JeoO65vreOJTz6676P77cMTO5tdAdve1FvT3ytZ5SpYTuaEfXihoqvp8Ii/iJ3tkHPum6y8LrtQp82DaMVQSOtAS8EPa3GF1sHmom2VO0YPrq1yoSZo+WHfkHdk33plFWupOiwAABzFSURBVDfhh/++fri2QTC5in+v8lLKYCUVrpuQd4Xijx/O8NZxY4F/gszenCv4dV19DSsystoGW4u+6XsuduFF9/fFbrfOhgXf1F8PNicy7UdmfLnoRKtgqni5ndOS3w893FHBaY0Vvm6PDqv/zk5N3Y009zTm/BtvzNUYeqqflv7mnA8ExxVaB5ppqeBbTSnbJ1TtfF1cSy5te13kLnXZnY5ttcfYLlD8e5XPFdFkpnhLkkxQk+0ZLPBPED9d/Eh5+3Xtw63EmmI5Y5TiCLHm6L6Xxiv1ps4N+9w39X6DXlVJX5tFJ8tfdjFHc4HxfQFpiSEt1Ql8OJrQFxHGPjDK8DODNHY2UN9WR//jvTz68nm8qMfZF0cYe/8IHaNtdIy2MfaBUc68b/hAKnT2s9TlxutC10t806rC7FinKVq4Nh+gzjv0MtXjwMbwT5C2oRbii4m88XnHc2jdVnrmuA6P/NI5Zq7PMnc37KXeOdpGz4WufY3b7/qmTi5R7fVrg7kEwYMSYe8ITn/hI9ZgJUVQaHalgnuho+pv9MNqr7ydiNB5pp3OM/nDNCJCS18zLX3NZJIZgh3fkIIgIJP08aJuRedx9reu8daHpNMcLXrOoRo97aXOw+lpCide7Wil7Z0/fauGlcMC/wTpGutg9uY8qXh6s0pDXKGhvZ6WvtxaY9dz6Hush77H8pcd3ItK39T7FUwUGcqB7JF6tOhEK398qeiRXTC5glvFI/wNewl9bt0m8uS9ikO/lORqijt/d39zHeRIncfws4OszcXDCi5VBOg+38nAe/r2/CFYjdeFNERwOusJ5hJ5gexWqS+U91gXfp2Lf385PEdQ5+Gdb6/opP1JZoF/griey6Mvn2f6+izz95ZwHKHzbDtd57aOWlWV5ckVZm7N46d92oda6TzbjuvtfUWrowp7yFaiFCFtdSU7YGrSL16tUmoYoUJbwzvLO0K/Pif0NyZoHVToB5mAa9++QWbb+HgqnubmD+6EM3Gz4arA9I05Al8Zfm/5i6xU83XhPdFD5uY8wURYcku9h3ehE7erOi0lxBG8cx24Y+2gNuvWAv+EcSMu/Zd66b/UW/D6+z99wPydhc2JN/GFBDM35nj05fN7WsbwKMMewOluxF9O5R/lu4I73FLyjSuNEXSuwCIZjiAdRU7mVlHYdG0ZyJ2Vu1HBtH1WbuTJe7Q+ubdZubtZGF8iyBTuK5PXUthXZm/NM/BEb1mvj1J9cTb+vr28LsQRIhc60fMdOYGsKR9dzyD1XsEZ1HslIsVP4NYQO2l7isQXE8xtC3sI39CpeJrp67NlP06xzoaFml1B9cMesn1xti9QDWFgN0VxShz9BYvrBPeLVHhEHNxDWtbxKGflJpbWC1ZzFSOOkCpUvrjDXsJ+swkapV8XGiiZe0ukfzxB6kfjpN+dI/XmQ1J/c4/065OkfnCf9Dszp2pNgKNkgX+KLE+uFHxjaKDMl1nXfBBv6v0QzyH6wiDuaCvS4CGNEdxz7eFkqxLjzZlrc0WHcyLP9iOH2E7iqEK/rjlWsCy3GA2USInF4Yf77uS/Ll78xbzXxZdvyebrolirhJztqpJ+Ywr/5kI4EzaRIbi/HK5LHJCty1eCh2tk3p0r++8xxVXl1S8ir4jINRG5ISK/WeD6mIj8m+z1PxaRM9XYrskljpTorV46AA7qTV0J8Ry8sXaiLw0TfXEIb6S15N+hu/TEz7zxkGBl/7109uMoQr99pLXoDOSdz584QvtwK1608LBJNfvi7KTz62FTtN2O3gMlmFxF9/CtxRRWceCLiAv8PvBx4BLwORG5tONm/wBYUNXzwP8J/K+VbtfkaxtspdBApbhC19niZWgH+aY+VELJcVqNp0n/ZBJN7D58UU2HHfqu5/LIR8aoa4khruC4Dm7Uof+JXpq6GkHCUt6wF1MLI88WXmTloF8X/lw8v+1xKcVmT5uyVeMI/wXghqreUtUU8OfAp3fc5tPAn2R//jLwUanFWQ8HLNYUpf+J3pyFUhzXoaG9nq4ifUNOTdgDpAOkfZeTsoGSqcIszr3aHvobrRi2h77U9+SFfsPY8L5Dv76ljksfe4RLH3uEMy8O4XouD6/OsDYfRyRcU/eJ/+QiZ983UrAW/zBeF+I5ezuRWuRbiClfNQJ/ELi/7ffx7GUFb6OqGWAJ6KzCts0OfRe7efSj5+l5pIvOs+2cfXGYRz48VnD5w9MS9poJwvbHf3MPXdplyEZBt/UdOkyF+u9shP7mrNyxs3mzcjeG2/YjUu9x7/IEqXiaIBMQZAI0UBbuL7LysPDEtsN6Xbh9TeV1dHOylVknpNnfcXbsnkER+byIXBaRywvJAqV1Zlf1rXUMPdXP6HNDtA4UXunotIQ9QObtaYLZbSf6diElTlAetMNuurY8uZJTtbUh8JWpqzN5l++rCdo++yVJQyS3h70Q/twUCZPJkbAMd7QV91xtzoyttmrU4U8Aw9t+H8peVug24yLiAa1AwdPuqvoq8CrA4x29Vou1i80Zt3uYUFJ+E7Tq9MU5SLqeIZhfL9kWOIcTBsjm/TXsvEnGR5pjh1LFc5itGFKJdNGSxvR67mIge22hUY3XhTfUgtvVgD8Ttj9wOhtwmqLhPqd9iLjFJ9ilfUgHYV+cGp9QVa5qBP5rwAUROUsY7J8F/rMdt/kq8OvA3wKfAb6ttdiqroqSaynu/2SC5enwa3lzdyPDzwxS11y6bcBpCnsATWSyq40UuUHEyakCcS924mRbK2g8TfpnD9H1zObiGO5oK+7ZwovCby7EMr+ORB3cviYktr+30GGFfmN7AyKFPw8b2us3f97fRLvqvC6kzsMbzm3qJ45AkedWUz7pt2fQhUQ4JOQI7oUOPFvKcFcVH85kx+S/AHwduAJ8SVXfFpF/JiKfyt7sD4FOEbkB/A9AXummKV8m5XPtmzdYfrgavpMVVqbXuPatm6TLWMJtY4hgu42hhO02uj8eZ9LgFS/r8xwivzBM5L39RJ7qI/qhUbzsxCsNlNTrk2G/9M1e7Ip/dwl/ajXsw7+S3JyZqn5A+ieTpH8+TXBvCf/mAqkfjuNP77ObJ4fTabOho576tvr8ckxXGHiiN+exDmJWtfoBwVwcfzZesl1GuVSV9OuTYdgr4b9dJsC/Noc/E6/48U+7qrRWUNWvAV/bcdlvb/t5Hfi1amzLwNydefxCy9j5AbM354q2XTiNJObhdDcSzMTzG3CdaQtPVhfofx/MJzYXQc+9QvHfmcXfqHTyHCLv6SGYS6Arya1vEgqoknl7FqejYd9DQZU2XSvHwK85vP03HvevpPEz0Nrl8J4PxegcnN68zUGEvT+zRubtbecJFNxHOzc/dHejiTTBWjrsetkUtrvWpWT4jWznZ3yg+LcWcLur04PntLJeOifQ6my84BKGGiirs7V3lONd6ibzbnYZRAAB90wb7kiJUCoUGtttPL++T/qnU+A6hYeNBIKZOG6RVs3l2G/Ttb146SK8qFqwgdjOWdVQhSP7RJrMWzN53778q3M4TdG8Redz7hto9kR8YnOoTRojRJ7qC7+RFfl3O+z5FSeRBf4JVNcYZSn7RsghUNdUfGHv00ocIfJoF3qhIzyJFy1+om/zPoUWSClGKfxtYOO6oDozQPfadA0g/Wb5H/CFGohVqwnaTpnx5cItqgPFv7+Mc6k7/7qN+96YD8N+24eFrqRIv/kQ73xH0dp9qbc42409QydQ17lOZm7O5ZXbiSN0n6/d6Q3iOuGReDm3bYkhzdFwav9u5QNB2LaXROHzI05HfcHL92NjiOe1b+aGPsS5llzKCf1g+t5mYFdit35J+yrHXS/eolrXM+HY/sO1cCjOc3AHm3Ha6lDVsFVyoZ5Qqyk06iB1XsEjffWVYHEdp+3gO6KeVBb4J1CsKcrZl0a58+N7mwdRIjD6/BB1xdZyNTlEhMjTfWSuz4Vr4wYKnkChtsKu4Aw0EdxbDo/0dfvlzVWv699L6Fcsr0Jr48RxZf2SpL0OZuP5we0I0hoj/dqDsMIqe30wvRZWSI22Fl+WUEBSPpFn+kn9/CEs7phkl8iQ/ukUkWf7NyuxTC4L/BOqtb+ZJz91ibW5OAo0dtYXnE1rSsgEOM0xpDmG01mPAKkfT+RP3nKdsGywv5nMvaWw137ExR1pKdmquRLlhn41HMREO7evCf/OYv4i5a5AQE7YA5sVUk5fY9hCodDi5grSGEWiLpFzHeG5lZ0fKNmTt87TfXve51pggX+CiSM0dTce9W6cSJlbC/jb1r31r4fr3Uae6SdzdXaz66a01RF5rGtzuChyoRMuHM4+7hb6j7ZVZ0cKhf0P/l0GL56hvWV/dfbiOUSfHyDz7jzB7BooOJ0NeI90kCoU1ACqBLMJ3PMd+Fdm86qunIEmJNtPJ1hOFl/G8pA7op4kFvim5qTvLBLcXsy73L8+j/PCANEXBsOaceHI+7dsD/0Pfszhy7cWN0N/a7y9chthH6OVy7+3TMtsEnGEtK5uVshIbG/NyyTmEXlP/vyOQvUGm1cAXl8TIpC5sRBWU0Uc3JHWnBnSEvPC1gsFqtUkarFWjI0BmJoSzMUJbi4UuVLxH6wA4RHqUYf9huJN16rz3/Yj+9f+cI3oXBJRNiejhRUyU1X7e5y+ptyVzDbJZh2929tE7APDRH/pDLEPjeKdyZ397HQ3FH4MR3DPtOZfbgALfFNjMtfnS98gdTwX2SjcdK3y5oI5LTT+g0/91DpS4CnQ1TTBWvHFZfbCHWkNSyh3LF/pnmnNOwFerIu6OELkmf5wGUw3bLKGQ3hepceGOYux7z6mpuhaick5Djid1SuxrLbCs3IrD/2Nvjg9dS2k/CIfiEI4vNJY+TwP8Rwizw8QTO8oy2zdW4WZ0xQl+oFhdDmJpgOc1lhVFjw/zSzwTW2JOOHkrEJiXsVHh5oJwqGQqFNy7d392hn6lcqrxKlzwxr6nVSRKk7qEzdcUL7SReVFBNnjB0Uts8A3NcUdbg3LBXdWiXhC5PmBfbfZ1fUM6Xdm0MXs4ioxj8hjXVWdlLVhe+hX6/E2uOeKVMj0Nu6rM6g/G8e/u4iu+zgtUdyz7Zt9cczhs8A3NcU90xo25Zpa3RxDlphH5OnefQ8HaKCkLj/IrR1fz5D+2UMiz/WX7BuzXwfVrtrrC3sC+TfnwyN9N1xtyi2xJnIxmWxX0c3JVesZgtkEkWf69jx8Y6rDAt/UFBEhcqkbHWsnWE0hURdpjlY0/BLMxIt33ry9iPPkyeheqpkAXVrHiXk4zw2ga2mkwcOp2/tMYs0EOWG/KVDSP50i+gsjh7LYjMllgW9qktR5uHXVefnrWqro0orB6t4rW9QPwr78D1bAV5zOerxz7Qe6NGPm/jL+jXlAt7qCZr8BOR31eE90o4vrZG4uoPE0EvNwz7aF69IW+huWk8UXKPeV9NvTRJ+y2bCHzQLfmApJvReWBRYIfachP6RVw9p2IO/bhWp4BLy9937wcI3UXILo+waRHR9Sqhr2iE+kkcbovnrIBAuJMOwLHI1DuHZA+vXJcCnI7GUaT5O5MouuZ/DOFDh57Jb+xqSzCTTtW1XNIbPAN6ZCTk8jXJ/PD/zsIizbBQsJ0m9Nb93WEbxL3bjZnjy6sB62ddg5QuQHZO4uEbm41Q1Vk2GzMN1Ym1ZBmqJE3tuHeE44RLOShKiLU6Kc0r+7VHzVMAgnXy0X+KaSHbJyh1vyJqlJS6zohyAArqDpwAL/kFngG1MBDRR/ciVs+LXRSdMBHAf3YkdOq15dz5B+42FuuPpK5ufTBP2N6GIyXJi7UEgqBFOr+J31YaM3EdI/n86bV6ArSdLvTOM0xcIg31hApN4j8lRvwWEhXd99WcyiJGxbvLM0UkTwnuwlc3my+F2rNKRmymfPuDH7pKqk35hCl5JbIS6EJZnPD+DsOHottShIMFHG2riZgMxb0+Gi3493bw4L5e4U6EwCf249dwGRtTSp16eIvn8o7wS1tMZKT0grRSl6lO621hGMtRPcXshtnuMI7lj7vktgzf7ZaXJj9imYTYQnJ7cfsSuQ9AkeruXfocTyfGXzNRw/v7lQ/KQoFB6iSfvownrexd6ZttJj7o5Ag5e/PQmXHpQC5ym2HrsV92Jn2AIBIObiXuzAG7F+N0fBjvCN2adgZq3w8EugBNNrMJRbKy8tMZhLlB4vL4eCzifCVW/2wlcyN+eJtOVOMJP6CJFn+8lcmwu/rcBWuDuCO9KKM9xC+o0p2D6WH3WJ7FJyKiJ4gy14gwczb8DsjQW+MftVopumFDhidgebdz9BWi4FZ6yV4PaOx3MkPJ9QZFxeV1Jkbi0QOd+Rc7nTHCP63AC6fcgpE4AXtojQlB9OLHPYOqGcDvBn43hDFuYnRUVDOiLSISJ/LSLXs/8vOB1PRHwReSP731cr2aYxx4XbX6TNrys4BY5oJeKG7RvaYuERtBD2rtnHULa0xoicacd9tHOz86Q0RvCe6CbyWFfxd7ZCML6cG+zbH1dk67+Iuznen7k5Dyk/t3ooUPzr8+GJZnMiVHqE/5vAt1T1d0TkN7O//88FbpdQ1acr3JYxx4rTEsMdbc09as/2nSnWddNpiBB9dgD1Nyp6hPTPsid+fc0P/0K57AjeI2F5ptffDAUakLmjbfgFFnkBwu0oe/qgCabjhfdFIJhLFJ2AZY6XSgP/08CHsz//CfAfKRz4xpxK3lg7Tm8jwcM1VBW3u7GsyU/b69YjT/ehi+v4cwnEc3Da68i8O781W9WR8JtAAE5rDPdMW15dvapm6/cVaY6Fw0d3FguHdJ1nFTI1qtLA71XVjULbKaDYGZw6EbkMZIDfUdW/qHC7xhwbTmMUZ2z/HSBFBGmvx2mvDxux/fD+ViO2jZWnEj6RZ/vzPkz8mTUy785vjdlnPyC8x7pw+psIptbyxvi9C3tvhOb0NBBMruZ/gOjxXkPA5No18EXkm0Chphe/tf0XVVURKXY2alRVJ0RkDPi2iPxcVW8W2d7ngc8D9DdU1ivbmJMmmFkr3ojtTm4jNv/hKpl3ZvPLQn0l884s3jN9SF0E//4SpAOk3sO90IG7j4XvvXMdpOYS4VoC24av3Ec6bLbsCbJr4Kvqy8WuE5GHItKvqpMi0g9MF3mMiez/b4nIfwTeCxQMfFV9FXgV4PGO3iqUMxhzcuhauqxGbKoaLvJdrOInUIL7y0Se6ME724aqVtQRVKIu0ReH8CdX0Pn1sJ5+sAWn2XrbnySVTrz6KvDr2Z9/HfjLnTcQkXYRiWV/7gI+ALxT4XaNOZWkIVJ0ElROI7ZAi5ZebtD41uzZaqy+JZ6DN9xK5KleIo92WdifQJUG/u8Avywi14GXs78jIs+JyBezt3kMuCwiPwO+QziGb4FvTAFOd0PhwHcE92xbzu8FS0K3kX10zjSnW0UnbVV1DvhogcsvA/8w+/MPgfdUsh1jaoW4DtFnB8LGaPH01knYi505q0SJCM5gM8HEcn5nTQBX8EatfYHJZTNtjTlmpCFC9H2D6HoGzQRhv5oCQzLe+Q4yiQzBfDw39BsjRC51H+iCKeZkssA35piSOq/k3ChxhMhTvWg8HZ7QjThIUzSvS6cxGyzwjTnhpCGCW6JjpTEbrD2yMcbUCAt8Y4ypERb4xhhTIyzwjTGmRljgG2NMjbDAN8aYGmGBb4wxNcIC3xhjaoQFvjHG1AgLfGOMqREW+DUofuv+5s/B9D0ANLF97Zo4AOv+1iLYDxPLh7JvxpiDY4FfY+5PnQHC0E+/GQZ78KPvAmHoP9p2AYDPjIUrKa37i3zwY+HLxELfmJPNAr8GWegbU5ss8GuUhb4xtccCv4btJfQ/M1ZvoW/MCWeBX+PuT53h/tSZ/NC/dTsn9CFuoW/MCWeBbwA2Q3/pL64B2eqdbOhfjG2sjboV+s+/HIa9hb4xJ4cFvtm0McSzM/SBvNAHLPSNOWEs8E0OC31jTi8LfJPHQt+Y06miwBeRXxORt0UkEJHnStzuFRG5JiI3ROQ3K9mmORwW+sacPpUe4b8F/H3ge8VuICIu8PvAx4FLwOdE5FKF2zWHYHvop9+M54V+WMGTG/of/JhjoW/MMVVR4KvqFVW9tsvNXgBuqOotVU0Bfw58upLtmsOzs1Z/e+hvlW3G8yZoPUwsW/Abc8wcxhj+IHB/2+/j2csKEpHPi8hlEbm8kEwc+M6Z3RUKfZuVa8zJs2vgi8g3ReStAv8dyFG6qr6qqs+p6nPtsfqD2ITZB2vFYMzJt2vgq+rLqvpEgf/+ssxtTADD234fyl5mThgLfWNOtsMY0nkNuCAiZ0UkCnwW+OohbNccgI3QBzZDf2NMH9gW+lvfzjZC3xhztCoty/xVERkHXgL+vYh8PXv5gIh8DUBVM8AXgK8DV4Avqerble22McaYvfIqubOqfgX4SoHLHwCf2Pb714CvVbItY4wxlbHv2sYYUyMs8I0xpkZY4BtjTI2wwDfGmBphgW+MMTXCAt8YY2qEBb4xxtQIC3xjjKkRFvjGGFMjLPCNMaZGWOAbY0yNsMA3xpgaYYFvjDE1wgLfGGNqhAW+McbUCAt8Y4ypERb4xhhTIyzwjTGmRljgG2NMjbDAN8aYGmGBb4wxNcIC3xhjakRFgS8ivyYib4tIICLPlbjdHRH5uYi8ISKXK9mmMcaY/fEqvP9bwN8H/p8ybvsRVZ2tcHvGGGP2qaLAV9UrACJSnb0xxhhzYA5rDF+Bb4jIT0Tk86VuKCKfF5HLInJ5IZk4pN0zxpjTb9cjfBH5JtBX4KrfUtW/LHM7v6CqEyLSA/y1iFxV1e8VuqGqvgq8CvB4R6+W+fjGGGN2sWvgq+rLlW5EVSey/58Wka8ALwAFA98YY8zBqPSk7a5EpBFwVHUl+/OvAP+snPu+szA9+/SXfncNsJO9oS7sudhgz8UWey622HMBo8WuENX9j5qIyK8C/xLoBhaBN1T1YyIyAHxRVT8hImPAV7J38YA/VdV/vodtXFbVoiWftcSeiy32XGyx52KLPRelVVql8xW2wnz75Q+AT2R/vgU8Vcl2jDHGVM5m2hpjTI04CYH/6lHvwDFiz8UWey622HOxxZ6LEioawzfGGHNynIQjfGOMMVVggW+MMTXiRAS+iPwLEbkqIm+KyFdEpO2o9+molNuh9LQSkVdE5JqI3BCR3zzq/TlKIvJHIjItIm8d9b4cJREZFpHviMg72ffGf3vU+3RcnYjAB/4aeEJVnwTeBf7JEe/PUdroUFpzM5VFxAV+H/g4cAn4nIhcOtq9OlJ/DLxy1DtxDGSA/1FVLwEvAv+oxl8XRZ2IwFfVb6hqJvvrj4Cho9yfo6SqV1T12lHvxxF5AbihqrdUNQX8OfDpI96nI5PtRzV/1Ptx1FR1UlVfz/68AlwBBo92r46nExH4O/xXwF8d9U6YIzEI3N/2+zj2xjbbiMgZ4L3Aj492T46nA++lU65yunKKyG8Rfn3714e5b4etSh1KjakpItIE/Fvgv1PV5aPen+Po2AT+bl05ReQ3gE8CH9VTPnmgGh1KT6kJYHjb70PZy0yNE5EIYdj/a1X9/456f46rEzGkIyKvAP8T8ClVjR/1/pgj8xpwQUTOikgU+Czw1SPeJ3PEJFxy7w+BK6r6fxz1/hxnJyLwgd8DmgkXT3lDRP7gqHfoqIjIr4rIOPAS8O9F5OtHvU+HJXvi/gvA1wlPzH1JVd8+2r06OiLyZ8DfAhdFZFxE/sFR79MR+QDwXwC/lM2HN0TkE0e9U8eRtVYwxpgacVKO8I0xxlTIAt8YY2qEBb4xxtQIC3xjjKkRFvjGGFMjLPCNMaZGWOAbY0yN+P8B3045Y1prUdoAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# License\n",
        "\n",
        "The MIT License (MIT) Copyright (c) 2020 Andrej Karpathy, 2022 Curtis Hawthorne\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"
      ],
      "metadata": {
        "id": "tPrVh6XCUkqS"
      }
    }
  ]
}