implicit_layers_ode.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a78ff981-3580-4741-93d2-592d4b47bac6",
   "metadata": {},
   "source": [
    "# Putting numerics into a neural network\n",
    "\n",
    "This notebook highlights some quick experimentation I did with putting arbitrary numerical solvers into neural networks, particularly for encoding solutions to ODEs and PDEs as a network structure. Note that this is different than a [neural ODE](https://arxiv.org/pdf/1806.07366.pdf) in that a neural ODE parameterizes the entire network as an ODE while here we want to specify a specific ODE and learn some parameters of it through the optimization.\n",
    "\n",
    "To get started I'll be using the `torch.autograd` package, particularly the `jacobian` function. This does exactly what you think it does - it takes the Jacobian of a function which is autodifferentiable (aka implemented in pure pytorch). Let's import some packages and start messing around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "994c99b1-91f3-4bb4-a48b-b612ba35e0f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "import xarray as xr\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "from collections import OrderedDict\n",
    "from typing import Optional, Tuple\n",
    "from functools import partial\n",
    "\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "from torch.autograd.functional import jacobian\n",
    "dtype = torch.float32\n",
    "\n",
    "# V will be our domain for testing functions\n",
    "v = torch.tensor(np.arange(-2.0, 2.0, step=0.1), dtype=dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0232885b",
   "metadata": {},
   "source": [
    "## Testing the `jacobian` on simple functions\n",
    "Just to get a feel for how the `jacobian` function works, let's look at some examples where we have analytic solutions. Here I show that the autodiff calculation of the derivatives of both ReLU and hyperbolic tangent are equivalent to their analytic counterparts.\n",
    "\n",
    "### ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "05958579",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:3: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    }
   ],
   "source": [
    "def drelu(x):\n",
    "    # Derivative of the relu function\n",
    "    return torch.tensor(x>0, dtype=torch.float32)\n",
    "\n",
    "torch_drelu_v = torch.hstack([jacobian(F.relu, vv) for vv in v])\n",
    "truth_drelu_v = drelu(v)\n",
    "\n",
    "assert np.allclose(torch_drelu_v.numpy(), truth_drelu_v.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d23813a",
   "metadata": {},
   "source": [
    "### Hyperbolic tangent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "56d56c38",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/torch/nn/functional.py:1794: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n"
     ]
    }
   ],
   "source": [
    "def dtanh(x):\n",
    "    return torch.tensor(1/np.power(np.cosh(x.numpy()), 2.0), dtype=dtype)\n",
    "\n",
    "truth_dtanh_v = dtanh(v)\n",
    "torch_dtanh_v = torch.hstack([jacobian(F.tanh, vv) for vv in v])\n",
    "assert np.allclose(truth_dtanh_v, torch_dtanh_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d9089b4-b5e7-4635-bfe9-910b567bd22d",
   "metadata": {},
   "source": [
    "## Testing the jacobian on a simple NN layer\n",
    "Great, those both work out of the box. Now can we take derivatives of simple neural networks? Let's find out. Here I'll define a basic feedforward type network, AKA a Multi Layer Perceptron. Rather than initializing weights randomly from a distribution I'm going to specify the values. That way we know what the derivative should be. I also note that our activation functions will be the ReLU and hyperbolic tangents that we know the `jacobian` function works on, but you could expand this.\n",
    "\n",
    "For the first test I'll just do the derivative of a layer with a single hyperbolic tangent activation. And for the second I'll do a two layer (each with single neurons) network of a ReLU activation followed by a hyperbolic tangent activation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3a09f68e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLPLayer(nn.Module):\n",
    "    \n",
    "    def __init__(self, width, activation=nn.ReLU, init=True):\n",
    "        super().__init__()\n",
    "        self.width = width\n",
    "        self.layer = nn.Linear(width, width)\n",
    "        self.activation = activation()\n",
    "        if isinstance(init, tuple):\n",
    "            self.init_parameters(*init)\n",
    "        elif init:\n",
    "            self.init_parameters(1, 0)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        X = self.layer(X)\n",
    "        X = self.activation(X)\n",
    "        return X\n",
    "    \n",
    "    def init_parameters(self, weight_value, bias_value=None):\n",
    "        if bias_value is None:\n",
    "            bias_value = weight_value\n",
    "        with torch.no_grad():\n",
    "            self.layer.weight[:, :] = weight_value\n",
    "            self.layer.bias[:] = bias_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "588464d5-5c0b-4581-a46e-b2b67ebd9e03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Single neuron with tanh activation')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA//0lEQVR4nO3dd3gUVdvH8e+dRgiE0HvvnST0IgKiAhaKoiCKoK+IiPVRQHlUrNh7xa4IqCCIiKIgRaQGCKH3FkINhJpAynn/2IEnxHQ2OZvN/bmuXNndmZ357WSz95yZ2XPEGINSSimVER/bAZRSSnk2LRRKKaUypYVCKaVUprRQKKWUypQWCqWUUpnSQqGUUipTWigKEREZJCJ/uGlZC0Tk/9yxLG8iItVF5LSI+GYyjxGRuvmZK9W6vxKRFyys123vvXSW/bGIPJUXy1YuWii8jIh0EpElInJCRI6JyD8i0hrAGPOdMeYa2xm9mTFmrzGmuDEmGS6/oIrIOBGZ6L6EeU9EajrF0O/CY+5674nIEBFZnPoxY8xwY8zzl7tslTG/rGdRBYWIlABmAfcBPwABwBXAOZu5bBERP2NMku0cShV02qLwLvUBjDGTjTHJxph4Y8wfxpgo+PfemLPXN1xEtonIcRH5QETEmeYrIm+IyFER2SUiI9PuJaYmIneJyCZnOXNEpEYG813Y27xTRPY6yx+barqPiIwRkR0iEisiP4hIaWdaFxGJTrO83SLS3bk9TkSmishEETkJDBGRyiIy02ldbReRe1I9d5yz/G9E5JSIbBCRVhnkflZE3nNu+4vIGRF51blfVEQSRKRU6r1pEXkRV6F+3zkc9X6qRXZPb7unWWcP4EngVuf5a53Hhzrb+pSI7BSRe1M9p4uIRIvIf0TksIgcEJGhaRZdSkR+dZ6/XETqpPeaneX9KCIHnRbqIhFpkmpaUec9sseZvlhEigKLnFninNztU7/3xHWo6PU06/lZRB51bl/4+58SkY0i0td5vBHwMdDeWW6c8/glh9NE5B7nb33M+dtXTjUtw/e8yoQxRn+85AcoAcQCXwM9gVJppg8BFqe6b3C1QEoC1YEjQA9n2nBgI1AVKAXMdeb3c6YvAP7Pud0H2A40wtVK/S+wJIOMNZ3lfAoUBVrgavE0cqY/DCxz1lsE+ASY7EzrAkSnWd5uoLtzexyQ6OTxcZa/EPgQCARCndd4Var5E4BegC8wHliWQe5uwDrndgdgB7A81bS1aV7fv7ZTdrZ7OusdB0xM89h1QB1AgCuBs0B4qm2UBDwH+Duv7eyF9wLwFXAMaOP8rb4DpmTynroLCHb+Fm8DkammfeC8virO9uvgzHfJNkj73gM6A/sAce6XAuKBys79/kBl5294K3AGqJTeezjVa3oh1d/iKBDuZHkPWJSbba8///vRFoUXMcacBDrxvw/iI84eVYVMnvayMSbOGLMXmI/rwxTgFuAdY0y0MeY48HImy7gXGG+M2WRch3peAkIzalU4njWuFs9aYC2ugnFhWWOd9Z7D9UF5c0YtmXQsNcbMMMakAGVxbY/RxpgEY0wk8BlwR6r5FxtjZhvXOYVvU+X413KBeiJSBtcH3edAFREpjuvDemE2812Q0XbPkjHmV2PMDuOyEPgDV8vlgkTgOWNMojFmNnAaaJBq+k/GmBXO3+q7zNZtjPnCGHMq1d+ihYiEiIgPriLykDFmv3G1YJc482Xlb1zv0QuZb8b1d4tx1vmjMSbGGJNijPke2IarsGXHIOALY8xqJ8sTuFogNVPNk+ttX1hpofAyzof1EGNMVaAprj2ztzN5ysFUt88CxZ3blXHt9V2Q+nZaNYB3RCTOORxwDNfebpVcrLcGMD3VsjYByUBmxS611DkrA8eMMadSPbYnTa60OQLTK0rGmHggAldR6IyrMCwBOpK7QpHR68+SiPQUkWXOoZU4XK2GsqlmiTWXnptJu/xsrVtchx9fdg4DncTVesNZV1lcrbQd2c19gXHt2k8BBjoP3YarYF1Y72ARiUz1HmjKpa8vM5Vx/Y0vrOs0rlZ2Zn/zbG/7wkoLhRczxmzG1SxvmounH8B1+OeCapnMuw+41xhTMtVPUWPMklysdx/QM82yAo0x+3Edggi6MKO4LkEtl+b5qbtDjgFKi0hwqseqA/tzkQtcxaAbEAasdO5fi2tvd1EGz7nc7pkveb6IFAGmAa8DFYwxJYHZuAqzu90G9Aa6AyG4DinhrOsorsN26Z3fyM5rnoyrpVgDaIvrNeHc/xQYCZRxXt96/vf6slp2DK6dDZzlFQPKkPu/uUILhVcRkYbOScyqzv1quPbaluVicT8AD4lIFREpCYzOZN6PgScunOh0Dk30z8U6LyzrxQuHrUSknIj0dqZtxbXHf52I+OM6F1IkowUZY/bh2usfLyKBItIcuJtUe685tBAYDGw0xpzHOf8A7DLGHMngOYeA2rlc34Xn13QO9YDrSrYiuI6tJ4lITyCvLnkOxnX+KBZXgX7pwgTn0N4XwJviumDA1zlpfSFbCpm8bmPMGme+z4A5xpg4Z1IxXMXgCLhO3HPpjs4hoKqIBGSw6EnAUBEJdbK8hOtc0u6cvHB1KS0U3uUUrr2z5SJyBleBWA/8JxfL+hTXse8oYA2uvdYkXIeBLmGMmQ68AkxxDlGsx3UyPTfeAWYCf4jIKVyvoa2znhPACFwfLhdaGNEZLOeCgbj2hGOA6cAzxpg/c5ltCa4T5BdaDxtx7VVn1JoA1+u52bnC5t1crPNH53esiKx2DqM9iKuQH8e11z8zF8vNjm9wHcbZj+u1pt3heAxYh6t1dQzXe8DHGHMWeBH4xzl81C6D5U/G1VqZdOEBY8xG4A1c54QOAc2Af1I95y9gA3BQRI6mXaAxZh7wFK4WygFcLZ4B2X/JKj0XrjpQKlPOnuvHxpjMTlArpbyQtihUupxr5HuJ6/sAVYBncO2RK6UKGW1RqHSJSBCuY/INcV3j/iuuSyFPWg2mlMp3WiiUUkplSg89KaWUypTXdQpYtmxZU7NmTdsxlFKqQFm1atVRY0za7yUBXlgoatasSUREhO0YSilVoIjInoym6aEnpZRSmdJCoZRSKlNaKJRSSmXK685RKKW8V2JiItHR0SQkJNiOUmAFBgZStWpV/P39s/0cLRRKqQIjOjqa4OBgatasiQ5Ml3PGGGJjY4mOjqZWrVrZfp61Q08i8oW4hmpcn8F0EZF3nSENo0QkPL8zKqU8S0JCAmXKlNEikUsiQpkyZXLcIrN5juIroEcm03sC9ZyfYcBH+ZBJKeXhtEhcntxsP2uHnowxi9IMT5hWb+AbZzSsZSJSUkQqGWMO5E9CpfLGkZjd7F7xK/HJQqVOd1C3fHH98FMezZOveqrCpcNaRpPB0JoiMkxEIkQk4siRjMaPUcqOs+eTiFw4nWUf3cvu55pRbkILWkc+yaE1v3H1W4to99JcFr8zlIhfPiH2UFbDayhPMH36dESEzZs353oZQ4YMYerUqZnO89JLL11yv0OHDrle3+Xw5EKR3i5Wuj0YGmMmGGNaGWNalSuX7jfQlcpXcUcP8tfkNxk4YRmhz/7JqbmvE3ZwGqf8y7Cs9oPs6Debdg9+zcv9mtG9agpNj/9Bq1WjKPNRE3Y8H8qyj0ewZ9Mq2y9DZWDy5Ml06tSJKVOm5Ol60haKJUtyM7rw5fPkQhHNpeM0V8U1SplSHm372sXEv9+Jzpuf5/zZOIZ0rEmRfh9gRu+m2RMLaDf4eeo070i18qUZ0KY6L955LcH/3cO23r+wrOb9JPgGE35gCm9Omsn0NdrC8DSnT5/mn3/+4fPPP79YKBYsWECXLl24+eabadiwIYMGDeJCz9zPPfccrVu3pmnTpgwbNoy0PXbPmzePvn37Xrz/559/0q9fP8aMGUN8fDyhoaEMGjQIgOLFi1+c79VXX6VZs2a0aNGCMWPG5Olr9uTLY2cCI0VkCq6hME/o+Qnl6VbOeJ9ma8YRJyHs7D2daeFdsvU8Xz8/6oV1pl5YZwCOxMZycOpWHvl+Lcc3LeKOm2/CPyDD4cELpWd/2cDGGPcOj9K4cgmeuaFJpvPMmDGDHj16UL9+fUqXLs3q1asBWLNmDRs2bKBy5cp07NiRf/75h06dOjFy5EiefvppAO644w5mzZrFDTfccHF53bp14/777+fIkSOUK1eOL7/8kqFDh3LDDTfw/vvvExkZ+a8Mv/32GzNmzGD58uUEBQVx7Ngx922EdNi8PHYyrnFxG4hItIjcLSLDRWS4M8tsYCewHdf4zSMsRVUqS+eTUljywT20jhzL9sAmBIxYRP1sFon0lCtThon/15ZHWhdl0JYH2PZaN44e3Ou+wCrXJk+ezIABrmG4BwwYwOTJkwFo06YNVatWxcfHh9DQUHbv3g3A/Pnzadu2Lc2aNeOvv/5iw4YNlyxPRLjjjjuYOHEicXFxLF26lJ49Mx9yfu7cuQwdOpSgoCAASpcu7eZXeSmbVz0NzGK6Ae7PpzhK5drhkwmM+G41DWOK4VPtNlrd/Q5+/gGXvVx/Xx8euqkbEUXG02TlWE59fCWbr/uUhq27uyF1wZfVnn9eiI2N5a+//mL9+vWICMnJyYgIvXr1okiR/7X4fH19SUpKIiEhgREjRhAREUG1atUYN25cut9huNCCCAwMpH///vj5Zf7RbIzJ1yvlPPkchVIeb/PyP3j5nbfYEHOStreMot3wj9xSJFJrdf0wDvSfRaL4U3vWLSz/8fV/HedW+WPq1KkMHjyYPXv2sHv3bvbt20etWrVYvHhxuvNfKAply5bl9OnTGV7lVLlyZSpXrswLL7zAkCFDLj7u7+9PYmLiv+a/5ppr+OKLLzh79iyA9x56UqogMykpLP/+FerMHsBw8wMzRrTnhhaV82x9tZu2pfgDi9lUNJxla9czeloUCYnJebY+lb7JkydfcuIZ4KabbmLSpEnpzl+yZEnuuecemjVrRp8+fWjdunWGyx40aBDVqlWjcePGFx8bNmwYzZs3v3gy+4IePXpw44030qpVK0JDQ3n99dcv41VlzevGzG7VqpXRgYtUXlv+/cu03TSetUXbUnPYJEJKlc2X9SYnJfHOvK28O38XAxr5M/6O7ohP4dnf27RpE40aNbIdI0+MHDmSsLAw7r777jxfV3rbUURWGWNapTe/J1/1pJRH2rj0N8I3vkpkUDuaPzYbH1/ffFu3r58fj17bmEocpeeSASz7dijt73wx39av8kbLli0pVqwYb7zxhu0o6dJCoVQOxMTFs/yP7wn2qUDteyfla5FIbcDVnVi9vi1td35A1Pwwmne92UoO5R6rVnn2lysLT5tVqcuUkJjMfRNX8UbKQBLv+pMSJctYyyI+PjS69yt2+9ag5sIH2b9zQ9ZPUiqXtFAolQ0mJYXFnzxEwv51vHlLC2pXq2o7EkHFQwi8YwoGOD9xIGdOxdmOpLyUFgqlsmHFj6/S/ei3PFN3J9c0qWg7zkWVazVib7cPWJtYlSdnbNDLZlWe0EKhVBY2XTh5XbQd7Ya8bDvOvzTr3JdD3d/j5w1xfLJgm+04ygtpoVAqE4eid1B+zr0ctHzyOivDOtfmjkY+dJnfj6gF02zH8Xo56Wb87bffvvjFuPzUpUsX3PVVAS0USmUgITGZiIlPE2jOkXLLRKsnr7MiIjxxUwcC/HypueABPbmdx3LSzbg7C0VSUpJblpNTWiiUysDTP6/n4bhbWHv1JGo0amk7TpaCiodQ5Pb/ndw+e/qE7UheKaNuxq+//vqL84wcOZKvvvqKd999l5iYGLp27UrXrl0BV5Fp1qwZTZs2ZfTo0Ref8/nnn1O/fn26dOnCPffcw8iRIwHXAEePPvooXbt2ZfTo0axYsYIOHToQFhZGhw4d2LJlCwDx8fEMGDCA5s2bc+uttxIfH++216zfo1AqHcv/mc+ciMPc2zWUDp0a2I6TbVVqN2Jd1/dpNn8oy74dTbv7PrYdKW99ed2/H2vSB9rcA+fPwnf9/z099DYIGwRnYuGHwZdOG/prlqvMqJvx9Dz44IO8+eabzJ8/n7JlyxITE8Po0aNZtWoVpUqV4pprrmHGjBm0adOG559/ntWrVxMcHEy3bt1o0aLFxeVs3bqVuXPn4uvry8mTJ1m0aBF+fn7MnTuXJ598kmnTpvHRRx8RFBREVFQUUVFRhIeHZ/lasksLhVJpnD1zkup/3sM3xSrQuPvftuPkWLMr+7F87XSKHljBxuhYGlf13ENmBdHkyZN5+OGHgf91M37ddekUrHSsXLmSLl26cGEkzkGDBrFo0SIArrzyyovdhffv35+tW7defF7//v3xdc6PnThxgjvvvJNt27YhIhc7DVy0aBEPPvggAM2bN6d58+aX/2IdWiiUSmPtxLG05whx3d/D37dgHp1tMPhdrnl3OVVmbmba8A74+ORfl9T5KrMWQEBQ5tOLlclWCyK1jLoZv/HGG0lJSbk4X3pdiQMZXr6c1WXNxYoVu3j7qaeeomvXrkyfPp3du3fTpUuXi9PyquvxgvlfoFQe2bUxglYx37GiZC8atc988BhPVrJkKUZf15Sde6OZN2e67TheI6NuxgE2btzIuXPnOHHiBPPmzbv4nODgYE6dOgVA27ZtWbhwIUePHiU5OZnJkydz5ZVX0qZNGxYuXMjx48dJSkpi2rSMr1w7ceIEVapUAeCrr766+Hjnzp357rvvAFi/fj1RUVFue91aKJRypCQnkzD9Qc5IUeoNetN2nMvWL7wKn5T8hjbLRxJ7SMfedofMuhm/5ZZbLnYJHhYWdnH6sGHD6NmzJ127dqVSpUqMHz+erl270qJFC8LDw+nduzdVqlThySefpG3btnTv3p3GjRsTEhKSboZRo0bxxBNP0LFjR5KT/9fV/H333cfp06dp3rw5r776Km3atHHb69ZuxpVy/LR0M8Vm30/psBtp3e8h23HcYs/m1VSa3J21JbvT+pEfbMe5bN7czfjp06cpXrw4SUlJ9O3bl7vuuutfRcldctrNuLYolAKOnTnP83/u5bMqz9Oy9wO247hNjYbhrKo6mNYn5rD+n19sx1GZGDduHKGhoTRt2pRatWrRp08f25Eu0pPZSgF/f/scpRNq8UKfW/ApoCewMxI26AX2vzabEvNGc67lVRQJDLIdSaUjr0epuxze9R+hVC5sXPY7vQ++xws1ImlQMdh2HLcLDCpObOeX2JJYkW8XFvxvbHvb4fL8lpvtp4VCFWrnz50j6I/HOUA5Wgx6wXacPNO8681Mb/gar/19lD2xZ2zHybXAwEBiY2O1WOSSMYbY2FgCAwNz9Dw99KQKtVXfv0D7lL1EdvqYSsXTv8rEWzx9fRN2bVnHmq9HUf3h9wrkWNtVq1YlOjqaI0eO2I5SYAUGBlK1as7GU9FCoQqtmN1bCN3xMWuKdSSs+0DbcfJcxZBAnmkYTbutE1k9pxXhPYfajpRj/v7+F7+3oPJPwdulUMpNxi84yBSuodKAd2xHyTet+o9iu28dqi5/TkfEU9mmhUIVSit2HeOXzac52+VZKlavZztOvvHzDyCpx6uU5xhRU8fbjqMKCC0UqtAxKSnE/XA/PYtt5a6Ohe8wRsPW3YkM6kDT3V8Td/Sg7TiqANBCoQqdqAU/cE38bIY0SKJogGeOWJfXSt7wPFOSu/H5kr22o6gCQAuFKlRSkpMpsXg80VKR8D4P2o5jTc1GrdjcfDSfrIjlwAn3DXCjvJMWClWorP71U2ql7OZgq8fwDyhiO45VD3evRweiWDN5nO0oysNpoVCFxvlzCVRa8yY7fGsT3uMu23Gsq1Y6iOGVt3PNgQns3RppO47yYFooVKHxQ8Q+Pjrfi5Odx+HjWzjPTaRV76ZnOEcAR2c+bTuK8mBaKFShcPZ8Em/P38PW6rcS2vlG23E8RpkKVYmqfjvhpxeybc0i23GUh9JCoQqFFT++zjXxsxl9bf08Gy6yoGp681iOE0zC78/YjqI8lLVCISI9RGSLiGwXkTHpTA8RkV9EZK2IbBCRgtffgPIIJ2IPEb7tHfqHbKJlzTK243ic4JDSrGv0CD+cbs6SbYdtx1EeyEqhEBFf4AOgJ9AYGCgijdPMdj+w0RjTAugCvCEiAfkaVHmFjVOfo7iJp+T1z9mO4rHa9HuYecVv5JU/tmnPrOpfbLUo2gDbjTE7jTHngSlA7zTzGCBYXMcJigPHgKT8jakKusP7dxEW8z2rSl5NrcatbcfxWIH+vjzSrTaNYn5i1bwfbcdRHsZWoagC7Et1P9p5LLX3gUZADLAOeMgYk5LewkRkmIhEiEiEdj+sUts17Wl8SKFKn+dtR/F4fcOrMKLIb5Rb8hxJiedtx1EexFahSO9sYtr27rVAJFAZCAXeF5ES6S3MGDPBGNPKGNOqXLly7sypCrCdR07zyaFGzK86gsq1GtqO4/H8/AOIbTOKGin7WD3rY9txlAexVSiigWqp7lfF1XJIbSjwk3HZDuwC9L9dZdvbc7exzDec8AFP2Y5SYIReM5htfvWovvZdzp9LsB1HeQhbhWIlUE9EajknqAcAM9PMsxe4CkBEKgANgJ35mlIVWHu2RNJgw1vc07oM5YILd1cdOSE+PsR3HENFjhA56yPbcZSHsFIojDFJwEhgDrAJ+MEYs0FEhovIcGe254EOIrIOmAeMNsYctZFXFTyHZ7/IXb6/MbhNJdtRCpxmV/ZjbpGrmbzVkJic7mlBVchYGwrVGDMbmJ3msY9T3Y4BrsnvXKrgi96xgbC4uURUvJV2FXI2NrBytSro/QHTv4mgY2QMN7fUbVjY6TezldeJmfUiyfhSt/cTtqMUWFc1Kk+bCsLhOa+TnKRXpRd2WiiUV4nZvYWwY78TWb43ZSvXsB2nwBIRnmh8hBHnv2TN71/YjqMs00KhvMqkZbuYZ1pSU1sTl61F99vZ5VODcqvfJSU52XYcZZEWCuU1DpyIZ8I6w99hb1Khal3bcQo8H19fjrV8kBop+1gz5xvbcZRFWiiU11gy/WOqmxju61LHdhSvEXrtEPb6VKFUxNvaqijEtFAor3A0Zg/X73qR8eXnUbVUkO04XsPXz4/DoQ+wM7EU86O2246jLLF2eaxS7rT95/G0IpnKN461HcXrhF53L1dtaUjw4sN0C9XxPAojbVGoAi/2UDQtDk5jTcnuVKndxHYcr+Pn68P9XetyKmYrK5ctsB1HWaCFQhV4W2e8QhESqXCdtibySt/QSkwMfI2QeaMxKfpt7cJGC4Uq0I6fOc/6mBOsCLmW6vVDbcfxWv5+fsQ0/j8aJG1h/d8zbMdR+UwLhSrQvvhnFy+eu5XSt31qO4rXC71hBIcog9/i17RVUchooVAF1onjR1n/z2x6Nq1I/YrpDlWi3KhIYBC7G95Do8SNbFj6q+04Kh9poVAF1sYZr/GljOPRcL0KJ7+06P0gMZQnYvk/tqOofKSFQhVIZ07F0WjPRCKD2lOvcbjtOIVGYNFi/N51FuMOX8Hqvcdtx1H5RAuFKpDWzXyXkpwmsNvjtqMUOre2q0PJIH9+mvOX7Sgqn+S6UIhIMRHxdWcYpbLjXMJZam/7kg0BLWjY6irbcQqdYkX8eK3OOl7Yfxc71y+3HUflg2wXChHxEZHbRORXETkMbAYOisgGEXlNROrlXUyl/mfe34vxNYmkdPqP7SiFVpset3PGBHJsziu2o6h8kJMWxXygDvAEUNEYU80YUw64AlgGvCwit+dBRqUuSkpO4eU1/gwv9zVNO91gO06hFVKmAlGVbybs5F9Eb19vO47KYzkpFN2NMc8bY6KMMSkAIpJsjDlmjJlmjLkJ+D5vYirlMm/5aqKPneaerk1cQ3Yqa+r1Hk0SfsT8Ot52FJXHsv2fZoxJTOdhycY8SrlFSnIy9efexcTi79G9UQXbcQq9shWrE1nuBmoeW8yB2GO246g8lGWhEJF5IvK6iAwUkfppJhtnHj2prfLc2nmTqZWyh4AWN+Hjo9+d8ARV+z5P98Q3+XTJQdtRVB7KTotiHlDSmfd2EZmczjwTRCQIQEQ6uy+eUi4mJYViK95hv1QgtMdQ23GUo0qVqlwdWpfJK3YRezzOdhyVR7IsFMaYl4DngXbAJmPMwHRmexr4XES+BVq7N6JSsP6fX6iftJXoxvfi5x9gO45KZUSnSvwsj7N16rO2o6g8kp1DT9cDtwEpQL8MDjM9D2zBdSjqB7cmVAo4tuRbjlCK0Bvusx1FpVGncnlOB9ehyf4pnIyLtR1H5YHsHHr6AGiG6xLYscaY9AbOHWWMGQfcBzzjvnhKwao9xxl6/E7mt/+SIoE6zKknCr56FCU4y4af37QdReWB7Bx6qgGMAs6S8TmKIBFpCSQB97o3oirsJvy1kZCgIlzX5QrbUVQG6oVeQVRgKxrs+ob4M6dsx1Fult3LY32AaODFNOcoRERG4PoS3n+Bz4C73BtRFWY71i3npd23MrZpHMWK6BDvnsyvy+OU5iSrZ31iO4pys+yco7ikEIjIPWlmaQc8BPgBdwK93B1SFV5xf7xMIIlc3aWr7SgqC43b9eDFUs8xZkdzzifpwEbeJDstivZkXAiMMWawMeY88DbwITDX3SFV4bRv+zpCT84nqvLNhJQuZzuOyoYOPQay72QiM1ZH246i3Cg75yjuSFUIPuLSQiCp5ptnjBlujPnA7SlVoXRg1ksk4ke93qNtR1HZ1KV+OUaWWUWj328hOSnJdhzlJjnpLOcvY8y9qQuBMeaS54uIfl1WuUVM9G7Cjs9hbfnelK1Y3XYclU0iwlXNa9AsZRNrfv/CdhzlJjnqPVZEHhCRS/5rRSRARLqJyNe4Dk0pddk+ijjN4OT/UqP3WNtRVA616H47u3yqU3b1e6Qkp3c1vSpoclIoegDJwGQRiRGRjSKyC9gGDATeMsZ8lQcZVSFz6GQC30fso2Z4dypWrW07jsohH19fYsMfoGbKXiLnTrIdR7lBTnqPTTDGfGiM6QjUAK4CwowxNYwx9xhjInOyYhHpISJbRGS7iIzJYJ4uIhLpDI60MCfLVwXXtkmjeFK+4r7OdWxHUbkUeu1Q9kllSqx4C5OiV0AVdLnq0N8Yk2iMOWCMicvN851uQD4AegKNgYEi0jjNPCVxXUV1ozGmCdA/N+tSBcuxw/sJPzCFJqUN1csWsx1H5ZKfvz87Wz/D2LO3sWDrEdtx1GWyNfJLG2C7MWanc0XVFKB3mnluA34yxuwFMMYczueMyoItP79KIOcp3+sJ21HUZepw7S1Eh4Tz3l/bMcbYjqMug61CUQXYl+p+tPNYavWBUiKyQERWicjgjBYmIsNEJEJEIo4c0b2XgurEsSM0i/6eyODO1GgQZjuOukz+vj480LEcN8S8w4Ylv9qOoy5DjguFuNwuIk8796uLSJucLiadx9LucvgBLYHrgGuBp9IZOMn1RGMmGGNaGWNalSunX8wqqDbNeI3iEk/ItU/ajqLcpE+r2lzvtwIWvWY7iroMuWlRfIjr29oX+nw6het8Q05EA9VS3a8KxKQzz+/GmDPGmKPAIqBFzuOqguD0uSRe2teEyaXvp06zdrbjKDcJLFqMHfXuoum5SDav+NN2HJVLuSkUbY0x9wMJAMaY40BOR5JZCdQTkVoiEgAMAGammedn4AoR8XNGz2sLbMpFXlUATFy2h6j4sjTqO8p2FOVmzXs/xHFKcO6vV2xHUbmUm0KR6Fy1dGG87HK4BjXKNmNMEjASmIPrw/8HY8wGERkuIsOdeTYBvwNRwArgM2PM+lzkVR4u/swpqi54hAE1TxNaraTtOMrNgoqHsLnWHbRIWMm2yL9tx1G5kJt+m98FpgPlReRF4GZcPcvmiDFmNjA7zWMfp7n/GqAHN73c2pnvcr1ZSO3Q+21HUXmkaZ/HmPrWZtasPs6LobbTqJzKcYvCGPMdroGMxgMHgD7GmB/dHUwVDucSzlJ7y2dsDGhG43bX2o6j8khwSGn2dXyZ77b6suWgDmxU0OTmqqdHgNPGmA+MMe87h4iUypXIXz6iPMdI7vQf21FUHhvasSYtA/ay/qeXbUdROZSbcxQlgDki8reI3C8iFdwdShUOiefPUW3jJ2z1q0/TTmm/b6m8TcmgAB6vsoE+hz5g37a1tuOoHMjNoadnnS417gcqAwtFRAcrUjn28+rdTE1sz5kOoxAfW9/9VPmpXp8xnCOAQ788bzuKyoHL+e88DBwEYoHy7omjCovzSSm8tWA/8yreQ2jXm23HUfmkTIWqrK3cn/ATc9mzebXtOCqbcnOO4j4RWQDMA8oC9xhjmrs7mPJuS3/9ikYnF/NI93roeFeFS8N+/yWeIhz99TnbUVQ25eby2BrAwzntVlypCxLiz9BwzQs8Vqw8Dern+MpqVcCVKleJRdWHErXrAMUPnKRBpRK2I6ks5OYcxRgtEupyrP35XSoQS1LnJ/TcRCHVfOBzfOJ7G2/P22Y7isqGbP+Xishi5/cpETmZ6ueUiJzMu4jKmyScPU3tzZ+wIaAZTTreYDuOsqRkUAB3dazJuY2z2b5+he04KgvZPvRkjOnk/A7OuzjK20XOeIt2HOdwlw+1NVHI3dWmHLLkI3bMXgJNf7MdR2UiNyez/9WzV3qPKZXW2fNJ/LI9iUVBV9OkQy/bcZRlISVLs6HGHYSdXcL2NYtsx1GZyM0u3dXpPNbzcoMo7/ft0j18d7YNQbdMsB1FeYim/UYTR3HO/KHfq/BkOTlHcZ+IrAMaiEhUqp9dwLq8i6i8wemTxzm64CO61QuhVc3StuMoDxEcUprNtYbSIn4FmyPm2Y6jMpCTFsUk4AZc40bckOqnpTFmUB5kU15k/U+vMdZ8ypiwJNtRlIdp1u8xNlOLWcs22I6iMpDtQmGMOWGM2W2MGQicBCrg+k5FUxHpnFcBVcF3Ki6WRru/Ym3RttQP72I7jvIwxYJL8nfXabwfXYeVu4/ZjqPSkZuT2f+Ha1jSOcCzzu9x7o2lvMn6n14hhDMU6/G07SjKQ93eviaVivmw+OfPbUdR6cjNyeyHgNbAHmNMVyAMOOLWVMprnDh2hCZ7v2VNUEfqtuhkO47yUEUDfHm93noeOf4C6//5xXYclUZuCkWCMSYBQESKGGM2Aw3cG0t5i58WrWZnSiVCej1jO4rycC17j+AwpfFZ8DImJUejK6s8lptCES0iJYEZwJ8i8jMQ485QyjscOXWO11cbPq3/KbWbtrUdR3m4wKLF2NVoOI0T1xO1YKrtOCqV3PT11NcYE2eMGQc8BXwO9HFzLuUF5k2bQNGkE/znWm1wquwJ6/MQ0VKJEotfIDlJr5DzFLkaClVEqgIYYxYaY2YaY867P5oqyPZujeTmXU/zbpW51C5X3HYcVUAEFAnkcJvRnE4Sfl2qo+B5itx0M35hKNRjwBRgqjHmkHtjqYIu9uexlCGABjfruQmVM2HX3slNO+uy/+/jXN02maIBvrYjFXo6FKpyu83L5xB2ZjHrag6hTIWqtuOoAkZ8fHjiuiacO3mUP36ZbDuOQodCVW5mUlKQP5/mMKVpcYsOSqRyp3XN0nxS9geuinqM2EPRtuMUejoUqnKrPyN3sONcCXY3e4iixbRHepV7lW58mkDOsf1H/aKmbToUqnKbxOQUXpoXTUDp//Jbnytsx1EFXPX6oSwveyPhR2awb9vDVKun+6O26FCoym0WzvoO32PbeKJnI3x9dVAidfnq9H+eRPw4MmOs7SiFmg6Fqtzi1IljtFzzJG8HT6JLg3K24ygvUbZiddbWGMKhk/Gs3qkXV9qSk95jO4mIAE2MMSVS/QQbY0rkYUZVAGz44VlKcZKiPZ/D9TZRyj1aDHqRpwPH8NKcHRhjbMcplHJ0fMC4/krT8yiLKqAOR+8kNPo7VpW4irph2uO8cq+gIv48enV9ju3dwPKFOra2Dbk5kLxMRFq7PYkqsHZPHYsPhkp9X7IdRXmp/uFV+LTo+1RZ+B8Sz5+zHafQyU2h6IqrWOxwhkJdJyJR7g6mCobNB06w+qgvKyvfRuVaDW3HUV7Kz8+X0x3GUM3EsHr6O7bjFDq5uTy2p9tTqALJGMOLszez1u8OFt3R1XYc5eWad7uVDSs+oN6m9zhxbCghpfWiifySmxbFXuAK4E5jzB7A4BoWVRUyK+f+SMqO+Tx6dX1KBgXYjqO8nPj4EHj9q4SYU2yeNMp2nEIlN4XiQ6A9MNC5fwr4IKcLEZEeIrJFRLaLyJhM5mstIskicnMusqo8cuZUHDX/Gc1zRb/n9rbVbMdRhUSd5h1YXuFW1hxMZH10nO04hUZuCkVbY8z9QAKAMeY4kKPdSRHxxVVcegKNgYEi0jiD+V7BNS638iBRk8ZSnmMkXvsafn65OYKpVO40GfIenwUO4b8/byAlRS+XzQ+5KRSJzge4ARCRckBOxy1sA2w3xux0xrKYAvROZ74HgGm4OiBUHmL35tW0ipnMypK9aNimu+04qpAJCQrgyV4NCdq/mCWzvrQdp1DITaF4F9d3KSqIyIvAYiCn10VWAfaluh/tPHaRiFQB+gIfZ7UwERkmIhEiEnHkyJEcRlE5YVJSOPPTw8RLIHVue912HFVI9Q2tzH+L/UKT1U8Td/Sg7TheLzd9PX0HvA58gmus7D7GmB9zuJj0vrqbtg35NjDaGJOcjUwTjDGtjDGtypXTKyHy0sy1+5l8Jpz1zcZQunyVrJ+gVB4QHx+K9nmDYHOGLZMetx3H62X74LLTfcczwEhcH/Q+QBJQBnguh+uNBlKfAa2Kq+ik1gqY4nQHURboJSJJxpgZOVyXcpNTCYm8OHsLFSvdzLN9O9qOowq5Wk3asqziLbQ5+D1bVy+gfngX25G8Vk5aFA8DHYHWxpgyxphSQFugo4g8ksP1rgTqiUgtEQkABgAzU89gjKlljKlpjKkJTAVGaJGwa9U3Y+hydg7P926Kr4/256Tsa3LbeGKlJPLrf0hOSrIdx2vlpFAMBgYaY3ZdeMAYsxO43ZmWbcaYJFwtkznAJuAHY8wGERkuIsNzsiyVP3ZtWE6n/V9wU4WDtKhW0nYcpQAIDinNrjbP8nXCFUxeudd2HK+Vk+sa/Y0xR9M+aIw5IiL+OV2xMWY2MDvNY+meuDbGDMnp8pX7mJQU4mc8yikpRoPbXrMdR6lLtO45mLeiGzBzzjZ6NqtCmeJFbEfyOjlpUZzP5TRVwEXM/JjGievZ1uw/lCxb0XYcpS4hIjzXuwk9kv5i3ZcP2o7jlXJSKFqkGbDo4sBFQLO8CqjsOnEijtqRL7PZryGt+ug/ofJM9SoE07faGbrETmHzij9tx/E6ORm4yDfNgEWpBy7K8aEnVTC8PG8v/zk/HL8b38bH19d2HKUy1Py2FzhEGQJ/f5RzCWdtx/EqOrCxytCijfuYvGIfDTr1pW7z9rbjKJWpYsElOdj5ZWqm7GX1N6Ntx/EqWihUuk4cO0rdH7rxUMnFPHJ1fdtxlMqWFt1uYUWp62iz/1vWr4+0HcdraKFQ6dr69QjKm6PccG1PAv31kJMqOBoNeZ9RAWN58Pc44s9n2bGDygYtFOpfVv8xkdYn5hBRbaiOga0KnOCQ0vS7dSg7j57hw1lLbMfxCloo1CWOHzlAjSVPssO3NuF36BjYqmDqWLcszzU5wMi1fdmwZHbWT1CZ0kKhLjLG8OO0KRQ1Cfj0+5iAIoG2IymVazf37c8RnzKU+vNhTp+Ksx2nQNNCoS76JeoAL+2uz+SOv1KrSVvbcZS6LEHFQzjV410qphxmw1cP2Y5ToGmhUAAcPbiX32ZMpEW1ktx5VUvbcZRyi0Ztr2VFxQG0jZ3BuoU/2Y5TYGmhUJiUFKK/voc3U17j7esq4+erbwvlPULvfJ2dPjWYt2gRJ+ITbccpkPQTQbHy5w8IjV/G2voPUKtmLdtxlHKrwKDinBo8l/fir+G5XzbajlMgaaEo5A7s2UKjyBfZ6N+UNgPG2o6jVJ5oUbM8911ZhyORv7Jq7ve24xQ4OelmXHmZc+fPcerb2wkGQgZO0L6clFd7sGtt9q6YSvnFB4mp04LKtRrajlRgaIuiEHvh1618Hn8l2zq8QpXaTWzHUSpPBQT4U3TQdwCcnXgbCfFnLCcqOLRQFFKzIrbz7fK9lOhwF2HX3mk7jlL5okrtRuzs9AZ1k3cQ9em9tuMUGFooCqHdmyJo90s3/q/STkb10Oa3KlxCuw9kaeXBtDn2Cwt+n2Y7ToGghaKQOX3yOD4/DgYR7u1/A/56KawqhFoPfYO3S43l3sVF2Rhz0nYcj6efEoWISUlh64Q7qZIcw6GrP6Rc5Zq2IyllhZ9/AIPueoiQogG8+O0sTsYdtR3Jo2mhKESWT3mJ8NMLWVnnAZp0vM52HKWsKhdchI/71+O9s4+zY8JgTEqK7UgeSwtFIRGx+xgbN21gTVAH2t7+rO04SnmE8Po12Fr/XsLO/sPy78bZjuOxtFAUAkdPn+P+Sav5usQwao+Yhvjon12pC9oO/C+ri3em1fb32Lj0N9txPJJ+Yni5pMTzrP/wdqqc3cKHg8IJKR5kO5JSHkV8fKh3z9cc8KlI+TnDOXJgr+1IHkcLhRczKSms/nAIXc7+wZiWKTSpHGI7klIeKTikNEn9v+Ev05K7v9/OqQTtPDA1LRRebNmXo2hz/FeWVb2bNn0fsB1HKY9Wq3Fryt/2MRsPJ/DYNws4fy7BdiSPoYXCS62Y+ibt933KipK9aHvX67bjKFUgdGlQnld712VU9ANEfXA7JiXZdiSPoIXCC/216SBno34mKrA1YSO+0pPXSuVAv7b1OVy7D61O/snyzx62Hccj6CeIl4ncF8f9kyJ5u+yz1Ll/Gv4BRWxHUqrAaTf4JZaX6UO7mG9YPmW87TjWaaHwItHb13Py837ULR7Pp0PbUyxYT14rlRvi40PL4Z+xJqgDrTe9wqo/JtuOZJUWCi8ReygavruJZmzlg351KBesLQmlLoefvz8N7/+BP4v24IFFri+tFlZaKLzA2dMniP20L2VSjnHo+q+pXq+57UhKeYWixYJp/cA3FClZiXu/WsbuHVtsR7JCC0UBlxB/hu3v30SdxG1sueJdGra6ynYkpbxK6WIBfD20DePkE4p8ex0H9hS+YmGtUIhIDxHZIiLbRWRMOtMHiUiU87NERFrYyOnJ4s8n89i3iwg+u5dVzccR2n2g7UhKeaXqZYJo1OdxinEW+fI6orevtx0pX1kpFCLiC3wA9AQaAwNFpHGa2XYBVxpjmgPPAxPyN6VnO33qBEO+WMrsXSmsvu5X2tz0sO1ISnm1ui06cbjvjxQhgSITr2fPlkjbkfKNrRZFG2C7MWanMeY8MAXonXoGY8wSY8xx5+4yoGo+Z/RYJ44fZf8719Bv/+u8PSCMm9rWsx1JqUKhbouOxN0yHcFgJg9kc8zxrJ/kBWwViirAvlT3o53HMnI3kGG3jiIyTEQiRCTiyJEjboromeKOHuTI+9dQK3Eb9TrdxI0tKtuOpFShUqtxa87e9jPP+j3EgM9Wsi76hO1Iec5WoZB0HjPpzijSFVehGJ3RwowxE4wxrYwxrcqVK+emiJ7n6MF9HP/wWqol7WXTlR8Tfu0dtiMpVShVrx/Ks/cNpngRP+Z89iSbV861HSlP2SoU0UC1VPerAjFpZxKR5sBnQG9jTGw+ZfNIB+POEjuhNxWTY9jW/XNadLvFdiSlCrXqZYL4cWhzBsg8qs0axAYvHsvCVqFYCdQTkVoiEgAMAGamnkFEqgM/AXcYY7ZayOgx9sfFc+uny3kleRC7e3xD0yt6Z/0kpVSeq1S+LIH3/M5R33LU/n0wUQun246UJ6wUCmNMEjASmANsAn4wxmwQkeEiMtyZ7WmgDPChiESKSISNrLZtWTWfr999huNnzvPA3XfRqH1P25GUUqmUrVyT4vf+zgHfyjT8626WT3vHdiS3E2PSPTVQYLVq1cpERHhHTYmY+RHNVj3FYZ+yxN+9iPpVy9uOpJTKwInjR9nzyS18dbI1JdoN5r/XNcLPt+B8p1lEVhljWqU3zS+/w6isJSclsfKzB2l38Ds2FGlOlXt+oFo5LRJKebKQUmVp/NgflPp9K58v3kWRvYsYMbAfIWUq2I522bRQeJhT8efZ/u6NtItfyvIyfQi/d4J2Fa5UAeHn58dT1zemaRlD99/v5sT77xB36yRqNAy3He2yFJx2USGw++gZ+n60lFmn6rG88VjaPvC1FgmlCqC+7Zuw/7pvKGrOUnpyL9b+9YPtSJdFWxQeYv3fP/PBvC3E+oRy1dBnaFunrO1ISqnL0LDNNRysOI+4r2+h2cJhLIvZQNtB4xBJ72tknk1bFJYlJyWx9JunaDh3CPf7/sTPIzrSQYuEUl6hYvV6VHpkAZHBnYnavI2HpkRyIj7Rdqwc0xaFRTG7NhM36W7aJ65ndXBn6g/7huIlitmOpZRyo6DiIYQ9Op1lC7bz69wdJOxcwoNdatC04/W2o2WbFgoLjDHM/ns5V87rQzCwMnw8rW4YjvhoA08pbyQ+vozo1oCO9Spw/qtXaPzHWpZGDSBsyBsEFvX8nUP9ZMpnsSfPMHziKu6ffZSZJW7l9F0Lad17hBYJpQqBFtVK0vSRX1hZri/tD03mwGvt2b5uqe1YWdJPp3y09q8pxL8Zyp7NkTzZqxG3PvI2lWo0sB1LKZWPihYvQduRX7L2ys8ITjlB9anXMXXmTJJTPPfLz1oo8sGpE8dY/t5gWiy6l0SfID4YFMqwznXw9Sl4Vz8opdyjRdf++N6/jN/LDObxJcKACUvZe9Azh0nQQpGHUlIMy2d8yLm3wmh9dCbLKg6i8qil1Gmc7rfklVKFTKlylbjhgbd445YwDh+IJuijliz99CHOnIqzHe0SWijySOS+OPp+tIS1EYuJ9avI9t4/0274hxQJDLIdTSnlQUSEfuFV+fG+9uwq2Zb2+7/izBthRPzyCSYlxXY8QDsFdLujB/eya8oo3jocxrZiLRl7TW1uDK+Bj6+vtUxKqYJj88q5+P4+mnrJ29nk34SU26fTpEbe9xelnQLmg/PnElj943iabvuEFpxnSL1GdLitC8WL6CZWSmVfw9bdSQlfwYoZ77Fz/TKe+DiCAa2r81i36pQpGWIlk36KXaak5BQi5kykysrxtDMxrA1qS6l+r3NNvea2oymlCigfX1/a3PQwDXolsm3eNpYs/RuJ6snSGoNo0ncUJUqWydc8WihyKfH8OWas2c/7i/bQOW4VdxUJYO0VE2jR7Vbb0ZRSXiKkqD9PXd+YPQ2S2PtTE9rv+ZiTb3/D0mq30bjPqHzrwlzPUeTQuYSzRP7yIdU2fsL7529gbYW+PNilFlc3qaTnIZRSeWr72sWc+mM8YWcWc9SE8HXbWQzp3IAyxS+/l+nMzlFoocimhLOnifz5XWpt+YwKxLLFrwFnr3iS0M69C2RvkEqpgmvXhuUs+ns+4/Y0I9DPh3dqLiP8urspW7F6rpepJ7Mvw75jZ5m0Yi/tl42gM6vY6N+Uw1e8QdNOvbXbDaWUFbWatKVWk7Z0PHyKab/P5aod7zDhw910HPYOzauWdPv6tFCkIynxPOvm/wCrv+X/TgwhTkLwrXEbZZo9RpMOvWzHU0opAOqWD2b04L7s216XcxvO0LhSiTxZjxaKVI7E7Gb77x9Se+9UwojlMKUZ1cqPzt27UimkqO14SimVrmp1m/Fw3bxbvhYKx4s/LuLx9X1oL8lEBbZkf9hzNO92C7f6B9iOppRSVmmhcJQuV4W5tR6n+RU30rxOE9txlFLKY2ihcNzXpQ7whO0YSinlcfSyHaWUUpnSQqGUUipTWiiUUkplSguFUkqpTGmhUEoplSktFEoppTKlhUIppVSmtFAopZTKlNd1My4iR4A9uXx6WeCoG+O4k2bLHc2WO5otdwpythrGmHLpTfC6QnE5RCQio/7YbdNsuaPZckez5Y63ZtNDT0oppTKlhUIppVSmtFBcaoLtAJnQbLmj2XJHs+WOV2bTcxRKKaUypS0KpZRSmdJCoZRSKlOFulCIyGsisllEokRkuoiUzGC+HiKyRUS2i8iYfMrWX0Q2iEiKiGR4SZuI7BaRdSISKSIRHpbNxnYrLSJ/isg253epDObLt+2W1XYQl3ed6VEiEp6XeXKYrYuInHC2U6SIPJ1Pub4QkcMisj6D6Ta3WVbZrGwzZ93VRGS+iGxy/kcfSmeenG87Y0yh/QGuAfyc268Ar6Qzjy+wA6gNBABrgcb5kK0R0ABYALTKZL7dQNl83m5ZZrO43V4Fxji3x6T3N83P7Zad7QD0An4DBGgHLM+nv2N2snUBZuXn+8tZb2cgHFifwXQr2yyb2axsM2fdlYBw53YwsNUd77dC3aIwxvxhjEly7i4DqqYzWxtguzFmpzHmPDAF6J0P2TYZY7bk9XpyI5vZrGw3Zx1fO7e/Bvrkwzozk53t0Bv4xrgsA0qKSCUPyWaFMWYRcCyTWWxts+xks8YYc8AYs9q5fQrYBFRJM1uOt12hLhRp3IWryqZVBdiX6n40/97wNhngDxFZJSLDbIdJxdZ2q2CMOQCufxqgfAbz5dd2y852sLWtsrve9iKyVkR+E5Em+ZArOzz9/9L6NhORmkAYsDzNpBxvOz+3JvNAIjIXqJjOpLHGmJ+decYCScB36S0incfcck1xdrJlQ0djTIyIlAf+FJHNzh6P7WxWtlsOFpMn2y0d2dkOebatspCd9a7G1QfQaRHpBcwA6uV1sGywtc2yw/o2E5HiwDTgYWPMybST03lKptvO6wuFMaZ7ZtNF5E7geuAq4xzASyMaqJbqflUgJj+yZXMZMc7vwyIyHdfhhMv+wHNDNivbTUQOiUglY8wBpzl9OINl5Ml2S0d2tkOebassZLne1B8yxpjZIvKhiJQ1xtju+M7WNsuS7W0mIv64isR3xpif0pklx9uuUB96EpEewGjgRmPM2QxmWwnUE5FaIhIADABm5lfGzIhIMREJvnAb18n5dK/EsMDWdpsJ3OncvhP4V+snn7dbdrbDTGCwczVKO+DEhcNneSzLbCJSUUTEud0G12dGbD5ky4qtbZYlm9vMWe/nwCZjzJsZzJbzbWfjzLyn/ADbcR2ri3R+PnYerwzMTjVfL1xXD+zAdeglP7L1xVX5zwGHgDlps+G6WmWt87PBk7JZ3G5lgHnANud3advbLb3tAAwHhju3BfjAmb6OTK5ys5BtpLON1uK64KNDPuWaDBwAEp332t0etM2yymZlmznr7oTrMFJUqs+1Xpe77bQLD6WUUpkq1IeelFJKZU0LhVJKqUxpoVBKKZUpLRRKKaUypYVCKaVUprRQKJUBETmdB8usKSK3ZTK9kojMymIZ14vIs+7OplRGtFAolb9qAhkWCuBR4NMslvErcKOIBLkrlFKZ0UKhVBac8QUWiMhUcY1f8l2qb97uFpFXRGSF81PXefwrEbk51TIutE5eBq5wxil4JJ3V3QT87jznURH5wrndTETWi0iQcX35aQGurmeUynNaKJTKnjDgYaAxrm92d0w17aQxpg3wPvB2FssZA/xtjAk1xryVeoKI1AKOG2POOQ+9DdQVkb7Al8C95n9dzUQAV+T61SiVA1oolMqeFcaYaGNMCq5uEWqmmjY51e/2l7GOSsCRC3ecdQ0BvgUWGmP+STXvYVzdkiiV57RQKJU951LdTubSnpdNOreTcP6/nMNUAdlYRzwQmOaxesBp/l0UAp35lcpzWiiUuny3pvq91Lm9G2jp3O4N+Du3T+EaojI9W0nVUhGREOAdXENvlkl9zgOoj+f0FKy8nBYKpS5fERFZDjwEXDhB/SlwpYisANoCZ5zHo4AkZ/SzS05mG2POADsunBAH3gI+NMZsxdVD6cvOQEsAXXFd/aRUntPeY5W6DCKyG1c3zW4ZlMY5cd3SGPPfTOapAEwyxlzljnUqlRWvH+FOqYLEGDNdRMpkMVt14D/5kUcp0BaFUkqpLOg5CqWUUpnSQqGUUipTWiiUUkplSguFUkqpTGmhUEoplan/B5U2EA8cuG3YAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "m = MLPLayer(1, activation=nn.Tanh)\n",
    "torch_dmodel_v = np.hstack([\n",
    "    jacobian(m, torch.tensor([[vv]])).detach().numpy().flatten() for vv in v])\n",
    "\n",
    "truth_dtanh_v = dtanh(v)\n",
    "\n",
    "plt.plot(v, truth_dtanh_v, label='Analytic')\n",
    "plt.plot(v, torch_dmodel_v, linestyle='--', label='Autograd')\n",
    "plt.xlabel('Input (x)')\n",
    "plt.ylabel(r'Derivative ($ \\frac{\\partial M}{\\partial x}$)')\n",
    "plt.legend()\n",
    "plt.title('Single neuron with tanh activation')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c8c47c95-1f6f-4a3d-b74b-5741949b615a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:3: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, '2 layer (relu, tanh)')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5I0lEQVR4nO3dd3hUZdrH8e89mfSEmlBDU4PSWwiIDdRVURQsKIhYF2ysbXcFO7uuupa1vXbXuiqoIOoqFkQRRZEmLRRFBAlFSmghdZL7/WMGNqSXmTmTyf25rlyZOc8z5/xylNw57XlEVTHGGGMq4nI6gDHGmNBmhcIYY0ylrFAYY4yplBUKY4wxlbJCYYwxplJWKIwxxlTKCoWpt0TkchH51ukcB4nIPBHpU8vPhtTPUpKIbBCRUyto6yki3wU7kwkuKxQmZIhItIi8JCIbRWS/iPwoIkOdzlUdInI2sF9VfwyBLHNE5I/B2JaqLgf2+H5+E6asUJhQ4gY2AScBjYG7gHdEpKOToUoSEXcFTdcA/6nF58LBm8DVTocwgWOFwoQMVT2gqpNVdYOqFqvqR8CvQL/qfF5EnhCRTSKyT0QWi8gJvuWtRCRHRJqX6NtPRHaISKTv/ZUislpEdovIZyLSoURfFZHrReRn4OdythsFnAx8XWLZZBGZJiJviMg+4HIRaew7YtoqIptF5B8iElHO+jr6tukusaxaRwkich9wAvCUiGSLyFOV7ZsSWd8Rkdd9R3IZIpJWatW9RWS5iOwVkbdFJKZE2xzgFBGJriqfqZ+sUJiQJSItgc5ARjU/shDoDTQD3gLeFZEYVd2G95fZhSX6XgJMVdVCERkB3A6cByQD3wBTSq17BDAA6FrOdlOBYlXNLLV8ODANaIL3r+7XAA9wFNAHOA3w6ykiVb0Db/4JqpqgqhN8TeXumxIfPQeY6sv6IfBUqVVfCJwBdAJ6ApeX2OZmoBA42p8/iwkdVihMSPL9pf8m8JqqrqnOZ1T1DVXdpaoeVf0XEM3/fnm9hrc44PsrfjT/O1V0NfCAqq5WVQ9wP96/oDuUWP0DqpqlqrnlbLoJsL+c5d+r6vuqWgw0AoYCN/mOnLYDjwGjqvOz1VUV+wbgW1WdqapFePdLr1KreFJVt6hqFvBfvEWnpP1494MJQ1YoTMgRERfeX1YFwIQqupf83J99p4/2isgevNc5knzNHwBdReQI4A/AXlVd4GvrADwhInt8n8sCBGhbYvWbKtn0biCxnOUlP9MBiAS2ltjO80CL6v58dVHFvgHYVuJ1DhBT6rpK6faEUptIBPb4L7EJJeF8gc3UQyIiwEtAS+BMVS2s5udOACYCpwAZqlosIrvx/sJHVfNE5B1gDHAMh1943gTcp6pvVrKJyoZZ/tkXva3vNEx5n9kE5ANJvqOWyhzwfY8D9vlet6riMxVmrWrf1JWItAGigLX+WJ8JPXZEYULNs0AX4OwKTvNUJBHv+f8dgFtE7sZ7uqek1/GeWz8HeKPE8ueA20SkG4DvovPI6m7YV8y+wHu3VkV9tgKfA/8SkUYi4hKRI0WkzGdUdQewGbhERCJE5ErgyIPtJS52d6xgc78DR5R4X519UxeDgS9VNd+P6zQhxAqFCRm+awJX4z3/vc131062iIypxsc/Az4BfgI2AnmUOl2kqvOAYmCJqm4osXwG8CAw1XeH0kq81xNq4nlgbBV9LsX7l/cqvKerpgGtK+g7DvgrsAvoBpR8qK0d3p9xczmfA3gCuMB3B9eTVGPf1NEYvMXWhCmxiYtMQyIiXwJvqeq/A7Dub4E/BfqhOxG5E9ihqs8HcjvVzNIDeEFVj3U6iwkcKxSmwRCR/sAsoJ2qlneXkjGmHHbqyTQIIvIa3usIN1mRMKZm7IjCGGNMpeyIwhhjTKXC7jmKpKQk7dixo9MxjDGmXlm8ePFOVU0ury3sCkXHjh1ZtGiR0zGMMaZeEZGNFbXZqSdjjDGVskJhjDGmUlYojDHGVCrsrlEYY8JXYWEhmZmZ5OXlOR2l3oqJiSElJYXIyMhqf8YKhTGm3sjMzCQxMZGOHTviHWjY1ISqsmvXLjIzM+nUqVO1P+fYqScReVlEtovIygraRUSeFJF1vikY+wY7ozEmtOTl5dG8eXMrErUkIjRv3rzGR2ROXqN4Fe/UihUZineKyVRgPN7hp40xDZwVibqpzf5zrFCo6ly8M4lVZDjwunrNB5qISEVDMhtTr2xcs4Qf3v4nP61ZTnGxDaNjQlso3/XUlsPHzM/k8KkpDxGR8SKySEQW7dixIyjhjKmL7Z88yIDVD/DKf16lz72zuPXlT5g/5T7Wr/yB4qIip+OZKsyYMQMRYc2aak3nXq7LL7+cadOmVdrn/vvvP+z9oEGDar29ugjlQlHe8VG5f3qp6guqmqaqacnJ5T6BbkxIifAcYLO05Ljhf+T0bi2J37aAgWsf4ohpp7H33g4seXgYy758x+mYpgJTpkzh+OOPZ+rUqQHdTulC8d1331XQM7BCuVBk4p3J66AUYItDWYzxK3dRHgciGjMsvSsPXdCLe26/h21XLmJh7/v5uckJtD2wiq5fX8Nj735BXqEdYYSS7Oxs5s2bx0svvXSoUMyZM4fBgwdzwQUXcMwxxzBmzBgOjsz997//nf79+9O9e3fGjx9P6RG7Z8+ezbnnnnvo/axZszjvvPOYNGkSubm59O7dmzFjvJM8JiQkHOr30EMP0aNHD3r16sWkSZMC+jOH8u2xHwITRGQqMADY65t32Jh6z12cR6Er5rBlrdqn0qp9KnA9+Xk5TH3vXZ5YnM+sLd/x9Hkd6NSuvTNhQ9Tf/pvBqi37/LrOrm0acc/Z3Srt8/7773PGGWfQuXNnmjVrxpIlSwD48ccfycjIoE2bNhx33HHMmzeP448/ngkTJnD33XcDMHbsWD766CPOPvvsQ+s7+eSTuf7669mxYwfJycm88sorXHHFFZx99tk89dRTLF26tEyGTz75hPfff58ffviBuLg4srIqu9xbd07eHjsF+B44WkQyReQqEblGRK7xdZkJrAfWAS8C1zkU1Ri/mxV1Ct8lVnzTX3RMHGMvvoyXLkvjqD3f0uLf/Vn4/tNBTGgqMmXKFEaNGgXAqFGjmDJlCgDp6emkpKTgcrno3bs3GzZsAOCrr75iwIAB9OjRgy+//JKMjIzD1icijB07ljfeeIM9e/bw/fffM3Ro5VO2f/HFF1xxxRXExcUB0KxZMz//lIdz7IhCVUdX0a7A9UGKY0xQfeQ6mdRmCVX2O6VLS7pfdSEbX/+Y/ktvZ+Gvc+j6xxeJT2wS+JAhrqq//ANh165dfPnll6xcuRIRoaioCBHhzDPPJDo6+lC/iIgIPB4PeXl5XHfddSxatIh27doxefLkcp9hOHgEERMTw8iRI3G7K//VrKpBvU04lK9RGBO24gu209hVvYeeWqYcydG3zuH7duPou2cWWY8ey7pl8wKc0JRn2rRpXHrppWzcuJENGzawadMmOnXqxLfffltu/4NFISkpiezs7ArvcmrTpg1t2rThH//4B5dffvmh5ZGRkRQWFpbpf9ppp/Hyyy+Tk5MDEL6nnoxpyF7Ou4URO1+odv8It5tjr3qENae9SbTm8eq7M/h4uV2yC7YpU6YcduEZ4Pzzz+ett94qt3+TJk0YN24cPXr0YMSIEfTv37/CdY8ZM4Z27drRtWvXQ8vGjx9Pz549D13MPuiMM87gnHPOIS0tjd69e/PII4/U4aeqWtjNmZ2WlqY2cZEJdTn3tGB5q/MYeO1zNf7s7qydjJu6luWb9/HWVX1JO6JlABKGptWrV9OlSxenYwTEhAkT6NOnD1dddVXAt1XefhSRxaqaVl5/O6IwJsi0uJgYCtDI2Fp9vmmzJP59eX+GJf5Mq9ePJ3NducOlmXqkX79+LF++nEsuucTpKOWyQmFMkOXn5+IShci4Wq+jSVwUt1xwMvHkom+OZM/ObX5MaIJt8eLFzJ0797AL4qHECoUxQZafkw2ARNW+UACkHNWd34e+TIviHWx5/nzy83L8Ec+YMqxQGBNkuRrB3wrHsie54gub1XXMgNNYkf4AXQtXsuKZsWhxsR8SGnM4KxTGBFmOxvBK0VDyk/zzHEDaWeOY3/F6fs0q4MlZtR+kzpiKWKEwJsjycrNJlUziJd9v6xxw6T9Y0PNeHvvqV6Yv2ui39RoDViiMCTrXtuXMir6VlnuW+m2d4nJx33k9GdGhgF7/HUrGdzP9tm5TVk2GGX/88ccPPRgXTIMHD8ZfjwpYoTAmyDy+i87umHi/rjfK7eJvo04gMsJFyud/ZPP6jKo/ZGqlJsOM+7NQeDwev6ynpqxQGBNknvwDAET6uVAANG6aROTYaQjK3qlX2yRIAVDRMOPDhg071GfChAm8+uqrPPnkk2zZsoUhQ4YwZMgQwFtkevToQffu3Zk4ceKhz7z00kt07tyZwYMHM27cOCZMmAB4Jzi65ZZbGDJkCBMnTmTBggUMGjSIPn36MGjQINauXQtAbm4uo0aNomfPnlx00UXk5ub67WcO5WHGjQlLRfnevy4jY6oeFLA22nQ6hgW9biN92V388O5DDBh1W0C2ExJeOavssm4jIH0cFOTAmyPLtve+GPqMgQO74J1LD2+74uMqN1nRMOPlueGGG3j00Uf56quvSEpKYsuWLUycOJHFixfTtGlTTjvtNN5//33S09O59957WbJkCYmJiZx88sn06tXr0Hp++uknvvjiCyIiIti3bx9z587F7XbzxRdfcPvttzN9+nSeffZZ4uLiWL58OcuXL6dv375V/izVZUcUxgRZke+IIjrW/0cUB/UfPoHlMf0pWv0xv+08ELDtNEQVDTNeHQsXLmTw4MEkJyfjdrsZM2YMc+fOZcGCBZx00kk0a9aMyMhIRo48vMCNHDmSiIgIAPbu3cvIkSPp3r07N99886Fhy+fOnXvoye6ePXvSs2dPf/y4gB1RGBN0mYm9mF44jkmNAjdtr7hcJF/+BmOf/ZEu7y3nrT8OxOUK3rDUQVPZEUBUXOXt8c2rdQRRUkXDjJ9zzjkUl3iGpbyhxIEys9tVtfxQ1Pj//VFx1113MWTIEGbMmMGGDRsYPHjwobZADT1uRxTGBNnvUe15u2gIsfGJAd1O61atuO2sbvy0fgNfzgzs3M4NRUXDjAOsWrWK/Px89u7dy+zZsw99JjExkf379wMwYMAAvv76a3bu3ElRURFTpkzhpJNOIj09na+//prdu3fj8XiYPn16hRn27t1L27ZtAXj11VcPLT/xxBN58803AVi5ciXLly/3289thcKYIIvcv4me8gvR7sD/87uofzuebjaVQQtvZMuGtQHfXrirbJjxCy+88NCQ4H369DnUPn78eIYOHcqQIUNo3bo1DzzwAEOGDKFXr1707duX4cOH07ZtW26//XYGDBjAqaeeSteuXWncuHG5GW699VZuu+02jjvuOIpK3Kxw7bXXkp2dTc+ePXnooYdIT0/3289tw4wbE2Tzn7uOXlvfJfZvO4KyvW2//UzCSyewIeZouk38CnHV378Pw3mY8ezsbBISEvB4PJx77rlceeWVZYqSv9gw48aEOPHkkifBGyW0VftUMrr/le75S1kw/dGgbdfUzOTJk+nduzfdu3enU6dOjBgxwulIh9jFbGOCzOXJJZ+YoG4z/fybWbHuv3Rf+TBb08+mdYejg7p9U7VAz1JXF3ZEYUyQuTy55LuCO++AuFw0H/08s+nPvZ/+WuVdNqGsPmcPBbXZf1YojAkyd1EuhUE89XRQm45Hs3fo08z8tYipCzcFffv+EBMTw65du6xY1JKqsmvXLmJianZEa6eejAmy9+IvIiImj7sd2PaY9PYsWrKElI/HsjPldZLadHAgRe2lpKSQmZnJjh3BuREgHMXExJCSklKjz1ihMCbIlskxJCY480/P5RL+eloqLf6zgqXvTCLppuo/VRwKIiMjDz23YILHTj0ZE2RH5S7jKP3Nse2nHNWdJa1Hkbb7E35e+o1jOUz9YYXCmCC76cATnLXP2Selu466lz2SiOfjiTZ9qqmSFQpjgixa8yl2xzqaoVGT5vzc7Sa6FGbw4+evO5rFhD4rFMYEWTTOFwqAtHNv5NnYcUxcmkxeoc1bYSpmhcKYIIvRfDQECkWE202v8yfx8x54+dv1TscxIcwKhTFBVFiQT5QUQaTzhQJg0FFJXH3ELk6ccyE7tzl3gd2ENisUxgRRnke5uOB2fmtzptNRDrnk5L50ZiPr357kdBQTohwrFCJyhoisFZF1IlLm/1ARaSwi/xWRZSKSISJXOJHTGH/KLYLvirtT2Lij01EOaXdUdxa3voi0rJmsWzbP6TgmBDlSKEQkAngaGAp0BUaLSNdS3a4HVqlqL2Aw8C8RiQpqUGP8LC97D2e55tPME1pPFncb9Q/2SCL5drusKYdTRxTpwDpVXa+qBcBUYHipPgokinduvwQgC/AEN6Yx/lWUtZGno56kVfZKp6Mcxnu77I10K1jB4i+nOR3HhBinCkVboOSoZJm+ZSU9BXQBtgArgBtVtdw/dURkvIgsEpFFNgaMCWUFuQcAiIiOr6Jn8PUbcQP3xU3klsXNyPfY7bLmf5wqFOXNAF56OMjTgaVAG6A38JSINCpvZar6gqqmqWpacnLgJqw3pq4K872Fwh2ChcIdGcXg88bz2+58Xv1mndNxTAhxqlBkAu1KvE/Be+RQ0hXAe+q1DvgVOCZI+YwJiKI8b6GIjAm9QgFw3FFJ3NJhPWfNOYs9O7c5HceECKcKxUIgVUQ6+S5QjwI+LNXnN+AUABFpCRwN2FNBpl4rys8BICo2NAsFwPDBg2jNTtZM+7vTUUyIcKRQqKoHmAB8BqwG3lHVDBG5RkSu8XW7FxgkIiuA2cBEVd3pRF5j/GVT03TOzf8b7uahO1R2hy79WNLkdHpvfYftmfa3mXFwPgpVnQnMLLXsuRKvtwCnBTuXMYG0TxL5UVOJjk1wOkql2p77d1yvzGLDe3fT4oY3nI5jHGZPZhsTRPG7MhgZMYdYd2hP5dmm49H82OJc+u76mN82/Ox0HOMwKxTGBFHr3+fwcOQLxLhD/5/ekedP5gq9m4e/z3Y6inFY6P/fakw4KcylQN24I0N/kIGkVu3oddyZ/HfZFjI273E6jnGQFQpjgsjlySFPop2OUW3jTjyC22Omk/fGxU5HMQ6yQmFMEIknlzzqT6FoHBtJzyPa0i93Hqvmf+p0HOMQKxTGBJHLk0e+xDgdo0Z6nX8rO2iKzP67DRjYQFmhMCaIpja9msmJ9zgdo0Zi4xNZ3/V6uhRmsHzOu07HMQ6wQmFMEP2uTdgd067qjiGm74gbyJRWxM57mOIiO6poaKxQGBNEaXtncXzRD07HqLHIqGh+Pf5hxuVcy39XbHU6jgkyKxTGBNFZB95jSO7nTseoleOGnE1sy1QenfUThTYMeYNihcKYIIoqzqMoItbpGLXicgkTh7Tlrn1/Y8mMx5yOY4LICoUxQRSteRRH1K+7nkoa3KMTKTH5dMp4mjzfJEwm/FmhMCaIosmn2F0/jygAxOXCc9JttCCLpXZU0WBYoTAmiGI0H42sv4UCoPvx55AR1YvUn14gJ3uv03FMEFihMCZIiouVE/KfYGmHK52OUmcRp9xJc/aybMajTkcxQWCFwpggyfMUsYMmuGKbOB2lzo4ZcBr/bvZnJq7vxf68QqfjmACzQmFMkORm7+Vm97u0zV3rdBS/SD/vBn7LjeaVeRucjmICzAqFMUFSsH8nN7pn0CJ3ndNR/KJnShPGH5HFsd9cyt6sHU7HMQFU60IhIvEiEuHPMMaEs4Jc7wRAEVFxDifxn1GDOtOfVayafp/TUUwAVbtQiIhLRC4WkY9FZDuwBtgmIhki8rCIpAYupjH1X4HvuYOI6HiHk/jPEd0HsDhhML0y3yJr+2an45gAqckRxVfAkcBtQCtVbaeqycAJwHzgnyJySQAyGhMWCvO8hcIdEz6FAiBp2D1EU8BP79lRRbhy16Dvqap62O0NIlKkqhHAdGC6iET6NZ0xYcRzsFCE0REFQIdj+rKwyR/ovfUddm6ZSFKbDk5HMn5W7SOK0kXCR6rRxxgDbEk6lm55L6Ft+jgdxe/aDJ/MvUWX8czCPU5HMQFQZaEQkdki8oiIjBaRzqWa1dfHLmobU4Vcj3KAWGKj689UqNXV9ohuFPe9jDcWbGHznlyn4xg/q84RxWygia/vJSIypZw+L4hIHICInOi/eMaEj4RtC7nD/QZxhOcv0gknp3KhzGbtW7c6HcX4WZWFQlXvB+4FBgKrVXV0Od3uBl4Skf8A/f0b0ZjwkJi1gnHumUSH6fF32yaxnN0qixN+f5PN61c7Hcf4UXVOPQ0DLgaKgfMqOM10L7AW76mod/ya0JhwUeg9koiNS3A4SOAced49FOFiywd3Ox3F+FF1Tj09DfTAewvsHapa3tRWt6rqZOBaoH7NHG9MsBTm4FEXkZFRTicJmKQ2HVja+kL67ZnFxtWLnY5j/KQ6p546ALcCOVR8jSJORPoBHuBq/0Y0JkwU5pJHNOIK75Fzjj7/LnKIYddHk52OYvykus9RuIBM4FNVzS+xXETkOrxHHK2AbGAu8KJfUxoTBrSogDyJJnxPPHk1TW7N56l/4c1VBfwlcy89Uho7HcnUUXWuUVyH92nsO4F/i8i4Ul0GAjfiLTqXAWdWZ8MicoaIrBWRdSIyqYI+g0VkqW+YkK+rs15jQtU7LW7ivNiXnI4RFMdecCPLYtJ45PPwGCm3oavOMfCxVFwIVFUvVdUC4HHgGeCLqlbouyD+NDAU6AqMFpGupfo08a3vHFXtBoysRlZjQlZuQRExUQ1j8ILEmEhuOK4Vx65/klXzP3M6jqmjKk89qepYABF5HHiWwwuBlOg3G+8zF9WRDqxT1fW+dU8FhgOrSvS5GHhPVX/zrX97NddtTEgavGsqgzwFwElORwmK0QM7kf3Nt+ycvR5N/0PYX5sJZzX5L/elql6tqk8fXKCqh31eRKTsx8rVFthU4n2mb1lJnYGmIjJHRBaLyKUVrUxExovIIhFZtGOHjYtvQlPPnO9JK2w4dwLFxieyvsu1dCnMYMXX7zkdx9RBjUaPFZE/iUj7kgtFJEpEThaR1/CemqqO8gqKlnrvBvoBZwGnA3eVM4SI94OqL6hqmqqmJScnVzOCMcEVWZyHxxXjdIyg6jPiRrZIC+K+vR8tLnY6jqmlmhSKM4AiYIqIbBGRVSLyK/AzMBp4TFVfrea6MoF2Jd6nAFvK6fOpqh5Q1Z1476bqVYO8xoSUyOI8PO5Yp2MEVVR0DJm9buKool/48fPXnY5jaqkmo8fmqeozqnoc0AE4Beijqh1UdZyqLq3BdhcCqSLSSUSigFHAh6X6fACcICJu3zhSAwAbF8DUW1GaT1FEwyoUAP2GXc2MqGE8tSKCouLSJw5MfVCrq0uqWqiqW1V1Ty0/7wEmAJ/h/eX/jqpmiMg1InKNr89q4FNgObAA+LeqrqzN9owJBfkaSWFkotMxgi7C7Sb67Ef4clcz3v/RZsGrj2oycZFfqepMYGapZc+Vev8w8HAwcxkTKGcVP8rFHdszyOkgDjijWyv+0CobZv6Fgu6vEhXdsK7V1Hd2v5oxQaCq5BYWERsVpkPHVsHlEib0cnF+0Sf8+METTscxNVTjQiFel4jI3b737UUk3f/RjAkf+Xm5PON+jGP2znM6imN6nnQ+qyO7ccSqZ8k9sN/pOKYGanNE8Qzep7UPzkuxH+9T1saYCuTn7GdoxEKaF251OopjxOVCT76LZHazdNo/nY5jaqA2hWKAql4P5AGo6m4gfMdNNsYP8nKzAXBFxzmcxFldjx3K0tiBdF//Ert3NNyiWd/UplAU+sZqOjhfdjLeSY2MMRUoOFgoohp2oQBocs79TCs6kRe/3eh0FFNNtSkUTwIzgBYich/wLXC/X1MZE2YKcg8AEBEd73AS53Xs0o+1fe7kxUVZ/LYrx+k4phpqXChU9U28Exk9AGwFRqjqu/4OZkw4yfcUkalJuGJtbgaAm//QmQGuNax86zano5hqqM1dTzcD2ar6tKo+5XswzhhTiazELhyf/yQF7Y5zOkpIaNkohus7ZHLmrlf5aYlNNRPqanPqqRHwmYh8IyLXi0hLf4cyJtzkFHgAiI1smM9RlKf7yDvJohGFn95hAwaGuNqcevqbbyKh64E2wNciUuVkRcY0ZAmbv+H1yAeIz7dpVQ5KbNyMn7tMoFvBCpZ/9Y7TcUwl6vJk9nZgG7ALaOGfOMaEJ/f+zZwYsYLYyOpO2dIw9D33JjZJG5rM+weewgKn45gK1OYaxbUiMgfvbHZJwDhV7envYMaEk+IC79090bEJDicJLZFR0WwfdDev55/E9MWbqv6AcURtBgXsANxUw2HFjWnQtDAXgNg4KxSl9T11FPf93J7/zv6Vs/t2IC7KsbFKTQVqc41ikhUJY2qo0HdEEWMP3JUmItx+ZheOPzCLBW8/5HQcU45qFwoR+db3fb+I7CvxtV9E9gUuojH1X7YkkqGdEJcN2FyetI7NuKRpBv3XPcGu3zOdjmNKqckMd8f7vieqaqMSX4mq2ihwEY2p/+Y2u4CxbptapTJJI+4jmgLWvXuX01FMKbW5mP1gdZYZY/4nt7DInqGoQvvUXixKHk7fHR+wcc0Sp+OYEmpzHPyHcpYNrWsQY8LZaVue5W+ex5yOEfI6X3gfuRLN/hm32EN4IaQm1yiuFZEVwNEisrzE16/AisBFNKb+a5m3no5q80VXpVmLtvzY/Q6e2D+EzzJ+dzqO8anJEcVbwNnAh77vB7/6qeqYAGQzJmy4i/IodNk80dVx3LnX8VvyEP4xczV5hUVOxzHU7GL2XlXdoKqjgX1AS7zPVHQXkRMDFdCYcOAuzqMwwgpFdbgjXEw+uyvn73uDRW/e7XQcQy0euBORPwI3AinAUmAg8D1wsl+TGRNGoorzyXElOR2j3jj2qCSimu6i268fsmXDpbTpeLTTkRq02lzMvhHoD2xU1SFAH2CHX1MZE2bWSXu2xx7hdIx6pe1F/6IYF9ve/bPTURq82hSKPFXNAxCRaFVdA1i5N6YSd7huZE7b8U7HqFdatTuKZZ2upO+Bb1gx9wOn4zRotSkUmSLSBHgfmCUiHwBb/BnKmHCTW1hEjD1HUWN9LrqLzdKShDl3UujxOB2nwarNWE/nquoeVZ0M3AW8BIzwcy5jwsp0vYXjdk13Oka9ExMbz9bBjzIh9xpe+/43p+M0WLWaClVEUgBU9WtV/VBVbSB5YypQWJBPZ8kkXnOcjlIv9TvxLJI79+fxL35m+z7bh06wqVCNCbDc3AMASJSNHFsbIsLdZ3XhzuLn2fDSlU7HaZBsKlRjAqzgwH4AJNIKRW0d0SKRDiltSd/7CWsW2q+bYLOpUI0JsPy8bABcdkRRJz0vvpftNMP96a02bWqQ2VSoxgRYbpGL2UV98DRKcTpKvRaf2IRN6XdyVNEvLHr7PqfjNCi1OaI4OBVqN1W9R1VX1WbDInKGiKwVkXUiMqmSfv1FpEhELqjNdoxx2v7oVlxV+Ffy2g50Okq91/eMK/gxbhBH/vwKG7ftdDpOg+HIVKgiEgE8jXd48q7AaBHpWkG/B4HP6rI9Y5yU6xvYzuajqDtxuUgZ+zyj5Z/c+sHPFBer05EaBKemQk0H1qnqet+ttVOB4eX0+xMwHe/1EGPqpbjfvmZ+9PU03f+T01HCQnLr9ow763h++HUXH331tdNxGoQaTYUqIgJ088NUqG2BTSXeZ/qWHSIibYFzgeeqWpmIjBeRRSKyaMcOG3bKhJbi3L20kt1ER0U6HSVsXJjWjieSPuTkuaP4PfMXp+OEvRqdelJVBWb4YbtS3upLvX8cmKiqVQ5Ir6ovqGqaqqYlJyf7IZ4x/uPJ9z5HERWT4HCS8CEipJ13My6K2fbmtTYbXoDV5mL2fBHpX8ftZgLtSrxPoex4UWnAVBHZAFwAPCMiI+q4XWOCTgu8TxNHxcY7nCS8tD2iC8uP/hO9cn9g8ccvOh0nrNWmUAzBWyx+8U2FukJEltdwHQuBVBHpJCJRwCi8M+cdoqqdVLWjqnYEpgHXqer7tchrjKO00FsoYuITHU4SfvpfeBtr3Mdw5OJ7ydpuU80GSm0KxVDgCLwTFZ0NDPN9rzZV9QAT8N7NtBp4R1UzROQaEbmmFpmMCVk7IlP4qGggMTF2ROFvEW43Mec/w16N56WPv3E6Ttiq8Qx3wG/AGOAIVf27iLQHWgEba7ISVZ0JzCy1rNwL16p6eS1yGhMSMhqdwMvFKQyLtIvZgdCxSz/+74QPePqLX+iz6ndO7WrDz/lbbY4ongGOBUb73u/H+0yEMaYceYVFxETWZbQcU5WrB3emR8sYfpr+N/bt2eV0nLBTm/97B6jq9UAegKruBqL8msqYMHL6hgf5gFucjhHWotwu/jU4iqs9b7HmtT85HSfs1KZQFPqemFYAEUkG7N40YyoQWZiNW+yfSKB17nMCC9peSvruj1k88xWn44SV2hSKJ/E+S9FSRO4DvgXu92sqY8KIuyiXAle00zEahLTLH2at+2hSF9zO1o1rnY4TNmoz1tObwCPA83iffRihqu/6O5gx4cJdlEuhxDgdo0GIjIomcczriCo73xiHp8iO5PyhJmM9iYhMFpGdeI8obgYmAyMDlM2YsOAuzscTYUcUwdKm0zFkDHqUv2aP5vEvfnY6TlioyRHFTcBxQH9Vba6qTYEBwHEicnMgwhkTDr5392dF/CCnYzQoA0+/mJ79BvH0nHUsyLBTUHVVk0JxKTBaVX89uEBV1wOX+NqMMeX4T8R5zEuyA+9gm3xONyYnfkind09n946tTsep12pSKCJVtcxMIaq6A7AniYypQEFBgc1F4YC4KDfHD7uMxrqfjS9fbgMH1kFNCkVlk9TaBLbGVOCjgis4//cnnI7RIB3ZcxBLjrmF3rnz+eHtB5yOU2/VpFD0KjVh0aGJi4AegQpoTH0Xo/ngtruenDLgottYGjuQvmseZd2yeU7HqZdqMnFRRKkJi0pOXGSnnowpR3FRETFSCJFxTkdpsMTlosOVr7JZWvHyJ99xIN/jdKR6xwagMSaA8nKzvS+irFA4qWlya7aO+ZIpe7sy6b0VeOdgM9VlhcKYAMrL8RYKsULhuEGpLfnLaUfTZOWrfP/anU7HqVesUBgTQDlFETznOZv9zbo7HcUA1510BMObZzJow1Ms+ew/TsepN6xQGBNAua54/ukZTW6Lvk5HMXivV3S/9nXWuo/mmO/+zC/Lv3M6Ur1ghcKYAMrNz6MRB4i1xyhCRkxsPM2vmsZ+SSD+vbHs3LbJ6UghzwqFMQHk3ryI5THjaLVnkdNRTAlJrduz/9zXSdT9vPHWq+R7ipyOFNKsUBgTQJ68AwC4o22+7FBzVK/j+f6sL3h8e19uszuhKmWFwpgAKirwForImASHk5jynJrek5tOTWXr0s/5bopNq1MRKxTGBFBRfg4A0bF2RBGqbjwllVuaz+fYtQ+zdPZUp+OEJCsUxgTQoUIRZ0cUoUpE6H7Na/ziPpLUuTfyy4r5TkcKOVYojAmgrfHH8GjhBUQlNHU6iqlEbHwija+cRrbE03T6SDauXep0pJBihcKYAMqM6cyTRecRG5fodBRThRZtO1EwZgbFuJj79qP8tivH6UghwwqFMQGkOVm0ce0mMkKcjmKqoV1qL/Ze8jmP6sVc/O/5bNmT63SkkGCFwpgA6v/bS3we+RdErFDUF0cedTSvX3UsCTmb2fR/Z7Jz229OR3KcFQpjAkg8ueRJtNMxTA31SGnMY2e1oocng/0vDGPPzm1OR3KUFQpjAijCk0u+FYp6qUv6H1h/6ou0KdrCjmfPYt+eXU5HcowVCmMCyFWUR4HY7Hb1VfcThrPmpKfo6PmVzU8PIyd7r9ORHGGFwpgAchflUeiyI4r6rNfJo1gx8F/k5hdw01s/kFPQ8GbIc6xQiMgZIrJWRNaJyKRy2seIyHLf13ci0suJnMbUxaexZzIz8QKnY5g66jv0CjYMn8EXvxZyxQtfs7uBXbNwO7FREYkAngb+AGQCC0XkQ1VdVaLbr8BJqrpbRIYCLwADgp/WmNr7NiKd5vFRTscwfnBev/bEx0TB22PZ//Rmci+bQZuORzsdKyicOqJIB9ap6npVLQCmAsNLdlDV71R1t+/tfCAlyBmNqbOk3A20cmU5HcP4yendWtH+rL/QRPcQ9eppDWa4D6cKRVug5Gwhmb5lFbkK+KSiRhEZLyKLRGTRjh07/BTRmLq778Bkzst62ekYxo+6DDidrIs+pIgIWkwbwcp5/3U6UsA5VSjKe/qo3MHgRWQI3kIxsaKVqeoLqpqmqmnJycl+imhM3UWTT7E71ukYxs86dkmDP85iV0QyTT+/iZk/bnA6UkA5VSgygXYl3qcAW0p3EpGewL+B4aracG9iNvVWjOajVijCUsuUI2k64UseTfoH17+TwcvfrHc6UsA4VSgWAqki0klEooBRwIclO4hIe+A9YKyq/uRARmPqRIuLiaEAjbRCEa4aN0vmvmsu5LSuLSn47G7mP3cdRZ7wu33WkUKhqh5gAvAZsBp4R1UzROQaEbnG1+1uoDnwjIgsFRGbdNjUK/n5ubhEwQpFWIuJjOCZi/vSp1UkA7e9yeqHT2Hntk1Vf7Aecew5ClWdqaqdVfVIVb3Pt+w5VX3O9/qPqtpUVXv7vtKcympMbeR54OaCa9nWcrDTUUyARUS4GDDhFRb0upej8jLQ505g1fxPnY7lN/ZktjEBklvsYkbxCeQ3O8bpKCZI0s+9gc0XfES+xND+k8t4bfZiVMu9T6desUJhTIDkHdhPmqyhEfudjmKC6MgeA2l04zxeansv98zaxtX/WczeffucjlUnViiMCZCineuYFv13Wu350ekoJsgaNWnODePGcfewriT89B7Zj6Wxbtm3TseqNSsUxgRIYd4BANzR8Q4nMU4QEa48vhPjh5+MWz20e28EP7z9T4qLipyOVmNWKIwJEI+vUETGWKFoyI7pfypR189jbWxvBqx+gDUPnsSmn5c5HatGrFAYEyCefCsUxqtpcmt63Po5C3rdS0rBev75+vs8M2cdhUXFTkerFisUxgRIkRUKU4K4XKSfewP51y2h+OhhPPTpWp589O+sWzbP6WhVskJhTIBsadSbqwtuIrKpDXxs/ie5RSuevaQfz4/uzqgDb9LxvWF8/8KfyMvJdjpahaxQGBMgWRHJfFacTkx8I6ejmBB0eq8OJNzwHUuansGxW15nx8P9WTF3htOxymWFwpgAidr7Kye6lhHrLm+wZGO8Y0Wl3zSFFSe/josius6+gluef5+Vm0Nrbm4rFMYEyBFbPuL1qAeJiYxwOooJcT1OHE7SxKV83udpvvw9jmH/9y3Tnr2bLRvWOh0NsEJhTOAU5pCj0YjL/pmZqkXHxDF0xBi+/usQ/jKoCWdte5akVwYx/9mr2ePwHN32f7AxASKeXPIl2ukYpp5pHBvJhHOOY9+4+Sxtehr9t72N66k+fP/aHRzIdmYoECsUxgSIy5NHPlYoTO20TDmS9Jum8NtFs1gf25Oe61/kjEdm8fBna9i+90BQs7iDujVjGpCIolzyXVYoTN106tofun7G8oyVdFuSzzNz1nHKvEvY0LwzLU//Cx2O6RvwDFYojAmQ9xqNJc+1m0ecDmLCQs9u3XmuG2zYtpPf3+1Gr50fEzP1Y5bGDiTyhBvpOvCMgF0Ps1NPxgTIL7RjY1x3p2OYMNOxVRID/vQaOdcv4/v24+mQu4pun4/mwcceZt32wDy0Z4XCmADpkj2fnsWrnY5hwlSzFm059sqHifnrauZ3u4cFkf1p0Sgwpzrt1JMxATI2+xV2e1KAq5yOYsJYbHwCA0fewnsB3IYdURgTIJGaR7E7xukYxtSZFQpjAiRG8yh2xzodw5g6s0JhTIBEa74VChMWrFAYEyAxFKBWKEwYsEJhTAAUeooYUfB31qSMdDqKMXVmhcKYAMj1FJOhnShKbOt0FGPqzAqFMQGQn72PMRFf0CL/N6ejGFNnViiMCYCCfdu4L/JlWmdnOB3FmDqzQmFMABTkeodSiIiJdziJMXVnhcKYACjI9Q4DHRFlhcLUf1YojAmAwnxvoXDbEYUJA1YojAkAT563UERaoTBhwLFCISJniMhaEVknIpPKaRcRedLXvlxEAj87hzF+sqVpf07OfwRp2cXpKMbUmSOFQkQigKeBoUBXYLSIdC3VbSiQ6vsaDzwb1JDG1MGB4ijWaxtiYhOcjmJMnTk1zHg6sE5V1wOIyFRgOLCqRJ/hwOuqqsB8EWkiIq1VdWsgAt0xYwWXrbicKPIPW/515PG8ET0a0WJeOPCnMp/7LPJUpkWfS6zm8OSBv5Zp/yDqLD6KOpMmxbt5OOfOMu3vRJ3HrKhTaFW8jXtz7i3T/p/o0cyNPJ6ORRu4I/fhMu0vRl/Bgsg0jvGs5c95T5Zp/7+Ya1nu7k5vzzKuz3uhTPvDsTfxU0QqAwoX8sf8V8u03xs7id8i2nFS4Tdckj+1TPsdcfew3dWC0wtmcUHB+2Xa/xz3APtcjRhe8BHDCj4p0z4h/lHyJZqL8qdzauGXh7UpwviEpwC4NO8tTvDMO6w9l1huSPDOHzc+72X6exYf1r5HmvDX+PsAuCH3GXoUHX6r6jZXS+6KuxuAW3MfI7Vo3WHtG1wduC/uVgDuznmAdsWZh7WvjejMI7E3AnDfgcm00B2H2gYXZ3OTewjRDALiyvzcxtQnThWKtsCmEu8zgQHV6NMWKFMoRGQ83qMO2rdvX6tAbZrEsie+I24tOGy5O641qY0TEC0ma1unMp+Ljm9JaqMEooojyPq9bHtcQgtSExOILyoia3vZ9oTEFqQmJNDY05isHWXbGzVOJjUugeSCpmTtKtvetEkSqbEJtMxvSlZW2fakps1IjUkgKa85WbvLtrds1hSNTqBpbnOy9pRtb5PUhOjIBBofSCJrX9n2dsmNaexOICG7BVn7y7Z3bJFIbkQCsftbkpVdtv2IFol4XFFE7WtJ1oHD2xUhtaX3L3L3nlZk5R7eXiAxh9pld2uy8g5vP+BqdKi9OCuFrPycw9pz3UmkJnvbC3elkFVQdFh7fmQbUpO87Xk72pHliTysvTAqhdTm3vac7e3JKvrf0UMWMDA+iuQmjcv8zMbUN+L9gz3IGxUZCZyuqn/0vR8LpKvqn0r0+Rh4QFW/9b2fDdyqqovLW+dBaWlpumjRosCFN8aYMCQii1U1rbw2py5mZwLtSrxPAbbUoo8xxpgAc6pQLARSRaSTiEQBo4APS/X5ELjUd/fTQGBvoK5PGGOMqZgj1yhU1SMiE4DPgAjgZVXNEJFrfO3PATOBM4F1QA5whRNZjTGmoXPqYjaqOhNvMSi57LkSrxW4Pti5jDHGHM6ezDbGGFMpKxTGGGMqZYXCGGNMpaxQGGOMqZQjD9wFkojsADbW8uNJwE4/xvEny1Y7lq12LFvt1OdsHVQ1ubyGsCsUdSEiiyp6MtFplq12LFvtWLbaCddsdurJGGNMpaxQGGOMqZQVisOVHYc7dFi22rFstWPZaicss9k1CmOMMZWyIwpjjDGVskJhjDGmUg26UIjIwyKyRkSWi8gMEWlSQb8zRGStiKwTkUlByjZSRDJEpFhEKrylTUQ2iMgKEVkqIkGZsakG2ZzYb81EZJaI/Oz73rSCfkHbb1XtB99Q+k/62peLSN9A5qlhtsEiste3n5aKyN1ByvWyiGwXkZUVtDu5z6rK5sg+8227nYh8JSKrff9GbyynT833nao22C/gNMDte/0g8GA5fSKAX4AjgChgGdA1CNm6AEcDc4C0SvptAJKCvN+qzObgfnsImOR7Pam8/6bB3G/V2Q94h9P/BBBgIPBDkP47VifbYOCjYP7/5dvuiUBfYGUF7Y7ss2pmc2Sf+bbdGujre50I/OSP/98a9BGFqn6uqh7f2/l4Z9ErLR1Yp6rrVbUAmAoMD0K21aq6NtDbqY1qZnNkv/m28Zrv9WvAiCBsszLV2Q/DgdfVaz7QRERah0g2R6jqXLxTj1fEqX1WnWyOUdWtqrrE93o/sBpoW6pbjfddgy4UpVyJt8qW1hbYVOJ9JmV3vJMU+FxEFovIeKfDlODUfmupvpkQfd9bVNAvWPutOvvBqX1V3e0eKyLLROQTEekWhFzVEer/Lh3fZyLSEegD/FCqqcb7zrGJi4JFRL4AWpXTdIeqfuDrcwfgAd4sbxXlLPPLPcXVyVYNx6nqFhFpAcwSkTW+v3iczubIfqvBagKy38pRnf0QsH1VhepsdwneMYCyReRM4H0gNdDBqsGpfVYdju8zEUkApgM3qeq+0s3lfKTSfRf2hUJVT62sXUQuA4YBp6jvBF4pmUC7Eu9TgC3ByFbNdWzxfd8uIjPwnk6o8y88P2RzZL+JyO8i0lpVt/oOp7dXsI6A7LdyVGc/BGxfVaHK7Zb8JaOqM0XkGRFJUlWnB75zap9Vyel9JiKReIvEm6r6XjldarzvGvSpJxE5A5gInKOqORV0WwikikgnEYkCRgEfBitjZUQkXkQSD77Ge3G+3DsxHODUfvsQuMz3+jKgzNFPkPdbdfbDh8ClvrtRBgJ7D54+C7Aqs4lIKxER3+t0vL8zdgUhW1Wc2mdVcnKf+bb7ErBaVR+toFvN950TV+ZD5QtYh/dc3VLf13O+5W2AmSX6nYn37oFf8J56CUa2c/FW/nzgd+Cz0tnw3q2yzPeVEUrZHNxvzYHZwM++782c3m/l7QfgGuAa32sBnva1r6CSu9wcyDbBt4+W4b3hY1CQck0BtgKFvv/XrgqhfVZVNkf2mW/bx+M9jbS8xO+1M+u672wID2OMMZVq0KeejDHGVM0KhTHGmEpZoTDGGFMpKxTGGGMqZYXCGGNMpaxQGFMBEckOwDo7isjFlbS3FpGPqljHMBH5m7+zGVMRKxTGBFdHoMJCAdwCvFjFOj4GzhGROH+FMqYyViiMqYJvfoE5IjJNvPOXvFniydsNIvKgiCzwfR3lW/6qiFxQYh0Hj07+CZzgm6fg5nI2dz7wqe8zt4jIy77XPURkpYjEqffhpzl4h54xJuCsUBhTPX2Am4CueJ/sPq5E2z5VTQeeAh6vYj2TgG9UtbeqPlayQUQ6AbtVNd+36HHgKBE5F3gFuFr/N9TMIuCEWv80xtSAFQpjqmeBqmaqajHeYRE6lmibUuL7sXXYRmtgx8E3vm1dDvwH+FpV55Xoux3vsCTGBJwVCmOqJ7/E6yIOH3lZy3ntwffvy3eaKqoa28gFYkotSwWyKVsUYnz9jQk4KxTG1N1FJb5/73u9Aejnez0ciPS93o93isry/ESJIxURaQw8gXfqzeYlr3kAnQmdkYJNmLNCYUzdRYvID8CNwMEL1C8CJ4nIAmAAcMC3fDng8c1+dtjFbFU9APxy8II48BjwjKr+hHeE0n/6JloCGIL37idjAs5GjzWmDkRkA95hmv0yKY3vwnU/Vb2zkj4tgbdU9RR/bNOYqoT9DHfG1CeqOkNEmlfRrT3w52DkMQbsiMIYY0wV7BqFMcaYSlmhMMYYUykrFMYYYyplhcIYY0ylrFAYY4yp1P8DEY1wwe/VrQcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "m2 = nn.Sequential(MLPLayer(1, nn.ReLU),\n",
    "                   MLPLayer(1, nn.Tanh))\n",
    "torch_dmodel2_v = np.hstack([\n",
    "    jacobian(m2, torch.tensor([[vv]])).detach().numpy().flatten() for vv in v])\n",
    "\n",
    "truth_dmodel2_v = drelu(v) * dtanh(v)\n",
    "\n",
    "plt.plot(v, truth_dmodel2_v, label='Analytic')\n",
    "plt.plot(v, torch_dmodel2_v, linestyle='--', label='Autograd')\n",
    "plt.legend()\n",
    "plt.xlabel('Input (x)')\n",
    "plt.ylabel(r'Derivative ($ \\frac{\\partial M}{\\partial x}$)')\n",
    "plt.title('2 layer (relu, tanh)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79ffacf8-2e1b-4ff9-b4e4-1c1176d4770e",
   "metadata": {},
   "source": [
    "## Onto the numerical solvers\n",
    "\n",
    "From the above, you should be getting more confident that we can autodiff through neural networks, but that's not entirely our goal here. Remember, we're trying to combine an ODE solver with a neural network where we will represent an unknown portion of an ODE as it's own neural network. To do that, we'll need to look at some simple numerical solutions to ODEs.\n",
    "\n",
    "To get started let's implement some [root finding algorithms](https://en.wikipedia.org/wiki/Root-finding_algorithms), which will be used to solve our ODE. Without any exposition, here's a Newton solver. Note that it takes an `fprime` argument. This is the derivative of the function which we are trying to find a root for. Now you may see where the `jacobian` method will fit into the ODE solver..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "75125582",
   "metadata": {},
   "outputs": [],
   "source": [
    "def newton_solve(f, fprime, x, tol=1e-5, max_iter=100, return_seq=False):\n",
    "    f_test = f(x)\n",
    "    x_list = [x]\n",
    "    it = 0\n",
    "    while torch.abs(torch.tensor(f_test)) > tol and it < max_iter:\n",
    "        x = x - (f_test / fprime(x))\n",
    "        f_test = f(x)\n",
    "        x_list.append(x)\n",
    "        it += 1\n",
    "    if return_seq:\n",
    "        return x_list\n",
    "    else:\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa836684",
   "metadata": {},
   "source": [
    "## Testing the solvers on a parabola\n",
    "For confidence, let's do an easy one, solving for the minimum of a parabola. No need for `jacobian`, neural networks, or anything fancy, we'll write it out from scratch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4684a68d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.1, 1.1, -0.0499959945678711, 1.0499998092651368)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqwUlEQVR4nO3deXyU9bX48c+ZJCSEAIaAyiZrQhI2A8omqIgLaMEFEK4KxaXgWku196r1quj9Wa3Xq0VrpW6IUkGtFqUKassiamWVxRBQLC4syhoIWchyfn98n2jEBALJzDPLeb9e/sFkMs/JmJznzHc5X1FVjDHGhEbA7wCMMSaWWNI1xpgQsqRrjDEhZEnXGGNCyJKuMcaEkCVdY4wJIUu6xhgTQpZ0jTEmhCzpGmNMCFnSNcaYEIr3O4BIsGLFiuPj4+OfBrphNypjalIBrCsrK7umd+/e3/kdTLiypFsL8fHxT5944olZLVq02BMIBKxZhTHVqKiokB07dmRv3779aWCE3/GEK6vaaqdbixYt9lnCNaZmgUBAW7RokY/7RGhqYEm3dgKWcI05Mu/vxPLKYQTlzRGRTBF5REQaBOP1jTEmmETkLhE5PRivHaw7UkfgV8DPgvT6Yami5KBsGX5D+pbhN6RXlByUmh47FsnJyTlHes6YMWParVixIgngtttuO7Hq13JycjLr4xrGRDsR6QhMAc4IxusHK+nOB7YA1wTp9cNORclB2XrxzZ2LV61PKV61PmXrJTd3Lt9XEPjRYxff3LkuifdIZs+e/WXv3r2LAaZOndqy6tdWrVqVF6zrGhNlrsKtxJgejBcPStJV1XLgOeA8EWkbjGuEm22jJncuWbMhhZLSACWlgZLVG1K+7HFJjx89tmZDyrZRkzvX5Tpz585t3KdPny5Dhw7t2KFDh64jRozoUFFRAUCfPn26LF68OPn6669vXVJSEsjMzMweMWJEB/ihis3Pzw/0798/Izs7OysjIyP7xRdfPO5I1/zNb37TskOHDl0HDBiQPnz48A533XXXCVWvB7Bt27b41q1bdwcoKytj0qRJbbp165aVkZGR/dBDDzUH+PLLLxNOOeWULpmZmdnp6eld582bl1JWVsbIkSPbp6end83IyMieMmXK8XV5f4ypCxGJByYA81X162BcI5hLxp4D7sT9APcF8TrhqaQ0oCWlQXnp9evXN/zkk0++aN++fWnv3r0z33333ZTzzjuvoPLrTzzxxJbp06cfn5eXl3vo9yYnJ1f8/e9//7xZs2YV27Zti+/bt2/mZZddtjcQqP7+u3jx4uQ333wzde3atbmlpaVy8sknZ+fk5BQeLr5HH320edOmTcvXrVu3vqioSE499dTM4cOH73vppZdShwwZkv/ggw9uLysrY//+/YGPPvooedu2bQmfffbZpwA7d+6Mq+PbY0xdnAe0Bn4ZrAsEbZZRVb8A/gFcJSJRP5vZ8tVHPk/s0aWAxISKap+QmFCR2LNLQctXH/m8rtfq3r37gU6dOpXGxcXRtWvXwk2bNtV6wrKiokJ+9atftcnIyMgePHhwxnfffdfgm2++qfHmu3DhwpRhw4btTUlJ0dTU1Ipzzjln75Gu8d577zV5+eWX0zIzM7NzcnKy9uzZE5+bm5vUr1+/Ay+99FLzX//6162WLl3aMDU1tSIzM7Pk66+/Tvz5z3/e9tVXX22SmppaXtufxZgguBrYAcwN1gWCnQyfAdoDZwX5Or4LJDbQlrMe2iTx8dUuLZP4eG016383BRIb1HnpWWJi4vevERcXR1lZWa3HiadNm9Zs165d8WvXrl2fl5eXm5aWVlpUVFTj78HhDi6Nj4/X8nKXIwsLC6XK98jDDz/8VV5eXm5eXl7uli1b1l5yySX7hg0bVrB48eINrVu3PjhhwoQOjz/+eFqLFi3K161blzt48OD9TzzxxPFjx45tX9ufxZj6JCInAsOB51X1YLCuE+yk+zqwhxiYUKsoOSjbxv6mk9aQALWsTLaOvbVTMCfSqoqPj9eSkpKfXCs/Pz+uefPmpYmJifrmm2823rp162Gr5DPPPLNg/vz5TQsLCyU/Pz/w3nvvHVf5tbZt25YsXbq0EcDMmTNTKx8/55xz8v/0pz+1qLz+mjVrEvft2xfYuHFjg9atW5fecsstO6+44oqdK1euTN62bVt8eXk5EyZM2Ps///M/W9auXZtcb2+CMUdnPG7I9ZlgXiSo24BVtVhEXgQmiUiaqu4K5vX89KOJtOp4k2vbRk3u3PrNP34W7Hguv/zyHVlZWdndunUrfOONN/5d+fg111yze9iwYZ27deuW1bVr18IOHToUH+51zjjjjMKhQ4fmZ2dnd23dunVJjx49DjRt2rQc4Lbbbvt2zJgxHWfNmpU2aNCgfZXfM3ny5J2bN29O7N69e5aqSrNmzUrfeuutTfPnz288derUE+Pj4zU5Obl85syZ/968eXPC1Vdf3b6iokIA7r333m+C9Z4YUxMREdzQwgeqGtSVPnK4j4/1cgGRHsBqYLKqPhrUiwXJ6tWrN/fs2XPn4Z6zZfgN6cWr1v+QdBMTKiQ+XrWsTKo+lpSTVRCKpFuf8vPzA02bNq3Yv39/oH///l2efPLJLwcOHHjYyTQTu1avXt28Z8+e7f2O42h4GyEWAVep6nPBvFbQJ7hUdQ3wMTDRu5tEpR9NpHmTZu3WvLbmR4/1qJ+JtFC74oor2mVmZmb36NEja/jw4Xss4ZooNBHIB2YH+0JBr3QBRORq4GlgoKp+EPQL1rPaVLrgjet663BbvvrI54HEBlrdY8GO1xg/RVqlKyLNgK3AM6p6Q7CvF6rWjrOBR3B3k4hLurUVSGyghw4dVPeYMSasjAMSgT+H4mIhWT+rqgXATOBSEUk90vONMSYUvCHPXwBLVXV1KK4Zyk0LfwaSgMtDeE1jjDmc/kBXQlTlQgiTrqquAlYQ5RNqxpiI8guggBBMoFUK9fbcPwPdgb4hvm5IFJcVy+nPnZ5++nOnpxeXFUtNjx0LEel90UUXdaj8d2lpKampqT0HDx7cGWDmzJlN77jjjhNrfgXYvHlzwtChQzseawx1tXXr1vgePXpkZmVlZc+bNy8lWNcZOXJk++eee+4nw1hVG/QcralTp6YFAoHeH3/8ccPKx9LT07tu2LChTj2jp06dmrZ58+aEurzGsYr1Vp4ichwwBpjpDYGGRKiT7ku4u8rEEF836IrLiuWs58/qvHTL0pSlW5amDJkxpHN+cX6g6mNnPX9W52NNvA0bNqzYsGFDw4KCAgF4/fXXm5xwwgnfd9S5/PLL8++///7th3uN9u3bl86bN++LY7l+fZg7d27jzp07F69fvz536NChIfslry8nnHDCwXvvvbflkZ9Zey+++GLzr776ypekWx/Kysr8DqEurgAaEsKhBQhx0lXV/bgJtbHRNqF27gvndl65bWVKSXlJoKS8JLBi64qUNo+06VH1sZXbVqac+8K5x9zacciQIfmvvPLKcQAvvfRSs5EjR+6u/NrUqVPTxo8ffxK4Sm/ChAltc3JyMtu0adO9surbsGFDg/T09K6Vzz/77LM7nXXWWZ1bt27d/f77729xzz33nJCVlZXds2fPzG+//TYOam7fWNvvr/Thhx82vPvuu9ssWLCgaWZmZnZBQYFMmzatWUZGRnZ6enrX6667rnXlc6tWYM8991zqyJEj2x/u56qoqGD8+PEnderUqeuZZ57ZeefOnTWuypk+fXpaTk5OZnp6etcFCxYkl5eX065du25bt26NBygvL+ekk07qtm3btp+8xpAhQ/I3btzYcPXq1YmHfu21115rcvLJJ2dmZ2dnDRs2rGN+fn5gwYIFyeeee24ngBdffPG4pKSkXsXFxVJYWCiV8a9bty55/PjxHSvfkzlz5jTOysrKzsjIyB49enT7oqIiAWjdunX3yZMnt6psyblq1aqkQ2NYvnx5Uvfu3bMyMzOzMzIysteuXZsIcM8995yQnp7eNT09veu99977k9aZF1xwQcfZs2c3rfz3yJEj20+fPv24mlp0zp07t3Hfvn0zhg8f3qFLly5da3qvw5k3xHktsExVV4by2n50/5qGu7uM8+HaIVNSXhIoOFgQV1JeUm/v8bhx43bPnj07tbCwUNavX5/cv3//AzU999tvv01Yvnx53pw5cz67++67W1f3nI0bNzb861//+sWyZcvW/+53v2udnJxcsX79+txTTjnlwLRp09KOFM/RfP+AAQOKbr/99q3Dhw/fk5eXl7tz5874e+65p/XChQs35ubmfrpq1apGL7zwwnFHumZ1P9cLL7xw3Oeff564YcOGT6dPn/7lypUraxy6KCwsDKxatSpv6tSpX06cOLFDXFwco0aN2vX00083A5gzZ06TrKysopYtW/6khAsEAtx8883bp0yZ8qNqd9u2bfH3339/y8WLF2/Mzc1d36tXr8L77rvvhIEDBxZ++umnyQCLFy9O6dy5c9HixYuTFyxY0CgnJ6fgyiuv3NOtW7fCGTNmfJGXl5cbCASYNGlSh9mzZ2/auHFjbllZGQ899FCLyus0b968LDc3d/1VV12144EHHjjh0Pgee+yxFtdff/23eXl5uWvWrFnfoUOHg++//37yX/7yl7QVK1asX758+foZM2a0+OCDDxpW/b4xY8bsnj17dipAcXGxfPDBB01GjRqVX7VF5+rVq9c///zzLfLy8hoArFmzptFDDz20ZdOmTZ8e6f9ZmDoNN4H2ZKgvHPKk602ofQxcG00Tau+Me+fzXi17FSTGJVbb2jExLrGid6veBe+Me+eYd6T17du36Jtvvkl86qmnmp199tn5h3vuiBEj9sbFxdG7d+/iXbt2VfvxdcCAAftTU1MrWrVqVZaSklI+evTovQDdu3cv3Lx580+qufr8/iVLljTq16/f/latWpUlJCQwZsyY3YsWLTriOG91P9eiRYsaX3rppbvj4+Np3759af/+/ffX9P2XXXbZboBhw4YVFBQUBHbu3Bl33XXX7Zw1a1YawLPPPtt8woQJNW6EmTRp0q6VK1emVCYfgIULFzbatGlTUp8+fTIzMzOzZ82alfbVV181SEhIoF27dsUrV65MWrlyZaObbrrp2wULFjRetGhR49NOO+0nwyurV69OatOmTUmPHj1KACZMmLBryZIljavEvgegT58+hV9//fVP3t/+/fsfePjhh1v+9re/PfGzzz5rkJKSogsXLkw5//zz9zZp0qSiadOmFRdccMGeBQsWNK76faNGjcr/8MMPmxQVFcmrr77atE+fPvtTUlK0phadAD169DiQmZkZtE5cIXAtsI8QTqBV8qvP7ZNAFjDIp+vXu6T4JH378rc3JcQlVLvjLCEuQd++/O1NSfFJddqRNnTo0L1333132/Hjx+8+3POSkn64Tk27Dhs0+GF3XCAQ+P57AoHA9+0ia2rfWNvvr8nhdkJWvRdXfrw+0s9V2/v3oc8TETp37lzavHnzsjfeeKPxqlWrGo0ePbrGG1pCQgI33njj9nvvvff7SUtVZeDAgfsqW1lu2rTp05dffvlLgAEDBhS88cYbTRMSEnT48OH7Pvroo5SPPvooZciQIT+5MRxpd2jlzx4fH6/Vvb/XXnvt7jlz5nzesGHDimHDhmW88cYbjWuz4zQ5OVn79eu3/7XXXmsye/bs1LFjx+724qm2Raf3PdX3jY4AItIcGA3MUNUaPy0Gi19J92VgL+5uExWKy4pl2MxhnUrLS6v96y8tL5VhM4d1qssKBoDrrrtu5y233LK1T58+RXV5ndqqqX1jXZ1++ukHPv7448bbtm2LLysr45VXXml25plnFgCkpaWVrly5Mqm8vJw5c+Yc8ZpnnHHG/ldeeaVZWVkZX375ZcK//vWvxjU996WXXkoFmD9/fkrjxo3L09LSygGuuuqqHddcc02HESNG7I6PP/xGzRtvvHHXkiVLmuzevTse4MwzzzywfPnylHXr1iUC7N+/P7BmzZpE72sF06ZNO/7UU08taNWqVdmePXviv/jii6TKs+xSUlLK8/Pz4wBOPvnk4i1btjSofJ0ZM2akDRo0qMaq/VC5ubkNsrKySu68887vzj333L2ffPJJw7POOqvgrbfeOm7//v2Bffv2Bd56663UwYMH/+Q1x44du3v69OnNly1b1rgysdbUorO28YSxCUAD3FBnyPnyBqpqITADGCkiLY70/EhQdSKtuq9XTq7VZSINoFOnTqX//d///V1dXuNo3Hbbbd8+88wzLXJycjIPN0F1tNq1a1d61113bTnjjDMysrKyuvbo0aPwiiuu2AswZcqULRdeeGHn/v37d6m6QqMm48aN29uxY8eSLl26dL366qtP6tOnT42JKjU1tTwnJyfzxhtvbDdt2rTNlY//x3/8R35hYWHcxIkTj9h+NCkpSSdOnPhdZdJt1apV2bRp0zaPHTu2Y0ZGRnbv3r0z165dmwQu6e7atSuh8oaSnZ1d1KVLl6LK45HGjx+/86abbmqXmZmZXVFRwZNPPrl59OjRnTIyMrIDgQC33nrrjiPFU+mFF15olpGR0TUzMzP7s88+S5o0adKugQMHFl522WW7evXqldW7d++scePG7TjttNN+csO++OKL9y1btqzxwIED91VW1JMnT96ZmZlZ3L1796z09PSuv/jFL9qVllZfVEQK7xSbicASVV3nSwyhaHhT7YVFsoFPgdtU9UFfgqil2jS8Of2509OXbln6fdJNjEusSIhL0NLyUqn6WJ/WfQoWX7nYejGEmcWLFydPnjy57YoVKzb4HUukC+eGNyJyNvAuME5VX/QjBt8+KqhqLrAQ1+A84g8jrDqRVjlp9s3kb9ZUfaxXy151mkgzwXHHHXecOHbs2E7333//Fr9jMUF3PbATeNWvAHyrdAFEZDRufPcCVX3Lt0COoLatHYvLiqVy+OCdce98nhSfpNU9Fux4jfFTuFa6ItIG+BJ4SFVv8yuOULV2rMnfgO24u0/YJl2goqKiQgKBwGETZlJ8kh46dFDdY8ZEK+/YpXBd2TAREHyaQKvk60ykqpYCTwHni0iHIz3fR+t27NjRtPIcL2PMT1VUVMiOHTuaAr5MUB2OiDTANbd5S1X/faTnB5PflS64fc93AJMA30r+wykrK7tm+/btT2/fvr0bPt+ojAljFcC6srKycDz9+yLgROAJn+Pwd0z3+yBEXsNtlGirqoc9ndYYY46WiCwETgLSVbXcz1jCpWp7AqjcJWKMMfVGRLoCZwBP+p1wIXwq3QCQC+xV1X5+x2OMiR4i8gRwFdBGVY+4CinYwqLSVdUK4I9AXxE51e94jDHRQUSaAuOBl8Ih4UKYJF3P87gG5zf6HYgxJmpMABoBj/kcx/fCYnihkog8jlvW0VZVQ9ZfwBgTfbxhyw3ADlUd4Hc8lcKp0gV4HNf9JxyXnBhjIsu5QGfCqMqFMKt0AUTkXSAT6KCqEX0AkzHGPyIyFzgFOElVw6bherhVuuCq3TbAhX4HYoyJTCLSCTgfmBZOCRfCs9KNAz4HvlLVM/yOxxgTeUTkEdykfDtV3ep3PFWFXaXrLV5+HDhdRE72ORxjTIQRkca4dbkvh1vChTBMup5ngULgZr8DMcZEnAlAE2Cqz3FUK+yGFyp5u0iuxpaPGWNqyVsmlgfsDtfdreFa6YK7SzXA9cA0xpjaGAqkA3/wO5CahG2lCyAi84AeQPtwm4E0xoQfEZkPdMPljCMeauqHcK50wVW7LYFRfgdijAlvIpKF2xDxp3BNuBD+lW4AWA/sA/poOAdrjPGViDyJm0Rrq6q1Pro+1MK60vW6jz2K21Vymr/RGGPClYg0x3UTeyGcEy6EedL1zAD2AJP9DsQYE7YmAQ1xRVpYC/ukq6oHgCeBi0Wko9/xGGPCi4gk4nafzVfVT/2O50jCPul6/giUA7/0OxBjTNgZgzt08hG/A6mNsJ5Iq0pEXsCd6NlGVfN9DscYEwZERICVuDX93SJhsj1SKl1wd7EUXJNzY4wBd+DkycAjkZBwIYIqXfj+GOWOQKdwXodnjAkNEXkT6IvrJlbkdzy1EUmVLsD/Am2xo9qNiXkikg38DHg8UhIuRF6lGwA+BYqBXpHyccIYU/9E5GngctxmiLA46bc2IqrS9TZLPIwbwznL32iMMX4RkROBccBzkZRwIcKSrudF4FvgVr8DMcb45kYgAfg/vwM5WhGXdFW1GHe651AR6eZ3PMaY0BKRFOB64HVV/dzveI5WxCVdz5O4kyWs2jUm9lwJpOIm1iNORE2kVSUij+H2W3dU1W/8jscYE3wikgB8BmxR1YhsghWplS64CbUA1gjHmFhyKdAOeNDvQI5VxFa6ACIyExgBnKSqe/yOxxgTPN6W30+AeKC7t5op4kRypQvwe9zW4Ov9DsQYE3RDccd3PRSpCRcivNIFEJG3gd5E0DZAY8zR89oAdMbN40TsmYmRXumCG9tpAfzc70CMMcEhIn1xzW0eieSEC9FR6QrwES7xdlHVMp9DMsbUMxF5DRiMm7/Z73c8dRHxla7Xf+EBXPexS30OxxhTz7zGNhfjGttEdMKFKKh04ftGOGuBCqBnJA+yG2N+TESeB0bh5m0iqs9CdSK+0oXvG+H8DugGXOBzOMaYeiIi7XGdxP4cDQkXoqTSBRCReNxOlW+B/tb20ZjIJyJ/xJ0WEzU7T6Oi0gXwJtAexHWRP9PfaIwxdeW1b7waeD5aEi5EUdL1TAe2A7/1OQ5jTN1NxrVv/L3fgdSnqEq6XtvHh4EhItLP73iMMcdGRNKAG4CXVfUzv+OpT1GVdD1PAruA//Y7EGPMMfsV0Aj4fz7HUe+iLumqagGum/z5ItLb73iMMUdHRI4Dfgm8pqrrfA6n3kVd0vU8DuwF7vQ5DmPM0bsJaALc53cgwRCVSVdV9wF/AC4SkR5+x2OMqR0RaYwbWnhTVT/xN5rgiMqk6/kDsB+rdo2JJNcDzYjSKheiaHNEdUTkfuA2XMPjT/2OxxhTMxFpBPwbWKmqQ/2OJ1iiudIFN6F2ALjL70CMMUd0A65b4BS/AwmmqK504UfVbo9onAk1Jhp4x6r/G1gRzVUuRH+lC26zxAFs3a4x4ewGoDlwj89xBF3UV7oAIvL/gNuxateYsONVuZuBZao6zOdwgi4WKl1wY7sF2NiuMeHoRiCNGKhyIUaSrqruAqYCo23drjHhQ0SaALcC81T1Y7/jCYWYSLqeh4F8onxm1JgIczOuyo2ZOZeYSbqqugc3zHCRiJzidzzGxDoRaYarcueo6nK/4wmVmEm6nkeB3UTxbhdjIsgtuB4LMTXXElNJ1+vJ8CAwVEQG+h2PMbFKRI7HDS3MVtU1fscTSjGxZKwqb6vhJiAPGGxnqRkTeiLyMK6xTVdVzfM5nJCKqUoXQFUPAPcDZwBn+xyOMTFHRNrgGtu8GGsJF2Kw0gUQkURgI/Ad0MeqXWNCR0T+DEwAMlR1s7/RhF7MVboAqloC3A2cAlziczjGxAwRyQCuAp6MxYQLMVrpAohIHLAWd+Pp5h3hbowJIhGZDVwAdFLVb/2Oxw8xWekCqGo57qj2LsB4n8MxJuqJSC/gUuCRWE24EMOVLoCICPAvoBWQ7h3hbowJAhGZB5wKdFTVfL/j8UvMVroA3gTa7UAbXNMNY0wQiMhZwHnAA7GccCHGK91KIvI20Bc3zrTH73iMiSYiEgCWAsfjVizE9CfKmK50q/gv4Dhc1WuMqV+XAr2BO2M94YJVut8TkenAWNyd+CufwzEmKnhr4tfjTubu5U1gxzSrdH9Q2XTDmuEYU3+uBToA/2UJ17FKtwoReRD4De6O/InP4RgT0UTkOOBz4BPgHNv56Vil+2O/A/YA/+stJzPGHLs7gGbAf1rC/YEl3SpUdS/uZIkhwPn+RmNM5BKRDrjWjTNUdaXf8YQTG144hIgkAOuACtzpwaU+h2RMxBGRWcAI3KajLX7HE06s0j2El2T/E8gEfuFzOMZEHBHpD4wBHrKE+1NW6VbDG8/9J9AN6BzrO2iMqS3vb+dDoB1u+WWBzyGFHat0q+EN+t9CjJ1Sakw9GAP0w22EsIRbDat0D0NEngHG4Vo/bvQ7HmPCmYgkAxv44XAAW5dbDat0D++3QDHwsN+BGBMB/hPXPOpmS7g1s6R7GKq6HbdD7Wcicp7f8RgTrkTkJFwPk1mqusTveMKZDS8cgbd3fB1QCvS0JWTG/FSVJWKZ1rvk8KzSPQLvPLVbgCzgBp/DMSbsiMjpuAm031vCPTKrdGvBWwbzNtAftwwmZo8aMaYqEYkHVgJNgSxVLfQ5pLBnlW4teEvIfgk0BB7wORxjwsl1QHdgsiXc2rFK9yiIyO+A24ABqvqR3/EY4ycROQG3ROxjYKg1takdS7pHQUQaAXnADuBUWxZjYpmIPAtcAXRX1Q1+xxMpbHjhKKjqAdykWg4wyedwjPGN11/hStxx6pZwj4JVukfJm1R7FzgF6GKTaibWeJNny3Hb5LNsu+/RsUr3KHnjVtfjJtVsp5qJRTcBPXE7zyzhHiWrdI+RiNyLa4YzRFX/6Xc8xoSCiLTBHTS5CBhuk2dHz5LuMRKRhsBaoBzX7LzE55CMCToReRV3qkpXVf233/FEIhteOEaqWoTboZaBa/RhTFQTkfOBkcB9lnCPnVW6deTtOb8YV+3aLK6JSiKSAnwKFAA5qnrQ55AillW6dfcroBD4s4jY+2mi1X3AScAvLOHWjSWJOvLaP94KnA5c5XM4xtQ7ETkVtw3+T6r6od/xRDobXqgH3trdBbhlNFleIjYm4nmnYy8DWgDZdl5g3VmlWw+8ZTMTcWt3H/M5HGPq0y24YuJGS7j1wyrdeiQitwP3A6NU9a9+x2NMXYhIJvAJ8HdVHelzOFHDkm498j6K/QtojVvHuMvnkIw5JiISB7wPdMH9LtuQWT2x4YV65B3lcxVuT/qj/kZjTJ3chGvaf7Ml3PpllW4QiMgU4C7cNsm5fsdjzNEQkU643Zb/AEbYVt/6ZUk3CESkAbACV/F2U9XdPodkTK14wwoLgB64YYUtPocUdWx4IQi8xeM/xy2zsdUMJpLcDAwCfmkJNzis0g0iEbkLmAKMVtVX/Y7HmMMRkSxgFTAPuNiGFYLDkm4QeasZPgLa4YYZrOG5CUve7+qHQAfcsIL9rgaJDS8Ekbea4edAY2Cat3PNmHB0O+40lGst4QaXJd0gU9VPgd8CFwJX+xyOMT8hIn1xq23+YsNgwWfDCyHgdR97F+gHnKyqn/kckjHA9y0bVwENgJ6qutffiKKfVbohoKoVuGGGEmCmN35mTDh4FOgEjLeEGxqWdENEVb/BHdt+Ku6jnDG+EpGLcUNeD6rqIr/jiRU2vBBiIvIcMB44y37RjV9EpC2umc1moL81Jg8dS7oh5o2hrQSSceO7O30OycQYEYkH/gnkAL1sjiG04v0OINaoaoGIjMWt331WRC60RegmGDa1GNQAmOP988JOO94/uKnFoAaXJZ346V+Kt3dOIjChSMst4YaYjen6QFVXAr8BhuO6ORlTr7yEOxc4w/tv7qYWg1LeKdn1wazi7Z0vSmxRvq75gMu955kQsuEFn3gbJeYAQ4HTVHWZzyGZKLKpxaC3ccm2ofdQ0Y6Kg6Uj9qxqnBKIl78ddzKNJK4IWNRpx/vD/Is09lil6xNvSOFKYBvwiog08zkkE8XKVRv+ev+GJvu0XB5rnEkjifM7pJhlSddH3skSo4FWwPN2hLupRxcCS4AigMcKv+Kj0nzuSelIZnwjvMeXeM8zIWR/5D5T1aW4w/9+hhvnNabOOu14/yBwCXBw8cE9/LHoa0YmHs/opBMrn3IQuNh7ngkhG9MNA9747ixgFHCOqv7T55BMhKucSPu6vHjQJXs/SWoRaMBfj+tJwx+GFSor3Z9Z4g0tq3TDgDe+ew2QB8wWkZN8DslEvjlFWj7whn3rk8pQnmiSVTXhgptgG8gPS8pMiFjSDROquh+4GNd45DURaXiEbzGmRhWq/Lbg8wbryw/wf4270D6uYRGQjzfGa/xjSTeMqOpG4AqgN/An679rjlW3XR/Of6NkR9yNDduWDm7QrHIooQ0/TK7ZRJpPbEw3DInIPcDdwGRVfdTfaEykEZEhwPwAvLU+7bSEOHfv/n5HGofsUvMt0BhlSTcMeUvHXsVVIReo6jyfQzIRQkTSgY+BrcAAVd3nc0jmEJZ0w5SINAI+wJ1Z1VdV83wOyYQ5EWkK/At3CnUfVf3C55BMNWxMN0yp6gFgBFAMvCkiaT6HZMKY1zlsFtAZGGkJN3xZ0g1jqvoVbkVDW+BvIpLkc0gmDHkTro/h+nhcb32aw5sl3TCnqh/ijvoZiGsFaf/PzKFuAa7FnQDxlN/BmMOzfroRQFVni0h74AHgC+BOfyMy4UJERgMPAS8Dd/gcjqkFm0iLEN5HyCeBicC1qjrN55CMz0RkEO6U6RXAEFUt9jkkUwuWdCOIN1nyOnA+MFpVX/M5JOMTEekOvA9sBwbasU+Rw5JuhBGRZOA9oBdwnk2axB5vqOlDoAK3FvcrfyMyR8OSbgTyGp4vAVoDZ6rqKp9DMiEiIsfjKtzjgUGqus7nkMxRspnwCKSqu4HzgL3AOyKS5W9EJhREJBV4B7eEcLgl3MhkSTdCqerXwNlAOfAPEenkc0gmiESkMfA2kAVcpKpLfA7JHCNLuhFMVT/DJd5EXOJt63NIJgi8cfw3gFOAS1X1HZ9DMnVgSTfCeR8xzwVSgYWWeKOLl3DfxJ3sO05Vrel4hLOkGwVUdQUu8TbHEm/UqJJwBwPjVfUln0My9cCSbpRQ1Y+Bc4A0XOK1I38imNdlbi5wJi7hvuhvRKa+WNKNIt7JwpWJ932vt6qJMCJyHDAfN6Twc0u40cXW6UYhEcnBLS0qx50uvNbnkEwtiUgLXMLtBlymqq/6HJKpZ1bpRiFvs8QgoAxYJCJ9fQ7J1IKItAEW45aFXWgJNzpZ0o1S3kkTA4HdwD9F5AKfQzKHISLdgI9wuwzPU9W3fQ7JBIkl3SimqpuB04BcYI6IXONvRKY6InI6bmtvHG5r72KfQzJBZEk3yqnqt7glR+8BT4nIFDvaPXyIyBjc+Pt2oL+qrvY5JBNklnRjgKoWAMOB54C7gFki0tDfqGKbOHfjzjVbhmvP+KXPYZkQsNULMcSrcH+DO4FiOW6yZpu/UcUe74b3LDAWeB6YpKol/kZlQsWSbgwSkYuAmbguZaNU9SNfA4ohItIOeA3IAW4Hfq/2RxhTbHghBqnq34D+uOPdF4nIdTbOG3wicg7uaJ1OuE8ZD1rCjT2WdGOUqq7Bda16F3gCeF5EUvyNKjqJSJyI3AnMA7YBp6rqmz6HZXxiSTeGqeoe3ATbPcAVwAoROdnPmKKNiLTErU64Dzdp1s9ryWlilCXdGKeqFao6BTgLSAE+FpFfioj9btSRiJwPrMYN5VwNXKGqB/yNyvjN/rAMAKq6EDgZN9zwB9wxQNap7BiISGMReQr4O/AtcIqqPmvjtwYs6ZoqVHUHbrhhEtAPWCsiV9okW+2JyGBgDa6y/T0u4eb6G5UJJ5Z0zY+o82egB/AJbj3pe9Ym8vBEJE1EngX+iWs0NFBV/8vW35pDWdI11VLVL3Dbh6/DrXJYKyJ3ikiSv5GFFxEJiMg4YD0wHrfxpKeqfuhvZCZc2eYIc0Qi0go3zjsK+DdwC/C3WB+jFJE+uPelH/AxMNFbimdMjazSNUekqltVdTTuVIpC3I6qf4jIqf5G5g8RaS8iM3CJtj1wJTDAEq6pDUu6ptZU9T3cCocbge7AUhF5TUS6+hpYiIjIiSLyOLARGA08CGSo6nRVrfA3OhMpbHjBHBMRaQz8CrgVaIyrfn/nnUwcVUSkPa5R0FVAPPAMcJ+qbvEzLhOZLOmaOhGRNGAyrvptiuvb+wfgbVUt9zO2uvLGbH+J6wZWAcwAHlDVz30NzEQ0S7qmXohIE+BaXPXbEjfh9idghtdIPSJ4/SdGATfgVm0UAE8B/6eq3/gZm4kOlnRNvRKRBOBiXNI6HXci8TzgBWBuXbbByhRpAMzx/nmh3q0Hq3vsGGKOxy2PGweMBJJxS8D+CLygqvuONWZjDmVJ1wSNiGTj1q5egTtwsQjX/OV1YN7RVMBecp2LO2wTYAlwCW4suepjP6tN4vUq88G4G8RwoBmQD7yMu0EsifUlcSY4LOmaoBOROFzVewlwEdDG+9I64B+4ZLkM+KqmRCdT5G3gDKDymKEi4CDQ4JDHFundOqyaGFoAp+KazwwB+uAOgtyLS+av48ahi479JzXmyCzpmpDy+jj0As7GJb+B/JA0v8P1LVgP5OHGhbcAW7ibF5EfJd0fU+AgRexkOU8xBWgHZHr/dcetpwU33LEMl+z/gatoS+v3pzSmZpZ0ja9EpAEuKfbFVZ9dcYny0IbqFSRQQRJxJPBDA55yXL1bgltf8GMHcWtqc3GJdimw0juo0xhfWNI1YcerhlvhqtXW3n9pxJNGV66mnAbfPzkON8CQQAnLuY+DbMZVx18DmyN92ZqJPpZ0TUQ4ZCKtuiGGIo5iIs0Yv9g2YBMp5lBzwsV7fCA/LB8zJizF+x2AMceoutULxoQ9q3RNpLgQN3xQxA9DCW2qeexCvwI0pjZsTNdEjGDtSDMmlCzpGmNMCNnwgjHGhJAlXWOMCSFLusYYE0KWdI0xJoQs6RpjTAhZ0jXGmBCypGuMMSFkSdcYY0LIkq4xxoSQJV1jjAmh/w8gs1s2uBXlHAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "f = lambda x: x**2\n",
    "fprime = lambda x: 2*x\n",
    "\n",
    "x = np.linspace(-1, 1, 100)\n",
    "plt.plot(x, f(x), color='black')\n",
    "x_init = 0.5\n",
    "x_min = newton_solve(f, fprime, x_init)\n",
    "plt.scatter([x_init], [f(x_init)], marker='X',s=100, c='crimson', label='Initial guess')\n",
    "plt.scatter([x_min], [f(x_min)], marker='X',  s=100, c='green', label='Minimum found by Newton solver')\n",
    "plt.legend()\n",
    "plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12279094",
   "metadata": {},
   "source": [
    "## Now let's test the solvers on a neural net\n",
    "\n",
    "To make this work, we'll need to define a neural network which is likely not useful in any real setting. I'm calling it `TroughLayer` because it essentially represents a \"trough\" where there is a low point in the middle of the domain surrounded by high walls. You'll see what I mean in a second. This is a carefully designed network with two sigmoid neurons whos weights and biases were chosen so that the output is what I want.\n",
    "\n",
    "Anyhow, despite being contrived, we can take the `jacobian` function and apply it to the network to get `fprime` that we can put into the Newton solver. As you will see, we find a good minimum. There was some tuning of the tolerance here, if you set it too low you will diverge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0e529714",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TroughLayer(nn.Module):\n",
    "    \n",
    "    def __init__(self, activation=nn.Sigmoid):\n",
    "        super().__init__()\n",
    "        self.layer = nn.Linear(1, 2)\n",
    "        self.activation = activation()\n",
    "        self.init_parameters()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.layer(x)\n",
    "        x = self.activation(x)\n",
    "        x = torch.sum(x)\n",
    "        return x\n",
    "    \n",
    "    def init_parameters(self):\n",
    "        with torch.no_grad():\n",
    "            self.layer.weight = nn.Parameter(torch.tensor([[ 1.5],\n",
    "                                                           [ -1.5]]))\n",
    "            self.layer.bias[:] =nn.Parameter(torch.tensor([[-12.0, -12.0]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "848687f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:5: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  \"\"\"\n",
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/numpy/ma/core.py:2830: FutureWarning: The input object of type 'Tensor' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'Tensor', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`.\n",
      "  order=order, subok=True, ndmin=ndmin)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(-21.994999980926515, 21.89499959945679, -0.05, 1.5)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAiCUlEQVR4nO3de1RU1/028GcPA4MIIiLe8AICwwCCIopivJtYaYLWqNUYNTZJtUmTlfomWStN2xhtX9u+/mz62jaNq2k1MTYa06ReYkzqW5WYmxdQUG5KghrFuyLIRWZmv3+cM4rK/TJnzpnns9aso2fOzHxh4GHP3vvsI6SUICIi9zBpXQARkTdh6BIRuRFDl4jIjRi6RERuxNAlInIjhi4RkRsxdImI3IihS0TkRgxdIiI3YugSEbmRWesCyPMcOnSoh9lsfhPAIPAPM1FDnACO2u32J1NSUi4090EMXbqH2Wx+s1evXnFhYWFXTSYTF+cgqofT6RQXL16MP3fu3JsApjb3cWzFUH0GhYWFXWfgEjXMZDLJsLCwMiifCJv/uA6qh/TNxMAlapr6e9KiHGXoEhG5EUOXWs1Zc1OcyfhpzJmMn8Y4a26Khva1RkBAQHJTx8yePXvAoUOH/AHgpZde6lX3vuTkZFt7vAZRe2PoUqs4a26Ks9Ofi67Ozg+szs4PPPvwc9GO6xWmO/ZNfy66LcHblE2bNp1MSUmpBoDVq1f3rntfdnZ2QUe9LlFbMHSpVUpnLomuySkMRE2tCTW1ppojhYEnkx5OumNfTmFg6cwl0W15ne3btwelpqbGTpkyZWBkZGTC1KlTI51OJwAgNTU1NjMzM+Dpp58Or6mpMdlstvipU6dGArdbsWVlZaa0tDRrfHx8nNVqjX/nnXe6NvWaL774Yu/IyMiEUaNGxWRkZES+8sorPeu+HgCUlpaaw8PDEwHAbrdj8eLFfQcNGhRntVrjV65c2R0ATp486Tts2LBYm80WHxMTk7Bz585Au92OGTNmRMTExCRYrdb4ZcuW9WjL94f0h1PGqH3U1JpkTW2HPHV+fn6nw4cPfxMREVGbkpJi+89//hP4ve99r8J1/+uvv35m3bp1PQoKCvLufmxAQIDzo48+OtGtWzdnaWmpecSIEba5c+deM5nqb29kZmYGbNu2LSQ3NzevtrZWDBkyJD45Obmysfr++Mc/dg8ODnYcPXo0v6qqSgwfPtyWkZFx/d133w2ZNGlS2e9///tzdrsd5eXlpi+//DKgtLTU9/jx48cA4NKlSz5t/PaQzjB0qVV6v//aibPTn7vd2r2bxddpGRxb0fv910609bUSExNvREVF1QJAQkJCZXFxsV9zH+t0OsXPfvazvl999VWgyWTChQsX/L777jtz//797fUdv2fPnsD09PRrgYGBEoB84IEHrjX1Grt27epSUFAQsHXr1hAAKC8v98nLy/MfOXLkjcWLF0fU1taaZs6ceXXUqFFVNput5vTp05bHHnusX0ZGRtn06dOvN/drIWNg9wK1isniJ3tvXFkszOZ6p5YJs1n22fg/xSaLX5unnlksllvP4ePjA7vd3ux+4jVr1nS7fPmyOTc3N7+goCAvNDS0tqqqqsGf+8Yu1Go2m6XD4QAAVFZWijqPEatWrTpVUFCQV1BQkHfmzJnchx9++Hp6enpFZmZmYXh4+M2FCxdG/vnPfw4NCwtzHD16NG/ChAnlr7/+eo85c+ZENPdrIWNg6FKrOGtuitI5L0bJBgJQ2u3i7JwXojpyIK0us9ksa2pq7nmtsrIyn+7du9daLBa5bdu2oLNnzzbaSh4/fnzFJ598ElxZWSnKyspMu3bt6uq6r1+/fjX79+/vDAAbNmwIce1/4IEHyv7617+GuV4/JyfHcv36dVNRUZFfeHh47fPPP39p3rx5l7KysgJKS0vNDocDCxcuvPab3/zmTG5ubkC7fRNIF9i9QK1yx0BafdTBtdKZS6LDt/3leEfX8+ijj16Mi4uLHzRoUOXWrVu/de1/8sknr6Snp0cPGjQoLiEhoTIyMrK6secZN25c5ZQpU8ri4+MTwsPDa5KSkm4EBwc7AOCll146P3v27IEbN24MHTNmzK1ugSVLllwqKSmxJCYmxkkpRbdu3Wp37NhR/MknnwStXr26l9lslgEBAY4NGzZ8W1JS4vvEE09EOJ1OAQDLly//rqO+J+SZRGMfp8g7HTlypGTw4MGXGjvmTMZPY6qz82+HrsXXKcxmKe12UXeff3JchTtCtz2VlZWZgoODneXl5aa0tLTYN9544+To0aMbHUwj73XkyJHugwcPjmju8exeoFbp/f5rJyxJsRWw+Dpdg2YDcj7IuWNfUvsMpLnbvHnzBthstvikpKS4jIyMqwxcak9s6dI9mtPSBdR+XXUebu/3XzthsvjJ+vZ1dL1EWmppS5d9utRqJoufvLvroL59RHQbuxeIiNyIoUtE5EYMXSIiN2LoUqtV26vF2LVjY8auHRtTba8WDe1rDSFEyg9+8INI1/9ra2sREhIyeMKECdEAsGHDhuCXX365V8PPAJSUlPhOmTJlYGtraKuzZ8+ak5KSbHFxcfE7d+4M7KjXmTFjRsTatWtD7t5fd4Gellq9enWoyWRK+frrrzu59sXExCQUFhY2+xTshp63pKTEty3P0VqespQnQ5dapdpeLSa+NTF6/5n9gfvP7A+c9Pak6LLqMlPdfRPfmhjd2uDt1KmTs7CwsFNFRYUAgA8//LBLz549b62o8+ijj5atWLHiXGPPERERUbtz585vWvP67WH79u1B0dHR1fn5+XlTpkypaPoRnqVnz543ly9f3rvpI5vvnXfe6X7q1ClNQrc92O31LtnRIgxdapXJ6ydHZ5VmBdY4akw1jhrTobOHAvu+1jep7r6s0qzAyesnt3ppx0mTJpVt3ry5KwC8++673WbMmHHFdd/q1atDFyxY0B9QWnoLFy7sl5ycbOvbt2+iq9VXWFjoFxMTk+A6/v7774+aOHFidHh4eOKKFSvCXn311Z5xcXHxgwcPtp0/f94HaHj5xuY+3uWLL77otHTp0r67d+8Ottls8RUVFWLNmjXdrFZrfExMTMJTTz0V7jq2bgts7dq1ITNmzIho7OtyOp1YsGBB/6ioqITx48dHX7p0qcFZSOvWrQtNTk62xcTEJOzevTvA4XBgwIABg86ePWsGAIfDgf79+w8qLS295zkmTZpUVlRU1OnIkSOWu+/74IMPugwZMsQWHx8fl56ePrCsrMy0e/fugMmTJ0cBwDvvvNPV399/aHV1taisrBSu+o8ePRqwYMGCga7vyZYtW4Li4uLirVZr/KxZsyKqqqoEAISHhycuWbKkj2tJzuzsbP+7azh48KB/YmJinM1mi7darfG5ubkWAHj11Vd7xsTEJMTExCQsX778nqUzH3zwwYGbNm0Kdv1/xowZEevWreva0BKd27dvDxoxYoQ1IyMjMjY2NqGh73VzMXSpXdQ4akwVNyt8ahw17fYzNX/+/CubNm0KqaysFPn5+QFpaWk3Gjr2/PnzvgcPHizYsmXL8aVLl4bXd0xRUVGnf/3rX98cOHAg/7e//W14QECAMz8/P2/YsGE31qxZE9pUPS15/KhRo6p+/vOfn83IyLhaUFCQd+nSJfOrr74avmfPnqK8vLxj2dnZndevX9+1qdes7+tav3591xMnTlgKCwuPrVu37mRWVlaDXReVlZWm7OzsgtWrV59ctGhRpI+PD2bOnHn5zTff7AYAW7Zs6RIXF1fVu3fve5pwJpMJzz333Llly5bd0dotLS01r1ixondmZmZRXl5e/tChQyt//etf9xw9enTlsWPHAgAgMzMzMDo6uiozMzNg9+7dnZOTkyt+9KMfXR00aFDl22+//U1BQUGeyWTC4sWLIzdt2lRcVFSUZ7fbsXLlyjDX63Tv3t2el5eX//jjj1/83e9+1/Pu+v70pz+FPf300+cLCgrycnJy8iMjI29+9tlnAf/85z9DDx06lH/w4MH8t99+O+zzzz/vVPdxs2fPvrJp06YQAKiurhaff/55l5kzZ5bVXaLzyJEj+W+99VZYQUGBHwDk5OR0Xrly5Zni4uJjTb1nTWHoUqt8Ov/TE0N7D62w+Fic9d1v8bE4U/qkVHw6/9NWn5E2YsSIqu+++87yt7/9rdv9999f1tixU6dOvebj44OUlJTqy5cv1/vxddSoUeUhISHOPn362AMDAx2zZs26BgCJiYmVJSUl97Tm2vPx+/bt6zxy5MjyPn362H19fTF79uwre/fubbKft76va+/evUE//OEPr5jNZkRERNSmpaWVN/T4uXPnXgGA9PT0ioqKCtOlS5d8nnrqqUsbN24MBYB//OMf3RcuXNjgiTCLFy++nJWVFegKHwDYs2dP5+LiYv/U1FSbzWaL37hxY+ipU6f8fH19MWDAgOqsrCz/rKyszs8+++z53bt3B+3duzfovvvuu6d75ciRI/59+/atSUpKqgGAhQsXXt63b19QndqvAkBqamrl6dOn7/n+pqWl3Vi1alXvX/ziF72OHz/uFxgYKPfs2RP4/e9//1qXLl2cwcHBzgcffPDq7t27g+o+bubMmWVffPFFl6qqKvH+++8Hp6amlgcGBspdu3Z1ee+990JtNlt8cnJy3NWrV815eXn+AJCUlHTDZrPdbOj71BI8OYJaxd/sLz9+9OPivq/1Tapx1Nxzv6+Pr/z40Y+L/c3+bTojbcqUKdeWLl3a79NPPy28cOFCgz+v/v63X6ehsyz9/G6fHWcymW49xmQy3VousqHlG5v7+IY0duanELcf6vp43dTXVfcxjbn7OCEEoqOja7t3727funVrUHZ2dud///vfDfZ7+/r64plnnjm3fPnyW4OWUkqMHj36+rZt2769+/hRo0ZVbN26NdjX11dmZGRcnzt3boTD4RB/+MMfTt99bFNnw7q+drPZLOv7/v7kJz+5MmbMmBsffvhhcHp6uvX1118vac4ZtgEBAXLkyJHlH3zwQZdNmzaFPPLII1fUesSqVatOzZgx4441jrdv3x4UEBBQb+OiNdjSpVaptleL9A3pUbWO2np/+2sdtSJ9Q3pUW2YwAMBTTz116fnnnz+bmppa1Zbnaa6Glm9sq7Fjx974+uuvg0pLS812ux2bN2/uNn78+AoACA0Nrc3KyvJ3OBzYsmVLk685bty48s2bN3ez2+04efKk71dffRXU0LHvvvtuCAB88skngUFBQY7Q0FAHADz++OMXn3zyycipU6deMZsbb3s988wzl/ft29flypUrZgAYP378jYMHDwYePXrUAgDl5eWmnJwci3pfxZo1a3oMHz68ok+fPvarV6+av/nmG3/XtewCAwMdZWVlPgAwZMiQ6jNnzvi5nuftt98OHTNmTIOt9rvl5eX5xcXF1fzyl7+8MHny5GuHDx/uNHHixIodO3Z0LS8vN12/ft20Y8eOkAkTJtzznHPmzLmybt267gcOHAh6+OGHrwMNL9HZ3Hqai6FLrVJ3IK2++12Da20ZSAOAqKio2l/96lcX2vIcLfHSSy+d//vf/x6WnJxsa2yAqqUGDBhQ+8orr5wZN26cNS4uLiEpKaly3rx51wBg2bJlZ6ZNmxadlpYWW3eGRkPmz59/beDAgTWxsbEJTzzxRP/U1NQGgyokJMSRnJxse+aZZwasWbOmxLX/kUceKausrPRZtGjR5aZez9/fXy5atOiCK3T79OljX7NmTcmcOXMGWq3W+JSUFFtubq4/oITu5cuXfV1/UOLj46tiY2OrXJdHWrBgwaVnn312gM1mi3c6nXjjjTdKZs2aFWW1WuNNJhNeeOGFi03V47J+/fpuVqs1wWazxR8/ftx/8eLFl0ePHl05d+7cy0OHDo1LSUmJmz9//sX77rvvnj/Y06dPv37gwIGg0aNHX3e1qJcsWXLJZrNVJyYmxsXExCT8+Mc/HlBbW3+joi244A3dozkL3oxdOzZm/5n9t0LX4mNx+vr4ylpHrai7LzU8tSLzR5lci8HDZGZmBixZsqTfoUOHCrWuRe+4tCO5Rd2BNNeg2XdLvsupu29o76FtGkijjvHyyy/3mjNnTtSKFSvOaF2LN2JLl+7R3KUdq+3VwtV98On8T0/4m/1lffs6ul4iLXFpR2oPTqfTKUwmU6OB6W/2l3d3HdS3j8io1MsutWhmA7sXqD5HL168GOy6jhcR3cvpdIqLFy8GAzjaksexpUv3sNvtT547d+7Nc+fODQL/MBM1xAngqN1uf7IlD2KfLhGRG7EVQ0TkRgxdIiI3YugSEbkRQ5eIyI0YukREbsTQJSJyI4YuEZEbMXSJiNyIoUtE5EYMXSIiN2LoEhG5EUOXiMiNGLpERG7E0CUiciOGLhGRGzF0iYjciKFLRORGDF0iIjdi6BIRuRFDl4jIjRi6RERuxNAlInIjhi4RkRsxdImI3IihS0TkRgxdIiI3YugSEbkRQ5eIyI0YukREbmTuiCcVQvQAEK/+VzZz25JjO2rb3GOdACrV2w0pZS2IvIAQwgzAH4AFgJ96c/3bDEDUczM1sb9dS2zH5zolpfymHZ8PQAeFLoBxAN7roOf2OEIIO5QArgBwDsBZ9XYKQC6AI1DeQNngkxBpTAghAPQFMByADUCEeusJIES9BWpUnhZ+D+Cl9n5S0RE5UKel6/qr09S2Ocd09LYlx/oACLjr1hlAEIBeAPqotx51nvtbADsBbAawhwFMnkAIYQIwAcAMANOg/Ny6XABQAqAUwFUA19TbDQA31VtNnX/boXwSrHtz1rOvvv3tqb2er0Nauh0SuqQQQgQCGARgGIAHAEyCEs4FAJYC2MzwJS2oYTsPwMsAYqF8UtsBYA+AAwCOSSlvaFaggTF03UgI0QnALAAvQgnjPQAWSClPa1kXeRchRDSAtQBGA8gGsBLAFillpaaFeQmGrgaEED4AHgfwGpSPZRlSys+1rYq8gRBiEpQuLgB4HsA6ftpyL4auhtQWx0dQBi8ypJT/1bgkMjAhxANQft6KAEztiP5KahpDV2NCiJ4AdgHoD2C4lLJI45LIgIQQQwFkAigGMF5KeVXjkrwWQ9cDCCEGADgI4CKAVCllhcYlkYGoA7rZUObTjpBSlmpcklfjGWkeQEp5EsAcAHFQZjUQtadVAKIAzGfgao8tXQ8ihHgTwEIAyVLKXI3LIQMQQqQC+BrAKinlC1rXQwxdjyKECAVQCOCwlPJ+reshfVPPMPsvgAQAUVLKco1LIrB7waNIKS8DWAFgkhBipNb1kO59D8B4AMsZuJ6DLV0Pow56nASwT0o5Tet6SL+EEP+F0pcbI6W8qXU9pGBL18OoMxdWA5gqhIhv6nii+gghEqCsqfBXBq5nYeh6ptehLB7yuNaFkG49DWUxmr9rXQjdid0LHkoI8SGANAB9pZR2resh/RBCdIayxOgHUsrHtK6H7sSWrud6C8o6ppO1LoR050Eo696u1boQuhdD13PtAHAZAFsq1FKzAJwH8JnWhdC9GLoeSh382Azg+0IIi9b1kD6oXQsPAviXlNKhdT10L4auZ9sG5WPiOK0LId14EEAn3F6+kTwMQ9ez7QZQBeAhrQsh3ZgGZeEkdi14KIauB5NSVkFZ9vEh9ZROogapl+C5H8Cn7FrwXAxdz7cdQCRuX9KeqCGJUC6GukvrQqhhDF3Pt1PdcgEcaorrZ+Q/mlZBjWLoejgp5Skol8Eeq3Ep5PkeAJAvpTyjdSHUMIauPuwFMJb9utQQdVrhWLBrweMxdPUhE0B3ADatCyGPlQJlqthurQuhxjF09SFT3XK+LjUkTd1+oWkV1CSGrj4UAygF+3WpYWkAvpVSnte6EGocQ1cHpLIU3GcARmtdC3mskQC+0roIahpDVz/2A+gnhOihdSHkWYQQ/QCEA/hS61qoaQxd/TioblM0rYI8kas/l6GrAwxd/cgCIAEM07oQ8jgjAVQDOKJ1IdQ0hq5OqFdzLQAwXOtayOOkAMiWUtZqXQg1jaGrLwfBli7VoS5yMwRAtsalUDMxdPXlIIDeQog+WhdCHiMCQBcAh7Utg5qLoasvrsE0tnbJJVndsqWrEwxdfclRt4M1rYI8yRAADgDHNK6DmomhqyNSygooZ6clal0LeYxkAAXqgvekAwxd/ckFQ5duGwJ2LegKQ1d/cgFYhRCdtC6EtCWECINyJtphjUuhFjBrXQA1T3HYGD8AW5Z1juq19EaxCUBccdiYowC2qIdMi7r42U3tKiQNuD7x5DR6FHkUtnR1QA3c7QDGjfANtgFAV2Ee5tqn3rarx5H3cF03j4NoOsKWrj5sgbLCWKcIn06wwIQfWHqsBuCEsnA11Pu3AEjXqEZyvwQA16As+0k6wdDVGbMQiPbphOOOSovWtZDm4gHkqUt/kk6we0EfpgHYB6AKAGLNnXHcUVn3/ir1/mnuL420oF4vLwHsWtAdhq4OqANkDwO4CQBRPgG44LyJcqfddchNANM5kOZVegAIBZCndSHUMgxdHVAHyD4A4AcAA32UbtxvHLfmw/sB+JADaV6Fg2g6xdDVh1sDaQAQdSt0b3UxdMLtgTTyDgnqli1dnWHo6lA/H/8qMwSOOyrtTR9NBpUAoAzAWa0LoZZh6OpD3YG0Kl9h2ueALMqxV1xx7QMH0ryNDUA+Zy7oD0NXB9QBsocA7FVvD0kg/+vasst193EgzatYARRqXQS1nOAfSn0SQvwOwP8CECClZDeDFxFCBAIoB/ALKeUKreuhlmFLV78KAPgCiNS6EHK7GHVbpGkV1CoMXf0qULc2TasgLVjVLUNXhxi6+uXqz2Poeh9X6J7QtApqFYauTkkprwK4ACBW61rI7awATkspK5s8kjwOQ1ffCsCWrjeygl0LusXQ1TeGrpdRF7ph6OoYQ1ffCgCECiG6a10IuU13AF3B0NUthq6+cTDN+3C6mM4xdPXNFboxjR5FRsLpYjrH0NW3kwDsuP2LSMZnhfKel2hcB7USQ1fH1NN/i8HQ9SZWAMU89Vu/GLr6VwSGrjfhzAWdY+jqXxGAGCEE30uDU9/jGADHta6FWo+/qPpXBMACoJ/WhVCH6wvAH2zp6hpDV/9cv4DsYjA+zlwwAIau/jF0vQdD1wAYuvpXCuAGGLrewAqgErwumq4xdHVOvUYWZzB4ByuAIl4XTd8YusbA0PUOnC5mAAxdYygCECGEsGhdCHUMIYQflEszMXR1jqFrDEVQ3suBWhdCHWYglPeYoatzDF1j4AwG4+PMBYNg6BqD6wwlhq5xud5bno2mcwxdA1Cvl3YRDF0jswK4JKW8onUh1DYMXeMoAtfVNTLOXDAIhq5xcNqYsTF0DYKhaxxFAHoLIYK0LoTal/qe9gZD1xAYusbh+oVkF4Px8LpoBsLQNQ5OGzOuWHXL0DUAhq5xFKtbhq7xxAKQ4HQxQ2DoGoSUsgrAKTB0jcgG4FspZbXWhVDbMXSNhTMYjMkGoEDrIqh9MHSNpQiAVQghtC6E2od6XbRYMHQNg6FrLEUAggGEaV0ItZv+UK6LxtA1CIausXAGg/HY1C1D1yAYusbC0DUe13SxQk2roHbD0DWWkwBqwdA1EhsA14JGZAAMXQORUtqhzNeNbepY0g0bgAJeF804GLrGU4jb/YCkf5wuZjAMXeM5BiBGvaYW6ZgQoiuAXmDoGgpD13jyAPiA/bpG4OomYugaCEPXeI6p2wRNq6D2wOliBsTQNZ5CAE4A8VoXQm1mA2AH8K3WhVD7YegajLrwzTdgS9cIBgEolFLWal0ItR+GrjEdA0PXCBIB5GpdBLUvhq4xcQaDzgkhggEMAJCjdS3Uvhi6xsQZDPo3SN2ypWswDF1jOqpuEzWtgtrC9d4xdA2GoWtM+VDWYBiicR3UeokArkO5GggZCEPXgKSUN6G0dodoXAq1XiKAXK65YDwMXeM6DCCZV5HQH/U9SwK7FgyJoWtch6FcQaK3xnVQy/WDcgUQzlwwIIaucWWr2yFaFkGtMkzdHtK0CuoQDF3jOqJuh2hZBLXKMCin/7Kla0AMXYOSUl6HsqB5sta1UIsNgzKIVq11IdT+GLrGlg0gResiqPnUQbRhAA5oXQt1DIausX0FIFII0VPrQqjZBgIIAXBQ60KoYzB0je1LdZumaRXUEq5BNIauQTF0jS0LyplpI7UuhJptOIAa3F6MngyGoWtg6kBMNtjS1ZP7AGSpZxWSATF0je9LAMOFEL5aF0KNE0J0htK9sFfrWqjjMHSN70sAnaCcVkqebSQAMxi6hsbQNb7P1e1YTaug5hgH5fp2X2hdCHUchq7BSSm/g3Kxyge0roWaNBZAtnpiCxkUQ9c77AIwjpfv8VxCCAuU7gV2LRgcQ9c7/AdAADiLwZONBmABsEfjOqiDMXS9wx4ADgD3a1wHNSwDQDWA/6d1IdSxBBem9w5CiC8A+EgpR2hdC91JXW/hBIACKeWDWtdDHYstXe/xEYBUIURfrQuhe8RCWXNhu9aFUMdj6HqPzep2pqZVUH0eUrcfaVoFuQW7F7yIEOIIgAop5X1a10K3CSG+BuAnpeTax16ALV3vshnAKHYxeA4hhA1AKoB3tK6F3IOh613eU7dzNa2C6noMysySDVoXQu7B7gUvI4TYC+VqszFSSofW9XgzIYQPgFNQzkJ7qKnjyRjY0vU+fwYQCSBd60IIGQD6AFincR3kRmzpehl1iccSKBc+nKJxOV5LnZv7FYDuAGKllHaNSyI3YUvXy0gpawH8BcD3hBA8LVg7E6EMoP0fBq53YUvXCwkhAqGcAVUEYJzkD4FbCSFMADKhnBAxkJda9y5s6XohKWUFgGUAxgCYpnE53ugxKJfl+SUD1/uwpeul1L7dQwB6AEiUUl7UuCSvIIQIA5APoADAWCmlU+OSyM3Y0vVSat/uPAAhAN5UP/JSB1L/0G0C0BnATxi43om/aF5MSpkD4EUAUwG8po6oUwdQv7f/F8AEAIuklEc1Lok0Yta6ANLcnwBEAFgCwCGEeJEnTbQv9SSIvwL4MYCVUsr1GpdEGmLoejkppRRCvADAB0rwJgghnpRSnu7I1xXLhB+ALep/p8ml8mZ9+zqyBncQQvSHsq7CGAD/G8CvtK2ItMaBNLpFCLEIwB+hXJH2fwD8SUp5ud1fRwnX7VAuUQMA+wA8DOCDu/Y9pNfgFUL0APAc1E8QAJ5mC5cAhi7dRQgRCeAPAH4AoAbAJ1DCcC+Ak+0xp1csEx9Dudx4J3VXFYCbAPzu2rdXLpW6OF1Z7bMdAOWqy+lQ1sj1BfAugJellCXaVUeehN0LdAcp5bcApgshEqD0Qc6AMtAGABeFEMegnEZcAuAcgKvq7RqASijhWXPX1gFA3rr9CgICuHVTgtYVtq2ihp5rILC+fzd2X0P/9ldvne66hUKZatcDytoJcQASAASrjz8N5ay/NVLKgrZ8XWQ8bOlSo9SpZEMAjAAwHIAVysBbH9wOqXZ6sTueUcKJWjQdmlpyArgIoBDAMQBHoVwENJ9n+VFDGLrUKkIIC5TFWrpCmevbFUor0A/KpcRdWwuUQTpxx80XFozECzDBAtePoGvrQA324y+w42ade2SdI1r775Y+plq9Vd11uwLgPIArnOlBLcXQJbe7ayCtvm6FKuh8II2oITw5grSwBQ0HLtT9o3F7+hiRYXAgjTxBfbMXiAyJLV3SwjQo3QeuPtJ9APrWs48roJHhsE+XNOEtZ6QR3Y2hS0TkRuxeICJyI4YuEZEbMXSJiNyIoUtE5EYMXSIiN2LoEhG5EUOXiMiN/j9N5vfaP68qbwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dtype = torch.float32\n",
    "v = torch.tensor(np.arange(-20.0, 20.0, step=0.1), dtype=dtype)\n",
    "\n",
    "m = TroughLayer()\n",
    "\n",
    "model_v = np.hstack([m(torch.tensor([vv])).detach().numpy().flatten() for vv in v])\n",
    "\n",
    "# Take the derivative fo the model\n",
    "dm_dx = partial(jacobian, m)\n",
    "# Easier application of the derivative of the model\n",
    "dm_dx_flat = lambda x: dm_dx(x)[0]\n",
    "\n",
    "x_init = torch.tensor([-8.5])\n",
    "x_min = newton_solve(m, dm_dx_flat, x_init, tol=1e-4)  # <--- NOTE: Have to tune tolerance here to get good performance!\n",
    "\n",
    "plt.plot(v, model_v, color='black')\n",
    "plt.scatter([x_init], \n",
    "            [m(x_init).detach().numpy()], marker='X',s=100, c='crimson', label='Initial guess')\n",
    "plt.scatter([x_min.detach().numpy()], \n",
    "            [m(x_min).detach().numpy()], marker='X',  s=100, c='green', label='Minimum found by Newton solver')\n",
    "plt.gca().set_ylim([-0.05, 1.5])\n",
    "plt.legend()\n",
    "plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa0bb775-40ea-4bf1-b506-fdea127c4f03",
   "metadata": {},
   "source": [
    "## Time for the linear reservoir!\n",
    "\n",
    "It's the hydrologist's favorite model! We can pretty easily solve this one analytically, so let's use it to make sure that our solvers are capable of producing good solutions. I'll leave it as an exercise for the reader on solving the ODE analytically, but it's a pretty simple one. As you can see we can solve this equation pretty well numerically. Sure, there's some discrepancy, but this is mainly showing how to get an end to end solution, rather than fine tuning each piece. I'll call this good enough. Let's move on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e335fd55-46ca-4d23-b755-d822a9b6ce67",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:5: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  \"\"\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'storage')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABCiUlEQVR4nO3deZzN9f7A8dfnzD5jFsNYBzPZYxaMNYSQIq4tahRSklu3br9baaFFVFqk5VYIJRclREgJKRVmbNmyzAwGjZkx+3rOzOf3xxmn2c0wxzFz3s/H4zyc7/7+fOc47/Pd3h+ltUYIIYT9Mtg6ACGEELYliUAIIeycJAIhhLBzkgiEEMLOSSIQQgg752jrACqrbt26OiAgwNZhCCFEtRIZGZmgtfYrbVq1SwQBAQFERETYOgwhhKhWlFKny5omp4aEEMLOSSIQQgg7J4lACCHsXLW7RiBKZzQaiY2NJTs729ahCCFsyNXVFX9/f5ycnCq8jCSCGiI2NhZPT08CAgJQStk6HCGEDWitSUxMJDY2lsDAwAovZ7VTQ0qpRUqpi0qpQ2VMV0qp95RSJ5VSB5VSHa0Viz3Izs6mTp06kgSEsGNKKerUqVPpMwPWvEawBBhUzvQ7gJYFr8nAR1aMhaSkJJYuXcqGDRusuRmbkiQghLia7wGrJQKt9Q7gUjmzDAM+12a/Az5KqYbWiGXDhg3Uq1eP+++/n7fnvG6NTQghRLVly7uGGgNnCw3HFowrQSk1WSkVoZSKiI+Pr/SGwsLCeKitC4mvujOz+UHi4uKuLmJxRWvWrEEpxbFjx656HRMmTGDVqlXlzjN79uwiwz169Ljq7Qlh72yZCEo7fim1lxyt9XytdZjWOszPr9QnpMtVv359GjZrgW9gJh3a57Ju7dpKr0NUzPLly+nZsycrVqyw6naKJ4Jff/3VqtsToiazZSKIBZoUGvYHzltrY163jefwJ835660O7F/6tbU2Y9fS09PZuXMnn376qSURbN++nT59+jBq1CjatGlDeHg4l3vFe+WVV+jcuTPt27dn8uTJFO8t78cff2T48OGW4R9++IERI0Ywbdo0srKyCA0NJTw8HIBatWpZ5pszZw5BQUGEhIQwbdo0azdbiGrPlolgHXB/wd1D3YAUrfUFa21s+IiRfBGRhzY6UPdgNMnJydbalM0ppaz2Ks/atWsZNGgQrVq1wtfXl7179wKwb98+3n33XY4cOUJUVBQ7d+4E4NFHH2XPnj0cOnSIrKwsvv322yLr69evH0ePHuXy6cDFixczceJEXn/9ddzc3Ni/fz/Lli0rssymTZtYu3Ytu3bt4sCBAzz99NNVtVuFqLGsefvocuA3oLVSKlYpNUkpNUUpNaVglo1AFHASWABMtVYsAE2bNiW6ufm00u2OPny7bp01N2eXli9fztixYwEYO3Ysy5cvB6BLly74+/tjMBgIDQ0lJiYGgG3bttG1a1eCgoLYunUrhw8fLrI+pRT33XcfX3zxBcnJyfz222/ccccd5cawZcsWJk6ciLu7OwC+vr5V3Eohah6rPVCmtb7nCtM18E9rbb80IWP/gY78iM63xrP3u4/h/vuv5+ZrtMTERLZu3cqhQ4dQSpGXl4dSijvvvBMXFxfLfA4ODphMJrKzs5k6dSoRERE0adKEl156qdR7nydOnMhdd92Fq6sro0ePxtGx/I+s1lpuoxWikuyq1tDIUaM465GKU90cOjgfIyMjw9YhWYXW2mqvsqxatYr777+f06dPExMTw9mzZwkMDOSXX34pdf7LX/p169YlPT29zLuEGjVqRKNGjXj11VeZMGGCZbyTkxNGo7HE/AMHDmTRokVkZmYCcOlSeXcwCyHAzhJBq1atWHK8Ic++7s0dS1PYvHmzrUOqMZYvX17kwi7AyJEj+d///lfq/D4+Pjz00EMEBQXxj3/8g86dO5e57vDwcJo0acLNN99sGTd58mSCg4MtF4svGzRoEEOHDiUsLIzQ0FDeeuuta2iVEPZBlfcr70YUFhamr6VjmhkzZjBz5kwA7r333hIXG6uro0eP0rZtW1uHYRWPPvooHTp0YNKkSbYORYhqobTvA6VUpNY6rLT57eqIAGDEiBGW999+u56cnBwbRiOupFOnThw8eJBx48bZOhQhaiy7qz4aEhJCh1aBLL4tgebNjWz9YTN3DBlq67BEGSIjI20dghA1nt0dESiluO2u4bRslUuthtn8teZ9W4ckhBA2ZXdHBAAjRo7is0e/YHCOPwlxFzCZTFe8LVEIIWoquzsiAOjatSvLz7tjjHPn1nwPfvnpJ1uHJIQQNmOXicBgMBA84k6iTJnUMTiz55Oltg5JCCFsxi4TAcCIkSPZV+88DR48Qv8635Kfn2/rkKq9woXfrsX27dsZMmQIAOvWreP118vvQ+JyCeqYmJgyn1uIiYnBzc2N0NBQy+vzzz8vd70vvfTSNT+HsG/fPh588EEAlixZwqOPPlpkep8+fbiW26ELCwgIICEh4ZrWUXjfV1ZF9tfatWs5cuRIpdddkc/W1fy9KvL5KkvxtsyYMYMtW7Zc1brGjh3LiRMnrmrZqmC3iaB3797sdTLhEXSJ9qEp7Pl1p61DEqUYOnToFSuIXi5BXV4iAGjevDn79++3vO6v4hIjJpOpxLjZs2fz2GOPVel2qrOrTQTWYDKZKvT5Kkvxtrzyyiv079//qtb1yCOPMGfOnKtatirYbSJwdHTEvfNgFn1am9uec2L1uvW2DqnG2L59O7feeit33303rVq1Ytq0aSxbtowuXboQFBTEqVOnAHMHNFOmTKFXr160atWqRPVRKPorOi4ujuHDhxMSEkJISIglAVz+tTht2jR+/vlnQkNDmTt3boXjLfxrc9WqVUVKWVx26tQpBg0aRKdOnejVq5el450JEybw5JNP0rdvX5555pkiy6SlpXHw4EFCQkIqFMcjjzxCWFgY7dq148UXX7SMDwgI4MUXX6Rjx44EBQVZtp2YmMjAgQPp0KEDDz/8cJklQL777js6duxISEgIt912GwC7d++mR48edOjQgR49evDnn3+WWC49PZ2JEycSFBREcHAwX3/9dYX314IFC+jcuTMhISGMHDmSzMxMfv31V9atW8dTTz1FaGgop06dKnO/RkdH0717dzp37sz06dPL3GezZs2idevW9O/fv0gbKvr3uvz5SklJISAgwHJmIDMzkyZNmmA0GivclssdKm3atIm7777bEsv27du56667APj+++/p3r07HTt2ZPTo0aSnpwPQq1cvtmzZUuqPievCmnVprPHq1KmTrirffvutxtwZjm7evLnOz8+vsnVfb0eOHCkyfLJuzzJfKZ99Y5kv5bNvyp23Mjw8PLTWWm/btk17e3vr8+fP6+zsbN2oUSM9Y8YMrbXW7777rn788ce11lqPHz9e33777TovL08fP35cN27cWGdlZelt27bpwYMHa621Xrx4sf7nP/+ptdb67rvv1nPnztVaa20ymXRycnKJ7V5errjo6Gjt6uqqQ0JCLK8dO3YUWV5rrb/66is9fvx4rbXWL774on7zzTe11lr369dPHz9+XGut9e+//6779u1racPgwYO1yWQqsc2tW7fqESNGWIYXL16s69atWyQGDw8PvWfPHq211omJiZa23XrrrfrAgQNaa62bNWum33vvPa211h9++KGeNGmS1lrrxx57TL/88sta678/y/Hx8UViuHjxovb399dRUVFFtpGSkqKNRqPWWusffvjBEmfhffj0009b/lZaa33p0qUK76+EhATLPM8//7wl/vHjx+uvvvrKMq2s/XrXXXfpzz77TGut9QcffFBkm5dFRETo9u3b64yMDJ2SkqKbN29e6b9X4c/X0KFD9datW7XWWq9YscKynyvalsvDRqNRN2nSRKenp2uttZ4yZYpeunSpjo+P17169bKMf/311y1/P6217t+/v46IiCjRzqtR/PtAa62BCF3G96pd3zPZv39/PD09SUtL49SpU0Tu2UNYly62DqtG6Ny5Mw0bmrugbt68OQMHDgQgKCiIbdu2Wea7++67MRgMtGzZkptuuqncLi63bt1qOa/v4OCAt7d3pWK6fGqostLT0/n1118ZPXq0ZVzhJ9JHjx6Ng4NDieUuXLhA8R71xowZwwcffGAZ7tOnj+X9l19+yfz58zGZTFy4cIEjR44QHBwM/P1EfKdOnVi9ejUAO3bssLwfPHgwtWvXLhHD77//Tu/evQkMDAT+LsudkpLC+PHjOXHiBEqpUgv4bdmypUhPc6WtvyyHDh3ihRdeIDk5mfT0dG6//fYS85S3X3fu3Gk5ArnvvvtKHG0B/PzzzwwfPtxScnzo0KFXXC+U/fcaM2YMK1eupG/fvqxYsYKpU6dWuC2FOTo6MmjQINavX8+oUaPYsGEDc+bM4aeffuLIkSPccsstAOTm5tK9e3fLcvXq1eP8+fN06tSp3PVbg10nAhcXF0aMGEHTY1/zxF357F/2b+hSM64VNI//uULzed0/FK/7q/7J6sKlpw0Gg2XYYDAUOfwtXjLaFiWkC2+ztFLY+fn5+Pj4lJlEPDw8Sh3v5uZW6vpKEx0dzVtvvcWePXuoXbs2EyZMKLLs5f13uYx3abGXRpdRlnv69On07duXNWvWEBMTUyQhXWnZK+0vMJ+CWbt2LSEhISxZsoTt27eXmOdK+7Uin4XS5rnav9fQoUN59tlnuXTpEpGRkfTr16/CbSluzJgxfPjhh/j6+tK5c2c8PT3RWjNgwABLPx3FZWdn4+bmdsV1W4PdXiO4LDw8nCbeCt/ATNrUOkReXp6tQ7IrX331Ffn5+Zw6dYqoqChat25d5ry33XYbH330EQB5eXmkpqYWmX756K6y6tevz9GjR8nPz2fNmjUlpnt5eREYGMhXX30FmL8gDxw4cMX1tm3blpMnT1YohtTUVDw8PPD29iYuLo5NmzZdcZnevXtbiiZu2rSJpKSkEvN0796dn376iejoaODvstwpKSk0btwYMF+HKc3AgQOLHL1cXv+V9heYr480bNgQo9FYpLBj4b9Refv1lltusRyNlFUYsnfv3qxZs4asrCzS0tJYv379Fddbnlq1atGlSxcef/xxhgwZYjlqqEhbiuvTpw979+5lwYIFjBkzBoBu3bqxc+dOy2ciMzOT48ePW5Y5fvw47dq1u2Kc1mD3iaBfv37895g7sUtakrnoZn76svQPtrCO1q1bc+utt3LHHXfw8ccf4+rqWua88+bNY9u2bQQFBdGpU6cSPZoFBwfj6OhISEhIqReLT506VeT20ffeew+A119/nSFDhtCvXz/L6azili1bxqeffkpISAjt2rXjm2++uWLb2rRpQ0pKSoWSU0hICB06dKBdu3Y88MADltMH5XnxxRfZsWMHHTt25Pvvv6dp06Yl5vHz82P+/PmMGDGCkJAQy5fS008/zbPPPsstt9xS5o+fF154gaSkJNq3b09ISIjllF5F9tfMmTPp2rUrAwYMoE2bNpbxY8eO5c0336RDhw6cOnWqzP06b948PvzwQzp37kxKSkqp2+jYsSNjxowhNDSUkSNH0qtXL8u0q/l7gfmX/BdffGHZT5VpS2EODg4MGTKETZs2WW7H9fPzY8mSJdxzzz0EBwfTrVs3y6nQuLg43Nzcytyf1mZ3ZahL88QTTxC46DuGuPixtbUvD/1SsQ/NjaQ6lqGeMGECQ4YMYdSoUbYOxWrmzp2Lp6en5VkCIUozd+5cvLy8qqzUupShvgrh4eGsyb4IwE3HzpFZQ3suE9ffI488UuR6iRCl8fHxYfz48TbbvhwRYD6H2LZVazYF5eDfJ45deVPp+e93qnQb1lYdjwiEENYhRwRXQSnFPePCifVKxqluDt5Rq20dkhBCXDeSCAqEh4fz/HfZzHjTh97zL0in50IIuyGJoECLFi3IaRbKzP3JJOfmsmrVKluHJIQQ14UkgkLCw8Mt71csk9LUQgj7IImgEHO5A8Xvk7zYfO/vnI3YbuuQqhUpQ11S4TLUlw0bNqxIaYHyXKmialkKl7e+8847SU5OBuC9996jbdu2hIeHk5OTQ//+/QkNDWXlypWV3kZhV/u3r0jZ6/3797Nx48ZKr7siJb6vpuz2+fPnr/qW5+JtuZYy2B988AGLFy++qmWLk0RQSIMGDejffwAN6ubj5Gnir+Uv2zoku1fTylAnJyezd+9ekpOTLU/7ludqE0FhGzduxMfHB4D//ve/bNy4kWXLlrFv3z6MRiP79+8v8gDVjeZqE4E1mEwmGjVqdNWnjou35VrKYD/wwAOWhyKvlSSCYsLDw3ljlQNnX+9A0v9yrryAKEHKUJuVVob666+/5q677mLs2LFFCrpdLmFcPKbibcrOzraUhu7QoYPlad+srCzGjh1LcHAwY8aMISsry7Kuyx3WTJkyhaioKIYOHcobb7zBuHHj2L9/v6WMcmEnT56kf//+hISE0LFjR06dOkV6ejq33XabpRx2WU/rzpkzh6CgIEJCQixfcoV/nSckJBAQEFBiudJKY+fm5jJjxgxWrlxpOXLJyMjggQceoHPnznTo0MESR3n7oLDvvvuONm3a0LNnT0vRPqDM9S5ZsoTRo0dz1113MXDgQGJiYmjfvj1g7va28BPuffr0ITIyssJtqUgZ7LI+e+7u7gQEBLB79+5S21kpZZUlvVFfVVmGujSpqana281dR/p20yfr9tR/rN1k1e1VlRJlZ5dhfhW2bYh53Nl1f4878Yl53O8P/T0u45x53OqGlYpBylAXVbwMtdZa33bbbXrHjh36zz//1EFBQZbxxUsal9Wmt956S0+YMEFrrfXRo0d1kyZNdFZWln777bf1xIkTtdZaHzhwQDs4OFjKWzdr1sxSnrrw+/L2V5cuXfTq1au11lpnZWXpjIwMbTQadUpKitZa6/j4+CKl2y/Hu3HjRt29e3edkZGhtf677PWtt95qiSc+Pl43a9asRAxllcYu/BnQWutnn31WL126VGutdVJSkm7ZsqVOT08vdx9clpWVpf39/fXx48d1fn6+Hj16tGX7Za138eLFunHjxpa2REdH63bt2mmttX7nnXcsn+3z58/rli1bVqotFSmDXdZnT2utX331Vf3WW2+V+PtJGepr5OnpyaChd/HttxGEuzXk1PsLaD9skK3DqnakDHXJMtRxcXGcPHmSnj17opTC0dGRQ4cOWX5dVsQvv/xiOdXUpk0bmjVrxvHjx9mxYwf/+te/AHPNpcvlq69GWloa586dY/jw4QCW+k9Go5HnnnuOHTt2YDAYOHfuHHFxcTRo0MCy7JYtW5g4caKlNPTlstcVUZHS2GDu3GXdunWW6zfZ2dmcOXOmQvvg2LFjBAYG0rJlSwDGjRvH/Pnzy10vwIABA0pty913382AAQN4+eWX+fLLLy2fkYq2pbDSymBf6bNXr169cv/PVJQkglKEh4fz6vpveXJiMk0Dd2FKS8LRs+K12G8I95byxHifUnphazHZ/CrMvVHpy1eClKEuWYZ65cqVJCUlWfoGSE1NZcWKFbz66qs4OjpaTgtorcnNzS11nbqcSgBVte/K2sayZcuIj48nMjISJycnAgICSuwvXUbp6sLtK6t0dUVKY1/extdff11qpdqrLV1d3np37dpV5t+4cePG1KlTh4MHD7Jy5Uo++eSTSrWlsNLKYGdkZJT72auq0tVyjaAUt99+OyddnTH5ZuJYy8jRT1+ydUg1lj2VoV6+fDnfffcdMTExxMTEEBkZablOEBAQQGRkJADffPON5Rdk8TYVLj19/Phxzpw5Q+vWrYuMP3ToEAcPHqz0fijcXn9/f9auXQuYf4FmZmaSkpJCvXr1cHJyYtu2bZw+fbrEsgMHDmTRokVkZmYCf5e9Lty+si60llUau/g+uP3223n//fctCWvfvn0AFdoHbdq0ITo62nJNpHDfAGWt90rGjh3LnDlzSElJISgoqFJtKay0MthX+uwdP368UkeUZbFqIlBKDVJK/amUOqmUKnFpXCnlrZRar5Q6oJQ6rJSaaM14KsrZ2Znw8HE8utBA33+7Mev3OFuHVGPZSxnqmJgYzpw5Q7du3SzTAwMD8fLyYteuXTz00EP89NNPdOnSpcgv0OJtmjp1Knl5eQQFBTFmzBiWLFmCi4sLjzzyCOnp6QQHBzNnzhy6XGNPe0uXLuW9994jODiYHj168NdffxEeHk5ERARhYWEsW7asSEnmywYNGsTQoUMJCwsjNDTUcprlP//5Dx999BE9evQgISGh1G2WVRq7b9++HDlyxHKBdfr06RiNRoKDg2nfvr2lT+OK7ANXV1fmz5/P4MGD6dmzJ82aNbNMK2u9VzJq1ChWrFhRpJ/iiraluNLKYJf32du5cyf9+/evUJzlKuviwbW+AAfgFHAT4AwcAG4uNs9zwBsF7/2AS4Bzeeu19sXiyw4cOGDpz9jZ2blEX7A3mtIuDt3oil8grYneeecdvWDBAluHIWqgvXv36nHjxpU6rbIXi615RNAFOKm1jtJa5wIrgGHF8xDgqcwn7WoVJIKSN2PbQHBwsOUXRW5uLuvmV/x2RCEukzLUwloSEhKYOXNmlazLmheLGwNnCw3HAl2LzfMBsA44D3gCY7TW+cVXpJSaDEwGSu2FyVoeeughTh3Yw9Hn3PBt8AZ5Sf/CoXb967b9mq6sLhJrEldXV+677z5bhyFqoAEDBlTZuqx5RFDapfnityPcDuwHGgGhwAdKKa8SC2k9X2sdprUOK3w7nrWNGTOGbEd3lDI35tQnr163bV8NXc36lhBCVL2r+R6wZiKIBZoUGvbH/Mu/sInA6oJTWCeBaKDkFSgb8fT05J577mH6AhdipnchfsHRG/bL1tXVlcTExBs2PiGE9WmtSUxMLPemi9JY89TQHqClUioQOAeMBe4tNs8Z4DbgZ6VUfaA1EGXFmCrtwQcfpPfCT5nkm0+D3Bwu7YykTs9SO/mxKX9/f2JjY4mPj7d1KEIIG3J1dcXf379Sy1gtEWitTUqpR4HNmO8gWqS1PqyUmlIw/WNgJrBEKfUH5rMvz2itS7+3zEa6dOlCq/bt+DoqjgfdGxM1bxZ1epa819zWnJycLA8qCSFEZVj1yWKt9UZgY7FxHxd6fx4YaM0YrpVSioceeoiPpz3Ny8/FEVg7h7yEszjUbXLlhYUQohqQJ4srYNy4cUTlQ8IlA6YMR6LXl1/HXgghqhNJBBXg6+vLyJEjGfahiXqPm3g38oKtQxJCiCojiaCCHnzwQSKSckjJy+eLL76w1FIRQojqThJBBfXp04cWLVoAkJ+Zwr73nrBtQEIIUUUkEVSQUopJkybh6QTxc53o0XgBxpORtg5LCCGumSSCSpgwYQKZ+Q5EHXcl94IH5xbMt3VIQghxzSQRVEKDBg0YNmwY9y/MI/bNUDI/P0Z+WoatwxJCiGsiiaCSHnvsMSIyM9llTMXFlE/cgq9sHZIQQlwTSQSVdOuttxIcHMynWecwuBvRe19D55be9Z4QQlQHkggqSSnF448/zrbcS9R55CCN7jxO1jdVUxNcCCFsQRLBVbj33nupU7cuX213Iv6IJ3+cTLZ1SEIIcdUkEVwFV1dXHn74YR7enEq9WWn8Z+PVdxQuhBC2JongKk2dOhXlaK7Z98svv7B3714bRySEEFdHEsFVatSoEaNHjwage31n3P43EuORn20clRBCVJ4kgmvw+OOPA/DxSFfadowh+7unbByREEJUniSCa9C1a1e6du3Kc9/mkrbbj8RlRvKSUm0dlhBCVIokgmv0+OOPs+FMNl8vrkfeGXcuLfra1iEJIUSlSCK4RqNGjaJRo0Z8mnUOgIQPl5OfKQ+YCSGqD0kE18jJyYmpU6fyizGZKN94/EfsI3v5I7YOSwghKkwSQRWYPHkyLi4u/OR9Ds+OCThlr0Tn5to6LCGEqBBJBFXAz8+P++67j1l70tm8ujbPf9sJ5exs67CEEKJCJBFUkaeffhqDwcCgr5N487tfiIyUTmuEENWDJIIq0rJlS+6++27L8OzZs8n7K9qGEQkhRMVIIqhCzz77LAAtvRz4oN1m8le3Q2dLxzVCiBubJIIqFBwczF133cWptDw8vfMwuOSSsWqBrcMSQohySSKoYs899xz5Gp79rytnXu5M3Js70EaTrcMSQogySSKoYt26daNfv378NyqZk2m5cD6etK9/sHVYQghRJkkEVvD888+TD/w36yygyVg1W7qzFELcsCQRWEHfvn3p1q0b63Pi8Zx4mIYjfiF7zYu2DksIIUolicAKlFI899xz5AGr9uZjTHEi85LcPSSEuDFJIrCSIUOGEBwczP9tT6Xlk5p5F3xtHZIQQpTKqolAKTVIKfWnUuqkUmpaGfP0UUrtV0odVkr9ZM14rqfLRwXZeXA628T7779Paqr0VSCEuPFYLREopRyAD4E7gJuBe5RSNxebxwf4LzBUa90OGG2teGxh1KhRtGzZEoDM1GSOzBxC1meTbRyVEEIUZc0jgi7ASa11lNY6F1gBDCs2z73Aaq31GQCt9UUrxnPdOTg48PzzzwMQ3tqNbh1+xiVvEXnnT9o4MiGE+Js1E0Fj4Gyh4diCcYW1AmorpbYrpSKVUveXtiKl1GSlVIRSKiI+Pt5K4VrHuHHjuPnmm1l8NIujO2oTv7IFSR+st3VYQghhYc1EoEoZp4sNOwKdgMHA7cB0pVSrEgtpPV9rHaa1DvPz86v6SK3IwcGBWbNmATB8YQ6pkXVJWfwNxpjzNo5MCCHMrJkIYoEmhYb9geLffrHAd1rrDK11ArADCLFiTDYxbNgwunbtyp95mazJuQimPC69/qGtwxJCCKASiUAp1Uwp1b/gvZtSyvMKi+wBWiqlApVSzsBYYF2xeb4BeimlHJVS7kBX4GjFw68elFK89tprALybeQb3nueo23EOOb9JR/dCCNurUCJQSj0ErAI+KRjlD6wtbxmttQl4FNiM+cv9S631YaXUFKXUlIJ5jgLfAQeB3cBCrfWhq2jHDa9v374MHDiQC/k5HPdKxKGWCdPOebYOSwghUFoXP21fykxK7cd8F9AurXWHgnF/aK2DrBteSWFhYToiIuJ6b7ZKREZGEhYWRrNaDkxp48lt8zbSuUd3W4clhLADSqlIrXVYadMqemoop+AW0MsrdKTkhV9xBZ06dWL06NGcTs/j2Yhknp0x3dYhCSFEhRPBT0qp5wA3pdQA4CtA7oG8CjNnzsTBwQGAH3/8kZ1ffkr2ZrlwLISwnYomgmlAPPAH8DCwEXjBWkHVZK1bt2bixIkA3FLfmW7pD+MU+wR550/YODIhhL2q0DWCG0l1vkZwWWxsLC1atCAnJ4fY5z3wyXMhK+9h6s6ZbevQhBA11DVfI1BK/aGUOljs9bNSaq5Sqk7Vhlvz+fv789hjjwEw4J08zi5sS8qSX8jee8TGkQkh7FFFTw1tAjYA4QWv9Zgf/voLWGKVyGq4559/nrp163I0K5tFmedAaxKmvYvOy7N1aEIIO1PRRHCL1vpZrfUfBa/ngT5a6zeAAOuFV3P5+Pgwe7b5VNCHmWdI8krDJ3gN2cufsG1gQgi7U9FEUEsp1fXygFKqC1CrYNBU5VHZiQceeICOHTuSST5bG0VRq2MCTmmfkp8pvZkJIa6fiiaCB4GFSqlopVQMsBB4SCnlAbxmreBqOgcHB9577z0AnvkljR/X+bDPaRYGdw8bRyaEsCeVumtIKeVdsEyy1SK6gppw11Bx48aNY9myZQC0a9eO/fv34+joaOOohBA1SVU8WYxSajDmZwj+pZSaoZSaUVUB2rs33ngDDw/zUcDhw4f56KOPyNm2GJ0nZ92EENZX0dtHPwbGAI9h7mdgNNDMinHZlcaNG/PCC38/n9fr6Iu4XHiA7KVTbBiVEMJeVPSIoIfW+n4gSWv9MtCdon0NiGv073//m+bNmwPwzX4j+TkG0r/9FeOZCzaOTAhR01U0EWQX/JuplGoEGIFA64Rkn1xcXJg7dy4AL/2WzqIXbiL1pzrEPzmH6vb0txCieqloIlivlPIB3gT2AjHAcivFZLeGDBnCoEGDAJh24TSpBk3WTxGk/W+DjSMTQtRkV0wESikD8KPWOllr/TXmawNttNZysbiKKaWYN28eLi4uJGojL6Ycx/3mS7hevAdTzEFbhyeEqKGumAi01vnA24WGc7TWKVaNyo61atWKl156CYD1OfEYe5zFuX46Oav/ZdvAhBA1VkVPDX2vlBqplFJWjUYA8H//93+EhoYCMHRBNr9svgmXB7+xbVBCiBqroongScyd0eQqpVKVUmlKqVQrxmXXnJyc+PTTT3FwcOBAopFen0fx6fIVtg5LCFFDVSgRaK09tdYGrbWT1tqrYNjL2sHZs44dO/LUU09Zhp9++mnOHTtI1v+esF1QQogaqTJPFg9VSr1V8BpizaCE2YwZM2jZsiUAmWmp+GzsjhvzyFo5zcaRCSFqkoo+Wfw68DhwpOD1eME4YUVubm4sXLgQAJOGLzY5kHPenYS3f8YYc97G0QkhaoqKHhHcCQzQWi/SWi8CBhWME1bWu3dvpkwxl5qYsiWNL95qTm60gbgpL6ONUotICHHtKnxqCPAp9N67iuMQ5XjjjTfw9/cH4MmLR0h2MZATeYTkuW/aODIhRE1Q0UTwGrBPKbVEKfUZEIn0Q3DdeHl58fHHHwOQqvN4JG4/tQefxqfhc+T8uMDG0QkhqruK3jW0HOgGrC54dS8YJ66TwYMHM3nyZAD2mFLZa0oHBaZDP9g4MiFEdVfRi8U/aq0vaK3Xaa2/0Vr/pZT60drBiaLeeecdWrduDcDg5Um8uqQ9zlOX2TgqIUR1V24iUEq5KqV8gbpKqdpKKd+CVwDQ6LpEKCw8PDxYvnw5Tk5O5ObDjB8P8eKLLwKg8/NtHJ0Qorq60hHBw5ivB7Qp+DcSiAC+AT6wbmiiNB06dGD27NmW4ddff539S17D+EE9ciM32jAyIUR1VW4i0FrP01oHArOA0IL3i4Eo4LfrEJ8oxZNPPkn//v0B0FrjEPkWznUTMX77MHnJaTaOTghR3VT0rqFRWutUpVRPYACwBPjIalGJchkMBj777DPq1KkDQK9Pkji9tRlxnzTh4iOvyGkiIUSlVDQR5BX8Oxj4WGv9DeB8pYWUUoOUUn8qpU4qpcqsi6CU6qyUylNKjapgPHavUaNGfPrppwCkGDW3LP6LHCd3Mrf8TtKcRTaOTghRnVQ0EZxTSn0C3A1sVEq5XGlZpZQD8CFwB3AzcI9S6uYy5nsD2FyZwAUMGzaMhx9+GIBz+Tk8knAQ7Qjq+EtkfzP7CksLIYRZRRPB3Zi/qAdprZMBX+CpcpeALsBJrXWU1joXWAEMK2W+x4CvgYsVjEUU8s4779C+fXsAfsqMZ3eHGGr3P4dT/Evk/XXGxtEJIaqDij5Qlqm1Xq21PlEwfEFr/f0VFmsMnC00HFswzkIp1RgYDnxc3oqUUpOVUhFKqYj4+PiKhGw33N3dWbNmDd7e5qof4Zti+XNPQzJrvYpDg6Y2jk4IUR1UptZQZZXWm5kuNvwu8IzWOq+Uef9eSOv5WuswrXWYn59fVcVXY7Ro0YJly/5+sKzNuxeYuTfBhhEJIaoTayaCWKBJoWF/oHjt5DBghVIqBhgF/Fcp9Q8rxlRjDR482NLXMcCbb77JV199Re7vX5O58F7bBSaEuOFZMxHsAVoqpQKVUs7AWGBd4Rm01oFa6wCtdQCwCpiqtV5rxZhqtOnTpzNkyN99Br34z0k4HB6Lu/tyMpdOt2FkQogbmdUSgdbaBDyK+SLzUeBLrfVhpdQUpdQUa23XnhkMBpYuXUqLFi0AOBqfxleb6pO2ux4Xpv1E5tZdNo5QCHEjUloXP21/YwsLC9MRERG2DuOGdujQIbp27UpmZiYAH7bvw+1/mVAebjRe/yEuQS1tHKEQ4npTSkVqrcNKm2bNU0PCRtq3b8/ixYstw48e2s4Rfy90VgbZ8wdjOrXXhtEJIW40kghqqLvvvpsXXngBMN+qNWr/JlzGxOPd/QR5629HG422DVAIccOQRFCDvfLKK9x3330A5KLpufgU6WfrkBc4E+XkZOPohBA3CkkENZhSioULF9K3b18ATqfnU2d6Cru929g4MiHEjUQSQQ3n7OzM6tWruflmc5mnXKOJ4cOHc+TIEXK2LSbrvZ7oPJONoxRC2JIkAjvg4+PDxo0badCgAQDJycmED70Dx5NTcKu7k7Q3w6lud48JIaqOJAI70axZMzZs2ICHhwcA+0+d4Y3VzUj5rRHx75wj8cUPJRkIYackEdiRjh078uWXX+Lg4ADA89+d4KltTcDRiZSPVpL0xgIbRyiEsAVJBHbmzjvvtHRoA7Dg8C7e9zOCMzhfeo7M+WNsGJ0QwhYkEdih8ePH89FHf/c0Ou/gTraFpeIRnIiLWo0x+qgNoxNCXG+SCOzUlClTmDt3rmX4oXUHWfdDWzLrzMMpsK0NIxNCXG+SCOzYE088wezZf3dp+Y/Pj3Lf599jLHjqOPfAFnR+vq3CE0JcJ5II7Nyzzz7L9Ol/l6j+5ptvuPfee8ncsgDHfbeT/X5XtDHXhhEKIaxNEoHg5Zdf5j//+Y9leNWqVXy+wHwNwXjyLBenvoo2ykNnQtRUjrYOQNieUoo5c+ZgNBqZN28eAI98uY/zSR2YcKIOOn0b+Zk51P/0FQyuLjaOVghR1eSIQADmZDB37twip4lm/rCPZzySUd61yPzxFzJndSHvwikbRimEsAZJBMJCKcUrr7zC22+/bRm36nAEj6iz1BnzF7XaHsT4v57ovDwbRimEqGqSCEQJTz75JAsXLsRgMH88vj95mDFbjGSdr4du+xqq4MlkIUTNIIlAlGrSpEmsWLECp4J+C74/cpaAN/LZ59PKMo/xxB5bhSeEqEKSCESZRo8ezbp163BzcwPgYkIC/fr1Y/ny5eT8uADH37qS9ckIedZAiGpOEoEo16BBg9iyZQt169YFICcnh3vvvZcD330GBk1OxC4uTn2V/IwsG0cqhLhakgjEFfXo0YNdu3bRtu3fpSe6vrWTxRt7c+mHtqR//QPn7pyC8dRZG0YphLhakghEhdx00038+uuvDBgwwDJu0vIdPFM3C0NAI4wnTmBaHkr2+jk2jFIIcTUkEYgK8/HxYcOGDTz88MOWcasif2Xwxd243+OI202XcIh5hfyUJBtGKYSoLEkEolKcnJz46KOPeOedd1BKAXD0TAzNPvidY0e6Y2z1MQbv2jaOUghRGZIIRKUppfj3v//NunXr8Pb2BiA310jbWb8xeel3ZGRkAJC1eBI5P0qvZ0Lc6CQRiKs2ZMgQIiMjCQ0NtYxbtmwZXbp0IXr1B7g6LcL5/GSSZr+KzjXaLlAhRLkkEYhr0rx5c3799VcefPBBy7gjR44QNnEaZ6K6kby1CZfmbubcnY+Qe/KMDSMVQpRFEoG4Zm5ubixYsIDFixfj6uoKwKXUDAJe/J0347ti8K9HzoE/ibtnDFlfPCoPoAlxg5FEIKrMhAkT2LVrFy1atLCMe23DKu5MiCDj1vbUG/0HboYPyZw/yYZRCiGKs2oiUEoNUkr9qZQ6qZSaVsr0cKXUwYLXr0qpEGvGI6wvODiYyMhIxo8fbxl3LPYMIV9/zM/nOpIb543L4Bk2jFAIUZzVEoFSygH4ELgDuBm4Ryl1c7HZooFbtdbBwExgvrXiEdePl5cXS5YsYdWqVdSpU8cyfsB7P9NlYWMOJaYAoI25ZL3XG+ORn20VqhAC6x4RdAFOaq2jtNa5wApgWOEZtNa/aq0vP330O+BvxXjEdTZy5Ej++OMP7rjjDsu4A0eO0LlzZ5577jkyPn8Yt7o/o3+4naQPvkCbpDtMIWzBmomgMVC4+ExswbiyTAI2WTEeYQMNGzZkw4YNfPTRR7i7uwNgMpl47bXXGPLOT1w63YrENYFcevkTYgdOJuv3AzaOWAj7Y81EoEoZp0udUam+mBPBM2VMn6yUilBKRcTHx1dhiOJ6UEoxZcoU9u3bR69evSzjfzoSTZ3njjPftQ2qYV1y/zhBxtvDyXo3FNPpQzaMWAj7Ys1EEAs0KTTsD5wvPpNSKhhYCAzTWieWtiKt9XytdZjWOszPz88qwQrra9WqFdu3b2f+/Pn4+PhYxr+0eTW9zv3M2QHt8Okfi1u9A+T8tNJ2gQphZ6yZCPYALZVSgUopZ2AssK7wDEqppsBq4D6t9XErxiJuEAaDgYceeoijR48yduxYy/jzlxLpu/wTHv5fGy6cvBX38Jcs04x//i7PHghhRVZLBFprE/AosBk4CnyptT6slJqilJpSMNsMoA7wX6XUfqVUhLXiETeWBg0asHz5cjZt2kRAQIBl/KIdf9DoxZ8Yc889REdHY4raj8Nvt5A7rynZu/faLmAhajCldamn7W9YYWFhOiJC8kVNkpmZyezZs3n77bfJzs62jHdxcWHVI/24o90Wso57Ebe4LR6De+P7/GScWzazYcRCVD9KqUitdVhp0+TJYmFz7u7uvPrqqxw7dqzI6aKcnBzuencTnWbWJuLMQJSrCxkbdnDhH6PJercLpqj9tgtaiBpEEoG4YTRr1ozly5ezc+dOunTpYhl/4MxF+q5czgh9ktjOLfG9Iwa3envI+VJKVQhRFSQRiBtOjx49+O233/jiiy/w9//7GcMD507TZ9NiJm9zJyEqANXzVcu03IM/yhPKQlwlSQTihmQwGAgPD+fEiRPMmzeP+vXrW6at2Hsav+kxdJj0BIsWLSI3N5f8Hx/EcW9vkqePIffPaBtGLkT1I4lA3NBcXV3517/+RVRUFHPmzClSu+j48eNMmjSJ1jcFcCnZQH62I0mfn+Zsz/u5MPYpMn/cLLedClEBkghEteDu7s5TTz1FVFQUr7zyiqWLTICYcxdo/EoUnZ/35nCzpuDiTOaPv+EQMRLjh3XI3b3ZhpELceOTRCCqFS8vL6ZPn87p06d5/fXXadCggWXavouJ3PXzV9ySuItdHd1x8DHh4JKBoWmwZZ78jBRbhC3EDU0SgaiWvL29eeaZZ4iOjmb+/PlFOsOJy84g/Psf8PlnDjO+asOKLT+Sk5ODzs0m//NGZM9tScbmzei8PBu2QIgbhzxQJmqEvLw81qxZw5w5c9izZ0+J6X5+frx7Xz/uCf0SY6ILZ2d1wrFJQ7zCh+A5tg+OjeUBNVGzlfdAmSQCUaNordm9ezcffPABX375Jbm5uUWmt/F25PmuQXSIaYDrpTSUi4lmL0VgTGuK0/jdOPjWKWPNQlRv8mSxsBtKKbp27crSpUs5e/Yss2bNKvIswrEUE/d9v4+g45uYaowioZMJg0seGFMx1Pa1zJe7Zz06TzrKEfZBjghEjWcymVi3bh0LFy5k8+bN5Be7pbSllwP92zSjzsB7ueeee2jlbsLhlxBMSZ6kJM6i1tDbcOnQFqVK62JDiOpBTg0JUeDcuXN8/vnnLFq0iJMnT5Y6z4wBLXj+H7HknvYgblFbAByb1Kfu+FwcO92Dc48xKIMcTIvqRRKBEMVordm5cyeLFi1i1apVpKWlFZluUNDHqxb/qN2OProWPt6XaPLMPvLSnWB0HA61a5vXk58vSUFUC5IIhChHdnY2GzduZMWKFaxfv75IKWww97k6pKEHM4a64enqy1/D53PLLbfgkG/CtLAuppxmmBrPxr1/Hxx8vUvfiBA2JolAiApKS0tj3bp1LF++nO+//x6j0VjqfN7e3sy4M4Qnh+wg96IrZ2d1AoMDLp1uxufOXJyC+uDcfTTK2fU6t0CI0kkiEOIqpKamsmnTJtauXcvGjRtJTU0tMU/3+s4MrudJm8QWBJmccXQwEfDa7ygHTW7wLlyCzeW08xLOYvBtLKeRhM1IIhDiGuXm5rJ9+3bWrl3L+vXriY2NLTGPh3JgoK8n04ZrmtRx5GPnqfTt149u3bph+KQpBpc0Uk48gFPoIFx7hOJ0k7/ciSSuG0kEQlQhrTVHjhxh06ZNfPfdd/z8888lHlwrrE4tVy68mYejh4noad3QuQ4A+I5KwK2VA4Q+h2vfe69X+MJOSSIQworS09PZvn07mzdvZtu2bRw+fLjEPAYFfRq40jipNl2dvOnu6ku7p/fj0iiTM6nP4z/5FQwGA9kb3yU/5ntUs5E4dRqGY4O6NmiRqIkkEQhxHcXFxbF9+3a2bt3Ktm3bOHHiRKnzDWriyvA2TrzyUxZpru506tSJJd2jadouhoQ1gaRsb4xDQz88bqlHrQ4JGJoPweW2Cde3MaLGkEQghA2dO3eOnTt3Wl779+8nr4zKp48EeTA6xIGMyHo0uehHLeWAZ/e/qDf2JKmnbsJt2jGcnJzQmWlkf3YvNOiKY+hEHJs1lAvRolySCIS4gaSnp7Nr1y5+/fVXdu/eze7du7l48WKJ+RQQ4ODG6NYu/KNbPruPG3h6dyZt2rRhfIgfTw76kdw4N87O7oTycMO5TSC1+53EULcxTnfMxLGhVFQVf5NEIMQNTGvN2bNn2b17N3v27GH37t3s27ePlJSyO9HpXt+Zmf1d8cpxxbQ1gHoGZzBobnrzV5SjZn3iOzTp1ZdWrVqhlo6HrCiMTsNxCOyNU4umOAU0Rjk7XcdWCluTRCBENaO15vTp0+zbt4/9+/ezf/9+9u3bx9mzZ0ud31c50t7NnQf6QAtfA72/SOZyab3EVz3wDczg3PvtyT7pA4DnLX9Re+AFcvMH4/HoSsDce5vpxC4cbwrD4OVb6nZE9SWJQIgaIjk5mSNHjnDo0KEir/j4+DKXGd/GnZ7NHPlrrw+NcmoR6OBO6Mhz1O57gS1r6/J/fzYiMDCQYf4wscc35MR6cGZhbxybNsQt0B/P4EMYajfE+c4ZOPjWu46tFVVJEoEQNVxiYiJ//vknx44ds/x77NgxoqKiMJlK9qtQyxG6NXAhMUOzL8n8DMTUIA/enGTCdKYW8Z+bq65i0Nz01k6UAzy+ZhQugYE0btyYsamf4O15gUtH+2Go1RWXxg1wbJCHk28GDoFdcWrd+Xo2X1SAJAIh7JTJZOLMmTOcPHmyyOvUqVPExMSQmZlZYhkDUN/gjL/BlZvcXBh/Zw71PBXdFqZiwvx9ET/Tnbo3ZXJuXhDZUeZCe959zlF3eDRRe5rwgRpFvXr1aOlmZJDxPXIy65H01914NG2EW+MGONc+jcGnDk7Bd2Dw8rmOe8R+SSIQQpSgtSYhIYHo6GhiYmKIjo7m9OnTnD17ljNnznD27FmSkpJKXballwMd6jmTHeeGt9GFBgZnbu2aQ6deaWz53Zn7vjcvF97KjS9ezCL7TC3OvR1qWb7Zy7tx9MnlmcXdOV2nKXXq1GFqrV8I8DvD+QNBpKbcjJNfbTyaOFC70XmcGofhOXiqZfn89GQMtXysuXtqHEkEQoirkp6eztmzZ4mNjeXcuXOcO3eO8+fPF3kfFxdX5nMR9d0MDLvJFW+ciD/lgZ/BmToGJ8Y88hfetU10ejmP6BzzqaljT3jSunMacZ+3Ij3SfC3Co0M8DSb8ydm93rT8IBsfHx8a+nqx74UT5GU7cHB6D3JdnDC6OtOkfzSudXLY7zSctFY98PT0pOGlP/FNPQaeN+NUvyO16tfFpbYnBjdQ3n4oB8frti9trbxEYD97QQhRabVq1aJt27a0bdu2zHny8/NJTEzkr7/+srwuXLhAfHw8Fy9e5Gx8PJEXLxJfL564uAvkZOYw7c2S6xmzNJv2G93ISIjDkJFMbYMToRfy6PmLF0ejFTk5OcTFxeGdnUi+SYHRgFeuhtxcSMulfoNzuAak892sT3n3yFwA1ozwInhkKklbGnNpfSDpgFP9TJo+t5fsv9zouDAADw8PPDw8+F+vw7i6mti/Lpgskxfa1Zn6rePxbphKlOrIucBbcHd3x9eYTOOkP8h3bYip1s04e3ni6uOFey0TLt51ca3nj6ubGw4ODlb6q1Q9qyYCpdQgYB7gACzUWr9ebLoqmH4nkAlM0FrvtWZMQoiqZTAY8PPzw8/Pj6CgoHLn1VqTmZlJQkICCQkJJCYmkpCQQHx8PElJSVy6dAkuXSIpKYnDly7x86VLzPw6v8gpquMpJhzGg7eTCde83XgpR7yVI7d+qWjq68kPsX93LPTHhXza7/Xk/GlFgjEND+VAfYdsGmcbyM0ycPToUcu8fmMdcfIycVNCNnmp5ptv/TqcwKtdHPuXnuHBWYsBeDbMg9n/ziDjcG3+mt/ucstoOHcnygCN7nfiQp4RJycn9k5xp3mLbI6sDOTiKS+MBkWdm1MI7B7HHzF1mBvfHGdnZ+q7wL/8D5NjdOHQobbg7ETv/rfRoGE0ytGI0y0P4di0VRX9xUqyWiJQSjkAHwIDgFhgj1Jqndb6SKHZ7gBaFry6Ah8V/CuEqIGUUpZf4M2aVfzJZ601WVlZJCcnk5ycTFJSEikpKaSmppKSkmJ5fyIlhU5NUmmVlkZaWhob0tJYscn8Pk2dJj09HdNBE0wCRUaRbYS/5Ug9DxdOXjiBa74j7sqBrntMtIz3YsPJvzsoOp+uiT3sQVKMEyeNabgrB9ycoGmGI8oAWfnm0+1GoxFfbxNudXNoqBzwyXOGPPDyysWvbRIJcZrNm839Znf2cybo3VyMSS7UXlnH3OZ9n+PwzF5cGmWSdTi4eiYCoAtwUmsdBaCUWgEMAwongmHA59p8oeJ3pZSPUqqh1vqCFeMSQlQzSinc3d1xd3enUaNG17Su3Nxc0tPTi7wyMjIsr44ZGWRmZpKRkUFSVhbbMzPx9MtkYq9MMjMzuZSZycTdWWRlZZHdIpusLPP7rJfqkpOTQ45rNio7D601fd7LpqG7MykpUeQbHXDFQODvmtbna3E8/u/S5XGZJlav8iIv18DuzLM4KwP3jbkHk3Yi/+IlDO2aX+suLJc1E0FjoPBjkLGU/LVf2jyNgSKJQCk1GZgM0LRp0yoPVAhhP5ydnfH19cXX13pPT2utMRqNZGdnk52dTU5OTol/b83OZnxuLrm5ueTk5JBZ8O9Ng83j/MaNw6NOHavFWJg1E0FpXS8Vv0WpIvOgtZ4PzAfzXUPXHpoQQliPUgpnZ2ecnZ3x8vKydThXZM26tbFAk0LD/sD5q5hHCCGEFVkzEewBWiqlApVSzsBYYF2xedYB9yuzbkCKXB8QQojry2qnhrTWJqXUo8BmzLePLtJaH1ZKTSmY/jGwEfOtoycx3z460VrxCCGEKJ1VnyPQWm/E/GVfeNzHhd5r4J/WjEEIIUT5pG87IYSwc5IIhBDCzkkiEEIIO1ftqo8qpeKB01e5eF0goQrDqQ6kzfZB2mwfrqXNzbTWfqVNqHaJ4FoopSLKKsNaU0mb7YO02T5Yq81yakgIIeycJAIhhLBz9pYI5ts6ABuQNtsHabN9sEqb7eoagRBCiJLs7YhACCFEMZIIhBDCztlNIlBKDVJK/amUOqmUmmbreKxBKbVIKXVRKXWo0DhfpdQPSqkTBf/WtmWMVU0p1UQptU0pdVQpdVgp9XjB+BrZbqWUq1Jqt1LqQEF7Xy4YXyPbW5hSykEptU8p9W3BcI1us1IqRin1h1Jqv1IqomCcVdpsF4mgUP/JdwA3A/copW62bVRWsQQYVGzcNOBHrXVL4MeC4ZrEBPyf1rot0A34Z8Hftqa2Owfop7UOAUKBQQUl3Gtqewt7HDhaaNge2txXax1a6NkBq7TZLhIBhfpP1lrnApf7T65RtNY7gEvFRg8DPit4/xnwj+sZk7VprS9orfcWvE/D/EXRmBrabm2WXjDoVPDS1ND2XqaU8gcGAwsLja7RbS6DVdpsL4mgrL6R7UH9y539FPxbz8bxWI1SKgDoAOyiBre74BTJfuAi8IPWuka3t8C7wNNAfqFxNb3NGvheKRVZ0G87WKnNVu2P4AZSob6RRfWllKoFfA08obVOVaq0P3nNoLXOA0KVUj7AGqVUexuHZFVKqSHARa11pFKqj43DuZ5u0VqfV0rVA35QSh2z1obs5YjAnvtGjlNKNQQo+PeijeOpckopJ8xJYJnWenXB6Brfbq11MrAd83WhmtzeW4ChSqkYzKd1+ymlvqBmtxmt9fmCfy8CazCf4rZKm+0lEVSk/+Saah0wvuD9eOAbG8ZS5ZT5p/+nwFGt9TuFJtXIdiul/AqOBFBKuQH9gWPU0PYCaK2f1Vr7a60DMP/f3aq1HkcNbrNSykMp5Xn5PTAQOISV2mw3TxYrpe7EfJ7xcv/Js2wbUdVTSi0H+mAuVRsHvAisBb4EmgJngNFa6+IXlKstpVRP4GfgD/4+f/wc5usENa7dSqlgzBcJHTD/kPtSa/2KUqoONbC9xRWcGvqP1npITW6zUuomzEcBYD6F/z+t9SxrtdluEoEQQojS2cupISGEEGWQRCCEEHZOEoEQQtg5SQRCCGHnJBEIIYSdk0QgxBUopXyUUlML3jdSSq2ydUxCVCW5fVSIKyioYfSt1rpGl3IQ9steag0JcS1eB5oXFHo7AbTVWrdXSk3AXP3RAWgPvA04A/dhLhd9p9b6klKqOeYy6H5AJvCQ1tpqdWOEqCw5NSTElU0DTmmtQ4Gnik1rD9yLuQ7MLCBTa90B+A24v2Ce+cBjWutOwH+A/16PoIWoKDkiEOLabCvoByFNKZUCrC8Y/wcQXFAVtQfwVaGKqC7XP0whyiaJQIhrk1PofX6h4XzM/78MQHLB0YQQNyQ5NSTElaUBnlezoNY6FYhWSo0Gc7VUpVRIVQYnxLWSRCDEFWitE4GdSqlDwJtXsYpwYJJS6gBwmBrYTaqo3uT2USGEsHNyRCCEEHZOEoEQQtg5SQRCCGHnJBEIIYSdk0QghBB2ThKBEELYOUkEQghh5/4fvt2j9Uc48p8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Reservoir constant\n",
    "k = -0.1\n",
    "\n",
    "# Initial value\n",
    "S0 = 1.0\n",
    "N_time = 50\n",
    "\n",
    "# Analytic solution\n",
    "t_range = np.arange(N_time)\n",
    "S_analytic = S0 * np.exp(k * t_range)\n",
    "\n",
    "# Format the ODE for the Newton solver\n",
    "def f(x, S, k):\n",
    "    return x - S - k*x\n",
    "\n",
    "# Derivative of the equation above\n",
    "def fprime(x, S, k):\n",
    "    return 1 - k\n",
    "\n",
    "# Start with the initial value\n",
    "S_newt = [S0]\n",
    "S_newt_ad = [S0]\n",
    "\n",
    "for i in range(N_time):\n",
    "    # Partials are getting rid of the initial condition and parameter args\n",
    "    fprime_Sk = partial(fprime, S=S_newt[-1], k=k)\n",
    "    f_Sk = partial(f, S=S_newt[-1], k=k)\n",
    "    # Calculate the update\n",
    "    S_newt.append(newton_solve(\n",
    "        f_Sk,                  # Function with conditions at current time\n",
    "        fprime_Sk,             # Derivative with conditions at current time\n",
    "        (1+k)*S_newt[-1])      # Initial guess - I just made this one up\n",
    "    )\n",
    "   \n",
    "    # Now do the same, but with an autodiff calculated derivative\n",
    "    fprime_Sk_ad = partial(jacobian, f_Sk)\n",
    "    S_newt_ad.append(newton_solve(f_Sk, fprime_Sk_ad, torch.tensor((1+k)*S_newt[-1])))\n",
    "\n",
    "plt.plot(S_analytic, color='black', linewidth=3, label='Analytic')\n",
    "plt.plot(S_newt, color='crimson', linestyle='--', linewidth=2, label='Implicit Euler (Hand calculated derivative)')\n",
    "plt.plot(S_newt, color='orange', linestyle=':', linewidth=2, label='Implicit Euler (Autodiff calculated derivative)')\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('storage')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcd38efc",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Now the nonlinear reservoir!\n",
    "\n",
    "Of course, a linear reservoir is too easy. Let's change the ODE so that the conductivity term, K, is now dependent on the current storage. I'll just make a nice little function here, where basically, if you have low storage you start filling up and if you have high storage you start draining. Steady state is at a nice even value of 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b08f7e01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Reservoir conductivity (K)')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEGCAYAAACpXNjrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyMUlEQVR4nO3deXxV1bn/8c83M1MYQwiTgDKUSdTgXIeiiIqitlq8tsXWlmurrba9t8Xb9tfh/vq73s612lb0Ol6rxaqFKgpKFUeUoMjkQASUQAjzDBmf3x9nBw7hkJwk52QnJ8/79dqvc/bea+39bIY82WvtvZbMDOeccy6R0sIOwDnnXOrx5OKccy7hPLk455xLOE8uzjnnEs6Ti3POuYTLCDuA1qBXr142aNCgsMNwzrk2ZcmSJVvNLC/WPk8uwKBBgygqKgo7DOeca1MkfXysfd4s5pxzLuE8uTjnnEs4Ty7OOecSzpOLc865hPPk4pxzLuFCTS6SJkn6QFKxpBkx9o+Q9Iakckn/Fk9dST0kPS9pdfDZvSWuxTnn3GGhJRdJ6cBdwMXASOBaSSPrFNsOfAv4VSPqzgAWmNlQYEGw7pxzrgWF+Z7LqUCxma0BkPQYMAVYVVvAzDYDmyVd2oi6U4DzgnIPAi8B30/GBWzadZC/vBk85i2hyAdCSMHm2m21G+qUUYxtwTUdWT96W2294Jyxjnn4/JENAtLTRHqayEhLIyNdZDRyPScznQ6Z6WRnpJGWdvh6nHOurjCTSz9gfdR6CXBaAurmm1kpgJmVSuod6wCSpgPTAQYOHNiIsA/btPsgf3ixmPY4JU5WRhodMtPJyaz9TD+UfLrkZNC1Q+ahJTfqe/dOWeTnZpPXOZuMdO/ycy5VhZlcYv3qG++P6ebUjRQ2mwnMBCgsLGxSehg3oBtr/+vwTZWZYXY4EDPDINhmwTYOfUZvszrla6/IaPiYHKp/uHyw+VBMADVmVNUYVdVGVU0N1TVGZbVRXXN4PbLPqK6piSprVFbXUF5ZzYHKGg5WVnOwqpqDFdUcrKzhQGU1ByurOVBZzYGKatZt28fuA1XsOlDJgcrqmH92EvTqnE1+bjb5XXIo6JbDoJ6dIkuvTgzo0YHsjPSm/LU451qBMJNLCTAgar0/sDEBdcskFQR3LQXA5mZHGifpcLNWsKWlTt1qVVTVsPtgJbsORJbteyso23OQst3lbN59kLLdBynddZCij3ew60DloXppgoE9OjKqX1dG9c1ldN+ujO7XlR6dskK8GudcvMJMLouBoZIGAxuAqcC/JKDuHGAacHvwOTuRQbvGycpIo1fnbHp1zm6w7I59Fazdto91WyPLh2V7eXf9Tp5ZVnqozLD8zpw+pCenD+nJaYN70DOO4zrnWp4sxA4DSZcAvwPSgfvM7OeSbgQwsz9L6gMUAblADbAXGGlmu2PVDY7ZE5gFDAQ+Aa42s+31xVFYWGg+cGXrtXN/Bas27uad9Tt5c+12itZtZ39FNRKcMrA7F43qw8RR+RzXs1PYoTrXrkhaYmaFMfeFmVxaC08ubUtldQ3LN+zi5Q+3MG9lGe+V7gbgpIHduKZwAJPHFtAlJzPkKJ1LfZ5cGuDJpW1bv30/c5eX8viSEoo376VDZjqfPaUfX/v0EL+bcS6JPLk0wJNLajAzlq7fyaNvfcLf39lIVU0NF48p4FufGcrwPl3CDs+5lOPJpQGeXFJP2e6D3P/aOh5Z9DH7Kqq4+pQBfGfiMPJzc8IOzbmU4cmlAZ5cUteOfRXc+WIxD72xjoy0NL47cRhfPmsw6T7CgHPNVl9y8VekXUrr3imLH00eyYLvnMeZx/fk/z7zHlf96XU+2LQn7NCcS2meXFy7MLBnR+6dVsgd157E+u37uewPr/Lwoo/xO3fnksOTi2s3JHH5iX15/tvncOYJPfnR31fwzUffYW95VdihOZdyPLm4dqdn52zumzae708awbMrNnH1n9+gdNeBsMNyLqV4cnHtUlqa+Pp5x3P/9eNZv30/V9z1Gis37go7LOdShicX166dMyyPv339DNIkpt69iHc+2RF2SM6lBE8urt0b0SeXJ75+Jj06Z/HF/3mLJR97gnGuuTy5OAf07daBx6afTq/OWUy77y3eXb8z7JCca9M8uTgXKOjagcemn0H3Tpl85YHFfLxtX9ghOddmeXJxLkqfrjk88OVTqTFj2n1vsW1vedghOdcmeXJxro7j8zpz77RCSncd5F8fXkJFVU3YITnX5nhycS6GU47rwS+vPpGij3fw/+a+F3Y4zrU5oSYXSZMkfSCpWNKMGPsl6Y5g/zJJJwfbh0taGrXslnRrsO8nkjZE7bukhS/LpYjLT+zLV84azAOvr2P20g1hh+Ncm5IR1oklpQN3ARcCJcBiSXPMbFVUsYuBocFyGvAn4DQz+wAYF3WcDcBTUfV+a2a/SvpFuJR32yUjWL5hJzOeWM6ovl05oXfnsENyrk0I887lVKDYzNaYWQXwGDClTpkpwEMWsQjoJqmgTpkJwEdm9nHyQ3btTWZ6Gnf+y8lkZ6bx3VlLqaz2/hfn4hFmcukHrI9aLwm2NbbMVODROttuDprR7pPUPdbJJU2XVCSpaMuWLY2P3rUb+bk5/PyKMbxbsou7XiwOOxzn2oQwk0us2Zrqjn9ebxlJWcDlwONR+/8EHE+k2awU+HWsk5vZTDMrNLPCvLy8RoTt2qNLxxZwxbi+/OGfxf6CpXNxCDO5lAADotb7AxsbWeZi4G0zK6vdYGZlZlZtZjXAPUSa35xrtp9OGU1e52y+/8Qybx5zrgFhJpfFwFBJg4M7kKnAnDpl5gBfCp4aOx3YZWalUfuvpU6TWJ0+mSuBFYkP3bVHXTtk8pPLR/H+pj088Nq6sMNxrlUL7WkxM6uSdDMwD0gH7jOzlZJuDPb/GZgLXAIUA/uBL9fWl9SRyJNm/1rn0L+QNI5I89m6GPuda7KLRuUzYURvfvvCh1w6toC+3TqEHZJzrZJ8mlcoLCy0oqKisMNwbcT67fu58LcLOXdYHnd/sTDscJwLjaQlZhbzP4G/oe9cIw3o0ZFvfmYo81aW8Xrx1rDDca5V8uTiXBPccPZg+nXrwM/nvkdNjd/9O1eXJxfnmiAnM51/v2g4KzfuZva7PjSMc3V5cnGuiS4/sS+j++Xyq3kfcrCyOuxwnGtVPLk410RpaeI/LvkUG3Ye4KE31oUdjnOtiicX55rhzON78emhvbh74Rr2V1SFHY5zrUa9yUXSGZLuCsbp2iLpE0lzJd0kqWtLBelca3bLhKFs21fBI4s+CTsU51qNYyYXSc8CXyXykuMkoAAYCfwQyAFmS7q8JYJ0rjUrHNSDs07oyd0vf8SBCu97cQ7qv3P5opndYGZzzGyjmVWZ2V4ze9vMfm1m5wGvt1CczrVqt0wYxta9FTzyps/84BzUn1wujbVRUqakRwHMzN8gcw44dXAPzjy+J3e/vIbyKr97ca6+5HKLpOnRGyR1IjLe1/6kRuVcG/SN805gy55yZr9Td3Bv59qf+pLLBcBXJX0LQFIe8BKwxMxuaIHYnGtTzjqhJ58qyGXmK2v8rX3X7h0zuZjZdiIJ5vOSbgdeBh42sxktFZxzbYkkpp8zmOLNe1n4oc9u6tq3+p4Wu4pIcpkJfA1YDpRIuirY55yrY/LYvvTJzWHmy2vCDsW5UNU3n8tlUd/n1NlmwJNJici5NiwzPY2vnD2I/zf3fZaX7GJMf38dzLVPx0wuZvblY+1LFEmTgN8TmSzsXjO7vc5+BfsvIfIQwfVm9nawbx2wB6gGqmrnFJDUA/grMIjIZGHXmNmOZF+Lc7WmnjqQ372wmofeWMcvrz4x7HCcC0V9zWJfkFTf/uMlnd3UE0tKB+4CLibycua1kkbWKXYxMDRYpgN/qrP/fDMbV2eymhnAAjMbCiwI1p1rMbk5mVxxUj/mvLuRXfsrww7HuVDU97RYT+AdSfcFw71cI+lLkn4maSHwC6CsGec+FSg2szVmVgE8BkypU2YK8JBFLAK6SSpo4LhTgAeD7w8CVzQjRuea5AunHUd5VQ2PL1kfdijOhaK+p8V+D5wMPArkAROC9Q1E3t7/rJmtbsa5+wHR//NKgm3xljFgvqQldd7HyTez0uAaSoHezYjRuSYZ2TeXU47rziNvfuKPJbt2qb4OfcysGng+WBJNsU7ZiDJnmdlGSb2B5yW9b2Yvx33ySEKaDjBw4MB4qzkXty+efhy3/nUpr320lU8PzQs7HOdaVJhD7pcAA6LW+wN1X20+Zhkzq/3cDDxFpJkNoKy26Sz43Bzr5GY208wKzawwL8//47vEu3hMH3p0yuLhN3y8Mdf+hJlcFgNDJQ2WlAVM5fAjz7XmAF9SxOnALjMrldRJUhc4NCTNRGBFVJ1pwfdpwOxkX4hzsWRnpHNN4QBeeK+MTbsOhh2Ocy2qweQSPNWVcGZWBdxMZEj/94BZZrZS0o2SbgyKzQXWAMXAPcA3gu35wKuS3gXeAp4xs+eCfbcDF0paDVwYrDsXiqnjB1Bj8MTbJWGH4lyLkln9nY2S1gJ/A+43s1UtElULKywstKKiorDDcCnqmj+/wZa95fzzu+cSeXXLudQgaUmdV0EOiadZbCzwIXCvpEWSpkvKTWiEzqWwzxX2Z+3WfSz52N/lde1Hg8nFzPaY2T1mdibwPeDHQKmkByWdkPQInWvjLh1TQMesdGYV+Tsvrv2Iq89F0uWSniIyFMuvgSHAP4j0iTjn6tEpO4NLxxTwzLJS9ldUhR2Ocy0inmax1UTeev+lmZ1kZr8xszIz+xvwXAN1nXPA1YUD2FdRzdzlm8IOxbkWEU9y+ZKZ3WBmr9dukHQWgJl9K2mROZdCxg/qzqCeHXncm8ZcOxFPcrkjxrY/JDoQ51KZJK46uT9vrt3Oxp0Hwg7HuaSrb9TjMyR9F8iT9J2o5SdEhsh3zjXClHF9AfjHu3UHonAu9dR355IFdCYy/liXqGU38Lnkh+ZcajmuZyfGDejG7KWeXFzqq2+ysIXAQkkPmJkPjuRcAkwZ15ef/mMVq8v2MDS/S9jhOJc09TWL/S74eqekOXWXlgnPudRy6dgC0oTfvbiUV9+Q+w8Hn79qiUCcaw96d8nhrBN6MfvdDXx34jAfDsalrPomC1sSfO0BLDKzhdFLy4TnXOqZMq4f67cf4J31O8MOxbmkiedR5MuBDyU9LOlSSfVOMOacq99Fo/LJzkhjjjeNuRQWz9hiXwZOAB4H/gX4SNK9yQ7MuVTVJSeTCZ/qzdPLNlLtUyC7FBXXZGFmVgk8CzwGLCEyHIxzrokuHdOXrXsreGvt9rBDcS4p4hm4cpKkB4hM2PU54F6gIMlxOZfSzh+RR05mGs+uKA07FOeSIp47l+uBvwPDzGyamc0NZpFstiBxfSCpWNKMGPsl6Y5g/zJJJwfbB0h6UdJ7klZKuiWqzk8kbZC0NFguSUSsziVSx6wMzh/em2dXbPKmMZeS4ulzmWpmfzez8kSeOJg++S7gYmAkcK2kkXWKXQwMDZbpwJ+C7VXAd83sU8DpwE116v7WzMYFi08L4FqlS8YUsGVPuU8i5lJSfS9Rvhp87pG0O2rZI2l3As59KlBsZmvMrIJIf07dvpwpwEMWsQjoJqnAzErN7G2ITGYGvAf0S0BMzrWYz4zoTXZGGnOXe9OYSz31vedydvDZxcxyo5YuZpaIaY77AdHjj5dwdIJosIykQcBJwJtRm28OmtHuk9Q91smD6ZqLJBVt2bKliZfgXNN1ys7gvOF5PLuilBpvGnMpJp4O/Yfj2dYEsV5Nrvs/rN4ykjoDTwC3mlnt3dSfgOOBcUApkZkzjz6I2UwzKzSzwry8vEaG7lxiXDKmgLLd5byz3pvGXGqJp0N/VPRK8BLlKQk4dwkwIGq9P1D3rbJjlpGUSSSxPGJmT9YWCGbJrDazGuAeIs1vzrVKnxnRm6yMNJ5Z5jNUutRSX5/LbZL2AGOj+1uAMmB2As69GBgqabCkLGAqUHdAzDnAl4Knxk4HdplZqSIDMv0P8J6Z/aZO3NGPSV8JrEhArM4lRZecTM4Z6k1jLvXU1+fyX2bWBfhlnf6WnmZ2W3NPHDzOfDMwj0iH/CwzWynpRkk3BsXmAmuIvGNzD/CNYPtZwBeBz8R45PgXkpZLWgacD3y7ubE6l0yXjOlD6a6DLC3ZGXYoziVMPOOEvSWpq5ntApDUDTjPzP7e3JMHjwnPrbPtz1HfDbgpRr1Xid0fg5l9sblxOdeSLhiZT1Z6Gs8uL+XkgTGfP3GuzYmnz+XHtYkFwMx2Aj9OWkTOtTO5OZmccXxP5q8qI/L7lHNtXzzJJVYZHxnZuQS6aFQfPt62nw/L9oYdinMJEU9yKZL0G0nHSxoi6bdEBq90ziXIBSN7I8H8lf7UmEsN8SSXbwIVwF+JDLt/kBj9IM65puvdJYeTBnRj3ipPLi41NNi8ZWb7gKMGlXTOJdbEUX24/dn32bDzAP26dQg7HOeaJZ439F+U9M+6S0sE51x7MnFkPgDPe9OYSwHxdMz/W9T3HOCzREYlds4l0JC8zgzt3Zn5q8q4/qzBYYfjXLPE0yxWt/P+NUkLkxSPc+3axFH5/HnhGnbsq6B7p6yww3GuyeJpFusRtfSSdBHQpwVic67dmTiyD9U1xj/f3xx2KM41SzzNYkuIjEQsIs1ha4EbkhmUc+3VmH5d6ZObw/xVm/jsKf3DDse5JounWcwbf51rIWlp4sKR+Ty+ZD0HKqrpkJUedkjONckxk4ukq+qrGD3MvXMucS4a1YeHF33MK6u3MHGUt0C7tqm+O5fLgs/ewJlA7ePH5wMvAZ5cnEuC04b0oEtOBvNXlXlycW3WMZOLmX0ZQNLTwEgzKw3WC4C7WiY859qfzPQ0JozozYL3yqiqriEjPZ6BNJxrXeL5VzuoNrEEyoBhSYrHOUfkbf0d+ytZvM6nP3ZtUzzJ5SVJ8yRdL2ka8AzwYpLjcq5dO3dYHlkZaTy/qizsUJxrkgaTi5ndDNwNnAiMA2aa2TcTcXJJkyR9IKlY0lHjlwXTG98R7F8m6eSG6gbv4zwvaXXw6bMvuTanU3YGZ5/Qi/mrNvkcL65Niqsx18yeNLNvB8tTiTixpHQifTcXAyOBayWNrFPsYmBosEwH/hRH3RnAAjMbCizAB910bdRFo/Ip2XGAVaW7ww7FuUZr8D2X4JHk/yby1JiCxcwst5nnPhUoNrM1wXkeA6YAq6LKTAEeCqY7XiSpW/BAwaB66k4BzgvqP0jkybbv1xfItm3beOCBB47YNmrUKMaPH09lZSWPPPLIUXXGjRvHuHHj2L9/P7NmzTpqf2FhIaNHj2bXrl089dTR+fiMM85g+PDhbN26laeffvqo/eeccw5Dhgxh06ZNPPfcc0ftnzBhAgMGDGD9+vUsWLDgqP2TJk2iT58+rFmzhpdffvmo/ZMnT6ZXr1588MEHvPHGG0ftv/LKK+natSsrVqygqKjoqP3XXHMNHTt2ZOnSpSxduvSo/ddddx2ZmZksXryYlStXHrX/+uuvB+D111/nww8/PGJfZmYm1113HQALFy5k7dq1R+zv2LEj11xzDQAvvPACJSUlR+zPzc3lqqsiT9I/99xzbNp05ECQPXv25LLLIg9D/uMf/2Dbtm1H7O/Tpw+TJk0C4Mknn2T37iN/uPfv358LLrgAgFmzZrF///4j9g8ePJhzzz0XgEceeYTKysoj9g8bNowzzzwT4Kh/d3D43945J/Tg4uz3eeqvn7C4++FRkv3fnv/bg+T+22vqz71o8dy5/AK43My6mlmumXVJQGIB6Aesj1ovCbbFU6a+uvm1DyAEn71jnVzSdElFkorq/gU41xr06pxNl+xMduyrCDsU5xpNDbXnSnrNzM5K+Imlq4GLzOyrwfoXgVOj+3MkPQP8l5m9GqwvAL4HDDlWXUk7zaxb1DF2mFm9/S6FhYUW6zck58J2z8tr+Pnc93jle+czoEfHsMNx7giSlphZYax98U5z/FdJ10q6qnZJQFwlwICo9f7AxjjL1Fe3LGg6q30nx0cAdG3WhcEcL/P9qTHXxsSTXHKB/cBEIm/tXwZMTsC5FwNDJQ2WlAVMBebUKTMH+FLw1NjpwK6gqau+unOAacH3acDsBMTqXCgG9erE8PwuzPcJxFwbE8/AlV9OxonNrErSzcA8IB24z8xWSrox2P9nYC5wCVBMJMF9ub66waFvB2ZJugH4BLg6GfE711ImjsrnrheL2b6vgh4+x4trI+Lpc+kP/AE4i8jQ+68Ct5hZSb0V2xDvc3Gt2fKSXVx256v88nNjubpwQMMVnGshze1zuZ9IU1NfIk9k/SPY5pxrAaP75dK3a473u7g2JZ7kkmdm95tZVbA8AOQlOS7nXEASE0f14ZXVWzhQUR12OM7FJZ7kslXSFySlB8sXgG0N1nLOJczEkfkcrKzh5dVbwg7FubjEk1y+AlwDbAJKgc8F25xzLWT84B507ZDJ/JXeNObahnieFvsEuLwFYnHOHcOhOV7e9zleXNvQ4L9QSQ9K6ha13l3SfUmNyjl3lImj8tnpc7y4NiKeX3/GmtnO2hUz2wGclLSInHMxnTMsj+yMNOb5C5WuDYgnuaRFz4kiqQdxNKc55xKrY1YGnx7ai+dXlfkcL67Viye5/Bp4XdJ/SvoZ8DqRkZKdcy1s4sg+bNh5gJUbfY4X17rFMxPlQ8BngTJgC3CVmT2c7MCcc0eb8KnepMkHsnStX1zNW2a2iiMn8XLOhaBn52wKB/Vg/spNfOfCYWGH49wx+fOMzrUxE0fm8/6mPXyybX/DhZ0LiScX59qYiSP7ADB/lT815lqvepNLMNzLCy0VjHOuYQN7dmREny7e7+JatXqTi5lVA/sldW2heJxzcZg4qg9F67azbW952KE4F1M8zWIHgeWS/kfSHbVLsgNzzh3bxJH51BgseN9n8XatUzzJ5RngR8DLwJKopckk9ZD0vKTVwWf3Y5SbJOkDScWSZkRt/6Wk9yUtk/RU7fA0kgZJOiBpabD8uTlxOtdajeqbS79uHXhuhfe7uNYpnoErH0zCeWcAC8zs9iBpzAC+H11AUjpwF3AhUAIsljQneCz6eeC2YLrj/wZui6r/kZmNS0LMzrUakrhkTB8eeH0duw5U0rVDZtghOXeEY965SJoVfC4P7hCOWJp53ilAbdJ6ELgiRplTgWIzW2NmFcBjQT3MbL6ZVQXlFgH9mxmPc23OpWP7UlltPO8d+64Vqu/O5Zbgc3ISzptvZqUAZlYqqXeMMv2A9VHrJcBpMcp9Bfhr1PpgSe8Au4EfmtkrsQKQNB2YDjBw4MDGX4FzITuxf1f6devAM8s28rlT/Pcr17ocM7lE/fD/WFI+MD7Y9ZaZNdiLGDzC3CfGrh/EGZtihVXnHD8AqoBHgk2lwEAz2ybpFODvkkaZ2VEDMZnZTGAmQGFhoY8C6NocSVw6toD7X1vLrv2VdO3oTWOu9YhnPpdrgLeAq4nMSPmmpM81VM/MLjCz0TGW2UCZpILg+AVArGRVAgyIWu8PbIyKaxqRu6rrLBgi1szKzWxb8H0J8BHgY2S4lHXpmAIqq81fqHStTjxPi/0AGG9m08zsS0T6Qn7UzPPOAaYF36cBs2OUWQwMlTRYUhYwNaiHpElEOvAvN7NDY2BIygseBEDSEGAosKaZsTrXao3t35X+3TvwzPLSsENx7ghxzedSpxlsW5z16nM7cKGk1USeBrsdQFJfSXMBgg77m4F5wHvALDNbGdS/E+gCPF/nkeNzgGWS3gX+BtxoZtubGatzrVZt09irq7eyc39F2OE4d0g8oyI/J2ke8Giw/nlgbnNOGjRdTYixfSNwSdT63FjnMrMTjnHcJ4AnmhObc23N5DF9uXvhGuavLOOa8QMaruBcC4hnPpd/B+4GxgInAjPN7Pv113LOtZTR/XIZ0KMDT3vTmGtFGrxzkfRt4HEze7IF4nHONZIkLh3Tl3teWcOOfRV075QVdkjOxdV3kgvMk/SKpJuCx5Kdc63I5LEFVNcYc1f43YtrHeJpFvupmY0CbgL6Agt9GH7nWpdRfXM5oXdnZr+zseHCzrWAxjz1tRnYRORpsVhv1DvnQiKJK0/qx1vrtrN+u89Q6cIXz0uUX5f0ErAA6AV8zczGJjsw51zjXH5iXwBmL90QciTOxXfnMhC41cxGmdmPg1GJnXOtzIAeHTl1cA+eemcDwaAVzoWmoWmO04DLzGxpy4TjnGuOK0/qx0db9rFiw1HD6TnXohqa5rgGeFeSDxvsXBtwyegCstLTeOodbxpz4YqnWawAWClpgaQ5tUuyA3PONV7XjplM+FRv5ry7karqmrDDce1YPMO//DTpUTjnEuaKk/rx7IpNvFK8lfOH+4OdLhzxvOeyEFgHZAbfFwNvJzku51wTnT+8Nz06ZfF40fqGCzuXJPE8ivw1IiMM3x1s6gf8PYkxOeeaISsjjc+e3I/nV5WxdW952OG4diqePpebgLOITBuMma3GX6J0rlX7/PgBVFYbT73tHfsuHPEkl3IzOzRRhKQM6kw37JxrXU7o3YVTjuvOY4s/8XdeXCjiSS4LJf0H0EHShcDjwD+ac1JJPSQ9L2l18Nn9GOUmSfpAUrGkGVHbfyJpQzBR2FJJl0Ttuy0o/4Gki5oTp3Nt2efHD+CjLftY8vGOsENx7VA8yWUGsAVYDvwrkcm7ftjM884AFpjZUCLDysyoWyCYrvgu4GJgJHCtpJFRRX5rZuOCZW5QZySR6ZBHAZOAP9ZOe+xce3PpmAI6Z2fw2GLv2HctL56nxWrM7B4zuxqYDrxpzb/PngI8GHx/ELgiRplTgWIzWxM0yz0W1GvouI+ZWbmZrQWKg+M41+50ys7gshP78vSyjew6UBl2OK6diedpsZck5UrqASwF7pf0m2aeN9/MSgGCz1gPCPQDon/lKgm21bpZ0jJJ90U1qzVUx7l25brTBnKwsoa/LSkJOxTXzsTTLNbVzHYDVwH3m9kpwAUNVZL0gqQVMZaG7j4OHSLGtto7pj8BxwPjgFLg13HUqRvfdElFkoq2bNkSZ0jOtS2j+3XllOO689Ab66ip8Y5913LiSS4ZkgqAa4Cn4z2wmV1gZqNjLLOBsuCYBJ+bYxyiBBgQtd4f2Bgcu8zMqoOxz+7hcNPXMevEiG+mmRWaWWFeXl68l+Vcm3P9mYP4eNt+Xvow1n8z55IjnuTyM2Ae8JGZLZY0BFjdzPPOAaYF36cBs2OUWQwMlTRYUhaRjvo5cCgh1boSWBF13KmSsiUNBoYCbzUzVufatEmj+5Cfm80Dr38cdiiuHYmnQ/9xMxtrZl8P1teY2Webed7bgQslrQYuDNaR1FfS3OA8VcDNRBLbe8AsM1sZ1P+FpOWSlgHnA98O6qwEZgGrgOeAm8ysupmxOtemZaancd1px/Hyh1v4aMvesMNx7YQaevBL0jAifRz5ZjZa0ljgcjP7vy0RYEsoLCy0oqKisMNwLmm27CnnrNv/ybWnDuCnU0aHHY5LEZKWmFlhrH3xNIvdA9wGVAKY2TIiTVTOuTYir0s2k8cW8PiSEnbur2i4gnPNFE9y6WhmdfstqpIRjHMueaafO4T9FdU89Ib3vbjkiye5bJV0PMEjvZI+R+TxX+dcGzKiTy4TRvTm/tfWsr/Cfz90yRXvqMh3AyMkbQBuBW5MZlDOueT4+nnHs2N/JbN8SBiXZPE8LbbGzC4A8oARwHnA2UmOyzmXBIWDejB+UHfueWUtlT4NskuiYyaXYMiX2yTdGYyGvJ/IOynFRF6odM61QV8/73g27DzAk2/7kDAueeq7c3kYGE5kNOSvAfOBq4ErzCzeIVycc63M+cN7c2L/rtyxoJjyKn8NzCVHfclliJldb2Z3A9cChcBkM1vaIpE555JCEt+dOJwNOw9434tLmvqSy6ExuoO33Nea2Z7kh+ScS7ZPD+3FqYN68Id/FnOgwu9eXOLVl1xOlLQ7WPYAY2u/S9rdUgE65xIvcvcyjM17ynl40bqww3Ep6JjJxczSzSw3WLqYWUbU99yWDNI5l3inDenJOcPyuPOfxezY52/tu8SK5z0X51yK+uGln2JfRTW/feHDsENxKcaTi3Pt2LD8Llx32kAeefMTPizzLlWXOJ5cnGvnvn3BMDpnZ/CfT6+ioVHSnYuXJxfn2rnunbK49YKhvLJ6K3OXbwo7HJciPLk45/ji6ccxpl9XfjxnhQ/J7xIilOQiqYek5yWtDj67H6PcJEkfSCqWNCNq+18lLQ2WdZKWBtsHSToQte/PLXRJzrVpGelp3P7ZMezYX8nPn3kv7HBcCgjrzmUGsMDMhgILgvUjSEoH7gIuBkYC10oaCWBmnzezcWY2DngCeDKq6ke1+8zMR292Lk6j+nZl+jlDeHxJCa+u3hp2OK6NCyu5TAEeDL4/CFwRo8ypQHEwKnMF8FhQ7xBJIjKI5qPJC9W59uOWCUMZkteJf3v8XX/3xTVLWMkl38xKAYLP3jHK9AOiBz4qCbZF+zRQZmaro7YNlvSOpIWSPn2sACRNl1QkqWjLli1NuwrnUkxOZjp3TD2JbfvKmfHkMn96zDVZ0pKLpBckrYixxDuismJsq/sv/VqOvGspBQaa2UnAd4C/SIo5moCZzTSzQjMrzMvLizMk51Lf6H5d+d5FI5i3soy/vPVJ2OG4NiojWQcOJhiLSVKZpAIzK5VUAGyOUawEGBC13h/YGHWMDOAq4JSoc5YD5cH3JZI+AoYBRc25FufamxvOHszLq7fws3+sYky/rozt3y3skFwbE1az2BwiE48RfM6OUWYxMFTSYElZwNSgXq0LgPfN7NCMR5LyggcBkDQEGAqsSUL8zqW0tDTxu8+Po1fnbKY/tITNew6GHZJrY8JKLrcDF0paDVwYrCOpr6S5AGZWBdwMzAPeA2aZ2cqoY0zl6I78c4Blkt4F/gbcaGbbk3olzqWonp2zmfmlU9h1oJIbH17iE4u5RpF32EFhYaEVFXnLmXOxzF1eyjceeZtLxxZwx9STSE+L1R3q2iNJS8ysMNY+f0PfOVevS8YU8B+XjOCZZaX8aPYKf4LMxSVpHfrOudQx/Zzj2bm/kj++9BFdcjKYMWkEkdfMnIvNk4tzLi7/ftFwdh+s5O6FayivrOH/TB5JmjeRuWPw5OKci4sk/nPKaLLS07nvtbXsLa/i9qvGkJHurevuaJ5cnHNxk8SPJn+KLjkZ/H7Basp2H+TOa0+ma8fMsENzrYz/yuGcaxRJfPvCYdx+1RgWrdnGFX98jeLNe8MOy7Uynlycc00y9dSB/OVrp7P7QCWX3/kqsxav9yfJ3CGeXJxzTTZ+UA+e/tbZnNi/G997Yhk3/eVttu4tDzss1wp4cnHONUtB1w7871dP4/uTRvD8qjI+86uX+N9FH1NT43cx7ZknF+dcs6Wnia+fdzzP3vJpRvbN5Yd/X8EVf3yN14p90rH2ypOLcy5hTujdhUe/djq//fyJbN1TznX3vsm1MxexaM02749pZ3xsMXxsMeeS4WBlNX958xP++FIxW/dWMKpvLl85azCTTywgOyM97PBcAtQ3tpgnFzy5OJdMByqqeeqdDdz32lqKN++lW8dMJo8t4MqT+nHywO4+jEwb5smlAZ5cnEs+M+PV4q08XlTC/FWbOFhZQ//uHZgwojfnjejNGUN6kpPpdzRtiSeXBnhyca5l7S2vYt6KTcxdXsprH23lYGUN2RlpjB/Ug1OO684px3Vn3MBu5Ob4m/+tmSeXBnhycS48ByureXPtdl58fzNvrt3OB5t2U2MgweBenRjRpwvD83MZ3qczw/K70L97R7Iy/Fmk1qC+5BLK2GKSegB/BQYB64BrzGxHjHL3AZOBzWY2Op76km4DbgCqgW+Z2bwkXopzrplyMtM5d1ge5w7LA2DPwUreXb+Loo+3s2rjblZu3M2zKzZR+3uwBH1ycxjQvSP9e3Sgf/eO5Odm07NTNnldsujZKZteXbLplJXu/TkhCuXORdIvgO1mdrukGUB3M/t+jHLnAHuBh+okl5j1JY0kMvXxqUBf4AVgmJnVOz+r37k417rtr6hiddlePizbQ8mOA6zfsZ+SHQco2b6f0t0HifVjLCczja4dMumcnUGXnEy65GREluxMOudk0Dk7g5zMdHIy08jOSCc7I43szDRyMtLJDrblZKaRlZFGRppIT4t8pqUpWI/+TDu03p6mIWh1dy7AFOC84PuDwEvAUcnFzF6WNKgR9acAj5lZObBWUjGRRPNGwiJ3zrW4jlkZnDigGycO6HbUvsrqGrbvq2DLnnK27atg655ytu6NLLsPVLGnvJI9B6vYc7CKjTsPsLc88n1/Rb2/czaZBBlpQhJpAiEkEJFBPxWUkWJtj952uG5acAcWqXfkMQ+f9/DaEelNMb8eKn/esDx+OHlkIv8IgPCSS76ZlQKYWamk3gmq3w9YFFWuJNh2FEnTgekAAwcObOTpnXOtRWZ6Gvm5OeTn5jSqXnWNUVFVQ3lVNQcrI5/lVTUcrIx8lkdtq64xqmuMqhqjuqYm+LQ6242q6qj9ZmBQY0bwNfgM1s2O3kaw3Y7eXhN8OVQm6lqi79yO3G4xt0evFHTr0Kg/t3glLblIegHoE2PXD5J1Tuok7EDMdj8zmwnMhEizWBJjcs61QulpokNWOh2y/PHnZEhacjGzC461T1KZpILgrqMA2NzIwx+rfgkwIKpcf2BjI4/tnHOumcJ6nm8OMC34Pg2YnaD6c4CpkrIlDQaGAm81M1bnnHONFFZyuR24UNJq4MJgHUl9Jc2tLSTpUSKd8cMllUi6ob76ZrYSmAWsAp4DbmroSTHnnHOJ5y9R4o8iO+dcU9T3KLK/5uqccy7hPLk455xLOE8uzjnnEs6Ti3POuYTzDn1A0hbg42YcohfQniYLb2/XC37N7YVfc+McZ2Z5sXZ4ckkASUXHemIiFbW36wW/5vbCrzlxvFnMOedcwnlycc45l3CeXBJjZtgBtLD2dr3g19xe+DUniPe5OOecSzi/c3HOOZdwnlycc84lnCeXZpA0SdIHkoolzQg7nmSQNEDSi5Lek7RS0i3B9h6Snpe0OvjsHnasiSQpXdI7kp4O1lP6egEkdZP0N0nvB3/fZ6T6dUv6dvDveoWkRyXlpNo1S7pP0mZJK6K2HfMaJd0W/Ez7QNJFTT2vJ5cmkpQO3AVcDIwErpWU+Imow1cFfNfMPgWcDtwUXOcMYIGZDQUWBOup5Bbgvaj1VL9egN8Dz5nZCOBEItefstctqR/wLaDQzEYD6cBUUu+aHwAm1dkW8xqD/9tTgVFBnT8GP+sazZNL050KFJvZGjOrAB4DpoQcU8KZWamZvR1830PkB04/Itf6YFDsQeCKUAJMAkn9gUuBe6M2p+z1AkjKBc4B/gfAzCrMbCcpft1EZuPtICkD6Ehk5tqUumYzexnYXmfzsa5xCvCYmZWb2VqgmMjPukbz5NJ0/YD1UeslwbaUJWkQcBLwJpBvZqUQSUBA7xBDS7TfAd8DaqK2pfL1AgwBtgD3B82B90rqRApft5ltAH4FfAKUArvMbD4pfM1RjnWNCfu55sml6RRjW8o+1y2pM/AEcKuZ7Q47nmSRNBnYbGZLwo6lhWUAJwN/MrOTgH20/eagegX9DFOAwUBfoJOkL4QbVegS9nPNk0vTlQADotb7E7mlTjmSMokklkfM7Mlgc5mkgmB/AbA5rPgS7CzgcknriDR1fkbS/5K611urBCgxszeD9b8RSTapfN0XAGvNbIuZVQJPAmeS2tdc61jXmLCfa55cmm4xMFTSYElZRDrB5oQcU8JJEpF2+PfM7DdRu+YA04Lv04DZLR1bMpjZbWbW38wGEfk7/aeZfYEUvd5aZrYJWC9peLBpArCK1L7uT4DTJXUM/p1PINKnmMrXXOtY1zgHmCopW9JgYCjwVlNO4G/oN4OkS4i0z6cD95nZz8ONKPEknQ28AizncB/EfxDpd5kFDCTyn/RqM6vbadimSToP+DczmyypJ6l/veOIPMSQBawBvkzkF9CUvW5JPwU+T+SpyHeArwKdSaFrlvQocB6RofXLgB8Df+cY1yjpB8BXiPyZ3GpmzzbpvJ5cnHPOJZo3iznnnEs4Ty7OOecSzpOLc865hPPk4pxzLuE8uTjnnEs4Ty7ONYGkHwSj6S6TtFTSacH2WyV1DDu+WpI6SFoYjPKcJumOYATg5ZIWB+8yIOmFtj76r2tdPLk410iSzgAmAyeb2Vgib3rXjsd0K5EBEBtzvCaNOhunrwBPmlk1kfc5+gJjzWwMcCWwMyj3MPCNJMbh2hlPLs41XgGw1czKAcxsq5ltlPQtIj+8X5T0IoCka4O7hBWS/rv2AJL2SvqZpDeBMyT9n+BOYoWkmcEb40gaH9wdvSHpl7VzcgR3Ir8M6iyT9K/HiPU6Dr99XQCUmllNEHeJme0I9s0Brk3kH5Jr3zy5ONd484EBkj6U9EdJ5wKY2R1ExmE638zOl9QX+G/gM8A4YLykK4JjdAJWmNlpZvYqcKeZjQ/mFelA5M4I4H7gRjM7A6iOiuEGIqP4jgfGA1+rbeKqFQxLNMTM1gWbZgGXBc14v5Z0Um3ZIMlkByMRONdsnlycayQz2wucAkwnMkz9XyVdH6PoeOClYGDEKuARInOmQCRRPBFV9nxJb0paTiQZjZLUDehiZq8HZf4SVX4i8CVJS4kMxdOTyDhQ0XpxuNkLMysBhgO3ERnKZ4GkCVHlNxO583Ku2TLCDsC5tijow3gJeClICNOIzPgXLdbw5bUOBsdAUg7wRyIzIq6X9BMgp4H6Ar5pZvPqKXMgOE503OXAs8CzksqITBK1INidE9Rxrtn8zsW5RpI0XFL0XcI44OPg+x6gS/D9TeBcSb2CTvtrgYUxDlmbALYG8+Z8Dg41Ve2RdHqwf2pUnXnA14PpEJA0LJjc65CgfnqQvJB0ctBUh6Q0YGxt3EEfTx9gXbx/Ds7Vx+9cnGu8zsAfgmarKiJTwU4P9s0kcldQGvS73Aa8SOROY66ZHTV8u5ntlHQPkZGn1xGZzqHWDcA9kvYRuVPaFWy/FxgEvB0khi3Eno53PnA28AKR2QbvkZQd7HsLuDP4fgqwKGi+c67ZfFRk51oxSZ2DPh4kzQAKzOyWRtQ/CfiOmX2xgXK/B+aY2YL6yjkXL79zca51uzS4+8kg0oR1fWMqm9k7kl6UlF7bx3MMKzyxuETyOxfnnHMJ5x36zjnnEs6Ti3POuYTz5OKccy7hPLk455xLOE8uzjnnEu7/A7k5JCYrIpiRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dtype = torch.float32\n",
    "v = torch.tensor(np.arange(0.0, 100.0, step=0.1), dtype=dtype)\n",
    "\n",
    "def nlres(t, x, k):\n",
    "    return k(x)*x\n",
    "\n",
    "def kx(x, b=50, s=-0.1):\n",
    "    return s * torch.tanh((x-b)/10)\n",
    "\n",
    "plt.plot(v, kx(v))\n",
    "plt.axhline(0, color='grey', linestyle='--')\n",
    "plt.xlabel('Storage (S)')\n",
    "plt.ylabel('Reservoir conductivity (K)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f641618c-0334-4693-8f4b-dcbb31fc82a5",
   "metadata": {},
   "source": [
    "## Solving the nonlinear reservoir\n",
    "\n",
    "Given all of our machinery developed so far, we can now solve the nonlinear version for all sorts of initial conditions. Note that I am using autodiff's `jacobian` to determine the derivative of the conductivity function $K(S)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b793ad6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:5: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  \"\"\"\n",
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:13: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  del sys.path[0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'storage')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2k0lEQVR4nO3deXgUVdbA4d9JQkgIS9gHiIosIoKAEhgVVBhgBBXFURBExXFBFB1wGUXUcRkXXMbtc9w33EBEGUBFQAQVZAvIziAwoAYQIvuWtc/3x62EgAGS0JXuTp/3eeqp6uqlTttyqnLr3ntEVTHGGBM9YkIdgDHGmLJlid8YY6KMJX5jjIkylviNMSbKWOI3xpgoExfqAIqjVq1a2rBhw1CHYYwxEWXBggW/qWrtQ/dHROJv2LAhaWlpoQ7DGGMiioj8VNR+a+oxxpgoY4nfGGOijCV+Y4yJMhHRxm+MMflycnJIT08nMzMz1KGEjYSEBFJSUqhQoUKxXm+J3xgTUdLT06lSpQoNGzZEREIdTsipKlu3biU9PZ0TTzyxWO/xtalHRIaIyDIRWS4iQ719NURkqois9tbV/YzBGFO+ZGZmUrNmTUv6HhGhZs2aJfoLyLfELyItgRuA9kBr4EIRaQoMA6apalNgmvfYGGOKzZL+wUr638PPK/7mwBxV3aequcA3wCXAxcBI7zUjgV6+RTB9OjzxhG8fb4wxkcjPxL8MOEdEaopIJeB84DigrqpuAvDWdYp6s4gMFJE0EUnLyMgoXQSffw7Dh8OqVaV7vzHGlEO+JX5VXQk8AUwFvgQWA7kleP9rqpqqqqm1a/9uxHHx3HUXJCTAI4+U7v3GGFMO+XpzV1XfVNXTVfUcYBuwGtgsIvUAvPUW3wKoUwcGD4YPP7SrfmNMUG3atIm+ffuSmprKSSedROfOnUv9Wc8//zwtW7akRYsWPPfccwX7v/zyS5o1a0aTJk0YMWJEEKJ2/O7VU8dbHw/8BRgFTAAGeC8ZAIz3Mwb+/ndITISHH/b1MMaY6HLVVVdxySWXkJaWxo8//sgLL7xQqs9ZtmwZr7/+OvPmzWPx4sV89tlnrF69mry8PAYPHsykSZNYsWIFo0aNYsWKFUGJ3e+Ru5+IyApgIjBYVbcDI4BuIrIa6OY99k/t2nDLLTBqFKxc6euhjDHRIS8vjxkzZnDuuecW7Dv11FNL9VkrV67kjDPOoFKlSsTFxXHuuecybtw45s2bR5MmTWjUqBHx8fH07duX8eODc53s6wAuVT27iH1bgS5+Hvd37rwTXnzRXfWPGlWmhzbG+GjoUFi0KLif2aYNFGpuKUpsbCxdu3aldevW9OrVi6uvvpoOHToc9vVnn302u3fv/t3+p59+mpYtW3LvvfeydetWEhMT+eKLL0hNTWXDhg0cd9xxBa9NSUlh7ty5pf1WB4mOkbu1asGtt7qunffdBy1ahDoiY0yEmzRpErNmzWLChAl0796d9957j27dunHzzTcTHx9Pp06d6N+/PwDffffdET/r7rvvplu3blSuXJnWrVsTFxeHqv7udcEavxAdiR8Ovur/6KNQR2OMCYajXJn7SUTo2LEjHTt2ZPv27SxZsoTdu3dz2WWX0bNnTy6//PKCxH+kK/6uXbty3XXXcd111wEwfPhwUlJSSElJ4Zdffil4bXp6OvXr1w9K7NGT+GvWhL/9DR5/HO6/H1q2DHVExpgINXnyZDp37kx8fDxbtmxh5syZvPXWW8yYMaOgrT82Nrbg9Ue74t+yZQt16tTh559/5tNPP2X27NlUqVKF1atXs27dOho0aMDo0aP58MMPgxJ/dE3LfMcdULkyPPRQqCMxxkSwsWPH0rx5c1q3bs2FF17IP//5T84880xSUlJIT08HIBAIFPvzLr30Uk455RR69uzJv//9b6pXr05cXBwvvvgi5513Hs2bN6dPnz60CFIztRTVjhRuUlNTNWilF++/3w3oWrIESnkX3hgTOitXrqR58+ahDqNIe/fu5ZZbbiEhIYGOHTsWNPWUhaL+u4jIAlVNPfS10dPUk+/22+GFF9xV/9ixoY7GGFOOJCUl8fbbb4c6jKOKrqYegOrVXRewTz6BxYtDHY0xxpS56Ev8ALfdBtWqWVu/MSYqRWfiT052yX/cuOAP/jDGmDAXnYkfYMgQd9X/4IOhjsQYY8pU9Cb+5GR3o3f8eFi4MNTRGGNMmYnexA/uqj852a76jTFRJboTf7VqblDXxIkQrHECxhgT5qI78YObxqFGDevhY4yJGpb4q1Z1E7h99hnMnBnqaIwxxneW+MFd9Tdo4Jp9SjC/hjEmegWr9OKqVato06ZNwVK1atWC8ot+lV5EVX1bgNuA5cAyXNnFBKAGrgD7am9d/Wif07ZtW/XdO++oguqHH/p/LGNMqa1YsSLUIaiqapcuXXT06NEFj5csWXLMn5mbm6t169bV9evXa25urjZq1EjXrl2rWVlZ2qpVK12+fPlh31vUfxcgTYvIqb5d8YtIA+BvQKqqtgRigb7AMGCaqjYFpnmPQ++qq+C002DYMNi/P9TRGGPCWDBLLxY2bdo0GjduzAknnBC5pRe9z08UkRygErARuAfo5D0/EpgB3O1zHEcXEwP/+hf86U/w/PPuBGCMCW8LhsL2RcH9zOptoO1zR3xJMEsvdu3ateDx6NGj6devH0Bkll5U1Q0i8jTwM7AfmKKqU0Skrqpu8l6zSUTqFPV+ERkIDAQ4/vjj/QrzYJ07Q8+e8NhjcO21UKfI0IwxJqilFwGys7OZMGECjz/+OEBkll4UkerAxcCJwA7gYxG5srjvV9XXgNfAzcfvR4xFeuopV53rwQfhpZfK7LDGmFI4ypW5n4JZehHcieT000+nbt26ABFberErsE5VMwBE5FPgLGCziNTzrvbrAVt8jKHkmjWDQYPg5ZddgfYwLfhgjAmdYJdeBBg1alRBMw9Au3btIrL04s/AGSJSSdzfJ12AlcAEYID3mgFAcO5WBNMDD7gSjX//e6gjMcaEoWCXXty3bx9Tp07lL3/5S8G+iC29KCIPAZcDucAPwPVAZWAMcDzu5NBbVbcd6XOCWnqxuJ56Cu66C6ZOhUI3X4wxoWWlF4tWktKL0Vdzt7gyM10zT9WqbvbOQn+2GWNCJ5wTfyiVJPHbyN3DSUiAJ55wRdlHjgx1NMYYEzSW+I+kd28480y47z7YsyfU0RhjTFBY4j8SETeoa9MmePrpUEdjjDFBYYn/aM48E/r0cTd7N2wIdTTGGHPMLPEXx4gRkJvrmnyMMSbCWeIvjhNPdGUaR46ERYtCHY0xxhwTS/zFNXy4q9R1xx0QAV1gjTHmcCzxF1d+Ufavv4ZPPgl1NMYYU2qW+Eti0CA4/XQ3h8+OHaGOxhhjSsUSf0nExcHrr0NGBtwd+hICxpjQCVbpRYCGDRty6qmn0qZNG1JTDwy09av0ot+FWMqf00+H225z/fr794dzzgl1RMaYELjqqqu44YYbuPzyywFYunTpMX3e9OnTqVWrVsHjvLw8Bg8ezNSpU0lJSaFdu3ZcdNFFnHLKKcd0HLDEXzoPPeTa+QcOdL18EhJCHZEx0WnzUMhcFNzPTGgDdZ874kvySy++//77BfuCUXqxsMKlF4GC0ovBSPzW1FMalSrBq6/CqlWuWpcxJqoULr144403MmvWrCO+/uyzz6ZNmza/W7766ivAFXX585//TNu2bXnttdeAoksvbgjSIFK74i+tbt1cgfYRI9zI3pYtQx2RMdHnKFfmfgpm6cVZs2ZRv359tmzZQrdu3Tj55JMjs/RiVHjmGZg0CW64AWbOtKmbjYkiwSy9mF9SsU6dOlxyySXMmzePDh06+FZ60Zp6jkWtWvDsszBnjivVaIyJCpMnTyY7OxugoPRit27dSE9PL2ieObT04qJFi363dO3alb179xacFPbu3cuUKVNo2bLlQaUXs7OzGT16NBdddFFQ4vct8YtIMxFZVGjZJSJDRaSGiEwVkdXeurpfMZSJ/v3hvPPgnnug0NnZGFN+BbP04ubNm+nYsSOtW7emffv2XHDBBXTv3j1ySy8WHEQkFtgA/BEYDGxT1REiMgyorqpH7BRf6gpcgVzYsw6qNi1F1CWwbp1r4+/SBcaPd9M5G2N8Ec4VuCKl9GJZtfF3Adaq6k8icjHQyds/EpgB+DMaat4NsGky9FgCCbWO/vrSOvFEePhhuPNOGDvWFXAxxkSdpKQk3n777VCHcVRl1cbfFxjlbddV1U0A3rpOUW8QkYEikiYiaRkZGaU7arMhkLUV5l3v/8RqQ4ZA27ZuOoft2/09ljHGHAPfE7+IxAMXAR+X5H2q+pqqpqpqau3atUt38OptoM0ISB8Pa14t3WcUV/50Dr/9Bnfd5e+xjDHmGJTFFX8PYKGqbvYebxaRegDeeouvR282BOqdBwtvg50rfD0Up50Gt98Ob7wB33zj77GMMaaUyiLx9+NAMw/ABGCAtz0AGO/r0SUGzngH4qrArCsgL9PXw/Hgg9CokZvOIdPnYxljTCn4mvhFpBLQDfi00O4RQDcRWe09F7wp5w4n8Q9wxtuwYzEsusffY1WqBK+8Aj/+6G72GmNMmPE18avqPlWtqao7C+3bqqpdVLWpt97mZwwFGlwAJ90Cq56DjV/6e6xu3VyTz7//DaNH+3ssY4wpoegaudvmSajWEuYMgP2bj/76YzFiBHToANdfDytX+nssY4wpgehK/HGJ0GEUZO+Eudf628WzQgX46CPX9HPppbBnj3/HMsaYEoiuxA+Q3BJOexo2fgE/vujvsRo0gFGj3PTNgwZZkXZjypFgVeDKzMykffv2tG7dmhYtWvDAAw8UPOdXBS5UNeyXtm3balAFAqrTL1AdVVF1+5LgfnZRHnlEFVRfesn/YxlTzq1YsSLUIaiqapcuXXT06NEFj5csKV0uCQQCunv3blVVzc7O1vbt2+vs2bM1NzdXGzVqpGvXrtWsrCxt1aqVLl++/LCfU9R/FyBNi8ip0XfFD24unTPegvhkmNUPcvf7e7x77oEePWDoUJg/399jGWN8l1+B69xzzy3YV9oKXCJC5cqVAcjJySEnJwcROagCV3x8fEEFrmCI3vn4E+rAGSNhRnf44e/Qzsdmn5gYeO89V6+3d29YsABq1vTveMZEiaFfDmXRr4uC+plt/tCG57o/d8TXFK7A1atXL66++mo6dOhw2NcfbT7+vLw82rZty5o1axg8eDB//OMfGTt27O8qcM2dO7fU36uw6E38APXPg2a3wapn3ejelJ7+HatmTTeBW4cOcPXVMHGiOyEYYyJSMCtwxcbGsmjRInbs2MEll1zCsmXLrAKXr9o8Dpu/hrl/hWpzoEoT/47Vrh089xwMHuy6ew4f7t+xjIkCR7sy91MwK3DlS05OplOnTnz55ZdWgctXsRWh4xi3Pf082L/J3+PddBP06wf33w/Tpvl7LGOML4JZgSsjI4MdO3YAsH//fr766itOPvnkyKzAFVGqngTnfgGZm2F6D9fP3y8i8Npr0KwZXHEFbNjg37GMMb4IZgWuTZs20blzZ1q1akW7du3o1q0bF154YeRX4DpWpa7AVVKbpsA3F0KtM6HzZIhN8O9YK1e6pp/TToOvv3YDvowxR2UVuIoWjhW4IkO9P7uePt/3d908O34MMT79J2re3M3ff8UVcOONbipnu9lrTESLlApclvgP1bAfZP0GC/4G8wdB+9f9q6Hbr58b1fvQQ+6xJX9jTBmwxF+UZrdCVgYs+ydUrANtHvPvWA8+6NYPPeSmdHjjDSh0U8gYY4LNEv/hnPoQZG6BFY+7wV4nD/XvWA8+6P6qePBBl/zffNOSvzHGN74mfhFJBt4AWgIKXAusAj4CGgLrgT6qGn7VyUUg9d+u2WfhbVCxNpzo442aBx5wx3zgAZf833rLkr8xxhd+Nyg/D3ypqicDrYGVwDBgmqo2BaZ5j8NTTCyc9T7U7QxzroGNk/w93j/+AQ8/DO++C3/9K+Tl+Xs8Y0xU8i3xi0hV4BzgTQBVzVbVHcDFwEjvZSOBXn7FEBSxCXDOfyD5VPjuUsiY7e/x7r8fHnnEze1zzTWW/I0xQefnFX8jIAN4W0R+EJE3RCQJqKuqmwC8dZ2i3iwiA0UkTUTSMjIyfAyzGCpUhU6TILEBfHMBbJnp7/HuvRcefRTefx8GDLDkb4wJKj8TfxxwOvCyqp4G7KUEzTqq+pqqpqpqau3atf2KsfgS68KfpkB8TZjWGVa96G9hleHD4bHH4IMP3KRuubn+HcsYE1X8TPzpQLqq5s8jOhZ3ItgsIvUAvPUWH2MIrsonQvf5UK87LLgVZg+A3H3+He+ee9xkbh9+aMnfGBM0viV+Vf0V+EVEmnm7ugArgAnAAG/fACA4lQXKSnwynDvedfdc/z5M7QB71vl3vLvvhieecCUcL7oINvtcJN4YUyzBKr0IcO2111KnTh1atmx50P6ILL0ItAHSgCXAf4DqQE1cb57V3rrG0T4n6KUXgyX9M9Ux1VQ/rqG6cbK/x3r5ZdWKFVVr11adMMHfYxkTxspb6UVV1W+++UYXLFigLVq0KNgXsaUXVXWRunb6VqraS1W3q+pWVe2iqk299TY/Y/BVgwugexok1ofp3WH54/61+w8a5Cp3NWjgrvwHDYK9e/05ljHmiIJZehHgnHPOoUaNGgfts9KL4axKEzhvDsy9HhYPh63z4cx3XE+gYGvRAubMcf39n3rKzer5wQdulk9jotDQobBoUXA/s00bVy/pSIJderEoGzZssNKLYS0uCc76EGq2d/V7J/8Rzh4H1U4O/rEqVnRt/j16uBu+Z53lRvsOGwZx9nMaU1aCWXqxKGqlFyOACJx8G1Q/DWb2gcntoNUj0HSQq/IVbJ06wZIlrozj/ffDpElu0FejRsE/ljFh6mhX5n7yo/RiYSkpKaEvvSgiJ4hIV287UUSqBCWC8qZuJ+ixEGqdAQuHwmcnw7oPQItXjadEkpNdU8+HH8Ly5dC6Nbzzjr/jC4wxQS29eDghL70oIjfg+uG/6u1KwfXSMUWplAJ/mgqdp0B8dZh9JUw6HTZO9icp9+vnrv5TU90cP+efD7N9nlrCmCgWzNKLAP369ePMM89k1apVpKSk8Oabb4a+9KKILALaA3PVjcJFRJaqaulvY5dAmZVe9IMG4KePYPG9sHcd1P0TtBkBNX24IRsIwPPPu7l+tm1zzUH33APduvlXTMaYMmalF4tWktKLxW3qyVLV7EIfFoebZtkcjcS4ql4X/hfavgA7lsDk9u4+wK7VwT1WTAzcdhv89BM88wysXg3nned6/Ywda3P+GOOz/NKLL7/8cpkm/ZIqbuL/RkSGA4ki0g34GJjoX1jlUGy8q+x10Vpo+Q/Y+AV8fgrMvxn2/C+4x6pc2Z0A1q51Fb127YLevV130Lffhuzso3+GMabcKm7iH4abaXMpcCPwBXCfX0GVaxWqQquHoOdaaHIjrHkdJjSGKR3gx5cg87fgHatiRbjuOli5Ej76CBIT4dproXFj1yRkA8CMiUrFSvyqGlDV11W1t6pe5m1bU8+xSKwL7V6Ei/4HrR+HnJ2QNhjG1YMZPd19gWBNABcbC336wMKFrttno0Zu5Eu9eu7G8Ecfub8KjDFRoVj9+EVkKb9v09+Jm4fnEVXdGuzAokbScdBiGJxyt2v/X/8BrP8QNn4GcZXhuEuhYX93UzjmGEsxikD37m6ZNcs1+0yYAKNHQ3w8dOkCvXq5KSH+8IegfD1j/KCqQRvMVB6U9Dq8uL16ngTygA+9XX299S6go6r2LNFRSyiie/WURiAPtnzjTgK/jIWcXZBYD+pfCLU7Qp2zIalhcHrq5OW5rp//+Q+MGwf/+5/73DPOgEsucSeCpk2P/TjGBMm6deuoUqUKNWvWtOSPS/pbt25l9+7dnHjiiQc9d7hePcVN/LNUtUNR+8qiW2fUJf7CcvfDxs/dSWDzdNckBG5iuNodofbZUKcjVDv12P8iUHUDwfJPAgsXuv3NmrkTQbt2bqxA69aQkHBsxzKmlHJyckhPTyczMzPUoYSNhIQEUlJSqFChwkH7jzXxLwYGqldURUTaA6+ramsR+SG/b79fojrxFxbIg53LIWOmt3wH+9xgESpUhVpnuZNBzXZQtTlUauC6k5bWzz/D+PEweTLMnw9bvJo5FSrAqaceOBG0a+d6DNlcQcaElWNN/O2At4DKgOCaeK4HlgMXqOqY4IZ7MEv8R7D3J1cDOP9ksHPZgediK0HVZlD1ZG/xtqs0hbhKJTuOKvzyC6SluZPA/Plue2f+XyCJ7i+Bk05yvYYaN3Y3kRs3htq1bQCZMSFwTIm/0IdU896zI4ixHZUl/hLI2uZuEu9aBbv+e2DZ+xMH7s8LJB0PlZu4vwoSG7imo0r13TqxvrunEFPhSEdyI4XXrj1wIvjhB1izBjZsOPh1lSsffCJo3Nj1KKpb98BSqYQnImPMUR1z4heRC4AWQEHjrqo+fJT3rAd2424M56pqqojUAD4CGgLrgT6quv1In2OJPwhy98Pu1d6JwDsp7FkL+zfC/k2gRdTzTajjTgIJf4D4GlCxhpt7qGCpcch2MsQmQmYmrF/vTgr5y//+59br1kFW1u+PVbnywSeCP/zBratXd5PRVat28JKcDFWrWvOSMUdwuMRf3O6crwCVgM7AG8BlwLxiHruzqhYelTQMmKaqI0RkmPf47mJ+limtuESo3soth9IAZP3mTgL7NngnA2/ZtxEyN8HuHyF7O2Tv4IizdUgMxCZBhcpufVJlOCXJdU2NOwVi20EWkJkH+3JgTzbsyYJd+2DnPti+Dbb9BCt3wfe7IRfvssFb5x3yOD4RkqpCpSqQUMlbkiAxCRIrQaUk99dEkrdOSHAD2+Lj3frQ7fzH8fHuXkZc3OHX+UtsrFusOctEiOJeLp2lqq1EZImqPiQi/wI+LeUxLwY6edsjgRn4lfg3D4XMRb58dLlWwVsKiohV9havGpDmQiAXNMdb50Igx601r9CyE3T7IfvyoGIeVAkEaarq/d5y+CL0GoDsgDvXZAcgJwA56q0Lb+dCTo57Ta5CnkJuwK3z9MC+Q7cDeGtvO1Bof6DQ8+o9Vtw+9Z773X7vOQUUOeTxgecp9JhCrym8XznwAi10Xir8fjh4+6B9R3gdRbzuoH2AuyV4ZAd9dkAI5FUgkFuRQF48gdx4b13RbQfi0Lw4NBBHIK+Ct3aP87cDgVjvcSyqMW7tLeQ/1tiC51Fxr9MY97zGoIED2wc/L4Cg3jr/uaL3H9h231MOeR2FfhQp9N/w4OeeuGEal1/37lH/O5ZEcRN/fr+pfSJSH9gKnHiE1+dTYIqIKPCqqr4G1FXVTQCquklE6hT1RhEZCAwEOP7444sZpikTEgexcRRq9TsGCqqo5rEnJ5ftWdnsyM5he5ZbdufksTc3fwmwNzePPbkB9ubksTcvwN7cAHtyA2TmqbcEyAq47QPrIIQZRDG4Pw5icP/EY+TgtXjPeynikMcHP0+hfRS1Xw5OvUVtF/5DRYrxukBuPJpZlUBWVQKZ3pJVBfUe52UV2pdTCc1JJJCTiOZU8taFt711bjyaFw+Bo9xXKkvinYYlAKJIwWMtWIv3XOH9csjjgjW45w7aT8FzSP5pnoOe27J1ZtC/WnET/0QRSQaeAhbiIn29GO/roKobveQ+VUT+W9zAvJPEa+Da+Iv7voPUfa5UbzPHTlXZnrmdTbs3sWnPpoPWv+79lW37t7F9/3Z2ZO5ge6Zb5waKuM9wiIqxFUmKTyKpQjUqx1d225WSqFWhEhXjKpIQl0DFWLcuvJ3/XHxsPBViKlAhtsJR17ESS1xMHLEx3lpif7edv46RGGIkhlgptF1of4zEIEhYDjjKznb34zduhIwM+O03tz7c9r5izCQSE+Nu21Sq5JbERKhU1a0TEw/sy18ObWXLXw5teYuPP7iV7XAtb4Vb4GJiDmwf+jgmpuhFBKSgK/Qxjo85ZsVJtSVz1MQv7ttP83ryfCIinwEJqrrzaO9V1Y3eeouIjMPN6b9ZROp5V/v1gC3H9A1MSAQ0wIZdG1i9bTWrt65mzbY1rN2+lo27N/Lrnl/5dc+vZOX9/iZupQqVqFe5HjUr1aRGYg0aVW9E9YTqVE+sXrBOTkgu2K4SX8VL9EkkxScRF2M3c0tq5043JOOnn4peb9xYdFNNUpLriVurFtSp44Zq1KoFNWu6++qHLlWqHNiuVMlueYSzo/4rUtWA16Z/pvc4C3d77ohEJAmIUdXd3vafgYeBCcAAYIS3Hl/68I3ftu/fzuLNi1m9dbVL8l6iX7t9LZm5B0ZOVoytyInVT6RBlQacfcLZ1Ktczy1VDl5XqWgVO/2yezcsW+aWpUvdsmyZu1IvLD4ejjsOTjjB1eg54QQ4/nho0MAl+vxkn5gYmu9h/Ffcy6cpInIp8GkJZuWsC4zz/rSNAz5U1S9FZD4wRkSuA34Gepc0aOOP/Tn7+eHXH5i/YT7zNs5j3oZ5rNm2puD5+Nh4GldvTJMaTTiv8Xk0rdmUpjWa0qRGE1KqphB7rFNGmGJRhVWr3LCJ/OS+dKnrQZsvKQlatnRTLZ100oHkfsIJrpdszDEM6DaRr7gjd3cDSbgOdPtx93tUVase8Y1BYv34gy8vkMeKjBXM2+AS/PyN81m6ZWlBO3uDKg1o16Ad7eu35/R6p9OsVjOOq3qcJfcQUHXF1KZPd8uMGbDZ68QUF+emUmrZ0s2ikb+ccIIld3OM/fhV1f4+Lwd2Z+1mytopTPxxIl+s/oKMfRkAJCck065+O+466y7aN2hPuwbtqF+lfoijjV6qbrxbfpKfPt21wwPUrw9du7pyyu3bu6RfsWIoozWRqNh3ykTkIuAc7+EMVf3Mn5BMMK3fsZ6JqyYy8ceJzFg/g5xADtUTqtOjaQ+6N+7OGSln0KRGk7DsbRJNcnPhq69gzBi3/uUXt79uXZfkO3d2S9OmdtPUHLvijtwdAbQDPvB2DRGRjqo6zLfITKnkBfKYu2EuE1dN5LPVn7Fsi5u0rVnNZgz54xB6NuvJWcedZb1jwoCqm+fugw9g1Cg3+Wm1au6G67BhLuE3b26J3gRfcf/1nw+0UXVDLUVkJPADbroFEwY279nMmz+8yStpr/DLrl+IlVjOOeEcnvnzM1x40oU0rWnFVMLF2rUu2X/wAfz4o+tl07Mn9O8P559vTTfGfyW57EsGtnnb1YIfiikpVeX7X77npbSX+Hj5x+QEcuhyYhdGdB3B+U3PJzkhOdQhGk9GhmvGef99mDPH7evUCe66Cy691M05Z0xZKW7ifxz4QUSm43r0nAMM9y0qc0R7s/fywdIPeGn+SyzevJiqFatyU+pN3NTuJk6udXKowzOFrFkDjz8O777r2vFPPRWeeMLVuD/uuFBHZ6JVcXv1jBKRGbh2fgHuVtVf/QzM/N6q31bx0vyXeGfxO+zK2kWruq149cJXueLUK6gcXznU4ZlCli+Hxx47UMd+0CC44QZoVcTkqMaUteLe3J2mql1wo24P3Wd8tvjXxdwz7R4mrZlEhZgKXHbKZQxuN5izjjvLeuOEmYUL4dFH4dNP3SCqO+6A22935QWMCRdHTPwikoCbh7+WiFTnwER9VQHr6O2z9F3p3Pf1fby7+F2SE5J5uNPDDGw7kLqV64Y6NHOI2bNdwv/8c9cz5/77YcgQN6+NMeHmaFf8NwJDcUl+Ad6IXVxVrRd9jSyK7czcyROznuDZOc8S0AB3nHkHw88eTvXE6qEOzRSiCt98A488AtOmuST/yCMweLDdrDXh7YiJX1WfB54XkX8Az6nqLhG5HzgdmF0WAUaT7LxsXk17lYe/fZjf9v1G/1P788ifHqFhcsNQh2YO8dNPcOutMHGiG2T19NNw441uKmJjwl1xe/VcpqoPi0hHoBvwL+Bl4I++RRZFVJVPVn7CPdPuYc22NXRu2Jmnuj1F2/ptQx2aOURODjz7LDz0kBtY9eSTcMstNpOliSzFTfx53voC4BVVHS8iD/oTUnT5/pfvuXPKncxOn02L2i34/IrP6dGkh920DUMzZ7reOcuXu1kvn3/ezXhpTKQp7vx9G0TkVaAP8IWIVCzBe00R9mbv5dYvbqXDWx1Yv2M9b/R8g0WDFnF+0/Mt6YeZrVvh+uvh7LPdnPfjx8O4cZb0TeQq7hV/H6A78LSq7vAqZ/3dv7DKt5k/z+Sa/1zD2u1r+Vv7v/FYl8dIik8KdVjmEKowciTceaerYnXXXfCPf7humsZEsuIO4NoHfFro8SZgk19BlVf7c/Zz79f38tyc52iY3JAZA2ZwbsNzQx2WKcKKFXDTTfDtt9ChA7z8sht1a0x54HtzjYjEisgPXq1eRKSGiEwVkdXeOir6KM5Jn0ObV9vw7JxnGZQ6iCU3LbGkH4Zyc+Hhh6F1a1fZ6o03XPK3pG/Kk7Jopx8CrCz0eBiueHtTYBrlfIbPzNxMhn01jA5vdSAzN5OpV03lpQtesikWwtDPP7s57x94AHr3hv/+F667zipZmfLH1/+lRSQF1xPojUK7LwZGetsjgV5+xhBKaRvTaPtaW56Y9QTXtrmWpTctpWujrqEOyxRh7Fh3lb9oEbz3Hnz4oSs6bkx55Pe1zHPAXUCg0L663j2C/HsFdYp6o4gMFJE0EUnLyMjwOczgysnL4f6v7+eMN85gZ+ZOJvWfxOsXvU7VimVSotiUwN69bvK03r1dUfJFi+DKK0MdlTH+8i3xi8iFwBZVXVCa96vqa6qaqqqptSPo0uuXnb/QaWQnHvnuEfq36s+ym5fRvUn3UIdlirBoEaSmwptvuopXM2dC48ahjsoY//lZf68DcJGInA8kAFVF5H1gs4jUU9VNXrfQLT7GUKYmrZ7EVeOuIisvi1GXjqJvy76hDskUQdUNvrr7bje/ztSp0MXmmTVRxLcrflW9R1VTVLUh0Bf4WlWvxE3tPMB72QBgvF8xlJXcQC7Dpw3n/A/Pp0HVBqTdkGZJP0xt2QIXXAC33QbnnQdLlljSN9EnFBW3RwBjROQ64GegdwhiCJqNuzfS75N+fPvTt1x/2vW80OMFEivYxC3haOpUuOoq2LEDXnwRbr7ZCpmb6FQmiV9VZwAzvO2tQLm4xpq6dir9P+3P3py9vHfJe1zZyu4KhqNAwM2V/8AD0Ly5OwFYv3wTzayHcinkBfJ4YPoDnPf+edRJqkPaDWmW9MPUtm1w4YVuqoX+/WHePEv6xoSiqSei/brnV/p/2p+v133NNW2u4cUeL9o8O2FqwQK49FLYuBFeesnNrGlNO8ZY4i+Rb9Z/Q99P+rIzcydvX/w217S5JtQhmSKouqkWbrnF1bqdORPatw91VMaED2vqKQZV5envn6bLu12oVrEa826YZ0k/TO3bB9deCwMHQqdO7qrfkr4xB7Mr/qPYlbWLv47/K5+u/JRLm1/KWxe/ZSNww9SaNXDZZa6L5gMPuILnsbGhjsqY8GOJ/wiWb1nOX8b8hbXb1vJ0t6e5/czbrUhKmBo/HgYMcIn+88+hR49QR2RM+LKmnsMYtXQU7d9oz66sXXw94GvuOOsOS/phKDcX7rnHlUJs0sQ17VjSN+bI7Ir/ENl52dw55U7+b97/0fH4joy5bAz1qtQLdVimCL/+Cv36wYwZrk3/+echISHUURkT/izxF5K+K50+H/dhdvpsbjvjNp7o+gQVYiuEOixThO++g8svd6Nw333Xjcg1xhSPJX7P9HXTuXzs5ezP3c+Yy8bQu0VEzyRRbqnCv/7lZtNs1AgmT7YBWcaUVNS38Qc0wOPfPU7X97pSq1It5l0/z5J+mNq5E/7yF/j73+GSSyAtzZK+MaUR1Vf8v+37javHXc2kNZO4vMXlvN7zdapUrBLqsEwRFi92o3B/+gmefRaGDLFRuMaUVtQm/lk/z6LvJ33ZsncLL53/EoNSB1mvnTD19ttuJs0aNdyN3A4dQh2RMZEt6pp6AhrgyVlPcu4751IxtiJzrpvDTe1usqQfhvbvh+uvdyNxO3SAH36wpG9MMETVFf/WfVsZ8J8BfL76cy475TLe6PkG1RKqhTosU4RVq6BvX1ce8b774MEHbRSuMcHiW+IXkQTgW6Cid5yxqvqAiNQAPgIaAuuBPqq63a848s1Jn0Ofj/uwee9mXuzxIje3u9mu8sOQKowc6SZYS0hwo3DPPz/UURlTvvjZ1JMF/ElVWwNtgO4icgYwDJimqk2Bad5j36gq//r+X5z99tnExcTx/bXfM7j9YEv6YWjXLjdn/l//Cu3auRu6lvSNCT4/a+6qqu7xHlbwFgUuBkZ6+0cCvfyKYfv+7fT6qBd3Tr2Ti5pdxMIbF9K2flu/DmeOwbx5cNppMGYMPPIIfPUVNGgQ6qiMKZ98vbkrIrEisgjYAkxV1blAXVXdBOCt6xzmvQNFJE1E0jIyMkp1/Fsm3cKk1ZN4vvvzjO09luSE5FJ9jvFPIABPPulu2ubmwrffwr33Wnu+MX4SVfX/ICLJwDjgVmCmqiYXem67qlY/0vtTU1M1LS2txMfdsGsDG3ZvoH0Dm5A9HP36q5tRc8oU10f/9deh+hH/TzDGlISILFDV1EP3l1Wx9R0iMgPoDmwWkXqquklE6uH+GvBFg6oNaFDV2gvC0eTJcPXVrl3/1VfhhhtsQJYxZcW3ph4Rqe1d6SMiiUBX4L/ABGCA97IBwHi/YjDhJyvLTbnQvTvUqeOmXRg40JK+MWXJzyv+esBIEYnFnWDGqOpnIjIbGCMi1wE/AzYxTpRYuNA17Sxb5gqfP/MMJCaGOipjoo9viV9VlwCnFbF/K9DFr+Oa8JOdDY8+6pa6da1vvjGhFlUjd03ZW7rUteUvWgRXXgkvvGA3cI0Jtaibq8eUjdxceOwxaNsWNm6EcePgvfcs6RsTDuyK3wTdypWuLX/+fOjTB/79b6hVK9RRGWPy2RW/CZq8PHj6aTcC93//g48+coslfWPCi13xm6BYudJNofz999CrF7zyiruRa4wJP3bFb47Jvn1uioXWrV3yf+89+PRTS/rGhDO74jel9sUXbvrkdetcm/6TT7pBWcaY8GZX/KbE0tPhssvgggugYkWYPh3eeceSvjGRwhK/KbbcXHjuOWje3A3CevRRN2d+p06hjswYUxLW1GOKZe5cN83CokXQowe8+CI0ahTqqIwxpWFX/OaItm2Dm26CM8+EjAwYO9Zd7VvSNyZyWeI3RcrOhmefhSZN3Dz5Q4e6XjuXXmozaRoT6Szxm4OowiefwCmnwO23Q/v28MMPbibNKlVCHZ0xJhgs8ZsC8+bBOee4HjuJifDll2459dRQR2aMCSZL/IaffoIrroA//hFWr4bXXnNX+eedF+rIjDF+sF49UWznTnj8cddFMyYG7rsP7rrLmnSMKe/8LL14nIhMF5GVIrJcRIZ4+2uIyFQRWe2tbaLeMpaZ6ebFb9IEnngCLr8cfvwR/vlPS/rGRAM/m3pygTtUtTlwBjBYRE4BhgHTVLUpMM17bMpAZqbrf9+4MQwZ4truFyyAkSMhJSXU0RljyopviV9VN6nqQm97N7ASaABcDIz0XjYS6OVXDMbJyoKXX3ZX+Lfe6tbTp8PXX8Ppp4c6OmNMWSuTm7si0hBXf3cuUFdVN4E7OQBFzvAiIgNFJE1E0jIyMsoizHInOxtefRWaNoWbb4aGDeGrr2DGDJtmwZho5nviF5HKwCfAUFXdVdz3qeprqpqqqqm1a9f2L8ByKCcH3ngDTjrJTbPQoAFMngzffQddutgALGOina+JX0Qq4JL+B6r6qbd7s4jU856vB2zxM4ZokpnpEn6zZnDDDW62zEmTXHGUP//ZEr4xxvGzV48AbwIrVfWZQk9NAAZ42wOA8X7FEC22bXOFzRs2dAm/Rg347DM3sVr37pbwjTEH87MffwfgKmCpiCzy9g0HRgBjROQ64Gegt48xlGvr17v5dN58E/budUn+zjvhT3+yZG+MOTzfEr+qzgQOl366+HXcaLBgATz1FHz8sRt4dcUVLuHb1ArGmOKwkbsRQtXNm/PUU64rZtWqcMcd8Le/WR98Y0zJWOIPczt3wrvvun74K1e6HjpPPeXa8qtVC3V0xphIZIk/TP3wg0v2H3wA+/ZBu3buBHD55RAfH+rojDGRzBJ/GNm/H8aMcQl/7lw3NfIVV7gKWG3bhjo6Y0x5YYk/DKxZA6+8Am+/7bpmnnwyPP88XH01JCeHOjpjTHljiT9E9u2D//wH3nkHpk6FuDi45BJ3dd+pk3XHNMb4xxJ/GQoEYOZMNxvmxx/D7t1wwgnw8MNw/fVQr16oIzTGRANL/GVg7Vp3Y/a992DdOqhcGXr3hgED4OyzXV98Y4wpK5b4fbJzp7tR++677ipfBLp2dcVOevWCpKRQR2iMiVaW+INo9243R86YMW5ytKwsd6P28cfhyittoJUxJjxY4j9GRSX7+vXhxhtdsk9NtRu1xpjwYom/FIpK9vXquWTfuzecdZa12xtjwpcl/mLKyIAvvnBdMAsn+4EDoU8fS/bGmMhhif8wVGH5cpg40S1z5rh99eu7ZN+7N3ToYMneGBN5LPEXkp0N33xzINmvX+/2n346/OMf0LOn27Y2e2NMJIv6xP/zz27k7Jdfurq0u3dDQoLrennPPXDBBW5GTGOMKS98S/wi8hZwIbBFVVt6+2oAHwENgfVAH1Xd7lcMRdm1C2bMcMl+yhT48Ue3v3596NvXXdV36QKVKpVlVMYYU3b8vOJ/B3gReLfQvmHANFUdISLDvMd3+xgDubkwb55L9FOnurb6vDyX2M89FwYNcoXITznFmnCMMdHBz9KL34pIw0N2Xwx08rZHAjPwMfH/85/w9NPuKl/ETW18993QrRuceSZUrOjXkY0xJnyVdRt/XVXdBKCqm0SkzuFeKCIDgYEAxx9/fKkOlpLiCpd06+YKkNesWaqPMcaYckVU1b8Pd1f8nxVq49+hqsmFnt+uqtWP9jmpqamalpbmW5zGGFMeicgCVU09dH9Z90LfLCL1vIDqAVvK+PjGGBP1yjrxTwAGeNsDgPFlfHxjjIl6viV+ERkFzAaaiUi6iFwHjAC6ichqoJv32BhjTBnys1dPv8M81cWvYxpjjDk6m2nGGGOijCV+Y4yJMpb4jTEmyljiN8aYKOPrAK5gEZEM4KdSvr0W8FsQwwkl+y7hp7x8D7DvEq6O5bucoKq1D90ZEYn/WIhIWlEj1yKRfZfwU16+B9h3CVd+fBdr6jHGmChjid8YY6JMNCT+10IdQBDZdwk/5eV7gH2XcBX071Lu2/iNMcYcLBqu+I0xxhRiid8YY6JMuU78ItJdRFaJyBqvxm/EEpH1IrJURBaJSMRUpRGRt0Rki4gsK7SvhohMFZHV3vqoxXjCwWG+y4MissH7XRaJyPmhjLE4ROQ4EZkuIitFZLmIDPH2R9zvcoTvEom/S4KIzBORxd53ecjbH/Tfpdy28YtILPAjbvrndGA+0E9VV4Q0sFISkfVAqqpG1KAUETkH2AO8W6gS25PANlUd4Z2Qq6uqb7WXg+Uw3+VBYI+qPh3K2ErCK4JUT1UXikgVYAHQC7iGCPtdjvBd+hB5v4sASaq6R0QqADOBIcBfCPLvUp6v+NsDa1T1f6qaDYzGFXs3ZUhVvwW2HbL7YmCktz0S9w817B3mu0QcVd2kqgu97d3ASqABEfi7HOG7RBx19ngPK3iL4sPvUp4TfwPgl0KP04nQ/yE8CkwRkQVeIfpIVldVN4H7hwvUCXE8x+oWEVniNQWFffNIYV5d7NOAuUT473LId4EI/F1EJFZEFuHK0k5VVV9+l/Kc+KWIfZHcrtVBVU8HegCDvWYHE3ovA42BNsAm4F8hjaYERKQy8AkwVFV3hTqeY1HEd4nI30VV81S1DZACtBeRln4cpzwn/nTguEKPU4CNIYrlmKnqRm+9BRiHa8qKVJu9ttn8NtotIY6n1FR1s/ePNQC8ToT8Ll4b8ifAB6r6qbc7In+Xor5LpP4u+VR1BzAD6I4Pv0t5TvzzgaYicqKIxAN9ccXeI46IJHk3rhCRJODPwLIjvyusTQAGeNsDgPEhjOWY5P+D9FxCBPwu3k3EN4GVqvpMoaci7nc53HeJ0N+ltogke9uJQFfgv/jwu5TbXj0AXheu54BY4C1VfTS0EZWOiDTCXeWDq5P8YaR8FxEZBXTCTS27GXgA+A8wBjge+Bnoraphf9P0MN+lE645QYH1wI357bHhSkQ6At8BS4GAt3s4rm08on6XI3yXfkTe79IKd/M2FndRPkZVHxaRmgT5dynXid8YY8zvleemHmOMMUWwxG+MMVHGEr8xxkQZS/zGGBNlLPEbY0yUscRvzCFEJFlEbva264vI2FDHZEwwWXdOYw7hzfnyWf4MnMaUN3GhDsCYMDQCaOxNlrUaaK6qLUXkGtzMiLFAS9z8L/HAVUAWcL6qbhORxsC/gdrAPuAGVf1vWX8JYw7HmnqM+b1hwFpvsqy/H/JcS+AK3NwvjwL7VPU0YDZwtfea14BbVbUtcCfwUlkEbUxx2RW/MSUz3Zv3fbeI7AQmevuXAq28WSLPAj5208gAULHswzTm8CzxG1MyWYW2A4UeB3D/nmKAHd5fC8aEJWvqMeb3dgNVSvNGby74dSLSG9zskSLSOpjBGXOsLPEbcwhV3QrM8oqqP1WKj+gPXCcii4HlWMlPE2asO6cxxkQZu+I3xpgoY4nfGGOijCV+Y4yJMpb4jTEmyljiN8aYKGOJ3xhjoowlfmOMiTL/D8Q6qmz4NjzzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f(x, S, k):\n",
    "    return x - S - k(x)*x\n",
    "\n",
    "def fprime(x, S, k):\n",
    "    return 1 - jacobian(k, x) * x - k(x)\n",
    "\n",
    "def my_odeint(f, fprime, S_newt, kx):\n",
    "    N_time = 30\n",
    "    for i in range(N_time):\n",
    "        f_Sk = partial(f, S=S_newt[-1], k=kx)\n",
    "        #fprime_Sk = partial(fprime, S=S_newt[-1], k=kx)     # <---- \"hand done\" derivative function\n",
    "        fprime_Sk = partial(jacobian, f_Sk)                  # <---- autodiff derivative function\n",
    "        S_newt.append(newton_solve(f_Sk, fprime_Sk, torch.tensor((1+k)*S_newt[-1])))\n",
    "    return S_newt\n",
    "\n",
    "all_S_newt = []\n",
    "for S0 in [90, 70, 50, 30, 10]:\n",
    "    all_S_newt.append(my_odeint(f, fprime, [S0], kx))\n",
    "\n",
    "plt.plot(all_S_newt[0], color='red',    label='$S_{0}$=90')\n",
    "plt.plot(all_S_newt[1], color='orange', label='$S_{0}$=70')\n",
    "plt.plot(all_S_newt[2], color='gold',   label='$S_{0}$=50')\n",
    "plt.plot(all_S_newt[3], color='green',  label='$S_{0}$=30')\n",
    "plt.plot(all_S_newt[4], color='blue',   label='$S_{0}$=10')\n",
    "plt.legend()\n",
    "plt.xlabel('time')\n",
    "plt.ylabel('storage')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5948bb88",
   "metadata": {},
   "source": [
    "## Getting to the good stuff: A reservoir with a neural network for $K(S)$\n",
    "\n",
    "Now imagine you're actually a hydrologist - you can't directly measure the conductivity of the \"reservoir\" but you can measure storage levels. If you're interested in determining $K(S)$ from data you have all sorts of avenues, but the one we're interested in is a neural network. In this case, imagine we \"know\" what the dynamics look like (aka the ODE defining the system), but we do not know the functional form for $K(S)$. Our neural network then, will solve the dynamics, and update the weights of a network that represents the conductivity during training. Once trained, we can pull out the network and look at what $K(S)$ was determined to be from the data.\n",
    "\n",
    "Below is the network in question. Note it's just a simple MLP with a single `width` hyperparameter. You might modify this to be a more complex structure, but for simplicity let's give this a go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "79136a9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralReservoir(nn.Module):\n",
    "    \n",
    "    def __init__(self, width):\n",
    "        super().__init__()\n",
    "        # Parameterize K(S) as a neural network\n",
    "        self.K = nn.Sequential(\n",
    "                    nn.Linear(1, width),\n",
    "                    nn.Tanh(),\n",
    "                    nn.Linear(width, width),\n",
    "                    nn.Tanh(),\n",
    "                    nn.Linear(width, 1),\n",
    "                    nn.Tanh())\n",
    "        self.dK_dx = partial(jacobian, self.K)\n",
    "   \n",
    "    def forward(self, S0):\n",
    "        # Reservoir equation (aka the \"dynamics\")\n",
    "        def f(x, S, k):\n",
    "            return x - S - k(x)*x\n",
    "        \n",
    "        f_Sk = partial(f, S=S0, k=self.K)\n",
    "        fprime_Sk = partial(jacobian, f_Sk) \n",
    "        S1_guess = torch.tensor(0.95*S0)\n",
    "        return newton_solve(f_Sk, fprime_Sk, S1_guess)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90ce48b4-81ab-47b9-8fca-87be9f6db02f",
   "metadata": {},
   "source": [
    "## Training data\n",
    "\n",
    "Before we can train the network we need some data - this is where we use our previous solution to generate some synthetic data and we will see how well the network can reconstruct the known conductivity function. I'll just run a few timesteps for a bunch of different initial conditions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4d024f8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:5: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  \"\"\"\n",
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:13: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  del sys.path[0]\n"
     ]
    }
   ],
   "source": [
    "all_S_newt = []\n",
    "for S0 in [0.1, 1, 2, 5, 10, 15, 20, 30, 40, 60, 70, 80, 85, 90, 92, 95, 97, 100, 110, 130, 150, 200]:\n",
    "    all_S_newt.append(my_odeint(f, fprime, [S0], kx))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3be1796-e3e2-4afb-a9a3-8a109a763de4",
   "metadata": {},
   "source": [
    "##  A standard epoch function\n",
    "\n",
    "As usual, let's define this and get it out of the way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "adbb5903",
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch(X, y, model, loss_fun, device=device, opt=None, monitor=None):\n",
    "    total_loss, total_err, total_monitor = 0.,0.,0.\n",
    "    model.eval() if opt is None else model.train()\n",
    "    n_iter = X.shape[0]\n",
    "    for i in tqdm(range(n_iter), leave=False):\n",
    "        Xd, yd = X[i].to(device), y[i].to(device)\n",
    "        if opt:\n",
    "            opt.zero_grad()\n",
    "        yp = model(Xd)\n",
    "        loss = loss_fun(yp, yd)\n",
    "        if opt:\n",
    "            loss.backward(retain_graph=True)\n",
    "            if sum(torch.sum(torch.isnan(p.grad)) for p in model.parameters()) == 0:\n",
    "              opt.step()\n",
    "        total_loss += loss.item() * X.shape[0]\n",
    "        if monitor is not None:\n",
    "            total_monitor += monitor(model)\n",
    "    return total_loss / len(X), total_monitor / len(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c075d39-952f-4785-91d6-006e266af400",
   "metadata": {},
   "source": [
    "## Split out the input/output data\n",
    "\n",
    "Our networks training task is to figure out the storage one timestep later, given it's initial storage. We have 660 samples for training, but you could add more by adding more `S0`'s under the `Training Data` heading or by increasing the number of timesteps in the `my_odeint` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "fa5179cf-eea8-4735-819c-35c15f8e7f97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([660, 1])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X = []\n",
    "train_Y = []\n",
    "for ss in all_S_newt:\n",
    "    train_X.append(ss[0:-1])\n",
    "    train_Y.append(ss[1:])\n",
    "\n",
    "train_X = torch.tensor(np.hstack(train_X).reshape(-1, 1), dtype=torch.float32)\n",
    "train_Y = torch.tensor(np.hstack(train_Y).reshape(-1, 1), dtype=torch.float32)\n",
    "\n",
    "train_X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73f78dfa-7154-4841-b3a0-16efb627b8af",
   "metadata": {},
   "source": [
    "## Let's train!\n",
    "\n",
    "I'll set up the NeuralReservoir with some user-defined width and train a user-defined number of epochs. Feel free to play with these. Following training we can look at the loss curve to see if we have learned anything..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "242c9b50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d3108395b5e4dfbbc7a14b3410cf424",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/30 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:22: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:5: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  \"\"\"\n",
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1, 1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n",
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/ipykernel_launcher.py:13: DeprecationWarning: Calling np.sum(generator) is deprecated, and in the future will give a different result. Use np.sum(np.fromiter(generator)) or the python sum builtin instead.\n",
      "  del sys.path[0]\n",
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/bzq/miniconda3/envs/all/lib/python3.7/site-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1, 1, 1, 1, 1, 1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/660 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "max_epochs = 30\n",
    "width = 16\n",
    "\n",
    "model = NeuralReservoir(width)\n",
    "loss_fun = torch.nn.MSELoss()\n",
    "opt = torch.optim.Adam(model.parameters())\n",
    "\n",
    "loss_history = []\n",
    "for i in tqdm(range(max_epochs)):\n",
    "    train_loss, train_mon = epoch(train_X, train_Y, model, loss_fun, opt=opt)\n",
    "    loss_history.append(train_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "95e88bee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'MSE Loss')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEGCAYAAACZ0MnKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnlElEQVR4nO3de3hV9Z3v8fc3V3IDEpJAyEVAIggpWghUa/UB1BY7jtg6Kk5trdrDjMep7XRm2jp9zpnOxVNPp9Mz2qmd0dapztQLVTvaVrQW8TIVwaAiiFDuJBBISCAkhNy/54+9kG1MwiZkZ2fvfF7Ps5+99m+tvfd3PRv48Fu/tX7L3B0REZFIJMW6ABERiR8KDRERiZhCQ0REIqbQEBGRiCk0REQkYimxLiBa8vPzfcqUKbEuQ0Qkrqxfv/6Quxf0tz5hQ2PKlClUVVXFugwRkbhiZnsGWq/DUyIiEjGFhoiIREyhISIiEVNoiIhIxKIWGmb2oJnVmdmmXu1fNrOtZvaumX03rP1OM9serPtUWPs8M9sYrLvXzCxaNYuIyMCi2dP4KbAkvMHMFgFLgTnuPhv4XtA+C1gGzA7ec5+ZJQdv+xGwHCgPHh/4TBERGT5RCw13fwVo7NV8G3C3u7cH29QF7UuBx9y93d13AduBBWZWBIx19zUemo73YeDqaNUsIiIDG+4xjXOAi81srZm9bGbzg/ZioDpsu5qgrThY7t3eJzNbbmZVZlZVX18/qAIfem03v9ywf1DvFRFJdMMdGilALnAB8FfAimCMoq9xCh+gvU/ufr+7V7p7ZUFBvxc0DmhFVTVPvVlz6g1FREah4Q6NGuApD1kH9AD5QXtp2HYlwP6gvaSP9qgpzc2k+vDxaH6FiEjcGu7Q+C9gMYCZnQOkAYeAZ4BlZpZuZlMJDXivc/daoNnMLgh6JF8Ano5mgWUTMqlubKWnR3c0FBHpLWpzT5nZo8BCIN/MaoC/AR4EHgxOw+0AbgoGuN81sxXAZqALuN3du4OPuo3QmVgZwMrgETWleZm0d/VQ39LOxLFjovlVIiJxJ2qh4e439LPqxn62vwu4q4/2KqBiCEsbUFleJgB7G1sVGiIiveiK8F5KczMAqG5sjXElIiIjj0Kjl+LcDMxCPQ0REfkghUYv6SnJFI0do9AQEemDQqMPJXmZ1DTqtFsRkd4UGn0oy8tUT0NEpA8KjT6U5WVy4GgbbZ3dp95YRGQUUWj04cRptzW6MlxE5AMUGn0ozQtOuz2sQ1QiIuEUGn0oDXoaulZDROSDFBp9KMhOZ0xqEnsbFBoiIuEUGn0wM0pzdQaViEhvCo1+lOVpinQRkd4UGv0ozQtNkR6ahFdERECh0a+yvExa2rs43NoZ61JEREYMhUY/dAaViMiHKTT6EX5fDRERCVFo9OPEBX4KDRGRkxQa/chMSyE/O02Hp0REwig0BlCal6mpREREwkQtNMzsQTOrM7NNfaz7SzNzM8sPa7vTzLab2VYz+1RY+zwz2xisu9fMLFo196Yp0kVEPiiaPY2fAkt6N5pZKXA5sDesbRawDJgdvOc+M0sOVv8IWA6UB48PfWa0lOVlsv9IG13dPcP1lSIiI1rUQsPdXwEa+1j1/4CvA+FXzS0FHnP3dnffBWwHFphZETDW3dd46Cq7h4Gro1Vzb6W5mXT3OLVNbcP1lSIiI9qwjmmY2VXAPnff0GtVMVAd9romaCsOlnu39/f5y82sysyq6uvrz7jeUp12KyLyAcMWGmaWCXwL+N99re6jzQdo75O73+/ule5eWVBQMLhCw5RNUGiIiIRLGcbvOhuYCmwIxrJLgDfNbAGhHkRp2LYlwP6gvaSP9mExaewYUpNNoSEiEhi2noa7b3T3Qnef4u5TCAXCXHc/ADwDLDOzdDObSmjAe5271wLNZnZBcNbUF4Cnh6vm5CSjeHyGrtUQEQlE85TbR4E1wAwzqzGzW/vb1t3fBVYAm4HngNvdvTtYfRvwY0KD4zuAldGquS8nZrsVEZEoHp5y9xtOsX5Kr9d3AXf1sV0VUDGkxZ2GsrxMnt1YG6uvFxEZUXRF+CmU5mVyuLWT5jZNkS4iotA4hbL3p0jXXfxERBQap6Ap0kVETlJonEJprm7GJCJygkLjFMZlpjJ2TIpmuxURQaERkbIJmu1WRAQUGhHRFOkiIiEKjQiU5mZS03icnp5+p70SERkVFBoRKM3LpKO7h7rm9liXIiISUwqNCOi0WxGREIVGBHRfDRGREIVGBIrHZ2CmazVERBQaEUhLSWLyOE2RLiKi0IhQSW6GDk+JyKin0IiQrtUQEVFoRKwsL5O65nbaOrtPvbGISIJSaESobELoDKoazUElIqOYQiNCJbk67VZERKERofcv8GtQaIjI6BW10DCzB82szsw2hbX9o5ltMbN3zOwXZjY+bN2dZrbdzLaa2afC2ueZ2cZg3b1mZtGqeSD52WlkpCZTfVh38BOR0SuaPY2fAkt6tb0AVLj7HOD3wJ0AZjYLWAbMDt5zn5klB+/5EbAcKA8evT9zWJgZpXk67VZERreohYa7vwI09mr7jbt3BS9fB0qC5aXAY+7e7u67gO3AAjMrAsa6+xp3d+Bh4Opo1XwqZXmZusBPREa1WI5p3AKsDJaLgeqwdTVBW3Gw3Lu9T2a23MyqzKyqvr5+iMsNzUFV3dhKKL9EREafmISGmX0L6AJ+dqKpj818gPY+ufv97l7p7pUFBQVnXmgvZXmZHOvopvFYx5B/tohIPBj20DCzm4Argc/5yf+y1wClYZuVAPuD9pI+2mOiVKfdisgoN6yhYWZLgG8AV7l7+L+8zwDLzCzdzKYSGvBe5+61QLOZXRCcNfUF4OnhrDnciQv8FBoiMlqlROuDzexRYCGQb2Y1wN8QOlsqHXghOHP2dXf/U3d/18xWAJsJHba63d1PzNdxG6EzsTIIjYGsJEZO9DRqdNqtiIxSUQsNd7+hj+afDLD9XcBdfbRXARVDWNqgZaQlk5+drgv8RGTU0hXhp6lM12qIyCim0DhNmiJdREYzhcZpKsvLpLbpOJ3dPbEuRURk2Ck0TlNJXiY9DvuPaDBcREYfhcZpen+2Wx2iEpFRSKFxmk6ERnWjehoiMvooNE7TxLFjSE029TREZFRSaJym5CSjJFez3YrI6KTQGIRSnXYrIqOUQmMQyvIyqD6s0BCR0UehMQiluZkcae2k6XhnrEsRERlWCo1BOHkGlXobIjK6KDQGoTTvxGy3Cg0RGV0UGoNQqgv8RGSUUmgMwriMVMZlpCo0RGTUUWgMUmi2W10VLiKji0JjkMryMqlRT0NERplThoaZXWRmWcHyjWb2fTM7K/qljWwleRnUHD5Od4/HuhQRkWETSU/jR0CrmZ0HfB3YAzwc1ariQFleJh3dPdQ26RCViIwekYRGl7s7sBS4x93vAXJO9SYze9DM6sxsU1hbnpm9YGbbgufcsHV3mtl2M9tqZp8Ka59nZhuDdfeamZ3eLkbHnOLxAKzb1RjbQkREhlEkodFsZncCNwK/NrNkIDWC9/0UWNKr7ZvAKncvB1YFrzGzWcAyYHbwnvuC74FQT2c5UB48en9mTMyePJaCnHRWbamLdSkiIsMmktC4HmgHbnX3A0Ax8I+nepO7vwL0/m/4UuChYPkh4Oqw9sfcvd3ddwHbgQVmVgSMdfc1QW/n4bD3xFRSkrFoRgGv/L5et34VkVEjop4GocNSr5rZOcD5wKOD/L6J7l4LEDwXBu3FQHXYdjVBW3Gw3Lu9T2a23MyqzKyqvr5+kCVGbvHMQprbuli/53DUv0tEZCSIJDReAdLNrJjQIaWbCR16Gkp9jVP4AO19cvf73b3S3SsLCgqGrLj+fKK8gNRkY7UOUYnIKBFJaJi7twKfBX7g7p8hNPYwGAeDQ04Ezyf+ta0BSsO2KwH2B+0lfbSPCNnpKXxs6gSNa4jIqBFRaJjZhcDngF8HbckDbD+QZ4CbguWbgKfD2peZWbqZTSU04L0uOITVbGYXBGdNfSHsPSPCopmFbK9r0Yy3IjIqRBIaXwXuBH7h7u+a2TRg9aneZGaPAmuAGWZWY2a3AncDl5vZNuDy4DXu/i6wAtgMPAfc7u7dwUfdBvyY0OD4DmBl5LsXfYtnhoZlXlRvQ0RGAQudlBTBhmY5gLt7S3RLGhqVlZVeVVU1LN+1+HsvUZKXycO3LBiW7xMRiRYzW+/ulf2tj2QakY+Y2VvAJmCzma03s8GOaSSkRTMLeX1nA60dXbEuRUQkqiI5PPVvwNfc/Sx3LwP+AnggumXFl8UzC+no6uF32xtiXYqISFRFEhpZ7v7+GIa7vwRkRa2iODR/Sh7Z6Ska1xCRhJcSwTY7zex/Af8RvL4R2BW9kuJPWkoSF5fns3pLHe7OCJkeS0RkyEXS07gFKACeCh75wBejWFNcWjSzkANH29hcezTWpYiIRM0pexrufhi4I7zNzB4nNCeVBBbOCF2BvnpLHbMnj4txNSIi0THYO/ddOKRVJIDCnDHMKRmncQ0RSWi63esQWjyzkLeqj9DQ0h7rUkREoqLf0DCzuf085hHZ/TRGncUzC3GHl38f/Rl2RURiYaAxjX8aYN2WoS4kEVRMHkd+djovbqnjs3NLTv0GEZE4029ouPui4SwkEZy4MdPz7x6gs7uH1GQd/RORxKJ/1YbYpecWclQ3ZhKRBKXQGGK6MZOIJDKFxhDLTk9hwdQ8nXorIglpoLOnbgxbvqjXuj+LZlHxbtGMQrbpxkwikoAG6ml8LWz5B73W3RKFWhLGpedOBHRjJhFJPAOFhvWz3NdrCTM1P4up+VkKDRFJOAOFhvez3Ndr6WXRjELW6MZMIpJgBgqNmWb2jpltDFs+8XrGMNUXt3RjJhFJRANdEX5utL7UzP4c+BKhHstG4GYgE3gcmALsBq4LZtjFzO4EbgW6gTvc/flo1TZUFkzNIystmRe31HH5rImxLkdEZEj029Nw9z3hD6AFmAvkB68HxcyKCU21XunuFUAysAz4JrDK3cuBVcFrzGxWsH42sAS4z8ySB/v9wyV0Y6YCXtoaujGTiEgiGOiU21+ZWUWwXARsInTW1H+Y2VfP8HtTgAwzSyHUw9gPLAUeCtY/BFwdLC8FHnP3dnffBWwHFpzh9w+LxTMLqW1q473a5liXIiIyJAYa05jq7puC5ZuBF9z9D4GPcQan3Lr7PuB7wF6gFmhy998AE929NtimFigM3lIMVId9RE3Q9iFmttzMqsysqr4+9jPNLpwZujHTi1sOxrgSEZGhMVBodIYtXwo8C+DuzUDPYL/QzHIJ9R6mApOBrPALCft6Sx9tfR7vcff73b3S3SsLCgoGW+KQ0Y2ZRCTRDBQa1Wb2ZTP7DKGxjOcAzCyDM7ufxmXALnevd/dOQvcd/zhwMDgMduJw2Il/aWuA0rD3lxA6nBUXFs0I3Zip8VhHrEsRETljA4XGrYQGn78IXO/uR4L2C4B/P4Pv3AtcYGaZZmaEejHvAc8ANwXb3AQ8HSw/Aywzs3QzmwqUA+vO4PuH1ckbM6m3ISLxb6D7adQBf9pH+2pg9WC/0N3XmtkTwJtAF/AWcD+QDawws1sJBcu1wfbvmtkKYHOw/e3u3j3Y7x9uHykeR0FOOj97fS9XnVdMcpIupheR+GX9nQ5qZs8M9EZ3vyoqFQ2RyspKr6qqinUZADy5voa/+PkG7ri0nK9dfk6syxER6ZeZrXf3yv7WD3Rx34WEzlp6FFiL5psatGvmlfD6zgZ+8OI2FkzJ4xPl+bEuSURkUAYa05gE/DVQAdwDXA4ccveX3f3l4Sgukfzd0grKC7P56uNvUXe0LdbliIgMykBXhHe7+3PufhOhwe/twEtm9uVhqy6BZKQl88M/nsux9m6+/OhbdHUP+qxlEZGYGfDOfcEZS58F/hO4HbiX0CmyMgjlE3P4h6srWLurkXtXbYt1OSIip63fMQ0ze4jQoamVwN+GXR0uZ+D98Y3V25k/NY+Ly2N/EaKISKQG6ml8HjgH+ArwmpkdDR7NZnZ0eMpLTO+Pbzz2Ngc1viEicWSgMY0kd88JHmPDHjnuPnY4i0w0J8Y3Wju6uUPjGyISRwYc05Do0fiGiMQjhUYMXTOvhGvnlfCD1dt5dVvsZ+UVETkVhUaMaXxDROKJQiPGNL4hIvFEoTECaHxDROLFQHNPyTAKv34jKz2F5ZdMIzRzvIjIyKHQGEH+/uoKWju6+c7KLWyra+Guz1SQnpIc67JERN6n0BhBxqQm84MbPsr0wmzuWbWNPQ3H+Ncb5zEhOz3WpYmIABrTGHGSkow/v/wcfnDDR3mnpomlP/wdWw80x7osERFAoTFi/eF5k1nxJxfS0dXDZ+/7HaveOxjrkkREFBoj2Xml43nmzz7B1IIsvvRwFfe/soP+7rQoIjIcFBoj3KRxY/j5n3ycT1cU8X+e3cJfPfEO7V1xc4t0EUkwMQkNMxtvZk+Y2RYze8/MLjSzPDN7wcy2Bc+5YdvfaWbbzWyrmX0qFjXHUkZaaID8K5eW88T6Gm788VoaWtpjXZaIjEKx6mncAzzn7jOB84D3gG8Cq9y9HFgVvMbMZgHLgNnAEuA+Mxt156H2NUC+blejDleJyLAa9tAws7HAJcBPANy9w92PAEuBh4LNHgKuDpaXAo+5e7u77yJ029kFw1nzSBI+QH7dv63h0u+/zH0vbedAk+atEpHoi0VPYxpQD/y7mb1lZj82syxgorvXAgTPhcH2xUB12PtrgrYPMbPlZlZlZlX19Yk7a+x5peN58S8X8t1r5pCflc53n9vKx+9exU0PruOXG/bT1qkxDxGJjlhc3JcCzAW+7O5rzewegkNR/ehrLo0+j8m4+/3A/QCVlZUJfdwmOz2F6+aXct38UnYfOsaTb9bw5PoavvzoW4wdk8JV50/mj+aVcl7JOE1HIiJDJhahUQPUuPva4PUThELjoJkVuXutmRUBdWHbl4a9vwTYP2zVxoEp+Vn8xSdn8OeXncNrOxp4Yn01P6+q4T9f38v0wmyunVfCsgVljMtIjXWpIhLnLBYDqWb2KvAld99qZt8GsoJVDe5+t5l9E8hz96+b2WzgEULjGJMJDZKXu/uAx2AqKyu9qqoqejsxwh1t6+TZd2r5+foa1u85TE56CjdeeBa3XDSVghxNSyIifTOz9e5e2e/6GIXG+cCPgTRgJ3AzofGVFUAZsBe41t0bg+2/BdwCdAFfdfeVp/qO0R4a4Tbta+JHL+/g2Y21pCUncV1lKcsvmUZpXmasSxOREWZEhsZwUGh82M76Fv7t5Z089VYNPQ5Lz5vMbQvPpnxiTqxLE5ERQqEhH1LbdJwHXtnFo+v2cryzm0/Omsj/XDSd80vHx7o0EYkxhYb0q/FYBz/93S5++tpujrZ1cdH0CXzp4mlcUl5AcpLOuBIZjRQackot7V08snYPD7y6i/rmdorHZ3BdZSnXzS+haFxGrMsTkWGk0JCIdXT18MLmgzy6bi//vf0QSQYLZxRyw4IyFs0oICVZ81uKJDqFhgzK3oZWHq/ay4qqGuqb25k4Np1r55Vy/fxSnXUlksAUGnJGOrt7eHFLHY+t28tLvw9NzfKJ6fksm1/G4pmFZKSNurkjRRKaQkOGzL4jx1nxRjUrqqqpbWojIzWZRTMLWFJRxOKZhWSn65bzIvFOoSFDrrvHWbOjgZWbann+3YMcamknLSWJS8rzuaKiiMvOnci4TE1ZIhKPFBoSVd09zvo9h0MBsukA+5vaSEkyPj49nysqJvHJWROZkK1pS0TihUJDho27s6GmiZWbalm58QB7G1tJMlgwNY8rKopYUjGJiWPHxLpMERmAQkNiwt3ZXHuU5zYdYOWmA2yvawFg3lm5XFExiSUVkyjJ1VlYIiONQkNGhO11zazceIBnNx3gvdqjAHykeBxLKiZxRcUkphVkx7hCEQGFRqzLkD7saTjGyqAHsqH6CAAzJ+WwpGISV84pYnqhJlAUiRWFhoxo+44c57lNB3huUy1Vew7jHgqQK+cU8QdzJjM1P+vUHyIiQ0ahIXHj4NE2nt1Yy6/fCQUIwOzJY7lyzmSunFOkK9FFhoFCQ+LS/iPHeXZjLb96p5a3g0NY55WM48o5k/n0nCKKx2siRZFoUGhI3KtubOXXQQ9k474mAOZPyeWzc0v4gzlFjB2jCwlFhopCQxLK7kPH+PXGWp56s4Yd9cdIS0nik7Mmcs3cEi4uz9dMvCJnSKEhCcndeaemiaferOGZDfs53NpJfnY6S8+fzGfnFjN78rhYlygSl0ZsaJhZMlAF7HP3K80sD3gcmALsBq5z98PBtncCtwLdwB3u/vypPl+hMXp0dPXw0tY6nnpzH6u2HKSz25k5KYdr5paw9PzJFOoqdJGIjeTQ+BpQCYwNQuO7QKO7321m3wRy3f0bZjYLeBRYAEwGfguc4+7dA32+QmN0Onysg1+9s58n39zH29VHSE4yLju3kM997Cw+MT2fJN3GVmRApwqNmMxlbWYlwB8AdwFfC5qXAguD5YeAl4BvBO2PuXs7sMvMthMKkDXDWLLEidysND5/4RQ+f+EUdtS3sKKqmieqanj+3YOU5WXyxx8r49p5JZpEUWSQYjVq+M/A14GesLaJ7l4LEDwXBu3FQHXYdjVBm8iAzi7I5s4rzuW1Oxdz7w0fpWjcGO5euYULv/Midzz6Fmt3NpCoY3oi0TLsPQ0zuxKoc/f1ZrYwkrf00dbn33QzWw4sBygrKxtsiZJg0lOSueq8yVx13mS21zXzs7V7eXJ9aAC9vDCbz32sjM/MLWFchk7dFTmVYR/TMLPvAJ8HuoAxwFjgKWA+sNDda82sCHjJ3WcEg+C4+3eC9z8PfNvdBzw8pTENGcjxjm5+9c5+frZ2L29XH2FMahKf/kgRfzS3hAumTdDYh4xaI3YgHCDoafxlMBD+j0BD2EB4nrt/3cxmA49wciB8FVCugXAZKpv2NfHIur388u39NLd3UTw+g898tJhr5pVo7isZdeIpNCYAK4AyYC9wrbs3Btt9C7iFUO/kq+6+8lSfrdCQ09XW2c1vNh/kyfU1vLqtnh4P3f/jmuDKcx2+ktFgRIdGNCk05EwcPNrGf721jyfW17CtruXklefzSrh4uq48l8Sl0BA5A+7Oxn1NPLm+hqc37OdIcOX55bMKuezciVw0PZ8xqcmxLlNkyCg0RIZIR1cPL26p45cb9vPy7+tpae9iTGoSF5cXcPm5E1k0s5CCHF3/IfFtRF7cJxKP0lKSWBLc37yjq4e1uxr47eaD/Pa9Ol7YfBAz+GjpeC6bNZHLzp1IeWE2ZjoLSxKLehoiZ8jdea+2md++d5DfvneQd2pC07eX5WVy6bmFLJpRyIKpeTqMJXFBh6dEhtmBpjZWbTnIbzcf5LUdDbR39ZCRmsxF0yewaGYhC2cU6iZSMmIpNERi6HhHN6/vbGD11jpe3FJHzeHjAMyYmMPCmQUsnlHI3LNySdXZWDJCKDRERgh3Z0d9C6u31LN6ax3rdjXS1ePkjEnhwmkTOGdiDlPzs5hakMW0/CzGZ6bFumQZhTQQLjJCmBnTC3OYXpjD/7hkGs1tnfxu+yFWb6ln3e5GVm2po7vn5H/icjNTQyGSn820gqxgOYvywmxdJyIxo56GyAjR2d1DdWMruw4dY9ehY+w8dIxd9aHlA0fb3t+uICeda+aWcP38Uk1zIkNOPQ2ROJGanMS0gmymFWR/aN2x9i52Nxxj28EWfvVOLQ+8upN/fXkHC6bmcX1lKZ/+SBEZaTo7S6JPPQ2ROFR3tI0n3qxhxRvV7G5oJSc9havOn8yy+WVUFI/V9SEyaBoIF0lg7s7aXY2seKOaX2+spb2rh3OLxrJsfqj3kZacRGdPD13dTmd3D109Tld3D53dTldP8NzdQ3pqMtnpKeSMSSErPYXM1GRNDz9KKTRERomm45088/Y+Hq+qZtO+o2f0WWaQnZZC9pgUstNPPo8dk8qE7DTys9PJz04PWw49Z6Ylq5cT5zSmITJKjMtIff/+6Jv2NfH6zgbMjNRkIyUpiZTkk8upyUbyibakJDq6u2lu66KlvYuWti6OtXfRHCy3tIcezW1d7Dt8nIZjHTQd7+yzhjGpSUGYpDN78lgunp7Px8/OZ1ymppVPFAoNkQRUUTyOiuJxUfv8jq4eGo6109DSQX1L6PlQSzsNLe0caumgrrmNZ97ezyNr95JkcF7peC6ens/F5xRwful4XcwYx3R4SkSiorO7h7erj/DqtkO8uq2eDdVH6HHITk/hgmkTuOScfC4uL2DKhEwd0hpBNKYhIiNCU2sna3Ye4pUgRKobQ1OqFI/PYP6UXCqn5LFgah7TC7LPeBC+p8c1kD9ICg0RGZH2NBzjlW2HWLPjEG/sPkx9czsQGpupPCsUIvOn5PKRknGkp3z4GpTO7h72Nrays/4YO+tbQs+HQs+NrR0U5qQzeXxG6DFuzPvLxeMzKBo3hrysNPVw+qDQEJERz93Z29jKG7sPU7W7kTd2N7Kj/hgQuo/J+SXjmXtWLt09PaGr5euPsbexla6waVfys9OYFky5kp+dzsGjbdQ2tbH/yHH2HTlOe1fPB74zPSWJ4vEZfGzaBK6fX8p5JeMUIozA0DCzUuBhYBLQA9zv7veYWR7wODAF2A1c5+6Hg/fcCdwKdAN3uPvzp/oehYZIfGtoaadqz4kQOcymfU0kJRlTJ2QxrSB4BCExrSCbcRn9n6Hl7hxu7Xw/QPYfOU5tUxt7Go7x8u/raevsYeakHK6tLOUzHy0mL+v0J4vcdegYL22t47UdDZQXZnPzRVPj8k6OIzE0ioAid3/TzHKA9cDVwBeBRne/28y+CeS6+zfMbBbwKLAAmAz8FjjH3bsH+h6Fhkhiae/qJiUpieQhHqs42tbJLzfsZ8Ub1WyoaSItOYnLZ03kuvmlfGJ6fr/f19bZzdpdjazeUsdLW+vY3dAKQEluBvuOHCctOYnrKktZfsk0SvMyh7TmaBpxofGhAsyeBv4leCx099ogWF5y9xlBLwN3/06w/fPAt919zUCfq9AQkdO15cBRHn+jml+8tY8jrZ0Uj8/gmnklXDuvhNK8TKobW3lpax2rt9bz2o5DtHX2MCY1iQunBTfYOqeQsgmZ7Khv4f6Xd/LUWzX0OPzhnCL+dOHZzJw0Nta7eEojOjTMbArwClAB7HX38WHrDrt7rpn9C/C6u/9n0P4TYKW7PzHQZys0RGSw2ru6eWHzQVZU1fDqtnoAJo8L9SAgdCvfxTMLWTijgAumTej3Vr4Hmtr48as7eWTdXlo7url0ZiH/c9HZzDsr74xr7Oru4VhHN60dXRxr/+Dz4pmFgx6fGbGhYWbZwMvAXe7+lJkd6Sc0fgis6RUaz7r7k3185nJgOUBZWdm8PXv2DMeuiEgC23fkOE9U1fDu/iYumDaBhTMKmJqfdVr/KB9p7eCh1/bw09d2cbi1kwVT8rht0dksPKcAM+N4Rzf1ze3Ut7RzqKWd+uYPPh9q6aC5rfNkOHR009FrYD/clr9fMuh70o/I0DCzVOBXwPPu/v2gbSs6PCUiCay1o4vH36jmgVd2sr+pjfzsdI4HIdCXvKy09+f1Gjsmlaz0FLLSk8lMSyErLZnM9F7PaaE5wmZNHjvosZ8RN/eUheL5J8B7JwIj8AxwE3B38Px0WPsjZvZ9QgPh5cC64atYRGRoZKalcPNFU/ncx87i6bf38dqOBnIz08jPSaMgO538nHQKstMpyEknLyttRE63Eou5py4CPg9sNLO3g7a/JhQWK8zsVmAvcC2Au79rZiuAzUAXcPupzpwSERnJ0lKSuLaylGsrS2Ndymkb9tBw9/8G+us3XdrPe+4C7opaUSIiEpGR1/cREZERS6EhIiIRU2iIiEjEFBoiIhIxhYaIiERMoSEiIhFTaIiISMRiPstttJhZPTDYyafygUNDWE6sJdr+QOLtU6LtDyTePiXa/kDf+3SWuxf094aEDY0zYWZVA829Em8SbX8g8fYp0fYHEm+fEm1/YHD7pMNTIiISMYWGiIhETKHRt/tjXcAQS7T9gcTbp0TbH0i8fUq0/YFB7JPGNEREJGLqaYiISMQUGiIiEjGFRhgzW2JmW81su5l9M9b1DAUz221mG83sbTOLy/vfmtmDZlZnZpvC2vLM7AUz2xY858ayxtPRz/5828z2Bb/T22b26VjWeDrMrNTMVpvZe2b2rpl9JWiP59+ov32Ky9/JzMaY2Toz2xDsz98G7af9G2lMI2BmycDvgcuBGuAN4AZ33xzTws6Qme0GKt09bi9KMrNLgBbgYXevCNq+CzS6+91BwOe6+zdiWWek+tmfbwMt7v69WNY2GGZWBBS5+5tmlgOsB64Gvkj8/kb97dN1xOHvFNxmO8vdW8wsFfhv4CvAZznN30g9jZMWANvdfae7dwCPAUtjXJMA7v4K0NireSnwULD8EKG/0HGhn/2JW+5e6+5vBsvNwHtAMfH9G/W3T3HJQ1qCl6nBwxnEb6TQOKkYqA57XUMc/yEJ48BvzGy9mS2PdTFDaKK710LoLzhQGON6hsKfmdk7weGruDmUE87MpgAfBdaSIL9Rr32COP2dzCzZzN4G6oAX3H1Qv5FC46S+7lueCMfuLnL3ucAVwO3BoREZeX4EnA2cD9QC/xTTagbBzLKBJ4GvuvvRWNczFPrYp7j9ndy9293PB0qABWZWMZjPUWicVAOUhr0uAfbHqJYh4+77g+c64BeEDsMlgoPBcecTx5/rYlzPGXH3g8Ff6h7gAeLsdwqOkz8J/Mzdnwqa4/o36muf4v13AnD3I8BLwBIG8RspNE56Ayg3s6lmlgYsA56JcU1nxMyygkE8zCwL+CSwaeB3xY1ngJuC5ZuAp2NYyxk78Rc38Bni6HcKBll/Arzn7t8PWxW3v1F/+xSvv5OZFZjZ+GA5A7gM2MIgfiOdPRUmOH3un4Fk4EF3vyu2FZ0ZM5tGqHcBkAI8Eo/7ZGaPAgsJTeN8EPgb4L+AFUAZsBe41t3jYnC5n/1ZSOiQhwO7gT85cax5pDOzTwCvAhuBnqD5rwmNAcTrb9TfPt1AHP5OZjaH0EB3MqHOwgp3/zszm8Bp/kYKDRERiZgOT4mISMQUGiIiEjGFhoiIREyhISIiEVNoiIhIxBQaIoNkZt1hs52+PZQzI5vZlPBZcEVGipRYFyASx44H0zKIjBrqaYgMseAeJv83uH/BOjObHrSfZWargsnuVplZWdA+0cx+EdzrYIOZfTz4qGQzeyC4/8Fvgit5RWJKoSEyeBm9Dk9dH7buqLsvAP6F0CwDBMsPu/sc4GfAvUH7vcDL7n4eMBd4N2gvB37o7rOBI8A1Ud0bkQjoinCRQTKzFnfP7qN9N7DY3XcGk94dcPcJZnaI0I19OoP2WnfPN7N6oMTd28M+Ywqh6avLg9ffAFLd/R+GYddE+qWehkh0eD/L/W3Tl/aw5W40BikjgEJDJDquD3teEyy/Rmj2ZIDPEbrlJsAq4DZ4/0Y5Y4erSJHTpf+5iAxeRnAntBOec/cTp92mm9laQv8xuyFouwN40Mz+CqgHbg7avwLcb2a3EupR3EboBj8iI47GNESGWDCmUenuh2Jdi8hQ0+EpERGJmHoaIiISMfU0REQkYgoNERGJmEJDREQiptAQEZGIKTRERCRi/x+SeSvALsB+lwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss_history)\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('MSE Loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce8df5c0-3b5e-4b20-984a-352107e5cc8f",
   "metadata": {},
   "source": [
    "## What did the network actually learn though?\n",
    "\n",
    "Of course, our network was evaluated on how it was able to predict the next timestep's storage, but we were interested in getting $K(S)$ out. Lucky for use, we can just pull it out with `model.K` and start inputting storage values. Let's see how we did!\n",
    "\n",
    "That's the end of what I've got here - but hopefully it's been informative and you can see some other applications of this type of approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "807244f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Reservoir constant')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEGCAYAAABLgMOSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA8R0lEQVR4nO3deZxN9f/A8df73jv7Zhb7NpOUELKlL8nSolJK+BJFylL8UiqRUtKqqEhCkSKprKWibKVki2TNzlhmxjZj9u3z++NevoMZc417587yfj4e53HnnPP5nPM+Y8x7zvl8zucjxhiUUkqpy2XxdABKKaWKJ00gSimlCkQTiFJKqQLRBKKUUqpANIEopZQqEJunAyhMERERJjIy0tNhKKVUsbJhw4bjxpiyF24vVQkkMjKS9evXezoMpZQqVkTkQG7b9RGWUkqpAtEEopRSqkA0gSillCqQUtUGopQq/jIyMoiOjiY1NdXToZQ4vr6+VKlSBS8vL6fKawJRShUr0dHRBAUFERkZiYh4OpwSwxjDiRMniI6OJioqyqk6+ghLKVWspKamEh4ersnDxUSE8PDwy7qz0wSilCp2NHm4x+V+X/URVjFl0tLJOHSMrLhTZJ2MJ/vkabITkjCZWZCVjcnOQry8sAT6YwnyxxIShK1SOWyVy2EJDdb/gEqpK6YJpIgzWVlk7DlE2sYdpG3aQfq/+8nYd5jM6Bgo4Fwu4u+L19XV8LnuKrxr18Cn/rX4NKyNxc/HxdErVfKcPn2aL7/8kieeeMKt55k/fz7XXHMNtWvXdut5roQmkCLGGEP6jn2krFxHyor1pKzZjElMBkD8/fCuFYnvjdfj1fVOvCIrYS0fgTU0GGt4CJaQILBaEasFrBZMeibZZ5IwiclknU4g83CsfYk+RvrO/SQvX8uZ2T/ZT+zthW+DWvjeVB//Njfi27QuYtMfD6UudPr0aT766COnE4gxBmMMFsvltRjMnz+f9u3bF+kEIqVpRsLGjRubggxlkjBzEakbtlJu7BA3RAUmO5vUNf+QtHA5iYt+JetoHABeNarid3NDfBrWxveG6/CqWQ2xWl167qzjp0j9azupf/5Nyuq/Sdu0AzKzsISFEHDbTQTcdTP+bZshPt4uPa9SBbV9+3auu+46j52/a9euLFiwgGuvvZbWrVuzefNmTp06RUZGBq+99hodOnRg//793HnnnbRu3ZrVq1czf/58Pv/8c2bOnEnVqlWJiIigUaNGPPvss+zZs4cBAwYQFxeHv78/U6ZM4eTJk7Rv356QkBBCQkKYM2cONWrUKJTry+37KyIbjDGNLyyrf2I6IWPvIc589SNlRw926V/l6bsPkvDFdyTO/YWsY8cRX2/82zbD/9be+LVqgleV8i47V16sEaEE3P4fAm7/DwDZickkL11D0uJVJC3+nTOzf8ISGkxgx1sJ7nYX3vWu0fYTVWQ89dRTbNq0yaXHbNCgAe+//36e+9966y22bNnCpk2byMzMJDk5meDgYI4fP06zZs249957Adi5cyfTpk3jo48+Yv369cyZM4eNGzeSmZlJw4YNadSoEQB9+/bl448/pmbNmqxZs4YnnniCZcuWce+999K+fXs6derk0utzJU0gTvC+JhIyMsnYfwTvq6td0bFMWjqJP/xKwvSFpP6+EWxW+y/wDm0IuP0/WAL9XRN0AVkC/Qns0JrADq0xGZkkr1xP4uwfOTPjexI+nYt33ZqE9OtM0P1t9a5ElXrGGF544QV+/fVXLBYLhw8fJiYmBoDq1avTrFkzAFatWkWHDh3w8/MD4J577gEgMTGRP/74g86dO587ZlpaWiFfRcFpAnGC17WRAKTv3F/gBJKx7zAJXywkYdYPZB8/ja16RcJe7EdQt7uwlQtzYbSuI142Am5tRsCtzcg6fYbE+UuJ/3Qucf/3Bidf/Zjg3vcR8ugDWEODPR2qKqUudadQGGbOnElcXBwbNmzAy8uLyMjIc+9RBAQEnCuXV1NBdnY2ZcqUcfldVGHR90CccDZpZOzcf1n1TFo6ifOWcqTjIA427crpj2bjd2M9Kn49hmprvyJ0UI8imzwuZC0TREiv+6j663QqfjMWn/rXcOrtqRxo2JmTb31C1ukzng5RqUIRFBTEmTP2n/f4+HjKlSuHl5cXy5cv58CBXEc9p0WLFnz33XekpqaSmJjIokWLAAgODiYqKopvvvkGsCeav//++6LzFFV6B+IES6A/XldXI3XD1kuWO9fldtMOkpeuIfmXP8lOSMRWrSJhw/oQ9OBd2CpEFFLU7iEi+Ldqgn+rJqRt38upd6Zxasx04qfMIeTxLpR5vCuWAD9Ph6mU24SHh9O8eXPq1q1LkyZN2LFjB40bN6ZBgwbUqlUr1zpNmjTh3nvvpX79+lSvXp3GjRsTEhIC2O9iHn/8cV577TUyMjLo2rUr9evXp2vXrvTp04dx48bx7bffFloj+uXQXlhOin36bZK+W0Hkv4sQR3e87ORUkr5bQfKyNaTv3E/G3kOYFPvzS0tEGQJub07gfW3wu6XxuTolUdrW3Zx6ZxpJi37FWiGC8Jf6Edjp9hJ9zcpzPN0Lq6ASExMJDAwkOTmZli1bMnnyZBo2bOjpsC6ivbDcwK9FQ87M+J7U1X/j1/wGkn/bQNzg0WTuP4K1fDg+19fE7+aGeNe5Gp/ra+JdK8rlXW6LKp86V1Phs9dJXfsPx18cT+yA14n/ZC4Rbw7Ct1EdT4enVJHQt29ftm3bRmpqKj179iySyeNyaQJxUsCdN2MJDebk65PxqlGVM1/9iFdUFSp+M9Z+h6FdW/Ftej2Vf/qYxG+XcGLUJA7f+Tghj3YkbHhfj/cuU8rTvvzyS0+H4HL6jMFJFn9fwl/qR+r6rZyZ8zNlBnajyopp+LdqoskjB7FYCOrSjmqrZxLc+37iP53LoRYPkbTkd0+HppRyMb0DuQzBD92L/x0tEC+bdl3NhyXQn7JvPU3QA7cRO3g0x7oPJei/7Yh462m9G1GqhNA7kMtkKxemyeMy+DapS9WlnxI6uCdnvlnCodaPkLpui6fDUkq5gCYQ5Xbi7UXYsMeotGA8ZBsO3zOQk+9Mw2RleTo0pdQV8GgCEZF2IrJTRHaLyNBc9tcSkdUikiYiz15OXVX0+DWrR5XlUwns2JZTo6dytOtzZJ047emwlCoWIiMjOX78+BUfZ8WKFfzxxx8uiMiDCURErMAE4E6gNtBNRC4ct/gk8CTwbgHqqiLIGhxI+Y9eouzYIaSu/pvoWx8jdeN2T4ellMdlFdIduSsTiCcb0ZsCu40xewFE5CugA7DtbAFjTCwQKyJ3X25dV1q0aBHr1q0719tKRM5bLtx2peuuOIbFYsFms+Hl5YXNZst3ubCcj48Pfn5++Pv74+Xl5fLvafBD9+B9fU1ier/E4fYDKPvW0wQ/dI/Lz6OUq50dqr1Fixb88ccfVK5cmQULFuDn55fr0Oy1atWiV69e542sGxgYSGJiIitWrGDkyJFUrFiRTZs2sW3bNu677z4OHTpEamoqgwYNom/fvpeMJzAwkEGDBvH999/j5+fHggULKF++PHFxcfTv35+DBw8C9nHDKleuzMcff4zVamXGjBmMHz+em2++ucDfC08mkMrAoRzr0cCNrq4rIn2BvgDVqhVsIMRFixYxceLEAtUtCaxWK/7+/vj5+Z1LKmc/g4KCKFOmDKGhoRd9hoaGUqFCBSpVqkRISMhF3Z19G9Siyi+fENNvJHGDR5O+5yDhL/UvNS9gqit3fPg40rbscukxferWJOL1Jy9ZZteuXcyaNYspU6bQpUsX5syZQ48ePfIcmv1S1q5dy5YtW4iKigJg6tSphIWFkZKSQpMmTXjggQcIDw/Ps35SUhLNmjXj9ddfZ8iQIUyZMoUXX3yRQYMG8fTTT9OiRQsOHjzIHXfcwfbt2+nfvz+BgYE8++yzeR7TWZ5MILm9POHsuCpO1zXGTAYmg30oEyePf56PPvqICRMmnD3eecuF26503VXHzMrKIjMzM98lIyMj121paWmkpKSQnJxMSkpKrl8nJycTFxfHrl27OHXqFKdPn87zNtzX15dKlSqdW6KioqhZsyY1a9bk6rGDCR4/m/gJX5GxN5ryE0foeFqqSIuKiqJBgwYANGrUiP379xd4aPamTZueSx4A48aNY968eQAcOnSIXbt2XTKBeHt70759+3Ox/PzzzwD88ssvbNv2v4cyCQkJLh+c0ZMJJBqommO9CnCkEOoWSM7HRCp3xhgSExPPJZOTJ09y7Ngxjhw5wpEjRzh69ChHjhxh48aNzJs3j4yMjHN1AwICGFzleh76cRXH/tOV7HeepO4tLfDx0XnaVd7yu1Nwl5w/l1arlZSUlEsOzW6z2cjOzgYc01anp5/bl3PY9xUrVvDLL7+wevVq/P39adWq1bnh4fPi5eV17veS1WolMzMTsA8Vv3r16nNzkLiDJxPIOqCmiEQBh4GuwIOFUFe5iYgQFBREUFBQvo8LMzMzz/11dXZZtXkzf8Xs4o3DWZzu+jx1U3dRtVkjWrVqRatWrbjxxhs1oagiK+fQ7J07d8YYw+bNm6lfvz6RkZFs2LCBLl26sGDBgvP+eMopPj6e0NBQ/P392bFjB3/++WeB47n99tv58MMPee655wDYtGkTDRo0ICgoiISEhAIfNyeP9cIyxmQCA4HFwHbga2PMVhHpLyL9AUSkgohEA4OBF0UkWkSC86rrmStRBWGz2YiKiuL2229nwIABvP/++yxbtozvTh4kYOpIKgWHMieiEcGx8bzyyivccsstRERE0KVLF2bNmkV8fLynL0Gpi8ycOZNPP/2U+vXrU6dOHRYsWABAnz59WLlyJU2bNmXNmjXn3XXk1K5dOzIzM6lXrx4vvfTSuRkNC2LcuHGsX7+eevXqUbt2bT7++GPAPhvivHnzaNCgAb/99luBjw86nLsqotJ37ONI58GYlFQCJo9g9ZlYFi1axIIFC4iNjcXb25v777+fxx57jDZt2mDRoeNLjeI6nHtxcTnDuev/OlUkedeKovKij7CGlSHxkRHcHl6VyZMnc+TIEVatWkW/fv1YsmQJt912G1dddRXvvPOOy27LlVLO0QSiiiyvahWp9P0EbNUqcvTBIaT8vhGr1Urz5s0ZN24cR44cYdasWVx11VUMGTKE6tWrM3z4cJe8rauUyp8mEFWk2cqFUWnuB/9LIn9sOrfP19eXrl27smzZMtauXUvbtm158803qVGjBqNHj86394oqvkrTo/fCdLnfV00gqsizlQ21J5Eq5TnabQgpq/++qEyTJk349ttv2bJlCy1btuT555+nVq1aLFq0yAMRK3fy9fXlxIkTmkRczBjDiRMn8PX1dbqONqKrYiMz5gRH7nuSzKNxVJrz3iWny126dCmDBg1i69at9OjRg/fff/+SL2Op4iMjI4Po6Gi9w3QDX19fqlSpctHwRXk1omsCUcVK5rHjHL5nINnxZ6j8/QS8r4nMs2xaWhpvvPEGb7zxBuHh4Xz55Ze0adOm8IJVqoTQXliqRLBViKDSN2MRLxtHOj9DRnRMnmV9fHwYOXIk69evJywsjNtuu41Ro0adeyNYKXVlNIGoYscrshIVZ4/BJCZztMsz+c4pUr9+fdauXUu3bt0YMWIEHTt2JCkpqXCCVaoE0wSiiiWfuldTYcZbZB46ytFuQ8hOSrlk+cDAQL744gs++OADvvvuO1q3bk1MTN53L0qp/GkCUcWW3031Kf/Jq6T9vZOY/q/mO0WuiPDkk08yb948tm7dyk033cSBAwcKKVqlSh5NIKpYC7ijORFvDCL5p1WcGOncnC333nsvK1as4NSpU9xyyy3s27fPzVEqVTJpAlHFXsijHQnp04n4ibOJ/2y+U3WaNGnC0qVLSUhIoFWrVuzfv9+tMSpVEmkCUSVC+KiB+N92E8eHvk/ysjVO1WnYsOG5JNKuXTtOnDjh5iiVKlk0gagSQaxWyk9+Be9akRx7dATp/+53qt4NN9zAwoUL2b9/P/fccw8pKZdujFdK/Y8mEFViWAL9qTjzbcTXm2M9h5N9xrmuujfffDNffvklf/75Jz169ND3RJRykiYQVaLYKpenwievkrHvMDEDXsM4mQw6duzImDFjmDt3Lm+//babo1SqZNAEokocv+Y3ED5yAMk/ruLUe587Xe+pp57iwQcfZPjw4SxZssSNESpVMmgCUSVSSN9OBHa+nVNvTyVpyR9O1RERJk+eTN26denWrRsHDx50c5RKFW+aQFSJJCKUffc5vOtcTezjo8jYG+1UvYCAAObOnUt6ejoPP/wwWfm8nKhUaaYJRJVYFn9fKkx/HawWjj06guzUNKfqXX311YwfP56VK1cyZswYN0epVPGlCUSVaF7VKlJu/Aukb9nFiRETnK7Xs2dPHnjgAV588UU2btzoxgiVKr40gagSL+CO5oQ8/l8Sps0jceFyp+qICJMmTSIiIoJevXqRkZHh5iiVKn40gahSIfzFfvg0vI64p94mY99h5+qEhzNhwgQ2b97Me++95+YIlSp+NIGoUkG8vSg/ZSRYhJg+L2PS0p2qd//999OhQwdeeeUVHXRRqQtoAlGlhle1ipQbN4y0v3dy4tWPna43fvx4rFYrjz/+OKVpCmil8qMJRJUqAXe1JPjRjsRP/obk5WudqlO1alVef/11Fi9ezIIFC9wcoVLFh5Smv6gaN25s1q9f7+kwlIdlp6QRfeujZMcnUvXX6VjDQvKtk5mZSf369UlLS2Pr1q34+PgUQqRKFQ0issEY0/jC7XoHokodi58P5SeOIOtkPHGD33HqsZTNZmPs2LHs2bOHDz/8sBCiVKro0wSiSiWfetcQNuwxkhat5MxXPzpV54477uDOO+/k1VdfJS4uzs0RKlX0aQJRpVaZJ7ri+58GHB/2Phn7jzhVZ8yYMSQlJfHqq6+6OTqlir58E4iIRDmzTaniRqxWyk14EbFaiX1iFCYzM9861113Hb1792bSpEkcOHCgEKJUquhy5g5kTi7bvnV1IEp5gleV8kSMHkzqui2c/mi2U3VeeuklRITXXnvNzdEpVbTlmUBEpJaIPACEiEjHHEsvwLfQIlTKzQI73kpA+1s4+fanpO/M/2XBqlWr0r9/f6ZNm8bu3bsLIUKliqZL3YFcC7QHygD35FgaAn3cHplShUREiBj9DJZAf2KffNOpR1nDhg3D29ubV155xf0BKlVE5ZlAjDELjDGPAO2NMY/kWJ40xjg3Q08+RKSdiOwUkd0iMjSX/SIi4xz7N4tIwxz79ovIPyKySUT05Q51RWxlQyn71tOk/bWd0xPzf5RVoUIF/u///o9Zs2axa9euQohQqaLHmTaQ3SLygohMFpGpZ5crPbGIWIEJwJ1AbaCbiNS+oNidQE3H0heYeMH+1saYBrm94KLU5Qq4rw0Bd9/Cqbenkv7v/nzLDx48GG9vb0aPHu3+4JQqgpxJIAuAEOAXYFGO5Uo1BXYbY/YaY9KBr4AOF5TpAHxu7P4EyohIRRecW6mL2B9lDUYC/Ij9vzfyfZRVvnx5evfuzfTp0zl82LkRfpUqSZxJIP7GmOeNMV8bY+acXVxw7srAoRzr0Y5tzpYxwBIR2SAifV0Qj1LYyoUR8dZTTj/Keu6558jOzmbs2LGFEJ1SRYszCeR7EbnLDeeWXLZdOKbEpco0N8Y0xP6Ya4CItMz1JCJ9RWS9iKzXt4eVMwLva+v0o6zIyEi6devGpEmTOHHiROEEqFQR4UwCGYQ9iaSISIKInBGRBBecOxqommO9CnDh68B5ljHGnP2MBeZhfyR2EWPMZGNMY2NM47Jly7ogbFXSnXuU5e9L3FNvY7KzL1n++eefJykpScfIUqVOvgnEGBNkjLEYY/yMMcGO9WAXnHsdUFNEokTEG+gKLLygzELgYUdvrGZAvDHmqIgEiEgQgIgEALcDW1wQk1KA/VFW+KsDSV23hYTPL/yxPF/dunVp3749EyZMIDU1tZAiVMrznBoLS0RCRaSpiLQ8u1zpiY0xmcBAYDGwHfjaGLNVRPqLSH9HsR+AvcBuYArwhGN7eWCViPwNrAUWGWN+utKYlMop6L/t8GvZiJOvfkzm0Us//nzqqaeIi4vjq6++KqTolPK8fOcDEZHHsD/GqgJsApoBq40xbdwenYvpfCDqcmXsjebQLT3xb9uMCp+9nmc5Ywz16tXDZrPx119/IZJb851SxdOVzAcyCGgCHDDGtAZuALQ1WpUKXldVIfS53iQt+pXERb/mWU5EePLJJ9m0aRO//fZbIUaolOc4k0BSjTGpACLiY4zZgX2YE6VKhTKP/xfvOldzfOh7ZCUk5lmue/fuhIWF8cEHHxRidEp5jjMJJFpEygDzgZ9FZAEX95ZSqsQSLxtl3xtCVuxJTr42Kc9y/v7+9O3bl/nz57N///7CC1ApD3GmF9b9xpjTxphXgJeAT7n4jXGlSjTfG64jpM8DJEybT8qazXmWGzBgACLCxIkXjrqjVMnjzIRSX5z92hiz0hizELjisbCUKm7Chj6GrUp54p55B5OWnmuZKlWqcM899zBt2jTS03Mvo1RJ4cwjrDo5VxyDIDZyTzhKFV2WQH8iRj9Dxs79nP4o7+66/fr1Iy4ujnnz5hVidEoVvktNKDVMRM4A9RxvoCc41mOxD7CoVKkTcNtNBLS/hVNjp5NxIPemwNtvv53IyEgmTcq7vUSpkuBS84G8aYwJAt5xvIF+9i30cGPMsEKMUakiJeL1J8Fq5fiw98ntPSqLxUKfPn1Yvnw5//77rwciVKpwODuYYgCAiPQQkbEiUt3NcSlVZNkqlSPs+d4k/7yapB9yf+ejd+/e2Gw2Jk+eXMjRKVV4nEkgE4FkEakPDAEOAJ+7NSqliriQPp3wrlOD4y98QHZi8kX7K1SoQIcOHfjss890fCxVYjmTQDKN/T69A/CBMeYDIMi9YSlVtInNRtnRz5B1JJaT707LtUy/fv04ceIEc+fOLeTolCocziSQMyIyDOgBLHL0wvJyb1hKFX2+Ta8nqPvdxH/8DWnb9ly0v23btkRFRTF1qvZ6VyWTMwnkv0Aa8Kgx5hj2GQHfcWtUShUT4SMexxISyPHnxlw0b4jFYqFnz54sW7aMAwcOeChCpdzHmTfRjxljxhpjfnOsHzTGaBuIUoA1LITwEY+TuvYfznz140X7e/bsiTGGzz/X/zKq5HHmTfSOIrJLROJdPCOhUiVCULc78W16PSdGTiTrZPx5+yIjI2ndujWfffZZrl1+lSrOnHmENRq41xgT4uIZCZUqEcRiIeKdZ8iOT+TEqI8v2t+rVy/27t3LqlWrPBCdUu7jTAKJMcZsd3skShVjPrVrENK/M2dmfE/q2n/O2/fAAw8QGBjItGm599ZSqrhyJoGsF5HZItLN8Tiro4h0dHtkShUzYc8+grVSOeKGjMFkZp7bHhAQQJcuXfj6669JTMx7PhGlihtnEkgwkAzcDtzjWNq7MyiliiNLoD8Rrz9J+tY9xH8y57x9jzzyCElJScyZMyeP2koVP/nOiV6S6Jzoyt2MMRx7cAgpq/+m2h8zsFUqd257zZo1qVq1KsuXL/dwlEpdngLPiS4iVURknojEikiMiMwRkSruCVOp4k1EiHjzacjK4vhLH563vVevXqxYsYJ9+/Z5MEKlXMeZR1jTgIVAJewvEX7n2KaUyoVXZCVCn3qYpIXLSV625tz2hx56CIAvv/zSU6Ep5VLOJJCyxphpxphMx/IZUNbNcSlVrJUZ2A2vq6pwfOj7ZKemAVC9enVuvvlmZsyYoe+EqBLBmQRy3DGMu9Wx9ABOuDswpYoz8fG2z164L5rT4/93x9G9e3d27NjBxo0bPRidUq7hTALpDXQBjgFHgU6ObUqpS/C/pTGB97fl9AczyNgbDUDnzp3x8vJi5syZHo5OqSvnzFhYB40x9xpjyhpjyhlj7jPG6MhwSjkh/NWB4GUjbuh7GGMICwvjrrvuYtasWWRlZXk6PKWuiDO9sKaLSJkc66EiouNTK+UEW4UIwob1IWX5WpK+WwFAjx49OHr0qHbnVcWeM4+w6hljTp9dMcacAm5wW0RKlTAhve/Du25Njr84nuzEZNq3b09wcDAzZszwdGhKXRFnEohFRELProhIGGBzX0hKlSxis1H2nWfIOnack+9Mw9fXl06dOjF37lySky+eDlep4sKZBDIG+ENERonIq8Af2EfoVUo5ybdxHYIfuof4Sd+QtnU33bt358yZM3z33XeeDk2pAnOmEf1z4AEgBogDOhpjvnB3YEqVNGHD+2IpY5+9sOXNN1O5cmXtjaWKNWfuQDDGbDPGfGiMGW+M2ebuoJQqic7NXrhuC8nfLKFbt278+OOPHD9+3NOhKVUgTiUQpZRrBHW9E98b63Fi5ER63HMfmZmZfPPNN54OS6kC0QSiVCESi4WI0YPJjk+k4vzfqVOnjj7GUsXWJROIY+iSXworGKVKA5/aNQjp15kzX3zHk23u5vfff9cRelWxdMkEYozJApJFJMQdJxeRdiKyU0R2i8jQXPaLiIxz7N8sIg2dratUURb23CNYK5al7YZorOgIvap4cuYRVirwj4h86vhlPk5Exl3piUXECkwA7gRqA91EpPYFxe4EajqWvsDEy6irVJF1dvZC/j3Iy3V0hF5VPDmTQBYBLwG/AhtyLFeqKbDbGLPXGJMOfAV0uKBMB+BzY/cnUEZEKjpZV6kiLaD9Lfi1uZEup22c/HevjtCrip183yg3xkx307krA4dyrEcDNzpRprKTdQEQkb7Y716oVq3alUWslAuJCGXfepqUFg/xYlANZsyYQcOGDfOvqFQRkecdiIh87fj8x9H+cN7ignNLLtsuvIfPq4wzde0bjZlsjGlsjGlctqzOg6WKFq+oyoQ+/RB3eYWza+Z8HaFXFSuXugMZ5Phs76ZzRwNVc6xXAY44WcbbibpKFQtlBj7IsalzGXQshWWLl3DbXXd6OiSlnJLnHYgx5qjj8wD2hvTrHUuKi+YDWQfUFJEoEfEGumKfez2nhcDDjt5YzYB4R1zO1FWqWLD4+lDpveeJsvpx8PWJng5HKac5Mx9IF2At0Bn7zIRrRKTTlZ7YGJMJDAQWA9uBr40xW0Wkv4j0dxT7AdgL7AamAE9cqu6VxqSUp5S5owXbKgVx084TJGzd5elwlHKK5Nd1UET+Bm4zxsQ61ssCvxhj6hdCfC7VuHFjs379ek+HoVSufp2zgKC+b8G11Wnw+yxEcmvqU6rwicgGY0zjC7c7NR/I2eThcMLJekqpy9Di/nuY5H2a4F2HOTNzkafDUSpfziSCn0RksYj0EpFe2N8L+cG9YSlV+lgsFoJ7tGdtZgJxL39I5jEdpVcVbc7MB/IcMAmoB9QHJhtjnnd3YEqVRj0eeogXEv4lKzmF4y984OlwlLokZxrRnwbWGmMGG2OeNsbMK4S4lCqV6tWrR2Dtq5kTkknSdytI+uFXT4ekVJ6ceYQVDCwWkd9EZICIlHd3UEqVZt27d2fEztVQsxpxQ8aSFX/G0yEplStnHmGNNMbUAQYAlYCVOsS7Uu7z4IMPkonhp4YVyYo7xclRkzwdklK5upzeVLHAMey9sMq5JxylVLVq1WjZsiXjf/mekH6dSZi+gJQ/Nnk6LKUu4kwbyOMisgJYCkQAfYwx9dwdmFKlWY8ePdi5cyf7b78BW/WKxA0eTXZKmqfDUuo8ztyBVAOeMsbUMca8bIzZ5u6glCrtOnXqhLe3NzPnfEvZMUPI2HOIk29/4umwlDpPflPaWoB7jDGbCiccpRRAaGgod999N7NmzcK7eQOCe3Yg/qPZpK79x9OhKXVOflPaZgN/i4hOpKFUIevevTsxMTEsW7aM8FeewFa1ArH/9wbZyameDk0pwLlHWBWBrSKyVEQWnl3cHZhSpd3dd99NSEgIM2bMwBLoT7kPhpKxN5qTr0/2dGhKAU7MSAiMdHsUSqmL+Pr60qlTJ2bPnk1SUhIBLRoS/GhH4id/Q8BdN+PX/AZPh6hKOWfeA1kJ7Ae8HF+vA/5yc1xKKey9sRITE5k3zz4ARPhL/fGKqkLsoDfJTkz2cHSqtHOmG28f4Fvs42GBfT7y+W6MSSnl0LJlS6Kiopg2bRoAlgA/yo4bRubBY5x4VSefUp7lTBvIAKA5kABgjNmFvkioVKGwWCw88sgjLFu2jH379gHg16weIf27kDBtPsnL13o4QlWaOZNA0owx6WdXRMQGXHoWKqWUy/Ts2RMR4bPPPju3LWxYH7yujST2/94g68Rpj8WmSjdnEshKEXkB8BOR24BvgO/cG5ZS6qxq1apx2223MW3aNLKysgCw+PlQ/uOXyTqVQOzTb5PfzKJKuYMzCWQoEAf8A/TDPpnUi+4MSil1vt69e3Po0CGWLVt2bptP3asJf6kfyT+uIuFz7VmvCp8zvbCyjTFTjDGdgb7AGqN/7ihVqDp06EBoaChTp049b3tI3874tWrCiZfGk77rgIeiU6WVM72wVohIsIiEAZuAaSIy1u2RKaXO8fX1pXv37sybN49Tp06d2y4WC+XGv4D4+RLTbyQmLf0SR1HKtZx5hBVijEkAOgLTjDGNgFvdG5ZS6kK9e/cmLS2NWbNmnbfdViGCch8MJf2fXZx8SwdcVIXHmQRiE5GKQBfgezfHo5TKww033ECDBg349NNPL9oX0K4Fwb06cPrDWdq1VxUaZxLIq8BiYI8xZp2IXAXscm9YSqncPPbYY/z111+sW7fuon3hIwfifd1VxDz+KplH4zwQnSptnGlE/8YYU88Y87hjfa8x5gH3h6aUutBDDz1EQEAAEyde/Ba6xd+X8lNHYVLTienzCiYj0wMRqtLEmUb0axwj8W5xrNcTEe3Gq5QHBAcH06NHD2bNmsXJkycv2u99dTXKjh1C6prNnHxzigciVKWJM4+wpgDDgAwAY8xmoKs7g1JK5e3xxx8nNTWV6dOn57o/qOOtBD9yH6fHf0nS4t8LOTpVmjiTQPyNMRe2yum9sVIeUr9+fW666SYmTpxIdnZ2rmXCXx2Id71riB34OhkHjxZyhKq0cCaBHBeRGjjGvxKRToD+RCrlQU888QS7du067830nCy+PlT4dBRkG2IeHUF2SlohR6hKA2dH450E1BKRw8BTQH93BqWUurROnToRHh6ea2P6WV6RlSg3YThpm3YQ9+w7Ol6WcjlnemHtNcbcCpQFagGtgBZujkspdQm+vr48+uijLFiwgAMH8h7CJKBdC0Kf703i14uJn/R1IUaoSoM8E4hj+JJhIvKhYxTeZKAnsBv7S4VKKQ8aOHAgAOPGjbtkudDBPQm4uyUnXv6I5JXrCyM0VUpc6g7kC+Ba7KPw9gGWAJ2B+4wxHQohNqXUJVStWpUuXbowZcoUEhIS8iwnFgvlPhyO1zXVienzMhn7jxRilKoku1QCucoY08sYMwnoBjQG2htjNhVKZEqpfD3zzDOcOXOGTz659BhYlkB/Kn7+JhjDsZ7DdD515RKXSiAZZ78wxmQB+4wxZ1xxUhEJE5GfRWSX4zM0j3LtRGSniOwWkaE5tr8iIodFZJNjucsVcSlV3DRq1IhbbrmFDz74gMzMS/eu94qqTPkpI0nfeYBjj47A5FNeqfxcKoHUF5EEx3IGqHf2axHJ+37ZOUOBpcaYmsBSx/p5RMQKTADuBGoD3USkdo4i7xljGjiWH64wHqWKrcGDB3Pw4EHmzJmTb1n/Vk0o+85gUpat4fjQ97VnlroieSYQY4zVGBPsWIKMMbYcXwdf4Xk7AGdfo50O3JdLmabAbkcvsHTgK0c9pVQO7du3p2bNmowZM8aphBD80L2UebI7CdMXcHrCrHzLK5UXZ94DcYfyxpijAI7PcrmUqQwcyrEe7dh21kAR2SwiU/N6BAYgIn1FZL2IrI+L0xFKVcljsVh45plnWLduHUuXLnWqTtjwvgTe14aTIyeSuGC5myNUJZXbEoiI/CIiW3JZnL2LkFy2nf3zaiJQA2iA/a34MXkdxBgz2RjT2BjTuGzZspdzCUoVG7169aJy5cqMHDnSqbsQsVgoO/4FfJvUJXbAa6Ss/rsQolQljdsSiDHmVmNM3VyWBUCMY5IqHJ+xuRwiGqiaY70KcMRx7BhjTJYxJhv7YI9N3XUdShUHPj4+DB06lFWrVrFy5Uqn6lh8fajwxZvYqpTnWPfnSdv8r5ujVCWNpx5hLcT+UiKOzwW5lFkH1BSRKBHxxj4C8EI4l3TOuh/Y4sZYlSoWHnvsMSpWrMioUaOcrmMNL0PFb9/DEhzAkf8+Q/qeg26MUJU0nkogbwG3icgu4DbHOiJSSUR+ADDGZAIDsc+GuB342hiz1VF/tIj8IyKbgdbA04V9AUoVNb6+vgwZMoRly5axatUqp+t5VSlPxW/fA2M42mkwmUdyeyCg1MWkNHXja9y4sVm/XodyUCVXcnIyUVFR1K1bl19++QWR3JoSc5f2906O3D8Ia4UIKi8cjzUiz74pqpQRkQ3GmMYXbvfUHYhSyg38/f154YUXWLZsGUuWLLmsuj71r6XCjLfIPHSUI50Gk3XitHuCVCWGJhClSpj+/fsTFRXF888/T1ZW1mXV9ftPAyp8/iYZew5y5IGnNImoS9IEolQJ4+PjwxtvvMHff//NzJkzL7u+f+umVPjiLTL2HOJIx0FkHT/lhihVSaBtIEqVQNnZ2TRt2pTY2Fj+/fdffH19L/sYySvXc6zH83hFVaHinPexldU2kdJK20CUKkUsFgujR4/m0KFDvP/++wU6hv8tjakw820y9h/myH3/p72z1EU0gShVQrVp04YOHTowatQoDh06lH+FXPi3bEzFWe+QeSSOw3c/QfpufU9E/Y8mEKVKsPfft4+4+/TTBX9Vyq/5DVSeP47s1DQOt3+C1I3bXRihKs40gShVgkVGRjJ8+HDmzJnD4sWLC3wcn/rXUvn7j7AE+HPkvkE6Na4CtBFdqRIvLS2N66+/nuzsbDZv3oy/v3+Bj5V57DhHuzxD+u6DlB3zHMHddC630kAb0ZUqpXx8fJg0aRJ79uzhhRdeuKJj2SpEUGnhh/g1q0fck29yYtTHmOxsF0WqihtNIEqVAq1bt2bAgAF88MEHTo/WmxdrmSAqzh5D8MP3cnrcTGJ6v0R2UoqLIlXFiT7CUqqUSEpKon79+uceZQUGBl7R8YwxxE/6hhMvT8C7Tg0qfvEmtsrlXRStKkr0EZZSpVxAQACfffYZ+/fvZ8CAAVc8H7qIUKZ/FyrMeIuMfYc51OZRkpevdVG0qjjQBKJUKdKiRQtGjBjB559/zrRp01xyzIDbbqLKL1OwlQ/n6H+f5eQ707RdpJTQBKJUKfPSSy/Rpk0bBgwYwD///OOSY3rXqEblHz8msPPtnBo9laNdn9OBGEsBTSBKlTJWq5WZM2cSEhJCx44dOXHihEuOawnwo9yHw4l491lSft/IoVaPkLxinUuOrYomTSBKlUIVKlRgzpw5HDx4kAceeID09HSXHFdECOnZgSo/fowlOICjnQdz/EX7W+yq5NEEolQp1bx5c6ZNm8bKlSvp27fvFTeq5+RT7xqq/PIpwY92JH7SN0Tf1oe0LbtddnxVNGgCUaoUe/DBB3nllVeYPn06w4YNc2kSsfj5UPatp6n41btkn4wn+vY+nHzrE70bKUE0gShVyo0YMYL+/fvz9ttvM2rUKJcf37/tjVT9dTqB97fl1JjpRLfuTcrqv11+HlX4NIEoVcqJCBMmTKBXr168/PLLvPHGGy69EwGwhpeh/IQXqTj7XUxaOkfuHUjcs++SdSrBpedRhUsTiFIKi8XCJ598Qvfu3Rk+fDjPPvss2W54l8O/zY1U/e1zQh7/LwlffMfBZg8SP20eJjPT5edS7qcJRCkF2Lv3fv755wwcOJCxY8fSs2dP0tJc315hCfAj4tWBVFn2Kd7XXcXxIWOJvvUxUn7f6PJzKffSBKKUOsdisTBu3DhGjRrFjBkzaNWqFUeOHHHLuXzqXE2leR9QfuooshOSOHLfkxzr+QLpO/a55XzK9TSBKKXOIyK8+OKLfPPNN/zzzz80atToikfwvdS5Au9pRdXfZxA29DGSf93AoZY9iRnwGhn73ZO4lOtoAlFK5apTp06sWbOGoKAgWrduzXPPPUdqaqpbzmXx8yH0mZ5U3/A1ZQZ0JWnhcg7e9CBxz75LxqFjbjmnunKaQJRSeapTpw5//fUX/fr1491336Vx48b88ccfbjufNSyE8JefoNq62QQ/dC8JXy7iYNOuxAx4XR9tFUGaQJRSlxQYGMjEiRP54YcfOH36NM2bN+fhhx/m6NGjbjunrUIEZUcPpvq6rwh5tCNJ36/g0M0Pc/ShYaSs2ezybsaqYHRCKaWU0xITE3nzzTd599138fLyYuDAgTzzzDOULVvWrefNOnGa+E/nEv/JHLJPJeBd7xpCencksOOtWPx83HpulfeEUppAlFKXbc+ePYwYMYJZs2bh7+9Pv379GDBgAFdddZVbz5udmMyZb5eQMHUe6dv3YgkNJrj73QQ/3AGvqMpuPXdppgkETSBKudqOHTsYNWoUs2fPJjs7m7vuuot+/fpxxx134O3t7bbzGmNI/WMT8Z/OJemH3yArC99m9Qn6bzsCO7TGEhTgtnOXRppA0ASilLscPnyYyZMnM2nSJGJiYggNDeWBBx6ga9eu3HzzzW5NJplH4zgz+yfOzP6JjN0HET8fAu5uSVCXdvi1aIh42dx27tJCEwiaQJRyt/T0dH7++WdmzZrF/PnzSUpKIjAwkDZt2tCuXTvatm1LzZo1ERGXn9sYQ9pf2zjz1Y8kzltKdnwiltBgAu68mYB7WuHfshHi7eXy85YGmkDQBKJUYUpOTmbJkiUsXryYxYsXs2+fvRtuWFgYzZo1o1mzZjRq1Ig6depQrVo1lyaV7NQ0UpavJfG7FST/9DvZZ5KwBAfi364FAXf8B79WTbAGB7rsfCVdkUogIhIGzAYigf1AF2PMqVzKTQXaA7HGmLqXW/9CmkCU8gxjDLt37+bXX39l9erV/Pnnn2zbtu1cd9zAwEBq165NnTp1qFGjBtWrVycyMpLq1atTqVIlrFZrwc+dlk7yyvUkfbeCpJ9WkX36DFit+Dapi3/bG/Fv2wzvule75a6opChqCWQ0cNIY85aIDAVCjTHP51KuJZAIfH5BAnGq/oU0gShVdMTHx7N582a2bt16btm2bRsxMTHnlbPZbJQvX55y5cpdtISGhhIcHExwcDAhISHnvg4ODiYoKOiixGMyM0ldv43kpX+SvPRP0v/ZBYC1fDh+zW/A9z8N8Gt+A141qmpCyaGoJZCdQCtjzFERqQisMMZcm0fZSOD7CxKI0/Vz0gSiVNGXkpLCwYMH2b9/PwcOHGD//v0cO3aM2NhYYmNjiYuLIyYmhpSUlHyP5eXlha+vb55LOasP9ZOF685kUyM+g+B0+xD2iT5WDob7E102gJhyQcSHByJeNmw2+2K1WrHZbFgsFkQkzyW//c6WuTCZ5Zbc8ivTtm1bKlcuWFfnopZAThtjyuRYP2WMCc2jbCQXJ5DLqd8X6AtQrVq1RgcOHHDJNSilPCsxMZHTp0+TkJCQ55KamurUkpmZSWZGJuUzoE6alboZXtTP8qEc9h5cqSabHSaZf7KS2JRxho1ppzmY5Z5xwdzlxx9/pF27dgWqm1cCcVv/NhH5BaiQy67h7jpnbowxk4HJYL8DKcxzK6XcJzAwkMBA9zWEG2PIPHiUtL+2k7pxO2U2bOOGzTt5KDUd/MESEohXravwui4Kr+uuwlYrCtu1kYi/L8aYc0t2dvZ567kt+ZW5MK7cYs2vTIUKuf06vjJuSyDGmFvz2iciMSJSMccjqNjLPPyV1ldKqUsSEbyqV8KreiUC728LgMnIJH3HPlL/2kb6ll2kb91D0rc/YxKTz9WzRVbGu1YUXjWq4F2jKl5XVcXr6mpYy4WVuHYVT71hsxDoCbzl+FxQyPWVUuqyiZcNn+tr4nN9zXPbjDFkHjpG+rY9pG/dQ9q2PWTsOkDK8rWYtPT/1Q30tyeUGlWxVauIV9UK2KqUx1a1ArYqFYrlmF6eagMJB74GqgEHgc7GmJMiUgn4xBhzl6PcLKAVEAHEAC8bYz7Nq35+59VGdKVUYTHZ2WQejiVj90Ey9hwiY88h0ncfJGNvNJmHYyEr67zy1rKh2Ko4kkrlcljLh2MrH57jMwJLSKBH7mKKVCO6p2gCUUoVBSYzk8xjJ8g8dIzM6GNkHooh49BRMqNj7NuOxGJSLp6PXny9sZZzJJVy4VjDQ7CEhWANC8YSGoI17H+LJSwES3AAYrnyWTsKvRFdKaVU7sRmw6tKebyqlAfqX7TfGINJTCYz5gRZMSccn8fPrWfFnCB99wGy18aTdTLhoruZc6xWLKFBWENDKPvus/j9p4FLr0MTiFJKFTEiggQF4B0UAFdXu2RZk51N9pkksk8mkHXyNFknE8g+GU/Wqfj/fX0yHkuI63usaQJRSqliTCwWrCFBWEOCCn1OFJ3SVimlVIFoAlFKKVUgmkCUUkoViCYQpZRSBaIJRCmlVIFoAlFKKVUgmkCUUkoViCYQpZRSBVKqxsISkTigoDNKRQDHXRhOcaDXXDroNZcOV3LN1Y0xZS/cWKoSyJUQkfW5DSZWkuk1lw56zaWDO65ZH2EppZQqEE0gSimlCkQTiPMmezoAD9BrLh30mksHl1+ztoEopZQqEL0DUUopVSCaQJRSShWIJhAniEg7EdkpIrtFZKin43E1EakqIstFZLuIbBWRQY7tYSLys4jscnyGejpWVxMRq4hsFJHvHesl+ppFpIyIfCsiOxz/3jeVgmt+2vFzvUVEZomIb0m7ZhGZKiKxIrIlx7Y8r1FEhjl+n+0UkTsKel5NIPkQESswAbgTqA10E5Hano3K5TKBZ4wx1wHNgAGOaxwKLDXG1ASWOtZLmkHA9hzrJf2aPwB+MsbUwj4Z93ZK8DWLSGXgSaCxMaYuYAW6UvKu+TOg3QXbcr1Gx//trkAdR52PHL/nLpsmkPw1BXYbY/YaY9KBr4AOHo7JpYwxR40xfzm+PoP9l0pl7Nc53VFsOnCfRwJ0ExGpAtwNfJJjc4m9ZhEJBloCnwIYY9KNMacpwdfsYAP8RMQG+ANHKGHXbIz5FTh5wea8rrED8JUxJs0Ysw/Yjf333GXTBJK/ysChHOvRjm0lkohEAjcAa4DyxpijYE8yQDkPhuYO7wNDgOwc20ryNV8FxAHTHI/tPhGRAErwNRtjDgPvAgeBo0C8MWYJJfiac8jrGl32O00TSP4kl20lsu+ziAQCc4CnjDEJno7HnUSkPRBrjNng6VgKkQ1oCEw0xtwAJFH8H91ckuO5fwcgCqgEBIhID89G5XEu+52mCSR/0UDVHOtVsN8Clygi4oU9ecw0xsx1bI4RkYqO/RWBWE/F5wbNgXtFZD/2x5JtRGQGJfuao4FoY8wax/q32BNKSb7mW4F9xpg4Y0wGMBf4DyX7ms/K6xpd9jtNE0j+1gE1RSRKRLyxNz4t9HBMLiUigv25+HZjzNgcuxYCPR1f9wQWFHZs7mKMGWaMqWKMicT+b7rMGNODkn3Nx4BDInKtY1NbYBsl+JqxP7pqJiL+jp/zttjb+EryNZ+V1zUuBLqKiI+IRAE1gbUFOYG+ie4EEbkL+/NyKzDVGPO6ZyNyLRFpAfwG/MP/2gNewN4O8jVQDft/xM7GmAsb6oo9EWkFPGuMaS8i4ZTgaxaRBtg7DXgDe4FHsP8hWZKveSTwX+y9DTcCjwGBlKBrFpFZQCvsQ7bHAC8D88njGkVkONAb+/fkKWPMjwU6ryYQpZRSBaGPsJRSShWIJhCllFIFoglEKaVUgWgCUUopVSCaQJRSShWIJhClrpCIDHeM9rpZRDaJyI0i8pSI+Hs6NqXcSbvxKnUFROQmYCzQyhiTJiIR2N+x+AP7CLDHL+NYVmNMlptCVcrl9A5EqStTEThujEkDcCSMTtjHXVouIssBRKSbiPzjmJPi7bOVRSRRRF4VkTXATSIyQkTWOcpNdrw9jYg0cdzhrBaRd87O++CYz+QdR53NItKvkK9flWKaQJS6MkuAqiLyr4h8JCK3GGPGYR9bqLUxprWIVALeBtoADYAmInKfo34AsMUYc6MxZhXwoTGmiWPuCj+gvaPcNKC/MeYmIOddyqPYR5htAjQB+jiGp1DK7TSBKHUFjDGJQCOgL/ah0meLSK8LijUBVjgG9MsEZmKflwPsyWBOjrKtRWSNiPyDPeHUEZEyQJAx5g9HmS9zlL8deFhENmEfeiYc+9hGSrmdzdMBKFXcOdotVgArHL/4e15QJLfhs89KPdvuISK+wEfY204OicgrgG8+9QX4P2PM4gKGr1SB6R2IUldARK4VkZx/8TcADgBngCDHtjXALSIS4Zg6tBuwMpfD+To+jzvmZukEYIw5BZwRkWaO/V1z1FkMPO4Yjh8RucYxSZRSbqd3IEpdmUBgvOMxUyb26UH7Yk8SP4rIUUc7yDBgOfY7hh+MMRcNH26MOS0iU7CPirwf+1QCZz0KTBGRJOx3O/GO7Z8AkcBfjgb3OIr59Kyq+NBuvEoVAyIS6GhvQUSGAhWNMYM8HJYq5fQORKni4W7HXYwN+yOyXp4NRym9A1FKKVVA2oiulFKqQDSBKKWUKhBNIEoppQpEE4hSSqkC0QSilFKqQP4f4XWPMJjtNBAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dtype = torch.float32\n",
    "v = torch.tensor(np.arange(0.0, 100.0, step=0.1), dtype=dtype)\n",
    "model_k = model.K\n",
    "yhat = np.hstack([model_k(torch.tensor([vv])).detach().numpy().flatten() for vv in v])\n",
    "plt.plot(v, kx(v), color='black', label='target')\n",
    "plt.plot(v, yhat, color='crimson', label='neural net')\n",
    "plt.legend()\n",
    "plt.xlabel('Storage')\n",
    "plt.ylabel('Reservoir constant')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e388eae4-7706-4a15-a872-15109649bdd2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "all",
   "language": "python",
   "name": "all"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}