Orthonormal init and LSUV.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# All you need is a good init and Orthonormal Initialization\n",
    "#### A simple implementation based on two papers:\n",
    "- Dmytro Mishkin, Jiri Matas (https://arxiv.org/abs/1511.06422) and \n",
    "- Andrew Saxe, James McClelland and Surya Ganguli (https://arxiv.org/abs/1312.6120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from fastai import datasets\n",
    "import gzip, pickle\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Layer Sequential Unit-Variance Initialization\n",
    "The main idea in the 'All you need is a good init' paper is an algorithm the authors call 'Layer Sequential Unit-Variance Initialization' or LSUV. Instead of trying to compute a formula for how to scale weights in terms of the dimensions of particular layers, the algorithm instead takes an empirical approach. You feed a batch of input data through the network layer by layer and adjust the initial weights of each layer until the scale of the outputs is sufficiently close to 1. \n",
    "\n",
    "```\n",
    "for each layer L do:\n",
    "    Initialize weights of L (WL) with some reasonable starting point \n",
    "    (see the discussion of the Saxe et al. paper below)\n",
    "    do:\n",
    "        increment iteration counter Ti++\n",
    "        do the forward pass with a mini-batch\n",
    "        calculate the variance of the output of the layer - Var(L(xb))\n",
    "        Scale the weights WL by sqrt(Var(L(xb))) i.e. WL = WL / sqrt(Var(L(xb)))\n",
    "    while \n",
    "        |Var(L(xb)) − 1.0| ≥ some tolerance and the Ti < max iterations```\n",
    "\n",
    "The function below is my implementation of the above pseudo-code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LSUV(model, tol_var=0.01, t_max=100):\n",
    "    o = x\n",
    "    for m in model:\n",
    "        if hasattr(m,'weight'):\n",
    "            t = 0\n",
    "            u = m(o)\n",
    "            while (u.var() - 1).abs() > tol_var and t < t_max:\n",
    "                t += 1\n",
    "                m.weight.data = m.weight.data/u.std()\n",
    "                u = m(o)\n",
    "            o = u\n",
    "        else:\n",
    "            o = m(o)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Orthogonal Initialization for Convolutional Layers\n",
    "\n",
    "The idea of using an orthogonal initialization was introduced by Andrew Saxe et al. in their paper *Exact solutions to the nonlinear dynamics of learning in deep linear neural networks* (https://arxiv.org/abs/1312.6120). The idea here is to choose the initial values of the weight matrices such that the dot product of any distinct rows is zero while each row has norm 1. This can be written in terms of the *Kronecker Delta* $\\delta_{ij}$:\n",
    "\n",
    "$$\\delta_{ij} = \\begin{cases}\n",
    "0 &\\text{if } i \\neq j,   \\\\\n",
    "1 &\\text{if } i=j.   \\end{cases}$$\n",
    "\n",
    "So we want for each pair of rows $\\mathbf w_{i}$, $ \\mathbf w_{j}$ of the weight matrix:\n",
    "\n",
    "$${\\mathbf w_{i}}^\\mathrm{T}\\mathbf w_{j} = \\delta_{ij}$$\n",
    "\n",
    "There are a couple of properties of rows that have this property that are useful in the context of neural networks:\n",
    "- They preserve scale, so for some vector $\\mathbf x$ we have $\\|\\mathbf W \\mathbf x\\|=\\|\\mathbf x\\|$. \n",
    "\n",
    "This will help with maintaining the scale of outputs through affine functions in the network\n",
    "\n",
    "- The rows are orthogonal to each other\n",
    "\n",
    "The intuitive benefit of this property is that different rows of the weight matrix will learn different features of the inputs.\n",
    "\n",
    "### Convolutional layers\n",
    "\n",
    "The idea of the initialization with the weight matrices of convolutional networks is similar but needs some tweaking. Each $k \\times k$ kernel $\\mathbf W$ is multiplied elementwise with a $k \\times k$ portion of our input $\\mathbf X$ and then summed up. If we flatten out these $k \\times k$ matrices into length $k^2$ vectors we can think of this part of the covolution as the dot product of these two length $k^2$ vectors.\n",
    "\n",
    "We would like the different kernels for the output channels of each kernel to learn different things. Again intuitively we want to choose the $k^2$ vector representations of kernels from each channel to be orthogonal to each other.\n",
    "\n",
    "One way to achieve this is to start with a matrix with random weights and use singular value decomposition (https://en.wikipedia.org/wiki/Singular_value_decomposition) to extract an matrix with orthogonal rows of the required shape. For our purposes what is important about singular value decomposition is that it allows you to express a matrix as a product of an orthonormal matrix $\\mathbf{M}$, a diagonal matrix $\\boldsymbol{\\Sigma}$ and another orthonormal matrix $\\mathbf{V}^{\\mathrm{T}$.\n",
    "\n",
    "$$\\mathbf{M} = \\mathbf{U} \\boldsymbol{\\Sigma} \\mathbf{V}^{\\mathrm{T}}$$ \n",
    "\n",
    "We can take the rows of $\\mathbf{V}^{\\mathrm{T}}$, reshape them to be $k \\times k$ and use them as initialisations for our kernels.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class OrthInitConv2D(torch.nn.Conv2d):\n",
    "    def __init__(self, in_channels, out_channels, kernel_size, stride=1,\n",
    "                 padding=0, dilation=1, groups=1, bias=True):\n",
    "        super().__init__(in_channels, out_channels, kernel_size, stride,\n",
    "                 padding, dilation, groups, bias)\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        with torch.no_grad():\n",
    "            self.weight.normal_(0,1)\n",
    "            self.bias.zero_()\n",
    "        W = self.weight.data.view([self.weight.shape[0],-1])\n",
    "        _, _, Vt = torch.svd(W)\n",
    "        self.weight.data = torch.Tensor(Vt).view(self.weight.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple example using this initialisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get MNIST data and normalize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "MNIST_URL='http://deeplearning.net/data/mnist/mnist.pkl'\n",
    "\n",
    "def get_data():\n",
    "    path = datasets.download_data(MNIST_URL, ext='.gz')\n",
    "    with gzip.open(path, 'rb') as f:\n",
    "        ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding='latin-1')\n",
    "    return map(torch.tensor, (x_train,y_train,x_valid,y_valid))\n",
    "\n",
    "def normalize(x, m, s): return (x-m)/s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train,y_train,x_valid,y_valid = get_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_mean,train_std = x_train.mean(),x_train.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = normalize(x_train, train_mean, train_std)\n",
    "x_valid = normalize(x_valid, train_mean, train_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get a batch of 1000 inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = x_train[:1000].view([-1,1,28,28])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a simple CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Lambda(torch.nn.Module):\n",
    "    def __init__(self, func):\n",
    "        super().__init__()\n",
    "        self.func = func\n",
    "\n",
    "    def forward(self, x): return self.func(x)\n",
    "\n",
    "def flatten(x):      return x.view(x.shape[0], -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model(convtype=torch.nn.Conv2d, extra_depth=1):\n",
    "    model = torch.nn.Sequential(\n",
    "                convtype(1,8,5,stride=2,padding=2),\n",
    "                convtype(8,16,3,stride=2,padding=1),\n",
    "                convtype(16,32,3,stride=2,padding=1),\n",
    "                *[convtype(32,32,3,stride=2,padding=1) for i in range(extra_depth)]\n",
    "            )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare variance from PyTorch `nn.Conv2d` init and `LSUV` init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Standard PyTorch `nn.Conv2d` init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "shallow_pytorch_stds    = [get_model()(x).std().item() for i in range(100)]\n",
    "deep_pytorch_stds       = [get_model(extra_depth=30)(x).std().item() for i in range(100)]\n",
    "shallow_orthnormal_stds = [LSUV(get_model(convtype=OrthInitConv2D))(x).std().item() for i in range(100)]\n",
    "deep_orthnormal_stds    = [LSUV(get_model(convtype=OrthInitConv2D, extra_depth=30))(x).std().item() for i in range(100)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABIEAAANrCAYAAADRToirAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xm8bXVdN/DPV69oigrExRDUi0Uq1ePQFSlLSSynFBxTU8HwRYOmlZo4PKmlT9hT2aBPRU444TyQ5IAoTil1MQcQDUSUKwjXASU1Ff09f6x1YHPY55595r3Per9fr/06e6/x9/utvff6nc8adrXWAgAAAMDmdp2NLgAAAAAAa08IBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBGLNVdUxVfXhZc57eFXtHHl9YVXdc/VKNxuq6oyqetxGl2O1VdW2qmpVtWWjy7KYqnpFVT1vFZf3nKp69Wotb4nrfkZVvWQF859TVYevYpEAmMBQ+1QrqfdGW+3+w0aZ//6ZZqvdb97IbVhV/1hV/3sF8/93Vd16NcvE7BMCsSqq6peq6t+q6ptV9fWq+khV3Xmjy7VU/T/mP+i/MC/v6/QLE8w3NTvGviytql48b/iHq+qYCZfRquqn1qSAK9CHF1/ot8/Oqnr9yLhNE5T12/BHfT3n6vqG1fpMtdb+T2ttorYa1/Fprf1Ma+2M1SgLANe0WfpUSVJVh1TVKX1drqiq91fVLy4yz8wcIFqpPtxqVfXUecN3TnKwZVrbqqr2qKq/6uvx333f7YUj42cmgFxMvw1/ONJn+0JVvbyqfno1lt9a+53W2p9NWJZr9YVba3u21i5YjbKweQiBWLGqukmSdyT5+yT7JDkgyXOTfG8jy7UCr2+t7Zlka5IPJ3lLVdVarnANdt7fTvKYqtq2ystdNUutc1UdneTRSe7Zb5/tSU5fi7Ktpuos57v24r6eN05yWJLPJvlQVR2xqgUEYGpspj5VVf1kko8k+XSSg5LcPMlbk7xnoQNsGx1mbND6v57kaf22n0rLaJenp+unHZquH/MrSf5ztcu12law/T/a99lumuSeSb6b5Kyq+tlVKxysIiEQq+Gnk6S1dnJr7Yette+21t7TWvvU6ERV9ZdV9Y0+Ib/PyPDHVtW5/RGiC6rqtydZaVVdv6r+pqou7h9/U1XX78d9oKoe3D//pf4oyX371/esqk8stvzW2g+SnJTkJ5Ls1x+N+7mR9e9XVd+tqlsleWeSm48cBbj5IuU7vD868rSq+kqSl/fDj6yqT1TVt6rq81V175Ei3ao/GnhFVb2nqvbdTfEvT/KKJM/eTfv9Vt/u36iqd/f1SFV9sJ/kk31dfmPS9qyq61TVs6rqi1V1WVW9sqpu2o+bO1p1bFV9Kcn7xpTpwf3RoXE7zTsneXdr7fP99vlKa+3Efr7nJ/nlJC/qy/yifvjfVtVFfXueVVW/PLKu51R3ds0r+zY9p6q2j4y/Y1V9vB/3+iQ3GBm3d1W9o6p29e33jqo6cGT8GVX1/Kr6SJLvJLl1VR3Ut+MVVXVakt1tv6u0zs7W2p8keUmSF4ys57ZVdVr/3vxcVT2sH35YVX2lqq47Mu0Dq+pTI3V/9ci4N/bTf7OqPlhVP9MPPy7Jbyb5475d/6UfftURvAnf50/u3w+XVNVjJ6k3wEBtpj7Vc9L9c/zM1trXW2tXtNb+Lsmr0u/LFugbzPVDLu/3PVcFRrup9xlV9We1QD+pqh7Q7+cv76e93ci4C6vrj30qyberaks/7KlV9amq+nZVvbSqblZV7+yX/96q2ntkGWP3oxM6N8lHk/zhuJHV9a2Or65f+LW+77JPP/pabVVdH+zn+3kf1bfvIf3rx1XV2/rnS+6nzivXE6vqMzXS/xlx5yRvba1d3PdjLmytvbKf71VJbpnkX/oy//FibVjdWckvrqpT+/Y/s7qQcW78r1bVZ/t5X5SkRsb9ZFW9r2+7r1bVa6pqr5Hx47b/gn3A3ek/s59vrf1ekg+k+wzMreew6s7wu7yqPln9mV5V9fCq2jGvbf+wqk4Zqfvz+ucL9j9r4b7wVWf3V9VNq+v37urfJ8+q/kBl9ZdcLvQZY3MRArEa/ivJD6vqpKq6z+hOccRdknwu3T++f5HkpVVXnV1zWZJfT3KTJI9N8sKqutME631mujMk7pDk9umONjyrH/eBJIf3z++W5IIkdx95/YHFFt7vCI9JsrO1dmmS1yV51Mgkj0jy3tbaF5PcJ/2ZG/3j4kXKl3Th0j5JbpXkuKo6NMkrkzw1yV59OS8cmf6R6dpnvyR7JHnKIlV4fpIHV9VtxtTtqCTPSPKgdGc8fSjJyUnSWrtbP9nt+7q8PpO35zH941eS3DrJnkleNG/1d09yuyT3mlemx6brFN6ztXb2mPp8LN3ZTU+tqu01EnC01p7Z1+EJfZmf0I/6j3Ttv0+S1yZ5Y1WN7sgfkG677pXklLmyVtUeSd6WrqO6T5I3JnnwyHzXSdchulW6jsx3x9Tz0UmOS3cE7Iv9+s9K9xn4syRHj6njYt6S5E5VdaOqulGS0/rl7pfu/fj/qupnWmsfS3c22D1G5n1kP+0470xycL+cjyd5TZL0IdtrkvxF3673HzPvJO/zm6Y7mn1skhcv8B0BwObqU/1quv3nfG9IctequuHIsNG+wVw/ZK9+3/PR/vXu6p0s0E+q7rKck5P8Qbo+z7+mCyD2GJn3EUnu16/zyn7Yg/s6/HSS+6fbVz6jX/91kjxxZP6x+9El+N9J/nAk3Bn1xCRHpWujmyf5RpK5S/7HtdWk22tJ/dTRAlV3j5pjkty9tTbudggfS/JHVfV7VfVzo9uptfboJF9Kcv++zH/Rj1qsDR+R7qy4vZOcn66fm+rCvjf3Zd83yeeT3HW0uEn+PF3b3S7JLTISzows+37p+oPXye77gJN6S7pQJlV1QJJTkzyvX+ZTkry5qram63/epqoOHpl3oT7bgv3P3fSFR/19uj7ZrdO9Jx6T7jMzZ7HPGJtFa83DY8WPdF+qr0iyM8mV6b7QbtaPOybJ+SPT3jBJS/ITCyzrbUme1D8/PF0IMzfuwnQhQdJ9yd93ZNy9klzYPz8iyaf65+9K8rgkH+tffyDJgxZY93OSfD/dmTSXpTsi9fP9uLskuSjJdfrXO5I8bFw5Jyjf4f16bjAy/p+SvHCBcp2R5Fkjr38vybsWmPaqsqT7An99//zDSY7pn78zybEj81wn3Rkrt+pftyQ/NTJ+ovZMd3nW743Md5skP0iyJcm2frm3Hhk/N+wpST6T5MBF3me/meS96QKOryU5fl4bPW6R+b+RLtya29bvHRl3SJLv9s/vluTiJDUy/t+SPG+B5d4hyTfmleVPR17fMt3n4kYjw16b5NWLbcN5w2/bt9cBSX4jyYfmjf+nJM/unz8vycv65zfu22xu+z5nN+veq1/HTfvXr5hf70z+OTw8XQdly8j4y5Ictrvt5OHh4THkRzZPn+rKJPceM3x0X7YtC/cNRvcdu613dtNPShewvGFk3HWSfDnJ4SPt8Fvzynhhkt8cef3mJP8w8vr3k7xtgXovuh+dV68P98/fkOQF/fOdI+U7N8kRI/Psn2v3rUbb6tgkp4zM+7gkr+tffzHJnSbY5ofn2v3Uw/t2++t0fcqb7uY9fN0kj093OeD30vWpjh733ltCG75kZPx9k3y2f/6Y9O/H/nX17Te2T5guUPvPeWX5rZHXS+0DXrUN5w2/d5If9M+fluRV88a/e65Nkrw6yZ/0zw9OckWSG07w/hnX/3zcvGlakp/qt8n3khwyMu63k5wxyWfMY3M9nAnEqmitndtaO6a1dmCSn02Xtv/NyCRfGZn2O/3TPZOkP9L1seouabk83Rf7JJfK3DzdzmzOF/thSXda7U9X1c3SfUG+Mskt+qMFh+bq02fHeUNrba/W2n6ttXu01s7qy31mun+k715Vt033hXrKMsuXJLtaa/8z8voW6XbIC/nKyPPvpG+/Rbwgyb2q6vbzht8qyd/2p6Renu569ErXIRtn0vYcV+ctSW42MuyiMct/apIXt/FHk67SWntNa+2e6ToHv5PkT6vqXgtNX91lSOf2pwdfnu7ox+h7a36b3qC668FvnuTLrXV7wZG6zC33hlX1T/2ptN9KV/+9Rs9OmlfPm6fbSX973PKW4IB0O+TL023Du8xtw75+v5nuyF3ShUwP6s9oe1CSj7furLVrqKrrVtUJ1Z1m/q1cffbZRJerZfH3+dfa1UdVk8nfuwCDtIn6VF9NF1jMt3+SH6U7MDNnXN9gvgXrPX98rrmvuUbdWms/6tc32ucZt/5LR55/d8zruTZf6X50zp8k+d2q+ol5w2+V5K0j+/pzk/ww1+xbjfpAkl/ul3PdJK9Pd+bVtnT9oLnL95baT026/tdxSf68tfbNhSrSusuiXtxau2s/z/OTvKxGLsMbNWEb7m77XrX9+r7bVa+ru33D66rqy/2yX51rb5v5fbYF+4BLcEC6/nXSbcOHzuuz/VKu/ny8Nt3ZSEl3FtDbRt7jV5mw/7mQfdOdITd/m49+Dhb7jLFJCIFYda21z6ZLrRe9GVr/D+qbk/xluqNce6U7TXeSUw8vTvelOueW/bC5L66zkjwpydmtte+nS/H/KMnnW2tfnbQ+85yU7pKwRyd508jOsY2ZdsHyLTDPRUl+Mquotfa1dB3H+b8qcFGS3+7DrrnHj7XW/m2B5UzanuPqfGWu2XEa11a/luRZ1d9zYIJ6/aC19sYkn8rV77NrLLe6+/88LcnDkuzdv7e+mcneW5ckOWDeKbC3HHn+5HRnOd2ltXaTXH069uj0o+W5JMne/SVc45Y3qQemC3O+nW4bfmDeNtyztfa7SdJa+0y6nft9svtLwR6Z5Mh0NzK8abqjiqN1Gbe9Ri32PgdgmWa8T/XeJA8dM/xh6e4VNPpPblvg+Wq4Rt36ffst0p3VshrrXGw/OpF+W78l3SVnoy5Kcp95+/sbtNa+PK7crbXz04UkT0zywdbaFen+uT8u3RkrP+onXWo/NemCu19P8vKquuuY8ePq9d3W2ov7eQ9ZYNkracNL0m3Pboart++cP+/X97/6Ptujxix3fp9td33AST0w3eVZSbcNXzVvG96otXZCP/49SfatqjukC4MW6rMt1v/c3fv4q+nOIJu/zb88fnI2MyEQK1bdzWmfXFffmOwW6b7APjbB7HskuX6SXUmurO4GZL824apPThccbO2PRv1JunR/zgeSPCFXX/t8xrzXy/GqdF/qj0p3JGzOpUl+vPqbIE9YvvlemuSxVXVEdTcBPKA/42il/jrJL6Y7vXzOPyZ5el19A+CbVtVoR+3SdNcLj5qkPU9Od037QVW1Z5L/k+5ytNEzQcY5J91psy+uqgeMm6C6G9bdr6pu3LfPfZL8TJIzFyjzjdMFULuSbKmqP0l3j4RJfLSf94nV3SDwQemOdo4u+7vpbsS4T3ZzA+4k6c/A2ZHkudX9bOovpbu/wKKqc0BVPTvdKd1zncN3pDsy++iqul7/uPO8o2yvTdcJvFvG35dhri7fS3d53Q3TbbNR494Lo5b6PgdgAZusT/XcJL9Y3Q8l7NPvv38/3eU7T9vNfLvSnSm0u33PUrwhyf36/tX10v0j/b10QdZqWGw/uhTPTXePlr1Ghv1jkufX1T/gsbWqjuzHLdRWk/bZlrz/bq2dke7M47dW1V3GTVNVf1DdzaV/rO9HHZ2uneZ+IWxcn225bXhqkp+pqgdVdzb3E3P1WdFzy/7vdH22A9Kdfb47i/UBF9Sf0XRQVf19usvnntuPenWS+1fVvfppbtC3z4FJ0veT35Tk/6a7Z9BpC6xisf7ngn221toP030Wnt9/Fm+VLsjVZxsgIRCr4Yp098s5s6q+na6jcna6nexu9UcnnpjuS+kb6Y4E7O4Sq1HPS/eP9afS/fzox/thcz6Q7svygwu8XrL+cqWPp0vaPzQy/LPpdqYX9Kd53nyC8s1f9r+nv4ljujNWPpBrpvXLLfO30t0baJ+RYW9Nd6nY6/rTSc9Od8bInOckOamvy8P6YZO058vSBWUfTPKFJP+T7rr5Scr5yXRHl/65xv8awbfSBSBfSnc51F8k+d3W2of78X+b5CHV/aLB36W71vqd6W6y+cW+LJOcbp7+KOeD0l0f/Y109995y8gkf5Pkx9IdVflYunskLOaR6T4nX0+3037l7ifvfm0uXcflP5L8XLr7A7ynL+MV6Tr3D0935O4r6bbp9UeWcXK6Tsj7dnOk9pXp2ufL6e7LNP8fjZcmOaR/L7xtzPxLep8DsFubpk/VWjsv3SUvt093ec8l6W6we6/W2kd2M9930l0+9JF+33PYhHVYaHmfS3fw7u/T7bfvn+6mxN9fyXJHLLYfnVhr7Qvp+lGjZw7/bbrt+J6quqJf/l366Rdqq0m217L3362109L1WU+p/pfI5vlukr9K1zf5arr7Az24tXZBP/7P0wVQl1fVU7KCNuz7Nw9NckK6EOngdPcimvPcJHdK17c+Ndfsz41b3mJ9wHF+oe+zfStd4HaTJHdurX26X+ZF6c50eka64O6idGHU6P/ir013JtQbd3PwdLH+5/y+8Hy/n+7WFheku6/Ta9P13RmYuubljsBiqupl6X4J7FmLTgwAAABTYstGFwBmSXU31XtQkjtubEkAAABgaVwOBhOqqj9Ld0r2/+1P1wUAAICZ4XIwAAAAgAFwJhAAAADAAKzrPYH23Xfftm3btvVcJQCwjs4666yvtta2bnQ5uCZ9MADY3Cbtg61rCLRt27bs2LFjPVcJAKyjqvriRpeBa9MHA4DNbdI+mMvBAAAAAAZACAQAAAAwAEIgAIANVFUvq6rLqurskWH7VNVpVXVe/3fvfnhV1d9V1flV9amqutPGlRwAmDVCIACAjfWKJPeeN+z4JKe31g5Ocnr/Oknuk+Tg/nFckn9YpzICAJuAEAgAYAO11j6Y5OvzBh+Z5KT++UlJjhoZ/srW+ViSvapq//UpKQAw64RAAADT52attUuSpP+7Xz/8gCQXjUy3sx92LVV1XFXtqKodu3btWtPCAgCzQQgEADA7asywNm7C1tqJrbXtrbXtW7duXeNiAQCzQAgEADB9Lp27zKv/e1k/fGeSW4xMd2CSi9e5bADAjBICAQBMn1OSHN0/PzrJ20eGP6b/lbDDknxz7rIxAIDFbNnoAgAADFlVnZzk8CT7VtXOJM9OckKSN1TVsUm+lOSh/eT/muS+Sc5P8p0kj133AgMAM0sIBACwgVprj1hg1BFjpm1JHr+2JQIANiuXgwEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAA+HUwBmXb8adudBGW5cIT7rfRRQAAYEbo8wILcSYQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZg0RCoqm5RVe+vqnOr6pyqelI//DlV9eWq+kT/uO/aFxcAAACA5dgywTRXJnlya+3jVXXjJGdV1Wn9uBe21v5y7YoHAAAAwGpYNARqrV2S5JL++RVVdW6SA9a6YAAAAACsniXdE6iqtiW5Y5Iz+0FPqKpPVdXLqmrvBeY5rqp2VNWOXbt2raiwAAAAACzPxCFQVe2Z5M1J/qC19q0k/5DkJ5PcId2ZQn81br7W2omtte2tte1bt25dhSIDAAAAsFQThUBVdb10AdBrWmtvSZLW2qWttR+21n6U5J+THLp2xQQAAABgJSb5dbBK8tIk57bW/npk+P4jkz0wydmrXzwAAAAAVsMkvw521ySPTvLpqvpEP+wZSR5RVXdI0pJcmOS316SEAAAAAKzYJL8O9uEkNWbUv65+cQAAAABYC0v6dTAAAAAAZpMQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAgClVVX9YVedU1dlVdXJV3aCqDqqqM6vqvKp6fVXtsdHlBABmgxAIAGAKVdUBSZ6YZHtr7WeTXDfJw5O8IMkLW2sHJ/lGkmM3rpQAwCwRAgEATK8tSX6sqrYkuWGSS5LcI8mb+vEnJTlqg8oGAMwYIRAAwBRqrX05yV8m+VK68OebSc5Kcnlr7cp+sp1JDhg3f1UdV1U7qmrHrl271qPIAMCUEwIBAEyhqto7yZFJDkpy8yQ3SnKfMZO2cfO31k5srW1vrW3funXr2hUUAJgZQiAAgOl0zyRfaK3taq39IMlbkvxikr36y8OS5MAkF29UAQGA2SIEAgCYTl9KclhV3bCqKskRST6T5P1JHtJPc3SSt29Q+QCAGSMEAgCYQq21M9PdAPrjST6drt92YpKnJfmjqjo/yY8neemGFRIAmClbFp8EAICN0Fp7dpJnzxt8QZJDN6A4AMCMcyYQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABiALRtdAGBx244/daOLsGwXnnC/jS4CAAAAcSYQAAAAwCAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAzAoiFQVd2iqt5fVedW1TlV9aR++D5VdVpVndf/3XvtiwsAAADAckxyJtCVSZ7cWrtdksOSPL6qDklyfJLTW2sHJzm9fw0AAADAFFo0BGqtXdJa+3j//Iok5yY5IMmRSU7qJzspyVFrVUgAAAAAVmZJ9wSqqm1J7pjkzCQ3a61dknRBUZL9FpjnuKraUVU7du3atbLSAgAAALAsE4dAVbVnkjcn+YPW2rcmna+1dmJrbXtrbfvWrVuXU0YAAAAAVmiiEKiqrpcuAHpNa+0t/eBLq2r/fvz+SS5bmyICAAAAsFKT/DpYJXlpknNba389MuqUJEf3z49O8vbVLx4AAAAAq2HLBNPcNcmjk3y6qj7RD3tGkhOSvKGqjk3ypSQPXZsiAgAAALBSi4ZArbUPJ6kFRh+xusUBAAAAYC0s6dfBAAAAAJhNQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAwpapqr6p6U1V9tqrOrapfqKp9quq0qjqv/7v3RpcTAJgNQiAAgOn1t0ne1Vq7bZLbJzk3yfFJTm+tHZzk9P41AMCihEAAAFOoqm6S5G5JXpokrbXvt9YuT3JkkpP6yU5KctTGlBAAmDVCIACA6XTrJLuSvLyq/rOqXlJVN0pys9baJUnS/91vIwsJAMwOIRAAwHTakuROSf6htXbHJN/OEi79qqrjqmpHVe3YtWvXWpURAJghQiAAgOm0M8nO1tqZ/es3pQuFLq2q/ZOk/3vZuJlbaye21ra31rZv3bp1XQoMAEw3IRAAwBRqrX0lyUVVdZt+0BFJPpPklCRH98OOTvL2DSgeADCDtmx0AQAAWNDvJ3lNVe2R5IIkj013EO8NVXVski8leegGlg8AmCFCIACAKdVa+0SS7WNGHbHeZQEAZp/LwQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgABYNgarqZVV1WVWdPTLsOVX15ar6RP+479oWEwAAAICVmORMoFckufeY4S9srd2hf/zr6hYLAAAAgNW0aAjUWvtgkq+vQ1kAAAAAWCNbVjDvE6rqMUl2JHlya+0b4yaqquOSHJckt7zlLVewOqbJtuNP3egiAAAAAEuw3BtD/0OSn0xyhySXJPmrhSZsrZ3YWtveWtu+devWZa4OAAAAgJVYVgjUWru0tfbD1tqPkvxzkkNXt1gAAAAArKZlhUBVtf/IywcmOXuhaQEAAADYeIveE6iqTk5yeJJ9q2pnkmcnObyq7pCkJbkwyW+vYRkBAAAAWKFFQ6DW2iPGDH7pGpQFAAAAgDWy3BtDAwAAADBDhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIASR0ZFJAAAgAElEQVQCAJhiVXXdqvrPqnpH//qgqjqzqs6rqtdX1R4bXUYAYDYIgQAAptuTkpw78voFSV7YWjs4yTeSHLshpQIAZo4QCABgSlXVgUnul+Ql/etKco8kb+onOSnJURtTOgBg1giBAACm198k+eMkP+pf/3iSy1trV/avdyY5YNyMVXVcVe2oqh27du1a+5ICAFNPCAQAMIWq6teTXNZaO2t08JhJ27j5W2sntta2t9a2b926dU3KCADMli0bXQAAAMa6a5IHVNV9k9wgyU3SnRm0V1Vt6c8GOjDJxRtYRgBghjgTCABgCrXWnt5aO7C1ti3Jw5O8r7X2m0nen+Qh/WRHJ3n7BhURAJgxQiAAgNnytCR/VFXnp7tH0Es3uDwAwIxwORgAwJRrrZ2R5Iz++QVJDt3I8gAAs8mZQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAsGgJV1cuq6rKqOntk2D5VdVpVndf/3XttiwkAAADASkxyJtArktx73rDjk5zeWjs4yen9awAAAACm1KIhUGvtg0m+Pm/wkUlO6p+flOSoVS4XAAAAAKtoufcEullr7ZIk6f/ut9CEVXVcVe2oqh27du1a5uoAAAAAWIk1vzF0a+3E1tr21tr2rVu3rvXqAAAAABhjuSHQpVW1f5L0fy9bvSIBAAAAsNqWGwKdkuTo/vnRSd6+OsUBAAAAYC1M8hPxJyf5aJLbVNXOqjo2yQlJfrWqzkvyq/1rAAAAAKbUlsUmaK09YoFRR6xyWQAAAABYI2t+Y2gAAAAANp4QCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAFs2ugDA5rbt+FM3ugjLcuEJ99voIgAAAKwqZwIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADsGWjCwAwjbYdf+pGF2FZLjzhfhtdBAAAYEo5EwgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAYApV1S2q6v1VdW5VnVNVT+qH71NVp1XVef3fvTe6rADAbBACAQBMpyuTPLm1drskhyV5fFUdkuT4JKe31g5Ocnr/GgBgUUIgAIAp1Fq7pLX28f75FUnOTXJAkiOTnNRPdlKSozamhADArBECAQBMuaraluSOSc5McrPW2iVJFxQl2W+BeY6rqh1VtWPXrl3rVVQAYIoJgQAAplhV7ZnkzUn+oLX2rUnna62d2Frb3lrbvnXr1rUrIAAwM4RAAABTqqquly4Aek1r7S394Eurav9+/P5JLtuo8gEAs0UIBAAwhaqqkrw0ybmttb8eGXVKkqP750cneft6lw0AmE1bNroAAACMddckj07y6ar6RD/sGUlOSPKGqjo2yZeSPHSDygcAzBghEADAFGqtfThJLTD6iPUsCwCwObgcDAAAAGAAVnQmUFVdmOSKJD9McmVrbftqFAoAAACA1bUal4P9Smvtq6uwHAAAAADWiMvBAAAAAAZgpWcCtSTvqaqW5J9aayfOn6CqjktyXJLc8pa3XOHqNp9tx5+60UUANpFZ/U658IT7bXQRAABg01vpmUB3ba3dKcl9kjy+qu42f4LW2omtte2tte1bt25d4eoAAAAAWI4VhUCttYv7v5cleWuSQ1ejUAAAAACsrmWHQFV1o6q68dzzJL+W5OzVKhgAAAAAq2cl9wS6WZK3VtXccl7bWnvXqpQKAAAAgFW17BCotXZBktuvYlkAAAAAWCN+Ih4AAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIAB2LLRBVgt244/daOLAAAAADC1nAkEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGYMtGFwAAth1/6kYXYVAuPOF+G10EAAA2gDOBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgALZsdAEAAACm0bbjT93oIgCsKmcCAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMAArCoGq6t5V9bmqOr+qjl+tQgEAsDB9MABgOZYdAlXVdZO8OMl9khyS5BFVdchqFQwAgGvTBwMAlmslZwIdmuT81toFrbXvJ3ldkiNXp1gAACxAHwwAWJYtK5j3gCQXjbzemeQu8yeqquOSHNe//O+q+twK1rlZ7JvkqxtdiAHR3utLe68/bb6+Zr696wVruvhbrenSSaavDzbzn4kJDKGOiXpuJjNZx2Xsn2aynsswhHoOoY7J2tZzoj7YSkKgGjOsXWtAaycmOXEF69l0qmpHa237RpdjKLT3+tLe60+bry/tzRSYqj7YED4TQ6hjop6byRDqmKjnZjKEOibTUc+VXA62M8ktRl4fmOTilRUHAIBF6IMBAMuykhDoP5IcXFUHVdUeSR6e5JTVKRYAAAvQBwMAlmXZl4O11q6sqickeXeS6yZ5WWvtnFUr2ebm8rj1pb3Xl/Zef9p8fWlvNtQU9sGG8JkYQh0T9dxMhlDHRD03kyHUMZmCelZr17qEHAAAAIBNZiWXgwEAAAAwI4RAAAAAAAMgBFpFVXXvqvpcVZ1fVcePGX/9qnp9P/7MqtrWD79eVZ1UVZ+uqnOr6unrXfZZtYI236OqXt63+Ser6vB1LvpMmqC971ZVH6+qK6vqIfPGHV1V5/WPo9ev1LNrhe39rqq6vKresX4lnn3LbfOqukNVfbSqzqmqT1XVb6xvyWFyVbVPVZ3Wfx+fVlV7LzDd2O/tqvr5fv95flX9XVXV7pZbVbftPx/fq6qnzFvH2M9cf9PrM/tlvb6/Afa017P66c7vvwfu1A//lar6xMjjf6rqqH7cK6rqCyPj7jCLdezH/XCkHqeMDN9M23LB7/rlbsuFPgMj48f2ZftxT++Hf66q7rXYMhfaFstZx1JNST3/qKo+02+706vqViPzjH3/zmg9j6mqXSP1edzIPCvuj09JHV84Ur//qqrLR+aZxW35hH5Yq6p9R4ZXLfydu/xt2VrzWIVHuhszfj7JrZPskeSTSQ6ZN83vJfnH/vnDk7y+f/7IJK/rn98wyYVJtm10nab9scI2f3ySl/fP90tyVpLrbHSdpvkxYXtvS/K/krwyyUNGhu+T5IL+79798703uk7T/FhJe/fjjkhy/yTv2Oi6zMpjhe/xn05ycP/85kkuSbLXRtfJw2PcI8lfJDm+f358kheMmWbB7+0k/57kF5JUkncmuc/ultvvZ++c5PlJnjKyjgU/c0nekOTh/fN/TPK7M1DP+/bTVZLDkpy5wPq+nuSG/etXzP/+ntU6JvnvBcq4abZldvNdv5xtmZX1ZQ/pp79+koP65Vx3d8tcaFssdR3L2H7TUs9fydWfvd+dW8fu3r8zWs9jkrxoKZ+RWavjvPX9frofSJjlbXnHdH3MC5PsO7KOhb6LVrQtnQm0eg5Ncn5r7YLW2veTvC7JkfOmOTLJSf3zNyU5oqoqSUtyo6rakuTHknw/ybfWp9gzbSVtfkiS05OktXZZksuTbF+XUs+uRdu7tXZha+1TSX40b957JTmttfb11to3kpyW5N7rUegZtpL2Tmvt9CRXrEtJN49lt3lr7b9aa+f1zy9OclmSretTbFiy0X3jSUmOGjPN2O/tqto/yU1aax9tXU/0lSPzj11ua+2y1tp/JPnBvHWM/cz1++l7pNtv766MU1XPfvgrW+djSfbqlzPqIUne2Vr7zjLqM8401vEqm21brsF3/Ur6skemO4j8vdbaF5Kc3y9vOZ+rpa5jJuvZWnv/yGfvY0kOXEZdpr6eu7Ea/fFprOMjkpy8xHosZt3qmSSttf9srV04phwLfeeuaFsKgVbPAUkuGnm9sx82dprW2pVJvpnkx9O9ab6d7mjCl5L8ZWvt62td4E1gJW3+yXRfLluq6qAkP5/kFmte4tk2SXuvxbxDpc3W36q0eVUdmu4Iz+dXqVyw2m7WWrskSfq/+42ZZqHPwwH98/nDJ13uJOv48SSX9/vt+etYivWu5yTfIQ/Ptf9ZeX5/mv8Lq+r6k1RsxDTV8QZVtaOqPlb95W7ZxNtyge/6pW7LlfRld1ffpX6ulrqOpZqWeo46Nt0ZFnPGvX+Xaprq+eD+vfimqpr7H2c1tuc01THVXdJ3UJL3jQyetW25nHKsaFtumXRCFlVjhrUJpzk0yQ/TnVa6d5IPVdV7W2sXrG4RN52VtPnLktwuyY4kX0zyb0muHDMtV5ukvddi3qHSZutvxW3eH515VZKjW2vXOkML1ktVvTfJT4wZ9cxJFzFmWNvN8OVY8TqmrJ67naf/fvi5JO8eGf/0JF9JFyacmORpSf70GgudnTresrV2cVXdOsn7qurTGX9m+2bZlvO/6xfdlktdzyLTLDR83EH+xdphrT/v01LPbkVVj0p3BcDdRwZf6/3bWlvqwZxpqee/JDm5tfa9qvqddGer3GPC8i1mWuo45+FJ3tRa++HIsFnblsspx4q2pRBo9ezMNc8kOTDJxQtMs7O/9Oum6a4Lf2SSd7XWfpDksqr6SLovJiHQ7i27zfvTgf9wbqKq+rck561tcWfeJO29u3kPnzfvGatSqs1rJe3N8qyozavqJklOTfKs/pRd2DCttXsuNK6qLu0vbbmk/2f2sjGTLfS9vTPXvIRi9HMyyXLnr2PcZ+6r6U5539IfXV3wszhl9VzsO+RhSd7a9/fmyn9J//R7VfXyJNe4cfYs1bG/PCqttQuq6ox097h4czbZtlzou36SbblAmZf7/8Pu5l3q52o561iKaalnquqe6YLFu7fWvjc3fIH371KDg6moZ2vtayPT/3OSF4ys+/B5yzpj4tpds/zjyjl/mjXdlr2Hp7vX61VmcFsupxwr2pYuB1s9/5Hk4OruYL5Hujfk/LuRn5Lk6P75Q5K8rw8jvpTkHtW5UbqbPn12nco9y5bd5lV1w76tU1W/muTK1tpn1qvgM2qS9l7Iu5P8WlXtXd0vb/xarnkklGtbSXuzPMtu8376t6a7bvuNa1hGWA2j+8ajk7x9zDRjv7f7f3SvqKrD+nsfPGZk/kmWO2rsZ67vG70/3X570mWNs971PCXJY/r+3GFJvjkSDCRj7lvRBw9z9845KsnZs1jHftnX7+uyb5K7JvnMZtuWu/uuX+a2XMn/D6ckeXh1v1B0UJKD090Aezmfq6WuY6mmop5Vdcck/5TkAa27J2j64WPfvzNcz9H7dD0gybn989Xoj09FHft63ibdVTQfHRk2c9tykXIstF9Z2bZsK7xztsc17hB+3yT/lS5pfGY/7E/TfdEkyQ2SvDHdzaH+Pcmt++F79sPPSfcmfepG12VWHito821JPpfuS/G9SW610XWZhccE7X3ndMn0t5N8Lck5I/P+Vr8dzk/y2I2uyyw8VtjeH0qyK8l3+2nutdH1mYXHcts8yaPS3fT2EyOPO2x0fTw8xj3S3bPg9HRnwJ6eZJ9++PYkLxmZbuz3dj/d2f3n5EVJapHl/kT/uflWuh9i2JnuRr1jP3P98Fv3++3z+/349WegnpXkxf30n06yfWRZ25J8OfN+iTTdfSw+3a/n1Un2nMU6JvnF/vUn+7/HbsZtmd181y93W2aZfdl+3DP7+T6X/hfPlvO5Ws46lrENp6Ge701y6ci2O2Wx9++M1vPP0/1f+cl0IcptF/uMzFod+3HPSXLCvLLN6rZ8Yrp945XpzvR5ye6+i1a6Lee+AAEAAADYxFwOBgAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAFOuqh5aVedU1Y+qavtyliEEAgAAAJgiVXV4Vb1i3uCzkzwoyQeXu9wtKykUAAAAAGuvtXZuklTVspfhTCAAAACAAXAmEAAAAMAUqKozk1w/yZ5J9qmqT/SjntZae/dKly8EAgAAAJgCrbW7JN09gZIc01o7ZjWX73IwAAAAgAEQAgEAAABMuap6YFXtTPILSU6tqiVfHlattdUvGQAAAABTxZlAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGID/z969x9tW1XXj/3zjiCiigBwJuXjQyGuPYCfELPOCeS0wb1gZ+mB00dK0FLVfommh5e156qWhKJiKIGqipokomiXoQVFRVBAQEYSjgoD6pOj4/THnhsVm77PvZ6115vv9eq3Xnmve1hhzzrXW2J855lxCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQDAhqur4qnrpuMuxFqqqVdUvjLscC6mqo6vqrau4vqdU1SdXa31LfO3fq6oPr2D5D1bV4atZJgCYT1U9sKouHXc5lmO12w/jUlUb+jbbunGXZSGr3W4e5z6sqhdU1RtXsPyXquqBq1gktnFCILaKqrq4qn5UVddW1dVV9d9V9cdVtdWPwf4f859W1XVVdU1VnVNVj17EchPzxThSlg/MGv/Wqjp6keu4uKoOXpMCrkBVHVFVX+mPlSuq6gNVtVM/bZsJykb24XX944qqen9VPXQ11t9ae1tr7TcXWZabNXxaa49orZ2wGmUBYPVMUpuqL89eVfW2qvpuVf2gqj69yHbVVJwgWqk+3GpV9c+zxn+yqp6yyHVM5Lbqw4uL+nbMpVV10si0M6rqaeMs32rp9+HPRtpsl1bVyVX1K6ux/tba37XWFrWt5moLt9bu2Vo7YzXKwjAIgdiafqu1tlOSOyU5Jsnzkhw3prJ8qrV2myQ792U4uap2XcsXXKPw6KCquv8arHdVLLXOVfUbSf4uyZP6Y+XuSU5ei7KttqrabpmL7twfi/dOclqS9yy2UQjAYE1Em6pvO30yyY+T3DPJbkleneTtVfW4eZYZ68m06mzt/4F+kOQPqmrDVn7dRVtGm+3wJE9OcnDfjtmY5PS1KNtqWsH+v6yv505JDkrylST/WVUPWdUCwlYgBGKra619v7V2apInJjm8qu6VJFV1y6r6x6q6pO8V8fqqutXMclX16L7XzsxZr/81Mu3iqnp+VX25qq6qqjdX1Q6LKMvPkrwpya2S3Lmqzq2q3xpZ7y2q6jtVtX+ST/Sjr+7PAtyvqn6uqv66qr5RVVdW1Vuq6nb9sjM9PY6oqkuSfLQf/2t9+a+uqm/O+od/l77ny7VVdVZV3WWBKrwiybw9Y+bbZlX1r0n2SfK+vi7PraoTquo5/fQ9+7L/af/8F6rqe1VV/fM/rKoL+nGnVtUdR16zVdXTq+r8JOfPUaZf6+v9oDmK/CvpArrPJUlr7XuttRNaa9dW1ZFJfi/Jc/syv69f31FV9fV+m325qh4z8lpP6c+0/WN/XFxUVY8Ymb5vVX28X/a0dI3X0bK+s6q+XVXfr6pPVNU9R6YdX1Wvq6p/r6ofJHlQVd2+3x7XVNWnkyy0/27QWvt2a+21SY5O8vKZBkpV3bGq3lVVm/vy//nI+B/VSHhZVQf0x+statalaFX12n67X1NVZ1fVr/fjH57kBUme2G/Xz/fjbziDt8jj/PDq3rvfqaoXLrbeACzfBLSp/iLJdUmO6L/HftRaOzHJy5K8cqTdcJO2QVXNtKk+33/3PHHk9Z/Tf9dcXlVPHRl/fFX9c83TTqqqX62qz/Tf2Z+pql8dmXZGVb2sqv4ryQ/TtfnOqKqX9vW/rqre13+Pv63/rvxMjYQ2832PLtLVSY5P8qL5Zqiq/11V5/Xb/D+q6k79+Jttq77t8th++q/12/eR/fODq+qcfnjJ7dRZZXpsfzzca44i/0qS/2itfT25oR1zbL/cy5L8epJ/6sv8Twttw+p6JZ/cl/Ha6i5x2jgy/YCq+mw/7aQkO4xM26W63tSb++33/qraa2T6XPt/39pCG3A+rXNpa+1vkrwxyctHXuduVXVade3jr1bVE/rxB1XXntxuZN7HVNUXRur+1pFpc7Y/a/628A29+6t777+mqi7rH6+pqlv20x5YXS+mOd9jDEhrzcNjzR9JLk53pmD2+EuS/Ek//JokpybZNV3K/r4kf99Pu0+SK5PcN8l2SQ7v13nLkfWfm2Tvfvn/SvLSecrylCSf7IfXJXlmkmuT3C7Jc5OcNDLvIUm+2A9vSNKSrBuZ/r+TXJDkzkluk+TdSf511vxvSbJjuqBpn/61npTkFklun2T/fv7jk3wvyYF9ud6W5B3z1GFm3bdJ8q2ZbZvkrUmOXsI2O3hWXd7XD/9ukq/PbIt+2nv74Qcn+U6//lsm+b9JPjGynpauR8uuSW41Mu4XkjwsyTeTHDhPvX49yY+SvDjJ/WfKOjL9+Nn7Ncnjk9wxXaj9xHRn2/YY2dc/SfKH/Tb4kySXJal++qeSvKqvxwP6ffPWWdtkp376a5KcM6ss3+/L+XPpGiPvSNdzacck9+r3zScX2IfrZo2/cz/+7v16z07yN0m276ddmORh/bwfTfKHI8v+Q5LXzz7O++e/n+54W5fkOUm+nWSHftrRo/Xux52R5GlLOM7fkO4Yv3eS/0ly93F/7nh4eHhsi49MVpvqzCQvnmP8vv13w1375/O2DUaWeWCS65O8JF0b6ZHp/mHfpZ9+fOZpJ/XrvSpdz5R16dpZVyW5fT/9jH773LOffot+3AXpTtjcLsmXk3wtycH9PG9J8uaR8i3pe3RWvS5N8vNJrhnZJp9M8pR++NC+LHfv1//XSf57ZB2zt9VLkvzffvgF6dpsLx+Z9tp+eKnt1Jlx65I8tV/2F+ap1+/3++Ov0vUC2m7W9DPStyOWsA3/X7/ft0vy90nO7Kdtn+Qb6ULHWyR5XLr23Uv76bdP8tgkt053vL8zyb/NKsvs/b/FNuBc+3CO8Q9O8rN+++2Yro371P417pOuvXzPft6vJ3noyLLvTHLUXMdPFm5/zm4LX5wb/xd4Sbr35R2SrE/y30n+djHvMY/hPMZeAI9hPDJ/g+XMJC9MUun+eb/LyLT7JbmoH37dzAfYyPSvJvmNkfX/8ci0Ryb5+jxleUr/AXh1/+F85sgH5x37L4Hb9s9PSfLcfnhDbh4CnZ7kT0ee37X/Ulo3Mv+dR6Y/P8l75inX8UneOKsOX5ln3hvKkuRPc+OX5GgItJhtNhoC3aXfJj+X5PVJ/ij9F16SE5I8ux8+LskrRpa7TV/nDf3zluTBs1639XX/RpJfWuBYeUS6xurV6c4uvip9wyJzfPHNsfw5SQ4Z2dcXjEy7dV+Wn08XyF2fZMeR6W/P/A2AnftlbzdSlreMTN+u3w53Gxn3d1l6CLRDP/7+6Rrol8ya/vz0jdIkT0vy0X640jU+HjBS9zlfu59+VZJ798NHz653bhoCLeY432tk+qeTHLbYzwcPDw8Pj8U/Zn9/j4wfR5vqgtF5R8bf8F3WP5+vbTA7BPpRbtrOujLJQf3w8ZmnnZQu/Pn0rPV/KjeGLGckecms6WckeeHI81cm+eDI89/KyD/fc9Rxi9+js+o10556RW48wTYaAn0wXW+qmWV+Lt0/53eaZ1s9JMkX+uEPpWsPzLQFP57kd/rhpbZTZ8b9ZbpQbK/56t/P/3tJPtIfb99NH2qMbN+nLbD87G34kZFp90jyo374ARk5ideP++/MH07un+SqWWV5ycjzpbYBb9iHs8bfrd9ee6Y7Efmfs6b/S5IX9cMvTfKmfninfpvN7N8tHT9ztT+3FAJ9PckjR6Y9LMnFi3mPeQzn4XIwxm3PdGcR1qf7B/3s6romX53uS219P9+dkjxnZlo/fe90oc2Mb44Mf2PWtNnObK3t3FrbrbV2UGvtI0nSWrss3Rmvx1bVzukCibdtYT137F9r9HXXJdl9nnLtne7DeT7fHhn+YbqAZSFvSLJ7jVzG1lvMNrtB67rzXpfui/PXk7w/yWVVddckv5GuUZHMqnNr7bp0X/x7jqxutM4znpXk5NbaF7dUmdbaB1trv5XurN4h6cKMeW+WV1V/UDd2ab86XQ+c0S69N2zT1toP+8Hb9PW4qrX2g5F5b6hXVW1XVcdUd6nZNem+YDNr3aP1XJ9u388+DpdqZjt+L90+vOOsffiC3Hh8nZLkftVdjveAdI2E/5xrpX3X3/P6rsVXpzvruaiuz1nccb6cYxeA1TOONtV3kuwxx/g9RqbPtc75fLe1dv3I83pnch0AACAASURBVNnfJ/N918z+nkr/fKG2yRUjwz+a4/kNr73C79EZL0/ysKq696zxd0ry2pH98b10Yd6es1fQ+1SSX6yq3dO1296SZO+q2i1dT6mZS8iW2k6d8VdJ/rm1tsVfa2vdj1AcnC6o+OMkL6mqh803/yK24ez9u0N19yq6Y5JvtdalFiN1mVnvravqX/rL3q5JV/+d66b3axyt5xbbgEuwZ7q219Xp9uF9Z72vfi/dicekC5l+p78063eSfLa1drPXXGT7c0vm2uej79+F3mMMgBCIsanujvp7pjsT8p10X7b37MOZnVtrt2vdDdiS7oP7ZSPTdm6t3bp1153P2HtkeJ90ZwyW44R03VUfn+7+NN/qx7c55r0s3Yf+6Oten5s2IkaX+2aWcJ+YxWit/STd5VN/m67BMPpaW9pmc9Xn4+m62G7f1/vjSf4gyS7petgks+pcVTum64b7rZH1zLXuxyc5tKqetch6/ay1dnq6S55mrkW/yXqru17+DUmeka7L987purCPbof5XJ7uHkw7jozbZ2T4d9OFUAena6RsmHnZ0WKODG9Ot+9nH4dL9Zh0Z2W+mm4fXjRrH+7UWntkkrTWrk7y4SRP6Mt74qwGUlfg7pr75/Xz7dJvp++P1GWu/TVqMcc5AGMyxjbVR9KdOJv9P8UT+tf52si4hb5rVmL291TSlXuhtsmiLOJ7dFFaa99Nd3nP386a9M0kfzRrn9yqtfbf86znh+kuF39mknNbaz9O1zPm2el6bc2Eb0ttp874zSR/Xf19hxZRr5+01t6Z5AuZv822km14eZI9q2p03tE21nPS9XK6b2vttulOjCXzt9kWagMu1mPShTk/SLcPPz5rH96mtfYnSdJa+3K6QOYR6dpsb59nnQu1P5fTZlvu/0Rso4RAbHVVddvqfjr0Hem6Pn6xdTdofkOSV1fVHfr59hw5m/CGJH9cVfetzo5V9ajqfzq89/TqfqZ013S9JU7K8vxbuut4n5nuzMqMzemu+73zyLgTk/xFdTeXu026y39OmpWwj3pbkoOr6glVta66GxDuv8xyjvrXdNcNP3xk3ELb7IpZdUm60OcZufEM0hlJ/izdZUU/7ce9PclTq2r//mzG3yU5q7V28QJlvCxd9+U/r/6G07NV1SFVdVh1N/irqjowXS+kM+cp847pvgw398s/NTc2PraoP/uyKcmLq2r7qvq1dF2/Z+yU7t423013RvXvFljfT9Nda390f0bqHunus7AoVbV7VT0j3U0jn9+/Jz6d5Jqqel5V3ao/O3SvuulPkr49XVD32MzfoNgpXaNvc5J1VfU3SW47Mv2KJBvmaMTPWOpxDsBWMAFtqlen+z45rqp+vqp2qKonpbss7a/mOjExYq52yHL9e7qeMb/bt6+emO5yovev0voX+h5dilcl+dV09/+Z8fokz68bbwB8u6p6/Mj0LbXZZnpqnzHrebL87+8vpWtT/nNV/fZcM1T3AxSPqqqdqrsB9SPS3XPnrHnKvJJt+Kl+2T/v9+/vpOvxNLruH6X78ZZds4UbcCeLagPOq3/P7FlVL0rXU/0F/aT3pzsGn1zdD3Tcoqp+papG9/Pbk/x5upDqnfO8xELtz4XeNyemC/DWV9cz7G/S3S4CbiAEYmt6X1Vdmy4pf2G6L8HRO9I/L9215WdW1/3xI+lS/bTWNqW7ue8/pbt++IJ0lwmNenu6XhEX9o95fzVrS1prP0ryrnQ3NXz3yPgfpvu1i/+qrpvnQel+Wexf04UmF6W7od2fbWHdl6S7hv056br6npPuRror0gcQL0p3CdXMuIW22d+n+5K4uqr+sh/38XRfPjMh0CfTfQHNPE/fO+f/S7eNLk/Xs+mwRZbzknRB0POq/+WpWa7qy3x+upsnvjXJP7TWZi7JOy7JPfoy/1t/VuWV6RoHVyT5pXSX8y3W76a778730m2/0dDvLenO2Hwr3XXxZ95s6Zt7Rroutd9Od832mxexzNXV/brYF9MdG49vrb0puWG//la6rt4XpTu7+8Z0Z4ZmnJpkvyRXtNY+P89r/Ee6+w18ra/T/8tNu0XPNES+W1WfnWP5JR3nAKy5iWhT9T1bfi3dPYC+nO4f12cneXJrbaGTcUcnOaH/Tn/CgjXegr4cj07Xvvpuuh/6ePRIj5iVWuh7dNFaa9ekuzfQaJvtPekuFXtHv7/OTddjZMbRufm2mt1mm/08WcH3d9+meHSSN9TIL6uOuCZdAHJJusuhXpHuxuQzv0z62iSPq+7Xuv5PVrAN+55Ov5PuOL0q3f133j0yy2vS3dh65l6fH1rEarfUBpzLHavqunS3TvhMujbnA1trH+7LeG26HlSHpTvx+e10+/SWI+s4Md19eT66hWNzofbnTdrCcyz/0nQB1xfStS0/m2X+T8S2q7Yc0MN0qKqL09187iOrtL6/SfKLrbXfX431AQBMg9VuUwEwWdaNuwAwafpupEek+6UJAAAA2Ca4HAxGVNUfpuua+sHW2icWmh8AAACmhcvBAAAAAAZATyAAAACAAVjwnkBVtXe6u5T/fLqfxz62tfbaqjo63S8LbO5nfUFr7d+3tK7ddtutbdiwYUUFBgAm19lnn/2d1tr6cZeDm9IGA4Bt22LbYIu5MfT1SZ7TWvtsVe2U5OyqOq2f9urW2j8utlAbNmzIpk2bFjs7ADBlquob4y4DN6cNBgDbtsW2wRYMgVprlye5vB++tqrOS7LnyooHAAAAwNa0pHsCVdWGJAckOasf9Yyq+kJVvamqdplnmSOralNVbdq8efNcswAAAACwxhYdAlXVbZK8K8mzWmvXJHldkrsk2T9dT6FXzrVca+3Y1trG1trG9evdIgAAAABgHBYVAlXVLdIFQG9rrb07SVprV7TWftpa+1mSNyQ5cO2KCQAAAMBKLBgCVVUlOS7Jea21V42M32NktsckOXf1iwcAAADAaljMr4PdP8mTk3yxqs7px70gyZOqav8kLcnFSf5oTUoIAAAAwIot5tfBPpmk5pj076tfHAAAAADWwpJ+HQwAAACA6SQEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADsG7cBQAWtuGoD4y7CMt28TGPGncRAAAGZVrbjtqNsPb0BAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgCYUFX1F1X1pao6t6pOrKodqmrfqjqrqs6vqpOqavtxlxMAmA5CIACACVRVeyb58yQbW2v3SrJdksOSvDzJq1tr+yW5KskR4yslADBNhEAAAJNrXZJbVdW6JLdOcnmSByc5pZ9+QpJDx1Q2AGDKCIEAACZQa+1bSf4xySXpwp/vJzk7ydWttev72S5Nsud4SggATBshEADABKqqXZIckmTfJHdMsmOSR8wxa5tn+SOralNVbdq8efPaFRQAmBpCIACAyXRwkotaa5tbaz9J8u4kv5pk5/7ysCTZK8llcy3cWju2tbaxtbZx/fr1W6fEAMBEEwIBAEymS5IcVFW3rqpK8pAkX07ysSSP6+c5PMl7x1Q+AGDKCIEAACZQa+2sdDeA/mySL6Zrtx2b5HlJnl1VFyS5fZLjxlZIAGCqrFt4FgAAxqG19qIkL5o1+sIkB46hOADAlNMTCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAJhAVXXXqjpn5HFNVT2rqnatqtOq6vz+7y7jLisAMB2EQAAAE6i19tXW2v6ttf2T/HKSHyZ5T5KjkpzeWtsvyen9cwCABQmBAAAm30OSfL219o0khyQ5oR9/QpJDx1YqAGCqLBgCVdXeVfWxqjqvqr5UVc/sx+uKDACwdRyW5MR+ePfW2uVJ0v+9w1wLVNWRVbWpqjZt3rx5KxUTAJhki+kJdH2S57TW7p7koCRPr6p7RFdkAIA1V1XbJ/ntJO9cynKttWNbaxtbaxvXr1+/NoUDAKbKgiFQa+3y1tpn++Frk5yXZM/oigwAsDU8IslnW2tX9M+vqKo9kqT/e+XYSgYATJUl3ROoqjYkOSDJWVlkV2QAAFbkSbnxUrAkOTXJ4f3w4Uneu9VLBABMpUWHQFV1myTvSvKs1to1S1jO9egAAMtQVbdO8tAk7x4ZfUySh1bV+f20Y8ZRNgBg+qxbzExVdYt0AdDbWmszjZArqmqP1trlW+qK3Fo7NsmxSbJx48a2CmUGABiE1toPk9x+1rjvpvu1MACAJVnMr4NVkuOSnNdae9XIJF2RAQAAAKbEYnoC3T/Jk5N8sarO6ce9IF3X45Or6ogklyR5/NoUEQAAAICVWjAEaq19MknNM1lXZAAAAIApsKRfBwMAAABgOgmBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAE6qqdq6qU6rqK1V1XlXdr6p2rarTqur8/u8u4y4nADAdhEAAAJPrtUk+1Fq7W5J7JzkvyVFJTm+t7Zfk9P45AMCChEAAABOoqm6b5AFJjkuS1tqPW2tXJzkkyQn9bCckOXQ8JQQApo0QCABgMt05yeYkb66qz1XVG6tqxyS7t9YuT5L+7x3GWUgAYHoIgQAAJtO6JPdJ8rrW2gFJfpAlXPpVVUdW1aaq2rR58+a1KiMAMEWEQAAAk+nSJJe21s7qn5+SLhS6oqr2SJL+75VzLdxaO7a1trG1tnH9+vVbpcAAwGQTAgEATKDW2reTfLOq7tqPekiSLyc5Ncnh/bjDk7x3DMUDAKbQunEXAACAef1ZkrdV1fZJLkzy1HQn8U6uqiOSXJLk8WMsHwAwRYRAAAATqrV2TpKNc0x6yNYuCwAw/VwOBgAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABiAdeMuALBt23DUB8ZdhGW5+JhHjbsIAAAAq0pPIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYAAWDIGq6k1VdWVVnTsy7uiq+lZVndM/Hrm2xQQAAABgJRbTE+j4JA+fY/yrW2v7949/X91iAQAAALCaFgyBWmufSPK9rVAWAAAAANbISu4J9Iyq+kJ/udgu881UVUdW1aaq2rR58+YVvBwAAAAAy7XcEOh1Se6SZP8klyd55XwzttaOba1tbK1tXL9+/TJfDgAAAICVWFYI1Fq7orX209baz5K8IcmBq1ssAAAAAFbTskKgqtpj5Oljkpw737wAAAAAjN+6hWaoqhOTPDDJblV1aZIXJXlgVe2fpCW5OMkfrWEZAQAAAFihBUOg1tqT5hh93BqUBQCAEVV1cZJrk/w0yfWttY1VtWuSk5JsSHcy7gmttavGVUYAYHqs5NfBAABYew9qre3fWtvYPz8qyemttf2SnN4/BwBYkBAIAGC6HJLkhH74hCSHjrEsAMAUEQIBAEyuluTDVXV2VR3Zj9u9tXZ5kvR/7zDXglV1ZFVtqqpNmzdv3krFBQAm2YL3BAIAYGzu31q7rKrukOS0qvrKYhdsrR2b5Ngk2bhxY1urAgIA00NPIACACdVau6z/e2WS9yQ5MMkVVbVHkvR/rxxfCQGAaSIEAgCYQFW1Y1XtNDOc5DeTnJvk1CSH97MdnuS94ykhADBtXA4GADCZdk/ynqpKujbb21trH6qqzyQ5uaqOSHJJksePsYwAwBQRAgEATKDW2oVJ7j3H+O8mecjWLxEAMO1cDgYAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGYN24CwDA6tlw1AfGXYRlufiYR427CAAAsM3TEwgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAGCCVdV2VfW5qnp//3zfqjqrqs6vqpOqavtxlxEAmA5CIACAyfbMJOeNPH95kle31vZLclWSI8ZSKgBg6giBAAAmVFXtleRRSd7YP68kD05ySj/LCUkOHU/pAIBpIwQCAJhcr0ny3CQ/65/fPsnVrbXr++eXJtlzHAUDAKaPEAgAYAJV1aOTXNlaO3t09ByztnmWP7KqNlXVps2bN69JGQGA6SIEAgCYTPdP8ttVdXGSd6S7DOw1SXauqnX9PHsluWyuhVtrx7bWNrbWNq5fv35rlBcAmHBCIACACdRae35rba/W2oYkhyX5aGvt95J8LMnj+tkOT/LeMRURAJgyQiAAgOnyvCTPrqoL0t0j6LgxlwcAmBLrFp4FAIBxaq2dkeSMfvjCJAeOszwAwHTSEwgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAFYMASqqjdV1ZVVde7IuF2r6rSqOr//u8vaFhMAAACAlVhMT6Djkzx81rijkpzeWtsvyen9cwAAAAAm1IIhUGvtE0m+N2v0IUlO6IdPSHLoKpcLAAAAgFW0bpnL7d5auzxJWmuXV9Ud5puxqo5McmSS7LPPPst8OSbNhqM+MO4iLMvFxzxq3EUAAACAsVjzG0O31o5trW1srW1cv379Wr8cAAAAAHNYbgh0RVXtkST93ytXr0gAAAAArLblhkCnJjm8Hz48yXtXpzgAAAAArIXF/ET8iUk+leSuVXVpVR2R5JgkD62q85M8tH8OAAAAwIRa8MbQrbUnzTPpIatcFgAAAADWyJrfGBoAAACA8RMCAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMADrxl0A2Jo2HPWBcReBKeFYAQAAtjV6AgEAAAAMgBAIAGACVdUOVfXpqvp8VX2pql7cj9+3qs6qqvOr6qSq2n7cZQUApoMQCABgMv1Pkge31u6dZP8kD6+qg5K8PMmrW2v7JbkqyRFjLCMAMEWEQAAAE6h1ruuf3qJ/tCQPTnJKP/6EJIeOoXgAwBQSAgEATKiq2q6qzklyZZLTknw9ydWttev7WS5Nsuc8yx5ZVZuqatPmzZu3ToEBgIkmBAIAmFCttZ+21vZPsleSA5Pcfa7Z5ln22NbaxtbaxvXr169lMQGAKSEEAgCYcK21q5OckeSgJDtX1bp+0l5JLhtXuQCA6SIEAgCYQFW1vqp27odvleTgJOcl+ViSx/WzHZ7kveMpIQAwbdYtPAsAAGOwR5ITqmq7dCfuTm6tvb+qvpzkHVX10iSfS3LcOAsJAEwPIRAAwARqrX0hyQFzjL8w3f2BAACWxOVgAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABiAdeMuwGrZcNQHxl2EZbn4mEeNuwgAwASqqr2TvCXJzyf5WZJjW2uvrapdk5yUZEOSi5M8obV21bjKCQBMDz2BAAAm0/VJntNau3uSg5I8varukeSoJKe31vZLcnr/HABgQUIgAIAJ1Fq7vLX22X742iTnJdkzySFJTuhnOyHJoeMpIQAwbYRAAAATrqo2JDkgyVlJdm+tXZ50QVGSO8yzzJFVtamqNm3evHlrFRUAmGBCIACACVZVt0nyriTPaq1ds9jlWmvHttY2ttY2rl+/fu0KCABMDSEQAMCEqqpbpAuA3tZae3c/+oqq2qOfvkeSK8dVPgBgugiBAAAmUFVVkuOSnNdae9XIpFOTHN4PH57kvVu7bADAdNpmfiIeAGAbc/8kT07yxao6px/3giTHJDm5qo5IckmSx4+pfADAlBECAQBMoNbaJ5PUPJMfsjXLAgBsG1wOBgAAADAAQiAAAACAAVjR5WBVdXGSa5P8NMn1rbWNq1EoAAAAAFbXatwT6EGtte+swnoAAAAAWCMuBwMAAAAYgJX2BGpJPlxVLcm/tNaOnT1DVR2Z5Mgk2WeffVb4ctueDUd9YNxFAAAAAAZgpT2B7t9au0+SRyR5elU9YPYMrbVjW2sbW2sb169fv8KXAwAAAGA5VtQTqLV2Wf/3yqp6T5IDk3xiNQoGAAAwTnrtA9uaZfcEqqodq2qnmeEkv5nk3NUqGAAAAACrZyU9gXZP8p6qmlnP21trH1qVUgEAAACwqpYdArXWLkxy71UsCwAAAABrxE/EAwAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADMC6cRcAADYc9YFxF2FQLj7mUeMuAgAAY6AnEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAEygqnpTVV1ZVeeOjNu1qk6rqvP7v7uMs4wAwHQRAgEATKbjkzx81rijkpzeWtsvyen9cwCARRECAQBMoNbaJ5J8b9boQ5Kc0A+fkOTQrVooAGCqCYEAAKbH7q21y5Ok/3uH+WasqiOralNVbdq8efNWKyAAMLmEQAAA26DW2rGttY2ttY3r168fd3EAgAkgBAIAmB5XVNUeSdL/vXLM5QEApogQCABgepya5PB++PAk7x1jWQCAKSMEAgCYQFV1YpJPJblrVV1aVUckOSbJQ6vq/CQP7Z8DACzKunEXAACAm2utPWmeSQ/ZqgUBALYZegIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYACEQAAAAAADIAQCAAAAGAAhEAAAAMAACIEAAAAABkAIBAAAADAAQiAAAACAARACAQAAAAyAEAgAAABgAIRAAAAAAAMgBAIAAAAYACEQAAAAwAAIgQAAAAAGQAgEAAAAMABCIAAAAIABEAIBAAAADIAQCAAAAGAAhEAAAAAAAyAEAgAAABgAIRAAAADAAAiBAAAAAAZACAQAAAAwAEIgAAAAgAEQAgEAAAAMgBAIAAAAYABWFAJV1cOr6qtVdUFVHbVahQIAYH7aYADAciw7BKqq7ZL8c5JHJLlHkidV1T1Wq2AAANycNhgAsFwr6Ql0YJILWmsXttZ+nOQdSQ5ZnWIBADAPbTAAYFnWrWDZPZN8c+T5pUnuO3umqjoyyZH90+uq6qsreM1ps1uS74y7EBPAdriRbdGxHW5kW3Rshxut+baol6/l2nOnNV07yeS1wYbw/h1CHRP13JZMZR2X8f00lfVchiHUcwh1TNa2notqg60kBKo5xrWbjWjt2CTHruB1plZVbWqtbRx3OcbNdriRbdGxHW5kW3RshxvZFizCRLXBhnDMDqGOiXpuS4ZQx0Q9tyVDqGMyGfVcyeVglybZe+T5XkkuW1lxAABYgDYYALAsKwmBPpNkv6rat6q2T3JYklNXp1gAAMxDGwwAWJZlXw7WWru+qp6R5D+SbJfkTa21L61aybYNg7wMbg62w41si47tcCPbomM73Mi2YIsmsA02hGN2CHVM1HNbMoQ6Juq5LRlCHZMJqGe1drNLyAEAAADYxqzkcjAAAAAApoQQCAAAAGAAhECLVFUPr6qvVtUFVXXUHNNvWVUn9dPPqqoN/fiHVtXZVfXF/u+DR5Z5Uj/+C1X1oarabevVaPlWsC0OrKpz+sfnq+oxi13nJFrt7VBVe1fVx6rqGTVT5QAAC8NJREFUvKr6UlU9c+vWaPnW4pjop29XVZ+rqvdvnZqszBq9N3auqlOq6iv9sXG/rVej5VujbfEX/Xvj3Ko6sap22Ho1Wp7lboeR6ftU1XVV9ZeLXScspKp2rarTqur8/u8u88x3eD/P+VV1+Mj4X+7bLxdU1f+pqtrSeqvqblX1qar6n9FjuZ825/Fc3U2vz+rXdVJ1N8Ce9HpWP98F1bXt7tOPf9DI59o5VfX/qurQftrxVXXRyLT9p7GO/bSfjtTj1JHx29K+3L8/lr/Uj3/iyGssa1/O9x4YmT7v90RVPb8f/9WqethC65xvXyznNZZqQur57Kr6cr/vTq+qO40sM+fxO6X1fEpVbR6pz9NGlpnzPTKFdXz1SP2+VlVXjywzjfvyGf24ViOZQHXm+8xd/r5srXks8Eh308WvJ7lzku2TfD7JPWbN86dJXt8PH5bkpH74gCR37IfvleRb/fC6JFcm2a1//ookR4+7rmu8LW6dZF0/vEdf/3WLWeekPdZoO+yR5D79+J2SfG3St8NabYuR5Z6d5O1J3j/ueo5rOyQ5IcnT+uHtk+w87rqOY1sk2TPJRUlu1U87OclTxl3XtdoOI9PfleSdSf5ysev08Fjoka7NcVQ/fFSSl88xz65JLuz/7tIP79JP+3SS+yWpJB9M8ogtrTfJHZL8SpKXzRzL/fh5j+f+PX5YP/z6JH8yBfV8ZD9fJTkoyVnzvN73kty6f358ksdN0b6ct45JrpunjNvMvkzyi0n264fvmOTy9N/Ly9mXWdn35T36+W+ZZN9+PdttaZ3z7YulvsYy9t+k1PNBufG99ycZ+c7NPMfvlNbzKUn+aSnvkWmr46zX+7N0P5AwzfvygCQbklycPh9Y4LNoRftST6DFOTDJBa21C1trP07yjiSHzJrnkHT/rCXJKUkeUlXVWvtca+2yfvyXkuxQVbdMtyMryY5VVUlum+SyTL6VbIsfttau78fvkKQtYZ2TZtW3Q2vt8tbaZ/vha5Ocl+4f30m3FsdEqmqvJI9K8sY1Lf3qWfXtUFW3TfKAJMclSWvtx621qzP51uSYSBcG3aqq1qULiyb9M3PZ2yFJquspcGG6746lrBMWMnrcnZDk0DnmeViS01pr32utXZXktCQPr6o9kty2tfap1rVE3zKy/Jzrba1d2Vr7TJKfzHqNOY/n/j3w4HTviS2VcaLq2Y9/S+ucmWTnfj2jHpfkg621Hy6jPnOZxDreYFvbl621r7XWzk+Svn1/ZZL1y6jPjJV8TxyS5B2ttf9prV2U5IJ+fct5Xy31Naaynq21j428985Mstcy6jLx9dyCOd8j20Adn5TkxCXWYyFbrZ5J0mcGF89Rjvk+c1e0L4VAi7Nnkm+OPL80N//n/IZ5+n9gvp/k9rPmeWySz/UHxE/SJdBfTPePzD3S/6M34Va0LarqvlX1pXT1/uN++mLWOWnWYjvcoO9OeECSs9ag7KttrbbFa5I8N8nP1q7oq2ottsOdk2xO8ubqLot7Y1XtuLbVWBWrvi1aa99K8o9JLkl35vX7rbUPr2ktVm7Z26Hfz89L8uJlrBMWsntr7fKkOwGRrqfObPMda3v2w7PHL3a9i3mN2ye5euT7YLnH+dau52Len4fl5v+svKzv5v/q/kThUkxSHXeoqk1VdWYfYifb8L6sqgPTnc3/+sjope7LlXxfbqm+S31fLfU1lmpS6jnqiHQ9LGbMdfwu1STV87H9sXhKVe29hPItZJLqmOou6ds3yUdHRk/bvlxOOVa0L4VAi1NzjGtLmaeq7pnk5Un+qH9+i3Qh0AHpupN+IcnzV6Owa2xF26K1dlZr7Z7puoU/v7p7eixmnZNmLbZDt1DVbdJdAvKs1to1q1TetbTq26KqHp3kytba2atb1DW1FsfEuiT3SfK61toBSX6Qrrv6pFuLY2KXdGdD9k33mbljVf3+KpZ5LaxkO7w4yatba9ctY52QqvpIdffPmv1YbM+x+Y611TwGV/waE1bPhdqCeyT5pST/MTL9+Unulu7zbtd04e9NVzo9ddyntbYxye8meU1V3WUprzFF9ZzZl/+a5KmttZmTVQvuy6W+zgLzrNb45bzGUk1KPbsX6toPG5P8w8jouY7fpZqUer4vyYbW2v9K8pHc2FtlNfbnpNRxxmFJTmmt/XRk3LTty+WUY0X7Ugi0OJcm2Xvk+V65+WUIN8zTX6pwu3TXfM9c1vKeJH/QWps5W7B/krTWvt53TT05ya+uVQVW0Yq2xYzW2nnp/qG91yLXOWnWYjvMhIPvSvK21tq716Tkq28ttsX9k/x2VV2crqvkg6vqrWtR+FW0Vu+NS1trMz3CTkkXCk26tdgWBye5qLW2ue9J+e5M/mfmSrbDfZO8on8PPCvJC6rqGYtcJ6S1dnBr7V5zPN6b5IqZS3j6v1fOsYr5jrVLc9NLKEaPwcWsdzGv8Z10Xd7XzfEak1zPhd6fT0jynv4zbKb8l/fd/P8nyZszx6U201LH1t/+oLV2YZIz0p3o3Ob2ZXWXan8gyV/3l2fMlH/BfbmEMs85z6zviS3Vd6nvq6W+xlJNSj1TVQcneWGS3+73VZJ5j9+lmoh6tta+O1K3NyT55SWUbyETUccRN+tdOYX7cjnlWNm+bCu8adIQHunOxl+Y7gz0zE2c7jlrnqfnpjeGOrkf3rmf/7Gz5p+5mdz6/vnfJnnluOu6xtti39x4w9c79QfqbotZ56Q91mg7VLrr1F8z7vqNe1vMWvaBmY4bQ6/Jdkjyn0nu2g8fneQfxl3XcWyLdKHIl9LdC6jSndX6s3HXda22w6x5js6NN4aeus9Lj8l7pDv7PXoz3FfMMc+u6W7Gvkv/uCjJrv20z6S7QWWlu5zikYtZ7+ix3D+f93hOd0P00ZuB/umk1zPdfexGb+D56VmvdWaSB80at0f/t9JdBn3MNNaxX+8t++HdkpyfG294us3sy/44PT1db+3Zr7/kfZmVfV/eMze9+eyF6W48u+T31VJfYxn7b1LqeUC6y/f2m/Xa8x6/U1rPPUZe7zFJzlzoPTJtdeyf3zXdjZRr2vflyDovzk1vDD3fZ9GK9uWSNsaQH+nuzP21dB8cL+zHvSRdipz/v737d5HiDOMA/n2CcI2kMCApg5VligMRUlydKgmkUzi0SWWbxkIr/wHtrrBId40EUgRyCaRTRE7vRAXtYpEuWItj8Y64LuGO271lZ28+HxhYdoZn5533nV/PvPNu2gCm22kDPz1Icq7//nrak+zdielsP++ntMF/n6R12/ti2eVc8La4nHYTt5vkUZLvDoo59Om4t0OSb9K68T2ZaCvfLrucy2oTE7E3sgJJoEVth7Regw/7dnEvR/wXhxO2LW4meZ5kP60b/tqyy7mo7TAV40Y+vXFeueOlaVhT2pgFO2kXxzv5eKO8nmRrYrkrfdt8mfbaSyaW2+/b4O30F+AHxP0y7anlmyT/9Z8/7+f9b3tOGxPtQf/b27Ps70soZyW50y+/l2R9ItZXSV4n+WxqHf/sl91P8kuS06tYxrSemXtpNzV7Sa6exLpMciltgPPJ6/qv56nLzHGeSOvR8irJi/T/eDbLfjXLb8xQh0Mo5x9J/p2ou18Pa78rWs5baddSj5P8leT8YfvIqpWxn3cjU8nWFa7La2nnxrdpD0C3DjoWzVuXHw6AAAAAAJxgxgQCAAAAGAFJIAAAAIARkAQCAAAAGAFJIAAAAIARkAQCAAAAGLiq+rGqnlbVu6panyWGJBAAAADAgFTVRlXdnfp6P8kPSf6eNe6peVYKAAAAgMXruu5ZklTVzDH0BAIAAAAYAT2BAAAAAAagqu4nWUtyOsmZqtrtZ/3cdd3v88aXBAIAAAAYgK7rLiRtTKAkm13XbR5nfK+DAQAAAIyAJBAAAADAwFXV91X1T5KLSX6rqiO/HlZd1x3/mgEAAAAwKHoCAQAAAIyAJBAAAADACEgCAQAAAIyAJBAAAADACEgCAQAAAIyAJBAAAADACEgCAQAAAIzAe2XK4hl5JrIvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1440x1080 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(2,2, figsize=(1,15))\n",
    "\n",
    "ax[0,0].hist(shallow_pytorch_stds, bins=10, density=False);\n",
    "ax[0,0].set_title('Shallow Pytorch Network Standard Deviation');\n",
    "\n",
    "ax[0,1].hist(shallow_orthnormal_stds, bins=10, density=False, range=[1-0.0001,1+0.0001]);\n",
    "ax[0,1].set_title('Shallow Orthnormal Network Standard Deviation');\n",
    "\n",
    "ax[1,0].hist(deep_pytorch_stds, bins=10, density=False);\n",
    "ax[1,0].set_title('Deep Pytorch Network Standard Deviation');\n",
    "\n",
    "ax[1,1].hist(deep_orthnormal_stds, bins=10, density=False,  range=[1-0.0001,1+0.0001]);\n",
    "ax[1,1].set_title('Deep Orthnormal Network Standard Deviation');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}