Attempt to reproduce Part 1: Baseline of "How to train your resnet" with fastai v1

how_to_train_your_resnet_fastaiv1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "how_to_train_your_resnet_fastaiv1.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OKqJD7RwMOKQ",
        "colab_type": "text"
      },
      "source": [
        "# \"How to train your ResNet: Part 1\" in fastai v1\n",
        "\n",
        "The goal of this notebook is to reproduce the results of the great article series called \"How to train your ResNet\" [Part 1](https://myrtle.ai/how-to-train-your-resnet-1-baseline/) by myrtle.ai. If all goes well, I might implement the entire article series, but we are starting with Part 1 for now.\n",
        "\n",
        "You can find the code for the experiments corresponding to the diffferent parts of the article series here: https://github.com/davidcpage/cifar10-fast/blob/master/experiments.ipynb."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HBNI1f-vCG_P",
        "colab_type": "text"
      },
      "source": [
        "# Setup\n",
        "\n",
        "Install & import fastai and download the CIFAR-10 dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "31Pw8JjsJP43",
        "colab_type": "code",
        "outputId": "33e8f1c8-4d3a-45e0-8b97-de86af1f7738",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 714
        }
      },
      "source": [
        "!pip install fastai"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: fastai in /usr/local/lib/python3.6/dist-packages (1.0.57)\n",
            "Requirement already satisfied: numpy>=1.15 in /usr/local/lib/python3.6/dist-packages (from fastai) (1.16.5)\n",
            "Requirement already satisfied: fastprogress>=0.1.19 in /usr/local/lib/python3.6/dist-packages (from fastai) (0.1.21)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from fastai) (19.1)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (from fastai) (0.3.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from fastai) (0.24.2)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from fastai) (3.0.3)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.6/dist-packages (from fastai) (4.3.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from fastai) (2.21.0)\n",
            "Requirement already satisfied: typing; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from fastai) (3.7.4.1)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from fastai) (0.6)\n",
            "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from fastai) (1.1.0)\n",
            "Requirement already satisfied: bottleneck in /usr/local/lib/python3.6/dist-packages (from fastai) (1.2.1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from fastai) (1.3.1)\n",
            "Requirement already satisfied: nvidia-ml-py3 in /usr/local/lib/python3.6/dist-packages (from fastai) (7.352.0)\n",
            "Requirement already satisfied: spacy>=2.0.18 in /usr/local/lib/python3.6/dist-packages (from fastai) (2.1.8)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from fastai) (3.13)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.6/dist-packages (from fastai) (4.6.3)\n",
            "Requirement already satisfied: numexpr in /usr/local/lib/python3.6/dist-packages (from fastai) (2.7.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from packaging->fastai) (1.12.0)\n",
            "Requirement already satisfied: attrs in /usr/local/lib/python3.6/dist-packages (from packaging->fastai) (19.1.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->fastai) (2.4.2)\n",
            "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas->fastai) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas->fastai) (2.5.3)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->fastai) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->fastai) (0.10.0)\n",
            "Requirement already satisfied: olefile in /usr/local/lib/python3.6/dist-packages (from Pillow->fastai) (0.46)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->fastai) (2.8)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->fastai) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->fastai) (2019.6.16)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->fastai) (1.24.3)\n",
            "Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (2.0.1)\n",
            "Requirement already satisfied: srsly<1.1.0,>=0.0.6 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (0.1.0)\n",
            "Requirement already satisfied: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (0.2.4)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (2.0.2)\n",
            "Requirement already satisfied: thinc<7.1.0,>=7.0.8 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (7.0.8)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (0.2.2)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (1.0.2)\n",
            "Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.18->fastai) (0.9.6)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib->fastai) (41.2.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /usr/local/lib/python3.6/dist-packages (from thinc<7.1.0,>=7.0.8->spacy>=2.0.18->fastai) (4.28.1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_NP5L7hrCF1D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%reload_ext autoreload\n",
        "%autoreload 2\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fTRD0vpIDQF-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fastai.vision import *"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uxPKhALbChdZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "path = untar_data(URLs.CIFAR)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zjx0ruKgj4HM",
        "colab_type": "text"
      },
      "source": [
        "# Transforms\n",
        "\n",
        "The below is supposed to give us the same transforms as in the Part 1: Baseline  experiment, namely `[Crop(32, 32), FlipLR()]`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AHUznZRXiNxm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "crop = rand_resize_crop(32, ratios=(1.,1.))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sY1zckY9kZnU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "268cf74c-280f-40ed-dd1d-7bbc52981673"
      },
      "source": [
        "crop"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[RandTransform(tfm=TfmCoord (zoom_squish), kwargs={'scale': (1.0, 2.0, 8), 'squish': (1.0, 1.0, 8), 'invert': (0.5, 8), 'row_pct': (0.0, 1.0), 'col_pct': (0.0, 1.0)}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True),\n",
              " RandTransform(tfm=TfmPixel (crop), kwargs={'size': 32}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PJSHFbUUu_8j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tfms = get_transforms(max_rotate=0., max_zoom=0., max_lighting=0., max_warp=0., xtra_tfms=[*crop])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E8VGEK7BkIXO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "8b50c5b2-6e0c-4521-d1d0-3ef9ad5f3159"
      },
      "source": [
        "tfms[0] # train set transforms"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[RandTransform(tfm=TfmCrop (crop_pad), kwargs={'row_pct': (0, 1), 'col_pct': (0, 1), 'padding_mode': 'reflection'}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True),\n",
              " RandTransform(tfm=TfmPixel (flip_lr), kwargs={}, p=0.5, resolved={}, do_run=True, is_random=True, use_on_y=True),\n",
              " RandTransform(tfm=TfmCoord (zoom_squish), kwargs={'scale': (1.0, 2.0, 8), 'squish': (1.0, 1.0, 8), 'invert': (0.5, 8), 'row_pct': (0.0, 1.0), 'col_pct': (0.0, 1.0)}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True),\n",
              " RandTransform(tfm=TfmPixel (crop), kwargs={'size': 32}, p=1.0, resolved={}, do_run=True, is_random=True, use_on_y=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdVDoIOYlV4Q",
        "colab_type": "text"
      },
      "source": [
        "TODO: check if the transforms are actually identical"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Em3uv7BzlYgp",
        "colab_type": "text"
      },
      "source": [
        "# create databunch"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6QIGq9O93XYo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 128"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ZGZf2KEix88",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data = ImageDataBunch.from_folder(path, valid='test', bs=batch_size, ds_tfms=tfms)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v_Lbl0l2z5qn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cifar_stats = ([0.491, 0.482, 0.447], [0.247, 0.243, 0.261])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "odwwR8Rj0ECj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "outputId": "7681fddc-33f3-4186-9d3c-b7f473022a85"
      },
      "source": [
        "data.normalize(cifar_stats)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ImageDataBunch;\n",
              "\n",
              "Train: LabelList (50000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "truck,truck,truck,truck,truck\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Valid: LabelList (10000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "truck,truck,truck,truck,truck\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Test: None"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TGnWaj3JKr9p",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 369
        },
        "outputId": "1e4f2c70-8a16-4df3-c3b8-0be3f00356d5"
      },
      "source": [
        "data.show_batch(rows=3, figsize=(5,5))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAAFgCAYAAADpZ/FJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvWm0JdlVHvjtiLjzm1/OWZVZpSqV\nhpKEECrEYGymRm4MjSw30wJs0WA3a4GBprEFNk1jFjQYN21g0d14YWiayQgJDGJopgbcCEoSmqUq\nqVRVWZmVleOb351v3IjTP/becc697+bLzLpVL9/N2t9ab8W7ESfGcyLOt2dyzsFgMBgMzx/Rnb4A\ng8FgmHXYh9RgMBimhH1IDQaDYUrYh9RgMBimhH1IDQaDYUrYh9RgMBimxEviQ0pEP0REv7rP9seI\n6AsP8JIMzwNE9EtE9CN3+joM04GIzhPRl05Y/wVE9MRtHutQjInkTl/AYYBz7uE7fQ0Gw0sdzrm/\nAvCKO30dzwcvCUZqMNwIRGRkYgZw2PvprvuQEtHbiegSETWJ6Aki+hLZVCaiX5b1jxHRG4N9ClFD\n1ADvIqJ3SNsPEdFn3JGbeYmDiD5Tnn+TiN4BoBps+woi+ggRbRPR3xDR64Jtp4jot4hojYieIaLv\nDLZp//4qEe0CeNuB3pRB8QgRPU5EW0T0fxFRlYi+kIie0wbyXr6diD4GoE1EyX5j4k7irvqQEtEr\nAHwHgEecc/MA3gzgvGz+bwD8BoAlAO8G8LP7HOqrALwTwAqAXwfwO0RUepEu2zABRFQG8DsAfgXc\nD+8E8I9k22cC+EUA/z2AVQD/AcC7iahCRBGA3wPwUQCnAXwJgO8mojcHh/8qAO8Cj4VfO5AbMozj\nG8Dv5wMAHgLwAzdo9/UA/gG4ryLcYEzcadxVH1IAGYAKgFcTUck5d94597Rse49z7g+dcxm4I/Zj\nmR90zr3LOZcC+N/As97nvKhXbhjH5wAoAfgp51zqnHsXgL+Vbf8MwH9wzr3POZc55/5vAH3Z5xEA\nR51zP+ycGzjnzgH4eQBfFxz7Uefc7zjncudc9+BuyRDgZ51zF51zmwB+FPzBnISfkXZd7D8m7igO\ntd7hduGce4qIvhvADwF4mIj+GMD3yOarQdMOgCoRJc654YRDXQyOmYu4cepFumzDZJwCcMmNZtW5\nIMuzAP4JEf3zYFtZ9skAnCKi7WBbDOCvgt8XYbjTCPvgAm78foXt9hsTdxR3GyOFc+7XnXN/B/yy\nOQD/9nkc5l79R0TFewBcfmGu0HCLuALgNBFRsO6MLC8C+FHn3FLwV3fO/SfZ9szYtnnn3JcHx7GU\nZ3ce9wb/n8GN36+wr/YbE3cUd9WHlIheQURfTEQVAD0AXQD58zjUZxHRW8VS+N1gsfG9L+ClGm6O\nRwEMAXwnEZWI6K0APlu2/TyAbyOiNxGjQUT/gIjmAbwfQFOMFDUiionoNUT0yB26D8NkfDsR3UNE\nKwD+NYB33MI++42JO4q76kMK1o/+OIB1sCh/DMD3P4/j/C6ArwWwBeCbALxV9KWGA4JzbgDgrWCr\n+ia4P35btn0AwD8FGwy3ADwl7SA68K8A8HoAz4DHwn8EsHiQ12+4KX4dwJ8AOAfgaQA3darfb0zc\naZAldh4FEf0QgAedc994p6/FYDDMBu42RmowGAwHDvuQGgwGw5Qw0d5gMBimhDFSg8FgmBIH6pD/\n4Dd/rwOAXp4V61LxCMuIv+kZeEWSe1exXFhzT1zKtA1D/s/Zy8nlw5F9wiZuyG1iWR0Fx8kyvqZi\nrzzYv/jfyeF4v5DNs7HYt4noxnNU7vRa915j9vu/SBN2OZT41rf/MwcAO62dYt0gZecGfTRJiSNr\nK2UfEl0ulXldwuvi4DmsLs0DAE6dZP/sTncAANjYaRZtUulrSvhRZVmff5OPrYhiPmaayngIxly5\nzNcUR9JHMtZKcbloU60mci4+djb0/TlI+ViRuDNWKrHca1y0SQd8jT/8XT86M/35+m/4GX5oE4XU\n8ds4vLc16mb6wuLDv/btEw9ujNRgMBimxIEy0mrMM/Yw88xhKNQlhy4hy+AbLzMMCRMpOb+NZP9M\njpkJ8wgnJTc2xSpbygJGmY/NwlF4AGW00t7ppqBNpNcrbHPk/HoeWZLTuwxOOoOq6p3dFgAgzbyL\nrTLxKBF2Jw9rOPBxEW7IfTV0bQDAQsXng1mtNQAAVXme17d3AQC7u56R5rGMh4iXyjZLJT8uHPH5\nhiKFVEr+HHMVZr3lhMdjt9sDAHS6Puy+05PrleMkSb3YFkcVOTbfR6vFrLVc9oyUZjH6WsYgBWxT\nh6VfQ3vW3BIOgMD6d+7GJ9t7Py8MjJEaDAbDlLAPqcFgMEyJA5U/chX7okAEEiOPiuQDWe98EySy\nn4r2GHoxkYSrR7kai7xRoTjvmIuXGzMeyRH4eCofBLK5ivJOBAJtE9qTSPaf5E5WSPZy3uLQMyjO\nj0BUFOWyvxFV2/Q6LO6L7QlJ4g05tWoNAFCp8PLowlKx7fj8HO8/YHG502FxezD0/VqInkO9DFG5\n5IH6IBKDkIyjSmAIKkuHzsUsoscl/t1ue9G+O+SRqH0cpX5bHPNNZZmcz+nY8+qDUhwM4BkBTeBV\ne01MYmgNV6pKYB95ueizfQ1Bo++lu235W1/UG79gkw45bpzS35Pu8UYwRmowGAxT4kAZaUfYZxR8\nv0uRuCTJ5K6kYhAkbSpciTI1IARuLrJUg5ITljQyQwxHWW8xvUSBsShmNkEUy3XsTRoVRaMz3Yj7\nk64bc5GSg+oBRo73InppHAgadXZfiir+vnp9Zmu9TgcAkGX8PJLYDzV1OysLS11ZXim2rSwxO710\nbQ0AMFT3pSyQQgrDnq7gf+LggVaqfOxhxswyGwyKbUMVdyJuE08ysohBU88VSlHFdRTSixpRPWt2\nGOxpf9hBt2GCoQk/XjBDDo0sJh57f4J4c1Hvtl2kaP9jGiM1GAyGKXGgjLTV51m6GvvvdxwJmxDd\nZklm9zTwR8qVUQ5VF+W3ZUpOVE8lrjhRFrBF1a8Jy4zUFSaYlWJ1pSl8oyYkzo9UDyqHcwFrvQV/\nZSp0rHu3zaK6VNl/Fugvy/Js5+bZXahcFveh2DvkV8rMBPXxdTq9Yltf9N9RxEOzJNJImTwjdFJQ\nMpM+V/enkBUs1ln/2uvLPqm/Rh1jser7ZKzkwZhRJlot1+R6/PlVWiFSKYSPPQwyLQ6HM8hIJwzM\n4oncQij5vvrHMZY56Wh+mzZ2e7ZN+u3G/nneDvlTiIjGSA0Gg2FKHCgjTYX1hbqkSPSdym503g9V\nlIV1VBlloNsswi01xFOYbB60IQkJJGUVEpo4MstqtOCA2REFc6buV9gUlUoF+0fKUnQfCq9RKewE\nR3zceNVhR6fFluwheUa5sMjO7nVhhKVEQ2a9Rbss1vJ8wNuurm8W2+pVDSll1lqrcV+pFR4A1Le/\nK+NCn2+o4yuL1bzS4OuJhv4BH19c5WuL+Fz97W25xsCbRD0tMtF/BvsPVUIqwoqFmZMftPkE75FD\nj4lS1D4bb+WQN2B5+60lTy33tJjkFXNj3W5w9S+U4vYGMEZqMBgMU8I+pAaDwTAlDlS07+dqrAm+\n3/J/pG5Lk+LQ81HxXQ0RABDL/ySZezRGP3TC1zh+EsOF0/kjUDFAjFRqeBgh8mr4KmLlVVwNRId8\ndFu4v7ZTlQBNyP7k3F53q8OOdKBqFN+f6pmm/ugk2+LIi/ZJiUV7J8+lm/aLbeeuXAEArCxyiaVS\nhft1qeSfaKvNqoT+mKoklPp6PTb2zNfZ6FWv+oCAYytH+fr7fLFrEs8fXqM69+9sNeV+vNivon0q\n0QaqXqI4GA/JDHOUUHou/OgPIlhel/s41O97HS/eNd7syDPc2waDwXA4cKCMdFHcQ3oBERwoOxU2\noEp6wt6inYmwgihgQBp2quQkSzV8L3Rf4vkkjjUzkTpbB6xV82iO7KEbR0MQi1DE0Ode85EKOwkN\naoVzv48VHTnu3hPOBijjPgtdg1zO60pVcdYXV6WYKkWbSplZokom3Y5n47sDduRPt7cAAEeWlgEA\ny7U5f15hhP2eMn1eDga+z9c3OEdqTwxiR+d8GGqzxgy00+Zz7W7vyP6eGXclNLXT5nXz874IaSSv\nTdpnZtwTt76wO0tlf7+zgxu75/km+/ju3Ur056Q2e4jnjY1N+78mo5mp3ARTmXednEC7x09/GxZg\nY6QGg8EwJQ6Ukf53X/xGAMB6y8/8F9ZZB/XUlesAgK0ms4TBiP9T4QEPAEgSf9k10X3Vyjwn1MXV\nqR4wgkT0qANhiVebzEjWt31m91RdV8RtKgozJsh5i3DUCTqcYqajsSWwR8+krkBhFv0D0UG9wChL\niGUv987nmkmeIOGX8uzDEFHNDZrJc20FzzERJpvJuq0m5yzNA7ZZkrFRF5VmQ9yp2n3vhtUUHWkk\niVHmGj6f6DCV/KNtHnutDudV3el2/H2I1DO3wMeuBDlT8yH/3+vKfrvMXkP+Uq7Mnj/bxBF4Czk+\nb0Wa0uE9kZiO5xopkoaEDvmjz3M03/D4hegyyE879n660Nlfw4HHu2zkFbYQUYPBYHhRYR9Sg8Fg\nmBIHKtp/9ee/AQDQGnhDzNVtFos+9exlAMBHPv1pAMCHLzxbtOlLnPS9SxyR8ur77iu23XPiGIBQ\ntOdbmpvzopwW1ttssUj3Fx95DACwtblRtBlI0Tx1kYrCciYaQZOPiv9xYPQq4ujzUTUEAGQYzxHA\nxhkKs0eN1zqZAVRL/Kx3mtvFulyyeVUksknj110Q9UMlFt9LIu5ryQ8AIPlf83nmkk3q2nZwDhGp\n58R4uCLlSZaSIHpKjFNLdY5sam15NU7TcV/3xSDZFyNomnv1QSIx9nMLvEwC96fNdVYbdLqyn7p8\nBTlPMSFb1KFHpJFFQRYs/acI9dobB39LqZk0h+9+RqdxEX+C0F64uk3YvRDXx9wUwxNrFBrC8ai5\niIvl3vPfLDmqMVKDwWCYEgfKSE8uMEvMAkPS6QbP+Pcv8raXH2UGcf+J5aLNjripnD12AgCw2mgU\n2yKZdVJpkzeZdXZbQbYedcXJ+HZXxXBwMmCtzZ4YGmQSSoJ5KZc467Rwe9JZOciZKswrE6NIFDrb\nq/JcGJQT1psGeVW1jPEsIRIH9GrFP+uS5CYdSs6CXJh9qeyftRryXKaBFv5Z9cUQlArbH4gbUi8w\nBEkVZsyLY3+1yuNhvu7HxTyU0UqBunar2NYR17pUrtuJ1aoWGP96Pd6vLe5P1Zrvn346kHvlC6mT\n5AMYMRjOnoQxCXsyMilChjae2Wmih9SoS9M+hDRgkrR368Rjq9uW5rvgvooDI6hW1VCXO80gBgBO\nkiFTIWHKcqQExv6c0xipwWAwTIkDZaRDcXgOov1QEp1YWXRR8/V7AQD3HfUO1FfX1gEAsbg0bYr7\nEgBcvMi61Y3L1wAA/V3ZFpykduQIAKAxx5nYTwkDmX/4oaKNOuvrbBgH2aMycaAfiE4tlWWz3S7a\nNLeZMbV2eRnqPHuikxvKjJcKa91teZaUZbPHSDX7UTkog6zPbdjnvlbn9SINPYA8FR1WvNfVLNZs\n8/qMxP3JVzfwGaKGmldUxsXq8qo/hzSPhAltlP01djrs7N9yfI0aMhxX/etQkkS3vQ63CUsw1Rvc\nrlpZluOJFNTzDKg/mL3+LLBfFqhbcXW6jbUjLYrw6tElAOTS1877SBXbtOKC2jUy0aE3t88Xbbo7\n3OcU8XcmrvrvSyz6/LJINEmFpac48S6UNwvgNkZqMBgMU+JAGemOhOvVKz6BhCZ86EvYn+b+PLXk\nQwIbUi5yq8vsIK34maJW5f8jZZQyNSSBJbgq1tRFYTKnRMe6shrWCmJ2oQlFwiQVeuxcLH/9obKl\nZtFmU5Jb7EpAQS/Qf15tscX42haz5evrO3KO4PEns2flJQmMKLtasU5zemr2+ESCD5JAJ1Xoq0Q3\nWgq25RJq2xsKW1RHiaA/lOynWUE7AQDVmh8XA/HQKKkjfd+fY6g66iJyN5frCPLkShhx3pX7KHkd\nb72mVVB5nVZObba8Hndnx0srs4L9gkJuJWDkViJF9z3/WMKfUY3z6NooYKsq7Ghu4/4u57fdvPhE\n0aa1yRJreY4T1jRW7yu2laWvVbKJSzI+b+PajZEaDAbDlLAPqcFgMEyJgxXt26yMb/e96rYlbi1D\ncYVpVMWRe847V1fFTamhZXTzIO6Z/fFRE/EwlbjpctW3iUSZPF9l16oFFe0XfUafhpxjWBRS84KF\nxvare4u2War6gm4n5Fjdbk/u0Rsb5tZYZGg2+dq0mF9S8o+/WvbqjllBVeLY0zSIaXaaD4H7qiHO\n8nNVLxon0o8qmieB83pHMilpQb2SxOWHsdFqSVLn6kTcmMqBG9agzWLeprjFrXe8GqYnrmbq7qJu\nM2F5kEgMaGUJ8MizQLUgYy0RP6z5JQkwqAZO3tEslhq5dfH99g9NI8uJJUPGzp+PpFfTKpdavt2/\nX1Xpt7zPqrPB1kUAwNalp4s26ZDfy9rycQDA/BFvmKws8rpYS7IX+Y5vvdCKMVKDwWCYEgfKSJMS\ns640zPvYYwOUuh/FUGbmDRh5xDP+QBxsd5peqZ9K+OjyChuOGidZmVwLMqJnwigjp0te32r747R6\nPGMNhPmkgZO4sqOuXLe6PTU73qCw29kdOWZv4Pe/tMGuF489fR4AsLHNzDTM44lo9ua0mhhbykFh\nOw0yKIvifnmB3UzqFc/eU2GdkKVKIwAKxlGQE/knH4YMT9iqVEWoifGyGjD8rlgg1rY4DHgj6KuB\n8KpEGWmieW79GUgc+jVDVzcwJPUL53JhtnLeYegAHs1exYMC+9DO8SxOIz9uwcpUNJ3IfkfXjQQ4\naFY2MexFmZcwaMDvV5yuAQBK+XO8DIsyHmO3yuWT7PJYW7nPn0fcnLzblV6NMVKDwWA4MBxsOWZh\nnyuBa9ORVf4/FyZTkm/7INBJXb7C7gzXNtgxv5d616JMnLorkrBiKCyvGegou+J21e/xuk5Xfvc9\ng1B3pU42kN9B+V9hvRoa2JOl/gaATK4py6UO0dDPYecu8fWvt1K5RtX3+HO4GXTgnq9y3yU1f68a\nrKD631pZM+WHc7boySb4uSSim4xSzXovOtPUP6tKom5T3LaqLlepZ4Gac1STpeRBOK/qNiPxlorl\nnHGQ53Yo/Rc3eNnr+jHXb/P4oUiTn/D1DJ1v0+3NHiO9pdSjk0I8bykf6S045I/9F8F/A4byrg2a\nzDq728/5Hbvs2pTkLBX22vy+VQI3y/klkVTnWTcaBSHLhS52vF7cRNo9GcZIDQaDYUocKCPVcNBe\nwC4ycbLvS3bzVBjI+o7XSW0Io+z2JYFFkAm9K+2dMNGS6GHDqpwDYYuaBs9Hb05ImKVVL8nPhhWN\ncpSM8Hof1ZLXDQ7EWr8uTveXNn3ato0dXpfrrCzO5SOp8/LZy5Bf1+zzc16frY9UdaUDYRJtkQIA\noKP66FTT2AWhlUP+X8cBJGFMOfbPuiaJUfT5qxTTb3vdu5bsOiK1ligIfuhJUos+yfllXGhVUQAY\nSsXbspxjfsEzmF25brX69yU0tNX147LTCfSlMwLaR0+/Z3ROTHEvvyZmrx9bMSkNnyYfKRipv55+\nm9+h5tULvLz8qWJb1ucUi5WKVqyVvqv5gJtShaUn1buGIdneSj9az8ntdx9jMEZqMBgMU8I+pAaD\nwTAlDlS0n19kV5jNHZ+9ScU8Ndb0ZTkMuHS1og7xLFqHhgN1XSn8e4WPB95LqAjVz5yWVeb1Yclk\nVQVkIpKG6oNWV9ydJKfljmSf2t7197Gzze4Ym2KI6JW9u08Wi+irGWzEmdgF55/FGW2YsSjd7gSu\nSWNuS2qQa7a9u8rOLv+vKp5SECNfrnEfD0XMrjRGcykAQDnmbbVYKw1w37V7oVsdr1to8PFOH1so\ntrVS7qOrIi5ek74eDP240nGo2Z/CXKflmlZI4HFZl2z6cN640ZJ471nErSmZ9roGuX3S31MhLk+w\nMO7JrC+ifSD+9zavAAC2L50DAOxev1Jsqy2zIanS4CxvkRa+HHj1ytaO5Pkocb8slPx4qC2ujJxf\nvyW3o2ybxffXYDAYDhUOlJGub7Bz9JPnfT2mrR1mJ31xV+qJ03sazCbpUENLedkdhNnKuX2uBiVx\nW9KMPACQSkjgYKAGEMkrGrgvOanXkw3lOoKM7G35X4MHlGWlgYtULiWBqaEzXTBjy7WQhp+q21Ng\nEJvFcswaWtkK2KZmaUqkLHYsblALS54BxLJtZ1dKLQcSRizPoSFhvQm4bTr0fR7JeCjJcyQxVoW1\nwFoSEkrSxQ3nXe7K0g9VyYJfErepOOiDshgth3KONHBn02RVOlZcj7el/SB/ph9+M4NbGYE0oVSy\nGzMSTTqmG6edofWmOGY08htByG4qOWRJcsjOBblnjzz4egBAVUI9E63dFQT+9MTYWZJaXuWgvhaN\nh6s+j3fRGKnBYDBMiQNlpE3RjV25fLlYd+nKdQBAV9iiLkO2pm4ITdGBtQJH+m6R5EIYi7qyBPlA\nOx1mlForaiBth0M/Y5VlgtKggWHgrjMUBpwVyTH5gpKy1+0l4grkEmYyWRj2KK4WpDfiNN+hbzN7\n7ttAlmvtIj+MNJ+sMtGyOEVHYX5YSfaiYaP9oddHk2Sy16QWmYTaBp5FqEGSzizwscsJM5fNjnc5\n6/RZn+0kn2g1yDWq7jGZSh1CH5Oa12vnY/W1uj0/nmLR0UbqnC6uUlmgY83z2Uta4tlmsE6W2h9e\nC3p7rG2PRnVS7aVxJhh8AzIJsEgkuc/cyrFi29Kpl/O2muQUljFImf9OVPoaRMG/o6DibPFaFsxa\nfk9IrHIjGCM1GAyGKWEfUoPBYJgSByraHzvCbgavf82ri3VnzpwFALTF3WVTDBDtQJbLhaq3xSiw\n3fUi+VaL23ek2JoamVScB4DeFiuqIxG3+tvsAjEI4vFVjBjKOQLPJDjS7EAiMohBJQqKYzmJ01YX\nqzBbEUnUEhV6dnUFCUs2YwYh2XJCCWisXIS6mA0Dxf9QjH2JlnOOvdvQMOb26hanmba6QaRQIqqE\nWAxRubhhdbrbRZuBinXSxVvtoBiiREkNNSpOStm4oIif9ofaIzuBqqdS5WusJazOUVXFfBYU2Ov5\n9rMGmmBI2kciD3eUxd6Iwb37TchHOrYuFK0zefeSBrtQzq2cKraVqhy9lksWslwiBqPIf96SufmR\nc+wnthfuT7dhdDJGajAYDFPiQBnp0gK7wKwu+1KoPfG83xVl/tV1ZhUbTZ8/siM65x1hraVOwCSl\npHFZnKq78rv7nM8OozlL1eijhqQscOBOxc3FO8kH6nF1pC+ye/P6MI9mLgYpZTUuyHYUZrEBAteR\ncMKLZ29OU8f6WikofhdrRnnJiCRmtNZIrD1LC2qsiQNDlJPMTk2pdNAVZqrlmQGg4ZgBdrtSFlmz\nSQVZwTSvQlfZSWitEtar5aT1dxy4YVWlaJ6y7SH89ZdlW1lyL1Q0i7/zfVjvesPVrOBWDEmerU3a\nemNj1Y3aTl6jGbsCd7RFNi7F4hJVW/LGJorE1a4QjeRdDk5RyIe0l1pTkeeCRpqYsclgMBgOEAfK\nSLekRO3yvM+ko9OG6s3UNSnMFnRVnPavShjmbuB43SlK80qJ5KscOnbt6SeLNjvrV+U/YaTCiFzg\nIpUW2bH3TrlU1Itx4WJEkeokk5Bmdoqj0CVIjzU+b4VOIbOnJN3YZunh6GJwr+Jm5CRQoiv9uS0M\nEwDaovfUHKNhqlJ1f9JHVZbw0VqQ/V7zhvYy1afKLi5gtk7d4CTAwnnpI5Z6SkN1sVJVd3AdWsZZ\n86O6UsAwZczGmTBRGTNBl6NSPdBX6wXBfiOQbkFJejuhpRMZrRvVqIZSXWPlJAAvxSRBDTAU79do\nGOqk7E0TWfcet6sbrN8HxkgNBoNhShzotPnJ55gZPnSvt7hpLsgrG8xuLq+zRf2Jc+eLNuee5aqA\nm01mpGkw1ZA46Kp1uHmVs2XvXvJhqPmQGWiOUUdqP5MFzsgTcpWOt9mbroGPDvgEGqEz8bhJPnJ6\nrlAHM3su+UURg7Kfj1Vv3BOL+K6w/92gZpJ6VpTFMT6s4ZVn/GzqixzKV2tIXaiyH6qSdhKZWtaL\nQAnfJtKaSRI+OghCjpU15+I1MJDXIA36LB0wzY0wXlkSSLUKrujWcq0nlASeAXOzx0gnYW8e0vF3\nYMI+txliOR4I4D0F/PtRW1iV89LIPkDgUD/+7k64jMmq3dF8qs9HODRGajAYDFPCPqQGg8EwJQ5W\ntL/EhaviQFG8ucMi/TOXOP5+bYt/f+zjnyjaaGz+XJ0V/pWKj5NtNdmI0RGLQy4F7pAFxgUpzaul\nFCgXx/pAmR2LkWgoBqiwVMk4158kuhQZZETFMLJ7pIYsOb9T8SRwAL91T4tDg2UpYliu+WFU2OGk\nWFws7m2VQPyfn+P9FmWpLk6AF69TUXV0JEY6sC+iJGJ+JrH6JJmawiw+taq4rA01v0JQMFG7TwxY\nJA7+eSBKpuJOlxSGpMDYJe5siVTP00xReezVB/EMxtpPi9sR6Se+Q2PrRhNESV/RhG2+0Y3Pd0sX\ndSuNJsMYqcFgMEyJA2WkT4qT/MXnLhTrLl6+BMC70vSFOWysrRVtUnGFWqzzzL9c8yGF6a5kjZKQ\n0EgcsAPfaCSSgaguhotBT4wNAWssCTvpyrk0hynjxqFrBfRgRd7MMHuV/C8M1CvTgylwBh3yTyxx\n2F2j4bPHK+vv9KRonDDDQZBPNBaJIpK8pNXUSxi7KmHs8FJjHoaRZ3hNIbCdOQ4NXF3hDOlx0C1Z\nzhJJRaSYctWfg2SM9MXoVfhjB2JEJBmhqsJ6K9XgHqvMRActuSZx48qCLFYum8Hid7TXkKO4Hef0\nSczw1hjhqBvTZJfAvYak8SCBSZe6x8PJ7d1WrHoeRqfZe3sNBoPhkOFAGenfvu9RAIALXFEGwvzU\nob4njtzhDFgVnVhLaiapQz0A9KVGkpbN1XLK/cClRnWTsdR8qtX4tzqGA0C/yHEqrixBadob55b0\n10j56DQYzuqa5KTIBC7LUI3qHREnAAAgAElEQVTqZtD96dX3nAAwmttR3Z/6A2Zt+TKz1ixgex3p\nm6YwuDjI45k5HgeLUt8rEhY/RJAPVNrUpM/7ksRmt+2d/ptaT0u6sRJIMf2OhAjLM6/WmdnW530W\nfw1trSYTMvSLvq4hutFUwoPTIC1+JR4NC54F7MdIn/cxx469H/Y4y9/qZYy1u5XLn9SG9vxz6zBG\najAYDFPCPqQGg8EwJQ5UtG9trAMAajWfLWhpgUW/XLIvbW5ygbxhkM9Ti5Jp0bpeOxCzRB5IJKdl\nLNmHytXAtUjj8MWAESeat9BfmxOXpLKUUXbOn38g6oYscKHZg3FxgCZpw8dK0oZtZnBKW1pgA0we\nqCVUJZLEvE3FtNB415GooY4WNQxzLxzlCJa6uMg5cZ9K81C05qWTjFA7mztyTi9O1+u8/1aTt60c\n88XStNSIqmyWFlmkbyx60b4rfT5XFxetjlfdRCQGqJjPoeMidcHrlHjj1KxhVAyfItxnz7FuU8Sf\nofQTM/j6GgwGw+HCgTLSBckkXqr6zPKS+BpVYYKV6AgAYGtjq2jTEePUUCgkBYagwoYh2zSjU5jj\nMi7J+SSvqMZdhwYlzSikLkpREIcfiZN9VqS4n5SUcSzDU5jSSBmQtMmL2PvAIOVmb06rLnCf5QF7\nr0jug6oYgiJ5HsOg9HWtx20aYpxZHTEs8raSZLEvKg0EbRJ5nuqqdllcx8pVb1AqSbamJ8+fAwDc\nc+p0sa0vrnIaT790hI1N8yuLRZuBsOWKuD91Kl4aqVbm5Rw8rpwwW+c8s468/W1mMNHtadqsT8Xu\nN6eX48aeWYpRmb2312AwGA4ZDpSRnj7F7jIuYIKbO8w8B6IDq8SamTxwLVKWmEiIZcAWE6dhn0JN\nJTN7FuQKLXLeq3tHpFns/XG0fHKWj+UlBZCPKW28e0aQgaZYp/k0gwz5ortTNyx16QlDRGcRpbrQ\nLvL0S/WUeXFv4k4W1GWqlLTPpGeC51CtMMtT8l8cJaAn2v1Rhc+1ICGmFIRxzjVY37m5y+PryOpy\nsa0vOU61hPfqUa4lthAw0qFWTJDzDup+PNWqcyOXncvYjRCU505m1/1pZJ0sx9lhmNBsTza0/cin\nu9ERw+Ppe3Zrufbd2H/Pm8nu57V/E8z2m2wwGAyHAAfKSBfFypsGFvl2S8MmGfkYMwR8QhFVI4YT\nh+rSSJNTZGphD7LXa2XLSPWfyghDRqrJRmQZzmuFTlQvAHsuRGdRN2Hm1POr032xe3iKPXsdflBJ\n2aYfRuqQn4v+U59xHFjUSfWnYuXOg5l/qMUI5DilWJNVBBUlNbGJuGzkcvo8CriJnK4sCUmq9aDi\nqzBIJ43mFphhzi94RqrncCLpZIERXll3mkmiHJGCkiBnqY7LmcK+ZvIbsz1XSGrFgW56iv3J3o09\nBSbx2Ztz3FvDNMcxRmowGAxTwj6kBoPBMCUOth6CKPCrQUmG+05x2ZHNbY6NvrrGTvuVqi82Nl9n\nB/6m5qYMYvULJ/mhipSjyxC0Ty3Z8VIjI9lu9ngr7Y1JVjcfFe1DcbVQGxR7jxmtbnBNhx16ySOi\nuYjEuebxVMtQvPdeveea339QGHm0H2WXKNSDSDkTEdF12fdFd1GSdVks6pRycP6yGrtknRip8iAD\nl15TkkigR9BXw6HG1g+k7UCu0RvUYpo90X7/EXhzQ5LakegWZONp4/kn732bMfpjKFSAEwJmbnZL\nxkgNBoNhStBt5Rk0GAwGwx4YIzUYDIYpYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUYDIYpYR9S\ng8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUYDIYp\nYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUY\nDIYpYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBL2ITUYDIYpYR9Sg8FgmBIz\n/SElovNE9KV3+joMBw8i+iUi+pE7fR0GAzDjH1KDwfDSwmElTy/5DykRJXf6GgwGw2zjbviQvp6I\nPkZEO0T0DiKqAgAR/VMieoqINono3UR0SncgIkdE305ETwJ4khj/noiuE9EuEX2ciF4jbStE9L8S\n0bNEdI2Ifo6IanfoXl+yIKLPJKIPEVGTiN4BoBps26+vv4yInpDx8X8Q0X8hom+9IzdhGAER3UtE\nv01Ea0S0QUQ/S0QPENGfy+91Ivo1IlqS9r8C4AyA3yOiFhH9yzt7Bx53w4f0awD8fQD3A3gdgLcR\n0RcD+DHZdhLABQC/MbbfWwC8CcCrAXwZgL8L4CEAi7LfhrT7cVn/egAPAjgN4AdfvNsxjIOIygB+\nB8CvAFgB8E4A/0i23bCviegIgHcB+H4AqwCeAPB5B3z5hgkgohjA74P76z7we/UbAAjcn6cAvArA\nvQB+CACcc98E4FkAX+mcm3PO/cSBX/iN4Jyb2T8A5wF8Y/D7JwD8HIBfAPATwfo5ACmA++S3A/DF\nwfYvBvBpAJ8DIArWE4A2gAeCdZ8L4Jk7fe8vpT/wJHcZAAXr/gbAj+zX1wD+MYBHx/rzIoBvvdP3\n9FL/k/doDUByk3ZvAfDh4Pd5AF96p69//O9u0A9eDf7vgGeyVQAf0pXOuRYRbYBnvfOy+mKw/c+J\n6GcB/O8AzhLRbwP4XrD4WAfwQSLS5gQgflHuxHAjnAJwycmbJLgQbLtRX5/CaD87InruAK7XcHPc\nC+CCc24YriSi4wB+GsAXAJgHS81bB395t4e7QbSfhMsAzuoPImqAP66XgjbhSwnn3M845z4LLOo/\nBOBfAFgH0AXwsHNuSf4WnXNzL/YNGEZwBcBpCmYzsK4M2L+vrwC4J9hG4W/DHcVFAGcmGHv/F/C7\n+Vrn3AKAbwSTF4XDIcTd+iH9TwC+mYheT0QVcOe8zzl3flJjInqEiN5ERCWwKN8DkDvncgA/D+Df\nE9ExaXuaiN58IHdhUDwKYAjgO4moRERvBfDZsm2/vv4DAK8lorfIC/vtAE4c/OUbJuD94Inux4mo\nQURVIvp8MAttAdghotNgQhPiGoCXHeyl3hx35YfUOfdnAP4nAL8F7qwHAHzdPrssgD+YW2CRcQPA\nv5NtbwfwFID3EtEugD8D8IoX58oNk+CcGwB4K4C3AdgE8LUAflu23bCvnXPrAL4arDvfAEsbHwDQ\nP9AbMOyBcy4D8JVgA+6zAJ4D9+u/AfAGADvgifC3x3b9MQA/QETbRPS9B3fF+4NG1U4Gw90LIorA\nL+w3OOf+4k5fj+HuwV3JSA0GBRG9mYiWROz/V2B923vv8GUZ7jLYh9Rwt+NzATwNNhx+JYC3OOe6\nd/aSDHcbTLQ3GAyGKWGM1GAwGKbEgTrk53m+l/7STVcAY6x51J3w9jGRhcsqivYeO5eNvW4LALB2\n9TIA4MJTTxVtWpvXAQBLdf7dSLyf8dqlKwCA1JUBAI/8/bcCABZP3Fe0KZV4G017cweI7/up73MA\n8IHH3lesKzm+/EapAgAYynPoxmnRJqrw/D0/x22Wl71b7uLKAgBgaV6WS7ys1ypFm2E24ONIWEQs\nT6zT7xVtBi4HAOQZ992wl/ltbd4/TviZJ2UO2282/f79Ll93a7vN9zXwY+beOb6mM7IcDPlcf/zR\njxZtjp99FQDgJ77rJ2amP9/9n//AAUCj0SjWVSqcViIpJbLkh16W8QoAZXl+UczbKHD1zIapLPn5\n5xkv09SPB0eyX8zHTAfcD4N+x1+cvLNJhV+w8DVxufS1nKu5uQ4AuHrp2aLNbpcdNVZXlgEAcyV/\njW7AfTxMeVy4iO81dT7uZrfD1/Rt3/M/TOzPA/2QOrn58EpIAhtcLjchD4wi31EUlW75HKMBSHpi\nPqYr2ux9FrotHfL1bG1tF9ue+vQTAIDHPvoBAMAnP/YRbvPcM0WbU3P80N/46tMAgGMr/uOwdZnD\n9gelRQDAxhUOrpk7crpok+iHdP/bO1T4wCfZZrO2XgQP4bVH7gMAPLiwBADo5TwArwx2izZrHe7r\nnYG8TMHEVpKPW0VerkrEH10XvHj6Aa1Uue2Q5GPdaxdtOn1Wg+rc7TL/ZLOM20fE44py/t2WMQgA\niXwUjs7xR+KB+kKx7d6IP+pHK/zB6cqL96fB/u/9qJ9cZgYy+QwH/j6i4p0tyW+Z0IIvB2mHyOgd\nDn1f5Zl85OQNc8T9GSX+/Y70wyUf0GFnh393mv7S9B3O+UOcBcGFJZm0o1iuscy/SzWfW6ie8Dko\n4fO3esFHOpVvkHzkdaIeumDMDPdXgZpobzAYDFPCPqQGg8EwJQ5UtE9bmwCA9uZasa67zf8PuyxK\nOxEvoooXpaoLRwAAjaUVAEBlzm9LSpKWUsQLt49wnIoObWuDz3n58uVi25WrrOO8vs5i+IULXlw9\n96lPcZvzTwIAsg7fx5llr7c7ffw4AGAVLFLSrhcdhm0Wa3PRCZ574hMAgGMPvLpoU66KGEKzkw+F\nwH21WC5Sg+K+RdZBveooR2K2RUxbbvuh1qzw/1tlnsf74S13+ccwYTFrJ2Xxrlf1wUiNBj+rWsL6\nsoHoQdOW5wWuw+eoV/naKPai2XaPdWittDNyrqPzR4s2Zxd5jK2ISLcKr14arLOuvC2i6PxpTn96\n/6kzRZtnd7z+fFYQJ/ysklJlz7a0z88/VxWL6CUBIIqTkWU29Nu8PYLfS7WSEAUcbsiitH4fhh1+\nX9zA66x1v6GoXFqBznpufnnkmBTz8vg9RVpaLCyyWk3VDttr/huEXPXnfN29Lr/Dw8zr1XET9aIx\nUoPBYJgSB8pIL33kLwEAu5eeLtb1d5gJRkNmB7HMCo4CZXRdLG0neMavBkaa6iIzn/ois4mSWByj\nkr+1fMCz6cbF8wCAxz7EhoCPfuwTRZsnzzMDvXKdZ8XtrZ1i26DNRoxFuaRXnWa28roHff6L+4/x\njBflMuOt+/37MkNHYhS5epHvv9fcLNosLK3Kzc4OI62U+VnHsTfyVCWZj87mzRZv6235e10SieLY\nCWbx7bl6sW1zyGyg1RFjkTChct8bQHpiHIiEJexK/2zttPzFCZNMwNdYIs9gmtf42J0+H+ceYZJv\nWH2gaHM85/O2r3CWxstXvPSSxPMAgHqFWVa5zec9tuCt3dXZcb4oQCINxUnAvpwYd+QdSjMe32lg\nkMrEoFSps4E1CvYnZaJqdBLrPeXeIJV1mYEOOiyVkoydcnAcETrQl/M2d3xMRRwzg45LzAsbYiA8\nc8Yz0hMnTgIAemJk2tr043HcxNvvdeW+PCNNAuPYJBgjNRgMhilxoIz06sf/PwBA3F4v1sXiHlOO\n+esfievBoOP1I71M9J9b5wEArUWvy3JldrOpL/KMU1thPWqp7vV2ucwwG8+w3mrnEx8EAKTnLxRt\n6Apf01yXz79U8Y+mMc8z3tkTzIxffh+f8/iKZyDDlK//8jWeXdc2vbtPeZkZ2LzcY2/Aer/O1rWi\nTXbiXgA3n/kOE2LRRSkTAICyEGr1BCInOsrc6922r3Ke3hUur4X7j3nd4hl5Vlf6zOibKbPNQeZ1\npEPR012/zs+x1RfWC8+S4ipfUzNitph0A/eVnJnOySWWKL7kwdcCAE4EvoXtTb7G/pa4BLUCdiTt\nsi6zm2qDz5Wn/vzwasKZQb8n7kap1+8nJb7vWNzQSF2VAne0dJvZXVUkr1pjvtimxHwo+s5cGGHe\n99KDG8g6cUMric69VPGSytDx+XfELbHT9d+Hsri9HRVd6enTx2XpGamqahuNuZFlCNXn5vneziuX\njZEaDAbDi4oDZaS1jFlGY97rASPIl97xDDcUvVXaCxiEOFeXZf+466OGdq6eAwB0UpkxxRk3qQQ6\nVvEE0OOcLvFy7rRnlCfKYjkUJ9xS7K+xIszrxHFmoqvHeL880LtdbTNjut7jtuu5P/aKXNtxOU5f\nZucr571ld/U06+cqNe+RcNihLMWVAr2u6KY1aqha5vtZXvGMtFtlBkExM44k89vuWWR2et8cM4ZM\nnllKniWkwlx6Q9F9N1nP/tyulzB6JYlISrg/56peQllMWKI5LZLN/WKh7171nhqRMOnFZZUUvL5s\nt8NsKu8zOxr0ZewOPS+ZX/BS06yg2+T7clX/7jSEkRbUUlhbHKgVc5Eid69zAYpOwN5ifY9Et+pS\nHvsUBC9ESheLpRw38GAZOB5XvZQ3lqv+/WqIjv3MWS5+sLrKUunVq17i29hgb5w5GVeDMOhAxnFf\n9PHr615iVnzu534uAKBWm1xA2BipwWAwTAn7kBoMBsOUOFDR/sg8n24QKJoHQvVzcbMYiudtJ3CA\nbooTbU0cbueq/rITUQ20tvmY3V1WfKeB64KKF5Uqi5A1MUSVSl6ZHSEdOX8SiPbqhNwQdxcScaeb\netG+LwryxfvFDQteXN145pMAgK0dFh0ubrJI2I6K4pdYPvNKXh4v6rgdejgxAmbk73Ug8c65xDTH\nZXGMr/lnXTvChkGq8DPLy9440etyvy3WeL+V5WO8z7xXeZC6xaiqJGNVzXObPknFs5ss5mcZj68j\n1aViW3nI/bgi6h9128GRIL+DuHEly9zHC4H9ob7Bot/WFRljJXH/mVsu2qwcm70CvZ2OGH2cf3f0\ntlWUj2QbhW1EbM9Tcdrv+3e3LKo2ZWzaxuXB+xmNqg1yec9dkDSkm/G6tiQfqc/7/mzUeWyp2K3v\n++6uN/h+SoJqVLSPg/e7Kw74zSYb24aSb+PMGW8EvRmMkRoMBsOUONBpcyhuCu2md1YfiiGIJDuL\nZnVJg0/8QJzzNRPaMFBCl+d5hklkFknEBSUPQshydcGQabUkyvT6vFdYR5KmLRPXh07Hu9t0tsVh\nWGbFVMIPt+EVz42zDwIAXveGv8vX1fDs5NE//SMAwMff918AAJc3mSVd6/vsUcc+yaz1NW/8u5gV\nROriRP45qB2wKY71ibhGlQPjX3mOGWhU5v3KQfCEMp18wPunXR4zcZjeMNJjcp8tCcuYP/FQ0eRY\nhQMkMmFLCzXPeksS9FAWQ1RZ2HOy5A0Q6e6O3BtLOktBajkIA26vSUYiYcZR4HIX1717zqyg05Nr\nDoyoA3HEVwd6NdwmgX0xEutQKRJDVOj7pe+xsEx1cRoG7lN6MA0I0IxdrZYP9NiVgRWX9rovlcS4\npUajkkiM1cDAqMYlZalhBrhWS0J+JbBjYYGln1OnvPtUyGAnwRipwWAwTIkDZaTdruYc9d/vWHRq\nmtw1lVDAftvrNyJxc3EDYa2BDqZUElZT42MOZRIbpN5FSqHnSESlt7jo9XaLi7xSJ6rdXc9ONiWp\nQiazaFt0nbUzvirzfa//ewCAs6/8DIwcCMCXf+3bAAAvf/3nAAD+4PffCQB4/6N/XbS5JKGIs4RE\npIe47J8jSbheKvquirrCJIFeWxhoQ2b+5SM+1HZpmZl8vcb7qSO0C56nShgk7MipI3jgtnNskRPd\npCKhVAN3maomB5Zh6CQnLgZeCllJeFwNGszSXOZZVln0c3W51rU26+X7mR9z5Xj2QkR3JdQ1C/Sf\nRWitvIOaPzi0IWjO0pq4wVHFM9qinSQ0UbY5EmIq4mcsSaR70mebbd8fg5yPc/YE68wXF7yO1Alb\nVvclZZvKLAHgyBEeD9vb2yNtAR9ckMgYXRB3uGPHjhVtjJEaDAbDiwz7kBoMBsOUONh8pAOtleNF\n81jycKro1h+wWOHC1P4iManHRL8XRCWIOJGI6KC1j/I4EO1FzIwky4uWFEiHQb5BUZDP11hBvbzg\nFdWQXKlX1lgcSI6yW8SZN3xB0eT0QxyvTRorH5TPqM6zoePhN7yBfy+Iojxwv3rVaz8LswaqcD+G\nYl5dSnJEYnjoqTExC4wLIrrV5JnHSVgWRvIRtNkVZXdTskAFz7Mmxqq5xZWRfdRtRQ4KAKiKW1q1\n5A1impFI99MyGklg9EoiddVjl6B216uaqkssVt7/2jcCAEprHNGzedm7X60OvUF1VrAjhphBkDOg\noX0skUia3SwNukyf51AtULl/1rGIyyU1NqmmIHj3BuoSJQdtSnRjUGYLS6L+OS5ZnEqJ54C7TTFi\nt7mvVFRXVycAeOABjhx85hk28KrLEwDUxX1KxffjkpWsVg/GzE2yeRkjNRgMhilxoIy0I0xyLsg/\nWdZYXvmk9+V3lPjZIO3xbDCQGWt727tFaN7RkmanyUVxHRTP06z7WgCtK0rsXuqLa2nhr2iR21Sr\nfjariYvVUoVnqsbDnw0AOPayVxZtSpqJXdlOMINRsUoU5vfdDwD4pn/yLUUbzeA9S0gabFhyqacO\nGgjRFXaoxoly4hmlMpdBm5n+1lW/f1MYTCqGGyWii1IdAQBKYpRIJXcC1FgUeV5QWeDnWRUn7ZBR\nqP1kPMnPCKuQ5lUJ4sh6fmtvl8dNqkaWOjPUU0f9fQRC08xAM8P3uj77UyoSWq0sroPyrKPAIJWI\nOxlJoEPwqJCI9JmrNCjPPgvy7rZEUt3auC7beAwcO3lP0ebeeyTnQcSdNuh7Rqn5QwdiqFbJpFLx\ngSL338/vnBqQQullPNuTGp1Uyg3bRNFk7mmM1GAwGKbEgTLS7ZboSIMvfa3OM1siblBa27rvvB61\nJTqUkihN4iDvpLrXqFO166mrkm+TF6Vcuc1AQju7Q+8CoS4cc5G63Xh3GZKyu/NSanjpHp7dwrBF\npTATSz1riWnZpmFzx094t58kmb2QQpLZOR16BrMjIboQ3XVdliFr1e5LOxLKFzhnl0QSiKsstRw9\nwU7RS0s+wEGTnbZ3OaPPUPKRVuq+z8oiIaSJ9HHsWYcGhvRarMfUuujh/lrqWeuxp4H700CU9V0p\nF3xd6qi3A5e7xRnK4uXB4zMZGcMiDaqHmDwHlwbvjozvqiSj1fLKABAl7D7WlxT3fXme24Gz/ZbU\nN1tY5CoR99zDNojjJ08WbfTYO5vX5Hr8mNnaUZcm6VfJeXoqyEeq7k/zYq8Ide4qRWrwgWbPP3fu\nXNFmdZWvTZntOIyRGgwGw5Q4UBq02xb2SUG9FWGUJUkcca3JM95T1/2Mtb3F1sQjkqV8vhw4/Irl\nWxlp3tVqh37GhIaulUb1Nc3OXkbameN11Tk/43WEQVfmWL9SbrC+brQ2zWi1xBDKRJWZqp4lZKE3\n0r0cZiw3mFHv9K4U6zTZhwY/ZKKQTHMfMjkQz4xuV0I9S95DYk50oSdOsk6sKl4Y1y/5XKHXL3Ee\n17THWezVOfv+hz6zaNPrMIXKJRlNnvjzb13+NADgucc5acyGZMNvLHsH7BNnmHmoZ0Co8uyLB4Lm\nlW0LM/30Jf8c0nhvJc7DjvMXzgMATklFXACoi460oXk4id+3Xs+/O10J403F0ybteGZeStXJX9oI\nM6UkyEF738v5vGKRny8Si/h3YiDn0MQqvSB4oi15VC9feg4A8ImPfxwAsBTYHR55E9s17nvZfXyv\nQXWMLclV2hGWvCGVhC9e9GNOE6J8/w/+a0zC7L29BoPBcMhgH1KDwWCYEgcq2rfEWFSOvOHBbTNV\nHw5YhLjcZVFunXwsbUc+9zvX1gAAR2v++1+qsNBVUWdgKfugpUsAIBLXm4rcbkkU1/3ci+GpiApt\nETvrA7//rrhqrFTZ4EHqGkXBPHQLodXjhqhZFOdDzFdZFM6qgapFSk4PHKtvupLpqxyI9rG4k8UV\nfta1oDRFXZyg1y6zmPapx1lMO3/uiaJNS4oGHl9m0e1Nn/dFfD2LXiSNSBzJtRxKUP43Ev5w8SKf\n48Mf5rLc17e80WxplfPKPvKmRwAAZx+8r9jWF7VRVXKs1hdY/K9c9yUqdra2MGvY2mSRthSonFSk\nnZPsV3VZFsEQAEjdn2TZ7/u+TuU9SkStNSeqmtDd74gYcuqSgU37qhfEw6trU1+O1w225eIql4mx\nb1dUgVtrvuRyKt+FJx7nvKTrG76vNDPUQNQVW+u83+6ud4+MzCHfYDAYXlwcKCNttpmlVIN8okPJ\nLl4WBnDifskU/7B3yO9v8+z+1IfeCwDYvXK+2HbuGiuIq3LIKgnrDdxdEnEmXpQSy4nkwcxjz4w3\nW+xCMb/NDKo+78/fqTETPSpGpjDbUYEb25r2NnU3bzMLSCJhluTDJwcpPz+d3Z04Sy8EBsI00zLO\nLIWUg+SWl55mxvDoex4FAPz1+7l0djMov3tMmGj5YXYxKs9xv1DZD+c053ExbPH5XeCAPRTXusZp\nNnLQBWYej3/40aLN9gfYIHXlErPft7z1zcW2Up1Z1ZpkElo5xkw4CYwjWWjsnBFoCPXurg9vXd/k\nsV6tapY1ZqSLQcWDOXED1ECJXj/I6yqubSSDXnPPjgbliGFSsqyl4r40DJjtUNhmJpnjXBYGBEjl\nDAn1zIS1bm95RnrhaQ4NfeYpdmlqBLmIT5xiI5c64F99ToyGWZDTGPu/tMZIDQaDYUocKCMlKakb\nlb27S2WZmeg9r+aEHmc+k/Vd5bp3wE6l/O39DzCD+Nh7fR7P68+yK0xzl3UeHXGyL6UBI5XQtaHo\nRjO5627fzzJtmUR3Wsyueql3qK6cfjUAoLbC7j5JSerQBJPUTVQoIxirbDuzSMVLO8xknkmCF5cy\nq5gXtlmreoa/JG5Gy4vcx71tr686/+STAICda5wARJ1kmplnrSlk/EhoZi7JJja31oo2mrVene2z\nwOm/KgERR0/eBwA4fYZ1g8eOfyq4D2Yz1y6xm8yH/va9xbZXvpalpk3RKV7Z4Fyy8ZxnOT0ESVpm\nBDUJYsgCtrcp+kISJ/t8gsilun7lmPp8AaBe1f/lfSxKLgcBEuI2lYnUkIr+cxhIIU7tGZoFP8wP\nqgEBUoUhl34YBIx2u8tSbV+qAKwc8Trek+K4r0mVnjvPY2/Q8/tTENI6CcZIDQaDYUocKCM9eoxn\ngZMnjxTrlh7g9HOnX+8kbKUAACAASURBVMHLlWM8O6hjNwCQhHU98oVfBgA49eCrim0XnnocAHD1\nHC93r7NOq9/0LEmrgPYznnGe3eBZtr3udSh1CQ9cmucZtBRUKl194GEAwLyk8lJH+tFZaPYyok+L\nrXWe5bd3fFXY2EmKPGEOGpK3tOx1YrHosXc3OUmFC3RRS0us/3zVq7n6wNIRZq/tgX++976MJZOH\nX/s6AMDqUW4TBXqsjmR7b4rlNQk8JDJhx7E0f+0ruN4W9bxu8KN/y8vNa5wi7/r1y8W21et8jbkk\n0GiKTm8w9GOum81eyK/TCroB+9KEIBcvsI5xe5vfmVbHV7s9K4xwXnSTlaA+l3oAlEQ3qkmKQof8\nXEJu1cI/KKRJfxxNfuLkPYt8LAxSCRfV4yhDrpT8/kWYuPzWMFAAeOKTT4zcf1/1+8F4yrO9FTdC\nGCM1GAyGKWEfUoPBYJgSByp/1CWefkEMTACwdJxjqmtLLJ5pjHZoh4k1K404bp992QPFttP3caaY\nXvfzAADdJoty/XaQU1EyQl19jg0Hf/0n7wYAPPmRjxZtHjrLLhAnlll8r1S9sWnxCG8rV7zBZC9e\neqL9wjyLuIOhV8pXS6wiqTp+/itzLIOVyBsX1kX9UhGjY2PBGxajBj/3RAIj5hdZpHrZMZ/J5/P+\nDhcaPHnqNABgmEvlgtiPmmGbr6l5lZ3uXZBzckGMEYkYH2OJi3/w5S8v2riU93/sgyzatgZefbHb\nZINkfZHHs9pPwvjzLNnfOHG44cdyVUTxgeQX2N1hdc6nP+3zZTSlmsEZydq0vOSDaTRGvyYx+8rc\nXCAq69m0i8qS+SuCf4ZOskYNJdYeQQY5zdCViYhPWnkhcEfTkbF6hJ3/48BVbnNNsneJOlCNbWGa\n0vGcpeMwRmowGAxT4kAZ6Zpkti9tegZTc6L4L7NRQsMu92ZEDDwnwlyCYpSqNXimmWuoIWuvb1FV\nMgs9/gQbprqBG1ZPHH03NsXgsOrDDakkIaGqhCdVXI8Urtlz3Xc7jh1VR+yAXQzZ/SkRw4FLmbms\nbWwXbTQDEJaYeaRB2d22MI7dLXGJkuPMlXx/uh12T7m49hgAoNdjRlRtzPvrkPHUE7eXbuBKk7Z5\nzDQ32F2qO+R+PXaPzzV58l6WlK5eZ0a7c96HqKpxqbTA5xhKntthQEsGNzFOHEYUrCuMfJb3opxo\nfS4piZ77/rj2HBvkOmLYW1nxrkWrq/w+rso6zcg03/DSXWEkFGk0Koy5/m3qS/91xMVtELg/6XOP\nJANcIqWwQ6FAbY0kLLVUDjO3MdT9Sr8v4XdmOFLfbS+MkRoMBsOUOFBG+uQlZgelYw8W6x6o84xV\nlkQgOjvs94UPk38Uk44bXbpgxtSD5bJuILWbaM7rck68il1p+hnPePGC18klNW6nTsn7+dG/lBip\ny5m9p71rfl1Xss4LE+1I6G03qAOUiA6MhG1mfc/etrfYyT0XV6SVGuvFo7532n/ucQ7h6+3IUmv1\nVL3ufTthieJyi0dIL/CPr4nbkmsyk2pe5WX5wx8o2tz/ag7CmFuV4IGBD3vMoHWppLclQCPLPbNu\ndbwr1KxAXYQoGOH+P2V7sgy2ZOJc323zPV8LcoVq/tDxOkpztcA1SQNUcq1kwcfLg6oEfQnKaYk+\nthuwRSchplr+LRZGmuVBGKmsa+7wuGq3/KevJ2zX60Hd2G9f9+1GMEZqMBgMU8I+pAaDwTAlDlS0\nf+Ya0/JX1r0I1lhhEazIzal0esTaJAaofQPa1QAky0DRrLvprFEVZfbiqr+OI/ez68v1T7H64UTZ\n50t0QWnnPdf2EsbadRaJW7te7J4v81OOilLFbJAp13x56yyS0iuy7He8K40TN5uyiJCaozIu+Q5t\nD6RsBViWqx9j97RPX/fX9q6/4DIi3QqrZSpz3hDVFReel63yOHjk5RxFdf3cx4o2n/oki/lHJKqt\ntuhdtDSzVSqifUlceyhwf6KbiIKHEfruhIXhIjf6XunYDwtYain0Yv9AJO+KG2JL3BK1MF2a+sim\nWAxHKtr3NMIpyCLVkzHSFSPeIBC7S5LFzQ342GokSrPgHDLWuhrl2PSqJlUBuLHkFzdzeQphjNRg\nMBimxIEy0q1UXA8WPROszTHzozEfJ7pN2lfYmujGpiCdKXMxNi3M+Zj/I6fZ9eXR//fPedu6d5d5\npeisC3eMIvfoS5uaunyv43Kzyc8tEefoFYm1b1T8UNvtMxvY2WXH9qzvLUGZxFmXS+weU2kwo2wP\nAoPUdTZIJZJhaSfl3+trvu/7Uqr50uZ5AAAFDCoRN5uTtXv4+mMpix2w5g0psVwXJpQ0vLvMQFiN\n3rZmj1+or/oHgS5mDYXbT2CodYWr31ibCanLyhLbHpX3Oi9q9q12S4xFgbFJc5SSSJ49MUy2295g\np/H3sbD/UjDo5iU/hsvU6MX7h1msKhJME0fcdivzeRWGPXV7EiPk88jKZozUYDAYpsSBMtJYQkQX\njwSMtNEYbfR8SV7hEL832aeuUydinagqFc9Ajp3g8LbdlGfKjZafzbLs5iz5pchNh/I8iyAG+BpN\ni1UpnV1nBpAMA+dm0X2RPteg9lXmxDVNXIo0EdC5Z332pStSaykb8saOMIrmMKg1tMj7NyRTfaft\nJYw5YZ7ZgFnj+z/I+tS052v05JJs9qhc4/E5P1ZUEilq/QijjiqetZbG9eozgIJtBlLdOBOdqDdU\nyUxYZxw4y9PYs1KW2ap7h/yK6L8rkg91OOR+yYJsWiqhHD3O0oP2HQDMSTnuOOEgip0dlnTW133d\nLGrI2Mh1zPjx0O3L+YTR6nse1lS7WX01Y6QGg8EwJQ6UkZZlxmkEs7uGamm4ZcHsbpPi3YpOlWRW\nKYueZWXV60iPyUy3dERCQ+Pw0Yxemxtb/1LFYMgsYbsbJIDYEMvpgiS7OMJMpFoOKr8KA0012Udg\nnS2YgiQgGa4xK3n2gnf6P/c0O/lv7fB+vYGKGP4ciTCeWPRv80s+HFi1m89c4Mz2A3Gerwc5aM9I\nEptMwj8p9mxzScaN6lG74oifZp7lVObHJK0ZwCT9Zy56w8hpjOWYnQCBLlL2G6m4Ke+chl8OJIFQ\nGHIZa3vRnQ8G3L/1mn/mR45ygIwmNGmm/lm3pcZUKtfRktpwKt0AwOpRTnCjPdzr+TE3kIz8u+JZ\noN4HoY41/H8SjJEaDAbDlLAPqcFgMEyJAxXtlc4Pg9K4ef48fA2eJ/RMFcmRePKUj6dfWGSFtRbC\nKpUDY8EL7OY060XvFOcusmj8zBXvkD8U8WghkUJq0udB2HOhYtG+bze9kacp7jFJid3iUhHbr+94\n48LVbTFcSIx+pcF9Nb/kVUZlMfxkObdpBIXpllfZuX79Gl/3+mUpuhYaxKRUSKXM+7W6/vyrNc5k\nlEjceC/h69nKfIYrdHz+0llDODzVuJSBnxHJJ0MzLY3sN0E1oG+OlmVWx/ydun+/5kQNo69ZWZ55\nreFzYUTitrQrpdk7QS4D9f/vSbE7jZ2PE6/O0eJ1iRTmWz16stjWlPyyagjri2EsNKyZaG8wGAwv\nMg6UkeqspAXJAK/orVZ19ii83W94nJDR3Q5ZLLLBSLjY6lHvhpWIM/HyKjtVD4JsP+42QsVuBXdL\nOeZnzz0NALh65UqxrhZzH/fmeY7upsxk6sF+JIki1RCk/QEAucz8qRig0iEvN7Z8SF+zM5T9xU1F\n3HXSIEQzT5UJy/Ha3um/22R2GYkBrK6MNigZfeQ4s6GylBOOwnK8wlwjddWq8Lb5IAw1D0oBzxom\nvVI6VPMJblCFa5AM7DDEVN2NlKUqa2y2PGNXNleUvJPw0RyedbY76mrGy92W/4Z0RFroCKPUQJGF\noPLC/BIz2WiBpYlKzY/Ixjzn1S1v8nm7fT+ObhXGSA0Gg2FKHCgjrUqZVs0pCAA9mU0WFhcn7jMJ\nt8JCwwQnOhsOJE9iT5bHgsAAdZ+qimvU2nXvbjMcKju9OVt+KaG/JTrBjmfvpQXu41Tm6FTYZh7q\n1NRNRvSYjSXf95s7fMztbXZpKYseMg4comNhoCTMMJfzu9Szzuq8lP0V5/BO1zOo6xt8bBJH8KUF\n7vP77/d6s5MneWxoHsuw/k8qetfU8TgqlSWL/9Eg7JHCcNHZQuhKGI29bD6xiX+e5EbbhNuKXCfS\nRu0k/SAsGAuid5V8v6qj7LSvFk1Umu31ef/t3ZCRaiIU3q8sfd4LktCSBEgsH+G2cw1fk00DCLTk\ns98ncMi/ifhojNRgMBimhH1IDQaDYUocqGh//DhHhDjnKXdPohn2OkzQnjXPF7kon3s9ViNodpko\n2RsTXBNXjOb2ZrEtlRyKrjBW6WML6f5LUNwXsX0kT6z8rxFK2y02AFSCOHQ13mmJ3VLJb4ulT7a3\n2CXliORDWA1K/G5cY8NBKgalhkQkHVv1BoRqiTnCQM4xIpoRi3klUTUdXWEj0eqiNxYtS2G7vuS/\ndKnfv6vZjRpSdqMuMeaBx9zpY2cxa5iU0Un7czwXcNh23O2JRiT70Tymmlkqz/z+c2IASqQYZXOL\nE8te3/Ci/doGu6pt7fI7nKZBcUGJlCtJNKKT4bQlMfcA0OuzYXRzh9/rIyvHim3pQA2bgbphHDfR\nJxojNRgMhilxoIz0oYdeBgC497SPca/oLC7TWC7l7KIpGekkrtgQQ9KJY2wIWFwKyvcKuzpxnGPt\nE/IOuKkWdMska1FheHhpz0OJxJ+ngatYf8D9mGbcsdtSbKwc+x7R9k5ck+YCVxQnheW0kNxAMgCt\nHvV9dWSD/+92uM3RFZYiluYC1is5Z7MWSxPlxJ9/tcHnu/9+Ho/3nOQ+z1PvdB+J1NQTx28KSy5A\ncq7O8z1WxVm81PHnqPZmb2wUrk0TDEk0zkxDY+5YbLoLjTR6HGmvb9Vw6M9Rr7Php1JlR/yOZBBr\ntn1/XFsTCVGkicV5byyqSPCMGo00m1iYV7UvsfUtKR195bLPJqZ3EtOo0Wm0+J0ZmwwGg+FFxYEy\n0tP3cEbyh17ximLdygrrRwpVygulagxVYjJDnjghmZ1e9zpZ728/krnyDZ/1RgBAknuH6vl5ds/R\nSUkdsV+KatEQ5ZLUKso9I9VSvt0uP5xt0WlFASPtdZnlqZsKhb7uym5kIPRSZpSLNd/oxD3MRrbW\n2am7Ns+MtBFkXIplaA/V7abn2c3yMZaIjshSwx2z3J9DCWhOwmS6Xn8WEx+rJuPngfu43ld9zitJ\n6+moK80sQMsqUx5Igzd4ISkK1wsfmxAi6kZbFG2GaaiPlHEgrbckDHRty+e5jRJm/UeWuM/i4PzK\nhDU/rW4Kr7Gk2fslsGKQBgE3er3JKOsOGWkY1j4JxkgNBoNhShwoI92RLOXv+Zv3FetOnGTL3Csf\nfj0A4NTpe1+EM4slucuz0GOPfwoAcP7Cs0WLf/jffg0A4N4zbG1d3/bhaZ/6wz8CAPy9L/mvAAD3\nP/AQACBODvTxHT5EYnUP8njOi+U7EWf3UlErxyeQcMJglfiEySVqdbbOd3vMEpIqH6cfWHlrks/W\nRcz6SpJNvxxY9hNhGcekNlC+5i3AdWG3pZIk5BB9aBRUKt2VGlFtYUvtgJGQWHc7YB3tc122Mp+d\nv6doc6zhwxNnBUUYaCDO3aiSZui0rzpStWvkYTb5Qm8qlWPlcIPgeaoXTavFwRjr62sAgDSo09UQ\n/Sm0ykWov8So/jLHXmZdtJFLC5MSFYxU2bJcW3jv+U2qwhojNRgMhilhH1KDwWCYEgcqmz77LLse\n7LZ8Jp9ylcWxnpS9zcWBOoxznd4CxZR9bp5Fwlc9/BoAwIlTXo3Q0Mw94jqxetznKm1LzG4mooIW\nyYpDjxiMlq19KeArvvLLAAB/+Z73FOsyzSkpMfI0lDIcgUN7pariOxuL+pl/kImUkkhKrFppizqo\nKlmYAKBSZnXB3JK6QXEbSnwbFU+Lksmr3gFbs05p2QotI9wLnLy76jiecBsinxlqY1fyr4rBotVm\nFZE761UUD77pjZg1eBE3WEejBiQVd8P4el8eiN+dkJ0VrlC5/hbxOfPPerfFxqVmi41LO+L+FHoc\nqUuS7jfZCOZGt03It1EYoCZs00ARLds+4vJkxe8MBoPhxQXdzNHUYDAYDPvDGKnBYDBMCfuQGgwG\nw5SwD6nBYDBMCfuQGgwGw5SwD6nBYDBMCfuQGgwGw5SwD6nBYDBMCfuQGgwGw5SwD6nBYDBMCfuQ\nGgwGw5SwD6nBYDBMCfuQGgwGw5SwD6nBYDBMCfuQGgwGw5SwD6nBYDBMCfuQGgwGw5SwD6nBYDBM\nCfuQGgwGw5SwD6nBYDBMCfuQGgwGw5SwD6nBYDBMCfuQGgwGw5SwD6nBYDBMCfuQGgwGw5SwD6nB\nYDBMCfuQGgwGw5SwD6nBYDBMCfuQGgwGw5SY2Q8pEX0DEf3JFPu/jYje80Jek+GFBxG9gog+QkRN\nIvrOO309hulwt/Zncqcv4PnCOfdrAH7tTl+H4UXHvwTwF86519/pCzG8ILgr+3NmGel+IKKZnSAM\ne3AWwGOTNhBRfMDXYpged2V/HvoPKRF9HxE9LaLA40T0D2X9iGhORI6Ivp2IngTwZLDuO4noHBGt\nE9G/I6KJ90xEP01EF4lol4g+SERfEGz7ISL6TSL6ZbmOx4jojcH2U0T0W0S0RkTP3E0iy50EEf05\ngC8C8LNE1CKiXyei/5OI/pCI2gC+iIgWpV/WiOgCEf2A9jERxUT0k9L3zxDRd8iYsIn2DuCu7k/n\n3KH+A/DVAE6BP/pfC6AN4CSAtwF4T9DOAfhTACsAasG6v5B1ZwB8GsC3yrbx/b8RwCpY3fE/ArgK\noCrbfghAD8CXA4gB/BiA98q2CMAHAfwggDKAlwE4B+DNd/rZ3Q1/AP4y6LNfArAD4PPluVcB/DKA\n3wUwD+A+6eNvkfbfBuBxAPcAWAbwZzImkjt9Xy/Vv7u1Pw89I3XOvdM5d9k5lzvn3gFmm599g+Y/\n5pzbdM51g3X/VtY9C+CnAHz9Dc7zq865Defc0Dn3kwAqAF4RNHmPc+4PnXMZgF8B8Bmy/hEAR51z\nP+ycGzjnzgH4eQBf97xv2rAfftc599fOuRxACn7O3++cazrnzgP4SQDfJG2/BsBPO+eec85tAfjx\nO3LFhv1wV/TnnafENwER/WMA3wOenQBgDsARANmE5hdvsu4CmN1OOs/3AvgW2e4ALMh5FFeD/zsA\nqiJSnAVwioi2g+0xgL+afEeGKRH25xEAJXC/Ki4AOC3/nxprP2l8GO4s7or+PNSMlIjOgtnddwBY\ndc4tAfgEALrBLm7CunuD/88AuDzhPF8AtiZ+DYBlOc/OPucJcRHAM865peBv3jn35bewr+H2Efbx\nOpjFnA3WnQFwSf6/AhYDFeFYMBwO3BX9eag/pAAa4Ae9BgBE9M0AXnObx/gXRLRMRPcC+C4A75jQ\nZh7AUM6TENEPghnpreD9AJpE9HYiqolC/DVE9MhtXqfhNiFqlt8E8KNENC8T7/cA+FVp8psAvouI\nThPREoC336FLNdwCZrk/D/WH1Dn3OFhH8iiAawBeC+Cvb/Mwvws2Bn0EwB8A+IUJbf4YwB+BFdsX\nwIalWxIbpPO/AsDrATwDnlX/I4DF27xOw/PDPwcbIM8BeA+AXwfwi7Lt5wH8CYCPAfgwgD8ET5iT\n1EKGw4GZ7E8Sa9hdCSJyAF7unHvqTl+L4c6DiP5rAD/nnDt708aGQ4/D1J+HmpEaDNNAVC1fTkQJ\nEZ0G8D8D+M93+roMzw+HuT/tQ2q4m0EA/g2ALbAo+Emwv69hNnFo+/OuFu0NBoPhIGCM1GAwGKbE\ngTrkv/v33ukAII7995to3FWTfyeRz19QSkq8jHkdRX4fNpoDw+EAAJDl/HuUZ0fhoRHJOfM8L1pk\nwf8AECf+0aTpUM4xHNkvjv01agh/kkTy2x+Lgzb8/lnm5Dh7pYG3fNXX34rv6qHA13z16xwAlIJR\nlKf8/LMB31u1XAMA1Kvlok0cpQCASoWfS6Xq949i3i+SPuv3uE2r1SvaDFJuU67wsYc594NDqWiz\nudkGAKyvNwEA7fag2OZykvPz/nHM+2VBfwwzGQ8y1gb9vr/HYqzwNhXqQumuXOb7/dBjT85Mf/7C\nv/pSBwBZkDtkkHPnXnn2OQBAf2cLADBX922WjrCDytVrawCA+ly92La8ssz/yPu8vsn7t4PnWUED\nAFCO5gEAzS73x3rbX9vKwhwAoLP2NF/PtY1iWy9a4f2XXw4AyMVzcSGIkWluPMnX31sHANTm/Fg5\neoqvsTJXkwPyiRN5XwGAUAEA/PT/88TE/jRGajAYDFPiQBmpMrGQLypz09m8mN1j30aJQlrMEH6b\nMtIsH8rvUbbAGJsvxs45ckjSc/r9M2En2lzZp3MhM74xy1QGMxxjonkW3see3Q49Bl2+rzwOnoN4\n9A06zADbGc/upePLRZv6Is/uS4vCVgN200+ZeV5fZzbRbvMBk7hStKk2+P9Bn8/fFLa6emS1aLMg\nxx6kPMSjyNObXo+vTcfecMgMOew6ilSKodHfAGiss1SqSlPvzpiPSTizgEaDmeXQ+ftIW8wcq0Lg\nqgv87GtV/+nIxY2zXBbJMfGM1GXMzPW1jCPeP8892xtk/Pz0HW53uHFv4MdFVOFjzy3xsVczn04j\nL3Ffl5e4/U6rxfdT9n3W7vD16njQsQMA7asdvu+I77WS8/hYqvhxvbKw/6fSGKnBYDBMCfuQGgwG\nw5Q4UNG+EHdGDEwi0su2XMSmLAvbpEFLAPC0nDCmLtDdXLg/jTTZLxWJGqKyUOwe222vgWyCaiIQ\n/8T+BeeikQN6NcRklcBhx6DLIlBc84r7RMX8ityPiGtLy15MO32Sxfzq/9/elyw3lijX5R0xgzO7\nq7qqX3er+0nPkqUX4QhHaOG9I+xv8MZfantjhUNeSJb8wq2uruqqYpFFghiI8Y5e5MmbCZJdCgUs\nquDIswEJXAAXdzyZefIkwsPF4q55bTTikP52wuFWVXNo2OtqsYptX4nygkOx6R2H9klLC1JRzBWs\n42MuRBweasdunqFAiVB8MpkREdFqnTfLBJGcGihMlibcvRe2P7bPy/JfvGvxn4yi4LA7jvT31Rnv\n4zjhfRynfTzqPt+UvEydYN+bYhWi5Oa8loh+YYqHUphM8fY84z+iWnlev8vPJR3sx8ikBioUeBM+\njjYJf2loitpRR9aH13s612LXFPs/SPiYGaAwOlvpdrhdzulTcEbqcDgcO+JJGWkGahY9UlkRtirM\n1C5R1fdYp0F0jx0GjzDSh/7V/DmBLXphkVCKCqVhvVJwwKOw1tDIsCIwsUbaZYtVYJ5SpJBPrivD\nWoL9K04Mhvwbux19LhI2g4ji+OiciIgOh71mmRiysemUCwbvP6iURYpMgwNmrUnCDEhkUERE49GU\niFQSdX3LjOJuafYnSUGQt3FipGqdDhcsOm3+7CTl0yA3+1yOmccm08SxFLD4NYkmikL352NRy+eO\nDMWdODXSPyGOAdgqdnacqGZtjYgiSHh/1oFuh6LkyKJEoTjLVnhURvrxmt/fDnm/9Nr82b2W7rP1\nLVuUHj1jaVN8puZsdyt8NpjxszNe/9uRyp9ms1siIhrfMRNdFEbiFTADPT19RkREL16wU19UqWRu\nfvOBPgVnpA6Hw7EjnpSRFmBgpc1x3ssvNYy0fpijbN6z9c82S6xFJG0W0RyWvKW2b93+TPxhmwYk\nlSnrFITygSYHA8FxEqNpwKxlJcwFb2vE3ub7t3PC+4EXL5kVdDuavxTmJ4L6TspM5upCWefVO5Yi\nrQveIGvz2zu9I3wmi7ODOsGymtNazpmBTCfMbiQduZ2XRBNFznnPjclHL1fMkg6HItvhdeyHSq2X\na5Hi8HfkueZPE8kXxpInDO497qf8ab7k7RKT7s/1in//fMPsLC95/6ZGWrSBnIyCBK+pVK3OeLut\n1vzZ2Zo/r9fTbX1Q8N8r7NdNyfu6H+vnxCEz2G6Hc96dnjLi/gEaKkhy7vxdvfSsWaZAE0dJzFKz\nsTLiFU7MTocjlFaLmXFd6P5Mu336FJyROhwOx454WkF+Le1/iqBJf97jnY/kmJocpXlOmGAUSk5M\nRO+6jBbEt5moZZ0x/g4jyX9qDqUhnpLrlPebHGnzV/2IMiGQir5QYlnGMKjwYf73c8fxITPS0PxW\nufMXOT83umO2cnGhlfkcy7R6yFUOVMCdphU+m3OqYc37oWVadhPsm+EATAZsyQofNmCQOVhvbWIU\nWd/jI2ZeUtEPY2ViEygBfnnLLCkvlZHWTY4defGQ1y1IdTsI89onjEecRzyIVeEQgh2Wa35cIZoq\nO0bhAFXNcMDRRBrpvlpveL8XhagfpLJv6wu8PwuU9Jcr3vbrXHOUS7Rpn3Eak3o93VftCKJ/XM5a\n+P7Dgf6OFkT7SYuZZdrR/OmbS25brZacax+9R4NGofswDfT3PgZnpA6Hw7Ej/ELqcDgcO+KJBfkP\nw9cmTd/o6UVqpMuIzERCeyt5ikHjo0aKgtDDypdou4Al7lGx6RGXApQIr224Ggbb3/+gekVENd5X\nZNvfQURU0Xa6QVIclelprvaw2X69QLFopSHQZIqiAmpD+Ya3w3Kp+6MFIX6Kx07bOH0lvE2Oj7jQ\nEBG/1jfLnB5woSFCKD6ZcfFqOtN++iVCUekbDxNTOECR6PTwkIiIBn1OIyQdTTGsUFRp9bjo9fr1\nO/3dd/i92Oc10g9W8hTsYYPF5JZlZXSmhaDDIW+rGvKnDL9LXM6IiAZD9jj48hlPTd6sZs1rYxz0\n4qBGIYqHc91X+Zq3Y5ahwLfh/xcmmi5wPi02vK2PjgfNa7LVV/B3SBI+Lg76Krn7zYsviIgohXyr\nPdDfGCDFNp9zGmJ9w4XRoNYVGHqvvcPhcPzz4l+ckYbCQJtntsXvRMpI5TEyImlJVEfhr7BG87cw\nUhHSW0H8as0J9KysYgAAIABJREFU7iVawQ4ODpvXWjDMbAT4jVTL+FfihluGD92CajwnTFR8L60H\n6h4SGHr3lv0np2NlIJMpM40w4m12cvQlERENDlRAnbbldzNrXEN2Q0TUPWbm0+vy9lvOmf0VucpV\nUhigHh0y80ggxen3jewG/pHNcbFVGOT31yh8FBmvc5Aq631+ziPTnz//LdZHixvv3vxCRES38Dpd\noTgShNo2SY8I+T93rMAEK9MccnICSRDagJdgjUlHGeGzL78lIqKvXvIMutHodfNaEvNn9bq8P1Yr\nnHPX42aZTQ73pZK3Wbs/xHeo5KgH+VGnz/Kn42OVNpUoSq2WH/lzUGhcmtbjA/iZxpDqhUa+FRK/\n/91rFt3PIatLzDFzOtBj4zHs3952OByOzwxPK39CC12wJS3aFtA3OVPLIO5JobbaR4XV3Zc4mVtE\no16qt8X7y6Xm9t69f09ERBd4/N3vfte8dnp6SkQqxBYmalmkMuuHXqUlpB/SQbhpHPdNjpT2D3/6\nZ8xEPlx8bJ4bjcTcgRlIAmY6X9w2y3SRo2zBYzIyx0Oc8N/LFbPVMdo/7eTyqMPbczzhXNbVDTPj\n+VqZ7eGQ851DEW6rxwW1usxqCAYcU8ix3v5y1Szz6id2Un/x9R8TEdF3L071s3u8t1694mPlzVvO\nLa43upJxak1W9gMl2j7nhXH6ryH/gsSpBWZ+cq7+ss+ec/6x12Mmmef6WgpJWFVIgwQ//90PesRP\n7/i12QJMEC2ixyfnzTIShQ57fMnqGqOcAq2hSZdfu7ni43G5mjbL9HA8tGC2Muxp/vSPvnnOnyl5\n2DG/L640R9prf7phxhmpw+Fw7Ai/kDocDseOeNLQXnqTt4eASLiNglLwsNikkKKRLdJIp0R9b8nq\n3rvUkGmJwtL0Tj0G5wgrxAEoM/KpuIXxCpDHlAVGh2yU+ufoKd6g8LAxr62RoM/gfyk9zauV9o/f\nTrjT4j//p4e/+nNFd8Db4eXXx81zfQwQG2Fy2e0Nh/SJqcOkCN+jSPanbiuRrcn+76Coka80bF5i\nkF2W8XZsw8VJ9hMRUSpjIhBhJ6aQlPTQKw9vy+iI/w/7pv96wt836PI+Pz9XadSwz+vYgTvRoMuF\nkx//z2WzzGJpcgl7ggW6d95NtOsnavP2P/+CHZEO4e86GGqxqdXmczerpCNJ92ecIKRGIVDPXT2/\nOl0+D442HG53e7w/+wMNv9+8fs3fgVElFGivfYiDa3jI7xv2eacXmaZ6lnCIurnlAtRyo9eL83MO\n7YcD/m1rXBeu3/3SLDO5vKBPwRmpw+Fw7IgnZaTiyrI2RZ6W3E0GfIdrG1YhqO/Ll8xrVeOML8wU\n7zElqQLC6RzFnQUE5HdLZSCbQqZz8d1slSmjuFus8Tkim+Lnc8M6Z3DZni/gbLQy44NlDDMKYxt8\n9tqMpL24UDazLxiNuMjTSVXcPISAu8LQs+sr3taDvvY9F8225e3RH2ph5uSEjwNRrx0dscylUnJC\n01suMi0w5GyAnv9Nrttzg5G64pPQOVApTQ1GXOPoF7f3PinL6UAmc3LMBY9eV/u+05TX8QDi7tMu\nf39gjuu/R7Fqn7Csef0vx8rkXqCQdHjKjPT8nOVsdlz5CudRFaLxJdDwo4KHRIFLTSEC/Y1uq0L8\nXHGebhAxpsYXtQX2X6LB4s4I+ltdPjgOhnyMpXDxygszoA/uUfEMHgxT/f7bay4uHZ9xkUyikMVY\nC6TvjKP+Y3BG6nA4HDviSRnp1RXLS+4mKkvoIx8ibkst8TK07Zf3/EPrLQEU5BXNICTC/4ocd7yV\neCPCLUgeiYiyHK2loDB3c71jXV0zA0rReiYfLrlSIqI7SCbmM87BZPaOC+eg4t5cqnxL0L9/bkFX\nH/g3HwyVmZ+d8l397IxZwuV73p8tIwdarCDaj3nfDfrKFo+O+P0fL3mb9zBiN0rMaN7GqYv/Fxev\nwsxc2gjbRwtxkRnOIG5cJMtAtB+bvN8Bs5u4wyw1MO4/LeT6e2gG6R7zfl2+1FzxL++VzewLJJdf\nm1HF6xIzjjAWO7/kyGu51nNnjGjs4JhbRY+MM1OMnVSFcGhCLWS51Jx3iqg0qnhfybm8Nt+RpGiK\ngd9vbiLGkhAFoY21RkNBYhz2u8jRHh/JOGb93Tc3nOMuNvw7zk+4GSfacmdz+ZPD4XD8s+JpGekl\n5wGXJr+xgblEByJcMZCws3Kk2/JeGhTPbQv6JY9aGkabCwPEZNAN2OfK3PFWWA9x+55NtaIvuVVh\nPo2dqGGk+R3yv2CkkqMjIsoxn2YFj8pMpgCYHJBlXPuCDBMZ12uzR7DZDwa8P19+xUL28US3dWPy\ngZJ6WGtevEA1dY7tH5XXeF639fwOgulEmKnsH/2O4QF/doS2zYVxRO9BpB+2wcAisM22HnMV8qaX\n8Oj8SjXmFEGoLbOKClR5rbFKq/Wkp9b/E0iK1/QV0HjG2/3DNXLOFT9eftSJB2/eswD+y+dc/f7z\nH142r339nHPMQxibhHCfz2vNo7YTRJwQ1legi5WZiiANOi20iiZmOkUdCSPlxxyt3+1Ec/fSTDM8\ngMLAMNoxcqEXP//I78t5nQOTc0+7D2s3Fs5IHQ6HY0f4hdThcDh2xJPGHx24KLVipfU1BNjjW074\ndsQv0HhDSsJarvp2sJw8KdFI4/VpnZWakJ5Dh+kE/pUTDd/XS6bxcbAdvhMRBQgjEoQQMYTklGuB\naD3BuNdXr4mIaD5WUXOJ3npJoucodmQmGV4NzUzjPUELEqGCrFSMt20HRYGzYw7FRrfvzTK8HxYo\nOISV8V5AlH044JTA6REXfdYr3SEhChcR+rhPTvj/82daLJKwf7mAEPtKUy01Ck9reMcWkOikpMel\nHHNv377lzxlpvDtI0DSA3v7FhEP8j2MNBSf/iFzmc8TsDufJwsjyrvkY/803HJofwsN1nuu2uvpf\nr4iI6H+/+isiIrobqfdC69/9JRERffnVd0RE1D3g/WpH+QQFn4f1BukDGZ1tUz6luE5x6q/f1X1d\nYdRLlvFxVcvI51yvEzncw9aNT60WDw+P+LM2I4yMEaexWo/r8B9J1TgjdTgcjh3xpIz0I1xZ7Jw3\n8Q4UtjYCs2ub8afN3QvvCw0jFSlM03gmUinzHcIuJQm9RNFnvdYChDDjFKwzWKgoWQT1ERz1A3xO\nYJzh5yMWYE9GnIRfjlXiVaMoJeso87+Wsf6O9Z16J+4Ljk4hN4m1yJPXvP8+ogAzSFjQfX5+0ixz\nByeoiyveRuOZ+pmOpswOvn7OzKULjXxoRN4wKWrGYot7+2yqDCLDcSUtp5UZMQziQjdoX70FeyyC\nD/odGJIWoO3xj/7ih+a1M7QiTiCLe/Oef8ebG41wFoWV6O0H1jlvo/FMf8d//59/ICKiDOL2v/y3\n/4aIiL7/7Z80y4Rtjh7/y3/9b/w5a2W0cs5/hPTxEE5b79/+3CwzuUG0gggvxIkexcpa2zgQwoT3\n9dA4QwWQRk2mHAVevefWzjLXKGKI4merhejSNP6cfcH7s5rz++OKD5A41Pcn/U/vT2ekDofDsSOe\nlJG+fvOaiLYZqfgUlui7DFPJfxm5QQC5DFxHIsNI40AE+QxRRVjWKvc1WSZDm5mdkyTSKsmNJaGR\nJuH7G5oLRhqZcbHllBmlGJIUhcmvyBcj/yqiZOOLQtnaaE72BKsV/9Zn3+q+OjzhdslyzQxycwvx\nusmJHR8zO1hCVnY70+14cc1saImBPXd4bTAwIm/8mXZ4H5Vj/ry3v2ib7XIFVgE3/XZqGC28UlfY\n5nezFd6jzLjT5r+fPTvGb9Xo5RJs9wathR+m/NqlaTu0pjf7gjpkZpaT5pOvRvz3X/313xAR0XTK\nv/n3f/6nzTLff/89ERH9x3//H4iIaHLxunmtghfvLz9zrvn6I0Yfl7o9rz6wxG2z5O8a9Jnhnpxo\nW/EdIraa+PhYmP0RgKWuNvzcbCbfoedgq837MUakEYV6PElLazrgduQE+dOhYcRfHXzahMYZqcPh\ncOyIpzUtgckEmYq65CbFjKC57lsXfSGOENKHRggfiVt98z5+CLamQIG1QtRbYJnK3kbEvg/LRkbw\nK+xWXfwrLKOMNpaJkrgLBib/2VgFCssFw62MVWBV7R8jvf7IDOLkpTKHcwiwjw44N/rxDazZftKc\nWG/I2++0ZqVClahZyOV7ZhOv33L+cjrhfNvZuaoajk6YQXRy3p7TOQxSzOwomUqQtNBumJl8F6Kf\nusQU0xZ/tmUpMq9HpoFeGQG6kM0RLNmuxmDRmTLrYA9nNpWYnBrFuq2lieTjiPOHNSzuq8xYH8K8\n5+tveGJCO1WHmQKi9ptrfn+EhpWzM+1wEEOTmxkvO0dTRtDS42K9HGPd+P+KdH+IjV6MBpcYIWDS\nsQ0vUGjgPMszEzFI+ycsA6XxZmMbAgyDfgz7t7cdDofjM4NfSB0Oh2NHPGlo//UL9jSsTCEmRzi0\nRn/tCv8LvSYiqhE65Bibm5vxvdLvHiJcq8DSK3OPKFGkykUQH22H+ETGUQrhdkjGbUjzBVuPoXGE\nkaCweSo26y8D+aRtQNbVfEfxaXOZzxLzKa/025+1OPH8Jf/GH37LvpUnR7xlbkavmmUShHAdSGqG\npYbUG3iLjq44BLwec2hZm1RJFfP27KDYF0a8TG+okrlWG54NSBFZ79gAEVuGYpeIzMNI12M552Ps\n+pZD0jvz/jbE4PMNH8dzyODMIbv1974gx3kZm4aZZqoFCq0bNDG8fauDAteQCl5DDjZIja8risdt\nCNo7fYjnTSqrwHeMlph8MEGjQ6X7vMjgNIaxyIHRN6b47MGAUxJ99MX3+roeeSWTK/j/KNYCaRTh\n74SPnxJpmdlMrzMzc815DHu4ux0Oh+PzwpMy0pfPnhERUWXGEBdgFRtISjKZ11rrMmu0W85w558b\nM8EAolthfRWsosrAsj0Ud6R9VFRM5q5Wyjwo/F+ZtsfqXuGgvsdM+Tu2Rz1XoSazSxSnqlLYs8yl\nsh6Z+3dPG2DGzfWFJv7/5q9ZXC2jdP/171ka8/IH9eqswA7qd3g0xcf4nNne9JYF/DcjLiBdXWuy\nfwpG2+3xNjv/kt/z4sWwWSZtw1kK7YZHJzpOuRdi7PCUmfQGdkfTufHYlDZWaU1tGWeoTBoB+LP7\n8Cwtc+tBS/uHCse8OfcSMEqRjCVg+MY8iT6OOHrIK3ZPaoXK9g4GXIh88ZylRd8c8n4dT7WF+hLn\n90c4qM3ueP+OjEVvv4PGGwQNg45GDy3Qf5kB1krlXDStyzNumEkSjj5Ozp43r6U9XqeyYkY7kblO\nSyuh/LQ72/6dvQ6Hw/GZ4UkZaYmWTDHxINL5LCs85sgfttt6xwlwvQ+bR5O/lJsG7ko1BNhxZKQP\neJ8IdAvceS0TEvlRVW9PscQ/vPwnflsjZcIK1abroJQeVXxf0pgy6HZI4/3zrywgCcmMochPf8/i\n6jz/H0REVMNZvmUE8SV0Z/M5jCRW5s6fIv/ZA7MfIa9tDCg6mBoqxhUVem4/XGprYweeowcHzFIj\nwxmWd8xEJyPOu92iRfX2TlnvOuflj+H6fv78u+a1b7/9DX43v+/nH/+OiIgu3moeuNg/PT5FOB7z\njeYDwwhTQDuY1ApGWmx0Wy1XmAqBmU/9ljLaGkYwX+H8ODnjyODyVo1NLiHS30CO1uoyiw0SzXmX\niDRHENtPblXqdoTF0pj/ODtiZlkYP9HlnNdRfGIzE9UWMWZF1Rw1/fiG2evATHX49puv6FNwRupw\nOBw7wi+kDofDsSOettf+Jw59aiN/yuAUIw5LIULzozMtDtRyvW+xpCXuGSkLYihxVBJvycIUkgoU\neTLIj+SxMMOtmvC7idD1/eH94lItob61mKLt50zXk0iz5DN7bQz7Mqan7WT/QnvpFrEjtOfw//wH\nhPhR+rdERPT7v9DQOKg4XE4Rxm9SM1oX3ShoNqJKxnqs9ZiJECYmkLF1MhmapsWiw2MUFks+dm42\n4+a11ZiXK8SPtHnPl80yz17y+v7m62+IiOj46LB5TYpl42t2i1ogbCSTDmq1NCzcFxwOcFxG6gW8\nyjlcriM+9yLs66SlhT3C8kXB3Wi12Q4y4DCS4xspuOmd7vOLDxivvebXWhg7FG50n3/E6J6q4GJm\nUmn64Yeved98/y2H5nJ8ZKbid4IUjYzHGY30eKiXnEr48Q17Nfz0mt2jzgca/ve65vc+AmekDofD\nsSOelAa9AiOlUu8UjRQKtK/Tw52vrWLaAHY/ufTKt40zFJisODqt4SWYl8pahdWUYKA1ZBKRuXOK\nrCMKxQ0/NK/BNZ9k9cGCze+QMbEFnitNIUssqWSIXx+D7tqGtbTT/WOkgwNMMzC/I2oYBxce/vC3\nmHwQqd/qAJFFXbTxHnPnh4i63WbGkYDR1KYwt4RLUFnwvm4vmWWExj/y6JBZRhcuYncr/Y4V/pZ6\nQwiW9cVQ+79fvuTmkQOI9acTI9f58JqIiK4+/IzXuPARGPOGw0P1X90XDCGW7/XVO2FR8O9YbuT8\n4MfYFmPFlwJDHjsd3dYdRBZhhJHVK5mOoOdnCD/ZOJRiMs6FWr9jA73VAhMY4koZ7SrD8VTLshLF\n6HcMUHQMIFkbj4z71OU7IiL6uz8w281QPPvqWLeDXBd+Dc5IHQ6HY0c8KQ2SnIWVHQnkTifG4uOJ\nOsyLoD2HnKE0PqDitlQj31hDCB9GmqOM0VLYAqMVkXZs8pKthFlVina11LyW4rUKbHMNR5zNRtej\n3eG7qtwNN8adu6rEWQr/g0mlgeaiesmnx71+jhhiztQmVQYSozU2Svm3XUNV/fofVK5yPMCEAWJW\nECVGkB+KIxMzUxmdvDYMJoOTk2x/OR46HZXFHQ04t/fNSxZez490xk+x4dzmxyteJ2kfnU31mHv/\n9jUREb35+RX+f9O8toTDf0iQdsUiANfvP9hDRprGkB72VHY0bHOtYjRB5AcP19jUBzpwiwo2fJ4M\nuqqkHx4gn42gZb6AZE5PHeoj/5hiOybiBmbaOMWhf53xNrZmTBvIKddgrTXxsVfWeg7nOb9vtcYk\njltltK/e8f68ueLj4ktMfjg71Zy5HIe/BmekDofDsSOelJG2+3yn22yUwUjbZC2zl5CLiIxAvdfl\n21mc8J0rNNXuQPKeeC4MJf+pd8xI7p5gwjJh1PqESGunuO8HYfTgtXWTE+XHNNVP6HWZyXbxWJqC\n/n2z9PWS79hWMLza7N+Mn80Gv6PU31GjLS+BR+RgwPvM+q3O4EkZh8ip9XQDZUt+fwo20sdU2fzA\nLJNtsyPpxzw2piVfHHNu88sTfhxtGdRwLmyN7b8Bw128Vpbyy3tmohEYcm1y7l20K/Z7qGBHIlLX\n35gZv859QVUyTUwi3dbDIzAxnA5zGIpEleash5j4m015oW7bTOiE2KHTxVRX5EjNod9MoOi3YT7S\n530eGp/a5WaC9/O1YLXR83OBz7yD6Umnz7nuotYvuR3zMffhkqOOt++06n9zw/uthXP/C5ifHPU1\n9y/Rx6/BGanD4XDsCL+QOhwOx4540tBewvXIyIYSuOr0uixh6KNI0OtpIabdwSiBBDIk08fe+BI2\n4nrx/tTwJLwnkpfxJrWNuevt0No6M8lLVbiUH8LrbgpSXYSgki+ojINOKWEtbKe6KEytjMdhUexf\nKDjFwL92S/voM4ioxxN+rUJBbdjX8RUJDrsOClLdtm7r9ZL3yS2G4OUb/r9l5GFthIAVDpEKIfWz\nZ1rgGfQgicK2T0I7zhlymxYKF5IWMgY/clw14W5ipHL4qAhpDFnW2DvQ6Eb9OvcFksawXp8B2hUG\nOL4jSNbiSo/XZxj9UvaQxilvm9fSWNInHFKLo9JibpyyULzt9/kaMOxiqKE9B5tCM1J5ppC0XKGA\nNOZUUVHxOlbmlL74wMXC1xiQOFFbBmq3Of8Q1JDeYRieHjHb6Y7H4IzU4XA4dsSTMtIYQnQrS0jh\nsNIb8N2oB0F+alhOiEKUaICtM5O9e/IT8r8Znifvk0KSLGM8Q6kpQIlEStdRfExD5NdjsJtOV1mz\nrK98dm0Yqfg7BmZ8HhFRu6MsTYTG+4TlEm5epe6PfIP2PDDJvGZm2jHqrgjthvM7ZjWTW9WylKAR\nScQFg6MD3o/rTLdPUUM7IwJ4tJpGkW7zuztuBJjNIM0h3dbC/nNoaMR7Non0mEuFduLgSczEAyk8\nZPDQlTZUOzwvyz89LO1zhPirRrEWeSJMDRAGmEQYc20ivggOXwSGnplBg+sV9lvAx0FZMmvtdnRA\nXlgx2zwAExU52dxYaNW1DJzEuHSzr1Yr/r6ffuK25F6bfVHLSjVW799eEBHRbMZUtAz0/XIYdTos\nwB8cIlLq6HZITbPHY3BG6nA4HDviiXOkfDdJSe/cKdoL5TVp7SxXKq+QGTsibQrCh4y0IanBw2Vk\n/LHImJrxTPVDyZHcWaJH5EtZM0NGkmT6OwowYOHBzehlIoK3BkV2nWh77G0d7J8gfwFPSuuWHpDk\ns8FkwP4yYzqSttHqC5H0dKKvLSAN++Pf8mjf4QEzAZG98N+cWxZn+zqVUEHXYwWR/hKmOC0z2rcP\netxHHlXsccUEhYgolblQYqBhoqAC2rYS3SMgSw2bJiLKVsbefU8QyHymSI9FMQyS81OiytqcOzJn\nLSjFLMTIE2EME7f4tRSSpmRg2H/I26rVBJG8QyozEr0tkSuc8TMTTeZgvTcfucHixxaMTVJlkUvM\nlZK1ju24dEQyrRZ/5hFaQ9sdjTip0oaSx+CM1OFwOHaEX0gdDodjRzxpaF9AiiJhgv1bRnzkMmLX\nuK3In1Jk2p4T9yvFJhOKVc0Y5O1lI/NBTWZAvsMkzKV/PkeYKg5HQWwahrcnjVBs+qbke0Ktlj1Y\nx7LaP/9K6Qwyk2OoDTlbF10/Zc6/aznXLpMNijVJDH9Z4225uuUwT1Iz0tXW7+nwvPWa++Yn6JCa\nzbmwVNsR2HAMWyP8LgqVmp2dcOgWRRinvMSojEwLROIdOxzw5xTmR5YVUgJZiPfJsavbwfow7AvE\nR7Q0/KpCukK6+MQvdlObziKcH62at0ttuhJL7McKw/OkkCWdb7wQ5EooKBbofCwr/ZwY51y3jRSa\nrRNj1E2JHNN6jZHu5niIZdRNLv34xosDaYJuF+kHyOPWuRlFlH26eOiM1OFwOHbEkzLS1ZpZQSs1\nXqO4lJeViJvFScewxYbt4bXQvibPCVt9yPaqe0UmZYbWBV+WAYMplVFMwXzkLibi+3xrwpk0AqAQ\nYRhpWIt8C96azbrV9969X5B9ZAsPsmtaGF5YgrhMRpqsz1Ys2D4542WOTpSR3o75tXfv2CPyy3MW\nS39xdtYscwjf0BQ+ouMpO6yvN6Yfv0AhC0x4facjo8/Pmd3+2Z/wqOiP1xf47ptmmRrHRpzCv3Ki\n6y9RlHAq8aclUxwpKt3/+wLx0LXHdYLzUtSAESRKda0FqdVC5H0Q9JvzW7oXSuLoQ7ZLFNliD5yd\ncpGl4RwKba87r0cbDlVkpG4VCoMhopCjQ45Yjk50PVZrlj2Nbzl6ERcqIqIU1wyJojIw0dFY93mV\nqx/tY3BG6nA4HDviaccxozU0L5TtNa5A927g4ZbofjvHGEU2fyru89HW/7QlfxIhvrDd7cetv8Gu\nsvyhk09ZmmQgaTOB+Qq69zH82U0jAByu7s93sm/cIwzgW1kWuu4yBlvE6tmKH4fGmUmdz9HSZ1p1\nT085fzm9ZQZxcw1x9kwF+QnaRTuIDE5PviAioncX2pY5nzKjlXbclpG7bDb8Ween/4qIiJ6fs+em\nyKGIiGZwtrq4ZI/KvNAcbwgJToxcYAomulro8dFumX7TPYG2RZtjsRK2iUdphzXnYAQmuBKWZ97e\nwUjlmPhxlUvO2jjkQzTY5FalASbU7TmA12wG7heX+v2dkJlnH808333Lx8PxmR5zyyUfT28goVwu\n9VgpkFvtdJjJ1qF4l5oW7vWn5WzOSB0Oh2NHPK0gP3p4lxZDj/q+acjW39uM0hqKSE5U8nXNa1uM\ndHvZB1V0C1Qprbi6xt85Kn4VJhhaRtrkaMEy7R3qYbUeX/X/CyM17KCAN6i0vNaomvdMS2AC4bds\npC3DlkAE+LyNcwjqbTODmIsk2P6DAzYrCQLdHzJHSYxVDoem+QEVZGGwwkTna42Urm45JzbD7KmO\naReUfVVjRwo5C80xvM6M4eaeYNhnRtY289KGPd5uLUQam7WYh5joDC3Qm4y3WWjYalxBmRGIqQ9E\n/+YMb8t0YExnLWs5dnTOV0x8PAS4YrUGuj/TiNf3YMCPhwM0SuQ68QBBRJNzD0wL+XwFZQFccEr0\njNoJGmloxPmPwBmpw+Fw7Ai/kDocDseOeNLQPk2TB89JSC/jPx5pf2+ekxC7Ji0ESehI2w9bcqLG\n/em+VGprpOz2d4Rb41fRG17I0DUOb0KTqtDlt2VQ+m56GL5v1cP2L7Q/PeaQer7UMHY25aR+gTHZ\nUmiz45RF8yb7fGPGcoj0JBGnMBQCIuMnKmHZcsGheFZySJkVus1zfD/he/PCeKaWHOdd3XLoN7rj\ndbxbqEnlLcYvyzrGRtwtxbEAaamwaHI1DSQE3iekKBq1Ej2uezJqG5t2jSJsaRoO8s0ci3D4Xdaa\nGshKCPHRxBDKdjTysCQQIT+/TxzD1suxLlMhxdKSQp9xhyOcj+jZp4rfH5nfIY5UJYqGsUk/tNuc\noqprTjFUOL46xoEuDTQ19RickTocDseOCO4XeRwOh8PxT4MzUofD4dgRfiF1OByOHeEXUofD4dgR\nfiF1OByOHeEXUofD4dgRfiF1OByOHeEXUofD4dgRfiF1OByOHeEXUofD4dgRfiF1OByOHeEXUofD\n4dgRfiF1OByOHeEXUofD4dgRfiF1OByOHeEXUofD4dgRfiF1OByOHeEXUofD4dgRfiF1OByOHeEX\nUofD4dgYLry6AAAAJUlEQVQRfiF1OByOHeEXUofD4dgRfiF1OByOHeEXUofD4dgR/xcocqqCvU8W\nhAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 360x360 with 9 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pDnf_yQxleHx",
        "colab_type": "text"
      },
      "source": [
        "# reproduce the architecture"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9NXBdCgzJyIB",
        "colab_type": "text"
      },
      "source": [
        "Custom resnet variant: ![ResNet18 variant graph](https://296830-909578-raikfcquaxqncofqfm.stackpathdns.com/wp-content/uploads/2019/06/Artboard-1-5.svg)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YS-_x9MUI-N6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def conv(ni, nf, ks=3, stride=1, padding=None, bias=False):\n",
        "    if padding is None: padding = ks//2\n",
        "    return nn.Conv2d(ni, nf, kernel_size=ks, stride=stride, padding=padding, bias=bias)\n",
        "\n",
        "def noop(x): return x\n",
        "  \n",
        "class ResBlock(Module):\n",
        "    def __init__(self, c_in, c_out, stride=1):\n",
        "        self.bn1   = nn.BatchNorm2d(c_in)\n",
        "        self.relu1 = nn.ReLU(True)\n",
        "        self.branch = nn.Sequential(\n",
        "            conv(c_in, c_out, ks=3, stride=stride, padding=1),\n",
        "            nn.BatchNorm2d(c_out),\n",
        "            nn.ReLU(True),\n",
        "            conv(c_out, c_out, ks=3, stride=1, padding=1)\n",
        "        )\n",
        "        projection = (stride != 1) or (c_in != c_out)\n",
        "        self.idconv = noop if not projection else conv(c_in, c_out, ks=1, stride=stride, padding=0)\n",
        "\n",
        "    def forward(self, x):\n",
        "      _out = self.relu1(self.bn1(x))\n",
        "      return self.branch(_out) + self.idconv(_out)\n",
        "\n",
        "class Pool(Module):\n",
        "  def __init__(self, concat=True):\n",
        "    self.concat = concat\n",
        "    self.maxpool = nn.MaxPool2d(4)\n",
        "    if concat:\n",
        "      self.avgpool = nn.AvgPool2d(4)\n",
        "    \n",
        "  def forward(self, x):\n",
        "    if self.concat:\n",
        "      return torch.cat([self.maxpool(x), self.avgpool(x)], 1)\n",
        "    return self.maxpool(x)\n",
        "\n",
        "def make_DAWN_Net(c=64, Block=ResBlock, prep_bn_relu=False, concat_pool=True):\n",
        "  if isinstance(c, int):\n",
        "    c = [c, 2*c, 4*c, 4*c]\n",
        "  layers = [conv(3, c[0], ks=3, stride=1, padding=1)]\n",
        "  if prep_bn_relu:\n",
        "    layers.append(nn.BatchNorm2d(c[0]))\n",
        "    layers.append(nn.ReLU(True))\n",
        "  layers.append(nn.Sequential(\n",
        "    Block(c[0], c[0], 1),\n",
        "    Block(c[0], c[0], 1)\n",
        "  ))\n",
        "  layers.append(nn.Sequential(\n",
        "    Block(c[0], c[1], 2),\n",
        "    Block(c[1], c[1], 1)\n",
        "  ))\n",
        "  layers.append(nn.Sequential(\n",
        "    Block(c[1], c[2], 2),\n",
        "    Block(c[2], c[2], 1)\n",
        "  ))\n",
        "  layers.append(nn.Sequential(\n",
        "    Block(c[2], c[3], 2),\n",
        "    Block(c[3], c[3], 1)\n",
        "  ))\n",
        "  layers.append(nn.Sequential(\n",
        "    Pool(concat=concat_pool),\n",
        "    Flatten(),\n",
        "    nn.Linear(2*c[3] if concat_pool else c[3], 10, bias=True),\n",
        "  ))\n",
        "  return nn.Sequential(*layers)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bDlv591gV7lG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "018dfcbd-2acb-44f9-93d1-d3d6ba31fe31"
      },
      "source": [
        "model = make_DAWN_Net(); model"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (2): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (3): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (4): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (5): Sequential(\n",
              "    (0): Pool(\n",
              "      (maxpool): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "      (avgpool): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "    )\n",
              "    (1): Flatten()\n",
              "    (2): Linear(in_features=512, out_features=10, bias=True)\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDh6LqLIl4EC",
        "colab_type": "text"
      },
      "source": [
        "TODO: find a way to test if the model is actually idendical at the PyTorch level"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b4zXSIhTl9Ro",
        "colab_type": "text"
      },
      "source": [
        "# reproduce optimizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9D-0FWxc7VM2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "opt = partial(optim.SGD, lr=1e-3, momentum=0.9, weight_decay=5e-4*batch_size, nesterov=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ixxilqyRmGIn",
        "colab_type": "text"
      },
      "source": [
        "TODO: make sure the optimizer is actually identical"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tl2MbkD1mJ5d",
        "colab_type": "text"
      },
      "source": [
        "# create learner"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dm1VX4lmMsQ",
        "colab_type": "text"
      },
      "source": [
        "TODO: find a way to use `nn.CrossEntropyLoss` as loss function like in Part 1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hFPuzq329mqP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = Learner(data, model, metrics=accuracy, loss_func=CrossEntropyFlat(), opt_func=opt)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u9JPIWkAICjc",
        "colab_type": "text"
      },
      "source": [
        "# learning rate schedule\n",
        "\n",
        "lr_schedule from the bag of tricks article:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-2EDNTj6Ejyv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class PiecewiseLinear(namedtuple('PiecewiseLinear', ('knots', 'vals'))):\n",
        "    def __call__(self, t):\n",
        "        return np.interp([t], self.knots, self.vals)[0]\n",
        "\n",
        "sched = PiecewiseLinear([0, 15, 30, 35], [0, 0.1, 0.005, 0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WhuxJLSNFIZc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jzrNsWmBFZgU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "outputId": "ab909914-84a1-4892-80d5-7f54f3680bd0"
      },
      "source": [
        "plt.plot([sched(o) for o in range(35)])"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f4b1f7ace10>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XtcVXW+//HXhzuCgAiiIgoqiqCA\nSdrF7GpZjpF5SX8zc5qZzqNmppuppd3M7KppVjN1znSmqU7T5D3TtLSyycrJROUiXvEKiIqKqKhc\nv78/9nYOkcpWNqx9+Twfjx5u1l5783aFb/Ze67PXEmMMSimlvIOP1QGUUkq1HC19pZTyIlr6Sinl\nRbT0lVLKi2jpK6WUF9HSV0opL6Klr5RSXkRLXymlvIiWvlJKeRE/qwM0FBUVZeLj462OoZRSbmX9\n+vWHjTHRja3ncqUfHx9PVlaW1TGUUsqtiMheR9bT3TtKKeVFtPSVUsqLaOkrpZQX0dJXSikvoqWv\nlFJexKHSF5EhIrJNRApEZPI57h8kIhtEpEZERja4724R2WH/725nBVdKKXXxGi19EfEF3gRuBZKB\nsSKS3GC1fcBvgH80eGwk8AwwAOgPPCMibZoeWyml1KVw5JV+f6DAGLPLGFMFzAEy669gjNljjMkF\n6ho89hbgC2PMUWNMGfAFMMQJuZVy2Mr8A+w+XGF1DKVcgiOlHwsU1vu6yL7MEQ49VkTuFZEsEckq\nLS118KmVatzq7aXc+8F6fv3OWo6fqbY6jlKWc4kDucaYt40xGcaYjOjoRj9FrJRDyk9XM2lhLrER\nwZSUn2HK4k1WR1LKco6UfjEQV+/rTvZljmjKY5Vqkuc+3cyhE5W8+cvLeOiGRBZn72fxRv3xU97N\nkdJfBySKSIKIBABjgCUOPv8K4GYRaWM/gHuzfZlSzeqLzQdZsL6IP17XjfS4CO6/vhsZXdrw1OJN\nFB49ZXU8pSzTaOkbY2qAB7CV9RZgnjEmX0SmicjtACJyuYgUAaOAv4hIvv2xR4HnsP3iWAdMsy9T\nqtkcraji8UV59OoQxoM3JALg5+vD7LvSEeDhORupqW04c6CUdxBjjNUZfiIjI8PoWTZVU9z/jw2s\nzD/AkgcG0qtD2E/u+yS7mIfnZPPwjYk8MriHRQmVcj4RWW+MyWhsPZc4kKuUsyzN2c+y3BLG3dTj\nZ4UPkJkey519Y/nTqh1k7dE3ncr7aOkrj3HoxBme/mQTaXER3Deo63nXezYzhU5tWvHwnGwd41Re\nR0tfeQRjDI8vzON0VS2zRqXh53v+H+3WQf68PiadA8fP8LSOcSovo6WvPMKC9UV8tfUQjw1Jonu7\n0EbX79u5DeNuTOST7P18vLGoBRIq5Rq09JXbKz52mmlLNzMgIZLfXhXv8OP+eH13+sdH8vTifPYd\n0TFO5R209JVbq6szTFqQS60xzByVho+POPxYXx9h9ph0RGDcXB3jVN5BS1+5tQ/X7uW7gsM8NTSZ\nuMhWF/342IhgXhzehw37jvHGVzuaIaFSrkVLX7mtvUcqeHH5Vgb1iGZs/7jGH3Aew9I6MuKyTvz5\n6wJ+3K1jnMqzaekrt1RbZ5g4Pwd/X2HGiFREHN+tcy7PZqYQF9mKR+ZmU35axziV59LSV27pb9/t\nZt2eMqbenkL78KAmP19ooB+v3WUb43xq8SZc7ZPqSjmLlr5yOzsOnuCVldu4OTmG4X0dvbRD4/p2\nbsMjNyWyNGc/izbo2TiVZ9LSV26luraOCfNzCA3044XhfZq8W6ehP1zXnf4JkUz5ZBN7j+jVtpTn\n0dJXbuW//rmT3KJynr+jN9GtA53+/L4+wuy70vH1ER6ek021jnEqD6Olr9zGpuJy3vhqB5npHbmt\nT4dm+z6xEcG8eGcfsgt1jFN5Hi195RYqa2qZMC+HyJAAnr09pdm/3y9SOzKyXyfe1DFO5WG09JVb\neO3LHWw7eILpI1KJaBXQIt9z6u06xqk8j5a+cnnr95bxl292MubyOK5Patdi3zc00I/Xx/Tl4PEz\nPPlxno5xKo+gpa9c2umqWibOz6FDeDBPDu3V4t8/PS6CRwb34NPcEhbqGKfyAFr6yqXNWLGV3Ycr\neGVkKq2D/C3J8PtruzEgIZJnPtnEnsM6xqncm5a+cllrdh7m3e/38Jur4rmqe5RlOX4yxjlXxziV\ne9PSVy7pZGUNj87PJSEqhElDkqyOQ8eIYF66M5WcwmO8/qWOcSr3paWvXNILyzZTUn6amaNSCQ7w\ntToOAENTOzCqXyfe/GcBP+w6YnUcpS6Jlr5yOV9vO8RHPxZy76Bu9OsSaXWcn5h6ewpdzo5xntIx\nTuV+tPSVSzl2qopJC3LpERPKI4MTrY7zMyH2Mc7SE5U8oWOcyg1p6SuXMnVJPkcrqnh1dDqBfq6x\nW6ehNPsY57K8Euav14uqK/eipa9cxmd5JSzO3s+DNyTSOzbc6jgXdHaMc+qSfHbrGKdyI1r6yiUc\nPlnJk4s30Sc2nD9e383qOI06O8bp7+vDuDkbdYxTuQ0tfWU5YwxPLMrj5JkaZo1Ow9/XPX4sO0YE\n8/KdfcgpKmf2F9utjqOUQ9zjX5fyaIuzi1m5+SATbu5Bj5jWVse5KLf26cBdGXH81zc7+ddOHeNU\nrk9LX1mqpPw0Uz7Jp1+XNvznNV2tjnNJpgxLJr5tCOPnZXPsVJXVcZS6IIdKX0SGiMg2ESkQkcnn\nuD9QROba718rIvH25f4i8r6I5InIFhF53LnxlTszxjBpYR41tYZZo9Lw9XHupQ9bim2MM13HOJVb\naLT0RcQXeBO4FUgGxopIcoPV7gHKjDHdgdnAdPvyUUCgMaYP0A+47+wvBKU++rGQ1dtLefy2JOKj\nQqyO0ySpnSKYcHNPlucdYH6WjnEq1+XIK/3+QIExZpcxpgqYA2Q2WCcTeN9+ewFwo9iuWG2AEBHx\nA4KBKuC4U5Irt7bvyCmeX7aZq7u35VcDulgdxynuG9SVK7u2ZepSHeNUrsuR0o8FCut9XWRfds51\njDE1QDnQFtsvgAqgBNgHzDTG6LXnvFxdneHRBTn4iDBjZBo+brpbpyEfH+HVu2zTRw/P2UhVjY5x\nKtfT3Ady+wO1QEcgAZggIj87Wici94pIlohklZaWNnMkZbV31+xh7e6jTPlFMrERwVbHcaoO4cFM\nH9GH3KJyZn+pY5zK9ThS+sVAXL2vO9mXnXMd+66ccOAI8P+Az40x1caYQ8D3QEbDb2CMedsYk2GM\nyYiOjr74v4VyGztLTzLj863cmNSOURmdrI7TLIb07sCYy+P47292smbnYavjKPUTjpT+OiBRRBJE\nJAAYAyxpsM4S4G777ZHAKmMbYdgH3AAgIiHAFcBWZwRX7qemto4J83IIDvDlpTv7YDvs45mmDEsm\noW0I4+fm6BincimNlr59H/0DwApgCzDPGJMvItNE5Hb7au8AbUWkABgPnB3rfBMIFZF8bL883jXG\n5Dr7L6Hcw19W7yK78BjTMnvTLizI6jjNqlWA7WycRyoqmbxQxziV6/BzZCVjzHJgeYNlU+rdPoNt\nPLPh406ea7nyPltKjvPal9sZ2qcDw1I7WB2nRfTpFM6Em3vy8mdbmbuukDH9O1sdSSn9RK5qflU1\ndYyfl0N4sD/P3dHbo3frNHTvNV25qltbnl26mZ2lJ62Oo5SWvmp+f1q1gy0lx3npzlQiQwKsjtOi\nfHzEdm0Afx/GzcnWMU5lOS191axyCo/x1j93MuKyTgxOjrE6jiXahwcxfUQqecXlzPpim9VxlJfT\n0lfN5kx1LRPm59CudSBThjU8c4d3uSWlPWP7d+bt1btYU6BjnMo6Wvqq2cxauY2CQyeZPiKV8GB/\nq+NY7ulf9CIhKoRH5mVTVqFjnMoaWvqqWfy4+yh//W43v7qiM4N66AfuwDbG+caYvhytqGLyolwd\n41SW0NJXTldRWcPE+TnEtWnF47f2sjqOS+kdG86jt/RkRf5B5qwrbPwBSjmZlr5yupc/20ph2Slm\njkojJNChj4J4lf8c2JWB3aOYpmOcygJa+sqpvt1Rygc/7OWeqxPonxBpdRyX5OMjzBqdRpC/no1T\ntTwtfeU0x89U89iCXLpFhzDxlp5Wx3FpMWG2Mc5NxceZtVLHOFXL0dJXTjNt6WYOnahk1uh0gvx9\nrY7j8m5Oac//G9CZv6zexfc6xqlaiJa+coovNh9kwfoi/nhdN9LjIqyO4zaeHppMt2jbRdV1jFO1\nBC191WRHK6p4fFEevTqE8eANiVbHcSvBAb68bh/jnLRQxzhV89PSV0329CebKD9dxauj0wjw0x+p\ni9U7NpzHbkli5eaD/OPHfVbHUR5O/4WqJlmas59luSWMu6kHvTqEWR3Hbd0zMIFrEqN47tPNFBw6\nYXUc5cG09NUlO3T8DE9/son0uAjuG/SzSx+ri+DjI8wclUawvy8PfZRNZU2t1ZGUh9LSV5fEGMPj\ni/I4XVXLrNFp+Pnqj1JTxYQFMWNkGptLjjNzhY5xquah/1LVJVmwvoivth7isSFJdIsOtTqOxxic\nHMMvB3Tmf77dzbc7Sq2OozyQlr66aMXHTjNt6WYGJETy26virY7jcZ4amkz3dqFMmJfDUR3jVE6m\npa8uSl2dYdKCXGqNYeaoNHx8vOfShy3FNsaZzrFTtk846xinciYtfXVRPly7l+8KDvPU0GTiIltZ\nHcdjpXQM57EhPflyy0E+XKtjnMp5tPSVw/YcruDF5VsZ1COasf3jrI7j8X53tW2M8/llOsapnEdL\nXzmkts4wcX4O/r7CjBGpiOhunebm4yPMGpVGqwA/HtQxTuUkWvrKIe98t4usvWU8m5lC+/Agq+N4\njXb2s3FuKTnOK5/rGKdqOi191ajtB08wc8V2bkmJ4Y70WKvjeJ3ByTH86orO/PW73azermOcqmm0\n9NUFVdfWMWFeDqFBfrwwvI/u1rHIk7fZxzjn53DkZKXVcZQb09JXF/TW1zvJKy7nhTt6ExUaaHUc\nrxUc4MsbY/pSfqpaz8apmkRLX53XpuJy/rRqB5npHbm1Twer43i95I5h9jHOQ/xdxzjVJdLSV+dU\nWVPL+HnZRIYE8OztKVbHUXa/uzqBQT2ief7Tzew4qGOc6uJp6atzmv3FDrYfPMn0EalEtAqwOo6y\ns52NM5XQQD8emqNjnOriaemrn1m/9yhvr97JmMvjuD6pndVxVAPtWgcxY6RtjHOGjnGqi+RQ6YvI\nEBHZJiIFIjL5HPcHishc+/1rRSS+3n2pIvIvEckXkTwR0SFvF3aqqoYJ83LoEB7Mk0N7WR1HnceN\nvWL4jyu78I6OcaqL1Gjpi4gv8CZwK5AMjBWR5Aar3QOUGWO6A7OB6fbH+gF/B35vjEkBrgOqnZZe\nOd2Mz7ex58gpXhmVSusgf6vjqAt44rZe9IjRMU51cRx5pd8fKDDG7DLGVAFzgMwG62QC79tvLwBu\nFNtA981ArjEmB8AYc8QYozshXdSagsO8t2YPv7kqnqu6RVkdRzUiyN92UfXy03o2TuU4R0o/Fiis\n93WRfdk51zHG1ADlQFugB2BEZIWIbBCRx871DUTkXhHJEpGs0lJ9q2qFE2eqeXRBLglRIUwakmR1\nHOWgXh3CmDwkia+2HuLvP+y1Oo5yA819INcPGAj80v7ncBG5seFKxpi3jTEZxpiM6OjoZo6kzuWF\nZVsoKT9tu05rgK/VcdRF+O3V8VzbI5rnl21hu45xqkY4UvrFQP3z6HayLzvnOvb9+OHAEWzvClYb\nYw4bY04By4HLmhpaOdfXWw8xZ10h913bjX5d2lgdR10kEdtF1UMD/Xjoo42cqdY9qOr8HCn9dUCi\niCSISAAwBljSYJ0lwN322yOBVca2g3EF0EdEWtl/GVwLbHZOdOUMx05VMWlhLj1jWjPupkSr46hL\nFN06kFdGpbL1wAkd41QX1Gjp2/fRP4CtwLcA84wx+SIyTURut6/2DtBWRAqA8cBk+2PLgFex/eLI\nBjYYY5Y5/6+hLtUzS/I5WlHFrNFpBPrpbh13dkNSDHdf2YW/fb+bf247ZHUc5aLE1Y74Z2RkmKys\nLKtjeIXP8kr4w4cbeOSmHjysr/I9wpnqWjL//D1HKqr4fNw1epI8LyIi640xGY2tp5/I9VKHT1by\n5OJN9IkN54/Xd7M6jnKSIH9fXh+bzvEz1Tw6P0fHONXPaOl7IWMMTyzK42RlDbNGp+Hvqz8GniSp\nfRhP3JrE19tK+d9/6Rin+in91+6FFmcXs3LzQSbe3IMeMa2tjqOawd1XxXN9z2heWL6FbQd0jFP9\nHy19L1NSfpopn+ST0aUN9wzsanUc1UxEhBkj0wgL0jFO9VNa+l7EGMOkhXnU1BpmjkrD10cvfejJ\nbGOcaWw7eIKXP9tqdRzlIrT0vchHPxayenspT9yWRHxUiNVxVAu4vmc7fnNVPO+t2cPXOsap0NL3\nGoVHT/HCss0M7B7FLwd0sTqOakGTb00iqX1rHp2fQ+kJPRunt9PS9wJ1dYaJ83PwEWH6yFR8dLeO\nVzl7Ns7jZ2p4bIGOcXo7LX0v8N6aPazdfZSnhyUTGxFsdRxlgZ7tW/Pkbb34elsp76/ZY3UcZSEt\nfQ+3s/Qk0z/fyo1J7RjVr5PVcZSF/uPKLtyQ1I4XP9vK1gPHrY6jLKKl78FqauuYMC+H4ABfXrqz\nD7br2ihvZRvjTCUsyJ+HP8rWMU4vpaXvwf6yehfZhcd4LrM37cL00sQKokIDmTkqVcc4vZiWvofa\nUnKc177cztDUDgxL62h1HOVCruvZjt9ebRvjXLX1oNVxVAvT0vdAVTV1jJ+XQ3hwAM9l9rY6jnJB\nk4acHePM5dCJM1bHUS1IS98D/WnVDraUHOelO/sQGRJgdRzlgoL8fXljbF9OVtbw6Pxc6up0jNNb\naOl7mOzCY7z1z52MuKwTg5NjrI6jXFiPmNY8ObQX32wv5T0d4/QaWvoe5Ex1LRPmZdOudSBThiVb\nHUe5gV9f0YUbk9rx8mdb2VKiY5zeQEvfg8xcsY2dpRVMH5FKeLC/1XGUGxD7p7TDgv31bJxeQkvf\nQ6zddYR3vt/Nr67ozKAe0VbHUW4kKjSQWaPT2HHoJC8u32J1HNXMtPQ9QEVlDRMX5BDXphWP39rL\n6jjKDV3bI5rfXZ3A//5rL19t0TFOT6al7wFeXL6ForLTzByVRkign9VxlJt6bEhP2xjnAh3j9GRa\n+m7um+2lfLh2H/dcnUD/hEir4yg3FuTvy5/G9rW9c9QxTo+lpe/Gyk9XM2lBLt3bhTLxlp5Wx1Ee\nIDGmNU8N7cXq7aW8q2OcHklL341NW7qZ0pOVzBqVRpC/r9VxlIf41RVduKlXO6Z/tpXN+3WM09No\n6buplfkHWLihiD9e1420uAir4ygPIiK2sd9W/jw8R8c4PY2Wvhs6WlHFEx/n0atDGA/ekGh1HOWB\n2oYG8qp9jPOFZTrG6Um09N2MMYanFudRfrqaV0enEeCn/wtV87gmMZr/HJjABz/s5cvNOsbpKbQx\n3MzS3BKW5x1g3E096NUhzOo4ysM9OqQnyR3CeGxhLoeO6xinJ9DSdyOHjp/h6cWbSI+L4L5BXa2O\no7xAoJ8vb4xN51RVDRPm5+gYpwfQ0ncTxhgmL8rjTHUts0an4eer/+tUy+jerjVPDU3m2x2H+dv3\nu62Oo5rIoeYQkSEisk1ECkRk8jnuDxSRufb714pIfIP7O4vISRGZ6JzY3md+VhGrth5i0pAkukWH\nWh1HeZlfDujM4OQYZny+jfz95VbHUU3QaOmLiC/wJnArkAyMFZGG5+29BygzxnQHZgPTG9z/KvBZ\n0+N6p6KyU0z7dDMDEiL5zVXxVsdRXujsGGdEK9vZOE9X6Rinu3LklX5/oMAYs8sYUwXMATIbrJMJ\nvG+/vQC4UUQEQETuAHYD+c6J7F3q6gyPLcjFGMPMUWn4+IjVkZSXigwJ4NXR6ewsreD5ZZutjqMu\nkSOlHwsU1vu6yL7snOsYY2qAcqCtiIQCk4Bnmx7VO/197V7W7DzCU79IJi6yldVxlJcbmBjFvYO6\n8uHafazMP2B1HHUJmvto4FRgtjHm5IVWEpF7RSRLRLJKS0ubOZL72H24gpeWb+XaHtGMuTzO6jhK\nATDx5p6kdAxj0sJcDuoYp9txpPSLgfqN08m+7JzriIgfEA4cAQYAM0RkDzAOeEJEHmj4DYwxbxtj\nMowxGdHRegEQgNo6w8T5Ofj72val2veWKWW5AD8fXh/Tl9PVtUyYp2Oc7saR0l8HJIpIgogEAGOA\nJQ3WWQLcbb89ElhlbK4xxsQbY+KB14AXjTF/dlJ2j/bXb3exfm8Zz2am0D48yOo4Sv1E93ahTPlF\nCt8VHOad73SM0500Wvr2ffQPACuALcA8Y0y+iEwTkdvtq72DbR9+ATAe+NlYp3Lc9oMnmLVyO7ek\nxHBHesPDJ0q5hrH947glJYYZK7ayqVjHON2FGONab80yMjJMVlaW1TEsU11bx/C3vmf/sTOsfGQQ\nUaGBVkdS6rzKKqoY8vpqQgP9+PTBawgO0FN8W0VE1htjMhpbTz/W6WLe+nonm4qP8+Lw3lr4yuW1\nsY9x7jpcwXM6xukWtPRdyKbicv60agd3pHdkSO8OVsdRyiFXd4/i3mu68o+1+1ihY5wuT0vfRVTW\n1DJ+XjZtQwN49vbeVsdR6qJMuLknvWPDmKxjnC5PS99FzP5iB9sPnuRl+xWLlHInZ8c4z1TXMX5e\nto5xujAtfRewfu9R3l69k7H947i+Zzur4yh1SbpFhzJlWDLfFxzhr9/tsjqOOg8tfYudrqpl4vxc\nOkYE8+TQhuexU8q9jLncNsb5yoptOsbporT0LTb9863sPlzBKyPTCA30szqOUk0iIrx8ZyptQwJ5\naM5GTlXVWB1JNaClb6E1Ow/z3po9/PbqeK7s1tbqOEo5hW2MM43dhyt47lO9qLqr0dK3yIkz1Tw6\nP5euUSE8dkuS1XGUcqqrukdx36BufPTjPj7fpGOcrkRL3yIvLNtCSflpZo5O008xKo80fnAP+sSG\nM3lRLgfKdYzTVWjpW+DrrYeYs66Q31/bjcs6t7E6jlLNwjbGmU5ldR0T5usYp6vQ0m9hx05VMWlh\nLkntW/PwTYlWx1GqWXWNDmXq7bYxzv/5Vsc4XYGWfgt7Zkk+RyuqmDkqjUA/3a2jPN/ojDhu7d2e\nmSt1jNMVaOm3oOV5JXySvZ+Hbkykd2y41XGUahEiwkt39iEqNJCHPtIxTqtp6beQ0hOVPLV4E6md\nwvnDdd2sjqNUi4poZTsb5+4jFTz3qZ6N00pa+i3AGMOTH+dxsrKGWaPS8PfVza68z5Xd2vL7a7vx\n0Y+FfJZXYnUcr6Xt0wI+3ljMys0HefTmniTGtLY6jlKWeeSmHqR2CmfyojxKyk9bHccraek3s5Ly\n0zyzJJ+MLm343cAEq+MoZamzZ+Osrq1j/NwcanWMs8Vp6TcjYwyPLcilptYwc1Qavj5idSSlLJcQ\nFcLUYSn8a9cR3l6tY5wtTUu/Gf3jx318u+MwT9yWRHxUiNVxlHIZozI6cVuf9sxauY3comNWx/Eq\nWvrNZN+RU7ywbAsDu0fxywFdrI6jlEsREV4ankp060AenpNNRaWOcbYULf1mUFdnmLggB18Rpo9M\nxUd36yj1M+Gt/Jl9Vzp7jlQwbamOcbYULf1m8O6aPfy4+yhThiUTGxFsdRylXNYVXdvyh2u7MTdL\nxzhbipa+kxUcOsmMz7dyU692jOzXyeo4Srm8Rwb3IM0+xrn/mI5xNjctfSeqqa1jwvwcggN8efHO\nPojobh2lGuPvW2+Mc162jnE2My19J/rL6l3kFB7juczetGsdZHUcpdxGfFQIU29P4YddR/nL6p1W\nx/FoWvpOsnn/cV77cjtDUzswLK2j1XGUcjuj+nViaGoHXl25nZxCHeNsLlr6TlBVY3tbGh4cwHOZ\nva2Oo5RbEhFevKMP7VoHMm6ujnE2Fy19J3jjqx1sPXCCl+7sQ2RIgNVxlHJb4a38edU+xvns0nyr\n43gkLf0m2rivjLf+WcDIfp0YnBxjdRyl3N4VXdvyx+u6MS+riOU6xul0WvpNcKa6lgnzc2gfFsSU\nYclWx1HKY4y7qQdpcRFMXpirY5xO5lDpi8gQEdkmIgUiMvkc9weKyFz7/WtFJN6+fLCIrBeRPPuf\nNzg3vrVeWbGNXaUVzBiZRliQv9VxlPIY/r4+vH5XOrV1hkfm6hinMzVa+iLiC7wJ3AokA2NFpOHL\n2nuAMmNMd2A2MN2+/DAwzBjTB7gb+MBZwa22dtcR/vb9bn59RRcGJkZZHUcpj3N2jHPt7qP89zc6\nxuksjrzS7w8UGGN2GWOqgDlAZoN1MoH37bcXADeKiBhjNhpj9tuX5wPBIhLojOBWqqisYeKCHDpH\ntmLyrUlWx1HKY420j3HO/kLHOJ3FkdKPBQrrfV1kX3bOdYwxNUA50LbBOiOADcaYyobfQETuFZEs\nEckqLS11NLtlXly+haKy07wyMo2QQD+r4yjlsc6OccaEBfHwnI06xukELXIgV0RSsO3yue9c9xtj\n3jbGZBhjMqKjo1si0iX7ZnspH67dx38OTKB/QqTVcZTyeGfPxrnv6CmmLtExzqZypPSLgbh6X3ey\nLzvnOiLiB4QDR+xfdwI+Bv7DGOPWO+bKT1czaUEu3duFMuHmnlbHUcpr9E+I5P7ruzN/fRHLcnWM\nsykcKf11QKKIJIhIADAGWNJgnSXYDtQCjARWGWOMiEQAy4DJxpjvnRXaKs8uzaf0ZCWzRqUR5O9r\ndRylvMpDNyaSHhfB44tyKdYxzkvWaOnb99E/AKwAtgDzjDH5IjJNRG63r/YO0FZECoDxwNmxzgeA\n7sAUEcm2/9fO6X+LFrAi/wCLNhRz/3XdSIuLsDqOUl7HdjZOHeNsKjHGtTZcRkaGycrKsjrGTxw5\nWcktr62mXesgFt9/NQF++pk2payycH0RE+bn8OgtPbn/+u5Wx3EZIrLeGJPR2HraXo0wxvD0J5so\nP13Nq3elaeErZbE7L4tlWFpHZn+xnWwd47xo2mCNWJpbwvK8AzwyuAdJ7cOsjqOU1xMRnr+j97/H\nOE/qGOdF0dK/gEPHz/D04k2pGcTWAAALmklEQVT07RzBvdd0tTqOUsouPNif18akU3j0FM98omOc\nF0NL/zyMMUxelEdlTS2zRqXh56ubSilXcnl8JA9c352FG4pYmrO/8QcoQEv/vOZnFbFq6yEmDUmi\na3So1XGUUufw0I2J9O0cwRMf51FUdsrqOG5BS/8cispOMe3TzVzRNZK7r4y3Oo5S6jz8fH14/a6+\nGAPj5+boGKcDtPQbqKszPLYgF2MMr4xMw8dHrI6klLqAzm1bMS0zhR/3HOWtrwusjuPytPQb+OCH\nvazZeYSnfpFMXGQrq+MopRwwvG8smekdee2rHWzYV2Z1HJempV/P7sMVvPTZFq7rGc2Yy+Maf4BS\nyiWICM/d0ZsO4UGMm5PNiTPVVkdyWVr6drV1hgnzsgn082X6iFREdLeOUu4kLMif1+5Kp6jsFM/o\n2TjPS0vf7q/f7mLDvmM8e3sKMWFBVsdRSl2CjPhIHrwhkUUbilmiY5znpKUPbD94glkrtzMkpT2Z\n6R2tjqOUaoIHb+hOvy5teHKRjnGei9eXfnVtHePnZdM6yI/nh/fW3TpKuTk/Xx9euysdgEfmZlNT\nW2dxItfi9aX/5tcFbCo+zgvDexMV6vaX71VKAXGRrXjujt6s21PGW/9062s3OZ1Xl35eUTl/XlXA\n8L6xDOndweo4SiknuqNvLHekd+T1r3awfq+OcZ7ltaV/prqWCfOzaRsawNRhKVbHUUo1g2lnxzjn\nbtQxTjuvLf3ZX25n+8GTTB+RSngrf6vjKKWaQViQP6+PSWf/sTNM0bNxAl5a+uv3HuXt1bsY2z+O\n63q65dUblVIO6tclkgdv6M7HG4v5JLvY6jiW87rSP1VVw4R5OcRGBPPk0GSr4yilWsAD13cno0sb\nnvp4E4VHvXuM0+tKf/pnW9lz5BSvjEwjNNDP6jhKqRbg5+vDbPsY5zgvH+P0qtJfU3CY9/+1l99e\nHc+V3dpaHUcp1YLiIlvx/PDerN9bxp+9+GycXlP6J85U8+iCXLpGhfDYLUlWx1FKWSAzPZbhfWN5\n46sdrN971Oo4lvCa0n/+0y2UlJ9m5ug0ggN8rY6jlLLItMwUYtsE8/CcbI574RinV5T+qq0HmZtV\nyH3XduOyzm2sjqOUslDrIH9eu6svJeVn+N2763j3+93kFB6jqsY79vN7/JHMsooqJi3Mo2dMa8bd\nlGh1HKWUC+jXpQ3P3p7Cn1cV8OzSzQAE+vnQJzacvp0juKxzG/p2bkP7cM87464Y41rXlMzIyDBZ\nWVlOe76HPtrI8rwSFt9/Nb1jw532vEopz1BSfpoNe4+xcV8ZG/aVsan4OFX26Z6O4UH07dzG9oug\nSxtSOoYR6Oeau4dFZL0xJqOx9Tz6lf6y3BKW5Oxn/OAeWvhKqXPqEB7M0NRghqbazr9VWVPL5v3H\n2bjvGBv2lbFx3zGW5ZUAEODrQ0psGH3j2nBZlwj6dm5Dx/Agtzo7r8e+0i89UcnNs78hLrIVC/9w\nFf6+XnH4QinVDA4eP8PGff/3biC3qJxK+zGAmLDAn/wS6BMbTpB/y78b8OpX+sYYHl+UR0VVLbNG\npWnhK6WaJCYsiCG92zOkd3vAdh2OLSU/fTfwef4BAPx8hOSOYfbjArbjA53aBLvMuwGPLP1FG4r5\ncstBnrytF4kxra2Oo5TyMP6+PqR2iiC1UwR3XxUP2PYuZBee/SVQxtx1hby3Zg8AUaGB9O0c8e9f\nAqmdwmkVYE39OvRdRWQI8DrgC/zVGPNyg/sDgf8F+gFHgLuMMXvs9z0O3APUAg8ZY1Y4Lf05lJSf\nZurSfC6Pb8PvBiY057dSSql/i24dyODkGAYnxwBQU1vHtoMn2LDvGBv3lrGx8BhfbD4IgK+PkNS+\n9U/eDXRp26pF3g00Wvoi4gu8CQwGioB1IrLEGLO53mr3AGXGmO4iMgaYDtwlIsnAGCAF6Ah8KSI9\njDG1zv6LgG23zmMLcqmpNcwclYavj2u8nVJKeR8/Xx9SOoaT0jGcX1/RBYCjFVVkF5b9e7fQxxuL\n+eCHvQC0aeXP6MvjePzWXs2by4F1+gMFxphdACIyB8gE6pd+JjDVfnsB8Gex/crKBOYYYyqB3SJS\nYH++fzkn/k/948d9fLvjMM/d0ZsubUOa41sopdQliwwJ4IakGG5Isr0bqK0z7Dh0wvZLYG8ZUSHN\nf8lWR0o/Fiis93URMOB86xhjakSkHGhrX/5Dg8fGXnLaC9h35BQvLNvCNYlR/GpA5+b4Fkop5VS2\n3TxhJLUPY2z/luktlxhrEZF7RSRLRLJKS0sv6TnqjKFflzZMH5HqMkfJlVLK1ThS+sVAXL2vO9mX\nnXMdEfEDwrEd0HXksRhj3jbGZBhjMqKjox1PX098VAgf3DOAjhHBl/R4pZTyBo6U/jogUUQSRCQA\n24HZJQ3WWQLcbb89ElhlbJ/6WgKMEZFAEUkAEoEfnRNdKaXUxWp0n759H/0DwApsI5t/M8bki8g0\nIMsYswR4B/jAfqD2KLZfDNjXm4ftoG8NcH9zTe4opZRqnMeehkEppbyJo6dhcIkDuUoppVqGlr5S\nSnkRLX2llPIiWvpKKeVFtPSVUsqLuNz0joiUAnub8BRRwGEnxWkJ7pYXNHNLcbfM7pYXPCtzF2NM\no59udbnSbyoRyXJkbMlVuFte0Mwtxd0yu1te8M7MuntHKaW8iJa+Ukp5EU8s/betDnCR3C0vaOaW\n4m6Z3S0veGFmj9unr5RS6vw88ZW+Ukqp8/CY0heRISKyTUQKRGSy1XkcISJ7RCRPRLJFxCXPMici\nfxORQyKyqd6ySBH5QkR22P9sY2XGhs6TeaqIFNu3dbaI3GZlxvpEJE5EvhaRzSKSLyIP25e77Ha+\nQGZX3s5BIvKjiOTYMz9rX54gImvt3THXfgp5y10g73sisrveNk6/qCc2xrj9f9hO+bwT6AoEADlA\nstW5HMi9B4iyOkcjGQcBlwGb6i2bAUy2354MTLc6pwOZpwITrc52nrwdgMvst1sD24FkV97OF8js\nyttZgFD7bX9gLXAFMA8YY1/+38AfrM7aSN73gJGX+rye8kr/3xdvN8ZUAWcv3q6ayBizGts1EurL\nBN63334fuKNFQzXiPJldljGmxBizwX77BLAF27WkXXY7XyCzyzI2J+1f+tv/M8ANwAL7cpfZzhfI\n2ySeUvrnuni7S/8A2hlgpYisF5F7rQ5zEWKMMSX22weAGCvDXIQHRCTXvvvHZXaV1Cci8UBfbK/q\n3GI7N8gMLrydRcRXRLKBQ8AX2PYQHDPG1NhXcanuaJjXGHN2G79g38azRSTwYp7TU0rfXQ00xlwG\n3ArcLyKDrA50sYztvac7jID9F9ANSAdKgFnWxvk5EQkFFgLjjDHH69/nqtv5HJldejsbY2qNMenY\nrtfdH0iyONIFNcwrIr2Bx7HlvhyIBCZdzHN6Suk7dAF2V2OMKbb/eQj4GNsPoTs4KCIdAOx/HrI4\nT6OMMQft/4DqgP/Bxba1iPhjK88PjTGL7ItdejufK7Orb+ezjDHHgK+BK4EIETl76ViX7I56eYfY\nd60ZY0wl8C4XuY09pfQduXi7SxGREBFpffY2cDOw6cKPchlLgLvtt+8GPrEwi0POlqfdcFxoW4uI\nYLvO9BZjzKv17nLZ7Xy+zC6+naNFJMJ+OxgYjO1YxNfASPtqLrOdz5N3a70XAoLt+MNFbWOP+XCW\nfTTsNf7v4u0vWBzpgkSkK7ZX92C7QP0/XDGziHwEXIftzH4HgWeAxdgmHjpjOyPqaGOMyxw4PU/m\n67DtcjDYpqbuq7e/3FIiMhD4FsgD6uyLn8C2j9wlt/MFMo/FdbdzKrYDtb7YXvDOM8ZMs/9bnINt\nV8lG4Ff2V9GWukDeVUA0tumebOD39Q74Nv68nlL6SimlGucpu3eUUko5QEtfKaW8iJa+Ukp5ES19\npZTyIlr6SinlRbT0lVLKi2jpK6WUF9HSV0opL/L/AS+8x3TLH6X4AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zbLvQmb5H97t",
        "colab_type": "text"
      },
      "source": [
        "So this is basically the one cycle scheduler.\n",
        "\n",
        "TODO: make sure it is actually the learning rate scheduler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_6sOwvl1mdkV",
        "colab_type": "text"
      },
      "source": [
        "# half precision training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dhy7_bcu7JX6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ee1ca0b2-f258-4404-9d86-819689b139e3"
      },
      "source": [
        "learn.to_fp16()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Learner(data=ImageDataBunch;\n",
              "\n",
              "Train: LabelList (50000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "truck,truck,truck,truck,truck\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Valid: LabelList (10000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "truck,truck,truck,truck,truck\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Test: None, model=Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (2): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (3): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (4): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (5): Sequential(\n",
              "    (0): Pool(\n",
              "      (maxpool): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "      (avgpool): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "    )\n",
              "    (1): Flatten()\n",
              "    (2): Linear(in_features=512, out_features=10, bias=True)\n",
              "  )\n",
              "), opt_func=functools.partial(<class 'torch.optim.sgd.SGD'>, lr=0.001, momentum=0.9, weight_decay=0.064, nesterov=True), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7f4b32569ea0>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/root/.fastai/data/cifar10'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[MixedPrecision\n",
              "learn: Learner(data=ImageDataBunch;\n",
              "\n",
              "Train: LabelList (50000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "truck,truck,truck,truck,truck\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Valid: LabelList (10000 items)\n",
              "x: ImageList\n",
              "Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32),Image (3, 32, 32)\n",
              "y: CategoryList\n",
              "truck,truck,truck,truck,truck\n",
              "Path: /root/.fastai/data/cifar10;\n",
              "\n",
              "Test: None, model=Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (2): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (3): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (4): Sequential(\n",
              "    (0): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (idconv): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "    )\n",
              "    (1): ResBlock(\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu1): ReLU(inplace)\n",
              "      (branch): Sequential(\n",
              "        (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (2): ReLU(inplace)\n",
              "        (3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (5): Sequential(\n",
              "    (0): Pool(\n",
              "      (maxpool): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "      (avgpool): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "    )\n",
              "    (1): Flatten()\n",
              "    (2): Linear(in_features=512, out_features=10, bias=True)\n",
              "  )\n",
              "), opt_func=functools.partial(<class 'torch.optim.sgd.SGD'>, lr=0.001, momentum=0.9, weight_decay=0.064, nesterov=True), loss_func=FlattenedLoss of CrossEntropyLoss(), metrics=[<function accuracy at 0x7f4b32569ea0>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/root/.fastai/data/cifar10'), model_dir='models', callback_fns=[functools.partial(<class 'fastai.basic_train.Recorder'>, add_time=True, silent=False)], callbacks=[...], layer_groups=[Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (2): ReLU(inplace)\n",
              "  (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (5): ReLU(inplace)\n",
              "  (6): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (8): ReLU(inplace)\n",
              "  (9): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (11): ReLU(inplace)\n",
              "  (12): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (13): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (14): ReLU(inplace)\n",
              "  (15): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (16): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (17): ReLU(inplace)\n",
              "  (18): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (19): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (20): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (21): ReLU(inplace)\n",
              "  (22): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (23): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (24): ReLU(inplace)\n",
              "  (25): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (26): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (27): ReLU(inplace)\n",
              "  (28): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (29): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (30): ReLU(inplace)\n",
              "  (31): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (32): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (33): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (34): ReLU(inplace)\n",
              "  (35): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (36): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (37): ReLU(inplace)\n",
              "  (38): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (39): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (40): ReLU(inplace)\n",
              "  (41): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (42): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (43): ReLU(inplace)\n",
              "  (44): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (45): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (46): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (47): ReLU(inplace)\n",
              "  (48): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (49): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (50): ReLU(inplace)\n",
              "  (51): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (52): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "  (53): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "  (54): Flatten()\n",
              "  (55): Linear(in_features=512, out_features=10, bias=True)\n",
              ")], add_time=True, silent=False, cb_fns_registered=False)\n",
              "loss_scale: 65536\n",
              "max_noskip: 1000\n",
              "dynamic: True\n",
              "clip: None\n",
              "flat_master: False\n",
              "max_scale: 16777216], layer_groups=[Sequential(\n",
              "  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (2): ReLU(inplace)\n",
              "  (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (5): ReLU(inplace)\n",
              "  (6): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (8): ReLU(inplace)\n",
              "  (9): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (11): ReLU(inplace)\n",
              "  (12): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (13): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (14): ReLU(inplace)\n",
              "  (15): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (16): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (17): ReLU(inplace)\n",
              "  (18): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (19): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (20): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (21): ReLU(inplace)\n",
              "  (22): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (23): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (24): ReLU(inplace)\n",
              "  (25): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (26): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (27): ReLU(inplace)\n",
              "  (28): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (29): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (30): ReLU(inplace)\n",
              "  (31): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (32): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (33): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (34): ReLU(inplace)\n",
              "  (35): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (36): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (37): ReLU(inplace)\n",
              "  (38): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (39): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (40): ReLU(inplace)\n",
              "  (41): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "  (42): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (43): ReLU(inplace)\n",
              "  (44): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (45): Conv2d(256, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "  (46): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (47): ReLU(inplace)\n",
              "  (48): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (49): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (50): ReLU(inplace)\n",
              "  (51): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "  (52): MaxPool2d(kernel_size=4, stride=4, padding=0, dilation=1, ceil_mode=False)\n",
              "  (53): AvgPool2d(kernel_size=4, stride=4, padding=0)\n",
              "  (54): Flatten()\n",
              "  (55): Linear(in_features=512, out_features=10, bias=True)\n",
              ")], add_time=True, silent=False, cb_fns_registered=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cvzaCwzmmgNy",
        "colab_type": "text"
      },
      "source": [
        "# learning rate"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JrQ2ctUqmbeq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# learn.lr_find()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tl8zAhGkmfUD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# learn.recorder.plot()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lh6GxXeBmkzn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lr = 1e-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yzu9P7NNmj-G",
        "colab_type": "text"
      },
      "source": [
        "# training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h9XbDER7N8Hc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        },
        "outputId": "7bed4bc9-cf56-4e3c-aa92-bbac51c505bd"
      },
      "source": [
        "# %%prun -s cumulative -l 40\n",
        "# learn.fit_one_cycle(1, max_lr=lr)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.877775</td>\n",
              "      <td>0.862519</td>\n",
              "      <td>0.696200</td>\n",
              "      <td>02:04</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            " "
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rSG-8f1x4q10",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "3379b007-8570-4242-dcac-b08da6a48264"
      },
      "source": [
        "# with torch.autograd.profiler.profile() as prof:\n",
        "#   learn.fit_one_cycle(1, max_lr=lr)\n",
        "# print(prof.key_averages().table(sort_by=\"self_cpu_time_total\"))"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.177362</td>\n",
              "      <td>1.135423</td>\n",
              "      <td>0.589600</td>\n",
              "      <td>00:55</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------\n",
            "Name                                  Self CPU total %   Self CPU total      CPU total %        CPU total     CPU time avg     CUDA total %       CUDA total    CUDA time avg  Number of Calls\n",
            "------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------\n",
            "_local_scalar_dense                            28.68%           7.683s           28.68%           7.683s        358.128us              NaN          0.000us          0.000us            21454\n",
            "to                                             18.97%           5.083s           19.37%           5.188s        197.461us              NaN          0.000us          0.000us            26274\n",
            "cudnn_convolution_backward                      8.82%           2.364s            8.82%           2.364s        303.060us              NaN          0.000us          0.000us             7800\n",
            "pin_memory                                      8.41%           2.254s            8.45%           2.265s          2.414ms              NaN          0.000us          0.000us              938\n",
            "cudnn_convolution                               5.20%           1.393s            5.20%           1.393s        148.479us              NaN          0.000us          0.000us             9380\n",
            "mul_                                            4.44%           1.189s            4.44%           1.189s         28.264us              NaN          0.000us          0.000us            42066\n",
            "add_                                            3.99%           1.070s            3.99%           1.070s         22.112us              NaN          0.000us          0.000us            48384\n",
            "add                                             3.43%        919.341ms            3.43%        919.341ms         32.018us              NaN          0.000us          0.000us            28713\n",
            "sum                                             2.60%        697.788ms            2.60%        697.788ms         32.531us              NaN          0.000us          0.000us            21450\n",
            "zero_                                           2.40%        641.927ms            2.40%        641.927ms         15.240us              NaN          0.000us          0.000us            42120\n",
            "cudnn_batch_norm                                2.02%        542.037ms            2.02%        542.037ms         72.233us              NaN          0.000us          0.000us             7504\n",
            "torch::autograd::AccumulateGrad                 2.02%        540.504ms            2.02%        540.504ms         25.665us              NaN          0.000us          0.000us            21060\n",
            "div_                                            1.70%        455.219ms            1.70%        455.219ms         21.615us              NaN          0.000us          0.000us            21060\n",
            "cudnn_batch_norm_backward                       1.38%        369.229ms            1.38%        369.229ms         59.171us              NaN          0.000us          0.000us             6240\n",
            "relu_                                           0.79%        210.723ms            0.79%        210.723ms         28.081us              NaN          0.000us          0.000us             7504\n",
            "threshold_backward                              0.67%        180.782ms            0.67%        180.782ms         28.971us              NaN          0.000us          0.000us             6240\n",
            "empty                                           0.41%        110.881ms            0.41%        110.881ms          8.058us              NaN          0.000us          0.000us            13761\n",
            "addmm                                           0.30%         81.596ms            0.30%         81.596ms        173.979us              NaN          0.000us          0.000us              469\n",
            "CudnnBatchNormBackward                          0.30%         80.290ms            1.73%        463.435ms         74.268us              NaN          0.000us          0.000us             6240\n",
            "_convolution                                    0.27%         71.046ms            5.53%           1.482s        158.011us              NaN          0.000us          0.000us             9380\n",
            "_batch_norm_impl_index                          0.26%         70.605ms            2.42%        649.537ms         86.559us              NaN          0.000us          0.000us             7504\n",
            "contiguous                                      0.26%         69.170ms            0.26%         69.170ms          1.302us              NaN          0.000us          0.000us            53140\n",
            "item                                            0.24%         65.112ms           28.92%           7.748s        361.163us              NaN          0.000us          0.000us            21454\n",
            "CudnnConvolutionBackward                        0.22%         58.983ms            9.04%           2.423s        310.622us              NaN          0.000us          0.000us             7800\n",
            "mul                                             0.21%         56.248ms            0.21%         56.248ms         34.340us              NaN          0.000us          0.000us             1638\n",
            "nll_loss_forward                                0.15%         39.802ms            0.15%         39.802ms         84.866us              NaN          0.000us          0.000us              469\n",
            "ReluBackward1                                   0.14%         37.225ms            0.81%        218.006ms         34.937us              NaN          0.000us          0.000us             6240\n",
            "cat                                             0.13%         33.744ms            0.13%         33.744ms         71.948us              NaN          0.000us          0.000us              469\n",
            "sub                                             0.11%         28.348ms            0.11%         28.348ms         60.444us              NaN          0.000us          0.000us              469\n",
            "_log_softmax                                    0.10%         27.921ms            0.10%         27.921ms         59.533us              NaN          0.000us          0.000us              469\n",
            "batch_norm                                      0.10%         26.593ms            2.52%        676.129ms         90.103us              NaN          0.000us          0.000us             7504\n",
            "detach_                                         0.09%         24.399ms            0.09%         24.399ms          0.567us              NaN          0.000us          0.000us            43058\n",
            "view                                            0.09%         22.999ms            0.09%         22.999ms         11.764us              NaN          0.000us          0.000us             1955\n",
            "mm                                              0.08%         22.175ms            0.08%         22.175ms         28.430us              NaN          0.000us          0.000us              780\n",
            "conv2d                                          0.08%         21.722ms            5.67%           1.520s        162.072us              NaN          0.000us          0.000us             9380\n",
            "max_pool2d_with_indices                         0.08%         20.861ms            0.08%         20.861ms         44.479us              NaN          0.000us          0.000us              469\n",
            "avg_pool2d_backward                             0.08%         20.476ms            0.08%         20.476ms         52.502us              NaN          0.000us          0.000us              390\n",
            "div                                             0.07%         18.515ms            0.07%         18.515ms         21.529us              NaN          0.000us          0.000us              860\n",
            "nll_loss_backward                               0.07%         18.503ms            0.07%         18.503ms         47.442us              NaN          0.000us          0.000us              390\n",
            "max_pool2d_with_indices_backward                0.06%         16.836ms            0.06%         16.836ms         43.170us              NaN          0.000us          0.000us              390\n",
            "convolution                                     0.06%         16.377ms            5.59%           1.499s        159.757us              NaN          0.000us          0.000us             9380\n",
            "avg_pool2d                                      0.05%         14.647ms            0.05%         14.647ms         31.230us              NaN          0.000us          0.000us              469\n",
            "_th_eq                                          0.05%         12.851ms            0.05%         12.851ms        162.674us              NaN          0.000us          0.000us               79\n",
            "_log_softmax_backward_data                      0.05%         12.598ms            0.05%         12.598ms         32.303us              NaN          0.000us          0.000us              390\n",
            "max                                             0.04%         10.821ms            0.04%         10.821ms        136.972us              NaN          0.000us          0.000us               79\n",
            "unsigned short                                  0.03%          8.241ms            0.03%          8.241ms          4.062us              NaN          0.000us          0.000us             2029\n",
            "unsqueeze                                       0.03%          7.963ms            0.03%          7.963ms          4.245us              NaN          0.000us          0.000us             1876\n",
            "_th_set_                                        0.03%          7.309ms            0.03%          7.309ms          3.896us              NaN          0.000us          0.000us             1876\n",
            "detach                                          0.03%          6.951ms            0.03%          6.951ms          6.645us              NaN          0.000us          0.000us             1046\n",
            "max_pool2d                                      0.01%          3.985ms            0.09%         24.846ms         52.976us              NaN          0.000us          0.000us              469\n",
            "mean                                            0.01%          3.941ms            0.01%          3.941ms         49.888us              NaN          0.000us          0.000us               79\n",
            "AddmmBackward                                   0.01%          3.941ms            0.11%         29.718ms         76.200us              NaN          0.000us          0.000us              390\n",
            "transpose                                       0.01%          3.430ms            0.01%          3.430ms          2.583us              NaN          0.000us          0.000us             1328\n",
            "NllLossBackward                                 0.01%          3.368ms            0.08%         21.871ms         56.079us              NaN          0.000us          0.000us              390\n",
            "set_                                            0.01%          3.015ms            0.04%         10.324ms          5.503us              NaN          0.000us          0.000us             1876\n",
            "reshape                                         0.01%          2.911ms            0.02%          5.542ms          7.105us              NaN          0.000us          0.000us              780\n",
            "MulBackward0                                    0.01%          2.782ms            0.07%         19.370ms         49.667us              NaN          0.000us          0.000us              390\n",
            "clone                                           0.01%          2.673ms            0.01%          2.673ms         24.747us              NaN          0.000us          0.000us              108\n",
            "as_strided                                      0.01%          2.631ms            0.01%          2.631ms          3.372us              NaN          0.000us          0.000us              780\n",
            "AddBackward0                                    0.01%          2.251ms            0.01%          2.251ms          0.721us              NaN          0.000us          0.000us             3120\n",
            "MaxPool2DWithIndicesBackward                    0.01%          2.173ms            0.07%         19.010ms         48.743us              NaN          0.000us          0.000us              390\n",
            "LogSoftmaxBackward                              0.01%          1.911ms            0.05%         14.509ms         37.203us              NaN          0.000us          0.000us              390\n",
            "torch::autograd::CopyBackwards                  0.01%          1.875ms            0.05%         14.374ms         36.856us              NaN          0.000us          0.000us              390\n",
            "log_softmax                                     0.01%          1.839ms            0.11%         29.760ms         63.455us              NaN          0.000us          0.000us              469\n",
            "slice                                           0.01%          1.693ms            0.01%          1.693ms          2.170us              NaN          0.000us          0.000us              780\n",
            "nll_loss                                        0.01%          1.651ms            0.15%         41.453ms         88.386us              NaN          0.000us          0.000us              469\n",
            "AvgPool2DBackward                               0.01%          1.611ms            0.08%         22.087ms         56.633us              NaN          0.000us          0.000us              390\n",
            "CatBackward                                     0.01%          1.346ms            0.02%          4.319ms         11.075us              NaN          0.000us          0.000us              390\n",
            "ViewBackward                                    0.00%          1.298ms            0.03%          6.840ms          8.769us              NaN          0.000us          0.000us              780\n",
            "narrow                                          0.00%          1.281ms            0.01%          2.974ms          3.813us              NaN          0.000us          0.000us              780\n",
            "TBackward                                       0.00%        558.003us            0.01%          1.927ms          4.942us              NaN          0.000us          0.000us              390\n",
            "TransposeBackward0                              0.00%        548.340us            0.00%          1.326ms          3.400us              NaN          0.000us          0.000us              390\n",
            "argmax                                          0.00%        451.589us            0.04%         11.272ms        142.688us              NaN          0.000us          0.000us               79\n",
            "eq                                              0.00%        334.114us            0.05%         13.185ms        166.904us              NaN          0.000us          0.000us               79\n",
            "is_floating_point                               0.00%        251.189us            0.00%        251.189us          0.561us              NaN          0.000us          0.000us              448\n",
            "torch::autograd::GraphRoot                      0.00%        234.270us            0.00%        234.270us          0.601us              NaN          0.000us          0.000us              390\n",
            "stack                                           0.00%        124.513us            0.00%        124.513us        124.513us              NaN          0.000us          0.000us                1\n",
            "random_                                         0.00%         49.978us            0.00%         49.978us         24.989us              NaN          0.000us          0.000us                2\n",
            "is_complex                                      0.00%          1.509us            0.00%          1.509us          0.755us              NaN          0.000us          0.000us                2\n",
            "------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------\n",
            "Self CPU time total: 26.789s\n",
            "CUDA time total: 0.000us\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7xiLFBFm91KX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "dc126c73-996e-478f-93bc-1a9d7bfbc0ba"
      },
      "source": [
        "learn.fit_one_cycle(35, max_lr=lr)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.755360</td>\n",
              "      <td>0.798539</td>\n",
              "      <td>0.721900</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.724884</td>\n",
              "      <td>0.740708</td>\n",
              "      <td>0.746300</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.689003</td>\n",
              "      <td>0.795453</td>\n",
              "      <td>0.729000</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.643591</td>\n",
              "      <td>0.749863</td>\n",
              "      <td>0.747000</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.619617</td>\n",
              "      <td>0.678545</td>\n",
              "      <td>0.772600</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>5</td>\n",
              "      <td>0.557650</td>\n",
              "      <td>0.712036</td>\n",
              "      <td>0.764300</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>6</td>\n",
              "      <td>0.525982</td>\n",
              "      <td>0.628469</td>\n",
              "      <td>0.789800</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>7</td>\n",
              "      <td>0.489476</td>\n",
              "      <td>0.601703</td>\n",
              "      <td>0.799400</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>8</td>\n",
              "      <td>0.446550</td>\n",
              "      <td>0.513119</td>\n",
              "      <td>0.827200</td>\n",
              "      <td>01:59</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>9</td>\n",
              "      <td>0.411602</td>\n",
              "      <td>0.566839</td>\n",
              "      <td>0.811200</td>\n",
              "      <td>01:59</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>10</td>\n",
              "      <td>0.406884</td>\n",
              "      <td>0.532525</td>\n",
              "      <td>0.819400</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>11</td>\n",
              "      <td>0.371704</td>\n",
              "      <td>0.536400</td>\n",
              "      <td>0.823000</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>12</td>\n",
              "      <td>0.368738</td>\n",
              "      <td>0.477462</td>\n",
              "      <td>0.836900</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>13</td>\n",
              "      <td>0.348643</td>\n",
              "      <td>0.509209</td>\n",
              "      <td>0.833700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>14</td>\n",
              "      <td>0.324770</td>\n",
              "      <td>0.483217</td>\n",
              "      <td>0.840300</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>15</td>\n",
              "      <td>0.311770</td>\n",
              "      <td>0.444769</td>\n",
              "      <td>0.849200</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>16</td>\n",
              "      <td>0.297338</td>\n",
              "      <td>0.446841</td>\n",
              "      <td>0.854700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>17</td>\n",
              "      <td>0.284720</td>\n",
              "      <td>0.420779</td>\n",
              "      <td>0.862800</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>18</td>\n",
              "      <td>0.267742</td>\n",
              "      <td>0.400152</td>\n",
              "      <td>0.866700</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>19</td>\n",
              "      <td>0.258534</td>\n",
              "      <td>0.410638</td>\n",
              "      <td>0.867700</td>\n",
              "      <td>01:58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>20</td>\n",
              "      <td>0.246835</td>\n",
              "      <td>0.391739</td>\n",
              "      <td>0.871300</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>21</td>\n",
              "      <td>0.228389</td>\n",
              "      <td>0.374244</td>\n",
              "      <td>0.878100</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>22</td>\n",
              "      <td>0.214114</td>\n",
              "      <td>0.377271</td>\n",
              "      <td>0.877700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>23</td>\n",
              "      <td>0.198912</td>\n",
              "      <td>0.329379</td>\n",
              "      <td>0.894000</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>24</td>\n",
              "      <td>0.185296</td>\n",
              "      <td>0.321134</td>\n",
              "      <td>0.896200</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>25</td>\n",
              "      <td>0.163980</td>\n",
              "      <td>0.353952</td>\n",
              "      <td>0.885100</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>26</td>\n",
              "      <td>0.158317</td>\n",
              "      <td>0.340016</td>\n",
              "      <td>0.891700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>27</td>\n",
              "      <td>0.143901</td>\n",
              "      <td>0.331729</td>\n",
              "      <td>0.898400</td>\n",
              "      <td>01:56</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>28</td>\n",
              "      <td>0.119838</td>\n",
              "      <td>0.330769</td>\n",
              "      <td>0.900500</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>29</td>\n",
              "      <td>0.104753</td>\n",
              "      <td>0.307226</td>\n",
              "      <td>0.907100</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>30</td>\n",
              "      <td>0.082374</td>\n",
              "      <td>0.285755</td>\n",
              "      <td>0.914900</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>31</td>\n",
              "      <td>0.076399</td>\n",
              "      <td>0.285294</td>\n",
              "      <td>0.914100</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>32</td>\n",
              "      <td>0.072202</td>\n",
              "      <td>0.276401</td>\n",
              "      <td>0.918600</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>33</td>\n",
              "      <td>0.062335</td>\n",
              "      <td>0.275956</td>\n",
              "      <td>0.917400</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>34</td>\n",
              "      <td>0.058365</td>\n",
              "      <td>0.274776</td>\n",
              "      <td>0.917700</td>\n",
              "      <td>01:57</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LKATetdEy86y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}