[Bug already fixed] RAdam implementation comparison to verify potential bug in buffer implementation

radam_experiments.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.1"
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    },
    "colab": {
      "name": "RAdam_experiments.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/sholderbach/a92e15fe8588d62f1804e9b2c508f0ce/radam_experiments.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PP_tOaRpc5Rg",
        "colab_type": "text"
      },
      "source": [
        "# Small experiments to test the behavior of RAdam implementations on different learning rates"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YCHF4qfadHKs",
        "colab_type": "text"
      },
      "source": [
        "Test of RAdam is done with the fastai API training a pretrained ResNet18 on MNIST. Fastai's ResNet is split in 3 layer groups (= optimizer param_groups), allowing for training at different learning rates of those groups for better transfer learning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vbw660rLeELB",
        "colab_type": "text"
      },
      "source": [
        "## Hypothesis\n",
        "The buffering functionality of the reference implementation [https://github.com/LiyuanLucasLiu/RAdam](https://github.com/LiyuanLucasLiu/RAdam) reuses a cached factor containing the learning rate for all layer groups and thereby breaks the chance to use different learning rates for different param groups."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9qaK-7COhMwA",
        "colab_type": "text"
      },
      "source": [
        "## Experiment\n",
        "\n",
        "Train a ResNet with different learning rates passed to the parameter groups with different implementations of RAdam\n",
        "\n",
        "To verify that the group wise learning rates are ignored compare the results/learning curve of the buffered implementation with training the whole network with the lowest learning rate (used for the first parameter group)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iy_RZWMsfA7A",
        "colab_type": "text"
      },
      "source": [
        "## Setup \n",
        "\n",
        "Apart from fastai I use a [library](https://github.com/mjendrusch/torchsupport) developed by a colleague to which I contributed another RAdam implementation (and checked out the original code into for this comparison).\n",
        "When including the code directly from [https://github.com/LiyuanLucasLiu/RAdam](https://github.com/LiyuanLucasLiu/RAdam) the install/imports of `torchsupport` can be omitted."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4_n1KIikWUPZ",
        "colab_type": "code",
        "outputId": "c8c11343-3ef1-49cf-e2eb-547a2e0cdb04",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 233
        }
      },
      "source": [
        "!pip install git+https://github.com/sholderbach/torchsupport.git@impl_radam"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting git+https://github.com/sholderbach/torchsupport.git@impl_radam\n",
            "  Cloning https://github.com/sholderbach/torchsupport.git (to revision impl_radam) to /tmp/pip-req-build-187og2_f\n",
            "  Running command git clone -q https://github.com/sholderbach/torchsupport.git /tmp/pip-req-build-187og2_f\n",
            "  Running command git checkout -b impl_radam --track origin/impl_radam\n",
            "  Switched to a new branch 'impl_radam'\n",
            "  Branch 'impl_radam' set up to track remote branch 'impl_radam' from 'origin'.\n",
            "Requirement already satisfied (use --upgrade to upgrade): torchsupport==0.0.1 from git+https://github.com/sholderbach/torchsupport.git@impl_radam in /usr/local/lib/python3.6/dist-packages\n",
            "Building wheels for collected packages: torchsupport\n",
            "  Building wheel for torchsupport (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for torchsupport: filename=torchsupport-0.0.1-cp36-none-any.whl size=108371 sha256=b246eac9006078fe2b1be591e9aa5581c766fbeb7f431cb0c8ca713fbf8a0c8f\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-uha8hwmk/wheels/46/9b/1f/ab957bedc9ebbc22611e4a6412c766f14caec5e6992b4e4d62\n",
            "Successfully built torchsupport\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:18:47.692114Z",
          "start_time": "2019-09-02T14:18:44.976794Z"
        },
        "id": "bCfViGSLWQsH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from fastai.vision import *\n",
        "from torchsupport.training.optim import RAdam\n",
        "from torchsupport.training.radam import PlainRAdam as pradam\n",
        "from torchsupport.training.radam import RAdam as bradam\n",
        "import torch\n",
        "from functools import partial"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:18:47.743789Z",
          "start_time": "2019-09-02T14:18:47.694360Z"
        },
        "id": "9t-A6k-4WQsT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "path = untar_data(URLs.MNIST)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:18:48.526878Z",
          "start_time": "2019-09-02T14:18:47.745610Z"
        },
        "id": "blXY6Iu8WQsf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data = ImageDataBunch.from_folder(path, train='training', valid='testing')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:18:48.531486Z",
          "start_time": "2019-09-02T14:18:48.528626Z"
        },
        "id": "UK9MnQQIWQsl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "opt_radam_own = partial(RAdam) # own implementation\n",
        "opt_radam_plain = partial(pradam) # PlainRAdam\n",
        "opt_radam_buffered = partial(bradam) # Buffered version RAdam "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i9sbkkn7gDOU",
        "colab_type": "text"
      },
      "source": [
        "## RAdam reference implementation (with buffer)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:18:48.925207Z",
          "start_time": "2019-09-02T14:18:48.533209Z"
        },
        "id": "6XPViYhXWQst",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = cnn_learner(data, models.resnet18, opt_func=opt_radam_buffered, metrics=[accuracy])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:18:48.932826Z",
          "start_time": "2019-09-02T14:18:48.926893Z"
        },
        "id": "110lk_pwWQsy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn.unfreeze()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.113881Z",
          "start_time": "2019-09-02T14:18:48.937959Z"
        },
        "id": "0MFrC3WSWQs4",
        "colab_type": "code",
        "outputId": "a4c3ba46-3648-4355-da46-5d2d3b14f012",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "source": [
        "learn.fit(1, [1e-5, 1e-4, 1e-3])"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.121780</td>\n",
              "      <td>0.792971</td>\n",
              "      <td>0.809100</td>\n",
              "      <td>01:12</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.121148Z",
          "start_time": "2019-09-02T14:18:44.983Z"
        },
        "id": "MiIBI5uxWQs_",
        "colab_type": "code",
        "outputId": "34527e09-829d-45e3-8e9a-5ba77b8ba237",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        }
      },
      "source": [
        "learn.recorder.plot_losses()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VFX6x/HPkwIhAUICoQYITULo\nSQCVIsWVsioWEBFWwVXsdXVFd9XV1dV1LaDrWlaBH66CFWxgAxQLUkLvvYQaQkkglJTz+2PGMSol\nQCaTZL7v12tezNx75ua5N0OeOefc+1xzziEiIgIQEugARESk9FBSEBERHyUFERHxUVIQEREfJQUR\nEfFRUhARER8lBRER8VFSEBERHyUFERHxCQt0AKeqRo0aLiEhIdBhiIiUKWlpabudc3Ena1fmkkJC\nQgLz5s0LdBgiImWKmW0qSjsNH4mIiI+SgoiI+CgpiIiIT5mbUxCR8iM3N5f09HQOHz4c6FDKjYiI\nCOLj4wkPDz+t9yspiEjApKenU6VKFRISEjCzQIdT5jnnyMzMJD09nUaNGp3WNjR8JCIBc/jwYapX\nr66EUEzMjOrVq59Rz0tJQUQCSgmheJ3p8Qza4aMvlu0g63Ae9WMqUTkijJZ1owMdkohIwAVlUli7\nK5sRb6T9YtnGJ38foGhEJFAyMzPp1asXADt27CA0NJS4OM9Fv3PmzKFChQon3cbw4cMZOXIkzZs3\n92usJSUok8J3a3b/ZtnRvAIqhGk0TSSYVK9enYULFwLwt7/9jcqVK3PPPff8oo1zDuccISHH/vsw\nduxYv8dZkoLyr+CqnQeoFhnO6sf60rZ+NQA+WbwtwFGJSGmxdu1akpKSGDJkCC1btmT79u2MGDGC\n1NRUWrZsyaOPPupr26VLFxYuXEheXh7VqlVj5MiRtG3blnPOOYddu3YFcC9OT1D1FI7mFVDgHBt2\nH6BJXGUqhIUw+eZz6T1qJq/OXM8l7eoREqJJL5FAeOTjZSzfllWs20yqW5WHL2p5Wu9duXIl48eP\nJzU1FYAnn3yS2NhY8vLy6NGjBwMGDCApKekX79m/fz/nnXceTz75JHfffTdjxoxh5MiRZ7wfJSmo\negoXvfAdiQ9+xo/r9xAT6RkrNDNu6dGUlTuyGfvDRg4cyQtwlCJSGjRp0sSXEAAmTJhAcnIyycnJ\nrFixguXLl//mPZUqVaJv374ApKSksHHjxpIKt9gEVU9h1c5s3/P0vTm+579vXYc7317I3z9Zzvdr\ndzNmWIdAhCcS1E73G72/REVF+Z6vWbOG0aNHM2fOHKpVq8bQoUOPeS1A4Ynp0NBQ8vLK3pfMoOop\nFFa3WiXf87DQEF4emgLA9JW76DNqJiPGz+PZL1ezbd+hQIUoIqVEVlYWVapUoWrVqmzfvp3PP/88\n0CH5TdAmhacGtPnF694ta7Py731IaRjDyh3ZfLF8J89PW8N97y8u8jb3HjzK1WPm8NnS7cUdrogE\nUHJyMklJSSQmJnL11VfTuXPnQIfkN+acC3QMpyQ1NdWdzk12juYVcNZfpwKw4MHfERN17POPc/ML\nmDR/KxsyD3LgcB5v/LiJYecmMLJvIhHhocd8z2dLd3Dn2ws4nFvgW3Zbz6ZcnhxPQo2oY75HRGDF\nihW0aNEi0GGUO8c6rmaW5pxLPc5bfIJmTuHQ0XwAHrww6bgJASA8NIQrOtQHIC+/gBCDcT9sZNwP\nG+mZWJPOTWuwM+swDWIjuapjAzbtyeH2iQs4mldA2/hoLm5Xj6lLtvPC9LX85+t1dEiI4bFLWhEf\nE3ncpCIiUloETVLIyfVM+ERWKPof5rDQEB7p34rKEWG8OGMd01fuYvrKn887/uvkpQBUjQhj+p/O\nIz4mEoBrOyewamc2D3+4jB/X7+H8Z2fSJj6a5AYxdGoUS9/WdYpxz0REik/wJAVvT+FUksJP7u2d\nyD0XNOdwbgGb9hykZpUIkv/+pW/9n/sk+hICeE5zTaxdlf9d14mvV2Vw/fh5LE7fz+L0/Yz7YSNX\ndqjPlR0b0M574ZyISGkRNEnhp+GjSqc5hGNmVKoQSmLtqgAsevgCKoaFcOho/nGHo8JDQ/hdUi3W\n/aMfs9dnciSvgMc+Xc7EuVt4Z94WBnVoQEL1SKIqhnFBy1rUrBJxejsnIlJMgiYp/NxTKJ5djq7k\nuatRUeYJQkOMc5vWAKBHYk3W7MzmmjFzmDBns6/Ngx8uZWSfRPq3q0dMVDirdmTTul60ygqLSIkK\noqTgmVOodBrDR8WtWa0q/HB/L75avpMpS7bTpVkNJi3YyhNTV/LE1JW+djd0a8w9vZvz4OSl5BU4\nDh7Jo0lcZRZv3U9MZDi9WtTiwtZ1VJpDRIpN0FyncOgM5hT85fykWjw7qB2XJcfz8tAU/nZREnf0\nasbv23gmol+ZuZ6khz5j4twtvJeWztSlO/j3jLXMXJ3Bhwu3cfuEBdz73mLfvonIqenRo8dvLkQb\nNWoUN91003HfU7lyZQC2bdvGgAEDjtmme/funOzU+VGjRpGT83NlhX79+rFv376ihu43QdNTaFKz\nMrf3akatqqVz3D6qYhjDOv98T9WHLzzMm7M3s3pnNr9vU4c60RFs2J3DzqzDXNu5EUfzC3hxxlpe\nnbmeeZv2MLJPIr1b1lavQeQUDB48mIkTJ9K7d2/fsokTJ/LUU0+d9L1169blvffeO+2fPWrUKIYO\nHUpkpOcklSlTppz2torVT7XCy8ojJSXFiUdBQYH7x5TlrvH9n7qG933i7ntvkSsoKAh0WCJFtnz5\n8oD+/MzMTBcXF+eOHDninHNuw4YNrn79+i4rK8v17NnTtW/f3rVq1cpNnjzZ956oqChf25YtWzrn\nnMvJyXGDBg1yiYmJ7pJLLnEdO3Z0c+fOdc45d+ONN7qUlBSXlJTkHnroIeecc6NHj3bh4eGuVatW\nrnv37s455xo2bOgyMjKcc84988wzrmXLlq5ly5buueee8/28xMREd91117mkpCT3u9/9zuXk5Bxz\nv451XIF5rgh/Y4Omp1AemRn3923B7T2b8fz0NbzyzXqWbtvPwJT69G9Xl2qRJ79rlEipMXUk7FhS\nvNus3Rr6Pnnc1bGxsXTs2JGpU6fSv39/Jk6cyBVXXEGlSpWYNGkSVatWZffu3Zx99tlcfPHFxz3x\n46WXXiIyMpIVK1awePFikpOTfesef/xxYmNjyc/Pp1evXixevJjbb7+dZ599lhkzZlCjRo1fbCst\nLY2xY8cye/ZsnHN06tSJ8847j5iYGNasWcOECRP473//yxVXXMH777/P0KFDi+dYeQXNnEJ5FlUx\njJF9ErmpexOWbs3i4Y+W0WfUt+zM+m0VRxH5pZ+GkMAzdDR48GCcczzwwAO0adOG888/n61bt7Jz\n587jbmPmzJm+P85t2rShTZufa6u98847JCcn0759e5YtW3bMktuFfffdd1x66aVERUVRuXJlLrvs\nMr799lsAGjVqRLt27QD/leZWT6GcMDP+3Ls5XZvWYN+hXO58eyHXjptLrxa1SG0YQ9dmNXR6q5Ru\nJ/hG70/9+/fnrrvuYv78+eTk5JCSksK4cePIyMggLS2N8PBwEhISjlkq+2Q2bNjA008/zdy5c4mJ\niWHYsGGntZ2fVKxY0fc8NDSUQ4eKv4qzegrliJnneoh+revwxKWtWb0zm+enreHqMXNodP8UHpi0\nhFU7sk++IZEgUrlyZXr06MG1117L4MGDAc8d1GrWrEl4eDgzZsxg06ZNJ9xGt27deOuttwBYunQp\nixd7qitnZWURFRVFdHQ0O3fuZOrUqb73VKlShezs3/5/7Nq1K5MnTyYnJ4eDBw8yadIkunbtWly7\ne1LqKZRTl6fEc17zOAoKHO/M28LTX6zmrdmbmbk6gy/u6kZ+gaNKRHigwxQpFQYPHsyll17qG0Ya\nMmQIF110Ea1btyY1NZXExMQTvv+mm25i+PDhtGjRghYtWpCS4rk/S9u2bWnfvj2JiYnUr1//FyW3\nR4wYQZ8+fahbty4zZszwLU9OTmbYsGF07NgRgOuuu4727duX2F3cgqZ0drDLzS/gm1UZXDfec+wq\nhIVwVccGhIYYl7SrR+v46ABHKMFIpbP9Q6Wz5aTCQ0M4P6kWr12dyo/rM5m9YQ/jftgIwPhZG7k8\nOZ6/XdxS5b1FgpySQpA5P6kW5yfVwjlH+t5DVAgL4R9TVjBx7hZW78zm/n4tSG0Yo0lpkSClieYg\nZWbUj42kVtUIRl/Znn9e3ppl27IY+PIs7np7IZ8v2xHoECVIlLUh7NLuTI+nkoIAMKhDA6be0ZUO\nCTFMXriNG95I+0UVVxF/iIiIIDMzU4mhmDjnyMzMJCLi9Mv5aKJZfmPH/sNcN34uS7dm0bB6JBe3\nrcvdvztLQ0pS7HJzc0lPTz+jc/fllyIiIoiPjyc8/JdnFxZ1ollJQY5pV/Zhrv+/eazPOEj2kTy6\nNK3BK39IIaqipqFEyqKiJgUNH8kx1awSwYe3dmH2X3px9+/OYtb6TC564TtW79TFbyLlmZKCnFBk\nhTBu79WMscM6sCfnKH1GzeTmN9PYfeBIoEMTET/QWIAUSbez4njj2k7c+94ipizZQURYKH1b1yE2\nKpyUhrGBDk9Eionf5hTMrD4wHqgFOOBV59zoX7XpDnwIbPAu+sA59+iJtqs5hcC7Y+ICPly4zfe6\ndb1oalWtSFyVCP50wVlEVwonPFSdUJHSJOATzWZWB6jjnJtvZlWANOAS59zyQm26A/c45y4s6naV\nFAKvoMDx/brdfLtmN3sOHmXljiyWbs3yrY8ID+GT27rStGblAEYpIoUFvMyFc247sN37PNvMVgD1\ngBMXE5dSLyTE6Nosjq7N4nzLnHMs3ZrFA5OWsGTrfu7/YDGv/iGVmCjd6EekLCmRPr6ZJQDtgdnH\nWH2OmS0ys6lm1rIk4pHiZ2a0jo/m49u68LeLkpi3aS9/GDObtE17Ah2aiJwCv080m1ll4H3gTudc\n1q9WzwcaOucOmFk/YDLQ7BjbGAGMAGjQoIGfI5YzNaxzI6Ijw3n04+Vc/tIsoiqEcllyPEPPbkjz\n2lUCHZ6InIBfL14zs3DgE+Bz59yzRWi/EUh1zu0+XhvNKZQdOUfzeGv2Zh77dAUA1aMq8L/rOtGi\nTtUARyYSfAJ+8Zp5aiK8Dqw4XkIws9redphZR288mf6KSUpWZIUwruvamPX/6MfY4R04cCSPi//9\nHYvT9wU6NBE5Dn/OKXQG/gD0NLOF3kc/M7vRzG70thkALDWzRcDzwJWurNXdkJMKCTF6NK/Jp7d3\npVpkBS7+9/c8OHkpy7btD3RoIvIrqn0kJWrp1v0MfX02+3JyCQsxLmlfj/Nb1KR3y9oquCfiRwG/\nTsFflBTKviN5+WRkH+Gpz1bxxfIdHM4toNtZcTSuEUX1qAoM6lCfmlVPv/SviPyWkoKUCfkFjkc+\nXsb4WZt8y8JDjUf7t2JwR51pJlJcAn7xmkhRhIZ4EsCtPZtSNSKc+Zv38tCHy/jr5KXUrhrBeWfF\nERKiYSWRkqKegpQ6WYdzueLlWazckY2Zp7bS2Y2rc1vPplSJCD/5BkTkNwJ+SqrI6aoaEc644R25\nrH09ejavSX6B49WZ67nsPz+w9+DRQIcnUq6ppyCl3tG8Ai564TtW7cymVtWKjL6yPWc3rh7osETK\nFPUUpNyoEBbC1Du6cuf5zTh4JJ9rxsxhy56cQIclUi4pKUiZEBJi3Hn+WXxxVzdCQ4zbJy5gV5Zu\n9i5S3JQUpEypW60Sd51/Fgs276PjP6bR6P5PmbdRlVhFiouSgpQ513VtxHs3nsOl7evhHPzh9TlK\nDCLFRElByhwzIzUhlucGtePbP/fgUG4+A16exXtp6YEOTaTMU1KQMq1+bCRPD2xLdKVw7nl3Ea99\nuz7QIYmUaUoKUuYNSIln9gO96NOyNo99uoKL//0di7aoPLfI6VBSkHIhIjyUF4ck8+c+zdm27xCX\n/ud7Js7ZzOHc/ECHJlKm6OI1KXe27MlhyGuz2ey9lmFgSjz/Gtg2wFGJBJYK4knQqh8byZd3d+P7\ntbuZtGAb76al06FRLBe3rUtEeGigwxMp1dRTkHItL7+AK1/9kXmb9gJQLTKcv/4+ifBQo1eLWlSu\nqO9FEhxU5kIECAsN4aGLkois4Okh7MvJ5Z53F3HHxIUMfHkWR/I05yBSmL4mSbnXJr4aix6+gPDQ\nEOZv3sv6jINMXbKdaSt38eo367mtV7NAhyhSaigpSFAID/V0ipMbxJDcIIYBKfHc/GYao6atIbJi\nGMPPTdDNfETQ8JEEsScua0OrulX5+yfL6fLP6Szduj/QIYkEnJKCBK3oSuG8f9O5/PPy1hQ4uOGN\nNNL35lBQULZOvhApTkoKEtTCQkMY1KEBLw5JJuPAEbr8cwa3TpivxCBBS0lBBEhpGMOXd3UDYMqS\nHTR+YAozV2cEOCqRkqekIOLVsHoUyx7pzc3dmxAbVYE7317I92t3U9au5RE5E7p4TeQY1mUcYMBL\nP7A3J5fuzeMIDw3hknb16NOqNqE6S0nKIJW5EDkDTeIq8/W9PfjTO4v4asVOAL5cvpOOCbE80r8l\nLepUDXCEIv6hnoLICTjneHP2ZhyQdSiXUV+tJjffcX/fREZ0a4yZeg1SNqinIFIMzIyhZzf0vR7U\noT7Xj5/HE1NXsih9Hy9elazEIOWKJppFTkGNyhV578ZzueG8xkxZsoOx328MdEgixUpJQeQUhYYY\nd/RqRseEWB79ZDkT52zWGUpSbigpiJyGyAphPHl5a6pUDGPkB0sY98PGQIckUiyUFEROU+O4ynx5\n93nUq1aJRz5ezqBXZvH2XN0CVMo2nX0kcob2HjzK6Glr+L9ZG3EOqkaEcX3Xxgzu1IAalSsGOjwR\noOhnHykpiBSTvPwCPl2ynTd/3MycjXuICA9h1KD29GlVO9ChiejOayIlLSw0hP7t6vHGdR0ZNagd\n8TGR3Pi/NP752UoOHdWQkpQN6imI+MmWPTncPnEBCzbvIyzEqB0dwVMD2nBukxqBDk2CkHoKIgFW\nPzaSSTd35rlBbWkdH0363kM89OEysg7nBjo0kePyW0/BzOoD44FagANedc6N/lUbA0YD/YAcYJhz\nbv6JtquegpRVb8/dzH3vL6FmlYpc1akBdaMrER9TiXObqucg/lcaylzkAX9yzs03sypAmpl96Zxb\nXqhNX6CZ99EJeMn7r0i5M6hDA5rXrsr14+cx6qs1vuWf39mN5rWrBDAykZ/5bfjIObf9p2/9zrls\nYAVQ71fN+gPjncePQDUzq+OvmEQCrV39avwwsidjhqXyzMC2VAgLofeomTz84VJd3yClQonMKZhZ\nAtAemP2rVfWALYVep/PbxCFSroSHhtAzsRaXp8Qz9Y6uXJBUi/+btYnEBz9j/KyNgQ5Pgpzfk4KZ\nVQbeB+50zmWd5jZGmNk8M5uXkaFbJEr50SSuMq9encqj/VsC8NCHy7jguW/I1mS0BIhfk4KZheNJ\nCG865z44RpOtQP1Cr+O9y37BOfeqcy7VOZcaFxfnn2BFAujqcxJY8WgfmsRFsXrnAUaMT2PvwaOB\nDkuCkN+SgvfMoteBFc65Z4/T7CPgavM4G9jvnNvur5hESrNKFUL58q7zuLd3c9I27aXbUzN4+Zt1\n5BeUrWuJpGzz59lHnYE/AEvMbKF32QNAAwDn3MvAFDyno67Fc0rqcD/GI1LqhYQYt/RoSpemNXj6\ni1U8OXUlS9L380j/lqqjJCVCVzSLlGKPfbKc/5u1kcoVw7iwTV1G9k0kqqJumCinTlc0i5QDf70w\niU9u60rHRrG88eMmzn1yOlv25AQ6LCnHipQUzKyJmVX0Pu9uZrebWTX/hiYiAM1rV+GVP6QyICWe\n/YdyueGNNA4eyQt0WFJOFbWn8D6Qb2ZNgVfxnDH0lt+iEpHfeOryNowa1I7l27O4/KUfmLxgqy54\nk2JX1KRQ4JzLAy4FXnDO3QvoymOREhQSYlzSvh5jh3VgU2YOd769kOFj53I0ryDQoUk5UtSkkGtm\ng4FrgE+8y8L9E5KInEiPxJpMv+c8OibEMmt9JrdPWKCL3aTYFDUpDAfOAR53zm0ws0bAG/4LS0RO\npE50Jd658Rzu75vIZ8t2cM4T05m9PpP9ObmUtTMKpXQ55VNSzSwGqO+cW+yfkE5Mp6SK/NIP63Zz\ny5vz2Zvj6S1c2KYOLwxuj+f6URGPYi2dbWZfAxd726cBu8zse+fc3WcUpYicsXOb1GDyLZ35ZnUG\nq3dm878fN/PNqgwevDCJKzrUP/kGRAop6lUw0c65LDO7Dk+p64fNLCA9BRH5rYbVo7j6nCicc9St\nVomnPlvFn99fzPPT13B918b0blmb2tERgQ5TyoCizimEee9zcAU/TzSLSCljZtzcvSnLH+3NZe3r\nkb73EA9/tIyez3zNhwu3UqA6SnISRU0KjwKfA+ucc3PNrDGw5iTvEZEAiawQxrOD2rH28b48PbAt\nEeGh3DFxIbe8NV/XNsgJqfaRSBA4nJvPs1+u5tWZ6+nStAYvXpVMdKTOKg8mxVr7yMzizWySme3y\nPt43s/gzD1NESkJEeCgP9GvBU5e34Yd1uxnxxjz1GOSYijp8NBbPvQ/qeh8fe5eJSBlyRYf6PHNF\nW+Zs3EPXp2Ywb+OeQIckpUxRk0Kcc26scy7P+xgH6BZoImXQpe3jGTWoHRnZR7htwgI2ZR4MdEhS\nihQ1KWSa2VAzC/U+hgKZ/gxMRPynf7t6TLr5XPYcPMrF//6eW96cz4rtp3ULdSlnipoUrsVzOuoO\nYDswABjmp5hEpAS0bxDDlDu60iEhhu/X7abv6G95YuoKlckIcqd99pGZ3emcG1XM8ZyUzj4SKX47\nsw7zl0lL+WrFTlrWrcrYYR2oWVUXu5UnJXHnNZW4ECknalWN4NU/pHB910Ys25bF9ePnsSv7cKDD\nkgA4k6Sgalsi5UhIiPGX3yfx+KWtWLYtixHj08jL170ags2ZJAUNPIqUQ0M6NeTpgW1ZuGUfN785\nn5yjuvVnMDlhUjCzbDPLOsYjG8/1CiJSDl3cti4DUuL5YvlObn1rgXoMQeSEVVKdc1VKKhARKT1C\nQoynB7alfYNq/GXSUq56bTZXdWzA0fwCLkiqRbXICoEOUfykqKWzRSQIDenUkOzDeTw5dSVzNniu\nfv4zcEm7ujw3qJ1u5FMOKSmIyAndeF4TWtSpypY9ORw4ksf7aelMXriNyQu38b8/dqJLsxqBDlGK\nkaqkisgpycsv4LKXfmBx+n4A7ujVjJt7NKFiWGiAI5MTKYnrFEQkCIWFhvDRrV348q5uAIyetoZO\n/5jGxt2qoVQeKCmIyGlpVqsK393XgyGdGrAvJ5d+z3+rC97KASUFETlt8TGRPH5pa14emkLO0Xw6\nPj6NzZk5gQ5LzoCSgoicsT6tavOXfi0IDTG6/WsGCzbvDXRIcpqUFESkWFzfrTFv/LEjACPeSOPR\nj5eTkX0kwFHJqVJSEJFic26TGnx0a2fqVqvEmO830OPpr/n39DXsP5Qb6NCkiJQURKRYtYmvxoe3\ndOaDm8+lXf1qPP3Fano98zXTVuwMdGhSBEoKIuIXyQ1i+N91nXjzuk5Uj6rIjf9L45PF2wIdlpyE\nkoKI+FXnpjV4+4azqR8byd8+Ws7uA5pnKM2UFETE76pFVuD5K9uzL+coqY99Rf9/f8ef3lnEwSMq\ny13aKCmISIloVS+apwa0oU50BIvS9/P+/HSuem02h47mBzo0KUS1j0SkxB3Jy+evk5byblo6FcNC\n+PT2LjStqUr9/qTaRyJSalUMC+VfA9vyrwFtOJJXQN/R3/Lsl6s5nKteQ6D5rXS2mY0BLgR2Oeda\nHWN9d+BDYIN30QfOuUf9FY+IlD4DU+uT3DCGZ75YxfPT1jBhzmZCzbgsuR739m6u+zUEgN+Gj8ys\nG3AAGH+CpHCPc+7CU9muho9EyqfPlm7nhelrWbYty7ds2LkJPNCvBRXCNKhxpoo6fOS3noJzbqaZ\nJfhr+yJSvvRpVYc+reqQkX2E17/bwMvfrGPcDxvJOHCEF69KDnR4QSPQ6fccM1tkZlPNrOXxGpnZ\nCDObZ2bzMjIySjI+ESlhcVUqMrJvIhuf/D139GrGp4u3M2zsHI7mFQQ6tKAQyKQwH2jonGsLvABM\nPl5D59yrzrlU51xqXFxciQUoIoF143lN6NuqNl+vyuDpL1YFOpygELCk4JzLcs4d8D6fAoSbmW72\nKiI+lSqE8tLQFIae3YBXZ67n758sJ7+gbJ1GX9b4bU7hZMysNrDTOefMrCOeBJUZqHhEpPR68MIk\nduz3zDXM27iHKzrUp3vzmtSrVinQoZU7/jwldQLQHahhZunAw0A4gHPuZWAAcJOZ5QGHgCtdWbuS\nTkRKRMWwUP57dQrPfbWG56etYVH6fsJDjeevbE/vlrUJCdGpq8VFVzSLSJmy+8ARPl60jf98vY6M\n7CM8cVlrBndsEOiwSj1d0Swi5VKNyhUZ3rkRX919HgDjvt/Ihwu3smH3wQBHVj4oKYhImRRdKZz/\nXp3Kqp3Z3DFxoe8ub9mHdZe3M6HhIxEp0+Zt3MPr321gb85Rfly/B4DkBtV44rI2NK+tIns/CfgV\nzSIiJSE1IZbUhFhyjuYx+qs1LNm6nx/WZXLFK7N48apkzmlSnVBNRBeZegoiUu6szzjA8HFz2ZSZ\nQ71qlbgitT4A4WHGBUm1WZdxgAuSagVVwb2i9hSUFESkXNp/KJdXvlnHlCXb2ZiZ85v1LepU5cWr\n2tM4rnIAoit5SgoiIl5L0veTefAIBc7x8aLtrMs4wOL0/cRGVeCGbo35asVOLkiqzfXdGgc6VL/R\nnIKIiFfr+Gjf856JtQBYuGUf97y7iCemrgRg7sa9rN11gDvOb0bdIL5SWj0FEQlaB4/k8fmyHdSO\njmD6il28/v0GQsy4t3dzbujWuFzNOWj4SETkFC3cso87Ji5gU2YOXZvV4MUhyVSNCA90WMVCVzSL\niJyidvWr8d6N53J5cjyz1mVyzZg55OYH130clBRERAqJq1KRZ65oy3OD2rFg8z7ufmcRew8eDXRY\nJUZJQUTkGC5qW5c/dmnEx4vvIYnGAAAMIklEQVS2MeS12ezLCY7EoKQgInIcD16YxLjhHVizK5t2\nj37J6K88tZWcc5S1+dii0kSziMhJzN24h1venM+u7CO+ZQNT4vnXwLYBjOrU6DoFEZFi0iEhlu/u\n68n789PZvu8Qb87ezLtp6aQmxDCoQ/m6l4N6CiIipygvv4CBr8xiweZ9tK4Xza09m5b6Wko6JVVE\nxE/CQkMYN7wjN3VvwpKt+7nhjTRSHvuKLXt+W2OprFFSEBE5DdGVwrmvTyJpfz2f23o2Zc/Bo1wz\nZg7TV+4s05PQmlMQETkD1StX5E8XNCehehR/encR146bR+O4KC5Pjmfo2Q2JrlS2rojWnIKISDHZ\nlHmQr1bs4qWv17L7wFFioyrwzMC2dGoci2FUqhAasNhU+0hEJEAO5+Yzetoa3p2Xzu4DP5/G+tNt\nQt+avYkPF22jZpWKjBnWgfiYSL/HpKQgIhJgB4/kMWXJdpZtyyK/wPH23C0c/VUtpbrREUy+tTM1\nq0T4NRYlBRGRUmbp1v28MWsTLetV5fLkeL5dk8FtExZQKTyU/wxJoUuzGn772UoKIiJlwLJt+7lz\n4kL25hxl6h3diKtS0S8/R9cpiIiUAS3rRvPikGSyD+cx5LUf+WZ1RkDjUVIQEQmws2pV4T9Dktmb\nk8s1Y+bwyjfr+Gr5TtbuOlDisWj4SESklMg6nMvVr89h4ZZ9vmVJdary1IA2tKoXfYJ3npzmFERE\nyqCDR/L4asVONuw+yJ6DR5k4dwtH8wp4aUgyfVvXOe3tqkqqiEgZFFUxjP7t6vleX9S2LgNfnsXd\n7yyiRZ2qJNSI8uvP15yCiEgp1iEhlh9G9qRT41hCQ/xfhVU9BRGRUq5utUqMG96xRH6WegoiIuKj\npCAiIj5KCiIi4qOkICIiPkoKIiLio6QgIiI+fksKZjbGzHaZ2dLjrDcze97M1prZYjNL9lcsIiJS\nNP7sKYwD+pxgfV+gmfcxAnjJj7GIiEgR+C0pOOdmAntO0KQ/MN55/AhUM7PTL+whIiJnLJBzCvWA\nLYVep3uX/YaZjTCzeWY2LyMjsLXGRUTKszIx0eyce9U5l+qcS42Liwt0OCIi5VYgk8JWoH6h1/He\nZSIiEiCBTAofAVd7z0I6G9jvnNsewHhERIKe36qkmtkEoDtQw8zSgYeBcADn3MvAFKAfsBbIAYb7\nKxYRESkavyUF59zgk6x3wC3++vkiInLqysREs4iIlAwlBRER8VFSEBERHyUFERHxUVIQEREfJQUR\nEfFRUhARER8lBRER8VFSEBERHyUFERHxUVIQEREfJQUREfFRUhARER8lBRER8VFSEBERHyUFERHx\nUVIQEREfJQUREfFRUhARER8lBRER8VFSEBERHyUFERHxUVIQEREfJQUREfEx51ygYzglZpYBbDrN\nt9cAdhdjOGWRjoGOAegYQPAdg4bOubiTNSpzSeFMmNk851xqoOMIJB0DHQPQMQAdg+PR8JGIiPgo\nKYiIiE+wJYVXAx1AKaBjoGMAOgagY3BMQTWnICIiJxZsPQURETmBoEkKZtbHzFaZ2VozGxnoePzF\nzOqb2QwzW25my8zsDu/yWDP70szWeP+N8S43M3vee1wWm1lyYPegeJhZqJktMLNPvK8bmdls736+\nbWYVvMsrel+v9a5PCGTcxcnMqpnZe2a20sxWmNk5Qfg5uMv7/2CpmU0ws4hg/CyciqBICmYWCrwI\n9AWSgMFmlhTYqPwmD/iTcy4JOBu4xbuvI4FpzrlmwDTva/Ack2bexwjgpZIP2S/uAFYUev1P4Dnn\nXFNgL/BH7/I/Anu9y5/ztisvRgOfOecSgbZ4jkfQfA7MrB5wO5DqnGsFhAJXEpyfhaJzzpX7B3AO\n8Hmh1/cD9wc6rhLa9w+B3wGrgDreZXWAVd7nrwCDC7X3tSurDyAezx+8nsAngOG5SCns158H4HPg\nHO/zMG87C/Q+FMMxiAY2/HpfguxzUA/YAsR6f7efAL2D7bNwqo+g6Cnw84fjJ+neZeWat/vbHpgN\n1HLObfeu2gHU8j4vj8dmFPBnoMD7ujqwzzmX531deB99++9dv9/bvqxrBGQAY73DaK+ZWRRB9Dlw\nzm0FngY2A9vx/G7TCL7PwikJlqQQdMysMvA+cKdzLqvwOuf5KlQuTzszswuBXc65tEDHEmBhQDLw\nknOuPXCQn4eKgPL9OQDwzpf0x5Mg6wJRQJ+ABlUGBEtS2ArUL/Q63rusXDKzcDwJ4U3n3AfexTvN\nrI53fR1gl3d5eTs2nYGLzWwjMBHPENJooJqZhXnbFN5H3/5710cDmSUZsJ+kA+nOudne1+/hSRLB\n8jkAOB/Y4JzLcM7lAh/g+XwE22fhlARLUpgLNPOedVABz2TTRwGOyS/MzIDXgRXOuWcLrfoIuMb7\n/Bo8cw0/Lb/ae/bJ2cD+QsMLZY5z7n7nXLxzLgHP73m6c24IMAMY4G326/3/6bgM8LYv89+enXM7\ngC1m1ty7qBewnCD5HHhtBs42s0jv/4ufjkFQfRZOWaAnNUrqAfQDVgPrgL8EOh4/7mcXPEMCi4GF\n3kc/PGOj04A1wFdArLe94Tkzax2wBM+ZGgHfj2I6Ft2BT7zPGwNzgLXAu0BF7/II7+u13vWNAx13\nMe5/O2Ce97MwGYgJts8B8AiwElgKvAFUDMbPwqk8dEWziIj4BMvwkYiIFIGSgoiI+CgpiIiIj5KC\niIj4KCmIiIiPkoKUOmaWb2YLzWyRmc03s3NP0r6amd1chO1+bWa6J28hZjbOzAacvKUECyUFKY0O\nOefaOefa4ile+MRJ2lcDTpoUAqXQ1bMipZ6SgpR2VfGUN8bMKpvZNG/vYYmZ9fe2eRJo4u1d/Mvb\n9j5vm0Vm9mSh7Q00szlmttrMunrbhprZv8xsrvdeAjd4l9cxs5ne7S79qX1hZrbRzJ7y/qw5ZtbU\nu3ycmb1sZrOBp7z3MZjs3f6PZtam0D6N9b5/sZld7l1+gZnN8u7ru95aVpjZk+a5V8ZiM3vau2yg\nN75FZjbzJPtkZvZv89xb5CugZnH+sqTs0zcYKY0qmdlCPFeY1sFTvwjgMHCpcy7LzGoAP5rZR3gK\nvbVyzrUDMLO+eAqhdXLO5ZhZbKFthznnOppZP+BhPPVx/oinrEMHM6sIfG9mXwCX4Smr/Lj3nhyR\nx4l3v3OutZldjadC64Xe5fHAuc65fDN7AVjgnLvEzHoC4/FccfzgT+/3xh7j3be/Auc75w6a2X3A\n3Wb2InApkOicc2ZWzftzHgJ6O+e2Flp2vH1qDzTHc1+RWnjKPowp0m9FgoKSgpRGhwr9gT8HGG9m\nrfCUYviHmXXDUxa7Hj+Xfi7sfGCscy4HwDm3p9C6nwoEpgEJ3ucXAG0Kja1H47nZzFxgjHkKDE52\nzi08TrwTCv37XKHl7zrn8r3PuwCXe+OZbmbVzayqN9Yrf3qDc26veSq9JuH5Qw5QAZiFp5TzYeB1\n89xR7hPv274HxpnZO4X273j71A2Y4I1rm5lNP84+SZBSUpBSzTk3y/vNOQ5PDac4IMU5l2ueSqgR\np7jJI95/8/n582/Abc65z3/d2JuAfo/nj+6zzrnxxwrzOM8PnmJsvh8LfOmcG3yMeDriKew2ALgV\n6Omcu9HMOnnjTDOzlOPtk7eHJHJcmlOQUs3MEvHcRjETz7fdXd6E0ANo6G2WDVQp9LYvgeFmFund\nRuHho2P5HLjJ2yPAzM4ysygzawjsdM79F3gNT+npYxlU6N9Zx2nzLTDEu/3uwG7nuc/Fl8AthfY3\nBvgR6FxofiLKG1NlINo5NwW4C88tNjGzJs652c65h/DcWKf+8fYJmAkM8s451AF6nOTYSJBRT0FK\no5/mFMDzjfca77j8m8DHZrYET/XPlQDOuUwz+97MlgJTnXP3mlk7YJ6ZHQWmAA+c4Oe9hmcoab55\nxmsygEvwVFm918xygQPA1cd5f4yZLcbTC/nNt3uvv+EZiloM5PBziebHgBe9secDjzjnPjCzYcAE\n73wAeOYYsoEPzSzCe1zu9q77l5k18y6bBizCUxn1WPs0Cc8czXI8paWPl8QkSKlKqsgZ8A5hpTrn\ndgc6FpHioOEjERHxUU9BRER81FMQEREfJQUREfFRUhARER8lBRER8VFSEBERHyUFERHx+X8AQKVT\nAPP7oQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YaSukEdPgeIU",
        "colab_type": "text"
      },
      "source": [
        "## My own reimplementation of RAdam following the paper"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.122368Z",
          "start_time": "2019-09-02T14:18:44.987Z"
        },
        "id": "R1NCCOwwWQtJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = cnn_learner(data, models.resnet18, opt_func=opt_radam_own, metrics=[accuracy])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.123427Z",
          "start_time": "2019-09-02T14:18:44.989Z"
        },
        "id": "ZAPECjibWQtR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn.unfreeze()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.124374Z",
          "start_time": "2019-09-02T14:18:44.991Z"
        },
        "id": "PVtNvpuRWQtZ",
        "colab_type": "code",
        "outputId": "75787a65-ee4e-4e4e-c0e6-70341e71b6f9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "source": [
        "learn.fit(1, [1e-5, 1e-4, 1e-3])"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.274110</td>\n",
              "      <td>0.175034</td>\n",
              "      <td>0.946800</td>\n",
              "      <td>01:07</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.125330Z",
          "start_time": "2019-09-02T14:18:44.993Z"
        },
        "id": "AMBrDaBBWQtg",
        "colab_type": "code",
        "outputId": "16b9caa4-89d5-42b4-8319-66ba49bb58bc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        }
      },
      "source": [
        "learn.recorder.plot_losses()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAELCAYAAAA2mZrgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4VOXd//H3N5N9DyGEsCVssm8h\nsrohLmhblZa61L22VLrZ2traPn20tfWpT2ut2qr9uVdbodaVquhjFYsrEJB9k52wJYQACdmT+/fH\nHGJEAgEyOUnm87quuZg5c8+Z7zkZ5jPn3Ofcx5xziIiIAET4XYCIiLQdCgUREWmgUBARkQYKBRER\naaBQEBGRBgoFERFpELJQMLNYM1tgZkvNbKWZ/eoIbWLM7B9mtt7M5ptZTqjqERGRYwvllkIVcLZz\nbgQwEphiZuMOa3MDUOKc6wf8EfjfENYjIiLHELJQcEFl3sMo73b4mXIXA3/17j8HTDYzC1VNIiJy\ndCHtUzCzgJktAQqBN51z8w9r0h3YBuCcqwX2A+mhrElERJoWGcqZO+fqgJFmlgq8aGZDnXMrjnc+\nZjYdmA6QkJAweuDAgS1cqYhIx7Zo0aI9zrmMY7ULaSgc4pzbZ2ZzgSlA41DYDvQECswsEkgBio/w\n+oeBhwHy8vJcfn5+6IsWEelAzGxLc9qF8uijDG8LATOLA84F1hzWbDZwrXd/GvC20wh9IiK+CeWW\nQhbwVzMLEAyfZ51zr5jZHUC+c2428BjwtJmtB/YCl4ewHhEROYaQhYJzbhkw6gjTb2t0vxL4aqhq\nEBGR49MqfQoiIkdSU1NDQUEBlZWVfpfSYcTGxtKjRw+ioqJO6PUKBRHxTUFBAUlJSeTk5KBTlE6e\nc47i4mIKCgro3bv3Cc1DYx+JiG8qKytJT09XILQQMyM9Pf2ktrwUCiLiKwVCyzrZ9RlWofD6il3s\nPqB9lyIiTQmbUKisqePGvy3iqkcPH2lDRMJVcXExI0eOZOTIkXTt2pXu3bs3PK6urm7WPK6//nrW\nrl0b4kpbT9h0NO89GPwDf1JYxsdbSxjVK83nikTEb+np6SxZsgSAX/7ylyQmJvLjH//4M22cczjn\niIg48m/oJ554IuR1tqaw2VI4FAoAUx/8gO37KnysRkTasvXr1zN48GCuvPJKhgwZws6dO5k+fTp5\neXkMGTKEO+64o6HtaaedxpIlS6itrSU1NZVbb72VESNGMH78eAoLC31cihMTdlsKh+Rv3kv3kd19\nqkZEDverf61k1Y4DLTrPwd2Suf1LQ07otWvWrOGpp54iLy8PgLvuuotOnTpRW1vLpEmTmDZtGoMH\nD/7Ma/bv38+ZZ57JXXfdxc0338zjjz/OrbfeetLL0ZrCZkuhpPyzoXDLc8vYvOegT9WISFvXt2/f\nhkAAmDlzJrm5ueTm5rJ69WpWrVr1udfExcVxwQUXADB69Gg2b97cWuW2mLDZUrhoRDciIyK4+dkl\n3HL+AH7z6mrOuvsdHroyl3MGZxIVCJt8FGmTTvQXfagkJCQ03P/kk0+47777WLBgAampqVx11VVH\nPBcgOjq64X4gEKC2trZVam1JYfNNaGZ8YXgWa39zAVeOzW6YPuPvi/nrB5v9K0xE2rwDBw6QlJRE\ncnIyO3fu5I033vC7pJAJm1BoLC46wOL/Prfh8XOLCnysRkTautzcXAYPHszAgQO55pprmDhxot8l\nhYy1t8sXtORFdtbsOsCsBdt48oPNvP2jM+mTkdgi8xWR5lm9ejWDBg3yu4wO50jr1cwWOefymnhJ\ng7DcUjhkYNdkpp/RB4A5K3b5XI2IiP/COhQAuqXGMSanE8/M30pNXb3f5YiI+CrsQwHgW2f2Yfu+\nCl5essPvUkREfKVQAM4e2IU+nRN4ecl2v0sREfGVQoHg4arj+qazZOs+6uvbV8e7iEhLUih4RvdK\no7SqlrW7S/0uRUTENwoFz2n9OxMdiODZ/G1+lyIirWTSpEmfOxHt3nvvZcaMGU2+JjExeOj6jh07\nmDZt2hHbnHXWWRzr0Pl7772X8vLyhscXXngh+/bta27pIaNQ8GQmxzK2Tyc+3FDsdyki0kquuOIK\nZs2a9Zlps2bN4oorrjjma7t168Zzzz13wu99eCi89tprpKamnvD8WopCoZFRvdJYt7uUsqr2N16J\niBy/adOm8eqrrzZcUGfz5s3s2LGDUaNGMXnyZHJzcxk2bBgvv/zy5167efNmhg4dCkBFRQWXX345\ngwYNYurUqVRUfDo0/4wZMxqG3L799tsBuP/++9mxYweTJk1i0qRJAOTk5LBnzx4A7rnnHoYOHcrQ\noUO59957G95v0KBBfPOb32TIkCGcd955n3mflhI2A+I1R152GvUOXlxcwNXjc/wuRyS8zLkVdi1v\n2Xl2HQYX3NXk0506dWLMmDHMmTOHiy++mFmzZnHppZcSFxfHiy++SHJyMnv27GHcuHFcdNFFTV7/\n+KGHHiI+Pp7Vq1ezbNkycnNzG56788476dSpE3V1dUyePJlly5bx/e9/n3vuuYe5c+fSuXPnz8xr\n0aJFPPHEE8yfPx/nHGPHjuXMM88kLS2NTz75hJkzZ/LII49w6aWX8vzzz3PVVVe1zLryaEuhkYn9\nOjOwaxLPLdahqSLhovEupEO7jpxz/PznP2f48OGcc845bN++nd27dzc5j3nz5jV8OQ8fPpzhw4c3\nPPfss8+Sm5vLqFGjWLly5RGH3G7svffeY+rUqSQkJJCYmMiXv/xl3n33XQB69+7NyJEjgdANza0t\nhUYCEcaFw7L447/XUVRaRUZSjN8liYSPo/yiD6WLL76YH/7whyxevJjy8nJGjx7Nk08+SVFREYsW\nLSIqKoqcnJwjDpV9LJs2beLuu+9m4cKFpKWlcd11153QfA6Jifn0OykQCIRk95G2FA5z1oAMnIMP\nNuzxuxQRaQWJiYlMmjSJr3/96w0dzPv376dLly5ERUUxd+5ctmzZctR5nHHGGTzzzDMArFixgmXL\nlgHBIbcTEhJISUlh9+7dzJkzp+E1SUlJlJZ+/hD4008/nZdeeony8nIOHjzIiy++yOmnn95Si3tM\nCoXDDM5KJjEmkgWb9vpdioi0kiuuuIKlS5c2hMKVV15Jfn4+w4YN46mnnmLgwIFHff2MGTMoKytj\n0KBB3HbbbYwePRqAESNGMGrUKAYOHMjXvva1zwy5PX36dKZMmdLQ0XxIbm4u1113HWPGjGHs2LF8\n4xvfYNSoUS28xE0L2dDZZtYTeArIBBzwsHPuvsPanAW8DGzyJr3gnLuDo2jJobObct0TCygoqeDf\nN58Z0vcRCXcaOjs0Tmbo7FD2KdQCP3LOLTazJGCRmb3pnDu8l+Vd59wXQ1jHcRvbO5131q5hT1kV\nnRPVryAi4SNku4+cczudc4u9+6XAaqB7qN6vJU3slw7AO2uLfK5ERKR1tUqfgpnlAKOA+Ud4eryZ\nLTWzOWbWJq7cPax7Ct1T43h9xU6/SxHp8Nrb1R/bupNdnyEPBTNLBJ4HfuCcO3DY04uBbOfcCOBP\nwEtNzGO6meWbWX5RUeh/vZsZ5w/pyrxP9lBaWRPy9xMJV7GxsRQXFysYWohzjuLiYmJjY094HiG9\nRrOZRQGvAG845+5pRvvNQJ5zrsnjQVujoxlgybZ9XPLA+/zqoiFcOyEn5O8nEo5qamooKCg4qWP3\n5bNiY2Pp0aMHUVFRn5nue0ezBc8HfwxY3VQgmFlXYLdzzpnZGIJbLm1iRLqRPVMZ1SuVx9/fxFXj\nsglEHPn0dhE5cVFRUfTu3dvvMqSRUO4+mghcDZxtZku824VmdqOZ3ei1mQasMLOlwP3A5a4NbUde\nOz6HLcXlLC3wfzhbEZHWELItBefce8BRf1475/4M/DlUNZysCd5RSO+u20NurzSfqxERCT2d0XwU\nXZJiOb1/Z/42fwu1dfV+lyMiEnIKhWO4/NReFJVWaReSiIQFhcIxjO3TiQiDmQt0mU4R6fgUCsfQ\nOTGGK8dm89LH2yks1WFzItKxKRSa4bqJOdTWO/724dGHzxURae8UCs3QNyORM0/JYPbSHX6XIiIS\nUgqFZprQN53NxeXsKavyuxQRkZBRKDTT6OzgeQqLtpT4XImISOgoFJppWI8UEqIDvLO20O9SRERC\nRqHQTDGRASYPyuSNlbt1IpuIdFgKheNw7uBM9h6sZvn2/X6XIiISEgqF4zChb3AspA82tImBXEVE\nWpxC4TikJ8YwvEcKT7y/mYrqOr/LERFpcQqF4/ST8weyp6yKV5frUp0i0vEoFI7TxH7p9MlIYNaC\nrX6XIiLS4hQKx8nMuCyvJ/lbSlhfWOZ3OSIiLUqhcAKm5nYnEGH8c5FGThWRjkWhcAK6JMVy9sAu\nPL9oOzU6Z0FEOhCFwgm6NK8ne8qqeGu1znAWkY5DoXCCJg3IoGtyLP/98gr2l9f4XY6ISItQKJyg\nyEAED16VS3FZFQ+8s97vckREWoRC4STk9krj3MGZzFqwVVdlE5EOQaFwkr53dn8OVNbyz/wCv0sR\nETlpCoWTNLR7ChP7pfPQOxsorVTfgoi0bwqFFnDL+QMpq6rlhcXb/S5FROSkKBRawMieqYzomcpT\nH27GOed3OSIiJ0yh0EKuOLUnG4oOsmL7Ab9LERE5YQqFFnLB0CyiAxE8p6EvRKQdC1komFlPM5tr\nZqvMbKWZ3XSENmZm95vZejNbZma5oaon1FLio/jC8CyeX7xdHc4i0m6FckuhFviRc24wMA74jpkN\nPqzNBUB/7zYdeCiE9YTcDaf1pqyqlkff3eR3KSIiJyRkoeCc2+mcW+zdLwVWA90Pa3Yx8JQL+ghI\nNbOsUNUUakO7p3DhsK488u5G9h6s9rscEZHj1ip9CmaWA4wC5h/2VHeg8U74Aj4fHO3KD845hfLq\nOmYt1EV4RKT9CXkomFki8DzwA+fcCR2aY2bTzSzfzPKLiopatsAWdkpmEhP6pvO3D7dQq2G1RaSd\nCWkomFkUwUD4u3PuhSM02Q70bPS4hzftM5xzDzvn8pxzeRkZGaEptgVdOyGHHfsr+ffq3X6XIiJy\nXEJ59JEBjwGrnXP3NNFsNnCNdxTSOGC/c25nqGpqLecMyqR7ahxPfrDZ71JERI5LKLcUJgJXA2eb\n2RLvdqGZ3WhmN3ptXgM2AuuBR4Bvh7CeVhOIMK4en81HG/eydlep3+WIiDRbZKhm7Jx7D7BjtHHA\nd0JVg58uy+vJH99cx93/t5b/d9VoIiKOuipERNoEndEcImkJ0cw4qy9vrtrNS0s0UJ6ItA8KhRC6\naXJ/+nVJ5K8fbvG7FBGRZlEohJCZ8bUxvVi6bR+rd2qgPBFp+xQKIXbJqO4kRAe4a84av0sRETkm\nhUKIdUqI5nuT+/OfdUXaWhCRNk+h0Aqmje5BYkwkf/i/dX6XIiJyVAqFVtA5MYbrJ+bw1prdbC0u\n97scEZEmKRRayZVjswmY8dSHm/0uRUSkSQqFVtI1JZYpQ7vy9/lbyd+81+9yRESOSKHQin5y/kDi\nogP88Nkl1GgEVRFpgxQKrahXejy/nzacbXsr+NtHOqFNRNoehUIrO3tgFyb0Tee3c9aw+0Cl3+WI\niHyGQqGVmRm/uWQo1bX1/GvpDr/LERH5DIWCD/pkJDKqVyp/+c8G9pRV+V2OiEgDhYJPfnPJUPaU\nVfOPhduO3VhEpJUoFHwypFsKE/qmM3PBVurrnd/liIgACgVfXTk2m4KSCh5/f5PfpYiIAAoFX104\nrCvnDc7kt3PWsGCTTmgTEf8pFHxkZvzh0hH06hTPD/+xhIrqOr9LEpEwp1DwWVJsFL/4wiC276vg\nwXfW+12OiIQ5hUIbMHlQJqf378xfP9jMtr0aRVVE/NOsUDCzvmYW490/y8y+b2apoS0tvNx5yTCq\n6+q5+dkl1OloJBHxSXO3FJ4H6sysH/Aw0BN4JmRVhaFe6fHccFpvFm4u4eUl2/0uR0TCVHNDod45\nVwtMBf7knLsFyApdWeHpx+cNYGj3ZO55c51GURURXzQ3FGrM7ArgWuAVb1pUaEoKX2bGTZNPoaCk\ngrfXFPpdjoiEoeaGwvXAeOBO59wmM+sNPB26ssLXpAEZdEmK4Zn5W/0uRUTCULNCwTm3yjn3fefc\nTDNLA5Kcc/8b4trCUmQggmvGZ/OfdUWs3LHf73JEJMw09+ijd8ws2cw6AYuBR8zsntCWFr6uHp9D\nUkwkD87d4HcpIhJmmrv7KMU5dwD4MvCUc24scM7RXmBmj5tZoZmtaOL5s8xsv5kt8W63HV/pHVdK\nXBRXj8/mtRU7WV9Y6nc5IhJGmhsKkWaWBVzKpx3Nx/IkMOUYbd51zo30bnc0c75h4YbTepMYHckv\nZ6/COZ23ICKto7mhcAfwBrDBObfQzPoAnxztBc65eYBGeTtB6Ykx/Pj8Aby3fg/ffCqfqlqNiyQi\nodfcjuZ/OueGO+dmeI83Oue+0gLvP97MlprZHDMb0lQjM5tuZvlmll9UVNQCb9s+XD0um1vOH8C/\nVxfyyLyNfpcjImGguR3NPczsRa+PoNDMnjezHif53ouBbOfcCOBPwEtNNXTOPeycy3PO5WVkZJzk\n27YfERHGdyb14/whmTz4zgZ27q/wuyQR6eCau/voCWA20M27/cubdsKccwecc2Xe/deAKDPrfDLz\n7Kh+8YXB1NU7fv3KKr9LEZEOrrmhkOGce8I5V+vdngRO6ie7mXU1M/Puj/FqKT6ZeXZUPTvF872z\n+/Ha8l28s1ZnOotI6DQ3FIrN7CozC3i3qzjGF7iZzQQ+BAaYWYGZ3WBmN5rZjV6TacAKM1sK3A9c\n7nSYTZO+eUYf+mQkcNvLK6msUaeziISGNed72MyyCe73Hw844APge865baEt7/Py8vJcfn5+a79t\nm/DB+j187dH5fH9yf24+9xS/yxGRdsTMFjnn8o7VrrlHH21xzl3knMtwznVxzl0CtMTRR3IcJvTr\nzMUju/GXdzZQUKKL8YhIyzuZK6/d3GJVSLP9dMpA6pzj9pdXUq+L8YhICzuZULAWq0KarVtqHN84\nvTdvrSnknjfX+V2OiHQwJxMK+pnqkx9MPoWuybE8/v4m9h6s9rscEelAjhoKZlZqZgeOcCsleL6C\n+CAuOsDTN4yhoqaOR9/Vmc4i0nKOGgrOuSTnXPIRbknOucjWKlI+r39mEl8c3o3H39/Eiu267oKI\ntIyT2X0kPvvplAEkREcy4++LOFhV63c5ItIBKBTasR5p8dz91RFs21vBPW+u0xDbInLSFArt3KSB\nXZg2ugePvbeJn7+4XMEgIidF/QIdwF1fHkZKXBSPvbeJAZlJXDext98liUg7pVDoACIDEfziC4NY\nt7uUO15ZRUxUgCvG9PK7LBFph7T7qIMwM/5y1WhGZ6fxsxeW88T7m/wuSUTaIYVCB5IQE8mT14/h\n1Jw0fjtnDat2HPC7JBFpZxQKHUxCTCQPfC2XuKgAP/jHx7q2s4gcF4VCB9QlOZY/fHUE63aX8dvX\n1uiIJBFpNoVCB3XO4EyuHZ/Nkx9s5rXlu/wuR0TaCYVCB3bbl4bQp3MC//v6GrYW6/oLInJsCoUO\nLBBh/GbqUHYdqOQrf/mAA5U1fpckIm2cQqGDm9C3M49ck0dRaRUPzF3vdzki0sYpFMLAmadkMG10\nDx6et5G31+z2uxwRacMUCmHijouHMLBrMt995mPW7S71uxwRaaMUCmEiPjqSJ647lfjoSM774zye\nW1Tgd0ki0gYpFMJI15RYXpgxgfjoAHe/sZbKGp3YJiKfpVAIM73S43n0mjx2Hahk5oKtfpcjIm2M\nQiEMje+bzoS+6fzu9bU8PG8DNXX1fpckIm2EQiEMmRm/mzacvJw0/ue1NfzxzXV+lyQibYRCIUz1\nSIvn6RvGNhyqOmf5To2RJCKhCwUze9zMCs1sRRPPm5ndb2brzWyZmeWGqhZp2n9dOIhe6fHM+Pti\nZi3c5nc5IuKzUG4pPAlMOcrzFwD9vdt04KEQ1iJNSEuI5vWbzuDUnDRuf3kl76/f43dJIuKjkIWC\nc24esPcoTS4GnnJBHwGpZpYVqnqkadGRETx6zan0yUjgW08vYtOeg36XJCI+8bNPoTvQeH9FgTdN\nfJASH8Xj151KZMCYcu88nv5oi/oYRMJQu+hoNrPpZpZvZvlFRUV+l9NhdUuN48Erc4mPDvDfL63g\n/rc0gJ5IuPEzFLYDPRs97uFN+xzn3MPOuTznXF5GRkarFBeuJvTtzKJfnMslI7tx31vrmLu20O+S\nRKQV+RkKs4FrvKOQxgH7nXM7faxHPBERxm+mDqNvRiI/enYpb63WyKoi4SKUh6TOBD4EBphZgZnd\nYGY3mtmNXpPXgI3AeuAR4NuhqkWOX2JMJA9dNZqEmADTn16kYBAJE9beOhPz8vJcfn6+32WEjbKq\nWqY99AFrd5dy/+Wj+NKIbn6XJCInwMwWOefyjtWuXXQ0i38SYyJ59sbxDO+ewvdmfswd/1qlo5JE\nOjCFghxTcmwUf/36GC4e2Y3H39/EqF+/yawFWzWQnkgHpFCQZkmNj+bey0byo3NPYV95Dbe+sJzc\nO97kpY+PeMCYiLRT6lOQ41ZZU8d/1hVxyz+XUl5dx9dP681lp/YkLipAt9Q4v8sTkSNobp+CQkFO\n2L7yam55bhlvrgoemZQcG8kj1+Qxtk+6z5WJyOHU0SwhlxofzSPX5PHitycwIDOJA5W1XPbwRzww\ndz319e3rx4aIBGlLQVrM3oPV3DTrY979ZA+nZCZy+5eGMLFfZ7/LEhG0pSA+6JQQzWPXnsrvpg2n\nuraebz29iPWFZX6XJSLHQaEgLSo6MoJL83ry16+PAeCce/7DjL8tYsX2/T5XJiLNoVCQkMhOT+Dl\n705k6qjuzFmxiy/+6T2e/miL32WJyDGoT0FCbv7GYm6fvZI1u0q5bkIOudlp9MtIZHC3ZL9LEwkb\nOiRV2pTy6lpueW4Zry77dCDci0Z045bzB9CzU7yPlYmEB4WCtEk791fw8pIdPLeogPWFZUQYnDs4\nk+sn9mZs706Ymd8linRICgVp05xzLN66j7dW7+aZBVvZV17Ddyb15ZbzB/pdmkiH1NxQiGyNYkQO\nZ2aMzk5jdHYa3zu7Pz9+bikPzN1AcVk1v/jiYBJj9NEU8YP+54nv4qID3HfZSKIijFkLt7F4awn3\nXT6KQVnqiBZpbTokVdqEyEAE914+ioevHk1haRVTH3yfx97bpOEyRFqZQkHalPOGdOX1m86gT+dE\nfv3KKqY++D4fbSzWhX1EWok6mqVNqq2rZ+aCrfzp7fUUllbRp3MCV47L5ovDs8hMjvW7PJF2R0cf\nSYdQXFbF7KU7eHjeRnburyQ6EMFp/TvzldwenD8kk8iANnZFmkOhIB3O2l2lPJu/jecWFbC/ooZu\nKbHkZqfRLTWOm889hdiogN8lirRZCgXpsCpr6nhrdSF3vrqKHfsrARjRM5WJfdMprazlrAEZnD2w\ni06EE2lEoSAdXk1dPRU1dfx71W7ufHU1JeXVHDpYaXBWMr+ZOpTcXmn+FinSRigUJKxU19ZTV+9w\nOJ78YDMPzt1AWVUtqfFRTB6Yybcn9aVvRqLfZYr4RqEgYa3kYDWzl+7gmflb2VBURiDC+OmUgVw/\nMUe7lSQsKRREPDv3V3DTzCUs2LyXS0Z249eXDCUpNsrvskRalcY+EvFkpcQxa/o4/jx3Pff+ex1v\nrtpNbnYaXZJiOb1/Zy4e2U1bDyIebSlIWFm8tYRH393Ikq37KCytorbeMaJnKucM7MKZAzIY3iPV\n7xJFQqJN7D4ysynAfUAAeNQ5d9dhz18H/B7Y7k36s3Pu0aPNU6EgLaW2rp6/fbSFP7y5jtLKWiIM\nxvdNB2DSgC6cPbALfdQ5LR2E76FgZgFgHXAuUAAsBK5wzq1q1OY6IM85993mzlehIC2torqOkvJq\n/jx3Pa8t38m+8pqG584dnEm3lFiSYqMY2j2ZKUOzfKxU5MS1hT6FMcB659xGr6BZwMXAqqO+SqSV\nxUUHiIuO43+mDuN/pg6juraet1bv5rUVu3hj5S6qa+sb2nZPjeNrY3sRExnBqTmdGNFTu5ukYwll\nKHQHtjV6XACMPUK7r5jZGQS3Kn7onNt2hDYirSY6MoILhmVxwbDgVkF9veNAZQ3PLNjKP/ML+P0b\naxvantavM9PP6MOpOZ2Ii9YwG9L+hXL30TRginPuG97jq4GxjXcVmVk6UOacqzKzbwGXOefOPsK8\npgPTAXr16jV6y5YtIalZ5Fjq6x0VNXXsr6jhmflbeey9TVTU1GEG5wzK5GcXDFQ/hLRJbaFPYTzw\nS+fc+d7jnwE4537bRPsAsNc5l3K0+apPQdqSHfsqWLBpL2+s3MW8dUXERAX4wTn9ufzUXkRHagRX\naTvaQihEEtwlNJng0UULga8551Y2apPlnNvp3Z8K/NQ5N+5o81UoSFu1Yvt+/uvF5Swt2M+Inql8\naXgWn+wuIzkuktp6R8+0eL40ohsZSTEcrKplW0k5PdLiKa+qZX9FDf0zk/xeBOnAfA8Fr4gLgXsJ\nHpL6uHPuTjO7A8h3zs02s98CFwG1wF5ghnNuzdHmqVCQtqy+3vHSku3c/vJKSqtqiQ5EUF1X/5k2\nGUkxFJVWARBhNAziNygrmc6J0fTvksT3J/cjKhBBRU0dlTV1FJdV0z8zkbiogE60kxPSJkIhFBQK\n0h7sPlDJtr3ljOiZSmllLSlxUSzfvp9/LNzKnrJqUuOiOCUziYKScmrqHXFRAdYXlrF9XwWb9hyk\n7gjXpo6OjKC6tp4BmUnkZqcysGsyp/XvTN+MRCpr6oiNCuCco6bO8eryHazcfoD46AAbig5yw+m9\nNWJsmFMoiLRTCzbt5T/rCkmIiSRgRnx0gKraet5YuYuFm0sYlJXM6p0HGtoHIoy6ekfX5Fj2Hqwm\nMmCUV9d9br5jencit1caM87qS0qcxn4KNwoFkQ7GOUdtvSMqEMH6wjJq6+v5aEMxy7bvJzLCWLur\nlBE9U6mpc5w/JJPR2WnsKavGgNlLd/C3j7ZQ6O22GtY9hejICAIRRkZiDF8aEbz2defEGErKq1my\nbR9jendiQGaSdld1EAoFEfkXTRiCAAAMW0lEQVSc11fs5ONt+3js3U1kJsfSPS2OjUUH2VNWdcT2\nOenx9M1IZMf+Sib2TWdEz1Q2FJWRnhjDsO4pLN5SwsCuSYzp3UnXy27jFAoi0qR95dUkxkQSGQj2\nUyzcvJei0ioqa+pIS4gmMSaSBZv28tyiArbvq2jWPLPT45nQN526ekdtneOikd3olhpHbGQAhyM7\nPSHESyVHo1AQkZNWU1dPeXUdKXFRrC8sZUtxOTmdEyitrOXjrSUkxkRSVVvPe5/s4fWVu446r0FZ\nyfTuHM+Fw7KoqaunX0YStfX19M9MIipgbCg8SOekaNITYghEGBXVdSzYvJfyqlrOOCWDhBiN9H8y\nFAoi0qqcc3y8bR856QnU1AWDoqS8mpLyakora1mwaS+7D1RS0mjAwabkZaexac9Big9WAxATGcHY\nPun85PwBDO1+1PNbpQkKBRFpc8qqapm/sZiICGNfeTU1dY5564pIjImkX5dElhbsZ8m2EkoO1tA9\nNY4Lh2Uxqlcq76wtYvbS7ewpqyYrJZaxvTsxOjuNvQdr2HWgkqLSKr5+Wg4T+nY+Zg3OOQpLq9hQ\nVEZqXDSZyTGkJ8a0wtL7S6EgIu3Woe+lxkc+7Smr4pF5G3l1+U4KSj7bz3HoHI6slFjOHZxJz7R4\naurr6Z4aR1Qggn3lNeyrqGZj0UFmL9nxmRMKIyOMS0/tScCM/C0lVNfWkZ4Yw7rdpVwwNIsZZ/al\ne1oc63aX0jcjkejICGrq6omMsHZ1ZJZCQUQ6rAOVNZQcrKamrp4eafEA/Pnt9cxeuoOi0ioqaj5/\nnsYhp+akcf6QruSkJ1B8sIrFW/bxj/xtREdGEBMZQa9O8dTWOZLjIlm0pYTG5xFmpcQyKCuZ/M17\niY6M4AvDsuieFsfWveXs2FfJmadkcNW4bAIRbS8sFAoiEpZq6uopKa8mMiKCkvJqDlbVEhkRQXRk\nBL07JxzxC3vvwWqSYiM/9+t/295ynv5oC+sLy5jYrzOvr9jJ0oL9pCdE0ykhmpU7gicRJsdGcqCy\nFoCE6AC90hP4wrCunDekK/27JLaJLQqFgohICDjnGr7kC0srqa4N7qYCeH3FLt5ZW8TGPWUs3FwC\nQM9OcVw6uicREUbX5Fj6ZyZS76CotIrICGNcn3T2llczf2Mxew9Wk5EUw459lew+UMno7DSmDO2K\nATv2VdIlOYbYqBO7bodCQUTER9v2lvPOuiLmLN/JBxuKW2Ses787keE9Tuxqf23hcpwiImGrZ6d4\nrh6XzdXjstm5v4KoQAQ79lXw8dZ9JMdF0qtTPIUHqli98wBJsVEM75FCVkocxQer6JIcS1p8FB9u\nKGbu2kKqa+sZ2TOtYYsklLSlICISBpq7paDBSkREpIFCQUREGigURESkgUJBREQaKBRERKSBQkFE\nRBooFEREpIFCQUREGigURESkgUJBREQaKBRERKSBQkFERBooFEREpIFCQUREGigURESkQbu7noKZ\nFQFbTvDlnYE9LVhOe6R1oHUAWgcQfusg2zmXcaxG7S4UToaZ5TfnIhMdmdaB1gFoHYDWQVO0+0hE\nRBooFEREpEG4hcLDfhfQBmgdaB2A1gFoHRxRWPUpiIjI0YXbloKIiBxF2ISCmU0xs7Vmtt7MbvW7\nnlAxs55mNtfMVpnZSjO7yZveyczeNLNPvH/TvOlmZvd762WZmeX6uwQtw8wCZvaxmb3iPe5tZvO9\n5fyHmUV702O8x+u953P8rLslmVmqmT1nZmvMbLWZjQ/Dz8EPvf8HK8xsppnFhuNn4XiERSiYWQB4\nALgAGAxcYWaD/a0qZGqBHznnBgPjgO94y3or8JZzrj/wlvcYguukv3ebDjzU+iWHxE3A6kaP/xf4\no3OuH1AC3OBNvwEo8ab/0WvXUdwHvO6cGwiMILg+wuZzYGbdge8Dec65oUAAuJzw/Cw0n3Ouw9+A\n8cAbjR7/DPiZ33W10rK/DJwLrAWyvGlZwFrv/v8DrmjUvqFde70BPQh+4Z0NvAIYwZOUIg//PABv\nAOO9+5FeO/N7GVpgHaQAmw5fljD7HHQHtgGdvL/tK8D54fZZON5bWGwp8OmH45ACb1qH5m3+jgLm\nA5nOuZ3eU7uATO9+R1w39wI/Aeq9x+nAPudcrfe48TI2LL/3/H6vfXvXGygCnvB2oz1qZgmE0efA\nObcduBvYCuwk+LddRPh9Fo5LuIRC2DGzROB54AfOuQONn3PBn0Id8rAzM/siUOicW+R3LT6LBHKB\nh5xzo4CDfLqrCOjYnwMAr7/kYoIB2Q1IAKb4WlQ7EC6hsB3o2ehxD29ah2RmUQQD4e/OuRe8ybvN\nLMt7Pgso9KZ3tHUzEbjIzDYDswjuQroPSDWzSK9N42VsWH7v+RSguDULDpECoMA5N997/BzBkAiX\nzwHAOcAm51yRc64GeIHg5yPcPgvHJVxCYSHQ3zvqIJpgZ9Nsn2sKCTMz4DFgtXPunkZPzQau9e5f\nS7Cv4dD0a7yjT8YB+xvtXmh3nHM/c871cM7lEPw7v+2cuxKYC0zzmh2+/IfWyzSvfbv/9eyc2wVs\nM7MB3qTJwCrC5HPg2QqMM7N47//FoXUQVp+F4+Z3p0Zr3YALgXXABuC//K4nhMt5GsFdAsuAJd7t\nQoL7Rt8CPgH+DXTy2hvBI7M2AMsJHqnh+3K00Lo4C3jFu98HWACsB/4JxHjTY73H673n+/hddwsu\n/0gg3/ssvASkhdvnAPgVsAZYATwNxITjZ+F4bjqjWUREGoTL7iMREWkGhYKIiDRQKIiISAOFgoiI\nNFAoiIhIA4WCtDlmVmdmS8xsqZktNrMJx2ifambfbsZ83zEzXZO3ETN70symHbulhAuFgrRFFc65\nkc65EQQHL/ztMdqnAscMBb80OntWpM1TKEhbl0xweGPMLNHM3vK2Hpab2cVem7uAvt7Wxe+9tj/1\n2iw1s7saze+rZrbAzNaZ2ele24CZ/d7MFnrXEviWNz3LzOZ5811xqH1jZrbZzH7nvdcCM+vnTX/S\nzP5iZvOB33nXMXjJm/9HZja80TI94b1+mZl9xZt+npl96C3rP72xrDCzuyx4rYxlZna3N+2rXn1L\nzWzeMZbJzOzPFry2yL+BLi35x5L2T79gpC2KM7MlBM8wzSI4fhFAJTDVOXfAzDoDH5nZbIIDvQ11\nzo0EMLMLCA6ENtY5V25mnRrNO9I5N8bMLgRuJzg+zg0Eh3U41cxigPfN7P+ALxMcVvlO75oc8U3U\nu985N8zMriE4QusXvek9gAnOuToz+xPwsXPuEjM7G3iK4BnH/33o9V7tad6y/QI4xzl30Mx+Ctxs\nZg8AU4GBzjlnZqne+9wGnO+c295oWlPLNAoYQPC6IpkEh314vFl/FQkLCgVpiyoafcGPB54ys6EE\nh2L4HzM7g+Cw2N35dOjnxs4BnnDOlQM45/Y2eu7QAIGLgBzv/nnA8Eb71lMIXmxmIfC4BQcYfMk5\nt6SJemc2+vePjab/0zlX590/DfiKV8/bZpZuZslerZcfeoFzrsSCI70OJvhFDhANfEhwKOdK4DEL\nXlHuFe9l7wNPmtmzjZavqWU6A5jp1bXDzN5uYpkkTCkUpE1zzn3o/XLOIDiGUwYw2jlXY8GRUGOP\nc5ZV3r91fPr5N+B7zrk3Dm/sBdAXCH7p3uOce+pIZTZx/+Bx1tbwtsCbzrkrjlDPGIIDu00Dvguc\n7Zy70czGenUuMrPRTS2Tt4Uk0iT1KUibZmYDCV5GsZjgr91CLxAmAdles1IgqdHL3gSuN7N4bx6N\ndx8dyRvADG+LADM7xcwSzCwb2O2cewR4lODQ00dyWaN/P2yizbvAld78zwL2uOB1Lt4EvtNoedOA\nj4CJjfonEryaEoEU59xrwA8JXmITM+vrnJvvnLuN4IV1eja1TMA84DKvzyELmHSMdSNhRlsK0hYd\n6lOA4C/ea7398n8H/mVmywmO/rkGwDlXbGbvm9kKYI5z7hYzGwnkm1k18Brw86O836MEdyUttuD+\nmiLgEoKjrN5iZjVAGXBNE69PM7NlBLdCPvfr3vNLgruilgHlfDpE82+AB7za64BfOedeMLPrgJle\nfwAE+xhKgZfNLNZbLzd7z/3ezPp7094ClhIcGfVIy/QiwT6aVQSHlm4qxCRMaZRUkZPg7cLKc87t\n8bsWkZag3UciItJAWwoiItJAWwoiItJAoSAiIg0UCiIi0kChICIiDRQKIiLSQKEgIiIN/j/bWtEf\n3mp5AAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPuRFIPPgoiB",
        "colab_type": "text"
      },
      "source": [
        "## PlainRAdam (bufferless version from the reference repo)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.127121Z",
          "start_time": "2019-09-02T14:18:45.000Z"
        },
        "id": "7g7gnhEeWQtv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = cnn_learner(data, models.resnet18, opt_func=opt_radam_plain, metrics=[accuracy],)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.128106Z",
          "start_time": "2019-09-02T14:18:45.003Z"
        },
        "id": "jk2o23hjWQtz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn.unfreeze()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.129086Z",
          "start_time": "2019-09-02T14:18:45.006Z"
        },
        "id": "SQUYikpHWQt5",
        "colab_type": "code",
        "outputId": "39baa231-3616-4dac-80cf-cc4bd492f68f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "source": [
        "learn.fit(1, [1e-5, 1e-4, 1e-3])"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.161931</td>\n",
              "      <td>0.089183</td>\n",
              "      <td>0.970600</td>\n",
              "      <td>01:10</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.130046Z",
          "start_time": "2019-09-02T14:18:45.009Z"
        },
        "id": "44i2RVOtWQt8",
        "colab_type": "code",
        "outputId": "450cf8a6-5f56-4ee3-c4be-5eb2d9f47e42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        }
      },
      "source": [
        "learn.recorder.plot_losses()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXJ/u+ErYECJvse0RQ\nqeK+tOJCrai12sWrtbW1rV7be1ut1/6uba211l5vbavWXivuu9RdcUEkICCyGfawhiUkELJ/fn/M\nGCMGCJDhJJn38/GYBzNnvnPmcyZD3jnb55i7IyIiAhATdAEiItJ+KBRERKSJQkFERJooFEREpIlC\nQUREmigURESkiUJBRESaKBRERKSJQkFERJrEBV3AwerSpYsXFhYGXYaISIcyd+7cre6ed6BxHS4U\nCgsLKS4uDroMEZEOxczWtGacNh+JiEgThYKIiDSJWCiYWZKZfWBmC8zsYzP7ZQtjEs3sETMrMbPZ\nZlYYqXpEROTAIrlPoQY4yd13mVk88I6ZzXD395uN+Raww90HmNlFwK+Br0WwJhFpR+rq6igtLaW6\nujroUjqNpKQkCgoKiI+PP6TXRywUPHShhl3hh/Hh294Xb5gC3By+/zhwt5mZ6yIPIlGhtLSU9PR0\nCgsLMbOgy+nw3J1t27ZRWlpK3759D2keEd2nYGaxZjYf2AK84u6z9xqSD6wDcPd6YCeQ28J8rjSz\nYjMrLisri2TJInIEVVdXk5ubq0BoI2ZGbm7uYa15RTQU3L3B3UcDBcB4Mxt+iPO5192L3L0oL++A\nh9mKSAeiQGhbh/t5HpGjj9y9HHgDOGOvp9YDvQDMLA7IBLZFqo4XP9rIpp3adikisi+RPPooz8yy\nwveTgVOBpXsNexb4Rvj+VOD1SO1PWLe9iu8+NI8bnlgYidmLSAe0bds2Ro8ezejRo+nevTv5+flN\nj2tra1s1jyuuuIJly5ZFuNIjJ5JHH/UA/m5msYTC51F3f97MbgGK3f1Z4G/AP8ysBNgOXBSpYuau\n2QFAY6P2YYtISG5uLvPnzwfg5ptvJi0tjZ/85CefG+PuuDsxMS3/DX3//fdHvM4jKWJrCu6+0N3H\nuPtIdx/u7reEp/8iHAi4e7W7f9XdB7j7eHdfGal6zh2TT25qAmmJHa6zh4gcYSUlJQwdOpRLLrmE\nYcOGsXHjRq688kqKiooYNmwYt9xyS9PY448/nvnz51NfX09WVhY33ngjo0aNYuLEiWzZsiXApTg0\nUfUbsmtGEv/6eBMPvLuKy487tMO1RCQyfvncxyzeUNGm8xzaM4ObvjLskF67dOlSHnzwQYqKigC4\n7bbbyMnJob6+nsmTJzN16lSGDh36udfs3LmTE044gdtuu40f/ehH3Hfffdx4442HvRxHUlS1uUiO\nDy3uzc8tRqdCiMj+9O/fvykQAB5++GHGjh3L2LFjWbJkCYsXL/7Ca5KTkznzzDMBGDduHKtXrz5S\n5baZqFpT+NlZQ5j6v7MAWLu9ij65qQFXJCKfOtS/6CMlNfWz3w+ffPIJf/jDH/jggw/Iysri0ksv\nbfFcgISEhKb7sbGx1NfXH5Fa21JUrSl0TU9qur+ibNd+RoqIfKaiooL09HQyMjLYuHEjL730UtAl\nRUxUrSl0zUhsuv9eyTYmD+qqE2dE5IDGjh3L0KFDGTx4MH369OG4444LuqSIsY62bb2oqMgP5yI7\nWyqrOfuudyirrGFQt3SmXzmB7NSEA79QRNrckiVLGDJkSNBldDotfa5mNtfdi/bxkiZRtfkIQpuQ\nbg5vu1y2uZLL7vuA+obGgKsSEWkfoi4UAM4e2YPB3dMB+Gj9Tk79/UzqFAwiItEZCgDPff94Ft58\nGpMH5bFq625uf7nznKYuInKoojYU4mNjyEiK597Lihien8GD761hQ/meoMsSEQlU1IbCp+JjY/if\ni8dRXd/A9Dnrgi5HRCRQUR8KAL1zUzi2fy6PFa+jdEdV0OWIiARGoRB27UkD2VVdz0X3vq8WGCJR\nYvLkyV84Ee3OO+/k6quv3udr0tLSANiwYQNTp05tccyJJ57IgQ6dv/POO6mq+uyP0LPOOovy8vLW\nlh4xCoWwY/rl8h9nD6F0xx6WbKwMuhwROQKmTZvG9OnTPzdt+vTpTJs27YCv7dmzJ48//vghv/fe\nofDiiy+SlZV1yPNrKwqFZk4a3JW0xDhuenYRDbrugkinN3XqVF544YWmC+qsXr2aDRs2MGbMGE4+\n+WTGjh3LiBEjeOaZZ77w2tWrVzN8eOgKw3v27OGiiy5iyJAhnHfeeezZ89lBK1dffXVTy+2bbroJ\ngLvuuosNGzYwefJkJk+eDEBhYSFbt24F4I477mD48OEMHz6cO++8s+n9hgwZwne+8x2GDRvGaaed\n9rn3aStR1ebiQLpmJHHTV4Zy/eML+f7D8/ifS8YFXZJI9JhxI2z6qG3n2X0EnHnbPp/Oyclh/Pjx\nzJgxgylTpjB9+nQuvPBCkpOTeeqpp8jIyGDr1q1MmDCBc845Z59tce655x5SUlJYsmQJCxcuZOzY\nsU3P/epXvyInJ4eGhgZOPvlkFi5cyLXXXssdd9zBG2+8QZcuXT43r7lz53L//fcze/Zs3J1jjjmG\nE044gezsbD755BMefvhh/vKXv3DhhRfyxBNPcOmll7bNZxWmNYW9nDcmH4AXP9rEozoaSaTTa74J\n6dNNR+7Oz372M0aOHMkpp5zC+vXr2bx58z7nMXPmzKZfziNHjmTkyJFNzz366KOMHTuWMWPG8PHH\nH7fYcru5d955h/POO4/U1FTS0tI4//zzefvttwHo27cvo0ePBiLXmltrCnuJi41h0S9PZ8rd7/Db\nl5dx+rDuZKbEB12WSOe3n7/oI2nKlClcd911zJs3j6qqKsaNG8cDDzxAWVkZc+fOJT4+nsLCwhZb\nZR/IqlWruP3225kzZw7Z2dlcfvnlhzSfTyUmftbUMzY2NiKbj7Sm0IK0xDjuuHA0ZZU1PPDe6qDL\nEZEISktLY/LkyXzzm99s2sG8c+dOunbtSnx8PG+88QZr1qzZ7zy+9KUv8c9//hOARYsWsXDhQiDU\ncjs1NZXMzEw2b97MjBkzml6Tnp5OZeUXD2qZNGkSTz/9NFVVVezevZunnnqKSZMmtdXiHpBCYR9G\n9crilCFdeeC9VVTVdrwLZYhI602bNo0FCxY0hcIll1xCcXExI0aM4MEHH2Tw4MH7ff3VV1/Nrl27\nGDJkCL/4xS8YNy60P3LUqFGMGTOGwYMHc/HFF3+u5faVV17JGWec0bSj+VNjx47l8ssvZ/z48Rxz\nzDF8+9vfZsyYMW28xPsWda2zD8bcNTu44J73uGXKMC6bWHhE3lMkmqh1dmSodXaEjOuTzYCuacz4\naFPQpYiIHBEKhQM4b0w+s1Zu4w+vfhJ0KSIiEadQOIDvTOrHeWPy+f2ry1m8oSLockQ6nY62Cbu9\nO9zPU6FwAAlxMfz8y0OJizGeXbAh6HJEOpWkpCS2bdumYGgj7s62bdtISko65HlE7DwFM+sFPAh0\nAxy4193/sNeYE4FngFXhSU+6+y2RqulQ5aQmMGlgF55bsIEbTh9ETEzLZzWKyMEpKCigtLSUsrKy\noEvpNJKSkigoKDjk10fy5LV64MfuPs/M0oG5ZvaKu+99Ot/b7v7lCNbRJs4Z3ZPrHlnA3LU7OLow\nJ+hyRDqF+Ph4+vbtG3QZ0kzENh+5+0Z3nxe+XwksAfIj9X6RdurQ7iTExfCIWl+ISCd2RPYpmFkh\nMAaY3cLTE81sgZnNMLNhR6KeQ5GWGMc5o3ry+NxSZq3YFnQ5IiIREfFQMLM04Angh+6+9+E784A+\n7j4K+CPw9D7mcaWZFZtZcZDbHm89dzgF2clc9X9z2VJ56P1LRETaq4iGgpnFEwqEh9z9yb2fd/cK\nd98Vvv8iEG9mXVoYd6+7F7l7UV5eXiRL3q+k+Fhu+sowdu6p4xv3zdEREyLS6UQsFCzUePxvwBJ3\nv2MfY7qHx2Fm48P1tOttM6cO7cZlE/uwZGMFry3ZEnQ5IiJtKpJrCscBXwdOMrP54dtZZnaVmV0V\nHjMVWGRmC4C7gIu8A/z5/YsvD6VbRiIPzd5/50QRkY4mYoekuvs7wH4P6Hf3u4G7I1VDpMTFxvC1\nol788Y0SSndUUZCdEnRJIiJtQmc0H6Kvje+NgQ5RFZFORaFwiPKzkjlxUFcembOOuobGoMsREWkT\nCoXDcPH43myprOEfs7RvQUQ6B4XCYTh5SFcmDezC3W+UaG1BRDoFhcJhMDO+MbGQ7btreadka9Dl\niIgcNoXCYZp0VBfSk+J4fsHGoEsRETlsCoXDlBgXyxnDuvPSx5uorK4LuhwRkcOiUGgDl07ow66a\neh2eKiIdnkKhDYzqlcWx/XP54+sllFfVBl2OiMghUyi0kRvOGMzOPXW8qn5IItKBKRTayKiCTLqm\nJ/LGUoWCiHRcCoU2YmacNqwbry7ZzLZdNUGXIyJySBQKbejrEwqpqW/kmfkbgi5FROSQKBTa0KDu\n6YzIz+SJeaVBlyIickgUCm3s/LH5fLyhgvnryoMuRUTkoCkU2tjUcQWkJ8bx0PtqkiciHY9CoY2l\nJ8Vz+vDu/GvRJqpq64MuR0TkoCgUImDa+N5U1tTz+1eWB12KiMhBUShEwLg+2Uwb35u/vbOKddur\ngi5HRKTVFAoR8t0T++PA7S8vC7oUEZFWUyhESK+cFL4/eQDPzN/A8s2VQZcjItIqCoUIumRCH2Jj\njAdnrQ66FBGRVlEoRFC3jCQuLOrFo8Wlan0hIh2CQiHCvnlcIbX1jUzXtRZEpANQKETYwG7pHD+g\nC/+YtYa6hsagyxER2a+IhYKZ9TKzN8xssZl9bGY/aGGMmdldZlZiZgvNbGyk6gnSFccVsqmimn8t\n2hR0KSIi+xXJNYV64MfuPhSYAFxjZkP3GnMmMDB8uxK4J4L1BGbyoK707ZLKrS8sZk9tQ9DliIjs\nU8RCwd03uvu88P1KYAmQv9ewKcCDHvI+kGVmPSJVU1BiYoxbzx3O5ooanl2wPuhyRET26YjsUzCz\nQmAMMHuvp/KB5ntgS/licHQKx/bPZXD3dO5/dzXuHnQ5IiItingomFka8ATwQ3evOMR5XGlmxWZW\nXFZW1rYFHiFmxuXHFrJ0UyUfrNoedDkiIi2KaCiYWTyhQHjI3Z9sYch6oFezxwXhaZ/j7ve6e5G7\nF+Xl5UWm2CNgyuh8slLi+es7q4IuRUSkRZE8+siAvwFL3P2OfQx7FrgsfBTSBGCnu2+MVE1BS06I\n5dJj+vDqks1s2lkddDkiIl8QyTWF44CvAyeZ2fzw7Swzu8rMrgqPeRFYCZQAfwG+G8F62oULxhXg\njnY4i0i7FBepGbv7O4AdYIwD10Sqhvaob5dUxhfmcN87q7lsYiFJ8bFBlyQi0kRnNAfgulOPYlNF\nNbfNWBp0KSIin6NQCMDE/rlcNrEPf5+1mrlrdgRdjohIE4VCQG44YzBd0hK5580VQZciItJEoRCQ\ntMQ4pozqyVvLt7Czqi7ockREAIVCoM4Z3ZO6BudfH3fao3BFpINRKARoRH4mfXJTeH6hQkFE2geF\nQoDMjAvGFvD2J1sp2bIr6HJERBQKQbuwKNTl47kFGwKuREREoRC47plJTB6Ux4OzVlNVWx90OSIS\n5RQK7cD3ThrAjqo6HisuDboUEYlyCoV2YFyfHEbkZ3LvzJXsrtHagogER6HQTtxwxiDWl+/hhY90\nJJKIBEeh0E4cP6AL/fNSeWTOugMPFhGJEIVCO2FmTBvfm7lrdrB00yFdoE5E5LApFNqRC8YWkBAX\nwz9nrw26FBGJUgqFdiQ7NYGzhnfnqXnrqaxWPyQROfIUCu3Mt47vR2VNPX99W9dxFpEjT6HQzowo\nyOTUod34+6zVVNc1BF2OiEQZhUI79K3j+1JeVceNTywMuhQRiTKtCgUz629mieH7J5rZtWaWFdnS\noteEfrlcfWJ/np6/gVkrtgVdjohEkdauKTwBNJjZAOBeoBfwz4hVJfzg5IF0TU/kzleXB12KiESR\n1oZCo7vXA+cBf3T364EekStLkuJj+e6J/Zm9ajtzVm8PuhwRiRKtDYU6M5sGfAN4PjwtPjIlyacu\nPLoXSfExPPT+mqBLEZEo0dpQuAKYCPzK3VeZWV/gH5ErSwBSEuK4sKgXT8/fwPMLdb0FEYm8VoWC\nuy9292vd/WEzywbS3f3XEa5NgP88eyh9clP40xsrcPegyxGRTq61Rx+9aWYZZpYDzAP+YmZ3RLY0\nAUiIi+H7Jw1kycYKXvxoU9DliEgn19rNR5nuXgGcDzzo7scAp+zvBWZ2n5ltMbNF+3j+RDPbaWbz\nw7dfHFzp0eO8Mfkc1S2N219eRl1DY9DliEgn1tpQiDOzHsCFfLaj+UAeAM44wJi33X10+HZLK+cb\ndWJjjOtPH8yqrbt5tFittUUkclobCrcALwEr3H2OmfUDPtnfC9x9JqBjKdvIKUO6UtQnmz+8+gl7\natX+QkQio7U7mh9z95HufnX48Up3v6AN3n+imS0wsxlmNqwN5tdpmRn/fuZgtlTWcP97apYnIpHR\n2h3NBWb2VHgfwRYze8LMCg7zvecBfdx9FPBH4On9vP+VZlZsZsVlZWWH+bYd19GFOZw8uCv3vLmC\n8qraoMsRkU6otZuP7geeBXqGb8+Fpx0yd69w913h+y8C8WbWZR9j73X3IncvysvLO5y37fBuOGMw\nu2rquW3G0qBLEZFOqLWhkOfu97t7ffj2AHBYv53NrLuZWfj++HAt6v52AIO6p3PZhD48NreUZZsq\ngy5HRDqZ1obCNjO71Mxiw7dLOcAvcDN7GJgFDDKzUjP7lpldZWZXhYdMBRaZ2QLgLuAi19lZrXL1\niQNITYjlty8tC7oUEelk4lo57puEtvv/HnDgPeDy/b3A3acd4Pm7gbtb+f7STPfMJL49qR93vLKc\nuWt2MK5PdtAliUgn0dqjj9a4+znunufuXd39XKAtjj6SQ/TN4/uSnhTHXa/t98hgEZGDcjhXXvtR\nm1UhBy0tMY5rTxrIW8vLeGPZlqDLEZFO4nBCwdqsCjkk3zi2kL5dUrntxaXUq/2FiLSBwwkF7RQO\nWEJcDNefPohlmyv5fy/qEFUROXz7DQUzqzSzihZulYTOV5CAnTWiB9+Y2If73l3FeyVbgy5HRDq4\n/YaCu6e7e0YLt3R3b+2RSxJhN545hH5dUrn+8YXqoioih+VwNh9JO5GcEMv1pw9iffke3lwWvW1A\nROTwKRQ6iRMHdaUgO5kfTP+QVVt3B12OiHRQCoVOIjkhloe/M4H6RudXLyzWpTtF5JAoFDqRXjkp\nXHvSAF5dsoVH5uhiPCJy8BQKncx3TxzAxH653PrCEjbu3BN0OSLSwSgUOpmYGOPXF4yktr6Rax6a\nR0V1XdAliUgHolDohHrnpvC7C0cxf105v/mXTmoTkdZTKHRSXxnVk8uP7ctDs9fy/kpdpkJEWkeh\n0In9+LSj6Jubyg+mf8jWXTVBlyMiHYBCoRNLTYzjT5eMpbyqjp88tkCHqYrIASkUOrkhPTL4j7OH\n8OayMm5QGwwROQD1L4oCX5/Qh5Itu3hw1hoS42O49dwRQZckIu2U1hSigJlxy5ThXDC2gEeLS9lS\nWR10SSLSTikUosg1k/sD8POnF2n/goi0SKEQRfrlpfGjU4/ipY838+yCDUGXIyLtkEIhynxnUj9G\n98ripmc/1mYkEfkChUKUiY0xbv/qKKpqG/jPpxYFXY6ItDMKhSg0oGsaPzxlIC8v3swHq7YHXY6I\ntCMKhSh1xbF9yU1N4O43SoIuRUTakYiFgpndZ2ZbzKzFbRQWcpeZlZjZQjMbG6la5IuSE2L51qS+\nzFxextw1WlsQkZBIrik8AJyxn+fPBAaGb1cC90SwFmnB1yf0IT8rmWsfns+O3bVBlyMi7UDEQsHd\nZwL7+xN0CvCgh7wPZJlZj0jVI1+UnhTP/1wylrLKGn7wyHwaGnXugki0C3KfQj7Q/JqRpeFpcgSN\n6pXFTecMZebyMv7jqY90UptIlOsQO5rN7EozKzaz4rKysqDL6XQuHt+bs0f0YPqcddz37uqgyxGR\nAAUZCuuBXs0eF4SnfYG73+vuRe5elJeXd0SKiyZmxt0Xj+H0Yd34r+cX879vrQi6JBEJSJCh8Cxw\nWfgopAnATnffGGA9Uc3M+N2FoxlfmMNvX1rGko0VQZckIgGI5CGpDwOzgEFmVmpm3zKzq8zsqvCQ\nF4GVQAnwF+C7kapFWictMY4/f30cmcnx/PTJj2jUjmeRqBOx6ym4+7QDPO/ANZF6fzk02akJ/PzL\nQ7jukQU8NHsNX59YGHRJInIEdYgdzXJknTs6n0kDu/Bfzy/hvZKtQZcjIkeQQkG+wMz447Qx9M5N\n4SePLaCyui7okkTkCFEoSIuyUhL4zdSRbKyo5jf/WhZ0OSJyhCgUZJ/G9s7mimP78o/31/CdB4vZ\nuqsm6JJEJMIUCrJfN5wxiG8e15e3lpXx0yd1xrNIZ6dQkP1Kio/lF18ZynWnHsUrizfzwHurgy5J\nRCJIoSCtctUJ/ZjYL5fbX1rGwtLyoMsRkQhRKEirmBm3njecjOR4vv33Yjbt1PWdRTojhYK0Wv+8\nNO6/4mh21dQz+fY3+d3Ly6hraAy6LBFpQwoFOSiDu2fw6L9NZEiPdP74egljbnmFVxdvDrosEWkj\nCgU5aMPzM3ni6mP562VFdE1P5Kr/m8sLC9XLUKQzUCjIITEzThnajcevPpZhPTO47pH5PDJnra7e\nJtLBKRTksOSkJnD/FeMpyE7m35/4iFN//xYlWyqDLktEDpFCQQ5bTmoCM344id9MHUnFnnrO/dN7\nPDpnnVpvi3RACgVpE4lxsVxY1IvHrppIVko8NzyxkGunf6gzoEU6GIWCtKm+XVKZ8YNJXHxMb55f\nuJGfP7NIh62KdCARu8iORK/0pHh+de5wEmJjmtpi3HruiGCLEpFWUShIRJgZN58zjMT4GP781kpy\nUhL44SlHERNjQZcmIvuhUJCIuv60QZRu38Ndr5dQU9/Ij08bREKctlqKtFf63ykRFRcbw90Xj+H8\nsfn8eeZKzr/nXVZt3R10WSKyDwoFiTgz43dfHcXdF49hzdYqzr7rbe5/dxX12gEt0u4oFOSIMDO+\nPLInz3zvOIb2yOCXzy3mqv+bx5YKdVsVaU8UCnJE9ctL47GrJnL96YOYubyMM//wNv/94hJmrdim\ncxpE2gGFghxxZsY1kwfw1DXHkpkSz59nrmTaX97nB9PnU1ldF3R5IlFNoSCBGdYzkyevPpYfnXoU\nE/vl8sJHG/nyH9/Rld1EAmQdbZW9qKjIi4uLgy5DIqB49XauffhDNlVUM3VcAd8/aSC9clKCLkuk\nUzCzue5edKBxEV1TMLMzzGyZmZWY2Y0tPH+5mZWZ2fzw7duRrEfat6LCHF64dhInHJXHo8WlTPrN\nG1z/2AIWrd8ZdGkiUSNiawpmFgssB04FSoE5wDR3X9xszOVAkbt/r7Xz1ZpCdFi2qZKHP1jL32et\nxh2O6ZtD/65pfPO4vgzomhZ0eSIdTntYUxgPlLj7SnevBaYDUyL4ftKJDOqezs3nDGPWjSdz1Qn9\nmb1qO/+cvZbTfv8Wt81YSnVdQ9AlinRKkWxzkQ+sa/a4FDimhXEXmNmXCK1VXOfu61oYI1Gqe2YS\nN545mB+dehRrt+/mnjdX8r9vreDemSsY3zeHW88dQW5qAtmpCUGXKtIpBH300XNAobuPBF4B/t7S\nIDO70syKzay4rKzsiBYo7UNCXAwDuqbzuwtH8dC3j+HowhzeX7mdU+54iwn//Rq/fO5jyiprgi5T\npMOL5D6FicDN7n56+PFPAdz9v/cxPhbY7u6Z+5uv9inIpz5cu4N5a8tZvKGCJ+aVkpEUx5TR+Vx7\n8kDy0hODLk+kXWntPoVIbj6aAww0s77AeuAi4OLmA8ysh7tvDD88B1gSwXqkkxnTO5sxvbMBuOK4\nQn738jKmz1nLcws3cNUJ/SnMTeXkIV2Jjw16hVik44hYKLh7vZl9D3gJiAXuc/ePzewWoNjdnwWu\nNbNzgHpgO3B5pOqRzm14fib3XzGeki2V3PD4Qm6bsbTpubNH9iA5PpaC7GQGdE3jpMFdSUlQ13iR\nlujkNemUyipreGb+eu567RN21dTTuNfXfFC3dKaOK8AMLjy6FxlJ8cEUKnKEtHbzkUJBOj13x8yo\nrK5j5vKtvLhoI3NX72BTuENrSkIs10wewMXje5MYH8PKst0UZCeTlaIjmqTzUCiI7EdDozN/XTmL\nN+zkuYUb+WDVdrplJLK7poFdNfXEGJw0uCv/dkJ/ji7MCbpckcOmUBA5CAtLy7nmn/PomZnM+WPz\nWb55F0/MK6W8qo5hPTP4yqienDOqJz2zkoMuVeSQKBREDtKnm5k+VVldx5/fWsmrSzazdFMlEFp7\nmDK6J2eN6KGjmqRDUSiItKEF68p5bO46Hp9bSnVdI6kJsZw2rDunDOlGSkIsCXExpCfFkZ2SQFZK\nPOnacS3tjEJBJAJq6ht4a1kZf3qjhEUbKmjY+7AmIMbghKPyGNs7mzNH9FADP2kXFAoiEVZd18CS\njRWU7thDZnI823fXsrmimrLKGqbPWceumnoAji7M5uwRPTiqWzpj+2RTXdfArBXbqG1oJDc1kWP6\n5TBn9XYGdk3XmdgSMQoFkQDtqW1gR1Utzy7YwPQP1rJ6W9U+xybGxVBT3wiETrSbdnRvigqzSYqP\npb6hkX+8v4byqjqyU+I5dkAXjuqWfqQWQzoRhYJIO9HY6HywejvlVbV8uLacT7bsYkyvLCYP7spH\n63fy9idlDOmewYqyXbzw0UbqGpy0xDgm9MulrLKaBaWfXWQoLsa4dEIfCrKTGZ6fyZAeGWQma/+F\nHJhCQaQDqqyu4/2V23lo9hpmLi8jJSGOc8f05HuTB+I4v56xlGcWbKD5f9vB3dO5+sT+nDq0m9p3\nyD4pFEQ6uL0Pkf1UTX0Dn2zexTslW9ldU8+T89azvnwPqQmxnD68OxP75fJuyVZGFGRx8fjeJCfE\nBlC9tDcKBZEoUd/QyAert3MeKdo4AAANA0lEQVTfO6t5dclmAJLiY6iuayQuxijskkqXtARGFmRx\nypBu9M9LJSc1AXfYuruGvLTEFsNHOheFgkgU2r67lqUbKxjbJ5v568p5Zv4GSndUUVldz4LS8qbN\nTvlZyZRV1lDb0EhmcjzdM5LomZXExp3V5KUncvyALnzpqDwGdUsnJkaB0RkoFETkczaU7+H9ldso\n3bGHD9fuoHxPHcf0zaWiuo45q7azettuhvTIYEdVLeu27wEgNsYY3jODbhlJFHZJ5d++1I+0pDge\nen8t8bFGXnoSZqFeUpnJ8XTPTKIwN5XYCARJY6MroA6DQkFEDkpDoxMbY9TWN7KwtJySLbtYtW03\nC9aVs3ZbFRt2VhMbYy2esLe37hlJ5GcnEx9rbK6oYXD3dNIS48hNS8QMausbOeGoPIb1zGDWym0s\n31RJQlwMvXNTyU6JZ96acrqkJ9AzKxkDpn+wjrlrd9AzM4ma+kb656XROzeFqpp60pLi6JGZzIR+\nufTPS6W6rpH15VX0ykkhMS6WxkZnS2UN3TKiezOZQkFE2tTyzZU8MmcdDY3O+L45DO6ezp66Buob\nnOq6BjZVVFNRXc/LH2+iYk8dK8p20y8vlV019VTsqaeuoZGq2nrcISYcPgcjPSmOvPRE+uamsmxz\nJaU79nxhTG5qApU19dTWN5KdEs+kgXks21TJss2VdM9IYmC3NHJSEyjqE7pqX4wZR3VLI26vPlb1\nDY18tH4nKQlx9M9L/cLzHZFCQUTancZGp77RaWh03luxlSUbK5jYPze82aqO9Tv2sKWymkkD81iz\nbXfT2eL98lLpkfn5DrVVtfVs3FlNz8xkSrbsYkFpOS8v3kxGUhxDemSwbFMlb39SRo/MZI4f2IUV\nW3axdXctWyqq2bizumk+uakJnDasG71zUimvquWpD9ezpbKm6fmEuBjiY4y89ETOGtGDs0b0wAx6\nZiaH1noaGslOSSA+NgZ3Z335HrqkJZIU//mjvpZuqqB0+x7Wbq8iJzWBssoakhJCJyiu3V7FwK7p\njMjPpLImFKgF2cmcMDCvzTaZKRRERPahZMsu5qzeTkJsDG8uL+P1JZvZXdsAwMCuaYzrk82w/EwS\nYo15a8pxnNXbqpi7Zsc+N5/lpiYQG2NsqawhPtYY2jOTvLREyqtq2VVT39Rp92B0SUugIDuF7btr\naXTn/suPZuAhntHe2lDQmS4iEnUGdE1ralR4wbgC6hoaqW9wEuJivrCT/GtH9266v6WimvdWbCM+\nNobV23aTHB9LVW09e+oa2Larltr6RobnZ1K2q4bZK7fx+tLNdElLDO2kP6EfA/JC71tZXU+f3BTc\nocGd5PhYyqvqKN0RaocysFs6C9aVM/OTMrZU1JCflUxcrB2RzVhaUxARiZC6hsZ2c92N1q4ptI9q\nRUQ6ofYSCAej41UsIiIRo1AQEZEmCgUREWmiUBARkSYRDQUzO8PMlplZiZnd2MLziWb2SPj52WZW\nGMl6RERk/yIWCmYWC/wJOBMYCkwzs6F7DfsWsMPdBwC/B34dqXpEROTAIrmmMB4ocfeV7l4LTAem\n7DVmCvD38P3HgZMtmjtWiYgELJKhkA+sa/a4NDytxTHuXg/sBHIjWJOIiOxHh9jRbGZXmlmxmRWX\nlZUFXY6ISKcVyVBYD/Rq9rggPK3FMWYWB2QC2/aekbvf6+5F7l6Ul5cXoXJFRCSSoTAHGGhmfc0s\nAbgIeHavMc8C3wjfnwq87h2tGZOISCcSsS6p7l5vZt8DXgJigfvc/WMzuwUodvdngb8B/zCzEmA7\noeAQEZGAdLguqWZWBqw5xJd3Aba2YTkdkT4DfQagzwCi7zPo4+4H3P7e4ULhcJhZcWtax3Zm+gz0\nGYA+A9BnsC8d4ugjERE5MhQKIiLSJNpC4d6gC2gH9BnoMwB9BqDPoEVRtU9BRET2L9rWFEREZD+i\nJhQO1Ma7szCzXmb2hpktNrOPzewH4ek5ZvaKmX0S/jc7PN3M7K7w57LQzMYGuwRtw8xizexDM3s+\n/LhvuD17Sbhde0J4eqdt325mWWb2uJktNbMlZjYxCr8H14X/Hywys4fNLCkavwsHIypCoZVtvDuL\neuDH7j4UmABcE17WG4HX3H0g8Fr4MYQ+k4Hh25XAPUe+5Ij4AbCk2eNfA78Pt2nfQahtO3Tu9u1/\nAP7l7oOBUYQ+j6j5HphZPnAtUOTuwwmdRHsR0fldaD137/Q3YCLwUrPHPwV+GnRdR2jZnwFOBZYB\nPcLTegDLwvf/DExrNr5pXEe9Eeqz9RpwEvA8YIROUorb+/tA6Iz7ieH7ceFxFvQytMFnkAms2ntZ\noux78GkX5pzwz/Z54PRo+y4c7C0q1hRoXRvvTie8+jsGmA10c/eN4ac2Ad3C9zvjZ3MncAPQGH6c\nC5R7qD07fH4ZO2v79r5AGXB/eDPaX80slSj6Hrj7euB2YC2wkdDPdi7R9104KNESClHHzNKAJ4Af\nuntF8+c89KdQpzzszMy+DGxx97lB1xKwOGAscI+7jwF289mmIqBzfw8AwvtLphAKyJ5AKnBGoEV1\nANESCq1p491pmFk8oUB4yN2fDE/ebGY9ws/3ALaEp3e2z+Y44BwzW03oan8nEdq2nhVuzw6fX8ZW\ntW/vgEqBUnefHX78OKGQiJbvAcApwCp3L3P3OuBJQt+PaPsuHJRoCYXWtPHuFMKXM/0bsMTd72j2\nVPM25d8gtK/h0+mXhY8+mQDsbLZ5ocNx95+6e4G7FxL6Ob/u7pcAbxBqzw5fXP5O177d3TcB68xs\nUHjSycBiouR7ELYWmGBmKeH/F59+BlH1XThoQe/UOFI34CxgObAC+I+g64ngch5PaJPAQmB++HYW\noW2jrwGfAK8COeHxRujIrBXAR4SO1Ah8OdroszgReD58vx/wAVACPAYkhqcnhR+XhJ/vF3Tdbbj8\no4Hi8HfhaSA72r4HwC+BpcAi4B9AYjR+Fw7mpjOaRUSkSbRsPhIRkVZQKIiISBOFgoiINFEoiIhI\nE4WCiIg0UShIu2NmDWY238wWmNk8Mzv2AOOzzOy7rZjvm2ama/I2Y2YPmNnUA4+UaKFQkPZoj7uP\ndvdRhJoX/vcBxmcBBwyFoDQ7e1ak3VMoSHuXQai9MWaWZmavhdcePjKzKeExtwH9w2sXvw2P/ffw\nmAVmdluz+X3VzD4ws+VmNik8NtbMfmtmc8LXEvi38PQeZjYzPN9Fn45vzsxWm9lvwu/1gZkNCE9/\nwMz+18xmA78JX8fg6fD83zezkc2W6f7w6xea2QXh6aeZ2azwsj4W7mWFmd1moWtlLDSz28PTvhqu\nb4GZzTzAMpmZ3W2ha4u8CnRtyx+WdHz6C0bao2Qzm0/oDNMehPoXAVQD57l7hZl1Ad43s2cJNXob\n7u6jAczsTEKN0I5x9yozy2k27zh3H29mZwE3EeqP8y1CbR2ONrNE4F0zexk4n1Bb5V+Fr8mRso96\nd7r7CDO7jFCH1i+HpxcAx7p7g5n9EfjQ3c81s5OABwmdcfzzT18frj07vGz/CZzi7rvN7N+BH5nZ\nn4DzgMHu7maWFX6fXwCnu/v6ZtP2tUxjgEGErivSjVDbh/ta9VORqKBQkPZoT7Nf8BOBB81sOKFW\nDP/PzL5EqC12Pp+1fm7uFOB+d68CcPftzZ77tEHgXKAwfP80YGSzbeuZhC42Mwe4z0INBp929/n7\nqPfhZv/+vtn0x9y9IXz/eOCCcD2vm1mumWWEa73o0xe4+w4LdXodSugXOUACMItQK+dq4G8WuqLc\n8+GXvQs8YGaPNlu+fS3Tl4CHw3VtMLPX97FMEqUUCtKuufus8F/OeYR6OOUB49y9zkKdUJMOcpY1\n4X8b+Oz7b8D33f2lvQeHA+hsQr9073D3B1sqcx/3dx9kbU1vC7zi7tNaqGc8ocZuU4HvASe5+1Vm\ndky4zrlmNm5fyxReQxLZJ+1TkHbNzAYTuoziNkJ/7W4JB8JkoE94WCWQ3uxlrwBXmFlKeB7NNx+1\n5CXg6vAaAWZ2lJmlmlkfYLO7/wX4K6HW0y35WrN/Z+1jzNvAJeH5nwhs9dB1Ll4Brmm2vNnA+8Bx\nzfZPpIZrSgMy3f1F4DpCl9jEzPq7+2x3/wWhC+v02tcyATOBr4X3OfQAJh/gs5EoozUFaY8+3acA\nob94vxHeLv8Q8JyZfUSo++dSAHffZmbvmtkiYIa7X29mo4FiM6sFXgR+tp/3+yuhTUnzLLS9pgw4\nl1CX1evNrA7YBVy2j9dnm9lCQmshX/jrPuxmQpuiFgJVfNai+VbgT+HaG4BfuvuTZnY58HB4fwCE\n9jFUAs+YWVL4c/lR+LnfmtnA8LTXgAWEOqO2tExPEdpHs5hQa+l9hZhEKXVJFTkM4U1YRe6+Neha\nRNqCNh+JiEgTrSmIiEgTrSmIiEgThYKIiDRRKIiISBOFgoiINFEoiIhIE4WCiIg0+f+RgJ1uZCPo\n/wAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcTaLpWfgxp0",
        "colab_type": "text"
      },
      "source": [
        "## Training the PlainRAdam with just the lowest lr (first in the param_groups)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uEtdo6QGaYKe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = cnn_learner(data, models.resnet18, opt_func=opt_radam_plain, metrics=[accuracy],)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5MBdoHs5WQuG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn.unfreeze()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S7wOlokdbIy1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        },
        "outputId": "7566107a-b1cf-49cc-84eb-b6af6039ab9f"
      },
      "source": [
        "learn.fit(1, 1e-5)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.213442</td>\n",
              "      <td>0.902260</td>\n",
              "      <td>0.773000</td>\n",
              "      <td>01:10</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hBGL1VxobJnj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "outputId": "ad9cf22d-4eb0-45dd-ecf2-8bc205e5a7bc"
      },
      "source": [
        "learn.recorder.plot_losses()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VGX2+PHPSSEhhQAhgRACoZMQ\nWogUKYIgzYIoKigqNhQrtt/Xtuq6q6ur6yJ2VkGxYMdKV3bpJUFCR0BaQk1ogVBSzu+PuWDABIaQ\nyaSc9+s1L2ae+9w7506GnDz3PkVUFWOMMeZsfLwdgDHGmPLBEoYxxhi3WMIwxhjjFksYxhhj3GIJ\nwxhjjFssYRhjjHGLxxKGiASKyGIRSRWRVSLy10LqBIjI5yKyQUQWiUhsgW2PO+XrRKSvp+I0xhjj\nHk+2MI4BF6tqG6At0E9EOp1W5zZgn6o2Af4NvAQgIvHAEKAl0A94S0R8PRirMcaYs/BYwlCXQ85L\nf+dx+ijBgcCHzvOvgF4iIk75Z6p6TFU3ARuADp6K1RhjzNn5efLgTqsgBWgCvKmqi06rEg1sA1DV\nXBE5AIQ75QsL1Etzys6oVq1aGhsbWwKRG2NM5ZCSkpKhqhHu1PVowlDVPKCtiFQHJolIgqquLMn3\nEJERwAiA+vXrk5ycXJKHN8aYCk1Etrhbt1R6SanqfmAWrvsRBaUDMQAi4geEAZkFyx31nLLCjj1W\nVZNUNSkiwq0kaYwxphg82UsqwmlZICJVgUuAtadV+x642Xk+GPhFXbMhfg8McXpRNQSaAos9Fasx\nxpiz8+QlqSjgQ+c+hg/whar+KCLPAcmq+j3wPvCRiGwA9uLqGYWqrhKRL4DVQC5wj3N5yxhjjJdI\nRZrePCkpSe0ehjHlX05ODmlpaRw9etTboVQYgYGB1KtXD39//1PKRSRFVZPcOYZHb3obY0xxpKWl\nERoaSmxsLK6e9uZ8qCqZmZmkpaXRsGHDYh/HpgYxxpQ5R48eJTw83JJFCRERwsPDz7vFZgnDGFMm\nWbIoWSXxeVrCcExduZP0/UcASN9/hEm/plGR7u8YY8z5snsYQG5ePnd9nEIVPx+WPNmbLi/+AkBQ\nFT/6tqzj5eiMMaUtMzOTXr16AbBz5058fX05Mc5r8eLFVKlS5azHuOWWW3jsscdo3ry5R2MtTZYw\ngH3ZOQAcz82nzV+nnyyfsmKHJQxjKqHw8HCWLVsGwLPPPktISAiPPPLIKXVUFVXFx6fwCzXjx4/3\neJylzS5JAXsPHy+0/Ntl2+n4wkwm/ZpGXr5y+FhuKUdmjClLNmzYQHx8PDfccAMtW7Zkx44djBgx\ngqSkJFq2bMlzzz13sm7Xrl1ZtmwZubm5VK9enccee4w2bdrQuXNndu/e7cWzKD5rYQCZh4+d8npU\n76b0jqvNDe8tYtfBYzz4eSoPfp4KwKvXtqFXXG3CqvoXdihjTAn76w+rWL39YIkeM75uNZ65vGWx\n9l27di0TJkwgKck1dOHFF1+kZs2a5Obm0rNnTwYPHkx8fPwp+xw4cICLLrqIF198kYceeohx48bx\n2GOPnfd5lDZrYQDp+1w3u/+vXwsAhnVqQEJ0GL88fBFXJ9Y7pe5DX6Ry07jF5OTll3qcxhjva9y4\n8clkATBx4kQSExNJTExkzZo1rF69+k/7VK1alf79+wPQvn17Nm/eXFrhlihrYQBrdmQR6O/DiO6N\nGNmj8cny8JAAXrq6FV8vTQOgTb0wNuw+ROq2/fR+9X88c3k8F7eoDcDC3zNJiA4jJMA+UmNKUnFb\nAp4SHBx88vn69et57bXXWLx4MdWrV2fYsGGFjnUoeJPc19eX3NzyeXnbfrsBGYeOUbtaIL4+f+6n\n7Ofrw88PX0TW0VzaxlTnyPE8Rs/8jUm/pnP7h8nc0a0R787+/WT96OpV6d4sAlUlMjSAu3s2IdDf\nFgs0piI6ePAgoaGhVKtWjR07djBt2jT69Tt9Uu6KwxIGkJufj18hyeKExhEhJ59XreLL4wPiuL9X\nU65+e/4pyQJcYzgmLt568vXvGYd55Zo2ljSMqYASExOJj4+nRYsWNGjQgC5dung7JI+yyQeBOyYk\ns21vNlNHdT+n/TIOHePuj5cyrHMDrmhTl+Vp+8k8fJz0fUdQYMmmvXyfup24qGqM6t2Ubk1rEVTl\nzDk6ddt+IkIDqF0tkOTNe4msFkjDWsFn3MeYimbNmjXExcV5O4wKp7DP1SYfPEe5efn4+Z77sPla\nIQF8cVfnk69b16t+yvYbOzWgUUQwo2eu586PUqji60Pd6oEM69SAvHzl57W7+XXrPqr6+/LgJc3Y\nl53DmJ/XA9A0MoT1u11Loj97eTzDuxR/wjBjjCkJljCA3HzFr4jBN+drVO9m9GweyYfzNzN7fQab\nM7P5+09rAIipWZWLmkUwa90e/vqDq2dFRGgAESEBiMAVbeqyOfMwz/6wmmO5+dx5UeMzvZUxxniU\nJQwgJy8f/2K0MNzVJqY6r17Xlty8fH7dtp+Ji7YyOKkenRu5ZuPcdfAoz3y3iuFdYunUKPyUfY/m\n5HHj+4v4x5S1hIcEMLh9vSLexRhjPMtjCUNEYoAJQG1AgbGq+tppdR4FbigQSxwQoap7RWQzkAXk\nAbnuXmMrjtw8xd/X80NS/Hx9uCC2JhfE1jylvHa1QN65sX2h+wT6+/Lx7R259YMlPPJlKnuyjnFH\nt4b4lUK8xhhTkCd/6+QCD6tqPNAJuEdEThn+qKovq2pbVW0LPA78T1X3FqjS09nusWQBkJOv+PuV\n3V/AAX6+vH/zBfRqEclLU9dy38RfWZl+gL2HjzNvQwYTF28tcnoTY4wpKR5rYajqDmCH8zxLRNYA\n0bjW6S7MUGCip+I5k9y8fPzP0K22LAj092XsTUnc88lSpqzcyZSVO0/Z/sLkNfSOq02TyBC6NqlF\nm5jqRRzJGGOKp1T+rBaRWKAdsKiI7UFAP+DrAsUKTBeRFBEZ4cn4cvO0WL2kSpuvj/D4gBaM6N6I\nK9rUZWiHGDo1qkm3prVoFBHCTyt28PK0dQx8cx5/+XalredhTDH17NmTadOmnVI2evRoRo4cWeQ+\nISGu8Vrbt29n8ODBhdbp0aMHZ+v6P3r0aLKzs0++HjBgAPv373c3dI/y+E1vEQnBlQhGqWpRM4hd\nDsw77XJUV1VNF5FIYIaIrFXV2YUcfwQwAqB+/frFijEnP7/c3BNoEB7MEwMK75+uqixPO8Ar09fx\n0cItxNetxtAOxftMjKnMhg4dymeffUbfvn1Pln322Wf885//POu+devW5auvvir2e48ePZphw4YR\nFBQEwOTJk4t9rJLm0d+SIuKPK1l8oqrfnKHqEE67HKWq6c6/u4FJQIfCdlTVsaqapKpJJxY4OVe5\neVrmL0m5Q0RoE1OdD2/pQLemtXhy0opTRp0bY9wzePBgfvrpJ44fd90b3Lx5M9u3b6ddu3b06tWL\nxMREWrVqxXffffenfTdv3kxCQgIAR44cYciQIcTFxTFo0CCOHDlyst7IkSNPTov+zDPPADBmzBi2\nb99Oz5496dmzJwCxsbFkZGQA8Oqrr5KQkEBCQgKjR48++X5xcXHccccdtGzZkj59+pzyPiXJk72k\nBHgfWKOqr56hXhhwETCsQFkw4OPc+wgG+gDPFXGI8+YauFc+Whju8PER3r2xPSM/Xsrj36zg00Vb\nebhPMy5qFsHRHFcX4op0vqaCm/IY7FxRsses0wr6v1jk5po1a9KhQwemTJnCwIED+eyzz7j22mup\nWrUqkyZNolq1amRkZNCpUyeuuOKKItfLfvvttwkKCmLNmjUsX76cxMTEk9uef/55atasSV5eHr16\n9WL58uXcf//9vPrqq8yaNYtatWqdcqyUlBTGjx/PokWLUFU6duzIRRddRI0aNVi/fj0TJ07kP//5\nD9deey1ff/01w4YNOz2c8+bJ3xpdgBuBi0VkmfMYICJ3ichdBeoNAqar6uECZbWBuSKSCiwGflLV\nqZ4KNCdfPToOwxuCqvjx3s1JXNO+HivSDzB8/BIu+fdsuv3zF5o8OYV+o2eX+BoDxlQkJy5Lgety\n1NChQ1FVnnjiCVq3bk3v3r1JT09n165dRR5j9uzZJ39xt27dmtatW5/c9sUXX5CYmEi7du1YtWpV\nodOiFzR37lwGDRpEcHAwISEhXHXVVcyZMweAhg0b0rZtW8Cz06d7spfUXOCsv4VV9QPgg9PKfgfa\neCSwQuTm5XtspLc3+fv68PI1bbgmKYZl2/bxzdJ0Mg65mthrd2Zx5Zvz+MtlcVx7QQxVfH2K/CvJ\nGK86Q0vAkwYOHMiDDz7I0qVLyc7Opn379nzwwQfs2bOHlJQU/P39iY2NLXQ687PZtGkTr7zyCkuW\nLKFGjRoMHz68WMc5ISAg4ORzX19fj12Sqni/JYuhvPSSKq4ODWsyontjpo7qzuYXL2Xt3/qR8lRv\nujQJ5y/fraL5U1Np+Phkhr23iIxDx85+QGMqgZCQEHr27Mmtt97K0KFDAdfKeZGRkfj7+zNr1iy2\nbNlyxmN0796dTz/9FICVK1eyfPlywDUtenBwMGFhYezatYspU6ac3Cc0NJSsrKw/Hatbt258++23\nZGdnc/jwYSZNmkS3bt1K6nTdYgkDVy+p0hjpXVYE+vsSHhLA+zdfwL+uaUMHZ+T53A0ZPPRFKrm2\nmqAxgOuyVGpq6smEccMNN5CcnEyrVq2YMGECLVq0OOP+I0eO5NChQ8TFxfH000/Tvr1rRoc2bdrQ\nrl07WrRowfXXX3/KtOgjRoygX79+J296n5CYmMjw4cPp0KEDHTt25Pbbb6ddu3YlfMZnZtObA7N/\n20Pd6lVpEhly9soVkKqyblcWizft5envVgEwuH09mkaGcPBoDt2bRlC3elViagZ5OVJTWdj05p5h\n05uXgO7Nitcdt6IQEVrUqUazyFB+SN3Oks37+Col7eT2N2dtBOBvVyZwZdu6hAb6eytUY4wXWcIw\nJ/n4CBPv6ERuvrJxzyF8RNiSeZhxczezePNe/vLtSv7y7Uoe6dOMJpGh9ImvjU8FGL9ijHGPJQxz\nCj9fH/x8oWXdMADioqrRLyGKozl53DfxV2as3sUr038DoHdcJAPbRtMgPOhPi0cZc75U1XrulaCS\nuP1gCcO4JdDfl//clETmoWNMXbWTzxZvY9GmvcxcsxuApy6N4/ZujbwcpakoAgMDyczMJDw83JJG\nCVBVMjMzCQwMPK/j2E1vU2zHcvP4MXUHD3+ZCsC/rmnD1bbAkykBOTk5pKWlndfYBHOqwMBA6tWr\nh7//qfcgz+WmtyUMc972Zx/nstfnkrbvCCO6N+Lei5uQtvcITWuHVKruysaUR5YwTKnbnXWUK9+Y\nx/YDf/xF2DgimBevbv2nFQaNMWXHuSQM+/PPlIjI0EB+eaQHb92QSEzNqnRvFkH6/iNc++4CJq/Y\n4e3wjDElwG56mxIT6O/LgFZRDGgVBUDGoWNc9+4C7p/4K3FR1WhYK9jLERpjzoe1MIzH1AoJ4K0b\n2pObr/T6138ZMSHZph0xphyzhGE8qnmdUMYPv4B8hemrd9F39Gzem/M7R3PyvB2aMeYcWcIwHtez\nRSTLn+1D1ya12LjnMH//aQ1XvjmPjEPHmP3bHg4fy/V2iMYYN1gvKVOq9mQd44vkbbw8bd3Jsip+\nPowZ0pZ+CVFejMyYyqlM9JISkRgRmSUiq0VklYg8UEidHiJyoMCKfE8X2NZPRNaJyAYRecxTcZrS\nFREawN09GvPPwa3p0TyC2PAgmkaGcNfHS3lz1gYOHs3xdojGmCJ4rIUhIlFAlKouFZFQIAW4UlVX\nF6jTA3hEVS87bV9f4DfgEiANWAIMLbhvYayFUT4dzcljyNiFLNu2nwbhQXx0a0fqh9tU6saUhjLR\nwlDVHaq61HmeBawBot3cvQOwQVV/V9XjwGfAQM9Earwt0N+XT+/oyMC2ddmSmc1Vb89jf/Zxb4dl\njDlNqdz0FpFYoB2wqJDNnUUkVUSmiEhLpywa2FagThpFJBsRGSEiySKSvGfPnhKM2pSmoCp+jL6u\nLe8Ma0/m4eNc++4CvkzexrFc601lTFnh8YQhIiHA18AoVT142ualQANVbQO8Dnx7rsdX1bGqmqSq\nSRERlXshpPJOROiXUId7ejTht12HePSr5TR/airTV+30dmjGGDycMETEH1ey+ERVvzl9u6oeVNVD\nzvPJgL+I1ALSgZgCVes5ZaYSGNW7Kf+5KYknB8RRKySAER+lMPDNeWzYneXt0Iyp1Dw2NYi4JrF/\nH1ijqq8WUacOsEtVVUQ64EpgmcB+oKmINMSVKIYA13sqVlO2+Pn6cEl8bQAGJUZz50cppGzZx5Cx\nC+nVojaNIoIZckF9woJsqVhjSpMne0l1BeYAK4AT80E8AdQHUNV3ROReYCSQCxwBHlLV+c7+A4DR\ngC8wTlWfP9t7Wi+piklV+XppOo84626AaybcGzs1YFBiPcKqWuIwprhsenNT4eTnKx8u2EzbmOqs\n3nGQJyetBKB2tQBeuro1PZpHejdAY8qpc0kYNlutKRd8fIRbujQEICE6jN92ZrFt3xFWpB/g9g+T\n+fKuzrSrX8PLURpTsVkLw5RrB7Jz6P/abLKO5vL4gDia1wmlXUx1fHxsHWhj3FEmBu4ZUxrCgvz5\n/M7ORNeoyhOTVnD12/O55t0FbNub7e3QjKlwLGGYci+mZhDf39uV14a05fqO9fltVxZ3fZxCjq29\nYUyJsoRhKoQqfj4MbBvNC4Na8fLg1qzafpCHv0ilIl1yNcbbLGGYCqdfQhR3dGvI96nbGT1zPcdz\nraVhTEmwhGEqpP/r14JLW0fx2s/rufbdBaxMP2ALNRlznqyXlKnQvliyjce+WU6+QmigH7d3bcSl\nraOICAmwkeLGYAP3vB2GKWNWph/ghclr2LYvm217jwAQGuDHxBGdSIgO83J0xniXJQxjCqGqLNq0\nlyWb9vLe3E3k5uXzSN/mDL8wFtfUZ8ZUPjYOw5hCiAidGoVzX6+mfH9vF1pGh/HXH1bz4OfLbN0N\nY9xgCcNUSg3Cg5l4RyfaxlTn22Xbaf7UVL5OSfN2WMaUaZYwTKXl6yN8NqITD/ZuBsDDX6Zy6Zg5\nHDiS4+XIjCmbLGGYSi3Q35cHejcl9Zk+xEVVY9X2gwx8Yy7b9x/xdmjGlDmWMIwBwqr6M+WBbrx7\nY3t2HDjKqM+XkZdfcTqEGFMSPJYwRCRGRGaJyGoRWSUiDxRS5wYRWS4iK0Rkvoi0KbBts1O+TESs\n65MpFX1b1uHO7o1YvGkvf/luJXuyjnk7JGPKDE+uh5ELPKyqS0UkFEgRkRmqurpAnU3ARaq6T0T6\nA2OBjgW291TVDA/GaMyfPHhJM3YdPMani7Yyb0MGX97VmcjQQG+HZYzXeayFoao7VHWp8zwLWANE\nn1Znvqruc14uBOp5Kh5j3CUivDS4NW9c346dB44y4LW5fLPUelAZUyor7olILNAOWHSGarcBUwq8\nVmC6iCjwrqqO9ViAxhTistZ1aVAzmKe+XcFDX6QycfFWdhw4ysC2dbk2KYYG4cHeDtGYUuXxkd4i\nEgL8D3heVb8pok5P4C2gq6pmOmXRqpouIpHADOA+VZ1dyL4jgBEA9evXb79lyxYPnYmprA4cyeHm\ncYtZvePgKTPf3t2jMY/2bW6jxE25VmamBhERf+BHYJqqvlpEndbAJKC/qv5WRJ1ngUOq+sqZ3s+m\nBjGeoqrk5ivHcvMZN3cTr85wfVWvaV+PpNga9GsZZZMZmnKpTCQMcf3Z9SGwV1VHFVGnPvALcJOq\nzi9QHgz4qGqW83wG8JyqTj3Te1rCMKUlL1+599OlTFm5E4BAfx++GdmF+LrVvByZMeemrMwl1QW4\nEbjY6Rq7TEQGiMhdInKXU+dpIBx467Tus7WBuSKSCiwGfjpbsjCmNPn6CGOGtmP4hbH0aB7B0Zx8\nrn13Abuzjno7NGM8xmarNaYErN15kP6vzUEVujWtxaN9m9O6XnVvh2XMWZ1LC6NUekkZU9G1qFON\nj27tyJSVO5ixehdXvDGPiNAAhl4Qw7DODWwch6kQrIVhTAnbd/g4L01dy1cpaeTmK1X9fXnx6lZc\n0aau9agyZU6ZuOntDZYwTFmiqmzOzGbUZ7+SmnaAgW3r8vRl8YSHBHg7NGNOKis3vY2p1ESEhrWC\n+eKuzozq3ZQfUrdz9dvz2ZqZ7e3QjCkWSxjGeFiAny+jejdjwq0d2Z11jOHjF3PoWK63wzLmnFnC\nMKaUdG1aizFD2rEp8zCD357P0RxbFtaUL5YwjClFveNr89b1iazdmcX/+2o5Fekeoqn4LGEYU8r6\nJdThtq4N+T51Ox8v2urtcIxxmyUMY0qZiPDEgDi6Na3F09+t5EFb3c+UE5YwjPECXx/hX9e0oUWd\nakz6NZ3h4xeTdTTH22EZc0aWMIzxkshqgfx4X1cubhHJnPUZDH57AWN+Xs/7czeRfdx6UZmyxwbu\nGeNlefnKv6av463/bjxZ5u8r3H9xU67vWN8G+hmPspHexpRDGYeOcSw3n62Z2TzyZSrp+48A0Duu\nNi8MSiCyms1HZUqejfQ2phyqFRJAdPWqdG4czsyHLuKmzg2Ij6rGzDW7uG7sQnLz8s9+EGM8yFoY\nxpRxPy7fzr2f/kpYVX9euro1reqFUTcs0CYyNCXCpjc3pgK5tFUUS7vsZ9y8Tdz1cQoAwVV8+e7e\nLjSJDPVydKYy8dglKRGJEZFZIrJaRFaJyAOF1BERGSMiG0RkuYgkFth2s4isdx43eypOY8o6EeHp\ny+P57yM9GH5hLPFR1Th8PI8hYxcxf0OGt8MzlYgn1/SOAqJUdamIhAIpwJWqurpAnQHAfcAAoCPw\nmqp2FJGaQDKQBKizb3tV3Xem97RLUqay2LD7EMPeW8TurKP8/cpWXNo6irCq/t4Oy5RDZeKmt6ru\nUNWlzvMsYA0QfVq1gcAEdVkIVHcSTV9ghqrudZLEDKCfp2I1prxpEhnClAe6Ua9GEE9MWsGgN+ex\nPG2/t8MyFVyp9JISkVigHbDotE3RwLYCr9OcsqLKjTGOGsFVeOuGRFrXC+P3jMNc8cY8Or4wk6kr\nd3g7NFNBuZUwRKSxiAQ4z3uIyP0i4tYK9yISAnwNjFLVg8UPtcjjjxCRZBFJ3rNnT0kf3pgyLSE6\njO/v7cq8xy7mnp6NyT6Wx10fL+Whz5dxPNe64ZqS5W4L42sgT0SaAGOBGODTs+0kIv7Ovp+o6jeF\nVEl3jnVCPaesqPI/UdWxqpqkqkkRERHunIsxFU509ao82rcF397bBYBvfk3n7k+W2hQjpkS5mzDy\nVTUXGAS8rqqPAlFn2kFcncTfB9ao6qtFVPseuMnpLdUJOKCqO4BpQB8RqSEiNYA+Tpkx5gwaR4Sw\n+cVLua1rQ2au2cVVb81n7+Hj3g7LVBDujsPIEZGhwM3A5U7Z2bpkdAFuBFaIyDKn7AmgPoCqvgNM\nxtVDagOQDdzibNsrIn8Dljj7Paeqe92M1ZhK7y+XxVMrJICXpq4l8W8zuKVLLKN6NSMsyHpSmeJz\nq1utiMQDdwELVHWiiDQErlXVlzwd4LmwbrXGnGrEhGSmr94FQP2aQXx0WwcahAd7OSpTlnh08kHn\nElGMqi4vTnCeZAnDmFMdzckjefM+fATu+XQp+7JzaF0vjBcGtSIhOszb4ZkyoMTHYYjIf0WkmjOg\nbinwHxEp6r6EMaaMCPT3pWvTWlzYpBZf3NmZEd0bsXVvNle8MZef1+zydnimnHH3pneY0yX2KlwD\n7ToCvT0XljGmpDWtHcoTA+KY/mB3mkSGcOdHKYydvZGKNAGp8Sx3E4afMwL7WuBHD8ZjjPGwyNBA\n3r/5AkIC/Xhh8loaPj6Zd/+38ew7mkrP3YTxHK5urRtVdYmINALWey4sY4wnxdQMYtETvbita0MA\n/jFlLc9+v4oDR2xdcVM0Ww/DmEruWG4ez/2wmk8WbaVPfG3G3uTW/U9TQXjipnc9EZkkIrudx9ci\nUu/8wjTGlAUBfr48P6gVD1/SjOmrd/HenN+9HZIpo9y9JDUe16jsus7jB6fMGFNBjLioEb3jIvn7\nT2tI2XLGlQRMJeVuwohQ1fGqmus8PgBs4iZjKpAAP19eG9KO0EA/Rs/8jfz8inO52pQMdxNGpogM\nExFf5zEMyPRkYMaY0hcc4McjfZozZ30Gz/24miPH87wdkilD3J1L6lbgdeDfuFbAmw8M91BMxhgv\nuqlzAzZlHOaD+Zv5fMk2ujatRZ/42vSKq42qEh4S4O0QjZcUu5eUiIxS1dElHM95sV5SxpQMVWXx\npr2M+WU98zb8cTEhJMCPpy+Lp3FkCIn1q+OalNqUZx6dS6rAm2xV1frF2tlDLGEYU/K27z/CrHW7\nWZl+gJlrdrMn6xgAA9vW5fH+cdQJC/RyhOZ8lFbC2KaqMWevWXosYRjjWdnHc5m7PoO3/ruRZdtc\na4h3aRLOR7d2xMfHWhvlkbUwjDEepaos2JjJyE+WcuBIDoH+PlwSX4d/XdOGKn7u9qUxZUGJDdwT\nkSwROVjIIwvXeAxjTCUkIlzYpBbLnr6ER/s2RxB+SN3O8PGLbTLDCuyMCUNVQ1W1WiGPUFU9Yw8r\nERnnjApfWcT2R0VkmfNYKSJ5zvTpiMhmEVnhbLMmgzFllIhwT88mLHvmEsKDqzB/YyavTF9nSaOC\n8mTb8QOgX1EbVfVlVW2rqm2Bx4H/nbYMa09nu01sY0wZF+Dny/zHL6Z2tQDenLWRd2f/bkmjAvJY\nwlDV2YC763APBSZ6KhZjjOcF+Pnyzd1d6BBbkxenrCX+6Wn8vueQt8MyJcjrd6dEJAhXS+TrAsUK\nTBeRFBEZcZb9R4hIsogk79mzx5OhGmPOIrp6VT66vQNDLojhSE4ew8cvYXnafvJsmpEKwaPTm4tI\nLPCjqiacoc51wDBVvbxAWbSqpotIJDADuM9psZyR9ZIypuyYtW43d05I4XhePv6+wjVJMXSIrcnl\nberia11wy4wSn97cw4Zw2uUoVU13/t0NTAI6eCEuY8x56Nk8kkn3XMjQDvUJ9PPl00VbGfX5Mh77\nerm1OMopryYMEQkDLgK+K1CBrQEUAAATL0lEQVQWLCKhJ54DfYBCe1oZY8q2lnXD+MdVrVj+bB/e\nviGRwe3r8WVKGle9Nc8mNiyH3J188JyJyESgB1BLRNKAZwB/AFV9x6k2CJiuqocL7FobmOTMUeMH\nfKqqUz0VpzHG80SE/q2i6JdQh8T6NXhi0goue30Or1zThrYxNidVeWFLtBpjSt33qdt57odVZBw6\nTq8Wkbx3c5IlDS8pb/cwjDGVzBVt6vLNyC40qx3Cz2t3M3dDhrdDMm6whGGM8Yr64UF8d09XGoQH\nceP7i7ll/GKyjuZ4OyxzBpYwjDFeU7WKL29en4i/rzBr3R7+9uNq60FVhlnCMMZ4VUJ0GOufH8Cd\n3RvxRXIaTZ+czKeLtno7LFMISxjGmDLhsf4teLRvc/IVnpi0gjd+We/tkMxpLGEYY8qEEzPfrn6u\nL5e2iuKV6b8x+O35HM/N93ZoxmEJwxhTpgRV8eO1IW0Z1C6a5C37eOTLVHLzLGmUBR4buGeMMcXl\n5+vDv69rS1hVfz6Yv5lV2w/w7o3taRIZ6u3QKjVLGMaYMuvpy+KpFujHmF820G/0HJpEhnB5m7rc\n3q0hAX6+3g6v0rFLUsaYMsvHR3ioT3PmP3YxN3WOZfv+I7w8bR1Dxy7k0LFcb4dX6djUIMaYcuNA\ndg5TVu7giUkryFd4tG9z7u7R2KYVOQ/nMjWIXZIyxpQbYUH+DOlQn7Cq/rw8fR0vT1tH2r5snhuY\ngL+vXTDxNPuEjTHlTv9WUfz80EXc0a0hExdv444Jyew9fNzbYVV41sIwxpRLIsKTl8ZTv2YQf/lu\nFYl/m0Gz2iF0ahTOg72bUSO4irdDrHAsYRhjyrUbO8cSERrI1JU7mLM+gwkLtvDRwi30ahHJg5c0\no2XdMG+HWGF4cgGlccBlwO7C1vQWkR64Vtrb5BR9o6rPOdv6Aa8BvsB7qvqip+I0xpR//RLq0C+h\nDgC/bt3HF8nbmLh4GzPX7CYuqhpjhrSlaW0bw3G+PHkP4wOg31nqzFHVts7jRLLwBd4E+gPxwFAR\nifdgnMaYCqRd/Rr846rWjL/lAoZfGMuG3Vlc/fZ8ftuV5e3Qyj2PJQxVnQ3sLcauHYANqvq7qh4H\nPgMGlmhwxpgKr2fzSJ69oiWT7u6Cn68PA9+Yx8zVu7wdVrnm7V5SnUUkVUSmiEhLpywa2FagTppT\nZowx5ywhOozJ93ejWe0Q7vgomXs+XUrGoWPeDqtc8mbCWAo0UNU2wOvAt8U5iIiMEJFkEUnes2dP\niQZojKkY6oQF8ukdnRh+YSxTV+7ksjFzLWkUg9cShqoeVNVDzvPJgL+I1ALSgZgCVes5ZUUdZ6yq\nJqlqUkREhEdjNsaUX8EBfjxzeUvevD6RjEPHuHncYvZn29iNc+G1hCEidcQZzy8iHZxYMoElQFMR\naSgiVYAhwPfeitMYU7H0S6jD84MSWLX9IO3/PpP+r83hg3mbqEjTJHmKJ7vVTgR6ALVEJA14BvAH\nUNV3gMHASBHJBY4AQ9T1E8sVkXuBabi61Y5T1VWeitMYU/lcd0F9mtYO5d3/bWTb3iM8+8NqlmzZ\nx5Vto4mLCqVejSBvh1gm2eSDxphKLT9feWHyGt6b6xoSVj3In09v70R83Wpejqx0nMvkg97uJWWM\nMV7l4yM8eWkcE+/oxEOXNEOAAWPmcO27C0jbl+3t8MoUa2EYY0wBCzZm8uSkFfyecRg/H6FvQh2u\nbBtN77jICjmN+rm0MCxhGGNMIf67bjcfzt/MrHWu7voBfj68c2N7ejaP9HJkJcsShjHGlJAVaQeY\nuWYXY35ZjyoMahfNdRfE0KlRuLdDKxGWMIwxpoRlHDrGS1PW8mVKGj4C/xzchivb1sWvnC/cZAnD\nGGM85ODRHC5/fS5bMrOJCA2gR7MIBrevR8dy2uKwXlLGGOMh1QL9mfJAN566NI7MQ8f4MiWN68Yu\n5IN5mziak8eG3Ycq7CBAa2EYY0wx7c8+TvLmffxnzu8s2vTH5NxXJUbzr2valIteVefSwrAV94wx\nppiqB1Whd3xtujatxZcpaSzZtJfvU7fzzdJ0dh44yrjhFxDo7+vtMEuMXZIyxpjzFOjvy42dGjBm\naDtWP9eX/gl1mL8xk/fm/O7t0EqUJQxjjClBQVX8eHtYe/rE1+bNWRtZ9Humt0MqMZYwjDHGA54Y\nEEdooB/XjV1I+7/N4KflO9i2N5u8/PJ739huehtjjIccyM7h9V/WM3dDBmt3utYUv6hZBO/e2L7M\n3NuwcRjGGFOGHM/N54XJa/htVxYLfs9EFR7o1ZRRvZt6vSeV9ZIyxpgypIqfD89e0RKAGat38fAX\ny3jt5/VszjxMs9qh9G1Zh8YRwV5PHmdjLQxjjCllR3PyuPuTpfyydvfJsqiwQD6+vSONI0JKNZYy\nMdJbRMaJyG4RWVnE9htEZLmIrBCR+SLSpsC2zU75MhGxDGCMqVAC/X15/+YkRl/XlktbR3Fxi0h2\nHDjKpWPm8O2v6Rw5nuftEAvlsRaGiHQHDgETVDWhkO0XAmtUdZ+I9AeeVdWOzrbNQJKqZpzLe1oL\nwxhTXn2ZvI3Hv1lBrtOL6v/6tWBkj8Yef98y0cJQ1dnA3jNsn6+q+5yXC4F6norFGGPKumuSYkh5\n6hLeviERgJemruXFKWu9HNWpyso4jNuAKQVeKzBdRFJEZISXYjLGmFIVFuRP/1ZRrPt7P3q1iOSd\n/23kklf/x+iZv5WJ8RsevektIrHAj4VdkipQpyfwFtBVVTOdsmhVTReRSGAGcJ/TYils/xHACID6\n9eu337JlS8mehDHGeEFOXj4fLdjCj8u3s3Trfvx9ha5NatE7vjadG4XTqIRujpeZcRhnSxgi0hqY\nBPRX1d+KqPMscEhVXznb+9k9DGNMRaOqvP7LBl6d8cevyOAqvtzYOZZ7L25CoJ/PeS3iVC4ShojU\nB34BblLV+QXKgwEfVc1yns8AnlPVqWd7P0sYxpiKan/2cfx9fZi8YgdfL01j4e9/3CL+euSFtG9Q\no1jHLRM3vUVkIrAAaC4iaSJym4jcJSJ3OVWeBsKBt07rPlsbmCsiqcBi4Cd3koUxxlRk1YOqEBzg\nxzVJMUy8oxP39myCr49roN+3v6aXSgw2cM8YY8opVWXNjixa1AnFx6d4o8RtahBjjKkERIT4utVK\n7f3KSrdaY4wxZZwlDGOMMW6xhGGMMcYtljCMMca4xRKGMcYYt1jCMMYY4xZLGMYYY9xiCcMYY4xb\nLGEYY4xxiyUMY4wxbrGEYYwxxi2WMIwxxrjFEoYxxhi3WMIwxhjjFksYxhhj3OLRhCEi40Rkt4is\nLGK7iMgYEdkgIstFJLHAtptFZL3zuNmTcRpjjDk7T7cwPgD6nWF7f6Cp8xgBvA0gIjWBZ4COQAfg\nGREp3oK1xhhjSoRHE4aqzgb2nqHKQGCCuiwEqotIFNAXmKGqe1V1HzCDMyceY4wxHubtexjRwLYC\nr9OcsqLK/0RERohIsogk79mzx2OBGmNMZefthHHeVHWsqiapalJERIS3wzHGmArL2wkjHYgp8Lqe\nU1ZUuTHGGC/xdsL4HrjJ6S3VCTigqjuAaUAfEanh3Ozu45QZY4zxEj9PHlxEJgI9gFoikoar55M/\ngKq+A0wGBgAbgGzgFmfbXhH5G7DEOdRzqnqmm+fGGGM8zKMJQ1WHnmW7AvcUsW0cMM4TcRljjDl3\n3r4kZYwxppywhGGMMcYtljCMMca4xRKGMcYYt1jCMMYY4xZLGMYYY9xiCcMYY4xbLGEYY4xxiyUM\nY4wxbrGEYYwxxi2WMIwxxrjFEoYxxhi3WMIwxhjjFksYxhhj3CKuGcYrBhHZA2wp5u61gIwSDKc8\nss/APgOwzwAq12fQQFXdWt+6QiWM8yEiyaqa5O04vMk+A/sMwD4DsM+gKHZJyhhjjFssYRhjjHGL\nJYw/jPV2AGWAfQb2GYB9BmCfQaHsHoYxxhi3WAvDGGOMWyp9whCRfiKyTkQ2iMhj3o7HU0QkRkRm\nichqEVklIg845TVFZIaIrHf+reGUi4iMcT6X5SKS6N0zKDki4isiv4rIj87rhiKyyDnXz0WkilMe\n4Lze4GyP9WbcJUVEqovIVyKyVkTWiEjnyvY9EJEHnf8HK0VkoogEVrbvQXFU6oQhIr7Am0B/IB4Y\nKiLx3o3KY3KBh1U1HugE3OOc62PAz6raFPjZeQ2uz6Sp8xgBvF36IXvMA8CaAq9fAv6tqk2AfcBt\nTvltwD6n/N9OvYrgNWCqqrYA2uD6LCrN90BEooH7gSRVTQB8gSFUvu/BuVPVSvsAOgPTCrx+HHjc\n23GV0rl/B1wCrAOinLIoYJ3z/F1gaIH6J+uV5wdQD9cvxIuBHwHBNUDL7/TvBDAN6Ow893PqibfP\n4TzPPwzYdPp5VKbvARANbANqOj/XH4G+lel7UNxHpW5h8McX54Q0p6xCc5rU7YBFQG1V3eFs2gnU\ndp5X1M9mNPD/gHzndTiwX1VzndcFz/PkZ+BsP+DUL88aAnuA8c5lufdEJJhK9D1Q1XTgFWArsAPX\nzzWFyvU9KJbKnjAqHREJAb4GRqnqwYLb1PUnVIXtNicilwG7VTXF27F4kR+QCLytqu2Aw/xx+Qmo\nFN+DGsBAXMmzLhAM9PNqUOVEZU8Y6UBMgdf1nLIKSUT8cSWLT1T1G6d4l4hEOdujgN1OeUX8bLoA\nV4jIZuAzXJelXgOqi4ifU6fgeZ78DJztYUBmaQbsAWlAmqoucl5/hSuBVKbvQW9gk6ruUdUc4Btc\n343K9D0olsqeMJYATZ3eEVVw3fj63ssxeYSICPA+sEZVXy2w6XvgZuf5zbjubZwov8npJdMJOFDg\nkkW5pKqPq2o9VY3F9bP+RVVvAGYBg51qp38GJz6bwU79cv2Xt6ruBLaJSHOnqBewmkr0PcB1KaqT\niAQ5/y9OfAaV5ntQbN6+ieLtBzAA+A3YCDzp7Xg8eJ5dcV1mWA4scx4DcF2L/RlYD8wEajr1BVcP\nso3AClw9Srx+HiX4efQAfnSeNwIWAxuAL4EApzzQeb3B2d7I23GX0Lm3BZKd78K3QI3K9j0A/gqs\nBVYCHwEBle17UJyHjfQ2xhjjlsp+ScoYY4ybLGEYY4xxiyUMY4wxbrGEYYwxxi2WMIwxxrjFEoYp\nV0QkT0SWiUiqiCwVkQvPUr+6iNztxnH/KyK2hnMBIvKBiAw+e01TWVjCMOXNEVVtq6ptcE0W+Y+z\n1K8OnDVheEuBkcXGlHmWMEx5Vg3XNNSISIiI/Oy0OlaIyECnzotAY6dV8rJT9/+cOqki8mKB410j\nIotF5DcR6ebU9RWRl0VkibMexJ1OeZSIzHaOu/JE/YJEZLOI/NN5r8Ui0sQp/0BE3hGRRcA/nbUo\nvnWOv1BEWhc4p/HO/stF5GqnvI+ILHDO9UtnfjBE5EVxrXeyXEReccquceJLFZHZZzknEZE3xLU+\nzEwgsiR/WKb8s79uTHlTVUSW4Rp9G4VrPiiAo8AgVT0oIrWAhSLyPa6J9RJUtS2AiPTHNfFcR1XN\nFpGaBY7tp6odRGQA8AyuOYduwzUdxgUiEgDME5HpwFW4pr9+3llXJaiIeA+oaisRuQnXTLmXOeX1\ngAtVNU9EXgd+VdUrReRiYAKu0dh/ObG/E3sN59yeAnqr6mER+T/gIRF5ExgEtFBVFZHqzvs8DfRV\n1fQCZUWdUzugOa61YWrjmi5jnFs/FVMpWMIw5c2RAr/8OwMTRCQB1xQWL4hId1xTl0fzxxTdBfUG\nxqtqNoCq7i2w7cSEjClArPO8D9C6wLX8MFyLCS0BxolrQsdvVXVZEfFOLPDvvwuUf6mqec7zrsDV\nTjy/iEi4iFRzYh1yYgdV3SeuGXfjcf2SB6gCLMA15fZR4H1xrST4o7PbPOADEfmiwPkVdU7dgYlO\nXNtF5JcizslUUpYwTLmlqgucv7gjcM2LFQG0V9Uccc1IG3iOhzzm/JvHH/83BLhPVaedXtlJTpfi\n+oX8qqpOKCzMIp4fPsfYTr4tMENVhxYSTwdcE+kNBu4FLlbVu0SkoxNnioi0L+qcnJaVMUWyexim\n3BKRFriW18zE9VfybidZ9AQaONWygNACu80AbhGRIOcYBS9JFWYaMNJpSSAizUQkWEQaALtU9T/A\ne7imCC/MdQX+XVBEnTnADc7xewAZ6lqrZAZwT4HzrQEsBLoUuB8S7MQUAoSp6mTgQVxLryIijVV1\nkao+jWvhpJiizgmYDVzn3OOIAnqe5bMxlYy1MEx5c+IeBrj+Ur7ZuQ/wCfCDiKzANRPrWgBVzRSR\neSKyEpiiqo+KSFsgWUSOA5OBJ87wfu/hujy1VFzXgPYAV+Ka7fZREckBDgE3FbF/DRFZjqv18qdW\ngeNZXJe3lgPZ/DGV9t+BN53Y84C/quo3IjIcmOjcfwDXPY0s4DsRCXQ+l4ecbS+LSFOn7GcgFdcs\ntYWd0yRc94RW45oCvKgEZyopm63WGA9xLoslqWqGt2MxpiTYJSljjDFusRaGMcYYt1gLwxhjjFss\nYRhjjHGLJQxjjDFusYRhjDHGLZYwjDHGuMUShjHGGLf8f5VXwMCOmq2oAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rb5n21yHhHiM",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ug7apL3Tg-e4",
        "colab_type": "text"
      },
      "source": [
        "## Adam for comparison purposes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.131076Z",
          "start_time": "2019-09-02T14:18:45.015Z"
        },
        "id": "rGZweQ31WQuM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn = cnn_learner(data, models.resnet18, metrics=[accuracy])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.132459Z",
          "start_time": "2019-09-02T14:18:45.017Z"
        },
        "id": "bcEJGdaNWQuR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learn.unfreeze()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.133861Z",
          "start_time": "2019-09-02T14:18:45.019Z"
        },
        "id": "8Z-huluiWQuV",
        "colab_type": "code",
        "outputId": "bcf6c5a2-d39e-4c75-8d7a-24c4fcb7d0d7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "source": [
        "learn.fit(1, [1e-5, 1e-4, 1e-3])"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>0.093938</td>\n",
              "      <td>0.051388</td>\n",
              "      <td>0.984900</td>\n",
              "      <td>01:09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-09-02T14:19:11.135059Z",
          "start_time": "2019-09-02T14:18:45.022Z"
        },
        "id": "IyMRiIhvWQua",
        "colab_type": "code",
        "outputId": "19eb6ef6-edc2-430e-96f4-44f971a2223e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        }
      },
      "source": [
        "learn.recorder.plot_losses()"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XWW97/HPL8nO3kl25qlpOiQd\noPOQhjILBY4CCqVQkR44CopVjl48Duce9HpFeemRo1xERfGgUg5HhCMgiAwiapFJhra2pQNtQ+mQ\nps3UZp6T5/6xV0No0zZts7OS7O/79dqv7r32Wmv/1spuvnnW8DzmnENERAQgzu8CRERk+FAoiIhI\nL4WCiIj0UiiIiEgvhYKIiPRSKIiISC+FgoiI9FIoiIhIL4WCiIj0SvC7gOOVk5PjioqK/C5DRGRE\nWb16dY1zLvdY8424UCgqKmLVqlV+lyEiMqKY2c6BzKfDRyIi0kuhICIivRQKIiLSa8SdUxCR0aOz\ns5Py8nLa2tr8LmXUCIVCjBs3jkAgcELLKxRExDfl5eWkpqZSVFSEmfldzojnnKO2tpby8nKKi4tP\naB06fCQivmlrayM7O1uBMEjMjOzs7JNqeSkURMRXCoTBdbL7M6ZCYcOeel7bXut3GSIiw1bMhIJz\njo/8+GWuufc1v0sRkWGitraWefPmMW/ePMaMGUNhYWHv646OjgGt44YbbmDLli1RrnToxMyJ5oa2\nLr9LEJFhJjs7m7Vr1wLwzW9+k3A4zFe+8pX3zeOcwzlHXFz/f0OvWLEi6nUOpZhpKVQ16JI3ERmY\nsrIyZsyYwbXXXsvMmTPZu3cvy5cvp7S0lJkzZ3Lbbbf1znvOOeewdu1aurq6yMjI4JZbbmHu3Lmc\neeaZVFVV+bgVJyZmWgqVDe1+lyAiR/Gt329kU0XDoK5zxtg0br1s5gkt+/bbb/PAAw9QWloKwO23\n305WVhZdXV0sWrSIpUuXMmPGjPctU19fz3nnncftt9/Ol770Je677z5uueWWk96OoRQzLYXGtk6/\nSxCREWTy5Mm9gQDw0EMPUVJSQklJCZs3b2bTpk2HLZOUlMQll1wCwIIFC9ixY8dQlTtoYqalcMns\nAj5z3iRWvLzD71JEpB8n+hd9tKSkpPQ+37ZtGz/84Q954403yMjI4Lrrruv3XoDExMTe5/Hx8XR1\njbxzmTHTUgAIJcTT0d2Dc87vUkRkBGloaCA1NZW0tDT27t3Lc88953dJURMzLQWAxIRIBrZ39RAK\nxPtcjYiMFCUlJcyYMYNp06YxceJEzj77bL9LihobaX81l5aWuhMdZOcXL23n209vZv03P0ha6MQ6\nixKRwbN582amT5/udxmjTn/71cxWO+dKj7BIr5g6fBQ82FLo7PG5EhGR4SlqoWBm481spZltMrON\nZvaFfuY538zqzWyt9/hGtOoBCCZEDhl1dCsURET6E81zCl3Al51za8wsFVhtZs875w69jusl59xH\nolhHr6TESCg0t4+8KwJERIZC1FoKzrm9zrk13vNGYDNQGK3PG4jscORysdqmgfVpIiISa4bknIKZ\nFQHzgdf7eftMM1tnZs+aWVQvVM4JBwGobdbdzSIi/Yn6JalmFgYeA/7FOXfoPexrgInOuSYzuxR4\nApjazzqWA8sBJkyYcMK1ZKdEWgo1jQoFEZH+RLWlYGYBIoHwoHPut4e+75xrcM41ec+fAQJmltPP\nfPc650qdc6W5ubknXE9mciJxBrXNOnwkIrBo0aLDbkS76667uOmmm464TDgcBqCiooKlS5f2O8/5\n55/PsS6dv+uuu2hpael9femll1JXVzfQ0qMmmlcfGfBLYLNz7s4jzDPGmw8zW+jVE7VRcOLijKyU\nIDVNaimICCxbtoyHH374fdMefvhhli1bdsxlx44dy6OPPnrCn31oKDzzzDNkZGSc8PoGSzRbCmcD\n/wRc0OeS00vN7LNm9llvnqXABjNbB/wIuMZF+W66nHAiNTrRLCLA0qVLefrpp3sH1NmxYwcVFRXM\nnz+fCy+8kJKSEmbPns3vfve7w5bdsWMHs2bNAqC1tZVrrrmG6dOns2TJElpbW3vnu+mmm3q73L71\n1lsB+NGPfkRFRQWLFi1i0aJFABQVFVFTUwPAnXfeyaxZs5g1axZ33XVX7+dNnz6dT3/608ycOZMP\nfvCD7/ucwRK1cwrOuZeBow4W6py7G7g7WjX0JyesloLIsPTsLbDvrcFd55jZcMntR3w7KyuLhQsX\n8uyzz7J48WIefvhhrr76apKSknj88cdJS0ujpqaGM844g8svv/yI4x/fc889JCcns3nzZtavX09J\nSUnve9/5znfIysqiu7ubCy+8kPXr13PzzTdz5513snLlSnJy3n/EfPXq1axYsYLXX38d5xynn346\n5513HpmZmWzbto2HHnqIn//851x99dU89thjXHfddYOzrzwxdUczQF5qkCqNrSAinr6HkA4eOnLO\n8bWvfY05c+Zw0UUXsWfPHiorK4+4jhdffLH3l/OcOXOYM2dO73u/+c1vKCkpYf78+WzcuLHfLrf7\nevnll1myZAkpKSmEw2GuvPJKXnrpJQCKi4uZN28eEL2uuWOqQzyA/PQQVY1t9PQ44uKO2pARkaF0\nlL/oo2nx4sV88YtfZM2aNbS0tLBgwQLuv/9+qqurWb16NYFAgKKion67yj6Wd999lzvuuIM333yT\nzMxMrr/++hNaz0HBYLD3eXx8fFQOH8VcSyE/NUhnt2N/i84riEjkaqJFixbxyU9+svcEc319PXl5\neQQCAVauXMnOnTuPuo4PfOAD/PrXvwZgw4YNrF+/Hoh0uZ2SkkJ6ejqVlZU8++yzvcukpqbS2Nh4\n2LrOPfdcnnjiCVpaWmhububxxx/n3HPPHazNPaaYaymMSQ8BUNnQ1nszm4jEtmXLlrFkyZLew0jX\nXnstl112GbNnz6a0tJRp06YddfmbbrqJG264genTpzN9+nQWLFgAwNy5c5k/fz7Tpk1j/Pjx7+ty\ne/ny5Vx88cWMHTuWlStX9k4vKSnh+uuvZ+HChQDceOONzJ8/f8hGcYuprrMB1uw6wJU/fZX7ri/l\ngmn5g1iZiBwvdZ0dHeo6+ziMSYu0FPbV62SziMihYi4UclODmMG+hhM/2SMiMlrFXCgE4uPIDQfZ\nVz/4Z+1F5PiNtEPYw93J7s+YCwWAgvQQe+vVUhDxWygUora2VsEwSJxz1NbWEgqFTngdMXf1EUTu\natbhIxH/jRs3jvLycqqrq/0uZdQIhUKMGzfuhJePyVBITw7w9r7Drw8WkaEVCAQoLi72uwzpIyYP\nH6UnBahv7fS7DBGRYScmQyEjKZGm9i46u3v8LkVEZFiJzVBIDgCotSAicgiFgoiI9IrJUEhPioRC\nXYtCQUSkr5gOhfpW9ZQqItJXTIZCRnIioJaCiMihYjIUclODxMcZ26qa/C5FRGRYiclQCAcTmDsu\nndU7DvhdiojIsBKToQCRwXY0+pqIyPvFbChkJidyoFmhICLSV8yGQlZKIgdaOujpUe+MIiIHxWwo\nZCYn0uOgsa3L71JERIaNmA2FrJTIZak6ryAi8p6YDYXMg6Gg8woiIr1iNhSyvBvYdLJZROQ9sRsK\n4UgoVDZqBDYRkYNiNhTGpofITA6wdled36WIiAwbUQsFMxtvZivNbJOZbTSzL/Qzj5nZj8yszMzW\nm1lJtOrp57OZVZjO1koNyykiclA0WwpdwJedczOAM4DPmdmMQ+a5BJjqPZYD90SxnsMUZaewo7Zl\nKD9SRGRYi1ooOOf2OufWeM8bgc1A4SGzLQYecBGvARlmVhCtmg41ISuZ+tZODbYjIuIZknMKZlYE\nzAdeP+StQmB3n9flHB4cUZOfHgKgskEnm0VEYAhCwczCwGPAvzjnGk5wHcvNbJWZraqurh602sak\nKRRERPqKaiiYWYBIIDzonPttP7PsAcb3eT3Om/Y+zrl7nXOlzrnS3NzcQavvYCjsq1coiIhAdK8+\nMuCXwGbn3J1HmO1J4OPeVUhnAPXOub3RqulQeWlBQC0FEZGDEqK47rOBfwLeMrO13rSvARMAnHM/\nA54BLgXKgBbghijWc5hQIJ6M5ACVDe1D+bEiIsNW1ELBOfcyYMeYxwGfi1YNA1GQnkSZhuUUEQFi\n+I7mgxadmstr79bS0qEutEVEYj4Uphek4Rzs2q+b2EREYj4UinNSANhRo1AQEYn5UCjMSAKgoq7V\n50pERPwX86GQkRwgFIhTKIiIoFDAzBibnsRe3cAmIqJQACjICFFRr5aCiIhCgci9Cjp8JCKiUAAi\nJ5urGttp6+z2uxQREV8pFIhcluoc7Na9CiIS4xQKwKTcyL0K22uafa5ERMRfCgXeu4Fte7VCQURi\nm0IBSA0FyE0Nsr1aHeOJSGxTKHim5IYpUyiISIxTKHim5ocpq2oi0pu3iEhsUih4puaFaWzroqpR\nA+6ISOxSKHgm54UB2FapQ0giErsUCp6peakAbKtq9LkSERH/KBQ8OeFEMpIDbNPQnCISwxQKHjNj\nal6YMh0+EpEYplDoY0peKlurGnUFkojELIVCH1PzwtS1dFLb3OF3KSIivlAo9DE1X1cgiUhsUyj0\ncfAKpDJdgSQiMUqh0Ed+WpDUYIKuQBKRmKVQ6MPMmJIf1uEjEYlZCoVDTM0Lq6UgIjFLoXCIqXmp\n1DS1c0BXIIlIDIpaKJjZfWZWZWYbjvD++WZWb2Zrvcc3olXL8ZjiXYGkbrRFJBZFs6VwP3DxMeZ5\nyTk3z3vcFsVaBmyqOsYTkRgWtVBwzr0I7I/W+qNlbHoSyYnx6hhPRGKS3+cUzjSzdWb2rJnN9LkW\nAOLijCl5kQF3RERijZ+hsAaY6JybC/wYeOJIM5rZcjNbZWarqquro17YlLwwb+9TH0giEnsGFApm\nNtnMgt7z883sZjPLOJkPds41OOeavOfPAAEzyznCvPc650qdc6W5ubkn87EDMn98BtWN7ZQfaI36\nZ4mIDCcDbSk8BnSb2RTgXmA88OuT+WAzG2Nm5j1f6NVSezLrHCwzC9MB2Fqp8woiElsSBjhfj3Ou\ny8yWAD92zv3YzP5+tAXM7CHgfCDHzMqBW4EAgHPuZ8BS4CYz6wJagWvcMDleMyknBYB3a5p9rkRE\nZGgNNBQ6zWwZ8AngMm9a4GgLOOeWHeP9u4G7B/j5QyojOZHc1CCbKhr8LkVEZEgN9PDRDcCZwHec\nc++aWTHw39Ery3/zx2ewZtcBv8sQERlSAwoF59wm59zNzrmHzCwTSHXO/UeUa/PV/AmZ7Khtobap\n3e9SRESGzECvPnrBzNLMLIvIpaQ/N7M7o1uav+ZPiFxctb683udKRESGzkAPH6U75xqAK4EHnHOn\nAxdFryz/zS5MJ85g7e46v0sRERkyAw2FBDMrAK4GnopiPcNGSjCBqXmprC9XKIhI7BhoKNwGPAe8\n45x708wmAduiV9bwMHd8OuvK63Vns4jEjIGeaH7EOTfHOXeT93q7c+6q6JbmvznjMtjf3MGu/S1+\nlyIiMiQGeqJ5nJk97o2PUGVmj5nZuGgX57czJmUB8ErZsLjRWkQk6gZ6+GgF8CQw1nv83ps2qk3O\nDZOZHGDtbt2vICKxYaChkOucW+Gc6/Ie9wPR75nOZ2bGzLHpbNqrO5tFJDYMNBRqzew6M4v3Htcx\nTDqvi7aZY9PYuq+Jzu4ev0sREYm6gYbCJ4lcjroP2EukM7vro1TTsDJjbBod3T0anlNEYsJArz7a\n6Zy73DmX65zLc85dAYz6q48AZo6NdKO9sUJ3NovI6HcyI699adCqGMaKc1JICsSzUT2mikgMOJlQ\nsEGrYhiLjzOmF6SqG20RiQknEwoxc5vvjLFpbNrbQE9PzGyyiMSoo4aCmTWaWUM/j0Yi9yvEhJlj\n02lq79KdzSIy6h115DXnXOpQFTKczRybBsDGigaKvKE6RURGo5M5fBQzTh2TSjAhjlU79/tdiohI\nVCkUBiCYEM/C4ixe3lbjdykiIlGlUBigc6bksK2qiX31bX6XIiISNQqFATpnag4AL2yp8rkSEZHo\nUSgM0IyCNCbnpvDwm7v9LkVEJGoUCgNkZlx3xkTW7q7jrXJ1eSEio5NC4ThcWTKOpEA8v3ptp9+l\niIhEhULhOKQnBbhi/lh+t24PLR1dfpcjIjLoFArH6UMzx9DW2cOqHRqNTURGH4XCcTqtKIuEOOPV\nd2JijCERiTEKheOUEkxg3vgM/rZdoSAio0/UQsHM7jOzKjPbcIT3zcx+ZGZlZrbezEqiVctgO2tK\nDm+V11HVqBvZRGR0iWZL4X7g4qO8fwkw1XssB+6JYi2D6vK5BfQ4eHJthd+liIgMqqiFgnPuReBo\nPcgtBh5wEa8BGWZWEK16BtOUvFTmjkvn0dXlfpciIjKo/DynUAj0vT243Js2Ily1YBxv72vU2M0i\nMqqMiBPNZrbczFaZ2arq6mq/ywHgsjljCcQbj63e43cpIiKDxs9Q2AOM7/N6nDftMM65e51zpc65\n0tzc3CEp7lgyUxK5cFo+T67bQ2d3j9/liIgMCj9D4Ung495VSGcA9c65vT7Wc9yuWjCOmqYOXtw6\nPFovIiIn66jDcZ4MM3sIOB/IMbNy4FYgAOCc+xnwDHApUAa0ADdEq5ZoOf/UXLJTEnlsTTkXTs/3\nuxwRkZMWtVBwzi07xvsO+Fy0Pn8oBOLjuHzeWB58bRd1LR1kJCf6XZKIyEkZESeah7OrSsbR0d3D\nk+t0z4KIjHwKhZM0c2wa88Zn8LMX3iHS+BERGbkUCifp4OA7FfVtrNlV53c5IiInRaEwCD40M5+0\nUAL3vviO36WIiJwUhcIgSA0FuP6sIp7bWMnWyka/yxEROWEKhUFyw9nFJAXiWfHKu36XIiJywhQK\ngyQzJZFLZo/hqfV7ae3o9rscEZETolAYRP+4cAKNbV384qXtfpciInJCFAqDqLQoizMnZbPi1R06\ntyAiI5JCYZB9e8ksepzj//1xi9+liIgcN4XCIJucG2bZwgk8v6mSt/c1+F2OiMhxUShEwY3nFJOV\nksiXf7OOLnWrLSIjiEIhCrLDQW5bPIuNFQ2seGWH3+WIiAyYQiFKLpk1hgun5XHn81vZvb/F73JE\nRAZEoRAlZsZtV8zCDJb9/DXqWzv9LklE5JgUClFUmJHEnVfPo/xAKz9dWeZ3OSIix6RQiLKLZ43h\nw7MLeGR1ucZyFpFhT6EwBK6YX8j+5g7ufVF3OovI8KZQGAKLTs3lH2bk8/3ntrBln+50FpHhS6Ew\nBBLi4/jeVXMIBeK472X1oioiw5dCYYhkpiSydME4Hl+7h+rGdr/LERHpl0JhCH3qnEn09Di+++xm\nv0sREemXQmEIFeek8NnzJvPbNXv406ZKv8sRETmMQmGIff6CKZySH+Yzv1rN0+v3+l2OiMj7KBSG\nWCgQz4M3nkFxTgqf+/Uafv36Lr9LEhHppVDwQW5qkMc+exbTC9L4+hNv8Zs3d/tdkogIoFDwTXpy\ngEc+eybnTM3lq4+/xZpdB/wuSUREoeCncDCBu/9xPvmpQT7+yzdYvVPBICL+Uij4LC0U4LF/PouU\nYDxX3fMqL2+r8bskEYlhUQ0FM7vYzLaYWZmZ3dLP+9ebWbWZrfUeN0aznuGqID2Je65bQFF2Mjf9\najUbK+r9LklEYlTUQsHM4oGfAJcAM4BlZjajn1n/xzk3z3v8Ilr1DHclEzL51Y2nk5QYz1ceWU9r\nR7ffJYlIDIpmS2EhUOac2+6c6wAeBhZH8fNGvHGZyfz7ktm8va+BG+5/g+b2Lr9LEpEYE81QKAT6\nXmtZ7k071FVmtt7MHjWz8f2tyMyWm9kqM1tVXV0djVqHjYtm5HPXx+bxxrv7+ecH19DT4/wuSURi\niN8nmn8PFDnn5gDPA//V30zOuXudc6XOudLc3NwhLdAPi+cVctviWfx1azU/fUEjtonI0IlmKOwB\n+v7lP86b1ss5V+ucO9hl6C+ABVGsZ0S59vQJXD53LHf8cSt/2LDP73JEJEZEMxTeBKaaWbGZJQLX\nAE/2ncHMCvq8vBxQ96EeM+N7S+cwuzCdrzyyjgdf3+l3SSISA6IWCs65LuDzwHNEftn/xjm30cxu\nM7PLvdluNrONZrYOuBm4Plr1jEShQDw/+Ng8inNS+D+Pb+C7z27WOQYRiSpzbmT9kiktLXWrVq3y\nu4wh1d7Vza2/28jDb+7moun5XHvGBBadmud3WSIygpjZaudc6bHmSxiKYuTkBBPi+e6Vsxmflcz3\nn9vCnzZXct4puSwszuKz500mPs78LlFERgm1FEaYts5u7nhuCw+9sYvmjm4ump7Hh+cUcEp+KjPH\npvtdnogMUwNtKSgURrAH/raDb/1+E909jjiDK+YV8ukPTGJ6QZrfpYnIMKNQiBFVDW3Ut3by7ac3\n89etkRv7lswv5FuLZ5IWCvhcnYgMFzqnECPy0kLkpYW4/4bTqGxo59tPb+Lxv+/hpW3VfOLMIq4+\nbTz5aSG/yxSREUIthVFo3e46vv/cFl4ui3TDvWzheL7+4RmkBPU3gEis0uEjYfXOA9z9l22s3FJN\nRnKAG88p5rozJpKRnOh3aSIyxAYaCn73fSRRtGBiJituWMj9N5xGR1cPd/xxK0t++iob9mi8BhHp\nn0IhBpx/ah6rv/4P/M/yM6hpbOeyu1/mnhfeoau7x+/SRGSY0eGjGHOguYOvPf4Wz27YR0F6iMKM\nJCZkJ3N6cRanF2dTlJPid4kiEgU6pyBH5JzjDxv28du/7+FAcwfvVDdxoKWT+DjjsjkFfLR0PGdP\nyfG7TBEZRAoFGTDnHKt2HuA///oOf9pcBcCH5xRw3im5lEzIYFxmMqFAvM9VisjJ0H0KMmBmxmlF\nWZxWlEVLRxc//ksZK155l6fX7wUgPy3I2ZNzaGzv4sxJ2XzglBwm54YxU59LIqONWgrSr7bObt7a\nU8+rZbW8UlbDmzv30/erkp8WJCWYwOzCdBYWZ3HW5ByKdT5CZNjS4SMZNM45unocnd097G/u4OVt\nNbxcVkNjW1dv1xoAxTkpXDm/kCvmF/JOdRNtnd2cMzWXsG6aE/GdQkGGRGd3D9urm3lhSxUveWHR\nn7njM5hdmMZF0/MJxMfR3tXNB6bmkhCvq6JFhoJCQXxRVtXEQ2/sIjM5wOxxGbxSVsPO2mberWlm\nW1XT+w5BjUkLUZiZRHZKIiUTM0mIMz48p4AxXl9ND76+CwcUZSezZmcdSYlxZKcEyQ5H5leHfyID\np1CQYaeqsY2dtS1U1LVyoLmDlVuq2X2ghcr6Npo7unvnm5idzM7alqOuK5gQR044SDAQx6yx6UzO\nDRMKxJEcTCAQZ1TUtVLX2smErGQunzuWvLQQnd09GOCArm7Hrv0tFOekkJig1oqMfgoFGTE6u3to\n6eimsqGNP22u5K9bqtlb38ZpRVl86pxiNlbUc/aUHFJDCexv7mBPXSt/3FhJ+YFW9je3s2t/CzVN\nHUf9jDFpIfY1tB02PTE+juxwIpNzw6QnBaiob6WlvZu4OGNybgqTcsOMz0yirKqJpMR4inNSGJeZ\nRDAhnqKcFN6paiIUiGdidjJm0NEVOe8yIStZV2fJsKJQkJhS39JJR3cPze1dBBLi2N/UwfisJMoP\ntPL79RVUNbQzJj1EU1sXualBunochRkhtlc3U1Hfxpvv7qeysY1pY9JIDSbQ3tXNnrpW9jd30HMC\n/0VSgwnMGZ9OfFwcCyZkkpcWpMJbH0BBeoiLZxUwJS88yHtCpH8KBZHj0NndQ1NbF5kp7+9BtrWj\nm+01TRRlp9DQ1kltUwf76tto6exmR00zRTkpOOfYVtlERX0rnd2OpEAce+vb2L2/ha4eR/mB1iN+\n7oyCNM6dmkNFfRuBOKPbORrbuqhtaqeyoZ2c1EQMo6Wji6rGdkomZDJjbBq54SCFmUnUt3aSGw4y\nszCNF7ZU86vXdrK+vJ5gQhxF2SkEEoy0UIAx6SFOzU/l7Ck5PLmugoL0EKUTsyjKSSa1z7mZA80d\n7NzfwtiMEHmpGodjNFEoiAwTtU3t/PntKibnhlkwMROAyoY2nlq/l6fWV7B2dx1j05No6+wmMSES\nKJNzU5iQlQxEbi6MMwgG4vn7zgNU1B9+GOygU/LDFGYkkRJMYGdtCy0dXQTi4zjQ0kFlQ3u/y+Sn\nBUlPCtDd49hR20K31zSamhdm0bQ8Zo5No7Gti5RgPOMyk2lu72JMeoiUxAQa2jppautiR23z+z5z\nZ20L9a2dXDAtj+kFacTHGTMK0nrH9Ojs7iEhzo55iK2hrZMHXt3BxooGZhWmM398BsW5KeSnhoiL\nO77Dc909jvg4Y8u+RirqWynMSGJ7dTNPrtvDjII0QoF4ntu4jzgzxmUmMyk3hXAwgQnZyRRlpzAm\nLURS4si9s1+hIDJCHPxldVB1Yzs54cR+f2F29zhqmtpJiDP21LXS46ClvYu39tRTkJHEpbPGHPEy\n3+rGdv6wYS854SDTC9JYV15HRV0bZVVNHGjpwICM5ETOnZrD3vo2/rq1ijU76+g4gd50E+KMnHDw\nsPM4qcEEQonx1Ld2kpOSyNzxGaQnBTjQEjkP4xyYwYGWTirqWnnj3f3eob4k9tS91+JKDSWQGw4y\nKTdMTjiRzXsbqG/tJDkxgZKJGeSEg4SDCWyrbGJbVSP7mzvYUdtCaiiBxrauI9adlxokKyWRHbXN\ntHUevt1j0kIkJsQRTIjDDJyDvLQggfg4Ts1PJS0pwCn5qZRMyCAjOZHyAy3kp4V6u4nZvb+Fe1/c\nTltnN3FmFGSEmDYmlYnZKUzKTSGYEL3QUSiIyEnr6OrhneomurodycF4du1vobWjm7bObmqbOshI\nDpCYEEdxTgoNrV2kJ0UORSUlxjElL5WtlY3s3t9CS0c3G/bUs7+5g/auHtKSEqht6mDT3gbqWjpJ\niDMa2joJ9Am0jKQAs8el85nzJlMyIZO6lg7W7q7j3Zpm3tpTT2tHNxsq6qlv6SQnHGRKXpjmji7W\n7KyjtbP7fdtRlJ1MOJTApJww0wpSmVGQRmVDGxnJiZyan0pWOJGdNS1MzQ8TCsTjnKOhrYvm9i4q\n6lrZvK+R/U0dbK9porm9m5rtG406AAAKD0lEQVSmdtKTAnR299Dc0U1nVw9lVU1HDNCC9BCZyYls\nrWwEIDkxns5ud1idhRlJZIcTyUsNRobaTQ2yYGImE7NSyElNJDnxxG8EVSiIyIjR0+PocW5Qbmbs\n6OrBDOpaOmnt6GZsRmhIbpJs7+pmf3MH71Q18/a+BnbUNjMxK4WWjm527m+murGd7JREvvzBUxmf\nlUxPj6OxvYuNe+opr2tl9/4W9hxopaa5g6qGNqob26ltfu+qOjN45DNnUlqUdUL1qUM8ERkx4uKM\nOAbnEt6D953kpgYHZX0DFUyIpyA9iYL0JM6Zeuyu5+PijPSkAGcdpZv6lo4uXi2r5UBL5FLsqfmp\ng1lyvxQKIiLDVHJiAhfNyB/Sz9StnCIi0iuqoWBmF5vZFjMrM7Nb+nk/aGb/473/upkVRbMeERE5\nuqiFgpnFAz8BLgFmAMvMbMYhs30KOOCcmwL8APiPaNUjIiLHFs2WwkKgzDm33TnXATwMLD5knsXA\nf3nPHwUuNHUYIyLim2iGQiGwu8/rcm9av/M457qAeiA7ijWJiMhRjIgTzWa23MxWmdmq6urqYy8g\nIiInJJqhsAcY3+f1OG9av/OYWQKQDtQeuiLn3L3OuVLnXGlubm6UyhURkWiGwpvAVDMrNrNE4Brg\nyUPmeRL4hPd8KfAXN9JusRYRGUWidvOac67LzD4PPAfEA/c55zaa2W3AKufck8Avgf82szJgP5Hg\nEBERn0T1jmbn3DPAM4dM+0af523AR6NZg4iIDNyIONEsIiJDY8T1kmpm1cDOE1w8B6gZxHJGIu0D\n7QPQPoDY2wcTnXPHvFJnxIXCyTCzVQPpOnY00z7QPgDtA9A+OBIdPhIRkV4KBRER6RVroXCv3wUM\nA9oH2gegfQDaB/2KqXMKIiJydLHWUhARkaOImVA41oA/o4WZjTezlWa2ycw2mtkXvOlZZva8mW3z\n/s30ppuZ/cjbL+vNrMTfLRgcZhZvZn83s6e818XeQE5l3sBOid70UTvQk5llmNmjZva2mW02szNj\n8HvwRe//wQYze8jMQrH4XTgeMREKAxzwZ7ToAr7snJsBnAF8ztvWW4A/O+emAn/2XkNkn0z1HsuB\ne4a+5Kj4ArC5z+v/AH7gDeh0gMgATzC6B3r6IfAH59w0YC6R/REz3wMzKwRuBkqdc7OIdLdzDbH5\nXRg459yofwBnAs/1ef1V4Kt+1zVE2/474B+ALUCBN60A2OI9/09gWZ/5e+cbqQ8iPfL+GbgAeAow\nIjcpJRz6fSDSN9eZ3vMEbz7zexsGYR+kA+8eui0x9j04OF5LlvezfQr4UKx9F473ERMtBQY24M+o\n4zV/5wOvA/nOub3eW/uAfO/5aNw3dwH/G+jxXmcDdS4ykBO8fxtH60BPxUA1sMI7jPYLM0shhr4H\nzrk9wB3ALmAvkZ/tamLvu3BcYiUUYo6ZhYHHgH9xzjX0fc9F/hQalZedmdlHgCrn3Gq/a/FZAlAC\n3OOcmw80896hImB0fw8AvPMli4kE5FggBbjY16JGgFgJhYEM+DNqmFmASCA86Jz7rTe50swKvPcL\ngCpv+mjbN2cDl5vZDiLjgl9A5Nh6hjeQE7x/Gwc00NMIVA6UO+de914/SiQkYuV7AHAR8K5zrto5\n1wn8lsj3I9a+C8clVkJhIAP+jApmZkTGqdjsnLuzz1t9BzT6BJFzDQenf9y7+uQMoL7P4YURxzn3\nVefcOOdcEZGf81+cc9cCK4kM5ASHb/+oG+jJObcP2G1mp3qTLgQ2ESPfA88u4AwzS/b+XxzcBzH1\nXThufp/UGKoHcCmwFXgH+D9+1xPF7TyHyCGB9cBa73EpkWOjfwa2AX8Csrz5jciVWe8AbxG5UsP3\n7RikfXE+8JT3fBLwBlAGPAIEvekh73WZ9/4kv+sexO2fB6zyvgtPAJmx9j0AvgW8DWwA/hsIxuJ3\n4XgeuqNZRER6xcrhIxERGQCFgoiI9FIoiIhIL4WCiIj0UiiIiEgvhYIMO2bWbWZrzWydma0xs7OO\nMX+Gmf3zANb7gplpTN4+zOx+M1t67DklVigUZDhqdc7Nc87NJdJ54XePMX8GcMxQ8Eufu2dFhj2F\nggx3aUS6N8bMwmb2Z6/18JaZLfbmuR2Y7LUuvu/N+2/ePOvM7PY+6/uomb1hZlvN7Fxv3ngz+76Z\nvemNJfAZb3qBmb3orXfDwfn7MrMdZvY977PeMLMp3vT7zexnZvY68D1vHIMnvPW/ZmZz+mzTCm/5\n9WZ2lTf9g2b2N29bH/H6ssLMbrfIWBnrzewOb9pHvfrWmdmLx9gmM7O7LTK2yJ+AvMH8YcnIp79g\nZDhKMrO1RO4wLSDSfxFAG7DEOddgZjnAa2b2JJGO3mY55+YBmNklRDpCO90512JmWX3WneCcW2hm\nlwK3Eukf51NEunU4zcyCwCtm9kfgSiLdKn/HG5Mj+Qj11jvnZpvZx4n00PoRb/o44CznXLeZ/Rj4\nu3PuCjO7AHiAyB3H//fg8l7tmd62fR24yDnXbGb/BnzJzH4CLAGmOeecmWV4n/MN4EPOuT19ph1p\nm+YDpxIZVySfSLcP9w3opyIxQaEgw1Frn1/wZwIPmNksIl0x/LuZfYBIt9iFvNf1c18XASuccy0A\nzrn9fd472EHgaqDIe/5BYE6fY+vpRAabeRO4zyIdDD7hnFt7hHof6vPvD/pMf8Q51+09Pwe4yqvn\nL2aWbWZpXq3XHFzAOXfAIj29ziDyixwgEfgbka6c24BfWmREuae8xV4B7jez3/TZviNt0weAh7y6\nKszsL0fYJolRCgUZ1pxzf/P+cs4l0odTLrDAOddpkZ5QQ8e5ynbv327e+/4b8L+cc88dOrMXQB8m\n8kv3TufcA/2VeYTnzcdZW+/HAs8755b1U89CIh27LQU+D1zgnPusmZ3u1bnazBYcaZu8FpLIEemc\nggxrZjaNyDCKtUT+2q3yAmERMNGbrRFI7bPY88ANZpbsraPv4aP+PAfc5LUIMLNTzCzFzCYClc65\nnwO/INL1dH8+1uffvx1hnpeAa731nw/UuMg4F88Dn+uzvZnAa8DZfc5PpHg1hYF059wzwBeJDLGJ\nmU12zr3unPsGkYF1xh9pm4AXgY955xwKgEXH2DcSY9RSkOHo4DkFiPzF+wnvuPyDwO/N7C0ivX++\nDeCcqzWzV8xsA/Csc+5fzWwesMrMOoBngK8d5fN+QeRQ0hqLHK+pBq4g0svqv5pZJ9AEfPwIy2ea\n2XoirZDD/rr3fJPIoaj1QAvvddH8beAnXu3dwLecc781s+uBh7zzARA5x9AI/M7MQt5++ZL33vfN\nbKo37c/AOiI9o/a3TY8TOUeziUjX0kcKMYlR6iVV5CR4h7BKnXM1ftciMhh0+EhERHqppSAiIr3U\nUhARkV4KBRER6aVQEBGRXgoFERHppVAQEZFeCgUREen1/wHxJXDNrJ6rugAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iW3MKr-Gilit",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "The current buffered version just uses the lr of the first param_group leading to suboptimal results.\n",
        "\n",
        "This is relevant for those using different learning rates e.g. for transfer learning."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yfsz1Xxqi_W4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}