Skip to content

Instantly share code, notes, and snippets.

@muellerzr

muellerzr/train_test_discrepancy.ipynb

Created June 22, 2022 02:06
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save muellerzr/42fba2665d84adbd2fbeb2d76a7e617e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save muellerzr/42fba2665d84adbd2fbeb2d76a7e617e to your computer and use it in GitHub Desktop.
Download ZIP
Testing the Train/Test Time Discrepancy
Display the source blob
Display the rendered blob
Raw
train_test_discrepancy.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/muellerzr/42fba2665d84adbd2fbeb2d76a7e617e/scratchpad.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# Experimenting with [Fixing the train-test resolution discrepancy](https://arxiv.org/pdf/1906.06423.pdf)\n",
"\n",
"> A very small experiment on testing this paper vs just fine-tuning for one more epoch"
],
"metadata": {
"id": "o1FayI_q5Iyi"
}
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "lIYdn1woOS1n"
},
"outputs": [],
"source": [
"from fastai.vision.all import *"
]
},
{
"cell_type": "markdown",
"source": [
"## Experiment Setup\n",
"\n",
"This experiment poses using the [ImageWoof](https://github.com/fastai/imagenette) data setup and using a resnet18 that was *not* pretrained to test if this test-train resolution discrepancy lives up to its name.\n",
"\n",
"I'm using code from my [Walk with fastai lesson](https://walkwithfastai.com/ImageWoof), which is why this looks like the lower level part of the API"
],
"metadata": {
"id": "jsDNuoDPIbWf"
}
},
{
"cell_type": "code",
"source": [
"set_seed(99, True)\n",
"path = untar_data(URLs.IMAGEWOOF) # Declare our dataset\n",
"tfms = [[PILImage.create], [parent_label, Categorize()]] # Our X and Y base transforms\n",
"item_tfms = [ToTensor(), Resize(448)] # Our item transforms\n",
"batch_tfms = [FlipItem(), RandomResizedCropGPU(224, min_scale=0.35),\n",
" IntToFloatTensor(), Normalize()] # The batch transforms\n",
"\n",
"items = get_image_files(path) # Get our items\n",
"split_idx = GrandparentSplitter(valid_name='val')(items) # Create our splits\n",
"dsets = Datasets(items, tfms, splits=split_idx) # Create our datasets\n",
"dls = dsets.dataloaders(after_item=item_tfms, after_batch=batch_tfms, bs=64) # Build our dataloaders"
],
"metadata": {
"id": "cqsZYOa35tZ4"
},
"execution_count": 106,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"We'll use a base `resnet18` as mentioned before:"
],
"metadata": {
"id": "i9AwxYsHI72T"
}
},
{
"cell_type": "code",
"source": [
"learn = vision_learner(dls, resnet18, pretrained=False, metrics=accuracy).to_fp16()"
],
"metadata": {
"id": "1Z3YEaMW6DGu"
},
"execution_count": 90,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"And fit for five epochs at a decent learning rate:"
],
"metadata": {
"id": "cARtrxM8I-SD"
}
},
{
"cell_type": "code",
"source": [
"learn.fit_one_cycle(5, 4e-3)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 206
},
"id": "HQjwM9fP6UyF",
"outputId": "ad91b584-d8a7-4341-9675-4b4fcb0d9074"
},
"execution_count": 72,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
"\n",
"<style>\n",
" /* Turns off some styling */\n",
" progress {\n",
" /* gets rid of default border in Firefox and Opera. */\n",
" border: none;\n",
" /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
" background-size: auto;\n",
" }\n",
" .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
" background: #F44336;\n",
" }\n",
"</style>\n"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>3.489550</td>\n",
" <td>2.619425</td>\n",
" <td>0.135912</td>\n",
" <td>00:58</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>3.022590</td>\n",
" <td>2.408455</td>\n",
" <td>0.163655</td>\n",
" <td>00:58</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>2.738046</td>\n",
" <td>2.283973</td>\n",
" <td>0.175108</td>\n",
" <td>00:58</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>2.579401</td>\n",
" <td>2.248823</td>\n",
" <td>0.181980</td>\n",
" <td>00:57</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>2.535550</td>\n",
" <td>2.256250</td>\n",
" <td>0.180962</td>\n",
" <td>00:58</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
]
},
"metadata": {}
}
]
},
{
"cell_type": "markdown",
"source": [
"Next we take the ground truth accuracy, which wound up being 18.09%"
],
"metadata": {
"id": "I8zvcbKwJBEY"
}
},
{
"cell_type": "code",
"source": [
"# Ground Truth:\n",
"learn.validate()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"id": "HhS-iUmO6YTd",
"outputId": "03b22d34-72fc-4591-8cae-a3fb18133e1e"
},
"execution_count": 73,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
"\n",
"<style>\n",
" /* Turns off some styling */\n",
" progress {\n",
" /* gets rid of default border in Firefox and Opera. */\n",
" border: none;\n",
" /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
" background-size: auto;\n",
" }\n",
" .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
" background: #F44336;\n",
" }\n",
"</style>\n"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
""
]
},
"metadata": {}
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(#2) [2.2562503814697266,0.18096207082271576]"
]
},
"metadata": {},
"execution_count": 73
}
]
},
{
"cell_type": "markdown",
"source": [
"This little func can change the size of a transform in a pipeline.\n",
"\n",
"> Note: This overrides both `dls.train.{pipe}` and `dls.valid.{pipe}`'s pipelines since they all point to the same reference point in the transform, visualization below:"
],
"metadata": {
"id": "t6KbHqJRJESO"
}
},
{
"cell_type": "code",
"source": [
"assert id(dls.train.after_item[0]) == id(dls.valid.after_item[0])"
],
"metadata": {
"id": "-WyHIdWSJQVc"
},
"execution_count": 107,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# With final size of 448x448\n",
"def change_size(\n",
" pipe:Pipeline, # A transform pipeline\n",
" augmentation:Transform, # A class type of a Transform \n",
" size # The size of the new transform\n",
"):\n",
" \"Dynamically changes transform size to size\"\n",
" for tfm in pipe:\n",
" if isinstance(tfm, augmentation):\n",
" tfm.size = size if isinstance(size, tuple) else (size,size)"
],
"metadata": {
"id": "LWjJrNUuINJX"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"We'll upscale to 448x448"
],
"metadata": {
"id": "XGq2WFXfJXk_"
}
},
{
"cell_type": "code",
"source": [
"change_size(learn.dls.valid.after_batch, RandomResizedCropGPU, 448)"
],
"metadata": {
"id": "8OrFHBtGDGMd"
},
"execution_count": 78,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"And perform inference:"
],
"metadata": {
"id": "Ft6SDXNWJZnI"
}
},
{
"cell_type": "code",
"source": [
"# 448 no fine-tuning\n",
"learn.validate()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"id": "LokqaMP1A03x",
"outputId": "51fb6633-081f-4215-cfa7-8ff429caf882"
},
"execution_count": 79,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
"\n",
"<style>\n",
" /* Turns off some styling */\n",
" progress {\n",
" /* gets rid of default border in Firefox and Opera. */\n",
" border: none;\n",
" /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
" background-size: auto;\n",
" }\n",
" .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
" background: #F44336;\n",
" }\n",
"</style>\n"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
""
]
},
"metadata": {}
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(#2) [2.277555227279663,0.18681597709655762]"
]
},
"metadata": {},
"execution_count": 79
}
]
},
{
"cell_type": "markdown",
"source": [
"We can see our new accuracy is 18.68%, which **is** an improvement. But is it better than one more epoch?"
],
"metadata": {
"id": "mMOzIWZiJauP"
}
},
{
"cell_type": "code",
"source": [
"learn.fit_one_cycle(1, 4e-3)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 81
},
"id": "FH_UZESLFMBu",
"outputId": "33ed836d-5dc2-4733-d719-b62e91f6498a"
},
"execution_count": 81,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
"\n",
"<style>\n",
" /* Turns off some styling */\n",
" progress {\n",
" /* gets rid of default border in Firefox and Opera. */\n",
" border: none;\n",
" /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
" background-size: auto;\n",
" }\n",
" .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
" background: #F44336;\n",
" }\n",
"</style>\n"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>epoch</th>\n",
" <th>train_loss</th>\n",
" <th>valid_loss</th>\n",
" <th>accuracy</th>\n",
" <th>time</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>0</td>\n",
" <td>2.445351</td>\n",
" <td>2.227491</td>\n",
" <td>0.191143</td>\n",
" <td>01:57</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
]
},
"metadata": {}
}
]
},
{
"cell_type": "markdown",
"source": [
"19.11% at the end, so no. But if you don't want to train for one more epoch (for whatever reason), it can boost your accuracy.\n",
"\n",
"Do take note this is an `n` of `1`, but given this change and fundamentally it would make sense that the one more epoch would perform better. We've shown that this paper is not incorrect as there was an accuracy increase, but just not as good as one more epoch.\n",
"\n",
"Table below of the results:\n",
"\n",
"| **Kind** | **Accuracy** |\n",
"|:---------------------------:|:------------:|\n",
"| Baseline | 18.09% |\n",
"| Test-Resolution Discrepancy | 18.68% |\n",
"| One More Epoch | 19.11% |"
],
"metadata": {
"id": "e0egiPo-JhOw"
}
}
],
"metadata": {
"colab": {
"name": "scratchpad",
"provenance": [],
"machine_shape": "hm",
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"accelerator": "GPU",
"gpuClass": "standard"
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment