Cologne AI and Machine Learning Meetup - CAIML #7 - Hands-on by Immanuel Weber

ai_ml_meetup_image_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ai_ml_meetup-image_classification.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KsP9H_m-PReQ",
        "colab_type": "text"
      },
      "source": [
        "# Image Classification using pytorch and fast.ai\n",
        "\n",
        "* for documentation and tutorials see https://docs.fast.ai/index.html\n",
        "* for the courses etc see https://www.fast.ai/"
        "* __make sure that you select a gpu in the runtime menu! (Runtime > Change runtime type)__"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1qMzFNWX15Vf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%matplotlib inline\n",
        "%reload_ext autoreload\n",
        "%autoreload 2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u-JXsZx01-oJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import fastai\n",
        "from fastai.vision import *\n",
        "from fastai.utils.show_install import *\n",
        "\n",
        "torch.cuda.set_device(0) # this controls which gpu should be used\n",
        "show_install()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JDrOUx1vAIZe",
        "colab_type": "text"
      },
      "source": [
        "## connect to your google drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8FOiD7l82PGH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive', force_remount=True)\n",
        "root_dir = \"/content/gdrive/My Drive/\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NhtpjkHjAOE4",
        "colab_type": "text"
      },
      "source": [
        "## define paths to data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "trZUBHR4Z1bS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "PATH = Path(root_dir + 'beer_pizza')\n",
        "DATA_PATH = PATH / 'training_data'\n",
        "EXAMPLE_PATH = PATH / 'test_data'\n",
        "DATA_PATH.ls()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mq7Mhn65fKCc",
        "colab_type": "text"
      },
      "source": [
        "## collect data\n",
        "\n",
        "* to run this file you need to provide some image data (as mentioned I can not provide the beer and pizza images)\n",
        "* you can either collect the imagery yourself using google image search or some other source\n",
        "* or you could use the fast.ai `download_google_images()` function like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Z4sqdA5fnP0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# from fastai.widgets import *\n",
        "# labels = ['beer', 'pizza']\n",
        "# for label in labels: \n",
        "#     download_google_images(DATA_PATH, label, size='>400*300', n_images=100)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "upGOcIRSfw2i",
        "colab_type": "text"
      },
      "source": [
        "* this will download images for the provided labels from google image search to the provided path\n",
        "* note that this most likely will result in some crashes of the function, however in my short experiments it still downloaded the images\n",
        "* you have to manually check if there are images which are not valid\n",
        "* note that for more than 100 images you need to install chromedriver, have look here: https://docs.fast.ai/widgets.image_cleaner.html#ImageDownloader"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RuGf-0uy2hgZ",
        "colab_type": "text"
      },
      "source": [
        "## define transformations\n",
        "\n",
        "* the transformations are defined as a tuple of two arrays\n",
        "* the first array contains the transformations for the training data\n",
        "* the second array contains the transformations for the validation data\n",
        "* you can use `get_transformations()` to directly get that tuple containing some default transformations\n",
        "* note that not every transformation makes sense for every dataset\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iqCVM08l2bDl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "transforms = ([\n",
        "    rotate(degrees=3.0, p=0.5),\n",
        "#     brightness(change=(0.05, 0.05), p=0.5),\n",
        "#     contrast(scale=(0.05, 0.05), p=0.5),\n",
        "    flip_affine(p=0.5)\n",
        "], [])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NDxDEVyq2krR",
        "colab_type": "text"
      },
      "source": [
        "## create dataset\n",
        "* the fast.ai library provides many different options to create a dataset, have a look at the documentation!\n",
        "* note that the `data.show_batch()` does not apply the final normalization, thats why the images look naturally\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DJlO4AVj2gNG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "image_size = 224 # the architecture is designed to consume images of size 224 x 224\n",
        "batch_size = 12\n",
        "\n",
        "data = (\n",
        "  ImageList.from_folder(DATA_PATH)\n",
        "    .split_by_rand_pct(seed=0, valid_pct=0.4)\n",
        "    .label_from_folder() # this selects the folder names as labels\n",
        "    .transform(transforms, size=image_size, resize_method=ResizeMethod.SQUISH, padding_mode='zeros')\n",
        "    .databunch(bs=batch_size)\n",
        "    .normalize(imagenet_stats)\n",
        ")\n",
        "print(data)\n",
        "data.show_batch()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jeu4Ohtz2uBF",
        "colab_type": "text"
      },
      "source": [
        "## creater learner object\n",
        "\n",
        "* the `cnn_learner` creates a network and loads a pretrained model (so transfer learning is the default)\n",
        "* it can automatically remove the fully connected head and create a new one for the classes provided in the dataset\n",
        "* it does all the heavy lifting: it runs the training loop, loads data, computes metrics, plots data, etc.\n",
        "* you can add stuff, like metrics, by implementing callbacks\n",
        "* we use a ResNet-34 \n",
        "* and add accuracy as an additional metric\n",
        "* note that the learner freezes the backend part initially, you can unfreeze it using `learner.unfreeze()` to also train the backend\n",
        "* by default fast.ai rel"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i_Zo3diO2spB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learner = cnn_learner(data, models.resnet34, metrics=accuracy, callback_fns=[ShowGraph])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H3OJi4Fd3FXz",
        "colab_type": "text"
      },
      "source": [
        "## run learning rate finder\n",
        "\n",
        "* use this to get an idea how different learning rates behave\n",
        "* this is more of an intuition guided thing than pure science\n",
        "* try to pick a learning rate which is quite high and in a steep area\n",
        "* stay away from the minimal point or fast changing regions\n",
        "* experiment and you will get a feeling for that\n",
        "* note that the `learner.recorder.plot()` removes some of the samples at the front and back, this can be changed using parameters "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "944UZZU23HpU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learner.lr_find()\n",
        "learner.recorder.plot()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kng7S07SD5P-",
        "colab_type": "text"
      },
      "source": [
        "## run training in a \"traditional\" way\n",
        "\n",
        "* pick one learning rate and let it run for some epochs\n",
        "* you can optionally provide a learning rate scheduler to the learner object or the `.fit()` function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3RythwbCHfYi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learner.fit(10, 1e-3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "abR6uP-0IvI4",
        "colab_type": "text"
      },
      "source": [
        "## run training using the \"one cycle\" scheme\n",
        "\n",
        "* use training scheme where the learning rate is changed using ramps\n",
        "* it is based on this paper https://arxiv.org/pdf/1803.09820.pdf\n",
        "* it should improve training times and lead to \"super convergence\"\n",
        "* the number epochs provided here defines how long the ramp is\n",
        "* experiment with the number of epochs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FW2QXqHjJWzQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# uncomment the the following to try the one cycle scheme\n",
        "# learner = cnn_learner(data, models.resnet34, metrics=accuracy, callback_fns=[ShowGraph]) # we recreate the learner to start from the beginning"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "TvtLj_1iJdsk",
        "colab": {}
      },
      "source": [
        "# lr = 1e-2\n",
        "# learner.fit_one_cycle(4, max_lr=lr)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yh62IacwKE8-",
        "colab_type": "text"
      },
      "source": [
        "#have a look at the data\n",
        "\n",
        "* predict\n",
        "* visualize \"difficult\" data\n",
        "* see heat maps for important parts of the images (see Grad-CAM)\n",
        "* compute a confusion matrix"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w0fipJZQla_M",
        "colab_type": "text"
      },
      "source": [
        "## get some visualizations for the networks performance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "p8eBmYwyJdsd",
        "colab": {}
      },
      "source": [
        "interp = ClassificationInterpretation.from_learner(learner)\n",
        "losses, idxs = interp.top_losses()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "l3g0dKbsJdsa",
        "colab": {}
      },
      "source": [
        "interp.plot_top_losses(9, figsize=(15,11))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7-WNfk1IJdsX",
        "colab": {}
      },
      "source": [
        "interp.plot_confusion_matrix(figsize=(6,6), dpi=100)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zlYY57bPKo8z",
        "colab_type": "text"
      },
      "source": [
        "## let the network do predictions on single images"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5aTzRzZHJdsO",
        "colab": {}
      },
      "source": [
        "test_files = [\n",
        "    'car.jpg',\n",
        "    'wine.jpg',\n",
        "    'beer_pizza1.jpg',\n",
        "    'beer_pizza2.jpg',\n",
        "    'cake.jpg'\n",
        "]\n",
        "\n",
        "image = open_image(EXAMPLE_PATH/test_files[4])\n",
        "image = image.resize(224)\n",
        "image_data = normalize(image.data, torch.tensor(imagenet_stats[0]), torch.tensor(imagenet_stats[1]))\n",
        "image = Image(image_data)\n",
        "image.show()\n",
        "learner.predict(image)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}