Pracw8-9

prac7.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Pracw8-9",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMcuHGDAxjoSge/FEZwaW8+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/llewynS/959cdcab41dd336d17eafa967370df8f/prac7.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3RtpIC9fDJAM"
      },
      "source": [
        "# PRAC W8-9\n",
        "## MLPS and Convolutional Neural Networks (CNNs) using Tensorflow 2.x\n",
        "\n",
        "A lot of this will be borrowed from the Tensorflow introduction found [here](https://www.tensorflow.org/tutorials/)\n",
        "\n",
        "You've already covered multilayer perceptrons in last weeks prac. CNNs are possibly part of the reason you're interested in this course due to their strengths in image classification.\n",
        "\n",
        "[This link](https://cs231n.github.io/convolutional-networks/) provides a good overview of CNNs and would be useful to read. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MyAHHQ7dEmH7"
      },
      "source": [
        "Some useful imports "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PW-AQDwnC9TT"
      },
      "source": [
        "import tensorflow as tf\n",
        "%load_ext tensorboard\n",
        "from tensorflow.keras.layers import Dense, Flatten, Conv2D\n",
        "from tensorflow.keras import Model\n",
        "import datetime\n",
        "!rm -rf ./logs/ \n",
        "print(tf.__version__) #Double check the colab has the instance of tensorflow we want"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EkZrSEmCE0Kb"
      },
      "source": [
        "Import the MNIST dataset and normalise"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qZKHRdCHEo9o"
      },
      "source": [
        "mnist = tf.keras.datasets.mnist\n",
        "\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
        "\n",
        "# Add a channels dimension\n",
        "x_train = x_train[..., tf.newaxis]\n",
        "x_test = x_test[..., tf.newaxis]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7dSeLLOOFIOj"
      },
      "source": [
        "Use `tf.data` to batch and shuffle the dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g6t38tFVFSWG"
      },
      "source": [
        "train_ds = tf.data.Dataset.from_tensor_slices(\n",
        "    (x_train, y_train)).shuffle(10000).batch(32)\n",
        "\n",
        "test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92gcjhflRFT8"
      },
      "source": [
        "# MLPs\n",
        "We can really easily create basic [sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) (the most common) models  in Tensorflow using the [keras](https://www.tensorflow.org/guide/keras) api. \n",
        "\n",
        "Here we'll create a single layer MLP that uses the relu activation in its hidden layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3oau4RLUREFv"
      },
      "source": [
        "model =  tf.keras.models.Sequential([\n",
        "            tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
        "            tf.keras.layers.Dense(512, activation='relu'),\n",
        "            tf.keras.layers.Dense(10, activation='softmax')\n",
        "          ])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nyn0Wb5AR6Hr"
      },
      "source": [
        "Then all we need to do is add an [optimiser](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/) and a [loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses)  function.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T32PJ7lOREIj"
      },
      "source": [
        "model.compile(optimizer='sgd',\n",
        "              loss='sparse_categorical_crossentropy',metrics=['accuracy'])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BMGRzLYYVGxh"
      },
      "source": [
        "We can then fit our model to the and evaluate it all in one simple step. Maybe we also want to be able to check out some sweet graphage - luckily for us, TensorFlow comes with a fantastic visualisation tool called TensorBoard"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eXzX9hqxRELa"
      },
      "source": [
        "log_dir = \"logs/fit/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") #datetime storage\n",
        "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1) #TB callbacks\n",
        "\n",
        "model.fit(train_ds, \n",
        "          epochs=5, \n",
        "          validation_data=test_ds, callbacks=[tensorboard_callback])\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YUma3lGLasch"
      },
      "source": [
        "This didn't work in my FireFox withough enabling cookies. Some googling informed me that it would require me to accept a level of cookies that I was unwilling to accept. If you're also feeling this way you can just download this noebook and run it locally :) "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HRZn3aArREQi"
      },
      "source": [
        "%tensorboard --logdir logs/fit"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7iZ9j5g6dw0c"
      },
      "source": [
        "#CNNs \n",
        "\n",
        "CNNs are neural networks that are structured in a certain way to take advantage of the inherent structure in images. They are just as easy to make in Keras as MLPs! :o"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IsVLWqrBhsL9"
      },
      "source": [
        "log_dir = \"logs/fit/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") #datetime storage\n",
        "tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1) #TB callbacks\n",
        "\n",
        "model =  tf.keras.models.Sequential([\n",
        "            tf.keras.layers.Conv2D(32,3),\n",
        "             tf.keras.layers.Flatten(),\n",
        "            tf.keras.layers.Dense(512, activation='relu'),\n",
        "            tf.keras.layers.Dense(10, activation='softmax')\n",
        "          ])\n",
        "\n",
        "model.compile(optimizer='sgd',\n",
        "              loss='sparse_categorical_crossentropy',metrics=['accuracy'])\n",
        "\n",
        "model.fit(train_ds, \n",
        "          epochs=5, \n",
        "          validation_data=test_ds, callbacks=[tensorboard_callback])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y7q5fVt1mSLw"
      },
      "source": [
        "%tensorboard --logdir logs/fit"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ndOUSoOzjMuP"
      },
      "source": [
        "# Tasks for you\n",
        "\n",
        "Now you've seen how easy it is to create an MLP/CNN in TensorFlow using Keras experiment with the parameters.\n",
        "\n",
        "Change:\n",
        "*   the activation functions from relu to tanh or sigmoids.\n",
        "*   the parameters of the optimiser. Hint use the actual optimiser instead of a string. \n",
        "*   the optimiser from SGD to Adam or AdaGrad.\n",
        "*   the model to have more or less layers.\n",
        "\n",
        "To do this it might be useful to design some kind of experiment where you sweep some/all/more than the parameters mentioned above. \n",
        "\n",
        "Try a different Dataset to MNIST.\n",
        "\n",
        "Hint: CNNs take some time to train so have a think about how many parameters are in a CNN and the best way you can reduce them. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zFDhhRdRb2Xo"
      },
      "source": [
        "#Extra\n",
        "\n",
        "Sometimes we want to create models at a lower level of abstraction. We can create a similar MLP and fit it using the subsequent cells. Keep in mind this is still using Keras which is a high level API. You can get even more control by doing all the operations yourself. The level of abstraction you want will be dependent on your application. In this prac we'll just stick with Keras."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xxmI-BmsFS8u"
      },
      "source": [
        "Define a basic MLP using the [model subclassing API](https://www.tensorflow.org/guide/keras#model_subclassing)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h3IKyzTCDNGo"
      },
      "source": [
        "class MLP(Model):\n",
        "  def __init__(self, hidden_layers = [512], activation = 'relu', output_dimensions=10):\n",
        "    super().__init__()\n",
        "    self._inp = Flatten() #ensure the input is flattened\n",
        "    self._densebois = []\n",
        "    for h in hidden_layers:\n",
        "      self._densebois.append(Dense(h, activation=activation))\n",
        "    self._out = Dense(output_dimensions)\n",
        "\n",
        "  def call(self, x):\n",
        "    x = self._inp(x)\n",
        "    for layer in self._densebois:\n",
        "      x = layer(x)\n",
        "    return self._out(x)\n",
        "\n",
        "# Create an instance of the model\n",
        "model = MLP()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dox4__6ZG-j4"
      },
      "source": [
        "Pick an arbitrary [optimiser](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/) for training and define the model [loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses) functions. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VevCeO7pG5M5"
      },
      "source": [
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
        "\n",
        "optimizer = tf.keras.optimizers.SGD()\n",
        "\n",
        "train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
        "train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')\n",
        "\n",
        "test_loss = tf.keras.metrics.Mean(name='test_loss')\n",
        "test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lYnAACjiI7vg"
      },
      "source": [
        "Tensorflow runs much faster if we create tensorflow functions for training and validating our models. We do this in Python using `@tf.function`.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v-qFyQjdHcRq"
      },
      "source": [
        "@tf.function\n",
        "def train_step(images, labels):\n",
        "  with tf.GradientTape() as tape:\n",
        "    # training=True is only needed if there are layers with different\n",
        "    # behavior during training versus inference (e.g. Dropout).\n",
        "    predictions = model(images, training=True)\n",
        "    loss = loss_object(labels, predictions)\n",
        "  gradients = tape.gradient(loss, model.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
        "\n",
        "  train_loss(loss)\n",
        "  train_accuracy(labels, predictions)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gzPfhwG5H1nY"
      },
      "source": [
        "@tf.function\n",
        "def test_step(images, labels):\n",
        "  # training=False is only needed if there are layers with different\n",
        "  # behavior during training versus inference (e.g. Dropout).\n",
        "  predictions = model(images, training=False)\n",
        "  t_loss = loss_object(labels, predictions)\n",
        "\n",
        "  test_loss(t_loss)\n",
        "  test_accuracy(labels, predictions)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mZyBPSDFJOUA"
      },
      "source": [
        "We can then simply run our model for a number of epochs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nfOF1eqxH1-C"
      },
      "source": [
        "EPOCHS = 5\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  # Reset the metrics at the start of the next epoch\n",
        "  train_loss.reset_states()\n",
        "  train_accuracy.reset_states()\n",
        "  test_loss.reset_states()\n",
        "  test_accuracy.reset_states()\n",
        "\n",
        "  for images, labels in train_ds:\n",
        "    train_step(images, labels)\n",
        "\n",
        "  for test_images, test_labels in test_ds:\n",
        "    test_step(test_images, test_labels)\n",
        "\n",
        "  template = 'Epoch {}, Loss: {}, Accuracy: {}, Test Loss: {}, Test Accuracy: {}'\n",
        "  print(template.format(epoch + 1,\n",
        "                        train_loss.result(),\n",
        "                        train_accuracy.result() * 100,\n",
        "                        test_loss.result(),\n",
        "                        test_accuracy.result() * 100))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljkjxlI_Jd6b"
      },
      "source": [
        "This training/validation looks pretty useful but we want to be able to do this to many models. So lets make a class for that :)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ihErHn_aJzDb"
      },
      "source": [
        "class ModelTester:\n",
        "  def __init__(self, model, optimiser = tf.keras.optimizers.SGD(), loss_object=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)):\n",
        "    self._model = model\n",
        "    self._optimiser = optimiser \n",
        "    self._loss_object = loss_object\n",
        "    self._train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
        "    self._train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')\n",
        "\n",
        "    self._test_loss = tf.keras.metrics.Mean(name='test_loss')\n",
        "    self._test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='test_accuracy')\n",
        "  \n",
        "  @tf.function\n",
        "  def train_step(self, images, labels):\n",
        "    with tf.GradientTape() as tape:\n",
        "      # training=True is only needed if there are layers with different\n",
        "      # behavior during training versus inference (e.g. Dropout).\n",
        "      predictions = self._model(images, training=True)\n",
        "      loss = self._loss_object(labels, predictions)\n",
        "    gradients = tape.gradient(loss, self._model.trainable_variables)\n",
        "    self._optimiser.apply_gradients(zip(gradients, self._model.trainable_variables))\n",
        "\n",
        "    self._train_loss(loss)\n",
        "    self._train_accuracy(labels, predictions) \n",
        "\n",
        "  @tf.function\n",
        "  def test_step(self, images, labels):\n",
        "    # training=False is only needed if there are layers with different\n",
        "    # behavior during training versus inference (e.g. Dropout).\n",
        "    predictions = self._model(images, training=False)\n",
        "    t_loss = self._loss_object(labels, predictions)\n",
        "\n",
        "    self._test_loss(t_loss)\n",
        "    self._test_accuracy(labels, predictions)\n",
        "\n",
        "  def train(self, train_ds, test_ds, epochs):\n",
        "    for epoch in range(epochs):\n",
        "      # Reset the metrics at the start of the next epoch\n",
        "      self._train_loss.reset_states()\n",
        "      self._train_accuracy.reset_states()\n",
        "      self._test_loss.reset_states()\n",
        "      self._test_accuracy.reset_states()\n",
        "\n",
        "      for images, labels in train_ds:\n",
        "        self.train_step(images, labels)\n",
        "\n",
        "      for test_images, test_labels in test_ds:\n",
        "        self.test_step(test_images, test_labels)\n",
        "\n",
        "      template = 'Epoch {}, Loss: {}, Accuracy: {}, Test Loss: {}, Test Accuracy: {}'\n",
        "      print(template.format(epoch + 1,\n",
        "                            self._train_loss.result(),\n",
        "                            self._train_accuracy.result() * 100,\n",
        "                            self._test_loss.result(),\n",
        "                            self._test_accuracy.result() * 100))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GGTYF2lJPBfE"
      },
      "source": [
        "tester = ModelTester(model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RKKn1Vq9PG19"
      },
      "source": [
        "tester.train(train_ds, test_ds, 5)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}