Skip to content

Instantly share code, notes, and snippets.

@bonthusaireddy

bonthusaireddy/transfer-learning-nlp-tf-hub.ipynb

Last active April 10, 2021 16:46
Show Gist options
  • Save bonthusaireddy/7e881834ca6f63f1ee97e9348e3ad930 to your computer and use it in GitHub Desktop.
Save bonthusaireddy/7e881834ca6f63f1ee97e9348e3ad930 to your computer and use it in GitHub Desktop.
Download ZIP
Transfer-Learning-NLP-TF-Hub.ipynb
Display the source blob
Display the rendered blob
Raw
transfer-learning-nlp-tf-hub.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Transfer-Learning-NLP-TF-Hub.ipynb",
"provenance": [],
"collapsed_sections": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/bonthusaireddy/7e881834ca6f63f1ee97e9348e3ad930/transfer-learning-nlp-tf-hub.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YOPcjnDDc5c7"
},
"source": [
"<h2 align=left> Transfer Learning for NLP with TensorFlow Hub</h2>\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bgBRL1nR_jdt"
},
"source": [
"This is a starter notebook for the guided project [Transfer Learning for NLP with TensorFlow Hub](https://www.coursera.org/projects/transfer-learning-nlp-tensorflow-hub/)\n",
"\n",
"A complete version of this notebook is available in the course resources.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ak95VdGNn3lJ"
},
"source": [
"### Overview\n",
"\n",
"[TensorFlow Hub](https://tfhub.dev/) is a repository of pre-trained TensorFlow models.\n",
"\n",
"In this project, you will use pre-trained models from TensorFlow Hub with [`tf.keras`](https://www.tensorflow.org/api_docs/python/tf/keras) for text classification. Transfer learning makes it possible to save training resources and to achieve good model generalization even when training on a small dataset. In this project, we will demonstrate this by training with several different TF-Hub modules."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cNNs4oBDoSf-"
},
"source": [
"### Learning Objectives"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MyoqkQFsoUzB"
},
"source": [
"By the time you complete this project, you will be able to:\n",
"\n",
"- Use various pre-trained NLP text embedding models from TensorFlow Hub\n",
"- Perform transfer learning to fine-tune models on your own text data\n",
"- Visualize model performance metrics with [TensorBoard](https://www.tensorflow.org/tensorboard)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n_BTLVEapAm0"
},
"source": [
"### Prerequisites"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-MAL1ZXnpCiF"
},
"source": [
"In order to be successful with this project, it is assumed you are:\n",
"\n",
"- Competent in the Python programming language\n",
"- Familiar with deep learning for Natural Language Processing (NLP)\n",
"- Familiar with TensorFlow, and its Keras API"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5d14nw7gpEth"
},
"source": [
"### Contents"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zGRlcriNpJGO"
},
"source": [
"This project/notebook consists of several Tasks.\n",
"\n",
"- **[Task 1]()**: Introduction to the Project.\n",
"- **[Task 2]()**: Setup your TensorFlow and Colab Runtime\n",
"- **[Task 3]()**: Download and Import the Quora Insincere Questions Dataset\n",
"- **[Task 4]()**: TensorFlow Hub for Natural Language Processing\n",
"- **[Task 5]()**: Define Function to Build and Compile Models\n",
"- **[Task 6]()**: Define Function to Build and Compile Models(Continued...)\n",
"- **[Task 7]()**: Train Various Text Classification Models\n",
"- **[Task 8]()**: Compare Accuracy and Loss Curves\n",
"- **[Task 9]()**: Fine-tuning Models from TF Hub\n",
"- **[Task 10]()**: Train Bigger Models and Visualize Metrics with TensorBoard"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kAxib00jpYbS"
},
"source": [
"## Task 2: Setup your TensorFlow and Colab Runtime."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GIPs2VnspY9l"
},
"source": [
"You will only be able to use the Colab Notebook after you save it to your Google Drive folder. Click on the File menu and select “Save a copy in Drive…\n",
"\n",
"![Copy to Drive](https://drive.google.com/uc?id=1CH3eDmuJL8WR0AP1r3UE6sOPuqq8_Wl7)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wcz2wdA_pez8"
},
"source": [
"### Check GPU Availability\n",
"\n",
"Check if your Colab notebook is configured to use Graphical Processing Units (GPUs). If zero GPUs are available, check if the Colab notebook is configured to use GPUs (Menu > Runtime > Change Runtime Type).\n",
"\n",
"![Hardware Accelerator Settings](https://drive.google.com/uc?id=1qrihuuMtvzXJHiRV8M7RngbxFYipXKQx)\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "PwWXcwno4wB9"
},
"source": [
"!nvidia-smi"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "YxB26qlms3qE"
},
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"import tensorflow as tf\n",
"import tensorflow_hub as hub\n",
"import tensorflow_datasets as tfds\n",
"\n",
"import matplotlib.pyplot as plt\n",
"plt.rcParams['figure.figsize'] = (12, 8)\n",
"from IPython import display\n",
"\n",
"import pathlib\n",
"import shutil\n",
"import tempfile\n",
"\n",
"!pip install -q git+https://github.com/tensorflow/docs\n",
"\n",
"import tensorflow_docs as tfdocs\n",
"import tensorflow_docs.modeling\n",
"import tensorflow_docs.plots\n",
"\n",
"print(\"Version: \", tf.__version__)\n",
"print(\"Hub version: \", hub.__version__)\n",
"print(\"GPU is\", \"available\" if tf.config.list_physical_devices('GPU') else \"NOT AVAILABLE\")\n",
"\n",
"logdir = pathlib.Path(tempfile.mkdtemp())/\"tensorboard_logs\"\n",
"shutil.rmtree(logdir, ignore_errors=True)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "J6qe4NYUp0vU"
},
"source": [
"## Task 3: Download and Import the Quora Insincere Questions Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dnRK0VlaDChm"
},
"source": [
"A downloadable copy of the [Quora Insincere Questions Classification data](https://www.kaggle.com/c/quora-insincere-questions-classification/data) can be found [https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip](https://archive.org/download/fine-tune-bert-tensorflow-train.csv/train.csv.zip). Decompress and read the data into a pandas DataFrame."
]
},
{
"cell_type": "code",
"metadata": {
"id": "hfF56xcgs8Eb"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "08kiCrj5tzcj"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "VfWdbhfYs8Ha"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "VX1UEsALs8J8"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "G_E6VWNTs8Mn"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "Hu6lFucLp4Bn"
},
"source": [
"## Task 4: TensorFlow Hub for Natural Language Processing"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2dDjc3ypg1uh"
},
"source": [
"Our text data consits of questions and corresponding labels.\n",
"\n",
"You can think of a question vector as a distributed representation of a question, and is computed for every question in the training set. The question vector along with the output label is then used to train the statistical classification model. \n",
"\n",
"The intuition is that the question vector captures the semantics of the question and, as a result, can be effectively used for classification. \n",
"\n",
"To obtain question vectors, we have two alternatives that have been used for several text classification problems in NLP: \n",
"* word-based representations and \n",
"* context-based representations"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PKLfQ-LAhdQZ"
},
"source": [
"#### Word-based Representations\n",
"\n",
"- A **word-based representation** of a question combines word embeddings of the content words in the question. We can use the average of the word embeddings of content words in the question. Average of word embeddings have been used for different NLP tasks.\n",
"- Examples of pre-trained embeddings include:\n",
" - **Word2Vec**: These are pre-trained embeddings of words learned from a large text corpora. Word2Vec has been pre-trained on a corpus of news articles with 300 million tokens, resulting in 300-dimensional vectors.\n",
" - **GloVe**: has been pre-trained on a corpus of tweets with 27 billion tokens, resulting in 200-dimensional vectors.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EoHvyDd7i7Hx"
},
"source": [
"#### Context-based Representations\n",
"\n",
"- **Context-based representations** may use language models to generate vectors of sentences. So, instead of learning vectors for individual words in the sentence, they compute a vector for sentences on the whole, by taking into account the order of words and the set of co-occurring words.\n",
"- Examples of deep contextualised vectors include:\n",
" - **Embeddings from Language Models (ELMo)**: uses character-based word representations and bidirectional LSTMs. The pre-trained model computes a contextualised vector of 1024 dimensions. ELMo is available on Tensorflow Hub.\n",
" - **Universal Sentence Encoder (USE)**: The encoder uses a Transformer architecture that uses attention mechanism to incorporate information about the order and the collection of words. The pre-trained model of USE that returns a vector of 512 dimensions is also available on Tensorflow Hub.\n",
" - **Neural-Net Language Model (NNLM)**: The model simultaneously learns representations of words and probability functions for word sequences, allowing it to capture semantics of a sentence. We will use a pretrained models available on Tensorflow Hub, that are trained on the English Google News 200B corpus, and computes a vector of 128 dimensions for the larger model and 50 dimensions for the smaller model.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3ZrM-BpDrxUG"
},
"source": [
"Tensorflow Hub provides a number of [modules](https://tfhub.dev/s?module-type=text-embedding&tf-version=tf2&q=tf2) to convert sentences into embeddings such as Universal sentence ecoders, NNLM, BERT and Wikiwords."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8TOn7rWN4BW-"
},
"source": [
"Transfer learning makes it possible to save training resources and to achieve good model generalization even when training on a small dataset. In this project, we will demonstrate this by training with several different TF-Hub modules."
]
},
{
"cell_type": "code",
"metadata": {
"id": "ePW3i1zos8PP"
},
"source": [
"module_url = \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\" #@param [\"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\", \"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\", \"https://tfhub.dev/google/tf2-preview/nnlm-en-dim128/1\", \"https://tfhub.dev/google/universal-sentence-encoder/4\", \"https://tfhub.dev/google/universal-sentence-encoder-large/5\"] {allow-input: true}"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "bTTELVrtqPwW"
},
"source": [
"## Tasks 5 & 6: Define Function to Build and Compile Models"
]
},
{
"cell_type": "code",
"metadata": {
"id": "B6di0eiFs8ch"
},
"source": [
"def train_and_evaluate_model(module_url, embed_size, name, trainable=False):\n",
" \n",
"\n",
" "
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "51YeToW1qpdi"
},
"source": [
"## Task 7: Train Various Text Classification Models"
]
},
{
"cell_type": "code",
"metadata": {
"id": "JmKhdRH1SsXG"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "GWfWIypew-lk"
},
"source": [
"module_url = \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\" #@param [\"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\", \"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\", \"https://tfhub.dev/google/tf2-preview/nnlm-en-dim128/1\", \"https://tfhub.dev/google/universal-sentence-encoder/4\", \"https://tfhub.dev/google/universal-sentence-encoder-large/5\"] {allow-input: true}"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "BJSxiRusq6CY"
},
"source": [
"## Task 8: Compare Accuracy and Loss Curves"
]
},
{
"cell_type": "code",
"metadata": {
"id": "taqgWQRHTfKq"
},
"source": [
"plt.rcParams['figure.figsize'] = (12, 8)\n",
"plotter = tfdocs.plots.HistoryPlotter(metric = 'accuracy')\n",
"plotter.plot(histories)\n",
"plt.xlabel(\"Epochs\")\n",
"plt.legend(bbox_to_anchor=(1.0, 1.0), loc='upper left')\n",
"plt.title(\"Accuracy Curves for Models\")\n",
"plt.show()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "cPxhmeAib7XP"
},
"source": [
"plotter = tfdocs.plots.HistoryPlotter(metric = 'loss')\n",
"plotter.plot(histories)\n",
"plt.xlabel(\"Epochs\")\n",
"plt.legend(bbox_to_anchor=(1.0, 1.0), loc='upper left')\n",
"plt.title(\"Loss Curves for Models\")\n",
"plt.show()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "54bNF4m-w08D"
},
"source": [
"## Task 9: Fine-tune Model from TF Hub"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TxNN-MzorJ9V"
},
"source": [
"## Task 10: Train Bigger Models and Visualize Metrics with TensorBoard"
]
},
{
"cell_type": "code",
"metadata": {
"id": "lG6fv6Mis8xd"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "4_Rwudlms80h"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "TFxWfePns83c"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "TsM0ZMt3s86C"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "MRO6Wkt4s889"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "jVdlhv3Ls8_6"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "7WavvT6vs9C3"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "DdL4FWe_s9Fj"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "7D9jHDdzs9IJ"
},
"source": [
""
],
"execution_count": null,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment