NLTK Excercises.ipynb

nltk-excercises.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLTK Excercises.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyM4xHbIhMAQes57clCuHzu9",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/samyumobi/56d020915b6dbd851e07a6b05940b2c2/nltk-excercises.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "What is Tokenize?\n",
        "\n",
        "Tokenization is the process of demarcating and possibly classifying sections of a string of input characters. The resulting tokens are then passed on to some other form of processing. The process can be considered a sub-task of parsing input.\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "bbhVHj3cbk34"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Write a Python NLTK program to split the text sentence/paragraph into a list of words.**\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "gKCkXjykbpQk"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qlmAWA8gbeaB",
        "outputId": "227b78c8-ac80-49a5-8e2d-c74d58ac8585"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original text \n",
            "Hello Deepa How are you? Kaise hai aaj ka mausam? Let's go on a date.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "text =  '''\n",
        "Hello Deepa How are you? Kaise hai aaj ka mausam? Let's go on a date.\n",
        "'''\n",
        "print('Original text',text)\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "from nltk.tokenize import sent_tokenize\n",
        "print([i for i in sent_tokenize(text)])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FythT7HLb_tI",
        "outputId": "f8e17df7-f301-47c0-ee94-47ddb5aadf8e"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['\\nHello Deepa How are you?', 'Kaise hai aaj ka mausam?', \"Let's go on a date.\"]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "l-NNKey0dosu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Write a Python NLTK program to tokenize sentences in languages other than English.**\n"
      ],
      "metadata": {
        "id": "GTlWQCsPclzz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = '''\n",
        "NLTK ist Open Source Software. Der Quellcode wird unter den Bedingungen der Apache License Version 2.0 vertrieben.  \n",
        "Die Dokumentation wird unter den Bedingungen der Creative Commons-Lizenz Namensnennung - Nicht kommerziell - Keine \n",
        "abgeleiteten Werke 3.0 in den Vereinigten Staaten verteilt.\n",
        "'''\n",
        "print(\"\\nOriginal string:\")\n",
        "print(text)\n",
        "from nltk.tokenize import sent_tokenize\n",
        "token_text = sent_tokenize(text, language='german')\n",
        "print(\"\\nSentence-tokenized copy in a list:\")\n",
        "print(token_text)\n",
        "print(\"\\nRead the list:\")\n",
        "for s in token_text:\n",
        "    print(s)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JyDVSL9EcsQ7",
        "outputId": "b111efa7-8f01-4a30-893b-7f2e5d0f3a3c"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Original string:\n",
            "\n",
            "NLTK ist Open Source Software. Der Quellcode wird unter den Bedingungen der Apache License Version 2.0 vertrieben.  \n",
            "Die Dokumentation wird unter den Bedingungen der Creative Commons-Lizenz Namensnennung - Nicht kommerziell - Keine \n",
            "abgeleiteten Werke 3.0 in den Vereinigten Staaten verteilt.\n",
            "\n",
            "\n",
            "Sentence-tokenized copy in a list:\n",
            "['\\nNLTK ist Open Source Software.', 'Der Quellcode wird unter den Bedingungen der Apache License Version 2.0 vertrieben.', 'Die Dokumentation wird unter den Bedingungen der Creative Commons-Lizenz Namensnennung - Nicht kommerziell - Keine \\nabgeleiteten Werke 3.0 in den Vereinigten Staaten verteilt.']\n",
            "\n",
            "Read the list:\n",
            "\n",
            "NLTK ist Open Source Software.\n",
            "Der Quellcode wird unter den Bedingungen der Apache License Version 2.0 vertrieben.\n",
            "Die Dokumentation wird unter den Bedingungen der Creative Commons-Lizenz Namensnennung - Nicht kommerziell - Keine \n",
            "abgeleiteten Werke 3.0 in den Vereinigten Staaten verteilt.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "WT7rLIYJdkRM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "** Write a Python NLTK program to create a list of words from a given string.**\n"
      ],
      "metadata": {
        "id": "_UBxvFlWdcmc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = ''' The majority of modern organisations collect large quantities of text data from their customers and \n",
        "business operations in the form of emails, chats, phone calls and many other interactions. \n",
        "This data can be invaluable in developing a deeper understanding of the problems that customers are facing, \n",
        "and insights from this data can help to solve some of these problems as well as automate and optimise customer facing business processes.'''\n",
        "\n",
        "## Technique1\n",
        "from nltk.tokenize import word_tokenize\n",
        "print(word_tokenize(text))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kpikKBHBdh2S",
        "outputId": "52569d53-c80e-4ce0-92e6-a54092005528"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['The', 'majority', 'of', 'modern', 'organisations', 'collect', 'large', 'quantities', 'of', 'text', 'data', 'from', 'their', 'customers', 'and', 'business', 'operations', 'in', 'the', 'form', 'of', 'emails', ',', 'chats', ',', 'phone', 'calls', 'and', 'many', 'other', 'interactions', '.', 'This', 'data', 'can', 'be', 'invaluable', 'in', 'developing', 'a', 'deeper', 'understanding', 'of', 'the', 'problems', 'that', 'customers', 'are', 'facing', ',', 'and', 'insights', 'from', 'this', 'data', 'can', 'help', 'to', 'solve', 'some', 'of', 'these', 'problems', 'as', 'well', 'as', 'automate', 'and', 'optimise', 'customer', 'facing', 'business', 'processes', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## Technique2\n",
        "print(list(text.split()))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TjWEVqjneKsP",
        "outputId": "ce793227-f2d7-4de2-e482-50ab78041689"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['The', 'majority', 'of', 'modern', 'organisations', 'collect', 'large', 'quantities', 'of', 'text', 'data', 'from', 'their', 'customers', 'and', 'business', 'operations', 'in', 'the', 'form', 'of', 'emails,', 'chats,', 'phone', 'calls', 'and', 'many', 'other', 'interactions.', 'This', 'data', 'can', 'be', 'invaluable', 'in', 'developing', 'a', 'deeper', 'understanding', 'of', 'the', 'problems', 'that', 'customers', 'are', 'facing,', 'and', 'insights', 'from', 'this', 'data', 'can', 'help', 'to', 'solve', 'some', 'of', 'these', 'problems', 'as', 'well', 'as', 'automate', 'and', 'optimise', 'customer', 'facing', 'business', 'processes.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## Technique3\n",
        "\n",
        "# Preserve punctuation marks\n",
        "\n",
        "import re\n",
        "words = re.compile(r'(\\w[\\w]*\\w|\\w)')\n",
        "print(words.findall(text))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8GuvlbD0eeat",
        "outputId": "e01073eb-c7c2-4a4d-9091-189b1eb36af8"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['The', 'majority', 'of', 'modern', 'organisations', 'collect', 'large', 'quantities', 'of', 'text', 'data', 'from', 'their', 'customers', 'and', 'business', 'operations', 'in', 'the', 'form', 'of', 'emails', 'chats', 'phone', 'calls', 'and', 'many', 'other', 'interactions', 'This', 'data', 'can', 'be', 'invaluable', 'in', 'developing', 'a', 'deeper', 'understanding', 'of', 'the', 'problems', 'that', 'customers', 'are', 'facing', 'and', 'insights', 'from', 'this', 'data', 'can', 'help', 'to', 'solve', 'some', 'of', 'these', 'problems', 'as', 'well', 'as', 'automate', 'and', 'optimise', 'customer', 'facing', 'business', 'processes']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "**Write a Python NLTK program to split all punctuation into separate tokens.**\n"
      ],
      "metadata": {
        "id": "RFMIKqEKi5qm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = '''Hello I'm a String !!!!!'''\n",
        "\n",
        "##technique1\n",
        "\n",
        "from nltk.tokenize import WordPunctTokenizer\n",
        "print(WordPunctTokenizer().tokenize(text))\n",
        "\n",
        "##technique2\n",
        "regex_pattern = r\"[\\w]+|[.,?/;:'!]\"\n",
        "print(re.findall(regex_pattern,text))\n",
        "\n",
        "###technique3\n",
        "l = re.split(\"(\\W+?)\",text)\n",
        "e = ['',' ']\n",
        "l2 = [j for j in l if j not in e]\n",
        "print(l,'\\n',l2)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ny_rEuyujDKR",
        "outputId": "444f82fd-cb03-40ed-c567-f2c2d92ff1e8"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Hello', 'I', \"'\", 'm', 'a', 'String', '!!!!!']\n",
            "['Hello', 'I', \"'\", 'm', 'a', 'String', '!', '!', '!', '!', '!']\n",
            "['Hello', ' ', 'I', \"'\", 'm', ' ', 'a', ' ', 'String', ' ', '', '!', '', '!', '', '!', '', '!', '', '!', ''] \n",
            " ['Hello', 'I', \"'\", 'm', 'a', 'String', '!', '!', '!', '!', '!']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Write a Python NLTK program to tokenize words, sentence wise.**"
      ],
      "metadata": {
        "id": "Az42wrJmn30W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = ''' Hi ! Busy people. Sam goes for a walk ### ,. She plays with her do%g !! Her dog#@love's ba@!rki$$ng'''\n",
        "\n",
        "from nltk.tokenize import word_tokenize, sent_tokenize\n",
        "# print(sent_tokenize(text))\n",
        "# print(word_tokenize(text))\n",
        "print([s for t in sent_tokenize(text) for s in word_tokenize(t) ])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0zCHJqn8n-XI",
        "outputId": "fde011c9-6460-40cd-9e2d-1634840daf18"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Hi', '!', 'Busy', 'people', '.', 'Sam', 'goes', 'for', 'a', 'walk', '#', '#', '#', ',', '.', 'She', 'plays', 'with', 'her', 'do', '%', 'g', '!', '!', 'Her', 'dog', '#', '@', 'love', \"'s\", 'ba', '@', '!', 'rki', '$', '$', 'ng']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " \n",
        "**Write a Python NLTK program to tokenize a twitter text.**"
      ],
      "metadata": {
        "id": "HsPxtMvFuHrA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.tokenize import TweetTokenizer\n",
        "tknzr = TweetTokenizer(strip_handles=True, reduce_len=True)\n",
        "tweet_text = \"NoSQL introduction - w3resource http://bit.ly/1ngHC5F  #nosql #database #webdev\"\n",
        "print(\"\\nOriginal Tweet:\")\n",
        "print(tweet_text)\n",
        "result = tknzr.tokenize(tweet_text)\n",
        "print(\"\\nTokenize a twitter text:\")\n",
        "print(result) "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ivyMzGH6ubvl",
        "outputId": "6c73a855-26f9-427d-9c96-f25ee731eaaf"
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Original Tweet:\n",
            "NoSQL introduction - w3resource http://bit.ly/1ngHC5F  #nosql #database #webdev\n",
            "\n",
            "Tokenize a twitter text:\n",
            "['NoSQL', 'introduction', '-', 'w3resource', 'http://bit.ly/1ngHC5F', '#nosql', '#database', '#webdev']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "**Write a Python NLTK program that will read a given text through each line and look for sentences. Print each sentence and divide two sentences with \"==============\".**"
      ],
      "metadata": {
        "id": "S1sFMEcXuiBH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = ''' Hi ! Busy people. Sam goes for a walk ### \n",
        ",. She plays with her do%g !! \n",
        "Her dog#@love's ba@!rki$$ng'''\n",
        "\n",
        "sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')\n",
        "print('\\n========\\n'.join(sent_detector.tokenize(text.strip())))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wuDePBVOuhcT",
        "outputId": "4fd0648d-d0f8-4c8c-cd8b-f9b260d0fbd4"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Hi !\n",
            "========\n",
            "Busy people.\n",
            "========\n",
            "Sam goes for a walk ### \n",
            ",.\n",
            "========\n",
            "She plays with her do%g !!\n",
            "========\n",
            "Her dog#@love's ba@!rki$$ng\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "**Find parenthesized expressions in a given string and divides into a sequence of substrings**"
      ],
      "metadata": {
        "id": "oxSUQJREwspb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.tokenize import SExprTokenizer\n",
        "text = '(a b (c d)) e f (g)'\n",
        "print(\"\\nOriginal Tweet:\")\n",
        "print(text)\n",
        "print(SExprTokenizer().tokenize(text))\n",
        "text = '(a b) (c d) e (f g)'\n",
        "print(\"\\nOriginal Tweet:\")\n",
        "print(text)\n",
        "print(SExprTokenizer().tokenize(text))\n",
        "text = '[(a b (c d)) e f (g)]'\n",
        "print(\"\\nOriginal Tweet:\")\n",
        "print(text)\n",
        "print(SExprTokenizer().tokenize(text))\n",
        "print(text)\n",
        "print(SExprTokenizer().tokenize(text))\n",
        "text = '{a b {c d}} e f {g}'\n",
        "print(\"\\nOriginal Tweet:\")\n",
        "print(text)\n",
        "print(SExprTokenizer().tokenize(text))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BOVgzgE2vsWy",
        "outputId": "555f6291-b02d-4e13-f319-9d3608c12507"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Original Tweet:\n",
            "(a b (c d)) e f (g)\n",
            "['(a b (c d))', 'e', 'f', '(g)']\n",
            "\n",
            "Original Tweet:\n",
            "(a b) (c d) e (f g)\n",
            "['(a b)', '(c d)', 'e', '(f g)']\n",
            "\n",
            "Original Tweet:\n",
            "[(a b (c d)) e f (g)]\n",
            "['[', '(a b (c d))', 'e', 'f', '(g)', ']']\n",
            "[(a b (c d)) e f (g)]\n",
            "['[', '(a b (c d))', 'e', 'f', '(g)', ']']\n",
            "\n",
            "Original Tweet:\n",
            "{a b {c d}} e f {g}\n",
            "['{a b {c d}} e f {g}']\n"
          ]
        }
      ]
    }
  ]
}