Creating a First Look at Your Text Collection.ipynb

{
 "worksheets": [
  {
   "cells": [
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Goal** In this assignment, you'll make a first pass look at your newly adopted text collection similar to the Wolfram Alpha's view.\n\n"
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Title, author, and other metadata**.  First, print out some summary information that gives the background explaining what this collection is and where it comes from:"
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "This collection of text contains Shakespeare's \"MacBeth\". It came from Project Gutenberg. "
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "import nltk\nfrom nltk import corpus\nimport re\nsent_tokenizer=nltk.data.load('tokenizers/punkt/english.pickle')",
     "prompt_number": 82,
     "outputs": [],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**First, load in the file or files below.**  First, take a look at your text.  An easy way to get started is to first read it in, and then run it through the sentence tokenizer to divide it up, even if this division is not fully accurate.  You may have to do a bit of work to figure out which will be the \"opening phrase\" that Wolfram Alpha shows.  Below, write the code to read in the text and split it into sentences, and then print out the **opening phrase**."
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "mb = nltk.corpus.gutenberg.raw('shakespeare-macbeth.txt')\nmb_tokenized = sent_tokenizer.tokenize(mb)\nmb_tokenized[5]",
     "prompt_number": 83,
     "outputs": [
      {
       "text": "'When shall we three meet againe?'",
       "output_type": "pyout",
       "metadata": {},
       "prompt_number": 83
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Next, tokenize.**  Look at the several dozen sentences to see what kind of tokenization issues you'll have.  Write a regular expression tokenizer, using the nltk.regexp_tokenize() as seen in class, to do a nice job of breaking your text up into words.  You may need to make changes to the regex pattern that is given in the book to make it work well for your text collection. \n\n*Note that this is the key part of the assignment.  How you break up the words will have effects down the line for how you can manipulate your text collection.  You may want to refine this code later.*"
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "mb_words = nltk.corpus.gutenberg.words('shakespeare-macbeth.txt')\nmb_sents = nltk.corpus.gutenberg.sents('shakespeare-macbeth.txt')\nmb_words_nopunc = [word for word in mb_words if word.isalnum()]\n\n# I tried tokenizing with regex, but since this text is part of the nltk gutenberg corpus, it was much cleaner to use\n# the .words and .sents methods \n\n\n# pattern = r'''(?x)    # set flag to allow verbose regexps\n# ...     ([A-Z]\\.)+        # abbreviations, e.g. U.S.A.\n# ...   | \\w+([-']\\w+)*        # words with optional internal hyphens\n# ...   | \\$?\\d+(\\.\\d+)?%?  # currency and percentages, e.g. $12.40, 82%\n# ...   | \\.\\.\\.            # ellipsis\n# ...   | [.,;\"'?():-_`]+  # these are separate tokens\n# ... '''\n\n# print nltk.regexp_tokenize(mb,pattern)[0:50]",
     "prompt_number": 84,
     "outputs": [],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Compute word counts.** Now compute your frequency distribution using a FreqDist over the words. Let's not do lowercasing or stemming yet.  You can run this over the whole collection together, or sentence by sentence. Write the code for computing the FreqDist below."
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "fd = nltk.FreqDist(mb_words_nopunc)",
     "prompt_number": 85,
     "outputs": [],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Creating a table.**\nPython provides an easy way to line columns up in a table.  You can specify a width for a string such as %6s, producing a string that is padded to width 6. It is right-justified by default, but a minus sign in front of it switches it to left-justified, so -3d% means left justify an integer with width 3.  *AND* if you don't know the width in advance, you can make it a variable by using an asterisk rather than a number before the '\\*s%' or the '-\\*d%'.  Check out this example (this is just fyi):"
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "print '%-16s' % 'Info type', '%-16s' % 'Value'\nprint '%-16s' % 'number of words', '%-16d' % 100000\n",
     "prompt_number": 86,
     "outputs": [
      {
       "output_type": "stream",
       "text": "Info type        Value           \nnumber of words  100000          \n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Word Properties Table** Next there is a table of word properties, which you should compute (skip unique word stems, since we will do stemming in class on Wed).  Make a table that prints out:\n1. number of words\n2. number of unique words\n3. average word length\n4. longest word\n\nYou can make your table look prettier than the example I showed above if you like!\n\nYou can decide for yourself if you want to eliminate punctuation and function words (stop words) or not.  It's your collection!  \n"
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "avg_word_len = (float(len(\"\".join(mb_words_nopunc)))) / float(len(mb_words_nopunc))\nlongest_words = [w for w in set(mb_words_nopunc) if len(w) >= (max(len(w) for w in mb_words_nopunc))]\n\nprint '%-16s' % 'Info type', '%-16s' % 'Value'\nprint '%-16s' % 'number of words', '%-16d' % len(mb_words_nopunc)\nprint '%-16s' % 'avg word len', '%-16f' % avg_word_len\nprint '%-16s' % 'longest word', '%-16s' % longest_words[0]",
     "prompt_number": 87,
     "outputs": [
      {
       "output_type": "stream",
       "text": "Info type        Value           \nnumber of words  18351           \navg word len     4.108713        \nlongest word     Voluptuousnesse \n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Most Frequent Words List.** Next is the most frequent words list.  This table shows the percent of the total as well as the most frequent words, so compute this number as well.  "
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "print '%-16s' % 'Word', '%-16s' % 'Frequency', '%-16s' % '% of total' \nfor i in range(30):\n    print '%-16s' % fd.keys()[i], '%-16d' % fd[fd.keys()[i]], '%-16f' % (float(fd[fd.keys()[i]]) * 100 / len(fd.keys()))",
     "prompt_number": 91,
     "outputs": [
      {
       "output_type": "stream",
       "text": "Word             Frequency        % of total      \nthe              531              13.261738       \nand              376              9.390609        \nI                333              8.316683        \nof               315              7.867133        \nto               311              7.767233        \nd                224              5.594406        \na                214              5.344655        \nyou              184              4.595405        \nin               173              4.320679        \nAnd              170              4.245754        \nmy               170              4.245754        \nis               166              4.145854        \nthat             158              3.946054        \nnot              155              3.871129        \nit               138              3.446553        \nMacb             137              3.421578        \nwith             134              3.346653        \ns                131              3.271728        \nhis              129              3.221778        \nbe               124              3.096903        \nThe              118              2.947053        \nhaue             117              2.922078        \nme               111              2.772228        \nyour             110              2.747253        \nour              103              2.572428        \nhim             ",
       "stream": "stdout"
      },
      {
       "output_type": "stream",
       "text": " 90               2.247752        \nfor              82               2.047952        \nEnter            80               1.998002        \nThat             80               1.998002        \nthis             79               1.973027        \n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Most Frequent Capitalized Words List** We haven't lower-cased the text so you should be able to compute this. Don't worry about whether capitalization comes from proper nouns, start of sentences, or elsewhere. You need to make a different FreqDist to do this one.  Write the code here for the new FreqDist and the List itself.  Show the list here."
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "caps = [word for word in mb_words_nopunc if word[0].isupper()]\nfd_caps = nltk.FreqDist(caps)\nfd_caps.items()[0:30]",
     "prompt_number": 89,
     "outputs": [
      {
       "text": "[('I', 333),\n ('And', 170),\n ('Macb', 137),\n ('The', 118),\n ('Enter', 80),\n ('That', 80),\n ('What', 74),\n ('To', 73),\n ('Macbeth', 61),\n ('But', 60),\n ('Macd', 58),\n ('King', 55),\n ('Rosse', 49),\n ('Lady', 48),\n ('Banquo', 39),\n ('Lord', 38),\n ('All', 36),\n ('He', 36),\n ('Which', 36),\n ('Ile', 35),\n ('As', 34),\n ('My', 33),\n ('Wife', 31),\n ('Exeunt', 30),\n ('For', 30),\n ('If', 28),\n ('In', 28),\n ('Lenox', 28),\n ('A', 27),\n ('Then', 27)]",
       "output_type": "pyout",
       "metadata": {},
       "prompt_number": 89
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    },
    {
     "metadata": {},
     "cell_type": "markdown",
     "source": "**Sentence Properties Table** This summarizes number of sentences and average sentence length in words and characters (you decide if you want to include stopwords/punctuation or not).  Print those out in a table here."
    },
    {
     "metadata": {},
     "cell_type": "code",
     "input": "print '%-25s' % '# of sentences', '%-16d' % len(mb_sents)\nprint '%-25s' % 'avg sent length in words', '%-16f' % (float(len(mb_words_nopunc)) / float(len(mb_sents)))\nprint '%-25s' % 'avg sent length in chars', '%-16f' % (float(len(\"\".join(mb_words_nopunc))) / float(len(mb_sents)))",
     "prompt_number": 90,
     "outputs": [
      {
       "output_type": "stream",
       "text": "# of sentences            ",
       "stream": "stdout"
      },
      {
       "output_type": "stream",
       "text": "1907            \navg sent length in words  9.622968        \navg sent length in chars  39.538018       \n",
       "stream": "stdout"
      }
     ],
     "language": "python",
     "trusted": true,
     "collapsed": false
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "name": "",
  "signature": "sha256:38301798e5d190f9c710be9f33b1d2206631a1ada117ad257ee5c40d285a4d1f"
 },
 "nbformat": 3
}