lirnli · December 4, 2018 05:18
diff --git a/Coursera Applied Text Mining in Python Assignment1.ipynb b/Coursera Applied Text Mining in Python Assignment1.ipynb
 {
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.1** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 1\n",
    "\n",
    "In this assignment, you'll be working with messy medical data and using regex to extract relevant infromation from the data. \n",
    "\n",
    "Each line of the `dates.txt` file corresponds to a medical note. Each note has a date that needs to be extracted, but each date is encoded in one of many formats.\n",
    "\n",
    "The goal of this assignment is to correctly identify all of the different date variants encoded in this dataset and to properly normalize and sort the dates. \n",
    "\n",
    "Here is a list of some of the variants you might encounter in this dataset:\n",
    "* 04/20/2009; 04/20/09; 4/20/09; 4/3/09\n",
    "* Mar-20-2009; Mar 20, 2009; March 20, 2009;  Mar. 20, 2009; Mar 20 2009;\n",
    "* 20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009\n",
    "* Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009\n",
    "* Feb 2009; Sep 2009; Oct 2010\n",
    "* 6/2008; 12/2009\n",
    "* 2009; 2010\n",
    "\n",
    "Once you have extracted these date patterns from the text, the next step is to sort them in ascending chronological order accoring to the following rules:\n",
    "* Assume all dates in xx/xx/xx format are mm/dd/yy\n",
    "* Assume all dates where year is encoded in only two digits are years from the 1900's (e.g. 1/5/89 is January 5th, 1989)\n",
    "* If the day is missing (e.g. 9/2009), assume it is the first day of the month (e.g. September 1, 2009).\n",
    "* If the month is missing (e.g. 2010), assume it is the first of January of that year (e.g. January 1, 2010).\n",
    "* Watch out for potential typos as this is a raw, real-life derived dataset.\n",
    "\n",
    "With these rules in mind, find the correct date in each note and return a pandas Series in chronological order of the original Series' indices.\n",
    "\n",
    "For example if the original series was this:\n",
    "\n",
    "    0    1999\n",
    "    1    2010\n",
    "    2    1978\n",
    "    3    2015\n",
    "    4    1985\n",
    "\n",
    "Your function should return this:\n",
    "\n",
    "    0    2\n",
    "    1    4\n",
    "    2    0\n",
    "    3    1\n",
    "    4    3\n",
    "\n",
    "Your score will be calculated using [Kendall's tau](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient), a correlation measure for ordinal data.\n",
    "\n",
    "*This function should return a Series of length 500 and dtype int.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                        03/25/93 Total time of visit (in minutes):\\n\n",
       "1                                      6/18/85 Primary Care Doctor:\\n\n",
       "2            sshe plans to move as of 7/8/71 In-Home Services: None\\n\n",
       "3                               7 on 9/27/75 Audit C Score Current:\\n\n",
       "4    2/6/96 sleep studyPain Treatment Pain Level (Numeric Scale): 7\\n\n",
       "dtype: object"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from functools import reduce\n",
    "pd.options.display.max_colwidth=1000\n",
    "pd.options.display.max_rows = 500\n",
    "\n",
    "doc = []\n",
    "with open('dates.txt') as file:\n",
    "    for line in file:\n",
    "        doc.append(line)\n",
    "\n",
    "df = pd.Series(doc)\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "04/20/2009; 04/20/09; 4/20/09; 4/3/09\n",
      "Mar-20-2009; Mar 20, 2009; March 20, 2009; Mar. 20, 2009; Mar 20 2009;\n",
      "20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009\n",
      "Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009\n",
      "Feb 2009; Sep 2009; Oct 2010\n",
      "6/2008; 12/2009\n",
      "2009; 2010\n",
      "\n",
      "rulex\n",
      "[('', '9', '1984')]\n",
      "rule2\n",
      "[('mar', '20', '2009'), ('mar', '20', '2009'), ('march', '20', '2009'), ('mar', '20', '2009'), ('mar', '20', '2009')]\n",
      "rule3\n",
      "[('20', 'mar', '2009'), ('20', 'march', '2009'), ('20', 'mar', '2009'), ('20', 'march', '2009')]\n",
      "rule4\n",
      "[('mar', '20', '2009'), ('mar', '21', '2009'), ('mar', '22', '2009')]\n",
      "rule5\n",
      "[('', 'feb', '2009'), ('', 'sep', '2009'), ('', 'oct', '2010')]\n",
      "rule6\n",
      "[('', '6', '2008'), ('', '12', '2009')]\n",
      "rule7\n",
      "[('', '', '2009'), ('', '', '2010')]\n"
     ]
    }
   ],
   "source": [
    "#play ground\n",
    "\n",
    "rule1 = r'(?P<month>\\d{1,2})[/-](?P<day>\\d{1,2})[/-](?P<year>\\d{2}|\\d{4})'\n",
    "rule2 = r'(?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\\w*)[.]?[ -](?P<day>\\d{1,2})(?:\\w{2})?[,]?[ -](?P<year>\\d{4})'\n",
    "rule3 = r'(?P<day>\\d{1,2}) (?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\\w*)[.,]? (?P<year>\\d{4})'\n",
    "rule4 = rule2\n",
    "rule5 = r'(?P<day>)(?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\\w*) (?P<year>\\d{4})'\n",
    "rule6 = r'(?P<day>)(?P<month>\\d{1,2})/(?P<year>\\d{4})'\n",
    "rule7 = r'(?P<day>)(?P<month>)(?:\\D|^)(?P<year>\\d{4})(?:\\D|$)'\n",
    "\n",
    "text = '''\n",
    "04/20/2009; 04/20/09; 4/20/09; 4/3/09\n",
    "Mar-20-2009; Mar 20, 2009; March 20, 2009; Mar. 20, 2009; Mar 20 2009;\n",
    "20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009\n",
    "Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009\n",
    "Feb 2009; Sep 2009; Oct 2010\n",
    "6/2008; 12/2009\n",
    "2009; 2010\n",
    "'''\n",
    "import re\n",
    "print (text)\n",
    "\n",
    "print ('rulex')\n",
    "print (re.findall(rule6, \"\"\"sChesterfield 9/1984 for 3 weeks for dual diagnosis alcohol and PTSDHx of Outpatient Treatment: Yes\\\n",
    "\"\"\".lower()))\n",
    "\n",
    "print ('rule2')\n",
    "print (re.findall(rule2, 'Mar-20-2009; Mar 20, 2009; March 20, 2009; Mar. 20, 2009; Mar 20 2009'.lower()))\n",
    "\n",
    "print ('rule3')\n",
    "print (re.findall(rule3, '20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009'.lower()))\n",
    "\n",
    "print ('rule4')\n",
    "print (re.findall(rule4, 'Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009'.lower()))\n",
    "\n",
    "print ('rule5')\n",
    "print (re.findall(rule5, 'Feb 2009; Sep 2009; Oct 2010'.lower()))\n",
    "\n",
    "print ('rule6')\n",
    "print (re.findall(rule6, '6/2008; 12/2009'.lower()))\n",
    "\n",
    "print ('rule7')\n",
    "print (re.findall(rule7, '2009; 2010'.lower()))\n",
    "# pd.DataFrame(\n",
    "#    df.str.extract(rule2),df\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def date_sorter(df):\n",
    "    \n",
    "    # Your code here\n",
    "    mon2int = dict([(m, i+1) for i, m in enumerate('jan,feb,mar,apr,may,jun,jul,aug,sep,oct,nov,dec'.split(\",\"))])\n",
    "    rule1 = r'(?P<month>\\d{1,2})[/-](?P<day>\\d{1,2})[/-](?P<year>\\d{2}|\\d{4})'\n",
    "    rule2 = r'(?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\\w*)[.]?[ -](?P<day>\\d{1,2})(?:\\w{2})?[,]?[ -](?P<year>\\d{4})'\n",
    "    rule3 = r'(?P<day>\\d{1,2}) (?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\\w*)[.,]? (?P<year>\\d{4})'\n",
    "    rule4 = rule2\n",
    "    rule5 = r'(?P<day>)(?P<month>(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\\w*) (?P<year>\\d{4})'\n",
    "    rule6 = r'(?P<day>)(?P<month>\\d{1,2})/(?P<year>\\d{4})'\n",
    "    rule7 = r'(?P<day>)(?P<month>)(?:\\D|^)(?P<year>\\d{4})(?:\\D|$)'\n",
    "    rule_list = [rule1, rule2, rule3, rule4, rule5, rule6, rule7]\n",
    "    extract_df_list = [df.str.lower().str.extract(rule) for rule in rule_list]\n",
    "    date_df = reduce(lambda x,y: x.fillna(y),extract_df_list)\n",
    "    date_df['text'] = df\n",
    "    date_df['month'] = date_df['month'].apply(lambda x: mon2int.get(str(x)[:3],x))\n",
    "    date_df['month'] = date_df['month'].fillna('1').apply(lambda x: '1' if x=='' else x)\n",
    "    date_df['year'] = date_df['year'].apply(lambda x: '19'+x if len(x)==2 else x)\n",
    "    date_df['day'] = date_df['day'].fillna('1').apply(lambda x: '1' if x=='' else x)\n",
    "    date_df = date_df[[\"year\",\"month\",\"day\"]].astype(int)\n",
    "    index_df = date_df.sort_values(['year','month','day'], axis=0, ascending=False).reset_index()\n",
    "    index_df = index_df['index']\n",
    "    return index_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel/__main__.py:13: FutureWarning: currently extract(expand=None) means expand=False (return Index/Series/DataFrame) but in a future version of pandas this will be changed to expand=True (return DataFrame)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0      413\n",
       "1      161\n",
       "2      186\n",
       "3      141\n",
       "4      231\n",
       "5      427\n",
       "6      253\n",
       "7      464\n",
       "8      235\n",
       "9      152\n",
       "10     257\n",
       "11     475\n",
       "12     401\n",
       "13     255\n",
       "14     439\n",
       "15     366\n",
       "16     381\n",
       "17     463\n",
       "18     198\n",
       "19     279\n",
       "20     431\n",
       "21     244\n",
       "22     286\n",
       "23     480\n",
       "24     383\n",
       "25     320\n",
       "26     139\n",
       "27     208\n",
       "28     243\n",
       "29     220\n",
       "30     282\n",
       "31     393\n",
       "32     460\n",
       "33     290\n",
       "34     227\n",
       "35     217\n",
       "36     229\n",
       "37     316\n",
       "38     170\n",
       "39     240\n",
       "40     467\n",
       "41     242\n",
       "42     364\n",
       "43     408\n",
       "44     411\n",
       "45     176\n",
       "46     448\n",
       "47     287\n",
       "48     273\n",
       "49     472\n",
       "50     268\n",
       "51     288\n",
       "52     379\n",
       "53     386\n",
       "54     389\n",
       "55     297\n",
       "56     371\n",
       "57     491\n",
       "58     360\n",
       "59     400\n",
       "60     293\n",
       "61     271\n",
       "62     396\n",
       "63     183\n",
       "64     187\n",
       "65     445\n",
       "66     497\n",
       "67     337\n",
       "68     428\n",
       "69     132\n",
       "70     341\n",
       "71     160\n",
       "72     289\n",
       "73     264\n",
       "74     325\n",
       "75     277\n",
       "76     391\n",
       "77     252\n",
       "78     314\n",
       "79     410\n",
       "80     490\n",
       "81     173\n",
       "82     307\n",
       "83     433\n",
       "84     270\n",
       "85     496\n",
       "86     348\n",
       "87     346\n",
       "88     344\n",
       "89     250\n",
       "90     339\n",
       "91     498\n",
       "92     451\n",
       "93     146\n",
       "94     302\n",
       "95     126\n",
       "96     207\n",
       "97     281\n",
       "98     425\n",
       "99     241\n",
       "100    350\n",
       "101    414\n",
       "102    188\n",
       "103    291\n",
       "104    306\n",
       "105    484\n",
       "106    144\n",
       "107    356\n",
       "108    362\n",
       "109    234\n",
       "110    452\n",
       "111    469\n",
       "112    184\n",
       "113    449\n",
       "114    128\n",
       "115    262\n",
       "116    446\n",
       "117    304\n",
       "118    494\n",
       "119    394\n",
       "120    367\n",
       "121    377\n",
       "122    147\n",
       "123    195\n",
       "124    159\n",
       "125    125\n",
       "126    328\n",
       "127    457\n",
       "128    374\n",
       "129    443\n",
       "130    158\n",
       "131    190\n",
       "132    417\n",
       "133    392\n",
       "134    420\n",
       "135    260\n",
       "136    305\n",
       "137    329\n",
       "138    456\n",
       "139    201\n",
       "140    145\n",
       "141    266\n",
       "142    407\n",
       "143    212\n",
       "144    384\n",
       "145    327\n",
       "146    432\n",
       "147    376\n",
       "148    321\n",
       "149    471\n",
       "150    120\n",
       "151    251\n",
       "152    387\n",
       "153     79\n",
       "154     12\n",
       "155    224\n",
       "156     20\n",
       "157    343\n",
       "158    222\n",
       "159     18\n",
       "160    338\n",
       "161    459\n",
       "162    114\n",
       "163    333\n",
       "164    101\n",
       "165    353\n",
       "166     87\n",
       "167    213\n",
       "168    365\n",
       "169    468\n",
       "170    166\n",
       "171    131\n",
       "172    121\n",
       "173    324\n",
       "174    124\n",
       "175      4\n",
       "176    142\n",
       "177    479\n",
       "178    292\n",
       "179    169\n",
       "180     90\n",
       "181    326\n",
       "182    203\n",
       "183    181\n",
       "184     60\n",
       "185    248\n",
       "186    110\n",
       "187    236\n",
       "188    372\n",
       "189     56\n",
       "190    388\n",
       "191    140\n",
       "192    193\n",
       "193    238\n",
       "194    311\n",
       "195    483\n",
       "196    406\n",
       "197    409\n",
       "198     42\n",
       "199    359\n",
       "200    133\n",
       "201     47\n",
       "202    196\n",
       "203    308\n",
       "204    163\n",
       "205    300\n",
       "206    450\n",
       "207    477\n",
       "208    331\n",
       "209    357\n",
       "210    172\n",
       "211     86\n",
       "212    192\n",
       "213    249\n",
       "214      0\n",
       "215    354\n",
       "216    272\n",
       "217    298\n",
       "218    478\n",
       "219    113\n",
       "220    301\n",
       "221    143\n",
       "222    310\n",
       "223    442\n",
       "224    100\n",
       "225    119\n",
       "226    269\n",
       "227     52\n",
       "228    211\n",
       "229     62\n",
       "230    138\n",
       "231    399\n",
       "232    487\n",
       "233     66\n",
       "234    322\n",
       "235     46\n",
       "236     67\n",
       "237    453\n",
       "238     33\n",
       "239     51\n",
       "240     91\n",
       "241     97\n",
       "242    226\n",
       "243    461\n",
       "244    122\n",
       "245    245\n",
       "246    206\n",
       "247    151\n",
       "248    194\n",
       "249     22\n",
       "250    412\n",
       "251    157\n",
       "252    177\n",
       "253    233\n",
       "254    482\n",
       "255    435\n",
       "256      7\n",
       "257     26\n",
       "258    149\n",
       "259     89\n",
       "260    210\n",
       "261    373\n",
       "262    385\n",
       "263    440\n",
       "264    202\n",
       "265    218\n",
       "266     71\n",
       "267    115\n",
       "268    265\n",
       "269    312\n",
       "270    476\n",
       "271    180\n",
       "272    174\n",
       "273    216\n",
       "274    256\n",
       "275    390\n",
       "276     37\n",
       "277    330\n",
       "278    462\n",
       "279     76\n",
       "280    116\n",
       "281    130\n",
       "282    416\n",
       "283     63\n",
       "284     41\n",
       "285    349\n",
       "286     32\n",
       "287     68\n",
       "288    284\n",
       "289    485\n",
       "290     54\n",
       "291    423\n",
       "292    437\n",
       "293     29\n",
       "294    168\n",
       "295    261\n",
       "296    404\n",
       "297    368\n",
       "298     96\n",
       "299    334\n",
       "300    230\n",
       "301    276\n",
       "302      8\n",
       "303    352\n",
       "304    199\n",
       "305    492\n",
       "306    135\n",
       "307    228\n",
       "308    185\n",
       "309    280\n",
       "310    447\n",
       "311      1\n",
       "312    178\n",
       "313     10\n",
       "314    274\n",
       "315    136\n",
       "316    489\n",
       "317    426\n",
       "318     48\n",
       "319    275\n",
       "320    421\n",
       "321    455\n",
       "322     99\n",
       "323    175\n",
       "324    285\n",
       "325    107\n",
       "326     61\n",
       "327    209\n",
       "328    247\n",
       "329    438\n",
       "330     35\n",
       "331    295\n",
       "332    137\n",
       "333    294\n",
       "334    397\n",
       "335    205\n",
       "336    358\n",
       "337    470\n",
       "338    179\n",
       "339     88\n",
       "340    429\n",
       "341    112\n",
       "342    103\n",
       "343     44\n",
       "344     70\n",
       "345    454\n",
       "346    127\n",
       "347     25\n",
       "348     16\n",
       "349    458\n",
       "350    403\n",
       "351     74\n",
       "352    263\n",
       "353    215\n",
       "354    246\n",
       "355    444\n",
       "356     80\n",
       "357    430\n",
       "358    355\n",
       "359    197\n",
       "360     39\n",
       "361    134\n",
       "362    466\n",
       "363    221\n",
       "364    267\n",
       "365    361\n",
       "366    441\n",
       "367    340\n",
       "368     78\n",
       "369    150\n",
       "370    347\n",
       "371    499\n",
       "372    259\n",
       "373     21\n",
       "374    167\n",
       "375     75\n",
       "376    254\n",
       "377    296\n",
       "378      5\n",
       "379    398\n",
       "380    424\n",
       "381    495\n",
       "382    313\n",
       "383    378\n",
       "384    164\n",
       "385    434\n",
       "386     65\n",
       "387     81\n",
       "388    200\n",
       "389      6\n",
       "390    336\n",
       "391    105\n",
       "392    148\n",
       "393    239\n",
       "394    318\n",
       "395    369\n",
       "396    493\n",
       "397    189\n",
       "398     72\n",
       "399    232\n",
       "400    117\n",
       "401     19\n",
       "402    123\n",
       "403    419\n",
       "404    309\n",
       "405    283\n",
       "406    395\n",
       "407    303\n",
       "408    488\n",
       "409     93\n",
       "410     27\n",
       "411    315\n",
       "412    258\n",
       "413    204\n",
       "414    342\n",
       "415     23\n",
       "416    237\n",
       "417    465\n",
       "418    219\n",
       "419    363\n",
       "420     50\n",
       "421      3\n",
       "422    370\n",
       "423    382\n",
       "424    165\n",
       "425    418\n",
       "426     40\n",
       "427    319\n",
       "428     11\n",
       "429     49\n",
       "430    317\n",
       "431    473\n",
       "432    223\n",
       "433    155\n",
       "434    214\n",
       "435    278\n",
       "436    351\n",
       "437    182\n",
       "438    332\n",
       "439    156\n",
       "440    108\n",
       "441     73\n",
       "442    402\n",
       "443    154\n",
       "444    162\n",
       "445    299\n",
       "446    104\n",
       "447    436\n",
       "448    481\n",
       "449     57\n",
       "450    345\n",
       "451    380\n",
       "452    375\n",
       "453    422\n",
       "454    323\n",
       "455    405\n",
       "456     36\n",
       "457    335\n",
       "458    415\n",
       "459    486\n",
       "460    191\n",
       "461    171\n",
       "462     31\n",
       "463    225\n",
       "464    111\n",
       "465     98\n",
       "466    129\n",
       "467     13\n",
       "468    153\n",
       "469    474\n",
       "470     28\n",
       "471     53\n",
       "472      2\n",
       "473     84\n",
       "474      9\n",
       "475     58\n",
       "476     83\n",
       "477    109\n",
       "478    102\n",
       "479     34\n",
       "480    118\n",
       "481     43\n",
       "482     92\n",
       "483    106\n",
       "484     15\n",
       "485     94\n",
       "486     69\n",
       "487     17\n",
       "488     55\n",
       "489     59\n",
       "490     85\n",
       "491     64\n",
       "492     38\n",
       "493     24\n",
       "494     82\n",
       "495     14\n",
       "496     95\n",
       "497     30\n",
       "498     45\n",
       "499     77\n",
       "Name: index, dtype: int64"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_sorter(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "LvcWI",
   "launcher_item_id": "krne9",
   "part_id": "Mkp1I"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
 }
	{
	"cells": [
	{
	"cell_type": "markdown",
	"metadata": {},
	"source": [
	"---\n",
	"\n",
	"_You are currently looking at version 1.1 of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
	"\n",
	"---"
	]
	},
	{
	"cell_type": "markdown",
	"metadata": {},
	"source": [
	"# Assignment 1\n",
	"\n",
	"In this assignment, you'll be working with messy medical data and using regex to extract relevant infromation from the data. \n",
	"\n",
	"Each line of the `dates.txt` file corresponds to a medical note. Each note has a date that needs to be extracted, but each date is encoded in one of many formats.\n",
	"\n",
	"The goal of this assignment is to correctly identify all of the different date variants encoded in this dataset and to properly normalize and sort the dates. \n",
	"\n",
	"Here is a list of some of the variants you might encounter in this dataset:\n",
	"* 04/20/2009; 04/20/09; 4/20/09; 4/3/09\n",
	"* Mar-20-2009; Mar 20, 2009; March 20, 2009; Mar. 20, 2009; Mar 20 2009;\n",
	"* 20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009\n",
	"* Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009\n",
	"* Feb 2009; Sep 2009; Oct 2010\n",
	"* 6/2008; 12/2009\n",
	"* 2009; 2010\n",
	"\n",
	"Once you have extracted these date patterns from the text, the next step is to sort them in ascending chronological order accoring to the following rules:\n",
	"* Assume all dates in xx/xx/xx format are mm/dd/yy\n",
	"* Assume all dates where year is encoded in only two digits are years from the 1900's (e.g. 1/5/89 is January 5th, 1989)\n",
	"* If the day is missing (e.g. 9/2009), assume it is the first day of the month (e.g. September 1, 2009).\n",
	"* If the month is missing (e.g. 2010), assume it is the first of January of that year (e.g. January 1, 2010).\n",
	"* Watch out for potential typos as this is a raw, real-life derived dataset.\n",
	"\n",
	"With these rules in mind, find the correct date in each note and return a pandas Series in chronological order of the original Series' indices.\n",
	"\n",
	"For example if the original series was this:\n",
	"\n",
	" 0 1999\n",
	" 1 2010\n",
	" 2 1978\n",
	" 3 2015\n",
	" 4 1985\n",
	"\n",
	"Your function should return this:\n",
	"\n",
	" 0 2\n",
	" 1 4\n",
	" 2 0\n",
	" 3 1\n",
	" 4 3\n",
	"\n",
	"Your score will be calculated using [Kendall's tau](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient), a correlation measure for ordinal data.\n",
	"\n",
	"This function should return a Series of length 500 and dtype int."
	]
	},
	{
	"cell_type": "code",
	"execution_count": 113,
	"metadata": {},
	"outputs": [
	{
	"data": {
	"text/plain": [
	"0 03/25/93 Total time of visit (in minutes):\\n\n",
	"1 6/18/85 Primary Care Doctor:\\n\n",
	"2 sshe plans to move as of 7/8/71 In-Home Services: None\\n\n",
	"3 7 on 9/27/75 Audit C Score Current:\\n\n",
	"4 2/6/96 sleep studyPain Treatment Pain Level (Numeric Scale): 7\\n\n",
	"dtype: object"
	]
	},
	"execution_count": 113,
	"metadata": {},
	"output_type": "execute_result"
	}
	],
	"source": [
	"import pandas as pd\n",
	"from functools import reduce\n",
	"pd.options.display.max_colwidth=1000\n",
	"pd.options.display.max_rows = 500\n",
	"\n",
	"doc = []\n",
	"with open('dates.txt') as file:\n",
	" for line in file:\n",
	" doc.append(line)\n",
	"\n",
	"df = pd.Series(doc)\n",
	"df.head(5)"
	]
	},
	{
	"cell_type": "code",
	"execution_count": 247,
	"metadata": {},
	"outputs": [
	{
	"name": "stdout",
	"output_type": "stream",
	"text": [
	"\n",
	"04/20/2009; 04/20/09; 4/20/09; 4/3/09\n",
	"Mar-20-2009; Mar 20, 2009; March 20, 2009; Mar. 20, 2009; Mar 20 2009;\n",
	"20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009\n",
	"Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009\n",
	"Feb 2009; Sep 2009; Oct 2010\n",
	"6/2008; 12/2009\n",
	"2009; 2010\n",
	"\n",
	"rulex\n",
	"[('', '9', '1984')]\n",
	"rule2\n",
	"[('mar', '20', '2009'), ('mar', '20', '2009'), ('march', '20', '2009'), ('mar', '20', '2009'), ('mar', '20', '2009')]\n",
	"rule3\n",
	"[('20', 'mar', '2009'), ('20', 'march', '2009'), ('20', 'mar', '2009'), ('20', 'march', '2009')]\n",
	"rule4\n",
	"[('mar', '20', '2009'), ('mar', '21', '2009'), ('mar', '22', '2009')]\n",
	"rule5\n",
	"[('', 'feb', '2009'), ('', 'sep', '2009'), ('', 'oct', '2010')]\n",
	"rule6\n",
	"[('', '6', '2008'), ('', '12', '2009')]\n",
	"rule7\n",
	"[('', '', '2009'), ('', '', '2010')]\n"
	]
	}
	],
	"source": [
	"#play ground\n",
	"\n",
	"rule1 = r'(?P<month>\\d{1,2})[/-](?P<day>\\d{1,2})[/-](?P<year>\\d{2}\|\\d{4})'\n",
	"rule2 = r'(?P<month>(?:jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\\w*)[.]?[ -](?P<day>\\d{1,2})(?:\\w{2})?[,]?[ -](?P<year>\\d{4})'\n",
	"rule3 = r'(?P<day>\\d{1,2}) (?P<month>(?:jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\\w*)[.,]? (?P<year>\\d{4})'\n",
	"rule4 = rule2\n",
	"rule5 = r'(?P<day>)(?P<month>(?:jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\\w*) (?P<year>\\d{4})'\n",
	"rule6 = r'(?P<day>)(?P<month>\\d{1,2})/(?P<year>\\d{4})'\n",
	"rule7 = r'(?P<day>)(?P<month>)(?:\\D\|^)(?P<year>\\d{4})(?:\\D\|$)'\n",
	"\n",
	"text = '''\n",
	"04/20/2009; 04/20/09; 4/20/09; 4/3/09\n",
	"Mar-20-2009; Mar 20, 2009; March 20, 2009; Mar. 20, 2009; Mar 20 2009;\n",
	"20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009\n",
	"Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009\n",
	"Feb 2009; Sep 2009; Oct 2010\n",
	"6/2008; 12/2009\n",
	"2009; 2010\n",
	"'''\n",
	"import re\n",
	"print (text)\n",
	"\n",
	"print ('rulex')\n",
	"print (re.findall(rule6, \"\"\"sChesterfield 9/1984 for 3 weeks for dual diagnosis alcohol and PTSDHx of Outpatient Treatment: Yes\\\n",
	"\"\"\".lower()))\n",
	"\n",
	"print ('rule2')\n",
	"print (re.findall(rule2, 'Mar-20-2009; Mar 20, 2009; March 20, 2009; Mar. 20, 2009; Mar 20 2009'.lower()))\n",
	"\n",
	"print ('rule3')\n",
	"print (re.findall(rule3, '20 Mar 2009; 20 March 2009; 20 Mar. 2009; 20 March, 2009'.lower()))\n",
	"\n",
	"print ('rule4')\n",
	"print (re.findall(rule4, 'Mar 20th, 2009; Mar 21st, 2009; Mar 22nd, 2009'.lower()))\n",
	"\n",
	"print ('rule5')\n",
	"print (re.findall(rule5, 'Feb 2009; Sep 2009; Oct 2010'.lower()))\n",
	"\n",
	"print ('rule6')\n",
	"print (re.findall(rule6, '6/2008; 12/2009'.lower()))\n",
	"\n",
	"print ('rule7')\n",
	"print (re.findall(rule7, '2009; 2010'.lower()))\n",
	"# pd.DataFrame(\n",
	"# df.str.extract(rule2),df\n",
	"# )"
	]
	},
	{
	"cell_type": "code",
	"execution_count": 243,
	"metadata": {
	"collapsed": true
	},
	"outputs": [],
	"source": [
	"def date_sorter(df):\n",
	" \n",
	" # Your code here\n",
	" mon2int = dict([(m, i+1) for i, m in enumerate('jan,feb,mar,apr,may,jun,jul,aug,sep,oct,nov,dec'.split(\",\"))])\n",
	" rule1 = r'(?P<month>\\d{1,2})[/-](?P<day>\\d{1,2})[/-](?P<year>\\d{2}\|\\d{4})'\n",
	" rule2 = r'(?P<month>(?:jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\\w*)[.]?[ -](?P<day>\\d{1,2})(?:\\w{2})?[,]?[ -](?P<year>\\d{4})'\n",
	" rule3 = r'(?P<day>\\d{1,2}) (?P<month>(?:jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\\w*)[.,]? (?P<year>\\d{4})'\n",
	" rule4 = rule2\n",
	" rule5 = r'(?P<day>)(?P<month>(?:jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\\w*) (?P<year>\\d{4})'\n",
	" rule6 = r'(?P<day>)(?P<month>\\d{1,2})/(?P<year>\\d{4})'\n",
	" rule7 = r'(?P<day>)(?P<month>)(?:\\D\|^)(?P<year>\\d{4})(?:\\D\|$)'\n",
	" rule_list = [rule1, rule2, rule3, rule4, rule5, rule6, rule7]\n",
	" extract_df_list = [df.str.lower().str.extract(rule) for rule in rule_list]\n",
	" date_df = reduce(lambda x,y: x.fillna(y),extract_df_list)\n",
	" date_df['text'] = df\n",
	" date_df['month'] = date_df['month'].apply(lambda x: mon2int.get(str(x)[:3],x))\n",
	" date_df['month'] = date_df['month'].fillna('1').apply(lambda x: '1' if x=='' else x)\n",
	" date_df['year'] = date_df['year'].apply(lambda x: '19'+x if len(x)==2 else x)\n",
	" date_df['day'] = date_df['day'].fillna('1').apply(lambda x: '1' if x=='' else x)\n",
	" date_df = date_df[[\"year\",\"month\",\"day\"]].astype(int)\n",
	" index_df = date_df.sort_values(['year','month','day'], axis=0, ascending=False).reset_index()\n",
	" index_df = index_df['index']\n",
	" return index_df"
	]
	},
	{
	"cell_type": "code",
	"execution_count": 244,
	"metadata": {
	"scrolled": false
	},
	"outputs": [
	{
	"name": "stderr",
	"output_type": "stream",
	"text": [
	"/opt/conda/lib/python3.6/site-packages/ipykernel/__main__.py:13: FutureWarning: currently extract(expand=None) means expand=False (return Index/Series/DataFrame) but in a future version of pandas this will be changed to expand=True (return DataFrame)\n"
	]
	},
	{
	"data": {
	"text/plain": [
	"0 413\n",
	"1 161\n",
	"2 186\n",
	"3 141\n",
	"4 231\n",
	"5 427\n",
	"6 253\n",
	"7 464\n",
	"8 235\n",
	"9 152\n",
	"10 257\n",
	"11 475\n",
	"12 401\n",
	"13 255\n",
	"14 439\n",
	"15 366\n",
	"16 381\n",
	"17 463\n",
	"18 198\n",
	"19 279\n",
	"20 431\n",
	"21 244\n",
	"22 286\n",
	"23 480\n",
	"24 383\n",
	"25 320\n",
	"26 139\n",
	"27 208\n",
	"28 243\n",
	"29 220\n",
	"30 282\n",
	"31 393\n",
	"32 460\n",
	"33 290\n",
	"34 227\n",
	"35 217\n",
	"36 229\n",
	"37 316\n",
	"38 170\n",
	"39 240\n",
	"40 467\n",
	"41 242\n",
	"42 364\n",
	"43 408\n",
	"44 411\n",
	"45 176\n",
	"46 448\n",
	"47 287\n",
	"48 273\n",
	"49 472\n",
	"50 268\n",
	"51 288\n",
	"52 379\n",
	"53 386\n",
	"54 389\n",
	"55 297\n",
	"56 371\n",
	"57 491\n",
	"58 360\n",
	"59 400\n",
	"60 293\n",
	"61 271\n",
	"62 396\n",
	"63 183\n",
	"64 187\n",
	"65 445\n",
	"66 497\n",
	"67 337\n",
	"68 428\n",
	"69 132\n",
	"70 341\n",
	"71 160\n",
	"72 289\n",
	"73 264\n",
	"74 325\n",
	"75 277\n",
	"76 391\n",
	"77 252\n",
	"78 314\n",
	"79 410\n",
	"80 490\n",
	"81 173\n",
	"82 307\n",
	"83 433\n",
	"84 270\n",
	"85 496\n",
	"86 348\n",
	"87 346\n",
	"88 344\n",
	"89 250\n",
	"90 339\n",
	"91 498\n",
	"92 451\n",
	"93 146\n",
	"94 302\n",
	"95 126\n",
	"96 207\n",
	"97 281\n",
	"98 425\n",
	"99 241\n",
	"100 350\n",
	"101 414\n",
	"102 188\n",
	"103 291\n",
	"104 306\n",
	"105 484\n",
	"106 144\n",
	"107 356\n",
	"108 362\n",
	"109 234\n",
	"110 452\n",
	"111 469\n",
	"112 184\n",
	"113 449\n",
	"114 128\n",
	"115 262\n",
	"116 446\n",
	"117 304\n",
	"118 494\n",
	"119 394\n",
	"120 367\n",
	"121 377\n",
	"122 147\n",
	"123 195\n",
	"124 159\n",
	"125 125\n",
	"126 328\n",
	"127 457\n",
	"128 374\n",
	"129 443\n",
	"130 158\n",
	"131 190\n",
	"132 417\n",
	"133 392\n",
	"134 420\n",
	"135 260\n",
	"136 305\n",
	"137 329\n",
	"138 456\n",
	"139 201\n",
	"140 145\n",
	"141 266\n",
	"142 407\n",
	"143 212\n",
	"144 384\n",
	"145 327\n",
	"146 432\n",
	"147 376\n",
	"148 321\n",
	"149 471\n",
	"150 120\n",
	"151 251\n",
	"152 387\n",
	"153 79\n",
	"154 12\n",
	"155 224\n",
	"156 20\n",
	"157 343\n",
	"158 222\n",
	"159 18\n",
	"160 338\n",
	"161 459\n",
	"162 114\n",
	"163 333\n",
	"164 101\n",
	"165 353\n",
	"166 87\n",
	"167 213\n",
	"168 365\n",
	"169 468\n",
	"170 166\n",
	"171 131\n",
	"172 121\n",
	"173 324\n",
	"174 124\n",
	"175 4\n",
	"176 142\n",
	"177 479\n",
	"178 292\n",
	"179 169\n",
	"180 90\n",
	"181 326\n",
	"182 203\n",
	"183 181\n",
	"184 60\n",
	"185 248\n",
	"186 110\n",
	"187 236\n",
	"188 372\n",
	"189 56\n",
	"190 388\n",
	"191 140\n",
	"192 193\n",
	"193 238\n",
	"194 311\n",
	"195 483\n",
	"196 406\n",
	"197 409\n",
	"198 42\n",
	"199 359\n",
	"200 133\n",
	"201 47\n",
	"202 196\n",
	"203 308\n",
	"204 163\n",
	"205 300\n",
	"206 450\n",
	"207 477\n",
	"208 331\n",
	"209 357\n",
	"210 172\n",
	"211 86\n",
	"212 192\n",
	"213 249\n",
	"214 0\n",
	"215 354\n",
	"216 272\n",
	"217 298\n",
	"218 478\n",
	"219 113\n",
	"220 301\n",
	"221 143\n",
	"222 310\n",
	"223 442\n",
	"224 100\n",
	"225 119\n",
	"226 269\n",
	"227 52\n",
	"228 211\n",
	"229 62\n",
	"230 138\n",
	"231 399\n",
	"232 487\n",
	"233 66\n",
	"234 322\n",
	"235 46\n",
	"236 67\n",
	"237 453\n",
	"238 33\n",
	"239 51\n",
	"240 91\n",
	"241 97\n",
	"242 226\n",
	"243 461\n",
	"244 122\n",
	"245 245\n",
	"246 206\n",
	"247 151\n",
	"248 194\n",
	"249 22\n",
	"250 412\n",
	"251 157\n",
	"252 177\n",
	"253 233\n",
	"254 482\n",
	"255 435\n",
	"256 7\n",
	"257 26\n",
	"258 149\n",
	"259 89\n",
	"260 210\n",
	"261 373\n",
	"262 385\n",
	"263 440\n",
	"264 202\n",
	"265 218\n",
	"266 71\n",
	"267 115\n",
	"268 265\n",
	"269 312\n",
	"270 476\n",
	"271 180\n",
	"272 174\n",
	"273 216\n",
	"274 256\n",
	"275 390\n",
	"276 37\n",
	"277 330\n",
	"278 462\n",
	"279 76\n",
	"280 116\n",
	"281 130\n",
	"282 416\n",
	"283 63\n",
	"284 41\n",
	"285 349\n",
	"286 32\n",
	"287 68\n",
	"288 284\n",
	"289 485\n",
	"290 54\n",
	"291 423\n",
	"292 437\n",
	"293 29\n",
	"294 168\n",
	"295 261\n",
	"296 404\n",
	"297 368\n",
	"298 96\n",
	"299 334\n",
	"300 230\n",
	"301 276\n",
	"302 8\n",
	"303 352\n",
	"304 199\n",
	"305 492\n",
	"306 135\n",
	"307 228\n",
	"308 185\n",
	"309 280\n",
	"310 447\n",
	"311 1\n",
	"312 178\n",
	"313 10\n",
	"314 274\n",
	"315 136\n",
	"316 489\n",
	"317 426\n",
	"318 48\n",
	"319 275\n",
	"320 421\n",
	"321 455\n",
	"322 99\n",
	"323 175\n",
	"324 285\n",
	"325 107\n",
	"326 61\n",
	"327 209\n",
	"328 247\n",
	"329 438\n",
	"330 35\n",
	"331 295\n",
	"332 137\n",
	"333 294\n",
	"334 397\n",
	"335 205\n",
	"336 358\n",
	"337 470\n",
	"338 179\n",
	"339 88\n",
	"340 429\n",
	"341 112\n",
	"342 103\n",
	"343 44\n",
	"344 70\n",
	"345 454\n",
	"346 127\n",
	"347 25\n",
	"348 16\n",
	"349 458\n",
	"350 403\n",
	"351 74\n",
	"352 263\n",
	"353 215\n",
	"354 246\n",
	"355 444\n",
	"356 80\n",
	"357 430\n",
	"358 355\n",
	"359 197\n",
	"360 39\n",
	"361 134\n",
	"362 466\n",
	"363 221\n",
	"364 267\n",
	"365 361\n",
	"366 441\n",
	"367 340\n",
	"368 78\n",
	"369 150\n",
	"370 347\n",
	"371 499\n",
	"372 259\n",
	"373 21\n",
	"374 167\n",
	"375 75\n",
	"376 254\n",
	"377 296\n",
	"378 5\n",
	"379 398\n",
	"380 424\n",
	"381 495\n",
	"382 313\n",
	"383 378\n",
	"384 164\n",
	"385 434\n",
	"386 65\n",
	"387 81\n",
	"388 200\n",
	"389 6\n",
	"390 336\n",
	"391 105\n",
	"392 148\n",
	"393 239\n",
	"394 318\n",
	"395 369\n",
	"396 493\n",
	"397 189\n",
	"398 72\n",
	"399 232\n",
	"400 117\n",
	"401 19\n",
	"402 123\n",
	"403 419\n",
	"404 309\n",
	"405 283\n",
	"406 395\n",
	"407 303\n",
	"408 488\n",
	"409 93\n",
	"410 27\n",
	"411 315\n",
	"412 258\n",
	"413 204\n",
	"414 342\n",
	"415 23\n",
	"416 237\n",
	"417 465\n",
	"418 219\n",
	"419 363\n",
	"420 50\n",
	"421 3\n",
	"422 370\n",
	"423 382\n",
	"424 165\n",
	"425 418\n",
	"426 40\n",
	"427 319\n",
	"428 11\n",
	"429 49\n",
	"430 317\n",
	"431 473\n",
	"432 223\n",
	"433 155\n",
	"434 214\n",
	"435 278\n",
	"436 351\n",
	"437 182\n",
	"438 332\n",
	"439 156\n",
	"440 108\n",
	"441 73\n",
	"442 402\n",
	"443 154\n",
	"444 162\n",
	"445 299\n",
	"446 104\n",
	"447 436\n",
	"448 481\n",
	"449 57\n",
	"450 345\n",
	"451 380\n",
	"452 375\n",
	"453 422\n",
	"454 323\n",
	"455 405\n",
	"456 36\n",
	"457 335\n",
	"458 415\n",
	"459 486\n",
	"460 191\n",
	"461 171\n",
	"462 31\n",
	"463 225\n",
	"464 111\n",
	"465 98\n",
	"466 129\n",
	"467 13\n",
	"468 153\n",
	"469 474\n",
	"470 28\n",
	"471 53\n",
	"472 2\n",
	"473 84\n",
	"474 9\n",
	"475 58\n",
	"476 83\n",
	"477 109\n",
	"478 102\n",
	"479 34\n",
	"480 118\n",
	"481 43\n",
	"482 92\n",
	"483 106\n",
	"484 15\n",
	"485 94\n",
	"486 69\n",
	"487 17\n",
	"488 55\n",
	"489 59\n",
	"490 85\n",
	"491 64\n",
	"492 38\n",
	"493 24\n",
	"494 82\n",
	"495 14\n",
	"496 95\n",
	"497 30\n",
	"498 45\n",
	"499 77\n",
	"Name: index, dtype: int64"
	]
	},
	"execution_count": 244,
	"metadata": {},
	"output_type": "execute_result"
	}
	],
	"source": [
	"date_sorter(df)"
	]
	},
	{
	"cell_type": "code",
	"execution_count": null,
	"metadata": {
	"collapsed": true
	},
	"outputs": [],
	"source": []
	}
	],
	"metadata": {
	"coursera": {
	"course_slug": "python-text-mining",
	"graded_item_id": "LvcWI",
	"launcher_item_id": "krne9",
	"part_id": "Mkp1I"
	},
	"kernelspec": {
	"display_name": "Python 3",
	"language": "python",
	"name": "python3"
	},
	"language_info": {
	"codemirror_mode": {
	"name": "ipython",
	"version": 3
	},
	"file_extension": ".py",
	"mimetype": "text/x-python",
	"name": "python",
	"nbconvert_exporter": "python",
	"pygments_lexer": "ipython3",
	"version": "3.6.0"
	}
	},
	"nbformat": 4,
	"nbformat_minor": 2
	}