afds01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://accessfuel.com/assets/images/logo/logo.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Scientist Position Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Thank you for taking the time to complete this challenge. As a data scientist for AccessFuel you will be responsible for sourcing data from many sources, data wrangling, and analysis. This is an opportunity to showcase your skills.**\n",
    "\n",
    "**Good luck! **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1- Download this notebook\n",
    "\n",
    "2- Make sure you have numpy, pandas, matplotlib and sklearn installed\n",
    "\n",
    "3- Read the instructions for each section and write code to accomplish each task within the designated area\n",
    "\n",
    "4- Save your final notebook (.ipynb) and email it to amir@accessfuel.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1- Read a CSV file into memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variables needed for this part (do not delete)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'\n",
    "headers = 'age,workclass,fnlwgt,education,education-num,marital-status,occupation,relationship,race,sex,capital-gain,capital-loss,hours-per-week,native-county,income'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Intrusctions\n",
    "Using the pandas library (if you are using Python) read the above url (url variable) into a dataframe. Make sure that you (1) skip the first 10 rows and (2) use the headers variables defined above as the column header.\n",
    "\n",
    "Note that adult.data is commad separated and has no header.\n",
    "\n",
    "**Output: print the dimensions (shape) of the dataframe as well as the top 5 rows**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# START CODE\n",
    "\n",
    "# END CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2- Filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instructions\n",
    "\n",
    "1- Create a new dataframe that excludes the \" Preschool\" variable (do not mutate the first dataframe). Call this dataframe df1.\n",
    "\n",
    "2- Create a new dataframe by filtering rows that fall within 1st and 3rd quartile of \"education-num\" (hint: describe gives you summary statistics for a variable). Call this dataframe df2.\n",
    "\n",
    "**Output: Print the percentage descrease in the number of rows after the filtering.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "education_exclude = [' Preschool']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# START CODE\n",
    "\n",
    "# END CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3- Grouping and visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instructions\n",
    "\n",
    "1- Group the rows in df2 by \"occupation\" and aggregate the age within each group.\n",
    "\n",
    "**Output: display a horizontal bar chart of average age by occupation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# START CODE\n",
    "\n",
    "# END CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4- Naiive Bayes Classification (Extra Credit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Attempt this section only if you have completed other sections and have extra time **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "url_train = 'https://s3-us-west-1.amazonaws.com/amirziai-accessfuel/nb_train.csv'\n",
    "url_test = 'https://s3-us-west-1.amazonaws.com/amirziai-accessfuel/nb_test.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instructions\n",
    "\n",
    "Train a Naiive Bayes classifier using the dataset in \"url_train\". Each row in the dataset consists of a \"Sentence\" and a \"Tag\" which encodes the sentiment. Your classifier should use the words in a sentence as features for that row. For this part simply count the number of occurences of each word in the dictionary (you can accomplish this using CountVectorizer from the scikit learn library if you are using Python).\n",
    "For more information refer to:\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html\n",
    "\n",
    "*If you are using pandas empty values are by defaul encoded as NaN. scikit learn does not play well with NaN. Make sure that you handle NaNs before you predict*\n",
    "\n",
    "**Output: Print the predictions of your NB model for the test dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# START CODE\n",
    "\n",
    "# END CODE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}