Data Science Workflow and Overview

overview.md

our trade/craft

gaining insights from data for actionable decisions data science -> applied interdisciplinary science around using data to make decision

practicing data science is a function

1. input
2. process
3. output

data science workflow

• plan
• acquire
• prepare
• explore
• model
• present
• maintain your data products

inputs:

• csv

• sql

• raw text

• images

• audio or video

• various types of data in various types of formats

• we have 2 main kinds of data

  • labeled data:
    • we have our targets
    • we have our labels (some human interaction)
      • example: hitting spam button is adding a lable to the data inside that email
        • sender, subject, recipient, attachments, language, wording
  • unlabeled data:
    • raw text
    • images where a human has not added any input

• sometimes our data is structured and sometimes not.

  • either way, it requires work to prepare or integrate

acquire

• sometimes, it's as easy as pd.read_csv()
• sometimes, we've got to go collect the data ourselves
• how do we sample a population to have a representative sample
• this is where you need to be good w/ sql
• may have to talk to different data sources
• overlap w/ other software dev skills
• you may get data from a data engineer
• most of the time, this is on you.

prepare

• 80% of the time, we're preparing the data
• pandas, pandas, pandas
• it's pandamonium.
• derived values like gross revenue if you have units and price
• feature engineering

explore

• statistics
• probability
• visualization
• we explore the data to figure out what model is a good candidate
• set aside 30% of our data for data for testing our model

model

• run a model or two(maybe three)
• measure the effectiveness
  • true positives, true negatives, false positive, false negatives
  • accuracy
  • precision
  • recall
• hyperparameter tuning (we're modifying the components of the ML equation that aren't only the inputs)
  • weights on weighted averages
  • number of groups in k-means
  • think of these like tweaking a performance automobile
  • experiment with your fuel-air mixture
• we need to test our model
• ideally, we've got new data coming in

present

• building data products to share
• write a paper or whitepaper
• produce a talk or a handout for talk
• bokeh
• tableau
• handouts

Maintain

• you may retrain your model
• get new data
• engineer your model to run on streaming inputs vs. a one time dump of data
• maintain your tool

Technologies in their place of the Data Science Workflow

Reminder: Workflow is plan, acquire, prepare, explore, model, present

Plan -> people think and people discuss and collaborate, project management like trello

Acquire

• Technologies: python, pandas, numpy, SQL, python libraries (web scraping, etc...)
• Skills: programming, troubleshooting, keeping the big picture in mind, keeping the next step in mind
• Tasks: getting raw data

Prepare

• Technologies: serious python. lots and lots of pandas.
• Skills: attention to detail, debug other people's data, debug our own code
• Tasks: cleaning the data, this is the "data wrangling" or "data munging" part, feature engineering

Explore

• Tasks: get a sense of the relationship of data's variables, visualization, statistical analysis
• Skills: statistics, visualization
• Technologies: matplotlib, seaborn, pandas, python, numpy, scipy, statistical packages

Model

• Technologies: scipy, sklearn (ML algorithms), keras or tensorflow
• Tasks: Evaluating the effacacy of a model, hyperparameter tuning
• Skills: determining overfitting vs. underfitting

Present

• Tasks: get your point across quickly, be prepared to support your thesis, but start w/ the bottom line (your point)
• Skills: public speaking, visual design and a sense for how people process information
• Technologies: Tableau, Microsoft Word to make a handout