Skip to content

Instantly share code, notes, and snippets.

@andrewdoss-bit
Last active August 24, 2021 00:50
Show Gist options
  • Save andrewdoss-bit/477b9fd42330ac42bf273cdb0d8900ff to your computer and use it in GitHub Desktop.
Save andrewdoss-bit/477b9fd42330ac42bf273cdb0d8900ff to your computer and use it in GitHub Desktop.
Adding the post-load transformation
#!/bin/bash
# Activate the venv and navigate to the location of main.py
source venv/bin/activate
cd simple_pipeline
# Uncomment the line below if you would like to re-run the population data pipeline
# The population data is only updated annually by the Census Bureau
# python main.py -local_source -name acs_population_counties \
# acs_5yr_population_data.csv bitdotio/simple_pipeline.population_counties
python main.py -name nyt_cases_counties \
'https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv' \
bitdotio/simple_pipeline.cases_counties
python main.py -name cdc_vaccines_counties \
'https://data.cdc.gov/api/views/8xkx-amqh/rows.csv?accessType=DOWNLOAD' \
bitdotio/simple_pipeline.vaccinations_counties
# New line shown below
python sql_executor.py ca_covid_data.sql
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment