Skip to content

Instantly share code, notes, and snippets.

View andrewdoss-bit's full-sized avatar

andrewdoss-bit

View GitHub Profile
@andrewdoss-bit
andrewdoss-bit / ca_covid_data.sql
Created August 18, 2021 23:46
PostgreSQL script for California COVID dataset
CREATE TABLE IF NOT EXISTS "bitdotio/simple_pipeline"."ca_covid_data_join" (
date date,
county TEXT,
fips TEXT,
cases INTEGER,
deaths INTEGER,
pct_vaccinations_complete REAL,
num_vaccinations_complete INTEGER,
total_population INTEGER,
population_16plus INTEGER,
@andrewdoss-bit
andrewdoss-bit / scheduled_run.sh
Last active August 24, 2021 00:50
Adding the post-load transformation
#!/bin/bash
# Activate the venv and navigate to the location of main.py
source venv/bin/activate
cd simple_pipeline
# Uncomment the line below if you would like to re-run the population data pipeline
# The population data is only updated annually by the Census Bureau
# python main.py -local_source -name acs_population_counties \
# acs_5yr_population_data.csv bitdotio/simple_pipeline.population_counties
python main.py -name nyt_cases_counties \
'https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv' \
@andrewdoss-bit
andrewdoss-bit / node_connect.js
Created August 27, 2021 23:29
node connect example
const { Client } = require('pg');
const client = new Client({
database: 'bitdotio',
host: 'db.bit.io',
port: 5432,
user: '<YOUR USERNAME HERE>',
password: '<YOUR PASSWORD HERE>'
});
@andrewdoss-bit
andrewdoss-bit / read_sensor.py
Created September 2, 2021 01:00
Reading SDS011 Sensor
import serial
ser = serial.Serial('/dev/ttyUSB0')
message = ser.read(10)
@andrewdoss-bit
andrewdoss-bit / create_record.py
Last active September 7, 2021 15:04
Creating a record
from datetime import datetime
def create_record(sample, measurement_config, period):
"""Create a record for insertion to a database."""
record = {'location': 'outside of house'}
record['sensor_id'] = parse_value(sample[0], 'little', [6, 2])
record['datetime'] = str(datetime.utcnow())
for measurement, parse_args in config['measurements'].items():
@andrewdoss-bit
andrewdoss-bit / parse_value.py
Last active September 2, 2021 02:20
Parse value
def parse_value(message, byte_order, start_byte, num_bytes, scale=None):
"""Returns a number from a sequence of bytes."""
value = message[start_byte: start_byte + num_bytes]
value = int.from_bytes(value, byteorder=byte_order)
value = value * scale if scale else value
return value
@andrewdoss-bit
andrewdoss-bit / create_table.sql
Created September 2, 2021 03:59
Create pm table
-- Aligns with logged columns in config.yaml
-- Update the first line with your table name
CREATE TABLE "air_quality_log_test/air_quality"."pm_measurements" (
"datetime" timestamp with time zone,
"location" text,
"sensor_id" integer,
"pm_2_5" real,
"pm_10" real
)
@andrewdoss-bit
andrewdoss-bit / insert_record.py
Last active September 2, 2021 04:20
Insert record
import bitdotio
columns = ['datetime', 'location', 'sensor_id', 'pm_2_5', 'pm_10']
record_list = [record[col] for col in columns]
bit = bitdotio.bitdotio(BITDOTIO_API_KEY)
# Replace the line below with your schema-qualified table name
qualified_table = '"air_quality_log_test/air_quality"."pm_measurements"'
@andrewdoss-bit
andrewdoss-bit / bit_connect.py
Created September 9, 2021 19:18
bit connect
# Connect to bit.io
import bitdotio
bit_conn = bitdotio.bitdotio(<YOUR_BITIO_KEY>)
# You can use the connection with the bit module, here we list repos
from bitdotio import bit
bit.list(bit_conn)
@andrewdoss-bit
andrewdoss-bit / create_repo.py
Created September 9, 2021 19:22
create repo
# Create a repo
import bitdotio
# Connect to bit.io
b = bitdotio.bitdotio(<YOUR_BITIO_KEY>)
# Construct a repo object
r = bitdotio.model.repo.Repo(name='my_new_repo',
description='My new repository.',
is_private=True)