This small subclass of the Pandas sqlalchemy-based SQL support for reading/storing tables uses the Postgres-specific "COPY FROM" method to insert large amounts of data to the database. It is much faster that using INSERT. To acheive this, the table is created in the normal way using sqlalchemy but no data is inserted. Instead the data is saved to a temporary CSV file (using Pandas' mature CSV support) then read back to Postgres using Psychopg2 support for COPY FROM STDIN.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# -*- coding: utf-8 -*- | |
""" | |
:DESCRIPTION:Code for managing soil moisture data | |
:REQUIRES: | |
:TODO:More error handling | |
:AUTHOR: John Franco Saraceno | |
:ORGANIZATION: U.S. Geological Survey, United States Department of Interior | |
:CONTACT: [email protected] | |
:VERSION: 1.0 | |
Fri Jul 29 18:48:40 2016 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# List unique values in a DataFrame column | |
pd.unique(df.column_name.ravel()) | |
# Convert Series datatype to numeric, getting rid of any non-numeric values | |
df['col'] = df['col'].astype(str).convert_objects(convert_numeric=True) | |
# Grab DataFrame rows where column has certain values | |
valuelist = ['value1', 'value2', 'value3'] | |
df = df[df.column.isin(valuelist)] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import urllib.request as request | |
import json | |
import matplotlib.pyplot as plt | |
import matplotlib.colors as colors | |
import matplotlib.cm as cm | |
import numpy as np | |
class BuoySpectraPlotter: | |
def __init__(self, buoy_station): |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#include <ESP8266WiFi.h> | |
#include <Wire.h> | |
#include <PubSubClient.h> | |
#include <Adafruit_Sensor.h> | |
#include <Adafruit_BME280.h> | |
#define wifi_ssid "YOUR_WIFI_SSID" | |
#define wifi_password "YOUR_WIFI_PASSWORD" | |
#define mqtt_server "YOUR_MQTT_SERVER_HOST" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
--- Config | |
SSID = "YOUR_WIFI_NAME" | |
PASSWORD = "YOUR_WIFI_PASSWORD" | |
TIMEOUT = 30000000 -- 30s | |
--- Station modes | |
STAMODE = { | |
STATION_IDLE = 0, | |
STATION_CONNECTING = 1, | |
STATION_WRONG_PASSWORD = 2, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
//nodeMCU v1.0 (black) with Arduino IDE | |
//stream temperature data DS18B20 with 1wire on ESP8266 ESP12-E (nodeMCU v1.0) | |
//shin-ajaran.blogspot.com | |
//nodemcu pinout https://github.com/esp8266/Arduino/issues/584 | |
#include <ESP8266WiFi.h> | |
#include <OneWire.h> | |
#include <DallasTemperature.h> | |
//Def | |
#define myPeriodic 15 //in sec | Thingspeak pub is 15sec |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import exifread | |
# based on https://gist.github.com/erans/983821 | |
def _get_if_exist(data, key): | |
if key in data: | |
return data[key] | |
return None |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/sh | |
if [ $# != 2 ]; then | |
echo "\nchris k's git update automater sh script. Edit to put a bunch of \`git add blah'" | |
echo "lines in the body then evoke with the branch name and commit description\n" | |
echo "Usage: $0 <branch_name> <\"Description of update\">\n" | |
echo " branch_name: the name of the git branch to be created" | |
echo " Description: Text for: git commit -m \"Description of update\"" | |
echo "" | |
echo "Example:" | |
BRANCH="core_override" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FROM python:3.8.5-slim-buster as base | |
LABEL maintainer="Your Name <[email protected]>" \ | |
version="1.0.0" | |
RUN apt-get update && apt-get install tzdata -y --no-install-recommends | |
ENV TZ="America/Los_Angeles" | |
ENV PIP_NO_CACHE_DIR=1 | |
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 |
OlderNewer