Skip to content

Instantly share code, notes, and snippets.

View LouisAmon's full-sized avatar

Louis Amon LouisAmon

View GitHub Profile
@LouisAmon
LouisAmon / avro_to_dataframe.py
Created January 15, 2017 11:55
Read Avro file from Pandas
import pandas
import fastavro
def avro_df(filepath, encoding):
# Open file stream
with open(filepath, encoding) as fp:
# Configure Avro reader
reader = fastavro.reader(fp)
# Load records in memory
records = [r for r in reader]
@LouisAmon
LouisAmon / pandas_to_elasticsearch.py
Last active November 4, 2023 08:34
Convert a `pandas.DataFrame` to a generator of ElasticSearch actions to be passed to a bulk helper
def pandas_to_elasticsearch(df):
"""
Generator that converts `pandas.DataFrame` rows into ElasticSearch actions
"""
# ElasticSearch will raise an exception on `NaN` values
df = df.dropna()
for i, row in df.iterrows():
action = {
'_op_type': 'index',
'_index': 'my_index',
@LouisAmon
LouisAmon / shovel.py
Created July 5, 2017 05:05 — forked from jdmaturen/shovel.py
SQS + Boto + Eventlet, re http://gist.github.com/434053
(jd@XXX) /home/jd/shovel> time ./shovel.py jd_test
got 901 messages
real 0m1.617s
user 0m0.558s
sys 0m0.044s
@LouisAmon
LouisAmon / shovel.py
Created July 5, 2017 05:06 — forked from jdmaturen/shovel.py
SQS + Boto + Eventlet, re http://gist.github.com/434053
(jd@XXX) /home/jd/shovel> time ./shovel.py jd_test
got 901 messages
real 0m1.617s
user 0m0.558s
sys 0m0.044s
@LouisAmon
LouisAmon / serialize-numpy-array.py
Last active November 16, 2017 10:03 — forked from alexland/serialize-numpy-array.py
serialize, persist, retrieve, and de-serialize a NumPy array as a binary string (any dimension, any dtype); exemplary use case: a web app calculates some result--eg, from a Machine Learning algorithm, using NumPy and the result is a NumPy array; it is efficient to just return that result to rather than persist the array then retrieve it via query
import time
import numpy
import redis
# a 2D array to serialize
A = 10 * numpy.random.randn(10000).reshape(1000, 10)
# flatten the 2D NumPy array and save it as a binary string
@LouisAmon
LouisAmon / fanout.js
Created March 16, 2018 08:46
Fanout AWS Lambda
'use strict';
const aws = require('aws-sdk');
const lambda = new aws.Lambda({region: 'eu-west-1'});
const async = require('async');
exports.handler = (event, context, callback) => {
/*
Invokes a given Lambda function `event.function` in parallel for each given
argument `event.events`
@LouisAmon
LouisAmon / build.sh
Created April 21, 2018 11:52
Building an AWS Lambda package that includes `numpy`, `pandas` & `scipy`
# System dependencies
yum update -y
yum install -y \
binutils \
findutils \
python36
# Create directory for our lambda
mkdir build
pip install -t build numpy pandas scipy
# Check uncompressed file size
def trigo():
product = 1.0
for counter in range(1, 1000, 1):
for dex in list(range(1, 360, 1)):
angle = radians(dex)
product *= sin(angle)**2 + cos(angle)**2
return product
def sha():
h = hashlib.sha1('secret')
@LouisAmon
LouisAmon / json_to_gzip.py
Created July 26, 2018 12:49
How to compress JSON-able data using gzip (Python 3)
import json
import gzip
def compress_data(data):
# Convert to JSON
json_data = json.dumps(data, indent=2)
# Convert to bytes
encoded = json_data.encode('utf-8')
# Compress
compressed = gzip.compress(encoded)
@LouisAmon
LouisAmon / lambda_handler.py
Created October 15, 2018 08:16
Run a system command in a Lambda
import subprocess
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
"""
Runs a shell command and returns the output code
STDOUT & STDERR are logged into Cloudwatch