Skip to content

Instantly share code, notes, and snippets.

View nmilford's full-sized avatar

Nathan Milford nmilford

View GitHub Profile
@nmilford
nmilford / api-batching-to-csv.py
Created September 3, 2019 20:41
Example on how to use limit/offset to collect and export data from the PerformLine API in bulk.
# This code demonstrates how to loop through an API using limit+offset.
#
# It will get all of the results as it pages through the data by limit and
# offset preventing the backend that serves it from getting overwhelemed.
#
# It will then put all of that collected data into a CSV for later processing.
#
# This code was thrown together as a demonstration, no warrenties are given or
# implied.
import requests
@nmilford
nmilford / build_redis_deb.sh
Last active June 4, 2018 18:35
redis 5.0 deb with rebloom and redis graph
# Create build root
mkdir -p /tmp/build/etc/redis/
mkdir -p /tmp/build/opt/redis/data/
mkdir -p /tmp/build/opt/redis/modules/
mkdir -p /tmp/build/etc/systemd/system/
# Get Redis
wget https://github.com/antirez/redis/archive/5.0-rc1.tar.gz
tar zxvf 5.0-rc1.tar.gz
cd redis-5.0-rc1/
@nmilford
nmilford / performline_example.py
Last active June 6, 2018 18:05
Sample script to build a report from PerformLine API and Export
#!/usr/bin/env python
import performline.embedded.stdlib.clients.rest.exceptions
from performline.client import Client
from glob import iglob
import argparse
import json
import sys
class PlineExportReport(object):
def __init__(self, token=None, path='.'):
@nmilford
nmilford / deps.sh
Created August 9, 2017 13:27
Create Native deb packages for Splash 3.0 dependencies for Ubuntu 16.04
# Make sure to have fpm installed as well as dependancies described here: https://github.com/scrapinghub/splash/blob/master/dockerfiles/splash/provision.sh
# Environment Setup
sudo mkdir -p /tmp/{downloads,build,target}
sudo mkdir -p /tmp/{downloads,build,target}
# Build qt-5.9.1 deb
sudo curl -L -o /tmp/downloads/qt-installer-noninteractive.qs https://raw.githubusercontent.com/scrapinghub/splash/master/dockerfiles/splash/qt-installer-noninteractive.qs
sudo sed -i 's|/opt/qt59|/tmp/target/opt/qt59|g' /tmp/downloads/qt-installer-noninteractive.qs
sudo curl -L -o /tmp/downloads/qt-installer.run http://download.qt.io/official_releases/qt/5.9/5.9.1/qt-opensource-linux-x64-5.9.1.run
@nmilford
nmilford / knife_completion.sh
Created September 7, 2016 03:04
knife bash completion with knife ec2 plugin completion (extended from https://github.com/stevendanna/knife-hacks)
#!/bin/bash
#
# Bash completion for Knife 0.10+
#
if [ $(uname) = "Darwin" ]; then
SED="gsed"
else
SED="sed"
fi
@nmilford
nmilford / splash_metrics.py
Created June 2, 2016 17:12
Splash metric collection
#!/usr/bin/env python
from statsd import StatsClient
import requests
import json
statsd = StatsClient()
url = 'http://127.0.0.1:8050/_debug'
resp = requests.get(url=url)
metrics = json.loads(resp.text)
@nmilford
nmilford / convert.py
Created April 19, 2016 17:37
Uncompress and Unpickle a compressed PickledObjectField
import pickle
from base64 import b64decode
from zlib import decompress
import xml.dom.minidom
def parse_pickled_transcript(value):
value = value.encode()
value = b64decode(value)
value = decompress(value)
result = pickle.loads(value)
@nmilford
nmilford / diskcheck
Last active October 10, 2015 21:05
Simple Diskcheck Daemon
#!/usr/bin/env ruby
require 'rubygems'
require 'fileutils'
require 'trollop'
require 'pstore'
require 'logger'
require 'dante'
require 'pony'
require 'erb'

Keybase proof

I hereby claim:

  • I am nmilford on github.
  • I am nathanmilford (https://keybase.io/nathanmilford) on keybase.
  • I have a public key whose fingerprint is EF54 4B5B B9E2 F051 46E3 9433 8AE5 0ECA 22FA 3F98

To claim this, I am signing this object:

@nmilford
nmilford / mcp.go
Created July 9, 2014 03:18
Simple file ledger example in Go for Cassandra.
package main
import (
"encoding/json"
"fmt"
"github.com/gocql/gocql"
"log"
"net/http"
"strings"
"time"