Skip to content

Instantly share code, notes, and snippets.

View simonw's full-sized avatar

Simon Willison simonw

17.1k followers · 141 following
View GitHub Profile
@simonw
simonw / batch_indexer.py
Created October 9, 2017 17:48
Batch indexer - a mechanism for indexing documents in batches
class BatchIndexer(object):
def __init__(self, batch_size):
self.batch_size = batch_size
def __enter__(self):
self.batch = []
return self
def index(self, item):
self.batch.append(item)
@simonw
simonw / comments.md
Last active October 5, 2017 15:25
Comment thread for "Implementing faceted search with Django and PostgreSQL"
@simonw
simonw / detected-dupes.txt
Created September 10, 2017 15:50
Duplicates detected in https://irma-api.herokuapp.com/ as of 8:49am PST 10th September 2017
Narcoossee Middle School
2700 N Narcoossee Rd, St Cloud, FL 34771
latitude=28.122861, longitude=-81.272736
Liberty High School
4250 Pleasant Hill Rd, Kissimmee, FL 34746
latitude=28.122861, longitude=-81.272736
-------
Livestock Shelter: Oconee Heritage Park
2380 Athens Hwy, Madison, GA, United States
latitude=33.6284624, longitude=-83.4531959
@simonw
simonw / puppeteer.ipynb
Created August 18, 2017 19:22
Using puppeteer to take screenshots from jupyter notebook
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@simonw
simonw / trees-friday-demos.ipynb
Last active August 18, 2017 19:50
San Francisco trees demo of jupyter notebook - dataset is CSV from https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@simonw
simonw / example_output.json
Created August 3, 2017 23:43
Python function for retrieving all git commits (from "git log" in the current directory) as a useful data structure
[
{
"date": "Mon Jan 23 13:03:00 2017 -0800",
"message": "Recommended by Django https://docs.djangoproject.com/en/1.10/ref/databases/#setting-sql-mode",
"title": "SET sql_mode='STRICT_TRANS_TABLES';",
"hash": "a395f401c9a6d3e8602b80ecc729d736da871f46",
"author": "Simon Willison <simonw@example.com>"
},
{
"date": "Mon Jan 23 11:37:38 2017 -0800",
@simonw
simonw / tcpdump-elasticsearch-docker.txt
Created July 24, 2017 23:36
Using tcpdump to spy on an elasticsearch docker container
# First, attach to the docker container
bay attach elasticsearch
# Now install tcpdump
apt-get update && apt-get install tcpdump
# Now run it, sniffing all traffic to port 9200
tcpdump -A -s 0 'tcp port 9200 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
@simonw
simonw / python-virtual-environment-howto.md
Created July 18, 2017 23:53
How to set up a Python virtual environment

How to set up a Python development environment

A Python development environment is a folder which you keep your code in, plus a "virtual environment" which lets you install additional library dependencies for that project without those polluting the rest of your laptop.

mkdir my-new-python-project
cd my-new-python-project
virtualenv --python=python2.7 venv

This will create a my-new-python-project/venv folder

@simonw
simonw / transform_cleaned_to_openelex.py
Created July 2, 2017 19:24
Script for converting my cleaned-up CSV version of the 2006 TN Governor election results into openelections-compatible format.
import csv
import argparse
import sys
def convert_cleaned_to_openelex(filepath, office='Governor'):
yield (
'county', 'precinct', 'office', 'district',
'candidate', 'party', 'votes'
)
@simonw
simonw / google-sheets-unix-timestamp-formula.txt
Created June 30, 2017 19:46
How to convert a unix timestamp to a human-readable datetime string in Google Sheets
=((A1-21600000)/86400000)+25569
# Then select the column and use Format -> Number -> Date time