Skip to content

Instantly share code, notes, and snippets.

View simonw's full-sized avatar

Simon Willison simonw

View GitHub Profile
@simonw
simonw / sqlite_progress.py
Last active January 18, 2021 04:45
Updating SQLite using a custom function and a click-powered progress bar
import click
import sqlite3
import dateparser
count = 0
c = sqlite3.connect("tweets.db")
with click.progressbar(
length=c.execute("select count(*) from tweets").fetchone()[0],
We can't make this file beautiful and searchable because it's too large.
gbifid datasetkey occurrenceid kingdom phylum class order family genus species infraspecificepithet taxonrank scientificname countrycode locality publishingorgkey decimallatitude decimallongitude coordinateuncertaintyinmeters coordinateprecision elevation elevationaccuracy depth depthaccuracy eventdate day month year taxonkey specieskey basisofrecord institutioncode collectioncode catalognumber recordnumber identifiedby license rightsholder recordedby typestatus establishmentmeans lastinterpreted mediatype issue
1837837577 f93694b5-e994-4c07-a95f-90b14d5cf5e7 urn:catalog:CMNH:Birds:74925 Animalia Chordata Aves Passeriformes Emberizidae Zonotrichia Zonotrichia albicollis SPECIES Zonotrichia albicollis (J.F.Gmelin, 1789) CA Municipal District of Bonnyville, Cold Lake 9c252ec3-7333-4cf8-a997-5441fcbad8dd 54.60115 -110.199233 2016-06-19T00:00:00Z 19 6 2016 5231140 5231140 PRESERVED_SPECIMEN CMNH Birds 74925 CC0_1_0 Andrew W. Jones, Benjamin M. Winger, Courtney L. Brennan NATIVE 2018-07-07T17:06:15.491Z
@simonw
simonw / Dockerfile
Created October 29, 2018 20:52
Lightweight Dockerfile for running a Python 3 webapp under the Zeit 100MB limit
FROM alpine:edge
RUN apk add --update \
python3 \
python3-dev \
build-base \
&& python3 -m ensurepip \
&& rm -r /usr/lib/python*/ensurepip \
&& pip3 install --upgrade pip setuptools \
&& pip3 install starlette uvicorn \
@simonw
simonw / a-yubikey-dvorak-karabiner.md
Last active July 22, 2022 17:05
How I got a Yubikey working with a Dvorak keyboard and Karabiner-Elements

How I got a Yubikey working with a Dvorak keyboard and Karabiner-Elements

I needed to set up a new Yubikey device, which plugs into my Mac laptop via USB-C and injects one-time codes by pretending to be a keyboard.

Problem: I use Dvorak... so the Yubikey codes were coming through incorrectly and failing to work.

https://superuser.do/2014/10/22/yubikey-support-for-non-qwerty-keyboard-layouts-on-mac-os-x/ has a recipe for fixing this using Karabiner. But... those instructions are for an older Karabiner which is configured using XML files.

The latest Karabiner - Karabiner-Elements - uses a JSON configuration instead.

@simonw
simonw / import_github.md
Created October 31, 2018 18:23
How to import a GitHub repository as a subdirectory of a new repository while maintaining commits and datestamps

How to import a GitHub repository as a subdirectory of a new repository while maintaining commits and datestamps

There is probably a better way to do this, but this worked for me.

I had a repository called docsearch that I had been building a prototype in.

I wanted to move the contents of that repository into an existing repository called search_experiments - but I wanted the contents to live in a docsearch/ subdirectory rather than living in the root of the repo.

I solved this using the combination of git format-patch and git apply.

@simonw
simonw / Dockerfile
Last active December 16, 2024 06:21
The Dockerfile used by the new Datasette Publish to generate images that are smaller than 100MB
FROM python:3.6-slim-stretch as csvbuilder
# This one uses csvs-to-sqlite to compile the DB, and then uses datasette
# inspect to generate inspect-data.json Compiling pandas takes way too long
# under alpine so we use slim-stretch for this one instead.
RUN apt-get update && apt-get install -y python3-dev gcc
COPY *.csv csvs/
RUN pip install csvs-to-sqlite datasette
RUN csvs-to-sqlite csvs/names.csv data.db -f "name" -c "legislature" -c "country"
@simonw
simonw / gargoyle-selective-exclude.md
Last active June 17, 2019 22:13
How gargoyle selective exclude rules work
@simonw
simonw / demo_bm25_bug.py
Last active January 6, 2019 22:06
Demonstrating a bug in Peewee's bm25 function - see https://github.com/coleifer/peewee/issues/1826
import math
import struct
import sqlite3
conn = sqlite3.connect(":memory:")
conn.executescript("""
CREATE VIRTUAL TABLE docs USING fts4(c0, c1);
INSERT INTO docs (c0, c1) VALUES ("this is about a dog", "more about that dog dog");
INSERT INTO docs (c0, c1) VALUES ("this is about a cat", "stuff on that cat cat");
@simonw
simonw / toss-up-one-liner.md
Last active January 21, 2019 04:21
toss-up.now.sh one-liner

Bash one-liner I used to create toss-up.now.sh

git clone https://github.com/dwillis/toss-up \
    && csvs-to-sqlite toss-up/data/*.csv toss-up.db \
    && datasette publish now toss-up.db 
        --source_url=https://github.com/dwillis/toss-up \
        --install=datasette-vega \
        --install=datasette-cluster-map \
        --alias=toss-up.now.sh