Simon Willison simonw

How to run a redis server using Docker-for-Mac

This will work with other flavours of Docker as well.

Run the following command:

docker run --name my-redis -p 6379:6379 --restart always --detach redis

This will download and run redis, set to auto-restart when your machine starts and bound to port 6379 on your machine.

React debouncing pattern for API calls

Classic debounce function:

const debounce = (fn, delay) => {
      let timer = null;
      return function (...args) {
          const context = this;
          timer && clearTimeout(timer);

timer = setTimeout(() => {

How I mada a Datasette out of a feather file

Fantastic data journalism by Christine Zhang at the LA Times: https://github.com/datadesk/homeless-arrests-analysis

First I grabbed the data - a zipped feather file:

wget https://github.com/datadesk/homeless-arrests-analysis/blob/master/arrests.zip?raw=true
mv arrests.zip\?raw\=true arrests.zip
unzip arrests.zip

How I fixed a tricky merge conflict

I switched to another branch before the conflict.

git checkout -b fixbranch

Then I cherry-picked each commit that was failing to merge, one at a time.

For each of those cherry-picked commits I manually cleaned up any conflicts.

How I made a Who's On First subset database

Using whosonfirst-data-latest.db from https://dist.whosonfirst.org/sqlite/ - technique inspired by Paul Ford: https://twitter.com/ftrain/status/957833410017361921

If you fire up the sqlite3 CLI against an existing database you can and then mount a brand new database using attach database - then you can create tables in that new database and populate them using a select statement, extracting data from JSON columns using the json_extract() function.

Having populated the new table, I also create a full-text index against the name column.

Here's an example query against the resulting database, using a wildcard to implement prefix autocomplete: https://whosonfirst-metadata-only.now.sh/whosonfirst-metadata-only-c11ebe3/whosonfirst?_search=san+fra%2A

Making a Datasette out of Freebase species data

I stumbled across an old export of Freebase data we used to build WildlifeNearYou:

https://github.com/devfort/wildlifenearyou/blob/master/freebase_data/data/organism_classification.tsv

It's a 6.89MB TSV file. Here's how I turned it into a Datasette instance at https://organisms.now.sh/

cd /tmp

wget https://raw.githubusercontent.com/devfort/wildlifenearyou/master/freebase_data/data/organism_classification.tsv

How to turn a list of JSON objects into a Datasette

This repository has a dataset of 184.879 crimes committed in Buenos Aires: https://github.com/ramadis/delitos-caba

Download the raw data like this:

wget 'https://github.com/ramadis/delitos-caba/releases/download/3.0/delitos.json'

Now use Pandas to load that into a dataframe:

	{
	"$schema": "https://vega.github.io/schema/vega-lite/v2.json",
	"width": 500,
	"height": 500,
	"data": {
	"url": "https://vice-police-shootings.now.sh/vice-bc7c892.jsono?sql=select+ViceNews_FullOISData.rowid%2CDate%2C+NumberOfSubjects%2C%0D%0A++Fatal.value+as+Fatal%2C%0D%0A++SubjectArmed.value+as+SubjectArmed%2C%0D%0A++SubjectRace.value+as+SubjectRace%2C%0D%0A++SubjectGender.value+as+SubjectGender%2C%0D%0A++Department.value+as+Department%2C%0D%0A++city.value+as+city%0D%0Afrom+ViceNews_FullOISData%0D%0A++join+Fatal+on+ViceNews_FullOISData.Fatal+%3D+Fatal.id%0D%0A++join+SubjectArmed+on+ViceNews_FullOISData.SubjectArmed+%3D+SubjectArmed.id%0D%0A++join+SubjectRace+on+ViceNews_FullOISData.SubjectRace+%3D+SubjectRace.id%0D%0A++join+SubjectGender+on+ViceNews_FullOISData.SubjectGender+%3D+SubjectGender.id%0D%0A++join+Department+on+ViceNews_FullOISData.Department+%3D+Department.id%0D%0A++join+city+on+ViceNews_FullOISData.city+%3D+city.id%0D%0Aorder+by+ViceNews_FullOISData.rowid+limit+1000",
	"format": {

	{
	"$schema": "https://vega.github.io/schema/vega-lite/v2.json",
	"width": 500,
	"height": 500,
	"data": {
	"url": "https://vice-police-shootings.now.sh/vice-bc7c892.jsono?sql=select+ViceNews_FullOISData.rowid%2CDate%2C+NumberOfSubjects%2C%0D%0A++Fatal.value+as+Fatal%2C%0D%0A++SubjectArmed.value+as+SubjectArmed%2C%0D%0A++SubjectRace.value+as+SubjectRace%2C%0D%0A++SubjectGender.value+as+SubjectGender%2C%0D%0A++Department.value+as+Department%2C%0D%0A++city.value+as+city%0D%0Afrom+ViceNews_FullOISData%0D%0A++join+Fatal+on+ViceNews_FullOISData.Fatal+%3D+Fatal.id%0D%0A++join+SubjectArmed+on+ViceNews_FullOISData.SubjectArmed+%3D+SubjectArmed.id%0D%0A++join+SubjectRace+on+ViceNews_FullOISData.SubjectRace+%3D+SubjectRace.id%0D%0A++join+SubjectGender+on+ViceNews_FullOISData.SubjectGender+%3D+SubjectGender.id%0D%0A++join+Department+on+ViceNews_FullOISData.Department+%3D+Department.id%0D%0A++join+city+on+ViceNews_FullOISData.city+%3D+city.id%0D%0Aorder+by+ViceNews_FullOISData.rowid+limit+1000",
	"format": {

	from django.db.models import Value, CharField


	def combined_recent(limit, **kwargs):
	datetime_field = kwargs.pop('datetime_field', 'created')
	querysets = []
	for key, queryset in kwargs.items():
	querysets.append(
	queryset.annotate(
	recent_changes_type=Value(