Simon Willison simonw

How I created dams.now.sh

Try it out at https://dams.now.sh/ - see this Twitter thread for background.

I started by grabbing the URLs to every downloadable Excel spreadsheet.

I navigated to the "Downloads (Public)" link starting from https://nid-test.sec.usace.army.mil/ - then I ran this JavaScript in my browser's console to extract all of the URLs as a JSON blob.

console.log(JSON.stringify(

Array.from(

Bash one-liner I used to create toss-up.now.sh

git clone https://github.com/dwillis/toss-up \
    && csvs-to-sqlite toss-up/data/*.csv toss-up.db \
    && datasette publish now toss-up.db 
        --source_url=https://github.com/dwillis/toss-up \
        --install=datasette-vega \
        --install=datasette-cluster-map \
        --alias=toss-up.now.sh

See this instead: https://gist.github.com/simonw/fe24758774d45ece0cf6

How to import a GitHub repository as a subdirectory of a new repository while maintaining commits and datestamps

There is probably a better way to do this, but this worked for me.

I had a repository called docsearch that I had been building a prototype in.

I wanted to move the contents of that repository into an existing repository called search_experiments - but I wanted the contents to live in a docsearch/ subdirectory rather than living in the root of the repo.

I solved this using the combination of git format-patch and git apply.

	def fetch_metadata_for_doc_ids(doc_ids, oauth_token):
	boundary = 'batch_boundary'
	headers = {
	'Authorization': 'Bearer {}'.format(oauth_token),
	'Content-Type': 'multipart/mixed; boundary=%s' % boundary,
	}
	body = ''
	for doc_id in doc_ids:
	req = 'GET https://www.googleapis.com/drive/v3/files/{}?fields=*'.format(doc_id)
	body += '--%s\n' % boundary

	import csv
	from dictdiffer import diff


	def load_trees(filepath):
	fp = csv.reader(open(filepath))
	headings = next(fp)
	rows = [dict(zip(headings, line)) for line in fp]
	return {r["TreeID"]: r for r in rows}

	csvs-to-sqlite https://candidates.democracyclub.org.uk/media/candidates-all.csv \
	--table=candidates \
	-c election \
	-f name \
	-f party_name \
	-f post_label \
	democracyclub.db

	datasette publish heroku democracyclub.db \
	--name="democracyclub-datasette" \

	[
	{
	"day": "Thursday",
	"description": "Get your badges and get some food (plus plenty of coffee), as you gear up for the first day of SRCCON!",
	"everyone": "y",
	"facilitators": "",
	"facilitators_twitter": "",
	"id": "thursday-breakfast",
	"length": "",
	"notepad": "",

	import math
	import struct
	import sqlite3


	conn = sqlite3.connect(":memory:")
	conn.executescript("""
	CREATE VIRTUAL TABLE docs USING fts4(c0, c1);
	INSERT INTO docs (c0, c1) VALUES ("this is about a dog", "more about that dog dog");
	INSERT INTO docs (c0, c1) VALUES ("this is about a cat", "stuff on that cat cat");

	FROM python:3.6-slim-stretch as csvbuilder

	# This one uses csvs-to-sqlite to compile the DB, and then uses datasette
	# inspect to generate inspect-data.json Compiling pandas takes way too long
	# under alpine so we use slim-stretch for this one instead.

	RUN apt-get update && apt-get install -y python3-dev gcc
	COPY *.csv csvs/
	RUN pip install csvs-to-sqlite datasette
	RUN csvs-to-sqlite csvs/names.csv data.db -f "name" -c "legislature" -c "country"