Simon Willison simonw

Thoughts on scrapers

I really like the "Git scraping" pattern - where scrapers run inside GitHub Actions and update pretty-printed JSON files on disk, which are then committed back to the repo. More on that here: https://simonwillison.net/2020/Oct/9/git-scraping/

These are really easy to write - you just need code that produces the JSON. Sometimes that's as simple as curl -o something.json https://some-api.com/some-path.json

Then you run a script which commits and pushes the file but ONLY if that file has changed. I wrote notes on doing that here: https://til.simonwillison.net/github-actions/commit-if-file-changed

Short version of that:

	[
	{
	"name": "Alabama",
	"abbreviation": "AL",
	"fips": "01"
	},
	{
	"name": "Alaska",
	"abbreviation": "AK",
	"fips": "02"

	Summary Level,State Code (FIPS),County Code (FIPS),County Subdivision Code (FIPS),Place Code (FIPS),Consolidtated City Code (FIPS),Area Name (including legal/statistical area description)
	010,00,000,00000,00000,00000,United States
	040,01,000,00000,00000,00000,Alabama
	050,01,001,00000,00000,00000,Autauga County
	050,01,003,00000,00000,00000,Baldwin County
	050,01,005,00000,00000,00000,Barbour County
	050,01,007,00000,00000,00000,Bibb County
	050,01,009,00000,00000,00000,Blount County
	050,01,011,00000,00000,00000,Bullock County
	050,01,013,00000,00000,00000,Butler County

	import toga
	from toga.style import Pack
	from toga.style.pack import COLUMN, ROW


	class HelloWorld(toga.App):
	def startup(self):
	main_box = toga.Box(style=Pack(direction=COLUMN))

	name_label = toga.Label("Your name: ", style=Pack(padding=(0, 5)))

	curl -s "https://github-to-sqlite.dogsheep.net/github.csv?sql=$(echo '
	select
	full_name
	from
	repos
	where
	rowid in (
	select
	repos.rowid
	from

	from collections import OrderedDict
	import httpx, dateutil, yaml

	url = "https://raw.githubusercontent.com/simonw/datasette/8ae0f9f7f0d644b0161165a1084f53acd2786f7c/README.md"

	content = httpx.get(url).text
	news = content.split("## News")[1].split("## Installation")[0].strip()

	# If a line starts with `* ` then it is a news item - if it does not it is a continuation of the previous item
	items = []

	import resource
	from datasette import hookimpl
	from datasette.utils.asgi import Response


	def rusage():
	return Response.text(
	repr(resource.getrusage(resource.RUSAGE_SELF))
	.replace(", ", ",\n ")
	.replace("(", "(\n ")

	Requirement already satisfied: datasette>=0.50.2 in /Users/simon/.local/share/virtualenvs/pipnext-HNJm0oR8/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (0.50.2)
	Collecting sqlite-utils>=2.17
	Using cached sqlite_utils-2.22-py3-none-any.whl (30 kB)
	Requirement already satisfied: click-default-group~=1.2.2 in /Users/simon/.local/share/virtualenvs/pipnext-HNJm0oR8/lib/python3.8/site-packages (from datasette>=0.50.2->-r requirements.txt (line 2)) (1.2.2)
	Requirement already satisfied: click~=7.1.1 in /Users/simon/.local/share/virtualenvs/pipnext-HNJm0oR8/lib/python3.8/site-packages (from datasette>=0.50.2->-r requirements.txt (line 2)) (7.1.2)
	Collecting twitter-to-sqlite~=0.20
	Using cached twitter_to_sqlite-0.21.3-py3-none-any.whl (26 kB)
	Collecting inaturalist-to-sqlite~=0.2
	Using cached inaturalist_to_sqlite-0.2-py3-none-any.whl (9.0 kB)
	Requirement already satisfied: click~=7.1.1 in /Users/simon/.local/share/virtualenvs/pipnext-HNJm0oR8/lib/python3.8/site-packages (from datasette>=0.50