Skip to content

Instantly share code, notes, and snippets.

View mvexel's full-sized avatar

Martijn van Exel mvexel

153 followers · 32 following
View GitHub Profile
@mvexel
mvexel / query-states-geojson.sql
Created October 18, 2017 16:11
query on a TIGER states table (Imported from Census SHAPE using shp2pgsql) to generate GeoJSON for each that osmium will understand.
COPY (
SELECT name, jsonb_build_object(
'type', 'Feature',
'id', gid,
'geometry', ST_AsGeoJSON(geom)::jsonb,
'properties', to_jsonb(row) - 'gid' - 'geom')
FROM (SELECT * FROM cb_2016_us_state_500k) row)
TO '/Users/martijnv/osm/planet/states.tsv';
@mvexel
mvexel / gist:5273c69d6b58587b5004e0a53ae34e75
Created October 18, 2017 16:10
osmium command to process N-America history into states using config file (see other gist)
osmium extract -H --overwrite -c osmium-config.json north-america.osh.pbf
@mvexel
mvexel / osmium-config.json
Created October 18, 2017 16:09
osmium config file to extract state history PBFs
{
"extracts": [
{
"description": "Full history PBF of Alabama",
"output": "Alabama.osh.pbf",
"output_format": "pbf",
"polygon": {
"file_name": "Alabama.geojson",
"file_type": "geojson"
}
@mvexel
mvexel / states.tsv
Created October 18, 2017 16:08
GeoJSON of all states (wrapped as Features this time with all TIGER attributes as properties)
We can't make this file beautiful and searchable because it's too large.
Alabama {"id": 1, "type": "Feature", "geometry": {"type": "MultiPolygon", "coordinates": [[[[-88.053375, 30.506987], [-88.051088, 30.508857], [-88.045647, 30.513306], [-88.041966, 30.517376], [-88.03867, 30.520405], [-88.035099, 30.521967], [-88.032407, 30.521447], [-88.029165, 30.519648], [-88.026222, 30.516609], [-88.023506, 30.492573], [-88.02433, 30.490774], [-88.026748, 30.491105], [-88.045043, 30.501189], [-88.052349, 30.505591], [-88.053375, 30.506987]]], [[[-88.211209, 30.322249], [-88.209999, 30.323199], [-88.209559, 30.322202], [-88.208954, 30.320445], [-88.208073, 30.319543], [-88.208733, 30.318878], [-88.209559, 30.31902], [-88.210824, 30.320777], [-88.211209, 30.322249]]], [[[-88.22512, 30.321802], [-88.221281, 30.322233], [-88.218694, 30.321903], [-88.221031, 30.32039], [-88.224653, 30.319886], [-88.226522, 30.320793], [-88.22512, 30.321802]]], [[[-88.264659, 30.335541], [-88.263674, 30.336335], [-88.261506, 30.335485], [-88.260061, 30.336449], [-88.259141, 30.337129], [-88.259141, 30.339], [-88
@mvexel
mvexel / states.txt
Created October 18, 2017 15:08
US state boundaries in geojson format
This file has been truncated, but you can view the full file.
Alabama {"type":"MultiPolygon","coordinates":[[[[-88.053375,30.506987],[-88.051088,30.508857],[-88.045647,30.513306],[-88.041966,30.517376],[-88.03867,30.520405],[-88.035099,30.521967],[-88.032407,30.521447],[-88.029165,30.519648],[-88.026222,30.516609],[-88.023506,30.492573],[-88.02433,30.490774],[-88.026748,30.491105],[-88.045043,30.501189],[-88.052349,30.505591],[-88.053375,30.506987]]],[[[-88.211209,30.322249],[-88.209999,30.323199],[-88.209559,30.322202],[-88.208954,30.320445],[-88.208073,30.319543],[-88.208733,30.318878],[-88.209559,30.31902],[-88.210824,30.320777],[-88.211209,30.322249]]],[[[-88.22512,30.321802],[-88.221281,30.322233],[-88.218694,30.321903],[-88.221031,30.32039],[-88.224653,30.319886],[-88.226522,30.320793],[-88.22512,30.321802]]],[[[-88.264659,30.335541],[-88.263674,30.336335],[-88.261506,30.335485],[-88.260061,30.336449],[-88.259141,30.337129],[-88.259141,30.339],[-88.258418,30.340417],[-88.257827,30.342742],[-88.257762,30.344329],[-88.259732,30.345633],[-88.262097,30.346541],[-88.26
@mvexel
mvexel / how_to_count.md
Last active October 17, 2017 22:06
how to generate a time series of object counts from an OSM history planet.

First, get your history extract at Geofabrik. Every region they define has an osh.pbf file.

Then, fork this gist and replace utah with the region name. Run it.

You end up with a bunch of files:

...
-rw-r--r--    1 martijnv    25M Oct 17 15:48 utah-20100101.osm.pbf
-rw-r--r--    1 martijnv    25M Oct 17 15:48 utah-20100201.osm.pbf
@mvexel
mvexel / generate_time_slices.sh
Created October 17, 2017 21:54
Generate time slices from a full history PBF file (HACK ALERT)
#!/bin/sh
osmium time-filter utah.osh.pbf 2007-01-01T00:00:00Z -o utah-20070101.osm.pbf
osmium time-filter utah.osh.pbf 2007-02-01T00:00:00Z -o utah-20070201.osm.pbf
osmium time-filter utah.osh.pbf 2007-03-01T00:00:00Z -o utah-20070301.osm.pbf
osmium time-filter utah.osh.pbf 2007-04-01T00:00:00Z -o utah-20070401.osm.pbf
osmium time-filter utah.osh.pbf 2007-05-01T00:00:00Z -o utah-20070501.osm.pbf
osmium time-filter utah.osh.pbf 2007-06-01T00:00:00Z -o utah-20070601.osm.pbf
osmium time-filter utah.osh.pbf 2007-07-01T00:00:00Z -o utah-20070701.osm.pbf
osmium time-filter utah.osh.pbf 2007-08-01T00:00:00Z -o utah-20070801.osm.pbf
@mvexel
mvexel / count_pbfs.py
Created October 17, 2017 21:53
Count objects in a series of OSM PBF files
"""
Count objects in a series of OSM files.
Build a CSV file with object counts for a series of PBF
time slices generated from a history file
"""
from __future__ import print_function
import osmium as o
from os.path import exists
import csv
@mvexel
mvexel / utah_bicycle_osm.geojson
Created October 4, 2017 18:25
http://overpass-turbo.eu/s/s99
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@mvexel
mvexel / create-postgis-tiger-boudaries.sh
Last active September 20, 2017 15:36
mkdir -p ~/tmp/tiger
cd ~/tmp/tiger
wget http://www2.census.gov/geo/tiger/GENZ2016/shp/cb_2016_us_county_500k.zip
unzip cb_2016_us_county_500k.zip
wget http://www2.census.gov/geo/tiger/GENZ2016/shp/cb_2016_us_state_500k.zip
unzip cb_2016_us_state_500k.zip
createdb tiger
psql -d tiger -c 'create extension postgis'
shp2pgsql -D -I cb_2016_us_state_500k.shp > cb_2016_us_state_500k.sql
shp2pgsql -D -I cb_2016_us_county_500k.shp > cb_2016_us_county_500k.sql