Skip to content

Instantly share code, notes, and snippets.

@tmcw
tmcw / foursquare_archive.py
Created August 14, 2012 15:19
Simple Foursquare Checkins Archive of one User
import requests, os, glob, json, sys, webbrowser
you = 'self'
data = 'checkins'
try: os.mkdir(data)
except Exception: pass
cid = 'YOUR_CLIENT_ID'
@tmcw
tmcw / foursquare_to_geojson.py
Created August 20, 2012 20:53
Turn your Foursquare Data Archive into a GeoJSON file
import glob, json
# this script loves this script
# https://gist.github.com/3350235
points = []
vids = set()
places = glob.glob("checkins/*.json")
for p in places:
@migurski
migurski / merge-geojsons.py
Created September 21, 2012 03:43
Merge multiple GeoJSON files into one
from json import load, JSONEncoder
from optparse import OptionParser
from re import compile
float_pat = compile(r'^-?\d+\.\d+(e-?\d+)?$')
charfloat_pat = compile(r'^[\[,\,]-?\d+\.\d+(e-?\d+)?$')
parser = OptionParser(usage="""%prog [options]
Group multiple GeoJSON files into one output file.

As an ex-University lecturer I could do this in class and students would enter the workforce able to make their analysis look good on a map and communicate properly… What about those who can't be bothered reading a few pages from a book or a web site that shows them some useful tips?

You understand the academic mindset, not the hacking mindset. Academics go like this:

lectures -> books -> grades -> make a good map

Hacking goes like this:

make a map -> websites -> make a better map -> make a good map
@andrewharvey
andrewharvey / 00-extract-from-log.sh
Created October 13, 2012 09:57
Web Server Logs Geographic Visualisation using Leaflet
#!/bin/sh
# To the extent possible under law, the person who associated CC0
# with this work has waived all copyright and related or neighboring
# rights to this work.
# http://creativecommons.org/publicdomain/zero/1.0/
if [ `which logrotatecat` ] ; then
# yes we found the binary
logrotatecat access.log | \ # a log-rotate aware method to cat access logs together
@iwek
iwek / find-in-json.js
Created October 20, 2012 21:43
Searching through JSON
//return an array of objects according to key, value, or key and value matching
function getObjects(obj, key, val) {
var objects = [];
for (var i in obj) {
if (!obj.hasOwnProperty(i)) continue;
if (typeof obj[i] == 'object') {
objects = objects.concat(getObjects(obj[i], key, val));
} else
//if key matches and value matches or if key matches and value is not passed (eliminating the case where key matches but passed value does not)
if (i == key && obj[i] == val || i == key && val == '') { //
@tmcw
tmcw / why.md
Created November 11, 2012 19:16

Why Shapely Instead of PostGIS

For small and medium datasets, I'd rather do things in Shapely than PostGIS.

  • Programming in SQL is weird and orphans code from my existing knowledge about portability, optimization, and so on.
  • Python scripts strike me as more reusable than SQL queries; they can make more contained assumptions about your data layout, they can be easily run as 'scripts' rather than sets of commands.
  • Installing and understanding PostGIS gives me the ragefits. Stuff designed for big complicated data does not scale well down to the silly hacks I do.
  • I/we do not have PostGIS running in production. What I do either ends up in GeoJSON in vectors or rendered by TileMill, and once it goes through TileMill, it's the same if I use something complicated or something simple.
  • Using Shapely lets you easily pull in other libraries to do other kinds of stuff. Like this delaunay map uses scipy for triangulation and Shapely/Fiona for making those thing
@emacgillavry
emacgillavry / load_dmp_2_cartodb.sh
Created November 12, 2012 13:31
Use Postgres dump files to populate CartoDB instance
#!/bin/bash
#------------------------------------------------------------
# We assume the dump files have been generated using pg_dump:
#
# pg_dump -a --column-inserts -x -O -t table_name database_name -f /tmp/dmp_file_name
#
#------------------------------------------------------------
# Provide details of your CartoDB account:
import lxml.html
def get_incidents():
incidents = []
url = "http://slonews.thetribunenews.com/police_log/slo_log.php"
tree = lxml.html.parse(url)
for inc in tree.xpath("//table[@class='report']"):
inc_keys = inc.xpath("tr/td[@class='c1']")
inc_keys = [key.text_content() for key in inc_keys]
@oeon
oeon / addfilenamecol.sh
Last active October 13, 2015 08:58
Add column with source filename from folder of shapefiles. Requires GDAL 1.10dev, my attempt to drop .shp extension isn't working - but overall, works.
for f in *.shp;
do
name=${f%.shp}
/Users/you/gdal_trunk/bin/ogrinfo $f -sql "ALTER TABLE $name ADD COLUMN filename character(21)"
/Users/you/gdal_trunk/bin/ogrinfo $f -dialect SQLite -sql "UPDATE $name SET filename = '$f'"
done;