Justin Myers myersjustinc

We used these two scripts at the PBS NewsHour to update the live election-night results in our Vote 2012 Map Center with data from the Associated Press.

I've removed NewsHour-specific credentials, of course, as well as the part that uploaded JSON files to a PBS FTP server, so these haven't been tested in the form in which they're provided here--but retrieve_data.py downloaded the AP data files periodically from their FTP server to an internal HTTP server for our use, and update_json.py processed the files from that HTTP server into JSONP and pushed them to S3 for consumption by our users.

This script was developed to take a column of a CSV and render it as JSON. Because it was meant for U.S. choropleth maps, it makes a few assumptions about the input CSV's structure--specifically that it has at least three columns:

FIPS contains a five-digit FIPS 6-4 county code for rows with county-level data. For rows with state-level data, a two-digit FIPS 5-2 state code is used by convention but is ignored.

This takes an HTML document that contains a client-side image map ( and elements) and creates an SVG image based on the shapes described in the image map (since SVG's a more general-purpose format than the HTML image map).

The HTML document doesn't necessarily have to be well-formed XML (there's a fallback to the BeautifulSoup parser for documents with some weirdness in them), which might be useful.

Dependencies

lxml
Python (of course)

Country codes

Just thought they might be handy to have around.

Methodology

The alpha-2 codes were copied from the ISO itself. The alpha-3 and numeric codes were copied from the UN Statistics Division, and any name discrepancies between the two organizations were resolved in favor of the UN version.

	<!doctype html>
	<html>
	<head>
	<title></title>
	</head>
	<body>
	<a href="iframe_content_2.html">another page?</a>
	</body>
	</html>

	#!/usr/bin/env python
	from collections import defaultdict, namedtuple
	from ftplib import FTP
	import logging
	import re
	import sys
	try:
	import json
	except ImportError:
	import simplejson as json

	<?xml version="1.0" encoding="utf-8"?>
	<!DOCTYPE eagle SYSTEM "eagle.dtd">
	<eagle version="6.1">
	<drawing>
	<settings>
	<setting alwaysvectorfont="no"/>
	<setting verticaltext="up"/>
	</settings>
	<grid distance="0.05" unitdist="inch" unit="inch" style="lines" multiple="1" display="no" altdistance="0.025" altunitdist="inch" altunit="inch"/>
	<layers>

	var formatNumber = function( value, decimalPlaces, alwaysDecimalize ) {
	// Set default decimal formatting values if undefined
	// decimalPlaces defaults to 0
	// Sets the number of places after the decimal point, if necessary
	// alwaysDecimalize defaults to false
	// If true, always shows a decimal and the set number of decimal
	// places, even for integral values (useful with currency values)
	decimalPlaces = typeof decimalPlaces === 'undefined' ? 0 : decimalPlaces;
	alwaysDecimalize = (typeof alwaysDecimalize === 'undefined' ? false :
	alwaysDecimalize);

	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
	<html xmlns="http://www.w3.org/1999/xhtml">
	<head>
	<title>Mercator craziness</title>
	<script type="text/javascript" src="http://maps.googleapis.com/maps/api/js?sensor=false"></script>
	<script type="text/javascript">//<![CDATA[
	var map;
	var circles = [];

	function degToRad(degrees) {return degrees * Math.PI / 180;}

	// Naming everything under a main "choro" namespace isn't necessary, but it's a bit cleaner.
	var choro = {};
	choro.ctrl = {};

	// One thing to keep in mind: In this data object, state names which normally have spaces
	// continue to do so. In the image map (see map.html), they have underscores instead of
	// spaces. (This is because the HTML class attribute treats spaces as separators so an
	// element can have multiple classes.) This is why there are different .replace() calls
	// throughout this script to add or remove the underscores as necessary.
	choro.data = {