Pulling ACS data from Infochimps Geo API in R

infochimps_geo.R

library(RJSONIO)
library(ggplot2)

api.uri <- "http://api.infochimps.com/"
acs.topline <- "social/demographics/us_census/topline/search?"
api.key <- "apikey=xxxxxxxxxx" # replace the x's with your Infochimps API key

radius <- 10000  # in meters
lat <- 44.768202
long <- -91.491603

uri <- paste(api.uri, acs.topline, api.key, "&g.radius=", radius, "&g.latitude=", lat, "&g.longitude=", long, sep="")

raw.data <- readLines(uri, warn="F")
results <- fromJSON(raw.data)

## Special thanks to Patrick Hausmann for the GetData function
GetData <- function(x) {
    L <- vector(mode="list", length = x$total)

    a1 <- sapply(x$results, function(z) sapply(z, length) )
    xn <- names( which(apply(a1, 1, function(z) all(z == 1) )) )
    a2 <- lapply(x$results, function(z) z[names(z) %in% xn] )

    for (i in seq_along(a2)  ) {
       x1 <- a2[[i]]
       x2 <- data.frame(x1)
       L[[i]] <- x2
    }

    x4 <- do.call(rbind, L)
    return(x4)
}

md <- GetData(results)
str(md)

qplot(median_household_income, data=md, geom="density")

Before Line 23 'qplot' there should be a conversion from factor to numeric as in your blog

md$median_household_income = as.numeric(as.character(md$median_household_income))
md$median_housing_value = as.numeric(as.character(md$median_housing_value))

If you know the columns that exists for every 'geography_name' the function 'GetData'

provides a way to avoid the manually conversion from factor to numeric

results <- fromJSON(raw.data)

GetData <- function(x, xcolumns) {

L <- vector(mode="list", length = length(x$results))

for (i in seq_along(x$results) ) {
   x1 <- x$results[[i]]
   x2 <- x1[lapply(x1, length) == 1]
   x3 <- data.frame(x2)
   L[[i]] <- x3
}

x4 <- lapply(L, FUN=function(x) x[xcolumns])
x4 <- do.call(rbind, x4)
return(x4)

}

md <- GetData(results, columns)
str(md)

qplot(median_household_income, data=md, geom="density")

Thanks, Patrick for the correction and the GetData function! Very nice! That really cleans up all the converting that I was doing.

Am 11.09.2011 14:22, schrieb Dave Kincaid:

Thanks, Patrick for the correction and the GetData function! Very nice! That really cleans up all the converting that I was doing.

Hi Dave, some 'improvements' of 'GetData' ... now you don't need to specify the columns explicitly. results <- fromJSON(raw.data) # Check the dimension of the variables: #1 = 1 value, 2 = 2 values (e.g. coordinates), etc. ( a1 <- sapply(results$results, function(z) sapply(z, length) ) ) # Only use variables with dimension == 1 to # bind together in a data.frame xn <- names( which(apply(a1, 1, function(z) all(z == 1) )) ) # I don't find a way to transform 'a2' into a data.frame without the # for-loop in 'GetData2'!! a2 <- lapply(results$results, function(z) z[names(z) %in% xn] ) GetData2 <- function(x) { ``` L <- vector(mode="list", length = length(x$total)) a1 <- sapply(x$results, function(z) sapply(z, length) ) xn <- names( which(apply(a1, 1, function(z) all(z == 1) )) ) a2 <- lapply(x$results, function(z) z[names(z) %in% xn] ) for (i in seq_along(a2) ) { x1 <- a2[[i]] x2 <- data.frame(x1) L[[i]] <- x2 } x4 <- do.call(rbind, L) return(x4) ``` } md <- GetData2(results) str(md) # To avoid the conversion from strings to factors just put # options(stringsAsFactors = FALSE) # at the top of the script P.S. I like our example, because here in Berlin a Opendata Portal will open next Wednesday. So playing around with JSON and an API is some fun and a perfect exercise :) Tschö Patrick

Thanks, Patrick. That works really well and is exactly what I was wanting to be able to do. I modified the Gist with your contribution.

Dave, there's an error in my example, so Line 19 should be:
L <- vector(mode="list", length = x$total)