Rufus Pollock rufuspollock

This is a short introduction to how to administer an Open Data Census.

Note: it assumes that a census instance has been booted for you (and is not about the technical side of deploying a census instance)

[toc]

Overview of How a Census is Structured

A Census is a survey built around 4 axes:

[notitle] [fullwidth]

Get your local open data census here!
Local data is often the most relevant to citizens on a daily basis - from rubbish collection times to local tax rates.
At the moment it’s hard to know what local open data is available.

HDX Common Humanitarian Dataset data into CKAN instance (we used datahub.io for convenience).

http://datahub.io/dataset/hdx-common-humanitarian-dataset

We've loaded (indicator) value table and indicator table separately in the CKAN DataStore (we have not bothered loading dataset table for the present) and we've also created a python script to automate this (which can also serve as an example of how to work with CKAN API).

Setting this up was pretty fast (most of the work was actually tidying up the data and then making some scripts to make this repeatable and testable).

Challenge 1

Your task: write a script to get a nice CSV file of natural gas prices.

Please publish your results in a git repo or a gist. Please include both script and your resulting data -- so the CSV files should be stored in the repo too!

More detail:

Prices should be Henry Hub gas prices. Use EIA data here: http://www.eia.gov/dnav/ng/hist/rngwhhdm.htm
Hint: you can get the data from any data source on the page ...

Currently Data Packagese must have a name attribute but do not have an id attribute.

There has been debate about both the semantics (e.g. uniqueness) of the name field and its usability for certain cases (e.g. importing datasets into a new catalog) - see #220 for extensive discussions.

Proposal

Two identifier fields:

name: SHOULD be present (and certainly required for installation etc). Name is human meaningful and is designed to support both resolution (protocol to be determined) and easy use by humans e.g. in data dependencies
- (?) Have this as a MUST?

	import os
	import shutil

	def convert():
	tmp = '/tmp/togif'
	if os.path.exists(tmp):
	print('Temp path %s already exists' % tmp)
	return
	os.makedirs(tmp)

	// replace this with your CKAN website
	var ckanSite = 'http://datahub.io'

	var sql = 'Your SQL goes here';

	// =================
	// Using jQuery only
	// =================

	var data = encodeURIComponent(JSON.stringify({sql: sql}));

	year,admin1,admin2,admin3,admin4,admin5,admin6,func1,func2,econ1,econ2,fin_source,exp_type,transfer,approved,adjusted,executed
	2009,Central,101 Parliament,0101 Parliament,,,010 Central apparatus (office) of ministries and other administrative authorities,01 General purpose state services,01.01 Legislative authorities,111 Remuneration of work,111.00 Remuneration of work,Base component,Personnel,Excluding transfers,30269300,30269300,27849186
	2009,Central,101 Parliament,0101 Parliament,,,010 Central apparatus (office) of ministries and other administrative authorities,01 General purpose state services,01.01 Legislative authorities,112 Mandatory state social insurance premiums,112.00 Mandatory state social insurance premiums,Base component,Personnel,Excluding transfers,5564000,5564000,5401021
	2009,Central,101 Parliament,0101 Parliament,,,010 Central apparatus (office) of ministries and other administrative authorities,01 General purpose state services,01.01 Legislative authorities,113 Payment for goods and servic

	name: opsd-time-series

	title: Time series

	description: Load, wind and solar, prices in hourly resolution

	long_description: This data package contains different kinds of time series data relevant for power system modelling, namely electricity consumption (load) for 36 European countries as well as...

	homepage: http://data.open-power-system-data.org/time_series/2016-07-14/

Rufus Pollock rufuspollock

Overview of How a Census is Structured

Get your local open data census here! Local data is often the most relevant to citizens on a daily basis - from rubbish collection times to local tax rates. At the moment it’s hard to know what local open data is available.

Challenge 1

Proposal

Get your local open data census here!
Local data is often the most relevant to citizens on a daily basis - from rubbish collection times to local tax rates.
At the moment it’s hard to know what local open data is available.