Skip to content

Instantly share code, notes, and snippets.

View tgherzog's full-sized avatar

Tim Herzog tgherzog

View GitHub Profile
@tgherzog
tgherzog / maturity-model.md
Created April 19, 2022 18:20
Data Maturity Model
Stage Open License Maturity Model Data Access Maturity Model
1 Is the license prominent on the website? Is data available online?
2 Is it clearly linked from each dataset? In non-proprietary/open formats?
3 Does it include both short and long/legal versions? With sufficient metadata?
4 Standard license (CC, OdBL)? API access?
5 Conform to Open Data principles (freely available for any purpose)? Interoperable standards?

MRY Age of Indicators

Ages are relative to the previous year. An age of 0 means 2018 for the "Original Report" and 2020 for "Oct, 2021".

indicatorID indicator Original Report Oct, 2021 Change
EN.ATM.NOXE.PC Nitrous oxide emissions (metric tons of CO2 equivalent per capita) 6 2 -4
EN.ATM.METH.PC Methane emissions (metric tons of CO2 equivalent per capita) 6 2 -4
AG.LND.FRST.ZS Forest area (% of land area) 2 0 -2
EN.ATM.CO2E.PC CO2 emissions (metric tons per capita) 4 2 -2

Overview

The Data Catalog API enables users to retrieve the contents of the data catalog along with data and metadata about individual datasets and their resources.

All API endpoints begin with the protocol and hostname: https://datacatalogapi.worldbank.org

@tgherzog
tgherzog / build.py
Last active October 31, 2023 17:58
Ranking hierarchy code and output for WBG data API
import wbgapi as wb
import pandas as pd
import json
# Background: this is an attempt to make part of the data API more consistent and coherent.
# The API back-end has a mapping table that specifies the canonical database for each
# indicator when indicators appear in more than one database and the user doesn't explicitly
# specify a database, e.g.:
# https://api.worldbank.org/v2/indicator/SP.POP.TOTL (database is ambiguous)
# https://api.worldbank.org/v2/indicator/SP.POP.TOTL?source=16 (database is clear)
@tgherzog
tgherzog / apitest1.py
Last active August 27, 2021 14:46
World Bank API: indicator endpoint consistency test
'''
Script that iterates the API databases to check if the indicator endpoints are internally consistent.
See https://gist.github.com/tgherzog/f213ff55b254a940d3e18e038da01495 for sample reports
Usage:
apitest1.py [--deep] [ID]
Options:
# Run this in a Jupyter Notebook
import wbgapi as wb
import pandas as pd
from pyquery import PyQuery # JQuery for Python, sort of
pd.options.display.max_rows = 300
# scrape a list of UN member states
@tgherzog
tgherzog / terre-biodiv.py
Last active May 22, 2019 21:46
terre-biodiv data uploader
"""
Upload files to a terre-biodiv s3 bucket
The first form uploads files from a local directory to an S3 bucket
The second form uses an S3 bucket as the source
Either BUCKET or SRCBUCKET can include a path prefix to control the
copy operation, e.g., wbg-terre-biodiv/data
@tgherzog
tgherzog / pyddh_example_dataset_add.py
Last active April 3, 2019 19:52
DDH module example
# from github.com/tgherzog/pyddh - be sure to get the dkan-2018-api branch
import ddh
ddh.load('ddh1stg.prod.acquia-sites.com')
ds = ddh.dataset.ds_template()
# populate required fields
ds['title'] = 'Python test'
ds['body'] = 'Blah blah blah'
@tgherzog
tgherzog / 01-overview.md
Last active November 5, 2018 18:58
How to Deposit Energy Datasets in DDH