Alex Kappel alexkapps

This is a list inspired by some of our current or potential lines of work at the World Bank Innovation Labs. The “Innovations in Big Data Analytics” program helps to strengthen the World Bank capabilities to effectively use big data in its operational and strategic work.

We are always looking for great Data Scientists. If you can solve any of these [using open software], you'll be heads down helping us from day one. Email us to [email protected]

(This list is updated frequently).

1. Nightlights from Satellite

We are building an open stack to process nightly data from satellite and query light output from all known villages. Currently we are doing 20 years of nightly data for 600,000 villages in India.

Himawari-8 animation tutorial

Here’s how to make animations like this one. It requires intermediate Unix command-line knowledge, to install some tools and to debug if they don’t work. You’ll need these utilities:

curl (or you can translate to wget)
convert and montage, part of ImageMagick
ffmpeg, plus whatever codecs
parallel, for iteration that’s nicer than shell for loops or xargs
run everything in zsh for leading 0s in numerical ranges to work

Useful Pandas Snippets

A personal diary of DataFrame munging over the years.

Data Types and Conversion

Convert Series datatype to numeric (will error if column has non-numeric values)
(h/t @makmanalp)

	from skimage import io, filters
	import numpy as np
	import sys

	# ./hi8-deband.py input.png output.png
	# Cherrypicked sample before/after: http://basecase.org/2016/1/hi8corr

	'''
	Himawari-8 has slight noise along scanlines. It seems basically uncorrelated
	between rows, and varies smoothly on the scale of about 100 columns.

	#!/bin/bash

	# see Development Seed landsat-api scene extent on geojson.io
	# example use: $0 LC81660362014196LGN00

	scene=$1
	# get scene metadata from devseed landsat-api
	curl -s "https://api.developmentseed.org/landsat?search=${scene}" \|\
	# get corner coords
	jq '.results[]\|[.lowerLeftCornerLongitude,.lowerLeftCornerLatitude,"",.lowerRightCornerLongitude,.lowerRightCornerLatitude,"",.upperRightCornerLongitude,.upperRightCornerLatitude,"",.upperLeftCornerLongitude,.upperLeftCornerLatitude,"",.lowerLeftCornerLongitude,.lowerLeftCornerLatitude]\|@csv' \|\

	#!/usr/bin/env python
	# Very simple rasterio NDVI (or general normalized difference ratio) demo.
	# python ndvi.py $landsat_scene/*B{4,5}.TIF ndvi.tif

	import rasterio as rio
	from sys import argv
	import numpy as np

	math_type = np.float32 # type for internal calculations

	# This is a shell function to quickly grab the data for a given Landsat 8 tile ID from Google's servers
	# For example:
	# l8get LC81690352014169LGN00

	# The echo at the end is to remind myself of the syntax for extracting bands 8, 4, 3, and 2. (Pansharp, Red, Green, Blue)

	# On OSX this would go into your ~/.bash_profile file.
	# Requires gsutil from https://developers.google.com/storage/docs/gsutil_install
	# Most useful in conjunction with USGS' Earth Explorer: http://earthexplorer.usgs.gov/

	# These are shell functions to quickly create pansharpened RGB images ready for color correction
	# from a set of raw Landsat 8 TIF files in a given directory.

	# For example:
	# l8_pansharp LC81690352014169LGN00
	# Creates:
	# LC81690352014169LGN00_rgb_pansharp.tif

	# l8_rush_pansharp downloads a L8 tile from Google Storage (if they have it), extracts the relevant bands, then creates the pansharpened output.