onyxfish

QCEW Data Loader

These scripts import the entire Bureau of Labor Statistics Quarterly Census of Employement and Wages (from 1990 to latest) into one giant PostgreSQL database.

The database created by this process will use about 100GB of disk space. Make sure you have enough space available before you start!

Configuration

Database name, table name, and more can be configured via config.sh.

onyxfish

import agate
import geojson

class LineString(agate.Aggregation):
    def __init__(self, lat_column, lng_column):
        self._lat_column_name = lat_column
        self._lng_column_name = lng_column

    def get_aggregate_data_type(self, table):
        return agate.Text()

onyxfish

onyxfish

onyxfish

Mr. Chartmaker: Impressive new responsive charting tool built by Gregor Aisch for Upshot
http://vis4.net/blog/posts/seven-features-youll-wantin-your-next-charting-tool/

markdowneyjr: Interesting (though probably fragile) approach to using Markdown for editing (i.e. how we use Google COPY docs)
https://github.com/veltman/markdowneyjr

Change Agent: neat, if maybe ill-advised usage of Github to version simple key/value data (written by someone at Github)
https://github.com/benbalter/change_agent

Wolfram Data Drop: Crazy new "give us all your data" platfrom that integrates with Wolfram Alpha

onyxfish

Federated login using 3rd party providers (LDAP)
Sign-in-as-a-Service
Concurrent logins with multiple accounts
Forced logins with single accounts (Drive)
API keys that use Oauth2
API keys that use don't use Oauth2
IP-restricted API keys
Domain-restricted API keys
Alternate accounts (password reset)
Account forwarding (email)

onyxfish

This script for Google Spreadsheets allows you to generate slugs for your data such as might be used for creating unique urls.

Use it like this!

#	A	B	C
1	a	b	slug
2	foo	baz bing	=slugify(A2:B4)
3	bar	BAZ
4	FOO	baz-bing

onyxfish

#!/usr/bin/env python

from collections import OrderedDict
import json

write_data = OrderedDict([
    ('a', '1'),
    ('b', '2'),
    ('c', '3')
])

onyxfish

#!/usr/bin/python
# -*- coding: utf-8 -*-

import argparse
import httplib2
import os
import sys

from apiclient import discovery
from apiclient.errors import HttpError

onyxfish

https://twitter.com/mdo/status/456520960481312769

"GitHub users: what do you love about, hate about, or want to see added to Issues?"

• Setting labels is inconsistent. On detail page it's click to set, on list its apply/remove.
• Setting assignee is inconsistent. On detail page "I" am first in the list but on list I'm alphabetized.
• Issues don't close from commit messages until merged to master. (This breaks our workflow.)
• Milestone deadlines are confusing. "Due on the 18th" rarely means "Finished by the 17th at midnight".
• "Closes #2, #3" only closes #2.