Skip to content

Instantly share code, notes, and snippets.

View konklone's full-sized avatar

Eric Mill konklone

684 followers · 80 following
View GitHub Profile
@konklone
konklone / calc.rb
Created June 3, 2011 14:35
>> Time.parse("2011-01-18") + 120.days
=> Wed May 18 00:00:00 -0400 2011
@konklone
konklone / elasticsearch.log
Created June 14, 2011 19:22
ElasticSearch running out of control
# Running normally, each request made generates one or two lines here
[2011-06-14 19:11:05,753][DEBUG][monitor.jvm] [Protector] [gc][ParNew][11] took [18ms]/[252ms], reclaimed [23.1mb], leaving [57.7mb] used, max [1.2gb]
[2011-06-14 19:11:06,753][DEBUG][monitor.jvm] [Protector] [gc][ParNew][14] took [2ms]/[340ms], reclaimed [28.9mb], leaving [60.3mb] used, max [1.2gb]
[2011-06-14 19:11:21,759][DEBUG][monitor.jvm] [Protector] [gc][ParNew][17] took [3ms]/[355ms], reclaimed [27.2mb], leaving [63.6mb] used, max [1.2gb]
[2011-06-14 19:11:31,762][DEBUG][monitor.jvm] [Protector] [gc][ParNew][19] took [17ms]/[374ms], reclaimed [26.4mb], leaving [63.7mb] used, max [1.2gb]
[2011-06-14 19:11:32,763][DEBUG][monitor.jvm] [Protector] [gc][ParNew][21] took [1ms]/[384ms], reclaimed [28.9mb], leaving [63.2mb] used, max [1.2gb]
[2011-06-14 19:11:36,765][DEBUG][monitor.jvm] [Protector] [gc][ParNew][25] took [4ms]/[404ms], reclaimed [27.8mb], leaving [66.3mb] used, max [1.2gb]
[2011-06-14 19:11:40,767][DEBUG][monitor.jvm] [Prot
@konklone
konklone / legislator_hunt.rb
Created August 31, 2011 14:41
Social media account finder for members of Congress, using the source code of their website to dig up leads.
#!/usr/bin/env ruby
require 'rubygems'
require 'open-uri'
require 'sunlight'
require 'fastercsv'
Sunlight::Base.api_key = '[your api key]'
def hunt(service)
@konklone
konklone / crash.txt
Created September 1, 2011 21:32
Crash on fetching hits for a search query - scoping it to specific fields does not cause this problem.
# offending line is here:
# https://github.com/elasticsearch/elasticsearch/blob/v0.17.6/modules/elasticsearch/src/main/java/org/elasticsearch/common/xcontent/support/XContentMapConverter.java
[2011-09-01 21:06:14,822][DEBUG][rest.action.search ] [Ant-Man] failed to execute search (building response)
java.lang.NullPointerException
at org.elasticsearch.common.xcontent.support.XContentMapConverter.writeValue(XContentMapConverter.java:145)
at org.elasticsearch.common.xcontent.support.XContentMapConverter.writeIterable(XContentMapConverter.java:131)
at org.elasticsearch.common.xcontent.support.XContentMapConverter.writeValue(XContentMapConverter.java:163)
at org.elasticsearch.common.xcontent.support.XContentMapConverter.writeMap(XContentMapConverter.java:121)
@konklone
konklone / 2009Q4-detail-diff-excerpt
Created September 8, 2011 17:24
2009Q4 diff between old and new detail files (excerpt from the top portion of each half of the file)
2,120841c2,120841
< "CLERK OF THE HOUSE","2009Q4","PRINTING AND REPRODUCTION","10-30","OFFICE","","","OF THE CLERK ................................................................................... PRINTING AND REPRODUCTION ","-80.00","FISCAL YEAR 2009","IV","","09A90100147","OFFICE"
< "COMMUNICATIONS","2009Q4","OTHER SERVICES","12-23","AMERISYS INC. ","12/02/09","12/02/09","NON-TECHNOLOGY SERVICE CONTRCT ","1,300.00","FISCAL YEAR 2010","P2","purchase order","MFP0003220","AMERISYS INC. "
< "COMMUNICATIONS","2009Q4","SUPPLIES AND MATERIALS","10-30","","10/01/09","10/31/09","OFFICE SUPPLY (TRANSFER) ","92.30","FISCAL YEAR 2010","S1","goods purchased through Office Supply Store","DY091000021",""
< "COMMUNICATIONS","2009Q4","SUPPLIES AND MATERIALS","11-30","","11/01/09","11/30/09","OFFICE SUPPLY (TRANSFER) ","115.90","FISCAL YEAR 2010","S1","goods purchased through Office Supply Store","DY091100019",""
< "COMMUNICATIONS","2009Q4","EQUIPMENT","10-30"
@konklone
konklone / example.json
Created October 14, 2011 20:21
example of CW API response
{
speaker_state: "TX"
speaker_first: "Rub��n"
congress: 112
title: "A CELEBRATION OF HISPANIC HERITAGE MONTH"
origin_url: "http://origin.www.gpo.gov/fdsys/pkg/CREC-2011-09-21/html/CREC-2011-09-21-pt1-PgH6330-6.htm"
number: 141
pages: "H6330-H6331"
volume: 157
chamber: "House"
@konklone
konklone / strip.py
Created January 5, 2012 15:53 — forked from lukerosiak/strip.py
Get rid of fluff on fields in a CSV
#!/usr/bin/env python
"""
Get rid of white space and periods on the old file, and ensure the new one uses the same CSV quoting conventions, so we can run a diff without being distracted those differences.
"""
import csv
directories = ["luke", "sunlight"]
base = "2011Q3-summary"
@konklone
konklone / icpsr_to_bioguide.rb
Created January 19, 2012 20:32
Matching up ICPSR IDs to Bioguide IDs using Charles Stewart's data, and the Sunlight Labs Congress API.
# house-icpsr.csv and senate-icpsr.csv are made by converting the XLS files found here to CSV:
# http://web.mit.edu/17.251/www/data_page.html#2
# Specifically, these files that list information and IDs for members from the 103rd to 112th Congress:
# http://web.mit.edu/cstewart/www/data/house_members_103-112-1.xls
# http://web.mit.edu/cstewart/www/data/senators_103-112-1.xls
# This script looks through the two original CSVs, caches the ICPSR ID of every member from the 110th Congress onward,
# then goes through every legislator in the Sunlight Labs Congress API and tries to match them up by a combination of
# last name, state, and party.
@konklone
konklone / http-regex.txt
Created February 16, 2012 03:12
http regex
/(?=(?-mix:http):)
([a-zA-Z][-+.a-zA-Z\d]*): (?# 1: scheme)
(?:
((?:[-_.!~*'()a-zA-Z\d;?:@&=+$,]|%[a-fA-F\d]{2})(?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*) (?# 2: opaque)
|
(?:(?:
\/\/(?:
(?:(?:((?:[-_.!~*'()a-zA-Z\d;:&=+$,]|%[a-fA-F\d]{2})*)@)? (?# 3: userinfo)
(?:((?:(?:(?:[a-zA-Z\d](?:[-a-zA-Z\d]*[a-zA-Z\d])?)\.)*(?:[a-zA-Z](?:[-a-zA-Z\d]*[a-zA-Z\d])?)\.?|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|\[(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:(?:[a-fA-F\d]{1,4}:)*[a-fA-F\d]{1,4})?::(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))?)\]))(?::(\d*))?))?(?# 4: host, 5: port)
|
@konklone
konklone / 01 - original.xml
Created February 22, 2012 03:56
US Code hypothetical example JSON
<section id="33ee3034-48af-1015-8668-ff2688424973" ref-id="USC_1_sa106_zn_nm_qb_oc4_sa2_.">
<enum>2</enum>
<header>SUBSEQUENT PREPARATION AND CERTIFICATION OF PRINTED ENROLLMENT</header>
<subsection id="340c3781-48af-1015-af4d-e7a0e25bee29" ref-id="USC_1_sa106_zn_nm_qb_oc4_sa2_sba_." style="OLC">
<enum>(a)</enum>
<header>Preparation</header>
<paragraph id="342a3f67-48af-1015-b220-9ba15ae2f7d9" ref-id="USC_1_sa106_zn_nm_qb_oc4_sa2_sba_pa1_." style="OLC">
<enum>(1)</enum>
<header>In general</header>
<text>If S. 2830 is presented to the President in the form of a hand enrollment pursuant to the authority of section 1, then upon the enactment of that bill the Secretary of the Senate shall prepare a printed enrollment of the bill as in the case of a bill to which sections 106 and 107 of title 1, United States Code, apply.</text>