Skip to content

Instantly share code, notes, and snippets.

View dwillis's full-sized avatar

Derek Willis dwillis

View GitHub Profile
http://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/
http://www.nrc.gov/reading-rm/doc-collections/event-status/reactor-status/Data_gov_PowerStatus_DataDictionary.xls
root@int-stg:/mnt/apps/polling/current# rake db:create --trace
(in /mnt/apps/polling/releases/20110407204716)
rake aborted!
no such file to load -- NYTPage
/usr/local/rvm/gems/ree-1.8.7-2011.03@global/gems/nytpage-0.1.0/lib/nytpage/railtie.rb:1:in `require'
/usr/local/rvm/gems/ree-1.8.7-2011.03@global/gems/nytpage-0.1.0/lib/nytpage/railtie.rb:1
/usr/local/rvm/gems/ree-1.8.7-2011.03@global/gems/nytpage-0.1.0/lib/nytpage.rb:3:in `require'
/usr/local/rvm/gems/ree-1.8.7-2011.03@global/gems/nytpage-0.1.0/lib/nytpage.rb:3
/usr/local/rvm/gems/ree-1.8.7-2011.03@global/gems/bundler-1.0.10/lib/bundler/runtime.rb:68:in `require'
/usr/local/rvm/gems/ree-1.8.7-2011.03@global/gems/bundler-1.0.10/lib/bundler/runtime.rb:68:in `require'
Derek-Willis-4901:politicians 200025$ rvm install rubygems 1.4.2
Removing old Rubygems files...
Installing rubygems dedicated to ree-1.8.7-2010.02...
Retrieving rubygems-1.4.2
curl: (3) <url> malformed
There was an error, please check /Users/200025/.rvm/log/ree-1.8.7-2010.02/*.log. Next we'll try to fetch via http.
curl: (3) <url> malformed
There was an error, please check /Users/200025/.rvm/log/ree-1.8.7-2010.02/*.log
@dwillis
dwillis / gist:963173
Created May 9, 2011 19:13
Wrestling with Python (with apologies to Charles Wesley)
COME, O Thou Traveller unknown,
Whom still I hold, but cannot see,
My company before is gone,
And I am left alone with Thee.
With Thee all night I mean to stay,
And wrestle till the break of day.
I need not tell Thee who I am,
My misery, or sin declare,
Thyself hast call'd me by my name,
Mr. INHOFE. Madam President, let me thank the Senator from California for allowing me to go first in this group that I am sure will appear down here to talk in morning business.
As the cap and trade continues to languish in the Senate, President Obama is trying to salvage international climate change talks that are on the brink of collapse. So he gave a climate change speech at the United Nations, hoping to inspire hope in the process marred by failure. His speech, however, fell short of expectations, offering only to talk of rising sea levels and climate refugees, sort of resurrecting things that have been refuted in the old Gore speeches.
President Obama's speeches have been delivered against a backdrop of confusion and disagreement in the international community over climate change. The European Union is angry that the Senate is stalling cap and trade. China and India refuse to accept binding emissions cuts. The New York Times admits that global temperatures ``have been stable for a decade and may e
@dwillis
dwillis / gist:1189275
Created September 2, 2011 17:46
Who Needs to Write Scrapers?
cd my_download_dir
curl http://apps.sd.gov/applications/ST12ODRS/LobbyistViewlist.asp?start=[1-8101:20] -o "lobby_#1.html"
# get a cup of coffee
cat lobby*.html > master.html # on mac/linux, on windows try copy lobby*.html master.html
-----
Then open master.html in a text editor, do some smart bulk find and replaces, and you'll be in business. After opening the cleaned up file in Excel, you'll need to un-merge the cells and do a little more cleanup, but it works.
@dwillis
dwillis / gist:1263335
Created October 5, 2011 01:04
RemoteTable CSV example
ree-1.8.7-2011.03 :001 > require 'rubygems'
=> true
ree-1.8.7-2011.03 :002 > require 'remote_table'
[remote_table] Ruby 1.8 detected, setting $KCODE to UTF8 so that ActiveSupport::Multibyte works properly.
=> true
ree-1.8.7-2011.03 :009 > table = RemoteTable.new "http://www.irs.gov/pub/irs-soi/stateoutflow0809.csv", :headers => true
=> #<RemoteTable:0x101c22750 @url="http://www.irs.gov/pub/irs-soi/stateoutflow0809.csv", @options={"headers"=>true}>
ree-1.8.7-2011.03 :010 > entry = table.entries.first
=> #<OrderedHash {"County_Code_Dest"=>"000", "State_Code_Origin"=>"00", "Return_Num"=>"3134167", "State_Abbrv"=>"US", "County_Code_Origin"=>"000", "State_Name"=>"US Total Mig - US & For", "Exmpt_Num"=>"5834133", "Aggr_AGI"=>"150297046", "State_Code_Dest"=>"96"}>
ree-1.8.7-2011.03 :011 > entry['Return_Num'].to_i
@dwillis
dwillis / gist:1263343
Created October 5, 2011 01:08
RemoteTable Fixed-Width Example
table = RemoteTable.new "ftp://ftp.fec.gov/FEC/webl12.zip", :filename => 'FECWEB/webl12.dat',
:format => :fixed_width, :cut => '-52,183-',
:schema => [
['candid', 9, { :type => :string }],
['name', 38, { :type => :string }],
['ico', 1, { :type => :string }],
['party1', 1, { :type => :string }],
['party3', 3, { :type => :string }],
['state', 2, { :type => :string }],
['district', 2, { :type => :string }],
>>> str = 'this has multiple spaces in multiple places'
>>> " ".join(str.split())
'this has multiple spaces in multiple places'
# sets is_super_pac boolean to false for PACs registered by Josue LaRose.
# here's why: http://www.tcpalm.com/news/2010/oct/05/self-styled-billionaire-single-handedly-doubles/?print=1
def self.remove_fake_superpacs!
update_all('is_super_pac = 0', 'treasurer= "LAROSE, JOSUE"')
end