1_pdfliberation_hackathon_activity.md

Three simple steps to using this gist

1. Fork this Gist
2. Edit your version to share what your team's activity
3. There's no third step

Markdown-Cheatsheet

2_who.md

Who

Who is working together?

Name	Email	Twitter	Organization
Seamus Kraft	[email protected]	@seamuskraft	OpenGov Foundation
Ross Tsiomenko	[email protected]	n/a	OpenGov Foundation

3_challenge.md

Challenge

Which challenge are you working on?

• Amnesty International Annual Reports – Torture Incident Database
• Comprehensive Annual Financial Reports
• Federal Communications Commission Daily Releases
• House of Representatives Financial Disclosures (OpenSecrets.org)
• IRS Form 990 – Not-for-Profit Organization Reports
• New York City Council and Community Board Documents
• New York City Economic Development Commission Monthly Snapshot
• New York City Environmental Impact Statements
• US Foreign Aid Reports (USAID)
• Other: Documenting What Everyone Else is Working On

4_pdfs.md

PDF Samples

How would you categorize the PDFs?

Sample documents

PDF URL	Document Title
http://www.domain.org/docs/docurl.pdf	Report of Economic Data 2012

Content category

• Disclosure (filing, forms, report, ...)
• Legislative doc (laws, analysis, ...)
• Financial (statements, reports)
• Government statistical data
• Non-Government statistical data
• Press (press releases, statements, ...)
• Government reports
• Non-Government reports
• Directory
• Other:

Number of pages

• 1 page
• 2 to 9 pages
• 10+ pages
• 100+ pages

Other observations

• Collection includes PDFs made from scanned documents
• PDFs include hand-written text

PDF Generation

• Human authored
• Machine generated
• God only knows

5_data.md

Type of data embedded in PDF

• Simple table of data
• Complex table of data
• Multiple tables of data from document
• Table data greater than one page in length
• Highly structured form data
• Loosely structured form data
• Has human-written text
• Structure text of a report (e.g., headings, subheadings, ...)
• Other:

Desired output of data

• CSV
• JSON
• text version (e.g., markdown)

6_tools.md

Tool

What tool(s) are you using to extract the data?

Tool	How we used it
Tabula	We used it to manually select and extract a table of data

Notes

7_how.md

How

How did you extracted the desired data that produced the best results?

Improvements

What would have to be changed/added to the tool or process to achieve success?

8_results.md

Results quality

• 99%+
• 90%+
• 80%+
• 50% to 75%
• less than 50%
• utter crap

Speed

How fast is the data extracted?

• < 10 seconds
• < 30 seconds
• < 1 minute
• < 5 minutes
• < 10 minutes
• < 20 minutes
• Other:

Notes

9_code.md

Code

Please list sample code, tips, and howtos of your processing pipeline.