hannah cushman garland hancush

🤖 Make some data

Your mission

Write a Makefile that gets data from Data.gov and creates a CSV of the 50 most recent datasets uploaded. For each dataset, the output should include:

The name of the dataset
The date it was uploaded
A download link

Install django-storages:

pip install django-storages

Add to settings:

INSTALLED_APPS = (

Where test.txt contains:

a string      with some  extra spaces




line

Assemble web_info by iterating over table cells parsed from Legistar GUI
Assemble terms by iterating over City Council terms from Legistar API, adding members to the web_info dict as we go
Assemble members by iterating over terms, storing names as keys, and Person objects, loaded with memberships for each City Council term, as values
Continue to assemble members by iterating over body offices for each committee, storing memberships on the appropriate Person objects (or minting

16:42:54 INFO scrapelib: GET - https://webapi.legistar.com/v1/nyc/matters/53903/texts/55398
Traceback (most recent call last):
  File "/Users/Hannah/.virtualenvs/ocd/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "/Users/Hannah/.virtualenvs/ocd/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1625, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "/Users/Hannah/.virtualenvs/ocd/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1431, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

	# usage: python interval.py /path/to/your/file.csv

	import csv
	import sys
	import re


	def days_from_interval(interval):
	if interval == 'never':
	return None

	from django.db.models import Q


	condition = Q()

	employers = Employer.objects.all()

	if params:
	if params.get('name'):
	employers = employers.annotate(search_vector=SearchVector('name'))

	TABLES=$(mdb-tables /path/to/YOUR_ACCDB.accdb); \
	for TABLE in $TABLES; \
	do mdb-export /path/to/YOUR_ACCDB.accdb $TABLE > /path/to/$TABLE.csv; \
	done;

	# header

	here's what this is

	<<>>=
	import pandas as pd
	@

	<<>>=
	data = pd.read_csv('path/to/your/data') # returns a dataframe

	from make_a_formset import formset

	# populate your formset

	for form in formset:
	form_data = form.clean_data
	# {'name': 'Hannah', 'favorite_color': 'yellow'}

	# do stuff with the data