Skip to content

Instantly share code, notes, and snippets.

@alexpetralia

alexpetralia/igm_scrape_detail_1.py

Created September 25, 2016 01:28
Show Gist options
  • Save alexpetralia/77e5b0ebeb174a00355eb245116dd6e6 to your computer and use it in GitHub Desktop.
Save alexpetralia/77e5b0ebeb174a00355eb245116dd6e6 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
igm_scrape_detail_1.py
data = []
for topic, handle in topics.items():
print(handle)
response = requests.get(domain + handle)
soup = BeautifulSoup(response.text, 'lxml')
survey_questions = list('ABCD')[:len(soup.findAll('h3', {'class': 'surveyQuestion'}))]
survey_date = soup.find('h6').get_text()
tables = soup.findAll('table', {'class': 'responseDetail'})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment