A short python script which takes a list of IPEDS IDs, opens a browser, navigates to the Data Center and looks up each institution in the list individually, then downloads the page IPEDS presents, after expanding all collapsed content on the page. This script does not download the image files or other resources like that. It's just text and tabl…

ipeds-simple-crawler.py

import time
from splinter import Browser
browser = Browser()

inst_list = [IPEDSID-1, IPEDSID-2, IPEDSID-3, IPEDSID-n]

browser.visit("http://nces.ed.gov/ipeds/datacenter/")
browser.execute_script('javascript:doSubmit(6)')
button = browser.find_by_id('ibtnLoginLevelOne')
button.click()

for i in inst_list:
  browser.fill('tbInstitutionSearch',i)
  browser.click_link_by_partial_href('InstitutionList.aspx')
  browser.execute_script('javascript:expandAll();')
  with open('IPEDS-PROFILE-'+str(i)+'.html','w') as file:
    file.write(browser.html.encode('utf-8'))
  time.sleep(2)
  browser.back()