Skip to content

Instantly share code, notes, and snippets.

@nailor
Created March 20, 2011 21:14
Show Gist options
  • Select an option

  • Save nailor/878686 to your computer and use it in GitHub Desktop.

Select an option

Save nailor/878686 to your computer and use it in GitHub Desktop.
Script that fetches pages from search.twitter.com as long as there's something to fetch
#!/usr/bin/python
import json
import urllib
import sys
def fetch_pages():
i = 1
while True:
result = urllib.urlopen('http://search.twitter.com/search.json?%s' % (
urllib.urlencode({
'q': '#tad011',
'rpp': 100,
'page': i,
})))
result = json.load(result)
if result['results']:
i = i+1
yield result['results']
else:
raise StopIteration
if __name__ == '__main__':
all_results = []
for page in fetch_pages():
print page
all_results.extend(page)
print >>sys.stderr, 'Total: %d tweets' % len(all_results)
json.dump(all_results, sys.stdout)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment