Skip to content

Instantly share code, notes, and snippets.

@satwikkansal
Created June 27, 2016 12:22
Show Gist options
  • Save satwikkansal/ffd6e81319123fa9b6d81fb937ab3212 to your computer and use it in GitHub Desktop.
Save satwikkansal/ffd6e81319123fa9b6d81fb937ab3212 to your computer and use it in GitHub Desktop.
Scraping top Stackoverflow posts using Scrapy
import scrapy
class Stackoverflowspider(scrapy.spider):
name = 'stackoverflow'
start_urls = ['http://stackoverflow.com/questions?sort=votes']
def parse(self, response):
for href in response.css('.question-summary h3 a::attr(href)'):
full_url = response.urljoin(href.extract())
yield scrapy.Request(full_url, callback=self.parse_question)
def parse_question(self, response):
yield {
'title': response.css('h1 a::text').extract()[0],
'votes': response.css('.question .vote-count-post::text').extract()[0],
'body': response.css('.question .post-text').extract()[0],
'tags': response.css('.question .post-tag::text').extract(),
'link': response.url,
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment