Skip to content

Instantly share code, notes, and snippets.

@keidrun

keidrun/README.md

Last active June 26, 2018 02:45
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save keidrun/52255e698e5b9c79fdafc19e7692f1f0 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save keidrun/52255e698e5b9c79fdafc19e7692f1f0 to your computer and use it in GitHub Desktop.
Download ZIP
How to start scraping with python and scrapy
Raw
README.md

How to start scraping with Python and Scrapy

Prerequisite

  • Python: 3.6.5
  • IPython: 6.4.0

Recommendation

  • Use VSCode Editor + Python Plugin for python coding
  • Follow autopep8 as a code style, then pip install autopep8 or simply say yes when Python Plugin ask you

Installation

How to use Scrapy

Shell

$ scrapy shell 'absolute_url_you_wanna_scrape'
# Type 'response'

Start your project

$ scrapy startproject your_project
$ cd your_project/
$ scrapy genspider your_spider_name domain_name_you_wanna_scrape
$ scrapy list
your_spider_name
# => Coding
$ scrapy crawl your_spider_name

My example

Break a leg!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment