Kiran S Hombal KSTARK007

🎯

Focusing

PhD Student CS UIUC. Systems researcher ( Linux and ESXi kernel )

KSTARK007 / crawler.md

Created September 13, 2017 22:17 — forked from typehorror/crawler.md

Simple Website Crawler (in python)

Simple Website Crawler

The following gist is an extract of the article Building a simple crawler. It allows crawling from a URL and for a given number of bounce.

from crawler import Crawler
crawler = Crawler()
crawler.crawl('http://techcrunch.com/')