Skip to content

Instantly share code, notes, and snippets.

@kshirsagarsiddharth

kshirsagarsiddharth/crawler1.py

Created September 1, 2020 09:39
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save kshirsagarsiddharth/71164e22a72dfeac9f01dabc5ac5797c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save kshirsagarsiddharth/71164e22a72dfeac9f01dabc5ac5797c to your computer and use it in GitHub Desktop.
Download ZIP
crawler 1
Raw
crawler1.py
from urllib.request import Request, urlopen, URLError
from urllib.parse import urlparse
import time
import threading
import queue
from bs4 import BeautifulSoup
import ssl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment