Skip to content

Instantly share code, notes, and snippets.

@nicksyna01

nicksyna01/Page_Links_Extraction.py

Created July 22, 2019 20:38
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save nicksyna01/780e48a1284141b80e76cbcbade0d1c7 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save nicksyna01/780e48a1284141b80e76cbcbade0d1c7 to your computer and use it in GitHub Desktop.
Download ZIP
Code to Extract Links of all the Pages in a Website
Raw
Page_Links_Extraction.py
import re
import urllib
import urllib.request as ur
from bs4 import BeautifulSoup
html_page = ur.urlopen("http://upscanthro.com/questions/")
soup = BeautifulSoup(html_page)
a = []
for link in soup.findAll('a', attrs={'href': re.compile("^http://")}):
a.append(link)
print(link.get('href'))
#print(a)
@nicksyna01
Copy link
Copy Markdown
Author

nicksyna01 commented Jul 22, 2019

It helps in extracting all the page links of a website

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment