Skip to content

Instantly share code, notes, and snippets.

@rs77
Last active April 2, 2023 07:12
Show Gist options
  • Save rs77/23d9c9b6c2b271a96d322fd701364a08 to your computer and use it in GitHub Desktop.
Save rs77/23d9c9b6c2b271a96d322fd701364a08 to your computer and use it in GitHub Desktop.
Download multiple files using requests and BeautifulSoup in Python. More here: https://pythonwebdevelopment.com/download-multiple-files-on-web-page-using-requests-and-beautifulsoup/
import requests
from bs4 import BeautifulSoup
response = requests.get("https://example.com")
soup = BeautifulSoup(response.text, 'html.parser')
# define what common attributes the html tags contain
anchors = soup.find_all("a")
links = [a for a in anchors if a['href'].lower().endswith(".pdf")]
for link in links:
download_url = link["href"]
# how are you going to save the file locally?
file_name = link.text.strip()
with open(file_name, "wb") as file:
response = requests.get(download_url)
file.write(response.content)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment