Imports and starter code for a scraping script with BeautifulSoup and Requests

scraper_boilerplate.py

from bs4 import BeautifulSoup
import requests
hdr = {'User-Agent': 'your user-agent info here'}
# find YOUR user-agent HERE: https://www.whatismybrowser.com/detect/what-is-my-user-agent/ 

url = 'https://www.some_domain.com/some_dir'
page = requests.get(url, headers=hdr)
soup = BeautifulSoup(page.text, 'html.parser')

'''
If you have a list of URLs to scrape, you need to loop over the list, and 
make page and soup each time the loop runs.
'''