Skip to content

Instantly share code, notes, and snippets.

@mhsmathew
mhsmathew / form-automation.py
Created November 27, 2020 20:32
Using selenium to automate form submissions
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Set chrome options object so you can set the size and headless preference
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1920x1080")
# Load the driver
chrome_driver = "chromedriver"
@mhsmathew
mhsmathew / proxy-scraping.py
Created November 27, 2020 20:07
More complex web scraping with proxies and headers
import requests
from bs4 import BeautifulSoup
from proxymanager import ProxyManager
import time
proxy_manager = ProxyManager('proxies.txt')
headers = {
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'Upgrade-Insecure-Requests': '1',
@mhsmathew
mhsmathew / basic-scraping.py
Created November 27, 2020 19:53
Basic web scraping monitor
import requests
from bs4 import BeautifulSoup
import time
def update():
request = requests.get('https://mathewsteininger.com/sample-product')
html = request.content
soup = BeautifulSoup(html, 'html.parser')
# Monitor this tag to check stock changes
stock = soup.find("button", {"id": "availability"})