Skip to content

Instantly share code, notes, and snippets.

@saharshg29
Created December 18, 2023 12:40
Show Gist options
  • Save saharshg29/6ffdf095e2c13ec978550cc5d72198c0 to your computer and use it in GitHub Desktop.
Save saharshg29/6ffdf095e2c13ec978550cc5d72198c0 to your computer and use it in GitHub Desktop.
Infinite Scroll Snippet in Python
import time
import requests
import pandas as pd
from tqdm import tqdm
import chromedriver_binary
from bs4 import BeautifulSoup
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('https://stockmages.netlify.app')
#Scroll to end
for i in tqdm(range(0,5000000,1000)):
browser.execute_script("window.scrollTo(0," + str(i) + ")")
time.sleep(.1)
#soup of all container
soup = BeautifulSoup(browser.page_source, 'html.parser')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment