Skip to content

Instantly share code, notes, and snippets.

@tzvc
Created October 12, 2022 16:34
Show Gist options
  • Save tzvc/4097ae3ef2a2cdcfc1d3a04988095ffb to your computer and use it in GitHub Desktop.
Save tzvc/4097ae3ef2a2cdcfc1d3a04988095ffb to your computer and use it in GitHub Desktop.

If you're getting blocked every 4 requests, it's likely that your IP address is being flagged. One way to work around this is to use a proxy server that will automatically rotate your IP address with each request. This way, you're less likely to get flagged and blocked.

There are a few different ways to do this, but one example is to use residential proxies. These are IP addresses that are associated with real physical locations (homes, businesses, etc.), so they're less likely to be flagged as suspicious.

Some allow scoping to a particular geo region and use "sticky" IPs (use the same IP for a set amount of time)

There are a number of different providers that offer this service, so you'll need to do some research to find one that meets your needs. Once you have a proxy server set up, you can use it in your code just like you would any other proxy.

Here's how it would work in your case:

from selenium import webdriver

PROXY = "http://username:[email protected]:22225"

chrome_options = WebDriver.ChromeOptions()
options.add_argument('--proxy-server=%s' % (PROXY))
driver = webdriver.Chrome(executable_path='chromedriver.exe', chrome_options=chrome_options)

# This will use a different IP address for
# each request so you wont get blocked
driver.get('https://www.transfermarkt.es/jadon-sancho/profil/spieler/'+str(id))

Hope this helps!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment