This is a guide for the best practices and uses for PyNode proxies in combination with Shopify Dashe bot. This guide will hopefully help you better understand good proxy/task ratios. There are two scenarios to consider when setting up tasks and proxies. This is monitor mode and restock mode. Most of this file will be about the best ratios, but I'll add a few notes and tips in here in addition.
When Dashe is monitoring a website (that is, you are using keywords or a link to look for a product which is not currently live on the site) the software will pull a new proxy every request from the proxy list in the settings. This is generally known as rotating proxies and is used to ensure that each proxy is only used at a set interval. Note that rotating your proxies in Dashe is different from rotating proxies. Rotating proxies, when referring to purchasing them change locations on a set time interval or with every request. Dashe rotates proxies by using a different proxy from the list for each web request.
For example, if you have 10 proxies and a monitor delay of 1000ms, and you are monitoring a single Shopify store, each proxy will be used once every 10 seconds, once the program cycles through. It's important to understand how this mechanism works and how it affects banning.
Another important fact here is that Dashe will only use a single monitor thread per each Shopify store monitored. Why is this important? Because, for example, if you are monitoring 2 stores, the program will cycle through the proxy list twice as fast. You can (and should, so you understand better) experiment here by setting up some dummy tasks to monitor a bunch of different web stores. The more stores that Dashe is monitoring (assuming you're using the same number of proxies), the faster they will likely get banned.
What does this mean? In general I reccommend using 50 proxies per each site monitored. For example, if you are monitoring Kith and SNK simultaneously, use 2 instances (100 proxies) to avoid your monitor threads being banned.
From my experience, using 50 proxies with my script (a single Linode instance), it is sustainable to monitor a single Shopify store anywhere from 500-1000ms.
Proxies will ban faster when they are being used to wait for a restock. You can think of this 'ban threshold' like a machine gun cool-down time in a video game. If you fire repeatedly and continuously, the gun will overheat after a few seconds of fire, however if you burst fire on a time interval, the gun will continue to work. Weird analogy, but it's probably the best way to explain it. The threshold for banning while monitoring the site is much higher than the threshold while checking out. I.e., proxies will ban faster while tasks are waiting for restocks, because the task is monitoring the individual Shopify checkout page.
When a product goes live, Dashe will use the checkout proxy to continue checking out. You should use checkout proxies for each individual task. That way, when a task goes to checkout, it uses an isolated proxy server and not a proxy that was being used to monitor the store.
You should do this by generating an additional 50 proxies for every individual task that you want to run. However, only use one proxy from each batch as a checkout proxy. Remember, each batch of 50 is effectively the same proxy server so you must keep them isolated.
For example, you want to run 6 tasks. 3 of them for a shoe on SNK and 3 of them for a shoe on Kith. You know that both sites are dropping at 10AM EST, so the tasks have to be ran simultaneously.
In this scenario, you should generate 2 batches of 50 proxies using the script. One for Kith and one for SNK. Once you These 100 proxies should all be pasted into the proxies list in Dashe settings. If you want to test them, you only need to test a single proxy from each batch. Don't test too close to drop time. Don't test while any tasks are running. These 100 proxies are 'monitoring proxies'.
Now for each of your 6 tasks, generate another 50 proxies. One proxy from each batch should be used as the checkout proxy for each task. Even though you'll have generated 300 new proxies, only use 1 from each batch as a checkout proxy and don't use them anywhere else.
For example, you know a store is going to restock an existing product on their website. You have the link and/or variants already. (Note: if a store takes down the product page, you may still (and should, because its faster) use the size variants to monitor for restocks.
If you only have the link, I reccommend using a batch of 50 proxies (in Dashe settings) for every separate link that you are watching.
If you are using variants and the variant is live—you can ATC—but the link is not live, use the variant and dont worry about monitor proxies. This is because each task will already be at checkout while waiting for restock, so you'll only need checkout proxies.
To get the best out of these proxies, you'll need a VPS. I highly reccommend AWS for server provider.
My usual server specs are:
- m5.4xlarge
- windows 2012 r2 base (dashe wont work on 2016 server OS)
- dedicated tenacy
More will be added to this list and this guide will be continually updated as I learn more about how hard these proxies can be pushed.
- If you're only running 1-2 tasks, just use localhost as checkout proxy
- If you checkout a bunch on the same proxy, you'll likely get cancelled for fraud.
- Keep separate batches of proxies on ice during releases. If you see a (430) error in Dashe, your proxies are banned and you should change them out.