All 31 collector pages. Each one is a versioned scraper you call with a semantic input (a keyword and a city, an ASIN, a domain, a handle) instead of a URL and selectors — published input and output schemas, an hourly health probe, and a price per delivered row.
Catalogue: https://quanticdata.io/collectors/ · API reference: https://quanticdata.io/docs/ · Examples: https://github.com/quantumproxies/collectors-api-examples
All 57 posts on quanticdata.io/blog, grouped by topic.
Every product, API and company page on quanticdata.io. Blog posts: see the QuanticData blog index gist. Collector pages: see the QuanticData collectors index gist.
| #!/usr/bin/env bash | |
| # What a server actually sees, and why rotating the user-agent alone changes nothing. | |
| # | |
| # ./qd-ua.sh # your current UA, parsed | |
| # ./qd-ua.sh "Mozilla/5.0 (…)" # parse any UA string | |
| # | |
| # Interactive lookup: https://quanticdata.io/tools/user-agent/ | |
| set -euo pipefail | |
| ECHO_SERVICE="${ECHO_SERVICE:-https://httpbin.org}" |
Chrome DevTools → Network → right-click a request → Copy as cURL is the fastest way to reproduce a request. Turning that into code by hand is the slow part.
Paste it here: quanticdata.io/tools/curl-converter — Python (requests), Node (fetch/axios), Go, PHP, Ruby, Java and more.
| """A robots.txt gate you can put in front of any crawl. | |
| python3 qd_robots_gate.py https://example.com/page https://example.com/private | |
| cat urls.txt | python3 qd_robots_gate.py --stdin > allowed.txt | |
| Standard library only, and it avoids the trap that makes most robots checks wrong: | |
| RobotFileParser.read() fetches with urllib's default user-agent. Behind Cloudflare | |
| that is frequently a 403 — and the parser treats 401/403 as "disallow everything". | |
| So a site whose robots.txt says `Allow: /` comes back as fully disallowed, and your |
Where the setting lives, what breaks, and the two patterns worth copying.
Gateways and the username syntax: QuanticData proxies · guide: how to use a proxy in n8n
The proxy setting is not in the main tab. Open the node → Options → Add Option →
| """Rotation with a retry policy that knows which failures a new IP can fix. | |
| The mistake that wastes most of a proxy budget: retrying everything. A 404 is a | |
| 404 from every exit in the world. Rotating on it costs bandwidth and gets you the | |
| same answer. | |
| worth a new exit 403, 407-after-success, 429, 503, connection reset, timeouts | |
| never worth it 400, 401, 404, 410 — the target answered, and it meant it | |
| pip install requests |
| """Authenticated proxies in Selenium + Chrome, without selenium-wire. | |
| Chrome's --proxy-server flag accepts host:port only. Credentials in the URL are | |
| ignored and you get a 407. The portable fix is a tiny MV3 extension generated at | |
| runtime that answers the auth challenge — no extra dependency, works headless. | |
| pip install selenium | |
| export QD_PROXY_USER=... QD_PROXY_PASS=... | |
| python3 qd_selenium_proxy.py us |
| /** | |
| * Puppeteer through an authenticated proxy. | |
| * | |
| * The trap: --proxy-server takes a host and port, and NOTHING ELSE. Credentials in | |
| * the URL are ignored by Chrome, and you get a 407 with no obvious explanation. | |
| * The username and password go through page.authenticate() instead — per page, | |
| * before the first navigation. | |
| * | |
| * npm i puppeteer | |
| * QD_PROXY_USER=... QD_PROXY_PASS=... node qd-puppeteer-proxy.mjs |