Hi Josh,
Thanks — the Enterprise Crawler model is exactly what we're after: we push URLs to your API, your crawler fetches and fully paginates each page, and you return the results to us via webhook, with a success-rate guarantee and throughput handled on your side. I'm setting up the crawler in your platform now. This email is the functional spec so your crawler/solutions team knows precisely what we need out of each page.
- We push Cardmarket product-page URLs to your API — high volume, ~7M product URLs/month at steady state (we'll validate first with the 500k-URL test you proposed), plus a smaller volume of filtered follow-up URLs (see §5).
- For each URL, your crawler renders the page, loads all offers (see §4), and returns the full HTML DOM via webhook.
- At this stage we only need the raw HTML DOM — we do all parsing ourselves. Later, if we contract, we're open to discussing you returning structured fields directly; not needed for the test.
- So we can reconcile results, please include the original submitted URL (and any job/correlation ID) alongside the HTML in each webhook payload.
- Pagination is bundled inside a single URL job — your crawler does the load-more internally, so one product = one URL = one billed unit on your side (this bundling is a big part of why this model works for us).
Example: https://www.cardmarket.com/en/Pokemon/Products?idProduct=273532
Each product page is a list of marketplace offers for one card. Every offer row contains: seller (and seller type — Professional / Powerseller / Private), language (shown as a flag), condition (Mint → Poor), price, quantity, and seller country.
We build a "cheapest-price matrix" per card. The important mechanics:
- Cardmarket sorts offers by price, cheapest first. So walking the offer list top-to-bottom, the first time a given combination appears, that offer is the cheapest for that combination.
- The combination we key on is (language × condition) — e.g. cheapest English / Near-Mint, cheapest Japanese / Mint, cheapest German / Played, etc.
- Potentially plus seller type (Pro vs non-Pro) as a third dimension — i.e. cheapest professional offer vs cheapest private offer for each language×condition cell.
So per page, the signal we care about is: the first (lowest-priced) offer for each (language, condition [, seller-type]) cell. We don't need every offer — but we do need enough of the price-sorted list loaded that we're confident we've seen the cheapest of each cell. That's why full pagination matters.
Cardmarket renders only 50 offers initially. Additional offers are loaded by a "Show more results" button that fires an AJAX request (.../Pokemon/AjaxAction/Product_LoadMoreArticles) which appends the next 50 rows into the DOM. This is capped by Cardmarket at 6 loads = 300 offers per URL.
Because you're running a real browser, the clean approach is:
- Render the page.
- Trigger the load-more repeatedly via JavaScript (click / dispatch the "show more" action) until there are no more offers to load, or the 300-offer cap is reached.
- Return the full DOM with all loaded offer rows present.
We need all offer rows in the returned HTML — not just the first 50 — so our parser sees the complete price-sorted list. A page that returns only the initial 50 (or a partial load) is not a usable result for us.
Cardmarket hard-caps the offer list at 300. On popular cards, those first 300 can be dominated by one segment (e.g. English / Near-Mint), so the cheapest of a rarer cell (e.g. Japanese / Played) may never appear within the 300.
To handle that, we generate filtered URLs. Cardmarket exposes its filters — language, condition, seller type, etc. — as URL query parameters, and we push each filtered variant as a separate URL job. For example, a Japanese-only or a specific-condition variant of the same product surfaces that segment's cheapest offer within the first page(s).
The key point for your side: all of the filtering logic is built into the URLs we send you. From the crawler's perspective, every job — base or filtered — is identical: "a Cardmarket product URL: render it, load-more until exhausted or capped at 300, return the full DOM." You don't need to build any Cardmarket-specific filter logic — you just fetch and fully paginate whatever URL we push.
Per URL, "done" means:
- the fully-paginated HTML DOM — all offer rows loaded, up to the natural end of the list or the 300-offer cap,
- delivered via webhook with the source URL,
- no Cloudflare block / challenge / interstitial page, and no partial (50-only) load.
This is exactly where your 100% success-rate guarantee matters most to us — a blocked or partially-loaded page is a silent data gap in our price feed.
- I'm creating the crawler in your platform now.
- Please confirm the crawler can trigger the load-more via JS and return the full post-load DOM as described in §4.
- Then let's run the 500k-URL test to validate success rate and throughput ahead of the yearly commitment.
Happy to jump on a call to walk your solutions/crawler team through any of this.
Best,
Tristan