Webhook receiver spec — Crawlbase Cardmarket crawl results (endpoint, run_result schema, S3 storage)

crawlbase-webhook-spec.md

Cardmarket price crawl — full brief + Crawlbase webhook receiver spec

Owner: CardNexus platform · For: the team hosting the webhook receiver (and Crawlbase integration) · Status: 🧪 temporary evaluation package Crawlbase details verified against /docs/crawler/receiving and /docs/crawler/api (read 2026-07-21).

🧪 This is a throwaway package — we are evaluating Crawlbase

This is not committed infrastructure. We're trialling Crawlbase as one candidate provider. The run_result table and the S3 artifacts here are temporary:

• If we don't go with Crawlbase → we delete the table, delete the S3 bucket/prefix, and remove the package. Nothing else should depend on them. Build it to eject cleanly (see C.9).
• If we do go with Crawlbase → we throw this away too and refactor into a proper, permanent data model aligned with the monorepo.

So treat everything here as a proposal to adapt to the monorepo's conventions — schema, table/column names, storage choices, framework, all of it — with exactly ONE fixed contract: the callback URL https://api.dev.cardnexus.com/webhooks/crawlbase-cardmarket (Crawlbase is configured to POST there). Don't polish this into permanence; make it work and make it removable.

Read this first (Part A) before the engineering spec (Part C). The receiver's job only makes sense once you know what we pull off each page and why full pagination matters.

---

PART A — What we're scraping and why

A.1 Mission

CardNexus aggregates trading-card market prices across marketplaces. Cardmarket (Europe's largest TCG marketplace, Cloudflare-protected) is our first and biggest source. We need the current cheapest price for every card, broken down by the attributes buyers actually filter on — language, condition, and seller type — refreshed continuously.

Scale: ~7 M product pages/month steady-state (≈200–250 K/day), starting with Pokémon singles. We push URLs; Crawlbase crawls them; we get the HTML back and parse prices ourselves.

A.2 The target page

Each Cardmarket product page is the marketplace offer list for one card:

https://www.cardmarket.com/en/Pokemon/Products?idProduct=273532

It's a list of individual seller offers, one row each. Every offer row exposes the fields we care about:

Field	Values	Why we need it
Price	EUR (e.g. 0,20 €)	the number we're ultimately after
Language	English, French, German, Spanish, Italian, Japanese, … (flag icon)	price varies a lot by language (e.g. Japanese vs English)
Condition	Mint (MT) › Near Mint (NM) › Excellent (EX) › Good (GD) › Light Played (LP) › Played (PL) › Poor (PO)	price varies a lot by condition
Seller type	Professional / Powerseller (= "pro") vs Private (= "non-pro")	pro sellers behave differently (stock, shipping, reliability); we may price them separately
Quantity, seller country	ints / country	secondary; captured if present

A.3 What we're trying to get: the "cheapest-price matrix"

For each card we build a matrix of the lowest available price per attribute combination:

cheapest price for each (language × condition) — and, potentially, × seller-type (pro vs private).

The key mechanic that makes this cheap to compute: Cardmarket sorts the offer list by price, cheapest first. So walking the list top-to-bottom, the first time a given combination appears, that offer is the cheapest for that combination. We don't need every offer — we need enough of the (already price-sorted) list to have seen the cheapest of each cell.

Worked example

Offer list as returned (already sorted cheapest→dearest):

#	Price	Language	Condition	Seller
1	€0.20	English	Near Mint	Private
2	€0.25	English	Near Mint	Professional
3	€0.30	French	Near Mint	Private
4	€0.50	English	Excellent	Private
5	€1.20	Japanese	Mint	Professional
…	…	…	…	…

The matrix we extract (first occurrence per cell = cheapest):

• (English, Near Mint) → €0.20 · (French, Near Mint) → €0.30 · (English, Excellent) → €0.50 · (Japanese, Mint) → €1.20
• With the pro dimension: (English, Near Mint, Private) → €0.20, (English, Near Mint, Pro) → €0.25

Rows 2 and later that repeat an already-seen cell are ignored (they're more expensive by definition).

What the crawler/receiver must deliver for this to work: the full post-pagination HTML DOM — every offer row that Cardmarket will show — so our parser sees the complete price-sorted list. A page truncated to the first 50 offers is a data gap. (We parse the HTML ourselves; at this stage Crawlbase returns raw HTML only, no structured extraction.)

A.4 The "load more" mechanic (why a browser is needed)

Cardmarket renders only the first 50 offers. More are loaded when the user clicks "Show more results", which fires an AJAX call (/en/Pokemon/AjaxAction/Product_LoadMoreArticles) that appends the next 50 rows into the page. This repeats, capped by Cardmarket at 6 loads = 300 offers.

So the crawler (a real browser) must: render the page → trigger "load more" via JS repeatedly until no more load or the 300-offer cap → return the full DOM with all rows present. This is exactly why we're using the Enterprise/Browser crawler and not a plain HTTP fetch.

A.5 Beyond 300 offers — filtered URLs (CardNexus's job, not the crawler's)

Cardmarket hard-caps the list at 300 offers. On popular cards the first 300 can be dominated by one segment (e.g. English/Near-Mint), so the cheapest of a rarer cell (e.g. Japanese/Played) may never appear in those 300.

To capture those, we generate filtered URLs — Cardmarket exposes language / condition / seller-type as URL query parameters — and push each filtered variant as a separate URL. From the crawler's and receiver's point of view every job is identical: "a Cardmarket product URL: render, load-more to end/cap, return full DOM." All filter logic lives in the URL we submit — no Cardmarket-specific logic is built into the crawler or the receiver. (Practically: a base URL plus a modest number of filtered follow-ups for high-liquidity cards.)

---

PART B — The pipeline & the receiver's role

CardNexus producer ── push URL (+ optional filters) ──▶ Crawlbase Crawler
   stores {rid → url}         returns {rid}                 │ render + JS load-more to end/300-cap
        ▲                                                   ▼
        └──────────────── POST callback (full HTML DOM) ──▶ api.dev.cardnexus.com/webhooks/crawlbase-cardmarket
                                                            │  (THIS SERVICE — must ACK ≤200 ms)
                          ┌─────────────────────────────────┼─────────────────────────────────┐
                          ▼                                  ▼                                 ▼
                 validate + rid-correlate          durably buffer + 200 ACK             (async worker)
                                                                              ├─ gunzip body
                                                                              ├─ gzip + PUT html → S3
                                                                              ├─ upsert run_result → DB
                                                                              └─ emit metrics
                                                                                        │
                                                            (downstream, separate job) ─┘─▶ parse HTML → cheapest-price matrix

This service's responsibility: authenticate-by-correlation, store the raw HTML artifact on S3, and record one run_result row of metadata per delivery. It does not parse prices — the matrix logic in Part A is a separate downstream consumer of the stored artifacts.

---

PART C — Webhook receiver spec

C.0 TL;DR

1. Accept POST https://api.dev.cardnexus.com/webhooks/crawlbase-cardmarket
2. Respond 200/201/204 within 200 ms, then process asynchronously (Crawlbase's hard requirement)
3. gunzip the body (Crawlbase always gzips it)
4. Store the raw HTML artifact on S3
5. Persist a run_result record (metadata + S3 pointer) per C.4
6. Be idempotent on rid (duplicate deliveries happen; every crawl attempt is billable)

Volume: ~7 M results/month (~3/s avg), bursty — the acceptance test pushes 500 K URLs. Push is capped at 30 URLs/s per token; each crawler queues ≤100 K URLs (C.6).

C.1 Endpoint

URL	https://api.dev.cardnexus.com/webhooks/crawlbase-cardmarket
Method	POST
Response	200/201/204 within 200 ms — else Crawlbase marks it failed and retries (retries are billable)
Body	always gzip (Content-Encoding: gzip) — gunzip before use; never base64. (Only docs exception: Zapier URLs, N/A)
Content-Type in	text/html (default/HTML mode) or application/json (if we push format=json)
Max payload	accept ≥ 10 MB decompressed (300-offer page ≈ 0.3–3 MB HTML; gzipped ≈ 50–600 KB)
TLS	required; endpoint must be publicly reachable by Crawlbase; HTTPS-only

The 200 ms rule drives the design

Hot path does almost nothing: cheap validate/correlate → durably buffer the raw payload → return 200; gunzip/S3/DB happen in async workers. Crawlbase's docs literally say "acknowledge immediately; process asynchronously if work exceeds the 200 ms window." A synchronous S3 PUT or DB write on the request path is a p99 risk under burst — measure during the 500 K test. If 200 ms can't be met reliably, plan B is Cloud Storage delivery (create the crawler without callback_url → Crawlbase stores results and we pull them).

Auth — Crawlbase does NOT sign callbacks

The receiving docs show no signature/secret on the callback. Treat this as an unauthenticated public sink, defended by:

1. Secret in the URL — the configured callback_url carries an unguessable path/token; reject the bare path.
2. rid correlation (strongest) — the producer records every rid returned by a push; only accept callbacks whose rid we pushed. Unknown rid → return 200 (avoid retry storm) + drop + alert.
3. IP allowlist — if Crawlbase publishes egress IPs (C.6 Q), restrict at the WAF/security group.

C.2 Inbound payload — both modes (verbatim from docs)

Crawlbase delivers one of two shapes based on the format we set on push. Build for both.

Mode A — format=html (default): metadata in HEADERS, HTML in body

POST /webhooks/crawlbase-cardmarket
Content-Type: text/html
Content-Encoding: gzip
CB-Status: 200            ← Crawlbase's status; ==200 ⇒ they crawled & billed it
Original-Status: 200      ← Cardmarket's status (200 / 404 / 410 / 5xx)
rid: a1B2c3D4e5F6         ← unique request id (idempotency + correlation key)
url: https://www.cardmarket.com/en/Pokemon/Products?idProduct=273532

<gzipped full HTML DOM>

Mode B — format=json: everything in the (gzipped) JSON body

{
  "cb_status": 200,          // JSON field is `cb_status` (header form is `CB-Status`)
  "original_status": 200,
  "rid": "a1B2c3D4e5F6",
  "url": "https://www.cardmarket.com/en/Pokemon/Products?idProduct=273532",
  "body": "<!doctype html>…" // full post-load HTML DOM as a string
}

Parsing rules: gunzip first; read status defensively — accept cb_status/pc_status (JSON) and CB-Status/PC-Status (header), normalize to one internal cb_status. In Mode A, rid/url/statuses come from headers.

C.3 S3 artifact storage

🧪 Throwaway + proposal. This bucket/prefix exists only for the Crawlbase trial, and the layout below is a suggestion — bucket name, key scheme, even "S3 vs another blob store" are yours to align with monorepo conventions. If we drop Crawlbase, the entire prefix is deleted (see C.9).

• Bucket: cardnexus-crawl-artifacts-dev (per-env; prod separate; SSE-KMS; block public access)
• Key: crawlbase-cardmarket/ingest_date=YYYY-MM-DD/id_product=<idProduct>/<rid>.html.gz (Hive-partitioned for lifecycle + Athena)
• Object: Content-Type: text/html; charset=utf-8, Content-Encoding: gzip; metadata tags cb_status, original_status, submitted_url, rid, crawler_name, received_at
• Idempotent: deterministic key on rid → re-delivery overwrites, no dupes
• Store failures too: blocked / non-200 pages stored (diagnostic) + flagged, never dropped
• Lifecycle (dev): IA at 30 d, expire at 90 d (tune later)

C.4 run_result schema — ⚠️ PROPOSAL, adapt freely

This schema is a proposition, not a mandate. Rename the table/columns, change the types, swap the store (Postgres / DynamoDB / whatever the monorepo uses), restructure it — align it with your conventions. The only thing that matters is that we retain the information: the correlation rid (unique), the two statuses (cb_status + original_status), the success/billed split, the S3 pointer, and hit_cap. How you model and name it is entirely your call. (Same goes for the framework, queue, and buffering choices below — proposals, not requirements. Only the callback URL is fixed.)

One row per delivered result; crawlbase_rid is the unique/idempotency key.

JSON Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "CrawlbaseCardmarketRunResult",
  "type": "object",
  "required": ["id","crawlbase_rid","submitted_url","cb_status","success",
               "s3_bucket","s3_key","received_at","processing_status"],
  "properties": {
    "id":               {"type":"string","format":"uuid"},
    "crawlbase_rid":    {"type":"string","description":"Crawlbase rid — UNIQUE, idempotency + correlation key"},
    "submitted_url":    {"type":"string","format":"uri"},
    "id_product":       {"type":["integer","null"],"description":"parsed from url ?idProduct="},
    "game":             {"type":"string","default":"Pokemon"},
    "is_filtered":      {"type":"boolean","description":"url carries filter params (language/condition/seller-type) beyond idProduct"},
    "filters":          {"type":"object","additionalProperties":true,"description":"parsed filter query params"},
    "cb_status":        {"type":["integer","null"],"description":"Crawlbase status (CB-Status / cb_status). ==200 ⇒ billed"},
    "original_status":  {"type":["integer","null"],"description":"Cardmarket status (Original-Status)"},
    "success":          {"type":"boolean","description":"cb_status==200 AND original_status==200"},
    "billed":           {"type":"boolean","description":"cb_status==200 (Crawlbase bills — may be true when success=false)"},
    "blocked":          {"type":["boolean","null"],"description":"200 but CF-challenge / no offer rows (heuristic)"},
    "offer_count":      {"type":["integer","null"],"description":"count of offer rows in the DOM (optional at ingest)"},
    "hit_cap":          {"type":["boolean","null"],"description":"300-offer cap reached (more offers may exist → needs filtered follow-up)"},
    "s3_bucket":        {"type":"string"},
    "s3_key":           {"type":"string"},
    "html_bytes":       {"type":["integer","null"],"description":"uncompressed size"},
    "html_sha256":      {"type":["string","null"]},
    "crawler_name":     {"type":["string","null"]},
    "received_at":      {"type":"string","format":"date-time"},
    "processing_status":{"type":"string","enum":["received","stored","parsed","failed"]},
    "error":            {"type":["string","null"]},
    "retry_count":      {"type":"integer","default":0,"description":"callback deliveries seen for this rid"},
    "raw_meta":         {"type":"object","additionalProperties":true,"description":"snapshot of inbound headers/JSON meta"}
  }
}

Postgres DDL

CREATE TABLE crawlbase_cardmarket_run_result (
  id                UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  crawlbase_rid     TEXT NOT NULL UNIQUE,          -- idempotency + correlation key
  submitted_url     TEXT NOT NULL,
  id_product        BIGINT,
  game              TEXT NOT NULL DEFAULT 'Pokemon',
  is_filtered       BOOLEAN NOT NULL DEFAULT FALSE,
  filters           JSONB  NOT NULL DEFAULT '{}',   -- {"language":"Japanese","minCondition":"NM",...}
  cb_status         INT,                            -- CB-Status / cb_status; ==200 ⇒ billed
  original_status   INT,                            -- Original-Status (Cardmarket)
  success           BOOLEAN NOT NULL,
  billed            BOOLEAN NOT NULL DEFAULT FALSE,
  blocked           BOOLEAN,
  offer_count       INT,
  hit_cap           BOOLEAN,
  s3_bucket         TEXT NOT NULL,
  s3_key            TEXT NOT NULL,
  html_bytes        INT,
  html_sha256       TEXT,
  crawler_name      TEXT,
  received_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
  processing_status TEXT NOT NULL DEFAULT 'received',
  error             TEXT,
  retry_count       INT NOT NULL DEFAULT 0,
  raw_meta          JSONB NOT NULL DEFAULT '{}',
  created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at        TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON crawlbase_cardmarket_run_result (id_product);
CREATE INDEX ON crawlbase_cardmarket_run_result (received_at);
CREATE INDEX ON crawlbase_cardmarket_run_result (success) WHERE success = FALSE;

billed vs success: Crawlbase bills whenever cb_status==200, even if Cardmarket didn't return 200 or the page had no offers — and every crawl retry is billable. Tracking both lets us reconcile their invoice against useful results (we've measured ~7–8 % of 200s come back with zero offers). hit_cap is the domain-critical flag: it tells the producer which products need filtered follow-up URLs (Part A.5).

C.5 Processing requirements

1. 200 ms ACK — cheap validate/correlate → durably buffer the raw (still-gzipped) payload + headers → return 200. Everything else async.
2. Durable buffer — raw payload → S3 landing prefix + enqueue the key to SQS (SQS's 256 KB cap is exceeded by large gzipped bodies, so buffer the bytes in S3 and pass a pointer).
3. Idempotency — upsert on crawlbase_rid (ON CONFLICT), bump retry_count on re-delivery; deterministic S3 key.
4. Decode & classify (worker): gunzip; handle Mode A (headers) vs Mode B (JSON); compute html_sha256, html_bytes;
   • success = cb_status==200 && original_status==200; billed = cb_status==200
   • blocked = 200 but body matches a Cloudflare-challenge signature or has no id="articleRow" markers
   • offer_count = count of offer rows; hit_cap = offer_count ≥ 300 (or last-page-full) — cheap substring counts, else null for the parser
5. rid correlation — check rid against the producer's pushed-rid store; unknown → 200 + drop + alert.
6. Never drop failures — store artifact + failed/blocked record so those URLs can be re-driven.
7. Backpressure — buffer unavailable → return 5xx so Crawlbase retries (accepting the billable retry) rather than losing data.

C.6 Push-side constraints (context for the producer + the 500 K test)

• Create crawler: POST https://api.crawlbase.com/crawler/<TOKEN> {"name","callback_url"}. Omit callback_url → Cloud Storage (plan B).
• Push: GET https://api.crawlbase.com/?token=…&crawler=NAME&callback=true&url=…&format=json → returns {"rid":"…"} (store for correlation).
• JS/browser options: javascript_token, page_wait, ajax_wait, css_click_selector (drives the load-more clicks in A.4).
• Push rate: 30 URLs/s per token. 500 K URLs ⇒ ~4.6 h to enqueue; steady 7 M/mo ≈ 2.7/s. Queue cap: 100 K URLs/crawler → producer must throttle to keep the waiting queue < 100 K.

C.7 Open questions for Crawlbase

1. format — run format=json (metadata+HTML in one JSON) or default HTML+headers? (We lean JSON.)
2. Callback auth — confirm no signature (so URL-secret + rid-correlation + IP allowlist is the plan), or share the signing scheme.
3. Egress IPs — fixed range for a WAF allowlist?
4. Callback retry policy — count, backoff, and whether a 200 ms timeout triggers a retry.
5. Max body size they'll deliver.
6. Higher push rate / queue cap for the 500 K test and 7 M/mo?

C.8 Observability & security

• Metrics: received_total, ack_latency_ms (p50/p95/p99 vs 200 ms), success_rate, billed_rate, blocked_rate, hit_cap_rate, unknown_rid_total, s3_write_errors, db_write_errors, processing_lag_s, payload_bytes histogram.
• Alerts: ACK p99 nearing 200 ms; success rate < 99 % / 15 min (guarantee is 100 %); sustained 5xx; unknown-rid spikes; buffer/S3/DB failures.
• Correlation: log crawlbase_rid + submitted_url at every step.
• Security: URL secret + rid-correlation + optional IP allowlist; TLS-only; 10 MB cap; SSE-KMS + block-public-access. Data is public marketplace listings (seller usernames) — low sensitivity; bucket stays private.
• Environments: dev now (api.dev.cardnexus.com, cardnexus-crawl-artifacts-dev); mirror to staging/prod with separate URLs, buckets, secrets.

C.9 Disposability — build it to eject cleanly

This is a trial (see the callout up top). Keep the blast radius small so that if we don't proceed, removal is a 10-minute checklist rather than an archaeology dig:

• One self-contained module/package. All Crawlbase-webhook code lives in a single directory/service; nothing else in the app imports from it. Removal = delete the package + its route.
• Everything namespaced. One table (crawlbase_cardmarket_*), one S3 prefix (crawlbase-cardmarket/…), one secret, one route. No shared/global schema changes.
• Behind a config/feature flag so the endpoint + producer can be switched off without a deploy.
• No downstream coupling. The (separate) matrix parser reads artifacts through a thin interface over S3/DB — if we switch providers, only this ingestion package changes, consumers don't.
• Ship a teardown checklist with the package, e.g.:
  1. disable the route / flag, 2. delete the Crawler in the Crawlbase dashboard, 3. DROP TABLE crawlbase_cardmarket_run_result, 4. delete the crawlbase-cardmarket/ S3 prefix (or let lifecycle expire it), 5. revoke the URL secret, 6. delete the package.

If we do adopt Crawlbase, none of this survives as-is anyway — it gets rebuilt as a first-class data model. So optimise this version for speed to a working test and clean removal, not for longevity.