Auction System Architecture (Bids via WhatsApp/SMS)

arch.md

Auction System Architecture (Bids via WhatsApp/SMS)

1) Goals & Requirements

• Accept bids sent via WhatsApp and SMS; optionally via HTTP API.
• Support CRUD for auctions (create/update/delete), each with start_time and end_time.
• Handle high concurrency, especially in the last minutes/seconds of an auction.
• Keep authoritative record of who is winning in real time and a complete bid ledger.
• Determine the winner and send notifications via WhatsApp, SMS, and email.
• Multi-auction support; each auction is logically isolated but shares infrastructure.
• Operate reliably with idempotency, exactly-once effects, observability, and security.

---

2) High-Level Architecture (Services & Data Flow)

Ingress

• Messaging Provider Webhooks: Twilio (SMS + WhatsApp) or WhatsApp Cloud API → Messaging Ingest API (validate HMAC/signature, normalize payloads, rate-limit, dedupe by provider message_id).
• Public Bid API (optional): POST /v1/auctions/{id}/bids for app/web clients.
• Admin API: CRUD for auctions & templates; backoffice dashboard reads.

Core

• Event Bus / Stream: Kafka (or Google Pub/Sub, AWS Kinesis). Topic: bids (partition key = auction_id), auction-lifecycle, notifications.
• Bid Matcher / Auction Engine (consumer):
  • Sharded consumers by auction_id to ensure per-auction ordering.
  • Validates business rules, updates Redis (hot state) atomically via Lua scripts, appends to Postgres (ledger) via Outbox pattern.
• Auction Scheduler: Starts/pauses/ends auctions at defined times. Emits AuctionStarted, AuctionClosingSoon, AuctionEnded events. Implement with Temporal.io / Quartz / AWS EventBridge / Cloud Tasks.
• Finalizer: On AuctionEnded, computes winner deterministically from ledger + Redis snapshot, writes WinnerDetermined and emits notifications event.
• Notification Service: Consumes notifications events; sends WhatsApp/SMS (provider) and Email (SES/SendGrid). Tracks delivery receipts.

State

• Postgres (or Cloud SQL): normalized OLTP store (Auctions, Bids, Bidders, Winners, AuditLog, OutboxEvents).
• Redis Cluster: hot state + locks + de-duplication; structures:
  • auction:{id}:leader (hash with amount, bidder_id, ts)
  • auction:{id}:bids (sorted set by server_ts as tiebreaker)
  • msg_dedupe:{provider_msg_id} (SET/TLL)
  • auction:{id}:status (enum active/pending/ended)
• Object Storage (optional): export CSVs, long-term audit archives.

Observability

• Centralized logs with correlation_id (trace): provider_msg_id → bid_id → notification_id.
• Metrics: bids_received/accepted/rejected, p99 bid latency, Kafka lag per partition, Redis CAS failures, finalization latency, notification success rate.
• Alerts: hot-auction detection (RPS threshold), scheduler drift, webhook error spikes.

---

3) Data Model (Relational)

-- Auctions
CREATE TABLE auctions (
  id UUID PRIMARY KEY,
  title TEXT NOT NULL,
  start_time TIMESTAMPTZ NOT NULL,
  end_time   TIMESTAMPTZ NOT NULL,
  min_increment NUMERIC(18,2) DEFAULT 1.00,
  currency CHAR(3) NOT NULL,
  status TEXT NOT NULL CHECK (status IN ('scheduled','active','ended','cancelled')),
  soft_close_seconds INT DEFAULT 0,  -- anti-sniping window; 0 = disabled
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Bidders
CREATE TABLE bidders (
  id UUID PRIMARY KEY,
  display_name TEXT NOT NULL,
  phone_e164 TEXT UNIQUE,
  whatsapp_id TEXT UNIQUE,
  email TEXT,
  is_blocked BOOLEAN DEFAULT FALSE,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Bids (immutable ledger)
CREATE TABLE bids (
  id UUID PRIMARY KEY,
  auction_id UUID REFERENCES auctions(id),
  bidder_id UUID REFERENCES bidders(id),
  amount NUMERIC(18,2) NOT NULL,
  provider_msg_id TEXT,
  source TEXT NOT NULL CHECK (source IN ('whatsapp','sms','api')),
  server_ts TIMESTAMPTZ NOT NULL DEFAULT now(),
  accepted BOOLEAN NOT NULL,
  reason TEXT, -- if rejected: below_min_increment, auction_not_active, etc.
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Winners
CREATE TABLE winners (
  auction_id UUID PRIMARY KEY REFERENCES auctions(id),
  bidder_id UUID REFERENCES bidders(id),
  final_amount NUMERIC(18,2) NOT NULL,
  decided_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Outbox (reliable events)
CREATE TABLE outbox_events (
  id UUID PRIMARY KEY,
  aggregate_type TEXT NOT NULL, -- auction, bid, notification
  aggregate_id UUID NOT NULL,
  type TEXT NOT NULL,          -- BidAccepted, AuctionEnded, WinnerDetermined, Notify
  payload JSONB NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  processed_at TIMESTAMPTZ
);

---

4) Event Schemas (examples)

• BidReceived (from Ingest): { auction_id, bidder_ref, amount, provider_msg_id, source, received_at }
• BidValidated (internal): { bid_id, auction_id, bidder_id, amount, accepted, reason, server_ts }
• AuctionStarted|Ended : { auction_id, at }
• WinnerDetermined : { auction_id, winner_bid_id, bidder_id, final_amount, decided_at }
• Notify : { channel: 'whatsapp'|'sms'|'email', to, template_id, variables }

---

5) Core Flows (Happy Paths)

A) Bid via WhatsApp/SMS

1. Provider calls Webhook → POST /v1/messaging/webhook with message.
2. Messaging Ingest validates signature, extracts auction_id & amount (see Parsing below), dedupes by provider_msg_id in Redis.
3. Emits BidReceived to Kafka partitioned by auction_id.
4. Bid Matcher (single consumer for that auction) consumes in-order:
   • Confirms auction status=active and now <= end_time (with grace window, e.g. 2s).
   • Reads current leader from Redis; validates amount >= leader + min_increment.
   • Atomically updates Redis (Lua script: CAS on auction:{id}:leader).
   • Appends immutable Bid row in Postgres within a local transaction that also writes an Outbox event (BidAccepted/BidRejected).
5. Dashboard/API shows new leader in <100ms from ingest; clients can poll or use Server-Sent Events / WebSockets from a Read API backed by Redis.

B) Auction Start/End

• Scheduler at start_time sets status→active and emits AuctionStarted.
• Near end_time:
  • If soft_close_seconds > 0 and a bid arrives within that window, extend end_time by that amount (anti-sniping). Emit AuctionExtended.
  • At final end: status→ended, Finalizer snapshots auction:{id}:bids and ledger to compute winner deterministically (highest amount; earliest server_ts tie-breaker), writes winners row, emits WinnerDetermined and Notify events.

C) Notifications

• Notification Service consumes Notify and dispatches using provider APIs:
  • WhatsApp/SMS (provider templates for outbound),
  • Email (SES/SendGrid). Records delivery receipts → updates outbox processed_at.

---

6) Parsing Free‑Text Messages → Auction & Amount

• Preferred format: #{auction_code} {amount} e.g., #A123 250.00.
• Fallbacks supported: BID 250 for A123, A123 250, currency symbols, commas.
• Use deterministic regex & locale parsing first; fallback to a lightweight NLU if ambiguous.
• Map sender → bidder_id via phone/wa_id; if unknown, auto-provision bidder (guarded by allowlist or KYC check).
• Reject with helpful message when parse fails or auction not active.

---

7) Concurrency & Ordering Strategy

• Partition by auction_id on the event bus ⇒ all bids for one auction are serialized through one consumer, preserving order.
• Atomic state in Redis guarded by Lua scripts ensures compare‑and‑set of leader & last amount.
• Idempotency:
  • Webhook: dedupe by provider_msg_id (Redis key TTL ~24h).
  • Public API: require Idempotency-Key header (stored in Redis/DB).
• Grace Window: treat bids with server_ts ≤ end_time + skew_ms as valid to account for network jitter. Record actual server_ts and provider sent_ts for audit.
• Backpressure: if consumer lag grows, auto-scale consumers (more partitions); hot auctions are isolated in their own partitions.

---

8) API Design (v1)

Admin

• POST /v1/auctions → create (title, start_time, end_time, min_increment, currency, soft_close_seconds)
• PATCH /v1/auctions/{id} → update editable fields before start; limited fields while active (e.g., extend end_time if policy allows)
• DELETE /v1/auctions/{id} → cancel (soft delete: status→cancelled)
• GET /v1/auctions/{id} / GET /v1/auctions?status=

Public

• POST /v1/auctions/{id}/bids (optional client route) body: { amount }. Require auth or signed link.
• GET /v1/auctions/{id}/leader → { bidder_display_name_masked, amount, ends_in_seconds }
• GET /v1/auctions/{id}/bids?limit=... (paginated read from Postgres or Redis recent cache)
• GET /v1/auctions/{id}/stream (SSE) → live updates.

Webhooks

• POST /v1/messaging/webhook (SMS/WhatsApp): validates signatures, returns 200 quickly, push to Kafka.

Security: OAuth2/JWT for Admin; HMAC verification for webhooks; WAF + per-bidder rate limits (e.g., 5 req/s burst 10).

---

9) Business Rules

• Bid must be ≥ current_leader + min_increment.
• Auction must be active (now between start_time and end_time, with grace window and soft-close logic).
• Ties on amount resolved by earliest server_ts.
• Optionally enforce max bid or proxy bidding (future extension).

---

10) Soft Close / Anti‑Sniping (Optional)

• If a bid arrives within the last N seconds (soft_close_seconds), push end_time += N (idempotently; store extension count if capped).
• Broadcast AuctionExtended so UIs and bidders know the new end.

---

11) Failure Handling & Consistency

• Outbox pattern in DB ensures reliable publication of domain events for notifications/analytics.
• At-least-once consumption with idempotent handlers. Redis CAS guarantees only one leader update per valid bid.
• Recovery: On restart, rebuild Redis leader from Postgres by replaying last K bids.
• Clock: Use NTP; compare to provider timestamps for audits; all comparisons use server_ts as authority.

---

12) Notifications (Winner + Participants)

• Templates:
  • Winner (WhatsApp/SMS): "Congrats, you won {title} with {amount}. Pay here: {payment_link}".
  • Non-winners (optional): "Auction ended. Winning bid: {amount}. Thanks for participating." (opt-in per policy).
• Email receipt with invoice/next steps.
• Respect WhatsApp template rules; fall back to SMS/email if out-of-session.

---

13) Deployment Reference (AWS & GCP)

AWS