Scaling & Surviving: Partition Keys, Blast Radius & Designed Degradation · resilience
Module 3 assumed a handful of replicas. Past one node, something is always failing — and the survivors aren't the ones with more capacity, they're the ones that decided how they'd fail in advance. This take-home asks you to audit a system you run for partitioning, resilience, and blast radius, and to write down its degradation before the spike. The grade is in a partition key that bounds failure as well as load.
Time: ~45–60 min · Due: before Module 5 · Submit: post your write-up in the cohort channel.
Pick one system you run, or know well enough to reason about concretely.
⚠️ Two systems are off-limits as your submission: the flash-sale checkout audited below (it's the worked demo), and the exam-results portal we audited together in class. Pick something else.
In a ~300–400 word audit, make the five moves:
- Name the partition axis and key — data, work, or geography. Then the real question: does that key also bound failure, or only load? The good key does both jobs.
- Place read/write behaviour on the fan-out spectrum — on-write (precompute), on-read (assemble on demand), or hybrid — and say why the ratio drives it.
- List the resilience pattern guarding each cross-component dependency — timeout, retry, circuit breaker, bulkhead, fallback. One per dependency.
- Find any single shared global state — the next Fail Whale, the counter/lock/table every request contends for — and say how you'd shard or approximate it.
- Specify degradation in advance — what you drop, what you keep, under overload. Decided before the spike, not during it.
- A single shared global state you've seen become the binding constraint — what everything contended for, and what it cost.
- A system with only one safety layer for a real hazard — what was the missing independent backstop?
- An emergent behaviour at scale nobody foresaw — could it have been contained and observed even if it couldn't be predicted?
Reply to one classmate's audit. Find a partition key that bounds load but not failure (so a failure spreads past it), or a dependency with no resilience pattern guarding it — and propose the fix.
Everything below is just these, applied to one system. Keep them in front of you:
- Something is always failing. At scale, failure is the normal case. The choice isn't whether failures happen — it's whether each one is routine or catastrophic, and that's decided in the design.
- One key, two jobs. The partition key that bounds load also bounds the blast radius. Choose it so a failure can't spread past it, not just so load balances.
- Resilience is designed in. Every cross-component dependency gets a guard. One unguarded slow call cascades into a whole-system outage.
- Count the independent layers. Defence in depth means multiple independent layers. Duplicating one fragile check buys nothing.
- Design degradation before you need it, and contain what you can't predict. Decide what to shed under overload in advance; bound and observe the emergent behaviour you couldn't foresee.
Let's run the audit on something that is not your assignment: a flash-sale checkout — an online store doing a limited-stock drop, where browsing traffic spikes 50× and thousands try to buy the same few items in the same minute. Watch the moves, then make them on your own system.
1 · Partition key — and does it bound failure? Partition the catalog and checkout by product (SKU), with users sharded underneath. One overwhelmed product shard — the one hot sneaker everyone wants — then touches only that product's buyers, not the whole store. That's the key doing two jobs: it spreads load and it stops a stampede on one SKU from taking down the entire catalog. A naive "one big checkout service for everything" bounds neither — one hot item melts the lot.
2 · Fan-out spectrum (read/write asymmetry). The asymmetry is extreme: browsing reads vastly outnumber purchase writes. So serve product pages from precomputed, cached, read-optimised views (fan-out-on-write style) that survive the read storm without touching the transactional path — and reserve the expensive, consistent write path for the rare actual checkout. The read storm never reaches the write path.
3 · Guard each dependency. Checkout depends on inventory, payments, and fulfilment. Timeout + circuit breaker on payments; if it trips, queue the order and confirm asynchronously rather than block. Bulkhead the inventory service so checkout load can't starve catalog browsing. Fallback: if recommendations are down, show a static list — never let a non-critical call stall the buy.
4 · Find the shared global state (the Fail Whale). Here it is: a single inventory counter on the hot SKU, which every concurrent buyer contends for — the textbook next-Fail-Whale. Left alone it serialises the entire sale onto one row. Fix it before it's the binding constraint: shard the stock into per-node allotments, or use an approximate-then-reconcile reservation so buyers aren't all hammering one counter.
5 · Degradation, decided in advance. Written down before the drop: under overload, drop personalization, reviews, and recommendations; keep the product page and the buy button. Show a "high demand — you're in the queue" state rather than collapsing. The core purchase path survives by shedding everything that isn't it.
Notice what just happened: I picked a key that bounds load and blast radius, placed the read/write split on the fan-out spectrum, guarded every dependency, killed the shared global counter before it killed the sale, and wrote the degradation in advance — while staying humble that some stampede behaviour I'll observe, not predict. That's the whole of Part A. Your job is to make those moves on your system.
A partitioned system where one shard's failure stops at its boundary, a guarded dependency that degrades instead of cascading, and the single shared counter that — left alone — becomes the chokepoint. Your audit finds all three on your system.
flowchart TD
LB["Incoming spike (50x)"]
S1["SKU shard A<br/>bounded load + blast radius"]
S2["SKU shard B<br/>(hot item) FAILING"]
S3["SKU shard C<br/>unaffected"]
PAY["Payments<br/>timeout + circuit breaker"]
CNT{{"Shared inventory counter<br/>the next Fail Whale<br/>shard / approximate"}}
DEG["Degrade: drop recs/reviews<br/>keep buy button<br/>(decided in advance)"]
LB --> S1 & S2 & S3
S2 -. contained .-> DEG
S1 --> PAY
S2 --> CNT
PAY -. trips -> queue .-> DEG
style S2 stroke:#EB7B5B,stroke-width:2px
style CNT stroke:#C97E12,stroke-width:1.5px
style DEG stroke:#6FCF97,stroke-width:1.5px
Copy this into your doc and replace each blank. Keep it tight — most of the marks are in points 1, 4, and 5.
SYSTEM (one line): _______________________________________
1. PARTITION KEY
Axis (data / work / geography) + key: ______________
Does it bound FAILURE too, or only load? ___________
2. FAN-OUT / READ-WRITE
Read:write asymmetry: ______________________________
Position on the spectrum (write / read / hybrid): __
3. RESILIENCE PER DEPENDENCY
Dependency -> guard: ________________________________
Dependency -> guard: ________________________________
Dependency -> guard: ________________________________
4. SHARED GLOBAL STATE (the next Fail Whale)
What everything contends for: ______________________
Shard / approximate it how: ________________________
5. DEGRADATION (decided in advance)
Under overload I DROP: ______________________________
Under overload I KEEP: ______________________________
If you want a diagram, copy the Mermaid block above and relabel it for your system — it renders as a flowchart in your Gist.
Part B — starter prompts (answer in your own words; don't just restate these):
- Find the counter, lock, queue, or single hot row that every request had to touch. When it became the bottleneck, what was the blast radius — and what would have sharded or approximated it away?
- Pick a real hazard in something you've shipped and count the independent safety layers between command and harm. If the count is one, name the backstop that should have been there.
- Recall a behaviour that emerged at scale nobody designed — a feedback loop, a thundering herd, a viral cascade. You couldn't have predicted it; could the system have bounded it and seen it?
- A partition key that balances load but not failure. If a failure in one shard can still spread, the key only did half its job. The master move is both.
- An unguarded dependency. Every cross-component call needs a timeout/breaker/bulkhead/fallback. The one you forgot is the one that cascades.
- Duplicating a fragile check and calling it depth. Two copies of the same flawed layer fail together. Independence is the property that matters.
- Degradation invented during the incident. If it isn't written down before the spike, it isn't a plan — it's improvisation under fire. Decide in advance.
- Pretending to foresee emergence. You can't predict the specific behaviour. Don't claim to — bound where it reaches and instrument so you can watch it.
- A partition key that bounds blast radius as well as load.
- Resilience patterns on every dependency — not just the obvious one.
- The shared global state named and a fix proposed before it's the binding constraint.
- Degradation specified in advance, and honesty that some behaviour must be observed, not predicted.
That's resilience. You don't survive a spike by hoping — you bound the blast radius, guard every dependency, kill the shared state, and decide the degradation ahead of time. Next module is the finale: the system still has to be built, shipped, and run — inside an organisation that doesn't care about your architecture. And at the end, all five modules become one instinct.