Raj's Notes

System design

Requirements, capacity, request paths, data systems, coordination, asynchronous work, reliability, and design cases.

Scope#

A method for producing and defending a design with traceable requirements, capacity math, storage and consistency choices, overload behavior, recovery targets, and observable acceptance criteria.

Read the notes in order. The first note shows how to turn a vague prompt into a bounded contract and explains the artifacts that appear in every later design. The sequence then follows one request through an API and the network before adding data modeling, relational-engine behavior, distribution, coordination, asynchronous work, caching, scheduling, and recovery. The final note combines those pieces in nine worked interview designs.

Orientation and decision ledger#

Every note continues the same illustrative system: merchants create and inspect tenant-scoped orders, update their status, and trigger email or push notifications. Payment execution, inventory ownership, template editing, and provider internals stay outside the boundary. The fixed workload is 1,000 average and 5,000 peak order creates each second, 20,000 peak reads each second, 4 KB per stored order, 30 days of hot history, p99 create latency below 250 ms, 99.9% monthly create availability, and 99% of notification dispatches starting within five seconds. The stress case assigns 20% of creates to one tenant; regional recovery targets a 15-minute RTO and 30-second RPO.

Reading sequence: SD1 → SD2 → SD3 → SD4 → SD5 → SD6 → SD7 → SD8 → SD9 → SD10 → SD11.

Note Main job Decision added to the running design Deeper derivation
SD1 Bound and size the prompt Contract, SLOs, 10.4 TB raw hot data, and exclusions DS1: remote calls and ambiguous outcomes
SD2 Define the request path Four API operations, one-second deadline, and tenant-scoped idempotency DS1: deadlines, retries, and operation identity
SD3 Choose data paths PostgreSQL authority, ordered tenant index, archive, and analytic projection DB1 through DB3
SD4 Open the relational engine Bounded pools, MVCC/WAL boundary, and observable concurrent index migration DB4 through DB7
SD5 Distribute ownership 64 shards, three zonal copies, and eight buckets for the hot tenant DS6: placement, DS7: replication, and DB11: Cassandra
SD6 State correctness Per-order versioning plus a transactional outbox and idempotent consumers DS5: consensus, DS7: transactions, and DB4: isolation
SD7 Buffer notification work At-least-once queue, retry identity, dead letters, and backlog arithmetic DS8: stream processing and DB15: CDC
SD8 Protect scarce capacity Terminal-receipt cache, 30,000-RPS admission ceiling, and priority shedding DB9: Redis and cache failure
SD9 Reconcile worker capacity 75 workers across three zones with explainable placement and spare nodes DS8: scheduling and shared resources
SD10 Recover a region Active-passive fencing, 15-minute RTO, 30-second RPO, and SLO evidence DS10: security and recovery and DB6: database recovery
SD11 Defend the complete design Final ledger plus a requirement perturbation that forces a redesign Revisit the derivation beside the decision that changed

These notes assume ordinary product-engineering experience. Product names such as Kafka, Cassandra, ECS, and EKS are defined before their mechanics are used. When a mechanism appears, ask four questions: what problem forced it into the design, what promise does it make, where does that promise end, and how would an operator know it stopped working?

Difficulty labels describe the depth inside a note, not another reading order. Follow the note numbers; a later Core note may apply an Advanced mechanism introduced earlier.

Reading the module will not by itself make someone ready for every system-design interview or production on-call rotation. Use it to build a repeatable reasoning method, then practice unfamiliar prompts, draw request and recovery paths, run load tests, restore backups, and review real incidents. Production judgment comes from making the written assumptions collide with measured systems.

Useful background#

If those foundations are rusty, read cloud infrastructure for service boundaries and low-level infrastructure for processes, memory, networks, and storage. Read distributed systems before or alongside SD5–SD10: that module derives the time, replication, consensus, and failure mechanisms that system design applies. The data systems foundations provide the slower path through storage engines, indexes, transactions, and the database products used from SD3 onward. None is a hidden prerequisite for starting SD1; the ledger links provide a second explanation when a mechanism needs more detail.

Notes

  1. Frame the Problem Before Drawing Boxes Convert a product prompt into explicit workloads, service indicators, reliability targets, and capacity estimates. Architecture starts only after the numbers expose the hard part.
  2. Design the API and Account for the Network Path Trace a request through DNS, transport, encryption, load balancing, and application code; then define API semantics that remain safe under retries and partial failure.
  3. Choose Storage from Access Patterns Turn product operations into a data model, select a storage family from its access paths and invariants, and design transactional and analytic projections for their real queries.
  4. Relational Engine Internals and Online Schema Changes Follow PostgreSQL and MySQL from connections through row versions, logs, cleanup, replication, and an online index build without confusing SQL syntax with the physical engine.
  5. Partition Data, Replicate It, and Defuse Hot Keys Distribute state across machines without confusing data volume with traffic distribution. Plan movement, replica placement, and skew before a celebrity key or current-time partition finds the limit.
  6. Reason About Time, Consistency, and Coordination Specify what concurrent clients may observe, decide what happens during a partition, and use consensus, leases, fencing, outboxes, or sagas only where the invariant demands them.
  7. Build Queues and Streams That Can Fall Behind Choose a work queue or durable log, define ordering and delivery semantics, and plan retries, dead letters, replay, and backpressure before consumers lag.
  8. Cache Deliberately and Survive Overload Place caches at the edge or service boundary, define freshness and invalidation, then contain overload with deadlines, jitter, retry budgets, shedding, circuit breakers, and shuffle shards.
  9. Build Control Planes and Schedulers Model desired state, reconciliation, leases, and placement as separate control-plane concerns; then schedule multi-resource workloads with explicit feasibility, packing, fairness, and interference policy.
  10. Design regional ownership, recovery, and the evidence to operate it Map failure domains and regional data ownership, set RTO and RPO, test backups, budget tail latency, limit correlated failure, and instrument the evidence needed to operate the design.
  11. System Design Interviews Through Nine Worked Designs Run a system-design interview from scope through recovery, choose cloud and data technologies by responsibility, and study nine bounded designs with explicit calculations and failure paths.