System Design Methodology and Trade-off Analysis Questions

Direct answer

Latency and throughput are not opposites by nature, they trade off through queueing: pushing more concurrent work through a fixed amount of processing capacity increases the time each request waits behind others, and holding latency low means keeping spare capacity in reserve rather than running it flat out. The right balance comes from setting an explicit target for both (a throughput floor and a tail-latency ceiling), then using queueing math plus load testing to find the utilization level where more throughput starts costing more latency than the business can absorb. What to measure at each load level: the full latency distribution, not just the average, including the 95th and 99th percentile (P95/P99), alongside the downstream business metric (conversion rate, task completion time) the latency target exists to protect.

Structured elaboration

Why the tension exists. Little's Law ties the three quantities together:

$L = \lambda W$

where $L$ is the average number of requests in the system (concurrency), $\lambda$ is the arrival rate (throughput), and $W$ is the average time a request spends in the system (latency). For a fixed amount of concurrency capacity $L$, pushing $\lambda$ up forces $W$ up. Throughput and latency are linked by whatever capacity sits between them, they only look independent at low load.

Decision criteria to walk through, in order:

• Is there a hard external constraint (a contractual service-level agreement, or SLA) versus a soft internal preference? Hard constraints bound the feasible region before you optimize anything.
• Is the load steady or bursty? A bursty workload needs headroom sized for the peak, not the average, or tail latency spikes during every burst.
• What is the true cost of extra capacity relative to the revenue or reliability cost of extra latency? If compute is cheap relative to the business impact of latency, buy headroom instead of accepting queueing.
• Which metric does the product actually care about, median latency almost never predicts user-visible pain, the tail does.

Process: baseline the current latency distribution and throughput, ramp load in steps while recording the full distribution at each step, locate the point where the P95 or P99 curve bends upward sharply (the "knee"), then correlate that knee to the business metric to decide whether operating past it is acceptable.

Worked example

Assume, for illustration, a single worker with an average service time of 10 ms per request ($S = 0.01\text{s}$), so its theoretical maximum throughput is $1/S = 100$ requests per second (RPS). Using the M/M/1 queueing approximation (a standard model for one server handling one request at a time, with randomly arriving requests and randomly varying service times, a common simplification for a single queue), the average wait time in queue at utilization $\rho = \lambda S$ is:

$W_q = \frac{\rho \cdot S}{1-\rho}$

Reproducing the middle row: $W_q = \dfrac{0.90 \times 0.01}{1-0.90} = \dfrac{0.009}{0.10} = 0.09\text{s} = 90\text{ms}$, so $W = 90 + 10 = 100\text{ms}$. Going from 70 to 90 RPS (a 29% throughput increase) roughly triples latency; the next 5.6% of throughput (90 to 95 RPS) roughly doubles it again. This is the shape of the trade-off: throughput gains near saturation cost latency disproportionately.

The same law sizes capacity to hit both targets at once. To sustain 5,000 RPS at an average latency target of 15 ms, the required in-flight concurrency is $L = \lambda W = 5000 \times 0.015 = 75$ concurrent request slots. If each server instance can hold 25 concurrent requests (its thread or connection budget), raw sizing needs $75 / 25 = 3$ instances, but running at 100% utilization guarantees queueing, so target roughly 65% utilization for headroom: $3 / 0.65 \approx 4.6$, round up to 5 instances.

Trade-offs & pitfalls

• Treating the median as the target metric hides exactly the users experiencing queueing delay, always instrument and alert on the tail, not the average.
• Adding raw compute capacity fixes queueing-induced latency but does nothing for latency caused by serialization cost or an inefficient algorithm, these are different bottleneck classes and need different fixes (see bottleneck-identification questions for the diagnostic process).
• Batching or coalescing requests can raise both average throughput and average latency-per-request while making the tail worse for whichever request lands first in a batch, batching trades individual completion time for aggregate efficiency and needs a separate tail-latency check.
• Autoscaling on CPU utilization alone can under-react to a pure queueing problem, alerting or scaling on the latency percentile itself, or on queue depth, catches the tension directly.
• Always tie the chosen operating point back to the business metric with real data (an A/B test or canary), a default like "P95 under 300 ms" is only correct if it is where the business metric actually degrades.

Direct answer

A global leaderboard for tens of millions of players needs three layers: a rate-limited ingest path that bounds how fast any single player's updates can enter the system, a per-region top-N structure that absorbs most of the update volume locally, and a periodic global merge that combines regional top-N lists into the global top-100. The handful of top players updating constantly is contained by capping their update rate at ingest, not by trying to process every update they generate.

Structured elaboration

Each piece exists to solve a specific problem:

Rate limiting at ingest. A per-player token-bucket cap (for example, a fixed number of accepted score updates per second per player) exists because a small number of hyperactive top players can otherwise generate a disproportionate share of total write volume; bounding it there protects everything downstream regardless of how aggressively any one player updates.

Regional top-N structures. Each region maintains its own bounded top-N set (slightly larger than 100, to keep merges accurate) in a fast in-memory structure, updated only when an incoming score would actually change that region's top-N; this keeps per-region write volume bounded by the number of players who could plausibly be in the top-N, not by total player count or total update volume.

Global merge. A periodic (sub-few-second) job reads each region's top-N snapshot and merges them into the global top-100, published to a cache the serving layer reads from. This decouples the strict local consistency of a region's own leaderboard from the necessarily looser consistency of the global view, which only needs to be accurate to within the merge interval.

flowchart LR
  A[Game clients] --> B[Ingest API]
  B --> C[Rate limiter: per-player token bucket]
  C --> D[Stream log]
  D --> E[Regional aggregator: top-N sorted set]
  E --> F[Global merge service]
  F --> G[Read cache]
  G --> H[Regional and global top-100 API]
  E --> I[(Cold store)]

Worked example

Assume, as illustrative pinned inputs, 50 million total players with 5% concurrently active and submitting a score update on average once every 10 seconds:
$\text{concurrent} = 50{,}000{,}000 \times 0.05 = 2{,}500{,}000 \text{ players}$
$\text{avg update rate} = \frac{2{,}500{,}000}{10\text{s}} = 250{,}000 \text{ updates/s}$
That 250,000 updates/s average is the volume the regional aggregation tier as a whole absorbs, but the reason a handful of top players don't overwhelm any single shard is the per-player cap applied at ingest. If the top 100 players were all capped at 10 updates/s each:
$\text{bounded hot-shard worst case} = 100 \text{ top players} \times 10 \text{ updates/s (token-bucket cap)} = 1{,}000 \text{ updates/s}$
Even in the worst case where every top player updates at the maximum allowed rate simultaneously, that's only 1,000 updates/s concentrated on the shard (one partition of the data spread across multiple servers, each holding a slice of the total) holding the current top-N, roughly 0.4% of total system update volume, which is exactly the point of capping at ingest rather than trying to absorb an unbounded rate from a handful of accounts.

Trade-offs & pitfalls

• Without the per-player rate cap, a handful of hyperactive top players can dominate write volume on the specific shard holding the top-N, since by definition they're the ones near the top; the cap is what keeps that from becoming a hot-key problem (one specific record, here the top-N list, getting a disproportionate share of traffic and overloading the single shard that holds it, even though the system overall has plenty of spare capacity elsewhere).
• The global merge is only as fresh as its merge interval; a design that promises true real-time global ranking is overpromising unless it accepts a correspondingly higher merge frequency and cost.
• Keeping each region's top-N slightly larger than the 100 actually served (for example, 200) protects merge accuracy against players near the boundary between regions' top lists, at a small extra memory cost.
• A common shallow answer proposes exact global ranking with no rate limiting and no batching, which looks correct on a diagram but doesn't survive a handful of players updating every fraction of a second; naming the rate-limiting mechanism explicitly is the senior signal here.

Direct answer

Build-vs-buy for a core platform capability like authentication or observability comes down to weighing three sets of criteria: technical (does the vendor cover the required functionality without excessive integration work), organizational (does the team have the skills and bandwidth to build and operate it, and is it a genuine differentiator worth owning), and financial (total cost of ownership over several years versus subscription cost, and the opportunity cost of the engineering time either path consumes). Commodity capabilities with real compliance or reliability requirements usually favor buying; capabilities that are a genuine competitive differentiator favor building.

Structured elaboration

Technical criteria: feature coverage against requirements (for authentication: single sign-on/OpenID Connect support, role-based access control; for observability: traces, metrics, logs, retention), integration complexity and API quality, scalability and the vendor's own reliability track record, and how hard it would be to migrate away later (portability, data export).

Organizational criteria: whether the team has the skills and spare capacity to build and operate this well, how urgent time-to-market pressure is, and whether this capability is a genuine long-term differentiator for the product or a commodity everyone needs and nobody differentiates on. Building a commodity capability is usually a distraction from what the team should be differentiating on.

Financial criteria: total cost of ownership (TCO) over a multi-year horizon (engineering time to build and maintain, hosting, licensing), the opportunity cost of the features that don't get built while the team builds this instead, and how predictable vendor pricing is versus the variability of an in-house maintenance burden.

This same framework generalizes beyond auth and observability. Adopting a search-as-a-service offering instead of operating a self-hosted search cluster turns on exactly the same criteria: cost, time-to-market, vendor lock-in, the quality of the vendor's service-level agreements (SLAs), and compliance requirements, the same list that applies to auth-as-a-service.

Worked example

Take authentication for a product with 500,000 monthly active users (MAU), comparing a managed identity provider against building in-house. Buy, at an illustrative $0.05 per MAU per month:
$\text{buy: } 500{,}000 \text{ MAU} \times \$0.05/\text{MAU-month} = \$25{,}000/\text{month} \Rightarrow \$25{,}000 \times 36 = \$900{,}000 \text{ over 3 years}$
Build, assuming two engineers dedicated to building and operating it at an illustrative fully loaded cost of $180,000/year each:
$\text{build: } 2 \text{ engineers} \times \$180{,}000/\text{year} = \$360{,}000/\text{year} \Rightarrow \$360{,}000 \times 3 = \$1{,}080{,}000 \text{ over 3 years}$
At this scale and time horizon, buying is roughly $180,000 cheaper over three years, before even counting the opportunity cost of the two engineers' time not going toward the product's actual differentiator. That gap would close or reverse at a different MAU count or a different per-MAU vendor price, which is exactly why this needs to be computed per situation rather than assumed.

Trade-offs & pitfalls

• The crossover point between build and buy moves with scale (MAU, request volume): a TCO comparison done once at launch can become wrong as the product grows, so it should be revisited, not treated as permanent.
• Vendor lock-in risk is real but is a cost to be mitigated (contract exit clauses, data portability, thin integration layers), not an automatic reason to build; building in-house has its own lock-in in the form of institutional knowledge walking out the door.
• A common weak answer treats "build" as inherently more control and "buy" as inherently faster, without pricing either side; the financial criterion is the one most often skipped under interview time pressure.
• Compliance requirements can flip the decision entirely regardless of cost: if a vendor can't meet a required certification, buy is off the table no matter how favorable the TCO looks.

Direct answer

Treat this as finding the cheapest lever that gets the 95th-percentile (P95) latency under 200 ms, not as a single yes-or-no spend decision: price out each candidate lever (a bigger cache, more instances, a faster database tier) in dollars per millisecond improved, then take the cheapest ones first until the target is hit or the budget runs out.

Structured elaboration

Levers and how to price them

Build a small table of candidate levers, each with its cost delta and its expected latency delta, measured or estimated from a canary or A/B test, then rank by dollars per millisecond and fund down the list until the 200 ms target is met or the budget is spent. Whichever lever is left over is what you tell the budget owner you couldn't afford.

Worked example: the absorbed cache lever

Assume 1,000 requests per second (RPS), a current cache hit rate of 70% (so 30% of requests, 300 RPS, reach the origin), and each origin instance safely handles 20 RPS at the target latency, with 1.5x headroom for safety margin:

Raising the hit rate to 90% (bigger cache, longer time-to-live) drops origin load to:

At an illustrative $0.10 per instance-hour (730 hours/month):

The cost being traded here is staleness: a longer time-to-live bounds how out of date a cached response can be, so the real trade is "up to N seconds of staleness" versus "$1,095/month of origin capacity," not latency versus cost in the abstract.

Comparing against a direct lever

If instead a database-tier upgrade costs an illustrative extra $4,000/month and takes P95 from 260 ms to 190 ms:

Compare that figure against the cache lever's effective dollars-per-ms and take whichever is cheaper first; reach for the database upgrade only once the cheaper levers are exhausted and the target is still not met.

Trade-offs & pitfalls

• Pitfall: comparing levers by their sticker cost instead of their cost per millisecond improved; a cheap lever that barely moves P95 can be worse value than an expensive one that clears the whole target.
• Caching trades latency and cost for staleness, not for nothing; a time-to-live long enough to matter for cost has to be checked against what the product can tolerate seeing stale.
• Pitfall: optimizing average latency instead of P95; a lever that helps the median can leave the tail, and the service-level objective (SLO) you're actually measured on, untouched.
• Validate every lever with a real canary or A/B test before committing budget; the arithmetic above is a planning estimate, not a substitute for measuring it.

Direct answer

The first-pass architecture has three jobs: shape and throttle traffic at the edge so the backend never sees the raw 10 million attempts at once, smooth the accepted traffic through a durable queue before it touches checkout logic, and keep an authoritative reconciliation step to catch and repair any inventory oversell that slips through. Each piece exists because the naive path (let every request hit the database directly) would collapse the datastore in the first seconds of the sale.

Structured elaboration

Edge shaping and throttling. A content delivery network (CDN) and web application firewall (WAF) sit in front, followed by an admission controller that applies token-bucket rate limits per IP/user (each client accrues "tokens" over time up to a fixed cap, and each request spends one token, so short bursts are allowed while the sustained rate stays capped) and can present a "waiting room" queue-position page instead of a raw error. This exists because most of the 10 million attempts are read-heavy page loads and refresh-spam, not real purchase intent; rejecting or queuing cheaply at the edge protects everything downstream.

Checkout queueing. Accepted checkout attempts become durable messages on a partitioned queue (partitioned per stock-keeping unit, so one hot item's traffic doesn't starve others), consumed by checkout workers at a controlled concurrency. This exists because the queue absorbs the burst and lets the backend process at a rate it can sustain, instead of every accepted request triggering an immediate synchronous database write.

Inventory reservation. Workers decrement inventory in a fast in-memory store guarded by atomic operations. Two named mechanisms exist for this, and the choice is a real trade-off: a pessimistic approach acquires a short-lived reservation lock before confirming, which guarantees no oversell but adds contention and reduces throughput under massive concurrent demand; an optimistic approach atomically decrements a counter and confirms after the fact, which sustains far higher throughput but can let the counter go transiently negative, requiring reconciliation. A common middle ground is optimistic decrement for most items with a pessimistic fallback check for low-stock or high-value items, where an oversell would be most costly.

Reconciliation. An event log of every reservation and decrement feeds a reconciliation job that compares committed orders against authoritative inventory after the sale, applying compensating actions (cancel lowest-priority orders, offer backorder or credit) for anything oversold.

flowchart LR
  A[Client] --> B[CDN and WAF edge]
  B --> C[Admission controller: token-bucket rate limit]
  C --> D[Checkout API]
  D --> E[Durable queue, per-SKU partitions]
  E --> F[Checkout workers]
  F --> G[Inventory cache: atomic decrement]
  G --> H[(Authoritative inventory DB)]
  F --> I[Event log]
  I --> J[Reconciliation job]
  J --> H
  J --> K[Compensation: cancel, refund, notify]

Worked example

The stated load is 10 million checkout attempts in a 10-minute window:
$\text{avg rate} = \frac{10{,}000{,}000 \text{ attempts}}{600 \text{ s}} \approx 16{,}667 \text{ req/s}$
That average masks the real shape of a flash sale: demand is heavily front-loaded in the opening seconds. Using an illustrative 8x opening-minute burst factor over the average (a labeled assumption, not a measured figure):
$\text{illustrative peak} = 16{,}667 \times 8 \approx 133{,}333 \text{ req/s}$
This is the number the edge admission controller and queue have to be sized for, not the smoothed 16,667 req/s average; sizing to the average alone would let the opening burst overwhelm the checkout tier before the throttle has a chance to shed load.

Trade-offs & pitfalls

• Sizing capacity to the average rate instead of the front-loaded peak is the most common mistake; a flash sale's demand curve is not uniform across the window.
• Pessimistic locking is safer but throttles throughput exactly when throughput matters most; optimistic decrement is faster but pushes correctness work into reconciliation, which only works if compensating actions (refunds, notifications) are also designed up front, not bolted on after an incident.
• A queue partitioned too coarsely (one partition for all items) recreates the same hot-item contention the partitioning was meant to avoid; partition by item or shard further for known hot stock-keeping units.
• Treating reconciliation as an afterthought is a red flag; for a hard system-design question the reconciliation and compensation path is as much a first-class component as the checkout path itself.