System Design Methodology and Trade-off Analysis Questions

Direct answer

Size the fleet from a single relationship, Little's Law, rather than guessing a replica count directly: pick a per-request service time and a concurrency budget per replica, derive that replica's sustainable throughput, divide the target queries per second (QPS) by it, then add headroom so the fleet runs below saturation, since running near 100% utilization is exactly what blows up the 95th-percentile (P95) tail this plan is trying to protect.

Structured elaboration

The relationship. Little's Law states $L = \lambda W$: the number of requests in flight ($L$) equals throughput ($\lambda$) times average time in the system ($W$). Inverting it per replica: if a replica holds $C$ requests concurrently and each takes $W$ seconds, its sustainable throughput is $C / W$.

Decision criteria for the assumptions:

• Service time budget: must leave room under the 50 ms P95 target for queueing and network overhead, not consume the whole budget on compute alone (recall from queueing math that latency blows up as utilization nears 100%, so some of the 50 ms has to be slack, not service time).
• Concurrency per replica should be tied to actual provisioned resources (for example, one in-flight request per vCPU core for a compute-bound serving path), not picked independently of the CPU budget, otherwise the CPU and throughput numbers won't reproduce each other.
• Headroom must cover both a safety margin against tail-latency blowup (target well under 100% utilization) and operational headroom (rolling deploys, node loss).

Worked example

Assumptions (illustrative, stated explicitly): average per-request service time $W = 20\text{ms} = 0.02\text{s}$, leaving roughly 30 ms of the 50 ms P95 budget as network and queueing slack; each replica is provisioned with $C = 10$ concurrent in-flight requests, matched to 10 vCPU cores (one request per core).

Per-replica throughput via Little's Law: $\lambda_{\text{replica}} = C / W = 10 / 0.02 = 500 \text{ QPS}$.

Replicas for raw throughput at the 10,000 QPS target: $10{,}000 / 500 = 20$ replicas.

Add headroom for two separate reasons, both real costs, not one combined guess:

• Utilization headroom: target roughly 70% utilization to keep queueing delay small and protect the P95 tail: $20 / 0.7 \approx 28.6 \to 29$ replicas.
• Rolling-update / failure headroom: reserve capacity equivalent to roughly 2 replicas being unavailable at any time: $29 + 2 = 31$ replicas.

Resource budget per replica, tied directly to the concurrency assumption above rather than picked independently: 10 vCPU (matching $C=10$), plus memory for the served model (illustrative 4 GB) and runtime/framework overhead (illustrative 1 GB), rounded up with a small buffer to 6 GB RAM per replica.

Trade-offs & pitfalls

• The most common mistake in this kind of estimate is stating a per-replica throughput number that isn't derived from the stated service-time and resource assumptions, for example claiming 200 QPS per replica on 2 vCPU with 30 ms of CPU work per request implies a maximum of roughly $2 / 0.03 \approx 67$ QPS per replica, not 200; always check that the throughput, service time, and resource assumptions are mutually consistent before sizing the fleet on them.
• Sizing purely for average throughput and skipping the utilization headroom step will hit the QPS target on paper while missing the P95 latency target in practice, because queueing delay is non-linear near saturation.
• CPU and memory are independent constraints, a replica sized correctly for CPU-bound throughput can still be starved on memory if the model or working set doesn't fit, both must be checked, not just one.
• Validate every assumption with a real load test before committing capacity: ramp to the target QPS, hold it, and confirm the measured P95 matches the plan; if it doesn't, the service-time or concurrency assumption was wrong, not the arithmetic.

Direct answer

Size against the throughput a single GPU can sustain while still meeting your latency ceiling, not its theoretical peak throughput, because batching trades latency for throughput and both constraints have to hold at once. Divide the required load by that per-GPU number, round up, and add headroom for failover and variance.

Structured elaboration

• Batching groups concurrent requests so GPU compute is shared across them, raising throughput per GPU but adding queuing and shared-compute wait time to each request.
• Larger batch sizes keep raising throughput per GPU only up to a point; eventually you hit the p95 latency budget or the GPU's memory limit (the KV-cache, per-request memory the model keeps around during generation so it doesn't have to recompute earlier tokens on every step, grows with both batch size and sequence length).
• Fleet size = required QPS divided by per-GPU sustainable QPS at your latency ceiling, plus N+1 redundancy; steady-state size and autoscaling headroom are two different numbers.

Worked example

Assume profiling shows batch size 1 takes 80ms per request (p95 latency, the 95th-percentile response time), well under the 300ms budget:

$\text{unbatched QPS per GPU} = \frac{1000\text{ms}}{80\text{ms}} = 12.5$
$\text{unbatched fleet} = \lceil 200 / 12.5 \rceil = 16 \text{ GPUs}$

Now assume batch size 8 takes 140ms per batch (still under the 300ms budget):

$\text{batched QPS per GPU} = \frac{1000\text{ms}}{140\text{ms}} \times 8 \approx 57.1$
$\text{batched fleet} = \lceil 200 / 57.1 \rceil = 4 \text{ GPUs}$

Batching cuts the fleet from 16 to 4 GPUs, a 4x reduction, while per-request latency rises from 80ms to 140ms, still inside the 300ms p95 budget. Add one GPU for failover: 5 GPUs total versus 17 unbatched.

Trade-offs and pitfalls

Bigger batches help only until you hit a new latency ceiling or run out of GPU memory; past that you are compute-bound, not batching-bound, and no batching buys back headroom. Static batching forces every request to wait for the batch to fill; dynamic or continuous batching captures most of the savings without that wait.

What the interviewer probes next

Expect questions on how you'd handle the request that arrives just after a batch closes, how you'd autoscale for a traffic spike without over-provisioning steady state, and what happens to this math when sequence lengths vary widely.

Direct answer

Quantization (representing weights and activations with fewer bits, such as 8-bit integers instead of 32-bit floats) is the cheaper first move: no retraining, and a small, well-characterized accuracy cost for a large memory and speed win. Distillation (training a smaller "student" model to mimic a larger "teacher") costs more engineering time but can recover more accuracy at a given size. Often you do both: distill to a smaller architecture, then quantize the result.

Structured elaboration

Quantization: changes numeric precision only, a calibration pass not a retrain, hours not days; 32-bit to 8-bit is a 4x memory reduction, with accuracy risk growing below 8-bit.

Distillation: changes architecture and parameter count, trained against the teacher's soft labels, days not hours plus a new evaluation cycle; preserves more accuracy per parameter than numeric coarsening alone.

Worked example

Original model 400MB in 32-bit floats, budget 100MB, baseline accuracy 92%, tolerance under 2 points. Quantizing to 8-bit:

$\text{size}_{\text{int8}} = \frac{400\text{MB}}{4} = 100\text{MB}$

exactly hits budget, and 8-bit post-training quantization commonly costs under 1 point, landing near 91-91.5%. If the measured drop is larger, say 88% (4 points, over budget), distill first to a 120MB float32 student, then quantize it:

$\text{size}_{\text{student-int8}} = \frac{120\text{MB}}{4} = 30\text{MB}$

comfortably inside budget, with the accuracy cost spent on a distillation step you can iterate on, not on coarsening you cannot.

Trade-offs and pitfalls

Quantization is fast but its accuracy hit is hard to fully control below 8-bit; distillation gives more control but is a multi-day project that may not converge on rare classes. A common pitfall is checking only aggregate accuracy and missing that compression hurt one rare, high-stakes class far more than average.

What the interviewer probes next

Checking per-class accuracy degradation rather than the aggregate, post-training quantization versus quantization-aware training, and how target hardware (INT8 kernel support) changes the plan.

Direct answer

Latency and throughput are not opposites by nature, they trade off through queueing: pushing more concurrent work through a fixed amount of processing capacity increases the time each request waits behind others, and holding latency low means keeping spare capacity in reserve rather than running it flat out. The right balance comes from setting an explicit target for both (a throughput floor and a tail-latency ceiling), then using queueing math plus load testing to find the utilization level where more throughput starts costing more latency than the business can absorb. What to measure at each load level: the full latency distribution, not just the average, including the 95th and 99th percentile (P95/P99), alongside the downstream business metric (conversion rate, task completion time) the latency target exists to protect.

Structured elaboration

Why the tension exists. Little's Law ties the three quantities together:

$L = \lambda W$

where $L$ is the average number of requests in the system (concurrency), $\lambda$ is the arrival rate (throughput), and $W$ is the average time a request spends in the system (latency). For a fixed amount of concurrency capacity $L$, pushing $\lambda$ up forces $W$ up. Throughput and latency are linked by whatever capacity sits between them, they only look independent at low load.

Decision criteria to walk through, in order:

• Is there a hard external constraint (a contractual service-level agreement, or SLA) versus a soft internal preference? Hard constraints bound the feasible region before you optimize anything.
• Is the load steady or bursty? A bursty workload needs headroom sized for the peak, not the average, or tail latency spikes during every burst.
• What is the true cost of extra capacity relative to the revenue or reliability cost of extra latency? If compute is cheap relative to the business impact of latency, buy headroom instead of accepting queueing.
• Which metric does the product actually care about, median latency almost never predicts user-visible pain, the tail does.

Process: baseline the current latency distribution and throughput, ramp load in steps while recording the full distribution at each step, locate the point where the P95 or P99 curve bends upward sharply (the "knee"), then correlate that knee to the business metric to decide whether operating past it is acceptable.

Worked example

Assume, for illustration, a single worker with an average service time of 10 ms per request ($S = 0.01\text{s}$), so its theoretical maximum throughput is $1/S = 100$ requests per second (RPS). Using the M/M/1 queueing approximation (a standard model for one server handling one request at a time, with randomly arriving requests and randomly varying service times, a common simplification for a single queue), the average wait time in queue at utilization $\rho = \lambda S$ is:

$W_q = \frac{\rho \cdot S}{1-\rho}$

Reproducing the middle row: $W_q = \dfrac{0.90 \times 0.01}{1-0.90} = \dfrac{0.009}{0.10} = 0.09\text{s} = 90\text{ms}$, so $W = 90 + 10 = 100\text{ms}$. Going from 70 to 90 RPS (a 29% throughput increase) roughly triples latency; the next 5.6% of throughput (90 to 95 RPS) roughly doubles it again. This is the shape of the trade-off: throughput gains near saturation cost latency disproportionately.

The same law sizes capacity to hit both targets at once. To sustain 5,000 RPS at an average latency target of 15 ms, the required in-flight concurrency is $L = \lambda W = 5000 \times 0.015 = 75$ concurrent request slots. If each server instance can hold 25 concurrent requests (its thread or connection budget), raw sizing needs $75 / 25 = 3$ instances, but running at 100% utilization guarantees queueing, so target roughly 65% utilization for headroom: $3 / 0.65 \approx 4.6$, round up to 5 instances.

Trade-offs & pitfalls

• Treating the median as the target metric hides exactly the users experiencing queueing delay, always instrument and alert on the tail, not the average.
• Adding raw compute capacity fixes queueing-induced latency but does nothing for latency caused by serialization cost or an inefficient algorithm, these are different bottleneck classes and need different fixes (see bottleneck-identification questions for the diagnostic process).
• Batching or coalescing requests can raise both average throughput and average latency-per-request while making the tail worse for whichever request lands first in a batch, batching trades individual completion time for aggregate efficiency and needs a separate tail-latency check.
• Autoscaling on CPU utilization alone can under-react to a pure queueing problem, alerting or scaling on the latency percentile itself, or on queue depth, catches the tension directly.
• Always tie the chosen operating point back to the business metric with real data (an A/B test or canary), a default like "P95 under 300 ms" is only correct if it is where the business metric actually degrades.

Direct answer

Use synchronous scoring only when the action on the score must happen before the transaction completes, such as blocking a fraudulent charge. If the action can wait (nightly review, retraining labels), batch scoring buys a heavier, more accurate model at a fraction of the cost. The deciding question: what does a delayed decision cost you, versus an always-on low-latency fleet?

Structured elaboration

Real-time: one hop in a hard end-to-end latency budget, needs near-zero-staleness features (a hot feature store: a system that serves the same precomputed input values, like a rolling transaction count, to the model at scoring time; "hot" means it's updated in near real time rather than nightly), and an always-on fleet sized for peak plus a fallback if the call times out.

Batch: runs on a schedule, so it can use a bigger, slower model on cheaper bursty compute, but adds a detection lag equal to the batch interval, and a partial failure is a silent under-score, not an outage.

Worked example

A 250ms authorize/decline budget, with auth, ledger, and notification already at 180ms:

$\text{fraud model budget} = 250\text{ms} - 180\text{ms} = 70\text{ms}$

Subtract 20ms network/serialization overhead:

$\text{compute budget} = 70\text{ms} - 20\text{ms} = 50\text{ms}$

That rules out heavy ensembles (combining predictions from several models, which multiplies the per-request compute cost) needing multiple feature joins (a feature store lookup that assembles several separate precomputed values into one input record, each join adding its own latency): illustratively, a 3-model ensemble with 2 extra feature joins might cost 90-120ms on its own, already over the 50ms budget, which is what "rules out" means here concretely. For batch: 10 million transactions overnight, 500 records per inference batch, 50ms per batch on one GPU:

$\text{batches} = \frac{10{,}000{,}000}{500} = 20{,}000, \quad \text{GPU time} = 20{,}000 \times 50\text{ms} \approx 16.7\text{ min}$

Across 4 GPUs, about 4 minutes wall time, no idle fleet cost during the day.

Trade-offs and pitfalls

Real-time buys speed but pays for peak capacity around the clock and widens the request's failure surface. Batch is cheaper but the fraud can complete before you act. A common mistake is defaulting to real-time without pricing the fleet against the actual cost of delay. The answer flips when either side of that equation moves: if fraud losses from a delayed decision start to dwarf the cost of an always-on fleet, or a cheaper real-time model becomes accurate enough to fit the leftover latency budget, you're pushed toward real-time; if the model needs more compute than the authorization path can spare, or false declines against legitimate customers become the bigger cost, you're pushed back toward batch.

What the interviewer probes next

Hybrid streaming or micro-batch designs, detecting a batch job that fails partway through, and how the answer shifts if a missed fraud case gets an order of magnitude costlier.