System Design Methodology and Trade-off Analysis Questions

Direct answer

Size against the throughput a single GPU can sustain while still meeting your latency ceiling, not its theoretical peak throughput, because batching trades latency for throughput and both constraints have to hold at once. Divide the required load by that per-GPU number, round up, and add headroom for failover and variance.

Structured elaboration

• Batching groups concurrent requests so GPU compute is shared across them, raising throughput per GPU but adding queuing and shared-compute wait time to each request.
• Larger batch sizes keep raising throughput per GPU only up to a point; eventually you hit the p95 latency budget or the GPU's memory limit (the KV-cache, per-request memory the model keeps around during generation so it doesn't have to recompute earlier tokens on every step, grows with both batch size and sequence length).
• Fleet size = required QPS divided by per-GPU sustainable QPS at your latency ceiling, plus N+1 redundancy; steady-state size and autoscaling headroom are two different numbers.

Worked example

Assume profiling shows batch size 1 takes 80ms per request (p95 latency, the 95th-percentile response time), well under the 300ms budget:

$\text{unbatched QPS per GPU} = \frac{1000\text{ms}}{80\text{ms}} = 12.5$
$\text{unbatched fleet} = \lceil 200 / 12.5 \rceil = 16 \text{ GPUs}$

Now assume batch size 8 takes 140ms per batch (still under the 300ms budget):

$\text{batched QPS per GPU} = \frac{1000\text{ms}}{140\text{ms}} \times 8 \approx 57.1$
$\text{batched fleet} = \lceil 200 / 57.1 \rceil = 4 \text{ GPUs}$

Batching cuts the fleet from 16 to 4 GPUs, a 4x reduction, while per-request latency rises from 80ms to 140ms, still inside the 300ms p95 budget. Add one GPU for failover: 5 GPUs total versus 17 unbatched.

Trade-offs and pitfalls

Bigger batches help only until you hit a new latency ceiling or run out of GPU memory; past that you are compute-bound, not batching-bound, and no batching buys back headroom. Static batching forces every request to wait for the batch to fill; dynamic or continuous batching captures most of the savings without that wait.

What the interviewer probes next

Expect questions on how you'd handle the request that arrives just after a batch closes, how you'd autoscale for a traffic spike without over-provisioning steady state, and what happens to this math when sequence lengths vary widely.

Direct answer

Prioritize by expected business impact combined with how quickly the failure mode compounds if unaddressed: a region outage first, because it's a full-availability event with no partial-degradation option; a sudden 10x traffic spike second, because it threatens availability but usually has partial mitigations (throttling, degraded modes) available immediately; and an upstream schema change third, because it's typically detectable and containable with fast rollback before it causes user-facing damage, even though it can silently corrupt data if left uncaught.

Structured elaboration

For each curveball, separate the immediate runbook response from the longer-term architectural change it justifies.

Region outage. Immediate: fail over reads and writes to a secondary region using health-checked traffic routing, and pause non-essential batch work to reduce write pressure during the transition. Architectural change: multi-region active-passive (or active-active) replication for the data layer, with regularly rehearsed failover drills; a design that was never built to fail over won't fail over correctly under real pressure, only under a rehearsed one.

Sudden 10x traffic spike. Immediate: autoscale the serving tier, shed or degrade non-critical functionality (serve cached or slightly stale results rather than fail outright), and throttle low-priority background jobs to protect the real-time path. Architectural change: pre-warmed capacity headroom, adaptive rate limiting, and a defined degraded mode that's tested before it's needed, not designed during the incident.

Upstream schema change breaking the data pipeline. Immediate: fail fast on schema-validation errors at ingestion rather than let malformed data propagate, quarantine the bad batch, and roll the downstream transform back to the last known-good schema. Architectural change: enforce a schema contract at the pipeline boundary (a strongly typed serialization format with a compatibility check, such as Avro or Protocol Buffers) so a breaking upstream change is caught at ingestion rather than discovered downstream after it has already corrupted derived data.

Worked example

An illustrative prioritization exercise, scoring each curveball on business impact (1 low to 5 high) and detectability/containability (1 hard to 5 easy) to make the ranking auditable rather than a gut call: region outage scores high impact (5/5: full outage, all users) and moderate containability (3/5: requires a rehearsed failover, not just a code fix); 10x traffic spike scores high impact if unmitigated (4/5) but higher containability (4/5: autoscaling and shedding are standard, fast-acting levers); schema break scores lower immediate user-facing impact (2/5: the pipeline can often keep serving stale-but-correct data while paused) but containability that depends entirely on whether validation exists at the ingestion boundary (2/5 without it), if it doesn't, undetected corruption can silently spread for a long time before anyone notices, which is exactly why validation is the priority architectural investment for that curveball specifically, even though it's ranked last for immediate response.

Trade-offs & pitfalls

• Ranking these purely by which is scariest in the abstract, rather than by business impact and how fast each compounds if left unaddressed, produces a plausible-sounding but ungrounded priority order; tie the ranking to a concrete criterion.
• A schema break that lacks ingestion-time validation is deceptively low-priority in the short term and highest-priority for silent, compounding damage; don't let "least immediately visible" become "least urgent to architect for."
• Building all three mitigations simultaneously from scratch during a single design pass is rarely realistic; sequence the architectural investments and say explicitly which curveball's mitigation ships first and why.
• Rehearsing failure (game days, chaos testing, restore drills) is what turns a runbook from theory into something that actually works under pressure; a runbook that has never been executed is a plan, not a capability.

Direct answer

Treat this as a constrained optimization, not a wishlist: list every cost lever, estimate each one's savings and its latency/service-level-objective (SLO) risk independently, combine the savings correctly (multiplicatively, since each lever applies to whatever cost remains after the prior ones, not additively), and sequence the lowest-risk, highest-confidence levers first so you are validating architecture changes only if the safe levers don't already close the gap.

Structured elaboration

Categorize levers by risk to latency and SLOs, not just by savings size:

• Commitment-based (reserved capacity, savings plans on predictable baseline usage): near-zero runtime risk, same infrastructure, different billing.
• Right-sizing and off-peak scheduling: low risk if headroom and monitoring are retained, touches capacity, not request-path logic.
• Caching improvements: moderate risk, changes the request path and introduces a staleness trade-off, needs a pilot.
• Consolidation or replacing a managed service: highest risk, changes topology or introduces new operational surface, needs a staged rollout with a rollback path.

Execution plan: run the low-risk levers first and measure actual savings against current spend, only reach for a higher-risk lever if the low-risk set doesn't clear the target, and size that higher-risk lever to close exactly the remaining gap rather than over-applying it.

Worked example

Assume four levers, sequenced from lowest to higher risk, each estimated independently:

Combined savings are multiplicative on remaining cost, not additive, because each lever's percentage applies to whatever spend is left after the prior levers:

$\text{remaining fraction} = (1-0.15)(1-0.10)(1-0.08)(1-0.05)$

Computing stepwise: $0.85 \times 0.90 = 0.765$; $0.765 \times 0.92 = 0.7038$; $0.7038 \times 0.95 = 0.66861$.

Remaining fraction $\approx 0.6686$, so total reduction $\approx 1 - 0.6686 = 0.3314 = 33.1\%$, clearing the 30% target with roughly 3 percentage points of margin for estimation error, using only levers with low-to-moderate individual latency risk and none requiring the highest-risk consolidation lever.

If these four levers had instead totaled, say, 24%, that is the point to reach for a higher-risk lever (service consolidation or replacing a managed component), sized with the same multiplicative method to close exactly the remaining gap, and gated behind a canary rollout given its higher risk to latency and SLOs.

Trade-offs & pitfalls

• Adding percentages linearly ($15+10+8+5=38\%$) overstates the true combined savings (33.1% here) and can make a plan look like it clears the ceiling when it doesn't, always combine sequential percentage savings multiplicatively.
• Reaching for the single biggest-percentage lever first, even when it's also the highest-risk one, instead of exhausting low-risk levers first, front-loads risk unnecessarily when a safer combination might already hit the target.
• Measuring "savings" against a stale baseline instead of current spend produces accounting surprises when finance reconciles the actual bill.
• Latency and SLO risk aren't uniform across levers, track a risk budget alongside the dollar target, a plan that hits 30% savings but blows through 15% latency increase on one lever has still failed the actual constraint.

Direct answer

A cascade, a cheap filter first and an expensive model only on the fraction it is unsure about, wins whenever the input mix is skewed toward easy cases, because it lets you pay the expensive model's cost only on the hard tail. A single large model wins when you cannot build a cheap filter with high enough recall on the escalation decision itself, since a weak filter silently drops cases that needed the expensive model's judgment.

Structured elaboration

• Cascade design: a cheap stage-1 classifier resolves confident cases directly; anything below a confidence threshold is escalated to the expensive stage-2 model.
• What matters most: not the cascade's average accuracy, but the stage-1 filter's recall on "this needs escalation," a miss there means the expensive model never sees a case that needed it.
• Complexity cost: two models to version and monitor for drift, and request latency now varies by path length, some requests take one hop, some take two.
• Single model: simpler to operate, but every request pays the expensive model's full cost, even the easy 90%.

Worked example

10 million posts/day, cheap filter $0.0001/post, expensive model $0.01/post, and the filter confidently resolves 90% while escalating the other 10%:

$\text{cascade cost} = 10{,}000{,}000 \times \$0.0001 + 1{,}000{,}000 \times \$0.01 = \$1{,}000 + \$10{,}000 = \$11{,}000/\text{day}$
$\text{all-expensive cost} = 10{,}000{,}000 \times \$0.01 = \$100{,}000/\text{day}$

That is roughly 89% cheaper. But if the filter's recall on "needs escalation" is only 95% against a true escalation population of 1,000,000:

$\text{missed escalations} = 1{,}000{,}000 \times (1 - 0.95) = 50{,}000/\text{day}$

50,000 posts a day get moderated by the wrong tier, a real cost that has to be weighed against the $89,000/day saved.

Trade-offs and pitfalls

Cascades only make sense if you can actually measure the filter's recall on the escalation decision, not just its overall resolve rate. Latency variance (a two-hop tail versus a one-hop median) can also blow past a p99 budget if not accounted for.

What the interviewer probes next

Expect follow-ups on how you would set and validate the confidence threshold, how to monitor drift when the two models retrain on different schedules, and how adversarial inputs crafted to slip past the cheap filter change the design.

Direct answer

Use synchronous scoring only when the action on the score must happen before the transaction completes, such as blocking a fraudulent charge. If the action can wait (nightly review, retraining labels), batch scoring buys a heavier, more accurate model at a fraction of the cost. The deciding question: what does a delayed decision cost you, versus an always-on low-latency fleet?

Structured elaboration

Real-time: one hop in a hard end-to-end latency budget, needs near-zero-staleness features (a hot feature store: a system that serves the same precomputed input values, like a rolling transaction count, to the model at scoring time; "hot" means it's updated in near real time rather than nightly), and an always-on fleet sized for peak plus a fallback if the call times out.

Batch: runs on a schedule, so it can use a bigger, slower model on cheaper bursty compute, but adds a detection lag equal to the batch interval, and a partial failure is a silent under-score, not an outage.

Worked example

A 250ms authorize/decline budget, with auth, ledger, and notification already at 180ms:

$\text{fraud model budget} = 250\text{ms} - 180\text{ms} = 70\text{ms}$

Subtract 20ms network/serialization overhead:

$\text{compute budget} = 70\text{ms} - 20\text{ms} = 50\text{ms}$

That rules out heavy ensembles (combining predictions from several models, which multiplies the per-request compute cost) needing multiple feature joins (a feature store lookup that assembles several separate precomputed values into one input record, each join adding its own latency): illustratively, a 3-model ensemble with 2 extra feature joins might cost 90-120ms on its own, already over the 50ms budget, which is what "rules out" means here concretely. For batch: 10 million transactions overnight, 500 records per inference batch, 50ms per batch on one GPU:

$\text{batches} = \frac{10{,}000{,}000}{500} = 20{,}000, \quad \text{GPU time} = 20{,}000 \times 50\text{ms} \approx 16.7\text{ min}$

Across 4 GPUs, about 4 minutes wall time, no idle fleet cost during the day.

Trade-offs and pitfalls

Real-time buys speed but pays for peak capacity around the clock and widens the request's failure surface. Batch is cheaper but the fraud can complete before you act. A common mistake is defaulting to real-time without pricing the fleet against the actual cost of delay. The answer flips when either side of that equation moves: if fraud losses from a delayed decision start to dwarf the cost of an always-on fleet, or a cheaper real-time model becomes accurate enough to fit the leftover latency budget, you're pushed toward real-time; if the model needs more compute than the authorization path can spare, or false declines against legitimate customers become the bigger cost, you're pushed back toward batch.

What the interviewer probes next

Hybrid streaming or micro-batch designs, detecting a batch job that fails partway through, and how the answer shifts if a missed fraud case gets an order of magnitude costlier.