System Design Methodology and Trade-off Analysis Questions

Direct answer

A cascade, a cheap filter first and an expensive model only on the fraction it is unsure about, wins whenever the input mix is skewed toward easy cases, because it lets you pay the expensive model's cost only on the hard tail. A single large model wins when you cannot build a cheap filter with high enough recall on the escalation decision itself, since a weak filter silently drops cases that needed the expensive model's judgment.

Structured elaboration

• Cascade design: a cheap stage-1 classifier resolves confident cases directly; anything below a confidence threshold is escalated to the expensive stage-2 model.
• What matters most: not the cascade's average accuracy, but the stage-1 filter's recall on "this needs escalation," a miss there means the expensive model never sees a case that needed it.
• Complexity cost: two models to version and monitor for drift, and request latency now varies by path length, some requests take one hop, some take two.
• Single model: simpler to operate, but every request pays the expensive model's full cost, even the easy 90%.

Worked example

10 million posts/day, cheap filter $0.0001/post, expensive model $0.01/post, and the filter confidently resolves 90% while escalating the other 10%:

$\text{cascade cost} = 10{,}000{,}000 \times \$0.0001 + 1{,}000{,}000 \times \$0.01 = \$1{,}000 + \$10{,}000 = \$11{,}000/\text{day}$
$\text{all-expensive cost} = 10{,}000{,}000 \times \$0.01 = \$100{,}000/\text{day}$

That is roughly 89% cheaper. But if the filter's recall on "needs escalation" is only 95% against a true escalation population of 1,000,000:

$\text{missed escalations} = 1{,}000{,}000 \times (1 - 0.95) = 50{,}000/\text{day}$

50,000 posts a day get moderated by the wrong tier, a real cost that has to be weighed against the $89,000/day saved.

Trade-offs and pitfalls

Cascades only make sense if you can actually measure the filter's recall on the escalation decision, not just its overall resolve rate. Latency variance (a two-hop tail versus a one-hop median) can also blow past a p99 budget if not accounted for.

What the interviewer probes next

Expect follow-ups on how you would set and validate the confidence threshold, how to monitor drift when the two models retrain on different schedules, and how adversarial inputs crafted to slip past the cheap filter change the design.

Direct answer

Use synchronous scoring only when the action on the score must happen before the transaction completes, such as blocking a fraudulent charge. If the action can wait (nightly review, retraining labels), batch scoring buys a heavier, more accurate model at a fraction of the cost. The deciding question: what does a delayed decision cost you, versus an always-on low-latency fleet?

Structured elaboration

Real-time: one hop in a hard end-to-end latency budget, needs near-zero-staleness features (a hot feature store: a system that serves the same precomputed input values, like a rolling transaction count, to the model at scoring time; "hot" means it's updated in near real time rather than nightly), and an always-on fleet sized for peak plus a fallback if the call times out.

Batch: runs on a schedule, so it can use a bigger, slower model on cheaper bursty compute, but adds a detection lag equal to the batch interval, and a partial failure is a silent under-score, not an outage.

Worked example

A 250ms authorize/decline budget, with auth, ledger, and notification already at 180ms:

$\text{fraud model budget} = 250\text{ms} - 180\text{ms} = 70\text{ms}$

Subtract 20ms network/serialization overhead:

$\text{compute budget} = 70\text{ms} - 20\text{ms} = 50\text{ms}$

That rules out heavy ensembles (combining predictions from several models, which multiplies the per-request compute cost) needing multiple feature joins (a feature store lookup that assembles several separate precomputed values into one input record, each join adding its own latency): illustratively, a 3-model ensemble with 2 extra feature joins might cost 90-120ms on its own, already over the 50ms budget, which is what "rules out" means here concretely. For batch: 10 million transactions overnight, 500 records per inference batch, 50ms per batch on one GPU:

$\text{batches} = \frac{10{,}000{,}000}{500} = 20{,}000, \quad \text{GPU time} = 20{,}000 \times 50\text{ms} \approx 16.7\text{ min}$

Across 4 GPUs, about 4 minutes wall time, no idle fleet cost during the day.

Trade-offs and pitfalls

Real-time buys speed but pays for peak capacity around the clock and widens the request's failure surface. Batch is cheaper but the fraud can complete before you act. A common mistake is defaulting to real-time without pricing the fleet against the actual cost of delay. The answer flips when either side of that equation moves: if fraud losses from a delayed decision start to dwarf the cost of an always-on fleet, or a cheaper real-time model becomes accurate enough to fit the leftover latency budget, you're pushed toward real-time; if the model needs more compute than the authorization path can spare, or false declines against legitimate customers become the bigger cost, you're pushed back toward batch.

What the interviewer probes next

Hybrid streaming or micro-batch designs, detecting a batch job that fails partway through, and how the answer shifts if a missed fraud case gets an order of magnitude costlier.

Direct answer

Quantization (representing weights and activations with fewer bits, such as 8-bit integers instead of 32-bit floats) is the cheaper first move: no retraining, and a small, well-characterized accuracy cost for a large memory and speed win. Distillation (training a smaller "student" model to mimic a larger "teacher") costs more engineering time but can recover more accuracy at a given size. Often you do both: distill to a smaller architecture, then quantize the result.

Structured elaboration

Quantization: changes numeric precision only, a calibration pass not a retrain, hours not days; 32-bit to 8-bit is a 4x memory reduction, with accuracy risk growing below 8-bit.

Distillation: changes architecture and parameter count, trained against the teacher's soft labels, days not hours plus a new evaluation cycle; preserves more accuracy per parameter than numeric coarsening alone.

Worked example

Original model 400MB in 32-bit floats, budget 100MB, baseline accuracy 92%, tolerance under 2 points. Quantizing to 8-bit:

$\text{size}_{\text{int8}} = \frac{400\text{MB}}{4} = 100\text{MB}$

exactly hits budget, and 8-bit post-training quantization commonly costs under 1 point, landing near 91-91.5%. If the measured drop is larger, say 88% (4 points, over budget), distill first to a 120MB float32 student, then quantize it:

$\text{size}_{\text{student-int8}} = \frac{120\text{MB}}{4} = 30\text{MB}$

comfortably inside budget, with the accuracy cost spent on a distillation step you can iterate on, not on coarsening you cannot.

Trade-offs and pitfalls

Quantization is fast but its accuracy hit is hard to fully control below 8-bit; distillation gives more control but is a multi-day project that may not converge on rare classes. A common pitfall is checking only aggregate accuracy and missing that compression hurt one rare, high-stakes class far more than average.

What the interviewer probes next

Checking per-class accuracy degradation rather than the aggregate, post-training quantization versus quantization-aware training, and how target hardware (INT8 kernel support) changes the plan.

Direct answer

A genuinely non-negotiable constraint (a legal, regulatory, or security requirement with no waiver path) changes trade-off analysis from optimizing across all options to first pruning the option set down to only what's compliant, and only then optimizing cost, performance, or time-to-market among what's left. It doesn't get a weight in a scoring matrix alongside other factors; it eliminates options before scoring starts.

Structured elaboration

Treat a hard constraint as a filter applied in a distinct first pass, before any cost or performance comparison: list every candidate architecture, remove any that violate the constraint outright (not "weight them lower", remove them), and only run the normal trade-off analysis (cost, latency, time-to-market) across what survives. This ordering matters because scoring an already-infeasible option wastes analysis effort and can create a false sense that it was seriously considered.

Two realistic examples of constraints that eliminate options outright, not just penalize them:

PCI-DSS (Payment Card Industry Data Security Standard) card-data scope. If a design stores raw card numbers to power broader analytics, that option is gone the moment PCI-DSS applies, regardless of how much better the analytics would be; the only surviving options tokenize card data (replace the real card number with a random, non-sensitive placeholder token that maps back to it only inside the certified payment vault) or route it through an already-certified payment gateway.

Regulatory data residency. A requirement that a jurisdiction's data (for example, European Union customer data under data-protection law) must remain within that jurisdiction's borders eliminates any single-region deployment outside it outright, even if that region is meaningfully cheaper or already has spare capacity; there's no scoring adjustment that makes a non-compliant region viable.

Worked example

An illustrative scenario: a new payments feature needs to store transaction detail for both fraud analytics and customer support. Three candidate designs exist: (A) store full raw card data plus transaction detail for maximum analytics flexibility, (B) tokenize card data and store only tokens plus transaction metadata, (C) tokenize card data and additionally keep only aggregated, non-identifying analytics rather than per-transaction detail. Once PCI-DSS scope is applied as a hard filter, option A is eliminated outright, not down-weighted, because storing raw card data outside a certified, PCI-scoped environment isn't a slower or costlier version of the same design, it's a design that isn't legally available. The remaining trade-off analysis, cost and analytics fidelity, runs only between B and C: B keeps more per-transaction detail at a higher tokenization and storage cost (illustratively, storing a token plus full transaction metadata for 10 million transactions/month at roughly $0.0004/record runs about $4,000/month), C is cheaper (aggregating to per-customer monthly summaries cuts that record volume by roughly 95%, to around $200/month) but sacrifices per-transaction granularity for fraud analysis. That second-stage comparison is where a normal cost-vs-capability trade-off analysis applies; the first stage had none, only elimination.

Trade-offs & pitfalls

• The most common mistake is treating a hard constraint as one more weighted factor in a scoring matrix; that understates it and risks a stakeholder pushing back with "can we just accept a bit more risk here," when the honest answer is there's no risk-acceptance path available.
• Document what was eliminated and why, not just what was chosen; a stakeholder who wasn't in the room needs to see that the more attractive option was never actually on the table, not that it lost a close call.
• Distinguish a genuinely non-negotiable constraint from a strongly-preferred one; treating a soft preference as a hard filter needlessly shrinks the option set and can be walked back once challenged, which undermines trust in the rest of the analysis.
• Residual risk still needs to be documented and mitigated even after the hard filter is applied; "compliant" doesn't mean "risk-free," it means the specific eliminated risk is off the table.

Direct answer

Size against the throughput a single GPU can sustain while still meeting your latency ceiling, not its theoretical peak throughput, because batching trades latency for throughput and both constraints have to hold at once. Divide the required load by that per-GPU number, round up, and add headroom for failover and variance.

Structured elaboration

• Batching groups concurrent requests so GPU compute is shared across them, raising throughput per GPU but adding queuing and shared-compute wait time to each request.
• Larger batch sizes keep raising throughput per GPU only up to a point; eventually you hit the p95 latency budget or the GPU's memory limit (the KV-cache, per-request memory the model keeps around during generation so it doesn't have to recompute earlier tokens on every step, grows with both batch size and sequence length).
• Fleet size = required QPS divided by per-GPU sustainable QPS at your latency ceiling, plus N+1 redundancy; steady-state size and autoscaling headroom are two different numbers.

Worked example

Assume profiling shows batch size 1 takes 80ms per request (p95 latency, the 95th-percentile response time), well under the 300ms budget:

$\text{unbatched QPS per GPU} = \frac{1000\text{ms}}{80\text{ms}} = 12.5$
$\text{unbatched fleet} = \lceil 200 / 12.5 \rceil = 16 \text{ GPUs}$

Now assume batch size 8 takes 140ms per batch (still under the 300ms budget):

$\text{batched QPS per GPU} = \frac{1000\text{ms}}{140\text{ms}} \times 8 \approx 57.1$
$\text{batched fleet} = \lceil 200 / 57.1 \rceil = 4 \text{ GPUs}$

Batching cuts the fleet from 16 to 4 GPUs, a 4x reduction, while per-request latency rises from 80ms to 140ms, still inside the 300ms p95 budget. Add one GPU for failover: 5 GPUs total versus 17 unbatched.

Trade-offs and pitfalls

Bigger batches help only until you hit a new latency ceiling or run out of GPU memory; past that you are compute-bound, not batching-bound, and no batching buys back headroom. Static batching forces every request to wait for the batch to fill; dynamic or continuous batching captures most of the savings without that wait.

What the interviewer probes next

Expect questions on how you'd handle the request that arrives just after a batch closes, how you'd autoscale for a traffic spike without over-provisioning steady state, and what happens to this math when sequence lengths vary widely.