System Design Methodology and Trade-off Analysis Questions

Direct answer

A genuinely non-negotiable constraint (a legal, regulatory, or security requirement with no waiver path) changes trade-off analysis from optimizing across all options to first pruning the option set down to only what's compliant, and only then optimizing cost, performance, or time-to-market among what's left. It doesn't get a weight in a scoring matrix alongside other factors; it eliminates options before scoring starts.

Structured elaboration

Treat a hard constraint as a filter applied in a distinct first pass, before any cost or performance comparison: list every candidate architecture, remove any that violate the constraint outright (not "weight them lower", remove them), and only run the normal trade-off analysis (cost, latency, time-to-market) across what survives. This ordering matters because scoring an already-infeasible option wastes analysis effort and can create a false sense that it was seriously considered.

Two realistic examples of constraints that eliminate options outright, not just penalize them:

PCI-DSS (Payment Card Industry Data Security Standard) card-data scope. If a design stores raw card numbers to power broader analytics, that option is gone the moment PCI-DSS applies, regardless of how much better the analytics would be; the only surviving options tokenize card data (replace the real card number with a random, non-sensitive placeholder token that maps back to it only inside the certified payment vault) or route it through an already-certified payment gateway.

Regulatory data residency. A requirement that a jurisdiction's data (for example, European Union customer data under data-protection law) must remain within that jurisdiction's borders eliminates any single-region deployment outside it outright, even if that region is meaningfully cheaper or already has spare capacity; there's no scoring adjustment that makes a non-compliant region viable.

Worked example

An illustrative scenario: a new payments feature needs to store transaction detail for both fraud analytics and customer support. Three candidate designs exist: (A) store full raw card data plus transaction detail for maximum analytics flexibility, (B) tokenize card data and store only tokens plus transaction metadata, (C) tokenize card data and additionally keep only aggregated, non-identifying analytics rather than per-transaction detail. Once PCI-DSS scope is applied as a hard filter, option A is eliminated outright, not down-weighted, because storing raw card data outside a certified, PCI-scoped environment isn't a slower or costlier version of the same design, it's a design that isn't legally available. The remaining trade-off analysis, cost and analytics fidelity, runs only between B and C: B keeps more per-transaction detail at a higher tokenization and storage cost (illustratively, storing a token plus full transaction metadata for 10 million transactions/month at roughly $0.0004/record runs about $4,000/month), C is cheaper (aggregating to per-customer monthly summaries cuts that record volume by roughly 95%, to around $200/month) but sacrifices per-transaction granularity for fraud analysis. That second-stage comparison is where a normal cost-vs-capability trade-off analysis applies; the first stage had none, only elimination.

Trade-offs & pitfalls

• The most common mistake is treating a hard constraint as one more weighted factor in a scoring matrix; that understates it and risks a stakeholder pushing back with "can we just accept a bit more risk here," when the honest answer is there's no risk-acceptance path available.
• Document what was eliminated and why, not just what was chosen; a stakeholder who wasn't in the room needs to see that the more attractive option was never actually on the table, not that it lost a close call.
• Distinguish a genuinely non-negotiable constraint from a strongly-preferred one; treating a soft preference as a hard filter needlessly shrinks the option set and can be walked back once challenged, which undermines trust in the rest of the analysis.
• Residual risk still needs to be documented and mitigated even after the hard filter is applied; "compliant" doesn't mean "risk-free," it means the specific eliminated risk is off the table.

Direct answer

Size against the throughput a single GPU can sustain while still meeting your latency ceiling, not its theoretical peak throughput, because batching trades latency for throughput and both constraints have to hold at once. Divide the required load by that per-GPU number, round up, and add headroom for failover and variance.

Structured elaboration

• Batching groups concurrent requests so GPU compute is shared across them, raising throughput per GPU but adding queuing and shared-compute wait time to each request.
• Larger batch sizes keep raising throughput per GPU only up to a point; eventually you hit the p95 latency budget or the GPU's memory limit (the KV-cache, per-request memory the model keeps around during generation so it doesn't have to recompute earlier tokens on every step, grows with both batch size and sequence length).
• Fleet size = required QPS divided by per-GPU sustainable QPS at your latency ceiling, plus N+1 redundancy; steady-state size and autoscaling headroom are two different numbers.

Worked example

Assume profiling shows batch size 1 takes 80ms per request (p95 latency, the 95th-percentile response time), well under the 300ms budget:

$\text{unbatched QPS per GPU} = \frac{1000\text{ms}}{80\text{ms}} = 12.5$
$\text{unbatched fleet} = \lceil 200 / 12.5 \rceil = 16 \text{ GPUs}$

Now assume batch size 8 takes 140ms per batch (still under the 300ms budget):

$\text{batched QPS per GPU} = \frac{1000\text{ms}}{140\text{ms}} \times 8 \approx 57.1$
$\text{batched fleet} = \lceil 200 / 57.1 \rceil = 4 \text{ GPUs}$

Batching cuts the fleet from 16 to 4 GPUs, a 4x reduction, while per-request latency rises from 80ms to 140ms, still inside the 300ms p95 budget. Add one GPU for failover: 5 GPUs total versus 17 unbatched.

Trade-offs and pitfalls

Bigger batches help only until you hit a new latency ceiling or run out of GPU memory; past that you are compute-bound, not batching-bound, and no batching buys back headroom. Static batching forces every request to wait for the batch to fill; dynamic or continuous batching captures most of the savings without that wait.

What the interviewer probes next

Expect questions on how you'd handle the request that arrives just after a batch closes, how you'd autoscale for a traffic spike without over-provisioning steady state, and what happens to this math when sequence lengths vary widely.

Direct answer

A cascade, a cheap filter first and an expensive model only on the fraction it is unsure about, wins whenever the input mix is skewed toward easy cases, because it lets you pay the expensive model's cost only on the hard tail. A single large model wins when you cannot build a cheap filter with high enough recall on the escalation decision itself, since a weak filter silently drops cases that needed the expensive model's judgment.

Structured elaboration

• Cascade design: a cheap stage-1 classifier resolves confident cases directly; anything below a confidence threshold is escalated to the expensive stage-2 model.
• What matters most: not the cascade's average accuracy, but the stage-1 filter's recall on "this needs escalation," a miss there means the expensive model never sees a case that needed it.
• Complexity cost: two models to version and monitor for drift, and request latency now varies by path length, some requests take one hop, some take two.
• Single model: simpler to operate, but every request pays the expensive model's full cost, even the easy 90%.

Worked example

10 million posts/day, cheap filter $0.0001/post, expensive model $0.01/post, and the filter confidently resolves 90% while escalating the other 10%:

$\text{cascade cost} = 10{,}000{,}000 \times \$0.0001 + 1{,}000{,}000 \times \$0.01 = \$1{,}000 + \$10{,}000 = \$11{,}000/\text{day}$
$\text{all-expensive cost} = 10{,}000{,}000 \times \$0.01 = \$100{,}000/\text{day}$

That is roughly 89% cheaper. But if the filter's recall on "needs escalation" is only 95% against a true escalation population of 1,000,000:

$\text{missed escalations} = 1{,}000{,}000 \times (1 - 0.95) = 50{,}000/\text{day}$

50,000 posts a day get moderated by the wrong tier, a real cost that has to be weighed against the $89,000/day saved.

Trade-offs and pitfalls

Cascades only make sense if you can actually measure the filter's recall on the escalation decision, not just its overall resolve rate. Latency variance (a two-hop tail versus a one-hop median) can also blow past a p99 budget if not accounted for.

What the interviewer probes next

Expect follow-ups on how you would set and validate the confidence threshold, how to monitor drift when the two models retrain on different schedules, and how adversarial inputs crafted to slip past the cheap filter change the design.

Direct answer

Approximate nearest-neighbor (ANN) indexes trade exact recall for speed and memory by avoiding a full comparison against every vector. The tuning knobs let you buy back recall by spending more query latency and memory, so the right setting is whichever point on that curve meets your product's minimum acceptable recall at the lowest cost.

Structured elaboration

Exact (brute-force) search gives 100% recall but scales linearly with corpus size, too slow at 50 million vectors for interactive search. Graph-based ANN indexes like HNSW (Hierarchical Navigable Small World: a graph structure where each stored vector links to a handful of its nearest neighbors, so a query can hop from node to node toward the target region instead of comparing against every vector) approximate the search, with a search-width parameter controlling how much of the graph is explored per query, raising both recall and latency together as it increases. Index memory scales with vector count and dimensionality, since it holds both the vectors and the graph edges.

Worked example

768-dimensional embeddings stored as 32-bit floats:

$\text{bytes per vector} = 768 \times 4 = 3{,}072 \text{ bytes} \approx 3\text{KB}$
$\text{raw vector memory} = 50{,}000{,}000 \times 3\text{KB} \approx 143\text{GB}$

Assuming a graph-based index adds 30% overhead for its structure:

$\text{index memory} \approx 143\text{GB} \times 1.3 \approx 186\text{GB}$

a real decision between one high-memory machine and a sharded cluster. If a lower-recall setting lets you quantize vectors to 8-bit integers instead of 32-bit floats:

$\text{raw vector memory}_{\text{int8}} = \frac{143\text{GB}}{4} \approx 36\text{GB}$

roughly a 4x memory reduction, at the cost of some recall loss on top of whatever the ANN approximation already gave up.

The recall-versus-latency side of the trade-off is just as traceable with illustrative figures (not measured against a real dataset here, since the actual curve depends on corpus and hardware, but self-consistent so the shape is real): if each additional graph node the search visits costs a fixed per-candidate distance computation, say roughly 0.02ms, then a low search-width setting of 64 nodes costs about $64 \times 0.02\text{ms} \approx 1.3\text{ms}$ and typically lands recall in the low 90s (~90%); a higher setting of 256 nodes costs about $256 \times 0.02\text{ms} \approx 5.1\text{ms}$ and typically reaches the high 90s (~98%); pushing further to 512 nodes roughly doubles the cost again to $512 \times 0.02\text{ms} \approx 10.2\text{ms}$ for only a small further recall gain (~99.3%), the same flattening-curve pattern discussed below.

Trade-offs and pitfalls

Chasing a small recall gain near the top of the curve, say 97% to 99%, often costs a disproportionate amount of latency and memory, because ANN recall curves flatten sharply. Measure recall against a labeled ground-truth set rather than eyeballing results, and re-measure after any re-embedding, since a new embedding model changes what "correct" nearest neighbors even are.

What the interviewer probes next

Expect questions on how you'd build a ground-truth recall evaluation set, sharding once the index outgrows one machine's memory, and how you'd handle index updates when embeddings change incrementally rather than in one batch.

Direct answer

Use synchronous scoring only when the action on the score must happen before the transaction completes, such as blocking a fraudulent charge. If the action can wait (nightly review, retraining labels), batch scoring buys a heavier, more accurate model at a fraction of the cost. The deciding question: what does a delayed decision cost you, versus an always-on low-latency fleet?

Structured elaboration

Real-time: one hop in a hard end-to-end latency budget, needs near-zero-staleness features (a hot feature store: a system that serves the same precomputed input values, like a rolling transaction count, to the model at scoring time; "hot" means it's updated in near real time rather than nightly), and an always-on fleet sized for peak plus a fallback if the call times out.

Batch: runs on a schedule, so it can use a bigger, slower model on cheaper bursty compute, but adds a detection lag equal to the batch interval, and a partial failure is a silent under-score, not an outage.

Worked example

A 250ms authorize/decline budget, with auth, ledger, and notification already at 180ms:

$\text{fraud model budget} = 250\text{ms} - 180\text{ms} = 70\text{ms}$

Subtract 20ms network/serialization overhead:

$\text{compute budget} = 70\text{ms} - 20\text{ms} = 50\text{ms}$

That rules out heavy ensembles (combining predictions from several models, which multiplies the per-request compute cost) needing multiple feature joins (a feature store lookup that assembles several separate precomputed values into one input record, each join adding its own latency): illustratively, a 3-model ensemble with 2 extra feature joins might cost 90-120ms on its own, already over the 50ms budget, which is what "rules out" means here concretely. For batch: 10 million transactions overnight, 500 records per inference batch, 50ms per batch on one GPU:

$\text{batches} = \frac{10{,}000{,}000}{500} = 20{,}000, \quad \text{GPU time} = 20{,}000 \times 50\text{ms} \approx 16.7\text{ min}$

Across 4 GPUs, about 4 minutes wall time, no idle fleet cost during the day.

Trade-offs and pitfalls

Real-time buys speed but pays for peak capacity around the clock and widens the request's failure surface. Batch is cheaper but the fraud can complete before you act. A common mistake is defaulting to real-time without pricing the fleet against the actual cost of delay. The answer flips when either side of that equation moves: if fraud losses from a delayed decision start to dwarf the cost of an always-on fleet, or a cheaper real-time model becomes accurate enough to fit the leftover latency budget, you're pushed toward real-time; if the model needs more compute than the authorization path can spare, or false declines against legitimate customers become the bigger cost, you're pushed back toward batch.

What the interviewer probes next

Hybrid streaming or micro-batch designs, detecting a batch job that fails partway through, and how the answer shifts if a missed fraud case gets an order of magnitude costlier.