End-to-End ML System Design Questions

Direct answer

The failure modes that bite hardest are the silent ones: training-serving skew (the input features seen at serving time subtly diverge from what training saw), a routine deployment that quietly regresses model quality without tripping any infrastructure alert, and adversarial manipulation of the very signals you'd otherwise rely on to catch the first two. Catch all three the same way: compare production against a frozen reference on distribution and prediction-quality metrics, not just system health metrics like latency and error rate.

Structured elaboration

A quick map across all four areas the question names, before narrowing to three:

1. Training-serving skew (data). This is the mismatch between the feature values, or the feature-computation logic, seen during training versus at serving time, often because offline and online paths compute the "same" feature differently. Detection: parity checks that compare offline-computed and online-computed feature values for the same logged request, plus ongoing feature-distribution monitoring using something like the population stability index (PSI, a measure of how much a distribution has shifted between two samples).

2. Silent post-deploy model regression (model). A deployment can look completely healthy on latency and error rate while quietly making worse predictions, because "was this prediction good" isn't a system-health signal at all. Detection: monitor delayed business/proxy metrics (e.g. click-through rate, CTR, the fraction of impressions that receive a click) against a frozen baseline, and run shadow or canary comparisons of the new model against the previous one on the same traffic before a full rollout, not only after.

3. Data poisoning targeting the monitoring signal itself (security). If any part of the label or feedback loop is influenced by user or third-party behavior (implicit feedback like clicks feeding back into training, for instance), an attacker can poison the feedback stream, not just the raw input distribution, which is harder to catch precisely because it can look like "the model learning" rather than "the model breaking." Detection: anomaly detection on input patterns per source, rate-limiting to bound any single actor's influence, and treating the label pipeline itself as a monitored, audited component rather than an implicit ground truth.

Worked example

The population stability index (PSI) formula:

Take a simplified two-bucket example. Baseline (expected) proportions: 70% / 30%. Production (actual) proportions observed today: 55% / 45%.

Using the common rule-of-thumb bands (PSI < 0.1: no significant shift, 0.1-0.25: moderate shift worth investigating, > 0.25: major shift), a computed PSI of roughly 0.097 sits just under the "investigate" threshold, meaning this shift is worth watching closely on the next check but wouldn't page anyone on its own, given these specific numbers.

Trade-offs & pitfalls

Distribution-based skew detection doesn't catch every model regression: a model can regress with an unchanged input distribution, for example after a code refactor to feature computation that happens to preserve the marginal distribution but changes correlations. Alerting on too many signals produces fatigue; prioritize a small number of high-precision signals, each with a clear runbook, over instrumenting everything. Poisoned feedback loops are the hardest of the three to catch precisely because they resemble normal model learning rather than an obvious break, which is why the label/feedback pipeline itself needs to be a monitored, access-controlled component, not an implicit trust boundary.

Direct answer

Priority order, not a shopping list: first decouple stateless request-serving from anything stateful (sessions, in-memory caches) so you can add replicas freely, second put an autoscaler in front of that stateless tier driven by a real load signal (queue depth or p95 latency, not just CPU), and third roll the whole thing out with staged traffic shifts (a canary) gated on service-level objectives (SLOs, the target thresholds for latency/availability you commit to) so a bad change is caught on 1% of traffic instead of 100%. Everything else (caching, async processing, monitoring) supports that spine.

Structured elaboration

1. Define the targets before touching infrastructure. Pin down p95/p99 latency (the 95th/99th percentile response time), an availability target (e.g. 99.9%), and a rough cost ceiling. Without these, "scale it up" has no stopping point and no way to know if a change helped.

2. Split the compute architecture into two paths.

• Synchronous, low-latency path: stateless model-serving replicas behind a load balancer, autoscaled on request-driven metrics (queue depth, in-flight requests, or p95 latency) rather than CPU alone, since CPU can look idle while requests queue on I/O.
• Asynchronous/batch path: anything that doesn't need an immediate response (bulk scoring, precomputation) goes through a durable queue consumed by an autoscaled worker pool, so a traffic spike on the async side doesn't compete with the latency-sensitive path for the same replicas.

3. Externalize state. Move sessions, feature lookups, and any per-request context out of the model-server process into a shared cache or store. This is what actually enables horizontal scaling: if state lives in the replica's memory, you can't add a second replica without splitting user traffic by session, which reintroduces the bottleneck you're trying to remove.

4. Add a caching layer for repeat/hot queries in front of the model-serving tier, sized to the fraction of traffic that's actually repeat, not universally, since caching stale predictions for a fast-moving model can itself be a correctness bug.

5. Observability before scale, not after. Golden signals (latency, traffic, errors, saturation) with alerting tied to the SLOs from step 1, so a slow rollout is visible before users report it.

6. Progressive rollout. Canary 1% of traffic, gated on automated SLO checks, then 5%, 25%, 100%, each stage paused until the previous stage's metrics are clean. Keep a one-click rollback path at every stage; this is what actually prevents "an embarrassing amount of downtime," not the capacity math itself.

Worked example

Say the prototype was validated at low volume, and the target is 2,000,000 daily active users, each issuing an average of 5 requests/day (a planning assumption, stated explicitly so the arithmetic is reproducible).

Convert to average queries per second (QPS, requests handled per second):

Traffic isn't flat across the day; assume a peak-to-average factor of 3x (a common planning multiplier for consumer traffic, stated as an assumption here):

Now assume load testing on a single replica measured a sustainable capacity of 20 QPS at the target p95 latency (this is the kind of number you'd get from your own load test, not a vendor benchmark, and it's the pinned input driving the rest of the math):

Add headroom for one replica's worth of failover (N+1) plus a burst buffer, say 30%:

So the autoscaler's target ceiling for the synchronous serving tier is roughly 24 replicas at this projected peak, with the floor set by off-peak QPS using the same per-replica capacity figure. The point of doing this arithmetic explicitly is that it's re-runnable the moment your real load test gives you a different per-replica capacity number or your usage assumptions change.

Trade-offs & pitfalls

Over-provisioning for a peak factor you guessed wrong wastes real money every hour of every day; under-provisioning turns "millions of users" into an incident. Prefer measuring your actual peak-to-average ratio from prototype traffic over guessing, and re-derive the replica count once you have it. Stateful services (sticky sessions, in-memory model caches keyed by user) quietly block horizontal scaling even after you've "added autoscaling," so audit for hidden state before trusting the replica math. A canary only protects you if the signal it watches is fast and sensitive enough: SLO checks based on hourly aggregates won't catch a regression that matters within minutes. Finally, resist scaling complexity ahead of evidence: building a five-region, multi-tier architecture for a prototype that hasn't proven its growth curve yet is itself a way to introduce downtime, just earlier.

graph LR
  A[Client request] --> B[Load balancer]
  B --> C[Stateless model-serving replicas]
  C --> D[Shared cache / session store]
  B --> E[Async queue]
  E --> F[Autoscaled worker pool]
  C --> G[Monitoring: SLO dashboards]
  G --> H[Canary gate]
  H --> I[Traffic ramp: 1% to 100%]

Direct answer

The three main ways to split a training job are data parallelism (each worker holds a full copy of the model and a different slice of the data, and workers exchange gradients to stay in sync), model or tensor parallelism (the model itself, or a single large layer, is split across workers, which exchange intermediate activations mid-computation), and pipeline parallelism (the model's layers are grouped into sequential stages across workers, which exchange only the activations at each stage boundary). Which one you reach for depends on whether the bottleneck is dataset size, model size, or both.

Structured elaboration

In practice, training very large models typically combines all three, sometimes called 3D parallelism: tensor parallelism within a machine over a fast local interconnect, pipeline parallelism across machines within a group, and data parallelism across groups, because each strategy's communication pattern fits a different tier of network speed: the fastest links carry the most frequent, tensor-parallel traffic, and the slowest cross-group links carry the least frequent, data-parallel gradient synchronization.

Worked example

Consider an 8-billion-parameter model stored at 16-bit precision (2 bytes per parameter): weights alone need roughly $8\times10^9 \times 2 = 16 \text{ GB}$. A common optimizer keeps additional per-parameter state that can add another 2 to 3x on top of the weights; at 3x, total memory needed is $16 \times 3 = 48$ GB. If each device has 24 GB of memory, this does not fit on one device at all, and no amount of extra data-parallel workers changes that, since every data-parallel worker would still need to hold the full 48 GB. Splitting the model across 2 devices (model or tensor parallelism) brings the per-device share to $48/2 = 24$ GB, right at the edge of fitting; that arithmetic, not preference, is what forces model parallelism here.

Separately, consider the communication cost that makes pure data parallelism expensive at this scale: an all-reduce of an 8-billion-parameter gradient tensor at 2 bytes each moves 16 GB of gradient data across workers every step. At a hypothetical 100 gigabit-per-second (12.5 GB/s) interconnect, transferring that volume alone takes on the order of:

$16 / 12.5 \approx 1.28 \text{ seconds of network time per step, in the naive case}$

Efficient ring all-reduce implementations move closer to $2\times(N-1)/N$ of that volume per node rather than the full amount, but the underlying point holds: gradient-synchronization cost scales with model size, not batch size, which is why pure data parallelism becomes network-bound long before it becomes compute-bound on very large models, and why the other two strategies exist at all.

Trade-offs & pitfalls

• Reaching for model parallelism when the real bottleneck is dataset throughput and the model fits comfortably in memory adds mid-layer communication overhead for no benefit, when data parallelism's simpler gradient-only synchronization would have been enough.
• Pipeline parallelism's bubble problem, workers idling while waiting on earlier or later stages, shrinks but never disappears with more micro-batches; more micro-batches reduce the idle fraction but increase per-batch overhead, so the choice is a tuning trade-off, not a solved problem.
• Data parallelism's communication cost scales with model size, so simply adding more data-parallel workers to speed up training on a very large model eventually stops helping once the job becomes network-bound.
• Combining all three strategies is powerful but adds real engineering and debugging complexity around exactly which workers communicate with which, in what order, at what precision; reaching for that combination before confirming plain data parallelism is actually insufficient is a common overengineering trap.

Direct answer

A production ML system can be attacked at four distinct points: the training data (poisoning), the input at inference time (evasion/adversarial examples), the model itself as an asset (extraction/stealing), and the training data as private information (membership inference and model inversion). Each has a different attacker goal, a different detection signal, and a different response, so a strong answer walks through them as a checklist rather than treating "ML security" as one problem.

Structured elaboration

Cutting across all of these:

• Access control and least privilege on who can write to training data sources, who can call the inference API at what rate, and who can pull model artifacts.
• An audit trail (immutable logging of training data provenance, model versions, and API access) so a suspected attack can be investigated after the fact, not just prevented in theory.
• Red-teaming: periodically running these attacks against your own system in a controlled way is the only way to know your detections actually fire before an adversary finds out first.

Worked example

Model extraction is the attack where a simple cost argument makes the "how would you realistically detect it" question concrete. Suppose training the model cost $500 (compute plus data), and the inference API charges (or costs to serve) $0.002 per query.

An attacker needs on the order of 250,000 queries against this API before cloning the model is cheaper than the API bill (real extraction attacks in the literature often need queries in a similar order of magnitude to closely approximate a decision boundary, so this is a reasonable planning number, not just a cost-accounting exercise). If you rate-limit each API credential to 1,000 queries/day:

Rate-limiting alone doesn't stop extraction, but it stretches the attack from "a bad afternoon" to "the better part of a year," which is exactly the window that makes query-pattern anomaly detection (one credential issuing sustained, unusually diverse queries every single day for months) realistic to catch, versus a single suspicious hour of traffic that's easy to miss in the noise.

Trade-offs & pitfalls

• Treating "ML security" as only adversarial-examples research is the most common narrow answer; a senior response covers the full lifecycle (data in, model as an asset, data out) because that's how real incidents are categorized.
• Differential privacy against membership inference has a real utility cost (it's a genuine accuracy/privacy trade-off, not a free lunch), so it's usually applied selectively to the sensitive fields or user segments that need it, not blanket-applied everywhere.
• Rate-limiting and query-pattern detection can be defeated by an attacker using many accounts/IPs; the response has to layer detection (unusual traffic shape) with hard limits (cost/quota), not rely on either alone.
• Prompt injection is easy to underweight if the system predates LLM-based components; any answer touching a RAG or tool-using LLM system needs to name it explicitly, since it's currently one of the most exploited attack surfaces in production.
• Over-investing in exotic attacks (model inversion) while under-investing in basic access control and data-provenance hygiene is a common miscalibration; most real incidents trace back to the boring failure (an open write path to training data, an unthrottled API), not a sophisticated adversarial-example attack.

Direct answer

A retrain-trigger policy is a small decision function that watches a handful of signals, data drift, a delayed quality or business metric, or a fixed calendar cadence, and fires a retrain only when a signal has been persistently and significantly out of bounds, not the moment it moves. That means the policy needs both a statistical bar (is this deviation large enough to be real) and a persistence bar (has it stayed that way long enough to not be noise) before it triggers anything, plus a floor cadence so the model never goes too stale even if every drift signal stays quiet.

Structured elaboration

Candidate trigger signals:

Debouncing mechanisms, so a signal blip does not fire a retrain:

• Statistical significance or minimum sample size: compare values with a test that accounts for how much data backs the comparison, since a tiny sample naturally swings more; this stops random noise on a low-traffic day from tripping the trigger.
• Consecutive-window requirement (hysteresis): require the signal to breach its threshold across several evaluation windows in a row, not once; this is what actually separates a blip from a trend.
• Cooldown period: once a retrain has fired, suppress the same trigger from firing again for a fixed minimum interval, since a fresh retrain needs time to take effect and be evaluated before deciding whether it worked.
• Magnitude floor: require the deviation to clear a minimum practically meaningful bar, not just a statistical one, since with enough traffic almost any tiny difference becomes statistically significant.

Composing the signals: use the scheduled cadence as a backstop; treat data drift alone as a low-trust signal that tightens monitoring (for example, shortens the evaluation window) rather than triggering a retrain by itself; reserve an automatic retrain trigger for a persistent, significant move in the delayed quality metric or business KPI; and let a severe data-drift signal alone escalate to a human decision rather than an automatic retrain, since drift without a confirmed quality impact might be a real, permanent shift the model should adapt to, or a one-off event a retrain would simply overfit to.

flowchart LR
  S[Drift and quality signals] --> D{Breached for N consecutive windows?}
  D -->|No| S
  D -->|Yes| C{Cooldown active?}
  C -->|Yes| S
  C -->|No| RT[Trigger retrain]
  RT --> V[Offline validation gates]
  V --> RO[Canary rollout]

Worked example

Suppose a trigger monitors weekly PSI on a key feature with a threshold of $0.10$, and daily values across a week, Monday through Sunday, are: $0.06, 0.12, 0.05, 0.11, 0.13, 0.14, 0.15$. A naive policy of "retrain the moment PSI exceeds 0.10" fires on Tuesday alone, a single-day spike that drops back below threshold on Wednesday. Adding a 3-consecutive-day hysteresis rule changes the outcome: Tuesday's breach does not count toward a streak because Wednesday falls back under threshold, breaking it; Thursday begins a new streak (Thursday $0.11$, Friday $0.13$, Saturday $0.14$), and the policy correctly fires on Saturday, the third consecutive day above threshold, treating Tuesday's spike as noise and the Thursday-onward run as a real, persistent shift.

Trade-offs and pitfalls

A too-sensitive policy causes retrain thrashing: burning compute and, worse, repeatedly resetting the model onto small, temporary populations, which can make it chase noise instead of tracking real drift. A too-conservative policy, long consecutive-window requirements or wide magnitude floors, means the model quietly serves stale weights through a real, sustained shift for longer than necessary. A common wrong turn is using data drift alone as the retrain trigger, since a distribution can shift with no effect on accuracy, or shift and matter greatly, and only the delayed quality signal actually tells those two apart. Scheduled retraining as the only mechanism is safe but wasteful when nothing has drifted, or dangerous when a real regression sits live while the policy waits for the calendar, which is why a floor cadence should be combined with, not replaced by, a faster, debounced signal-driven trigger. A triggered retrain should still be auto-kicked-off, not auto-promoted: the resulting model passes through the same offline validation and canary rollout gates as any other new version before it ever reaches full traffic.