End-to-End ML System Design Questions

Direct answer

Priority order, not a shopping list: first decouple stateless request-serving from anything stateful (sessions, in-memory caches) so you can add replicas freely, second put an autoscaler in front of that stateless tier driven by a real load signal (queue depth or p95 latency, not just CPU), and third roll the whole thing out with staged traffic shifts (a canary) gated on service-level objectives (SLOs, the target thresholds for latency/availability you commit to) so a bad change is caught on 1% of traffic instead of 100%. Everything else (caching, async processing, monitoring) supports that spine.

Structured elaboration

1. Define the targets before touching infrastructure. Pin down p95/p99 latency (the 95th/99th percentile response time), an availability target (e.g. 99.9%), and a rough cost ceiling. Without these, "scale it up" has no stopping point and no way to know if a change helped.

2. Split the compute architecture into two paths.

• Synchronous, low-latency path: stateless model-serving replicas behind a load balancer, autoscaled on request-driven metrics (queue depth, in-flight requests, or p95 latency) rather than CPU alone, since CPU can look idle while requests queue on I/O.
• Asynchronous/batch path: anything that doesn't need an immediate response (bulk scoring, precomputation) goes through a durable queue consumed by an autoscaled worker pool, so a traffic spike on the async side doesn't compete with the latency-sensitive path for the same replicas.

3. Externalize state. Move sessions, feature lookups, and any per-request context out of the model-server process into a shared cache or store. This is what actually enables horizontal scaling: if state lives in the replica's memory, you can't add a second replica without splitting user traffic by session, which reintroduces the bottleneck you're trying to remove.

4. Add a caching layer for repeat/hot queries in front of the model-serving tier, sized to the fraction of traffic that's actually repeat, not universally, since caching stale predictions for a fast-moving model can itself be a correctness bug.

5. Observability before scale, not after. Golden signals (latency, traffic, errors, saturation) with alerting tied to the SLOs from step 1, so a slow rollout is visible before users report it.

6. Progressive rollout. Canary 1% of traffic, gated on automated SLO checks, then 5%, 25%, 100%, each stage paused until the previous stage's metrics are clean. Keep a one-click rollback path at every stage; this is what actually prevents "an embarrassing amount of downtime," not the capacity math itself.

Worked example

Say the prototype was validated at low volume, and the target is 2,000,000 daily active users, each issuing an average of 5 requests/day (a planning assumption, stated explicitly so the arithmetic is reproducible).

Convert to average queries per second (QPS, requests handled per second):

Traffic isn't flat across the day; assume a peak-to-average factor of 3x (a common planning multiplier for consumer traffic, stated as an assumption here):

Now assume load testing on a single replica measured a sustainable capacity of 20 QPS at the target p95 latency (this is the kind of number you'd get from your own load test, not a vendor benchmark, and it's the pinned input driving the rest of the math):

Add headroom for one replica's worth of failover (N+1) plus a burst buffer, say 30%:

So the autoscaler's target ceiling for the synchronous serving tier is roughly 24 replicas at this projected peak, with the floor set by off-peak QPS using the same per-replica capacity figure. The point of doing this arithmetic explicitly is that it's re-runnable the moment your real load test gives you a different per-replica capacity number or your usage assumptions change.

Trade-offs & pitfalls

Over-provisioning for a peak factor you guessed wrong wastes real money every hour of every day; under-provisioning turns "millions of users" into an incident. Prefer measuring your actual peak-to-average ratio from prototype traffic over guessing, and re-derive the replica count once you have it. Stateful services (sticky sessions, in-memory model caches keyed by user) quietly block horizontal scaling even after you've "added autoscaling," so audit for hidden state before trusting the replica math. A canary only protects you if the signal it watches is fast and sensitive enough: SLO checks based on hourly aggregates won't catch a regression that matters within minutes. Finally, resist scaling complexity ahead of evidence: building a five-region, multi-tier architecture for a prototype that hasn't proven its growth curve yet is itself a way to introduce downtime, just earlier.

graph LR
  A[Client request] --> B[Load balancer]
  B --> C[Stateless model-serving replicas]
  C --> D[Shared cache / session store]
  B --> E[Async queue]
  E --> F[Autoscaled worker pool]
  C --> G[Monitoring: SLO dashboards]
  G --> H[Canary gate]
  H --> I[Traffic ramp: 1% to 100%]

Direct answer

A production ML system can be attacked at four distinct points: the training data (poisoning), the input at inference time (evasion/adversarial examples), the model itself as an asset (extraction/stealing), and the training data as private information (membership inference and model inversion). Each has a different attacker goal, a different detection signal, and a different response, so a strong answer walks through them as a checklist rather than treating "ML security" as one problem.

Structured elaboration

Cutting across all of these:

• Access control and least privilege on who can write to training data sources, who can call the inference API at what rate, and who can pull model artifacts.
• An audit trail (immutable logging of training data provenance, model versions, and API access) so a suspected attack can be investigated after the fact, not just prevented in theory.
• Red-teaming: periodically running these attacks against your own system in a controlled way is the only way to know your detections actually fire before an adversary finds out first.

Worked example

Model extraction is the attack where a simple cost argument makes the "how would you realistically detect it" question concrete. Suppose training the model cost $500 (compute plus data), and the inference API charges (or costs to serve) $0.002 per query.

An attacker needs on the order of 250,000 queries against this API before cloning the model is cheaper than the API bill (real extraction attacks in the literature often need queries in a similar order of magnitude to closely approximate a decision boundary, so this is a reasonable planning number, not just a cost-accounting exercise). If you rate-limit each API credential to 1,000 queries/day:

Rate-limiting alone doesn't stop extraction, but it stretches the attack from "a bad afternoon" to "the better part of a year," which is exactly the window that makes query-pattern anomaly detection (one credential issuing sustained, unusually diverse queries every single day for months) realistic to catch, versus a single suspicious hour of traffic that's easy to miss in the noise.

Trade-offs & pitfalls

• Treating "ML security" as only adversarial-examples research is the most common narrow answer; a senior response covers the full lifecycle (data in, model as an asset, data out) because that's how real incidents are categorized.
• Differential privacy against membership inference has a real utility cost (it's a genuine accuracy/privacy trade-off, not a free lunch), so it's usually applied selectively to the sensitive fields or user segments that need it, not blanket-applied everywhere.
• Rate-limiting and query-pattern detection can be defeated by an attacker using many accounts/IPs; the response has to layer detection (unusual traffic shape) with hard limits (cost/quota), not rely on either alone.
• Prompt injection is easy to underweight if the system predates LLM-based components; any answer touching a RAG or tool-using LLM system needs to name it explicitly, since it's currently one of the most exploited attack surfaces in production.
• Over-investing in exotic attacks (model inversion) while under-investing in basic access control and data-provenance hygiene is a common miscalibration; most real incidents trace back to the boring failure (an open write path to training data, an unthrottled API), not a sophisticated adversarial-example attack.

Approach
I would trace one request from ingress to response using distributed tracing. A trace is a single request’s timeline, broken into spans, which are timed steps like feature retrieval, preprocessing, model inference, and downstream calls. CPU can look normal because the bottleneck may be waiting on network, locks, queueing, or a slow dependency, not raw compute.

How I would isolate it

1. Start with one trace ID from a slow p99 request.
2. Check span timings for each stage: feature store, preprocessing, batching, model execution, postprocessing, and any outbound calls.
3. Compare slow traces to fast ones. I care about where the extra time accumulates and whether it is wait time or compute time.
4. Add or inspect metrics per stage, such as feature fetch latency, batch queue time, inference time, and downstream timeout rate.
5. If tracing is coarse, I would add sub-spans in the service and temporary structured logs with the trace ID to pinpoint the jump.

Worked example
If median requests are 80 ms and p99 is now 160 ms, I might find: feature retrieval 15 ms, preprocessing 10 ms, batch queue 55 ms, model execution 20 ms, downstream call 45 ms. That tells me the issue is batching or a dependency, not the model itself.

Next checks

• Feature retrieval slow only on cache misses, then look at cache hit rate and feature store latency.
• Batching spikes, then inspect queue depth and batch timeout settings.
• Model execution spikes, then profile the serving container.
• Downstream dependency spikes, then check retries, timeouts, and circuit breakers.

If autoscaling did not trigger, I would also verify the scaling signal. CPU-based scaling often misses latency problems, so I would prefer queue depth, request rate, or custom p95/p99 latency signals for this workload.

Direct answer

The gap that let this ship is a validation process that checked the model's outputs but never directly compared the staging and production feature pipelines against each other on the same inputs. The fix is to add an explicit parity check, a statistical test that compares the distribution of every feature as it lands in production against the distribution seen in staging (or training), gated as a hard blocker before blue traffic is ramped, not an optional dashboard someone glances at after the fact.

Structured elaboration

Where the parity check sits in the pipeline

flowchart LR
A[Training data] --> B[Preprocessing spec v1: versioned and hashed]
B --> C[Staging pipeline]
B --> D[Production pipeline]
C --> E[Feature distribution sample: staging]
D --> F[Feature distribution sample: prod]
E --> G[PSI distribution-diff test]
F --> G
G --> H{PSI within threshold}
H -->|No| I[Block blue rollout]
H -->|Yes| J[Shadow traffic on blue]
J --> K[Canary ramp with rollback gate]

1. Pipeline parity, verified, not assumed

• Preprocessing logic (scalers, encoders, tokenizers, normalization constants) has to be a single versioned artifact loaded identically by staging and production, not two independently maintained code paths that happen to be intended to match.
• Even with a shared artifact, a parity test still matters: run the same batch of real (or replayed) inputs through both environments and diff the outputs field-by-field. A silent mismatch (log1p applied in one place and log10 in another, a timezone offset in a time-based feature, a different null-fill value) shows up as a diff here even when both pipelines "look correct" individually.

2. Distribution-diff testing as an automated gate
This is the check that catches the class of bug in this scenario: nothing crashed, no schema changed, but the feature values are subtly on a different scale. Bucket each feature into bins and compare the proportion of production traffic landing in each bin against the expected (staging or training) distribution using the population stability index (PSI), a standard measure of how much a distribution has shifted:

where $a_i$ is the actual (production) proportion in bin $i$ and $e_i$ is the expected (staging) proportion. A PSI above roughly 0.2 is the common industry rule of thumb for "this is a material shift, not noise" and should block promotion.

3. Where this sits in the deployment pipeline

• Schema and contract tests (types, ranges, required fields) run first in CI, on every change, and catch structural breaks.
• The distribution-diff test runs against a production-like traffic sample before blue gets any real traffic, and again continuously once blue is in shadow mode, comparing shadow predictions and their input features against the green baseline on the same live traffic.
• Shadow mode: route a copy of real production traffic through blue without acting on its output, and compare blue's predictions and confidence distribution against green's on the same requests. A processing mismatch that changes the input distribution will usually show up as a shift in blue's output distribution too, not just its accuracy on a later-arriving label.
• Canary ramp (a few percent of real traffic) with an automatic rollback gate tied to the same distribution-diff and bias metrics, not just latency and error rate.

4. Governance around the pipeline itself

• A pre-deploy checklist with explicit sign-off from whoever owns the data/feature pipeline, separate from whoever owns the model, since this bug sits exactly at the seam between those two areas of ownership.
• An automated diff tool that flags any change to normalization constants, encoders, or tokenizer vocabulary as a reviewed, called-out change, not a side effect buried in an unrelated pull request.

Worked example

Suppose a feature (say, a scaled transaction amount) has this expected (staging/training) distribution across four bins, and this is what's actually observed in production after the scaling mismatch:

A PSI of 0.216 clears the ~0.2 "material shift" threshold, which is exactly the kind of quiet mass-shift toward the "very high" bin a scaling mismatch (for example, a log1p transform in staging versus a log10 transform in production) produces. Wired into the promotion pipeline as a hard gate, this catches the bug before blue takes real traffic, instead of after clinicians, users, or downstream consumers see biased output.

Trade-offs & pitfalls

• Schema tests alone are not enough: this bug passed every type and range check because nothing was structurally wrong, only the values were subtly rescaled. The distribution-diff test is the piece that closes that gap, and it's easy to skip because it takes real engineering effort to define good bins and thresholds per feature.
• Setting the PSI (or equivalent) threshold too loose defeats the purpose; setting it too tight creates alert fatigue and teams start ignoring it, which is its own failure mode. The threshold needs to be tuned per feature against historical natural variation, not copy-pasted as a single global number.
• Comparing distributions once at deploy time and never again misses drift that develops after a clean launch; the same test needs to run continuously as a monitoring signal, not just as a pre-deploy gate.
• Bias specifically (as opposed to a generic accuracy regression) requires checking the diff broken out by subgroup, not just in aggregate, since a shift that is invisible in the pooled distribution can be concentrated in one subgroup.
• Rollback has to be automatic and fast (traffic-weight based, not a redeploy), or the gate finding the problem doesn't actually limit the blast radius.

Direct answer

The choice of index is driven mainly by whether the embedding set fits, in a form that's still fast to search, on the memory you're willing to provision per node. For very large embedding sets, exact nearest-neighbor search is off the table on latency grounds, so the real decision is between approximate nearest-neighbor (ANN) approaches: an inverted-file-plus-product-quantization index (IVF+PQ) when memory is the binding constraint, or a graph-based index like HNSW (Hierarchical Navigable Small World) when the set fits comfortably in memory and you want the best recall-per-millisecond.

Structured elaboration

Locality-sensitive hashing (LSH). Multiple hash tables map similar vectors into the same bucket; a query hashes into the same tables and unions the candidates for exact re-ranking. It's simple and easy to shard, but needs many tables for good recall, which costs memory, and bucket sizes vary, so tail latency is unpredictable. Generally the least attractive of the three for a tight latency budget.

Inverted file plus product quantization (IVF+PQ). Vectors are clustered into coarse centroids (the inverted file); at query time, only the few nearest centroids' lists are searched. Each vector in those lists is stored as a compact quantized code (product quantization: split the vector into sub-vectors, each replaced by the id of its nearest of a small set of learned sub-centroids) instead of the full vector. This is extremely memory-efficient and the compressed-code lookup is fast and cache-friendly, at the cost of some accuracy loss from quantization, usually recovered with a final re-ranking pass against a smaller set of true vectors.

HNSW. A multi-layer proximity graph is built once; queries do a greedy graph walk from an entry point, expanding the most promising neighbors. It typically gives the best recall for a given latency of the three, and supports incremental inserts, but it stores full vectors plus per-node graph links, so its memory footprint is higher, and building the graph is comparatively expensive.

Worked example

Reason about memory, not a benchmark, for a set of 100 million embeddings at 768 dimensions stored as 32-bit floats:

That alone is already a large fraction, or more, of a typical single high-memory node's RAM, before adding a graph index's edge storage on top. With product quantization compressing each vector to, for example, 16 bytes (16 sub-vectors, each quantized to 8 bits):

That fits comfortably in memory with enormous headroom, which is exactly why IVF+PQ is the default choice once a set reaches the tens-to-hundreds-of-millions scale under a tight per-node memory budget. HNSW, needing the full 307.2 GB of vectors plus per-node graph edges (roughly another 25 GB for a moderate edge count), would need on the order of 330 GB or more, requiring either a very large single node or sharding across several, whereas the same 100 million vectors as IVF+PQ fit on one modest node with room to spare. If instead the two sets were an order of magnitude smaller (say low tens of millions), HNSW's memory cost becomes affordable on ordinary hardware and its latency/recall advantage would make it the better default.

Trade-offs & pitfalls

Product quantization's compression is not free: it introduces quantization error, so a re-ranking step against a smaller candidate set's true (uncompressed) vectors is usually needed to recover accuracy, adding one more tunable knob (how many candidates to re-rank) to the latency/recall trade-off. The number of clusters probed at query time (for IVF) or the search-beam width (for HNSW) both trade recall against latency directly, and both need re-tuning if the data distribution shifts. A common wrong turn is picking an index purely on published benchmark recall numbers without checking whether the memory footprint at YOUR scale actually fits your infrastructure; an index that's "better" in isolation is useless if it forces you into a much larger and more expensive fleet. Update frequency also matters: if the embedding sets are largely static, build cost is a one-time expense; if they update continuously, index maintenance overhead (rebuilding IVF clusters periodically, or graph-edge upkeep in HNSW) becomes an ongoing operational cost that has to be budgeted for.