End-to-End ML System Design Questions

Design
I would keep a small pool of warm replicas and use dynamic batching, an autoscaler, and request admission control. The goal is to absorb bursts without letting queueing time blow up p95 latency.

Key components

• API gateway or router that sends requests to healthy replicas
• Inference workers with a batcher that waits only a short time before running a batch
• Autoscaler that looks at queue depth, GPU utilization, and recent latency
• Warm capacity so scale-up is not from zero during traffic spikes
• Priority or deadline-aware routing for urgent requests

How I would control latency
I would set a maximum batch wait time so batching improves throughput without causing long queues. During quiet periods, I would scale down gradually but keep a floor of ready GPUs. During spikes, I would prefer a small amount of extra capacity over missing the p95 target.

Worked example
If a batcher waits up to 5 ms and batches up to 8 requests, it can improve GPU efficiency while still keeping queue delay bounded.

Tradeoff
More batching lowers cost per request, but too much batching hurts tail latency. The right answer is a controlled batch size, warm headroom, and fast scaling signals rather than aggressive scale-to-zero.

Direct answer

A feature store is a shared system that computes, versions, and serves the transformed inputs ("features") a model consumes, so the training pipeline and the live serving path read the exact same feature definitions instead of each team reimplementing them separately. Teams build one because ad hoc computation means the same transformation logic gets written twice (a batch job for training, application code for serving), and those two implementations drift apart over time with nobody noticing. It keeps training and serving consistent by having one canonical, registered transformation materialize into two stores, an offline store for historical training joins and an online store for low-latency point lookups at request time, instead of two independently maintained code paths.

Structured elaboration

Why not ad hoc:

• Duplication risk: a feature like "days since last purchase" gets implemented once in a training notebook and once in a serving service; different rounding, null handling, or time windows creep in between the two.
• No shared registry: two teams may build the same feature independently, slightly differently, with nobody able to tell the definitions have diverged.
• No lineage: without a central store, nobody can answer "which exact feature version did the model currently in production train on."

Offline vs online planes:

How consistency is achieved:

• A single registered feature transformation is executed once and fanned out to both stores, not maintained as two separate codebases.
• Point-in-time correctness: the offline store must answer "what was this feature's value at the time the training label was observed," not "what is its value now"; otherwise information from after the label leaks into training.
• Shared materialization job: one streaming or batch job writes to the online row and the offline partition at the same time, so the two stores never disagree on freshness semantics.
• Contract tests: sample a serving key and compare its online value against what the offline store would have produced at that same event time.

Worked example

Consider a feature "purchases in the last 7 days" for a user with purchases on day 1, day 3, day 9, and day 12. The training label for this user was observed on day 10. A correct, point-in-time join asks "what was this feature's value as of day 10," which counts purchases in the window [day 3, day 10): that is day 3 and day 9, a count of 2. A naive join that instead counts all of the customer's purchases up to whenever the pipeline happens to run (say day 20) would include the day-12 purchase as well, a count of 3, silently leaking a purchase that had not happened yet at label time into the training feature. This single miscounted purchase is training-serving skew in miniature: the model trains on a feature value ("3 recent purchases") that could never have existed at serving time for that label, and a feature store's point-in-time join discipline is precisely what prevents it.

Trade-offs and pitfalls

Running two stores (offline and online) is genuine operational cost, so a small team with only a handful of features and one consuming model may be better off with a well-tested shared library than a full feature store; the store earns its cost once several teams are reusing the same features and skew has already bitten someone. The most common pitfall is skipping the point-in-time join because it is harder to implement than "join on the most recent value," which quietly reintroduces skew even after a feature store is in place. A second pitfall is treating the online store's eventual consistency (a brief lag between an event happening and its feature updating) as if it were instantaneous, which matters most for features that are supposed to reflect very recent behavior.

Direct answer

Batch scoring runs predictions over a large set of inputs on a schedule and writes the results somewhere for later use; a low-latency hosted endpoint keeps a model warm and always-on behind an API to answer individual requests in milliseconds; serverless inference spins compute up on demand per request and back down when idle, trading an occasional cold-start latency hit for not paying for idle capacity. Reach for batch scoring when nobody is waiting synchronously for that specific prediction; reach for a hosted endpoint when request volume is high and steady enough to justify always-on capacity and latency must be predictable; reach for serverless when traffic is bursty or low-volume enough that paying for an idle endpoint is wasteful and occasional cold starts are acceptable.

Structured elaboration

Worked example

Suppose keeping a model warm on a small always-on instance costs $0.50 per hour, while a serverless platform charges $0.0004 per invocation, and every request genuinely needs an individual synchronous answer (batching is not an option).

At 200 requests per day:

At this volume serverless is roughly $360/2.40 \approx 150$ times cheaper, so the decision comes down entirely to whether occasional cold-start latency is tolerable. The two costs are equal at the volume $V$ where $V \times 30 \times 0.0004 = 360$:

Below roughly 30,000 requests per day, serverless is cheaper; above it, the always-on hosted endpoint becomes the cheaper and lower-tail-latency choice. Batch scoring sits outside this comparison entirely: scoring 10 million items once overnight on a shared batch cluster is typically far cheaper per prediction than serving each one synchronously through either of the other two options, precisely because nothing is paying for an always-on or per-invocation serving layer; the cost is traded for freshness, since results are only as current as the last run.

Trade-offs and pitfalls

A common wrong turn is choosing serverless for a latency-critical path without checking cold-start behavior under the platform's real concurrency model, since a burst of concurrent requests can each trigger a separate cold start rather than sharing one warm instance. Another is defaulting to a hosted endpoint out of habit for something that is genuinely a batch problem, such as a once-a-day digest score that never needs an always-on API. Batch scoring's biggest pitfall is treating it as maintenance-free simply because nothing is synchronous: a broken nightly job can silently serve yesterday's scores for days if nobody watches the job's completion status or output row count.

Direct answer

The gap that let this ship is a validation process that checked the model's outputs but never directly compared the staging and production feature pipelines against each other on the same inputs. The fix is to add an explicit parity check, a statistical test that compares the distribution of every feature as it lands in production against the distribution seen in staging (or training), gated as a hard blocker before blue traffic is ramped, not an optional dashboard someone glances at after the fact.

Structured elaboration

Where the parity check sits in the pipeline

flowchart LR
A[Training data] --> B[Preprocessing spec v1: versioned and hashed]
B --> C[Staging pipeline]
B --> D[Production pipeline]
C --> E[Feature distribution sample: staging]
D --> F[Feature distribution sample: prod]
E --> G[PSI distribution-diff test]
F --> G
G --> H{PSI within threshold}
H -->|No| I[Block blue rollout]
H -->|Yes| J[Shadow traffic on blue]
J --> K[Canary ramp with rollback gate]

1. Pipeline parity, verified, not assumed

• Preprocessing logic (scalers, encoders, tokenizers, normalization constants) has to be a single versioned artifact loaded identically by staging and production, not two independently maintained code paths that happen to be intended to match.
• Even with a shared artifact, a parity test still matters: run the same batch of real (or replayed) inputs through both environments and diff the outputs field-by-field. A silent mismatch (log1p applied in one place and log10 in another, a timezone offset in a time-based feature, a different null-fill value) shows up as a diff here even when both pipelines "look correct" individually.

2. Distribution-diff testing as an automated gate
This is the check that catches the class of bug in this scenario: nothing crashed, no schema changed, but the feature values are subtly on a different scale. Bucket each feature into bins and compare the proportion of production traffic landing in each bin against the expected (staging or training) distribution using the population stability index (PSI), a standard measure of how much a distribution has shifted:

where $a_i$ is the actual (production) proportion in bin $i$ and $e_i$ is the expected (staging) proportion. A PSI above roughly 0.2 is the common industry rule of thumb for "this is a material shift, not noise" and should block promotion.

3. Where this sits in the deployment pipeline

• Schema and contract tests (types, ranges, required fields) run first in CI, on every change, and catch structural breaks.
• The distribution-diff test runs against a production-like traffic sample before blue gets any real traffic, and again continuously once blue is in shadow mode, comparing shadow predictions and their input features against the green baseline on the same live traffic.
• Shadow mode: route a copy of real production traffic through blue without acting on its output, and compare blue's predictions and confidence distribution against green's on the same requests. A processing mismatch that changes the input distribution will usually show up as a shift in blue's output distribution too, not just its accuracy on a later-arriving label.
• Canary ramp (a few percent of real traffic) with an automatic rollback gate tied to the same distribution-diff and bias metrics, not just latency and error rate.

4. Governance around the pipeline itself

• A pre-deploy checklist with explicit sign-off from whoever owns the data/feature pipeline, separate from whoever owns the model, since this bug sits exactly at the seam between those two areas of ownership.
• An automated diff tool that flags any change to normalization constants, encoders, or tokenizer vocabulary as a reviewed, called-out change, not a side effect buried in an unrelated pull request.

Worked example

Suppose a feature (say, a scaled transaction amount) has this expected (staging/training) distribution across four bins, and this is what's actually observed in production after the scaling mismatch:

A PSI of 0.216 clears the ~0.2 "material shift" threshold, which is exactly the kind of quiet mass-shift toward the "very high" bin a scaling mismatch (for example, a log1p transform in staging versus a log10 transform in production) produces. Wired into the promotion pipeline as a hard gate, this catches the bug before blue takes real traffic, instead of after clinicians, users, or downstream consumers see biased output.

Trade-offs & pitfalls

• Schema tests alone are not enough: this bug passed every type and range check because nothing was structurally wrong, only the values were subtly rescaled. The distribution-diff test is the piece that closes that gap, and it's easy to skip because it takes real engineering effort to define good bins and thresholds per feature.
• Setting the PSI (or equivalent) threshold too loose defeats the purpose; setting it too tight creates alert fatigue and teams start ignoring it, which is its own failure mode. The threshold needs to be tuned per feature against historical natural variation, not copy-pasted as a single global number.
• Comparing distributions once at deploy time and never again misses drift that develops after a clean launch; the same test needs to run continuously as a monitoring signal, not just as a pre-deploy gate.
• Bias specifically (as opposed to a generic accuracy regression) requires checking the diff broken out by subgroup, not just in aggregate, since a shift that is invisible in the pooled distribution can be concentrated in one subgroup.
• Rollback has to be automatic and fast (traffic-weight based, not a redeploy), or the gate finding the problem doesn't actually limit the blast radius.

Direct answer

Treat the offline metric as a PREDICTION of the online result, not as ground truth to be confirmed. Build the platform so the two are compared explicitly (deterministic assignment, clean exposure logging, and a dedicated offline-versus-online agreement check), and when they diverge, check the health of the ONLINE read first (is the split clean, is exposure being captured correctly) before concluding the offline metric was simply wrong.

Structured elaboration

Platform components.

flowchart TD
  SDK[Client SDK: deterministic hash-based bucketing] --> Exposure[Exposure log]
  Exposure --> Ingest[Stream ingestion, joins exposure with events]
  Ingest --> Aggregate[Per-user daily aggregates]
  Aggregate --> LiveStats[Live stats: sample ratio mismatch check, guardrails]
  OfflineEval[Offline eval: replay against logged data] --> Compare[Offline vs online agreement check]
  LiveStats --> Compare
  Compare -->|agree| Ramp[Automated ramp]
  Compare -->|disagree| Diagnose[Diagnose: skew, exposure lag, feedback loop]
  Diagnose --> Ramp

Deterministic bucketing (a stable hash of user id plus a per-experiment salt) keeps a user's assignment consistent across app restarts and platforms. Metrics are aggregated per user per day, not per raw event, to avoid a small number of highly active users dominating the statistic. A sample ratio mismatch (SRM) check confirms the traffic split itself is clean before any metric from that split is trusted.

Why an offline metric and the live result can genuinely disagree, not just from noise:

What to do when offline and online disagree: first, gate on data quality (a clean SRM check, exposure-logging health) before trusting either number. Then segment the disagreement: is it uniform across the whole population, or concentrated in a segment with a known telemetry quirk (for example, users on a slow update cadence)? If a real effect was missed offline (a feedback loop, for example), add that as a new evaluation slice for next time rather than trusting the offline metric blindly going forward.

Worked example

A new ranking model shows a 4% offline gain on a logged holdout, but the live experiment shows a flat click-through rate (CTR) at day 7. Walk the diagnosis in order: the sample ratio mismatch check passes (the split is clean), so the disagreement is not a broken assignment. Next, check exposure capture: suppose 40% of the nominal treatment arm is on mobile clients that only receive the new model bundle on their own app-update cycle, and by day 7 only 40% of that segment has actually updated and been exposed. If the TRUE lift among users actually exposed is +5% CTR, but 60% of the nominal treatment arm hasn't been exposed yet and contributes 0% effect, the arm-level effect observed is diluted:

and with normal sampling noise at that sample size, a 2.0% true underlying effect can easily read as statistically flat at day 7. The correct read is not "the offline metric was wrong," it's "the online read is still diluted by incomplete exposure, extend the measurement window until enough of the arm has actually updated, or compute the effect only over users confirmed exposed."

Trade-offs & pitfalls

Skipping offline evaluation entirely and shipping straight to a live experiment wastes real traffic on obviously broken candidates; offline evaluation is still the right first filter, just not a sufficient one on its own. The most common wrong turn when online contradicts offline is to blame "noise" and simply run longer, without first checking a concrete, fixable confound like exposure lag or a feedback loop. Guardrail metrics catch an obviously broken rollout, but they will not catch a subtler bias where the average effect looks fine while a real harm or a real lag is concentrated in one segment; segment-level analysis is what surfaces that.