End-to-End ML System Design Questions

Direct answer

Build one system with two paths that share the same infrastructure: entities with enough history are forecast either by their own model or by a pooled model that uses their history as a strong feature, while entities with almost no history fall back to a pooled estimate derived from similar entities, and that fallback's weight is gradually replaced by the entity's own signal as its history accumulates. The mechanism that makes this smooth rather than a hard switch is statistical shrinkage toward the group estimate.

Structured elaboration

Architecture: streaming/batch ingestion → feature store (per-entity time-series features plus entity metadata/attributes) → training orchestration (scheduled, e.g. nightly) → model registry → batch and on-demand serving → drift monitoring feeding back into retraining.

Modeling approach for entities with sufficient history: at scale (tens or hundreds of thousands of entities), the practical default is one global model conditioned on entity identity or entity features (rather than literally one model per entity), so seasonal and cross-entity patterns are learned once and shared, and the registry/serving overhead stays bounded regardless of entity count.

Cold-start handling (the entity with almost no history): this is the part the base architecture above does not solve on its own, and it needs a deliberate design:

1. Use the same global/pooled model, but for a near-empty-history entity its own-history features are near-empty, so the model leans on entity metadata (category, geography, cluster assignment) instead, which a covariate-conditioned global model degrades to gracefully.
2. Assign the new entity to a cluster of similar existing entities using available attributes, and initialize its forecast from that cluster's aggregate historical pattern.
3. Blend the entity's own (short, noisy) observed average with the group/cluster average, weighting toward the group when the entity's own sample is small and toward the entity's own signal as it accumulates. This is a shrinkage estimator, and it is the mechanism that makes the transition continuous instead of a hard cutover at some arbitrary history-length threshold.

Worked example

A standard shrinkage formula blends an entity's own observed mean with the group mean, weighted by how much of the entity's own history exists:

where $n_{entity}$ is the number of periods of history observed for this entity and $k$ is a tunable prior-strength constant (chosen, for example, so that $w = 0.5$ once the entity has $k$ periods of history).

Take $k = 27$, group mean $\bar{y}_{group} = 100$, and an entity with only 3 observed periods averaging $\bar{y}_{entity} = 40$:

The forecast starts close to the group's typical behavior because there isn't yet enough of the entity's own signal to trust. Once the same entity has accumulated 30 periods with the same observed average of 40:

The forecast has moved substantially toward the entity's own (lower) observed average as evidence accumulates, without ever requiring a hard rule like "use the group model for the first 10 periods, then switch."

Trade-offs & pitfalls

Pooling too aggressively (a $k$ set too high, or a cluster assignment that's too coarse) suppresses genuinely different entities' real signal, quietly averaging away a real pattern; pooling too little defeats the purpose of borrowing strength for new entities. The constant $k$ and the cluster/grouping logic both need to be validated by backtesting against entities that DID have full history (withhold their early history and see whether the shrinkage estimate at low $n$ tracks their eventual true pattern), not chosen by intuition. Maintaining literally separate models per entity at scale creates real registry, monitoring, and serving overhead that grows linearly with entity count; most systems avoid this specifically by using one pooled/global model with entity-level features, reserving separate models only for a small number of unusually large or unusually distinct entities.

graph LR
  A[Ingestion] --> B[Feature store: history + entity attributes]
  B --> C{Sufficient history?}
  C -- yes --> D[Global model, own-history features]
  C -- no --> E[Cluster assignment + shrinkage toward group mean]
  D --> F[Forecast store]
  E --> F
  F --> G[Serving]
  G --> H[Drift monitor]
  H --> B

Direct answer

Batch scoring runs predictions over a large set of inputs on a schedule and writes the results somewhere for later use; a low-latency hosted endpoint keeps a model warm and always-on behind an API to answer individual requests in milliseconds; serverless inference spins compute up on demand per request and back down when idle, trading an occasional cold-start latency hit for not paying for idle capacity. Reach for batch scoring when nobody is waiting synchronously for that specific prediction; reach for a hosted endpoint when request volume is high and steady enough to justify always-on capacity and latency must be predictable; reach for serverless when traffic is bursty or low-volume enough that paying for an idle endpoint is wasteful and occasional cold starts are acceptable.

Structured elaboration

Worked example

Suppose keeping a model warm on a small always-on instance costs $0.50 per hour, while a serverless platform charges $0.0004 per invocation, and every request genuinely needs an individual synchronous answer (batching is not an option).

At 200 requests per day:

At this volume serverless is roughly $360/2.40 \approx 150$ times cheaper, so the decision comes down entirely to whether occasional cold-start latency is tolerable. The two costs are equal at the volume $V$ where $V \times 30 \times 0.0004 = 360$:

Below roughly 30,000 requests per day, serverless is cheaper; above it, the always-on hosted endpoint becomes the cheaper and lower-tail-latency choice. Batch scoring sits outside this comparison entirely: scoring 10 million items once overnight on a shared batch cluster is typically far cheaper per prediction than serving each one synchronously through either of the other two options, precisely because nothing is paying for an always-on or per-invocation serving layer; the cost is traded for freshness, since results are only as current as the last run.

Trade-offs and pitfalls

A common wrong turn is choosing serverless for a latency-critical path without checking cold-start behavior under the platform's real concurrency model, since a burst of concurrent requests can each trigger a separate cold start rather than sharing one warm instance. Another is defaulting to a hosted endpoint out of habit for something that is genuinely a batch problem, such as a once-a-day digest score that never needs an always-on API. Batch scoring's biggest pitfall is treating it as maintenance-free simply because nothing is synchronous: a broken nightly job can silently serve yesterday's scores for days if nobody watches the job's completion status or output row count.

Direct answer

A human needs to be in the loop at the point where the assistant's output is both hard to fully verify automatically and has real-world consequence if wrong: specifically, before any output reaches a customer or triggers an action outside the system, unless an automated risk score is confidently low. The design decision isn't "always require a human" (that doesn't scale) or "never require one" (that's how a bad draft or a wrong action ships); it's a risk-scored router that sends the low-risk majority straight through and the flagged minority to a reviewer.

Structured elaboration

Architecture

flowchart TD
A[User or workflow trigger] --> B[Retrieval: grounding documents]
B --> C[LLM drafts content or action]
C --> D[Automated safety and policy classifiers]
D --> E{Risk score}
E -->|Low risk| F[Auto-publish or execute]
E -->|High risk| G[Human review queue]
G --> H{Reviewer decision}
H -->|Approve| F
H -->|Edit needed| C
H -->|Reject| I[Discard and log]
F --> J[Audit log: version, risk score, reviewer]

Key design decisions and why

1. Retrieval grounds the draft before the LLM ever generates. Pulling relevant, authoritative source documents into the prompt reduces hallucination and gives the output something to be checked against, both by the automated classifiers and by a human reviewer, rather than a reviewer having to fact-check from scratch.
2. A risk score, not a binary rule, decides whether a human sees it. The safety/policy classifier stage scores each draft or proposed action on dimensions like policy-sensitive content, factual claims that are hard to verify automatically, and irreversibility of the action (a drafted email is reversible before sending; a completed refund or a live-published page is not). This is the design decision that makes human review scale: it's not "review everything" or "review nothing," it's "review the fraction that's actually risky."
3. Irreversibility is the single strongest trigger for mandatory human review, independent of how confident the risk classifier is otherwise. An action that can't be undone once taken (sending a customer communication, executing a payment, publishing externally) should route to a human by default even when the content looks fine, because the cost of being wrong is asymmetric: a bad automated classification on a reversible draft costs an edit; on an irreversible action it costs an incident.
4. The reviewer's decision feeds back into the loop, not just forward. An "edit needed" outcome sends the draft back through generation (with the reviewer's correction as added context), not just a one-off manual fix; this is what keeps human review from becoming a dead-end patch that never improves the underlying system.
5. Every decision, automated or human, is logged with the model version and risk score. This is what makes "who approved this and why" answerable after the fact, which matters both for a customer complaint and for tuning the risk classifier's threshold over time based on which auto-approved outputs later turned out to be problems.

Worked example

Sizing the human-review queue is a straightforward capacity calculation once you know the flag rate and reviewer throughput. Suppose the assistant generates 100,000 drafts/actions per day, and the risk classifier flags the top 5% as high-risk:

If each reviewer can process 200 items/hour over an 8-hour shift:

Four reviewers clear the daily flagged queue with same-day turnaround. This number is what turns "human in the loop" from an abstract safety principle into an actual staffing and SLA (service-level agreement) commitment: if the flag rate crept up to 15% instead of 5%, the same math needs 10 reviewers, not 4, which is exactly the kind of capacity conversation this design should force before launch, not after the queue backs up and outputs start sitting un-reviewed for days.

Trade-offs & pitfalls

• Setting the risk threshold too conservative (flagging too much) doesn't make the system safer in practice, it makes the human queue the bottleneck, and a backed-up queue creates pressure to rubber-stamp reviews quickly, which defeats the purpose; the threshold has to be tuned against real reviewer capacity, not set to the most cautious number in isolation.
• Routing purely on content risk while ignoring reversibility is a common miscalibration: a mundane-sounding draft that triggers an irreversible action (an automated refund, a message to a large distribution list) deserves review even if the classifier sees nothing alarming in the text itself.
• Treating human review as a one-way gate rather than feeding "edit needed" corrections back into generation wastes the signal reviewers are generating; over time that signal should also inform retraining or prompt refinement, not just fix one instance.
• A retrieval layer reduces hallucination but doesn't eliminate it, and it's a mistake to treat "grounded in retrieved documents" as equivalent to "verified"; the retrieved documents themselves can be stale or wrong, so the safety pipeline still needs to check the draft's claims, not just confirm it cited something.
• Audit logging is easy to under-scope to just the final output; capturing the risk score, the model version, and the specific reviewer decision (not just "approved") is what makes it possible to later diagnose whether a bad outcome was a classifier miss, a reviewer miss, or a genuinely hard edge case, and those have different fixes.

Direct answer

When the vehicle loses its connection, nothing about on-path decision-making changes: perception and control run entirely on the vehicle's own compute using the last model it had, because the design never depends on the cloud being reachable for a safety-critical decision in the first place. What connectivity actually provides is periodic model improvement, delivered as signed, validated updates the vehicle pulls down when it can, applied atomically so a partial or corrupted download can never leave the vehicle running a broken model.

Structured elaboration

Architecture

flowchart TD
A[Sensors: camera, lidar, radar] --> B[On-vehicle perception: distilled model]
B --> C[Safety monitor: latency and confidence check]
C -->|OK| D[Planning and control]
C -->|Fail or timeout| E[Fallback safe-stop controller]
D --> F[Vehicle actuation]
E --> F
G[Cloud: full training and validation] --> H[Signed model artifact and canary rollout]
H --> I[Secure OTA to fleet]
I --> B
B --> J[Telemetry buffer, upload when connected]
J --> G

Key design decisions and why

1. On-path perception and control are entirely self-contained on the vehicle. The model running in the hot path is a distilled, quantized version of a larger cloud-trained model (compressed via knowledge distillation and quantization to fit the on-vehicle compute and latency budget), so it never makes a network call as part of a driving decision. This is the single decision that makes the "loses connection" question a non-event for safety: connectivity was never in the critical path to begin with.
2. A safety monitor sits between perception and control, checking that the perception output arrived within its time budget and above a confidence threshold. If either check fails, for any reason, sensor degradation, a corrupted model, or simply an edge case the model isn't confident on, control hands off to a deterministic fallback (a conservative, pre-verified safe-stop or minimal-risk maneuver), not to "keep going with a lower-confidence answer."
3. Model updates flow one way: cloud trains, vehicle receives, never the reverse in real time. The vehicle's job is to buffer telemetry (interesting edge cases, disengagements, near-misses) locally and upload it when connected; a slow or missing connection only delays how quickly the fleet's model improves, it never blocks how the vehicle drives right now.
4. Every update is signed, versioned, and staged before any single vehicle acts on it. A hardware root of trust (a secure enclave or TPM, a small tamper-resistant chip that stores and verifies cryptographic keys) checks the signature before an update is even loaded, and the rollout itself is staged (a canary vehicle or small fleet subset first, broader fleet only after health metrics clear).
5. Atomic swap, not in-place patch, for the deployed model. The vehicle keeps two banks (A/B), the currently active model and the newly downloaded one; the switch is atomic, and if the new model fails a post-install self-check, the vehicle instantly reverts to the previous known-good bank rather than running a half-applied update.

Worked example

The reason the on-path perception model has to be entirely local, not just "usually" local, comes straight out of the latency budget. If the on-path decision needs to complete in 50ms:

allocating, say, 15ms for sensor fusion, 20ms for the distilled perception model's inference, and 10ms for planning and control, leaves 5ms of margin, all of it spent on the vehicle's own compute. A single network round trip to even a nearby edge data center, on a good cellular connection, routinely costs tens of milliseconds and is highly variable (it can spike far higher under poor signal, exactly when the vehicle might be in an unusual or higher-risk environment). There is no way to fit a cloud round trip inside this budget reliably, which is the concrete reason the architecture puts perception and control fully on-vehicle rather than "on-vehicle when the network is fast enough": the design has to hold under the worst case, and a network-dependent worst case is not one you can bound in advance.

Trade-offs & pitfalls

• The most common wrong turn is describing a hybrid architecture where the vehicle "falls back to local processing" when disconnected, implying the cloud is normally in the loop for driving decisions; a safety-critical real-time system can't be designed this way; the on-vehicle path has to be the only path for control, always, and cloud connectivity only affects how fast the model improves over time.
• Distillation and quantization trade some model capacity for the latency and compute budget; the mitigation is that the cloud can run a much larger, more capable model for offline validation, replay analysis, and generating better training labels for the next distilled version, even though it never runs live.
• OTA (over-the-air) rollout speed is a genuine trade-off against safety: rolling out to the full fleet quickly gets everyone the improvement sooner but increases blast radius if the validation missed something; a staged canary rollout is slower but bounds the exposure, and the right pace depends on how confident the offline and shadow validation made you.
• Telemetry buffering has a storage and bandwidth cost trade-off: buffering everything gives the richest signal for improving the model but is expensive to store and upload; most designs prioritize buffering disengagements, near-misses, and low-confidence events over routine driving, accepting that some rare useful signal from "boring" driving gets discarded.
• A corrupted or malicious update is the scenario the signed-artifact and dual-bank design specifically defends against; skipping the atomic-swap-with-rollback design in favor of an in-place patch is a common shortcut that removes the vehicle's ability to self-heal from a bad update without a physical service visit.

Direct answer

The failure modes that bite hardest are the silent ones: training-serving skew (the input features seen at serving time subtly diverge from what training saw), a routine deployment that quietly regresses model quality without tripping any infrastructure alert, and adversarial manipulation of the very signals you'd otherwise rely on to catch the first two. Catch all three the same way: compare production against a frozen reference on distribution and prediction-quality metrics, not just system health metrics like latency and error rate.

Structured elaboration

A quick map across all four areas the question names, before narrowing to three:

1. Training-serving skew (data). This is the mismatch between the feature values, or the feature-computation logic, seen during training versus at serving time, often because offline and online paths compute the "same" feature differently. Detection: parity checks that compare offline-computed and online-computed feature values for the same logged request, plus ongoing feature-distribution monitoring using something like the population stability index (PSI, a measure of how much a distribution has shifted between two samples).

2. Silent post-deploy model regression (model). A deployment can look completely healthy on latency and error rate while quietly making worse predictions, because "was this prediction good" isn't a system-health signal at all. Detection: monitor delayed business/proxy metrics (e.g. click-through rate, CTR, the fraction of impressions that receive a click) against a frozen baseline, and run shadow or canary comparisons of the new model against the previous one on the same traffic before a full rollout, not only after.

3. Data poisoning targeting the monitoring signal itself (security). If any part of the label or feedback loop is influenced by user or third-party behavior (implicit feedback like clicks feeding back into training, for instance), an attacker can poison the feedback stream, not just the raw input distribution, which is harder to catch precisely because it can look like "the model learning" rather than "the model breaking." Detection: anomaly detection on input patterns per source, rate-limiting to bound any single actor's influence, and treating the label pipeline itself as a monitored, audited component rather than an implicit ground truth.

Worked example

The population stability index (PSI) formula:

Take a simplified two-bucket example. Baseline (expected) proportions: 70% / 30%. Production (actual) proportions observed today: 55% / 45%.

Using the common rule-of-thumb bands (PSI < 0.1: no significant shift, 0.1-0.25: moderate shift worth investigating, > 0.25: major shift), a computed PSI of roughly 0.097 sits just under the "investigate" threshold, meaning this shift is worth watching closely on the next check but wouldn't page anyone on its own, given these specific numbers.

Trade-offs & pitfalls

Distribution-based skew detection doesn't catch every model regression: a model can regress with an unchanged input distribution, for example after a code refactor to feature computation that happens to preserve the marginal distribution but changes correlations. Alerting on too many signals produces fatigue; prioritize a small number of high-precision signals, each with a clear runbook, over instrumenting everything. Poisoned feedback loops are the hardest of the three to catch precisely because they resemble normal model learning rather than an obvious break, which is why the label/feedback pipeline itself needs to be a monitored, access-controlled component, not an implicit trust boundary.