Meta Machine Learning Engineer Interview Preparation Guide - Mid Level (2-5 Years)

Direct answer: Proving a new feature's business impact requires a real randomized experiment measuring the actual business outcome, not just an offline metric bump, and needs a deliberate plan for the specific, recurring failure mode where offline validation improves but the live experiment's business metric doesn't (or gets worse), plus an explicit ROI case made before any of the engineering work starts.

Structured elaboration:
The experiment needs randomized treatment assignment (which requires the feature computation itself to be deterministic and reproducible per treatment group, since non-deterministic feature computation can itself leak treatment assignment or contaminate variants), instrumentation via feature flags so the new feature can be toggled cleanly per arm, and guards against confounds like time-of-day or user-segment imbalance between arms (stratified or blocked randomization, checked post-hoc for balance across arms before trusting the result).

When offline validation area-under-the-curve improves but a live experiment shows a business-metric regression (a genuinely common and important pattern), the investigation needs a structured hypothesis list spanning three layers: a DATA problem (the offline evaluation set doesn't represent the live population the same way, or there's subtle leakage inflating the offline number), a MODEL problem (an offline-metric improvement that doesn't translate to a better real decision, e.g. improved ranking precision that doesn't move the metric users actually care about), or a DEPLOYMENT problem (a bug in how the feature is actually served in the live path versus how it was computed offline, which circles back to training-serving skew). Diagnostic steps: segment the live results (does the regression concentrate in one user segment or time period, suggesting a specific root cause); check the feature's live distribution against its offline-validated distribution (a mismatch here points at a data or deployment issue, not a modeling one); and, if needed, roll back to isolate whether the new feature specifically, versus something else that shipped concurrently, caused the regression.

Determinism and treatment isolation: the feature computation for a given entity must produce the identical value regardless of which experiment arm is evaluating it at that instant (no shared mutable cache keyed only on entity ID that a control-arm read could accidentally populate with a treatment-arm-computed value, for example); treatment/control-specific feature computation paths should be fully isolated (separate cache namespaces or separate materialization jobs keyed by arm), and computation for a given entity-timestamp pair should be idempotent, so a retried or replayed computation doesn't silently drift between arms mid-experiment.

Building the ROI case before doing the work: a simple, explicit framework to reason about whether the engineering investment is worth making:

ROI = (Expected business-metric lift x affected traffic volume x per-unit business value) / (engineering cost + ongoing serving/compute cost)

with each term explicitly estimated and its uncertainty stated: the expected lift comes from a cheap early offline check (a quick ablation showing the feature's plausible ceiling contribution, clearly labeled as an ESTIMATE, not a measurement); affected traffic volume and per-unit business value come from existing production data (a MEASURED baseline); engineering cost is a rough effort/timeline estimate for building and validating the feature; ongoing cost includes the feature's serving latency and compute/storage footprint at scale. A feature whose estimated numerator is dwarfed by its denominator (a large engineering lift for a marginal, low-traffic-affecting signal) is a candidate to deprioritize before any experiment is even designed, not after a costly build-and-test cycle confirms it wasn't worth it.

Worked example: A model with a newly engineered feature shows improved offline validation accuracy but a live A/B test shows conversion drop. Segmenting the live results reveals the regression concentrates entirely on mobile traffic; checking the feature's live distribution on mobile against its offline-validated distribution shows a meaningful mismatch, pointing to a deployment-layer bug (the feature was computed correctly offline but served incorrectly for mobile requests specifically) rather than the feature itself being a bad idea.

Trade-offs and pitfalls: The ROI estimate's expected-lift term is the weakest link in the formula, since it's necessarily based on an offline proxy before any live experiment has run; treating that number as a confident projection rather than a rough, clearly-labeled estimate is exactly the kind of overclaiming that erodes trust when the eventual live result differs from the pre-experiment projection.

Direct answer: A resource-management model for a shared feature platform needs namespace-level quotas that are enforced (not advisory), autoscaling boundaries so no single tenant's burst starves others, priority scheduling so critical production workloads pre-empt best-effort ones, and cost attribution that ties usage back to teams to keep the fair-share system honest over time.

Structured elaboration:

• Namespace quotas. Each team's namespace gets hard caps on storage, compute-seconds for materialization jobs, and online-store operation rate; quotas are enforced at admission (a job that would exceed quota is queued or rejected, not silently allowed to run and degrade shared infrastructure).
• Autoscaling boundaries. The platform autoscales shared compute (materialization workers, online-store shards) within a budget, but per-tenant autoscaling has a ceiling, so one team's traffic spike triggers platform-wide scale-up only up to a bound, beyond which that team's own requests get throttled rather than the platform scaling unboundedly on their behalf.
• Priority scheduling. Classify workloads (production-serving-critical, scheduled-batch, best-effort/experimental) and give the scheduler a priority queue so a production materialization job pre-empts a lower-priority ad-hoc backfill during resource contention, similar to how a Kubernetes cluster uses priority classes and pod preemption.
• Cost attribution and fair-share. Track actual resource consumption per tenant continuously, and use it both for chargeback reporting and as an input to the fair-share algorithm itself (a tenant that has consumed more than its fair share recently gets lower scheduling priority for new work until the moving average rebalances), which is the mechanism that prevents a single well-resourced team from perpetually crowding out smaller teams even within their nominal quota.
• Noisy-neighbor prevention. Beyond quotas, isolate the blast radius of one tenant's misbehaving job (a runaway query, a job stuck in a retry loop) using resource limits (CPU/memory caps, circuit breakers on repeated failures) so it cannot degrade the shared control plane or other tenants' jobs even transiently.

Worked example: Using a priority-class scheme with three tiers (critical production, scheduled batch, best-effort), if the platform is at 90% compute utilization and a critical production materialization job arrives, the scheduler pre-empts a running best-effort job (checkpointing its progress if possible, or simply killing and requeuing it) to make room, rather than queuing the critical job behind lower-priority work; this mirrors how many organizations run Spark or Kubernetes scheduling in shared clusters today.

Trade-offs & pitfalls: A fair-share algorithm based purely on historical consumption can create a "rich get richer, poor stay poor" dynamic if not careful: a team that legitimately needs to scale up (a new product launch) gets penalized by the fair-share mechanism just when it needs more resources, so the model needs an explicit override or request path for legitimate, approved growth rather than treating all consumption spikes identically. Priority pre-emption without checkpointing support means pre-empted jobs restart from scratch, which can waste more resources than it saves if pre-emption happens frequently; investing in checkpoint-and-resume for long-running materialization jobs pays for itself once pre-emption becomes common at scale.

Direct answer

Justifying a simpler, interpretable model against a stakeholder fixated on maximum accuracy means reframing "accuracy" as only one of several things the business actually needs, and showing concretely what the more accurate model would cost on the other dimensions that matter.

Structured elaboration

1. Name the full set of things that matter, not just accuracy. Interpretability (can you explain individual decisions), maintainability (can the team actually keep this running and updated), and regulatory risk (does a black-box model expose the business to compliance or legal risk) are all real costs, not soft preferences.
2. Quantify the accuracy gap's actual value. As with other trade-offs, translate the accuracy difference into a business-impact number so it's comparable to the other three dimensions, rather than treating "more accurate" as an unquestionable win by default.
3. Quantify the regulatory and interpretability risk concretely. What's the realistic cost if a regulator or auditor challenges a decision the black-box model can't adequately explain? This is often the single most persuasive point for a stakeholder focused purely on model performance.
4. Propose the simpler model with a credible path to close the gap. Rather than presenting simplicity as a permanent ceiling, show what would need to be true (more mature tooling, a regulatory change, proven business value justifying the added risk) before the more complex model becomes the right choice.

Worked example

In a credit-decisioning context, the interpretable model isn't just "safer" in the abstract, it directly avoids a specific, quantifiable regulatory exposure (the requirement to provide an adverse-action reason a black-box model can't reliably produce), which is a concrete cost the accuracy-focused stakeholder likely hadn't fully priced in when framing the decision purely as an accuracy contest.

Trade-offs and pitfalls

The common mistake is arguing for simplicity on principle rather than with a concrete cost tied to the specific business and regulatory context; a stakeholder fixated on accuracy will reasonably discount a vague appeal to "best practice." The other risk is overstating the regulatory or maintainability risk to win the argument, which can backfire if it's later shown to be exaggerated.

Direct answer: HNSW (Hierarchical Navigable Small World graphs) gives logarithmic-ish query time ($O(\log N)$ empirically, via greedy graph traversal across hierarchical layers) with strong recall but higher memory and index-build cost; IVF+PQ (inverted file index with product quantization) compresses vectors aggressively for much lower memory at the cost of recall, using a coarse clustering to restrict search to a subset of the data; LSH (locality-sensitive hashing) gives probabilistic sub-linear query time with weaker recall guarantees than either, but simpler theoretical underpinnings and easy distributed/sharded implementation. All trade exactness for speed and memory versus brute-force O(N x d) linear scan.

Structured elaboration

• HNSW: builds a multi-layer graph where higher layers have progressively fewer nodes (a "skip list"-like structure over the vector space), allowing search to start at a sparse top layer and greedily navigate toward the query's nearest neighbors, refining through denser lower layers. Query time is empirically close to logarithmic in N; build time is superlinear (each insertion involves graph-edge construction, roughly $O(N \log N)$ overall) and memory overhead is significant (storing the multi-layer graph structure, not just the raw vectors) - HNSW typically gives the BEST recall-per-query-latency trade-off among common ANN methods, at the highest memory cost.
• IVF+PQ: first clusters the dataset into coarse "cells" via k-means (the Inverted File index); a query only searches within the few nearest cells to its own location, avoiding the need to compare against the full dataset. Product Quantization further compresses each vector by splitting it into sub-vectors and quantizing each sub-vector to a small codebook, shrinking memory dramatically (often 10-30x smaller than storing raw float vectors) at the cost of the quantization introducing approximation error that reduces recall.
• LSH: hashes vectors such that similar vectors are more likely to collide into the same hash bucket than dissimilar ones (the opposite goal of a normal, collision-avoiding hash function); a query hashes and only checks the colliding bucket(s). Simpler to reason about and to shard/distribute (each hash table can live on a different machine), but typically needs multiple hash tables to achieve competitive recall, and its probabilistic guarantees are looser than HNSW's empirical graph-navigation quality.

Worked example

For 100 million 768-dimensional embeddings (a realistic large-scale semantic search setting): raw storage at float32 is $100\text{M} \times 768 \times 4\text{B} \approx 307\text{GB}$. IVF+PQ compressing each vector to, say, 64 bytes (a common PQ configuration) shrinks storage to $100\text{M} \times 64\text{B} = 6.4\text{GB}$ - roughly 48x smaller, making the difference between needing a multi-machine memory-resident index and fitting comfortably on a single large-memory machine. HNSW's graph overhead instead ADDS to the raw vector storage (typically an additional 30-100+ bytes per vector for the graph edges across layers, so roughly 100M x (3072+~60) bytes ≈ 313GB total) - HNSW does not compress the base vectors, so its memory footprint stays close to (or above) the raw storage size, trading memory for its superior recall/latency at query time.

Trade-offs & pitfalls

• Recall (fraction of TRUE nearest neighbors actually returned) versus latency versus memory is a three-way trade across all these methods - there's no universally "best" choice; the right one depends on which axis your system's constraints bind on hardest.
• Build/update cost matters for dynamic datasets: HNSW graph updates (inserting new vectors) are more expensive than IVF+PQ's (which can often just assign a new vector to its nearest existing cluster cheaply), a real consideration if the embedding collection changes frequently rather than being built once and queried many times.
• Hybrid approaches (e.g. IVF+HNSW combinations, or IVF+PQ with a re-ranking pass using exact distances on a small candidate set) are common in production to balance these trade-offs rather than committing to one pure method.

Situation: At my previous company I led an ML effort to replace a rules-based fraud filter with a deep-learning classifier. After three full experimental cycles (different architectures, feature sets, and data-augmentation strategies) the model repeatedly missed the production success criteria: precision at target recall stayed below 72% vs required 85%, and false positives in a live shadow run increased operational load.

Task: I had to decide whether to continue iterating on the model or pivot strategy to meet business SLAs without burning more time.

Action:

• I ran a rapid root-cause analysis: error analysis on false positives, data drift checks, ablation studies, and consulted ops about label quality. That showed two issues: noisy labels in a class of edge transactions and that real-time latency constraints prevented using richer context features.
• I created a decision checklist: estimated marginal gain from further model iteration (low, based on diminishing returns), cost of more data-labeling (high), time-to-value, and business risk.
• Based on that, I recommended a pivot: instead of a single offline DL model, we would adopt a hybrid approach—keep the existing rules for high-risk cases, add a lightweight gradient-boosted model for real-time scoring, and schedule a longer-term data-quality initiative to enable DL later.
• I communicated the pivot to stakeholders via a one-page decision memo and a 30-minute cross-functional meeting showing evidence (charts from error analysis, estimated impact), the alternative plan, and rollback criteria.
• To help the team adopt the new approach I:
  • Broke work into short sprints: deliver a GBDT prototype, integrate with the serving stack, and run an A/B test.
  • Paired ML engineers with data engineers to fix label pipelines and added monitoring dashboards for precision/recall and latency.
  • Ran a workshop demonstrating the prototype and the rationale, and updated the roadmap so everyone saw how the pivot led back to the DL goal once data was clean.

Result: Within four weeks the hybrid system met the precision target (86%) and reduced false positives by 30%, restoring stakeholder confidence. The data-quality work completed over the next quarter enabled retraining a DL model with reliable labels; once deployed it improved detection by another 6% without violating latency constraints.

This taught me to let evidence—error analysis, cost/benefit, and operational constraints—drive the iterate vs pivot decision, and that clear, data-backed communication plus incremental deliverables makes pivots adoptable and low-risk.

Direct answer
For a fixed model class, adding more training data primarily reduces variance; it does little to nothing for bias, because bias comes from the model's inherent inability to represent the true relationship, not from noisy parameter estimates.

Structured elaboration
Variance comes from the model's parameters (or splits, or weights) being estimated from a finite, noisy sample; as the sample grows, the law of large numbers pulls the estimated parameters toward their true expected values, so different training sets produce increasingly similar fitted models. That's a direct reduction in variance. Bias, by contrast, is baked into the model's functional form: a linear model fit to a genuinely curved relationship is wrong in the same systematic way no matter how much data you feed it, because a straight line simply cannot represent a curve.
When more data does NOT help: if you're in a genuinely high-bias regime (a linear model on a strongly non-linear target, or a shallow tree on data with deep interactions), the learning curve will show training and validation error converging to a high, roughly-equal error floor. Adding another 10x rows to the training set will barely move that floor. The signal to look for is exactly this: if validation error has already flattened out and closely tracks training error, more data buys you very little and the money is better spent on a more expressive model or better features.

Worked example
Fit a linear model to data generated from $y = \sin(3x) + \epsilon$. At n=50, 500, and 5000 training points, plot training and validation MSE. Training error stays roughly flat and validation error converges to nearly the same (high) value by n=500, since the line can never represent a sine wave; further data does not shrink either curve.

Trade-offs & pitfalls
Diagnosing "more data won't help" purely from a learning curve requires that the curve has actually had a chance to plateau; a curve that is still steadily declining at your current sample size may just need one more order of magnitude, not a model change. Don't conclude bias-limited too early from a curve that hasn't converged yet.

Direct answer

After a product launch I owned came in well short of its adoption target, I ran a structured review with the two other people closest to the work, and it surfaced something I genuinely hadn't seen while we were building: the target itself had been set based on a comparable launch that wasn't actually comparable, which meant part of the shortfall was a bad target, not just execution. What changed because of it was both a fix to the immediate rollout and a change to how I set targets for anything similar going forward.

How I structured the review

I scheduled it two weeks after launch, once we had a real signal instead of just launch-week noise, and kept it to the two people who had been closest to the build and rollout decisions. Before the meeting, I pulled the original planning document, the actual usage data, and a timeline of the decisions we'd made along the way, so the conversation could be grounded in what we'd actually said and done rather than what we remembered saying. I ran it around a small set of questions: what did we expect and why, where did the plan and reality diverge, and what would we have needed to know earlier to catch it.

What it surfaced

Working through the timeline, we found that the adoption target had been benchmarked against a previous launch in a different user segment with meaningfully different existing habits, something none of us had flagged while setting the target because the comparison felt intuitively reasonable at the time. We also found a smaller, genuinely execution-related gap: onboarding for the new feature was buried two screens deep, which the usage data showed was where a real chunk of users dropped off, something none of us had noticed during testing because we already knew where to find it.

What changed

The onboarding placement was fixed within a week, and usage on that path improved measurably, though I won't claim a precise before-and-after number beyond that it was a clear, visible shift in the funnel. The more durable change was to how I set targets afterward: I now require an explicit note on any benchmark comparison stating what's actually different about the comparison case, rather than letting a comparison stand just because it feels close enough. I tracked that change by checking, on the next two launches, whether that note existed before the target was finalized, rather than just trusting that I'd remember to do it.

Trade-offs and pitfalls

The hindsight in a review like this is easy to feel foolish about, since the flawed benchmark seemed obviously reasonable in the room when we set it. The pitfall is treating that hindsight as evidence the team was careless, when the more useful conclusion is usually that a specific assumption needed to be made explicit and checked, which is a fixable process gap rather than a character flaw.

Situation: I need a repeatable CI test plan that verifies a TensorFlow SavedModel -> ONNX -> ONNX Runtime roundtrip produces numerically equivalent outputs within acceptable tolerances.

Test plan (high-level steps)

1. Export & convert pipeline

• Scripted steps: (a) export SavedModel, (b) convert with tf2onnx/onnx-tf, (c) run ONNX Runtime inference. Fix RNG seeds and TF/ONNX runtime versions.

2. Test data selection

• Unit tests: small hand-crafted vectors that exercise edge cases (zeros, ones, large/small magnitudes, negative, inf/nan).
• Functional tests: random inputs with fixed seeds across distributions (uniform, normal, skewed).
• Coverage tests: inputs that trigger different ops, dynamic shapes, batch sizes, and quantized/dtype variants.
• Real-data smoke test: 50–200 real samples from production-ish dataset.

3. Tolerance rules & metrics

• Per-output checks:
  • Exact equality for integer outputs.
  • For floating types, use combined metrics:
    • max_abs = max(|y_tf - y_onnx|)
    • rms = sqrt(mean((y_tf - y_onnx)^2))
    • cosine_sim for embeddings/vectors.
  • Default thresholds (float32): rtol=1e-5, atol=1e-6; practical thresholds: max_abs < 1e-4 or rms < 1e-6. For fp16 or quantized: relax (rtol=1e-2, atol=1e-3).
• Special checks: NaN/Inf parity (fail if TF has finite and ONNX has NaN/Inf or vice versa).
• Relative per-output scaling: normalize by max(|y_tf|, epsilon) when outputs span orders of magnitude.

4. Pass/fail rules

• Per-test: pass if metrics under thresholds and NaN/Inf parity holds.
• Aggregate: allow up to a tiny percentage (e.g., 1–2%) of samples to exceed soft thresholds for flaky ops; failing tests trigger investigation.

5. Automation & CI hooks

• Integrate into CI pipeline (GitHub Actions / Jenkins):
  • Matrix run across runtime versions, hardware (CPU/GPU), and dtypes.
  • Store artifacts: SavedModel, ONNX model, test inputs/outputs, diff reports, and serialized failure cases.
  • Auto-generate human-readable report with metric summaries and example failing cases (show inputs, TF vs ONNX outputs, diffs).
  • Alerting: fail the PR on hard failures; for soft failures, open a ticket with attached artifacts.
  • Regression baselines: keep golden outputs and only allow changes via approved updates.

6. Additional practices

• Add randomized fuzzing tests periodically (nightly).
• Maintain converter-version compatibility tests.
• Add model-level unit tests for deterministic ops and stochastic ops (ensure seeds or compare distributions).

This plan provides deterministic, reproducible checks, clear numeric criteria per dtype, and CI automation to catch regressions early while producing helpful artifacts for debugging.

Direct answer

The first hypothesis is that the input pipeline, not the model or the GPU, is the bottleneck: the CPU-side work that prepares each batch (decoding, augmentation, tokenization, collation) can't produce batches fast enough to keep the GPU fed, so the GPU sits idle waiting while the CPU is maxed out doing that preparation. This is different from a disk/network stall (which would show LOW CPU, workers blocked on I/O rather than pinned on compute) and different from a distributed-training straggler problem (which shows up as uneven step times across ranks, not a single job's own GPU sitting idle next to a saturated CPU).

Structured elaboration

Reading the three signals together. GPU utilization at 20% says the accelerator is idle most of the time. CPU pinned at 80% says something on the host is doing real, sustained work, not waiting. An empty data-loading queue, combined with those two, is the tell: it means the CPU-side workers are being drained by the training loop as fast as they can fill it, and still can't build up a buffer, they are running at their ceiling and that ceiling is below what the GPU can consume. An empty queue on its own can look "healthy" on a dashboard; it only means starvation once you also see the GPU idle and the producer maxed out.

Why this points at CPU-bound preprocessing specifically, and not I/O. If the stall were disk or network I/O, the CPU would typically be LOW while workers block waiting on reads, not pinned. Pinned CPU is the signature of compute-bound work happening in the data pipeline itself: image decode and resize, text tokenization, feature crossing, or other per-sample transforms that are often single-threaded Python work per worker process.

Why this is not the distributed-straggler pattern. A straggler problem is about variance ACROSS workers/ranks in a multi-node synchronous job, one rank finishing slower than the others repeatedly. It doesn't explain a single node's own accelerator sitting idle while its own CPU is maxed and its own queue is empty; that is a local producer/consumer imbalance, a different layer of the system entirely.

How to confirm it.

1. Break down step time with a framework profiler (for example the PyTorch profiler or a TensorFlow Profiler trace) into "waiting for data" versus "on-device compute." A large data-wait fraction confirms the hypothesis directly.
2. Run an isolation test: swap the real dataset for a synthetic generator producing randomly-initialized tensors of the same shape, skipping real decode/augment. If GPU utilization jumps toward its ceiling, the pipeline (not the model) was the limiting factor.
3. Check worker configuration: number of data-loader worker processes versus available CPU cores, prefetch depth, and whether workers persist across epochs. Too few workers relative to per-sample CPU cost is the most common root cause.
4. Identify the specific expensive stage (decode, augment, tokenize) rather than stopping at "the pipeline is slow"; the fix differs by stage.

Worked example

Reason through the throughput math, not a wall-clock measurement. Suppose (illustrative numbers, not measured) the GPU can consume one batch's worth of preprocessed data every 4 ms once it arrives (a throughput ceiling of 250 samples/sec-equivalent), and a single CPU worker needs 8 ms to prepare one sample-equivalent (a throughput of 125 samples/sec). For the queue to stay non-empty, the combined worker throughput must at least match the GPU's consumption rate:

$$W \times \frac{1}{8\,\text{ms}} \ge \frac{1}{4\,\text{ms}} \quad\Rightarrow\quad W \ge 2$$

If the job is configured with a single data-loading worker ($W=1$), that single misconfiguration fully explains the observed pattern: the one worker is pinned doing 125 samples/sec of work while the GPU, capable of 250 samples/sec, drains the queue as fast as it fills and then idles. Raising the worker count toward the CPU core count (or reducing per-sample CPU cost by moving augmentation onto the GPU, e.g. with a GPU-side data-loading library, or by precomputing static features once instead of per-epoch) is the concrete next step to test.

Trade-offs & pitfalls

Adding data-loader workers has a ceiling: once you've used all available CPU cores, more workers just add contention, not throughput; past that point the fix is reducing per-sample CPU cost, not adding workers. The common wrong turn here is assuming the GPU or model architecture is at fault and reaching for a bigger GPU or mixed-precision training, when the accelerator was never actually the bottleneck. Also worth naming explicitly: multi-node straggler mitigation (re-sharding, speculative workers, elastic training) is real and useful, but it solves a different failure mode than this one, jumping straight to distributed-training fixes here is solving the wrong layer of the stack.

Goal: test whether moving inference from CPU to GPU improves latency/throughput and reduces cost per useful request without increasing error rate. Measure performance, cost, and correctness under production-like load.

Experiment design

• Type: randomized A/B test (traffic split at request/endpoint level) with deterministic routing per user/session.
• Variants: A = CPU baseline, B = GPU serving.
• Duration: run for at least one full weekly cycle to capture diurnal patterns (7–14 days).

Key metrics (KPIs)

• Latency: p50, p90, p95, p99 (ms); report mean and std.
• Throughput: requests/sec and max sustainable QPS before tail latency > SLO.
• Cost per request: $/inference including infra (instance amortized), GPU utilization overhead, and queuing.
• Error rate: model output differences (e.g., label/score drift), request failures, and downstream business metric delta.
• Reliability: CPU/GPU memory errors, OOMs, retry counts.
• Composite KPI: cost-per-successful-request (cost * (1/(1-error_rate))).

Sample size & power

• Choose primary metric (e.g., p95 latency). Estimate baseline mean μ0 and std σ from production logs.
• Use two-sample t-test (or bootstrap for non-normal). Required N per arm:
  N = 2 * (Z_{1-α/2} + Z_{1-β})^2 * σ^2 / Δ^2
  where Δ = minimum detectable effect (e.g., 15% reduction in p95), α=0.05, power 1-β=0.8.
• If latency has heavy tails, use log-transform or nonparametric test; bootstrap CIs for percentiles.
• For cost metric, treat as continuous; for error rate (binary), use proportion test sample size formula.

Statistical considerations

• Adjust for multiple comparisons (Bonferroni or control FDR) if testing multiple percentiles/metrics.
• Use stratified randomization by traffic source, model input size, and request type to reduce variance.
• Pre-register primary metric and analysis plan.

Success criteria (example)

• p95 latency decreased by ≥15% with p < 0.05
• Cost per request reduced or net cost saving after considering GPU idle costs
• No statistically significant increase in model error rate (non-inferiority margin e.g., +0.1%)
• No increase in failures or operational incidents

Safe rollout strategy

1. Canary (1% traffic, 24–48h): monitor KPIs and alerts; automatic rollback if thresholds breached.
2. Phased rollout: 1% → 5% → 25% → 50% → 100%; at each step wait N hours/requests and validate SLOs and KPIs with automated checks.
3. Automated guardrails:
   • Immediate rollback triggers: p95 > baseline_p95 * 1.25, error_rate increase > pre-specified delta, GPU OOMs > threshold, or cost-per-request increases beyond ROI threshold.
   • Alerting and anomaly detection on tails and throughput.
4. AB analysis at each phase: confirm effect persists and is not due to transient load differences.
5. Post-rollout monitoring: 2–4 weeks tightened SLO monitoring; collect cost amortization data; run periodic A/B re-checks under peak loads.

Implementation notes

• Ensure identical model binary/precision unless GPU-specific optimizations are part of the experiment; if using mixed precision, treat as separate variant.
• Measure end-to-end latency including queuing and network; isolate inference time as internal metric.
• Account for batching: GPU may benefit from larger batch sizes; run parallel experiments or control batch sizes to compare fairly.
• Use traffic replay and load testing to validate capacity before live rollout.

This framework balances statistical rigor, operational safety, and business-relevant KPIs to decide whether GPU inference is production-improving.