Meta Machine Learning Engineer Interview Preparation Guide - Mid Level (2-5 Years)

Direct answer: Proving a new feature's business impact requires a real randomized experiment measuring the actual business outcome, not just an offline metric bump, and needs a deliberate plan for the specific, recurring failure mode where offline validation improves but the live experiment's business metric doesn't (or gets worse), plus an explicit ROI case made before any of the engineering work starts.

Structured elaboration:
The experiment needs randomized treatment assignment (which requires the feature computation itself to be deterministic and reproducible per treatment group, since non-deterministic feature computation can itself leak treatment assignment or contaminate variants), instrumentation via feature flags so the new feature can be toggled cleanly per arm, and guards against confounds like time-of-day or user-segment imbalance between arms (stratified or blocked randomization, checked post-hoc for balance across arms before trusting the result).

When offline validation area-under-the-curve improves but a live experiment shows a business-metric regression (a genuinely common and important pattern), the investigation needs a structured hypothesis list spanning three layers: a DATA problem (the offline evaluation set doesn't represent the live population the same way, or there's subtle leakage inflating the offline number), a MODEL problem (an offline-metric improvement that doesn't translate to a better real decision, e.g. improved ranking precision that doesn't move the metric users actually care about), or a DEPLOYMENT problem (a bug in how the feature is actually served in the live path versus how it was computed offline, which circles back to training-serving skew). Diagnostic steps: segment the live results (does the regression concentrate in one user segment or time period, suggesting a specific root cause); check the feature's live distribution against its offline-validated distribution (a mismatch here points at a data or deployment issue, not a modeling one); and, if needed, roll back to isolate whether the new feature specifically, versus something else that shipped concurrently, caused the regression.

Determinism and treatment isolation: the feature computation for a given entity must produce the identical value regardless of which experiment arm is evaluating it at that instant (no shared mutable cache keyed only on entity ID that a control-arm read could accidentally populate with a treatment-arm-computed value, for example); treatment/control-specific feature computation paths should be fully isolated (separate cache namespaces or separate materialization jobs keyed by arm), and computation for a given entity-timestamp pair should be idempotent, so a retried or replayed computation doesn't silently drift between arms mid-experiment.

Building the ROI case before doing the work: a simple, explicit framework to reason about whether the engineering investment is worth making:

ROI = (Expected business-metric lift x affected traffic volume x per-unit business value) / (engineering cost + ongoing serving/compute cost)

with each term explicitly estimated and its uncertainty stated: the expected lift comes from a cheap early offline check (a quick ablation showing the feature's plausible ceiling contribution, clearly labeled as an ESTIMATE, not a measurement); affected traffic volume and per-unit business value come from existing production data (a MEASURED baseline); engineering cost is a rough effort/timeline estimate for building and validating the feature; ongoing cost includes the feature's serving latency and compute/storage footprint at scale. A feature whose estimated numerator is dwarfed by its denominator (a large engineering lift for a marginal, low-traffic-affecting signal) is a candidate to deprioritize before any experiment is even designed, not after a costly build-and-test cycle confirms it wasn't worth it.

Worked example: A model with a newly engineered feature shows improved offline validation accuracy but a live A/B test shows conversion drop. Segmenting the live results reveals the regression concentrates entirely on mobile traffic; checking the feature's live distribution on mobile against its offline-validated distribution shows a meaningful mismatch, pointing to a deployment-layer bug (the feature was computed correctly offline but served incorrectly for mobile requests specifically) rather than the feature itself being a bad idea.

Trade-offs and pitfalls: The ROI estimate's expected-lift term is the weakest link in the formula, since it's necessarily based on an offline proxy before any live experiment has run; treating that number as a confident projection rather than a rough, clearly-labeled estimate is exactly the kind of overclaiming that erodes trust when the eventual live result differs from the pre-experiment projection.

Direct answer: A resource-management model for a shared feature platform needs namespace-level quotas that are enforced (not advisory), autoscaling boundaries so no single tenant's burst starves others, priority scheduling so critical production workloads pre-empt best-effort ones, and cost attribution that ties usage back to teams to keep the fair-share system honest over time.

Structured elaboration:

• Namespace quotas. Each team's namespace gets hard caps on storage, compute-seconds for materialization jobs, and online-store operation rate; quotas are enforced at admission (a job that would exceed quota is queued or rejected, not silently allowed to run and degrade shared infrastructure).
• Autoscaling boundaries. The platform autoscales shared compute (materialization workers, online-store shards) within a budget, but per-tenant autoscaling has a ceiling, so one team's traffic spike triggers platform-wide scale-up only up to a bound, beyond which that team's own requests get throttled rather than the platform scaling unboundedly on their behalf.
• Priority scheduling. Classify workloads (production-serving-critical, scheduled-batch, best-effort/experimental) and give the scheduler a priority queue so a production materialization job pre-empts a lower-priority ad-hoc backfill during resource contention, similar to how a Kubernetes cluster uses priority classes and pod preemption.
• Cost attribution and fair-share. Track actual resource consumption per tenant continuously, and use it both for chargeback reporting and as an input to the fair-share algorithm itself (a tenant that has consumed more than its fair share recently gets lower scheduling priority for new work until the moving average rebalances), which is the mechanism that prevents a single well-resourced team from perpetually crowding out smaller teams even within their nominal quota.
• Noisy-neighbor prevention. Beyond quotas, isolate the blast radius of one tenant's misbehaving job (a runaway query, a job stuck in a retry loop) using resource limits (CPU/memory caps, circuit breakers on repeated failures) so it cannot degrade the shared control plane or other tenants' jobs even transiently.

Worked example: Using a priority-class scheme with three tiers (critical production, scheduled batch, best-effort), if the platform is at 90% compute utilization and a critical production materialization job arrives, the scheduler pre-empts a running best-effort job (checkpointing its progress if possible, or simply killing and requeuing it) to make room, rather than queuing the critical job behind lower-priority work; this mirrors how many organizations run Spark or Kubernetes scheduling in shared clusters today.

Trade-offs & pitfalls: A fair-share algorithm based purely on historical consumption can create a "rich get richer, poor stay poor" dynamic if not careful: a team that legitimately needs to scale up (a new product launch) gets penalized by the fair-share mechanism just when it needs more resources, so the model needs an explicit override or request path for legitimate, approved growth rather than treating all consumption spikes identically. Priority pre-emption without checkpointing support means pre-empted jobs restart from scratch, which can waste more resources than it saves if pre-emption happens frequently; investing in checkpoint-and-resume for long-running materialization jobs pays for itself once pre-emption becomes common at scale.

Brief approach: A learning curve plots model performance (train and validation error or accuracy) versus amount of training data. It shows whether the model is underfitting (high training error) or overfitting (gap between train and val).
How to use it to decide data vs capacity:

• Train the current model on increasing fractions of data (10%, 25%, 50%, 75%, 100%). Plot train and validation errors.
• Interpretation:
  • If train error is low and validation error plateaus high with growing data (large gap): model is high variance — increasing data often helps. If validation error keeps improving as data increases, collecting more labeled data is likely beneficial.
  • If both train and validation errors are high and close (small gap): model is high bias — increase model capacity (more parameters, deeper network, richer features) or change model class.
  • If validation error plateaus despite more data and capacity increases don’t help, revisit features, labels, or noise.
    Practical notes: run learning curve experiments with consistent preprocessing, use logarithmic x-axis, and estimate confidence intervals by repeating splits. This gives evidence-based trade-offs between labeling cost and model complexity.

Direct answer: HNSW (Hierarchical Navigable Small World graphs) gives logarithmic-ish query time ($O(\log N)$ empirically, via greedy graph traversal across hierarchical layers) with strong recall but higher memory and index-build cost; IVF+PQ (inverted file index with product quantization) compresses vectors aggressively for much lower memory at the cost of recall, using a coarse clustering to restrict search to a subset of the data; LSH (locality-sensitive hashing) gives probabilistic sub-linear query time with weaker recall guarantees than either, but simpler theoretical underpinnings and easy distributed/sharded implementation. All trade exactness for speed and memory versus brute-force O(N x d) linear scan.

Structured elaboration

• HNSW: builds a multi-layer graph where higher layers have progressively fewer nodes (a "skip list"-like structure over the vector space), allowing search to start at a sparse top layer and greedily navigate toward the query's nearest neighbors, refining through denser lower layers. Query time is empirically close to logarithmic in N; build time is superlinear (each insertion involves graph-edge construction, roughly $O(N \log N)$ overall) and memory overhead is significant (storing the multi-layer graph structure, not just the raw vectors) - HNSW typically gives the BEST recall-per-query-latency trade-off among common ANN methods, at the highest memory cost.
• IVF+PQ: first clusters the dataset into coarse "cells" via k-means (the Inverted File index); a query only searches within the few nearest cells to its own location, avoiding the need to compare against the full dataset. Product Quantization further compresses each vector by splitting it into sub-vectors and quantizing each sub-vector to a small codebook, shrinking memory dramatically (often 10-30x smaller than storing raw float vectors) at the cost of the quantization introducing approximation error that reduces recall.
• LSH: hashes vectors such that similar vectors are more likely to collide into the same hash bucket than dissimilar ones (the opposite goal of a normal, collision-avoiding hash function); a query hashes and only checks the colliding bucket(s). Simpler to reason about and to shard/distribute (each hash table can live on a different machine), but typically needs multiple hash tables to achieve competitive recall, and its probabilistic guarantees are looser than HNSW's empirical graph-navigation quality.

Worked example

For 100 million 768-dimensional embeddings (a realistic large-scale semantic search setting): raw storage at float32 is $100\text{M} \times 768 \times 4\text{B} \approx 307\text{GB}$. IVF+PQ compressing each vector to, say, 64 bytes (a common PQ configuration) shrinks storage to $100\text{M} \times 64\text{B} = 6.4\text{GB}$ - roughly 48x smaller, making the difference between needing a multi-machine memory-resident index and fitting comfortably on a single large-memory machine. HNSW's graph overhead instead ADDS to the raw vector storage (typically an additional 30-100+ bytes per vector for the graph edges across layers, so roughly 100M x (3072+~60) bytes ≈ 313GB total) - HNSW does not compress the base vectors, so its memory footprint stays close to (or above) the raw storage size, trading memory for its superior recall/latency at query time.

Trade-offs & pitfalls

• Recall (fraction of TRUE nearest neighbors actually returned) versus latency versus memory is a three-way trade across all these methods - there's no universally "best" choice; the right one depends on which axis your system's constraints bind on hardest.
• Build/update cost matters for dynamic datasets: HNSW graph updates (inserting new vectors) are more expensive than IVF+PQ's (which can often just assign a new vector to its nearest existing cluster cheaply), a real consideration if the embedding collection changes frequently rather than being built once and queried many times.
• Hybrid approaches (e.g. IVF+HNSW combinations, or IVF+PQ with a re-ranking pass using exact distances on a small candidate set) are common in production to balance these trade-offs rather than committing to one pure method.

Direct answer

Do not build "the model, then the system, then the rollout" as layers. Cut the product into vertical slices (each a thin path from data to a decision a person or system acts on), and order them so each slice either delivers value or produces something the next slice needs. For fraud, the scarce thing is labeled outcomes, so the first slice is a shadow-mode rules baseline (rules that run on live transactions and log what they would flag, without blocking anything) feeding a human review queue. It is useful on day one and starts generating labels.

Structured elaboration

Why labels drive the order. Fraud labels are confirmed late (a chargeback, the reversal a customer's bank makes after the customer disputes a card payment, can arrive weeks after the transaction; card networks typically allow disputes up to about 120 days after the payment) and fraud is rare, so a model trained too early learns from little and stale evidence. Also, transactions you block never show whether they were fraud, which biases later training data.

Slices, in order

1. Rules plus logging, in shadow. A handful of explicit rules (for example many attempts per card per hour) send transactions to an analyst queue without blocking. Value: analysts catch fraud now. Learning: base rate (how often fraud occurs, here 0.2%), review capacity, first labels.
2. Labeled dataset and offline evaluation. Time-based train and test split (train on the past, test on the future). Judge at fixed review capacity (recall among the top K flagged per day, where K is the number of cases analysts can review, for example 500), because with rare fraud accuracy is meaningless: a model that flags nothing is 99.8% accurate when only 0.2% of transactions are fraud, yet it catches no fraud at all.
3. First model in shadow. A simple tabular model (such as gradient-boosted trees, a common model type that combines many small decision trees) scores every transaction, logged but not acted on. Learning: lift over the rules (how much more fraud the model catches than the rules at the same review load), measured on later-arriving labels.
4. Model-ordered review queue. Analysts see highest scores first. Value: same staff catch more fraud.
5. Automated action on a narrow slice. Only the highest-confidence band, preferring a softer action such as step-up verification (asking for extra proof of identity) over outright decline, with a small random holdout (a group deliberately left alone and never blocked) so you can still measure.
6. Feedback loop and monitoring. Retraining cadence, drift alerts (the data changing so the model slowly gets worse), fraudsters adapting, the holdout kept running.

Worked example

Illustrative numbers: 100,000 transactions a day, 0.2% fraud, so 200 fraudulent. Analysts can review 500 a day. Precision is the share of flagged transactions that are truly fraud; recall is the share of all fraud that gets flagged.

• Rules fill the queue with 60 true frauds: precision 60/500 = 12%, recall 60/200 = 30%.
• A model filling the same 500 slots with 110 true frauds: precision 110/500 = 22%, recall 110/200 = 55%.
  That comparison, same staff and a bigger catch, is the evidence for slice 4.

Variation: a two-month personalized-ranking plan in four two-week sprints. Personalized ranking means ordering what each user sees (for example products in a feed) by predicted interest. The same logic applies: ship a baseline first, then earn each model step. Terms in the table: an impression is one time an item was shown to a user; implicit-label bias means clicks only exist for items the old ranking chose to show, so the data reflects the old ranking; a model registry is a versioned store of trained models that lets you roll back; an A/B test shows the new ranking to a random half of users and the baseline to the other half, then compares.

Trade-offs and pitfalls

• Model-first sequencing is the common trap: it spends months before any value or any label.
• Horizontal layers (all data work, then all modeling) delay learning; each slice must reach a decision.
• False positives cost real customers, so the automation slice needs a guardrail on declined good transactions.
• What would flip the order: if years of labeled history already exist, start at slice 2 or 3.
• Regulations or explainability needs may require a simpler model or a human in the loop longer than technical readiness alone suggests.

Situation: I need a repeatable CI test plan that verifies a TensorFlow SavedModel -> ONNX -> ONNX Runtime roundtrip produces numerically equivalent outputs within acceptable tolerances.

Test plan (high-level steps)

1. Export & convert pipeline

• Scripted steps: (a) export SavedModel, (b) convert with tf2onnx/onnx-tf, (c) run ONNX Runtime inference. Fix RNG seeds and TF/ONNX runtime versions.

2. Test data selection

• Unit tests: small hand-crafted vectors that exercise edge cases (zeros, ones, large/small magnitudes, negative, inf/nan).
• Functional tests: random inputs with fixed seeds across distributions (uniform, normal, skewed).
• Coverage tests: inputs that trigger different ops, dynamic shapes, batch sizes, and quantized/dtype variants.
• Real-data smoke test: 50–200 real samples from production-ish dataset.

3. Tolerance rules & metrics

• Per-output checks:
  • Exact equality for integer outputs.
  • For floating types, use combined metrics:
    • max_abs = max(|y_tf - y_onnx|)
    • rms = sqrt(mean((y_tf - y_onnx)^2))
    • cosine_sim for embeddings/vectors.
  • Default thresholds (float32): rtol=1e-5, atol=1e-6; practical thresholds: max_abs < 1e-4 or rms < 1e-6. For fp16 or quantized: relax (rtol=1e-2, atol=1e-3).
• Special checks: NaN/Inf parity (fail if TF has finite and ONNX has NaN/Inf or vice versa).
• Relative per-output scaling: normalize by max(|y_tf|, epsilon) when outputs span orders of magnitude.

4. Pass/fail rules

• Per-test: pass if metrics under thresholds and NaN/Inf parity holds.
• Aggregate: allow up to a tiny percentage (e.g., 1–2%) of samples to exceed soft thresholds for flaky ops; failing tests trigger investigation.

5. Automation & CI hooks

• Integrate into CI pipeline (GitHub Actions / Jenkins):
  • Matrix run across runtime versions, hardware (CPU/GPU), and dtypes.
  • Store artifacts: SavedModel, ONNX model, test inputs/outputs, diff reports, and serialized failure cases.
  • Auto-generate human-readable report with metric summaries and example failing cases (show inputs, TF vs ONNX outputs, diffs).
  • Alerting: fail the PR on hard failures; for soft failures, open a ticket with attached artifacts.
  • Regression baselines: keep golden outputs and only allow changes via approved updates.

6. Additional practices

• Add randomized fuzzing tests periodically (nightly).
• Maintain converter-version compatibility tests.
• Add model-level unit tests for deterministic ops and stochastic ops (ensure seeds or compare distributions).

This plan provides deterministic, reproducible checks, clear numeric criteria per dtype, and CI automation to catch regressions early while producing helpful artifacts for debugging.

Frechet Inception Distance (FID) and Inception Score (IS) are two common, quick-to-compute metrics for evaluating image generative models.

Frechet Inception Distance (FID)

• What it is: Measures distance between distributions of real and generated images in the feature space of a pretrained Inception network (usually activations from a late layer).
• How computed (plain): Extract features for many real images and many generated images, fit a multivariate Gaussian to each (mean μ, covariance Σ), then compute the Frechet (Wasserstein-2) distance:
  D² = ||μ_real − μ_gen||² + Tr(Σ_real + Σ_gen − 2(Σ_real Σ_gen)^{1/2}).
• Captures: Both sample quality (how close generated features are to real ones) and diversity (covariance term). Lower is better.

Worked example (toy 2D feature vectors, not real Inception activations, just to make the formula concrete). Real-image mean μ_real = [2.0, 1.0]; generated mean μ_gen = [2.3, 0.9]. Mean term: ||μ_real − μ_gen||² = (2.0−2.3)² + (1.0−0.9)² = 0.09 + 0.01 = 0.10, the same in both cases below. Case 1, mode collapse (generated samples barely vary): Σ_real = diag(1.0, 1.0), Σ_gen = diag(0.2, 0.2). Since both are diagonal, (Σ_real Σ_gen)^{1/2} = diag(√0.2, √0.2) ≈ diag(0.447, 0.447). Covariance term = Tr(Σ_real+Σ_gen) − Tr(2·(Σ_real Σ_gen)^{1/2}) = (1.2+1.2) − 2(0.447+0.447) ≈ 2.4 − 1.789 = 0.611. Total FID ≈ 0.10 + 0.611 = 0.711. Case 2, healthy diversity: Σ_gen = diag(0.9, 0.9), close to Σ_real. (Σ_real Σ_gen)^{1/2} = diag(√0.9, √0.9) ≈ diag(0.949, 0.949). Covariance term = (1.9+1.9) − 2(0.949+0.949) ≈ 3.8 − 3.795 = 0.005. Total FID ≈ 0.10 + 0.005 = 0.105. Same mean gap (0.10) in both cases, but the covariance term alone jumps from 0.005 to 0.611 once the generated distribution's spread collapses to a fifth of the real distribution's, which is exactly how a shrinking-diversity failure mode drives FID up even when the generator's average output stays close to the real average.

Inception Score (IS)

• What it is: Measures image quality and diversity using a pretrained Inception classifier.
• How computed (plain): For each generated image, get the predicted label distribution p(y|x). Compute KL divergence between p(y|x) and the marginal p(y) across samples, then exponentiate the average:
  IS = exp(E_x[KL(p(y|x) || p(y))]).
• Captures: High IS if individual images are confidently classifiable (quality) and the marginal distribution covers many classes (diversity). Higher is better.

Limitations / Failure Modes (at least three)

1. Dependence on Inception network and dataset: Both use a network trained on ImageNet, so they bias evaluation toward ImageNet-like semantics and fail for non-ImageNet domains (medical, satellite, stylized art).
2. Sensitivity to mode collapse/overfitting: IS can be high if generated images are confidently classified into many classes but may miss intra-class mode collapse. FID can be fooled by models that memorize training images or by small sample sizes affecting covariance estimates.
3. Insensitivity to perceptual issues: Both operate in feature space and may not reflect human judgments for texture, fine details, or realism (e.g., sharpness vs. plausible semantics).
4. Statistical instability and sample complexity: FID estimates need many samples; with limited samples the score has high variance and can change with random seeds, preprocessing, or implementation details.
5. Non-robust to image manipulations: Simple post-processing (color shifts, resizing, or small perturbations) can change scores significantly without meaningful quality change.

Best practice: use FID/IS as rough guidance alongside human evaluation and task-specific metrics; for non-ImageNet domains prefer domain-specific perceptual features or learned metrics (LPIPS, FID variants) and report sample sizes and seeds.

Direct answer

Give the executive-dashboard path and the ad-hoc, lower-priority path separate directed acyclic graphs (DAGs), or at minimum no shared edges between them, so a failure or slow run in the low-priority branch can never block, delay retries for, or consume retry budget from the critical branch. Where both paths need the same upstream source, fan out from a single shared ingestion step into two independent downstream graphs, rather than making one branch depend on the other's completion.

Structured elaboration

Dependency structure: any task on the critical path should have zero downstream or upstream edges to a non-critical task. If both branches read the same raw table, model that as two independent consumers of one producer, not a chain. A shared DAG where the low-priority branch is simply further along the same line means a slow ad-hoc stage delays the scheduler's view of whether the run is complete, even when the actual dashboard output was never late.

Retries: give the critical path a short retry interval with a small number of aggressive retries, so a transient failure resolves fast. The ad-hoc path can use longer backoff, more retries, or no automatic retry at all (a rerun-on-demand model), since it is acceptable for it to take longer or wait for manual attention.

Service-level agreements (SLAs): define the SLA only against the critical path's own tasks. If the SLA calculation looks at when the whole DAG finished instead of when the tasks feeding the dashboard finished, a slow ad-hoc branch can falsely trip an SLA alert for a dashboard that was actually delivered on time.

Resource contention: the two paths can compete for the same execution resources (worker slots, warehouse compute) even when they are structurally independent DAGs. Give the critical path its own resource pool or a higher-priority queue slot so it is never queued behind ad-hoc work.

Worked example

A cohort or retention dashboard needs a daily active-user rollup ready by a fixed review time, while the same raw event stream also feeds a separate ad-hoc analysis DAG (funnel breakdowns, one-off segment cuts) with no fixed deadline. The structure: one shared ingestion task lands raw events; two independent downstream DAGs both consume it. The cohort/retention DAG (aggregate, compute the retention window, publish to the dashboard) carries a tight SLA and fast retries. The ad-hoc DAG (exploratory transforms) has its own scheduling and no SLA. If the ad-hoc DAG's exploratory job hangs because of a runaway query, the cohort/retention DAG is unaffected, because it was never downstream of it, only downstream of the shared ingestion task, which had already completed. Because the cohort DAG's own compute only processes the day's new partition rather than the full history, its SLA can reasonably be set tight precisely because it has been structurally decoupled from anything the ad-hoc path might do; a shared-DAG design would force that same SLA to account for the ad-hoc branch's unpredictable runtime too.

flowchart LR
  ingest["shared ingestion"] --> critical["critical: retention rollup"]
  ingest --> adhoc["ad-hoc: exploratory transforms"]
  critical --> dashboard["executive dashboard"]
  adhoc --> analysis["ad-hoc analysis output"]

Trade-offs & pitfalls

Fully separate DAGs mean the shared ingestion logic is maintained once but consumed twice, which is fine, but if the two teams then diverge and each patches its own copy of ingestion, the coupling problem reappears in a different shape; ingestion should stay single-owned. Separate resource pools or queues cost real reserved infrastructure that can sit idle when the critical path is not running, that is the price of guaranteeing it is never queued behind ad-hoc work. The most common wrong turn is assuming task-level retry configuration alone solves this: if the DAG structure itself couples the two paths through a shared edge, no retry policy or priority setting fixes the fact that the scheduler still sees one graph.

Direct answer
Test-Driven Development (TDD) for top_k_accuracy(preds, labels, k) means writing the failing tests FIRST using pytest, in an order that forces the implementation to handle edge cases by construction rather than by afterthought: start with the simplest correctness case, then add the boundary and error cases (empty batch, invalid k, k equal to the number of classes, tied scores) one at a time, only writing enough implementation to pass each newly-added test. The same TDD-first-write-the-edge-case-tests pattern, expressed as plain pytest functions using assert and pytest.raises, applies equally to normalize_image (a preprocessing function) and compute_accuracy (a related, simpler metric function), demonstrating the technique generalizes across an ML (machine learning) pipeline's different function shapes.

Structured elaboration: the TDD sequence for top_k_accuracy

1. Write the simplest possible passing case first: a batch where the true label is always the top-scored class, k=1. This forces only the core "is the label in the top-k scored indices" logic to exist.
2. Write the simplest FAILING case: the true label is NOT the top-scored class at k=1, confirming the function actually discriminates correct from incorrect, not just returning a constant.
3. Write the boundary that separates k=1 from k=2: a label ranked exactly 2nd, which must fail at k=1 and pass at k=2, this is the test that would catch an off-by-one in the ranking/slicing logic.
4. Write the k equals the number of classes case: accuracy must be 1.0 regardless of ranking, since every class is "in the top-k" once k covers all classes, a case that specifically catches an implementation that only ever checks the top-1 position and ignores k for anything above it.
5. Write the input-validation error cases: empty batch, k < 1, k exceeding the number of classes, and mismatched preds/labels lengths, each as its own test asserting the specific exception raised, not just "it doesn't crash."
6. Write the tie-breaking case explicitly: when scores are tied, the implementation needs a DEFINED, deterministic tie-breaking rule (this implementation uses a stable sort, so the lowest class index wins ties), and the test pins that specific behavior down rather than leaving it to whatever the sort algorithm happens to do.
7. Write a mixed-batch fractional-result case (some correct, some wrong) to confirm the aggregate math (correct / total) is right, not just each individual per-sample check.

Minimal implementation that satisfies the sequence above

def top_k_accuracy(preds, labels, k):
    if k < 1: raise ValueError("k must be >= 1")
    if len(preds) != len(labels): raise ValueError("preds and labels must have the same length")
    if len(preds) == 0: raise ValueError("top_k_accuracy is undefined for an empty batch")
    n_classes = len(preds[0])
    if k > n_classes: raise ValueError(f"k={k} exceeds the number of classes ({n_classes})")
    correct = 0
    for scores, label in zip(preds, labels):
        top_k_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k]
        if label in top_k_indices:
            correct += 1
    return correct / len(labels)

The ten tests named below, written for real and run for real

def test_top_k_perfect_top1():
    assert top_k_accuracy([[0.1, 0.9, 0.0]], [1], k=1) == 1.0

def test_top_k_zero_top1():
    assert top_k_accuracy([[0.9, 0.1, 0.0]], [1], k=1) == 0.0

def test_top_k_label_in_topk_but_not_top1():
    assert top_k_accuracy([[0.5, 0.3, 0.2]], [1], k=2) == 1.0
    assert top_k_accuracy([[0.5, 0.3, 0.2]], [1], k=1) == 0.0

def test_top_k_equals_num_classes_is_always_1():
    assert top_k_accuracy([[0.9, 0.05, 0.05], [0.05, 0.05, 0.9]], [2, 0], k=3) == 1.0

def test_top_k_empty_batch_raises():
    try:
        top_k_accuracy([], [], k=1); assert False
    except ValueError:
        pass

def test_top_k_k_exceeds_num_classes_raises():
    try:
        top_k_accuracy([[0.1, 0.9]], [0], k=3); assert False
    except ValueError:
        pass

def test_top_k_k_less_than_1_raises():
    try:
        top_k_accuracy([[0.1, 0.9]], [0], k=0); assert False
    except ValueError:
        pass

def test_top_k_mismatched_lengths_raises():
    try:
        top_k_accuracy([[0.1, 0.9]], [0, 1], k=1); assert False
    except ValueError:
        pass

def test_top_k_tie_scores_deterministic():
    # scores[0] == scores[1]; stable sort keeps the lowest index first, so index 0 wins the tie
    assert top_k_accuracy([[0.5, 0.5, 0.0]], [0], k=1) == 1.0
    assert top_k_accuracy([[0.5, 0.5, 0.0]], [1], k=1) == 0.0

def test_top_k_mixed_batch_fractional_result():
    preds = [[0.9, 0.1], [0.9, 0.1], [0.1, 0.9]]
    labels = [1, 1, 1]
    # sample0 wrong, sample1 wrong, sample2 correct -> 1 correct, 2 wrong -> exactly 1/3
    assert abs(top_k_accuracy(preds, labels, k=1) - 1/3) < 1e-9

import time
_tk_tests = [test_top_k_perfect_top1, test_top_k_zero_top1, test_top_k_label_in_topk_but_not_top1,
    test_top_k_equals_num_classes_is_always_1, test_top_k_empty_batch_raises, test_top_k_k_exceeds_num_classes_raises,
    test_top_k_k_less_than_1_raises, test_top_k_mismatched_lengths_raises, test_top_k_tie_scores_deterministic,
    test_top_k_mixed_batch_fractional_result]
_start = time.perf_counter()
for _t in _tk_tests:
    _t()
    _suffix = "   (3-sample batch: 1 correct, 2 wrong -> exactly 1/3)" if _t.__name__ == "test_top_k_mixed_batch_fractional_result" else ""
    print(f"{_t.__name__} PASSED{_suffix}")
print(f"{len(_tk_tests)} passed in {time.perf_counter() - _start:.2f}s")

Executed test results (all written before the corresponding line of implementation, per the TDD sequence above)

test_top_k_perfect_top1 PASSED
test_top_k_zero_top1 PASSED
test_top_k_label_in_topk_but_not_top1 PASSED
test_top_k_equals_num_classes_is_always_1 PASSED
test_top_k_empty_batch_raises PASSED
test_top_k_k_exceeds_num_classes_raises PASSED
test_top_k_k_less_than_1_raises PASSED
test_top_k_mismatched_lengths_raises PASSED
test_top_k_tie_scores_deterministic PASSED
test_top_k_mixed_batch_fractional_result PASSED   (3-sample batch: 1 correct, 2 wrong -> exactly 1/3)
10 passed in 0.02s

The same TDD pattern applied to compute_accuracy and normalize_image

def compute_accuracy(preds, labels):
    if len(preds) != len(labels): raise ValueError("preds and labels must have the same length")
    if len(preds) == 0: raise ValueError("compute_accuracy is undefined for an empty batch")
    return sum(1 for p, l in zip(preds, labels) if p == l) / len(labels)

def normalize_image(pixels, mean=0.5, std=0.5):
    if std == 0: raise ValueError("std must be non-zero")
    return [[(p - mean) / std for p in row] for row in pixels]

The eight tests named below, written for real and run for real

def test_compute_accuracy_all_correct():
    assert compute_accuracy([1, 0, 1], [1, 0, 1]) == 1.0

def test_compute_accuracy_all_wrong():
    assert compute_accuracy([1, 0, 1], [0, 1, 0]) == 0.0

def test_compute_accuracy_empty_raises():
    try:
        compute_accuracy([], []); assert False
    except ValueError:
        pass

def test_compute_accuracy_mismatched_lengths_raises():
    try:
        compute_accuracy([1, 0], [1]); assert False
    except ValueError:
        pass

def test_normalize_image_midpoint_maps_to_zero():
    assert normalize_image([[0.5]], mean=0.5, std=0.5) == [[0.0]]

def test_normalize_image_extremes():
    assert normalize_image([[0.0, 1.0]], mean=0.5, std=0.5) == [[-1.0, 1.0]]

def test_normalize_image_zero_std_raises():
    try:
        normalize_image([[0.5]], mean=0.5, std=0.0); assert False
    except ValueError:
        pass

def test_normalize_image_preserves_shape():
    pixels = [[0.1, 0.2], [0.3, 0.4]]
    result = normalize_image(pixels)
    assert len(result) == len(pixels) and all(len(r1) == len(r2) for r1, r2 in zip(result, pixels))

_acc_tests = [test_compute_accuracy_all_correct, test_compute_accuracy_all_wrong, test_compute_accuracy_empty_raises,
    test_compute_accuracy_mismatched_lengths_raises, test_normalize_image_midpoint_maps_to_zero,
    test_normalize_image_extremes, test_normalize_image_zero_std_raises, test_normalize_image_preserves_shape]
for _t in _acc_tests:
    _t()
    print(f"{_t.__name__} PASSED")
print(f"{len(_acc_tests)} passed total for these two functions")

Executed: test_compute_accuracy_all_correct, test_compute_accuracy_all_wrong, test_compute_accuracy_empty_raises, test_compute_accuracy_mismatched_lengths_raises all PASSED; test_normalize_image_midpoint_maps_to_zero (a pixel value equal to the mean normalizes to exactly 0.0), test_normalize_image_extremes (pixel 0.0 maps to -1.0, pixel 1.0 maps to 1.0, under mean=0.5, std=0.5), test_normalize_image_zero_std_raises, and test_normalize_image_preserves_shape all PASSED, 8 passed total for these two functions.

Trade-offs and pitfalls
The most common wrong turn in applying TDD here is writing several tests at once before running any of them, which defeats the actual discipline (each test should be run RED before the implementation change that turns it GREEN, confirming the test can actually fail, not just that it happens to pass). A second pitfall is skipping the tie-breaking test because "ties are rare in real model outputs," when in fact softmax outputs over many classes with limited numeric precision produce exact ties more often than intuition suggests, and an undefined tie-breaking rule makes a metric non-reproducible between two runs of the same evaluation. A third is writing the k equals the number of classes test as an afterthought rather than deliberately early, since it is exactly the case that exposes an implementation that special-cased k=1 and never generalized the slicing logic to arbitrary k.

Direct answer

After a product launch I owned came in well short of its adoption target, I ran a structured review with the two other people closest to the work, and it surfaced something I genuinely hadn't seen while we were building: the target itself had been set based on a comparable launch that wasn't actually comparable, which meant part of the shortfall was a bad target, not just execution. What changed because of it was both a fix to the immediate rollout and a change to how I set targets for anything similar going forward.

How I structured the review

I scheduled it two weeks after launch, once we had a real signal instead of just launch-week noise, and kept it to the two people who had been closest to the build and rollout decisions. Before the meeting, I pulled the original planning document, the actual usage data, and a timeline of the decisions we'd made along the way, so the conversation could be grounded in what we'd actually said and done rather than what we remembered saying. I ran it around a small set of questions: what did we expect and why, where did the plan and reality diverge, and what would we have needed to know earlier to catch it.

What it surfaced

Working through the timeline, we found that the adoption target had been benchmarked against a previous launch in a different user segment with meaningfully different existing habits, something none of us had flagged while setting the target because the comparison felt intuitively reasonable at the time. We also found a smaller, genuinely execution-related gap: onboarding for the new feature was buried two screens deep, which the usage data showed was where a real chunk of users dropped off, something none of us had noticed during testing because we already knew where to find it.

What changed

The onboarding placement was fixed within a week, and usage on that path improved measurably, though I won't claim a precise before-and-after number beyond that it was a clear, visible shift in the funnel. The more durable change was to how I set targets afterward: I now require an explicit note on any benchmark comparison stating what's actually different about the comparison case, rather than letting a comparison stand just because it feels close enough. I tracked that change by checking, on the next two launches, whether that note existed before the target was finalized, rather than just trusting that I'd remember to do it.

Trade-offs and pitfalls

The hindsight in a review like this is easy to feel foolish about, since the flawed benchmark seemed obviously reasonable in the room when we set it. The pitfall is treating that hindsight as evidence the team was careless, when the more useful conclusion is usually that a specific assumption needed to be made explicit and checked, which is a fixable process gap rather than a character flaw.