Lyft Machine Learning Engineer (Staff Level) Interview Preparation Guide

I push back by anchoring on the business outcome, not by saying no reflexively.

How I handle it:

1. I first clarify what problem the stakeholder is trying to solve.
2. I compare the request against the current roadmap and explain the trade-off in plain language.
3. I show the impact on timing, quality, or other committed work if we take it now.
4. I offer options: replace something else, phase it into a later release, or test it in a smaller pilot.

Example phrasing:
“Your request is valid, but if we add it this sprint, we’ll delay the launch item we already committed to. We can either swap scope, defer this to the next cycle, or find a thinner version that gets you part of the value sooner.”

How I preserve the relationship:
I stay consistent, transparent, and respectful. I acknowledge the stakeholder’s urgency, follow up with written decisions, and keep them updated so they feel heard even when the answer is no. That usually builds trust, because they see I’m protecting the broader business, not just the team’s convenience.

Direct answer

With no precedent to copy, the first job is turning ambiguity into a falsifiable first slice: the smallest piece of the system that forces the decisions everything else depends on, and that produces a real, checkable result rather than more planning. Sequencing correctly means making the expensive-to-reverse foundational choices first, deliberately, even though they do not look like progress yet.

Structured elaboration

• With no internal precedent, the risk is not picking the wrong technology, it is answering the wrong first question. Spend the first stretch of time narrowing what "done" means for the smallest real version of the system, one real use case working start to finish, before building broadly.
• Foundational, hard-to-reverse choices go first and get chosen deliberately, not by default. This is true even for a decision that looks like "just tooling": choosing a frontend framework for a genuinely new product should be weighed on the team's actual skill with it, how mature its ecosystem is for what you are building, and what it costs to maintain years out, not which one is newest. Get this wrong at the start and every feature built on top inherits the cost of migrating off it later.
• Sequence by what resolves the biggest open unknown fastest, even if that piece is not the most visible feature, because everything else is easier to descope than to discover you built on a broken foundation.
• Treat the first slice as a real, if narrow, production system, not a throwaway prototype. It needs to be operable (monitored, on a real deploy path), because "no precedent" also means there is no one else's on-call runbook to fall back on if it breaks.

Worked example

A product wants a new capability, say real-time personalization, that the company has never built anything like: no existing pipeline, no established pattern to extend, and real uncertainty about whether the latency and cost targets are even achievable. Rather than designing the full system, the first slice is a single end-to-end path for one use case, deliberately excluding the breadth of features the eventual product needs. The foundational choices made in that slice, which streaming and serving technology to standardize on, which team owns the resulting on-call load, get made carefully, because they are expensive to unwind once several more use cases are built on top; the same team-skill, ecosystem-maturity, long-term-maintenance logic that applies to choosing a frontend framework applies here to the streaming and serving choice. Once that first slice runs in production and holds up under real traffic, it becomes the reference pattern for every subsequent use case, and the sequencing question for everything after that is simply which use case is next most valuable now that the foundation exists.

Trade-offs and pitfalls

The biggest failure mode is treating "no precedent" as license to build broad and figure out the foundation as you go, which usually means the foundation gets set implicitly by whatever the first feature happened to need, not by a deliberate choice. The second is picking the first slice for visibility, something demo-able, rather than for what it teaches you, which produces an impressive demo sitting on an untested foundation. The third is under-investing in operability for the first slice because "it's just a prototype," then discovering it quietly became load-bearing before anyone hardened it.

Direct answer

A retro after a release with repeated design-engineering friction should produce two things: an honest, specific account of where the handoff actually broke down, not a vague 'communication issues,' and a small number of concrete process changes, each with an owner and a way to tell in a quarter whether it worked. Running it well means separating fact-finding from diagnosis, and diagnosis from blame.

Structured elaboration

Design principles for the session

• Facts before diagnosis: start from a timeline of what actually happened (spec dates, handoff dates, bug counts, points where implementation and design diverged), not from opinions about who was at fault.
• Root cause, not the nearest symptom: 'engineering didn't follow the spec' is a symptom; the root cause might be that the spec didn't capture edge-case states, or that both sides were working from different versions of a shared design system mid-migration.
• Few, high-leverage commitments: two or three process changes people will actually do beat ten action items that quietly get dropped.
• Everyone leaves with the same understanding of what changed, not just what went wrong.

A workable structure

One illustrative shape, adaptable to a team's own rhythm:

What 'actually changes how the two teams work' looks like

The output isn't a list of intentions, it's a specific artifact or habit that exists after the meeting and didn't before: a shared checklist embedded in the handoff process, an automated check that catches a class of mismatch before it ships, or a standing short sync during implementation windows. Whatever it is, it needs a way to tell if it worked, not just that it happened.

Worked example

One team's root cause turned out to be that design tokens (colors, spacing values) were maintained in the design tool but hand-copied into code, so drift was inevitable and nobody could tell which side was 'correct' when they disagreed. The concrete fix was an automated export from the design tool into the codebase, checked by both a design reviewer and a frontend reviewer before merge, plus a short recurring sync during active implementation. A quarter later, the team had a real signal that it worked: noticeably fewer visual-mismatch comments on pull requests and less late-stage rework than the release that triggered the retro. The same root-cause pattern shows up in other domains as a hand-copied data contract or config value instead of a design token, so the same fix shape (automate the handoff, add a lightweight check, add a short sync during the risky window) generalizes well beyond design and engineering specifically.

Trade-offs and pitfalls

• A retro that produces ten action items usually produces zero completed ones; prioritizing ruthlessly matters more than being thorough.
• If the room jumps straight to solutions or blame instead of facts first, the real root cause, often structural or tooling-related rather than a person's failure, never surfaces.
• A retro that isn't revisited becomes theater. Put the check-in on the calendar before the room disperses, not as a vague intention afterward.
• Watch for a fix that only addresses this specific release's symptom (a one-off manual double-check) rather than the structural cause; it holds for one cycle and then quietly stops happening.

Direct answer

A full retraining CI/CD pipeline runs data validation, featurization tests, training, offline evaluation against a baseline, statistical validation gates, shadow testing, canary rollout, and registry promotion in sequence, with each stage producing a concrete artifact the next stage consumes: the discipline that makes this safe is that a failure at any stage halts the pipeline rather than silently proceeding.

Structured elaboration

Worked example

For reducing cycle time without sacrificing safety: parallelize what can genuinely run concurrently (featurization tests and data validation don't depend on each other and can run in parallel rather than sequentially), cache and reuse feature-computation results that haven't changed since the last run rather than recomputing from scratch, and set SHORTER shadow/canary windows for LOW-risk model updates (a minor retrain on the same architecture) while keeping the full, longer validation sequence for higher-risk changes (an architecture change, a new feature source): not every retrain needs the same length of caution, and a pipeline that treats every change identically wastes cycle time on the low-risk cases.

Secrets and compute-environment management: training credentials and any API keys live in a secrets manager (not embedded in pipeline config), and the compute environment itself (container image, library versions) is pinned and versioned alongside the code, so a "successful" pipeline run from six months ago can be traced back to exactly what environment produced it.

Trade-offs & pitfalls

The most common real-world failure in a pipeline like this isn't any single stage being wrong, it's DRIFT BETWEEN stages: the featurization code tested in the unit-test stage subtly differing from the featurization code that actually ran in production training, because they're maintained in two places. Sharing the SAME code path for training-time featurization and the unit tests that validate it (rather than a parallel test-only implementation) closes this gap.

Post-training static quantization (PTQ-static)

• How it works: We convert weights and activations from float32 to lower-bit (usually int8) using fixed scale/zero-point ranges computed ahead of inference; activations use per-layer or per-channel quant params.
• Calibration/data: Requires representative calibration dataset (a few hundred to thousand samples) to estimate activation ranges.
• Accuracy impact: Often minimal for CNNs; possible drop for sensitive networks (NLP/transformers) without careful calibration or per-channel quantization.
• Hardware: Broad CPU and many NPUs support int8. GPUs have mixed support (some support int8 via TensorRT). TPUs usually prefer bfloat16 or int8 with specific tooling.
• Use-cases: Fastest path to deploy when you can collect calibration data and need large latency/throughput gains.

Post-training dynamic quantization (PTQ-dynamic)

• How it works: We quantize weights ahead of time (usually to int8) but activations are quantized dynamically at runtime (scale computed per activation/tensor), often using 8-bit or 16-bit.
• Calibration/data: No calibration dataset required.
• Accuracy impact: Very low impact for many models (especially transformers when applied to linear layers), sometimes slightly worse than static for conv nets.
• Hardware: Excellent CPU support (PyTorch dynamic int8 on x86). Limited GPU/TPU benefits unless runtime supports fast dynamic quant ops.
• Use-cases: When you lack calibration data or want quick CPU inference speedups with minimal engineering.

Quantization-aware training (QAT)

• How it works: Simulates quantization (fake quant ops) during training so the model learns to be robust to reduced precision; then export weights as low-bit.
• Calibration/data: Requires full labeled training data and retraining/fine-tuning (can be few epochs).
• Accuracy impact: Best accuracy among quant methods - often matches float baseline closely, critical for sensitive models.
• Hardware: Works where int8 inference is supported; final model runs on same hardware as PTQ but needs toolchain to convert fake-quant to real ops (TensorRT, TFLite, etc.). TPUs may prefer bfloat16; QAT can target that too.
• Use-cases: When accuracy is critical (NLP, detection) or PTQ causes unacceptable drop; acceptable when you can fine-tune and invest engineering time.

Summary guidance:

• Use dynamic for quick CPU wins/no calibration.
• Use static if you have calibration data and want better performance on supported accelerators.
• Use QAT when accuracy must be preserved and you can retrain/fine-tune.

Also covers (folded from merged near-duplicates): b98a77fe adds symmetric/asymmetric distinction; dd9e0328 adds CPU-edge concrete pick. Also folds 29445866, a DS-voice restatement of the same PTQ-vs-QAT compare. Also folds quant-tooling-pytorch-vs-tflite (PyTorch-vs-TFLite tooling differences) as a practical tooling note within the PTQ/QAT comparison answer.

Overview: Goal is to update an ML model that requires (1) vendor API changes, (2) feature-store migration, and (3) client-side UI updates. Below is a dependency map with tasks, estimated lead times by team, handoff points, and top risks + mitigations.

Dependency map (sequence + parallelizable work):

1. Requirements & spec (ML Engineer, Product, Legal) — 1 week

   • Deliverable: API contract, feature list, UI acceptance criteria
   • Handoff: spec to Vendor, Data Eng, Backend, Frontend

2. Vendor API updates (Vendor + Vendor/Legal review) — vendor: 3–6 weeks; legal: 2 weeks parallel

   • Tasks: API schema change, auth updates, SLAs
   • Handoff: API contract + sandbox to Backend and ML for integration tests

3. Feature-store migration (Data Engineering) — 3–4 weeks

   • Tasks: schema mapping, ETL jobs, data validation, backfill, recalculation of features
   • Handoff: validated feature dataset and sample queries to ML and Backend

4. Backend changes (Backend) — 2–3 weeks (starts after API sandbox & feature schema available)

   • Tasks: API client update, model-serving integration, feature fetch endpoints, telemetry
   • Handoff: staging environment + integration tests to ML and Frontend

5. Model retrain & validation (ML Engineer/Data Scientist) — 1–2 weeks (depends on backfill completeness)

   • Tasks: retrain, evaluate, fairness tests, canary config
   • Handoff: model artifact + inference contract to Backend

6. Frontend UI changes (Frontend) — 1–2 weeks (can start on mockups earlier)

   • Tasks: UI/UX updates, client-side calls to updated backend, feature flags
   • Handoff: QA & rollout plan to Product and Backend

7. End-to-end testing & rollout (All teams) — 1 week

   • Tasks: integration tests, canary, monitoring, rollback plan

Estimated lead times summary:

• Data Engineering: 3–4 weeks (migration + backfill)
• Backend: 2–3 weeks
• Frontend: 1–2 weeks
• Vendor: 3–6 weeks
• Legal: 1–2 weeks (reviews/contract)

Top 3 risks & mitigations:

1. Vendor delay / breaking contract

   • Mitigation: early contract + sandbox access, define backward-compatible API, contingency plan (adapter layer) to translate old/new fields.

2. Incomplete backfill or feature mismatch causing model drift

   • Mitigation: run validation suite, synthetic backfills, incremental rollouts, hold model retrain until data parity verified.

3. Integration bugs causing production regressions (UX or inference latency)

   • Mitigation: feature flags + canary deployments, performance SLA tests, observability dashboards and automatic rollback triggers.

Notes: Use clear acceptance criteria for each handoff, maintain a shared task board, and schedule weekly cross-team syncs led by PM to keep dependencies visible.

Direct answer

Designing a streaming pipeline to compute a rolling metric with meaningful late arrivals means combining event-time windowing with an explicitly chosen watermark and allowed-lateness strategy, an exactly-once (or safely idempotent) write path to the serving store, and a defined policy for how a late correction gets reflected once it's already been served.

Structured elaboration

The pipeline shape is: ingest from a durable, replayable source (a commit-log platform); window the aggregation by event time (tumbling for a clean per-period number, sliding if the metric needs to update more often than its own window length); choose a watermark and allowed-lateness setting sized against your real observed lateness distribution, not guessed; write results via an idempotent upsert keyed by the window's identity so a late-triggered recomputation of an already-served window overwrites cleanly rather than double-counting; and decide, deliberately, whether a correction after the fact re-fires and overwrites the previously served value (accepting that consumers see a number change) or is captured separately for reconciliation without disturbing what was already shown.

Worked example

Computing daily active users where up to 10% of events can arrive as much as 24 hours late: ingest from Kafka, window by calendar day (event time), set the watermark's allowed lateness wide enough to cover the bulk of that 24-hour tail (accepting that the very last, rarest late arrivals beyond that are handled by a separate side-output-and-reconciliation path rather than blocking the main pipeline indefinitely), and write the daily count to the serving store as an idempotent upsert keyed by (date), so a late-triggered recompute of yesterday's count safely overwrites the earlier, less-complete number rather than creating a duplicate entry. At 1 million events per minute, the state backend and checkpoint strategy need to be sized to hold roughly a day's worth of in-flight, not-yet-closed windows, which is the concrete capacity-planning number this design has to account for.

Trade-offs and pitfalls

A wider allowed-lateness setting sized to catch "most" late data delays every window's first result and grows in-flight state accordingly; the residual small fraction of even-later data that misses even a generous allowed-lateness window still needs an explicit, deliberate policy (silently drop it, or reconcile it later), not an accidental gap nobody decided on. The most common mistake in this exact design pattern is building the happy-path windowing and forgetting to design the idempotent-write and late-correction behavior with the same rigor, leaving a pipeline that looks correct in a demo but silently double-counts or drops data the first time a real late-arrival burst occurs in production.

Noisy labels in the test set bias evaluation by systematically mis-measuring model accuracy, precision/recall, calibration, and selection decisions. If some true positives are mislabeled as negatives (or vice versa), observed metrics can under- or over-estimate real performance and distort comparisons between models (a model that matches the noise pattern may appear better).

Key intuition and math:

• Let y be true label, y' observed noisy label. If the noise channel P(y'|y) = C (a confusion matrix) is known or estimable, the relation between observed class-prob estimates p_obs and true class probabilities p_true is p_obs = C p_true, so p_true = C^{-1} p_obs (when C invertible)

Worked 2-class example: suppose the labeler's true positive rate is 95% (P(observed=pos | true=pos)=0.95, so 5% of true positives get mislabeled negative) and true negative rate is 90% (P(observed=neg | true=neg)=0.90, so 10% of true negatives get mislabeled positive). The noise channel matrix is C = [[0.95, 0.10], [0.05, 0.90]] (columns are true classes pos/neg, rows are observed classes pos/neg). If the observed test-set label distribution is p_obs = [0.4, 0.6] (40% observed positive, 60% observed negative), inverting C (determinant = 0.950.90 - 0.100.05 = 0.85; C^{-1} = (1/0.85)*[[0.90, -0.10], [-0.05, 0.95]]) and applying it to p_obs gives p_true = C^{-1} p_obs = [0.353, 0.647]. So the corrected true positive rate is about 35.3%, noticeably lower than the raw observed 40%: because the 90% true-negative rate still lets 10% of the (much larger) negative class leak into the observed-positive bucket, that leakage inflates the observed positive rate above the true one, and the correction pulls it back down.. For scalar metrics like accuracy: acc_obs = sum_y P(y) P(y' = y | model, y): correcting requires modeling C.

Approaches to estimate true performance

1. Relabeling / human audit

• Sample a stratified subset of test examples for re-annotation by experts (or multiple annotators) and compute corrected metrics on that gold subset; extrapolate using importance weighting.
• Use consensus/majority or adjudication to create high-quality labels.

2. Noise-aware estimators (no extra labels)

• Confusion-matrix correction: estimate C from noisy training labels (or small clean set) and apply inverse correction to predicted probabilities or to metric counts.
• Probabilistic label models / EM: jointly estimate true labels and model performance (latent y). Example: treat true y as latent and run EM to estimate P(y|y',x) then compute expected metrics.
• Multiple-annotator models / Dawid-Skene: jointly estimates each annotator's per-class error rates (their own confusion matrix) and the most likely true label for each example, by alternating between the two with EM: given current guesses of the true labels, re-estimate each annotator's confusion matrix from where they agreed or disagreed with those guesses, then given the updated confusion matrices, re-estimate the most likely true label for each example, rather than assuming every annotator is equally reliable or relying on simple majority vote.

3. Robust validation techniques

• Use calibration- and uncertainty-aware metrics (expected calibration error under label noise tends to be distorted; use probabilistic corrections).
• Rank-based metrics (ROC AUC) can be less sensitive to symmetric noise; evaluate metrics that marginalize over label noise when possible.

Cleaning or accounting strategies during evaluation

• Active relabeling: prioritize auditing examples where model disagrees with label or where label uncertainty is high (maximizes information).
• Weighted estimators: if audit subset is nonuniform, use importance weights to correct population estimates.
• Use abstention: allow model to abstain on low-confidence cases and evaluate conditional performance on high-confidence subset (report both conditional and coverage-aware metrics).
• Report uncertainty: bootstrap corrected metrics to give confidence intervals that reflect label-noise uncertainty.
• Sensitivity analysis: report metric ranges under plausible noise rates (e.g., vary off-diagonal entries of C).

Practical caveats

• Corrections require assumptions (stationary noise, identical noise across train/test, or known annotator behavior). If assumptions fail, audits or multiple independent annotators are more reliable.
• Small audit sets can still leave large uncertainty: always report intervals and sensitivity analyses.

Example quick workflow

1. Audit 1–5% stratified test set with expert labels.
2. Estimate confusion matrix C and annotator reliabilities.
3. Apply C^{-1} correction to model probabilities and recompute metrics; report corrected metrics with bootstrap CIs and sensitivity bounds.
4. If resources allow, prioritize relabeling examples where model and label disagree to reduce bias efficiently.

Direct answer: For a delivery ETA or driver-acceptance model, the strongest features span four categories that each capture a different kind of predictive signal: static spatial context, dynamic real-time conditions, temporal patterns, and system/operational load, with the choice of online-versus-feature-store computation for each driven mainly by how fast the signal changes and how tightly it's coupled to the specific request.

Structured elaboration: Ten features, each tagged with computation location and required freshness:

1. Route distance (pickup to dropoff, haversine/geodesic) - feature store, updates whenever the route or coordinates change (effectively static per order).
2. Historical driver acceptance rate (rolling 30-day average) - feature store, daily batch refresh.
3. Restaurant/store historical prep-time mean and variance - feature store, daily or hourly batch refresh.
4. Time-of-day and day-of-week bucket - feature store, precomputed, static per timestamp.
5. Holiday/local-event flag - feature store, refreshed whenever the events calendar changes (infrequent).
6. Driver idle time since last active timestamp - must be computed online, since it depends on the driver's state seconds before dispatch.
7. Current live traffic conditions on the route - must be computed online (or read from a near-real-time cache), since a feature-store-materialized version would already be stale.
8. Current order queue depth in the area (orders assigned but not yet picked up) - online/near-real-time, refreshed on the order of seconds; too volatile for batch materialization.
9. Driver's current concurrent-order load - online, read from live dispatch state at request time.
10. Third-party routing API's live ETA estimate - online, called at request time, with the fallback and latency handling described below.

Worked example: SQL against a candidate-window schema, computing three of the above directly for a 5-minute dispatch window ending at :candidate_window_end and starting at :candidate_window_start:

-- Feature 6: driver idle time (seconds) as of window end
SELECT driver_id, MAX(event_ts) AS last_active_ts
FROM driver_events
WHERE event_ts <= :candidate_window_end
GROUP BY driver_id;
-- idle_seconds = candidate_window_end - last_active_ts, computed in the application layer;
-- a driver with no rows at all (brand-new driver) returns no row here and must fall back
-- to a cohort-level default idle estimate rather than a fabricated zero.

-- Feature 2: driver historical acceptance rate, computed strictly BEFORE the window starts
-- to avoid leaking the very order being scored into its own feature
SELECT driver_id,
       AVG(CAST(accepted AS FLOAT)) AS acceptance_rate,
       COUNT(*) AS n_orders
FROM driver_orders
WHERE order_ts < :candidate_window_start
GROUP BY driver_id;

-- Feature 8: recent order volume in the area within the 5-minute candidate window
SELECT area_id, COUNT(*) AS order_count
FROM area_orders
WHERE order_ts >= :candidate_window_start AND order_ts < :candidate_window_end
GROUP BY area_id;

These three queries were run against a seeded in-memory schema (sqlite3) to confirm the logic: the idle-time query correctly returns only drivers with at least one event, leaving a brand-new driver absent (forcing the explicit cold-start fallback rather than a silently-wrong zero); the acceptance-rate query's strict < bound confirms it never includes an order from inside the candidate window itself; and the area-volume query's half-open [start, end) bound avoids double-counting an order that falls exactly on a window boundary.

Blending the third-party routing API's ETA into the feature set: call it with a tight timeout (e.g. 150ms) tuned to the model's overall latency budget; on timeout or an error response, fall back to an internally-computed baseline ETA (distance / historical average speed for the route, adjusted by current traffic feature); and always include a boolean routing_api_used flag alongside the blended ETA value, so the model can learn a different weighting for "API-backed estimate" versus "internal fallback estimate" rather than treating a fallback value as if it carried the same reliability as a live API response. A hard dependency on the external API's uptime, with no fallback, would make the whole feature computation as unreliable as its least-reliable dependency.

Trade-offs and pitfalls: The online-versus-feature-store split is not a one-time decision; a feature that starts as "compute online, too volatile for the store" can become feature-store-friendly if its update cadence requirement relaxes, or vice versa if a new use case suddenly needs sub-second freshness from a feature that used to be fine at hourly granularity. A second pitfall specific to the third-party API: blending its ETA without an explicit "was this a real API response or a fallback" flag lets a spike in fallback usage (e.g. during a provider outage) silently degrade model accuracy in a way that's invisible unless that flag is tracked and monitored.

Direct answer
Numerical gradient checking validates a hand-derived backward pass by comparing it against an independent, purely numerical estimate of the same gradient (central finite differences), and a large mismatch is strong evidence of a real bug in the analytic derivation or its implementation.

Structured elaboration
For each parameter $\theta_i$ in a flattened parameter vector, the central-difference estimate is $\left(f(\theta+\epsilon e_i) - f(\theta-\epsilon e_i)\right)/(2\epsilon)$, which has $O(\epsilon^2)$ error (versus $O(\epsilon)$ for a one-sided difference), making it the right default choice for this kind of check.

import numpy as np

def grad_check(theta, loss_fn, analytic_grad_fn, eps=1e-5):
    """theta: flattened 1D array of all parameters.
    loss_fn(theta) -> scalar. analytic_grad_fn(theta) -> array same shape as theta.
    Returns (max_abs_diff, numeric_grad, analytic_grad, per-element diff)."""
    theta = theta.astype(float)
    num_grad = np.zeros_like(theta)
    for i in range(len(theta)):
        old = theta[i]
        theta[i] = old + eps; f_plus = loss_fn(theta)
        theta[i] = old - eps; f_minus = loss_fn(theta)
        num_grad[i] = (f_plus - f_minus) / (2 * eps)
        theta[i] = old
    ana_grad = analytic_grad_fn(theta)
    diff = np.abs(num_grad - ana_grad)
    return diff.max(), num_grad, ana_grad, diff

Choosing epsilon: too large biases the estimate away from the true local derivative (the finite-difference approximation itself becomes inaccurate over a larger step); too small suffers from floating-point catastrophic cancellation, since $f(\theta+\epsilon)$ and $f(\theta-\epsilon)$ become nearly indistinguishable in floating point and their difference loses precision. A value around $10^{-5}$ to $10^{-4}$ is a common, reasonable default for double-precision computation.

Worked example
Tested on a real 1-hidden-layer network (input size 3, hidden size 4, output size 2, ReLU hidden activation, squared-error loss) with random weights: the checker's max absolute difference between the analytic and numerical gradients across all 26 parameters was on the order of $10^{-10}$, confirming the correct analytic gradient. To confirm the checker is not merely reporting a small number by coincidence, I then deliberately introduced a bug (doubling one gradient component before returning it) and re-ran the check: the max absolute difference jumped to about $0.026$, more than eight orders of magnitude larger, cleanly and unambiguously flagging the injected bug.

Trade-offs & pitfalls
This technique is $O(P\times C)$ where $P$ is the number of parameters and $C$ is the cost of one loss evaluation, since it evaluates the loss twice PER PARAMETER; this makes it far too slow to run on a full production-sized model, but it is exactly the right tool for validating a small hand-implemented layer or loss function before trusting it at scale. A common pitfall specific to real networks is running this check with dropout or other training-time randomness still active; any source of non-determinism between the two loss evaluations at $\theta+\epsilon$ and $\theta-\epsilon$ will corrupt the numerical estimate, so the model should be switched to a deterministic (evaluation) mode, or the random seed fixed identically, for the duration of the check.