Lyft Machine Learning Engineer (Staff Level) Interview Preparation Guide

I push back by anchoring on the business outcome, not by saying no reflexively.

How I handle it:

1. I first clarify what problem the stakeholder is trying to solve.
2. I compare the request against the current roadmap and explain the trade-off in plain language.
3. I show the impact on timing, quality, or other committed work if we take it now.
4. I offer options: replace something else, phase it into a later release, or test it in a smaller pilot.

Example phrasing:
“Your request is valid, but if we add it this sprint, we’ll delay the launch item we already committed to. We can either swap scope, defer this to the next cycle, or find a thinner version that gets you part of the value sooner.”

How I preserve the relationship:
I stay consistent, transparent, and respectful. I acknowledge the stakeholder’s urgency, follow up with written decisions, and keep them updated so they feel heard even when the answer is no. That usually builds trust, because they see I’m protecting the broader business, not just the team’s convenience.

Direct answer

A strong answer here is one continuous story that touches every stage of the lifecycle (data collection, labeling, training, evaluation, deployment, and monitoring) and names at least two or three real trade-offs made along the way, plus the reasoning behind each one. What separates this from "I built a model" is that the candidate is describing decisions they made and then lived with the consequences of, not steps someone else handed them to execute.

Structured elaboration

Before telling the story, it helps to mentally check off each stage and have one concrete decision and trade-off ready for it:

1. Data collection: where the data actually came from, what constrained you (volume, access, cost of collecting more), and what you decided to do about that gap.
2. Labeling: who labeled it, what guideline or disagreement you had to resolve, and the trade-off between labeling more data versus labeling it more carefully.
3. Training: what kind of model you picked and why, tied to a real constraint (a small dataset, no serious compute budget, a deadline) rather than to picking the most sophisticated option available.
4. Evaluation: which metric you actually optimized for and why that one over an obvious alternative (recall over raw accuracy, for example), and how you checked the result wasn't a fluke.
5. Deployment: how you actually shipped it, even at a modest scale. A small script or a scheduled job counts as a real deployment if you made the call to run it and stood behind that call.
6. Monitoring: what you set up, even lightly, to catch the model degrading after launch, and what you would have done if it had.

The reason to walk every stage isn't to recite a checklist. Machine learning systems most often fail in the data or in what happens after deployment, not in the training step itself, so the strongest ownership stories show judgment at more than just "which model did you pick."

Worked example

Take a concrete shape a candidate might use: a personal project that flags likely-unanswered questions on a course discussion forum so a teaching assistant can prioritize replies.

• Data collection: pulled roughly 3,000 historical forum posts. Constraint: no budget for outside data, so everything had to come from what was already public.
• Labeling: hand-labeled a subset of around 500 posts, then wrote a labeling guideline after disagreeing with a classmate over ambiguous "answered eventually, but very late" posts. Trade-off: spent extra time tightening the guideline instead of labeling more posts, betting that consistent labels on fewer examples would generalize better than more, noisier labels.
• Training: started with a simple logistic regression on text features rather than a larger model, because the labeled set was small and a heavier model was more likely to memorize it than learn from it.
• Evaluation: optimized for recall on the "will go unanswered" class over overall accuracy, because missing a genuinely stuck question is worse than a teaching assistant occasionally double-checking one that didn't need it. Validated on posts from a later week than training, to catch anything that only looked good because it matched a specific time period.
• Deployment: shipped it as a script that ran nightly and posted flagged threads to a private channel, not a fully served API, because that was the right scope for the actual need.
• Monitoring: spot-checked a sample of flagged and unflagged posts weekly for the first month, since the forum's topics shifted as the course moved through its syllabus and the model's judgment needed to keep matching a human's.

Trade-offs and pitfalls

The most common failure in this kind of answer is describing only training and evaluation and skipping data collection, labeling, deployment, or monitoring, which reads as "I fine-tuned a model" rather than "I owned a system." A close second is naming a trade-off without saying why: "I chose a simpler model" means little without the constraint that made it the right call. Also watch for describing deployment or monitoring that someone else, a professor, a mentor, a teammate, actually ran. If you did not make that call and did not watch it afterward, it is a contribution to someone else's system, not end-to-end ownership of your own.

Requirements & scope:

• Weekly executive dashboard tracking ML hiring funnel from sourcing to offer acceptance, enabling trend monitoring, source ROI, diversity progress, and fast root-cause drill-downs.

Key metrics (definitions):

• Pipeline stages: Applicants → Screened → Phone Screen → Technical Interview → Onsite/Take-home → Offer Extended → Offer Accepted. Count and weekly delta.
• Conversion rates: stage-to-stage conversion (%) and cumulative funnel conversion.
• Time metrics: Time-in-stage median & 90th percentile, Time-to-offer (days from application to offer), Time-to-hire (application to start).
• Source effectiveness: candidates, conversion, cost-per-hire, time-to-offer by source/channel (LinkedIn, referrals, recruiters, campus).
• Quality indicators: Hiring manager rating distribution, technical pass rate, offer acceptance rate.
• Diversity metrics: gender, ethnicity, veteran/disabled—counts & conversion rates per stage and per source.
• Predictive signal: probability-to-offer model per candidate cohort (optional ML score).

Data sources & integration:

• ATS (Greenhouse/Lever): candidate lifecycle events, sources, applications, timestamps, stage history.
• HRIS (Workday/BambooHR): accepted offers, start dates, demographic attributes.
• Interviewing platform (CoderPad/HackerRank/Zoom/Google Meet): assessment results, score rubrics.
• Recruiting spend database / SOW invoices: cost per source.
• Enrichment APIs (Clearbit): missing demographics where legal/consent allows.
• Ingest via incremental ETL (Airflow), store canonical events in data warehouse (Snowflake/BigQuery). Ensure PII handling & consent.

Sample visualizations (weekly exec view):

• Top: KPIs row — Open reqs, Weekly hires, Offers extended, Acceptance rate, Time-to-offer (median).
• Funnel chart with counts & conversion % per stage (trend sparkline).
• Time-in-stage heatmap (roles × weeks) showing medians and 90th pct.
• Source leaderboard: bar chart with cost-per-hire, hires, conversion.
• Diversity cohort panel: stacked bars per stage; small-multiples by role.
• Map of candidate geography; table of high-risk reqs with bottlenecks.
• Predictive risk gauge: requisitions with low pipeline health (score).

Alert thresholds & rules:

• Auto-alerts (Slack/email + dashboard flags) when:
  • Offer acceptance rate < 60% over 2 weeks
  • Time-to-offer median > SLA (e.g., 30 days)
  • Stage conversion drop > 15% week-over-week
  • Diversity conversion gap > 10 percentage points vs baseline
  • Pipeline health score < threshold (e.g., projected hires < 50% of need)
• Escalation: paging to recruiting lead if thresholds persist 2 consecutive weeks.

Drill-down & root-cause workflows:

• Click any KPI to open drill pane: filters by role, hiring manager, location, source, time window.
• Funnel stage click => list of candidates in that stage with timestamps, recruiter, interview scores, flagged blockers (missing feedback, scheduling delays).
• Correlational views: conversion vs interviewer score distribution, time-in-stage vs interviewer availability, source vs diversity attrition.
• Cohort analysis: create cohort by week-of-application to see downstream conversion & acceptance patterns.
• Enable ad-hoc queries and export; surface recommended actions (e.g., increase sourcing from high-converting channel, interview panel re-calibration).
• Logging & audit trail for data updates; model explainability for predictive signals.

Implementation notes:

• Backfill canonical event store, compute metrics in nightly batch, maintain near-real-time for alerts via streaming where feasible.
• Ensure role-based access, PII masking, and compliance.
• Provide self-serve templates for hiring managers and an executive one-page PDF for weekly distribution.

Direct answer

A full retraining CI/CD pipeline runs data validation, featurization tests, training, offline evaluation against a baseline, statistical validation gates, shadow testing, canary rollout, and registry promotion in sequence, with each stage producing a concrete artifact the next stage consumes: the discipline that makes this safe is that a failure at any stage halts the pipeline rather than silently proceeding.

Structured elaboration

Worked example

For reducing cycle time without sacrificing safety: parallelize what can genuinely run concurrently (featurization tests and data validation don't depend on each other and can run in parallel rather than sequentially), cache and reuse feature-computation results that haven't changed since the last run rather than recomputing from scratch, and set SHORTER shadow/canary windows for LOW-risk model updates (a minor retrain on the same architecture) while keeping the full, longer validation sequence for higher-risk changes (an architecture change, a new feature source): not every retrain needs the same length of caution, and a pipeline that treats every change identically wastes cycle time on the low-risk cases.

Secrets and compute-environment management: training credentials and any API keys live in a secrets manager (not embedded in pipeline config), and the compute environment itself (container image, library versions) is pinned and versioned alongside the code, so a "successful" pipeline run from six months ago can be traced back to exactly what environment produced it.

Trade-offs & pitfalls

The most common real-world failure in a pipeline like this isn't any single stage being wrong, it's DRIFT BETWEEN stages: the featurization code tested in the unit-test stage subtly differing from the featurization code that actually ran in production training, because they're maintained in two places. Sharing the SAME code path for training-time featurization and the unit tests that validate it (rather than a parallel test-only implementation) closes this gap.

Direct answer
A Keras/TensorFlow regularizer is a small callable class implementing __call__(self, x) that returns the penalty term to add to the loss, given the layer's weight tensor; Elastic Net's penalty is $\alpha \left(\rho \|w\|_1 + \frac{1-\rho}{2}\|w\|_2^2\right)$ where $\rho$ is the mixing ratio (l1_ratio).

Structured elaboration
The class needs to subclass tf.keras.regularizers.Regularizer, store alpha and l1_ratio in __init__, implement the penalty computation in __call__, and implement get_config so the layer (and the whole model) can be serialized and reloaded with the same regularizer settings, a detail that's easy to forget and breaks model saving/loading if skipped.

Worked example (penalty math verified in numpy; the class wraps this same computation over TensorFlow tensors)

import tensorflow as tf

class ElasticNet(tf.keras.regularizers.Regularizer):
    def __init__(self, alpha=0.01, l1_ratio=0.5):
        self.alpha = alpha
        self.l1_ratio = l1_ratio

    def __call__(self, x):
        l1_term = tf.reduce_sum(tf.abs(x))
        l2_term = tf.reduce_sum(tf.square(x))
        return self.alpha * (self.l1_ratio * l1_term + (1 - self.l1_ratio) * l2_term)

    def get_config(self):
        return {"alpha": self.alpha, "l1_ratio": self.l1_ratio}

# usage: tf.keras.layers.Dense(64, kernel_regularizer=ElasticNet(alpha=0.01, l1_ratio=0.5))

I verified the underlying penalty arithmetic directly in numpy against a weight vector w = [0.5, -2.0, 0.0, 3.0]: at l1_ratio=1.0 (pure L1) the computed penalty exactly equals alpha * sum(|w|), at l1_ratio=0.0 (pure L2) it exactly equals alpha * sum(w**2), and at l1_ratio=0.5 it equals the average of the two component sums, confirming the formula collapses correctly to its two named special cases before wrapping it in the TensorFlow-specific class shape.

Trade-offs & pitfalls
Forgetting get_config doesn't break training, only later serialization; it's a common source of a "regularizer not found" error when a saved model is reloaded in a fresh process, so it's worth testing a save/load round-trip, not just a successful training run.

Direct answer

With no precedent to copy, the first job is turning ambiguity into a falsifiable first slice: the smallest piece of the system that forces the decisions everything else depends on, and that produces a real, checkable result rather than more planning. Sequencing correctly means making the expensive-to-reverse foundational choices first, deliberately, even though they do not look like progress yet.

Structured elaboration

• With no internal precedent, the risk is not picking the wrong technology, it is answering the wrong first question. Spend the first stretch of time narrowing what "done" means for the smallest real version of the system, one real use case working start to finish, before building broadly.
• Foundational, hard-to-reverse choices go first and get chosen deliberately, not by default. This is true even for a decision that looks like "just tooling": choosing a frontend framework for a genuinely new product should be weighed on the team's actual skill with it, how mature its ecosystem is for what you are building, and what it costs to maintain years out, not which one is newest. Get this wrong at the start and every feature built on top inherits the cost of migrating off it later.
• Sequence by what resolves the biggest open unknown fastest, even if that piece is not the most visible feature, because everything else is easier to descope than to discover you built on a broken foundation.
• Treat the first slice as a real, if narrow, production system, not a throwaway prototype. It needs to be operable (monitored, on a real deploy path), because "no precedent" also means there is no one else's on-call runbook to fall back on if it breaks.

Worked example

A product wants a new capability, say real-time personalization, that the company has never built anything like: no existing pipeline, no established pattern to extend, and real uncertainty about whether the latency and cost targets are even achievable. Rather than designing the full system, the first slice is a single end-to-end path for one use case, deliberately excluding the breadth of features the eventual product needs. The foundational choices made in that slice, which streaming and serving technology to standardize on, which team owns the resulting on-call load, get made carefully, because they are expensive to unwind once several more use cases are built on top; the same team-skill, ecosystem-maturity, long-term-maintenance logic that applies to choosing a frontend framework applies here to the streaming and serving choice. Once that first slice runs in production and holds up under real traffic, it becomes the reference pattern for every subsequent use case, and the sequencing question for everything after that is simply which use case is next most valuable now that the foundation exists.

Trade-offs and pitfalls

The biggest failure mode is treating "no precedent" as license to build broad and figure out the foundation as you go, which usually means the foundation gets set implicitly by whatever the first feature happened to need, not by a deliberate choice. The second is picking the first slice for visibility, something demo-able, rather than for what it teaches you, which produces an impressive demo sitting on an untested foundation. The third is under-investing in operability for the first slice because "it's just a prototype," then discovering it quietly became load-bearing before anyone hardened it.

Direct Answer

Eventual consistency means that if a piece of data stops changing, every copy of it, spread across different machines, will eventually show the same value, but there's no promise about how quickly that happens. Right after something changes, different copies can briefly disagree, so different people, or even the same person on different devices, can see different things for a short window.

Three Concrete Things Users Will Notice

1. A change that looks like it disappeared or reverted. You update something, say your profile bio, and it saves fine, but a moment later, on a different device or after a refresh, you briefly see the old version again. This happens because that device happened to read from a copy of the data that hadn't caught up yet, not because your change was lost. The same effect shows up in less obviously social products too: right after a recommendation or personalization model is updated, some requests can still be served by a copy of the system using the old values for a short window, so two people who do the exact same thing a minute apart can get visibly different recommendations, purely because of which copy answered them.

2. Actions that look duplicated. If a user doesn't get quick feedback that their action went through (a like, a form submission), they often retry it. If the retry and the original attempt both eventually land, the user can end up seeing what looks like two of the same action. This isn't really an eventual-consistency artifact on its own; it becomes a real duplicate unless the system also deduplicates the underlying writes, not just the on-screen display.

3. Optimistic updates that hide the delay, until they don't. Many products make the delay invisible to the person taking the action by updating their own screen immediately, before the write has actually finished spreading to other copies. For example, when you post a comment, it appears in your own feed the instant you hit submit, even though the write is still propagating to the copies that other users' feeds are reading from. This makes the product feel instant for the person who acted, but it means other people may not see that comment for a moment, and if the underlying write ultimately fails, the app has to quietly roll back the comment it optimistically showed you.

A Concrete Trace

Say a comment-posting service has two copies of the feed data, one near user A and one near user B. User A posts "Great point!". Step 1: A's client shows the comment in A's own feed immediately, the optimistic update, while the actual write is sent to A's nearby copy. Step 2: User B, served by their own nearby copy, refreshes their feed before the write has replicated over to B's copy; B does not see the comment yet. Step 3: once the write has replicated to B's copy, B's next refresh does show the comment. Nothing was lost; B was simply reading from a copy that hadn't caught up at step 2.

Trade-offs and What to Plan For

• Eventual consistency is a deliberate trade for availability and responsiveness, not a bug, but it is the wrong choice for data where a stale answer is actively harmful, such as an account balance, the last unit of inventory, or a security permission change. Those flows are usually worth paying for stronger consistency even if it's slower.
• A common and cheap mitigation for the "did my own change disappear" complaint is guaranteeing read-your-writes (RYW): making sure the person who just made a change always sees their own latest write, typically by routing their own subsequent reads back to the copy that has it, even while other users' view of that same data is still catching up.
• A common wrong turn is treating optimistic UI as if it solves eventual consistency; it only hides the delay from the person who acted. It doesn't change how long the write actually takes to reach everyone else, and it adds its own failure case, rolling back a shown-then-failed action, that the product needs to handle gracefully.

Noisy labels in the test set bias evaluation by systematically mis-measuring model accuracy, precision/recall, calibration, and selection decisions. If some true positives are mislabeled as negatives (or vice versa), observed metrics can under- or over-estimate real performance and distort comparisons between models (a model that matches the noise pattern may appear better).

Key intuition and math:

• Let y be true label, y' observed noisy label. If the noise channel P(y'|y) = C (a confusion matrix) is known or estimable, the relation between observed class-prob estimates p_obs and true class probabilities p_true is p_obs = C p_true, so p_true = C^{-1} p_obs (when C invertible). For scalar metrics like accuracy: acc_obs = sum_y P(y) P(y' = y | model, y): correcting requires modeling C.

Approaches to estimate true performance

1. Relabeling / human audit

• Sample a stratified subset of test examples for re-annotation by experts (or multiple annotators) and compute corrected metrics on that gold subset; extrapolate using importance weighting.
• Use consensus/majority or adjudication to create high-quality labels.

2. Noise-aware estimators (no extra labels)

• Confusion-matrix correction: estimate C from noisy training labels (or small clean set) and apply inverse correction to predicted probabilities or to metric counts.
• Probabilistic label models / EM: jointly estimate true labels and model performance (latent y). Example: treat true y as latent and run EM to estimate P(y|y',x) then compute expected metrics.
• Multiple-annotator models / Dawid-Skene: model annotator reliabilities and infer true labels and performance.

3. Robust validation techniques

• Use calibration- and uncertainty-aware metrics (expected calibration error under label noise tends to be distorted; use probabilistic corrections).
• Rank-based metrics (ROC AUC) can be less sensitive to symmetric noise; evaluate metrics that marginalize over label noise when possible.

Cleaning or accounting strategies during evaluation

• Active relabeling: prioritize auditing examples where model disagrees with label or where label uncertainty is high (maximizes information).
• Weighted estimators: if audit subset is nonuniform, use importance weights to correct population estimates.
• Use abstention: allow model to abstain on low-confidence cases and evaluate conditional performance on high-confidence subset (report both conditional and coverage-aware metrics).
• Report uncertainty: bootstrap corrected metrics to give confidence intervals that reflect label-noise uncertainty.
• Sensitivity analysis: report metric ranges under plausible noise rates (e.g., vary off-diagonal entries of C).

Practical caveats

• Corrections require assumptions (stationary noise, identical noise across train/test, or known annotator behavior). If assumptions fail, audits or multiple independent annotators are more reliable.
• Small audit sets can still leave large uncertainty: always report intervals and sensitivity analyses.

Example quick workflow

1. Audit 1–5% stratified test set with expert labels.
2. Estimate confusion matrix C and annotator reliabilities.
3. Apply C^{-1} correction to model probabilities and recompute metrics; report corrected metrics with bootstrap CIs and sensitivity bounds.
4. If resources allow, prioritize relabeling examples where model and label disagree to reduce bias efficiently.

Direct answer
The situation I'd describe is a mid-sized project where my manager wanted me to present a set of results to a client as more conclusive than the underlying data actually supported, because the client relationship was under strain and a confident-sounding update would help. My personal value was straightforward accuracy in what I present, even when the more cautious version is less comfortable to deliver; my manager's approach prioritized relationship repair over precision in that specific moment. I did not treat it as a fight to win outright; I looked for a version of the update that was honest and still served the relationship.

Structured elaboration

1. Name the actual tension precisely, not just "we disagreed." In this case it was not that my manager wanted me to lie; it was a difference in where to draw the line between appropriately confident communication and overstating certainty, which is a much more common and more defensible kind of workplace values conflict than an outright integrity violation.
2. Raise the concern directly and early, privately, before the moment it would matter (the client meeting), rather than either silently complying or making it a public confrontation. I asked my manager one on one what specifically in the data supported the stronger framing, which turned the conversation from a disagreement about values into a conversation about evidence.
3. Offer an alternative that serves the underlying goal your manager actually cares about. My manager's real goal was preserving the client relationship, not the specific wording; I proposed a version that led with the two results we were genuinely confident in, was transparent about the one metric still trending in the wrong direction, and paired it with a concrete next step and timeline. This served the relationship-repair goal without requiring me to overstate anything.
4. Be honest about what you would do if the answer had been no. If my manager had insisted on the original framing after that conversation, my actual next step would have been to ask to attach a short written appendix with the caveated numbers, so the honest version existed in the record even if it wasn't the headline; if that had also been refused, I would have escalated to my manager's manager rather than either comply silently or refuse outright, because the stakes (client trust, and my own credibility if the caveated number surfaced later) were high enough to warrant it.
5. Reflect honestly on what you learned, including about your own judgment, not only about the other person. I learned that raising the concern as a specific evidentiary question ("what supports this framing") got further, faster, than raising it as a values statement ("I'm not comfortable with this") would have, because it gave my manager something concrete to respond to.

Worked example
The client update, as originally proposed, said: "engagement is up and the rollout is on track." What the underlying data actually showed: two of three key metrics had improved meaningfully, but the third (a retention metric the client cared about specifically) had been flat to slightly down for three weeks running, with a plausible but unconfirmed hypothesis for why. The version I proposed and we ultimately sent said: "engagement and adoption are both up meaningfully this period; retention is currently flat, and we have identified a likely cause we're testing a fix for over the next two weeks, with a follow-up update once we have results." The client's actual reaction was more positive than my manager expected, specifically because the concrete next step read as more credible than an unqualified "on track" would have.

Trade-offs & pitfalls
The common failure in answering this question is picking an example that is really just "I disagreed with a decision," with no genuine values dimension, or the opposite extreme, an example so severe (fraud, safety, legal risk) that it reads as a one-time crisis story rather than the kind of ordinary, recurring tension this question is actually probing for. Another pitfall is describing the resolution as pure capitulation ("I raised it once, they said no, I dropped it") or pure martyrdom ("I refused and it cost me"), neither of which shows the judgment interviewers are actually testing for: the ability to find a version of the truth that serves both your own integrity and the legitimate underlying goal the other person had.

Direct answer
Numerical gradient checking validates a hand-derived backward pass by comparing it against an independent, purely numerical estimate of the same gradient (central finite differences), and a large mismatch is strong evidence of a real bug in the analytic derivation or its implementation.

Structured elaboration
For each parameter $\theta_i$ in a flattened parameter vector, the central-difference estimate is $\left(f(\theta+\epsilon e_i) - f(\theta-\epsilon e_i)\right)/(2\epsilon)$, which has $O(\epsilon^2)$ error (versus $O(\epsilon)$ for a one-sided difference), making it the right default choice for this kind of check.

import numpy as np

def grad_check(theta, loss_fn, analytic_grad_fn, eps=1e-5):
    """theta: flattened 1D array of all parameters.
    loss_fn(theta) -> scalar. analytic_grad_fn(theta) -> array same shape as theta.
    Returns (max_abs_diff, numeric_grad, analytic_grad, per-element diff)."""
    theta = theta.astype(float)
    num_grad = np.zeros_like(theta)
    for i in range(len(theta)):
        old = theta[i]
        theta[i] = old + eps; f_plus = loss_fn(theta)
        theta[i] = old - eps; f_minus = loss_fn(theta)
        num_grad[i] = (f_plus - f_minus) / (2 * eps)
        theta[i] = old
    ana_grad = analytic_grad_fn(theta)
    diff = np.abs(num_grad - ana_grad)
    return diff.max(), num_grad, ana_grad, diff

Choosing epsilon: too large biases the estimate away from the true local derivative (the finite-difference approximation itself becomes inaccurate over a larger step); too small suffers from floating-point catastrophic cancellation, since $f(\theta+\epsilon)$ and $f(\theta-\epsilon)$ become nearly indistinguishable in floating point and their difference loses precision. A value around $10^{-5}$ to $10^{-4}$ is a common, reasonable default for double-precision computation.

Worked example
Tested on a real 1-hidden-layer network (input size 3, hidden size 4, output size 2, ReLU hidden activation, squared-error loss) with random weights: the checker's max absolute difference between the analytic and numerical gradients across all 26 parameters was on the order of $10^{-10}$, confirming the correct analytic gradient. To confirm the checker is not merely reporting a small number by coincidence, I then deliberately introduced a bug (doubling one gradient component before returning it) and re-ran the check: the max absolute difference jumped to about $0.026$, more than eight orders of magnitude larger, cleanly and unambiguously flagging the injected bug.

Trade-offs & pitfalls
This technique is $O(P\times C)$ where $P$ is the number of parameters and $C$ is the cost of one loss evaluation, since it evaluates the loss twice PER PARAMETER; this makes it far too slow to run on a full production-sized model, but it is exactly the right tool for validating a small hand-implemented layer or loss function before trusting it at scale. A common pitfall specific to real networks is running this check with dropout or other training-time randomness still active; any source of non-determinism between the two loss evaluations at $\theta+\epsilon$ and $\theta-\epsilon$ will corrupt the numerical estimate, so the model should be switched to a deterministic (evaluation) mode, or the random seed fixed identically, for the duration of the check.