Lyft Data Scientist (Staff Level) Interview Preparation Guide

Direct answer

Keep a stalled dependency peer-to-peer as long as direct conversation is still making progress. Escalate when you hit a concrete trigger: a scope change that neither side can unilaterally absorb, genuinely conflicting priorities that only someone with visibility into both roadmaps can arbitrate, or a hard deadline-driven blocker where peer-to-peer conversation has already stalled.

Framework

Default: work it peer-to-peer. Most stalls are under-communication or unclear ownership, and a direct conversation or a short written proposal usually unsticks them without anyone else getting involved.

Concrete triggers to escalate.

• Scope change: the fix now requires work neither team budgeted for, and only a manager can reprioritize that.
• Conflicting priorities: both sides are acting rationally from their own team's goals, and the trade-off needs someone with visibility into both roadmaps to arbitrate.
• Hard blocker with a deadline: a fixed external date is genuinely at risk, and peer-to-peer conversation has already stalled past a reasonable window, for example no movement after two direct attempts over several days.
• Repeated pattern: the same kind of stall keeps recurring with the same team, which means the real issue is the working relationship or process, not this one dependency.

What to bring when you escalate. A short brief: what's blocked, what you've already tried peer-to-peer, the realistic options and their trade-offs, and the specific decision you need.

Worked example (applying the criteria)

Situation: your team's deliverable needs a schema change from another team that they've deprioritized for two weeks despite two direct requests.
Applying the criteria: this isn't just a communication gap, direct conversation was already tried twice with no movement. It's a conflicting-priorities case, the other team's roadmap has no room for this without reprioritizing something else, combined with a hard blocker, a fixed external deadline in three weeks that this schema change sits on the critical path for (meaning if this dependency slips, the final deadline slips by the same amount, unlike a dependency with buffer to absorb delay).
Action: escalated to the shared manager with a one-page brief covering what's blocked, the two peer-to-peer attempts and their outcome, and two options: the other team reprioritizes one sprint of work, or your team ships a temporary workaround with known limitations, along with the deadline risk if neither happens within the week.
Result: the shared manager reprioritized one sprint item, unblocking the schema change with two weeks to spare before the deadline. Both teams also agreed to flag scope-affecting asks earlier next time, so the same dependency doesn't reach this point again.

Trade-offs and pitfalls

• Escalating too early over normal friction burns trust and reads as an inability to work horizontally.
• Escalating too late, repeatedly trying peer-to-peer past the point it's actually working, puts the deadline at real risk and looks like poor judgment in hindsight.
• A vague escalation with no options and no specific ask wastes the leader's time compared with a brief that names the decision needed.

Direct answer

A binary heap is a complete binary tree (every level full except possibly the last, which fills left to right) stored compactly in an array. A min-heap keeps every parent less than or equal to its children, so the root is always the minimum; a max-heap keeps every parent greater than or equal to its children, so the root is always the maximum. Insert and extract-min/max both cost $O(\log n)$ because each only has to fix a single root-to-leaf path, while peek is $O(1)$ since the answer always sits at the root. Reach for a heap over a balanced binary search tree (BST) or a plain hash table specifically when the operation you actually need is "give me the current min or max, repeatedly, while other items keep arriving": a heap does that with a simpler structure and lower constant factors than a full BST, and a hash table can't do it at all without a full scan.

Structured elaboration

Array representation and core operations (0-indexed; for a node at index $i$, its children sit at $2i+1$ and $2i+2$)

• Insert: append the new value at the next free array slot, then "sift up", swapping it with its parent while the heap property is violated. Time $O(\log n)$ (tree height), space $O(1)$ auxiliary.
• Peek: return the root value directly. Time $O(1)$.
• Extract-min/max: read the root, move the last array element into the root position, shrink the array by one, then "sift down" from the root, swapping with the smaller (or larger) child until the heap property holds. Time $O(\log n)$.
• Build-heap (heapify) from an existing array: run sift-down starting from the last non-leaf node back to the root, not one insert at a time. This costs $O(n)$ total, not $O(n \log n)$: most nodes sit near the bottom of the tree and only need a short sift-down, and the sum of sift-down work across all levels is a convergent series bounded by $O(n)$, tighter than treating it as $n$ separate $O(\log n)$ inserts.

Heap versus balanced BST versus hash table

Use a heap when you repeatedly need the current best item and nothing else about ordering; use a BST when you also need range queries, in-order traversal, or predecessor/successor lookups; use a hash table when you only need arbitrary-key existence or lookup and never need the min, max, or any ordering.

Worked example

A concrete use of a min-heap of bounded size $k$ to track the top-$k$ largest values in a stream: push each new value; once the heap holds $k$ items, only replace the root (the current smallest of the kept set) if the new value is larger.

import heapq


def top_k_largest(nums: list[int], k: int) -> list[int]:
    heap: list[int] = []
    for x in nums:
        if len(heap) < k:
            heapq.heappush(heap, x)
        elif x > heap[0]:
            heapq.heapreplace(heap, x)
    return sorted(heap, reverse=True)


if __name__ == "__main__":
    nums = [7, 2, 9, 4, 1, 8, 3, 10, 5, 6]
    print(top_k_largest(nums, 3))

Running this prints [10, 9, 8]: the heap only ever holds the 3 largest values seen so far, each of the other 7 values is checked against the current smallest kept value (the root) in $O(1)$ and, if larger, replaces it in $O(\log k)$, for a total cost of $O(m \log k)$ across $m$ input values.

Trade-offs & pitfalls

• Inserting $n$ elements one at a time into an empty heap costs $O(n \log n)$ total; heapify builds the same final structure in $O(n)$. Both produce a valid heap, only the construction cost differs, and conflating the two is a common mistake.
• A heap does not support efficient arbitrary-key lookup or a "decrease this specific key's priority" operation without extra bookkeeping (an auxiliary map from key to its current array position); this matters for algorithms like Dijkstra's shortest-path algorithm that rely on decrease-key.
• $k$-ary heaps (each node has $k$ children instead of 2) trade a shallower tree, so insert and decrease-key touch fewer levels, for a more expensive sift-down, since each step now compares against $k$ children instead of 2; they help when insert/decrease-key frequency dominates extraction frequency.
• Duplicate keys are allowed by the heap property; the ordering among equal keys is unspecified and shouldn't be relied upon.

Direct answer

I would not pick yet. Three vague KPIs (key performance indicators, the numbers a team watches to judge success) such as "engagement", "growth" and "customer happiness" are labels, not metrics. I would ask what decision the metric must drive, turn each label into a measurable definition, and then choose the one that best reflects the launch goal. The other two are dropped or demoted if they do not fit. Separately I pick guardrail metrics (numbers that must not get worse while we chase the primary one). I justify the pick against explicit criteria, not taste.

Questions I ask, in order

1. What will you do differently if this number goes up, or down? (Ship wider, roll back, fund phase two. If nobody can say, the metric is decoration.)
2. What behaviour is the launch supposed to change, for which users? (Names the action the metric must count.)
3. By when must we know? (A 90-day outcome cannot judge a 2-week launch window.)
4. For each KPI, how is it measured today, from what source, and who trusts it? (Separates a product problem from a measurement problem: if the event is not logged, or two dashboards disagree, a "movement" may just be instrumentation, meaning the tracking code that records user actions as events. A change to that code can make a number jump with no change in user behaviour.)
5. What must not get worse? (Produces the guardrails.)

If the stakeholder says "just be data-driven" and has no metrics at all, I accept proxies (stand-in measures close to the real goal): a count straight from a database table, a support-ticket tag, or a manual review of a small sample of sessions. I label these directional, not conclusive.

Choosing and justifying

Worked example (illustrative numbers)

A launch adds a setup checklist to a team-collaboration app. KPIs offered: "engagement", "growth", "happiness".

• Engagement becomes: share of new accounts that create a first project within 7 days. This is the activation rate, the share of new users who reach a first meaningful action. Baseline 30%, target 36% (both illustrative).
• Growth (signups) is driven by marketing, not by this launch, so it fails the "tied to the goal" test.
• Happiness (survey score) is slow and sparse in a 2-week window, so it fails "sensitive in the window".

Primary: activation rate. Guardrails: support tickets per 100 new accounts, and 7-day return rate (share of new accounts that come back in the week after signup), support tickets per 100 new accounts is readable throughout the launch window, while the 7-day return rate, like the 7-day activation rate, is readable for early signups inside the window and for the last signups only about day 21. 30-day retention (share of new accounts still active a month later) is a longer-horizon guardrail: it cannot be observed inside a 2-week window, so I schedule its read for after the window closes and say so up front. Likewise a 7-day activation rate counts an account only once it is 7 days old, so for a 2-week launch the last signups are readable about day 21, and I tell the stakeholder when each number becomes available.

Trade-offs and pitfalls

• A "comprehensive model" that blends several KPIs into one weighted score (for example 40% engagement plus 30% growth plus 30% happiness added into one number) hides trade-offs and cannot be explained to the stakeholder. I prefer one primary metric with guardrail constraints.
• A primary metric with no guardrail can be gamed (activation rises because the checklist nags users).
• Changing definitions after seeing results. I write the definition and target before launch.
• What would change my pick: if the launch is explicitly a retention play, retention becomes primary.

Direct answer: An Employee Resource Group (ERG) is a voluntary, employee-led group organized around a shared identity or affinity (for example, a Women in Engineering group, a Black employee network, a Parents/Caregivers group, or an LGBTQ+ employee group), providing community, mentorship, and often a channel for feedback to leadership on issues affecting that group; an individual contributor can support one technically (building or maintaining a small internal tool the group needs, like an event sign-up or resource directory) and non-technically (mentoring within the group, showing up to events, or acting as an ally who amplifies the group's asks in spaces the group itself isn't in).

Structured elaboration:

1. What an ERG is and isn't. It's a voluntary community and support structure, not an official decision-making body with authority over HR policy, though healthy organizations do give ERGs a real feedback channel to leadership; it's not a substitute for the company's own accountability on representation and inclusion, and treating it as such (outsourcing DEI work to unpaid volunteer labor) is a common organizational failure mode.
2. Technical support example. Build or maintain a small internal tool the ERG actually needs: an event RSVP page, a searchable mentor-matching directory, or automating a manual process (like tracking event attendance) the group's volunteer organizers were doing by hand. This is real engineering work with a clear, appreciative user base, and a good way for a technical IC to contribute meaningfully without needing to be the group's public face.
3. Non-technical support example. Show up consistently to events (attendance itself signals allyship and gives the group visibility), volunteer as a mentor within the ERG's mentorship track if one exists, or use your own standing to amplify the group's asks in rooms the group's members aren't in (sponsorship: advocacy on someone's behalf, in their absence).
4. Measurable outcome. A concrete signal an IC could track for their own contribution: whether the tool they built is actually used (event sign-ups via the tool versus the old manual process, adoption rate); for mentorship, a simple before/after check-in with the mentee on whether the relationship felt useful, since ERG mentorship impact is best measured qualitatively at the individual level rather than with an invented aggregate statistic an individual contributor wouldn't actually have access to.

Worked example: An engineer notices their company's ERG for parents/caregivers manually tracks event RSVPs via a shared spreadsheet that frequently gets out of sync, causing confusion about headcount for catered events. They build a small internal form-and-dashboard tool over a couple of evenings, cutting RSVP-tracking overhead for the volunteer organizers from roughly an hour per event to a few minutes, and separately volunteer as a mentor for a newer parent navigating return-to-work logistics, checking in with them after two months on whether the mentoring relationship has been useful.

Trade-offs and pitfalls: Be careful not to frame ERG support as a substitute for the company's own investment (budget, paid organizer time, leadership sponsorship); an ERG that runs entirely on unpaid volunteer labor, however well-intentioned the individual contributions, is not a sustainable or fully credible signal of organizational commitment, which connects to the broader question of institutionalizing that work. Also avoid treating ERG participation as something only members of the group's identity should do; ally participation (attending, amplifying, technical support) matters and shouldn't require group membership.

Direct answer

Nested cross-validation is two CV loops in one: an outer loop that estimates how well the whole modeling pipeline generalizes, and an inner loop, run entirely inside each outer training fold, that picks hyperparameters or features. You need it whenever the same data is used both to tune the model and to report its performance, because tuning on the same data you evaluate on leaks information and inflates the reported score.

Structured elaboration

Why plain k-fold CV is not enough here. If you run k-fold CV once to pick the best hyperparameters (by, say, taking the config with the highest mean CV score) and then report that same mean CV score as your generalization estimate, you have used the test folds to make a selection decision, so the reported number is optimistically biased. The gap grows with the size of the hyperparameter search space: the more configurations you try, the more likely one of them fits the validation folds' noise, not just signal.

Outer/inner structure.

• Outer loop ($K_\text{outer}$ folds): each outer fold is held out entirely and touched only once, at the very end, purely for scoring. It never influences any modeling decision.
• Inner loop ($K_\text{inner}$ folds, run on the outer-training portion only): performs the hyperparameter search or feature selection, using its own train/validation splits. Whatever it selects (say, the config with the best mean inner-validation score) is refit on the full outer-training set.
• The refit model is then scored once on the untouched outer-test fold. Averaging that score across all outer folds gives an unbiased estimate of how well "the pipeline, including its tuning procedure" generalizes.
• Any preprocessing that looks at the target (target encoding, feature selection by correlation with $y$, scaling parameters) must be fit only on the current inner-training data, never on the inner-validation or outer-test data, or the same leakage reappears one level down.

Computational cost. Every hyperparameter configuration gets fit $K_\text{inner}$ times per outer fold just to be scored, then the winner gets refit once more on the full outer-training set. Total model fits:

where $G$ is the number of hyperparameter configurations evaluated. This is roughly $K_\text{inner}$ times more expensive than a single k-fold search (precisely $K_\text{inner} + 1/G$), which is why nested CV is usually reserved for the final reported generalization number rather than for every exploratory tuning pass.

Worked example

Suppose $K_\text{outer} = 5$, $K_\text{inner} = 3$, and a grid search over $G = 10$ hyperparameter configurations.

Compare that to a single (non-nested) k-fold grid search used just to pick hyperparameters, at $G \times K = 10 \times 5 = 50$ fits, with no separately reported unbiased generalization estimate. Nested CV costs about $155 / 50 \approx 3.1\times$ more fits here, and the ratio grows directly with $K_\text{inner}$: doubling $K_\text{inner}$ to 6 gives $5 \times (10 \times 6 + 1) = 305$ fits, roughly double, because the inner search dominates the total.

Trade-offs & pitfalls

• Nested CV answers "how good is this tuning procedure, on average," not "what hyperparameters should I ship." The winning configuration can differ across outer folds; for deployment, refit once on the full dataset using the inner-loop procedure (or the single most frequently selected configuration) after nested CV has validated that the procedure is trustworthy.
• Under a tight compute budget, replace grid search with randomized or Bayesian search in the inner loop to shrink $G$ without shrinking the search space explored, or reduce $K_\text{inner}$ to 3 (a common compromise, since the inner loop only needs to rank configurations relatively, not report a final number).
• Skipping the inner loop and just using a single train/val split inside each outer fold is a cheaper approximation, but reintroduces some tuning variance into the outer score; it is a reasonable trade-off for very expensive models, not for cheap ones where full nested CV is affordable.
• The single most common implementation bug: fitting a preprocessing step (scaler, target encoder, feature selector) once on the whole dataset before either loop starts. That silently defeats the entire point of nesting.

Direct answer

For an individual coefficient in a logistic regression, use a Wald test ($z = \hat\beta_j / SE(\hat\beta_j)$) or, more reliably, a likelihood ratio test comparing nested models. For the model as a whole, compare it against the null (intercept-only) model with a likelihood ratio test. Confidence intervals are built on the log-odds scale and then exponentiated to get an interpretable odds ratio. Prefer the LRT over the Wald test whenever coefficients are large, samples are small, or you're near separation (a predictor, or combination of predictors, that perfectly or almost perfectly divides the positive and negative outcomes, which pushes the fitted coefficient and its standard error toward infinity), since the Wald test's SE estimate becomes unstable in exactly those conditions.

Structured elaboration

Testing an individual coefficient

• Wald test: $z = \hat\beta_j / SE(\hat\beta_j)$, compared to a standard normal; two-sided p-value $= 2(1 - \Phi(|z|))$.
• Likelihood ratio test: fit the full model and a reduced model with $\beta_j$ fixed at 0, then compute the deviance difference:

Testing the model as a whole

Compare the fitted model's log-likelihood to the null (intercept-only) model's log-likelihood using the same LRT formula, with degrees of freedom equal to the number of added predictors. This is the standard omnibus test, analogous to the F-test in OLS.

Confidence intervals and odds ratios

Build the CI on the log-odds (linear) scale first, then exponentiate:

Never build the CI directly on the odds-ratio scale and average the endpoints; the log-odds scale is where the sampling distribution is (approximately) symmetric.

Wald vs. likelihood ratio test

Worked example

Simulate purchase (binary outcome) driven by two standardized predictors, fit with Newton-Raphson (IRLS), pinned seed:

import numpy as np

rng = np.random.default_rng(seed=123)
n = 500
x1 = rng.normal(0, 1, n)   # e.g. standardized time-on-site
x2 = rng.normal(0, 1, n)   # e.g. standardized past purchases
true_b0, true_b1, true_b2 = -0.5, 0.8, 0.4
p = 1 / (1 + np.exp(-(true_b0 + true_b1*x1 + true_b2*x2)))
y = rng.binomial(1, p)

X = np.column_stack([np.ones(n), x1, x2])

def fit_logreg(X, y, iters=50):
    beta = np.zeros(X.shape[1])
    for _ in range(iters):
        mu = 1/(1+np.exp(-(X @ beta)))
        W = np.clip(mu*(1-mu), 1e-8, None)
        beta = beta + np.linalg.solve(X.T @ (X*W[:,None]), X.T @ (y-mu))
    mu = 1/(1+np.exp(-(X @ beta)))
    cov = np.linalg.inv(X.T @ (X * (mu*(1-mu))[:,None]))
    return beta, cov

beta, cov = fit_logreg(X, y)
se = np.sqrt(np.diag(cov))
# beta = [-0.439, 0.874, 0.580], se = [0.102, 0.117, 0.107]

Fitting gives $\hat\beta_1 = 0.874$ (SE 0.117), so the Wald z-statistic for $x_1$ is $0.874/0.117 = 7.49$, far into significance ($p < 0.0001$). The odds ratio is $e^{0.874} = 2.40$, with a 95% CI of roughly (1.91, 3.01), meaning a one-SD increase in $x_1$ is associated with about 2.4x the odds of purchase. Comparing the full model to a reduced model that drops $x_2$ gives a log-likelihood of $-291.17$ (full) versus $-307.55$ (reduced), so $LR = -2(-307.55 - (-291.17)) = 32.76$ on 1 df, again $p < 0.0001$. In this well-behaved, moderate-sample simulation the Wald and LRT p-values agree closely, which is exactly when you'd expect them to.

Trade-offs & pitfalls

• Near-perfect separation breaks the Wald test badly: coefficients and their SEs diverge toward infinity, so the Wald z-statistic can shrink toward 0 even though the effect is enormous (the Hauck-Donner effect). The LRT stays sane in this regime.
• Odds ratios are not risk ratios. For common outcomes (baseline probability well above ~10%), an odds ratio overstates the relative risk; say so explicitly if the audience will read it as "X times more likely."
• The omnibus LRT tells you the model beats the null; it doesn't tell you it's well-calibrated or has good discrimination. Follow up with calibration plots and a discrimination metric (e.g. AUC) rather than stopping at significance.
• Multiple coefficient tests need multiplicity correction if you're screening many predictors and treating each Wald p-value as a keep/drop decision.
• CIs must be built on the log-odds scale, not by exponentiating a normal-approximation interval computed directly on the OR scale; the OR distribution is right-skewed, not symmetric.

I would treat this as a change-management problem, not just a rollout problem.

First, I would diagnose where adoption is breaking down. I would review usage data, interview a few people from each function, and compare the new process to the old one. I want to know whether people are resisting because the process is too slow, unclear, misaligned with incentives, or simply not useful in their day-to-day work.

Then I would test the rollout design. I would ask: did we train people, give them a reason to care, and remove the old path? For example, if support keeps using the old escalation template, maybe the new process adds friction and does not solve their problem fast enough.

If the issue is execution, I would tighten enablement, add team champions, and publish a clear operating cadence. If the issue is the process itself, I would change it based on the feedback rather than forcing adoption of a bad design.

I would judge success by outcomes, not attendance at meetings. If adoption improves, cycle time drops, and fewer teams revert to the old habit, the rollout is working. If not, I would change the process before asking for more compliance.

For example, when a company rolled out a new cross-functional incident-escalation process across product, engineering, and support, usage data after three weeks showed only 40% of support tickets were being routed through the new template, the rest were still going through the old one. Interviews with five support agents revealed the real problem: the new template required them to fill in a business-impact field that only engineering had the context to answer, so agents defaulted back to the old, faster template rather than get stuck. That pointed to a process-design gap, not a training gap. The fix was to move the business-impact classification to a follow-up step engineering completed after triage, instead of asking support to guess it up front. Within two weeks of that change, template usage rose to 92%, and average escalation cycle time (the time from a ticket being flagged to a fix being assigned) dropped from about 3.5 days to just under 2 days.

Situation: A PM pushed to launch a new feature immediately because a short A/B test showed p < 0.05. I felt the result was fragile due to multiple unregistered comparisons, short runtime, and potential peeking.

Task: My goal was to prevent a premature company-wide rollout that could harm metrics or trust, while preserving the PM relationship and moving to a data-backed decision.

Action:

• I started with a collaborative, non-confrontational framing: “This looks promising—can we align on some quick checks so we’re confident at scale?”
• I presented concrete evidence:
  • Re-ran the analysis with Bonferroni / Benjamini-Hochberg corrections for multiple hypotheses and showed how significance changed.
  • Displayed sequential monitoring plots (p-value and effect-size over time) to show volatility from early stopping.
  • Ran simple sensitivity checks: removing early days, stratifying by user cohort, and running bootstrap CIs for the effect size.
  • Calculated minimum detectable effect (MDE) vs observed effect and showed power at current sample size.
  • Checked for allocation or instrumentation issues and pre/post experiment covariate balance.
• Proposed compromise rollout options:
  1. Phased rollout: 10% -> 30% -> 100% with pre-specified gates (stable effect size & no negative secondary metric drift).
  2. Additional holdouts: keep a long-term control cohort for residual monitoring (e.g., 4–8 weeks).
  3. Extended run with pre-registered analysis plan and stopping rules to increase power and control Type I error.
  4. Feature flag + quick rollback plan so PMs can iterate without full exposure.
• Emphasized business trade-offs (time-to-market vs risk) and recommended the option minimizing downside while allowing learning (phased rollout + extended monitoring).
• Sought input: asked the PM which secondary metrics matter most and agreed on clear success/failure criteria.

Result / Documentation:

• We documented everything in a short decision memo: hypothesis, primary/secondary metrics, pre-registered analysis plan, multiple comparison method, rollout schedule, monitoring dashboard links, and rollback criteria.
• I logged the rationale and minutes in our product analytics wiki and scheduled checkpoints (after each phase) to reassess.
• This preserved trust: PMs appreciated the pragmatic options and the shared ownership of the decision; leadership approved a phased rollout that caught a cohort-specific negative churn signal before full launch.

Learning: Present statistical nuance as actionable risk management, not obstruction—give alternatives, quantify trade-offs, and document the agreed path so decisions remain transparent and reversible.

Direct answer. Type the boundaries and the highest-risk logic first, allow the type checker to run in a permissive/lenient mode initially, and tighten strictness incrementally so the codebase is never in a state where adding types blocks unrelated work -- never attempt a single flag-day conversion of the whole codebase.

A staged plan (TypeScript over JavaScript, the same shape applies to Python type hints)

1. Enable the type checker in its most permissive mode first (allowJs, checkJs: false initially, or --strict off) so it can run alongside 100% untyped code without generating a wall of errors on day one.
2. Type new files fully from the start -- this stops the untyped surface from growing, even before a single existing file is converted.
3. Type the highest-risk/highest-value boundaries first: public API surfaces between modules, and code that's caused type-related bugs before (a wrong shape passed across a module boundary) -- these are where types pay off fastest, versus purely-internal, rarely-touched helper functions.
4. Convert existing files opportunistically, when you're already touching a file for a real change, rather than a dedicated 'add types everywhere' project competing with feature work indefinitely.
5. Tighten strictness flags incrementally (enable noImplicitAny, then strictNullChecks, etc., one at a time) once enough of the codebase is typed that each new flag doesn't produce an unmanageable wall of new errors -- each flag addition is its own small, reviewable milestone.
6. Track progress with a simple metric (percentage of files/lines type-checked) visible to the team, so the migration has visible momentum rather than feeling like an invisible, indefinite background task.

Keeping the codebase shippable throughout

Because the type checker runs in permissive mode against a MIX of typed and untyped code the whole time, there's no moment where 'half-typed' is a broken state -- untyped code is simply treated as any (escape-hatch, unchecked) until it's converted, and the app builds and ships normally at every point in the migration.

Trade-offs and pitfalls

• Converting files just to hit a coverage percentage, without actually thinking through the types (reflexively typing everything any to make the checker pass), defeats the purpose and gives false confidence that the codebase is now type-safe when it isn't -- track QUALITY of typing (how many any escape hatches remain), not just raw percentage converted.
• Watch for type definitions that drift from runtime reality over time (a type says a field is required, but the actual data sometimes omits it) -- types are only as trustworthy as the discipline behind keeping them accurate, and a stale type is arguably worse than no type, since it actively misleads.

Direct answer

Treat the offline metric as a PREDICTION of the online result, not as ground truth to be confirmed. Build the platform so the two are compared explicitly (deterministic assignment, clean exposure logging, and a dedicated offline-versus-online agreement check), and when they diverge, check the health of the ONLINE read first (is the split clean, is exposure being captured correctly) before concluding the offline metric was simply wrong.

Structured elaboration

Platform components.

flowchart TD
  SDK[Client SDK: deterministic hash-based bucketing] --> Exposure[Exposure log]
  Exposure --> Ingest[Stream ingestion, joins exposure with events]
  Ingest --> Aggregate[Per-user daily aggregates]
  Aggregate --> LiveStats[Live stats: sample ratio mismatch check, guardrails]
  OfflineEval[Offline eval: replay against logged data] --> Compare[Offline vs online agreement check]
  LiveStats --> Compare
  Compare -->|agree| Ramp[Automated ramp]
  Compare -->|disagree| Diagnose[Diagnose: skew, exposure lag, feedback loop]
  Diagnose --> Ramp

Deterministic bucketing (a stable hash of user id plus a per-experiment salt) keeps a user's assignment consistent across app restarts and platforms. Metrics are aggregated per user per day, not per raw event, to avoid a small number of highly active users dominating the statistic. A sample ratio mismatch (SRM) check confirms the traffic split itself is clean before any metric from that split is trusted.

Why an offline metric and the live result can genuinely disagree, not just from noise:

What to do when offline and online disagree: first, gate on data quality (a clean SRM check, exposure-logging health) before trusting either number. Then segment the disagreement: is it uniform across the whole population, or concentrated in a segment with a known telemetry quirk (for example, users on a slow update cadence)? If a real effect was missed offline (a feedback loop, for example), add that as a new evaluation slice for next time rather than trusting the offline metric blindly going forward.

Worked example

A new ranking model shows a 4% offline gain on a logged holdout, but the live experiment shows a flat click-through rate (CTR) at day 7. Walk the diagnosis in order: the sample ratio mismatch check passes (the split is clean), so the disagreement is not a broken assignment. Next, check exposure capture: suppose 40% of the nominal treatment arm is on mobile clients that only receive the new model bundle on their own app-update cycle, and by day 7 only 40% of that segment has actually updated and been exposed. If the TRUE lift among users actually exposed is +5% CTR, but 60% of the nominal treatment arm hasn't been exposed yet and contributes 0% effect, the arm-level effect observed is diluted:

and with normal sampling noise at that sample size, a 2.0% true underlying effect can easily read as statistically flat at day 7. The correct read is not "the offline metric was wrong," it's "the online read is still diluted by incomplete exposure, extend the measurement window until enough of the arm has actually updated, or compute the effect only over users confirmed exposed."

Trade-offs & pitfalls

Skipping offline evaluation entirely and shipping straight to a live experiment wastes real traffic on obviously broken candidates; offline evaluation is still the right first filter, just not a sufficient one on its own. The most common wrong turn when online contradicts offline is to blame "noise" and simply run longer, without first checking a concrete, fixable confound like exposure lag or a feedback loop. Guardrail metrics catch an obviously broken rollout, but they will not catch a subtler bias where the average effect looks fine while a real harm or a real lag is concentrated in one segment; segment-level analysis is what surfaces that.