Lyft Machine Learning Engineer Interview Preparation Guide - Entry Level

Direct answer

For a fraud model with a 7-to-30-day label delay, detection has to lean on signals that don't require the delayed label at all, retraining cadence has to be sized against how long it actually takes to accumulate enough NEW labels to matter, and validation of a retrained candidate needs a scheme that doesn't just wait passively for the full label set to arrive.

Structured elaboration

1. Detection signals that work despite the delay: feature and input drift (needs zero labels, detectable immediately), proxy signals correlated with eventual fraud (manual-review flag rates, customer dispute rates that arrive faster than a confirmed-fraud label), and model confidence/score distribution shifts (a model suddenly producing far more borderline scores than usual is itself a signal, independent of ground truth).
2. Retraining cadence given the delay: retraining more often than roughly the label-delay period mostly retrains on the SAME small trickle of newly-confirmed labels: a practical cadence here is weekly-to-biweekly (matched to how long it takes a MEANINGFUL new batch of confirmed labels to accumulate), not daily.
3. Validating a retrained candidate under partial/delayed labels: use backtesting against the LATEST fully-labeled historical window (excluding the most recent labels still trickling in) as the primary validation gate, plus a secondary "surrogate" check against the delayed-but-partial recent data using proxy signals, understanding that the primary confirmed-quality read on the new model itself won't be available until its own labels mature.
4. Rollback criteria: because full validation is inherently delayed, deploy the new candidate behind a canary or shadow rather than a full swap, and set rollback criteria on the FAST-available proxy signals (score-distribution shift, review-flag rate) rather than waiting for confirmed fraud labels to justify a rollback decision: by the time confirmed labels show a problem, weeks of exposure have already accrued.

Worked example

Concretely: a candidate model retrained on data through week N is validated first against week N-1's now-fully-labeled data (the most recent window where labels have had time to mature) as the primary go/no-go gate. Once deployed via canary, if the canary's proxy signals (review-flag rate, score distribution) look stable relative to the champion over the following two weeks, promote to full traffic: the eventual confirmed-label read (arriving weeks later) becomes a LAGGING confirmation of a decision already made on faster proxy evidence, not the gate itself, since waiting for it would mean carrying full-traffic exposure to a potentially-bad model for the entire label-delay period.

Trade-offs & pitfalls

The core tension is that the metric you ultimately care about (confirmed fraud caught) is exactly the one that's slowest to materialize: over-relying on proxy signals risks promoting a model that looks fine on fast signals but is genuinely worse on the metric that matters, while over-relying on the slow, confirmed signal means carrying real exposure for weeks before you can act. The canary/shadow deployment structure is what lets you hedge this: limited exposure while waiting for the slow signal to eventually validate (or invalidate) the fast-signal-based decision.

If a DP fills an n × m table with O(1) work per cell, time = O(nm) and space = O(nm). Common techniques to reduce space to O(min(n,m)) or O(1):

1. Two-row (rolling array) technique: keep only the previous row (or column) when transitions depend on the immediate previous row. Space → O(m) (or O(n)).
2. In-place overwrite / single array with careful ordering: reuse one array when dependencies are earlier indices (e.g., prefix DP). Space → O(m).
3. Divide-and-conquer reconstruction (Hirschberg): compute only costs forward/backward to find split points, reducing memory while still yielding full solution. Space → O(min(n,m)).

Concrete example — Edit Distance (Levenshtein) reduced to O(min(n,m)) space:
Approach: compute DP row-by-row storing only two rows; swap so the shorter string indexes the columns to get O(min(n,m)) space.

def edit_distance(a: str, b: str) -> int:
    # ensure b is the shorter string to use O(len(b)) space
    if len(b) > len(a):
        a, b = b, a
    n, m = len(a), len(b)
    prev = list(range(m + 1))  # distance from empty prefix of a to prefixes of b
    cur = [0] * (m + 1)
    for i in range(1, n + 1):
        cur[0] = i
        ai = a[i - 1]
        for j in range(1, m + 1):
            cost = 0 if ai == b[j - 1] else 1
            cur[j] = min(prev[j] + 1,        # deletion
                         cur[j - 1] + 1,    # insertion
                         prev[j - 1] + cost) # substitution
        prev, cur = cur, prev  # reuse arrays
    return prev[m]

Complexity: Time O(nm). Space O(min(n,m)) for the two rows. For full alignment output, use Hirschberg to retain O(min(n,m)) space while reconstructing the sequence.

Direct answer

Quantization (representing weights and activations with fewer bits, such as 8-bit integers instead of 32-bit floats) is the cheaper first move: no retraining, and a small, well-characterized accuracy cost for a large memory and speed win. Distillation (training a smaller "student" model to mimic a larger "teacher") costs more engineering time but can recover more accuracy at a given size. Often you do both: distill to a smaller architecture, then quantize the result.

Structured elaboration

Quantization: changes numeric precision only, a calibration pass not a retrain, hours not days; 32-bit to 8-bit is a 4x memory reduction, with accuracy risk growing below 8-bit.

Distillation: changes architecture and parameter count, trained against the teacher's soft labels, days not hours plus a new evaluation cycle; preserves more accuracy per parameter than numeric coarsening alone.

Worked example

Original model 400MB in 32-bit floats, budget 100MB, baseline accuracy 92%, tolerance under 2 points. Quantizing to 8-bit:

$\text{size}_{\text{int8}} = \frac{400\text{MB}}{4} = 100\text{MB}$

exactly hits budget, and 8-bit post-training quantization commonly costs under 1 point, landing near 91-91.5%. If the measured drop is larger, say 88% (4 points, over budget), distill first to a 120MB float32 student, then quantize it:

$\text{size}_{\text{student-int8}} = \frac{120\text{MB}}{4} = 30\text{MB}$

comfortably inside budget, with the accuracy cost spent on a distillation step you can iterate on, not on coarsening you cannot.

Trade-offs and pitfalls

Quantization is fast but its accuracy hit is hard to fully control below 8-bit; distillation gives more control but is a multi-day project that may not converge on rare classes. A common pitfall is checking only aggregate accuracy and missing that compression hurt one rare, high-stakes class far more than average.

What the interviewer probes next

Checking per-class accuracy degradation rather than the aggregate, post-training quantization versus quantization-aware training, and how target hardware (INT8 kernel support) changes the plan.

Linearizability and serializability sound similar but answer different questions. Linearizability is about a single object: every operation on it must appear to happen instantaneously at some point between when it was invoked and when it returned, and that ordering must match real time. Serializability is about multiple objects touched by a transaction: the outcome of running several transactions concurrently must be equivalent to running them in some serial order, but that order does not have to match real time or even the order the transactions actually started in. A system can have one property without the other, and assuming the wrong one silently breaks a different class of guarantee.

Two mechanisms that actually enforce serializability

• Two-phase locking (2PL): a transaction acquires every lock it needs before releasing any of them, and once it starts releasing locks it may acquire no more. This physically prevents conflicting concurrent access, at the cost of blocking and potential deadlock.
• Optimistic concurrency control (OCC): transactions proceed without locking, then get validated at commit time; if another transaction's concurrent writes conflict with what this one read, it aborts and retries. This avoids blocking under low contention but wastes work under high contention.

When you need one but not the other

Consider a key-value store advertising single-copy semantics: every replica must behave as if there is exactly one physical copy of the data, so any client reading a key right after a write, from any client, on any replica, sees that write or a later one, never a stale value. The same requirement shows up as a highly available configuration service needing linearizable reads: if a client reads a feature flag or a routing rule right after it changed, it must get the new value, since acting on a stale one applies the wrong policy. Neither of these needs serializability: there is no multi-key transaction to isolate, just one key's recency.

The mirror case: a reporting system running multi-row aggregate queries across many tables needs those queries to see an internally consistent snapshot (serializability, or at least snapshot isolation), but does not need that snapshot to be the absolute latest possible instant in real time. A report built from data a few hundred milliseconds behind the live system is fine, as long as every row it reads is mutually consistent with every other row it reads.

Worked example: what breaks if you assume the wrong one

Linearizable but not serializable, no cross-key transaction: a key-value store gives linearizable single-key reads and writes but has no multi-key transactions. A funds transfer moves 30 units from account A (currently 100) to account B (currently 50) as two separate linearizable writes: write A=70, then write B=80. A concurrent reader can land exactly between the two writes and read A=70 and B=50. Both individual reads are linearizable, each reflects the latest write to that specific key at the moment it was read, but the reader just observed a total of 70+50=120, when the true, fully-settled total is 70+80=150: 30 units appear to have vanished mid-transfer. That is the anomaly linearizability alone does not prevent, because it says nothing about atomicity across two different keys.

Serializable but write-skew possible, snapshot isolation only: a hospital scheduling system enforces one invariant, that at least one doctor remains on call.

$\text{doctors on call} \ge 1$

Two doctors, Alice and Bob, are both currently on call, so the on-call count is 2. Both, concurrently, read a snapshot showing 2 doctors on call and each independently decide it is safe to go off-call, and both commit that decision under snapshot isolation, since neither transaction's write conflicts with what the other actually wrote (each only writes their own on-call flag). The result: 0 doctors on call, violating the invariant, even though each transaction, viewed alone against its own snapshot, looks perfectly valid. Full serializability, not just snapshot isolation, would detect that these two transactions' reads and writes interfere and force one to abort; snapshot isolation's weaker check does not.

Trade-offs & pitfalls

• Common wrong turn: treating serializable as automatically meaning fresh or linearizable. It is not: transactions can be serialized in an order that does not match when they actually ran.
• Common wrong turn: treating a single-key linearizable store as if it gives transactional safety across several keys. It does not, by itself, unless the store also offers multi-key transactions on top.
• Snapshot isolation is cheaper than full serializability, since it does not need to detect every possible interleaving, only genuine write-write conflicts, and is what most production databases default to, which is exactly why the write-skew anomaly above shows up in practice more often than people expect.

To merge k sorted linked lists in O(N log k) time, use a min-heap (priority queue) that always yields the smallest current node among the k heads. Push each non-null head into the heap keyed by value; repeatedly pop the smallest node, append it to the result list, and if that node has a next, push next into the heap. Each node is pushed/popped once, costing O(log k) each → O(N log k).

import heapq

class ListNode:
    def __init__(self, val=0, next=None):
        self.val = val
        self.next = next

def merge_k_lists(lists):
    """
    Merge k sorted linked lists using a min-heap.
    Time: O(N log k), Space: O(k) extra for heap
    """
    heap = []
    # Heap entries: (value, unique_id, node) -> unique_id avoids comparison of nodes
    uid = 0
    for node in lists:
        if node:
            heapq.heappush(heap, (node.val, uid, node))
            uid += 1

    dummy = ListNode()
    tail = dummy

    while heap:
        val, _, node = heapq.heappop(heap)
        tail.next = node
        tail = tail.next
        if node.next:
            heapq.heappush(heap, (node.next.val, uid, node.next))
            uid += 1

    return dummy.next

Key points:

• Time: O(N log k) where N is total nodes.
• Space: O(k) for heap; output uses original nodes (O(1) extra beyond heap).
• Use unique id to break ties when node objects aren’t comparable.

Trade-offs:

• Heap gives best general guarantee. For small k, repeated pairwise merges may be simpler; for very large k with few elements each, heap overhead dominates.
• If memory is tight and lists are extremely imbalanced, an external k-way merge (streaming from disk) with buffered reads may be needed.

ML pipeline application:

• Prediction logs are often sharded by time or source and already sorted locally. Treat each shard as a sorted linked stream; this algorithm merges shards into a single global time-ordered stream for downstream aggregation, deduplication, or metric computation. Using a heap allows streaming merge without loading all logs into memory and preserves O(N log k) performance—important when k (number of shards) is much smaller than N (total records).

Direct answer

An implicit graph never materializes its full node or edge set up front: neighbors are computed on demand by calling produce_neighbors(state), and breadth-first search (BFS) still applies unmodified, since BFS only ever needs "give me the neighbors of this node right now," never the whole graph at once. This is exactly what a state-space search (a puzzle, a game position, an abstract configuration space) needs, since materializing every reachable state ahead of time is often impossible or wasteful.

Structured elaboration

Why BFS still guarantees shortest path here. BFS's core guarantee, the first time a state is discovered it is at its true minimum distance, depends only on exploring states in non-decreasing distance order, never on knowing the graph's shape in advance. Calling produce_neighbors lazily, one node at a time, preserves this exactly: each call happens precisely when that node is dequeued, in the same layer-by-layer order BFS always uses.

Pruning strategies. A visited set is the baseline prune (never re-expand a state already discovered), but implicit graphs often support domain-specific pruning too: reject a neighbor immediately if it violates an invariant (an illegal board position, a state outside a known-safe region) before it is ever added to the queue, saving both the memory to store it and the future work of expanding it. Pruning by domain invariant must never prune a state that could still be on SOME shortest path, or the "shortest" guarantee breaks; a prune based purely on redundancy (already visited) is always safe, a prune based on heuristic judgment is not automatically safe unless proven admissible.

Heuristics. If a heuristic estimate of remaining distance to the goal is available, switching from plain BFS to A* (using the heuristic to prioritize which state to expand next via a priority queue instead of a FIFO queue) can dramatically reduce the number of states visited, at the cost of losing BFS's simplicity and requiring the heuristic to be admissible (never overestimate the true remaining distance) to keep the shortest-path guarantee.

Bidirectional search. When both start and goal are known in advance (not always true in state-space search, but common), growing frontiers from both ends and stopping when they meet bounds the search to roughly the square root of the single-direction node count for a typical branching factor, the same principle as bidirectional BFS on any explicit graph.

Worked example

from collections import deque
from typing import Callable, Iterable, List, Optional, TypeVar

State = TypeVar("State")

def find_shortest_sequence(start: State, goal: State,
                            produce_neighbors: Callable[[State], Iterable[State]]) -> Optional[List[State]]:
    if start == goal:
        return [start]
    parent = {start: None}
    q = deque([start])
    while q:
        s = q.popleft()
        for nxt in produce_neighbors(s):
            if nxt in parent:
                continue
            parent[nxt] = s
            if nxt == goal:
                path = [nxt]
                cur = nxt
                while parent[cur] is not None:
                    cur = parent[cur]
                    path.append(cur)
                return list(reversed(path))
            q.append(nxt)
    return None

if __name__ == "__main__":
    # Abstract 12-state space: from state s, neighbors are s+1, s-1, s+5 (mod 12).
    # Deliberately not a literal grid or puzzle, to make the point that BFS
    # does not care what a "state" actually represents.
    N = 12
    def produce_neighbors(state):
        return [(state + 1) % N, (state - 1) % N, (state + 5) % N]

    path = find_shortest_sequence(0, 7, produce_neighbors)
    print("Shortest sequence 0 -> 7:", path)
    print("Steps:", len(path) - 1 if path else None)

    def is_valid(path, produce_neighbors):
        return all(path[i+1] in set(produce_neighbors(path[i])) for i in range(len(path)-1))
    print("Path uses only real transitions:", is_valid(path, produce_neighbors))

    def full_bfs_dist(start, produce_neighbors):
        dist = {start: 0}
        q = deque([start])
        while q:
            u = q.popleft()
            for v in produce_neighbors(u):
                if v not in dist:
                    dist[v] = dist[u] + 1
                    q.append(v)
        return dist
    ref = full_bfs_dist(0, produce_neighbors)
    print("Matches independent full-BFS distance table:", ref[7] == len(path) - 1)

    print("No path from a state that only transitions to itself:", find_shortest_sequence(100, 200, lambda s: [s]))

Output (actually executed with python3):

Shortest sequence 0 -> 7: [0, 1, 2, 7]
Steps: 3
Path uses only real transitions: True
Matches independent full-BFS distance table: True
No path from a state that only transitions to itself: None

The independent full_bfs_dist helper computes distances to every reachable state from scratch, without reusing the path-returning function's logic, and its distance to state 7 (3) matches len(path) - 1 exactly, confirming the lazily-called produce_neighbors version is genuinely finding a true shortest sequence, not just any sequence.

Complexity

Time $O(N + E)$ where $N$ is the number of reachable states and $E$ is the number of transitions actually explored (both unknown in advance for a true implicit graph, unlike an explicit one where $V$ and $E$ are given). Space $O(N)$ for parent and the queue. Each call to produce_neighbors is charged whatever it costs to compute (in the example above, $O(1)$; in a real puzzle, it might be proportional to board size).

Edge cases

• start == goal: returns [start] immediately, a zero-move sequence, without ever calling produce_neighbors.
• No path exists (as demonstrated by the self-loop-only state 100 in the worked example): the queue drains completely, parent never gains an entry for goal, and the function returns None.
• produce_neighbors yielding a state already on the current path (a state with a transition back to itself, or a cycle in the state graph): handled the same as any other implicit-graph cycle, since the parent dict doubles as the visited set, a state already discovered is never re-enqueued.
• produce_neighbors raising an exception mid-search (a real risk if computing a state's neighbors can fail, for example an invalid board configuration): not handled by the implementation above, and worth flagging explicitly as something a production version would need to decide on: abort the whole search, or treat that state as having no valid neighbors and continue.

Trade-offs and pitfalls

• Common mistake: calling produce_neighbors more than once for the same state (for example, once to check if any neighbor is the goal, and again to actually enqueue them). Since neighbor generation can be expensive in a real state space, the implementation above calls it exactly once per dequeued state and processes each yielded neighbor as it arrives.
• Common mistake: pruning by a heuristic that is not admissible (can overestimate true remaining distance), which silently breaks BFS's shortest-path guarantee; a state that gets pruned because it "looks unpromising" might still sit on the actual shortest path.
• Infinite or unbounded state spaces. Unlike a finite explicit graph, an implicit state space can be infinite (an unbounded counter, an open-ended configuration). Plain BFS on such a space either runs forever if goal is unreachable, or needs an explicit depth cap or iterative-deepening strategy layered on top to guarantee termination even on a "no path exists" input.
• When to reach for A instead.* If a genuinely admissible heuristic exists (an under-estimate of remaining distance that is cheap to compute), A* dominates plain BFS by visiting fewer states while preserving the same shortest-path guarantee; if no such heuristic is available or trustworthy, plain BFS remains the safe default.

Direct answer

An online-learning recommendation system updating continuously from millions of users needs monitoring that catches degradation FAST (since a bad update compounds quickly), explicit safety constraints (exposure caps, diversity floors) baked into the serving logic itself, and an auditable trail so any harmful update can be traced and reverted precisely.

Structured elaboration

1. Monitoring for degradation: track rolling, short-window versions of the usual signals (score-distribution stability, diversity of recommendations served, engagement-rate trend) at a MUCH tighter cadence than a batch-retrained system would need, since a continuously-updating model can degrade meaningfully within hours, not days.
2. Safety constraints as guardrails, not just alerts: exposure caps (no single item or narrow category can exceed some maximum share of impressions, regardless of what the model currently believes is optimal) and diversity controls (a minimum floor on recommendation variety) are enforced directly in the SERVING logic, as a hard constraint the model operates within: this is different from an alert that fires after the fact, since these constraints prevent certain bad outcomes from ever being SERVED at all, rather than just detecting them after the harm has already reached users.
3. Rollback and shadow testing for candidate updates: rather than every incremental update going straight to full production, validate candidate updates in shadow first (score live traffic without serving) even in an online-learning setting, and maintain the ability to revert to a specific PRIOR model state (a periodic checkpoint) if a bad update needs undoing: pure online learning without checkpoints makes "rollback" meaningless, since there's no discrete prior state to return to.
4. Offline evaluation of updates: periodically (even if updates themselves are continuous) run a full offline evaluation against a held-out benchmark, to catch a slow, cumulative degradation that no single incremental update looked bad enough to trigger an alert on its own.
5. Auditable records: log enough about each incremental update (what data it trained on, what changed) that a harmful drift can be traced back to roughly WHEN it started accumulating, not just noticed once it's already severe.

Worked example

The algorithm-choice dimension matters here too: for a time-series-like setting with frequent concept drift, adaptive online learners (adaptive random forests, or online gradient boosting with a decay factor on older data) are built to track a moving target more gracefully than a model with no explicit forgetting mechanism, trading some stability for responsiveness, but that responsiveness is exactly what makes the guardrails above non-optional, since a model that adapts FAST to real drift also adapts fast to a bad batch of poisoned or anomalous data.

Trade-offs & pitfalls

The core tension in a continuously-updating system is that the same property that makes it valuable (fast adaptation to real changes) also makes it dangerous (fast adaptation to noise, bugs, or adversarial input): hard guardrails baked into serving, not just monitoring, are what keep the downside bounded while still capturing the upside of genuine responsiveness.

Direct answer: Feature engineering for survival analysis (modeling time-to-event with right-censoring, like time-to-churn) needs to represent both fixed baseline characteristics at the start of observation and genuinely time-VARYING covariates that update as the observation period unfolds, with the preprocessing differing meaningfully between a proportional-hazards-style model and a more flexible tree-based or deep survival model.

Structured elaboration:
Baseline features at time zero: characteristics known at the start of the observation window (signup date, initial plan type), which are static for the duration of that entity's observation. Time-varying covariates: features that genuinely change during the observation period (a rolling engagement count, updated at each time step) need to be represented as a SEQUENCE of values over time, not a single snapshot, since the whole point of survival analysis with time-varying covariates is capturing how CHANGES in behavior relate to the hazard of the event occurring.

Preprocessing differences by model: a Cox proportional-hazards model has specific structural assumptions (the proportional-hazards assumption itself, and typically a linear combination of covariates) that constrain how time-varying covariates can be incorporated (often requiring the data to be reshaped into a specific "counting process" format with one row per covariate-change interval); a random survival forest or a deep survival model (like DeepSurv) can generally handle time-varying covariates more flexibly and doesn't require the same structural linearity or proportional-hazards assumption, at the cost of typically needing more data to fit reliably and being less directly interpretable.

Worked example: For a subscription-churn survival model, a baseline feature might be the customer's initial plan tier (fixed at signup), while a time-varying covariate is their rolling 30-day engagement score, updated at each observation interval; reshaped into the counting-process format a Cox model needs, each customer contributes multiple rows, one per interval during which their covariates stayed constant, with the model then estimating how a CHANGE in the time-varying engagement score relates to the instantaneous hazard of churning at that point, which a single-snapshot feature representation could never capture.

Trade-offs and pitfalls: The temptation to simplify by using only baseline (time-zero) features and ignoring genuine time-variation loses exactly the signal that's often most predictive in a survival context (a RECENT change in behavior, not the initial state, frequently drives the actual event), so the added complexity of properly representing time-varying covariates is usually worth its engineering cost for this class of problem specifically.

High-level goal: show clearly and quantitatively why the new model improves business outcomes, explain trade-offs (FP vs FN), and propose a safe rollout + monitoring plan, backed by a persistent executive dashboard rather than a one-time deck.

Metrics I'd present (with plain-language translations)

• Confusion matrix (counts): "How many real positives/negatives are acted on?"
• Precision / Recall / F1: Precision = percent of flagged cases that are correct (reduces wasted effort); Recall = percent of true cases we catch (reduces missed opportunities/risks). F1 balances them.
• ROC AUC and PR AUC: overall discriminative ability; PR AUC is more meaningful for imbalanced targets.
• Calibration (reliability diagram / Brier score): "If model says 70% risk, how often is it true?" (important for risk-based decisions).
• Business-weighted cost (expected cost / benefit per decision): translate FP/FN into $ or UX impact (e.g., cost of manual review, lost revenue, regulatory fines).
• Latency/resource metrics: inference time, throughput, memory.

Visualizations

• Confusion matrix heatmap with raw counts and rates (easy for non-technical audiences).
• Bar chart mapping cost per FP vs cost per FN and total expected cost for baseline vs new model.
• ROC and Precision-Recall curves with highlighted operating point (threshold).
• Calibration plot (predicted vs observed).
• Cumulative gain / lift chart to show business value (e.g., how many positives captured in top X%).
• Threshold slider demo (interactive or static snapshots) showing how changing threshold trades FP vs FN and the business cost curve.

Presentation plan (30-40 minutes, one-time, to get the go/no-go decision)

1. Context & objective (3-5 min): business goal, current model baseline, decision stakes.
2. Summary headline (2 min): one-sentence verdict, e.g. "New model reduces missed events by X% and lowers expected cost by $Y/month at the recommended threshold."
3. Evidence overview (10-12 min): confusion matrix comparison, precision/recall/F1/AUCs in plain language, business-cost comparison, calibration/reliability for trust.
4. Trade-offs and threshold choice (5-8 min): trade-off curves, recommended threshold and why.
5. Risk & rollout plan (5 min): canary/A-B testing, KPIs to watch, rollback criteria.
6. Q&A and decision request (5 min).

The executive dashboard (persistent artifact, separate from the one-time deck)
The deck gets the go/no-go decision; the dashboard is what leadership and I both look at afterward, so it must be a SMALL, curated set, not the full technical metrics list above. I would build it with exactly these KPIs:

1. Net business value vs baseline ($/week): expected revenue/cost impact of the switch, the single number that answers "is this working".
2. Recall (or catch-rate) at the deployed operating point, vs baseline: the plain-language "how many of the bad things are we still catching" number.
3. Manual-review / customer-friction volume, vs baseline: the plain-language "how much extra work or annoyance are we causing" number, which is the direct proxy for false-positive cost.
4. Model health flag (calibration drift / data drift indicator): a simple green/yellow/red status, not a raw statistic, so a non-technical viewer knows when to escalate to the team rather than trying to interpret a distribution shift metric themselves.
5. Rollout stage and status (shadow / 10% canary / ramping / 100%): so leadership always knows how much of the traffic is currently exposed to the new model.
   Each tile links through to the full technical dashboard (the confusion matrix, PR curves, calibration plots listed above) for anyone who wants to go deeper, but the executive view itself stays to these five.

Monitoring follow-up commitments

• Phased rollout: shadow mode, then 10% traffic canary for 2 weeks, then gradual ramp to 100% if KPIs meet targets.
• Production monitoring cadence: the five KPIs above refreshed daily; underlying technical metrics (precision, recall, calibration, drift tests) reviewed daily by the ML team and escalated to the executive dashboard only when they cross an alert threshold.
• Alerting & SLOs: alert if expected cost increases more than X% or recall drops more than Y% relative to baseline for 24 hours; latency SLA (e.g., 95th percentile < 200ms).
• Retraining policy: retrain schedule (monthly or drift-triggered), with automated validation and an A/B test before production swap.
• Governance: a post-launch review at 2 weeks and 8 weeks with business owners to validate realized ROI against the dashboard's net-value KPI and adjust threshold/policy if needed.

Why this works for non-technical stakeholders

• Confusion matrix and business-cost framing directly map model errors to operational and financial consequences in the one-time deck.
• The five-KPI executive dashboard gives leadership a standing, glanceable answer to "is this still working" without requiring them to read a PR curve every week.
• Phased rollout + monitoring reduces risk and provides measurable checkpoints for go/no-go decisions.

Final ask for stakeholders

• Confirm the prioritized business objective (minimize cost? maximize recall? reduce false positives?) so we lock the threshold, the dashboard's alert thresholds, and the rollout cadence.

Direct answer

Ordinary least squares (OLS) assumes: (1) linearity of the mean relationship, (2) independence of errors, (3) homoscedasticity (constant error variance), (4) no perfect multicollinearity among predictors, and, only if you need exact small-sample inference, (5) normally distributed errors. The first four are about getting unbiased, well-behaved coefficient estimates; the fifth is about trusting the p-values and confidence intervals you compute from them.

Structured elaboration

Two things worth being precise about. First, OLS point estimates are unbiased under just linearity and mean-zero, uncorrelated errors, homoscedasticity is only needed for the standard variance formula $\sigma^2(X^\top X)^{-1}$ to be correct, not for unbiasedness. Second, the diagnostics form a natural order: check linearity first, because a mis-specified mean can itself produce a curved or fanned residual plot that looks like heteroscedasticity but is really a wrong functional form.

Worked example

Simulate $y = 2.0 + 1.5x + \varepsilon$ where $\varepsilon$'s standard deviation grows with $x$ (a textbook heteroscedasticity case), fit OLS, then run the auxiliary regression the Breusch-Pagan test is built on (regress squared residuals on the predictor):

import numpy as np
np.random.seed(0)
n = 500
x = np.random.uniform(1, 10, n)
noise = np.random.randn(n) * (0.3 * x)      # std scales with x: heteroscedastic
y = 2.0 + 1.5 * x + noise

X = np.column_stack([np.ones(n), x])
beta_hat = np.linalg.solve(X.T @ X, X.T @ y)          # [1.893, 1.491], close to [2.0, 1.5]
resid = y - X @ beta_hat

Xr = np.column_stack([np.ones(n), x])
gamma = np.linalg.solve(Xr.T @ Xr, Xr.T @ (resid**2))
pred = Xr @ gamma
r2_bp = 1 - np.sum((resid**2 - pred)**2) / np.sum((resid**2 - resid.var())**2)
lm_stat = n * r2_bp                                   # ~ chi-square(1) under homoscedasticity

This runs (verified) to beta_hat = [1.893, 1.491], close to the true [2.0, 1.5], and an auxiliary $R^2 = 0.137$, giving an LM statistic of 68.4 against a $\chi^2(1)$ reference, a p-value on the order of $10^{-16}$. Coefficients are still roughly unbiased, exactly as the theory predicts (homoscedasticity isn't needed for that), but the test flags, correctly and decisively, that the default OLS standard errors are wrong here.

A second check, separately, on multicollinearity:

np.random.seed(1)
n2 = 300
x1 = np.random.randn(n2)
x2 = 0.95*x1 + 0.05*np.random.randn(n2)   # near-collinear with x1
x3 = np.random.randn(n2)                  # independent

def vif(Xmat, j):
    y_j = Xmat[:, j]
    Xother = np.delete(Xmat, j, axis=1)
    Xother = np.column_stack([np.ones(n2), Xother])
    beta = np.linalg.solve(Xother.T @ Xother, Xother.T @ y_j)
    pred = Xother @ beta
    r2 = 1 - np.sum((y_j-pred)**2) / np.sum((y_j-np.mean(y_j))**2)
    return 1/(1-r2)

Xvif = np.column_stack([x1, x2, x3])
vifs = [vif(Xvif, j) for j in range(3)]

Running this (verified) gives vifs = [293.97, 294.10, 1.01]: the near-collinear pair comes back with VIF around 294 for both, versus 1.01 for the independent column, exactly the "blows up" signature VIF is meant to catch.

Trade-offs & pitfalls

• Don't diagnose in the wrong order: a curved residual plot from a mis-specified mean can masquerade as fanning variance. Fix the functional form first, then re-check for heteroscedasticity on the corrected model.
• A VIF threshold of 10 is a convention, not a law. In some fields (e.g. survey data with many derived variables) even VIF of 5 is treated as a problem; judge it against how unstable your specific coefficients get.
• Normality of errors is the assumption most often over-weighted. With even a moderately large sample, the coefficient estimates are approximately normal by the central limit theorem regardless, so a failed Q-Q plot on the residuals is rarely a reason to abandon OLS outright.
• Heteroscedasticity-robust standard errors fix your inference but not your efficiency: OLS is no longer the minimum-variance estimator once errors are heteroscedastic, weighted least squares can do better if you know (or can estimate) the variance structure.