Lyft Applied Scientist (Senior Level) - Comprehensive Interview Preparation Guide

Direct answer

When offline metrics favor a new model but the online test doesn't, work through instrumentation and measurement issues first (they're common and cheap to rule out), then feature-computation differences between offline and online paths, before concluding the model itself is genuinely worse online.

Structured elaboration

1. Instrumentation verification: confirm the online experiment is actually routing the claimed 5% of traffic to Model B, and that the CTR metric is computed identically for both arms (same attribution window, same bot-filtering, same definition of a "click"). A surprising number of "the model regressed online" incidents turn out to be a metric-computation bug in the experiment pipeline, not the model.
2. Feature-distribution comparison: pull the actual features Model B saw in the online experiment and compare them against the offline evaluation set's features: a training-serving skew (a feature computed differently online than offline) means Model B's offline AUC was measured against inputs that don't match what it actually receives live.
3. Cohort analysis: check whether the 3% CTR drop is uniform across user segments or concentrated in one: a concentrated drop points at a specific interaction (a segment where Model B's behavior genuinely differs) rather than a uniform regression.
4. Labeling delays: if CTR requires a post-click event to register (say, a delayed conversion signal folded into the metric), an early read on a short experiment window can look worse than the eventual steady-state number simply because Model B's traffic hasn't had time to accumulate its delayed positive signals yet.
5. Production-only differences: check for anything that exists online but not in offline evaluation: real-time feature freshness, cold-start behavior for new users the offline eval set didn't include, or latency-driven fallback behavior under load.

Worked example

A concrete case this checklist catches: offline evaluation used a feature computed as a same-day rolling average; the online serving path computed the "same" feature using a different timezone boundary for what counts as "same day." Every offline metric looked fine because offline evaluation used the SAME feature-computation code as training. Online, Model B's actual inputs differed subtly from what it was evaluated against, degrading its real online CTR even though nothing about the MODEL itself was worse: the checklist's feature-distribution-comparison step is specifically designed to catch exactly this class of bug.

Trade-offs & pitfalls

The instinct to distrust the online result and defer to the (more favorable) offline number is exactly backwards: the online A/B test is closer to ground truth by construction, since it measures real user behavior rather than a proxy metric on held-out historical data. The discipline here is trusting the online signal and investigating WHY the offline number was misleading, not looking for a reason to dismiss the online result.

Definition (concise)

• Data drift: change in the distribution of input features p_train(x) → p_prod(x).
• Concept drift: change in the conditional relationship between inputs and targets p(y|x) over time (model no longer maps inputs to correct outputs).

Three practical detection methods

1. Statistical two-sample tests

   • Use Kolmogorov–Smirnov for continuous features, Chi-square for categorical; run per-feature and aggregate with Bonferroni or BH correction.
   • Example: KS p < 0.01 on feature "session_length" → raise suspicion.

2. Proxy metrics / population-level features

   • Monitor feature distribution summaries (mean, std, percentiles), derived ratios, and missingness rate. Sudden shifts in feature importance (via SHAP on recent batch) also signal drift.
   • Example: median transaction_amount up 40% vs baseline.

3. Label-informed drift and performance monitors

   • When labels available, track model metrics (AUC, calibration, PSI, lift). Use Population Stability Index (PSI) for grouped distributions; for concept drift, monitor calibration drift and degradation in AUC over sliding windows.
   • Example: sustained 5% AUC drop over 7 days triggers investigation.

Alerting strategy (latency vs noise)

• Multi-tier alerts:
  1. Quiet anomalies: minor stat test p-value triggers logging and increased sampling / automated data snapshot (no human alert).
  2. Elevated: aggregated signals across multiple features or proxy metric breaches (e.g., 3 features fail KS or PSI > 0.2) trigger on-call notification.
  3. Critical: performance degradation (AUC drop + calibration error) or business KPI impact triggers immediate paging.
• Use rate-limiting and rolling-window confirmation (require anomaly persists for N consecutive windows or affects >M% of traffic) to reduce false positives.
• Add automated remediation: shadow model retraining pipeline and feature-validation checks before human intervention.
• Include runbook documenting checks, likely root causes, and rollback/retrain steps.

Direct answer: An end-to-end RLHF platform at this scale (roughly 100M preference pairs, daily fine-tuning runs) needs to separate the always-on serving path from the heavy offline training path, use tiered storage matched to access patterns, and build in checkpointing and immutable artifact versioning so daily runs are reproducible and cheap to roll back.

Structured elaboration:

• Major components: an ingestion and validation service (schema checks, deduplication, PII filtering) sitting in front of a preference store; an annotation service backing human-labeling UI; reward-model training on a GPU cluster; a policy-optimization cluster (PPO-style, with actor/learner separation); an artifact repository for versioned models and checkpoints; a serving layer for both the reward model (batched scoring) and the policy (rollout generation); and a monitoring/governance layer over all of it.
• Data flow: production comparisons, annotator judgments, and any active-learning-selected pairs all land in the ingestion pipeline, get validated and deduplicated, and are written both to a hot store (fast reads for recent pairs, used for sampling and reward-model training minibatches) and a cold, partitioned object store (bulk historical data, used for large offline training runs).
• Storage and sharding: partition the cold store's preference data by date, model version, and a shard key (so training readers can each own a disjoint prefix range and read in parallel without hotspotting); keep a document-store index over the data for fast filtering during active sampling; use a columnar, compressed format for the bulk of the 100M pairs, since most access patterns are large sequential reads during training rather than point lookups.
• Compute and storage sizing: reward-model training and the PPO policy update both need GPU/TPU capacity, but the compute and storage considerations are asymmetric, reward-model training reads large batches of static preference data (I/O-bound, benefits from streaming and prefetch), while policy optimization needs to also generate rollouts and re-score them, so it typically needs a mix of generation-optimized and training-optimized hardware; storage for model checkpoints (versioned, kept for rollback) usually dominates over raw preference-data storage once you retain many days of daily fine-tuning history, so a clear checkpoint retention policy is part of the cost model, not an afterthought.
• Cost-saving opportunities: use preemptible/spot GPU capacity for the less time-critical reward-model training, with frequent checkpointing to tolerate preemption; run a full retrain on a weekly cadence and incremental fine-tuning on daily deltas rather than a full 100M-pair retrain every day; tier storage (hot store for the last few weeks of active data, cold archival storage for older history); and batch/autoscale the serving layer aggressively, since reward-model and policy inference both benefit heavily from batching GPU requests.

flowchart TB
    Ann[Annotation Service] --> PrefStore[(Preference Store hot/cold)]
    Ingest[Ingestion + Validation] --> PrefStore
    PrefStore --> RM[Reward Model Training]
    RM --> Artifacts[(Artifact Repository)]
    Artifacts --> PPO[Policy Optimization Cluster]
    PPO --> Artifacts
    Artifacts --> Serving[Serving Layer]
    Serving --> Monitor[Monitoring + Rollback]
    Monitor -.-> PPO

Worked example: A concrete daily cycle: an Airflow-style DAG triggers each morning, pulls the latest promoted reward model and the delta of new preference pairs since the last run (rather than the full 100M), runs an incremental PPO fine-tuning pass on a fixed GPU-hour budget, validates the resulting policy against a held-out evaluation set and safety regression suite, and only promotes the new checkpoint to the canary serving stage if it clears both bars; a full reward-model retrain on the entire accumulated dataset runs on a separate, weekly schedule, since it is far more expensive and does not need to happen daily to keep the policy improving incrementally.

Trade-offs and pitfalls: Chasing strong consistency everywhere in this pipeline is unnecessary and expensive, the cold historical store can tolerate eventual consistency, while only the hot store used for live sampling and dedup checks needs to be strongly consistent. Using preemptible compute for cost savings adds real operational complexity (checkpointing must be frequent and reliable, or a preemption mid-run wastes GPU-hours), so the savings need to be weighed against the added reliability engineering it requires. Immutable, versioned artifacts (models, datasets, sampling-policy snapshots) are what make daily runs auditable and reversible, skipping this discipline to move faster is a common shortcut that makes a bad daily run very expensive to diagnose after the fact.

Linearizability and serializability sound similar but answer different questions. Linearizability is about a single object: every operation on it must appear to happen instantaneously at some point between when it was invoked and when it returned, and that ordering must match real time. Serializability is about multiple objects touched by a transaction: the outcome of running several transactions concurrently must be equivalent to running them in some serial order, but that order does not have to match real time or even the order the transactions actually started in. A system can have one property without the other, and assuming the wrong one silently breaks a different class of guarantee.

Two mechanisms that actually enforce serializability

• Two-phase locking (2PL): a transaction acquires every lock it needs before releasing any of them, and once it starts releasing locks it may acquire no more. This physically prevents conflicting concurrent access, at the cost of blocking and potential deadlock.
• Optimistic concurrency control (OCC): transactions proceed without locking, then get validated at commit time; if another transaction's concurrent writes conflict with what this one read, it aborts and retries. This avoids blocking under low contention but wastes work under high contention.

When you need one but not the other

Consider a key-value store advertising single-copy semantics: every replica must behave as if there is exactly one physical copy of the data, so any client reading a key right after a write, from any client, on any replica, sees that write or a later one, never a stale value. The same requirement shows up as a highly available configuration service needing linearizable reads: if a client reads a feature flag or a routing rule right after it changed, it must get the new value, since acting on a stale one applies the wrong policy. Neither of these needs serializability: there is no multi-key transaction to isolate, just one key's recency.

The mirror case: a reporting system running multi-row aggregate queries across many tables needs those queries to see an internally consistent snapshot (serializability, or at least snapshot isolation), but does not need that snapshot to be the absolute latest possible instant in real time. A report built from data a few hundred milliseconds behind the live system is fine, as long as every row it reads is mutually consistent with every other row it reads.

Worked example: what breaks if you assume the wrong one

Linearizable but not serializable, no cross-key transaction: a key-value store gives linearizable single-key reads and writes but has no multi-key transactions. A funds transfer moves 30 units from account A (currently 100) to account B (currently 50) as two separate linearizable writes: write A=70, then write B=80. A concurrent reader can land exactly between the two writes and read A=70 and B=50. Both individual reads are linearizable, each reflects the latest write to that specific key at the moment it was read, but the reader just observed a total of 70+50=120, when the true, fully-settled total is 70+80=150: 30 units appear to have vanished mid-transfer. That is the anomaly linearizability alone does not prevent, because it says nothing about atomicity across two different keys.

Serializable but write-skew possible, snapshot isolation only: a hospital scheduling system enforces one invariant, that at least one doctor remains on call.

$\text{doctors on call} \ge 1$

Two doctors, Alice and Bob, are both currently on call, so the on-call count is 2. Both, concurrently, read a snapshot showing 2 doctors on call and each independently decide it is safe to go off-call, and both commit that decision under snapshot isolation, since neither transaction's write conflicts with what the other actually wrote (each only writes their own on-call flag). The result: 0 doctors on call, violating the invariant, even though each transaction, viewed alone against its own snapshot, looks perfectly valid. Full serializability, not just snapshot isolation, would detect that these two transactions' reads and writes interfere and force one to abort; snapshot isolation's weaker check does not.

Trade-offs & pitfalls

• Common wrong turn: treating serializable as automatically meaning fresh or linearizable. It is not: transactions can be serialized in an order that does not match when they actually ran.
• Common wrong turn: treating a single-key linearizable store as if it gives transactional safety across several keys. It does not, by itself, unless the store also offers multi-key transactions on top.
• Snapshot isolation is cheaper than full serializability, since it does not need to detect every possible interleaving, only genuine write-write conflicts, and is what most production databases default to, which is exactly why the write-skew anomaly above shows up in practice more often than people expect.

Direct answer

Treat a single surprising segment finding, mobile shows a significant lift while the overall test is flat, as a hypothesis to validate, not a result to act on. Work through data-integrity checks, a formal interaction test with a multiplicity correction (since this segment was very likely noticed after the fact rather than pre-specified), and a set of robustness checks; then use an explicit decision process that weighs the statistical uncertainty against the business value and cost of being wrong, rather than a pure significance threshold, to choose between shipping to that segment, running a confirmatory follow-up, or abandoning the finding.

Structured elaboration

Step 1: verify the data before trusting the effect

• Check assignment balance within mobile specifically: treatment and control counts, and balance on key covariates, within the mobile slice alone, not just in aggregate.
• Check for instrumentation differences: missing events, a different SDK version, or a different exposure window on mobile that could produce a spurious effect having nothing to do with the treatment.
• Check for timing issues: did the mobile rollout start at the same time as the rest of the experiment, and is there any cross-over where a user appears in both device buckets across the test window.

Step 2: test the interaction formally

Fit an interaction model rather than comparing the mobile-only conversion rate to the mobile-only control rate informally:

import statsmodels.formula.api as smf

df["treat"] = df["assignment"].map({"control": 0, "treatment": 1})
model = smf.logit("conversion ~ treat + mobile + treat:mobile + signup_channel", data=df).fit()
print(model.summary())

Illustrative output (a hypothetical summary row, not a real run) would show a coefficient, standard error, z-value, and p-value for each term; the row that matters most here is treat:mobile. A row reading something like treat:mobile coef = 0.18, p = 0.02, alongside a treat main-effect coefficient close to zero and non-significant, is the pattern that supports a genuine mobile-specific effect: the interaction term carries the real signal while the main treatment effect alone looks flat, consistent with the original observation that the overall test showed no lift. A significant coefficient on treat:mobile is what actually supports "the effect really differs by device," rather than the mobile-only point estimate on its own, which can look large purely from within-mobile noise.

Step 3: correct for multiplicity honestly

Ask directly whether mobile was a subgroup chosen before the test ran or one noticed afterward because it happened to look interesting. If it was not pre-specified, and in practice it usually was not when this kind of question comes up, apply a multiplicity correction appropriate to however many segments were actually eyeballed (even informally) before mobile stood out, or at minimum treat the raw p-value as an optimistic upper bound on how surprising this finding really is.

Step 4: check power on the mobile slice itself

Compute the sample size and event count within mobile alone and the confidence interval width on its effect estimate. A wide interval or a small mobile sample means the "significant" reading is fragile, and this matters even more when mobile is a genuinely small-traffic segment (a specific device class or platform with limited volume) rather than merely a smaller slice of a large population: in that case a confirmatory follow-up restricted to the same segment may take a long time to reach adequate power, or may never fully reach the same statistical bar as the overall test, which is itself part of the decision, not a reason to ignore the finding.

Step 5: robustness checks

• Look at related metrics (engagement, retention, complaint or refund rate) to see whether they move in a direction consistent with the primary metric's mobile-specific lift, or whether the primary metric is moving alone in a way that is harder to explain.
• Check whether the effect is stable over the test window or concentrated in a short burst of days.
• Check finer sub-slices of mobile (iOS versus Android, OS version) to rule out the effect actually being driven by one narrow slice within "mobile" rather than the device class as a whole.
• Re-run with alternative covariate adjustment and see whether the interaction coefficient is stable.

Worked example: the decision process

Rather than a bare "p < 0.05 so ship it" rule, weigh four inputs explicitly: how strong the statistical evidence is after the checks above, how large and reliable the resulting business value would be if the effect is real, how costly it is if the segment is shipped and the effect turns out not to be real, and how long a confirmatory follow-up on that segment alone would realistically take to reach adequate power given the segment's own traffic volume.

• Strong evidence, low cost of being wrong, fast to confirm: ship a small, reversible rollout to the segment while a confirmatory read continues, since the downside of being wrong is small and quickly detected.
• Moderate evidence, or the segment is small enough that a proper confirmatory test would take a long time to reach power: this is the case worth naming explicitly, since waiting for full statistical certainty may never be practical for a genuinely small segment. Here, the decision becomes an explicit risk-tolerance call: state the estimated cost of shipping on an unconfirmed finding versus the estimated cost of never acting on a real effect because the segment could never generate enough data to confirm it on its own, and make that trade-off visible to the decision-maker rather than deferring it to a p-value the segment may structurally never be able to produce.
• Weak evidence, or a moderate cost of being wrong: run a dedicated, pre-specified confirmatory experiment targeted at the segment before making any production change, treating the original finding purely as the hypothesis that justified the follow-up.
• Evidence disappears after the data-integrity and robustness checks: abandon the finding and document why, so the same slice does not get re-litigated the next time someone happens to look at it.

Trade-offs & pitfalls

• Treating an unadjusted subgroup p-value as decisive. The interaction test plus a multiplicity correction is what separates a real segment effect from one of several plausible slices that happened to look significant.
• Waiting indefinitely for a small segment to reach the same statistical bar as the overall test. For a genuinely low-traffic segment, that bar may not be reachable on a useful timeline; the decision framework needs to say what happens in that case rather than defaulting to inaction.
• Ignoring instrumentation as a candidate explanation. A device-specific logging or SDK difference is a mundane but common cause of an apparent segment effect and should be ruled out before any statistical machinery is trusted.
• Shipping on a single significant slice with no plan to re-check it. Even a reversible segment rollout should carry a defined follow-up read, not be treated as a closed decision the moment it ships.

Direct answer

Decide what to delegate by looking at two things: where the task sits relative to the person's current skill level, and what happens if they get it wrong. Delegate work that stretches them but is reversible or cheap to fix. Keep for yourself work that needs context you can't hand off in time, decisions whose blast radius exceeds the trust you've built with this person so far, or one-off tasks where teaching would take longer than doing it. Treat each handoff as a deliberate intervention, not an offload: pick the task for the specific gap it targets, define what "done" looks like up front, and calibrate how much support comes with it.

Decision framework

Match difficulty to their zone of growth. Too easy and it's busywork with no development value. Too hard with no support and it's discouraging or risky. The sweet spot is a task just past what they've done independently before.

Weigh reversibility, not just difficulty. Prefer delegating decisions that are cheap to undo (a first draft, a component design, a low-stakes customer interaction) over ones that are hard to walk back (a commitment made externally, a change with security or compliance exposure). Trust for higher-stakes delegation gets built incrementally through the reversible tasks.

Compare time-to-teach against time-to-do. If explaining the task well would take meaningfully longer than doing it yourself, and it's a one-off with no repeat value, do it yourself. If it's a skill they'll use again, the teaching cost is an investment that pays back on the second and third time.

Define the support structure explicitly. Delegating isn't handing off and disappearing. Decide in advance: what checkpoints happen, what they can decide alone versus what needs a quick check-in, and what "stuck enough to escalate" looks like.

What you keep. Work that needs institutional context you can't transfer in the available time, early-relationship politically sensitive conversations, and anything where a mistake would damage a stakeholder's trust in the team broadly rather than just cost you some rework time.

Worked example

Say you're leading a project with three distinct pieces. One is well-scoped, reversible, and slightly above where this person has worked before: a strong candidate to delegate as a growth task, with a design check-in before they start building and a review before it ships. Another piece is customer-facing with real cost if it goes wrong: you either delegate it with heavy pairing so you catch problems before they land, or you keep it yourself this round and delegate the next similar piece once trust is established. The third is a one-off internal chore with no growth value: you delegate it purely for your own capacity, not as a coaching move, and you say so, because dressing up busywork as a growth opportunity erodes trust.

Trade-offs and pitfalls

Delegating only "safe" tasks because failure is expensive to you personally caps the person's growth. They never build judgment under real stakes if you only ever hand them things that can't go wrong.

Delegating and then vanishing looks like empowerment but is often abdication. The failure mode shows up late, when it's expensive to fix, because there was no checkpoint designed to catch it earlier.

Over-specifying the implementation defeats the purpose. If you hand someone a task but dictate every step, there's no room left for them to exercise judgment, which is the actual thing you're trying to develop.

The honest trade-off: delegating a stretch task usually costs you more short-term time, in reviewing and coaching, than doing it yourself would. That extra cost is the investment, and it's worth naming rather than pretending delegation is free.

Direct answer

Making model training reproducible across machines and runs requires fixing every source of randomness, enabling deterministic GPU kernel implementations, controlling data-loading order, pinning package versions inside a containerized environment, and logging enough environment metadata alongside every run that a later attempt to reproduce it knows exactly what to reconstruct.

Structured elaboration

• Seed every RNG source: Python's random.seed(), NumPy's np.random.seed(), and the framework's own CPU and per-GPU RNG seeds.
• Enable deterministic operations: PyTorch's torch.use_deterministic_algorithms(True) combined with torch.backends.cudnn.deterministic = True and torch.backends.cudnn.benchmark = False forces slower but bit-reproducible kernel choices.
• Control data loading order: fix the data loader's shuffling seed and, for multi-worker loading, ensure each worker's seed is deterministically derived (e.g. via worker_init_fn).
• Pin environment and versions: pin CUDA, cuDNN, and framework versions, since different versions can produce different numerical results for the same operations.
• Containerization: package the pinned environment (OS libraries, CUDA/cuDNN, Python/framework dependencies) into a versioned, immutable container image (e.g. a tagged Docker image, never :latest) rather than relying on a host machine's environment staying unchanged over time; this is what actually makes "pin the versions" enforceable across different physical machines, since two hosts running the same container image are guaranteed to see the identical software stack, whereas two hosts each independently following a version-pinning document can still drift (a slightly different OS-level library, a manually-installed system package) in ways a lockfile alone doesn't catch.
• Logging environment metadata: at the start of every run, log (to the experiment tracker or alongside the checkpoint) the exact container image tag/digest, the git commit hash of the training code, the full resolved dependency versions (not just the top-level pinned ones, since transitive dependencies can also vary), the hardware used (GPU model/count, driver version), and the seeds used; without this, a run from three months ago that needs to be reproduced or audited has no reliable record of what actually produced it, making "reproducible in principle" (the seeds and code exist) meaningless in practice if nobody can reconstruct which exact environment they applied to.

Worked example

A checklist applied to a PyTorch training run: random.seed(42), np.random.seed(42), torch.manual_seed(42), torch.cuda.manual_seed_all(42), torch.backends.cudnn.deterministic = True, torch.backends.cudnn.benchmark = False, a DataLoader with worker_init_fn deriving each worker's seed from the base seed plus worker ID, run inside a container image tagged train:v1.4.2-cuda12.1-torch2.1.0, with the run's metadata log recording that image tag, the code's git commit hash, and the resolved pip freeze output; running the same image on a different physical machine six months later, using the logged metadata to reconstruct the exact setup, reproduces bit-identical (or numerically identical to floating-point tolerance) loss curves.

Trade-offs & pitfalls

Deterministic mode has a real performance cost, so it's typically enabled specifically for debugging/reproducibility-validation runs. Skipping containerization in favor of "just document the versions" is a common shortcut that looks equivalent on paper but fails in practice, since undocumented OS-level or transitive-dependency drift between two "identically configured" bare-metal hosts is a frequent, hard-to-diagnose source of irreproducibility that containerization eliminates by construction.

The idea in plain words

The system you are considering rests on one belief: given the data you already collect, you can tell which impressions will be clicked better than a simple rule can. That belief can be tested on logged history, with no new system, in days rather than months. An impression is one occasion on which an item was shown to a user, and a click is what some impressions lead to. Train a few models on older logs, score them on newer logs they have never seen (a time-forward test), and compare them with the simplest rule that is already in use (the incumbent rule). If the models barely beat the rule, the premise fails for these features, and you stop before building. The test can kill the premise cheaply; it cannot prove the full system will raise clicks, which needs a live experiment later.

Step 1: pin down the premise as a testable statement

"Click propensity (the probability that a given impression is clicked) can be predicted from today's features" becomes: on impressions from a later period, a model using only features available at the moment of serving ranks clicked impressions above unclicked ones noticeably better than the current simple rule does.

Step 2: build the cheapest credible test

1. Take the logs you have. One row per impression, a label (clicked or not), and the features as they were at that moment. Use only features that will exist at serving time. A feature computed after the click, such as "the item's click count for the day" including this click, leaks the answer and makes any model look brilliant.
2. Split by time, not at random. Train on the earlier weeks and test on the last one or two weeks. A random split lets the model see the same users and items on both sides and overstates what production will deliver.
3. Fit the ladder of baselines. (a) Always predict the average click rate. (b) The incumbent rule, for example each item's past click rate. (c) A simple logistic regression (a basic model that adds up weighted features and turns the total into a probability) on a handful of features. (d) A gradient-boosted tree model (many small decision trees added one after another, each correcting the last).
4. Score with AUC (area under the ROC curve, a plot of how many true clicks the model catches against how many false alarms it raises as you lower the score cut-off; the area equals the probability that a randomly chosen clicked impression gets a higher score than a randomly chosen unclicked one, and 0.5 is no better than a coin flip). Also check calibration (when the model says 5 percent, do about 5 percent click?) and the click rate in the top-scored 10 percent of impressions.
5. Run a sanity check that must fail. Shuffle the labels and retrain. The AUC should come out near 0.5. If it does not, there is leakage or a bug.
6. Slice the result by new versus returning users and by item age, because a model can be good on average and useless where you most need it.

Worked example (assumed numbers and an illustrative readout)

Assume the later-period test set has 200,000 impressions with a 3 percent click rate (plausible range 1 to 8 percent): $200{,}000 \times 0.03 = 6{,}000$ clicks and 194,000 non-clicks.

What chance alone does to AUC. Even a score with no signal at all gets an AUC a little above or below 0.5, because a finite set of clicks and non-clicks can line up by luck. The size of that wobble (its standard deviation, the typical distance from 0.5) shrinks as the counts grow. With no real signal it is about

where $n_1$ is the number of clicks and $n_0$ the number of non-clicks (the 12 is a constant of the standard result for a score with no signal; a reader only needs to know that more clicks and non-clicks make the wobble smaller). A 95 percent range for chance alone is therefore about $0.5 \pm 1.96 \times 0.0038 = 0.5 \pm 0.0074$. An AUC of 0.52 is clearly above chance and may still be worth nothing, so the comparison that matters is against the incumbent rule.

What the system has to be worth. Assume 10 million impressions a month at the 3 percent rate, which is 300,000 clicks, and a value of 0.40 dollars per click (plausible range 0.05 to 1 dollar), so about 120,000 dollars a month. Assume a build cost of 300,000 dollars, running cost of 5,000 dollars a month, and a goal of paying back in 24 months. The monthly gain needed is $300{,}000 / 24 + 5{,}000 = 17{,}500$ dollars, which is $17{,}500 / 120{,}000 \approx 14.6\%$ more clicks than today.

How the two numbers connect. No fixed formula turns an AUC gap into a click gain, because the gain also depends on how the product uses the score (which items get the best slots, how many slots there are). So the AUC gap is a cheap first screen and the 14.6 percent is the target the later steps must show: if the best model cannot even clearly beat the incumbent rule on ranking quality, a 14.6 percent click gain is very unlikely, and if it can, the offline replay (re-ranking past impressions with the model and checking the click rate among the top-scored 10 percent) and then the small live test measure the gain in clicks itself.

Illustrative readouts on the time-forward test set. The columns Readout A and Readout B are two alternative outcomes the same test could produce, shown side by side. The row "Gap over incumbent" is the best model's AUC minus the incumbent rule's AUC (A: 0.662 - 0.640 = 0.022; B: 0.648 - 0.640 = 0.008).

Readout B: the gap of 0.008 is below the 0.02 bar, so the premise is not supported beyond what the incumbent rule already captures. Stop, or spend one more cheap round testing one new feature family. Readout A: a gap of 0.022 clears the bar, so continue to the next cheap step: an offline replay and then a small live test.

What result would make me stop

These thresholds are illustrations. Three checks drive the stop decision: the time-forward split, the comparison with the incumbent rule and the shuffled-label run. A real process then sharpens the call with the interval on the gap (a paired bootstrap: resample the test impressions many times with replacement, score both models on each resample, and see how much the gap varies), the segment results and any shift in the data between training and test periods.

1. The best model beats the incumbent rule by less than 0.02 AUC on the time-forward test. The 0.02 is a judgment, reached like this: chance alone moves one AUC by up to about 0.0074, and the gap between two models scored on the same impressions has its own wobble (the bootstrap measures it), for which that figure gives a first sense of scale. Asking for a gap of 0.02, about 2.7 times that half-range ($0.02 / 0.0074 \approx 2.7$), keeps the bar clearly outside noise without demanding a huge improvement. It should be tuned to what a past launch in your own product taught you about AUC versus clicks.
2. The score drops sharply from the random split to the time split (say 0.71 to 0.66), which says the model is memorising users and items, not learning a lasting pattern.
3. The shuffled-label run is not near 0.5. That is a bug to fix, not a reason to stop, but nothing can be concluded until it is fixed.
4. The model works only where the incumbent already works (returning users) and fails where the business needs it (new users).

Each stop condition is written before the test is run, so a disappointing readout cannot be explained away afterwards.

Judgment

I would run this as a few days of work for one data scientist, commit to the stop rules in advance, and treat a pass as permission to run the next small step, not as proof of the 14.6 percent uplift. Logged clicks reflect what the old system chose to show, and top positions get more clicks regardless of quality, so the live test is the only fair measure of uplift. A failed premise test does not mean clicks are unpredictable in principle; it means these features add too little over the rule you have.

Common mistakes

• Random split with leaked features, giving a flattering AUC.
• Comparing only against a constant, not against the incumbent rule.
• Reading a small AUC gap without a chance range.
• Deciding the stop threshold after seeing the numbers.

Direct answer
Single exponential smoothing maintains one smoothed level, updated each period as a weighted blend of the new observation and the previous smoothed value ($S_t = \alpha y_t + (1-\alpha) S_{t-1}$); the forecast for ALL future periods is simply the final smoothed level $S_n$ (SES has no trend or seasonal component to extrapolate further), and alpha controls how much weight recent observations get versus the smoothed history.

Structured elaboration and worked example (executed, cross-checked against statsmodels)

import numpy as np

def simple_exp_smoothing(values, alpha):
    smoothed = np.empty_like(values, dtype=float)
    smoothed[0] = values[0]                       # initialize at the first observation
    for t in range(1, len(values)):
        smoothed[t] = alpha * values[t] + (1 - alpha) * smoothed[t - 1]
    return smoothed, smoothed[-1]                  # forecast = last smoothed level

Run on [112, 118, 132, 129, 121, 135, 148, 142, 130, 145] with alpha=0.3:

from-scratch smoothed:      [112. 113.8 119.26 122.182 121.827 125.779 132.445 135.312 133.718 137.103]
from-scratch next forecast: 137.103

Cross-checked against statsmodels.tsa.holtwinters.SimpleExpSmoothing with the same alpha (optimized=False): the fitted values matched exactly once accounting for statsmodels' indexing convention (its fittedvalues[t] is the one-step-ahead forecast made BEFORE seeing $y_t$, i.e. it equals our smoothed[t-1], not our post-update smoothed[t]) - after aligning on that offset, max abs diff = 0.0 and both implementations agree on the final forecast, 137.103.

This cross-check caught a genuine implementation bug on the first attempt: an initial version computed "next forecast" as alpha * values[-1] + (1-alpha) * smoothed[-1], which double-applies the last observation (since smoothed[-1] already incorporated it in the recursion), producing 139.47 instead of the correct 137.10. The correct SES forecast is simply the final smoothed level itself, $\hat{y}_{n+h} = S_n$ for every horizon $h \ge 1$ - SES has no mechanism to extrapolate differently at different horizons, which is itself an important limitation to flag (see below).

• Initialization choices: initializing $S_1 = y_1$ (used above) is the simplest and most common convention; an alternative is to initialize with the average of the first few observations, which reduces sensitivity to a noisy first data point at the cost of a slightly more complex setup. The choice mostly matters for a short series; its influence fades geometrically as more observations are smoothed in.
• How alpha controls responsiveness: alpha close to 1 makes the smoothed level track new observations almost immediately (responsive, but passes through more noise); alpha close to 0 makes it change very slowly (smooth, but slow to react to a genuine shift). Alpha is typically chosen by minimizing in-sample or backtested one-step-ahead error rather than picked by hand.

Trade-offs & pitfalls
SES has no trend and no seasonality component - its forecast is FLAT (the same value) at every horizon, which makes it a poor choice for any series with a visible trend or seasonal pattern (that's exactly what Holt's linear method and Holt-Winters extend it to handle). The implementation trap surfaced above generalizes: any time you're tempted to apply "one more update step" to produce a forecast from a recursive smoothing state, check the model's actual mathematical definition of its forecast function rather than assuming intuition about "the next step" is correct - for SES specifically, the forecast function is constant in the smoothed level, not a further-updated value.

Direct answer. Bin both the reference and current samples on the SAME bin edges (derived from the reference distribution's quantiles), convert each bin's count to a share of its total, and sum $(\text{current}-\text{reference})\times\ln(\text{current}/\text{reference})$ across bins, flooring any zero-share bin so the log never blows up.

Code (executed and verified below).

import numpy as np

def psi(reference, current, bins=10, eps=1e-4):
    reference, current = np.asarray(reference), np.asarray(current)
    quantiles = np.quantile(reference, np.linspace(0, 1, bins + 1))
    quantiles[0] -= 1e-9; quantiles[-1] += 1e-9   # make sure the edge values fall inside the range
    ref_counts, _ = np.histogram(reference, bins=quantiles)
    cur_counts, _ = np.histogram(current, bins=quantiles)
    ref_pct = np.where(ref_counts / ref_counts.sum() == 0, eps, ref_counts / ref_counts.sum())
    cur_pct = np.where(cur_counts / cur_counts.sum() == 0, eps, cur_counts / cur_counts.sum())
    contributions = (cur_pct - ref_pct) * np.log(cur_pct / ref_pct)
    return contributions.sum(), contributions

Worked example (recomputed on 5,000-point synthetic samples). Reference vs. an IDENTICALLY-distributed current sample gave PSI = 0.0019, correctly reading as "no meaningful change." Reference vs. a current sample with a shifted mean (+0.5) and inflated spread (std x1.3) gave PSI = 0.2362, landing in the "moderate to large change" band under the common 0.10 / 0.25 rule-of-thumb thresholds (below 0.10: no significant change; 0.10 to 0.25: moderate, investigate; above 0.25: large, likely real drift).

Structured elaboration. Using the REFERENCE distribution's quantiles to define bin edges (rather than fixed-width bins) means every reference bin starts with roughly equal weight, which is what makes the metric behave sensibly across features with very different scales or shapes without hand-tuning bin widths per feature. The epsilon floor exists purely so a bin that got zero observations in either sample doesn't produce a division-by-zero or a $\ln(0)$; it's a numerical guard, not a modeling choice, so keep it small relative to a real bin's typical share.

Trade-offs and pitfalls. Too few bins (say, 4 to 5) can hide a real shift that only affects the tails; too many bins on a modest sample size makes individual bin counts noisy and can produce a PSI that looks alarmingly high purely from sampling noise, not real drift. Always compute PSI on a big enough current sample that each bin has a meaningful count (a common rule of thumb is at least a few dozen observations per bin), and re-derive the reference quantile edges only once (at training time), not every time you recompute PSI, or you lose the ability to compare PSI values across time on a consistent scale.