DoorDash Data Scientist Interview Preparation Guide - Mid Level

Situation: A preprocessing bug from a junior data scientist caused bad model inputs and a production incident that affected downstream reports for our sales team last quarter.

Task: As their mentor and the project owner, I needed to address the mistake with the individual, restore stakeholder trust, and implement fixes so we wouldn't repeat it—while keeping the culture blameless.

Action:

• Immediate: I paused the pipeline, reverted to the last stable model, and communicated transparently to stakeholders with a timeline for resolution.
• One-on-one with the junior DS: I used a coaching approach—asked them to walk me through their code and thought process, then we mapped the root cause together (a normalization step applied in the wrong order). I focused on facts, asked what they would do differently, and reinforced responsibility without shaming.
• Learning: We ran a joint code review, added unit tests for preprocessing (schema checks, range tests), and created a small checklist for deploy readiness (data validation, model input assertions).
• Team-level: I led a blameless postmortem documenting timeline, causes, and action items; shared learnings in a short demo; and introduced pair-review for any preprocessing changes and CI checks that validate data schemas before deployment.

Result: The junior engineer owned the fix and implemented tests; similar incidents dropped to zero in the next two quarters. Stakeholders appreciated the transparent communication. The team adopted the checklist and CI validations, which reduced production data issues by 40% and improved our onboarding for new hires.

Learning: Mistakes are teaching moments—focus on systems and safeguards, coach for growth, and make process changes that remove single-person failure points.

Direct answer
Draw each trial's configuration independently from a seeded random generator (per-parameter distributions for continuous ranges, uniform choice for discrete options), evaluate via cross-validation, and track the best; fixing the seed on the RNG used for sampling makes the exact sequence of trials, and therefore the result, reproducible.

Structured elaboration
Each parameter in the space specification carries its own sampling rule (a log-uniform range for something like a regularization strength, a plain integer range for tree depth, a discrete choice list for a categorical option); the tuner draws one full configuration per trial from all these rules using a single seeded RandomState, so re-running with the same seed reproduces the identical sequence of configurations tried, not just a statistically similar one.

Worked example (executed)

import numpy as np

def random_search_tuner(estimator_cls, param_space, X, y, n_trials=15, cv=4, seed=0):
    rng = np.random.RandomState(seed)
    best_score, best_params = -np.inf, None
    for _ in range(n_trials):
        params = {}
        for name, spec in param_space.items():
            if spec[0] == "int_uniform":
                params[name] = int(rng.randint(spec[1], spec[2] + 1))
            elif spec[0] == "log_uniform":
                params[name] = float(np.exp(rng.uniform(np.log(spec[1]), np.log(spec[2]))))
            elif spec[0] == "choice":
                params[name] = spec[1][rng.randint(len(spec[1]))]
        est = estimator_cls(**params, random_state=seed)
        score = cross_val_score(est, X, y, cv=cv).mean()
        if score > best_score:
            best_score, best_params = score, params
    return best_params, best_score

Run against a RandomForestClassifier search space (n_estimators int-uniform 20-200, max_depth int-uniform 2-10) on synthetic classification data with 12 trials: found best params {'n_estimators': 159, 'max_depth': 4} at CV accuracy 0.88. Re-running with the identical seed reproduced the exact same trial sequence and result.

Trade-offs & pitfalls
Passing random_state=seed (the same fixed value) to the estimator itself, as done here for simplicity, means every trial's OWN internal randomness (e.g. a random forest's bootstrap sampling) is identical across trials too, which is fine for reproducibility but means trial-to-trial score differences are due entirely to the hyperparameters, not model-internal randomness; a more thorough implementation would vary the estimator's own seed per trial (from the same master RNG) and average over a couple of estimator seeds per configuration to separate hyperparameter effects from model-training noise.

Situation: In constrained experiments (stratified randomization, availability-based assignment, or forced treatment caps) units have unequal probabilities of treatment. To get unbiased effect estimates and correct SEs you must use probability-weighted estimators and variance formulas that respect the assignment mechanism.

Estimators:

• Horvitz–Thompson (HT): estimate the mean potential outcome under treatment as (1/N) * sum_i (Z_i Y_i / p_i), where Z_i is treatment indicator and p_i the unit’s treatment probability. HT is unbiased for finite-population estimands.
• Hajek (IPW normalized): \hat\tau = [sum_i Z_i Y_i / p_i] / [sum_i Z_i / p_i] − [sum_i (1−Z_i) Y_i / (1−p_i)] / [sum_i (1−Z_i)/(1−p_i)]. More stable; approximately unbiased and often lower MSE.

Variance estimation:

• For HT/IPW under independent assignments use Var(HT)=sum_i sum_j ( (π_ij − π_i π_j) / (π_i π_j) ) y_i y_j / N^2, where π_i, π_ij are 1- and 2-unit inclusion probs. In practice compute linearized (sandwich) variance or use robust heteroskedasticity-consistent SEs.
• With stratified or blocked randomization, compute variance within strata and combine (stratum-weighted variances) using the known randomization distribution.
• For clustered assignment or interference, use cluster-robust (Huber-White) variance estimators or compute randomization-based permutation/empirical variances at cluster level.

Practical considerations:

• Weight capping/trimming: cap extreme weights (e.g., p_min) or use stabilized weights to reduce variance; report sensitivity analyses because capping introduces bias-variance tradeoff.
• Covariate adjustment: incorporate prognostic covariates via weighted regression (IPW + outcome model) or doubly robust estimators for efficiency and protection against misspecification.
• Effective sample size: monitor ESS = (sum w_i)^2 / sum w_i^2 to assess weight variability.
• Diagnostics: report distribution of p_i and weights, balance after weighting, and run randomization-based checks (placebo outcomes, permutation tests) when possible.
• Implementation: compute p_i from design (availability rules, strata, caps); do not estimate p_i from data unless unavoidable—if so, account for estimation in SEs.

Takeaway: Use HT/Hajek/IPW aligned to design, compute variances using randomization-aware formulas or robust/clustered SEs, cap/stabilize weights carefully with sensitivity checks, and prefer doubly-robust adjustments for efficiency.

Direct answer

A naive rolling z-score (mean and standard deviation over the trailing N days, current day included or not) breaks down whenever the metric has weekly seasonality, because it compares a day to a mixed population of weekdays and weekends rather than to its own kind of day. The fix is to compute the baseline per day-of-week: compare a Monday only to prior Mondays.

Structured elaboration

• Naive approach: AVG/STDDEV_SAMP over ROWS BETWEEN 6 PRECEDING AND CURRENT ROW, ordered by date, then z = (value - roll_mean) / roll_std.
• Why it breaks: a 7-day window spanning both weekday and weekend values estimates a standard deviation that reflects the SIZE of the seasonal swing, not genuine day-to-day noise. That inflated standard deviation does two things at once: it makes structurally-low weekend days score a moderate, harmless-looking negative z (a false signal of concern where none exists), and it makes a real anomaly on a weekday look small relative to that inflated spread (a missed true anomaly).
• Fix: partition the window by day-of-week, ordered by date, so each day is compared only to its own recent history of the same weekday: PARTITION BY EXTRACT(dow FROM day) ORDER BY day ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING.
• Remaining pitfalls even after the fix: a day-of-week baseline still needs several prior instances of that weekday before its own standard deviation is meaningful (same minimum-history guard as any rolling statistic), and it will not by itself catch a genuine change that happens to be a full week long, since day-of-week baselines are blind to same-weekday drift over a multi-week span.

Worked example

Five weeks of a metric with clean weekly seasonality (weekdays around 100-104, weekends around 40-42) and ONE planted real anomaly: a Friday reading of 130 where every other Friday in the series was 100-104. Concretely:

Naive rolling 7-day z-score on that Friday:

SELECT day, metric_value,
  AVG(metric_value) OVER w AS roll_mean,
  STDDEV_SAMP(metric_value) OVER w AS roll_std,
  (metric_value - AVG(metric_value) OVER w) / STDDEV_SAMP(metric_value) OVER w AS z
FROM daily_metric
WINDOW w AS (ORDER BY day ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)

gives roll_mean = 89.00, roll_std = 33.98, z = 1.21 -- computed directly from the trailing 7 calendar days ending on that Friday (week 4's Sat, Sun: 41, 42, then week 5's Mon-Fri: 104, 102, 103, 101, 130): mean = (41+42+104+102+103+101+130)/7 = 623/7 = 89.00, and the sample standard deviation of those same 7 values is 33.98, comfortably under any reasonable alert threshold. The genuine 25-30% spike is missed, because the naive window's standard deviation is inflated to 33.98 by mixing weekday and weekend values.

Day-of-week-adjusted baseline on the identical Friday:

SELECT day, metric_value,
  AVG(metric_value) OVER w AS dow_mean,
  STDDEV_SAMP(metric_value) OVER w AS dow_std,
  (metric_value - AVG(metric_value) OVER w) / STDDEV_SAMP(metric_value) OVER w AS z_dow
FROM daily_metric
WINDOW w AS (PARTITION BY dayofweek(day) ORDER BY day ROWS BETWEEN 4 PRECEDING AND 1 PRECEDING)

gives dow_mean = 102.00, dow_std = 1.826, z_dow = 15.34 -- from the prior 4 Fridays in the table (weeks 1-4: 101, 103, 100, 104): mean = 408/4 = 102.00, sample standard deviation 1.826, and z_dow = (130 - 102.00) / 1.826 = 15.34. Comparing only to prior Fridays gives a tight, honest standard deviation and correctly flags this as a dramatic outlier. On week 5's Saturday (41), an ordinary non-anomalous weekend day in the same table: the naive method's trailing 7-day window shifted one day later ([42, 104, 102, 103, 101, 130, 41], the same mean 89.00 and std 33.98 as above) reports z = (41 - 89.00) / 33.98, which works out to roughly -1.41 (looks like a mild, spurious negative signal), while the day-of-week method, using the prior 4 Saturdays (41, 40, 42, 41, mean 41.00, std 0.816), reports z_dow = (41 - 41.00) / 0.816 = 0.00 exactly (correctly recognizing that value as completely normal for that day type).

Trade-offs & pitfalls

• The naive method is not just "less accurate", it fails in both directions at once: false positives on structurally low days, false negatives on real anomalies, because both failures share the same root cause (an inflated, conflated standard deviation).
• Day-of-week baselining costs sample size: with only 5 weeks of history, the day-of-week window has just 4 prior points, which is a thin standard deviation estimate. In practice you want at least 6-8 weeks of history before trusting this baseline, and you should still apply a minimum-history guard exactly as in a simpler per-entity outlier check.
• Other seasonality shapes (monthly billing cycles, holiday effects) need the equivalent treatment: partition by whatever unit of seasonality actually governs the metric, not just day-of-week by default.

The retention window you choose as the primary KPI encodes a bet about what kind of value your product delivers, and 7-day, 30-day, and 90-day retention each optimize a team toward a different, sometimes conflicting, set of product decisions.

Comparison across windows

Short-term engagement optimization versus long-term monetization

Optimizing for 7-day retention alone can reward onboarding gimmicks and notification-driven re-engagement that inflate the near-term number without building a durable habit, the classic 'looks retained, isn't really retained' trap; a product can show excellent 7-day numbers while 90-day retention (and, downstream, monetization from users who actually stick around long enough to convert or renew) quietly craters. Conversely, only watching 90-day retention means you learn about a broken onboarding flow three months too late to act on it cheaply.

Cohort stability and experiment interpretation

A/B tests analyzed on a 7-day window reach statistical conclusions fast but risk declaring a win on an effect that reverses by day 90 (novelty effects are a classic culprit); tests analyzed only on 90-day windows are more trustworthy but require holding an experiment open, and its opportunity cost, for three months. The common resolution is a layered approach: use 7-day retention as an early, cheap SIGNAL to kill clearly bad ideas fast, but require 30-day (and periodically 90-day) confirmation before declaring a genuine, durable win and rolling out broadly.

Trade-offs and pitfalls

Picking a single window as THE metric, rather than treating the three as a layered decision system, is the core mistake; a mature team reports all three together and treats disagreement between them (strong 7-day, weak 90-day) as itself a diagnostic signal, not noise to average away.

What I would look at
Spill to disk usually means the engine ran out of memory for a large sort or window step. Since the query has several window functions, I would inspect whether they share the same PARTITION BY and ORDER BY clauses. If not, the engine may sort the same billion-row fact table multiple times.

Tuning steps

• Keep only the columns needed before the window step.
• Filter as early as possible, especially on date ranges.
• Align window definitions so the engine can reuse one sort.
• Pre-aggregate to the lowest grain that still preserves the result.
• Check table clustering or partitioning on the hottest filters, such as day.

Example
If one window is over (customer_id, day) and another is over (customer_id, country, day), I would ask whether the business result really needs both raw row-level windows. If a daily summary is enough, I would stage one row per customer, country, and day first, then run the windows on that smaller set.

Why this helps
You are not changing the answer, just reducing the amount of data that must be sorted and carried through each window calculation. That is usually the fastest way to stop spills.

Direct answer

"MRR as of a date, prorating partial months" has two legitimate but different conventions, and the definition needs pinning before writing SQL. This answer uses month-to-date proration: an active subscription contributes monthly_price x (days elapsed so far in the calendar month containing as_of_date) / (days in that month), capped at as_of_date and floored at the later of the month's start or the subscription's own start date. The alternative convention, full run-rate (count the full monthly price the moment a subscription exists, prorating only for accounting revenue recognition, not the headline MRR figure) is common at many SaaS companies and gives a different number for any subscription that started mid-month; state clearly which one you're using.

Structured elaboration

Approach:

1. Filter to subscriptions active on as_of_date: start_date <= as_of_date AND (end_date IS NULL OR end_date >= as_of_date).
2. For each, compute the overlap window within the calendar month containing as_of_date: overlap_start = GREATEST(start_date, month_start), overlap_cap = LEAST(month_end, as_of_date).
3. Prorate: monthly_price_cents * (overlap_cap - overlap_start + 1) / days_in_month, summed across active subscriptions.

Key points:

• days_in_month must come from the actual calendar month (DATE_TRUNC plus one month minus a day), not a hardcoded 28-31; getting this wrong systematically skews every partial-month subscription in the same direction.
• Keep the whole calculation in integer cents through the arithmetic, converting to dollars only at final display, to avoid compounding floating-point rounding across many subscriptions.

Worked example

Seed data, as_of_date = 2026-03-15 (March has 31 days):

Point-in-time prorated MRR:

WITH params AS (
  SELECT
    DATE '2026-03-15' AS as_of_date,
    DATE_TRUNC('month', DATE '2026-03-15')::DATE AS month_start,
    (DATE_TRUNC('month', DATE '2026-03-15')::DATE + INTERVAL '1 month' - INTERVAL '1 day')::DATE AS month_end
),
active_subs AS (
  SELECT s.subscription_id, s.customer_id, s.start_date, s.monthly_price_cents, p.*
  FROM subscriptions s
  CROSS JOIN params p
  WHERE s.start_date <= p.as_of_date
    AND (s.end_date IS NULL OR s.end_date >= p.as_of_date)
),
calc AS (
  SELECT
    subscription_id, customer_id, monthly_price_cents,
    GREATEST(start_date, month_start) AS overlap_start,
    LEAST(month_end, as_of_date) AS overlap_cap,
    (month_end - month_start + 1) AS days_in_month
  FROM active_subs
)
SELECT
  subscription_id, customer_id, monthly_price_cents, overlap_start, overlap_cap,
  (overlap_cap - overlap_start + 1) AS active_days_in_month,
  days_in_month,
  ROUND(monthly_price_cents * (overlap_cap - overlap_start + 1)::DOUBLE / days_in_month, 2) AS prorated_mrr_cents
FROM calc
ORDER BY subscription_id;

Result (run in DuckDB; T-SQL/Postgres users would use DATEADD/EOMONTH or date_trunc equivalents, the arithmetic is identical):

subscription_id | customer_id | monthly_price_cents | overlap_start | overlap_cap | active_days_in_month | days_in_month | prorated_mrr_cents
----------------+-------------+----------------------+---------------+-------------+-----------------------+----------------+---------------------
              1 |         101 |                 5000 |    2026-03-01 |  2026-03-15 |                    15 |             31 |             2419.35
              2 |         102 |                 3000 |    2026-03-10 |  2026-03-15 |                     6 |             31 |              580.65
              5 |         104 |                 6000 |    2026-03-01 |  2026-03-15 |                    15 |             31 |             2903.23
              6 |         105 |                 1000 |    2026-03-01 |  2026-03-15 |                    15 |             31 |              483.87

Subscriptions 3 and 4 correctly don't appear, they aren't active on the as_of_date (subscription 3 churned March 5; subscription 4 was superseded February 28). Summed and converted to dollars: $63.87 total prorated MRR as of March 15.

Extension: the new / expansion / churned MRR bridge

A point-in-time figure alone doesn't say WHY MRR is moving. The standard SaaS decomposition classifies subscription-level events within the month using each subscription's FULL monthly price (a different, non-prorated convention from the point-in-time figure above; the two must never be added together or expected to reconcile directly).

Detecting continuity (an upgrade, not a churn-then-new pair) uses LAG() per customer to compare each subscription to that same customer's immediately prior one, treating it as a continuation only when the previous subscription's end_date is exactly one day before the new one's start_date:

WITH ordered AS (
  SELECT
    subscription_id, customer_id, start_date, end_date, monthly_price_cents,
    LAG(monthly_price_cents) OVER (PARTITION BY customer_id ORDER BY start_date) AS prev_price_cents,
    LAG(end_date) OVER (PARTITION BY customer_id ORDER BY start_date) AS prev_end_date,
    LEAD(start_date) OVER (PARTITION BY customer_id ORDER BY start_date) AS next_start_date
  FROM subscriptions
),
classified AS (
  SELECT
    subscription_id, customer_id, start_date, end_date, monthly_price_cents, prev_price_cents,
    CASE
      WHEN start_date BETWEEN DATE '2026-03-01' AND DATE '2026-03-31' AND prev_price_cents IS NULL THEN 'new'
      WHEN start_date BETWEEN DATE '2026-03-01' AND DATE '2026-03-31'
           AND prev_end_date = start_date - INTERVAL 1 DAY AND monthly_price_cents > prev_price_cents THEN 'expansion'
      WHEN start_date BETWEEN DATE '2026-03-01' AND DATE '2026-03-31'
           AND prev_end_date = start_date - INTERVAL 1 DAY AND monthly_price_cents < prev_price_cents THEN 'contraction'
      ELSE NULL
    END AS start_event,
    CASE
      WHEN end_date BETWEEN DATE '2026-03-01' AND DATE '2026-03-31'
           AND (next_start_date IS NULL OR next_start_date <> end_date + INTERVAL 1 DAY) THEN 'churned'
      ELSE NULL
    END AS end_event
  FROM ordered
)
SELECT 'new' AS event_type, subscription_id, customer_id, monthly_price_cents AS mrr_delta_cents
FROM classified WHERE start_event = 'new'
UNION ALL
SELECT 'expansion', subscription_id, customer_id, (monthly_price_cents - prev_price_cents)
FROM classified WHERE start_event = 'expansion'
UNION ALL
SELECT 'contraction', subscription_id, customer_id, (monthly_price_cents - prev_price_cents)
FROM classified WHERE start_event = 'contraction'
UNION ALL
SELECT 'churned', subscription_id, customer_id, -monthly_price_cents
FROM classified WHERE end_event = 'churned'
ORDER BY event_type, subscription_id;

Result:

event_type | subscription_id | customer_id | mrr_delta_cents
-----------+------------------+-------------+-----------------
churned    |                3 |         103 |           -2000
expansion  |                5 |         104 |            2000
new        |                2 |         102 |            3000

Customer 104's plan swap is correctly classified as expansion (+$20), not as a churn-then-new pair, because subscription 4's end_date (Feb 28) is exactly one day before subscription 5's start_date (Mar 1), the continuity condition. Customer 103's subscription (subscription 3) is churned (-$20) because nothing replaces it. Customer 102 is genuinely new (+$30), a first subscription with no prior one for that customer.

Net MRR movement for March: SUM(mrr_delta_cents) = 3000 cents, +$30.00 net new MRR for the month.

Complexity

The point-in-time query is a single filter and scalar computation per active subscription, O(n). The bridge adds two window-function passes (LAG/LEAD partitioned by customer), O(n log n) for the per-customer sort, then a constant number of UNION ALL branches over the already-classified rows. Both are cheap at any realistic subscription-table size; the only real cost driver is whether customer_id and start_date are indexed for the partition/sort.

Edge cases

• A subscription active exactly on as_of_date with start_date = as_of_date: overlap_start = overlap_cap = as_of_date, one active day, correctly prorated to a single day's worth of the monthly price, not zero.
• Leap years and variable month lengths: days_in_month is computed from the actual calendar month, not hardcoded, so February in a leap year prorates correctly against 29 days.
• A customer with two subscriptions active at once (e.g., a base plan plus a separately-billed add-on) breaks the "one active subscription per customer" assumption this bridge logic relies on; each subscription line needs to be classified independently rather than assuming a single prior subscription per customer.
• The one-day-gap continuity rule is a specific assumption about this business's billing semantics (old plan ends the day before the new one starts). A business whose old and new subscriptions overlap by a day, or cut over same-day, needs that condition adjusted to match its actual data model; state the assumption explicitly whenever this query ships.

Trade-offs & pitfalls

• The point-in-time prorated MRR ($63.87) and the bridge's net movement (+$30.00) are two different, non-reconcilable numbers built on different conventions (prorated vs full monthly price). Presenting them on the same dashboard without labeling which is which invites a stakeholder to try to reconcile them and conclude the numbers are broken.
• The continuity/gap detection is a judgment call encoded in SQL, not a SQL universal. Every business's billing system handles plan swaps slightly differently; verify the actual gap convention against real data before trusting the classification, rather than assuming the one-day rule used here transfers unchanged.
• This model assumes one active subscription per customer at a time. Multi-subscription accounts need every subscription LINE classified on its own, or a genuine new add-on gets misclassified as an expansion of an unrelated existing plan.
• Keep monetary values in integer cents through every intermediate step; converting to floating-point dollars early and summing many rows is a classic source of cents-level drift that shows up as "why doesn't this tie out to the penny" in a finance review.

Direct answer

A data-quality rule documentation template needs, at minimum: a plain-language description, the detection logic (the actual query or code implementing the check), severity, an owner, remediation steps, and worked examples of both a passing and a failing record, stored somewhere discoverable and versioned so it stays accurate as the underlying pipeline evolves.

Structured elaboration

• Description: a plain-language statement of what the rule checks and why it matters, written for someone who is not the rule's author.
• Detection logic: the literal, runnable check (SQL, a framework expectation, a script), not a paraphrase of it, so anyone can verify exactly what is being tested rather than relying on the description alone, which can drift out of sync with the actual implementation.
• Severity and owner: an explicit escalation path, who gets paged, and how urgently, when this rule fails.
• Remediation steps: what to actually do when the rule fires, not just what the rule detects; a rule with no documented remediation path is only half-useful.
• Worked examples: a concrete passing record and a concrete failing record, since an abstract description of "what counts as a violation" is often ambiguous at the edges in a way a real example resolves immediately.
• Discoverability and staying current: store the template alongside the pipeline code itself (versioned in the same repository) rather than in a separate wiki that drifts out of sync, and treat a change to the detection logic as requiring an update to the documentation in the same code review, not as a separate, easily-forgotten step.

Worked example

A rule for "duplicate order detection": description = "flags orders with the same (customer_id, order_total, order_date) appearing more than once, which usually indicates a retry-induced duplicate insert rather than a genuine repeat purchase"; detection logic, inlined rather than just referenced, so the documentation and the runnable check are the same artifact:

WITH ranked AS (
  SELECT order_id, customer_id, order_total, order_date, created_at,
         ROW_NUMBER() OVER (
           PARTITION BY customer_id, order_total, order_date
           ORDER BY created_at
         ) AS rn
  FROM orders
)
SELECT * FROM ranked WHERE rn > 1;

Every row this returns has rn > 1, meaning it is not the first-created row within its (customer_id, order_total, order_date) group, so it is flagged as a probable duplicate rather than the original; severity = medium, does not block the pipeline, routes to a daily digest rather than paging; owner = the orders-pipeline team; remediation = "confirm via the source system's request ID whether this is a genuine retry duplicate, and if so, mark the later record for exclusion in the next reconciliation pass"; worked examples = one clearly-duplicate pair and one legitimate same-day repeat purchase that should NOT be flagged, illustrating exactly where the boundary of the rule lies.

Trade-offs and pitfalls

The template's biggest practical risk is documentation drift: a rule's underlying SQL gets updated during an incident fix, but the corresponding documentation is not, and six months later the documented detection logic no longer matches what actually runs, misleading whoever reads it next. Requiring the documentation update as part of the same pull request that changes the detection logic, enforced by convention or by a lightweight CI check, is what actually keeps the two in sync rather than relying on someone remembering to update a separate wiki page.

Goal: produce an offline protocol that yields unbiased (or corrected) estimates of ranking metrics from click logs affected by position bias.

1. Setup and basic idea

• Clicks c_i on item at position i are influenced by relevance r and exposure e_i (position bias). Naïve CTR/ranking metrics are biased toward top positions.
• Use Inverse Propensity Scoring (IPS) to reweight observed clicks by 1/propensity to correct for exposure differences.

2. IPS estimator

• For a binary click signal, estimate expected utility of a policy π (re-ranking) from logged policy π0:
  IPS(π) = (1/N) Σ_{t, i} w_{t,i} · c_{t,i}, where w_{t,i} = I[item_i under π would be shown at pos j] · (π(pos=j | context,item)/π0(pos=j | context,item)) often simplified to 1 / propensity_{t,j} if evaluating deterministic re-rankings.
• For ranking metrics like DCG, compute weighted gain: weighted_DCG = Σ_j (gain(item_j) * discount(j) * I[click observed]) / propensity(j).

3. Propensity estimation

• Best: instrument randomized exposure (shuffle or position-randomization) to directly measure propensity p(j | context). If not possible, estimate via click models:
  • Position-Based Model (PBM): P(click|item, pos) = α_item * β_pos. Fit β_pos by likelihood / EM.
  • Logistic regression / IPS with features: model propensity as function of pos and context.
• Validate propensities (e.g., holdout randomization checks).

4. Variance and bias trade-offs: practical techniques

• IPS is unbiased if propensities correct, but high variance when propensities are small (top-heavy). Mitigations:
  • Clipping: replace weights w = min(w, w_max) reduces variance at cost of bias.
  • Self-normalized IPS (SNIPS): SNIPS = Σ w·y / Σ w reduces variance and stabilizes.
  • Doubly Robust (DR) estimators: combine a direct model μ(x) for reward with IPS: DR = μ + (w)(y - μ): lower variance and remains unbiased if either model or propensities correct.
  • Stratify by propensity or use control variates.
• Practical choices: collect some randomized impressions to get reliable propensities; use clipping/winsorizing and DR for production evaluation.

5. Interleaving (complementary technique)

• Interleaving compares two rankers by combining their lists into a single impression and attributing clicks to rankers. In online/offline logged randomized settings, interleaving yields low-variance pairwise comparisons without full propensity modeling. Offline interleaving requires logs with per-item exposure randomness or logged pairwise assignments.
• Use interleaving for quick A/B-like comparisons; use IPS/DR for absolute metric estimation.

6. Protocol checklist

• Instrument small amount of randomization to estimate propensities.
• Fit PBM or context-aware propensity model; validate.
• Use SNIPS/DR and clipping to control variance.
• Report confidence intervals (bootstrap or analytic variance for IPS).
• For pairwise decisions, use interleaving when feasible.

This yields principled, practical offline evaluation that balances unbiasedness and variance for ranking with position bias.

Direct answer

Build a decision framework around three questions: how much does the predictor actually matter to the model (measured, not assumed), how much would it cost to actually collect the missing 50%, and is there a cheaper alternative that gets most of the same signal, then let the answer to those three drive whether you drop, impute, or find a proxy, rather than defaulting to whichever option is easiest to implement.

Structured elaboration

Step 1, information value analysis: before deciding anything, measure how much this specific feature actually contributes to model performance using only the 50% of rows where it's present (train with and without it on that subset, compare); a feature that stakeholders BELIEVE is important but that measurably adds little to model performance changes the whole conversation, since the "important" framing may be a business assumption worth challenging with data rather than accepting at face value.

Step 2, collection cost: get a real estimate of what it would cost (in money, time, or process change) to actually fill the gap by collecting the missing 50%, rather than working around it; sometimes the honest answer is that fixing the underlying data-collection process is cheaper and more valuable long-term than any imputation workaround.

Step 3, model sensitivity to imputation choice: if dropping the column and collecting real data both prove infeasible in the near term, test how SENSITIVE the model's predictions actually are to different imputation strategies for this feature (mean/median versus a more sophisticated model-based imputation), since a feature the model is highly sensitive to deserves more careful imputation than one it barely uses even when present.

Step 4, proxy features: investigate whether a correlated, more-complete feature (present for a higher fraction of rows) can substitute for most of the missing predictor's signal, a proxy that's 90% as informative but present for 95% of rows may be strictly better for the model than the "true" feature that's only 50% present, worth quantifying explicitly rather than assuming the original feature is irreplaceable just because a stakeholder named it first.

Worked example

If information-value analysis shows the 50%-missing predictor improves validation AUC by only 0.003 when included (versus excluded) on the subset where it's present, that's strong evidence against investing heavily in fixing its missingness, regardless of how strongly a stakeholder BELIEVES it matters; conversely, if it improves AUC by 0.04, that's a real signal worth the cost of either better collection or careful imputation.

Trade-offs and pitfalls

The single most valuable habit in this whole framework is separating "the stakeholder's BELIEF that a feature matters" from "the MEASURED contribution that feature actually makes," since these can and do diverge, and presenting the measured evidence back to the stakeholder, rather than either dismissing their intuition or accepting it uncritically, is usually what actually moves the decision forward productively.