Data Driven Recommendations and Impact Questions

Covers the end to end practice of using quantitative and qualitative evidence to identify opportunities, form actionable recommendations, and measure business impact. Topics include problem framing, identifying and instrumenting relevant metrics and key performance indicators, measurement design and diagnostics, experiment design such as A B tests and pilots, and basic causal inference considerations including distinguishing correlation from causation and handling limited or noisy data. Candidates should be able to translate analysis into clear recommendations by quantifying expected impacts and costs, stating key assumptions, presenting trade offs between alternatives, defining success criteria and timelines, and proposing decision rules and go no go criteria. This also covers risk identification and mitigation plans, prioritization frameworks that weigh impact effort and strategic alignment, building dashboards and visualizations to surface signals across HR sales operations and product, communicating concise executive level recommendations with data backed rationale, and designing follow up monitoring to measure adoption and downstream outcomes and iterate on the solution.

HardTechnical

24 practiced

You are asked to build, validate, and deploy an uplift model to target users most likely to be positively influenced by an email campaign. Describe in detail the data collection needs (including randomization), model training approach, evaluation metrics (Qini, AUUC), how to avoid leakage, and how you would run an online validation test to confirm uplift before full deployment.

Sample Answer

Situation: I need to build an uplift model to target users most likely to be positively influenced by an email. Below is a production‑grade plan covering data, modeling, evaluation, leakage avoidance, and online validation.

Data collection & randomization- Run a properly randomized RCT: randomly assign users to treatment (email) and control (no email) at the unit-level before any selection. Record randomization seed and assignment probability.- Capture pre‑treatment covariates (demographics, past behavior, recency/frequency/monetary metrics, device, channel preferences) and coarse time context (day, campaign). Log exposure, send timestamp, opens/clicks, conversions and revenue with event timestamps.- Ensure sufficient sample size and power for detecting heterogenous effects; oversample rare segments if needed. Store raw logs immutable for audit.

Model training approach- Framing: estimate Individual Treatment Effect (ITE). Candidate methods: - Two‑model (S-learner/T-learner) baseline: separate outcome models for treated and control and take difference. - Meta‑learners: X‑learner for imbalanced treatment sizes, DR-learner or R-learner for robustness (doubly robust). - Causal forests / generalized random forests to capture nonlinear heterogeneity.- Pipeline: - Use pre‑treatment features only. - Estimate propensity score if using weighting/DR learners. - Cross‑validation stratified by treatment assignment; ensure folds preserve randomization. - Produce point estimates of uplift and uncertainty intervals (e.g., bootstrap or jackknife+).

Evaluation metrics- Global ATE: sanity check.- Qini curve & Qini coefficient: rank users by predicted uplift, compute incremental gain curve relative to random targeting; Qini summarizes area between model and random.- AUUC (Area Under Uplift Curve): equivalent concept; report uplift@k (e.g., uplift@top1%, top10%) and incremental conversion rate or revenue.- Calibration: compare predicted uplift bins vs observed uplift in held-out randomized data.- Statistical tests: compute confidence intervals for incremental metrics (bootstrap or permutation respecting randomization).

Avoiding leakage- Never use post‑treatment features (opens, clicks, downstream events) as predictors.- Temporal leakage: split by randomization time (train on past users, test on later users) and ensure feature engineering only uses data available at decision time.- Preprocess train/test separately to avoid information bleed (fit scalers/encoders on train only).- Log full lineage so any feature can be audited as pre‑treatment.

Online validation before full deployment- Holdout A/B test (champion/challenger): deploy model to a randomized subset, keep another randomized control group and a random targeting baseline. Options: - Targeting test: randomly route users into (1) model-targeted sends, (2) random-targeted sends, (3) no-send control — measure incremental conversion/revenue. - Intent-to-treat validation: compare incremental lift per user among those assigned to receive vs not.- Power the test using expected uplift and conversion variance; run until pre-specified sample or sequential testing with alpha spending to avoid peeking bias.- Monitor primary metric (incremental conversions/revenue), secondary metrics (opens, unsubscribes, complaints), and model calibration drift.- If validated, gradually scale (canary) and continuously monitor real-world uplift; periodically retrain on new randomized data.

Why this works- Randomization gives unbiased ground truth for uplift. DR/meta-learners and causal forests provide robustness and heterogeneity. Qini/AUUC directly measure uplift ranking value. Leakage controls preserve causal validity. A staged online validation ensures the model’s offline promises translate to real lift before full rollout.

HardTechnical

26 practiced

In Python, implement a function that computes the required sample size per group for a two-sided two-proportion z-test. Inputs: baseline conversion p0, desired relative lift (e.g., 0.10 for 10% relative), alpha, power. Return the sample size per group. State assumptions and show formula references in comments. (You may use numpy/scipy.)

Sample Answer

Approach: convert relative lift to absolute p1 = p0*(1+rel_lift). Use standard large-sample (z-test) formula for equal sized groups (two-sided alpha): n = [ (z_{1-alpha/2} * sqrt(2*p_bar*(1-p_bar)) + z_{power} * sqrt(p0*(1-p0)+p1*(1-p1)) )^2 ] / (p1 - p0)^2 This is a common approximation (see Fleiss, Statistical Methods for Rates and Proportions) and assumes normal approximation valid (np and n(1-p) both >= ~5–10).

python

import math
import numpy as np
from scipy.stats import norm

def sample_size_two_proportions(p0, rel_lift, alpha=0.05, power=0.8):
    """
    Compute required sample size per group for two-sided two-proportion z-test.
    Assumptions:
    - Equal sample sizes in both groups
    - Large-sample normal approximation (np, n(1-p) >= ~5-10)
    - Two-sided test
    Formula reference:
    - Fleiss: n = [ (z_{1-alpha/2}*sqrt(2*p_bar*(1-p_bar)) + z_power*sqrt(p0*(1-p0)+p1*(1-p1)))^2 ] / (p1-p0)^2
    """
    if not (0 < p0 < 1):
        raise ValueError("p0 must be in (0,1)")
    p1 = p0 * (1 + rel_lift)
    if not (0 < p1 < 1):
        raise ValueError("resulting p1 must be in (0,1); adjust rel_lift or p0")
    z_alpha = norm.ppf(1 - alpha/2)
    z_beta = norm.ppf(power)
    p_bar = (p0 + p1) / 2.0

    numerator = (z_alpha * math.sqrt(2 * p_bar * (1 - p_bar))
                 + z_beta * math.sqrt(p0 * (1 - p0) + p1 * (1 - p1))) ** 2
    delta = p1 - p0
    n = numerator / (delta ** 2)
    return int(math.ceil(n))

# Example:
# p0=0.1 baseline, 10% relative lift -> p1=0.11, alpha=0.05, power=0.8
# print(sample_size_two_proportions(0.1, 0.10))

Key points:- Time/space: O(1) compute.- Edge cases: very small p0 or extreme lifts may violate normal approx; consider exact binomial methods or simulations for small n or rare events.- Alternatives: use pooled-variance formulas, continuity corrections, or exact power/sample-size via iterative root-finding with scipy.stats.binom for more accuracy.

MediumSystem Design

32 practiced

Design an experimentation platform (high-level) for a mid-size company: include components for randomization, assignment storage, telemetry ingestion, metric computation, guarding against Sample Ratio Mismatch, and an API for product teams to create experiments. Sketch versioning, reproducibility, and how teams get metric reports.

Sample Answer

Requirements:- Functional: create/launch experiments, deterministic user/randomization, store assignments, ingest telemetry, compute metrics, report results via API/UI.- Non-functional: reproducibility, low-latency assignment service, scalable telemetry, audit/versioning, guardrails (Sample Ratio Mismatch, p-hacking controls).

High-level architecture (components & flow):1. Experiment API Service- REST/gRPC for product teams to create experiments (variants, targeting rules, traffic %, start/end, metrics to track).- Validates schema, registers experiment spec in Experiment Registry (versioned).

2. Experiment Registry (metadata + versioning)- Stores immutable versions of experiment specs (ID, version, seed, allocation algorithm, start/end).- Each change creates new version; API returns spec version to clients.

3. Randomization & Assignment Service- Deterministic hashing: H(user_id, experiment_id, version, salt) -> uniform [0,1).- Apply bucketing rules to assign variant.- Returns assignment and records assignment event to Assignment Store (eventually consistent + write-through).- Supports SDKs for client/server side.

4. Assignment Store (audit log)- Append-only store (e.g., Kafka topic + compacted DB / BigQuery) with assignment events: user_id, experiment_id, version, variant, timestamp.- Enables reproducibility by replaying assignments.

5. Telemetry Ingestion Pipeline- Client/server SDKs send events (exposure, conversions, custom events) to ingestion (Kafka).- Stream processing validates schema, enriches with assignment (lookup or join by user_id + recent assignment), tags with experiment_id/version/variant.- Persist raw events to data lake (Object store) and pre-aggregates to OLAP store for metrics.

6. Metric Computation Engine- Batch (daily) + near-real-time pipelines compute metrics by experiment/version/variant.- Metrics computed using exposure-based joins (only consider users with exposure assignment).- Implementations: SQL jobs in warehouse (BigQuery/Redshift) or Spark; also maintain user-level aggregates for unbiased estimates.- Support statistical tests (t-test, bootstrap, Bayesian) and multiple-testing corrections.

7. SRM Guard (Sample Ratio Mismatch)- Compute expected vs observed assignment counts per variant per experiment version.- Run chi-squared test on exposures; if p < threshold, flag experiment and send alerts; surface cause (filtering, instrumentation loss).- Automatic blocking of metric computation or marking results as unreliable until resolved.

8. Reproducibility & Versioning- All inputs are versioned: experiment spec, assignment events, telemetry schema, metric code (stored in repo with commit hash).- Re-runability: can re-compute metrics by selecting experiment version + time window + code version to reproduce published results.

9. Reporting & Access- Results API + UI dashboards per experiment (trend charts, tabular stats, CI, effect sizes, SRM status).- Auto-generated PDF reports and scheduled emails to stakeholders.- Access controls: who can create/publish experiments, who can view results.

Trade-offs & considerations:- Deterministic hashing is simple and stable but requires careful salt/versioning to change allocations.- Server-side assignment gives correctness; client-side reduces latency but needs secure hashing to prevent tampering.- Near-real-time metrics require careful deduplication and stateful joins.

Example deterministic assignment (pseudocode):

python

def assign(user_id, experiment_id, version, buckets):
    score = hash_fn(f"{user_id}:{experiment_id}:{version}") % 100000 / 100000
    for variant, (start, end) in buckets:
        if start <= score < end:
            return variant

This design balances reproducibility, statistical rigor, and product-team usability for a mid-size company.

HardTechnical

28 practiced

You must design an experiment and analysis plan for a global feature rollout across markets that have different seasonal patterns and baselines. Describe blocking/stratification, choice of control groups, adjustments for seasonality (e.g., time-series models), how to pool results, and decision criteria that account for heterogeneous effects.

Sample Answer

Framework: treat this as a stratified, multi-market A/B experiment with pre-planned analysis that explicitly models seasonality and heterogeneity.

Design / Blocking & randomization:- Block markets by seasonality pattern and baseline (e.g., hemisphere, retail vs non-retail, climate, baseline conversion quantiles). Within each block, randomize at the smallest safe unit (users/regions) to treatment vs control to balance unobserved confounders and allow within-block comparisons.- For markets with few clusters, use cluster randomization (regions/cities) and account for cluster-level variance in power calculations.

Control groups:- Use concurrent randomized control groups in each block. For markets with rollout constraints, use staggered rollouts (stepped-wedge) or synthetic controls built from similar markets and pre-period matching.- Maintain holdout duration >= one full seasonal cycle if possible; if not, ensure rich pre-period baseline (several cycles) for time-series adjustment.

Adjustments for seasonality / modeling:- Fit market-level time-series models to estimate counterfactuals: e.g., SARIMA or SARIMAX (with covariates: holidays, promotions), or Bayesian Structural Time Series (BSTS) for markets with complex patterns. Use pre-period to estimate model and forecast post-period counterfactuals; treatment effect = observed − counterfactual.- For daily/hourly metrics, include day-of-week, holiday indicators, and temperature/promotion covariates. Use robust standard errors / bootstrap to account for model uncertainty.

Pooling & heterogeneous effects:- Combine market estimates with a hierarchical (multilevel) model: market-level treatment effects as draws from a population distribution. This supports partial pooling—stabilizes noisy small-market estimates while preserving heterogeneity.- Alternatively, perform random-effects meta-analysis when using independent market-level estimates from time-series fits.

Decision criteria:- Pre-specify primary metric, minimum detectable effect (MDE) by block, and risk thresholds.- Use Bayesian posterior probabilities or frequentist tests with multiplicity control: - Bayesian: require P(effect > business-MDE) > 0.95 to roll out globally; report posterior distribution per market and overall pooled effect. - Frequentist: control false discovery rate for per-market decisions (e.g., Benjamini–Hochberg) and require overall pooled effect significant at alpha (e.g., 0.05) with pre-specified direction.- For heterogeneous results, use decision rules: global rollout only if pooled effect meets criteria and no >X% of markets have negative effects beyond acceptable loss; consider targeted rollout to segments with positive effects.

Diagnostics & robustness:- Pre-registered analysis plan, simulation-based power analysis that incorporates seasonality and intra-cluster correlation.- Placebo/holdout tests, sensitivity to modeling choices (SARIMAX vs BSTS), check pre-period balance and parallel trends, and run uplift by covariates.- Monitor post-launch; implement rollback triggers based on real-time monitoring against predicted counterfactuals.

This plan balances causal identification across seasonal markets, models temporal structure per-market, pools evidence appropriately, and uses explicit rules to act on heterogeneous outcomes.

MediumTechnical

25 practiced

List and explain at least five A/B test diagnostics (e.g., Sample Ratio Mismatch, outlier analysis, baseline imbalance) you would run during or after an experiment. For each diagnostic, describe the SQL or analytic check to perform and what corrective actions you might take if the diagnostic flags an issue.

Sample Answer

Here are six essential A/B test diagnostics, each with the check you’d run (SQL/analytic) and possible corrective actions.

1) Sample Ratio Mismatch (SRM)- What: Treatment/control allocation deviates from expected random split.- Check (SQL): count by variant and compare to expected using chi-square or binomial.

sql

SELECT variant, COUNT(*) AS n
FROM assignments
WHERE experiment_id=123
GROUP BY variant;

Then run a chi-square test on counts.- Action: If SRM significant, investigate logging/launch bugs, targeting rules, duplicate IDs, or non-random assignment. Stop analysis until fixed; consider rerun or exclude affected periods/users.

2) Baseline Imbalance (covariate imbalance)- What: Key pre-treatment metrics differ across groups.- Check (SQL):

sql

SELECT variant,
 AVG(age) AS avg_age,
 SUM(past_purchases) AS total_purchases
FROM users u JOIN assignments a USING(user_id)
WHERE a.experiment_id=123
GROUP BY variant;

Run t-tests or standardized mean differences.- Action: If imbalance, verify randomization, stratify or adjust with regression (covariate adjustment), or use blocking/stratified randomization in future.

3) Pre-period metric drift / Trending- What: Metrics change over time differently by variant (time confounder).- Check (SQL): compute daily counts/metrics by variant and plot or test interaction.

sql

SELECT assign_date, variant, COUNT(*) AS n
FROM assignments WHERE experiment_id=123
GROUP BY assign_date, variant;

- Action: If drift, control for time (time fixed effects), restrict to stable window, or retrace deployment that caused temporal bias.

4) Outlier influence / heavy-tail users- What: A few users dominate metric (e.g., revenue).- Check (SQL):

sql

SELECT variant, PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY revenue) AS p99,
 SUM(revenue) AS total_rev
FROM events e JOIN assignments a USING(user_id)
WHERE a.experiment_id=123
GROUP BY variant;

Compare mean vs median and run winsorized analyses.- Action: Report robust metrics (median, trimmed mean, bootstrapped CI), cap/winsorize, or analyze both with/without outliers.

5) Incomplete/late data (missingness)- What: Some events not logged or delayed, differing by variant.- Check (SQL):

sql

SELECT variant,
 COUNT(DISTINCT user_id) AS users_with_events,
 COUNT(*) AS total_events
FROM events e JOIN assignments a USING(user_id)
WHERE a.experiment_id=123 AND event_time < NOW() - INTERVAL '48 hours'
GROUP BY variant;

Compare event completion rates and time-to-event distributions.- Action: Wait for data to stabilize, backfill missing logs, exclude recent window, or flag for instrumentation fixes.

6) Peeking / Multiple testing inflation- What: Repeated looks inflate false positives.- Check (analytic): Track number/timing of significance checks; compute alpha spending or use sequential testing boundaries.- Action: Apply proper sequential methods (alpha spending, Bayesian monitoring, or Bonferroni for multiple metrics), or revert claims if improper peeking occurred.

For each diagnostic, document findings, rerun after fixes, and report both primary (pre-registered) and sensitivity analyses.

Unlock Full Question Bank

Get access to hundreds of Data Driven Recommendations and Impact interview questions and detailed answers.

Join thousands of developers preparing for their dream job.