Consultative Discovery and Requirements Gathering Questions

Hypotheses

• Null (H0): Showing the recommendation does not change the repeat-purchase rate within 30 days (p_treatment = p_control).
• Alternative (H1): Showing the recommendation increases the repeat-purchase rate within 30 days (p_treatment > p_control).

Primary metric

• Primary: Repeat-purchase rate within 30 days per user (binary: did the user make another purchase within 30 days of index purchase). Report absolute uplift and relative lift.
• Secondary: Revenue per user (RPU) within 30 days, average order value for repeat orders, click-through and add-to-cart rates on recommended items, negative signals (uninstalls, complaints).

Minimal sample size considerations (power calc for proportions)

• Choose α = 0.05 (two-sided) and power = 0.8 (β = 0.2).
• Need baseline p0 (control repeat rate) and Minimum Detectable Effect (MDE) as absolute uplift you care about.
• Sample size formula (approx for two proportions):
  n_per_arm ≈ [ (z_{α/2} √(2 p̄ (1−p̄)) + z_{β} √(p0(1−p0)+p1(1−p1)) )^2 ] / (p1−p0)^2
  where p̄=(p0+p1)/2.
• Example: if p0=0.10 and we want MDE = 0.02 (to 0.12), z_{α/2}=1.96, z_β=0.84 → n ≈ 3,840 users per arm (so ~7,680 total).
• If metric is continuous (e.g., RPU), use t-test sample size with baseline mean, std dev, and desired absolute change.

Practical adjustments

• Inflate sample by ~10–20% for non-compliance, instrumentation loss, and attrition.
• If randomization unit is session or device, adjust for intra-class correlation (effective sample size reduction) using design effect: n_effective = n_observed / (1 + (m−1) ICC), where m is cluster size.

Dependencies / questions to clarify before starting

• Randomization unit: user-level (recommended) vs session/device/account? Are users shared across devices?
• Definition of repeat purchase: timeframe (30 days?), what counts as a purchase (refunds, fraudulent orders)?
• Baseline rate & historical variance: what is current repeat-purchase rate and variability by cohort/segment?
• Traffic & timeframe: expected eligible users per day/week; any business seasonality or upcoming promotions?
• Exposure logic: how and when is recommendation shown (frequency caps, position, UI variants)? Are we guaranteed exposure when assigned?
• Instrumentation: Are recommendation impressions, clicks, cart adds, and purchases reliably logged and linkable to user IDs?
• Overlap with other experiments or campaigns: any concurrent tests or marketing that could confound results?
• Segmentation needs: Do stakeholders want results overall and by key segments (new vs returning users, geography, product category)?
• Guardrails: acceptable negative impacts (e.g., lower AOV or increased complaints), rollback thresholds.
• Privacy/legal constraints: any PII/consent issues with tracking or personalization.

Analysis plan (brief)

• Pre-specify primary metric, MDE, test duration, and stopping rule (avoid peeking).
• Use ITT analysis; also report treatment-on-treated (exposure) as secondary.
• Run balance checks on covariates; if imbalanced, consider covariate-adjusted models (logistic regression for binary outcome).
• Report confidence intervals for uplift and p-values, and practical significance.

This gives a testable hypothesis, the metric and sample-size method, and the operational clarifications needed to run a robust experiment.

To deliver a usable “high LTV vs low LTV” segmentation I’d first clarify definitions, windows, assumptions, and downstream actions. Key questions grouped by topic, with why they matter and example options:

Label construction

• How does the business define LTV? (gross revenue, net revenue, contribution margin, or discounted cash flows?) — affects target calculation.
• Is LTV absolute (dollars) or relative (top X% of customers)? — determines thresholding strategy.
• Do we include refunds, promotions, or returns in LTV? — impacts accuracy.

Lookback window

• Over what historical period should we compute features? (30/90/365 days) — balances recency vs sample size.
• Any seasonality or cohort effects to respect? — may require aligned windows (e.g., 12-month rolling).

Evaluation period / prediction horizon

• For labeling, what forward horizon should LTV predict? (next 3, 6, 12 months) — sets target window for supervised learning.
• How will we handle censored customers with short tenure? — decide on exclusion or survival-modeling.

Monetization & business assumptions

• Do we discount future revenue? At what rate? — impacts present-value LTV.
• Should we model ARPU, purchase frequency, and churn separately or directly predict revenue? — influences model choice.
• Which revenue sources to include (subscriptions, ads, in-app purchases)?

Data & labeling practicalities

• Which data sources are authoritative for transactions, refunds, and cost? Any latency issues?
• Minimum activity/tenure to include a user in training? (e.g., >=30 days)
• How to handle bots, test accounts, or corporate customers?

Evaluation metrics & success criteria

• Business KPIs to optimize: revenue uplift, retention, margin, cost per acquisition?
• Preferred model metrics: AUC, precision@k, calibration, or expected value lift?
• What is acceptable false-positive/false-negative trade-off? (e.g., cost of mis-targeting high-value treatment)

Converting model output into product actions

• How will labels trigger actions? (personalized offers, CRM tiers, targeting budgets)
• Operational constraints: realtime vs batch scoring, latency, and data availability.
• Experiment design: propose A/B test or champion-challenger with metrics (incremental revenue, retention, CAC).
• Action mapping examples: top 5% -> VIP offers and dedicated support; next 20% -> targeted promotions; bottom -> cost-effective retention or win-back flows.

Deliverables & timeline

• Preferred cadence for model refresh (weekly/monthly) and retraining triggers.
• Acceptable rollout plan: shadow mode → small pilot → scaled deployment.
• Compliance/ethical constraints (GDPR, promotional fairness).

Example concrete label setup (if PM wants one quickly)

• Label = total net revenue (sales − refunds) over next 6 months, discounted 0% for short horizon.
• High LTV = top 20% by that revenue in a cohort with >=90 days tenure.
• Train on customers with at least 12 months of historical data, evaluate uplift via a 6-week A/B test measuring incremental revenue per user.

These questions align modeling choices with business needs and will let us produce a labeled dataset, pick features and models appropriately, define evaluation, and create actionable, testable product treatments.

Situation: A stakeholder requests "real-time fraud detection." I would ask a focused set of clarifying questions across latency SLAs, signal availability, feature freshness, labeling lag, and validation so we build a realistic, measurable system.

Latency SLA

• What is "real-time" in business terms? (e.g., response within 100 ms, 1s, 1 min)
• Is online blocking required (prevent transaction) or just alerting/investigation?
• What throughput and peak QPS must we support?
  Reason: Blocking requires much tighter latency, different architecture (inference at edge vs async scoring).

Signal availability & feature sources

• Which signals exist in-stream vs batch (payment fields, device fingerprint, user history, 3rd-party risk lookups)?
• Are any signals intermittent or rate-limited (third-party fraud scores)?
• Can we accept degraded feature sets or fallback rules if some signals are missing?
  Reason: Defines feature retrieval architecture and fallback logic.

Feature freshness

• For each feature, what freshness is acceptable? (e.g., last 5s for session features, daily for long-term risk)
• Which features must be updated continuously (session velocity) vs can be windowed?
  Reason: Drives streaming feature store design and windowing semantics.

Labeling lag

• How quickly do we receive ground truth (chargeback, dispute, manual review)? Typical delays?
• Can we create surrogate labels (reversed transactions, manual flags) to bootstrap?
• What label quality / noise level is acceptable for online learning?
  Reason: Affects model training cadence, delayed-feedback correction, and bias.

Validation & performance in streaming

• Which metrics matter in production? (precision at k, false positive rate, decision latency, time-to-detect)
• How to balance business cost: cost of false positives vs fraud loss?
• Do we need online A/B/Canary, shadow mode, or progressive rollout?
  Validation approach:
• Shadow testing: run model in parallel to measure predicted vs eventual labels without impacting users.
• Time-windowed backtesting with realistic feature staleness and arrival order (simulate streaming).
• Use delayed-label evaluation (labeling lag aware): compute metrics with survival analysis or inverse propensity weighting to correct for censoring.
• Monitor concept drift with drifting feature distributions, calibration, and real-time KPI alerts; retrain triggers when performance degrades.
• Use reservoir sampling or streaming metrics (approx quantiles) to keep evaluation light at scale.

Operational considerations

• SLOs for model inference, feature retrieval, and overall decision path; logging for every decision for audit and retraining.
• Fall-back rule engine when features unavailable or model offline.
• Retraining cadence: continuous online learning vs minibatch nightly depending on label lag and stability.

Example acceptance criteria (concrete)

• Block transactions within 200 ms for 99th percentile latency
• Precision@top100 alerts >= 85% in shadow for 7-day rolling window after label reconciliation
• Channel: run shadow for 30 days, monitor FP cost < $X per day, then canary 1% traffic before global rollout

This set of questions and validation plan ties SLA, data realities, and business costs into a measurable implementation path.

Clarifying questions

• What exact business metric do you mean by “better”? (e.g., conversion rate, revenue per user, average order value, retention)
• What is the current baseline rate/value for that metric?
• What minimum improvement would be meaningful to the business (in absolute or percent terms)?
• How long can we run the test (days, traffic %, business cycles)?
• What risk of a false positive is acceptable? (common choices: 5% or 1%)
• Do you need a quick early signal or a final definitive result?
• Are there subgroups (mobile vs desktop, new vs returning) we must consider?

Converting into statistical success definitions

• Metric: primary metric = Checkout conversion rate.
• Baseline: baseline conversion = 8.0%.
• Minimum Detectable Effect (MDE): business says “better” means at least +10% relative → MDE = 0.8 percentage points (8.8%).
• Significance & power: choose α = 0.05 (two-sided) and power = 0.8.
• Success definition: “Variant is successful if we reject the null of equal conversion rates at α=0.05 and observed uplift ≥ 0.8 ppt.”

Sample size and stopping rules

• Compute fixed-horizon sample size for proportions (standard formula) → N_per_arm ≈ 2 * (Z_{1-α/2}√(2p(1-p)) + Z_{power}√(p(1-p)+ (p+MDE)(1-(p+MDE))/2))^2 / MDE^2. (I’d calculate exact N with code; for p=0.08, MDE=0.008, α=0.05, power=0.8 -> ~50k users per arm.)
• Stopping rule options:
  1. Fixed-horizon (recommended): Run until both arms reach N_per_arm, then analyze once. No peeking.
  2. Sequential monitoring (if business needs early stop): use an alpha-spending approach (e.g., O’Brien–Fleming) or adopt a pre-registered group-sequential plan and adjust significance thresholds per look to control overall α.
  3. Bayesian alternative: predefine a decision threshold like P(Δ > 0.8 ppt | data) > 0.975 to call success, and P(Δ < 0 | data) > 0.95 to call failure; allows flexible peeking.

Practical example (explicit stopping rules)

• Fixed-horizon: “Collect 50k users per arm (~7 days). After reaching sample, run two-sided z-test at α=0.05. If p-value <0.05 and uplift ≥0.8 ppt → ship; else no lift or inconclusive.”
• Sequential with O’Brien–Fleming: “Planned looks at 25%, 50%, 100% of data with adjusted critical p-values (≈0.0005, 0.014, 0.045). Stop early only if p < threshold and uplift ≥ MDE.”

Why this works

• Converts vague ‘better’ into measurable MDE tied to business value.
• Predefining α, power, sample size and stopping rules prevents biased decisions from peeking.
• Offer calculation and runbook; I’d implement the exact sample-size calc and monitoring code and share a one-page experiment spec for stakeholder sign-off.

Situation: Given a brief "clean the data and build the best model", I would first ask focused clarifying questions, then define cleaning scope, reproducibility standards, and speed-vs-quality trade-offs.

Clarifying questions to define success:

• What is the business objective / target metric (e.g., AUC, RMSE, revenue lift, recall at p95)?
• Success criteria and threshold, and how model will be used in production (batch vs real-time)?
• Allowed features and data sources, privacy/regulatory constraints, label availability and quality?
• Deadline and resource constraints (compute, team members)?
• Evaluation data: is there a holdout/test set and how should temporal leakage be handled?

Degree of cleaning expected:

• Minimal: remove obvious corruptions, normalize types, handle missingness with simple imputations for quick prototyping.
• Standard: detect and correct duplicates, outliers, inconsistent categories, feature-specific imputations, create validation to ensure cleaning didn't leak label information.
• Deep: resolve upstream data pipeline issues, enrich data, robust anomaly detection — for production-ready models.

Reproducibility standards:

• Notebook + modular scripts (data ingestion -> cleaning -> feature engineering -> modeling), version-controlled (Git).
• Use deterministic seeds, containerized environment (Docker), pinned deps (requirements.txt/conda.yml).
• Store intermediate artifacts and hashes, include tests for data schema and unit tests for key transformations.
• CI to run training/eval and save model artifacts with metadata (hyperparams, data snapshot).

Trade-offs (speed vs model quality):

• Rapid prototype: lean cleaning, simple models (logistic, tree) to validate signal quickly.
• Iterative improvement: invest in targeted cleaning and feature engineering where error analysis shows gains.
• Production: prioritize robustness, monitoring, and reproducibility even if training cost increases.
• Decide by ROI: if small data fixes yield large metric gains, invest time; if marginal, prefer automation and monitoring.

This approach balances short-term validation with long-term production quality and clear criteria for success.