Applying Data Science Techniques to Business Problems Questions
Recognizing when A/B testing is appropriate vs observational analysis. Suggesting SQL queries or analysis approaches that would answer the business question. Understanding when you'd need advanced modeling vs simpler analysis. Connecting technical approaches to business decisions (e.g., 'This cohort analysis would tell us whether the decline is from existing users or new users').
MediumTechnical
77 practiced
A key metric is rare (conversion 0.1%). How would you design an experiment to detect a meaningful uplift? Discuss sample size implications, alternative proxy metrics, event aggregation strategies, and whether to use sequential testing tailored for rare events.
Sample Answer
Situation: We need to test an intervention where the primary conversion is extremely rare (0.1% baseline). The standard A/B design would need huge samples, so we must reason about power, alternatives, and practical experiment design.Sample-size implications (quantitative example)- For a two-arm test (α=0.05 two-sided, power=80%), detecting a 20% relative uplift (0.001 → 0.0012, absolute Δ=0.0002) requires a lot of users.- Using normal approximation for proportions: N_per_arm ≈ ((Zα/2+Zβ)^2 * 2 * p̄(1−p̄)) / Δ^2.- Plugging p̄≈0.0011, Zα/2=1.96, Zβ=0.84 gives N_per_arm ≈ 430k (≈860k total). Smaller effects or higher confidence raise N dramatically.- Conclusion: direct testing on the raw rare event is often infeasible.Alternative/proxy metrics- Use higher-frequency leading indicators correlated with the rare conversion (e.g., “added to cart”, sign-up started, model-predicted propensity score). Validate correlation and ensure proxy is predictive of downstream business value.- Construct a composite metric (weighted combination) that increases signal while preserving interpretation.- Use time-to-event or engagement measures (continuous) which have higher power.Event aggregation and modeling- Aggregate events over longer windows (e.g., weekly rather than per-session) or at user-level counts to increase observed events per unit. Beware leakage and time-varying confounders.- Target segments with higher baseline conversion (power uplift): users with prior intent, premium cohort—run stratified or targeted experiments and pre-specify heterogeneity analyses.- Covariate adjustment (logistic regression, Poisson/negative-binomial) improves precision: include strong predictors (past behavior, demographics) to reduce residual variance and thus required N.- Hierarchical/Bayesian models can borrow strength across segments/time and produce probabilistic uplift estimates with smaller samples.Sequential testing for rare events- Sequential (or continuous) monitoring is attractive but dangerous with rare events because early looks will be noisy and increase false positives if unadjusted.- Use group-sequential designs or alpha-spending functions (O’Brien–Fleming, Pocock) to control Type I error, or prefer fully Bayesian decision rules (posterior probability of uplift > threshold).- For very rare events, pre-specify minimum information (e.g., minimum number of events per arm) before stopping to avoid misleading early stops.Practical recipe1. Validate proxies and their lift→conversion correlation. If correlation strong, run experiments on proxy metrics and back-validate.2. Precompute sample-size and minimum-events thresholds; consider targeting higher-incidence segments to reduce N.3. Use covariate-adjusted analysis (GLM) and Bayesian/hierarchical models to improve power.4. If sequential testing, use proper alpha-spending or Bayesian stopping rules and require minimum events before any decision.5. Report absolute and relative effects, uncertainty, and business impact (expected incremental conversions) so stakeholders can weigh trade-offs.This balances statistical rigor with business practicality for rare-event experimentation.
HardSystem Design
70 practiced
Design an end-to-end daily experiment metrics analytics pipeline that covers event ingestion, sessionization, deduplication, metric computation, reporting, and rollback support. Describe core components, storage choices, latency SLAs, and how you would ensure reproducibility, lineage, and versioned metric definitions.
Sample Answer
Requirements & constraints:- Daily experiments: compute metrics once per day for A/B tests; SLAs: fresh results within 2 hours after daily cut (nearline), archival recompute within 24 hours.- Functional: event ingestion, sessionization, deduplication, metric computation, reporting, rollback to prior metric definitions, reproducibility and lineage, versioned metric definitions.High-level architecture:1. Event ingestion: client SDK → Kafka (partitioned by user_id) for exactly-once-friendly stream; schema enforced via Confluent Schema Registry (Avro/Protobuf).2. Stream processing: Flink (or Spark Structured Streaming) jobs that perform real-time deduplication (using event IDs with time-windowed state), enrich events (user properties), and write cleaned events to a raw event lake (object storage like S3/GCS) and to a feature/event warehouse (Delta Lake / Iceberg) for fast queries.3. Sessionization: Stateful stream job that identifies sessions (idle timeout config), emits session records to warehouse; also store user-session mapping for attribution.4. Batch compute & metric engine: A daily Airflow job triggers a reproducible metric computation registry. Jobs read from versioned Delta/Iceberg tables plus a snapshot of transformation code (containerized) and metric SQL/DSL stored in Git + metadata DB. Compute writes outputs to a versioned metrics store (timeseries DB like ClickHouse or BigQuery partitioned by date).5. Reporting & dashboarding: BI reads from metrics store; experiment dashboard pulls metric metadata (definition version) and links to lineage.6. Rollback & re-compute: Metric definitions are immutable; new versions created for edits. Rollback = point-in-time recompute using stored code container image + exact input table snapshot (Delta/Iceberg time travel or S3 object versions), then write a new metric version flagged as “rolled-back” or promote old version.Storage choices & rationale:- Kafka: durable, low-latency ingestion.- S3/GCS + Delta Lake or Iceberg: cheap durable raw layer, support time travel, ACID for reproducible recompute.- Warehouse (BigQuery / Snowflake / ClickHouse): OLAP queries, dashboards.- Metadata DB (Postgres): stores metric definitions, code hashes, container images, experiment configs, lineage graph.Latency SLAs:- Ingestion to cleaned event store: <5 minutes (streaming).- Nearline experimental metrics (approx): <2 hours after daily cut.- Full archival recompute: <24 hours.Reproducibility, lineage, and versioning:- All processing code packaged as versioned containers; builds hashed and stored in artifact registry.- Metric definitions stored in Git + metadata DB with semantic versioning; each metric version includes SQL/DSL, owner, tests, and change diff.- Input snapshots: use Delta/Iceberg time travel or store manifest of S3 object versions; metadata DB records exact table snapshot IDs used for each metric run.- Lineage: record DAG linking metric version → code image → input snapshot(s) → output artifact. Expose lineage UI and store in metadata DB (or use OpenLineage/Marquez).- Tests & validation: every metric version runs unit tests and small-sample backtests; guardrails (thresholds) require human approval before production promotion.Deduplication & correctness details:- Use event_id + max(timestamp) semantics; streaming job maintains short-lived dedupe state and emits canonical events; long-term dedupe enforced at batch compute by dedup SQL using windowed aggregates and watermark logic.- Idempotent sinks and write-once metric artifacts.Operational & ML-specific considerations:- Experiment metadata (variants, allocation, exposure windows) kept in metadata DB and joined during metric compute.- Drift & significance: compute uplift, confidence intervals, and optional bootstrap; store p-values and power metrics.- Monitoring: data quality checks (rate, schema, cardinality), alerting, and automated rollback triggers if anomalies detected.Trade-offs:- Nearline SLA vs cost: faster SLAs need more streaming state and compute; choose two-hour daily SLA to balance cost and business value.- Delta/Iceberg chosen for reproducibility; if org uses BigQuery, use snapshot exports and partitioned tables.This design ensures end-to-end reproducible daily experiment metrics, clear lineage, safe versioning, and straightforward rollback/recompute.
HardTechnical
72 practiced
A city implemented a policy affecting ride-hailing usage and randomization was not possible. Explain how you'd construct a synthetic control from other cities to estimate the policy effect: discuss donor-pool selection, predictor choice, optimization for weights, and inference using placebo tests.
Sample Answer
Situation: We need an estimate of a city-level policy effect on ride-hailing where randomization wasn't possible. Synthetic control (SC) constructs a weighted combination of other cities (donor pool) that matches the treated city prior to the policy, then attributes post-policy divergence to the policy.Donor-pool selection:- Exclude cities with concurrent interventions or clear spillovers (geographic neighbors if commuting effects likely).- Keep cities with similar pre-policy trends in key outcomes and covariates (population, public transit supply, GDP per capita, smartphone penetration).- Prefer a reasonably large pool (20–50) to allow flexible weighting but prune extreme outliers.Predictor choice:- Use predictors that predict the outcome and are unaffected by the policy: lagged levels and trends of ride-hailing usage, seasonality dummies, mobility metrics, socio-economic covariates, pre-policy shocks indicators.- Include multiple pre-treatment periods of the outcome (e.g., monthly rides for 24 months) to force close trajectory matching.- Normalize predictors (z-score) so scale doesn’t distort weights.Optimization for weights:- Solve for non-negative weights w_j that sum to 1 minimizing the pre-treatment mean squared prediction error (MSPE): minimize_w (X1 - X0 w)' V (X1 - X0 w) subject to w_j >= 0, sum w_j = 1 where X1 = vector of treated predictors, X0 = matrix of donor predictors, and V is a diagonal matrix of predictor importance (choose via cross-validation to minimize MSPE on holdout pre-period or via LASSO-like regularization).- If many donors, consider adding an L2 penalty to avoid overconcentration: minimize MSPE + lambda ||w||^2.- Check fit: pre-treatment MSPE should be small; visually compare trajectories.Inference using placebo tests:- Conduct placebo (permutation) tests by reassigning the intervention to each donor city and computing their post/pre MSPE ratios or treatment effects. Build distribution of placebo effects.- Use ratio of post-treatment MSPE to pre-treatment MSPE as a test statistic to account for fit quality.- P-value = proportion of placebos with effect at least as large as treated city's effect.- Report effect size, uncertainty bands from placebo distribution (e.g., percentiles) and robustness checks: leave-one-out donors, alternative predictor sets, varying pre-period length.- Address staggered adoption by using extensions (Generalized Synthetic Control or matrix completion) and check for time-varying confounders.This approach yields a transparent, data-driven counterfactual with principled inference via permutation-based placebo tests.
EasyTechnical
76 practiced
Explain when you would use an A/B test versus an observational analysis to answer a product question. Give two concrete business scenarios (one suited to randomized experiment, one to observational study), list the key assumptions for each approach, and describe the main limitations an ML engineer should communicate to stakeholders.
Sample Answer
Use an A/B test when you can randomize exposure to treatments and need a causal estimate of the effect. Use observational analysis when randomization is infeasible, unethical, too slow, or too costly; observational work can still estimate associations and, with careful methods, causal effects under strong assumptions.Scenario 1 — Randomized experiment (A/B):- Business question: Does a new recommendation model increase click-through rate (CTR) and downstream purchases?- Design: Randomly assign 50% of traffic to current model (A) and 50% to new model (B), run for an adequate sample and time window, measure CTR and purchases.- Key assumptions: - Randomization is implemented correctly (no leakage). - Stable Unit Treatment Value Assumption (SUTVA): one user’s assignment doesn’t affect another’s outcome. - No differential attrition or measurement bias across groups.- Main limitations to communicate: - External validity: experiment population or time window may not generalize to all users or seasons. - Short-term vs long-term effects: immediate CTR lift could harm long-term retention. - Infrastructure constraints: rollout complexity, instrumentation coverage. - Metric selection: uplift on proxy metric (CTR) may not equal business value.Scenario 2 — Observational analysis:- Business question: Did onboarding changes implemented last quarter increase user retention?- Approach: Use historical data, adjust for confounders with methods like propensity score matching, regression adjustment, or difference-in-differences if there's a clear pre/post and control group.- Key assumptions: - No unmeasured confounding (conditional ignorability) after adjustment. - Correct model specification for adjustment methods. - For DiD: parallel trends between treated and control groups pre-intervention.- Main limitations to communicate: - Causal claims are weaker: residual confounding or selection bias may remain. - Sensitivity to model choices and covariates; results should include robustness checks. - Measurement timing and data quality issues can bias estimates. - Actionability: observational effect sizes may be biased and should be validated (ideally with a targeted experiment).As an ML engineer, recommend experiments when feasible for clear causal answers; when not, run rigorous observational analyses with sensitivity analyses, transparently report assumptions, confidence intervals, and practical risks so stakeholders understand how much trust to place in the results.
MediumBehavioral
74 practiced
Behavioral: Tell me about a time you convinced product or marketing stakeholders to run an experiment instead of relying on observational analysis. Use the STAR framework: describe the situation, what you proposed, objections you encountered, how you addressed them, and the outcome. If you lack a direct example, outline how you would handle such a conversation.
Sample Answer
Situation: At my last company I was the ML engineer responsible for a personalization model for email subject lines. The product and marketing teams were debating whether open-rate differences they’d seen in historical logs justified rolling out a new model globally.Task: I needed to convince stakeholders that an experiment (A/B test) was necessary to measure causal impact on opens, downstream clicks, and long-term engagement rather than trusting observational lift from past data.Action:- I presented a short, focused slide that explained confounders in the observational data (time-of-day, audience segments, campaign types) and showed a simple simulation demonstrating how selection bias could produce spurious gains.- Proposed an A/B test design: randomized assignment at recipient level, primary metric = 7-day click-through rate, safety guardrails (monitor for spam complaints, unsubscribes), minimum detectable effect and sample size calculations, and a 2-week rollout with interim checkpoints.- Anticipated objections: marketing worried about losing short-term revenue from a “worse” variant; product worried about implementation cost. I addressed these by including an uplift threshold for early stop rules, a rollback plan, and estimating engineering effort (one-week integration using our feature flag system).- Aligned on success criteria and included an analysis plan (pre-registered metrics and segmentation) to avoid p-hacking.Result: The experiment ran as planned. The new model produced a 3.2% improvement in 7-day CTR (p < 0.01) and no negative impact on unsubscribes. Because we had pre-registered metrics and safety checks, stakeholders felt confident and we deployed the model gradually. The process also increased trust: product adopted A/B testing as the default for future personalization changes.This taught me that translating technical risks into business terms (what could go wrong, how we'll detect it, and how much effort it needs) is the most effective way to get stakeholder buy-in.
Unlock Full Question Bank
Get access to hundreds of Applying Data Science Techniques to Business Problems interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.