Covers fundamental probability theory and statistical inference from first principles to practical applications. Core probability concepts include sample spaces and events, independence, conditional probability, Bayes theorem, expected value, variance, and standard deviation. Reviews common probability distributions such as normal, binomial, Poisson, uniform, and exponential, their parameters, typical use cases, computation of probabilities, and approximation methods. Explains sampling distributions and the Central Limit Theorem and their implications for estimation and confidence intervals. Presents descriptive statistics and data summary measures including mean, median, variance, and standard deviation. Details the hypothesis testing workflow including null and alternative hypotheses, p values, statistical significance, type one and type two errors, power, effect size, and interpretation of results. Reviews commonly used tests and methods and guidance for selection and assumptions checking, including z tests, t tests, chi square tests, analysis of variance, and basic nonparametric alternatives. Emphasizes practical issues such as correlation versus causation, impact of sample size and data quality, assumptions validation, reasoning about rare events and tail risks, and communicating uncertainty. At more advanced levels expect experimental design and interpretation at scale including A B tests, sample size and power calculations, multiple testing and false discovery rate adjustment, and design choices for robust inference in real world systems.
HardTechnical
60 practiced
You are the lead data scientist asked to define the primary metric and guardrail metrics for launching a new one-click checkout flow. Prepare a short measurement plan: define primary metric, at least three guardrail metrics, sample size considerations, risk thresholds for stopping the experiment, and rollout criteria for launch.
Sample Answer
Primary metric- Conversion rate of completed purchases per user-session (one-click checkout conversion): proportion of sessions that start checkout and finish purchase within 30 minutes. Rationale: directly measures business goal (friction reduction → revenue).Guardrail metrics (at least 3)1. Revenue per user-session (RPU): ensures conversion gains aren't driven by large discounts or basket manipulation.2. Average order value (AOV): detect if checkout biases toward low-value purchases.3. Payment failure rate: percent of attempted payments that fail — catches regressions in payment integrations.4. Customer support contacts about checkout (CS contacts per 1k sessions): user confusion or errors.5. Fraud rate / chargebacks: protects against increased fraud.Sample size & testing setup- Target minimum detectable effect (MDE): 3% relative lift on conversion (business decides). Baseline conversion = 12% → absolute MDE = 0.36 pp.- Power = 80%, alpha = 0.05 (two-sided). Using standard proportions formula, required N per arm ≈ 40k sessions. Adjust for multiple segments and expected attrition; plan for 50–60k sessions per arm.- Run experiment for at least one business cycle (14 days) to capture weekday/weekend patterns.Risk thresholds (stop criteria)- Stop immediately if payment failure rate increases by >0.5 percentage points AND absolute increase is statistically significant (p<0.01).- Stop if fraud/chargeback rate increases >25% relative and significant.- Pause and investigate if CS contacts per 1k sessions increase >50% or AOV drops by >5% with significance.- Interim checks: use conservative alpha-spending (e.g., O’Brien-Fleming) to avoid false positives.Rollout criteria- Primary metric: statistically significant lift (p<0.05) with practical significance ≥ MDE.- All guardrails: no statistically significant adverse impact beyond predefined thresholds.- Qualitative sign-off: no major UX or merchant-reported issues from beta users; support and fraud teams cleared.- Phased rollout: 10% → 25% → 50% → 100% over 1–2 weeks, monitoring guardrails at each step; rollback if any guardrail breaches.Notes- Pre-register metrics and analysis plan; include segment analyses (new vs returning users, device, region) and A/A to validate instrumentation.
MediumSystem Design
70 practiced
Design a medium-scale A/B testing platform for a consumer web product that serves ~1M daily active users. Describe how you would implement random assignment, experiment configuration, metric collection, handling of multiple variants, and how you would detect and handle sample ratio mismatch or instrumentation issues. Focus on statistical considerations more than system-level code.
Sample Answer
Requirements & goals:- 1M DAU, medium-scale experiments, low-latency assignment, reproducible randomization, robust metric collection, detect sample-ratio-mismatch (SRM) and instrumentation errors, support multi-variant.Random assignment:- Choose a unit (user-id or session). Use deterministic hashing: bucket = hash(user_id + experiment_id) mod 100000. Map buckets to variants by precomputed allocation. This is stateless, reproducible, and supports rollbacks and staging.- Support stratified/randomized by segments (country, platform) by applying hashing within strata to ensure balance.Experiment configuration:- Store config in a central service: experiment_id, start/end, variants & weights, targeting rules, unit, metrics list, exposure criteria. Version configs; clients cache and periodically refresh.- Include exposure criteria (viewed experiment page) vs assignment (assigned but not exposed).Metric collection:- Collect raw events with schema: user_id, experiment_id, variant, event_type, timestamp, properties. Record both assignment and exposure events; treat exposure as authoritative denominator.- Use deterministic join keys and idempotent event IDs to avoid double-counting. Aggregate via daily batch jobs + real-time streaming for dashboards.- Precompute derived metrics (rates, means) and compute variance using robust formulas (bootstrap or delta method).Multiple variants & stats:- Use contrasts vs control with adjustment for multiple comparisons (Benjamini-Hochberg for FDR, or Bonferroni for strict control). For continuous metrics, use t-test/ANCOVA adjusting for covariates (pre-experiment baseline) to increase power. For proportions, use chi-square or z-test with pooled variance.- Predefine hypothesis, minimum detectable effect (MDE), power, and required sample. Consider sequential testing with alpha spending (e.g., O'Brien-Fleming) or use Bayesian posterior inference to allow continuous monitoring.Detecting SRM & instrumentation issues:- SRM test: daily compare observed assignment proportions vs expected using a chi-square or binomial test. Compute p-value and alert when p < threshold (e.g., 0.001) and effect size > practical threshold.- Data integrity checks: compare assignment counts (assignments vs exposures), user-level churn, repeat assignment rates, and duplicate events.- Automatic sanity checks on metrics (unexpected jumps, missing cohorts, timezone mismatches) using control charts and upstream/downstream consistency checks.- When SRM/instrumentation flagged: pause inferential reporting for affected experiments, run a lineage/debug workflow: check rollout config, caching, SDK versions, hashing function, targeting rules, and sampling at client side. Reprocess events with corrected mappings if possible.Practical notes:- Log both assignment and exposure; use exposure as denominator to avoid inflating power with unexposed assigned users.- Maintain experiment registry and guardrails (max simultaneous experiments per user, overlapping exclusion rules).- Document assumptions and enforce pre-registration of primary metric and analysis plan to avoid p-hacking.
HardSystem Design
62 practiced
You need to allow stakeholders to continuously monitor an experiment but avoid inflated false positives from continuous peeking. Describe a plan using group sequential methods or alpha spending to control Type I error under continuous monitoring. Explain trade-offs between sample size, frequency of looks, and complexity of implementation.
Sample Answer
Requirements & constraints:- Stakeholders must view interim results frequently (near real-time) without inflating Type I error (overall α, e.g., 0.05).- Maintain statistical validity, reasonable sample size, and operational simplicity.- Support binary or continuous metrics, fixed-horizon trial.High-level plan:1. Choose a group-sequential framework with an alpha-spending function (Lan–DeMets) so looks can be irregular/frequent while controlling cumulative Type I error.2. Pre-specify maximum sample size Nmax and maximum number of looks K (or allow looks by time/accumulated information fraction).3. Select an alpha-spending function: O’Brien–Fleming-like (conservative early, more liberal later) or Pocock-like (equal spending). Implement Lan–DeMets to map information fraction t to cumulative α(t).4. At each look compute test statistic Z (e.g., difference in means / pooled SE). Compare Z to boundary z_alpha(t) derived from inverse normal mapping of remaining alpha: reject H0 if Z >= z_boundary(t).5. Adjust final sample size/power: design initial Nmax using planned power and effect size under group-sequential correction (inflation factor relative to fixed design).Core components and data flow:- Data pipeline: event ingestion → aggregation into analysis-ready snapshots.- Monitoring service: on each snapshot compute information fraction t = observed information / planned information; call alpha-spending function to get boundary; compute Z and decision.- Audit/logging: persist decisions, boundaries, seeds, and exact info fractions for reproducibility.- UI: show observed effect, p-value, current boundary, and remaining α.Trade-offs:- Frequency of looks vs. sample size: more frequent looks with stringent early boundaries (O’Brien–Fleming) usually need little sample-size inflation; Pocock gives easier early stopping but requires larger Nmax to retain power.- Complexity: Lan–DeMets + irregular looks supports continuous monitoring and is flexible but requires careful engineering to compute information fractions and boundaries; simpler fixed K group-sequential (pre-specified times) is easier to validate.- Stakeholder clarity vs. rigor: frequent "peeks" require communicating that near-term significance thresholds are stricter; avoid ad-hoc stopping rules (optional stopping) unless included in the plan.Implementation notes & best practices:- Pre-register plan: α, spending function, metric definitions, handling of missing/stratified data, and maximum sample size.- Use information-based fraction: for Bernoulli, information ≈ p(1−p)*n; update using pooled estimates to avoid bias.- Simulate operating characteristics (Type I error, power, expected sample size) across realistic scenarios before launch.- Automate reproducible analyses with locked code, and provide Explainable UI messaging: "Current Z = X, boundary = Y; overall α preserved via Lan–DeMets OBF."This approach balances stakeholder need for monitoring with rigorous Type I control; choose O’Brien–Fleming/Lan–DeMets for many frequent looks, or Pocock if earlier stopping is prioritized but accept larger N.
EasyTechnical
56 practiced
You receive a binary diagnostic signal for fraud on transactions. Explain conditional probability and Bayes theorem in this context: if the fraud detector has 98% true positive rate and 1% false positive rate, and baseline fraud prevalence is 0.1%, compute the posterior probability that a flagged transaction is actually fraudulent. Show your reasoning and discuss implications for decision thresholds.
Sample Answer
Situation: We have a binary fraud detector with known performance and a low base rate of fraud. We want P(Fraud | Flagged).Bayes’ rule (conditional probability):P(Fraud | Flag) = P(Flag | Fraud) * P(Fraud) / P(Flag)where P(Flag) = P(Flag | Fraud)*P(Fraud) + P(Flag | NotFraud)*P(NotFraud).Plugging numbers:- P(Fraud) = 0.001 (0.1% prevalence)- P(Flag | Fraud) = TPR = 0.98- P(Flag | NotFraud) = FPR = 0.01Numerator = 0.98 * 0.001 = 0.00098Denominator = 0.00098 + 0.01 * 0.999 = 0.00098 + 0.00999 = 0.01097P(Fraud | Flag) ≈ 0.00098 / 0.01097 ≈ 0.0893 → about 8.9%Interpretation and implications:- Even with 98% sensitivity, because fraud is rare, the positive predictive value is low (~9%). Most flagged transactions are false positives.- Decision thresholding: optimize threshold by trading off FPR vs TPR using ROC/PR curves, but in low-prevalence settings precision (PR curve) matters more.- Operational actions: add secondary checks (risk scores, rules, human review) for flagged cases; consider cost-sensitive decisions (expected loss of letting fraud through vs investigation cost) and calibrate threshold to minimize expected cost.- Monitor prevalence and model calibration over time—posterior changes if base rate shifts.
EasyTechnical
51 practiced
Define expected value, variance, and standard deviation. Given a discrete distribution of user lifetime values with values {0:0.5, 10:0.3, 100:0.2} (value:probability), compute the expected value and variance and interpret what they mean for business decision-making.
Sample Answer
Expected value, variance, and standard deviation (brief):- Expected value (E[X]) is the probability-weighted average of a random variable — the long-run average outcome.- Variance (Var[X]) measures the average squared deviation from the mean; it quantifies spread/risk.- Standard deviation (SD) is the square root of variance, giving spread in the original units.Compute for distribution {0:0.5, 10:0.3, 100:0.2}:- E[X] = 0·0.5 + 10·0.3 + 100·0.2 = 0 + 3 + 20 = 23- E[X^2] = 0^2·0.5 + 10^2·0.3 + 100^2·0.2 = 0 + 30 + 2000 = 2030- Var[X] = E[X^2] − (E[X])^2 = 2030 − 23^2 = 2030 − 529 = 1501- SD = sqrt(1501) ≈ 38.73Business interpretation:- Average LTV per user is $23, which is the useful number for budgeting, forecasting, and comparing to CAC.- Large variance and SD (~$39) indicate high dispersion and skew: many users (50%) give $0, a minority yield high value ($100). Median is 0, so half users give no lifetime value.- Implications: relying only on the mean can be misleading — consider segmentation, target high-LTV cohorts, use medians or percentile-based metrics for risk assessment, and incorporate variance into ROI and sensitivity analyses (e.g., required sample sizes, worst-case scenarios).
Unlock Full Question Bank
Get access to hundreds of Probability and Statistical Inference interview questions and detailed answers.
