Quantitative Research and Analysis Questions

Covers end to end quantitative research methods used to measure and validate product and user behavior hypotheses. Topics include experimental and quasi experimental design, split testing and controlled experiments, metric definition and success criteria, sample size calculation and statistical power, selection of appropriate statistical tests and interpretation of statistical significance and effect sizes, confidence intervals, correlation versus causation, common statistical pitfalls and biases, analytics instrumentation and metric tracking, survey design and quantitative measurement, and data analysis workflows and tools used to analyze large scale user data. Candidates should be able to design experiments, justify metric choices, calculate sample size and duration, analyze results rigorously, and make data driven recommendations.

EasyTechnical

73 practiced

What are common pitfalls in survey design for quantitative product feedback? Provide examples of question wording that introduces bias and rewrite them into neutral, high-quality survey questions. Also explain how you would combine survey responses with behavioral data to strengthen conclusions.

Sample Answer

Common pitfalls in survey design for quantitative product feedback:- Leading or loaded questions that push respondents toward an answer- Double-barreled questions asking two things at once- Ambiguous or fuzzy wording (e.g., “often”, “regularly”)- Poor response scales (inconsistent anchors, no neutral option) and scale bias- Recall bias and long/complex surveys causing satisficing- Sampling bias and low response rates that make results non-representative

Examples of biased wording and neutral rewrites:- Biased: “How much do you like our excellent new checkout flow?” Neutral: “How satisfied are you with the new checkout flow?” (1 = Very dissatisfied, 5 = Very satisfied)- Double-barreled: “How easy and fast was placing an order?” Split & neutral: “How easy was it to place an order?” and “How fast did the checkout process feel?” (separate scales)- Ambiguous: “Do you use the app frequently?” Neutral: “How many times did you open the app in the past 7 days?” (provide numeric ranges)- Leading scale: “Would you strongly agree that our pricing is fair?” Neutral: Use balanced Likert: Strongly disagree → Strongly agree, with a clear statement: “The product’s price is fair.”

Combining survey responses with behavioral data:- Use surveys for subjective sentiment and attach user IDs (with consent) to link to actual events (e.g., number of purchases, feature usage, time-to-complete-task).- Validate self-report with observed behavior: if users say “I use feature X weekly” but event logs show <1 use/month, prioritize behavioral signal.- Segment analyses: compute Net Promoter Score or satisfaction by cohort defined by behavior (power users vs occasional) to uncover heterogeneity.- Use causal approaches where possible: A/B test changes and collect targeted in-app surveys to measure perceived impact while observing conversion metrics.- Best practices: ensure privacy, account for non-response, weight survey results to match population, and report both survey and behavioral metrics together in dashboards so stakeholders see sentiment + hard outcomes.

HardTechnical

99 practiced

Design a monitoring strategy for metric drift in a production recommendation system. Specify which statistical tests and distance measures (e.g., population stability index, KL divergence, KS test) you would use on feature and outcome distributions, thresholds for alerts, strategies to attribute drift to model vs data vs upstream changes, and recommended rollback or mitigation steps.

Sample Answer

Requirements & goals:- Detect metric/feature/outcome drift in daily/weekly windows for a production recommendation system.- Alert with low false positives, attribute root cause (model vs data vs upstream), and prescribe mitigations/rollback.

Monitoring layers:1. Data pipeline health (schema, counts, null rates, cardinality) - Alert if row count deviates >20% vs 7-day rolling average or if new columns appear/drop. - Monitor null% per key feature; alert if change >10 pp.

2. Feature distribution drift (input features to model) - Continuous features: use Population Stability Index (PSI) per day vs baseline (training or rolling 30d). Alert if PSI > 0.2 (moderate) and >0.3 (high). - Categorical features: use Chi-square or Jensen-Shannon divergence; alert if JS > 0.15. - Univariate supplement: Kolmogorov–Smirnov (KS) test for numeric features; use p-value < 0.01 plus effect size (KS statistic > 0.1).

3. Joint / multivariate drift - Monitor covariance change / correlation matrix Frobenius norm vs baseline; alert if norm increases >25%. - Track distribution of model score (predicted relevance): KL divergence or JS between score histograms; alert if JS > 0.1.

4. Outcome/label drift - Monitor conversion rate / CTR / purchase rate: use proportion z-test for daily vs baseline; alert if absolute difference > 2 standard errors and relative change >10%. - If delayed labels, use time-lagged windows and aggregate.

Threshold rationale:- PSI thresholds: widely used benchmarks (0–0.1 stable, 0.1–0.2 small, >0.2 significant).- Combine statistical significance (p-values) with effect-size thresholds to avoid hypersensitivity with large samples.

Attribution strategy (model vs data vs upstream):1. Pipeline triage (automated): - If schema/row-count/null alerts triggered → upstream data issue. - If new high-cardinality categories appear or distribution shifts concentrated on a single upstream source → upstream change.

2. Model vs data: - Replay inputs: score a recent baseline sample with current model and score current inputs with archived model. - If current inputs scored by archived model shift similarly to current model → data drift. - If current model on baseline inputs shows different outputs vs archived model → model change/bug. - Feature importance / SHAP drift: identify which features contributed most to prediction change. If features with large drift also have high importance → model performance impact.

3. A/B / experiment check: - Check release/deploy logs and upstream feature-engineering commits. Correlate timestamps of deploys with drift onset.

Mitigation & rollback steps:1. Immediate automated: - Suppress non-critical downstream actions (e.g., personalized emails) if outcome drift > critical threshold. - Trigger canary rollback: route small % of traffic to previous model while monitoring.

2. Investigation: - Snapshot sample of raw inputs + model scores + downstream outcomes; run replay tests. - Notify stakeholders with dashboard: per-feature drift metrics, timestamps, top contributing features.

3. Remediation: - If upstream data change: coordinate with data engineering to fix ETL or add transformations; patch and resume. - If model regression/bug: rollback to previous stable model; schedule urgent retrain if concept drift confirmed. - If gradual concept drift: schedule model retraining using recent labeled data, validate in staging, and deploy via canary.

Operational recommendations for BI role:- Build dashboards in Looker/Tableau showing per-feature PSI/KS, score distributions, label rates, and pipeline health with colored thresholds and links to example records.- Automate daily jobs to compute metrics, run tests, and post alerts to Slack with contextual links (samples, diffs).- Keep audit logs of alerts, actions, and outcomes to refine thresholds.

Example metric definitions (for implementation):- PSI(feature) = sum over bins ( (actual% - expected%) * ln(actual%/expected%) )- JS(P,Q) = 0.5 * (KL(P||M) + KL(Q||M)) with M=(P+Q)/2

This strategy balances sensitivity and practicality: combine univariate tests (PSI, KS) with multivariate checks and replay-based attribution to decide safe, documented rollback or retrain actions.

MediumTechnical

60 practiced

Explain how covariate-adjusted analysis (ANCOVA or regression adjustment) can increase power in randomized experiments. Describe necessary assumptions for unbiasedness, how to choose covariates, and give an example formula for adjustment when analyzing a continuous outcome.

Sample Answer

Covariate-adjusted analysis (ANCOVA / regression adjustment) increases power by explaining outcome variation with baseline covariates, thereby reducing residual variance and shrinking the standard error of the treatment effect estimate—so you detect smaller effects with the same sample size.

How it works (formula):Y_i = α + β T_i + γ' X_i + ε_i- Y_i: continuous outcome- T_i: treatment indicator (0/1)- X_i: vector of pre-treatment covariates- β is the adjusted treatment effect; with randomization β is an unbiased estimate of the ATE.

Necessary assumptions for unbiasedness:- Treatment is randomized (or unconfounded given X): T independent of potential outcomes.- Covariates X are pre-treatment (no post-treatment controls).- OLS model need not be perfectly specified for unbiased β if randomization holds—model misspecification affects efficiency but not bias.- Use robust or cluster-robust SEs if heteroskedasticity or clustering is present.

Why power increases:- Var(β̂) ≈ σ² / [n Var(T)] where σ² is residual variance after regressing out X. Good covariates lower σ² ⇒ smaller SE ⇒ higher power.

How to choose covariates:- Include strong prognostic (predictive) pre-treatment variables (baseline outcome, demographics, historical metrics).- Avoid post-treatment variables and variables measured with high noise.- Keep model parsimonious to avoid multicollinearity; pre-specify covariates to prevent fishing.- Consider interactions if you expect treatment effect heterogeneity (e.g., T*baseline).

Practical example (BI context):If evaluating an A/B test on weekly revenue:Y = weekly_revenue_postX = baseline_weekly_revenue, customer_segment, tenureEstimate: weekly_revenue_post = α + β A/B_group + γ1 baseline_revenue + γ2 segment + γ3 tenure + εReport β̂ with robust SE, adjusted means, and explain how baseline_revenue reduced residual variance and improved detection power.

Best practices:- Pre-register adjustment plan, check covariate balance, report both unadjusted and adjusted estimates, and use robust/clustered SEs when appropriate.

EasyTechnical

67 practiced

Explain the difference between user-level, session-level, and event-level units of analysis. Give an example where using event-level aggregation would mislead a product decision (e.g., a chat 'send' event where power users trigger many events), and recommend the correct unit-of-analysis for measuring feature adoption.

MediumTechnical

66 practiced

Design an experiment to evaluate a new recommendation ranking algorithm. Provide decisions and trade-offs for: unit of randomization (user vs session), metric selection (primary, secondary, guardrails), sample size considerations given 10M MAU and expected relative CTR lift of 2%, stratification or blocking strategies, rollout plan, and how you'd monitor for negative downstream effects (e.g., decreased retention).

Sample Answer

Requirements & goal:- Measure whether new ranking increases click-through rate (CTR) by ~2% relative vs control, without harming retention or downstream metrics.

Unit of randomization — choice & trade-offs:- User-level randomization (assign each user to treatment/control permanently): pros — avoids cross-session contamination, measures persistent effects (retention); cons — larger sample needed, slower to ramp; preferred when downstream metrics (retention, lifetime value) matter.- Session-level randomization: pros — smaller sample, quicker signal for immediate CTR; cons — contamination (user sees both experiences), can't measure long-term effects reliably.Recommendation: Use user-level as primary (since retention downstream matters). Consider a parallel session-level A/B as a quick diagnostic.

Metrics:- Primary: Impression-weighted CTR on recommended items (per-user aggregated to avoid heavy-hitters bias).- Secondary: Engagement depth (clicks per session on recommendations), conversions (if applicable), time-on-site attributable to recommendations.- Guardrails: 7- & 28-day retention (DAU/MAU), churn rate, downstream conversion rate, complaint/negative feedback rate, recommendation diversity and novelty metrics (to detect echo chambers).

Sample size (10M MAU, expected 2% relative CTR lift):- Let baseline CTR = p0 (e.g., 5%). Relative 2% -> absolute Δ = 0.02 * p0 = 0.001 if p0=0.05 (0.1pp).- For alpha=0.05, power=0.8, detecting Δ≈0.001 with p≈0.05 requires ~1–2 million users per arm. So with 10M MAU you can allocate ~20–40% of population (e.g., 2M control / 2M treatment) to get reliable detection.- If baseline CTR higher, required n decreases; if lower, n increases. Use standard two-proportion power calculator to compute exact n before launch.

Stratification / blocking:- Block randomization by key covariates to reduce variance: device type (mobile vs desktop), geography/timezone, new vs existing users, heavy vs light users. Implement stratified assignment to ensure balanced arms within each stratum. Consider post-stratification weighting in analysis.

Rollout plan:1. Offline validation / canary: run the ranking on a shadow traffic and compare scoring distributions and safety checks.2. Small user-level pilot: 1% treatment vs 1% control for 1–2 weeks to validate instrumentation and immediate regressions.3. Intermediate ramp: 5% → 20% while monitoring metrics and guardrails.4. Full roll if no negative signals and primary metric statistically significant and business-relevant.

Monitoring & detecting downstream harms:- Real-time dashboards for primary/secondary/guardrail metrics with alerting thresholds (stat sig and practical significance). Monitor both relative and absolute changes.- Cohort-based retention analysis (7/28/90 day) to detect delayed harm; run survival analysis and compute cumulative retention curves.- Sliced analysis: check effects across strata (new users, power users, regions).- Automated anomaly detection on revenue/churn/complaints.- Predefine kill-switch criteria (e.g., >1% absolute drop in 7-day retention or statistically significant drop beyond practical threshold) and rollback process.

Analysis notes:- Use intention-to-treat (assignments) for primary analysis. Adjust for covariates (via regression) to improve precision.- Correct for multiple comparisons when evaluating many secondary slices.- Report both statistical and practical significance (confidence intervals, uplift in % and absolute pp).

This design balances detection power for a small relative lift, protects long-term business metrics, and allows fast diagnostics via session-level checks and staged rollout.

Unlock Full Question Bank

Get access to hundreds of Quantitative Research and Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.