Data Analysis and Insight Generation Questions

Ability to convert raw data into clear, evidence based business insights and prioritized recommendations. Candidates should demonstrate end to end analytical thinking including data cleaning and validation, exploratory analysis, summary statistics, distributions, aggregations, pivot tables, time series and trend analysis, segmentation and cohort analysis, anomaly detection, and interpretation of relationships between metrics. This topic covers hypothesis generation and validation, basic statistical testing, controlled experiments and split testing, sensitivity and robustness checks, and sense checking results against domain knowledge. It emphasizes connecting metrics to business outcomes, defining success criteria and measurement plans, synthesizing quantitative and qualitative evidence, and prioritizing recommendations based on impact feasibility risk and dependencies. Practical communication skills are assessed including charting dashboards crafting concise narratives and tailoring findings to non technical and technical stakeholders, along with documenting next steps experiments and how outcomes will be measured.

MediumTechnical

67 practiced

You suspect the analytics pipeline undercounts mobile users due to a tracking SDK bug. Explain how you would detect sampling bias using available data (device metrics, server logs), design an analysis to estimate the magnitude of undercounting (backfill comparisons, control groups), and propose corrective steps and validation checks to ensure accuracy going forward.

Sample Answer

Approach summary:1) Detect bias using signals orthogonal to the SDK (device metrics, server logs).2) Quantify undercounting with backfill comparisons, control groups, and statistical adjustment.3) Remediate instrumentation and validate with ongoing monitoring and audits.

Detection steps:- Compare device-side counts (app launches, session starts from device telemetry) vs. analytics events. Compute ratio by OS version, device model, app version, and region. Large systematic deviations concentrated in mobile segments indicate SDK loss.- Use server logs (API hits, auth tokens, push tokens) as ground-truth proxies for active users and compare daily active user (DAU) trends to tracked analytics DAU.- Look for sudden drops correlated with SDK rollouts, feature flags, or OS updates. Plot cumulative distribution of session lengths and inter-event times to spot missing short sessions.

Analysis to estimate magnitude:- Backfill comparison: For a period, instrument a parallel reliable pipeline (e.g., server-side event capture or a lightweight fallback beacon). Compare counts per cohort (device model, app version) and compute undercount factor f = true_count / observed_count. Report f with 95% CI using bootstrap.- Control groups: Roll out an A/B where a small percent of clients use patched SDK or server-side tracking. Estimate lift in captured users; use difference-in-differences to control for temporal trends.- Modeling: Fit logistic regression predicting "event captured" using device/app covariates; use inverse-probability weighting to estimate true totals. Cross-check with backfill and control estimates.- Uncertainty: Propagate errors and give ranges by cohort; flag cohorts with high variance.

Corrective steps:- Fix SDK bug and release patch; add Canary rollout + feature flags to limit blast radius.- Implement server-side deduplication and fallback beacons for critical events (session start, install).- Add deterministic identifiers (hashed device id) where privacy allows to join device/server signals and detect misses.- Add telemetry in SDK: counters for dropped events, buffer overflow, network errors, and version tags.

Validation checks and monitoring:- Post-release A/B: measure convergence of patched vs. legacy capture ratios; expect ratio -> 1 within CI.- Continuous monitoring dashboards: device vs server DAU ratio by cohort, SDK error rates, event latency, retransmit counts, and capture rate by app version.- Automated alerts for anomalies (threshold or trained anomaly detector).- Periodic audits: sample device logs, replay events through analytics pipeline and validate end-to-end integrity.- Data-quality tests in ETL: row-count reconciliation, survival analysis of session durations, and re-run backfill comparisons monthly.

This combination provides detection, a statistically principled estimate of undercounting, targeted fixes, and ongoing validation to prevent regression.

HardTechnical

56 practiced

You observed a 4% increase in retention after a new onboarding flow. Craft a concise executive one-paragraph narrative that includes: the metric with baseline and lift, statistical evidence and robustness checks summary, business impact estimate (e.g., ARR or retention lift translation), recommended next steps, and brief measurement plan for rollout. Then list 4 key dashboard bullets that support that narrative.

MediumTechnical

51 practiced

You run 30 A/B tests in parallel each reporting p-values for the same primary metric. Explain why multiple testing corrections are necessary, compare family-wise error rate control (Bonferroni) vs false discovery rate (Benjamini-Hochberg), and demonstrate how to apply Benjamini-Hochberg to a list of p-values in brief pseudocode or Python.

Sample Answer

When you run 30 A/B tests (30 hypothesis tests) on the same metric, multiple testing corrections are necessary because the probability of at least one false positive (Type I error) increases with the number of tests. If each test uses α=0.05 independently, the chance of ≥1 false positive ≈ 1 − (1−0.05)^30 ≈ 0.79, so many "significant" results will be spurious unless adjusted.

Comparison: Bonferroni vs Benjamini–Hochberg (BH)- Bonferroni (Family-Wise Error Rate control): divides α by m (α' = α/m). Very conservative; controls probability of any false positive. Good when false positives are very costly, but reduces power (more false negatives).- Benjamini–Hochberg (False Discovery Rate control): controls expected proportion of false discoveries among declared positives. Less conservative, higher power—suitable when you expect some true effects and tolerate a controlled fraction of false positives (common in A/B testing pipelines).

When to choose:- Use Bonferroni if a single false alarm causes major cost (safety-critical decisions).- Use BH for analytics/experimentation where you prefer discovering more true effects and can tolerate some false discoveries.

Benjamini–Hochberg pseudocode / Python:

python

def benjamini_hochberg(pvals, alpha=0.05):
    """
    pvals: list of p-values
    returns: boolean list indicating which hypotheses to reject
    """
    m = len(pvals)
    indexed = sorted(enumerate(pvals), key=lambda x: x[1])  # (idx, p)
    rejects = [False]*m
    # find largest k such that p_(k) <= (k/m)*alpha
    max_k = -1
    for rank, (idx, p) in enumerate(indexed, start=1):
        if p <= (rank/m)*alpha:
            max_k = rank
    if max_k >= 1:
        # reject all p_(1..max_k)
        for rank in range(1, max_k+1):
            orig_idx = indexed[rank-1][0]
            rejects[orig_idx] = True
    return rejects

Key notes: BH assumes p-values are independent or positively dependent; it controls FDR at level α. Always report adjusted method and chosen α when presenting experiment results.

HardTechnical

63 practiced

A product analyst monitored an A/B test daily and stopped when p<0.05, then reported significance. Explain why this is problematic (optional stopping) and describe statistical methods that allow valid sequential monitoring: alpha-spending functions (O'Brien-Fleming, Pocock), sequential probability ratio test (SPRT), and Bayesian monitoring. Provide guidance on practical implementation and how to pre-register monitoring plans.

Sample Answer

Why stopping an A/B test the first day p<0.05 is problematic- Stopping as soon as you see p<0.05 is “optional stopping” (data-dependent stopping). Repeated looks inflate the Type I error — the nominal 5% no longer holds. If you peek daily, the chance of a false positive can rise dramatically (often >>5%) because each look is another opportunity to reject H0 by chance.- Consequence: decisions, model rollouts or business changes based on those inflated p-values are likely to be unstable and non-reproducible.

Valid sequential-monitoring approaches1. Alpha-spending / group-sequential methods- Idea: allocate the overall alpha (e.g., 0.05) across multiple planned looks using an alpha-spending function. Two common boundaries: - O’Brien–Fleming: very conservative early looks, most alpha spent near the end. Good when you want to avoid early false positives. - Pocock: more equal alpha across looks, easier early stopping but requires larger final sample for same power.- Implementation: pre-specify number/timing of interim analyses or use information time (fraction of total information). Use standard libraries (e.g., gsDesign in R, Python ports) to compute critical z-values for each look, and only reject if Z_k > boundary_k.

2. Sequential Probability Ratio Test (SPRT)- Idea: continuous sequential test comparing likelihood ratio to two thresholds (accept H0, accept H1, or continue). Minimizes expected sample size for given Type I/II errors.- Good when you have a well-specified alternative (effect size) and want efficient stopping.- Implementation: compute cumulative log-likelihood ratio after each observation or batch; stop when above upper or below lower threshold. Use Wald bounds adjusted for desired alpha/beta.

3. Bayesian monitoring- Idea: compute posterior probability of treatment effect (or Bayes factor) after each look. No fixed Type I error paradigm; inference conditions on observed data. Stopping rules can be based on posterior prob(Δ>0) > p_threshold or Bayes factor thresholds.- Advantages: coherent updating, intuitive decision rules, can incorporate priors. Caveat: priors and decision thresholds must be justified; stakeholders may still want frequentist calibrations.- Practical calibration: run simulations under H0 and plausible alternatives to estimate false-positive rates and power given your prior and stopping rule.

Practical implementation guidance- Pre-plan: decide in advance the monitoring frequency (batched daily/weekly), the statistical approach, maximum sample size, minimum detectable effect, and the decision thresholds.- Use batch looks, not one-by-one updates, unless using SPRT or carefully calibrated Bayesian rules.- Simulate: before running the experiment, simulate under null and alternatives to estimate Type I error, power, average sample size, and expected stopping times for your chosen method and monitoring cadence.- Tooling: use established libraries (R: gsDesign, rpact; Python: sequential, custom likelihood code) and validate with unit tests and simulation notebooks.- Reporting: when reporting results, include the pre-registered plan, number/timing of looks, alpha-spending boundaries or Bayes thresholds, actual information time, and results of pre-run simulations.

How to pre-register monitoring plans- Record: primary metric, hypothesis, one-sided/two-sided, target MDE, baseline rate, planned maximum N (or information), monitoring frequency, chosen method (alpha-spending/OBF/Pocock/SPRT/Bayesian), exact stopping thresholds, and analysis population.- Share: publish to internal experiment registry or public pre-registration (e.g., GitHub/GitLab, OSF).- Lock: timestamp the plan and avoid post-hoc changes; if changes are necessary, document them and treat the result as exploratory.- Include governance: who may stop the experiment, who interprets results, and rollout rules if stopped early.

Bottom lineOptional stopping without correction invalidates p-values. Use a pre-registered sequential method (alpha-spending, SPRT, or Bayesian) and simulate your plan so decisions about ML model rollouts are statistically sound and defensible.

EasyTechnical

99 practiced

Given a new dataset for product engagement with columns: user_id, session_id, session_start (timestamp), session_duration_seconds, and events_count, describe an exploratory analysis plan. List the summary statistics and visualizations (histogram, boxplot, time series decomposition, pivot tables) you would compute, the objective for each (detect skew, seasonal patterns, anomalies, distribution tails), and three suspicious patterns you would flag for deeper investigation.

Sample Answer

Overview: I’d run descriptive stats, distributions, temporal analysis, and cross-tabs to understand user engagement, data quality, and signals useful for modeling (features, skew, outliers, seasonality).

Summary statistics to compute (with pandas):- Row counts, unique user/session counts, missingness per column — objective: data completeness and duplication.- session_duration_seconds: mean, median, std, min, max, IQR, skew, kurtosis — objective: central tendency, dispersion, tail-heaviness.- events_count: mean, median, std, zero-rate, quantiles — objective: usage intensity and zero-inflation.- sessions per user distribution (groupby user_id).session_id.count(): mean, median, top percentiles — objective: heavy users vs. casual.- session_rate per time-window (daily/weekly): counts and percent change — objective: trend detection.

Visualizations and objective:- Histogram (session_duration_seconds, events_count, log-transformed): detect skew, multi-modality, need for transformation.- Boxplot (by device/segment if available or by weekday): detect outliers and distribution shifts across segments.- Time series plot (daily total sessions, avg duration, avg events): detect trends, jumps, drops.- Time series decomposition (trend/seasonal/residual) on daily session counts: reveal weekly/daily seasonality and underlying trend.- Heatmap (hour-of-day × day-of-week session counts): detect usage cadence and peak windows.- ECDF or percentile plot for session_duration_seconds: understand tail behavior for feature capping.- Pivot tables: weekday vs hour average duration/events; user cohort retention (first-session week vs subsequent activity): objective — behavioral patterns and cohort decay.

Three suspicious patterns to flag:1) Large spike of sessions with identical short duration (e.g., 1–2s) and zero events_count — possible bot traffic, instrumenting error, or app crash.2) Many duplicate session_ids or session_start timestamps per user — ingestion/ID generation bug.3) Heavy right-tail with extremely long durations but zero events_count or constant events_count — likely session never closed, background process, or logging issue.

Next steps if flagged: filter and inspect raw logs, join with device/app-version metadata, sample user timelines, and consider feature engineering (log transforms, capping, bot filters) before modeling.

Unlock Full Question Bank

Get access to hundreds of Data Analysis and Insight Generation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.