Growth & Business Optimization Topics
Growth strategies, experimentation frameworks, and business optimization. Includes A/B testing, conversion optimization, and growth playbooks.
Segmentation & Dimensionality in Metrics
Learn to think about how metrics vary across dimensions: user segment, geography, traffic source, device type, etc. Practice deciding which dimensions are critical to track separately. Understand why slicing metrics reveals insights (e.g., desktop vs. mobile retention may differ significantly).
Statistical Rigor & Avoiding Common Pitfalls
Demonstrate deep understanding of statistical concepts: power analysis, sample size calculation, significance levels, confidence intervals, effect sizes, Type I and II errors. Discuss common mistakes in test interpretation: peeking bias (checking results too early), multiple comparison problem, regression to the mean, selection bias, and Simpson's Paradox. Discuss how you've implemented safeguards against these pitfalls in your testing processes. Provide examples of times you've caught flawed analyses or avoided incorrect conclusions.
Impact Driven Mindset
Approach and habits that prioritize measurable impact over activity. Topics include defining success criteria and hypotheses, using data to compare and prioritize initiatives, selecting work with the highest expected business return, balancing short term wins and long term investments, time boxing experiments and minimum viable solutions to learn quickly, and communicating impact oriented choices to stakeholders. Candidates should be ready to show examples of how they set impact goals, measured results, and redirected effort based on outcomes.
Experimentation Methodology and Rigor
Focuses on rigorous experimental methodology and advanced testing approaches needed to produce reliable, actionable results. Topics include statistical power and minimum detectable effect trade offs, multiple hypothesis correction, sequential and interim analysis, variance reduction techniques, heterogenous treatment effects, interference and network effects, bias in online experiments, two stage or multi component testing, multivariate designs, experiment velocity versus validity trade offs, and methods to measure business impact beyond proximal metrics. Senior level discussion includes designing frameworks and practices to ensure methodological rigor across teams and examples of how to balance rapid iteration with safeguards to avoid false positives.
A and B Test Design
Designing and running A and B tests and split tests to evaluate product and feature changes. Candidates should be able to form clear null and alternative hypotheses, select appropriate primary metrics and guardrail metrics that reflect both product goals and user safety, choose randomization and assignment strategies, and calculate sample size and test duration using power analysis and minimum detectable effect reasoning. They should understand applied statistical analysis concepts including p values confidence intervals one tailed and two tailed tests sequential monitoring and stopping rules and corrections for multiple comparisons. Practical abilities include diagnosing inconclusive or noisy experiments detecting and mitigating common biases such as peeking selection bias novelty effects seasonality instrumentation errors and network interference and deciding when experiments are appropriate versus alternative evaluation methods. Senior candidates should reason about trade offs between speed and statistical rigor plan safe rollouts and ramping define rollback plans and communicate uncertainty and business implications to technical and non technical stakeholders. For developer facing products candidates should also consider constraints such as small populations cross team effects ethical concerns and special instrumentation needs.
Experimentation and Product Validation
Designing and interpreting experiments and validation strategies to test product hypotheses. Includes hypothesis formulation, experimental design, sample sizing considerations, metrics selection, interpreting results and statistical uncertainty, and avoiding common pitfalls such as peeking and multiple hypothesis testing. Also covers qualitative validation methods such as interviews and pilots, and using a mix of methods to validate product ideas before scaling.
Hypothesis and Test Planning
End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.
Metrics Selection and Diagnostic Interpretation
Addresses how to choose appropriate metrics and how to interpret and diagnose metric changes. Includes selecting primary and secondary metrics for experiments and initiatives, balancing leading indicators against lagging indicators, avoiding metric gaming, and handling conflicting signals when different metrics move in different directions. Also covers anomaly detection and root cause diagnosis: given a metric change, enumerate potential causes, propose investigative steps, identify supporting diagnostic metrics or logs, design quick experiments or data queries to validate hypotheses, and recommend remedial actions. Communication of nuanced or inconclusive results to non technical stakeholders is also emphasized.
Experiment Design and Execution
Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.