Experimentation & Growth Metrics Topics
Growth strategies, experimentation frameworks, and business optimization. Includes A/B testing, conversion optimization, and growth playbooks.
A/B Test Design & Statistical Rigor
Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.
Feature Success Measurement
Judging whether a shipped feature worked: defining success criteria before launch, measuring adoption and impact, and separating a feature's effect from background trends. Covers post-launch readouts, tying a feature to a target metric, and deciding whether to iterate, keep, or roll back. The scope is evaluating feature impact rather than designing the test that produced it.
Experiment Analysis & Result Interpretation
Reading out an experiment after it runs: interpreting the treatment effect, deciding ship/no-ship, and reconciling conflicting or flat results. Covers reasoning under uncertainty, acting on inconclusive or limited data, and translating a measured effect into a business decision. The emphasis is turning experiment output into a defensible recommendation.