Experimentation and Validation Questions

A) Technical-level (CTO)
Netflix uses a modular experimentation stack that separates control plane (experiment definition & targeting) from data plane (metrics collection & analysis) and the runtime (feature delivery & rollout).

Architecture components:

• Experimentation platform: authoring UI + API where engineers/product define variants, sample ratios, targeting rules, and duration. Stores experiment metadata and enforces assignment consistency.
• Feature-flag / rollout service: low-latency SDKs in clients/servers evaluate flags locally using deterministic user bucketing (hashed IDs) and exposure logging; supports staged rollouts, kill-switch, and config overrides.
• Metrics pipeline: high-throughput event ingestion (streaming) → enrichment → denormalized metrics store for near-real-time dashboards. Events include exposures, conversions, and diagnostics.
• Analysis engine: batch/stream statistical engine applying sequential testing corrections, Bayesian or frequentist models, automated A/A checks, and false-discovery controls. Produces significance, MDE, uplift and power reports.
• Monitoring & safety: anomaly detection, automated guardrails (SLO/health monitors), canary jobs, and automatic rollback triggers integrated with deployment pipelines.
  Data flow: define experiment → SDK assigns bucket & logs exposure → events flow to metrics pipeline → analysis engine computes results → rollout service uses outcomes to ramp or rollback.

B) Product-level (PM)
Think of it as a safe way to test ideas on real users with measurable business impact and minimal risk.

How it works:

• You design an experiment: who sees it, what the variants are, and which KPIs matter (primary metric + safety metrics like playback error rate).
• A feature-flag system delivers the variant to a randomized sample of users and lets you ramp exposure gradually (1% → 10% → 50%).
• Real-time metric collection and dashboards show effect on your primary metric and guardrail metrics. Automated checks flag anomalies.
• Statistical analysis tells you whether observed differences are meaningful (confidence, minimum detectable effect) and recommends when to stop, ramp, or roll back.
  Safety & impact controls:
• Start small with canaries, monitor health metrics continuously, have kill-switches and rollbacks automated, and require predefined success criteria before full rollout.
  Outcome focus:
• You get quantified lift (or harm) with clear confidence, and the platform enforces safety so you can move quickly without endangering core user experience.

Approach: Treat this as an end-to-end A/B experiment — define clear success criteria, ensure statistical rigor, and design the implementation so Engineering can safely deploy, measure, and roll back.

Primary & secondary metrics

• Primary metric: one business-focused, leading indicator tied to the feature goal. Example: “Conversion rate from trial to paid within 14 days” or “task completion rate.” Must be a single metric for hypothesis testing.
• Secondary metrics: safety and health signals (error rate, latency, CPU), engagement (time-on-task, DAU), downstream business KPIs (retention, revenue per user). Include uplift/downgrade thresholds to detect harm.

Sample size & duration

• Compute sample size using baseline conversion p0, minimum detectable effect (MDE, e.g., 5%), alpha=0.05, power=0.8. Use standard two-proportion formulas or an online calculator.
• Set duration to cover at least one full business cycle and account for user behavior periodicity (min 2–4 weeks for consumer; can be longer for low-traffic B2B). Ensure enough unique users, not sessions.

Rollout criteria

• Success: statistically significant improvement on primary metric and no degradation on safety metrics; secondary metrics within defined bounds.
• Stop/rollback rules: statistically significant harm on primary or any safety metric exceeds threshold (e.g., >2x baseline error rate), or infra limits reached.
• Progressive ramp: start with small percent (1–5%) -> canary (10–25%) -> full sample once checks pass.

Engineering collaboration

• Instrumentation: define event schema, naming, and dimensions (user-id, cohort, timestamp, variant). Use idempotent, strongly-typed telemetry events and include sampling flags.
• Measurement pipeline: ensure events flow to analytics (streaming and batch), validate data quality with smoke queries and backfills.
• Feature-flag design: implement server-side flag with kill-switch and rollout percentage; tie flag state to audit logs and deploy controls.
• Safety: add circuit-breakers, rate-limits, health checks, and observability dashboards (latency, error-rate, resource usage) before ramp.
• Reproducibility: log cohort assignment and seed so users are consistently bucketed; provide exportable datasets for stats team.
• Verification: run an experiment readiness checklist (event counts per cohort, schema validation, end-to-end test) before opening traffic.
• Communication: document success criteria, sample-size calc, rollback plan, and runbook; schedule a post-mortem and metric dashboard for real-time monitoring.

This ensures the experiment is statistically sound, measurable, safe to roll back, and engineered for reliable instrumentation and observability.