Clarify goal & constraints
- Goal: detect subtle regressions during short canaries (minutes) for a service with historically very low error rates, while keeping false positives low.
- Constraints: low baseline error → need statistical power; canary runs minutes → small sample sizes.
SLOs & signals
- Primary SLOs: Success rate ≥ 99.995% (error rate ≤ 0.005%) and p99 latency ≤ X ms.
- Secondary SLOs: p50/p95 latency, throughput, CPU/memory, user-facing business metrics (e.g., checkout completion).
- Instrument: per-request success/failure, latency histogram, request attributes (route, region, client version).
Aggregation windows & baselines
- Canary windows: 1m and 5m rolling windows for realtime detection; compare to baseline computed from 1h rolling historical distribution (same traffic pattern, day-of-week).
- Minimum sample threshold: require Nmin requests (e.g., 10k) or fall back to longer windows; if below, extend canary or delay promotion.
Statistical tests
- Use control-vs-canary comparison (two-sample Poisson/binomial test for error counts) or Fisher’s exact when small counts.
- Prefer Sequential Probability Ratio Test (SPRT) or Bayesian A/B with posterior credible intervals for fast decisions on streaming data.
- Example thresholds: require posterior probability > 99% that error rate increased by at least relative delta (e.g., 50% relative increase) OR p-value < 0.01 with multiple-testing correction (Bonferroni or BH).
- For latency, use bootstrap on p99 estimates or compare distributions with Mann–Whitney U plus effect-size threshold.
Alerting strategy
- Tiered alerts:
- Canary-warning (informational): single 1m signal crossing threshold but below sustained criteria.
- Canary-blocking (pager): sustained for 3 consecutive 1m windows OR statistical test meets confidence AND min sample size.
- Require corroboration across signals (e.g., error rate spike + p99 rise) to reduce false positives.
- Use burn-rate style: convert deviation into error-budget burn; automated rollback if burn rate exceeds threshold quickly.
- Suppress noisy alerts: auto-snooze if traffic below threshold; track multiple comparisons to avoid alert storms.
Operational practices
- Pre-run shadow traffic or traffic amplification to increase sample size in short canaries.
- Run control cohort with identical traffic to separate noise from regression.
- Continuously tune delta and sample thresholds based on historical detection power analysis; run offline power estimation to pick delta that product team cares about.
Why this works: combining control comparisons, sequential/Bayesian tests, minimum-sample rules, multi-signal corroboration, and tiered alerting gives high sensitivity for subtle regressions within short windows while keeping false positives low and aligning detection to business-impact thresholds.
