Monitoring, Logging, and Observability Questions

Direct answer

Monitor the business-critical user journeys, not just individual endpoints: track success rate for flows like login, checkout, or search, run synthetic checks that replay those flows continuously, and compare the new release's error and journey-success rates against the previous release's baseline using a statistical comparison rather than a fixed threshold, so a noisy small sample doesn't page someone unnecessarily.

Structured elaboration

What to monitor post-release

• Journey-level success rate for the flows that matter to the business (not raw endpoint 200/500 counts alone, since a flow can "succeed" at the HTTP layer while still failing the user, e.g. a checkout that returns 200 but never charges the card).
• Synthetic canaries: scripted runs of critical flows on a schedule, independent of real user traffic, so you get signal even during low-traffic windows.
• Client-side signal where relevant: crash-free session rate, JS error rate, rage-clicks/dead-clicks if you have RUM (real user monitoring) instrumentation.
• Canary/cohort comparison: route a percentage of traffic to the new release and compare its error rate against the stable baseline cohort, rather than comparing today's absolute number to a fixed historical number.

How to alert without triggering on noise
A newly-deployed canary usually serves far fewer requests than the baseline, so comparing raw percentages directly is misleading: a small sample can show a "higher" error rate purely from chance. Use a two-proportion significance test (comparing whether two observed rates are different by more than sampling noise would explain) before paging, and require the difference to be both statistically significant and above a minimum practical size (e.g., at least a few percentage points), so a "significant" but trivial 0.01% difference doesn't fire.

Feeding it back into testing
Every regression that monitoring catches but tests missed is a test-escape: the incident review should produce a new automated test (unit, integration, or synthetic) that would have caught it, and the monitoring signal that caught it (which journey, which metric) should be reviewed to see if it needs to trigger earlier in the pipeline (e.g., as a pre-production canary gate) next time.

Worked example

Suppose a release goes to a 2,000-request canary cohort while the stable baseline serves 10,000 requests in the same window. Baseline shows 40 errors, canary shows 25 errors.

$$\hat p_1 = \frac{40}{10000} = 0.004, \qquad \hat p_2 = \frac{25}{2000} = 0.0125$$

Pooled error rate across both cohorts:

$$\hat p = \frac{40+25}{10000+2000} = 0.005417$$

Two-proportion z-statistic:

$$z = \frac{\hat p_2 - \hat p_1}{\sqrt{\hat p(1-\hat p)\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$$

Plugging in: standard error $\approx 0.001798$, so $z \approx \frac{0.0085}{0.001798} \approx 4.73$. A z-score that far from zero (well above the ~1.96 threshold for a 95% confidence two-sided test) means this isn't noise: the canary's error rate is a genuine regression, not sampling variance, and it should page. If the canary had instead shown, say, 3 errors out of 2,000 (a rate of 0.15%, actually below baseline), the same test would correctly stay quiet even though 3 is a "different number" from 40.

Trade-offs & pitfalls

• Comparing raw percentages without accounting for sample size is the single most common cause of false canary alarms early in a rollout, when the canary cohort is still small.
• Journey-level monitoring requires actual investment in defining and maintaining synthetic scripts; if they aren't kept in sync with real user flows, they silently stop testing what matters.
• Overly sensitive significance thresholds (alerting on any statistically significant difference, however tiny) reintroduce the alert-fatigue problem from a different angle; pair the significance test with a minimum effect-size floor.
• A test-escape process without an actual ticket/tracking loop back to the test suite tends to catch the same class of regression repeatedly.