Site Reliability Engineering Principles Questions

Thirty days is enough time to get real minimum-viable monitoring in place and land a few concrete fixes, but not enough to justify an aggressive SLO target; the plan needs to be honest about that distinction from day one.

Structured elaboration

Week 1: instrument the golden signals (latency, traffic, error rate, saturation) even before any target exists, using whatever the org already has available rather than introducing new tooling under time pressure, e.g. Prometheus + Grafana (or a hosted equivalent like Datadog or a cloud provider's native metrics, such as CloudWatch) for metric collection and dashboards, and the existing on-call/paging tool (e.g. PagerDuty or Opsgenie) for alert routing, since standing up brand-new tooling in week 1 of a 30-day window competes directly with the time needed to actually instrument and observe. Set up basic alerting on obvious failure conditions (5xx spike, service down) that don't require historical baselining to be useful. Week 2: with a few days of real data, propose a deliberately conservative INITIAL SLO (looser than hoped-for, since there's no real baseline yet) and identify 1-2 quick-win reliability fixes visible from the new telemetry (an obvious N+1 query, a missing timeout, an unbounded retry loop). Weeks 3-4: implement the quick wins, tighten alerting thresholds based on the now-larger dataset, and present a plan for the NEXT phase (a real SLO, set with proper headroom) rather than pretending 30 days is enough to also validate a long-term target. The same underlying discipline applies whether the deadline is a relatively relaxed 30 days or a much tighter two-week window with no maintenance window available at all: the sequence (instrument first, then quick wins, then a real target) just compresses, with even more conservatism applied to any interim SLO or error-budget accounting during that short bootstrap period, since there isn't enough data yet to trust a tight number.

Worked example

For a service with no prior monitoring: Day 1-3, deploy request-count, error-rate, and p95-latency instrumentation using the org's existing Prometheus/Grafana stack (or the closest equivalent already running elsewhere in the company), wired into the existing PagerDuty rotation rather than a new one. Day 4-10, observe; find that 15% of requests hit an endpoint making an unindexed database query under load, an easy, well-understood fix. Day 11-20, ship the fix, observe improvement (p95 latency drops from 900ms to 200ms). Day 21-30, propose an initial SLO (e.g. 99% success / p95 < 400ms, comfortably above what's now being observed) as a placeholder, explicitly flagged to stakeholders as provisional pending a longer observation window.

Trade-offs and pitfalls

The biggest risk under a hard 30-day (or shorter) deadline is a product manager interpreting "we set an SLO" as "the reliability problem is solved," when in reality only the FIRST cheap fixes have landed and the number itself is still a conservative placeholder; that expectation needs managing explicitly and early, not left implicit. It's also tempting to skip instrumentation and jump straight to "fixes" under deadline pressure, but without baseline telemetry there's no way to demonstrate the fixes actually helped, which undermines the whole exercise's credibility with the very stakeholder who's watching for measurable improvement. Introducing an entirely new observability stack from scratch inside a 30-day window is itself a common mistake: reusing whatever tooling the org already trusts and knows how to operate is almost always faster than evaluating and onboarding a new platform under deadline pressure.

Static thresholds fail on traffic with real daily/weekly seasonality; adapting the threshold from the data itself via a smoothed baseline and a spread estimate is the standard fix, but the naive single-pass version of this hides a real accuracy bug.

Structured elaboration

A sliding-window z-score approach scores each new point against the mean and standard deviation of the last N points, flags it as anomalous if the deviation exceeds a chosen multiple of sigma, then adds it to the window. Testing revealed that an EWMA-based variance estimator (a common textbook alternative that avoids keeping a window in memory) has a real flaw here: with a smoothing factor high enough to track real level changes responsively, its effective sample size for the VARIANCE estimate is far too small (roughly $1/\alpha$ points), so the estimated standard deviation itself swings noisily, and whenever it happens to dip low, an entirely ordinary point reads as a spurious multi-sigma anomaly.

Worked example (executed; python3, 5+ seeds, before/after comparison)

The first (EWMA-variance) version, tested against 80 points of injected Gaussian noise (mean 100, std 5) plus one genuine spike at 300, flagged the real spike correctly but ALSO flagged 3 ordinary points (values 91.7, 89.9, 106.4, none more than 1.8 true standard deviations from the mean) as false anomalies purely because the EWMA-variance estimate had randomly dipped. Switching to a fixed 30-point rolling window for both the mean and standard deviation:

from collections import deque
import statistics

def sliding_window_anomaly(values, window_size=30, z_thresh=3.0, min_periods=20):
    window = deque(maxlen=window_size)
    out = []
    for v in values:
        flagged = False
        if len(window) >= min_periods:
            mu = statistics.fmean(window)
            sd = statistics.pstdev(window)
            if sd > 1e-9:
                flagged = abs(v - mu) / sd > z_thresh
        out.append((v, flagged))
        window.append(v)
    return out

flagged ONLY the genuine spike, with zero false positives on the same test data, and a false-positive rate of 0.35%-0.80% across five separate pure-noise seeds of 2,000 points each (close to the ~0.27% a true 3-sigma threshold implies on i.i.d. Gaussian data, with the small remaining excess coming from estimating standard deviation off a finite 30-point sample rather than the true population value).

Trade-offs and pitfalls

The window size trades detection speed for stability: a smaller window adapts faster to legitimate seasonality but re-introduces the same kind of variance instability seen in the EWMA version; a larger window is more stable but slower to adapt after a genuine, permanent level shift (e.g. after a deploy that legitimately changes baseline latency). min_periods matters just as much as the window size: evaluating z-scores before the window has enough points to estimate a meaningful standard deviation is what let the EWMA version misfire early in its own warm-up period.

An automated burn-rate mitigation system needs tiered detection windows (fast for severe spikes, slower to confirm sustained burns) each mapped to a specific, increasingly aggressive automated action, not a single on/off trigger.

Structured elaboration

A layered design: a SHORT window (e.g. 5 minutes) at a HIGH burn-rate threshold (e.g. 10x) catches a sudden severe outage quickly and can trigger the most conservative automated response (temporary feature rollback of the most recent risky change) with minimal delay; a LONGER window (e.g. 6 hours) at a LOWER threshold (e.g. 2x) confirms a slow, sustained degradation that a short window alone would miss, and triggers a less drastic response (throttle non-critical feature flags, or scale up capacity if the degradation looks saturation-related rather than error-related). Each preemption level maps to a specific, reversible action: level 1 (early warning) notifies on-call with no automated change; level 2 (moderate, sustained burn) throttles specific feature flags identified as recently changed; level 3 (severe or fast burn) automatically rolls back the most recent deploy and pages.

Worked example

At a 99.9% SLO (30-day error budget of 43.2 minutes): level 1 fires at burn rate > 2x sustained over 6 hours (notify only), which by the time it fires has consumed about 2*(6/720) = 1.67% of the monthly budget; level 2 fires at burn rate > 6x sustained over 1 hour (auto-throttle the most recently enabled feature flags), consuming about 6*(1/720) = 0.83% of the monthly budget by the time it fires; level 3 fires at burn rate > 14x sustained over 5 minutes (auto-rollback the last deploy and page), consuming only about 14*(5/60/720) = 0.16% of the monthly budget by the time it fires. This is the standard multi-window multi-burn-rate pattern from the Google SRE workbook's chapter on alerting, but the design goal is NOT that every level consumes the same fraction of budget; they don't (1.67%, 0.83%, and 0.16% here). The actual goal is the opposite: the fastest, highest-severity alert (short window, high multiplier) is tuned to fire having consumed as LITTLE budget as possible, because it exists to catch a severe spike before much damage is done, while a slower, lower-multiplier alert is allowed to consume more budget before firing, because distinguishing a genuine sustained low-grade burn from ordinary noise requires a longer observation window, and firing too early on a short window at a low multiplier would false-positive constantly.

Trade-offs and pitfalls

Every automated mitigation (feature rollback, auto-scaling, throttling) needs a safety valve: an automatic rollback that itself has a bug, or that fires on a false-positive burn signal, can cause more disruption than the incident it was meant to contain, so each level should have a manual override and a rate limit on how often it can fire automatically before requiring human confirmation. It's also worth distinguishing SATURATION-caused burn (where auto-scaling is the right response) from CODE-defect-caused burn (where rollback is the right response); triggering the wrong automated action for the wrong root cause can make an incident worse, so the system needs at least a rough signal (e.g. correlating burn with a recent saturation metric versus a recent deploy event) to pick the right lever.

A per-service burn-rate report needs the allowed-errors figure computed from each service's OWN request volume, not a single fleet-wide number, since services with different traffic naturally have different absolute error budgets even at the same SLO percentage.

Structured elaboration

For each service, total requests and observed errors are aggregated over the 30-day window, and the allowed-errors figure is simply the SLO's allowed-bad-fraction multiplied by that service's own total request count; burn rate is then observed errors divided by allowed errors, giving a comparable, unitless number across services regardless of how much traffic each one handles.

Worked example (executed; sqlite3, two services with different traffic and error counts)

WITH agg AS (
  SELECT service_name,
         COUNT(*) AS total_requests,
         SUM(CASE WHEN success = 0 THEN 1 ELSE 0 END) AS observed_errors
  FROM service_metrics
  WHERE ts >= date('2026-07-01') AND ts < date('2026-07-01', '+30 day')
  GROUP BY service_name
)
SELECT service_name, total_requests, observed_errors,
       total_requests * 0.0005 AS allowed_errors,
       ROUND(observed_errors * 1.0 / NULLIF(total_requests * 0.0005, 0), 2) AS burn_rate
FROM agg;

Against a synthetic dataset with checkout (2 requests, 0 errors) and images (4 requests, 1 error) at a 99.95% SLO: checkout correctly returned a burn rate of 0.0, and images returned 500.0 (1 observed error against an allowed-errors figure of 4 * 0.0005 = 0.002, meaning even a single error on a very small sample massively exceeds the allowed rate), correctly illustrating why burn rate on a low-traffic service needs a minimum sample-size guard before it's trusted.

Trade-offs and pitfalls

The NULLIF guard prevents a divide-by-zero for a service with zero requests in the window, but it does not solve the more subtle problem the images result illustrates: at very small request counts, a tiny absolute number of errors produces an enormous, statistically unstable burn-rate multiple that looks alarming but reflects noise more than a real trend; a production version should suppress or flag burn-rate values for services below a minimum request-count threshold rather than reporting them at face value.

Reporting a daily error rate that must show 0% on days with zero traffic requires generating the full calendar of days first and left-joining actual data onto it, rather than grouping only the rows that exist.

Structured elaboration

A plain GROUP BY date(occurred_at) silently omits any day with no matching rows, which is exactly the case this question calls out as a requirement to handle correctly. The fix is a recursive date-spine (or an equivalent calendar table) covering the full 30-day range, LEFT JOINed to the aggregated daily counts, with COALESCE converting missing aggregates to zero before computing the percentage (and guarding the percentage calculation itself against a divide-by-zero when total is zero).

Worked example (executed; sqlite3, verified against a dataset with an intentional zero-request day)

WITH RECURSIVE days(d) AS (
  SELECT date('2026-07-01')
  UNION ALL
  SELECT date(d, '+1 day') FROM days WHERE d < date('2026-07-01', '+29 day')
),
daily AS (
  SELECT date(occurred_at) AS d, COUNT(*) AS total,
         SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) AS bad
  FROM http_logs
  WHERE service = 'checkout'
    AND occurred_at >= datetime('2026-07-01') AND occurred_at < datetime('2026-07-01','+30 day')
  GROUP BY date(occurred_at)
)
SELECT days.d,
       COALESCE(daily.total, 0) AS total_requests,
       COALESCE(daily.bad, 0) AS errors,
       ROUND(CASE WHEN COALESCE(daily.total,0) = 0 THEN 0.0
                  ELSE daily.bad * 100.0 / daily.total END, 2) AS error_pct
FROM days LEFT JOIN daily ON days.d = daily.d
ORDER BY days.d;

Against a dataset with requests on July 1st and 3rd but NONE on July 2nd, the query correctly returned 50.0% error for July 1st (1 of 2 requests failed), and exactly 0.0% (not a missing row, not NULL) for the zero-traffic July 2nd, confirming the calendar-spine approach satisfies the stated requirement.

Trade-offs and pitfalls

A date-spine built with a recursive CTE is portable but can be slow at very large day-ranges in some engines; PostgreSQL's generate_series is the more idiomatic and typically faster equivalent for the same purpose. It's also worth being explicit about WHY a zero-traffic day reports 0% rather than being excluded or shown as NULL: for an SLO dashboard, a silent gap (day missing from the output entirely) is easy to misread as "no problem that day" when it might actually mean the service was down so hard it received no traffic at all, which a naive GROUP BY would hide.