Site Reliability Engineering Principles Questions

Translating a business goal into SLIs and an SLO is the practical skill behind the SLI/SLO/SLA distinction: you start from what the business is promising, and work backward to what must be measured and enforced to keep that promise honestly.

Structured elaboration

SLI = the measurement (e.g. proportion of checkout attempts that complete successfully), SLO = the internal target on that measurement (e.g. 99.9% of checkout attempts succeed over a rolling 30-day window), SLA = the external, often contractual, commitment (looser than the SLO, with defined consequences). Working from "99.9% checkout success" backward: first define what "success" means precisely (a completed order with payment confirmed, not just a 200 status from the checkout API, since a technically-200 response that fails payment silently is not a business success); then instrument that precise definition as the SLI; then set the SLO target with headroom, and define the error-budget action (e.g. below a certain burn threshold, checkout-related feature releases pause automatically).

Worked example

Business goal: "99.9% of checkout attempts should succeed." SLI: proportion of checkout sessions where payment is confirmed and an order ID is returned, out of all checkout sessions initiated, measured over a 5-minute rolling window for alerting and 30 days for the SLO. SLO: 99.9% over 30 days. Error-budget action: if burn rate exceeds 3x over a 1-hour window, checkout-path deploys are automatically restricted to hotfix-only until an SRE manually clears the restriction.

Trade-offs and pitfalls

The most common mistake translating a business goal into an SLI is measuring something adjacent but not equivalent to the actual business outcome (e.g. "API returned 200" instead of "payment actually confirmed"), which can look healthy on a dashboard while the business goal is quietly being missed. It's also worth deciding early whether abandoned carts (a user who leaves before completing checkout, not a system failure) should be excluded from the denominator; conflating user abandonment with system failure either inflates or deflates the SLI in a way that misrepresents actual reliability.

A simple availability alert compares a directly-computed success ratio against a fixed threshold and requires that condition to persist for a sustained window, which is the right first alert to build before adding burn-rate sophistication.

Structured elaboration

The ratio itself comes from two counters (successful requests over total requests for the service), aggregated with sum(rate(...)) over a short window (5 minutes here) so the ratio reflects RECENT behavior rather than an all-time average; a for: clause on the alert then requires the low-availability condition to hold continuously for the stated duration before paging, which filters out single-scrape noise.

Worked example (executed; promtool unit test against a real Prometheus binary)

- record: job:availability:5m
  expr: sum(rate(http_requests_total{service="images",code="200"}[5m]))
      / sum(rate(http_requests_total{service="images"}[5m]))
- alert: LowAvailability
  expr: job:availability:5m < 0.9995
  for: 15m
  labels:
    severity: page

Verified against a synthetic counter pair (994 good : 6 bad requests per interval, i.e. an observed availability of 99.4%): the alert correctly fired after the 15-minute persistence window against a 99.95% threshold.

Trade-offs and pitfalls

A short 5-minute evaluation window is sensitive to real changes quickly but is also the window most vulnerable to a low-traffic service producing a wildly noisy ratio from just one or two errors; if this service handles only a handful of requests per 5 minutes, widen the window or add a minimum-request-count guard so the alert doesn't fire on statistically meaningless samples. This alert alone also does not distinguish a slow, sustained low-grade degradation from a sudden severe outage, both look identical once the 15-minute persistence has elapsed; that distinction is exactly what a second, burn-rate-based alert (see the companion burn-rate design) is for.

Static thresholds fail on traffic with real daily/weekly seasonality; adapting the threshold from the data itself via a smoothed baseline and a spread estimate is the standard fix, but the naive single-pass version of this hides a real accuracy bug.

Structured elaboration

A sliding-window z-score approach scores each new point against the mean and standard deviation of the last N points, flags it as anomalous if the deviation exceeds a chosen multiple of sigma, then adds it to the window. Testing revealed that an EWMA-based variance estimator (a common textbook alternative that avoids keeping a window in memory) has a real flaw here: with a smoothing factor high enough to track real level changes responsively, its effective sample size for the VARIANCE estimate is far too small (roughly $1/\alpha$ points), so the estimated standard deviation itself swings noisily, and whenever it happens to dip low, an entirely ordinary point reads as a spurious multi-sigma anomaly.

Worked example (executed; python3, 5+ seeds, before/after comparison)

The first (EWMA-variance) version, tested against 80 points of injected Gaussian noise (mean 100, std 5) plus one genuine spike at 300, flagged the real spike correctly but ALSO flagged 3 ordinary points (values 91.7, 89.9, 106.4, none more than 1.8 true standard deviations from the mean) as false anomalies purely because the EWMA-variance estimate had randomly dipped. Switching to a fixed 30-point rolling window for both the mean and standard deviation:

from collections import deque
import statistics

def sliding_window_anomaly(values, window_size=30, z_thresh=3.0, min_periods=20):
    window = deque(maxlen=window_size)
    out = []
    for v in values:
        flagged = False
        if len(window) >= min_periods:
            mu = statistics.fmean(window)
            sd = statistics.pstdev(window)
            if sd > 1e-9:
                flagged = abs(v - mu) / sd > z_thresh
        out.append((v, flagged))
        window.append(v)
    return out

flagged ONLY the genuine spike, with zero false positives on the same test data, and a false-positive rate of 0.35%-0.80% across five separate pure-noise seeds of 2,000 points each (close to the ~0.27% a true 3-sigma threshold implies on i.i.d. Gaussian data, with the small remaining excess coming from estimating standard deviation off a finite 30-point sample rather than the true population value).

Trade-offs and pitfalls

The window size trades detection speed for stability: a smaller window adapts faster to legitimate seasonality but re-introduces the same kind of variance instability seen in the EWMA version; a larger window is more stable but slower to adapt after a genuine, permanent level shift (e.g. after a deploy that legitimately changes baseline latency). min_periods matters just as much as the window size: evaluating z-scores before the window has enough points to estimate a meaningful standard deviation is what let the EWMA version misfire early in its own warm-up period.

The right approach to inheriting a low-maturity, incident-heavy system is to resist the urge to fix everything at once, and instead sequence a small number of high-leverage, fast wins that reduce the most painful and frequent failures first, while building the observability needed to make every later decision with data instead of guesswork. A credible 6 to 12 month roadmap has three phases: stop the bleeding, build the foundation, and invest in structural fixes, in that order.

Phase 1 (weeks 0 to 4): stop the bleeding

• Instrument the system enough to see what's actually failing (before this, teams are usually reacting to whichever incident is loudest, not the ones costing the most).
• Fix or work around the single most frequent or costly failure mode, even with a blunt, temporary mitigation (a manual restart script, a stricter rate limit), to buy breathing room for the rest of the plan.
• Stand up a minimal on-call rotation and the first one or two runbooks, for whatever is currently paging most.

Phase 2 (months 1 to 3): build the foundation

• Establish the first real reliability target (a service-level objective, or SLO) grounded in the data gathered in phase 1, not guessed at.
• Close the biggest gaps a production-readiness review would have caught (missing monitoring, no rollback path, no capacity margin).
• Introduce a lightweight blameless postmortem process so recurring root causes stop repeating silently.

Phase 3 (months 3 to 12): structural fixes

• Address the underlying architectural or process causes the earlier phases only worked around (a single point of failure, a fragile deploy process, a monolith that makes any change risky).
• Automate away the highest-volume manual, repetitive operational work identified along the way.
• Re-baseline the reliability target against the now-healthier system, and set the cadence (quarterly review, ongoing error-budget policy) that keeps the gains from eroding after the initial push ends.

Worked example

A team inherits a data platform with no defined service-level indicators (SLIs) and outages roughly weekly, each investigated from scratch because nothing is written down. Weeks 0 to 4: add basic pipeline-completion and freshness monitoring, discover 60% of incidents trace back to one upstream, unversioned schema change breaking downstream jobs, and put a schema-validation check in front of ingestion as an immediate mitigation. Months 1 to 3: define an SLO for data freshness per critical dataset, write runbooks for the two most common failure classes, and start a blameless postmortem for every incident above a defined severity. Months 3 to 12: replace the ad hoc schema-validation patch with a proper schema-registry and contract-testing setup, and automate the manual reprocessing steps that on-call had been doing by hand after every incident. Incident frequency drops from roughly weekly to roughly monthly by the end of the roadmap, tracked against the pipeline-completion monitoring put in place in phase 1, not asserted without a baseline to compare to.

Trade-offs and pitfalls

The most common failure is starting with the most architecturally satisfying fix (a full rewrite, a major re-platform) before the team has the monitoring to know whether it actually helped, or the credibility with stakeholders that comes from a visible early win. A second pitfall is declaring victory after phase 1's quick wins and never funding phase 3's structural work, which lets the same class of incident resurface once the temporary mitigations degrade or the system grows past what they were sized for. The roadmap's sequencing (visibility first, targets second, structural change third) matters more than any specific fix on it; a team that skips straight to structural change without phase 1's visibility is often solving the wrong problem first.

An SLI is only as trustworthy as its correlation with real user experience, so validating it means actively looking for evidence the metric could be biased by exactly HOW it's collected, not just trusting that a well-designed-sounding SLI must be accurate.

Structured elaboration

Validation plan: (1) synthetic tests, deliberately injecting a known failure or degradation and confirming the SLI actually moves as expected, in the expected direction and magnitude, a direct causal check that the instrumentation actually detects what it claims to; (2) correlating against RUM (real-user monitoring), checking that periods where the SLI shows good health also show healthy real-user-observed experience, and vice versa, since a persistent disagreement between the two is exactly the kind of bias signal worth investigating (e.g. a server-side SLI staying green while RUM shows real degradation, suggesting the SLI is measuring something too narrow, like server-side-only latency, missing a real client-side or network-level problem); (3) shadow traffic, routing a copy of real production traffic through an alternate measurement path and comparing its SLI reading against the primary path's, which can reveal instrumentation-specific bias (e.g. if the shadow path samples differently or measures at a different point in the request lifecycle and gets a meaningfully different number, that difference itself is diagnostic).

Worked example

Acceptance criteria: the synthetic-failure-injection test should show the SLI degrading by an expected, roughly-predictable amount for a known-severity injected failure (not necessarily exact, but in the right direction and rough magnitude); the RUM correlation should show no SUSTAINED divergence beyond an agreed tolerance over a representative period (occasional brief divergence is expected noise, a persistent gap is a validation failure); shadow-traffic comparison should show the two measurement paths agreeing within an agreed tolerance band, with any persistent, unexplained gap treated as an open validation defect requiring investigation before the SLI is trusted for a real SLO commitment.

Trade-offs and pitfalls

Sampling bias is a specific, easy-to-miss failure mode worth checking explicitly: if the SLI's underlying data collection systematically excludes a subset of real traffic (e.g. requests from users on unstable connections failing to successfully report their own telemetry, a pattern discussed elsewhere), the SLI will look healthier than reality specifically for the population it's failing to measure, and neither a synthetic test (which doesn't naturally exercise this bias) nor a simple RUM-correlation check (if RUM has the SAME sampling bias) would necessarily catch it; shadow traffic with a DELIBERATELY different sampling or measurement mechanism is specifically valuable here because it's less likely to share the exact same bias as the primary path. This validation shouldn't be a one-time exercise either: as the underlying system and its traffic patterns evolve, a periodic re-validation (not just a one-off at initial SLI design time) is what catches bias that develops later, rather than assuming a metric validated once stays valid forever.