Site Reliability Engineering Principles Questions

Static thresholds fail on traffic with real daily/weekly seasonality; adapting the threshold from the data itself via a smoothed baseline and a spread estimate is the standard fix, but the naive single-pass version of this hides a real accuracy bug.

Structured elaboration

A sliding-window z-score approach scores each new point against the mean and standard deviation of the last N points, flags it as anomalous if the deviation exceeds a chosen multiple of sigma, then adds it to the window. Testing revealed that an EWMA-based variance estimator (a common textbook alternative that avoids keeping a window in memory) has a real flaw here: with a smoothing factor high enough to track real level changes responsively, its effective sample size for the VARIANCE estimate is far too small (roughly $1/\alpha$ points), so the estimated standard deviation itself swings noisily, and whenever it happens to dip low, an entirely ordinary point reads as a spurious multi-sigma anomaly.

Worked example (executed; python3, 5+ seeds, before/after comparison)

The first (EWMA-variance) version, tested against 80 points of injected Gaussian noise (mean 100, std 5) plus one genuine spike at 300, flagged the real spike correctly but ALSO flagged 3 ordinary points (values 91.7, 89.9, 106.4, none more than 1.8 true standard deviations from the mean) as false anomalies purely because the EWMA-variance estimate had randomly dipped. Switching to a fixed 30-point rolling window for both the mean and standard deviation:

from collections import deque
import statistics

def sliding_window_anomaly(values, window_size=30, z_thresh=3.0, min_periods=20):
    window = deque(maxlen=window_size)
    out = []
    for v in values:
        flagged = False
        if len(window) >= min_periods:
            mu = statistics.fmean(window)
            sd = statistics.pstdev(window)
            if sd > 1e-9:
                flagged = abs(v - mu) / sd > z_thresh
        out.append((v, flagged))
        window.append(v)
    return out

flagged ONLY the genuine spike, with zero false positives on the same test data, and a false-positive rate of 0.35%-0.80% across five separate pure-noise seeds of 2,000 points each (close to the ~0.27% a true 3-sigma threshold implies on i.i.d. Gaussian data, with the small remaining excess coming from estimating standard deviation off a finite 30-point sample rather than the true population value).

Trade-offs and pitfalls

The window size trades detection speed for stability: a smaller window adapts faster to legitimate seasonality but re-introduces the same kind of variance instability seen in the EWMA version; a larger window is more stable but slower to adapt after a genuine, permanent level shift (e.g. after a deploy that legitimately changes baseline latency). min_periods matters just as much as the window size: evaluating z-scores before the window has enough points to estimate a meaningful standard deviation is what let the EWMA version misfire early in its own warm-up period.

An automated burn-rate mitigation system needs tiered detection windows (fast for severe spikes, slower to confirm sustained burns) each mapped to a specific, increasingly aggressive automated action, not a single on/off trigger.

Structured elaboration

A layered design: a SHORT window (e.g. 5 minutes) at a HIGH burn-rate threshold (e.g. 10x) catches a sudden severe outage quickly and can trigger the most conservative automated response (temporary feature rollback of the most recent risky change) with minimal delay; a LONGER window (e.g. 6 hours) at a LOWER threshold (e.g. 2x) confirms a slow, sustained degradation that a short window alone would miss, and triggers a less drastic response (throttle non-critical feature flags, or scale up capacity if the degradation looks saturation-related rather than error-related). Each preemption level maps to a specific, reversible action: level 1 (early warning) notifies on-call with no automated change; level 2 (moderate, sustained burn) throttles specific feature flags identified as recently changed; level 3 (severe or fast burn) automatically rolls back the most recent deploy and pages.

Worked example

At a 99.9% SLO (30-day error budget of 43.2 minutes): level 1 fires at burn rate > 2x sustained over 6 hours (notify only), which by the time it fires has consumed about 2*(6/720) = 1.67% of the monthly budget; level 2 fires at burn rate > 6x sustained over 1 hour (auto-throttle the most recently enabled feature flags), consuming about 6*(1/720) = 0.83% of the monthly budget by the time it fires; level 3 fires at burn rate > 14x sustained over 5 minutes (auto-rollback the last deploy and page), consuming only about 14*(5/60/720) = 0.16% of the monthly budget by the time it fires. This is the standard multi-window multi-burn-rate pattern from the Google SRE workbook's chapter on alerting, but the design goal is NOT that every level consumes the same fraction of budget; they don't (1.67%, 0.83%, and 0.16% here). The actual goal is the opposite: the fastest, highest-severity alert (short window, high multiplier) is tuned to fire having consumed as LITTLE budget as possible, because it exists to catch a severe spike before much damage is done, while a slower, lower-multiplier alert is allowed to consume more budget before firing, because distinguishing a genuine sustained low-grade burn from ordinary noise requires a longer observation window, and firing too early on a short window at a low multiplier would false-positive constantly.

Trade-offs and pitfalls

Every automated mitigation (feature rollback, auto-scaling, throttling) needs a safety valve: an automatic rollback that itself has a bug, or that fires on a false-positive burn signal, can cause more disruption than the incident it was meant to contain, so each level should have a manual override and a rate limit on how often it can fire automatically before requiring human confirmation. It's also worth distinguishing SATURATION-caused burn (where auto-scaling is the right response) from CODE-defect-caused burn (where rollback is the right response); triggering the wrong automated action for the wrong root cause can make an incident worse, so the system needs at least a rough signal (e.g. correlating burn with a recent saturation metric versus a recent deploy event) to pick the right lever.

Direct answer

Cascading failures across a fleet of DAGs happen when a cross-DAG dependency, an external-task sensor, a dataset trigger, or an informal "wait for the other DAG" convention, blocks on an upstream DAG that has failed, with nothing in the downstream DAG's own configuration able to tell the difference between "the upstream just has not run yet" and "the upstream failed and will never succeed today." Detect and prevent this with a mix of failure isolation designed in up front (preventative) and fast blast-radius containment once a failure has actually started (reactive).

Structured elaboration

Preventative measures.

• Bound every cross-DAG wait with an explicit timeout, not an unbounded poll, so a downstream DAG blocked on a permanently failed upstream eventually fails visibly instead of silently occupying resources forever.
• Design cross-DAG dependencies to react to failure, not only success: check the upstream DAG's actual terminal state directly (failed versus still running versus succeeded) rather than only waiting for a success signal, so a downstream DAG can fail fast and alert the moment its upstream failed, instead of only discovering the problem after riding out a full timeout window.
• Avoid deep dependency chains where many DAGs transitively depend on one hub DAG for reasons that do not actually require strict ordering (a shared "wait for the daily ingestion to finish" convention used far beyond what genuinely needs it); a failure in the hub then cascades to everything downstream of it, including DAGs that never strictly needed to wait.
• Use resource pools or queues per DAG family so a failing or retrying DAG's stuck tasks cannot starve worker capacity that unrelated DAGs need; this is itself a mechanism by which one DAG's failure becomes another's, through resource contention rather than an explicit dependency edge.
• A circuit-breaker pattern: if an upstream DAG has failed a set number of times in a row, automatically pause the DAGs that depend on it rather than letting each keep attempting and timing out on its own schedule, converting a slow, spread-out cascade into one clear, actionable signal.

Reactive measures.

• A dashboard or alert that shows dependency-graph blast radius the moment a DAG fails, which other DAGs are currently blocked on it, not just that this one DAG failed, so the on-call engineer understands scope immediately instead of discovering it DAG by DAG as pages trickle in over the following hours.
• A documented runbook action to pause the fleet's dependent DAGs, whether triggered manually or by the circuit breaker above, to stop the bleeding immediately while the root cause is investigated.
• Root-cause the specific upstream failure, fix it, then resume the downstream DAGs in their own correct dependency order rather than all at once; an un-paused fleet resuming simultaneously can recreate the same resource-contention cascade at the exact moment of recovery.

Worked example

A fleet of 40 DAGs includes 12 that wait on daily_ingestion through an external-task-style sensor. daily_ingestion fails at 02:15. Each of the 12 downstream sensors is configured with a 4-hour timeout and a 5-minute poll interval, in reschedule mode, and only checks for success, not failure.

Without a fail-fast check, all 12 keep polling for the full window and finally time out and fail at 06:15, four hours after the root cause was already known, and the on-call engineer, who is only paged when a DAG itself fails, does not learn about any of the 12 downstream failures until that same 06:15.

Adding a fail-fast check, the sensor also examines whether daily_ingestion's latest run state is failed, not only success, changes this: all 12 downstream DAGs fail immediately at 02:16, one poll interval after the upstream failure. Detection-to-alert latency drops from

$06{:}15 - 02{:}15 = 240 \text{ minutes} \quad \text{to} \quad 02{:}16 - 02{:}15 = 1 \text{ minute}$

a roughly 240 times improvement, and the on-call engineer sees the full 12-DAG blast radius in a single page instead of a slow trickle spread across four hours.

Trade-offs and pitfalls

An automatic circuit breaker tuned to pause downstream DAGs on a single upstream failure, rather than a set number in a row, can cause unnecessary disruption for a one-off transient blip that would have self-healed on the upstream's own retry; the "how many failures in a row" threshold should be tuned against the upstream's own observed transient-failure rate, not set to the most cautious possible value by default.

A dependency-graph blast-radius view is only as good as the dependency edges it actually knows about. An undeclared, convention-based dependency, two DAGs that happen to need to run in a particular order with no explicit sensor or trigger connecting them, is invisible to this kind of tooling and needs to be made explicit before it can be protected against at all.

A per-service burn-rate report needs the allowed-errors figure computed from each service's OWN request volume, not a single fleet-wide number, since services with different traffic naturally have different absolute error budgets even at the same SLO percentage.

Structured elaboration

For each service, total requests and observed errors are aggregated over the 30-day window, and the allowed-errors figure is simply the SLO's allowed-bad-fraction multiplied by that service's own total request count; burn rate is then observed errors divided by allowed errors, giving a comparable, unitless number across services regardless of how much traffic each one handles.

Worked example (executed; sqlite3, two services with different traffic and error counts)

WITH agg AS (
  SELECT service_name,
         COUNT(*) AS total_requests,
         SUM(CASE WHEN success = 0 THEN 1 ELSE 0 END) AS observed_errors
  FROM service_metrics
  WHERE ts >= date('2026-07-01') AND ts < date('2026-07-01', '+30 day')
  GROUP BY service_name
)
SELECT service_name, total_requests, observed_errors,
       total_requests * 0.0005 AS allowed_errors,
       ROUND(observed_errors * 1.0 / NULLIF(total_requests * 0.0005, 0), 2) AS burn_rate
FROM agg;

Against a synthetic dataset with checkout (2 requests, 0 errors) and images (4 requests, 1 error) at a 99.95% SLO: checkout correctly returned a burn rate of 0.0, and images returned 500.0 (1 observed error against an allowed-errors figure of 4 * 0.0005 = 0.002, meaning even a single error on a very small sample massively exceeds the allowed rate), correctly illustrating why burn rate on a low-traffic service needs a minimum sample-size guard before it's trusted.

Trade-offs and pitfalls

The NULLIF guard prevents a divide-by-zero for a service with zero requests in the window, but it does not solve the more subtle problem the images result illustrates: at very small request counts, a tiny absolute number of errors produces an enormous, statistically unstable burn-rate multiple that looks alarming but reflects noise more than a real trend; a production version should suppress or flag burn-rate values for services below a minimum request-count threshold rather than reporting them at face value.

A centralized SRE model pools reliability engineers into one team that serves many product groups, which maximizes consistency and lets scarce reliability expertise scale across the company; an embedded model places SRE engineers inside each product team, which maximizes context and speed of iteration but risks each team reinventing its own standards and tooling. For a large, many-product organization, a hybrid usually wins: a small central platform team owns the shared reliability substrate (tooling, SLO framework, incident process, training) while embedded or team-aligned SREs apply it to their specific product's context, reporting a dotted line back to the central team for standards and career growth.

Comparing the two pure models

Designing the hybrid

1. Central platform layer: owns the service-level objective (SLO) framework and tooling, the incident-command process, the production-readiness bar, and a small set of non-negotiable standards (what "on-call ready" means).
2. Team-aligned reliability engineers: embedded in or paired with each product group, applying the shared framework to that team's actual service, running that team's on-call rotation, and feeding real-world friction back to the central team.
3. Governance loop: a lightweight review (quarterly or per-major-launch) where the central team audits embedded practice against the shared standard, and the embedded engineers propose changes to the standard itself.

How the structure evolves with scale

At small scale (a handful of services), a single centralized team is usually enough: there isn't enough volume to justify per-team specialists, and consistency is cheap to maintain by hand. As the organization grows into dozens or hundreds of engineers across many product lines, the central team's queue becomes the bottleneck, and the case for embedding grows; the central team's job shifts from doing reliability work directly to building the tooling, training, and standards that let embedded engineers do it well without central review on every decision.

Trade-offs and pitfalls

The most common failure of the hybrid is letting the "central standard" calcify into bureaucracy that embedded teams route around, which quietly reverts the organization to a fragmented model with extra process on top. A second pitfall is treating a strong service-ownership culture, where individual teams already feel deep accountability for what they ship, as a substitute for any shared reliability structure at all; without a common vocabulary and toolset, cross-service incidents (a failure in one team's dependency causing another's outage) have no shared playbook and take longer to resolve than they should. Note that the specific mechanics being standardized (SLOs, error budgets, runbook format) are set by dedicated practices elsewhere; the organizational question here is who owns applying them and at what scale, not how those mechanics themselves work.