Site Reliability Engineering Principles Questions

Prioritizing reliability engineering work against feature requests is not a one-time policy decision, it's a recurring negotiation that works best when it's anchored to an agreed, visible signal rather than relitigated from scratch every sprint. The strongest approach ties the trade-off to a shared reliability target (a service-level objective, or SLO, and the resulting error budget) so that whether reliability work takes priority becomes a mostly mechanical question once that target is trending badly, and a genuine judgment call the rest of the time.

A structure for the recurring decision

1. Anchor to a shared number. When the team has an agreed reliability target and the current state relative to it is visible to both engineering and product, "should we ship this feature or fix reliability" stops being a values argument and becomes "are we inside or outside our agreed margin."
2. Separate toil from real engineering investment. Toil (repetitive manual operational work) competes with both features and deeper reliability engineering; automating it away is rarely controversial and should usually win by default, since it pays back quickly and frees capacity for everything else.
3. Make the cost of deferral visible. Reliability work deferred long enough compounds (a known issue causes repeat incidents, each one costing far more engineering time than the original fix would have); surface that compounding cost in terms product stakeholders can weigh, not just "we should be doing this."
4. Reserve explicit capacity, don't just compete for it. Many teams that get this right allocate a standing percentage of each cycle to reliability and toil work up front, so it isn't re-litigated feature by feature.

Worked example (a concrete trade-off)

A team's checkout API is inside its reliability target, but a recurring, low-severity issue (an intermittent third-party payment-processor timeout) has caused three minor incidents in the past quarter, none severe enough to trigger the error-budget policy on their own. Meanwhile product wants a new checkout feature shipped this sprint. The negotiation: the team quantifies that the recurring timeout has already cost roughly two engineer-days of incident response this quarter and is trending worse, proposes shipping the feature this sprint as planned but committing the next sprint's reliability-work allocation specifically to that fix rather than leaving it in the general backlog, and gets product's buy-in by showing the incident-time cost directly rather than arguing reliability work is important in the abstract.

Trade-offs and pitfalls

Treating every reliability request as equally urgent erodes trust with product stakeholders, who correctly notice when "everything is a fire" and start discounting reliability asks across the board; the credibility of a hard stop (this must happen now) depends on it being genuinely rare. The opposite failure, always deferring reliability work in favor of the next feature, quietly accumulates risk until an incident forces the trade-off on the team's behalf, usually at a worse time and a higher cost than if it had been scheduled deliberately. The healthiest teams treat this as a standing, visible allocation decision rather than a fight that has to be re-won every time.

Static thresholds fail on traffic with real daily/weekly seasonality; adapting the threshold from the data itself via a smoothed baseline and a spread estimate is the standard fix, but the naive single-pass version of this hides a real accuracy bug.

Structured elaboration

A sliding-window z-score approach scores each new point against the mean and standard deviation of the last N points, flags it as anomalous if the deviation exceeds a chosen multiple of sigma, then adds it to the window. Testing revealed that an EWMA-based variance estimator (a common textbook alternative that avoids keeping a window in memory) has a real flaw here: with a smoothing factor high enough to track real level changes responsively, its effective sample size for the VARIANCE estimate is far too small (roughly $1/\alpha$ points), so the estimated standard deviation itself swings noisily, and whenever it happens to dip low, an entirely ordinary point reads as a spurious multi-sigma anomaly.

Worked example (executed; python3, 5+ seeds, before/after comparison)

The first (EWMA-variance) version, tested against 80 points of injected Gaussian noise (mean 100, std 5) plus one genuine spike at 300, flagged the real spike correctly but ALSO flagged 3 ordinary points (values 91.7, 89.9, 106.4, none more than 1.8 true standard deviations from the mean) as false anomalies purely because the EWMA-variance estimate had randomly dipped. Switching to a fixed 30-point rolling window for both the mean and standard deviation:

from collections import deque
import statistics

def sliding_window_anomaly(values, window_size=30, z_thresh=3.0, min_periods=20):
    window = deque(maxlen=window_size)
    out = []
    for v in values:
        flagged = False
        if len(window) >= min_periods:
            mu = statistics.fmean(window)
            sd = statistics.pstdev(window)
            if sd > 1e-9:
                flagged = abs(v - mu) / sd > z_thresh
        out.append((v, flagged))
        window.append(v)
    return out

flagged ONLY the genuine spike, with zero false positives on the same test data, and a false-positive rate of 0.35%-0.80% across five separate pure-noise seeds of 2,000 points each (close to the ~0.27% a true 3-sigma threshold implies on i.i.d. Gaussian data, with the small remaining excess coming from estimating standard deviation off a finite 30-point sample rather than the true population value).

Trade-offs and pitfalls

The window size trades detection speed for stability: a smaller window adapts faster to legitimate seasonality but re-introduces the same kind of variance instability seen in the EWMA version; a larger window is more stable but slower to adapt after a genuine, permanent level shift (e.g. after a deploy that legitimately changes baseline latency). min_periods matters just as much as the window size: evaluating z-scores before the window has enough points to estimate a meaningful standard deviation is what let the EWMA version misfire early in its own warm-up period.

An SLA negotiation template needs to name every component a real dispute would eventually turn on, so it functions as a genuine starting point for negotiation rather than a document that looks complete but has gaps a sophisticated enterprise counterparty will immediately probe.

Structured elaboration

Measurable components: availability, latency (with explicit percentiles, e.g. p95 and p99, not just an average), and throughput, each defined precisely enough that both sides agree in advance what "meeting the target" means. Measurement windows: state both the evaluation period (typically monthly) and, separately, any shorter windows used for real-time internal alerting versus the contractual reporting period, since these can legitimately differ. Exclusions and blackout windows: scheduled maintenance (with a defined maximum monthly/quarterly allowance and advance-notice requirement) and force-majeure categories, named specifically rather than left vague. Monitoring sources for disputes: state whose data is authoritative (provider logs, a named third-party monitor, or a reconciliation process between both parties' data) to avoid the classic "our numbers don't match yours" stalemate. Remedy/credit structure: tiered service credits scaled to severity, with a stated cap. Dispute resolution: a defined escalation path (technical review, then a management-level conversation, then, if unresolved, a named arbitration or mediation process) before any resort to litigation. Alignment to internal SLOs and capacity planning: internally, confirm the negotiated SLA sits comfortably inside the existing internal SLO's safety margin before signing, since committing externally to a number your internal SLO can't sustainably support creates immediate structural risk.

Worked example

A template clause: "Availability shall be measured as [defined ratio] over each calendar month, using Provider's production monitoring system, cross-checked quarterly against [named third-party monitor]. Scheduled maintenance, not exceeding 4 hours per quarter with 72 hours' advance notice, is excluded. Should monthly availability fall below 99.9%, Customer is entitled to a service credit of 10% of that month's fees; below 99.5%, 25%; three consecutive months below 99.9% entitles Customer to terminate for cause without penalty. Disputes regarding measured availability shall first be reviewed jointly by both parties' technical teams within 10 business days, escalating to management review if unresolved, and to [named] mediation before any litigation."

Trade-offs and pitfalls

Negotiating a tighter SLA than your internal SLO can sustainably support, purely to win the deal, creates a structural mismatch that surfaces painfully later, either as chronic credit payouts or as constant internal pressure to hit a number the architecture wasn't built for; the internal-alignment check should happen BEFORE the negotiation, not be discovered as a problem after signing. It's equally risky to leave the measurement-source question vague ("availability shall be measured appropriately"), since that vagueness is exactly what a real dispute exploits; naming the authoritative source and a reconciliation process up front removes the single most common point of later conflict.

Direct answer

Cascading failures across a fleet of DAGs happen when a cross-DAG dependency, an external-task sensor, a dataset trigger, or an informal "wait for the other DAG" convention, blocks on an upstream DAG that has failed, with nothing in the downstream DAG's own configuration able to tell the difference between "the upstream just has not run yet" and "the upstream failed and will never succeed today." Detect and prevent this with a mix of failure isolation designed in up front (preventative) and fast blast-radius containment once a failure has actually started (reactive).

Structured elaboration

Preventative measures.

• Bound every cross-DAG wait with an explicit timeout, not an unbounded poll, so a downstream DAG blocked on a permanently failed upstream eventually fails visibly instead of silently occupying resources forever.
• Design cross-DAG dependencies to react to failure, not only success: check the upstream DAG's actual terminal state directly (failed versus still running versus succeeded) rather than only waiting for a success signal, so a downstream DAG can fail fast and alert the moment its upstream failed, instead of only discovering the problem after riding out a full timeout window.
• Avoid deep dependency chains where many DAGs transitively depend on one hub DAG for reasons that do not actually require strict ordering (a shared "wait for the daily ingestion to finish" convention used far beyond what genuinely needs it); a failure in the hub then cascades to everything downstream of it, including DAGs that never strictly needed to wait.
• Use resource pools or queues per DAG family so a failing or retrying DAG's stuck tasks cannot starve worker capacity that unrelated DAGs need; this is itself a mechanism by which one DAG's failure becomes another's, through resource contention rather than an explicit dependency edge.
• A circuit-breaker pattern: if an upstream DAG has failed a set number of times in a row, automatically pause the DAGs that depend on it rather than letting each keep attempting and timing out on its own schedule, converting a slow, spread-out cascade into one clear, actionable signal.

Reactive measures.

• A dashboard or alert that shows dependency-graph blast radius the moment a DAG fails, which other DAGs are currently blocked on it, not just that this one DAG failed, so the on-call engineer understands scope immediately instead of discovering it DAG by DAG as pages trickle in over the following hours.
• A documented runbook action to pause the fleet's dependent DAGs, whether triggered manually or by the circuit breaker above, to stop the bleeding immediately while the root cause is investigated.
• Root-cause the specific upstream failure, fix it, then resume the downstream DAGs in their own correct dependency order rather than all at once; an un-paused fleet resuming simultaneously can recreate the same resource-contention cascade at the exact moment of recovery.

Worked example

A fleet of 40 DAGs includes 12 that wait on daily_ingestion through an external-task-style sensor. daily_ingestion fails at 02:15. Each of the 12 downstream sensors is configured with a 4-hour timeout and a 5-minute poll interval, in reschedule mode, and only checks for success, not failure.

Without a fail-fast check, all 12 keep polling for the full window and finally time out and fail at 06:15, four hours after the root cause was already known, and the on-call engineer, who is only paged when a DAG itself fails, does not learn about any of the 12 downstream failures until that same 06:15.

Adding a fail-fast check, the sensor also examines whether daily_ingestion's latest run state is failed, not only success, changes this: all 12 downstream DAGs fail immediately at 02:16, one poll interval after the upstream failure. Detection-to-alert latency drops from

$06{:}15 - 02{:}15 = 240 \text{ minutes} \quad \text{to} \quad 02{:}16 - 02{:}15 = 1 \text{ minute}$

a roughly 240 times improvement, and the on-call engineer sees the full 12-DAG blast radius in a single page instead of a slow trickle spread across four hours.

Trade-offs and pitfalls

An automatic circuit breaker tuned to pause downstream DAGs on a single upstream failure, rather than a set number in a row, can cause unnecessary disruption for a one-off transient blip that would have self-healed on the upstream's own retry; the "how many failures in a row" threshold should be tuned against the upstream's own observed transient-failure rate, not set to the most cautious possible value by default.

A dependency-graph blast-radius view is only as good as the dependency edges it actually knows about. An undeclared, convention-based dependency, two DAGs that happen to need to run in a particular order with no explicit sensor or trigger connecting them, is invisible to this kind of tooling and needs to be made explicit before it can be protected against at all.

A 99.9% freshness SLA on derived dashboard metrics needs to be decomposed into SLIs that each capture a different stage where freshness could be lost, since a single blended "is it fresh" number hides WHERE in the pipeline a delay is actually occurring.

Structured elaboration

Four SLIs: (1) source-to-ingestion lag - time from the source event occurring to it landing in the raw ingestion layer; (2) ingestion-to-transform lag - time from raw ingestion to the derived-metric computation completing; (3) end-to-end freshness - the combined total, source event to dashboard-visible metric, which is the number that actually maps to the 99.9% SLA commitment; (4) computation success rate - proportion of scheduled metric-computation runs that complete successfully at all, since a freshness number computed only from SUCCESSFUL runs silently ignores runs that failed entirely, which is itself a freshness (and correctness) problem.

Worked example

SLO: 99.9% of end-to-end freshness measurements land under a defined threshold (e.g. under 10 minutes) over a 30-day rolling window, with the three component SLIs (source-to-ingestion, ingestion-to-transform, computation success rate) tracked as diagnostic breakdowns rather than separately-committed SLAs. If the end-to-end SLI degrades, the three component SLIs let you immediately localize whether the delay is happening at ingestion (a source-system problem) or at transform (a compute-pipeline problem), without which you'd only know "it's slow" with no actionable next step.

Trade-offs and pitfalls

Reporting only the end-to-end number without the component breakdown is fine for the customer-facing SLA report but nearly useless for diagnosing an actual regression internally; both views need to exist, aimed at different audiences. It's also worth being careful about how the computation-success-rate SLI interacts with the freshness SLI: a completely FAILED computation run has no freshness reading at all (there's no metric to measure the lag of), so it must be explicitly counted against the SLO as a worst-case freshness violation, not silently excluded from the freshness average simply because it produced no valid data point to measure.