On-Call Practices and Runbook Design Questions

A runbook being written isn't the same as it being trustworthy under pressure: writing tests whether the author understood the system, while readiness tests whether someone else, half-awake at 3am, can follow it and get the right outcome. Check readiness by having someone who didn't write it actually execute it against a real (or realistic) system, not by reading it for completeness.

What to check before trusting a runbook

• Has anyone other than the author run it? A runbook the author has never handed to someone else is unverified by definition; the author's own mental model fills gaps a stranger will trip on (an assumed tool is installed, an assumed permission is already granted, a step that says "check the dashboard" without saying which one).
• Are the steps executable as written, not just described? "Restart the service" is a description; "run systemctl restart payments-api on each of the 3 hosts listed in the service registry" is executable. If a step requires judgment the runbook doesn't supply (how do you know which hosts?), that's a gap, not an acceptable level of abstraction.
• Does it state what success looks like? A remediation step without a stated verification step (what metric or log line confirms this worked) leaves the responder guessing whether to move to the next step or escalate.
• Is it safe to run when the diagnosis is wrong? Incident responders under pressure sometimes run the wrong runbook, or run the right one when the actual cause differs from what it assumes. Check whether each destructive step is reversible, and whether the runbook states a precondition to verify before acting ("only run this if X").
• Is it current? Check for an owner and a last-verified date; a runbook referencing a deprecated tool, an old cluster name, or a rotation that no longer exists is worse than no runbook, because it costs time before the responder realizes it's wrong.

How to actually verify these, not just check for their presence

1. Tabletop walkthrough: someone unfamiliar with the runbook reads it aloud, step by step, without help from the author, narrating what they'd actually type or click. Gaps surface immediately as "wait, what do I do here?" moments.
2. Staging or canary drill: run the actual remediation against a staging environment or a single canary instance, and time it. This catches steps that look right on paper but fail against the real system (a command with an outdated flag, a permission the on-call role doesn't actually have).
3. Cold-open test: hand it to someone with zero context on this specific service (not zero context on the systems generally) and see if they can act on it in under a target time, without pinging the original author. If they can't, the runbook is only usable by the person who wrote it, which defeats the point.

Worked example

A runbook for "database replica lag alert" says: "Check replica lag, if high, failover to standby." A cold-open drill immediately exposes three gaps: no link to where replica lag is displayed, no threshold for what counts as "high" (the alert already fired, so this should already be answered, but the runbook re-asks the question), and "failover to standby" doesn't say which standby if there are multiple, or what to verify afterward to confirm the failover succeeded rather than made things worse. Fixing it: link the specific dashboard panel, state the alert's own threshold so the runbook doesn't require re-deciding it, name the failover command with the specific standby-selection logic, and add a verification step ("confirm write latency on the new primary is under 50ms and replica lag on remaining replicas is decreasing"). Re-running the cold-open drill after the fix, the same tester completes it without asking a clarifying question, which is the actual pass condition.

Trade-offs and pitfalls

Running live drills has a real cost in engineering time and, for staging drills, some risk if the environment isn't well isolated from production; the return is worth it for any runbook covering a high-severity or destructive action, and can be scaled down to tabletop-only for low-risk, easily reversible ones. A common wrong turn is treating runbook review as a documentation-quality pass (is it well written, does it have headers) rather than an execution test; a beautifully formatted runbook that's never been run by anyone but its author is still unverified.

A production-readiness review should verify four things before a service starts taking pages: it fails safely (degrades or rolls back instead of cascading), it's observable enough that on-call can diagnose without guessing, on-call actually knows how to respond to it, and someone specific owns it. Structure the review around those four, not a flat checklist, so gaps are obvious by category rather than buried in a long list.

What the review checks, by category

Process for running the review

1. The owning team self-certifies against the checklist first, providing evidence (load-test results, a link to the rollback runbook, a screenshot of the dashboard) rather than a checked box with no backing.
2. A reviewer outside the owning team (SRE or a peer team) spot-checks the evidence, focusing on the failure-containment and observability rows, since those are the ones teams under launch pressure are most likely to overstate.
3. Run one live-fire test before go-live: trigger the most likely failure mode in staging (or a controlled prod canary) and confirm the alert fires, the runbook's diagnosis steps actually find the cause, and the rollback works. A checklist that's never been exercised is a hypothesis, not a verified readiness state.
4. Sign-off is explicit and time-bound, not a one-time gate that's forgotten: re-review triggers on major architecture changes, not just at initial launch.

flowchart TD
  A[Owning team self-certifies checklist] --> B[Provide evidence: load tests, runbook links, dashboards]
  B --> C[Outside reviewer spot-checks evidence]
  C --> D{Gaps found?}
  D -->|Yes| E[Team remediates gap]
  E --> C
  D -->|No| F[Live-fire test in staging or canary]
  F --> G{Alert fires, runbook works, rollback succeeds?}
  G -->|No| E
  G -->|Yes| H[Sign-off: service takes pages]

Worked example

A new recommendations service is going live. Self-certification claims load testing was done "at expected traffic." The outside reviewer asks for the actual load-test report and finds it tested at the current expected peak (500 req/s) with no margin, while the service also sits behind a feature flag that product plans to ramp to three times that within a month. That's a real gap: the review isn't asking for perfection, but it should require either testing at the higher number now or an explicit, documented plan (with an owner and date) to re-test before the ramp, rather than letting "tested at expected traffic" silently mean "tested at today's traffic." The live-fire test then finds the circuit breaker on the downstream recommendation-model call has no timeout configured, so a slow model response would hang the request instead of failing fast; that's flagged as a blocking issue, not a follow-up ticket, because it directly causes cascading failure under exactly the load condition the service is meant to handle.

Trade-offs and pitfalls

The live-fire test step is the one teams most often skip under launch deadline pressure, and it's also the one that catches the gaps self-certification checklists miss (an alert that's configured but never actually fires, a runbook step that references a dashboard that doesn't exist); treat it as non-negotiable for anything customer-facing, and reserve a lighter self-certification-only path for low-risk internal services. A common wrong turn is treating the checklist as complete once every box is checked, without weighting which gaps are load-bearing; a missing rollback plan for a payments-adjacent service is not the same severity as a missing dashboard for an internal admin tool, and the review should say so explicitly rather than gating everything equally.

Direct answer

Guardrails come from three layers working together: classify every automatable action by risk, reversibility and blast radius, so low-risk actions can run unattended while destructive ones require signed, multi-party approval; rate-limit and circuit-break execution so a misfiring trigger can't repeat itself into a bigger outage; and make every execution auditable and traceable to a specific signed, version-pinned runbook so a compromised trigger's blast radius stays bounded even if it does fire.

Structured elaboration

Threat model

• Misfire: a legitimate alert misclassifies severity or triggers the wrong action, such as restarting the wrong service.
• Compromised automation pipeline: an attacker forges or replays an alert to trigger a destructive action.
• Insider abuse: someone with legitimate access triggers an action outside its intended use.

Each needs a different control. Misfire needs validation and rate limits. Compromise needs signing and short-lived credentials. Insider abuse needs approval gates and an audit log that can't be edited after the fact.

Guardrail decision flow

flowchart TD
  A[Alert triggers automated action] --> B{Reversible and low blast radius?}
  B -->|Yes, low risk| C{Under rate-limit threshold?}
  C -->|Yes| D[Execute in sandboxed, least-privilege runner]
  C -->|No| E[Circuit-break: block further auto-actions]
  B -->|No, high risk or destructive| F[Require signed approval from two operators]
  F --> G{Approved within TTL?}
  G -->|Yes| D
  G -->|No| H[Escalate to human on-call, no auto-execution]
  D --> I[Write signed, tamper-evident audit entry]

Core controls

• Risk classification: every automatable action is tagged low, medium, or high at authoring time, reviewed like code, based on reversibility and blast radius.
• Rate limiting and circuit breaking: cap executions per action per time window, and auto-disable an action after repeated failures instead of letting it keep firing.
• Signing and provenance: only signed, version-pinned runbook releases execute in production; the executor verifies the signature first, so a compromised trigger can request an action but can't smuggle in unreviewed logic.
• Least privilege and short-lived credentials: the executor pulls scoped, time-limited credentials per execution rather than holding standing broad access.
• Immutable audit trail: every execution, who or what triggered it, the parameters, and the outcome, is written to an append-only log kept separate from the systems it can act on, so it survives a compromise of the executor itself.

Worked example

Consider an attacker who can forge an alert payload. Against the pod-restart action, they're bounded by the rate limit and the action's inherently small blast radius. Against the index-drop action, they're blocked at the approval gate no matter how convincing the forged alert looks, because approval requires a human signature that an alert payload can't fake.

Trade-offs and pitfalls

• Classifying everything as needing approval defeats the point of automation, which is unattended response to the common case. Keep destructive steps as a separate, explicitly gated action rather than bundling them with routine remediation, so most actions land in the low-risk tier by design.
• More approval gates reduce blast radius but increase mean time to remediate; mitigate by keeping the low-risk tier wide and reserving gates for the genuinely destructive minority.
• Signing the runbook but not validating its parameters leaves a gap: a signed, but freely parameterizable, action like "restart <service>" can still be misused if the parameter itself isn't checked against an allowlist.

Direct answer

"Blameless" means the postmortem analyzes the incident as a failure of systems and processes that made a reasonable person's normal action produce a bad outcome, not as a failure of that person's competence or effort. It matters because the moment a postmortem starts assigning individual fault, people stop giving you the honest details (what they clicked, what they assumed, what they skipped) that you actually need to fix the underlying gap, and near-misses stop getting reported at all.

Structured elaboration

What blameless does not mean. It is not "no accountability." Individuals still own action items and are still expected to do their jobs well. Blameless means the analysis stops at "why did this look like the right thing to do at the time, given the information and tools available" instead of stopping at "who made the mistake."

Essential components of a postmortem document:

• Summary and impact: severity, duration, which customers/systems were affected, business impact.
• Timeline: timestamped sequence from first signal to full resolution, including detection and every mitigation attempt.
• Contributing factors: plural, not a single root cause. A "five whys" style chain that includes technical gaps (missing test, no lock-duration check) and process gaps (review checklist didn't require it, no staging environment with prod-sized data).
• What went well / what didn't: honest assessment of detection speed, mitigation effectiveness, and communication, separate from the technical cause.
• Action items: each with an owner, a due date, and a verification step, tracked to closure rather than left as a paragraph nobody revisits.
• Lessons learned: shared broadly enough that other teams with the same pattern can act on it before they hit the same incident.

Worked example

A deploy runs a database migration that locks a hot table for several minutes, causing a customer-facing outage. A blameful writeup says: "Engineer X pushed a migration without checking lock duration." A blameless writeup for the same incident says: "The migration tool doesn't warn about lock duration before merge, and the review checklist doesn't require a dry run against a prod-sized dataset. Both are now action items: add a lock-duration check to the migration tool (owner: platform team, verify by testing against a 10M-row table), and add a prod-sized dry-run step to the migration checklist (owner: DBA lead, verify by auditing the next five migrations)." The second version identifies exactly the same failure but produces two concrete, ownable fixes instead of a warning to be more careful next time, which is not something that reliably prevents a repeat.

Trade-offs and pitfalls

• Pitfall: blameless in the document but not in the room. Teams sometimes write a passive-voice, name-free doc while the actual retro meeting is full of pointed questions at one person. The doc's tone has to match the meeting's tone, or the culture stays blameful regardless of what's written down.
• Pitfall: treating blameless as zero consequences ever. Repeated, egregious negligence (ignoring a known policy, skipping a required review on purpose) is a people-management conversation, but it happens outside the technical postmortem, not inside it.
• Senior signal: distinguishing proximate cause from contributing factors. A junior answer stops at "the migration locked the table." A senior answer keeps asking why the tooling, the review process, and the testing environment all failed to catch it, because single-root-cause thinking tends to produce a single, shallow fix that doesn't survive the next incident with a different trigger but the same underlying gap.

The first few minutes after a page are for narrowing down scope and cause fast enough to act, not for finding the root cause. I validate the alert is real, figure out how big the blast radius is, check whether it lines up with the deploy that just went out, and pick a mitigation I can undo, escalating the moment the clock runs out on any of those steps.

The first-minutes checklist

1. Validate (0-2 min): confirm the page against the actual dashboard, not just the alert text. Is p95 latency and error rate really elevated right now, or is this a flapping alert that already recovered?
2. Determine scope (2-5 min): is this one service, one region, or several services at once? Check whether other, unrelated-looking alerts fired in the same window, they're often the same root cause fanning out, not five separate problems.
3. Correlate with the deploy (2-5 min): the timing (a spike a few minutes after a deploy) is a strong signal but not proof; pull the diff (code, config, feature flags, schema changes) to see if anything plausibly explains a latency or error increase.
4. Choose a reversible mitigation (5-15 min): rollback if the deploy is the clear cause and rollback is safe and fast; scale out or shed load if the cause looks like saturation; flip a feature flag if the change is flag-gated. Prefer whatever gets users out of pain fastest and can be undone if it turns out to be wrong.
5. Escalate on the clock, not on ego (by 15 min): if scope, cause, or mitigation isn't clear by 15 minutes, escalate, that's a threshold, not a judgment call to second-guess in the moment.

Decision flow

flowchart TD
  A[Page: p95 latency + error rate up, deploy went out minutes ago] --> B[0-2m: Validate against real dashboards]
  B --> C[2-5m: Determine scope: one service or several]
  C --> D{Timing matches the deploy?}
  D -->|Yes| E[Pull diff: code, config, feature flag, schema]
  D -->|No| F[Check shared dependency: DB, network, upstream API]
  E --> G[5-15m: Pick a reversible mitigation]
  F --> G
  G --> H{Recovering by 15m?}
  H -->|Yes| I[Validate recovery, stand down, write timeline]
  H -->|No| J[Escalate: page secondary and service owner, open incident]

Telling apart the three usual causes when several services fail together

When multiple services degrade at once, the fastest disambiguation is checking one thing first: are the affected services related by a shared dependency, or only by being on the same network path?

• Shared dependency outage (a database, cache, or upstream API multiple services call): errors cluster around calls to that one dependency across otherwise-unrelated services; the fix is usually failover or shedding load to the dependency, not touching the calling services.
• Network partition: services can't reach each other or a shared resource, but each service is individually healthy when checked in isolation (CPU, memory, internal logic all fine); the fix is on the networking layer, not application code.
• Code bug from the recent deploy: failures track cleanly to services that received the deploy, and services that didn't receive it stay healthy; the fix is rollback.
  The tell is the failure pattern: dependency outages show a common downstream call in every affected service's error logs, network partitions show healthy services that simply can't connect, and deploy-caused bugs map exactly to the set of services the deploy touched.

Two scenario variants worth knowing

A gradual climb (latency creeping up over 20 to 30 minutes rather than spiking) usually points at a resource exhaustion pattern, a connection leak or a slowly growing queue, rather than a bad deploy; the checklist is the same but step 3 shifts from "what deployed" to "what's been running long enough to leak." A regional error spike on a payments API narrows scope immediately to that region's infrastructure or a regional dependency, and raises the severity bar (payments plus regional means check whether it's revenue-impacting before deciding the escalation urgency).

Trade-offs and pitfalls

The most common mistake under pressure is skipping validation and mitigating against the alert text instead of the live dashboard, which can mean fixing a problem that already resolved itself while missing the real one. The second most common mistake is treating the 15-minute escalation threshold as optional because "I'm close", escalating early and being wrong costs a few minutes of someone else's attention; escalating late during a real outage costs users.