On-Call Practices and Runbook Design Questions

Direct answer

"Blameless" means the postmortem analyzes the incident as a failure of systems and processes that made a reasonable person's normal action produce a bad outcome, not as a failure of that person's competence or effort. It matters because the moment a postmortem starts assigning individual fault, people stop giving you the honest details (what they clicked, what they assumed, what they skipped) that you actually need to fix the underlying gap, and near-misses stop getting reported at all.

Structured elaboration

What blameless does not mean. It is not "no accountability." Individuals still own action items and are still expected to do their jobs well. Blameless means the analysis stops at "why did this look like the right thing to do at the time, given the information and tools available" instead of stopping at "who made the mistake."

Essential components of a postmortem document:

• Summary and impact: severity, duration, which customers/systems were affected, business impact.
• Timeline: timestamped sequence from first signal to full resolution, including detection and every mitigation attempt.
• Contributing factors: plural, not a single root cause. A "five whys" style chain that includes technical gaps (missing test, no lock-duration check) and process gaps (review checklist didn't require it, no staging environment with prod-sized data).
• What went well / what didn't: honest assessment of detection speed, mitigation effectiveness, and communication, separate from the technical cause.
• Action items: each with an owner, a due date, and a verification step, tracked to closure rather than left as a paragraph nobody revisits.
• Lessons learned: shared broadly enough that other teams with the same pattern can act on it before they hit the same incident.

Worked example

A deploy runs a database migration that locks a hot table for several minutes, causing a customer-facing outage. A blameful writeup says: "Engineer X pushed a migration without checking lock duration." A blameless writeup for the same incident says: "The migration tool doesn't warn about lock duration before merge, and the review checklist doesn't require a dry run against a prod-sized dataset. Both are now action items: add a lock-duration check to the migration tool (owner: platform team, verify by testing against a 10M-row table), and add a prod-sized dry-run step to the migration checklist (owner: DBA lead, verify by auditing the next five migrations)." The second version identifies exactly the same failure but produces two concrete, ownable fixes instead of a warning to be more careful next time, which is not something that reliably prevents a repeat.

Trade-offs and pitfalls

• Pitfall: blameless in the document but not in the room. Teams sometimes write a passive-voice, name-free doc while the actual retro meeting is full of pointed questions at one person. The doc's tone has to match the meeting's tone, or the culture stays blameful regardless of what's written down.
• Pitfall: treating blameless as zero consequences ever. Repeated, egregious negligence (ignoring a known policy, skipping a required review on purpose) is a people-management conversation, but it happens outside the technical postmortem, not inside it.
• Senior signal: distinguishing proximate cause from contributing factors. A junior answer stops at "the migration locked the table." A senior answer keeps asking why the tooling, the review process, and the testing environment all failed to catch it, because single-root-cause thinking tends to produce a single, shallow fix that doesn't survive the next incident with a different trigger but the same underlying gap.

A production-readiness review should verify four things before a service starts taking pages: it fails safely (degrades or rolls back instead of cascading), it's observable enough that on-call can diagnose without guessing, on-call actually knows how to respond to it, and someone specific owns it. Structure the review around those four, not a flat checklist, so gaps are obvious by category rather than buried in a long list.

What the review checks, by category

Process for running the review

1. The owning team self-certifies against the checklist first, providing evidence (load-test results, a link to the rollback runbook, a screenshot of the dashboard) rather than a checked box with no backing.
2. A reviewer outside the owning team (SRE or a peer team) spot-checks the evidence, focusing on the failure-containment and observability rows, since those are the ones teams under launch pressure are most likely to overstate.
3. Run one live-fire test before go-live: trigger the most likely failure mode in staging (or a controlled prod canary) and confirm the alert fires, the runbook's diagnosis steps actually find the cause, and the rollback works. A checklist that's never been exercised is a hypothesis, not a verified readiness state.
4. Sign-off is explicit and time-bound, not a one-time gate that's forgotten: re-review triggers on major architecture changes, not just at initial launch.

flowchart TD
  A[Owning team self-certifies checklist] --> B[Provide evidence: load tests, runbook links, dashboards]
  B --> C[Outside reviewer spot-checks evidence]
  C --> D{Gaps found?}
  D -->|Yes| E[Team remediates gap]
  E --> C
  D -->|No| F[Live-fire test in staging or canary]
  F --> G{Alert fires, runbook works, rollback succeeds?}
  G -->|No| E
  G -->|Yes| H[Sign-off: service takes pages]

Worked example

A new recommendations service is going live. Self-certification claims load testing was done "at expected traffic." The outside reviewer asks for the actual load-test report and finds it tested at the current expected peak (500 req/s) with no margin, while the service also sits behind a feature flag that product plans to ramp to three times that within a month. That's a real gap: the review isn't asking for perfection, but it should require either testing at the higher number now or an explicit, documented plan (with an owner and date) to re-test before the ramp, rather than letting "tested at expected traffic" silently mean "tested at today's traffic." The live-fire test then finds the circuit breaker on the downstream recommendation-model call has no timeout configured, so a slow model response would hang the request instead of failing fast; that's flagged as a blocking issue, not a follow-up ticket, because it directly causes cascading failure under exactly the load condition the service is meant to handle.

Trade-offs and pitfalls

The live-fire test step is the one teams most often skip under launch deadline pressure, and it's also the one that catches the gaps self-certification checklists miss (an alert that's configured but never actually fires, a runbook step that references a dashboard that doesn't exist); treat it as non-negotiable for anything customer-facing, and reserve a lighter self-certification-only path for low-risk internal services. A common wrong turn is treating the checklist as complete once every box is checked, without weighting which gaps are load-bearing; a missing rollback plan for a payments-adjacent service is not the same severity as a missing dashboard for an internal admin tool, and the review should say so explicitly rather than gating everything equally.

Whether to fully automate, keep human-in-the-loop, or leave manual comes down to four questions applied to each specific action: how often does it happen, how bad is it if it goes wrong, can it be safely retried, and can success be verified automatically. High frequency, low blast radius, idempotent, and observable pushes toward full automation; anything destructive or hard to verify stays manual or gated behind a human, no matter how routine it feels.

The criteria

Applying it to the three actions

Restarting a cached worker instance: high frequency, low blast radius (stateless, replaceable), fully idempotent, and easily verified with a health check. This is a strong automate candidate: drain connections, spin up a replacement, run a health check, cut traffic over, roll back automatically if the health check fails.

Reattaching a detached volume: lower frequency, meaningfully higher blast radius (attaching to the wrong instance or double-attaching can corrupt data), and only moderately idempotent, reattaching twice isn't necessarily safe. This sits in the middle: automate the pre-checks and the mechanical steps (verify volume ID, verify target instance, snapshot before attaching), but require a human to confirm before the final attach executes.

Running a database schema migration: low frequency, high blast radius (can be destructive and hard to reverse), low idempotency for anything involving DDL (data definition language: schema-altering SQL statements like ALTER TABLE), and success often isn't verifiable by a simple automated check, it needs someone to look at whether the data actually came out right. This stays manual, or more precisely, human-gated: automation handles the mechanical parts (schema diff, pre-migration validation, backup, staged rollout to a canary), but a person approves the production apply.

The underlying argument for phasing automation in gradually

Automating a step doesn't just remove toil, it also removes the moment a human would have caught something unusual about this particular instance of the problem. That's fine for the worker-restart case, where "unusual" mostly doesn't exist, but risky for the migration case, where every migration is a little different. The practical path is phasing: run a new automation in shadow mode first (it proposes the action but a human executes), then human-in-the-loop (it executes after one-click approval), and only promote to full automation once it has a track record across enough real incidents that its false-positive and false-negative rate are actually known, not assumed.

Guarding automated actions with least privilege

Whatever is automated should run with only the permissions that specific action needs, a worker-restart automation shouldn't hold credentials that could also run a schema migration, and every automated action should be logged with who (or what) triggered it and why. For the human-in-the-loop tier, the approval step itself should require a specific person's action (not a shared bot token anyone can trigger), so there's a real approval trail, not a rubber stamp.

Trade-offs and pitfalls

The common wrong turn is automating based on how annoying a task feels rather than how safe it is, restarting workers manually is annoying but safe to automate; migrations are also annoying, but the annoyance is not the variable that should decide it. The other pitfall is leaving a human-in-the-loop step gated behind an approval that nobody actually reads before clicking, if the approval doesn't include enough context (what will run, what's the blast radius, what's the rollback) to make a real judgment, it's automation with an extra click, not a genuine safety gate.

The first few minutes after a page are for narrowing down scope and cause fast enough to act, not for finding the root cause. I validate the alert is real, figure out how big the blast radius is, check whether it lines up with the deploy that just went out, and pick a mitigation I can undo, escalating the moment the clock runs out on any of those steps.

The first-minutes checklist

1. Validate (0-2 min): confirm the page against the actual dashboard, not just the alert text. Is p95 latency and error rate really elevated right now, or is this a flapping alert that already recovered?
2. Determine scope (2-5 min): is this one service, one region, or several services at once? Check whether other, unrelated-looking alerts fired in the same window, they're often the same root cause fanning out, not five separate problems.
3. Correlate with the deploy (2-5 min): the timing (a spike a few minutes after a deploy) is a strong signal but not proof; pull the diff (code, config, feature flags, schema changes) to see if anything plausibly explains a latency or error increase.
4. Choose a reversible mitigation (5-15 min): rollback if the deploy is the clear cause and rollback is safe and fast; scale out or shed load if the cause looks like saturation; flip a feature flag if the change is flag-gated. Prefer whatever gets users out of pain fastest and can be undone if it turns out to be wrong.
5. Escalate on the clock, not on ego (by 15 min): if scope, cause, or mitigation isn't clear by 15 minutes, escalate, that's a threshold, not a judgment call to second-guess in the moment.

Decision flow

flowchart TD
  A[Page: p95 latency + error rate up, deploy went out minutes ago] --> B[0-2m: Validate against real dashboards]
  B --> C[2-5m: Determine scope: one service or several]
  C --> D{Timing matches the deploy?}
  D -->|Yes| E[Pull diff: code, config, feature flag, schema]
  D -->|No| F[Check shared dependency: DB, network, upstream API]
  E --> G[5-15m: Pick a reversible mitigation]
  F --> G
  G --> H{Recovering by 15m?}
  H -->|Yes| I[Validate recovery, stand down, write timeline]
  H -->|No| J[Escalate: page secondary and service owner, open incident]

Telling apart the three usual causes when several services fail together

When multiple services degrade at once, the fastest disambiguation is checking one thing first: are the affected services related by a shared dependency, or only by being on the same network path?

• Shared dependency outage (a database, cache, or upstream API multiple services call): errors cluster around calls to that one dependency across otherwise-unrelated services; the fix is usually failover or shedding load to the dependency, not touching the calling services.
• Network partition: services can't reach each other or a shared resource, but each service is individually healthy when checked in isolation (CPU, memory, internal logic all fine); the fix is on the networking layer, not application code.
• Code bug from the recent deploy: failures track cleanly to services that received the deploy, and services that didn't receive it stay healthy; the fix is rollback.
  The tell is the failure pattern: dependency outages show a common downstream call in every affected service's error logs, network partitions show healthy services that simply can't connect, and deploy-caused bugs map exactly to the set of services the deploy touched.

Two scenario variants worth knowing

A gradual climb (latency creeping up over 20 to 30 minutes rather than spiking) usually points at a resource exhaustion pattern, a connection leak or a slowly growing queue, rather than a bad deploy; the checklist is the same but step 3 shifts from "what deployed" to "what's been running long enough to leak." A regional error spike on a payments API narrows scope immediately to that region's infrastructure or a regional dependency, and raises the severity bar (payments plus regional means check whether it's revenue-impacting before deciding the escalation urgency).

Trade-offs and pitfalls

The most common mistake under pressure is skipping validation and mitigating against the alert text instead of the live dashboard, which can mean fixing a problem that already resolved itself while missing the real one. The second most common mistake is treating the 15-minute escalation threshold as optional because "I'm close", escalating early and being wrong costs a few minutes of someone else's attention; escalating late during a real outage costs users.

MTTD is how long a problem existed before anything noticed it. MTTA is how long a human took to acknowledge the alert once it fired. MTTR is how long it took to fully resolve once someone was working it. The most common mistake is reporting a single blended average and treating it as typical, when one long outage in the set is doing all the work.

Definitions

(Some teams instead measure MTTR from detection to resolve rather than ack to resolve; either is defensible, but the convention has to be fixed and stated, because mixing them across teams silently changes what the number means.)

Worked example: one incident timeline

Worked example: averaging across three incidents, and where it goes wrong

The mean of 36.7 minutes is being pulled up almost entirely by incident 3's 54-minute outlier: the mean sits above two of the three data points (20 and 36), with only the outlier itself larger. The median of {20, 36, 54} is 36, the middle value itself rather than a value inflated by the outlier, so it is a better single-number stand-in for the typical incident than the mean. Reporting mean MTTR alone, without the incident count or a percentile, makes a single bad incident look like the typical case.

Trade-offs and pitfalls

Comparing MTTR across teams that use different start-point conventions is comparing two different metrics wearing the same name; agree on the convention org-wide before benchmarking teams against each other. A dropping mean MTTR can hide a rising incident count: if you're resolving more small incidents faster while one rare severe incident still takes hours, the mean improves and the tail risk hasn't moved at all. Improving MTTD without improving MTTA or MTTR just means you find out about the same slow response faster; treat the three as stages of one pipeline, not independent wins to report separately.