On-Call Practices and Runbook Design Questions

In the moment, redirect the conversation from the person to the timeline: acknowledge what was said without amplifying it, then immediately steer the group back to reconstructing what happened and why the system allowed it, while making clear that accountability for the fix is not going away.

In-the-moment response

• Interrupt with a redirect, not a confrontation. Something like: "Let's hold on names for a second and walk the timeline: what did the system show at each step?" This isn't ignoring what was said; it's refusing to let the postmortem's structure reward the blame framing by continuing down that thread.
• Reframe the specific claim into a system question. If the IC (Incident Commander, the person directing the response) says "this happened because Priya deployed without checking the dashboard," the redirect is: "So the deploy process didn't require a dashboard check before going out. Is that a gap in the checklist, or did the checklist exist and get skipped? Either answer tells us what to fix." This keeps the factual content (a deploy went out without a check) while stripping the blame framing.
• Do not let it pass silently either. Staying quiet when a peer is singled out in front of the team reads as agreement, and it's the fastest way to make the next engineer afraid to be transparent in their own postmortem. A short, calm correction in the room is better than a private word afterward, because the damage (and the culture signal) happened publicly.
• Follow up with the IC privately, separate from the room. The public redirect handles the moment; a private conversation afterward addresses the pattern, especially if this IC does it repeatedly.

Keeping accountability intact

Blameless does not mean no one owns the fix. The distinction to hold onto:

• Blame assigns fault for what already happened, to a person, and looks backward.
• Accountability assigns ownership for what happens next, to a role or system, and looks forward.
  So the postmortem should still end with a named owner for each remediation item (a person, because someone has to actually do the work) and a deadline, but the framing is "you're the best person to close this gap because you understand the deploy path," not "this is your fault so you have to fix it." The action items get assigned based on who has the context and capability, independent of who gets blamed.

Worked example

During a payments-outage postmortem, the IC says: "Marcus rolled back the config and that's what caused the second outage." The redirect: "Let's look at what the rollback runbook told him to check before rolling back. Did it call out this specific config's downstream dependency?" The team pulls up the runbook and finds it didn't mention that this particular config was read by two other services; the rollback step existed, but the pre-check for downstream impact didn't. The postmortem action items become: (1) add a downstream-dependency check to the rollback runbook for this config, owned by the platform team, due in two weeks, and (2) audit other high-fanout configs for the same missing check, owned by Marcus, since he now has the clearest picture of what that gap looks like, due in one month. Marcus ends up with an action item, but it's framed as "you're positioned to close this" rather than "you caused this," and the runbook gap, not Marcus's judgment, is recorded as the finding.

Trade-offs and pitfalls

The main pitfall is overcorrecting into vagueness, where "blameless" gets used to avoid naming any specific decision point, and the postmortem ends up too soft to actually change anything; the fix is to be precise about the decision and the missing guardrail while staying impersonal about who made the decision. A related pitfall specific to this scenario: correcting an incident commander in front of the team carries real interpersonal risk if done poorly, so the redirect has to stay factual and calm rather than accusatory itself. This is a leadership-culture issue that shows up at scale too: it typically takes deliberate, sustained work, roughly a couple of quarters of consistent leadership behavior, published blameless postmortems, and visible non-punitive handling of pages, to shift a team's on-call culture away from a punitive default, and it has to be reinforced the same way every time, including in the exact moment someone in authority breaks the pattern.

When an automated remediation makes things worse, the first move is to stop trusting the automation, not to debug it live: disable the trigger (feature flag or scheduler pause) so it can't fire again while you investigate, then treat the automation's own actions as the incident's primary evidence trail.

Investigation approach

• Pull the automation's own audit log first. What did it decide to do, on what input, at what timestamp? Most remediation frameworks log the triggering condition and the action taken; if this one doesn't, that's itself a finding.
• Reconstruct the precondition it evaluated against. Was the health check it used stale (cached metrics, delayed scrape) or narrower than reality (checked one replica's health, not cluster quorum)?
• Check for concurrency. Did two instances of the same remediation run at once, or did it run while a human was mid-deploy? Interleaved writes to the same resource are a common cause of "fix that broke things."
• Diff the assumed environment against the actual one. Runbooks and remediation scripts encode assumptions (resource names, API versions, cluster topology) that drift silently; check whether the automation was written against a topology that's since changed.

Framework for the fix

1. Add a pre-check gate: the remediation must verify the system is in the state it assumes (quorum present, no in-flight deploy, dependency healthy) before acting, and abort loudly if not.
2. Make the action idempotent and reversible: re-running it, or running it against a system already in the target state, should be a no-op, and every destructive step needs a paired rollback.
3. Bound the blast radius: act on one node/instance first (canary), verify success, then proceed, rather than acting cluster-wide in one shot.
4. Add a concurrency guard: a lock or lease so two triggers of the same remediation can't run simultaneously.
5. Gate high-impact actions behind a second signal: require the automation to see the problem confirmed by two independent signals (e.g., an alert plus a direct health check) before taking a destructive action, not just one noisy metric.

Worked example

Suppose the remediation is: "if a node reports high memory for 3 consecutive scrapes, cordon and drain it." The postmortem finds the metrics scraper had a 90-second collection lag during a load spike, so by the time the automation cordoned the third node it was actually reading data that was already 4.5 minutes stale (three 90-second-lagged scrapes), and it drained three nodes in the same 2-minute window because the memory spike was cluster-wide, not node-specific. Losing three nodes at once dropped the cluster below quorum for its replicated service, which is the partial outage.
The fix that follows directly from that trace: (a) the pre-check should compare current live memory, not the lagged scrape, before acting; (b) the automation should check how many nodes it has already drained in the current window and refuse to exceed a cap (e.g., no more than one node per 10 minutes) until a human confirms; (c) it should check that the remaining fleet still satisfies quorum before draining another node.

Trade-offs and pitfalls

Adding pre-checks and rate caps makes the remediation slower to react, which is the right trade for anything that can cause an outage of its own; reserve fully unthrottled auto-remediation for actions that are cheap to reverse (like restarting a single stateless pod) and keep caps and human gates on anything that removes capacity or touches shared state. A common wrong turn is to respond to this incident by simply disabling the automation permanently and reverting to manual remediation: that trades a rare automation bug for a much larger population of slower, inconsistent manual responses. The senior move is to narrow what the automation is trusted to do unsupervised, not to abandon automation.

Direct answer

Runbooks go stale because nothing automatically ties them to the systems they describe: ownership is unclear, updates aren't triggered by the changes that invalidate them, and nobody is rewarded for maintaining them, so they drift silently until an incident exposes it. The fix for each cause is the same shape: build the update into a workflow that already has to happen, like a deploy, a PR review, or a drill, rather than relying on someone remembering.

Structured elaboration

Worked example

A payments team migrates from a single database instance to a managed cluster with a different failover tool. The failover runbook still references the old promote command. Nobody touches the runbook because the migration's review process had no requirement to touch documentation tied to it, which is exactly the "no trigger tied to system changes" row above. Months later, an on-call engineer hits a real primary failure, runs the stale command, gets an error, and has to rediscover the correct procedure live instead of following a runbook that already had it. The root cause traces cleanly to the missing trigger, not to the engineer who wrote the original doc.

Trade-offs and pitfalls

• Quarterly "please review this doc" reminders without a named owner tend to become checkbox theater: marked reviewed without anyone actually re-verifying the steps.
• Gating merges on documentation updates adds friction to every infra change; scope the gate to changes that touch a documented procedure specifically, or teams will route around it entirely.

When the same incident keeps recurring despite repeated fixes, the prior RCAs were almost certainly treating a symptom as the root cause; run this RCA by explicitly listing every prior "fix" and asking why each one didn't hold, because the pattern across failed fixes usually points straight at the real systemic gap.

Framework for a systemic RCA

1. Build a fix history first, before investigating the current occurrence. For each prior incident of this same recurring issue: what was diagnosed as the cause, what was changed, and did the change actually address that diagnosis or just the immediate symptom? A pattern of "different symptom fixed each time, same underlying trigger every time" is the tell that root cause was never actually found.
2. Separate the trigger from the vulnerability. The trigger (a specific deploy, a specific load pattern, a specific external dependency hiccup) may vary each month, but if the same class of trigger keeps causing an outage, the system has a standing vulnerability to that trigger class that no single fix removed. The RCA's job is to name the vulnerability, not just the latest trigger.
3. Use the fishbone categories (code, config, infrastructure, process, tooling) to check whether every prior fix landed in the same category. If four consecutive fixes were all code patches but the incident keeps returning, that's evidence the real gap is in process or tooling (no regression test for this class of failure, no canary catching it before full rollout) rather than in any specific line of code.
4. Test the systemic hypothesis, don't just assert it. If the hypothesis is "the nightly batch job and the backup window contend for the same database connection pool," verify by reproducing that contention in a controlled environment (isolated run of the batch job during a simulated backup window), not by pattern-matching from the incident timeline alone.

Worked example

A nightly batch job has caused three partial outages in three consecutive months. Prior fixes: month 1, increased the job's timeout (fix addressed "job was timing out"); month 2, added a retry with backoff (fix addressed "job failed transiently"); month 3 is the current incident, and the job is again failing, this time differently, a connection pool exhaustion error. Building the fix history shows a pattern: every fix targeted why the job failed on that specific night, and none asked why the job's failure mode changes every month while the timing (always during the nightly backup window) stays constant. Testing the systemic hypothesis, that the batch job and backup process share a connection pool and the backup's duration has been slowly growing as data volume grows, month-over-month backup duration logs confirm the backup window has grown from roughly 12 minutes to 40 minutes over the quarter, now overlapping the batch job's peak connection usage. That is the systemic cause: the job and backup were never intentionally isolated, and it was invisible for months because the backup was short enough not to overlap.
The durable fix follows from the systemic cause, not the latest symptom: allocate the batch job a dedicated connection pool separate from ad hoc processes, and alert on backup-duration trend (not just backup failure) so a slowly growing resource conflict is visible before it causes an outage again.

Trade-offs and pitfalls

The main pitfall is that a systemic RCA takes longer and produces a less satisfying immediate answer than "here's the line that broke," which creates pressure to ship another symptom-level fix under time pressure; the way to resist that is to make the fix-history review a required first step, not an optional deep-dive, so the systemic question gets asked before anyone commits to a scope. A real trade-off: broadening the RCA to a process or tooling gap usually means the fix is slower to land (new alerting, resource isolation, a regression test suite) than a code patch, so it's worth explicitly stating in the postmortem that a fast interim mitigation (in this example, a manual connection-pool bump) is being paired with the slower structural fix, rather than letting the slow fix block any near-term relief.

Direct answer

Base the rubric on customer-facing impact and business consequence, not internal technical severity: P0 is a full outage or a safety or data-loss risk with immediate paging and an aggressive SLA, scaling down through partial-impact and degraded-but-functional states to P3, a backlog item with no paging at all. Each level pairs a concrete definition with a response SLA, a resolution SLA, and exactly who gets paged, so classification is a lookup an on-call engineer can make under pressure, not a judgment call.

Structured elaboration

What determines the level

• Scope: how many customers or what fraction of traffic is affected.
• Reversibility of harm: data being lost or corrupted pushes toward P0 regardless of how few customers are affected, versus a fully recoverable error.
• Whether a workaround exists for the customer.
• Contractual exposure: whether this breaches an SLA the company is financially on the hook for.

Reclassifying as more information arrives

Declare an initial severity fast, from the first available signal, and treat it as provisional. Many incidents start looking like P1 and get upgraded to P0 once data loss is confirmed, or start as P0, a total outage, and downgrade to P1 once a workaround is found. Reclassification in either direction should be cheap and require no approval, because holding onto an inaccurate severity either under-pages a real emergency or burns unnecessary on-call attention.

Worked example

A payment-processing API returns errors for all merchants for several minutes before an automatic circuit breaker reroutes traffic to a backup provider, after which errors drop back to baseline. Applying the table: the initial signal, all merchants affected and revenue-blocking, classifies this as P0 and pages the on-call engineer, the service owner, and the incident commander immediately. Once the reroute confirms the impact is contained and no data was lost, the incident is reclassified to P1, SLA-covered functionality degraded with a workaround in place via the backup provider, for the remainder of the response. That reclassification changes the resolution SLA from continuous work under 4 hours to a target under 24 hours, but it doesn't stand down the already-paged responders mid-incident.

Trade-offs and pitfalls

• Defining severity by an internal technical signal, like an error-rate threshold, instead of customer impact, treats a high error rate on a low-traffic internal endpoint the same as the same error rate on checkout, which it isn't.
• Too many severity levels creates ambiguity at classification time under pressure; four tiers is usually enough resolution to route paging and SLAs correctly without forcing a judgment call between two levels that don't functionally differ.
• Making reclassification require a meeting or approval means on-call will just leave the severity wrong, which quietly corrupts incident metrics later.