On-Call Practices and Runbook Design Questions

Direct answer

"Blameless" means the postmortem analyzes the incident as a failure of systems and processes that made a reasonable person's normal action produce a bad outcome, not as a failure of that person's competence or effort. It matters because the moment a postmortem starts assigning individual fault, people stop giving you the honest details (what they clicked, what they assumed, what they skipped) that you actually need to fix the underlying gap, and near-misses stop getting reported at all.

Structured elaboration

What blameless does not mean. It is not "no accountability." Individuals still own action items and are still expected to do their jobs well. Blameless means the analysis stops at "why did this look like the right thing to do at the time, given the information and tools available" instead of stopping at "who made the mistake."

Essential components of a postmortem document:

• Summary and impact: severity, duration, which customers/systems were affected, business impact.
• Timeline: timestamped sequence from first signal to full resolution, including detection and every mitigation attempt.
• Contributing factors: plural, not a single root cause. A "five whys" style chain that includes technical gaps (missing test, no lock-duration check) and process gaps (review checklist didn't require it, no staging environment with prod-sized data).
• What went well / what didn't: honest assessment of detection speed, mitigation effectiveness, and communication, separate from the technical cause.
• Action items: each with an owner, a due date, and a verification step, tracked to closure rather than left as a paragraph nobody revisits.
• Lessons learned: shared broadly enough that other teams with the same pattern can act on it before they hit the same incident.

Worked example

A deploy runs a database migration that locks a hot table for several minutes, causing a customer-facing outage. A blameful writeup says: "Engineer X pushed a migration without checking lock duration." A blameless writeup for the same incident says: "The migration tool doesn't warn about lock duration before merge, and the review checklist doesn't require a dry run against a prod-sized dataset. Both are now action items: add a lock-duration check to the migration tool (owner: platform team, verify by testing against a 10M-row table), and add a prod-sized dry-run step to the migration checklist (owner: DBA lead, verify by auditing the next five migrations)." The second version identifies exactly the same failure but produces two concrete, ownable fixes instead of a warning to be more careful next time, which is not something that reliably prevents a repeat.

Trade-offs and pitfalls

• Pitfall: blameless in the document but not in the room. Teams sometimes write a passive-voice, name-free doc while the actual retro meeting is full of pointed questions at one person. The doc's tone has to match the meeting's tone, or the culture stays blameful regardless of what's written down.
• Pitfall: treating blameless as zero consequences ever. Repeated, egregious negligence (ignoring a known policy, skipping a required review on purpose) is a people-management conversation, but it happens outside the technical postmortem, not inside it.
• Senior signal: distinguishing proximate cause from contributing factors. A junior answer stops at "the migration locked the table." A senior answer keeps asking why the tooling, the review process, and the testing environment all failed to catch it, because single-root-cause thinking tends to produce a single, shallow fix that doesn't survive the next incident with a different trigger but the same underlying gap.

Direct answer

Base the rubric on customer-facing impact and business consequence, not internal technical severity: P0 is a full outage or a safety or data-loss risk with immediate paging and an aggressive SLA, scaling down through partial-impact and degraded-but-functional states to P3, a backlog item with no paging at all. Each level pairs a concrete definition with a response SLA, a resolution SLA, and exactly who gets paged, so classification is a lookup an on-call engineer can make under pressure, not a judgment call.

Structured elaboration

What determines the level

• Scope: how many customers or what fraction of traffic is affected.
• Reversibility of harm: data being lost or corrupted pushes toward P0 regardless of how few customers are affected, versus a fully recoverable error.
• Whether a workaround exists for the customer.
• Contractual exposure: whether this breaches an SLA the company is financially on the hook for.

Reclassifying as more information arrives

Declare an initial severity fast, from the first available signal, and treat it as provisional. Many incidents start looking like P1 and get upgraded to P0 once data loss is confirmed, or start as P0, a total outage, and downgrade to P1 once a workaround is found. Reclassification in either direction should be cheap and require no approval, because holding onto an inaccurate severity either under-pages a real emergency or burns unnecessary on-call attention.

Worked example

A payment-processing API returns errors for all merchants for several minutes before an automatic circuit breaker reroutes traffic to a backup provider, after which errors drop back to baseline. Applying the table: the initial signal, all merchants affected and revenue-blocking, classifies this as P0 and pages the on-call engineer, the service owner, and the incident commander immediately. Once the reroute confirms the impact is contained and no data was lost, the incident is reclassified to P1, SLA-covered functionality degraded with a workaround in place via the backup provider, for the remainder of the response. That reclassification changes the resolution SLA from continuous work under 4 hours to a target under 24 hours, but it doesn't stand down the already-paged responders mid-incident.

Trade-offs and pitfalls

• Defining severity by an internal technical signal, like an error-rate threshold, instead of customer impact, treats a high error rate on a low-traffic internal endpoint the same as the same error rate on checkout, which it isn't.
• Too many severity levels creates ambiguity at classification time under pressure; four tiers is usually enough resolution to route paging and SLAs correctly without forcing a judgment call between two levels that don't functionally differ.
• Making reclassification require a meeting or approval means on-call will just leave the severity wrong, which quietly corrupts incident metrics later.

Direct answer

A good handoff transfers three things: current state (what's broken or at risk right now), context (what's already been tried and what's scheduled), and ownership (who is accountable for what next), and it has to be verifiable rather than a status dump, meaning the incoming engineer confirms they can actually act (they have access, they can reproduce the symptom, they understand the next step) before the outgoing engineer is done.

Structured elaboration

Handoff checklist, in order:

1. Status snapshot: current owner and contact, "all green" or a one-line count of active incidents/unstable services.
2. Active incidents: for each, severity, start time, impact, current owner, and a link to the ticket, not a re-explanation from scratch.
3. Recent alert history: what's fired in the last several hours, and specifically any alert that's been flapping, since that's exactly what an incoming responder will misdiagnose as new if nobody flags it.
4. Ongoing mitigations and runbook links: what's been tried, what's blocked, and the specific next action, with a link to the runbook rather than a paraphrase of it.
5. Scheduled changes: upcoming deploys, migrations, or maintenance windows during the next shift, with rollback plans linked.
6. Degraded-but-not-incident services: anything running hot or close to a threshold that isn't paging yet.
7. Access and tooling: pager rotation, incident channel, dashboards, runbook repo, and an explicit note if the incoming engineer is missing access to any of them.
8. Explicit confirmation, not implied: "Can you access the dashboards I linked? Can you reproduce the symptom in incident #123? Do you agree you own the DB migration follow-up?" Each gets an actual yes, not a thumbs-up emoji on a wall of text.

Making it lightweight (chatops mechanics). For day-to-day handoffs without an active incident, a short structured chat message beats a long document nobody reads: header (shift window, owner, escalation contact), one-line status, action items with owners, upcoming risks, and links. Anything in that message that turns out to matter beyond the shift boundary (a workaround that becomes permanent, a gotcha that will recur) gets tagged for follow-up and folded into the actual runbook within a day or two, so the team's durable documentation doesn't quietly live and die in chat history.

Automating the tedious part. The status snapshot, active-incident list, and recent-alert-history sections don't need to be typed by hand: a handoff template can be pre-populated from the monitoring and paging systems (current alert state, open incident IDs, last-deploy timestamp) so the outgoing engineer is editing and confirming pre-filled facts rather than writing a report from a blank page. That reduces both the time cost of handoff and the chance that something gets left out because the outgoing engineer forgot it existed.

Worked example

Friday, 6pm, end of a shift. One active Sev2 incident (checkout latency degraded for a subset of EU traffic, mitigation in progress: a feature flag was flipped to route around a slow dependency, error rate has dropped but root cause isn't fixed), and a database migration scheduled for 2am that night. The outgoing engineer posts:

Handoff | Fri 18:00-Sat 02:00 UTC
Owner: @outgoing -> @incoming | Escalation: @oncall-lead
Status: Degraded (checkout latency, EU) -> incident #482, mitigated not resolved
Action items:
 - Watch checkout error rate; if it climbs above 2% again, re-check the feature flag is still on
 - DB migration at 02:00 UTC (runbook: <link>, rollback: <link>) - I'll be asleep, this is yours
Risks: migration touches the same table implicated in incident #482; if latency spikes right after, check the migration first
Links: <dashboard> <incident #482> <migration runbook>

The incoming engineer confirms: dashboard access works, they can see incident #482's current state, and they explicitly acknowledge owning the migration watch. That confirmation, not the message itself, is what makes the handoff complete.

Trade-offs and pitfalls

• Pitfall: a "read the ticket" handoff with no verification step lets the incoming responder discover gaps at 3am instead of at 6pm when the person with context is still reachable.
• Pitfall: too much ceremony (a mandatory 45-minute call every single handoff) burns out the outgoing engineer and makes people avoid going on-call at all; reserve synchronous overlap for when there's an active Sev1/Sev2, not as the default for a quiet shift.
• Trade-off: synchronous handoff transfers tacit knowledge best but costs both people's time; async structured notes are cheaper but only as good as their template and discipline. The right default is async-by-default, with synchronous overlap triggered automatically whenever an incident is still open at shift boundary.

Readiness is a checklist, not a countdown: get access and tooling working first, have them study and sign off on the runbooks for their services, shadow several live pages, then run one supervised tabletop and one supervised live (or simulated) incident before they take a shift alone with a mentor reachable but not present.

A four-week ramp

Sign-off checklist before the first unsupervised shift

• Access confirmed end to end (paging tool, dashboards, deploy/rollback permissions) with a real test, not just "provisioned."
• Runbooks for their assigned services reviewed and any ambiguous steps flagged and fixed.
• At least one supervised tabletop and one supervised live or injected-fault incident completed.
• Mentor sign-off plus the engineer's own confidence self-assessment, not mentor judgment alone.

Extending the ramp for a complex or high-stakes service

Four weeks is often enough for a straightforward service, but for a complex hybrid-cloud system or one with many downstream dependents, add explicit competency checkpoints tied to named systems, for example certifying someone independently on the database failover path as a separate sign-off from general on-call readiness, rather than declaring them ready across the board at once. Longer term, treat the first 90 days as a structured mentorship arc rather than stopping at week four: scheduled 30/60/90-day check-ins, a second mentor pairing on a different service, and a distinct milestone for graduating from "supervised" to "primary" status rather than just a date on the calendar.

Trade-offs and pitfalls

Rushing readiness to fill a rotation gap is the most common failure mode, and it produces confident-sounding but wrong incident responses, which is worse than an obviously under-prepared response because it takes longer to catch. A checklist with no live-incident component only validates that someone can read, not that they can act under time pressure; keep at least one supervised live or simulated incident before signing off. Sign-off criteria should be service-specific rather than one generic "on-call ready" badge, since readiness on a well-instrumented service doesn't automatically transfer to a fragile legacy one with thin runbooks.

Direct answer

Keep the incident moving without trusting the stale command: switch to safe, read-only discovery to re-derive the actual current state instead of assuming the runbook's exact syntax still matches reality, get a second responder to sanity-check any ad-hoc workaround before running it, and narrate every command and its outcome in the incident channel as you go. Afterward, treat the correction as a first-class, owned follow-up: file it with the incident as evidence and require the same review the runbook normally gets, rather than merging a fix drafted under adrenaline with no second pair of eyes.

Structured elaboration

Keeping the response moving

• Don't keep retrying the stale command hoping it starts working; that burns clock on a Sev1.
• Fall back to read-only discovery to find what actually changed: list the current resource names or config instead of assuming the runbook's exact prior values, and check whether a known change (a migration, a rename, a tool upgrade) explains the mismatch.
• If a workaround command is genuinely needed, treat it like an experiment: scope it to the smallest possible blast radius (a single pod, host, or canary) if at all possible, and have a second responder review it before running anything destructive.
• Narrate in the incident channel as you go: exact command, who ran it, what happened. This live log is what makes the eventual runbook fix accurate instead of a reconstruction from memory the next morning.

Getting the runbook actually fixed afterward

• File the correction as its own owned action item, not "someone should update this."
• Attach evidence: the incident timeline entries showing what actually worked, captured live rather than recalled later.
• Route the fix through the same review the runbook would normally require. A fix drafted under incident pressure is exactly the kind of change that benefits from a second reviewer, not an exception to needing one.
• If the drift has a systemic cause, such as no process tying documentation updates to the change that invalidated it, raise that separately as its own postmortem action item, not just a one-off doc patch.

Worked example

kubectl rollout restart deployment/checkout -n prod fails with "deployment not found." The responder falls back to read-only discovery: kubectl get deploy -A | grep checkout shows the deployment now lives in namespace checkout-prod, after a namespace-per-service migration weeks earlier that never touched the runbook. The corrected command runs, the service recovers, and both the failed and working commands are logged with timestamps in the incident doc as they happen. The resulting follow-up action item reads: "update runbook RB-042's namespace reference and add a namespace-lookup step instead of a hardcoded name; owner: platform team; verified by a peer review plus a sandboxed dry-run before merge."

Trade-offs and pitfalls

• Fixing the runbook file directly, mid-incident, with no review is a common shortcut; the correct fix ships as a follow-up change through the normal review gate, informed by what was learned live.
• Treating the ad-hoc working command as tribal knowledge instead of writing it down immediately is how the same staleness reappears at the next incident; capture it in the channel the moment it works, not after the retro.
• Verifying a corrected command in a sandbox before trusting it live is safer, but a Sev1 usually doesn't have that time; this is really an argument for building runbooks with idempotent, safe-to-retry commands and dynamic lookups in the first place, so on-call isn't forced into that trade-off during the incident.