On-Call Practices and Runbook Design Questions

Sequence the work from cheap and immediate (alert routing and runbook hygiene) to structural (instrumentation, then automation) over roughly two quarters, and prove it's real improvement rather than gaming by tracking reopen rate and severity-classification consistency alongside the headline MTTR number, not just the number alone.

Where the 90 minutes actually goes

Before proposing fixes, break the baseline down into the phases every incident passes through:

The phased program

Worked example: tracking the breakdown phase by phase

Guarding against gaming the metric

Pair MTTR with reopen rate and a severity-distribution audit, not just the headline number. If MTTR drops but the 24-hour reopen rate rises, for example from roughly 1 reopened incident a month to roughly 4 out of the same ~40 monthly incidents, that's a sign incidents are being closed to hit the target rather than actually fixed; treat a rising reopen rate as an automatic invalidator of the MTTR win until it's addressed. Separately, audit whether the mix of logged severities shifted (more incidents suddenly classified a notch lower than before); that alone can lower average MTTR without anything getting faster.

Trade-offs and pitfalls

Automation is the highest-leverage phase but also the highest-risk one; gate it behind canary rollout and audit logging so a bad automated fix doesn't become its own incident. A program driven purely by the MTTR number invites exactly the gaming described above; always report it alongside reopen rate and incident volume, never alone. Returns are not evenly spread across the phases, and they don't simply shrink over time: Phase 2 (instrumentation) delivers the largest single incremental win, 20 of the 90 minutes, bigger than either Phase 1's 11-minute alerting and runbook fix or Phase 3's 17-minute automation gain, because Phase 2 targets diagnosis, the single largest chunk of the baseline (35 of the 90 minutes). The lesson isn't that early phases always win biggest; it's that whichever phase targets the largest remaining bottleneck wins biggest, so sequence by where the time actually goes, not just by what's cheapest to ship first.

A strong answer to this picks one real incident, states plainly what your specific role was (not "the team"), walks through how status was communicated to the people who needed it, and ends with a concrete process change that came out of the postmortem, not just a lesson learned in the abstract. The bar an interviewer is checking is whether you can drive coordination under pressure and whether the org actually got better afterward, not whether the incident itself was dramatic.

Structuring the answer

• Situation: what broke, which teams were involved, and why it needed cross-team coordination rather than one person fixing it alone.
• Task: your specific role, on-call engineer, incident commander, SME being pulled in, and what you were responsible for.
• Action: what you actually did, especially the communication piece: how often you posted updates, who you looped in and when, how you decided what to escalate.
• Result: how the incident resolved, and the one concrete thing that changed afterward as a direct result of the postmortem.

Worked example (illustrative story skeleton)

Situation: I was on-call when a payment-processing service started failing a meaningful share of transactions during a peak traffic window, and it quickly became clear the cause spanned two teams, ours and the database team, not something either of us could fix alone.

Task: As the on-call engineer, my first job was triage and coordination, not fixing it single-handedly, since the fix needed the database team's context on a recent schema change.

Action: I opened a dedicated incident channel, pulled in the database on-call and our service owner, and posted a short status update on a fixed cadence so people weren't pinging me individually for progress. We traced the failures to a missing index from a recent migration, coordinated a rollback with the database team while I kept support and the affected team's manager updated with plain-language impact summaries (not the technical detail I was sharing in the incident channel).

Result: The rollback resolved the failure rate back to its normal baseline within the same on-call shift. In the blameless postmortem, the concrete change we made was adding a required index-verification check to the deploy pipeline for any migration, so a similar schema change can't ship without it being caught automatically rather than during a live incident.

Variant: when the story is about a communication breakdown

The strongest version of this question for a communication-focused answer isn't one where everything went smoothly, it's one where something specific broke in how status was shared, and what changed as a result. For example: during a similar incident, the customer-support team found out about the outage from customers before the incident channel posted a status update, because the communications role wasn't assigned and everyone assumed someone else was posting it. The fix that came out of that postmortem wasn't "communicate better", it was specific: the incident-declaration checklist now requires naming a communications owner in the first two minutes, before technical triage even starts, so that role is never implicitly assumed.

Trade-offs and pitfalls

The most common mistake is describing only the technical fix and skipping the coordination and communication details entirely, which is exactly the part this question is testing for. A close second is ending on "we learned to communicate better" without naming the specific process change that came out of it; a blameless postmortem is only as good as the concrete, trackable action item it produces, and an answer that mirrors that (a specific fix, not a vague lesson) is what separates a senior-sounding story from a generic one.

Direct answer

A good handoff transfers three things: current state (what's broken or at risk right now), context (what's already been tried and what's scheduled), and ownership (who is accountable for what next), and it has to be verifiable rather than a status dump, meaning the incoming engineer confirms they can actually act (they have access, they can reproduce the symptom, they understand the next step) before the outgoing engineer is done.

Structured elaboration

Handoff checklist, in order:

1. Status snapshot: current owner and contact, "all green" or a one-line count of active incidents/unstable services.
2. Active incidents: for each, severity, start time, impact, current owner, and a link to the ticket, not a re-explanation from scratch.
3. Recent alert history: what's fired in the last several hours, and specifically any alert that's been flapping, since that's exactly what an incoming responder will misdiagnose as new if nobody flags it.
4. Ongoing mitigations and runbook links: what's been tried, what's blocked, and the specific next action, with a link to the runbook rather than a paraphrase of it.
5. Scheduled changes: upcoming deploys, migrations, or maintenance windows during the next shift, with rollback plans linked.
6. Degraded-but-not-incident services: anything running hot or close to a threshold that isn't paging yet.
7. Access and tooling: pager rotation, incident channel, dashboards, runbook repo, and an explicit note if the incoming engineer is missing access to any of them.
8. Explicit confirmation, not implied: "Can you access the dashboards I linked? Can you reproduce the symptom in incident #123? Do you agree you own the DB migration follow-up?" Each gets an actual yes, not a thumbs-up emoji on a wall of text.

Making it lightweight (chatops mechanics). For day-to-day handoffs without an active incident, a short structured chat message beats a long document nobody reads: header (shift window, owner, escalation contact), one-line status, action items with owners, upcoming risks, and links. Anything in that message that turns out to matter beyond the shift boundary (a workaround that becomes permanent, a gotcha that will recur) gets tagged for follow-up and folded into the actual runbook within a day or two, so the team's durable documentation doesn't quietly live and die in chat history.

Automating the tedious part. The status snapshot, active-incident list, and recent-alert-history sections don't need to be typed by hand: a handoff template can be pre-populated from the monitoring and paging systems (current alert state, open incident IDs, last-deploy timestamp) so the outgoing engineer is editing and confirming pre-filled facts rather than writing a report from a blank page. That reduces both the time cost of handoff and the chance that something gets left out because the outgoing engineer forgot it existed.

Worked example

Friday, 6pm, end of a shift. One active Sev2 incident (checkout latency degraded for a subset of EU traffic, mitigation in progress: a feature flag was flipped to route around a slow dependency, error rate has dropped but root cause isn't fixed), and a database migration scheduled for 2am that night. The outgoing engineer posts:

Handoff | Fri 18:00-Sat 02:00 UTC
Owner: @outgoing -> @incoming | Escalation: @oncall-lead
Status: Degraded (checkout latency, EU) -> incident #482, mitigated not resolved
Action items:
 - Watch checkout error rate; if it climbs above 2% again, re-check the feature flag is still on
 - DB migration at 02:00 UTC (runbook: <link>, rollback: <link>) - I'll be asleep, this is yours
Risks: migration touches the same table implicated in incident #482; if latency spikes right after, check the migration first
Links: <dashboard> <incident #482> <migration runbook>

The incoming engineer confirms: dashboard access works, they can see incident #482's current state, and they explicitly acknowledge owning the migration watch. That confirmation, not the message itself, is what makes the handoff complete.

Trade-offs and pitfalls

• Pitfall: a "read the ticket" handoff with no verification step lets the incoming responder discover gaps at 3am instead of at 6pm when the person with context is still reachable.
• Pitfall: too much ceremony (a mandatory 45-minute call every single handoff) burns out the outgoing engineer and makes people avoid going on-call at all; reserve synchronous overlap for when there's an active Sev1/Sev2, not as the default for a quiet shift.
• Trade-off: synchronous handoff transfers tacit knowledge best but costs both people's time; async structured notes are cheaper but only as good as their template and discipline. The right default is async-by-default, with synchronous overlap triggered automatically whenever an incident is still open at shift boundary.

In the moment, redirect the conversation from the person to the timeline: acknowledge what was said without amplifying it, then immediately steer the group back to reconstructing what happened and why the system allowed it, while making clear that accountability for the fix is not going away.

In-the-moment response

• Interrupt with a redirect, not a confrontation. Something like: "Let's hold on names for a second and walk the timeline: what did the system show at each step?" This isn't ignoring what was said; it's refusing to let the postmortem's structure reward the blame framing by continuing down that thread.
• Reframe the specific claim into a system question. If the IC (Incident Commander, the person directing the response) says "this happened because Priya deployed without checking the dashboard," the redirect is: "So the deploy process didn't require a dashboard check before going out. Is that a gap in the checklist, or did the checklist exist and get skipped? Either answer tells us what to fix." This keeps the factual content (a deploy went out without a check) while stripping the blame framing.
• Do not let it pass silently either. Staying quiet when a peer is singled out in front of the team reads as agreement, and it's the fastest way to make the next engineer afraid to be transparent in their own postmortem. A short, calm correction in the room is better than a private word afterward, because the damage (and the culture signal) happened publicly.
• Follow up with the IC privately, separate from the room. The public redirect handles the moment; a private conversation afterward addresses the pattern, especially if this IC does it repeatedly.

Keeping accountability intact

Blameless does not mean no one owns the fix. The distinction to hold onto:

• Blame assigns fault for what already happened, to a person, and looks backward.
• Accountability assigns ownership for what happens next, to a role or system, and looks forward.
  So the postmortem should still end with a named owner for each remediation item (a person, because someone has to actually do the work) and a deadline, but the framing is "you're the best person to close this gap because you understand the deploy path," not "this is your fault so you have to fix it." The action items get assigned based on who has the context and capability, independent of who gets blamed.

Worked example

During a payments-outage postmortem, the IC says: "Marcus rolled back the config and that's what caused the second outage." The redirect: "Let's look at what the rollback runbook told him to check before rolling back. Did it call out this specific config's downstream dependency?" The team pulls up the runbook and finds it didn't mention that this particular config was read by two other services; the rollback step existed, but the pre-check for downstream impact didn't. The postmortem action items become: (1) add a downstream-dependency check to the rollback runbook for this config, owned by the platform team, due in two weeks, and (2) audit other high-fanout configs for the same missing check, owned by Marcus, since he now has the clearest picture of what that gap looks like, due in one month. Marcus ends up with an action item, but it's framed as "you're positioned to close this" rather than "you caused this," and the runbook gap, not Marcus's judgment, is recorded as the finding.

Trade-offs and pitfalls

The main pitfall is overcorrecting into vagueness, where "blameless" gets used to avoid naming any specific decision point, and the postmortem ends up too soft to actually change anything; the fix is to be precise about the decision and the missing guardrail while staying impersonal about who made the decision. A related pitfall specific to this scenario: correcting an incident commander in front of the team carries real interpersonal risk if done poorly, so the redirect has to stay factual and calm rather than accusatory itself. This is a leadership-culture issue that shows up at scale too: it typically takes deliberate, sustained work, roughly a couple of quarters of consistent leadership behavior, published blameless postmortems, and visible non-punitive handling of pages, to shift a team's on-call culture away from a punitive default, and it has to be reinforced the same way every time, including in the exact moment someone in authority breaks the pattern.

An escalation system for a multi-team, multi-timezone org needs three separate mechanisms working together: routing (getting an alert to the right on-duty person without a human deciding that in the moment), noise suppression (so one root cause doesn't fan out into ten pages), and a severity model that decides who gets pulled in and how fast, because a revenue-impacting outage and a data-sensitive incident need different people in the room, not just different urgency.

Core building blocks

• Alert gateway: every alert is deduplicated by fingerprint, enriched with service/team/severity tags, and checked against maintenance windows before it's allowed to page anyone.
• Routing table: maps service ownership and team schedule (including timezone-local business-hours windows) to whoever is currently on duty, kept in the paging tool as the single source of truth rather than a wiki page someone forgets to update.
• Severity model: decides who gets paged first and how many people, based on impact type, not just raw error rate.
• Escalation ladder: a fixed sequence of who gets paged next if no one acknowledges, with a hard time budget at each step.

Severity and routing matrix

Escalation flow

flowchart TD
  A[Alert fires] --> B[Gateway: dedupe, enrich, tag severity]
  B --> C{Severity type}
  C -->|Revenue-impacting| D[Page Primary, ack SLA 5m]
  C -->|Data-sensitive| E["Page Data on-call AND Compliance lead in parallel; notify Legal/Compliance on fixed clock, independent of ack"]
  D --> F{Acked by 5m?}
  F -->|No| G[Escalate to Secondary, ack SLA +10m]
  G --> H{Acked by 15m?}
  H -->|No| I[Escalate to Service Lead, open incident bridge]
  F -->|Yes| J[Primary mitigates]
  H -->|Yes| J
  E --> K{Acked by 5m?}
  K -->|No| L[Escalate both chains in parallel, ack SLA +10m]
  K -->|Yes| M[Data on-call + Compliance mitigate]
  L --> M

Preventing alert-noise from cascading into unnecessary escalations

Most alert storms come from one root cause tripping many downstream checks at once (a database going down pages every service that depends on it). The gateway groups alerts by a correlation key (same root dependency, same time window) before routing, so the escalation ladder above runs once for the incident, not once per symptom. Escalation timers also only start on the first page for a correlated group; late-arriving duplicates reset nothing.

Keeping the matrix trustworthy

A routing matrix that's wrong is worse than no matrix, because it creates false confidence. Two things keep it honest: primary/backup contacts are pulled live from the scheduling tool rather than hand-maintained, and if the on-duty person marks themselves absent (leave, travel) in that same tool, pages route to the next person automatically instead of timing out silently first. A silent timeout during a real on-call absence is exactly the failure mode that erodes trust in the whole system.

Trade-offs and pitfalls

Stricter deduplication reduces noise but risks folding two genuinely unrelated incidents into one correlation group if the correlation key is too broad; the fix is scoping correlation to a real dependency graph, not just a time window. A common wrong turn is building one severity ladder for everything, which either pages security teams for routine downtime or under-escalates a compliance-relevant incident because it didn't look revenue-critical on the dashboard. The severity model has to be impact-type aware, not just impact-size aware.