On-Call Practices and Runbook Design Questions
Running a sustainable on-call function: rotation design, production-readiness handoffs, and authoring runbooks that let responders act quickly. Covers runbook automation, on-call culture, escalation-ready documentation, and readiness reviews before a service takes production traffic. The operational-preparedness discipline that makes incidents survivable.
During a postmortem, the incident commander singles out one engineer as the cause of the outage. How do you respond in the moment to preserve a blameless culture, without letting accountability for the fix slide?
Sample Answer
In the moment, redirect the conversation from the person to the timeline: acknowledge what was said without amplifying it, then immediately steer the group back to reconstructing what happened and why the system allowed it, while making clear that accountability for the fix is not going away.
In-the-moment response
- Interrupt with a redirect, not a confrontation. Something like: "Let's hold on names for a second and walk the timeline: what did the system show at each step?" This isn't ignoring what was said; it's refusing to let the postmortem's structure reward the blame framing by continuing down that thread.
- Reframe the specific claim into a system question. If the IC (Incident Commander, the person directing the response) says "this happened because Priya deployed without checking the dashboard," the redirect is: "So the deploy process didn't require a dashboard check before going out. Is that a gap in the checklist, or did the checklist exist and get skipped? Either answer tells us what to fix." This keeps the factual content (a deploy went out without a check) while stripping the blame framing.
- Do not let it pass silently either. Staying quiet when a peer is singled out in front of the team reads as agreement, and it's the fastest way to make the next engineer afraid to be transparent in their own postmortem. A short, calm correction in the room is better than a private word afterward, because the damage (and the culture signal) happened publicly.
- Follow up with the IC privately, separate from the room. The public redirect handles the moment; a private conversation afterward addresses the pattern, especially if this IC does it repeatedly.
Keeping accountability intact
Blameless does not mean no one owns the fix. The distinction to hold onto:
- Blame assigns fault for what already happened, to a person, and looks backward.
- Accountability assigns ownership for what happens next, to a role or system, and looks forward.
So the postmortem should still end with a named owner for each remediation item (a person, because someone has to actually do the work) and a deadline, but the framing is "you're the best person to close this gap because you understand the deploy path," not "this is your fault so you have to fix it." The action items get assigned based on who has the context and capability, independent of who gets blamed.
Worked example
During a payments-outage postmortem, the IC says: "Marcus rolled back the config and that's what caused the second outage." The redirect: "Let's look at what the rollback runbook told him to check before rolling back. Did it call out this specific config's downstream dependency?" The team pulls up the runbook and finds it didn't mention that this particular config was read by two other services; the rollback step existed, but the pre-check for downstream impact didn't. The postmortem action items become: (1) add a downstream-dependency check to the rollback runbook for this config, owned by the platform team, due in two weeks, and (2) audit other high-fanout configs for the same missing check, owned by Marcus, since he now has the clearest picture of what that gap looks like, due in one month. Marcus ends up with an action item, but it's framed as "you're positioned to close this" rather than "you caused this," and the runbook gap, not Marcus's judgment, is recorded as the finding.
Trade-offs and pitfalls
The main pitfall is overcorrecting into vagueness, where "blameless" gets used to avoid naming any specific decision point, and the postmortem ends up too soft to actually change anything; the fix is to be precise about the decision and the missing guardrail while staying impersonal about who made the decision. A related pitfall specific to this scenario: correcting an incident commander in front of the team carries real interpersonal risk if done poorly, so the redirect has to stay factual and calm rather than accusatory itself. This is a leadership-culture issue that shows up at scale too: it typically takes deliberate, sustained work, roughly a couple of quarters of consistent leadership behavior, published blameless postmortems, and visible non-punitive handling of pages, to shift a team's on-call culture away from a punitive default, and it has to be reinforced the same way every time, including in the exact moment someone in authority breaks the pattern.
How do you define severity levels for production incidents (say Sev1 through Sev4), and how does severity map to expected response time and who gets notified?
Sample Answer
Direct answer
Severity is a fixed classification of an incident's technical and business impact right now (how bad is it), and each severity tier maps to a specific acknowledgment SLA, escalation path, and notification list so the response scales automatically with how bad things are. Severity is often confused with priority: severity measures blast radius and impact, while priority additionally weighs urgency and business context, and the two usually move together but can diverge.
Structured elaboration
| Severity | Definition | Ack SLA | Who's paged | Update cadence |
|---|---|---|---|---|
| Sev1 | Full outage, data loss, or security breach affecting all or most customers | 5 minutes | Primary + secondary on-call, engineering manager, exec on-call | Every 15-30 min until resolved |
| Sev2 | Major feature broken or severe degradation for a large subset of users | 15 minutes | Primary on-call, secondary auto-paged if unacked | Every 30-60 min |
| Sev3 | Partial degradation with a workaround, or impact limited to a small subset | Next business hour | Routed to on-call as a ticket, no page | Daily until closed |
| Sev4 | Cosmetic or non-user-facing issue | Best effort | Backlog, no page | None required |
Severity vs. priority. Severity is a property of the system: what fraction of functionality is broken and for whom. Priority is a property of the response: how urgently the organization needs to act on it right now, which factors in severity plus things like contract SLAs, timing, and who is affected. A Sev2 bug (partial degradation, workaround exists) affecting one enterprise customer with a contractual one-hour response commitment can get treated with P1 urgency even though its technical severity classification stays Sev2. Conversely, a technically Sev1-caliber bug discovered in a staging-only environment has low priority because there is no live customer impact yet. Conflating the two leads to two failure modes: under-resourcing a contractually urgent-but-technically-narrow issue, or paging the whole org for something with real severity but zero current business urgency.
Worked example
Two incidents happen the same week. Incident A: the primary API returns errors for 70% of requests across all customers. That's Sev1 by impact (majority of users, core path) and P1 by urgency (acknowledge in 5 minutes, exec on-call notified). Incident B: a non-critical reporting endpoint used by one enterprise customer returns stale data. By impact alone that's Sev3 (small subset, workaround exists: refresh manually). But that customer's contract has a 30-minute response SLA for any reported defect, so it gets routed with P1 priority: acknowledged within the contract window and staffed immediately, even though the severity label on the incident stays Sev3. The postmortem for B should note this divergence explicitly, since it's exactly the kind of nuance a severity-only view misses.
Trade-offs and pitfalls
- Pitfall: over-classifying everything as Sev1 "to be safe." This burns out on-call and trains people to treat pages as noise, defeating the purpose of having tiers at all.
- Pitfall: assigning severity once at triage and never revisiting it. Initial severity is frequently wrong (scope looks narrow until the second wave of impact shows up); the postmortem should include a severity-accuracy check as a standard field.
- Pitfall: letting priority silently override severity without documenting why, which erodes trust in the severity scale over time because people start reading "severity" as "whatever got the fastest response," rather than a consistent, calibratable measure of impact.
Your organization's average time to resolve incidents has been stuck around 90 minutes for months. How would you design a program over the next couple of quarters to meaningfully bring that down, and how would you know it's actually working rather than teams gaming the metric?
Sample Answer
Sequence the work from cheap and immediate (alert routing and runbook hygiene) to structural (instrumentation, then automation) over roughly two quarters, and prove it's real improvement rather than gaming by tracking reopen rate and severity-classification consistency alongside the headline MTTR number, not just the number alone.
Where the 90 minutes actually goes
Before proposing fixes, break the baseline down into the phases every incident passes through:
| Phase | Baseline time |
|---|---|
| Detect lag | 8 min |
| Triage | 12 min |
| Diagnosis | 35 min |
| Fix | 25 min |
| Verify | 10 min |
| Total | 90 min |
The phased program
| Phase | Timeframe | What changes | Segment targeted | Effect |
|---|---|---|---|---|
| 1: Alert & runbook hygiene | Weeks 0-4 | Fix alert ownership/routing, write missing runbooks | Triage, detect lag | Triage 12 to 4 min, detect 8 to 5 min |
| 2: Instrumentation | Months 1-3 | Distributed tracing, standardized incident dashboards | Diagnosis | Diagnosis 35 to 15 min |
| 3: Automation & drills | Months 3-6 | One-click remediation scripts, automated post-fix health checks, game days | Fix, verify | Fix 25 to 12 min, verify 10 to 6 min |
Worked example: tracking the breakdown phase by phase
| Checkpoint | Total | Reduction from baseline |
|---|---|---|
| Baseline | 90 min | - |
| After Phase 1 (5+4+35+25+10) | 79 min | 12.2% |
| After Phase 2 (5+4+15+25+10) | 59 min | 34.4% |
| After Phase 3 (5+4+15+12+6) | 42 min | 53.3% |
Guarding against gaming the metric
Pair MTTR with reopen rate and a severity-distribution audit, not just the headline number. If MTTR drops but the 24-hour reopen rate rises, for example from roughly 1 reopened incident a month to roughly 4 out of the same ~40 monthly incidents, that's a sign incidents are being closed to hit the target rather than actually fixed; treat a rising reopen rate as an automatic invalidator of the MTTR win until it's addressed. Separately, audit whether the mix of logged severities shifted (more incidents suddenly classified a notch lower than before); that alone can lower average MTTR without anything getting faster.
Trade-offs and pitfalls
Automation is the highest-leverage phase but also the highest-risk one; gate it behind canary rollout and audit logging so a bad automated fix doesn't become its own incident. A program driven purely by the MTTR number invites exactly the gaming described above; always report it alongside reopen rate and incident volume, never alone. Returns are not evenly spread across the phases, and they don't simply shrink over time: Phase 2 (instrumentation) delivers the largest single incremental win, 20 of the 90 minutes, bigger than either Phase 1's 11-minute alerting and runbook fix or Phase 3's 17-minute automation gain, because Phase 2 targets diagnosis, the single largest chunk of the baseline (35 of the 90 minutes). The lesson isn't that early phases always win biggest; it's that whichever phase targets the largest remaining bottleneck wins biggest, so sequence by where the time actually goes, not just by what's cheapest to ship first.
How would you decide whether a runbook is actually ready for on-call use, not just written? What would you check before trusting it during a real incident?
Sample Answer
A runbook being written isn't the same as it being trustworthy under pressure: writing tests whether the author understood the system, while readiness tests whether someone else, half-awake at 3am, can follow it and get the right outcome. Check readiness by having someone who didn't write it actually execute it against a real (or realistic) system, not by reading it for completeness.
What to check before trusting a runbook
- Has anyone other than the author run it? A runbook the author has never handed to someone else is unverified by definition; the author's own mental model fills gaps a stranger will trip on (an assumed tool is installed, an assumed permission is already granted, a step that says "check the dashboard" without saying which one).
- Are the steps executable as written, not just described? "Restart the service" is a description; "run
systemctl restart payments-apion each of the 3 hosts listed in the service registry" is executable. If a step requires judgment the runbook doesn't supply (how do you know which hosts?), that's a gap, not an acceptable level of abstraction. - Does it state what success looks like? A remediation step without a stated verification step (what metric or log line confirms this worked) leaves the responder guessing whether to move to the next step or escalate.
- Is it safe to run when the diagnosis is wrong? Incident responders under pressure sometimes run the wrong runbook, or run the right one when the actual cause differs from what it assumes. Check whether each destructive step is reversible, and whether the runbook states a precondition to verify before acting ("only run this if X").
- Is it current? Check for an owner and a last-verified date; a runbook referencing a deprecated tool, an old cluster name, or a rotation that no longer exists is worse than no runbook, because it costs time before the responder realizes it's wrong.
How to actually verify these, not just check for their presence
- Tabletop walkthrough: someone unfamiliar with the runbook reads it aloud, step by step, without help from the author, narrating what they'd actually type or click. Gaps surface immediately as "wait, what do I do here?" moments.
- Staging or canary drill: run the actual remediation against a staging environment or a single canary instance, and time it. This catches steps that look right on paper but fail against the real system (a command with an outdated flag, a permission the on-call role doesn't actually have).
- Cold-open test: hand it to someone with zero context on this specific service (not zero context on the systems generally) and see if they can act on it in under a target time, without pinging the original author. If they can't, the runbook is only usable by the person who wrote it, which defeats the point.
Worked example
A runbook for "database replica lag alert" says: "Check replica lag, if high, failover to standby." A cold-open drill immediately exposes three gaps: no link to where replica lag is displayed, no threshold for what counts as "high" (the alert already fired, so this should already be answered, but the runbook re-asks the question), and "failover to standby" doesn't say which standby if there are multiple, or what to verify afterward to confirm the failover succeeded rather than made things worse. Fixing it: link the specific dashboard panel, state the alert's own threshold so the runbook doesn't require re-deciding it, name the failover command with the specific standby-selection logic, and add a verification step ("confirm write latency on the new primary is under 50ms and replica lag on remaining replicas is decreasing"). Re-running the cold-open drill after the fix, the same tester completes it without asking a clarifying question, which is the actual pass condition.
Trade-offs and pitfalls
Running live drills has a real cost in engineering time and, for staging drills, some risk if the environment isn't well isolated from production; the return is worth it for any runbook covering a high-severity or destructive action, and can be scaled down to tabletop-only for low-risk, easily reversible ones. A common wrong turn is treating runbook review as a documentation-quality pass (is it well written, does it have headers) rather than an execution test; a beautifully formatted runbook that's never been run by anyone but its author is still unverified.
During a P1 outage, the first responder doesn't restore service within a few minutes and doesn't acknowledge the page. Walk through what happens next: escalation timeouts, who gets paged, which channels you use, and who ultimately declares a major incident.
Sample Answer
Direct answer
Escalation has to be a timed staircase, not a hope that someone notices: each tier gets an explicit timeout, and crossing it triggers the next tier automatically rather than waiting for a human to decide to escalate. The critical branch is distinguishing "primary hasn't acknowledged at all" (a much stronger, faster signal that they're unreachable) from "primary acknowledged but hasn't fixed it yet" (a slower, more judgment-based signal), and a specific person needs to be unambiguously the incident commander once the incident crosses into major-incident territory.
Structured elaboration
flowchart TD
A[Page fires to primary] --> B{Acked within 5 min?}
B -->|No| C[Auto-escalate to secondary]
B -->|Yes| D{Mitigated or owned by 15 min?}
C --> D
D -->|No| E[Notify SRE lead and EM at 20 min]
E -->|Sev1 criteria still met at 30 min| F[IC declares Major Incident]
F --> G[Comms lead posts updates every 15 min]
| Time | Trigger | Action | Channel |
|---|---|---|---|
| T+0 | Page fires | Primary paged | PagerDuty + phone |
| T+5m | No ack from primary | Auto-escalate to secondary (unreachable-primary path) | PagerDuty + phone |
| T+15m | Acked but not mitigated or no clear owner | Secondary/backup engages as acting responder | Incident Slack channel |
| T+20m | Still unresolved | SRE lead and engineering manager notified | PagerDuty + phone |
| T+30m | Sev1 criteria still met, no imminent fix | Incident Commander formally declares a Major Incident, opens a bridge | Incident channel + status page |
Redundancy for an unreachable primary. The T+5m no-ack escalation exists specifically because "didn't acknowledge" is not the same failure mode as "acknowledged but stuck." An unreachable primary should trigger the fastest possible escalation, on multiple channels at once (page, SMS, phone call) rather than a single retry, since every minute the whole rotation is effectively uncovered.
Who declares Major Incident, and when. The role of Incident Commander should be predefined (whoever is the senior-most engaged responder at the moment of declaration, or a designated on-call IC rotation), not assigned ad hoc during the incident. Declaration criteria should be objective: customer-facing impact confirmed and no imminent fix, not a vibes call.
Audit trail. Every escalation event (page sent, ack received, tier crossed, IC declared) should be logged automatically by the paging tool with timestamps, not reconstructed from memory afterward. This is what makes the eventual postmortem's timeline accurate instead of approximate.
Worked example
A P1 outage affects payment processing. T+0: primary is paged, acknowledges within 2 minutes, and starts investigating (restart attempts, checking recent deploys). T+15m: the issue isn't mitigated and the primary flags they need help; the secondary engages and effectively becomes acting IC. T+20m: still unresolved, SRE lead and EM are notified via page and phone. T+30m: payment processing is still materially impaired with no fix in sight, so the SRE lead formally declares a Major Incident: opens an incident bridge, assigns a comms lead to post external status-page updates and an internal lead to keep driving the technical fix, and sets a 15-minute update cadence. Every one of these steps (ack time, escalation trigger, IC declaration) is logged automatically with a timestamp by the paging tool, which becomes the backbone of the postmortem timeline.
Trade-offs and pitfalls
- Pitfall: "someone will notice and escalate" with no explicit timer is the default at many small companies and reliably fails exactly when it matters most, at 3am with a skeleton crew.
- Pitfall: leaving IC as "whoever showed up" instead of a defined selection rule creates coordination ambiguity in the first minutes of a major incident, which is the worst possible time to negotiate who's in charge.
- Trade-off: the T+5m no-ack timeout should be noticeably tighter than the T+20m and T+30m escalation-for-non-progress timeouts, because "didn't acknowledge at all" is a much stronger signal of a coverage gap than "acknowledged and is still working on it." Making every tier the same length either escalates too aggressively on ordinary, in-progress incidents or too slowly on a genuinely unreachable primary.
Unlock Full Question Bank
Get access to all On-Call Practices and Runbook Design interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.