On-Call Practices and Runbook Design Questions

Readiness is a checklist, not a countdown: get access and tooling working first, have them study and sign off on the runbooks for their services, shadow several live pages, then run one supervised tabletop and one supervised live (or simulated) incident before they take a shift alone with a mentor reachable but not present.

A four-week ramp

Sign-off checklist before the first unsupervised shift

• Access confirmed end to end (paging tool, dashboards, deploy/rollback permissions) with a real test, not just "provisioned."
• Runbooks for their assigned services reviewed and any ambiguous steps flagged and fixed.
• At least one supervised tabletop and one supervised live or injected-fault incident completed.
• Mentor sign-off plus the engineer's own confidence self-assessment, not mentor judgment alone.

Extending the ramp for a complex or high-stakes service

Four weeks is often enough for a straightforward service, but for a complex hybrid-cloud system or one with many downstream dependents, add explicit competency checkpoints tied to named systems, for example certifying someone independently on the database failover path as a separate sign-off from general on-call readiness, rather than declaring them ready across the board at once. Longer term, treat the first 90 days as a structured mentorship arc rather than stopping at week four: scheduled 30/60/90-day check-ins, a second mentor pairing on a different service, and a distinct milestone for graduating from "supervised" to "primary" status rather than just a date on the calendar.

Trade-offs and pitfalls

Rushing readiness to fill a rotation gap is the most common failure mode, and it produces confident-sounding but wrong incident responses, which is worse than an obviously under-prepared response because it takes longer to catch. A checklist with no live-incident component only validates that someone can read, not that they can act under time pressure; keep at least one supervised live or simulated incident before signing off. Sign-off criteria should be service-specific rather than one generic "on-call ready" badge, since readiness on a well-instrumented service doesn't automatically transfer to a fragile legacy one with thin runbooks.

The Incident Commander (IC) owns the incident, not the fix. Their job is coordination and decision-making, deciding priorities, deciding when to escalate, keeping the response moving, while the technical work of diagnosing and fixing the problem belongs to subject matter experts (SMEs) the IC coordinates but doesn't have to be one of. That separation is the whole point: it lets someone stay focused on the shape of the response instead of getting pulled into a single technical rabbit hole.

Roles and how they differ

Why the IC role has to be distinct from the SME role

If the IC is also the person elbow-deep in a stack trace, two things suffer at once: the technical dive doesn't get their full attention, and no one is watching the overall picture (are we escalating too slowly, is communications falling behind, has severity changed). Separating the roles means the IC can pull in a second or third SME without needing to personally understand every system, and can make a call like "stop investigating, roll back now" even when an SME would rather keep digging for the root cause, because the IC's job is minimizing impact, not necessarily finding the deepest explanation in the moment.

Escalation triggers and handoffs

An IC should escalate (bring in a more senior IC, or a specific SME) when the current responder hits the edge of their authority or context: severity increasing beyond what the current team can safely own, the fix requiring a decision (like a risky rollback) above the current IC's authorization level, or the incident running long enough that fatigue is a real risk. Handoffs between ICs during a long-running incident follow the same discipline as an on-call shift handoff: the outgoing IC states current status, open decisions, and what's already been tried, in the incident channel, with the incoming IC explicitly confirming they've taken over before the outgoing IC steps back. An IC handoff that happens silently, with no explicit confirmation, is a common source of dropped context in long incidents.

Trade-offs and pitfalls

On a small team, it's tempting to skip a dedicated IC and let the most senior engineer both fix and coordinate; that works for short, simple incidents but breaks down exactly when it matters most, a complex, multi-team incident, because that's when coordination and deep technical focus can no longer be done well by the same person at once. The other common pitfall is an IC who defers every decision back to SMEs instead of actually deciding, which turns the incident into a discussion instead of a response; the IC's authority to make the call, even an imperfect one, quickly, is the actual value of the role.

A production-readiness review should verify four things before a service starts taking pages: it fails safely (degrades or rolls back instead of cascading), it's observable enough that on-call can diagnose without guessing, on-call actually knows how to respond to it, and someone specific owns it. Structure the review around those four, not a flat checklist, so gaps are obvious by category rather than buried in a long list.

What the review checks, by category

Process for running the review

1. The owning team self-certifies against the checklist first, providing evidence (load-test results, a link to the rollback runbook, a screenshot of the dashboard) rather than a checked box with no backing.
2. A reviewer outside the owning team (SRE or a peer team) spot-checks the evidence, focusing on the failure-containment and observability rows, since those are the ones teams under launch pressure are most likely to overstate.
3. Run one live-fire test before go-live: trigger the most likely failure mode in staging (or a controlled prod canary) and confirm the alert fires, the runbook's diagnosis steps actually find the cause, and the rollback works. A checklist that's never been exercised is a hypothesis, not a verified readiness state.
4. Sign-off is explicit and time-bound, not a one-time gate that's forgotten: re-review triggers on major architecture changes, not just at initial launch.

flowchart TD
  A[Owning team self-certifies checklist] --> B[Provide evidence: load tests, runbook links, dashboards]
  B --> C[Outside reviewer spot-checks evidence]
  C --> D{Gaps found?}
  D -->|Yes| E[Team remediates gap]
  E --> C
  D -->|No| F[Live-fire test in staging or canary]
  F --> G{Alert fires, runbook works, rollback succeeds?}
  G -->|No| E
  G -->|Yes| H[Sign-off: service takes pages]

Worked example

A new recommendations service is going live. Self-certification claims load testing was done "at expected traffic." The outside reviewer asks for the actual load-test report and finds it tested at the current expected peak (500 req/s) with no margin, while the service also sits behind a feature flag that product plans to ramp to three times that within a month. That's a real gap: the review isn't asking for perfection, but it should require either testing at the higher number now or an explicit, documented plan (with an owner and date) to re-test before the ramp, rather than letting "tested at expected traffic" silently mean "tested at today's traffic." The live-fire test then finds the circuit breaker on the downstream recommendation-model call has no timeout configured, so a slow model response would hang the request instead of failing fast; that's flagged as a blocking issue, not a follow-up ticket, because it directly causes cascading failure under exactly the load condition the service is meant to handle.

Trade-offs and pitfalls

The live-fire test step is the one teams most often skip under launch deadline pressure, and it's also the one that catches the gaps self-certification checklists miss (an alert that's configured but never actually fires, a runbook step that references a dashboard that doesn't exist); treat it as non-negotiable for anything customer-facing, and reserve a lighter self-certification-only path for low-risk internal services. A common wrong turn is treating the checklist as complete once every box is checked, without weighting which gaps are load-bearing; a missing rollback plan for a payments-adjacent service is not the same severity as a missing dashboard for an internal admin tool, and the review should say so explicitly rather than gating everything equally.

Direct answer

A good handoff transfers three things: current state (what's broken or at risk right now), context (what's already been tried and what's scheduled), and ownership (who is accountable for what next), and it has to be verifiable rather than a status dump, meaning the incoming engineer confirms they can actually act (they have access, they can reproduce the symptom, they understand the next step) before the outgoing engineer is done.

Structured elaboration

Handoff checklist, in order:

1. Status snapshot: current owner and contact, "all green" or a one-line count of active incidents/unstable services.
2. Active incidents: for each, severity, start time, impact, current owner, and a link to the ticket, not a re-explanation from scratch.
3. Recent alert history: what's fired in the last several hours, and specifically any alert that's been flapping, since that's exactly what an incoming responder will misdiagnose as new if nobody flags it.
4. Ongoing mitigations and runbook links: what's been tried, what's blocked, and the specific next action, with a link to the runbook rather than a paraphrase of it.
5. Scheduled changes: upcoming deploys, migrations, or maintenance windows during the next shift, with rollback plans linked.
6. Degraded-but-not-incident services: anything running hot or close to a threshold that isn't paging yet.
7. Access and tooling: pager rotation, incident channel, dashboards, runbook repo, and an explicit note if the incoming engineer is missing access to any of them.
8. Explicit confirmation, not implied: "Can you access the dashboards I linked? Can you reproduce the symptom in incident #123? Do you agree you own the DB migration follow-up?" Each gets an actual yes, not a thumbs-up emoji on a wall of text.

Making it lightweight (chatops mechanics). For day-to-day handoffs without an active incident, a short structured chat message beats a long document nobody reads: header (shift window, owner, escalation contact), one-line status, action items with owners, upcoming risks, and links. Anything in that message that turns out to matter beyond the shift boundary (a workaround that becomes permanent, a gotcha that will recur) gets tagged for follow-up and folded into the actual runbook within a day or two, so the team's durable documentation doesn't quietly live and die in chat history.

Automating the tedious part. The status snapshot, active-incident list, and recent-alert-history sections don't need to be typed by hand: a handoff template can be pre-populated from the monitoring and paging systems (current alert state, open incident IDs, last-deploy timestamp) so the outgoing engineer is editing and confirming pre-filled facts rather than writing a report from a blank page. That reduces both the time cost of handoff and the chance that something gets left out because the outgoing engineer forgot it existed.

Worked example

Friday, 6pm, end of a shift. One active Sev2 incident (checkout latency degraded for a subset of EU traffic, mitigation in progress: a feature flag was flipped to route around a slow dependency, error rate has dropped but root cause isn't fixed), and a database migration scheduled for 2am that night. The outgoing engineer posts:

Handoff | Fri 18:00-Sat 02:00 UTC
Owner: @outgoing -> @incoming | Escalation: @oncall-lead
Status: Degraded (checkout latency, EU) -> incident #482, mitigated not resolved
Action items:
 - Watch checkout error rate; if it climbs above 2% again, re-check the feature flag is still on
 - DB migration at 02:00 UTC (runbook: <link>, rollback: <link>) - I'll be asleep, this is yours
Risks: migration touches the same table implicated in incident #482; if latency spikes right after, check the migration first
Links: <dashboard> <incident #482> <migration runbook>

The incoming engineer confirms: dashboard access works, they can see incident #482's current state, and they explicitly acknowledge owning the migration watch. That confirmation, not the message itself, is what makes the handoff complete.

Trade-offs and pitfalls

• Pitfall: a "read the ticket" handoff with no verification step lets the incoming responder discover gaps at 3am instead of at 6pm when the person with context is still reachable.
• Pitfall: too much ceremony (a mandatory 45-minute call every single handoff) burns out the outgoing engineer and makes people avoid going on-call at all; reserve synchronous overlap for when there's an active Sev1/Sev2, not as the default for a quiet shift.
• Trade-off: synchronous handoff transfers tacit knowledge best but costs both people's time; async structured notes are cheaper but only as good as their template and discipline. The right default is async-by-default, with synchronous overlap triggered automatically whenever an incident is still open at shift boundary.

No single number proves burnout. Track three families of signals together: raw load (pages per person per week), response burden (after-hours percentage, time-to-resolve), and human signals (fatigue self-reports, PTO usage), and watch for the same people repeatedly crossing thresholds across categories, not just one bad week.

What to track

Worked example: reading a four-week rotation block

Suppose the primary on-call received a combined 88 pages across the last 4-week rotation block (one week per engineer).

Against the roughly-under-10/week guideline, 22 pages/week is more than double, a sustainability flag on its own. If 39 of those 88 pages fired between 20:00 and 08:00:

well above the roughly-25% guideline, corroborating that this isn't just a high-volume rotation, it's specifically disrupting sleep.

Trade-offs and pitfalls

These metrics can be gamed by suppressing alerts; pair volume metrics with an independent audit (a sampled review of closed incidents) so under-alerting doesn't masquerade as improvement. Self-reported fatigue data is noisy and subject to survey fatigue itself; treat it as a leading indicator alongside hard metrics, not as the sole trigger for action. A single bad week (one major outage) will spike every metric at once; look for a sustained pattern across at least a full rotation cycle before concluding the rotation itself, rather than the incident, is the problem.