On-Call Practices and Runbook Design Questions

Direct answer

Action items get done when they're tracked in the same system and cadence as regular engineering work rather than in a postmortem doc nobody revisits, each has exactly one named owner and a due date, and a recurring, lightweight review surfaces overdue items instead of letting them go quiet. Lessons reach teams that didn't experience the incident when postmortems are indexed centrally by system and failure pattern, not just filed per-team, and a short summary is actively pushed to adjacent teams rather than waiting for someone to search the archive.

Structured elaboration

Getting action items done

Categorize each root cause as process, technical, or people and communication before assigning it. A technical fix routes to the engineering backlog with the same prioritization as other work; a process fix might mean updating a runbook or a review gate; making this distinction up front stops "add more monitoring" from becoming the default answer to everything. Each item gets one owner, one due date, and one acceptance criterion, since items without a single owner reliably don't get done. A weekly or sprint-cadence review of all open incident action items across teams escalates anything overdue to the owning manager, not just the assignee, and the remediation deadline itself scales with the severity of the incident that produced it.

Spreading lessons beyond the team that had the incident

A central, searchable postmortem index tagged by system, failure mode, and root-cause category, not filed only under the owning team's folder, makes cross-cutting patterns findable. A short, standardized summary, what broke, why, what changed, gets pushed to a cross-team channel or a recurring digest rather than relying on other teams to search for it. If the same root-cause category, such as "connection-pool exhaustion" or "no alert on saturation," shows up across multiple postmortems, that's a systemic gap worth its own initiative, and it only becomes visible if postmortems are tagged consistently enough to query across them.

Worked example

A quarter's incident action-item tracker opens 12 action items from postmortems and closes 9 of them within their assigned deadline by end of quarter, a $9/12 = 0.75$, or 75 percent, on-time closure rate. The weekly review flags the 3 still-open items each week; two turn out to be cross-team items stuck on a dependency from a third team that was never explicitly assigned ownership of its part. That gap itself becomes a new process action item: cross-team action items need an explicit dependency owner, not just an assignee, added to the next postmortem review template.

Trade-offs and pitfalls

• Tracking action items only in the postmortem document itself means they compete for attention against the team's regular backlog and usually lose; they need to live in the same system as everything else.
• Broadcasting every postmortem summary to the whole organization trains people to ignore the channel; scope the push to teams that own adjacent or dependent systems, and keep a searchable index for everyone else.
• A strict remediation deadline drives closure but can pressure a team into a shallow fix just to hit the date; pairing the deadline with an explicit acceptance criterion is what keeps "closed" meaning "verified" rather than just "closed."

Compensation and schedule design are two separate levers, and both need to move: pay a base standby stipend for availability, add per-incident pay or time-off-in-lieu for the work actually done, and size the rotation and rest guarantees so the schedule itself isn't relying on money to make an unsustainable load tolerable.

Compensation model components

Worked example: one on-call week

Stipend $250 + 6 hours of actual incident work at $40/hr:

$$\text{Pay} = 250 + (6 \times 40) = 250 + 240 = \$490$$

If the same 6 hours bank as TOIL at a 1.5x rate for after-hours work:

$$\text{TOIL banked} = 6 \times 1.5 = 9 \text{ hours}$$

Schedule practices that reduce the load pay has to compensate for

Primary/secondary tiers so one page doesn't always land on the same person; a cap on consecutive on-call weeks per engineer; shorter rotations (fewer consecutive days of stress, more handoffs) traded against longer rotations (fewer handoffs, more concentrated fatigue), sized to team headcount rather than picked arbitrarily; and follow-the-sun coverage once the team is large and distributed enough to make timezone handoffs cheaper than overnight pages.

Trade-offs and pitfalls

Per-incident pay can invite gaming in both directions, either padding logged hours or under-reporting to avoid looking like a "high maintenance" service; review incident-hour claims against the paging log rather than trusting self-reports alone. Contractors and salaried employees often need different structures (cash versus TOIL), and a single model rarely fits both cleanly. Regional labor law varies significantly, some jurisdictions treat standby time itself as compensable working time, so confirm with legal or HR before setting a global policy rather than assuming one region's rules generalize. There's no dominant answer on rotation length; it has to be sized to team size and incident frequency, not copied from another team's policy.

No single number proves burnout. Track three families of signals together: raw load (pages per person per week), response burden (after-hours percentage, time-to-resolve), and human signals (fatigue self-reports, PTO usage), and watch for the same people repeatedly crossing thresholds across categories, not just one bad week.

What to track

Worked example: reading a four-week rotation block

Suppose the primary on-call received a combined 88 pages across the last 4-week rotation block (one week per engineer).

$$\text{Pages per on-call week} = \frac{88}{4} = 22 \text{ pages/week} \approx 3.1/\text{day}$$

Against the roughly-under-10/week guideline, 22 pages/week is more than double, a sustainability flag on its own. If 39 of those 88 pages fired between 20:00 and 08:00:

$$\text{After-hours share} = \frac{39}{88} \times 100 \approx 44.3\%$$

well above the roughly-25% guideline, corroborating that this isn't just a high-volume rotation, it's specifically disrupting sleep.

Trade-offs and pitfalls

These metrics can be gamed by suppressing alerts; pair volume metrics with an independent audit (a sampled review of closed incidents) so under-alerting doesn't masquerade as improvement. Self-reported fatigue data is noisy and subject to survey fatigue itself; treat it as a leading indicator alongside hard metrics, not as the sole trigger for action. A single bad week (one major outage) will spike every metric at once; look for a sustained pattern across at least a full rotation cycle before concluding the rotation itself, rather than the incident, is the problem.

A strong answer to this picks one real incident, states plainly what your specific role was (not "the team"), walks through how status was communicated to the people who needed it, and ends with a concrete process change that came out of the postmortem, not just a lesson learned in the abstract. The bar an interviewer is checking is whether you can drive coordination under pressure and whether the org actually got better afterward, not whether the incident itself was dramatic.

Structuring the answer

• Situation: what broke, which teams were involved, and why it needed cross-team coordination rather than one person fixing it alone.
• Task: your specific role, on-call engineer, incident commander, SME being pulled in, and what you were responsible for.
• Action: what you actually did, especially the communication piece: how often you posted updates, who you looped in and when, how you decided what to escalate.
• Result: how the incident resolved, and the one concrete thing that changed afterward as a direct result of the postmortem.

Worked example (illustrative story skeleton)

Situation: I was on-call when a payment-processing service started failing a meaningful share of transactions during a peak traffic window, and it quickly became clear the cause spanned two teams, ours and the database team, not something either of us could fix alone.

Task: As the on-call engineer, my first job was triage and coordination, not fixing it single-handedly, since the fix needed the database team's context on a recent schema change.

Action: I opened a dedicated incident channel, pulled in the database on-call and our service owner, and posted a short status update on a fixed cadence so people weren't pinging me individually for progress. We traced the failures to a missing index from a recent migration, coordinated a rollback with the database team while I kept support and the affected team's manager updated with plain-language impact summaries (not the technical detail I was sharing in the incident channel).

Result: The rollback resolved the failure rate back to its normal baseline within the same on-call shift. In the blameless postmortem, the concrete change we made was adding a required index-verification check to the deploy pipeline for any migration, so a similar schema change can't ship without it being caught automatically rather than during a live incident.

Variant: when the story is about a communication breakdown

The strongest version of this question for a communication-focused answer isn't one where everything went smoothly, it's one where something specific broke in how status was shared, and what changed as a result. For example: during a similar incident, the customer-support team found out about the outage from customers before the incident channel posted a status update, because the communications role wasn't assigned and everyone assumed someone else was posting it. The fix that came out of that postmortem wasn't "communicate better", it was specific: the incident-declaration checklist now requires naming a communications owner in the first two minutes, before technical triage even starts, so that role is never implicitly assumed.

Trade-offs and pitfalls

The most common mistake is describing only the technical fix and skipping the coordination and communication details entirely, which is exactly the part this question is testing for. A close second is ending on "we learned to communicate better" without naming the specific process change that came out of it; a blameless postmortem is only as good as the concrete, trackable action item it produces, and an answer that mirrors that (a specific fix, not a vague lesson) is what separates a senior-sounding story from a generic one.

Direct answer

A runbook is a fixed set of step-by-step instructions for a known failure mode: it tells you exactly what to type. A playbook is a decision framework for handling an incident more broadly: it tells you how to figure out what to do, who needs to be involved, and which runbook to reach for. Reach for a runbook when you already know the cause and the fix is mechanical; reach for a playbook when you're still diagnosing, coordinating multiple people, or the right response depends on judgment.

Structured elaboration

Minimum fields for each:

• Runbook: title, service, owner, trigger/precondition, required permissions and tools, exact step-by-step commands, verification steps, rollback steps, expected impact, last-reviewed date.
• Playbook: title, scope and severity thresholds, incident commander and stakeholder roles, the decision tree itself, links to the relevant runbooks, communication templates, escalation matrix, last-reviewed date.

Worked example

A database replica's lag exceeds a threshold: this is a runbook. It lists the exact commands to promote a replica, the steps to reconfigure the application's read preference, verification queries to confirm the fix, and the rollback commands if the promotion causes a new problem. Now compare that to a major outage affecting payments: this is a playbook. It guides the incident commander through detecting the actual scope, declaring severity, deciding between routing traffic to a fallback payment path versus draining traffic entirely, coordinating the app, infra, and comms teams, and linking out to the specific runbooks (including the replica-promotion one, if that turns out to be the fix) for whichever technical action the decision tree leads to.

Trade-offs and pitfalls

• Pitfall: writing a "runbook" for something that actually needs judgment, for example "when in doubt, restart the service." That hides a decision a playbook should make explicit, and someone follows it verbatim during exactly the incident it doesn't fit.
• Pitfall: letting a playbook go stale is worse than letting one runbook go stale, because the playbook is what everyone reaches for first during ambiguity; a stale escalation matrix (wrong names or numbers) breaks the whole response, not just one specific fix.
• Trade-off: automating a runbook into a one-click execution is great for high-confidence, low-blast-radius fixes (restarting a stateless service) and dangerous for high-blast-radius ones (promoting a database replica). The more damage a wrong click can do, the more the runbook should require an explicit human confirmation step before executing, not less.
• Both belong in a versioned, reviewed repository rather than an unowned wiki page, with periodic review, and runbooks specifically benefit from occasional dry-run or game-day testing to confirm the steps still work against the current system rather than an outdated one.