On-Call Practices and Runbook Design Questions

A runbook being written isn't the same as it being trustworthy under pressure: writing tests whether the author understood the system, while readiness tests whether someone else, half-awake at 3am, can follow it and get the right outcome. Check readiness by having someone who didn't write it actually execute it against a real (or realistic) system, not by reading it for completeness.

What to check before trusting a runbook

• Has anyone other than the author run it? A runbook the author has never handed to someone else is unverified by definition; the author's own mental model fills gaps a stranger will trip on (an assumed tool is installed, an assumed permission is already granted, a step that says "check the dashboard" without saying which one).
• Are the steps executable as written, not just described? "Restart the service" is a description; "run systemctl restart payments-api on each of the 3 hosts listed in the service registry" is executable. If a step requires judgment the runbook doesn't supply (how do you know which hosts?), that's a gap, not an acceptable level of abstraction.
• Does it state what success looks like? A remediation step without a stated verification step (what metric or log line confirms this worked) leaves the responder guessing whether to move to the next step or escalate.
• Is it safe to run when the diagnosis is wrong? Incident responders under pressure sometimes run the wrong runbook, or run the right one when the actual cause differs from what it assumes. Check whether each destructive step is reversible, and whether the runbook states a precondition to verify before acting ("only run this if X").
• Is it current? Check for an owner and a last-verified date; a runbook referencing a deprecated tool, an old cluster name, or a rotation that no longer exists is worse than no runbook, because it costs time before the responder realizes it's wrong.

How to actually verify these, not just check for their presence

1. Tabletop walkthrough: someone unfamiliar with the runbook reads it aloud, step by step, without help from the author, narrating what they'd actually type or click. Gaps surface immediately as "wait, what do I do here?" moments.
2. Staging or canary drill: run the actual remediation against a staging environment or a single canary instance, and time it. This catches steps that look right on paper but fail against the real system (a command with an outdated flag, a permission the on-call role doesn't actually have).
3. Cold-open test: hand it to someone with zero context on this specific service (not zero context on the systems generally) and see if they can act on it in under a target time, without pinging the original author. If they can't, the runbook is only usable by the person who wrote it, which defeats the point.

Worked example

A runbook for "database replica lag alert" says: "Check replica lag, if high, failover to standby." A cold-open drill immediately exposes three gaps: no link to where replica lag is displayed, no threshold for what counts as "high" (the alert already fired, so this should already be answered, but the runbook re-asks the question), and "failover to standby" doesn't say which standby if there are multiple, or what to verify afterward to confirm the failover succeeded rather than made things worse. Fixing it: link the specific dashboard panel, state the alert's own threshold so the runbook doesn't require re-deciding it, name the failover command with the specific standby-selection logic, and add a verification step ("confirm write latency on the new primary is under 50ms and replica lag on remaining replicas is decreasing"). Re-running the cold-open drill after the fix, the same tester completes it without asking a clarifying question, which is the actual pass condition.

Trade-offs and pitfalls

Running live drills has a real cost in engineering time and, for staging drills, some risk if the environment isn't well isolated from production; the return is worth it for any runbook covering a high-severity or destructive action, and can be scaled down to tabletop-only for low-risk, easily reversible ones. A common wrong turn is treating runbook review as a documentation-quality pass (is it well written, does it have headers) rather than an execution test; a beautifully formatted runbook that's never been run by anyone but its author is still unverified.

When the same incident keeps recurring despite repeated fixes, the prior RCAs were almost certainly treating a symptom as the root cause; run this RCA by explicitly listing every prior "fix" and asking why each one didn't hold, because the pattern across failed fixes usually points straight at the real systemic gap.

Framework for a systemic RCA

1. Build a fix history first, before investigating the current occurrence. For each prior incident of this same recurring issue: what was diagnosed as the cause, what was changed, and did the change actually address that diagnosis or just the immediate symptom? A pattern of "different symptom fixed each time, same underlying trigger every time" is the tell that root cause was never actually found.
2. Separate the trigger from the vulnerability. The trigger (a specific deploy, a specific load pattern, a specific external dependency hiccup) may vary each month, but if the same class of trigger keeps causing an outage, the system has a standing vulnerability to that trigger class that no single fix removed. The RCA's job is to name the vulnerability, not just the latest trigger.
3. Use the fishbone categories (code, config, infrastructure, process, tooling) to check whether every prior fix landed in the same category. If four consecutive fixes were all code patches but the incident keeps returning, that's evidence the real gap is in process or tooling (no regression test for this class of failure, no canary catching it before full rollout) rather than in any specific line of code.
4. Test the systemic hypothesis, don't just assert it. If the hypothesis is "the nightly batch job and the backup window contend for the same database connection pool," verify by reproducing that contention in a controlled environment (isolated run of the batch job during a simulated backup window), not by pattern-matching from the incident timeline alone.

Worked example

A nightly batch job has caused three partial outages in three consecutive months. Prior fixes: month 1, increased the job's timeout (fix addressed "job was timing out"); month 2, added a retry with backoff (fix addressed "job failed transiently"); month 3 is the current incident, and the job is again failing, this time differently, a connection pool exhaustion error. Building the fix history shows a pattern: every fix targeted why the job failed on that specific night, and none asked why the job's failure mode changes every month while the timing (always during the nightly backup window) stays constant. Testing the systemic hypothesis, that the batch job and backup process share a connection pool and the backup's duration has been slowly growing as data volume grows, month-over-month backup duration logs confirm the backup window has grown from roughly 12 minutes to 40 minutes over the quarter, now overlapping the batch job's peak connection usage. That is the systemic cause: the job and backup were never intentionally isolated, and it was invisible for months because the backup was short enough not to overlap.
The durable fix follows from the systemic cause, not the latest symptom: allocate the batch job a dedicated connection pool separate from ad hoc processes, and alert on backup-duration trend (not just backup failure) so a slowly growing resource conflict is visible before it causes an outage again.

Trade-offs and pitfalls

The main pitfall is that a systemic RCA takes longer and produces a less satisfying immediate answer than "here's the line that broke," which creates pressure to ship another symptom-level fix under time pressure; the way to resist that is to make the fix-history review a required first step, not an optional deep-dive, so the systemic question gets asked before anyone commits to a scope. A real trade-off: broadening the RCA to a process or tooling gap usually means the fix is slower to land (new alerting, resource isolation, a regression test suite) than a code patch, so it's worth explicitly stating in the postmortem that a fast interim mitigation (in this example, a manual connection-pool bump) is being paired with the slower structural fix, rather than letting the slow fix block any near-term relief.

Direct answer

My rule is: an alert pages me only if it's actionable, urgent, and real, meaning there's something I can actually do about it, it needs a response within minutes rather than hours, and it reflects genuine or imminent customer/business impact. Anything that fails one of those three tests goes to Slack or a ticket, not my phone. I protect my own sustainability by treating a repeatedly noisy page as a bug in the alert that needs fixing, not as a toughness test I'm supposed to pass.

Structured elaboration

The paging bar, in practice. Before an alert is allowed to page a human overnight, it should answer yes to all three:

• Actionable: is there a specific thing a person can do right now, or does it just need to be visible on a dashboard?
• Urgent: does waiting until business hours meaningfully worsen the outcome?
• Real: does it correlate with actual customer or business impact, not just an internal metric that moves for benign reasons?

Anything that's informational, non-urgent, or has historically resolved itself before a human could act belongs in Slack or a ticket queue, not a page.

Boundaries I set for myself and expect from a team:

• Rotation limits: a cap on consecutive on-call weeks and mandatory rest between rotations, so on-call load is a scheduling property of the team, not a matter of individual endurance.
• Compensation for the bad weeks: comp time or a stipend that scales with how bad the week actually was, so a rotation with three overnight Sev1s isn't treated the same as a quiet one.
• A standing agreement that any alert paging the same person more than once or twice without a code change in response gets flagged as noise to fix, not accepted as the cost of doing business.
• Tracking my own load: if my pages-per-week start trending up, that's something I bring to the team, not something I quietly absorb, because sustainable on-call is a team-level property (rotation depth, alert hygiene) and not an individual willpower contest.

Worked example

In a recent on-call rotation, an alert on a background job queue started paging me nightly around 2am even though the queue reliably drained on its own within a few minutes every time. I didn't just mute it and move on; I posted the pattern in our team channel with the timestamps and outcomes for the last several nights (fires, self-resolves, no human action taken each time), and we agreed together that the alert needed a sustained-duration requirement before it could page overnight, since a brief backlog wasn't actually urgent or actionable at 2am. I made the config change, had a teammate review it since it affected everyone's rotation, and rolled it out. The nightly pages for that specific alert stopped after the change. The underlying job still occasionally backed up during traffic spikes, but that now showed up as a daytime ticket for the team to investigate instead of an overnight page for whoever happened to be on call.

Trade-offs and pitfalls

• Pitfall: "I'm always reachable, page me for anything" sounds committed but actually signals a lack of judgment about what deserves urgency, and it's a fast path to burnout that eventually degrades response quality for the alerts that really matter.
• Pitfall: the opposite extreme, "I basically never get paged after hours," can mean either genuinely excellent alert hygiene or quietly under-covered risk; the honest answer distinguishes which one it is with a concrete example, not just an assertion.
• Trade-off: a team that pays well for on-call (stipends, real comp time) can sustain a slightly higher page volume than one that doesn't, but that changes how much noise is tolerable, not whether the actionable/urgent/real bar applies. A well-compensated but chronically noisy alert is still a bug to fix, not a cost center to accept.

The line is intent, not tooling: a runbook assumes the failure isn't malicious and defaults to fast, reversible operational actions, while a security playbook assumes it might be and defaults to preserving evidence and escalating. The handoff between them has to be a written, specific trigger list, not a judgment call made under pressure.

Runbook versus security playbook

Handoff triggers (explicit, not a judgment call)

• A known indicator-of-compromise match (signature, known-bad IP, tampered binary).
• Automated remediation failing repeatedly while a new suspicious artifact appears.
• Any sign of lateral movement or a cross-service authentication anomaly.
• Any alert crossing a predefined confidence-plus-impact threshold agreed with security in advance.

Handoff flow

flowchart TD
  A[Runbook step in progress] --> B{Indicator of compromise?}
  B -->|No| C[Continue runbook as operational issue]
  B -->|Yes| D[Isolate and preserve evidence, non-destructive]
  D --> E[Open incident ticket with artifacts]
  E --> F[Notify security on-call, SLA clock starts]
  F --> G[Security triages within SLA]
  G --> H[Security leads containment and investigation]
  H --> I[SRE executes infra changes under security direction]

Keeping a live credential out of the runbook in the first place

The mechanism isn't "tell people not to paste secrets," it's removing the need to. Runbooks reference secret IDs resolved from a vault at execution time, the same underlying mechanism used to control access to any sensitive step, so there's rarely a reason to type a raw value into the document at all. For the cases where a responder is genuinely debugging and tempted to paste a live value for reference, a secret-scanning check on the runbook-editing tool (pattern and entropy-based) that blocks the save and asks for a redacted reference instead is a cheaper and more reliable control than relying on discipline during an incident.

Trade-offs and pitfalls

Treating every operational failure as a potential security incident slows down the vast majority of pages that are genuinely just a bad deploy; keep the trigger list narrow and specific rather than "when in doubt, escalate." The more dangerous direction is the opposite: treating every alert as operational until proven otherwise is how evidence gets destroyed by a well-meaning restart before anyone realizes it mattered, so the trigger list has to be actively taught, not just written down somewhere. Keeping runbooks and playbooks in separate systems owned by separate teams is how the handoff criteria silently drift out of sync with what each team actually expects; version-control both in one place with cross-references and joint review.

A credible automation case has three parts: the annual cost of doing the task manually (engineer time plus any revenue or SLA impact), the fully loaded cost of building and maintaining the automation, and a residual-risk line for when the automation itself misfires. Turn those into a payback period and an annual ROI, then restate the same numbers for a non-technical audience as "what we spend today versus a one-time investment that pays for itself in N months."

Inputs the model needs

The formula

Worked example: a noisy alert that fires 300 times a year

Assumptions (pinned): F = 300/year, H = 0.4 hours, C = $75/hr, R = $100/incident, D = $20,000, N = 2 years, M = $4,000/year, p = 2%, C_f = $175/misfire.

Translating it for a non-technical stakeholder

The pitch a CFO or exec sponsor needs is the same numbers, stripped of the formula: "this task currently costs the team about $39k a year in engineer time and SLA impact. A $20k investment, amortized over two years plus $4k a year to keep it working, pays for itself in about ten months and then keeps saving roughly $24k every year after that, with a 2% chance any given automated run still needs a human to clean up after it." Leading with the payback date and the residual-risk number, not just the savings, is what makes the pitch credible instead of salesy.

Trade-offs and pitfalls

If F is seasonal or unstable, a single annual estimate is misleading; revisit the model quarterly rather than trusting a number computed once. Don't drop the residual-failure term to make the case look better: an automation that works 98% of the time but corrupts state on the other 2% can be a net negative if C_f is large, especially for destructive actions. A common wrong turn is pitching automation using only "engineer-hours saved" and skipping maintenance and residual-risk costs entirely; that consistently overstates ROI and makes the next proposal harder to trust once someone notices the gap.