On-Call Practices and Runbook Design Questions

A runbook being written isn't the same as it being trustworthy under pressure: writing tests whether the author understood the system, while readiness tests whether someone else, half-awake at 3am, can follow it and get the right outcome. Check readiness by having someone who didn't write it actually execute it against a real (or realistic) system, not by reading it for completeness.

What to check before trusting a runbook

• Has anyone other than the author run it? A runbook the author has never handed to someone else is unverified by definition; the author's own mental model fills gaps a stranger will trip on (an assumed tool is installed, an assumed permission is already granted, a step that says "check the dashboard" without saying which one).
• Are the steps executable as written, not just described? "Restart the service" is a description; "run systemctl restart payments-api on each of the 3 hosts listed in the service registry" is executable. If a step requires judgment the runbook doesn't supply (how do you know which hosts?), that's a gap, not an acceptable level of abstraction.
• Does it state what success looks like? A remediation step without a stated verification step (what metric or log line confirms this worked) leaves the responder guessing whether to move to the next step or escalate.
• Is it safe to run when the diagnosis is wrong? Incident responders under pressure sometimes run the wrong runbook, or run the right one when the actual cause differs from what it assumes. Check whether each destructive step is reversible, and whether the runbook states a precondition to verify before acting ("only run this if X").
• Is it current? Check for an owner and a last-verified date; a runbook referencing a deprecated tool, an old cluster name, or a rotation that no longer exists is worse than no runbook, because it costs time before the responder realizes it's wrong.

How to actually verify these, not just check for their presence

1. Tabletop walkthrough: someone unfamiliar with the runbook reads it aloud, step by step, without help from the author, narrating what they'd actually type or click. Gaps surface immediately as "wait, what do I do here?" moments.
2. Staging or canary drill: run the actual remediation against a staging environment or a single canary instance, and time it. This catches steps that look right on paper but fail against the real system (a command with an outdated flag, a permission the on-call role doesn't actually have).
3. Cold-open test: hand it to someone with zero context on this specific service (not zero context on the systems generally) and see if they can act on it in under a target time, without pinging the original author. If they can't, the runbook is only usable by the person who wrote it, which defeats the point.

Worked example

A runbook for "database replica lag alert" says: "Check replica lag, if high, failover to standby." A cold-open drill immediately exposes three gaps: no link to where replica lag is displayed, no threshold for what counts as "high" (the alert already fired, so this should already be answered, but the runbook re-asks the question), and "failover to standby" doesn't say which standby if there are multiple, or what to verify afterward to confirm the failover succeeded rather than made things worse. Fixing it: link the specific dashboard panel, state the alert's own threshold so the runbook doesn't require re-deciding it, name the failover command with the specific standby-selection logic, and add a verification step ("confirm write latency on the new primary is under 50ms and replica lag on remaining replicas is decreasing"). Re-running the cold-open drill after the fix, the same tester completes it without asking a clarifying question, which is the actual pass condition.

Trade-offs and pitfalls

Running live drills has a real cost in engineering time and, for staging drills, some risk if the environment isn't well isolated from production; the return is worth it for any runbook covering a high-severity or destructive action, and can be scaled down to tabletop-only for low-risk, easily reversible ones. A common wrong turn is treating runbook review as a documentation-quality pass (is it well written, does it have headers) rather than an execution test; a beautifully formatted runbook that's never been run by anyone but its author is still unverified.

Readiness is a checklist, not a countdown: get access and tooling working first, have them study and sign off on the runbooks for their services, shadow several live pages, then run one supervised tabletop and one supervised live (or simulated) incident before they take a shift alone with a mentor reachable but not present.

A four-week ramp

Sign-off checklist before the first unsupervised shift

• Access confirmed end to end (paging tool, dashboards, deploy/rollback permissions) with a real test, not just "provisioned."
• Runbooks for their assigned services reviewed and any ambiguous steps flagged and fixed.
• At least one supervised tabletop and one supervised live or injected-fault incident completed.
• Mentor sign-off plus the engineer's own confidence self-assessment, not mentor judgment alone.

Extending the ramp for a complex or high-stakes service

Four weeks is often enough for a straightforward service, but for a complex hybrid-cloud system or one with many downstream dependents, add explicit competency checkpoints tied to named systems, for example certifying someone independently on the database failover path as a separate sign-off from general on-call readiness, rather than declaring them ready across the board at once. Longer term, treat the first 90 days as a structured mentorship arc rather than stopping at week four: scheduled 30/60/90-day check-ins, a second mentor pairing on a different service, and a distinct milestone for graduating from "supervised" to "primary" status rather than just a date on the calendar.

Trade-offs and pitfalls

Rushing readiness to fill a rotation gap is the most common failure mode, and it produces confident-sounding but wrong incident responses, which is worse than an obviously under-prepared response because it takes longer to catch. A checklist with no live-incident component only validates that someone can read, not that they can act under time pressure; keep at least one supervised live or simulated incident before signing off. Sign-off criteria should be service-specific rather than one generic "on-call ready" badge, since readiness on a well-instrumented service doesn't automatically transfer to a fragile legacy one with thin runbooks.

Don't gate sensitive steps behind standing credentials a responder already holds. Gate them behind short-lived, narrowly scoped credentials issued by the runbook orchestrator at the moment of use, with a pre-authorized fast path for the highest severities so speed during a real incident doesn't require quietly bypassing the audit trail.

Comparing access models

Worked example: sizing the credential TTL

If the median observed time to complete a given remediation step across past incidents is 12 minutes, a 15-minute TTL leaves almost no margin:

$$\text{Margin} = 15 - 12 = 3 \text{ min}$$

a responder who hits a snag is interrupted 3 minutes short of done, exactly when stopping is most disruptive. A TTL of roughly 20 minutes, the median plus a working buffer rather than an open-ended grant, gives room to finish without leaving a long-lived credential outstanding. A 4-hour TTL "to be safe" instead means a credential compromised from a responder's terminal during that window stays valid for the rest of the shift; that's the trade being made for the extra convenience.

Keeping it auditable without slowing the responder down

• Runbooks reference secret IDs, never raw values, so the document itself is safe to read even if it leaks.
• The orchestrator executes the sensitive step server-side where practical, so the responder never sees the decrypted secret at all, only the outcome.
• Every credential issuance logs identity, ticket or incident ID, scope, and TTL to an immutable log, correlated automatically rather than reconstructed after the fact.
• The highest severities get pre-authorized issuance, no waiting on a human approver, precisely because the TTL and logging, not a manual gate, are what keep it auditable.

Trade-offs and pitfalls

A break-glass path needs more audit rigor than the normal path, not less; pair any emergency bypass with mandatory post-incident review and automatic rotation of whatever it touched. Auto-approval for the highest severities removes a human gate exactly during the highest-risk window (a real incident, adrenaline, and possibly an actor exploiting the chaos), so the TTL and logging have to carry that weight instead. Orchestrator-executed remediation is safer for the responder but adds its own risk surface; the automation itself now needs the same change-review rigor as production code, not less because "it's just a script."

Direct answer

Flapping (rapid healthy/unhealthy cycling) almost always means the alert's evaluation window is shorter than the system's normal, benign recovery time, not that the underlying issue is genuinely intermittent, so investigate the window/threshold mismatch first. Fix it by requiring the bad state to sustain before paging, without raising the threshold so far that it stops catching a real, more slowly building incident.

Structured elaboration

Investigation steps:

• Pull a baseline over a representative window (24-72h): how often the alert fires, how long each firing lasts, and what fraction resolve within a very short window on their own.
• Correlate firing timestamps against deploys, autoscaling events, and traffic patterns. Flapping that clusters right after every autoscale-up event or every deploy is a strong clue about the actual trigger.
• Check the alert's configuration for what's actually missing: no minimum-consecutive-samples requirement, an evaluation interval shorter than the system's normal transient-recovery time, or per-instance alerting fanning out instead of an aggregate view.
• Reproduce deliberately if possible: trigger a controlled scale-up or load test and watch whether the same benign pattern shows up.

Fix tiers:

1. Immediate, low-risk: require the condition to persist across N consecutive evaluation samples before paging, instead of firing on any single breach.
2. Medium-term: move from an instant threshold to a rate- or percentile-based check over a rolling window (for example, sustained error rate over 5 minutes rather than any single elevated sample).
3. Long-term: tie alerting to SLOs and error-budget burn so the alert reflects actual customer impact rather than a raw infrastructure metric that's naturally noisy during normal operation.

Validation. Run the changed threshold in shadow mode first, track the flap count and true-incident detection rate before and after, and get sign-off from the service owner on the specific numbers before flipping it back to paging.

Worked example

Investigation shows the alert evaluates every 15 seconds and pages on any single sample crossing the threshold, and that the service's autoscaler routinely causes the metric to cross that threshold for one to two evaluation cycles during a normal scale-up event before settling back down. Requiring 3 consecutive breaching samples at a 15-second evaluation interval means the condition has to stay unhealthy for at least (3-1) x 15 seconds = 30 seconds beyond the first breach before it pages. That's long enough to ride out the 15-30 second autoscale blip the investigation identified, while still catching anything that stays broken past that window. This is a design decision derived directly from the observed autoscale-blip duration, not an arbitrary "let's add a delay and see."

Trade-offs and pitfalls

• Pitfall: "just raise the threshold" without addressing the window/sustain mismatch masks the same noisy pattern at a higher, later-triggering severity instead of fixing it, and can end up hiding a real, slower-building degradation that would previously have crossed the old, lower threshold.
• Pitfall: fixing the sustain window and declaring victory without a tiered fast path means a genuinely fast, severe failure now waits out the same delay as a routine autoscale blip.
• Trade-off: percentile- or rate-based alerting over a rolling window is more robust to this kind of noise but reacts more slowly than an instant threshold, which is the right trade for capacity-style, gradually-building issues and the wrong trade for a hard-down, all-or-nothing failure.

Direct answer

My rule is: an alert pages me only if it's actionable, urgent, and real, meaning there's something I can actually do about it, it needs a response within minutes rather than hours, and it reflects genuine or imminent customer/business impact. Anything that fails one of those three tests goes to Slack or a ticket, not my phone. I protect my own sustainability by treating a repeatedly noisy page as a bug in the alert that needs fixing, not as a toughness test I'm supposed to pass.

Structured elaboration

The paging bar, in practice. Before an alert is allowed to page a human overnight, it should answer yes to all three:

• Actionable: is there a specific thing a person can do right now, or does it just need to be visible on a dashboard?
• Urgent: does waiting until business hours meaningfully worsen the outcome?
• Real: does it correlate with actual customer or business impact, not just an internal metric that moves for benign reasons?

Anything that's informational, non-urgent, or has historically resolved itself before a human could act belongs in Slack or a ticket queue, not a page.

Boundaries I set for myself and expect from a team:

• Rotation limits: a cap on consecutive on-call weeks and mandatory rest between rotations, so on-call load is a scheduling property of the team, not a matter of individual endurance.
• Compensation for the bad weeks: comp time or a stipend that scales with how bad the week actually was, so a rotation with three overnight Sev1s isn't treated the same as a quiet one.
• A standing agreement that any alert paging the same person more than once or twice without a code change in response gets flagged as noise to fix, not accepted as the cost of doing business.
• Tracking my own load: if my pages-per-week start trending up, that's something I bring to the team, not something I quietly absorb, because sustainable on-call is a team-level property (rotation depth, alert hygiene) and not an individual willpower contest.

Worked example

In a recent on-call rotation, an alert on a background job queue started paging me nightly around 2am even though the queue reliably drained on its own within a few minutes every time. I didn't just mute it and move on; I posted the pattern in our team channel with the timestamps and outcomes for the last several nights (fires, self-resolves, no human action taken each time), and we agreed together that the alert needed a sustained-duration requirement before it could page overnight, since a brief backlog wasn't actually urgent or actionable at 2am. I made the config change, had a teammate review it since it affected everyone's rotation, and rolled it out. The nightly pages for that specific alert stopped after the change. The underlying job still occasionally backed up during traffic spikes, but that now showed up as a daytime ticket for the team to investigate instead of an overnight page for whoever happened to be on call.

Trade-offs and pitfalls

• Pitfall: "I'm always reachable, page me for anything" sounds committed but actually signals a lack of judgment about what deserves urgency, and it's a fast path to burnout that eventually degrades response quality for the alerts that really matter.
• Pitfall: the opposite extreme, "I basically never get paged after hours," can mean either genuinely excellent alert hygiene or quietly under-covered risk; the honest answer distinguishes which one it is with a concrete example, not just an assertion.
• Trade-off: a team that pays well for on-call (stipends, real comp time) can sustain a slightly higher page volume than one that doesn't, but that changes how much noise is tolerable, not whether the actionable/urgent/real bar applies. A well-compensated but chronically noisy alert is still a bug to fix, not a cost center to accept.