On-Call Practices and Runbook Design Questions

Direct answer

Stop the flap by putting a circuit breaker on top of the remediation itself: require a minimum number of consecutive failures before acting at all, cap how many automated attempts happen before backing off exponentially, and open the breaker entirely, stopping automated action, after repeated failures so the automation isn't the thing keeping the service oscillating. The automation should get less aggressive the more it fails, not equally aggressive every time.

Structured elaboration

State machine

stateDiagram-v2
  [*] --> Closed
  Closed --> Remediating: threshold breached
  Remediating --> Closed: health recovers
  Remediating --> Open: still failing after attempt
  Open --> HalfOpen: cooldown elapses
  HalfOpen --> Closed: canary healthy
  HalfOpen --> Open: canary fails
  Open --> HumanApproval: max attempts reached

Why each piece prevents flapping

• A failure-count threshold before acting at all, requiring multiple consecutive failed health checks within a window rather than one, filters out single transient blips that don't need remediation.
• An exponential cooldown between attempts means each retry backs off further, so a persistent problem gets fewer, more spaced-out remediation attempts instead of a tight restart loop.
• The circuit breaker's open state, where the system stops acting entirely and escalates to a human after enough failed attempts, is what actually breaks the flap, not just slows it down.
• The half-open probe tests recovery with a single canary instance or request after the cooldown, before trusting the system enough to resume full automated remediation.
• Staggered, single-instance restarts that respect the dependency graph prevent one service's remediation from cascading into a restart storm on services that depend on it.

Worked example

With a base cooldown of 60 seconds and a cap of 1800 seconds (30 minutes), the cooldown after the nth failed attempt is:

$\text{cooldown}_n = \min(\text{base} \cdot 2^{n-1}, \text{max})$

$\text{cooldown}_1 = \min(60 \cdot 2^{0}, 1800) = 60\text{ s}$
$\text{cooldown}_2 = \min(60 \cdot 2^{1}, 1800) = 120\text{ s}$
$\text{cooldown}_3 = \min(60 \cdot 2^{2}, 1800) = 240\text{ s}$
$\text{cooldown}_4 = \min(60 \cdot 2^{3}, 1800) = 480\text{ s}$
$\text{cooldown}_5 = \min(60 \cdot 2^{4}, 1800) = 960\text{ s}$
$\text{cooldown}_6 = \min(60 \cdot 2^{5}, 1800) = \min(1920, 1800) = 1800\text{ s}$

After the sixth failed attempt the breaker opens: no further automated action until a human intervenes. This pattern applies whether the underlying trigger is a transient availability-zone network partition, a database connection-pool spike, or intermittent upstream timeouts. The automation doesn't need to know which cause it's looking at, because the backoff and breaker behave the same regardless.

Trade-offs and pitfalls

• Tuning the failure-count threshold too low means genuinely fast-recurring problems get no remediation attempt before the breaker opens; too high delays help for the transient blips the system was meant to catch.
• A longer maximum cooldown reduces flap risk but leaves a genuinely fixable problem unremediated for longer; the cap should be informed by how long a typical transient cause, like an AZ partition or a connection spike, usually takes to self-resolve.
• Forgetting the dependency graph is the most common miss: fixing one service's flap by restarting it can trigger a second flap downstream if that service health-checks against the still-recovering upstream.

Direct answer

A runbook is a fixed set of step-by-step instructions for a known failure mode: it tells you exactly what to type. A playbook is a decision framework for handling an incident more broadly: it tells you how to figure out what to do, who needs to be involved, and which runbook to reach for. Reach for a runbook when you already know the cause and the fix is mechanical; reach for a playbook when you're still diagnosing, coordinating multiple people, or the right response depends on judgment.

Structured elaboration

Minimum fields for each:

• Runbook: title, service, owner, trigger/precondition, required permissions and tools, exact step-by-step commands, verification steps, rollback steps, expected impact, last-reviewed date.
• Playbook: title, scope and severity thresholds, incident commander and stakeholder roles, the decision tree itself, links to the relevant runbooks, communication templates, escalation matrix, last-reviewed date.

Worked example

A database replica's lag exceeds a threshold: this is a runbook. It lists the exact commands to promote a replica, the steps to reconfigure the application's read preference, verification queries to confirm the fix, and the rollback commands if the promotion causes a new problem. Now compare that to a major outage affecting payments: this is a playbook. It guides the incident commander through detecting the actual scope, declaring severity, deciding between routing traffic to a fallback payment path versus draining traffic entirely, coordinating the app, infra, and comms teams, and linking out to the specific runbooks (including the replica-promotion one, if that turns out to be the fix) for whichever technical action the decision tree leads to.

Trade-offs and pitfalls

• Pitfall: writing a "runbook" for something that actually needs judgment, for example "when in doubt, restart the service." That hides a decision a playbook should make explicit, and someone follows it verbatim during exactly the incident it doesn't fit.
• Pitfall: letting a playbook go stale is worse than letting one runbook go stale, because the playbook is what everyone reaches for first during ambiguity; a stale escalation matrix (wrong names or numbers) breaks the whole response, not just one specific fix.
• Trade-off: automating a runbook into a one-click execution is great for high-confidence, low-blast-radius fixes (restarting a stateless service) and dangerous for high-blast-radius ones (promoting a database replica). The more damage a wrong click can do, the more the runbook should require an explicit human confirmation step before executing, not less.
• Both belong in a versioned, reviewed repository rather than an unowned wiki page, with periodic review, and runbooks specifically benefit from occasional dry-run or game-day testing to confirm the steps still work against the current system rather than an outdated one.

Spend the first two weeks measuring where pages actually come from, not guessing, then rank the recurring drivers by pages eliminated per engineer-day of effort and fund the top of that list first. Feature work gets whatever capacity is left after the pager-volume target for the quarter is funded, not the other way around.

Step 1: baseline before you prioritize anything

Pull a two-week alert log and count pages by source, time of day, and whether each one required a real action or was noise. Prioritizing off memory or the loudest recent incident produces a list that optimizes for what people remember, not what's actually costing the most on-call time.

Step 2: score every recurring driver

Worked example: allocating a 20-day quarterly toil budget

Take the items in score order until the budget runs out: item 1 (1 day) + item 2 (3 days) + item 3 (4 days) + item 4 (5 days) = 13 of 20 days, leaving 7 days of buffer rather than starting the 15-day redesign this quarter.

Against a baseline of 220 pages/month, that's a reduction of

funded by 13 of 20 available toil-reduction days, with the remaining 7 days as margin for whatever the alert log surfaces next.

Trade-offs and pitfalls

Silencing an alert to hit the score is only a win if it was genuinely non-actionable; verify that before suppressing it, since a silenced alert that was catching a real problem just moves the cost from "pages" to "undetected incidents." The full redesign scores lowest on pages-per-day but may be the only fix that prevents an outage-class failure; the score is an input to prioritization, not the whole decision, and a high-blast-radius item can justify funding even at a low score. Communicate the capacity trade explicitly to product as a quarter-long commitment rather than letting it get silently reprioritized sprint by sprint. Re-score the remaining list after each phase; fixing the top four items usually promotes a previously mid-ranked item to the top as the dominant offender changes.

Whether to fully automate, keep human-in-the-loop, or leave manual comes down to four questions applied to each specific action: how often does it happen, how bad is it if it goes wrong, can it be safely retried, and can success be verified automatically. High frequency, low blast radius, idempotent, and observable pushes toward full automation; anything destructive or hard to verify stays manual or gated behind a human, no matter how routine it feels.

The criteria

Applying it to the three actions

Restarting a cached worker instance: high frequency, low blast radius (stateless, replaceable), fully idempotent, and easily verified with a health check. This is a strong automate candidate: drain connections, spin up a replacement, run a health check, cut traffic over, roll back automatically if the health check fails.

Reattaching a detached volume: lower frequency, meaningfully higher blast radius (attaching to the wrong instance or double-attaching can corrupt data), and only moderately idempotent, reattaching twice isn't necessarily safe. This sits in the middle: automate the pre-checks and the mechanical steps (verify volume ID, verify target instance, snapshot before attaching), but require a human to confirm before the final attach executes.

Running a database schema migration: low frequency, high blast radius (can be destructive and hard to reverse), low idempotency for anything involving DDL (data definition language: schema-altering SQL statements like ALTER TABLE), and success often isn't verifiable by a simple automated check, it needs someone to look at whether the data actually came out right. This stays manual, or more precisely, human-gated: automation handles the mechanical parts (schema diff, pre-migration validation, backup, staged rollout to a canary), but a person approves the production apply.

The underlying argument for phasing automation in gradually

Automating a step doesn't just remove toil, it also removes the moment a human would have caught something unusual about this particular instance of the problem. That's fine for the worker-restart case, where "unusual" mostly doesn't exist, but risky for the migration case, where every migration is a little different. The practical path is phasing: run a new automation in shadow mode first (it proposes the action but a human executes), then human-in-the-loop (it executes after one-click approval), and only promote to full automation once it has a track record across enough real incidents that its false-positive and false-negative rate are actually known, not assumed.

Guarding automated actions with least privilege

Whatever is automated should run with only the permissions that specific action needs, a worker-restart automation shouldn't hold credentials that could also run a schema migration, and every automated action should be logged with who (or what) triggered it and why. For the human-in-the-loop tier, the approval step itself should require a specific person's action (not a shared bot token anyone can trigger), so there's a real approval trail, not a rubber stamp.

Trade-offs and pitfalls

The common wrong turn is automating based on how annoying a task feels rather than how safe it is, restarting workers manually is annoying but safe to automate; migrations are also annoying, but the annoyance is not the variable that should decide it. The other pitfall is leaving a human-in-the-loop step gated behind an approval that nobody actually reads before clicking, if the approval doesn't include enough context (what will run, what's the blast radius, what's the rollback) to make a real judgment, it's automation with an extra click, not a genuine safety gate.

The version of this story that lands isn't "I said no," it's "I translated an impossible ask into priced options and let the business choose one." Quantify what each availability tier actually costs in downtime and dollars, then let the person asking pick a tier instead of arguing about whether 100% uptime is achievable.

Situation

On a multi-region migration project, the client's leadership wanted a contractual guarantee of 100% uptime with no additional budget, written directly into the renewal. Procurement had already fixed the budget, and the engineering side knew the ask was not just expensive but physically meaningless as written; nothing achieves literal 100% availability.

What availability tiers actually mean

Every "nines" figure translates to a concrete amount of allowed downtime per year:

Task and action

My job was to protect delivery and financial risk while still meeting the underlying business need for reliability. I brought three concrete options, each priced relative to a simple baseline (this relative-cost framing was an illustrative assumption for the workshop, not a benchmarked figure): a baseline single-region setup at roughly 99.5% availability, a warm-standby setup with automated failover at roughly 99.95%, and a full active-active geo-redundant setup at roughly 99.99%+, at meaningfully higher infrastructure cost. I ran a workshop with the CIO, procurement, and product owners to map which transactions were actually revenue-critical (payment flows) versus which weren't (internal telemetry, admin dashboards), rather than treating the whole system as one uniform target.

Result

The client accepted a hybrid SLA: the higher-availability tier for payment flows, the baseline tier for non-critical systems, phased with a roadmap to increase availability later if the business case justified it. The negotiation replaced an unenforceable "100%, no exceptions" clause with numbers everyone in the room had agreed to and could actually be held to.

Trade-offs and pitfalls

Showing up with only "that's technically impossible" and no alternative loses the argument on tone even when you're right; always bring priced options, not just a rejection. Watch for the agreed tier quietly becoming a number nobody revisits; put an explicit review date on it rather than treating the negotiation as a one-time event. Not every ask is negotiable this way; if leadership genuinely won't move off a literal 100% guarantee, that's a signal to escalate the conversation above your level rather than keep re-explaining the same math to the same audience.