On-Call Practices and Runbook Design Questions

The first few minutes after a page are for narrowing down scope and cause fast enough to act, not for finding the root cause. I validate the alert is real, figure out how big the blast radius is, check whether it lines up with the deploy that just went out, and pick a mitigation I can undo, escalating the moment the clock runs out on any of those steps.

The first-minutes checklist

1. Validate (0-2 min): confirm the page against the actual dashboard, not just the alert text. Is p95 latency and error rate really elevated right now, or is this a flapping alert that already recovered?
2. Determine scope (2-5 min): is this one service, one region, or several services at once? Check whether other, unrelated-looking alerts fired in the same window, they're often the same root cause fanning out, not five separate problems.
3. Correlate with the deploy (2-5 min): the timing (a spike a few minutes after a deploy) is a strong signal but not proof; pull the diff (code, config, feature flags, schema changes) to see if anything plausibly explains a latency or error increase.
4. Choose a reversible mitigation (5-15 min): rollback if the deploy is the clear cause and rollback is safe and fast; scale out or shed load if the cause looks like saturation; flip a feature flag if the change is flag-gated. Prefer whatever gets users out of pain fastest and can be undone if it turns out to be wrong.
5. Escalate on the clock, not on ego (by 15 min): if scope, cause, or mitigation isn't clear by 15 minutes, escalate, that's a threshold, not a judgment call to second-guess in the moment.

Decision flow

flowchart TD
  A[Page: p95 latency + error rate up, deploy went out minutes ago] --> B[0-2m: Validate against real dashboards]
  B --> C[2-5m: Determine scope: one service or several]
  C --> D{Timing matches the deploy?}
  D -->|Yes| E[Pull diff: code, config, feature flag, schema]
  D -->|No| F[Check shared dependency: DB, network, upstream API]
  E --> G[5-15m: Pick a reversible mitigation]
  F --> G
  G --> H{Recovering by 15m?}
  H -->|Yes| I[Validate recovery, stand down, write timeline]
  H -->|No| J[Escalate: page secondary and service owner, open incident]

Telling apart the three usual causes when several services fail together

When multiple services degrade at once, the fastest disambiguation is checking one thing first: are the affected services related by a shared dependency, or only by being on the same network path?

• Shared dependency outage (a database, cache, or upstream API multiple services call): errors cluster around calls to that one dependency across otherwise-unrelated services; the fix is usually failover or shedding load to the dependency, not touching the calling services.
• Network partition: services can't reach each other or a shared resource, but each service is individually healthy when checked in isolation (CPU, memory, internal logic all fine); the fix is on the networking layer, not application code.
• Code bug from the recent deploy: failures track cleanly to services that received the deploy, and services that didn't receive it stay healthy; the fix is rollback.
  The tell is the failure pattern: dependency outages show a common downstream call in every affected service's error logs, network partitions show healthy services that simply can't connect, and deploy-caused bugs map exactly to the set of services the deploy touched.

Two scenario variants worth knowing

A gradual climb (latency creeping up over 20 to 30 minutes rather than spiking) usually points at a resource exhaustion pattern, a connection leak or a slowly growing queue, rather than a bad deploy; the checklist is the same but step 3 shifts from "what deployed" to "what's been running long enough to leak." A regional error spike on a payments API narrows scope immediately to that region's infrastructure or a regional dependency, and raises the severity bar (payments plus regional means check whether it's revenue-impacting before deciding the escalation urgency).

Trade-offs and pitfalls

The most common mistake under pressure is skipping validation and mitigating against the alert text instead of the live dashboard, which can mean fixing a problem that already resolved itself while missing the real one. The second most common mistake is treating the 15-minute escalation threshold as optional because "I'm close", escalating early and being wrong costs a few minutes of someone else's attention; escalating late during a real outage costs users.

Direct answer

"Blameless" means the postmortem analyzes the incident as a failure of systems and processes that made a reasonable person's normal action produce a bad outcome, not as a failure of that person's competence or effort. It matters because the moment a postmortem starts assigning individual fault, people stop giving you the honest details (what they clicked, what they assumed, what they skipped) that you actually need to fix the underlying gap, and near-misses stop getting reported at all.

Structured elaboration

What blameless does not mean. It is not "no accountability." Individuals still own action items and are still expected to do their jobs well. Blameless means the analysis stops at "why did this look like the right thing to do at the time, given the information and tools available" instead of stopping at "who made the mistake."

Essential components of a postmortem document:

• Summary and impact: severity, duration, which customers/systems were affected, business impact.
• Timeline: timestamped sequence from first signal to full resolution, including detection and every mitigation attempt.
• Contributing factors: plural, not a single root cause. A "five whys" style chain that includes technical gaps (missing test, no lock-duration check) and process gaps (review checklist didn't require it, no staging environment with prod-sized data).
• What went well / what didn't: honest assessment of detection speed, mitigation effectiveness, and communication, separate from the technical cause.
• Action items: each with an owner, a due date, and a verification step, tracked to closure rather than left as a paragraph nobody revisits.
• Lessons learned: shared broadly enough that other teams with the same pattern can act on it before they hit the same incident.

Worked example

A deploy runs a database migration that locks a hot table for several minutes, causing a customer-facing outage. A blameful writeup says: "Engineer X pushed a migration without checking lock duration." A blameless writeup for the same incident says: "The migration tool doesn't warn about lock duration before merge, and the review checklist doesn't require a dry run against a prod-sized dataset. Both are now action items: add a lock-duration check to the migration tool (owner: platform team, verify by testing against a 10M-row table), and add a prod-sized dry-run step to the migration checklist (owner: DBA lead, verify by auditing the next five migrations)." The second version identifies exactly the same failure but produces two concrete, ownable fixes instead of a warning to be more careful next time, which is not something that reliably prevents a repeat.

Trade-offs and pitfalls

• Pitfall: blameless in the document but not in the room. Teams sometimes write a passive-voice, name-free doc while the actual retro meeting is full of pointed questions at one person. The doc's tone has to match the meeting's tone, or the culture stays blameful regardless of what's written down.
• Pitfall: treating blameless as zero consequences ever. Repeated, egregious negligence (ignoring a known policy, skipping a required review on purpose) is a people-management conversation, but it happens outside the technical postmortem, not inside it.
• Senior signal: distinguishing proximate cause from contributing factors. A junior answer stops at "the migration locked the table." A senior answer keeps asking why the tooling, the review process, and the testing environment all failed to catch it, because single-root-cause thinking tends to produce a single, shallow fix that doesn't survive the next incident with a different trigger but the same underlying gap.

Spend the first two weeks measuring where pages actually come from, not guessing, then rank the recurring drivers by pages eliminated per engineer-day of effort and fund the top of that list first. Feature work gets whatever capacity is left after the pager-volume target for the quarter is funded, not the other way around.

Step 1: baseline before you prioritize anything

Pull a two-week alert log and count pages by source, time of day, and whether each one required a real action or was noise. Prioritizing off memory or the loudest recent incident produces a list that optimizes for what people remember, not what's actually costing the most on-call time.

Step 2: score every recurring driver

Worked example: allocating a 20-day quarterly toil budget

Take the items in score order until the budget runs out: item 1 (1 day) + item 2 (3 days) + item 3 (4 days) + item 4 (5 days) = 13 of 20 days, leaving 7 days of buffer rather than starting the 15-day redesign this quarter.

Against a baseline of 220 pages/month, that's a reduction of

funded by 13 of 20 available toil-reduction days, with the remaining 7 days as margin for whatever the alert log surfaces next.

Trade-offs and pitfalls

Silencing an alert to hit the score is only a win if it was genuinely non-actionable; verify that before suppressing it, since a silenced alert that was catching a real problem just moves the cost from "pages" to "undetected incidents." The full redesign scores lowest on pages-per-day but may be the only fix that prevents an outage-class failure; the score is an input to prioritization, not the whole decision, and a high-blast-radius item can justify funding even at a low score. Communicate the capacity trade explicitly to product as a quarter-long commitment rather than letting it get silently reprioritized sprint by sprint. Re-score the remaining list after each phase; fixing the top four items usually promotes a previously mid-ranked item to the top as the dominant offender changes.

Direct answer

My rule is: an alert pages me only if it's actionable, urgent, and real, meaning there's something I can actually do about it, it needs a response within minutes rather than hours, and it reflects genuine or imminent customer/business impact. Anything that fails one of those three tests goes to Slack or a ticket, not my phone. I protect my own sustainability by treating a repeatedly noisy page as a bug in the alert that needs fixing, not as a toughness test I'm supposed to pass.

Structured elaboration

The paging bar, in practice. Before an alert is allowed to page a human overnight, it should answer yes to all three:

• Actionable: is there a specific thing a person can do right now, or does it just need to be visible on a dashboard?
• Urgent: does waiting until business hours meaningfully worsen the outcome?
• Real: does it correlate with actual customer or business impact, not just an internal metric that moves for benign reasons?

Anything that's informational, non-urgent, or has historically resolved itself before a human could act belongs in Slack or a ticket queue, not a page.

Boundaries I set for myself and expect from a team:

• Rotation limits: a cap on consecutive on-call weeks and mandatory rest between rotations, so on-call load is a scheduling property of the team, not a matter of individual endurance.
• Compensation for the bad weeks: comp time or a stipend that scales with how bad the week actually was, so a rotation with three overnight Sev1s isn't treated the same as a quiet one.
• A standing agreement that any alert paging the same person more than once or twice without a code change in response gets flagged as noise to fix, not accepted as the cost of doing business.
• Tracking my own load: if my pages-per-week start trending up, that's something I bring to the team, not something I quietly absorb, because sustainable on-call is a team-level property (rotation depth, alert hygiene) and not an individual willpower contest.

Worked example

In a recent on-call rotation, an alert on a background job queue started paging me nightly around 2am even though the queue reliably drained on its own within a few minutes every time. I didn't just mute it and move on; I posted the pattern in our team channel with the timestamps and outcomes for the last several nights (fires, self-resolves, no human action taken each time), and we agreed together that the alert needed a sustained-duration requirement before it could page overnight, since a brief backlog wasn't actually urgent or actionable at 2am. I made the config change, had a teammate review it since it affected everyone's rotation, and rolled it out. The nightly pages for that specific alert stopped after the change. The underlying job still occasionally backed up during traffic spikes, but that now showed up as a daytime ticket for the team to investigate instead of an overnight page for whoever happened to be on call.

Trade-offs and pitfalls

• Pitfall: "I'm always reachable, page me for anything" sounds committed but actually signals a lack of judgment about what deserves urgency, and it's a fast path to burnout that eventually degrades response quality for the alerts that really matter.
• Pitfall: the opposite extreme, "I basically never get paged after hours," can mean either genuinely excellent alert hygiene or quietly under-covered risk; the honest answer distinguishes which one it is with a concrete example, not just an assertion.
• Trade-off: a team that pays well for on-call (stipends, real comp time) can sustain a slightly higher page volume than one that doesn't, but that changes how much noise is tolerable, not whether the actionable/urgent/real bar applies. A well-compensated but chronically noisy alert is still a bug to fix, not a cost center to accept.

Whether to fully automate, keep human-in-the-loop, or leave manual comes down to four questions applied to each specific action: how often does it happen, how bad is it if it goes wrong, can it be safely retried, and can success be verified automatically. High frequency, low blast radius, idempotent, and observable pushes toward full automation; anything destructive or hard to verify stays manual or gated behind a human, no matter how routine it feels.

The criteria

Applying it to the three actions

Restarting a cached worker instance: high frequency, low blast radius (stateless, replaceable), fully idempotent, and easily verified with a health check. This is a strong automate candidate: drain connections, spin up a replacement, run a health check, cut traffic over, roll back automatically if the health check fails.

Reattaching a detached volume: lower frequency, meaningfully higher blast radius (attaching to the wrong instance or double-attaching can corrupt data), and only moderately idempotent, reattaching twice isn't necessarily safe. This sits in the middle: automate the pre-checks and the mechanical steps (verify volume ID, verify target instance, snapshot before attaching), but require a human to confirm before the final attach executes.

Running a database schema migration: low frequency, high blast radius (can be destructive and hard to reverse), low idempotency for anything involving DDL (data definition language: schema-altering SQL statements like ALTER TABLE), and success often isn't verifiable by a simple automated check, it needs someone to look at whether the data actually came out right. This stays manual, or more precisely, human-gated: automation handles the mechanical parts (schema diff, pre-migration validation, backup, staged rollout to a canary), but a person approves the production apply.

The underlying argument for phasing automation in gradually

Automating a step doesn't just remove toil, it also removes the moment a human would have caught something unusual about this particular instance of the problem. That's fine for the worker-restart case, where "unusual" mostly doesn't exist, but risky for the migration case, where every migration is a little different. The practical path is phasing: run a new automation in shadow mode first (it proposes the action but a human executes), then human-in-the-loop (it executes after one-click approval), and only promote to full automation once it has a track record across enough real incidents that its false-positive and false-negative rate are actually known, not assumed.

Guarding automated actions with least privilege

Whatever is automated should run with only the permissions that specific action needs, a worker-restart automation shouldn't hold credentials that could also run a schema migration, and every automated action should be logged with who (or what) triggered it and why. For the human-in-the-loop tier, the approval step itself should require a specific person's action (not a shared bot token anyone can trigger), so there's a real approval trail, not a rubber stamp.

Trade-offs and pitfalls

The common wrong turn is automating based on how annoying a task feels rather than how safe it is, restarting workers manually is annoying but safe to automate; migrations are also annoying, but the annoyance is not the variable that should decide it. The other pitfall is leaving a human-in-the-loop step gated behind an approval that nobody actually reads before clicking, if the approval doesn't include enough context (what will run, what's the blast radius, what's the rollback) to make a real judgment, it's automation with an extra click, not a genuine safety gate.