On-Call Practices and Runbook Design Questions

When the same incident keeps recurring despite repeated fixes, the prior RCAs were almost certainly treating a symptom as the root cause; run this RCA by explicitly listing every prior "fix" and asking why each one didn't hold, because the pattern across failed fixes usually points straight at the real systemic gap.

Framework for a systemic RCA

1. Build a fix history first, before investigating the current occurrence. For each prior incident of this same recurring issue: what was diagnosed as the cause, what was changed, and did the change actually address that diagnosis or just the immediate symptom? A pattern of "different symptom fixed each time, same underlying trigger every time" is the tell that root cause was never actually found.
2. Separate the trigger from the vulnerability. The trigger (a specific deploy, a specific load pattern, a specific external dependency hiccup) may vary each month, but if the same class of trigger keeps causing an outage, the system has a standing vulnerability to that trigger class that no single fix removed. The RCA's job is to name the vulnerability, not just the latest trigger.
3. Use the fishbone categories (code, config, infrastructure, process, tooling) to check whether every prior fix landed in the same category. If four consecutive fixes were all code patches but the incident keeps returning, that's evidence the real gap is in process or tooling (no regression test for this class of failure, no canary catching it before full rollout) rather than in any specific line of code.
4. Test the systemic hypothesis, don't just assert it. If the hypothesis is "the nightly batch job and the backup window contend for the same database connection pool," verify by reproducing that contention in a controlled environment (isolated run of the batch job during a simulated backup window), not by pattern-matching from the incident timeline alone.

Worked example

A nightly batch job has caused three partial outages in three consecutive months. Prior fixes: month 1, increased the job's timeout (fix addressed "job was timing out"); month 2, added a retry with backoff (fix addressed "job failed transiently"); month 3 is the current incident, and the job is again failing, this time differently, a connection pool exhaustion error. Building the fix history shows a pattern: every fix targeted why the job failed on that specific night, and none asked why the job's failure mode changes every month while the timing (always during the nightly backup window) stays constant. Testing the systemic hypothesis, that the batch job and backup process share a connection pool and the backup's duration has been slowly growing as data volume grows, month-over-month backup duration logs confirm the backup window has grown from roughly 12 minutes to 40 minutes over the quarter, now overlapping the batch job's peak connection usage. That is the systemic cause: the job and backup were never intentionally isolated, and it was invisible for months because the backup was short enough not to overlap.
The durable fix follows from the systemic cause, not the latest symptom: allocate the batch job a dedicated connection pool separate from ad hoc processes, and alert on backup-duration trend (not just backup failure) so a slowly growing resource conflict is visible before it causes an outage again.

Trade-offs and pitfalls

The main pitfall is that a systemic RCA takes longer and produces a less satisfying immediate answer than "here's the line that broke," which creates pressure to ship another symptom-level fix under time pressure; the way to resist that is to make the fix-history review a required first step, not an optional deep-dive, so the systemic question gets asked before anyone commits to a scope. A real trade-off: broadening the RCA to a process or tooling gap usually means the fix is slower to land (new alerting, resource isolation, a regression test suite) than a code patch, so it's worth explicitly stating in the postmortem that a fast interim mitigation (in this example, a manual connection-pool bump) is being paired with the slower structural fix, rather than letting the slow fix block any near-term relief.

Direct answer

A chronically low-precision page is a design bug in the alert, not a discipline problem for the responder: fix it by classifying WHY each false positive happened (transient blip versus a signal that's sustained but wrong for a different reason) and attacking each class with a targeted change, then re-measure in shadow mode before it pages anyone again. Separately, decide what generally deserves to page a human with an explicit bar: actionable (there's something to do right now), urgent (waiting meaningfully worsens the outcome), and real (it correlates with actual impact, not an internal metric that moves for benign reasons).

Structured elaboration

1. Classify the false positives, don't guess. Pull a sample of historical firings and tag each one: true incident, transient blip (self-resolved with no action), or sustained-but-non-actionable (stayed unhealthy but for a known, benign reason). Compute today's precision from that sample before changing anything.
2. Attack each class differently. Transient blips get a sustain/consecutive-sample requirement, since by definition they don't last. Sustained-but-wrong firings need a better underlying signal, typically moving from a cause-based check (an internal metric that spikes for reasons unrelated to customer impact) to a symptom-based one (customer-facing error rate or SLO burn).
3. Re-measure before re-enabling paging. Run the changed alert in shadow/log-only mode against real traffic for at least a full weekly cycle, and compare precision and recall against the original, not just precision alone, since a change that also silently drops true positives isn't actually a win.
4. Set the general page-or-not bar (actionable, urgent, real) so the next alert someone proposes gets evaluated against a fixed standard instead of being added because it seemed prudent at the time.

Worked example

Say the alert fired 100 times last month, and by the question's premise it's "right only a fraction of the time": assume 20 of those 100 firings were real incidents and 80 were false positives.

$$\text{precision} = \frac{TP}{TP+FP} = \frac{20}{20+80} = 0.20$$

Re-tagging the 80 false positives by how long they stayed unhealthy: suppose 65 of them self-resolved within a single evaluation cycle with no human action (transient blips), and 15 stayed unhealthy for several minutes but turned out to be a known, benign condition, for example a scheduled batch job that always spikes this particular metric (sustained-but-wrong).

Adding a "must stay unhealthy for 3 consecutive evaluation samples" requirement removes the 65 transient blips entirely, since none of them lasted past a single sample, while every one of the 20 true incidents is unaffected, because a real incident is sustained by nature and was never at risk of being filtered by a sustain window. Precision after this one change:

$$\text{precision}_{\text{new}} = \frac{20}{20+15} = \frac{20}{35} \approx 0.571$$

That's a jump from 20% to roughly 57% precision from a single, low-risk change, with the true-positive count completely untouched. The remaining 15 sustained-but-wrong firings need a second, more targeted fix, such as excluding the known benign batch-job window or moving to a symptom-based check, since a sustain window alone can't distinguish "sustained and wrong" from "sustained and real." That second change should also be shadow-tested before it's trusted to page.

Trade-offs and pitfalls

• Pitfall: a blanket sustain window can blind you to a genuinely fast, severe failure (a hard crash) that needs to page in well under the sustain threshold. The fix should be tiered: a fast, low-latency path for total-outage-class signals, and a sustain-gated path for degradation-class signals, not one delay applied to every alert on the service.
• Pitfall: re-measuring precision only in the days immediately after a change is vulnerable to regression to the mean; hold the shadow test open long enough to capture the conditions that caused the original noise (a full weekly cycle, including whatever batch job or traffic pattern triggered the sustained-but-wrong case).
• Trade-off: symptom-based alerting (SLO burn rate, customer-facing error rate) is more precise about whether something actually matters, but it's slower to point at a root cause than a cause-based internal metric. Good practice pages on the symptom and links a runbook that walks through cause-based diagnostics from there, rather than choosing one approach exclusively.

A strong answer to this picks one real incident, states plainly what your specific role was (not "the team"), walks through how status was communicated to the people who needed it, and ends with a concrete process change that came out of the postmortem, not just a lesson learned in the abstract. The bar an interviewer is checking is whether you can drive coordination under pressure and whether the org actually got better afterward, not whether the incident itself was dramatic.

Structuring the answer

• Situation: what broke, which teams were involved, and why it needed cross-team coordination rather than one person fixing it alone.
• Task: your specific role, on-call engineer, incident commander, SME being pulled in, and what you were responsible for.
• Action: what you actually did, especially the communication piece: how often you posted updates, who you looped in and when, how you decided what to escalate.
• Result: how the incident resolved, and the one concrete thing that changed afterward as a direct result of the postmortem.

Worked example (illustrative story skeleton)

Situation: I was on-call when a payment-processing service started failing a meaningful share of transactions during a peak traffic window, and it quickly became clear the cause spanned two teams, ours and the database team, not something either of us could fix alone.

Task: As the on-call engineer, my first job was triage and coordination, not fixing it single-handedly, since the fix needed the database team's context on a recent schema change.

Action: I opened a dedicated incident channel, pulled in the database on-call and our service owner, and posted a short status update on a fixed cadence so people weren't pinging me individually for progress. We traced the failures to a missing index from a recent migration, coordinated a rollback with the database team while I kept support and the affected team's manager updated with plain-language impact summaries (not the technical detail I was sharing in the incident channel).

Result: The rollback resolved the failure rate back to its normal baseline within the same on-call shift. In the blameless postmortem, the concrete change we made was adding a required index-verification check to the deploy pipeline for any migration, so a similar schema change can't ship without it being caught automatically rather than during a live incident.

Variant: when the story is about a communication breakdown

The strongest version of this question for a communication-focused answer isn't one where everything went smoothly, it's one where something specific broke in how status was shared, and what changed as a result. For example: during a similar incident, the customer-support team found out about the outage from customers before the incident channel posted a status update, because the communications role wasn't assigned and everyone assumed someone else was posting it. The fix that came out of that postmortem wasn't "communicate better", it was specific: the incident-declaration checklist now requires naming a communications owner in the first two minutes, before technical triage even starts, so that role is never implicitly assumed.

Trade-offs and pitfalls

The most common mistake is describing only the technical fix and skipping the coordination and communication details entirely, which is exactly the part this question is testing for. A close second is ending on "we learned to communicate better" without naming the specific process change that came out of it; a blameless postmortem is only as good as the concrete, trackable action item it produces, and an answer that mirrors that (a specific fix, not a vague lesson) is what separates a senior-sounding story from a generic one.

Allocate people by comparing the two incidents' business impact and required expertise, not by splitting the team evenly, and default to running them as separate incidents with separate commanders unless you find a shared root cause; escalate to a higher-level incident commander as soon as the resource conflict itself (not just the technical severity) becomes the bottleneck.

Allocating people across concurrent incidents

• Score each incident independently first: user-facing impact, revenue impact, data-integrity risk, and blast radius (one team's problem versus platform-wide). Two incidents rarely score identically, and the higher-scored one gets first claim on the strongest responders.
• Check for a shared root cause before splitting resources. If both services depend on the same failing component (a shared database, a shared auth service), that's actually one incident with two symptoms, and it should be run as a single incident with one commander, not two competing efforts pulling on the same underlying fix.
• Staff each independent incident with a minimum viable team: one incident commander, one primary responder, one communications owner. Resist over-staffing the incident that's louder or more visible if the other one is actually higher severity but quieter.
• Protect against double-booking the same expert. If one person is the only one who understands a shared piece of infrastructure, they can advise both incidents briefly but should not be the sole owner of fixing both; pull in a secondary responder even if slower.

When to escalate to a higher-level incident commander

Escalate when any of these is true, not just when severity is high:

• Resource contention itself is blocking progress: both incidents need the same scarce specialist or the same change-freeze exception, and someone above both incident commanders needs to arbitrate.
• Combined blast radius crosses an organizational boundary: the two incidents together affect enough of the business (multiple product lines, a shared customer segment) that unified external communication is needed, even if each incident alone wouldn't trigger that.
• One incident commander is starting to context-switch between both incidents. A single IC trying to run two incidents at once is a bigger risk than the incidents themselves; that's a signal to bring in a second commander or an overall coordinator, not to push through.
• Duration crosses a threshold where sustained dual-incident load starts to fatigue the responding team; a higher-level commander can pull in fresh responders or make the call to deprioritize the lower-severity incident explicitly (and communicate that decision) rather than let it silently starve.

Worked example

Two incidents fire eleven minutes apart: the checkout service is returning 500s for roughly a third of requests (revenue-impacting, high severity), and the internal analytics dashboard is showing stale data (no customer impact, low severity). The correct allocation: full incident-commander-plus-primary-plus-comms team goes to checkout immediately; analytics gets a single responder to investigate on a non-paging basis, because pulling more people onto analytics wouldn't shorten its resolution meaningfully and would strip capacity from checkout. If, twenty minutes in, the checkout investigation discovers the 500s trace back to the same message queue that feeds the analytics pipeline, the two incidents are merged under checkout's commander, because they share a root cause and running them separately would mean two people independently investigating the same queue.
Escalation in this example would trigger only if a third, unrelated incident arrived while checkout was still active and unresolved: three concurrent incidents makes single-IC-per-incident coordination itself the bottleneck, which is exactly the resource-contention trigger above.

Trade-offs and pitfalls

A common mistake is allocating headcount proportional to how loud or visible each incident is (how many people are asking about it in Slack) rather than its actual business impact; loud and low-impact will out-compete quiet and high-impact if you let it. Another is treating "escalate to a higher IC" as an admission of failure, so teams delay it past the point where a fresh coordinator would have resolved the resource conflict faster. The trade-off with merging incidents on a suspected shared root cause is real: merge too eagerly and you lose the separate investigation threads that might have found the divergence faster; the mitigation is to merge the coordination and communication, but keep separate technical workstreams until the shared cause is actually confirmed.

Readiness is a checklist, not a countdown: get access and tooling working first, have them study and sign off on the runbooks for their services, shadow several live pages, then run one supervised tabletop and one supervised live (or simulated) incident before they take a shift alone with a mentor reachable but not present.

A four-week ramp

Sign-off checklist before the first unsupervised shift

• Access confirmed end to end (paging tool, dashboards, deploy/rollback permissions) with a real test, not just "provisioned."
• Runbooks for their assigned services reviewed and any ambiguous steps flagged and fixed.
• At least one supervised tabletop and one supervised live or injected-fault incident completed.
• Mentor sign-off plus the engineer's own confidence self-assessment, not mentor judgment alone.

Extending the ramp for a complex or high-stakes service

Four weeks is often enough for a straightforward service, but for a complex hybrid-cloud system or one with many downstream dependents, add explicit competency checkpoints tied to named systems, for example certifying someone independently on the database failover path as a separate sign-off from general on-call readiness, rather than declaring them ready across the board at once. Longer term, treat the first 90 days as a structured mentorship arc rather than stopping at week four: scheduled 30/60/90-day check-ins, a second mentor pairing on a different service, and a distinct milestone for graduating from "supervised" to "primary" status rather than just a date on the calendar.

Trade-offs and pitfalls

Rushing readiness to fill a rotation gap is the most common failure mode, and it produces confident-sounding but wrong incident responses, which is worse than an obviously under-prepared response because it takes longer to catch. A checklist with no live-incident component only validates that someone can read, not that they can act under time pressure; keep at least one supervised live or simulated incident before signing off. Sign-off criteria should be service-specific rather than one generic "on-call ready" badge, since readiness on a well-instrumented service doesn't automatically transfer to a fragile legacy one with thin runbooks.