On-Call Practices and Runbook Design Questions

Direct answer

A runbook for a junior on-call unfamiliar with the model internals should never ask them to reason about the model itself. It should map symptoms, a false-positive spike or stale features, to a small set of safe, reversible actions such as rolling back to the last known-good model version, failing open to cached or default features, or conservatively raising a decision threshold, with a clear line for when to stop and page the ML on-call. The goal for this runbook is containment and triage, not diagnosis.

Structured elaboration

Design principle: no model-internals knowledge required

Every step should compare a dashboarded metric against a documented normal range and trigger a small, pre-approved action, never "investigate the model." Every action must also be reversible: a version rollback, not a retrain; a threshold change, not a code change.

Triage flow

flowchart TD
  A[Alert: FP rate or staleness spike] --> B{Deviation vs baseline metric}
  B -->|Small| C[Log and monitor, no action]
  B -->|Large| D{Recent deploy or feature-pipeline change?}
  D -->|Yes| E[Roll back to last known-good model version]
  D -->|No| F{Feature staleness or pipeline lag detected?}
  F -->|Yes| G[Fail open to cached, last-good features]
  F -->|No| H[Raise score threshold, page ML on-call]
  E --> I[Confirm metric recovers]
  G --> I
  H --> I

Detection and quick checks

• False-positive spike: compare the current false-positive rate or precision to the trailing baseline on the same dashboard, and check whether the spike's start time lines up with a recent deploy. A deploy-timed spike is the most common cause.
• Stale features: check the feature-freshness or last-updated metric for the serving pipeline. A stale-features symptom is usually an upstream data job failure, not a model problem, and shouldn't send anyone digging into model code.

Safe mitigations a junior engineer can execute alone

• Roll back to the last known-good model version through a single documented command or console action.
• If features are stale, fail open to a cached or default feature set rather than blocking legitimate traffic on missing data.
• If it's neither a recent deploy nor staleness, conservatively raise the decision threshold to cut false-positive volume while paging the ML on-call for root cause.

When to page immediately

Page right away if the metric doesn't recover after rollback or fail-open, or if it keeps degrading despite the mitigation; that's the signal this is outside the runbook's scope.

Worked example

The trailing seven-day baseline false-positive rate sits steady around 2.1%. During the incident window it reads 7.4%, which is $7.4/2.1 \approx 3.5\times$ the baseline, well past the runbook's "far more false positives than normal" trigger. The responder checks the deploy log first: a model deploy shipped 40 minutes before the spike started. Per the triage flow, that routes to the rollback branch: the responder executes the documented rollback to the previous model version and pages the ML on-call to confirm recovery and investigate what the failed deploy actually changed.

Trade-offs and pitfalls

• A runbook that requires model literacy to execute defeats its own purpose; every step must be doable without understanding what the model predicts or why.
• Raising the decision threshold cuts false positives immediately but also increases false negatives on whatever the model is meant to catch. The runbook should say this trade-off explicitly and mark it as a temporary measure, not a fix.
• Rolling back without confirming the previous model version is still compatible with the current feature schema can turn one incident into two; the rollback step should name which versions are known-compatible, not just "the last one."

The Incident Commander (IC) owns the incident, not the fix. Their job is coordination and decision-making, deciding priorities, deciding when to escalate, keeping the response moving, while the technical work of diagnosing and fixing the problem belongs to subject matter experts (SMEs) the IC coordinates but doesn't have to be one of. That separation is the whole point: it lets someone stay focused on the shape of the response instead of getting pulled into a single technical rabbit hole.

Roles and how they differ

Why the IC role has to be distinct from the SME role

If the IC is also the person elbow-deep in a stack trace, two things suffer at once: the technical dive doesn't get their full attention, and no one is watching the overall picture (are we escalating too slowly, is communications falling behind, has severity changed). Separating the roles means the IC can pull in a second or third SME without needing to personally understand every system, and can make a call like "stop investigating, roll back now" even when an SME would rather keep digging for the root cause, because the IC's job is minimizing impact, not necessarily finding the deepest explanation in the moment.

Escalation triggers and handoffs

An IC should escalate (bring in a more senior IC, or a specific SME) when the current responder hits the edge of their authority or context: severity increasing beyond what the current team can safely own, the fix requiring a decision (like a risky rollback) above the current IC's authorization level, or the incident running long enough that fatigue is a real risk. Handoffs between ICs during a long-running incident follow the same discipline as an on-call shift handoff: the outgoing IC states current status, open decisions, and what's already been tried, in the incident channel, with the incoming IC explicitly confirming they've taken over before the outgoing IC steps back. An IC handoff that happens silently, with no explicit confirmation, is a common source of dropped context in long incidents.

Trade-offs and pitfalls

On a small team, it's tempting to skip a dedicated IC and let the most senior engineer both fix and coordinate; that works for short, simple incidents but breaks down exactly when it matters most, a complex, multi-team incident, because that's when coordination and deep technical focus can no longer be done well by the same person at once. The other common pitfall is an IC who defers every decision back to SMEs instead of actually deciding, which turns the incident into a discussion instead of a response; the IC's authority to make the call, even an imperfect one, quickly, is the actual value of the role.

MTTD is how long a problem existed before anything noticed it. MTTA is how long a human took to acknowledge the alert once it fired. MTTR is how long it took to fully resolve once someone was working it. The most common mistake is reporting a single blended average and treating it as typical, when one long outage in the set is doing all the work.

Definitions

(Some teams instead measure MTTR from detection to resolve rather than ack to resolve; either is defensible, but the convention has to be fixed and stated, because mixing them across teams silently changes what the number means.)

Worked example: one incident timeline

Worked example: averaging across three incidents, and where it goes wrong

The mean of 36.7 minutes is being pulled up almost entirely by incident 3's 54-minute outlier: the mean sits above two of the three data points (20 and 36), with only the outlier itself larger. The median of {20, 36, 54} is 36, the middle value itself rather than a value inflated by the outlier, so it is a better single-number stand-in for the typical incident than the mean. Reporting mean MTTR alone, without the incident count or a percentile, makes a single bad incident look like the typical case.

Trade-offs and pitfalls

Comparing MTTR across teams that use different start-point conventions is comparing two different metrics wearing the same name; agree on the convention org-wide before benchmarking teams against each other. A dropping mean MTTR can hide a rising incident count: if you're resolving more small incidents faster while one rare severe incident still takes hours, the mean improves and the tail risk hasn't moved at all. Improving MTTD without improving MTTA or MTTR just means you find out about the same slow response faster; treat the three as stages of one pipeline, not independent wins to report separately.

A credible automation case has three parts: the annual cost of doing the task manually (engineer time plus any revenue or SLA impact), the fully loaded cost of building and maintaining the automation, and a residual-risk line for when the automation itself misfires. Turn those into a payback period and an annual ROI, then restate the same numbers for a non-technical audience as "what we spend today versus a one-time investment that pays for itself in N months."

Inputs the model needs

The formula

Worked example: a noisy alert that fires 300 times a year

Assumptions (pinned): F = 300/year, H = 0.4 hours, C = $75/hr, R = $100/incident, D = $20,000, N = 2 years, M = $4,000/year, p = 2%, C_f = $175/misfire.

Translating it for a non-technical stakeholder

The pitch a CFO or exec sponsor needs is the same numbers, stripped of the formula: "this task currently costs the team about $39k a year in engineer time and SLA impact. A $20k investment, amortized over two years plus $4k a year to keep it working, pays for itself in about ten months and then keeps saving roughly $24k every year after that, with a 2% chance any given automated run still needs a human to clean up after it." Leading with the payback date and the residual-risk number, not just the savings, is what makes the pitch credible instead of salesy.

Trade-offs and pitfalls

If F is seasonal or unstable, a single annual estimate is misleading; revisit the model quarterly rather than trusting a number computed once. Don't drop the residual-failure term to make the case look better: an automation that works 98% of the time but corrupts state on the other 2% can be a net negative if C_f is large, especially for destructive actions. A common wrong turn is pitching automation using only "engineer-hours saved" and skipping maintenance and residual-risk costs entirely; that consistently overstates ROI and makes the next proposal harder to trust once someone notices the gap.

Direct answer

What changes between internal and external incident communication is technical depth and certainty: engineering updates include specifics, hypotheses, exact systems and logs, mitigation steps, because that audience can act on them, while external updates stick to confirmed customer-visible impact and next-update timing, because speculation shared with customers erodes trust if it turns out wrong. What stays the same is cadence discipline and honesty: both audiences get updates on a predictable schedule, and neither gets a false ETA.

Structured elaboration

Tailoring further by audience

Enterprise customers with contractual SLAs often get a more detailed, sometimes one-to-one update from their account team in addition to the public status page, while free-tier customers rely on the public page alone. The underlying facts should be identical; only the delivery channel and level of individual attention differ. Escalation also has its own trigger: open a live bridge call and involve leadership once the outage crosses a defined severity or duration threshold, such as a sustained top-severity incident running longer than the org's own SLA commitment, not just because internal chat is busy.

What never changes regardless of audience

No blame is assigned, internally or externally, while the incident is still active; naming a cause before it's confirmed just gets walked back later. No ETA gets promised that isn't actually known; "investigating" is an honest status, a fabricated timeline is not.

Worked example

Internal update at the 15-minute mark: "checkout-service 5xx rate is elevated since 14:02 UTC, correlates with a config deploy at 13:58; on-call is rolling back now, expect confirmation in a few minutes; questions in the incident channel." External status page at the 30-minute mark, built only from the confirmed facts in that internal update: "We're investigating an issue causing checkout errors for a subset of users. Our team has identified a likely cause and is applying a fix. Next update in 30 minutes." The external version omits the specific deploy detail, since it isn't customer-actionable and would need walking back if the rollback doesn't fully resolve things, but keeps the same cadence commitment as the internal update.

Trade-offs and pitfalls

• Copy-pasting the internal technical update into the external channel either leaks unnecessary detail or reads as more alarming than the plain-language version would.
• More frequent external updates build trust but risk announcing something not yet confirmed if the situation is still moving fast; anchor the cadence to confirmed milestones, not the clock alone.
• Skipping communications or product sign-off on external messages risks a technically accurate but poorly worded update going out while everyone is heads-down on the actual fix.