On-Call Practices and Runbook Design Questions

Direct answer

Wire the alert directly to a runbook execution layer, not just a link: the alert carries a runbook identifier, the paging tool surfaces the matching runbook inline with a one-click execute action for pre-approved steps, and every click passes through validation, RBAC (role-based access control, the system that decides which actions a given user is permitted to run), and policy evaluation before anything runs, with the result and a full audit entry logged back to the incident. What actually stops an unsafe action isn't the button in the UI, it's the policy layer sitting between the click and the execution.

Structured elaboration

Component flow

sequenceDiagram
  participant M as Monitoring
  participant PD as PagerDuty
  participant R as Responder
  participant E as Runbook Executor
  participant V as Audit Log
  M->>PD: Alert fires with runbook_id attached
  PD->>R: Page plus incident link
  R->>E: Open runbook, click approved step
  E->>E: Validate params, RBAC, policy check
  E->>V: Write audit entry: who, what, when
  E->>PD: Post execution result to incident

What each layer does

• Alert enrichment: alerting rules carry a runbook identifier and a list of allowed actions as labels, so the alert itself knows which remediation it maps to.
• Paging tool integration: the incident view embeds the runbook with each step clearly labeled manual versus automated, along with its risk level.
• Execution broker: on click, it validates the caller's identity through RBAC, confirms the incident is actually open and acknowledged, and evaluates policy, such as whether this action needs extra approval given current severity or a maintenance window.
• Audit log: append-only, recording who clicked, what parameters were used, what the action did, and the result, written before the action is considered complete rather than best-effort afterward.

What logs, and what actually stops an unsafe action

Nothing about the UI stops an unsafe action; the broker re-validates everything server-side even if the button already looked available. Any action tagged destructive requires approval regardless of who clicks it, a circuit breaker disables an action automatically after repeated failures, and execution always runs with short-lived, scoped credentials so even a successful unsafe action is bounded in what it can touch.

Worked example

A "restart cache node" action is clicked from the incident view. The broker checks: is the incident open, yes; is the clicker on the paging on-call roster for this service, yes; is this action tagged low-risk, yes, a single-node restart; is it within the rate limit, zero executions in the last five minutes against a cap of one. All checks pass, the executor runs the restart with a scoped, time-limited credential, and writes the audit entry, who, incident ID, action, parameters, result, before returning success to the incident view.

Trade-offs and pitfalls

• Client-side checks are fine for UX, greying out an unavailable button, but the broker must re-validate everything server-side; the client should never be the actual enforcement point.
• Real-time policy evaluation on every click adds latency to remediation; keep the policy check itself a fast local lookup rather than a call to an external system on the critical path, so safety doesn't undercut automation's main benefit.
• Logging "success" based on the action returning without error, rather than verifying the target metric actually recovered, is a common gap; a runbook step can exit cleanly and still not have fixed anything.

For a junior, half-asleep responder, the runbook needs to be a strict linear sequence of copy-pasteable commands with expected output shown at each step, not a description of the problem. A connection pool is just the fixed set of open database connections an application reuses instead of opening a new one per request; "exhausted" means every connection in that set is busy or leaked, so new requests queue or fail waiting for one to free up. Below is the runbook, followed by how the same template extends to a few other common recurring failures.

Runbook: Database connection pool exhausted

Applies when: alert fires for "DB connections at capacity" or the app logs show timeouts waiting for a pooled connection.

1. Confirm the symptom

-- Postgres: how many connections are open right now, and what state are they in
SELECT state, count(*) FROM pg_stat_activity GROUP BY state;

If active + idle in transaction is at or near the configured pool max, this is confirmed.

2. Check for the two common causes

-- Long-running or stuck queries holding a connection
SELECT pid, state, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC
LIMIT 10;

A handful of queries running for minutes, not seconds, points at a stuck query holding connections hostage. Many short connections in idle in transaction for the same app host points at a connection leak in that service (it opened a connection and never closed it).

3. Immediate mitigation

• If one or two queries are clearly stuck (duration in minutes, not part of a normal batch job): terminate them.

SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid = <PID>;

• If one app instance is leaking connections: restart that instance only, not the whole fleet, to limit blast radius.

# Example: restart a single app instance behind a load balancer
kubectl rollout restart deployment/<app-name> --namespace=<ns>

4. Verify recovery
Re-run the query from step 1. Connection count should drop back under 80% of pool max within a couple of minutes, and application error logs should stop showing connection-timeout errors.

5. Escalate if

• Step 1 confirms the symptom but step 2 shows no stuck queries and no obvious leaking host, escalate to the on-call DBA.
• Terminating queries and restarting the instance doesn't bring connections back down within 10 minutes, escalate to the on-call DBA and the service owner together.

6. Log it
Note in the incident channel: which queries or instance were involved, what you ran, and the connection count before/after. This becomes the input for the next runbook review.

Applying the same template to other recurring failures

The shape (confirm symptom, check the one or two common causes, apply a scoped mitigation, verify, escalate on a threshold) is the same regardless of what's failing:

• A worker process stuck or crash-looping: confirm via the process manager's status command, check recent logs for the crash reason, restart just that worker (not the whole host), verify it stays up for a few minutes, escalate if it crash-loops again within 15 minutes.
• A Kubernetes service unresponsive: confirm via kubectl get pods showing CrashLoopBackOff or 0 ready replicas, check kubectl describe pod for the failure reason, roll the deployment, verify readiness probes pass, escalate if the rollout itself fails.
• High database CPU: confirm via the DB's CPU metric, check pg_stat_activity for a small number of expensive queries dominating, kill the worst offender if it's not part of an expected batch job, verify CPU drops, escalate if CPU stays high with no single query responsible (that usually means aggregate load, not one query, and needs a scaling decision above a junior responder's authority).

Keeping it usable day to day, not just during an incident

Part of what keeps a junior responder capable of running this cold at 3am is having already looked at the same dashboard during calmer moments. A daily on-call checklist (glance at connection count, query latency, and error rate once per shift, even with no alert firing) means the responder already knows what "normal" looks like on this system before the night it matters, instead of learning it for the first time under a page.

Trade-offs and pitfalls

The runbook deliberately doesn't cover permanent fixes (increasing pool size, fixing the leak in application code, adding query timeouts), those are follow-up work items, not something a junior responder should do live against production under a page; bundling them in would tempt someone under pressure into a riskier change than the incident calls for. The other pitfall is giving a vague duration threshold like "long-running query"; "minutes, not seconds" is deliberately concrete so a junior responder isn't left guessing whether a 40-second query counts.

A production incident moves through seven stages: detection, triage, investigation, escalation, remediation, recovery validation, and postmortem. Each stage has a specific job and produces a specific artifact, and skipping the artifact (not just the activity) is usually where lifecycle discipline actually breaks down, a team can do all the right actions and still lose the value if nothing gets written down for the next stage or the next incident to build on.

The lifecycle

flowchart LR
  A[Detection] --> B[Triage]
  B --> C[Investigation]
  C --> D[Escalation]
  D --> E[Remediation]
  E --> F[Recovery validation]
  F --> G[Postmortem]
  D -.->|if needed| C

Escalation isn't strictly linear, an investigation can loop back through another round after escalating in a specialist, which is why that arrow feeds back rather than only forward.

Stage by stage

Why the artifacts matter as much as the actions

An incident that's handled well but produces no artifacts is invisible to everyone who wasn't in the room: the next on-call engineer can't learn from it, the postmortem has no evidence trail to work from, and "what actually happened" becomes reconstructed memory instead of a timeline. The scribe role during a live incident exists specifically to make sure the investigation and remediation artifacts get captured in real time, because reconstructing a timeline after the fact from memory is unreliable and slow.

Blameless postmortem practice

The postmortem stage only works if it stays blameless in substance, not just in name: it asks what allowed the incident to happen (a missing safeguard, a gap in monitoring, an undocumented dependency) rather than who made the mistake. A postmortem that names an individual as the cause teaches everyone watching to hide problems rather than surface them, which is the opposite of what the stage exists to produce. The concrete output is a short list of action items, each with a named owner and a tracked ticket, not a narrative that ends with "we'll be more careful."

Trade-offs and pitfalls

Some teams try to skip investigation and jump straight to remediation under pressure, which sometimes works for well-understood failure modes but risks applying the wrong fix when the pattern only looks familiar. The more common failure across the whole lifecycle is letting the postmortem's action items go untracked once the adrenaline of the incident fades, an incident that produces a postmortem with no completed follow-through is functionally the same as one with no postmortem at all, just with more paperwork.

Compensation and schedule design are two separate levers, and both need to move: pay a base standby stipend for availability, add per-incident pay or time-off-in-lieu for the work actually done, and size the rotation and rest guarantees so the schedule itself isn't relying on money to make an unsustainable load tolerable.

Compensation model components

Worked example: one on-call week

Stipend $250 + 6 hours of actual incident work at $40/hr:

If the same 6 hours bank as TOIL at a 1.5x rate for after-hours work:

Schedule practices that reduce the load pay has to compensate for

Primary/secondary tiers so one page doesn't always land on the same person; a cap on consecutive on-call weeks per engineer; shorter rotations (fewer consecutive days of stress, more handoffs) traded against longer rotations (fewer handoffs, more concentrated fatigue), sized to team headcount rather than picked arbitrarily; and follow-the-sun coverage once the team is large and distributed enough to make timezone handoffs cheaper than overnight pages.

Trade-offs and pitfalls

Per-incident pay can invite gaming in both directions, either padding logged hours or under-reporting to avoid looking like a "high maintenance" service; review incident-hour claims against the paging log rather than trusting self-reports alone. Contractors and salaried employees often need different structures (cash versus TOIL), and a single model rarely fits both cleanly. Regional labor law varies significantly, some jurisdictions treat standby time itself as compensable working time, so confirm with legal or HR before setting a global policy rather than assuming one region's rules generalize. There's no dominant answer on rotation length; it has to be sized to team size and incident frequency, not copied from another team's policy.

A credible automation case has three parts: the annual cost of doing the task manually (engineer time plus any revenue or SLA impact), the fully loaded cost of building and maintaining the automation, and a residual-risk line for when the automation itself misfires. Turn those into a payback period and an annual ROI, then restate the same numbers for a non-technical audience as "what we spend today versus a one-time investment that pays for itself in N months."

Inputs the model needs

The formula

Worked example: a noisy alert that fires 300 times a year

Assumptions (pinned): F = 300/year, H = 0.4 hours, C = $75/hr, R = $100/incident, D = $20,000, N = 2 years, M = $4,000/year, p = 2%, C_f = $175/misfire.

Translating it for a non-technical stakeholder

The pitch a CFO or exec sponsor needs is the same numbers, stripped of the formula: "this task currently costs the team about $39k a year in engineer time and SLA impact. A $20k investment, amortized over two years plus $4k a year to keep it working, pays for itself in about ten months and then keeps saving roughly $24k every year after that, with a 2% chance any given automated run still needs a human to clean up after it." Leading with the payback date and the residual-risk number, not just the savings, is what makes the pitch credible instead of salesy.

Trade-offs and pitfalls

If F is seasonal or unstable, a single annual estimate is misleading; revisit the model quarterly rather than trusting a number computed once. Don't drop the residual-failure term to make the case look better: an automation that works 98% of the time but corrupts state on the other 2% can be a net negative if C_f is large, especially for destructive actions. A common wrong turn is pitching automation using only "engineer-hours saved" and skipping maintenance and residual-risk costs entirely; that consistently overstates ROI and makes the next proposal harder to trust once someone notices the gap.