Incident Response and Management Questions

Direct answer
A production incident moves through seven phases: preparation, detection, triage, containment, mitigation, recovery, and post-incident review. Preparation happens before anything breaks (runbooks written, on-call staffed, alerts wired up); detection is the moment you learn something is wrong; triage scopes and prioritizes it; containment stops it from getting worse; mitigation reduces the pain customers feel; recovery restores full normal service; and the post-incident review turns the experience into a lasting fix. Each phase has a different owner, a different artifact, and a different question it answers.

Structured elaboration

• Preparation. Activity: writing and testing runbooks, defining on-call rotations, wiring alerts to real signals. Artifact: the runbook itself and the on-call schedule. Owner: the team that runs the service, done continuously, not reactively.
• Detection. Activity: an alert fires or a human notices something is off. Artifact: the alert or the first ticket. Owner: whoever is paged, or whoever notices first.
• Triage. Activity: scoping how bad it is and who needs to know. Artifact: an incident ticket with severity, scope, and a first status note. Owner: the first responder, sometimes handed to an incident commander for anything large.
• Containment. Activity: stopping the blast radius from growing (isolating a host, throttling a bad client, disabling a feature flag). Artifact: a decision log entry noting what was done and why. Owner: whoever is closest to the failing component.
• Mitigation. Activity: making the customer-visible symptom smaller even before the root cause is fixed (failing over, serving cached data, degrading gracefully). Artifact: an updated status note describing customer impact before and after. Owner: the responder or incident commander.
• Recovery. Activity: restoring full functionality and confirming it holds, not just that one metric blipped green. Artifact: a recovery validation checklist and the all-clear message. Owner: the responder, with sign-off from anyone whose data or workflow was affected.
• Post-incident review. Activity: reconstructing the timeline, finding the root cause, and turning it into owned action items. Artifact: the postmortem document. Owner: usually the incident lead, with input from everyone involved.

Worked example
An API starts returning 5xx errors to 15% of traffic. Preparation already exists: there's a runbook for 'elevated 5xx rate' and an on-call SRE. Detection: a synthetic check pages the on-call engineer. Triage: the engineer opens a ticket, sees the error rate and which endpoints are affected, and judges this a SEV2. Containment: they notice the errors correlate with a recent deploy and freeze further deploys to that service so nothing else changes mid-investigation. Mitigation: they roll back the deploy, which drops the error rate from 15% to under 1% within two minutes. Recovery: they watch the error rate and latency stay at baseline for 20 minutes before declaring the incident resolved, since a single good data point after a rollback isn't proof the fix held. Post-incident review: a review two days later finds the deploy introduced a null-pointer bug in an edge case, and the action items are a missing test case plus a canary step that would have caught it before full rollout.

Trade-offs and pitfalls
The most common mistake is skipping straight from detection to mitigation without a real triage step, which means responders end up mitigating the wrong thing or missing that three separate alerts are actually one incident. The second common mistake is calling recovery too early: one healthy-looking dashboard refresh is not the same as a service that has held steady long enough to trust. A third, subtler pitfall is treating containment and mitigation as the same step; containment is about preventing the problem from spreading (a freeze, an isolation), while mitigation is about reducing what customers currently feel (a rollback, a failover) - conflating them means teams sometimes stop at containment and believe the incident is handled when customers are still seeing errors.

Direct answer
In the first 15 to 30 minutes: acknowledge the page immediately so others know someone is on it, check your two or three primary signal sources (error-rate dashboard, recent deploys, recent config changes) to get a rough sense of scope and cause, apply an obviously safe temporary mitigation if one exists, post a first status update even if it just says 'investigating,' and decide whether you need more people. What you don't do: silently investigate alone for the whole window, or start deep root-causing before you know how big the problem is.

Structured elaboration

• Acknowledge and orient (minute 0-2). Confirm the page is real, not a flapping alert, and check the primary dashboard to see current error rate, latency, and traffic. This tells you if you're dealing with a full outage or a partial degradation.
• Check for an obvious cause (minute 2-8). Look at what changed recently: deploys, config pushes, feature flag flips, infrastructure changes. Most production incidents correlate with a recent change, and this is the highest-value early check.
• Decide on scope and escalation (minute 5-10). Based on what you've seen, decide if this affects one service or many, one region or all, and whether you need to page anyone else. Getting help early is cheap; struggling alone for 25 minutes before asking is expensive.
• Apply a safe temporary action if available (minute 10-20). If a recent deploy correlates with the problem, rolling it back is usually the safest first move, since it's typically reversible. If there's no obvious cause, avoid guessing with an irreversible action.
• Post a first status update (by minute 15-ish). Even 'we're aware, investigating, will update in 15 minutes' is far better than silence, both for stakeholders and for your own discipline of staying on a cadence.
• What not to do: don't start a deep root-cause investigation before you understand the blast radius; don't make an irreversible change (a manual database edit, a permanent config change) under time pressure without a second person's input; don't go quiet.

Worked example
Page fires at 14:02 for elevated error rate on the checkout service. 14:02-14:03: engineer acknowledges, opens the dashboard, sees error rate at 8% (up from a normal 0.1%), affecting checkout only, not the whole site. 14:03-14:06: checks recent deploys, finds one shipped at 13:55, seven minutes before the alert. 14:06-14:08: decides this is likely deploy-related, pages a second engineer to help verify while they prepare a rollback, and opens an incident ticket marked SEV2 (partial, revenue-affecting feature). 14:08-14:11: rolls back the deploy. 14:11-14:14: watches error rate drop back to 0.1% and holds there. 14:15: posts 'checkout errors have returned to normal following a rollback of a recent deploy; monitoring to confirm and will follow up with a summary.'

Trade-offs and pitfalls
Spending the first 15 minutes purely gathering information without acting risks letting an easily-mitigated problem run longer than necessary; acting immediately without any scoping risks mitigating the wrong thing entirely, or worse, taking an action whose blast radius you don't understand. The 'lone hero' pattern, where a responder tries to fully resolve the incident single-handedly before looping anyone in, is a recurring failure mode: it delays getting a second set of eyes on a decision made under pressure, and it delays stakeholder communication that people are actively waiting on.

Direct answer
An incident is unplanned degradation you did not choose the timing of, and it demands an immediate, ad hoc response. A change is planned work you scheduled, reviewed, and can roll back on your own terms. The core difference is control: with a change you set the clock and the safety net in advance; with an incident, both are decided under pressure, in real time.

Structured elaboration

• Response process. A change follows a pre-agreed plan (a rollout schedule, a rollback procedure written before you started). An incident has no such plan available; the responder is improvising against a runbook at best, from first principles at worst.
• Communication expectations. A change is usually announced in advance ('deploying at 2pm, expect brief latency') and confirmed complete afterward. An incident communication starts reactively ('we're aware of X, investigating') and needs a cadence of updates because nobody agreed to this timing.
• Approvals. A change typically needs sign-off before it happens (a peer review, a change-advisory step for higher-risk changes). An incident response needs no advance approval to act, since delay itself has a cost, though bigger interventions (a full rollback, a customer-facing statement) may still need someone with authority to say go.
• Post-activity documentation. A completed change gets a short record that it happened and worked as intended. An incident gets a fuller review, because the whole point is extracting a lasting lesson from something nobody planned for.

Worked example
A team schedules a canary rollout of a new caching layer for 2pm, reviewed and approved the day before, with an automatic rollback if error rate crosses a threshold. That's a change: planned, approved, monitored against a pre-set safety trigger. At 2:15pm the canary's error rate stays low but an unrelated dependency the caching layer talks to starts timing out, and the on-call engineer gets paged for a 5xx spike unrelated to the rollout. That's now an incident: nobody scheduled it, there's no pre-agreed plan for this specific failure, and the engineer has to improvise triage in real time, even though it happened to start during a change window.

Trade-offs and pitfalls
A common failure mode is applying incident-level ceremony to every change, which breeds change fatigue and makes people route around the process. The opposite failure is worse: not declaring an incident when a change goes wrong, often because the team feels responsible ('we did this to ourselves, let's just quietly fix it') and skips the visibility, communication, and review that an incident would otherwise get. A bad outcome caused by your own planned work is still an incident and deserves the same rigor as one caused by anything else.

Direct answer
When the root cause lives in a service another team owns, my first move is getting the right person from that team engaged directly and fast, usually by paging their on-call rather than routing through a manager, and then working in parallel rather than blocking: I keep making progress on whatever I can control while they investigate their side.

Structured elaboration

• Get the right person, not just any person. Page the owning team's on-call directly if your investigation clearly points at their service, rather than escalating through management layers that add delay without adding expertise.
• Be specific about what you need from them. Rather than a vague 'something's wrong with your service,' share exactly what you've observed and why you believe the root cause is there, which lets them start from your findings instead of re-deriving them from scratch.
• Work in parallel, not sequentially. While the owning team investigates their side, continue anything you can independently do on your side (further mitigation, additional monitoring, keeping stakeholders updated), rather than sitting idle waiting for their update.
• Don't take over their system without context. Even if you technically have the access to poke at their service directly, doing so without their domain knowledge risks causing a second problem; the better move is close collaboration, not unilateral action on a system you don't own.
• Don't silently wait either. If you've reached out and haven't heard back within a reasonable window given the severity, escalate again rather than assuming they're already on it.

Worked example
An incident's root cause traces to a shared authentication service owned by a different team. Rather than waiting for a formal handoff process, the responder directly pages that team's on-call with specific findings ('auth requests from our service are timing out starting at 14:02, correlating with your deploy at 13:58'), which lets the other team's engineer start investigating their deploy immediately rather than starting from scratch. While waiting, the original responder adds a client-side retry with backoff on their own service as a partial mitigation, something within their own control, rather than being fully blocked on the other team's fix.

Trade-offs and pitfalls
The most common failure here is silently waiting on the other team without actively escalating, which can leave an incident stalled far longer than necessary if that team is slow to notice or prioritize it. The opposite failure, someone outside the owning team taking matters into their own hands and directly modifying a system they don't fully understand, risks introducing a second, unrelated incident on top of the first. The right balance is proactive, specific engagement paired with continuing to make progress on what you do control, rather than either extreme.

Direct answer
My immediate response has three simultaneous threads: contain and mitigate the technical impact, communicate what I actually know (not what I'm guessing) on a steady cadence, and validate that the fix genuinely held before standing down. None of these wait for the others; a responder who only does the technical work and goes quiet for six hours has failed the incident just as much as one who only communicates and never fixes anything.

Structured elaboration

• Contain and mitigate. First isolate whatever is failing so it can't spread (pull a bad instance from rotation, disable a feature flag, freeze further changes), then apply the fastest safe action that reduces customer pain, which is very often a rollback to the last known-good state if a recent change correlates with the onset.
• Communicate on a cadence, not just when there's news. Post a first update as soon as you have a rough sense of scope, even if it's just 'investigating, will update in 15 minutes.' Keep to that cadence even when the honest update is 'still investigating,' because silence reads as worse than an uneventful update, and stakeholders making their own decisions (support teams, other engineering teams, leadership) need to know what you know, not what you hope.
• Validate real recovery before standing down. Don't declare resolved on the first green data point; watch the key metrics hold at baseline for a sustained window, and get explicit confirmation from anyone whose data, transactions, or workflows were affected before formally closing.
• Handle degraded evidence gracefully. Sometimes your own tooling is part of the problem: if the logging or monitoring pipeline itself is down, you're triaging with less information than normal. In that case fall back to direct, lower-level checks (host-level metrics, spot-checking a handful of real requests, canary user reports) and be more conservative about declaring anything resolved, since you're flying with reduced visibility.
• Some failures don't have a clean single-instance fix. A storage array with a degraded RAID rebuilding for 12 hours, or a network partition that's caused two database replicas to diverge (a split-brain), both require deliberately not rushing: forcing a failover mid-rebuild can worsen data loss, and blindly failing over during a split-brain risks writing to the wrong side of a diverged pair. The instinct to 'just fail over' has to be checked against whether failover itself is currently safe.

Worked example
A six-hour outage: a database failover during peak traffic silently sends a subset of write traffic to a replica that hadn't fully caught up, so some writes appear to succeed but are later lost on failback. Immediate response: the on-call engineer confirms the scope (which customers, which write types affected) within the first 20 minutes and posts a first update. Containment: they freeze automated failover so the system can't flip again mid-investigation. Mitigation: they route new writes exclusively to the confirmed-primary replica, restoring correct behavior for new activity even though the earlier lost writes aren't yet recovered. Communication: updates go out every 30 minutes to internal stakeholders and, once the customer-facing impact is confirmed, to affected customers, describing what's known and what's still being investigated, not speculating about root cause before it's confirmed. Recovery: after the fix, the team monitors write consistency across replicas for two hours (not two minutes) before declaring the incident resolved, given how deceptive an initially-healthy-looking replica had already proven to be.

Trade-offs and pitfalls
The most senior-discriminating mistake here is premature stand-down: declaring victory on the first healthy-looking check when the underlying problem (like replica divergence) can silently resurface. A second common mistake is communicating guesses as facts under pressure ('this was caused by X') before root cause is actually confirmed, which erodes trust when the real cause turns out to be different. A third is treating every outage the same way regardless of its actual shape: a clean single-service outage tolerates a fast rollback, while a data-divergence or hardware-degradation scenario often calls for patience and extra verification precisely when the instinct is to act fastest.