Incident Response and Management Questions

Direct answer
For a CPU spike causing timeouts: containment is isolating the affected host or throttling the traffic causing the spike so the problem stops spreading to healthy instances; mitigation is shedding non-critical load or scaling out so timeouts stop for most users even before you know the root cause; recovery is restoring the service to its normal capacity and confirming latency has genuinely returned to baseline, not just dropped once.

Structured elaboration

• Containment answers 'how do I stop this from getting worse.' For the CPU spike: pull the overloaded instance out of the load balancer pool so it stops receiving new traffic and can't drag healthy instances down with it (for example through a shared connection pool or retry storm). Safety check: confirm the load balancer has enough remaining healthy capacity before pulling the instance, or you've just made the outage worse for everyone else.
• Mitigation answers 'how do I reduce what customers feel right now,' independent of understanding the root cause yet. For the CPU spike: shed non-critical requests (defer background jobs, disable an expensive feature) or scale out horizontally to spread load. Safety check: shedding load has to be reversible and clearly logged, so nobody forgets a feature is disabled after the incident ends.
• Recovery answers 'is the service actually back to normal.' For the CPU spike: bring capacity back to its normal level, watch CPU and latency hold at baseline for a sustained window, and re-enable anything that was shed. Safety check: recovery isn't declared on a single good data point, since a transient dip can look like recovery for a minute and then relapse.

Worked example
Suppose CPU on the affected instance is pinned at 95% and p99 latency has risen from a normal 120ms to 4 seconds, causing client-side timeouts. Containment: the on-call engineer removes the instance from the load balancer pool, redistributing its traffic across the remaining healthy instances. Mitigation: because the remaining instances are now each carrying more load, they scale out from 4 to 6 instances and temporarily disable a CPU-heavy recommendation feature; CPU across the fleet drops to a more sustainable 60% and p99 latency returns to around 300ms, well below the timeout threshold even if not yet fully back to baseline. Recovery: once the root cause (an inefficient query introduced by a recent deploy) is identified and fixed, the team restores the original instance count, re-enables the recommendation feature, and watches p99 latency hold near 120ms for 30 minutes before declaring the incident resolved.

Trade-offs and pitfalls
Containment that's too aggressive (pulling too many instances, or isolating a component that other services still depend on) can turn a partial degradation into a full outage; containment that's too narrow (missing that the real blast radius includes a shared resource like a connection pool or cache) leaves the spread unaddressed. Mitigation can also mask the symptom in a way that delays real diagnosis: shedding load makes CPU look healthy again, but if nobody tracks that the shed feature is still disabled, the team can lose track of what 'fully recovered' actually means. The general pattern (same containment options: read-only mode, feature toggles, traffic shaping, or temporary scaling) applies just as well to a database write-outage as it does to a CPU spike; the phase you're in, not the specific technology, determines what action is appropriate.

Direct answer
A production incident moves through seven phases: preparation, detection, triage, containment, mitigation, recovery, and post-incident review. Preparation happens before anything breaks (runbooks written, on-call staffed, alerts wired up); detection is the moment you learn something is wrong; triage scopes and prioritizes it; containment stops it from getting worse; mitigation reduces the pain customers feel; recovery restores full normal service; and the post-incident review turns the experience into a lasting fix. Each phase has a different owner, a different artifact, and a different question it answers.

Structured elaboration

• Preparation. Activity: writing and testing runbooks, defining on-call rotations, wiring alerts to real signals. Artifact: the runbook itself and the on-call schedule. Owner: the team that runs the service, done continuously, not reactively.
• Detection. Activity: an alert fires or a human notices something is off. Artifact: the alert or the first ticket. Owner: whoever is paged, or whoever notices first.
• Triage. Activity: scoping how bad it is and who needs to know. Artifact: an incident ticket with severity, scope, and a first status note. Owner: the first responder, sometimes handed to an incident commander for anything large.
• Containment. Activity: stopping the blast radius from growing (isolating a host, throttling a bad client, disabling a feature flag). Artifact: a decision log entry noting what was done and why. Owner: whoever is closest to the failing component.
• Mitigation. Activity: making the customer-visible symptom smaller even before the root cause is fixed (failing over, serving cached data, degrading gracefully). Artifact: an updated status note describing customer impact before and after. Owner: the responder or incident commander.
• Recovery. Activity: restoring full functionality and confirming it holds, not just that one metric blipped green. Artifact: a recovery validation checklist and the all-clear message. Owner: the responder, with sign-off from anyone whose data or workflow was affected.
• Post-incident review. Activity: reconstructing the timeline, finding the root cause, and turning it into owned action items. Artifact: the postmortem document. Owner: usually the incident lead, with input from everyone involved.

Worked example
An API starts returning 5xx errors to 15% of traffic. Preparation already exists: there's a runbook for 'elevated 5xx rate' and an on-call SRE. Detection: a synthetic check pages the on-call engineer. Triage: the engineer opens a ticket, sees the error rate and which endpoints are affected, and judges this a SEV2. Containment: they notice the errors correlate with a recent deploy and freeze further deploys to that service so nothing else changes mid-investigation. Mitigation: they roll back the deploy, which drops the error rate from 15% to under 1% within two minutes. Recovery: they watch the error rate and latency stay at baseline for 20 minutes before declaring the incident resolved, since a single good data point after a rollback isn't proof the fix held. Post-incident review: a review two days later finds the deploy introduced a null-pointer bug in an edge case, and the action items are a missing test case plus a canary step that would have caught it before full rollout.

Trade-offs and pitfalls
The most common mistake is skipping straight from detection to mitigation without a real triage step, which means responders end up mitigating the wrong thing or missing that three separate alerts are actually one incident. The second common mistake is calling recovery too early: one healthy-looking dashboard refresh is not the same as a service that has held steady long enough to trust. A third, subtler pitfall is treating containment and mitigation as the same step; containment is about preventing the problem from spreading (a freeze, an isolation), while mitigation is about reducing what customers currently feel (a rollback, a failover) - conflating them means teams sometimes stop at containment and believe the incident is handled when customers are still seeing errors.

Direct answer
The trigger to shift from firefighting to careful investigation is reaching 'stable but not understood': the immediate symptom has stopped getting worse and key metrics are holding, which buys you the room to investigate properly instead of guessing under pressure. Conversely, if your mitigation isn't actually holding, that itself is the signal you need to go deeper immediately rather than trying another quick fix.

Structured elaboration

• Concrete trigger to slow down: error rate or the relevant health metric has returned to and held near baseline for a sustained window (not a single good data point), and there's no immediate reason to believe it will relapse.
• Concrete trigger to go deeper immediately, even mid-firefight: the same symptom keeps recurring despite mitigation attempts, which usually means you're treating a symptom rather than the actual cause, and continuing to apply quick fixes without understanding why they keep failing wastes time and erodes confidence.
• Irreversibility as a forcing function. If the action you're considering is hard to undo (a data migration, a customer-facing communication, an action that affects money or legal exposure), that pushes you toward more investigation before acting, even at the cost of a slower initial response, because getting an irreversible action wrong is worse than a slightly slower recovery.
• Confidence in your own read of the situation matters. If your working theory of the mitigation is based on a guess rather than confirmed evidence, that's itself a reason to invest in confirming it before declaring things under control, since acting on an unconfirmed theory can create a false sense of resolution.

Worked example
During a live outage with suspected data loss, the team applies an immediate mitigation (stopping the affected pipeline) within minutes. At this point error rate on downstream systems stops climbing, but that's not yet the trigger to slow down, since the underlying data-loss question is unresolved and any further action (like deciding to restart the pipeline, or attempting a repair) is hard to undo cleanly. The team spends the next chunk of time specifically investigating the scope of potential data loss and confirming, through an independent check, whether any data was actually lost or only delayed. Once that's confirmed (data was delayed, not lost, and can be safely reprocessed) and the pipeline has been safely paused for a sustained period with no further impact, the team shifts fully into the more careful, deliberate reprocessing plan rather than continuing to firefight.

Trade-offs and pitfalls
Declaring victory and shifting to leisurely investigation too early, based on a single good-looking metric, risks missing that the mitigation is fragile and the problem resurfaces once attention has moved elsewhere. The opposite mistake, staying in pure firefighting mode too long out of an instinct to 'just keep trying things' without ever stepping back to actually understand the cause, means repeated quick fixes accumulate risk (especially when each attempted fix is itself an action with its own blast radius) without making real progress. Legal or compliance constraints, and genuine uncertainty about whether a rollback is even correct, both push the decision toward more caution and more investigation before acting, even when the pressure to move fast is real.

Direct answer
An incident is unplanned degradation you did not choose the timing of, and it demands an immediate, ad hoc response. A change is planned work you scheduled, reviewed, and can roll back on your own terms. The core difference is control: with a change you set the clock and the safety net in advance; with an incident, both are decided under pressure, in real time.

Structured elaboration

• Response process. A change follows a pre-agreed plan (a rollout schedule, a rollback procedure written before you started). An incident has no such plan available; the responder is improvising against a runbook at best, from first principles at worst.
• Communication expectations. A change is usually announced in advance ('deploying at 2pm, expect brief latency') and confirmed complete afterward. An incident communication starts reactively ('we're aware of X, investigating') and needs a cadence of updates because nobody agreed to this timing.
• Approvals. A change typically needs sign-off before it happens (a peer review, a change-advisory step for higher-risk changes). An incident response needs no advance approval to act, since delay itself has a cost, though bigger interventions (a full rollback, a customer-facing statement) may still need someone with authority to say go.
• Post-activity documentation. A completed change gets a short record that it happened and worked as intended. An incident gets a fuller review, because the whole point is extracting a lasting lesson from something nobody planned for.

Worked example
A team schedules a canary rollout of a new caching layer for 2pm, reviewed and approved the day before, with an automatic rollback if error rate crosses a threshold. That's a change: planned, approved, monitored against a pre-set safety trigger. At 2:15pm the canary's error rate stays low but an unrelated dependency the caching layer talks to starts timing out, and the on-call engineer gets paged for a 5xx spike unrelated to the rollout. That's now an incident: nobody scheduled it, there's no pre-agreed plan for this specific failure, and the engineer has to improvise triage in real time, even though it happened to start during a change window.

Trade-offs and pitfalls
A common failure mode is applying incident-level ceremony to every change, which breeds change fatigue and makes people route around the process. The opposite failure is worse: not declaring an incident when a change goes wrong, often because the team feels responsible ('we did this to ourselves, let's just quietly fix it') and skips the visibility, communication, and review that an incident would otherwise get. A bad outcome caused by your own planned work is still an incident and deserves the same rigor as one caused by anything else.

Direct answer
When the problem is silently wrong output rather than downtime, the first job is bounding the blast radius in time and scope, not immediately trying to fix the pipeline. I'd find the last point where output was verifiably correct, stop the corruption from spreading further with the smallest possible change, and only then decide between rolling back to reconstruct the correct state or fixing forward and reprocessing.

Structured elaboration

• Determine the affected time window. Find the most recent known-good checkpoint by comparing current output against an independent source of truth (a snapshot, a source system, an upstream event log) and working backward until you find where they diverge. This is usually the hardest and most time-consuming part, because silent corruption by definition wasn't caught by any existing check.
• Stop it from getting worse. Pause or disable the specific step that's producing bad output (not necessarily the whole pipeline, if only one stage is at fault), so you stop adding more corrupted data while you investigate. This is a narrow, reversible action, not a fix.
• Decide rollback versus forward-fix. Roll back when you can cheaply reconstruct the correct state from an upstream source and downstream consumers haven't yet acted irreversibly on the bad data. Fix forward and reprocess when a rollback would lose legitimate data that arrived after the corruption started, or when downstream consumers have already consumed and acted on the bad output in ways that can't simply be undone by reverting the pipeline.
• A materially easier variant worth distinguishing: a pipeline job that fails outright and produces no output at all is a much simpler case than one that runs successfully and produces wrong output, because a failed job is self-evidently incomplete, while silently wrong output can go unnoticed for a long time and requires you to first prove where correctness broke down before you can fix anything.
• Communicate honestly, and only once scope is bounded. Resist announcing a scope estimate before you've actually confirmed it; an early guess that turns out to be wrong (in either direction) damages trust more than a slightly later but accurate one.

Worked example
A billing metrics pipeline has been silently under-reporting a subset of transactions for three months due to a filter condition introduced in a change that nobody flagged as a behavior change. To bound the window: compare monthly transaction counts from the pipeline's output against the upstream transaction log, and find the first month where the two diverge, narrowing the corruption window to a specific start date. To stop it from getting worse: disable the filter step (a single, reversible change) so new data stops being under-reported, without touching anything else in the pipeline. To decide rollback versus forward-fix: because the upstream transaction log still has the correct raw data for the entire affected window, a full historical backfill (reprocessing the affected months from the source) is both possible and safer than trying to patch the already-corrupted output in place, so the team chooses to reprocess rather than attempt a partial in-place correction.

Trade-offs and pitfalls
The most common mistake is rushing to fix forward before the blast radius is actually understood, which risks fixing the pipeline while leaving a chunk of already-corrupted historical data unaddressed. A close second is under-scoping the impact to avoid a harder conversation with stakeholders, announcing 'a few days affected' when the real window is months; this is worse for trust than taking longer to give an accurate number. A genuinely hard case is when downstream consumers have already made real-world decisions based on the bad data (a customer was billed incorrectly, or a business decision was made on a wrong metric); you can correct the data going forward, but you can't retroactively undo an action someone already took based on it, which is a different and harder remediation problem than a pure data-correctness fix. The same underlying challenge (bounding scope, deciding rollback vs. forward-fix, and being honest about what's still unknown) applies whether the corrupted data is a metric, a set of duplicate billing records from a faulty sync job, or a downstream executive dashboard that broke because an upstream team changed a schema without warning; the specific technical trigger varies, but the response shape does not.