Incident Response and Management Questions

Direct answer
For a CPU spike causing timeouts: containment is isolating the affected host or throttling the traffic causing the spike so the problem stops spreading to healthy instances; mitigation is shedding non-critical load or scaling out so timeouts stop for most users even before you know the root cause; recovery is restoring the service to its normal capacity and confirming latency has genuinely returned to baseline, not just dropped once.

Structured elaboration

• Containment answers 'how do I stop this from getting worse.' For the CPU spike: pull the overloaded instance out of the load balancer pool so it stops receiving new traffic and can't drag healthy instances down with it (for example through a shared connection pool or retry storm). Safety check: confirm the load balancer has enough remaining healthy capacity before pulling the instance, or you've just made the outage worse for everyone else.
• Mitigation answers 'how do I reduce what customers feel right now,' independent of understanding the root cause yet. For the CPU spike: shed non-critical requests (defer background jobs, disable an expensive feature) or scale out horizontally to spread load. Safety check: shedding load has to be reversible and clearly logged, so nobody forgets a feature is disabled after the incident ends.
• Recovery answers 'is the service actually back to normal.' For the CPU spike: bring capacity back to its normal level, watch CPU and latency hold at baseline for a sustained window, and re-enable anything that was shed. Safety check: recovery isn't declared on a single good data point, since a transient dip can look like recovery for a minute and then relapse.

Worked example
Suppose CPU on the affected instance is pinned at 95% and p99 latency has risen from a normal 120ms to 4 seconds, causing client-side timeouts. Containment: the on-call engineer removes the instance from the load balancer pool, redistributing its traffic across the remaining healthy instances. Mitigation: because the remaining instances are now each carrying more load, they scale out from 4 to 6 instances and temporarily disable a CPU-heavy recommendation feature; CPU across the fleet drops to a more sustainable 60% and p99 latency returns to around 300ms, well below the timeout threshold even if not yet fully back to baseline. Recovery: once the root cause (an inefficient query introduced by a recent deploy) is identified and fixed, the team restores the original instance count, re-enables the recommendation feature, and watches p99 latency hold near 120ms for 30 minutes before declaring the incident resolved.

Trade-offs and pitfalls
Containment that's too aggressive (pulling too many instances, or isolating a component that other services still depend on) can turn a partial degradation into a full outage; containment that's too narrow (missing that the real blast radius includes a shared resource like a connection pool or cache) leaves the spread unaddressed. Mitigation can also mask the symptom in a way that delays real diagnosis: shedding load makes CPU look healthy again, but if nobody tracks that the shed feature is still disabled, the team can lose track of what 'fully recovered' actually means. The general pattern (same containment options: read-only mode, feature toggles, traffic shaping, or temporary scaling) applies just as well to a database write-outage as it does to a CPU spike; the phase you're in, not the specific technology, determines what action is appropriate.

Direct answer
For a sudden drop in a key business metric, the fastest useful triage is checking, in order: is this a real business event, a data-pipeline problem, or an instrumentation or dashboard bug, because each has a completely different next step and a completely different message to leadership. I start by cross-checking against a second, independent data source before saying anything definitive.

Structured elaboration

• Rule out instrumentation first, cheaply. Check whether the dashboard or tracking pixel itself is broken (a recent change to the dashboard, a tagging or tracking change) before assuming the underlying business reality actually changed; this is often the fastest check and, if true, means there's no real business problem at all.
• Check pipeline health next. Look at whether the data pipeline feeding the dashboard actually ran, ran on time, and ran completely; a stalled or partially-failed pipeline job can produce a metric that looks like a real drop but is actually just incomplete data.
• Cross-check against an independent source. Compare the metric against a different system that measures something related (a payments processor's own numbers, a different analytics tool, raw transaction logs) rather than relying on a single dashboard; agreement across independent sources is the strongest signal you're looking at a real event.
• Loop in the right people based on what you find, not before: if it's confirmed a real business event, engineering, the business owner, and possibly leadership need to know quickly; if it's a data or instrumentation issue, it's primarily an engineering fix and the messaging to leadership should say so clearly rather than implying a real business problem.
• What to tell decision-makers while still investigating: be explicit about what's confirmed versus what's still unknown ('the drop is real and confirmed by two independent sources, we don't yet know the cause' is very different from 'we're not sure if this is even real yet'), since leadership may make decisions based on what you tell them.

Worked example
An executive dashboard shows revenue down 60% at 8am. First check: was the dashboard or its underlying tagging changed recently? No. Second check: did the pipeline feeding it run on schedule and completely? It ran, but a spot check shows the row count for the past few hours is lower than a typical morning, suggesting a partial data issue rather than a true 60% revenue collapse. Third check: compare against the payments processor's own transaction volume for the same window, which shows only a minor, explainable dip, not a 60% drop. Conclusion: this is a data pipeline problem, not a real business event. Communication to leadership: 'the dashboard's 60% drop is not reflecting reality; real transaction volume from an independent source shows normal activity; we've identified a pipeline issue and are fixing the data, not the business.'

Trade-offs and pitfalls
The biggest risk is announcing 'revenue is down 60%' to leadership before confirming it's real, which can trigger disproportionate reactions (panic, hasty decisions) that are worse than the actual underlying problem. The opposite risk, taking too long to 'double-check' a genuinely real and urgent business event, delays a response that might actually need fast leadership attention. The calibration is doing the cheap checks (instrumentation, pipeline health) fast, since those rule out the most common false alarms quickly, before spending more time on a deeper investigation.

Direct answer
My default is to build a temporary mitigation myself rather than wait, unless the vendor's ETA is both short and credible, because customer experience shouldn't depend on a timeline I don't control and can't verify. The key criteria are how reliable the vendor's own status communication has historically been, how cheap and low-risk a mitigation is to build, and whether building a mitigation might itself mask a vendor problem that needs to stay escalated.

Structured elaboration

• Assess the vendor's ETA credibility. Vendors are often optimistic about their own timelines; check their status page's track record, whether they've given a specific, committed time or a vague 'we're looking into it,' and weigh that against your own tolerance for continued impact.
• Assess mitigation cost and risk. A cheap, well-understood mitigation (a circuit breaker to fail fast instead of hanging, a cached fallback response, a degraded-but-functional mode) is usually worth building even for a short outage; a mitigation that requires new, untested code under time pressure carries its own risk and might not be worth it for a vendor issue expected to resolve in minutes.
• Don't let your mitigation hide a problem that still needs escalating. If you build a good enough fallback, make sure someone is still tracking and escalating the underlying vendor issue; a well-built mitigation can quietly make an important problem invisible to leadership or to the vendor relationship owner.
• Verify the vendor issue is what you think it is, not just assumed: check the vendor's own status page or reach out directly rather than purely inferring from your own symptoms, since 'looks like a vendor is down' and 'confirmed the vendor is down' warrant different confidence levels.

Worked example
A third-party payment processor starts intermittently failing, causing checkout errors for a subset of users. The vendor's status page shows 'investigating' with no ETA, and their historical incidents have often run longer than initially communicated. The team decides not to wait: they add a circuit breaker so failing calls fail fast rather than hanging and degrading the whole checkout flow, and enable a secondary, lower-priority payment path they'd already built for exactly this kind of situation. They keep the primary vendor issue actively tracked and continue monitoring the vendor's status page, so once the vendor recovers they can cleanly revert to the primary path, rather than letting the fallback silently become permanent.

Trade-offs and pitfalls
Building a mitigation under pressure risks shipping throwaway code that never gets properly cleaned up and becomes unplanned permanent technical debt; it's worth explicitly flagging a mitigation as temporary and following up after the incident. Waiting on a vendor whose communicated ETA turns out to be optimistic (vendors very often are) means your customers experience longer impact than necessary, purely because you trusted someone else's timeline you had no way to verify. The same underlying judgment applies whether the failing dependency is a payment processor, a cloud provider's specific service, or a network transit provider throttling traffic to a region: the calibration is always about ETA credibility versus mitigation cost, not about the specific vendor.

Direct answer
A strong answer to this needs to show calm, specific ownership: what you actually noticed, what you actually did in triage and containment (not a vague 'we fixed it'), how you kept people informed while you were still working the problem, and a concrete, real follow-up change that came out of it, not just a general lesson learned.

Structured elaboration
Walking through the STAR structure and what an interviewer is actually listening for at each part:

• Situation. Set up enough context to matter: what service, what kind of impact, roughly how severe. Interviewers are listening for whether you can quickly orient someone else, since that's exactly the skill needed during a real incident.
• Task. Your specific role and responsibility in that moment, not the whole team's. Were you the first responder, the person who happened to notice, someone brought in partway through? This matters because it sets the frame for whether the actions you describe next were really yours.
• Action. This is the section that carries the most weight, and it needs actual specifics: what you checked first, what hypothesis you formed and how you tested it, what mitigation you applied and why you chose it over alternatives, and specifically how you communicated (to whom, how often, what you said versus what you didn't yet know). Vague answers ('I investigated the issue and resolved it') are the single most common weakness here.
• Result. The concrete outcome: how long it took, what the actual resolution was, and critically, what changed afterward as a genuine consequence, whether that's a new test, a new alert, a runbook that didn't exist before, or a process change. A result without a real follow-up change suggests the lesson wasn't actually internalized.

Worked example
An illustrative skeleton, not a claimed real transcript: 'I was on call when an alert fired for elevated error rates on our checkout service (Situation). As the first responder, my job was to assess scope and either resolve it or escalate within the first 15 minutes (Task). I checked our error dashboard and saw it correlated closely with a deploy that had gone out about ten minutes earlier, so I rolled that deploy back rather than trying to debug it live, and posted a status update to our incident channel explaining what I'd found and what I'd done, then a follow-up once error rates had held steady for a while confirming things looked resolved (Action). The rollback resolved the issue within a few minutes, and afterward I worked with the team to add a canary step to that deploy pipeline so a similar regression would be caught on a small percentage of traffic before a full rollout next time (Result).'

Trade-offs and pitfalls
The most common weak answer stays entirely at the process level ('we followed our runbook and resolved it') without any of the actual technical or judgment specifics an interviewer is trying to assess. A second common weakness is claiming sole credit for what was clearly a team effort, which reads as either dishonest or as a poor understanding of how real incidents get resolved; naming your specific individual contribution within a team response is stronger and more credible than implying you did it all alone. A third weakness is a result with no real follow-up change, which suggests the incident was treated as a one-off rather than a source of a lasting improvement.

Direct answer
My immediate response has three simultaneous threads: contain and mitigate the technical impact, communicate what I actually know (not what I'm guessing) on a steady cadence, and validate that the fix genuinely held before standing down. None of these wait for the others; a responder who only does the technical work and goes quiet for six hours has failed the incident just as much as one who only communicates and never fixes anything.

Structured elaboration

• Contain and mitigate. First isolate whatever is failing so it can't spread (pull a bad instance from rotation, disable a feature flag, freeze further changes), then apply the fastest safe action that reduces customer pain, which is very often a rollback to the last known-good state if a recent change correlates with the onset.
• Communicate on a cadence, not just when there's news. Post a first update as soon as you have a rough sense of scope, even if it's just 'investigating, will update in 15 minutes.' Keep to that cadence even when the honest update is 'still investigating,' because silence reads as worse than an uneventful update, and stakeholders making their own decisions (support teams, other engineering teams, leadership) need to know what you know, not what you hope.
• Validate real recovery before standing down. Don't declare resolved on the first green data point; watch the key metrics hold at baseline for a sustained window, and get explicit confirmation from anyone whose data, transactions, or workflows were affected before formally closing.
• Handle degraded evidence gracefully. Sometimes your own tooling is part of the problem: if the logging or monitoring pipeline itself is down, you're triaging with less information than normal. In that case fall back to direct, lower-level checks (host-level metrics, spot-checking a handful of real requests, canary user reports) and be more conservative about declaring anything resolved, since you're flying with reduced visibility.
• Some failures don't have a clean single-instance fix. A storage array with a degraded RAID rebuilding for 12 hours, or a network partition that's caused two database replicas to diverge (a split-brain), both require deliberately not rushing: forcing a failover mid-rebuild can worsen data loss, and blindly failing over during a split-brain risks writing to the wrong side of a diverged pair. The instinct to 'just fail over' has to be checked against whether failover itself is currently safe.

Worked example
A six-hour outage: a database failover during peak traffic silently sends a subset of write traffic to a replica that hadn't fully caught up, so some writes appear to succeed but are later lost on failback. Immediate response: the on-call engineer confirms the scope (which customers, which write types affected) within the first 20 minutes and posts a first update. Containment: they freeze automated failover so the system can't flip again mid-investigation. Mitigation: they route new writes exclusively to the confirmed-primary replica, restoring correct behavior for new activity even though the earlier lost writes aren't yet recovered. Communication: updates go out every 30 minutes to internal stakeholders and, once the customer-facing impact is confirmed, to affected customers, describing what's known and what's still being investigated, not speculating about root cause before it's confirmed. Recovery: after the fix, the team monitors write consistency across replicas for two hours (not two minutes) before declaring the incident resolved, given how deceptive an initially-healthy-looking replica had already proven to be.

Trade-offs and pitfalls
The most senior-discriminating mistake here is premature stand-down: declaring victory on the first healthy-looking check when the underlying problem (like replica divergence) can silently resurface. A second common mistake is communicating guesses as facts under pressure ('this was caused by X') before root cause is actually confirmed, which erodes trust when the real cause turns out to be different. A third is treating every outage the same way regardless of its actual shape: a clean single-service outage tolerates a fast rollback, while a data-divergence or hardware-degradation scenario often calls for patience and extra verification precisely when the instinct is to act fastest.