Incident Response and Management Questions

Direct answer
In the first 15 to 30 minutes: acknowledge the page immediately so others know someone is on it, check your two or three primary signal sources (error-rate dashboard, recent deploys, recent config changes) to get a rough sense of scope and cause, apply an obviously safe temporary mitigation if one exists, post a first status update even if it just says 'investigating,' and decide whether you need more people. What you don't do: silently investigate alone for the whole window, or start deep root-causing before you know how big the problem is.

Structured elaboration

• Acknowledge and orient (minute 0-2). Confirm the page is real, not a flapping alert, and check the primary dashboard to see current error rate, latency, and traffic. This tells you if you're dealing with a full outage or a partial degradation.
• Check for an obvious cause (minute 2-8). Look at what changed recently: deploys, config pushes, feature flag flips, infrastructure changes. Most production incidents correlate with a recent change, and this is the highest-value early check.
• Decide on scope and escalation (minute 5-10). Based on what you've seen, decide if this affects one service or many, one region or all, and whether you need to page anyone else. Getting help early is cheap; struggling alone for 25 minutes before asking is expensive.
• Apply a safe temporary action if available (minute 10-20). If a recent deploy correlates with the problem, rolling it back is usually the safest first move, since it's typically reversible. If there's no obvious cause, avoid guessing with an irreversible action.
• Post a first status update (by minute 15-ish). Even 'we're aware, investigating, will update in 15 minutes' is far better than silence, both for stakeholders and for your own discipline of staying on a cadence.
• What not to do: don't start a deep root-cause investigation before you understand the blast radius; don't make an irreversible change (a manual database edit, a permanent config change) under time pressure without a second person's input; don't go quiet.

Worked example
Page fires at 14:02 for elevated error rate on the checkout service. 14:02-14:03: engineer acknowledges, opens the dashboard, sees error rate at 8% (up from a normal 0.1%), affecting checkout only, not the whole site. 14:03-14:06: checks recent deploys, finds one shipped at 13:55, seven minutes before the alert. 14:06-14:08: decides this is likely deploy-related, pages a second engineer to help verify while they prepare a rollback, and opens an incident ticket marked SEV2 (partial, revenue-affecting feature). 14:08-14:11: rolls back the deploy. 14:11-14:14: watches error rate drop back to 0.1% and holds there. 14:15: posts 'checkout errors have returned to normal following a rollback of a recent deploy; monitoring to confirm and will follow up with a summary.'

Trade-offs and pitfalls
Spending the first 15 minutes purely gathering information without acting risks letting an easily-mitigated problem run longer than necessary; acting immediately without any scoping risks mitigating the wrong thing entirely, or worse, taking an action whose blast radius you don't understand. The 'lone hero' pattern, where a responder tries to fully resolve the incident single-handedly before looping anyone in, is a recurring failure mode: it delays getting a second set of eyes on a decision made under pressure, and it delays stakeholder communication that people are actively waiting on.

Direct answer
A strong answer to this needs to show calm, specific ownership: what you actually noticed, what you actually did in triage and containment (not a vague 'we fixed it'), how you kept people informed while you were still working the problem, and a concrete, real follow-up change that came out of it, not just a general lesson learned.

Structured elaboration
Walking through the STAR structure and what an interviewer is actually listening for at each part:

• Situation. Set up enough context to matter: what service, what kind of impact, roughly how severe. Interviewers are listening for whether you can quickly orient someone else, since that's exactly the skill needed during a real incident.
• Task. Your specific role and responsibility in that moment, not the whole team's. Were you the first responder, the person who happened to notice, someone brought in partway through? This matters because it sets the frame for whether the actions you describe next were really yours.
• Action. This is the section that carries the most weight, and it needs actual specifics: what you checked first, what hypothesis you formed and how you tested it, what mitigation you applied and why you chose it over alternatives, and specifically how you communicated (to whom, how often, what you said versus what you didn't yet know). Vague answers ('I investigated the issue and resolved it') are the single most common weakness here.
• Result. The concrete outcome: how long it took, what the actual resolution was, and critically, what changed afterward as a genuine consequence, whether that's a new test, a new alert, a runbook that didn't exist before, or a process change. A result without a real follow-up change suggests the lesson wasn't actually internalized.

Worked example
An illustrative skeleton, not a claimed real transcript: 'I was on call when an alert fired for elevated error rates on our checkout service (Situation). As the first responder, my job was to assess scope and either resolve it or escalate within the first 15 minutes (Task). I checked our error dashboard and saw it correlated closely with a deploy that had gone out about ten minutes earlier, so I rolled that deploy back rather than trying to debug it live, and posted a status update to our incident channel explaining what I'd found and what I'd done, then a follow-up once error rates had held steady for a while confirming things looked resolved (Action). The rollback resolved the issue within a few minutes, and afterward I worked with the team to add a canary step to that deploy pipeline so a similar regression would be caught on a small percentage of traffic before a full rollout next time (Result).'

Trade-offs and pitfalls
The most common weak answer stays entirely at the process level ('we followed our runbook and resolved it') without any of the actual technical or judgment specifics an interviewer is trying to assess. A second common weakness is claiming sole credit for what was clearly a team effort, which reads as either dishonest or as a poor understanding of how real incidents get resolved; naming your specific individual contribution within a team response is stronger and more credible than implying you did it all alone. A third weakness is a result with no real follow-up change, which suggests the incident was treated as a one-off rather than a source of a lasting improvement.

Direct answer
For a CPU spike causing timeouts: containment is isolating the affected host or throttling the traffic causing the spike so the problem stops spreading to healthy instances; mitigation is shedding non-critical load or scaling out so timeouts stop for most users even before you know the root cause; recovery is restoring the service to its normal capacity and confirming latency has genuinely returned to baseline, not just dropped once.

Structured elaboration

• Containment answers 'how do I stop this from getting worse.' For the CPU spike: pull the overloaded instance out of the load balancer pool so it stops receiving new traffic and can't drag healthy instances down with it (for example through a shared connection pool or retry storm). Safety check: confirm the load balancer has enough remaining healthy capacity before pulling the instance, or you've just made the outage worse for everyone else.
• Mitigation answers 'how do I reduce what customers feel right now,' independent of understanding the root cause yet. For the CPU spike: shed non-critical requests (defer background jobs, disable an expensive feature) or scale out horizontally to spread load. Safety check: shedding load has to be reversible and clearly logged, so nobody forgets a feature is disabled after the incident ends.
• Recovery answers 'is the service actually back to normal.' For the CPU spike: bring capacity back to its normal level, watch CPU and latency hold at baseline for a sustained window, and re-enable anything that was shed. Safety check: recovery isn't declared on a single good data point, since a transient dip can look like recovery for a minute and then relapse.

Worked example
Suppose CPU on the affected instance is pinned at 95% and p99 latency has risen from a normal 120ms to 4 seconds, causing client-side timeouts. Containment: the on-call engineer removes the instance from the load balancer pool, redistributing its traffic across the remaining healthy instances. Mitigation: because the remaining instances are now each carrying more load, they scale out from 4 to 6 instances and temporarily disable a CPU-heavy recommendation feature; CPU across the fleet drops to a more sustainable 60% and p99 latency returns to around 300ms, well below the timeout threshold even if not yet fully back to baseline. Recovery: once the root cause (an inefficient query introduced by a recent deploy) is identified and fixed, the team restores the original instance count, re-enables the recommendation feature, and watches p99 latency hold near 120ms for 30 minutes before declaring the incident resolved.

Trade-offs and pitfalls
Containment that's too aggressive (pulling too many instances, or isolating a component that other services still depend on) can turn a partial degradation into a full outage; containment that's too narrow (missing that the real blast radius includes a shared resource like a connection pool or cache) leaves the spread unaddressed. Mitigation can also mask the symptom in a way that delays real diagnosis: shedding load makes CPU look healthy again, but if nobody tracks that the shed feature is still disabled, the team can lose track of what 'fully recovered' actually means. The general pattern (same containment options: read-only mode, feature toggles, traffic shaping, or temporary scaling) applies just as well to a database write-outage as it does to a CPU spike; the phase you're in, not the specific technology, determines what action is appropriate.

Direct answer
Of the three, I'd respond to the payment failures first, then the public API errors, then the batch job. The payment failures affect a small percentage but include high-value customers and touch money directly, which makes the per-incident cost disproportionately high even at low volume. The API errors affect a much larger share of users (a fifth), which makes them close behind despite lower per-user stakes. The batch job failing on non-critical reports has the lowest urgency because nothing customer-facing depends on it in the next few minutes.

Structured elaboration
Prioritization among simultaneous alerts should weigh three factors together, not any one alone:

• Impact per affected user. A failed payment is a much sharper failure than a slow page load; money and trust are on the line in a way that compounds if it isn't caught quickly.
• Scope. How many users or how much revenue is actually affected right now, not how loud the alert is. A noisy alert with low real impact should never outrank a quiet one with high impact.
• Business criticality and reversibility. Some failures get worse the longer they run (money moving incorrectly, data corrupting further); others are simply delayed (a report that's a few hours late is annoying but recoverable).
  A good way to communicate this under pressure: state the ranking and the one-line reason out loud or in the incident channel immediately, then explicitly assign or acknowledge who is covering the lower-priority items so they aren't silently dropped, even while you focus on the top one.

Worked example
For the three alerts given: payment failures affecting 1% of transactions, including high-value customers, go first, because even a small volume of incorrect financial outcomes needs immediate first-action (typically confirming whether payments are failing safely, i.e., not charging without fulfilling, versus failing unsafely). Second, the public API 5xx errors affecting 20% of users across regions, because the sheer scope makes it the next highest-impact item even though each individual failure is less severe than a payment error. Third, the internal nightly batch job, which is deprioritized to 'someone will pick this up after the first two are stable' since it affects no external users right now. First concrete action on the payment issue: check whether failures are being safely rejected (no charge, clear error to the user) or unsafely processed (charge succeeds but order fails), since that distinction determines whether this needs an urgent stop-the-bleeding action or can tolerate a few more minutes of investigation.

Trade-offs and pitfalls
A common pitfall is anchoring on whichever alert is loudest or has the highest raw volume, rather than the one with the highest actual business impact; a batch job failing nightly can generate more page noise over time than a rare but severe payment issue. Another pitfall specific to this pattern: sometimes a single root cause produces what looks like several simultaneous alerts (a bad deploy triggering a flood of unrelated-looking alerts across a fleet), and prioritizing them as independent problems wastes time that a single fix would resolve; it's worth a quick check for a shared root cause before treating three alerts as three separate incidents. At a higher level, a manager juggling multiple simultaneous incidents across teams faces the same decision with an added dimension: whether the aggregate is severe enough to formally declare a larger, cross-team incident rather than three parallel smaller ones.

Direct answer
An incident is unplanned degradation you did not choose the timing of, and it demands an immediate, ad hoc response. A change is planned work you scheduled, reviewed, and can roll back on your own terms. The core difference is control: with a change you set the clock and the safety net in advance; with an incident, both are decided under pressure, in real time.

Structured elaboration

• Response process. A change follows a pre-agreed plan (a rollout schedule, a rollback procedure written before you started). An incident has no such plan available; the responder is improvising against a runbook at best, from first principles at worst.
• Communication expectations. A change is usually announced in advance ('deploying at 2pm, expect brief latency') and confirmed complete afterward. An incident communication starts reactively ('we're aware of X, investigating') and needs a cadence of updates because nobody agreed to this timing.
• Approvals. A change typically needs sign-off before it happens (a peer review, a change-advisory step for higher-risk changes). An incident response needs no advance approval to act, since delay itself has a cost, though bigger interventions (a full rollback, a customer-facing statement) may still need someone with authority to say go.
• Post-activity documentation. A completed change gets a short record that it happened and worked as intended. An incident gets a fuller review, because the whole point is extracting a lasting lesson from something nobody planned for.

Worked example
A team schedules a canary rollout of a new caching layer for 2pm, reviewed and approved the day before, with an automatic rollback if error rate crosses a threshold. That's a change: planned, approved, monitored against a pre-set safety trigger. At 2:15pm the canary's error rate stays low but an unrelated dependency the caching layer talks to starts timing out, and the on-call engineer gets paged for a 5xx spike unrelated to the rollout. That's now an incident: nobody scheduled it, there's no pre-agreed plan for this specific failure, and the engineer has to improvise triage in real time, even though it happened to start during a change window.

Trade-offs and pitfalls
A common failure mode is applying incident-level ceremony to every change, which breeds change fatigue and makes people route around the process. The opposite failure is worse: not declaring an incident when a change goes wrong, often because the team feels responsible ('we did this to ourselves, let's just quietly fix it') and skips the visibility, communication, and review that an incident would otherwise get. A bad outcome caused by your own planned work is still an incident and deserves the same rigor as one caused by anything else.