Incident Response and Troubleshooting Questions

Approach to diagnosing and resolving production incidents, outages, and critical failures under time pressure. Covers systematic triage, identifying root causes, maintaining service availability, coordinating with stakeholders, prioritizing safety and mitigation steps, postmortem practices, and learning from incidents to prevent recurrence. Interviewers expect examples showing technical troubleshooting, communication during crises, decision making under pressure, and follow through in remediation and documentation.

MediumTechnical

68 practiced

You rolled out a canary and observe the canary user cohort's error rate rose from 0.2% to 3.0%. Describe how you decide whether to stop the rollout, rollback, or mitigate. Explain the criteria you use, how to perform a safe rollback, and how to validate that rollback succeeded.

Sample Answer

First, treat this as an operational incident: gather facts, limit blast radius, then choose action based on risk and mitigations.

Decision criteria- Severity vs SLOs: 3.0% vs baseline 0.2% -> absolute +2.8%. Compare to error budget and business impact (user-facing errors, data loss).- Trend & scope: is error rate rising, steady, or spiking? Is it constrained to the canary cohort or leaking to others?- Root-cause confidence: Do logs/trace/span data point to a specific change? Is there a quick mitigation (feature flag, config tweak)?- Rollout size & velocity: small canary (1–5%) vs larger. For small canary, higher tolerance to investigate; for larger, more urgency.

Actions- If errors are causing customer-visible failures or breaching error budget and no safe mitigation exists → stop rollout and rollback.- If root cause is identified and a low-risk mitigation (config, circuit-breaker, rate-limit) can be applied quickly → pause and mitigate.- If impact is moderate, scoped, and you can ramp down the canary to 0% while investigating → pause and rollback incrementally.

How to perform a safe rollback1. Pause automated ramps immediately and notify stakeholders.2. Perform a single-step rollback via the deployment system (rollback to prior image/manifest) or set feature flag off. Prefer declarative rollback (git/manifest) to avoid drift.3. Do rollback incrementally if needed: reduce canary percentage to 0% then redeploy stable version to remaining hosts, or perform a rolling rollback with health checks.4. Ensure orchestration health probes and readiness checks are enabled so unhealthy pods aren’t promoted.5. Preserve telemetry and logs (don’t delete canary instances) for postmortem.

Validating rollback succeeded- Monitor the same error-rate metric and related traces: expect return toward baseline within one SLA window (e.g., 1–5 minutes for service-level, longer for background jobs).- Check canary cohort metrics: error rate drops to near baseline; key user journeys succeed end-to-end.- Verify system health: CPU/memory, latency, and other KPIs are back to normal.- Confirm no new top-level alerts and that the stable version’s deployment/replica counts match desired state.- Sanity-check logs/traces to ensure error signatures tied to the change are gone.

Communicate: post-incident alert to stakeholders, capture timeline, and schedule postmortem to prevent recurrence.

HardTechnical

87 practiced

Design an incident response process for an enterprise handling regulated financial data. Requirements: preserve evidence for audits, complete regulatory reporting within 24 hours of detection, restrict communications appropriately, and automate data preservation where possible. Describe roles, data preservation steps, templates, and validation/testing of the process.

Sample Answer

Requirements clarification:- Preserve chain-of-custody and immutable evidence for audits- Meet regulatory report filing within 24 hours of detection- Limit external/internal communications per policy- Automate preservation where possible

High-level process (phases):1. Detect & Triage (0–1h) - Automated alerts enrich with context (asset owner, data classification, regulatory flags). SRE runbook triggers containment if availability impacted.2. Contain & Preserve (1–4h) - Immediately invoke automated preservation playbook: snapshot VMs/containers, enable write-block on affected storage, export relevant logs (app, OS, network, cloud audit) to WORM storage (immutable S3/GCS bucket) and SIEM with retention tags. - Capture volatile memory where required using forensics tool (automated via orchestrator on affected host). - Record chain-of-custody in an audit ledger (timestamped, signed entries).3. Investigate & Remediate (4–18h) - Forensic team (SRE + Security + Legal + Compliance + Business owner) performs analysis on preserved artifacts. - All actions recorded; no destructive changes to preserved copies.4. Report & Communicate (within 24h) - Compliance completes regulatory report using prefilled template (see below). Communications must follow approved whitelist: internal incident channel, legal-approved statements, and regulator contact.5. Post-Incident (after 24h) - Postmortem, root-cause, remediation plan, lessons learned, and evidence retention verification.

Roles & responsibilities:- Incident Commander (rotating SRE on-call): coordinate technical response, declare containment.- Forensics Lead (Security): direct evidence capture, maintain chain-of-custody.- Compliance Officer: prepare regulatory filings, liaise with regulator.- Legal: approve external communications and escalation decisions.- Service Owners/Dev Teams: supply context, implement fixes.- Automation Engineer (SRE): author/maintain automation playbooks.

Data preservation steps (automated where possible):- Tag incident assets and trigger IaC job to snapshot disks & containers.- Export logs: systemd/journal, application logs, auth logs, cloud provider audit logs, network PCAP for a limited window to immutable storage.- Lock relevant user accounts and API keys (automated rotation workflows).- Store checksums and digital signatures of artifacts in the audit ledger.

Templates (pre-approved):- Regulatory Report Template: incident ID, detection time, systems affected, data classification, containment actions, preservation artifacts (locations + checksums), estimated impact, next steps, contact points.- Internal Communication Template (tiered): internal urgent, business stakeholders, approved external statement.- Chain-of-Custody Form: artifact ID, timestamp, captured-by, capture-method, storage location, integrity hash, signer.

Automation examples (implementation snippets):- IaC/Orchestration job triggers snapshot and uploads logs to WORM bucket; sample pseudocode (cron/CI job) automates S3 put with ObjectLock.- SIEM integration rule auto-tags artifacts and creates incident ticket with prefilled regulatory template.

Validation & testing:- Quarterly tabletop exercises with Compliance, Legal, Security, SRE, and Business to validate timelines and decision paths.- Monthly automated drill: simulate low-severity incident that runs preservation playbook end-to-end (snapshots, log export, ledger entry) in staging; verify immutability and hashes.- Annual full-scale live drill with regulator-facing reporting exercise to confirm 24-hour reporting pipeline.- KPIs: time-to-preserve (target <2h), time-to-report (<=24h), preservation completeness (percentage of required artifact types captured).- Audit checks: periodic verification that WORM storage enforces retention policies and ledger integrity (signed hashes match stored artifacts).

Trade-offs and controls:- Limit PCAP collection window to balance privacy and storage costs; preserve metadata as fallback.- Use role-based access control and approval workflows to prevent unauthorized communication or evidence tampering.

This design ensures technical automation for rapid preservation, clear roles for compliance demands, templates to accelerate reporting, and regular validation so the 24-hour regulatory requirement and auditability are reliably met.

MediumTechnical

64 practiced

A postmortem lists 12 action items across infra, app, and process. Describe a method to prioritize and assign these items to maximize reliability impact within a quarter. Include how you'd score items, assign owners, track progress, and report back to stakeholders.

Sample Answer

Situation: After a high-severity incident our postmortem produced 12 action items across infra, app, and process that must be completed this quarter to close the reliability gaps.

Method (prioritization + assignment):

1. Score each item (quick 4‑factor rubric, 1–5 each; higher = more favorable)- Impact on availability/SLOs (how much reduction in errors/MTTR) — weight 40%- Probability of preventing similar incidents (preventive value) — weight 25%- Implementation effort (engineering hours) — weight inverted, lower effort scores higher — weight 20%- Risk/complexity (risk of deployment/regression) — weight 15%Compute weighted score and rank items. Example: automated failover might score Impact 5, Preventive 5, Effort 3, Risk 4 → weighted score = 5*.4+5*.25+3*.2+4*.15 = 4.25.

2. Triage into 3 buckets for the quarter- Must-do (top ~30% by score or required by compliance): deliver in first 6 weeks- High-value (next ~40%): deliver by week 7–12- Deferred/minor (remaining): backlog/follow-up beyond quarter or smaller quick wins

3. Assign owners- Assign primary owner by domain expertise (infra items → infra SRE; app fixes → service dev lead) plus an SRE reviewer as co-owner to ensure reliability practices.- Consider capacity: check team sprint allocation; break large items into 2‑week deliverables and assign phased owners.- Document clear acceptance criteria: expected metric change (e.g., reduce p99 latency by X, lower MTTR by Y), tests, rollout plan, rollback criteria.

4. Track progress- Create tickets in Jira with priority labels, subtasks, due dates mapped to sprint cadence.- Add a Reliability Board (Kanban) with swimlanes: In Progress / Blocked / QA / Done; link to dashboards.- Weekly SRE sprint check-in to surface blockers; daily standups where needed.- Measure progress against objective metrics (SLOs, error budget burn, MTTR) and task burn-down.

5. Report back to stakeholders- Weekly executive one‑pager: status (Green/Yellow/Red), top 3 risks, mitigation, completed items, and expected impact on SLOs.- Biweekly demo or RFC for significant infra or app changes; invite product and support leads.- End-of-quarter report: mapping of actions → measurable outcomes (e.g., "Completed automated failover; MTTR reduced from 45m to 8m; error budget conserved by 30%").

Why this works:- Quantitative scoring balances impact vs effort so we maximize reliability ROI within the quarter.- Domain owners + SRE reviewers ensure knowledge transfer and reduce rework.- Clear acceptance criteria, tracking, and stakeholder reporting maintain alignment and visibility.

Example outcome target for the quarter: complete 4 must-do and 5 high-value items such that MTTR improves by 50% and p99 errors reduce by 30%, keeping us within SLOs for the next quarter.

MediumTechnical

67 practiced

During a backend slowdown, explain how you would use feature flags and circuit breakers to degrade functionality safely and reduce load while preserving critical user journeys. Give concrete steps you would take, safety checks, and monitoring needed to verify the mitigation.

Sample Answer

Approach (overview): use feature flags to selectively turn off or degrade non‑critical features, and circuit breakers to stop hitting overloaded downstream services. The goal is graceful degradation that preserves critical user journeys (login, checkout, read content).

Steps:1. Triage & prioritize user journeys: identify critical flows and dependent services (auth, payments, content API).2. Create feature flags scoped by percentage, user role, and session: - e.g., disable “recommendations” and “async image processing” for 100% of traffic. - add gradual flags: “search_fallback_v2” routed to cached results for X% traffic.3. Deploy circuit breakers at client and service boundaries (client-side SDK + server middleware): - configure thresholds: error rate > 20% in 1m or latency P95 > 2s → open. - use exponential backoff and half‑open probes to test recovery.4. Rollout plan: - Step 0: enable non‑critical flags in staging. - Step 1: enable for internal users (5%). - Step 2: ramp to 25/50/100% while monitoring. - Simultaneously enable circuit breakers globally for problematic downstreams.

Safety checks:- Circuit breaker short‑circuit returns cached/fallback responses with clear telemetry.- Feature flags default to safe state (on for critical flows).- Predefined kill-switch by runbook and automated playbook.- Rate limiting and quota checks to avoid thundering herd when breakers close.

Monitoring & verification:- Dashboards: SLOs for latency, error rate, request throughput for critical flows; flag adoption metrics; breaker state metrics (open/closed/half‑open).- Alerts: when critical SLOs breach, when breaker opens, or when flag rollback fails.- Golden metrics: signup/login/checkout success rate, end‑to‑end latency, cache hit rate.- Post‑action validation: synthetic checks hitting critical journeys, real‑user monitoring (RUM) samples, and logs to confirm fallbacks are used and no data loss occurred.

Outcome and rollback:- If mitigation improves SLOs, progressively re-enable features and close breakers; if not, execute rollback runbook and escalate.This approach minimizes blast radius, preserves user trust on core journeys, and provides observability to safely restore full functionality.

MediumTechnical

52 practiced

A third-party payment provider is experiencing a global outage causing reduced throughput. As the SRE handling the incident, how do you coordinate with product, legal, and engineering to decide on mitigation steps (e.g., retry/backoff, degrade experience), customer messaging, and whether to roll back a recent change that might be unrelated?

Sample Answer

Situation: During a peak period our payments pipeline showed rising error rates and latency. External vendor status confirmed a global throughput degradation. This impacted checkout success for many customers.

Task: As SRE on-call I needed to stabilize user experience, coordinate cross-functional decisions (product, legal, engineering), own technical mitigations, and decide whether a recent deployment should be rolled back.

Action:- Rapid assessment (first 10–15 min): validated alerts, looked for error patterns, spike timing, and whether the recent deploy correlated with onset. Confirmed vendor outage via provider status + telemetry.- Triage meeting (15–30 min): I convened a short war-room with product, engineering, and legal. I owned the incident timeline and metrics; asked product for customer impact priority; engineering for rollback cost/risk; legal for regulatory/customer-notice constraints.- Proposed mitigations with engineering: - Implement client-side and server-side exponential backoff + jitter for retries and a circuit-breaker to avoid cascading load. - Temporarily degrade non-critical payment flows (one-click upsells, analytics) to preserve core checkout throughput. - Add throttling and queueing for background payment reconciliation.- Rollback decision framework presented: - If telemetry showed high correlation between deploy and errors (>90% tied to new code path) -> rollback immediately. - If no strong correlation and vendor is root cause -> avoid risky rollback; prefer runtime mitigations.- Customer messaging coordinated with product & legal: - Drafted an emergency status page update and templated customer emails describing impact, ETA, and mitigations — legal reviewed wording for compliance. - Product approved tone/prioritization; support and sales given a FAQ.- Execution: engineering implemented circuit-breaker + degraded features within 45 minutes; I validated metrics and adjusted thresholds. Messaging published and support briefed.

Result: Core checkout success rate recovered to acceptable SLOs within an hour despite vendor slowness. No rollback was needed because telemetry showed the deploy was not causal. Post-incident we ran a blameless postmortem, updated runbooks to include vendor outage playbooks, and added synthetic tests to detect vendor throughput regressions earlier.

What I demonstrated: decisive, data-driven coordination; clear roles (SRE drives metrics/mitigations, product sets customer priority, legal signs off on messaging, engineering implements changes); low-risk rollback policy tied to signal strength; and fast, transparent customer communication.

Unlock Full Question Bank

Get access to hundreds of Incident Response and Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.