Incident Command and Crisis Leadership Questions

Incident Control Plan — One Page (Outage spanning multiple teams)

Who calls the incident

• Any engineer, on-call, product or support rep who observes service-impacting behavior escalates to PagerDuty/alert channel. If impact meets Severity 1 (customer-facing outage, major functionality down, widespread P99 failures), the first responder declares an incident and pages the Incident Commander (IC).

Roles & responsibilities

• Incident Commander (IC): owns coordination, decisions, and timeline.
• Tech Leads (Service Owners): own diagnostics and mitigation for their domains.
• Communications Lead: drafts external/internal updates.
• Scribe: records timeline, actions, rollbacks, and decisions.
• Liaison: product/ops contact for business stakeholders.
• Safety/QA: verifies fixes on staging/production as needed.

Communication cadence

• T0 (declare): IC opens Incident Channel, posts initial summary (impact, scope, contacts, meeting link), starts 5–10 min check-ins.
• T+15, T+30, then every 30 min: concise status updates (what we know, what we’re doing, blockers, ETA).
• If progress stalls, cadence increases to every 10–15 min.
• Use a shared doc for live notes and an audio bridge for coordination when debugging.

Public communication

• Communications Lead and Liaison approve public messages.
• Initial public message within 30 min: acknowledge issue, affected services, and that teams are investigating.
• Regular public updates aligned with internal cadence; set expectation for next update.
• Post-resolution: clear summary of impact, mitigation steps, and ETA for a postmortem.

Decision & resolution criteria

• Incident declared resolved when: 1) affected user-facing symptoms return to acceptable SLOs for 30 continuous minutes, 2) root cause mitigations are in place or an approved rollback applied, and 3) monitoring shows stable metrics.
• IC announces “Resolved” in channel, pauses cadence, schedules postmortem within 72 hours, and assigns action items.

Post-incident

• Blameless postmortem with timeline, RCA, mitigations, and owner-assigned follow-ups tracked to closure. Continuous improvements: update runbooks, alerts, and on-call procedures.

Situation: A critical customer reports they lost data after using a feature I own. I’m the primary owner and first responder.

Immediate steps (first hour)

• Acknowledge receipt to the customer within 30 minutes: empathic, factual, and set expectations (“We’ve received your report, I’m the primary owner. I’ll investigate and update every 60 minutes. If urgent, call X.”).
• Triage & contain: put the feature behind a kill-switch/feature-flag or rollback the deployment to stop further writes. If rollback risks more harm, disable incoming write paths or queue writes.
• Preserve evidence: take system snapshots, preserve logs, export relevant DB backups, and capture metrics/alerts. Avoid destructive changes.
• Assemble incident channel (Slack/Zoom) and notify SRE, Prod DB, Security, PM, Support, and Engineering Manager with a one-line impact summary and immediate actions taken.

Customer & stakeholder communication (first 2 hours)

• Customer: quick status message: what we know, actions taken to stop further loss, ETA for next update (60–120 min). Promise transparency.
• Internal: incident summary in channel (impact, scope, mitigation in place, ownership, next steps). Assign roles: investigator, recovery lead, customer comms, RCA lead.

Next 72 hours plan
Hours 0–6: Investigation

• Confirm scope (which accounts, time window, objects affected) by querying immutable logs, audit trails, and backups.
• Reproduce in sandbox if safe to understand the bug path.
• Identify whether data was deleted, overwritten, or never persisted.

Hours 6–24: Containment & initial remediation

• If deletions: attempt point-in-time DB restores to a staging environment and build export of missing data for verification.
• If overwrites: use write-ahead logs / binlogs to reconstruct previous state; coordinate with DB/SRE to replay or rollback safely.
• If unrecoverable from system: determine whether customer-side exports/backups exist.
• Deliver a technical update to customer and stakeholders every 4–8 hours with concrete progress and timelines.
• Provide temporary mitigations (read-only access, export tools, or scripts to reconstruct data) where possible.

Day 2–3: Recovery, validation, and permanent fixes

• Execute validated recovery plan in a controlled window (coordinate maintenance window, get customer sign-off).
• Validate recovered data with customer (sample checks, hashes) and run end-to-end tests.
• Implement code fix/patch and deploy behind feature-flag; add extra automated tests and data-integrity checks.
• Harden: add retention snapshots, stricter validation on destructive operations, RBAC guardrails, rate-limits, and pre-deploy canary checks.

Post-incident (by 72 hours)

• Postmortem plan and timeline: collect timeline, root cause, contributing factors, remediation, and preventive actions. Share a blameless postmortem within 72 hours with the customer and stakeholders.
• Compensation/Customer care: propose remediation (data recreation help, credits, dedicated support) as per policy.
• Metrics & follow-up: add monitoring for regressions (SLOs/alerts), schedule a retrospective with the team, and track action items to closure.

Why this works

• Fast containment minimizes blast radius.
• Evidence preservation enables safe investigation and restores trust.
• Frequent, honest customer communication reduces anxiety and demonstrates ownership.
• Combining immediate fixes, validated recovery, and systemic changes prevents recurrence.

Situation: A major feature we shipped caused regressions in production, resulting in 8% customer churn over 2 weeks and many support tickets. As engineering lead I coordinated the technical and communication response.

Immediate stabilization (0–24 hours)

• Toggle feature flag / disable rollout globally to stop new impact.
• If rollback is safer, perform controlled rollback to last known-good release with canary verification.
• Deploy emergency monitoring dashboards (errors, latency, affected user count) and raise alerting thresholds.
• Assign a 3-person incident squad (frontend, backend, ops) with a clear incident commander and rotation.

Stakeholder & customer communication

• Hour 0: Send internal “incident initiated” to PM, Support, Sales, Legal with scope and ownership.
• Hour 4: Public acknowledgement to affected customers: explain we’re investigating, mitigation applied, expected next update in 24 hrs. Use email + status page + CS playbook.
• Daily updates until resolved; post-resolution: detailed incident report and recovery timeline.

Root-cause investigation (24–72 hours)

• Freeze changes; capture logs, traces, and metrics; reproduce in staging with production data snapshot (sanitized).
• Run RCA using 5 Whys and include test/CI artifacts: deployment pipeline, feature flag logic, schema migrations, performance tests.
• Produce a clear causal diagram and list contributing factors (e.g., missing integration tests, unsafe DB migration, rollout ramp misconfiguration).

Remediation roadmap (short/medium/long)

• Short (0–7 days): Hotfixes/rollback, add rate-limiting, fix immediate bug, expand monitoring, update runbooks.
• Medium (1–4 weeks): Add end-to-end tests and integration tests that reproduce the failure mode, improve canary metrics and automated rollback triggers, required code review checklist for migrations.
• Long (1–3 months): Re-architect risky path if needed, invest in chaos testing, train team on incident response, and incorporate learnings into SDLC.

Restore trust & metrics to show recovery

• Immediate KPIs: MTTD (target <1 hr), MTTR (target <4 hrs for similar incidents), daily active affected users trending to zero.
• Business KPIs: churn reduction (aim to recover to baseline churn within 8 weeks), feature adoption vs pre-incident baseline.
• Customer sentiment: CSAT for affected customers (target >4/5 within 6 weeks), NPS delta improvement and number of renewed contracts.
• Transparency: publish postmortem within 7 days with actionable items and owner+ETA; track completion of remediation items in sprintboard and report weekly to stakeholders.

What I’d personally do

• Lead the incident squad, keep stakeholders informed, ensure a blameless postmortem, and own delivery of the highest-priority remediation items until metrics demonstrate recovery.

Situation: A deploy caused a surge of errors and noisy logs across thousands of microservices right after release.

Triage & diagnosis (first 15–30 minutes)

• I’d become Incident Commander (IC) and open an incident channel + video room. I declare severity and ETA cadence (every 15 min).
• Quickly gather signals: global error rate, latency, success-rate, request volume from metrics (Prometheus/Grafana), distributed traces (Jaeger/Zipkin), and centralized logs (ELK/Datadog).
• Run blast-radius query: which services show spike, which versions were deployed, common upstream dependencies or shared libraries/configs. Use release tags and deployment timelines to correlate.
• Identify a small set of “smoking guns”: the service(s) with high error-per-minute increase that are upstream of many failing services or a shared library/config change used by many.

Prioritization (how I pick rollbacks/fixes)

• Rank by impact: user-facing errors (conversion/payment pages) > high-traffic APIs > internal non-critical jobs.
• Consider blast radius: if a shared lib/config broke many services, revert that first.
• Consider recovery speed and risk: prefer rollbacks for risky deploys that are fast and reversible; prefer feature-flag toggles for single-service risky features.

Actions to minimize customer impact

• Immediate mitigations: throttle traffic (API gateway), enable circuit breakers, toggle feature flags to disable new behavior, apply emergency rate-limits or queueing to protect downstream systems.
• For each candidate service: attempt an automated rollback to the last known-good version if it’s quick and safe. If rollback is risky, apply patch or config fix and test in prod-shadow traffic if possible.

Coordination & roles

• Incident Commander (me initially): overall decisions, prioritize, communicate exec summary.
• Triage Leads (2): one for backend, one for infra/SRE — analyze traces/logs and validate root cause hypotheses.
• Rollback Owners (per service/team): execute rollback/runbooks and verify health.
• Comms Lead: customer-facing status updates, internal Slack/-email updates, coordinate with PM/support.
• QA/Validation: run smoke tests and synthetic transactions to confirm recovery.
• Postmortem lead: start collecting timelines and artifacts.

Communication cadence

• Initial public status within 10 minutes: severity, scope, next update time.
• Short updates every 15 minutes, detailed update when root cause isolated and after recovery.
• Notify customers proactively for prolonged impact and expected remediation time.

Post-incident

• Freeze related deploys, create action items: fix root cause, add better alerts/tracing, improve deployment guardrails (canary % ramp, automated rollback, stricter pre-deploy tests), update runbooks.
• Run a blameless postmortem within 48–72 hours with timeline, decisions, and measurable follow-ups.

Why this approach

• Fast correlation + blast-radius focus isolates high-leverage mitigations first.
• Clear roles and rapid, predictable communication minimize confusion and customer impact.
• Emphasis on automatable safeguards reduces recurrence.

Situation: As the on-call lead I wake to alerts that payments are failing and customers are reporting financial impact; logs show a systemic error affecting payment processing and it's been ongoing ~24 hours with potential regulatory exposure.

Immediate actions (first 0–60 minutes)

• Declare SEV1 and create a single source of truth (incident doc/War Room Slack + Zoom). I own the incident channel.
• Triage: gather triage info (scope: # of failed txns, affected regions/products, error codes, timeline, deploys/config changes). Pull metrics and recent CI/CD/deploy, config, and third-party payment gateway status.
• Assemble a cross-functional core team with clear roles:
  • Engineering: two squads (investigation & mitigation, rollback path)
  • Ops/SRE: logs, metrics, capacity, circuit breakers
  • Product: customer/business impact, prioritize flows
  • Legal/Compliance: regulatory reporting requirements, document preservation
  • Customer Success/Support: customer communication drafts and prioritization
  • Finance/payments ops: reconcile failed payments
• Set cadence: 15-minute standups, continuous logging to incident doc, decision owner assignment.

Containment & mitigation (0–3 hours)

• If safe, enact immediate mitigations: enable degraded mode (read-only or queuing), enable retries with backoff, turn off non-critical features interacting with payments, throttle traffic to reduce load.
• If a recent deploy/config caused it, prepare and test rollback on canary then promote. Engineering leads execute rollback if it reduces risk.
• If third-party failure, switch to fallback gateway or queued processing and preserve transaction integrity.

External communication (within 1–4 hours)

• Draft and publish a brief, factual “we’re aware and investigating” message to customers and status page. Include impact, affected regions/products, and expected next update time.
• With Legal, prepare regulatory notification timeline and gather required evidence (logs, timelines, decisions). If regulation mandates immediate reporting, file accordingly.
• Customer Success provides prioritized outreach to high-risk customers and offers temporary remediation guidance (e.g., manual reconciliation steps).

Restitution & transactional integrity (3–24 hours)

• Stop further failed transactions; capture all failed transaction IDs and affected customers for reconciliation.
• Build a reconciliation plan with Finance: calculate lost/misapplied funds, fees, penalties. Define restitution policy (refunds, credits, fee waivers) in coordination with Product and Legal.
• Begin automated reprocessing where safe; otherwise arrange manual reprocessing with audit trails. Ensure idempotency to avoid double-charges.
• Track every restitution action in incident doc and customer tickets.

Ownership through resolution

• I remain incident commander until handoff: make final calls on rollback, mitigations, and customer messaging. I keep stakeholders updated (CTO/VP Eng, Head of Product, Legal).
• Maintain incident timeline, decisions, and evidence for regulators and audits.

Post-incident (24–72+ hours)

• Convene a blameless postmortem within 72 hours with contributors across teams. Publish a detailed incident report: root cause, timeline, decisions, impact metrics (customers affected, transactions failed, financial/regulatory impact), and gaps (monitoring/automation/process) with severity-tagged action items.
• Create SMART remediation: code fixes, test coverage, automated canaries, circuit breakers, improved observability (high-cardinality tracing for payments), runbooks, and on-call playbooks.
• Assign owners and deadlines; I track to closure through a remediation board and weekly status updates until all actions complete.
• Update customer and regulator with final report and restitution confirmation; provide remediation evidence if required.

Learnings & prevention

• Institutionalize lessons: add synthetic payment transactions, improved alert thresholds tied to business metrics (failed payment rate), scheduled regulatory readiness drills, and a clear RACI for future incidents.

Why this approach

• It balances immediate containment, customer protection, and regulatory compliance while preserving evidence and enabling an accountable, auditable remediation and restitution process. I keep ownership by centralizing decisions, ensuring cross-functional alignment, and tracking remediation to completion.