Incident Command and Crisis Leadership Questions

Executive 5-line update (clear, non-technical, impact-focused — to send now, then every 30 min):

• Incident: Production API latency >3x affecting checkout — started 08:12 UTC.
• Scope: ~35% of customers impacted; order throughput down 40%.
• Current status: Mitigation in progress — traffic throttled & failover to secondary cluster enabled.
• Next action: Rolling config rollback and capacity scale; ETA 25 minutes.
• Ask: No action required from executives; we'll notify on resolution or if customer-facing escalation needed.

Operational engineering update (detailed — post to war room, every 10–15 min):

• Timeline & symptoms: latency spike at 08:12 UTC, spike correlated with increased p95 CPU on primary app nodes and elevated DB connection churn.
• Hypothesis: recent config push (v2.3.7) increased connection pooling aggressiveness causing DB pool exhaustion.
• Actions taken: throttled ingress via API GW, enabled failover DB replicas, rolling rollback of v2.3.7 on service-a (nodes 1–4), increased DB max_connections by 200, collected heap/proc and tcp stats.
• Evidence: attach logs (links), p95 latency graph, DB wait events, tcp/ephemeral port exhaustion sample.
• Next steps: complete rollback (ETA 10m), monitor DB connections for 2x steady-state window, if > threshold open incident for DB team to perform deeper tuning; post-mortem owner assigned (me).
• Escalation: if rollback fails or error rate >5% persist at 3 consecutive 1-min checks, escalate to SRE manager and DBA on-call immediately.

Distinguishing rationale:

• Execs get impact, ETA, business ask — no technical detail.
• Engineers get root-cause hypothesis, precise actions, telemetry, and clear escalation thresholds to enable fast remediation.

Overview (role in one line)
As incident commander (IC) I own coordination, priority decisions, communication, and lifecycle management for a high-severity production incident while empowering subject-matter owners to execute fixes.

Key responsibilities

• Triage scope and business impact (users affected, SLO/SLAs, customer-facing vs internal).
• Declare severity and escalation path.
• Set short-term priorities: stabilize service, protect data, restore critical functionality.
• Coordinate cross-team actions, external comms, and post-incident review ownership.

Priority decisions

• Stop-the-bleed vs experiment: roll back vs test fixes in prod.
• Resource allocation: who works on mitigation vs root cause.
• When to escalate to leadership or legal/comms.

Delegation model

• IC → Leads: designate a Tech Lead for remediation, a Communications Lead, a Monitoring/Telemetry Lead, and a Logistics Scribe.
• Tech Lead runs hands-on fixes; IC approves risky actions and coordinates dependencies.

Handoff criteria

• IC hands off when IC role is needed elsewhere, or a stable triage-to-repair transition occurs and another IC has bandwidth/authority. Handoff includes concise briefing, incident timeline, decision log, and contact list.

Maintaining situational awareness without bottlenecking

• Use a single source-of-truth (incident channel + runbook) and delegate decision authority for bounded actions.
• Require status cadence (5–15 min check-ins), a live incident board, and a scribe logging actions/RTT.
• Make only high-impact decisions; trust delegated leads for execution and confirm via quick checkpoints.

Decision Framework — Incident Commander (Systems Engineer)

1. Triage & Objectives

• Primary: protect data integrity and regulatory compliance (transactions must be correct, auditable).
• Secondary: maximize business continuity and minimize customer harm.

2. Rapid Data Collection (quantitative)

• Error rate: failed transactions per minute (threshold: >1% sustained).
• Data divergence: % of reconciliation mismatches (threshold: >0.1%).
• Latency P50/P95/P99 (ms).
• Throughput: TPS vs expected.
• Rollback safety score: % of schema migrations in release, idempotent change count, DB irreversible ops.
• Regulatory window: time-to-report requirement (hours).

3. Decision matrix (weighted scores)

• Data integrity (40%): if divergence > threshold → prefer rollback.
• Rollback safety (25%): if unsafe (non-idempotent or cross-service state) → avoid rollback.
• Business continuity (15%): revenue impact $/min or blocked payment volume.
• Time-to-implement (10%): estimated minutes to rollback vs throttle.
• Compliance risk (10%): fines/mandates if transactions lost.

Score each option (0–10) then compute weighted total; pick higher score.

4. Qualitative checks

• Downstream effects (clearing houses, partners).
• Operational readiness: available runbook, on-call roster.
• Stakeholder tolerance for partial degradation.

5. Mitigation actions

• If rollback chosen: freeze writes, run DB consistency checks, run canary rollback on 1 region then full.
• If throttling chosen: apply token-bucket limits, prioritize high-value/regulated flows, enable backpressure, monitor reconciliation.

6. Communication cadence

• Initial: 5-min incident bulletin with decision, ETA, mitigations.
• Operational updates: every 15 min with metrics (error rate, TPS, reconciliation %).
• Major change (rollback/throttle): pre-change confirm, post-change verification window 5–10 min, then status update.
• Postmortem deadline: within 72 hours, regulatory report as required.

7. Exit criteria

• Error rate < threshold for 30 consecutive minutes, reconciliation stable, latency acceptable.

Rationale: prioritize preserving transactional correctness and compliance; use quantitative thresholds to remove bias, qualitative checks to surface systemic risks, and tight communication for alignment.