Crisis Management and Decision Making Questions

Evaluates how a candidate responds to urgent, high stakes, or time sensitive incidents such as production outages, security incidents, regulatory investigations, compliance failures, customer escalations, or other critical operational problems. Interviewers assess the candidate's ability to rapidly gather and prioritize incomplete or ambiguous information, perform quick diagnosis and root cause analysis, triage and prioritize multiple competing issues, and make pragmatic decisions under time pressure using clear decision criteria. The scope includes short term containment actions, trade offs between temporary workarounds and longer term fixes, risk identification and mitigation, escalation thresholds, and knowing when to pause for more information or to delegate and call for help. Candidates should demonstrate clear and concise stakeholder communication, documentation of rationale, attention to accuracy and quality under deadlines, stress and resilience strategies, and mechanisms to follow up and prevent recurrence by implementing safeguards and lessons learned. At senior levels this also includes leading teams through incidents, setting priorities under pressure, coordinating cross functional stakeholders, maintaining team morale, and measuring outcomes and impact. Strong answers use concrete examples of specific incidents, the decision criteria used, trade offs made when data was limited, how uncertainty and stress were managed, and what was learned and institutionalized afterward.

HardTechnical

71 practiced

You are leading response to a 12-hour outage that affects regulatory reporting deadlines in multiple jurisdictions. Describe how you would escalate the incident, involve legal & compliance, decide about filing extensions or notifications, allocate engineering resources across priorities, and communicate status to executives and regulators while ensuring auditability of decisions and actions.

Sample Answer

Situation: During a major batch-window deployment, our core reporting pipeline failed and remained down for 12 hours during a day with regulatory reporting deadlines across three jurisdictions. Missing those deadlines could incur fines and reputational damage.

Task: As the on-call SRE lead, I needed to rapidly contain and restore service, assess regulatory exposure, coordinate legal/compliance decisions about filings/extensions/notifications, allocate engineering effort effectively, and keep executives and regulators informed — all with an auditable trail.

Action:- Immediate escalation: I declared a Sev-1 and paged the incident response war room (SRE, platform engineers, product owner). I invoked the incident command structure (ICM): Incident Commander (me), Communications Lead, Technical Leads, and Scribe.- Rapid impact assessment: Within 30 minutes we quantified affected reports, jurisdictions, deadlines, and estimated time-to-repair (TTR) options (workaround vs full restore).- Legal & Compliance engagement: I opened a formal incident channel and invited Legal & Compliance into the war room. I provided a concise impact brief (what’s affected, data scope, TTR options, mitigation steps) so they could evaluate regulatory obligations and decide whether to request filing extensions or submit immediate notifications.- Decision framework for filings: Together with Compliance we used criteria: deadline imminence, materiality, likelihood of restoration before cutoff, and precedent. If extension was prudent, Legal prepared the extension request templates and Compliance approved the content.- Resource allocation: I split engineering into two squads: Restore Squad (hotfix/rollback to known-good pipeline) and Workaround Squad (generate manual/extracted reports or replay mechanisms). I assigned senior engineers to restore work and mid-level engineers to build manual export tools, with clear timeboxed goals and checkpoints.- Communications: I ran 30-minute update cadence for stakeholders and hourly exec summaries. Communications Lead drafted messages for executives (impact, mitigations, ask: approve extension/notification) and regulators (if Legal advised immediate notification). All messages were routed through Legal/Compliance before sending.- Auditability: The Scribe maintained a running decision log in our incident tracker (timestamped entries for decisions, approvers, and rationale). All commands, deploys, and rollbacks were done via CI/CD with immutable artifacts and linked to the incident ID. Compliance copies of all regulator communications and evidence (logs, timelines, test results) were archived to our secure evidence store.

Result: We restored the automated pipeline in 7.5 hours via a targeted rollback and delivered manual reports to two jurisdictions before their cutoffs. For the third jurisdiction, Legal filed an extension request within the regulatory timeframe; the regulator acknowledged receipt. Post-incident, we produced a detailed postmortem, preserved the decision log and artifacts for audit, and implemented a permanently automated fallback export to prevent recurrence.

Learning: Pre-established ICM roles, documented decision criteria for regulatory actions, and automatic evidence collection drastically reduce risk in regulatory incidents. I advocated for quarterly tabletop exercises with Legal to shorten future response times.

MediumTechnical

70 practiced

A major customer reports partial data loss after a recent rollout. Outline an investigation plan to determine blast radius and affected users, steps to restore or reconcile data (including safe restore order and verification), how to communicate remediation timelines to the customer, and what internal process changes you would enact to prevent recurrence.

Sample Answer

Investigation plan (scope & triage)- Immediately declare incident, assemble on-call SRE + product + DB owner + customer success.- Gather facts: rollout artifacts (deploy IDs, migrations), timestamps, affected services, logs, metrics, backup snapshots, and recent schema/ETL changes.- Define blast radius: query audit logs and write paths to find time window and affected shards/partitions; run targeted queries to count missing records by tenant/user, region, and data type.- Prioritize customers by impact (major customer first), produce a temporary status page with known scope and next update ETA.

Determine affected users & data- Use immutable audit/event logs, write-sequence IDs, and CDC (if available) to enumerate missing keys.- Cross-check with backups, replicas, and read-repair consistency states to identify divergence.- Produce an export: list of affected user IDs, record types, counts, and last-good-timestamp.

Restore / reconcile steps (safe order & verification)1. Freeze writes for affected scopes (feature flag or routing rule) to prevent further divergence.2. Snapshot current state for forensics.3. Restore from most recent consistent source in this order: - Regional live replica with correct data (fastest, minimal lag) - Point-in-time backups (PITR) to a staging cluster - Reconstruct via CDC/event-sourcing if available4. Reconcile in staging: run automated diff scripts that compare restored vs expected using checksums and record-level invariants.5. Gradual roll-forward: apply restores in small batches (by tenant), validate application-level integrity and business tests, enable read-only access for customer validation, then allow writes.6. Final verification: smoke tests, end-to-end user flows, customer acceptance tests, and monitoring for error spikes.

Communication plan- Initial customer communication within 30–60 minutes acknowledging issue, scope (partial data loss), and immediate mitigation (writes frozen), plus expected next update in 2 hours.- Regular updates: every 2 hours while active, hourly during critical operations, and a targeted update after reconciliation of their data.- Provide remediation timeline ranges (ETA for isolated restores vs full reconciliation) and offer a dedicated support channel and post-incident report timeline (24–72 hours).

Prevent recurrence (process & technical changes)- Require pre-deploy automated canary validation for data path changes and mandatory synthetic tests exercising read/write/restore flows.- Enforce schema/migration review checklist and staged rollout gates (canaries → percentage rollout → full).- Enable continuous backups + PITR with automated restore drills quarterly.- Improve observability: add alerts for abnormal write-to-backup divergence, CDC lag, and per-tenant anomaly detection.- Update runbooks for data-loss incidents, add postmortem with blameless RCA, and track remediation tasks to closure.

This approach balances rapid containment, safe recovery, clear customer communication, and systemic fixes to reduce future risk.

HardTechnical

60 practiced

Design an automated incident responder (bot) that can execute pre-approved containment actions such as disabling a feature flag, scaling a deployment, or updating a firewall rule. Discuss authentication/authorization, approval workflows for destructive actions, audit logging, idempotency, testing strategy, and safe rollback mechanisms.

Sample Answer

Start by treating the responder as a controlled automation platform (playbook engine + connector layer) that executes only pre-approved actions. Key design areas:

Architecture & flow- Playbook store (versioned, signed runbooks) → Orchestration engine → Connectors (k8s, feature-flag service, firewall API) → Audit + Approval service.- Trigger: alert or manual. Engine validates playbook signature and required approvals, then executes step-by-step with state checkpointing.

Authentication & authorization- Mutual TLS between engine and connectors; mTLS client certs issued by internal CA rotated regularly.- Connectors use short-lived service tokens (OIDC/JWT) or workload identities (Kubernetes ServiceAccount, AWS STS) scoped to the smallest privilege necessary.- RBAC: role-based approval policies; playbooks declare required roles for each destructive step.- Secrets stored in a Vault (dynamic credentials, lease/renew).

Approval workflows for destructive actions- Two modes: automated (pre-approved low-risk steps) and gated (destructive: feature-flag kill-all, firewall-wide open).- Gated steps require multi-channel approval: at least one approver from SRE on-call + one from owning team, approval recorded via signed JWT from approval service (can be via CLI, UI, or ChatOps with enforced identity).- Time-bound approvals (expire) and can be conditional (e.g., require telemetry showing >= X% failure rate).

Audit logging & observability- Immutable, append-only audit log (WORM) in centralized logging with event signatures: who, when, playbook id/version, step, inputs, outputs, request-id, and a hash chain for tamper-evidence.- Emit structured events to SIEM and incident timeline; include snapshots (pre/post state) where possible.- Alerts on unexpected connector responses or manual intervention.

Idempotency & safe execution- Every run has a globally unique request-id; each action writes intent and result to checkpoint store before/after execution.- Connectors implement idempotent APIs or use compare-and-swap semantics (e.g., PATCH with expected-version).- Example idempotency pattern (simplified):

python

def run_action(request_id, action):
    if done_store.exists(request_id):
        return done_store.get(request_id)
    result = execute(action)
    done_store.put(request_id, result)
    return result

- Use optimistic concurrency: validate current state before applying changes; if state mismatch, abort and surface to operator.

Testing strategy- Unit tests for engine logic; contract tests for connectors (wiremock, localstack).- Integration tests against staging environment that mirrors production (subset of resources, synthetic traffic).- Chaos/chaotic verification: runbooks executed under failure injection (network partitions, API timeouts).- Dry-run mode that simulates steps and records intended API calls; approval-required actions are disabled in dry-run.- Canary runs: apply changes to a small percentage of targets first, validate metrics, then proceed.

Safe rollback mechanisms- Every destructive action includes an automatic compensating step in the runbook (pre-defined rollback).- Maintain checkpointed pre-change snapshots (config, iptables rules, deployment revision) to restore.- Use transactional-like patterns where possible (blue/green or canary deployments instead of in-place destructive changes).- Automated health checks after each step; if SLO regressions exceed thresholds, orchestration triggers rollback automatically and notifies on-call.- Human-in-the-loop kill switch: abort and initiate immediate rollback with high-priority alert.

Trade-offs & operational considerations- Strong approvals increase MTTR; mitigate with fine-grained pre-approvals for well-tested fast actions.- Balance audit granularity vs. log volume; summarize high-frequency non-destructive steps.- Regularly review and re-audit playbooks, rotate credentials, and run tabletop exercises.

This design enforces least privilege, auditability, reproducibility, and safe automatic remediation while preserving human oversight for high-risk operations.

MediumTechnical

54 practiced

How would you instrument a complex distributed job consisting of thousands of dependent tasks so that an on-call engineer can quickly perform root cause analysis during failures? Describe the design of logs, structured context, correlation IDs, tracing spans, critical metrics, and sample dashboard panels that would help narrow issues rapidly.

Sample Answer

Start with the goals: enable fast triage (who/what/where/when), follow causality across thousands of tasks, and surface high-level health so on-call can zoom into problematic tasks.

1) Correlation IDs & propagation- Assign a root job_id for the whole run and a task_id for each task. Propagate both through RPC headers, queues, and workers.- Also include parent_task_id to reconstruct dependency trees.

2) Tracing spans- Create spans per logical unit: scheduler.enqueue, worker.start, task.exec, task.retry, storage.read/write.- Record attributes: job_id, task_id, parent_task_id, attempt, queue_name, worker_id, host.- Use sampling but ensure all error spans are retained.

3) Structured logs- Emit JSON logs with fields: timestamp, level, job_id, task_id, span_id, parent_span_id, event, message, metadata.Example:

json

{"ts":"2025-01-01T12:00:00Z","level":"ERROR","job_id":"J123","task_id":"T45","span_id":"S7","event":"task.failed","attempt":3,"error":"timeout","duration_ms":120000}

- Log lifecycle events: scheduled, dequeued, started, succeeded, failed, retried, cancelled.

4) Critical metrics (per job and aggregated)- Job success rate, tasks_failed_per_min, tasks_latency_p50/95/99, queue_depth, worker_cpu/mem, retry_rate, blocking_dependencies_count.- Tag metrics with job_type, queue, priority.

5) Dashboards & panels (fast-fail triage)- Overview: active jobs, success rate, failing jobs heatmap (by job_type).- Job drilldown: given job_id panel showing dependency graph, per-task status, latency histogram, timeline of events (logs + traces).- Task failure explorer: top error types, tasks by host, retry counts.- Trace waterfall: show spans for a failing job_id to find slow component.- Alerting panel: recent incidents, correlated alerts (queue spike + worker OOM).

6) Playbook tips- Provide a “search by job_id” quick command that opens trace + logs + dashboard.- Preserve full context on errors (stack, input IDs, config version) but scrub secrets.- Postmortem: store sampled traces and representative logs for 30–90 days for analysis.

Why this works: correlation IDs link logs, metrics, and traces; structured logs enable filtering and quick searches; spans show causal timing; metrics and dashboards surface systemic vs localized problems so on-call narrows root cause in minutes.

EasyTechnical

56 practiced

You're an SRE and a major customer emails that the web UI is extremely slow. Your monitoring shows no active alerts. You have 15 minutes before more customers escalate. Describe a step-by-step triage plan: which dashboards, metrics, and logs you check first, how you form and validate quick hypotheses, what short-term mitigations you might apply, and how you would communicate status to stakeholders while you investigate.

Sample Answer

Situation: Customer reports web UI extremely slow; monitoring shows no alerts. With 15 minutes before escalation, I run a focused, time-boxed triage.

Immediate actions (first 2 minutes)- Acknowledge the customer and state we’re investigating; give an estimated 10–15 min update.- Open critical dashboards: frontend RUM/synthetic checks, gateway/edge latency, API gateway/ingress latency, app service latency, DB/query latency, cache hit rate, error rate, infra CPU/mem, network packet drops, and recent deploys.

Step-by-step triage (minutes 2–10)1. Correlate timelines: align customer report time with metric graphs (spikes in p95/p99 latency across components).2. Check frontend indicators: - RUM: increased client-side load times? Large asset downloads? - CDN/edge: cache miss spike or origin latency increase?3. Check backend/API: - p50/p95/p99 request latency, request rates (RPS), and 5xx error rates on ingress and app pods. - Pod restart counts, OOMs, GC spikes, threadpool exhaustion.4. Check persistence and caches: - DB slow queries, connection pool saturation, replication lag. - Redis/memcached hit rates and eviction rates.5. Infra and network: - Host CPU, memory, disk I/O, network errors, and packet drops on load balancers.6. Logs: - Tail web server and app logs for latency traces, timeouts, upstream errors, DB timeouts. - Search for correlated error messages or stack traces across services.7. Recent changes: - Look at deploy history and config changes in last hour; check if a release, config, or infra change correlates.

Form quick hypotheses and validate (minutes 6–12)- Hypothesis A: CDN/cache regression causing origin overload — validate: CDN logs show cache-miss surge + origin RPS spike; cache hit rate down.- Hypothesis B: Backend DB slow queries — validate: DB p99 latency spike, long-running queries in slow query log, connection saturation.- Hypothesis C: Client-side regression (large asset) — validate: RUM shows increased resource download times, bundle size change in deploy.Use cross-correlation (metrics + logs + traces) to confirm one hypothesis quickly.

Short-term mitigations (minutes 8–15)- If CDN/cache issue: purge problematic cache keys selectively or increase TTL for static assets; add CDN routing rule back to cached edge if misrouted.- If backend overload: scale up replicas or DB read replicas, increase connection pool temporarily, enable circuit breaker/rate-limit non-critical endpoints.- If recent deploy caused regression: rollback or hotfix that deploy; if risky, route a fraction of traffic to previous version (canary rollback).- If DB slow queries: kill runaway queries, revert problematic migration or disable heavy endpoint; enable degraded mode (turn off expensive UI features).- If infra node failure: cordon/replace and autoscale.

Communication cadence- Immediate ack to the reporter (within 2 min): restate impact, owner, ETA for next update (10–15 min).- Short updates every 5–10 minutes: concise status (what we checked, hypothesis, mitigation in progress, ETA).- Notify stakeholders when mitigations deployed and whether customer issue improved; if resolved, confirm with customer and begin root-cause analysis.- Post-incident: document timeline, root cause, mitigations, and action items (alerts to add, runbook updates, permanent fixes) and schedule postmortem.

Why this approach- Time-boxed, evidence-driven steps avoid chasing noise.- Prioritize high-impact, low-effort mitigations first (scaling, rollback, cache fixes).- Clear communication reduces customer frustration while enabling focused technical work.

Unlock Full Question Bank

Get access to hundreds of Crisis Management and Decision Making interview questions and detailed answers.

Join thousands of developers preparing for their dream job.