Incident Response and Runbook Design Questions

Covers the design and operation of incident response programs and the creation and maintenance of actionable runbooks and playbooks for production systems. Candidates should be able to explain the incident lifecycle from detection and classification through investigation, escalation, remediation, and post incident analysis. Topics include severity definitions and assessment, escalation procedures, team roles and responsibilities, communication protocols during incidents, on call rotations, alert triage, and coordination across teams during outages. Also includes designing automated remediation steps where appropriate, integrating runbooks with monitoring and alerting systems, maintaining playbooks for common failure modes such as malware, data exfiltration, denial of service, and account compromise, and conducting blameless post incident reviews and continuous improvement. Candidates should be able to discuss metrics for measuring response effectiveness such as mean time to detect, mean time to repair, and response success rate, and describe approaches to improve those metrics over time.

HardSystem Design

64 practiced

You need to design an orchestration engine for automated remediation that supports idempotent steps, concurrent executions across regions, persistent execution state, and safe rollbacks. Provide component architecture, data model for step state, and how you would ensure at-most-once or exactly-once semantics for remediation steps.

Sample Answer

Requirements clarification:- Orchestrate multi-step remediation workflows across regions concurrently.- Steps must be idempotent; persist execution state; support safe rollbacks.- Provide at-most-once or exactly-once semantics per remediation step.

High-level architecture:- API / Controller: Accepts remediation requests, validates, returns execution IDs.- Orchestrator Service: Core engine that plans step execution, maintains DAG, retries/rollbacks.- Execution Store (persistent): Durable store for workflow & step state (e.g., CockroachDB / PostgreSQL with strong consistency; or DynamoDB with conditional writes).- Task Queue / Dispatcher: Regional workers subscribe (Kafka / SQS + FIFO or regional topics).- Worker Agents (regional): Execute steps against resources, implement idempotent handlers and local locks.- Saga Manager / Compensator Registry: Stores compensating actions for each step.- Distributed Locking / Lease: Use etcd / Redis TTL locks or DB row leases to prevent duplicate concurrent execution.- Observability: Audit log, tracing, metrics.

Data model for step state (ExecutionID, StepID combination unique):- workflow_execution: - execution_id, workflow_id, status (PENDING/IN_PROGRESS/SUCCEEDED/FAILED/ROLLED_BACK), created_at, updated_at- step: - step_id, execution_id, order, action_type, target_region, payload, status (PENDING/LOCKED/IN_PROGRESS/SUCCEEDED/FAILED/COMPENSATED), attempt_count, last_attempt_at, lease_owner, lease_expiry, result, compensating_action- event_log: - id, execution_id, step_id, event_type, timestamp, metadata

Ensuring at-most-once / exactly-once semantics:- Idempotent handlers: Each action must be implemented to be idempotent (use resource-level idempotency keys, safe PUT semantics, compare-and-swap, or declared idempotency tokens).- Leader/Lease + Conditional Writes (At-most-once): - Orchestrator obtains a lease for a step (write lease_owner + expiry). Workers attempt to acquire via conditional DB update; only holder executes. - Before executing, worker checks step.status via conditional update: PENDING -> IN_PROGRESS. If conditional update fails, it aborts (prevents duplicate). - Upon success, worker writes result and sets status to SUCCEEDED atomically.- Exactly-once (stronger, costly): - Use deterministic execution + dedup table keyed by (step_id, idempotency_key). Worker first inserts a dedup record using conditional write; if insert succeeds, proceed; else return existing result. - Combine with transactional DB (single-transaction write of dedup + state) or two-phase commit pattern using coordinator for cross-service effects. - Use idempotency tokens for external side-effects; providers should support idempotent APIs.- Failure & retries: - On worker crash, lease expiry lets orchestrator reassign; worker re-evaluates by reading state and using dedup keys to avoid reapplication.- Rollback / Compensations: - Each successful step registers compensating action (stored in step.compensating_action). On failure of later steps, orchestrator runs compensations in reverse order, using the same at-most-once/lease/dedup mechanism. - Support checkpoints: persist intermediate invariants so rollback works even after crashes.- Concurrency across regions: - Use per-target-region queues and regional workers; global orchestrator coordinates via consistent execution store. - Use causal ordering from DAG; allow parallel steps whose dependencies are satisfied.

Trade-offs:- Exactly-once requires stronger transactional support or external idempotency; increases latency and complexity.- Using leases + idempotent handlers provides pragmatic at-most-once with safe semantics for most remediation tasks.- Strong consistency store simplifies correctness but may affect global latency; consider regional DB with global consensus for cross-region workflows.

Operational practices:- Define clear idempotency contract for plugin developers.- Limit compensating actions to be reversible and idempotent.- Run chaos and failure injection tests for lease expiry, duplicate delivery, and partial failures.

MediumTechnical

63 practiced

Given an incidents table with schema incidents(incident_id, service, detected_at, acknowledged_at, resolved_at), write a SQL query to compute MTTA and MTTR per service for the last 90 days. Mention assumptions you make about null resolved_at rows and daylight savings/timezones.

Sample Answer

Approach: compute average time-to-acknowledge (MTTA) as avg(acknowledged_at - detected_at) and mean time-to-repair (MTTR) as avg(resolved_at - acknowledged_at). For in-progress incidents you can either exclude them or include time-to-now (common for current operational metrics). Below is a PostgreSQL example that (a) considers last 90 days by detected_at, (b) treats timestamps as timestamptz (UTC), and (c) includes unresolved incidents by using now().

Assumptions:- detected_at, acknowledged_at, resolved_at are timestamptz (or stored in UTC). If not, convert to UTC first.- MTTA: exclude rows where acknowledged_at IS NULL if you want only acknowledged incidents; query below shows both variants.- MTTR: we use COALESCE(resolved_at, now()) to include ongoing incidents; change to WHERE resolved_at IS NOT NULL to exclude them.- Daylight savings handled by storing/operating in UTC (timestamptz) and using now() (UTC-aware).

SQL (Postgres):

sql

SELECT
  service,
  COUNT(*) AS incidents_count,
  AVG(EXTRACT(EPOCH FROM (acknowledged_at - detected_at))) / 60.0 AS mtta_minutes,
  AVG(EXTRACT(EPOCH FROM (COALESCE(resolved_at, now()) - acknowledged_at))) / 60.0 AS mttr_minutes_including_open
FROM incidents
WHERE detected_at >= now() - INTERVAL '90 days'
-- optional: only include incidents that were acknowledged for MTTA/MTTR calculations
-- AND acknowledged_at IS NOT NULL
GROUP BY service
ORDER BY service;

Notes:- EXTRACT(EPOCH) gives seconds; dividing by 60 yields minutes. Use AVG(...) directly on interval in Postgres if preferred: AVG(acknowledged_at - detected_at).- To exclude ongoing incidents from MTTR, replace COALESCE(resolved_at, now()) with resolved_at and add WHERE resolved_at IS NOT NULL.- If timestamps are in different timezones, convert: detected_at AT TIME ZONE 'UTC'.- Validate outliers (e.g., negative intervals) and filter bad data (acknowledged_at < detected_at).

MediumTechnical

58 practiced

Write a runbook for suspected account compromise of a privileged service account used by multiple microservices. Steps should include immediate containment, secret rotation, session invalidation, dependency mitigation, communication, and validation tests to verify the compromise has been resolved.

Sample Answer

Runbook: Suspected compromise of a privileged service account used by multiple microservices

1) Triage & scope- Confirm alert source (IDS, logs, IAM alert). Capture timestamps, IPs, user-agent, affected services, last secret change.- Mark incident severity and assign incident owner.

2) Immediate containment (minutes)- Disable account access in IAM or apply a restrictive policy (deny all) — if immediate disable would break critical customer-facing flows, place account into read-only or emergency lockout group.- Isolate affected hosts (quarantine pods/instances) and block suspicious IPs at network/ingress layer.

3) Secret rotation (hours)- Generate new credentials/keys in secrets manager (unique per service if possible).- Use automated rotation pipelines (CI/CD or secret-manager APIs) to update each microservice; prefer staged rollout (canary) to reduce blast radius.- Revoke old credentials immediately after rotation via IAM key deletion/disable.

4) Session/token invalidation- Invalidate all active sessions: rotate JWT signing keys or increment token version stored in user metadata so services reject old tokens.- Revoke refresh tokens and API keys; force re-auth for interactive components.

5) Dependency mitigation- Audit downstream systems the account accesses (databases, message queues, cloud APIs). Rotate any credentials the compromised account could access.- Apply least-privilege: tighten permissions to minimal required during recovery.

6) Communication- Notify SRE, security, product leads, and impacted teams with succinct status: scope, actions taken, next steps, and rollback plan.- If customer data impacted, follow legal/PR policy and notify stakeholders per compliance requirements.

7) Forensic capture & preserve evidence- Collect logs, memory snapshots, process lists, and secret usage history. Snapshot affected instances before changes.- Note timeline of actions for post-incident review.

8) Validation tests (post-recovery)- Verify services successfully authenticate with new secrets in canary then full rollout.- Confirm revoked credentials are rejected: attempt API calls with old keys/tokens and expect 401/403.- Re-run intrusion detection and scan for persistence (scheduled tasks, new users, unusual IAM policies).- Monitor for anomalous activity for 72+ hours with elevated logging and alerting.

9) Post-incident actions- Conduct root-cause analysis, document lessons, update runbook to remove single privileged shared account: adopt per-service identities, short-lived credentials (OIDC, workload identity), MFA for human use, and automated secret rotation.- Implement preventive controls: least-privilege, credential scanning, alerting for high-risk actions, and periodic access reviews.

HardSystem Design

74 practiced

At scale, you need to generate runbooks programmatically from service metadata and templates. Propose a runbook-as-code pipeline that validates, lints, and publishes runbooks to an internal runbook service. Explain how you would handle secrets, test the runbooks, and rollback a bad runbook change.

HardTechnical

76 practiced

Hard coding problem: Implement a function in Python that given a sequence of incident events (detected, acknowledged, action_started, action_completed, resolved) computes MTTA and MTTR per incident and flags incidents where automation was started before acknowledgment. Input is a list of (incident_id, event_type, timestamp). Provide code and explain complexity.

Sample Answer

Approach: parse events per incident, sort by timestamp, then walk each incident's timeline to compute MTTA (time from first detected to acknowledged) and MTTR (time from detected to resolved or action_completed→resolved depending on definition). Also flag if automation (action_started) occurred before acknowledge.

python

from collections import defaultdict
from datetime import datetime
from typing import List, Tuple, Dict, Any

def compute_metrics(events: List[Tuple[str,str,str]]) -> Dict[str, Dict[str, Any]]:
    """
    events: list of (incident_id, event_type, timestamp_iso)
    Returns per-incident dict with mtta_seconds, mttr_seconds, automation_before_ack (bool)
    """
    incidents = defaultdict(list)
    for iid, etype, ts in events:
        incidents[iid].append((datetime.fromisoformat(ts), etype))
    result = {}
    for iid, evs in incidents.items():
        evs.sort()  # sort by timestamp
        detected_ts = ack_ts = resolved_ts = None
        automation_before_ack = False
        # track earliest detected and acknowledged/resolved times
        for ts, etype in evs:
            if etype == "detected" and detected_ts is None:
                detected_ts = ts
            if etype == "acknowledged" and ack_ts is None:
                ack_ts = ts
            if etype == "action_started":
                if ack_ts is None:
                    # automation started before ack
                    automation_before_ack = True
            if etype == "resolved":
                resolved_ts = ts
                break  # resolved final state
            if etype == "action_completed" and resolved_ts is None:
                # could use action_completed as resolution proxy if resolved missing
                resolved_ts = ts
        mtta = (ack_ts - detected_ts).total_seconds() if detected_ts and ack_ts else None
        mttr = ((resolved_ts - detected_ts).total_seconds() if detected_ts and resolved_ts else None)
        result[iid] = {"mtta_seconds": mtta, "mttr_seconds": mttr, "automation_before_ack": automation_before_ack}
    return result

Key points:- Sort per-incident events (stable, O(k log k) per incident).- Time complexity: O(N log N) dominated by sorting all events per incident (N total events). If events pre-bucketed chronological, can be O(N).- Space: O(N) to store events.Edge cases:- Missing detected/acknowledged/resolved events → return None for metric.- Multiple detected/acknowledged: uses first detected and first acknowledged, first resolved (or action_completed if resolved missing).- Timestamps must be ISO; adapt parsing for other formats.

Unlock Full Question Bank

Get access to hundreds of Incident Response and Runbook Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.