Incident Response and Runbook Design Questions

Covers the design and operation of incident response programs and the creation and maintenance of actionable runbooks and playbooks for production systems. Candidates should be able to explain the incident lifecycle from detection and classification through investigation, escalation, remediation, and post incident analysis. Topics include severity definitions and assessment, escalation procedures, team roles and responsibilities, communication protocols during incidents, on call rotations, alert triage, and coordination across teams during outages. Also includes designing automated remediation steps where appropriate, integrating runbooks with monitoring and alerting systems, maintaining playbooks for common failure modes such as malware, data exfiltration, denial of service, and account compromise, and conducting blameless post incident reviews and continuous improvement. Candidates should be able to discuss metrics for measuring response effectiveness such as mean time to detect, mean time to repair, and response success rate, and describe approaches to improve those metrics over time.

MediumTechnical

75 practiced

A critical Kafka consumer group begins showing increasing lag across partitions, delaying downstream jobs. Describe a step-by-step triage: how to determine whether the producer, broker, or consumer is the bottleneck; quick mitigations to reduce lag; and longer-term fixes to avoid recurrence (repartitioning, consumer scaling, retention changes).

Sample Answer

Triage approach (step-by-step)1. Gather immediate facts - Which topic(s) and consumer group; overall lag and per-partition lag (kafka-consumer-groups.sh --describe --group X). - Time window when lag started; any deploys/traffic spikes.2. Isolate layer: producer vs broker vs consumer - Producer: check producer throughput (records/sec, bytes/sec), retry/error rates, and timestamps on produced messages. If incoming rate jumped or producers show retries/acks >0, suspect producer-side surge. - Broker: use JMX/Cloud metrics for leader balance, ISR size, request latency, network IO, disk utilization, controller state, and UnderReplicatedPartitions. Also run kafka-topics.sh --describe to check partition leaders and replication. High broker request queue or disk full indicates broker bottleneck. - Consumer: check consumer client metrics (poll/commit rate, processing time per record, max.poll.interval.ms hits), GC pauses, thread pool saturation, and whether lag is concentrated on partitions assigned to particular consumers. If consumer processing time >> produce inter-arrival, consumer is the bottleneck.

Quick mitigations to reduce lag (minutes)- Scale consumers horizontally: add consumer instances (if consumer group has spare partitions) or increase parallelism in consumers (worker threads or async processing).- Temporarily throttle producers or reroute noncritical producers to a buffer to reduce input rate.- Increase max.poll.records and/or max.poll.interval.ms if safe, to process larger batches per poll.- Pause lower-priority downstream jobs to free resources.- Add brokers or increase broker resources (CPU/disk/network) if broker is saturated.- If retention causes compaction pressure or high I/O, temporarily increase retention to avoid segment deletion spikes.

Longer-term fixes to avoid recurrence- Repartitioning: increase partition count for high-throughput topics so consumers can scale (remember partition count is immutable—create a new topic + mirror/produce there or use ReassignPartitions tool). Balance partition leadership across brokers.- Consumer scaling & architecture: make consumers horizontally scalable and stateless; use worker pools, idempotent processing, batch processing, and async I/O to improve throughput. Ensure commit semantics (enable/avoid auto-commit depending on processing).- Producer controls: implement backpressure, rate limiting, batching, and proper acks/retries to avoid overload spikes.- Broker capacity planning: add brokers, improve disk layout (faster disks/SSDs), tune configs (num.network.threads, queued.max.requests), and ensure monitoring/alerts for ISR/under-replicated partitions.- Retention and compaction: tune retention.ms and segment.bytes to match downstream processing; consider compaction for changelog-style topics to reduce storage churn.- Observability & runbooks: add end-to-end monitoring (per-partition lag, consumer processing time, producer throughput), alerts for sustained lag and GC pauses, and a documented runbook with steps above.- Load testing & SLAs: simulate producer load and validate consumer scaling; set capacity targets and autoscaling policies.

Key checks/commands- kafka-consumer-groups.sh --describe --group G- kafka-topics.sh --describe --topic T- JMX metrics: kafka.server:type=BrokerTopicMetrics*, kafka.server:type=ReplicaManager*- Consumer client metrics (Micrometer/JMX): poll/records/latency, commit/latency- Broker logs for disk/full/leader-election errors

This sequence quickly identifies which layer is saturated, applies short-term relief, and implements structural fixes (repartitioning, scaling, retention tuning, monitoring) so the issue doesn't recur.

MediumTechnical

77 practiced

Propose an operational process for maintaining runbooks in a medium-sized data engineering org: define runbook ownership, review cadence, testing expectations, deprecation policy, linking to CI/CD pipelines, and incentives for engineers to update runbooks after incidents.

Sample Answer

Situation: In a medium-sized data engineering org we needed a sustainable way to keep operational runbooks accurate so on-call responders and engineers could resolve incidents quickly and safely.

Process (ownership & review cadence):- Ownership: each pipeline/service has a primary runbook owner (service owner) and a secondary (backup) recorded in the team roster; platform-runbooks owned by platform team.- Review cadence: owners must review and sign off quarterly, and every time a change touches infra/data contracts; ownership rotations are reviewed at each sprint planning.

Testing expectations:- Treat runbooks as code: store runbooks in the same repo as service code where possible (or a central docs repo) and require PRs to update runbooks alongside code changes.- Automated checks: CI runs markdown lint, front-matter validation (owner, last-reviewed, severity), link-checks, and a checklist that ensures runbook steps are executable (commands, playbook variables).- Tabletop drills: quarterly tabletop exercises for high-severity runbooks and annual live drills for critical pipelines; drill outcomes produce actionable PRs.

Deprecation policy:- Version and annotate runbooks with lifecycle state: active, deprecated (read-only), archived.- Deprecation flow: owner files a deprecation PR explaining replacement and migration plan; mark deprecated for 90 days with redirects; after successful migration and no incidents, archive after 180 days.

Linking to CI/CD:- CI requires “runbook-updated” when PRs change behavior affecting reliability. PR template includes a runbook update section and auto-links to runbook files.- CD deployment notes automatically include runbook version and diff; pipelines fail gating for changes that increase blast radius unless a runbook exists or a documented mitigation is present.- Automation: build a bot that comments on PRs missing runbook updates and auto-opens runbook PR templates.

Incentives to update runbooks after incidents:- Post-incident policy: every incident ticket must include a runbook action item or a justification; unresolved runbook items block incident closure.- Recognition & metrics: runbook updates count toward on-call credit and are visible on team dashboards; quarterly “Operational Excellence” awards and a small budget for team celebrations tied to measurable reductions in mean time to recovery (MTTR).- Performance alignment: include runbook quality and responsiveness metrics in engineering goals (OKRs), and tie part of on-call compensation/rotation priority to demonstrated maintenance.

Example enforcement pattern:- After an incident, a blameless postmortem assigns a runbook PR. The PR is linked to the incident ticket; CI enforces lint/tests; once merged, the incident can be closed. Repeated missed updates trigger a follow-up in the retro with manager support.

This approach balances clear ownership, automated guardrails, practiced validation, predictable lifecycle, CI/CD integration, and behavioral incentives so runbooks stay current and useful.

MediumTechnical

56 practiced

Draft an incident communication protocol template for a data-platform outage that affects downstream analytics. Include initial notification content, cadence of status updates, stakeholders to notify (engineers, analytics, product, customers), recommended communication mediums, and when to open a bridge call or involve leadership.

Sample Answer

Incident Communication Protocol — Data-Platform Outage Affecting Downstream Analytics

Initial notification (send within 15 minutes of detection)- Subject: [INCIDENT] Data Platform Outage — Impacting Analytics | {Short ID}- Body: - What: Brief one-line summary (e.g., "Streaming ingestion to raw-events topic failed; downstream nightly ETL failing") - Scope: Affected systems, datasets, and teams (e.g., analytics reports, dashboards, ML training) - Impact: Business/user impact (delayed reports, stale dashboards, blocked experiments) - Time detected: ISO timestamp and reporter - Initial severity: P1/P2 and rationale - Next update ETA: e.g., "Status update in 30 minutes" - Bridge call details if opened (see rules below)

Cadence of status updates- P1 (severe, customer/business impact): Every 15 minutes until mitigation; then every 30–60 minutes during recovery; final post-incident report within 48 hours.- P2 (moderate): Every 60 minutes until resolved; final post-incident report within 72 hours.- P3 (minor): End-of-day summary if not resolved same day.- Each update MUST include: current status, progress/actions taken, blockers, next steps, ETA to next update.

Stakeholders & notification order1. On-call Engineers (data infra, ETL, SRE) — immediate2. Analytics/BI leads and owners — immediate3. Product owners & PMs for affected products — within initial notification4. Customer-facing teams (Support/CS/Account Managers) — within 30 minutes if customers are affected5. Leadership (Head of Data / CTO) — escalate per "when to involve leadership"6. External customers — coordinated by CS/PR with factual advisory if SLA/customer impact

Recommended communication mediums- Internal: Primary — Incident channel in Slack/MS Teams (create #inc-{id}); Email for broad stakeholder lists only for P1/P2; PagerDuty for on-call alerts.- Synchronous: Bridge call (Zoom/Google Meet) with pinned chat; use shared doc (confluence/GDrive) for live timeline.- External: Status page + templated email via CS or status system; avoid speculative technical detail.

When to open a bridge call- Open bridge immediately for P1 incidents or P2 that meet any: >1 hour unresolved, cross-team blockers, or customer-impacting SLAs breached.- Owner: on-call SRE/Data lead opens and posts link in incident channel.

When to involve leadership- Immediately notify leadership for P1 incidents, or if any: imminent or actual regulatory/SLA/customer-impact, multi-region outage, or need for additional org resources.- Provide brief: impact estimate, mitigation plan, decisions needed, and recommended ask (e.g., customer communication approval).

Post-incident- Run blameless postmortem within 3–7 days: timeline, root cause, action items with owners and deadlines, and communication review.- Update runbooks and RCA shared to stakeholders and status page.

Templates and checklists should be stored with runbooks; practice this protocol in tabletop drills quarterly.

MediumTechnical

70 practiced

List the logging and telemetry artifacts you must capture to enable a forensic investigation for suspected data exfiltration or pipeline tampering. Include database access logs, object store access logs, job execution traces, schema-change events, network flow logs, and retention/access-control policies for these logs.

Sample Answer

Situation: You need a forensic-grade telemetry baseline to investigate suspected data exfiltration or pipeline tampering. Capture the following artifacts, with why each matters and basic retention/access-control guidance.

Essential artifacts to collect- Database access logs - What: timestamps, user/service account, client IP, connection ID, SQL text (full/obfuscated), rows returned/affected, query plan ID, auth method, success/failure, session metadata. - Why: identify suspicious queries, lateral movement, privileged queries.

- Object-store access logs (S3/GCS/Azure Blob) - What: bucket/key, operation (GET/PUT/DELETE/ACL), requester ID, requester IP, timestamp, user-agent, referrer, response code, object version/etag, bytes transferred. - Why: detect large downloads, unauthorized reads, object deletions/rewrites.

- Job execution traces (orchestration + compute) - What: orchestrator (Airflow, Prefect) DAG/task start & end, task owner, command/entrypoint, container image, environment variables masked, logs, exit codes, resource usage, lineage IDs, downstream inputs/outputs. - Why: reveal malicious job scheduling, injected tasks, or tampered transformation steps.

- Schema-change & DDL events - What: DDL statements, who issued them, timestamps, prior schema state, migration IDs, table/column drops or permission changes. - Why: detect tampering that hides exfiltration paths or corrupts data.

- Data lineage & provenance - What: source datasets, transformations applied, checksums/hashes, version IDs, timestamps. - Why: trace origin and mutation of suspect records; verify integrity.

- Network flow & perimeter logs - What: VPC flow logs, firewall logs, proxy/forwarder logs, NAT/gateway records, DNS queries, VPN auth events. - Why: identify exfil destinations, data transfer volumes, anomalous connections.

- Host & container telemetry - What: process trees, user sessions, file access events (open/read/write), privileged escalation events, kernel audit logs, container start/stop, image hashes. - Why: catch insider tools, exfil tooling, or file staging.

- Authentication & authorization audit trails - What: IAM role assumption, token issuance/refresh, MFA events, permission grants/revocations, policy changes. - Why: detect credential misuse or privilege escalation.

- Application-level logs & metrics - What: API request logs, rate anomalies, pagination patterns, error rates, request sizes. - Why: spot scripted downloads or abnormal usage patterns.

- Integrity artifacts and hashes - What: periodic checksums of critical tables/objects, Bloom filters or Merkle trees, tamper-evident timestamps. - Why: prove data alteration or absence.

Retention and storage policies- Retention durations: align with compliance — minimum 1 year for operational forensics, 3–7 years where regulated. Keep high-fidelity logs (DB slow/statement logs, object access, DDL) longer than verbose debug logs.- Immutable and versioned storage: ingest logs into WORM-capable, versioned object storage (S3 Object Lock, GCS Object Hold) and enable object versioning for object stores.- Centralized, indexed store: ship logs to SIEM/ELK/Managed logging (Splunk, Datadog) for correlation and quick search; keep raw archives offline/cheap (cold) but accessible.- Encryption at rest & in transit; sign logs where possible to detect tampering.

Access control and operational controls- Principle of least privilege for log access; separate duties (forensics vs. ops). Require MFA and ephemeral credentials for forensic access.- Audit who accesses logs and queries them; log the log-access.- Retain an offline, air-gapped copy of critical logs for high-risk incidents.- Ensure log collection is resilient: instrument agents in init phase, validate pipeline alerts if logging stops.- Alerting & baseline analytics: set volume/delta alerts (large egress, spikes in GETs), and integrate UEBA to flag anomalies.

Edge cases and practical notes- Mask sensitive fields (PII) in logs but preserve context (hashes).- Balance verbosity vs. cost: sample high-volume telemetry while always capturing full metadata for access and DDL events.- Test with tabletop exercises to verify logs suffice to reconstruct attacks.

This set enables reconstructing who did what, when, from where, and how data moved across systems—critical to determine scope of exfiltration and remediate.

HardTechnical

110 practiced

Provide a runbook-as-code YAML template for automating a Kafka consumer-group rebalance and restart. Include fields: metadata (name, owner, severity), preconditions, steps (shell commands), expected_output, timeout, rollback_steps, and a dry_run boolean. Then show an example filled for consumer group 'analytics-consumer' with concrete placeholders for cluster and topic.

Unlock Full Question Bank

Get access to hundreds of Incident Response and Runbook Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.