Incident Management and Response Questions

Covers operational handling of production outages and service incidents across the full lifecycle from preparation through detection, triage, containment, mitigation, recovery, and post incident review. Interviewers assess monitoring and observability signals, alerting thresholds and on call rotation, severity classification and escalation paths, incident command and coordination, runbooks and playbooks, immediate containment and mitigation techniques to minimize customer impact, restoration and recovery procedures, and evidence capture when relevant. Candidates should be able to describe root cause analysis practices, blameless post incident reviews, tracking remediation and follow up actions, driving cross functional ownership of fixes, and how incident learnings feed into long term reliability improvements and tooling or automation. Senior level expectations include organizing incident response teams for production reliability, defining severity levels and escalation policies, balancing rapid decisions with risk management, and continuously improving processes, runbooks, and instrumentation.

MediumTechnical

69 practiced

Write a Python 3 script that reads a newline-delimited log file where each line is a JSON object with keys: "timestamp" (ISO 8601), "service", "level", "message". The script should output per-minute error counts (level == "ERROR") for a given service over the last 60 minutes, printing lines like: 2025-03-12 14:05 3. The log may be out-of-order and can be large (~10GB): prioritize streaming and bounded memory.

Sample Answer

Approach:- Stream the file line-by-line, parse JSON, filter by service and level.- Normalize timestamps to UTC minute buckets (truncate seconds).- Maintain a dict of counts keyed by minute; keep only the last 60 minutes relative to the maximum timestamp seen so far to bound memory.- After processing, print per-minute counts for the 60-minute window in chronological order, including zero counts.

python

#!/usr/bin/env python3
import sys
import json
from collections import defaultdict
from datetime import datetime, timedelta, timezone

WINDOW_MINUTES = 60

def parse_iso8601(s):
    # Accept e.g. "2025-03-12T14:05:23Z" or with offset
    if s.endswith("Z"):
        s = s[:-1] + "+00:00"
    return datetime.fromisoformat(s)

def minute_bucket(dt):
    # Ensure timezone-aware (convert naive to UTC)
    if dt.tzinfo is None:
        dt = dt.replace(tzinfo=timezone.utc)
    dt = dt.astimezone(timezone.utc)
    return dt.replace(second=0, microsecond=0)

def purge_old(counts, newest_minute):
    cutoff = newest_minute - timedelta(minutes=WINDOW_MINUTES - 1)
    # remove keys older than cutoff
    to_delete = [k for k in counts if k < cutoff]
    for k in to_delete:
        del counts[k]

def main(log_path, target_service):
    counts = defaultdict(int)
    newest_minute = None

    with open(log_path, 'r', encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                obj = json.loads(line)
            except json.JSONDecodeError:
                continue  # skip malformed lines
            if obj.get("service") != target_service:
                continue
            if obj.get("level") != "ERROR":
                continue
            ts = obj.get("timestamp")
            if not ts:
                continue
            try:
                dt = parse_iso8601(ts)
            except Exception:
                continue
            mb = minute_bucket(dt)
            counts[mb] += 1
            if (newest_minute is None) or (mb > newest_minute):
                newest_minute = mb
                purge_old(counts, newest_minute)

    if newest_minute is None:
        # No matching entries; print last 60 minutes relative to now
        newest_minute = minute_bucket(datetime.now(timezone.utc))

    start = newest_minute - timedelta(minutes=WINDOW_MINUTES - 1)
    cur = start
    while cur <= newest_minute:
        print(cur.strftime("%Y-%m-%d %H:%M"), counts.get(cur, 0))
        cur += timedelta(minutes=1)

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Usage: script.py /path/to/logfile service_name", file=sys.stderr)
        sys.exit(2)
    main(sys.argv[1], sys.argv[2])

Key points:- Streaming parse keeps memory bounded; counts dict holds at most ~60 keys.- Handles out-of-order logs by tracking the newest minute seen and purging older buckets.- Uses timezone-aware UTC minute buckets to avoid DST/offset issues.

Complexity:- Time: O(N) for N lines.- Space: O(WINDOW_MINUTES) (bounded ~60 keys).

Edge cases:- Malformed JSON or missing fields are skipped.- If no matching ERRORs appear, prints 60 minutes ending at current UTC minute.- If timestamps are from different timezones, they’re normalized to UTC.

EasyTechnical

89 practiced

Explain the difference between a runbook and a playbook. Provide a concise example runbook entry for 'database-primary-unavailable' that an on-call SRE can follow, including safety checks and verification steps. Also list what metadata (last-tested, owner, risk-level) must appear on every runbook entry.

Sample Answer

Runbook vs Playbook (concise):- Runbook: A step-by-step, low-ambiguity operational procedure for a specific condition (what to do). Targeted at on-call engineers to resolve known failures safely and quickly.- Playbook: Higher-level decision framework that includes diagnostics, multiple remediation paths, escalation criteria, and business context (how and when to choose actions). Often references multiple runbooks.

Example runbook entry — "database-primary-unavailable"

Metadata:- owner: db-sre-team- last-tested: 2025-10-01- risk-level: high- contact: oncall-db@company.com; Pager: #db-oncall

Symptom:- Primary DB is unreachable or read/write errors from apps; alert: DB_PRIMARY_DOWN

Pre-Preconditions / Safety:- Ensure you have access to runbook, cluster credentials, and maintenance window if required.- Notify stakeholders if estimated recovery >15min.- Do NOT failover if transaction replication lag > 5s unless instructed.

Steps:1) Confirm alert and scope- Check alert details and timestamp.- Verify primary status:

bash

kubectl exec -it svc/db-admin -- pg_isready -h primary.db.cluster
# or run cloud provider console check

2) Quick health checks- Check primary host ping, system load, disk:

bash

ssh db-primary "uptime; df -h /var/lib/postgresql; journalctl -u postgresql -n 200 --no-pager | tail -n 50"

- Check replication lag on a replica:

bash

psql -h replica1 -c "SELECT now()-pg_last_xact_replay_timestamp() AS lag;"

3) If process crashed but host healthy- Attempt safe restart:

bash

ssh db-primary "sudo systemctl start postgresql"

- Wait 30s, re-run pg_isready and check application error rate.

4) If primary host unreachable or data corruption suspected- Initiate controlled failover (only if replication lag <=5s and read-only replicas healthy):

bash

# use orchestrator/tooling:
orchestrator -c failover --candidate replica1 --force

- Monitor failover progress and application connections.

5) Post-action verification- Verify new primary accepts writes:

bash

psql -h new-primary -c "CREATE TEMP TABLE t(id int); INSERT INTO t VALUES(1); SELECT * FROM t;"

- Confirm replication from new primary to other replicas and app error rate returns to baseline.

6) Escalation- If failover fails or data inconsistency: page senior DB engineer and engineering manager; mark incident Sev1.

Rollback / Cleanup:- If original primary recovers, do NOT auto-promote back; follow controlled rejoin procedure documented in db-rejoin runbook.

What must appear on every runbook entry:- Title and clear trigger/symptom- Owner and contact info- Last-tested date- Risk-level (low/medium/high)- Preconditions and safety checks- Step-by-step actions with exact commands- Verification steps and success criteria- Escalation path and rollback guidance- Links to related docs/playbooks and post-incident review template

HardTechnical

73 practiced

You're leading an incident where, over several hours, hundreds of hosts show intermittent disk I/O latency spikes that cause sporadic application errors. Logs are sparse because services rely heavily on in-memory caches. Describe a step-by-step approach to detect and triage the issue with limited logs, including live tools to run (iostat, ioping, eBPF tracing), what instrumentation to add quickly, short-term containment to reduce customer impact, and long-term mitigations to prevent recurrence.

Sample Answer

Situation: Hundreds of hosts over several hours show intermittent disk I/O latency spikes causing sporadic app errors; logs are sparse due to heavy in-memory caching.

Step-by-step triage1. Rapid scope and impact- Query orchestration/CMDB to find affected hosts, services, zones, and timeline.- Correlate with alerts (SRE pager, monitoring graphs) to prioritize customer-facing services.

2. Live diagnostics (non-disruptive first)- On representative affected host run: iostat -x 1 10 to view util/await/%util per device.- Use ioping -c 20 /dev/sdX and ioping -c 50 /path/to/mount to measure latency distribution.- Check kernel IO stats: cat /proc/diskstats and iostat -x for queue depth.- Inspect dmesg/syslog for device errors (timeouts, reassigns).- Use atop/htop to spot processes with high I/O wait or blocked state.- If on VMs/cloud, check hypervisor/host-level metrics and storage pool health.

3. Deep tracing with eBPF (if available)- Run simple eBPF scripts (bcc tools): sudo biosnoop, cachestat, or biolatency to capture per-process / per-device IO latency histograms.- Example: sudo biolatency 1 > /tmp/biolatency.log (short run) and biosnoop to map pids causing spikes.- Use opensnoop/execsnoop to see sudden process activity.

4. Quick instrumentation to add- Add short-lived metrics: per-host disk latency histograms (histograms buckets), per-process IO ops/latency via node_exporter textfile or statsd.- Enable verbose OSS/driver logging on a small subset.- Add request-level tracing in app to capture cache miss rates and downstream storage calls (sampled traces).

5. Short-term containment- Shift traffic away from worst-affected hosts (load balancer draining/cordon nodes).- If bursty background jobs are causing IO, reduce concurrency / pause them.- Increase cache TTLs to reduce backend I/O temporarily.- If on shared storage, move critical volumes to healthy backend or switch to replicas.

6. Root-cause analysis & long-term mitigations- Analyze collected eBPF traces + device errors to determine cause: hardware failing, driver bug, noisy neighbor, backup jobs, or storage controller saturation.- Fixes: - Replace/repair failing disks or move to healthier pool. - Tune IO scheduler, adjust queue depths, enable writeback throttling (blkio/cgroups). - Improve app-side resilience: exponential backoff, circuit breakers on storage calls, better retries with jitter. - Improve observability: persistent per-host IO latency histograms, per-process IO metrics, sampled traces for cache misses, and SLOs/alerts on backend latency and error rates. - Automate mitigation: playbook to drain nodes and trigger live-migration or temporary traffic routing.- Post-incident: run blameless postmortem, capacity planning, and runbooks; add chaos tests for storage degradation.

Key reasoning: start with low-risk diagnostics, quickly gather per-host and per-process latency data (iostat, ioping, eBPF), contain impact by isolating hosts and throttling heavy writers, then implement fixes that address hardware, kernel, orchestration, and app resilience while improving observability to detect future incidents earlier.

MediumTechnical

61 practiced

You are facilitating a blameless post-incident review for a P0 outage. Outline a meeting agenda and the artifacts you will produce: a detailed timeline, evidence collected, RCA, corrective actions with owners and deadlines, impact quantification, and communication artifacts. Describe rules and facilitation techniques to keep the review blameless and constructive, and how remediation items are tracked to closure.

Sample Answer

Meeting agenda (60–90m):- 0–5m: Welcome, objective (learn, not blame), review norms- 5–15m: Incident summary (what customers saw, timeline snapshot)- 15–35m: Detailed timeline walkthrough (owner reads, evidence shown)- 35–55m: Root cause analysis discussion (why each contributing factor existed)- 55–70m: Corrective actions and mitigations (propose, assign owners, set deadlines)- 70–80m: Impact quantification & customer comms review- 80–90m: Wrap-up: decisions, next steps, follow-up meeting schedule

Artifacts produced- Detailed timeline: timestamped events, actions, alerts, runbook steps, responsible person — shared as a living Google Doc/Confluence page.- Evidence bundle: alert graphs, logs, trace spans, deployment manifests, runbook excerpts, comms (Slack/PagerDuty snippets), and screenshots — attached to the doc.- RCA: structured “5 whys” or causal factor tree with contributing factors categorized (human, process, code, infra).- Corrective actions: table with action, description, owner, priority, deadline, verification method, acceptance criteria.- Impact quantification: customer count, affected regions/tenants, downtime minutes, SLO burn, revenue/ops impact estimate.- Communication artifacts: postmortem executive summary, incident timeline for customers, status templates, and suggested status emails.

Blameless rules & facilitation techniques- Explicit blameless charter at start; focus on systemic causes.- Use facts-first: require evidence before attribution.- Facilitate with neutral language; interrupt finger-pointing, refocus on processes.- Ask “What allowed this to happen?” not “Who caused it?”- Time-box dives; defer deep technical debates to follow-ups with owners.- Encourage psychological safety: invite junior voices, anonymize sensitive comments if needed.- Use whiteboard or shared doc to map causal chains live.

Tracking remediation to closure- Track actions in a ticketing system (Jira/Asana) with link to postmortem doc.- Define verification steps (test, runbook update, monitoring alert created) and acceptance criteria.- Assign SLA for remediation (e.g., immediate mitigations 48h, medium 2 weeks, long-term 90 days).- Weekly status review for open items in a reliability board; escalate overdue items to engineering managers.- Post-implementation verification: run a tabletop or chaos test where applicable, and close only after verification artifact attached.- Quarterly review: ensure recurring themes are consolidated into reliability roadmap and SLO adjustments.

This approach ensures disciplined, evidence-based learning, timely remediation, and organizational improvement without assigning blame.

EasyTechnical

75 practiced

You are designing alerting for a microservice. Describe at least five alerting best practices (signal selection, actionable thresholds, aggregation, deduplication, ownership). Provide three examples of noisy or misleading alerts and show how you would improve each into an actionable alert with clear responder and remediation guidance.

Sample Answer

1) Alerting best practices- Signal selection: Alert on symptoms (user-visible errors, SLO breaches, latency) not on every internal metric. Prioritize signals that correlate with customer impact.- Actionable thresholds: Use thresholds tied to runbooks/SLOs (e.g., error rate > 5% for 5m) so alerts require a clear action, not informational noise.- Aggregation/windowing: Aggregate over sensible dimensions and short rolling windows (e.g., 5m) to avoid flapping from single-point spikes; group by service/region.- Deduplication & suppression: Collapse repeated alerts (rate-limit, backoff, or dedupe by fingerprint) and suppress lower-severity alerts during incidents to focus responders.- Ownership & routing: Each alert must include an owner/team, severity, and escalation policy. Attach a short runbook with first-step checks and remediation steps.

2) Noisy/misleading alert examples and improvements

Example A — Noisy: “CPU > 70%” fires continuously for stateless pods.Improvement: Replace with “CPU > 85% for 5m causing pod restarts or OOMs” routed to infra team. Runbook: check HPA, recent deployments, pod restart count; remediation: scale HPA, roll back if spike post-deploy.

Example B — Misleading: “Disk utilization > 80%” on ephemeral worker nodes.Improvement: Alert only on “root fs > 90% for 10m OR inode exhaustion OR ready disk errors” to storage team. Runbook: identify large files, rotate logs, replace failing disk; escalation to infra if remediation fails.

Example C — Spammy: “Service X downstream 5xx > 0” fires per endpoint every minute.Improvement: Aggregate to “Service X downstream 5xx rate > 1% sustained 3m” with fingerprinting by downstream service, sent to owning service team. Runbook: check downstream health, circuit-breaker state, retry/backoff metrics; remediation: enable fallback, roll back release, or open incident.

For each alert include severity, owner, annotated runbook steps, expected diagnostic queries, and post-incident tagging to improve future signal quality.

Unlock Full Question Bank

Get access to hundreds of Incident Management and Response interview questions and detailed answers.

Join thousands of developers preparing for their dream job.