Comprehensive operational topic covering the responsibilities, processes, and practices involved in supporting production systems and managing incidents. Candidates should be able to describe on call scheduling models and burden distribution across teams, expected incident volume and typical severity levels, incident triage steps and severity assessment to prioritize and escalate appropriately, and criteria for involving security teams or external vendors. It includes monitoring and alerting strategy, alert thresholds and noise reduction, service level objectives and service level indicators, and tooling for incident management. Candidates should also be able to explain runbooks and playbooks for common incident types, hands on troubleshooting during live incidents, root cause analysis approaches, deployment and rollback practices, and measures to reduce mean time to detection and mean time to recovery. The topic also covers incident communication practices, escalation procedures, post incident activities such as blameless postmortems and follow up actions for continuous improvement, and considerations about allocation of time between maintenance and feature work to preserve production readiness.
MediumSystem Design
83 practiced
Design a runbook for a primary database outage affecting the order service. Include quick diagnostic checks (connectivity, replica status, disk usage), safe mitigation steps (read-only mode, failover), verification steps, rollback criteria, stakeholders to notify (product, legal, customers), and post-incident follow-up actions.
Sample Answer
Goal: Restore order-service DB availability safely, preserve data integrity, and communicate clearly.Immediate actions (first 5 minutes)- Declare incident and assign incident lead (IL).- Post “Order service DB outage” in incident channel with affected scope and ETA.Quick diagnostic checks (run in parallel)- Connectivity: from app host: curl http://order-service/health; psql -h primary -U monitor -c '\l' (check TCP, DNS).- DB process: ps aux | grep postgres or systemctl status.- Replica status: on primary: SELECT pid, state, application_name, client_addr FROM pg_stat_replication; on replica: SELECT pg_is_in_recovery();- Disk: df -h; SELECT pg_size_pretty(pg_database_size('orders'));- Error logs: tail -n 200 /var/lib/postgres/data/log/postgresql-*.log- Resource: top/iostat, and cloud provider console for instance health.Safe mitigation steps (ordered)1. If write latency high but DB up: enable read-only on non-critical paths; throttle writers in app config.2. If primary unresponsive and replica healthy: initiate controlled failover to replica (pg_auto_failover, Patroni or RDS failover). Follow automated failover tool steps; do NOT promote if replication lag > configured threshold.3. If disk full: free space (rotate logs, temp file cleanup) then restart DB; avoid DROP unless safe.4. If corruption detected: stop writes; promote replica from latest consistent WAL; engage DBA.Verification after mitigation- Health: application /health returns OK; DB accepts writes (insert test record) and reads.- Consistency: run validation job comparing order counts/recent orders between backup and new primary for last 1h.- Replication: pg_stat_replication shows replicas streaming and lag < threshold.Rollback criteria- Rollback if promoted node shows data loss vs backups, or application errors persist >30 minutes, or replication lag exceeds safe threshold causing inconsistent reads. Rollback means revert app traffic to read-only primary and restore from snapshot/backup under DBA direction.Stakeholders to notify- Immediate: Product manager, Engineering manager, On-call DBA, SRE, Support lead.- Within 1 hour: Legal (if PII or regulatory risk), Security (if breach suspected).- Customer communications: Support drafts status page and templated customer notification; PM approves.Post-incident follow-up- Postmortem within 72 hours: timeline, root cause, actions, owners.- Action items: add runbook gaps, automation (health checks, auto-failover tuning), improved monitoring/alerts (replication lag, disk), rehearsal of failover twice quarterly.- Metrics: MTTR, number of failed transactions, customer impact.- Close loop: verify fixes in staging, schedule RCA presentation to stakeholders.
MediumTechnical
98 practiced
Describe a concrete plan to measure and reduce Mean Time To Detection (MTTD) for a microservice over a quarter. Include instrumentation changes, alert routing improvements, synthetic checks vs real-user metrics, dashboards, and an experiment you would run to validate improvement.
Sample Answer
Goal: cut MTTD for the Orders microservice from 30m to <10m in one quarter.Plan (12 weeks):Week 0–2: baseline & instrumentation- Measure current MTTD from alert timestamps vs incident start (define incident = first user-impacting error).- Add structured logs (request_id, trace_id, user_id, latency, error_code) and ensure correlation with distributed tracing (OpenTelemetry).- Emit fine-grained metrics: per-endpoint error_rate, p95_latency, saturation (queue depth, threadpool), and business KPIs (checkout-failure-rate).- Add RUM events for real-user errors and latency (sampled).Week 3–6: detection surface & synthetic checks- Create synthetic probes: critical-path health checks that exercise checkout flows every 30s from 3 regions; check HTTP status, response body assertions, and business validation (cart->checkout->payment stub).- Instrument health endpoints to return dependency health (DB, payment gateway, cache).- Configure error budget alert rules: fast alerts for sharp spikes (error_rate > 3% in 1m), and slow-burn alerts for sustained degradation (error_rate > 1% for 15m).Week 7–9: alert routing & dashboards- Route alerts by severity: P1 to on-call via PagerDuty with runbook link and playbook; P2 to Slack incidents channel and on-call watchlist; noisy alerts to internal metrics team.- Enrich alerts with context: last 5 traces, top 5 failing endpoints, recent deploys, and synthetic check status.- Build dashboards: Executive KPI (checkout success, MTTD), SRE triage view (top errors, top traces, host/container failures), synthetic map (region probe statuses), and RUM heatmap (client-side errors by browser/region).Week 10–12: experiments & validation- Run an A/B experiment: split traffic 80/20 to two detection configs. Variant A = baseline alerts only. Variant B = added synthetic + RUM-derived alerts + enriched context. Randomly route new users’ traffic for two weeks.- Measure MTTD, false positive rate, time-to-ack, and MTTR for both groups.- Success criteria: Variant B reduces median MTTD by ≥40% and does not increase noisy alerts by >15%.Ongoing:- Triage weekly false positives, adjust thresholds, apply anomaly-detection on trace error signatures, and run post-incident reviews to refine runbooks.Why this works:- Combined synthetic checks catch deterministic availability regressions quickly; RUM catches real-user regressions that aren’t covered by probes. Correlated traces/logs reduce cognitive load for on-call, speeding detection and diagnosis. The controlled experiment validates that instrumentation and routing changes actually reduce MTTD without excess noise.
HardSystem Design
105 practiced
Design an automated playbook runner that can execute runbook steps (toggle feature flag, restart service, scale-up) in response to alerts. Describe required safety features: confirmation gates, idempotency of steps, audit logging, RBAC, rate limits, test/canary mode, and automated rollback if remediation fails.
Sample Answer
Requirements (clarify):- Triggered by alerts, pick and run a playbook composed of discrete steps (feature-flag toggle, service restart, scale-up).- Safety: confirmation gates, idempotency, audit logging, RBAC, rate limits, test/canary mode, automated rollback.- Non-functional: low-latency for critical remediations, high reliability, observable, auditable.High-level architecture:Alerting → Playbook Orchestrator (state machine) → Step Workers (connectors for feature flags, orchestration layer for services/infra) → Execution Store / Audit Log → Monitoring + Rollback EngineCore components and responsibilities:1. Playbook Orchestrator (stateful):- Represents playbook as a finite state machine with step metadata: idempotency key, timeout, retry policy, compensating action.- Enforces confirmation gates and rate limits before executing steps.- Stores checkpoints so execution can resume/replay.2. Step Workers / Connectors:- Implement APIs to target systems; each worker ensures idempotency (use resource-level idempotency keys, e.g., toggle request includes desired state + request-id).- Return canonical result + diagnostics.3. Execution Store & Audit Log:- Immutable append-only log (WAL or event store) capturing who/what/when/input/output. Use signed entries and retention/archival policies for compliance.4. RBAC & Approval Service:- Evaluate caller identity and required approvals. Support: - Auto approvals for low-risk steps. - Manual multi-actor confirmation (2-of-3) for high-risk steps. - Time-bound approval tokens and escalation flows.5. Rate Limiter & Circuit Breaker:- Token-bucket per-target and global rate limiter to prevent flapping.- Circuit breaker that pauses further actions on a target when error thresholds exceeded.6. Test / Canary Mode:- Dry-run mode that validates step inputs, simulates calls, and records predicted changes without applying.- Canary execution: run on a subset (single instance / isolated environment) first; observe metrics and only proceed if success criteria met.7. Rollback Engine / Compensating Actions:- Every mutating step must register a compensating step (explicit rollback) or snapshot prior state (e.g., previous scale, previous flag state, deployment image, service PID).- Orchestrator monitors post-step health checks and SLO metrics. If health deteriorates or checks time out, it triggers compensating actions in reverse order or hits an automated rollback policy.- Support semi-automated rollback: attempt automatic rollback, notify on failure and require human approval to continue.Data flow:1. Alert arrives; playbook selected via matching rules.2. Orchestrator checks RBAC and required approvals.3. If manual confirmation required, pause and notify approvers (UI/Slack/email).4. On approval, orchestrator executes steps sequentially (or in parallel where safe).5. Each step recorded to Execution Store; monitoring probes evaluate system health.6. On failure, orchestrator invokes rollback engine, records outcome, and alerts operators.Idempotency details:- Use idempotency keys and declare desired final state instead of imperative operations (e.g., "set feature X = off" rather than "toggle").- Workers implement compare-and-set or use provider-supported idempotent APIs.- Replay safe: orchestrator uses checkpoints and deduplicates based on execution-id.Confirmation gates:- Configurable per-step risk level: AUTO, SINGLE_APPROVER, MULTI_APPROVER.- Support “time-window” approvals and policy-driven skip for emergency roles (safeguarded by audit and post-mortem requirements).Audit & observability:- Immutable audit log with queryable events, change diffs, and links to monitoring graphs.- Emit structured events to SIEM and incident management (PagerDuty).- Retain run artifacts (inputs, outputs, logs) for N days.Scalability & reliability:- Stateless orchestrator frontends with a durable execution store (e.g., PostgreSQL + event store or DynamoDB + SQS for tasks).- Horizontal workers autoscaled; backpressure via rate limiter.- Leader election for orchestrator instances to run scheduled tasks and maintain single-writer semantics for a playbook run.Security & governance:- Principle of least privilege for connectors; temporary credentials issued for runs.- Approval audit trails, tamper-evident logs (WORM), retention/purge policies.- Change review workflow integrated with CI/CD for playbook updates (tests, canary runs).Trade-offs:- Strong safety (multi-approvals, canaries) increases latency for remediation; mitigate via risk tiers/roles for emergency fast-paths.- Storing snapshots for rollback increases storage and complexity; for some targets rely on explicit compensating actions instead.- Complex orchestration logic increases system complexity; mitigate by building a small core orchestrator and pluggable worker SDKs.Example sequence (feature flag rollback scenario):- Playbook: Canary toggle flag -> monitor error rate -> scale-up if needed -> full toggle -> monitor -> complete.- Each step: desired-state API call with idempotency key, recorded snapshot of previous flag state.- If error spike after canary, orchestrator triggers rollback: set flag to snapshot state, scale back, log events, notify.This design balances safety and speed by combining policy-driven approvals, idempotent operations, immutable audit trails, controlled rollouts, and deterministic rollback mechanisms.
MediumTechnical
83 practiced
You are the on-call engineer and receive an alert indicating a spike of HTTP 500 errors for the payments API across a region. Walk through your triage steps: what dashboards and logs do you check first, how do you determine impact and scope, which quick mitigations do you attempt, when do you escalate, and how do you communicate status?
Sample Answer
Situation: I get an alert showing a spike of HTTP 500 errors for the payments API across a region while on-call.Triage steps I follow (priority-first):1. Rapid assessment (first 5 minutes)- Check SLO/alert dashboard (PagerDuty/Alertmanager, Grafana): confirm alert trend, start time, error-rate vs. baseline.- Open APM/tracing (Datadog/NewRelic/Jaeger): look for increased latency, exception traces, which service/component is failing.- Check global and regional rate/traffic dashboards (CDN, load balancer, API gateway) to see if traffic surge or bad clients coincide.2. Logs and focused debugging (5–15 minutes)- Tail structured logs for payments service (ELK/CloudWatch): search for 500 stack traces, common exception message, upstream failures (DB, auth, third-party payment processor).- Inspect recent deploys / config changes (CI/CD dashboard, deploy history) and infra events (autoscaler, instance health, infra alerts).- Correlate with feature flags or config drifts.3. Determine impact & scope- Query user-facing metrics: number of failed transactions, percent of traffic affected, business impact (revenue, highest-value customers).- Check region/container/instance distribution: single AZ, whole region, or global.- Identify affected endpoints (charge, refund, webhook) and customer segment (internal vs external).4. Quick mitigations (while investigating)- If a bad deploy suspected: rollback to last green release.- If overload or DB connection exhaustion: scale up replicas or increase DB connection pool, enable circuit breaker or throttling at gateway.- If third-party payment provider outage: fail fast with informative 503 or route to backup provider.- Apply temporary rate limit or feature-flag disable for noncritical flows to reduce load.5. Escalation criteria & timing- Escalate to Senior/On-call Lead and SRE within 10–15 minutes if business-critical traffic is impacted or mitigation not found/working.- Loop in product/merchant support when customer-visible incidents exceed SLAs or large customers affected.- Engage vendor/third-party support if external dependency implicated.6. Communication- Post an initial incident message (Slack/Incident channel) within 10 minutes: summary, scope, initial mitigation, and next update cadence (e.g., every 15 minutes).- Update status page/public incident board with high-level impact and ETA for fix.- Provide clear updates: actions taken, findings, impact metrics, next steps. After resolution, publish a postmortem with root cause, timeline, and remediation plan.Why this approach: prioritize real-user impact, isolate scope fast, attempt low-risk rollbacks or throttles, and surface collaboration early to restore service while keeping stakeholders informed.
MediumTechnical
135 practiced
Describe the process and escalation path for a Sev1 incident in an enterprise: list roles (incident commander, communications lead, triage engineers, exec liaison), initial responsibilities, expected timelines for first response and containment, and when to escalate to a war room or executive briefing.
Sample Answer
Situation: For a Sev1 outage I follow a clear incident command process so work is rapid, coordinated, and auditable.Roles & quick responsibilities:- Incident Commander (IC): assumes overall control, sets priority, assigns owners, and drives decisions. IC tracks timeline and declares containment/mitigation.- Communications Lead: owns stakeholder comms (status pages, Slack, email), maintains scheduled updates and templates, and escalates externally when required.- Triage Engineers (on-call + SMEs): own technical diagnosis, run runbooks, implement hotfixes or rollbacks, and report progress to IC.- Exec Liaison: informs leadership, prepares executive briefing and business impact summary, arranges approvals for high-risk actions.- Scribe/Recorder: logs timeline, actions, and artifacts for postmortem.Initial responsibilities & timelines:- First response: acknowledge and assemble within 5–15 minutes of alert.- Triage/start mitigation: within 15–30 minutes; determine blast radius and whether to mitigate (e.g., circuit-breaker, rollback).- Containment goal: initial containment or workaround within 60–120 minutes.Escalation to war room:- Open a war room (voice/video + persistent chat) when the event exceeds the containment SLA (30–60 min without clear path), affects multiple customers/regions, or requires cross-team coordination. IC declares war room and invites triage leads, product, SRE, and security as needed.When to brief executives:- Immediately notify Exec Liaison if the incident likely causes significant revenue/customer impact or regulatory exposure.- Provide an executive briefing within 60–120 minutes if service degradation persists or major customers are affected; update at agreed cadence (e.g., every 60 minutes).Closure & follow-up:- IC declares recovery and hands off to post-incident owner.- Within 24–72 hours conduct a blameless postmortem with action items, owners, and timelines.This cadence balances speed with clear ownership so engineers can act decisively while leadership stays informed.
Unlock Full Question Bank
Get access to hundreds of On Call and Production Readiness interview questions and detailed answers.
