Reliability and Operational Excellence Questions

Covers design and operational practices for building and running reliable software systems and for achieving operational maturity. Topics include defining, measuring, and using Service Level Objectives, Service Level Indicators, and Service Level Agreements; establishing error budget policies and reliability governance; measuring incident impact and using error budgets to prioritize work. Also includes architectural and operational techniques such as redundancy, failover, graceful degradation, disaster recovery, capacity planning, resilience patterns, and technical debt management to improve availability at scale. Operational practices covered include observability, monitoring, alerting, runbooks, incident response and post incident analysis, release gating, and reliability driven prioritization. Proactive resilience practices such as fault injection and chaos engineering, as well as trade offs between reliability, cost, and development velocity and scaling reliability practices across teams and organizations, are included to capture both hands on and senior level discussions.

MediumSystem Design

103 practiced

Create a runbook template for an on-call responder handling a payment gateway outage. The template must include triage checklist, health checks/commands, mitigation steps (circuit-breakers, fallback payments), rollback criteria, stakeholder communication templates, and post-incident follow-ups.

Sample Answer

Title: Payment Gateway Outage — On-Call Runbook (Template)

1) Incident Summary- Purpose: Steps to triage, mitigate, and restore payment gateway during outage.- Pager details: [Primary on-call], escalation: [SRE lead], SLA target: [X mins]

2) Triage checklist (first 10 mins)- Confirm alert source & scope (monitoring, error rates, customer reports)- Identify impact: payment auth failures, settlement delays, partial vs full outage- Check recent deploys / config changes (last 60 mins)- Capture timestamps, transaction IDs, sample failed requests- Set incident severity (sev1/2) and start timeline log

3) Health checks / commands (run & paste output)- Check gateway status page / upstream provider: curl https://provider/status- App logs for payment errors: kubectl logs -n payments deployment/pay-api --since=15m | grep -i "payment|auth|timeout"- Queue depth: redis-cli LLEN payment_queue- DB connectivity: pg_isready -h payments-db && psql -c "SELECT count(*) FROM payments WHERE status='processing';"- Circuit-breaker state: GET /admin/circuit-breakers (or consul/etcd key)- Latency/error metrics: grafana query or prometheus: increase(http_request_errors_total[5m])

4) Mitigation steps (priority order)- Step A: Open circuit-breakers to upstream provider if error rate > threshold to prevent cascading- Step B: Enable fallback payment processor (flip feature flag / DNS / routing) — validate credentials and limits- Step C: Throttle non-essential payment flows (subscriptions, retries) via rate-limit config- Step D: Serve cached payment-confirmation UIs with clear messaging- Step E: If queue backpressure, pause enqueues and increase worker horizontal autoscale

5) Rollback / restore criteria- Safe to rollback last deploy if: - Error rate returns to baseline on previous version in canary - Key health checks (DB, queue, downstream) green for 10 minutes- Criteria to close incident: - Success rate >99% for 15 mins, backlog drained < threshold, stakeholders acknowledged

6) Stakeholder communications (templates)- Initial (to execs/PMs): "Observed payment failures affecting X% of transactions since [time]. Impact: [customers affected]. Investigating; mitigation in progress. Next update in 15 mins."- Technical update (to engineering/partners): "Root cause hypothesis: [e.g., upstream auth latency]. Actions taken: opened circuit-breaker, enabled fallback processor, paused retries. Current metrics: success rate X%. Next steps: [list]."- Customer-facing: "We're experiencing payment delays. We're working to restore service; no customer card data exposed. Expected update in 30 mins. For urgent cases contact [support link]."

7) Post-incident follow-up- Timeline + annotated logs- RCA: root cause, contributing factors, deploy/config changes- Action items (owners + due dates): e.g., add synthetic tests, improve circuit-breaker thresholds, failover drills- Metrics to track: mean time to detect/mitigate/resolve, recurrence risk- Schedule postmortem review within 3 business days; share learnings with Sales/Customer Success and update runbook

8) Quick links & runbook metadata- Runbook owner: [name]- Links: dashboards, runbooks, provider status, playbooks for rollback, pager rota

Use conservative changes during incident; prefer routing to fallback over code changes.

EasyTechnical

76 practiced

Explain the differences between observability, monitoring, and logging. For a service that experiences a 2s latency regression, describe which telemetry you would consult first and why, and how traces, logs and metrics work together to identify root cause.

HardSystem Design

84 practiced

Design disaster recovery for a Kafka-based streaming platform that requires exactly-once semantics and has global consumers with stateful processing. Discuss replication (cluster vs MirrorMaker), controller responsibilities, consumer group recovery, state stores (RocksDB/changelog topics), RTO/RPO considerations, and cross-region failover implications.

Sample Answer

Requirements & constraints:- Exactly-once end-to-end for stateful stream processing (producer -> Kafka -> consumers with state stores).- Global consumers across regions, low RTO (minutes) and low RPO (seconds/minutes).- Support cross-region failover, regulatory data residency concerns.

High-level approach:- Deploy active-active Kafka clusters per region with topic-level replication using Confluent Replicator / MirrorMaker 2 (MM2) for cross-region replication, plus intra-cluster Replication Factor (RF) >=3 and Rack Awareness. Use MM2 for asynchronous geo-replication and rely on local clusters for read/write locality.

Replication: cluster vs MirrorMaker- Intra-region: rely on Kafka’s ISR/RF for durability and leader election.- Cross-region: use MM2 (consumer/producer based) for topology flexibility, metadata sync, and offset translation. For true synchronous guarantees across regions (very high RPO requirements) consider WAN-optimized storage replication (rare) — but synchronous cross-region Kafka is impractical for latency-sensitive workloads.

Controller responsibilities- Keep controllers per cluster (KIP-500 considerations). Controllers handle partition leadership, replicas, and reassignments. Ensure controller quorum health via ZK (older) or KRaft; monitor controller lag and partition reassignment churn during failover. Automate safe leader election (preferred-leader settings off) and use prefer-unchanged-leader policies to limit flapping.

Stateful consumers & consumer-group recovery- Use Kafka Streams/Fluent/Beam with EOS (exactly-once) enabled: transactional producers + read-process-write pattern with consumer isolation.level=read_committed.- For consumer group failover, ensure committed offsets are stored in Kafka (consumer offsets topic) and that MM2 replicates offsets and consumer group metadata when crossing regions. On failover, restart consumers to resume from last committed transactional offsets. Implement deterministic partition assignment and sticky assignment to limit state movements.

State stores (RocksDB & changelog topics)- Local RocksDB for low-latency state; every state change is written to a changelog topic via EOS transactions. Changelog topics must have RF>=3 and be replicated by MM2 to remote region(s). On failover, reconstruct RocksDB by consuming changelog topics up to the last committed transaction offset (ensures exactly-once recovery). Tune RocksDB checkpoints and enable incremental snapshots to speed restore.

RTO / RPO trade-offs- RPO: Determined by replication model. With asynchronous MM2, expect RPO = replication lag (seconds–minutes). To lower RPO, increase replication throughput, use higher bandwidth links, and reduce replication batch sizes (at cost of throughput).- RTO: Time to rehydrate state and resume consumers. Improve by (1) periodic RocksDB snapshots uploaded to shared durable storage (S3/GCS) per region, (2) parallel changelog replay, (3) pre-warmed consumer instances and automated orchestration (K8s + operators) for quicker restart. Target RTO in minutes by combining snapshots + incremental changelog.

Cross-region failover implications & runbook- Decide failover model: active-passive (simpler, deterministic) vs active-active (complex: conflict resolution & idempotency). For exactly-once, active-passive is safer: promote target region, pause producers in primary, ensure all local transactions have been replicated/consumed, then switch DNS/ingress.- Runbook steps: (1) detect outage & freeze writes upstream, (2) ensure MM2 replication caught up for critical topics and changelogs, (3) promote replicated topics (update topic configs and leaders), (4) start consumers in target region; ensure they validate offsets/transactional markers, (5) resume producers with transactional producerId resets.- Testing: regular DR drills: full failover rehearsals, partial partition failovers, and state rehydration tests. Validate end-to-end exactly-once invariants.

Operational controls & monitoring- Monitor: ISR sizes, replication lag (intra & inter-region), transaction coordinator metrics (pending txns), changelog consumer lag, RocksDB restore times.- Automate: leader reassignment scripts, MM2 lag alerts, automated snapshot and restore pipelines, and post-failover reconciliation checks (hashes, counts).- Security & compliance: encryption in transit, restricted cross-region replication, and audit of failover events.

Trade-offs summary- Exactly-once globally: achievable but costly. Recommended pattern: local active clusters with MM2 async replication + transactional processing + changelog replication + snapshots — use active-passive for critical exactly-once workloads. For lower-cost/latency-tolerant bounded-duplication, relax to at-least-once with idempotent sinks.

This design balances exactly-once semantics, operational feasibility, and pragmatic cross-region recovery while making explicit RTO/RPO trade-offs and automation needs.

MediumTechnical

90 practiced

Design a test plan for validating a disaster recovery (DR) plan on an annual cadence. Include scope of tests (failover vs failback), automation opportunities, verification steps / acceptance criteria, rollback plans, scheduling to minimize customer impact, and reporting that satisfies compliance and leadership.

Sample Answer

Overview / objectives:- Validate that critical workloads meet RTO/RPO targets, runbooks are effective, and stakeholders (ops, security, compliance, customers) have confidence. Annual full DR test with quarterly partial exercises.

Scope of tests:- Failover tests (primary → DR): full-production simulation for non-customer-impacting environments; partial for customer-impacting services (subset of traffic or pilot tenants).- Failback tests (DR → primary): verify data sync, configuration drift resolution, and cutback procedures.- Ancillary tests: network re-routing, DNS TTL changes, storage snapshots, identity and secrets access, third-party integrations.

Automation opportunities:- Infrastructure as Code to reprovision DR environments (Terraform/CloudFormation).- Automated failover orchestration (runbooks encoded in scripts/Orchestration tools like AWS Step Functions or Azure Automation).- Smoke-test suites that validate service endpoints, auth, and DB integrity post-failover.- Automated rollback triggers on health-check failures.

Verification / acceptance criteria:- Pre-check: latest backups exist, replication lag < threshold.- Post-failover verification: services respond to synthetic transactions, key metrics within SLA (latency, error rate), data consistency checks pass, security controls active.- Acceptance: all critical business transactions complete end-to-end and RTO/RPO met within agreed window.

Rollback plans:- Clear rollback decision points and automated/manual triggers.- Snapshot-based restore or DNS rollback playbooks with estimated time and owners.- Contingency: if failback not possible within defined window, escalate to execs and extend DR runbook.

Scheduling to minimize impact:- Use maintenance windows and low-traffic windows per region; select pilot tenants or read-only modes where possible; notify customers and provide status pages; run full-production simulation in isolated staging that mirrors production.

Reporting & compliance:- Test report template: objectives, scope, timeline, pre-checks, step-by-step actions, verification logs, metrics (RTO/RPO actual vs target), incidents, root cause, remediation plan, sign-offs.- Store artifacts: audit logs, screenshots, automated test outputs, IaC commits, runbook versions.- Deliverables: executive summary for leadership, technical appendix for ops, compliance pack with evidence and attestation.

Governance & continuous improvement:- Post-test retrospective, update runbooks/IaC, prioritize fixes, and schedule follow-up mini-tests for remediations.

HardTechnical

85 practiced

You are asked to reduce on-call burnout while improving incident handling. Propose a program that includes runbook automation, schedule design, pager reduction strategies, role of automation and playbooks, training, and measurable metrics (pager frequency, MTTA, MTTD, psychological metrics). Explain how you'd pilot and evaluate success.

Sample Answer

Situation: On a recent engagement the client had high on-call burnout—frequent noisy pages, long toil, and inconsistent incident response. They asked for a scalable program to reduce burnout while improving incident outcomes.

Program proposal (summary):1) Runbook automation- Convert high-frequency manual runbooks into executable playbooks (IaC/automation pipelines) using tools like Ansible/RunDeck or orchestration via service catalog.- Implement one-click remediation actions exposed in incident UI (e.g., paged alert shows “Restart worker” button that runs safe, idempotent script).- Add automated pre-checks (canary checks, rate-limited rollbacks) to reduce risk.

2) Schedule design- Move to 2-week rotations with 8-12 hour shifts, primary/secondary/owner model, protected weekends for each engineer every 4 rotations.- Enforce hard caps: max 2 weekend calls/month, max 1 week of continuous coverage.- Use follow-the-sun for global teams; enforce handover checklist.

3) Pager reduction strategies- Triage alerts into severity buckets; route informational events to dashboards not pages.- Raise alert thresholds with dynamic baselines and rate-limiting; use anomaly-detection to reduce duplicates.- Implement suppressions during planned ops and noisy deploys; auto-group related alerts.

4) Role of automation and playbooks- Playbooks contain decision trees, automated steps, and escalation paths; preferred action is automated where safe.- Automation handles remediation and data collection (logs, snapshots) so humans handle decisions.- Version control for playbooks with postmortem linkage and audit trail.

5) Training and culture- Mandatory runbook run-throughs and tabletop drills quarterly.- Pair on-call shadowing: new on-call shadows an experienced on-call for 2 rotations.- Psychological safety: blameless postmortems, guaranteed recovery time after severe incidents, EAP info.

6) Measurable metrics & targets- Pager frequency: reduce actionable pages per engineer/week from X to ≤2.- MTTD: target <= 5 min for critical; MTTA (time-to-ack) <= 2 min; MTTR <= 30 min for priority-1.- Noise ratio: percent of pages requiring human intervention <= 30%.- Psychological metrics: weekly pulse (5-question survey), burnout score, Net Emotional Value; target 20% improvement in wellbeing over 6 months.- Adoption: % automated remediation coverage for top-10 incident types >= 70%.

Pilot and evaluation- Pilot scope: one service/domain with highest page volume for 8–12 weeks.- Phase 1 (weeks 1–2): baseline metrics, inventory runbooks and alerts.- Phase 2 (weeks 3–6): implement automated playbooks for top 3 incident types, adjust thresholds, introduce schedule changes and shadowing.- Phase 3 (weeks 7–12): run drills, collect metrics, iterate.- Evaluation: compare baseline vs. pilot on pager frequency, MTTD/MTTR, human-handled pages, and pulse survey. Success criteria: >40% reduction in actionable pages, MTTD improvement to target, and measurable improvement in wellbeing. Rollout plan tied to meeting targets; if automation introduces risk, rollback to narrower automations and repeat.

Why this works- Targets both symptom (pages) and root causes (noisy alerts, manual toil).- Automation reduces cognitive load and accelerates recovery; schedules and culture preserve human capacity.- Clear metrics + short pilot allow data-driven decisions and safe, phased rollout.

Unlock Full Question Bank

Get access to hundreds of Reliability and Operational Excellence interview questions and detailed answers.

Join thousands of developers preparing for their dream job.