Reliability and Operational Excellence Questions

Covers design and operational practices for building and running reliable software systems and for achieving operational maturity. Topics include defining, measuring, and using Service Level Objectives, Service Level Indicators, and Service Level Agreements; establishing error budget policies and reliability governance; measuring incident impact and using error budgets to prioritize work. Also includes architectural and operational techniques such as redundancy, failover, graceful degradation, disaster recovery, capacity planning, resilience patterns, and technical debt management to improve availability at scale. Operational practices covered include observability, monitoring, alerting, runbooks, incident response and post incident analysis, release gating, and reliability driven prioritization. Proactive resilience practices such as fault injection and chaos engineering, as well as trade offs between reliability, cost, and development velocity and scaling reliability practices across teams and organizations, are included to capture both hands on and senior level discussions.

MediumTechnical

85 practiced

A payment service experienced a 30-minute incident that affected approximately 2% of transactions. Describe how you would quantify customer impact (including monetary exposure and UX degradation), choose an incident severity level, and draft the initial and follow-up communications for internal teams and external stakeholders.

Sample Answer

Quantify customer impact- Scope: 2% of transactions over 30 minutes. If avg TPS = T, total impacted transactions = 0.02 * T * 1800. Use metrics: transaction count, dollar value per transaction (mean, 95th), failure types (failed, delayed, duplicate).- Monetary exposure: compute lost/failed value = sum(value of failed transactions) + estimated cost of retries/chargebacks/refunds. Example: if 10k tx during window, 2% = 200 tx, avg $50 → immediate exposure ≈ $10k. Add downstream costs (refund ops, interchange fees) as percent estimate.- UX degradation: measure latency spike, error rates, successful-but-delayed percentage, user-visible errors (payment declined). Map to SLOs: compare to payment success rate SLO and latency SLO to estimate error budget burn.

Severity selection (SRE rationale)- Severity = P2 (High) if: financial impact measurable, partial customer-facing failure affecting payments (2% non-trivial), SLO breach likely, requires cross-team coordination. Justification: partial but customer-visible payment failures, potential revenue loss and reputational risk; not full outage so not P1.

Initial communications- Internal (15-min incident banner):Subject: INCIDENT — Payments degraded (2% tx) — InvestigatingBody: Timeline: detected 14:02. Impact: ~2% of transactions failing or delayed (30m). Scope: payment gateway / region X / all methods? Actions: paging on-call, throttling retries, enabling fallback gateway, collecting logs/traces, preserving data. Owner: SRE on-call (name). Next update: 15 min or sooner. Severity: P2.

- External (customer-facing status page update):Title: Partial payment failures reportedBody: We are investigating an issue affecting a subset (~2%) of payment attempts between 14:02–14:32 UTC. Some customers experienced failed or delayed payments. We have identified impacted subsystem and are working on mitigation. No action required unless transaction failed — then retry or contact support. Next update: within 60 minutes.

Follow-up communications (post-mitigation)- Internal (incident resolution summary within 1 hour):Subject: RESOLVED — Payments incident; postmortem plannedBody: Root cause (initial): e.g., degraded upstream gateway timeout + retry storm; Mitigation: disabled aggressive retries, rolled back deploy, switched to fallback gateway at 14:18, service normalized by 14:32. Impact: ~2% tx (200 tx), estimated immediate monetary exposure $10k, error budget burn X%. Actions: retain logs, collect metrics for RCA, schedule postmortem within 72 hours with dev, product, and payments ops. Owner: SRE lead.

- External (status page and customer email if affected/high-value clients):Status page: RESOLVED — Issue affecting subset of payments between 14:02–14:32 UTC. All services restored. If you saw a failed payment, please retry or contact support.Customer email (to affected merchants): Apology, summary of impact (timeframe, % transactions), what we did to remediate, estimated exposure and reimbursements/refund policy, mitigation to prevent recurrence (e.g., improved circuit-breaker, retry backoff, monitoring), timeline for formal postmortem and contact for questions.

Metrics & next steps- Quantify exact counts/values, update SLA/SLO dashboards, calculate error budget impact, run retrospective RCA with concrete action items (deploy safer retry logic, enhance synthetic checks for payment success, add real-time monetary-impact alerts).

EasyTechnical

78 practiced

Explain the practical difference between monitoring and observability. Give a real example of a production problem that a monitoring-only setup might miss but an observable system would reveal, and suggest the instrumentation changes required to surface that problem.

Sample Answer

Monitoring vs observability — short and practical:- Monitoring: predefined checks and metrics (CPU, memory, request rate, error rate, synthetic healthchecks) with thresholds and alerts. It tells you "something is wrong" when known signals cross thresholds.- Observability: a property of the system that lets you ask new questions about unknown failures by using high-cardinality telemetry (structured logs, traces, metrics/histograms) and correlating them. It lets you discover root cause without having pre-defined alerts for that exact failure mode.

Concrete production example:Situation: Users intermittently see extremely high tail latency (P99 spikes) for an API. A monitoring-only setup has CPU/memory, request-rate, and 5xx error-rate alerts—none fire, so on-call assumes healthy. The problem persists and is hard to reproduce.

Why monitoring-only misses it:- Aggregate metrics (average latency, error-rate) can remain normal while tail latency spikes for a small subset of requests.- No request-level context exists, so you can’t correlate high latency with specific downstream calls, user IDs, client versions, or retry chains.

What observability reveals:- Distributed traces show a pattern: a small fraction of requests invoke a downstream cache miss that triggers a long database read and then client retries amplify load. Traces show large retry counts and long downstream spans causally linked to the P99 requests. Structured logs tied by request-id show the same user-agent and tenant. A latency histogram exposes the heavy tail.

Instrumentation changes to surface this:- Add distributed tracing (propagate a request-id/trace-id) across services; sample enough traces to capture tail behavior (adaptive or tail-based sampling).- Export latency histograms (not just averages) with high-resolution percentiles (p50/p95/p99/p999) per endpoint, per instance, and tagged by important dimensions (tenant, endpoint, client-version).- Emit structured logs including request-id, downstream call IDs, retry count, cache hit/miss, and relevant headers.- Add counters/gauges for downstream call latency and retry counts; break them out by dependency.- Implement tail-based tracing or increase sampling during SLA breaches to capture rare events.

Result: With these changes, on-call can quickly pivot from “latency looks fine” to a traced path showing the cache-miss → DB-read → client retry amplification, allowing a fix (cache TTL tuning or retry backoff) to be implemented and validated with p99 and trace dashboards.

HardTechnical

104 practiced

How would you scale reliability practices—SLOs, error budgets, runbooks, and incident-review processes—across a 200-engineer organization? Propose organizational changes, tooling and templates, incentive mechanisms, and quantitative metrics to measure adoption and effectiveness over time.

Sample Answer

Situation / goal: Scale consistent SRE practices (SLOs, error budgets, runbooks, post-incident reviews) across ~200 engineers so reliability is owned, measurable, and improves iteratively.

Organizational changes:- Create a Reliability Guild (cross-functional reps from each product team + SRE leads) that meets biweekly to set standards, unblock teams, and approve SLO templates.- Embed 1–2 SRE advocates per product squad (rotation + mentorship) responsible for local adoption.- Executive sponsorship: quarterly reliability review included in OKRs with an SRE leader reporting to CTO.

Tooling & templates:- Standard SLO template (service, user journey, metric, window, alert thresholds, error-budget policy).- Central SLO dashboard (Grafana/Chronograf) fed by Prometheus/OpenTelemetry; single source of truth.- Runbook template stored in company wiki with playbook sections: symptoms, mitigation steps, key logs/queries, rollback process, owner, runbook test procedure.- Incident Review template (timeline, root cause, contributing factors, corrective actions, owners, deadlines) in Jira/Confluence with required SLA for completion.

Incentive mechanisms:- Make error-budget burn a gate for non-urgent launches; teams that respect budgets earn “speed credit.”- Include reliability goals in performance reviews and team OKRs (e.g., maintain SLO >= target; close P1 action items within 30 days).- Recognition program: monthly “Reliability Hero” and team-level badges for consistently low P1 recurrences.

Metrics to measure adoption & effectiveness (quantitative):- Coverage: % of customer-facing services with approved SLOs (target: 100% in 9 months).- Runbook completeness: % of services with runbooks passing an automated validation checklist (target: 90%).- Incident process SLAs: % of incidents with completed post-incident reviews within 7 days (target: 95%).- Mean Time to Detect (MTTD) and Mean Time to Repair (MTTR) trends — aim to reduce MTTR by 30% year-over-year.- Error-budget policy adherence: % of releases blocked due to error-budget burn (track false positives/negatives).- Action closure rate: % of PI (post-incident) action items closed within target window.- Business impact: downtime minutes per quarter and number of customer-impacting incidents.

Rollout plan:1. Pilot with 4 teams (6–8 weeks): refine templates, dashboards, and gating.2. Train: 1-day hands-on SLO/runbook workshop + office hours for next 12 weeks.3. Automate: integrate SLO checks into CI/CD and release gates.4. Governance: quarterly audits by Reliability Guild + iterative improvements.

Why this works:- Combines top-down governance with bottom-up ownership, practical tooling, measurable incentives, and clear metrics so adoption is visible, enforced, and improvable.

HardSystem Design

91 practiced

Design an SLO-based release gating system that can scale across hundreds of services. Describe the architecture (centralized vs decentralized), how SLIs are ingested and validated, enforcement mechanisms (CI/CD pre-deploy checks, automated gating), handling of flaky metrics and partial outages, and how teams are onboarded or may opt-in/opt-out.

Sample Answer

Requirements:- Functional: block or flag releases when SLO breaches or insufficient data; support pre-deploy CI checks and automated gating in pipelines; provide per-service SLOs/SLIs and global policy.- Non-functional: support hundreds of services, low-latency gating decisions, secure multi-tenant access, auditability, fault-tolerant metrics ingestion.

High-level architecture (hybrid): decentralized SLI collection + centralized policy/enforcement.- Decentralized agents (sidecar/daemon) collect and pre-aggregate SLIs (latency, error-rate, saturation) and export to a centralized SLI store (time-series DB/Kafka + TSDB).- Central SLO Engine holds SLO definitions, rolling-window calculations, error-budget state, and gating policy. Exposes APIs and webhook integrations for CI/CD.- CI/CD plugins call SLO Engine pre-deploy (blocking or advisory). Orchestrators (Argo/Jenkins) can subscribe to automated gating webhooks.ASCII:[Service] -> [SLI Agent] -> Kafka -> [Metrics Pipeline] -> TSDB -> [SLO Engine] -> CI/CD

SLI ingestion & validation:- Agents send typed SLI events with schema, service_id, provenance, timestamp, and sample_count.- Ingestion pipeline enforces schema, time-sync checks, and cardinality limits. Validate freshness and sampling rate; run lightweight anomaly detection to flag outliers.- Store raw and pre-aggregated windows; compute rolling-window SLOs deterministically.

Enforcement mechanisms:- CI pre-deploy: CI plugin queries SLO Engine for current error budget and recent trend; returns PASS/WARN/BLOCK. Plugin can fail builds or require manual approval.- Automated gating: SLO Engine emits webhook or orchestration API call to block rollout (can integrate with feature flagging for phased rollouts).- Safe defaults: soft-gate (warn) on first breaches, hard-gate if breach persists or critical SLO.

Handling flaky metrics & partial outages:- Use multi-window analysis (5m, 1h, 7d) and statistical smoothing (EWMA) to avoid transient spikes causing blocks.- Require sustained breach criteria: e.g., breach > X minutes AND >Y% deviation, or consecutive windows.- Support multi-region redundancy and compare cross-region SLIs; isolate partial outages by tagging host/zone and only affect services impacted.- Provide manual override with audit trail and temporary "maintenance mode" to pause gating during known incidents.

Onboarding / opt-in / opt-out:- Default stance: opt-in with guardrails. Provide templates for SLOs and automatic SLI instrumentation libraries.- Self-service UI and CLI to register service, choose SLO template, tune thresholds. Central SRE reviews critical services or non-standard SLOs.- Opt-out allowed for experimental services with limits (timebox, risk acknowledgement) and requires review for production promotion.- Education: docs, runbooks, and periodic reviews; telemetry to track adoption and false-positives.

Scalability & trade-offs:- Decentralized ingestion reduces central load; Kafka+stream processors scale horizontally. Central engine stores compact state per SLO.- Trade-off: central engine single source of truth — make it highly available (multi-AZ, leader-election). Favor deterministic computation over ad-hoc queries to keep gating latency low.

Metrics for success: mean time to gate decision (<30s), false-positive rate <1%, adoption rate, reduced post-release incidents.

MediumTechnical

87 practiced

Your service is forecasted to grow 3x in traffic over the next six months. Outline a capacity planning approach that includes the telemetry you will monitor, the cadence of load testing, autoscaling policy decisions, cost mitigation (reserved vs on-demand), and contingency runbooks for sudden unexpected spikes.

Sample Answer

High-level approach: translate the 3x traffic forecast into resource needs using current telemetry + headroom targets, validate with progressive load tests, implement autoscaling tuned for SLOs, optimize cost with a mix of reserved capacity and on-demand/spot, and prepare runbooks for sudden spikes.

Telemetry to monitor (real-time + historical):- Business/traffic: RPS, concurrent users, request size, peak/minute patterns- Latency/error SLOs: p50/p95/p99 latency, error rate (5xx), SLA breaches- Resource: CPU, memory, network I/O, disk IOPS, socket counts, file descriptor usage- Platform: pod/container restarts, JVM heap/GC, thread pool saturation, DB connection pool usage, queue lengths- Cost/capacity: instance utilization, autoscaler activity, instance billing metrics

Load-testing cadence:- Baseline: weekly synthetic tests at current load to validate monitoring and alerting- Progressive: biweekly ramp tests at 1.5x, 2x, then 3x traffic — include burst/spike scenarios and long-duration soak tests- Pre-launch: full-scale rehearsals (3x sustained + sudden 2x burst) one month before expected growth milestones- Include chaos tests (network partitions, backend slowdowns) during non-production and limited production windows

Autoscaling policy decisions:- Scale on business signals (RPS or queue length) combined with resource signals (CPU, request latency)- Use target-based horizontal autoscaler: maintain p95 latency under SLO by scaling replicas; set min replicas to cover baseline plus safety buffer (e.g., 20%)- Cooldown and stabilization: short scale-up (10–30s) for burst detection, longer cooldown for scale-down (5–10 min) to avoid flapping- Vertical adjustments: use HPA + VPA in recommendation mode; schedule rolling instance type changes during low traffic- Rate-limit/inlet controls: implement ingress-level leaky-bucket to protect downstream

Cost mitigation:- Reserve steady-state baseline capacity with 1–3 year Reserved/Committed Instances or Savings Plans covering predicted minimum utilization- Keep remaining capacity on-demand and use spot instances for non-critical batch workers with graceful eviction handling- Rightsize instances using utilization percentiles from telemetry; use autoscaler to prefer smaller instances scaled horizontally- Monitor cost per RPS and set alerts when cost deviates >10% of forecast

Contingency runbooks for sudden spikes:- Detection: alert on sudden delta(RPS) >50% over 1 min, concurrent connection surge, or p95 latency breach- Immediate actions: 1. Enable emergency ingress throttling / degrade non-essential endpoints (feature flags) 2. Increase autoscaler caps and temporarily raise max replicas or spin up pre-warmed larger instances 3. Promote cached responses / enable read-only mode for downstream systems (DB) 4. Shift traffic via load balancer to healthy regions or traffic-split to DR cluster- Communication: notify on-call, product stakeholders, and downstream ops; publish customer-facing status with ETA- Post-incident: collect timeline, root-cause, update SLO/alert thresholds, and runbook improvements

Why this works: combines data-driven forecasting, progressive validation, conservative autoscaling tuned to latency SLOs, cost-efficient purchasing for steady baseline, and clear operational playbooks for resilience.

Unlock Full Question Bank

Get access to hundreds of Reliability and Operational Excellence interview questions and detailed answers.

Join thousands of developers preparing for their dream job.