Site Reliability Engineering Questions

Covers the principles, practices, and operational responsibilities for keeping production systems reliable while enabling ongoing feature delivery. Topics include balancing reliability and development velocity; incident response, on call operations, and incident management including post incident learning; observability, monitoring, and alerting; automation and tooling to reduce manual toil; infrastructure and platform design; application reliability engineering; capacity planning, performance, and scaling; service level indicators, service level objectives, and error budget management; reliability engineering culture and interactions with product and development teams; and the metrics used to measure reliability and performance such as availability, latency, error rate, throughput, saturation, time to detection, and time to recovery. Candidates should be able to describe typical operational tasks at entry and senior levels, reason about trade offs, propose architectural or operational improvements, and justify decisions with metrics; senior engineers are expected to show deep expertise in some areas and demonstrable competence across many domains.

HardSystem Design

61 practiced

Design an automated self-healing system for a container orchestration platform that detects unhealthy services (CPU or memory spikes, failing health checks, memory leaks), triggers safe remediation actions (restart, scale, replace, or rollback), and escalates to humans when thresholds are exceeded. Include risk controls such as automation circuit breakers, rate limits on automation, and detailed audit trails for every automated action.

Sample Answer

Requirements:- Detect unhealthy services by metrics (CPU, memory, GC, OOM), health checks, error rates, latency; support custom detectors and anomaly detection.- Remediate automatically with safe actions: restart pod, scale replica count, replace node, rollout-rollback, or run diagnostic job.- Escalate to humans when thresholds, retries, or blast-radius limits exceeded.- Provide safety controls: automation circuit breakers, rate limits, canary-first rollouts, and complete tamper-proof audit trails.

High-level architecture:Monitoring → Analyzer (rules + ML) → Decision Engine (policy + safety) → Orchestrator (executes on cluster) → Audit & Escalation → Dashboard/Runbooks

Components and responsibilities:1. Monitoring: Prometheus, cAdvisor, health endpoints, distributed tracing. Push and pull metrics into TSDB.2. Analyzer: Rule engine (SLO/SLA rules, thresholds, moving averages) + optional anomaly detector for nonstationary patterns.3. Decision Engine: - Policy store (per-service remediation policies): allowed actions, max retries, cool-down, blast-radius, required approvals. - Safety module: automation circuit breaker (global & per-service), rate limiter (ops/min), dependency-aware impact calculator. - Action planner: chooses minimal-impact action (e.g., restart one pod, scale up by 1) and simulates expected outcome.4. Orchestrator: Uses Kubernetes API to perform actions atomically with leader election and idempotency.5. Audit & Observability: Immutable event log (append-only, signed entries), action metadata, before/after snapshots, metrics on automation efficacy.6. Escalation: PagerDuty/Slack integration with contextual diagnostics, rollback knobs, and manual approval workflows.

Data flow:1. Metric anomaly -> Analyzer raises incident.2. Decision Engine evaluates policy, checks circuit breakers/rate limits, computes safe remediation plan.3. Orchestrator executes action with pre/post health checks and rollback timers.4. If healthy, event closed; if not, retry per policy or escalate.

Risk controls:- Automation circuit breakers: trip on rate of failed remediations or injected chaos tests; require human approval to reset.- Rate limits: per-service and global caps, exponential backoff and cool-down windows.- Canary-first actions: apply to single pod/zone before wide rollout.- Dependency & blast-radius checks: deny actions that would violate SLOs or reduce capacity below safety thresholds.- Immutable audit trails: write-ahead log (WAL) stored in secure object store, signed and versioned; accessible for forensic and compliance.- Simulation/safety gates: dry-run mode and predictive simulation using historical data.

Operational practices:- Define SLOs and error budgets per service; tie automation aggressiveness to remaining error budget.- Regular chaos engineering to validate remediations and ensure circuit breakers are functional.- Post-incident review recording automated decisions and tuning policies.

Scalability & trade-offs:- Stateful decision engine can be sharded by namespace/team. ML detection adds sensitivity but increases false positives—mitigate via human-in-the-loop for low-confidence cases.- Conservative defaults favor minimal actions (restart single pod) to reduce risk; allow aggressive policies for low-risk services.

This design balances automated fast recovery with strong safety, auditability, and human escalation when needed.

HardTechnical

64 practiced

You're facilitating a blameless postmortem where the root cause spans multiple teams and one team is reluctant to accept responsibility. Describe how you would run the postmortem to surface systemic issues, ensure corrective actions are assigned without finger-pointing, document accountable owners and deadlines, and secure executive alignment on cross-team fixes.

Sample Answer

Situation: I run a blameless postmortem for an outage where the root cause spans multiple teams and one team resists owning part of the problem.

Task: My goal is to surface systemic causes, produce non‑blaming corrective actions with clear owners and deadlines, and get executive buy‑in for cross‑team fixes.

Action:- Prework: gather timeline, logs, alerts, runbooks and invite representatives (SRE, App, Infra, PM, Product) plus an executive sponsor. Share data and a neutral agenda in advance.- Set norms at the start: blameless language, focus on systems & decisions, not people; the purpose is learning and prevention.- Build a factual timeline together (timeline review forces alignment on “what happened when”).- Use structured RCA (fishbone + 5 whys) to map contributing factors across teams (e.g., missing alert thresholds in monitoring, unsafe deployment process, unclear API SLAs).- When a team resists, reframe ownership as “who can change this contributing factor?” — assign by capability not blame. Use “action owner” (role or person) and “stakeholder” (affected teams).- Convert each contributing factor into SMART action items: what, owner, due date, success metric. Record these in a shared tracker (JIRA/Confluence) with links to PRs/runbook updates.- For cross-team work, propose a RACI for each action and request a short escalation path if conflicts reappear.

Result / Exec alignment:- Present a one‑page remediation plan to execs showing risks, cost of delay, owners, deadlines, and measurable outcomes (reduced MTTR, alert coverage %). Ask execs to commit to removing organizational blockers (e.g., dedicating sprint capacity or resolving priority conflicts).- Schedule a follow‑up review in 2–4 weeks to validate progress against metrics; keep exec sponsor copied on updates.

This approach surfaces systemic issues, avoids finger‑pointing by assigning ownership by capability, documents accountable owners and deadlines, and secures executive alignment through a concise risk‑and‑remediation plan and follow‑up cadence.

EasyTechnical

61 practiced

Create a concise runbook/playbook template for responding to a common production problem such as 503 Service Unavailable errors on a web service. The template should include detection criteria, immediate mitigation steps, rollback or quick-fix steps, root-cause investigation checkpoints, stakeholder communication templates, and how to capture post-incident actions.

Sample Answer

Runbook: 503 Service Unavailable — Web Service

Detection criteria- Alert: >X% 503 rate over Y minutes (e.g., ≥5% of requests in 5m) OR traffic spike + elevated latency- Key metrics: 5xx rate, error budget burn, avg latency, backend health checks, pod/container restarts, CPU/memory, queue depths, DB connection errors- Sources: Prometheus/Grafana, ELK, CDN/Load Balancer logs, Synthetic checks

Immediate mitigation (first 10–15 min)1. Triage: Confirm alert, capture timestamps, severity, impact (users affected, regions).2. Traffic control: - Enable circuit-breaker or degrade non-essential features. - Reduce traffic via WAF/Rate-limit or scale out LB if capacity-bound.3. Failover: - Route traffic to healthy region/replica if available.4. Restart/Isolate: - Restart unhealthy pods/instances; cordon problematic nodes.5. Short-term rollback: - If incident follows deployment, trigger fast rollback to previous version.

Rollback / quick-fix steps- Verify last deploys (CI/CD): rollback to known-good artifact with automated job.- Revert config flags/feature toggles.- If DB or cache overloaded: throttle writes, increase pool sizes or add read-replicas.- Apply emergency patch only if low-risk and tested in staging.

Root-cause checkpoints (investigation)- Timeline: correlate deploys, config changes, infra events.- Logs: error stack traces, upstream 502/504s, auth failures.- Resource: CPU/memory, GC pauses, file descriptors, connection pool exhaustion.- Dependencies: third-party API, DB, cache, service mesh.- Network: packet loss, DNS resolution, TLS handshake errors.- Reproduce: run load test against canary if safe.

Stakeholder communication templates- Incident alert (to on-call/chat): "INCIDENT: 503 spike on service-X. Start: <time>. Impact: ~<percent>% requests failing, regions: <list>. Action: triage started. Owner: @<oncall>."- Update (every 15–30m): "UPDATE: <time>. Current status: <mitigations applied>. Impact reduced? <yes/no>. Next step: <action>. ETA: <xx>."- Resolution: "RESOLVED: <time>. Root cause (initial): <summary>. Mitigation: <what we did>. Follow-ups: <PRs/Tasks>. Postmortem: <link>."

Post-incident capture (postmortem checklist)- Collect timeline, metrics, logs, runbook steps executed.- Root cause analysis: 5 whys, contributing factors.- Action items: owner, priority, due date (e.g., improved alerts, automation for rollback, capacity changes).- Update runbook with lessons and test changes in staging.- Schedule review with dev and product stakeholders; link documented RCA and tracker.

Playbook notes- Keep playbook accessible, single-click runbook link in alert.- Practice tabletop drills quarterly; automate safe mitigation steps.

HardTechnical

72 practiced

Design how to instrument and enforce SLOs for critical third-party dependencies (like a payment gateway or auth provider) whose SLA varies. Include synthetic probing, in-band telemetry to measure dependency latency and error rates, fallback and circuit breaker strategies, and how you would account third-party reliability in your own SLOs and error budgets.

Sample Answer

Requirements & goals:- Measure availability/latency/error-rate of critical third-party calls (payment, auth).- Detect regressions before customers notice, provide graceful degradation, and account third-party impact in our SLOs/error budget.

Instrumentation1. Synthetic probing (out-of-band):- Periodic scripted probes that exercise representative flows (auth token request, payment authorization) from multiple regions and networks at 1–5 min intervals.- Collect latency, HTTP status, response body correctness, and TLS/CA/redirect checks.- Tag probes by region, network, and time-of-day.

2. In-band telemetry (real traffic):- Emit per-request telemetry from service: dependency_latency_ms, dependency_status_code, dependency_error_type, business_impact (e.g., payment_hold).- Use distributed tracing to correlate user transaction spans with dependency spans and capture sample traces for high-latency/error requests.- Aggregate into histograms (p50/p95/p99), error-rate counters, and SLIs.

Example synthetic probe (simplified):

python

import requests, time
start=time.time()
r=requests.post("https://payments.example/authorize", json=payload, timeout=3)
lat=(time.time()-start)*1000
emit_metric("payment_probe_latency_ms", lat, tags={"region":"us-east"})
emit_metric("payment_probe_status", r.status_code)

Detection & Alerts- Define SLIs for latency (p95), success rate (HTTP 2xx and semantic success), and functional correctness.- Alert on synthetic degradation (e.g., probe success rate < 99% for 5m) and on in-band user-impacting errors (increase in user-facing error rate or sustained p95 latency increase).- Use anomaly detection on traces and dependency error spikes.

Resilience: Fallbacks & Circuit Breakers- Per-dependency circuit breaker with these states: CLOSED, OPEN, HALF-OPEN.- Tripping conditions: rolling-window error-rate threshold (e.g., >5% errors over 1m) or latency SLA breach.- Automatic backoff with exponential cool-down; allow a small rate of probes in HALF-OPEN to test recovery.- Granularity: circuit breakers per endpoint and per customer-tenant for noisy neighbors.- Fallback strategies ordered by business impact: 1. Local cached success (e.g., cached token / last-known-good settings) 2. Reduced-functionality path (e.g., queue payment for later capture with user-visible note) 3. User-visible error with clear remediation- Ensure fallbacks are safe (idempotent, reversible) and audited.

Accounting third-party reliability in our SLOs & error budgets- Split SLOs into two components: - End-to-end SLO (user-visible success & latency) - Dependency attribution SLO (fraction of errors attributable to third-party)- Maintain an "external dependency budget": classify errors as internal vs external using telemetry/traces. Example: If end-to-end availability SLO = 99.95% monthly, allow a portion (e.g., 0.03%) to be attributed to third-party failures; the rest must be internal.- If third-party consumes > X% of our error budget, trigger vendor escalation: open support ticket, failover to backup provider, enable mitigation runbook.- Negotiate contractual SLAs and map vendor SLA windows to our SLO tolerances; do not blindly accept vendor SLA as our SLO.

Operationalization & Runbooks- Automate remediation: circuit-breaker actions, automatic route to backup providers, and queuing.- Runbooks: steps for triage, vendor escalation, customer communications, rollback of new integrations.- Regularly review synthetic vs in-band divergence to detect monitoring gaps.- Post-incident: blameless postmortem, update SLO allocations, and add tests for uncovered failure modes.

Trade-offs & notes- Synthetic probes catch availability but may miss real-user network paths—always pair with in-band telemetry.- Conservative fallbacks preserve safety but may reduce functionality; align fallbacks with product/PM for acceptable UX.- Over-attributing to vendor may hide internal issues—ensure strict, auditable attribution logic (trace IDs, error codes, upstream latency contributions).

This design gives early detection through probes, precise attribution via tracing and metrics, automated protection via circuit breakers and fallbacks, and a governance model that accounts for third-party reliability in our SLOs and error budgets.

EasyTechnical

68 practiced

Explain SLI, SLO, and SLA in the context of a customer-facing web service. Define each term, describe who the primary consumers of each are (engineering, product, legal), give a concrete example SLI and a corresponding SLO, explain how SLA differs technically and contractually from SLO, and describe operational responses for violations.

Sample Answer

SLI (Service Level Indicator): a measurable metric that reflects user experience (e.g., request success rate, request latency p95, DB error rate). Primary consumers: SRE/engineering for monitoring, product for feature impact analysis.

SLO (Service Level Objective): a target or threshold on one or more SLIs over a time window (error budget). Primary consumers: SRE/engineering to drive operational behavior and alerting; product to set reliability expectations and prioritize work.

SLA (Service Level Agreement): a formal, often legal, contract with customers that specifies promised levels, remedies/penalties for breaches. Primary consumers: legal and product for negotiation; customers; SREs implement monitoring to prove compliance.

Concrete example:- SLI: fraction of successful HTTP GETs to /api/v1/orders that return 2xx within 300ms measured per minute.- SLO: 99.9% of those requests succeed within 300ms measured over a 30-day window (implies 43.2 minutes of allowable failure).

How SLA differs:- Technically an SLA maps to one or more SLOs/SLIs but is legally binding and specifies remediation (credits, termination). An SLO is an internal operational target used to manage error budget and prioritization. SLAs often require stricter evidence collection, audit logs, and defined reporting periods.

Operational responses for violations:- If SLO breach (approaching/exhausting error budget): throttle feature launches, halt risky deploys, increase on-call staffing, run focused blameless postmortem, allocate engineering time to reliability work until error budget restored.- If SLA breach (legal): trigger executive and legal notifications, collect compliance-grade telemetry and incident timeline, engage customer success to communicate remediation/credits, perform root cause analysis, implement corrective actions and review contractual terms.

Best practice: instrument SLIs with high fidelity, automate SLO alerting (warning at 50% error budget, page at 80–100%), keep SLAs aligned to achievable SLOs and ensure monitoring meets evidentiary standards.

Unlock Full Question Bank

Get access to hundreds of Site Reliability Engineering interview questions and detailed answers.

Join thousands of developers preparing for their dream job.