Failure Detection and Automated Response Questions

How systems detect that something is wrong (health checks, external monitoring, client-side detection). Distinction between fast failure detection and robust failure detection. At Staff level, discuss sophisticated detection approaches (combination of signals to reduce false positives), automated response systems, and designing for graceful handling of detection errors.

EasyTechnical

70 practiced

What is SLO-based alerting and how do error budgets influence automated responses? As an SRE, when would you trigger automated rollback, restrict new deployments, or pause automation based on SLO breaches versus simply alerting the on-call team?

HardSystem Design

143 practiced

Design an error-budget-driven automation framework where predefined mitigations (e.g., disable noncritical features, reduce new deployments, throttle batch jobs) are executed automatically as a service's error budget burn rate crosses thresholds. Describe policy definition, approval workflow, safe execution, rollback, audit trails, and coordination across dependent services.

Sample Answer

Requirements & constraints:- Functional: automatically execute predefined mitigations when a service’s error-budget burn rate crosses thresholds (e.g., warn at 25%, action at 50%, strong action at 90%).- Non-functional: must be safe (avoid cascading failures), auditable, reversible, support approvals for production-impacting actions, and coordinate across dependent services. Low-latency detection, high-availability control plane, role-based access.

High-level architecture (text diagram):Metrics/Tracing → SLO Engine → Policy Engine → Approval Workflow + Orchestrator → Executors (K8s API, Feature-flag service, CI/CD) ← Service Catalog & Dependency GraphAll actions logged to Audit DB and Notifications.

Key components:1. SLO Engine: computes real-time burn rate from metrics (Prometheus/Thanos or Datadog). Emits events when thresholds crossed.2. Policy Definition Store: declarative YAML policies per service: - thresholds, ordered mitigations (id, action type, target, safety constraints), required approval level, cooldowns, preconditions. Example: threshold: 50% mitigations: - id: disable-noncritical-feature type: feature_flag target: featureX require_approval: team_lead safety: canary_percent=53. Policy Engine: validates policy, matches events, enforces preconditions (dependency checks) and escalation rules.4. Approval Workflow: integrates with IAM and ticketing (Jira/ServiceNow). For low-impact mitigations, auto-approve; for higher-impact, send to on-call + approver UI (Slack/Email + Web dashboard). Include timeouts and fallback.5. Orchestrator / Executor: executes mitigations safely using canaryed steps: - dry-run simulation (policy simulator) - staged execution (e.g., 5% -> 20% -> 100%) - health probes after each stage; roll forward or rollback Executors use K8s APIs (scale deployments), feature flag SDKs, CI/CD pipelines to block new releases.6. Dependency & Coordination: Service Catalog + dependency graph ensures actions consider downstream/upstream effects. Orchestrator consults graph, simulates impact, optionally applies coordinated mitigations (e.g., throttle batch jobs and pause nonessential downstream tasks).7. Rollback & Safety: automatic rollback triggers on health-probe regressions or increased error-rate after mitigation; manual rollback via UI; immutable runbooks and kill-switch to abort all automation.8. Audit & Observability: append-only Audit DB (immutable logs), signed action records, ability to replay actions; dashboard for timeline, who/what/why; integrate with SIEM for compliance.9. Security & Governance: RBAC, policy signing, change approval windows, rate limits on automated actions.10. Testing & Validation: staging environment, chaos engineering exercises, policy linting, policy unit tests and canary pipelines.

Trade-offs:- Automation reduces MTTR but risks bad mitigations — mitigated via staged canaries and approval gates.- Complexity of dependency coordination vs speed — favor safety: allow safe-mode (auto small mitigations) and manual for disruptive actions.

Example flow:Prometheus detects 10x burn rate → SLO Engine emits event → Policy Engine picks mitigation: “reduce deploy frequency” (auto) + “disable noncritical feature” (requires approver). Orchestrator throttles CI jobs, executes feature flag canary at 5%, monitors 10 minutes; success → ramp to 100%. All steps logged; if downstream latency increases, orchestrator rolls back and notifies on-call.

This design balances safety, auditability, and speed — enabling SREs to automate error-budget responses while preserving control and traceability.

EasyTechnical

89 practiced

Compare active (synthetic/blackbox) monitoring vs passive (instrumentation/whitebox) monitoring. Provide concrete examples of signals each provides, strengths and weaknesses, and when you'd choose one over the other for detecting failures across a global e-commerce stack (CDN, API gateway, backend services). Discuss cadence, cost, and blind spots.

Sample Answer

Active (synthetic/blackbox) vs Passive (instrumentation/whitebox) monitoring

Quick definition- Active (blackbox): external probes simulate user traffic from many locations (HTTP checks, synthetic transactions, RUM is a hybrid). Treats system as a black box.- Passive (whitebox): collects internal telemetry (metrics, logs, traces, health endpoints, profilers) from services themselves.

Concrete signals- Active: global HTTP status codes, end-to-end latency, DNS resolution time, TLS handshake time, CDN cache hit/miss as seen by a probe, payment checkout flow success rates from probes.- Passive: per-service CPU/memory, internal error counters, request latencies by span, DB connection pool saturation, application logs with stack traces, deployment/version tags.

Strengths & weaknesses- Active strengths: detects customer-visible outages (global reach), validates routing/edge config, catches problems even when telemetry pipeline is down. Weakness: limited internal visibility, synthetic load may not hit all code paths, cost grows with cadence × locations.- Passive strengths: rich root-cause data, fine-grained SLOs, resource and error context. Weakness: blind to CDN/DNS/ISP issues and to telemetry pipeline failures; requires instrumentation and access.

When to choose (global e‑commerce stack)- CDN: active probes from multiple ISPs/regions to detect cache misconfig, TLS/DNS problems, or POP-level outages.- API gateway: combine both—active health checks for availability and end-to-end latency; passive metrics/traces for routing/middleware errors and per-backend error rates.- Backend services: prioritize passive (metrics, traces, logs) to diagnose CPU spikes, DB latency, or code exceptions; add synthetic tests for critical user journeys (checkout) that exercise backend integration.

Cadence, cost, blind spots- Cadence: critical paths (checkout, login) run synthetic checks every 15–60s globally; less critical every 5–15m. Passive is continuous but watch cardinality of metrics/logs to control cost.- Cost: active cost = probes × locations × frequency (can be substantial at high cadence globally). Passive cost = ingest volume (high-cardinality traces/logs).- Blind spots: active misses internal state and low-volume edge-cases; passive misses network/DNS/CDN reachability and failures of the monitoring pipeline itself.

Recommended strategy- Use both: deploy global synthetic probes for customer-facing availability and edge issues; rely on rich instrumentation for root-cause analysis and SLOs. Alerting: trigger high-level page- or checkout-service pager from active outage; attach passive-context playbooks that surface traces/metrics automatically when paging.

HardTechnical

67 practiced

How do you measure and ensure the reliability of your failure-detection and automated-response systems themselves? Define SLOs for monitoring systems, which key metrics to track (e.g., detection MTTD, false positive rate), and approaches to test and validate detection engines and remediations without risking production stability.

Sample Answer

Start by treating your monitoring and automated-response (A/R) system as a product that needs its own SLOs, observability, and testing lifecycle. Define clear SLOs, instrument everything, and use safe test patterns to validate behavior without destabilizing production.

SLOs (examples)- Monitoring availability: alerting/telemetry pipeline uptime >= 99.99% (monthly).- Data freshness/latency: metrics/logs/traces ingestion latency < 30s 99.9%.- Detection MTTD: mean time to detect incidents <= 60s (P99 requirement configurable per service).- False positive rate (FPR): alerts that are actionable <= 5% over a 30d window.- Remediation success rate: automated remediation completes successfully without human rollback >= 99% (P90).- MTTR for automated actions (time from detection to resolution) <= X minutes per service.- Alert fatigue: mean alerts per on-call per week <= threshold; paging rate SLO.

Key metrics to track- MTTD (mean & P99), MTTR, and remediation success/failure counts.- FPR and false negative rate (FNR) where detectability permits.- Precision/recall of detectors (for ML-based rules).- Time-series completeness and ingestion lag.- Alert-to-acknowledge time, paging noise, duplicate alerts.- Rate of automated-runbook rollbacks / safety-trigger invocations.- Canary failure rates and rollback counts.

Validation & testing approaches (safe, production-aware)- Shadow / passive mode: run detectors and remediations in read-only mode; record decisions and predicted outcomes without taking actions. Compare against ground truth and historical incidents.- Staging & replay: replay production traffic/logs/metrics into staging to validate detectors and remediation logic against real scenarios.- Canary automation: enable automation for a small subset (one cluster/region/percentage of traffic) with strict safety gates, metrics checks and automated rollback.- Chaos & fault-injection in controlled windows: introduce failures in canary/staging and gradually in production with guardrails and blast-radius limits.- Synthetic tests & health checks: inject synthetic errors to validate detection and end-to-end remediation path (heartbeat tests).- Game days/postmortems: run scheduled tabletop and live drills to validate human+automation interaction and runbooks.- A/B or shadow remediations: have automation propose a remediation and log what it would do; operators approve before actual run for a period to build trust.- Gradual escalation & circuit breakers: automated responses should start conservative (throttled) and escalate only if metrics confirm correctness. Include kill switch & manual override.- Replay historical incidents: verify detectors trigger and remediations would have succeeded on past incidents.

Instrument & iterate- Log every decision: detector input, rule evaluation, confidence, chosen action, outcome, timing. Use this to compute metrics and run ML model calibration.- Continuous feedback: run periodic precision/recall analysis, tune thresholds, and update SLOs based on business impact.- Safety practices: built-in cooldowns, rate limits, idempotent remediations, canary rollbacks, role-based approvals for high-risk actions.- Governance: deploy change review, automated tests for rule logic, test coverage for playbooks and remediation scripts.

Trade-offs- Tight thresholds reduce MTTD but may increase FPR; balance via error budget and business impact.- Automation aggressiveness should match confidence and blast-radius tolerance.

This approach ensures monitoring and A/R systems are measurable, testable, and evolve safely while minimizing production risk.

MediumTechnical

92 practiced

Given these tables:

slos(service text PRIMARY KEY, allowed_error_rate numeric)service_requests(request_id uuid, service text, outcome text CHECK (outcome in ('ok','error')), timestamp timestamptz)

Write a Postgres SQL query that computes for each service the errors, total requests, burn_rate = (errors/total) / allowed_error_rate, and error_budget_remaining_pct = max(0, 1 - burn_rate) over the past 7 days. Return service, errors, total, burn_rate, error_budget_remaining_pct.

Sample Answer

Approach: aggregate requests for the last 7 days per service, join SLOs to get allowed_error_rate, compute burn_rate = (errors/total)/allowed_error_rate with safe guards for zero totals or zero/NULL allowed rates, and compute error_budget_remaining_pct = max(0, 1 - burn_rate).

sql

WITH reqs_7d AS (
  SELECT
    service,
    COUNT(*) FILTER (WHERE outcome = 'error') AS errors,
    COUNT(*) AS total
  FROM service_requests
  WHERE timestamp >= now() - interval '7 days'
  GROUP BY service
)
SELECT
  s.service,
  COALESCE(r.errors, 0)         AS errors,
  COALESCE(r.total, 0)          AS total,
  -- safe burn_rate: NULL when allowed_error_rate IS NULL or <= 0; 0 when total = 0
  CASE
    WHEN COALESCE(r.total,0) = 0 THEN 0
    WHEN s.allowed_error_rate IS NULL OR s.allowed_error_rate <= 0 THEN NULL
    ELSE (r.errors::numeric / r.total) / s.allowed_error_rate
  END AS burn_rate,
  -- error budget remaining pct: if burn_rate is NULL -> NULL, else max(0, 1 - burn_rate)
  CASE
    WHEN COALESCE(r.total,0) = 0 THEN 1.0
    WHEN s.allowed_error_rate IS NULL OR s.allowed_error_rate <= 0 THEN NULL
    ELSE GREATEST(0.0, 1 - ((r.errors::numeric / r.total) / s.allowed_error_rate))
  END AS error_budget_remaining_pct
FROM slos s
LEFT JOIN reqs_7d r ON r.service = s.service
ORDER BY s.service;

Key points:- Uses FILTER counts for clarity and single-scan aggregation.- Handles zero total requests (treats burn_rate=0 and budget remaining 100%) and invalid allowed_error_rate (returns NULL).- You can round results with ROUND(..., 4) if desired.- For large tables add an index on service_requests(timestamp) and (service, timestamp) to speed the 7-day filter.

Unlock Full Question Bank

Get access to hundreds of Failure Detection and Automated Response interview questions and detailed answers.

Join thousands of developers preparing for their dream job.