Covers identifying system failure modes and designing resilient distributed systems, plus proactive resilience testing through controlled failure injection. Topics include common failure modes such as network partitions, increased latency, resource exhaustion, cascading failures, and data corruption; resilience design patterns like graceful degradation, retries with backoff, circuit breakers, bulkheads, timeouts, rate limiting, redundancy, and replication; and operational practices such as monitoring, distributed tracing, metrics and alerting to detect and diagnose failures. Also includes chaos engineering methodologies: defining steady state and hypotheses, designing safe experiments, controlling blast radius, tooling and frameworks, running game days, producing recovery runbooks and playbooks, handling test induced outages versus real incidents, and feeding lessons learned into postmortems and system improvements. Emphasis is on designing experiments that validate assumptions without causing uncontrolled production outages and on translating chaos results into concrete reliability improvements.
EasyTechnical
52 practiced
As a Solutions Architect preparing resilience tests, explain the differences between SLA, SLO, and SLI. Provide concrete guidance on selecting three SLIs for a customer-facing HTTP API (one latency, one availability, one correctness), propose example SLO targets, and explain how you would use these SLOs to decide whether a chaos experiment is safe to run.
Sample Answer
SLA vs SLO vs SLI — short definitions- SLA (Service Level Agreement): a formal contractual commitment to the customer; usually includes penalties if violated.- SLO (Service Level Objective): an internal measurable target for a service (e.g., availability 99.95% over 30 days) used to drive reliability decisions.- SLI (Service Level Indicator): the raw metric that measures service behavior (e.g., request latency p95, success rate). SLIs feed SLOs; SLOs inform SLAs/policy.Three concrete SLIs for a customer-facing HTTP API1. Latency SLI — p95 request latency for successful (2xx) responses measured at the API edge over a 1-minute scrape and aggregated over a 30-day window. Example measurement: p95_latency_ms.2. Availability SLI — ratio of successful HTTP responses (2xx) to total requests per minute, aggregated to rolling 30-day percentage. Example measurement: availability_pct = successful_requests / total_requests.3. Correctness SLI — percentage of requests returning correct payloads (schema + business validations). Implement by validating responses against a schema or checksum in production canaries; aggregated over 30 days. Example: correctness_pct.Example SLO targets (practical, customer-facing)- Latency: p95 < 200 ms over 30 days.- Availability: 99.95% successful responses over 30 days (≈ 22 minutes downtime/month).- Correctness: 99.99% correct responses over 30 days.Using SLOs to decide if a chaos experiment is safe1. Error budget concept: error_budget = 1 - SLO_target. For availability 99.95%, monthly error budget = 0.05% of requests/time.2. Pre-run checks: - Verify current burn rate and remaining error budget for each SLO over relevant windows (1h, 24h, 30d). - Require remaining error budget > safety threshold (e.g., >30–50%) for the SLOs the experiment may affect. - Ensure short-term burn rate is low (e.g., burn rate < 1 for past 24h). - Confirm monitoring, alerts, automated rollback, and runbooks are in place and tested. - Start with limited blast radius (canary region, small percentage of traffic).3. Decision rule example: - Proceed if: remaining error budget >50% AND short-term burn rate < 0.5 AND dependent SLIs (correctness, latency, availability) show stable baselines. - Abort or postpone if remaining error budget <30% OR burn rate rising rapidly (e.g., >2) OR correctness SLI already degraded.4. During/after experiment: - Continuously stream SLIs and compute live burn rate; halt if thresholds exceeded. - Postmortem: update SLOs, runbook, and experiment design based on results.Rationale- Latency, availability, and correctness cover user experience, reachability, and functional integrity.- Using error budgets and burn rate turns SLOs into operational guardrails for safe experimentation while protecting customer contractual expectations (SLAs).
MediumTechnical
58 practiced
Design the behavior and configuration for a circuit breaker used by a high-volume HTTP client in a microservice environment. Specify failure thresholds, rolling window duration, minimum request volume, half-open probe strategy, and how you would integrate metrics/tracing to observe its behavior. Describe how you'd tune parameters given a p95 latency target and the traffic profile.
Sample Answer
Requirements & goals:- Protect client from cascading failures, fail fast for downstream outages, and maintain p95 latency SLA under high QPS while minimizing false positives.Behavior & configuration (concrete starting point for a high-volume HTTP client):- Failure definition: count 5xx responses, connection/timeouts, and request latency > p95_target (treat as failure for latency-sensitive flows).- Rolling window: 60 seconds, implemented as 12 buckets of 5s (low memory, fine granularity).- Minimum request volume: 100 requests in rolling window (avoids tripping on low-volume noise).- Failure threshold to open: >= 50% failures over rolling window AND absolute failures >= 20 (so both relative and absolute).- Open state duration (cooldown): 30 seconds initially.- Half-open (probe) strategy: allow a small controlled sample of requests through: - Probe size: min(10, 1% of current concurrency) parallel requests, spaced by token-leaky rate (e.g., 1 probe every 500ms) to avoid avalanche. - Require N_success = 8/10 successive successes to transition closed; any failure returns to open and doubles cooldown (exponential backoff up to max 5 minutes).- Slow-start: after closed, gradually ramp allowed concurrency for that service (e.g., 10% increments per minute) to avoid immediate overload.Instrumentation & observability:- Metrics: - Counters & rates: total requests, successes, failures (by type), timeouts, latency histograms (high-resolution, e.g., HDR or Prometheus summaries for p50/p95/p99). - Circuit state gauge: OPEN/HALF-OPEN/CLOSED per downstream service/endpoint. - Probe metrics: probe success/failure counts and latency. - Emit events when state changes (with timestamps and reason).- Tracing: - Tag spans with circuit-breaker state and decision (e.g., "cb_decision=open", "cb_probe=true") so traces show whether requests were blocked or allowed probes. - Sample probes and failing requests more aggressively.- Dashboards & alerts: - Dashboard: p95 latency, failure rate, circuit state, request volume over time per downstream. - Alerts: circuit enters OPEN for >1 min, knockout count > X, p95 > target for 5m, or repeated half-open failures.- Logs: structured log when breaker trips, include window stats and recent errors.Tuning approach given a p95 latency target and traffic profile:1. Establish baseline: measure normal failure rates and latency distribution at current load.2. Set p95_target as latency threshold included in failure definition only for latency-sensitive endpoints.3. For high QPS, increase minimum volume (e.g., 500) and shorten bucket width (more buckets) to get faster reaction without noise.4. If backend is bursty: shorten rolling window to 30s for faster reaction; if steady but occasionally noisy, lengthen to 120s to avoid flipping.5. Adjust failure threshold based on cost of false-open vs false-closed: - If false-open is very costly (affects SLAs), raise threshold to 60–70% and increase min volume. - If protecting system is paramount, lower to 30–40% and reduce cooldown.6. Tune half-open probe aggressiveness by observing probe success rate: if probes frequently fail, reduce probe rate and increase cooldown/backoff.7. Use canary/ramp experiments: start with conservative settings in production for a single downstream, analyze metrics/traces for 24–72h, then iterate.Trade-offs & rationale:- 60s window + 50% threshold balances responsiveness and noise tolerance for high-volume systems.- Minimum volume prevents spurious trips on low traffic.- Controlled probes and exponential backoff prevent repeated thundering restarts of a degraded downstream.- Integrating latency into failure criteria ensures the breaker protects the p95 SLA, not just error counts.Operational practices:- Tie circuit config to service-level config (per endpoint) and make adjustable at runtime.- Run chaos tests and failure-injection to validate breaker behavior.- Maintain runbooks: when a breaker trips, how to diagnose (error types, recent deploys, load spikes).This design gives a practical, observable, and tunable circuit breaker suitable for high-volume HTTP clients while focusing on meeting a p95 latency target.
MediumSystem Design
48 practiced
An order-processing microservice synchronously calls inventory, payment, and shipping microservices. During a traffic spike, inventory becomes slow and request latency increases across the pipeline, creating cascading failures. As a Solutions Architect propose a redesign using resilience patterns (bulkheads, timeouts, circuit breakers, async queues, backpressure, rate limiting). Provide a migration plan (incremental steps) that minimizes customer impact and describes how you'd validate each migration step.
Sample Answer
Requirements:- Keep user-facing latency low and availability high during spikes.- Prevent cascading failures when Inventory is slow.- Minimize customer impact; allow incremental rollout and validation.High-level redesign (architecture):- Front door API Gateway -> Order Service (orchestrator) -> async Order Queue -> Worker pool.- Worker interacts with downstream services via resilient clients implementing timeouts, circuit breakers, and bulkheads.- Inventory, Payment, Shipping each expose two paths: fast synchronous check (cache / read-only) and async fulfillment via events/queues.- Rate limiter at API Gateway and per-service; backpressure via queue size and consumer scaling.- Observability: metrics (latency, error rates), tracing, SLOs, dashboards, and automated alerts.Key resilience patterns applied:- Timeouts: short client-side timeouts for sync calls to avoid thread starvation.- Circuit Breakers: open when error/latency thresholds exceeded; fallback to async path or offer degraded functionality.- Bulkheads: separate thread pools/processes per downstream call to avoid resource contention.- Async Queues: place intent in durable queue (e.g., Kafka/Rabbit) for retries and smoothing spikes.- Backpressure & Rate Limiting: API-level rate limiting + queue length-based throttling; return 429 with Retry-After when overloaded.- Idempotency & Compensation: idempotent order operations and compensating workflows for partial failures.Migration plan (incremental, low-risk):1) Instrumentation baseline (0–1 week)- Add tracing, metrics, dashboards for latency/error per service and call path.Validation: Baseline metrics captured; alerting working.2) Add short client-side timeouts + bulkhead thread pools in Order Service (1–2 weeks, deploy to staging then canary)- No behavior changes, just fail fast.Validation: Reduced tail latency under load tests; simulated slow Inventory shows limited thread exhaustion.3) Deploy circuit breakers around Inventory calls (1 week)- Configure to open on latency/error thresholds and to fallback to a cached synchronous response or trigger async path.Validation: Fault-injection tests (chaos) cause breakers to open; Gateway continues to respond with graceful degradation.4) Introduce read-cache for Inventory (cache invalidation strategy) and a fast read path (2 weeks)- Reduces sync dependency; use Redis or CDN for SKU availability.Validation: Cache hit rate increase; reduction in Inventory call rate and end-to-end latency.5) Add async Order Queue + worker processors (3–4 weeks, feature-flagged)- New orders placed on queue; immediate response returns order-accepted (or accepted-pending) to customer while background workers call downstream services with retries and circuit breakers.Validation: Canary traffic routed to queue; end-to-end correctness tests; confirm eventual consistency and customer notifications.6) Implement backpressure and rate-limiting at Gateway and queue (1 week)- Enforce per-tenant and global limits; when queue > threshold return 429 or degrade features.Validation: Load tests with spike reproduction; system gracefully rejects excess requests with correct headers.7) Move Payment & Shipping to async/dual-path (2–3 weeks)- Similar pattern as Inventory; ensure idempotency and compensating transactions.Validation: End-to-end business tests, monitoring of order completion rates, SLOs met.8) Clean-up & optimization (ongoing)- Tune thresholds, autoscaling policies, and SLAs; remove temporary fallbacks once stable.Validation approach per step:- Automated integration & chaos tests (simulate slow/failed downstream).- Canary deployments with traffic shadowing and real-user monitoring.- Pre- and post-change KPIs: p50/p95/p99 latency, error rate, throughput, queue depth, order success rate, customer-visible errors.- Rollback plan for every change and communication plan for customer-facing degradations.Trade-offs:- Async improves resilience and throughput but increases complexity and eventual consistency. Use clear UX messaging (order pending).- Circuit breakers + cache trade accuracy for availability—acceptable if business can tolerate short inconsistencies.This phased approach isolates risk, validates each resilience pattern, and moves the system from brittle synchronous chaining to a robust, observable, and scalable architecture.
MediumTechnical
46 practiced
Create a vendor evaluation checklist to help a client choose between Gremlin, Chaos Toolkit, AWS Fault Injection Service (FIS), and Litmus. Include criteria such as integration with CI/CD, Kubernetes support, safety controls, RBAC/audit, supported failure-injection types, observability integrations, cost model, and enterprise support. For each criterion provide scoring guidance and one sample enterprise requirement.
Sample Answer
Vendor Evaluation Checklist — Gremlin | Chaos Toolkit | AWS FIS | LitmusScoring: 0 = none, 1 = limited/manual, 2 = good/partial automation, 3 = best-in-class/full automation & enterprise features.1) Integration with CI/CD- Scoring guidance: evaluate native plugins, CLI automation, API, templates, and sample pipelines for Jenkins/GitHub Actions/GitLab/ArgoCD.- Sample enterprise requirement: "Must support automated experiment runs in CI pipelines with pass/fail gating and REST API triggers."2) Kubernetes support- Scoring guidance: evaluate native k8s primitives, CRDs/operators, namespace/label targeting, Helm/OCI charts.- Sample enterprise requirement: "Run scoped experiments via CRD/operator with safe defaults and support for multi-cluster (EKS/AKS/GKE)."3) Safety controls (blast radius, approvals, rollback)- Scoring guidance: automatic blast-radius limits, scheduled windows, canary targeting, abort/rollback APIs, pre-checks.- Sample enterprise requirement: "Experiments must enforce blast-radius policies, require approver workflow, and support automatic rollback on health-check failures."4) RBAC & Audit- Scoring guidance: fine-grained RBAC, SSO (SAML/OIDC), audit logs with export, immutable activity trail.- Sample enterprise requirement: "Integrate with SSO and send detailed experiment/audit logs to SIEM (Splunk/CloudWatch) for 7+ years retention."5) Supported failure-injection types- Scoring guidance: network, CPU/memory, disk I/O, process kill, latency, chaos at infra/cloud API, service-level faults; extensibility for custom probes.- Sample enterprise requirement: "Must support network partition/latency, pod/process kills, disk fill, and custom scripts/plug-ins."6) Observability integrations- Scoring guidance: native connectors for Prometheus, Grafana, Datadog, New Relic, OpenTelemetry, alerting hooks; automatic metric dashboards.- Sample enterprise requirement: "Automatically generate Grafana dashboards and correlate experiment timeline with traces and metrics."7) Cost model- Scoring guidance: licensing (per-agent, per-node, per-seat), SaaS vs self-host TCO, hidden infra costs, enterprise discounts, free/open-source options.- Sample enterprise requirement: "Total cost must be predictable and ≤ X% of SRE budget including support and multi-region agent costs."8) Enterprise support & SLAs- Scoring guidance: dedicated CSM/Technical account, 24x7 P1 response, runbooks, professional services, training, vetted security/compliance docs.- Sample enterprise requirement: "24x7 enterprise support with P1 <1 hour, quarterly health reviews, and SOC2/ISO compliance documentation."How to use: score each vendor by criterion (0–3), weight by customer priorities (e.g., k8s 25%, safety 20%), sum for comparison. Include hands-on proof-of-concept covering representative failure types and pipeline integration before procurement.
MediumTechnical
49 practiced
A prospective client is skeptical about investing in a chaos engineering program. As a Solutions Architect, prepare an outline of key business and technical KPIs (3–5 each) you would present to justify the program and explain how you'd estimate quantifiable benefits and compare them to costs over a 12-month period. Include examples of typical benefit assumptions and how you'd validate them post-deployment.
Sample Answer
Business KPIs (3–5):- Mean Time to Detect (MTTD) reduction — % decrease in time to surface incidents.- Mean Time to Recover (MTTR) reduction — % faster recovery from failures.- Production incident frequency — reduction in sev1/sev2 incidents per quarter.- Customer-facing metrics — reduction in SLA breaches, increase in uptime (%).- Cost of incidents — $ saved from fewer outages (business impact per hour × hours avoided).Technical KPIs (3–5):- Failure injection coverage — % of critical services/scenarios exercised by chaos tests.- Recovery automation rate — % of incidents mitigated automatically vs manual.- Time-to-remediation playbook effectiveness — average time for runbook success.- Regression rate — % of new releases that pass chaos checks before deploy.- Observability completeness — % of services with end-to-end monitoring and SLOs.Estimating quantifiable benefits vs costs (12 months):1. Baseline: measure current MTTD, MTTR, incident count, average outage cost/hour, and deployment cadence for prior 6–12 months.2. Assumptions: e.g., chaos program reduces MTTR by 30%, incidents by 25%, automates 40% of common remediations.3. Translate to dollars: annual incident-hours avoided = incident_count × incident_duration × reduction%. Cost savings = incident-hours avoided × cost/hour. Add productivity gains from reduced firefighting (engineer-hours saved × fully-burdened rate).4. Costs: initial setup (tools, integration, workshops), recurring (licenses, engineer time for tests, runbook automation), training and governance.5. ROI calc: (Total annual benefits − Total annual costs) / Total annual costs; present payback period.Example assumptions and validation:- Assumption: MTTR drops 30% within 6 months. Validate: compare MTTR rolling averages pre/post 3,6,12 months; run significance tests.- Assumption: 40% of incidents automated. Validate: track remediation automation count vs manual and measure time delta.- Assumption: 25% fewer production incidents. Validate: track incident frequency by type and correlate with coverage of chaos experiments.Post-deployment validation & risk controls:- Set short-term milestones (30/90/180 days) with measurable targets.- Run A/B or phased rollout (pilot teams) to compare baselines.- Use dashboards (SLOs, incident costs, experiment coverage) and monthly reviews to refine assumptions and adjust scope.This approach ties technical improvements to clear business dollars and gives a quantifiable, auditable path to justify the chaos program.
Unlock Full Question Bank
Get access to hundreds of Resilience and Chaos Engineering interview questions and detailed answers.
