Resilience and Chaos Engineering Questions

Covers identifying system failure modes and designing resilient distributed systems, plus proactive resilience testing through controlled failure injection. Topics include common failure modes such as network partitions, increased latency, resource exhaustion, cascading failures, and data corruption; resilience design patterns like graceful degradation, retries with backoff, circuit breakers, bulkheads, timeouts, rate limiting, redundancy, and replication; and operational practices such as monitoring, distributed tracing, metrics and alerting to detect and diagnose failures. Also includes chaos engineering methodologies: defining steady state and hypotheses, designing safe experiments, controlling blast radius, tooling and frameworks, running game days, producing recovery runbooks and playbooks, handling test induced outages versus real incidents, and feeding lessons learned into postmortems and system improvements. Emphasis is on designing experiments that validate assumptions without causing uncontrolled production outages and on translating chaos results into concrete reliability improvements.

EasyTechnical

49 practiced

Define 'blast radius' in the context of chaos engineering. Provide three concrete examples of techniques to limit blast radius in production experiments, such as restricting to a percentage of traffic, a single namespace, or specific labels/tenants. Explain when each approach is preferable.

HardTechnical

60 practiced

You have a service dependency graph for a complex system. Describe how you would model and analyze the quantitative risk of cascading failures: include per-service failure probabilities, load amplification factors, and capacity constraints. Outline algorithmic steps or simulation approaches to identify critical services that lead to the largest downstream impact.

Sample Answer

Situation: We need a quantitative, reproducible model to find which services, when they fail, cause large downstream cascades given per-service failure probabilities, load amplification, and capacity limits.

Modeling:- Represent system as directed graph G=(V,E). Each node v has: base load Lv, capacity Cv, intrinsic failure probability pv (independent or time-varying), and amplification factor αuv on edge u→v meaning when u fails it adds αuv·Lu extra load to v (can be fractional or conditional).- Failure rule: node fails if incoming load > Cv or via intrinsic failure.

Analysis approach — two complementary methods:

1) Deterministic propagation (fast approximate):- For a given initial failed set S0, propagate in topological/level order: - For each failed u, for each outgoing edge u→v: Lv := Lv + αuv·Lu. - If Lv > Cv and v not yet failed, mark v failed and continue.- Use this to compute deterministic downstream reachability and final failed set F(S0). Complexity O(|V|+|E|) per S0.- Use for “what-if” single-node or small-set failure scenarios, computing lost capacity and service impact.

2) Probabilistic simulation (accurate):- Monte Carlo trials: each trial sample intrinsic failures by pv; pick initial failed seeds or allow spontaneous failures; then run deterministic propagation per trial. Record metrics: final failure count, total unavailable capacity, probability a target service fails.- Use importance sampling for rare events (bias pv upwards and weight outcomes) to estimate tail risks.- Use many trials to estimate expected impact E[impact | seed], conditional probabilities, and distribution tails.

Analytical/linearized approximation:- For small amplifications, approximate as branching process: expected secondary failures ≈ sum_u αuv·Pr[u fails] / Cv scaling — compute spectral radius of weighted adjacency to estimate whether cascades are supercritical.

Criticality ranking:- Compute marginal expected impact per-node: Delta(v) = E[impact | force-fail v] - E[impact | baseline].- Use deterministic reachability as quick proxy; refine with Monte Carlo estimates for top-k candidates.- Also compute centrality-weighted metrics: eigenvector centrality on amplification-weighted graph, and expected lost capacity = sum_w Cv·Pr[w fails | seed v].

Edge cases & validation:- Model correlated failures (shared infrastructure) by adding common-mode events in simulations.- Validate against historical incidents, tune α and pv via telemetry (latencies, retries, queue lengths).- Use sensitivity analysis to prioritize mitigations: capacity increases, rate-limiting (reduce α), redundancy (reduce pv).

Outcome metrics to report:- Probability of >X% system capacity loss, mean time to cascade, marginal risk per service, and recommended mitigations with cost-benefit.

This combined deterministic + Monte Carlo pipeline gives fast prioritization and rigorous probabilistic risk estimates for SRE decision-making.

MediumTechnical

61 practiced

An increase in end-to-end latency is observed. Using distributed tracing and metrics, outline a step-by-step triage process to determine whether the bottleneck is in the network, a downstream microservice, or an external third-party API. Mention specific tracing signals, span analysis, histogram aggregation, and alerts you'd check.

Sample Answer

1) Confirm scope & severity- Check alert firing: p50/p95/p99 latency, error-rate, and SLO burn rate. Note which endpoints/services are affected and time window.

2) Pull representative traces- Use distributed traces (service map + waterfall). Filter for slow traces (p95/p99) and compare to baseline traces.- Key signals: total trace duration, per-span duration, span start/end timestamps, span.kind (client/server), tags (http.status, db.statement, net.peer.ip), error tags.

3) Determine where time is spent (span analysis)- Aggregate spans by operation and parent service; compute histogram of span durations (p50/p95/p99).- If a single downstream span dominates trace duration (child span >> other spans), suspect that downstream microservice or external API.- If many small increases across multiple spans (network, serialization), suspect network or client-side queuing.

4) Network checks (if suspected)- Look at client vs server timestamps: high client-to-server gap indicates network latency.- Check TCP metrics: RTT, retransmits, packet loss, interface errors, route changes from host/network monitoring (eBPF, Cilium, VPC flow logs).- Correlate with traces showing increased transport-level times (e.g., net.peer.transport tag) and with histogram of round-trip time.

5) Downstream microservice checks- Inspect server-side spans in downstream service: CPU/memory, GC pauses, thread pool queues, DB call durations inside that service.- Look for heavy child spans (DB, cache) inside downstream; check its own metrics/alerts: CPU, GC, request queue length, connection pool saturation.- Use flame graphs or service-level spans to locate expensive operations.

6) External third-party API checks- Identify spans where net.peer.ip or peer.service points to external host; check client-side connect + wait-for-response durations.- Correlate with outbound HTTP status codes, retries, backoff events, and increased error rates.- Cross-check vendor status pages and synthetic tests.

7) Correlate with metrics & histograms- Use histogram aggregation for latency buckets over time per operation to see whether tail latency increased.- Cross-correlate with infra metrics (CPU, GC, thread_pool.queue, DB slow-queries), network metrics, and request rates to rule out load-induced queuing.

8) Validate hypothesis and act- If network: involve network team, apply routing fixes, enable retries/backoff, or regional failover.- If downstream service: scale replicas, increase connection pool, fix hot code path.- If third-party: implement caching, circuit breaker, degrade gracefully, or contact provider.

9) Post-incident- Create alerts: detect divergence between client and server span durations, p99 per-span, and sudden histogram bucket shifts. Add synthetic probes and SLA-oriented SLOs for external dependencies.

Signals/alerts to check: p95/p99 latency, span-duration histograms per operation, increased client-server timestamp gap, TCP retransmits, connection pool saturation, error-rate spikes, retry/backoff counts, SLO burn.

HardTechnical

45 practiced

Compare and contrast the trade-offs between aggressive retries, client-side rate limiting, circuit breakers, and backpressure mechanisms when protecting services from overload. For each approach discuss latency impact, resource utilization, user experience, and scenarios where the mechanism can worsen overall system health.

Sample Answer

High-level summary: all four are tools to protect services from overload but they make different trade-offs across latency, resource use, UX, and failure modes. Below I compare them as an SRE would.

Aggressive retries- Latency: increases tail latency dramatically (retries add cumulative wait).- Resource utilization: multiplies load on downstream services, causing cascading overload.- UX: may hide transient failures for users if quick success occurs, but often causes longer waits or timeouts.- When harmful: under systemic overload or network partition, retries amplify demand and can drive collapse (retry storm).

Client-side rate limiting- Latency: reduces latency variability by preventing bursts; rejected requests fail fast.- Resource utilization: lowers demand to sustainable levels, conserving downstream capacity.- UX: users see earlier errors (429s) instead of long waits—better for predictability but less transparent.- When harmful: misconfigured limits can starve critical workloads or create uneven fairness; if clients don't respect limits, benefits vanish.

Circuit breakers- Latency: when open, fail fast (low latency); when half-open, may add probing delays.- Resource utilization: prevents waste on doomed calls, giving the system time to recover.- UX: short-term failures surface quickly; progressive recovery avoids repeated wasted work.- When harmful: incorrect thresholds or slow failure detection can split-brain behavior or hide degraded service leading to cascading failures elsewhere (e.g., pushing load to a less resilient fallback).

Backpressure (e.g., producer throttling, client pushback, reactive flow control)- Latency: can increase end-to-end latency due to buffering and slowdowns, but keeps system responsive rather than failing.- Resource utilization: smooths consumption, avoids bursts, and preserves overall throughput.- UX: may slow progress but maintains correctness; better for long-running pipelines than interactive requests.- When harmful: poor flow-control policies cause head-of-line blocking, queue bloat, or unfairness; if upstream can’t adapt, queues fill and memory or downstream timeouts cause failures.

Practical guidance for SREs- Combine approaches: e.g., client rate limits + circuit breakers + measured retry with exponential backoff and jitter; use backpressure where protocol supports it (HTTP/2, gRPC).- Tune for SLOs: prefer fail-fast (rate limit/circuit-breaker) for interactive paths to protect latency SLOs; use backpressure for batch/streaming workloads.- Monitor key signals: error rates, queue lengths, p99 latency, CPU/memory saturation; run chaos experiments to validate thresholds.- Beware config drift: automated throttles and breakers must be observable and dynamically adjustable during incidents.

HardSystem Design

46 practiced

You're using a service mesh (e.g., Istio). Design a fault injection strategy integrated with the mesh to run controlled latency and abort experiments across services while ensuring observability and safe rollback. Discuss virtual services/destination rules, traffic mirroring, RBAC, and how to correlate traces and metrics to injected faults.

Sample Answer

Requirements:- Run controlled latency/abort experiments (percent-based, per-path/service)- Full observability (traces, metrics, logs) and correlation to injected faults- Safe rollout and automated/manual rollback- RBAC to restrict who can create experiments- Minimal blast radius, ability to mirror traffic for analysis

Design summary:- Use Istio VirtualService HTTP fault injection (delay/abort) with weighted routes for percent-based experiments.- Use DestinationRule subsets to target specific versions/instances.- Mirror subset traffic to a non-production shadow service for offline analysis.- Enforce Istio RBAC (AuthorizationPolicy + Kubernetes RBAC) so only SRE/chaos teams can apply fault injections.- Correlate faults by injecting a custom header and propagating trace IDs; enrich metrics and spans with an "experiment_id" tag.- Automate safe rollout with canary percentages, guardrails (Prometheus alerts on SLO violation) and an automated rollback controller.

Example VirtualService fault injection (10% abort, 20% delay for a canary):

yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: orders
spec:
  hosts: ["orders.svc.cluster.local"]
  http:
  - match:
    - uri:
        prefix: /checkout
    route:
    - destination:
        host: orders
        subset: v1
      weight: 80
    - destination:
        host: orders
        subset: v2
      weight: 20
    fault:
      abort:
        httpStatus: 503
        percent: 10
      delay:
        fixedDelay: 200ms
        percent: 20
    mirror:
      host: orders-shadow

DestinationRule with subsets:

yaml

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: orders-dr
spec:
  host: orders
  subsets:
  - name: v1
    labels: { version: "v1" }
  - name: v2
    labels: { version: "v2-canary" }

RBAC (restrict who can change VirtualServices):- Kubernetes RBAC: limit ability to update istio networking CRDs to group "sre-team".- Istio AuthorizationPolicy: restrict traffic-management endpoints (e.g., using Gatekeeper or admitting webhook) and require mTLS for admin paths.

Observability and correlation:- Add experiment metadata: when creating an experiment, inject a header X-Experiment-ID and X-Fault-Type via EnvoyFilter or sidecar Envoy header manipulation for requests routed into the faulted subset. Ensure trace context (traceparent / x-request-id) is preserved.- Instrument services to read X-Experiment-ID and add as span attribute and a Prometheus label (use OpenTelemetry/OTLP exporters). Example span tag: experiment_id=<id>, fault_injected=true.- Metrics: create Prometheus recording rules and dashboards that slice latency/error-rate by experiment_id, subset, endpoint.- Tracing: configure Jaeger/Zipkin to include experiment_id in span tags so you can query traces affected by faults.

Safe rollout & rollback:- Start with 0.1–1% traffic for destructive faults, higher for non-destructive latency.- Use short time-boxed windows (e.g., 10m) and observe SLOs (p99 latency, error-rate).- Automated guardrail: Prometheus alert fires if error-rate or latency breach threshold; a controller (Argo Rollouts/Flux/CD) watches alerts and reverts VirtualService to last known-good by applying stored YAML.- Maintain GitOps: experiments are created via PRs to a "chaos" repo requiring approvals; changes auto-applied only after policy checks.

Additional safety:- Only run during low-risk windows or on dev/staging clusters; when in prod, require multi-person approval.- Use traffic mirroring to shadow production requests to a non-critical environment before injecting active faults.- Log all experiment actions centrally (audit logs) for postmortems.

Trade-offs:- Envoy-level faults are fast and low-overhead but may not simulate every backend failure mode (e.g., resource exhaustion).- Mirroring increases load on shadow systems — provision accordingly.

Why this works:- Istio VirtualService/fault injection provides precise per-route control.- DestinationRule subsets + weights enable canary percent control.- Header/trace enrichment ensures end-to-end observability and easy filtering.- RBAC + GitOps + automated guardrails provide safe, auditable, reversible experiments.

Unlock Full Question Bank

Get access to hundreds of Resilience and Chaos Engineering interview questions and detailed answers.

Join thousands of developers preparing for their dream job.