Dependency Failures and Graceful Degradation Questions
Handling failures in external services or dependencies: rate limiting (HTTP 429), timeouts, quota exhaustion. Understanding circuit breakers, intelligent retries, and how to design services that behave well when dependencies fail. Knowing when to disable features vs. when to queue/cache.
EasyTechnical
25 practiced
Describe best practices for setting timeouts and request cancellation on outbound RPC/HTTP calls as an SRE. Include how to choose timeout values relative to upstream SLAs, how to implement cascading cancellation, how to propagate context, and how to avoid 'zombie' requests tying up resources.
Sample Answer
Start with clear SLAs and percentile measurements. Choose timeouts as a fraction of the upstream SLA/latency SLO—for example, if upstream 99th percentile latency = 500ms and your caller needs headroom for retries/processing, set outbound RPC timeout to ~50–70% (e.g., 250–350ms) or use p95 as baseline. Always measure real latencies and tune; prefer percentile-based targets (p95/p99) not averages.Cascading cancellation and context propagation- Use a single request context/deadline that flows through your call graph so child calls inherit the same deadline; cancel upstream work when the parent times out or client disconnects.- Propagate trace and request IDs (traceparent, x-request-id) and deadlines if supported (e.g., grpc/HTTP headers) so downstream systems can make informed decisions.Implementation patterns (example in Go)For Node/Java, pass cancellable tokens/promises tied to the incoming request.Avoiding "zombie" requests- Enforce both connection and request (read/write) timeouts at the transport layer.- Respect cancellations immediately in handlers and clean up resources (goroutines, file handles).- Use circuit breakers, concurrency limits, and request hedging with global limits to prevent resource exhaustion.- Ensure server queuing limits and request queues are bounded; drop or reject early when overloaded.Retries and backoff- Limit retries and ensure each retry fits within the original parent deadline; prefer idempotent-only retries and exponential backoff with jitter.Observability- Emit metrics for timeouts, cancellations, latency percentiles, and active requests. Alert on rising timeout/cancellation rates.Summary: pick percentile-based timeouts with headroom, propagate a single cancellable context and trace IDs to all downstream calls, enforce transport/read/write timeouts, use circuit breakers/concurrency limits, and instrument cancellations to avoid zombies.
go
// ctx has incoming deadline; pass it to outbound call
req = req.WithContext(ctx)
resp, err := httpClient.Do(req)
// for gRPC: client.Call(ctx, ...)HardTechnical
33 practiced
Formulate an optimization problem for selecting backpressure thresholds across a network of services to minimize end-to-end latency subject to per-service quota constraints. Describe variables, objective function, constraints, and outline an approach to compute thresholds in near real-time as loads change.
Sample Answer
Clarify objective: choose per-service backpressure thresholds (watermarks) that throttle upstream requests so end-to-end latency is minimized while each service respects a quota (max allowed throughput) and system remains stable under changing load.Variables- For N services in path order, let t_i be threshold at service i (requests in-flight or queue length where backpressure engages).- λ_in is external offered load; r_i(t) is effective service rate at i as function of t (reduces arrivals when upstream throttled).- q_i(t,λ) expected queue length or latency contribution at service i (use queueing model or empirical map).Objective (minimize expected end-to-end latency L):Minimize L(t) = sum_{i=1..N} q_i(t,λ)/μ_i(or more generally E[latency]=Σ_i f_i(t_i, λ) )Constraints1. Per-service quota: throughput_i(t,λ) ≤ Q_i for all i.2. Capacity/stability: throughput_i(t,λ) ≤ μ_i (service capacity).3. Threshold bounds: 0 ≤ t_i ≤ T_i_max.4. Safety SLA: P(latency > S) ≤ α (chance constraint) — can be approximated by deterministic bounds.Modeling choices- Use M/M/1 or M/G/1 approximations to express q_i(·) as convex in arrival rate; r_i(t) links thresholds to arrival reduction (monotone).- If f_i convex, the problem can be cast as convex program: minimize convex L subject to linear/convex constraints.Approach for near-real-time computation1. Instrumentation: continuously estimate arrival rates, service rates μ_i, current queues, tail latencies; fit lightweight parametric maps r_i(t) and q_i(·) (e.g., linear or quadratic).2. Fast optimizer: - Centralized: solve convex QP/MINLP every control interval (e.g., 1–10s) using efficient solvers (OSQP, CVXGEN) — warm-start using previous solution. - Distributed: use dual decomposition or ADMM so each service optimizes locally given shadow prices for quotas.3. Adaptive control: use Model Predictive Control (MPC) — predict next k intervals, optimize thresholds, apply first step, repeat.4. Fallback heuristics: PID or reinforcement-learning policy for extremely tight latency budgets.5. Safety and smoothing: enforce rate limits on threshold changes, add regularization (||t - t_prev||^2) to avoid oscillation.6. Validation: run A/B or shadow testing; tune model lag and solver frequency.Trade-offs- Complexity vs accuracy: richer queue models improve optimality but cost solver time.- Centralized optimality vs distributed robustness.- Use conservative quotas and chance constraints for tails.This yields a practical pipeline: estimate models → solve convex/ADMM optimization with warm-start/MPC → apply smoothed thresholds → monitor and adapt.
EasyTechnical
54 practiced
List and justify at least six key observability metrics and alerts an SRE should have to detect and triage dependency degradation. Include metrics for error rates, latency percentiles, saturation, retries, circuit breaker state, and dependency-specific quotas. Explain why each is useful.
Sample Answer
1) Upstream error rate (5m rate; 500/4xx ratio) — Alert when >= X% of calls return 5xx or error codes (e.g., >1% sustained for 5m). Why: first indicator a dependency is failing; helps separate client vs upstream faults. Triage: check dependency logs, fallback behavior, and recent deploys.2) Latency percentiles (p50/p95/p99) — Alert on sustained p95/p99 increase (e.g., p95 > 500ms or >2x baseline for 5m). Why: higher tail latency often precedes timeouts and user impact. Triage: identify slow endpoints, look at service maps and traces.3) Saturation (connection pool utilization, CPU, socket/file descriptors against capacity) — Alert when utilization >80–90%. Why: resource exhaustion causes cascading failures. Triage: scale, increase pool sizes, investigate leaks.4) Retry/backoff volume and success-after-retry rate — Alert when retries increase >X% or success-after-retry falls. Why: rising retries indicate transient failures and add load; falling success-after-retry signals persistent degradation. Triage: reduce client retries, enable circuit breaker.5) Circuit-breaker state / open-rate and recent trip count — Alert when CB opens frequently or stays open >N seconds. Why: indicates upstream instability and client protection kicking in. Triage: inspect root cause, adjust thresholds, route traffic to fallback.6) Dependency-specific quotas / throttling responses (HTTP 429, quota metrics) — Alert when throttles exceed normal (e.g., 429 rate spikes). Why: service limits cause errors/latency; distinguishes quota issues from outages. Triage: request quota increase, implement client-side rate limiters, backoff.Additional useful items: end-to-end SLO error budget burn rate alert; distributed tracing span error ratio to pinpoint failing dependency. Each alert should include severity, runbook link, and owner to speed triage.
HardSystem Design
35 practiced
Design a global distributed service that depends on a rate-limited external API. Requirements: 99.95% availability from the user's perspective, 100M active users, peak total QPS 20k, external API quota 1k QPS per account with per-region quotas. Describe architecture to maintain SLOs, including caching, aggregation, quota brokers, regional proxies, and graceful degradation strategies.
Sample Answer
Requirements (clarify): 99.95% user-facing availability, 100M active users, peak global QPS 20k, external API quota 1k QPS per account and per-region quotas. Goal: absorb user traffic while respecting external API limits and degrade gracefully.High-level architecture:- Global ingress → Regional Proxies (multi-AZ per region) → Local Cache & Aggregator → Quota Broker(s) per region → External API clients (multiple accounts across regions) → Fallback services.Key components:1. Regional Proxies: terminate client traffic, enforce per-user and per-tenant rate limits, route to local aggregator or cache. Deployed in at least 3 regions for geo redundancy.2. Local Cache (read-through, TTL + staleness bounds): store recent external API responses to satisfy cache hits instantly. Use Redis cluster per region with multi-AZ replication.3. Request Aggregator / Coalescer: collapse duplicate concurrent requests for same key into single external call; respond to waiting callers from that single response.4. Quota Broker: central per-region service tracking available external API quota across all provider accounts. Implements token-bucket with distributed consensus (lightweight: Redis with atomic ops or Raft-backed service). Brokers allocate tokens to regional proxies and can reassign across regions if provider supports global quotas.5. Multi-account External Clients: rotate requests across multiple provider accounts and regions to respect per-account/per-region caps. Circuit-breakers per account.6. Graceful Degradation: prioritized feature levels — for low-priority calls serve stale cache or synthetic fallback; for mid-priority return reduced fidelity; for high-priority queue with user-visible wait indicator. If quotas exhausted, degrade nonessential paths first.7. Backpressure & Queuing: local bounded queues with latency SLAs; requests exceeding queue/time budget get degraded response.8. Observability & Automation: per-request tracing, quota metrics, error budgets, alerts when broker nearing exhaustion; auto-scale proxies and cache; automated failover to alternate provider accounts/regions.Data flow summary:Client -> Regional Proxy -> Check cache -> if miss, Aggregator asks Quota Broker for token -> Broker grants token -> External call via chosen account -> cache result -> respond to clients.Scalability & resilience:- 20k QPS distributed across regions — combine caches + aggregation to reduce external calls by >90% (target).- Use multiple provider accounts per region to reach effective 1k*#accounts capacity; Broker balances usage.- Brokers replicated and use optimistic allocation with periodic reconciliation to avoid oversubscribe.- Circuit breakers isolate failing accounts; automated failover to other accounts/regions.Trade-offs:- Strong consistency in quota allocation increases coordination overhead; prefer eventual consistency with short reconciliation windows to reduce latency.- Stale-cache serving improves availability but may reduce freshness—mitigate with TTL tuning per priority.- Cost vs availability: more accounts/regions increases cost but increases throughput and isolation.SLO enforcement:- Define per-region error budgets; monitor cache hit rate, external-call rate, queue latency, token exhaustion events; run playbooks that scale proxies, increase caching aggressiveness, or trigger feature degradation when approaching error budget.This design keeps user latency low, respects external quotas via Quota Brokers and multi-account rotation, and maintains 99.95% availability using caching, aggregation, regional isolation, and graceful degradation.
HardTechnical
28 practiced
Design a client library (pseudocode or API spec) that ensures safe retries across microservices participating in long-running workflows implemented with sagas. Requirements: deduplication across retries, idempotency support through idempotency keys, integration with distributed tracing, and minimal developer friction.
Sample Answer
Approach: provide a small client library API that (1) normalizes idempotency via keys, (2) performs reliable deduplication using a central datastore (Redis/etcd/RDBMS) with CAS, (3) integrates trace context propagation, (4) exposes minimal primitives so developers only call one method per saga step. Use optimistic locking + response caching to return prior results on retries.API (pseudocode / Python-style):Key concepts / reasoning:- Deduplication key composition includes workflow_id + step_name + supplied idempotency_key to scope keys per saga step and workflow.- Use SETNX (or DB insert with unique constraint) to ensure a single executor per logical operation — prevents duplicate side-effects.- Cache the authoritative response so subsequent retries return the same result (strong consistency for callers).- Propagate tracing context so all retries/duplicate-suppression operations appear in the same trace; helps SRE debugging.- Short TTLs for ERROR to allow quick reattempts; longer TTL for DONE to allow late retries.- Minimal developer surface: one call_step per step; they only provide idempotency_key and workflow metadata.Operational considerations / edge cases:- Store availability: degrade to circuit-breaker/fallback mode; annotate traces and emit alerts.- Clock skew: use store's TTL semantics; prefer Redis server time or DB transactions.- Very long-running sagas: rotate/extend TTLs or use persistent DB with explicit completion records.- Idempotency key leaks: keys should be scoped and have expiry to avoid unbounded growth.- Partial failure where service actually performed side-effect but crashed before storing DONE: use outbox pattern on service side or ensure service returns before acknowledging.Alternatives / trade-offs:- Use distributed unique constraint in RDBMS for stronger durability (but higher latency).- Use leader-election per key vs SETNX; SETNX is simpler and lower friction.- Exactly-once semantics require distributed transactions or outbox pattern; here we aim practical at-least-once with dedupe+cache.This design balances reliability, low developer friction (single API call), observability (trace propagation), and operational safety (dedupe + cached responses).
python
class SagaClient:
def __init__(self, store, tracer, default_ttl=3600):
self.store = store # e.g., Redis client supporting SETNX/GET/DEL
self.tracer = tracer # OpenTelemetry-compatible tracer
self.default_ttl = default_ttl
def call_step(self, service_url, workflow_id, step_name, idempotency_key, payload, timeout=30):
"""
Atomically dedupe + call remote service and cache response.
Returns (status, response).
"""
trace_ctx = self.tracer.current_context()
dedupe_key = f"saga:{workflow_id}:{step_name}:{idempotency_key}"
# Try to reserve the key (SETNX)
reserved = self.store.set_if_not_exists(dedupe_key, "IN_PROGRESS", ttl=self.default_ttl)
if reserved:
# We are the executor for this idempotent step
self.tracer.inject(trace_ctx, carrier=payload) # propagate trace
try:
resp = http_post(service_url, payload, headers={"Idempotency-Key": idempotency_key, "Trace": trace_ctx})
# store full response atomically with status marker
self.store.set(dedupe_key, {"status": "DONE", "response": resp}, ttl=self.default_ttl)
return ("DONE", resp)
except Exception as e:
# mark failure so other retries can know
self.store.set(dedupe_key, {"status": "ERROR", "error": str(e)}, ttl=short_ttl)
raise
else:
# Another worker is or was executing this key; wait/observe
entry = self.store.get(dedupe_key)
if not entry:
# race: retry attempt to reserve
return self.call_step(service_url, workflow_id, step_name, idempotency_key, payload, timeout)
if entry["status"] == "DONE":
return ("DONE", entry["response"])
if entry["status"] == "ERROR":
raise RemoteStepError(entry["error"])
# IN_PROGRESS: optionally block with backoff and re-check
wait_for_completion(dedupe_key, timeout)
entry = self.store.get(dedupe_key)
if entry and entry["status"] == "DONE":
return ("DONE", entry["response"])
raise TimeoutError("step did not complete in time")Unlock Full Question Bank
Get access to hundreds of Dependency Failures and Graceful Degradation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.