Dependency Failures and Graceful Degradation Questions

Handling failures in external services or dependencies: rate limiting (HTTP 429), timeouts, quota exhaustion. Understanding circuit breakers, intelligent retries, and how to design services that behave well when dependencies fail. Knowing when to disable features vs. when to queue/cache.

EasyTechnical

36 practiced

Explain eventual consistency and how dependency failures complicate consistency guarantees. As a Solutions Architect, how would you present trade-offs between strong and eventual consistency to product and operations teams when designing systems that must continue operating during outages?

HardSystem Design

28 practiced

Design API contract semantics for partial success in a batch endpoint: how to represent per-item statuses, global error metadata, retry advice, and backward compatibility rules. Provide an example JSON response shape for a 10-item batch where 3 items failed and explain how clients should interpret and act on it.

Sample Answer

Requirements & goals:- Allow batch endpoints to return per-item success/failure, global metadata (request id, timestamps), retry advice, and preserve backward compatibility (v1 clients).- Convey deterministic machine-readable statuses + human messages.- Keep small, stable top-level fields; extend via namespaced objects.

Design:- Top-level: requestId, processedCount, successCount, failureCount, retryAfter (seconds, optional), errors (global list), items (array of per-item results).- Per-item result: id (client-provided item id), status (enum: "success","failed","skipped","pending"), code (machine error code), message (human), retryable (bool), details (opaque object for diagnostics).- Semantic rules: If any items fail, HTTP 207 Multi-Status for partial success; 200 for all-success; 4xx/5xx for total failure. Clients must parse body even on non-200 when 207 used.- Backward compatibility: v1 responded with 200 + items[].status only. Preserve items[] order and id field; keep status values unchanged. New fields additive and optional. Use version header/response-v to signal schema.

Example response (10 items, 3 failed):

json

{
  "requestId": "req_20251206_01",
  "processedCount": 10,
  "successCount": 7,
  "failureCount": 3,
  "retryAfter": 30,
  "errors": [
    {"code":"RATE_LIMIT","message":"Request partly throttled; see per-item codes"}
  ],
  "items": [
    {"id":"i1","status":"success"},
    {"id":"i2","status":"failed","code":"INVALID_PAYLOAD","message":"Missing field 'email'","retryable":false},
    {"id":"i3","status":"success"},
    {"id":"i4","status":"success"},
    {"id":"i5","status":"failed","code":"DEPENDENCY_ERROR","message":"External service unavailable","retryable":true,"details":{"service":"payments","eta_seconds":45}},
    {"id":"i6","status":"success"},
    {"id":"i7","status":"success"},
    {"id":"i8","status":"failed","code":"DUPLICATE","message":"Item already exists","retryable":false},
    {"id":"i9","status":"success"},
    {"id":"i10","status":"success"}
  ],
  "responseVersion":"v2"
}

How clients should interpret & act:- Always check HTTP status: 200 => all success; 207 => inspect body for per-item statuses; 4xx/5xx => treat as global failure unless body indicates otherwise.- For each item: - success: proceed. - failed && retryable=true: schedule retry using retryAfter or item.details.eta_seconds; apply backoff. - failed && retryable=false: surface error to user/operator for fix.- Use requestId for support/tracing.- If client is older (v1) and doesn't recognize 207, advise falling back to reading items[] and treating non-success statuses as failures; server may include compatibility header X-Compat-Status: 200 to ease older clients (transitional only).

Scaling & trade-offs:- Keep per-item payloads small; support pagination for huge batches.- 207 allows semantic clarity but some clients/HTTP libs mis-handle; provide clear SDKs and a header for compatibility.- Provide mappings of machine codes to remediation docs (URL in errors) to reduce back-and-forth.

EasyTechnical

56 practiced

Compare synchronous versus asynchronous dependency calls in the context of graceful degradation. Provide recommendations of when asynchronous architectures (message queues, eventing) offer better resilience and when synchronous calls are still appropriate, including effects on latency, consistency, and user experience.

Sample Answer

Situation: When designing systems that must continue operating under partial failures, choosing between synchronous and asynchronous dependency calls affects resilience, latency, consistency, and user experience.

Short answer / definitions:- Synchronous call: client waits for dependency response (HTTP/RPC). Easy to reason about strong consistency and immediate success/failure semantics.- Asynchronous call: decoupled via message queues/events; caller returns quickly and work is processed later. Enables loose coupling and higher resilience.

When async is better (recommendations):- Non‑blocking work: notifications, analytics, background processing where eventual consistency is acceptable.- Spiky or unreliable downstreams: message queues buffer load, smooth bursts, and allow retries/backoff without blocking users.- Graceful degradation: surface core functionality immediately and degrade non‑critical features (e.g., show cached view, enqueue update for later).- Example: order placement — respond to user immediately and enqueue downstream invoicing, fraud checks, shipping.

When sync is appropriate:- User-facing actions that require immediate confirmation or strict consistency: payment authorization, real‑time inventory checks, authentication flows.- Simple low‑latency dependencies with high availability SLAs where rollback/compensation is harder.

Effects:- Latency: Sync increases end-to-end latency (user waits); async reduces perceived latency but shifts completion later.- Consistency: Sync provides stronger immediate consistency; async leads to eventual consistency and requires idempotency and compensating actions.- User experience: Async improves perceived responsiveness but must communicate state (e.g., “Processing — you’ll receive confirmation”). Provide status APIs, notifications, and UX fallbacks.

Operational considerations:- Implement durable queues, dead‑letter handling, monitoring, and backpressure strategies.- Design idempotent consumers, versioned events, and compensation workflows.- Use hybrid patterns: synchronous for critical path, async for best‑effort and long‑running tasks.

Recommendation summary:- Default to synchronous for critical, consistency-sensitive user flows.- Use asynchronous architectures to increase resilience and scalability for non‑critical or long‑running work, and combine with clear UX patterns for state visibility and graceful degradation.

HardTechnical

33 practiced

Design a safe chaos engineering program to simulate dependency failures in pre-production and limited production. Define blast radius controls, roll-back automation, runbook requirements, metrics to evaluate resilience, and gating criteria for expanding experiments across teams and regions.

Sample Answer

Framework: run chaos in three phases—pre-production (dev/stage), limited-production (canary subset), then broader production. Start small, iterate, and require automated safety gates at every step.

Blast-radius controls- Scope: target specific service instances, namespaces, or AZs only; avoid global resources (auth, billing).- Percentage limits: cap affected traffic/users (e.g., 1% → 5% → 20% rollout).- Resource limits: CPU/memory, one DB replica, one region.- Timebox: experiments auto-stop after short windows (e.g., 5–15 minutes).- Access & approvals: RBAC and change tickets; scheduled maintenance windows.

Rollback & automation- Automated rollback triggers: breach of health checks (error rate, latency, SLI degradation), failed canary metrics, SLO breach, or manual abort.- Implementation: orchestration tool (Chaos Mesh/Gremlin/Litmus) integrated with CI/CD and feature-flag system to flip experiments off and roll back to last known-good infra via IaC (Terraform/CloudFormation) and orchestration scripts.- Safety actions: circuit-breaker activation, route traffic away (load balancer), restart affected pods, restore DB replica, and execute blue/green switch if necessary.

Runbook requirements- Clear detection steps: alert sources, runbook ID, owner.- Immediate mitigation commands (CLI/API): stop experiment, toggle feature flag, scale down service, failover steps.- Escalation matrix: on-call contacts, SRE, product, legal.- Validation & postmortem checklist: data collection, timeline, RCA, remediation plan.- Pre-run checklist: observability available, backups current, canary routing ready, rollback tested.

Metrics to evaluate resilience- SLOs/SLIs: error rate, p99 latency, request success rate, throughput.- Availability & user impact: affected user count, session loss.- Detection & recovery: MTTD, MTTR, % successful automated rollbacks.- Business impact: revenue loss estimate, support ticket volume.- Experiment health: false-positive aborts, roll-forward success.

Gating criteria to expand- Quantitative thresholds: zero SLO breaches in pre-prod; in limited-prod allow only transient minor deviations (e.g., error rate < 0.5% above baseline) and automated rollback success > 95%.- Observability maturity: full tracing, metrics, logs, alerting with runbook links.- Team readiness: runbook drills completed, paging ownership, postmortem culture.- Security & compliance signoff for region-specific experiments.- Pilot duration & reproducibility: at least 3 successful runs over 2 weeks with consistent metrics.

Example scenario- Simulate third-party API latency: run in pre-prod, verify fallbacks, instrument feature flag in limited-prod for 2% traffic with 10-minute timebox. If p99 latency rises >50% or error rate >0.5% above baseline, abort automatically, flip feature flag, and failover to cache.

Trade-offs & rationale- Conservative blast radius reduces business risk but slows confidence building. Automated rollbacks and robust observability accelerate safe scaling. Emphasize frequent, small experiments and strong postmortems to improve system resilience incrementally.

HardSystem Design

25 practiced

Design a resilient social feed service that must continue serving read-heavy feeds during sudden spikes when the external recommendation engine is unavailable. Describe cache hierarchy, precomputed feeds, fallback ordering (time-decay), feature flags to toggle enhanced ranking, and user experience trade-offs for ordering quality versus availability.

Sample Answer

Requirements & constraints:- Read-heavy: millions of feed reads/sec; writes (new posts, interactions) relatively lower.- Must continue serving reasonable feeds during sudden spikes or when external recommendation engine (RE) is down.- Low latency (<200ms p95), eventual freshness, tunable ranking features.

High-level architecture:- Ingest & write path: Posts → Write API → Event stream (Kafka) → Fan-out workers produce per-user entries & update precomputed store.- Recommendation Engine (RE): external service that computes personalized scores; used to enrich ranking when available.- Feed API (read path): Layered cache → Precomputed feed store → Fallback ranking service → Time-decay ordering.- Observability + Feature Flag service for ranking toggles.

Cache hierarchy:1. L1: Edge CDN / POP-level cache for hot users (short TTL ~5–30s) to absorb spikes.2. L2: In-memory distributed caches (Redis Cluster / KeyDB) per region storing precomputed feed slices (e.g., top N per user) with TTLs and version keys.3. Persistent store: Sharded NoSQL (Cassandra/DynamoDB) storing durable precomputed feed lists and metadata.

Precomputed feeds:- Continuous fan-out (or hybrid fan-out-on-read for high-fanout) builds per-user sorted lists containing post IDs + light metadata and baseline scores (engagement, recency).- Store feeds as time-ordered segments (e.g., last 1000 items) to support pagination; maintain per-user version and tombstones for deletes.- Periodic recompute jobs enrich feeds with RE scores when RE is available; store enrichment separately to allow fast fallback.

Fallback ordering (when RE unavailable):- Use deterministic multi-factor scoring computed locally: Score = w1 * recency_decay(ts) + w2 * normalized_engagement + w3 * social_weight + w4 * machine-learned_local_model- recency_decay(t) = 1 / (1 + alpha * age_hours) or exponential decay; tune alpha per product need.- If enrichment missing, sort precomputed items by fallback score (time-decay dominant).- For freshest content, merge last-minute writes from write-ahead buffer (Kafka) into top results.

Feature flags & rollout:- Global and per-user flags to toggle enhanced ranking, RE-enrichment, or aggressive personalization.- Flags control: use_RE_score, use_local_ml_model, recency_bias_strength.- Canary rollout: enable enhanced ranking for small % of users; monitor latency, CTR, engagement, error rates.- Emergency kill-switch to route all reads to fallback-time-decay ordering.

Handling spikes & RE outages:- Circuit breaker around RE client with exponential backoff; on trip, mark RE_unavailable and notify feed API.- When RE down, increase cache TTLs slightly & serve precomputed/fallback-ranked feeds to preserve availability.- Backpressure: degrade non-essential enrichment calls, batch score requests, and prioritize top-K users.

Consistency & freshness trade-offs:- Precomputed feeds favor availability & read latency; slightly stale (seconds–minutes).- Fallback ordering sacrifices per-user optimal relevance for guaranteed availability.- Provide UI signals: “Recent” vs “For You” tabs — when RE unavailable, show Recent (time-ordered) and mark “Personalized” offline to set expectations.

User experience trade-offs:- Ordering quality vs availability: - Prioritize availability: serve time-decay or baseline personalized feeds — consistent low latency but lower precision. - Prioritize quality: wait for RE enrichment (higher latency/failure risk) or degrade gracefully for fewer users.- UX mitigation: show hybrid UI — top 3 slots reserved for freshest content (time-decay), remaining slots use enriched ranking when available. Show subtle indicator or reduced personalization when fallback active to maintain trust.

Operational considerations:- SLOs: p95 latency, availability during RE outage > 99.9%.- Metrics: cache hit rate, RE error rate, engagement delta between enriched vs fallback.- Capacity planning: autoscale Redis & API layers; warm caches for anticipated spikes.- Security & privacy: store minimal PII in caches, respect entitlements.

Summary:Design centers on layered caching + precomputed feeds with a robust fallback scoring (time-decay + local signals), feature-flag driven control for ranking behavior, circuit breakers for the RE, and clear UX trade-offs (explicit tabs/indicators) so the service stays highly available during RE outages while degrading ranking quality gracefully.

Unlock Full Question Bank

Get access to hundreds of Dependency Failures and Graceful Degradation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.