System Design and Reliability Questions

Design principles and trade offs for building highly scalable and reliable distributed systems. Expect discussion of capacity planning, partitioning and sharding, caching and load balancing strategies, replication and consistency models, latency and throughput trade offs, fault tolerance, graceful degradation, redundancy, disaster recovery, monitoring and alerting, and postmortem culture. Candidates should reason about non functional requirements and propose architectures meeting targets for scale, performance, and operational resilience.

HardTechnical

106 practiced

You have budget to hire one new person: an SRE or a backend engineer. As EM, create a decision framework to choose which hire better reduces operational risk and improves velocity over the next 12 months. Include evaluation criteria, data to collect, and how hiring choice affects team composition.

MediumTechnical

97 practiced

You need to create runbooks for three common incidents: high error-rate, region outage, and degraded downstream dependency. For each incident type, list the key steps, ownership, safe mitigations, and escalation criteria. Explain how you'd keep runbooks maintained and discoverable across teams.

Sample Answer

**High error-rate**

- Situation & ownership: Primary on-call service owner (SRE/Backend lead) owns response; engineering manager notified if impact > SLO or 30+ minutes.- Key steps: confirm alert > threshold, collect logs/traces, identify recent deploys/config changes, roll forward debugging, apply safe mitigation (rate-limit, circuit-breaker, rollback).- Safe mitigations: rollback to known-good release, enable feature flags, increase autoscaling modestly.- Escalation criteria: error-rate > 2× SLO for 15m despite mitigations, customer-facing outage, or rollback fails → escalate to Eng Manager, Product, and Incident Commander.

**Region outage**

- Ownership: SRE + Platform lead; Eng Manager coordinates cross-team communication.- Key steps: detect region health, failover traffic to healthy regions, validate data consistency, runbook: activate cross-region DNS/load-balancer failover.- Safe mitigations: route to fallback region, enable read-only mode if writes unsafe.- Escalation: failover unsuccessful after 20m, data loss risk, or major customer impact → exec-level notification and war room.

**Degraded downstream dependency**

- Ownership: Consumer service owner; dependency owner notified.- Key steps: verify degradation, identify impact surface, switch to cached responses or degraded feature mode, throttle calls to dependency.- Safe mitigations: enable cache, circuit-breaker, queue requests for retry.- Escalation: SLA breach for >30m, business-critical flows blocked → escalate to Eng Manager and vendor/contact.

**Maintenance & discoverability**

- I maintain runbooks as living docs in a central runbook repo (versioned in Git) linked to PagerDuty services and service README. Each runbook includes owner, play-by-play, checklists, contact matrix, and post-incident review template. Quarterly reviews assigned as part of sprint for owners; automated tests validate playbook steps where possible (e.g., failover drills). Discoverability: integrate runbook links into on-call rota, Slack shortcuts, and onboarding docs; run periodic tabletop exercises to surface gaps.

MediumTechnical

87 practiced

Design an observability plan to support distributed tracing for a polyglot microservices platform. Specify what to instrument (spans, metadata), sampling strategy, aggregation points, dashboards, and alert rules that help detect cross-service latency and error propagation.

Sample Answer

**Approach (Engineering Manager perspective)** I’d deliver a pragmatic, standards-driven observability plan that balances signal quality, cost, and adoption across languages/teams.

**What to instrument**- Spans: entry/exit (HTTP/gRPC), DB queries, RPCs, message produce/consume, cache ops, background jobs, long-running tasks.- Metadata/tags: trace_id, span_id, service, env, version, endpoint, user_id (pseudonymized), http_status, db_statement (hashed), retry_count, error_type, sampling_reason, latency_ms.- Logs: attach trace_id/span_id to all logs; structured JSON.

**Sampling strategy**- Default head-based low-rate (e.g., 1%) for volume control.- Tail-based/conditional retention: retain 100% of traces with errors, high latency (P95/P99 > threshold), or unusual patterns.- Adaptive sampling per-service (higher for critical paths), and debug-mode for on-demand burst captures.

**Aggregation & pipelines**- Use OpenTelemetry collectors → processing layer (batching, enrichment) → tracing backend (Jaeger/Tempo) + metrics exported to Prometheus/Longview.- Link traces → logs → metrics via trace_id; store spans for X days, metrics for SLO window.

**Dashboards**- Service map with dependency graph.- Latency panels: P50/P95/P99 per service & per downstream call.- Trace count, error traces, tail latency heatmaps, flame graphs for hotspots.- SLO dashboards (latency/error) with burn-rate panels.

**Alert rules**- SLO burn-rate alert (e.g., 5-minute burn > 2x target).- P95 service latency breach (example: P95 > 500 ms for 5m).- Cross-service latency propagation: if root span latency increases >30% AND 3+ downstream services show >20% P95 increase in same time window → high-priority alert.- Error cascade detection: correlation alert when error-rate spike (>1% absolute) in one service coincides with downstream errors/traces containing failed RPCs > threshold.- Anomaly detection baselines for unusual trace volumes or tail-latency spikes.

**Governance & rollout**- Instrumentation standards, SDKs, code-review checklist, tracing playbook, quarterly audits, and training. Start with critical user journeys, iterate by impact.

This plan balances actionable signal, cost control, and team adoption while enabling fast detection and root-cause of cross-service latency and error propagation.

MediumSystem Design

96 practiced

Design a graceful-degradation plan for a payment authorization service that must maintain some level of operation during partial failures (e.g., regional network partition). Explain what functionality you would degrade, how you'd prioritize transactions, and how you'd ensure eventual consistency of queued transactions.

Sample Answer

**Clarify requirements & constraints**- Objective: keep high-value, low-risk payments flowing during partial failures (e.g., regional partition) while preserving safety and legal constraints (fraud checks, settlement rules).- Constraints: limited external connectivity, degraded downstream services, SLA tiers by customer/txn.

**High-level graceful-degradation plan**- Tier functionality into A (must-have), B (nice-to-have), C (nonessential). - A: authorize based on cached card/token state, reserve funds, return deterministic decision. - B: delayed/async fraud checks, reduced risk scoring, simplified logging. - C: promotions, analytics, cross-region reconciliation in real time.

**Transaction prioritization**- Priority order: high-value merchants / regulators / recurring subscriptions > new/guest checkouts > low-value one-offs.- Use SLA metadata and merchant-configured policies to compute priority. Apply token buckets to enforce throughput per tier.

**Queueing & eventual consistency**- Local durable queues per region (WAL-backed) accept degraded approvals with a reconciliation flag.- Backfill workers attempt to sync to central authorization/exchange when connectivity restores; use idempotent APIs and vector clocks or monotonic sequence numbers to order retries.- Store original decision, rationale, and risk score to allow human/automated rollback if post-check fails.

**Operational controls**- Feature flags to toggle degradation modes; dashboard for queue depth, retry success rates, fraud anomaly metrics.- Runbooks: when partition detected, escalate to on-call, inform stakeholders, enable degrade mode automatically if thresholds hit.- Post-incident: reconcile metrics, audit logs, and policy adjustments; blameless postmortem with engineering and product.

**Team & governance**- Cross-functional design with payments, security, SRE, and product. Define SLA objectives and acceptance criteria, own automated tests and drills (chaos/partition tests) to validate behavior.

HardSystem Design

83 practiced

Design an automated canary analysis system that can automatically promote or rollback a deployment across services and regions based on a configurable set of golden metrics (latency, error-rate, business conversion). Discuss metrics selection, statistical tests, thresholds, sample sizes, and rollback safeguards.

Sample Answer

**Clarify requirements & goals**- Objective: automated canary promotion/rollback across services & regions using golden metrics (latency, error-rate, conversion). Allow config per-service/region, SLAs, and business-owner overrides.

**High-level architecture**- Canary Orchestrator service triggers canary deployments (k8s/CF). - Metrics Collector (Prometheus/CloudWatch) + Feature Flags store.- Analysis Engine runs statistical tests and decision engine.- Execution Layer performs promote/rollback and notifies incident/owners.- Dashboard + Audit log + Pause/override UI.

**Metrics selection**- Tiered: SLI (latency p95, error-rate 5xx), SLO-derived thresholds, and business metric (conversion, revenue per session). Use both local service signals and downstream user-impact metrics.

**Statistical tests & thresholds**- Use sequential testing: non-parametric Mann-Whitney U for latency, proportion z-test or Fisher’s exact for error-rate/conversion. Apply Benjamini-Hochberg to control false discovery across multiple metrics.- Define delta thresholds: e.g., relative increase > 10% for error-rate or absolute p95 latency > SLA + 50ms. Combine with significance (p < 0.01) and minimum effect size.

**Sample sizes & cadence**- Minimum sample size computed from power analysis (target 80% power). For low-traffic services, extend canary duration or increase traffic % to reach n. Start with conservative traffic ramp (1%→5%→20%) with minimum observation windows per ramp (e.g., 5–15 min depending on metric variance).

**Decision logic & safeguards**- Multi-metric policy: require X out of Y metrics to fail to trigger rollback, or weight business metric higher. Use cooldown windows to avoid oscillations.- Safety nets: automatic rollback on catastrophic signals (error-rate spike > 5x baseline), circuit breaker to halt new canaries, hard limits per region.- Human-in-loop: require manual approval for changes impacting revenue, and allow emergency force-rollback.- Observability & runbooks: automated alerts, playbooks attached to failures, detailed audit trail for compliance.

**Operational & people considerations**- Define ownership for metric thresholds and runbook owners. Run quarterly reviews of golden metrics with product and SRE. Use post-mortems after rollbacks to improve tests and thresholds.

This design balances statistical rigor, safety, and operational transparency while enabling safe automated promotion across services and regions.

Unlock Full Question Bank

Get access to hundreds of System Design and Reliability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.