System Design and Reliability Questions

Design principles and trade offs for building highly scalable and reliable distributed systems. Expect discussion of capacity planning, partitioning and sharding, caching and load balancing strategies, replication and consistency models, latency and throughput trade offs, fault tolerance, graceful degradation, redundancy, disaster recovery, monitoring and alerting, and postmortem culture. Candidates should reason about non functional requirements and propose architectures meeting targets for scale, performance, and operational resilience.

HardSystem Design

95 practiced

Design a multi-tenant platform service that prevents noisy neighbors from degrading other tenants. Propose isolation and enforcement mechanisms (CPU/memory/network quotas, request throttling, per-tenant queues), describe quota calculation methodology, and detail autoscaling and billing interactions to maintain fairness and predictability.

MediumSystem Design

83 practiced

For a multi-region service with RTO = 1 hour and RPO = 15 minutes, design a backup and disaster recovery approach. Discuss replication vs periodic snapshots, warm standby vs active-active, failover orchestration, and how you would validate that recovery objectives are met through drills and testing.

Sample Answer

**Clarify requirements**- RTO = 1 hour (max time to restore service), RPO = 15 minutes (max data loss).- Multi-region: assume primary region and at least one secondary region.

**High-level approach**- Use continuous replication for critical data + periodic snapshots for point-in-time retention.- Maintain a warm-standby cluster in secondary region for quick failover; active-active optional if cost and consistency requirements permit.

**Replication vs Snapshots**- Primary: synchronous or semi-synchronous replication for critical DBs where latency allows; otherwise async with WAL shipping and replication lag monitoring to meet 15-min RPO.- Snapshots: hourly incremental snapshots retained for compliance/longer-term recovery and for recovering from logical corruption (point-in-time restores).

**Warm-standby vs Active-active**- Warm-standby: provisioned compute and up-to-date replicas (read-only) in secondary; autoscale and DNS failover reduces RTO to <1h with automated promotion scripts.- Active-active: only if application supports multi-master and conflict resolution; lowers RTO further but increases complexity.

**Failover orchestration**- Orchestrate with runbooks + IaC (Terraform/Ansible) and an automation engine (Argo Workflows, AWS Step Functions).- Steps: detect -> quiesce writes -> promote replica -> update DNS/route53 + health checks -> scale services -> smoke tests.- Include health probes, circuit-breakers, and traffic shifting (canary failover) to avoid DNS TTL pitfalls.

**Validation and drills**- Automated unit recovery tests (CI): restore backups to ephemeral env nightly.- Quarterly full failover drills: simulate region outage, execute scripted failover, measure RTO/RPO.- Tabletop exercises monthly for runbook review.- Metrics to collect: time to detect, time to promote, data loss measured by binlog offsets, application-level success rate.- Post-mortem and improvement backlog after each drill.

This plan balances cost, complexity, and the stated RTO/RPO while providing operational discipline through automation and regular testing.

MediumTechnical

132 practiced

Explain strong consistency, eventual consistency, and causal consistency at the service level (not database internals). Give realistic service-level examples for each model, discuss latency and availability trade-offs, and explain when a systems engineer should prefer one model over another.

Sample Answer

**Strong consistency — definition & example**- Strong consistency: after a successful write, any subsequent read returns that write globally.- Service-level example: an account-balance microservice for bank transfers that uses a leader-coordinated write path and routes reads through the leader or a linearizable cache. After a transfer completes, any service reading the balance sees the new value.- Latency & availability: higher read/write latency (synchronous coordination), reduced availability under partitions (leader unavailable means operations stall).- When to prefer: financial, inventory-reservation, or configuration writes where correctness is mandatory.

**Eventual consistency — definition & example**- Eventual: writes propagate asynchronously; replicas converge eventually, so reads may return stale data short-term.- Example: user profile edits in a social network where updates are propagated via background replication and caches; showing old profile picture briefly is acceptable.- Latency & availability: low latency and high availability (continue serving reads/writes during partitions), but risk of stale reads and write conflicts.- When to prefer: high-throughput, user-facing features where availability and responsiveness matter more than immediate freshness.

**Causal consistency — definition & example**- Causal: preserves causally related orders (if A led to B, all nodes see A before B); unrelated writes can be seen in different orders.- Example: comment and reply service: if user posts a comment (A) and then replies (B), all readers will see the comment before the reply; independent comments may arrive in any order.- Latency & availability: stronger freshness than eventual with modest coordination (tracking causal dependencies), better availability than strong during partitions but higher metadata/latency than eventual.- When to prefer: collaboration, messaging, or social interactions where preserving user action order matters but global linearizability is unnecessary.

**Decision guidance**- Choose strong for correctness-critical operations.- Choose causal when user-perceived ordering matters but you need higher availability and lower latency than strong.- Choose eventual for max throughput and availability where brief staleness is acceptable.- Also consider operational costs: complexity of implementation, metadata overhead, and testing/observability needs.

MediumTechnical

95 practiced

A downstream payment provider becomes slow and intermittently times out, causing your checkout service to back up. Design a resilience strategy using timeouts, retries with backoff, circuit breakers, bulkheads, and fallback behavior. Explain the reasoning and ordering of these techniques.

HardSystem Design

82 practiced

Design an observability strategy (logs, metrics, traces) for an ecosystem of ~200 microservices to meet an MTTD target of less than 5 minutes and an MTTR target of less than 30 minutes. Address data aggregation, retention, sampling, alert fatigue reduction, and how to instrument new services consistently.

Sample Answer

**Goal & constraints**MTTD < 5 min, MTTR < 30 min for ~200 microservices. Prioritize signal-to-noise, fast detection, and quick root-cause workflows.

**High-level architecture**- Central telemetry pipeline: Fluentd/Vector → Kafka → processing layer (OpenTelemetry Collector) → storage backends (metrics: Prometheus+Thanos, traces: Jaeger/Tempo, logs: Loki/Elastic).- Single SSO-enabled observability UI that links traces ↔ logs ↔ metrics (Grafana + plugins).

**Data aggregation & routing**- Service agents emit OpenTelemetry: metrics (Prometheus exposition), traces (OTLP), structured JSON logs.- Collector performs enrichment, schema validation, tenant/cluster tagging, and routing to appropriate backends.- Use Kafka for buffering to absorb spikes and enable replay for debugging.

**Retention & tiering**- Hot tier: 30d full-resolution (metrics 15s, traces full), stored in Thanos/Loki.- Warm: 90d aggregated metrics (1m), sampled traces (1:10) for trend analysis.- Cold: 1y aggregated metrics only; logs indexed by policy (errors only).

**Sampling**- Adaptive trace sampling: tail-based (capture traces with errors/high latency) + probabilistic (1-5%) for baseline. Increase sampling on incidents via dynamic rules.- Metrics: high-cardinality labels trimmed at collector; cardinality guardrails and automatic rollups.

**Alerting & alert fatigue**- Alert rules co-located in GitOps repo, reviewed by SRE on-call policy.- Use multi-signal alerts: require metric threshold + correlated increase in error traces or logs before paging.- Escalation tiers: page critical (SLA breach), notify (Slack) for warnings. Implement alert deduplication and suppression windows.

**Instrumentation & consistency**- Provide company OpenTelemetry SDK wrapper with auto-instrumentation, standard resource attributes, semantic conventions, and helper libs for context propagation, error tagging, and business transaction names.- Templates and CI checks: linting for telemetry, unit tests that assert presence of metrics/traces/log correlation IDs; pre-deploy gate.

**Operational practices**- Playbooks linking alerts → runbooks → dashboards → jump-to-trace/log actions.- Run regular noise hunts, pager duty burn-downs, and capacity tests. Use SLOs to tune alert thresholds.

**Trade-offs**- Tail-based sampling increases compute but improves MTTD/MTTR; Kafka adds complexity but provides resilience. Retention tiering balances cost vs. forensic capability.

This design emphasizes fast, correlated detection and guided troubleshooting to meet the MTTD/MTTR targets while controlling cost and alert noise.

Unlock Full Question Bank

Get access to hundreds of System Design and Reliability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.