System Design and Scalability Questions

Covers architectural thinking and design tradeoffs for building reliable, high performance systems. Topics include design decision reasoning given constraints such as cost, latency and availability; scaling strategies including horizontal and vertical scaling, load balancing, caching patterns, database partitioning and sharding, read replicas, and asynchronous processing; capacity planning and observability; spotting and explaining bottlenecks such as hot partitions, single points of failure, database locks and network limits; and communicating technical impact in business terms. Candidates should be able to justify choices, compare alternatives, and articulate metrics and monitoring approaches to validate design decisions.

EasyTechnical

23 practiced

Explain the circuit breaker pattern and exponential backoff strategies. As a Cloud Engineer, describe how to apply them in a microservices environment to protect downstream services from overload, what configuration parameters you would expose (error thresholds, timeout windows, half-open behavior), and how to monitor and alert based on circuit state.

HardSystem Design

18 practiced

Design a high-throughput, fault-tolerant event delivery system that guarantees in-order delivery of events per user stream while partitioning across many nodes to scale. Discuss partitioning strategy, ensuring per-partition ordering, producer and consumer semantics (ack/commit), replay and reprocessing capabilities, and operational issues such as rebalancing and retention.

Sample Answer

**Clarify goals & constraints**- High throughput, per-user in-order delivery, partitioned across many nodes, fault-tolerant, supports replay/reprocessing, cloud-native (e.g., AWS/GCP).

**Partitioning strategy**- Partition by user-stream key (user_id or hashed user_id) so all events for a user map to one partition -> preserves ordering.- Use consistent hashing with virtual shards to scale partitions without remapping many keys.- Store mapping in a service discovery/config (or use Kafka partitioner/Kinesis shard keys).

**Per-partition ordering & replication**- Each partition has a single leader replica that serializes writes; followers replicate (sync or async depending SLA).- Use quorum-based replication (e.g., Kafka ISR, or Kinesis multi-AZ) to tolerate node failures.- Ensure single-writer semantics per partition; producers route to leader.

**Producer semantics**- Support idempotent producers (sequence numbers / producer IDs) to handle retries without duplicates.- Ack model: require leader commit + replication to majority before acknowledging for "durable" ack. Optionally allow lower acks for latency.

**Consumer semantics & commits**- Consumers track offsets per partition. Use at-least-once by default; support exactly-once semantics via transactional writes or idempotent consumers and deduplication.- Commit protocol: consumer commits offsets only after processing and checkpointing (store in durable service like Kafka __consumer_offsets or DynamoDB).

**Replay & reprocessing**- Retain append-only log per partition for configurable retention; enable long-term retention via tiered storage (S3).- Support replay by resetting consumer offsets or spawning reprocessing jobs reading from a timestamp/offset range.- Provide snapshots/checkpoints to accelerate reprocessing and maintain side-effect idempotency.

**Rebalancing & operational concerns**- Rebalance handled by consumer group coordinator with sticky assignment to minimize movement; use graceful handoff (drain in-flight messages).- When increasing partitions: use virtual shards to remap gradually; full re-sharding requires careful coordination and temporary routing layer mapping old->new.- Monitor lag, ISR size, broker health, disk, GC, network. Auto-scale shards/nodes based on throughput/lag.- Retention: balance business needs vs cost; use cold tier for long-term storage and compacted topics for latest state per key.- Security: IAM, encryption in transit/at rest, VPC, private endpoints.

**Trade-offs**- Strong ordering per user limits cross-user parallelism only if many users map to few partitions—use many partitions or virtual shards.- Strict durability (majority acks) increases latency; tune per SLA.

This design mirrors Kafka/Kinesis patterns implemented on cloud (MSK/Kinesis + S3 tiering + IAM + autoscaling) and provides ordered, fault-tolerant, replayable user-stream delivery.

MediumTechnical

24 practiced

Design an observability pipeline for a cloud-based microservices architecture to support capacity planning and real-time incident response. Include collection (logs, metrics, traces), sampling strategies, retention policies, alerting thresholds, dashboards, and cost-control measures. Recommend managed versus open-source components and explain trade-offs.

Sample Answer

**Situation & goals**Design an observability pipeline that supports real-time incident response and long-term capacity planning for cloud microservices (AWS/GCP/Azure).

**High-level architecture**- Collect: instrument services with OpenTelemetry for traces/metrics; structured JSON logs to stdout.- Ingress: Fluentd/Fluent Bit (agent) or managed log collector (CloudWatch Logs / GCP Logging).- Store: metrics to Prometheus (managed Thanos/Prometheus Operator) or CloudWatch Metrics / Managed Prometheus; traces to Jaeger/Tempo or AWS X-Ray; logs to Elasticsearch/OpenSearch or managed CloudWatch/GCP Logging / Elastic Cloud.- Visualization & alerting: Grafana (managed Grafana Cloud or self-hosted) + Alertmanager or cloud-native alerting.

**Collection & sampling**- Metrics: high-cardinality metrics exported at 10–60s resolution.- Traces: 100% capture for top-level (ingress/egress) spans; adaptive tail-sampling for detailed traces (e.g., 1–5% baseline, 100% for error/latency > p95).- Logs: ingest structured logs with severity tags; sample DEBUG logs at 1–5% in steady state; keep 100% for errors/warnings.

**Retention policies**- Metrics: hot storage (90 days at full resolution), downsampled mid-term (1 year at 5m resolution), aggregated long-term (3+ years) for capacity trends.- Traces: full-resolution 14–30 days; aggregated trace-derived metrics longer.- Logs: full-text searchable for 30 days; archived to cheaper object storage (S3/Blob/GS) for 1–3 years with lifecycle rules.

**Alerting thresholds & strategy**- Use multi-tier alerts: - P0 (pager): service unavailable, error rate > 5% and increased latency > 2x p95. - P1 (on-call): sustained CPU/RAM > 85% for 10m, queue backlog growth > 20%/5m. - P2 (ticket): capacity threshold forecasts: projected resource exhaustion in < 7 days.- Use anomaly detection (ML-based) for noisy signals; notify via PagerDuty/Teams/Slack.

**Dashboards**- Incident RT dashboards: service health, error rate, latency histogram, top traces, recent logs.- Capacity planning dashboards: cluster utilization, allocation vs request, growth trends, forecasted capacity gaps.- Maintain runbooks linked from dashboards.

**Cost-control measures**- Ingest controls: sampling, log-level filtering, route debug logs to cheaper buckets.- Compression and batching at collectors; set retention/lifecycle rules to move cold data to S3/Glacier.- Use managed services where operative cost < ops cost; reserve capacity or use committed-use discounts for heavy storage.

**Managed vs Open-source trade-offs**- Managed (CloudWatch, Managed Prometheus, Grafana Cloud, X-Ray): faster setup, SLA, integrated IAM/billing; higher recurring cost, less control over tuning.- OSS (Prometheus + Thanos, Grafana + Loki, Jaeger/Tempo, OpenSearch): lower licensing cost and full control, better customization, but higher ops burden (scaling, HA, upgrades).- Recommendation: hybrid — use OpenTelemetry + managed ingestion (for reliability) and managed metrics/tracing where available; for heavy log/query workloads consider OSS with object storage-backed long-term retention (Thanos/Loki + S3) to optimize cost.

**Why this design**- OpenTelemetry standardizes instrumentation across teams.- Sampling + retention balance real-time needs and long-term capacity analysis.- Hybrid managed/OSS gives rapid time-to-value while controlling long-term costs and flexibility.

EasyTechnical

22 practiced

You are given expected peak traffic of 10,000 requests per second with a p95 latency requirement of under 200ms for a stateless web API. As a Cloud Engineer, outline how you'd estimate capacity and choose instance types on AWS or GCP. Include steps for load testing, headroom calculations, autoscaling policy design, network considerations, and cost trade-offs.

Sample Answer

**Situation & goal**Estimate capacity to serve 10,000 RPS p95 < 200ms for a stateless HTTP API on AWS or GCP.

**1) Baseline measurement & per-instance throughput**- Run micro-benchmarks of the service (container or binary) on candidate instance types (e.g., AWS c6i/c6a, m6i; GCP n2/n2d/n1-standard) using realistic payloads and auth/DB mocks.- Use k6/Locust/JMeter behind a gateway to measure stable RPS at p95<200ms per instance.- Record CPU%, memory, network (Mbps), and open connections at target load.

**2) Headroom & capacity calc**- If one instance sustains X RPS at p95=200ms, add 30–50% headroom for traffic spikes and variability.- Required instances = ceil( 10,000 / (X * (1 - safety) ) ). Example: X=1,000 RPS → with 40% headroom need ceil(10000 / 600)=17 instances.

**3) Load testing plan**- Gradual ramp (10%→100%), spike tests, soak tests (several hours), chaos (terminate instances).- Use realistic traffic patterns, keep-alive, TLS overhead, and upstream latency.

**4) Autoscaling policy**- Prefer target tracking based on request latency or custom metrics (p95 latency from tracing/Prometheus) and fallback on CPU/network.- Configure scale-out: aggressive (scale when p95 >170ms or CPU >60% for 1–2 min). Scale-in: conservative (cooldown 5–10 min) to avoid flapping.- Min instances = capacity for baseline traffic (e.g., 30–50% of peak), max instances = capacity for DDoS/upper bound budget.

**5) Network & infra considerations**- Use ALB/NGINX/GCP Load Balancer with HTTP/2 and keep-alive.- Ensure subnet IP and connection limits, NAT Gateway bandwidth (or use private egress with ENIs) and allow sufficient ephemeral ports.- Use placement/zone spread and health checks for fast failover.

**6) Cost trade-offs**- Higher vCPU per instance reduces management overhead but may have higher wasted headroom. Use cost/RPS metric from tests.- Consider autoscaling groups with mixed instance types or spot/preemptible for non-critical capacity (with on-demand fallback).- Reserve savings plans or committed use discounts for baseline.

**7) Validation & monitoring**- Continuously monitor p95, error rate, 95th connection counts. Run periodic chaos and revisit instance sizing after code changes.

This approach balances empirical measurement, conservative headroom, responsive autoscaling, network limits, and cost-optimization.

HardTechnical

21 practiced

You observe that 0.1% of keys receive 50% of requests, causing hot partitions in your sharded datastore. As a Cloud Engineer propose strategies to detect and mitigate hot partitions with minimal disruption. Include hot-key splitting, rerouting, adaptive caching, sharding changes, consistent-hashing options, and operational monitoring needed to validate the mitigation.

Sample Answer

**Situation & goal**Detect and mitigate hot partitions causing 0.1% of keys to take 50% of traffic with minimal disruption, preserving availability and consistency.

**Detection / instrumentation**- Per-key telemetry: collect request counts, latency, error-rate and CPU/IO per shard. Use CloudWatch / Prometheus / Datadog with high-cardinality metrics sampled into histograms and top-k queries.- Alerts: top-10 key QPS > threshold, shard CPU > 80% with skew, tail latency increase.- Tracing: distributed traces to find client sources and request patterns.

**Mitigation strategies (ordered by risk)**1. Adaptive caching- Insert an edge / in-cluster cache (CloudFront, Redis cluster, memcached). Cache hot-key results with short TTL and write-through/invalidation to reduce read pressure.- Use local application cache for ultra-hot keys.

2. Rerouting / request-level split- Use reverse-proxy or service-mesh routing rules to send hot-key reads to specialized replicas or scaled worker pool. Implement feature-flagged routing so rollback is instant.

3. Hot-key splitting (logical)- Split a hot logical key into N subkeys (append suffix, hashed buckets) and update clients gradually via boolean flag or middleware. Use a proxy to translate transparently to avoid client changes.- For writes, implement aggregation on read or background compaction to maintain logical view.

4. Sharding changes (online)- Move affected key ranges to new shards with live resharding tools (DynamoDB auto-shard, Bigtable split, custom consistent rehash with streaming migration). Throttle movement to limit impact.

5. Consistent-hashing improvements- Add virtual nodes to smooth distribution.- Consider rendezvous hashing to reduce remapping when scaling.- Use weighted nodes if hardware differs.

**Operational validation**- Canary rollout and compare QPS/latency per-shard before/after.- Dashboards: per-key QPS, shard CPU/disk, latency P50/P99, cache hit-rate.- Post-mortem: verify 90/50/10 distribution shifts and validate no loss of consistency.- Automate rollback and warm-up scripts.

Reasoning: prefer low-risk layer (cache, routing) first to reduce load quickly; then persistently fix distribution with splitting or re-sharding.

Unlock Full Question Bank

Get access to hundreds of System Design and Scalability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.