Ability to decompose complex systems into components and define clear responsibilities, interfaces, and interactions. Evaluate architectural alternatives and articulate core trade offs such as consistency versus availability, latency versus throughput, simplicity versus extensibility, and cost versus performance. Explain how design choices affect scalability, resilience, failure modes, and operational burden, and justify architecture decisions based on expected load patterns and business requirements.
EasyTechnical
68 practiced
Define idempotency in the context of HTTP APIs. Design a POST /payments endpoint to be idempotent for retried client requests: specify what headers or body fields to require (e.g., Idempotency-Key), how to persist and lookup deduplication records, how long to retain dedup keys, and how to handle partial failures and race conditions.
Sample Answer
**Definition (short)** Idempotency for HTTP APIs means repeated identical requests have the same effect as a single request — the server must not create duplicate payments when a client retries.**Endpoint design: POST /payments** - Require header: Idempotency-Key (UUIDv4 suggested). Optional: Idempotency-Key-TTL in body for client hint. - Request body: payment details (amount, currency, payee), client-generated payment_reference.**Dedup record model** - Store entries: idempotency_key, request_fingerprint (hash of relevant body), status (IN_PROGRESS / COMPLETED / FAILED), result_payload (response or payment_id), created_at, last_updated. - Use a strongly consistent store (Postgres row with UNIQUE(idempotency_key) or DynamoDB conditional write).**Flow / lookup** 1. On request, start transaction: attempt insert idempotency_key with status=IN_PROGRESS; if UNIQUE violation, read existing record. 2. If record.status==COMPLETED return stored result (200/201). 3. If IN_PROGRESS, either wait with short backoff or return 409/202 indicating in-flight. 4. If FAILED or no record, process payment, then update record atomically to COMPLETED with result.**Retention** - Keep keys for 7–30 days (typical 7 days for payments), then garbage collect. Store enough to reconcile disputes longer as business requires.**Partial failures & race conditions** - Use DB transactions and conditional updates (compare-and-set) to move IN_PROGRESS→COMPLETED only once. - Ensure payment processor calls are idempotent or record external transaction id; if processor request times out, keep IN_PROGRESS until confirmed/finalized or mark FAILED after deadline and allow retry. - Record external_transaction_id to detect duplicates from downstream. - Log and expose reconciliation jobs to detect orphan IN_PROGRESS entries and retry or compensate.**Notes** - Validate Idempotency-Key uniqueness per customer/account to avoid cross-customer reuse. - Return clear status codes and Idempotency-Key in responses.
HardSystem Design
61 practiced
Design a messaging platform that guarantees at-least-once delivery, supports ordered delivery per user, and allows replaying message history. Describe partitioning strategy to preserve per-user ordering, offset and cursor management for consumers, deduplication mechanics for at-least-once semantics, retention and compaction policies, storage choices for scaling to billions of messages, and approaches for efficient fan-out to many subscribers.
Sample Answer
**Clarify requirements & constraints**- At-least-once delivery, per-user ordering, replay history, scale to billions of messages, many subscribers (fan-out), backend developer focus on pragmatic components.**High-level partitioning**- Partition by user_id (or conversation_id for multi-user threads) using consistent hashing so all messages for a user map to same partition. Use partition key = user_id mod N_shards; allow shard rebalancing with consistent-hash ring + shard owner metadata.**Storage & scaling**- Use append-only log storage (e.g., Kafka-like or cloud object-backed log) where each partition is an ordered topic. Persist segments to S3/Blob for long-term; keep recent segments on SSD for low-latency. For billions of messages: compact indices (time+offset) and segment-level metadata to seek.**Offsets & cursors**- Expose per-consumer-group offsets (offset = partition + sequence number). For user clients, cursor = (partition, offset). Support named cursors/checkpoints in durable store (DynamoDB/Postgres) for resumes and replay by setting offset to earlier value or time-based seek.**Deduplication**- Assign each message a client-generated or server-generated monotonic sequence id and globally unique message_id. Consumers detect duplicates by tracking last_processed_seq per partition; for cross-restart safe dedupe use compacted store keyed by message_id with TTL (or bloom filter + checkpointed watermark) to avoid reprocessing within retention window.**Retention & compaction**- Two policies: time-based retention for raw segments (e.g., 7–90 days) with cold archival to S3; log compaction keyed by message_id or user sequence to keep latest state for each key and enable long-term replay of latest states. Provide snapshot/export APIs for long-term compliance.**Fan-out strategies**- Push model: broker pushes to subscriber endpoints (WebSocket/APNs) using per-subscriber cursors and backpressure-aware queues; maintain per-subscriber checkpoint to allow retries.- Pull model: maintain consumer groups where each subscriber reads partition(s). For many subscribers, use a tiered fan-out: write-once log → streaming workers (stateless) that fan-out to subscriber queues (sharded) or use multicast via cache layer (Redis Streams) to avoid reading same log per subscriber.- Use batching, rate-limiting, and per-client backpressure windows; for heavy fan-out use async delivery with durable retry queues (SQS/Kafka topics per subscriber segment).**Trade-offs & reliability**- At-least-once with dedupe simplifies reliability at cost of idempotency logic. Per-user partitioning ensures ordering but limits cross-user batching. Archival lowers primary storage cost but increases seek latency for deep replays.I’d implement prototype with Kafka (partitioned topics), Kafka Connect to archive to S3, DynamoDB for cursors/dedupe state, and stateless fan-out workers behind an autoscaled service.
EasyTechnical
62 practiced
Define observability in distributed systems and name three concrete telemetry signals you would collect (metrics, logs, traces). Given a sudden increase in latency for the /checkout endpoint, describe a practical debugging workflow using those signals to identify whether the cause is application code, a downstream dependency, or infrastructure.
Sample Answer
**Definition (observability)** Observability is the ability to understand internal system state from external outputs (telemetry) so you can detect, localize, and fix issues in distributed systems. For a backend developer it means designing systems that emit rich, correlated signals to answer “what,” “where,” and “why” when something goes wrong.**Three concrete telemetry signals** - Metrics: request P95/P99 latency, error rate, throughput for /checkout. - Logs: structured request logs with request_id, user_id, stack traces, DB/HTTP call durations. - Traces: distributed traces showing spans for web handler, DB queries, downstream HTTP/gRPC calls.**Practical debugging workflow for sudden /checkout latency spike** 1. Check metrics: confirm spike in P95/P99 and whether throughput or error rate changed; compare hosts/regions. 2. Use traces: sample slow traces, inspect span durations to see which component (handler, DB, downstream service, auth) consumes time. Correlate trace IDs to logs. 3. Inspect logs for affected request_ids: look for exceptions, retries, circuit-breaker events, or slow SQL queries (include query text and timings). 4. Narrow: if traces show long DB spans → likely DB (application waiting); examine DB metrics (CPU, locks, query plans). If external HTTP spans slow → downstream dependency; check its health and retry/backoff. If spans show CPU waits or garbage collection on host, or many connection errors across services → infrastructure (CPU, network, autoscaling). 5. Mitigation: roll forward a quick fix (rate-limit, circuit-breaker, fallback), add alerting on root-cause metric, collect more traces/logs, and create a postmortem with concrete action items.
MediumSystem Design
66 practiced
Provide an architecture for a global, low-latency, read-heavy API serving content to users in 50+ countries with a p95 latency target under 100ms. Discuss multi-region deployment, data replication (async vs sync), read locality strategies, geo-routing (DNS, anycast, edge proxies), consistency model for writes, and trade-offs between single-leader vs multi-leader writes.
Sample Answer
**High-level approach**Design for read-locality: serve reads from the nearest edge/CDN or regional read replica; keep writes constrained to avoid cross-region sync latency. Use CDN + regional API edge proxies + per-region read replicas of the datastore.**Multi-region deployment**- Deploy API gateways and edge proxies (CloudFront/Cloudflare + regional Envoy/ALB) in each region.- Run stateless app servers in each region behind regional autoscaling groups.- Maintain per-region read replicas of the primary datastore (RDS read replicas, DynamoDB global tables, or MongoDB replica sets).**Data replication: async vs sync**- Use async cross-region replication for reads: low-latency reads, eventual consistency, minimal write latency impact.- Use synchronous replication only inside a region (for local strong consistency). Global sync replication would blow write latency and break the <100ms p95 target.**Read locality strategies**- CDN for static content and cacheable API responses (edge caching, TTL, stale-while-revalidate).- Regional read replicas for dynamic reads; route clients to nearest region.- Implement cache warming and short TTLs plus stale-while-revalidate to keep p95 low.**Geo-routing**- Use Anycast + global DNS (Route53 latency-based routing) or Cloudflare Anycast for directing users to nearest POP.- Edge proxies terminate TLS and forward to the nearest regional API cluster.- Health checks + weighted failover for outages.**Consistency model for writes**- Default: eventual consistency for reads (acceptable for read-heavy workloads).- Provide stronger guarantees where needed: read-your-writes/session affinity (cookie or token), or route write-read pairs to the same region or leader.- For critical operations, synchronous local commits and read-after-write from that region.**Single-leader vs multi-leader writes (trade-offs)**- Single-leader (global primary): easy consistency, simple conflict-free model, but global write latency can be high if leader is remote.- Multi-leader: writes accepted locally → very low write latency globally, better availability. But introduces conflict resolution complexity (need CRDTs, version vectors, or app-level merges) and operational complexity.- Recommendation: For most read-heavy APIs, use regional leaders with a single-writer per data partition (or single global leader for strongly consistent entities) and async cross-region replication. Offer multi-leader only for workloads that require low-latency multi-region writes and can tolerate or resolve conflicts.**Operational & monitoring**- SLOs: monitor p95 latency per region, replication lag, cache hit ratio.- Circuit breakers, retry/backoff, and fallbacks to serve slightly-stale data when necessary.Why this meets p95 <100ms: reads hit CDN/nearest read replica, avoiding cross-region round trips; writes avoid global sync; trade-offs are transparent to API consumers via session affinity and targeted consistency endpoints.
MediumSystem Design
57 practiced
You need to design an Orders microservice that must guarantee exactly-once processing of orders involving payment and inventory services. Compare three approaches: two-phase commit (2PC), idempotent operations with dedup keys + at-least-once messaging, and eventual compensating transactions (sagas). For each approach analyze latency, operational complexity, failure modes, and which business requirements push you to choose one over the others. Make a recommendation and justify it.
Sample Answer
**Clarifying assumptions**- Order processing involves: charge payment service, reserve/decrement inventory, persist order in Orders DB.- Exactly-once = business-visible effect occurs once: single charge, single inventory decrement, single persisted order.**Two-Phase Commit (2PC)**- Latency: high — blocking prepare/commit across services; synchronous round-trips.- Operational complexity: very high — requires XA/coordinator support, distributed transactions, careful DB/driver support.- Failure modes: coordinator crash, resource locking leads to timeouts and blocked resources; hard recovery.- When to choose: only if strict ACID across heterogeneous transactional stores is mandatory and you control all services and drivers (rare in microservices).**Idempotent ops + dedup keys + at-least-once messaging**- Latency: moderate — async messaging; can be near-real-time.- Operational complexity: medium — requires dedup store, idempotency keys, idempotent handlers, message broker.- Failure modes: duplicate attempts handled by idempotency; storage of dedup keys grows; partial failures if services aren't idempotent.- When to choose: when services can be made idempotent and business tolerates eventual ordering of side-effects; good balance for resiliency and simplicity.**Sagas (compensating transactions)**- Latency: low to moderate — orchestrated or choreographed async steps.- Operational complexity: medium-high — design compensating actions, ensure compensation correctness and idempotency.- Failure modes: compensation failing or non-reversible side-effects (refund delays), temporary inconsistency window, complex error paths.- When to choose: when operations are across bounded contexts and you can model compensations (e.g., refund payment, restock inventory). Best when eventual consistency is acceptable.**Recommendation**Use idempotent operations with dedup keys + at-least-once messaging as default: it offers strong operational resilience, relatively low latency, and simpler recovery. For flows where compensation is natural (e.g., refund + restock) combine with a Saga-style orchestrator to manage business logic and retries. Reserve 2PC only for rare cases where synchronous ACID is non-negotiable and all services support XA.
Unlock Full Question Bank
Get access to hundreds of System Architecture and Tradeoffs interview questions and detailed answers.
