Large Scale System Architecture and Evolution Questions

Design and evolution of architectures to support massive user bases large data volumes and very high request rates. Topics include global distribution strategies such as geographic partitioning and multi region replication; high throughput low latency design choices including careful partitioning efficient data pipelines and edge caching; storage and data lifecycle strategies for petabyte scale including tiered storage and efficient compaction; federation and aggregation patterns for global services; migration strategies for rewarding systems and rolling upgrades; and operational concerns for large fleets including monitoring alerting incident response and cost management. Interviewers assess the candidate on ability to reason about long term maintainability operational scaling and trade offs required to run systems at extreme scale.

MediumSystem Design

28 practiced

Design a global read-write architecture for a social timeline service: 200M users, peak 500k RPS reads, 50k RPS writes, read latency target <200ms for most users, and eventual consistency acceptable within ~2 seconds. Describe replication, write routing, fan-out-on-write vs fan-out-on-read trade-offs, conflict handling, and how to scale read-heavy workloads.

Sample Answer

Requirements recap: global active user base (200M), peak reads 500k RPS, writes 50k RPS, read latency <200ms for most users, eventual consistency acceptable (~2s).

High-level approach:- Active‑active multi‑region deployment (regions near users). Each region serves reads locally; writes are accepted in the nearest region and asynchronously replicated to other regions to meet the 2s eventual consistency SLA.- Partitioning: shard by user-id (consistent hashing) so each user's write/timeline lives on a small set of timeline shards. Use consistent shard mapping with capacity-aware rebalancing.

Write routing & replication:- Client writes go to nearest region API gateway → local write service → persist to local primary storage (append-only timeline log + follower list store).- Durable local commit returns fast; write is appended to a per-user write-ahead log and assigned a causal timestamp (HLC or Lamport + server id) for ordering.- Asynchronous cross-region replication via a durable change stream (Kafka/CPS) with per-shard topics; consumers in remote regions apply entries to regional materialized views. Replication uses batching and parallelism to meet <2s propagation.

Fan-out strategies (trade-offs and recommended hybrid):- Fan-out-on-write (push): - On write, push update into all followers' inboxes (materialized timelines). Pros: fast reads (single local lookup). Cons: heavy at write-time for users with millions of followers (hot producers), storage amplification.- Fan-out-on-read (pull): - Store only per-author activity; on read, merge recent activities from followed authors. Pros: cheap writes, low storage. Cons: higher read latency and compute; worst when user follows many people.- Hybrid (recommended): - For most users, use fan-out-on-write to maintain low read latency: push to follower inboxes stored in Redis/SSD. For celebrity/high-fanout writers, use fan-out-on-read or background sharded fan-out jobs that async materialize the top N items; keep a merged “fallback” read path that merges per-author feeds on demand. - Use threshold-based policy: if follower count < X, push; else enqueue background fan-out job and mark followers as served by pull path.

Conflict handling and ordering:- Use HLC timestamps + server ID to produce total order per user; attach causal metadata to each event.- For conflicting edits (rare): define operations as immutable posts; for edits/deletes use last-write-wins by HLC plus tombstones. For stronger semantics (e.g., edits merged), apply OT/CRDT for comment counters or reactions.- Ensure idempotency via unique client-generated IDs and dedupe during replication.

Scaling read-heavy workloads:- Materialized per-user inbox stored in fast key-value stores (Redis for hot, SSD-backed store for cold).- Multi-layer caching: CDN for static assets, edge caches for popular feeds, regional Redis LRU caches for hot timelines, per-request in-memory merge for tails.- Read replicas: each region has many read replicas for timeline storage; serve reads from nearest replica.- Pre-compute top-K timelines and “For You” ranking using streaming jobs (Flink/Spark Streaming), maintain ranked caches.- Autoscale consumers/producers for replication pipeline. Backpressure: if replication falls behind, temporarily serve slightly-stale data (within SLA).- Monitoring and SLOs: replication lag, tail latency, error rates; circuit for heavy writes to fallback to pull-mode.

Operational notes & trade-offs:- Prioritize read latency by materializing feeds; accept storage cost and complexity.- Use per-user sharding and capacity-aware placement to avoid hotspots; handle celebrity fan-out with hybrid approach.- Keep write path local and fast; rely on reliable async replication to meet the 2s eventual consistency goal.- Security, compliance, and data residency: place user PII where required and ensure cross-region replication respects policies.

This design balances sub-200ms reads for most users, accepts ~2s eventual consistency, and scales to 500k RPS reads by combining regional materialized timelines, multi-layer caching, and hybrid fan-out.

EasyTechnical

32 practiced

An enterprise wants to manage petabyte-scale logs and telemetry while minimizing cost and meeting compliance. Propose a high-level tiered data lifecycle (hot/warm/cold/archive), retention windows, deletion/aggregation patterns, and search/restore expectations that balance queryability, cost, and regulatory holds.

Sample Answer

Requirements (assumptions): petabyte-scale logs/telemetry, cost-sensitive, must support fast recent queries, ad-hoc historical search on subsets, and regulatory/legal holds.

Proposed tiered lifecycle:- Hot (0–7 days): SSD-backed index store (Elasticsearch/Opensearch or cloud-managed equivalent). Full raw events retained. Low latency for dashboards/alerts. Replication level for HA.- Warm (7–90 days): Cold-able nodes or cheaper NVMe/HDD with compressed indexed data; retain full events but use coarser indexing (time-based indices, fewer replicas). Still queryable with slightly higher latency.- Cold (90 days–2 years): Columnar object storage with partial indexes/metadata (e.g., Parquet in S3 + secondary index of keys/timestamps). Keep aggregated summaries and sampled raw data (e.g., 1–5% sampling) plus retained full raw for high-value sources only.- Archive (>2 years): Immutable, low-cost object archive (Glacier/Deep Archive) containing compressed raw files and catalog metadata. Query impossible without restore; restores are batch and take hours–days.

Retention, deletion & aggregation:- Default deletion: hard-delete raw after configured retention per data class (e.g., telemetry 2y, security logs 7y). Implement automated lifecycle policies.- Aggregation: after warm window, precompute rollups (hourly/daily) for metrics and keep alongside metadata. Use downsampling and deduplication for low-importance sources.- Legal/Compliance holds: apply hold flags at dataset or ID level to suspend lifecycle policies; maintain immutable copies (WORM) where required.

Search & restore expectations:- Hot: <1s queries; Warm: 1–5s; Cold: 10s–minutes via object store queries using maintained indexes; Archive: restore required (hours–days).- Provide a fast path: maintain lightweight time+ID index for archived objects to enable targeted restores rather than full restores.

Operational considerations & trade-offs:- Cost vs queryability: more indexing increases cost; offset by selective full retention (critical sources) and aggressive sampling elsewhere.- Security/compliance: encryption at rest, access logging, audit trails, immutability for regulated datasets.- Automation: lifecycle policies, monitoring of storage costs, retention attestations, and regular validation of restores.

This balances low-latency needs for recent data, cost-effective long-term storage, and compliance flexibility.

MediumSystem Design

48 practiced

Design a global search API that must query multiple regional indices and aggregate results within a 300ms 95th-percentile latency budget. Explain orchestration of parallel queries, scoring normalization across regions, early-return strategies, timeout handling, and how to serve useful partial results when some regions are slow or down.

Sample Answer

Requirements & constraints:- Global query fan‑out to regional indices, 95th‑percentile latency ≤300ms, return ranked top-K, tolerate regional slowdowns or failures, consistent UX (relevance across regions).

High-level architecture:- API Gateway → Orchestrator (stateless) → Regional Query Workers (near regions) → Aggregator & Normalizer → Cache / Result Store → Client.

Orchestration of parallel queries:- Orchestrator fans out async HTTP/gRPC requests in parallel to regional endpoints with per-region concurrency pools. Use pinned connections and protobuf for low overhead. Collect responses via futures/promises and merge.

Scoring normalization:- Each region returns raw scores + metadata (doc-level features, model version, sample score distribution histogram). Aggregator applies normalization: map raw scores to a global scale using per-region z-score or percentile calibration (compute mean/std or quantile mapping from histogram). Optionally run a small learned combiner model that ingests region score + signals (region latency, freshness) to produce global score.

Early-return and timeouts:- Global deadline = request deadline minus buffer (e.g., 260ms). Per-region timeouts derived from historical latency SLAs and remaining budget. Use adaptive early-return: if top-K stabilized (no incoming response can outrank current results with statistical confidence), return immediately.

Timeout handling & partial results:- If region times out or fails, return partial results with provenance (which regions included), best-effort estimate of missing contribution, and graceful degradation: (1) indicate that results are partial, (2) attach confidence score and retriable token for client to request updated results later or subscribe to updates.

Additional considerations:- Progressive / streaming responses: stream initial top results quickly and patch with late arrivals. Cache hot queries and use replicas for high availability. Monitor per-region drift and periodically re-calibrate normalization. Trade-offs: heavier normalization model improves quality but increases latency; prefer lightweight calibration in the hot path and offline/realtime re-ranking for final quality.

MediumTechnical

31 practiced

You own APIs used by thousands of clients across regions. Propose an API versioning and deprecation strategy that minimizes client breakage and supports continuous improvement: describe versioning scheme, deprecation windows, monitoring for deprecated usage, and mechanisms for negotiating contract changes (feature flags, client adapters).

Sample Answer

Versioning scheme- Use semantic, resource-level versions: major.minor.patch (v1.2.3). Only increment major when you introduce breaking changes. Expose version via URL (https://api.example.com/v1/...), and via Accept/Prefer headers for clients that prefer header negotiation.- Support backward-compatibility by favoring additive changes (new endpoints/fields) and defaulting new fields to safe values.

Deprecation windows- Breaking major version: announce at least 6–12 months depending on customer SLA & region; provide milestone schedule: announcement → soft-deprecation → hard-removal.- Minor/patch: no forced breaking changes; if remove deprecated fields, give 90–180 days.- Publish changelogs, migration guides, and per-client timelines for enterprise customers.

Monitoring deprecated usage- Emit structured deprecation warnings in responses (Warning header + X-Deprecation-Date). Instrument gateway to log deprecated-API and deprecated-field usage.- Create dashboards (per-region, per-client, per-endpoint) showing call volume, error rates, and time-to-migrate. Alert if high-volume clients remain on deprecated endpoints approaching removal.

Negotiating contract changes- Use feature flags to roll out behavior changes per client/tenant; default to opt-in for risky changes.- Provide client adapters / SDK shims that translate vN → vN+1 for common languages, and a compatibility layer in the API gateway for legacy clients.- For major customers, propose joint migration plans: committed timelines, test environments, and rollback playbooks.- Offer a “compat mode” header allowing clients to opt for legacy semantics temporarily.

Operational practices & governance- Service contract registry + automated contract tests (consumer-driven contracts) in CI/CD to catch breakages early.- Review board for breaking changes including product, legal, security, and major customer reps.- Communication cadence: announcement email, targeted outreach, migration webinars, and a public deprecation calendar.

This approach minimizes breakage by defaulting to backward-compatible changes, gives measurable migration windows, provides visibility into who is impacted, and offers technical mechanisms (feature flags, adapters, SDKs) to negotiate transitions with clients.

MediumTechnical

33 practiced

Your service partitions data by user-id and periodically experiences 'hot keys'—individual users generating 10x normal traffic (celebrity events). Describe automated and architectural mitigation strategies: dynamic re-sharding, per-key throttles, dedicated caches, write aggregation, or temporary routing to dedicated resources.

Sample Answer

Situation: A partitioned user-id-based service occasionally sees “hot keys” (single users 10x normal traffic) during celebrity events, causing tail latency, backend overload, and uneven resource usage.

Approach: Use layered, automated defenses plus architectural patterns so hot keys are localized, controllable, and routed to appropriate capacity without global disruption.

Automated & architectural strategies:- Dynamic re-sharding: detect sustained key-frequency > threshold and migrate that key’s shard to a new or underutilized node (or split shard). Automate via metrics pipeline (per-key QPS, latency) and orchestration (k8s job or controller). Trade-off: migration cost and complexity; use for multi-minute-plus events.- Per-key throttles & contract limits: enforce soft/hard QPS caps per key at edge/gateway with token bucket + backpressure signals to clients. Provide tiered SLA options (paid burst capacity). Advantage: immediate protection with predictable behavior.- Dedicated caches (hot-key cache tier): route detected hot keys to an in-memory cache cluster or edge cache (Redis cluster with dedicated shards). Cache reads heavily; invalidate or write-through on updates. Low latency, cheap for read-heavy bursts.- Write aggregation / batching: buffer small writes for hot keys and coalesce periodic flushes (per-key queues). Reduces write amplification and DB pressure; acceptable if slightly relaxed consistency is okay.- Temporary routing to dedicated resources: spin up ephemeral workers or separate instances (autoscale with pre-warmed images) and route hot-key traffic there via feature-flagged routing or service mesh. Use short-lived dedicated buckets for extremely hot events.- Observability + automation: real-time detection (Prometheus + alerting), playbooks for thresholds, and automated remediation pipelines (detect → isolate → route/migrate → notify).

Recommendation: combine edge throttles + dedicated caching for immediate protection; dynamic re-sharding or temporary routing for sustained hotspots; write aggregation when consistency tolerance exists. Always include instrumentation, SLA-aware throttling, and cost/complexity review before applying each mitigation.

Unlock Full Question Bank

Get access to hundreds of Large Scale System Architecture and Evolution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.