Distributed Systems Design and Trade-offs Questions

Evaluate the candidate's ability to solve complex, multi-layered distributed-systems design problems by making reasonable assumptions, articulating trade-offs, and handling edge cases. Candidates should show how to decompose problems that span networking, caching, persistence, and performance optimization; select architectures and algorithms with explicit trade-off analysis (e.g. speed versus simplicity, consistency versus availability, synchronous versus asynchronous communication); and consider failure modes including network failures, device or resource limitations, and concurrent access patterns. Strong responses include clear assumption statements, alternative approaches, complexity and cost considerations, testing and validation strategies, and plans to monitor and mitigate operational risk (circuit breakers, rate limiting, backpressure, observability).

EasySystem Design

73 practiced

Design a simple REST API for fetching pre-aggregated metrics for dashboards with a freshness constraint of up to 1 minute and expected load of 1,000 RPS. Describe endpoints, caching strategy, TTLs, and how you would support queries for multiple time windows (e.g., last 1m, 1h, 24h). State assumptions about metric precomputation frequency.

Sample Answer

Requirements & assumptions:- Freshness ≤ 1 minute (acceptable up to 60s).- 1,000 RPS read load peak.- Metrics are pre-aggregated at fixed intervals (assume 1s raw ingestion, precompute aggregates every 30s and 60s).- Support windows: last 1m, 1h, 24h and arbitrary windows aligned to aggregation buckets.- SLA: p95 latency < 100ms.

High-level design:- Precompute service (Spark/Beam job or streaming job in Flink) maintains time-bucketed aggregates at multiple granularities: 1s (hot), 30s, 1m, 5m, 1h. Aggregates written to a time-series store (Redis TS / ClickHouse / Bigtable) and to a cache layer (Redis).- API layer (stateless, autoscaled behind LB) serves requests by reading cache first, falling back to DB.

Endpoints:- GET /metrics/{metric_name}?start=ISO&end=ISO&granularity=auto|1m|5m - Returns pre-aggregated values and metadata (bucket_size, computed_at).- GET /metrics/{metric_name}/summary?window=1m|1h|24h&agg=sum|avg|max|min

Caching strategy & TTLs:- Use Redis as L1 cache for API responses keyed by (metric, start, end, granularity).- Cache freshness policy: TTL = max(30s, bucket_size). For 1m window TTL = 30s (so cache is refreshed at least every 30s) to honor ≤60s staleness.- Also use short-lived per-bucket cache: store aggregate per (metric,bucket_start,bucket_size) with TTL = 3 * bucket_size to allow reuse across queries.- Read-through or write-through population from precompute jobs: when a new bucket completes, precompute job updates DB and invalidates or updates related Redis keys.

Handling multiple time windows:- Query planner in API chooses smallest set of precomputed buckets covering requested window (e.g., 1h = use 1m buckets; 24h = use 5m or 1h buckets depending on desired resolution).- For "auto" granularity choose highest resolution where number_of_points ≤ 1,000.

Scaling & performance:- 1,000 RPS served by Redis + time-series DB with connection pooling. Cache hit target ≥ 95% to keep latency low.- Precompute frequency: write aggregates every 30s for 1m resolution; hourly/day aggregates computed from lower-level buckets periodically (hourly, daily).

Failure & consistency:- Stale-by-design: document freshness ≤ 60s. On cache miss, API can return last-known aggregate and a computed_at timestamp.- Provide cache-bypass param (fresh=true) to force read-from-store for critical dashboards.

Metrics & monitoring:- Monitor cache hit ratio, API latency, precompute lag (time between bucket end and availability), and error rates. Automate alerts if precompute lag > 10s.

Trade-offs:- More frequent precompute increases freshness/cost. Chosen 30s/1m balances load vs freshness.- Using Redis for hot reads keeps low latency; time-series DB provides long-term storage and larger-window aggregation.

MediumSystem Design

80 practiced

You're designing dashboards for a high-cardinality user-facing data service. Explain strategies to control metric cardinality, sample traces, and still retain useful debugging information. Include ideas for aggregation, dynamic sampling, and tag bucketing.

Sample Answer

Requirements & constraints:- User-facing service with many dynamic dimensions (user_id, session_id, feature flags) — must avoid explosion of time-series and cost while preserving debuggability for incidents and analytics.

High-level approach:1. Enforce cardinality budget & tiering- Classify tags into high-cardinality (user_id, session_id), medium (region, plan), low (service, endpoint).- Allow only low+medium as first-class metric labels in dashboards; treat high-cardinality as searchable, not as time-series keys.- Set quotas per team/service and reject or coalesce metrics beyond budget at ingest.

2. Aggregation & rollups- Emit pre-aggregated metrics at ingest: counts, sums, latency histograms (ex: HDR or Prometheus histograms) keyed by low/medium tags.- Maintain rollups at multiple resolutions (1s/1m/1h) and time horizons (hot vs cold) to support both realtime and historical analysis.

3. Tag bucketing / normalization- Map high-cardinality values into buckets: user_id -> "top-1000", "power-user", "other"; numeric -> quantized ranges; geo -> country/region.- Use deterministic hashing for fingerprinting (store hash but not raw id) so you can group suspects while preserving privacy/compliance.

4. Sampling strategies for traces & events- Head-based sampling: always keep traces for errors and slow requests, sample successful fast requests.- Dynamic/adaptive sampling: increase sample rate during anomalies or for users in "investigation" cohorts; decrease during steady-state.- Tail and reservoir sampling: ensure rare/long-tail traces are captured; keep a bounded reservoir per service/endpoint keyed by fingerprint to maintain diversity.- Exemplar linking: attach exemplar trace IDs to aggregated metrics (e.g., latency histogram exemplar -> trace id) so dashboards can jump to representative traces without storing all traces.

5. Ingest-time enrichment & filtering- Compute derived attributes (is_internal_user, feature_flag_bucket) at ingestion to reduce downstream joins.- Drop or redact non-actionable tags (full URLs, UUIDs) or store them only in trace store, not in metrics DB.

6. Query & UX patterns- Provide pre-built dashboard dimensions (low/medium tags) and a "drill-to-trace" action using exemplar/tracing links.- Offer ad-hoc search over raw events/traces with rate-limited lookups for debugging.

Trade-offs and considerations:- Lossiness vs cost: bucketing/hash reduces resolution — mitigate by allowing temporary high-fidelity captures for incident windows.- Consistency: ensure same bucketing rules across producers to avoid split-series.- Privacy: hashing alone may be reversible for small sets; consider one-way hashing + salt or storing PII only in secure trace store.

Example flow:- Producers send metric + full tags to ingestion. Ingest pipeline: 1) Validate & apply tag whitelist/budget 2) Bucket high-cardinality tags, compute histograms, emit rollups 3) Head/tail sampler decides trace persistence and attaches exemplar IDs to metric aggregates 4) Short-term full-fidelity store for 7d, long-term rollups/cold storage thereafter

This design controls series cardinality, keeps dashboards responsive and cheap, yet preserves actionable debugging paths via exemplars, sampled traces, and temporary high-fidelity captures during incidents.

HardSystem Design

65 practiced

Design a GDPR-compliant deletion propagation system so a user's request to be deleted is honored across services, caches, logs, backups, and third-party exports. Describe data discovery, propagation channels, proofs/audits of deletion, and strategies for handling immutable backups and analytics aggregates without violating compliance.

Sample Answer

Requirements & constraints:- Functional: on user deletion request, remove or render inaccessible all personal data across services, caches, logs, backups, third-party exports, and downstream analytics.- Non-functional: auditable proof of deletion, bounded time SLA (e.g., 30 days), minimal downtime, explainability for regulators.

High-level architecture:- Central Deletion Orchestrator (DO): authenticated API that receives GDPR requests, validates identity, computes scope (IDs, PII types), issues deletion job.- Global Data Catalog / Index: metadata store mapping logical user identifiers to physical locations (datastores, topics, buckets, backups, exports) maintained by instrumented ingestion pipelines.- Service Adapters: per-service endpoints/consumers that apply deletion (soft/hard) and return attestations.- Message Bus (immutable audit log): ordered event stream (e.g., Kafka) for propagation; DO emits DeletionIntent -> ServiceAcknowledgement -> DeletionComplete events.- Audit Store: append-only ledger (WORM) that stores attestations and proofs, with signed receipts.- Monitoring & Retry Engine: tracks failures, exponential backoff, escalation.

Data discovery:- Instrument all ETL jobs, schemas, and connectors to register datasets and the columns that contain PII into the Global Data Catalog.- Periodic scan jobs: column-level scans and fingerprinting (hashes) to find residual references.- Tagging policy: propagate user_id tags through pipelines; lineage metadata enables mapping from source user to derived tables.

Propagation channels:- Primary: push via authenticated gRPC/REST to Service Adapters for immediate removal.- Secondary: consume DeletionIntent from Message Bus for services listening asynchronously.- For third-parties: deliver signed revocation notifications and require contractual attestations; track via Audit Store.

Proofs & audits:- Each Service Adapter returns a signed DeletionAck including: service id, affected partitions, timestamp, operation type (soft/hard), cryptographic hash of pre/post state metadata (not raw PII).- Store these in the Audit Store with an immutable index; support auditor queries and generate time-stamped deletion receipts for users.- Periodic reconciliation: run verification jobs that re-scan datasets for any reappearing identifiers; escalate mismatches.

Handling immutable backups:- Logical deletion markers: mark records as deleted in catalog and ensure restores respect deletion markers.- Backup lifecycle: shorten retention where feasible; for long-term immutable backups use cryptographic slicing—encrypt per-user keys and destroy keys for deleted users (crypto-erase).- Document exception policy and obtain lawful basis if retention is necessary (e.g., legal hold).

Analytics aggregates & derived data:- Favor privacy-preserving pipelines: store aggregates without direct identifiers; use differential privacy & k-anonymity thresholds.- For aggregates containing small cohorts: recompute aggregates excluding deleted user; if full recompute impractical, apply algorithmic suppression (remove or fuzz contributions) and log changes.- Maintain provenance to identify which aggregates include a user to trigger selective recomputation.

Trade-offs & notes:- Strong consistency vs availability: some services may only support eventual deletion—use SLA and clear user communication.- Crypto-erase is powerful but requires careful key management.- Governance: contracts with third-parties, SLAs, and regular audits are essential.

This design provides discoverability, reliable propagation channels, cryptographically verifiable attestations, and practical strategies for immutable backups and analytics while balancing cost and compliance.

EasyTechnical

65 practiced

For monitoring a distributed ETL service, explain the distinct roles of logs, metrics, and traces. Provide concrete examples of what each should capture for an ingestion job that processes per-file transforms (e.g., file received, parsing errors, duration, per-record failures) and how you would use them to investigate a spike in error rate.

Sample Answer

Logs, metrics, and traces each serve a distinct observability role:

- Logs (high-cardinality, detailed): capture per-event context and human-readable errors. For a file ingestion job log examples: - "FileReceived: file_id=..., size=..., source=..., timestamp" - "ParseError: file_id=..., line=123, record_id=..., error='invalid date'" - "RecordFailed: file_id=..., record_id=..., reason=validation_failed, payload_snippet=..." - "JobCompleted: file_id=..., status=FAILED, processed=1234, failed=12, duration_ms=..." Use logs to read exact stack traces, inspect offending record payloads, and correlate file IDs.

- Metrics (aggregated, numeric, time-series, for alerting): low-cardinality counters/gauges/histograms: - Counter: files_received_total, files_failed_total - Gauge: in_flight_jobs - Histogram: file_processing_duration_ms - Counter: records_processed_total, records_failed_total, parse_errors_total Use metrics for dashboards and alerts (e.g., sustained spike in records_failed_total per minute).

- Traces (distributed request-level timing and causality): show end-to-end flow across services for a single file or request. - Span for "IngestService.receive" → "Parser.parse" → "Transformer.transform" → "Writer.write" - Attach IDs: file_id, trace_id; add tags for status and error types Use traces to find where latency or failures cluster across components.

Investigating a spike in error rate:1. Alert from metric (records_failed_total spike) triggers investigation.2. Query logs filtered by time and file_id ranges to inspect error messages and sample offending records; identify common error types (e.g., parsing date format).3. Use traces for recent failed files to see which component fails most often and whether failures correlate with upstream latency or retries.4. Cross-reference metrics (e.g., increased upstream latency, memory pressure) to find systemic causes.5. Remediate (hotfix parser, backfill) and confirm via reduced metrics and absence of new error logs; add specific metric tags/alerts if new error class recurs.

HardSystem Design

71 practiced

Design a scalable metadata service for distributed job coordination that supports leader election, per-job metadata sharding, and low-latency reads. Discuss consensus choices (Raft vs Paxos vs managed services), how to avoid single-leader bottlenecks, backup/restore, and handling leader CPU/memory overload.

Sample Answer

Requirements:- Strong consistency for coordination operations (leader election, job state updates).- Per-job metadata sharded for scale; low-latency reads (near real-time).- High availability, horizontal scaling, operational simplicity for a data engineering org.

High-level design:- Shard metadata by job-id using a consistent hash ring. Each shard is a logical partition served by a small Raft group (3–5 nodes) elected per-shard leader. Clients locate shard leader via a lightweight routing layer (gossip + consistent hashing or a metadata router service).- For global coordination (e.g., cluster-wide leader election, config), use a separate small Raft cluster or a managed consensus (Cloud Spanner/etcd/Consul). Per-shard Raft gives strong consistency without a single global leader bottleneck.

Consensus choices:- Raft (self-managed): simple mental model, easier operational debugging, leader reads/writes via leader for linearizability. Good for per-shard groups.- Paxos: more complex; use only if integrating with existing Paxos infra.- Managed services (etcd/Consul, Cloud Spanner, Zookeeper): reduce ops burden. etcd is Raft-based and matches needs; Cloud Spanner gives global transactions but cost/complexity higher.

Avoiding single-leader bottlenecks:- Shard aggressively so no single leader handles all jobs.- Allow follower reads for low-latency non-strict reads with bounded staleness (lease-based reads or read-index in Raft). Use read-through caches (local LRU, Redis) for hot-job metadata.- Client-side partitioning + routing to spread load; autoscale shards (split hot shards).

Backup & restore:- Periodic snapshots + WAL shipping per Raft group. Store snapshots in durable blob storage (S3/GCS). For restore, replay snapshot then WALs to target generation. Provide cross-shard consistent export by coordinating at a global snapshot version (two-phase snapshot across shards using monotonic snapshot IDs).

Handling leader CPU/memory overload:- Monitor per-leader metrics (QPS, latency, CPU/memory). If overloaded: - Split shard (rehash subset job-ids) to create new Raft groups. - Offload read traffic via follower reads or cache layer. - Move leadership (Raft transfer) to a less-loaded node if node-level resource constrained. - Throttle clients or apply backpressure for heavy ops (rate limits, bulk API for batched writes).- For memory spikes, enforce per-shard limits, evict long-running in-memory structures to disk or external store (e.g., RocksDB).

Trade-offs:- More shards = more Raft groups = more overhead; tune shard count.- Strong consistency vs read latency: use follower reads/leases for lower latency at the cost of bounded staleness.- Managed service reduces ops but may limit sharding/custom behavior.

Operational notes:- Chaos testing (leader failover, network partitions), automated shard rebalancer, clear migration tooling, and runbooks for snapshot restore are essential.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Design and Trade-offs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.