Distributed Systems Fundamentals Questions

Core principles and theory that underlie distributed computing systems. Includes understanding trade offs between consistency, availability, and partition tolerance, common consistency models such as eventual and strong consistency, replication and sharding strategies, load balancing and data partitioning, consensus algorithms and their guarantees, scalability and fault tolerance patterns, and how these concepts apply to infrastructure components such as databases, caches, service meshes, and load balancers. Candidates are expected to explain design choices, common failure modes, and how fundamental concepts influence architecture decisions for resilient and scalable systems.

MediumTechnical

57 practiced

Explain 'eventual consistency with monotonic reads' guarantee. Provide an example system where monotonic reads are important and how you would implement monotonic reads for clients in a multi-region read-replica setup.

Sample Answer

Eventual consistency with monotonic reads means the system is eventually consistent (updates propagate asynchronously), but each client sees a non-decreasing sequence of versions: once a client reads version V of an object, any later read by the same client will never return a version older than V. It prevents “time travel” for a client even when replicas lag.

Example system where this matters:- A multi-region user profile or shopping-cart service accessed from multiple devices. A user updates their cart in one region, then switches devices or moves to another region—monotonic reads ensure they never see an older cart state after having observed the update.

How to implement monotonic reads in a multi-region read-replica setup1. Requirements & primitives- Leader or primary applies writes and assigns a monotonically increasing commit sequence number (CSN), logical timestamp, or HLC per object/partition.- Replicas apply updates asynchronously but expose their current applied CSN per partition.- Clients maintain a session state: last_seen_CSN for each partition they read/write.

2. Client protocol (session guarantee)- On each read, the client sends the last_seen_CSN for the target partition (could be per-user key).- A replica receiving the read compares its applied CSN for that partition to last_seen_CSN: - If applied_CSN >= last_seen_CSN: serve the read immediately (monotonic read satisfied). - Else: either a) Wait (block) until replica catches up to last_seen_CSN (use WAL tailing/replication progress) — lower staleness, higher latency. b) Redirect the read to the primary/closer up-to-date replica (sticky routing) — higher cross-region cost, lower latency if leader reachable. c) Return a “stale” marker and let client retry after fetching from leader (less user-friendly).- For writes, after successful commit, the client updates last_seen_CSN to the returned commit CSN.

3. Practical choices & optimizations- Use hybrid logical clocks (HLC) or monotonically increasing CSNs to avoid clock skew problems.- Track last_seen per logical partition (not globally) to keep metadata small.- Provide configurable policy per client: strict wait, redirect to leader, or best-effort (trades latency for availability).- Materialize fast replica catch-up via prioritized replication for “hot” sessions or use client-affinity (session stickiness) to a region for a session lifetime.

4. Trade-offs- Waiting or redirecting increases read latency or cross-region traffic.- More metadata (per-partition last_seen) for mobile clients; mitigate by coarse-grained keys or session timeouts.- Stronger guarantees than plain eventual consistency, but weaker than linearizability — appropriate when per-client monotonicity suffices.

This design meets the Solutions Architect goal: translates a business need (consistent user experience across devices/regions) into concrete session protocols, replication progress checking, and pragmatic trade-offs (latency vs. staleness vs. complexity).

HardSystem Design

57 practiced

Design an approach for safe schema evolution for a set of microservices that share events via a message bus. Requirements: zero-downtime deploys, backward and forward compatibility where possible, and a rollback plan if consumers break. Outline tooling and governance you would recommend to the customer.

Sample Answer

Requirements clarification:- Zero-downtime deploys- Backward and forward compatibility where possible- Safe rollback if consumers break- Operational visibility and governance

High-level approach:1. Schema format & registry- Use a binary/typed schema (Avro or Protobuf) for strict typing and compact messages; JSON Schema where human readability matters.- Use a centralized Schema Registry (Confluent Schema Registry or Apicurio) to store schemas, enforce compatibility rules, and serve schemas to producers/consumers.

2. Compatibility strategy- Enforce additive-only changes by default: adding optional fields with defaults, deprecating fields (not removing), avoiding renames without aliasing.- Use backward-compatible changes for consumers (producer can add fields); support forward compatibility via defaults and handling unknown fields.- For breaking changes, create a new schema version and new topic name (topic-v2) or use topic namespacing.

3. CI/CD and automated gates- Producer pipeline: compile schema, run unit tests, publish schema to registry with automated compatibility check (fail on incompatibility), run contract tests.- Consumer pipeline: run consumer-driven contract tests (Pact or custom test harness) against the new producer schema and run integration tests against a test bus.- Use branch-based deployments + CI gating: schema merge only after passing automated checks and review.

4. Deployment pattern- Canary and progressive rollout: deploy producer to canary subset, monitor consumer error rates and business metrics.- Support dual-writing/dual-read strategy for migrations: - Dual-write: producer writes both topic and topic-v2 until consumers migrate. - Consumer-side adapters: keep a transformation layer (Kafka Streams / KSQL / Lambda) to project v2->v1 for legacy consumers.

5. Rollback plan- If consumers break: - Stop producer writes to new schema (feature flag) or switch back to previous topic. - Use consumer-side backfill or republish from logs (Kafka) from previous schema to rehydrate. - If issues are consumer-specific, revert consumer deployment rather than producer when possible.- Maintain ability to replay data (retention or log compaction) and to re-publish transformed messages.

6. Observability and safety nets- Schema and compatibility dashboards (registry metrics)- Per-topic consumer lag, error rates, serdes exceptions, message schema id mismatches- Automated alerts (SLOs on message processing errors)- Dead-letter and poison-queue handling

7. Governance & process- Schema ownership: each bounded-context owns its schemas; register owners and reviewers.- Change process: lightweight RFC + schema PR that includes compatibility check results, migration plan, and rollout/rollback steps.- Automation-first governance: enforce compatibility checks in CI; require manual approval only for breaking changes.- Versioning policy doc (allowed changes, deprecation timeline, naming conventions).- Periodic audits and sandbox environments for cross-team integration tests.

Recommended tooling- Messaging: Kafka (or managed Kafka: Confluent Cloud/MSK)- Schema Registry: Confluent Schema Registry / Apicurio- Serialization: Avro or Protobuf (proto3) + codegen- Contract testing: Pact (HTTP) or custom schema-driven contract tests, Testcontainers + Kafka for integration- CI/CD: GitHub Actions / Jenkins with schema validation plugin- Feature flags: LaunchDarkly / internal flags for dual-write toggles- Observability: Prometheus + Grafana, Kafka Cruise Control, Sentry/Datadog for consumer errors

Trade-offs- Strict schema enforcement adds developer friction but prevents runtime failures.- Dual-writing increases complexity and storage costs but enables safe migrations and zero-downtime.- Creating new topics for breaking changes preserves safety at cost of consumer migration effort.

Summary:Enforce schema registry checks in CI, prefer additive changes, use versioned topics for breaking changes, adopt dual-write or transformation layers for migration, perform canary rollouts with strong monitoring, and govern via owner-driven RFCs + automated gates so zero-downtime and safe rollbacks are achievable.

MediumTechnical

71 practiced

Describe the transactional outbox pattern and how it helps bridge between a relational DB and an asynchronous message bus. As a solutions architect, provide an implementation checklist for production (DB schema changes, delivery worker, idempotency, monitoring).

Sample Answer

The transactional outbox pattern ensures reliable, atomic handoff between a relational DB and an async message bus by writing "events" into an outbox table inside the same DB transaction that mutates application state. Since the DB commit contains both state change and the outbox row, you avoid distributed transactions. A separate delivery worker reads committed outbox rows and publishes them to the message bus, then marks them delivered.

Implementation checklist for production

1) DB schema changes- Outbox table columns: id (UUID/PK), aggregate_id, event_type, payload (JSONB), created_at, published_at (nullable), status (pending/published/failed), attempts, next_try_at, dedup_key (nullable).- Indexes: created_at/next_try_at + status; dedup_key unique partial index (if used).- Consider partitioning/archival strategy (date-based) to limit table growth.- Enforce small payloads or reference external blob store for large payloads.

2) Delivery worker- Read-only: poll by (status = pending and next_try_at <= now) in consistent, bounded batches with FOR UPDATE SKIP LOCKED to allow concurrency.- Publish to bus transactionally where possible (produce and then update outbox row); update published_at/status/attempts in DB after successful publish.- Backoff strategy: exponential backoff, cap attempts, set next_try_at.- Support concurrency, sharding workers by mod(id) or lease rows to workers.

3) Idempotency & semantics- Design for at-least-once delivery: ensure consumers are idempotent.- Provide dedup_key in events and use broker-side deduplication (if available) or consumer-side idempotency store (dedup table with event id + ttl).- Include event version/sequence when ordering matters.- For exactly-once needs, evaluate transactional producers (Kafka transactions) + consumer idempotency.

4) Monitoring, observability, and operations- Metrics: queue depth (pending rows), publish rate, success/failure rate, attempts histogram, average lag (created_at → published_at).- Alerts: growing pending rows, repeated failures, delivery latency > SLO, high attempt rates.- Logs & traces: correlate DB transaction id / aggregate id / event id through distributed tracing.- Dead-letter handling: move rows exceeding max attempts to dead-letter table with failure reason, expose manual reprocess UI.- SLA/Runbook: escalation steps, reprocessing procedures, safe replay guidance.- Security: encrypt payloads at rest, limit DB permissions for delivery worker, authenticate to message bus.

Trade-offs and notes- Simpler and robust for eventual consistency; introduces small lag and operational complexity (cleanup/worker scale).- Use SKIP LOCKED for horizontal scaling; tune batch sizes and poll interval to balance latency vs DB load.- Test failure scenarios: DB commit success + broker outage, worker crash mid-publish, duplicate deliveries. Validate idempotency end-to-end.

This design yields durable, transactional handoff between relational state and async systems with clear operational controls for production.

HardSystem Design

85 practiced

How would you design rate limiting for a multi-tenant API gateway so that each tenant has independent quotas, and burst behavior is allowed per tenant but protected system-wide? Describe algorithms and data stores to implement this at scale and how to ensure correctness under partition.

Sample Answer

Requirements & constraints:- Per-tenant independent quotas (rate & burst), configurable per tenant.- Allow bursts up to tenant-specific limits while protecting overall system capacity.- Low latency at API gateway path, high throughput (100k+ rps), multi-region, resilient to partitions.- Strong-enough correctness: no tenant can exceed quota long-term; temporary slight overage acceptable during partitions with reconciliation.

High-level architecture:- Edge API Gateways (stateless) + local in-memory fast token-bucket caches → Regional rate-enforcement layer → Global quota store & policy service.- Policy store: authoritative tenant configs in a consistent DB (e.g., Spanner/CockroachDB) replicated across regions.- Fast counters & token buckets: Redis Cluster (sharded by tenant id) with Lua scripts for atomic token operations; fall back to local leaky bucket when Redis unavailable.- Telemetry & reconciliation: Kafka stream of enforcement events to offline/nearline processor that corrects long-term state and triggers throttles/alerts.

Algorithm & implementation:1. Token Bucket per tenant: capacity = burst_limit, refill_rate = steady_rate.2. At gateway: attempt atomic token consume via Redis script: - Read and update last_refill_ts, tokens atomically, return allow/reject + remaining tokens. - This keeps correctness and allows burst up to capacity.3. For scale, shard Redis by tenant id (consistent hashing) and autoscale shards.4. To protect system-wide capacity: a global admission controller keeps a regional headroom counter (coarse-grained) in Redis or use a centralized circuit-breaker service. Gateways also check region-level capacity before allowing large consumes.5. Partition handling: - If Redis for tenant unreachable: use local in-memory leaky bucket (smaller capacity than normal) to allow emergency ops but conservative to prevent system overload. - Emit events marking degraded enforcement; when connectivity restored, reconcile by comparing gateway-local consumption events with global counters and apply compensating throttles/credits. - For stronger correctness where overage is unacceptable, require synchronous RPC to authoritative store (higher latency) or use consensus (Raft) per shard — tradeoff performance vs strict correctness.

Data stores & consistency:- Authoritative tenant config: globally-consistent DB (Spanner/Cockroach).- Fast state & counters: Redis Cluster with persistence and replication; use Lua scripts for atomicity.- Long-term audit: write events to Kafka + cold store (Cassandra/S3) for billing and reconciliation.

Scalability & operational notes:- Shard by tenant id; hot-tenant mitigation via token-bucket split across multiple keys or rate-limiting by API key subsets.- Use metrics/alerts for hot-spots and autoscale Redis/gateways.- Test chaos scenarios (partition, failover) and validate reconciliation logic.

Trade-offs:- Redis+Lua = low latency and atomicity but eventual consistency across regions; partitions require conservative local fallback.- Strong consistency (consensus per shard) eliminates overages but increases latency and reduces throughput.

This design balances per-tenant flexible bursts with system protection, provides low-latency enforcement, scales horizontally, and contains mechanisms to handle and reconcile state under partitions.

EasyTechnical

62 practiced

What is idempotency in the context of distributed systems and APIs? Provide an example of an idempotent operation and outline three strategies a solutions architect can use to make non-idempotent operations safe to retry across a flaky network.

Unlock Full Question Bank

Get access to hundreds of Distributed Systems Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.