Data Consistency and Distributed Transactions Questions
In depth focus on data consistency models and practical approaches to maintaining correctness across distributed components. Covers strong consistency models including linearizability and serializability, causal consistency, eventual consistency, and the implications of each for replication, latency, and user experience. Discusses CAP theorem implications for consistency choices, idempotency, exactly once and at least once semantics, concurrency control and isolation levels, handling race conditions and conflict resolution, and concrete patterns for coordinating updates across services such as two phase commit, three phase commit, and the saga pattern with compensating transactions. Also includes operational challenges like retries, timeouts, ordering, clocks and monotonic timestamps, trade offs between throughput and consistency, and when eventual consistency is acceptable versus when strong consistency is required for correctness (for example financial systems versus social feeds).
MediumTechnical
57 practiced
Describe the transactional outbox pattern and explain why it helps achieve consistency between a service's database and its messaging system. Provide the sequence of steps (application transaction, outbox insert, poller publish, mark-published), explain failure modes (crash between commit and publish), and compare with CDC-based approaches (Debezium) including pros and cons.
Sample Answer
The transactional outbox pattern ensures atomicity between state changes in a service DB and the messages emitted to other systems by writing outbound events into the same transactional database as part of the business transaction, then reliably publishing them to the messaging system from that outbox table.Sequence of steps:1. Application transaction: perform business write(s) (e.g., update order) inside a DB transaction.2. Outbox insert: within the same transaction, insert an outbox row (event payload, headers, topic, status).3. Commit: commit the DB transaction — both state change and outbox row are durable together.4. Poller/publisher: a separate process (poller or daemon) periodically reads un-published outbox rows, publishes them to Kafka (or other broker).5. Mark-published: after successful publish (ack from broker), update outbox row as published (or delete it) in the DB.Why it helps consistency:- Because the outbox insert is in the same DB transaction as the state change, there is no window where the DB reflects the state change but no corresponding outbox row exists. This provides atomic “transactional write → eventual delivery” guarantee (effectively exactly-once semantics at the service boundary when combined with idempotent consumers).Failure modes and mitigations:- Crash after commit but before poller publishes: message not yet in broker, but outbox row persists; poller will pick it up later — eventual delivery guaranteed.- Poller publishes but crashes before mark-published: risk of duplicate publish. Mitigate via idempotent producer keys, transactional produce-with-commit (Kafka transactions), or deduplication in consumers.- Poller partial failure (only some rows published): use at-least-once semantics and idempotency; ensure poller updates mark-published inside a DB transaction for its bookkeeping where possible.- Outbox growth: implement retention/compaction, batch publish, and backpressure monitoring.CDC (Debezium) comparison:Pros of CDC:- Non-invasive to app logic: no code changes; reads DB transaction log and streams changes (low developer overhead).- High throughput and low latency in many setups; good for replicating entire tables.- Works well for multi-service reads of many tables.Cons of CDC:- Operational complexity: Debezium + Kafka Connect + connectors, schema evolution handling, snapshotting, and ordering concerns.- Ordering and atomic multi-table semantics: while CDC preserves commit order per-transaction log, correlating related events across tables may be tricky.- Exactly-once guarantees require careful tooling; handling of tombstones, schema changes, and filtering can be complex.- May expose internal schema/events you don’t want in downstream topics without transformation.Pros of Outbox:- Simple, explicit, and controlled by the service; stronger semantic mapping between business action and event.- Easier to guarantee atomicity between state and event since same DB transaction used.- Simpler to reason about for single-service message emission.Cons of Outbox:- Requires application changes to write outbox rows.- Poller becomes an operational component (scaling, idempotency, monitoring).- Extra writes and storage in primary DB; can increase DB load.Recommendation:- Use outbox when you need tight, explicit coupling of state change→event with minimal coupling to tooling and easy semantics for event generation.- Use CDC when you want low-effort broad table streaming, need to capture many tables without touching services, or for analytical replication — but plan for operational complexity, schema evolution, and ensure downstream idempotency/ordering strategies. Hybrid approach: outbox for critical domain events, CDC for broad data replication.
MediumSystem Design
29 practiced
Design a distributed transaction mechanism for a booking system that spans payment and reservation services. Requirements: avoid double-booking, ensure atomic transfer where both steps commit or the system compensates, support retries, and produce audit logs. Compare two-phase commit and saga patterns, pick one with justification, and outline coordinator behavior, timeouts, and recovery procedures.
Sample Answer
Clarify requirements- Functional: payment + reservation must appear atomic (both succeed or compensate), no double-booking, retries allowed, full audit trail.- Non‑functional: high availability, reasonable latency, eventual consistency acceptable, durable logs for audits.Choose pattern: Saga (orchestrated) — justification- Two‑Phase Commit (2PC) enforces strong atomicity but requires distributed locking/coordinator that blocks resources, is brittle across microservices, and hurts availability and throughput (bad for high-scale booking and payment flows).- Saga (orchestrated) provides non‑blocking, compensating transactions, better fault tolerance and scalability. With careful design (idempotency, reservation holds, short locks) it meets business needs while enabling reliable audit logs and retries.High-level flow (orchestrated saga coordinator)1. Client requests booking → coordinator creates saga instance (unique saga_id) and writes “Started” event to durable event store / audit log.2. Reserve seat: coordinator sends ReserveRequest(reservation_id, hold_until, saga_id) to Reservation service. - Reservation service places a short exclusive hold (soft lock) for hold_until, responds Reserved or Failed; writes event.3. On Reserved → coordinator sends ChargeRequest(payment_id, amount, saga_id) to Payment service. - Payment service authorizes and captures (or just authorize then capture) and writes event.4. If Charge succeeds → coordinator sends ConfirmReservation(reservation_id, saga_id); Reservation converts hold → confirm and writes event; saga completes with “Committed”.5. If any step fails → coordinator triggers compensating actions in reverse order (e.g., Refund/CancelCharge, ReleaseHold), logs each action, marks saga “Compensated” or “Failed”.Key components and responsibilities- Saga Coordinator (stateless but persists saga state): - Persist events/state to durable event store (append-only) for audit and recovery. - Enforce at-least-once message delivery with retries and exponential backoff. - Track timeouts, retries, and compensation triggers.- Reservation Service: - Implements holds (soft locks) with TTL and idempotent APIs: ReserveHold(id, client, hold_until), Confirm(id), Release(id). - Use DB-level optimistic locking or unique constraint to avoid double-booking.- Payment Service: - Supports idempotent operations: Authorize(id), Capture(id), Refund(id). - Persist payment events.- Event Store / Audit Log: - Immutable events (saga_id, step, status, timestamps, traces) for reproducibility and analytics. - Stream events to data pipelines (Kafka → data lake) for downstream reporting.Avoiding double-booking- Use short-duration holds at reservation step stored in the reservation DB (row-level lock or availability table with unique constraints).- Hold TTL must exceed expected payment time; coordinator extends hold proactively if payment is slow (heartbeats).- On Confirm, atomically flip hold→booking in DB using single transaction to prevent race conditions.Timeouts & retries- Each step has configurable timeout (e.g., Reserve: 5s, Payment: 30s). Coordinator retries idempotently (max N attempts) with backoff.- If timeout exceeded without success, coordinator triggers compensation.- Coordinator heartbeat to extend holds when payment is in-progress beyond initial TTL.Recovery procedures- Durable saga state in event store allows replay: on coordinator restart, read “incomplete” sagas and resume based on last persisted state.- Idempotent service endpoints prevent duplicate effects during retries.- Background reconciler job scans long-running or inconsistent sagas and enforces business policy (force-release holds, manual intervention alerts).- All compensations are recorded; unresolved sagas escalate to on-call with clear audit trail.Auditability & observability- Append-only saga events + service logs = complete audit trail (who, when, what, compensation reason).- Emit structured events to streaming platform (Kafka) and sink to data warehouse for analytics and compliance.- Expose metrics: sagas started/succeeded/compensated, avg latency, retry counts, hold expirations.Trade-offs & notes- Saga trades strict ACID for availability — acceptable if compensations are well-defined and fast.- If strict strong consistency is required across many heterogeneous stores (rare in this domain), 2PC could be considered but expect operational complexity and lower availability.This design meets requirements: avoids double-booking via holds/constraints, ensures atomic outcome via compensations, supports retries and recovery through persisted saga state and idempotent APIs, and provides full audit logs for downstream data pipelines and compliance.
HardSystem Design
28 practiced
Architect a multi-region analytics ingestion pipeline that must replicate writes from any of three regions to a central analytics store within 5 seconds and must avoid data loss. Detail ingestion buffers, cross-region replication strategies (fan-in via durable queues, CDC tailing), ordering and deduplication approaches, backpressure handling, and how you will monitor SLA compliance and test under region failure.
Sample Answer
Requirements & constraints- Functional: Accept writes in any of 3 regions, replicate to central analytics store within 5s, no data loss.- Non-functional: high availability, ordering where required, deduplication, observability, graceful backpressure.High-level architecture (text diagram)Region A/B/C write endpoints → local ingestion buffer (durable) → cross-region replication layer → central ingestion queue → central consumer/transform → analytics store (data lake/warehouse)Components and rationale1. Local ingestion buffers (per region)- Use durable append-only log (e.g., Kafka/Managed Kafka, Kinesis, Pub/Sub) per region to absorb spikes and persist writes immediately to disk.- Each event receives region-timestamp + GUID + causal metadata (optional logical clock/sequence per producer).2. Cross-region replicationTwo complementary strategies:A. Fan-in via durable queues (primary)- Mirror local topic to a central "ingest" topic using topic mirroring (Kafka MirrorMaker/Confluent Replicator or cloud cross-region replication). Replication is at-least-once and persistent, ensuring no data loss.- Replication pushes batches continuously; tune linger/flush to meet 5s end-to-end.B. CDC Tailing (optional for DB-origin writes)- For DB-origin data, use CDC (Debezium / cloud CDC) to tail binlog and stream into local topic; ensures low-latency capture of committed writes.Why both: CDC for database origins; queue mirroring for app events and uniform fan-in.3. Central ingestion & processing- Central durable queue receives mirrored partitions. Consumers (autoscaled) read, apply lightweight enrichment, dedup, and write to analytics store (e.g., S3 + Iceberg/Delta or BigQuery).- Use exactly-once semantics where possible: Kafka Streams/KS or Flink with checkpointing and idempotent sinks to avoid duplicates.Ordering and deduplication- Ordering: preserve partitioning keys (customer_id/time) end-to-end. Mirror partitions 1:1 to central to preserve intra-key order. For cross-region concurrent writes for same key, rely on event timestamps and sequence numbers; define conflict resolution (last-writer-wins by causal sequence) documented in schema.- Deduplication: include unique event_id per write. At central consumer use a bounded dedup cache (LRU/Redis with TTL >= maximum replication window, e.g., 1 hour) plus idempotent writes to the sink (e.g., upserts in Iceberg/Delta). For large scale, use stateful stream processors keyed by event_id or key+sequence and compacted storage.Backpressure and flow-control- Local buffers decouple producers from cross-region links. If cross-region link is congested: - Throttle producers at edge (rate-limit / token-bucket) with graceful 429/Retry-After. - Increase local retention and scaling of replication links. Use autoscaling replication workers. - Emit alerts when local retention approaching limit; provide emergency spill to cold storage.- For central ingestion overload, autoscale consumers and increase parallelism by re-partitioning keys if necessary (careful with ordering).SLA tuning to meet 5s- Budget: local write persist (<=50ms), replication propagation (<=2000ms typical; tune batch size/linger), central consumer lag processing (<=2500ms). Use pipelining, small batches, and parallel consumers.- Measure p99 latencies and tune batching and network paths (dedicated inter-region links or private VPC).Monitoring and SLA compliance- Instrument metrics: - End-to-end ingestion latency per-event (ingest time at region → write time central). - Replication lag (offset lag for each mirrored partition). - Delivery success rate, retry counts, local retention usage. - Error rates in consumers and sink commits.- Alerts: - Replication lag > threshold (e.g., >3s) per partition. - Local buffer consumption >80%. - End-to-end p95/p99 >5s.- Dashboards: Grafana with traces correlated to event_id; distributed tracing for timing breakdown.- SLOs: define SLO e.g., 99% of events delivered within 5s, 100% durability.Testing and fault-injection- Chaos tests: - Simulate region outage: kill Region B's replication workers and assert Region B's local buffer persists and replication resumes without loss; replay test to ensure idempotency. - Network partition: impose high-latency / dropped packets between regions to validate backpressure and retention safe-guards. - Scale failure: throttle central consumers to force backlog; verify local retention doesn't overflow and producers get throttled gracefully.- Load tests: - Synthetic event generator per region at 2–3x expected peak; measure latency and correctness. - End-to-end verification: inject events with known IDs and assert arrival order/dedup within 5s in central store.- Disaster recovery: - Periodic restore-from-backup tests and re-replication dry-run (replay topics) to validate no-loss replay.Trade-offs and alternatives- Synchronous cross-region commit would reduce complexity but increase write latency and reduce availability—avoided.- Using strong global ordering (single leader) simplifies ordering but is single-point and harms write-locality.- Choice of storage: choose file-based lake (Iceberg/Delta) for cost and analytics-read patterns, ensure upsert-friendly sinks.Key operational practices- Schema evolution governance, contract tests, and validators at ingestion.- Runbook for region failures: increase central consumer parallelism, expand retention, coordinate failover.- Regular audits of replication offsets and end-to-end tests.This design provides durable local buffering, low-latency cross-region replication, deterministic ordering per key, robust deduplication, backpressure handling, and thorough monitoring & testing to meet the 5s replication SLA with no data loss.
HardSystem Design
30 practiced
Architect a cross-service transactional system for moving money between accounts managed by separate services. Requirements: atomic transfer semantics (debit+credit or compensated rollback), durable audit trail for regulation, and ability to reconcile and prove conservation of funds. You cannot use a single distributed database. Propose an architecture (saga, 2PC, or hybrid), specify protocol steps, failure handling, idempotency guarantees, and reconciliation processes.
Sample Answer
Requirements clarification:- Atomic transfer semantics between AccountService-A and AccountService-B without a single DB.- Durable, regulatory audit trail and ability to reconcile/prove conservation of funds.- High throughput, eventual consistency allowed, but transfer must be atomic (either completed or compensated).High-level architecture (hybrid Saga + coordinator + durable ledger):- Coordinator Service (Transfer Orchestrator) — stateless, issues transfer requests and tracks state.- AccountService (debit side) & AccountService (credit side) — own their DBs; expose idempotent APIs and publish events.- Durable Ledger (append-only, immutable) — write-once audit store (e.g., cloud object store + signed manifests or an append-only log like Kafka with topic retention + ledger snapshots).- Message bus (Kafka/RabbitMQ) for reliable event delivery and CDC pipelines feeding data warehouse for reconciliation.- Optional: Lightweight 2PC-like locking for intra-data-center high-value transfers (hybrid).Protocol steps (Saga with coordinator):1. Client -> Coordinator: POST /transfers {transfer_id, from, to, amount}.2. Coordinator writes TransferIntent to Durable Ledger (immutable) synchronously (audit trail).3. Coordinator CALL -> FromAccount: /debit {transfer_id, amount}. FromAccount: - Validates idempotency by transfer_id. - If sufficient funds: create pending hold in local DB, emit DebitSucceeded event to bus and append DebitRecord to Durable Ledger. - If insufficient or error: emit DebitFailed, append to ledger.4. Coordinator on DebitSucceeded -> CALL -> ToAccount: /credit {transfer_id, amount}. ToAccount: - Idempotent: if transfer_id seen, no double-credit. - Create credit record, append CreditRecord to ledger, emit CreditSucceeded.5. Coordinator on CreditSucceeded -> send Commit to FromAccount to remove hold (finalize debit). Append TransferCompleted to ledger. Respond to client success.6. If any step fails (DebitFailed or CreditFailed/timeouts), Coordinator triggers compensating actions: - If debit succeeded but credit failed: Coordinator calls FromAccount /compensateCreditFailed {transfer_id} which releases the hold and records CompensationSucceeded. Append CompensationRecord to ledger. - If debit failed: mark TransferFailed in ledger, no further action.Failure handling & guarantees:- Durable Ledger acts as source-of-truth for all intents/steps. Every command and outcome is appended synchronously (or synchronously with strong consistency) to guarantee auditability.- Idempotency: All account operations accept transfer_id and store a transfer record with status. Repeated messages are no-ops returning the stored result.- Exactly-once semantics per service achieved via idempotent APIs + local transactions (create transfer-record + update balance in same DB transaction).- Coordinator is stateless; its progress is reconstructed by replaying Durable Ledger events (or reading TransferIntent status). Coordinator retries are safe.- Network partitions/timeouts: use exponential backoff + persistent retry queue; after N attempts escalate to human/operator with ledger state.Hybrid 2PC option (for very high-value transfers where strong consistency required):- Use a lightweight prepare/commit between services using a distributed lock service (e.g., Zookeeper/etcd) and two-phase commit only for flagged transfers. Prepare: each AccountService reserves funds and writes a PREPARED entry in its DB and ledger; commit: both commit. Trade-off: blocks resources and reduced throughput; use sparingly.Reconciliation & proof of conservation:- CDC pipeline (Debezium or DB WAL) streams all account DB mutations + ledger events into a data lake / analytical warehouse.- Periodic reconciliation jobs compute: - Sum of all beginning balances + net of all appended ledger credits/debits/compensations = current balances by account. - Invariant check: total system debits == total system credits + net external inflows/outflows. Any mismatch is an exception.- Reconciliation process: 1. Consumer job reads Durable Ledger (canonical) and account snapshots. 2. Recompute per-account final balances from ledger-only projection; compare to operational DB balances. 3. Produce reconciliation report with mismatches, root-cause via transfer_id traceability.- Maintain immutable snapshots and Merkle-tree hashes of ledger segments for tamper-proof proofs; regulators can verify segments and signatures.Operational concerns:- Monitoring: SLOs for step latency, retry counts, orphan holds, reconciliation drift. Alerts for compensations or persistent pending transfers.- Security: sign ledger entries, encrypt PII, RBAC for manual compensation.- Performance: partition ledger by date and account ranges; use batching for high-throughput.- Edge cases: duplicate transfer requests (idempotency handles), partial failures during compensation (compensation retried until ledger shows CompensationSucceeded), long-lived holds (timeout policy + automated compensation + alert).Why this design:- Saga + durable, append-only ledger provides atomic transfer semantics at application level without one DB.- Durable Ledger gives regulatory audit trail and canonical event stream for reconciliation.- Idempotent service APIs, coordinator statelessness, and CDC-based reconciliation enable provable conservation of funds and operational recoverability.
MediumTechnical
39 practiced
Explain how Kafka's transactional API and idempotent producers work together to achieve exactly-once semantics for producer→Kafka→consumer cycles. Cover transactional.id configuration, producer fencing, transaction begin/commit/abort, consumer read_committed mode, and limitations when integrating transactional writes with external datastores.
Sample Answer
Start with the idempotent producer: when producer.enable.idempotence=true Kafka assigns a producerId (PID) and sequence numbers per partition. This guarantees each produce request (per partition + PID + seq) is applied exactly once on the broker leader even if retries occur — preventing duplicates from network retries.Transactional API builds on idempotence to achieve end-to-end exactly-once for multi-partition writes and for atomically committing produced records together with consumer offsets. Key pieces:- transactional.id: a stable identifier the client sets. It ties application state to a transactional producer instance across restarts. Only one active producer can use a transactional.id at a time.- Producer fencing: when a new producer with the same transactional.id starts, it gets a new PID; brokers fence (invalidate) the old PID so any in-flight/uncommitted writes from the previous instance are rejected, preventing split-brain duplicates.- Transaction lifecycle: initTransactions(), beginTransaction(), send() (to topics/partitions), optionally sendOffsetsToTransaction() to include consumer offsets, then commitTransaction() or abortTransaction(). commitTransaction() makes the batch visible to consumers in read_committed mode atomically; abortTransaction() discards them.- Consumer read_committed: consumers set isolation.level=read_committed to see only committed transactional records; read_uncommitted will see all records. For offset management, using sendOffsetsToTransaction ensures offsets are committed atomically with produced data so downstream reprocessing won’t double-process.- Limitations with external datastores: Kafka transactions only cover Kafka. If your pipeline writes to an external DB or S3 within the same logical transaction, you cannot get true distributed XA transactions. Common patterns: - Two-phase commit via outbox table in Kafka topic or DB, then relay to external store (idempotent consumers or CDC). - Use idempotent writes on the datastore (upserts, dedupe keys) and idempotent consumers. - Accept at-least-once to external systems and design for deduplication.Summary: idempotent producers + transactional API + producer fencing + read_committed consumer = atomic, duplicate-free producer→Kafka→consumer cycles within Kafka boundaries; integrating external systems requires additional patterns for idempotency or eventual reconciliation.
Unlock Full Question Bank
Get access to hundreds of Data Consistency and Distributed Transactions interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.