Event-Driven Architecture and Asynchronous Messaging Questions
Designing systems around events and message passing: publish/subscribe, message queues, event streaming, choreography versus orchestration, and decoupling producers from consumers. Covers delivery semantics (at-least-once, at-most-once), ordering, backpressure, dead-letter handling, and the operational tradeoffs of asynchronous flows. Includes async processing patterns for offloading slow work.
Explain Command Query Responsibility Segregation (CQRS). As a data engineer, when is CQRS valuable for analytics or operational workloads? Discuss trade-offs including complexity, eventual consistency of read-models, and strategies to make reads 'fresh' when required.
Sample Answer
Direct answer
Command Query Responsibility Segregation (CQRS) is the pattern of using a different model for writes (commands that change state) than for reads (queries that return state), instead of forcing one schema to serve both well. As a data engineer, it earns its added machinery when the write side's transactional shape and the read side's analytical or lookup shape diverge enough that one schema serves neither well, for example narrow row-level operational writes versus wide, pre-aggregated reporting reads. Treat it as a deliberate trade: schema simplicity for the ability to scale, model, and store reads and writes independently.
Structured elaboration
What CQRS actually separates
- Command side: an authoritative model (a normalized transactional store, or an event log) that enforces write-time invariants and produces state changes.
- Query side: one or more purpose-built read models (denormalized tables, search indexes, in-memory caches), each shaped for a specific access pattern rather than for correctness enforcement.
- A projector connects the two asynchronously: it consumes the write side's changes (domain events, or change-data-capture (CDC) records) and updates the read model(s).
When CQRS is valuable for analytics workloads
- Reporting or business intelligence (BI) queries need aggregation shapes (rollups by category, hour, region) that would otherwise require expensive joins or full scans against the transactional schema.
- Several independent consumers need different projections of the same data (a finance rollup, a fraud-detection view, a customer dashboard); three denormalized read models are cheaper to operate than three sets of ad-hoc joins against the online transaction processing (OLTP) store.
- Analytical queries would otherwise contend for locks and I/O with operational writes on the same tables.
When CQRS is valuable for operational workloads
- Write throughput and read throughput need to scale independently and at different rates (write-heavy ingestion feeding a low-cardinality operational dashboard).
- Write-side invariants are complex enough (state machines, multi-step validation) that mixing them with read-optimization concerns would make the write model harder to reason about.
- Not valuable: a small application with one read pattern that already matches the write schema. There CQRS adds a projector, extra storage, and extra failure modes with no offsetting benefit.
Trade-off: complexity
You now operate an additional pipeline (the projector), additional storage (one or more read stores), and additional failure modes: projector lag, projector crashes mid-batch, and schema drift between the write shape and the read shape.
Trade-off: eventual consistency of read models
Because the read model updates asynchronously, a query issued immediately after a write can observe stale data. The size of that staleness window is a direct function of projector throughput and batching, not something a team can design around by ignoring it.
Strategies to make reads "fresh" when required
- Read-your-writes for the writer: return enough state in the command response (or a version/sequence number) that the client who just wrote never needs to trust the read model for its own write.
- Tighten the pipeline: smaller batches and event-driven push instead of periodic batch pull shrinks the staleness window, at the cost of more frequent projector invocations.
- Expose staleness explicitly: attach a last-updated version or timestamp to read-model responses so callers can judge whether the data is fresh enough, instead of the system silently presenting stale data as current.
- Selective synchronous update: for a small, well-identified set of critical fields, update the read model synchronously in the write path (accepting some coupling) while everything else stays asynchronous.
Worked example
An order system accepts writes at 500 orders per minute (about 8 to 9 orders per second) into a transactional order table. A "revenue by category, per hour" read model is built by a projector that drains the order-events stream every 60 seconds and applies that batch of updates.
- Worst-case staleness for that read model equals the batch interval: 60 seconds. An order committed just after a batch run will not appear until the next run.
- Average staleness is roughly half the batch interval, about 30 seconds, if orders arrive close to uniformly across the minute.
If the product requirement is "the dashboard must reflect a new order within 10 seconds," this projector cadence fails outright: 60 seconds worst case exceeds the 10-second bound. The fix is either to drop the batch interval below 10 seconds, or to read the specific "orders placed today" counter synchronously from the write side while the rest of the dashboard stays on the 60-second cadence.
Trade-offs and pitfalls
- Common wrong turn: adopting CQRS because it sounds architecturally sophisticated for a workload that has a single read pattern already matching the write schema. That is pure overhead with no payoff.
- Common wrong turn: treating "eventually consistent" as a detail to sort out later. Staleness needs an explicit, stated bound (or an explicit "no bound" with a user-facing affordance for it) decided at design time, not discovered in production when a user cannot see the order they just placed.
- Senior signal: naming a concrete staleness budget and matching the pipeline's cadence to it, rather than discussing CQRS only in the abstract.
As a Solutions Architect, detail the decision criteria you use to choose synchronous (HTTP/REST, gRPC) versus asynchronous (message queues, event streams) service-to-service communication. Discuss trade-offs around latency, reliability, coupling, operational complexity, developer productivity, and how each choice affects deployment independence.
Sample Answer
Direct answer
Choose synchronous communication (HTTP/REST, or gRPC, a high-performance remote-call framework built on protocol buffers) when the caller needs an immediate answer to proceed and strong end-to-end reliability guarantees matter more than decoupling; choose asynchronous communication (message queues, event streams) when the work can be deferred, the caller does not need the result to continue, or you need producers and consumers to fail, scale, and deploy independently. The decision criteria are latency requirements, reliability/failure-isolation needs, coupling tolerance, operational complexity budget, developer productivity, and how much deployment independence the teams involved actually need.
Structured elaboration
Walk each axis explicitly rather than picking a style by habit:
- Latency. Synchronous calls give the caller a result (or an error) within the request's timeout window, which is required whenever a human or an immediately-dependent step is waiting (e.g., "is this card valid"). Asynchronous messaging trades immediate response for throughput and resilience: the producer gets an acknowledgment that the message was accepted, not that the work finished.
- Reliability and failure isolation. A synchronous call couples the caller's availability to the callee's: if the downstream service is slow or down, the caller blocks or fails too, and a chain of synchronous calls compounds failure probability (if three downstream services each have 99.9% availability, the chain that calls all three synchronously is bounded by roughly 0.9993≈0.997, i.e. about 3x the failure rate of any single hop). Asynchronous messaging inserts a durable buffer (the queue or log) between producer and consumer, so a consumer outage delays processing but does not fail the producer's request.
- Coupling. Synchronous calls create temporal coupling (both sides must be up at the same instant) and often contract coupling (the caller depends on the callee's exact response shape and latency profile). Asynchronous messaging only requires agreement on the event/message schema; producer and consumer do not need to be online simultaneously.
- Operational complexity. Synchronous systems are simpler to trace (a single call stack, straightforward distributed tracing) and debug. Asynchronous systems add a broker to run and monitor, delivery-semantics decisions (at-least-once handling, ordering), dead-letter queues (DLQs, queues that hold messages a consumer could not process after retries) for poison messages, and eventual-consistency reasoning that developers have to learn.
- Developer productivity. Synchronous request/response is the default mental model most engineers already have; it is faster to build and test for simple CRUD-style interactions. Asynchronous flows require additional skills (idempotent consumers, correlation IDs for tracing a request across services, compensating logic) and slower local iteration (you cannot just curl an endpoint and see the final state).
- Deployment independence. This is the axis most often underweighted. A synchronous caller that depends on a callee's API contract must coordinate deploys carefully around breaking changes (versioned endpoints, backward-compatible fields). An asynchronous consumer reading from a durable topic can be redeployed, scaled, or even paused independently of the producer, because the event log absorbs the gap; this is what lets teams ship on independent cadences, which is usually the real reason "event-driven" gets proposed for a set of otherwise unrelated services.
A simple framework: ask "does the caller need the result to proceed, right now, in this request?" If yes, go synchronous. Then ask "if the callee is degraded, should the caller degrade too, or should the caller succeed and the work catch up later?" If the caller should still succeed, go asynchronous even if latency were not a constraint, because the failure-isolation property is what you are actually buying.
Worked example
A checkout service calling three downstream services synchronously (inventory check, tax calculation, fraud score) with each at 99.9% availability has a combined dependency availability of 0.9993≈0.997, meaning roughly 3 requests in 1,000 fail purely from the chaining, even though each service individually is healthy 999 times in 1,000. If two of those three calls (tax calculation and fraud score) can tolerate a few hundred milliseconds of extra latency and their results are not needed to authorize the immediate step, moving them to asynchronous event handlers that publish a "checkout.enriched" event removes two of the three synchronous dependencies from the critical path, leaving only inventory check (which genuinely must block, since you cannot confirm an order for stock you do not have) synchronous. The chain's availability floor improves to roughly 0.9991=0.999, and the tax/fraud services can now be redeployed or scaled without coordinating a maintenance window with checkout.
Trade-offs and pitfalls
A common wrong turn is defaulting to asynchronous "for scalability" on a step the caller genuinely needs the result of right now; that just relocates the wait (the caller polls or blocks on a callback) while adding a broker, a correlation mechanism, and eventual-consistency bugs, with no user-facing benefit. The opposite pitfall is defaulting to synchronous everywhere because it is simpler to write, then discovering that one flaky downstream service now takes the whole call chain down with it. A senior answer treats each interaction independently on these six axes rather than applying one style architecture-wide, and explicitly names deployment independence as a first-class criterion, not an afterthought, since it is usually the criterion that determines whether decoupling was worth the added operational complexity.
You are on call for an asynchronous data-ingestion pipeline with an SLA to persist every event within 60 seconds. Design the observability and alerting strategy: what would you track, what would actually be worth paging someone for, and what would your on-call runbook tell them to check first?
Sample Answer
Direct answer
For a 60-second persist service-level agreement (SLA), the metric that actually matters is the projected time-to-persist for the oldest unprocessed event. Everything else, producer rate, consumer processing rate, dead-letter-queue rate, error rate, raw throughput and latency, exists to explain why that projection is moving, and paging should be tied to SLA risk crossing a threshold, not to any single metric wobbling on its own.
Structured elaboration
What to track.
- Producer rate: events published per second, the input side of the balance.
- Consumer processing rate (throughput): events actually persisted per second, the output side.
- Consumer lag: how far behind the consumer is, in event count or in the age of the oldest unprocessed event, the single number that most directly answers whether the pipeline is on track to breach the 60-second SLA.
- Dead-letter queue (DLQ) rate and depth: events that failed processing and were routed out of the normal flow. Growth here means a subset of events are silently not going to be persisted on time, or at all, until someone intervenes.
- Error rate: the fraction of processing attempts failing, whether or not they eventually land in the DLQ, often a leading indicator that rises before lag does.
- End-to-end latency: measured from event creation to successful persist, the metric the SLA is actually stated in terms of, and it should be tracked directly wherever the event carries a creation timestamp, not just inferred from lag and rate.
What's worth paging for. Not every metric wobble. Page on SLA risk specifically: when the projected additional latency for the oldest currently unprocessed event crosses a threshold meaningfully inside the 60-second budget, enough time left for someone to actually act, not an alert that fires the same moment the SLA is already breached, or when DLQ depth grows continuously rather than staying flat. A flat, near-zero DLQ is healthy; a climbing one means a systemic issue is actively producing failures faster than anyone is triaging them. A momentary lag blip that self-resolves within the next reporting interval, or a brief error-rate spike that recovers, is exactly the kind of thing that belongs on a dashboard, not something that should wake someone up.
Runbook: what to check first. With lag rising or an SLA-risk page firing, the ordered checks are: first, is the producer rate elevated, an unusual traffic spike that may just need consumer autoscaling to catch up, or is the consumer processing rate depressed, a downstream dependency slowdown, a bad deploy, resource exhaustion on the workers. This distinguishes "we need more capacity" from "something is broken." Second, check the DLQ: is it growing, and if so, sample a few entries to see whether they share a common cause, one bad event shape from one producer versus a systemic downstream failure. Third, check recent deploys and recent consumer restarts or scaling events; a consumer-group churn event, workers restarting or the assignment of work across them changing, is a common, usually self-resolving cause of a short lag spike, worth ruling out before assuming a deeper problem. Fourth, check the downstream dependency the persist step actually writes to, since a slow or unavailable datastore on the write side shows up first as processing-rate degradation, not as anything on the ingestion side.
Worked example
Suppose consumer lag is currently 12,000 unprocessed events and the consumer is processing at a steady 500 events/sec:
additional latency=50012000=24 sThe oldest unprocessed event would take about 24 seconds to be reached and persisted at the current rate, comfortably inside the 60-second SLA. If lag instead grows to 27,000 events at the same 500 events/sec processing rate, the projected additional latency is 27000/500 = 54 seconds, past a page threshold set at 75% of the SLA budget:
page threshold=0.75×60=45 sand closing in on the 60-second breach itself. That is the SLA-risk crossing that should page, not the raw lag number in isolation, the same 12,000-event lag would be a non-event if the processing rate were higher, or would already be an active breach if the rate had dropped low enough.
Trade-offs and pitfalls
- Paging on raw lag count without dividing by current processing rate produces false urgency during high-throughput periods and false calm during low-throughput periods. The SLA is stated in time; the alert should be too.
- Alerting on every DLQ message individually instead of on DLQ growth rate and depth creates alert fatigue fast. A slow, steady trickle into the DLQ from known bad producer data is a triage-queue item, not a page; a sudden acceleration is.
- Treating a transient consumer-group reassignment as an incident before checking whether lag self-resolved wastes on-call attention on a self-healing event; conversely, dismissing every lag spike as probably just a reassignment, without checking, risks missing a real regression. The runbook should say check it, not assume it.
- Tracking throughput and latency as separate, unrelated numbers instead of connecting them to the SLA budget makes the dashboard descriptive but not actionable. The projection, lag divided by rate, compared to the SLA, is what turns raw metrics into a decision.
Design a CQRS system for an online auction where commands place bids and queries read current highest bid. Explain how to implement the write model, project read models asynchronously, deal with stale reads, and resolve conflicting bids arriving near the same time.
Sample Answer
Direct answer
This is Command Query Responsibility Segregation (CQRS) applied to a specific write invariant. Split the auction into a write model that owns one invariant (a bid must beat the current highest bid) and enforces it with optimistic concurrency, and a read model that is a denormalized "current highest bid" projection updated asynchronously from the write model's event stream. The hard part is not the happy path; it is two bids landing at nearly the same time against the same auction, which optimistic concurrency turns into an explicit, retryable conflict instead of a silent race.
Structured elaboration
Write model: command handling with optimistic concurrency
- Model the auction as an aggregate: one append-only, version-checked event stream per auction, holding only
BidPlacedevents (the actual state transitions).BidRejectedoutcomes are recorded to a separate audit log that does NOT consume a version slot: if a rejected (invalid) bid instead bumped the version-checked stream, an unrelated concurrent bidder's genuinely valid, non-conflicting bid could fail with a spuriousConcurrencyErrorit never earned, purely because someone else's bad low-ball bid landed first. - A command handler loads the aggregate's current version and current highest bid, evaluates the incoming bid against the single invariant (must exceed the current highest, optionally by a minimum increment), and, only if the bid is accepted, appends the resulting
BidPlacedevent with anexpected_versionequal to what it just read; a rejected bid is logged for audit but never reaches the version-checked append call. - The event store rejects the append if the stream's actual version has moved on since the handler read it. That rejection (a concurrency conflict) is exactly the signal that another bid landed first; the handler does not need locks to detect the race, only a version check on write.
Projecting read models asynchronously
- A separate projector consumes the event stream in order and maintains a small denormalized table: auction identifier, highest amount, highest bidder, and the event sequence number the projection has processed through.
- The projector is decoupled from the write path entirely: writes succeed or fail based only on the event store's version check, never on whether the read model has caught up.
Dealing with stale reads
- Because the read model updates asynchronously, a query issued immediately after a write can return the previous highest bid for a short window. For a public "current highest bid" display this is usually acceptable; state that trade-off explicitly rather than silently accepting or silently trying to eliminate it.
- For the bidder who just placed the winning bid, do not make them wait on the read model at all: the command's own response (accepted, new version) is authoritative for their own write.
Resolving conflicting bids arriving near the same time
- Two bidders load the aggregate at the same version, both decide to bid, and both try to append at that same expected version. Only one append can succeed (the event store enforces this atomically); the second gets a concurrency conflict, not a silently wrong accepted bid.
- On conflict, the losing client's handler re-reads the now-current highest bid and re-evaluates: if the client's amount still beats the new highest, resubmit at the new version; if not, reject with a clear reason (below current highest) rather than looping forever. Cap retries so a losing bidder does not retry indefinitely against a fast-moving auction.
flowchart LR
Client[Bidder client] -->|PlaceBid command| Handler[Command handler]
Handler -->|append with expected_version| Store[(Auction event store)]
Store -->|BidPlaced / BidRejected| Bus[[Event stream]]
Bus --> Projector[Async projector]
Projector --> ReadModel[(Read model: highest bid)]
Client -->|GetHighestBid query| ReadModel
Complexity
- Command decision and append: O(1) amortized per bid, given the current highest is cached on the in-memory aggregate (or O(k) to rehydrate from k events since the last snapshot if the aggregate is loaded cold).
- Conflict retry: O(r) where r is the number of retries before the bidder either wins or is out-bid; bounded by a max-retry cap.
- Projection: O(1) per event applied, since the read model is a single row per auction keyed by auction identifier.
Edge cases
- Equal-amount bids: reject the later one as "not exceeding" the current highest (never accept a tie as a new highest bid).
- A bidder retries after a conflict with an amount that is now below the new highest: reject with an explicit reason, do not auto-increment on the bidder's behalf.
- Out-of-order projection delivery: the projector must only overwrite the read model if the incoming event's sequence number is greater than what is already projected, so a redelivered or delayed older event cannot regress the projection backward.
- Retry storm at auction close: cap retries and surface a clear "outbid, please resubmit" response instead of an unbounded retry loop when many bidders converge in the closing seconds.
Worked example
The following script implements the write model, the optimistic-concurrency conflict, the retry path, and the async projection described above, and was executed exactly as shown (no wall-clock timing is measured; a plain integer sequence counter stands in for ordering).
"""
Pinned, deterministic demo of a CQRS write model + async projection for an
online auction,
No wall-clock timing is measured or asserted. A logical counter stands in
for time so the demo is fully reproducible. Run with: python3 auction_cqrs.py
Rejected bid attempts (BidRejected) are recorded to a separate audit log,
NOT appended to the version-checked event stream. Only accepted state
changes (BidPlaced) consume a version slot. This matters under contention:
if a rejected attempt bumped the version, a concurrently-reading bidder's
genuinely valid, non-conflicting bid would fail with a spurious
ConcurrencyError it never actually earned, purely because someone else's
invalid low-ball bid happened to land first.
"""
from dataclasses import dataclass
from typing import List, Dict, Optional
class ConcurrencyError(Exception):
pass
@dataclass
class BidPlaced:
auction_id: str
bidder: str
amount: int
seq: int # logical clock, not wall time
@dataclass
class BidRejected:
auction_id: str
bidder: str
amount: int
reason: str
seq: int
class EventStore:
"""Append-only store, one stream per aggregate (auction). Only accepted
BidPlaced events are appended here, so `version()` always reflects the
auction's real state and never advances on a rejected attempt."""
def __init__(self):
self._streams: Dict[str, List[object]] = {}
self._audit_log: Dict[str, List[object]] = {}
def version(self, auction_id: str) -> int:
return len(self._streams.get(auction_id, []))
def append(self, auction_id: str, event: object, expected_version: int):
current = self.version(auction_id)
if current != expected_version:
raise ConcurrencyError(
f"expected version {expected_version}, store is at {current}"
)
self._streams.setdefault(auction_id, []).append(event)
def read(self, auction_id: str) -> List[object]:
return list(self._streams.get(auction_id, []))
def log_rejection(self, auction_id: str, event: "BidRejected"):
"""Audit-only: rejected attempts are kept for observability but do
not participate in optimistic concurrency."""
self._audit_log.setdefault(auction_id, []).append(event)
def read_audit_log(self, auction_id: str) -> List[object]:
return list(self._audit_log.get(auction_id, []))
class AuctionAggregate:
"""Write-model invariant: a bid must exceed the current highest bid."""
def __init__(self, auction_id: str, min_increment: int = 1):
self.auction_id = auction_id
self.min_increment = min_increment
self.highest_amount: Optional[int] = None
self.highest_bidder: Optional[str] = None
@classmethod
def rehydrate(cls, auction_id: str, events: List[object], min_increment: int = 1):
agg = cls(auction_id, min_increment)
for e in events:
if isinstance(e, BidPlaced):
agg.highest_amount = e.amount
agg.highest_bidder = e.bidder
return agg
def decide_place_bid(self, bidder: str, amount: int, seq: int):
floor = (self.highest_amount or 0) + (
self.min_increment if self.highest_amount is not None else 0
)
if amount <= (self.highest_amount or 0):
return BidRejected(self.auction_id, bidder, amount, "below current highest", seq)
if self.highest_amount is not None and amount < floor:
return BidRejected(
self.auction_id, bidder, amount,
f"must exceed highest by at least {self.min_increment}", seq,
)
return BidPlaced(self.auction_id, bidder, amount, seq)
class ReadModel:
"""Denormalized projection updated asynchronously from the event stream."""
def __init__(self):
self.current_highest: Dict[str, Dict] = {}
self.projected_through_seq: Dict[str, int] = {}
def apply(self, event, seq: int):
if isinstance(event, BidPlaced):
row = self.current_highest.get(event.auction_id, {"version": -1})
if seq > row.get("version", -1):
self.current_highest[event.auction_id] = {
"amount": event.amount,
"bidder": event.bidder,
"version": seq,
}
self.projected_through_seq[event.auction_id] = seq
def handle_place_bid(store: EventStore, auction_id: str, bidder: str, amount: int, seq: int):
"""Command handler: load current version, decide, append with optimistic
concurrency. A rejected command is logged for audit but never touches the
version-bearing stream. Returns (accepted: bool, event, version_after)."""
version = store.version(auction_id)
agg = AuctionAggregate.rehydrate(auction_id, store.read(auction_id))
event = agg.decide_place_bid(bidder, amount, seq)
if isinstance(event, BidRejected):
store.log_rejection(auction_id, event)
return False, event, version
try:
store.append(auction_id, event, expected_version=version)
return True, event, store.version(auction_id)
except ConcurrencyError as e:
return False, e, version
def handle_place_bid_with_retry(store: EventStore, auction_id: str, bidder: str,
amount: int, seq: int, max_retries: int = 2):
"""Retries on optimistic-concurrency conflict by re-checking the amount
against the now-current highest bid before resubmitting."""
for attempt in range(max_retries + 1):
accepted, result, version = handle_place_bid(store, auction_id, bidder, amount, seq)
if accepted:
return True, result, version, attempt
if isinstance(result, BidRejected):
return False, result, version, attempt
# ConcurrencyError: someone else's bid landed first, retry the check
return False, result, version, max_retries
if __name__ == "__main__":
store = EventStore()
reads = ReadModel()
AUCTION = "auction-42"
seq_counter = 0
def project_all():
for i, e in enumerate(store.read(AUCTION)):
reads.apply(e, i)
print("=== Sequential bids, no conflict ===")
for bidder, amount in [("alice", 100), ("bob", 120), ("carol", 150)]:
seq_counter += 1
accepted, event, version = handle_place_bid(store, AUCTION, bidder, amount, seq_counter)
print(f"bid {bidder}=${amount} -> accepted={accepted} version_after={version}")
project_all()
print("read model after projection:", reads.current_highest[AUCTION])
assert reads.current_highest[AUCTION] == {"amount": 150, "bidder": "carol", "version": 2}
print("\n=== Near-simultaneous conflicting bids (both read version=3) ===")
version_seen = store.version(AUCTION)
assert version_seen == 3
seq_counter += 1
dave_seq = seq_counter
seq_counter += 1
erin_seq = seq_counter
accepted_dave, event_dave, version_after_dave = handle_place_bid(
store, AUCTION, "dave", 160, dave_seq
)
print(f"dave bid $160 against stale version={version_seen} -> "
f"accepted={accepted_dave}, version_after={version_after_dave}")
try:
store.append(AUCTION, BidPlaced(AUCTION, "erin", 155, erin_seq), expected_version=version_seen)
erin_conflict = False
except ConcurrencyError as e:
erin_conflict = True
print(f"erin bid $155 against stale version={version_seen} -> ConcurrencyError: {e}")
assert erin_conflict is True, "expected the stale write to be rejected"
print("\n=== Erin retries: re-reads current highest (now $160) and rebids above it ===")
accepted_erin, event_erin, version_after_erin, attempts = handle_place_bid_with_retry(
store, AUCTION, "erin", 165, erin_seq
)
print(f"erin retry bid $165 -> accepted={accepted_erin}, version_after={version_after_erin}, attempts={attempts}")
assert accepted_erin is True
assert version_after_erin == 5
print("\n=== A too-low bid ($155, below erin's $165) is rejected and logged to the audit trail, NOT the version stream ===")
version_before_frank = store.version(AUCTION)
accepted_low, event_low, version_low, _ = handle_place_bid_with_retry(
store, AUCTION, "frank", 155, seq_counter + 1
)
print(f"frank bid $155 (below current $165) -> accepted={accepted_low}, reason={getattr(event_low, 'reason', None)}")
print(f"store version before frank's attempt={version_before_frank}, after={store.version(AUCTION)} (unchanged)")
assert accepted_low is False
assert isinstance(event_low, BidRejected)
assert store.version(AUCTION) == version_before_frank, "a rejected bid must not consume a version slot"
project_all()
print("\nfinal read model:", reads.current_highest[AUCTION])
assert reads.current_highest[AUCTION]["amount"] == 165
assert reads.current_highest[AUCTION]["bidder"] == "erin"
print("\nfinal version-bearing event stream (auction_id, type, bidder, amount, seq):")
for e in store.read(AUCTION):
kind = type(e).__name__
print(f" {kind:12s} bidder={e.bidder:6s} amount={e.amount:4d} seq={e.seq}")
print("\naudit log (rejected attempts, kept for observability, never gates concurrency):")
for e in store.read_audit_log(AUCTION):
print(f" BidRejected bidder={e.bidder:6s} amount={e.amount:4d} reason={e.reason}")
print("\n=== Adversarial: an invalid low-ball bid must never cause a spurious conflict for an unrelated concurrent bidder ===")
store2 = EventStore()
handle_place_bid(store2, "auction-99", "alice", 100, 1)
version_bob_saw = store2.version("auction-99")
print(f"bob reads version={version_bob_saw} (highest=$100) and prepares to bid $200")
handle_place_bid(store2, "auction-99", "attacker", 1, 2)
print(f"attacker submits an obviously-too-low $1 bid -> version after={store2.version('auction-99')} (must be unchanged: {version_bob_saw})")
assert store2.version("auction-99") == version_bob_saw
try:
store2.append("auction-99", BidPlaced("auction-99", "bob", 200, 3), expected_version=version_bob_saw)
print("bob's $200 append against his originally-seen version SUCCEEDED (no spurious conflict)")
except ConcurrencyError as e:
print("BUG: bob's valid bid was spuriously rejected:", e)
raise
print("\nALL SCENARIOS PASSED")
Actual output from running python3 auction_cqrs.py:
=== Sequential bids, no conflict ===
bid alice=$100 -> accepted=True version_after=1
bid bob=$120 -> accepted=True version_after=2
bid carol=$150 -> accepted=True version_after=3
read model after projection: {'amount': 150, 'bidder': 'carol', 'version': 2}
=== Near-simultaneous conflicting bids (both read version=3) ===
dave bid $160 against stale version=3 -> accepted=True, version_after=4
erin bid $155 against stale version=3 -> ConcurrencyError: expected version 3, store is at 4
=== Erin retries: re-reads current highest (now $160) and rebids above it ===
erin retry bid $165 -> accepted=True, version_after=5, attempts=0
=== A too-low bid ($155, below erin's $165) is rejected and logged to the audit trail, NOT the version stream ===
frank bid $155 (below current $165) -> accepted=False, reason=below current highest
store version before frank's attempt=5, after=5 (unchanged)
final read model: {'amount': 165, 'bidder': 'erin', 'version': 4}
final version-bearing event stream (auction_id, type, bidder, amount, seq):
BidPlaced bidder=alice amount= 100 seq=1
BidPlaced bidder=bob amount= 120 seq=2
BidPlaced bidder=carol amount= 150 seq=3
BidPlaced bidder=dave amount= 160 seq=4
BidPlaced bidder=erin amount= 165 seq=5
audit log (rejected attempts, kept for observability, never gates concurrency):
BidRejected bidder=frank amount= 155 reason=below current highest
=== Adversarial: an invalid low-ball bid must never cause a spurious conflict for an unrelated concurrent bidder ===
bob reads version=1 (highest=$100) and prepares to bid $200
attacker submits an obviously-too-low $1 bid -> version after=1 (must be unchanged: 1)
bob's $200 append against his originally-seen version SUCCEEDED (no spurious conflict)
ALL SCENARIOS PASSED
Walking the numbers: after alice ($100), bob ($120), and carol ($150) bid in sequence, the event store is at version 3. Dave and erin both read that version 3 (highest = $150) and each decide to bid. Dave's append reaches the store first and succeeds, moving the store to version 4. Erin's append is still expressed against version 3 (the version she read), so the store rejects it with a ConcurrencyError: "expected version 3, store is at 4". Erin's client retries: it re-reads the current highest ($160, dave's bid) and resubmits at $165, which succeeds and moves the store to version 5. A later bid of $155 from frank is rejected outright, not as a concurrency conflict but as a genuine invariant violation ("below current highest"), since $155 is below the now-current $165; frank's rejection is written to the audit log only, so the store's version stays at 5, exactly as the assertion in the script confirms. The adversarial check at the end proves the actual fix: an obviously-invalid $1 bid from an attacker does not advance the version at all, so bob's later, genuinely valid $200 bid against the version he originally saw succeeds without a spurious conflict. The final read model, built by projecting the whole stream in order, shows the correct winner: erin at $165.
Trade-offs and pitfalls
- Common wrong turn: resolving the conflict by having the event store silently accept both bids and picking a winner later based on wall-clock timestamps. Client clocks are not trustworthy for this and server-received order is already what the event store's version check gives you for free; do not reinvent it with timestamps.
- Common wrong turn: making the loser's retry re-submit the exact same amount instead of re-checking against the new highest bid; that either fails again for no new information or, worse, silently succeeds against a stale check if the retry logic is not careful about re-reading state.
- Common wrong turn: coupling the write path to the read model's freshness, for example by having the command handler wait for the projector to catch up before returning. That reintroduces the coupling CQRS is meant to remove and turns projector slowness into write-path latency.
- Common wrong turn, caught by adversarial testing: letting a rejected bid attempt append to the same version-checked stream as accepted bids. An earlier draft of this design did exactly that, and it means an obviously-invalid low-ball bid from any bidder (or an automated bid-spam attempt) silently bumps the auction's version number even though the real highest-bid state never changed, which then produces a false
ConcurrencyErrorfor a completely unrelated, genuinely non-conflicting concurrent bidder whose bid was never actually in a race with anything. The fix is to routeBidRejectedoutcomes to a separate audit log that never touches the version-checked stream, so only real state transitions (BidPlaced) can ever cause a legitimate optimistic-concurrency conflict. - This same write-model-plus-async-projection split is the general CQRS pattern; a bank-account ledger or a product-catalog price update would follow the identical structure. What makes the auction the right canonical worked example here specifically is the near-simultaneous conflicting write, which a simpler CRUD-style worked example would not exercise.
- Senior signal: naming optimistic concurrency (not locking) as the mechanism, and being explicit that a concurrency conflict is a normal, expected outcome under contention, not an error condition to be hidden from the caller.
Propose a backpressure/flow-control design when a fast producer floods a slow consumer connected via a queue system. Include mechanisms on both producer and broker sides (bounded queues, rate-limiting, pause/resume, token buckets), and describe how to implement graceful degradation while preserving important messages.
Sample Answer
Direct answer
When a fast producer floods a slow consumer behind a queue, the fix is layered: constrain the producer's effective send rate so it can never overwhelm the broker in the first place, cap what the broker will hold so an unconstrained burst fails fast instead of growing without limit, and give the consumer an explicit signal to slow the producer down. On top of that, degrade by shedding low-value messages first, never by silently dropping everything.
Structured elaboration
Producer-side mechanisms
- Token bucket rate limiter in front of the publish call, sized to the consumer's sustained throughput plus a small burst allowance, not to the producer's natural output rate.
- Pause/resume (credit-based flow control): the broker or consumer publishes a credit or high/low watermark signal; the producer pauses publishing when credits are exhausted and resumes when the consumer signals capacity again. This is the mechanism that actually closes the loop. A token bucket alone just smooths a rate, it does not react to real backlog.
- Client-side buffering with its own bounded size, so a paused producer does not itself become an unbounded memory sink. Once that local buffer is full, callers see backpressure (a blocking call, or a rejected/deferred write) instead of the process growing without limit.
Broker-side mechanisms
- A bounded queue with an explicit capacity ceiling. Once full, the broker rejects new publishes (or blocks the producer, depending on the client contract) rather than growing memory without bound. This turns "the consumer is a little slow" into a visible, actionable signal instead of a silent memory leak.
- Priority lanes or multiple queues by message class, so that once the bound is reached, the broker can shed low-priority traffic first while still admitting high-priority messages.
- Consumer-side autoscaling triggered off queue depth or lag, so backpressure is a symptom you fix by adding capacity over time, not just a state you tolerate forever.
Multi-stage and hierarchical topologies. In a pipeline with more than one hop (producer, broker, an intermediate aggregator, final consumer), apply backpressure independently at each hop rather than only at the outermost edge. A hierarchical fan-out (broker to regional relays to final consumers) needs the same bounded-queue-plus-token-bucket pattern at each stage, otherwise one stage's overflow just moves the flood one hop downstream instead of resolving it. On the signaling side, a producer that receives an explicit "too busy" response (an HTTP 429 status code from an API-fronted queue, or a broker-specific backpressure response) should treat repeated instances as its own local circuit breaker: stop sending for a cooldown window rather than retrying immediately into a system that just told you it is full.
Graceful degradation, preserving important messages
- Classify messages into priority tiers at publish time (a header or routing key), not after the fact.
- When the bounded queue approaches capacity, drop or defer the lowest tier first (an analytics ping before a checkout confirmation), and reject new low-priority publishes at the producer's client library so the rejection happens as close to the source as possible.
- Keep the shed decision observable: emit a metric and a structured log entry for every dropped or throttled message, so shedding is a visible design choice, not silent data loss.
Worked example
Say the producer bursts at 5,000 messages/sec, and the consumer side is a pool of 8 workers, each sustaining about 100 messages/sec, for a total of 800 messages/sec.
net fill rate=5000−800=4200 msg/sIf the broker's bounded queue is capped at 50,000 messages, an unmitigated burst fills it in:
time to fill=420050000≈11.9 sUnder 12 seconds from full-speed burst to a broker that starts rejecting or blocking. That is the argument for a producer-side token bucket sized to consumer capacity (800/sec) with a small burst allowance (say 960/sec, 20% headroom) rather than relying only on the broker's cap: throttling at the source means the queue never gets anywhere near 50,000 in the first place, and the 12-second figure above becomes the worst case only if the token bucket is missing or misconfigured.
Trade-offs and pitfalls
- A token bucket sized to the producer's natural rate instead of the consumer's sustained rate just moves the flood downstream. Size it to what the consumer can actually drain.
- Bounding the queue without pause/resume just converts "slow consumer" into "producer errors," which is safer than unbounded memory growth but is still an outage if nothing tells the producer to slow down instead of retry-storming.
- Dropping messages under pressure without a priority scheme treats a checkout confirmation the same as a page-view ping. Senior answers always tier the shed.
- Consumer autoscaling has ramp-up lag (new workers take time to become ready), so it complements but does not replace bounded queues and rate limiting. It is the medium-term fix, not the instantaneous one.
Unlock Full Question Bank
Get access to all 28 Event-Driven Architecture and Asynchronous Messaging interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.