Event-Driven Architecture and Asynchronous Messaging Questions

Direct answer

At-most-once means a message is delivered zero or one times (no retries, so it can be lost but never duplicated); at-least-once means it is delivered one or more times (retries until acknowledged, so it can be duplicated but not lost); exactly-once means it is delivered, and its effect applied, exactly one time, neither lost nor duplicated. Each is acceptable in a different kind of scenario, and each is achieved with a different, specific technique rather than by simply picking a label.

Structured elaboration

At-most-once

• Definition: the sender makes one delivery attempt and does not retry if it fails; duplicates cannot happen, loss can.
• Acceptable scenario: non-critical telemetry feeding a live dashboard, for example a fleet of sensors publishing temperature readings every second, where an occasional dropped reading is invisible in an aggregate view but a duplicated reading would visibly distort a running average.
• Typical technique to achieve it: a best-effort, non-persistent send with no retry logic and no durable queue backing it, often a fire-and-forget UDP-style transport or a message queue configured with no redelivery policy.

At-least-once

• Definition: the sender or broker persists the message and retries until it receives an acknowledgment; duplicates can happen, loss (in the steady state, once the network and consumer recover) cannot.
• Acceptable scenario: a background job queue processing "send this welcome email" tasks, where the job can safely run twice (the second run is a harmless no-op if the handler is idempotent) but a lost task means a real user never gets their email.
• Typical technique to achieve it: durable, persistent queues with acknowledgment-based retry, plus an idempotent consumer that deduplicates by a unique message identifier (a dedup store keyed by that identifier, or a naturally idempotent write like an upsert).

Exactly-once

• Definition: the message's effect is applied exactly one time, with neither loss nor duplication, end to end.
• Acceptable scenario: applying a financial ledger entry or decrementing inventory for a single order, where either a lost update or a duplicated update produces an incorrect, customer-visible, or legally significant result.
• Typical technique to achieve it: either transactional coordination provided by the messaging platform (atomically tying together "consume this" and "produce/commit that" so a crash cannot leave a partially-applied state), or an application-level idempotency key combined with at-least-once delivery, which is the more portable and more commonly used approach in practice since it does not depend on every hop in the pipeline supporting native transactions.

Worked example

A ticket-booking system needs to decrement available seat count by 1 when a booking event is processed.

• At-most-once booking events: if a booking event is silently dropped, the seat count is never decremented for that booking, and the system can oversell (or, worse, hold seats no one actually booked, if the decrement direction were reversed). Not acceptable here; this is exactly the wrong semantic for inventory movement.
• At-least-once with a naive handler (decrement seat_count by 1 on every delivery, no idempotency): a redelivered booking event decrements the seat count twice for one real booking, silently under-reporting availability. This is a duplication bug caused by picking the right base semantic (at-least-once, so no bookings are lost) but skipping the required application-side idempotency.
• At-least-once with an idempotent handler (if booking_id not already applied, decrement seat_count by 1 and record booking_id as applied, keyed by the booking's own unique identifier): redelivery is a no-op, seat count is decremented exactly once per real booking, achieving the exactly-once business outcome from at-least-once transport plus an idempotency key, without needing the broker to support native transactions.

Trade-offs and pitfalls

• Common wrong turn: picking at-most-once for something that clearly needs a durability guarantee, purely because it requires the least code; the missing-event failure mode does not show up until an audit or a customer complaint.
• Common wrong turn: treating "we use at-least-once delivery" as equivalent to "we have exactly-once correctness," when the idempotency work on the application side is what actually closes that gap, not the delivery semantic alone.
• Common wrong turn: reaching for full transactional exactly-once machinery when a simple idempotency key and an at-least-once queue would have solved the same business problem with far less operational overhead.
• Senior signal: matching the technique to the semantic explicitly (non-persistent send for at-most-once, durable-retry-plus-dedup for at-least-once, transactions-or-idempotency-key for exactly-once), rather than describing the three semantics only in the abstract without saying how each is actually implemented.

Direct answer

Do not trust the third team's "we don't use it" self-report as the basis for removing the field: verify usage empirically (via consumer-driven contract tests or a lineage/usage audit against the registry) before touching anything, then make the change additive and reversible rather than destructive: deprecate the field with a stated window while the two dependent teams migrate to whatever replaces it, and only physically remove it after the registry and the audit both agree nobody depends on it. What has to be in place beforehand for this (and every future change like it) to be low-risk is a schema registry with enforced compatibility, consumer-driven contracts wired into the producer's continuous integration (CI) pipeline, and field-level usage visibility, not policy written down but unchecked by tooling.

Structured elaboration

Step 1: verify the claims, don't just count votes

• Two teams say removal breaks them; one says they don't use it. Before acting on any of these statements, check field-level usage against real traffic (a lineage/audit view keyed to the registry, or the third team's own consumer-driven contract, if one exists) rather than relying on memory or a quick grep of code that might miss dynamic field access. A team that "doesn't use" a field today may still have a downstream job or dashboard reading it that the team itself is not aware of.
• If no usage-tracking tooling exists yet, this is itself the finding: the producer team cannot safely make this change (or any future one) without it, and building that visibility becomes the first deliverable, not an afterthought.

Step 2: never remove directly, deprecate first

• Mark the field deprecated in the schema registry with a stated time-to-live (a concrete window, for example 60 to 90 days depending on how disruptive the change is), rather than removing it in the next release. Deprecation is reversible; removal is not.
• If the field is being replaced rather than dropped outright (a common case: a flatter carrier_code string replaced by a richer carrier object), ship the replacement as an additive, backward-compatible field alongside the old one first, so the two dependent teams can migrate to the new field on their own schedule while the old one still works.

Step 3: give the two dependent teams a real migration path, not a deadline

• Communicate the deprecation and its window through the same channel every consumer team already watches (the registry's deprecation surface, not a one-off message that is easy to miss).
• Track each dependent team's migration status against the deprecation window on a shared dashboard, so "day 89 and two teams still on the old field" is visible before it becomes an incident, not discovered at the deadline.
• If a team cannot realistically migrate within the standard window, that is a scheduling negotiation with a visible tracked extension, not silent, indefinite delay and not a forced break either.

Step 4: remove only once verified safe

• Remove the field only after both signals agree: the deprecation window has closed, and the usage audit independently confirms zero real traffic reads it, including the third team's prior "we don't use it" claim, now verified rather than assumed.

What must be in place beforehand for this, and future changes like it, to be possible

• A schema registry that enforces compatibility on every change automatically, so an accidental breaking change cannot ship silently regardless of what any individual engineer intends.
• Consumer-driven contracts (or equivalent field-level usage tracking / lineage) wired into the producer's continuous integration (CI) pipeline, so "who actually depends on this field" is a query, not a Slack thread across three teams.
• A documented, tooled deprecation workflow (a way to mark a field deprecated with a TTL, and tooling that surfaces that to consumers and eventually blocks removal until the window and the usage audit both clear), so this becomes a repeatable, low-drama process rather than a one-off negotiation every time a producer wants to evolve its data.
• Without these three things in place, the producer team's only honest options for a genuinely breaking change are: negotiate with every consumer team by hand each time (slow, does not scale past a handful of consumers), or ship the breaking change and hope (which is what created this exact situation).

Worked example

Say the field in question is legacy_shipping_zone on a shipment-events topic with 8 consumer teams subscribed. Team A and team B say they read it (matches the scenario's "will break" teams); team C says they don't.

• A usage audit against 30 days of production traffic (using consumer group lag/read metrics keyed to which fields a consumer's deserializer actually accesses, or the registry's usage tracking if wired up) confirms: team A genuinely reads it in a nightly job, team B reads it in a real-time dashboard, and team C's claim holds, zero reads from team C's consumer group in the 30-day window.
• The field is marked deprecated with a 60-day TTL. A replacement shipping_zone_v2 field ships additively alongside it in the same event, so team A and team B can migrate independently: team A's nightly job is updated in week 2 (low urgency, batch job, easy to redeploy); team B's real-time dashboard, wired into a customer-facing screen, is updated in week 7 after more careful testing.
• At day 60, the audit is re-run: both team A and team B now read shipping_zone_v2 exclusively; nobody reads legacy_shipping_zone. The field is removed. Team C, whose original claim was correct, was never blocked or delayed by any of this.

Trade-offs and pitfalls

• Common wrong turn: trusting a consumer team's self-reported "we don't use it" without verification. Self-reports are honest but incomplete; a dynamic field access, a downstream job the reporting team forgot about, or stale documentation can all make a well-intentioned "we don't use it" wrong.
• Common wrong turn: removing the field once the two dependent teams say they've migrated, without an independent audit confirming zero remaining traffic. "We think we're done migrating" and "the traffic confirms nobody reads the old field" are different claims, and only the second one is safe to act on.
• Common wrong turn: treating this as a one-time negotiation to get through, rather than as evidence that the team needs standing tooling (registry, contracts, usage visibility) so the next field removal does not require the same manual, three-team back-and-forth.
• Senior signal: naming the prerequisite tooling explicitly as the actual answer to "what would you need in place beforehand," rather than only describing the sequence of steps for this one field.

Direct answer

Storing only events gives perfect auditability and deterministic rebuilds, but rehydration cost (the work to replay events into current state) grows with the event count, so as a stream ages, reads and rebuilds get slower and more expensive. Periodic snapshots cap that cost by giving rehydration a recent starting point instead of the beginning of time, at the price of extra storage and a validation problem: a snapshot must be provably consistent with the events it claims to summarize. As a data engineer, pick a snapshot cadence and compaction policy driven by measured rehydration cost, not by a fixed rule of thumb.

Structured elaboration

Snapshotting frequency

• Event-count-based: snapshot every N events (commonly in the low thousands, tuned to the aggregate). Predictable worst-case replay cost regardless of how much wall-clock time has passed.
• Time-based: snapshot daily or hourly, better for aggregates with low or bursty event velocity where event count alone is not a reliable trigger.
• Hybrid: aggressive event-count triggers for hot, frequently-updated aggregates; time-based triggers for long-lived but rarely-updated ones.
• Whichever trigger is chosen, tune it against measured rehydration cost (replay time and CPU per rehydration), not a value picked without data.

Snapshot storage

• Store snapshots as immutable objects in cost-efficient storage, tagged with the aggregate identifier, the schema version, and the sequence number of the last event folded into the snapshot.
• Keep hot, frequently-accessed snapshots in fast storage or a cache; move cold snapshots to cheaper storage tiers with lifecycle rules.

Snapshot validation

• A snapshot must carry: the last-applied event sequence number, a schema version, and a checksum of its contents.
• On load, verify the checksum and confirm sequence continuity (no gap between the snapshot's last-applied sequence and the next event to replay). On mismatch, fall back to a full replay from the last known-good snapshot or from the beginning.
• Run periodic background verification jobs that independently rebuild a sample of aggregates from events alone and diff against the stored snapshot, to catch silent snapshot corruption before it is relied upon.

Rehydration cost

• Rehydration cost is: load the snapshot, then replay only the events recorded after that snapshot's sequence number. Cost scales with events-since-snapshot, not with the aggregate's total lifetime event count.
• The snapshot cadence directly bounds the worst case: with a snapshot taken every N events, no rehydration ever replays more than N-1 events.

Compaction and archival to control event-store growth (folding the replay/backfill and derived-dataset-reprocessing nuance)

• Compact by taking a full snapshot and, only where retention rules permit it, archiving or truncating events older than that snapshot's sequence to cold, cheaper storage rather than deleting them outright; legal or audit requirements often forbid true deletion.
• The archival tier still has to remain replayable: any derived dataset (an analytics table, a machine learning feature store, a rebuilt read model) that was originally built by consuming the full event history needs that same history available if it must ever be reprocessed, for example after a bug fix in the transformation logic. A retention policy that only optimizes for "rehydrate a live aggregate quickly" and quietly discards old events breaks backfill for any derived dataset that depended on that full history, even though the live aggregates themselves are unaffected. Decide retention and compaction against both use cases explicitly, not just the aggregate-rehydration one.
• Use a tiered retention: hot recent events in the primary event store, older events moved to compressed, partitioned cold storage with an index that supports selective replay by aggregate and time range.
• Mark truncation points explicitly (a compaction marker event, or metadata) so any consumer replaying the stream can detect where the live tier's history starts and knows to fetch older ranges from the archive if it needs them.

Worked example

An account aggregate receives 50 events per day on average.

• Without snapshots, rehydrating that account after 5 years of history means replaying 5 * 365 * 50 = 91,250 events.
• With a snapshot taken every 1,000 events, the worst-case replay after any snapshot is 999 events (999 / 91,250 = 1.09%, so rehydration cost is bounded to roughly 1% of the no-snapshot case at year 5, not strictly under 1%: it is a hair over).
• At 50 events/day, a snapshot every 1,000 events fires roughly every 20 days (1,000 / 50 = 20), so the account accumulates at most 20 days' worth of unsnapshotted events at any time.
• If the business later needs to reprocess the full 5-year history to backfill a new derived dataset (say, a new fraud-scoring feature that needs every historical event, not just the latest snapshot), the archived event range covering all 91,250 events must still be retrievable even though live rehydration never touches most of them.

Trade-offs and pitfalls

• Common wrong turn: choosing a snapshot cadence without measuring actual rehydration cost, then discovering it under- or over-snapshots (too frequent wastes storage and write bandwidth on snapshotting; too infrequent leaves rehydration slow).
• Common wrong turn: treating snapshot validation as optional. An unvalidated, silently corrupt snapshot is worse than no snapshot: it produces confidently wrong current state instead of forcing a (correct, if slow) full replay.
• Common wrong turn: designing retention purely around live-aggregate rehydration speed and discovering, only when a derived dataset needs to be rebuilt, that the events required for that rebuild were already archived out of reach or deleted.
• Senior signal: naming the specific numeric relationship between snapshot cadence and worst-case replay cost, and treating archival policy as a decision that serves more than one consumer (live rehydration and derived-dataset reprocessing), not just the first one that comes to mind.

Direct answer

For 100k emails/day with a hard 5-second "queued" acknowledgment and resilience to SMTP (Simple Mail Transfer Protocol) outages, the design decouples the fast, synchronous part (accept the request and durably queue it) from the slow, unreliable part (actually talking to an email provider), fronted by a pub/sub topic that fans out to a durable queue, drained by an autoscaled worker fleet that calls the provider and reports status back to a store the dashboard reads.

Structured elaboration

Components and message flow

1. The application publishes an EmailRequested event to a pub/sub topic, for example SNS (Amazon's Simple Notification Service, a managed pub/sub topic), immediately after basic validation and an idempotency-key check. This is the step that must land inside the 5-second budget.
2. The topic fans out to a durable queue that the worker fleet actually polls, for example SQS (Amazon's Simple Queue Service, a managed message queue). Using a topic-plus-queue split rather than publishing straight to a queue means a second subscriber (an audit log, analytics) can be added later without touching the producer.
3. A worker fleet pulls from the queue, calls the provider, for example SES (Amazon's Simple Email Service, a transactional email-sending service) or a direct SMTP relay, and on success marks the item sent; on a provider or SMTP-level failure it lets the message become visible again for retry instead of acknowledging it away.
4. Every state transition (queued, sent, failed) writes a row to a small status table that the sales dashboard reads, keyed by message id, so the dashboard reads data the pipeline already produces rather than polling the provider itself.

This same event shape generalizes across transactional email types, order confirmations, password resets, account-activation emails, by carrying a type field on the event rather than needing a separate pipeline per email type.

Duplicate prevention. Generate an idempotency key at the point of triggering the send (an order id plus email type, or a client-supplied request id), and check it against a store (a table with a unique constraint, or a cache with a time-to-live matching the outage window you want to tolerate) before publishing. If the key already exists, return the existing status instead of publishing again. This check has to happen before the publish, not after, otherwise two racing requests both pass the check and the email goes out twice.

Retry and dead-letter-queue behavior. The queue's visibility timeout controls how long a message stays invisible to other workers while one worker is processing it. If the SMTP call is retried with capped exponential backoff (say 1s, 2s, 4s) inside the worker for a small number of attempts, a worker crash mid-send just makes the message visible again for another worker. After a bounded number of redeliveries (the queue's own receive count, not in-worker retries), route the message to a dead-letter queue so a genuinely bad message (a malformed address, a recipient that permanently rejects) does not spin forever and starve the queue for everyone else.

Resilience to SMTP outages. The queue is the buffer. If the provider is down, workers back off (their retries fail, they let visibility expire) and the queue simply grows, durably, up to whatever retention window is configured. No messages are lost as long as the queue's retention exceeds the expected outage length. When the provider recovers, the worker fleet drains the backlog, autoscaling on queue depth so the drain is fast rather than trickling out at the steady-state worker count.

Monitoring. Queue depth and the age of the oldest message (a proxy for how close the pipeline is to the SLA and to retention limits), send success/failure rate by provider response code, dead-letter-queue depth (should stay near zero, growth signals a systemic problem), and the queued/sent/failed counts the dashboard itself needs, sourced from the same status table.

Worked example

100k emails/day averages to:

$$\text{avg rate} = \frac{100000}{86400} \approx 1.16 \text{ msg/s}$$

That average is trivial to handle; the real design driver is peak, not average. As an illustrative planning assumption, if 20% of the day's volume goes out inside a single one-hour campaign send:

$$\text{campaign peak} = \frac{0.20 \times 100000}{3600} \approx 5.56 \text{ msg/s}$$

Even a modest worker fleet (5 to 10 workers, each handling one send at a time) clears that comfortably. That is also why the 5-second budget is spent almost entirely on the accept-and-queue path (validate, check the idempotency key, publish), not on waiting for the SMTP call: the API handler returns a fast "queued" response the moment the publish acknowledges, and the actual send happens after the response, off the request's critical path.

Trade-offs and pitfalls

• Doing the idempotency check and the publish as two separate, non-atomic steps under concurrent requests reintroduces the duplicate you were trying to prevent. Use a store with a unique constraint or a conditional write so the check-and-set is effectively one operation.
• Sizing queue retention shorter than the outage duration you actually need to tolerate silently discards messages once retention expires during a long provider outage. Size it to the SLA (service-level agreement) you promise, not to a default.
• Treating every SMTP failure the same wastes retries: a 5xx from the provider is worth retrying, a permanent bounce (an invalid address) is not, and should go straight to failed or the dead-letter queue instead of consuming retry budget.
• A dashboard that queries the provider's API directly for status, instead of reading the pipeline's own status table, couples dashboard latency and availability to a third party you do not control.

Direct answer

For a 60-second persist service-level agreement (SLA), the metric that actually matters is the projected time-to-persist for the oldest unprocessed event. Everything else, producer rate, consumer processing rate, dead-letter-queue rate, error rate, raw throughput and latency, exists to explain why that projection is moving, and paging should be tied to SLA risk crossing a threshold, not to any single metric wobbling on its own.

Structured elaboration

What to track.

• Producer rate: events published per second, the input side of the balance.
• Consumer processing rate (throughput): events actually persisted per second, the output side.
• Consumer lag: how far behind the consumer is, in event count or in the age of the oldest unprocessed event, the single number that most directly answers whether the pipeline is on track to breach the 60-second SLA.
• Dead-letter queue (DLQ) rate and depth: events that failed processing and were routed out of the normal flow. Growth here means a subset of events are silently not going to be persisted on time, or at all, until someone intervenes.
• Error rate: the fraction of processing attempts failing, whether or not they eventually land in the DLQ, often a leading indicator that rises before lag does.
• End-to-end latency: measured from event creation to successful persist, the metric the SLA is actually stated in terms of, and it should be tracked directly wherever the event carries a creation timestamp, not just inferred from lag and rate.

What's worth paging for. Not every metric wobble. Page on SLA risk specifically: when the projected additional latency for the oldest currently unprocessed event crosses a threshold meaningfully inside the 60-second budget, enough time left for someone to actually act, not an alert that fires the same moment the SLA is already breached, or when DLQ depth grows continuously rather than staying flat. A flat, near-zero DLQ is healthy; a climbing one means a systemic issue is actively producing failures faster than anyone is triaging them. A momentary lag blip that self-resolves within the next reporting interval, or a brief error-rate spike that recovers, is exactly the kind of thing that belongs on a dashboard, not something that should wake someone up.

Runbook: what to check first. With lag rising or an SLA-risk page firing, the ordered checks are: first, is the producer rate elevated, an unusual traffic spike that may just need consumer autoscaling to catch up, or is the consumer processing rate depressed, a downstream dependency slowdown, a bad deploy, resource exhaustion on the workers. This distinguishes "we need more capacity" from "something is broken." Second, check the DLQ: is it growing, and if so, sample a few entries to see whether they share a common cause, one bad event shape from one producer versus a systemic downstream failure. Third, check recent deploys and recent consumer restarts or scaling events; a consumer-group churn event, workers restarting or the assignment of work across them changing, is a common, usually self-resolving cause of a short lag spike, worth ruling out before assuming a deeper problem. Fourth, check the downstream dependency the persist step actually writes to, since a slow or unavailable datastore on the write side shows up first as processing-rate degradation, not as anything on the ingestion side.

Worked example

Suppose consumer lag is currently 12,000 unprocessed events and the consumer is processing at a steady 500 events/sec:

$$\text{additional latency} = \frac{12000}{500} = 24 \text{ s}$$

The oldest unprocessed event would take about 24 seconds to be reached and persisted at the current rate, comfortably inside the 60-second SLA. If lag instead grows to 27,000 events at the same 500 events/sec processing rate, the projected additional latency is 27000/500 = 54 seconds, past a page threshold set at 75% of the SLA budget:

$$\text{page threshold} = 0.75 \times 60 = 45 \text{ s}$$

and closing in on the 60-second breach itself. That is the SLA-risk crossing that should page, not the raw lag number in isolation, the same 12,000-event lag would be a non-event if the processing rate were higher, or would already be an active breach if the rate had dropped low enough.

Trade-offs and pitfalls

• Paging on raw lag count without dividing by current processing rate produces false urgency during high-throughput periods and false calm during low-throughput periods. The SLA is stated in time; the alert should be too.
• Alerting on every DLQ message individually instead of on DLQ growth rate and depth creates alert fatigue fast. A slow, steady trickle into the DLQ from known bad producer data is a triage-queue item, not a page; a sudden acceleration is.
• Treating a transient consumer-group reassignment as an incident before checking whether lag self-resolved wastes on-call attention on a self-healing event; conversely, dismissing every lag spike as probably just a reassignment, without checking, risks missing a real regression. The runbook should say check it, not assume it.
• Tracking throughput and latency as separate, unrelated numbers instead of connecting them to the SLA budget makes the dashboard descriptive but not actionable. The projection, lag divided by rate, compared to the SLA, is what turns raw metrics into a decision.