Event-Driven Architecture and Asynchronous Messaging Questions

Direct answer

For 100k emails/day with a hard 5-second "queued" acknowledgment and resilience to SMTP (Simple Mail Transfer Protocol) outages, the design decouples the fast, synchronous part (accept the request and durably queue it) from the slow, unreliable part (actually talking to an email provider), fronted by a pub/sub topic that fans out to a durable queue, drained by an autoscaled worker fleet that calls the provider and reports status back to a store the dashboard reads.

Structured elaboration

Components and message flow

1. The application publishes an EmailRequested event to a pub/sub topic, for example SNS (Amazon's Simple Notification Service, a managed pub/sub topic), immediately after basic validation and an idempotency-key check. This is the step that must land inside the 5-second budget.
2. The topic fans out to a durable queue that the worker fleet actually polls, for example SQS (Amazon's Simple Queue Service, a managed message queue). Using a topic-plus-queue split rather than publishing straight to a queue means a second subscriber (an audit log, analytics) can be added later without touching the producer.
3. A worker fleet pulls from the queue, calls the provider, for example SES (Amazon's Simple Email Service, a transactional email-sending service) or a direct SMTP relay, and on success marks the item sent; on a provider or SMTP-level failure it lets the message become visible again for retry instead of acknowledging it away.
4. Every state transition (queued, sent, failed) writes a row to a small status table that the sales dashboard reads, keyed by message id, so the dashboard reads data the pipeline already produces rather than polling the provider itself.

This same event shape generalizes across transactional email types, order confirmations, password resets, account-activation emails, by carrying a type field on the event rather than needing a separate pipeline per email type.

Duplicate prevention. Generate an idempotency key at the point of triggering the send (an order id plus email type, or a client-supplied request id), and check it against a store (a table with a unique constraint, or a cache with a time-to-live matching the outage window you want to tolerate) before publishing. If the key already exists, return the existing status instead of publishing again. This check has to happen before the publish, not after, otherwise two racing requests both pass the check and the email goes out twice.

Retry and dead-letter-queue behavior. The queue's visibility timeout controls how long a message stays invisible to other workers while one worker is processing it. If the SMTP call is retried with capped exponential backoff (say 1s, 2s, 4s) inside the worker for a small number of attempts, a worker crash mid-send just makes the message visible again for another worker. After a bounded number of redeliveries (the queue's own receive count, not in-worker retries), route the message to a dead-letter queue so a genuinely bad message (a malformed address, a recipient that permanently rejects) does not spin forever and starve the queue for everyone else.

Resilience to SMTP outages. The queue is the buffer. If the provider is down, workers back off (their retries fail, they let visibility expire) and the queue simply grows, durably, up to whatever retention window is configured. No messages are lost as long as the queue's retention exceeds the expected outage length. When the provider recovers, the worker fleet drains the backlog, autoscaling on queue depth so the drain is fast rather than trickling out at the steady-state worker count.

Monitoring. Queue depth and the age of the oldest message (a proxy for how close the pipeline is to the SLA and to retention limits), send success/failure rate by provider response code, dead-letter-queue depth (should stay near zero, growth signals a systemic problem), and the queued/sent/failed counts the dashboard itself needs, sourced from the same status table.

Worked example

100k emails/day averages to:

That average is trivial to handle; the real design driver is peak, not average. As an illustrative planning assumption, if 20% of the day's volume goes out inside a single one-hour campaign send:

Even a modest worker fleet (5 to 10 workers, each handling one send at a time) clears that comfortably. That is also why the 5-second budget is spent almost entirely on the accept-and-queue path (validate, check the idempotency key, publish), not on waiting for the SMTP call: the API handler returns a fast "queued" response the moment the publish acknowledges, and the actual send happens after the response, off the request's critical path.

Trade-offs and pitfalls

• Doing the idempotency check and the publish as two separate, non-atomic steps under concurrent requests reintroduces the duplicate you were trying to prevent. Use a store with a unique constraint or a conditional write so the check-and-set is effectively one operation.
• Sizing queue retention shorter than the outage duration you actually need to tolerate silently discards messages once retention expires during a long provider outage. Size it to the SLA (service-level agreement) you promise, not to a default.
• Treating every SMTP failure the same wastes retries: a 5xx from the provider is worth retrying, a permanent bounce (an invalid address) is not, and should go straight to failed or the dead-letter queue instead of consuming retry budget.
• A dashboard that queries the provider's API directly for status, instead of reading the pipeline's own status table, couples dashboard latency and availability to a third party you do not control.

Direct answer

A message queue is a transient work-distribution mechanism: a message is delivered to one consumer and then typically removed. An event stream is a durable, ordered, append-only log that many consumers can read and re-read independently, each tracking their own position. Use a queue for background jobs that need to happen once; use an event stream when multiple consumers need to replay or independently process the same history of events, such as for analytics.

Structured elaboration

(1) Simple high-level definitions. A message queue holds discrete units of work; once a consumer successfully processes (and acknowledges) a message, it is generally gone from the queue. An event stream holds an ordered, append-only sequence of events that persists for a configured retention period regardless of whether any consumer has read them yet, and multiple independent consumers can read the same stream at their own pace.

(2) Step-by-step architecture differences.

• Delivery model: a queue is point-to-point: competing consumers pull from the same backlog, and each message goes to exactly one of them. An event stream is publish-subscribe at the storage layer: every consumer group gets its own full view of the log, so 5 independent consumer groups can each read every event, not compete for it.
• Retention: a queue typically discards a message once acknowledged (or after a short retention window for safety); an event stream retains events for a configured duration (hours to indefinitely) independent of consumption, which is what makes replay possible.
• Consumer offsets: a queue broker tracks per-message state (delivered, in-flight, acknowledged) on the broker's behalf; a consumer of an event stream tracks its own offset (a pointer into the log marking "I have processed up through here"), which the consumer can reset backward to reprocess history or forward to skip it, something a queue's consume-once model does not support.

(3) Concrete use cases. A task queue for background jobs (resize an uploaded image, send a single email, generate a PDF) is the queue use case: each job runs once, and once done there is no reason to revisit it. An event log for analytics (every purchase event, retained for 30+ days, read independently by a real-time dashboard consumer, a daily batch-aggregation consumer, and a fraud-detection consumer, each at their own offset) is the event-stream use case: the same history needs to serve multiple, independently-paced readers, and a new consumer added next month should be able to read events from before it existed.

(4) Delivery semantics and operational trade-offs. Both queues and streams commonly offer at-least-once delivery (a message may be delivered more than once, so consumers must be idempotent) as the practical default, since it is the easiest guarantee to build reliably. At-most-once (a message may be silently lost but never duplicated) is rarely chosen deliberately; it falls out of not retrying failed deliveries and is generally the wrong default for anything that matters. Exactly-once, from the consumer's experience, is achievable at the application layer by combining at-least-once delivery with a consumer-side idempotency check (a deduplication store keyed by message ID with a time-to-live, TTL), rather than relying on a broker guarantee alone; broker-level exactly-once semantics (EOS) protocols exist on some platforms but their internals are a streaming-platform implementation detail, not something the consuming application needs to reason about. Operationally, queues are simpler to run (bounded backlog, straightforward monitoring of queue depth) but lose replay ability; event streams give you replay and multiple independent consumer groups at the cost of needing retention-size planning, offset-lag monitoring per consumer group, and generally a heavier broker to operate.

Worked example

An e-commerce platform's "PurchaseCompleted" event needs to reach a real-time inventory-decrement consumer (must process within seconds), a nightly analytics-aggregation job (reads the whole day's events once, in batch), and a newly-added fraud-review consumer (needs to backfill and reprocess the last 7 days of purchases to retrain a rule set). This is only possible on an event stream: the inventory consumer reads from the current offset forward, the analytics job resets its offset to the start of each day, and the fraud-review consumer resets its offset back 7 days on rollout, all three reading the same retained log independently. If "PurchaseCompleted" had instead been placed on a point-to-point queue, only one of the three would ever have received a given event, and none of them could replay history that had already been consumed.

Trade-offs and pitfalls

The most common pitfall is reaching for an event stream by default because it seems more capable, then paying its operational cost (retention planning, per-consumer-group offset-lag monitoring, a heavier broker) for a workload that was really a simple once-only task queue. The opposite pitfall is placing something that needed independent replay, like the fraud-review backfill above, onto a queue and discovering months later that the history needed to replay it was already gone. A senior answer treats "does more than one independent party need to read this, possibly at different times or replaying history" as the deciding question, not "which technology is newer or more scalable."

Direct answer

Derive the deduplication (dedup) key from the payment's own business identity, payment_id, not from anything the broker generates, then make claiming that key and processing the payment a single atomic step: check-and-claim, process, mark done. Redelivery of the same message must always produce the same payment_id, so a redelivered message collapses onto the same key and gets skipped instead of charged again.

Structured elaboration

Why payment_id, not the other fields. The message is {payment_id, user_id, amount, currency, timestamp}. user_id alone is wrong (one user can legitimately make two separate payments). (user_id, amount, timestamp) is also wrong: two real, distinct charges can share the same amount, and timestamp is set by the producer, so a retried publish can carry a different timestamp for the same logical payment while a coincidentally identical tuple could describe two different real charges. payment_id is the one field that is supposed to be stable across every redelivery of the same logical event, which is exactly the property a dedup key needs.

The atomicity requirement. "At-least-once delivery from the broker" means the same message will arrive more than once, by design, not as an edge case. If the consumer does exists = store.get(payment_id) followed by a separate store.set(payment_id, ...), two redeliveries handled concurrently (two consumer threads, or two instances in the same consumer group during a rebalance) can both read "not found" before either writes, and both charge. The claim has to be a single atomic operation: "insert this key only if it does not already exist," succeed-or-fail in one round trip.

Store choice: in-memory vs. Redis vs. relational database.

For a payment consumer, Redis is usually the right default for the hot path because "maintain low latency" is an explicit requirement and the claim is a single command. The relational option earns its extra latency specifically when you want the dedup row and the ledger write to be atomically consistent with each other, for example inserting the dedup row in the same transaction as the row that records the charge, so a crash between "claim" and "charge" cannot leave the dedup store and the ledger disagreeing.

Probabilistic pre-filter (folding in the absorbed dedup-technique survey). At high volume, a Bloom or cuckoo filter can sit in front of the authoritative store as a cheap "have I possibly seen this before" check: a negative result is certain (definitely new, skip the store lookup and go straight to processing), a positive result is only "maybe," and must still fall through to the authoritative store, because treating a false positive as certain would silently drop a legitimate first-time charge. A cuckoo filter additionally supports deletion, which matters if you want the filter itself to track a rolling window rather than growing forever. Either filter is an optimization that reduces load on the durable store for the (usually large) fraction of genuinely-new events; it never replaces the store as the source of truth.

TTL strategy. The dedup key's time-to-live (TTL) must be at least as long as the broker's own maximum redelivery window plus a safety margin. If the broker can, in the worst case, redeliver a message up to 12 hours after first delivery (for example, a long consumer outage followed by catch-up), a TTL shorter than that lets the key expire and a legitimate-looking "new" charge slip through on a very late, otherwise-normal redelivery. TTL should not be sized off how long you want to retain history; it should be sized off how long the broker can still legally hand you a duplicate.

Cleanup considerations. Redis expires keys natively via the TTL, no separate job needed. A relational dedup table has no native per-row expiry in most databases, so it needs an explicit periodic sweep (a scheduled DELETE WHERE created_at < now() - retention_window, i.e. a dedup-table-plus-TTL pattern implemented as a cron job rather than a database feature), or the table grows without bound and both storage cost and index lookup latency degrade over time.

flowchart TD
  A[Payment event arrives] --> B{Dedup key exists in store?}
  B -- Yes --> C[Skip: return prior result, no charge]
  B -- No --> D[Atomically claim key: SETNX with TTL]
  D --> E[Process payment]
  E --> F[Mark key as done, keep TTL]
  F --> G[Ack message to broker]
  C --> G

Worked example

Assume 500 payment events per second sustained, and a 24 hour TTL matching the broker's stated maximum redelivery window.

At roughly 150 bytes per key plus value plus Redis's own per-key overhead:

That is a meaningful working set for a single Redis instance. It is exactly the kind of number that motivates either sharding the dedup store across a Redis cluster keyed by payment_id, shortening the TTL if the broker's actual redelivery window is smaller than assumed, or moving completed, already-settled payments' dedup records into the cheaper relational table once they age past the point where redelivery is still plausible.

Trade-offs and pitfalls

• Using a coarse or derived key (amount, timestamp, or a hash of the whole payload) instead of payment_id is the single most common mistake: it either misses real duplicates (payload varies slightly across redeliveries, for example a re-serialized timestamp) or falsely merges two distinct legitimate charges.
• A GET-then-SET pattern instead of a single atomic command reintroduces the exact race the dedup layer exists to prevent; this only shows up under real concurrent redelivery, so it can pass casual testing and still double-charge in production.
• Redis becoming unavailable forces an explicit choice: fail closed (stop processing new payments until Redis is back, safe but reduces availability) or fail open (keep processing without dedup protection, risks double-charges). Silently defaulting to one without deciding on purpose is itself the pitfall.
• Sizing the relational table's cleanup job off total historical volume instead of the broker's actual redelivery window wildly over-retains data and slows the unique-constraint index down for no correctness benefit.

Direct answer

For an internal eventing platform at 50,000 events/sec with durable replay for analytics, at-least-once delivery for many consumers, and per-key ordering for some consumers, Kafka is the better fit: its partitioned, replayable log natively supports per-key ordering (all events for a given key land on the same partition) and long retention, which RabbitMQ does not provide as first-class capabilities. RabbitMQ remains the better choice when the workload is closer to task distribution with complex routing needs and no replay requirement, which this workload is not.

Structured elaboration

Evaluation criteria, applied to the four stated constraints:

• Sustained throughput at 50k events/sec: both platforms can reach this rate, but the design work to get there differs. Kafka scales throughput by increasing partition count (more parallelism), which is a capacity-planning exercise done up front. RabbitMQ scales by adding queues and consumers, and can also sustain very high throughput, but does not offer partition-based ordered parallelism as a native primitive.
• Durable replay for analytics: this is the decisive criterion. Kafka retains events for a configured window regardless of consumption and lets any consumer group replay from an arbitrary offset. RabbitMQ removes messages once acknowledged; replaying "already-processed" history is not a built-in capability and would require bolting on a separate durable store.
• At-least-once delivery for many consumers: both support at-least-once delivery natively. Kafka's consumer-group model lets many independent consumer groups each read the full stream at-least-once without competing with each other; RabbitMQ supports at-least-once via manual acknowledgment, but achieving "many independent full-stream readers" specifically (rather than competing consumers on one queue) requires fanning the exchange out to a separate queue per consumer group, which is more configuration to maintain as consumers are added.
• Ordered delivery per key for some consumers: Kafka provides this natively via partition-key hashing, exactly the "some consumers require ordered delivery per key" requirement. RabbitMQ can approximate per-key ordering with a single active consumer per queue and a routing key per entity, but that limits parallelism for that queue to one consumer, which is a real constraint at 50k events/sec.

Recommendation: Kafka, specifically because the combination of durable replay and per-key ordering are both native to its architecture, whereas RabbitMQ would need workarounds for both.

Mitigations for Kafka's weaknesses. Kafka's two well-known weak points relative to RabbitMQ are operational complexity (a broker cluster, plus metadata quorum management, to run and monitor) and the lack of complex, content-based routing that RabbitMQ's exchange types provide natively. Mitigate the operational-complexity weakness by using a managed offering (e.g., Amazon MSK or Confluent Cloud) rather than self-hosting the cluster. Mitigate the routing-flexibility weakness by keeping routing logic in the consuming application (a consumer subscribes to specific topics and filters or re-publishes as needed) rather than expecting the broker to route by message content, which is a reasonable trade given this workload's requirements are about ordering and replay, not complex routing.

Worked example

Sizing partitions for 50,000 events/sec with per-key ordering: assume, as a planning input for this example (not a general Kafka benchmark), that a single partition on the chosen broker instance size can sustainably sustain roughly 5,000 events/sec for this message payload size. The minimum partition count is:

$N_{\text{partitions}} \ge \left\lceil \frac{50{,}000 \text{ events/s}}{5{,}000 \text{ events/s per partition}} \right\rceil = 10$

Ten partitions is the bare minimum to sustain the target throughput with zero headroom; in practice you would provision with headroom for growth and for uneven key distribution (a small number of very active keys, or "hot keys," can overload their assigned partition even when the average across all partitions is fine), so a reasonable starting point is roughly double the bare minimum, i.e. 20-24 partitions, revisited once real key-distribution data from production traffic is available. This calculation is illustrative: the 5,000 events/sec per-partition figure is a stated planning assumption for this example, to be replaced with a measured number from a load test against your actual message size and hardware before committing to a partition count in production.

Trade-offs and pitfalls

The main pitfall is under-provisioning partitions based only on average throughput and ignoring hot-key skew; a partition count that comfortably covers the 50k events/sec average can still fall over if one key (e.g., one very active customer or entity) accounts for a disproportionate share of traffic, since that key's events are all pinned to a single partition by design. A second pitfall is choosing partition count too high without a plan to ever reduce it: Kafka does not support shrinking partition count on an existing topic without breaking the per-key-to-partition mapping (and therefore ordering) for every key, so partition sizing decisions are effectively one-directional and should be made deliberately, not just rounded up generously. A third pitfall specific to this scenario is assuming "at-least-once for many consumers" and "ordered per key for some consumers" are the same requirement: ordering is a per-partition property that some consumers need to respect (by not parallelizing within a key), while at-least-once is a delivery guarantee every consumer needs to handle via idempotency regardless of whether it cares about ordering.

Direct answer

At-most-once means a message is delivered zero or one times (no retries, so it can be lost but never duplicated); at-least-once means it is delivered one or more times (retries until acknowledged, so it can be duplicated but not lost); exactly-once means it is delivered, and its effect applied, exactly one time, neither lost nor duplicated. Each is acceptable in a different kind of scenario, and each is achieved with a different, specific technique rather than by simply picking a label.

Structured elaboration

At-most-once

• Definition: the sender makes one delivery attempt and does not retry if it fails; duplicates cannot happen, loss can.
• Acceptable scenario: non-critical telemetry feeding a live dashboard, for example a fleet of sensors publishing temperature readings every second, where an occasional dropped reading is invisible in an aggregate view but a duplicated reading would visibly distort a running average.
• Typical technique to achieve it: a best-effort, non-persistent send with no retry logic and no durable queue backing it, often a fire-and-forget UDP-style transport or a message queue configured with no redelivery policy.

At-least-once

• Definition: the sender or broker persists the message and retries until it receives an acknowledgment; duplicates can happen, loss (in the steady state, once the network and consumer recover) cannot.
• Acceptable scenario: a background job queue processing "send this welcome email" tasks, where the job can safely run twice (the second run is a harmless no-op if the handler is idempotent) but a lost task means a real user never gets their email.
• Typical technique to achieve it: durable, persistent queues with acknowledgment-based retry, plus an idempotent consumer that deduplicates by a unique message identifier (a dedup store keyed by that identifier, or a naturally idempotent write like an upsert).

Exactly-once

• Definition: the message's effect is applied exactly one time, with neither loss nor duplication, end to end.
• Acceptable scenario: applying a financial ledger entry or decrementing inventory for a single order, where either a lost update or a duplicated update produces an incorrect, customer-visible, or legally significant result.
• Typical technique to achieve it: either transactional coordination provided by the messaging platform (atomically tying together "consume this" and "produce/commit that" so a crash cannot leave a partially-applied state), or an application-level idempotency key combined with at-least-once delivery, which is the more portable and more commonly used approach in practice since it does not depend on every hop in the pipeline supporting native transactions.

Worked example

A ticket-booking system needs to decrement available seat count by 1 when a booking event is processed.

• At-most-once booking events: if a booking event is silently dropped, the seat count is never decremented for that booking, and the system can oversell (or, worse, hold seats no one actually booked, if the decrement direction were reversed). Not acceptable here; this is exactly the wrong semantic for inventory movement.
• At-least-once with a naive handler (decrement seat_count by 1 on every delivery, no idempotency): a redelivered booking event decrements the seat count twice for one real booking, silently under-reporting availability. This is a duplication bug caused by picking the right base semantic (at-least-once, so no bookings are lost) but skipping the required application-side idempotency.
• At-least-once with an idempotent handler (if booking_id not already applied, decrement seat_count by 1 and record booking_id as applied, keyed by the booking's own unique identifier): redelivery is a no-op, seat count is decremented exactly once per real booking, achieving the exactly-once business outcome from at-least-once transport plus an idempotency key, without needing the broker to support native transactions.

Trade-offs and pitfalls

• Common wrong turn: picking at-most-once for something that clearly needs a durability guarantee, purely because it requires the least code; the missing-event failure mode does not show up until an audit or a customer complaint.
• Common wrong turn: treating "we use at-least-once delivery" as equivalent to "we have exactly-once correctness," when the idempotency work on the application side is what actually closes that gap, not the delivery semantic alone.
• Common wrong turn: reaching for full transactional exactly-once machinery when a simple idempotency key and an at-least-once queue would have solved the same business problem with far less operational overhead.
• Senior signal: matching the technique to the semantic explicitly (non-persistent send for at-most-once, durable-retry-plus-dedup for at-least-once, transactions-or-idempotency-key for exactly-once), rather than describing the three semantics only in the abstract without saying how each is actually implemented.