Event-Driven Architecture and Asynchronous Messaging Questions

Direct answer

For 100k emails/day with a hard 5-second "queued" acknowledgment and resilience to SMTP (Simple Mail Transfer Protocol) outages, the design decouples the fast, synchronous part (accept the request and durably queue it) from the slow, unreliable part (actually talking to an email provider), fronted by a pub/sub topic that fans out to a durable queue, drained by an autoscaled worker fleet that calls the provider and reports status back to a store the dashboard reads.

Structured elaboration

Components and message flow

1. The application publishes an EmailRequested event to a pub/sub topic, for example SNS (Amazon's Simple Notification Service, a managed pub/sub topic), immediately after basic validation and an idempotency-key check. This is the step that must land inside the 5-second budget.
2. The topic fans out to a durable queue that the worker fleet actually polls, for example SQS (Amazon's Simple Queue Service, a managed message queue). Using a topic-plus-queue split rather than publishing straight to a queue means a second subscriber (an audit log, analytics) can be added later without touching the producer.
3. A worker fleet pulls from the queue, calls the provider, for example SES (Amazon's Simple Email Service, a transactional email-sending service) or a direct SMTP relay, and on success marks the item sent; on a provider or SMTP-level failure it lets the message become visible again for retry instead of acknowledging it away.
4. Every state transition (queued, sent, failed) writes a row to a small status table that the sales dashboard reads, keyed by message id, so the dashboard reads data the pipeline already produces rather than polling the provider itself.

This same event shape generalizes across transactional email types, order confirmations, password resets, account-activation emails, by carrying a type field on the event rather than needing a separate pipeline per email type.

Duplicate prevention. Generate an idempotency key at the point of triggering the send (an order id plus email type, or a client-supplied request id), and check it against a store (a table with a unique constraint, or a cache with a time-to-live matching the outage window you want to tolerate) before publishing. If the key already exists, return the existing status instead of publishing again. This check has to happen before the publish, not after, otherwise two racing requests both pass the check and the email goes out twice.

Retry and dead-letter-queue behavior. The queue's visibility timeout controls how long a message stays invisible to other workers while one worker is processing it. If the SMTP call is retried with capped exponential backoff (say 1s, 2s, 4s) inside the worker for a small number of attempts, a worker crash mid-send just makes the message visible again for another worker. After a bounded number of redeliveries (the queue's own receive count, not in-worker retries), route the message to a dead-letter queue so a genuinely bad message (a malformed address, a recipient that permanently rejects) does not spin forever and starve the queue for everyone else.

Resilience to SMTP outages. The queue is the buffer. If the provider is down, workers back off (their retries fail, they let visibility expire) and the queue simply grows, durably, up to whatever retention window is configured. No messages are lost as long as the queue's retention exceeds the expected outage length. When the provider recovers, the worker fleet drains the backlog, autoscaling on queue depth so the drain is fast rather than trickling out at the steady-state worker count.

Monitoring. Queue depth and the age of the oldest message (a proxy for how close the pipeline is to the SLA and to retention limits), send success/failure rate by provider response code, dead-letter-queue depth (should stay near zero, growth signals a systemic problem), and the queued/sent/failed counts the dashboard itself needs, sourced from the same status table.

Worked example

100k emails/day averages to:

That average is trivial to handle; the real design driver is peak, not average. As an illustrative planning assumption, if 20% of the day's volume goes out inside a single one-hour campaign send:

Even a modest worker fleet (5 to 10 workers, each handling one send at a time) clears that comfortably. That is also why the 5-second budget is spent almost entirely on the accept-and-queue path (validate, check the idempotency key, publish), not on waiting for the SMTP call: the API handler returns a fast "queued" response the moment the publish acknowledges, and the actual send happens after the response, off the request's critical path.

Trade-offs and pitfalls

• Doing the idempotency check and the publish as two separate, non-atomic steps under concurrent requests reintroduces the duplicate you were trying to prevent. Use a store with a unique constraint or a conditional write so the check-and-set is effectively one operation.
• Sizing queue retention shorter than the outage duration you actually need to tolerate silently discards messages once retention expires during a long provider outage. Size it to the SLA (service-level agreement) you promise, not to a default.
• Treating every SMTP failure the same wastes retries: a 5xx from the provider is worth retrying, a permanent bounce (an invalid address) is not, and should go straight to failed or the dead-letter queue instead of consuming retry budget.
• A dashboard that queries the provider's API directly for status, instead of reading the pipeline's own status table, couples dashboard latency and availability to a third party you do not control.

Direct answer

Treat the dead-letter queue (DLQ) as a state machine, not just a holding queue: every quarantined message moves through explicit states, and nothing gets replayed until it passes two specific automated checks, has this exact message already been applied downstream (idempotency), and does its payload still match what the current consumer code expects (schema compatibility). Skipping either check on replay is the single most common way a "fixed" DLQ incident turns into a second incident.

Structured elaboration

Idempotency check before replay. A message can land in the DLQ after a later step in its processing failed, not the first one, for example a payment charge that succeeded but a subsequent confirmation-email step that did not. Replaying that message from the top without first checking whether it already partially or fully succeeded would double-apply the parts that already worked. Before replay, re-run the exact same dedup or idempotency lookup the normal delivery path would use, so a message that actually did succeed is recognized and skipped rather than blindly reprocessed.

Schema compatibility check before replay. A message can sit in the DLQ for days or weeks while the consumer code keeps evolving. Replaying it against the current code without checking whether its payload still matches the current expected schema risks a deserialization failure (landing right back in the DLQ) or, worse, being silently misinterpreted by code that has since changed its assumptions about a field's meaning. Validate the payload against the current schema before replay; a compatible message replays normally, an incompatible one gets transformed if a safe mapping exists, or archived with a note for manual handling rather than replayed blind.

Visibility into failure reasons. Every quarantined message should carry its classification (why it failed, and which state it is in) so a reviewer, or the automation itself, can act without re-diagnosing from scratch.

Automated replay or archive. Auto-classify at arrival (the same failure-reason tagging as the DLQ architecture question), auto-archive messages that pass a retention cutoff without being addressed, and auto-replay only for a narrowly pre-approved class of situations (for example, "downstream service X was down for known maintenance window Y, safe to replay everything quarantined during that window") with the idempotency and schema checks still applied even to auto-replays, not just manual ones. Anything outside that narrow, pre-approved class gates on human approval.

Monitoring DLQ health. Depth alone is not enough. Track the age of the oldest quarantined message (a slow leak looks fine on depth if replay keeps pace, age catches it), the replay success rate (a replay that lands right back in the DLQ signals the underlying fix did not actually work), and the arrival rate of new quarantines relative to a normal baseline.

stateDiagram-v2
  [*] --> Quarantined: retries exhausted
  Quarantined --> UnderReview: on-call or automation inspects
  UnderReview --> SchemaCheck: candidate for replay
  SchemaCheck --> IdempotencyCheck: schema compatible
  SchemaCheck --> Archived: schema incompatible
  IdempotencyCheck --> Replayed: safe to reprocess
  IdempotencyCheck --> Archived: unsafe or already applied
  Replayed --> [*]
  Archived --> [*]

Worked example

An illustrative incident, numbers chosen for the walkthrough, not measured: a downstream service outage sends 200 messages to the DLQ. On recovery, the on-call engineer runs the automated checks before batch-replaying. Of the 200: 12 had actually already succeeded on a delayed retry that landed just before the DLQ escalation was processed, the idempotency check filters these out and marks them replayed-as-already-done rather than reprocessing them; 3 have a payload shape that predates a schema migration made two weeks earlier, these route to archive with a note for manual follow-up; the remaining 185 pass both checks and replay cleanly. As a sanity check on the walkthrough's own numbers: 12 + 3 + 185 = 200, accounting for the full batch.

Trade-offs and pitfalls

• Skipping the idempotency re-check on replay is the most common way a DLQ "fix" causes a new incident, double-processing something that had already partially succeeded.
• An auto-replay policy that is too broad (for example, "replay everything older than a fixed age" with no per-message check) reintroduces exactly the failure mode this workflow exists to prevent.
• Monitoring only depth misses a slow, steady quarantine growth that never actually gets worked down, age and replay-success-rate catch what depth alone cannot.
• Not capturing why a message was archived (versus replayed) leaves nothing for someone auditing the incident later to reconstruct the decision.

Direct answer

Do not trust the third team's "we don't use it" self-report as the basis for removing the field: verify usage empirically (via consumer-driven contract tests or a lineage/usage audit against the registry) before touching anything, then make the change additive and reversible rather than destructive: deprecate the field with a stated window while the two dependent teams migrate to whatever replaces it, and only physically remove it after the registry and the audit both agree nobody depends on it. What has to be in place beforehand for this (and every future change like it) to be low-risk is a schema registry with enforced compatibility, consumer-driven contracts wired into the producer's continuous integration (CI) pipeline, and field-level usage visibility, not policy written down but unchecked by tooling.

Structured elaboration

Step 1: verify the claims, don't just count votes

• Two teams say removal breaks them; one says they don't use it. Before acting on any of these statements, check field-level usage against real traffic (a lineage/audit view keyed to the registry, or the third team's own consumer-driven contract, if one exists) rather than relying on memory or a quick grep of code that might miss dynamic field access. A team that "doesn't use" a field today may still have a downstream job or dashboard reading it that the team itself is not aware of.
• If no usage-tracking tooling exists yet, this is itself the finding: the producer team cannot safely make this change (or any future one) without it, and building that visibility becomes the first deliverable, not an afterthought.

Step 2: never remove directly, deprecate first

• Mark the field deprecated in the schema registry with a stated time-to-live (a concrete window, for example 60 to 90 days depending on how disruptive the change is), rather than removing it in the next release. Deprecation is reversible; removal is not.
• If the field is being replaced rather than dropped outright (a common case: a flatter carrier_code string replaced by a richer carrier object), ship the replacement as an additive, backward-compatible field alongside the old one first, so the two dependent teams can migrate to the new field on their own schedule while the old one still works.

Step 3: give the two dependent teams a real migration path, not a deadline

• Communicate the deprecation and its window through the same channel every consumer team already watches (the registry's deprecation surface, not a one-off message that is easy to miss).
• Track each dependent team's migration status against the deprecation window on a shared dashboard, so "day 89 and two teams still on the old field" is visible before it becomes an incident, not discovered at the deadline.
• If a team cannot realistically migrate within the standard window, that is a scheduling negotiation with a visible tracked extension, not silent, indefinite delay and not a forced break either.

Step 4: remove only once verified safe

• Remove the field only after both signals agree: the deprecation window has closed, and the usage audit independently confirms zero real traffic reads it, including the third team's prior "we don't use it" claim, now verified rather than assumed.

What must be in place beforehand for this, and future changes like it, to be possible

• A schema registry that enforces compatibility on every change automatically, so an accidental breaking change cannot ship silently regardless of what any individual engineer intends.
• Consumer-driven contracts (or equivalent field-level usage tracking / lineage) wired into the producer's continuous integration (CI) pipeline, so "who actually depends on this field" is a query, not a Slack thread across three teams.
• A documented, tooled deprecation workflow (a way to mark a field deprecated with a TTL, and tooling that surfaces that to consumers and eventually blocks removal until the window and the usage audit both clear), so this becomes a repeatable, low-drama process rather than a one-off negotiation every time a producer wants to evolve its data.
• Without these three things in place, the producer team's only honest options for a genuinely breaking change are: negotiate with every consumer team by hand each time (slow, does not scale past a handful of consumers), or ship the breaking change and hope (which is what created this exact situation).

Worked example

Say the field in question is legacy_shipping_zone on a shipment-events topic with 8 consumer teams subscribed. Team A and team B say they read it (matches the scenario's "will break" teams); team C says they don't.

• A usage audit against 30 days of production traffic (using consumer group lag/read metrics keyed to which fields a consumer's deserializer actually accesses, or the registry's usage tracking if wired up) confirms: team A genuinely reads it in a nightly job, team B reads it in a real-time dashboard, and team C's claim holds, zero reads from team C's consumer group in the 30-day window.
• The field is marked deprecated with a 60-day TTL. A replacement shipping_zone_v2 field ships additively alongside it in the same event, so team A and team B can migrate independently: team A's nightly job is updated in week 2 (low urgency, batch job, easy to redeploy); team B's real-time dashboard, wired into a customer-facing screen, is updated in week 7 after more careful testing.
• At day 60, the audit is re-run: both team A and team B now read shipping_zone_v2 exclusively; nobody reads legacy_shipping_zone. The field is removed. Team C, whose original claim was correct, was never blocked or delayed by any of this.

Trade-offs and pitfalls

• Common wrong turn: trusting a consumer team's self-reported "we don't use it" without verification. Self-reports are honest but incomplete; a dynamic field access, a downstream job the reporting team forgot about, or stale documentation can all make a well-intentioned "we don't use it" wrong.
• Common wrong turn: removing the field once the two dependent teams say they've migrated, without an independent audit confirming zero remaining traffic. "We think we're done migrating" and "the traffic confirms nobody reads the old field" are different claims, and only the second one is safe to act on.
• Common wrong turn: treating this as a one-time negotiation to get through, rather than as evidence that the team needs standing tooling (registry, contracts, usage visibility) so the next field removal does not require the same manual, three-team back-and-forth.
• Senior signal: naming the prerequisite tooling explicitly as the actual answer to "what would you need in place beforehand," rather than only describing the sequence of steps for this one field.

Direct answer

Use user_id itself as the partition key so Kafka's partitioner (a mod-based hash of the key) always routes every event for a given user to the same partition, which is what gives per-user ordering, it falls directly out of using the same key for the same entity, not something built separately. Balance across the cluster then comes from having enough distinct keys and a reasonably uniform hash, with a deliberate, separate remediation for the minority of disproportionately hot keys.

Structured elaboration

Hashing strategy. Kafka's default partitioner computes hash(key) % num_partitions (using a murmur2-based hash internally); the exact hash function matters less than the property that it is deterministic, the same key always maps to the same partition number for a fixed partition count. Because the partition assignment is a pure function of the key, ordering per key is automatic once you commit to using that key consistently, not an extra mechanism you build on top.

Why "balanced" is a statistical property, not a guarantee. With many distinct user ids and a well-distributed hash, the law of large numbers makes total load spread roughly evenly across partitions, if every key generates traffic at roughly the same rate. The question specifically names the case where that assumption breaks: hot users.

Handling hot users. Two remediations, with different costs:

• Key salting (sub-partitioning): append a bounded suffix to a hot user's key only once their rate crosses a threshold (user_id#0 through user_id#K-1), spreading that one user's traffic across up to K partitions. The explicit cost: per-user ordering across the salted shards is no longer guaranteed, only ordering within each shard is. This is a real trade-off to name out loud, not a free win, if the consuming system genuinely needs strict per-user order even for hot users, salting defeats that and a downstream re-sequencing step would be needed, which mostly defeats the purpose of salting in the first place.
• Isolating known hot keys: if hot users are identifiable ahead of time (service accounts, bots, high-volume enterprise integrations), route them to a separate topic or partition set with its own capacity plan, keeping the main topic's "roughly uniform" assumption valid for the long tail of ordinary users.

Multi-tenant fairness. Partitioning purely by user_id does nothing to stop one tenant (an account with many users) from occupying a disproportionate share of partitions and consumer capacity relative to a smaller tenant, since partition assignment has no concept of tenant at all. If fairness across tenants is a hard requirement: give large or noisy tenants a dedicated topic or partition range so they cannot starve others' consumer lag budget, and layer per-tenant rate limiting or quota enforcement on the consumer or producer side (the same token-bucket idea used for rate limiting generally), since a static partition assignment cannot adapt to traffic that changes over time the way an explicit quota can.

Worked example

A small, executed illustration of the mechanism (using Python's built-in zlib.crc32 purely as a stand-in deterministic hash function to demonstrate the mod-based mechanic; Kafka's actual default partitioner uses a different, murmur2-based hash internally, the mechanism, not the specific hash, is what this illustrates).

import zlib
num_partitions = 6
for uid in ["user-1042", "user-8831", "user-2207", "user-77519"]:
    h = zlib.crc32(uid.encode())
    print(uid, "partition=", h % num_partitions)

user-1042  partition= 3
user-8831  partition= 4
user-2207  partition= 4
user-77519 partition= 0

Every event for user-1042 always lands on partition 3, giving that user strict per-user order for free. Now simulate salting the hot user user-8831 into 12 sub-keys (user-8831#0 through user-8831#11):

counts = {}
for shard in range(12):
    key = f"user-8831#{shard}"
    p = zlib.crc32(key.encode()) % num_partitions
    counts[p] = counts.get(p, 0) + 1
print(counts)

{4: 5, 0: 1, 5: 5, 1: 1}

The 12 salted sub-keys land on 4 of the 6 partitions instead of all landing on partition 4 as the unsalted key would. The spread is not perfectly even at this small sample size (partitions 4 and 5 got 5 each, partitions 0 and 1 got 1 each), which is expected statistical noise at n = 12, not a flaw in the technique; at real production volumes for a genuinely hot key, the same law of large numbers that balances ordinary keys also evens out a hot key's salted shards.

Trade-offs and pitfalls

• Salting trades away strict per-user order for the salted user specifically, in exchange for spreading their load; state this cost explicitly when proposing it, do not present it as a free fix.
• Changing the partition count later reshuffles the mod-based assignment for every existing key, not just new ones, since hash(key) % num_partitions changes for essentially every key when num_partitions changes. Under-provisioning partition count up front makes a later increase a disruptive, coordinated migration, not an incremental capacity add.
• "Balanced across the cluster" and "fair across tenants" are different guarantees: the first is about hash uniformity over many keys, the second is about the business meaning of who owns which keys, no hashing scheme fixes tenant fairness on its own.
• Picking the wrong entity granularity for the key, for example partitioning by session_id when the actual ordering requirement is per-user across sessions, silently breaks the guarantee the system was supposed to provide.

Direct answer

Classify the failure first: only retry errors that look transient (timeouts, connection resets, a 5xx-class downstream error), and send anything that looks permanent (a validation error, a malformed payload) straight to the dead-letter queue (DLQ) without spending retry budget on it. For the retryable path, compute an exponentially growing delay capped at a maximum, then actually sleep a random value between zero and that delay (full jitter), not the delay itself, since randomizing what many failing consumers do at once is what prevents them from all retrying in lockstep.

Structured elaboration

Backoff with a cap. Each retry's base delay grows exponentially with the attempt number, but is capped so it never grows unreasonably large:

With base = 100 ms, mult = 2, cap = 5000 ms, the uncapped sequence of computed delays for attempts 1 through 5 is 100, 200, 400, 800, 1600 milliseconds (each simply doubling the previous one, 100 x 2^(i-1)), well under the cap in this range.

Why jitter specifically prevents a thundering herd. If every failing consumer instance computed and slept the exact same delay sequence, an outage affecting many instances at once would cause them all to retry in a synchronized wave, at roughly the same moment as each other, again and again, which can itself overload a downstream service that was only briefly struggling. Sleeping a uniformly random value between zero and the computed cap (full jitter) spreads that same population of retries out across the whole window instead of concentrating them, so the retries decorrelate from each other even though every instance is running identical logic.

Max retries and DLQ escalation. Once the retryable path exhausts its retry budget, escalate to the DLQ rather than retrying forever; a message that failed every attempt within the budget is very unlikely to succeed on attempt N+1 without something changing.

Transient versus permanent classification (the absorbed nuance). Not every failure deserves the same treatment. A connection timeout or a 5xx-class error from the external API is plausibly transient, worth the full retry budget. A 4xx-class validation error or a schema mismatch is not going to resolve itself on retry, sending it straight to the DLQ preserves retry budget for messages that might actually succeed and gets the unretryable ones in front of a human faster.

Observability of intermediate attempts. Log or emit a metric on every attempt, not just the final success or DLQ outcome: attempt number, computed delay, and the error, tagged with the message or event identifier. Without this, a message that failed on attempt 3 of 5 and then recovered is invisible in aggregate dashboards that only show terminal outcomes, which makes diagnosing a slow-failing dependency much harder during an incident.

flowchart TD
  A[Consume message] --> B[Call external API]
  B -->|success| C[Ack message]
  B -->|failure| D{attempt <= max_retries?}
  D -- No --> E[Send to DLQ, ack original]
  D -- Yes --> F["Compute backoff = min(cap, base*mult^attempt)"]
  F --> G["Sleep random(0, backoff): full jitter"]
  G --> H[Redeliver / retry]
  H --> B

Worked example

A rolling outage takes the external API down for several minutes. A hundred consumer instances all hit their first failure within the same second.

• Without jitter: every instance computes the identical sequence 100, 200, 400, 800, ... milliseconds and sleeps exactly that long, so all hundred retry at (roughly) the same instant repeatedly, a synchronized retry wave hitting the API the moment it starts to recover, which can knock it back down.
• With full jitter: each instance independently sleeps a value drawn uniformly between 0 and the computed cap for that attempt, so the same hundred retries spread across the whole window instead of landing together, giving the recovering API a smoothly ramping load instead of a spike.

This is a structural argument about what jitter changes (the retry timing distribution across instances), not a claim about a specific measured outcome for any particular API or incident.

Trade-offs and pitfalls

• Setting max retries too high delays DLQ visibility of a dependency that is actually broken, not just slow.
• Setting max retries too low routes ordinary transient blips to the DLQ, creating needless manual-review work for on-call.
• Skipping the transient-versus-permanent classification wastes retry budget on errors that were never going to succeed and delays the human attention a permanent failure actually needs.
• Not tagging attempt number and computed delay in logs or metrics makes a slowly-degrading dependency far harder to diagnose, since only the terminal outcome (success or DLQ) is visible without it.