Event-Driven Architecture and Asynchronous Messaging Questions

Direct answer

When a fast producer floods a slow consumer behind a queue, the fix is layered: constrain the producer's effective send rate so it can never overwhelm the broker in the first place, cap what the broker will hold so an unconstrained burst fails fast instead of growing without limit, and give the consumer an explicit signal to slow the producer down. On top of that, degrade by shedding low-value messages first, never by silently dropping everything.

Structured elaboration

Producer-side mechanisms

• Token bucket rate limiter in front of the publish call, sized to the consumer's sustained throughput plus a small burst allowance, not to the producer's natural output rate.
• Pause/resume (credit-based flow control): the broker or consumer publishes a credit or high/low watermark signal; the producer pauses publishing when credits are exhausted and resumes when the consumer signals capacity again. This is the mechanism that actually closes the loop. A token bucket alone just smooths a rate, it does not react to real backlog.
• Client-side buffering with its own bounded size, so a paused producer does not itself become an unbounded memory sink. Once that local buffer is full, callers see backpressure (a blocking call, or a rejected/deferred write) instead of the process growing without limit.

Broker-side mechanisms

• A bounded queue with an explicit capacity ceiling. Once full, the broker rejects new publishes (or blocks the producer, depending on the client contract) rather than growing memory without bound. This turns "the consumer is a little slow" into a visible, actionable signal instead of a silent memory leak.
• Priority lanes or multiple queues by message class, so that once the bound is reached, the broker can shed low-priority traffic first while still admitting high-priority messages.
• Consumer-side autoscaling triggered off queue depth or lag, so backpressure is a symptom you fix by adding capacity over time, not just a state you tolerate forever.

Multi-stage and hierarchical topologies. In a pipeline with more than one hop (producer, broker, an intermediate aggregator, final consumer), apply backpressure independently at each hop rather than only at the outermost edge. A hierarchical fan-out (broker to regional relays to final consumers) needs the same bounded-queue-plus-token-bucket pattern at each stage, otherwise one stage's overflow just moves the flood one hop downstream instead of resolving it. On the signaling side, a producer that receives an explicit "too busy" response (an HTTP 429 status code from an API-fronted queue, or a broker-specific backpressure response) should treat repeated instances as its own local circuit breaker: stop sending for a cooldown window rather than retrying immediately into a system that just told you it is full.

Graceful degradation, preserving important messages

• Classify messages into priority tiers at publish time (a header or routing key), not after the fact.
• When the bounded queue approaches capacity, drop or defer the lowest tier first (an analytics ping before a checkout confirmation), and reject new low-priority publishes at the producer's client library so the rejection happens as close to the source as possible.
• Keep the shed decision observable: emit a metric and a structured log entry for every dropped or throttled message, so shedding is a visible design choice, not silent data loss.

Worked example

Say the producer bursts at 5,000 messages/sec, and the consumer side is a pool of 8 workers, each sustaining about 100 messages/sec, for a total of 800 messages/sec.

If the broker's bounded queue is capped at 50,000 messages, an unmitigated burst fills it in:

Under 12 seconds from full-speed burst to a broker that starts rejecting or blocking. That is the argument for a producer-side token bucket sized to consumer capacity (800/sec) with a small burst allowance (say 960/sec, 20% headroom) rather than relying only on the broker's cap: throttling at the source means the queue never gets anywhere near 50,000 in the first place, and the 12-second figure above becomes the worst case only if the token bucket is missing or misconfigured.

Trade-offs and pitfalls

• A token bucket sized to the producer's natural rate instead of the consumer's sustained rate just moves the flood downstream. Size it to what the consumer can actually drain.
• Bounding the queue without pause/resume just converts "slow consumer" into "producer errors," which is safer than unbounded memory growth but is still an outage if nothing tells the producer to slow down instead of retry-storming.
• Dropping messages under pressure without a priority scheme treats a checkout confirmation the same as a page-view ping. Senior answers always tier the shed.
• Consumer autoscaling has ramp-up lag (new workers take time to become ready), so it complements but does not replace bounded queues and rate limiting. It is the medium-term fix, not the instantaneous one.

Direct answer

Use user_id itself as the partition key so Kafka's partitioner (a mod-based hash of the key) always routes every event for a given user to the same partition, which is what gives per-user ordering, it falls directly out of using the same key for the same entity, not something built separately. Balance across the cluster then comes from having enough distinct keys and a reasonably uniform hash, with a deliberate, separate remediation for the minority of disproportionately hot keys.

Structured elaboration

Hashing strategy. Kafka's default partitioner computes hash(key) % num_partitions (using a murmur2-based hash internally); the exact hash function matters less than the property that it is deterministic, the same key always maps to the same partition number for a fixed partition count. Because the partition assignment is a pure function of the key, ordering per key is automatic once you commit to using that key consistently, not an extra mechanism you build on top.

Why "balanced" is a statistical property, not a guarantee. With many distinct user ids and a well-distributed hash, the law of large numbers makes total load spread roughly evenly across partitions, if every key generates traffic at roughly the same rate. The question specifically names the case where that assumption breaks: hot users.

Handling hot users. Two remediations, with different costs:

• Key salting (sub-partitioning): append a bounded suffix to a hot user's key only once their rate crosses a threshold (user_id#0 through user_id#K-1), spreading that one user's traffic across up to K partitions. The explicit cost: per-user ordering across the salted shards is no longer guaranteed, only ordering within each shard is. This is a real trade-off to name out loud, not a free win, if the consuming system genuinely needs strict per-user order even for hot users, salting defeats that and a downstream re-sequencing step would be needed, which mostly defeats the purpose of salting in the first place.
• Isolating known hot keys: if hot users are identifiable ahead of time (service accounts, bots, high-volume enterprise integrations), route them to a separate topic or partition set with its own capacity plan, keeping the main topic's "roughly uniform" assumption valid for the long tail of ordinary users.

Multi-tenant fairness. Partitioning purely by user_id does nothing to stop one tenant (an account with many users) from occupying a disproportionate share of partitions and consumer capacity relative to a smaller tenant, since partition assignment has no concept of tenant at all. If fairness across tenants is a hard requirement: give large or noisy tenants a dedicated topic or partition range so they cannot starve others' consumer lag budget, and layer per-tenant rate limiting or quota enforcement on the consumer or producer side (the same token-bucket idea used for rate limiting generally), since a static partition assignment cannot adapt to traffic that changes over time the way an explicit quota can.

Worked example

A small, executed illustration of the mechanism (using Python's built-in zlib.crc32 purely as a stand-in deterministic hash function to demonstrate the mod-based mechanic; Kafka's actual default partitioner uses a different, murmur2-based hash internally, the mechanism, not the specific hash, is what this illustrates).

import zlib
num_partitions = 6
for uid in ["user-1042", "user-8831", "user-2207", "user-77519"]:
    h = zlib.crc32(uid.encode())
    print(uid, "partition=", h % num_partitions)

user-1042  partition= 3
user-8831  partition= 4
user-2207  partition= 4
user-77519 partition= 0

Every event for user-1042 always lands on partition 3, giving that user strict per-user order for free. Now simulate salting the hot user user-8831 into 12 sub-keys (user-8831#0 through user-8831#11):

counts = {}
for shard in range(12):
    key = f"user-8831#{shard}"
    p = zlib.crc32(key.encode()) % num_partitions
    counts[p] = counts.get(p, 0) + 1
print(counts)

{4: 5, 0: 1, 5: 5, 1: 1}

The 12 salted sub-keys land on 4 of the 6 partitions instead of all landing on partition 4 as the unsalted key would. The spread is not perfectly even at this small sample size (partitions 4 and 5 got 5 each, partitions 0 and 1 got 1 each), which is expected statistical noise at n = 12, not a flaw in the technique; at real production volumes for a genuinely hot key, the same law of large numbers that balances ordinary keys also evens out a hot key's salted shards.

Trade-offs and pitfalls

• Salting trades away strict per-user order for the salted user specifically, in exchange for spreading their load; state this cost explicitly when proposing it, do not present it as a free fix.
• Changing the partition count later reshuffles the mod-based assignment for every existing key, not just new ones, since hash(key) % num_partitions changes for essentially every key when num_partitions changes. Under-provisioning partition count up front makes a later increase a disruptive, coordinated migration, not an incremental capacity add.
• "Balanced across the cluster" and "fair across tenants" are different guarantees: the first is about hash uniformity over many keys, the second is about the business meaning of who owns which keys, no hashing scheme fixes tenant fairness on its own.
• Picking the wrong entity granularity for the key, for example partitioning by session_id when the actual ordering requirement is per-user across sessions, silently breaks the guarantee the system was supposed to provide.

Direct answer

Cost-efficient batch large-media processing trims cost on four independent levers: keep data in the cheapest storage tier it needs to be in, trigger compute directly off object-storage events instead of a separate polling layer, choose compute (serverless versus spot instances) based on the job's tolerance for interruption, and process media as a stream rather than a full local copy wherever the transform allows it. Each lever also happens to reduce egress or cold-start overhead as a side effect.

Structured elaboration

Storage tiering. Land the raw upload in a standard, hot tier only as long as it needs fast access, during and shortly after initial processing, then transition it to a cooler, cheaper tier via a lifecycle policy once it has been processed and access frequency drops. Most object stores support this as a built-in rule rather than something you have to build yourself.

Triggering via object storage events. Wire the object store's native "object created" event to invoke the processing pipeline directly, an event notification to a queue or a serverless function trigger, rather than running a separate poller that lists the bucket on an interval. This removes an idle compute component entirely and reacts immediately instead of on the poller's schedule, which also means not paying for a component that spends most of its time finding nothing new.

Compute scaling, serverless versus spot instances. Serverless (pay-per-invocation, no idle capacity) fits short, bursty, unpredictable-arrival jobs where the alternative is paying for idle worker capacity between infrequent uploads, and it fits workloads that tolerate the platform's execution-time and memory limits. Spot or preemptible instances fit longer-running, batchable jobs (large video transcodes) where the job can checkpoint and resume, or where the batch is broken into small enough units that losing one unit to preemption is cheap to redo. Spot capacity trades a real risk of interruption for a substantial cost reduction versus on-demand pricing, and that trade only pays off if the workload is actually resilient to being interrupted.

Priority across competing jobs. Where multiple jobs compete for capacity, a rush-priority transcode alongside routine batch jobs, a priority queue or priority field lets urgent jobs preempt routine ones for the next available worker slot, the same tiering principle used for backpressure-driven shedding elsewhere in this space, applied here to scheduling order rather than to dropping messages.

Streaming transforms versus copying. Many media transforms (transcoding, thumbnailing) can read the source as a stream and write the transformed output as a stream, without ever materializing the full file on local disk. This avoids provisioning large local disk or memory for large files, reduces the time before the job can start producing output, and avoids one extra full read-then-write pass over the data that a naive design (copy to local, then process, then upload) tends to add without realizing it.

Minimizing egress. Run compute in the same region as the storage bucket, transfer within a region, or between compute and storage in the same provider, is typically far cheaper than cross-region or cross-provider transfer, and avoid unnecessary intermediate copies between storage locations that would each incur their own transfer cost. This is less about a specific figure and more about counting how many times the bytes cross a network or region boundary in the design, each crossing is a cost decision, not a free operation.

Minimizing cold-start costs. For serverless compute, keep the deployed function or container lean, the transform's actual dependencies, not a general-purpose image, so cold starts are cheap when they happen. If job latency genuinely matters, not just cost, a small warm pool of provisioned capacity for the base load, with serverless or spot absorbing burst above that, avoids paying for constant idle capacity while still avoiding a cold start on every single burst-driven invocation.

Worked example

A batch of large video files, several hundred files, each significantly larger than the images an interactive upload pipeline would handle, uploaded overnight for a scheduled early-morning transcoding run: land the files in standard storage, trigger transcoding jobs off each object's creation event, run the transcode on spot instances sized for the batch (accepting that a preempted instance's partial work is cheap to redo because each file is its own checkpointable unit), stream-read the source and stream-write the transcoded output directly to the target bucket in the same region as the source, and transition the raw source to a cold storage tier once the transcoded output is confirmed written and verified. Every choice in that trace is one of the levers above, not a separate idea.

Trade-offs and pitfalls

• Spot or preemptible compute for a job that cannot checkpoint, a single monolithic multi-hour transcode with no intermediate save points, turns a cost optimization into a reliability problem. Break the job into smaller resumable units first, or do not use spot for it.
• Aggressive storage-tier transitions on data that turns out to still be accessed frequently reintroduce cost as retrieval fees from the colder tier, which can exceed what tiering saved if access patterns were misjudged.
• Streaming transforms that require random access to the whole file, some codecs, some multi-pass transforms, cannot always avoid a local copy. Verify the specific transform actually supports streaming before designing around it.
• Chasing egress savings by co-locating compute and storage in one region can conflict with a separate requirement, data residency or disaster recovery, that wants replication elsewhere. Cost is one input to the region decision, not the only one.

Direct answer

For 10k events/sec with at-least-once delivery and per-user ordering across email and push, partition by user id so every event for a given user always lands on the same ordered partition and is processed by one consumer at a time. Deduplication, backpressure, and multi-region replication are then layered on top of that partitioning choice rather than around it.

Structured elaboration

Components. Ingress producers publish NotificationRequested events carrying user_id, channel (email or push), and the payload. A partitioned event bus, keyed on user_id, sits between producers and channel-specific consumer groups: one group for email, one for push, each scaled independently. Per-channel delivery workers call the respective provider. A dedup store and a cross-region replication layer sit alongside the bus.

Achieving per-user ordering. Hash user_id to a fixed number of partitions and route every event for that user to the same partition. Within a partition, a single consumer processes events strictly in arrival order, it does not start event N+1 until event N is acknowledged (or is using only a small bounded in-order window, not arbitrary concurrency). This gives ordering per user without requiring a global order across all 10k events/sec, which would be both unnecessary and a throughput bottleneck.

Deduplication strategy. At-least-once delivery means a consumer crash-and-retry, or a producer retry after a network timeout, can each cause a duplicate. Every event carries an idempotency key the producer assigns once at creation (a user id plus a monotonic sequence, or a UUID, a universally unique identifier, generated once and never regenerated on retry). Before a delivery worker actually sends, it performs a conditional check-and-set against a dedup store keyed on that id, with a time-to-live comfortably longer than the maximum expected redelivery delay. If the key already exists, the worker skips the send and still acknowledges the event, it has already been delivered from the pipeline's point of view.

Backpressure handling. Each channel's consumer group scales on its own lag metric, a slow push provider should not stall email delivery and vice versa, and a bounded per-partition in-flight window caps how many unacknowledged events a consumer holds at once. A stuck downstream call blocks further progress on that one partition (preserving order) without unboundedly growing memory across the whole system.

Multi-region replication. Replicate the partitioned bus across regions so a regional outage does not stop delivery, and keep two things consistent across the replica: the partition-key-to-partition mapping (so per-user ordering survives a failover) and the dedup store (replicated or shared, otherwise failing over to a region with a cold dedup cache reintroduces duplicates the local store would otherwise have caught).

Worked example

Targeting roughly 500 events/sec per partition, chosen so a single consumer with typical provider-call latency and a modest in-flight window can keep up without becoming the bottleneck, 10,000 events/sec needs at least:

Route by hash(user_id) mod 20. A user who generates a disproportionate share of the traffic becomes a hot key on their one partition; since ordering is a per-user contract, that user's partition assignment cannot be split further without breaking their own ordering guarantee. The standard remediation is to rate-limit at the producer for that one user (cap and shed excess volume) rather than to try to rebalance partitions around them.

Trade-offs and pitfalls

• Hashing on a low-cardinality key, say tenant instead of user, preserves ordering at the wrong granularity and creates much hotter partitions.
• At-least-once delivery plus no explicit idempotency key is the classic bug: it is tempting to dedupe on payload content, but two legitimately different events with identical payloads then collapse into one silently. Always dedupe on an explicit event id, not content.
• Blocking a whole partition on one slow user degrades every other user sharing that partition. A bounded in-flight window with a timeout and a dead-letter path prevents one pathological event from stalling the partition indefinitely.
• Replicating the event bus across regions but not the dedup store means a failover reintroduces duplicates exactly when the system is already under stress.

Direct answer

When downstream push providers, the Apple Push Notification service (APNs), Firebase Cloud Messaging (FCM), or an email API, are rate-limited or down, the notification subsystem should queue rather than drop, prioritize what gets sent first, trip a circuit breaker so a struggling provider is not hammered by continued retries, fall back to an alternate channel where one exists, and communicate the delay honestly to both the end user and the team that will get paged about it.

Structured elaboration

Queueing and prioritization. Tier notifications at publish time (critical/transactional versus informational/marketing), and give the critical tier its own queue or a priority field so degradation sheds the low-priority tier first when the backlog needs to be bounded, the same principle as general backpressure design, tuned here to notification-specific business tiers rather than message size or type.

Circuit breakers. Track failure rate per downstream provider independently, APNs failing does not mean FCM is failing, open the breaker on a sustained failure rate over a short window, and use a half-open state that only lets a small canary volume through before fully reopening the gate. This avoids an immediate re-trip the instant a provider blips back to life.

Fallbacks. Define a real fallback path, not just "retry harder." If push fails and the notification matters, fall back to email (or SMS for the most critical class); if the fallback channel is also degraded, the message stays queued for the original channel with a bounded retry window rather than looping forever.

Retry policy. Capped exponential backoff with jitter, so retries spread out over time instead of synchronizing into a new burst against a provider that is already struggling, and a maximum retry count or time window after which the message moves to a dead-letter queue for manual or delayed reprocessing.

Communicating delays. To end users, this typically means an honest, generic in-app state ("notifications may be delayed") rather than silence, especially for anything the user is actively waiting on. To the product team, it means a dashboard and an alert channel showing current breaker state, queue depth by priority tier, and the estimated time to drain the backlog, so a rate-limit incident is visible as a known, bounded event rather than something someone has to reconstruct from logs afterward.

Worked example

With a circuit breaker configured to open after 5 consecutive failures within a 60-second window, and a 60-second cooldown before moving to half-open: a sustained APNs outage produces at most 5 failed attempts against the provider, roughly 1+2+4+8+16 = 31 seconds of retry spacing, before the breaker opens and sending stops. From that point, the queue simply holds critical-tier messages for retry once the breaker's half-open canary confirms recovery, rather than continuing to retry against a provider that has already signaled it is down.

Trade-offs and pitfalls

• Retrying without backoff or a circuit breaker turns a provider rate-limit into a self-inflicted denial-of-service against your own outbound traffic, and can get an account or IP range throttled or blocked further.
• Treating all notifications as equally important during degradation means low-value marketing pushes consume the same retry budget as a security alert. Tiering has to exist before the incident, not be improvised during it.
• A fallback channel with no cap of its own can silently overload email, or SMS, which usually carries a real per-message cost, the moment push degrades. Size and rate-limit the fallback too.
• Silence to the user reads as though the notification never happened. Even a generic delay indicator is better than nothing, but overpromising a specific recovery timeline the system cannot guarantee just moves the trust problem downstream.