Scalability Patterns and Techniques Questions

Direct answer

Read replicas are read-only copies of a primary relational database that let you route read-heavy traffic away from the primary, so read throughput scales roughly with the number of replicas instead of being capped by one machine's capacity. The two common replication modes trade off write latency against durability: asynchronous replication is fast but can lag, semi-synchronous replication waits for at least one replica to acknowledge before confirming a write, trading some write latency for a stronger durability guarantee. Replication lag, the gap between a write landing on the primary and appearing on a replica, is the operational pitfall that follows directly from choosing asynchronous replication for speed.

Structured elaboration

Why read replicas scale reads

A single primary database has a ceiling on how many queries per second (QPS, the standard measure of database or API load) it can serve before CPU, memory, or I/O saturates. Since most application workloads are read-heavy relative to writes, adding replicas that each hold a full copy of the data lets read queries fan out across many machines while writes still funnel through the one primary that owns correctness. This is a read-scaling pattern specifically: it does nothing for write throughput, which is bounded by the primary alone (write scaling is a separate problem, addressed by partitioning or sharding rather than replicas).

Replication modes

• Asynchronous: the primary commits and returns success to the client without waiting for any replica to apply the change. Write latency stays low and unaffected by replica health, but a replica can fall arbitrarily behind under load, and if the primary fails before a replica caught up, those last writes are lost from that replica's perspective.
• Semi-synchronous: the primary waits for acknowledgment from at least one replica (that the write was received, not necessarily fully applied) before confirming the commit to the client. This bounds the worst-case data loss to writes that hadn't yet reached any replica, at the cost of added write latency and a risk that a slow replica introduces a stall on every write.

This is standard terminology in an online transaction processing (OLTP) context, meaning a workload of many small, individual reads and writes (as opposed to large analytical scans); read replicas are one of the first tools reached for once a single OLTP primary starts to strain under read load.

Replication lag as the operational pitfall

Lag arises from network delay, I/O contention on the replica, or the replica processing a backlog of changes slower than the primary produces them. Its consequence is stale reads: a client that just wrote data may query a replica and not see its own write, or two clients may observe the data in different states depending on which replica they hit.

Read-routing design to minimize stale reads while maximizing throughput

The application layer, not just the database, needs a policy for which reads are allowed to be stale:

• Reads that must reflect the client's own very recent write (a user viewing the profile they just edited) should go to the primary, or to a replica only after confirming its lag has caught past that write's position.
• Reads that tolerate a small staleness window (a public dashboard, a search index, an analytics report) should go to replicas by default, since that is where the throughput gain comes from.
• A hybrid policy, sometimes called read-your-writes routing, pins an individual client to the primary (or to a replica known to be caught up) for a short window right after that client writes, then lets subsequent reads fall back to any replica.

Monitoring and safeguards

Safeguards built on that monitoring: alert when lag crosses a threshold tied to the application's staleness tolerance; throttle or move expensive ad hoc/analytical queries off replicas that also serve latency-sensitive reads; and, for any workflow that promotes a replica (to primary, during a failure), require lag to be at or near zero before promotion, since promoting a lagging replica means accepting the unreplicated writes as lost. That promotion and failover mechanics belong to the high-availability side of the system, not to the read-scaling pattern itself, but the monitoring described here is exactly what feeds that decision when it happens.

Worked example

A social-media-style application serves 9,000 reads per second and 1,000 writes per second against a single primary that is now CPU-saturated on reads. Adding 3 asynchronous read replicas and routing all reads except "read-your-own-write" cases to a round-robin pool across them reduces the read load on the primary from 9,000 QPS to roughly 0 (reads move off entirely), leaving the primary handling only the 1,000 writes/second plus the small share of reads that require read-your-writes freshness. Each replica now carries roughly 9,000 / 3 = 3,000 reads/second on average, well within a single replica's typical headroom, illustrating the linear-ish scaling read replicas provide as long as write volume itself stays within what one primary can sustain.

Trade-offs & pitfalls

• Read replicas scale reads only; teams sometimes reach for them to fix a write-contention problem, which they cannot, because writes still funnel through one primary.
• Asynchronous replication's low write latency is attractive, but skipping the read-routing design above (treating every replica as equally fresh) is the most common way stale reads leak into user-facing behavior.
• Semi-synchronous replication reduces data-loss risk but can introduce write stalls if the acknowledging replica itself becomes slow; it shifts risk from data loss to latency, it does not eliminate risk.
• Promoting a lagging replica during an incident, without checking lag first, can silently drop the most recent committed writes; this is a data-loss event dressed up as a recovery action.

Direct answer

A thundering herd happens when a large number of concurrent requests converge on the same cache key or origin at the same instant, whether that is one hot key expiring, a cache flush after a deploy, or a backend service coming back online after an outage. The strategy has two halves: prevent the herd from forming in the first place (spread out when things expire and who has to pay for a miss), and contain it once it starts (make sure only a small, bounded number of requests actually reach the origin while everyone else waits on or reuses that result). The right mix of techniques is a complexity-versus-protection trade-off: cheap local techniques handle the common case, and the more expensive coordinated techniques get added only where the blast radius or the cost of getting it wrong justifies them.

The simplest version, as a baseline

Two inexpensive techniques already cover a large fraction of real thundering-herd cases: adding random jitter to time-to-live (TTL) values (so a batch of keys set at the same moment do not all expire at the same moment) combined with request coalescing, sometimes called singleflight (so that when a key does expire, only one of the many concurrent requests for it actually fetches from origin, and the rest wait for that one result). Everything below either extends this baseline for a bigger blast radius or handles a different trigger for the same underlying problem.

sequenceDiagram
    participant R1 as Request 1..N
    participant Svc as Service
    participant Lock as Singleflight lock
    participant Origin as Origin / DB
    R1->>Svc: request for expired key
    Svc->>Lock: acquire per-key lock
    alt lock acquired
        Lock->>Origin: single fetch
        Origin-->>Lock: fresh value
        Lock-->>Svc: value + refresh cache
    else lock held by another request
        Lock-->>Svc: wait, then reuse result
    end
    Svc-->>R1: response (fresh or briefly stale)

Preventative techniques (stop the herd from forming)

• Jittered TTLs: instead of a flat TTL, add randomness (for example, base TTL plus or minus 5-20%) so a set of keys populated together do not expire together.
• Staggered renewals: refresh keys on a schedule spread out over time rather than all at once, particularly for keys that were all warmed at the same moment (after a deploy, a region failover, or a bulk cache-priming job).
• Background refreshers: proactively refresh a known hot key before it expires, so readers never actually observe a miss for that key under normal conditions.

Reactive techniques (contain the herd once it starts)

• Request coalescing / singleflight: on a miss, one request fetches from origin while concurrent requests for the same key wait on that result instead of issuing their own origin calls.
• Leader-based warmers: elect one instance (via a lease, not a permanent role) to be responsible for refreshing a given hot key or key range, so refresh work itself does not become another source of duplicate origin load.
• Per-key locks: a more granular form of coalescing, useful when singleflight is implemented per-process and needs a cross-process equivalent (a short-lived distributed lock) to coordinate across many service instances.
• Circuit breakers: once the origin's error rate or latency crosses a threshold, stop sending it more traffic and serve a fallback (often the stale cached value) instead of piling more failing requests onto an already-struggling origin. In plain terms, a breaker has three states: closed, where traffic flows normally while the breaker just watches the error/latency rate; open, where it stops sending calls to the origin entirely once that rate crosses its threshold, serving the fallback instead; and half-open, where it periodically lets one trial request through to check whether the origin has recovered before fully closing again. The full mechanics of breaker states and how they interact with retries go deeper than that summary; here, the breaker's job is specifically to stop a cache-miss storm from becoming an origin-overload storm.
• Rate limits: cap the request rate a single key or a single origin dependency can absorb, as a hard backstop under the other techniques.
• Grace caches serving stale-while-revalidating: keep serving the last known-good value past its nominal expiration while a refresh happens in the background, rather than making every reader wait on that refresh.
• Queuing / coordinator patterns: for the largest blast-radius cases, put the refresh work behind a queue with a single (or small, bounded) set of workers pulling from it, so origin load during a mass-invalidation event is capped by queue-consumer concurrency rather than by however many callers happened to miss at once.

Thundering herd is not only a cache problem

The same failure shape hits connection pools: a backend service recovering from an outage can see every client that was queued or retrying suddenly reconnect and check out a database or HTTP connection at once, exhausting the pool the same way a cache stampede exhausts an origin. The mitigations are the same family of ideas applied to a different resource: jittered reconnect/retry backoff instead of jittered TTL, and a bounded admission queue at the pool instead of request coalescing at the cache.

Worked incident case study: a stale-price outage

A pricing update was pushed, but the cache continued serving stale cached prices for several minutes, an invalidation gap, not a stampede, but the kind of incident that motivates having both invalidation correctness and stampede protection in the same design: a fix that improves one without the other still leaves revenue exposure on the table. The response has three time horizons:

• Immediate: force-invalidate the specific affected keys (or, if scope is unclear, accept a short blanket TTL reduction) to stop the bleeding, while watching origin load closely since a mass invalidation of price keys is itself a thundering-herd trigger, so this step should go out with jittered expiration or through the coordinator pattern above, not as one giant simultaneous purge.
• Short-term: add an event-driven invalidation path (publish an invalidation message on price update) so the next price change does not depend on a TTL window at all.
• Long-term: add a monitoring check that compares cached price against source-of-truth price on a sample basis, so a future invalidation gap is caught by an alert instead of by a customer or an audit.

Worked scenario: a flash sale

A flash sale that invalidates millions of product keys at once is a deliberately engineered thundering herd, not an accidental one, so it needs a pre-sale runbook rather than relying on steady-state defenses alone:

1. Days before: identify the key set that will be invalidated and pre-warm replacement values (new prices, new stock) into a shadow key space.
2. Hours before: switch traffic to read from the shadow key space via a fast pointer/alias flip rather than invalidating the live keys directly, avoiding a simultaneous mass-miss entirely.
3. At sale start: keep circuit breakers and rate limits armed on the origin regardless, as a backstop for any key the pre-warm missed.
4. Post-sale: retire the old key space and confirm hit ratios and origin load returned to baseline.

Concrete trigger: one hot key expiring

The narrowest version of this problem, a single popular key expiring and producing a flood of simultaneous requests that overloads the database, is mitigated specifically by a read-through warmer: a background process that refreshes that specific key proactively, on a schedule tied to its own TTL, so normal readers never observe the miss that would otherwise trigger the flood.

Shipping the mitigation itself safely

Rolling out a stampede fix is its own small project, not a one-shot deploy: use a staged rollout strategy (enable coalescing or jitter for one key namespace or one region first) rather than flipping it on globally, since a bug in the mitigation's own locking logic can create a new outage rather than preventing one. Pair the rollout with validation experiments, a controlled comparison of origin load and P95 (95th-percentile) latency with the mitigation on versus off for equivalent traffic, so the team has evidence the change worked rather than an assumption that it did.

Monitoring and alerting

Track cache miss rate per key (a sudden spike for one key is the earliest signal of a forming herd), waiter/queue depth at the coalescing layer, origin error rate and latency, connection-pool saturation, and circuit-breaker state transitions. Alert on a sharp rise in per-key miss rate or waiter count before it turns into an origin-side incident, not after.

Trade-offs and pitfalls

Local, per-process singleflight combined with TTL jitter is cheap and covers most real traffic patterns; it is the right default. Cross-process coordination (distributed locks, leader election for warmers, a queue-based coordinator) adds real protection for large blast-radius events like a flash sale, but it adds operational complexity, another thing that can itself fail, and some latency overhead from the coordination itself. The main pitfall is reaching for the expensive coordinated techniques everywhere out of caution: that adds failure surface without adding proportionate protection for keys that were never going to produce a large herd in the first place. The second pitfall is validating a mitigation with a synthetic load test but never with a real fault-injection or game-day exercise: a load test proves the happy path holds up under volume, not that the coalescing and breaker logic actually engage correctly the moment a real dependency starts failing.

Direct answer

For a global retailer with bursty, multi-region traffic, neither pure full replication nor pure partitioning wins outright: full replication gives the best latency and simplest rebalancing but pays for it in memory and cross-region sync cost, while partitioning is cheaper but concentrates risk into hotspots when demand shifts. The right default is a hybrid: keep a small, region-local cache of the hottest slice of the catalog fully replicated in every region for latency, and back it with a sharded cache for the long tail, promoting items into the local cache when a region's traffic to them justifies it.

Structured elaboration

Why a global retailer with bursty traffic needs the hybrid, not either extreme

Bursty, multi-region traffic on a retail catalog is rarely uniform: a small set of products (a flash sale, a viral item) drive a disproportionate share of reads at any given time, and which products are hot can shift quickly. Pure partitioning puts that risk on a single shard, since consistent-hashing-style assignment (products and cache shards are placed as points on a circular hash space, so only nearby points move when shards are added; the mechanics of the hash ring itself are covered in more depth under load balancing's consistent-hashing pattern, and what matters here is the caching consequence) doesn't know a key is about to become hot until it already is. Pure full replication avoids that risk entirely but pays a flat memory and cross-region sync tax for the entire long tail of the catalog, most of which is rarely read in any given region.

The hybrid keeps region-local, fully replicated caches sized to each region's actual working set (the products that region's users actually read), backed by a sharded cache holding the full catalog. A traffic-based promotion rule (an item crossing a per-region hit-rate threshold gets pushed into that region's local cache) handles the shifting-hotspot case without requiring the whole catalog to be replicated everywhere.

Worked example

Assume, as a planning input rather than a measured fact, a product catalog sized at 50 GB, served across 6 regions.

Full replication cost:

$50\,\text{GB} \times 6\,\text{regions} = 300\,\text{GB total cache memory}$

Partitioned cost (no duplication, split evenly across 6 shards, plus a replication factor of 2 within each shard for availability rather than for cross-region latency):

$\frac{50\,\text{GB}}{6\,\text{shards}} \approx 8.3\,\text{GB per shard}, \qquad 50\,\text{GB} \times 2 = 100\,\text{GB total with the availability replica}$

The partitioned approach uses roughly a third of the memory of full replication (100 GB versus 300 GB) at this illustrative catalog size. The hybrid sits between the two: if each region's working set is, say, 10% of the catalog (5 GB), replicating just that slice to all 6 regions costs:

$5\,\text{GB} \times 6 = 30\,\text{GB}$

on top of the 100 GB sharded backing store, for roughly 130 GB total, a fraction of full replication's 300 GB while still giving most reads (the ones hitting each region's working set) a local hit.

Trade-offs & pitfalls

• The hybrid's promotion rule needs a threshold and a demotion path; without demotion, the region-local cache grows unbounded as items get promoted but never removed, eventually approaching full replication's cost anyway.
• Cross-region invalidation is still required for the sharded backing store even in the hybrid; underestimating that traffic (versioned, pub/sub-style invalidation messages rather than synchronous broadcasts) is a common way the "cheaper" option ends up not being cheaper.
• A single globally hot product (a flash sale item) can still overload the shard that owns it even with promotion in place, if promotion reacts slower than the traffic spike; this is the scenario that specifically motivates proactive cache warming ahead of known events rather than purely reactive promotion.
• A content delivery network (CDN, a network of edge servers that cache content close to users) is a natural complement for static product assets (images, descriptions) but doesn't solve the dynamic pricing/inventory caching problem this comparison is about; don't conflate the two layers.
• Getting the region-local cache's time-to-live (TTL, how long a cached value is considered valid before refresh) too long trades staleness (wrong price or stock shown) for the latency win; too short and the hybrid starts behaving like the sharded-only design under load.

Direct answer

A shard or partition is "hot" when it receives a disproportionate share of traffic, writes, or resource consumption relative to its peers, so that it becomes a bottleneck (elevated latency, saturated CPU/I/O, or errors) even though the cluster as a whole still has spare capacity. The response has two parts that are easy to conflate but should stay separate: detect hotness quickly with the right signals, then apply a mitigation whose cost matches how long the hotspot is expected to last, from cheap short-term throttling and caching up to an actual re-shard.

Structured elaboration

Detecting hot partitions

• Load and resource metrics: per-shard requests per second (RPS), CPU, I/O, and queue depth, compared against the fleet's own distribution rather than a fixed threshold, since "hot" is relative to peers.
• Latency as a detection signal: a shard's p99 latency drifting upward while its peers stay flat is often a leading indicator of hotness that shows up before request-rate counters cross an alerting threshold, because contention (lock waits, queueing) degrades tail latency before it degrades throughput. Watching per-shard latency distributions, not just per-shard RPS, catches a hotspot earlier.
• Skew statistics: characterize how concentrated traffic is across keys, not just across shards, since a shard can look fine in aggregate while one key inside it is saturating a lock or a cache line. As a worked illustration: suppose telemetry for a system with 1,000,000 distinct keys and 200,000 total requests per second shows that 0.1% of keys account for 50% of request volume. That is $0.001 \times 1{,}000{,}000 = 1{,}000$ hot keys sharing $0.5 \times 200{,}000 = 100{,}000$ requests per second, or $100{,}000 / 1{,}000 = 100$ requests per second per hot key on average, against the remaining $999{,}000$ keys sharing the other $100{,}000$ requests per second, or roughly $100{,}000 / 999{,}000 \approx 0.1$ requests per second per ordinary key. That puts an average hot key at close to $100 / 0.1 \approx 1{,}000\times$ an ordinary key's traffic, which is the kind of skew that breaks any partitioning scheme assuming roughly-even key access.

Mitigation, short-term versus long-term

Short-term (apply within minutes, buy time without touching the data layout):

1. Caching: put a read-through or write-through cache in front of the hot key/shard so repeat reads don't reach the origin store at all.
2. Throttling and per-key rate limiting: apply backpressure (token buckets) specifically on the identified hot keys, so a burst degrades gracefully for that traffic instead of starving unrelated keys on the same shard.
3. Write-aggregation: for a celebrity-event burst of writes to one key (a viral post accumulating likes or comments), batch and aggregate writes at the edge (coalescing many small increments into periodic aggregated updates) before they hit the shard, trading a small amount of write latency for a large reduction in write volume against the hot key.

Long-term (structural, applied once a hotspot is confirmed sustained rather than transient):
4. Adaptive hashing / dedicated placement: move a confirmed-hot key to its own shard, or add salting (appending a small, rotating value to a key before hashing it, so one logical key's traffic spreads across several physical shards instead of landing on just one) or finer virtual-node granularity (giving each physical shard more small positions on the hash ring, the circular numeric space that a consistent-hashing scheme assigns keys and nodes to) so the hashing scheme can isolate it without redesigning the whole ring.
5. Re-sharding / splitting: split the hot partition's range or key-space so the load spreads across more physical shards, the durable fix when a hotspot is sustained rather than transient.

That is five distinct mitigations, and any three of caching, throttling, and re-sharding are enough to answer "how would you mitigate a hot key in a distributed cache in production": start with the cheapest (cache), add throttling if the cache doesn't fully absorb it, and re-shard only if the hotspot persists.

Choosing the underlying partitioning strategy with celebrity-style hot users in mind
The choice between hash-based, range-based, and directory-based partitioning has a direct hotspot consequence:

• Hash-based: spreads keys evenly in the common case but offers no lever to isolate one specific hot key without extra machinery (salting, sub-sharding), because the hash function alone doesn't know which keys are hot.
• Range-based: naturally groups related keys, which is good for locality but bad for hotspots, since a viral single key or a contiguous range containing a celebrity account concentrates load on whichever node owns that range.
• Directory-based: an explicit key-to-shard lookup table, which costs a metadata hop but is the only one of the three that lets you place a specific known-hot key on its own dedicated shard deliberately, rather than hoping the hash function happens to isolate it. For a workload dominated by a small number of celebrity-style hot users, a directory (or a hybrid: hash-based by default, directory override for the confirmed-hot subset) is usually the right choice specifically because it gives you that placement lever.

Trending-topic-driven hotspots and dynamic response
Some hotspots aren't a single hot key but a hot topic (many different keys suddenly correlated by a trending event, all spiking together). A static per-key mitigation doesn't catch this because no single key crosses the threshold; detection needs a topic- or tag-level aggregate metric, and the response is a dynamic re-sharding action triggered automatically off that aggregate signal (spin up additional shard capacity for the affected key range, or fan the trending topic's traffic out across a temporary pool of shards) rather than a human manually re-sharding after the fact.

Worked example

Kafka topic partitioning with a small number of extreme-volume users. A Kafka topic uses the user id as the partition key so that all events for a given user land on the same partition in the order they were produced, which is what preserves per-user ordering downstream. Under normal load this distributes evenly. When a small number of users generate extreme event volume, hashing on plain user_id sends all of that volume to one partition (and the one broker leading it), saturating it while other partitions sit idle. The fix is a compound key: for identified hot users only, key on user_id:bucket, where bucket is a small rotating value (for example, derived from a counter or the event's minute-of-hour), spreading that one user's events across a set of partitions instead of one. This compound key is exactly the salting technique described above (appending a rotating value to a key before hashing it so one logical key's traffic spreads across multiple physical shards), applied selectively to the confirmed-hot user subset rather than to every key in the topic. This is applied selectively (hot-user isolation): ordinary users keep the plain user_id key and their full ordering guarantee; only the confirmed-hot subset gets the compound key. The explicit trade-off is that a hot user's events are no longer strictly totally ordered across the whole topic, only ordered within each bucket, so a consumer that needs a global per-user order for that hot subset has to merge and sort the bucketed sub-streams by timestamp downstream, an added consumer-side cost accepted specifically because the alternative (one partition absorbing the full event volume) would fall behind and delay every user's events on that partition, not just the hot user's.

Trade-offs & pitfalls

• Rolling out a mitigation (a new hashing scheme, a directory override, a compound key) without a verification step, comparing per-shard load, error rate, and latency before and after, in a canary or staged rollout, risks trading a visible hotspot for an invisible correctness bug (as in the Kafka example, silently broken ordering for a subset of users); track the specific metrics the mitigation is supposed to move, not just "did the alert clear."
• Reaching for re-sharding as the first response to every hotspot is expensive and slow to reverse; try caching and throttling first and reserve re-sharding for hotspots that persist past the short-term mitigations' capacity.
• A skew statistic measured once and never rechecked goes stale: traffic concentration shifts as products and user behavior change, so hotspot detection needs to run continuously, not as a one-time audit.
• Assuming directory-based partitioning is strictly better because it handles hot users well ignores its cost: every request now pays a metadata lookup, and the directory itself can become a bottleneck or a single point of failure if it isn't replicated and cached carefully.

Direct answer

A stateless service keeps no client- or session-specific data in process memory between requests: every request carries (or looks up) everything needed to handle it. A stateful service holds that context in memory across requests, such as an open socket or an in-memory session. Statelessness enables horizontal scaling because any instance can serve any request, so a load balancer can distribute traffic with no per-node reconciliation and instances become disposable: add, remove, or replace them freely.

Structured elaboration

Why statelessness is the enabler, mechanically

• Interchangeability: since no instance holds unique data, a request routed to any instance gets the same result, which is what makes simple, even load distribution possible.
• Painless scaling events: adding capacity means starting new instances with no data migration; removing capacity means stopping instances with no data loss, because nothing lived there that mattered.
• Simpler recovery: a crashed stateless instance is replaced, not repaired; there is no in-memory state to reconstruct.
• Simpler rolling deploys: instances can be cycled one at a time without draining session state first.

Strategies to externalize state

• Durable application state moves to a database (relational or NoSQL) instead of process memory.
• Fast shared access to hot data moves to a distributed cache (an in-memory key-value store shared across instances) rather than a per-instance local cache.
• Session data specifically has two common paths: a centralized session store, or client-held tokens (such as a signed JSON Web Token, JWT) that let the server stay stateless entirely because the client presents its own state on every request.
• Large binary objects move to an object store rather than local disk.
• Asynchronous or queued state (work in flight, not yet durable) moves to a message queue.
• Small amounts of shared coordination state (configuration, leader pointers, locks) move to a dedicated coordination service built for that job.

Where a stateful service is genuinely necessary

The SRE angle: operating a stateful service you cannot avoid. When a stateful service is genuinely required, the operational burden shifts from "how do I scale it" to "how do I keep it reliable despite being pinned." That means treating the stateful nodes as a smaller, more carefully managed subset of the fleet: automated health checks tied to a fast failover or re-election path, regular drills that exercise that failover before it is needed under real incident pressure, and monitoring that specifically watches for the failure modes unique to statefulness (a stuck leader, a connection that never drains, a checkpoint that falls behind). The mitigation is never "make it stateless anyway"; it is "shrink the stateful surface to the minimum and instrument that minimum heavily."

Worked example

A web application currently stores logged-in session data in each server's memory, so a user's second request must land on the same server (sticky routing) or their session appears to vanish. To make the tier horizontally scalable: move session data out of process memory into a shared session store, or better, switch to a signed token that the client holds and presents on each request so the server does not need to look anything up at all. Either change means any instance can now serve any request, sticky routing is no longer required, and the fleet can be scaled up or down purely on load, with new instances immediately able to serve full traffic with zero data migration.

Trade-offs & pitfalls

• Do not confuse "stateless service" with "no state exists": the state still exists, it has just moved to a system designed to hold it reliably. That external system becomes a new dependency and a new potential bottleneck, so it needs its own scaling plan.
• Client-side tokens remove server lookups but push size and revocation concerns onto the client and the token design; server-side session stores keep revocation simple but add a network hop and a shared-store scaling problem.
• Treating "sticky sessions" as a free fix for statefulness is a common wrong turn: it works, but it silently reintroduces a form of per-node state ownership and undermines the failure-isolation benefit that horizontal scaling was meant to provide.
• For the genuinely stateful cases, skipping the failover-drill discipline is the most common production failure: the coordination logic works in testing and then fails silently the first time it is needed in an incident, because it was never exercised under realistic conditions.