Scalability Patterns and Techniques Questions

Direct answer

Consistent hashing gives even load distribution and cheap rebalancing but destroys query locality, forcing every range scan or join into a scatter-gather across many nodes; range-based partitioning gives excellent locality for exactly those queries but creates hotspot risk on skewed keys and makes rebalancing expensive. For a datastore that genuinely needs both complex range queries and elastic scaling, the answer is a hybrid: partition coarsely by range on an attribute that preserves most query locality, then hash within each coarse range to spread load and avoid hotspots inside it.

Structured elaboration

What consistent hashing actually is, before comparing properties. Consistent hashing places both data keys and node identifiers as points on a circular numeric range, called a ring, typically by hashing each one with the same hash function. Each key is then owned by whichever node's point is the first one reached going clockwise from the key's own point on the ring. That single mechanism is what produces both properties compared below: rebalancing is cheap because adding or removing one node only reassigns the keys between it and its ring neighbors, not the whole dataset the way a plain modulo hash (hash(key) mod N) would when N changes; and locality is poor because two keys that are numerically close in the original ID space, like adjacent order IDs, get hashed to essentially random, unrelated points on the ring, so there is no guarantee they land near each other, or on the same node, at all.

Consistent hashing

• Query locality: poor. Related keys are scattered pseudo-randomly across nodes by design, so a range scan or a join across a key range has to fan out to many (potentially all) nodes and merge results, a scatter-gather pattern that is expensive and has a tail-latency cost equal to its slowest participant.
• Rebalancing: cheap. Adding or removing a node (especially with virtual nodes) only moves a small, bounded slice of the keyspace.
• Scaling: straightforward; new capacity absorbs a proportional share of load automatically.

Range-based partitioning

• Query locality: strong. Keys that are close in sort order live on the same node, so range scans and joins over a contiguous key range are single- or few-node operations, and storage engines benefit from sequential I/O and better compression on sorted data.
• Rebalancing: expensive. A skewed key distribution concentrates load on one range, and fixing it means splitting or merging large contiguous chunks of data, a much bigger operation than moving a handful of hash buckets.
• Scaling: harder to automate; adding capacity typically requires a deliberate split-planning step rather than capacity absorbing load on its own.

Hybrid approach

1. Choose a coarse partitioning attribute that captures most of the locality your queries need (a time window, a tenant id range, a geographic region) and range-partition on it. This keeps queries that filter or join on that attribute confined to a small number of coarse partitions.
2. Within each coarse range, sub-partition by consistent hashing (or a hash of a secondary key) into several sub-shards spread across physical nodes. This is what prevents a single popular coarse range from becoming a hotspot: instead of one range living on one node, it's spread across the sub-shard set.
3. Maintain a lightweight metadata/routing layer mapping coarse range to its set of sub-shard endpoints, so the query planner knows which sub-shards a given range maps to without a full scatter.
4. Handle skew with adaptive split/merge scoped to the coarse range: if one coarse range's sub-shards are overloaded, split that range's sub-shard count, an operation that only touches that range's data, not the whole dataset.

flowchart TD
    Q[Incoming query: range filter] --> P[Router: which coarse ranges match?]
    P --> R1[Coarse range 1]
    P --> R2[Coarse range 2]
    R1 --> H1[Hash sub-shard a]
    R1 --> H2[Hash sub-shard b]
    R2 --> H3[Hash sub-shard a]
    R2 --> H4[Hash sub-shard b]
    H1 --> M[Merge results]
    H2 --> M
    H3 --> M
    H4 --> M

This bounds scatter-gather to the coarse ranges a query actually overlaps, times the sub-shard fan-out within each, instead of every shard in the cluster.

Worked example

A metadata/search index over 1 billion objects needs to support both point/range lookups by ingestion time and elastic scaling as volume grows. The team range-partitions coarsely by ingestion week, giving roughly 100 coarse ranges for two years of retained data (a stated design choice, not a derived figure), and hash-partitions each coarse range into 50 sub-shards to spread load evenly within the week, giving $100 \times 50 = 5{,}000$ total shards across the cluster. A typical query filtering on a 3-week window touches only the 3 matching coarse ranges, fanning out to $3 \times 50 = 150$ sub-shards, or $150 / 5{,}000 = 3\%$ of the cluster, instead of a full scatter-gather across all 5,000 shards that pure consistent hashing on object id would require for the same query. A query that needs a single object by id, with no time filter, still does a targeted hash lookup within whichever coarse range the object's timestamp maps to, so point lookups stay single- or few-shard regardless of the hybrid layout.

Trade-offs & pitfalls

• The hybrid scheme adds a real second layer of metadata and routing logic (which coarse ranges exist, which sub-shards belong to each) that a pure hash or pure range scheme doesn't need; that complexity has to be justified by an actual mixed workload of range queries plus elastic-scaling needs, not adopted by default.
• Choosing the coarse partitioning attribute poorly (one that doesn't align with how queries actually filter) gives you the rebalancing cost of range partitioning without recovering the locality benefit that was the whole point of choosing it.
• A coarse range that itself becomes hot (a recent, actively-written time window, for instance) still needs the adaptive split/merge step; sub-sharding by hash inside it helps distribute load but doesn't eliminate the need to watch for a coarse range outgrowing its allocated sub-shard count.
• Cross-coarse-range joins (joining two objects that fall in different weeks, say) are not free just because the hybrid scheme handles range queries well; that join still fans out across whichever coarse ranges and sub-shards are involved, so it's worth being explicit about which query shapes the hybrid design is actually optimizing for.

Direct answer

Reach for asynchronous processing when a task doesn't need to block the user's response, has variable or long duration, benefits from being retried independently of the request that triggered it, or needs to scale at a different rate than the request path that produces the work. The trade-off is always the same shape: lower perceived user latency and higher burst-absorption capacity, in exchange for longer end-to-end completion time and real added complexity (idempotency, ordering, observability).

Structured elaboration

Tasks that belong on a queue: long-running jobs (video transcoding, PDF generation, model training), background work (email/SMS delivery, webhook retries, batch extract-transform-load jobs), rate-limited external calls (payment-gateway retries respecting a provider's own limits), and fan-out work (thumbnail generation, notification distribution to many recipients).

Benefits versus trade-offs:

Choosing the messaging technology:

• Simple message queue (point-to-point task queue with retry semantics, one consumer per message): use for straightforward background jobs where ordering isn't required.
• Pub/sub (one message, many independent subscribers): use for event-driven fan-out across decoupled services, where each subscriber reacts to the same event differently.
• Streaming platform (ordered, replayable log, supports stateful consumers): use when ordering, replay, or exactly-once-style processing semantics matter, such as event sourcing or analytics pipelines.

Decision checklist: need ordering or replay, use streaming; need to fan out to many independent consumers, use pub/sub; need a simple worker queue with retries and nothing fancier, use a message queue.

Worked example

Migrating an existing synchronous pipeline to async: a concrete pilot. Say order-confirmation emails are currently sent synchronously inside the checkout request handler, adding real latency to checkout and coupling checkout availability to the email provider's uptime. The migration:

1. Introduce a queue behind the checkout handler; the handler publishes an "order confirmed" message and returns to the user immediately instead of waiting on the email call.
2. Define an idempotency key (the order ID) so that if the message is redelivered (a realistic possibility under at-least-once delivery), the consumer can detect and skip a duplicate send rather than emailing the customer twice.
3. Decide on ordering: for this task, per-order ordering doesn't matter across different orders, so no partition-by-key requirement beyond what's needed for the idempotency check itself.
4. Add a bounded retry (say, 3 attempts with backoff) and a DLQ for emails that still fail, so a transient provider outage doesn't silently drop confirmations.
5. Pilot on a low-traffic checkout path first, and communicate the user-facing change explicitly: the email is no longer guaranteed to have been sent by the time the checkout page renders, which may require a small copy change ("confirmation email on its way" rather than implying it already arrived).

Precompute layer for a legacy synchronous aggregation service. A service that computes an expensive aggregate (say, a rolling usage summary) synchronously on every request can be converted to a precompute pattern: an async job recomputes the aggregate on a schedule or on a triggering event and writes it to a fast-read store; the read path serves from that store. On a cache miss (a query for a window that hasn't been precomputed yet, common right after rollout), the service falls back to computing it synchronously that one time and warms the cache with the result, so misses shrink over time instead of recurring. A one-time backfill job for historical windows at rollout avoids a long tail of slow cache-miss requests immediately after launch.

Batch versus online inference for ML serving. The same sync-versus-async trade-off shows up in model serving: batch (async) inference runs predictions over a group of inputs on a schedule or triggered job, giving higher throughput and lower cost per prediction at the expense of staleness equal to the batch interval; online (sync) inference serves a live request through the model on the request path, giving the freshest possible result but bounded by a per-request latency budget and a harder cost curve to scale at high request volume. The choice depends on whether the feature consuming the prediction can tolerate staleness measured in minutes or hours, or genuinely needs a fresh result per request.

Trade-offs & pitfalls

• Skipping idempotency is the single most common mistake in this pattern. At-least-once delivery is the realistic default for most queueing systems; a consumer that isn't safe to run twice on the same message will eventually double-charge, double-send, or double-write something.
• Ordering requirements are easy to miss until they bite. If two messages about the same entity can be processed out of order and produce a different, wrong final state, that entity's messages need to be routed to preserve order (typically via a partition key), which the simplest queue setups don't guarantee by default.
• The user-facing latency change needs to be communicated, not just implemented. Moving a task from synchronous to asynchronous changes what "done" means from the user's perspective; shipping that silently creates confusing, inconsistent-feeling product behavior even when the backend change is technically correct.
• A precompute layer with an unbounded cache-miss fallback can silently degrade into "synchronous, but slower and more complex." If misses stay common because a backfill never ran, or the precompute schedule can't keep up with query patterns, you've added the complexity of async without capturing its latency benefit.

Direct answer

Resharding a live database with minimal downtime is an online migration: keep the old topology serving traffic while you copy data to the new one under a throttle, use dual writes or change-data-capture (CDC) to keep the copy current, validate it against the source before trusting it, and cut routing over in a small, reversible step rather than a single big-bang switch. The hard parts are rarely the copy itself; they are choosing a shard key/count that won't need redoing soon, keeping writes ordered correctly during the overlap window, and having a rollback that is actually exercised, not just documented.

Structured elaboration

Choosing the new shard key or shard count

• Analyze real access patterns (write hotspots, cardinality, range-vs-hash trade-offs) from production telemetry rather than guessing; target per-shard headroom (commonly keeping any shard well under its saturation point) so the new topology doesn't need revisiting immediately.
• For a hot single shard suffering write contention, the two concrete techniques are rekeying (choosing a shard key that spreads the contended writes, if the current key is the root cause) and range-splitting (dividing the shard's existing key range into two or more ranges, each becoming its own shard, when the key itself is fine but the range has outgrown one node).
• For a lookup-based versus algorithmic shard-mapping layer: an algorithmic mapping (consistent hashing, which places nodes and keys as points on a circular hash space called a ring, so a topology change only reassigns the small slice of keys near it, or the simpler hash-mod-N) needs no metadata lookup and scales trivially, but it distributes tenants roughly evenly by key, which fails badly when tenant sizes vary by orders of magnitude (a large enterprise tenant next to hundreds of tiny ones). A lookup-based mapping (an explicit tenant-to-shard table) costs a metadata hop per request but lets you place large tenants on dedicated shards and pack small tenants together deliberately. For a multi-tenant system with that kind of size skew, prefer lookup-based mapping, or a hybrid: algorithmic mapping for the default pool of ordinary tenants, with a lookup override for the small number of outsized ones.

Data-migration strategy

• Online copy under CDC: stream changes from the source shard(s) via a CDC pipeline (e.g., a log-based change stream) into the target topology while a background job bulk-copies existing data in ranges. The CDC stream and the bulk copy have to be reconciled (apply CDC events that land inside an already-copied range, buffer or replay ones that land ahead of the copy cursor).
• Dual writes with a quiesce window: to guarantee transaction ordering is preserved for records that are actively being migrated, briefly pause (quiesce) writes to the specific key range being cut over, drain the in-flight CDC backlog for that range so source and target agree, flip routing for just that range, then resume writes. This bounds the ordering risk to a small, deliberately-chosen window instead of trying to reason about ordering across an open-ended dual-write period for the whole dataset.
• Split-and-merge at scale: for a large, contended shard, resharding is a split (one shard's range divides into two, each getting its own routing-table entry) rather than a full-dataset migration; the reverse operation, merge, combines two under-loaded shards back into one when growth was over-provisioned. At terabyte scale, both operations are driven by range boundaries and routing-table updates rather than moving the whole dataset, which keeps a single split or merge bounded in time and blast radius.
• A cross-topology special case: migrating a stateful store (for example, a session store) from a single-region deployment to a multi-region sharded cluster follows the same dual-write skeleton, but adds a proxy layer in front of both topologies so the application never talks to the store directly during the migration; the proxy performs parity verification (comparing responses from the old and new paths for the same key) before fully cutting traffic over, which catches migration bugs before they're user-visible.

flowchart TD
    A[Old shard topology] -->|CDC stream| B[Change pipeline]
    A -->|throttled bulk copy| C[New shard topology]
    B --> C
    C --> D[Shadow reads: compare old vs new]
    D -->|parity confirmed| E[Routing cutover, range by range]
    D -->|mismatch| F[Pause cutover, reconcile]
    E --> G[Quiesce window per range: drain, flip, resume]
    G --> H[Old topology retired]

Routing updates
Introduce a migration-aware routing layer (gateway or client-side routing table) so application code never encodes shard topology directly. Roll the new routing out in stages: internal/test traffic, a small canary of production traffic, then full cutover, with each stage able to fall back to the previous routing table without a code deploy.

Throttling
Throttle the copy and CDC-apply workers adaptively against replication lag, tail latency, and source-shard CPU/I/O, not a fixed rate, so migration traffic backs off automatically when it's competing with production load. As a worked illustration of sizing a per-shard step against an operational target: suppose the team's service-level agreement (SLA) is for each per-shard rebalance step to complete in well under two minutes even across a 200+ node cluster, and a given shard holds roughly 50 GB of data with the copier capped at 500 MB/s per shard to protect production I/O. The data-transfer portion alone takes

which leaves about 20 seconds of headroom under the 120-second target for updating ring/routing metadata and completing leader election (the process by which the new shard's replica-set nodes agree on which one of them becomes the active primary/writer) for the new shard's replica set, before the step is considered done. That headroom is the number to watch: if metadata propagation or leader election routinely eats into it, the fix is a faster metadata path, not a bigger copy-bandwidth cap.

Automating rebalancing across many shards
When rebalancing hundreds of shards simultaneously rather than one at a time, the automation has to cap total concurrent migrations (global throttle, not just per-shard), and it must keep secondary-index consistency in scope explicitly: an index that's rebuilt or re-pointed on a different schedule than its base table will silently return wrong results for the window in between, so index cutover needs to be part of the same per-range quiesce step as the base data, not a follow-up job.

Validation

• Row counts and per-range checksums between source and target.
• Shadow reads: serve a fraction of real production reads from the new topology in parallel with the old one (without returning the shadow result to the client) and compare, which surfaces correctness bugs under real traffic patterns before any client depends on the new path.
• Client-side cache invalidation during cutover: if clients or edge caches hold entries keyed by the old routing, cutting a range over without invalidating those cached entries serves stale-routed responses; the cutover step has to include a cache-invalidation signal (a version bump on the routing key, or explicit invalidation) for anything caching downstream of the router.
• End-to-end application-level smoke tests against the new topology before removing fallback to the old one.

Rollback
Keep dual writes (or the CDC stream) active until validation passes; if a problem surfaces, stop applying new CDC events, revert routing to the old topology for the affected range, and resume from the last confirmed-consistent checkpoint rather than restarting the whole migration. Because cutover happens range-by-range behind a quiesce window, rollback scope is bounded to the ranges already flipped, not the whole dataset.

Trade-offs & pitfalls

• A migration that skips the quiesce window to avoid any write pause will have an ordering gap for records touched during the exact moment of cutover; a short, well-instrumented pause on a narrow key range is a better trade than an unbounded consistency risk.
• Algorithmic shard mapping is simpler to build and reason about, but for multi-tenant systems with orders-of-magnitude tenant-size variance, defaulting to it produces shards that are balanced by tenant count and wildly unbalanced by load; verify the mapping choice against actual per-tenant load, not tenant count.
• Skipping shadow reads to save migration time is the single most common way correctness bugs escape into production, because checksums catch data-copy errors but not query-behavior differences (an index that returns results in a different order, a range boundary edge case).
• Treating secondary-index rebuild as a background job that can lag the base-table cutover is a frequent source of silent incorrect reads immediately after a migration; keep index and base-data cutover atomic per range.

Direct answer

A pragmatic decomposition starts from real bounded contexts, not code modules, keeps data integrity intact by translating between the old and new models instead of cutting over all at once, and replaces any temptation toward a distributed two-phase commit with sagas built from idempotent, compensable steps. The strangler fig pattern, routing traffic through a facade so old and new implementations coexist during the transition, is the mechanism that makes all of this incremental and reversible rather than a risky big-bang cutover.

Identifying service boundaries

Use domain-driven design to find bounded contexts along business-capability lines (orders, billing, inventory) rather than along existing code-module lines, which usually reflect implementation history more than the current business domain. Within a candidate boundary, look for a vertical slice: API (application programming interface), business logic, and the data that logic owns, so the resulting service can operate independently without still calling back into the monolith for its own core data. Start with a boundary that is both low-risk (not on the critical write path) and clearly bounded (its data isn't heavily shared with other domains), since that combination validates the migration mechanics without betting the business on the first attempt.

Data integrity during migration

• Anti-corruption layer (ACL): a translation layer between the monolith's data model and the new service's API, so the new service isn't forced to adopt the monolith's legacy shape, and the monolith doesn't need to understand the new service's internals.
• Change data capture (CDC): stream changes from the monolith's database into the new service so reads stay consistent while write ownership is still transitioning.
• Dual writes, done carefully: if the monolith and the new service both need to write during the transition, make every write idempotent and reconcile the two stores regularly; prefer event-driven replication (the monolith publishes a change event that the new service consumes) over synchronous dual-writes, which fail if either side is briefly unavailable.

Avoiding distributed-transaction anti-patterns

Don't reach for two-phase commit (2PC) across the new service boundary. It requires every participant to be available and responsive during the commit window, which directly works against the availability and independent-deployability goals that motivated decomposition in the first place. Instead, model multi-step business flows as sagas: a sequence of local transactions, each with a defined compensating action if a later step fails, so the overall flow degrades gracefully instead of blocking on a distributed lock.

A common practical split is to orchestrate the core, high-stakes transaction (say, order to payment to shipment) where compensations matter, and let secondary, lower-stakes integrations (send a confirmation email, update an analytics feed) stay choreographed.

When to reach for the strangler fig pattern

Reach for it whenever you need the old and new implementations to coexist safely during a transition, which in practice is almost always for anything beyond a trivial, low-traffic domain. The mechanics, applied specifically at a monolith's gateway:

• Reverse-proxy routing: put a reverse proxy or API gateway in front of both the monolith and the new service, and route requests to one or the other by path, feature flag, or percentage, so the switch is a routing change rather than a deployment event.
• Traffic shadowing: mirror a copy of live production requests to the new service without using its response, so you can compare its behavior and performance against the monolith's real response under real traffic before the new service is trusted to actually serve anyone.

flowchart LR
  C[Client] --> RP[Reverse proxy /<br/>API gateway]
  RP -->|serves response| M[Monolith]
  RP -.->|mirrored, response discarded| NS[New service]
  RP -->|once validated,<br/>cut over by %| NS

Only retire the monolith's code path for a given slice once traffic has been fully cut over and the data behind it has been reconciled, not merely once the new service has been deployed.

Worked example: extracting a payments capability

Define the Payments bounded context and its API contract, then stand up an ACL that maps the monolith's existing payment calls onto that contract. Use CDC to populate the new Payments service's data store from the monolith's database so both stay consistent during the transition. Put a reverse proxy in front of the payments endpoints and start with traffic shadowing: every real payment request is also sent to the new service, its result compared against the monolith's, but only the monolith's response reaches the client. During this shadow period, downstream systems the payments path calls (a fraud check, a ledger write) effectively see close to double their steady-state call volume, since both the monolith and the shadowed new service are calling them, which is worth flagging to whoever owns that downstream capacity before shadowing begins. Once results match consistently, cut real traffic over gradually (a small percentage, then a majority, then all of it) via the reverse proxy, with the monolith's code path kept in place and able to take traffic back at any stage until the migration is fully reconciled.

Trade-offs and pitfalls

• Treating dual writes as sufficient data-integrity strategy on their own, without idempotency keys or a reconciliation job; a single missed or duplicated write during the transition window leaves the two stores silently inconsistent.
• Reaching for orchestration everywhere out of caution; it adds a coordinator every downstream service must trust and depend on, which is unnecessary overhead for flows that don't actually need centralized compensation logic.
• Shadowing traffic without accounting for its load on shared downstream dependencies, which can turn a validation exercise into a self-inflicted capacity incident.
• Declaring the migration done once the new service is receiving 100% of traffic, while the monolith's old code path and data are still live as a fallback; the migration isn't actually finished, and the risk isn't actually retired, until that fallback is deliberately removed.

Direct answer

There are three practical ways to scale a stateful service that keeps per-user ephemeral data in memory: keep the state local and pin each user to one instance (sticky sessions), replicate the state across instances so any of them can serve a user (active-active replication), or move the state out of the instances entirely into a shared store (externalization). Sticky sessions are the cheapest to build but the most fragile; externalization is the most operationally mature and is the default recommendation for anything beyond a small fleet; active-active replication is the highest-complexity option, justified mainly when you need both local-memory latency and multi-node resilience together.

Structured elaboration

1. Sticky sessions (session affinity at the load balancer)

• How it works: the load balancer hashes a stable client identifier and routes all of that client's requests to the same instance, which keeps the session in local memory. The routing mechanism itself (which hashing scheme, which health checks) is load-balancer territory; what matters for scaling the service is the consequence: correctness now depends on that one instance staying up.
• Operational complexity: low to moderate. Configure affinity on the load balancer, size instance capacity for peak per-node session count, add health checks and connection draining so deploys don't silently drop sessions.
• Failure modes: an instance failing loses every session pinned to it unless you add a fallback (short-time-to-live (TTL) backup write, or accept the loss and force re-authentication). Autoscaling and rolling deploys cause rebalancing, which cold-starts sessions on whichever instance a client lands on next. Hot users concentrated on one instance cause uneven load that autoscaling on aggregate metrics won't catch.
• Performance: the best case for latency, since reads and writes never leave process memory. Effective capacity is limited by the least-loaded-instance problem: total fleet capacity is high, but any single hot instance can be saturated while others are idle.

2. Active-active replication

• How it works: session state is synchronously or asynchronously replicated across a set of instances (via a gossip protocol, where nodes periodically exchange state with random peers; a consensus protocol, where nodes vote to agree on a value before it counts as committed; or conflict-free replicated data types, data structures built to merge concurrent updates without conflicts) so more than one node holds a current copy and can serve a read or accept a write.
• Operational complexity: high. You are now running a small distributed system inside your application tier: membership management, conflict resolution for concurrent writes to the same session, and monitoring for replication health.
• Failure modes: network partitions are the central risk. Without a consensus mechanism enforcing a single writer or a quorum, a partition produces split-brain, where two nodes both believe they hold the authoritative copy of a session and diverge. Even without a partition, replication lag means a read on one node can be stale relative to a very recent write on another.
• Performance: write latency goes up (synchronous replication waits on other nodes; asynchronous replication is fast but reintroduces the staleness problem it was meant to avoid). Read latency can stay low if reads are served from local replicas. Write throughput is bounded by replication and, if used, consensus overhead.

3. Externalizing state (shared cache or durable store)

• How it works: instances stop holding session state at all. Every request reads from and writes to a shared store (Redis, Memcached, or a durable database), which makes every instance interchangeable.
• Operational complexity: moderate. You now operate one more piece of infrastructure (a cache or database cluster: sizing, eviction policy if it's a cache, persistence if it needs to be durable) and the application changes to fetch and save state instead of holding it.
• Failure modes: the store itself becomes the thing that must not go down, so it needs its own high availability, replicas, and a defined fallback if it's briefly unreachable. Cache-invalidation bugs (serving stale state after a write, or evicting state that should have survived) are the main correctness risk. Network dependency between every app request and the store adds a new failure surface that didn't exist with local memory.
• Performance: a network hop per read/write adds latency relative to local memory, but it is predictable and independent of which instance serves the request, so it scales cleanly with the store's own capacity. A local read-through cache in front of the store recovers most of the latency for repeat reads.

Worked example

Consider a live-chat product where each open chat session tracks typing indicators, unread counts, and the last 20 messages in a rolling buffer, all ephemeral and rebuilt from durable message history if lost. The team starts with sticky sessions during the MVP: single-digit instance count, acceptable risk of a lost typing-indicator state on deploy. At 30 instances and daily autoscaling swings, rebalancing during scale-events starts dropping a visible fraction of active typing indicators on every scale-up, and hot users (large group chats) overload single instances that aggregate metrics don't flag as unhealthy. The team considers active-active replication to keep the local-memory latency but rejects it: the correctness bar for typing indicators is low (occasional staleness is invisible to users), so the complexity of consensus or CRDTs isn't earning its keep. They externalize instead: session state moves to a Redis cluster, keyed by chat id, with a short TTL so abandoned sessions clean up automatically. Every instance becomes interchangeable, rebalancing no longer drops state, and the added latency (one Redis round trip per message event) is well under the product's tolerance for a chat UI.

Trade-offs & pitfalls

• Choosing active-active replication for its latency profile without a real need for both local-memory speed and multi-node write availability at once is the most common overreach; most services that reach for it would be better served by externalization plus a local read-through cache.
• Assuming sticky sessions "just work" at scale and only discovering the rebalancing-churn problem after autoscaling is already live is a frequent operational surprise; load-test the rebalance path, not just steady-state.
• With externalized state, forgetting that the store is now a dependency of every request (not an optional side-system) leads to under-provisioning its availability relative to the application tier it's supposed to be simplifying.
• With active-active replication, treating eventual consistency as "good enough" without checking whether the specific session data has a correctness requirement (e.g., a shopping cart total versus a typing indicator) can silently introduce bugs that only show up under network partition, which is exactly when they're hardest to debug.