Infrastructure Scaling, Capacity Planning, and High Availability Questions

The two approaches

• Sticky sessions: the load balancer routes a given user's requests to the same backend instance every time (usually via a cookie), and that instance holds the cart state in memory. It is fast, since there is no extra network hop to fetch state, but it directly fights horizontal autoscaling: scaling in an instance loses whatever cart state it was holding, and the load balancer has to respect affinity, which complicates even traffic distribution across instances.
• Externalized session state: cart state lives in a shared, fast key-value store (for example a replicated in-memory cache), keyed by user or session ID, and any application instance can read and write it. Any instance can now serve any request, which is what makes clean horizontal autoscaling and instant failover possible; the cost is a small extra network round trip on every cart read or write.

Recommendation

Externalize the state. A shopping cart's affinity requirement is really a data-locality requirement, not a compute-locality one: the cart data needs to be reachable from wherever the request lands, it does not need to live inside the same process that is handling the request. Externalizing it solves both the affinity problem and the autoscaling problem at once, at the cost of a small, well-understood latency addition.

Trade-offs

• Consistency: with sticky sessions, cart state simply vanishes if that one instance is lost, a hard failure, not just staleness. With an externalized store as the single source of truth, cart data survives any individual app instance dying; the remaining consistency question shifts to the store itself (if it is replicated, is it strongly or eventually consistent), which is a much better-understood and more manageable problem than "we lost this user's cart."
• Latency: externalizing adds a network round trip to fetch and update cart state on every relevant request, typically single-digit milliseconds for a well-placed in-memory store, an acceptable cost given what it buys in resilience and scaling flexibility.
• Failover: sticky sessions make failover destructive by construction, an instance failing means its carts are gone unless painstakingly reconstructed. Externalized state makes failover close to invisible to the user, since a new instance can pick up exactly where the old one left off by simply reading the same key.
• Operational complexity: sticky sessions are simple at the load-balancer level but push complexity into every part of the system that has to respect affinity (deploys, scale-in, health checks). Externalizing state removes that constraint from the app tier but adds a new stateful dependency (the shared store) that itself needs to be made highly available, so the statefulness has not disappeared, it has moved somewhere more manageable.
• Cost: a managed, replicated cache cluster is a real, ongoing cost line, but it is usually far cheaper in practice than the engineering cost of building and maintaining correct sticky-session behavior across deploys, health checks, and autoscaling events.

Bottom line: keep the application tier fully stateless, externalize cart state to a fast, replicated shared store, and avoid sticky sessions entirely, since the small latency cost of the extra hop is consistently a better trade than tying user experience to the survival of one specific instance.

Round-robin

How it works: cycles through backends in order, sending each new request to the next one in the list.
Performs well: stateless services where every request costs roughly the same amount of work, so equal distribution by count is a good proxy for equal distribution by actual load, for example a simple content-serving API.
Performs poorly: when request cost varies a lot (round-robin can send an expensive request and a cheap request to the same backend back to back and call it "fair"), or when backend capacity is uneven, since round-robin has no concept of a backend being slower or busier.
Example service: a stateless, read-only REST API behind a fleet of identically sized instances.

Least-connections

How it works: sends each new request to whichever backend currently has the fewest active connections.
Performs well: workloads with variable request duration, since it naturally avoids piling more work onto a backend already busy with long-running requests, better than round-robin's blind cycling.
Performs poorly: when "active connections" doesn't actually track real load, for example long-lived idle connections like WebSockets or keep-alive connections inflate the connection count without reflecting real work, or when per-request cost varies so much that "fewest connections" still isn't "least actual work."
Example service: a general-purpose backend API with mixed-duration requests, where some endpoints are fast lookups and others do heavier processing.

Consistent hashing

How it works: maps each request to a backend deterministically based on a hash of some key, like client IP, session ID, or cache key, so the same key routes to the same backend repeatedly, and only a small fraction of keys remap when a backend is added or removed, unlike plain modulo hashing, where adding one backend remaps almost everything.
Performs well: stateful services or anything where routing the same key to the same backend repeatedly has real value, most commonly cache locality, for example a caching proxy tier where hitting the same backend for the same key means hitting a warm local cache instead of a cold one.
Performs poorly: when request cost is very uneven across keys, since a "hot key" can overload one backend while others sit idle (the hash doesn't know or care about load, only about the key), and it adds real operational complexity: rebalancing cost when the backend set changes, and needing a fallback strategy when a backend is unhealthy.
Example service: a distributed cache tier, like a fleet of Redis proxies, or a stateful session-affinity requirement where losing the mapping is expensive.

Metrics to collect first

• Per-endpoint or per-feature request rate over time, and specifically its variance: does a given feature's traffic spike independently of the rest of the application, or does it always move roughly in step with overall traffic?
• Per-feature resource consumption (CPU, memory, IO attributable to that specific code path), gathered via profiling or application performance tracing, since raw request count alone does not tell you how expensive a feature actually is to serve.
• Correlation between features' load patterns over time: if two features' traffic always rises and falls together, there is little benefit to scaling them separately, since they would need to scale together anyway.

Dependency analysis

• Trace inter-feature calls and shared data access: does the feature you are considering extracting need to call back into the rest of the monolith constantly to do its job? If so, extracting it adds network overhead and operational surface without real isolation, since it remains tightly coupled in practice even if it is deployed separately.
• Identify which features share the same database tables or the same hot rows: tight data coupling argues for keeping features co-located, at least until the data layer itself can also be separated, since splitting the compute tier while the data tier stays shared does not remove the real bottleneck.
• Identify features that are largely self-contained, with their own data and minimal calls back into the rest of the system: these are the strongest candidates for extraction, since they can genuinely scale on their own.

The extraction decision

Rank features by three things: load variance (does its traffic spike independently), resource intensity (is it expensive to serve), and coupling (how tightly is it tied to shared code or data). Extract first whichever features score high on variance and resource intensity but low on coupling, since those get the largest scaling benefit for the smallest added operational cost. A low-traffic, cheap feature is rarely worth extracting even if perfectly self-contained, since dedicated infrastructure for it is pure overhead with no meaningful scaling benefit to justify it.

Cost vs operational complexity trade-offs

Extracting a feature into its own independently scalable service means you stop paying to scale unrelated code just because one feature spikes, and you gain blast-radius isolation (a problem in that one feature does not directly threaten the rest of the monolith's capacity). It costs you a new deployable to operate, its own monitoring and alerting, a network call where there used to be an in-process function call, and potentially new data-consistency questions if the underlying data eventually needs to be separated too. Keep features co-located when their load moves together, when they are tightly coupled to shared data, or when the traffic and resource cost are simply too low to justify dedicated infrastructure regardless of how self-contained the code is.

A concrete worked example: suppose a monolith serves a search feature and an account-settings feature. Search shows sharp, independent traffic spikes tied to marketing campaigns and is CPU-heavy; account settings has low, steady traffic and touches the same user table search barely reads from. Search is a strong extraction candidate (high variance, high resource cost, low coupling); account settings is not (low traffic, low resource cost, tightly coupled to shared user data), so it stays co-located in the monolith.

What it is and why it matters

Connection draining, also called deregistration delay on some cloud load balancers, is the process of stopping new requests from being routed to an instance that's about to be removed, while letting its already-in-flight requests finish normally, instead of killing the instance immediately and dropping whatever it was in the middle of serving.

Cloud load balancers

Set a deregistration delay, the maximum time the load balancer keeps forwarding health checks as "draining" and allows existing connections to complete before force-closing them, typically 30 to 300 seconds depending on typical request duration. A fast JSON API might use 30 seconds, while a service with longer-running requests, like report generation or a streaming response, needs something closer to 120 to 300 seconds.

NGINX

On receiving a termination signal, an operator or orchestrator issues a graceful shutdown command rather than an immediate stop. Graceful shutdown stops accepting new connections on the listening socket right away but lets worker processes finish serving requests already in progress before exiting.

Kubernetes

Uses a preStop hook combined with a configured terminationGracePeriodSeconds value. The sequence for a pod being terminated: Kubernetes first removes the pod from the Service's endpoint list, so in-cluster routing stops sending it new traffic, then runs the preStop hook, commonly just a short pause of a few seconds to cover the propagation delay before every routing component has actually noticed the endpoint removal, then sends a termination signal (SIGTERM, the standard "please shut down" signal) to the container's main process, which should catch it and stop accepting new work while finishing in-flight requests, and finally force-kills it (SIGKILL) if the process hasn't exited once the grace period, commonly 30 to 60 seconds, longer for long-running requests, has elapsed.

Termination-signal-to-shutdown walkthrough

Putting it together for a typical HTTP service: termination is requested; the instance is removed from load-balancer or Service routing so no new requests arrive; a short buffer of a few seconds covers routing-table propagation lag; the application receives the termination signal and stops accepting new connections while letting in-flight requests complete; once all in-flight requests finish or the grace period expires, whichever comes first, the process exits cleanly; if it hasn't exited by the deadline, a forced kill ends it, which should be rare and usually indicates the grace period is set too short or a request handler is stuck.

Choosing the drain duration

Set it close to the p99 (99th-percentile) request duration, not the average: a duration long enough that the overwhelming majority of in-flight requests finish naturally is worth far more than shaving a few seconds off deployment speed. For long-lived connections, like WebSockets, gRPC streams, or Server-Sent Events, a fixed drain timeout doesn't really work the same way, since those connections can legitimately stay open for a long time; the practical approach there is closer to actively signaling the client to reconnect elsewhere than passively waiting out a drain timer.

Interaction with autoscaling and rolling deployments

Scale-in events must go through the same draining path as any other termination; an autoscaler removing an instance without draining is exactly the "kill it immediately" failure mode this whole mechanism exists to prevent, so scale-down cooldowns should always assume drain time is part of the total time to actually free that capacity, not instantaneous. Rolling deployments should combine draining with a maximum-unavailable or surge setting (surge: temporarily running more instances than the target count during the rollout, so replacements are up before old ones are removed) that only removes old instances after their replacements are confirmed healthy and after the old instance has finished draining, so capacity and in-flight requests are both protected during the rollout, not just one or the other.

Stateless versus stateful

A stateless service handles each request using only the information in that request, plus maybe a shared external store it reads/writes, it does not keep anything in its own memory that the next request depends on. A stateful service keeps something request-to-request that lives only on that instance: in-memory session data, a local cache, an open connection with accumulated context.

Why statelessness makes load balancing and autoscaling easier

If every instance is interchangeable (no instance knows anything the others don't), a load balancer can send any request to any healthy instance with zero coordination, and an autoscaler can add or remove instances freely: a new instance is immediately as capable as any existing one, and removing an instance loses nothing except the request it was mid-handling. Stateful instances break both assumptions: the load balancer has to route related requests to the specific instance that holds the relevant state (sticky routing), and the autoscaler has to worry about where that state goes when it scales an instance in, since just terminating it can lose or orphan data.

Scaling a genuinely stateful component horizontally

There is no way around dealing with the state directly; three real strategies, each with a real trade-off:

• Sharding: split the state itself across multiple instances by a key, similar to database sharding, each instance owns a slice of the total state. Scales well and each shard is simpler than one giant instance, but needs a routing layer to find the right shard, and rebalancing shards when adding or removing instances is real engineering work.
• Sticky sessions: keep the state local to one instance, but have the load balancer consistently route a given client's requests to that same instance. Simple to implement, but that instance becomes a mini single point of failure for that client's session, scaling that instance in without a graceful drain loses or disrupts the session, and it can unbalance load if some sessions are much heavier than others.
• Externalizing state: move the state out of the instance entirely into a shared external store (a database, Redis, a distributed cache) that every instance can read and write, so the instances themselves become stateless again from the load balancer's point of view. Usually the cleanest long-term answer, since it gets back to the easy stateless-scaling case, but adds a new dependency, that external store, which now has to be scaled and made highly available itself, and adds a network hop's worth of latency to every state access.

Choosing between them is really about where the complexity should live: sharding and sticky sessions keep it inside the service, at the cost of a smarter routing/instance-management layer, while externalizing moves it into a separate, purpose-built stateful system, usually the better trade unless that system itself becomes the bottleneck.