Scalability Patterns and Techniques Questions
Scaling a system to handle growth in traffic and data: horizontal versus vertical scaling, statelessness, sharding and partitioning strategies, read replicas, and connection pooling. Covers capacity estimation, identifying bottlenecks, and the tradeoffs each scaling axis introduces. The general toolkit for taking a design from thousands to millions of users.
When should you reach for asynchronous processing in a cloud architecture? Give examples of tasks that belong on a queue, the benefits and trade-offs (latency, throughput, complexity), and how you'd choose between a simple message queue, a pub/sub system, and a streaming platform for different workloads.
Sample Answer
Direct answer
Reach for asynchronous processing when a task doesn't need to block the user's response, has variable or long duration, benefits from being retried independently of the request that triggered it, or needs to scale at a different rate than the request path that produces the work. The trade-off is always the same shape: lower perceived user latency and higher burst-absorption capacity, in exchange for longer end-to-end completion time and real added complexity (idempotency, ordering, observability).
Structured elaboration
Tasks that belong on a queue: long-running jobs (video transcoding, PDF generation, model training), background work (email/SMS delivery, webhook retries, batch extract-transform-load jobs), rate-limited external calls (payment-gateway retries respecting a provider's own limits), and fan-out work (thumbnail generation, notification distribution to many recipients).
Benefits versus trade-offs:
| Dimension | Sync | Async |
|---|---|---|
| Perceived latency | Higher (user waits for the full task) | Lower (user gets an immediate acknowledgment) |
| End-to-end completion | Bounded by the request | Can be longer; work finishes on its own timeline |
| Throughput under burst | Limited by request-handling capacity | Higher; a queue absorbs bursts the request path can't |
| Complexity | Lower | Higher (idempotency, ordering, retry/dead-letter queue (DLQ), status tracking, monitoring) |
Choosing the messaging technology:
- Simple message queue (point-to-point task queue with retry semantics, one consumer per message): use for straightforward background jobs where ordering isn't required.
- Pub/sub (one message, many independent subscribers): use for event-driven fan-out across decoupled services, where each subscriber reacts to the same event differently.
- Streaming platform (ordered, replayable log, supports stateful consumers): use when ordering, replay, or exactly-once-style processing semantics matter, such as event sourcing or analytics pipelines.
Decision checklist: need ordering or replay, use streaming; need to fan out to many independent consumers, use pub/sub; need a simple worker queue with retries and nothing fancier, use a message queue.
Worked example
Migrating an existing synchronous pipeline to async: a concrete pilot. Say order-confirmation emails are currently sent synchronously inside the checkout request handler, adding real latency to checkout and coupling checkout availability to the email provider's uptime. The migration:
- Introduce a queue behind the checkout handler; the handler publishes an "order confirmed" message and returns to the user immediately instead of waiting on the email call.
- Define an idempotency key (the order ID) so that if the message is redelivered (a realistic possibility under at-least-once delivery), the consumer can detect and skip a duplicate send rather than emailing the customer twice.
- Decide on ordering: for this task, per-order ordering doesn't matter across different orders, so no partition-by-key requirement beyond what's needed for the idempotency check itself.
- Add a bounded retry (say, 3 attempts with backoff) and a DLQ for emails that still fail, so a transient provider outage doesn't silently drop confirmations.
- Pilot on a low-traffic checkout path first, and communicate the user-facing change explicitly: the email is no longer guaranteed to have been sent by the time the checkout page renders, which may require a small copy change ("confirmation email on its way" rather than implying it already arrived).
Precompute layer for a legacy synchronous aggregation service. A service that computes an expensive aggregate (say, a rolling usage summary) synchronously on every request can be converted to a precompute pattern: an async job recomputes the aggregate on a schedule or on a triggering event and writes it to a fast-read store; the read path serves from that store. On a cache miss (a query for a window that hasn't been precomputed yet, common right after rollout), the service falls back to computing it synchronously that one time and warms the cache with the result, so misses shrink over time instead of recurring. A one-time backfill job for historical windows at rollout avoids a long tail of slow cache-miss requests immediately after launch.
Batch versus online inference for ML serving. The same sync-versus-async trade-off shows up in model serving: batch (async) inference runs predictions over a group of inputs on a schedule or triggered job, giving higher throughput and lower cost per prediction at the expense of staleness equal to the batch interval; online (sync) inference serves a live request through the model on the request path, giving the freshest possible result but bounded by a per-request latency budget and a harder cost curve to scale at high request volume. The choice depends on whether the feature consuming the prediction can tolerate staleness measured in minutes or hours, or genuinely needs a fresh result per request.
Trade-offs & pitfalls
- Skipping idempotency is the single most common mistake in this pattern. At-least-once delivery is the realistic default for most queueing systems; a consumer that isn't safe to run twice on the same message will eventually double-charge, double-send, or double-write something.
- Ordering requirements are easy to miss until they bite. If two messages about the same entity can be processed out of order and produce a different, wrong final state, that entity's messages need to be routed to preserve order (typically via a partition key), which the simplest queue setups don't guarantee by default.
- The user-facing latency change needs to be communicated, not just implemented. Moving a task from synchronous to asynchronous changes what "done" means from the user's perspective; shipping that silently creates confusing, inconsistent-feeling product behavior even when the backend change is technically correct.
- A precompute layer with an unbounded cache-miss fallback can silently degrade into "synchronous, but slower and more complex." If misses stay common because a backfill never ran, or the precompute schedule can't keep up with query patterns, you've added the complexity of async without capturing its latency benefit.
Design an operational plan and technical implementation to reshard a live sharded database with minimal downtime. Cover choosing the new shard key or shard count, the data-migration strategy (online migration, dual writes, change-data-capture), routing updates, throttling the migration, validation steps, and your rollback procedure.
Sample Answer
Direct answer
Resharding a live database with minimal downtime is an online migration: keep the old topology serving traffic while you copy data to the new one under a throttle, use dual writes or change-data-capture (CDC) to keep the copy current, validate it against the source before trusting it, and cut routing over in a small, reversible step rather than a single big-bang switch. The hard parts are rarely the copy itself; they are choosing a shard key/count that won't need redoing soon, keeping writes ordered correctly during the overlap window, and having a rollback that is actually exercised, not just documented.
Structured elaboration
Choosing the new shard key or shard count
- Analyze real access patterns (write hotspots, cardinality, range-vs-hash trade-offs) from production telemetry rather than guessing; target per-shard headroom (commonly keeping any shard well under its saturation point) so the new topology doesn't need revisiting immediately.
- For a hot single shard suffering write contention, the two concrete techniques are rekeying (choosing a shard key that spreads the contended writes, if the current key is the root cause) and range-splitting (dividing the shard's existing key range into two or more ranges, each becoming its own shard, when the key itself is fine but the range has outgrown one node).
- For a lookup-based versus algorithmic shard-mapping layer: an algorithmic mapping (consistent hashing, which places nodes and keys as points on a circular hash space called a ring, so a topology change only reassigns the small slice of keys near it, or the simpler hash-mod-N) needs no metadata lookup and scales trivially, but it distributes tenants roughly evenly by key, which fails badly when tenant sizes vary by orders of magnitude (a large enterprise tenant next to hundreds of tiny ones). A lookup-based mapping (an explicit tenant-to-shard table) costs a metadata hop per request but lets you place large tenants on dedicated shards and pack small tenants together deliberately. For a multi-tenant system with that kind of size skew, prefer lookup-based mapping, or a hybrid: algorithmic mapping for the default pool of ordinary tenants, with a lookup override for the small number of outsized ones.
Data-migration strategy
- Online copy under CDC: stream changes from the source shard(s) via a CDC pipeline (e.g., a log-based change stream) into the target topology while a background job bulk-copies existing data in ranges. The CDC stream and the bulk copy have to be reconciled (apply CDC events that land inside an already-copied range, buffer or replay ones that land ahead of the copy cursor).
- Dual writes with a quiesce window: to guarantee transaction ordering is preserved for records that are actively being migrated, briefly pause (quiesce) writes to the specific key range being cut over, drain the in-flight CDC backlog for that range so source and target agree, flip routing for just that range, then resume writes. This bounds the ordering risk to a small, deliberately-chosen window instead of trying to reason about ordering across an open-ended dual-write period for the whole dataset.
- Split-and-merge at scale: for a large, contended shard, resharding is a split (one shard's range divides into two, each getting its own routing-table entry) rather than a full-dataset migration; the reverse operation, merge, combines two under-loaded shards back into one when growth was over-provisioned. At terabyte scale, both operations are driven by range boundaries and routing-table updates rather than moving the whole dataset, which keeps a single split or merge bounded in time and blast radius.
- A cross-topology special case: migrating a stateful store (for example, a session store) from a single-region deployment to a multi-region sharded cluster follows the same dual-write skeleton, but adds a proxy layer in front of both topologies so the application never talks to the store directly during the migration; the proxy performs parity verification (comparing responses from the old and new paths for the same key) before fully cutting traffic over, which catches migration bugs before they're user-visible.
flowchart TD
A[Old shard topology] -->|CDC stream| B[Change pipeline]
A -->|throttled bulk copy| C[New shard topology]
B --> C
C --> D[Shadow reads: compare old vs new]
D -->|parity confirmed| E[Routing cutover, range by range]
D -->|mismatch| F[Pause cutover, reconcile]
E --> G[Quiesce window per range: drain, flip, resume]
G --> H[Old topology retired]
Routing updates
Introduce a migration-aware routing layer (gateway or client-side routing table) so application code never encodes shard topology directly. Roll the new routing out in stages: internal/test traffic, a small canary of production traffic, then full cutover, with each stage able to fall back to the previous routing table without a code deploy.
Throttling
Throttle the copy and CDC-apply workers adaptively against replication lag, tail latency, and source-shard CPU/I/O, not a fixed rate, so migration traffic backs off automatically when it's competing with production load. As a worked illustration of sizing a per-shard step against an operational target: suppose the team's service-level agreement (SLA) is for each per-shard rebalance step to complete in well under two minutes even across a 200+ node cluster, and a given shard holds roughly 50 GB of data with the copier capped at 500 MB/s per shard to protect production I/O. The data-transfer portion alone takes
which leaves about 20 seconds of headroom under the 120-second target for updating ring/routing metadata and completing leader election (the process by which the new shard's replica-set nodes agree on which one of them becomes the active primary/writer) for the new shard's replica set, before the step is considered done. That headroom is the number to watch: if metadata propagation or leader election routinely eats into it, the fix is a faster metadata path, not a bigger copy-bandwidth cap.
Automating rebalancing across many shards
When rebalancing hundreds of shards simultaneously rather than one at a time, the automation has to cap total concurrent migrations (global throttle, not just per-shard), and it must keep secondary-index consistency in scope explicitly: an index that's rebuilt or re-pointed on a different schedule than its base table will silently return wrong results for the window in between, so index cutover needs to be part of the same per-range quiesce step as the base data, not a follow-up job.
Validation
- Row counts and per-range checksums between source and target.
- Shadow reads: serve a fraction of real production reads from the new topology in parallel with the old one (without returning the shadow result to the client) and compare, which surfaces correctness bugs under real traffic patterns before any client depends on the new path.
- Client-side cache invalidation during cutover: if clients or edge caches hold entries keyed by the old routing, cutting a range over without invalidating those cached entries serves stale-routed responses; the cutover step has to include a cache-invalidation signal (a version bump on the routing key, or explicit invalidation) for anything caching downstream of the router.
- End-to-end application-level smoke tests against the new topology before removing fallback to the old one.
Rollback
Keep dual writes (or the CDC stream) active until validation passes; if a problem surfaces, stop applying new CDC events, revert routing to the old topology for the affected range, and resume from the last confirmed-consistent checkpoint rather than restarting the whole migration. Because cutover happens range-by-range behind a quiesce window, rollback scope is bounded to the ranges already flipped, not the whole dataset.
Trade-offs & pitfalls
- A migration that skips the quiesce window to avoid any write pause will have an ordering gap for records touched during the exact moment of cutover; a short, well-instrumented pause on a narrow key range is a better trade than an unbounded consistency risk.
- Algorithmic shard mapping is simpler to build and reason about, but for multi-tenant systems with orders-of-magnitude tenant-size variance, defaulting to it produces shards that are balanced by tenant count and wildly unbalanced by load; verify the mapping choice against actual per-tenant load, not tenant count.
- Skipping shadow reads to save migration time is the single most common way correctness bugs escape into production, because checksums catch data-copy errors but not query-behavior differences (an index that returns results in a different order, a range boundary edge case).
- Treating secondary-index rebuild as a background job that can lag the base-table cutover is a frequent source of silent incorrect reads immediately after a migration; keep index and base-data cutover atomic per range.
Design sharding for a real-time pub/sub service that has a very large number of channels. Compare client-side sharding, where clients pick their partition, against server-side partitioning, where the broker assigns it. Discuss rebalancing cost, fairness, latency, and client churn from mobile users with intermittent connectivity, and recommend an approach for a client base that is mostly mobile and frequently offline.
Sample Answer
Direct answer
For a pub/sub service with a huge number of channels and a client base that is mostly mobile and frequently offline, prefer server-side partitioning over pure client-side sharding. Client-side sharding (clients hash the channel to pick their partition themselves) is operationally cheap but turns every flaky-network reconnect into a fresh routing decision made independently by thousands of clients. Server-side partitioning (the broker assigns and owns the mapping) keeps that complexity centralized, where it can buffer subscription state through a drop and rebalance in a controlled, batched way instead of leaving it to uncoordinated client behavior.
Structured elaboration
The two placement strategies
- Client-side sharding: the client computes
partition = hash(channel_id) % n(or a similar scheme) and connects directly to the broker owning that partition. No broker-side coordination service is needed. - Server-side partitioning: clients connect to a stateless gateway or routing tier; a partition-metadata service tells the gateway which broker shard owns the channel, and the broker fleet can move ownership without the client ever recomputing anything.
Compare on the dimensions that matter here
- Rebalancing cost: under client-side sharding, changing shard count means every client must recompute its mapping, and if the mapping uses naive modulo hashing, nearly all clients reconnect to a different shard at once. Server-side partitioning can migrate ownership of a subset of channels while the gateway keeps existing client connections open, so the client never sees the rebalance directly.
- Fairness: client-side hashing has no visibility into actual load, so it can leave one shard hot while another idles. A server-side metadata service can rebalance based on measured load per shard, not just key distribution.
- Latency: client-side sharding has one fewer hop if the client connects straight to the right broker. Server-side partitioning adds a routing lookup, but that lookup can be cached at the gateway and is usually a small fraction of a mobile network's own round-trip time.
- Client churn from intermittent connectivity: this is the deciding factor for a mostly-mobile, frequently-offline client base. Client-side sharding treats every reconnect as a brand-new decision, so a client that drops and reconnects seconds later re-derives its partition and may miss messages published in between. Server-side partitioning lets the broker hold the subscription state (with a bounded grace period) so a quick reconnect resumes the same logical subscription instead of starting over.
A brief note on mechanics: whichever side owns the mapping, production systems generally implement the underlying hash-to-shard assignment with a consistent-hashing-style ring rather than plain modulo, specifically to bound how many keys move when shard count changes: nodes and channels are placed as points on a circular hash space (the ring), and virtual nodes give each physical shard many small points on it instead of one, so only the moved points' channels need to migrate. The ring construction and virtual-node tuning that make that work: giving each physical shard many small points scattered around the ring instead of one large contiguous range means more virtual points per physical shard smooths load distribution further but adds more routing metadata to maintain, while fewer points is cheaper to track but risks uneven load; the point relevant here is just that the choice of hashing scheme, not just who runs it, determines how disruptive a rebalance is.
Worked example
Assume, as a planning input rather than a measured fact, a fleet of 50 broker shards holding about 2 million channels between them, and that the service needs to grow to 60 shards to handle load. Under a consistent-hashing-style ring, the expected fraction of channels that move when going from n to n + m shards is approximately:
n+mm=6010≈16.7%
Under naive modulo hashing (partition = hash(channel_id) mod n), changing the divisor n reassigns nearly every key, so close to 100% of channels remap even though only 10 shards were added. That gap, roughly 17% of clients reconnecting elsewhere versus nearly all of them, is why a server-side partitioning layer that controls the hashing scheme matters more than which side technically "does" the sharding: a mobile client base that reconnects constantly on its own cannot absorb a near-100% remap event without a visible spike in dropped or duplicated messages.
Trade-offs & pitfalls
- A server-side metadata/routing tier becomes a new critical-path dependency. If it is a single instance rather than a redundant, health-checked fleet, it becomes the system's weakest point exactly when you added it to improve reliability.
- A mass reconnect event (an app release, a regional network outage recovering) can still overwhelm a well-designed server-side system if all clients retry at once; client SDKs need jittered backoff regardless of which sharding approach is chosen.
- Client-side sharding is not always wrong: for a small number of large, stable shards and a client base with reliable connectivity (internal services, desktop clients), its simplicity and lack of a coordination service can be the better trade.
- Buffering subscription state server-side to smooth over churn has a cost: unbounded buffering for offline clients grows broker memory. A grace period with a hard cap forces an explicit decision about how "offline" a client can be before it must fully resubscribe.
flowchart LR
subgraph Server-side partitioning
C1["Mobile client"] --> GW["Stateless gateway"]
GW --> MD["Partition metadata service"]
MD --> GW
GW --> B1["Broker shard 1"]
GW --> B2["Broker shard 2"]
end
Your service is stateful and stores per-user ephemeral session data in memory. Walk through three practical approaches to scale it horizontally without breaking correctness: sticky sessions, active-active replication, and externalizing the state. For each, describe the operational complexity, failure modes, and performance characteristics.
Sample Answer
Direct answer
There are three practical ways to scale a stateful service that keeps per-user ephemeral data in memory: keep the state local and pin each user to one instance (sticky sessions), replicate the state across instances so any of them can serve a user (active-active replication), or move the state out of the instances entirely into a shared store (externalization). Sticky sessions are the cheapest to build but the most fragile; externalization is the most operationally mature and is the default recommendation for anything beyond a small fleet; active-active replication is the highest-complexity option, justified mainly when you need both local-memory latency and multi-node resilience together.
Structured elaboration
1. Sticky sessions (session affinity at the load balancer)
- How it works: the load balancer hashes a stable client identifier and routes all of that client's requests to the same instance, which keeps the session in local memory. The routing mechanism itself (which hashing scheme, which health checks) is load-balancer territory; what matters for scaling the service is the consequence: correctness now depends on that one instance staying up.
- Operational complexity: low to moderate. Configure affinity on the load balancer, size instance capacity for peak per-node session count, add health checks and connection draining so deploys don't silently drop sessions.
- Failure modes: an instance failing loses every session pinned to it unless you add a fallback (short-time-to-live (TTL) backup write, or accept the loss and force re-authentication). Autoscaling and rolling deploys cause rebalancing, which cold-starts sessions on whichever instance a client lands on next. Hot users concentrated on one instance cause uneven load that autoscaling on aggregate metrics won't catch.
- Performance: the best case for latency, since reads and writes never leave process memory. Effective capacity is limited by the least-loaded-instance problem: total fleet capacity is high, but any single hot instance can be saturated while others are idle.
2. Active-active replication
- How it works: session state is synchronously or asynchronously replicated across a set of instances (via a gossip protocol, where nodes periodically exchange state with random peers; a consensus protocol, where nodes vote to agree on a value before it counts as committed; or conflict-free replicated data types, data structures built to merge concurrent updates without conflicts) so more than one node holds a current copy and can serve a read or accept a write.
- Operational complexity: high. You are now running a small distributed system inside your application tier: membership management, conflict resolution for concurrent writes to the same session, and monitoring for replication health.
- Failure modes: network partitions are the central risk. Without a consensus mechanism enforcing a single writer or a quorum, a partition produces split-brain, where two nodes both believe they hold the authoritative copy of a session and diverge. Even without a partition, replication lag means a read on one node can be stale relative to a very recent write on another.
- Performance: write latency goes up (synchronous replication waits on other nodes; asynchronous replication is fast but reintroduces the staleness problem it was meant to avoid). Read latency can stay low if reads are served from local replicas. Write throughput is bounded by replication and, if used, consensus overhead.
3. Externalizing state (shared cache or durable store)
- How it works: instances stop holding session state at all. Every request reads from and writes to a shared store (Redis, Memcached, or a durable database), which makes every instance interchangeable.
- Operational complexity: moderate. You now operate one more piece of infrastructure (a cache or database cluster: sizing, eviction policy if it's a cache, persistence if it needs to be durable) and the application changes to fetch and save state instead of holding it.
- Failure modes: the store itself becomes the thing that must not go down, so it needs its own high availability, replicas, and a defined fallback if it's briefly unreachable. Cache-invalidation bugs (serving stale state after a write, or evicting state that should have survived) are the main correctness risk. Network dependency between every app request and the store adds a new failure surface that didn't exist with local memory.
- Performance: a network hop per read/write adds latency relative to local memory, but it is predictable and independent of which instance serves the request, so it scales cleanly with the store's own capacity. A local read-through cache in front of the store recovers most of the latency for repeat reads.
Worked example
Consider a live-chat product where each open chat session tracks typing indicators, unread counts, and the last 20 messages in a rolling buffer, all ephemeral and rebuilt from durable message history if lost. The team starts with sticky sessions during the MVP: single-digit instance count, acceptable risk of a lost typing-indicator state on deploy. At 30 instances and daily autoscaling swings, rebalancing during scale-events starts dropping a visible fraction of active typing indicators on every scale-up, and hot users (large group chats) overload single instances that aggregate metrics don't flag as unhealthy. The team considers active-active replication to keep the local-memory latency but rejects it: the correctness bar for typing indicators is low (occasional staleness is invisible to users), so the complexity of consensus or CRDTs isn't earning its keep. They externalize instead: session state moves to a Redis cluster, keyed by chat id, with a short TTL so abandoned sessions clean up automatically. Every instance becomes interchangeable, rebalancing no longer drops state, and the added latency (one Redis round trip per message event) is well under the product's tolerance for a chat UI.
Trade-offs & pitfalls
- Choosing active-active replication for its latency profile without a real need for both local-memory speed and multi-node write availability at once is the most common overreach; most services that reach for it would be better served by externalization plus a local read-through cache.
- Assuming sticky sessions "just work" at scale and only discovering the rebalancing-churn problem after autoscaling is already live is a frequent operational surprise; load-test the rebalance path, not just steady-state.
- With externalized state, forgetting that the store is now a dependency of every request (not an optional side-system) leads to under-provisioning its availability relative to the application tier it's supposed to be simplifying.
- With active-active replication, treating eventual consistency as "good enough" without checking whether the specific session data has a correctness requirement (e.g., a shopping cart total versus a typing indicator) can silently introduce bugs that only show up under network partition, which is exactly when they're hardest to debug.
Your relational database allows a maximum of 500 concurrent connections. Your application runs on 25 identical JVM instances plus 3 background job workers. Walk through how you'd compute a safe default connection-pool size per instance, the factors you'd weigh, and your recommended pool size. What would you monitor, and what would you do if you saw connection saturation in production?
Sample Answer
Direct answer
Work backward from the database's hard ceiling, not forward from a per-instance number that feels generous. Reserve headroom for admin connections, failover, and bursts, divide what remains across every process that opens a connection (application instances and background workers alike), and floor the result. For 25 application instances (each running as its own process, commonly a JVM, a Java Virtual Machine, instance) plus 3 background workers against a 500-connection ceiling, that works out to a safe default of 14 connections per process, with room to tune down from there once you have real utilization data.
Structured elaboration
The formula
- Total connection-opening processes: 25 application instances + 3 background workers = 28.
- Reserve headroom for database maintenance tooling, admin connections, and burst capacity: a common starting assumption, stated here as a planning input rather than a measured fact, is 20%, leaving 80% of the ceiling as usable pool capacity.
- Safe per-process pool size: divide usable capacity across all processes and round down, so no single process's pool can push the fleet over the ceiling if every process is fully utilized simultaneously.
usable=500×0.80=400
per-process pool=⌊28400⌋=⌊14.29⌋=14
Factors that push the number up or down
- Transaction duration: long-running transactions hold a connection for longer, so a workload with slow queries needs a smaller pool per process (or faster queries) to avoid connections queuing up behind a few slow holders.
- Background workers are often burstier than request-serving instances; giving them the same per-process budget as application instances can be wasteful most of the time and insufficient during a burst. A separate, smaller steady-state budget with the ability to borrow from an external pooler (below) handles this better than a single uniform number.
- Read-heavy workloads can shift some of this load to read replicas, reducing the connection pressure on the primary specifically.
- Connection acquisition overhead and idle timeout settings affect how much of the pool is actually "available" at any moment versus tied up in idle-but-not-yet-reaped connections.
Monitoring and response to saturation
- Track: active connections per process, pool wait count and wait time, database-side connection count versus the 500 ceiling, and application-level timeout errors on connection acquisition.
- Alert when pool utilization sustains above roughly 75%, or when any request has to wait for a connection for more than a short, explicitly agreed threshold.
- On saturation: first apply backpressure (queue or reject new work rather than let requests pile up waiting on connections), then check whether the pool size itself is oversized relative to actual concurrent demand (overcommitted pools waste database-side capacity even when the app-side pool looks "full" only occasionally), then consider moving read traffic to a replica, and only then treat raising the database's connection ceiling or adding an external pooler as the longer-term fix.
Worked example
The formula above holds at moderate fleet size, but it breaks down as the fleet grows faster than the database's connection ceiling. Assume, again as a planning input, a fleet that has grown to 200 identical instances against a database with a 1,000-connection ceiling. Applying the same approach:
usable=1000×0.80=800
per-instance pool=⌊200800⌋=4
Four connections per instance is thin: if a single instance briefly needs more than four concurrent database calls, for example during a request burst, it will queue locally even though the database as a whole has spare capacity sitting idle in other instances' unused pool slots. This is also exactly the scenario where a rolling deploy becomes dangerous: if all 200 instances restart in a short window and each immediately tries to open its full pool of 4 connections, that is a connection storm, a spike toward the 1,000-connection ceiling driven by simultaneous reconnects rather than steady-state load, and it can exhaust the ceiling before the deploy even finishes.
The fix at this scale is usually an external pooler such as PgBouncer, placed between the 200 application instances and the database, multiplexing many app-side connections onto a much smaller database-side pool. PgBouncer's two relevant modes make different trade-offs:
- Session pooling mode: a database connection is assigned to one client connection for its entire session (until it disconnects). This preserves session-scoped features like prepared statement caching and
SETcommands, but it does not reduce database-side connections below the number of concurrently active client sessions, so it does not solve the 200-instances-against-1,000-connections problem by itself. - Transaction pooling mode: a database connection is held only for the duration of a single transaction, then released back to the pool immediately for another client to use. This lets a small database-side pool (for example, 50 to 100 connections) serve far more app-side connections than session mode can, at the cost of breaking anything that relies on session state persisting across transactions, such as advisory locks held between transactions or
SET-scoped session variables.
Given the transaction-pooling trade-off, a deploy in this configuration should also stagger restarts (for example, capping simultaneous restarts to a small percentage of the fleet with a ramp-up delay between batches) rather than restart all 200 instances at once, regardless of whether an external pooler is in place.
Trade-offs & pitfalls
- Sizing the pool from "what feels safe" rather than the database's actual ceiling is the most common mistake: it works until a deploy, a traffic spike, or a new service instance pushes the fleet over the limit all at once.
- A pool that is too large per instance doesn't help throughput once the database itself is saturated; it just moves the queue from the application to the database, where diagnosing it is harder.
- An external pooler adds an operational dependency and a new failure mode (the pooler itself can become a bottleneck or single point of failure) in exchange for solving a problem the simple per-instance formula cannot solve at high instance counts.
- Any pool-sizing decision should be re-validated with load testing and real monitoring data, not treated as a one-time calculation; traffic mix and query patterns change the safe number over time.
Unlock Full Question Bank
Get access to all Scalability Patterns and Techniques interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.