Scalability Patterns and Techniques Questions

Direct answer

Cache-aside, read-through, write-through, and write-behind differ in who is responsible for moving data between the cache and the database, and on which side of a write the durability guarantee sits. Cache-aside and read-through both leave reads on a lazy-load path (the application checks cache first, loading from the database only on a miss), while write-through and write-behind differ from each other in whether a write is confirmed synchronously to the database (write-through) or acknowledged immediately and flushed later (write-behind). The choice comes down to how much staleness and how much write latency the use case can tolerate.

Structured elaboration

Cache-aside (lazy-loading)

• Read flow: application checks the cache; on a miss it reads the database itself, populates the cache, and returns the result.
• Write flow: application writes the database, then explicitly invalidates or updates the affected cache entry.
• Latency and consistency: fast on hits, a first-read penalty on misses; a small race window exists between the database write and the cache invalidation, so a concurrent reader can briefly see stale data.
• Failure handling: if the cache crashes or is flushed, the application transparently falls back to reading the database on every subsequent miss; nothing is lost, throughput just degrades to database-only speed until the cache warms back up. This graceful-degradation property is cache-aside's main durability advantage over the write-behind pattern below.
• Example: a product catalog, where reads dominate heavily and a brief staleness window after a price or description edit is acceptable.

Read-through

• Read flow: the application asks the cache client for a key; the cache itself is responsible for loading from the database on a miss (rather than the application doing it), which centralizes the loading logic instead of duplicating it in every caller.
• Write flow: typically paired with write-through or write-behind, since read-through only defines the read side.
• Latency and consistency: same miss-cost profile as cache-aside; the benefit is code simplicity, not a different consistency guarantee.
• Failure handling: same graceful degradation as cache-aside on a cache failure, since the loader logic still falls through to the database.
• Example: session lookups, where centralizing the load-on-miss logic in the cache layer avoids repeating it across every service that needs a session.

Write-through

• Read flow: same as cache-aside or read-through.
• Write flow: the application writes to the cache, and the cache synchronously writes through to the database before acknowledging the write as successful.
• Latency and consistency: higher write latency than write-behind, because every write pays the database round trip, but the cache and database are never out of sync from the client's perspective.
• Failure handling: if the origin (database) write fails, the cache write must be rolled back or the entry invalidated, or the cache would hold a value the database never actually has; a cache-side crash after a successful database write just loses the cached copy, not any data, since the database is always the durable copy of record.
• Example: a shopping cart, or the broader e-commerce cart use case generally, where the user needs to see their own change reflected immediately and correctly, and the write rate is low enough that the added latency is acceptable.

Write-behind (write-back, asynchronous)

• Read flow: reads are served from cache, populated by writes or by a read-miss loader.
• Write flow: the application writes to the cache, which acknowledges immediately and batches or asynchronously flushes accumulated writes to the database.
• Latency and consistency: the lowest write latency of the four, since the client never waits on the database, but the database is only eventually consistent with the cache, and durability now depends entirely on the cache surviving until its next flush.
• Failure handling, in depth: this is the pattern with a real durability analysis to do, not just a caveat. If the cache crashes before a batch of writes has been flushed, every write in that unflushed batch is lost, a crash-recovery data-loss window whose size is exactly the flush interval (a 5-second flush interval means up to 5 seconds of writes are at risk on any crash). Recovery after a crash means restoring from the last durable flush and accepting that anything after it is gone, unless the cache itself is backed by a write-ahead log or replicated before acknowledging, which narrows the window but adds back some of the latency write-behind was chosen to avoid. Batching writes together to flush also introduces write amplification when a key is updated many times before its flush: instead of every individual write reaching the database, only the final value per flush cycle is written, which reduces database load but means intermediate values are never durably recorded at all, not just delayed.
• Example: high-throughput counters, such as a leaderboard, or an analytics-ingestion pipeline, where the flush interval is analogous to a batching window and losing a small, bounded amount of the most recent data on a rare crash is an acceptable trade for sustaining a write rate the database could not absorb directly. The durability trade-off is the same shape in both cases: low per-write latency and reduced database load, in exchange for a bounded, quantifiable data-loss window plus write amplification on frequently-updated keys.

Choosing between them

Worked example

An e-commerce cart service needs the cart total to be correct the instant a user adds an item (write-through: synchronous database write, cache mirrors it, no staleness). The same platform's "trending products" counter, incremented on every product view across millions of views per day, uses write-behind: each view increments the cached counter and returns instantly, and the counter is flushed to the database on a fixed interval. If the cache process crashes, the trending counter loses at most that interval's worth of increments, which is invisible on a counter fed by millions of events, whereas losing even one cart write would be a customer-facing correctness bug. The same reasoning applies directly to an analytics-ingestion pipeline: individual event counts can tolerate the same bounded loss window, but a financial ledger update could not, which is why write-behind is scoped to the specific fields where that trade-off is safe rather than applied to a whole write path uniformly.

Trade-offs & pitfalls

• Treating write-behind's flush interval as a tuning knob without stating the resulting data-loss window explicitly is a common gap in design reviews; the window should be a named, deliberate number, not an implicit side effect of the batch size chosen for throughput.
• Cache-aside's invalidate-after-write race window is small but real under high concurrency; if a use case cannot tolerate any stale read, write-through is the safer default despite its latency cost.
• Read-through's benefit is purely architectural (centralized loading logic); choosing it under the assumption that it also improves consistency versus cache-aside is a misunderstanding of what the pattern actually changes.
• For any pattern, a cache failure that is treated as a full outage rather than a graceful fallback to the database is a design gap, not an inherent property of caching: cache-aside, read-through, and write-through can all fall back cleanly; only write-behind has real, quantifiable data at risk on a cache crash.

Direct answer

For a pub/sub service with a huge number of channels and a client base that is mostly mobile and frequently offline, prefer server-side partitioning over pure client-side sharding. Client-side sharding (clients hash the channel to pick their partition themselves) is operationally cheap but turns every flaky-network reconnect into a fresh routing decision made independently by thousands of clients. Server-side partitioning (the broker assigns and owns the mapping) keeps that complexity centralized, where it can buffer subscription state through a drop and rebalance in a controlled, batched way instead of leaving it to uncoordinated client behavior.

Structured elaboration

The two placement strategies

• Client-side sharding: the client computes partition = hash(channel_id) % n (or a similar scheme) and connects directly to the broker owning that partition. No broker-side coordination service is needed.
• Server-side partitioning: clients connect to a stateless gateway or routing tier; a partition-metadata service tells the gateway which broker shard owns the channel, and the broker fleet can move ownership without the client ever recomputing anything.

Compare on the dimensions that matter here

• Rebalancing cost: under client-side sharding, changing shard count means every client must recompute its mapping, and if the mapping uses naive modulo hashing, nearly all clients reconnect to a different shard at once. Server-side partitioning can migrate ownership of a subset of channels while the gateway keeps existing client connections open, so the client never sees the rebalance directly.
• Fairness: client-side hashing has no visibility into actual load, so it can leave one shard hot while another idles. A server-side metadata service can rebalance based on measured load per shard, not just key distribution.
• Latency: client-side sharding has one fewer hop if the client connects straight to the right broker. Server-side partitioning adds a routing lookup, but that lookup can be cached at the gateway and is usually a small fraction of a mobile network's own round-trip time.
• Client churn from intermittent connectivity: this is the deciding factor for a mostly-mobile, frequently-offline client base. Client-side sharding treats every reconnect as a brand-new decision, so a client that drops and reconnects seconds later re-derives its partition and may miss messages published in between. Server-side partitioning lets the broker hold the subscription state (with a bounded grace period) so a quick reconnect resumes the same logical subscription instead of starting over.

A brief note on mechanics: whichever side owns the mapping, production systems generally implement the underlying hash-to-shard assignment with a consistent-hashing-style ring rather than plain modulo, specifically to bound how many keys move when shard count changes: nodes and channels are placed as points on a circular hash space (the ring), and virtual nodes give each physical shard many small points on it instead of one, so only the moved points' channels need to migrate. The ring construction and virtual-node tuning that make that work: giving each physical shard many small points scattered around the ring instead of one large contiguous range means more virtual points per physical shard smooths load distribution further but adds more routing metadata to maintain, while fewer points is cheaper to track but risks uneven load; the point relevant here is just that the choice of hashing scheme, not just who runs it, determines how disruptive a rebalance is.

Worked example

Assume, as a planning input rather than a measured fact, a fleet of 50 broker shards holding about 2 million channels between them, and that the service needs to grow to 60 shards to handle load. Under a consistent-hashing-style ring, the expected fraction of channels that move when going from n to n + m shards is approximately:

$\frac{m}{n+m} = \frac{10}{60} \approx 16.7\%$

Under naive modulo hashing (partition = hash(channel_id) mod n), changing the divisor n reassigns nearly every key, so close to 100% of channels remap even though only 10 shards were added. That gap, roughly 17% of clients reconnecting elsewhere versus nearly all of them, is why a server-side partitioning layer that controls the hashing scheme matters more than which side technically "does" the sharding: a mobile client base that reconnects constantly on its own cannot absorb a near-100% remap event without a visible spike in dropped or duplicated messages.

Trade-offs & pitfalls

• A server-side metadata/routing tier becomes a new critical-path dependency. If it is a single instance rather than a redundant, health-checked fleet, it becomes the system's weakest point exactly when you added it to improve reliability.
• A mass reconnect event (an app release, a regional network outage recovering) can still overwhelm a well-designed server-side system if all clients retry at once; client SDKs need jittered backoff regardless of which sharding approach is chosen.
• Client-side sharding is not always wrong: for a small number of large, stable shards and a client base with reliable connectivity (internal services, desktop clients), its simplicity and lack of a coordination service can be the better trade.
• Buffering subscription state server-side to smooth over churn has a cost: unbounded buffering for offline clients grows broker memory. A grace period with a hard cap forces an explicit decision about how "offline" a client can be before it must fully resubscribe.

flowchart LR
  subgraph Server-side partitioning
    C1["Mobile client"] --> GW["Stateless gateway"]
    GW --> MD["Partition metadata service"]
    MD --> GW
    GW --> B1["Broker shard 1"]
    GW --> B2["Broker shard 2"]
  end

Direct answer

Connection pooling matters at scale because opening a new database or HTTP connection is expensive relative to a request (TCP handshake, and for a database, authentication and session setup), so reusing a small set of warm connections instead of creating one per request lowers latency and prevents the backend from being overwhelmed by connection churn. The core sizing problem is that a pool is a per-instance setting but the backend has a fleet-wide connection ceiling, so pool size has to be planned across the whole fleet, not tuned in isolation on one instance.

Structured elaboration

Why pooling matters at scale, mechanically

• Connection setup cost: a TCP handshake, TLS negotiation (for HTTP), and for a database, authentication plus session/state initialization, all add latency if paid on every request.
• Backend resource limits: every open connection holds memory and, for a database, often a whole backend process or thread; a backend with a hard maximum connection count can be pushed into refusing connections or degrading badly under connection churn even if query volume itself is modest.
• Reuse turns a per-request cost into a one-time cost amortized across many requests on the same warm connection.

Sizing pools: the fleet-wide constraint

The number one mistake is sizing a pool as if the instance owns the whole backend. It does not; every other instance is drawing from the same ceiling:

If the database allows 500 total connections and the service runs behind 20 instances, each instance's pool must stay at or below $\lfloor 500/20 \rfloor = 25$ connections, or a fleet at full pool utilization exceeds the database's ceiling and starts getting connection refusals, exactly when load is highest and refusals hurt the most. This constraint must be revisited every time the fleet is resized by autoscaling, which is the part teams most often forget: a pool size tuned for 20 instances silently becomes unsafe the moment autoscaling adds a 21st.

Core pool parameters

Behavior under a load spike: the connection-storm problem

The specific failure mode worth naming: a deploy, a failover, or a sudden traffic spike can cause many instances to simultaneously reconnect or spin up new pooled connections at once, a connection storm, which can itself exceed the database's connection ceiling even though steady-state pool sizing was correct. This has a process/thread-model dimension too: a backend that spawns one OS process or thread per connection (a common relational-database architecture) pays a much higher per-connection memory and context-switch cost under a storm than one built around lightweight connection handling, which changes how conservatively you should size db_max_connections in the first place. Mitigations: stagger reconnects with jitter (small random delays) instead of reconnecting all instances at once, keep pool warm-up gradual rather than instantaneous on instance startup, and prefer acquisition timeouts with backoff over unbounded retry storms.

Testing and tuning before production

• Load-test at realistic peak concurrency and burst shape, not just average throughput, since spikes and connection storms are what actually break pool sizing.
• Vary the number of app instances in the test to confirm the fleet-wide formula holds at the target autoscaling range, not just at today's instance count.
• Watch active/idle/wait-count and wait-time metrics from the pool itself, plus backend-side connection and CPU/IO metrics, and tune size, idle timeout, and lifetime to minimize wait time while keeping the backend under its ceiling.
• Explicitly test the failure path: kill connections mid-flight, simulate a slow backend, and confirm acquisition timeouts and backpressure behave as designed rather than hanging.

Worked example

A service runs 20 instances against a database capped at 500 total connections. Using the formula above, each instance is capped at 25 pooled connections. During a load test that simulates a rolling deploy (all 20 instances restarting within a short window), every instance attempts to rebuild its pool of 25 connections at once: $20 \times 25 = 500$ simultaneous reconnect attempts against a ceiling of exactly 500, with zero margin for any connection still draining from the old instances. Adding jittered reconnect delays and reducing per-instance pool size to 20 (giving $20 \times 20 = 400$, leaving 100 connections of headroom during a rollover) eliminates the connection-storm failures observed in the unthrottled test.

Trade-offs & pitfalls

• Sizing a pool against a single instance's peak load, without dividing by the fleet size, is the most common and most damaging mistake; it works until autoscaling adds instances, then fails exactly under peak traffic.
• A pool with no acquisition timeout turns backend overload into cascading request pile-ups instead of fast, visible failures; for services making many short-lived connections, pairing the pool with a circuit breaker (a resilience pattern that stops sending requests to a struggling dependency, covered under high-availability patterns rather than here) prevents that pile-up from spreading further upstream.
• Idle timeouts set too aggressively cause needless reconnection churn during normal traffic dips; set too loosely, they let leaked or stale connections accumulate unnoticed.
• Connection leaks (code paths that acquire a connection and never release it, often on an error path) are the quiet failure mode: the pool looks correctly sized until leaked connections slowly starve it, and only a saturation metric with alerting catches this before an outage.

Direct answer

A stateless service keeps no client- or session-specific data in process memory between requests: every request carries (or looks up) everything needed to handle it. A stateful service holds that context in memory across requests, such as an open socket or an in-memory session. Statelessness enables horizontal scaling because any instance can serve any request, so a load balancer can distribute traffic with no per-node reconciliation and instances become disposable: add, remove, or replace them freely.

Structured elaboration

Why statelessness is the enabler, mechanically

• Interchangeability: since no instance holds unique data, a request routed to any instance gets the same result, which is what makes simple, even load distribution possible.
• Painless scaling events: adding capacity means starting new instances with no data migration; removing capacity means stopping instances with no data loss, because nothing lived there that mattered.
• Simpler recovery: a crashed stateless instance is replaced, not repaired; there is no in-memory state to reconstruct.
• Simpler rolling deploys: instances can be cycled one at a time without draining session state first.

Strategies to externalize state

• Durable application state moves to a database (relational or NoSQL) instead of process memory.
• Fast shared access to hot data moves to a distributed cache (an in-memory key-value store shared across instances) rather than a per-instance local cache.
• Session data specifically has two common paths: a centralized session store, or client-held tokens (such as a signed JSON Web Token, JWT) that let the server stay stateless entirely because the client presents its own state on every request.
• Large binary objects move to an object store rather than local disk.
• Asynchronous or queued state (work in flight, not yet durable) moves to a message queue.
• Small amounts of shared coordination state (configuration, leader pointers, locks) move to a dedicated coordination service built for that job.

Where a stateful service is genuinely necessary

The SRE angle: operating a stateful service you cannot avoid. When a stateful service is genuinely required, the operational burden shifts from "how do I scale it" to "how do I keep it reliable despite being pinned." That means treating the stateful nodes as a smaller, more carefully managed subset of the fleet: automated health checks tied to a fast failover or re-election path, regular drills that exercise that failover before it is needed under real incident pressure, and monitoring that specifically watches for the failure modes unique to statefulness (a stuck leader, a connection that never drains, a checkpoint that falls behind). The mitigation is never "make it stateless anyway"; it is "shrink the stateful surface to the minimum and instrument that minimum heavily."

Worked example

A web application currently stores logged-in session data in each server's memory, so a user's second request must land on the same server (sticky routing) or their session appears to vanish. To make the tier horizontally scalable: move session data out of process memory into a shared session store, or better, switch to a signed token that the client holds and presents on each request so the server does not need to look anything up at all. Either change means any instance can now serve any request, sticky routing is no longer required, and the fleet can be scaled up or down purely on load, with new instances immediately able to serve full traffic with zero data migration.

Trade-offs & pitfalls

• Do not confuse "stateless service" with "no state exists": the state still exists, it has just moved to a system designed to hold it reliably. That external system becomes a new dependency and a new potential bottleneck, so it needs its own scaling plan.
• Client-side tokens remove server lookups but push size and revocation concerns onto the client and the token design; server-side session stores keep revocation simple but add a network hop and a shared-store scaling problem.
• Treating "sticky sessions" as a free fix for statefulness is a common wrong turn: it works, but it silently reintroduces a form of per-node state ownership and undermines the failure-isolation benefit that horizontal scaling was meant to provide.
• For the genuinely stateful cases, skipping the failover-drill discipline is the most common production failure: the coordination logic works in testing and then fails silently the first time it is needed in an incident, because it was never exercised under realistic conditions.

Direct answer

Read replicas are read-only copies of a primary relational database that let you route read-heavy traffic away from the primary, so read throughput scales roughly with the number of replicas instead of being capped by one machine's capacity. The two common replication modes trade off write latency against durability: asynchronous replication is fast but can lag, semi-synchronous replication waits for at least one replica to acknowledge before confirming a write, trading some write latency for a stronger durability guarantee. Replication lag, the gap between a write landing on the primary and appearing on a replica, is the operational pitfall that follows directly from choosing asynchronous replication for speed.

Structured elaboration

Why read replicas scale reads

A single primary database has a ceiling on how many queries per second (QPS, the standard measure of database or API load) it can serve before CPU, memory, or I/O saturates. Since most application workloads are read-heavy relative to writes, adding replicas that each hold a full copy of the data lets read queries fan out across many machines while writes still funnel through the one primary that owns correctness. This is a read-scaling pattern specifically: it does nothing for write throughput, which is bounded by the primary alone (write scaling is a separate problem, addressed by partitioning or sharding rather than replicas).

Replication modes

• Asynchronous: the primary commits and returns success to the client without waiting for any replica to apply the change. Write latency stays low and unaffected by replica health, but a replica can fall arbitrarily behind under load, and if the primary fails before a replica caught up, those last writes are lost from that replica's perspective.
• Semi-synchronous: the primary waits for acknowledgment from at least one replica (that the write was received, not necessarily fully applied) before confirming the commit to the client. This bounds the worst-case data loss to writes that hadn't yet reached any replica, at the cost of added write latency and a risk that a slow replica introduces a stall on every write.

This is standard terminology in an online transaction processing (OLTP) context, meaning a workload of many small, individual reads and writes (as opposed to large analytical scans); read replicas are one of the first tools reached for once a single OLTP primary starts to strain under read load.

Replication lag as the operational pitfall

Lag arises from network delay, I/O contention on the replica, or the replica processing a backlog of changes slower than the primary produces them. Its consequence is stale reads: a client that just wrote data may query a replica and not see its own write, or two clients may observe the data in different states depending on which replica they hit.

Read-routing design to minimize stale reads while maximizing throughput

The application layer, not just the database, needs a policy for which reads are allowed to be stale:

• Reads that must reflect the client's own very recent write (a user viewing the profile they just edited) should go to the primary, or to a replica only after confirming its lag has caught past that write's position.
• Reads that tolerate a small staleness window (a public dashboard, a search index, an analytics report) should go to replicas by default, since that is where the throughput gain comes from.
• A hybrid policy, sometimes called read-your-writes routing, pins an individual client to the primary (or to a replica known to be caught up) for a short window right after that client writes, then lets subsequent reads fall back to any replica.

Monitoring and safeguards

Safeguards built on that monitoring: alert when lag crosses a threshold tied to the application's staleness tolerance; throttle or move expensive ad hoc/analytical queries off replicas that also serve latency-sensitive reads; and, for any workflow that promotes a replica (to primary, during a failure), require lag to be at or near zero before promotion, since promoting a lagging replica means accepting the unreplicated writes as lost. That promotion and failover mechanics belong to the high-availability side of the system, not to the read-scaling pattern itself, but the monitoring described here is exactly what feeds that decision when it happens.

Worked example

A social-media-style application serves 9,000 reads per second and 1,000 writes per second against a single primary that is now CPU-saturated on reads. Adding 3 asynchronous read replicas and routing all reads except "read-your-own-write" cases to a round-robin pool across them reduces the read load on the primary from 9,000 QPS to roughly 0 (reads move off entirely), leaving the primary handling only the 1,000 writes/second plus the small share of reads that require read-your-writes freshness. Each replica now carries roughly 9,000 / 3 = 3,000 reads/second on average, well within a single replica's typical headroom, illustrating the linear-ish scaling read replicas provide as long as write volume itself stays within what one primary can sustain.

Trade-offs & pitfalls

• Read replicas scale reads only; teams sometimes reach for them to fix a write-contention problem, which they cannot, because writes still funnel through one primary.
• Asynchronous replication's low write latency is attractive, but skipping the read-routing design above (treating every replica as equally fresh) is the most common way stale reads leak into user-facing behavior.
• Semi-synchronous replication reduces data-loss risk but can introduce write stalls if the acknowledging replica itself becomes slow; it shifts risk from data loss to latency, it does not eliminate risk.
• Promoting a lagging replica during an incident, without checking lag first, can silently drop the most recent committed writes; this is a data-loss event dressed up as a recovery action.