Scalability Patterns and Techniques Questions

Direct answer

Project monthly active users (MAU) forward with compound growth, $\text{MAU}(t) = \text{MAU}_0 \times (1 + r)^t$, convert that into a traffic curve, and solve for when it crosses 2x today's traffic by taking a logarithm. For 200,000 MAU growing 10% month over month, capacity roughly doubles in about 7.3 months, so plan the next major capacity jump around month 7-8, not month 12. Seasonality and marketing campaigns then layer multiplicatively on top of that sustained-growth curve as separate, temporary spikes, not as part of the underlying trend line.

Projection

After 12 months:

When does capacity double?

Solve $(1.10)^t = 2$ for $t$:

So baseline traffic, and the required requests-per-second (RPS) capacity that scales with it, doubles in roughly 7.3 months, meaning capacity planning should target month 7 or 8 (rounding up for safety) for the next major scale-up, not month 12 as a straight annual-planning cycle might assume.

Seasonality and campaign effects, layered on top

Both effects are multiplicative spikes on top of the baseline curve above, not adjustments to the growth rate itself:

• Seasonal spike (November, +50%): if November falls at month $k$ in the projection, expected peak MAU (and peak RPS) for that month is $\text{MAU}(k) \times 1.5$.
• Marketing campaign (2x traffic for 2 weeks): averaged over a 4-week month, this is $(2\ \text{weeks} \times 2\times + 2\ \text{weeks} \times 1\times) / 4\ \text{weeks} = 1.5\times$ for that month's average, though the campaign's actual peak within those 2 weeks is the full 2x, which matters more for capacity than the monthly average does.
• If both land in the same month: the combined peak multiplier is $1.5 \times 1.5 = 2.25\times$ baseline for that month, not simply additive.

A different worked comparator: layering a seasonal spike on a sustained-growth target

A related planning problem shows up for a data-ingestion pipeline sized against a 10x sustained-growth target (a longer planning horizon than the 7-month doubling above): solving $(1.10)^t = 10$ gives $t = \ln(10)/\ln(1.10) \approx 24.2\ \text{months}$, roughly two years to reach 10x baseline at the same 10% monthly growth rate. Layering a 2x seasonal spike on top of that 10x sustained target means the pipeline's peak provisioning needs to cover $10\times \times 2\times = 20\times$ baseline, not just the 10x sustained figure, since the seasonal spike and the multi-year growth trend both have to be true at the same time during that one peak month. This is a materially different exercise from the 2-week campaign case above: a 20x peak on a data pipeline is usually validated with a dedicated load test that replays a realistic ingestion pattern at that multiple, rather than trusted from the arithmetic alone, since ingestion systems often have hard limits (storage throughput, downstream processing capacity) that do not scale as smoothly as a stateless request-handling tier.

Operational planning recommendations

• Use autoscaling with fast ramp policies so capacity can respond within the campaign's 2-week window, not just the multi-month growth trend, since a policy tuned only for gradual growth will react too slowly to a 2x spike that starts and ends within days.
• Load test at the predicted peak multiplier (2.25x for a combined seasonal-plus-campaign month, or 20x for the layered sustained-growth-plus-seasonal case) plus an additional safety margin, rather than testing only at the smoothed monthly-average multiplier.
• Use caching and content delivery network (CDN) offload to reduce the RPS that actually reaches origin infrastructure during a spike, which lowers how much raw compute capacity has to scale to cover the same user-facing multiplier.
• Track real-time metrics against the projected curve, so a campaign or season that outperforms (or underperforms) the assumed multiplier is caught early rather than discovered at the point of user-facing degradation.

Trade-offs and pitfalls

The most common mistake in this kind of projection is treating a marketing campaign's monthly average multiplier (1.5x) as the number to provision for, when the campaign's actual burst behavior within those 2 weeks, and especially within the peak hours of the peak days, is what capacity needs to survive; averaging over a full month smooths away exactly the spike that matters. A second pitfall is planning capacity purely off the compound-growth trend line and treating seasonality as an afterthought, when in a case like the layered 10x-plus-2x data-ingestion example, the seasonal multiplier applied to an already-large sustained-growth baseline can dwarf the trend-line number the team spent the most time modeling.

Direct answer

Cache hit ratio is the fraction of lookups a cache can answer without going to the origin (hits divided by total lookups); a cache miss is a lookup where the value isn't present or has expired, forcing a fetch from the origin and usually a write back into the cache; cache warmup is the process of populating a cache, at startup or after a restart, before it can offer a useful hit ratio. Whether adding an application cache (memcached, Redis, or similar) is worth it comes down to a simple cost comparison: what a miss costs you (a database round trip, a slow computation, an external API call) versus what running the cache costs you (infrastructure, and the risk of serving stale data), not a fixed hit-ratio threshold you're supposed to hit.

Structured elaboration

When to add a cache

• The workload is read-heavy with requests that repeat: the same or similar queries recur often enough that a cached answer is reused rather than computed once and discarded.
• The thing being cached is expensive relative to a cache lookup: a slow database query, an external API call, or a CPU-heavy computation are all good candidates; something already fast to compute gains little.
• The data can tolerate the staleness a cache implies, or the system can actively invalidate the cache on writes; if every read must reflect the absolute latest write, caching adds a consistency problem you have to solve, not just a performance win.

What hit ratio would justify the cost

There's no universal number, because "justified" depends on the ratio between the cost avoided per hit and the cost of running the cache, not on the hit ratio in isolation. A commonly cited planning heuristic for latency- and cost-sensitive systems is to aim for roughly 70 to 80% hit ratio; treat that as a starting heuristic to validate against your own workload, not as a target derived from anything specific to your system. A system where each miss is very expensive (an external, rate-limited, or metered API call) can be worth caching even at a 50% hit ratio, while a system where the origin call is already cheap may not justify caching even at 90%.

Three practical ways to improve an existing cache's effectiveness

1. Tune time-to-live (TTL, how long a cached value is considered valid before it must be refreshed) per key type rather than using one global value: longer TTLs for stable data, shorter for volatile data, so you're not needlessly re-fetching stable data or serving stale volatile data.
2. Improve cache key design: normalize keys (strip user-specific noise that doesn't actually change the result, use consistent prefixes) so semantically identical requests share a cache entry instead of each generating its own miss.
3. Warm proactively and avoid a thundering herd on expiry: pre-populate known-hot keys at deploy or restart, and use a read-through or refresh-ahead pattern where a background job refreshes a key just before it expires, rather than letting many concurrent requests all miss on the same expired key at once.

Worked example

Assume, as a planning input rather than a measured fact, that an endpoint currently receives 10,000 requests per minute with no caching, and that each request costs 20 ms of database time. Total database time consumed per minute today:

$10{,}000 \times 20\,\text{ms} = 200{,}000\,\text{ms} = 200\,\text{s of database time per minute}$

If a cache reaches a 75% hit ratio on this endpoint, the number of requests still reaching the database is:

$10{,}000 \times (1 - 0.75) = 2{,}500 \text{ requests per minute}$

$2{,}500 \times 20\,\text{ms} = 50{,}000\,\text{ms} = 50\,\text{s of database time per minute}$

That's a 75% reduction in database load, directly tracking the hit ratio, from 200 seconds of database time per minute down to 50. Whether that reduction is "worth it" then depends on what those 150 seconds of freed-up database time are worth relative to running the cache, not on the 75% figure by itself.

Trade-offs & pitfalls

• Chasing a higher hit ratio as a goal in itself, rather than as a proxy for reduced backend cost, can lead to caching things that barely help (already-cheap lookups) while a genuinely expensive but less frequent lookup goes uncached.
• A TTL that's too long trades staleness risk for hit ratio; too short and the cache behaves closer to no cache at all under the same access pattern.
• Skewed (heavily concentrated, or Zipfian) access patterns mean a handful of keys drive most hits; eviction policy choice (LRU, least-recently-used, versus LFU, least-frequently-used) matters more under this kind of skew than under uniform access, since LRU can evict a very-frequently-used-but-not-most-recent key that LFU would keep.
• Expose hit and miss counters through your application performance monitoring (APM, application performance monitoring) or metrics stack; without that visibility, a regression in hit ratio after a code change (a key-format change, a new high-cardinality parameter) can go unnoticed until backend load spikes.

Direct answer

Once you scale past a handful of instances, in-memory session state stops working: sticky sessions (routing every request from a user to the same backend instance) keep working but make the fleet fragile and hard to rebalance, while externalizing session state to a shared store (Redis, DynamoDB, or similar) makes every instance stateless at the cost of a network hop and a new piece of infrastructure to run. For most cloud-native, multi-region systems the externalized store wins once you account for failover and elastic scaling, but the right choice depends on session size, read/write rate, and how much latency and operational cost you can absorb.

Structured elaboration

Sticky sessions (load-balancer affinity)
The load balancer hashes a client identifier (cookie or IP) to consistently route it to one backend instance, which keeps the session data in that instance's local memory. This is the load-balancer mechanism; the algorithm and health-check details behind it belong to load-balancing design, not to this decision. What matters here is the state-management consequence: the instance becomes a single point of failure for every session pinned to it.

Externalized session store
The application writes session data to a shared store instead of local memory, so any instance can serve any request. This is what makes the fleet stateless and lets you autoscale, blue/green deploy, and route traffic freely.

Trade-off table

Two problems specific to externalizing state that a senior candidate should raise

1. Data locality/regulatory constraints on where the store lives. Once session state leaves the process and lives in a shared store, where that store is physically located becomes a compliance question in a way it never was for sticky sessions. If you operate in a jurisdiction with a data-residency requirement (for example, keeping EU user data on EU infrastructure), the session store's region placement is now something you have to design around explicitly: either run per-region session stores keyed so a user's session never leaves their region, or make sure the store you pick supports region-pinned partitions. This is a design constraint, not an afterthought bolted on after the store is chosen.
2. Token revocation/invalidation once you drop server-side sticky sessions. With sticky sessions, "logging a user out" or forcibly invalidating a session is trivial: the state lives in one place and you delete it there. Once you move to a stateless architecture backed by client-held tokens (JWTs, JSON Web Tokens), you lose that for free, because a JWT is self-contained and valid until it expires, regardless of what the server thinks. You need an explicit revocation mechanism: keep a server-side allow/deny list (defeats some of the point of a stateless token, but is often still cheaper than full session storage), keep JWT lifetimes short and pair them with a refresh-token that IS checked against the store on each renewal, or store only a session identifier in the token and look up the actual session state (and its revocation flag) in the externalized store on every request. Whichever you pick, decide it up front: it is the first thing that breaks in incident response ("how do we kill this user's access right now") if you didn't.

Worked example

A mid-size SaaS product runs 40 stateless application instances behind a load balancer in two regions (US and EU) for latency and the EU data-residency requirement above. Session state (user id, cart contents, feature flags, roughly 2 KB per session) is stored in a regional Redis cluster with cross-region replication disabled by design, so an EU user's session data never crosses into the US store. Each app instance reads the session on request start and writes it back on request end; a local, short-time-to-live (TTL) cache in front of Redis (5 seconds) absorbs the read traffic for users making several requests per second, so the added latency in the common case is effectively the local cache lookup, not a Redis round trip. Auth uses short-lived JWTs (a few minutes) plus a longer-lived refresh token that is checked against the Redis-backed session record on every refresh; revoking a user (support ticket, password reset, or a fraud flag) sets a revoked flag on that record, so their access token stops renewing once it expires and, for the small number of write-sensitive endpoints, is checked directly against the record on every call. This gives the team elastic autoscaling and blue/green deploys without cold-session churn, at the cost of running and monitoring two regional Redis clusters instead of zero.

Trade-offs & pitfalls

• Reflexively "always externalize" is a mistake for small, low-scale, or lift-and-shift systems where the operational cost of running a session store isn't yet justified; sticky sessions with a short-TTL fallback can be the pragmatic starting point.
• Under-provisioning the session store's replication is the most common failure: teams externalize state assuming it fixed failover, then discover the store itself is a new single point of failure because they ran it as a single node.
• Storing large or unbounded session payloads in the shared store (shopping-cart history, uploaded-file references) rather than keeping sessions small and pushing bulk data to its own store is a frequent scaling mistake that shows up as store memory pressure and slow serialization, not as an obvious session bug.
• Treating token revocation as solved by "just make JWTs short-lived" without also handling the case where a token must be killed immediately (compromised account) is a common gap; short expiry reduces the blast radius but does not give you an immediate kill switch on its own.

Direct answer

Size each pool from the request rate implied by the concurrency, not from the concurrency number itself: work out how many requests per second actually need a database connection or an outbound HTTP connection at steady state, multiply by how long each connection is held (Little's Law), then add an explicit safety margin. Timeouts must be shorter than the end-to-end service level objective (SLO), and connections need a maximum lifetime shorter than any upstream infrastructure's idle-connection timeout so sockets get rotated before they are silently dropped. A circuit breaker sits in front of each pool so a failing dependency fails fast instead of exhausting the pool while everyone queues for a resource that will not respond.

Sizing the pools

Step 1: convert concurrency to throughput. 500 concurrent handlers is a snapshot of how many requests are in flight, not a rate. Little's Law relates them: concurrency equals arrival rate times average time in system ($L = \lambda W$). If the average end-to-end request latency is known (or targeted), the implied arrival rate falls out directly.

Step 2: split by dependency. Not every request touches the database, and not every request calls the same upstream. Measure (from application performance monitoring, not guesswork) what fraction of requests hit each dependency and how long that call takes, then apply Little's Law again, this time scoped to that dependency, to get the concurrent connections it needs.

Step 3: timeouts and lifetimes.

• Connect timeout: short, on the order of a few hundred milliseconds for calls inside the same network, since a connection that cannot even establish is not going to succeed on retry within budget.
• Request/read deadline: a slice of the overall SLO, sized so that even if this call takes its full budget, the caller still has time left to respond.
• Idle timeout: closes unused connections after they've sat idle for a while, freeing capacity for other requests.
• Max connection lifetime: rotates connections periodically so no single socket outlives an upstream load balancer's or NAT device's own idle-reap window, which would otherwise cause requests to fail against a connection that looks alive locally but was already dropped upstream.
• Pool checkout timeout: kept short. If a caller cannot get a connection quickly, failing fast and shedding load beats piling requests into an ever-growing queue.

Step 4: circuit breakers and bulkheads. Wrap each pool (or each upstream host) with its own breaker: once the failure rate or consecutive-failure count crosses a threshold, the breaker opens and callers get an immediate fallback or error instead of waiting out a timeout against a resource that is already failing. A bulkhead means giving a critical dependency its own pool, separate from a non-critical one, so a slow or saturated non-critical dependency cannot starve connections a critical path needs. The state-machine mechanics of breakers (closed / open / half-open, backoff between probes, how they interact with retries and cascading failure) are a deeper topic in their own right; what matters here is that the pool and the breaker are wired to the same dependency boundary, so a breaker trip and a pool-exhaustion alert are talking about the same failure.

Step 5: test it. Load test by ramping to the target concurrency and then spiking beyond it, watching pool wait time and saturation rather than just end-to-end latency. Separately, inject faults (added latency, forced errors) on the database and the upstream to confirm the breaker actually trips and the fallback path is exercised, not just that it exists in code. Run a soak test long enough to exercise the max-lifetime rotation at least once, to catch connection leaks that a short test would miss.

Worked example

Assumptions used below (explicitly stated, not measured facts):

• Concurrency $L = 500$ handlers (given).
• Target average end-to-end latency $W = 200\,\text{ms}$ (an illustrative SLO target, not a measured value).
• 30% of requests make a database call, averaging 20 ms per call (illustrative).
• 50% of requests call an upstream HTTP service across 5 upstream hosts, averaging 50 ms per call (illustrative).
• Safety factor of 2x on the computed steady-state connection counts (illustrative, tune from observed variance).

Database pool:

With the 2x safety factor, target pool size = 30, capped at whatever the database server's own connection limit allows (if the DB caps total connections lower than 30 times the number of service instances, add a proxy/pooler in front of the database rather than growing the per-instance pool further).

HTTP client pool:

Spread across 5 upstream hosts: $63 / 5 \approx 12.6 \rightarrow 13$ per host minimum, times the 2x safety factor = 26 per host target. (For HTTP/2 upstreams that multiplex many logical requests over one connection, this per-connection count would be far smaller; the formula is the same, only the effective "one connection handles many concurrent calls" changes the divisor.)

flowchart LR
    C[Request handlers x500] --> CB1{Circuit breaker: DB}
    C --> CB2{Circuit breaker: upstream HTTP}
    CB1 -->|closed| DBPool[DB connection pool target 30]
    CB1 -->|open| FailFast1[Fail fast / fallback]
    DBPool --> DB[(Database)]
    CB2 -->|closed| HTTPPool[HTTP client pool per host 26]
    CB2 -->|open| FailFast2[Fail fast / fallback]
    HTTPPool --> Upstream[Upstream service]

Trade-offs and pitfalls

Over-provisioning a database pool is not free: every open connection costs the database server memory and, for some engines, a background process, so "just make the pool bigger" is not a safe default. Under-provisioning causes callers to queue for a checkout instead of failing fast, which turns a capacity problem into a latency-cascade problem for every caller of the service, not just the ones touching the saturated dependency. A common ordering mistake is setting an inner call's timeout longer than the budget the outer caller has left, which wastes the outer request's remaining time waiting on a call that can no longer help it finish in budget; timeouts should shrink, not grow, as you move deeper into a call chain. Finally, a load test that only ramps to the steady-state target and never spikes past it will not catch pool exhaustion, since the whole point of a pool is that it behaves fine until it doesn't.

Direct answer

For a pub/sub service with a huge number of channels and a client base that is mostly mobile and frequently offline, prefer server-side partitioning over pure client-side sharding. Client-side sharding (clients hash the channel to pick their partition themselves) is operationally cheap but turns every flaky-network reconnect into a fresh routing decision made independently by thousands of clients. Server-side partitioning (the broker assigns and owns the mapping) keeps that complexity centralized, where it can buffer subscription state through a drop and rebalance in a controlled, batched way instead of leaving it to uncoordinated client behavior.

Structured elaboration

The two placement strategies

• Client-side sharding: the client computes partition = hash(channel_id) % n (or a similar scheme) and connects directly to the broker owning that partition. No broker-side coordination service is needed.
• Server-side partitioning: clients connect to a stateless gateway or routing tier; a partition-metadata service tells the gateway which broker shard owns the channel, and the broker fleet can move ownership without the client ever recomputing anything.

Compare on the dimensions that matter here

• Rebalancing cost: under client-side sharding, changing shard count means every client must recompute its mapping, and if the mapping uses naive modulo hashing, nearly all clients reconnect to a different shard at once. Server-side partitioning can migrate ownership of a subset of channels while the gateway keeps existing client connections open, so the client never sees the rebalance directly.
• Fairness: client-side hashing has no visibility into actual load, so it can leave one shard hot while another idles. A server-side metadata service can rebalance based on measured load per shard, not just key distribution.
• Latency: client-side sharding has one fewer hop if the client connects straight to the right broker. Server-side partitioning adds a routing lookup, but that lookup can be cached at the gateway and is usually a small fraction of a mobile network's own round-trip time.
• Client churn from intermittent connectivity: this is the deciding factor for a mostly-mobile, frequently-offline client base. Client-side sharding treats every reconnect as a brand-new decision, so a client that drops and reconnects seconds later re-derives its partition and may miss messages published in between. Server-side partitioning lets the broker hold the subscription state (with a bounded grace period) so a quick reconnect resumes the same logical subscription instead of starting over.

A brief note on mechanics: whichever side owns the mapping, production systems generally implement the underlying hash-to-shard assignment with a consistent-hashing-style ring rather than plain modulo, specifically to bound how many keys move when shard count changes: nodes and channels are placed as points on a circular hash space (the ring), and virtual nodes give each physical shard many small points on it instead of one, so only the moved points' channels need to migrate. The ring construction and virtual-node tuning that make that work: giving each physical shard many small points scattered around the ring instead of one large contiguous range means more virtual points per physical shard smooths load distribution further but adds more routing metadata to maintain, while fewer points is cheaper to track but risks uneven load; the point relevant here is just that the choice of hashing scheme, not just who runs it, determines how disruptive a rebalance is.

Worked example

Assume, as a planning input rather than a measured fact, a fleet of 50 broker shards holding about 2 million channels between them, and that the service needs to grow to 60 shards to handle load. Under a consistent-hashing-style ring, the expected fraction of channels that move when going from n to n + m shards is approximately:

$\frac{m}{n+m} = \frac{10}{60} \approx 16.7\%$

Under naive modulo hashing (partition = hash(channel_id) mod n), changing the divisor n reassigns nearly every key, so close to 100% of channels remap even though only 10 shards were added. That gap, roughly 17% of clients reconnecting elsewhere versus nearly all of them, is why a server-side partitioning layer that controls the hashing scheme matters more than which side technically "does" the sharding: a mobile client base that reconnects constantly on its own cannot absorb a near-100% remap event without a visible spike in dropped or duplicated messages.

Trade-offs & pitfalls

• A server-side metadata/routing tier becomes a new critical-path dependency. If it is a single instance rather than a redundant, health-checked fleet, it becomes the system's weakest point exactly when you added it to improve reliability.
• A mass reconnect event (an app release, a regional network outage recovering) can still overwhelm a well-designed server-side system if all clients retry at once; client SDKs need jittered backoff regardless of which sharding approach is chosen.
• Client-side sharding is not always wrong: for a small number of large, stable shards and a client base with reliable connectivity (internal services, desktop clients), its simplicity and lack of a coordination service can be the better trade.
• Buffering subscription state server-side to smooth over churn has a cost: unbounded buffering for offline clients grows broker memory. A grace period with a hard cap forces an explicit decision about how "offline" a client can be before it must fully resubscribe.

flowchart LR
  subgraph Server-side partitioning
    C1["Mobile client"] --> GW["Stateless gateway"]
    GW --> MD["Partition metadata service"]
    MD --> GW
    GW --> B1["Broker shard 1"]
    GW --> B2["Broker shard 2"]
  end