Scalability Patterns and Techniques Questions

Direct answer

Yes, the downstream system is a clear bottleneck, by roughly 10x. At 200 incoming requests per second (RPS) with a fan-out of 50 calls each, the service generates 10,000 downstream calls per second, but the downstream system only accepts 1,000 RPS. Fixing this requires either reducing the number of downstream calls per incoming request (caching, batching, coalescing) or decoupling the response from the downstream work (async processing), not simply adding more capacity on your own side.

The math

Throughput check. Fan-out multiplies the incoming rate directly:

$200 \text{ RPS} \times 50 \text{ calls/request} = 10{,}000 \text{ downstream calls/s required}$

$\text{required} : \text{capacity} = 10{,}000 : 1{,}000 = 10 : 1$

Required load is 10 times the downstream cap. That alone confirms a bottleneck: no amount of retrying or connection pooling on your side changes a system that is already saturated at its own ceiling.

Concurrency cross-check (Little's Law). It helps to sanity check the same conclusion from a different angle: how many downstream calls must be in flight simultaneously, not just per second. Little's Law relates throughput $X$ and average time-in-system $R$ to the average number of concurrent items $N$:

$N = X \times R$

At the P95 latency (the response time that 95% of requests come in faster than) of about 100 ms (0.1 s), the concurrency the fan-out actually demands is:

$N_{\text{required}} = 10{,}000 \text{ RPS} \times 0.1\text{s} = 1{,}000 \text{ concurrent downstream calls}$

Whereas the downstream system, operating at its own stated cap with that same latency, is only structured to sustain:

$N_{\text{capacity}} = 1{,}000 \text{ RPS} \times 0.1\text{s} = 100 \text{ concurrent downstream calls}$

Both views agree: you need about 10x the concurrency the downstream system is built to hold. This cross-check matters in an interview because it shows the bottleneck isn't just a rate-limit number on a dashboard, it is a real resource constraint (connections, threads, or queue slots) that a naive retry loop would make worse, not better, by piling on more concurrent attempts against an already-saturated system.

Mitigation options and what each one requires

Different mitigations close the 10x gap in different ways. It is worth deriving the minimum each one needs before choosing, rather than picking whichever sounds most familiar:

The queueing row is the one candidates most often get wrong: a queue is a shock absorber for bursts, not a source of extra downstream capacity. If the arrival rate into the queue is sustained above the rate downstream can drain (1,000 RPS here), the backlog and its latency grow without bound over time; it only helps if the 10,000 RPS demand is a transient spike layered on top of a steady-state average that downstream can actually absorb.

Worked example: combining two mitigations

A single technique often has to hit an aggressive threshold alone (90% cache hit rate, or a batch factor of 10). Combining two moderate mitigations is usually more realistic. Assume, as an illustrative starting point (not a measured figure), a cache hit rate of 80% and a batch factor of 3 for the remaining traffic:

$\text{uncached calls} = (1 - 0.8) \times 10{,}000 = 2{,}000 \text{ RPS}$

$\text{after batching by 3} = \frac{2{,}000}{3} \approx 666.7 \text{ RPS}$

666.7 RPS is below the 1,000 RPS cap, with about 33% headroom. This is a useful pattern to point out explicitly: two moderate, individually achievable improvements (an 80% hit rate is realistic for many read-heavy access patterns; batching 3 calls together is a small API change) can beat needing one extreme, harder-to-sustain number from a single technique.

Trade-offs and pitfalls

• Treating "add a queue" as the fix without checking the sustained drain rate. A queue converts an overload into a growing backlog; it does not remove the overload.
• Choosing a hit rate or batch factor that meets the cap with zero margin. Production traffic is bursty and cache hit rates drift, so design to a threshold with headroom, not the exact breakeven point.
• Applying caching or batching uniformly across all 50 downstream calls when only some of them are actually cacheable or batchable in practice; the real achievable reduction is bounded by whichever calls are eligible.
• Retrying failed downstream calls without first fixing the 10x overload. Retries against an already-saturated system amplify load and can turn a slow degradation into a full outage.
• Skipping the concurrency cross-check. Throughput alone can hide a resource-exhaustion story (thread pools, connection limits) that shows up as timeouts before the RPS counter ever looks alarming.

Direct answer

For a user-profile service growing from 10 million to 1 billion users, I would shard on a hashed user ID for even distribution, migrate via a dual-write-then-cutover process driven by a routing layer (not a hard-coded formula the application computes itself), and rebalance by moving individual partitions with change-data-capture (CDC) to keep the source and destination in sync until a near-instant cutover. Consistent hashing wins for this workload because access is almost always by single user ID with no need for range scans, and its whole advantage is minimizing data movement when shards are added.

Structured elaboration

Shard-key selection criteria

1. Cardinality and uniform distribution: the key must have enough distinct values, and typical access patterns must spread evenly across them. A hashed user ID satisfies this; a raw signup-timestamp-derived ID does not, because recent users cluster on a few shards while old ones sit cold.
2. Stability: the key must never change after assignment. user_id qualifies; email or username do not, since a rename would require moving the row.
3. Query locality: does the workload need range scans or joins across users? For a profile service, almost all reads and writes are single-user lookups, so locality by an ordered range is not needed and a hash-based spread is safe.
4. Hot-key exposure: a small number of accounts (very active or high-visibility profiles) can dominate traffic on their shard regardless of key choice; the design needs an explicit way to detect and mitigate that (traffic-based alerting plus optional per-key overrides), not an assumption that hashing alone prevents hotspots.
5. Geographic skew: if the user base concentrates in specific regions rather than distributing evenly worldwide, a pure hash on user ID can still produce uneven infrastructure load even though key distribution is even, because a shard's users may all be far from the datacenter serving them, or a region's traffic may spike in a way that maps to a subset of shards. This is a distinct problem from key skew: it is about where a shard's traffic originates and lands, not how many keys it owns. The mitigation is to let shard placement (which datacenter/region a shard physically lives in) be a separate decision from the shard key itself.

Routing design: algorithmic versus lookup-service-based

A pure algorithmic router (client computes hash(user_id) mod N or applies consistent hashing directly) is simple and needs no extra network hop, but it hard-codes the shard topology into every client, which makes it painful to move individual users off a hot shard or to change placement for geographic reasons.

The alternative, which is what I would use at this scale, is a shard-map-management routing layer: a small, highly available service that owns the authoritative mapping of user_id → shard, so clients ask the routing layer (or a cached copy of its map) rather than computing the answer themselves. This costs an extra lookup (mitigated with aggressive client-side caching of the map, invalidated on change) but buys the ability to move individual users, override placement for geographic locality, and steer around a hot shard without a client rollout. At the higher end of this growth curve, once the user base is well past a billion (some teams see this discussion framed around roughly two billion users), the routing layer typically also needs sticky routing: caching a client's resolved shard for the duration of a session so repeated requests for the same user do not re-hit the routing layer on every call, trading a small staleness window (the cache might briefly point at a user's old shard mid-migration) for materially lower routing-layer load.

Migration approach: single shard to many

1. Stand up the routing layer first, initially mapping 100% of users to the single existing database, so all future migration is just a mapping change.
2. Provision the target shards and, for each cohort of users being moved, start a backfill copy from the source into the destination shard.
3. Enable dual-write for that cohort: once backfill completes for a user, writes go to both source and destination while reads still come from source.
4. Verify the destination has caught up (row counts, checksums, or a CDC-lag check near zero), then flip the routing-layer entry for that cohort to the destination and stop dual-writing.
5. Retire the source copy for that cohort once a safety window has passed with no fallback reads observed.

Rebalancing with minimal downtime

flowchart LR
    A[Mark partition as migrating in routing layer] --> B[Async bulk copy source to destination]
    B --> C[CDC stream captures deltas during copy]
    C --> D[Apply captured deltas to destination]
    D --> E{Destination caught up?}
    E -- No --> C
    E -- Yes --> F[Brief write pause, apply final delta]
    F --> G[Atomically flip routing-layer entry]
    G --> H[Old source marked stale, retired after safety window]

The only downtime is the brief pause in step F while the last few milliseconds of writes replay, typically small enough to be invisible to users; everything else happens while the partition continues serving live traffic from its current location.

Consistent hashing versus range-based sharding, for this workload

Consistent hashing places nodes and keys as points on a circular hash space (the ring); virtual nodes give each physical node many small points on that ring instead of just one, so a topology change only reassigns the affected points.

Given a read-heavy, single-user-lookup access pattern with no meaningful range-query requirement, consistent hashing on a hashed user_id, fronted by a shard-map-management routing layer rather than pure client-side computation, is the right choice: it minimizes data movement as the shard count grows from a handful of shards at 10 million users to however many are needed at 1 billion, and the routing layer absorbs the geographic-skew and hot-key concerns that hashing alone does not solve.

Worked example

Starting state: one database serving 10 million users. The trigger to shard is a saturation signal, not a fixed user count: sustained high CPU/IOPS (input/output operations per second) on the primary (concretely: sustained CPU utilization above roughly 70-80%, or IOPS approaching the storage volume's provisioned ceiling). When that signal fires, the team provisions an initial set of shards behind the routing layer, sized so projected per-shard load sits comfortably under those same saturation thresholds, and migrates user cohorts onto them following the steps above. As an illustrative planning input rather than a measured fact, assume a single shard can comfortably sustain roughly 2 million active user profiles' worth of steady-state read/write traffic before crossing that same 70-80%-CPU / near-IOPS-ceiling threshold: that puts the initial split at $10{,}000{,}000 / 2{,}000{,}000 = 5$ shards when the 10-million-user primary first hits saturation. From then on, shards are added incrementally whenever the routing layer's per-shard load metrics approach saturation again, all the way to whatever shard count is needed at 1 billion users, which at that same illustrative 2-million-users-per-shard ceiling works out to $1{,}000{,}000{,}000 / 2{,}000{,}000 = 500$ shards. Because the shard key is a hash and the routing layer decouples the shard count from the shard-key formula, each incremental addition only requires migrating the specific user ranges assigned to the new shard, not a full re-partition of the dataset.

Trade-offs & pitfalls

• A pure algorithmic router is simpler to build first but becomes the bottleneck to fix later: teams that skip the routing layer to save one network hop early usually end up building it anyway once they need to move a single hot user off a shard without a client deploy.
• Sticky routing at very large scale trades a small staleness window for routing-layer relief; that staleness window must be bounded and understood, or a mid-migration read can silently hit stale data.
• Consistent hashing does not, by itself, solve a single extremely hot key (a celebrity account); that requires a separate detection-and-override mechanism layered on top, not a property of the hashing scheme.
• Geographic skew and key skew are easy to conflate: even key distribution can still coexist with badly distributed physical load if shard placement ignores where traffic originates.

Direct answer

Headroom is the fraction of total fleet capacity not currently in use: $\text{headroom} = (\text{total capacity} - \text{load}) / \text{total capacity}$. For 20 servers at 1,000 requests per second (RPS) each against an observed 95th-percentile (P95) load of 12,000 RPS, current headroom is exactly 40%, which means the fleet is already at the stated target and needs zero additional steady-state instances. The more interesting part of this problem is that "40% headroom" is not one number once operational realities like rolling deployments enter the picture, since taking servers offline to redeploy them temporarily reduces the same denominator that headroom is computed against.

Step-by-step: current headroom

Since the target is also 40% headroom, the fleet already meets it: 0 additional instances needed for steady-state P95 load as given.

Extending the answer: headroom under rolling deployment

A steady-state headroom number does not survive a rolling deployment unchanged, because a rolling deploy takes a batch of servers offline (to restart and warm up) while the rest of the fleet absorbs the same load. If a target recovery time objective (RTO) bounds how long a batch may be down, and each server needs, illustratively, a 2-minute warm-up before it serves at full capacity again, then the fleet needs to keep at least the minimum serving capacity above throughout the rollout, not just at rest.

Solving for the minimum number of servers that must remain in service to hold 40% headroom during a drained window, using the same 12,000 RPS load:

That is the same 20 servers as the steady-state fleet, which means a rolling deploy that takes any servers offline at all will temporarily breach the 40% target unless extra servers are provisioned specifically to cover the batch that is mid-restart or mid-warm-up. With an illustrative batch size of 2 servers drained at a time (a deliberately conservative choice to bound blast radius and keep the 2-minute warm-up window short in aggregate):

So provisioning 22 servers instead of 20, two more than the steady-state minimum, keeps 20 servers always serving even while 2 are cycling through the 2-minute restart-plus-warm-up window, preserving the 40% headroom target throughout the rollout rather than only at rest. This same per-minute-granularity view, "how much serving capacity is available right now, given who's mid-warm-up," is what feeds a rolling capacity forecast into an autoscaler policy; because the forecast window is short (on the order of the 2-minute warm-up lead time itself), a lower steady-state buffer, for example a 20% headroom target rather than 40%, is often sufficient for that forecast layer, since it only has to smooth over the next couple of minutes rather than absorb a full traffic-growth cycle.

A second worked example: rolling maintenance at larger scale

The same batch-drain formula applies at a different fleet size with different constraints. Take a 100-server fleet undergoing rolling maintenance where each server needs a 2-minute restart followed by a 3-minute warm-up, and the operational requirement is to keep at least 80% capacity serving throughout:

With a maximum batch of 20 servers per wave and 100 servers total, that's 5 waves ($100 / 20$). At roughly 5 minutes per wave (2-minute restart plus 3-minute warm-up), a fully serial rollout takes about 25 minutes; waves could be shortened by running them with some overlap once a wave's warm-up phase no longer needs to block the next wave's restart phase, but that adds coordination complexity in exchange for a shorter total window.

Validating the headroom target with load testing

A headroom number computed from stated per-server capacity is only as good as that capacity figure. Before trusting it operationally:

• Stress test: push a single server (or a small cluster) past its stated 1,000 RPS to find its actual breaking point, confirming the capacity figure used in the headroom math is not optimistic.
• Soak test: hold the fleet at target load for an extended period to catch degradation that only shows up over time (memory growth, connection exhaustion), which a short burst test would miss.
• Spike test: apply a sudden jump well above the P95 load figure to confirm the stated headroom actually absorbs a real burst, not just the smoothed average the P95 number represents.
• Ramp-up schedule and success criteria: define the load curve in advance (for example, step up by 20% of capacity every few minutes) and a clear pass/fail bar (P95 latency stays under target, error rate stays near zero) rather than eyeballing dashboards during the test.

Trade-offs and pitfalls

The most common mistake here is computing headroom once at rest and treating it as a constant, when in practice every rolling deployment, maintenance window, or partial-zone failure temporarily changes the denominator; a fleet sized exactly to its steady-state headroom target has effectively zero headroom the moment any servers are intentionally taken offline. The second common mistake is picking a batch size for rolling operations based on deployment speed alone, without checking that the resulting drained capacity still clears the headroom bar, which is exactly the kind of gap that surfaces as a latency spike during otherwise-routine maintenance rather than during an actual traffic surge.

Direct answer

At 1 million requests per second (RPS), even a fully stateless service hits six distinct application-layer ceilings before the database ever enters the picture: raw network throughput, the thread/concurrency model used to handle requests, how connections are accepted and held, the CPU (central processing unit) cost of TLS (Transport Layer Security, the encryption protocol behind HTTPS) termination, the CPU and allocation cost of serialization, and garbage-collection (GC) pauses in managed runtimes. Each has its own symptom, its own mitigation, and its own way to measure how much it is actually costing you, so the right approach is to profile for all six rather than assume which one dominates.

Bottleneck, mitigation, and how to quantify it

Worked example: why the thread model has to change at this scale

As an illustrative assumption, not a measured figure, suppose the in-application processing budget (excluding network transit) needed to hit a reasonable P95 (95th-percentile latency) target is 20 ms per request. By Little's Law, the number of requests that must be handled concurrently to sustain 1,000,000 RPS at that latency is:

$N = X \times R = 1{,}000{,}000 \text{ RPS} \times 0.02\text{s} = 20{,}000 \text{ concurrent requests}$

If each of those were handled by a dedicated OS thread, and again as an illustrative assumption, suppose each thread reserves roughly 2 MB of stack:

$20{,}000 \text{ threads} \times 2\text{ MB/thread} = 40{,}000\text{ MB} = 39.06 \text{ GiB in thread stacks alone}$

That is memory spent before a single byte of request data or response buffer is allocated, and it scales linearly with load: doubling throughput or the latency budget doubles the stack overhead in this model. This is the concrete argument for an event-driven or lightweight-concurrency runtime: the same 20,000-way concurrency can be held by a small, fixed pool of OS threads multiplexing many logical requests, so memory and context-switch overhead stop scaling with request count.

Trade-offs and pitfalls

• Optimizing the bottleneck that is easiest to fix instead of the one profiling actually shows dominates. All six of these can look plausible from first principles; the profiling step is what separates a real fix from a guess.
• Fixing serialization or TLS cost with a change that only helps if both client and server can be changed together (a binary format, session resumption). Confirm you actually control both ends before committing to the approach.
• Tuning a garbage collector for lower pause times without checking the throughput cost; low-pause collectors typically trade some raw throughput or memory overhead for shorter, more predictable pauses, which is usually the right trade at the tail but should be a deliberate choice.
• Solving connection-handling and thread-model problems independently when they interact: an async runtime that still blocks on a synchronous crypto or serialization call in the hot path reintroduces the same contention it was meant to remove.
• Load-testing with unrealistic request shapes (uniform tiny payloads, no TLS) that hide exactly the bottlenecks (serialization at scale, real handshake cost) this question is about; test with payload sizes and TLS configuration that match production.

Direct answer

As incident commander, the first move is to stabilize without waiting on the cloud provider: manually add capacity wherever quota headroom still exists, and shed or queue the traffic you can't serve so the SLO (service level objective, the target you've committed to for availability or latency) breach doesn't get worse. In parallel, escalate to the provider and give stakeholders a clear, honest status. The incident is not resolved by root-causing; it's resolved by capping the damage now and then building the quota monitoring, predictive scaling, and updated runbooks that make sure a known quota ceiling never again gets discovered mid-spike.

Structured elaboration

Immediate mitigations (first 0-30 minutes)

• Manually provision capacity in a region, account, or instance family that still has quota headroom, preferring larger instance types over more instances if the limiting quota is instance count rather than vCPU.
• Throttle non-critical traffic at the edge: return 429 (rate limited) or 503 (unavailable) with a Retry-After header for low-priority requests (bulk exports, background jobs), preserving capacity for the traffic that actually matters.
• Apply admission control instead of dropping traffic uniformly: if only a fraction of demand can be served, let requests in up to capacity (first-come or a fair queue with an ETA) rather than randomly failing a percentage of everyone's requests. This is what separates "everyone gets a slow, unfair experience" from "most users are unaffected and the rest see a clear queue."
• If a queue sits in front of a write-heavy path, let it absorb the burst rather than pushing writes straight through to the database at spike rate.

Communication

• Cloud provider: open a severity-1 support case immediately with the quota metric, current usage, and requested limit; this is not a channel to wait on for immediate relief, so treat it as a parallel track, not the mitigation itself.
• Internal: post a structured status (what's affected, current mitigation, next update time) to a single incident channel on a fixed cadence, not ad hoc.
• Stakeholders and, if customer-facing, a status page: state the actual impact honestly and give a concrete next-update time rather than a resolution promise you can't back.

Medium-term fixes

The underlying gap this incident exposes is that scaling was purely reactive with no advance knowledge of demand. The fix is to design ahead for the traffic patterns you can actually anticipate, which fall into a few recognizable shapes:

• A short, extreme, scheduled burst (for example, a flash sale expected to run around 100,000 requests/second for roughly 10 minutes): because the timing is known in advance, pre-scale capacity ahead of the event rather than relying on the autoscaler to react to it live, put a queue in front of the write path to smooth the burst instead of hitting the database at peak rate, and stage the pre-scale as a monitored, reversible step (canary the added capacity, keep an automatic rollback if error rates rise) rather than a one-way commit.
• A sharp, less-precisely-timed spike (for example, a marketing email that drives a 10x jump within about 30 minutes): the exact start time is fuzzier than a scheduled sale, so the plan needs a short-term component (aggressive reactive scaling plus throttling as a backstop for the first few minutes) and a separate medium-term component (pre-notifying the team before large sends go out, and pre-warming capacity ahead of known send windows so the reactive layer isn't starting from a cold baseline).
• An extreme, unscheduled-feeling spike far above baseline (for example, roughly 100x normal traffic during a flash sale): at that magnitude, no autoscaler reacts fast enough on its own, so the design has to include pre-warmed capacity sized to the expected peak and admission control that treats all waiting users fairly (a first-come or randomized queue with a visible position or ETA) instead of an uncontrolled scramble where whoever's request happens to land first wins and everyone else gets errors.

Across all three shapes, the common fix is the same: stop treating "quota is sufficient" as an assumption and start treating it as a monitored, tested constraint, with permanent quota increases requested ahead of realistic peak-plus-buffer, not discovered during an incident.

Runbook and alert updates

• Add a dedicated quota-exhaustion playbook: exact commands for manual provisioning in each region/account, the throttling levers available, the provider escalation contact path, and a decision matrix for when to degrade vs. scale vs. fail over.
• Add quota-utilization alerts at conservative thresholds (for example, flagged at 60/75/90% of the current limit as an illustrative staging, not a universal standard) so the team requests an increase before hitting the wall, not after.
• Add an alert specifically for "autoscaler issued a scale-out that did not result in additional serving capacity within an expected window," which is a different failure than "no scale-out was attempted" and needs its own signal.
• Schedule runbook drills (tabletop or live) that specifically simulate a quota ceiling being hit, since a runbook that has never been rehearsed against this exact failure mode is unlikely to be followed correctly under real pressure.

Worked example

Assume, as stated planning inputs rather than measured facts: steady-state traffic of 5,000 requests/second (RPS) served by 50 instances, giving a baseline capacity ratio of

A flash-sale spike hits 100,000 RPS for about 10 minutes, which is a total request volume of

At 100 RPS/instance, serving the full spike needs 100,000 / 100 = 1,000 instances. If the account's instance quota is capped at 200, the achievable capacity at that ceiling is

which is only a fifth of demand, so the fraction of traffic that has to be shed or queued once the quota ceiling is hit is

an 80% shortfall. That number is the case for admission control over random shedding: dropping 80% of requests indiscriminately produces a bad experience for everyone, while admitting exactly the 20,000 RPS the fleet can serve and fairly queuing the rest (with a visible wait, not a silent failure) turns the same shortfall into a bounded, predictable degradation instead of a chaotic one. It's also the case for the medium-term fix: a permanent quota request sized to at least the 1,000-instance peak, not the 200-instance historical average, is what prevents this specific ceiling from being hit again.

Trade-offs & pitfalls

• An emergency quota increase request is not instant relief; the mitigation plan cannot depend on the provider responding within the incident window, only the medium-term fix (a pre-approved higher baseline quota) removes that dependency.
• Manually bringing up capacity in a different region can violate data-residency or added-latency assumptions the service normally relies on; that trade-off needs to be made consciously during the incident, not discovered afterward.
• Throttling everyone equally punishes both low- and high-value traffic the same way; admission control that's blind to request importance is only marginally better than dropping randomly.
• Alerting only on "the autoscaler failed to scale" misses the earlier, more useful signal: quota utilization climbing toward its ceiling before a scale-out attempt ever fails.