Scalability & Capacity Planning Questions

Data sources

Historical resource utilization at fine enough granularity to see daily and weekly cycles (not just daily averages, which smooth away the actual peak), the business calendar for known seasonal events specific to this service, the product roadmap for planned launches or marketing pushes that would shift the baseline, and, because this is a stateful service, the data growth rate itself: storage size and replication lag trends, since state doesn't reset the way a stateless tier's load does.

Forecasting method

A plain trend line on raw traffic would smooth the seasonal peak away and under-forecast it. Instead I'd decompose trend and seasonality separately: take last year's comparable peak, apply this year's underlying growth rate to it. For example, if last year's seasonal peak was 40,000 requests per second (RPS) and the trend growth since then has run about 25% year over year, this year's forecast peak = 40,000 x 1.25 = 50,000 RPS. I'd cross-check that 25% growth figure against the last several weeks of baseline traffic to make sure the trend is still holding before betting the forecast on it.

Headroom policy

Target ceiling = forecast peak plus a buffer for forecast error, say 25%: 50,000 x 1.25 = 62,500 RPS. Because it's a stateful service, I'd also decide how much of that headroom needs to be pre-warmed ahead of time (cache populated, connection pools ramped, replication caught up) rather than scaled into reactively during the spike, since a stateful tier generally can't absorb a sudden load increase as fast as a stateless one can.

Cost trade-offs

Holding a 62,500 RPS-capable fleet year-round is expensive if the spike only lasts a couple of weeks a year. For example, if the non-peak fleet costs $10K/month and the full peak-ready fleet would cost $25K/month, holding it year-round costs roughly $180K/year extra, versus holding the extra capacity only for a 3-week window around the peak, which costs a small fraction of that. The middle path is usually a modestly larger always-on baseline plus a temporary reservation for the known spike window.

Testing the assumptions before the peak

I'd run a full load test at the 62,500 RPS ceiling several weeks before the actual peak window, leaving time to fix anything that fails and, if needed, procure more capacity. For a stateful service specifically, that test needs to cover cache warm-up time, replication catch-up under sustained write load, and connection pool exhaustion, since those are the failure modes that don't show up in a stateless load test but do show up here.

Definition

Amdahl's Law describes the ceiling on speedup from adding more parallel workers to a workload that has both a parallelizable part and a fraction that must run serially (one step at a time, no matter how many machines are available). If P is the parallelizable fraction of the work and N is the number of workers:

Speedup(N) = 1 / ((1 - P) + P/N)

Intuition: as N grows large, P/N shrinks toward zero, so speedup approaches 1 / (1 - P). The serial fraction alone sets a hard ceiling on how much adding machines can ever help, no matter how many you add.

A concrete example

Say a batch job is 95% parallelizable (P = 0.95) and 5% serial, for example a final merge step that has to run on one node after everything else finishes:

• N = 10: Speedup = 1 / (0.05 + 0.095) = about 6.9x
• N = 100: Speedup = 1 / (0.05 + 0.0095) = about 16.8x
• N = 1,000: Speedup = 1 / (0.05 + 0.00095) = about 19.6x
• N approaching infinity: Speedup approaches 1 / 0.05 = 20x

Going from 10 to 100 machines, a 10x increase in hardware, only takes speedup from 6.9x to 16.8x. Going from 100 to 1,000 machines, another 10x increase, only buys 16.8x to 19.6x. The fixed 5% serial cost dominates once N is large, and the returns fall off a cliff well before infinite hardware would suggest.

Where the serial bottleneck actually comes from

In real systems it's rarely a literal single-threaded step, it's usually coordination: a global lock, a leader-election protocol like Raft or Paxos (algorithms that let a cluster of machines agree on a single coordinator for writes, so only one node accepts changes at a time) that every write has to go through, or a service where reads are horizontally scaled but every write still funnels through one primary database. Amdahl's Law treats the serial fraction as fixed, but coordination overhead often grows as N grows (a leader has to coordinate with more followers), which is what the Universal Scalability Law adds on top: a term for cost that increases with node count, capturing the real-world case where adding machines past some point can make throughput worse, not just plateau. Concretely, USL adds two terms beyond Amdahl's serial fraction: contention (time lost waiting to access a shared resource, like that one primary database) and coherency (time lost keeping multiple copies of the data in sync with each other), and both of those terms grow larger as N grows, which is why USL's curve can turn downward past some node count instead of just flattening the way Amdahl's does.

What I'd look for to judge if a system is a good candidate for horizontal scaling

• Statelessness or partitionability: can the work be split into independent units, or does shared state have a natural key to shard it by?
• How much of the critical path is truly global: any step that must touch a single writer, a global counter, or a distributed lock caps the achievable speedup regardless of node count.
• Whether coordination cost stays roughly constant or grows with node count: a design where nodes only talk to their own shard's leader scales very differently from one where every node talks to every other node.
• Whether the serial fraction can actually be shrunk by redesign, for example replacing a single merge step with a tree-shaped reduce, rather than being accepted as fixed.

Why this happened

An outage from running out of capacity usually means there was no ongoing visibility into how much headroom existed and how fast it was shrinking, capacity was being assessed reactively instead of tracked as a trend. Building a practice means turning a one-time sizing exercise into a recurring process with an early-warning signal.

Estimating headroom

For whatever resource caused the outage, say it was disk space on a stateful data store, I'd start from the current usage against the provisioned ceiling and the recent growth rate. For example: 3.2TB used out of 4TB provisioned (80%), growing at roughly 60GB per week. Time to saturation = (4,000GB - 3,200GB) / 60GB per week = about 13.3 weeks of runway at the current growth rate. That runway number, not just the current percentage, is what tells you when action is actually needed.

Metrics to catch the trend early

• The utilization trend over weeks, not just the current snapshot, since a static "80% full" reading tells you nothing about urgency.
• Leading indicators specific to the resource: queue depth or connection pool usage tend to move before latency does, and latency tends to move before hard failures do.
• A runway metric (time-to-saturation) computed from the trend, so it shows up as a number of weeks remaining rather than a percentage that has to be mentally translated.

Setting safety margins

Systems rarely degrade smoothly up to 100%; most show a latency or error "knee" (the point on the utilization curve where latency or errors start climbing sharply, well before the resource is literally 100% full) well before full saturation, often somewhere around 80-90% utilization for a database's disk or connection pool. So the safety margin should target staying comfortably below that knee, not below 100%, and the trigger threshold should account for how long it takes to actually add capacity. If procurement or migration takes 3 weeks, I would not wait until 3 weeks of runway remain, I'd set the alert at something like 8 weeks of runway so there's time to review, decide, and execute before it becomes urgent.

Communicating capacity risk before it's an incident

I'd set up a recurring capacity review (say monthly) with a simple dashboard showing runway per key resource, shared with the product team, plus an automatic alert when projected runway crosses the trigger threshold. The goal is that product teams see "13 weeks of runway, trending down" as a routine planning input well before it becomes "we're out of disk and the service is down."

Confirming the network really is the binding constraint

I wouldn't take the symptom (link saturation, rising query latency) at face value without ruling out other causes first. I'd check network interface utilization on both the app server and database side during the exact peak window, and correlate its onset with the onset of the latency increase, while also checking CPU and disk I/O on both tiers over the same window. For example: NIC (network interface) throughput at 950 Mbps sustained out of a 1 Gbps link (95% utilized), with TCP retransmissions (packets being resent because they weren't acknowledged in time, a sign the network link is congested) rising right when p99 latency (the 99th percentile response time, the slowest 1% of requests) crosses 300ms, while app server CPU sits at 40% and database CPU at 55%, both well under their own ceilings. That combination, network metric pegged and rising retransmits at the same moment latency spikes, while compute stays comfortable, is what actually confirms the network link as the binding constraint rather than assuming it from the topology alone.

Buying headroom quickly

Without touching the architecture, I'd reduce the bytes crossing that link: stop over-fetching (select only the columns actually needed instead of every column), compress payloads, batch queries to cut per-query overhead, and move some read traffic to a local cache or replica so it never has to cross the saturated link at all. These changes can ship in days and buy real headroom while a longer-term fix is planned.

The longer-term architectural fix

The quick fixes reduce load on the shared link, but the underlying issue is that traffic growth keeps colliding with a fixed-bandwidth path between two tiers that are architecturally coupled. Longer term I'd look at moving the app and database tiers onto a higher-bandwidth or lower-latency path (a dedicated link, tighter physical or network placement to cut hop count), and reducing chattiness structurally, colocating read replicas closer to the app servers, or adding a caching layer so most reads never round-trip to the database at all.

What happens next

Once the network stops being the ceiling, growth will eventually push the next resource, likely database CPU or the app tier itself, into becoming the new binding constraint. This isn't a one-time fix, it's the same identify-the-constraint exercise that needs to run again at the next growth checkpoint.

What I'd want to know about the workload first

• What the writes actually look like: single-row inserts versus batches, and whether the current bottleneck is disk I/O throughput, lock contention, or the overhead of committing (fsyncing) every individual write.
• Consistency requirements: can writes tolerate being asynchronous or eventually consistent, or does every write need to be immediately readable?
• Whether the data has a natural partition key (tenant ID, user ID, region) that writes could be split across, which is the precondition for horizontal scaling to even be viable here.
• Whether this is a one-time ceiling to clear or a trajectory that will keep climbing, since that changes how much re-architecture effort is worth investing now.

The decision framework

• Vertical scaling (a bigger instance, faster disks, more provisioned I/O): the fastest to implement and a reasonable choice if the headroom needed is modest and short-term, but it has a hard ceiling and doesn't fix a bottleneck that's inherently serial, like a single-writer commit log.
• Horizontal scaling (sharding writes across multiple database instances by a partition key): the sustainable choice if growth is ongoing and a good partition key exists, but it's a real re-architecture: a routing layer, and any query that spans shards gets harder.
• Changing the write pattern itself (batching writes, moving to an async queue, or a storage engine built for write-heavy workloads): often the highest-leverage option when the writes are small and frequent, because it reduces the cost per write instead of adding infrastructure.

A worked example

Say the primary is doing 4,000 writes/sec against a disk I/O ceiling of about 4,500 IOPS (input/output operations per second), with each write triggering its own commit. Batching writes into groups of 50 before committing turns 4,000 individual writes into roughly 80 batched commits/sec, which is far under the ceiling and buys significant headroom without touching hardware. That's why I'd usually check whether the write pattern itself can be improved before reaching for sharding.

What could go wrong with each approach

• Vertical: hits a cost or instance-type ceiling eventually, and the single instance stays a single point of failure the whole time.
• Horizontal: a poorly chosen partition key creates a hot shard that's just as bottlenecked as before, and operational complexity goes up.
• Batching/async: introduces a delay before a write is visible (staleness), needs idempotent retry handling (idempotent: safe to retry, because re-running the same write produces the same end result as running it once), and if the downstream queue consumer falls behind, the bottleneck just moves to the queue instead of being fixed.