Scalability & Capacity Planning Questions

Definition

Amdahl's Law describes the ceiling on speedup from adding more parallel workers to a workload that has both a parallelizable part and a fraction that must run serially (one step at a time, no matter how many machines are available). If P is the parallelizable fraction of the work and N is the number of workers:

Speedup(N) = 1 / ((1 - P) + P/N)

Intuition: as N grows large, P/N shrinks toward zero, so speedup approaches 1 / (1 - P). The serial fraction alone sets a hard ceiling on how much adding machines can ever help, no matter how many you add.

A concrete example

Say a batch job is 95% parallelizable (P = 0.95) and 5% serial, for example a final merge step that has to run on one node after everything else finishes:

• N = 10: Speedup = 1 / (0.05 + 0.095) = about 6.9x
• N = 100: Speedup = 1 / (0.05 + 0.0095) = about 16.8x
• N = 1,000: Speedup = 1 / (0.05 + 0.00095) = about 19.6x
• N approaching infinity: Speedup approaches 1 / 0.05 = 20x

Going from 10 to 100 machines, a 10x increase in hardware, only takes speedup from 6.9x to 16.8x. Going from 100 to 1,000 machines, another 10x increase, only buys 16.8x to 19.6x. The fixed 5% serial cost dominates once N is large, and the returns fall off a cliff well before infinite hardware would suggest.

Where the serial bottleneck actually comes from

In real systems it's rarely a literal single-threaded step, it's usually coordination: a global lock, a leader-election protocol like Raft or Paxos (algorithms that let a cluster of machines agree on a single coordinator for writes, so only one node accepts changes at a time) that every write has to go through, or a service where reads are horizontally scaled but every write still funnels through one primary database. Amdahl's Law treats the serial fraction as fixed, but coordination overhead often grows as N grows (a leader has to coordinate with more followers), which is what the Universal Scalability Law adds on top: a term for cost that increases with node count, capturing the real-world case where adding machines past some point can make throughput worse, not just plateau. Concretely, USL adds two terms beyond Amdahl's serial fraction: contention (time lost waiting to access a shared resource, like that one primary database) and coherency (time lost keeping multiple copies of the data in sync with each other), and both of those terms grow larger as N grows, which is why USL's curve can turn downward past some node count instead of just flattening the way Amdahl's does.

What I'd look for to judge if a system is a good candidate for horizontal scaling

• Statelessness or partitionability: can the work be split into independent units, or does shared state have a natural key to shard it by?
• How much of the critical path is truly global: any step that must touch a single writer, a global counter, or a distributed lock caps the achievable speedup regardless of node count.
• Whether coordination cost stays roughly constant or grows with node count: a design where nodes only talk to their own shard's leader scales very differently from one where every node talks to every other node.
• Whether the serial fraction can actually be shrunk by redesign, for example replacing a single merge step with a tree-shaped reduce, rather than being accepted as fixed.

What I'd want to know first

Before I put a number on anything, I'd pull four things: the traffic profile (is it steady, bursty, or spiky through the day), the read/write mix (reads and writes scale very differently), the resource cost per request (how much CPU, memory, and database load one typical request consumes), and the existing SLA (service level agreement) or error budget (how much latency and error headroom I'm actually allowed to spend). Skipping this step and jumping straight to "just double everything" is the most common mistake, because growth is rarely uniform across every resource.

Headroom calculation

Say the service peaks at 3,000 requests per second (RPS) today, running on 20 instances. Load testing shows each instance holds a safe ceiling of about 175 RPS before p99 latency (the 99th percentile response time, the slowest 1 in 100 requests) starts climbing. That puts the fleet's tested ceiling at 20 x 175 = 3,500 RPS, so today's headroom is about 500 RPS, roughly 14% of the tested ceiling (500 of 3,500 RPS still unused).

If traffic doubles in 6 months, peak becomes 6,000 RPS. I'd size for that peak plus a buffer for forecast error, say 20%, giving a target ceiling of 7,200 RPS. At 175 RPS per instance that's 7,200 / 175 = about 42 instances, roughly doubling the fleet plus a bit more because the buffer also grows.

Scaling strategy

For a stateless web tier fronting a database, horizontal scaling (adding more instances of the same size) is usually the right default: no single-instance ceiling to hit, and it's cheaper to add commodity capacity than to keep buying bigger boxes. But I'd check the database write path separately, since it often becomes the actual constraint before the web tier does. If the database is the binding resource, the plan has to include read replicas, connection pooling limits, or a write-path change, not just more web servers.

What I'd benchmark

• A steady-state load test at the projected 6,000 RPS peak with the real production read/write mix, not a synthetic uniform load.
• A soak test (sustained load over hours) to catch memory leaks and connection pool exhaustion that only show up over time.
• A burst test that ramps quickly to the new ceiling, since sustained-load tests can hide how the system behaves under a sudden spike.

Cost versus performance

If each instance costs roughly $140/month, the fleet grows from $2,800/month to about $5,880/month, a cost increase of around 110% to support a 100% traffic increase, which is close to linear and reasonable. If the math came out sharply super-linear (say 300% cost for 100% traffic), that would push me to look at right-sizing the instance type or fixing an inefficiency instead of just buying more of the same box.