Scalability & Capacity Planning Questions

Translating the burst requirement into a plan

Start from a tested number, not a guess: say load testing shows the current fleet's ceiling is 40,000 requests per second (RPS) before p99 latency (the slowest 1% of requests) breaches the service level agreement (SLA). The target is 100,000 RPS, plus a buffer for forecast and measurement error, say 20%, giving a target ceiling of about 120,000 RPS, three times the current tested ceiling.

Metrics and thresholds as evidence

I'd bring the load-test result itself ("today's tested ceiling is 40,000 RPS before SLA breach"), the cost-per-throughput-unit (cost per 1,000 RPS of capacity), and the consequence of not funding it: above 40,000 RPS today, the service doesn't degrade gracefully, it fails outright, because of a specific downstream constraint (for example, connection pool exhaustion). Grounding the ask in a tested number and a specific failure mode makes it evidence rather than an opinion.

Presenting the trade-off to finance

I'd translate infrastructure units into business terms rather than lead with RPS:

• Reframe the ceiling as a business-legible number: "up to X peak concurrent customers" or "X peak orders per minute" instead of "120,000 RPS."
• Reframe cost per unit of the ask, not just the total: "$Y per month buys headroom for Z% more peak customers," rather than a raw infrastructure dollar figure with no anchor.
• Present tiers instead of one fixed number, so finance chooses the risk level rather than being asked to approve or reject a single figure.

Worked example

Current infra costs $50K/month at the 40,000 RPS ceiling, a unit rate of about $1.25 per RPS of capacity per month. Scaling to the 120,000 RPS target is roughly 3x capacity, so at that same linear unit rate that's about $150K/month, a $100K/month increase. I'd present it as three tiers built on that same rate: Tier A, minimal (80,000 RPS ceiling, about $100K/month total, a $50K/month increase, still short of the 100,000 RPS target and risks falling over on the largest expected burst), Tier B, recommended (120,000 RPS, about $150K/month total, a $100K/month increase, covers the full target with buffer), and Tier C, max safety (150,000 RPS, about $188K/month total, a $138K/month increase). Each tier is tied to a specific quantified cost and residual risk, so finance is choosing a risk level with eyes open, not just approving a single number.

What I'd want to know first

Before I put a number on anything, I'd pull four things: the traffic profile (is it steady, bursty, or spiky through the day), the read/write mix (reads and writes scale very differently), the resource cost per request (how much CPU, memory, and database load one typical request consumes), and the existing SLA (service level agreement) or error budget (how much latency and error headroom I'm actually allowed to spend). Skipping this step and jumping straight to "just double everything" is the most common mistake, because growth is rarely uniform across every resource.

Headroom calculation

Say the service peaks at 3,000 requests per second (RPS) today, running on 20 instances. Load testing shows each instance holds a safe ceiling of about 175 RPS before p99 latency (the 99th percentile response time, the slowest 1 in 100 requests) starts climbing. That puts the fleet's tested ceiling at 20 x 175 = 3,500 RPS, so today's headroom is about 500 RPS, roughly 14% of the tested ceiling (500 of 3,500 RPS still unused).

If traffic doubles in 6 months, peak becomes 6,000 RPS. I'd size for that peak plus a buffer for forecast error, say 20%, giving a target ceiling of 7,200 RPS. At 175 RPS per instance that's 7,200 / 175 = about 42 instances, roughly doubling the fleet plus a bit more because the buffer also grows.

Scaling strategy

For a stateless web tier fronting a database, horizontal scaling (adding more instances of the same size) is usually the right default: no single-instance ceiling to hit, and it's cheaper to add commodity capacity than to keep buying bigger boxes. But I'd check the database write path separately, since it often becomes the actual constraint before the web tier does. If the database is the binding resource, the plan has to include read replicas, connection pooling limits, or a write-path change, not just more web servers.

What I'd benchmark

• A steady-state load test at the projected 6,000 RPS peak with the real production read/write mix, not a synthetic uniform load.
• A soak test (sustained load over hours) to catch memory leaks and connection pool exhaustion that only show up over time.
• A burst test that ramps quickly to the new ceiling, since sustained-load tests can hide how the system behaves under a sudden spike.

Cost versus performance

If each instance costs roughly $140/month, the fleet grows from $2,800/month to about $5,880/month, a cost increase of around 110% to support a 100% traffic increase, which is close to linear and reasonable. If the math came out sharply super-linear (say 300% cost for 100% traffic), that would push me to look at right-sizing the instance type or fixing an inefficiency instead of just buying more of the same box.

Benchmark plan

I'd run three kinds of workloads: a steady-state throughput test to find each option's ceiling, a latency-under-load test at several utilization levels (say 50%, 70%, 90%) to find the point where latency starts climbing sharply (the "knee"), and a realistic production-mix workload rather than a synthetic uniform one, since real traffic mixes reads, writes, and payload sizes unevenly.

Metrics to collect

• Operations per second (ops/sec) sustained before the latency knee, this is the real usable ceiling, not the theoretical maximum.
• Latency percentiles at each load level, especially p95 and p99 (the 95th and 99th percentile response times, i.e. the slower tail of requests), since averages hide exactly the degradation that matters.
• Cost per instance-hour for each option, including any reserved or committed-use discount.

Amortized cost computation, worked example

Vertical option: one large instance at $8/hour, sustaining 10,000 ops/sec at acceptable p99 latency. Cost per request = $8 / (10,000 x 3,600) = about $0.22 per million requests.

Horizontal option: five smaller instances at $2/hour each ($10/hour total), sustaining 4,000 ops/sec each, 20,000 ops/sec combined, at similar p99 latency. Cost per request = $10 / (20,000 x 3,600) = about $0.14 per million requests.

Here horizontal is both cheaper per request and has twice the throughput ceiling. That gap can shrink or reverse for workloads that don't parallelize cleanly, where the per-node coordination and network overhead starts eating into the combined throughput as more nodes are added, the same diminishing-returns effect that Amdahl's Law describes for compute.

What tips the recommendation

• If amortized cost is within roughly 10-15% between options, I'd lean horizontal anyway for the resilience of not having a single point of failure and the ability to scale incrementally.
• If horizontal's per-request cost is clearly worse because coordination overhead dominates, I'd stay vertical up to its ceiling.
• If the growth trajectory will outlive a single vertical instance's ceiling within the planning horizon, I'd go horizontal regardless of near-term cost, since re-architecting mid-flight later is more expensive than starting distributed now.

Data sources

Historical resource utilization at fine enough granularity to see daily and weekly cycles (not just daily averages, which smooth away the actual peak), the business calendar for known seasonal events specific to this service, the product roadmap for planned launches or marketing pushes that would shift the baseline, and, because this is a stateful service, the data growth rate itself: storage size and replication lag trends, since state doesn't reset the way a stateless tier's load does.

Forecasting method

A plain trend line on raw traffic would smooth the seasonal peak away and under-forecast it. Instead I'd decompose trend and seasonality separately: take last year's comparable peak, apply this year's underlying growth rate to it. For example, if last year's seasonal peak was 40,000 requests per second (RPS) and the trend growth since then has run about 25% year over year, this year's forecast peak = 40,000 x 1.25 = 50,000 RPS. I'd cross-check that 25% growth figure against the last several weeks of baseline traffic to make sure the trend is still holding before betting the forecast on it.

Headroom policy

Target ceiling = forecast peak plus a buffer for forecast error, say 25%: 50,000 x 1.25 = 62,500 RPS. Because it's a stateful service, I'd also decide how much of that headroom needs to be pre-warmed ahead of time (cache populated, connection pools ramped, replication caught up) rather than scaled into reactively during the spike, since a stateful tier generally can't absorb a sudden load increase as fast as a stateless one can.

Cost trade-offs

Holding a 62,500 RPS-capable fleet year-round is expensive if the spike only lasts a couple of weeks a year. For example, if the non-peak fleet costs $10K/month and the full peak-ready fleet would cost $25K/month, holding it year-round costs roughly $180K/year extra, versus holding the extra capacity only for a 3-week window around the peak, which costs a small fraction of that. The middle path is usually a modestly larger always-on baseline plus a temporary reservation for the known spike window.

Testing the assumptions before the peak

I'd run a full load test at the 62,500 RPS ceiling several weeks before the actual peak window, leaving time to fix anything that fails and, if needed, procure more capacity. For a stateful service specifically, that test needs to cover cache warm-up time, replication catch-up under sustained write load, and connection pool exhaustion, since those are the failure modes that don't show up in a stateless load test but do show up here.

What I'd want to know about the workload first

• What the writes actually look like: single-row inserts versus batches, and whether the current bottleneck is disk I/O throughput, lock contention, or the overhead of committing (fsyncing) every individual write.
• Consistency requirements: can writes tolerate being asynchronous or eventually consistent, or does every write need to be immediately readable?
• Whether the data has a natural partition key (tenant ID, user ID, region) that writes could be split across, which is the precondition for horizontal scaling to even be viable here.
• Whether this is a one-time ceiling to clear or a trajectory that will keep climbing, since that changes how much re-architecture effort is worth investing now.

The decision framework

• Vertical scaling (a bigger instance, faster disks, more provisioned I/O): the fastest to implement and a reasonable choice if the headroom needed is modest and short-term, but it has a hard ceiling and doesn't fix a bottleneck that's inherently serial, like a single-writer commit log.
• Horizontal scaling (sharding writes across multiple database instances by a partition key): the sustainable choice if growth is ongoing and a good partition key exists, but it's a real re-architecture: a routing layer, and any query that spans shards gets harder.
• Changing the write pattern itself (batching writes, moving to an async queue, or a storage engine built for write-heavy workloads): often the highest-leverage option when the writes are small and frequent, because it reduces the cost per write instead of adding infrastructure.

A worked example

Say the primary is doing 4,000 writes/sec against a disk I/O ceiling of about 4,500 IOPS (input/output operations per second), with each write triggering its own commit. Batching writes into groups of 50 before committing turns 4,000 individual writes into roughly 80 batched commits/sec, which is far under the ceiling and buys significant headroom without touching hardware. That's why I'd usually check whether the write pattern itself can be improved before reaching for sharding.

What could go wrong with each approach

• Vertical: hits a cost or instance-type ceiling eventually, and the single instance stays a single point of failure the whole time.
• Horizontal: a poorly chosen partition key creates a hot shard that's just as bottlenecked as before, and operational complexity goes up.
• Batching/async: introduces a delay before a write is visible (staleness), needs idempotent retry handling (idempotent: safe to retry, because re-running the same write produces the same end result as running it once), and if the downstream queue consumer falls behind, the bottleneck just moves to the queue instead of being fixed.