Scalability & Capacity Planning Questions

Benchmark plan

I'd run three kinds of workloads: a steady-state throughput test to find each option's ceiling, a latency-under-load test at several utilization levels (say 50%, 70%, 90%) to find the point where latency starts climbing sharply (the "knee"), and a realistic production-mix workload rather than a synthetic uniform one, since real traffic mixes reads, writes, and payload sizes unevenly.

Metrics to collect

• Operations per second (ops/sec) sustained before the latency knee, this is the real usable ceiling, not the theoretical maximum.
• Latency percentiles at each load level, especially p95 and p99 (the 95th and 99th percentile response times, i.e. the slower tail of requests), since averages hide exactly the degradation that matters.
• Cost per instance-hour for each option, including any reserved or committed-use discount.

Amortized cost computation, worked example

Vertical option: one large instance at $8/hour, sustaining 10,000 ops/sec at acceptable p99 latency. Cost per request = $8 / (10,000 x 3,600) = about $0.22 per million requests.

Horizontal option: five smaller instances at $2/hour each ($10/hour total), sustaining 4,000 ops/sec each, 20,000 ops/sec combined, at similar p99 latency. Cost per request = $10 / (20,000 x 3,600) = about $0.14 per million requests.

Here horizontal is both cheaper per request and has twice the throughput ceiling. That gap can shrink or reverse for workloads that don't parallelize cleanly, where the per-node coordination and network overhead starts eating into the combined throughput as more nodes are added, the same diminishing-returns effect that Amdahl's Law describes for compute.

What tips the recommendation

• If amortized cost is within roughly 10-15% between options, I'd lean horizontal anyway for the resilience of not having a single point of failure and the ability to scale incrementally.
• If horizontal's per-request cost is clearly worse because coordination overhead dominates, I'd stay vertical up to its ceiling.
• If the growth trajectory will outlive a single vertical instance's ceiling within the planning horizon, I'd go horizontal regardless of near-term cost, since re-architecting mid-flight later is more expensive than starting distributed now.

Confirming the network really is the binding constraint

I wouldn't take the symptom (link saturation, rising query latency) at face value without ruling out other causes first. I'd check network interface utilization on both the app server and database side during the exact peak window, and correlate its onset with the onset of the latency increase, while also checking CPU and disk I/O on both tiers over the same window. For example: NIC (network interface) throughput at 950 Mbps sustained out of a 1 Gbps link (95% utilized), with TCP retransmissions (packets being resent because they weren't acknowledged in time, a sign the network link is congested) rising right when p99 latency (the 99th percentile response time, the slowest 1% of requests) crosses 300ms, while app server CPU sits at 40% and database CPU at 55%, both well under their own ceilings. That combination, network metric pegged and rising retransmits at the same moment latency spikes, while compute stays comfortable, is what actually confirms the network link as the binding constraint rather than assuming it from the topology alone.

Buying headroom quickly

Without touching the architecture, I'd reduce the bytes crossing that link: stop over-fetching (select only the columns actually needed instead of every column), compress payloads, batch queries to cut per-query overhead, and move some read traffic to a local cache or replica so it never has to cross the saturated link at all. These changes can ship in days and buy real headroom while a longer-term fix is planned.

The longer-term architectural fix

The quick fixes reduce load on the shared link, but the underlying issue is that traffic growth keeps colliding with a fixed-bandwidth path between two tiers that are architecturally coupled. Longer term I'd look at moving the app and database tiers onto a higher-bandwidth or lower-latency path (a dedicated link, tighter physical or network placement to cut hop count), and reducing chattiness structurally, colocating read replicas closer to the app servers, or adding a caching layer so most reads never round-trip to the database at all.

What happens next

Once the network stops being the ceiling, growth will eventually push the next resource, likely database CPU or the app tier itself, into becoming the new binding constraint. This isn't a one-time fix, it's the same identify-the-constraint exercise that needs to run again at the next growth checkpoint.

What I'd want to know first

Before I put a number on anything, I'd pull four things: the traffic profile (is it steady, bursty, or spiky through the day), the read/write mix (reads and writes scale very differently), the resource cost per request (how much CPU, memory, and database load one typical request consumes), and the existing SLA (service level agreement) or error budget (how much latency and error headroom I'm actually allowed to spend). Skipping this step and jumping straight to "just double everything" is the most common mistake, because growth is rarely uniform across every resource.

Headroom calculation

Say the service peaks at 3,000 requests per second (RPS) today, running on 20 instances. Load testing shows each instance holds a safe ceiling of about 175 RPS before p99 latency (the 99th percentile response time, the slowest 1 in 100 requests) starts climbing. That puts the fleet's tested ceiling at 20 x 175 = 3,500 RPS, so today's headroom is about 500 RPS, roughly 14% of the tested ceiling (500 of 3,500 RPS still unused).

If traffic doubles in 6 months, peak becomes 6,000 RPS. I'd size for that peak plus a buffer for forecast error, say 20%, giving a target ceiling of 7,200 RPS. At 175 RPS per instance that's 7,200 / 175 = about 42 instances, roughly doubling the fleet plus a bit more because the buffer also grows.

Scaling strategy

For a stateless web tier fronting a database, horizontal scaling (adding more instances of the same size) is usually the right default: no single-instance ceiling to hit, and it's cheaper to add commodity capacity than to keep buying bigger boxes. But I'd check the database write path separately, since it often becomes the actual constraint before the web tier does. If the database is the binding resource, the plan has to include read replicas, connection pooling limits, or a write-path change, not just more web servers.

What I'd benchmark

• A steady-state load test at the projected 6,000 RPS peak with the real production read/write mix, not a synthetic uniform load.
• A soak test (sustained load over hours) to catch memory leaks and connection pool exhaustion that only show up over time.
• A burst test that ramps quickly to the new ceiling, since sustained-load tests can hide how the system behaves under a sudden spike.

Cost versus performance

If each instance costs roughly $140/month, the fleet grows from $2,800/month to about $5,880/month, a cost increase of around 110% to support a 100% traffic increase, which is close to linear and reasonable. If the math came out sharply super-linear (say 300% cost for 100% traffic), that would push me to look at right-sizing the instance type or fixing an inefficiency instead of just buying more of the same box.

Headroom sizing

Say baseline traffic is 2,000 requests per second (RPS); 10x gives a forecast peak of 20,000 RPS. Because this is a one-off event with limited prior data for this specific sale, I'd add a larger buffer than I would for a recurring, well-understood pattern, say 30% for forecast uncertainty, putting the target ceiling at about 26,000 RPS.

Hold-ready versus scale-into-it

For a 72-hour hard deadline with no room to react mid-event, I'd hold the full 26,000 RPS capacity ready in advance rather than plan to scale into it reactively. The reasons: there's essentially zero time to safely validate new capacity once the sale is live, and retail checkout paths usually have state that needs warming, caches, database connection pools, CDN edge caches, which can't be spun up instantly under load. This is different from a slow, sustained growth scenario, where reactive scaling is fine because there's time to react.

Cost versus risk

Holding the extra 26,000 RPS-capable capacity ready for about a week around the event, covering setup, the sale itself, and teardown, might cost on the order of a few thousand dollars in extra infrastructure spend. Against that, a marquee 72-hour sale falling over mid-event risks a large multiple of that in lost revenue and reputational damage, on a fixed, unrepeatable window (unlike ongoing growth, where over-provisioning forever really would be wasteful). The asymmetry between a small, bounded extra cost and a large, irrecoverable failure cost is what justifies over-provisioning for this specific case.

Three measurable commitments I'd propose to the client

1. A latency commitment: for example, checkout p99 latency (the slowest 1% of checkout requests) stays under 500ms up to 22,000 RPS, comfortably inside the tested 26,000 RPS ceiling.
2. An error-rate commitment: error rate stays under 0.1% for the full 72-hour window.
3. A headroom commitment as a business metric: at least 15% spare capacity above observed peak traffic is maintained throughout the event, with an internal alert if it drops below 10%, so there's a documented safety margin the client can see, not just a promise.

Data sources

Historical resource utilization at fine enough granularity to see daily and weekly cycles (not just daily averages, which smooth away the actual peak), the business calendar for known seasonal events specific to this service, the product roadmap for planned launches or marketing pushes that would shift the baseline, and, because this is a stateful service, the data growth rate itself: storage size and replication lag trends, since state doesn't reset the way a stateless tier's load does.

Forecasting method

A plain trend line on raw traffic would smooth the seasonal peak away and under-forecast it. Instead I'd decompose trend and seasonality separately: take last year's comparable peak, apply this year's underlying growth rate to it. For example, if last year's seasonal peak was 40,000 requests per second (RPS) and the trend growth since then has run about 25% year over year, this year's forecast peak = 40,000 x 1.25 = 50,000 RPS. I'd cross-check that 25% growth figure against the last several weeks of baseline traffic to make sure the trend is still holding before betting the forecast on it.

Headroom policy

Target ceiling = forecast peak plus a buffer for forecast error, say 25%: 50,000 x 1.25 = 62,500 RPS. Because it's a stateful service, I'd also decide how much of that headroom needs to be pre-warmed ahead of time (cache populated, connection pools ramped, replication caught up) rather than scaled into reactively during the spike, since a stateful tier generally can't absorb a sudden load increase as fast as a stateless one can.

Cost trade-offs

Holding a 62,500 RPS-capable fleet year-round is expensive if the spike only lasts a couple of weeks a year. For example, if the non-peak fleet costs $10K/month and the full peak-ready fleet would cost $25K/month, holding it year-round costs roughly $180K/year extra, versus holding the extra capacity only for a 3-week window around the peak, which costs a small fraction of that. The middle path is usually a modestly larger always-on baseline plus a temporary reservation for the known spike window.

Testing the assumptions before the peak

I'd run a full load test at the 62,500 RPS ceiling several weeks before the actual peak window, leaving time to fix anything that fails and, if needed, procure more capacity. For a stateful service specifically, that test needs to cover cache warm-up time, replication catch-up under sustained write load, and connection pool exhaustion, since those are the failure modes that don't show up in a stateless load test but do show up here.