Scalability & Capacity Planning Questions

Data sources

Historical resource utilization at fine enough granularity to see daily and weekly cycles (not just daily averages, which smooth away the actual peak), the business calendar for known seasonal events specific to this service, the product roadmap for planned launches or marketing pushes that would shift the baseline, and, because this is a stateful service, the data growth rate itself: storage size and replication lag trends, since state doesn't reset the way a stateless tier's load does.

Forecasting method

A plain trend line on raw traffic would smooth the seasonal peak away and under-forecast it. Instead I'd decompose trend and seasonality separately: take last year's comparable peak, apply this year's underlying growth rate to it. For example, if last year's seasonal peak was 40,000 requests per second (RPS) and the trend growth since then has run about 25% year over year, this year's forecast peak = 40,000 x 1.25 = 50,000 RPS. I'd cross-check that 25% growth figure against the last several weeks of baseline traffic to make sure the trend is still holding before betting the forecast on it.

Headroom policy

Target ceiling = forecast peak plus a buffer for forecast error, say 25%: 50,000 x 1.25 = 62,500 RPS. Because it's a stateful service, I'd also decide how much of that headroom needs to be pre-warmed ahead of time (cache populated, connection pools ramped, replication caught up) rather than scaled into reactively during the spike, since a stateful tier generally can't absorb a sudden load increase as fast as a stateless one can.

Cost trade-offs

Holding a 62,500 RPS-capable fleet year-round is expensive if the spike only lasts a couple of weeks a year. For example, if the non-peak fleet costs $10K/month and the full peak-ready fleet would cost $25K/month, holding it year-round costs roughly $180K/year extra, versus holding the extra capacity only for a 3-week window around the peak, which costs a small fraction of that. The middle path is usually a modestly larger always-on baseline plus a temporary reservation for the known spike window.

Testing the assumptions before the peak

I'd run a full load test at the 62,500 RPS ceiling several weeks before the actual peak window, leaving time to fix anything that fails and, if needed, procure more capacity. For a stateful service specifically, that test needs to cover cache warm-up time, replication catch-up under sustained write load, and connection pool exhaustion, since those are the failure modes that don't show up in a stateless load test but do show up here.

What I'd want to know first

Before I put a number on anything, I'd pull four things: the traffic profile (is it steady, bursty, or spiky through the day), the read/write mix (reads and writes scale very differently), the resource cost per request (how much CPU, memory, and database load one typical request consumes), and the existing SLA (service level agreement) or error budget (how much latency and error headroom I'm actually allowed to spend). Skipping this step and jumping straight to "just double everything" is the most common mistake, because growth is rarely uniform across every resource.

Headroom calculation

Say the service peaks at 3,000 requests per second (RPS) today, running on 20 instances. Load testing shows each instance holds a safe ceiling of about 175 RPS before p99 latency (the 99th percentile response time, the slowest 1 in 100 requests) starts climbing. That puts the fleet's tested ceiling at 20 x 175 = 3,500 RPS, so today's headroom is about 500 RPS, roughly 14% of the tested ceiling (500 of 3,500 RPS still unused).

If traffic doubles in 6 months, peak becomes 6,000 RPS. I'd size for that peak plus a buffer for forecast error, say 20%, giving a target ceiling of 7,200 RPS. At 175 RPS per instance that's 7,200 / 175 = about 42 instances, roughly doubling the fleet plus a bit more because the buffer also grows.

Scaling strategy

For a stateless web tier fronting a database, horizontal scaling (adding more instances of the same size) is usually the right default: no single-instance ceiling to hit, and it's cheaper to add commodity capacity than to keep buying bigger boxes. But I'd check the database write path separately, since it often becomes the actual constraint before the web tier does. If the database is the binding resource, the plan has to include read replicas, connection pooling limits, or a write-path change, not just more web servers.

What I'd benchmark

• A steady-state load test at the projected 6,000 RPS peak with the real production read/write mix, not a synthetic uniform load.
• A soak test (sustained load over hours) to catch memory leaks and connection pool exhaustion that only show up over time.
• A burst test that ramps quickly to the new ceiling, since sustained-load tests can hide how the system behaves under a sudden spike.

Cost versus performance

If each instance costs roughly $140/month, the fleet grows from $2,800/month to about $5,880/month, a cost increase of around 110% to support a 100% traffic increase, which is close to linear and reasonable. If the math came out sharply super-linear (say 300% cost for 100% traffic), that would push me to look at right-sizing the instance type or fixing an inefficiency instead of just buying more of the same box.

The three options and what drives the choice

• Standby headroom held ready for the hour: right when the time needed to scale into the spike is longer than the ramp-up time available, for example if there's meaningful warm-up work (cache population, connection pool ramp) or the spike arrives as a near-instant step rather than a gradual ramp.
• Dynamic scale-into-it: right when the workload is fast to provision and stateless enough that elastic scaling can track the ramp in time, say if the spike builds over 10-15 minutes rather than arriving instantly; this is the cheapest option since capacity is only paid for while it's actually needed.
• Reshaping the workload: right when a meaningful share of the spike isn't actually latency-sensitive, for example report generation or non-urgent notification sends that could be deferred to run outside the peak hour instead. This shrinks the size of the problem itself rather than just paying to meet it, which usually makes it the highest-leverage option when it's available.

The right sequence is to profile what's actually inside that daily hour's traffic first. If a chunk of it is deferrable, reshape that part out before deciding between standby and dynamic for what's left.

A worked example

Baseline is 2,000 requests per second (RPS); the spike hour hits 20,000 RPS (10x). Say profiling shows 6,000 RPS of that spike is deferrable batch and reporting traffic. Reshaping (queueing it to run off-peak) cuts the peak that has to be served live from 20,000 to 14,000 RPS, a 30% reduction in what needs to be provisioned for, before even choosing standby versus dynamic for the remaining 14,000 RPS.

Cost control for the other 23 hours

Because this spike is daily and at a predictable time, unlike a one-off event or an unpredictable surge, I'd favor scheduling the elevated capacity to the known clock time rather than holding it around the clock: provision it for roughly the spike hour plus a warm-up and cooldown buffer, say 90 minutes total, and run at baseline the other 22.5 hours. Worked example: if the baseline fleet costs $6K/month and the surge capacity addition would cost $18K/month if held 24/7, scheduling it for 1.5 hours a day instead cuts that addition to roughly $18K x (1.5/24), about $1,125/month, a large saving as long as the daily timing is reliable enough to schedule against confidently.

From SLOs to a headroom target

A service level objective (SLO) of 99.99% availability implies an error budget, the amount of allowed failure before the SLO itself is breached:

Allowed downtime = (1 - 0.9999) x 30 days x 24 hours x 60 minutes

That works out to about 4.3 minutes per month. Intuition: four nines of availability doesn't mean zero room for error, it means a very small, explicit, spendable budget of failure per month.

For the p99 latency target (the 99th percentile response time must stay under 200ms), the key idea is that latency doesn't rise smoothly with load, it stays flat and then climbs sharply past some utilization point, the "knee." The capacity plan has to keep peak load safely below that knee, not riding right at it.

Determining headroom

Say load testing shows the knee, where p99 crosses 200ms, sits at about 75% of the fleet's tested ceiling. Operating right at 75% is fragile: any small load variance breaches the SLO immediately. So I'd target sustained peak utilization well below that, say 55-60% of ceiling, treating the 75% point as a true emergency reserve rather than the normal operating target.

Using the error budget to know when it's too close

I'd track how much of the monthly error budget gets consumed specifically by capacity-related incidents (isolated from other causes like deploy bugs), and watch the burn rate, not just the total consumed so far. Worked example: 10 days into a 30-day month, capacity-linked errors have already burned 2.5 of the 4.3-minute budget. At that rate, the full budget would be exhausted by around day 17, well before month end. That trend, not the raw "2.5 of 4.3 minutes used" snapshot, is the actual trigger to add headroom immediately.

Communicating to stakeholders who only care that the SLO is met

I'd skip the RPS and CPU numbers and lead with SLO risk directly: "at current headroom, we estimate an elevated chance of breaching the 99.99% target under this quarter's expected peak growth; closing that gap costs about $X per month." I'd frame under-provisioning explicitly as a decision to accept a stated increase in SLO-breach risk, since stakeholders who only care that the SLO holds need to see that not funding headroom is itself a choice with a cost, just a hidden one.

Headroom sizing

Say baseline traffic is 2,000 requests per second (RPS); 10x gives a forecast peak of 20,000 RPS. Because this is a one-off event with limited prior data for this specific sale, I'd add a larger buffer than I would for a recurring, well-understood pattern, say 30% for forecast uncertainty, putting the target ceiling at about 26,000 RPS.

Hold-ready versus scale-into-it

For a 72-hour hard deadline with no room to react mid-event, I'd hold the full 26,000 RPS capacity ready in advance rather than plan to scale into it reactively. The reasons: there's essentially zero time to safely validate new capacity once the sale is live, and retail checkout paths usually have state that needs warming, caches, database connection pools, CDN edge caches, which can't be spun up instantly under load. This is different from a slow, sustained growth scenario, where reactive scaling is fine because there's time to react.

Cost versus risk

Holding the extra 26,000 RPS-capable capacity ready for about a week around the event, covering setup, the sale itself, and teardown, might cost on the order of a few thousand dollars in extra infrastructure spend. Against that, a marquee 72-hour sale falling over mid-event risks a large multiple of that in lost revenue and reputational damage, on a fixed, unrepeatable window (unlike ongoing growth, where over-provisioning forever really would be wasteful). The asymmetry between a small, bounded extra cost and a large, irrecoverable failure cost is what justifies over-provisioning for this specific case.

Three measurable commitments I'd propose to the client

1. A latency commitment: for example, checkout p99 latency (the slowest 1% of checkout requests) stays under 500ms up to 22,000 RPS, comfortably inside the tested 26,000 RPS ceiling.
2. An error-rate commitment: error rate stays under 0.1% for the full 72-hour window.
3. A headroom commitment as a business metric: at least 15% spare capacity above observed peak traffic is maintained throughout the event, with an internal alert if it drops below 10%, so there's a documented safety margin the client can see, not just a promise.