Infrastructure Scaling, Capacity Planning, and High Availability Questions

Direct answer

A multi-year capacity roadmap for a global, compliance-bound SaaS provider has to answer four questions together: how much raw compute/storage/network capacity growth do we need, where geographically must that capacity physically sit (data residency), how do we guarantee the SLA (service-level agreement, the contractual uptime and performance promise) despite that geographic spread, and how do we keep procurement and cost decisions honest as the plan unfolds. The roadmap is a living document reforecast quarterly, not a one-time prediction.

Structured elaboration

Network architecture. Use regional points of presence in at least one EU region and one APAC region to satisfy data residency (the legal requirement that certain user data physically stays within a jurisdiction), each with its own compute and storage footprint, fronted by global load balancing (anycast or DNS-based routing sends each user to their nearest compliant region) so failover and residency routing are the same mechanism. Cross-region traffic (replication, control plane) rides on private interconnects rather than the public internet, both for latency and for the SLA's predictability.

Autoscaling for network functions. "Network functions" here means the network-layer components that also need to scale with load, not just application servers: load balancers, NAT gateways (translate private instance addresses to public ones), firewalls/inspection appliances, and DNS resolvers. These are commonly under-planned because engineers reflexively autoscale compute and forget the network tier has its own connection and throughput ceilings (a managed load balancer has a maximum new-connections-per-second rate; a NAT gateway has a maximum concurrent-connection count). Each of these needs its own scaling policy and its own capacity ceiling tracked in the roadmap, not an assumption that "the cloud handles it."

Circuit procurement model. For guaranteed inter-region bandwidth and the tightest SLA tiers, teams often lease dedicated network circuits (private, physically provisioned links between datacenters or into cloud provider backbones) rather than relying solely on best-effort internet paths. These have long lead times, commonly weeks to months, so the roadmap must place circuit orders AHEAD of the traffic growth that will need them, sized with headroom, and procured in diverse physical paths so a single fiber cut cannot take down the redundant link too.

Cost estimation. Roll up compute, storage, egress (data leaving the network to the internet, billed separately and often the most expensive traffic), and circuit costs per region per year of the roadmap, and separate the ALWAYS-ON baseline (steady growth) from PEAK provisioning (traffic spikes, failover capacity that mostly sits idle). Track this as a range, not a point estimate, since compounding growth assumptions are unreliable that far out (see the worked example).

Quarterly validation. Each quarter, compare actual measured growth against the forecast, re-run the capacity math with the corrected trend, and adjust circuit and reservation orders accordingly. Run small experiments (regional load tests, synthetic failover drills) to validate that the SLA actually holds under the new capacity, not just that the capacity exists on paper.

Worked example

Twenty percent annual growth compounds fast: over 3 years that is $1.2^3 \approx 1.73\times$ current load, and over 5 years $1.2^5 \approx 2.49\times$. A roadmap that only plans for a straight-line 20%-per-year addition would under-provision by year 3. Practically: forecast the near-term year (year 1) with high confidence, and widen the range for years 2 through 5 (for example, plan capacity for a 2x to 2.5x band by year 5, not a single number), re-forecasting quarterly as real data replaces the assumption.

Two related growth-planning shapes are worth walking through explicitly, since they stress-test the same roadmap differently:

Starting from a single-region monolith. Suppose the current footprint is a single-region deployment sized at 50 vCPU (virtual CPU cores) and 200 GB of memory. At the same 20%/year compounding rate, the 3-year target is roughly 50 * 1.73 ≈ 86 vCPU and 200 * 1.73 ≈ 346 GB, before any regional split. The roadmap has to decide not just "add more of the monolith" but when the growth curve crosses the point where a single region can no longer meet the SLA or the residency requirement, which is when the architecture must become genuinely distributed (the monolith becomes several regional deployments, each independently scaled).

Starting from a smaller base with an explicit multiplier and new regions. Suppose the starting point is 50,000 monthly active users (MAU) in one region, and the target is 5x that user base plus two new regions (matching the EU/APAC residency requirement). That is 250,000 MAU total; split evenly across three regions (the original plus two new ones) as a starting assumption gives roughly 83,000 MAU per region, though the real split should follow where those users actually are, not an even division. Each new region needs its OWN capacity roadmap, its own circuit procurement lead time, and its own SLA validation, it is not simply "divide the existing capacity by three."

Trade-offs & pitfalls

• The single biggest failure mode is treating a compounding growth rate as gospel for the full multi-year horizon; build in quarterly re-forecasting and a widening confidence band instead of a single locked number.
• Circuit procurement lead time is the easiest thing to forget until it becomes the actual bottleneck: compute can be added in minutes, a leased circuit cannot.
• Data residency is not just "which region hosts the data," it also constrains where backups, logs, and cross-region replication can flow, which affects the DR (disaster recovery, the plan for restoring service after a major outage) design, not just steady-state routing.
• A roadmap with only ONE cost number invites false confidence; present a range tied explicitly to the growth-assumption range.

Baseline math (everything kept in requests per second, RPS)

Current: 50,000 RPS at 60% CPU
Capacity at 100% CPU (today's fleet) = 50,000 / 0.60 = 83,333 RPS

Utilization at 60 percent while serving 50,000 RPS tells us the fleet's real ceiling today is about 83,333 RPS if it ran flat out, which we never actually want to do.

Baseline in 6 months with 30% growth = 50,000 x 1.30 = 65,000 RPS
Capacity needed to hold 60% target utilization = 65,000 / 0.60 = 108,333 RPS

Holding the same 60 percent target utilization at the grown baseline needs about 108,333 RPS of capacity, roughly 30 percent more fleet than today, which makes sense: if the target utilization does not change, fleet size scales linearly with traffic.

Flash peak on top of the grown baseline = 65,000 x 5 = 325,000 RPS
Capacity to hold 60% utilization at that peak = 325,000 / 0.60 = 541,667 RPS

That is about 6.5x today's compute footprint (541,667 / 83,333). Holding that permanently, year-round, for an "occasional" spike is not economical, so the plan treats it as an elastic ceiling, not a standing fleet.

Horizontal vs vertical scaling

Horizontal (more stateless instances) is the primary lever: it scales close to linearly and improves fault tolerance. Vertical scaling (bigger instances) is reserved for anything that cannot be sharded easily, such as a single-writer database, and even there it is a stopgap since it has a hard ceiling and usually needs a resize/restart.

Autoscaling policy and buffer sizing

Run target-tracking autoscaling around the same 60 percent utilization target the fleet runs at today, with the maximum instance count set high enough to cover the 5x flash case (the 541,667 RPS figure above). If flash events are known in advance (an announced sale), add scheduled pre-warming ahead of the event, because reactive-only autoscaling cannot launch, boot, and warm 6.5x the fleet inside a sudden spike without some queueing and latency degradation first.

Peak concurrency modeling

Use Little's Law (a queueing-theory result: average concurrent requests in flight equals throughput multiplied by average request latency) to size not just instance counts but connection pools and downstream (database, cache) query rates at each of the three traffic levels above, since the DB or cache can become the real ceiling even when the app tier autoscales fine.

Validating with load testing

Run a ramp test to the 6-month baseline (65,000 RPS) to confirm steady-state headroom holds, and a separate spike test that ramps from baseline to the 325,000 RPS flash level over a realistic ramp time to confirm the autoscaler (or pre-warmed buffer) keeps p95 latency inside SLO during the spike, not just after it settles.

Presenting cost vs headroom to stakeholders

Frame it as a curve, not a single number: holding permanent capacity for the 5x case year-round costs roughly 6.5x today's compute bill for a scenario that only happens occasionally, versus an autoscaling-plus-pre-warm approach that costs close to the 30 percent growth-driven increase most of the time, with a stated, honestly communicated risk window (the minutes it takes to scale out) for any flash event that arrives without advance notice.

Why deterministic single-service forecasting is not enough here

At org scale, with high business-metric uncertainty, a single point-forecast per service (this year's traffic times a growth factor) hides two real problems: individual service forecasts have wide, correlated error bars, and the org needs one coherent capacity story across all of them, not fifty independent guesses that may not even be internally consistent with each other.

Inputs

• Historical utilization and growth trend per service, at whatever granularity capacity decisions are actually made (per-cluster, per-region).
• Business inputs with real uncertainty attached: revenue/user growth targets, planned feature launches, marketing calendar, sales pipeline for large accounts that could cause step-changes.
• Known one-time events (planned migrations, deprecations, a scheduled traffic-driving launch) that a purely statistical model would miss entirely.

Scenario planning

Build best, base, and worst case scenarios, not as arbitrary percentage bands but grounded in specific, named assumptions (base case: current growth trend continues; best case: the enterprise deal pipeline closes as forecast; worst case: growth flattens as it did during a prior slow quarter). Naming the assumption behind each scenario is what makes it possible to tell, months later, which scenario is actually playing out.

Probabilistic techniques: Monte Carlo

For services where multiple uncertain inputs combine (growth rate, feature-launch timing, seasonal multiplier), run Monte Carlo simulation: repeatedly sample each uncertain input from its own distribution, propagate through to a resulting capacity need, and look at the resulting distribution of outcomes rather than a single number. This surfaces the actual shape of the risk, for instance whether the tail risk (99th percentile need) is close to the median or dramatically higher, which a single best/base/worst case with fixed percentages cannot show.

SLA-driven capacity choices

Different services carry different SLA (Service Level Agreement, a committed uptime or performance target, often with a contractual consequence for missing it) commitments (99.9 percent vs 99.99 percent uptime, or different latency targets), and higher SLA tiers need to be sized against a more conservative percentile of the demand distribution (say, the 95th or 99th percentile forecast, not the median) with correspondingly more headroom. This ties the probabilistic forecast directly to a concrete provisioning decision instead of leaving "how much headroom" as a separate, disconnected judgment call.

Stakeholder reporting

Executives need the range and the named scenario assumptions, not the simulation mechanics: a short one-page summary per major capacity area (compute, storage, network) with the base-case number, the range, and the specific triggers that would signal a scenario shift. Engineering leadership needs more detail: per-service breakdowns, procurement lead-time implications, and where the roadmap is most sensitive to a wrong assumption.

Update cadence

Quarterly for the org-wide roadmap as a default, but with explicit guardrails (a defined percentage deviation from base-case forecast, sustained over more than one reporting period) that trigger an off-cycle update rather than waiting for the next quarter, since a five-year, org-wide plan that only updates on a fixed calendar will lag real signal by a quarter or more exactly when it matters most.

Direct answer

With per-unit rates given (vCPU, memory, storage, network egress), estimating monthly cost when scaling from 100 to 10,000 concurrent connections is mostly a matter of stating clean per-connection resource assumptions, applying one consistent redundancy/headroom factor across the compute footprint, and then multiplying every component by hours or bytes per month using the same basis throughout.

Structured elaboration

Assumptions needed. Per-connection vCPU (virtual CPU core) and memory usage, since the given unit costs are priced per vCPU-hour and per GB-hour; per-connection network egress rate (network egress: data sent out to the internet, billed per GB and separate from inbound traffic), since egress is priced per GB transferred; and, separately, average session duration, which is less about the monthly cost formula itself (concurrency is already given directly) and more about validating that the concurrency number is internally consistent (total daily connections roughly equals concurrent connections divided by average session length, times seconds in a day), and about planning for connection churn during autoscaling events (draining long-lived sessions gracefully takes longer when average session duration is high).

Redundancy and autoscaling headroom. Combine both into one explicit capacity multiplier rather than two separate vague buffers, for example, targeting 70% average utilization implies provisioning $\frac{1}{0.7} \approx 1.43\times$ the bare compute footprint, covering both redundancy (spare capacity if a node fails) and autoscaling headroom (room to absorb a burst before new capacity comes online) in one number.

Applying the given unit rates. Convert hourly rates to monthly using a standard 730 hours/month basis (365 days times 24 hours, divided by 12 months), and apply the redundancy multiplier to compute and memory (the elastic, per-connection resources) but not to storage (typically durable, replicated separately rather than scaled with a utilization-headroom factor) or egress (a pure pass-through cost proportional to actual traffic, not capacity).

Worked example

Illustrative per-connection assumptions (state and validate your own before using this for a real budget): 0.02 vCPU, 5 MB memory, 1 MB/month of stored session/audit data, and a steady 2 KB/sec of egress per active connection.

At N=100 connections, with the 1.43x redundancy/headroom factor on compute and memory: roughly 2.9 vCPU and 0.7 GB memory provisioned, plus 0.1 GB storage and about 526 GB/month of egress (2 KB/sec sustained per connection adds up over a full month). Monthly cost: compute about $42, memory about $5, storage under $0.01, egress about $47, totaling roughly $94/month.

At N=10,000, everything scales by the same 100x factor: roughly 286 vCPU and 71.5 GB memory provisioned, 10 GB storage, and about 52,560 GB/month of egress. Monthly cost: compute about $4,176, memory about $522, storage under $1, egress about $4,730, totaling roughly $9,429/month, almost exactly 100x the 100-connection cost, since every cost component in this model is linear in connection count and the redundancy factor is held constant at both scales.

Trade-offs & pitfalls

• This simple model shows cost scaling perfectly linearly with connection count (100x connections, 100x cost), which is itself worth calling out as a LIMITATION, not a finding: a real environment usually shows cost growing more slowly than connections at higher scale, because fixed costs (load balancer, control plane, a fixed minimum redundant baseline) get amortized across more connections, and because larger commitments often unlock volume discounts this linear per-unit-rate model does not capture.
• Egress dominates the bill in this example (roughly half of total cost at both scales), a common surprise for teams that budget mainly for compute; validate the per-connection egress assumption carefully, since it is doing more work in the total than either compute or memory.
• Applying the redundancy/headroom factor to storage or egress the same way it is applied to compute would overstate those costs, since they don't work the same way (storage is usually already durably replicated by the storage layer itself, and egress is billed on actual bytes moved, not on provisioned capacity); keep the multiplier scoped to what it actually represents.

Start by questioning the growth assumption itself

Before building a roadmap, check whether 20% month-over-month compounding for three more years is actually plausible, because the arithmetic gets extreme fast:

A 709x traffic multiplier in three years is very rarely realistic; sustained hyper-growth phases almost always decelerate. Flagging this to stakeholders, rather than quietly building a roadmap on it, is itself part of the deliverable: a good capacity plan says explicitly where its input assumption is likely to break.

Converting traffic projections into RPS and instance counts

Worked example with a baseline of 100 RPS and one instance sustaining 20 RPS at target CPU:

import math

def instances_needed(month, baseline_rps=100, growth=0.20, rps_per_instance=20):
    projected_rps = baseline_rps * (1 + growth) ** month
    return projected_rps, math.ceil(projected_rps / rps_per_instance)

for m in [0, 6, 12]:
    rps, n = instances_needed(m)
    print(f"month {m}: {rps:.0f} RPS -> {n} instances")

for m in [6, 12, 36]:
    print(f"1.20^{m} = {1.20**m:.1f}x")

month 0: 100 RPS -> 5 instances
month 6: 299 RPS -> 15 instances
month 12: 892 RPS -> 45 instances
1.20^6 = 3.0x
1.20^12 = 8.9x
1.20^36 = 708.8x

Handling the long-tail years honestly

Rather than extending 20% MoM compounding through year 3, split the roadmap into two regimes: a high-confidence near-term window (roughly the next 2 to 3 quarters) where the historical trend is trusted directly, and a decelerating-growth range for years 2 to 3, modeled as a range of scenarios (e.g., growth tapering toward a more sustainable 5 to 10% MoM, or toward flat) rather than one number. Re-forecast quarterly against actuals instead of committing the full three years up front.

Discrete step events, kept separate from the smooth trend

A known event like a major feature launch expected to add a step change in load (say, roughly 50% more traffic in a specific quarter) should be added as a discrete addition on top of the smooth compounding curve for that quarter, not blended into the growth-rate assumption. Baking it into the rate would silently imply that step keeps compounding every month after, which it doesn't, it's a one-time shift.

Safety margins

Carry explicit headroom above whatever the current forecast says, commonly 20 to 30%, sized larger in the near term (where you're committing capital) and can be a wider planning range further out (where you're mostly setting expectations, not buying hardware yet).

Procurement lead times

If reserved capacity, hardware, or vendor commitments take, say, 2 to 3 months to arrive, trigger the order when the forecast crosses (target_capacity - lead_time_worth_of_growth), not when you actually hit the ceiling. This means the roadmap's real output isn't just "how many instances by when", it's "the calendar date each procurement action needs to be triggered."

Assumptions to document for finance and execs

State explicitly: the growth-rate assumption used for the near-term window and the deceleration assumption used beyond it, that per-request CPU/memory usage is held constant (no efficiency work assumed, flag separately if any is planned), the peak:average ratio used to translate RPS into provisioned instances, and any known step events (like the feature launch above) called out as separate line items rather than folded into the trend.