Infrastructure Scaling, Capacity Planning, and High Availability Questions

How the topology changes network capacity planning

Within a single AZ, inter-service network hops are effectively free to plan for: bandwidth is high, latency is low and stable, and cloud providers typically do not charge for traffic that stays within the same AZ. The moment a service spreads across AZs or regions, network capacity planning has to account for three things that were not there before: added latency, the bandwidth the replication/traffic itself needs, and the transfer fees the provider charges for that traffic.

Latency

Cross-AZ traffic within a region adds a small, fairly predictable latency (commonly single-digit milliseconds), usually fine to absorb into a synchronous call path. Cross-region traffic adds latency driven by physical distance (commonly tens to well over a hundred milliseconds), generally too slow for a synchronous request path, so it has to be planned for as asynchronous replication instead. That changes the bandwidth math: you are now sizing for a steady replication stream, not a single request/response pair.

Inter-region bandwidth and replication costs

Size cross-region bandwidth for the actual replication volume, not the request volume: for a database, that is write throughput times average write size times the number of regions receiving a copy; for event/log shipping, it is the raw event volume. Build in headroom above the steady-state rate, because if a target region falls behind (a network blip, a slow follower), it needs to catch up faster than steady-state without permanently falling further behind, which needs spare bandwidth on top of the baseline.

Factoring transfer fees into sizing

Cloud providers typically charge nothing, or very little, for traffic within an AZ, a small per-GB fee for cross-AZ traffic within a region, and a meaningfully larger per-GB fee for cross-region traffic. When sizing, that means: minimize what actually needs to leave the region (only replicate what genuinely needs a remote copy, not everything by default), track the recurring transfer cost as an ongoing line item that scales with traffic growth, not a one-time setup cost, and weigh it in the same trade-off as compute cost when deciding how many regions to run in and how much to replicate to each.

Direct answer

Shard by a geography-aware key so each user's data has one home region for writes, route reads to the nearest replica and writes to the shard's home region, and keep enough spare capacity in every region to absorb one region's traffic if it goes down, since that failure mode is what a sub-50ms design actually has to survive.

Structured elaboration

Data placement: give every shard a home (primary) region based on where its data is written most, with replicas in the other two regions. It helps to separate two distinct user populations rather than treating all traffic as one class: continent-scoped ACTIVE users, whose writes and low-latency reads must stay within their home continent, and a separate GLOBAL read-only population (dashboards, admin and support tooling) that can tolerate cross-region reads and does not need the sub-50ms guarantee. Sizing the whole fleet for the tightest requirement when only part of the traffic needs it wastes capacity.

Routing: writes go to the shard's home region only, so there is a single writer per shard and no cross-region write conflicts to resolve. Reads go to the nearest replica of that user's shard, which may lag slightly behind the primary.

Limiting cross-region latency: nothing on the hot read or write path makes a synchronous cross-region call; only the asynchronous replication stream crosses regions.

Consistency: replicas are eventually consistent. Define and expose a staleness bound (a target replication lag and an alert threshold) rather than implying real-time consistency a sub-50ms budget cannot support, and route a user back to the home region for a short window right after their own write so they never see a version older than what they just wrote.

Failover and rebalancing: when a region fails, its shards' home role fails over to whichever surviving replica has the least replication lag (tracked as a first-class metric, not assumed), and a rebalancing job then redistributes some of the newly promoted shards across the remaining regions gradually, rather than concentrating all of the failed region's load onto whichever single region absorbed it first.

Worked example

Because any one of three regions can fail and its load must land on the remaining two:

For n = 3, that works out to 1/2, meaning each surviving region needs 50% extra headroom above its own normal peak to absorb a failed region's traffic without breaching the latency target during the failover window.

Trade-offs and pitfalls

Sizing every region only for its own normal peak, with no failover headroom, means a single region loss cascades into overloading the survivors exactly when the system needs to hold up.

Promoting a replica without tracking per-replica lag risks promoting a stale one and silently losing recent writes; make lag an explicit, monitored number, not an assumption.

Rebalancing after a heal should move shards gradually with rate limiting; moving everything back at once risks a second, self-inflicted overload during recovery.

Approach

Model each historic job type's expected concurrent CPU demand as (average CPU-seconds consumed per run / average wall-clock runtime) x (average number of concurrent runs). The first factor is how many vCPUs (virtual CPUs) one running instance of that job consumes on average while it runs; a job that used 90 CPU-seconds over 120 seconds of wall time averaged 0.75 vCPU, for example because part of it was I/O-bound rather than fully single-threaded. Multiplying by how many copies typically run at once gives that job type's contribution to total concurrent demand. Summing across job types gives total expected concurrent demand, and dividing by a target utilization (headroom) turns that into a recommended cluster size, so the cluster is not run right at the edge of saturation.

import csv
import io
import math

# Sample historic batch-job data (columns match the question's schema).
# In production this would be `open("job_history.csv")`; embedded here so
# the script runs standalone with no external file.
CSV_DATA = """job_id,avg_runtime_seconds,avg_cpu_seconds,avg_concurrent_runs
etl_ingest,120,90,4
report_gen,300,450,2
ml_scoring,60,48,10
"""

TARGET_UTILIZATION = 0.7  # keep the cluster at <=70% busy so queueing delay
                           # stays bounded instead of growing sharply near 100%


def recommend_cluster_vcpus(csv_text: str, target_utilization: float = TARGET_UTILIZATION):
    reader = csv.DictReader(io.StringIO(csv_text))
    rows = []
    total_demand_vcpus = 0.0

    for row in reader:
        runtime_s = float(row["avg_runtime_seconds"])
        cpu_s = float(row["avg_cpu_seconds"])
        concurrent_runs = float(row["avg_concurrent_runs"])

        # vCPUs one running instance of this job consumes on average while it runs
        concurrency_factor = cpu_s / runtime_s
        # expected concurrent vCPU demand contributed by this job type
        job_demand_vcpus = concurrency_factor * concurrent_runs

        total_demand_vcpus += job_demand_vcpus
        rows.append({
            "job_id": row["job_id"],
            "concurrency_factor": round(concurrency_factor, 2),
            "demand_vcpus": round(job_demand_vcpus, 2),
        })

    # headroom: size the cluster so expected demand only uses target_utilization
    # of capacity, leaving room to absorb bursts without runtime latency growing
    recommended_vcpus = math.ceil(total_demand_vcpus / target_utilization)

    return rows, total_demand_vcpus, recommended_vcpus


if __name__ == "__main__":
    rows, total_demand, recommended = recommend_cluster_vcpus(CSV_DATA)

    print(f"{'job_id':<12}{'vcpu/run':<10}{'demand_vcpus':<14}")
    for r in rows:
        print(f"{r['job_id']:<12}{r['concurrency_factor']:<10}{r['demand_vcpus']:<14}")

    print(f"\nTotal expected concurrent demand: {total_demand:.2f} vCPUs")
    print(f"Target utilization: {TARGET_UTILIZATION:.0%}")
    print(f"Recommended minimum cluster size: {recommended} vCPUs")

Sample output

job_id      vcpu/run  demand_vcpus  
etl_ingest  0.75      3.0           
report_gen  1.5       3.0           
ml_scoring  0.8       8.0           

Total expected concurrent demand: 14.00 vCPUs
Target utilization: 70%
Recommended minimum cluster size: 20 vCPUs

Key points

• The concurrency factor, CPU-seconds divided by wall-clock seconds, captures whether a job is CPU-bound (factor near or above 1) or partly I/O- or wait-bound (factor well below 1); it is not simply "1 vCPU per job."
• Target utilization is the headroom knob: lowering it (for example to 0.5) recommends more spare capacity and shorter queueing delay under bursts, at higher steady-state cost; raising it toward 1.0 saves cost but leaves no room to absorb variance.
• This model assumes the historic averages are representative of near-future load; a job mix that is changing quickly needs more frequent re-runs of this calculation, or a rolling window instead of an all-time average.

Complexity

Reading and processing n historic job rows is O(n) time and O(n) space, dominated by the single pass over the CSV; this scales to a very large job catalog without needing anything more sophisticated.

Edge cases

• A job with avg_runtime_seconds of 0 would divide by zero; real data should filter or flag such rows before this runs.
• A job type with avg_concurrent_runs of 0 contributes nothing and can be safely included, since it adds zero demand.
• If the CSV has duplicate job_id rows, this script sums both as independent demand sources; decide whether that is correct or whether duplicates should be merged first, depending on what a duplicate row actually represents in the source data.

What "24 hours ahead" actually requires

A useful 24-hour-ahead forecast needs to capture the normal daily and weekly shape of your traffic well enough that a genuine trend toward exhaustion stands out from that normal shape, not from raw noise.

Data to feed it

Historical utilization (or the specific resource metric you are forecasting: disk, connection pool, queue depth) at a consistent resolution, covering enough history to include at least a few full weekly cycles, since day-of-week effects (weekday versus weekend) are often as large as the day's own daily cycle. Tag known anomalies in the history (a past incident, a one-off traffic event) so the forecasting method is not trained to treat a past outage as normal seasonality.

Forecasting approach

Start with the simplest method that captures daily and weekly seasonality well: a seasonal-naive baseline (assume tomorrow looks like last week, same day, same hour) is a surprisingly strong baseline and a good sanity check for anything fancier. A step up from that, Holt-Winters exponential smoothing (a classic time-series method that separately models level, trend, and seasonality) handles a genuine upward trend layered on top of daily and weekly seasonality, which is usually exactly the shape a capacity-exhaustion forecast needs. Reserve heavier approaches, like a learned model such as Prophet or an LSTM, for cases with complex, multi-scale seasonality (say, both a daily cycle and a monthly billing-cycle effect) that a simpler model demonstrably fails to capture; for most single-resource capacity forecasts, that added complexity is rarely worth its own instability and is not what most interviews are actually probing for.

Alerting thresholds and runbooks

• Forecast crossing 80 percent of capacity within the 24-hour window: file a ticket for the owning team to review same-day, no page.
• Forecast crossing 90 percent within the window: page on-call.
• Runbook on either alert: first confirm the forecast is not being thrown off by a known anomaly (a one-off event, a recent deploy), then decide between adding capacity ahead of the predicted exhaustion or, if the forecast turns out to be an artifact, tuning the model or excluding the anomalous period from its training data.

Keeping false positives low in a noisy environment

• Require the forecast to cross the threshold on N consecutive evaluation runs (for example, three in a row) before alerting, rather than acting on a single noisy forecast, since a forecast recomputed every few minutes will occasionally spike for reasons that resolve on their own.
• Backtest the model against every past real exhaustion event and every past false alarm you can identify, and use that history to tune how wide a confidence band you require before treating a forecast as actionable rather than just watching it.
• Track the false-positive rate itself as an ongoing metric (how often did an alert fire with no real exhaustion following), since a forecasting model that is well-tuned today can drift as traffic patterns change, and a rising false-positive rate is the signal that it needs retuning.

CPU

SLI (service level indicator, the actual measured value): average and p95 (the value below which 95 percent of samples fall) CPU utilization per instance over a rolling 5-minute window.
Why it helps: it's the most direct signal of how close a compute-bound service is to needing more capacity, and it's what most autoscaling policies key off.
Alert or dashboard: alert when p95 CPU exceeds 75% sustained for 15 minutes, leaving headroom before saturation actually degrades latency; dashboard it as a fleet-wide heatmap over time so uneven load across a few hot hosts is visible separately from a genuine capacity shortage.

Memory

SLI: resident memory usage as a percentage of instance memory, plus, for garbage-collected runtimes like the JVM, time spent in garbage collection per minute.
Why it helps: memory exhaustion causes a hard failure (an out-of-memory kill) rather than gradual degradation, so it needs a different response than CPU. Memory problems are usually a leak or a workload-size regression, not something you autoscale away.
Alert or dashboard: alert at 85% memory utilization, before the OS starts swapping or the kernel's out-of-memory killer engages, and separately alert if garbage-collection time exceeds 10% of wall-clock time; dashboard it as a trend line per instance to catch slow leaks a point-in-time snapshot would miss.

Storage I/O

SLI: disk read and write latency (p95, in milliseconds) and IOPS (input/output operations per second) as a percentage of the volume's provisioned IOPS limit.
Why it helps: storage capacity planning is really about staying under a provisioned ceiling, since most cloud block storage has a hard IOPS or throughput limit, so "I/O latency spiking" usually means you're hitting that ceiling, not that the disk failed.
Alert or dashboard: alert when IOPS utilization exceeds 80% of the provisioned limit for 10 minutes; dashboard p95 I/O latency next to IOPS utilization so it's visible exactly when latency starts climbing as utilization approaches the ceiling.

Network throughput

SLI: bytes per second in and out as a percentage of the instance's or interface's provisioned bandwidth cap.
Why it helps: bandwidth-bound services, like large file transfer or video, can be fully healthy on CPU, memory, and disk and still degrade because they're capped on the network, so this metric catches a capacity limit the other three would miss entirely.
Alert or dashboard: alert at 70% of the provisioned bandwidth cap, earlier than the 75 to 85% range used across CPU, memory, and storage I/O above (75% for CPU, 85% for memory, 80% for storage I/O), since network saturation degrades sharply near the limit; dashboard it as a stacked area chart of inbound vs outbound traffic per instance type, since asymmetric patterns often reveal a specific workload driving the need.

Practical note on collection

A concrete example: emit all four via a host agent at 60-second resolution, retained at full resolution for 15 days and downsampled to 5-minute or hourly aggregates beyond that for long-term capacity trend analysis.