Infrastructure Scaling, Capacity Planning, and High Availability Questions

Framing

This is a tool-building question, design a system that recommends right-sizing changes (adjusting an instance or container's allocated CPU, memory, or storage to match its actual observed usage rather than an arbitrarily chosen size) for a human to review, not "personally right-size this one workload." The deliverable is a pipeline plus a safety framework, not a single answer.

Data pipeline

Ingest historical resource-utilization telemetry, CPU, memory, disk, and network, per instance or container at fine granularity, for example one-minute samples, from the existing metrics store, over a window long enough to capture natural cycles: at least weekly seasonality, longer if the workload has monthly or quarterly patterns like a billing batch job. Join that with the currently allocated size, what's being paid for, and, where available, application-level signals like error rate and latency, so a recommendation can be checked against outcome, not just cost.

Feature engineering

Build per-workload summary features beyond a single average: p50, p95, and p99 utilization (the utilization level that 50%/95%/99% of your one-minute samples fall below), since a workload averaging 20% CPU but spiking to 95% needs different treatment than one flat at 20%; a peak-to-trough ratio capturing how bursty the workload is; out-of-memory (OOM) kill history or throttling events, a strong safety signal that current sizing is already too tight and should weight against shrinking; and restart or crash frequency correlated with resource pressure.

Modeling approach by recommendation type

For steady, low-variance workloads, a straightforward statistical rule, for example sizing to a percentile of observed peak plus a safety margin, is sufficient and easy to explain to a human reviewer. For bursty, spiky workloads, model the full distribution rather than only mean or peak, and reason explicitly about acceptable throttling risk, since naive peak-based sizing either wastes money on a rare spike or risks throttling on average load; a burst-tolerant recommendation might size below peak while relying on the workload's own autoscaling to absorb bursts. For workloads with strong seasonality or trend, use a lightweight forecasting approach rather than only backward-looking statistics, so the recommendation anticipates the next cycle's peak rather than reacting to the last one, particularly for workloads that are still growing.

Evaluating whether a recommendation is safe to act on

Require a minimum observation window and a minimum confidence that the workload's usage pattern is a genuine, stable shape rather than a one-off spike, before recommending any change. Explicitly compute and surface the recommendation's downside risk alongside its projected savings, for example backtesting the change against the workload's own history to report how many OOM events it would have caused had it been applied retroactively, never present savings without the paired risk estimate. Flag anything touching a workload with recent incidents, or classified as customer-critical, for mandatory human review regardless of how confident the model is.

Rolling out with a human in the loop

Never auto-apply a shrink recommendation directly to production. Start with a suggest-only mode, the recommendation surfaced in a dashboard or ticket with its backing evidence, for a probation period. For changes a team approves, apply gradually, resize one instance or replica of a fleet first and monitor before proceeding, rather than fleet-wide at once, and keep an automatic rollback path tied to the same alerting thresholds used elsewhere, so a bad recommendation that slips through review self-heals quickly.

A repeatable right-sizing process

1. Data collection: pull utilization telemetry (CPU, memory, network, and disk IO) per resource over a window long enough to smooth out weekly cycles, a 30-day window at 1 to 5 minute granularity is a reasonable default, since it captures multiple weekday and weekend cycles rather than one atypical week.
2. Percentile-based sizing analysis: never size to the average, which hides real peaks, and never size to the raw maximum, which lets one outlier spike dictate cost forever. Size to a high percentile of the observed distribution, commonly p95 or p99 (the value that 95% or 99% of observed samples fall at or below) across the full window, then apply the organization's required burst headroom on top of that.
3. Pilot change execution: apply the recommendation to a small canary slice first, one availability zone or 5 to 10 percent of the fleet, and observe it through at least one more full cycle before a fleet-wide rollout.
4. Rollback plan: keep the prior instance type or launch template version pinned and ready, with a one-step revert, and define explicit rollback triggers up front (for example, error rate or p99 latency crossing a stated threshold), so rollback is a decision made in advance, not improvised under pressure.
5. Quantify validated savings: compare actual billed cost before and after over an equivalent window, normalized by request volume if traffic isn't constant (cost per 1000 requests, for example), while confirming SLA-relevant metrics (latency percentiles, error rate) stayed within target throughout.

Worked example applying a burst-headroom policy

Suppose an instance is provisioned at 8 vCPU, and 30 days of telemetry show p95 CPU utilization of 25 percent. Converting to a vCPU-equivalent load:

Observed p95 load = 25% x 8 vCPU = 2.0 vCPU-equivalent
Company policy requires 20% burst headroom, so target max utilization on the new size = 80%
Minimum capacity needed = 2.0 vCPU / 0.80 = 2.5 vCPU
Round up to the next standard size that covers 2.5 vCPU: 4 vCPU

Right-sizing from 8 vCPU down to 4 vCPU roughly halves compute spend on this resource, while the 20 percent headroom policy is preserved: at the observed p95 load, the new 4 vCPU instance sits at 50 percent utilization (2.0 / 4.0), comfortably inside the 80 percent target ceiling, not right up against it.

Extending the process to network services and a heterogeneous EC2-style fleet

The same percentile-and-headroom logic applies to network services (right-sizing bandwidth or connection-pool limits against observed p95/p99 traffic), but a mixed compute fleet adds two more variables to the recommendation step:

• CPU architecture differences: a more efficient chip architecture can deliver the same throughput at a lower cost per unit of work, but may require recompiling or re-validating the workload, so factor in migration/testing cost, not just the sticker price difference, before recommending a cross-architecture move.
• Bursting behavior: burstable instance families that rely on accumulated CPU credits are a good fit for genuinely spiky, low-average workloads, and a bad fit for anything with sustained high CPU, which will exhaust its credit balance and get throttled exactly when it needs capacity most. A right-sizing recommendation must check whether the observed load is actually bursty (low average, occasional spikes) before recommending a burstable family, not just look at the average utilization number in isolation.
• Purchasing-option trade-offs layer on top of the instance-family recommendation: a steady, always-on baseline workload is a good candidate for a committed purchase, a fault-tolerant, interruptible workload is a good candidate for spot capacity, and a genuinely unpredictable or short-lived workload is best left on-demand, this decision is separate from, and should be made after, the instance-family and size recommendation itself.

Step 1: read the scheduler's own explanation

Run kubectl get pods to confirm which pods are Pending, then kubectl describe pod <name> and read the Events section. This is the fastest diagnostic step because the scheduler writes a FailedScheduling event with the actual reason spelled out, for example "0/12 nodes are available: 8 Insufficient cpu, 4 Insufficient memory". Don't skip straight to guessing, this event usually tells you exactly which resource and how many nodes failed on it.

Step 2: check real cluster capacity

kubectl describe nodes (or kubectl top nodes for current usage) shows allocatable versus requested/used CPU and memory across the pool. This distinguishes two very different situations: genuinely out of capacity everywhere, versus capacity existing somewhere but being blocked by a constraint (a taint the pod doesn't tolerate, an affinity rule, a topology spread constraint) even though raw numbers look fine.

Step 3: check the pod's own spec

Look for typos or overly strict nodeSelector/affinity values (a common cause: requesting a label value no node actually has), and confirm the pod's requested CPU/memory isn't larger than any single node's total allocatable capacity, in which case it can never schedule no matter how much capacity exists in aggregate.

Step 4: check namespace-level limits

kubectl describe resourcequota -n <namespace> and any LimitRange: a ResourceQuota can block scheduling before the scheduler even runs, independent of actual node capacity.

Step 5: check the autoscaler

Look at cluster autoscaler or Karpenter logs and events for why it isn't adding capacity: it may have hit a node group's maxSize, lack a node group that supports a requested resource (GPU), or be stalled on a cloud-side issue (instance type capacity shortage in the target AZ (Availability Zone, a physically separate section of a cloud region), an API rate limit or quota error).

Short-term remediation

Manually raise the relevant node group's max size, free capacity by evicting or rescheduling lower-priority pods (respecting PodDisruptionBudgets), or temporarily relax an overly strict affinity/anti-affinity rule that's blocking placement despite available capacity.

Medium-term remediation

Right-size requests using VPA (Vertical Pod Autoscaler, which recommends or adjusts a pod's own CPU/memory requests) recommendations if pods are asking for far more than they use, add a PriorityClass with preemption so critical pods can bump non-critical ones under pressure, pre-provision a small standing buffer of headroom so scale-up isn't always starting cold, and raise autoscaler limits proactively for the node groups that repeatedly hit this failure mode.

Framing

For a read-heavy, globally distributed service where writes can tolerate eventual consistency, the natural shape is a single writer per record with asynchronous replication out to regional read replicas, optimizing for the common case, a nearby, fast read, while keeping the write path simple.

Replication topology

One primary region owns writes for a given piece of data, and streams changes asynchronously to read-replica copies in every other served region. This avoids the coordination cost of true multi-writer conflict handling while still giving every region a local, low-latency copy for reads.

Routing for read locality

Reads route to the nearest healthy regional replica via latency-based DNS or a global load balancer with real-time health checking. Writes route to the single owning primary region regardless of where the request originated, accepting the extra round-trip latency for writes as the deliberate trade for simplicity and consistency on that path.

Estimating replication bandwidth

Suppose the primary handles 2,000 writes per second fleet-wide, each replicated record (including write-ahead-log overhead, the durability log a database writes before applying a change) averages 1.5 KB, and there are 4 secondary regions to replicate to:

Per-destination-region bandwidth = 2,000 writes/sec x 1.5 KB = 3,000 KB/s = 3 MB/s
Total egress from the primary = 3 MB/s x 4 regions = 12 MB/s (about 96 Mbps)

That is the steady-state bandwidth the primary's egress (outgoing traffic) and each secondary's ingress (incoming traffic) link must sustain continuously, before adding margin for catching up after any network interruption.

Failure modes that affect capacity planning

• Replication lag growth: if a secondary's link degrades, lag builds into a backlog, and that secondary must process at a rate ABOVE steady state to catch back up once the link recovers, so each secondary's ingest capacity needs headroom above the steady-state number, not just enough to match it.
• Primary region failure: whichever region is promoted to replace it must absorb both its own regional read-serving load and become the sole global write sink. Every region's capacity plan has to assume it might become primary, sized for full write load, not merely its normal regional read share.
• Network partition between regions: a partitioned secondary can keep serving (possibly stale) reads under eventual consistency, so read capacity is unaffected, but it still accumulates a backlog that needs the same above-steady-state catch-up capacity once the partition heals.

Tiering reads between an edge cache and the regional replica

Not every read needs to hit a database replica at all. For content that is cacheable and can tolerate some staleness, product pages, catalog listings, anything not requiring up-to-the-second freshness, put a CDN or edge cache layer in front of the regional replicas. The edge absorbs the bulk of read volume directly, and the regional replica only needs enough capacity for cache misses and for reads that genuinely cannot be edge-cached, such as personalized or authenticated content. Draw that boundary by cacheability and staleness tolerance, not by region, it changes how much replica capacity you actually need to provision.

Handling the case where some writes do need to happen in more than one region

If a subset of the workload genuinely needs multi-region writes rather than pure single-primary, an explicit conflict-resolution rule is required. Options include last-writer-wins by timestamp (simple, but a legitimate concurrent update can silently lose), CRDT-based merges (CRDT: conflict-free replicated data type, a data structure designed so independent concurrent updates can always be merged automatically without conflicts) for data types that tolerate them (counters, sets), or, usually the safest default, application-level partitioning where each entity has one designated home region for writes, so real conflicts only arise during a failover, not during normal operation.

Framing

This experiment validates the autoscaler's own behavior, does new capacity show up correctly, quickly, and safely at a specific large scale, 10,000 pods, which is different from validating whether the application under it can handle load. The thing under test is the scaling machinery itself.

Generating load

Drive demand for pods indirectly, through whatever actually triggers the autoscaler being tested, for example a scripted wave of Deployments, Jobs, or HPA (Horizontal Pod Autoscaler)-target-triggering load, rather than manually creating pods, so you exercise the real decision path: metrics collection, HPA and cluster-autoscaler decision logic, node provisioning. Ramp in stages, for example 1,000, then 5,000, then 10,000 pods, rather than one instantaneous jump, so you can observe where behavior first degrades instead of only seeing the end state.

Injecting realistic pod start times and custom metrics

Use synthetic workloads whose readiness and startup timing match your real workloads' distribution; a workload that's instantly ready doesn't exercise the same scheduling and admission pressure as one with a realistic multi-second startup. Drive any custom metrics the HPA scales on, for example queue depth or request rate, through the same metrics pipeline production uses, not a shortcut or a mocked value, so the real signal path is validated end to end.

Simulating node provisioning delays and failures

Inject artificial node-join latency, or use an environment where node bring-up is genuinely slow, spinning up real cloud instances rather than pre-warmed nodes, to see how the autoscaler and the pending-pod queue behave under realistic, not instant, node readiness. Separately inject node provisioning failures, for example simulating a regional instance-type capacity shortage, to confirm the autoscaler retries and falls back sensibly rather than looping forever or silently giving up.

Collecting telemetry

Capture, with timestamps, every stage of the pipeline: metric-triggers-decision time, decision-to-node-request time, node-request-to-node-ready time, node-ready-to-pod-scheduled time, and pod-scheduled-to-pod-ready time, so a slow overall scale-up can be attributed to the specific stage responsible rather than treated as one opaque number. Also capture stability signals: count of scale-up and scale-down flaps (an unwarranted scale-down immediately following a scale-up, or vice versa), count of pods scheduled and then evicted or rescheduled, and API server and etcd load during the ramp, since a scale event pushing 10,000 pod creations is itself significant control-plane load.

Pass and fail criteria

Scaling latency: total time from trigger to the target pod count being Ready stays under an agreed bound, with the per-stage breakdown available to diagnose any miss. Stability: zero or near-zero oscillation, an unwarranted scale-down immediately following a scale-up, during and shortly after the ramp, and no pod-eviction storms caused by the scaling process itself. Correctness: the final pod count matches the intended target exactly, no persistent overshoot or undershoot, and no control-plane component breached its own health thresholds during the test.