Infrastructure Scaling, Capacity Planning, and High Availability Questions

Translating the SLOs into raw headroom

An availability SLO of 99.95 percent, measured monthly, allows about 21.6 minutes of downtime a month (0.05 percent of 30 days x 24 hours x 60 minutes). That is the error budget the redundancy design has to protect. The latency SLO, p95 under 100 milliseconds (p95 meaning 95 percent of requests must complete faster than this), constrains utilization directly: queueing delay rises sharply as a system approaches full utilization, so capacity is typically kept well under 100 percent, often in the 50 to 65 percent range, specifically to keep that latency tail short rather than to save headroom for growth.

Redundancy levels: N+1 vs N+2

N+1 (glossed: enough spare capacity that losing any one unit, such as one instance, still leaves full needed capacity in place) is the minimum for surviving a routine instance failure, which happens regularly and should never be an incident. N+2 (surviving the loss of any two units simultaneously) is used when failures might be correlated, for example wanting to survive a full availability zone (AZ) loss while also tolerating one more ordinary instance failure on top of it during that same window.

Instance loss vs AZ loss

Instance loss is routine and should always be covered by at least N+1. AZ loss is a bigger event: if capacity is spread evenly across 3 AZs, losing one AZ removes about a third of total capacity. To survive that and still meet the SLO, the remaining AZs must alone be able to carry 100 percent of peak load:

With k=3 AZs, surviving the loss of 1 AZ requires
total capacity >= peak load x k / (k-1) = peak x 3/2 = 1.5x peak

That is 50 percent more total fleet capacity than the raw peak load, spread across the three zones, on top of whatever CPU or latency headroom (the 50 to 65 percent utilization target above) is already built in for normal operation.

Instance counts and load balancing choices

Instance counts should be set from the 1.5x-peak-across-3-AZs figure above, translated into instances using whatever per-instance throughput number a load test confirms. Load balancing needs multi-AZ target groups with health checks fast enough to detect and reroute around a lost instance or AZ well inside the 21.6-minute monthly error budget: a slow health-check-and-failover process that takes even 5 minutes to react eats nearly a quarter of the entire month's downtime allowance in a single event. Connection draining (letting in-flight requests finish instead of dropping them mid-failover) protects the latency SLO during the failover itself, not just the availability SLO.

Detection

Watch node conditions, not just uptime. The kubelet reports heartbeats to the control plane; if a node misses them for a configured window (commonly tens of seconds), the node controller marks it NotReady, and pods on it become eligible for eviction after a further toleration period (often around 5 minutes by default) if the node doesn't recover. Also watch the finer-grained conditions: DiskPressure, MemoryPressure, PIDPressure, plus cloud-provider instance health checks (for example AWS EC2 status check failures), since a node can be technically Ready while still failing in a way that will soon cause pod failures. A node-problem-detector style agent catches kernel-level and hardware issues the kubelet heartbeat alone won't surface.

Cordon, drain, reprovision

1. Cordon the node first: mark it unschedulable (kubectl cordon) so no new pods land on a node you're about to remove, without disturbing what's already running.
2. Drain: evict existing pods, respecting PodDisruptionBudgets and each pod's grace period, so the workload moves off gracefully rather than being hard-killed.
3. If a pod won't evict cleanly (stuck Terminating, often from a hung volume), force-delete after a bounded timeout, documenting the real risk: for workloads using local storage or non-idempotent writes, a forced delete can leave a duplicate instance briefly running before the old one is confirmed dead.
4. Reprovision: terminate the underlying instance and let the cluster autoscaler, Karpenter, or the managed node group's own desired-count mechanism (an ASG (Auto Scaling Group) replacing a terminated instance from the same launch template) bring up a fresh, healthy node.

Avoiding cascading failures during simultaneous repairs

• Rate-limit repairs: cap how many nodes can be cordoned/repaired at once, for example no more than one node or 10% of the pool concurrently, similar to how a rolling node upgrade is staged.
• Respect PodDisruptionBudgets across the whole repair loop, not just per individual drain, so many simultaneous repairs don't collectively violate a budget that each single drain would have respected on its own.
• Add a circuit breaker for correlated failures: if an unusually large fraction of the pool (for example over 20%) goes unhealthy at once, that's much more likely a systemic issue, a bad AMI rollout, a control-plane problem, a network partition, than independent hardware failures. Auto-repair should pause and page a human in that case rather than mass-repairing, since cycling many nodes simultaneously during a systemic event compounds the outage instead of fixing it.

Metric choice

Primary: target tracking (a policy that continuously compares the live metric to your chosen target and adds or removes capacity in proportion to how far off target it is, similar to a thermostat) on CPU utilization at a 55 to 65% target. The load balancer's own request-count-per-target metric is a good secondary signal if request cost is uneven. For a stateless, CPU-bound web tier, CPU predicts saturation directly and every major cloud target-tracking policy supports it out of the box, so it's the simplest policy to reason about and debug first.

Sampling window and alarm thresholds

• Scale-up: evaluate 3 consecutive 60-second periods; if average CPU exceeds 65% across all 3, trigger scale-out.
• Scale-down: evaluate 15 consecutive 60-second periods (15 minutes); if average CPU stays under 30% across all of them, trigger scale-in. The longer window on the way down is deliberate, so a brief traffic lull doesn't remove capacity.

Cooldowns

Scale-up cooldown of 60 seconds (react quickly, but let one action complete before another can fire). Scale-down cooldown of 300 seconds.

Minimum and maximum instance counts

Minimum of 2 instances split across 2 availability zones (independent data-center-like failure domains within a cloud region), so a single zone failure never drops you to zero capacity. Maximum set from a load test result, for example 20 instances if that's the point where a backing database connection pool becomes the bottleneck rather than compute.

Health-check and warm-up considerations

Load-balancer health check: 15-second interval, 2 consecutive successes to mark healthy, 3 consecutive failures to mark unhealthy, so a flapping instance is pulled from rotation within about 45 seconds. Give a new instance a health-check grace period of 90 seconds (or however long your bootstrap script actually takes) so the autoscaler doesn't kill an instance that's still warming up caches or JIT-compiling. If using target tracking, set the same value as the metric's instance-warmup period, so a freshly launched instance's low initial CPU doesn't get counted as evidence you need even more capacity.

Safety limits against oscillation

• Hard min/max bounds so no single bad signal can scale to zero or runaway.
• A step-size cap on scale-in, for example never remove more than 10% of fleet size in one action.
• Prefer target tracking over hand-tuned step scaling for the primary policy since it's self-correcting; add a step-scaling policy only for a documented big-spike case.
• Alarm on the fleet's own scaling-activity rate (for example more than 4 scaling actions in 15 minutes) as a meta-alert that your policy itself is flapping and needs review.

Framing

The manager's ask has two parts that both need proving, not just one: cost went down by roughly 20%, and reliability, p95 latency (the response time 95% of requests come in under) and other service-level objectives, stayed intact. A plan that
only shows the cost number isn't a complete answer, since a naive way to cut cost is to simply
under-provision, which would show a cost drop while quietly degrading the service level.

Key metrics, including a cost-efficiency metric

Raw cost isn't the right primary metric on its own, because traffic volume changes day to day
independent of the capacity change, comparing raw cost before and after conflates "the change saved
money" with "traffic happened to be lower this week." The metric that isolates the change's own
effect is cost per successful user action, total infrastructure cost attributable to this service
divided by the count of successfully-completed user actions in the same window, a ratio metric that
stays comparable across different traffic volumes. Alongside it: p95 and p99 latency (the same idea at the stricter 99th-percentile bound), error rate,
and, if capacity-related, queue depth or saturation indicators, tracked in the same windows as the
cost-efficiency metric, so a reviewer sees cost and reliability together, not in separate reports.

Dashboards

One dashboard showing cost per successful action as a time series spanning both the before and
after window, not just two summary numbers, so a reviewer can see the actual trend and whether the
drop happened cleanly at the change's deploy time or drifted gradually, a gradual drift is a red
flag it's confounded with something else, like a traffic-mix shift. A second, paired view with the
reliability metrics on the same time axis, directly under or beside the cost panel, so a viewer can
visually confirm the service levels held steady across the exact same window the cost dropped.

Experiment or rollback plan

Ideal: an A/B-style rollout, the capacity or autoscaling change applied to a random slice of
traffic or a random subset of instances, with the rest as a control running the old configuration
concurrently, this isolates the change's effect from time-based traffic variance far better than a
simple before-and-after comparison, because alternating or randomly splitting the old and new
configuration concurrently means whatever else varies over time, traffic mix, time of day, an
unrelated deploy, affects both configurations equally instead of confounding a sequential
before-then-after read. If a true concurrent A/B isn't feasible for an infrastructure-wide
change, the fallback is a clean before-and-after window matched as closely as possible on external
conditions, same day-of-week pattern, no other deploys or incidents in either window, with an
explicit rollback trigger defined in advance, for example immediate rollback if p95 latency or
error rate crosses its threshold at any point, agreed before the change ships, not decided
reactively if something looks off.

Worked example: statistical tests and confidence intervals

Treat each day, or window, in the before and after periods as one sample of the cost-per-successful-action
ratio, and compute a confidence interval on the percentage change, the same way a percentage
reduction claim needs an interval, not a single point comparison.

Worked example, computed for real: 14 days pre-change and 14 days post-change, daily cost per 1,000 successful actions computed for each day. Mean pre equals $8.055 per 1,000 actions, standard deviation $0.024, mean post equals $6.527 per 1,000 actions, standard deviation $0.031, a measured 19.0% reduction. A Welch's t-test (a statistical test for whether two group means differ by more than sampling noise would explain, used here instead of a plain t-test because the two 14-day samples are not assumed to share the same variance) comparing the two means gives t is approximately 145.8 on about 24 degrees of freedom (Welch-Satterthwaite approximation), with a resulting p-value on the order of 1e-37, i.e. effectively zero and nowhere close to the usual 0.05 significance threshold, meaning the drop is very unlikely to be explained by day-to-day noise alone. This comes in just under the manager's claimed
20%, close enough that I'd report it as "measured roughly 19%, consistent with the claimed 20%
within normal estimation variance," rather than either rubber-stamping the exact claimed number or
disputing a difference this small.

Attributing the change to the capacity change rather than traffic variance

The ratio metric already controls for raw traffic volume changes, but not for traffic mix changes,
a shift toward cheaper-to-serve request types would also lower this ratio with no capacity change
involved. Check that the request-type or endpoint mix was stable across the two windows, or if it
shifted, control for it by computing the ratio per request type and comparing like for like, rather
than trusting the blended number. Confirm no other deploy landed in either window that could
independently affect cost or latency, a clean attribution needs the capacity change to be the only
variable that moved.

What I'd report to my manager

The measured percentage reduction with its confidence interval, not just a single point number, the
reliability metrics proving service levels held across the same window, and an explicit statement
of what was controlled for, traffic volume via the ratio metric, traffic mix via the per-type check,
other concurrent changes via a deploy-log check, so the claim is defensible under scrutiny, not
just asserted.

The core requirement

At-least-once processing means every message is guaranteed to be processed at least one time, but might occasionally be processed more than once (which is why consumers of a queue-processing system need to be idempotent, safe to run twice, as a general rule, not specific to autoscaling). The autoscaling danger is scale-in: if a consumer is killed while it still owns unprocessed or in-flight messages, those messages are either lost (bad) or left invisible until a timeout expires (a processing delay, tolerable but should be minimized).

What happens to a consumer from removal to exit, step by step

1. Marked for removal. The autoscaler (or orchestrator) picks a consumer to remove based on the scale-in policy, but does not kill it immediately.
2. Stop assigning new work. The consumer is told to stop pulling new messages or, for a partitioned system like Kafka, is signaled to give up its partition assignments. It does not touch anything it has not already started.
3. Drain in-flight work. The consumer finishes processing whatever it already pulled (its current batch), it does not abandon a message mid-processing. This is why the drain step needs a bounded maximum time: a message that is stuck (a poison message, or a slow downstream dependency) cannot be allowed to block scale-in forever.
4. Commit offsets or acknowledge messages. Only after a message is fully processed does the consumer commit its offset (Kafka) or delete/ack the message (SQS or a similar queue). This ordering, process fully before acknowledging, is what makes at-least-once delivery hold even if the consumer dies partway: an unacknowledged message simply becomes visible again for another consumer to pick up.
5. Trigger rebalance. For a partitioned queue, giving up ownership triggers a rebalance (partitions reassigned to the remaining consumers). The removed consumer should not exit until it receives confirmation the rebalance completed, otherwise there is a window where its former partitions are owned by no one.
6. Deregister and terminate. Only now does the instance actually shut down. This is implemented with a lifecycle hook, for example an Auto Scaling Group lifecycle hook, or in Kubernetes a preStop hook combined with a generous terminationGracePeriodSeconds, which delays the actual termination signal until the application signals it is safely drained.

How cooldowns and thresholds differ from a stateless service

• Scale-in cooldown must be at least as long as the worst-case drain time (max message processing time plus rebalance time), not a fixed short number picked for a stateless web tier. A stateless service's scale-in cooldown only needs to cover connection draining, which is usually seconds; here it can legitimately be minutes.
• The scaling signal itself should be per-consumer backlog, not just aggregate queue depth. Removing a consumer based only on the aggregate looking low can still remove one that happens to own a disproportionate share of unprocessed messages; check that the specific consumer being removed has a small, close-to-empty backlog before selecting it.
• Scale-in should be more conservative than scale-out overall. Fast, generous scale-out (add consumers quickly when backlog grows, since adding capacity is low-risk) paired with slow, cautious scale-in (only remove a consumer after a sustained period of low backlog on that specific consumer, since removing capacity risks a stuck drain or a rebalance storm if too many consumers leave at once).
• Limit how many consumers can scale-in at once. Removing several consumers simultaneously multiplies rebalance activity and can itself cause a processing stall; a stateless service rarely needs this kind of rate limit on its own scale-in.