Infrastructure Scaling, Capacity Planning, and High Availability Questions

What it is

A health check is a periodic probe a load balancer (LB) sends to each backend instance to decide whether to keep routing traffic to it. Get the tuning wrong and you either keep sending users to a broken instance or pull healthy capacity out of rotation for no reason.

Three types

• TCP check: opens and closes a connection to the port. Confirms the process is listening, nothing about the app itself. Cheapest and fastest.
• HTTP/HTTPS check: sends a GET to a path like /healthz, expects a 200 (sometimes a body match). Confirms the web server and app framework are actually responding.
• Application-level script: a custom endpoint that also checks the app's own dependencies (database connection, cache). Most thorough, but if it checks a shared dependency, one outage can mark every instance unhealthy at once, exactly when you need capacity most. Keep it scoped to what that instance itself needs to serve traffic.

Tuning parameters, worked example

Interval (how often to probe, e.g. 10s), timeout (how long to wait before counting a failure, e.g. 5s, must be under the interval), unhealthy threshold (consecutive failures before pulling the instance, e.g. 3), healthy threshold (consecutive successes before adding it back, e.g. 2). With interval=10s and unhealthy threshold=3, an instance survives up to 30 seconds of trouble before it is pulled, so a 5 second blip never trips it. Tighten either number and you react faster, but to more noise.

Long warm-up phases

An instance that needs 60 seconds to warm a cache before it can serve traffic will fail checks during that window under normal settings, and some orchestrators will restart it in a loop so it never finishes starting. Fix: a startup grace period (checks do not count against the instance until it elapses), sized past the worst-case warm-up time, paired with a readiness check that only turns healthy once startup work is actually done.

How misconfiguration causes flapping or false positives

• Threshold too tight (unhealthy threshold=1): one slow response flips the instance out, it passes the next check and flips back in, repeatedly (flapping), and each flap costs real capacity.
• Timeout shorter than the app's worst-case healthy response time: the check itself becomes the false positive.
• No grace period on a slow starter: a permanent restart loop that looks like an outage but is a config bug.
• A deep check tied to a flaky shared dependency: a correlated failure across the whole fleet, the opposite of what health checks exist to prevent.

Concrete implementation

1. Create one weighted DNS record set per region, with weights matching the desired traffic split, for example a 70/30 split between two regional endpoints, each record pointing at that region's own load balancer or endpoint.
2. Attach a DNS-level health check to each weighted record, probing a lightweight endpoint (such as /healthz) in that region with a defined check interval and failure threshold.
3. Configure the DNS service so that when a record's health check fails, its weight is automatically excluded from the responses returned to clients, and traffic is redistributed proportionally among the remaining healthy weighted records.
4. Set the record TTL low enough to allow reasonably fast failover propagation, commonly in the 30 to 60 second range, while recognizing that a very low TTL increases DNS query volume and cost since clients must re-resolve far more often.

Worked failover-latency example

Health check: interval 10s, unhealthy threshold 3 consecutive failures -> up to 30s to mark unhealthy
DNS record TTL: 60s, a client holding a cached answer can take up to 60s more to naturally re-resolve
Worst-case total = 30s (detection) + 60s (TTL expiry) = up to 90s before all clients have moved off

Considerations

• DNS caching on clients: not every resolver or client strictly respects the configured TTL, some corporate resolvers, carrier networks, and browser-level caches hold answers longer, which means real-world failover time for a meaningful minority of clients can exceed the 90-second worst case above, no matter how the health check itself is tuned. This is a structural ceiling on pure DNS-based failover, not a configuration mistake to fix.
• Health check aggressiveness: a shorter interval and lower failure threshold detects a real outage faster but risks marking a merely momentarily-slow, healthy region unhealthy, the same false-positive-versus-detection-speed trade-off that applies to any health check. Providers typically probe from multiple external vantage points and require a majority to agree before declaring a record unhealthy, which adds a little latency but avoids one bad network path triggering an unnecessary failover.
• Because of the TTL-driven ceiling described above, DNS-based failover alone is usually not sufficient for a sub-few-seconds recovery target. Pairing it with a global anycast or edge load balancer, which fails over at the network or health-check layer rather than waiting on DNS caches to expire, is the right choice when faster, more deterministic failover is required than DNS alone can structurally provide.

Goal

Catch capacity saturation while it is still a leading indicator, before it becomes a user-visible latency or error problem, by instrumenting every layer a request actually passes through, not just the infrastructure underneath it.

Metrics by layer

• App layer: in-process request queue depth (requests waiting for a worker thread or handler), thread pool or connection pool utilization, and garbage-collection pause time for managed runtimes. These often rise before CPU or memory looks unusual, since a service can be capacity-constrained on concurrency long before it is constrained on raw resources.
• Infra layer: CPU, memory, disk I/O, and network throughput, the traditional signals, still necessary but rarely sufficient on their own.
• Queueing layer: for any service that sits behind a message broker or work queue, consumer lag (how far behind the latest message the consumer group is) and broker-side queue depth. A queue that is quietly growing is one of the clearest leading indicators of saturation there is, since it shows demand is already outpacing processing capacity, even while current latency for messages already being processed still looks fine.

Recommended alert thresholds

Alert on trend, not just absolute level: a request queue depth or consumer lag that is growing steadily over a sustained window (say, 10 minutes) is a stronger, earlier signal than a single absolute threshold crossing, since the absolute number that indicates trouble varies a lot by service and by time of day.

Dashboards

A layered dashboard mirrors the layered instrumentation: app-layer queue and pool metrics on top, infra metrics in the middle, queueing-layer lag at the bottom, all on the same time axis, so an operator can see which layer moved first when investigating an incident.

Synthetic checks to validate capacity continuously

Real traffic metrics can hide a capacity problem during genuinely quiet periods, low traffic can mask that a single node is already struggling, or that a dependency has degraded, simply because there is not enough real load to expose it yet. Address this with:

• Continuous synthetic transactions: a scripted request against the critical path, run from outside the service on a fixed interval (for example, every minute), measuring latency and success independently of real user traffic, so a problem is visible even during a quiet 3am window.
• Periodic synthetic load tests: not continuous, but run on a schedule (say, monthly, or after any significant capacity or code change) to validate that the headroom assumptions from your capacity plan still hold, since the actual per-request cost of a workload drifts as code changes, and a headroom number computed six months ago can be stale.

Together, continuous synthetic checks catch a currently-broken system that real traffic is too light to expose, and periodic load tests catch a headroom assumption that has quietly become wrong.

Single primary with cross-region reads

All writes go to one region, potentially over a long-haul link if the writing client is not co-located with the primary, which adds real round-trip latency, on the order of tens to over a hundred milliseconds for a trans-continental hop. Reads are served locally from replicas in every region. Capacity: the primary must be sized for 100 percent of global write volume by itself, while each replica is sized only for its local read share. Replication bandwidth is one-directional fan-out from the primary to N replicas, proportional to write rate times replica count. Operational complexity is comparatively low, there is exactly one writer, so no conflict resolution is needed, and failover is a straightforward promote-a-new-primary operation.

Multi-primary with conflict resolution

Each region accepts writes locally, at local latency, which is the main appeal for users everywhere. This requires an explicit conflict-resolution strategy: last-writer-wins by timestamp (simple, but a legitimate concurrent update can silently lose); vector clocks (a per-writer counter that lets you detect, without a shared or synchronized clock, whether one update causally preceded another or whether two updates are truly concurrent; a vector clock does NOT tell you which of two genuinely concurrent updates happened first, that is exactly the case it identifies as unorderable, so a real conflict still needs a separate resolution rule, such as last-writer-wins or an application-level merge, layered on top); CRDTs (conflict-free replicated data types, data structures designed so concurrent updates merge automatically) for data types that tolerate merging; or, usually the practical default, application-level partitioning where each region owns a disjoint set of keys, since concurrent writes to the same record in different regions can otherwise collide. Capacity: write load is naturally distributed, each region absorbs its own local writes, but every region must ALSO process the replication stream of every other region's writes, so a region's total write-processing load is its own local writes plus the incoming replicated writes from every other region. Replication bandwidth grows as a full mesh between regions rather than one-directional fan-out, so it scales faster as regions are added. Operational complexity is substantially higher: conflict-resolution logic, harder-to-reason-about consistency, and failure debugging that now has to ask which region's version actually won.

When each is appropriate

Single-primary with cross-region reads fits when writes are a small fraction of total traffic and can tolerate the extra cross-region latency, or when strict write ordering matters enough that true multi-master conflict risk is unacceptable, a financial ledger is the classic example. Multi-primary fits when low local write latency is a hard product requirement, for example a collaborative application where users everywhere write frequently, and the data model can define conflict resolution safely, or better, can be partitioned so each region owns a disjoint key range, which sidesteps true conflicts entirely and captures much of multi-primary's latency benefit without its full operational cost.

Testing the choice rather than deciding on paper

Simulate the real write pattern, including a realistic estimate of concurrent-write frequency, against both topologies in staging before committing to either. For single-primary, measure actual write latency from each target region under load. For multi-primary, measure the real observed conflict rate and verify the resolution logic behaves correctly under concurrent writers that resemble production traffic, not synthetic single-writer tests. The deciding factor in practice is usually operational overhead relative to what you actually get: if the simulated conflict rate turns out to be near zero, for example because writes are already naturally partitioned by user or tenant, multi-primary's added complexity buys little extra correctness benefit, and single-primary or an explicitly partitioned-ownership model wins on simplicity. If conflicts are genuinely frequent and the business truly needs local write latency everywhere, the multi-primary investment is justified by that measurement, not by architectural preference alone.

Definitions

Capacity planning is the process of forecasting future demand and provisioning infrastructure (compute, storage, network) to meet it within service commitments, without wildly over- or under-spending. Headroom is the portion of provisioned capacity deliberately kept unused at any given moment, a buffer against normal variance and unplanned events, expressed as a gap between what is provisioned and what is currently needed.

Short-term autoscaling vs long-term capacity planning

Short-term autoscaling is reactive and automated: it adjusts the running instance count within pre-set minimum and maximum bounds, in minutes, to match current load. It cannot change those bounds itself, provision new hardware, or exceed an account or region quota. Long-term capacity planning is proactive and typically human-driven: it decides, over months or quarters, what those autoscaling bounds should be, whether to reserve capacity or commit to discounted pricing, and whether the current architecture and quotas can support projected demand at all.

Both matter together because they cover each other's blind spot: autoscaling without capacity planning can hit an invisible ceiling right when you need capacity most, such as a cloud account quota, an exhausted subnet IP range, or a database that cannot accept more connections, none of which autoscaling can fix by itself. Capacity planning without autoscaling wastes money running a peak-sized fleet around the clock even during quiet hours.

Example metrics per resource type

• Compute: CPU utilization percent, memory utilization percent, request or job queue latency, thread pool saturation.
• Storage: bytes used versus provisioned, IOPS (input/output operations per second) or throughput utilization versus the provisioned limit, and growth rate in GB per day.
• Network: bandwidth utilization versus link capacity, packets per second, active connection count versus any configured limit, and cross-AZ or cross-region egress (outbound traffic leaving the network or region, as opposed to inbound/ingress traffic) volume (which often carries its own cost and capacity limits separate from raw bandwidth).
• Application-level: requests per second, queue depth or backlog age, error rate, and latency percentiles such as p95 (the value below which 95 percent of observed requests complete, a common way to capture "the experience for almost everyone" rather than a possibly-misleading average).