Infrastructure Scaling, Capacity Planning, and High Availability Questions

Direct answer

Shard by a geography-aware key so each user's data has one home region for writes, route reads to the nearest replica and writes to the shard's home region, and keep enough spare capacity in every region to absorb one region's traffic if it goes down, since that failure mode is what a sub-50ms design actually has to survive.

Structured elaboration

Data placement: give every shard a home (primary) region based on where its data is written most, with replicas in the other two regions. It helps to separate two distinct user populations rather than treating all traffic as one class: continent-scoped ACTIVE users, whose writes and low-latency reads must stay within their home continent, and a separate GLOBAL read-only population (dashboards, admin and support tooling) that can tolerate cross-region reads and does not need the sub-50ms guarantee. Sizing the whole fleet for the tightest requirement when only part of the traffic needs it wastes capacity.

Routing: writes go to the shard's home region only, so there is a single writer per shard and no cross-region write conflicts to resolve. Reads go to the nearest replica of that user's shard, which may lag slightly behind the primary.

Limiting cross-region latency: nothing on the hot read or write path makes a synchronous cross-region call; only the asynchronous replication stream crosses regions.

Consistency: replicas are eventually consistent. Define and expose a staleness bound (a target replication lag and an alert threshold) rather than implying real-time consistency a sub-50ms budget cannot support, and route a user back to the home region for a short window right after their own write so they never see a version older than what they just wrote.

Failover and rebalancing: when a region fails, its shards' home role fails over to whichever surviving replica has the least replication lag (tracked as a first-class metric, not assumed), and a rebalancing job then redistributes some of the newly promoted shards across the remaining regions gradually, rather than concentrating all of the failed region's load onto whichever single region absorbed it first.

Worked example

Because any one of three regions can fail and its load must land on the remaining two:

For n = 3, that works out to 1/2, meaning each surviving region needs 50% extra headroom above its own normal peak to absorb a failed region's traffic without breaching the latency target during the failover window.

Trade-offs and pitfalls

Sizing every region only for its own normal peak, with no failover headroom, means a single region loss cascades into overloading the survivors exactly when the system needs to hold up.

Promoting a replica without tracking per-replica lag risks promoting a stale one and silently losing recent writes; make lag an explicit, monitored number, not an assumption.

Rebalancing after a heal should move shards gradually with rate limiting; moving everything back at once risks a second, self-inflicted overload during recovery.

Incident summary

Autoscaling groups in multiple regions independently observed the same low-traffic lull, for example an overnight trough, and scaled in at roughly the same time, each judging its own regional headroom safe in isolation. When traffic returned, the combined fleet across regions had far less spare capacity than the system as a whole actually needed, and the return of traffic outpaced how fast new instances could come up, producing packet drops and a partial outage.

Probable root causes

• Correlated scale-in decisions: every region used the same signal shape (regional traffic dropping in the same overnight window) and the same aggressive scale-in policy, so all regions acted at once with no coordination or global view of total remaining capacity.
• No cross-region minimum-capacity floor: nothing enforced a fleet-wide minimum instance count; each region's policy was locally correct but globally unsafe.
• Asymmetric reaction speed: removing an instance is nearly instant, while adding one on the return of traffic requires boot and warm-up time, so the fleet was undersized for exactly the length of that gap.
• Load-balancer connection draining not accounted for: instances mid-scale-in may not have finished draining in-flight connections before traffic rebounded, adding to the drop count on top of pure under-capacity.
• Missing headroom alerting: nothing paged when aggregate spare capacity across all regions crossed an unsafe floor, so the risky state existed with no human visibility before impact.

Immediate mitigations

• Manually raise the minimum instance count in each affected region's autoscaling group to restore headroom immediately.
• Temporarily disable or widen scale-in thresholds fleet-wide until the root cause is understood, trading cost for safety.
• Shift traffic away from disproportionately degraded regions via the global load-balancing or DNS layer to protect overall availability while capacity recovers.

Architectural changes

• Enforce a global minimum-capacity floor, sized to the largest region's expected peak rather than today's regional trough, that no individual region's autoscaler can go below regardless of its own local metrics.
• Stagger or jitter scale-in decisions across regions, for example a small random delay or a rolling scale-in that removes capacity from at most one region at a time, so a correlated lull cannot remove headroom everywhere simultaneously.
• Make scale-in more conservative than scale-out by design: longer cooldowns and smaller step sizes on the way down than on the way up, since under-provisioning is far more costly than a few extra idle instances.
• Add pre-emptive scale-out ahead of the historically known return-of-traffic time, rather than relying purely on reactive scaling to catch the rebound in time.

Process changes

• Add an alert on aggregate cross-region spare capacity, not just per-region, so a global headroom deficit pages someone before traffic returns.
• Require any change to scale-in aggressiveness to go through a review that explicitly considers cross-region correlation, since this incident's root cause was a policy that was locally reasonable but globally unsafe.
• Add simultaneous multi-region scale-in as an explicit game-day (a scheduled exercise where a failure is deliberately triggered to rehearse the response) or chaos-test (automated, randomized failure injection) scenario, so this failure mode is exercised deliberately instead of discovered in production.

DNS-based load balancing

A DNS service (for example one offering latency-based or geo routing) returns different backend IPs depending on where the query came from. Failover time is bounded by the record's TTL plus however long individual clients and resolvers actually honor that TTL, which in practice can stretch to several minutes because some resolvers and clients cache longer than the TTL specifies. Granularity is coarse, decisions are made per DNS query, typically at the level of a whole geographic region, and generally without real-time knowledge of backend health unless health checks are explicitly wired into the DNS records. Operational complexity is low: there is no data plane to run, just DNS record configuration.

Anycast routing

The same IP address is announced from multiple physical network locations via BGP (Border Gateway Protocol, the protocol routers use across the internet to agree on the best path to a given address), and network routers deliver a client's packets to whichever announcement is topologically nearest. Failover happens at the network layer, typically on the order of seconds once BGP reconverges, since there is no DNS caching involved at all, a client already has the one IP and the network simply routes it differently. Granularity is based on network path proximity, not real backend health or true user-experienced latency, and BGP route flaps can themselves cause instability. Operational complexity is high if you run it yourself, you need your own autonomous system (a network under one organization's control, the unit that runs BGP) and BGP peering, which is why anycast is usually consumed via a CDN or cloud provider rather than self-operated.

Cloud provider global load balancers and CDNs

These are often anycast at the network edge, but layered with real health checking and TLS termination at each edge point of presence (a physical facility where the provider's network connects to the wider internet), then carried over the provider's private backbone to a healthy backend region. Failover can be sub-minute because it is driven by real-time health checks at the edge rather than DNS caching. Granularity can combine geography, real-time health, and latency simultaneously. Operational complexity is medium: it is a managed service, but comes with real cost and some amount of vendor lock-in to that provider's edge network.

Where each is appropriate

DNS-based routing suits a small team running a multi-region service who wants simple, cheap geographic distribution and can tolerate a failover time measured in minutes, for example splitting traffic between two regions by latency-based DNS records. Anycast fits use cases needing very fast, network-level failover and DDoS resilience without owning BGP infrastructure yourself, which is why it underlies most large CDNs and public DNS resolver services. A cloud provider's global load balancer or CDN is the right choice when you want health-check-driven failover in well under a minute plus fine-grained routing, and are comfortable paying for and depending on that provider's edge network, for example a latency-sensitive API or a real-time application that cannot tolerate DNS-caching-driven failover delay.

Why service discovery matters alongside load balancing

In a microservices architecture, the set of healthy backend instances for a given service changes constantly as instances scale, deploy, and fail. Service discovery is the mechanism by which a caller (or the load balancer in front of it) learns the current, correct set of addresses to route to, instead of relying on a static, stale list.

DNS-based discovery

Instances register into a DNS record, often an SRV or A record, sometimes via a sidecar agent (a small helper process that runs alongside the main service instance and handles supporting work, like registration, on its behalf) that manages registration for them. Callers resolve that DNS name, either per call or cached for a TTL. This is the simplest pattern and works with almost any existing tooling, but its main weakness is staleness: the DNS TTL and the actual rate of instance churn rarely match perfectly, a caller can hold a cached address for an instance that has already been terminated. Shortening the TTL helps but increases query volume, and does not fully solve it since many resolvers and clients do not strictly honor very short or zero TTLs.

Client-side discovery with a registry (Consul, Eureka)

Instances register themselves directly with a registry and send periodic heartbeats to prove they are still alive. Each client (often via a client library or local agent) queries the registry for the current healthy instance list and does its own load-balancing among them, for example round robin over whatever the registry currently returns. Deregistration happens two ways: an explicit deregister call on graceful shutdown, plus a heartbeat lease that expires if the instance disappears without a clean shutdown, for example a heartbeat renewed every 30 seconds with eviction after 90 seconds of silence is a common pattern, giving you a backstop against ungraceful failures. There is still a window of staleness between a client's periodic refresh of its cached list and the registry's actual current state, but it is typically much tighter than DNS TTL staleness.

Server-side discovery with a gateway or service mesh

A central gateway or the mesh's own sidecar owns the lookup: the calling service just talks to a fixed local address (the gateway or its own sidecar), and that component queries the registry and routes on the caller's behalf. Registration works the same way as client-side discovery, but the calling code carries no discovery logic or client library at all, which matters most in a polyglot fleet where you would otherwise need a discovery client for every language in use.

Comparing the three

DNS-based discovery is the simplest and most broadly compatible, at the cost of the weakest freshness guarantee. Client-side discovery with a registry gives the most precise, near-real-time membership view and lets each client apply smarter load-balancing logic, at the cost of a client library dependency in every service. Server-side discovery centralizes that complexity into shared infrastructure, which is the best fit for a large, polyglot fleet, but adds an extra network hop and a new piece of infrastructure that itself needs to be highly available.

Approach

Combine CPU, memory, network throughput, and a custom metric (active sessions) into one weighted composite score, then base the scaling decision on a smoothed (EMA, exponential moving average, a running average that weights recent samples more heavily and damps out single-sample noise) version of that score rather than the raw instantaneous value, with a dead zone around the target and a cooldown after any action. Each of those three mechanisms defends against a different flapping cause: EMA smooths per-sample noise, the dead zone stops the decision from firing on small, harmless deviations, and cooldown stops repeated actions from stacking before the previous action's effect has actually shown up in the metrics.

import random

class MetricAutoscaler:
    def __init__(self, target_score=0.60, weights=None, ema_alpha=0.3,
                 dead_zone=0.10, cooldown_ticks=3, max_step=2):
        self.weights = weights or {"cpu": 0.4, "mem": 0.2, "net": 0.2, "sessions": 0.2}
        self.target_score = target_score
        self.ema_alpha = ema_alpha
        self.dead_zone = dead_zone
        self.cooldown_ticks = cooldown_ticks
        self.max_step = max_step
        self.ema_score = None
        self.cooldown_remaining = 0
        self.instances = 4
        self.history = []

    def normalize(self, raw):
        cpu = raw["cpu_pct"] / 100.0
        mem = raw["mem_pct"] / 100.0
        net = raw["net_pct"] / 100.0
        sessions = min(raw["sessions"] / raw["sessions_capacity"], 1.5)
        return {"cpu": cpu, "mem": mem, "net": net, "sessions": sessions}

    def composite_score(self, norm):
        return sum(self.weights[k] * norm[k] for k in self.weights)

    def tick(self, raw_metrics):
        norm = self.normalize(raw_metrics)
        raw_score = self.composite_score(norm)
        if self.ema_score is None:
            self.ema_score = raw_score
        else:
            self.ema_score = self.ema_alpha * raw_score + (1 - self.ema_alpha) * self.ema_score

        action, delta = "hold", 0
        if self.cooldown_remaining > 0:
            self.cooldown_remaining -= 1
        else:
            deviation = self.ema_score - self.target_score
            if deviation > self.dead_zone:
                delta = min(self.max_step, 1 + int(deviation / self.dead_zone))
                action = "scale_out"
            elif deviation < -self.dead_zone:
                delta = -1
                action = "scale_in"
            if action != "hold":
                self.instances = max(2, self.instances + delta)
                self.cooldown_remaining = self.cooldown_ticks

        self.history.append({"raw_score": round(raw_score, 3), "ema_score": round(self.ema_score, 3),
                              "action": action, "instances": self.instances})
        return self.history[-1]

def make_noisy_metrics(base_cpu, base_mem, base_net, base_sessions, sessions_capacity, rng):
    return {
        "cpu_pct": max(0, base_cpu + rng.gauss(0, 8)),
        "mem_pct": max(0, base_mem + rng.gauss(0, 5)),
        "net_pct": max(0, base_net + rng.gauss(0, 10)),
        "sessions": max(0, base_sessions + rng.gauss(0, sessions_capacity * 0.05)),
        "sessions_capacity": sessions_capacity,
    }

if __name__ == "__main__":
    rng = random.Random(1234)
    scaler = MetricAutoscaler()
    SESSIONS_CAP = 1000
    base_loads = [50 + min(35, t * 3) if t <= 12 else 50 + max(0, 35 - (t - 12) * 3) for t in range(30)]
    for t, base in enumerate(base_loads):
        metrics = make_noisy_metrics(base, base * 0.8, base * 0.9, SESSIONS_CAP * base / 100, SESSIONS_CAP, rng)
        result = scaler.tick(metrics)
        print(f"t={t:2d} base_load={base:5.1f}% raw={result['raw_score']:.3f} "
              f"ema={result['ema_score']:.3f} action={result['action']:9s} instances={result['instances']}")
    actions_taken = [h["action"] for h in scaler.history if h["action"] != "hold"]
    print(f"Total scaling actions over {len(scaler.history)} ticks: {len(actions_taken)} -> {actions_taken}")

Output (seeded, re-run twice, identical both times):

t= 0 base_load= 50.0% raw=0.546 ema=0.546 action=hold      instances=4
t= 1 base_load= 53.0% raw=0.525 ema=0.540 action=hold      instances=4
t= 2 base_load= 56.0% raw=0.560 ema=0.546 action=hold      instances=4
t= 3 base_load= 59.0% raw=0.528 ema=0.541 action=hold      instances=4
t= 4 base_load= 62.0% raw=0.618 ema=0.564 action=hold      instances=4
t= 5 base_load= 65.0% raw=0.597 ema=0.574 action=hold      instances=4
t= 6 base_load= 68.0% raw=0.608 ema=0.584 action=hold      instances=4
t= 7 base_load= 71.0% raw=0.602 ema=0.589 action=hold      instances=4
t= 8 base_load= 74.0% raw=0.699 ema=0.622 action=hold      instances=4
t= 9 base_load= 77.0% raw=0.774 ema=0.668 action=hold      instances=4
t=10 base_load= 80.0% raw=0.752 ema=0.693 action=hold      instances=4
t=11 base_load= 83.0% raw=0.860 ema=0.743 action=scale_out instances=6
t=12 base_load= 85.0% raw=0.802 ema=0.761 action=hold      instances=6
t=13 base_load= 82.0% raw=0.796 ema=0.771 action=hold      instances=6
t=14 base_load= 79.0% raw=0.845 ema=0.794 action=hold      instances=6
t=15 base_load= 76.0% raw=0.744 ema=0.779 action=scale_out instances=8
t=16 base_load= 73.0% raw=0.684 ema=0.750 action=hold      instances=8
t=17 base_load= 70.0% raw=0.640 ema=0.717 action=hold      instances=8
t=18 base_load= 67.0% raw=0.646 ema=0.696 action=hold      instances=8
t=19 base_load= 64.0% raw=0.548 ema=0.652 action=hold      instances=8
t=20 base_load= 61.0% raw=0.489 ema=0.603 action=hold      instances=8
t=21 base_load= 58.0% raw=0.529 ema=0.581 action=hold      instances=8
t=22 base_load= 55.0% raw=0.571 ema=0.578 action=hold      instances=8
t=23 base_load= 52.0% raw=0.455 ema=0.541 action=hold      instances=8
t=24 base_load= 50.0% raw=0.465 ema=0.518 action=hold      instances=8
t=25 base_load= 50.0% raw=0.478 ema=0.506 action=hold      instances=8
t=26 base_load= 50.0% raw=0.478 ema=0.498 action=scale_in  instances=7
t=27 base_load= 50.0% raw=0.520 ema=0.505 action=hold      instances=7
t=28 base_load= 50.0% raw=0.394 ema=0.471 action=hold      instances=7
t=29 base_load= 50.0% raw=0.412 ema=0.454 action=hold      instances=7
Total scaling actions over 30 ticks: 3 -> ['scale_out', 'scale_out', 'scale_in']

Key points

• Notice ticks 8 through 10: the raw score jumps around a lot (0.699, 0.774, 0.752) well above the 0.60 target, but no action fires until t=11, because the EMA is still catching up and the dead zone (target plus or minus 0.10) has not yet been cleared by the smoothed value. That gap is exactly what prevents flapping on noisy samples.
• Only 3 scaling actions happen across 30 ticks despite the raw score crossing the target repeatedly, that ratio (few actions relative to how often the raw signal alone would suggest acting) is a reasonable rough health check for a real deployment: if your actual autoscaler's action count per hour is close to its raw-signal-crossing count, your smoothing or dead zone is too weak.
• The scale-out step size grows with how far the EMA is past the dead zone (1 + int(deviation / dead_zone), capped at max_step), so a large sustained spike scales out faster than a marginal one, while scale-in always moves by exactly 1, deliberately more conservative than scale-out to avoid overshooting into under-capacity.

Complexity

O(1) work per tick (fixed number of metrics, fixed weight lookup), independent of fleet size; this scales fine even at very high tick frequency.

Failure modes this mitigates

• Flapping on noisy single samples: EMA plus dead zone.
• Runaway scaling from a sustained real spike: capped max_step limits how much a single decision can add.
• Rapid oscillation right after an action, before its effect shows up in the metrics: cooldown.
• One metric spiking alone (say, a network blip) dominating the decision: the weighted composite means no single metric alone can trigger action unless its movement is large enough to shift the blended score past the dead zone.

Edge cases

Cold start (no prior EMA) initializes the EMA to the first raw sample rather than an arbitrary default, so the very first decision is not biased by a warm-up value. A metric that goes missing entirely (a scrape failure) is not handled by this minimal version, in production that needs an explicit fallback (hold action, or fail safe to a conservative default) rather than silently treating a missing value as zero, which would look like low load and wrongly trigger scale-in.