Infrastructure Scaling, Capacity Planning, and High Availability Questions

Definitions first

An SLO (service level objective) is the internal performance target a team holds itself to, here 99.9 percent of requests finishing within 200ms. An SLA (service level agreement) is the external, usually contractual, commitment made to customers, and is typically set looser than the internal SLO so there is margin to catch problems before a customer-facing breach. The error budget is simply 100 percent minus the SLO: at 99.9 percent, you have a 0.1 percent budget of requests allowed to miss the 200ms target (or be otherwise unreliable) over the measurement window, usually a rolling 28 or 30 days.

Translating the SLO into a capacity target

The key idea is that latency does not degrade linearly as load increases, it stays flat for a while and then rises sharply as a system approaches saturation, because requests start queueing behind each other rather than being served immediately. So provisioning capacity right at your expected peak load is dangerous: any small spike above that peak pushes you past the point where latency spikes, which is exactly the region where you'd blow through a 200ms target.

Concretely: suppose your service handles a peak of 10,000 requests per second on a normal day. If you load test this service and find that p99.9 latency (the worst latency experienced by the slowest 0.1 percent of requests, which is exactly the population your SLO is protecting) crosses 200ms once sustained throughput passes about 16,000 requests per second, then your real usable capacity ceiling for this SLO is 16,000 req/s, not whatever the hardware can theoretically push. You then provision so your normal peak (10,000) sits comfortably below that ceiling, leaving roughly 60 percent headroom rather than running near the edge where any burst causes an SLO breach.

Accounting for traffic variability

Do not size capacity off average load; size it off a high percentile of your actual demand distribution (for example the busiest 5 minutes of your busiest day in recent history), since that is the load level your capacity actually has to survive. If traffic has a known daily or weekly pattern, use the peak of that pattern plus a buffer for unplanned spikes, not the daily average.

Using the error budget

The 0.1 percent error budget is a spending account, not just a compliance number. If you are consuming it quickly (frequent near-breaches under normal load), that is a signal you are under-provisioned or your safety margin is too thin, and it should trigger tightening capacity or investigating a regression before it becomes a real SLA breach. If the budget is barely touched, that is a signal you may have room to take calculated risk elsewhere, for example a planned maintenance window or a riskier deploy, without endangering the target.

Deciding which requests to prioritize

Classify requests into priority tiers before an incident, not during one, for example critical (checkout, authentication) above standard (browsing, search) above low-priority (background sync, analytics beacons, non-critical batch calls). Under shedding, drop from the bottom tier upward. Within a tier, prefer shedding new or low-value work over work already in flight, since a request that is nearly done represents sunk cost, and shedding it wastes the resources it already consumed.

What triggers shedding

Tie the trigger to the same signal the autoscaler or capacity plan already tracks, rather than inventing a separate one: for example, utilization or queue depth crossing a threshold that would also have triggered scale-out, or a scale-out that has already hit its configured maximum and still cannot keep up, which is the real signal that capacity itself, not just current instance count, is the constraint. Combine that with a hard signal like request queue wait time exceeding an SLA-derived (service-level-agreement-derived) threshold, since utilization alone can look acceptable while queueing delay has already blown past acceptable latency.

Mechanics

• Shed as close to the edge as possible, at the load balancer or API gateway, so shed requests never consume backend resources at all; shedding deep in the stack after work has started wastes the resources you were trying to protect.
• Return a distinct response, for example an HTTP 503 status with a Retry-After header, rather than a generic error, so well-behaved clients back off instead of retrying immediately and making the problem worse, sometimes called a retry storm.

Communicating the degraded state to clients

For internal or API clients, use a standard status code and a machine-readable reason so client code can distinguish "shed, try later" from "your request was invalid." For end users, surface a clear, honest degraded-mode message, for example "search temporarily limited, checkout still available," rather than a generic error, since silent degradation erodes trust more than an honest one.

Rolling shedding back safely

Do not flip from fully shedding a tier to fully restored the instant the trigger metric dips below threshold. Ramp back gradually, for example restoring a quarter of shed capacity, watching for a few minutes, then continuing, to avoid immediately re-triggering the same overload, especially since recovery itself often creates a burst of retried or queued client requests. Use a cooldown window, requiring the healthy signal to hold for a few minutes, before considering a tier fully restored, mirroring the same anti-flapping logic you would want in the autoscaler itself.

What I would validate

Load-test the shedding logic itself under the exact overload condition it is meant to handle. A shedding policy that has never been exercised is a policy you are finding out about for the first time during a real incident.

Vertical vs horizontal for GPU-served models

Vertical scaling here means moving to a bigger or more capable single GPU (or a tightly coupled multi-GPU node with fast interconnect between the GPUs). Horizontal scaling means running more, smaller GPU instances as independent replicas behind a load balancer, each serving its own share of requests.

Latency SLOs (service level objectives, the internal performance targets a team holds itself to)

If a single request must return within a strict latency budget, keeping the model entirely on one GPU avoids the cross-device communication overhead that model or tensor parallelism (splitting a model's computation across multiple GPUs) introduces. For a strict per-request latency target, vertical scaling (a single capable GPU handling the full forward pass) tends to win, since it avoids that extra communication hop.

Batching efficiency

Batching efficiency actually favors consolidation, the same direction as the latency-SLO argument above, though for a different reason. Dynamic batching groups requests that arrive within a short window into one GPU forward pass, and how useful that batch is depends on how full it gets before the window closes, which depends on how many concurrent requests land at that one batching queue. Pooling all traffic onto fewer, larger GPU instances (or fewer replicas that each carry more traffic) means each batching queue sees a larger share of the aggregate request rate, so batches fill faster and fuller, raising GPU utilization per request served. Splitting the same total traffic across many independent horizontal replicas divides that request rate by the replica count, so each replica's own batching queue sees proportionally less traffic and fills batches more slowly and thinner, unless the per-replica traffic is still high enough on its own to keep batches full, which only holds once total volume is very large relative to replica count. So vertical, or fewer-and-larger, scaling tends to win on raw batching efficiency for a given total request volume, not horizontal. Horizontal scaling's real batching-relevant value is different: it adds independent execution capacity once a single node's compute is already saturated even with maximally full batches, which is a raw-throughput-ceiling argument rather than a batching-efficiency one, and it gives more, smaller, independently tunable batching pools you can route specific traffic segments to, for example isolating a latency-sensitive feature's batching pool from a throughput-tolerant one, which is a routing-flexibility win rather than a per-request GPU-utilization win.

GPU memory limits and licensing

If a model's parameters simply do not fit in one GPU's memory, vertical scaling (a bigger GPU or a multi-GPU node with fast interconnect) is not a preference, it is a requirement; there is no horizontal alternative to fitting a model that needs more memory than a single device provides. Some accelerator or software licensing structures also charge per node rather than per GPU, which can favor consolidating onto fewer, larger nodes to minimize licensing overhead.

Failure isolation

Horizontal scaling isolates failure to whichever fraction of total replicas is affected, losing one of many independent replicas degrades capacity without taking the whole service down. A single large vertically-scaled node concentrates the entire serving capacity into one failure domain: losing it takes down all serving capacity behind it at once. This is one of the strongest arguments for horizontal scaling whenever the model fits comfortably on a smaller GPU, since the failure-isolation benefit is essentially free.

Scaling elasticity

Horizontal scaling tracks demand naturally: add or remove replicas as traffic changes, usually within seconds to a couple of minutes. Vertical scaling (resizing a GPU instance) commonly requires a restart or redeploy and is not something you do reactively in real time, so it responds to demand changes far more slowly.

Where each wins, concretely

• A small-to-medium transformer model with a high query volume and a hard per-request latency budget, where the model fits comfortably on a single mid-size GPU: horizontal scaling wins, since failure isolation and elasticity dominate and the model has no memory constraint forcing a bigger node.
• A very large model whose parameters exceed a single GPU's memory: vertical scaling (or a hybrid, using model parallelism across a few large, tightly interconnected GPUs within one node) wins by necessity, regardless of the failure-isolation and elasticity trade-offs it accepts, because there is no way to serve the model at all without it.

What "probability of breach" actually requires

A single point forecast, "we'll need 100,000 units of capacity next month," hides the fact that growth rate, incident frequency, and how fast you can add capacity are all uncertain, not fixed numbers. Missing the SLA (service level agreement, the contractual target you're sized against) even once counts as a breach, so what leadership actually wants is a probability, not a guess. A Monte Carlo simulation, running the same calculation thousands of times, each time drawing a random but plausible value for every uncertain input, to see how often the outcome crosses a line you care about, turns that uncertainty into an actual number.

Building the model

Model monthly effective demand as the point forecast scaled by a random growth rate, and available capacity as the point forecast plus a candidate buffer, reduced by two further random effects: incident-driven capacity loss (some months have outages or degraded nodes eating into usable capacity) and a provisioning-lead-time penalty (if new capacity can't land before it's needed, the effective buffer that month is smaller than the number on paper). Reasonable input distributions: growth rate as a normal distribution centered on your historical trend with a standard deviation from historical month-to-month volatility; incident count from a Poisson-like process (models random, independent events happening at a steady average rate, appropriate for how many incidents occur in a given month) based on historical incident frequency, each incident consuming some percentage of capacity; provisioning lead time as a bounded distribution, for example triangular between your fastest and slowest historical procurement or scale-out time. For each of N random draws, check whether demand exceeds available capacity; the fraction of draws where it does is the estimated breach probability for that buffer size.

Sample count

Because the target, 0.1% per month, is 1 in 1,000, you need enough samples that a single unlucky or lucky draw doesn't swing the estimate. As a rule of thumb, aim for at least one to two orders of magnitude more samples than the inverse of the probability you're resolving, so tens of thousands to a few hundred thousand samples for a 0.1% target, not the few hundred that would be fine for estimating a 50% probability.

Worked run (toy numbers, to show the mechanics)

Running exactly this model, 200,000 samples per buffer size, growth centered at 3% with 4% standard deviation, incident-driven loss and lead-time penalty as described above, against a range of buffer sizes gives a monotonically decreasing breach-probability curve:

buffer= 5.0%  breach_probability=50.213%
buffer=10.0%  breach_probability=15.703%
buffer=15.0%  breach_probability= 3.682%
buffer=20.0%  breach_probability= 0.892%
buffer=25.0%  breach_probability= 0.236%
buffer=30.0%  breach_probability= 0.066%   <- first buffer under the 0.1% target

The exact percentages are specific to these toy input distributions, but the shape is the real lesson: breach probability falls off steeply, not linearly, as you add buffer, so the right buffer is wherever your own curve crosses your own risk tolerance, not a round number picked in advance.

Alternative to Monte Carlo

Without simulation infrastructure handy, a normal-approximation shortcut gives a fast sanity check: if effective demand is roughly normally distributed with a known mean and standard deviation, a 0.1% one-sided breach tolerance corresponds to roughly 3.1 standard deviations above the mean, from the standard normal distribution's tail, so buffer is approximately 3.1 times the standard deviation of monthly demand. This misses the fatter tails that incident spikes and lead-time risk actually add, so treat it as a cross-check against the simulation's answer, not a replacement for it.

Turning this into a headroom recommendation

Present leadership the curve, not a single number: buffer size on one axis, breach probability on the other, with their 0.1% target marked as one point on that curve. That framing makes the trade-off explicit, more buffer costs more and sits idle most months, and makes clear the 0.1% figure is a choice they're making, not a fact you're reporting.

Approach

The scheduler needs two mechanisms working together: best-fit bin-packing (placing each task on the node that leaves the least unused capacity, so free space stays consolidated rather than scattered thin across every node) for the common case where a task fits somewhere, and priority-based preemption for the case where it doesn't. Tasks are processed in arrival order, not sorted by priority up front, because real clusters receive work over time, not as one static batch; that ordering choice is what actually makes preemption possible, since a lower-priority task has to already be placed before a later, higher-priority task can evict it.

from dataclasses import dataclass, field

@dataclass
class Node:
    name: str
    cpu_total: float
    mem_total: float
    gpu_total: int
    cpu_used: float = 0.0
    mem_used: float = 0.0
    gpu_used: int = 0
    placed: list = field(default_factory=list)

    def free(self):
        return (self.cpu_total - self.cpu_used, self.mem_total - self.mem_used,
                self.gpu_total - self.gpu_used)

    def fits(self, task):
        fc, fm, fg = self.free()
        return fc >= task.cpu and fm >= task.mem and fg >= task.gpu

    def place(self, task):
        self.cpu_used += task.cpu
        self.mem_used += task.mem
        self.gpu_used += task.gpu
        self.placed.append(task.name)

@dataclass
class Task:
    name: str
    cls: str       # "latency" | "batch" | "ml"
    priority: int   # higher = more important, used for preemption
    cpu: float
    mem: float
    gpu: int = 0

def best_fit_score(node, task):
    # Prefer the node left with the LEAST leftover capacity (tightest fit),
    # to keep packing dense and preserve large free blocks elsewhere.
    fc, fm, fg = node.free()
    return (fc - task.cpu) + (fm - task.mem) + (fg - task.gpu) * 10

def find_task(tasks, name):
    return next(t for t in tasks if t.name == name)

def schedule(nodes, tasks):
    evicted, unschedulable = [], []
    for task in tasks:
        candidates = [n for n in nodes if n.fits(task)]
        if candidates:
            best = min(candidates, key=lambda n: best_fit_score(n, task))
            best.place(task)
            print(f"PLACE  {task.name} -> {best.name}")
            continue

        preempted = False
        for node in nodes:
            victims = sorted(
                [t for t in node.placed if find_task(tasks, t).priority < task.priority],
                key=lambda tn: find_task(tasks, tn).priority)
            # Evict victims cumulatively, lowest priority first, checking fit after
            # each one -- a task that needs the freed space of MORE than one victim
            # must still be placeable. If even evicting every eligible victim on this
            # node is not enough, undo the whole batch and try the next node.
            removed = []
            for vname in victims:
                v = find_task(tasks, vname)
                node.cpu_used -= v.cpu; node.mem_used -= v.mem; node.gpu_used -= v.gpu
                node.placed.remove(vname)
                removed.append(vname)
                if node.fits(task):
                    node.place(task)
                    evicted.extend(removed)
                    for rv in removed:
                        print(f"PREEMPT {rv} on {node.name} to fit {task.name} "
                              f"(prio {task.priority} > {find_task(tasks, rv).priority})")
                    preempted = True
                    break
            if preempted:
                break
            for vname in removed:
                v = find_task(tasks, vname)
                node.cpu_used += v.cpu; node.mem_used += v.mem; node.gpu_used += v.gpu
                node.placed.append(vname)
        if not preempted:
            unschedulable.append(task.name)
            print(f"UNSCHEDULABLE {task.name}")
    return evicted, unschedulable

if __name__ == "__main__":
    nodes = [Node("node-a-gpu", 16, 64, 1), Node("node-b-cpu", 16, 64, 0)]
    tasks = [
        Task("batch-1", "batch", 10, cpu=10, mem=40),
        Task("batch-2", "batch", 10, cpu=10, mem=40),
        Task("train-1", "ml", 50, cpu=8, mem=32, gpu=1),
        Task("web-1", "latency", 100, cpu=4, mem=16),
        Task("web-2", "latency", 101, cpu=10, mem=40),
    ]
    evicted, unschedulable = schedule(nodes, tasks)
    for n in nodes:
        print(n.name, "placed=", n.placed, "free=", n.free())
    print("evicted:", evicted, "unschedulable:", unschedulable)

Output (deterministic, no randomness involved):

PLACE  batch-1 -> node-b-cpu
PLACE  batch-2 -> node-a-gpu
PREEMPT batch-2 on node-a-gpu to fit train-1 (prio 50 > 10)
PLACE  web-1 -> node-b-cpu
PREEMPT train-1 on node-a-gpu to fit web-2 (prio 101 > 50)
node-a-gpu placed= ['web-2'] free= (6.0, 24.0, 1)
node-b-cpu placed= ['batch-1', 'web-1'] free= (2.0, 8.0, 0)
evicted: ['batch-2', 'train-1'] unschedulable: []

Key points

• Best-fit bin-packing minimizes leftover fragmentation per placement, but is evaluated fresh for every arriving task, so packing decisions adapt as the cluster fills up.
• Preemption only ever evicts a strictly lower-priority task, and only enough of them, one at a time, checking fit after each eviction, to fit the new arrival, never more capacity than necessary.
• This eviction is cumulative: the loop removes one victim, rechecks fit, and keeps removing the next-lowest-priority victim if that alone was not enough, rather than giving up after a single victim. Tested against an adversarial case (a node with 2 free cpu, two priority-5 victims using 3 cpu each, and an incoming priority-10 task needing 6 cpu, where evicting either victim alone only frees 3 cpu but evicting both frees 6): the loop correctly evicts both and places the task, instead of wrongly reporting it unschedulable.
• The run above shows a realistic priority tier in action: the batch job is evicted to make room for the ML training job, and the training job is later evicted to make room for a latency-critical service, matching the intended priority ordering (latency over ML over batch) even though neither eviction was hand-scripted, it falls out of the priority comparison.
• In production, an evicted task is re-queued for a fresh placement attempt (potentially on a different node, or triggering a cluster-autoscaler scale-up if nothing fits anywhere), rather than simply discarded as this simplified demo does.

Complexity

For each of T tasks, evaluating all N nodes for best fit is O(N); the preemption path in the worst case checks every node and, per node, sorts and tries each lower-priority victim, O(N * K log K) where K is placed tasks per node. For cluster-scale scheduling this is why real schedulers batch and cache candidate node lists rather than doing a full re-scan per pod.

Edge cases

• A task whose resource request exceeds any single node's total capacity is unschedulable regardless of preemption, since no eviction can create capacity that doesn't exist.
• Two equal-priority tasks: the code's strict less-than comparison means neither can preempt the other, matching the usual policy that equal priority does not authorize eviction.
• GPU-only tasks are correctly blocked from CPU-only nodes by the fits check regardless of how much CPU and memory is free there.