Infrastructure Scaling, Capacity Planning, and High Availability Questions

Basis and demand

200 pods, each requesting 0.5 vCPU and 1 GiB memory:

• Total requested CPU: 200 x 0.5 = 100 vCPU
• Total requested memory: 200 x 1 = 200 GiB

Apply the 20% bin-packing buffer to demand (real clusters never tile perfectly onto node boundaries, some capacity is always stranded):

• Buffered CPU: 100 x 1.2 = 120 vCPU
• Buffered memory: 200 x 1.2 = 240 GiB

Picking a node type

The pod's own resource ratio is 0.5 CPU : 1 GiB, that is 1:2. Matching the node type's CPU:memory ratio to that reduces stranded capacity on whichever resource isn't the bottleneck. A compute-optimized instance like an 8 vCPU / 16 GiB node has a raw 1:2 ratio, an exact match, versus a general-purpose 8 vCPU / 32 GiB node at 1:4, which would strand memory.

Every node also loses some capacity to overhead before pods can use it: kubelet and OS system reservation (roughly 10% of CPU and a chunk of memory on a mid-size node, this varies by cloud and AMI, always confirm with kubectl describe node on the real node type) plus per-node DaemonSets (CNI, kube-proxy, log/metrics agents, typically a few hundred millicores and a few hundred MiB). For an 8 vCPU / 16 GiB node, a reasonable planning estimate for USABLE allocatable is about 7 vCPU and about 13 GiB per node.

Node count

• By CPU: 120 / 7 = 17.1, round up to 18 nodes.
• By memory: 240 / 13 = 18.5, round up to 19 nodes.

Memory is the binding constraint here (barely), so plan for 19 nodes. Sanity check: 19 x 7 = 133 vCPU (>= 120 needed) and 19 x 13 = 247 GiB (>= 240 needed), both hold with only modest slack, confirming the node type was a good ratio match rather than badly stranding one resource.

Spot and mixed node pools

Split into at least two node groups:

• An on-demand baseline, sized to whatever minimum keeps the service available if spot capacity vanished entirely (for example enough on-demand nodes to cover 30-40% of the fleet), used for pods that are stateful, coordination-critical, or otherwise expensive to interrupt.
• A spot burst pool for the bulk of stateless replicas, spread across multiple instance types and AZs (Availability Zones, physically separate, independently-powered sections of a cloud region) so a single spot pool's simultaneous reclaim doesn't take out a large share of capacity at once.

Handle the 2-minute AWS spot interruption notice (or equivalent) with a PodDisruptionBudget and graceful termination hooks so the cluster autoscaler or Karpenter can reschedule affected pods before the instance actually terminates, and let the autoscaler treat spot and on-demand as separate, independently-scaling groups rather than one blended pool.

Framing: two workload shapes sharing one cluster

Nightly ETL (Extract, Transform, Load: the nightly batch data-processing pipeline) is batch and throughput-oriented, tolerant of queuing as long as it finishes inside its window (e.g., before business hours). Daytime dashboards are interactive and latency-sensitive, with a concurrency peak tied to when people actually look at them. Size for both shapes explicitly rather than one blended average.

Peak concurrency

Profile dashboard concurrency separately from ETL's footprint: find the actual busy window (commonly a morning spike as people check dashboards) and size interactive capacity to that peak, not the day's average query rate. ETL's "concurrency" is really about how many parallel tasks it runs within its own window, sized against the deadline it must finish by.

Buffer for failover

Size the cluster so that losing one node still lets ETL finish inside its window AND still serves the dashboard peak within its latency target, an N+1 (or more, depending on cluster size and criticality) node buffer, verified against BOTH constraints, not just checked against average-case load.

A small worked illustration: if ETL needs 8 nodes to finish inside its overnight window and the dashboard peak needs 6 nodes to stay within its latency target, provision to the higher of the two (8) plus an N+1 buffer of 1, a 9-node cluster, so losing any single node still leaves enough capacity to satisfy whichever constraint is binding at that moment.

Storage IOPS

ETL jobs (scans, shuffles) are IOPS (Input/Output Operations Per Second, how many read/write operations storage can handle per second)/throughput-hungry; dashboards are often read-cache-friendly since users repeatedly query similar recent data, so a caching or semantic layer disproportionately reduces the dashboard side's IOPS pressure. Provision for the higher of the two peaks if the workloads are reliably time-separated; provision for their SUM if there's real risk of ETL running late into business hours, that overlap risk is exactly the kind of thing a failover buffer should also protect against.

Cost forecasting

Track ETL cost and dashboard cost as separate lines, since they scale on different drivers: ETL scales with data volume processed per night, dashboards scale with active users and query volume. A single blended cost-per-node number hides which driver is actually pushing costs up.

Modeling growth, and add-nodes vs. optimize-jobs

Track data volume growth (for ETL) and query/user growth (for dashboards) as separate trend lines feeding the same node-count model used elsewhere in capacity planning (translate the trend into required throughput, then into node count with a safety margin). Before adding nodes, check whether the workload is inefficient rather than genuinely resource-bound: for illustration, a nightly job that scans a full multi-terabyte table when it only needs the last day's partition is a common and expensive pattern, partitioning or an incremental-processing rewrite can cut scanned data by an order of magnitude for a fraction of the cost of new hardware. A simple decision rule: if a job-level optimization would pay for the engineering time within a reasonable number of months, do that first; only scale out once you've confirmed the workload is genuinely resource-bound rather than inefficiently written.

Definitions first

An SLO (service level objective) is the internal performance target a team holds itself to, here 99.9 percent of requests finishing within 200ms. An SLA (service level agreement) is the external, usually contractual, commitment made to customers, and is typically set looser than the internal SLO so there is margin to catch problems before a customer-facing breach. The error budget is simply 100 percent minus the SLO: at 99.9 percent, you have a 0.1 percent budget of requests allowed to miss the 200ms target (or be otherwise unreliable) over the measurement window, usually a rolling 28 or 30 days.

Translating the SLO into a capacity target

The key idea is that latency does not degrade linearly as load increases, it stays flat for a while and then rises sharply as a system approaches saturation, because requests start queueing behind each other rather than being served immediately. So provisioning capacity right at your expected peak load is dangerous: any small spike above that peak pushes you past the point where latency spikes, which is exactly the region where you'd blow through a 200ms target.

Concretely: suppose your service handles a peak of 10,000 requests per second on a normal day. If you load test this service and find that p99.9 latency (the worst latency experienced by the slowest 0.1 percent of requests, which is exactly the population your SLO is protecting) crosses 200ms once sustained throughput passes about 16,000 requests per second, then your real usable capacity ceiling for this SLO is 16,000 req/s, not whatever the hardware can theoretically push. You then provision so your normal peak (10,000) sits comfortably below that ceiling, leaving roughly 60 percent headroom rather than running near the edge where any burst causes an SLO breach.

Accounting for traffic variability

Do not size capacity off average load; size it off a high percentile of your actual demand distribution (for example the busiest 5 minutes of your busiest day in recent history), since that is the load level your capacity actually has to survive. If traffic has a known daily or weekly pattern, use the peak of that pattern plus a buffer for unplanned spikes, not the daily average.

Using the error budget

The 0.1 percent error budget is a spending account, not just a compliance number. If you are consuming it quickly (frequent near-breaches under normal load), that is a signal you are under-provisioned or your safety margin is too thin, and it should trigger tightening capacity or investigating a regression before it becomes a real SLA breach. If the budget is barely touched, that is a signal you may have room to take calculated risk elsewhere, for example a planned maintenance window or a riskier deploy, without endangering the target.

Framing

The goal is to run most of the fleet on spot or preemptible instances for cost savings, while guaranteeing a minimum reliable capacity floor that never depends on spot availability, so an SLA is never at the mercy of a preemption wave.

The reliable floor

Always run a minimum number of instances on reserved or on-demand capacity, sized to the absolute minimum below which the SLA is breached even momentarily. Spot or preemptible capacity is layered strictly above that floor, never as a substitute for it.

Admission and allocation logic

Request capacity spread across multiple instance types and multiple availability zones simultaneously (a diversified fleet request) rather than bidding on a single pool, since a preemption or capacity crunch typically hits one instance type in one zone at a time, not a well-diversified set all at once. Weight allocation toward pools with deeper available capacity and a lower recent interruption rate over pools that are merely cheapest, since an interruption that forces rescheduling can cost more in wasted, redone work than a modest per-hour price difference saves.

Detecting and replacing lost capacity

Subscribe to the cloud provider's interruption warning signal, but size checkpointing to the SHORTEST notice window you might actually get, not the longest: AWS EC2 Spot gives about a 2-minute warning before reclaiming an instance, while GCP preemptible/Spot VMs and Azure Spot VMs give only about 30 seconds, a 4x difference that matters directly for how much in-flight work you can safely lose. Treat the signal exactly like a scale-in drain event: stop assigning new work immediately, checkpoint whatever is in flight. For long-running batch tasks, checkpoint at a granularity meaningfully smaller than the shortest of those windows, for example every 10 to 15 seconds or every few hundred records processed if you might run on a 30-second-notice provider, not "every few minutes" as a one-size-fits-all number tuned only to AWS's longer window, since a multi-minute checkpoint interval would lose far more in-flight work than a 30-second warning gives you time to save. If the fleet spans more than one provider, design checkpointing for the tightest window across all of them, never the most generous one. If the autoscaler cannot fill a capacity request from any spot pool within a bounded wait, a couple of minutes is a reasonable bound, fall back to launching the shortfall on on-demand capacity rather than silently leaving the reliable floor unmet, and treat that fallback as a visible, monitored cost escape valve, not a silent failure.

Dynamically weighting instance types by price and reliability

Maintain a rolling estimate of interruption rate per instance-type-and-zone pool, combining the provider's own interruption-frequency guidance with your own observed history, and combine it with current price into a cost-per-reliable-unit-of-work score. Bias allocation toward the better-scoring pool rather than the raw cheapest pool, since the cheapest pool is often cheap precisely because it is oversubscribed and close to being reclaimed.

Metrics to catch a cost-versus-reliability regression

• Interruption rate and mean time between interruptions, per pool.
• The fraction of current capacity running on the on-demand fallback path, this should normally sit near zero, a rising trend is an early warning that spot supply is drying up and effective cost is climbing even though the nominal spot price has not changed.
• Effective blended cost per unit of completed work, not just the headline spot price, to catch the case where frequent interruptions quietly erase the nominal discount through repeated, wasted work.
• SLA-relevant metrics (error rate, latency, completion time) segmented by whether serving capacity was on-demand or spot at the time, which can surface a reliability regression that pure infrastructure metrics would miss.

Which workloads belong on spot at all

Not every workload is a good candidate. Batch, offline, and fault-tolerant work (ETL, transcoding, CI runners, checkpointed model training) tolerates an interruption as "resume from checkpoint," making it a strong fit for spot's savings. Latency-sensitive or stateful interactive serving without a checkpointing story does not tolerate it, an interruption there directly hits users, and belongs on the reliable floor instead. Quantify this explicitly rather than by instinct: expected net savings should be the raw discount minus an estimated rework cost, interruption rate multiplied by the average wasted work per interruption, so a workload with a high interruption rate and no cheap way to resume can show negative net savings despite a lower sticker price.

Applying this to a batch pipeline already violating its SLA by 20 percent due to preemptions

This is almost always a checkpointing and retry-granularity problem, not a fundamental case against spot. The fix is fourfold: add fine-grained checkpointing so a preempted worker resumes rather than restarts from the beginning; break one large job into many smaller, idempotent tasks, so a single preemption only loses that one task's progress, not the whole run; add a small on-demand fallback pool sized to the recently observed preemption rate, so the pipeline can absorb the expected interruption load within its SLA rather than being surprised by it; and make the retry and backoff logic preemption-aware, distinguishing "preempted, reschedule elsewhere immediately" from "genuine task failure, back off," so it does not hammer an already-exhausted pool right after every preemption. This restores the SLA while keeping the pipeline predominantly on spot capacity, rather than abandoning the cost savings entirely.

Metrics to track

• Forecast error against actuals, using a scale-independent metric such as MAPE (mean absolute percentage error, the average of |actual minus forecast| divided by actual) to compare across metrics of different scale, alongside a metric that penalizes large misses more heavily, such as RMSE (root mean squared error), since a percentage-only view can hide a few very large misses that matter operationally more than many small ones.
• Bias, the signed average error (forecast minus actual), not just error magnitude, since a model that is consistently 10 percent low is a more dangerous problem than one that is randomly off by plus or minus 10 percent, because it means you under-provision every single time.
• Interval coverage, if the forecast produces a range rather than a single point: what fraction of actuals actually fall inside the stated confidence interval. If a "90 percent confidence" interval only contains the actual value 60 percent of the time, the intervals are miscalibrated even if the point forecast looks fine on average.

Worked trace: for one period, say the forecast was 100 units and the actual came in at 112. That period's absolute percentage error is |112 - 100| / 112 ≈ 10.7%, which is what gets averaged with every other period's percentage error to produce the overall MAPE. The bias for that same period is forecast minus actual, 100 - 112 = -12, a negative bias meaning the forecast under-shot. A single -12 miss is just noise; the dangerous pattern described above is that same negative sign recurring period after period instead of bouncing between positive and negative.

Backtesting and calibration

• Backtest with rolling-origin, or walk-forward, validation: train on data up to time T, forecast the next period or periods, compare against what actually happened, then roll T forward and repeat. A single random train/test split leaks future information into training, since capacity data is a time series.
• Calibrate by tracking the bias metric across multiple rolling windows; if a persistent bias appears, apply a correction factor or retrain with more recent data weighted more heavily, rather than assuming the model is inherently correct.
• Compare against a naive baseline, such as "next period equals this period" or "same period last year plus trend," every time you evaluate a more sophisticated model. If the sophisticated model does not consistently beat the naive baseline, it is not earning its added complexity.

Detecting structural breaks and concept drift

A structural break is a sudden, sustained shift in the underlying pattern, for example a product launch or a pricing change, that makes recent history a poor predictor going forward. Detect it by watching for a sudden, sustained jump in rolling forecast error, not a single noisy day, especially around known business events; a simple rolling-window error threshold that requires several consecutive breaching periods before alerting is often good enough in practice.

Concept drift is a slower, ongoing change in the relationship the model learned, for example seasonality gradually shifting or the growth rate slowly changing, rather than a sudden break. Detect it by watching whether the bias metric trends in one direction across successive backtest windows rather than staying centered near zero; a model with slowly degrading accuracy needs scheduled retraining before it silently crosses into unreliable.

In both cases the practical safeguard is the same: treat the forecast as its own monitored production system, alert when forecast error itself exceeds a threshold, and have a defined retraining or manual-override cadence, rather than waiting for a capacity incident to notice the model went stale.