Infrastructure Scaling and Capacity Planning Questions

Operational and infrastructure level planning to ensure systems meet current demand and projected growth. Topics include forecasting demand headroom planning and three to five year capacity roadmaps; autoscaling policies and metrics driven scaling using central processing unit memory and custom application metrics; load testing benchmarking and performance validation methodologies; cost modeling and right sizing in cloud environments and trade offs between managed services and self hosted solutions; designing non disruptive upgrade and migration strategies; multi region and availability zone deployment strategies and implications for data placement and latency; instrumentation and observability for capacity metrics; and mapping business growth projections into infrastructure acquisition and scaling decisions. Candidates should demonstrate how to translate requirements into capacity plans and how to validate assumptions with experiments and measurements.

HardTechnical

53 practiced

Design and describe an experiment to validate assumptions in a five-year capacity roadmap when you only have six months of historical telemetry. Explain how you will quantify uncertainty, choose priors, run stress tests, use leading indicators, set guardrails, and iterate the roadmap as more data becomes available.

Sample Answer

Situation: We need a defensible five-year capacity roadmap but only have six months of telemetry. The goal: validate assumptions, quantify uncertainty, and produce an iterative plan that becomes more precise as new data arrives.

Approach summary:1) Formalize assumptions into testable priors.2) Use probabilistic forecasting (Bayesian + bootstrap) to quantify uncertainty.3) Run stress and scenario tests to probe nonlinear failure modes.4) Track leading indicators and set operational guardrails (SLOs, error budgets, headroom).5) Automate re-evaluation and iterate roadmap monthly/quarterly.

Detailed plan

Priors and uncertainty- Translate business/technical assumptions into priors (e.g., traffic growth = 40%±15% annual; per-request CPU = 50ms±10ms). Choose priors using domain knowledge: weakly informative Gaussians for metrics with some historical signal, log-normal for multiplicative growth, wide-tailed Student-t where uncertainty is high.- Fit a Bayesian hierarchical model combining the six months of telemetry with priors to get posterior predictive distributions for key drivers (QPS, p95 latency, CPU, memory).- Quantify uncertainty by sampling posteriors (MCMC or variational inference) and producing percentiles (P10, P50, P90, P99) for capacity needs at 1/3/5 years.

Stress tests and scenarios- Define scenarios: steady growth, viral spike (x5 QPS for 24h), gradual degradation (resource leaks), sudden change in user behavior (longer sessions).- Implement chaos/stress tests in staging: traffic replay + synthetic load, large fanout events, network packet loss, node evictions.- For each scenario measure leading indicators (request rate, queue length, CPU steal, tail latency, error rate) and identify nonlinear thresholds where autoscaling or degradation triggers.

Leading indicators & guardrails- Select leading indicators that predict capacity stress: request concurrency, queue depth, CPU utilization per pod, backpressure events, p95/p99 latency, error budget burn rate.- Define guardrails: - SLOs and error budgets (e.g., 99.9% availability monthly). If error budget burn > X%, scale or roll back feature. - Capacity headroom policy: maintain P90 capacity >= expected load + buffer (e.g., 30%). - Autoscaling thresholds tied to leading indicators (not just CPU): queue length and request latency triggers.- Implement monitoring dashboards and alerting (Prometheus/Grafana, Alertmanager) to surface when guardrails approach thresholds.

Execution & automation- Build a pipeline: ingest telemetry -> update Bayesian model -> run Monte Carlo capacity projections -> generate reports and recommended actions. Automate monthly runs; notify stakeholders when P90 capacity gap crosses decision thresholds.- Use canary and A/B deployment to validate per-user resource changes; instrument canaries with fine-grained telemetry.

Iteration and decision rules- Define decision cadence: monthly small updates, quarterly roadmap revision, annual capacity purchases.- Decision-rule examples: - If posterior P90 for year-1 capacity increases >15% vs roadmap => trigger procurement/architectural review. - If stress-tests reveal behavior change (e.g., CPU per request doubles under certain workload) => re-evaluate architecture (sharding, caching).- As new telemetry arrives, posteriors tighten; reduce headroom proportionally and reallocate budget.

What success looks like- Roadmap with probabilistic bands (P50/P90) and actionable triggers.- Early detection via leading indicators prevents SLO breaches.- Measurable reduction in capacity surprises and more efficient capital allocation as uncertainty shrinks over time.

MediumTechnical

69 practiced

Write a Python function or clear pseudocode that calculates the required number of identical instances given the following inputs: target throughput in requests per second, average CPU used per request (cores), and single instance CPU capacity (cores). Include handling for rounding, minimum headroom percentage, and the case where memory is the limiting factor instead of CPU.

Sample Answer

Approach: compute required instances based on throughput and per-request CPU; factor in CPU capacity per instance, add headroom percentage, round up to whole instances, and also compute instances required by memory and take the max. Support using average or p95 CPU per request.

python

import math

def required_instances(target_rps,
                       cpu_per_request,
                       cpu_per_instance,
                       headroom_pct=20,
                       mem_per_request=None,
                       mem_per_instance=None,
                       percentile_label="avg"):
    """
    Calculate required identical instances.
    - target_rps: target requests/sec (int/float)
    - cpu_per_request: CPU cores used per request (avg or p95)
    - cpu_per_instance: CPU cores available per instance
    - headroom_pct: percent extra capacity to reserve (e.g., 20)
    - mem_per_request: memory bytes used per request (optional)
    - mem_per_instance: memory bytes available per instance (optional)
    - percentile_label: descriptive string for logs ("avg" or "p95")
    Returns: dict with cpu_based, mem_based (if applicable), final_instances, notes
    """
    if target_rps <= 0:
        return {"final_instances": 0, "notes": "No traffic"}
    # total CPU cores needed to handle requests concurrently per second:
    total_cpu_needed = target_rps * cpu_per_request
    # apply headroom
    safety_multiplier = 1 + headroom_pct / 100.0
    total_cpu_with_headroom = total_cpu_needed * safety_multiplier
    # instances from CPU (round up)
    instances_cpu = math.ceil(total_cpu_with_headroom / cpu_per_instance)

    note = f"Calculated from {percentile_label} CPU per request."
    result = {"cpu_based": instances_cpu, "notes": note}

    # memory-based calculation (if data provided)
    if mem_per_request is not None and mem_per_instance is not None:
        total_mem_needed = target_rps * mem_per_request
        total_mem_with_headroom = total_mem_needed * safety_multiplier
        instances_mem = math.ceil(total_mem_with_headroom / mem_per_instance)
        result["mem_based"] = instances_mem
        final = max(instances_cpu, instances_mem)
        result["final_instances"] = final
        result["notes"] += " Memory considered; taking max of CPU and memory requirements."
    else:
        result["final_instances"] = instances_cpu
    return result

Key points:- Use percentile (p95) when traffic spike safety is needed.- Headroom protects against bursts, background processes, scheduler overhead.- Rounds up with ceil to avoid under-provisioning.Complexity: O(1) time and space.Edge cases: zero/negative inputs, extremely small per-request CPU (ensure not zero), missing memory info, integer overflow for large numbers. Alternative: model concurrency per request duration (if requests overlap) by using per-request CPU*time instead of simple per-second multiplicative model.

MediumTechnical

58 practiced

Describe how you would design a predictive autoscaling system that uses historical traffic patterns, including daily and weekly seasonality, to proactively schedule scale-outs before expected peaks. What forecasting models or heuristics would you consider, how would you handle anomalies, and how would you validate the model?

Sample Answer

Requirements & constraints:- Predictive autoscaling for service X with daily/weekly seasonality, target SLOs (latency/availability), scale-up lead time (provisioning + warm-up), cost cap.- Input: historical traffic (RPS, concurrency), deployments, incidents, holidays.

High-level design:- Offline pipeline: ingest time-series (RPS, CPU, memory), enrich (calendar, promotion flags), clean (impute, remove outliers), train model per service/region.- Real-time pipeline: produce forecasts for next T hours, convert forecast to required capacity using performance model (capacity = f(RPS)->replicas/CPU), emit scale actions with lead time buffer and cooldowns.

Forecasting models / heuristics:- Baseline: exponential smoothing with seasonality (ETS / Holt-Winters) — simple, fast, handles daily/weekly patterns.- Production model: SARIMA or Prophet for interpretable seasonality and holiday effects.- ML option: Gradient boosted trees or LSTM if many regressors (features, marketing signals) and non-linearities.- Ensemble: combine fast ETS + ML to hedge model risk; weighted by recent accuracy.

Anomaly handling:- Preprocessing: detect anomalies with median absolute deviation and remove or downweight during training.- Runtime guardrails: compare short-term real-time metric to forecast; if real > forecast by threshold, trigger immediate reactive scale (safety path). If forecast spikes but no corroborating signals (e.g., no increase in upstream), suppress and mark for review.- Use rollback/cooldown to avoid oscillations.

Validation & monitoring:- Backtest with rolling-window cross-validation (walk-forward), compute MAPE, RMSE, and impact-oriented metric: predicted shortage time (%) where forecasted capacity < actual demand before provisioning lead time.- SLO-driven validation: simulate scale actions using forecast + provisioning latency on historical traces and measure SLO violations and cost delta.- Production validation: shadow mode where forecasts produce planned actions in dry-run; compare against reactive scaling.- Continuous learning: track forecast errors, recalibrate models weekly, auto-retrain when error drift exceeds threshold.

Operational considerations:- Safety caps, min/max instances, manual overrides.- Explainability: surface drivers of forecasts to ops/PMs.- Alerting on model drift, unusual forecast-triggered cost increase, and missed predictions.

EasyTechnical

65 practiced

Describe a simple method to estimate the number of worker processes required for a background job queue given average job duration, target concurrency, desired throughput, and SLA. Explain assumptions and how Little's Law applies in this estimation.

Sample Answer

Approach (brief): use Little’s Law (L = λ * W) where L = average number of items in system (workers busy), λ = arrival rate (jobs/sec), W = average job duration (seconds). Rearranged for workers: required_workers = λ * W. Add SLA and headroom to ensure latency/throughput targets.

Steps:1. Determine inputs: - avg job duration W (s) - target throughput λ (jobs/sec) — derived from desired throughput or observed peak arrival rate - SLA: e.g., 95th‑pct latency ≤ T_sla (s) or minimum throughput2. Base calculation: workers_base = λ * W. - Example: W = 3s, desired throughput = 20 jobs/sec → workers_base = 20 * 3 = 60 workers.3. Map SLA: if SLA is max latency T_sla and the system must serve arrivals with average service time W, ensure that average concurrency (workers busy) stays below capacity so queuing delay is acceptable. If you want average response ≤ T_sla and W is part of response, ensure queuing time Q ≤ T_sla − W. Using M/M/c approximations or target occupancy ρ = (λ * W) / c, choose c so ρ ≤ target (e.g., 0.7–0.8) to keep queuing low. - With ρ_target=0.75, workers = ceil(workers_base / 0.75) = ceil(60/0.75)=80.4. Add safety: include margin for spikes (e.g., +20–50%) and health‑check overhead.

Assumptions:- Job service time approximated by average W (variance ignored).- Arrivals approximated as stationary Poisson for queueing estimates.- Workers are identical; no significant start-up/shutdown cost.- No external throttling or resource contention (CPU/memory limits).

Why Little’s Law fits: it directly relates average jobs in system to arrival rate and service time regardless of distribution, letting you compute needed concurrent workers to sustain a given throughput. For SLA-sensitive scenarios, complement with queueing formulas (M/M/c or simulations) to account for variability and tail latency. Edge cases: high variance in job times, bursty arrivals, CPU-bound tasks needing fewer workers per host, or shared resource limits—these require measured profiling and iterative tuning.

MediumTechnical

61 practiced

Given a historical traffic series showing 20 percent month-over-month growth and measured per-request CPU and memory usage, outline a three year capacity roadmap. Include how you convert traffic projections to instance counts, how you add safety margins, and what assumptions you must document for finance and exec stakeholders.

Sample Answer

Approach: convert month-over-month (MoM) growth into request-rate projections, translate rate into required CPU/memory using measured per-request footprints, apply utilization targets and safety margins, and produce quarterly/yearly instance counts and cost/ops implications for three years.

1) Traffic projection- 20% MoM → monthly multiplier 1.20. Project R_t = R_0 * 1.20^t for t in months (or aggregate to quarters/years). Example: R_0 = 1M requests/month → Year1 ≈ 1M*1.20^12 ≈ 8.9M.

2) Convert traffic → resource demand- Measure: avg CPU-ms per request (C_req) and memory per concurrent request (M_req) or steady memory per process.- Choose target utilization U (e.g., 60% CPU to allow headroom/burst).- Estimate peak concurrency: assume peak-to-average ratio P (e.g., 3x) and request duration d.- Required CPU cores = ceil( (R_peak * C_req) / (1000 ms * seconds_per_month * core_ms_per_sec * U) ). Simpler: required vCPU = (peak RPS * C_req_ms) / (1000 ms * U * cpu_capacity_per_core_ms_per_sec).- Memory: required memory = baseline + concurrent_requests_peak * M_req; divide by instance memory.

3) Instance counts and safety margins- Pick instance type (vCPU, mem) and map required cores/memory to instances: instances = max( ceil(cpu_needed / vcpu_per_instance), ceil(mem_needed / mem_per_instance) ).- Add safety margin: operational buffer (20-40%) + redundancy for zones (N+1) and rollout capacity (extra 10-15% for deployments). Example total margin 30-50% depending on SLA.- Provide quarterly snapshots for 3 years, showing instances and projected monthly spend.

4) Operational plan- Use autoscaling for short-term elasticity; plan for reserved/spot usage for baseline to save cost.- Capacity exercises: load tests at projected peaks, validate scaling policies.- Monitor: SLOs, latency, CPU/memory headroom, autoscaler activity.

5) Assumptions to document for finance/exec- Base traffic R_0 and measurement period- 20% MoM is sustained and applies evenly (no seasonal variation)- Measured C_req and M_req are stable; no major feature changes- Target utilization U, peak-to-average ratio P, request duration d- Instance type, pricing (on-demand vs reserved/spot), region, network overhead- Desired SLA/SLO and N+1 redundancy model- Contingency margin percentage and review cadence (quarterly)

6) Deliverables & cadence- Provide spreadsheet with month-by-month RPS, CPU/mem, instance counts, cost.- Quarterly review to update assumptions, adjust margins, and re-run load tests.

This gives finance an auditable projection and ops a runnable plan with measurable validation steps.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Scaling and Capacity Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.