Capacity Planning and Forecasting Questions

Covers forecasting demand and planning infrastructure and platform capacity to meet expected business needs reliably and cost effectively. Candidates should be able to analyze historical usage and growth trends, build and validate capacity models, define capacity metrics and thresholds, estimate headroom and safety margins, and translate business growth scenarios into procurement or cloud provisioning plans and timelines. Includes storage and compute lifecycle planning such as archiving and retention strategies, upgrade and rollout planning to avoid disruption, and trade offs between overprovisioning and right sizing. Also addresses design for scale and redundancy, autoscaling and elasticity patterns, load balancing and failover planning, capacity testing and stress testing, monitoring and alerting for capacity signals, and techniques to measure and improve forecast accuracy. Finally it covers operational governance and decision making including cross team resource allocation, capacity reviews, cost optimization and budgeting, runbooks and change control, and alignment of capacity plans with service level objectives and business projections.

EasyTechnical

101 practiced

Explain the difference between capacity, utilization, headroom, and error budget in an SRE context. Give concise definitions and one concrete example of how each term would be used when planning capacity for a web service that handles 1,000,000 daily requests.

EasyTechnical

95 practiced

List common capacity-related alerts and signals you would configure to detect capacity degradation early (for example: queue depth, CPU steal, disk saturation, GC pause time, thread pool exhaustion). For each signal suggest sensible thresholding guidelines and explain why it matters.

Sample Answer

Below are common capacity-related signals I’d instrument, sensible threshold guidance, and why each matters (SRE focus: early detection, high signal-to-noise).

- Queue depth (work queues / message broker queues) - Thresholds: alert at sustained > 70% of per-queue configured capacity or growth rate > 20%/5m. - Why: rising queue depth is the earliest sign consumers are slower than producers; prevents backlog explosions and latency SLO breaches.

- Queue latency / time-in-queue - Thresholds: p95 > target SLO (e.g., 200ms) or p99 > 2×SLO. - Why: indicates processing delays even if depth looks ok.

- CPU utilization (per core) - Thresholds: sustained > 75–80% for 5–10m on multi-tenant hosts; >90% on single-purpose nodes. - Why: high CPU reduces headroom for spikes; sustained high ties into scheduling and latency.

- CPU steal (virtualized environments) - Thresholds: >5% alert, >15% critical over 5m. - Why: indicates hypervisor contention — performance unpredictable despite low guest CPU.

- Memory utilization and swap usage - Thresholds: resident memory >80% of capacity; swap >5% used or any sustained swap activity. - Why: prevents OOM kills, GC pressure, and severe latency spikes.

- Disk utilization and I/O wait / disk queue length - Thresholds: disk usage >85% (capacity); iowait >20% or queue length consistently > number_of_disks*2. - Why: disk saturation causes requests to back up and increases latencies.

- GC pause times (JVM/.NET) - Thresholds: p95 GC pause > 100ms or p99 > 500ms; frequency > 1/min. - Why: long or frequent pauses cause request spikes, timeouts, and cascading failures.

- Thread pool exhaustion / rejected executions - Thresholds: active threads > 80% of pool size for >1m; any rejected-task rate >0 should alert at high priority. - Why: signals inability to accept work; leads to queueing at higher layers or failed requests.

- Request latency and error rate (app-level) - Thresholds: latency p95 > SLO; error rate > baseline + 3σ or >1% absolute depending on service. - Why: direct customer impact metric — often the first visible symptom.

- Connection counts / ephemeral port exhaustion / file descriptors - Thresholds: connections >80% of limit; fd usage >85%. - Why: resource exhaustion causes immediate failures.

- Pod/node eviction pressure / scheduler backpressure (k8s) - Thresholds: node diskPressure/memoryPressure events; pod pending > small baseline. - Why: prevents loss of capacity and ensures autoscaling triggers are healthy.

Guidelines: prefer short sustained windows (5–10m) to avoid flapping, use multi-tier alerts (warning → critical), bind some alerts to capacity trends (growth rates) and some to absolute limits, and route high-confidence, actionable alerts to on-call with runbooks.

HardTechnical

94 practiced

For a microsecond-sensitive service, design capacity and redundancy so the system tolerates a 2-node rack failure with minimal RTO while staying cost-conscious. Discuss replication strategies, network topology, cross-rack placement, failure domains, and how you would test and validate the design.

Sample Answer

Situation & goal: design capacity/redundancy for a microsecond-sensitive service so the system tolerates simultaneous failure of any two nodes in the same rack with minimal RTO while controlling cost.

Design principles (high level)- Keep the data path as local/fast as possible (low latency) while ensuring independent failure domains for replicas.- Optimize for read/write latency separately: sync writes require stricter placement than reads.- Use capacity headroom to failover instantly without scaling latency penalties.

Replication strategy- Synchronous replication to 3 replicas for critical metadata/leader state placed across distinct failure domains (rack + power PDUs). Use quorum (majority) writes to maintain consistency; choose leader-follower with fast leader election.- Asynchronous read replicas for scaling reads within same availability domain to reduce cross-rack hops when acceptable.- For microsecond sensitivity: prefer local SSD/NVMe and bypass extra serialization layers; use RDMA or kernel bypass (DPDK) where feasible.

Placement & topology- Network topology: spine-leaf with ECMP, redundant ToR switches per rack. Ensure ToR dual-homed NICs and MLAG to avoid plane single-points.- Cross-rack placement: ensure replicas for any critical shard are spread across at least 3 racks and 2 PDUs. For 2-node rack failure tolerance, ensure no two replicas are colocated on same rack beyond one non-critical replica.- Failure domains: define node, rack, PDU, ToR, and pod. Map replication policy to avoid correlated failures (e.g., avoid same batch firmware).

Capacity planning- Maintain N+2 replica capacity for traffic headroom; size for 2-node-per-rack loss worst-case: ensure remaining replicas can absorb load with <X% latency increase.- Use resource throttling and CPU pinning to guarantee tail-latency bounds.

Failure handling & minimal RTO- Fast detection: aggressive health probes + BFD for network, application-level heartbeats.- Automated failover: pre-warmed standby replicas with warmed caches and warmed connections. Use fast reconfiguration (local quorum shifts) and leader election within <500ms target.- Circuit-breakers and backpressure to prevent cascading overload.

Testing & validation- Chaos experiments: scheduled chaos that kills 2 nodes and single ToR per rack during production-like load; run long enough to evaluate tail latencies.- Fault injection in staging: network partition, PDU loss, packet loss, increased latency; measure RTO, error rates, p99/p999 latencies.- Capacity validation: load tests that simulate failover traffic and confirm SLOs.- Observability: distributed tracing, fine-grained metrics (p50/p99/p999), synthetic transactions, and automated rollback playbooks.- Runbook & drills: on-call runbooks, blameless postmortems, and regular incident drills.

Trade-offs & cost controls- Use erasure coding for large, non-latency-critical data; keep critical hot-path data fully replicated.- Right-size replica count and hardware (mix of NVMe for hot-tier, cheaper disks for cold).- Use auto-scaling for non-latency-critical components; keep fixed redundancy for microsecond path.

This design balances microsecond latency, tolerates 2-node rack failure, keeps RTO low via fast detection and warm failover, and controls cost by mixing replication types and targeted hardware investments.

EasyTechnical

84 practiced

How do you decide whether to provision capacity based on the 95th percentile, 99th percentile, or absolute peak metrics? Explain the trade-offs in cost and user experience, how each choice affects SLOs and error budget, and provide a simple numeric example converting a percentile-based latency budget into provisioned instances.

Sample Answer

Deciding whether to provision to the 95th, 99th percentile, or absolute peak depends on acceptable user impact, cost constraints, and SLO/error-budget policy.

- When to pick each: - 95th percentile: Good for cost-sensitive services where occasional slower requests are acceptable (e.g., analytics batch jobs). Balances cost and experience. - 99th percentile: For user-facing latency-sensitive features where rare long tails harm UX (search, checkout). More cost to reduce tail risk. - Absolute peak: For safety-critical or regulated systems where any degradation is unacceptable (payment authorization, emergency systems). Highest cost.

- Trade-offs: - Cost: higher percentile -> more capacity -> higher infra cost and lower utilization. - User experience: lower percentile target (closer to peak) reduces outliers and improves consistency. - Operational complexity: peak provisioning simpler (overprovision) vs. autoscaling + buffering for percentile targets.

- SLOs and error budget: - If SLO is latency ≤ X for 99% of requests, provisioning to 99th reduces consumption of the error budget and lowers incident risk. Provisioning only to 95th will consume more error budget, increasing likelihood of paging and remediation actions when the budget is exhausted.

Numeric example:- Observed per-instance throughput: 100 requests/sec. Traffic peak (1 min): 10,000 rps.- Goal: meet latency SLO for 99% of requests. From profiling, to hit latency X, an instance can sustain 80 rps at the 99th percentile (because tails reduce throughput).- Provisioned instances = ceil(peak_rps / per_instance_rps_at_target_percentile) = ceil(10,000 / 80) = 125 instances.- If you provision to 95th where per-instance rps_at_95th = 100, you need ceil(10,000/100)=100 instances — cheaper but allows more 95–99th percentile latency violations.

Summary: choose percentile aligned to SLO and acceptable error-budget burn; prefer autoscaling + buffers and mitigation (circuit breakers, graceful degradation) before always provisioning to peak.

HardTechnical

87 practiced

Design capacity dashboards and KPIs that detect emergent saturation before SLO breach using streaming anomaly detection. Describe which leading indicators to show (e.g., queue growth, 95th latency), detection model choices, alert latency budget, and ways to keep alerts actionable with low false positive rates.

Sample Answer

Situation: We need dashboards and streaming anomaly detection that surface emergent capacity saturation early—before SLOs are breached—so ops can remediate proactively.

Leading indicators to show (real-time, per service / per region / per tier):- Ingress rate (rps) and short-term derivative (rps/min) — detect bursts- Queue length / backlog and growth rate (items/min) — earliest backlog signal- 95th and 99th latency (P95, P99) and slope over last 1–5m — tail amplification- Service time / processing time distribution (P50/P95)- CPU % busy, CPU steal, run queue length- Memory used vs. usable and GC pause rate (for JVM)- Concurrency (active requests) and connection pool saturation- Retry / throttled / 429 / circuit-breaker events- Error rate and error fraction among slow requestsDisplay: top-left heatmap of derivatives (growth rates), time-series for each KPI with small-multiple panels, resource maps, and per-request waterfall for tail cases.

Detection model choices (streaming, low-latency, explainable):- Fast univariate detectors per metric: EWMA + adaptive threshold (for smoothing) and robust z-score on rolling window (1–5m windows) for immediate spikes.- Trend-aware detectors: Holt-Winters or online STL to separate seasonality vs trend for metrics with diurnal patterns.- Changepoint detection: Bayesian Online Changepoint Detection for sudden regime shifts (queue growth starts increasing).- Multivariate correlation layer: real-time correlation / causation scoring (e.g., streaming PCA or online isolation forest) to require simultaneous anomalies across related signals (e.g., queue growth + rising P95 + CPU run-queue) before paging.- Optional ML: streaming LSTM-autoencoder or Online Isolation Forest for complex patterns where enough labeled incidents exist; keep as secondary scoring to avoid opaque alerts.

Alert latency budget:- Goal: alert within 1–3 minutes of earliest reliable signal for fast user impact services; 5–10 minutes for batch/backfill-like systems.- Define detection window sizes: 30s-1m sample, 3–5m evaluation window for transient suppression, 15–30m sustained threshold for severity escalation.

Keeping alerts actionable & low FPR:- Multi-signal corroboration: require N related signals or a composite score above threshold (e.g., queue growth > X and P95 rising > Y).- Context-aware thresholds: normalize by capacity (requests per CPU), deploy-phase suppression, feature flags, and maintenance windows.- Adaptive thresholds: percentile-based baselines that update slowly (e.g., 6–24h) to avoid alerting on daily cycles.- Rate-limit and dedupe: group similar alerts by service/region; escalate only if persists after cooldown (e.g., 3–5m) or increases severity.- Enrich alerts: include causal hints (top endpoints, recent deploys, shard hot-spots), suggested runbook steps, rollback flags, and recent config changes.- False-positive feedback loop: allow on-call to mark and feed incidents to retrain thresholds/models.- Test alarms: synthetic traffic and chaos tests to validate detection sensitivity/precision.

Operational practices:- Define SLO lead-time: convert SLO burn rate to required lead-time (e.g., if SLO allows 1% errors per hour, compute how long sustained anomaly would take to breach) and tune alert thresholds earlier than that.- Runbooks per alert with playbooks, ownership, and mitigation primitives (scale up, shed load, increase rate limits).- Post-incident: tune models, add features, and annotate dashboards with lessons.

This design balances early detection (queue growth, slope of tail latencies) with conservative multi-signal confirmation and contextual enrichment so alerts are timely, actionable, and have low false positives.

Unlock Full Question Bank

Get access to hundreds of Capacity Planning and Forecasting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.