Capacity Planning and Forecasting Questions

Covers forecasting demand and planning infrastructure and platform capacity to meet expected business needs reliably and cost effectively. Candidates should be able to analyze historical usage and growth trends, build and validate capacity models, define capacity metrics and thresholds, estimate headroom and safety margins, and translate business growth scenarios into procurement or cloud provisioning plans and timelines. Includes storage and compute lifecycle planning such as archiving and retention strategies, upgrade and rollout planning to avoid disruption, and trade offs between overprovisioning and right sizing. Also addresses design for scale and redundancy, autoscaling and elasticity patterns, load balancing and failover planning, capacity testing and stress testing, monitoring and alerting for capacity signals, and techniques to measure and improve forecast accuracy. Finally it covers operational governance and decision making including cross team resource allocation, capacity reviews, cost optimization and budgeting, runbooks and change control, and alignment of capacity plans with service level objectives and business projections.

EasyTechnical

80 practiced

Explain capacity planning in the context of data engineering. Cover objectives, typical outputs (forecasts, headroom, procurement timelines), common metrics (throughput, storage growth, latency, concurrency) and time horizons (short/medium/long). Give examples of when you'd choose short-term autoscaling vs long-term procurement.

Sample Answer

Capacity planning for data engineering is the process of forecasting resource needs to keep pipelines, storage, and serving layers within SLOs while minimizing cost and procurement risk.

Objectives:- Meet SLOs (latency, freshness, availability)- Ensure enough headroom for traffic spikes and growth- Optimize cost (right-size infra, avoid overprovisioning)- Plan procurement and contractual lead times

Typical outputs:- Forecasts: CPU, memory, IOPS, network, and storage growth by time horizon- Headroom report: current utilization vs. safe operating band (e.g., 60–70% steady-state)- Procurement timelines: lead times for reserved instances, hardware, or contract negotiations- Actionable recommendations: autoscale rules, instance sizing, retention policy changes

Common metrics:- Throughput: records/sec, bytes/sec processed- Storage growth: TB/day, retention curves- Latency: job completion time, end-to-end freshness- Concurrency: simultaneous jobs, Spark executors, query sessions- Resource utilization: CPU, memory, I/O, network, and queue lengths

Time horizons:- Short (hours–days): handle bursts, incident mitigation, autoscale policies- Medium (weeks–months): seasonal peaks, sprint-level capacity changes, reserved instances- Long (quarters–years): projected growth, procurement of hardware, architectural changes (sharding, archive tiers)

When to choose:- Short-term autoscaling: sudden traffic spike, ephemeral ETL jobs, unpredictable upstream load, or cost-efficient scaling in cloud (use horizontal autoscaling, serverless jobs). Example: nightly ingestion surge after a marketing campaign—scale executors dynamically.- Long-term procurement/architecture change: sustained growth trend, regulatory storage needs, or when predictable baseline justifies committed discounts or on-prem hardware. Example: steady 30% YoY data growth leading to multi-year storage purchase or re-architecting into partitioned data lake and colder tiers.

Always combine monitoring, safety buffers, and periodic re-forecasting to adapt decisions.

EasyTechnical

95 practiced

You are designing a retention and archival policy for application logs and raw event data in a budget-conscious company. Propose a policy that balances cost, compliance, and access latency. Include retention durations, storage tiers (hot/warm/cold/archival), deletion and legal-hold considerations, and retrieval SLA expectations.

Sample Answer

Policy summary (goal: minimize cost while meeting compliance and acceptable access latency)

1) Data classes- Application logs (structured/unstructured): used for debugging, alerting, audits.- Raw event data (telemetry/analytics): used for model training and analytics.

2) Retention durations & tiers (example)- Hot (frequent access): App logs 0–30 days; Raw events 0–7 days — store on low-latency storage (S3 Standard / GCS Standard / SSD-backed cluster).- Warm (infrequent but reasonably quick access): App logs 31–90 days; Raw events 8–30 days — S3 Standard-IA / GCS Nearline or compressed columnar datasets.- Cold (rare access, analytical): App logs 91–365 days; Raw events 31–365 days — S3 Glacier Instant/Deep Archive alternative with retrieval options / cold object stores, or cold BigQuery partitions.- Archival (compliance/long-term): 1–7+ years depending on regulation (e.g., 7 years for finance) — S3 Glacier Deep Archive, tape, or cold archival buckets.

3) Deletion and lifecycle enforcement- Use automated lifecycle policies to transition and delete; implement soft-delete/tombstone window (e.g., 7 days) before permanent purge to recover accidental deletes.- Retention metadata stored with each dataset (retention expiry timestamp) and enforced by a central service/policy engine.- Ensure immutable write-once storage options where required (WORM).

4) Legal-hold & compliance- Legal hold overrides lifecycle: mark datasets as “hold” in metadata store; prevent transitions/deletions until release.- Audit trail: every hold/release/retain/delete action logged to immutable audit logs (write-once) for compliance.- Encryption at-rest/in-transit and access controls (IAM, RBAC) aligned with policy.

5) Retrieval SLA expectations (budget-conscious)- Hot: <1 minute- Warm: <1–4 hours (acceptable for investigation)- Cold: 6–24 hours (batch retrieval; can provide faster retrieval for pay)- Archival: 24–72 hours (or longer depending on archive class); offer expedited retrieval for critical compliance requests at extra cost

6) Cost vs. latency trade-offs & operational notes- Default to cheaper colder tiers for >30 days; preserve hot for troubleshooting window.- Offer paid-on-demand expedited retrieval path for legal/compliance urgency.- Use compression, partitioning, and deduplication to reduce storage and egress costs.- Monitor usage/requests monthly and tune thresholds (e.g., move hot→warm sooner if accesses low).- Tag data with owner/team and retention policy; chargeback/showback to incentivize cleanup.

7) Implementation checklist- Define lifecycle rules in cloud storage + periodic job to enforce metadata checks.- Implement legal-hold API & audit logging.- Alerting for approaching expirations and for failed transitions.- Quarterly policy review with security, legal, compliance, and analytics stakeholders.

This balances cost by expiring/archiving old data, meets compliance via holds and audit logs, and keeps acceptable retrieval SLAs with an expedited option when needed.

HardTechnical

103 practiced

Compare trade-offs between overprovisioning dedicated clusters and using serverless/transient compute options (serverless Spark, FaaS, spot VMs) for variable ETL workloads. Discuss cost models (including idle cost), cold-start penalties, predictability, operational overhead, and how to choose based on SLOs and team maturity.

Sample Answer

Situation: You’re evaluating how to run variable ETL workloads—steady baseline plus spikes—and must choose between always-on dedicated clusters (overprovisioned) vs. transient/serverless compute (serverless Spark, FaaS, spot VMs).

Trade-offs (by axis)

- Cost model & idle cost - Dedicated clusters: predictable monthly cost; high idle cost when utilization <80%. Good when long-running jobs or high baseline. Example: 10-node Spark cluster billed 24/7 → wasted when idle. - Serverless/FaaS/spot: pay-per-use; near-zero idle cost for FaaS/serverless Spark. Spot VMs cheap for long jobs but risk preemption; need checkpointing.

- Cold-start & latency - Dedicated: near-zero cold-start; consistent job startup. - Serverless/FaaS: cold starts (container spin, JVM warm-up) add seconds–minutes; serverless Spark often mitigates with warm pools but can still add latency. - Spot VMs: provisioning + rehydration can be slow; resume from checkpoints takes engineering.

- Predictability & SLOs - Dedicated: best for strict latency/SLA (SLOs requiring bounded start-to-complete times). - Serverless: fine for flexible SLOs (batch windows) but less predictable for tail latency. - Spot: unpredictable due to revocations—suitable if SLOs tolerate retries.

- Operational overhead - Dedicated: manage cluster scaling, upgrades, capacity planning (higher ongoing ops). - Serverless/FaaS: less ops—provider handles infra; focus on pipeline code. - Spot: moderate ops—implement checkpointing, fallback, and orchestration.

How to choose (practical rules)- If baseline utilization high, tight latency SLOs, and team experienced managing clusters → prefer dedicated (or autoscaling clusters with warm pools).- If workloads are spiky, cost-sensitive, and SLOs allow variability → serverless Spark or FaaS.- If cost-critical for long-running, fault-tolerant jobs and team can build robust checkpointing → use spot VMs with fallback to on-demand.- Hybrid: keep a small warm dedicated pool for low-latency needs + burst to serverless/spot for spikes.

Operational maturity guide- Junior teams/new orgs: start serverless to reduce ops and iterate.- Mature teams with SRE/data-platform: adopt hybrid or spot + dedicated where you can automate reliability.

Quantify decision- Model TCO: (compute cost + idle cost + ops cost + failure/retry cost) per month vs. SLO breach cost. Run experiments with representative workloads to measure cold-starts and preemption impact before committing.

EasyTechnical

103 practiced

Explain the difference between 'headroom' and 'safety margin' in capacity planning. Provide concrete examples for both storage and compute, and describe how you'd quantify each for a data processing cluster.

Sample Answer

Headroom vs safety margin — short answer:- Headroom is the operational spare capacity you keep to absorb short-term spikes or variance (elastic buffer).- Safety margin is the longer-term buffer you provision above expected demand to cover forecasting error, growth, or outages (planning buffer).

Concrete examples

Compute (data-processing cluster):- Headroom: If your cluster normally uses 60 of 100 cores during daily batch windows, headroom = 40 cores (40%). That lets an unexpected job or retry spike run without hurting SLAs. Operational rule: keep CPU utilization below a threshold (e.g., 60–75%) so headroom exists for bursts.- Safety margin: If projected peak demand in 12 months is 120 cores, you might provision 150 cores (25% safety margin). That protects against forecasting error, new pipelines, or node failures during maintenance.

Storage:- Headroom: Live usable storage 100 TB; current active datasets use 70 TB → headroom = 30 TB (30%). This prevents ingest failures from sudden data surges and gives room for compacting/replication overhead.- Safety margin: If expected growth over a year is 40 TB, you may provision 120 TB total (20% safety) or plan capacity to 160 TB if risk tolerance is low.

How to quantify for a data-processing cluster- Define metric per resource: CPU cores / vCPU, memory, network bandwidth, IOPS, usable storage.- Measure baseline demand (median, p95) and spike behavior (max observed, distribution).- Headroom calculation (operational): Headroom (%) = 1 − (p95 utilization / threshold). Practically: target max operational utilization U_target (e.g., 70%). Headroom = U_target − current_utilization. Example: current p95 CPU = 55%, U_target = 75% → headroom = 20 percentage points.- Safety margin calculation (planning): Safety margin (%) = (Provisioned_capacity / Projected_peak_demand) − 1. Example: projected 12-month peak = 120 cores, provision 150 → safety margin = 25%.

Practical guidance- Use different values by resource and stage: aggressive headroom for CPU/latency-sensitive jobs, larger safety margin for storage where re-provisioning is slower.- Align with SLOs: if jobs must finish within X hours, set headroom to the level that keeps p99 runtime within SLO during spikes.- Automate monitoring and alerts on both: low headroom (immediate scaling) vs safety-margin breach (capacity purchase/resize planning).

HardTechnical

94 practiced

Design a capacity test to validate an ingest pipeline targeting 10 million transactions per second (10M TPS). Describe workload generation (distributed generators and test harness), network and storage bottlenecks, end-to-end latency and tail-latency measurement, integrity checks, and safe scaling strategies for running this test in staging.

Sample Answer

Requirements & constraints:- Target sustained ingest: 10M TPS (define event size: e.g., 200 bytes) → ~2 GB/s. Define protocol (HTTP/gRPC/TCP/Kafka) and SLAs: p50/p95/p99/p999 latency goals and data integrity.

Approach overview:- Use distributed workload generators that mimic real producers (client libraries, protocol, auth) and a central test harness to coordinate, collect metrics, and drive safe ramps.

Workload generation & test harness:- Generators: many lightweight instances (containers/VMs) each producing N TPS. Use language-native clients to avoid synthetic artefacts (e.g., librdkafka, gRPC stubs). Use fixed-size and variable-size event mixes.- Orchestration: Kubernetes job or fleet manager to deploy generators with consistent clock sync (PTP/NTP) and config. Central harness coordinates start/stop, ramp schedule, collects generator-side telemetry (send rates, failures, latency histograms).- Distribution: Place generators across AZs/racks to avoid single-host limits. Each generator logs event IDs, sequence numbers, checksums, send timestamps.

Network & storage bottlenecks:- Network: provision NICs and link capacity > peak (10M TPS * avg size * replication factor). Use 25/40/100Gbps NICs; verify host TCP stack tuning (net.core.rmem_max, etc.), number of sockets, and enable kernel bypass (DPDK/RDMA) where supported. Monitor interface utilization, switch port oversubscription, and packet drops.- Storage: if writing to disk (log store), provision write IOPS and throughput (consider SSD NVMe, RAID, or cloud block storage), and ensure write path replication (e.g., Kafka ISR) can sustain throughput. Measure end-to-end single-partition throughput vs partitioned topology (Kafka: scale partitions).- Monitor: host CPU, NIC queues, kernel drops, disk latency, broker GC, network switch counters.

Latency & tail-latency measurement:- Instrument end-to-end timing: client-send timestamp, ingest-ack timestamp, processing-complete timestamp. Use HdrHistogram on both client and server to capture microsecond resolution and compute p50/p95/p99/p999.- Correlate spans using IDs to measure per-stage latencies. Export histograms to central metrics (Prometheus + Grafana). Pay special attention to long-tail causes: GC pauses, compaction, full GC, network requeues, broker leader movements.

Integrity checks:- Deterministic sequence IDs and checksums in payload. Server-side verify and write back an acknowledgement containing sequence ID and checksum. After test, run reconciliation: counts per shard/partition, missing ranges, duplicate detection, checksum mismatches. Include end-to-end cryptographic hash over windows to detect silent corruption.

Safe scaling strategies in staging:- Canary & progressive ramp: start with small subset of generators -> increase by geometric steps (x2) while monitoring key metrics and health checks. Set hard abort thresholds (error rate, p99 latency spike, CPU>85%, packet drops) that trigger automated rollback/throttle.- Isolation: run in isolated staging network with similar topology but rate-limited ingress to avoid impacting other environments. Use resource quotas and cgroups to prevent generator overloads from saturating hosts.- Partitioning: scale ingestion horizontally (more partitions/shards) rather than single-partition saturation; pre-create partitions and rebalance carefully.- Backpressure & throttling: ensure consumers/brokers have backpressure and retry strategies; include circuit breakers in generators to avoid infinite retries.- Dry-run validations: validate metadata, auth, routing at low TPS before high-load runs.

Observability & post-test analysis:- Collect logs, histograms, system metrics, network flow telemetry, and reconciler reports. Produce a runbook: what failed, root cause (network vs storage vs app), and actionable remediation (add brokers, increase partitions, tune GC, upgrade NICs).

Trade-offs:- Synthetic generators may not capture all production behaviors (e.g., JVM pauses) — include a small subset of real producers or run a shadow traffic test if possible.- Prefer horizontal scaling and partitioning to maximizing single-node throughput for operational resilience.

This plan yields reproducible, measurable validation for 10M TPS with safety gates and integrity guarantees.

Unlock Full Question Bank

Get access to hundreds of Capacity Planning and Forecasting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.