Infrastructure Scaling and Capacity Planning Questions

Operational and infrastructure level planning to ensure systems meet current demand and projected growth. Topics include forecasting demand headroom planning and three to five year capacity roadmaps; autoscaling policies and metrics driven scaling using central processing unit memory and custom application metrics; load testing benchmarking and performance validation methodologies; cost modeling and right sizing in cloud environments and trade offs between managed services and self hosted solutions; designing non disruptive upgrade and migration strategies; multi region and availability zone deployment strategies and implications for data placement and latency; instrumentation and observability for capacity metrics; and mapping business growth projections into infrastructure acquisition and scaling decisions. Candidates should demonstrate how to translate requirements into capacity plans and how to validate assumptions with experiments and measurements.

HardTechnical

54 practiced

Design an IaC-driven auto-provisioning pipeline capable of scaling infrastructure predictably: include gitops patterns, feature-flagged capacity changes, terraform module versioning, safe rollouts (canary infra changes), state locking, and validation steps to apply changes affecting thousands of instances.

Sample Answer

Requirements & constraints:- Predictable, auditable infra changes for thousands of instances- GitOps-driven, automated approvals, safe canary rollouts, feature-flagged capacity, Terraform module versioning, state locking, automated validation and rollback.

High-level design:- GitOps control plane: repos (infra-modules, env-configs) + ArgoCD/Flux to reconcile env-configs into target clusters/accounts.- Terraform modules stored in a private registry (or monorepo) with semantic versioning (MAJOR.MINOR.PATCH). Consumers pin module versions in env-configs (Terraform required_version and module source with version constraint).- Remote state + locking: S3 + DynamoDB (AWS) or GCS + Cloud KMS + Firestore locks. Use workspace per environment and per canary cohort.- CI pipeline (GitHub Actions/GitLab CI) triggered on PR: runs terraform fmt, init, plan (remote backend with -lock-timeout), policy checks (OPA/Conftest/terraform-compliance), unit tests (terratest), and posts plan as a Git PR comment.- Validation gates: automated pre-apply checks (cost estimate, security scans, resource quotas), and manual approval for production.- Canary infra rollouts: implement canary cohorts via staged workspaces/environments: 1) Create canary workspace (e.g., prod-canary-1) with capacity scaled to a small percent via env-specific variables. 2) Apply to canary cohort automatically after PR approval. 3) Run automated verification: health checks, load tests, metric baselines, integration tests. 4) If OK, promote: update env-configs to next cohort or full production; promotion is another Git commit merged through GitOps.- Feature-flagged capacity changes: use a feature flag system (LaunchDarkly or home-grown flags stored in a service). Capacity-increasing Terraform applies set a low initial limit and gate the activation of additional capacity via a flag-read step in the pipeline or runtime autoscaler configured to consult flags/parameters.- Safe rollbacks: all applies are immutable commits; rollback performed by reverting the Git commit which triggers GitOps reconciliation to previous state. Ensure state snapshots/backups and change logs.- Observability & automated rollback: integration with monitoring (Prometheus, CloudWatch), define SLO-based alerting and automation (e.g., Terraform apply can create a rollback job via Argo Rollouts if post-apply metrics cross thresholds).- Module versioning & promotion: develop modules in feature branches, publish prerelease versions, test in staging and canary, then tag release (increase semver) and update consumer env-configs. Enforce policy that production only accepts modules >= stable tag.- Access & security: least privilege TF state encryption, CI runners use short-lived creds (OIDC), signed commits + branch protection, Terraform Cloud/Enterprise for runs if available (remote run + policy checks).Example Terraform backend (AWS S3 + DynamoDB):

hcl

terraform {
  backend "s3" {
    bucket         = "org-terraform-state"
    key            = "prod/vpc/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

Why this works:- GitOps provides auditable, declarative single source of truth.- Module versioning + pinning ensures reproducible builds.- Canary cohorts + test/metric gates make large-scale changes predictable and safe.- State locking + remote backend prevents concurrent corruption.- Feature flags decouple capacity activation from infra provisioning, enabling controlled ramp-ups.

Operational playbooks:- Pre-merge: CI runs plan + policy checks; PR reviewer validates plan and module versions.- Post-merge: GitOps applies to non-prod; canary pipeline triggers prod-canary workspace, runs verification, then staged promotion commits.- Failure: automated alarms trigger a Git revert or Argo Rollouts reversal; runbook defines metrics and rollback thresholds.

Tradeoffs:- More orchestration complexity vs safety. Use Terraform Cloud or managed GitOps to reduce custom glue.- Canary cohorts require environment/account isolation; cost vs safety tradeoff must be communicated to stakeholders.

MediumTechnical

53 practiced

Create a metrics and observability plan to support capacity planning: list critical metrics (infra, app, business), aggregation windows, dashboards/visualizations for trend analysis, alert thresholds for capacity risks, and retention strategy to support 3-year forecasting.

Sample Answer

High-level goal: provide continuous visibility into resource utilization and business load so we can predict and mitigate capacity risks over 3 years while balancing storage cost and signal fidelity.

Critical metrics- Infrastructure: CPU avg/95/99, memory used/available, disk IOPS and latency, disk used (%), network bytes in/out, ephemeral storage pressure, pod/node counts, autoscaler events, GC/VM churn.- Application: request rate (RPS), latency p50/95/99, error rates (4xx/5xx), concurrency, queue/backlog length, DB connections, cache hit ratio, batch job runtimes.- Business: user signups/day, transactions/hour, revenue/minute (or cohort metrics), active MAU/DAU, peak concurrent users, feature-flag adoption.

Aggregation windows & retention policy (to support 3-year forecasting)- Raw/stamped (high-res): 1s–10s for critical infra/app metrics; retain 30–90 days.- Short-term rollups: 1m aggregates for 6 months.- Medium-term rollups: 1h aggregates for 2 years.- Long-term rollups: 1d aggregates retained 3+ years (for trend & seasonality).- Store business events in a data warehouse (hourly/day) with full history and joins to metrics.

Dashboards & visualizations (for trend analysis)- Capacity overview: heatmap of CPU/memory/disk percent used across clusters, sortable by cost center.- Trend panels: time-series with raw vs. baseline (rolling 90d median) and seasonal bands; overlay forecasts (ARIMA/ETS/Prophet) with confidence intervals.- Percentile panels: latency and CPU at p50/p95/p99.- Consumption by tenant/service: stacked area charts and top-N contributors.- Anomaly/trend dashboard: changepoint detection, growth-rate table (week-over-week, month-over-month).- Forecasting dashboard: projected growth curves, reservation vs. on-demand gap, recommended capacity buys.

Alert thresholds & escalation (capacity risk focused)- Use sustained windows to avoid noise (e.g., condition persists > 5–15m).- Warning: sustained utilization > 70–80% for 1h (predictive alert if 7–14 day growth projects crossing 85%).- Critical: sustained utilization > 85–90% for 15–30m or sudden >10% jump in 5m.- Latency/error-based: p95 latency increase > 2x baseline or error rate > 1% with increasing trend → immediate investigation.- Business-backed alerts: transaction throughput drop > 15% with concurrent error rise.- Alert actions: automated scaling runbooks, capacity purchase recommendations, paging for critical.

Forecasting approach & tooling- Combine time-series models (Exponential smoothing, Prophet, ARIMA) with causal features (marketing campaigns, sales forecasts, product launches).- Run weekly automated forecasts and monthly executive capacity reports.- Tools: Prometheus (short-term), Grafana (dashboards), Loki/Elastic for logs, CloudWatch/Datadog for managed clouds, and a data warehouse (Snowflake/BigQuery) for long-term business joins and ML forecasting.

Retention storage/cost trade-offs- Keep high-resolution only where necessary (hot services), downsample others aggressively.- Use compressed columnar storage for long-term business metrics.- Archive raw telemetry to cheap object storage (S3/Glacier) for compliance.

Governance & process- Quarterly capacity review with engineering, finance, and product; include forecast, headroom, and reservation ROI.- Define SLOs/SLA-backed runbooks and tie forecasts to procurement/purchase lead times.

This plan ensures operational alerts for immediate risk, dashboards for root-cause and trend analysis, and a retention/forecasting pipeline that supports accurate 3-year capacity planning.

MediumSystem Design

65 practiced

Design capacity isolation and elasticity for a SaaS multi-tenant platform where tenants range from hobbyists to large enterprises. Include tenancy models (pooled vs single-tenant), admission control, per-tenant limits, burst policies, and how capacity decisions map to pricing tiers.

Sample Answer

Requirements:- Support tenants from hobby (low, bursty) to enterprise (SLA, predictable high throughput).- Strong isolation for noisy neighbors, cost-effective pooling, autoscaling, metering for billing.- Low-latency for premium tiers, configurable burst allowances, enforcement and observability.

High-level architecture:- API Gateway → Admission Controller → Router → Tenant Execution Plane (Pooled Shared Nodes + Optional Dedicated Nodes) → Monitoring & Billing

Tenancy models:- Pooled multi-tenant: default, cost-efficient. Shared compute with per-tenant resource controls (CPU, memory, IOPS, concurrency).- Single-tenant/dedicated: optional for enterprise tiers—entire node(s) or cluster per tenant for strict isolation and custom SLAs.

Core components and responsibilities:1. Admission Controller: authenticates, enforces per-tenant quotas (RPS, concurrent jobs, storage IO), evaluates burst eligibility, applies backpressure (429/queue token).2. Resource Scheduler: assigns requests to pooled worker pools or tenant-dedicated nodes; enforces cgroups/containers or K8s resourceQuota/LimitRange.3. Elastic Autoscaler: scales pooled pools by global load; scales dedicated clusters per-tenant metrics + scheduled policies.4. Burst Controller: token-bucket per tenant with refill rules tied to subscription; short-lived borrow from reserve pool if available, otherwise queue or degrade.5. Observability & Billing: real-time metrics, anomaly detection, metering for usage-based billing and alerts.

Admission control & per-tenant limits:- Static baseline quotas per tier (e.g., Hobby: 100 RPS, Pro: 1k RPS, Enterprise: 10k RPS).- Dynamic soft limits with burst tokens: allowed bursts = baseline * burst_factor for short windows (e.g., 5s). Track sustained overage -> throttle or require upgrade.- Priority queuing for premium tiers; circuit breaker to protect system under overload.

Elasticity and capacity decisions:- Two-layer scaling: horizontal autoscaling of pooled workers by aggregate metrics; per-tenant autoscaling for dedicated clusters.- Reserve capacity pool for bursts sized from observed percentile usage (p95+p buffer).- Proactive scaling using predictive models for enterprise tenants with predictable schedules (cron/business hours).

Mapping capacity to pricing tiers:- Tiers define baseline quotas, burst budgets, priority, and isolation level: - Free/Hobby: low baseline, minimal burst, best-effort SLAs, shared pool, metered overage charges. - Standard/Pro: higher baseline, moderate burst tokens, higher priority queue, minimal dedicated options. - Enterprise: guaranteed capacity, customizable dedicated nodes or soft guarantees with committed spend, higher burst allowance, 24/7 support and SLAs.- Pricing components: base subscription (reserving baseline), usage-based overage (per CPU-minute, GB storage, IOPS), premium features (dedicated nodes, priority routing, guaranteed capacity).

Trade-offs:- Pooling maximizes utilization but needs strong enforcement and monitoring to prevent noisy neighbors.- Dedicated tenants increase cost but offer predictable performance—sell as value-add.- Conservative burst policies improve stability but reduce perceived responsiveness; use telemetry-driven tuning.

Operational considerations:- Clear billing and upgrade paths when sustained overages occur.- Expose tenant dashboards with usage, alerts, and recommended tier.- Run regular chaos/soak tests to validate admission, scaling, and billing behavior.

This design balances cost-efficiency for small tenants with isolation and predictability for enterprises while making capacity and pricing transparent and enforceable.

MediumTechnical

65 practiced

Design an autoscaling policy for a Kubernetes deployment serving a latency-sensitive API using CPU, memory and a custom p95-latency metric. Explain concrete values for thresholds, stabilization windows, cooldowns and how you'd prevent oscillation and ensure cluster autoscaler interaction is safe.

Sample Answer

Goal: keep p95 latency under SLO (e.g., 200ms) while avoiding unnecessary scale events. Use multi-metric HPA (CPU, memory, external p95) with stabilization, hysteresis and safe interaction with Cluster Autoscaler.

Suggested values and reasoning:- SLO: p95 <= 200ms. Use external metric "api_p95_latency_ms".- CPU target: 60% average across pods — keeps headroom for bursts.- Memory target: 75% to avoid OOM-triggered churn.- p95 target: 200ms; scale up if >200, scale down only when <150 (hysteresis).

HPA snippet (Prometheus Adapter providing external metric):

yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: api-hpa}
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: api}
  minReplicas: 3
  maxReplicas: 30
  metrics:
  - type: Resource
    resource: {name: cpu, target: {type: Utilization, averageUtilization: 60}}
  - type: Resource
    resource: {name: memory, target: {type: Utilization, averageUtilization: 75}}
  - type: External
    external: {metric: {name: api_p95_latency_ms}, target: {type: AverageValue, averageValue: "200"}}
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent; value: 100; periodSeconds: 60
      - type: Pods; value: 5; periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent; value: 20; periodSeconds: 300

Key choices:- stabilizationWindowSeconds: short for scaleUp (30s) to respond quickly to latency spikes; long for scaleDown (300s) to avoid flapping and wait for traffic drop confirmation.- scaleUp policies cap burst growth (max 5 pods or 100%) to prevent sudden overshoot; scaleDown limited to 20% per 5min to avoid oscillation.- Hysteresis: require p95 < 150ms before scaling down in custom controller or via two-tier metric (use a Prometheus recording rule that only reports scale-down metric when sustained <150ms for Xs).

Cluster Autoscaler safety:- Ensure pod requests are realistic (not zero); request values determine node packing and CA decisions.- Use PodDisruptionBudget to keep availability during node scale-down.- Use node groups with sufficient headroom (e.g., buffer nodes or use CA's balanced-similar-node-groups) and set max surge for node provisioning latency.- Set HPA maxReplicas to what cluster can host until CA adds nodes; coordinate with CA max nodes to avoid "pending pods" storms.- Add a scale-up delay to CA-sensitive metrics: prefer CPU-driven immediate scale-up, but allow external latency to trigger additional pods gradually to give CA time to provision nodes.

Prevent oscillation:- Multi-metric decision reduces false positives (require any metric to indicate need; tune weights).- Use averaged metrics (p95 over 30s window) and recording rules to smooth spikes.- Limit scaling frequency with policies and stabilization windows.- Monitor and iterate: capture metrics (scale events, queue lengths, p95) and adjust thresholds.

Operational notes:- Use Prometheus Adapter with recording rules: compute a sustained p95 and a "scale_down_safe" boolean metric.- Load-test (k6) to validate thresholds and node provisioning delays.- Document failure modes and rollback plan.

This balances latency SLOs, stability and safe Cluster Autoscaler interaction.

MediumTechnical

72 practiced

You must decide between a managed Kafka service and self-hosted Kafka on VMs for a real-time analytics platform expected to scale 10x in 3 years. Compare cost drivers, operational effort, scalability, feature parity, upgrade/patching risk, and failure-mode operational playbooks.

Sample Answer

Situation: Client must choose between managed Kafka (e.g., Confluent Cloud, MSK) and self-hosted Kafka on VMs for a real-time analytics platform projected to grow 10x over 3 years.

Recommendation summary: For predictable rapid scale with limited ops headcount, prefer managed Kafka; for strict cost control, custom tuning, or regulatory constraints, prefer self-hosted. Below is a side-by-side evaluation.

Cost drivers- Managed: predictable per-MB / per-hour pricing, additional charges for data egress, connectors, and storage; OPEX-heavy but less capital outlay.- Self-hosted: CAPEX for instances, storage, networking; ongoing OPEX for ops staff, monitoring, backups; lower unit costs at very large scale if fully optimized.

Operational effort- Managed: minimal day‑to‑day (provisioning, backups, upgrades handled); still need schema management, topic/config tuning, and SLAs.- Self-hosted: full ops lifecycle—capacity planning, OS/patching, zookeeper/cluster ops, HA, security hardening.

Scalability- Managed: elastic scaling, automated rebalancing in many providers, faster time-to-scale.- Self-hosted: offers fine-grained control but requires planned capacity increases, rolling upgrades, and manual rebalances — more risk/time at 10x growth.

Feature parity- Managed: near-parity; some providers offer extras (fully-managed connectors, ksqlDB) and enterprise features.- Self-hosted: full access to plugins/custom configs; better for non-standard extensions.

Upgrade/patching risk- Managed: vendor-controlled rolling upgrades with tested compatibility; lower risk but less control over timing.- Self-hosted: higher risk — you control scheduling but must test and execute rolling upgrades; rollback complexity.

Failure-mode operational playbooks- Managed: playbooks focus on client-side mitigation (retry/backoff, circuit breakers, multi-region failover, increase partitions), vendor escalation, and data-export steps.- Self-hosted: playbooks include broker restart procedures, partition reassignment, controller failover, disk replacement, Zookeeper recovery, and disaster recovery runbooks.

Trade-offs & final decision factors- Team maturity and headcount: low ops → managed.- Cost horizon: short-term OPEX acceptable vs long-term at-scale cost sensitivity → choose accordingly.- Compliance/customization needs: strict controls/custom plugins → self-host.- Availability/latency requirements: multi-AZ/multi-region managed offerings reduce complexity; self-host can be optimized but at higher ops cost.

Concrete next steps1. Run a TCO 3‑year model including egress, staffing, and incident MTTR.2. Prototype on managed service for 3 months to validate performance and integration.3. If self-host chosen, budget a dedicated SRE team, CI/CD for infra, and automated playbook runbooks for scaling and failure scenarios.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Scaling and Capacity Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.