Infrastructure Scaling and Capacity Planning Questions

Operational and infrastructure level planning to ensure systems meet current demand and projected growth. Topics include forecasting demand headroom planning and three to five year capacity roadmaps; autoscaling policies and metrics driven scaling using central processing unit memory and custom application metrics; load testing benchmarking and performance validation methodologies; cost modeling and right sizing in cloud environments and trade offs between managed services and self hosted solutions; designing non disruptive upgrade and migration strategies; multi region and availability zone deployment strategies and implications for data placement and latency; instrumentation and observability for capacity metrics; and mapping business growth projections into infrastructure acquisition and scaling decisions. Candidates should demonstrate how to translate requirements into capacity plans and how to validate assumptions with experiments and measurements.

HardTechnical

63 practiced

Design a rigorous load-testing methodology to benchmark the maximum concurrent TCP sessions and connection churn a NAT gateway cluster can sustain. Include traffic generator orchestration, parameter sweeps (connection rate, idle timeout, payload size), statistical analysis to identify capacity limits and failure modes, and how you would extrapolate test results to predict production behavior and required headroom.

Sample Answer

**Overview & goals**I would design a controlled, reproducible test plan to measure: (1) max concurrent TCP sessions the NAT gateway cluster can maintain, (2) maximum new-connection rate (churn) it can sustain without degraded forwarding, and (3) failure modes (timeouts, connection drops, CPU/memory exhaustion, NAT port exhaustion).

**Testbed & tooling**- Use traffic generators supporting stateful TCP (T-Rex or IXIA for scale; TRex for open-source). Supplement with pktgen or custom Scapy scripts for fine-grained control.- Orchestrate agents with Kubernetes or Ansible + a central controller to coordinate ramps, collect telemetry, and trigger probes.- Collect telemetry: NAT instance metrics (CPU, memory, conn table size), kernel metrics (conntrack), network stats (pkt/byte drops, retransmits), and generator stats (latency, SYN/SYN-ACK rates, failures).- Time-sync all hosts (NTP/chrony) and store logs/pcaps centrally.

**Parameter sweep matrix**- Dimensions: - Connection rate (new conns/sec per node): sweep geometric series (10, 30, 100, 300, 1k, 3k, ...). - Concurrent sessions: ramp until resource failure (10k, 50k, 100k, 250k, 500k+). - Idle timeout: default vs reduced (30s, 60s, 300s) to simulate cleanup behavior. - Payload size / throughput per-connection: 0 bytes (keepalive-only), 1KB, 64KB bursts, sustained streaming. - NAT mapping type: symmetric vs endpoint-independent, port reuse policies.- Run each cell multiple times (≥5) to capture variance.

**Test choreography**- Warm-up phase (60–300s) to reach steady-state.- Ramp-up new-connection rate linearly for each sweep to observe thresholds.- Hold periods at target concurrency to check memory/aging behavior.- Controlled ramp-down and recovery to observe cleanup time and session leakage.

**Statistical analysis**- Record time-series of success rate, p99/p99.9 latency, retransmits, CPU, conntable size.- Use change-point detection to find when error rates spike.- Fit survival/decay curves to connection lifetimes to verify timeout behavior.- Use logistic regression or piecewise-linear fits to model probability of failure as function of new-conn rate and concurrency.- Report confidence intervals and coefficient of variation. Create heatmaps showing safe operating regions.

**Identify failure modes**- NAT port exhaustion (peak mapping count equals ephemeral port space) — detect by mapping collisions/failures.- CPU/interrupt saturation — rising p99 latency with increased CPU sys time.- Memory/conntrack leaks — conntable growth without expected decay.- Hash table thrashing or lock contention — increased tail latencies during mid-load.

**Extrapolation to production**- Map testbed scale to production scale using linearity checks: if NAT cluster scales horizontally, verify per-instance limits and aggregate using Amdahl-like model.- Use headroom formula: Required capacity = Observed 95th-percentile production load × safety margin (typically 1.5–2×) + observed burst cushion from churn tests.- If non-linear effects discovered, fit queuing models (M/G/∞ approximations for session arrival + service time) or use measured regression to predict required instances for target SLAs.- Validate extrapolation with a smaller number of higher-fidelity runs at target projected load.

**Deliverables**- Test harness (scripts, configs), raw telemetry, analysis notebook (plots, regression models), recommended capacity table with headroom and failure-mode mitigations (tuning conntrack timeouts, increasing ephemeral port range, horizontal autoscaling triggers).

This methodology gives reproducible, statistically-sound capacity numbers and actionable guidance to size NAT clusters for production with quantified headroom.

HardTechnical

60 practiced

Given uncertain and highly variable business growth projections, design procurement and contract strategies for WAN capacity (committed vs on-demand) to minimize cost while providing sufficient headroom for bursts. Explain financial trade-offs, risk exposure, and propose monitoring triggers and contractual clauses that can be used to scale capacity with measured demand.

HardTechnical

111 practiced

You must perform a non-disruptive rolling upgrade of BGP routers across three geographically separated data centers while maintaining active traffic flows. Provide a detailed step-by-step upgrade procedure, safeguards you would put in place, BGP traffic-engineering techniques (such as local-preference, MED, AS-path prepending) you would use to control traffic during the upgrade, testing/validation checks for each step, and rollback mechanisms.

Sample Answer

**Situation & Goal**I need to upgrade BGP routers in three DCs (A/B/C) with zero traffic disruption and full rollback capability.

**Step-by-step procedure**1. Prep & inventory - Document IOS/OS versions, config, BGP neighbors, route reflectors, RIB/FIB counts, maintenance windows. - Stage images and configs on a jump host; verify signatures.2. Pre-change validation - Baseline metrics (flow samples, BGP session states, route counts, latency). - Create pre-upgrade snapshots and backup configs.3. Canary in DC-A (single router) - Place router into maintenance mode: gracefully clear non-essential sessions; avoid soft-reset all. - Upgrade and reboot. - Re-establish BGP with session-refresh/soft-reconfiguration where supported.4. Verify traffic steering (10–30 min) - Test reachability, confirm prefixes, check BGP Adj-RIBs, RIB-to-FIB, ECMP/hash distribution, and flow continuity.5. Incremental rollout - Repeat per router in DC-A, then DC-B, then DC-C, limiting changes to one router per POD. - Allow >=30–60 min stabilization after each router.6. Post-upgrade validation - Full topology health-check, traffic distribution, convergence times, CPU/mem, error counters.

**Safeguards**- Maintenance VLAN/Out-of-band console access and spare transit links.- Pre-created rollback playbooks and automated config push with versioning.- Scheduling windows and on-call escalation.

**BGP traffic-engineering techniques**- Local-Preference: set higher local-pref in a DC to attract traffic during upgrade (applied on IGP-edge RR or PE).- MED: lower MED on preferred egress to influence remote AS selection without changing local-pref.- AS-path Prepend: add prepends on routers being upgraded to steer traffic away temporarily.- Communities: use well-known/no-export or provider-specific tags to control propagation.- ECMP/Weight: adjust weight on local routers for vendor-specific immediate steering.

**Testing & Validation per step**- Control-plane: show bgp summary, show bgp neighbors, verify route counts.- Data-plane: ping/tcp traceroute to critical prefixes, iperf or live flow compare, BFD session stability.- Convergence: measure route withdrawal/announce latency; confirm no flap storms.- Application tests: synthetic transactions and passive flow checks (Netflow/sFlow).

**Rollback mechanisms**- If issues, immediately: - Reapply pre-upgrade config or restore image via console. - Reintroduce withdrawn routes: remove prepends, revert local-pref/MED. - Use traffic-shift playbooks to move traffic to healthy DCs. - If persistent, failover to standby routers or circuits.- Post-rollback: validate same checks as above and conduct RCA.

I’d run this with staged automation (Ansible + network-test suite), tight monitoring (alerts on BGP session loss, route count delta >5%, packet-loss), and clear abort criteria before each step.

EasyTechnical

70 practiced

List and briefly explain common network-level metrics and indicators you would consider when creating autoscaling triggers for network-reliant services. Cover metrics such as bandwidth utilization, packets-per-second (pps), connection count, flow-table usage, interface errors, and latency. For each metric say whether it is typically a leading or lagging indicator and how you might choose thresholds safely.

EasyBehavioral

72 practiced

Behavioral: Describe a time when you discovered a network capacity or scaling issue under production load. Use the STAR method (Situation, Task, Action, Result) to explain what happened, how you diagnosed it, how you communicated with stakeholders, the mitigation you implemented, and measurable results after the fix.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Scaling and Capacity Planning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.