Infrastructure Scaling and Capacity Planning Questions
Operational and infrastructure level planning to ensure systems meet current demand and projected growth. Topics include forecasting demand headroom planning and three to five year capacity roadmaps; autoscaling policies and metrics driven scaling using central processing unit memory and custom application metrics; load testing benchmarking and performance validation methodologies; cost modeling and right sizing in cloud environments and trade offs between managed services and self hosted solutions; designing non disruptive upgrade and migration strategies; multi region and availability zone deployment strategies and implications for data placement and latency; instrumentation and observability for capacity metrics; and mapping business growth projections into infrastructure acquisition and scaling decisions. Candidates should demonstrate how to translate requirements into capacity plans and how to validate assumptions with experiments and measurements.
HardSystem Design
68 practiced
Your active-active multi-region database uses synchronous replication, but write latency is suffering due to geographic distance. Provide alternative architectures and migration steps: asynchronous replication with reconciliation, leaderless/CRDT approaches for eventual consistency, or region-local writes with global change propagation. Discuss trade-offs in latency, consistency, and recovery.
Sample Answer
**Clarify constraints & goal**- Need lower write latency for geographically distant regions while preserving acceptable consistency and recovery SLAs. Assume: relational or distributed KV store, current sync replication across N regions, RPO=0, RTO low.**Option A — Asynchronous replication + reconciliation**- High-level: Primary writes locally, replicate to other regions asynchronously; run background reconciliation/CDC with causal metadata (timestamps, version vectors).- Migration steps: 1. Enable async replication in parallel for a subset of tenants/keys. 2. Add write-ahead logs and durable local commit before ack. 3. Deploy CDC + reconciliation worker; surface conflict detection metrics. 4. Gradually increase traffic; implement compensation logic for detected conflicts.- Trade-offs: - Latency: local write latency minimal. - Consistency: eventual; potential lost-update or ordering anomalies until reconciled. - Recovery: faster local recovery; reconciliation needed for correctness.**Option B — Leaderless / CRDT (eventual consistency)**- High-level: Each region accepts writes; merge using commutative CRDTs or vector clocks.- Migration steps: 1. Model domain objects as CRDTs where possible (counters, sets, maps). 2. Implement client-side or middleware merge; run canary features on non-critical flows. 3. Instrument merge outcomes and fallbacks for non-CRDT data.- Trade-offs: - Latency: best (local). - Consistency: eventual with deterministic merges; no global linearizability. - Recovery: robust to partitions; merging on restore.**Option C — Region-local writes + global change propagation (sharding by region or affinity)**- High-level: Route writes to a regional owner for a key-space; replicate ownership mapping globally.- Migration steps: 1. Define partitioning/affinity rules; provision regional owners. 2. Implement routing layer (sticky sessions, DNS/anycast + region-aware proxies). 3. Migrate subsets of keys/users to regional owners, monitor cross-region read patterns.- Trade-offs: - Latency: low for local owner; cross-region operations incur extra hop. - Consistency: can provide strong consistency for owned keys; global operations require coordination. - Recovery: owner failover complexity; need takeover protocols and leader election.**Recommendation & operational notes**- If writes are latency-sensitive but tolerable eventual consistency: CRDTs or async with reconciliation.- If some data require strong consistency: combine region-local ownership for critical keys + async/CRDT for others.- Key operational items: observability (conflicts, replication lag), replayer tools for reconciliation, well-tested failover procedures, and clear SLAs per data class.
HardTechnical
67 practiced
You run sporadic batch ML inference jobs that require thousands of CPU cores for short bursts. Propose an orchestration design and pricing model that combines job scheduling, spot/preemptible instances, autoscaling, checkpointing, and reserved capacity to minimize cost while meeting latency/throughput goals. Explain how you handle preemption and fairness between jobs.
Sample Answer
**Approach summary (one line)**I’d build a scheduler that mixes spot/preemptible instances with a small pool of reserved/on‑demand capacity, using autoscaling, checkpointing and fair-share gang scheduling to minimize cost while meeting SLAs.**Architecture**- Central scheduler (Kubernetes custom controller or YARN-like) + worker fleet manager that requests spot and reserved VMs from cloud provider.- Job queue with metadata: cpu/core request, runtime estimate, latency/throughput SLA, priority, checkpoint frequency.**Scheduling & Autoscaling**- Autoscaler requests spot instances aggressively; maintains a baseline reserved pool sized for worst-case SLA headroom.- Scale-up: allocate spot first, fall back to reserved/on‑demand if not enough spot within X seconds to meet latency targets.- Scale-down: drain idle nodes; use bin-packing to consolidate.**Checkpointing & Preemption handling**- Enforce application-level incremental checkpoints (S3/remote store) every N minutes or logical epochs.- On spot preemption notification: scheduler issues in‑cluster SIGTERM handler to checkpoint; migrate remaining tasks to warm reserved nodes.- If preemption happens before checkpoint, roll back to last checkpoint and reschedule with exponential backoff.**Fairness & Prioritization**- Weighted fair-share per team/job class: each job has weight and SLA; scheduler enforces minimum guaranteed share via reserved capacity and max share via quotas.- Gang-aware scheduling for parallel jobs: only dispatch when enough capacity available (avoid partial progress cost).- Preemption policy: low-priority spot tasks evicted first; use “preemption budgets” to limit how often a job can be preempted; long-running high‑priority jobs get placement on reserved pool.**Pricing model (simple expected cost)**
- Choose reserved capacity to cover baseline SLA at lowest effective hourly cost; buy R cores such that P(preemption-induced fallback) * cost_on_demand < savings from using fewer reserved cores.- Monitor actual savings and adapt R monthly using observed spot availability and job preemption statistics.**Tradeoffs & metrics**- Track: job completion latency, throughput, preemption rate, cost-per-job. Tune checkpoint interval vs. wasted work.- Tradeoffs: more frequent checkpoints increase I/O/storage cost but reduce recompute; larger reserved pool reduces preemption risk but raises baseline cost.This design optimizes cost while bounding SLA risk via reserved headroom, intelligent fallback, and fairness controls.
HardTechnical
111 practiced
Propose a quantitative method to compute the probability that capacity will breach SLA given uncertain traffic forecasts. Describe the required inputs (forecast distribution, capacity model, headroom policy), the statistical approach (Monte Carlo simulation or analytic convolution), and how to derive buffer sizes to meet a specified business risk tolerance (e.g., <0.1% chance of breach per month).
Sample Answer
**Problem framing & required inputs**- Forecast distribution of demand over the target window (per-minute/hourly): mean μ(t) and uncertainty (std σ(t)) or full predictive distribution (e.g., log-normal or ensemble quantiles).- Capacity model C: mapping of provisioned units -> service capacity (requests/sec, throughput) including degradation modes.- Headroom policy H: how extra capacity is allocated (static buffer, auto-scale delay, cooldown).- SLA definition: threshold S (e.g., latency/queue length) and business risk tolerance p* (e.g., 0.1% monthly breach).**Statistical approaches**1) Monte Carlo simulation (recommended for complex, non-Gaussian, time-correlated forecasts)- Sample N demand trajectories from forecast distribution (preserving temporal correlation).- For each trajectory apply capacity model + headroom policy to compute instantaneous utilization U(t) and whether U(t) > C -> breach event.- Aggregate: estimate breach probability P̂ = (# trajectories with ≥1 breach in month) / N. Increase N until standard error sqrt(P̂(1-P̂)/N) meets precision.2) Analytic convolution (when linear, Gaussian, and independent)- If demand and capacity uncertainty are Gaussian and independent, compute distribution of margin M = C - D.- Breach probability = P(M < 0) = Φ( (μ_M) / σ_M ) where μ_M = C - μ_D, σ_M^2 = σ_C^2 + σ_D^2.- Use convolution for time aggregation.**Deriving buffer sizes**- Solve for buffer B so that P_breach(B) ≤ p*.- In MC: run binary search on B (increase capacity C := baseline + B) and re-estimate P̂ until P̂ ≤ p*.- Analytic: set
text
B = μ_D - C_base + z_{p*} * σ_M
where z_{p*} is inverse normal quantile for tail p* (use extreme-value correction for rare p*).**Practical considerations**- Model auto-scale latency, correlated demand spikes (use copulas or bootstrapped residuals), and seasonality.- Validate with historical backtesting and include safety margin for model risk.- Example: for p*=0.1% monthly (~3.3e-3 daily), z≈2.7; compute B accordingly and verify via MC.
EasyTechnical
63 practiced
You manage a three-tier application: stateless front-end behind a load balancer, an API tier with moderate CPU and occasional spikes, and a stateful database. For the front-end and API tiers, describe which autoscaling metrics you would use (CPU, memory, request latency, concurrent requests, queue length, custom app metrics), the type of policy (target-tracking, step, scheduled), and how you'd choose cooldowns, evaluation periods, and min/max instances to avoid oscillation.
Sample Answer
**Situation & approach (short)** I’d treat the front-end and API differently because the front-end is fully stateless and latency-sensitive, while the API has sustained moderate CPU with occasional spikes.**Front-end (stateless, latency-sensitive)** - Metrics: target-tracking on request latency or concurrent requests; fallback on requests/sec per instance. Also use unhealthy/error-rate as custom metric. - Policy: target-tracking autoscaling (keeps latency at target), with a short step policy for sudden bursts. - Cooldown/eval: short cooldown (30–60s), evaluation window 30–60s. - Min/max: min = enough for baseline traffic + redundancy (e.g., 2–3), max = cap by budget/throughput.**API tier (moderate CPU, spikes)** - Metrics: primary = CPU utilization (target-tracking for steady load), secondary = request queue length or inflight requests for spike detection, plus custom app metric (e.g., 95th percentile latency or job backlog). - Policy: combine target-tracking on CPU with step scaling triggered by queue-length thresholds to add capacity quickly on spikes. - Cooldown/eval: longer cooldown (3–5 min) for CPU target-tracking to avoid thrash; shorter immediate evaluation (30–60s) for queue-based steps. Use multiple evaluation periods (3 samples) for CPU. - Min/max: min sized for baseline throughput; max sized to absorb known peak plus safety buffer.**Oscillation prevention (general)** - Use stabilization windows or evaluation periods (3 consecutive periods) before scaling down. - Add hysteresis by setting different thresholds for scale-up vs scale-down. - Use scheduled scaling for predictable diurnal patterns. - Monitor and tune thresholds after deployment; prefer conservative scale-downs and faster scale-ups.
EasyTechnical
64 practiced
Explain the differences and trade-offs between deploying across multiple Availability Zones (AZs) within a region versus across multiple geographic regions. Discuss implications for high availability, latency, data residency, cost (network egress), and operational complexity.
Sample Answer
**Direct answer (summary)**Deploying across AZs gives low-latency redundancy inside a region; multi-region gives stronger disaster isolation and data locality but increases latency, cost, and operational complexity. Choose AZs for most HA needs; add regions for DR, compliance, or global scale.**High availability & disaster tolerance**- AZs: protects against rack/network power failures; fast failover (minutes). RTO/RPO typically small when using synchronous or quorum replication.- Regions: protects against entire-region outages (natural disasters, provider faults). Required for true geographically isolated DR.**Latency**- AZs: sub-ms–tens-ms between AZs—good for synchronous replication and user-facing services.- Regions: inter-region latency often tens–hundreds ms; avoid sync writes—use async replication or read replicas.**Data residency & compliance**- AZs: data stays in single country/region — may satisfy many residency rules.- Regions: necessary when legal requirements mandate data reside in multiple countries or avoid specific jurisdictions.**Cost (egress & resources)**- AZs: usually no or low intra-region transfer costs.- Regions: inter-region egress charges can be significant for heavy replication or cross-region traffic; duplicate infrastructure raises compute/storage costs.**Operational complexity**- AZs: simpler—same region tooling, single control plane, fewer network/config differences.- Regions: more complex networking (VPN/Transit), DNS failover, config drift, testing DR, cross-region backup strategies, and multi-region deployment pipelines.**Practical guidance**- Start with multi-AZ for HA. Add a passive or active-active secondary region if regulatory, RTO/RPO, or global latency needs demand it. Quantify trade-offs: acceptable RTO/RPO, egress cost budget, and runbooks for cross-region failover.
Unlock Full Question Bank
Get access to hundreds of Infrastructure Scaling and Capacity Planning interview questions and detailed answers.
