Capacity Planning and Resource Optimization Questions

Covers forecasting, provisioning, and operating compute, memory, storage, and network resources efficiently to meet demand and service level objectives. Key skills include monitoring resource utilization metrics such as central processing unit usage, memory consumption, storage input and output and network throughput; analyzing historical trends and workload patterns to predict future demand; and planning capacity additions, safety margins, and buffer sizing. Candidates should understand vertical versus horizontal scaling, autoscaling policy design and cooldowns, right sizing instances or containers, workload placement and isolation, load balancing algorithms, and use of spot or preemptible capacity for interruptible workloads. Practical topics include storage planning and archival strategies, database memory tuning and buffer sizing, batching and off peak processing, model compression and inference optimization for machine learning workloads, alerts and dashboards, stress and validation testing of planned changes, and methods to measure that capacity decisions meet both performance and cost objectives.

EasyTechnical

26 practiced

Describe a practical storage and archival strategy for ML training data and model artifacts. Include retention policy, hot vs cold storage selection, cost vs retrieval time trade-offs, encryption and compliance aspects, and how the strategy supports model reproducibility and lineage.

Sample Answer

Situation: We need a practical, compliant archival strategy for ML training data and model artifacts that balances cost, retrieval speed, and reproducibility.

Strategy:- Data classification & retention: - Raw immutable ingests: retain 1–3 years (or per regulation), store full provenance. - Processed training sets / feature snapshots: retain 6–12 months hot, move to cold for 1–3 years. - Model checkpoints & final artifacts: retain all production releases forever; experimental checkpoints: retain 90 days unless promoted.- Hot vs cold storage: - Hot (S3 Standard / SSD-backed object store): active experiments, recent feature tables, serving models — low latency. - Warm (S3 Intelligent-Tiering / infrequent-access): less-used versions with moderate retrieval cost. - Cold (S3 Glacier / archive blob): long-term compliance copies and raw historical data — low cost, higher retrieval latency.- Cost vs retrieval trade-offs: - Use lifecycle policies to transition objects automatically; keep most-accessed artifacts in hot for fast iteration, archive older ones to cut storage costs. Estimate retrieval rates to decide tier thresholds.- Encryption & compliance: - Server-side and client-side encryption (KMS-managed keys); role-based access via IAM, VPC endpoints for private access, audit logging (CloudTrail). Apply PII masking and retention rules per GDPR/HIPAA; include data retention/destruction workflows.- Reproducibility & lineage: - Store immutable dataset snapshots, preprocessing code, dependency manifests (conda/pip/requirements, Dockerfile), training config, random seeds, and evaluation metrics alongside model artifacts in a model registry (MLflow, ModelDB). Link dataset snapshot IDs and code commit hashes in artifact metadata. - Versioning and automated metadata capture enable repeatable retraining and audits.Outcome: Low operational cost through tiering, fast iteration from hot storage, and full reproducibility/compliance via immutable snapshots, encryption, and audited lineage.

EasyTechnical

23 practiced

List common model compression techniques such as quantization, pruning, and distillation. For each technique, briefly explain how it reduces inference resource usage (memory, compute, bandwidth) and indicate one potential drawback or limitation when applied to production models.

HardTechnical

45 practiced

Implement a streaming algorithm in Python to approximate the 95th percentile latency using bounded memory. Provide working code for a P^2 algorithm or t-digest approximation and explain the memory/accuracy trade-offs. Assume latencies arrive as a generator of floats.

Sample Answer

Approach: I'll implement the P² (P-square) algorithm which keeps a fixed small set of markers to approximate a single quantile (here 95th) in streaming fashion with O(1) memory. I’ll also summarize t-digest trade-offs (more accurate, slightly more memory).

python

# python
class P2Quantile:
    """
    P^2 algorithm for a single quantile (q in (0,1)).
    Usage:
      p2 = P2Quantile(0.95)
      for x in latency_stream: p2.add(x)
      estimate = p2.get()
    """
    def __init__(self, q=0.95):
        assert 0 < q < 1
        self.q = q
        self.n = 0
        self.markers = []    # heights h[0..4]
        self.pos = [0,0,0,0,0]   # positions n[0..4]
        self.des = [0, q/2, q, (1+q)/2, 1]  # relative desired positions

    def add(self, x):
        if self.n < 5:
            # bootstrap: store initial samples
            self.markers.append(x)
            self.n += 1
            if self.n == 5:
                self.markers.sort()
                for i in range(5):
                    self.pos[i] = i+1
                self.des = [1, 1+2*self.q, 1+4*self.q, 3+2*self.q, 5]
            return

        # find k: marker interval containing x
        k = 0
        if x < self.markers[0]:
            self.markers[0] = x
            k = 0
        elif x >= self.markers[-1]:
            self.markers[-1] = x
            k = 3
        else:
            for i in range(4):
                if self.markers[i] <= x < self.markers[i+1]:
                    k = i
                    break

        # increment positions
        for i in range(5):
            if i <= k:
                self.pos[i] += 1
            else:
                self.pos[i] += 0
        self.n += 1
        # update desired positions
        for i in range(5):
            self.des[i] += [0, self.q/2, self.q, (1+self.q)/2, 1][i]

        # adjust heights
        for i in range(1,4):
            d = self.des[i] - self.pos[i]
            if (d >= 1 and self.pos[i+1]-self.pos[i]>1) or (d <= -1 and self.pos[i]-self.pos[i-1]>1):
                d_sign = int(d/abs(d))
                # parabolic prediction
                p = self.pos; m = self.markers
                delta = d_sign
                num = delta*(p[i]-p[i-1]+delta)*(m[i+1]-m[i])/(p[i+1]-p[i]) + \
                      delta*(p[i+1]-p[i]-delta)*(m[i]-m[i-1])/(p[i]-p[i-1])
                den = (p[i+1]-p[i-1])
                newh = m[i] + num/den
                # if newh not monotonic, use linear
                if m[i-1] < newh < m[i+1]:
                    m[i] = newh
                else:
                    m[i] = m[i] + delta*(m[i + delta] - m[i])/(p[i + delta] - p[i])
                self.pos[i] += delta

    def get(self):
        if self.n == 0:
            return None
        if self.n < 5:
            return sorted(self.markers)[int((self.n-1)*self.q)]
        return self.markers[2]  # marker 2 estimates the quantile

Key concepts:- P² maintains 5 markers (heights & positions) and updates in O(1) per sample.- Memory: constant (5 floats + small bookkeeping). Very cheap.- Accuracy: Good for moderate distributions and single quantile, but can be biased with extreme tails or highly skewed/ multimodal data.

Trade-offs vs t-digest:- t-digest stores centroids, adaptive compression (configurable), better tail accuracy for high percentiles (95th/99th). Memory grows with desired accuracy but remains bounded by compression parameter.- P²: fixed tiny memory, faster, simpler; less accurate for complex/volatile distributions.- Choose P² when memory is extremely tight and one quantile is needed. Choose t-digest when tail accuracy matters and slightly more memory/CPU is acceptable.

HardBehavioral

22 practiced

Tell me about a time you had to make a capacity-related trade-off that impacted product features or delivery timelines (for example delaying model training to reduce cost or reducing concurrency to meet SLOs). Describe the situation, the options you considered, the decision you made, how you communicated it, and the outcome.

Sample Answer

Situation: At my last company we were rolling out a personalized recommendations model for a high-traffic e-commerce checkout flow. Peak traffic was 10K reqs/s and our initial design used a heavy transformer-based reranker that required GPU pods for low-latency inference. After load-testing, projected infra cost for expected scale would exceed budget by ~3x and risked missing Q4 launch.

Task: I needed to choose a capacity trade-off that kept SLOs (p95 latency < 120ms, availability 99.9%) while delivering a meaningful recommendation feature on schedule.

Action:- I evaluated three options: 1) Delay launch and optimize the transformer (3–4 weeks) to reduce cost. 2) Serve a lightweight distilled model at full concurrency (meets SLOs, slightly lower accuracy). 3) Serve the full model but reduce concurrency via request queuing and feature sampling (meets accuracy, risks user experience).- I ran A/B microbenchmarks: distilled model lost ~3% NDCG but reduced latency by 60% and cost by 70%. Queuing added ~80ms median latency under load and risked SLO breaches.- I chose option 2: deploy the distilled model for initial launch, along with telemetry to monitor user metrics and a clear roadmap to roll in the full model once we completed GPU optimization.- Communication: I prepared a one-page decision brief with trade-offs, performance metrics, cost impact, and rollout plan. I presented it in the weekly product-engineering sync, aligned with PM and infra, and got executive approval. I set up dashboards and an incident playbook.

Result: We launched on time, stayed within budget, and met SLOs. Conversion uplift matched forecast within the A/B error bounds; user engagement dropped negligibly. Over the next 6 weeks we implemented mixed-precision and model sharding, enabling the full model in a targeted high-value cohort without breaking budget. Lesson: quantify trade-offs with experiments, pick reversible decisions that prioritize user experience and delivery, and communicate a clear mitigation/upgrade path.

MediumTechnical

23 practiced

Explain how queuing theory (for example M/M/1 or M/M/c models) applies to capacity planning for synchronous inference services. Show how to estimate required service rate mu and number of parallel servers c given arrival rate lambda and a target average wait time W, and explain limitations of these simple models.

Sample Answer

Start by clarifying metrics: λ = arrival rate (requests/sec). μ = service rate per server (requests/sec). c = number of identical parallel servers. Distinguish Wq (average waiting time in queue) and W (average time in system = Wq + 1/μ). Most SLAs target W or tail percentiles; queuing formulas give means.

M/M/1 (single exponential server, Poisson arrivals):- ρ = λ/μ must be < 1.- W = 1/(μ − λ). So if you want average system latency W_target, choose μ = λ + 1/W_target.- Wq = ρ/(μ(1−ρ)) = λ/(μ(μ − λ)). To solve μ from Wq you get a quadratic: μ^2 Wq − μ(λ Wq + 1) + λ = 0 and take the >λ root.

M/M/c (Erlang C, c parallel servers):- Define ρ = λ/(c μ) < 1.- Compute Erlang C (probability a job waits): P_wait = [ ( (λ/μ)^c / c! ) * ( c / (c − ρ c) ) ] / [ Σ_{k=0}^{c−1} (λ/μ)^k / k! + ( (λ/μ)^c / c! ) * ( c / (c − ρ c) ) ] (Commonly written with ρ = λ/(cμ); ensure ρ<1.)- Then Wq = P_wait * 1/(c μ − λ). And W = Wq + 1/μ.- Given λ and target Wq (or W), choose μ from profiling (1/mean service time). Then find smallest integer c such that computed Wq ≤ target (iterate c upwards). Alternatively, if μ is tunable (e.g., faster model), you can solve by searching (c, μ) pairs that meet capacity and cost constraints.

Practical workflow:1. Measure service time distribution (use mean for μ, but account for variance).2. Use M/M/c to estimate c for required mean latency; iterate with observed μ.3. Validate with load tests; adjust for p95/p99 using simulations or G/G/c approximations.

Limitations:- Assumes Poisson arrivals and exponential service times — real traffic and ML inference latencies often have burstiness and heavy tails.- Erlang C models infinite queue and identical servers; ignores batching, GPU resource contention, cold starts, network delays, backpressure and queue size limits.- Mean-based targets hide tail behavior; SLOs often require p95/p99 guarantees not captured by these means.- Use as first-order capacity guide; always validate with empirical load testing, consider queuing networks, G/G/c, simulation, or reserve headroom for burstiness and degraded nodes.

Key takeaway: M/M/1 and M/M/c give closed-form intuition and quick estimates (μ ≈ λ + 1/W for M/M/1; iterate c for M/M/c), but production capacity planning must layer profiling, empirical tests, and tail-focused analysis.

Unlock Full Question Bank

Get access to hundreds of Capacity Planning and Resource Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.