Performance Engineering and Cost Optimization Questions

Engineering practices and trade offs for meeting performance objectives while controlling operational cost. Topics include setting latency and throughput targets and latency budgets; benchmarking profiling and tuning across application database and infrastructure layers; memory compute serialization and batching optimizations; asynchronous processing and workload shaping; capacity estimation and right sizing for compute and storage to reduce cost; understanding cost drivers in cloud environments including network egress and storage tiering; trade offs between real time and batch processing; and monitoring to detect and prevent performance regressions. Candidates should describe measurement driven approaches to optimization and be able to justify trade offs between cost complexity and user experience.

MediumSystem Design

47 practiced

Design a canary deployment strategy for a latency-sensitive service where tail latency matters. Explain traffic weights, metrics for success (including p99), warmup behaviors, and automated rollback triggers. How would you avoid false positives due to natural traffic variability?

Sample Answer

Requirements and constraints:- Latency-sensitive service with strict tail latency SLOs (example: p99 < 100ms, p95 < 40ms).- Low risk of user-visible regressions; need automated rollout/rollback.- Traffic patterns may vary (diurnal spikes, A/B cohorts).

High-level canary plan:1. Canary scope and size- Start with small fraction: 0.5–1% traffic or a fixed small pool of users (e.g., internal/test tenants).- Target absolute request counts: ensure canary sees >= 1k requests/min for stable stats; if traffic is low, extend duration.

2. Progressive traffic weights- Example schedule (hold-and-verify): 1% → 5% (10–15 min hold) → 20% (15–30 min) → 50% (30 min) → 100%- Each step requires passing automated checks before proceeding.

3. Metrics for success (primary & secondary)- Primary: p99 latency compared to baseline (relative + absolute). Accept if new_p99 <= baseline_p99 * 1.05 and new_p99 <= absolute_limit (e.g., 100ms).- Secondary: p95, p50, error rate (5xx), success rate, request rate, CPU/memory, GC pause times, thread-pool saturation, queue lengths, tail-QoS signals.- Use both relative (percent change) and absolute thresholds. Require both to be satisfied.

4. Warmup behavior- Gradually increase traffic to give caches, JIT, connection pools time to stabilize.- Run “cold-start” synthetic warmup: preheat caches by issuing representative load to canary instances before routing real user traffic.- Include a stabilization window after each ramp equal to at least one p99 measurement interval (e.g., 5–15 minutes) and a few GC cycles.

5. Automated rollback triggers- Hard triggers: error rate > threshold (e.g., +0.5% absolute or 2x baseline) OR absolute errors above SLO.- Latency triggers: p99 increases by >10% AND absolute p99 exceeds SLO; OR sustained p99 delta for >2 consecutive measurement windows.- Resource triggers: CPU/heap > 90% or thread saturation.- Combine signals: require either a single catastrophic signal (high error spike) OR at least two correlated degradations (p99 + resource).- On trigger: immediate traffic cutback to previous safe weight and auto-create incident with logs, traces, and metrics snapshot.

6. Avoiding false positives (statistical rigor)- Baseline: maintain rolling baseline windows (same time-of-day past N days) to account for diurnal effects.- Use statistical tests: compare distributions (e.g., bootstrap or Mann–Whitney U) rather than single-point percentiles; require significance (p < 0.01) and effect size (e.g., >5% change).- Use aggregated windows and consecutive failures: only roll back after N consecutive failing windows (e.g., 2 of 3).- Control groups: route fixed percentage to control (previous release) and compare canary vs control concurrently to cancel noise.- Outlier mitigation: cap influence of single large requests by tracking request counts and using trimmed percentiles or median-of-windows.

7. Observability & automation- Collect per-request traces, histograms (HDR or HdrHistogram), and expose p99 computed from histograms.- Automated canary orchestration (CI/CD) integrates checks, runs synthetic tests, and applies ramps/rollbacks.- Post-rollout diagnostics: capture heap dumps, thread dumps, flame graphs, and error traces.

Trade-offs:- Slower ramps reduce risk but increase time-to-deploy. Use risk-based policy: for critical releases, enforce slower holds and more telemetry.- Statistical rigor reduces false positives but may delay detection of real regressions; tune significance and windows for acceptable time-to-detect.

This strategy balances safety (control groups, statistical tests, multi-signal triggers) with practicality (progressive weights, warmup, automation) to protect tail latency SLOs while enabling continuous delivery.

EasyTechnical

63 practiced

Define a latency budget in the context of an API and explain how it differs from throughput targets. Given a microservice with 200 req/s average and burst to 5k req/s, describe a simple approach to allocate a latency budget across frontend, middleware, and database layers.

EasyTechnical

44 practiced

Explain 'right-sizing' compute instances for a containerized service. Describe a measurement-driven process to determine right sizes for VMs or container resource requests/limits for a service with diurnal traffic and 99.9% availability SLO.

Sample Answer

Situation: We have a containerized service with diurnal traffic and a 99.9% availability SLO. Right-sizing means choosing VM types and Kubernetes pod resource requests/limits so we meet the SLO reliably while minimizing cost.

Measurement-driven process:

1. Gather telemetry- Collect CPU, memory, thread/goroutine, latency, and request-rate metrics at high resolution (1m) over multiple diurnal cycles.- Record p50/p95/p99 latencies, error rates, and instance/pod-level resource utilization.

2. Characterize load patterns- Identify baseline, peak, and ramp windows; compute percentiles of CPU/RAM per pod at baseline and peak (e.g., 95th CPU usage during peak).

3. Determine target utilization and headroom- For availability SLO 99.9%, target pod CPU utilization ~50–60% at peak (lower if noisy or GC-heavy). Memory target should exceed observed 99th percentile plus buffer (e.g., 15–25%).

4. Translate to requests/limits- Set request = observed 50–75th percentile usage + safety margin; set limit = p99 usage + additional headroom or a multiplier (1.2–1.5) to avoid OOMs/CPU throttling.- Example: if p95 CPU = 400m, p99 = 600m → request = 300–400m; limit = 800–900m.

5. Choose VM instance sizes- Pack an integer number of pods onto VMs at desired utilization (e.g., 60% CPU). Consider node overhead (kube-reserved) and pod density. Prefer instance types that minimize fragmentation (e.g., 8 vCPU if pods of ~1 vCPU).

6. Autoscaling & safety- Use HPA (CPU/requests or custom metrics) for pod scaling and Cluster Autoscaler for nodes. Consider VPA (recommendation mode) for requests or controlled mode for automated adjustments.- Configure PodDisruptionBudget and Pod Anti-Affinity to preserve availability during scale events.

7. Validate with load tests and chaos- Run load tests across diurnal patterns and verify p99 latency and error rate within SLO. Inject node drains and ensure autoscaling meets demand within acceptable convergence time.

8. Iterate with continuous measurement- Monitor cost per successful request, observe during multiple cycles, and tighten margins where safe. Automate alerts on sustained utilization shifts.

Trade-offs and notes:- Tighter requests save cost but increase risk of throttling/OOMs; overly conservative sizing wastes money.- For 99.9% SLO prioritize availability: smaller cost savings in exchange for more headroom and faster autoscaling policies.

This process produces reproducible, measured requests/limits and VM choices that balance reliability and cost for diurnal traffic patterns.

HardTechnical

47 practiced

Tail latency is hurting 0.5% of users due to queuing and head-of-line blocking in a downstream service. Propose system-level changes to reduce p99: hedged requests, priority queues, admission control, timeouts, and client-side fallbacks. For each change, explain expected impact on latency, throughput, and cost.

Sample Answer

Situation: 0.5% of users experience p99 spikes caused by queuing and head-of-line blocking in a downstream service. Below are system-level mitigations, their expected impact on latency, throughput, cost, and implementation notes.

1) Hedged requests (duplicate-request policy)- Latency: Reduces p99 significantly by racing a second request after a short delay (e.g., 10-50ms) — often cuts long tails.- Throughput: Increases downstream request rate (extra load proportional to hedging frequency); negligible for rare tails, larger if aggressive.- Cost: Higher CPU/network and request charges; potential extra cost for downstream services.- Notes: Use adaptive hedging (only when latency > threshold), idempotency or de-duplication at receiver, measure wasted requests and back off under sustained overload.

2) Priority queues (request classification + differentiated scheduling)- Latency: Lowers p99 for high-priority traffic by avoiding head-of-line blocking from expensive lower-priority jobs.- Throughput: Overall throughput unchanged but lower-priority requests may see higher latency; avoids systematic tail amplification for critical flows.- Cost: Small infra/software cost to implement; possible need for more instances if lower-priority backlog grows.- Notes: Define clear priority rules, fairness controls, starvation prevention (weighted fair queuing).

3) Admission control (reject/ shed low-value work when overloaded)- Latency: Improves p99 for accepted requests by preventing queues from growing; rejects increase error rate but protects latency SLO.- Throughput: Reduces accepted throughput under overload; keeps system operating within capacity.- Cost: Saves backend resources; may require capacity planning to align SLOs with business priorities.- Notes: Implement graded degradation, circuit breakers, and clear client signals (429/503 + Retry-After).

4) Timeouts and retries (client-side timeouts + backoff)- Latency: Prevents clients from waiting indefinitely; reduces cascading queues if combined with proper retry/backoff.- Throughput: Avoids request pile-up from long waits; aggressive retries can worsen load if not exponential and capped.- Cost: Minimal; careful tuning avoids wasted work.- Notes: Use conservative timeouts for p95 and shorter for p99-sensitive flows; implement jittered exponential backoff and retry budgets.

5) Client-side fallbacks (cached responses, degraded features, stale read)- Latency: Eliminates many downstream calls, dramatically improving p99 for affected users.- Throughput: Lowers downstream load, improving overall system headroom.- Cost: Development/operational cost to build and maintain fallbacks and cache invalidation; potential consistency trade-offs.- Notes: Use TTLs, stale-while-revalidate, and clear UX signaling when degraded.

Recommendation: Combine measures — prioritize critical traffic, apply conservative timeouts + retries, add hedging selectively for rare long tails, and implement admission control + fallbacks for overload protection. Instrument: measure p99, hedged hit rate, wasted work, rejection rate, and business impact (errors vs latency) to tune policies.

MediumTechnical

61 practiced

You have a service that averages 500 requests/s and needs to ensure p95 latency < 200ms with 99.9% availability. Each pod can handle 50 concurrent requests with comfortable CPU/memory. Calculate the minimum number of pods needed to meet target under normal load and with a 2x burst, and explain how much headroom you'd keep for autoscaling and node failures.

Unlock Full Question Bank

Get access to hundreds of Performance Engineering and Cost Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.