Performance Engineering and Cost Optimization Questions

Engineering practices and trade offs for meeting performance objectives while controlling operational cost. Topics include setting latency and throughput targets and latency budgets; benchmarking profiling and tuning across application database and infrastructure layers; memory compute serialization and batching optimizations; asynchronous processing and workload shaping; capacity estimation and right sizing for compute and storage to reduce cost; understanding cost drivers in cloud environments including network egress and storage tiering; trade offs between real time and batch processing; and monitoring to detect and prevent performance regressions. Candidates should describe measurement driven approaches to optimization and be able to justify trade offs between cost complexity and user experience.

MediumSystem Design

47 practiced

Design an autoscaling policy for a microservice to minimize cost while meeting a 200ms p95 latency SLO. Specify metrics to drive scaling (e.g., latency, CPU, request queue depth), scale-up/down thresholds and cooldowns, predictive vs reactive scaling options, and how to handle sudden spikes or noisy neighbors.

Sample Answer

**Requirements & goal**Meet p95 ≤ 200 ms while minimizing cost across steady and burst traffic for a microservice in Kubernetes (or cloud autoscaling group).

**Metrics to drive scaling**- Primary: application p95 latency (measured by in-app metrics/exporter).- Secondary: request queue depth (requests waiting), CPU utilization, error rate (5xx).- Guardrail: pod restart rate / OOM events.

**Reactive policy (baseline)**- Scale-up triggers (any of): - p95 > 180 ms sustained for 30s → +25% replicas (min 1 pod) - request queue depth per pod > 50 for 30s → +25% - CPU > 75% for 60s → +20%- Scale-down triggers: - p95 < 120 ms and queue depth per pod < 10 for 5m → -20% (but not below min replicas)- Cooldowns: - After scale-up: 60s before next scale-up action; after scale-down: 5m before next scale-down.

**Predictive / proactive scaling**- Use short-term forecasting (5–15 min) via time-series (Prometheus/ML model) + schedule-based scaling for known traffic patterns (cron/workday).- When forecasted load increase >20% in next 10m, pre-scale up gradually to anticipated target to avoid cold-start latency.

**Handling sudden spikes / burst capacity**- Maintain a small warm buffer (e.g., +2 pods or 20% headroom) during high-risk windows.- Use burstable instance pool / node group with fast start-up (pre-warmed images) or use serverless (FaaS) for sporadic spikes.- Apply request-level backpressure: rate limiting, short queue depth limits and 429 responses with Retry-After where acceptable.

**Noisy neighbor & stability**- Pod anti-affinity and node taints: isolate latency-sensitive pods to dedicated node pool.- CPU/Memory requests set correctly; use QoS classes to prevent eviction; enable cgroup CPU shares.- Monitor noisy metrics; if noisy neighbor detected, evict to dedicated pool or scale node pool horizontally.

**Trade-offs & monitoring**- Prioritize p95 SLO; accept slight cost increase for automated warm buffer and predictive scaling to avoid SLO breaches.- Continuous feedback: alert on p95 > 200ms, track cost per request, tune thresholds via canary experiments.- Observability: trace, p95, queue depth, pod start time; log actions for postmortems.

EasyBehavioral

59 practiced

Tell me about a time you, as a Cloud Architect or senior engineer, identified and executed performance improvements that significantly reduced cloud spend without degrading user experience. Use STAR: describe the situation, the specific actions you took, the measurement-driven approach, the results (quantified), and the trade-offs or risks you managed.

EasyTechnical

52 practiced

Define 'tail latency' and explain why p99 and p999 percentiles matter for user experience. As a Cloud Architect, list three architectural strategies to reduce tail latency across distributed services and briefly explain their cost implications.

HardTechnical

52 practiced

How would you design a global database for a social feed that must serve reads under 100ms for 99% of users worldwide while keeping storage and egress costs reasonable? Discuss consistency model options, replication strategies (multi-master vs read-replicas), caching layers, and fanout-on-write vs fanout-on-read trade-offs.

Sample Answer

**Clarify requirements & constraints**- SLA: 99th percentile read <100ms globally.- Cost: minimize storage and cross-region egress.- Workload: mostly reads (feeds), writes are user posts/likes; read amplification high.

**High-level approach**- Geo-partitioned primary storage + regional read caches/replicas.- Use eventual-consistent read-replicas for cheap, low-latency reads; offer strong consistency for a small subset (recent posts, user timelines) via targeted techniques.

**Consistency model options**- Eventual consistency (default): low-latency, cheap; acceptable for most feeds.- Causal/Read-after-write for UX-sensitive flows: implement session stickiness or per-user master write region.- Strong consistency for critical operations (e.g., payments, privacy flags) using distributed transactions sparingly.

**Replication strategies**- Read-replicas: single-writer, multi-readers. Pros: lower egress (replicate once), simpler conflict model. Cons: replication lag—mitigate with TTL-based staleness SLAs and client-side session reads.- Multi-master (active-active): lower write latency for geo-local users, but increases conflict resolution complexity and cross-region anti-entropy costs—use only if writes are frequent globally.

**Caching layers**- CDN + regional edge caches for static assets.- Global edge cache (e.g., CloudFront + Lambda@Edge) for precomputed short-lived timeline fragments.- Regional in-memory caches (Redis clusters) storing recent timelines and fanout results; place caches close to users to meet 100ms.

**Fanout trade-offs**- Fanout-on-write: precompute and store per-follower timelines in regional storage/caches. Pros: reads are low-latency; lower read compute and egress per request. Cons: high write amplification and storage costs for users with many followers—mitigate by hybrid: fanout-on-write for normal users, fanout-on-read for celebrities.- Fanout-on-read: compute timeline at read time by merging followed users' recent posts. Pros: lower storage and write egress; Cons: higher read latency and CPU—offset with sharded parallel reads, caching of partial merges, and bounded merge window.

**Concrete hybrid design**- Author writes to home region. Fanout pipeline: - If follower count < threshold (e.g., 10k): fanout-on-write to follower timeline storage in follower's home region and regional caches. - If > threshold: mark as “heavy” and keep in creator shard; fanout-on-read merges popular creators at read time with cached recent chunks.- Replicate timeline storage to regional read-replicas asynchronously; use consistent hashing to route reads to nearest replica.- Use edge cache for first N posts of timeline; fall back to regional cache/replica for full timeline.

**Cost & optimization**- Limit replica retention windows, tier cold storage (object store) for older posts.- Compress and batch replication; use differential/CDC replication.- Monitor egress hotspots and move popular creators’ primary shards closer to predominant audiences.

**Trade-offs summary**- Prioritize read locality and caches to meet 100ms for 99% while using hybrid fanout to balance storage/egress. Choose eventual/casual consistency for scale, with targeted strong guarantees where UX requires it.

MediumSystem Design

92 practiced

Design a benchmarking plan for a stateful microservice you plan to migrate to the cloud. Cover objectives, representative workloads to simulate, required metrics (latency percentiles, throughput, CPU, memory, network, IOPS), test harness design, warm-up strategy, and how you will use results to size instances and set SLOs.

Unlock Full Question Bank

Get access to hundreds of Performance Engineering and Cost Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.