Performance Engineering and Cost Optimization Questions

Engineering practices and trade offs for meeting performance objectives while controlling operational cost. Topics include setting latency and throughput targets and latency budgets; benchmarking profiling and tuning across application database and infrastructure layers; memory compute serialization and batching optimizations; asynchronous processing and workload shaping; capacity estimation and right sizing for compute and storage to reduce cost; understanding cost drivers in cloud environments including network egress and storage tiering; trade offs between real time and batch processing; and monitoring to detect and prevent performance regressions. Candidates should describe measurement driven approaches to optimization and be able to justify trade offs between cost complexity and user experience.

MediumSystem Design

53 practiced

Design a caching strategy for a read-heavy API that requires eventual consistency for user profile data. Describe where you'd place caches (client, CDN, edge, app-level, DB), cache invalidation or TTL strategies, and how you'd measure cache hit ratio and its impact on latency and cost.

Sample Answer

**Clarify requirements**- Read-heavy, user profiles, eventual consistency tolerated (stale reads acceptable for short window), scale & cost sensitivity, low read latency.

**High-level placement**- Client: short in-memory/browser cache (1–5s) to reduce duplicate rapid reads.- CDN/Edge: cache profile avatars and public metadata (e.g., display name, picture) using CDN with edge caches (TTL configurable).- App-level (API cache/Redis): authoritative read cache for assembled profile objects; regional Redis clusters for low latency.- DB: write-through/direct to primary DB; use DB as source-of-truth, no DB-level caching beyond DB buffer pool.

**TTL and invalidation**- CDN/Edge: set TTL = 30–300s for public fields; use soft purge (stale-while-revalidate) to serve while refreshing.- App cache: short-ish TTL 60–120s for eventual consistency. For profile edits, emit cache invalidation events to Redis (key delete) via message bus (Kafka/SNS). For user-sensitive changes, perform targeted invalidation + cache-warm by fetching updated profile.- Client: very short TTL + ETag-based validation to minimize wasted bandwidth.

**Consistency model**- Eventual: writes update DB, publish change event; consumers invalidate app cache and send cache-control purge to CDN asynchronously. Accept window of staleness ~ TTL + propagation.

**Measuring and impact**- Cache hit ratio: measure per layer (edge, app) via metrics: hits/(hits+misses). Instrument Redis INFO stats + CDN metrics.- Latency: track p50/p95 read latency split by cache hit vs miss; calculate latency savings = (miss latency - hit latency) * hits.- Cost: compute requests and egress saved = baseline DB reads avoided * cost per DB read / CDN egress saved. Monitor cache memory vs DB RCU costs to optimize TTL.- Alerting: low hit ratio or rising miss latency triggers TTL/invalidation investigation.

**Trade-offs**- Short TTLs reduce staleness but increase origin load; invalidation complexity increases with more layers. Use event-driven invalidation and metrics to tune.

EasyTechnical

43 practiced

Describe common asynchronous processing patterns (e.g., worker queues, event-driven, publish/subscribe) for decoupling services. For each pattern, note when it's appropriate, how it affects latency and cost, and how you'd ensure resiliency and idempotency.

EasyTechnical

51 practiced

Explain the primary drivers of network egress cost in public clouds. For a multi-region application with cross-region reads and writes, list three strategies to reduce egress cost while keeping user-facing latency reasonable.

HardTechnical

60 practiced

Production occasionally shows intermittent latency regressions that you cannot reproduce locally. Describe a plan to instrument and capture sufficient evidence to find the root cause without overwhelming production (sampling, adaptive tracing, flamegraphs, flamegraphs for native code), and how you'd correlate traces with system-level metrics.

Sample Answer

**Clarify goal & constraints**- Capture reproducible evidence of intermittent latency in production with minimal overhead and no PII leakage.- Target 1–5% overhead, safe for peak traffic.

**Plan — phased instrumentation**1. Baseline metrics (low overhead) - Ensure high-cardinality request metrics (p50/95/99), error rates, throughput, host/container tags, GC/CPU/io latencies in Prometheus.2. Sampled distributed tracing - Enable tracing (e.g., OpenTelemetry) with dynamic sampling: default low sample rate (0.1%) and higher rate for slow requests (adaptive tail sampling). - Use header-based forced sampling for canaries or customer reports.3. Adaptive tracing rules - Trigger full traces when: latency > p95 threshold, error returned, or unusual spike in retry rate. - Use short retention for full traces; store trace IDs and summary longer.4. Lightweight stack-profiler & flamegraphs - For services with native or managed runtimes, periodically collect async stack samples (e.g., 1–10 Hz, 1–5% sessions) and generate flamegraphs. For native code, use eBPF-based profilers (BCC/pyflame/FlameGraph) to avoid pausing. - When adaptive trace triggers, increase profiler sampling for affected host/service window (e.g., 30s–2m).5. Correlation and enrichment - Attach a unique trace_id to each request and push to logs/metrics. Export trace_id to Prometheus as labels or to a central telemetry store (Elasticsearch/ClickHouse). - Correlate trace timelines with host-level metrics (CPU, iowait, network, disk latency), container resource throttling, and kernel events (dropped interrupts).6. Analysis workflow - Use trace_id to join traces → logs → flamegraph snippets. Build dashboards that show traces overlaid on host metrics and service heatmaps. - Automate anomaly detection: when many slow traces correlate with elevated iowait or GC, open an incident with evidence and suggested root causes.7. Safety & limits - Rate-limit escalation, sample budgets per service, and rollbacks. Mask sensitive fields before storage.

Result: low-overhead continuous visibility plus on-demand deeper sampling that produces traces + flamegraphs correlated to system metrics to find root causes of intermittent latency.

MediumTechnical

58 practiced

A production service sometimes experiences high tail latency (p99) during traffic bursts. Design an experiment to benchmark and reproduce tail behavior. Describe workload characteristics to simulate, the metrics to capture, and how you would isolate whether the bottleneck is CPU, IO, locks, GC, or network.

Sample Answer

**Goal & scope**Reproduce p99 spikes under controlled bursts, measure system behavior, and isolate root cause among CPU, I/O, locks, GC, or network.

**Experiment design**- Create repeatable traffic generator that can produce steady baseline + configurable bursts (duration, QPS ramp, concurrent connections, request size, payload type). Use tools like wrk2, k6, or custom client with fixed inter-arrival distribution (Poisson + occasional heavy-tailed inter-arrival to mimic real bursts).- Run experiments against a stable deployment (single instance and scaled cluster) with identical input each run; vary one parameter at a time (burst magnitude, concurrency, payload).

**Workload characteristics to simulate**- Short vs long requests mix (90/10), variable payload sizes, connection reuse vs new connections, authentication/path variations, backend dependency latencies (simulate downstream slow calls).

**Metrics to capture**- Latency percentiles (p50, p90, p99, p999) and request traces- CPU utilization per core, load average- Disk I/O: IOPS, queue depth, latencies- Network: bandwidth, packets/sec, retransmits, socket queue lengths- Memory: RSS, page faults, swap- GC: pause times, allocation rate, heap size (for managed runtimes)- Locks/contention: thread states, mutex wait times, lock profiling- System call rates, context switches- Application-level logs and traces (distributed tracing, correlation IDs)

**Isolation checklist (how to identify bottleneck)**- CPU: corroborate high p99 with >80–90% CPU and per-thread hot spots using perf/top/htop; flamegraphs to find hot functions.- I/O: high I/O wait, high disk latency, queue depth; reproduce with fio; if removing disk dependency (cache) removes spikes, I/O is culprit.- Locks/contention: thread dump shows many threads blocked; latency correlates with mutex wait profiles; use lock profiling tools (perf lock, JVM async-profiler).- GC: observe GC pause spikes aligned with latency; heap pressure and long GC pauses indicate GC-induced tail.- Network: packet drops, retransmits, increased latency between components, socket backlog growth; test by saturating link or adding artificial latency with tc/netem to reproduce.

**Procedure**- Run baseline + burst while collecting metrics (Prometheus/Grafana, perf, eBPF tools, tcpdump, strace/blkparse).- Introduce targeted mitigations (pinning threads, increasing connection pool, warming caches, tuning GC) to confirm causality.- Repeat experiments, document configuration and results.

**Success criteria**- Deterministic reproduction of p99 spikes and identification of dominant cause with supporting metrics and validated mitigation that reduces p99 under same burst.

Unlock Full Question Bank

Get access to hundreds of Performance Engineering and Cost Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.