Performance Engineering and Cost Optimization Questions

Engineering practices and trade offs for meeting performance objectives while controlling operational cost. Topics include setting latency and throughput targets and latency budgets; benchmarking profiling and tuning across application database and infrastructure layers; memory compute serialization and batching optimizations; asynchronous processing and workload shaping; capacity estimation and right sizing for compute and storage to reduce cost; understanding cost drivers in cloud environments including network egress and storage tiering; trade offs between real time and batch processing; and monitoring to detect and prevent performance regressions. Candidates should describe measurement driven approaches to optimization and be able to justify trade offs between cost complexity and user experience.

MediumTechnical

55 practiced

Serverless functions in production suffer occasional cold starts that violate a 100ms P95 latency SLO. Describe mitigation techniques (for example provisioned concurrency, keep-alive warmers, reducing package size), the cost and operational trade-offs of each, and an experimental plan to measure whether each approach meets the SLO while minimizing incremental cost.

Sample Answer

**Approach summary**Briefly: evaluate three mitigations (provisioned concurrency, keep‑alive warmers, package/runtime optimizations), compare cost & ops trade-offs, then run controlled experiments measuring P95 and incremental cost.

**Mitigations, trade-offs**- Provisioned Concurrency (e.g., AWS Lambda) - Benefit: near-zero cold starts for provisioned count. - Cost: continuous hourly charge per provisioned unit + execution cost; scales poorly for spiky traffic. - Ops: simple to configure, needs autoscaling policy and monitoring.- Warmers / Keep‑alive pings - Benefit: low incremental cost if scheduled carefully; reduces cold-start frequency. - Cost: extra invocations (small execution cost); risk of ineffective coverage for scale events or platform evictions. - Ops: requires scheduler (CloudWatch Events / EventBridge), health-check logic, and telemetry to detect failures.- Reduce package size / faster runtimes - Benefit: lowers cold-start latency without recurring cost. - Cost: engineering time to strip deps, use native libs or smaller runtimes (Go, Node 18), use layers. - Ops: CI changes, testing, dependency management; long‑term win.

**Experimental plan**1. Baseline: collect 1–2 weeks of production-like traces; measure P95, invocation patterns, mem/CPU.2. A/B tests with traffic split (20/80): - Enable provisioned concurrency for 20% of traffic; measure P95 and incremental cost for provisioned units. - Run scheduled warmers for 20% target functions; measure P95 and extra invocation cost. - Deploy optimized package to 20% of traffic; measure P95 and engineering effort.3. Metrics: P95 latency, error rate, cost delta (provisioned hourly + invocation), and operational alerts.4. Success criteria: approach that brings P95 ≤100ms with lowest total cost per hour and acceptable ops burden.5. Rollout: choose best approach or hybrid (provision for steady base, warmers for burst, optimize code) and monitor for 2 weeks.

This plan balances measurable performance gains, cost accounting, and operational complexity—allowing data-driven selection.

MediumTechnical

43 practiced

A production API shows occasional, significant P99 latency spikes while P50 and P95 remain relatively stable. Describe a step-by-step investigation plan covering application tracing, runtime (GC, thread pool), database checks, network and infra measurements. Name specific tools or metrics you'd use at each step and quick mitigations you might try while the investigation continues.

Sample Answer

**High-level goal**Find root cause of occasional P99 latency spikes with low P50/P95 by isolating layers: app traces, runtime, DB, network/infra. Triage quickly while collecting detailed telemetry.

**1) Application tracing**- What I do: Inspect distributed traces for P99 transactions to see slow spans and hotspots.- Tools/metrics: Jaeger/Zipkin, AWS X‑Ray, Datadog APM; look at span duration, dependency breakdown, error tags.- Quick mitigations: Enable sampling of full traces for recent window; add temporary request logging for P99 trace IDs.

**2) Runtime (GC, thread & connection pools)**- What I do: Correlate spike timestamps with JVM/CLR/Go runtime metrics.- Tools/metrics: Prometheus + Grafana, CloudWatch (JVM GC pause time, heap usage, safepoint, thread count), async queue latency, thread-pool/worker utilization.- Quick mitigations: Increase thread pool or worker concurrency, tune GC settings, restart unhealthy instance(s), set backpressure/circuit-breaker on endpoints.

**3) Database**- What I do: Check slow queries, connection pool exhaustion, locks, replication lag.- Tools/metrics: pg_stat_statements, MySQL slow_log, RDS Performance Insights, Cloud SQL Insights, Datadog DB monitoring.- Quick mitigations: Add read-replicas, increase DB connection pool, enable query result caching (Redis), kill blocker queries, add indexes for culprit queries.

**4) Network & infra**- What I do: Verify network latency, packet drops, host-level contention, storage latency.- Tools/metrics: VPC Flow Logs, tcpdump/pcap, 5xx counts, EC2/EBS metrics (latency, IOPS), ENI metrics, Kubernetes CNI metrics, k8s pod events.- Quick mitigations: Route around bad AZ, cordon/replace node, scale out service, move to instances with better network/IO, enable connection reuse.

**5) Correlation & automated detection**- Correlate trace IDs with logs, metrics, and events; use alerting for p99 > threshold and runbook actions.- Tools: Grafana alerting, CloudWatch Alarms, Datadog monitors, Sentry/Logs.

**6) Longer-term fixes & root-cause steps**- Capture flamegraphs/profiler on affected instances, reproduce with load tests, add request-level timeouts/retries with jitter, capacity planning.- Document changes, add synthetic p99 monitors, and schedule postmortem.

This plan lets me triage quickly, apply short-term mitigations to reduce user impact, and gather the telemetry necessary for a permanent fix.

EasyTechnical

55 practiced

Explain asynchronous processing patterns such as background workers, message queues, and event streams. Describe when to prefer asynchronous designs over synchronous ones, and how asynchronous architectures influence capacity planning, operational complexity, and cost profiles.

HardTechnical

61 practiced

Create a plan to instrument a backend service to measure the cost and latency contributions of serialization/deserialization and network I/O. Specify which trace spans, metrics, and profiling data you would collect, how to attribute end-to-end latency to serialization vs transport, and what experiments you would run to justify changing formats or enabling compression.

Sample Answer

**Plan overview**- Objective: measure and attribute end-to-end request latency and cost between serialization/deserialization (ser/de) and network transport; collect evidence to justify format or compression changes.

**Tracing spans**- Instrument request path with these spans (parent = request): - span.request.handler — application processing - span.serialize.send — time to serialize outbound payload - span.network.send — socket write / bytes queued / TCP ack wait - span.network.rtt (optional) — client-side TCP/HTTP connect/handshake - span.deserialize.recv — time to deserialize inbound payload - span.network.recv — socket read / time-to-first-byte / time-to-last-byte

Include attributes: payload.size.bytes, compressed.flag, serialization.format, compression.type, peer.ip, transport.protocol, retries.

**Metrics to collect**- Per-endpoint histograms: request.latency.total, latency.serialize, latency.network.send, latency.network.recv, latency.deserialize- Counters: bytes.sent, bytes.recv, compression.ratio, serialization.cpu-ms- Cost estimates: CPU-seconds per 1M requests, network GB per month per endpoint

**Profiling & sampling**- CPU profiles (pprof/jfr) sampled during high-latency requests; annotate profiles with active span ID.- Flamegraphs for hot serialization code paths; allocation profiles to measure GC overhead from formats (JSON vs protobuf).- Packet-level captures (pcap) for representative flows to validate transport timing and retransmits.

**Attribution method**- Use trace-span durations: attribute end-to-end = sum(span durations) + queueing delta. Normalize by measuring client and server-side network spans: transport_time ≈ (network.send + network.recv) / 2 if symmetric, otherwise use client-observed TTFB and server-side send-time. Subtract ser/de spans from total to get transport + application overhead; validate with pcap RTT and CPU profiles to account for kernel/queueing.

**Experiments to justify changes**1. Controlled A/B rollout: enable protobuf vs JSON on 10% traffic, measure latency histograms, CPU usage, error rates, compression ratio.2. Compression toggle matrix: none, gzip, snappy on same payloads; measure CPU cost (profiles), effective bandwidth, end-to-end latency at varying payload sizes.3. Load tests: synthetic traffic with realistic payload distributions to measure cost per million requests and tail latency (p95/p99).4. Network-constrained tests: throttle bandwidth/introduce latency to quantify benefit of smaller payloads vs CPU cost.5. Cost model: combine cloud egress costs + added CPU instance hours to compute ROI for format/compression change.

**Success criteria**- Measurable p99 latency reduction or cost savings with ≤ acceptable CPU increase; validated by tracing, profiles, and controlled experiments.

MediumTechnical

46 practiced

Write (or describe in pseudocode) a Python script that parses an HTTP access log where each entry includes timestamp, request path, status, and latency-ms. The script should compute request rate, p50, p95, p99 latencies, and error rate for a given time window. Explain how you would modify your approach to work on streaming logs and very large datasets without storing all latencies in memory.

Sample Answer

**Approach (brief)**

- Read file line-by-line, parse timestamp/path/status/latency.- Filter entries inside the target time window.- Maintain counters: total requests, error requests, and a list of latencies (for medium datasets).- Compute rate = total / window_seconds, error_rate = errors / total, and percentiles via numpy.percentile.- For streaming/large data: use a streaming quantile structure (t-digest or HDRHistogram) and keep only counters in memory.

**Example Python (batch mode, medium data)**

python

import datetime, numpy as np

def parse_line(line):
    # assume CSV: timestamp, path, status, latency_ms
    ts_s, path, status_s, lat_s = line.strip().split(',')
    ts = datetime.datetime.fromisoformat(ts_s)
    return ts, path, int(status_s), float(lat_s)

def analyze_log(path, start, end):
    latencies = []
    total = errors = 0
    with open(path) as f:
        for line in f:
            ts, _, status, lat = parse_line(line)
            if not (start <= ts <= end): continue
            total += 1
            if status >= 500: errors += 1
            latencies.append(lat)
    p50,p95,p99 = np.percentile(latencies, [50,95,99]) if latencies else (None,None,None)
    rate = total / (end - start).total_seconds()
    return {'rate': rate, 'p50': p50, 'p95': p95, 'p99': p99, 'error_rate': errors/total if total else None}

**Streaming / large-scale modifications**- Replace latencies list with a t-digest (python-tdigest) or HDRHistogram to compute percentiles online with bounded memory.- Aggregate counts per time-bucket (e.g., 1s or 1m) for sliding windows; use tumbling or sliding window implemented with a deque of bucket summaries.- For distributed logs, run map-reduce: each worker builds a t-digest and counts; merge t-digests and sums of counters centrally.

**Why**- t-digest/HDR provide accurate tail quantiles with small memory.- Bucketing enables windowed rates without retaining every event.- Merging supports parallel/cloud processing (Kinesis/Firehose -> Lambda workers -> aggregator).

**Complexity**- Batch: O(n) time, O(n) memory for latencies.- Streaming with t-digest: O(n) time, O(1) memory (configurable).

Unlock Full Question Bank

Get access to hundreds of Performance Engineering and Cost Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.