Performance Fundamentals and Troubleshooting Questions

Core skills for identifying, diagnosing, and resolving general performance problems across applications and systems. Topics include establishing baselines and metrics, using monitoring and profiling tools to determine whether issues are CPU bound, memory bound, input output bound, or network bound, and applying systematic troubleshooting workflows. Candidates should be able to prioritize fixes, recommend temporary mitigations and long term solutions, and explain when to escalate to specialists. This canonical topic covers general performance awareness, common diagnostic tools, and basic remediation approaches for slow systems and resource exhaustion.

HardTechnical

62 practiced

Create a post-incident action plan for a performance outage that exhausted the service's error budget. The plan should include prioritized engineering tasks, SLO/Sli adjustments, observability improvements, and team/process changes to reduce recurrence.

Sample Answer

Situation: A production performance outage exhausted the service’s monthly error budget (SLO violation), causing customer-facing latency and increased error rates.

Goal: Restore confidence and reduce recurrence by addressing root causes across code, infra, observability, and process so we don’t repeat the outage and we rebuild error budget safety margin.

1) Prioritized engineering tasks (owner / timeline / success metric)- High (0–2 weeks) - Emergency throttling/fail-fast & circuit breaker rollout (Owner: SRE lead): add rate-limits and graceful degradation for non-critical flows. Success: 50% drop in peak tail latency within 1 deploy. - Hotfix for root-cause bottleneck (Owner: service team): fix contention/DB query/GC issue discovered in postmortem. Success: reproduce load test shows <SLO latency at target QPS.- Medium (2–6 weeks) - Capacity & autoscaling tuning (Owner: Platform): right-size instances, add CPU/memory headroom, improve horizontal scaling. Success: sustained 2x planned spare capacity. - Back-pressure propagation across services (Owner: SRE/Dev): implement token bucket or reactive signals. Success: upstream flows limited before overload.- Low (6–12 weeks) - Architecture refactor (Owner: Eng): shard heavy writes, add caching layer. Success: long-term throughput improvements and cost-neutral performance.

2) SLI/SLO adjustments- Short-term: tighten SLIs for p95/p99 latency and increase monitoring cadence. Temporarily lower SLO burn threshold alert to 5% remaining error budget.- Medium-term: re-evaluate SLO targets against customer impact — if SLO unrealistic, adjust with stakeholders and increase error budget or add explicit degradation tiers (gold/silver).

3) Observability improvements- Immediate: add detailed p99 latency, queue depth, GC pause, thread pool saturation, DB slow-query histogram. (Owner: Observability)- Implement distributed tracing for main request path; add error budget dashboard and automated burn-rate alerts integrated with PagerDuty and Slack.- Add synthetic transactions and chaos tests to exercise degraded paths. Success: 100% trace coverage for top 10 request types.

4) Team/process changes- Incident playbook updates: include runbook for throttling, rollback, capacity expansion (Owner: Incident Commander).- Introduce weekly SLO review meeting; owner rotates between SRE and product. Actions: track burn-rate and upcoming releases against budget.- Blameless postmortems within 48 hours with assigned PIs and deadlines. Track action items in Jira with SLO of closure within 30 days.- Release gating: block risky releases when error budget < 20% unless mitigation in PR.

Verification and metrics- Short-term: 30-day window of no SLO breaches; error budget burn rate < 0.5x under expected load.- Medium-term: load tests at 2x production show <SLO latencies; trace coverage and dashboards in place.- Report weekly to stakeholders until 3 consecutive months of compliance.

Trade-offs- Quick mitigation (throttling) prioritizes availability at cost of degraded features; long-term work focuses on capacity and architectural fixes.

This plan pairs tactical fixes to stop the immediate bleeding with strategic investments in observability and process to prevent recurrence and ensure accountable tracking until SLO health is restored.

HardSystem Design

62 practiced

Design an end-to-end tracing and logging strategy to speed up performance debugging for a microservices environment of ~50 services. Address sampling decisions, how to correlate logs and traces, and how to avoid overwhelming storage/ingest costs.

Sample Answer

Requirements / constraints:- 50 microservices, sub-100ms latency targets for most RPCs, SLOs defined per service, budget cap for ingest/storage.- Goals: fast root-cause discovery, trace-to-log correlation, low-cost long-term retention for key signals.

High-level design:- Use distributed tracing (OpenTelemetry) + structured logging (JSON) + a tracing backend (e.g., Jaeger/Tempo or SaaS like Lightstep) and log store (e.g., Elasticsearch/ClickHouse/Cloud logging) with a correlator (trace IDs propagated into logs).- Central pipeline: services emit traces & logs → collector (OTel Collector) performs sampling/enrichment → stream to trace store + log store + metrics system.

Core components & responsibilities:1. Instrumentation: standardized SDK + middleware to auto-instrument HTTP/gRPC, and inject trace_id/span_id into logs and metrics tags.2. Collector: receives traces/logs, applies sampling, enrichment (user id, svc version, env), and routing.3. Trace store: stores full sampled traces for 7–30 days; supports tail-based sampling ingestion.4. Log store: hot short-term store (7 days full) + cold long-term (30–365 days aggregated/indexed) with tiered retention.5. UI/Correlator: link traces ↔ logs by trace_id/span_id; provide trace search by error, latency, user, or request id.

Sampling strategy:- Hybrid sampling: - Always-sample: errors, traces crossing SLO thresholds (latency > X ms), specific user-impacting customers, or flagged feature-ids. - Adaptive tail-based sampling: initially sample a percentage (e.g., 1–5%) of traces; when collector sees rare/error/high-latency traces, keep correlated prior spans (tail-based keep) to preserve causality. - Head-based probabilistic sampling for high-volume internal calls (0.1–1%). - Dynamic adjustment: auto-scale sampling rates per service based on traffic, error rate, and recent incidents.

Correlation between logs and traces:- Propagate trace_id and span_id in headers and include them in structured logs (trace_id, span_id, request_id). Enrich logs at ingress to include service, pod, git sha, and user id.- Store logs with indexed trace_id field for quick join; in UI allow "open trace from log" and "show logs for trace" features.

Cost-control & storage optimization:- Tiered retention: full traces/logs 7–30 days; aggregated metrics and span-level indexes longer.- Store high-cardinality attributes only in trace index, not full events.- Use sampling to reduce ingest volume; compress and batch events at collector; drop debug-level verbose logs in high-throughput paths unless toggled.- Pre-aggregation: convert frequent low-value logs into metrics or summaries (counts/histograms) before long-term retention.- Use cold storage (object store) for raw traces older than hot window; keep only indices and a pointer to cold objects for on-demand retrieval.- Cost-aware routing: route traces/logs from high-cost services through stricter sampling.

Operational practices:- SLO-driven alerts: trigger deeper tracing capture (sampling increase or debug flag flip) automatically during incidents.- Runbooks: include trace/log query patterns for common failure modes.- Canary and feature flags to enable verbose tracing for small subsets of traffic or tenants.- Regular audits of high-cardinality fields and index usage to reduce cost.

Trade-offs:- Tail-based sampling preserves debugability but increases collector complexity.- Tiered retention lowers cost but adds retrieval latency for old traces.- Always-sample for errors increases storage but yields highest ROI for incident resolution.

Outcome:- Fast correlation from symptom (metric/alert) → trace → logs → root cause, with bounded cost through hybrid sampling, tiered retention, pre-aggregation, and SLO-driven adaptive capture.

HardTechnical

72 practiced

You are given a CPU flamegraph showing a function `serialize_response()` consuming 45% of samples and a hot path inside serializing a large optional field. Describe concrete code-level optimizations you would propose and how you'd validate performance improvement without causing regressions.

Sample Answer

Situation: Flamegraph shows serialize_response() at 45% of CPU; hot inner path is serializing a large optional field (often absent or large when present). Goal: reduce CPU/time spent without regressions.

Concrete code-level optimizations:- Short-circuit when optional absent: check optional early and avoid entering heavy serializer.

cpp

if (!resp.large_field.has_value()) return serialize_without_large_field(out, resp);

- Lazy / on-demand serialization: serialize the large field only when the client requested it (via query param/Accept header).- Streaming / chunked serialization to avoid buffering large in-memory copies; write to output stream/IO buffer directly.

python

def serialize_large_field(stream, field):
      for chunk in field.iter_chunks():
          stream.write(encode_chunk(chunk))

- Zero-copy / borrow data where language allows (avoid constructing intermediate strings/byte arrays). Use memory views or serializers that operate on buffers.- Use a faster serializer or proto option (e.g., use binary protobuf vs JSON, or use simd-json / yajl).- Parallelize or offload heavy work: move large-optional serialization to background task or a separate worker if request latency allows.- Micro-optimizations: reuse buffers, pre-allocate capacity, avoid expensive formatting in hot loop.

Validation and avoiding regressions:- Add unit tests ensuring serialization correctness for present/absent large field and edge cases.- Create microbenchmarks (bench harness) for serialize_response() with representative inputs (absent, small, very large). Measure p50/p95/p99 latency and CPU.- Use flamegraphs and perf runs before/after to confirm CPU sample reduction and that hot path moved/removed.- Canary deployment: roll out change to small % of traffic, compare metrics (latency, CPU, error rate, tail latency). Monitor SLOs, GC/heap, and memory for regressions.- Automate regression tests in CI: benchmarks and fuzz tests; set performance gates (e.g., p95 must not increase).- If changing wire format, maintain compatibility and provide feature-flagged switch to revert.

Why: Early checks and lazy/streamed serialization reduce wasted work; zero-copy and buffer reuse reduce allocations/GC; benchmarking + canary + CI tests ensure performance gains and safety.

EasyTechnical

68 practiced

Explain why p50 alone is insufficient for performance monitoring of user-facing services. Provide an example showing how p50 and p99 could tell different stories and one practical consequence of relying only on p50.

HardTechnical

105 practiced

A recent deployment caused performance regressions only in Region A. Describe an investigation plan to determine whether the regression is due to the deployment, regional infra differences, or traffic profile differences. Include what metrics/logs to compare and how to test hypotheses with minimal user impact.

Sample Answer

Situation overview / goal: determine whether Region A's recent performance regression is caused by the code deployment, regional infra differences, or differing traffic profile — and do so with safe, minimal customer impact.

Investigation plan (steps):

1. Confirm scope & timing- Verify deployment metadata (build id, config, feature flags) rolled to Region A and other regions. Check deploy timestamps and rollout windows.- Correlate onset time of regression with deployment time.

2. Quick comparisons (baseline vs now; Region A vs Region B/C)Metrics to compare (per-region, per-service, per-instance):- User-visible: p50/p95/p99 latency, request rate (RPS), error rate (4xx/5xx), connection failures, timeouts.- Infra: CPU, memory, GC pause, threadpool/queue depth, disk IO, network throughput, socket/file descriptor usage.- Platform: container restart counts, OOMs, kube node metrics, autoscaler events.Logs/traces:- Aggregated logs for exceptions, stack traces; distributed traces (Jaeger/Zipkin) for span duration and hotspots.- Slow queries / DB wait times; cache hit/miss rates.

3. Hypothesis formation & targeted testsA. Deployment-caused:- Test: Gradual rollback or canary test: divert a small % (1-5%) of Region A traffic to the previous build (or route a small % of Region B traffic to new build). If latency/error improves only when using prior build, implicates code/config.- Additional: Run same build in Region B with identical infra (use same instance types/AMI) to see if issue reproduces.

B. Infra-caused:- Test: Compare VM/Node images, kernel, instance types, and autoscaler behavior. Spin up a new node pool in Region A with same template as Region B and move traffic there; measure.- Check cloud provider status & cross-region network health. Run synthetic latency/network tests (iperf, traceroute) to DB/cache.

C. Traffic-profile-caused:- Test: Replay Region A traffic (sampling) against Region B or staging environment; or replay Region B traffic in Region A. If heavy payloads or different client patterns trigger regression, you'll reproduce it.- Compare client-side metrics: request size, concurrency per client, TLS handshake counts.

4. Low-impact execution strategy- Use canary releases, weighted routing, and feature flags to limit blast radius.- Run synthetic load tests on staging and on newly provisioned small-scale nodes in Region A before broad changes.- Prefer read-only or non-critical traffic for heavier probes; use shadowing (duplicate traffic to test system without affecting user experience).

5. Root-cause confirmation and remediation- Use traces/flamegraphs/heap dumps to pinpoint code hotspots if deployment implicated; fix and redeploy canary.- If infra: change node types, kernel, or cloud config and monitor.- If traffic profile: implement autoscaling tuning, request shaping, caching, or optimize expensive code paths.

Key tooling to use: Prometheus/Grafana, ELK/CloudLogging, Jaeger/Zipkin, cloud metrics, APM profiler, packet captures. Always document findings, timeline, and post-incident action items (alerts, tests, regression suite).

Unlock Full Question Bank

Get access to hundreds of Performance Fundamentals and Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.