Performance Debugging and Latency Investigation Questions

Finding the root cause of latency spikes: checking CPU/memory/disk/network utilization, profiling applications, querying slow logs, and identifying bottlenecks. Understanding the difference between resource exhaustion and an algorithmic problem. Using monitoring and tracing tools to narrow down where time is spent.

EasyTechnical

55 practiced

For diagnosing CPU, memory, disk I/O and network issues, explain how you would use top/htop, iostat, vmstat, sar and ss. For each tool specify the columns or fields you check (e.g., %idle, iowait, context switches) and list two threshold patterns or signs that would indicate a problem for that resource.

MediumTechnical

98 practiced

You are given a single distributed trace for a user request that shows the total end-to-end latency is 1.2s. Spans (parent->child) with durations:- A (frontend): total 1.2s, calls B- A->B network: 0.1s- B (service): total 0.8s, internal CPU work 0.5s, B->C network 0.05s, waiting for DB 0.25s- C (downstream): total 0.1sBased on these spans, where is the largest portion of time spent and what initial remediation steps would you take to reduce p99 latency for this flow?

EasyTechnical

54 practiced

Compare the use-cases for strace and perf top when diagnosing latency issues on Linux. For each tool give one class of problem it helps reveal (e.g., blocking syscalls vs CPU hotspots) and describe a short workflow for using it safely on a production node.

Sample Answer

strace — class of problem: blocking or slow syscalls (I/O waits, futex waits, DNS, file locks)Workflow (safe on prod):- Goal: confirm a process is blocked in syscalls and identify which ones and call sites.- Steps: 1. Narrow target: get PID(s) from monitoring/alerting (top/ps, pstack). 2. Attach selectively: strace -p <pid> -s 200 -f -ttt -e trace=network,desc,process — this limits traced syscalls and shows timestamps; use -c for a summary if you just need counts. 3. Run briefly (10–30s) to avoid heavy overhead; use ionice/nice for strace itself if needed. 4. Analyze: look for long timestamp gaps between enter/exit, repeated EINTR/EAGAIN, long read/write or futex sleeps, blocking open/stat, or DNS (getaddrinfo) delays. 5. If necessary, capture to file and share with devs; always detach quickly and avoid tracing large PID trees on busy nodes.

perf top — class of problem: CPU hotspots and sampling-based performance (hot functions, kernel vs user time, lock contention)Workflow (safe on prod):- Goal: find where CPU cycles are spent with low overhead.- Steps: 1. Prepare: ensure perf permissions (root or perf_event_paranoid tuned) and limit scope: perf top -p <pid> --freq 99 --call-graph dwarf -o perf.out 2. Use moderate sampling rate (50–200 Hz) and short duration (30–60s). Prefer --pid or --cgroup to isolate service; avoid system-wide unless necessary. 3. Run with nice/ionice and monitor node load; collect folded stacks for flamegraph later if deeper analysis is needed. 4. Interpret: hot symbols point to CPU-bound code paths (user funcs, syscalls in kernel). Correlate with strace/metrics to decide if CPU or blocking is root cause. 5. Clean up and share perf report; for production, prefer non-invasive sampling and limited windows.

Summary: Use strace for pinpointing blocking syscalls and short traced sessions; use perf top for low-overhead CPU sampling to find hotspots. Combine both (small perf sample + short strace) and correlate with metrics (latency, iowait, runqueue) before making changes.

EasyTechnical

67 practiced

Explain the difference between metrics, logs and traces for performance debugging. For a service experiencing intermittent latency spikes, give one concrete example of the kind of information you expect to find in each signal and describe how you would correlate them to find a root cause.

MediumTechnical

72 practiced

Explain hedged (backup) requests as a technique for reducing tail latency. Describe how to implement hedging in a client library (timers, cancellation, idempotency concerns), and discuss three trade-offs of using hedged requests in a high-scale service.

Sample Answer

Hedged (backup) requests reduce tail latency by issuing one or more additional duplicate requests for the same operation if the original request hasn't completed within a short threshold. The first successful response is used; the rest are cancelled. This cuts long-tail outliers due to transient queuing, slow hosts, or GC pauses.

Implementing hedging in a client library:- Timer/threshold: measure baseline p95/p99 and pick a hedge delay slightly shorter than the target tail bound (e.g., if target is 200ms and p95=120ms, hedge at 140–160ms). Optionally use adaptive timers that track recent latency distributions per endpoint.- Concurrency: allow at most N outstanding hedges per request to limit amplification (commonly N=1 or 2).- Cancellation: when one response returns, immediately cancel in-flight backups (send TCP FIN/abort or use request-level cancelation tokens). Ensure cancellation is best-effort—servers may still work on the request.- Idempotency & routing: only hedge idempotent operations (GET, idempotent POSTs) or require servers to handle duplicate executions safely. Use unique request IDs and de-duplication at the server or implement at-most-once semantics using request tokens if side-effects exist.- Observability: emit metrics for hedge rate, success-by-original-or-hedge, and amplification ratio to tune settings.

Three trade-offs:1. Resource amplification: hedging increases request volume and CPU/network load, which can worsen overload during high traffic. Mitigate with rate limits, adaptive hedging, and backpressure.2. Complexity & correctness: handling cancellations, de-duplication, and idempotency adds complexity across client/server. Improper handling can cause double side-effects or hard-to-debug inconsistencies.3. Cost vs benefit: hedging helps when tail latency stems from occasional slow nodes, but when tail is systemic (network partitions, overloaded cluster) hedging wastes resources without improving latency. Require monitoring and dynamic disabling when ineffective.

In practice, start with conservative hedging (single backup, idempotent ops), measure impact on tail latency and system load, then iterate with adaptive controls and server-side de-duplication if needed.

Unlock Full Question Bank

Get access to hundreds of Performance Debugging and Latency Investigation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.