Performance Profiling & Bottleneck Analysis Questions

Before-and-after measurement plan

Re-run the SAME benchmark/harness used to establish the original baseline, under matched conditions: same load shape, same environment, and the same time-of-day window if traffic has a daily pattern. The change under test should be the only thing that's different between the two runs. Take multiple repetitions on each side, not one run each.

Statistical significance

Single numbers are noisy, so compare distributions instead of points. A practical, interview-safe approach: run N repetitions of before and after, compute the mean and spread (e.g. standard deviation) for each, and check whether the difference is bigger than the natural run-to-run variance, either with a simple significance test (a t-test, a standard statistical check for whether two averages differ by more than you'd expect from random chance) or, since latency data is usually skewed (bunched to one side with a long tail of slow outliers) rather than normally distributed (evenly spread in the familiar bell-curve shape), by comparing medians/percentiles directly or bootstrapping a confidence interval on the difference (repeatedly resampling the data to estimate a likely range for the true difference).

Concretely: if before averages 200ms with a spread of about 5ms (a typical range of roughly 195-205ms across repetitions) and after averages 180ms with a spread of about 5ms (roughly 175-185ms), the two ranges sit fully apart, a clear 10ms gap between the top of the after range and the bottom of the before range, so that's a real improvement. If the two ranges overlap heavily instead, for example if the after spread widened to +/-15ms (165-195ms, swallowing most of that gap), you cannot yet claim a win, you need more runs or a bigger effect.

Regression analysis for non-target metrics

A fix aimed at one metric (say, CPU time) can silently regress another (memory, error rate, or a different endpoint sharing the changed code path). Re-run the FULL metric dashboard before and after, not just the metric you optimized, paying particular attention to anything else that shares the changed component.

Communicating to stakeholders

Report: what metric moved and by how much, WITH its uncertainty or range rather than a single point number; the conditions it was measured under (e.g. "under X requests/sec of synthetic load" versus "confirmed against live production traffic"), so the audience knows the claim's scope; what else was checked and found unaffected; and what will be watched going forward (e.g. an alert threshold tied to the improved metric). Avoid presenting one lucky run as proof, since that's exactly the trap this whole validation process exists to avoid.

Framing

A feature-serving pipeline with occasional p95 (95th percentile latency) spikes to 2 seconds needs the spike CHARACTERIZED before it can be triaged: is it periodic (tied to a batch job, cache eviction, or GC pause) or effectively random? Pulling a trace from a few actual spike instances, if distributed tracing exists, is the fastest way to see where time in ONE slow request actually went.

1. Break the pipeline into stages and instrument each

Split the flow into stages: fetch raw data, compute/transform features, I/O to a feature store or model, serialize and respond. Time each stage (via existing tracing spans, or added timing) so you can see which stage owns the tail, since an average across the whole pipeline will hide a spike that lives entirely in one stage.

2. Apply the right diagnostic lens per stage

• If the slow stage is CPU-bound feature computation, take a CPU profile during an actual spike and read the flame graph.
• If it's I/O (feature-store lookups, calls to another service), check connection pool exhaustion, cache miss rate, and the downstream service's own latency.
• If it correlates with memory pressure, check whether GC (garbage collection, the runtime automatically reclaiming unused memory, which can briefly pause the program) pauses line up with the spike timing.

3. Prioritize using tail behavior, not averages

A stage that's only 5% of TYPICAL latency isn't where an occasional 2s spike is hiding even if it's individually inefficient; focus on whichever stage's tail (p99, not its average) is worst, since the reported symptom is a tail event, not a general slowdown.

4. Fix by category

• Infrastructure: add caching or connection pooling, autoscale ahead of load, isolate noisy-neighbor workloads onto separate capacity.
• Algorithmic: eliminate an accidental quadratic feature join, batch many small calls into one.
• I/O: cut round-trips to the feature store, add a local cache with a TTL (Time To Live, an expiration window) so repeat lookups don't hit the network every time.

5. Validate under a production-like load test

Reproduce the fix under the SAME request mix and concurrency the production traffic actually has, not a single-request smoke test, and compare the FULL latency distribution before and after, specifically p95/p99, not the average, since a fix that only improves the average while leaving the tail untouched hasn't solved the reported 2-second-spike problem. Run it long enough to catch periodic effects (cache warm-up cycles, GC cycles) that a short test would miss entirely.

The two approaches

• Sampling profilers periodically interrupt execution (e.g. every 1 to 10ms) and record the current call stack. After enough samples, functions that show up often are statistically the hot ones.
• Instrumentation profilers insert explicit measurement code (timers, counters) at function entry/exit or around chosen blocks, recording exact call counts and durations for every invocation.

Overhead

• Sampling: low and roughly constant, since cost is proportional to the sampling rate, not to how many times the profiled code runs. This is what makes it safe to leave running in production.
• Instrumentation: proportional to how often the instrumented code is called. A function called a million times a second, wrapped in instrumentation, can itself become a meaningful fraction of total runtime, this is called the probe effect (the act of measuring changes the thing being measured), and it can even distort WHICH function looks hottest.

Accuracy

• Sampling gives a statistical approximation. Long-running, frequently-hit functions are represented accurately; very short-lived functions that happen to fall between samples can be under-counted or missed entirely.
• Instrumentation is exact for whatever it wraps: precise call counts and durations, including rare or short-lived calls that sampling could miss. The trade-off is it only sees what you deliberately instrumented, unlike sampling's whole-process view.

When to choose which, as an SRE

• Default to sampling for open-ended, production, "where is the time going in this whole service" investigations, because its overhead is safe to run live and it needs no code changes.
• Reach for instrumentation once sampling has already narrowed the problem to a specific function or code path and you need EXACT counts or durations (e.g. confirming a cache is actually being hit 95% of the time, or timing a specific external call), and preferably run it in staging, or scope it tightly and temporarily if it must run in production.
• A common combined pattern: use sampling first to find the candidate hotspot cheaply and broadly, then apply narrow instrumentation only around that hotspot to confirm and quantify it precisely.

What a heap dump is

A heap dump is a snapshot of every live object on a process's managed heap (for a garbage-collected runtime such as the JVM, Java Virtual Machine) at one instant, including each object's fields, size, and references to other objects. It's effectively a full memory census you can load into an analyzer afterward.

When to capture one in production

• When heap usage is trending toward an OOM (Out Of Memory, when the process is killed or fails because it exceeded available memory) and you need to identify exactly which objects are being retained.
• When you have a specific, already-suspected leak and need to confirm which class is actually growing.
• At the moment of an OOM itself, many runtimes support a "dump heap automatically on OOM" flag, so you capture the exact state that caused the failure instead of trying to reconstruct it afterward.

Live heap dump vs allocation sampling

• A live heap dump captures everything CURRENTLY retained, exactly, you see every live object and its reference chain. The cost is real: taking one is expensive, it commonly triggers a full stop-the-world pause (the runtime freezes application threads to take a consistent snapshot) and produces a large file, so it's not something to do routinely, only for a deliberate, specific investigation.
• Allocation sampling instead records a statistical sample of allocation call sites over time (WHERE objects are being created), with much lower overhead, low enough to run continuously or on demand without freezing the process. It tells you where churn is happening, which is great for finding an allocation hotspot driving GC (garbage collection) pressure, but it does not give you the full retained-object graph a heap dump gives you.

Minimizing production impact

• Default to allocation sampling for ongoing, low-overhead visibility.
• Reserve full heap dumps for one deliberate capture, ideally on a single canary instance pulled out of the load balancer's rotation first, not across the whole fleet simultaneously.
• Use dump-on-OOM flags so the one dump you actually need gets captured automatically, without needing to guess the right moment.

Define what "net gain" means before collecting data

"Faster" is not the same as "net win." Fix the comparison basis up front: net gain means better cost-normalized throughput or lower latency at the same or lower cost per 1,000 inferences, measured end-to-end from request in to response out, not just kernel execution time on the accelerator. An experiment that only measures GPU kernel time and ignores everything around it will overstate the win.

Metrics to collect, on both the CPU baseline and the GPU treatment

• Latency distribution: p50, p95, and especially p99, not just the mean; GPU wins on average latency are common even when p99 gets worse under bursty load because of batching and queueing delay.
• Throughput: successful requests per second at a fixed, acceptable error rate, not raw requests attempted.
• GPU utilization: percentage of time the device is actually computing versus idle waiting for data, via nvidia-smi for a single box or a fleet-level tool like NVIDIA DCGM.
• CPU pre/post time: tokenization, feature construction, and postprocessing usually still run on CPU even in a "GPU" request; profile this separately (a CPU profiler such as py-spy for a Python inference server) so it does not get invisibly absorbed into "GPU time."
• Host-to-device memory copy overhead: the time to move the batch onto the GPU and the result back; for small batches or small models this copy can dominate the actual compute time.
• Cold-start latency: time to load the model onto the device and warm up kernels after a scale-up event, measured separately from steady-state latency.

Tools that map to each metric

Use a profiler with device-level visibility (NVIDIA Nsight Systems, or the framework's own profiler such as the PyTorch Profiler or TensorFlow Profiler) to get a timeline that separates host compute, the memory copy, and device kernel execution as distinct spans, instead of one lumped "inference time" number. Pair that with a load-testing setup that drives realistic concurrency and batch sizes (a scripted client, or a load tool such as k6 or vegeta adapted to the model's API) to produce the latency distribution and throughput numbers under conditions that resemble production traffic, not a single warmed-up request in a notebook.

Experiment design

Hold the traffic pattern (arrival rate, batch size distribution, payload size) identical between the CPU baseline and the GPU treatment; only the execution path changes. Run both warm (already loaded, steady traffic) and cold (right after a scale event), since GPU offloading's cold-start cost is usually worse than CPU's. Sweep batch size explicitly: GPU throughput advantages are typically batch-size dependent and can shrink or invert at batch size 1, which matters a great deal if the endpoint serves low-latency single-request traffic rather than batch scoring jobs.

Reasoning about cost, cold starts, and maintainability

Compute a cost-per-1,000-inferences figure for each path: instance hourly cost divided by sustained throughput, scaled to 1,000 requests, and make sure the GPU number includes the CPU pre/post and copy time you measured, not just kernel time, since those costs don't disappear just because they're not on the accelerator. If traffic is spiky and low-volume, GPU cold-start and idle-instance cost can make the CPU path cheaper overall even though the GPU is faster per request once warm. On maintainability, weigh the added operational surface honestly: CUDA and driver version pinning, harder local development and debugging, and a second deployment target to keep patched, against the measured, cost-normalized win. Recommend the switch only if the gain holds after including all of the above, not just the headline kernel speedup.