Performance Profiling & Bottleneck Analysis Questions

Define what "net gain" means before collecting data

"Faster" is not the same as "net win." Fix the comparison basis up front: net gain means better cost-normalized throughput or lower latency at the same or lower cost per 1,000 inferences, measured end-to-end from request in to response out, not just kernel execution time on the accelerator. An experiment that only measures GPU kernel time and ignores everything around it will overstate the win.

Metrics to collect, on both the CPU baseline and the GPU treatment

• Latency distribution: p50, p95, and especially p99, not just the mean; GPU wins on average latency are common even when p99 gets worse under bursty load because of batching and queueing delay.
• Throughput: successful requests per second at a fixed, acceptable error rate, not raw requests attempted.
• GPU utilization: percentage of time the device is actually computing versus idle waiting for data, via nvidia-smi for a single box or a fleet-level tool like NVIDIA DCGM.
• CPU pre/post time: tokenization, feature construction, and postprocessing usually still run on CPU even in a "GPU" request; profile this separately (a CPU profiler such as py-spy for a Python inference server) so it does not get invisibly absorbed into "GPU time."
• Host-to-device memory copy overhead: the time to move the batch onto the GPU and the result back; for small batches or small models this copy can dominate the actual compute time.
• Cold-start latency: time to load the model onto the device and warm up kernels after a scale-up event, measured separately from steady-state latency.

Tools that map to each metric

Use a profiler with device-level visibility (NVIDIA Nsight Systems, or the framework's own profiler such as the PyTorch Profiler or TensorFlow Profiler) to get a timeline that separates host compute, the memory copy, and device kernel execution as distinct spans, instead of one lumped "inference time" number. Pair that with a load-testing setup that drives realistic concurrency and batch sizes (a scripted client, or a load tool such as k6 or vegeta adapted to the model's API) to produce the latency distribution and throughput numbers under conditions that resemble production traffic, not a single warmed-up request in a notebook.

Experiment design

Hold the traffic pattern (arrival rate, batch size distribution, payload size) identical between the CPU baseline and the GPU treatment; only the execution path changes. Run both warm (already loaded, steady traffic) and cold (right after a scale event), since GPU offloading's cold-start cost is usually worse than CPU's. Sweep batch size explicitly: GPU throughput advantages are typically batch-size dependent and can shrink or invert at batch size 1, which matters a great deal if the endpoint serves low-latency single-request traffic rather than batch scoring jobs.

Reasoning about cost, cold starts, and maintainability

Compute a cost-per-1,000-inferences figure for each path: instance hourly cost divided by sustained throughput, scaled to 1,000 requests, and make sure the GPU number includes the CPU pre/post and copy time you measured, not just kernel time, since those costs don't disappear just because they're not on the accelerator. If traffic is spiky and low-volume, GPU cold-start and idle-instance cost can make the CPU path cheaper overall even though the GPU is faster per request once warm. On maintainability, weigh the added operational surface honestly: CUDA and driver version pinning, harder local development and debugging, and a second deployment target to keep patched, against the measured, cost-normalized win. Recommend the switch only if the gain holds after including all of the above, not just the headline kernel speedup.

CPU time vs GPU time

CPU time is time spent executing on the host processor: request handling, pre- and post-processing such as tokenizing text or resizing images, batching requests, and orchestration. GPU time is elapsed time the accelerator spends actually executing the inference kernels, the matrix and tensor operations that make up a neural network's forward pass. A request's total latency is the sum of both, plus the time spent transferring data between host memory and GPU memory (PCIe transfer, often overlooked but real), and the GPU can sit idle while the CPU works and vice versa.

When an SRE should reach for GPU profiling

Reach for GPU-specific profiling when a CPU profile of the service shows most time simply "waiting" inside the inference call, a synchronous wait for the GPU to finish that looks like idle time to a CPU profiler rather than a hotspot, or when nvidia-smi shows the GPU near 100% utilized yet throughput still is not meeting target, which points at needing to speed up or better batch the GPU-side work itself rather than anything on the host. Conversely, if GPU utilization is low while requests still queue up, the bottleneck is upstream, either CPU-side preprocessing or the input pipeline starving the GPU, and no amount of GPU profiling will fix that.

Tools and metrics

• nvidia-smi, or DCGM for continuous fleet-wide monitoring, for GPU utilization percentage, memory usage, and power or thermal state, at a coarse level.
• A framework-native profiler, such as the PyTorch Profiler or TensorFlow Profiler, to break down time per individual operation on the GPU, showing which layer or op dominates.
• NVIDIA Nsight Systems for a timeline view that shows CPU and GPU activity side by side, which is what actually reveals whether the two are overlapping efficiently (CPU preparing the next batch while the GPU computes the current one) or serializing (CPU idle while GPU runs, then GPU idle while CPU prepares).

Distinguishing the two bottleneck types

High GPU utilization with a growing request queue means the GPU is the bottleneck; consider batching more requests per inference call or a faster or larger accelerator. Low GPU utilization with high CPU utilization or a data-loading thread pegged near 100% means the bottleneck is upstream of the GPU entirely, and profiling the GPU further would be looking in the wrong place.