Performance Troubleshooting & Incident Response Questions

CPU steal time is the percentage of time a virtual CPU wanted to run but the physical host's hypervisor gave that CPU cycle to a different tenant's virtual machine instead, meaning your workload was ready to go but simply wasn't scheduled. It matters for latency-sensitive services because steal time produces unpredictable latency spikes that don't correlate with anything in your own application's metrics: from inside the VM, your own CPU utilization can look completely healthy while you're still being starved, since idle time and steal time are reported as separate things.

Why it especially hurts latency: a request that would normally take 5ms of actual compute can take far longer in wall-clock time if the VM's vCPU (virtual CPU) is repeatedly paused mid-execution while a neighboring tenant runs on the same physical core. The raw accounting does expose this: the Linux kernel tracks steal as its own field in /proc/stat, separate from idle, and top's CPU line, vmstat and iostat all print it as a distinct st or %steal column sitting right next to id or %idle. What makes it easy to overlook is one layer above the raw accounting: the number most dashboards, alerts and autoscalers graph as "CPU utilization" is user plus system time, and that stays low and reassuring precisely because the stolen cycles were never yours to spend. So the signal is present in the per-host tools and absent from the summary view almost everyone actually looks at, which is why exporting the steal field explicitly is part of the fix rather than an afterthought.

Three remediation options

1. Short-term: move the workload to different physical hardware. Stopping and restarting the instance (on most cloud providers this reschedules it to a different host) often relieves a noisy-neighbor situation immediately, or scale up to a larger instance size, since larger instance types are typically given more predictable, less-shared CPU allocation.
2. Short-term: reduce the application's sensitivity to jitter, for example by adding a small amount of timeout headroom, or spreading load across more, smaller instances so a single steal event on one instance doesn't dominate the aggregate tail latency.
3. Long-term: move latency-sensitive workloads to dedicated or reserved instance types with guaranteed, non-shared CPU allocation, or to bare-metal instances where the provider offers them, which structurally removes the possibility of steal since there's no other tenant sharing the core. Alternatively, choose a provider, region, or instance family with a track record of lower oversubscription for this workload class.

Worked example: a service's own CPU utilization graph shows a comfortable 40% average, yet p99 latency (99th percentile) intermittently spikes to 5 times baseline. Checking vmstat 1 with the steal (st) column visible shows steal time jumping to 15 to 20% during exactly those windows, meaning the hypervisor is preempting this VM's vCPUs for another tenant somewhere between a sixth and a fifth of the time during the spike. Note how the two numbers fit together: 40% utilization plus 15 to 20% steal says the VM is being denied a chunk of CPU comparable to half of what it is actually managing to use, which is more than enough to move a tail even though the utilization graph looks relaxed. Migrating to a larger instance size in the same family, which the provider allocates more predictably, drops steal time to under 1%, and the p99 spikes disappear over the following 24 hours of observation.

Trade-offs and pitfalls: a bigger instance or dedicated hardware costs more, so size the fix to the actual latency sensitivity of the workload rather than applying it fleet-wide. Steal is also intermittent and host-dependent, a fix validated on one host doesn't guarantee the next host you land on after any restart or migration behaves the same, so ongoing monitoring of the steal metric itself belongs in the long-term fix, not just a one-time check.

Direct answer: With one variable, region, isolating the regression, the fastest path is a controlled comparison that holds two of three factors constant while testing the third: is Region A different because of what was deployed there, where it physically runs, or who it serves. Each of those has a distinct, low-impact test.

Structured elaboration:

• Confirm the deployment actually differs by region. Check rollout logs: did Region A get a different version, a different rollout percentage, or the same version at a different time (a canary region (one that intentionally receives a new release before the rest, to catch problems on a small slice of traffic first) rolled out first might simply be further along a slow-burn issue that hasn't reached other regions yet)?
• Compare regional infrastructure. Instance types, node pool age or kernel version, network topology (does Region A cross an availability zone or region boundary to reach a dependency that's co-located everywhere else?), and dependency latency measured specifically from that region.
• Compare traffic profile. Request mix by endpoint, payload sizes, client mix (mobile versus desktop, bot traffic), and absolute traffic volume, since a smaller or larger region can exercise a code path differently.
• Test each hypothesis with minimal user impact:
  • To isolate the deployment: route a small percentage of Region A's traffic back to the old version, on the same region, same infrastructure, same traffic mix, controlling for everything except version.
  • To isolate infrastructure: if your architecture supports cross-region routing at the load balancer or CDN (content delivery network: the geographically distributed layer that caches and routes traffic close to users) layer, send a small slice of Region A traffic to process on Region B's infrastructure and see if the regression follows the region or the deployment.
  • To isolate traffic profile: shadow or replay a sample of Region A's real traffic against Region B's stack (or vice versa) to see if the same requests reproduce the same latency elsewhere, without exposing real users to a risky reroute.

Worked example: The deploy log shows every region received the identical version at the same time, ruling out a rollout-timing difference. Comparing regional infrastructure, Region A's app tier calls a metadata service that, only in Region A, is not co-located in-region (it routes cross-region, adding roughly 40ms of extra network round trip per call versus under 5ms elsewhere) due to a capacity gap from that region's original buildout. That 40ms tax was tolerable before the new deploy, but the deploy added three more calls to that same metadata service per request, multiplying an existing, previously survivable regional tax rather than introducing a new bug. The fix is either adding a regionally co-located instance of the metadata service, or reducing the new deploy's call count to it, validated by shadowing Region A's real traffic against a build with the reduced call count before rolling the fix forward.

Trade-offs and pitfalls: assuming identical code means identical behavior everywhere ignores that infrastructure and traffic differences can turn an otherwise harmless change into a region-specific regression. Testing hypotheses by actually shifting live user traffic between regions carries real risk, added latency, and in some cases data-residency constraints, so prefer shadow or replay traffic over live rerouting wherever compliance allows it.

Direct answer: The workload is write-heavy (four times more writes than reads, and eight times more write throughput than read throughput) and the disk metrics show queueing rather than raw saturation: await (the time each request waits, including time queued) is ten times svctm (the actual time the device spends servicing a request), even though %util is only 62.5 percent. Combined with 40 percent CPU I/O wait (wa), the most likely bottleneck is the storage device becoming a queueing point under this write load, not the CPU.

Structured elaboration:

• Read the ratio, not just %util. await at 25.0ms against svctm at 2.50ms gives a ratio of 25.0 / 2.50 = 10. That means, on average, roughly nine-tenths of the time a request spends "in flight" at the device, it's waiting in queue rather than actually being serviced, even though %util (62.5%) hasn't hit its ceiling.
• Put a number on the queue with Little's law, then reconcile it against %util, because those two figures are in tension and the tension is the finding. Completions are r/s + w/s = 250 per second and each request is resident at the device for await = 25.0ms, so the average number outstanding is 250 x 0.025 = 6.25 requests. A queue averaging 6.25 deep cannot coexist with a device that was idle 37.5% of the interval if arrivals were smooth, because a device with anything outstanding is by definition busy, and a steadily 6-deep queue would pin %util near 100. The only arrival pattern that produces both numbers is a bursty one: the device sits idle for part of each second and is then handed a clump of writes it has to drain. That is itself the diagnostic clue, and it points at a periodic writer (a flush, a checkpoint, a compaction, a log rotation) rather than steady user traffic, which is why the very next step below is to attribute the writes to a process. It also means 62.5% is a misleading headroom number: the device is not 37.5% spare, it is oversubscribed during the bursts and idle between them. The three fields are at least arithmetically consistent with each other, which is worth checking before trusting any of them: (50 + 200) x 2.50ms = 625ms of device-busy time per second, exactly the 62.5% reported.
• Treat svctm as a derived number, not a measurement. iostat computes it back out of %util and the completion rate rather than observing it, which makes it meaningless on any device that services requests in parallel (anything with a real queue depth: SSDs, NVMe, most array-backed volumes), and later sysstat releases dropped it from iostat -x output for that reason. On a current machine the fields to reach for are aqu-sz (average queue size, older builds call it avgqu-sz, and it is the same 6.25 derived above) together with r_await and w_await split by direction, since on a load that is 4-to-1 writes a single averaged await can be hiding fast reads behind very slow writes or the reverse.
• Confirm the write-heavy shape. w/s (200) is four times r/s (50); wkB/s (8192) is eight times rkB/s (1024). Whatever is generating load here is writing far more than it's reading.
• Cross-check against vmstat. CPU time shows wa=40 (40% of CPU cycles reported as idle are actually blocked on I/O), alongside us=40 and sy=10. High %wa corroborates that I/O, not compute, is where time is going. (Note: vmstat's b column, processes in uninterruptible sleep, is a point-in-time snapshot rather than an interval average, so seeing b=0 at the instant it was sampled doesn't contradict a high %wa measured as an average over the whole sampling interval.)
• Immediate action: identify what's generating the write volume, for example with pidstat -d 1 to attribute disk I/O to a specific process, and if a non-essential write-heavy job (a backup, compaction run, or log rotation) is running concurrently with user traffic, pause or deprioritize it to relieve pressure while you investigate further.
• Additional metrics to confirm: iotop for an interactive per-process view, lsof to see what has the device's files open, and a longer window of iostat -x samples (this was only a brief snapshot) to confirm the pattern is sustained rather than a short burst.

Worked example (alternate, for contrast): a healthy write-heavy device might report w/s around 120 and wsec/s (sectors written per second, one sector is typically 512 bytes) around 307200, with await around 12.5ms. Converting sectors to throughput: 307200 sectors/s times 512 bytes = 157,286,400 bytes/s, divided by 1,048,576 bytes per MB, gives about 150 MB/s of sustained write throughput. At 120 writes/sec that works out to roughly 1.25MB per write, consistent with large, sequential-style writes, and a 12.5ms await on that pattern is fast, not queued. That's what a device comfortably keeping up with a big-block write load looks like, in contrast to the given scenario's 10x await-to-svctm ratio, which is the actual signature of a device falling behind.

Trade-offs and pitfalls: it's tempting to read %util alone and conclude the disk still has headroom at 62.5%; %util measures how often the device was busy at all, not whether requests are queueing once it is, which is exactly why the await-to-svctm ratio matters more here. Acting on a single snapshot without a longer sampling window risks mistaking a brief burst for a sustained bottleneck.

Direct answer: Compare the runtime's own reported memory usage against the process's total memory footprint (RSS, resident set size, the physical memory the OS sees the whole process using). If both climb and drop together around a garbage collection, it's a leak inside the managed heap; if total memory keeps climbing while the runtime's own heap stays flat, the growth is happening in native memory the garbage collector has no visibility into.

Structured elaboration:

• Compare the two trends side by side. A managed runtime like the JVM exposes its own heap usage (via JMX, Java Management Extensions, the standard interface for reading a running JVM's internal metrics, or built-in metrics) separately from OS-level RSS. The number to plot is not heap usage at an arbitrary instant but the live set, meaning heap occupancy sampled immediately after each full collection, drawn over hours. If the live set climbs steadily toward the configured maximum, that is a managed-heap object leak. If the live set is flat after every collection while RSS keeps climbing past the maximum heap plus the JVM's known fixed overheads (metaspace, code cache, thread stacks, direct and mapped buffers, the collector's own structures), that gap is native memory growth.
• Investigate a managed-heap leak the way you would any object-retention bug: take two heap histograms some minutes apart under similar load and look for an object type whose count keeps growing well past what live traffic could require.
• Investigate native growth by checking subsystems known for it: off-heap or direct memory buffers (memory allocated outside the managed heap, often used for I/O buffering), a native library called through a foreign-function interface (FFI: a mechanism that lets code written in one language directly call functions written in another, such as Java calling into C), or simply a leaked thread count, since every OS thread reserves a fixed native stack size that adds up if threads are created and never cleaned up.
• Do not use "RSS did not fall after a full collection" as the discriminator, because it is not one. Most collectors keep the heap committed to the process rather than handing pages back to the operating system, so RSS normally sits at its high-water mark and stays there whether the leak is managed or native (G1 and ZGC can uncommit, but only under particular settings and usually after a delay, so it is not something to reason from either way). The discriminator that does work is the pair above: a flat live set alongside an RSS that keeps climbing past max-heap-plus-overheads means native.
• On the JVM there is a tool that settles it outright, and not reaching for it is the gap in most answers to this question. Start the process with -XX:NativeMemoryTracking=summary, take jcmd <pid> VM.native_memory baseline, and some hours later jcmd <pid> VM.native_memory summary.diff prints a per-category delta (Java heap, class, thread, code, GC, compiler, internal, other) that names which category grew, which turns "it is native somewhere" into a specific subsystem. For direct byte buffers there is an even cheaper read that needs no restart flag: the java.nio:type=BufferPool,name=direct MBean reports live buffer count and bytes, and a count that climbs and never falls is conclusive on its own. Know the one blind spot before you trust it: native memory tracking accounts for allocations the JVM itself makes, so a third-party native library leaking through its own malloc shows up in RSS and not in the report, and for that case you need a malloc-level profiler or a pmap -X diff across time.
• Immediate, minimal-disruption mitigation: if it's a managed-heap leak, add headroom (raise the max heap size, or disable a specific known-suspect cache) as a stopgap while the actual leak is found. If it's native and traceable to a specific subsystem, disable or shrink that subsystem through a runtime configuration flag if one exists, no code deploy required. In either case, if the service runs multiple replicas, a graceful, one-at-a-time rolling restart buys the most time with the least disruption while the real fix is developed.

Worked example: Heap occupancy read via JMX peaks around 1.2GB between collections and returns to about 900MB after every full collection, and that 900MB post-collection floor is unchanged across two days, so the live set is not growing at all. Meanwhile process RSS climbs from 2GB to 6GB over the same 48 hours. With a 2GB maximum heap configured, an RSS of 6GB is about 4GB beyond what the heap plus the usual fixed overheads can account for, and that gap is what is growing. Note what is NOT evidence here: RSS failing to drop after those collections proves nothing on its own, since the JVM keeps the heap committed either way. The flat post-collection floor is the evidence. jcmd <pid> VM.native_memory summary.diff against a baseline taken at the start of the window attributes nearly all of the growth to the category direct buffers land in rather than to Java heap, and the java.nio direct BufferPool MBean shows buffer count climbing monotonically and never falling. Checking the app's configuration reveals a metrics-reporting library allocating unbounded direct byte buffers for outbound metrics and never releasing them, a known pattern for that library's older versions. The immediate, config-only mitigation is reducing or disabling that library's buffering through its existing configuration flag, no deploy needed, while the team upgrades the library or patches the buffering behavior properly.

Trade-offs and pitfalls: "the language has a garbage collector, so it can't leak" is a common and wrong assumption used to wave off native-memory investigations. Forcing repeated full garbage collections just to test "does the heap actually drop back down" has a real latency cost on some collectors (a stop-the-world pause), so do it deliberately and sparingly on a live, loaded production instance, not as a repeated diagnostic loop; in most cases the collections the service is already doing give you the post-collection floor for free, and reading it off the existing GC log costs nothing at all.

Direct answer: Having "the right indexes" only guarantees the query planner picks a good access path, not that reading through that index stays cheap as data grows. The actual cause is one of index bloat, the planner drifting away from the index, the working set no longer fitting in cache, or a physical storage throughput ceiling, and each has its own distinct evidence.

Structured elaboration, evidence at each layer:

• Index inefficiency or bloat: many databases (Postgres, for example) accumulate dead space in a b-tree index if maintenance (vacuum) can't keep up, growing the index's physical size and lookup cost disproportionately to row count. Check the index's actual size over time against row-count growth.
• Full table scans: run EXPLAIN ANALYZE (a command that shows the actual query plan the database chose, along with real measured row counts and timing for each step, not just an estimate) on the slow query. If it shows a sequential scan instead of the expected index scan, check whether table statistics are stale, since a planner working from outdated row-count or selectivity estimates can silently switch away from the index it used to pick.
• Cache eviction: check the database's buffer cache hit ratio (the fraction of reads served from memory rather than disk) over time with the same query pattern. A declining hit ratio as the table grows, with nothing else changing, means the working set no longer fits in the cache it used to fit in.
• Storage throughput: check IOPS (I/O operations per second) and throughput utilization plus latency at the storage-volume level. If utilization is pinned near the provisioned ceiling as data grows, that's a physical limit, unrelated to the query plan at all.

Worked example: A table grows from 5 million to 50 million rows over six months. The buffer cache hit ratio drops from 98% to 82% over the same period, with the query's WHERE-clause pattern unchanged. EXPLAIN ANALYZE still shows an index scan being used, ruling out a planner regression to a full scan, but its Buffers output (which reports actual hit versus read counts for that execution) shows a much higher proportion of disk reads relative to cache hits than before. Index size has grown roughly in line with row count, not disproportionately, ruling out bloat. Storage IOPS utilization sits around 40% of provisioned capacity, well below its ceiling. The conclusion: cache eviction is the actual cause, the working set no longer fits in the buffer cache as the table grew, forcing more physical reads per query even though the index itself is correctly chosen and structurally healthy. The fix is increasing the buffer cache size, partitioning or archiving old, rarely queried rows out of the hot table to shrink the effective working set, or adding a dedicated caching layer in front of the hottest lookups.

Trade-offs and pitfalls: assuming "the index exists, so it must be fast" ignores that even a correctly used index still costs a real disk read if its relevant pages aren't cached. Rebuilding indexes as a reflexive bloat fix, when the real cause is cache eviction, wastes a maintenance window for no benefit. Seeing acceptable IOPS utilization doesn't rule out storage as a problem either, since a device can have headroom on throughput but still show elevated per-request latency under a bursty access pattern, so check latency, not just utilization.