Latency Analysis & Optimization Questions

How probes can cause or worsen latency spikes

• A liveness probe that does real I/O, such as a database ping, competes for the same resources (CPU, connection-pool slots, I/O bandwidth) as actual user traffic. Under a traffic surge, the probe's own request can be slow to answer, and if it misses its timeout, Kubernetes kills and restarts the pod. That's strictly worse than a slow-but-working pod: the pod is now completely gone from the load balancer for the restart duration.
• Losing a pod to a false-positive restart shifts its share of traffic onto the remaining pods, pushing them closer to saturation and making the next probe on those pods more likely to time out too. This is a positive feedback loop that turns "briefly overloaded" into a cascade of restarts.
• A readiness probe, which controls whether a pod receives traffic at all (distinct from liveness, which controls whether the pod gets killed), running the same slow, I/O-heavy check can flap a healthy pod in and out of the load balancer under load, adding latency by shrinking the effective pool size exactly when full capacity is needed most.

Concrete fixes

• Split the checks: liveness should answer "is this process alive and not deadlocked," a cheap in-memory check, such as an HTTP handler that just returns 200 without touching the database, never something that can be slow under load. Readiness can check real dependencies, but with its own short timeout and a failureThreshold (how many consecutive failures before Kubernetes acts) high enough to survive a brief blip rather than one slow response.
• Tune probe intervals, timeouts, and thresholds explicitly. For example, periodSeconds: 10, timeoutSeconds: 2, failureThreshold: 3 means a pod needs 3 consecutive failed checks, 30 seconds of trouble, before Kubernetes acts, instead of one slow response causing an immediate restart.
• Move any I/O-based health check off the request-serving path: have a background thread periodically check the dependency and cache the result, so the probe handler just reads that cached status instead of doing live I/O on every probe call. This decouples probe cost from live traffic load entirely.
• Add a startupProbe for slow-starting applications so the liveness probe doesn't even begin until startup is confirmed done, avoiding a normal (slow) boot being mistaken for a hang.
• Set reasonable resource requests and limits, and a PodDisruptionBudget (a Kubernetes policy that limits how many of a service's pods can be taken down at once, protecting capacity during maintenance or node changes), so the scheduler doesn't pack pods so tightly that a traffic surge starves the probe handler of CPU in the first place.

What's happening

CPU steal time (visible as %st in top, or as a dedicated metric from most cloud providers) is the percentage of time a virtual CPU wants to run but the hypervisor instead gives the physical core to a different tenant's VM. High steal time with periodic latency spikes is the classic noisy-neighbor fingerprint: the affected service's own code isn't slower, it's simply not getting scheduled onto a physical core when it needs to be.

Quick host-level measures

• cgroups (control groups, a Linux kernel mechanism for limiting and isolating a group of processes' resource usage) with CPU shares/quota: guarantee a CPU floor for the affected service's containers so a noisy co-tenant can't starve it entirely, or, if you control the noisy tenant, cap its cpu-shares directly instead.
• CPU pinning/affinity: pin the latency-sensitive processes or containers to specific cores so at least part of the host isn't shared with the noisy workload.

Scheduler or hypervisor options

• Move latency-sensitive workloads to an instance type or hypervisor configuration with stronger isolation guarantees, such as dedicated/bare-metal instances or a cloud provider's dedicated-host tier that guarantees no co-tenancy, if the steal time is coming from cloud multi-tenancy.
• Ensure NUMA-aware scheduling (NUMA: Non-Uniform Memory Access, where a CPU accesses memory attached to its own socket faster than memory on a different socket) keeps a VM's vCPUs and memory pinned to the same NUMA node, since cross-node memory access latency compounds on top of a scheduling problem rather than being independent of it.

Longer-term capacity or placement changes

• Anti-affinity rules so latency-sensitive workloads are never co-scheduled on the same physical host as known bursty or noisy workloads.
• Move to larger instance sizes (fewer, bigger tenants per host reduces how many neighbors you can have) or to a provider/region with better bin-packing transparency and co-tenancy controls.
• Right-size capacity generally, so the affected service isn't itself contributing to the contention it's suffering from.

Measuring improvement

Track CPU steal-time percentage (it should trend toward zero) side by side with p95/p99 latency for the affected service over the same window. Specifically check whether latency spikes stop correlating in time with steal-time spikes, rather than only checking whether the average latency improved, since noisy-neighbor impact is typically bursty and periodic rather than a constant, steady degradation.

Diagnose the shape of the problem

0.1% of keys driving 80% of reads is an extreme power-law (Zipfian) skew. The important detail is where those keys physically live: if the cache uses simple consistent hashing (a scheme that maps each key to a point on a hash ring, so a given key always lands on the same shard node), a handful of hot keys can pin themselves to one or two of many shard nodes, so those specific nodes are overloaded even though the cluster-wide average utilization looks completely healthy. Confirm this by mapping the sampled per-key request counts to shard/node ownership before proposing a fix.

Plan

1. Local (near) cache in the client with a short TTL (time-to-live, how long a cached value is trusted before re-fetching), for example 200-500ms. This bounds how often the hot key is even asked of the shared cache: with a 300ms TTL, a shared near-cache tier needs to refresh a given key at most once every 300ms (about 3 refreshes/sec) no matter how many reads arrive for it, regardless of whether that demand is 1,000 or 100,000 reads/sec.
2. Key-level replication for identified hot keys: store the value under N suffixed keys (key:0 ... key:N-1) on N different nodes, and have clients pick a replica at random or round-robin. This spreads the same 80% of reads across N nodes instead of concentrating them on one.
3. Request coalescing (single-flight): when a hot key misses, ensure only one in-flight fetch to the backend happens per key per node; concurrent requests for that key wait on the one fetch in progress instead of each triggering a separate origin call. This specifically protects against a stampede when a hot key's TTL expires.
4. If the hot-key population is small and known in advance, consider a dedicated cache tier or extra replicas just for it, so a spike there cannot degrade the shared cluster.

Trade-offs

Local caching trades a bounded staleness window for load reduction. Key replication costs extra memory and write amplification (every write to a hot key must fan out to N replicas) in exchange for read scalability. Coalescing adds a small wait for the "unlucky" requests behind an in-flight fetch, but caps backend load during a burst.

What to monitor to confirm it worked

Per-key (or per-key-bucket) request rate and per-node CPU/network, to confirm the skew moved from concentrated to spread. The p99 latency specifically on the previously-hot keys, not just the cluster aggregate, since an aggregate p99 can look fine while one shard is still degraded. Cache hit ratio and the actual observed staleness vs. the TTL you set. If you replicated keys, the write-amplification rate those replicas now cost.