Performance Troubleshooting & Incident Response Questions

Frequent TCP retransmissions mean packets are being resent because the sender didn't get an acknowledgment (ACK) in time, and to the application this shows up as added latency and sometimes intermittent errors. It points to something dropping or delaying packets between the two services, not necessarily a bug in either application's own code.

Root cause 1: network congestion or packet loss on the path (a saturated link, a lossy hop, a misbehaving switch or network interface). Validate with ping for baseline loss and jitter, a hop-by-hop tool like mtr to see which specific hop introduces loss, and OS-level retransmit and drop counters (ss -i or netstat -s for retransmission counts, ethtool -S for network-interface drop and error counters).

Root cause 2: receiver-side resource exhaustion, where the receiving host can't keep up with arriving traffic (for example because it's CPU-saturated or the application's read loop is slow). Be precise about the mechanism here, because two different things happen on a struggling receiver and only one of them produces retransmissions. If the application is slow to read but the kernel still accepts the packets, the receive buffer fills and the receiver advertises a zero window; the sender then stops sending and issues periodic window probes on a backing-off persist timer. That is a stall, not a retransmission: the data already in the buffer was acknowledged, and when the window reopens the sender sends new data rather than re-sending anything. It shows up in a capture as zero-window advertisements and probe packets, and in ss -i as a high rwnd_limited percentage, with the retransmission counter flat. Retransmissions come from the other half of receiver exhaustion: when the NIC ring buffer or the socket receive queue actually overflows, the kernel drops arriving segments before they are ever acknowledged, so the sender's retransmission timer fires. Validate by checking the receiver's CPU and load at the same timestamps as the retransmits, then look specifically for drop counters rather than window stats, netstat -s for socket buffer and backlog drops and pruned receive queues, ethtool -S for per-queue rx_dropped and ring overrun counters, alongside receive-window and buffer stats from ss -i and the receiving application's own read-loop latency. The practical decision rule: if you find a zero-window stall but no drops and no retransmits, the slow reader is real but it is not what is causing the retransmissions you were sent to explain, and you should keep looking at causes 1 and 3.

Root cause 3: a middlebox or MTU (maximum transmission unit) mismatch, for example a firewall silently dropping specific packet sizes, or a path-MTU issue where large packets never arrive intact. Validate with a packet capture (tcpdump, or Wireshark's retransmission filter) to see whether retransmits cluster around a specific packet size, and test the actual path MTU directly with a tool like tracepath.

Worked example: a tcpdump capture between the two services shows the exact same full-size segment being retransmitted repeatedly, a 1500-byte IP packet carrying the 1460-byte maximum segment size that a 1500-byte Ethernet MTU leaves after the 20-byte IP and 20-byte TCP headers, while smaller requests never retransmit. Running tracepath between the hosts shows a hop reporting a path MTU of 1400, well below the assumed 1500-byte Ethernet MTU, meaning a VPN or tunnel hop along the path is fragmenting or dropping the larger packets. Lowering the effective segment size (or fixing MTU discovery so it isn't blackholed) removes the retransmissions entirely.

Trade-offs and pitfalls: retransmissions alone don't tell you which of the three causes applies, you need a second correlated signal (loss location, receiver load, or packet-size pattern) to distinguish them. Jumping straight to "add more bandwidth" without confirming which cause is actually at play can spend money without fixing anything, a congestion-focused fix does nothing for an MTU blackhole.

Direct answer: A strong answer to this question shows three things in sequence: how you triaged and made a decision before you had a confirmed root cause, what you shipped as a reversible stop-gap versus what you shipped as the durable fix, and what changed on the team afterward so the same failure mode doesn't come back. "I fixed it and it worked" tells the interviewer nothing.

Situation and Task: I was the on-call backend engineer for a checkout service. About twenty minutes after a routine deploy, our p95 latency (the 95th percentile response time: 95% of requests come in faster than this number, so it captures the slow tail an average would hide) jumped from around 300ms to over two seconds, and the 5xx error rate started climbing as some requests began timing out. My job in the first few minutes wasn't to find the root cause, it was to stop the customer-facing bleeding and figure out fast whether the deploy was even the cause.

Action and Result: I pulled the deploy timeline next to the latency graph and split the latency metric by build version. It was a rolling rollout, so twenty minutes in most instances were already serving the new build while a handful had not been updated yet: the new-build instances showed the spike and the not-yet-updated ones sat at their normal 300ms. That split is what let me act on twenty minutes of data instead of waiting for a root cause, and it is also why the version label on the latency metric matters more than it looks: without it the graph is one line that went up and every hypothesis stays alive. Rather than wait for a full root cause, I rolled the deploy back right away since it was low-risk and reversible, and asked a teammate to pull a thread dump and a slice of traces from one struggling instance before it was torn down, so we wouldn't lose the evidence. Latency recovered within a few minutes of the rollback finishing. Working from the preserved traces afterward, we found that the new deploy had added a synchronous call to a rarely used fraud-check service with no timeout set, so a handful of slow fraud-check responses were tying up request threads and building a backlog. Longer term, we added an explicit timeout and circuit breaker (a safeguard that stops sending requests to a dependency once its failure rate crosses a threshold, so it gets a cooldown to recover instead of being hit while already failing) around that call, added a load test that specifically exercises dependency slowness before merge, put a real canary stage in front of this service's rollouts (a deliberately small first slice of traffic on the new build, watched against the old build's percentiles before the rollout proceeds, so a regression like this one is caught at a few percent of exposure instead of after it has reached almost every instance), and turned the incident writeup into the template our team now uses for "mitigate first, diagnose second" decisions.

Trade-offs and pitfalls: A weaker version of this story stops at "I rolled it back and it got better," with no explanation of why the rollback was the right call at that moment or what evidence justified it. It's also a mistake to only tell the mitigation half of the story: an interviewer specifically wants to hear the long-term change, because a stop-gap that never gets followed up on is the same incident waiting to happen again.

Direct answer: When only one region shows errors and latency while the rest are fine, and your application code is presumably identical across regions, the fastest path is to check the cheapest, most likely regional causes first, provider status, network path, regional infrastructure health, before assuming it's an application bug.

Structured elaboration, a prioritized runbook (a step-by-step checklist for handling this specific type of incident):

• Check the cloud provider's status page for that region first. It's the cheapest possible check: is there a known provider-side incident already?
• Check the regional network path. DNS resolution and routing into that region, and load balancer or CDN health checks specific to that region's edge nodes, ideally from more than one vantage point.
• Check regional infrastructure health. Are the compute instances or pods in that region's cluster healthy (node status, restart counts), and is a region-pinned dependency, like a regional database replica or cache, degraded?
• Only then suspect application-level causes. If network and infrastructure both look clean, check whether that region is running a different deploy version, config, or feature-flag state than the healthy regions; region-scoped rollouts are common and easy to overlook.
• Quick mitigations: if your architecture supports it, fail over that region's traffic to a healthy region via DNS or a global load balancer, the fastest way to restore service, at the cost of added latency and possibly data-locality considerations for those users. If failover isn't available, scale up capacity in that region in case it's a capacity issue, or roll back a region-scoped deploy if one is found.
• Telemetry to capture for post-incident analysis: the exact start and end time of impact, provider status-page timestamps if there was an outage, per-region deploy and config diffs at the time, the traffic-shift decisions and their timestamps, and an estimate of customer impact (percent of traffic, duration).

Worked example: The provider status page shows nothing unusual. Synthetic probes hitting that region's load balancer show elevated TCP connect time, about 150ms versus 5ms elsewhere, pointing at the network rather than the application. Checking that region's routing configuration turns up a routing-table change pushed a few hours earlier that unintentionally restricted a previously direct path, adding extra hops. As a quick mitigation, that region's traffic is failed over to the next-nearest healthy region via the global load balancer, accepting added latency for those users temporarily, while networking reverts the routing change; traffic is failed back once synthetic probes confirm connect times have returned to baseline.

Trade-offs and pitfalls: failing traffic over to another region isn't free, it adds latency and can conflict with data-residency requirements for that user base, so it needs a fast, deliberate go or no-go decision, not silence. Checking application-level causes last is a deliberate ordering choice (cheapest and most likely first), not a reason to skip it: if network and infrastructure both check out clean, a region-scoped rollout is a very common actual cause and needs to be explicitly ruled in or out, not assumed innocent just because "the code is the same everywhere."