Direct answer. Requests hanging (rather than failing outright) after a network partition is a specific and important clue: it usually means a caller is waiting on a connection or response that will never arrive because the far side is unreachable, not that the far side is actively rejecting requests, so the mitigation has to focus on TIMEOUTS as much as on the partition itself.
Structured elaboration.
- Confirm the partition's scope. Which specific network path or region boundary is affected, and which services or calls cross it? A partition rarely means 'everything is down'; it usually means 'calls crossing this specific boundary are affected', and calls that don't cross it should be unaffected. Concretely, this means checking per-destination error-rate and latency dashboards (segmented by downstream service and region) to see exactly which calls are degraded, connection-pool and thread-pool utilization/saturation metrics to see which resources are being tied up by hung requests, host- and network-level logs (TCP connection resets, SYN timeouts, retransmission counts) to confirm packets genuinely aren't getting through rather than just running slow, and any circuit-breaker or timeout-trip logs that are already firing, since those tell you which calls the system itself has already identified as failing.
- Understand why some requests succeed and others hang. Requests that don't need to cross the partition succeed normally. Requests that DO need to cross it will hang if there's no timeout configured on that call, or fail relatively quickly if there is one; a service with inconsistent timeout configuration across its various outbound calls will show exactly this kind of mixed 'some succeed, some hang' pattern.
- Immediate mitigation. Apply or tighten timeouts on the specific calls that cross the partition, so hanging requests fail fast instead of consuming resources (a connection, a thread, a request slot) indefinitely; a service where every incoming request eventually hangs waiting on an unresponsive downstream can exhaust its own capacity even though ITS code has no bug. If there's a fallback path or cached data available, serving degraded results instead of hanging is often better for users than waiting.
- Longer-term fixes. Ensure every outbound call has an explicit, sane timeout as a standing practice, not just for this partition. Add circuit breakers so that once a downstream is detected as unreachable, subsequent calls fail immediately instead of each one re-attempting the same doomed wait. Consider whether critical paths need a documented fallback behavior for partition scenarios specifically (serve stale data, degrade a feature, queue the write for later) rather than leaving the behavior undefined and discovering it live during an incident.
- Restore consistency once the partition heals. Any writes that were queued, retried, or that partially succeeded during the partition need reconciliation: check for duplicate side effects (from client-side retries during the hang), and confirm read paths are serving current data again, not still routed to a stale fallback.
Worked example. Suppose the partition separates region A from region B, and a payment service in region A calls a fraud-check service in region B with no configured timeout on that specific call. Requests through the normal path (fraud-check reachable) take their usual 30 to 50ms; requests during the partition hang until the underlying TCP connection itself times out, which at the OS level might take 60 to 130 seconds by default, far longer than any reasonable user-facing request should ever wait, and each hung request holds a thread or connection slot the whole time. Adding an explicit 2-second application-level timeout on that specific call (well above its normal 30 to 50ms but far below the OS default) means a partitioned call fails fast, freeing the resource, rather than hanging for over a minute; combined with a circuit breaker, subsequent calls during the same partition would fail immediately without even attempting the call, once the breaker trips.
Trade-offs and pitfalls. Setting timeouts too aggressively can cause false failures during normal, brief latency blips, so the value needs to be based on the call's actual normal latency distribution, not an arbitrary round number. It's also worth checking whether the SAME issue (missing or overly long timeouts on cross-partition calls) exists on other calls beyond the one that caused this specific incident, since a partition is exactly the kind of event that exposes every under-configured timeout across a system at once.
