Framing
Softirq (a deferred, lower-priority kernel interrupt-handling mechanism run after the initial hardware interrupt returns) spikes coinciding with packet loss have at least four plausible causes: the NIC (network interface card) hardware genuinely cannot keep up, IRQ (interrupt request) affinity is misconfigured so one CPU core does all the packet-processing work, a kernel network-stack bottleneck is in play, or the traffic itself is adversarial. The investigation should be layered from cheapest/safest checks to most invasive, and should not assume the network team's 'NICs are healthy' claim covers software-layer drops.
1. Application socket layer first
Check ss -s and ss -tanp for socket-level drop counters, and nstat for TcpExtListenDrops / TcpExtTCPBacklogDrop (or RcvbufErrors for UDP). If the application's socket receive buffer (SO_RCVBUF) is too small for a traffic burst, packets are dropped at the socket layer, which looks identical to a NIC problem from the outside but has nothing to do with the network hardware. Also check /proc/net/softnet_stat column 2 (packets dropped because the per-CPU backlog queue was full): a nonzero, rising value there points at the kernel processing softirqs slower than packets arrive, or net.core.netdev_max_backlog being sized too small.
2. IRQ affinity
cat /proc/interrupts shows whether the NIC's interrupts are spread across cores or all pinned to one. mpstat -P ALL 1 showing one core pegged near 100 percent %soft while others sit idle is the classic signature of an IRQ affinity misconfiguration: all NIC interrupts landing on a single core saturates it long before the NIC's actual throughput limit is reached. The fix is enabling irqbalance, manually spreading /proc/irq/<n>/smp_affinity, or enabling RPS/RSS (Receive Packet Steering / Receive Side Scaling, kernel and hardware features that distribute incoming packet processing across multiple cores).
3. NIC ring buffers and offloads
ethtool -g eth0 shows current versus maximum RX/TX ring buffer sizes; undersized rings overflow under bursty traffic and drop packets before they even reach the IP stack. ethtool -S eth0 exposes hardware-level counters like rx_fifo_errors and rx_missed_errors, which are distinct from the software drops in softnet_stat and specifically indicate the NIC itself could not keep up. ethtool -k eth0 shows offload state (checksum offload, TSO/GSO: TCP/Generic Segmentation Offload, features that let the NIC or kernel batch segmentation work); a misbehaving offload, occasionally seen after a driver or firmware update, can masquerade as random packet loss. Disabling one offload at a time (ethtool -K eth0 tso off gso off) and observing whether the symptom changes isolates a driver bug from a genuine capacity limit.
4. Isolating host vs NIC vs network path
This is the step that actually answers 'where' the loss happens, and needs the application-socket-layer check from step 1 as a baseline first, since a clean isolation test still means nothing if the drops were happening in the socket layer all along.
- Run
iperf3 between the affected host and a different host on the same rack/segment: clean results there but bad results to the original destination points at the network path, not this host.
- Run
iperf3 loopback (host to itself) under deliberate CPU load to see whether softirq spikes reproduce purely from CPU contention, isolating host-CPU-bound processing from anything NIC- or network-related.
- Capture with
tcpdump -i eth0 -w capture.pcap on this host, and if possible simultaneously at a switch SPAN port or the peer, then compare: packets present in the host's own capture but never reaching the application indicate a socket-layer drop; packets never reaching the host's NIC capture at all despite being sent from the peer indicate network-path loss; packets visible at the peer but absent from this host's capture entirely indicate a NIC hardware or driver drop.
5. Ruling in or out something adversarial
A genuine DDoS or traffic spike typically shows as an increase in total packet/byte rate (ethtool -S rx_packets rate, or ip -s link) correlated with the softirq spike and originating from many distinct source addresses (visible via the tcpdump capture or conntrack -L growth). A misconfiguration, by contrast, usually shows constant or only modestly elevated traffic volume with a disproportionate drop or softirq rate relative to that volume, which is the tell that distinguishes 'too much traffic' from 'the host cannot handle even normal traffic efficiently'.
