Network Troubleshooting and Diagnostics Questions
Systematically diagnosing network problems: a layered troubleshooting methodology, diagnostic tools and commands (ping, traceroute, tcpdump, packet capture), root-cause analysis, and connectivity and user-access issues. Covers isolating faults across the stack, reading packet-level evidence, and driving from symptom to root cause. The diagnostic discipline that spans all networking layers.
Construct a BPF filter for tcpdump or a display filter for Wireshark/tshark that isolates TCP retransmissions on a busy interface, and explain the difference between what a capture filter and a display filter can each see and why that distinction matters when you're trying to keep a production capture small.
Sample Answer
Direct answer
A capture filter (tcp[tcpflags] & (tcp-syn|tcp-ack) != 0 alone won't isolate retransmissions specifically, since BPF has no built-in concept of "this segment I've seen before") is fundamentally limited here; retransmission detection genuinely requires STATE across multiple packets (has this exact sequence number been seen already), which a stateless capture filter can't express, so isolating retransmissions is really a DISPLAY-filter (or post-processing) job, not a capture-filter one.
Structured elaboration
- Why capture filters (BPF) can't do this well: BPF filters evaluate each packet independently, with no memory of prior packets; "is this a retransmission" requires comparing THIS packet's sequence number against what's already been seen for this same TCP stream, which is inherently stateful and outside what a capture filter can express.
- What Wireshark's display filter CAN do, because it operates AFTER full protocol dissection with state:
tcp.analysis.retransmissionis a Wireshark-computed field, built by Wireshark's own stateful TCP stream tracking during dissection, not something derivable from a stateless per-packet filter; this display filter, applied in Wireshark or viatshark -Y, correctly isolates retransmissions because Wireshark has already done the stateful bookkeeping. - The practical implication for capturing on a busy interface: since you can't cheaply filter for retransmissions AT CAPTURE TIME, the practical approach is to capture the full (or reasonably filtered by host/port) traffic to a file, then apply the display filter afterward during analysis, accepting a larger capture file in exchange for being able to ask this specific, stateful question after the fact.
- Capture filters versus display filters, the general principle: capture filters (BPF) are for REDUCING VOLUME at the point of capture using only per-packet, stateless criteria (host, port, protocol, flags); display filters (Wireshark's own syntax) can express much richer, STATEFUL, cross-packet logic, because they run against already-captured, already-parsed data where the tool has built up state across the whole stream.
- Why you'd prefer one over the other in production: a capture filter is cheaper (reduces what's written to disk or memory at all) and appropriate when you know in advance a simple, stateless criterion (a specific host/port) will scope the capture usefully; a display filter is necessary whenever the question itself requires state (retransmissions, duplicate ACKs, a specific stream's full analysis), and in that case you must capture broadly enough first, then filter afterward.
Worked example
On a busy interface, capture with a modest capture filter scoping to the relevant host/port (tcp and host 10.1.1.10) to keep the file a manageable size, then open it in Wireshark or run tshark -r file.pcap -Y 'tcp.analysis.retransmission' to isolate the retransmissions specifically; attempting to write an equivalent BPF capture filter for "retransmissions only" isn't achievable, because BPF has no mechanism to remember which sequence numbers it's already seen for a given stream.
Trade-offs & pitfalls
A common misunderstanding is assuming any packet-matching criterion can be expressed as a capture filter if you just find the right syntax; retransmission detection specifically cannot, because it requires state BPF fundamentally doesn't carry across packets. Recognizing this distinction (stateless capture-time filtering versus stateful post-capture analysis) up front saves time that would otherwise be spent trying to force a capture filter to do something it structurally cannot.
A service behind a VXLAN overlay is experiencing connectivity flaps between two VTEPs. Describe the troubleshooting steps and tools (show vxlan, bridge fdb, capture VXLAN/UDP packets, check underlay routing, MTU) to determine whether the problem lies in the VXLAN control plane (e.g., EVPN), underlay IP path, or MAC learning/forwarding in the bridge domain. What capture filters would you use to isolate VXLAN traffic?
Sample Answer
Direct answer
VXLAN connectivity flaps between two VTEPs (VXLAN Tunnel Endpoints) can originate in four genuinely different layers: the VXLAN control plane itself (if using EVPN, BGP EVPN session or route issues), the underlying UNDERLAY IP path between the VTEPs, MAC learning/forwarding within the bridge domain, or an MTU/fragmentation issue causing larger encapsulated frames to be silently dropped; work through them in that order since each is progressively harder to fix if misdiagnosed.
Structured elaboration
- Check VXLAN/EVPN control-plane state first:
show vxlan(or the platform equivalent) and, if EVPN is in use,show bgp evpnor equivalent BGP-session status between the VTEPs; a flapping EVPN BGP session directly explains flapping VXLAN reachability, since EVPN is what advertises which VTEP owns which MAC/IP, and an unstable session means that information itself is intermittently unavailable or withdrawn. - Check underlay IP routing between the VTEPs independently: VXLAN traffic is itself just IP/UDP traffic between two VTEP IP addresses at the underlay level; confirm the underlay ROUTING between those two specific IPs is stable (no flapping routes, no asymmetric path issues) using standard IP-layer troubleshooting (traceroute, checking underlay routing protocol stability) completely independent of anything VXLAN-specific, since an underlay instability would manifest as VXLAN flapping without VXLAN itself being misconfigured at all.
- Check MAC learning and forwarding within the bridge domain: on Linux-based VTEPs,
bridge fdb showlists the VXLAN forwarding database directly (each entry mapping a MAC to the remote VTEP IP it was learned behind); on vendor platforms,show mac address-tableor the VXLAN-specificshow bridge-domainequivalent serves the same purpose. Confirm whether MAC addresses are being learned consistently and correctly associated with the expected VTEP, or whether they're flapping between VTEPs (which itself can be a SYMPTOM of an underlying loop or duplicate MAC, similar to physical-network MAC flapping, but occurring within the VXLAN overlay's logical bridge domain instead). - Check MTU along the full underlay path: VXLAN adds roughly 50 bytes of overhead (8-byte VXLAN header, 8-byte outer UDP header, 20-byte outer IP header, 14-byte outer Ethernet header) to every original frame; if any hop along the underlay path enforces a standard 1500-byte MTU while inner traffic runs at or near 1500 bytes itself, the resulting encapsulated packet exceeds that MTU and is fragmented or, more commonly with the Don't-Fragment bit set, silently dropped, producing intermittent flaps that correlate with larger packets specifically (bulk transfers, large frames) rather than small keepalive-style traffic. Confirm with
ping -M do -s <size>style MTU-discovery tests from one VTEP toward the other's underlay IP, increasing size until drops start, and compare that ceiling against the VXLAN overhead requirement. - Capture and filter specifically for VXLAN/UDP traffic between the VTEPs: filter on the VXLAN UDP port (commonly 4789, though configurable) between the two VTEP IPs to directly confirm whether encapsulated traffic is actually flowing consistently at the underlay level. Concrete filter syntax: a tcpdump/capture filter of
udp port 4789(orhost <vtep-ip> and udp port 4789to scope to one VTEP pair) isolates VXLAN traffic at capture time; the equivalent Wireshark/tshark display filter isudp.port==4789or simplyvxlan(Wireshark's dissector recognizes and labels VXLAN packets once the UDP port matches, so thevxlandisplay filter works directly against an already-captured file). This was verified directly: capturing loopback traffic withtcpdump -i lo 'udp port 4789'and then applyingtshark -r capture.pcap -Y 'vxlan'andtshark -r capture.pcap -Y 'udp.port==4789'both parse and run without filter-syntax errors. - Determine which layer is responsible by process of elimination across these four checks together: a stable EVPN session plus a stable underlay plus MAC flapping in the bridge domain points at something overlay-specific (perhaps a genuine duplicate MAC, or a loop within the overlay itself); an unstable EVPN session with a stable underlay points at the control plane; underlay instability, regardless of what VXLAN/EVPN state looks like, points at the physical/IP layer beneath everything else; drops that correlate specifically with larger packet sizes point at an MTU mismatch somewhere on the underlay path.
Worked example
show bgp evpn shows the EVPN session between the two VTEPs stable, with no flaps. Underlay traceroute between the VTEP IPs is consistently clean, no packet loss, stable path for small packets. bridge fdb show for the affected VNI shows no MAC flapping. A capture filtered with udp port 4789 shows encapsulated packets leaving the source VTEP consistently, but larger encapsulated frames (above roughly 1500 bytes) never arrive at the far VTEP while smaller ones do; an MTU-discovery ping (ping -M do -s 1473 <underlay-peer-ip>, accounting for the ICMP/IP header) confirms one intermediate hop still enforces a 1500-byte MTU that was never raised during a partial jumbo-frame rollout. Raising that hop's MTU to at least 1550 bytes resolves the flaps.
Trade-offs & pitfalls
Jumping straight to investigating VXLAN/EVPN CONFIGURATION when the actual cause is an underlay instability, a duplicate MAC, or an overlooked MTU gap on one hop wastes time on the wrong layer; checking all four layers (control plane, underlay, MAC/bridge-domain, MTU) SEPARATELY and in this order efficiently narrows down which one is actually responsible, rather than assuming it's inherently a 'VXLAN problem' just because VXLAN is the technology in use. An MTU gap is especially easy to miss because it only manifests intermittently, correlating with packet size rather than being a constant failure.
List and interpret common ICMP types and codes you may encounter during ping/traceroute: Destination Unreachable (with host/network/port codes), TTL Expired, Fragmentation Needed, Redirect, and more. For each, explain what the ICMP message tells you about the network problem and what diagnostic step you would take next.
Sample Answer
Direct answer
The ICMP messages you'll see most often during ping/traceroute each map to a specific, distinct network condition: Destination Unreachable (with several sub-codes for host, network, or port), TTL/Time Exceeded, Fragmentation Needed, and Redirect, and reading which specific one you got (not just "it failed") tells you what to check next.
Structured elaboration
- Destination Unreachable, host unreachable: a router along the path has no route to the specific destination HOST; check routing tables at the router that generated this message.
- Destination Unreachable, network unreachable: a router has no route to the destination NETWORK at all, a broader failure than "host unreachable," usually indicating a missing or withdrawn route rather than just an ARP/reachability issue for one specific host.
- Destination Unreachable, port unreachable: the destination host IS reachable, but nothing is listening on the specific port being probed; this is actually the NORMAL, expected response that classic UDP-based traceroute relies on to detect it has reached the final destination (since it deliberately probes an unlikely-to-be-open port).
- TTL Exceeded (Time Exceeded): as covered by traceroute's core mechanism, a router decremented TTL to zero and discarded the packet; seeing this from an intermediate hop is normal and expected during traceroute, but seeing it unexpectedly during regular traffic (not traceroute) can indicate a routing loop or an unusually long path exceeding a low starting TTL.
- Fragmentation Needed (with the "don't fragment" bit set): a router needs to fragment the packet to forward it but is prohibited from doing so, and reports the MTU of the constrained link in its reply, which is exactly the message Path MTU Discovery depends on to let the sender adjust; its absence (being filtered somewhere) is the classic PMTUD black-hole failure mode.
- Redirect: a router is telling the sender that a better next hop exists for this destination than the one currently being used, typically because the sender's default gateway assumption is suboptimal for that specific destination; on modern networks, redirects are frequently disabled or ignored for security reasons (they can be spoofed to redirect traffic maliciously), so seeing them, or specifically NOT seeing them where you might expect one, both carry diagnostic meaning depending on context.
Worked example
A traceroute to a destination shows normal TTL-Exceeded replies through several hops, then a "Destination Unreachable, network unreachable" from an intermediate router instead of ever reaching the destination. This specific sub-code tells you the FAILURE is at the ROUTING layer at that specific router (it has no path to the destination network at all), distinct from what you'd see if the destination itself were simply not responding (which would produce silence or timeouts, not an explicit unreachable message) or if only a specific port were closed (which would be "port unreachable," implying the network path and host are both fine).
Trade-offs & pitfalls
Treating all "Destination Unreachable" messages as equivalent is a common mistake; the sub-code (host, network, port, and several others) carries real diagnostic information about WHERE and WHAT kind of failure occurred, and conflating them (for example, treating a normal "port unreachable" completion of a UDP traceroute as an error) leads to misreading perfectly healthy tool output as a failure.
You notice a sudden spike in traffic to one destination and need to decide, quickly, whether it's a legitimate surge, a misconfiguration, or the start of a DDoS attack. Walk through what flow and packet-level evidence you'd pull first, how you'd tell those three apart, and what you would and wouldn't do while you're still deciding.
Sample Answer
Direct answer
Deciding quickly between a legitimate surge, a misconfiguration, and a DDoS attack means pulling flow and packet-level evidence FIRST, before taking any action, since acting on a wrong guess (blocking legitimate traffic, or failing to act against a real attack) has real cost in either direction.
Structured elaboration
- Check the SOURCE diversity and pattern: a legitimate traffic surge (a viral event, a marketing campaign) typically comes from a large, geographically and behaviorally diverse set of real client IPs with normal-looking request patterns; a DDoS attack often shows either a smaller set of sources sending disproportionately high volume, or a very large set of sources with suspiciously uniform, repetitive request patterns (a signature of automated/botnet traffic); a misconfiguration (a retry loop, a broken client) often shows a SMALL number of sources sending an abnormally HIGH, repetitive rate.
- Check what's actually being requested: legitimate surges typically request a diverse mix of real content/endpoints; an attack or a broken retry loop often hits the SAME endpoint repeatedly, sometimes with malformed or unusual parameters.
- Check protocol-level signatures: a SYN flood (many SYNs, few or no completed handshakes) is a clear DDoS signature distinguishable at the packet level; a legitimate surge completes handshakes normally at whatever elevated rate it's arriving.
- Check timing correlation against known events: does this align with a marketing push, a scheduled batch job, or a deploy that might have introduced a client-side retry bug? Correlating against your OWN recent changes and known external events is often the fastest way to explain a spike without needing deep packet analysis at all.
- What NOT to do while still deciding: avoid broad, aggressive blocking (which risks dropping legitimate traffic if this turns out to be a real surge) and avoid doing nothing while evidence accumulates that it IS an attack; apply narrow, reversible measures (rate-limiting rather than an outright block) as a middle ground while you continue gathering evidence.
Worked example
A spike shows requests arriving from several hundred thousand distinct source IPs, geographically distributed in a pattern that roughly matches the service's normal user base, hitting a diverse mix of endpoints, with completed TCP handshakes and normal-looking HTTP responses. This pattern strongly resembles a legitimate surge, not an attack; correlating the timing against a company blog post that was published and went viral on social media roughly 20 minutes before the spike began, confirms it. No blocking action is taken; instead, capacity is scaled up to absorb the legitimate demand.
Trade-offs & pitfalls
The cost of misjudging this in either direction is real: blocking too aggressively on a false-positive "attack" assessment turns a legitimate traffic surge, potentially a business win, into a self-inflicted outage; failing to act on a real attack because you're waiting for more certainty lets damage continue. Narrow, reversible measures (rate limiting specific abusive patterns rather than broad IP blocks) let you respond proportionally while evidence is still being gathered, rather than forcing an all-or-nothing decision immediately.
During a release, new iptables rules were applied via automation and some services became unreachable. Explain a safe process for applying iptables/nftables changes in production, including validation steps, atomicity concerns, rollback strategies, and how to use 'iptables-restore' or nftables sets to minimize disruptions.
Sample Answer
Direct answer
A safe process for applying iptables/nftables changes in production treats every change as something that must be validated before it's permanent, and reversible within seconds if it isn't: validate the ruleset syntactically and logically before applying, apply it in a way that's atomic (not a sequence of individual commands that could leave a half-applied, inconsistent state if interrupted), and have an automatic rollback if the change breaks connectivity to the very system managing it.
Structured elaboration
- Validate before applying:
iptables-restore --test(or the nftables equivalent, checking syntax without committing) catches malformed rules before they ever touch the live ruleset, a cheap, fast check that prevents an entire class of avoidable outage. - Apply atomically, not incrementally:
iptables-restore(loading a complete, prepared ruleset in one atomic operation) avoids the risk inherent in running a SEQUENCE of individualiptables -A/-Dcommands, where an interruption partway through (a script failure, a connection drop mid-execution) can leave the system in an inconsistent, partially-applied state that's neither the old ruleset nor the new one;nftablessets and atomic rule replacement provide an equivalent guarantee. - Build in an automatic, timed rollback: apply the new ruleset with a companion mechanism (a scheduled job that reverts to the previous, known-good ruleset after a short window, UNLESS explicitly confirmed) so that if the new rules break the very access needed to manage the device (locking yourself out), the system self-heals without requiring physical or out-of-band access to fix it.
- Test connectivity to a few known-critical endpoints immediately after applying, before considering the change complete, and only cancel the scheduled rollback once that's confirmed; this closes the gap between "the ruleset loaded without a syntax error" and "the ruleset actually preserves the connectivity we need."
- Stage the change through a representative, lower-risk environment first, where feasible, particularly for a change generated by automation that's new or has been recently modified, since automation itself can have bugs that only manifest against real, more complex production rule sets.
- Keep every prior ruleset version retrievable, not just the most recent one, so a rollback can go back further than one step if a chain of recent changes turns out to be implicated together.
Worked example
An automated pipeline generates a new ruleset, tests it with iptables-restore --test (passing), then applies it via iptables-restore (atomic load) rather than a sequence of individual commands. Immediately after applying, an automated check confirms SSH management access and a handful of critical service ports remain reachable; a scheduled rollback job, set to fire in 5 minutes unless explicitly cancelled, is cancelled only after that confirmation succeeds. In a prior incident (motivating this exact process), a ruleset applied without this safeguard had inadvertently blocked the very management port needed to fix it, requiring out-of-band console access to recover; the timed-rollback mechanism specifically exists to prevent a repeat of that failure mode.
Trade-offs & pitfalls
Applying a validated-but-unconfirmed ruleset change without a rollback safeguard risks a scenario where the change breaks access to the device managing it, turning a routine change into an access incident requiring physical or out-of-board recovery; the timed, automatic rollback specifically closes that gap. Testing only for SYNTACTIC validity (the ruleset parses correctly) is not the same as testing that it preserves the SPECIFIC connectivity actually needed; both checks matter and neither substitutes for the other.
Unlock Full Question Bank
Get access to all Network Troubleshooting and Diagnostics interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.