Network Troubleshooting and Diagnostics Questions
Systematically diagnosing network problems: a layered troubleshooting methodology, diagnostic tools and commands (ping, traceroute, tcpdump, packet capture), root-cause analysis, and connectivity and user-access issues. Covers isolating faults across the stack, reading packet-level evidence, and driving from symptom to root cause. The diagnostic discipline that spans all networking layers.
What is MTU, and how can fragmentation or an oversized packet cause application failures even when basic connectivity tests succeed?
Sample Answer
Direct answer
Maximum Transmission Unit (MTU) is the largest packet size a given network link can carry without fragmenting it. When a packet larger than the path's MTU is sent with the "don't fragment" bit set, it's dropped rather than split, and depending on whether the resulting ICMP error makes it back to the sender, this shows up as failures that look nothing like a typical connectivity problem: basic reachability tests succeed because they use small packets, while anything larger silently fails or hangs.
Structured elaboration
- Why small tests can pass while real traffic fails: a default
pingtypically sends a small packet (often 56 or 64 bytes of payload), far under any realistic MTU, so it succeeds even when a path has an MTU problem that only manifests on larger packets, like a full-size TLS handshake message or a large file-transfer segment. - What happens with fragmentation allowed: a router along the path that supports fragmentation can split an oversized packet into smaller pieces, at some CPU and latency cost; if any device along the path doesn't support it, or if the "don't fragment" (DF) bit is set (as it commonly is by default for TCP, via Path MTU Discovery), the packet is simply dropped and an ICMP "fragmentation needed" message should be returned to the sender.
- The failure mode when that ICMP message is filtered: some firewalls or middleboxes drop ICMP entirely as a matter of policy; when that happens, the sender never learns its packets are too large and never reduces its segment size, so the connection appears to hang indefinitely on anything beyond a small handshake, a classic and specifically named failure mode (a PMTUD, Path MTU Discovery, black hole).
- How this differs across encapsulation: any tunnel or overlay (VPN, VXLAN) adds its own header overhead, effectively lowering the usable MTU for the traffic it carries; a path that has no MTU problem for plain traffic can develop one the moment that traffic starts traversing a tunnel, if the tunnel's overhead isn't accounted for.
Worked example
A user can load a small webpage (a few KB, fitting easily in a handful of packets) but a large file download over HTTPS hangs. The TLS handshake itself may complete (its messages are often small enough to fit), but once the server starts sending full-size, DF-set segments of application data, one exceeds a 1400-byte link MTU somewhere in the path; the router there sends an ICMP "fragmentation needed, MTU 1400" message, but an intervening firewall drops all ICMP, so the server's sender never receives it, never shrinks its segment size, and keeps retransmitting the same oversized segment that keeps being dropped, producing a hang rather than a clean error.
Trade-offs & pitfalls
"Basic connectivity works" is not evidence against an MTU problem, since the tests that establish basic connectivity (a ping, a handshake) are usually far smaller than the packets that will actually trigger the fault; always test explicitly with larger packet sizes (or transfer a real file) before ruling MTU out. Do not assume ICMP being unreachable across a link means ICMP is entirely broken there; some networks deliberately filter only certain ICMP types, and "fragmentation needed" specifically is one that must be allowed through for Path MTU Discovery to function at all.
Describe how TTL (Time To Live) works in IP and how traceroute exploits TTL to discover network hops. Explain what it means when TTL values decrease from multiple vantage points and how you would interpret a traceroute that shows a long pause at a particular hop or a hop that alternates between responding and '*'.
Sample Answer
Direct answer
Time To Live (TTL) is a field in the IP header that's decremented by one at every router hop; when it reaches zero, the packet is discarded and the router that dropped it sends back an ICMP "time exceeded" message. Traceroute exploits this deliberately: it sends a series of probes with TTL starting at 1 and incrementing by one each time, so each successive probe dies at one hop further along the path, and each dying hop's ICMP reply reveals its identity.
Structured elaboration
- Why TTL exists at all: it's a safety mechanism to prevent a packet from looping forever if a routing misconfiguration creates a loop; without it, a looping packet would circulate indefinitely, consuming bandwidth and router resources.
- How traceroute turns this into a diagnostic tool: probe 1 goes out with TTL=1, dies at the first router, which replies with "time exceeded," revealing hop 1's address; probe 2 goes out with TTL=2, survives hop 1, dies at hop 2, revealing hop 2's address; and so on, until a probe finally reaches the destination and gets a normal reply (or, for UDP-based traceroute, an ICMP "port unreachable," since the destination port is deliberately chosen to be unused).
- What a hop that alternates between responding and showing
*typically means: that specific router is inconsistently generating or sending the ICMP time-exceeded reply, often because it deprioritizes or rate-limits ICMP generation relative to normal forwarding (a very common, largely benign router behavior, not necessarily a sign that FORWARDING through that hop is actually failing); the packet may well be forwarded correctly even when the ICMP reply informing you of that fact is dropped or delayed. - What a long pause at a specific hop, followed by normal timing at the next, usually means: that specific router is slow to GENERATE the ICMP reply (often because ICMP generation is handled by a slower control-plane path on that device, deprioritized relative to fast-path forwarding), not necessarily that forwarding through it is actually slow; compare this against the round-trip time to hops AFTER it, since if those return to normal, the delay was specific to that hop's ICMP generation, not a real, cumulative path problem.
Worked example
A traceroute shows hops 1 through 4 all responding in 1 to 3 ms. Hop 5 shows * * * (no reply at all) for all three probes, but hop 6 responds normally at 4 ms and the destination (hop 7) replies successfully. This pattern, an unresponsive hop sandwiched between two responsive ones with normal timing on either side, is the classic signature of a router that simply doesn't generate or deprioritizes ICMP time-exceeded replies (a common, deliberate configuration on some devices), not evidence that the path is actually broken at hop 5; the fact that hop 6 and the final destination both respond normally confirms packets ARE making it through hop 5 just fine.
Trade-offs & pitfalls
The most common misreading of traceroute output is treating a non-responding hop as proof of a problem AT that hop; in most cases, especially when later hops respond normally, it means that specific device isn't generating (or is rate-limiting) its own ICMP replies, which is unrelated to whether it's actually forwarding your traffic correctly. Only treat a hop as genuinely suspect if the path either never recovers after it (all subsequent hops also fail) or if there's other corroborating evidence, like actual application-level loss or latency, pointing at the same location.
Describe the minimum steps and capture settings you would use to perform a packet capture that is useful for troubleshooting a connectivity issue. Cover capture location (host vs tap vs SPAN), interface selection, capture filters, snaplen, buffering, and legal/privacy considerations when capturing production traffic.
Sample Answer
Direct answer
A packet capture that's actually useful for troubleshooting a connectivity issue needs deliberate choices made before you start, not just running tcpdump with no arguments: where you capture, what you filter for, how much of each packet you keep, and what legal/privacy constraints apply, all affect whether the capture actually answers your question later.
Structured elaboration
- Capture location: decide whether you need to capture on the host itself, at a switch SPAN/mirror port, at a network TAP, or (in cloud environments) via VPC traffic mirroring; a host-based capture sees only what that host sends/receives (and may miss packets dropped before reaching the NIC), while a SPAN or TAP sees the wire itself, which matters when you suspect a problem BETWEEN two points rather than at either endpoint.
- Interface selection: capturing on the wrong interface (a management interface instead of the data interface, or the wrong VLAN sub-interface) is a common, entirely avoidable mistake that produces an empty or irrelevant capture.
- Capture filters, applied at capture time: a BPF filter (host, port, protocol) applied while capturing reduces the volume you need to store and analyze, at the cost of not being able to go back and look at anything outside that filter later; decide this trade-off deliberately rather than by default.
- Snaplen (how much of each packet to keep): capturing only headers (a small snaplen) is enough for most connectivity troubleshooting and keeps files small; capturing full payload is needed only if you specifically need to inspect application-layer content, and comes with real privacy implications.
- Buffering and rotation: for anything beyond a quick, short capture, set file-size-based rotation (so a long-running capture doesn't fill the disk) and consider ring-buffer behavior, so an unattended capture degrades gracefully rather than crashing the host.
- Legal and privacy considerations: capturing production traffic can capture sensitive data (credentials, personal data) depending on snaplen and what's being captured; know your organization's policy on this before capturing broadly, and prefer the smallest scope (narrowest filter, smallest snaplen) that still answers your question.
Worked example
Troubleshooting an intermittent connectivity failure between a client and a specific backend: capture at the host closest to where the symptom is reported (not several hops away), filtered to the specific host/port pair involved (host <client-ip> and port 443), with a modest snaplen (enough to see full TCP/TLS headers, not necessarily full payload), rotating to 100MB files so an extended capture window doesn't fill the disk while you wait for the intermittent failure to recur.
Trade-offs & pitfalls
Capturing everything with no filter "just in case" produces a file that's expensive to store, slow to analyze, and more likely to contain sensitive data you didn't need; conversely, filtering too narrowly before you fully understand the failure risks missing the very packet that would have explained it. Start with a reasonably scoped filter based on what you already know, and widen it only if the first capture doesn't answer the question.
An application uses UDP for real-time telemetry and reports jitter and packet loss. Describe which packet capture indicators you would use to measure jitter, out-of-order delivery, and loss for UDP flows, and what network-layer causes (bufferbloat, congestion, incorrect QoS, NIC offload behavior) you would investigate. Include how to use tc and queue discipline to test remediation.
Sample Answer
Direct answer
For a UDP-based real-time telemetry service, use packet-level indicators that do not depend on any built-in protocol acknowledgment (since UDP has none): sequence gaps (if the application includes its own sequence numbers), inter-arrival timing variance for jitter, and correlate against network-layer causes like bufferbloat, congestion, misapplied QoS, or NIC offload behavior, since the same symptom (jitter and loss) can originate from any of these distinct places. Once you have a candidate cause, use tc (traffic control) and its queue disciplines (qdiscs) both to reproduce the symptom in a controlled way and to test a remediation before touching production.
Structured elaboration
- Measuring loss without TCP's help: since UDP itself carries no sequence guarantee, loss detection depends on whatever the application includes; if the application embeds its own sequence numbers in the payload, a capture (or the application's own logs) can directly count gaps in that sequence to quantify loss precisely; without application-level sequencing, you are limited to comparing packet counts between sender and receiver captures over the same window, a cruder but still useful measure.
- Measuring jitter directly from capture timestamps: jitter is the variation in inter-arrival time between consecutive packets; comparing the actual arrival-time deltas at the receiver against the expected, constant sending interval (if the sender transmits at a known, fixed rate) directly quantifies jitter, and a capture at the receiver alone is sufficient for this specific measurement.
- Out-of-order delivery: if the application's own sequence numbers show packets arriving in a different order than they were sent, that is reordering, distinguishable from loss by whether the missing sequence number eventually shows up later, just out of order, rather than never arriving at all.
- Bufferbloat as a specific network-layer cause: an oversized buffer somewhere on the path does not drop packets, but holds them long enough to introduce significant, variable delay, which for real-time telemetry specifically manifests as exactly the jitter being reported, even without any actual packet loss at all.
- Misapplied QoS as a cause: if this UDP traffic is not being correctly classified into an appropriate priority queue (perhaps due to a missing or incorrect DSCP marking), it can be treated as best-effort and suffer disproportionate jitter/loss compared to other traffic sharing the same congested link.
- NIC offload behavior as a cause: some hardware offload features can affect the precise timing of when packets are actually handed to the network versus when the application believes it sent them; corroborating with a capture taken as close to the physical wire as feasible helps rule this out.
- Using tc and queue disciplines to reproduce the symptom in a controlled test:
tc qdisc show dev eth0shows the interface's current qdisc (often a plain FIFO); to reproduce bufferbloat-like jitter and loss deliberately on a lab or staging link,tc qdisc add dev eth0 root handle 1: netem delay 20ms 5ms loss 0.5%injects a controllable amount of delay variance and loss so you can confirm the telemetry client actually reports the same jitter/loss signature under a known, reproducible condition, before assuming the production root cause. - Using tc to test the bufferbloat remediation specifically: if an oversized, undifferentiated FIFO queue is the suspected cause, replacing it with a modern queue discipline built specifically to counter bufferbloat,
tc qdisc replace dev eth0 root fq_codel(orcakewhere available), and re-running the same traffic pattern, directly tests whether active queue management reduces the observed jitter;tc -s qdisc show dev eth0after the change reports drop and backlog counters so you can compare before and after quantitatively rather than by feel. - Using tc to test the QoS/DSCP remediation specifically: to confirm that correctly classifying this UDP flow into a priority queue actually protects it under contention, set up a
prioqdisc and a filter matching the flow's DSCP marking, for exampletc qdisc add dev eth0 root handle 1: priofollowed bytc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip tos 0xb8 0xfc flowid 1:1(0xb8 matches DSCP EF, Expedited Forwarding, in the 8-bit ToS byte), then generate contending background traffic and confirm the marked telemetry flow's jitter/loss improves relative to the unmarked case.
Worked example
Application-embedded sequence numbers show zero true gaps (nothing is actually missing), but inter-arrival timing at the receiver shows highly variable delays, sometimes several times the expected interval. Checking DSCP markings on the captured packets reveals they are arriving unmarked (best-effort), despite an expectation that this telemetry traffic should carry a priority marking. To confirm this is actually the mechanism (not just a plausible story), a controlled test is run: tc qdisc add dev eth0 root handle 1: netem delay 20ms 5ms reproduces comparable jitter on a staging link with synthetic background load, matching the reported symptom; applying tc qdisc add dev eth0 parent 1: handle 10: prio with a DSCP-matching tc filter for the telemetry flow's expected EF marking, then correctly marking the telemetry traffic at the source, is tested end to end and shown (via tc -s qdisc show counters and a repeat of the jitter measurement) to bring the flow's jitter back down to baseline even while the background load continues, confirming DSCP misclassification, not bufferbloat or NIC offload, was the actual production cause.
Trade-offs & pitfalls
Without application-level sequence numbers, distinguishing true loss from severe jitter/reordering is genuinely harder for UDP than for TCP; embedding sequence numbers at the application layer is a small design cost that pays for itself substantially in future troubleshooting capability. Bufferbloat specifically is easy to misdiagnose as loss if you only check aggregate counts rather than looking at whether missing data actually eventually arrives, just late. When testing with tc qdisc add ... netem or prio/fq_codel changes, always test on a lab/staging interface or a clearly scoped, reversible change window first, since misapplying a prio or netem qdisc on a live production interface can itself introduce the exact symptom (added delay, reordering) you are trying to diagnose; keep the original qdisc configuration on hand so it can be restored with a single tc qdisc del dev eth0 root if a test does not go as expected.
From a troubleshooting perspective, how do TCP and UDP differ in the kinds of failures you can observe in packet traces?
Sample Answer
Direct answer
From a troubleshooting perspective, TCP failures show up as visible, protocol-level SIGNALS (retransmissions, duplicate ACKs, resets, a stuck handshake) because TCP tracks state and reports on it, while UDP failures show up as SILENCE, since UDP has no built-in mechanism to report loss, reordering, or a failed delivery at all; this single difference shapes almost everything about how you'd investigate each.
Structured elaboration
- TCP gives you built-in state to read: sequence numbers, acknowledgments, retransmissions, duplicate ACKs, and the handshake itself all provide direct, in-protocol evidence of what's succeeding and what isn't, which is why so much of TCP troubleshooting is literally "read what the protocol itself is telling you" from a capture.
- UDP gives you almost nothing built in: there's no handshake to fail partway through, no sequence numbers guaranteeing order, no automatic retransmission; a lost UDP packet simply never arrives, with nothing in the protocol itself to flag that it happened, unless the APPLICATION built its own tracking on top (sequence numbers, application-level ACKs).
- What this means for troubleshooting TCP: you can often diagnose a TCP problem largely from ONE capture, since the protocol's own state (retransmissions, resets, window behavior) tells a fairly complete story on its own.
- What this means for troubleshooting UDP: you typically need EXTERNAL corroboration, application-level logging (did the receiver get message N), synthetic traffic with known content and expected arrival, or comparing send-side and receive-side captures directly, since UDP itself won't tell you anything went wrong.
- What failures look like differently for each: a TCP connection that can't be established shows a stuck handshake, visible directly; a UDP-based service that can't be reached shows... nothing, from the protocol's own perspective; you'd only know by the application reporting no response, or by comparing captures on both ends to see packets sent but never arriving.
Worked example
Investigating "the service seems unreachable": for a TCP-based service, a capture immediately shows whether the handshake is stuck (and at which step), giving a fast, direct answer. For a UDP-based service (like a DNS resolver or a real-time telemetry feed), the SAME symptom requires synchronized captures at BOTH sender and receiver (to compare what was sent against what arrived) or application-level logging, since a single capture at just one end can only show "packets were sent" or "packets arrived," never both halves of the story at once the way TCP's own acknowledgments would.
Trade-offs & pitfalls
It's easy to bring TCP-style intuitions (look for retransmissions, look at the handshake) to a UDP problem and come away empty-handed, since UDP simply doesn't generate those signals; recognize early which protocol you're dealing with and adjust your evidence-gathering approach accordingly, leaning on external corroboration and paired captures for UDP rather than expecting the protocol itself to tell you what happened.
Unlock Full Question Bank
Get access to all 41 Network Troubleshooting and Diagnostics interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.