Network Troubleshooting and Diagnostics Questions
Systematically diagnosing network problems: a layered troubleshooting methodology, diagnostic tools and commands (ping, traceroute, tcpdump, packet capture), root-cause analysis, and connectivity and user-access issues. Covers isolating faults across the stack, reading packet-level evidence, and driving from symptom to root cause. The diagnostic discipline that spans all networking layers.
A host can reach other systems but cannot reach its default gateway. Describe how to use ARP and interface commands (for example 'ip link', 'ip addr', 'ip neigh' or 'arp -n') to determine whether the issue is a Layer 2 problem (missing ARP entry, bad link) or a Layer 3 routing problem. Explain the outputs you expect in each case.
Sample Answer
Direct answer
When a host can reach other systems but not its default gateway, the fault is almost always local: either the host never learned the gateway's MAC address (a Layer 2 / ARP problem) or the routing table itself is wrong (a Layer 3 problem). The fastest way to tell these apart is to check the ARP cache before touching routing at all.
Structured elaboration
- Check the ARP/neighbor table first:
ip neigh show <gateway-ip>(orarp -non older systems). An entry in stateREACHABLEorSTALEwith a real MAC address means Layer 2 is fine and the problem is elsewhere (routing, or the gateway itself is down). An entry in stateFAILEDorINCOMPLETE, or no entry at all, means the host sent an ARP request and got no reply: a Layer 2 problem (bad cable/port, VLAN mismatch, the gateway's interface down, or a switch not forwarding broadcast/ARP traffic between the host and gateway). - Check the interface and link state:
ip link showshould showstate UPand a carrier. A down link explains missing ARP replies trivially, and rules out anything more subtle. - Check the routing table:
ip route showand confirm the host's subnet mask and default route point at the correct gateway IP, on the correct interface. A wrong subnet mask is a classic cause: the host computes the gateway as "on-link" when it is not (or vice versa), so it never even sends the ARP request that would establish reachability. - Force a fresh ARP resolution:
ip neigh flush <gateway-ip>followed by a ping will re-trigger ARP and let you watch the outcome cleanly, useful when a stale, wrong MAC is cached (for example after the gateway's NIC was replaced).
Worked example
Say ip route show reports default via 10.0.0.1 dev eth0 and ip addr show eth0 reports 10.0.0.55/25. The /25 mask covers 10.0.0.0-10.0.0.127, so .1 is correctly on-link and the host should ARP for it directly. Now ip neigh show 10.0.0.1 returns nothing, or FAILED. Because the route calculation is fine, this narrows the problem to Layer 2: check ip link show eth0 for carrier, and if that's up, suspect the switch port (wrong VLAN, port down, or a security policy dropping ARP) rather than anything on the host's IP configuration.
Trade-offs & pitfalls
The order matters: many engineers immediately start a packet capture, which is unnecessary overhead for what a single ip neigh command already answers. The most common mistake is assuming "can't reach the gateway" is a routing problem and jumping straight to route tables, missing that an incorrect subnet mask is itself a routing misconfiguration that manifests as an ARP failure. On a switch, also confirm the gateway's own interface and VLAN membership; a host-side fix cannot repair a gateway that has silently gone down or been moved to the wrong VLAN.
A DNS record was updated as part of a failover, but many clients still resolve the old IP. Explain the end-to-end places where DNS caching can occur (OS resolver, local DNS forwarder, CDN, browser), how you would identify which cache is serving stale results, and commands to force-cache checks and clears. Include how dig or nslookup could help isolate the layer serving stale data.
Sample Answer
Direct answer
DNS caching happens at several independent layers between a client and the authoritative record, and a failover's DNS update only takes effect for a given client once EVERY cache layer between them and the authoritative server has expired its old entry, so "many clients still resolve the old IP" usually means one specific layer's cache outlived the TTL you expected.
Structured elaboration
- Operating-system resolver cache: many OSes cache DNS answers locally, sometimes for the record's stated TTL, sometimes with their own separate minimum caching behavior that can outlast the record's actual TTL.
- Local DNS forwarder or corporate resolver: many networks route through an internal recursive resolver (a corporate DNS forwarder) that caches independently of any individual client, meaning a whole office or site can be affected by one shared, stale cache entry even if every individual laptop's own cache is clean.
- CDN or edge-caching layers: if the service sits behind a CDN, the CDN's own DNS resolution or its edge nodes' cached upstream mappings can lag independently of client-side DNS entirely.
- Browser-level DNS cache: many browsers maintain their own short-lived DNS cache separate from the OS, which can occasionally outlast a very short record TTL due to a browser-specific minimum caching floor.
- How to identify which layer is serving stale data:
digdirectly against the authoritative server (dig @<authoritative-ns> <name>) confirms what the CORRECT, current answer is;dig @<specific-resolver-ip> <name>against the specific resolver a client actually uses (matching what that client would get) reveals whether that resolver's cache is the one serving stale data; checking the TTL remaining on the STALE answer (rather than just its value) tells you roughly how much longer that specific layer's cache will keep serving it, which helps set expectations rather than guessing.nslookup <name> <resolver-ip>gives the same isolation for engineers more familiar with that tool. - Force-checking and clearing at each layer, with concrete commands per layer:
- Force-check a specific resolver:
dig @<resolver-ip> <name>ornslookup <name> <resolver-ip>. - Clear the OS resolver cache on macOS:
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder. - Clear the OS resolver cache on Linux with systemd-resolved:
sudo resolvectl flush-caches(older systemd:sudo systemd-resolve --flush-caches); on systems using nscd:sudo systemctl restart nscd. - Clear the OS resolver cache on Windows:
ipconfig /flushdns. - Clear a browser's internal DNS cache: in Chrome/Edge, visit
chrome://net-internals/#dns(oredge://net-internals/#dns) and click "Clear host cache"; Firefox exposes an equivalent underabout:networking#dns. - For a corporate forwarder you don't administer directly, you typically cannot force a clear yourself; instead confirm its remaining TTL via
dig @<forwarder-ip>and escalate to whoever manages it if it's ignoring the record's stated TTL. - Confirm the record's TTL was actually set low enough BEFORE the failover (not just after), since a record kept at a long TTL right up until the failover means every cache that fetched it in the preceding window will hold the stale answer for that record's full original TTL, failover or not.
- Force-check a specific resolver:
Worked example
The DNS record was updated to point at the new IP as part of a failover. dig @<authoritative-ns> <name> correctly shows the new IP. A specific affected client resolves via a corporate DNS forwarder; dig @<forwarder-ip> <name> for the same name returns the OLD IP with 240 seconds of TTL remaining. This isolates the stale answer specifically to that corporate forwarder's cache, which fetched the old answer before the failover and, per its TTL, won't naturally refresh for another 4 minutes. On the affected client itself, running sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder (macOS) or ipconfig /flushdns (Windows) clears any locally cached copy immediately, though it cannot force the upstream corporate forwarder to drop its own entry before that entry's TTL naturally expires; other clients using a different resolver that happened to query AFTER the failover already see the new IP correctly.
Trade-offs & pitfalls
The common mistake is treating "the DNS record is updated" as equivalent to "clients see the new IP," without accounting for every caching layer between the client and the authoritative server independently holding onto the old answer for up to its own TTL. Clearing the OS or browser cache on an individual client only fixes that one client; it does nothing for a shared corporate forwarder or CDN edge node serving many other clients from the same stale entry, so identify which specific layer is actually responsible (via dig @<layer>) before assuming a local flush will resolve the wider complaint. For any service expecting to fail over, set the record's TTL LOW well BEFORE the planned failover so the actual cutover propagates as fast as the shortened TTL allows.
From a troubleshooting perspective, how do TCP and UDP differ in the kinds of failures you can observe in packet traces?
Sample Answer
Direct answer
From a troubleshooting perspective, TCP failures show up as visible, protocol-level SIGNALS (retransmissions, duplicate ACKs, resets, a stuck handshake) because TCP tracks state and reports on it, while UDP failures show up as SILENCE, since UDP has no built-in mechanism to report loss, reordering, or a failed delivery at all; this single difference shapes almost everything about how you'd investigate each.
Structured elaboration
- TCP gives you built-in state to read: sequence numbers, acknowledgments, retransmissions, duplicate ACKs, and the handshake itself all provide direct, in-protocol evidence of what's succeeding and what isn't, which is why so much of TCP troubleshooting is literally "read what the protocol itself is telling you" from a capture.
- UDP gives you almost nothing built in: there's no handshake to fail partway through, no sequence numbers guaranteeing order, no automatic retransmission; a lost UDP packet simply never arrives, with nothing in the protocol itself to flag that it happened, unless the APPLICATION built its own tracking on top (sequence numbers, application-level ACKs).
- What this means for troubleshooting TCP: you can often diagnose a TCP problem largely from ONE capture, since the protocol's own state (retransmissions, resets, window behavior) tells a fairly complete story on its own.
- What this means for troubleshooting UDP: you typically need EXTERNAL corroboration, application-level logging (did the receiver get message N), synthetic traffic with known content and expected arrival, or comparing send-side and receive-side captures directly, since UDP itself won't tell you anything went wrong.
- What failures look like differently for each: a TCP connection that can't be established shows a stuck handshake, visible directly; a UDP-based service that can't be reached shows... nothing, from the protocol's own perspective; you'd only know by the application reporting no response, or by comparing captures on both ends to see packets sent but never arriving.
Worked example
Investigating "the service seems unreachable": for a TCP-based service, a capture immediately shows whether the handshake is stuck (and at which step), giving a fast, direct answer. For a UDP-based service (like a DNS resolver or a real-time telemetry feed), the SAME symptom requires synchronized captures at BOTH sender and receiver (to compare what was sent against what arrived) or application-level logging, since a single capture at just one end can only show "packets were sent" or "packets arrived," never both halves of the story at once the way TCP's own acknowledgments would.
Trade-offs & pitfalls
It's easy to bring TCP-style intuitions (look for retransmissions, look at the handshake) to a UDP problem and come away empty-handed, since UDP simply doesn't generate those signals; recognize early which protocol you're dealing with and adjust your evidence-gathering approach accordingly, leaning on external corroboration and paired captures for UDP rather than expecting the protocol itself to tell you what happened.
Users report packet loss but interface counters on involved devices show no errors or drops. Describe advanced areas to investigate: per-queue egress drops/tail drops, microbursts leading to transient drops, QoS shaping/policing, bufferbloat and large buffers increasing latency, hardware offload masking counters, and how to gather high-resolution telemetry (ASIC counters, per-queue stats) to find the root cause.
Sample Answer
Direct answer
When packet loss is reported but interface counters on the involved devices show no errors, look above and below where standard counters measure: transient microbursts and per-queue tail drops that come and go faster than a counter's polling interval can capture, QoS shaping or policing discarding traffic by policy rather than by fault, and hardware-level buffering behavior (bufferbloat, offload features) that hides the real picture from a simple errors/drops counter.
Structured elaboration
- Understand what standard interface counters actually measure, and their blind spot: most polled counters (SNMP or show interface) sample at intervals of seconds; a microburst that fills a queue and causes a tail drop for a few milliseconds, then clears, can produce zero visible increment in a counter polled every 30 or 60 seconds, even though real packets were genuinely dropped.
- Look for per-queue, high-resolution telemetry instead: many modern switch ASICs expose per-queue drop counters (as opposed to aggregate interface-level counters) at a finer resolution; if available, these can reveal drops on a specific priority queue that never surface in the aggregate interface statistics.
- Check whether QoS shaping or policing is discarding traffic by design, not by fault: a policer enforces a committed rate and deliberately discards or remarks traffic above that rate at ingress, usually incrementing a policy-specific counter (a conform/exceed/violate counter on the policy-map) rather than the generic interface error/drop counter an engineer checks first; a shaper, by contrast, delays and queues excess traffic rather than dropping it outright, so it manifests as added latency and jitter rather than loss, unless its own buffer also overflows. A recent QoS policy change (a lowered committed rate, or a class reclassified into a stricter policer) is a common, entirely policy-driven cause of loss that will never show up as an interface error.
- Consider bufferbloat as a related but distinct pattern: an oversized buffer does not drop packets outright, but holds them long enough to inflate latency dramatically under load; this can look like loss to an application with a tight timeout (the packet was never actually dropped, but arrived too late to be useful), so distinguish true loss from excessive queuing delay using timestamps, not just counters.
- Check for hardware offload masking the real picture: some NICs and switch ASICs handle certain processing (checksums, some queueing decisions) in hardware in ways that are not reflected in the counters the OS or standard management interface exposes; a discrepancy between what the application experiences and what standard counters report can be a sign that the relevant activity is happening below where those counters look.
- Correlate timing precisely: gather the highest-resolution telemetry available (ASIC-level counters, per-queue stats, or policy-map conform/exceed/violate counters if accessible) and correlate the exact timestamps of reported application-level loss against any spike in queue depth, utilization, or policing activity at that same moment, even a spike too brief for a standard 30-second poll to register.
Worked example
An application reports occasional lost requests. Standard show interface counters on every device in the path show zero errors or drops over the reporting period. The policy-map attached to that egress interface, however, shows a nonzero and growing exceed counter under a QoS policer applied to this traffic class; a recent change lowered the committed rate for that class as part of a broader capacity reallocation. Enabling per-queue statistics on the relevant egress interface (polled every 1 second instead of every 60) corroborates this, showing brief spikes where the policed class's queue hits its maximum and experiences tail drops lasting under two seconds, precisely correlated with the timestamps of the application's reported failures; neither the standard 60-second interface counters nor a naive check of the interface's own drop counter would have surfaced this, since the drop is a deliberate policy action recorded in a QoS-specific counter.
Trade-offs & pitfalls
'The counters are clean' is often treated as proof there is no network-side loss, but standard interface counters have a real, specific blind spot for both short-duration events and policy-driven drops recorded elsewhere (in QoS policy-map counters, not the interface's own error/drop counters); before concluding the network is innocent, confirm you have looked at the highest time-resolution telemetry actually available on that hardware, and at any QoS policy applied to the affected traffic class, not just the generic interface counters. Distinguishing true drops (tail drop, policing) from bufferbloat-induced delay matters because the fixes are different: one calls for capacity, queue-management, or policy-rate changes, the other for buffer-sizing and queue-discipline tuning.
Explain the differences between ping, traceroute, and mtr. For each tool, describe what types of symptoms they help identify, what their outputs mean (ICMP TTL expiry vs ICMP unreachable vs UDP/TCP-based traceroute), and how you would use their results to progress your troubleshooting.
Sample Answer
Direct answer
Ping, traceroute, and mtr answer three different questions: ping tells you whether a destination is reachable and roughly how long a round trip takes; traceroute tells you the path packets take and where along that path something stops responding; mtr combines both, continuously, so you can see loss and latency per hop over time rather than a single snapshot. Reading their outputs correctly also means knowing which ICMP message each tool relies on and how the underlying probe type (ICMP, UDP, or TCP) changes what success looks like.
Structured elaboration
- Ping: sends ICMP echo requests and waits for ICMP echo replies; a successful reply confirms basic reachability and round-trip time; ping tells you nothing about where along the path a problem is, only whether the whole round trip succeeded.
- Traceroute: sends probes with increasing TTL (starting at 1) so each successive router along the path replies with an ICMP time-exceeded message as its TTL hits zero, revealing the path hop by hop. Classic Unix/Linux traceroute does this with UDP probes to a high, deliberately unlikely-to-be-open destination port; when a probe finally reaches the real destination, that host replies with ICMP destination-unreachable, port-unreachable, which is the normal, expected signal that tells traceroute it has reached the end of the path, not an error. Windows' tracert instead uses ICMP echo probes throughout, so the final hop returns a normal ICMP echo reply rather than a port-unreachable message. A TCP-based traceroute variant (tcptraceroute, or traceroute/mtr run in TCP mode) sends TCP SYN segments with increasing TTL instead; this is useful specifically when UDP or ICMP probes are filtered somewhere in the path (a common firewall policy) but the actual TCP port you care about is open, since a SYN-based probe follows the exact path and protocol handling that real application traffic would experience.
- mtr: repeatedly sends traceroute-style probes to every hop continuously and aggregates loss percentage and latency statistics per hop over the run, rather than a single traceroute's one-shot snapshot; this makes it far better at catching intermittent loss at a specific hop, which a single traceroute would likely miss entirely by chance. mtr can typically be run in ICMP, UDP, or TCP probe mode, inheriting the same completion-signal differences described above depending on which mode is selected.
- How to progress your troubleshooting using their results: start with ping to confirm there is a problem at all and get a rough sense of loss/latency; if there is a problem, use mtr (not just one traceroute) to localize which hop it correlates with, since a single traceroute run can easily miss an intermittent issue or be misled by ECMP; and if UDP- or ICMP-based probes show the path is blocked somewhere but you suspect that is only true for those protocols, rerun in TCP mode against the actual port in question to see whether the real application traffic's path differs.
Worked example
A user reports intermittent slowness to a remote service. A single ping shows occasional but not consistent high latency. A single UDP-based traceroute shows all hops responding normally with a final ICMP port-unreachable from the destination, confirming it reached the end of the path successfully; this happened to sample during a good moment, so it does not show the intermittent issue. Running mtr for several minutes reveals hop 6 specifically showing 15% loss and elevated latency, while every other hop shows 0% loss consistently, precisely the kind of transient, hop-specific pattern a one-shot traceroute is likely to miss by chance but that mtr's continuous sampling reliably catches. Separately, a different host appears completely unreachable by traceroute (every hop past a firewall shows only asterisks), while the actual web service on that host works fine in a browser; switching to a TCP-mode traceroute on port 443 succeeds cleanly, confirming that ICMP and UDP probes are being filtered by policy while the real TCP path is healthy.
Trade-offs & pitfalls
Treating a single clean traceroute as proof there is no path problem is a common mistake when the issue is intermittent; mtr's continuous sampling exists specifically to catch what a one-shot traceroute would miss. Also remember that ICMP responses (used by ping and, often, traceroute/mtr) can be deprioritized or rate-limited by routers relative to real data traffic, so a hop showing loss in these tools does not always mean real user traffic experiences the same loss; corroborate with a protocol-appropriate test (like iperf, curl, or a TCP-mode trace against the real port) when precision matters. A related and easy-to-miss mistake is reading a fully starred-out (blocked-looking) UDP or ICMP traceroute as proof the destination itself is unreachable, when it may only mean those specific probe types are filtered; a TCP-based trace against the actual service port is often what resolves that ambiguity.
Unlock Full Question Bank
Get access to all 26 Network Troubleshooting and Diagnostics interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.