Network Troubleshooting and Diagnostics Questions
Systematically diagnosing network problems: a layered troubleshooting methodology, diagnostic tools and commands (ping, traceroute, tcpdump, packet capture), root-cause analysis, and connectivity and user-access issues. Covers isolating faults across the stack, reading packet-level evidence, and driving from symptom to root cause. The diagnostic discipline that spans all networking layers.
From a troubleshooting perspective, how do TCP and UDP differ in the kinds of failures you can observe in packet traces?
Sample Answer
Direct answer
From a troubleshooting perspective, TCP failures show up as visible, protocol-level SIGNALS (retransmissions, duplicate ACKs, resets, a stuck handshake) because TCP tracks state and reports on it, while UDP failures show up as SILENCE, since UDP has no built-in mechanism to report loss, reordering, or a failed delivery at all; this single difference shapes almost everything about how you'd investigate each.
Structured elaboration
- TCP gives you built-in state to read: sequence numbers, acknowledgments, retransmissions, duplicate ACKs, and the handshake itself all provide direct, in-protocol evidence of what's succeeding and what isn't, which is why so much of TCP troubleshooting is literally "read what the protocol itself is telling you" from a capture.
- UDP gives you almost nothing built in: there's no handshake to fail partway through, no sequence numbers guaranteeing order, no automatic retransmission; a lost UDP packet simply never arrives, with nothing in the protocol itself to flag that it happened, unless the APPLICATION built its own tracking on top (sequence numbers, application-level ACKs).
- What this means for troubleshooting TCP: you can often diagnose a TCP problem largely from ONE capture, since the protocol's own state (retransmissions, resets, window behavior) tells a fairly complete story on its own.
- What this means for troubleshooting UDP: you typically need EXTERNAL corroboration, application-level logging (did the receiver get message N), synthetic traffic with known content and expected arrival, or comparing send-side and receive-side captures directly, since UDP itself won't tell you anything went wrong.
- What failures look like differently for each: a TCP connection that can't be established shows a stuck handshake, visible directly; a UDP-based service that can't be reached shows... nothing, from the protocol's own perspective; you'd only know by the application reporting no response, or by comparing captures on both ends to see packets sent but never arriving.
Worked example
Investigating "the service seems unreachable": for a TCP-based service, a capture immediately shows whether the handshake is stuck (and at which step), giving a fast, direct answer. For a UDP-based service (like a DNS resolver or a real-time telemetry feed), the SAME symptom requires synchronized captures at BOTH sender and receiver (to compare what was sent against what arrived) or application-level logging, since a single capture at just one end can only show "packets were sent" or "packets arrived," never both halves of the story at once the way TCP's own acknowledgments would.
Trade-offs & pitfalls
It's easy to bring TCP-style intuitions (look for retransmissions, look at the handshake) to a UDP problem and come away empty-handed, since UDP simply doesn't generate those signals; recognize early which protocol you're dealing with and adjust your evidence-gathering approach accordingly, leaning on external corroboration and paired captures for UDP rather than expecting the protocol itself to tell you what happened.
You need to process a set of packet captures (or a live traffic feed) to answer a concrete question, e.g. which flows are retransmitting the most, or what the handshake latency looks like per flow, without opening each one by hand. Describe how you'd script this (naming the library or tool you'd reach for), what fields you'd extract, and what you'd have to be careful of, like out-of-order packets or multiple capture points, for the numbers to be trustworthy.
Sample Answer
Direct answer
For a concrete question like "which flows are retransmitting the most" or "what's the per-flow handshake latency," script it with a pcap-parsing library (Python's scapy, or shelling out to tshark's field-extraction mode) rather than opening captures by hand; the key engineering concerns are correctness (handling out-of-order and duplicate packets) and scale (not loading an entire large capture into memory at once).
Structured elaboration
- Choose the extraction path:
tshark -r file.pcap -T fields -e ip.src -e ip.dst -e frame.lenstreams field values without building Python objects for every packet, and scales best for very large captures; scapy'sPcapReader(as opposed tordpcap) reads packets one at a time, which is the right choice when you need custom logic per packet in Python. - Aggregate incrementally: keep running totals in a dictionary keyed by the flow tuple (source IP, destination IP, and for TCP also ports), rather than storing every packet, so memory use stays flat regardless of capture size.
- Handle multiple capture points and out-of-order packets explicitly: if flows are captured at more than one point in the path, do not assume packet order in the file matches wall-clock order; sort by timestamp within each flow before computing anything that depends on ordering (like handshake latency, which needs to match a SYN to its SYN-ACK, not just the Nth and N+1th packet).
- Validate against a known-answer capture before trusting the script's output on real production data.
Worked example
Here is a minimal top-talkers-by-bytes script, executed against a synthetic 60-packet capture (3 flows) built with scapy for this validation:
from collections import defaultdict
from scapy.all import PcapReader, IP
def top_talkers(path, n=10):
counts = defaultdict(int)
byte_totals = defaultdict(int)
with PcapReader(path) as reader:
for pkt in reader:
if IP in pkt:
key = (pkt[IP].src, pkt[IP].dst)
counts[key] += 1
byte_totals[key] += len(pkt)
return sorted(byte_totals.items(), key=lambda kv: kv[1], reverse=True)[:n]
Run against the test capture, this correctly reported two flows: 10.1.1.10 -> 10.2.2.20 with 40 packets totaling 28,700 bytes, and 10.1.1.11 -> 10.2.2.21 with 20 packets totaling 11,700 bytes, matching the known composition of the synthetic file exactly. For handshake-latency specifically, the same pattern applies but keyed by the full 4-tuple, matching each SYN to the SYN-ACK sharing its source/destination ports and computing the timestamp delta.
Trade-offs & pitfalls
Loading an entire multi-gigabyte capture with rdpcap (which reads the whole file into memory as a list) is a common and expensive mistake; use a streaming reader instead. When captures come from multiple points, do not assume a packet appearing in one capture but not another means it was dropped; capture loss at the tap/collection point itself is a real, separate failure mode from network packet loss, and conflating the two produces wrong conclusions.
You're looking at a flow with out-of-order segments and repeated duplicate ACKs on a WAN link, and a colleague says "it's just packet loss." Explain how you would use the capture itself (sequence numbers, SACK blocks, timestamps) to tell true loss apart from simple reordering, a load-balancer or ECMP path change, and delayed ACKs from the receiver, and what additional evidence you'd gather before you're confident which one it is.
Sample Answer
Direct answer
Out-of-order segments and duplicate ACKs on their own do not prove packet loss; they prove the receiver saw a gap in the sequence space, which could be caused by real loss, by reordering (the data arrives late but intact), or by a load balancer or path change splitting a flow across inconsistent paths. Telling these apart requires looking at whether the "missing" data ever actually shows up.
Structured elaboration
- The core diagnostic question: does the segment that triggered the duplicate ACKs eventually arrive at all, just later than expected, or does it never arrive until a retransmission delivers it? If it eventually arrives on its own, this was reordering, not loss; if it only ever shows up via retransmission, real loss occurred somewhere on the path.
- Check for a load-balancer or ECMP path change mid-flow: if a flow's packets start taking a different path partway through (visible as a change in TTL, or timing characteristics, between packets of the same flow), a path with different, larger latency for a subset of packets is a classic cause of reordering that looks like loss until you check whether the data eventually arrives.
- Check for delayed ACKs specifically: a receiver that delays sending ACKs (a normal TCP optimization, to acknowledge more data per ACK) can produce ACK patterns that superficially resemble duplicate ACKs to an inexperienced reader; distinguish this by checking whether the acknowledgment number is actually repeating (a true duplicate ACK) versus simply arriving less frequently than expected (a delayed but still-progressing ACK).
- Use SACK (Selective Acknowledgment) information if present: SACK blocks in the TCP options explicitly tell the sender which out-of-order ranges the receiver HAS actually received, which directly distinguishes "received out of order" (data present, just reordered) from "not received at all" (a true gap), without needing to wait and see what happens next.
- Correlate with independent evidence: interface counters, queue-depth telemetry, or a second capture point closer to a suspected congestion point can corroborate whether real loss is occurring on the path, versus the symptom being purely an artifact of reordering.
Worked example
A capture shows segment 5000-6460 missing when segment 6460-7920 arrives, triggering three duplicate ACKs for byte 5000. If SACK information in those duplicate ACKs shows the receiver acknowledging it HAS received bytes 6460-7920 (just not the 5000-6460 range yet) and, two packets later, the 5000-6460 segment arrives WITHOUT ever being retransmitted by the sender, this proves it was reordering: the data took a slower path (perhaps a different ECMP hash bucket) and arrived late, not lost. If instead the sender explicitly retransmits 5000-6460 and that retransmission is what finally delivers it, real loss occurred.
Trade-offs & pitfalls
Your colleague's "it's just packet loss" instinct is a reasonable prior but not something to accept without checking; duplicate ACKs and out-of-order delivery are the SAME observable signature whether the cause is loss or reordering, and only checking whether the data eventually arrives (or reading SACK information directly) tells them apart. Treating reordering as loss leads you to investigate congestion or a failing link when the real fix might be addressing inconsistent ECMP hashing or multi-path routing instead.
You capture a TCP connection and see SYN, SYN-ACK, but no final ACK from the client. What are the most likely root causes, and how would you prove which side is dropping or delaying the packet?
Sample Answer
Direct answer
A handshake that reaches SYN-ACK but never gets a final ACK from the client points at one of a few specific causes, all downstream of the server's SYN-ACK actually being sent: it never reached the client, the client rejected it for a reason not visible in this capture (like a spoofed or already-closed original connection), or the client's own ACK was lost or blocked on the way back.
Structured elaboration
- Confirm the SYN-ACK actually left the server and reached the client: check whether the server-side capture shows the SYN-ACK being sent, and whether a capture closer to (or at) the client shows it arriving; if it never arrives at the client, a firewall or path issue between server and client is the cause, and the client's silence (no ACK, but also no visible reason from its side) is simply a natural consequence of never having received the SYN-ACK to respond to.
- Consider whether the original SYN was spoofed: if an attacker sent a SYN with a spoofed source address (as in a SYN-flood scenario), the server correctly sends its SYN-ACK toward that address, but the REAL owner of that spoofed address never sent the original SYN and has no reason to respond with an ACK at all; this produces exactly this pattern and is a security-relevant possibility to keep in mind alongside purely benign explanations.
- Check for a firewall specifically dropping the ACK on its way back to the server: if the SYN-ACK did reach the client and the client DID respond, but that response's ACK is lost between the client and server (perhaps due to an asymmetric path where the return leg passes through a different, more restrictive firewall), the server would see exactly this symptom despite the client behaving correctly.
- Check whether the client already closed or abandoned the connection attempt: a client-side timeout that gave up on this specific attempt just as the SYN-ACK was arriving (a race between the client's own timeout and the server's response) can produce this pattern without any packet actually being lost or blocked at all.
- Prove which of these applies with capture placement: a capture at the client, alongside the server-side one already available, is what actually distinguishes "SYN-ACK never arrived at the client" from "client received it but its ACK was lost on the way back" from "client had already given up"; without a client-side capture, several equally plausible explanations remain open.
Worked example
A capture at the client (obtained for this specific investigation) shows the SYN-ACK DID arrive, and shows the client DID send an ACK in response, promptly. But the server-side capture never shows that ACK arriving. This conclusively narrows the cause to something dropping the ACK specifically on its RETURN path from client to server, ruling out both a spoofed-source scenario (the client clearly did originate this legitimately) and a simple SYN-ACK delivery failure (it clearly arrived); investigation now focuses specifically on the return path, likely an asymmetric-routing-related firewall issue given the return-only nature of the loss.
Trade-offs & pitfalls
Without a capture at the client specifically, several genuinely different explanations (SYN-ACK never arrived, ACK lost on the way back, spoofed source, client gave up) all look identical from the server side alone, just "no ACK ever came"; resist concluding which one applies without that additional evidence. Also don't dismiss the spoofed-source possibility purely because most instances of this symptom are benign; a small volume of spoofed SYNs mixed into normal traffic can produce a few of these unexplained half-open connections that are worth distinguishing from a legitimate but lossy path.
A set of firewall or security-group rules is intermittently blocking legitimate traffic, and you don't yet know which rule or which layer (stateful firewall, ACL, NAT) is responsible. Describe a systematic, minimally invasive way to narrow this down using captures and logs, and how your approach differs between an on-prem stateful firewall and a cloud model where security groups and network ACLs behave differently.
Sample Answer
Direct answer
When you suspect a firewall or security-group rule set is intermittently blocking legitimate traffic but do not know which specific rule, a controlled bisection, splitting the rule set in half and testing each half's effect, finds the offending rule in a bounded number of steps without needing to understand every rule up front. The mechanics of that bisection differ meaningfully between an on-prem stateful firewall and a cloud model split across security groups and network ACLs.
Structured elaboration
- Reproduce the failure reliably first: confirm you can trigger the block on demand (a specific request, from a specific source, at a specific time) before starting bisection; bisecting against a failure you cannot reliably reproduce will produce misleading results.
- Split the rule set and test each half: apply the first half of the current rule set in a test/staging path (or, if testing in production is unavoidable, during a low-traffic window with a fast rollback plan) and check whether the failure still occurs; whichever half reproduces the failure contains the offending rule, and you discard the other half from consideration.
- Repeat on the implicated half: continue splitting the remaining candidate rules in half each time; this converges to a single rule in roughly log2(N) test cycles rather than testing N rules one at a time.
- Use packet captures alongside bisection, not instead of it: a capture at each side of the firewall during a test cycle can confirm which packets are actually being dropped at that stage, corroborating (or correcting) what the bisection result implies.
- Adapt the mechanics to the platform: on an on-prem stateful firewall, the rule set is usually one ordered list evaluated top to bottom with implicit session/connection tracking, so bisecting means temporarily disabling or reordering blocks of rules on the same device, and a rollback is a single configuration revert on that device; because the firewall is stateful, return traffic is handled automatically once the forward direction is allowed. In a cloud model, you are usually bisecting across at least two independently evaluated layers, security groups (stateful, return traffic implicit) and network ACLs (stateless, every direction must be explicitly allowed), often owned by different teams; bisection needs to identify not just which rule but which layer first, since a security group change and a NACL change can look identical from the traffic's perspective but require editing entirely different resources. Cloud rule changes are frequently safer to test than on-prem ones because you can often clone a security group or duplicate a NACL and attach it to a single test instance or subnet without touching the shared, in-place resource that live traffic depends on.
- Roll back safely: keep every intermediate rule-set version so you can restore the last known-good state immediately if a test step causes unexpected impact, and always have an explicit, tested rollback path, not just a plan to revert if needed.
Worked example
A rule set of 32 rules is suspected on an on-prem firewall. Applying rules 1 through 16 only: failure still occurs. Applying rules 1 through 8: failure occurs. Applying rules 1 through 4: failure does not occur. Applying rules 5 through 8: failure occurs. Applying rule 5 alone: failure occurs. Five test cycles isolated the single offending rule out of 32, versus up to 32 cycles testing them one at a time, and because it is one on-prem device, each test cycle is a single config push and revert. Contrast this with the equivalent cloud case: the same intermittent block could be caused by either the instance's security group or the subnet's NACL. Here the first bisection step is not 'which rule' but 'which layer': temporarily attach a maximally permissive NACL to a cloned test subnet while leaving the security group untouched; if the failure disappears, the NACL (a stateless, easy-to-misconfigure return-path rule) is implicated, and only then do you bisect within that specific NACL's rule list, without ever needing to touch the security group or risk the shared production subnet.
Trade-offs & pitfalls
Bisection assumes the failure is caused by exactly one rule (or a small, testable subset); if two rules interact (each individually harmless, together causing the block), naive bisection can miss it, since removing either half individually may appear to fix the symptom while the real interaction goes undetected. This risk is higher in the cloud case specifically, because a security group rule and a NACL rule can interact (one allows the connection out, the other silently drops the stateless return), so a bisection that only ever tests one layer at a time can converge on a wrong or incomplete answer; test across both layers, not just within one. State this assumption explicitly, and if bisection does not converge cleanly, consider that you may be looking at an interaction rather than a single rule. Testing rule-set changes in production carries real risk on-prem, where there is often only one shared device; the cloud model's ability to clone a security group or NACL onto a single test resource lowers that risk considerably and should be preferred over testing directly against the shared production resource whenever it is available.
Unlock Full Question Bank
Get access to all 41 Network Troubleshooting and Diagnostics interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.