Network Troubleshooting and Diagnostics Questions

Direct answer

A packet capture that's actually useful for troubleshooting a connectivity issue needs deliberate choices made before you start, not just running tcpdump with no arguments: where you capture, what you filter for, how much of each packet you keep, and what legal/privacy constraints apply, all affect whether the capture actually answers your question later.

Structured elaboration

1. Capture location: decide whether you need to capture on the host itself, at a switch SPAN/mirror port, at a network TAP, or (in cloud environments) via VPC traffic mirroring; a host-based capture sees only what that host sends/receives (and may miss packets dropped before reaching the NIC), while a SPAN or TAP sees the wire itself, which matters when you suspect a problem BETWEEN two points rather than at either endpoint.
2. Interface selection: capturing on the wrong interface (a management interface instead of the data interface, or the wrong VLAN sub-interface) is a common, entirely avoidable mistake that produces an empty or irrelevant capture.
3. Capture filters, applied at capture time: a BPF filter (host, port, protocol) applied while capturing reduces the volume you need to store and analyze, at the cost of not being able to go back and look at anything outside that filter later; decide this trade-off deliberately rather than by default.
4. Snaplen (how much of each packet to keep): capturing only headers (a small snaplen) is enough for most connectivity troubleshooting and keeps files small; capturing full payload is needed only if you specifically need to inspect application-layer content, and comes with real privacy implications.
5. Buffering and rotation: for anything beyond a quick, short capture, set file-size-based rotation (so a long-running capture doesn't fill the disk) and consider ring-buffer behavior, so an unattended capture degrades gracefully rather than crashing the host.
6. Legal and privacy considerations: capturing production traffic can capture sensitive data (credentials, personal data) depending on snaplen and what's being captured; know your organization's policy on this before capturing broadly, and prefer the smallest scope (narrowest filter, smallest snaplen) that still answers your question.

Worked example

Troubleshooting an intermittent connectivity failure between a client and a specific backend: capture at the host closest to where the symptom is reported (not several hops away), filtered to the specific host/port pair involved (host <client-ip> and port 443), with a modest snaplen (enough to see full TCP/TLS headers, not necessarily full payload), rotating to 100MB files so an extended capture window doesn't fill the disk while you wait for the intermittent failure to recur.

Trade-offs & pitfalls

Capturing everything with no filter "just in case" produces a file that's expensive to store, slow to analyze, and more likely to contain sensitive data you didn't need; conversely, filtering too narrowly before you fully understand the failure risks missing the very packet that would have explained it. Start with a reasonably scoped filter based on what you already know, and widen it only if the first capture doesn't answer the question.

Direct answer

For a concrete question like "which flows are retransmitting the most" or "what's the per-flow handshake latency," script it with a pcap-parsing library (Python's scapy, or shelling out to tshark's field-extraction mode) rather than opening captures by hand; the key engineering concerns are correctness (handling out-of-order and duplicate packets) and scale (not loading an entire large capture into memory at once).

Structured elaboration

1. Choose the extraction path: tshark -r file.pcap -T fields -e ip.src -e ip.dst -e frame.len streams field values without building Python objects for every packet, and scales best for very large captures; scapy's PcapReader (as opposed to rdpcap) reads packets one at a time, which is the right choice when you need custom logic per packet in Python.
2. Aggregate incrementally: keep running totals in a dictionary keyed by the flow tuple (source IP, destination IP, and for TCP also ports), rather than storing every packet, so memory use stays flat regardless of capture size.
3. Handle multiple capture points and out-of-order packets explicitly: if flows are captured at more than one point in the path, do not assume packet order in the file matches wall-clock order; sort by timestamp within each flow before computing anything that depends on ordering (like handshake latency, which needs to match a SYN to its SYN-ACK, not just the Nth and N+1th packet).
4. Validate against a known-answer capture before trusting the script's output on real production data.

Worked example

Here is a minimal top-talkers-by-bytes script, executed against a synthetic 60-packet capture (3 flows) built with scapy for this validation:

from collections import defaultdict
from scapy.all import PcapReader, IP

def top_talkers(path, n=10):
    counts = defaultdict(int)
    byte_totals = defaultdict(int)
    with PcapReader(path) as reader:
        for pkt in reader:
            if IP in pkt:
                key = (pkt[IP].src, pkt[IP].dst)
                counts[key] += 1
                byte_totals[key] += len(pkt)
    return sorted(byte_totals.items(), key=lambda kv: kv[1], reverse=True)[:n]

Run against the test capture, this correctly reported two flows: 10.1.1.10 -> 10.2.2.20 with 40 packets totaling 28,700 bytes, and 10.1.1.11 -> 10.2.2.21 with 20 packets totaling 11,700 bytes, matching the known composition of the synthetic file exactly. For handshake-latency specifically, the same pattern applies but keyed by the full 4-tuple, matching each SYN to the SYN-ACK sharing its source/destination ports and computing the timestamp delta.

Trade-offs & pitfalls

Loading an entire multi-gigabyte capture with rdpcap (which reads the whole file into memory as a list) is a common and expensive mistake; use a streaming reader instead. When captures come from multiple points, do not assume a packet appearing in one capture but not another means it was dropped; capture loss at the tap/collection point itself is a real, separate failure mode from network packet loss, and conflating the two produces wrong conclusions.

Direct answer

Without decrypting anything, the TLS handshake itself is unencrypted metadata (ClientHello, ServerHello, certificate exchange, and alert messages all travel in the clear before encryption begins), so a capture can distinguish a plain network problem, a middlebox performing TLS interception, and a server-side certificate or cipher mismatch just from what's visible in that initial exchange and how far it gets before failing.

Structured elaboration

1. Plain network-level failure: if the capture shows no ClientHello ever reaching the server at all (or the underlying TCP handshake itself failing), the problem is below TLS entirely, no different from any other connectivity failure; TLS never even gets a chance to fail on its own terms here.
2. Certificate or cipher-negotiation mismatch: if the ClientHello arrives and a ServerHello comes back, but is followed by a TLS Alert (visible as a distinct, unencrypted record type) rather than proceeding to Certificate and key exchange, the alert's specific code (readable in the clear) usually names the exact issue: an unsupported cipher suite, a protocol version mismatch, or (later in the handshake) a certificate validation failure.
3. Middlebox performing TLS interception: a device doing TLS inspection terminates the client's real TLS session and re-originates its own toward the actual server, presenting a DIFFERENT certificate (typically issued by an internal or corporate CA) than the server's genuine one; comparing the certificate seen in the capture against the server's actual, expected certificate (subject, issuer, serial number, all visible in the clear during the handshake) reveals this immediately, and a capture on both the client side and, if reachable, closer to the real server will show two DIFFERENT certificates in play for what the client believes is one connection.
4. SNI (Server Name Indication) issues: the ClientHello carries the hostname being requested in the clear (SNI), which servers hosting multiple certificates on one IP use to select the right certificate; if the wrong certificate comes back relative to the SNI sent, that's a server-side virtual-hosting misconfiguration, visible without decrypting anything since SNI itself is unencrypted.

Worked example

A capture shows the ClientHello leaving the client with SNI api.example.com, and a ServerHello and Certificate returning promptly, but the client immediately sends a TLS Alert (fatal, "bad certificate"). Examining the certificate in the capture shows it was issued to internal-proxy.corp.local by an internal CA, not to api.example.com by a public CA the client trusts. This is conclusive evidence of a middlebox performing TLS interception (intentional or otherwise), entirely from unencrypted handshake metadata; no payload decryption was needed to reach this conclusion.

Trade-offs & pitfalls

It's tempting to assume any TLS failure requires decrypting traffic to diagnose, but the handshake's own metadata (SNI, the negotiated cipher suite, the certificate chain, and any Alert messages) is deliberately unencrypted and answers most practical diagnostic questions on its own. The one thing this approach genuinely cannot tell you is anything about the APPLICATION data exchanged after the handshake completes successfully; if the handshake itself is healthy and the failure is deeper in the application protocol, this technique has reached its limit.

Direct answer

A CLI tool that runs DNS resolution across multiple resolvers plus parallel ICMP pings from multiple vantage points needs a concurrency model that runs independent checks in parallel (since they don't depend on each other) while handling partial failures gracefully, summarizing results in a way an operator can scan quickly rather than reading raw tool output for each check.

Structured elaboration

1. Architecture: a small set of independent "checker" functions (DNS-across-resolvers, ping-from-vantage-points, TCP-traceroute-to-port, SNMP-interface-counters, optional remote tcpdump via SSH), each returning a structured result (success/failure plus relevant detail), run concurrently via a thread pool or async event loop, since these checks are I/O-bound (waiting on network responses) rather than CPU-bound, making concurrency a clear win with limited complexity cost.
2. Concurrency model: a bounded thread pool (or asyncio with a semaphore limiting concurrent operations) prevents launching an unbounded number of simultaneous pings or SSH sessions, which could itself look like a mini denial-of-service against the target or overwhelm the operator's own machine's resources.
3. Error handling, deliberately per-check: each checker should catch its own exceptions and report a structured failure (which resolver failed, which vantage point was unreachable) rather than letting one failing check crash the whole run; partial results (4 of 5 resolvers answered, 1 timed out) are still useful and should be reported as such, not discarded.
4. Credentials management for SSH-based remote tcpdump: never hardcode credentials; use the operator's existing SSH key-based authentication and existing SSH agent/config, and make remote tcpdump strictly opt-in (given its higher risk, both operationally and in terms of what data it might capture) rather than a default behavior.
5. Summarizing for the operator: rather than dumping raw output from every tool, the tool should synthesize a short, prioritized summary of PROBABLE causes based on the pattern of results (for example: "DNS resolved consistently across all resolvers; ICMP succeeded from 3 of 4 vantage points, failing specifically from Vantage Point B; suggests a path-specific issue near Vantage Point B, not a DNS or destination-host problem").

Worked example

Running the tool against a hostname reporting no answer only from one recursive resolver and a failed ping from exactly one of four configured vantage points, both consistently failing from the SAME general network region: the tool's summary output would flag "resolver X and vantage point Y both show failures consistent with a regional network issue near [that region], rather than a problem with the destination host itself," letting an operator skip straight to investigating that specific region's path rather than manually correlating five separate tool outputs by hand.

Trade-offs & pitfalls

Running every check with unlimited concurrency, with no bound on parallel SSH sessions or pings, risks the tool itself becoming a burden on the network or the operator's machine, and in the worst case can look like abusive traffic to the very systems being diagnosed; always bound concurrency deliberately. A tool that crashes entirely on the first failed check (rather than isolating failures per-checker) is far less useful during an actual incident, when partial, "here's what we know so far" information is often exactly what's needed while other checks are still running.

Direct answer

Asymmetric routing (the forward and return paths differ) is often normal on the internet, since inbound and outbound path selection are controlled independently: your AS chooses its own outbound path, but the return path is chosen by the neighbor's routing policy, which you don't control. The first job is to confirm the asymmetry is real and correlated with the loss, not just an artifact of how you're measuring it, before treating it as the root cause.

Structured elaboration

1. Confirm the asymmetry with paired traceroutes: run traceroute A to B and B to A close together in time, and compare the AS-path (or hop sequence) each direction takes. A genuinely different path in each direction, especially crossing different upstream providers, confirms real routing asymmetry rather than a measurement artifact (ICMP rate-limiting or per-flow ECMP hashing can make a single traceroute look inconsistent even on a symmetric path).
2. Correlate loss with the asymmetric leg specifically: capture packets or use flow telemetry at both ends and confirm loss occurs specifically on the direction that takes the "worse" path (more hops, a congested transit link, or a stateful device that only sees one direction of the flow). This step matters because asymmetric routing is common and usually harmless; it only becomes the root cause when a device in the return path is stateful (a stateful firewall or NAT) and drops return traffic it never saw the outbound half of.
3. Inspect the routing decision on your side: show ip bgp <prefix> to see the attributes driving your outbound path choice (LOCAL_PREF, AS_PATH length, MED, communities). If your own policy is pushing outbound traffic onto a suboptimal path, that's the more tractable half to fix.
4. Propose a fix proportional to the risk: policy-based routing to force symmetric egress for a specific stateful device's traffic is a targeted, low-blast-radius fix; adjusting BGP attributes (LOCAL_PREF, MED, or communities sent to a peer) to influence the return path affects everyone using that path and needs more caution and peer coordination.

Worked example

Traceroute A to B goes A to ISP-X to B (3 hops); B to A goes B to ISP-Y to ISP-Z to A (4 hops, an extra transit AS). A capture at A shows outbound SYNs leaving cleanly but no SYN-ACKs arriving for a subset of flows. A capture at the return path's stateful firewall (sitting only on the ISP-Y/ISP-Z leg) shows it dropping the return traffic because it never saw the original SYN traverse it, since that packet took the ISP-X path instead. This is the textbook failure mode: the asymmetry itself isn't the bug, a stateful device sitting on only one of the two paths is.

Trade-offs & pitfalls

Do not assume asymmetric routing is itself the problem; a large fraction of internet traffic is asymmetric and works fine. The mistake is treating "the paths are different" as sufficient evidence without confirming a stateful device is actually affected. When proposing a fix, weigh a narrow, host- or device-specific policy route against a BGP-attribute change: the former is safer and reversible, the latter can shift traffic for many unrelated flows and should go through your normal peering-change process.