Network Troubleshooting and Tools Questions

Hands on skills for diagnosing and resolving network problems using standard command line and packet analysis tools. Topics include systematic troubleshooting workflows (start with reachability tests and escalate to captures), use of ping and traceroute to verify connectivity and routing paths, netstat and ss to inspect sockets and listening ports, arp and interface commands to check layer two mappings and interface state, router and switch show commands to view routing tables and interface status, and DNS troubleshooting using nslookup and dig. Deep packet capture and analysis with tcpdump and Wireshark is covered, including capture filters, interpreting packet headers and flows, identifying retransmissions, latency sources, and protocol errors. Emphasis is on interpreting tool output, correlating results across layers, and choosing the right tool at each step of an investigation.

HardSystem Design

68 practiced

Design an automated health-check script or pipeline (languages/tools optional) that runs synthetic network tests (ping, traceroute, dig, iperf3) from multiple locations to a set of application endpoints, aggregates the results, detects anomalies (loss >1%, RTT jump >100ms, DNS resolution failures), and alerts. Describe the data model, storage, alerting thresholds, and how you would avoid alert storms during transient spikes.

Sample Answer

**Clarify requirements / constraints**- Run synthetic network tests (ping, traceroute, dig, iperf3) from multiple geographic locations to N application endpoints, store results, detect anomalies (loss >1%, RTT jump >100ms, DNS failures), alert reliably without storming.

**High-level architecture**- Agents (Go/Python) deployed in multiple PoPs or cloud regions run scheduled tests; push events to Kafka.- Stream processor (Flink/Beam) normalizes, computes short-window aggregates and writes to TSDB (VictoriaMetrics) and object store (Parquet in S3) for raw traces.- Alerting: Prometheus Alertmanager or custom rule engine subscribing to aggregated metrics; Slack/pager/email notifications via escalation channels.- Dashboarding: Grafana with trace drill-down (stored traceroutes/iperf logs).

**Data model (per test event)**- test_id, timestamp, source_location, target_endpoint (URL/IP), test_type (ping/traceroute/dig/iperf3)- metrics: rtt_ms, packet_loss_pct, jitter_ms, throughput_mbps, dns_rcode, dns_ip_list- traceroute_hops: [ {hop, ip, rtt_ms} ]- raw_output_url (S3)- tags: env, app_team

**Storage & retention**- TSDB (VictoriaMetrics/Prometheus remote write) for 1s–1m aggregates, retention 30d for high-res.- Parquet S3 for raw outputs and traces, retention 90–365d.- Index on source_location, target_endpoint, test_type, timestamp.

**Anomaly detection & thresholds**- Primary rules: - packet_loss_pct > 1% (instant) -> candidate anomaly - rtt_ms increase > 100ms compared to baseline (median of last 1h) -> candidate - DNS: non-zero rcode OR no A records -> immediate failure- Use two-stage detection: 1. Statistical check: compare current value to baseline (median + 3*IQR) and absolute thresholds above. 2. Persistence check: condition must hold for K samples or T seconds (e.g., 3 consecutive 1-min samples or sustained 5 minutes) before firing.

**Avoiding alert storms**- Debounce / aggregation: - Require sustained window (see persistence check). - Group alerts by root-cause dimensions (target_endpoint, region, AS) to avoid per-agent duplicates. - Use deduplication keys in Alertmanager: alertname + target_endpoint + root_cause_hash.- Rate limits & escalation: - Per-target rate limit (e.g., max 1 alert / 10 min). - Severity tiers: transient (warning) -> sustained (critical) -> outage (page).- Correlation: - If multiple sources show simultaneous loss to same target, escalate faster. - Use traceroute diffs to detect same failing hop and suppress per-source noise.- Auto-suppression windows: - Short maintenance windows and known rolling deploy windows read from calendar API to suppress expected noisy periods.

**Operational details**- Tests schedule: ping/dig every 30s, traceroute every 5min, iperf3 every 15min (sampled).- Health of agents monitored; missing agents treated separately.- Playbooks linked in alerts with remediation steps and traceroute/iperf raw output links.- Regular baseline re-calculation with rolling 7-day windows to account for diurnal patterns.

This design balances sensitivity (detect real degradation) with resilience to transients (debounce, grouping, correlation), gives teams actionable context (hops, throughput, DNS), and scales via Kafka + stream processing + TSDB.

MediumTechnical

94 practiced

A Linux server's 'ip neigh' shows multiple entries for the gateway IP in different VLANs and some entries are 'FAILED/INCOMPLETE'. Describe the steps and tcpdump commands you would use to determine whether ARP requests are being sent and replied to, whether there is a proxy-ARP or duplicate IP issue, and how to remediate stale ARP entries.

Sample Answer

**Situation & goal**I need to verify whether ARP requests for the gateway are sent and answered, detect proxy‑ARP or duplicate IPs, and remove stale ARP entries.

**Step 1 — Observe ip neigh**- Run: ip neigh show dev <iface> to see which VLANs and states (REACHABLE, FAILED, INCOMPLETE).

**Step 2 — Capture ARP traffic on the server**- Capture ARP on the server interface:

bash

sudo tcpdump -n -i <iface> arp -vv

- Filter for gateway IP:

bash

sudo tcpdump -n -i <iface> arp and host <gateway_ip>

Look for: ARP requests from the server (who-has <gateway_ip> tell <server_ip>) and replies (is-at <gateway_mac>).

**Step 3 — Capture on upstream/vlan trunk or switch mirror**- If server sends requests but no reply, mirror traffic on upstream switch or run tcpdump on gateway/router:

bash

sudo tcpdump -n -i <upstream_iface> arp and host <gateway_ip>

Compare MAC addresses in replies.

**Step 4 — Detect proxy‑ARP or duplicate IP**- If replies appear from different MACs on multiple VLANs or switch ports, suspect proxy‑ARP or duplicate IP.- Use: arp-scan or nmap ARP to find duplicates:

bash

sudo arp-scan --interface=<iface> --localnet

- On routers, check proxy-arp settings and ARP tables for that IP.

**Step 5 — Remediate stale/failed entries**- Clear specific entry:

bash

sudo ip neigh flush to <gateway_ip> dev <iface>

- Or clear all:

bash

sudo ip neigh flush all

- If duplicates: shutdown/conflict host or correct config; disable proxy-arp where inappropriate; set static ARP/ND for critical gateways.

**Notes & follow-up**- If INCOMPLETE persists, verify L2 connectivity, VLAN membership, MTU, and switch port security. Document changes and monitor for recurrence.

HardTechnical

127 practiced

You need to locate where excessive latency is introduced for a TCP flow. Describe how you'd use tcpdump with timestamps, Wireshark's TCP stream analysis (time delta columns), and application logs to attribute delay to the network (queueing/bufferbloat), OS scheduling, or application-level processing. Provide concrete steps and filters for capturing the minimal data necessary to prove the source of delay.

Sample Answer

**Approach (summary)**- Capture minimal TCP packets with precise timestamps on both endpoints, analyze inter-packet timing in Wireshark’s “Time Delta” columns, and correlate with application logs (request/response times) to attribute delay to network queueing, OS scheduling, or application work.

**Step 1 — tcpdump captures (minimal, timestamped)**- On client and server capture only the flow (IP/port) with full timestamps and sequence/ack info:

bash

sudo tcpdump -i eth0 -s 0 -w client.pcap -tttt -n 'tcp and host 10.0.0.1 and host 10.0.0.2 and port 5432'
sudo tcpdump -i eth0 -s 0 -w server.pcap -tttt -n 'tcp and host 10.0.0.1 and host 10.0.0.2 and port 5432'

- If disk space tight, capture headers only:

bash

sudo tcpdump -i eth0 -s 96 -w client.pcap ...

**Step 2 — Wireshark analysis**- Open both pcaps (or merge with mergecap). Enable columns: - Time (absolute) - Time Delta (delta from previous packet in this view) - Time Delta from Previous Displayed Packet (shifted) - TCP Seq/Ack/TCP TimeStamp option (if present)- Filter to the stream:

text

tcp.stream eq 3

- Look for patterns: - Large one-way gap seen only on client-side capture between data send and ACK = network queuing/drop (bufferbloat or shaping). - Symmetric gaps in both captures indicate network delay (propagation/queue). - Server sees packet quickly but responds much later -> application processing or blocked thread. - Packet seen late at NIC (pcap timestamp jumps) preceded by syscall timestamp in app logs -> OS scheduling.

**Step 3 — Correlate with application and OS logs**- Instrument app to log high-resolution timestamps on request receive, processing start, send/write completion. Example log lines: - [ts] RX request id=X - [ts] START processing id=X - [ts] WRITE complete id=X- On server, collect dmesg/latencytop or pidstat to detect scheduling preemption or long syscalls.- Correlate pcap timestamps: if packet arrives at NIC (server.pcap) at t0, app START at t0 + small delta => network caused delay earlier. If app START much later than NIC arrival => OS scheduling or kernel backlog (e.g., high softirq or CPU starvation).

**Diagnosis heuristics**- Network queueing / bufferbloat: - Client send bursts; server sees those bursts delayed by large gaps; RTT increases progressively; retransmits/ECN/CWND growth patterns.- OS scheduling: - NIC timestamp shows packet arrival, but application-level START delayed, system-wide CPU busy, high run-queue, long context-switch gaps.- Application processing: - Server ACKs/response are generated only after application logs show long processing interval; pcap shows immediate ACK after write syscall.

**Minimal filters to prove source**- Capture SYN, SYN-ACK, FIN to verify handshake times:

bash

tcpdump -n -i eth0 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 and host A and host B'

- Capture only data and ACKs:

bash

tcpdump -n -i eth0 'tcp and host A and host B and (tcp[13] & 16 != 0 or tcp[13] & 16 == 0)'

(Use stream filter in Wireshark after capture.)

**What to include in your final report**- Timeline table: packet timestamps (client/server), app log events, observed deltas and probable cause.- Evidence snippets: pcap screenshots of big deltas, app log excerpt showing exact timestamps, CPU/memory metrics if OS scheduling suspected.

This workflow gives provable attribution by comparing NIC-level packet times with application-level timestamps and observing asymmetry between endpoint captures.

HardTechnical

98 practiced

Write a Python script (using pyshark or scapy) that reads a pcap file and outputs the top 10 source-destination IP pairs by packet count and total bytes. Include memory considerations for large pcaps and suggest how you would adapt the approach for streaming or very large files (e.g., tshark invocation, chunked reads). Provide a concise, working code sketch and explain key parts.

Sample Answer

**Approach (brief)** Use Scapy to iterate packets, aggregate counts and bytes per (src,dst) tuple in a dict. For large pcaps prefer streaming via tshark or Scapy's RawPcapReader in chunks to avoid loading all packets into memory.

**Code sketch (Scapy, memory-friendly)**

python

from scapy.all import RawPcapReader, Ether, IP
from collections import defaultdict, Counter

def top_pairs(pcap_path, top_n=10):
    stats = defaultdict(lambda: [0,0])  # (count, bytes)
    for pkt_data, meta in RawPcapReader(pcap_path):
        try:
            pkt = Ether(pkt_data)
            ip = pkt.getlayer(IP)
            if not ip: continue
            key = (ip.src, ip.dst)
            stats[key][0] += 1
            stats[key][1] += len(pkt_data)
        except Exception:
            continue
    # sort by count and bytes
    by_count = Counter({k:v[0] for k,v in stats.items()}).most_common(top_n)
    by_bytes = Counter({k:v[1] for k,v in stats.items()}).most_common(top_n)
    return by_count, by_bytes

if __name__ == "__main__":
    c,b = top_pairs("capture.pcap")
    print("Top by packets:", c)
    print("Top by bytes:", b)

**Key parts explained**- RawPcapReader yields raw packets one at a time; avoids building full list.- stats dict stores [count, bytes] per (src,dst).- We catch exceptions to skip malformed frames.

**Memory & scaling**- For very large pcaps: use tshark to filter/stream fields: tshark -r capture.pcap -T fields -e ip.src -e ip.dst -e frame.len and pipe into a streaming Python aggregator (line-by-line). That keeps minimal memory and allows distributed/map-reduce counting by sharding by hash of (src,dst).

**Alternatives / trade-offs**- Pyshark is higher-level but heavier (more memory). Use tshark+parsing or raw readers for performance. Consider an external DB (Redis) for counts if state won't fit in RAM.

HardTechnical

68 practiced

A load-balanced HTTPS service is failing TLS handshakes with clients intermittently. Packet captures show repeated TCP connections that reset before any TLS records; no TLS Alerts are seen. Describe a thorough approach to determine whether the failure is due to TCP-level issues (RSTs, retransmissions), a middlebox performing TLS inspection, misconfigured backend, or an application-level SSL library crash. Include capture placement and logs to check.

Sample Answer

**Clarify scope & reproduce**- Confirm frequency, affected client IPs, times, and topology (client → LB → backend).- Try to reproduce from test clients and run simultaneous captures.

**Capture placement (must capture at all hops)**- On client (tcpdump -s0 -w client.pcap host LB): proves client saw SYN/ACK or RST.- On LB frontend (capture on ingress interface): shows what LB receives from clients.- On LB backend/egress (capture toward servers): shows what LB forwards.- On backend server (tcpdump on server interface): shows what server receives.- If possible, mirror a span on the path through any firewall/IDS/middlebox.

**What to inspect in pcaps**- TCP flags sequence: SYN → SYN/ACK → ACK? Where RST occurs and who sends it (client, LB, server, middlebox).- Retransmissions and RTT: use tshark tcp.analysis.retransmission and tcp.time_delta.- TTL/IP ID differences to infer packet origin (middlebox rewriting).- TCP resets origin: check IP, port, sequence numbers; RST without prior SYN/ACK implies closed port/iptables REJECT vs RST.- Look for truncated captures where no TLS records are present (handshake never started).- Compare ClientHello between client→LB and LB→server to detect TLS inspection rewriting SNI/ciphers.

**Logs & system checks**- LB logs (HAProxy/nginx/AWS ALB): connection accept/close, health check failures, backend selection, backend state transitions, and LB kernel conntrack table.- Server logs: application, web server, and system logs (journalctl, dmesg) for crashes, OOM, or SSL library stack traces.- SSL library logs: enable OpenSSL/BoringSSL debug or set SSLKEYLOGFILE to correlate session keys with captures; enable verbose SSL logging in app server.- Kernel and firewall: iptables/nftables rules, conntrack limits (conntrack -L), tcp_tw_reuse, ephemeral port exhaustion (ss -s), sysctl net.* for drops.- Middlebox/IDS logs: TLS proxy logs, certificate inspection events, policy blocks.

**Hypothesis tests**- TCP-level issue: captures show RSTs/retrans before TLS; consistent origin IP equals router/firewall; correlate with conntrack exhaustion or iptables REJECT rules.- Middlebox TLS inspection: client→LB ClientHello differs from LB→server; LB or middlebox presents its own cert to client; check cert chain and SNI changes.- Misconfigured backend: LB forwards to closed port (server never answered SYN) or health checks failing; server capture shows no listening process on port.- SSL library crash: server accepted connection (SYN/SYN-ACK/ACK) then connection closed abruptly by server process with no TLS; check server stderr, core dumps, strace of worker, and journalctl for segfaults or OpenSSL errors.

**Commands to run**- tcpdump -i any -s0 -w /tmp/cap.pcap 'host A and port 443'- tshark -r cap.pcap -Y 'tcp.flags.reset==1' -T fields -e ip.src -e tcp.srcport -e frame.time- ss -tnp state all | grep :443- sudo journalctl -u myapp -o short-iso --since "1 hour ago"- sudo cat /proc/net/netfilter/nf_conntrack || conntrack -L

**Next actions**- Correlate timestamps across captures and logs to attribute RST origin.- If middlebox suspected, test end-to-end bypassing it or compare fingerprints of ClientHello.- If server crash suspected, reproduce with verbose SSL logging and capture core/strace.- Implement temporary increased logging and mirrored captures until root cause proven.

Unlock Full Question Bank

Get access to hundreds of Network Troubleshooting and Tools interview questions and detailed answers.

Join thousands of developers preparing for their dream job.