Network Troubleshooting and Tools Questions

Hands on skills for diagnosing and resolving network problems using standard command line and packet analysis tools. Topics include systematic troubleshooting workflows (start with reachability tests and escalate to captures), use of ping and traceroute to verify connectivity and routing paths, netstat and ss to inspect sockets and listening ports, arp and interface commands to check layer two mappings and interface state, router and switch show commands to view routing tables and interface status, and DNS troubleshooting using nslookup and dig. Deep packet capture and analysis with tcpdump and Wireshark is covered, including capture filters, interpreting packet headers and flows, identifying retransmissions, latency sources, and protocol errors. Emphasis is on interpreting tool output, correlating results across layers, and choosing the right tool at each step of an investigation.

HardTechnical

66 practiced

You detect an intermittent blackhole where SYN packets leave your host but no SYN-ACK returns. Outline a multi-layer diagnostic plan that starts at layer 2 and escalates up to layer 7, naming the specific commands and checks you'd perform at each layer (e.g., 'ethtool', 'ip link', 'ip route', 'traceroute', 'tcpdump', application logs). Emphasize correlating evidence across layers.

Sample Answer

Situation: Intermittent blackhole where SYNs leave host but no SYN-ACKs return. I’d perform an ordered, evidence-driven multilayer diagnosis (L2→L7) and correlate timestamps/captures across layers.

Layer 2 — Link/PHY- Commands: ethtool eth0; ip link show dev eth0; dmesg | grep -i eth0- Checks: link up/down, negotiated speed/duplex, driver errors, Rx/Tx drops, physical flaps, NIC firmware or SFP errors.- Evidence to gather: ethtool statistics, kernel logs, timestamps of flaps.

Layer 3 — Network/IP- Commands: ip addr show; ip route show; ip neigh show; ip -s link; ip -s route- Checks: correct source IP, route table, ARP resolution, asymmetric routing, route changes, MTU mismatches.- Evidence: arp table entries, route changes in syslog, MTU differences.

Layer 4 — Transport/TCP- Commands: tcpdump -i eth0 -s 0 -w syn-only.pcap 'tcp[tcpflags] & (tcp-syn) != 0'; ss -tn sport :80; iptables -L -v -n; conntrack -L | grep <ip>- Checks: confirm SYN egress, no SYN-ACK ingress, retransmits, RSTs, firewall/NAT rules, connection tracking drops, TCP offload anomalies (disable TSO/GSO via ethtool -K).- Evidence: pcap showing SYNs with no replies, kernel conntrack logs, firewall counters.

Layer 5 — Session/Path- Commands: traceroute -T -p <port> <dst> (TCP), mtr --tcp; tcptraceroute- Checks: where packets stop along path, intermittent hops, ICMP unreachable messages, rate-limited devices.- Evidence: correlated traceroute hop where responses cease; router logs if accessible.

Layer 6 — Presentation/Security (TLS, proxies)- Commands: openssl s_client -connect host:port -msg; curl -v --connect-timeout 5- Checks: TLS handshake failures, proxy/edge devices dropping packets, SNI/misconfiguration.- Evidence: client-side handshake logs, proxy/access logs with timestamps.

Layer 7 — Application- Commands: tail -F app.log; strace -f -e network app; curl -vv; check load balancer logs (ELB/NGINX/Haproxy)- Checks: app listening, accept queue drops, file-descriptor exhaustion, rate-limiting, per-process firewall.- Evidence: accept errors, backlog drops, LB target health checks.

Correlation & Workflow- Centralize: collect pcaps, syslogs, kernel logs, router/LB logs with synchronized timestamps (use NTP).- Correlate: match a SYN seen in tcpdump to kernel/conntrack entries, firewall counters, and upstream router logs to locate where SYN-ACKs vanish.- Reproduce/Test: run controlled tests (hping3 -S) with increased verbosity and simultaneous tcpdump on host and upstream point-of-presence.- Short-term mitigations: disable offloads, adjust MTU, add explicit route, temporarily bypass suspect device (if feasible).- Escalation: if path stops at provider/router, provide pcap + traceroute + timestamps to network provider or hardware vendor for deeper LUT/hardware trace.

Key principles: gather immutable evidence (pcaps, counters), timestamp alignment, minimal-change testing, and escalate with precise artifacts.

MediumTechnical

74 practiced

A service migration introduced an intermittent service failure that only appears when a specific firewall rule set is applied. Describe how you would perform a controlled binary search of firewall rule sets, use packet captures to identify which rule causes the failure, and how to implement a safe rollback or exception while preserving security posture.

Sample Answer

Situation: After migrating a service, it intermittently fails only when a particular firewall rule set is active. As an SRE I need to find the offending rule quickly, limit user impact, and restore functionality without weakening security.

Approach — controlled binary search of rule sets1. Snapshot current state: export active rules (IaC or firewall API) and record timestamps and traffic examples that reproduce failure.2. Split ruleset into two halves (A and B). Disable B (or enable only A) in a controlled maintenance window or on a test segment that mirrors production.3. Test service. If failure persists with A, offending rule is in A; otherwise it’s in B. Repeat: split the failing half into two, continue until you identify a single rule. This reduces checks from N to O(log N).

Automation example (pseudo-steps)- Use firewall API/Ansible to toggle rule groups atomically and rollback on error.- Maintain a runbook that logs each step and test outcome.

Using packet captures to identify the rule- Capture on both client and server sides with synchronized clocks (NTP) and include firewall interface(s).- Use targeted filters: tcp and port for service, host IPs, and time window:

bash

sudo tcpdump -i eth0 host 10.0.1.5 and host 10.0.2.3 and tcp -w capture.pcap

- Look for: - TCP handshake failures (SYN sent, no SYN/ACK) - RST or ICMP unreachable messages (type/code reveal blocking/blackholing) - MSS, TTL, fragmentation, or TCP options stripped by firewall - Asymmetric routing: packet seen on one interface but not the other- Correlate capture timestamps with firewall logs (dropped packets, rule IDs, hit counts). If firewall logs include rule IDs, match dropped packet entries to observed packets.- If encrypted traffic (TLS), focus on transport-level flags and sizes; inspect initial handshake packets and IP/port fields.

Implement safe rollback or exception preserving security1. Short-term emergency: apply a narrow exception rather than global disable - Create an allow rule scoped by source/destination IP, port, protocol, and time window. - Log and monitor the exception (audit trail).2. Long-term fix: - Patch or adjust offending rule (e.g., correct direction, order, or match criteria). - Validate in staging and a canary subset of production traffic.3. Automation and controls: - Use IaC (Terraform/Ansible) so rollbacks are versioned and reproducible. - Require change approvals and post-change automated tests (synthetic transactions).4. Monitoring & rollback plan: - After change, increase monitoring/alerts around SLOs and firewall metrics. - If errors rise above threshold, automatic rollback to previous IaC commit or disable exception.5. Post-incident: - Root-cause analysis, update runbooks, add unit tests for firewall rules (simulated traffic), and add better logging (rule hit counts, correlation ids).

Why this works- Binary search minimizes time-to-identify with minimal disruption.- Combining packet captures and firewall logs provides definitive evidence of which rule and why (dropped, malformed, asymmetry).- Scoped exceptions and IaC rollback preserve security posture while restoring availability and ensure future reproducibility and auditability.

HardTechnical

83 practiced

Explain how to use eBPF-based tools (bcc/eBPF or tools like 'tcplife', 'xdp', 'bpftool') to perform low-overhead network troubleshooting on Linux hosts. Give examples of use cases where eBPF provides advantages over tcpdump and outline safety considerations when deploying eBPF probes in production.

Sample Answer

Start with the goal: use eBPF to observe kernel networking paths with very low overhead and targeted instrumentation. eBPF tools (bcc, libbpf-based tools, bpftool, XDP) let you attach small programs to hooks (kprobes, tracepoints, XDP, TC, socket filters) to capture metrics, aggregated histograms, or selectively sample packets without copying full packets to userspace.

Typical workflow:- Prototype with bcc scripts (faster dev): tcplife, trace_tcp_conn, biolatency-style scripts.- Production use: compile safe, audited eBPF programs with libbpf CO-RE or use prebuilt tools (bpftool, xdpdump).- Inspect/attach with bpftool and verify maps, programs, and pinning.

Example commands:

bash

# Show live TCP connection lifetimes with bcc's tcplife
sudo /usr/share/bcc/tools/tcplife -p 1234

# Attach XDP to NIC for early drop/blackhole or measurements
sudo ip link set dev eth0 xdp obj my_xdp_prog.o sec xdp_pass

# Use bpftool to list loaded programs and maps
sudo bpftool prog show
sudo bpftool map show

Use cases where eBPF beats tcpdump:- High-cardinality aggregated metrics (per-socket latency histograms) without copying packets: eBPF can aggregate in-kernel then read compact maps.- Very high throughput NICs where tcpdump would drop packets or cause CPU overload: XDP can drop or sample at driver level.- Correlating kernel events (sock connect, accept, retransmit) with minimal context switches using tracepoints/kprobes.- Runtime safety: filter at kernel entry to capture only the subset (e.g., specific PID, port), avoiding user-space parsing of all traffic.

Safety and production considerations:- Limit complexity: keep programs bounded (loops must be unrolled or use bounded iteration) to ensure verifier acceptance and predictable CPU.- Use CO-RE/clang to avoid kernel-dependent behavior; test across kernel versions.- Prefer read-only probes and maps for observability; avoid modifying live packet paths unless necessary.- Resource caps: limit pinned map sizes and attach points. Monitor CPU overhead and BPF program run time (bcc tools like biolatency style can add overhead).- Use staging: test in canary hosts or namespaces, use perf/cgroups to scope probes to specific containers.- Security: restrict who can load eBPF (CAP_BPF/CAP_SYS_ADMIN), audit code, sign eBPF programs if kernel supports it.- Fallback: have a plan to quickly detach programs (ip link set ... xdp off; bpftool prog detach) and automated health checks.

Key takeaways: eBPF provides precise, low-overhead, in-kernel observability enabling high-throughput measurement and correlation that tcpdump cannot do efficiently. Deploy carefully: restrict scope, resource-use, and verify across kernels to keep production safe.

MediumTechnical

71 practiced

Explain how to interpret and use 'ethtool -S', 'ip -s link', and NIC vendor counters to diagnose transmit/receive drops, ring buffer overflow, and checksum offload behavior. Provide an example sequence of checks when you see packets dropped according to application logs but interface RX/TX counters show no drop.

Sample Answer

Start by understanding what each tool reports:- ethtool -S <iface>: shows driver/NIC-specific hardware counters (rings, drops, errors, checksum offload stats). These are the most granular counters for ring overflows, queue drops, and offload processing.- ip -s link show <iface>: kernel-level per-interface TX/RX packet/byte/errors/dropped counters. Good quick view but may miss NIC-level drops that occur before kernel accounting.- Vendor/NIC counters (via ethtool -S or vendor tools like ixgbe-cli): expose per-ring (RX/TX) descriptor allocation drops, packet truncation, LRO/TSO/CSUM offload stats.

Example diagnosis sequence when application reports drops but ip shows none:1. Confirm app-level drops and timestamps. Correlate to host and NIC timeline.2. Run ip -s link show to verify kernel RX/TX and queuing drops — if zero, proceed to NIC-level.3. Run ethtool -S <iface> and inspect counters: rx_missed, rx_no_buf, rx_over_errors, rx_discards, rx_queue_*_drops, tx_ring_*_full. Per-ring drops or rx_no_buf indicate ring/descriptors exhausted or skbuff allocation failures.4. Check NIC offload counters: rx_csum_none/err, tso_lro_* to see if checksum offload or LRO/TSO is discarding or coalescing incorrectly.5. Inspect dmesg and driver logs for memory allocation failures (skbuff clones) or XDP/eBPF/iptables that might drop pre-stack.6. If per-ring drops present, check ring sizes (ethtool -g) and increase with ethtool -G if appropriate; verify CPU affinity and rx/tx queue distribution with irqbalance or irqaffinity to avoid one busy CPU.7. If offload issues suspected, disable relevant offloads temporarily (ethtool -K <iface> tx off rx off gro off gso off) and retest—if drops disappear, narrow to offload bug or driver/NIC mismatch.8. Use tcpdump or pktgen on host to reproduce and capture both on-host and on-path to confirm where packets vanish.

Key reasoning: ip counters report kernel-level bookkeeping; ethtool/vendor counters reveal hardware/driver-level drops and offload behavior. Follow kernel → NIC → offload → CPU/IRQ → application order to localize the fault.

MediumTechnical

80 practiced

Provide an example of a BPF/tcpdump capture filter that captures only TCP flows with retransmissions and show how to use tcpdump or tshark to detect and output retransmission packets. Explain the limitations of capture filters versus display filters for this use case and why you might prefer one over the other in production captures.

Unlock Full Question Bank

Get access to hundreds of Network Troubleshooting and Tools interview questions and detailed answers.

Join thousands of developers preparing for their dream job.