Network Troubleshooting and Diagnostics Questions

Assess the candidate s ability to diagnose and resolve network problems with a systematic, layer by layer approach. Expect demonstration of methods for isolating faults at the physical layer, the data link layer, the network layer, the transport layer, and the application layer. Core skills include verifying physical connectivity and interface status and counters, examining media access control tables and address resolution protocol entries, checking routing tables and next hop reachability, validating access control lists and firewall behavior, inspecting service binding and listening ports, and analyzing packet captures. Candidates should be familiar with tools and techniques such as ping and traceroute, packet capture with Wireshark and tcpdump, socket inspection with ss and netstat, interface diagnostics with ethtool and ip link, and telemetry such as NetFlow and sFlow. Common scenarios to practice include a host that can reach external Internet Protocol addresses but not a particular subnet, domain name resolution that succeeds while an application fails to connect, application services reachable internally but not from outside, maximum transmission unit mismatch on tunnels and virtual private network links, duplicate address or address resolution protocol conflicts, virtual local area network misconfiguration and inter virtual local area network routing issues. Interviewers will look for clear stepwise narrowing of scope, reproducible tests and evidence collection, and reasonable mitigations or next steps.

MediumTechnical

56 practiced

A Kubernetes pod cannot reach an external API while the node can. Detail checks across layers to determine whether the issue is CNI plugin misconfiguration, kube-proxy iptables rules, network policy, host firewall, or namespace isolation. Include exact commands you would run inside the pod, on the node, and in the control plane to collect evidence.

Sample Answer

**Approach — layered checks (pod → node → control plane)**

**1) From inside the Pod**- Verify DNS and external connectivity:

bash

# Exec into pod
kubectl exec -it <pod> -- /bin/sh
# DNS
nslookup api.example.com || dig +short api.example.com
# Ping/traceroute (if available) and TCP check
ping -c 3 1.1.1.1
traceroute -m 10 api.example.com
curl -v --connect-timeout 5 https://api.example.com:443 || nc -vz api.example.com 443
# Check pod IP and routes
ip addr show
ip route show
cat /proc/net/iptables

Reasoning: isolates whether app-level/DNS or basic L3 reachability fails and whether pod has correct IP/routes.

**2) On the Node hosting the pod**- Compare node vs pod connectivity and inspect CNI and iptables rules:

bash

# identify node
kubectl get pod <pod> -o wide
# SSH node then:
ip addr show
ip route show
# Test same connectivity as pod
curl -v https://api.example.com:443
# CNI plugin: check interfaces and CNI logs
ip link show | egrep 'cni|flannel|cali|weave'
ps aux | egrep 'calico|flannel|weave'
journalctl -u kubelet -u calico-node -u flanneld --no-pager | tail -n 200
# kube-proxy iptables rules
iptables-save | egrep 'KUBE-SVC|KUBE-SEP|KUBE-FIREWALL' -n
iptables -t nat -S
# Host firewall
iptables -L -n -v
ss -tunlp | egrep 443

Reasoning: confirms node can reach API, inspects CNI interfaces and kube-proxy rules that affect pod SNAT/forwarding, and host firewall blocking.

**3) Control plane / cluster config**- Check NetworkPolicy, CNI config, kube-proxy mode, and pod namespace isolation:

bash

# NetworkPolicies affecting pod
kubectl get networkpolicy -n <ns> -o yaml
# Describe pod and endpoints/services
kubectl describe pod <pod> -n <ns>
kubectl get svc -n <ns> -o wide
# kube-proxy configmap and mode
kubectl -n kube-system get configmap kube-proxy -o yaml
# CNI DaemonSet and logs
kubectl -n kube-system get ds
kubectl -n kube-system logs ds/calico-node | tail -n 200
# Check if iptables FORWARD policy default DROP on nodes via daemonset logs or node config
kubectl get nodes -o json | jq '.items[].metadata.name'

Reasoning: NetworkPolicy can block egress; kube-proxy mode (iptables vs ipvs) affects rules; CNI config errors appear in daemonset logs.

**How to interpret evidence (quick mapping)**- CNI misconfiguration: pod lacks expected veth/route, CNI logs show errors, pod IP unreachable from node network namespace.- kube-proxy iptables: iptables-save lacks KUBE- rules or wrong SNAT chains; services not NAT’ing.- NetworkPolicy: explicit deny/egress policy targeting namespace/pod blocks traffic.- Host firewall: iptables/ufw DROP/REJECT rules on node blocking pod traffic but node-initiated connections allowed (different conntrack/SNAT).- Namespace isolation: check network namespaces; if pod network namespace lacks proper veth pair or has iptables blocking locally.

**Next steps**- Recreate with a debug pod (hostNetwork: true) to compare, patch suspected policy/CNI, or run tcpdump on node/veth to trace packets:

bash

# On node
tcpdump -i <vethX> host api.example.com and port 443 -w /tmp/pod-trace.pcap

MediumTechnical

53 practiced

Users report one-way audio in VoIP calls: they can hear remote parties but remote parties cannot hear them. Provide a systematic troubleshooting approach focused on RTP media path, including separating SIP signaling from RTP media, checking NAT traversal and pinholes, verifying symmetric RTP, confirming media port ranges and ACLs, and using packet capture or RTP tools to confirm media packets are flowing in both directions.

Sample Answer

**Brief approach / goal**Isolate SIP (signaling) from RTP (media). Confirm signaling is OK, then verify RTP flows both directions, NAT pinholes, symmetric RTP, port ranges, and ACL/firewall rules.

**Step-by-step troubleshooting**1. Verify SIP: confirm INVITE/200 OK/ACK complete and SDP contains candidate IP/ports for RTP. Use sngrep or SIP logs.2. Capture media: run packet capture at endpoints and at SBC/firewall:

bash

tcpdump -i any host <caller_ip> and udp and port 10000-20000 -w caller_rtp.pcap
tcpdump -i any host <callee_ip> and udp and port 10000-20000 -w callee_rtp.pcap

3. Confirm RTP bidirectional: open with tshark or Wireshark, check RTP streams and packet counts/RTCP SR. Look for one-way: packets arrive only to one side.4. NAT traversal / pinholes: check if caller behind NAT created outbound pinhole to remote IP/port in SDP. If asymmetric NAT, verify symmetric RTP support or use SBC/TURN. Verify pinhole timeouts and that re-INVITE/keepalives (RTP/AVP keepalive or STUN) refresh NAT.5. Symmetric RTP: ensure network path preserves source IP/port; if remote sends to different port than caller opened, firewall drops them. Configure RTP symmetric mode on media servers/SBCs.6. Media port ranges & ACLs: validate firewall allows configured RTP range (UDP) and ACLs permit both source/dst. Confirm no port translation changes.7. SBC/Media flow: check if SBC is anchoring media; inspect its media IP in SDP. If media anchored, ensure SBC can reach both endpoints.8. Tools: use sngrep, tshark, rtptools or rtpbreak to analyze jitter, sequence numbers, SSRC. Check RTCP reports for packet loss direction.9. Fix & verify: adjust NAT rules, open/allow port ranges, enable symmetric RTP or set media to flow through SBC/TURN, restart call and re-capture to confirm two-way RTP.

**Example tshark check**

bash

tshark -r caller_rtp.pcap -Y "rtp" -T fields -e rtp.ssrc -e ip.src -e udp.srcport -e frame.time_relative

Look for packets from both endpoints and matching sequence progression.

MediumTechnical

54 practiced

After a failover, NATed connections from a pool of clients fail to reach stateful services; logs show 'connection reset' and old translations remain. Describe how to inspect conntrack and NAT tables on Linux, how to safely clear or synchronize conntrack entries during failover, and how enterprise load balancers expose or migrate session state to avoid disruption.

Sample Answer

**Brief diagnosis**Connection resets + old NAT translations indicate stale conntrack state on the active box after a failover: back-end sees new source/port or different node, but conntrack still translates to the old tuple so packets get dropped or reset.

**How to inspect conntrack & NAT tables**- View conntrack entries (requires conntrack-tools):

bash

# List all connection tracking entries
sudo conntrack -L

# Filter for a client IP or proto/port
sudo conntrack -L -s 10.0.1.5
sudo conntrack -L -p tcp --dport 80

- Show kernel conntrack stats:

bash

cat /proc/net/nf_conntrack
sudo sysctl net.netfilter.nf_conntrack_count

- Inspect NAT table rules/translations:

bash

sudo iptables -t nat -L -n -v     # legacy
sudo nft list table nat          # nftables

**Safely clear or synchronize conntrack entries during failover**- Prefer surgical deletes over global flush: - Delete entries for specific client/server tuples:

bash

sudo conntrack -D -s 10.0.1.5   # remove all conntrack for client
  sudo conntrack -D -p tcp --orig-port-src 52344 --orig-port-dst 443

- Remove only RELATED/ESTABLISHED as needed; avoid flushing entire table in production (breaks many live flows).- Graceful approach: 1. Drain the old active node (stop accepting new sessions, mark as offline at LB). 2. Let existing flows finish or expire (shorten net.netfilter.nf_conntrack_tcp_timeout_established temporarily only with caution). 3. Delete remaining affected entries for known client pools.- For automated sync across HA pair: - Use conntrackd (part of conntrack-tools) to replicate state between nodes:

bash

# conntrackd config enables nf_conntrack sync over TCP
  sudo systemctl enable --now conntrackd

- Or use platform feature (Linux-HA, keepalived + conntrackd) to mirror states so failover preserves sessions.

**How enterprise load balancers avoid disruption**- Session/state replication: commercial LBs (F5, Big-IP, A10, Citrix) offer connection mirroring / connection table sync so the standby has active session/NAT state.- Persistent session mechanisms: cookie-based stickiness or source-IP persistence reduce reliance on conntrack for long-lived affinity.- Connection draining + graceful failover: active node signals connections to drain while standby picks up new flows; some vendors stream TCP state (connection mirroring) to the peer.- Design alternatives: use symmetric NAT (SNAT on same path) or keep NAT at edge so backends see consistent source, or use service meshes / proxies that handle session handoff.

**Practical tips**- Always test conntrack deletion steps in staging.- Log and backup current conntrack list before mass operations.- Prefer targeted deletions, decrease timeouts only when you can tolerate brief breaks, and use conntrackd or vendor state sync for production HA.

HardSystem Design

48 practiced

Design an automated diagnostics 'runbook runner' for on-call responders that collects a standardized set of evidence (interface state, route tables, selected packet capture, application binding info) when an incident is triggered. Explain which commands to run, how to securely collect and redact sensitive data, how to store artifacts, and how to make the outputs easy to consume for post-incident analysis.

Sample Answer

**Clarify requirements & constraints**- Trigger types: pager alert, manual on-call start, scheduled health-check.- Targets: routers, switches, firewalls, Linux hosts, cloud VPCs.- Security: least privilege, encrypted transit/storage, PII redaction, auditable access.

**High-level architecture**- Runbook Runner service (serverless or container) invoked by alert or CLI.- Agentless (SSH/NETCONF/REST) or lightweight agent (signed binary) on hosts.- Central artifact store (S3-compatible encrypted bucket + metadata DB).- Web UI / CLI to view summarized evidence and download artifacts.

**Standardized evidence & example commands**- Interface state:

bash

show interfaces description
show ip interface brief
ip -s link show

- Routing tables:

bash

show ip route
show bgp summary
netstat -rn

- Packet capture (selective):

bash

tcpdump -c 200 -s 0 -w /tmp/capture.pcap host 10.0.0.5 and port 443

- Application binding/info:

bash

ss -tunlp
show running-config | include <app-acl|nat|svc>
curl -s http://localhost:9000/health

- Logs (tail + timestamped slice):

bash

journalctl -u app -n 500 --since "5 minutes ago"

**Collection workflow**1. Authenticate to target via ephemeral certs (Vault-signed), scoped roles.2. Run pre-defined command set, capture stdout/stderr, timestamps, host metadata.3. Limit captures (size, duration) and sample by filters to reduce sensitive payloads.

**Secure collection & redaction**- Transport: mTLS + client certs; agent signs evidence.- Store encrypted-at-rest (KMS).- Redaction steps: - Regex-based masking for IPs/credentials, configurable whitelist. - PII detector for headers/body; replace with tokens and store original in HSM-only-bounded access audit.- Audit trail: who triggered, commands run, keys used.

**Artifact storage & indexing**- Store artifacts as versioned objects with metadata: incident-id, runbook-id, target, checksums.- Metadata DB (Postgres/Elastic) for fast search (by IP, device, time, symptom).- Retention policy tiering (short-term full artifacts, long-term summaries).

**Make outputs easy to consume**- Auto-generated summary report: key findings (down interfaces, missing routes, high error rates), one-line remediation suggestions.- Attach visualizations: interface graphs, BGP neighbor timelines, pcap quick-filter links.- Provide a UI with diff views (current vs baseline config) and downloadable tarball.- Provide CLI/jupyter integration for post-incident deeper analysis.

**Scalability & trade-offs**- Agentless simpler to deploy but higher latencies; agents enable richer telemetry.- Sampling PCAP reduces sensitivity and size but may miss transient packets — allow on-demand full capture for escalations.

**Metrics & validation**- Track run success rate, time-to-collect, artifact size, redaction false-positive/negative via periodic audits.

EasyTechnical

48 practiced

Explain differences to consider when using traceroute for IPv4 versus IPv6 in a troubleshooting session. Include common flags or parameters for TCP/UDP/ICMP probes, how intermediate devices may respond differently (ICMPv6 vs ICMP), and how to interpret occasional '!' markers, MPLS labels, or rate-limited ICMP replies in traceroute output.

Sample Answer

**Approach** Compare behavior, tool flags, probe types, intermediate-device responses, and common artifacts (MPLS, rate-limiting, special markers). Speak as a Network Engineer troubleshooting connectivity.

**IPv4 vs IPv6: key differences**- Use traceroute -6 or traceroute6 for IPv6; many platforms auto-select family via destination.- ICMPv6 is core to IPv6 (Neighbor Discovery, PMTU via "Packet Too Big"); lack of ICMPv6 can break path MTU and diagnostics.- IPv6 addresses are longer; prefer -n to avoid DNS delays.

**Common flags / probe types**- Linux traceroute (classic): - UDP default: traceroute -n -q 3 -w 2 destination - ICMP: traceroute -I destination - TCP: traceroute -T destination (or use tcptraceroute/hping3) - IPv6: traceroute -6 destination- Other tools: tracepath (Linux), tcptraceroute, mtr, and Windows: tracert (ICMP), Windows 10+ supports -d numeric.- Important options: -f (first TTL), -m (max TTL), -q (probes/TTL), -w (timeout).

**How intermediates respond (ICMP vs ICMPv6)**- IPv4: routers send ICMP Time Exceeded / Destination Unreachable (port unreachable for UDP probes).- IPv6: routers send ICMPv6 Time Exceeded and ICMPv6 Packet Too Big (used for PMTU). Firewalls or routers may block ICMPv6, causing silent failures.- RFC 4950 allows devices to include MPLS label stack in ICMP/ICMPv6 responses; not all vendors implement it.

**Interpreting artifacts**- '*' or timeouts: probe lost or ICMP rate-limited/blocked. Multiple '*' at same hop often indicates rate limiting or firewalling of TTL-exceeded messages.- '!' markers (Cisco-style): in extended traceroute or ping, symbols denote specific ICMP replies (e.g., !H host unreachable, !N network unreachable, !P protocol unreachable, !F fragmentation needed). Check device docs for exact meanings.- MPLS labels: traceroute output may show "MPLS" entries or label values when intermediate LSRs include RFC 4950 info. Seeing MPLS indicates traffic traverses an LSP; differing RTTs reflect penultimate hop popping or LSP behavior.- Asymmetric paths: different RTTs and differing hop lists for forward vs reverse probes are normal; correlate with return-path routing.

**Practical tips**- Use TCP probes (port 80/443) to test through firewalls that allow only established services.- If ICMPv6 Packet Too Big seen, adjust MTU or enable PMTU tests.- Combine traceroute with packet captures (tcpdump/wireshark) to confirm probe handling and MPLS label presence.- When seeing intermittent replies, suspect ICMP rate-limiting—confirm by increasing probe count or using different probe types.

This framing helps choose the right probe, interpret intermediate device behavior, and distinguish real path issues from rate-limited or policy-driven traceroute artifacts.

Unlock Full Question Bank

Get access to hundreds of Network Troubleshooting and Diagnostics interview questions and detailed answers.

Join thousands of developers preparing for their dream job.