Systematic Troubleshooting and Debugging Questions

Covers structured methods for diagnosing and resolving software defects and technical problems at the code and system level. Candidates should demonstrate methodical debugging practices such as reading and reasoning about code, tracing execution paths, reproducing issues, collecting and interpreting logs metrics and error messages, forming and testing hypotheses, and iterating toward root cause. Topic includes use of diagnostic tools and commands, isolation strategies, instrumentation and logging best practices, regression testing and validation, trade offs between quick fixes and long term robust solutions, rollback and safe testing approaches, and clear documentation of investigative steps and outcomes.

EasyTechnical

56 practiced

Explain what the following tcpdump filter captures and how you would use it to reproduce a web application issue: sudo tcpdump -i eth0 tcp port 443 and host 10.0.1.5 -w capture.pcap. Describe how to open and analyze the resulting capture and what indicators you would look for (handshake timing, retransmits, RSTs).

Sample Answer

**What the filter captures**

The command captures all TCP traffic to/from host 10.0.1.5 on port 443 (HTTPS) on interface eth0 and writes it to a file:

bash

sudo tcpdump -i eth0 tcp port 443 and host 10.0.1.5 -s 0 -w capture.pcap

(Note: add -s 0 to avoid truncating packet payloads if you need TLS records/headers.)

**How to reproduce a web app issue**

- Run the capture on the client-facing host or the server at the time you reproduce the problem.- Capture both sides if possible (client and server) to compare timings.- Start capture a bit before reproducing and stop shortly after the failure to limit file size.

**Open & analyze the capture**

- Open in Wireshark (GUI) or use tshark for CLI:

bash

wireshark capture.pcap
# or
tshark -r capture.pcap

- Useful Wireshark display filters: - TLS handshakes: tls.handshake - TCP retransmits: tcp.analysis.retransmission or tcp.analysis.fast_retransmission - RSTs: tcp.flags.reset == 1 - Round-trip time & conversation: Statistics → TCP Stream Graphs → Round Trip Time

**Indicators to look for and interpretation**

- Handshake timing: Check SYN → SYN/ACK → ACK spacing. Long gaps indicate high latency or packet loss (SYN retransmits).- TLS handshake: Look for ClientHello and ServerHello; time between them shows server responsiveness or TLS negotiation delays.- Retransmits: Frequent tcp.analysis.retransmission means packet loss or buffer drops. Correlate with interface drops or firewall logs.- RSTs: Abrupt connection resets (RST) indicate the peer closed the socket or middlebox terminated the session; look at preceding packets for reasons (e.g., zero-window, TCP options).- Zero window / delayed ACKs: tcp.analysis.zero_window or many duplicate ACKs indicate congestion or receive-side resource issues.- TLS alerts/application errors: Look for TLS Alert messages (tls.alert_message) or application-layer errors if decrypted.

**Next steps**

- Correlate with server logs, firewall rules, and interface counters.- If needed, capture packet traces with TLS key logging or on the server to decrypt TLS and inspect HTTP payloads.- Use timestamps and both-side captures to determine whether issue is network (latency/loss) or application (slow response, crashes).

HardTechnical

35 practiced

Services behind a NAT device are showing intermittent resets under high load; you suspect NAT/PAT ephemeral port exhaustion or port reuse collisions. Describe how to confirm ephemeral port exhaustion (metrics to collect), how to reproduce in a lab, mitigation techniques (port range expansion, multiple NATs, dedicated SNAT pool), and long-term fixes.

Sample Answer

**Situation / Goal**Explain how to confirm NAT ephemeral port exhaustion, reproduce it in lab, short-term mitigations, and long-term fixes—targeted for a Network Engineer role.

**Metrics to collect (what to measure)**- NAT translations in-use / peak concurrent mappings (conntrack count on Linux: conntrack -C and /proc/net/nf_conntrack).- Ephemeral port utilization per external IP (sockets bound per 10.0.0.0/8 source).- TCP resets and SYN/SYN-ACK retransmits (tcpdump, sFlow, NetFlow, or packet captures).- TCP state distribution (TIME_WAIT, ESTABLISHED) and average connection lifetime.- Port allocation churn (how often same source-port reused within TCP TIME_WAIT).- System logs for “no buffer” or allocator errors and NAT device CPU/memory metrics.

**How to confirm**- Correlate spikes in conntrack / NAT table nearing configured limits with increases in TCP RSTs and failed sessions.- Capture packets at NAT: observe new connections mapped to external ports already in use (or reused while peer still active) causing RSTs.- Reproduce by watching counters while load increases; if RSTs begin when translations ~100% allocated → exhaustion.

**Lab reproduction**- Build topology: many clients → NAT device (Linux iptables/nftables or commercial NAT) → server.- Use load generator (wrk, Tsung, custom scripts) creating many concurrent short-lived TCP connections from many client IPs; tune client source-ports to saturate ephemeral range.- On Linux NAT: set small ip_local_port_range to accelerate exhaustion:

bash

sysctl -w net.ipv4.ip_local_port_range="30000 30100"

Start thousands of connections and watch conntrack:

bash

conntrack -L | wc -l
  tcpdump -n -i eth0 'tcp[tcpflags] & (tcp-rst) != 0'

- Observe resets when range depleted.

**Short-term mitigations**- Expand ephemeral port range on NAT device and clients (increase ip_local_port_range).- Increase conntrack / NAT table size and tune timeouts (reduce TIME_WAIT, but carefully).- Implement multiple external IPs / SNAT pool so mapping space multiplies.- Use multiple NAT devices with source-IP hashing (stateless load-share) or ECMP.- Use dedicated SNAT pool for high-volume services to avoid port collisions with other tenants.- Increase TCP TIME_WAIT reuse timers cautiously; prefer reducing connection churn.

**Long-term fixes**- Move high-volume services off shared NAT: give services public IPs or use load balancers that preserve source ports.- Use connection pooling or keep-alive to reduce ephemeral allocation (application changes).- Implement stateful connection tracking scaling: NAT44-S pinholing, scalable NAT (e.g., NAT64 appliances, CGNAT with larger pools).- Adopt architectures that avoid source-NAT for east-west traffic (routing, private networking, overlay).- Instrument and autoscale NAT function (add more SNAT addresses or NAT instances automatically).- Monitor and alert on NAT-utilization thresholds; include capacity planning in runbook.

**Tradeoffs / Risks**- Reducing TIME_WAIT risks cross-connection collisions; expanding port ranges requires IP/OS support.- Multiple NATs add complexity for troubleshooting and logging; prefer dedicated SNAT for critical services.

This approach shows how to validate exhaustion, reproduce reliably in lab, apply immediate mitigations, and design durable architectural fixes.

HardTechnical

36 practiced

Core switch ports report increasing CRC and FCS errors affecting application traffic with occasional corrupt packets. Describe a methodical plan to diagnose whether this is caused by physical media (cable/SFP), optics, switch ASIC bugs, or upstream devices. Include exact commands, diagnostic tests (TDR, port swap, loopback, error counters), safe testing procedures, and criteria for hardware replacement.

Sample Answer

**Approach (brief)** I would follow a staged, non-disruptive-to-production workflow: collect data, isolate the port, test physical optics/cable, test switch ASIC/firmware, test upstream device, then decide replacement.

**1) Collect baseline and symptoms**- Check counters and errors:

bash

show interfaces <intf> counters
show interfaces <intf> | include CRC|FCS|errors
show logging | include <intf>
show tech-support

- Note affected VLANs, flows, time pattern, recent changes.

**2) Safe isolation (low-impact)**- If port is carrying critical traffic, move a single test VM/app to a maintenance window or mirror traffic:

bash

monitor session 1 source interface <intf>
monitor session 1 destination interface <lab-intf>

- Avoid physical disconnects during business hours unless scheduled.

**3) Physical checks and optics**- Inspect SFP type and transceiver mismatch:

bash

show interfaces transceiver detail | include <intf>
show inventory

- Verify SFP/DAC model and firmware compatibility with HCL.- Swap SFP with a known-good same-type unit (one change at a time). Test.- Replace cable with a known-good certified cable. For copper run TDR (if switch supports) or use a handheld tester.

bash

test cable-diagnostics tdr interface <intf>
show cable-diagnostics tdr interface <intf>

- Criteria: errors clear after SFP/cable swap → replace suspect element.

**4) Optical tests**- Check optical levels and OSNR:

bash

show interfaces transceiver optics | include RX|TX|power

- Look for out-of-spec RX power, high loss or excessive power. If optics readings abnormal, replace optics and re-test.

**5) Layer-1 loopback and bit-error tests**- Where supported, enable local loopback to isolate PHY vs link partner.

bash

test interface <intf> loopback local

- Run traffic generator or iperf from both ends to check CRC generation.

**6) Port swap and upstream isolation**- Swap the active port’s connection to another known-good port on same switch or a different line card. If errors follow cable/SFP to new port → media/optics. If errors stay on original port number → switch ASIC/port hardware.- Move link to an alternate upstream device (if possible) to rule out upstream NIC/driver.

**7) Firmware/ASIC checks**- Search vendor advisories for ASIC bugs matching symptoms. Capture debug logs and counters over time:

bash

show platform hardware fwd-asic errors
show logging onboard | include asic

- If replication occurs with known-good optics/cable and on different ports after swap, escalate to vendor TAC with packet captures and tech-support tar.

**8) Criteria for replacement**- Replace cable/SFP if errors vanish after swap.- Replace line card/port ASIC if: - Errors remain after good cable/SFP and follow the physical port (not the link). - Vendor confirms hardware fault or clear internal CRC/ECC errors on ASIC.- Replace entire switch if multiple ports/line cards fail or vendor recommends RMA after reproducible tests.

**9) Documentation & rollback**- Document commands, timestamps, swaps, and test results. Schedule maintenance for any replacements and confirm restoration of normal error-free counters after replacement (run tests for 24–72 hours).

MediumTechnical

32 practiced

Backends behind a load balancer intermittently fail health checks even though application logs report successful responses for the same requests. Outline a debugging approach, considering health probe source IP, ephemeral port exhaustion on backends, TLS handshake timing, sticky sessions, and how to validate with pcap and synthetic probes.

Sample Answer

**Approach summary (high level)**I would treat this as a deterministic troubleshooting flow: identify if probes originate where you expect, confirm backend resource exhaustion/state, examine TLS timing/handshake behavior, check session persistence effects, then validate with packet captures and synthetic probes.

**Steps**1. Verify health-probe source IP and path- Confirm LB health-source IP(s) and ensure backends accept them (firewall/ACLs). Check LB docs for probe CIDRs.- On backend, confirm probe arrival:

bash

sudo tcpdump -nni any host <load-balancer-probe-ip> and port <probe-port> -c 50

2. Check ephemeral port exhaustion / connection state- Look for many TIME_WAIT/ESTABLISHED from LB IPs; ephemeral exhaustion can cause refused connects.

bash

ss -tn sport = :<probe-port> | wc -l
ss -s

- Inspect kernel ephemeral port range and TIME_WAIT settings and tune if needed.

3. Examine TLS handshake timing- If health checks use HTTPS, verify TLS handshake completes within probe timeout. Reproduce from LB subnet (or simulate source) with:

bash

openssl s_client -connect backend:443 -servername host.example.com -tlsextdebug

- Use tcpdump to capture SYN/ACK/TLS records and measure time between SYN and ClientHello/ServerHello.

4. Consider sticky sessions/session persistence- Sticky sessions can route probes to different backend instances or reuse connections unexpectedly. Ensure health probes bypass persistence or that persistence is configured correctly.

5. Validate with pcap and synthetic probes- Correlate LB logs, backend app logs, and pcap timestamps to match a failed probe to the TCP/TLS trace.- Generate synthetic probes from a host in LB’s source subnet:

bash

curl -v --resolve host:443:backend_ip https://host/health --max-time 5
tcpdump -w probe.pcap host <load-balancer-ip> or host <backend-ip>

- Analyze pcap in Wireshark: filter by probe IP and check TCP retransmits, RSTs, or TLS handshake delays.

**Expected findings & remediation**- If ephemeral ports exhausted: increase ephemeral range or reduce TIME_WAIT via tcp_tw_reuse/tcp_fin_timeout; scale NAT/connection tracking.- If TLS handshake exceeds probe timeout: increase probe timeout or optimize cert chain/cipher settings; enable session resumption.- If probes blocked by firewall: allow LB probe CIDRs.- If sticky sessions misrouting probes: disable persistence for health checks or ensure consistent hashing.

This method ties LB logs, backend state, kernel networking metrics, and pcap evidence to pinpoint root cause and a concrete fix.

EasyTechnical

39 practiced

List the SNMP OIDs or generic metrics (interfaces, input/output errors, utilization, queue drops, discards, temperature) you would monitor to detect early signs of link degradation and congestion on routers and switches. Explain why each metric is helpful and suggest threshold levels for alerts.

Unlock Full Question Bank

Get access to hundreds of Systematic Troubleshooting and Debugging interview questions and detailed answers.

Join thousands of developers preparing for their dream job.