Networking Fundamentals and Troubleshooting Questions
Comprehensive coverage of core computer networking principles and the practical diagnostic and operational skills required to design, operate, and troubleshoot production systems. Fundamental concepts include the Open Systems Interconnection model layers, the Transmission Control Protocol and the Internet Protocol stack, the User Datagram Protocol, socket and port semantics, address notation and subnetting, Network Address Translation, Dynamic Host Configuration Protocol, and the Domain Name System resolution process. Infrastructure and architectural topics include switching and virtual local area networks, routing concepts and routing table behavior including Border Gateway Protocol basics, load balancing strategies and failure modes, firewall and access control, virtual private network technologies, and container and service network communication patterns. Diagnostic and tooling skills cover connectivity testing and path analysis, process and socket inspection, packet capture and analysis, and common command line tools and utilities used for network investigation. Performance and reliability topics include latency, bandwidth and throughput, packet loss, congestion and congestion control, connection pooling, timeout and retry strategies, and approaches to optimization. Observability, monitoring, and security practices include collecting and interpreting network metrics, logs, and traces, using packet capture tools for root cause analysis, and understanding how network issues surface in distributed applications. At senior levels expect discussion of network performance tuning, capacity planning, load balancer behavior at scale, and design decisions that affect system reliability and security.
HardTechnical
55 practiced
You're responsible for capacity planning for a fleet of edge load balancers. Describe how to estimate required network throughput, maximum concurrent connections, conntrack table size, and backend connection rates based on historical peak RPS, average request size, TLS overhead, keepalive behavior, and desired headroom. Which metrics and stress tests will you run to validate your model?
Sample Answer
Approach: turn observable inputs (peak RPS, avg request size, TLS overhead, keepalive, headroom) into capacity targets for bandwidth, concurrent connections, conntrack size, and backend connection rate using simple formulas, then validate with metrics and stress tests.Estimation formulas (per-node or fleet-wide):- Network throughput (bps) = peak_RPS * avg_request_size_bytes * 8 * (1 + tls_overhead_fraction) * (1 + response_size_fraction). Multiply by desired_headroom (e.g., 1.3).- Max concurrent client connections = peak_RPS * avg_request_latency_seconds * keepalive_multiplier - If requests are short-lived: concurrency ≈ peak_RPS * avg_latency. - For keepalive: add idle keepalive sockets = active_clients * pct_idle_keepalive.- Conntrack table size = max_concurrent_connections * safety_factor (1.5–2) + ephemeral NAT entries for backend connections.- Backend connection rate = peak_RPS * backend_connections_per_request (1 for one-request-per-conn; lower if connection re-use) * headroom.- Backend concurrent connections = backend_conn_rate * backend_avg_response_time.Concrete example: 10k peak_RPS, 1KB avg request, TLS 10% overhead, avg latency 50ms:- throughput ≈ 10k * 1024 * 8 *1.1 ≈ 90 Mbps → with 30% headroom ≈ 120 Mbps per fleet divided by nodes.- concurrency ≈ 10k * 0.05 = 500 concurrent active; if 30% idle keepalive add 150 → 650; conntrack ≈ 650*2 = 1,300 entries.Metrics to collect:- RPS (per-node, 5s/1m), bytes_in/out, TLS handshakes/sec, active TCP connections, TIME_WAIT counts, established sockets, conntrack stats (entries, alloc failures), backend connections/sec, backend pool reuse ratio, latency percentiles, retransmits, SYN retries.Stress tests to validate:- Synthetic traffic generator (wrk/fortio/vegeta) reproducing peak RPS and request-size mix with TLS and keepalive variants.- Ramp tests: step RPS up to 150% of modeled peak, hold, and spike tests (burst for short duration).- Connection churn tests: many short-lived connections to exercise conntrack and ephemeral port exhaustion.- Keepalive-heavy scenarios: large number of idle sockets to validate fd and memory limits.- Backend saturation: limit backend capacity to observe LB behavior (queueing, connection reuse).- Long-duration soak (24–72h) to catch leaks (fd, memory, conntrack).Validation criteria:- No packet drops, no conntrack or FD exhaustion, p95/p99 latency within SLO, TLS handshake rate sustainable, CPU and NIC util under thresholds, autoscaling triggers work. Iterate model based on measured reuse ratios, observed TLS overhead, and actual latency.
MediumTechnical
26 practiced
Explain TCP congestion control basics including slow start, congestion avoidance, fast retransmit, and fast recovery. How does packet loss impact throughput and what metrics should SREs collect to detect congestion early in production?
Sample Answer
TCP congestion control prevents the network from being overloaded by adapting the sender’s window based on inferred network capacity.- Slow start: Sender begins with a small congestion window (cwnd, often 1–2 MSS) and exponentially grows cwnd each RTT (cwnd += bytes ACKed) until a threshold (ssthresh) or loss occurs. This probes available capacity quickly.- Congestion avoidance: After crossing ssthresh, growth becomes linear (AIMD — Additive Increase, Multiplicative Decrease) to cautiously find fair share: cwnd += MSS*(MSS/cwnd) per RTT.- Fast retransmit: When 3 duplicate ACKs arrive, sender infers a lost segment and retransmits immediately without waiting for RTO, reducing recovery latency.- Fast recovery: Instead of dropping to slow-start, sender halves cwnd (multiplicative decrease), sets ssthresh, and enters congestion avoidance, allowing continued transmission and faster recovery.Packet loss reduces effective throughput because TCP backs off cwnd; throughput ≈ (MSS/RTT) * sqrt(1/p) for loss probability p. Loss spikes cause throughput collapse and increased latency.SRE metrics to detect congestion early:- RTT and RTT variance (per-flow or flow-proxy)- Packet loss rate and retransmissions (per interface/service)- TCP congestion window and inflight packets (via host TCP stats)- TCP RTO count and tail latencies- ECN markings if available- Link utilization, queue lengths, and bufferbloat indicators- SYN retransmits and connection establishment failuresAlert on rising RTT/RTT variance, increased retransmits/loss, and sustained high queue lengths. Correlate with traffic spikes and deploy rate-limiting or traffic shaping as mitigations.
EasyTechnical
29 practiced
During an incident where a service fails to accept connections, outline a sequence of commands and the expected indicators you would look for using 'ss -tanp', 'netstat -plant', and 'lsof -i' to determine whether the process is listening, whether ports are bound to the expected interfaces, and whether the firewall is blocking traffic. Provide example output snippets you would inspect.
Sample Answer
Approach: run ss/netstat/lsof to confirm (1) a process is actually LISTENing on the expected port, (2) which interface/address it's bound to (0.0.0.0 / :: / 127.0.0.1 / specific IP), (3) PID/process owner, and (4) whether packets are being accepted vs dropped (follow up with firewall rules). Sequence of commands, what to inspect, and example snippets:1) Check TCP listeners with ss
bash
ss -tanp | grep ':8080'
Look for: "LISTEN" state, local address (0.0.0.0:8080 vs 127.0.0.1:8080), and pid/name.Example:tcp LISTEN 0 128 0.0.0.0:8080 0.0.0.0:* users:(("myapp",pid=1234,fd=7))Interpretation: process 1234 is listening on all interfaces — not a bind issue.If you see:tcp LISTEN 0 128 127.0.0.1:8080 0.0.0.0:* users:(("myapp",pid=1234,fd=7))Interpretation: bound to loopback only — remote connections will fail.If nothing returned: process not listening on that port.2) Equivalent with netstat (older systems)
bash
sudo netstat -plant | grep ':8080'
Inspect: "LISTEN", Local Address IP:port, PID/Program name.Example:tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1234/myapp3) Find which process holds open sockets with lsof
bash
sudo lsof -iTCP:8080 -sTCP:LISTEN -Pn
Example:COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEmyapp 1234 app 7u IPv4 12345 0t0 TCP *:8080 (LISTEN)Interpretation: confirms PID and user; if lsof empty — no listener.4) Inspect connection states (inbound SYNs)
bash
ss -tn state syn-recv '( sport = :8080 or dport = :8080 )'
ss -tan | grep 8080
If many SYN_RECV entries from remote IPs, server is receiving SYNs but not completing handshake — could be app slow to accept or firewall interfering with responses or backend overloaded.5) Check firewall (local)- iptables:
bash
sudo iptables -L -n --line-numbers
Look for DROP/REJECT rules for the port or default policy DROP on INPUT.- nftables:
bash
sudo nft list ruleset
- ufw:
bash
sudo ufw status verbose
Example iptables snippet:Chain INPUT (policy ACCEPT)num target prot opt source destination1 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:222 DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080Interpretation: DROP on dpt:8080 blocks traffic even if process listens.Investigation flow summary:- If no LISTEN: restart service / check logs.- If LISTEN on 127.0.0.1: change bind config to 0.0.0.0 or proxy.- If LISTEN on 0.0.0.0 but clients can't connect: check firewall rules, security groups (cloud), and SYN_RECV (indicates handshake issues).- If listener exists and no firewall blocking, capture tcpdump (tcpdump -nni any port 8080) to observe packets.
HardTechnical
35 practiced
Describe important Linux kernel TCP tuning knobs (for example net.ipv4.tcp_rmem, tcp_wmem, tcp_congestion_control, tcp_moderate_rcvbuf) and how they impact throughput and latency for high-bandwidth, high-latency links. Show how you would calculate and set buffer sizes given a 10 Gbps link with 100 ms RTT (give the BDP calculation and recommended change).
Sample Answer
Situation/Goal: For a 10 Gbps link with 100 ms RTT you need TCP buffers and congestion control tuned so the pipe can be filled (high throughput) without adding excessive latency (bufferbloat). Key Linux TCP knobs, why they matter, and how to calculate/set them follow.Important knobs and impacts- net.ipv4.tcp_rmem / tcp_wmem (three values: min, default, max) - Control receive/send socket buffer sizes and autotuning bounds. If max < BDP, the sender/receiver cannot fill the link → throughput limited. Too-large buffers can add latency (bufferbloat) if queues build.- net.ipv4.tcp_congestion_control - Selects algorithm (cubic, reno, bbr, etc.). CUBIC is default and good for many networks; BBR (Bottleneck Bandwidth and RTT) often yields higher throughput and lower queueing latency on high-BDP links because it paces to measured bandwidth/RTT instead of filling buffers.- net.ipv4.tcp_moderate_rcvbuf - When enabled, allows kernel to expand receive buffer beyond advertised window using heuristics (beneficial if receive side can allocate memory). Helps on high-BDP links if tcp_rmem max is large enough.- net.core.rmem_max / net.core.wmem_max - Global kernel limits that cap per-socket tcp_*mem values. Must be >= tcp_rmem/tcp_wmem max.- net.ipv4.tcp_window_scaling, tcp_sack, tcp_timestamps - Window scaling must be enabled to support >64KB windows. SACK helps recovery on lossy/high-latency links.- QoS/pacing: fq_codel fq/pacing and tcp pacing (via BBR or fq) reduce queuing latency and bufferbloat.BDP calculation for 10 Gbps, 100 ms RTT- BDP (bits) = bandwidth (bits/s) * RTT (s) = 10 × 10^9 bits/s * 0.100 s = 1 × 10^9 bits- BDP (bytes) = 1e9 / 8 = 125,000,000 bytes ≈ 125 MB (≈119.2 MiB)So to fully utilize the link you need ~125 MB of send (and/or receive) buffer capacity per flow. In practice you may need slightly more to cover bursts and retransmissions (allow ~1.2×), but also consider aggregate flows.Recommended settings (example)- Enable features: sysctl -w net.ipv4.tcp_window_scaling=1 sysctl -w net.ipv4.tcp_moderate_rcvbuf=1 sysctl -w net.ipv4.tcp_sack=1- Increase kernel caps to exceed BDP: sysctl -w net.core.rmem_max=268435456 # 256MB sysctl -w net.core.wmem_max=268435456- Set tcp autotune bounds (min, default, max) — make max >= BDP: sysctl -w net.ipv4.tcp_rmem="4096 87380 167772160" # max 160MB sysctl -w net.ipv4.tcp_wmem="4096 87380 167772160"- Choose congestion control: sysctl -w net.ipv4.tcp_congestion_control=bbr (or use cubic with pacing if BBR not available; test both)Why these numbers:- tcp_{r,w}mem max must be >= BDP (~125 MB) so autotuning can scale socket buffers to fill the pipe.- net.core.wmem_max and rmem_max must be at least as large as tcp_* max.- Using BBR or pacing reduces queue fill and latency; fq_codel on egress queues mitigates bufferbloat.Trade-offs and operational notes- Large buffers per-flow increase system memory usage; account for concurrent flows (memory = buffer × concurrent sockets).- Large buffers can increase latency if queues are filled; use fq_codel + pacing or BBR to avoid bufferbloat.- Test: measure throughput with iperf3 and measure tail latency (ping, tc qdisc stats). Start with conservative increases, validate, then raise.- Monitor memory, retransmits, cwnd, RTT (ss, netstat -s, bpf tools).Example quick commandssysctl -w net.core.rmem_max=268435456sysctl -w net.core.wmem_max=268435456sysctl -w net.ipv4.tcp_rmem="4096 87380 167772160"sysctl -w net.ipv4.tcp_wmem="4096 87380 167772160"sysctl -w net.ipv4.tcp_congestion_control=bbrsysctl -w net.ipv4.tcp_moderate_rcvbuf=1Summary: Calculate BDP = bandwidth × RTT, set tcp_{r,w}mem max and core r/wmem to exceed BDP (here ~125 MB), enable window scaling and moderate_rcvbuf, and prefer BBR or pacing + fq_codel to achieve high throughput while minimizing added latency. Test and iterate in production-like conditions.
MediumTechnical
34 practiced
Write the tcpdump command to capture only TCP retransmits on interface eth0 between hosts 10.1.2.3 and 10.2.3.4, print timestamps and packet lengths, and save to rotated files of 100MB each. Explain the choices for snaplen, rotation flags, and the limitations of detecting retransmits purely via BPF filters.
Sample Answer
Command (rotate files of 100 MB, capture full packets on eth0 between two hosts):
bash
sudo tcpdump -i eth0 -n -s 0 -U -w /var/log/tcp_retx.pcap -C 100 'tcp and host 10.1.2.3 and host 10.2.3.4'
If you also need human timestamps and packet-length summaries on-screen, run a second reader (recommended) to avoid disabling writing:
bash
sudo tcpdump -r /var/log/tcp_retx.pcap -tt -n -v
Why these options- -i eth0: interface- -n: numeric addresses/ports (faster, avoids DNS)- -s 0: snaplen = 0 captures full packets (prevents truncation of TCP headers/options and payload useful for retransmit analysis). If storage is tight, use a snaplen large enough to include TCP headers + needed payload (e.g., 262144).- -U: make file writes packet-buffered (helps when rotating / reading simultaneously)- -w /path -C 100: write pcap files, rotate when file reaches 100 MB (-C value is in megabytes). Pair with -W N to limit number of files if you want bounded ring-buffer.- BPF filter 'tcp and host A and host B' restricts to TCP between the two hosts.Limitations detecting retransmits with BPF- BPF filters operate on single packets (no state). Retransmits require sequence-number/time correlation across packets; plain BPF cannot reliably detect retransmits.- Heuristic markers (like duplicate ACKs or identical seq/len/flags) can be matched but are error-prone (reordering, legitimate duplicates, or fragmented captures).- Tools like tcpdump post-processing (when reading pcap with -v/-vv) can annotate retransmissions by tracking sequence numbers, but this is done in user-space after capture, not by the kernel BPF filter.- For live RT detection you need stateful capture/analysis (tcptrace, Wireshark/tshark, custom script) or kernel eBPF programs that maintain state.
Unlock Full Question Bank
Get access to hundreds of Networking Fundamentals and Troubleshooting interview questions and detailed answers.
