Network Protocol Internals and Edge Cases Questions
Deep understanding of how protocols work internally (TCP congestion control, IP fragmentation, DNS resolution under failure, IPv6 transition), edge cases that cause problems, and protocol troubleshooting.
EasyTechnical
25 practiced
Explain TCP window scaling: why the 16-bit window field became insufficient, how the scaling factor is negotiated via options during handshake, and operational implications for high bandwidth-delay product links. As an SRE, what misconfigurations lead to throughput problems?
Sample Answer
TCP originally used a 16-bit window field in the TCP header, so the maximum advertised window was 65,535 bytes. On high bandwidth-delay product (BDP) links (e.g., long fat networks), that limit prevents the sender from keeping the pipe full, capping throughput = window / RTT. Window scaling (RFC 7323) extends the effective window by applying a left-shift (scale) to the 16-bit value, multiplying it by 2^S where S is a 0–14 shift, so the receive window can be >>64KB.The scale factor is negotiated during the three-way handshake using the TCP Window Scale option in SYN and SYN-ACK packets. Each side advertises its own shift count; the agreed scale is the value the receiver of data advertised. The option is only valid if present in SYN; if absent, no scaling is used. Because it’s an option, middleboxes that drop unknown options can break scaling.Operational implications for high BDP links:- Without scaling throughput is limited. Example: 100 Mbps link with 100 ms RTT needs ~1.25 MB window (~10 Mb) — far above 64KB.- Requires correct receiver buffer sizing and autoscaling in OS (sysctl like net.ipv4.tcp_rmem/tcp_wmem on Linux).- Path MTU, delayed ACKs, congestion control and packet loss interact with achievable throughput.Common misconfigurations SREs should watch for:- Disabled window scaling (kernel or firewall stripping TCP options) → throughput cap.- Small socket buffer limits (tcp_rmem/tcp_wmem too low) preventing kernel from using advertised window.- Middleboxes (NATs, load balancers, DPI) that remove or alter TCP options.- Mismatched or static application-level limits (e.g., app sets SO_RCVBUF too small).- Mis-tuned congestion control or aggressive pacing that reduces effective throughput.- Lack of monitoring: not tracking RTT, retransmits, zero-window or window-update stalls.Detection & remediation tips:- Capture SYN/SYN-ACK to verify window scale option.- Monitor throughput vs. BDP (calc required window = bandwidth * RTT).- Check /proc/sys/net/ipv4/tcp_* buffer settings and increase safely; ensure middleboxes pass TCP options.
EasyTechnical
21 practiced
Explain the TCP three-way handshake (SYN, SYN-ACK, ACK) and describe how a SYN flood attack exploits this process. As an SRE responsible for production mitigation, list short-term mitigations (while under attack) and longer-term architectural changes you would implement to prevent recurrence.
Sample Answer
TCP three-way handshake:- Client sends SYN (synchronize) with initial sequence number to request a connection.- Server replies with SYN-ACK: acknowledges client’s SYN and sends its own SYN with server sequence number.- Client sends final ACK acknowledging server’s SYN; connection is established and state moves from half-open to established.How SYN flood exploits this:- Attacker sends many SYNs with forged/unused source IPs. Server allocates a half-open connection entry and waits for the final ACK. With enough SYNs, the server’s backlog fills, preventing legitimate clients from connecting (resource exhaustion).Short-term mitigations (while under attack):- Increase SYN backlog and reduce TCP SYN timeout (temporarily).- Enable SYN cookies (statelessly validate the final ACK).- Rate-limit SYN packets via host firewall (iptables nf_conntrack limits, nftables) or edge router ACLs.- Deploy upstream DDoS scrubbing (cloud provider / ISP) to filter attack traffic.- Apply connection tracking / TCP stack tuning (net.ipv4.tcp_max_syn_backlog, tcp_synack_retries).Longer-term architectural changes:- Use load balancers / anycast + global traffic distribution to absorb attacks.- Move to managed DDoS protection / scrubbing services and IP anycast for edge filtering.- Implement autoscaling for front-end capacity combined with SYN-proxying on reverse proxies (nginx, HAProxy) or dedicated SYN-proxy appliances.- Enforce TLS (offload) and stronger authentication earlier in the flow to reduce impact of unauthenticated halves.- Improve monitoring/alerting for SYN rates and automate mitigations (auto-apply rate-limits, failover to scrubbing).- Regularly run capacity and chaos tests to validate defenses and incident runbooks.
MediumTechnical
20 practiced
In a Kubernetes cluster, pods intermittently experience connection resets when calling an external API. Describe how you would triage across layers: pod network namespace, CNI plugin, kube-proxy/iptables or IPVS, node networking, and cloud VPC. Include commands and observability checkpoints you would use.
Sample Answer
Start by reproducing the problem and collecting context (pod names, node, timestamps, examples of failing requests). Then triage top-down through layers, collecting evidence and isolating scope.1) Pod network namespace (app-level + netns)- Check app logs, request timestamps, and error type (RST vs timeout).- From the pod:- Observability: pod-level request rate, error rate, latencies (Prometheus: http_request_duration_seconds, http_errors_total). Look for local RSTs or FINs.2) CNI plugin (veth, iptables rules CNI inserted)- On node, map pod to host veth pair:- Check CNI logs (/var/log/ or systemd for plugin: kubelet/journalctl -u kubelet) and plugin-specific logs (calicoctl, flannel logs).- Observability: CNI metrics (calico/node, flannel metrics), dropped packets, policy (NetworkPolicy) denial counts.3) kube-proxy / iptables or IPVS- Check mode and rules:- Verify conntrack entries and timeouts:- Observability: kube-proxy metrics (sync errors), conntrack table fullness, NAT translation counts.4) Node networking / kernel- Node-level packet drops, NIC errors, TC shaping:- Check local firewall (iptables FORWARD rules), resource pressure (CPU, memory), and ephemeral port exhaustion:- Observability: node network errors, CPU spikes, kernel logs correlating to timestamps.5) Cloud VPC / external path- Validate security groups, NACLs, route tables, and external API health:- Inspect VPC Flow Logs for RST flags, denied traffic, or unusual retransmits over time window. Run traceroute/mtr from node to API:- Observability: cloud metrics (NSG drops, LB logs), API provider status, regional network events.Cross-layer checks & correlation- Correlate timestamps across pod logs, node journal, CNI and kube-proxy logs, VPC Flow Logs and external API logs.- Look for patterns: specific nodes, certain times, connection reuse vs new connections (keepalive), source port reuse, SYN->RST vs established RST.Short-term mitigations- If conntrack exhaustion: increase nf_conntrack_max and tune timeouts or enable conntrack hashing; restart offending pod/node after safe maintenance.- If CNI policy misconfiguration: temporarily relax NetworkPolicy to confirm.- If cloud-side resets: whitelist IPs, enable longer timeouts, or open TCP ephemeral port ranges.Key observability checkpoints to report in incident:- Pod socket states and tcpdump showing RSTs- CNI plugin logs & metrics- kube-proxy mode and conntrack usage- Node dmesg, NIC errors, TC drops- VPC Flow Logs and cloud firewall deny counts- External API logs/healthThis structured approach isolates whether resets originate in the pod, host, cluster networking plane, or upstream cloud path and yields concrete next steps.
bash
kubectl exec -n <ns> <pod> -- /bin/sh -c "env; date; curl -vS --max-time 10 https://api.example.com/"
kubectl exec -n <ns> <pod> -- ss -tpan # active TCP sockets and states
kubectl exec -n <ns> <pod> -- ip addr; ip route
kubectl exec -n <ns> <pod> -- tcpdump -i eth0 -w /tmp/pod.pcap host api.example.com and tcpbash
kubectl get pod -o wide -n <ns>
sudo ip link show # find veth peer names
sudo ethtool -S <veth> # if supported
sudo tcpdump -i <veth> -w /tmp/node_veth.pcap host api.example.combash
kubectl -n kube-system get daemonset kube-proxy -o yaml
sudo ps aux | grep kube-proxy
sudo iptables -t nat -L -n -v
sudo ipvsadm -L -n # if ipvs mode
kubectl logs -n kube-system <kube-proxy-pod>bash
sudo conntrack -L | grep <pod-ip> | wc -l
sysctl net.netfilter.nf_conntrack_max
sysctl net.netfilter.nf_conntrack_tcp_timeout_establishedbash
sudo dmesg | tail -n 200
ip -s link show <iface>
sudo ethtool -S <iface>
sudo tc qdisc show dev <iface>
sudo sar -n DEV 1 1bash
ss -s
cat /proc/sys/net/ipv4/ip_local_port_rangebash
aws ec2 describe-security-groups --group-ids ...
aws ec2 describe-network-acls --filters ...
# or equivalent gcloud/az CLIbash
kubectl exec -it -n <ns> <pod> -- traceroute api.example.com
kubectl exec -it -n <ns> <pod> -- mtr -rw api.example.comEasyTechnical
22 practiced
Explain the difference between TCP flow control and TCP congestion control. Give concrete examples of mechanisms used by each (for example receive window vs slow start) and describe a real cloud scenario where confusing the two could lead to an incorrect mitigation.
Sample Answer
TCP flow control vs TCP congestion control — short, practical distinction:Flow control:- Purpose: protect the receiver from being overwhelmed by the sender.- Mechanism: receiver advertises a receive window (rwnd) in TCP headers; sender limits in-flight bytes to min(rwnd, cwnd).- Example: receiver with a slow application or small buffer reduces rwnd → sender slows, but network is fine.Congestion control:- Purpose: avoid overloading the network (links/routers) and reduce packet loss/latency.- Mechanisms: slow start, congestion avoidance (AIMD), fast retransmit/fast recovery, and modern variants like CUBIC, BBR.- Example: on packet loss or ECN marks, sender reduces congestion window (cwnd) and ramps up slowly.Real cloud SRE scenario where confusing them causes wrong mitigation:- Symptom: many retransmits and high latency to a service. An engineer assumes “network congestion” and scales up instance NICs or adds more load balancer capacity. But root cause is a bug in the consumer app reading slowly, causing rwnd=0 and stalling senders. Scaling network capacity doesn't help; correct mitigation is fixing the slow consumer (increase socket buffers, optimize processing, or backpressure through application-level flow control). Conversely, treating true network congestion as receiver flow control and only increasing socket buffers could worsen packet loss and congestion collapse.Key takeaway: rwnd relates to receiver capacity; cwnd/slow-start relate to network capacity. Diagnose by inspecting rwnd, cwnd, retransmit patterns, ECN, and host CPU/I/O before choosing fixes.
MediumTechnical
26 practiced
Write a Python script (using scapy or pyshark) that parses a pcap file and outputs a summary of fragmented IPv4 packets: source, destination, identification, fragment offsets, and whether all fragments were observed. The script should handle large pcaps efficiently (streaming) and report incomplete reassemblies with timestamps.
Sample Answer
To detect and summarize IPv4 fragment sets efficiently, use scapy.PcapReader to stream packets, index fragment records by (src,dst,proto,identification), accumulate fragment offsets and timestamps, and mark completeness when a fragment with MF flag 0 arrives and offsets cover the range. The script below is memory-conscious (periodically times out old incomplete sets) and prints per-flow summaries including whether all fragments were seen and the first/last timestamps.Key points:- Streams using PcapReader to handle large pcaps without loading all into memory.- Uses (src,dst,proto,id) to correlate fragments.- Tracks offsets and MF flag; reports incomplete sets after timeout with timestamps.Complexity: O(n) time (single pass), O(f) memory where f = concurrent fragmented flows.Edge cases: overlapping/duplicate fragments, varying fragment lengths (more precise reassembly would track byte ranges), IPv6 fragmentation different handling. Alternative: use pyshark/tshark for higher-level reassembly but less control and heavier resources.
python
import time
from scapy.all import PcapReader, IP
from collections import defaultdict
PCAP_FILE = "capture.pcap"
# timeout in seconds to report incomplete reassemblies if no new fragment seen
INCOMPLETE_TIMEOUT = 60
def summarize_fragments(pcap_file):
sets = {} # key -> { offsets:set, seen_last:bool, first_ts, last_ts, tot_bytes_est }
def key(pkt):
ip = pkt[IP]
return (ip.src, ip.dst, ip.proto, ip.id)
def report(k, meta, complete):
src,dst,proto,ident = k
offsets = sorted(meta['offsets'])
print(f"{meta['first_ts']:.6f} - {meta['last_ts']:.6f} | {src}->{dst} id={ident} proto={proto} offsets={offsets} complete={complete}")
now = time.time()
with PcapReader(pcap_file) as pcap:
for pkt in pcap:
if IP not in pkt:
continue
ip = pkt[IP]
k = (ip.src, ip.dst, ip.proto, ip.id)
ts = float(pkt.time)
off = ip.frag # fragment offset in 8-byte units
mf = bool(ip.flags.MF)
m = sets.get(k)
if not m:
m = {'offsets': set(), 'seen_last': False, 'first_ts': ts, 'last_ts': ts, 'last_seen': ts}
sets[k] = m
m['offsets'].add(off)
if not mf:
m['seen_last'] = True
m['last_fragment_offset'] = off
m['last_ts'] = ts
m['last_seen'] = ts
# quick completeness heuristic: if seen_last and offsets include 0..last_offset contiguous
if m.get('seen_last'):
last_off = m['last_fragment_offset']
expected = set(range(0, last_off + 1)) # offsets are in units; assumes all fragments present
# note: in practice lengths matter; we use offset units contiguous check
if expected.issubset(m['offsets']):
report(k, m, True)
del sets[k]
# periodic cleanup every 1000 packets could be added; simple time-based cleanup:
now = ts
expired = [kk for kk,v in sets.items() if now - v['last_seen'] > INCOMPLETE_TIMEOUT]
for kk in expired:
report(kk, sets[kk], False)
del sets[kk]
# final report for leftover sets
for k,m in sets.items():
report(k,m, False)
if __name__ == "__main__":
summarize_fragments(PCAP_FILE)Unlock Full Question Bank
Get access to hundreds of Network Protocol Internals and Edge Cases interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.