Network Troubleshooting and Diagnostics Questions
Systematically diagnosing network problems: a layered troubleshooting methodology, diagnostic tools and commands (ping, traceroute, tcpdump, packet capture), root-cause analysis, and connectivity and user-access issues. Covers isolating faults across the stack, reading packet-level evidence, and driving from symptom to root cause. The diagnostic discipline that spans all networking layers.
Explain how you would use eBPF or other kernel tracing facilities to instrument packet processing on Linux to diagnose a production latency spike. Describe which kernel hooks you'd attach to, how to collect per-socket or per-packet statistics safely, how to minimize overhead when deploying eBPF programs in production, and how to aggregate results across many hosts.
Sample Answer
Direct answer
eBPF (extended Berkeley Packet Filter) lets you attach small, sandboxed programs to specific kernel hooks in the packet-processing path, collecting fine-grained, per-packet or per-socket statistics with far less overhead than tools like tcpdump that copy full packets to userspace, which is exactly what you need to instrument a production latency spike without the instrumentation itself becoming a new source of overhead.
Structured elaboration
- Which kernel hooks to attach to, and why: for a latency spike, relevant attach points include the socket layer (tracing
tcp_sendmsg/tcp_recvmsgor similar kernel functions to time how long data spends at each stage), and XDP (eXpress Data Path, the earliest possible hook, at the network driver level) if you need to observe packets before the kernel's normal networking stack even processes them; choosing the hook depends on WHERE in the stack you suspect the latency is introduced. - Collecting per-socket or per-packet statistics safely: eBPF programs run in a restricted, verified execution environment (the kernel verifier rejects programs that could loop indefinitely or access memory unsafely), so collecting timestamps at entry and exit of a specific kernel function, and computing the delta, is a safe, low-overhead way to build a latency histogram without needing to capture and inspect full packet contents at all.
- Minimizing overhead when deploying in production: use efficient in-kernel data structures (BPF maps, particularly per-CPU maps to avoid lock contention across cores) to aggregate statistics IN THE KERNEL before exposing them to userspace, rather than exporting every individual event; sample rather than trace every single packet if the traffic volume is very high, accepting a statistical rather than exhaustive view in exchange for materially lower overhead.
- Aggregating results across many hosts: since eBPF programs run per-host, aggregating a fleet-wide view requires a collection layer (each host's eBPF program periodically exports its aggregated map contents to a central time-series store) rather than trying to correlate raw eBPF program instances directly across machines.
- Safety considerations specific to production deployment: eBPF programs are verified for safety by the kernel (bounded loops, safe memory access) before being loaded, which is what makes this safer than, say, an arbitrary kernel module, but still requires care: test on a representative but non-critical host first, monitor the eBPF program's OWN CPU/memory overhead (usually small but not zero), and have a clean, tested way to unload the program if it needs to be removed quickly.
Worked example
Attaching a probe to tcp_sendmsg and tcp_recvmsg (userspace-to-kernel and kernel-to-userspace timestamps for socket writes/reads) on a host experiencing sporadic latency spikes reveals a specific pattern: most of the time is being spent between the KERNEL receiving data and the APPLICATION actually calling read() to consume it, rather than in any kernel-level packet processing at all, pointing toward application-level scheduling or backpressure as the actual cause, a finding that would have been much harder to obtain via full packet capture alone, since capture cannot see INTO the kernel's socket-buffer handling and the application's own read timing the way an eBPF probe placed at those specific points can.
Trade-offs & pitfalls
Full packet capture (tcpdump) copies data to userspace for every packet, meaningful overhead at high traffic volumes, and only tells you about NETWORK-visible events, not what happens between the kernel and the application; eBPF's low-overhead, targeted instrumentation is the better tool specifically when you suspect the latency lives in that kernel-to-application boundary, which a network capture alone cannot see into at all. Don't over-instrument, though: attaching probes to many kernel functions simultaneously, or tracing every packet rather than sampling, can itself introduce measurable overhead exactly when you're trying to diagnose a performance problem, undermining the investigation.
You must explain to a junior SRE how to interpret an excerpt from a tcpdump capture showing a TCP three-way handshake followed by a retransmission and an ICMP 'fragmentation needed' message. Provide a stepwise explanation focused on the packet roles, what the ICMP message indicates for MTU and PMTUD, and why retransmission occurred afterwards.
Sample Answer
Direct answer
Walk the junior SRE through the excerpt in the same order the packets actually happened, naming what each one represents and why the NEXT one occurred as a direct consequence, rather than describing all three events (handshake, retransmission, ICMP message) as separate, disconnected facts.
Structured elaboration
- Start with what a healthy handshake looks like, briefly: point out the SYN, SYN-ACK, and ACK in the excerpt and confirm this part completed normally, so the junior engineer has a clear "this part worked fine" anchor before looking at what went wrong afterward.
- Introduce the ICMP message next, and what it's reporting: explain that "fragmentation needed" specifically means a device along the path needed to split this packet to forward it but wasn't allowed to (because the "don't fragment" bit was set), and that the message includes the MTU (Maximum Transmission Unit, the largest packet size that specific link can carry) of the constraining link.
- Connect the ICMP message to the retransmission explicitly: explain that because the original, oversized packet was dropped rather than delivered, the sender's data never arrived, so from the sender's perspective this looks exactly like ordinary packet loss, triggering a normal retransmission, just with a root cause (path MTU) rather than random loss.
- Use a small, concrete analogy if it helps: sending a letter that's too big for a mail slot; the mail carrier can't force it through and it comes back to you, so you have to send it again in a smaller envelope. The retransmission in the capture is the "resending in a smaller envelope" step (technically the sender should now use Path MTU Discovery information to shrink its segment size, not just blindly retry).
- End by naming what to check if this pattern recurs: whether subsequent segments got smaller (a sign PMTUD worked and the sender adapted) or whether the SAME size keeps being retransmitted and dropped repeatedly (a sign the ICMP message isn't getting back to the sender at all, the PMTUD black-hole failure mode covered elsewhere).
Worked example
Walking through the excerpt line by line: "Here's the SYN, here's the SYN-ACK, here's the ACK, so the connection is fully established at this point, that part's healthy. Then we see this ICMP message; that number after 'mtu' is telling us a router along the path can only carry packets up to that size, and this packet was bigger than that with the don't-fragment flag set, so it got dropped rather than split. Because it got dropped, the data inside it never reached the destination, so from the sender's point of view this looks just like ordinary loss, which is why we see a retransmission right afterward, the sender resending because it never got an acknowledgment for that specific data."
Trade-offs & pitfalls
Presenting the three events (handshake, ICMP message, retransmission) as a list of disconnected facts, rather than a causal chain, leaves a junior engineer able to recite what each one is without being able to actually diagnose the NEXT similar capture they see on their own; always connect each event to what it CAUSES, not just what it IS. Avoid introducing jargon (PMTUD, black-hole) before the underlying mechanism is clear; name the concept only after the junior engineer has already understood, in plain terms, what's actually happening.
After a rolling update, many pods cannot resolve service DNS names in Kubernetes. Describe how you would use 'kubectl exec', 'dig', CoreDNS logs, kube-dns configmap, and packet captures to determine if the issue is due to CoreDNS pods crashing, CNI blocking DNS, or upstream resolver failures. Include the sequence of checks and commands.
Sample Answer
Direct answer
Widespread pod DNS-resolution failure after a rolling update, when it's affecting MANY pods across the cluster rather than just one, points at the shared, cluster-wide DNS infrastructure itself (CoreDNS, its ConfigMap, or the CNI's handling of DNS-destined traffic) rather than anything specific to individual pods, so investigate CoreDNS's own health and configuration before looking at any individual pod.
Structured elaboration
- Confirm the scope first: is this every pod, or a specific subset (a specific node, a specific namespace)? Start with the assumption of a shared cause given the "many pods" framing, but confirm it's genuinely widespread.
- Check CoreDNS pod health directly:
kubectl get pods -n kube-system -l k8s-app=kube-dnsto see if CoreDNS pods are crash-looping, pending, or otherwise unhealthy after the rolling update. - Check CoreDNS logs:
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100for repeated upstream-resolver timeouts, configuration-reload failures, or crash-loop reasons. - Check the kube-dns/CoreDNS ConfigMap directly:
kubectl get configmap coredns -n kube-system -o yaml(orkube-dnson older clusters) to inspect the actual Corefile in effect; a rolling update to the cluster or to CoreDNS itself can silently ship a ConfigMap change (a badforwardupstream address, a droppedkubernetesplugin zone, a strictercache/errorssetting) that CoreDNS picks up on its next restart, which is exactly what a rolling update triggers; diff this against the last known-good ConfigMap (kubectl rollout historywon't cover a ConfigMap, so keep it in version control or checkkubectl get configmap coredns -n kube-system -o yaml --exporthistory via your GitOps source) to catch this specific class of cause. - Test resolution directly from inside an affected pod:
kubectl exec -it <pod> -- dig @<coredns-clusterip> kubernetes.default.svc.cluster.local(bypassing the pod's own/etc/resolv.confand hitting the cluster DNS service IP directly) to confirm whether CoreDNS itself is unreachable or unresponsive, versus reachable but returning errors (a ConfigMap or upstream-forwarding problem), versus the pod's own/etc/resolv.confbeing wrong. - Capture traffic to see the query actually leave and (not) return:
kubectl exec -it <pod> -- tcpdump -n -i eth0 udp port 53run in parallel with thedigcommand above shows directly whether the DNS query packet leaves the pod's network namespace at all and whether any response comes back; if the query never leaves, suspect the CNI or the pod's own network namespace; if it leaves but nothing returns, suspect the CNI blocking return traffic, an intervening NetworkPolicy, or CoreDNS itself not responding. A parallel capture on a CoreDNS pod itself (kubectl exec -it <coredns-pod> -n kube-system -- tcpdump -n -i eth0 udp port 53) shows whether the query even arrives at CoreDNS, directly separating "never reached CoreDNS" from "CoreDNS received it and failed to answer." - Check the CNI's effect on DNS traffic specifically: if CoreDNS itself looks healthy and its ConfigMap is unchanged, but the capture shows queries not arriving at CoreDNS, confirm a NetworkPolicy or CNI-level rule introduced by the rolling update isn't selectively blocking traffic to the DNS Service's ClusterIP on UDP/TCP port 53.
- Check whether upstream resolvers are reachable from CoreDNS's own pods, since CoreDNS forwards external queries upstream per its ConfigMap's
forwarddirective; if the rolling update affected node-level networking CoreDNS itself depends on, in-cluster name resolution might work while external-name resolution through CoreDNS fails.
Worked example
kubectl get pods -n kube-system shows CoreDNS pods running and healthy (not crash-looping). kubectl exec -it <pod> -- dig @10.96.0.10 kubernetes.default.svc.cluster.local times out with no answer. kubectl exec -it <pod> -- tcpdump -n udp port 53 during the same dig shows the query leaving the pod's namespace but no response ever returning. A capture on a CoreDNS pod itself shows the query never arrives there at all. kubectl get configmap coredns -n kube-system -o yaml shows the Corefile is unchanged from the pre-update version, ruling out a config regression; the rolling update instead changed a NetworkPolicy's pod selector in a way that now also matches the CoreDNS pods, blocking ingress UDP/53 to them. Removing CoreDNS from that policy's selector immediately resolves DNS resolution across every affected pod.
Trade-offs & pitfalls
Investigating individual pods' /etc/resolv.conf one at a time, when the actual failure is a widespread, shared-infrastructure issue, wastes time; the "many pods" framing is itself the strongest signal pointing at CoreDNS's own health and configuration, checked first. Skipping the ConfigMap diff specifically is a common miss, since a rolling update to the DNS deployment (or to anything that triggers a CoreDNS restart and Corefile reload) can carry forward a subtly broken configuration that looks, from kubectl get pods, like everything is fine. Also don't skip the packet capture step: it's the only evidence that definitively separates "query never left the pod" from "query left but CoreDNS never got it" from "CoreDNS got it but didn't answer," three genuinely different root causes that a ConfigMap read and a health check alone can't distinguish.
Explain how NAT gateway or firewall connection-tracking table exhaustion can cause intermittent connection failures at scale. Describe how you would detect conntrack exhaustion, what metrics to monitor, and how you'd decide between raising the table's capacity versus addressing what's actually driving the churn. Include example commands to view the conntrack table on Linux-based appliances.
Sample Answer
Direct answer
A NAT gateway or firewall's connection-tracking (conntrack) table has a finite size; once it fills up, new connections can't be tracked and are dropped or fail intermittently, and this failure mode is distinctive because it appears only under LOAD (a specific connection count or churn rate), not as a constant, deterministic failure.
Structured elaboration
- Understand what conntrack tracks and why it can exhaust: every distinct connection passing through a stateful NAT or firewall device consumes an entry in its connection-tracking table until that connection closes (or its entry times out); a very large number of concurrent connections, or a high RATE of short-lived connections that don't clean up entries fast enough, can exhaust the table's capacity even if the device's raw throughput is nowhere near its limit.
- Detect conntrack exhaustion specifically: on Linux-based appliances,
conntrack -C(or reading/proc/sys/net/netfilter/nf_conntrack_countagainstnf_conntrack_max) shows current usage against the configured maximum; a count consistently near or at the maximum, correlated with the timing of reported failures, is the direct signature. - Monitor the right metric proactively: track conntrack table utilization as a percentage of its maximum, not just raw connection counts, since the maximum itself may have been sized for a different traffic pattern than what's actually occurring now.
- Remediation options, each with real trade-offs: increasing
nf_conntrack_maxraises the ceiling directly but consumes more memory and doesn't address WHY the table is filling (a genuine traffic-pattern change, or a bug causing connections to accumulate without properly closing); tuning conntrack timeouts more aggressively (reclaiming entries for idle or half-closed connections sooner) helps if many entries are lingering longer than they need to; deploying additional NAT gateways to spread the load reduces per-device table pressure but adds complexity and cost; and stateless NAT (where architecturally feasible) avoids the conntrack limit entirely for the traffic it handles, at the cost of losing some of the visibility and flexibility a stateful table provides. - Consider port-preservation strategies: some NAT implementations can be tuned to reuse or preserve source ports more efficiently, reducing table churn for high-connection-rate workloads without simply raising the ceiling.
Worked example
conntrack -C shows a count sitting at 98% of nf_conntrack_max during the exact windows when clients report intermittent connection failures, and comfortably lower outside those windows. Correlating this against traffic patterns shows a recent change introduced a much higher rate of short-lived connections (a client library switched from connection pooling to opening a fresh connection per request), each consuming a conntrack entry briefly but at a volume the existing table sizing didn't anticipate. Raising nf_conntrack_max provides immediate relief, while the client library's connection-pooling regression is flagged as the actual root cause worth fixing at the source.
Trade-offs & pitfalls
Simply raising the conntrack table's maximum size treats the symptom, not necessarily the cause; if a client-side regression (like the pooling example above) is generating far more connections than intended, raising the ceiling delays the next exhaustion event rather than preventing it. Always correlate exhaustion against what changed in the actual traffic pattern, not just the device's own capacity, before deciding whether to fix the ceiling or the traffic generating the load.
In a cloud VPC an instance cannot reach a public API. Describe how to use traceroute, VPC flow logs (or equivalent), security group/NSG rules, and route tables to determine whether packets are dropped on the host, at the VPC edge, or by cloud networking. Include the exact checks and commands/logs you would use on the instance and in the cloud console.
Sample Answer
Direct answer
A cloud VPC instance that can't reach a public API is almost always blocked by one of four things, checked in order of how cheap they are to verify: the security group attached to the instance, the network ACL on its subnet, the route table's path to an internet gateway or NAT gateway, and the instance's own OS-level firewall or DNS configuration.
Structured elaboration
- Security group (stateful): confirm outbound rules allow the destination port (and that inbound rules allow the return traffic if the security group has been made unusually restrictive; by default security groups are stateful so return traffic is normally automatic, but an explicit deny or a non-default configuration can break that assumption).
- Network ACL (stateless): unlike security groups, NACLs are stateless, so you must explicitly allow both the outbound request AND the inbound return traffic on the ephemeral port range. A very common self-inflicted failure is an NACL that allows outbound 443 but forgets the inbound ephemeral-port return rule.
- Route table: confirm the subnet's route table has a default route (0.0.0.0/0) pointing at an internet gateway (for a public subnet) or a NAT gateway (for a private subnet reaching the internet). A route table associated with the wrong subnet after a recent change is a common cause that looks identical to a security-group problem from the instance's perspective.
- From inside the instance:
traceroute(or the cloud-native equivalent) to see how far packets get before failing, and VPC flow logs (or the equivalent) to see whether traffic is being ACCEPTed or REJECTed and at which stage, which tells you directly whether the block is a security group, NACL, or something upstream.
Worked example
traceroute 8.8.8.8 from the instance times out at the first hop (the instance's own default gateway inside the VPC), while a security group check shows outbound 443 explicitly allowed. Flow logs for that ENI show entries marked REJECT for the outbound attempt. Since the security group allows it, the rejection is happening at the NACL or route-table layer; checking the subnet's route table reveals it lacks a default route to any gateway; a change reassociated this subnet with a route table meant for a fully private/isolated tier. Attaching the correct route table (with a route to the NAT gateway) restores connectivity without touching the security group at all.
Trade-offs & pitfalls
The most common mistake is only checking the security group, since it's the most visible and most commonly discussed control, and stopping there when the block is actually at the NACL or routing layer. A cloud provider's reachability-analysis tool (where available) can trace this whole chain automatically and is worth reaching for before manual traceroutes on a complex topology; treat manual traceroute and flow-log inspection as the fallback when that tooling isn't available or doesn't cover the specific path in question.
Unlock Full Question Bank
Get access to all Network Troubleshooting and Diagnostics interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.