Network Monitoring and Performance Questions
Network telemetry and performance operations: SNMP polling and traps (including counter wraparound and SNMPv3 access), NetFlow, sFlow and IPFIX flow export, sampling and its accuracy, streaming telemetry (gNMI), and eBPF or packet-capture telemetry; interface-level metrics (utilization, errors, discards, queue depth, microbursts), active synthetic probing alongside passive counters, link-flap detection, baselining and anomaly detection on network signals including elephant-flow spotting, network SLIs and SLOs, alerting, alert-storm suppression and NOC dashboards, telemetry pipeline design, storage, retention, downsampling and cardinality for network data (including securing the collection path and handling bursty remote sites), BGP and link-state monitoring including prefix hijack and route-leak detection, and network capacity monitoring, percentile utilization and bandwidth headroom planning. Also covers measuring and tuning network-level latency, jitter, packet loss and throughput (bufferbloat, queueing, TCP tuning for long paths). Excludes the generic metrics, logs and traces stack and alert design, the layered fault-isolation method and packet-capture troubleshooting, TCP and protocol fundamentals, application and CDN latency engineering, cloud VPC design and security detection.
How would you monitor network connectivity for a microservices platform on Kubernetes? Which signals would you collect at pod and node level, and how would you detect policy or datapath problems?
Sample Answer
Direct answer
Monitor in layers and let each layer name a different failure: pod and node signals (is the datapath on this host healthy), cluster services (DNS and service routing), and active probes (does pod A reach service B across nodes, now). Detect policy problems by looking at the verdict the enforcing network plugin gave each flow, not by guessing from application errors. The Kubernetes NetworkPolicy resource is only enforced by the network plugin (the CNI, Container Network Interface), so the plugin's flow and drop visibility is your best source.
Signals to collect
| Layer | Signal | Source | What it tells you |
|---|---|---|---|
| Pod | Flow verdicts (forwarded vs dropped) and drop reasons | Hubble (Cilium's flow-observability component) metrics hubble_flows_processed_total (label verdict) and hubble_drop_total (labels reason, protocol), when Cilium is the plugin | Policy denials versus other drops |
| Pod | TCP flag counts, for example SYN (the first packet of a TCP handshake) with no matching reply | hubble_tcp_flags_total | Connections that start and never complete |
| Pod | DNS queries and responses by response code | hubble_dns_responses_total (label rcode) | Lookups failing inside the cluster |
| Cluster | DNS request duration and response codes | CoreDNS (the cluster's DNS server) coredns_dns_request_duration_seconds and coredns_dns_responses_total{rcode}, served on port 9153 at /metrics | Slow or failing resolution (a rising SERVFAIL rate, the DNS server failed to answer, or NXDOMAIN rate, the name does not exist) |
| Node | Interface counters, TCP statistics, connection-tracking (conntrack, the kernel's per-connection state table) stats | node_exporter collectors netdev, netstat, conntrack (enabled by default) | NIC errors and drops, retransmits, conntrack pressure |
| Path | Synthetic probes between pods, across nodes and to external dependencies | blackbox_exporter (probes HTTP, HTTPS, DNS, TCP, ICMP, gRPC) | End-to-end reachability and latency now, independent of real traffic |
| Service | Request success rate and latency from the application or mesh | App or proxy metrics | Whether users are affected |
Read the table top to bottom as the order of suspicion in the next section: pod verdicts and DNS first, then node counters, then path probes. The Hubble metrics are disabled by default and are switched on in the Helm values with hubble.metrics.enabled, for example {drop,flow,tcp,dns}.
Detecting policy and datapath problems
- Is it policy? Remember the semantics: with no NetworkPolicy selecting a pod, all inbound and outbound connections are allowed. Once a policy selects it, only what a policy allows is permitted. So a service that broke right after a policy rollout is a suspect. With Cilium, list dropped flows for the pod:
hubble observe --pod <pod> --verdict DROPPED. The output shows source and destination namespace/pod:port and the verdict, so you can see the exact flow that a missing rule blocked. The lines look like this (pod names are illustrative; the layout follows the Cilium documentation):
May 4 13:23:47.852: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
May 4 13:23:48.854: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
Reading it: a timestamp, then source namespace/pod:port, the arrow, destination namespace/pod:port, the reason (Policy denied), the verdict DROPPED, and the TCP flags. A SYN dropped by policy and then sent again about a second later is a client retrying a connection that a rule never lets through, so the fix is the rule, not the application.
2. Is the policy even enforced? A NetworkPolicy created on a cluster whose plugin does not support it has no effect. Test with a deliberate deny rule in a test namespace and confirm the probe fails.
3. Is it DNS? Check the CoreDNS rcode counters and the request duration histogram first, because many "network" outages are failed lookups.
4. Is it the node datapath? Look at node interface drops and errors, and conntrack usage against its limit; a full conntrack table drops new connections on that node only.
5. Is it cross-node? Compare probe results same-node versus cross-node. If only cross-node fails, suspect the overlay (the tunnel the plugin builds to carry pod traffic between nodes) or the underlay (the physical or cloud network beneath it): a tunnel adds header bytes, so a full-size packet may no longer fit the MTU (maximum transmission unit, the largest packet a link carries) and is dropped, while small requests still pass; a firewall between nodes causes the same cross-node-only symptom.
Alerts
Alert on symptoms with rate and ratio: drop rate by reason rising, DNS error ratio, probe success under its target, and conntrack above a set fraction of its maximum. Page on service-level symptoms; keep drop counters as the diagnostic dashboard.
Pitfalls
- Per-pod metric labels that grow without bound as pods churn.
- Watching only application error rates, which cannot separate DNS, policy and datapath causes.
- Trusting that a policy works because it applied without an error.
Write a Python script using only the standard library that measures TCP connect latency to a list of endpoints concurrently and reports p50, p95 and p99 of successful connects. Handle timeouts and failures sensibly.
Sample Answer
Direct answer
Run every attempt as an asyncio task, cap how many run at once with a semaphore, time only the connection setup with time.perf_counter() (a high-resolution timer for short durations; only the difference between two calls means anything) around asyncio.open_connection, wrap it in a timeout, classify each failure (timeout, refused, DNS or other OS error), and compute percentiles over successful connects only while reporting the failure rate beside them. A p99 computed after silently dropping failures flatters the endpoint, so failures are always reported next to the percentiles.
The script
#!/usr/bin/env python3
"""Concurrent TCP connect-latency probe (standard library only)."""
import argparse
import asyncio
import math
import sys
import time
from collections import Counter
def percentile(sorted_vals, p):
"""Nearest-rank percentile: the smallest value with at least p% of samples at or below it."""
if not sorted_vals:
return None
rank = max(1, math.ceil(p / 100 * len(sorted_vals)))
return sorted_vals[rank - 1]
def parse_endpoint(text):
host, sep, port = text.strip().rpartition(":")
if not sep or not port.isdigit():
raise ValueError(f"bad endpoint {text!r}, expected host:port or [v6]:port")
return host.strip("[]"), int(port)
async def probe(host, port, timeout, payload, sem):
"""One attempt. Returns (kind, connect_ms, mbps); kind is 'ok' or a failure class."""
async with sem:
t0 = time.perf_counter()
try:
reader, writer = await asyncio.wait_for(
asyncio.open_connection(host, port), timeout)
except asyncio.TimeoutError:
return "timeout", None, None
except ConnectionRefusedError:
return "refused", None, None
except OSError as exc: # DNS failure, unreachable, reset, ...
return type(exc).__name__, None, None
connect_ms = (time.perf_counter() - t0) * 1000
mbps = None
try:
if payload:
t1 = time.perf_counter()
writer.write(b"x" * payload)
await asyncio.wait_for(writer.drain(), timeout)
await asyncio.wait_for(reader.readexactly(payload), timeout)
secs = time.perf_counter() - t1
mbps = (2 * payload * 8) / secs / 1e6 # bytes sent plus echoed back
except (asyncio.TimeoutError, asyncio.IncompleteReadError, OSError):
pass # connect succeeded, so keep its latency; throughput stays None
finally:
writer.close()
try:
await asyncio.wait_for(writer.wait_closed(), 1)
except (asyncio.TimeoutError, OSError):
pass
return "ok", connect_ms, mbps
async def run(endpoints, attempts, timeout, concurrency, payload):
sem = asyncio.Semaphore(concurrency)
plan = [ep for ep in endpoints for _ in range(attempts)]
done = await asyncio.gather(*(probe(h, p, timeout, payload, sem) for h, p in plan))
results = {ep: [] for ep in endpoints}
for ep, row in zip(plan, done):
results[ep].append(row)
return results
def report(results, payload):
print(f"{'endpoint':24} {'n':>4} {'ok':>4} {'fail%':>6} {'p50':>8} {'p95':>8} {'p99':>8} failures"
+ (" mbps_p50" if payload else ""))
for (host, port), rows in results.items():
lat = sorted(r[1] for r in rows if r[0] == "ok")
fails = Counter(r[0] for r in rows if r[0] != "ok")
pct = lambda p: "-" if not lat else f"{percentile(lat, p):.1f}"
line = (f"{host + ':' + str(port):24} {len(rows):>4} {len(lat):>4} "
f"{100 * (len(rows) - len(lat)) / len(rows):>6.1f} "
f"{pct(50):>8} {pct(95):>8} {pct(99):>8} {dict(fails) or '-'}")
if payload:
mb = sorted(r[2] for r in rows if r[2] is not None)
line += f" {percentile(mb, 50):.1f}" if mb else " -"
print(line)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("endpoints", nargs="+", help="host:port or [v6]:port")
ap.add_argument("-n", "--attempts", type=int, default=50)
ap.add_argument("-t", "--timeout", type=float, default=2.0)
ap.add_argument("-c", "--concurrency", type=int, default=20)
ap.add_argument("--payload-bytes", type=int, default=0,
help="also send this many bytes to an echo service and time the round trip")
a = ap.parse_args()
try:
eps = [parse_endpoint(e) for e in a.endpoints]
except ValueError as exc:
sys.exit(str(exc))
results = asyncio.run(run(eps, a.attempts, a.timeout, a.concurrency, a.payload_bytes))
report(results, a.payload_bytes)
if __name__ == "__main__":
main()
Usage: python3 tcpprobe.py 192.0.2.10:443 [2001:db8::1]:443 -n 100 -c 20 -t 2. Add --payload-bytes 65536 against an echo service (a server that sends back whatever it receives) to get the per-connection round-trip throughput the same way.
What the script prints
#!/usr/bin/env python3
"""Minimal TCP echo server on 127.0.0.1:9001 for testing tcpprobe.py."""
import asyncio
async def handle(reader, writer):
try:
while data := await reader.read(65536):
writer.write(data)
await writer.drain()
except OSError:
pass
writer.close()
async def main():
server = await asyncio.start_server(handle, "127.0.0.1", 9001)
async with server:
await server.serve_forever()
asyncio.run(main())
Save the script above as tcpprobe.py and this one as echo.py. Start the echo server with python3 echo.py &, then run python3 tcpprobe.py 127.0.0.1:9001 127.0.0.1:9002 -n 100; nothing listens on port 9002, so every attempt there is refused. Add --payload-bytes 65536 to the same command to exercise the throughput column.
The output of that command (loopback, so the times are tiny and will differ on every run; the values below are illustrative):
endpoint n ok fail% p50 p95 p99 failures
127.0.0.1:9001 100 100 0.0 0.5 0.8 1.2 -
127.0.0.1:9002 100 0 100.0 - - - {'refused': 100}
Reading the columns: n is attempts, ok is successful connects, fail% is (n minus ok) divided by n, p50, p95 and p99 are connect times in milliseconds over the successful attempts only, and failures counts each failure class. The second row shows why ok and fail% sit beside the percentiles: with no successes there is nothing to take a percentile of, so the cells show - and the failure column carries the story. With --payload-bytes 65536 one more column, mbps_p50, shows the median round-trip throughput.
How the asyncio pieces fit
asyncio runs many tasks in one thread: a task runs until it hits an await, then hands control to another, so hundreds of connection attempts overlap without threads.
async with semtakes one permit from the semaphore (a counter of free slots, here-c); a task that finds none waits, and the permit is returned automatically when the block ends, even after an error.asyncio.wait_for(x, timeout)cancelsxand raisesasyncio.TimeoutErrorif it has not finished in time; that is how a silent host becomes atimeoutrow instead of hanging the run.writer.drain()waits until the data handed towritehas been pushed out to the network, andreader.readexactly(n)waits for exactly n bytes back (and raisesIncompleteReadErrorif the connection closes first).asyncio.gather(...)starts all the probe tasks and returns their results in the order given, which is what letszip(plan, done)pair each result with its endpoint.
Why it is built this way
- What is timed.
open_connectionreturns when the TCP three-way handshake (client sends SYN, server answers SYN-ACK, client sends ACK) completes, so the figure is roughly one network round trip plus local overhead. If the host is a name, DNS resolution is inside that time. Probe addresses, or accept that resolution is part of the figure. - Concurrency is capped. Without the semaphore, hundreds of simultaneous attempts queue inside the client and inflate the numbers it is trying to measure.
-cbounds the client-side load; the tasks for all endpoints share it. - Failures are classified, not merged.
refusedmeans the host answered with a reset (path works, service down).timeoutmeans no answer within the limit (filtered or lost).gaierroris a DNS failure. They have different owners. - Percentiles use the nearest-rank rule (rank = ceil(p/100 x n)). With few successes p99 equals the maximum, which is why
nandokare printed. - Throughput is optional and honest. It needs an echo service to read the bytes back, covers the bytes in both directions, and a failure there keeps the connect sample.
- Cleanup is explicit.
writer.close()runs in afinally, so a failed read does not leak sockets.
Self-check of the percentile function
from tcpprobe import percentile
vals = sorted([12, 15, 11, 14, 13, 90, 12, 16, 11, 250])
print(vals)
for p in (50, 95, 99):
print(p, percentile(vals, p))
print(percentile([], 50))
Output when run:
[11, 11, 12, 12, 13, 14, 15, 16, 90, 250]
50 13
95 250
99 250
None
With 10 samples the 5th value (13) is the p50, and the p95 and p99 ranks both round up to 10, the maximum. That is the arithmetic by hand: ceil(0.95 x 10) = 10.
Pitfalls
- Probe from where users are; loopback or same-rack numbers say nothing about the WAN.
- Alert on a sustained change in p95 or failure rate over several runs, never on one sample. Percentiles of 20 attempts are noisy.
- Repeated short connections can trip rate limits or intrusion detection on the target, so coordinate and keep the attempt rate modest.
Inside a Kubernetes cluster, how would you measure per-flow network latency, spot microbursts and tell queueing delay from other causes, without adding heavy overhead across thousands of containers?
Sample Answer
Direct answer
Measure on each node, inside the kernel, with eBPF (small programs the kernel verifies and then runs at hook points, so no packet copying and no sidecar in every pod). Read the round-trip time (RTT) the kernel already tracks for every TCP connection, fold it into histograms in the kernel, and export only per-node, per-workload-pair summaries. Catch microbursts with byte counters kept in 1 ms bins on the node's network interface. Separate queueing from other causes by comparing three signals: kernel RTT, application request latency, and a same-node control path. Queueing is the explanation only when RTT rises in step with the burst bins and recovers the moment they end.
Background in plain language
End-to-end delay is propagation (distance), serialization (putting bits on the wire), processing (host and switch CPUs) and queueing (waiting in a buffer because the output port is busy). Of these four, queueing is the delay that grows directly with load on a link (host processing can also grow when a CPU is saturated, which is why the ordered checks below rule it out). A microburst is traffic that arrives faster than the output port can drain for a few milliseconds, fills the buffer, then disappears, so a counter polled every 10 to 60 seconds shows a near-idle link.
Worked example, computed with python3: four senders each push 10 Gb/s at one 10 Gb/s switch port for 1 ms. Ingress is 4 x 10 Gb/s x 1 ms = 5.0 MB; the port drains 1.25 MB in that millisecond, so 3.75 MB queue up. If the port's buffer is 4 MB, all 3.75 MB are held and the last byte waits 3.75 MB / (10 Gb/s) = 3.0 ms, with no loss. If the buffer is only 2 MB, the queue is capped at 2 MB, so the worst wait is 2 MB / (10 Gb/s) = 1.6 ms and 1.75 MB are dropped. Averaged over 10 seconds the same burst is 5 MB / 10 s = 4 Mb/s, which is 0.04% of the link. Averaged over the 1 ms bin it is 400% of line rate. Same event, 10,000 times apart, purely from the measurement interval.
Per-flow latency without heavy overhead
- One agent per node, not per container. A DaemonSet (Kubernetes runs one copy on every node) loads the eBPF programs. Thousands of containers share one agent per node, so overhead scales with nodes, not containers.
- Use RTT the kernel already computes.
ss -iprints the per-connectionrttandrttvar(mean deviation) in milliseconds, per the ss man page. The bcctcprtttool does the same job as a histogram: it traces TCP RTT, can break histograms down by local or remote address (-b,-B), and is described by its authors as a way to tell whether latency comes from a user process or the physical network. A production agent applies that idea continuously.
A real line fromss -ti dst 127.0.0.1:9000on a Linux test host (a loopback connection, so the values are tiny):
cubic wscale:10,10 rto:201 rtt:0.016/0.008 mss:32741 pmtu:65535 rcvmss:536 advmss:65483 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 164Gbps lastsnd:1010 lastrcv:1010 lastack:1010 pacing_rate 327Gbps delivered:1 app_limited rcv_space:65495 rcv_ssthresh:65495 minrtt:0.016 snd_wnd:65483
Read the fields that matter here: cubic is the congestion-control algorithm; rtt:0.016/0.008 is the smoothed RTT (0.016 ms) and its mean deviation, rttvar (0.008 ms), both in milliseconds per the ss man page; minrtt is the lowest RTT seen, a baseline; cwnd:10 is the congestion window in segments; wscale:10,10 the send and receive window-scale factors. On a real pod-to-pod path across nodes the rtt value is larger and rttvar shows how much it wobbles.
- Aggregate before exporting. Key a log-scale histogram by (source workload, destination workload, destination node), not by pod, so pod churn does not explode label cardinality (the number of distinct label combinations, each of which is a separate stored time series). Map IPs to workloads using the Kubernetes API or the CNI (container network interface plugin) data. Export every 15 seconds.
- Raw per-flow records only when interesting. Emit a full record for 1 in 100 connections plus any connection whose RTT exceeds a threshold. Hubble (Cilium's flow observability layer) is useful for flow verdicts and drops, but its documented
tcpmetrics handler counts TCP flags, so it does not give RTT; the eBPF layer does. - Non-TCP traffic has no kernel RTT. Use a small active probe mesh: each node probes a rotating sample of peer nodes once a second, which also provides the baseline.
Spotting microbursts
- Host side: an eBPF program on the node's egress interface adds packet bytes to a counter selected by the kernel's monotonic clock divided into 1 ms bins. Every interval it exports the maximum bin and the number of bins above, say, 80% of line rate. That is a few numbers per interface per interval, not a packet stream.
- Switch side: the queue forms at the switch's output port, so queue depth or buffer high-watermark telemetry from the switch shows the effect, if the platform exports it. The host bins show the cause (which nodes transmit in lockstep, as in a shuffle or all-to-one read).
- Align both on timestamps (clocks synchronized with NTP or PTP, network time protocols, PTP being the more precise) before correlating.
Telling queueing from other causes (ordered checks)
- Is it the network at all? Compare kernel RTT with application request latency. Kernel RTT flat while request latency rises points at the application, CPU throttling, or garbage collection.
- Run the control path. RTT between two pods on the same node never crosses the NIC or a switch. If that is also slow, the cause is host CPU, softirq load (the kernel's deferred packet-processing work, which runs on a CPU) or connection tracking (the kernel's table of connections that firewall and NAT rules depend on), not the fabric.
- Check load correlation. Queueing has a signature: RTT jumps by queue depth divided by line rate (3 ms in the 4 MB-buffer case above, at most 1.6 ms in the 2 MB-buffer case), affects every flow sharing that egress port at once, and returns to baseline within milliseconds of the burst ending. A cause like a bad cable or a CPU stall does not track the burst bins.
- Look at loss. Retransmits that appear only when the burst exceeds the buffer (in the 2 MB case, the 1.75 MB overflow) fit queueing; steady retransmits with flat bins fit a faulty link.
- Group by path. Only flows hashed onto one uplink (ECMP, equal-cost multipath) affected means a hot port, not a cluster-wide problem.
Worked trace (illustrative numbers, built on the 4 MB-buffer case above, where a burst adds 3 ms of queueing). Users report request latency rising from 8 ms to 12 ms. Check 1: kernel RTT for the same flows went from 0.4 ms to 3.4 ms, so about 3 ms of the 4 ms rise is in the network path, not the application. Check 2: the same-node pod pair stays at 0.05 ms, so host CPU and connection tracking are fine. Check 3: the 3 ms jumps line up with the 1 ms bins above 80% of line rate and fall back to 0.4 ms within a few milliseconds of each burst. Check 4: retransmits appear only in the few largest bursts, the ones that overflow the buffer. Check 5: only flows hashed onto uplink B are affected. Conclusion: queueing at one egress port, so spread the lockstep senders or add buffer or capacity there.
Trade-offs and pitfalls
- Kernel RTT is a smoothed per-connection estimate: it hides a single 3 ms spike on a long connection. That is why the 1 ms bins exist.
- eBPF needs a privileged agent and a kernel with the needed hooks; canary it on a few nodes, and put a hard cap on map memory and export rate.
- Do not mirror packets or inject sidecars cluster-wide for this: cost scales with total traffic or pod count, which is the overhead the question rules out.
- Recommendation: node-level eBPF histograms plus 1 ms bin counters as the always-on layer. It would change to targeted packet capture only on the two or three nodes the histograms implicate.
You must collect per-interface metrics from 200,000 interfaces across thousands of devices and keep your metrics system stable and queryable. Describe the collection architecture, where collectors run, how you shape labels, and what you do when it starts to fall over.
Sample Answer
Direct answer
Do not poll everything the same way. Prometheus (a time-series monitoring system that pulls metrics over HTTP) is the storage and query layer here. Split the 200,000 interfaces into tiers by how fast their state matters (about 20,000 uplinks and trunks every 30 seconds, about 180,000 access ports every 5 minutes), poll only the eight columns you will actually use (eight IF-MIB interface objects, named in the sizing section below), run SNMP collectors (snmp_exporter instances) inside each region's management network, and let several Prometheus servers shard the device list between them. Keep labels to a small bounded set (device, interface name, a few per-device attributes), and have a pre-planned ladder of load-shedding steps for when series count or scrape time starts to climb. The binding limit is the number of active time series (each unique metric name plus label set), not the number of devices. Cardinality is that count: one series exists for each distinct combination, for example ifHCInOctets{instance="sw1", ifName="Gi0/1"} is one series (illustrative names) and the same counter on the next interface is another.
Sizing: why the default is the wrong starting point
SNMP (Simple Network Management Protocol) polling reads counters from each device on a timer. The stock if_mib module in snmp_exporter's default snmp.yml defines 39 metrics, one of them (ifNumber) per device and the rest per interface, and its lookups attach ifAlias, ifDescr and ifName to every series.
| Design | Series | Samples per second |
|---|---|---|
| Default module, all 200,000 interfaces, 30 s | 200,000 x 38 = 7,600,000 | 253,333 |
| Eight columns, all interfaces, 30 s | 200,000 x 8 = 1,600,000 | 53,333 |
| Eight columns, tiered: 20,000 uplinks at 30 s plus 180,000 access ports at 300 s | 1,600,000 | 5,333 + 4,800 = 10,133 |
Tiering at the same series count cuts ingest 5.3 times against flat 30-second polling (53,333 / 10,133). As series, the eight columns for one interface are eight separate time series, one per column, each carrying the same device and interface labels (illustrative): ifOperStatus{instance="sw1", ifName="Gi0/1"} 1, ifHCInOctets{instance="sw1", ifName="Gi0/1"} 8234719442 and so on; 200,000 interfaces x 8 columns is the 1,600,000 in the table. The eight columns are ifOperStatus, ifAdminStatus, ifHCInOctets, ifHCOutOctets, ifInErrors, ifOutErrors, ifInDiscards and ifOutDiscards. The 64-bit HC (high capacity) counters are not optional: a 32-bit octet counter wraps in about 34 seconds at 1 Gbit/s (RFC 2863), far shorter than a 300 s access poll.
Collection architecture and where collectors run
- snmp_exporter near the devices. SNMP uses UDP, walks make many round trips, and loss or delay inflates scrape time. Run exporter instances in each region's management network and have Prometheus pull from them over HTTP across the WAN (one job per region, each relabelled to its regional exporter). The exporter is called as
/snmp?target=<device>&auth=<auth>&module=<module>, so one exporter serves many devices. - A custom module. Generate a module with snmp_exporter's generator that contains only the eight columns. Fewer OIDs per walk means shorter scrapes as well as fewer series.
- Scrape and relabel. A scrape is one HTTP request in which Prometheus pulls the current values. Relabeling rewrites a target's labels before the scrape, and labels whose names start with two underscores (
__address__,__param_target,__tmp_shard) are internal working labels that Prometheus discards afterwards, so they never become stored labels. The file in the next section shows each one. - Sharding. Split the target list across Prometheus servers with the hashmod relabel action: hash the target address modulo the shard count, and each server keeps the targets whose hash equals its index.
- Long-term and global view. Each shard holds recent data. Send everything to a horizontally scalable long-term store with
remote_writefor cross-shard queries and retention beyond weeks.
The configuration
global:
scrape_interval: 60s
scrape_configs:
- job_name: snmp_uplinks
scrape_interval: 30s
scrape_timeout: 25s
sample_limit: 20000
metrics_path: /snmp
params:
auth: [corp_v3]
module: [if_slim]
file_sd_configs:
- files: [targets/snmp-uplinks.yml]
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: snmp-exporter.dc1.example:9116
- job_name: snmp_access
scrape_interval: 300s
scrape_timeout: 120s
sample_limit: 20000
metrics_path: /snmp
params:
auth: [corp_v3]
module: [if_slim]
file_sd_configs:
- files: [targets/snmp-access.yml]
relabel_configs:
- source_labels: [__address__]
modulus: 4
target_label: __tmp_shard
action: hashmod
- source_labels: [__tmp_shard]
regex: "1"
action: keep
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: snmp-exporter.dc1.example:9116
metric_relabel_configs:
- regex: ifDescr
action: labeldrop
rule_files: [snmp-capacity-rules.yml]
Read the access job's relabel_configs top to bottom; each step runs on the target's labels in order. (1) hashmod hashes __address__ (the device address from the targets file) and stores the remainder after dividing by 4 in __tmp_shard, a number 0 to 3. (2) keep with regex: "1" drops every target whose __tmp_shard is not 1, so this server scrapes about a quarter of the devices. (3) The device address is copied into __param_target, which becomes the target= URL parameter of the request to the exporter. (4) The same value is copied to instance so the stored series are labelled with the device, not the exporter. (5) Finally __address__ is overwritten with the exporter's address, which is where Prometheus actually sends the request. The params block supplies auth and module, and metric_relabel_configs runs after the scrape on the returned series: labeldrop with regex: ifDescr removes that label from every stored series. file_sd_configs (file-based service discovery) reads the device list from a YAML file that can be edited without restarting Prometheus. remote_write is the Prometheus setting that streams each sample to the long-term store.
This is shard 1 of 4 for the access job (the same file with regex: "0", "2" and "3" on the other three servers). The 180,000 access ports x 8 = 1,440,000 series then split to about 360,000 series per shard. The 160,000 uplink series fit on one server and are not sharded. sample_limit (default 0, meaning no limit) rejects a scrape that returns more than 20,000 samples: a 48-port switch returns 48 x 8 = 384 samples, a 576-port chassis 4,608, so 20,000 only trips on a runaway.
Shaping labels
- Keep:
instance(the device),ifName,ifIndex(needed to join one metric to another), plus per-device attributes (site, role, vendor) added as target labels from the inventory, so the label count is bounded by devices rather than interfaces. - Drop on the bulk job: ifDescr (it duplicates ifName) and, if you do not need it, ifAlias. Alias is free text that people edit, and each edit creates a new series (churn). Keep it on the small uplink job where the description names the circuit.
- Never put a counter value, an IP address that changes, or a timestamp in a label.
- Put descriptive text that changes slowly (customer, circuit ID) on a separate, slowly polled info series and join it at query time.
Dashboards need windows that fit the interval
rate() works out a per-second increase from the counter samples inside a time window, written in square brackets. A rate over [5m] (the last five minutes) on a 300 s series holds one or two samples and can return nothing. Use rate(ifHCInOctets[15m]) or longer for access ports. A 300 s interval has a second problem: Prometheus treats an instant value as stale after 5 minutes by default (the lookback delta, set with --query.lookback-delta), so a 300 s scrape sits exactly on that limit and one late or jittered scrape leaves a gap, which makes a state panel such as ifOperStatus flicker to no data. Wrap state queries for the access tier in last_over_time(ifOperStatus[15m]), raise the lookback delta, or poll the access tier every 120 s where the budget allows: 1,440,000 / 120 = 12,000 samples per second for the access ports (17,333 with the uplink tier), against 4,800 (10,133) at 300 s. Always take rate() before aggregating with sum(), because only rate() sees counter resets.
When it starts to fall over, in order
Alert before it breaks:
groups:
- name: snmp-collection-health
rules:
- alert: SnmpScrapeNearInterval
expr: scrape_duration_seconds{job="snmp_uplinks"} > 0.8 * 30
for: 10m
labels:
severity: ticket
- alert: ScrapeSampleLimitHit
expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
labels:
severity: ticket
- Read the signals. Scrape duration approaching the interval, targets going
up == 0(upis the 1 or 0 series Prometheus records for every scrape, so 0 means the scrape failed),prometheus_tsdb_head_series(Prometheus's own count of active series in memory) rising, memory rising. Query/api/v1/status/tsdbforseriesCountByMetricNameandlabelValueCountByLabelNameto see what is eating series. - Name the cause. Steady growth with device count means capacity. Steps in series count with no new devices mean churn (alias edits, interface renames, flapping labels). A few targets with long scrape times mean slow devices holding exporter capacity.
- Shed load by value. Drop labels and metrics at ingestion (
metric_relabel_configs), lengthen the access interval (300 s to 600 s), cut the column list. - Add shards. Raise the modulus (4 to 8). Targets move to different servers when the modulus changes, so their history is split across servers: the long-term store is what makes a device's whole history queryable.
- Move the heaviest, most valuable devices to streaming telemetry (gNMI, the gRPC Network Management Interface, subscriptions, where the device pushes updates on a sample interval) so the busiest cores stop being polled.
Trade-offs
- A 5-minute access poll cannot catch short blips and leaves no slack against the 5-minute staleness window. Pair it with traps or syslog for state changes, and keep the 30 s tier for links whose flapping matters.
- More shards add query complexity. Do not shard until a measured limit forces it.
How would you use eBPF to collect network telemetry from Linux hosts? What can you observe, what does it cost the host, and how do you get the data into an existing metrics and logging stack?
Sample Answer
Direct answer
eBPF (extended Berkeley Packet Filter) lets you load small, kernel-verified programs that run when the kernel hits a hook (a named point in the kernel where a program can be attached), such as a TCP connect, a retransmit, or a packet arriving at the network driver. For network telemetry you attach programs to the events you care about, aggregate in kernel memory (in a map, a table in kernel memory that both the program and a user-space agent can read), and have a user-space agent read the results and expose them to your existing stack: counters and histograms to Prometheus, discrete events to your log pipeline. It gives per-host, per-process visibility that SNMP and flow data cannot, at a cost that scales with how many events you attach to.
What you can observe
- Connection events with the owning process and container. The bcc (BPF Compiler Collection) tools show the idea:
tcpconnectprints one line for every active TCP connection,tcpacceptone line for every passive connection, andtcpretransone line for every TCP retransmit, with addresses and the kernel state of the connection. Because the hook is in the kernel you can attach the process or cgroup (control group, the kernel's container boundary) that owned the socket. For connect and accept the current task is normally the owner, so the current task's cgroup is the right label. Retransmits are different: they often fire from a timer or interrupt where the running task is unrelated (the PID 0 in thetcpretransoutput below is exactly that), so a program that counts retransmits per cgroup must read the cgroup from the socket itself (thebpf_sk_cgroup_idhelper returns a socket's cgroup v2 id) and not from the current task (bpf_get_current_cgroup_idreturns the cgroup of the task that is running). - Retransmits and drops per destination, which turn into "this host sees 2% retransmits toward that subnet" without a packet capture.
- Latency distributions (connect time, time between events) as histograms rather than averages.
- Dropped packets with a reason, by attaching where the kernel discards a packet.
- Limits: it sees what the host kernel sees. It cannot see the middle of the network, and encrypted payload stays encrypted (you see sizes and timing, not content).
What the tools print (addresses and process names are illustrative; the column layout is the one in the bcc documentation):
$ tcpconnect
PID COMM IP SADDR DADDR DPORT
1469 curl 4 10.201.219.236 54.245.105.25 80
$ tcpretrans
TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE
01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED
Reading it: each tcpconnect row is one outbound connection, naming the process (PID, COMM), the source and destination address and the destination port. Each tcpretrans row is one retransmitted segment: the local and remote address and port, R> meaning a retransmit, and the connection state at that moment. PID 0 means the kernel was running, not an application, when the retransmit fired. Many rows for the same remote address and port is one unhealthy connection, and rows spread over many remote addresses point at a wider path problem.
What it costs the host
- Overhead is per event. Attaching to a rare event (connect, retransmit) costs almost nothing in practice. Attaching a program to every packet on a 40G host adds work to every packet and can be measurable (ESTIMATE, depends on the program).
- Aggregate in the kernel (a map keyed by cgroup and destination subnet, incrementing counters or histogram buckets) instead of emitting one event per packet. Streaming every event to user space is where the CPU goes.
- Operational cost: you need a kernel with BTF (BPF Type Format, a description of the kernel's own data structures) so that CO-RE (compile once, run everywhere) programs can adapt to each kernel version without recompiling, and privileges (CAP_BPF and CAP_PERFMON, Linux capabilities that split root power into narrow permissions) for the agent. A first deployment on one kernel version needs the connection and retransmit hooks, one in-kernel counter and the exporter; BTF and CO-RE start to matter once the fleet runs several kernels. Programs are checked by the kernel verifier (a checker that proves the program terminates and only touches memory it is allowed to) before loading, so a bad program is rejected rather than crashing the host, but a correct program can still burn CPU. Test per kernel version in a canary and compare CPU with the probe on and off.
- Cardinality cost on the metrics side: labelling by client IP and port explodes the time series count. Label by service, cgroup or subnet.
Getting the data into the stack
- Metrics. Run
ebpf_exporter(Cloudflare's Prometheus exporter for eBPF). You describe counters and histograms in a YAML config, the exporter reads the kernel maps and converts binary keys to labels using decoders (a decoder turns a raw kernel value into readable text, for example a cgroup ID into its path or an address into a function name). Prometheus scrapes it like any exporter. The shape of a counter definition (the metric and label names here are illustrative; the fields and thecgroupdecoder are from the ebpf_exporter documentation):
metrics:
counters:
- name: tcp_retransmits_total
help: TCP retransmits by cgroup
labels:
- name: cgroup
size: 8
decoders:
- name: cgroup
Each entry names one counter, and each label says how many bytes of the map key to read (size) and how to decode them. A destination-subnet label is a second entry under labels with a decoder that matches how the program stores it.
2. Events. For things you want as log lines (every retransmit burst above a threshold, every drop with a reason), have the program write to a BPF ring buffer. The kernel documentation describes it as a buffer shared across CPUs that preserves ordering of events, which a per-CPU perf buffer (the older design, one separate buffer per CPU) does not. A small agent reads it, rate-limits, adds host and pod metadata, and ships JSON lines through the existing log shipper.
3. Correlate. Join on host and cgroup labels so a retransmit spike in the metrics can be traced to the log lines for the same pod.
Worked example
Goal: find which service is hurting from retransmits. Deploy a retransmit counter keyed by the socket's cgroup (not the running task's) and destination /24, export it as a counter, alert on the rate of change per service. Roll out to 5% of hosts first, compare node CPU for those hosts against the rest for a day, then widen. When the alert fires, run tcpretrans on one affected host to see the actual destinations and connection states.
Pitfalls
- Per-packet probes on high-rate interfaces without measuring overhead.
- Unbounded label cardinality (client IP as a label).
- Assuming one program works on every kernel: test the kernels you actually run.
- Treating host-side numbers as network truth: a host that is overloaded can show drops that the network did not cause.
Unlock Full Question Bank
Get access to all 8 Network Monitoring and Performance interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.