Network Monitoring and Performance Questions

Approach: define concrete probe parameters per protocol, add hysteresis and aggregation rules, and explain trade-offs between sensitivity and false positives.

HTTP service (web app)

• Probe: HTTP GET to a lightweight health endpoint (/health or /ready) that checks app-level dependencies minimally.
• Frequency: 10s (can be 5s for mission-critical low-latency services).
• Timeout: 2s (abort slow responses).
• Retry strategy: retry immediately up to 2 times with small backoff (100–200ms).
• Failure threshold: mark instance Unhealthy after 3 consecutive failures; mark Healthy after 2 consecutive successes.
• Checks: verify 200 OK and small JSON payload; optionally include latency and content checks.

TCP database service

• Probe: TCP connect + simple query/heartbeat (e.g., SELECT 1) over the DB port.
• Frequency: 15s (less frequent than HTTP to reduce DB load).
• Timeout: 1s for TCP connect, 2s for query execution.
• Retry strategy: 1 retry after 200ms.
• Failure threshold: mark Unhealthy after 2 consecutive failures; require 3 consecutive successes to mark Healthy.
• Use non-privileged, low-cost queries to avoid load.

Combining multiple probes

• Per-instance: use sliding window (e.g., last N probes or time window) and require consecutive failures to trigger transition. Track states: Healthy → Probing → Unhealthy.
• Multi-region/multi-probe aggregation: use quorum/majority for a pool (e.g., if >50% instances healthy, service considered available) or weighted scoring (weights for instance capacity/priority).
• For layered services (app + DB), treat upstream dependency as critical: degrade app to Degraded if DB pool < threshold even if app instances pass HTTP checks.

Trade-offs (sensitivity vs false positives)

• More frequent/shorter timeouts => faster detection but higher false positive rate (transient network blips, GC pauses).
• Longer timeouts and higher thresholds => fewer false alarms but slower remediation and risk of serving bad traffic.
• Mitigations: add jitter to avoid synchronized probes, use exponential backoff on retries, correlate probe failures with metrics/alerts before paging, and apply progressive remediation (e.g., mark Unavailable in load balancer but avoid immediate paging until secondary checks fail).
• Tune per service based on SLOs, cost of false positives, and probe impact on system.

SNMP is a lightweight protocol SREs use to collect device and interface metrics from network gear (switches, routers, firewalls) and to receive device-originated notifications. It standardizes access via MIBs (Management Information Bases) so monitoring systems can query consistent OIDs.

Polling vs Traps/Notifications

• Polling: The monitoring system periodically queries OIDs on devices (GET/GETNEXT/GETBULK). Reliable for regular sampling, trending, capacity planning, and threshold evaluation because you control frequency and can compute deltas (e.g., bytes/sec).
• Traps/Notifications: Devices push asynchronous messages (TRAP/INFORM) when events occur (link down, auth failure). Faster for immediate alerts; INFORMs add acknowledgement for delivery assurance.

IF-MIB examples

• ifInOctets / ifOutOctets: cumulative byte counters per interface — use deltas between polls to compute throughput.
• ifOperStatus: current operational state (up/down/testing) — good for immediate availability checks.
• ifDescr, ifIndex: metadata to map counters to interfaces.

When to poll vs trap

• Prefer polling when you need continuous metrics, historical trending, accurate rates (throughput/packet loss), or when devices don’t reliably send traps.
• Prefer traps for low-latency incident detection (link down, high-priority alarms), reducing polling load, or when immediate notification reduces MTTR.

Practical tips

• Combine both: poll counters at a reasonable interval (30–300s) and subscribe to traps for critical events. Handle counter wrap/64-bit counters, clock sync for deltas, and implement deduplication/rate-limiting for noisy traps.

Steps to map heavy traffic to container ID:

1. Identify top flows on host using tcpdump or nethogs.

sudo tcpdump -i eth0 -nn net 0/0 -w /tmp/traffic.pcap
sudo nethogs eth0

2. Correlate flows to local sockets with ss and conntrack:

ss -tunp | grep remote-ip:port
sudo conntrack -L | grep <remote-ip>

ss output shows local port and PID of process owning socket. If container processes run under containerd/docker, PID maps to container via /proc/<pid>/cgroup.

3. Map PID to container:

Example

cat /proc/<pid>/cgroup

or use docker

docker ps --filter "status=running" --format "{{.ID}} {{.Names}}" && docker inspect --format '{{.State.Pid}} {{.Id}}' $(docker ps -q)

Alternatively use crictl or podman commands in k8s.

4. Use tc or iptables to filter flows for further capture:

sudo tc filter add dev eth0 protocol ip parent 1:0 u32 match ip dst <ip> 0xffffffff action mirred egress redirect dev ifb0

5. Tools: iftop, nethogs, netstat, ss, conntrack, docker inspect, nsenter. For Docker, use docker top and docker stats for per-container activity.

Outcome: combine flow capture, ss to get PID, and cgroup/docker inspect to resolve the container ID. Automate this in scripts for faster incident response.

SLI, SLO, SLA — short definitions for an internal network fabric:

• SLI (Service Level Indicator): a measurable metric that indicates network health (e.g., packet delivery, latency, jitter, link utilization).
• SLO (Service Level Objective): target value/range for an SLI that the team commits to meet.
• SLA (Service Level Agreement): formal contract with stakeholders (internal) that may include penalties or remediation if SLOs are repeatedly missed.

Measurable SLI formulas (examples):

• Successful packet delivery rate = 1 - (dropped_packets / total_packets)
• P99 latency = value at 99th percentile of packet RTTs
• Jitter = stddev(inter-packet-arrival-time) or 95th-percentile inter-arrival variance
• Link utilization (%) = (bytes_transferred / link_capacity_bytes) * 100

Example SLOs for internal fabric (reasonable defaults; adjust by business needs):

• Successful packet delivery rate ≥ 99.99% over 30 days
• P99 latency ≤ 5 ms within the datacenter over 30 days
• Packet loss per flow ≤ 0.01% (same window)

Measuring and acting on error budget:

• Error budget = 1 - SLO_target (e.g., for 99.99% target, monthly error budget = 0.01% of packets or time).
• Continuously measure SLIs with telemetry (sFlow, NetFlow, telemetry agents, BGP/EVPN counters) aggregated to time windows (minute/5m/30d).
• Track consumption: convert observed violations into budget consumed (e.g., dropped_packets_excess / total_packets_allowed).
• Responding to consumption:
  • Green (low consumption): allow normal changes and deploys.
  • Yellow (moderate consumption, e.g., >50%): freeze risky changes, increase monitoring, run targeted diagnostics (path traces, queue analysis), schedule remediation work.
  • Red (budget exhausted or trending fast): halt all non-essential changes, assemble incident response, prioritize root-cause fix, communicate to stakeholders, and perform postmortem.
• Automate alerts on burn rate, and create dashboards showing budget, root-cause tagged incidents, and mitigation actions. Regularly review SLOs vs. business needs and adjust instrumentation or capacity planning accordingly.

Detection approach — use three complementary signals (SNMP ifOperStatus, syslog link-state messages, and SNMP/CLI traps). Correlate them into a single “link-flap detector” service that ingests events with timestamps and interface identifiers and applies sliding-window logic + suppression during BGP reconvergence.

Detection logic (concrete):

• Primary rule: raise a flap-warning if an interface shows >= 3 state transitions (up→down or down→up) within 5 minutes (X=3, Y=5m).
• Escalation rule: if flaps continue and reach >= 10 transitions in 30 minutes, escalate to page on-call.
• Confirmation: require at least one corroborating source (e.g., syslog "LINK-3-UPDOWN" or an SNMP trap) within ±30s of an SNMP ifOperStatus change to avoid false positives from noisy polling.

Required historical data:

• Keep the last 30 minutes of raw events per interface (timestamped ifOperStatus changes, syslog messages, trap payloads).
• Maintain counters and the exact timestamps of the last N transitions (N >= 10) to run sliding-window counts.
• Store a flap state for each interface: last alert time, flap count window, suppression expiry.

Threshold rationale:

• 3 in 5m catches symptomatic flapping without reacting to a single maintenance bounce.
• 10 in 30m separates persistent instability from transient issues — good balance for production links.

Avoiding duplicate alerts during BGP reconvergence:

• Detect BGP reconvergence by watching BGP neighbor state events (syslog "BGP-5-ADJCHANGE", SNMP BGP peer state OIDs) for the same router pair. When a BGP neighbor reset is detected:
  • Temporarily suppress link-flap alerts for the neighboring interface(s) for a short window (e.g., 2 minutes) while reconvergence completes.
  • Instead of immediate alert, record events as “suppressed” and at suppression end evaluate whether flapping threshold was exceeded during suppression; if so, emit a single aggregated alert summarizing counts and first/last timestamps.
• Use alert fingerprinting: key = (device, interface, neighbor IP). If an alert already active for that fingerprint, update the alert (add counts/timestamps) rather than create new alerts. Add a cooldown (e.g., 15 minutes) after an alert before sending another notification for same fingerprint.

Additional robustness:

• Hysteresis: only clear a flap alert after 10 continuous minutes with no transitions.
• Backoff and dedupe: exponential backoff on repeated notifications; always include evidence (timestamps, source types) in the alert.
• Validation: correlate physical interface counters (SNMP ifInErrors/ifOutErrors) and link utilization spikes to prioritize real outages.

Simple pseudocode outline:

on_event(e): // e: {device, iface, type, state, ts}
  append_to_history(device, iface, ts)
  count = transitions_in_window(device, iface, now, 5m)
  if bgp_neighbor_reset_recent(device, iface, 2m):
    suppress_and_record(device, iface, e)
    return
  if count >= 3:
    if alert_active(device, iface):
      update_alert(device, iface, count, ts)
    else:
      create_alert(device, iface, severity=warning, details)

Operational notes:

• Tune X/Y per link criticality: carrier links may need tighter thresholds; lab links looser.
• Test with simulated flaps and BGP resets, and include runbooks in alerts describing next steps (check carrier, replace cable, escalate to network team).