Network Monitoring and Performance Questions

Direct answer

A fixed threshold fires when a metric crosses a number you chose in advance (for example link utilization above 85 percent). Anomaly detection fires when a metric deviates from what is normal for that metric at that time (for example Tuesday 03:00 traffic at four times its usual level). Fixed thresholds are simple, predictable and right for limits that are physical or contractual, but blind to anything that is abnormal yet below the line. Anomaly detection catches those, but needs history, is harder to explain, and can quietly learn a problem as the new normal. For network metrics I would use both: fixed thresholds for hard limits and error counters, anomaly detection for volume and latency shapes with no natural limit.

Where each works and where each fails

Executable comparison

This script builds hourly utilization percent with a daily cycle (12 percent overnight, up to 40 percent in the early afternoon) plus Gaussian noise (standard deviation 1.5, seed 7). The anomaly detector compares each hour with the median of the same hour over the previous 14 days and alerts when the gap exceeds 4 times a robust spread, with a floor of 1.5 points. A robust spread is a measure of how much the values normally vary that, unlike the standard deviation, is not thrown off by a few extreme values. Here it is the median absolute deviation (the median of each value's distance from the median) multiplied by 1.4826, the factor that makes it match the standard deviation when the data is bell-shaped, so "4 times" reads like four standard deviations. The floor stops a very quiet hour from making the rule over-sensitive. The fixed rule alerts above 85 percent. The scenarios mirror the table: each one injects a fault into otherwise clean data and asks which detector fires.

import random, math, statistics as st

def series(days, ramp_per_day=0.0, seed=7):
    r = random.Random(seed)
    out = []
    for d in range(days):
        for h in range(24):
            daily = 12 + 28 * max(0.0, math.sin(math.pi * (h - 6) / 14)) if 6 <= h <= 20 else 12
            out.append(daily + ramp_per_day * d + r.gauss(0, 1.5))
    return out

def baseline(x, i, window_days=14):
    same = [x[i - 24 * k] for k in range(1, window_days + 1)]
    med = st.median(same)
    mad = st.median(abs(v - med) for v in same) * 1.4826
    return med, max(mad, 1.5)

def evaluate(x, start_day, fixed=85.0, k=4.0):
    fixed_hits, anom_hits = [], []
    for i in range(start_day * 24, len(x)):
        med, scale = baseline(x, i)
        if x[i] > fixed: fixed_hits.append(i)
        if abs(x[i] - med) / scale > k: anom_hits.append(i)
    return fixed_hits, anom_hits

def fmt(i): return f"day{i // 24:02d} {i % 24:02d}:00"

x = series(44)
f, a = evaluate(x, 14)
print("clean 30 days:", "fixed alerts", len(f), "| anomaly alerts", len(a))

x = series(44); i = 40 * 24 + 3; x[i] = 45.0
f, a = evaluate(x, 14)
print("03:00 job at 45%:", "fixed", [fmt(j) for j in f if j == i], "| anomaly", [fmt(j) for j in a if j == i])

x = series(44); i = 40 * 24 + 14; x[i] = 2.0
f, a = evaluate(x, 14)
print("14:00 collapse to 2%:", "fixed", [fmt(j) for j in f if j == i], "| anomaly", [fmt(j) for j in a if j == i])

x = series(150, ramp_per_day=0.5)
f, a = evaluate(x, 14)
print("ramp 0.5pt/day, 150 days: first fixed alert", fmt(f[0]) if f else None, "| anomaly alerts total", len(a))
print("util at first fixed alert: %.1f" % x[f[0]])

Output:

clean 30 days: fixed alerts 0 | anomaly alerts 0
03:00 job at 45%: fixed [] | anomaly ['day40 03:00']
14:00 collapse to 2%: fixed [] | anomaly ['day40 14:00']
ramp 0.5pt/day, 150 days: first fixed alert day85 13:00 | anomaly alerts total 22
util at first fixed alert: 86.8

Reading it. On clean data neither detector alarms, so the later results are signal. The 45 percent overnight job and the 2 percent collapse are invisible to the fixed rule (45 and 2 are both below 85) and obvious to the anomaly rule, which knows the 03:00 and 14:00 norms. The slow ramp reverses it: utilization rises half a point per day, the fixed rule fires cleanly on day 85 at 86.8 percent, a clear "this link is about to saturate" signal, while the anomaly detector produced 22 isolated alerts across the 136 evaluated days (the longest run was two consecutive hours) and never a sustained one. Its 14-day median baseline lags the ramp by about 3.75 points (the window's median is about 7.5 days old, and the ramp adds 0.5 points per day), under the threshold of at least 6 points, so the trend is absorbed as normal and only noise occasionally tips an hour over. The demo uses a daily cycle only. Real traffic also has a weekly cycle, so a production baseline must compare the same hour of the same weekday.

Recommendation and pitfalls

• Use fixed thresholds for anything with a physical or contractual ceiling and for counters that should be zero (discards, cyclic redundancy check or CRC errors). Require the condition to persist, for example 10 minutes, to avoid paging on one noisy sample.
• Use anomaly detection for volume and latency shapes, set a minimum absolute change as well as a statistical one so a quiet link going from 0.1 to 0.3 percent does not page, and send those alerts to a ticket or a review queue first until false positives (alerts when nothing is actually wrong) are measured.
• Cover the anomaly blind spot (slow drift) with a forecast or a fixed capacity line, as the ramp case shows.
• What would flip the choice: with no history (new site), start with fixed; with metrics that have no safe absolute number, lean on anomaly detection.

Direct answer

I would first prove the problem is geographic and time-correlated, then look for repeated route changes (announce, withdraw, announce) on the paths serving that geography, using my own routing telemetry and passive flow data, then compare with outside views from public BGP collectors, and only then engage providers with evidence. If there is no update churn, it is not flapping and I move to congestion or application causes. Flapping means the same prefix changing state repeatedly; a single path change is not a flap.

Ordered checks

1. Slice the symptom. Break p99 by client country or ASN and by the entry point (PoP, point of presence, a site where your service connects to the internet) that serves it. If only one geography spikes, and the spikes recur at intervals, routing is plausible. Compare active probe loss and round-trip time (RTT) from that region.
2. Own routing data. From BMP (the Border Gateway Protocol, BGP, Monitoring Protocol, RFC 7854: your routers report each route update they receive, each time a neighbor session comes up or goes down, and counters) or router logs, count updates and withdrawals per prefix per minute for the providers or exchange serving that region, and look for session resets. Repeated withdrawals and re-announcements of the same prefixes in short cycles are the flap signature.
3. Passive flow data (NetFlow or IPFIX records). Look at traffic from that geography's source prefixes: if the ingress interface (the router port the traffic entered on) alternates between two links in step with the latency spikes, the path is changing under the traffic. That is also how to detect flaps without any routing feed.
4. Rule that raises an incident automatically (starting values, tuned to baseline): at least 5 changes for the same prefix within 10 minutes AND probe loss above 1% or p99 above the objective (the p99 target in your SLO, service level objective) for the same geography in the same window. The reasoning behind the numbers: one or two changes are normal churn or a single failover and recovery, so 5 asks for repeated cycles; 10 minutes keeps the rule about a problem happening now; and 1% loss is a value to set just above the loss each region normally shows on healthy days, so check it against your own baseline. Severity scales with the share of customer traffic from the affected region. Flaps with no customer impact open a ticket instead.
5. Outside data. RIPE RIS (RIS Live, which streams BGP messages as they arrive and can filter by prefix or AS path) and RouteViews (public collectors that record BGP updates from many networks) show whether the prefixes involved are churning from many vantage points; BGPStream (an open-source tool from CAIDA, the Center for Applied Internet Data Analysis) can process live and historical data for the same time window. If they show churn for the client-side network's prefixes, the instability is not inside my network. Limit: collectors are a sample of networks.
6. Rule out alternatives. Steady loss with no churn means congestion; a step change with no churn means a one-time path change or geo-DNS change (DNS answers that send users to a different site depending on where they are); a deploy at the same time means the application.

Who to engage

Send each a short evidence package: prefixes, UTC timestamps, update counts, the flow ingress-interface plot, probe loss.

Worked example of the alert rule

Prefix 198.51.100.0/24 (a documentation range, used here as an example) changes state 7 times in 10 minutes, and probe loss from the region is 3% over the same window. The seven events, as a collector would show them (illustrative):

14:02:10  withdraw   198.51.100.0/24
14:03:05  announce   198.51.100.0/24
14:04:40  withdraw   198.51.100.0/24
14:06:15  announce   198.51.100.0/24
14:07:30  withdraw   198.51.100.0/24
14:09:00  announce   198.51.100.0/24
14:10:25  withdraw   198.51.100.0/24

The fifth event (14:07:30) is the first time 5 changes fall within 10 minutes, and the seventh (14:10:25) is 8 minutes 15 seconds after the first, so the count of 7 is inside the window. Both conditions hold (7 is at least 5, 3% is above 1%), so an incident opens. If the same prefix had changed 7 times but loss stayed at 0.1%, it would be a ticket.

Pitfalls

• Averaging p99 across regions hides a single-region spike.
• Treating one path change as a flap sends you to providers with nothing repeatable to show.
• Public collectors lag and see only some networks, so no churn there does not clear my own view.

Pull means the monitoring server contacts each host and fetches metrics. Push means each host sends metrics out to a collector. Compare them for hosts that appear and vanish quickly and sit behind strict firewalls, then combine.

Pull in this environment

• For: you get a built-in health signal (the up metric: Prometheus records 1 for each target whose last scrape worked and 0 for each that failed), the server controls scrape rate, and hosts hold no credentials for the backend.
• Against: the server needs an inbound path to every host, which a strict firewall policy denies by default; targets must be found through service discovery (automatic lookup of which hosts exist, instead of a fixed list) as addresses churn; and a host that lives less than one scrape interval is never seen.

Push in this environment

• For: only outbound traffic from the host (typically one allowed HTTPS destination), no discovery of targets, and short-lived hosts can send a final report before they exit.
• Against: hosts need credentials and the receiver needs rate limiting; a host that stops sending looks the same as one that never existed, so you lose the automatic up signal. The Prometheus Pushgateway is not the answer: its guidance limits it to service-level batch job outcomes because it becomes a single point of failure, removes up, and never forgets a series until it is deleted.

Recommended hybrid: scrape inside each zone, forward outward.

1. In every firewall zone run a Prometheus in agent mode (--agent). Agent mode scrapes and remote-writes only; it has no local querying, no recording rules and no local alerting. It discovers hosts with local service discovery, so the firewall only has to allow zone-internal scrapes, which are usually permitted.
2. Each agent remote-writes (streams samples over HTTP to another store) outbound over HTTPS to a central store, tagging data with an external label such as zone (a label Prometheus attaches to everything it sends outward, so the central store can tell zones apart). If the link drops, the agent buffers locally; the buffer is documented as about two hours, so outages longer than that lose data.
3. Alerting and recording rules run centrally on the combined data.
4. Catch a silent zone with an absence rule, shown in the rules file below. absent_over_time(up{zone="eu-1"}[5m]) returns a value only when no up sample labelled zone="eu-1" has arrived for the whole 5 minutes, so the alert fires when that zone's agent stops delivering anything. This replaces the up signal push loses.
5. Hosts whose life is shorter than the scrape interval, and batch jobs, push through OTLP (the OpenTelemetry protocol, a standard format for sending metrics) or remote write straight to a collector, outbound only; only batch-job outcomes may use the Pushgateway.

# agent.yml, validated with: promtool check config --agent agent.yml
global:
  scrape_interval: 30s
  external_labels: {zone: eu-1}
scrape_configs:
  - job_name: hosts
    dns_sd_configs:
      - names: [hosts.eu-1.internal.example]
        type: A
        port: 9100
remote_write:
  - url: https://metrics.example.com/api/v1/write

Reading it: scrape_interval: 30s fetches every host every 30 seconds. external_labels: {zone: eu-1} stamps every outgoing sample with its zone. The hosts job uses dns_sd_configs, which is service discovery by DNS: it looks up the A (IPv4 address) records for hosts.eu-1.internal.example and scrapes each returned address on port 9100, so targets follow the DNS answer as hosts come and go, with no config edits. remote_write gives the central store's URL that the agent pushes to, outbound only. The hostnames and URL are illustrative.

The absence rule for that zone, in a separate rules file that runs on the central store:

# zone-rules.yml, validated with: promtool check rules zone-rules.yml
groups:
  - name: zone-silence
    rules:
      - alert: ZoneSilent
        expr: absent_over_time(up{zone="eu-1"}[5m])
        labels:
          severity: critical
        annotations:
          summary: No up samples from zone eu-1 for 5 minutes

A promtool test rules run with the zone's up series going stale showed no alert early on and a ZoneSilent alert with severity=critical and zone=eu-1 once 5 minutes had passed with no samples. alert names the rule, expr is the condition, labels are attached to the alert so routing can page on severity, and annotations carry the human-readable text.

Trade-off summary. The hybrid keeps pull's health signal and central control inside each zone, uses push only across the firewall boundary where pull cannot go, and costs one agent per zone to run and one absence rule per zone to maintain.

What would flip it: if no zone may host any collector at all, fall back to per-host push with an OpenTelemetry Collector agent on each host and a heartbeat metric per host. If hosts live long enough, zones are few, and firewall change control is fast, plain central pull is simpler.

Direct answer

Store a mergeable histogram per source IP per time window instead of raw samples: fixed, geometrically spaced latency buckets (each bucket 5 percent wider than the last) with a count in each. Counts add, so hourly histograms merge into daily or five-year ones and the 99th percentile can be read from the merged result with a known error. What becomes impossible is anything that needs the individual samples: an exact value, a different bucketing later, the order of events inside a window, and any breakdown you did not keep as a key.

Why not store the percentile itself

A 99th percentile (p99: 99 percent of samples are at or below it) cannot be averaged or combined. The p99 of a day is not the mean of 24 hourly p99s, nor their maximum. You can only combine distributions, so keep the distribution at reduced resolution.

Design

1. Buckets: the boundaries are $1.05^k$ ms for whole numbers k (negative k gives boundaries below 1 ms), so each bucket spans a factor of 1.05. Indices -95 to 188 run from about 0.0097 ms to 10,111 ms, which covers 10 microseconds to 10 seconds, in 284 buckets. Report a bucket's geometric midpoint: its lower edge times $\sqrt{1.05}$, the point halfway between the two edges on a multiplicative scale. A true value in the bucket is at worst a factor of $\sqrt{1.05}$ away from that midpoint, so the worst-case relative error is $\sqrt{1.05} - 1 \approx 2.47\%$ (2.41 percent in the other direction).
2. Per window record: bucket counts plus sample count, minimum, maximum and sum (these four are exact and merge trivially).
3. Tiers: raw 1 second samples for 7 days (for investigation), hourly histograms for 30 days, daily histograms for the rest of 5 years.
4. Sparse encoding: most buckets are empty in any one window (the demo's day touches 73 of the 284), so store only non-empty buckets as (index, count) pairs.

Worked example

The script below simulates 24 hours of one-per-second samples (lognormal around 10 ms, a skewed shape typical of latency with most samples near the middle and a long tail of slow ones, with five minutes of 12x congestion in hour 14), stores hourly histograms and compares three answers for the day's p99. It is complete and seeded.

import math, random
random.seed(59)
HOURS=24; N=3600
def true_q(v,q):
    v=sorted(v); return v[math.ceil(q*len(v))-1]
hours=[]
for h in range(HOURS):
    base=[random.lognormvariate(math.log(10),0.25) for _ in range(N)]   # ms
    if h==14:  # 5 minutes of congestion
        for i in range(1200,1500): base[i]*=12
    hours.append(base)
allv=[x for h in hours for x in h]
R=1.05
def bucket(x): return int(math.floor(math.log(x,R)))
def mid(b): return R**(b+0.5)
hist=[{} for _ in range(HOURS)]
for h,vals in enumerate(hours):
    for x in vals: hist[h][bucket(x)]=hist[h].get(bucket(x),0)+1
def hq(hists,q):
    m={}
    for hh in hists:
        for b,c in hh.items(): m[b]=m.get(b,0)+c
    tot=sum(m.values()); need=math.ceil(q*tot); run=0
    for b in sorted(m):
        run+=m[b]
        if run>=need: return mid(b)
print("true p99 over 24h raw: %.2f ms" % true_q(allv,0.99))
print("merged-histogram p99:  %.2f ms" % hq(hist,0.99))
hp=[true_q(v,0.99) for v in hours]
print("mean of 24 hourly p99: %.2f ms" % (sum(hp)/24))
print("hourly p99 of hour 14: %.2f ms, max hourly: %.2f" % (hp[14],max(hp)))
print("worst-case relative error of a bucket midpoint: %.2f%%" % ((math.sqrt(R)-1)*100))
print("distinct buckets used: %d" % len({b for hh in hist for b in hh}))
lo,hi=0.01,10000.0   # 10 us to 10 s in ms
print("buckets spanning 10us-10s at ratio 1.05:", math.ceil(math.log(hi/lo,R)))
yrs_hours=int(5*365.25*24)
print("hourly windows in 5 years:",yrs_hours)
row = 75 * 8   # budget 75 non-empty buckets per window (the demo used 73), 8 bytes each (index + count)
hourly_30d = 30 * 24
daily_rest = int(5 * 365.25) - 30
per_ip = (hourly_30d + daily_rest) * row
print("per IP: hourly for 30 days + daily for the rest = %.2f MB" % (per_ip / 1e6))
print("100,000 IPs: %.0f GB" % (per_ip * 100000 / 1e9))
print("raw 1/s samples, 8 bytes each, 100,000 IPs, 5 years: %.0f TB" % (100000 * 5 * 365.25 * 86400 * 8 / 1e12))
raw7 = 7 * 86400 * 8   # the 7-day raw tier: 1 sample per second, 8 bytes each
print("raw 7-day tier per IP: %.2f MB, 100,000 IPs: %.0f GB" % (raw7 / 1e6, raw7 * 100000 / 1e9))
total = per_ip + raw7
print("whole design, 100,000 IPs: %.0f GB, %.0f times smaller than 5 years raw" % (total * 100000 / 1e9, 100000 * 5 * 365.25 * 86400 * 8 / (total * 100000)))

Output:

true p99 over 24h raw: 18.59 ms
merged-histogram p99:  18.23 ms
mean of 24 hourly p99: 24.12 ms
hourly p99 of hour 14: 166.82 ms, max hourly: 166.82
worst-case relative error of a bucket midpoint: 2.47%
distinct buckets used: 73
buckets spanning 10us-10s at ratio 1.05: 284
hourly windows in 5 years: 43830
per IP: hourly for 30 days + daily for the rest = 1.51 MB
100,000 IPs: 151 GB
raw 1/s samples, 8 bytes each, 100,000 IPs, 5 years: 126 TB
raw 7-day tier per IP: 4.84 MB, 100,000 IPs: 484 GB
whole design, 100,000 IPs: 635 GB, 199 times smaller than 5 years raw

How the merged p99 is read: a day holds 86,400 samples, so the 99th percentile is the 85,536th smallest (ceil of 0.99 x 86,400). Add the merged counts from the lowest bucket upward. Everything below bucket 59 totals 85,159, which is not enough. Bucket 59 (17.79 to 18.68 ms) holds 407 more, taking the running total to 85,566, past 85,536. So the p99 lies in that bucket, and the reported value is its midpoint, 18.23 ms.

The merged histograms give 18.23 ms against a true 18.59 ms (1.9 percent low, inside the 2.47 percent bound). The mean of the 24 hourly p99s gives 24.12 ms, 30 percent too high, because one bad hour dominates the average. The 5-year sizing: the histogram tiers need 1.51 MB per IP, 151 GB for 100,000 IPs. The 7-day raw tier is the larger part: 7 x 86,400 x 8 bytes is 4.84 MB per IP, 484 GB for 100,000 IPs, so the whole design is about 635 GB, roughly 199 times smaller than the 126 TB that 5 years of raw samples would take (assuming 1 sample per second and 8 bytes each, which is an estimate of the input volume). To cut the raw tier, keep it only for the source IPs under investigation.

Questions that become impossible

• Exact values: p99 is only known to within its bucket (about 2.5 percent here), and a value in the top bucket is only known to be "at least" the bucket's edge, apart from the exact stored maximum.
• Re-slicing in time: a window that is not a union of stored windows (for example 09:30 to 10:30) cannot be reconstructed. The congestion in hour 14 shows as 166.82 ms at p99, and the counts show about 300 samples were slow, but not when inside the hour.
• Dimensions not kept: if the key is source IP only, p99 to one destination or port is gone.
• Different bucket boundaries: a new need for 1 percent accuracy cannot be applied to history.
• Rare events in short windows: with fewer than 100 samples in a window, the p99 is just the maximum, so sparse IPs give weak answers.
• Which packet or flow was slow.

Trade-offs

Finer buckets cost bytes and coarser windows lose the shape of time. The mitigation is to keep fine tiers short and cheap-to-query ones long, and to record the bucket scheme and its error bound with the data so a future reader can state the uncertainty. Other mergeable sketches (HDR histogram, t-digest) exist, the property to require is mergeability.

Direct answer

Run every attempt as an asyncio task, cap how many run at once with a semaphore, time only the connection setup with time.perf_counter() (a high-resolution timer for short durations; only the difference between two calls means anything) around asyncio.open_connection, wrap it in a timeout, classify each failure (timeout, refused, DNS or other OS error), and compute percentiles over successful connects only while reporting the failure rate beside them. A p99 computed after silently dropping failures flatters the endpoint, so failures are always reported next to the percentiles.

The script

#!/usr/bin/env python3
"""Concurrent TCP connect-latency probe (standard library only)."""
import argparse
import asyncio
import math
import sys
import time
from collections import Counter


def percentile(sorted_vals, p):
    """Nearest-rank percentile: the smallest value with at least p% of samples at or below it."""
    if not sorted_vals:
        return None
    rank = max(1, math.ceil(p / 100 * len(sorted_vals)))
    return sorted_vals[rank - 1]


def parse_endpoint(text):
    host, sep, port = text.strip().rpartition(":")
    if not sep or not port.isdigit():
        raise ValueError(f"bad endpoint {text!r}, expected host:port or [v6]:port")
    return host.strip("[]"), int(port)


async def probe(host, port, timeout, payload, sem):
    """One attempt. Returns (kind, connect_ms, mbps); kind is 'ok' or a failure class."""
    async with sem:
        t0 = time.perf_counter()
        try:
            reader, writer = await asyncio.wait_for(
                asyncio.open_connection(host, port), timeout)
        except asyncio.TimeoutError:
            return "timeout", None, None
        except ConnectionRefusedError:
            return "refused", None, None
        except OSError as exc:  # DNS failure, unreachable, reset, ...
            return type(exc).__name__, None, None
        connect_ms = (time.perf_counter() - t0) * 1000
        mbps = None
        try:
            if payload:
                t1 = time.perf_counter()
                writer.write(b"x" * payload)
                await asyncio.wait_for(writer.drain(), timeout)
                await asyncio.wait_for(reader.readexactly(payload), timeout)
                secs = time.perf_counter() - t1
                mbps = (2 * payload * 8) / secs / 1e6  # bytes sent plus echoed back
        except (asyncio.TimeoutError, asyncio.IncompleteReadError, OSError):
            pass  # connect succeeded, so keep its latency; throughput stays None
        finally:
            writer.close()
            try:
                await asyncio.wait_for(writer.wait_closed(), 1)
            except (asyncio.TimeoutError, OSError):
                pass
        return "ok", connect_ms, mbps


async def run(endpoints, attempts, timeout, concurrency, payload):
    sem = asyncio.Semaphore(concurrency)
    plan = [ep for ep in endpoints for _ in range(attempts)]
    done = await asyncio.gather(*(probe(h, p, timeout, payload, sem) for h, p in plan))
    results = {ep: [] for ep in endpoints}
    for ep, row in zip(plan, done):
        results[ep].append(row)
    return results


def report(results, payload):
    print(f"{'endpoint':24} {'n':>4} {'ok':>4} {'fail%':>6} {'p50':>8} {'p95':>8} {'p99':>8}  failures"
          + ("  mbps_p50" if payload else ""))
    for (host, port), rows in results.items():
        lat = sorted(r[1] for r in rows if r[0] == "ok")
        fails = Counter(r[0] for r in rows if r[0] != "ok")
        pct = lambda p: "-" if not lat else f"{percentile(lat, p):.1f}"
        line = (f"{host + ':' + str(port):24} {len(rows):>4} {len(lat):>4} "
                f"{100 * (len(rows) - len(lat)) / len(rows):>6.1f} "
                f"{pct(50):>8} {pct(95):>8} {pct(99):>8}  {dict(fails) or '-'}")
        if payload:
            mb = sorted(r[2] for r in rows if r[2] is not None)
            line += f"  {percentile(mb, 50):.1f}" if mb else "  -"
        print(line)


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("endpoints", nargs="+", help="host:port or [v6]:port")
    ap.add_argument("-n", "--attempts", type=int, default=50)
    ap.add_argument("-t", "--timeout", type=float, default=2.0)
    ap.add_argument("-c", "--concurrency", type=int, default=20)
    ap.add_argument("--payload-bytes", type=int, default=0,
                    help="also send this many bytes to an echo service and time the round trip")
    a = ap.parse_args()
    try:
        eps = [parse_endpoint(e) for e in a.endpoints]
    except ValueError as exc:
        sys.exit(str(exc))
    results = asyncio.run(run(eps, a.attempts, a.timeout, a.concurrency, a.payload_bytes))
    report(results, a.payload_bytes)


if __name__ == "__main__":
    main()

Usage: python3 tcpprobe.py 192.0.2.10:443 [2001:db8::1]:443 -n 100 -c 20 -t 2. Add --payload-bytes 65536 against an echo service (a server that sends back whatever it receives) to get the per-connection round-trip throughput the same way.

What the script prints

#!/usr/bin/env python3
"""Minimal TCP echo server on 127.0.0.1:9001 for testing tcpprobe.py."""
import asyncio


async def handle(reader, writer):
    try:
        while data := await reader.read(65536):
            writer.write(data)
            await writer.drain()
    except OSError:
        pass
    writer.close()


async def main():
    server = await asyncio.start_server(handle, "127.0.0.1", 9001)
    async with server:
        await server.serve_forever()


asyncio.run(main())

Save the script above as tcpprobe.py and this one as echo.py. Start the echo server with python3 echo.py &, then run python3 tcpprobe.py 127.0.0.1:9001 127.0.0.1:9002 -n 100; nothing listens on port 9002, so every attempt there is refused. Add --payload-bytes 65536 to the same command to exercise the throughput column.

The output of that command (loopback, so the times are tiny and will differ on every run; the values below are illustrative):

endpoint                    n   ok  fail%      p50      p95      p99  failures
127.0.0.1:9001            100  100    0.0      0.5      0.8      1.2  -
127.0.0.1:9002            100    0  100.0        -        -        -  {'refused': 100}

Reading the columns: n is attempts, ok is successful connects, fail% is (n minus ok) divided by n, p50, p95 and p99 are connect times in milliseconds over the successful attempts only, and failures counts each failure class. The second row shows why ok and fail% sit beside the percentiles: with no successes there is nothing to take a percentile of, so the cells show - and the failure column carries the story. With --payload-bytes 65536 one more column, mbps_p50, shows the median round-trip throughput.

How the asyncio pieces fit

asyncio runs many tasks in one thread: a task runs until it hits an await, then hands control to another, so hundreds of connection attempts overlap without threads.

• async with sem takes one permit from the semaphore (a counter of free slots, here -c); a task that finds none waits, and the permit is returned automatically when the block ends, even after an error.
• asyncio.wait_for(x, timeout) cancels x and raises asyncio.TimeoutError if it has not finished in time; that is how a silent host becomes a timeout row instead of hanging the run.
• writer.drain() waits until the data handed to write has been pushed out to the network, and reader.readexactly(n) waits for exactly n bytes back (and raises IncompleteReadError if the connection closes first).
• asyncio.gather(...) starts all the probe tasks and returns their results in the order given, which is what lets zip(plan, done) pair each result with its endpoint.

Why it is built this way

• What is timed. open_connection returns when the TCP three-way handshake (client sends SYN, server answers SYN-ACK, client sends ACK) completes, so the figure is roughly one network round trip plus local overhead. If the host is a name, DNS resolution is inside that time. Probe addresses, or accept that resolution is part of the figure.
• Concurrency is capped. Without the semaphore, hundreds of simultaneous attempts queue inside the client and inflate the numbers it is trying to measure. -c bounds the client-side load; the tasks for all endpoints share it.
• Failures are classified, not merged. refused means the host answered with a reset (path works, service down). timeout means no answer within the limit (filtered or lost). gaierror is a DNS failure. They have different owners.
• Percentiles use the nearest-rank rule (rank = ceil(p/100 x n)). With few successes p99 equals the maximum, which is why n and ok are printed.
• Throughput is optional and honest. It needs an echo service to read the bytes back, covers the bytes in both directions, and a failure there keeps the connect sample.
• Cleanup is explicit. writer.close() runs in a finally, so a failed read does not leak sockets.

Self-check of the percentile function

from tcpprobe import percentile
vals = sorted([12, 15, 11, 14, 13, 90, 12, 16, 11, 250])
print(vals)
for p in (50, 95, 99):
    print(p, percentile(vals, p))
print(percentile([], 50))

Output when run:

[11, 11, 12, 12, 13, 14, 15, 16, 90, 250]
50 13
95 250
99 250
None

With 10 samples the 5th value (13) is the p50, and the p95 and p99 ranks both round up to 10, the maximum. That is the arithmetic by hand: ceil(0.95 x 10) = 10.

Pitfalls

• Probe from where users are; loopback or same-rack numbers say nothing about the WAN.
• Alert on a sustained change in p95 or failure rate over several runs, never on one sample. Percentiles of 20 attempts are noisy.
• Repeated short connections can trip rate limits or intrusion detection on the target, so coordinate and keep the attempt rate modest.