Apple Network Engineer Interview Preparation Guide - Entry Level

Program summary
Run controlled, hypothesis-driven chaos experiments that progress from lab → staging → limited production canary → broader production. Start small, automate safety checks, measure observable impact, and feed learnings into runbooks and network design.

Example experiments

• Link failure: shut a leaf-spine link or VRF interface in staging; in prod run on a single non-critical PoP.
• Inject latency/jitter: use Linux tc or network emulator on path between services.
• BGP route changes: withdraw a route from a test ASN or simulate route flap via routing policy on a single router.

Risk controls

• Blast-radius limits: scope via PoP, fabric pod, or prefix-set; avoid edge-customer-facing circuits early.
• Approval & scheduling: change control ticket + ops lead sign-off; run in maintenance windows for higher-risk tests.
• Safety kill-switch: automated rollback if thresholds exceeded (packet loss, P50/P99 latency, control-plane instability).
• Pre-checks: health gates (redundant paths, BFD up, control-plane stable) before test; post-check validation before expanding scope.
• Use simulation/lab and BGP sandbox where possible; never inject into primary transit without approval.

Escalation & runbook

• Define thresholds for automated alerting (e.g., >1% loss, BGP neighbor down >30s).
• On alarm: paging to on-call network engineer → senior network/SRE → principal engineer. Include specific mitigation steps: undo ACLs, re-enable interfaces, clear routes, BGP clear, rollback config.
• Postmortem owner assigned within 24h.

Scheduling & cadence

• Weekly small experiments in staging; monthly canaries; quarterly broad resilience tests.
• Tie schedule to release cycles; avoid peak business periods.

Observability requirements

• Control-plane: BGP session state, route propagation times, RIB/FIB diffs, BFD.
• Data-plane: latency/packet-loss (active probes, ping/synthetic), NetFlow/sFlow, per-link utilization.
• Metrics: Prometheus/Grafana dashboards, alerts, logs, SNMP traps, packet captures when needed.
• Correlate app-level SLOs (error rate, latency) to network events.

Measuring learning & improvements

• Define hypotheses (e.g., "Link flap should not affect customer SLA > 99.9%") and success criteria before test.
• Track MPD/MTTD/MTTR, routing convergence time, packet-loss spike magnitude, number of impacted prefixes/services.
• Convert findings into action items: config changes, automation (fast failover, adjust BFD timers), capacity upgrades, runbook updates, and repeat tests to validate fixes.
• Maintain a chaos playbook and an experiment register with outcomes and owners.

This program balances safety with rigorous validation to iteratively strengthen network resilience.

Direct answer

Use ss (or the older netstat) to see exactly what's listening, on which address and port, and confirm whether the bind is scoped to loopback only or to all interfaces, since that single distinction explains a large share of it's-listening-but-nothing-external-can-connect reports. Beyond the bind address and process ownership, a separate class of local blockers, the host firewall, SELinux, and network namespaces, can each independently prevent an otherwise healthy listener from being reached, and each needs its own specific check rather than being lumped in with the network.

Structured elaboration

1. List listening sockets with process ownership: ss -ltnp (listening, TCP, numeric, show process) lists every listening TCP socket along with the PID and process name holding it; this immediately answers whether anything is actually listening on this port, and whether it's the process you expect.
2. Read the local address field carefully: 0.0.0.0:8080 means the process is listening on all interfaces and is reachable from outside the host (firewall permitting); 127.0.0.1:8080 means it is bound only to loopback and is fundamentally unreachable from any other host, no matter what firewall rules say, because the OS never even considers external interfaces for that socket.
3. Check established connections and their state with ss -tn (without the -l) to see active connections; a large number stuck in SYN-RECV suggests the three-way handshake is not completing (possibly a firewall dropping the client's ACK, or a backlog queue issue on the server), while a large number in TIME_WAIT on a busy server is often benign churn rather than a problem, unless it is approaching ephemeral port exhaustion.
4. Check the host firewall explicitly, as its own distinct layer from anything upstream: on iptables-based hosts, iptables -L -n -v (or iptables -S) shows whether a rule is dropping or rejecting the port in question, and on firewalld-based hosts, firewall-cmd --list-all shows the active zone's allowed services and ports; a socket can be correctly listening on 0.0.0.0 and still be unreachable purely because the host's own firewall drops the inbound SYN before it ever reaches that socket.
5. Check SELinux (on systems that enforce it) as a separate, non-firewall local blocker: getenforce confirms whether SELinux is enforcing at all, and if it is, a service listening on a non-standard port that was never labeled for that port's SELinux port-type will be denied at the kernel security-module level even though the bind and firewall are both correct; ausearch -m avc -ts recent (or sealert on systems that have it) surfaces the specific denial, and semanage port -l | grep <port> shows what port-type is currently associated with that port, with semanage port -a -t <type> -p tcp <port> as the fix once the correct type is identified.
6. Check network namespaces on containerized or namespace-isolated hosts: a process can be listening perfectly well, but inside a different network namespace than the one you are inspecting from (for example inside a container's own namespace rather than the host's default namespace), so ss run in the wrong namespace will show nothing at all, not a loopback-only bind or a blocked port. ip netns list enumerates namespaces on the host, and ip netns exec <ns> ss -ltnp (or nsenter --net=<path> ss -ltnp) runs the same check inside the namespace that actually owns the socket; a 0.0.0.0 bind inside a container's namespace is still only reachable from outside according to whatever port-publishing or bridging rule (for example Docker's own iptables-based NAT rules) connects that namespace to the host and the network beyond it.
7. Distinguish not-listening-at-all from listening-but-blocked-further-out: if ss -ltnp shows nothing on the expected port in the correct namespace, the application itself never started or crashed, a fact entirely independent of firewalls, SELinux, or networking; if it does show a correct, externally-bound listener and external clients still cannot connect, the fault has moved to one of the local blockers above, or to something outside the host entirely.

Worked example

ss -ltnp shows LISTEN 0 128 127.0.0.1:8080 0.0.0.0:* users:(("myapp",pid=4521,fd=6)). The process is confirmed running and listening, but bound specifically to 127.0.0.1, loopback only; changing the bind address to 0.0.0.0 is the first fix, before any firewall or SELinux check is even relevant. Suppose instead ss -ltnp shows the same process correctly bound to 0.0.0.0:8080, and external clients still cannot connect. iptables -L -n -v on the host shows no DROP or REJECT rule referencing port 8080, ruling out the host firewall. getenforce reports Enforcing, and ausearch -m avc -ts recent shows an AVC denial for that process attempting to bind to port 8080, because the application was reconfigured to use a nonstandard port that was never added to SELinux's port-type list for that daemon; semanage port -a -t http_port_t -p tcp 8080 (or the appropriate type for the service) resolves it. In a third case, the process runs inside a container; ss -ltnp on the host shows nothing at all for port 8080, not because the process is not listening but because it is listening inside the container's own network namespace; docker exec <container> ss -ltnp (or ip netns exec <ns> ss -ltnp) confirms the process is listening correctly inside its namespace, and the actual question becomes whether the container's port-publishing rule correctly maps the host port to it.

Trade-offs & pitfalls

It is easy to see the process is listening in ss output and conclude the service is correctly exposed, without reading the bind address carefully enough to notice it is loopback-only, or without checking that you are even looking in the right network namespace. Also do not confuse ss -ltnp's absence of a listener with a firewall or SELinux problem; if nothing is listening in the namespace you are inspecting, no firewall rule or SELinux policy anywhere will make the connection succeed, and time spent checking those first is wasted until is-it-listening-and-where is ruled out. SELinux denials in particular are easy to miss because the application logs may show nothing useful at all (the bind or connection attempt is blocked below the application's own visibility), so always check ausearch or the audit log specifically once a correctly-bound, correctly-firewalled listener still is not reachable.

Direct answer

Work from the physical layer up, and compare both ends at each step: link state and speed, then the negotiation mode and protocol (LACP or PAgP), then member-port consistency (speed, duplex, access VLAN or trunk settings, native VLAN, allowed VLANs), then the port-channel interface itself. Members get suspended when the switch detects a mismatch, so the first command on both sides is show etherchannel summary. When members sit on different physical switches, ordinary EtherChannel (Cisco's name for bundling several physical Ethernet links into one logical link, which Cisco describes as fault-tolerant and high-speed) does not work unless those switches act as one logical device (a switch stack, or a multichassis feature such as vPC on Nexus).

LACP (Link Aggregation Control Protocol, labelled IEEE 802.3ad in Cisco's guide; link aggregation is now specified in IEEE 802.1AX) is the standard negotiation; PAgP (Port Aggregation Protocol) is Cisco's proprietary one.

Ordered checks (do each on both switches)

1. State and flags. show etherchannel summary. The letters that matter first are P (healthy, bundled), s (suspended, a configuration mismatch) and D (down); the rest are secondary. Illustrative output in Cisco's format, with one member suspended:

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi1/0/47(P)  Gi1/0/48(s)

Read it left to right: group 1 is Po1, flags S and U mean a Layer 2 channel that is in use, the protocol is LACP, and member Gi1/0/47 is bundled (P) while Gi1/0/48 is suspended (s), so compare Gi1/0/48's configuration against Gi1/0/47's. Cisco's full flag legend: P is bundled in the port-channel, s is suspended, I is stand-alone (the port is not in any bundle), D is down, H is hot-standby (LACP only), w is waiting to be aggregated, M means not in use because minimum links are not met, u is unsuitable for bundling, d is the default port, f is failed to allocate aggregator, A is formed by Auto-LAG. U on the port-channel means in use, S means Layer 2, R means Layer 3. An (SD) port-channel is a Layer 2 channel that is down; (SU) is up.
2. Physical. show interfaces status: are all members connected and the same speed and duplex? One member at a different speed will not bundle with the others.
3. Negotiation modes. In LACP, active talks first and passive only answers: active-active and active-passive form a channel, passive-passive never does. In PAgP, desirable and auto follow the same pattern. Mode on forms a channel only against another on, with no negotiation. LACP and PAgP cannot interoperate, so one side set to PAgP (desirable or auto) and the other to LACP stays down. Check channel-group <n> mode ... in the config on both sides, and channel-protocol if used.
4. Member consistency. The ports must share the same speed and duplex, the same access VLAN or the same trunk configuration, the same native VLAN, and the same allowed VLAN list. Cisco states that when misconfigurations are detected in a port mode or VLAN mask, the ports are suspended. In the same-trunk-settings line, native VLAN means the VLAN whose frames cross the trunk untagged, and the allowed VLAN list is the set of VLANs the trunk may carry. Compare show running-config interface for each member against the others and against the far end.
5. LACP view. show lacp neighbor (does the partner answer, with which system) and show lacp internal (local state of each member), and show etherchannel <n> detail for per-port detail.
6. Port-channel interface. Configuration on the port-channel interface applies to all members, while changes on one physical port apply to that port only, so put the trunk and VLAN settings on the port-channel and keep members identical. Check the port-channel's trunk allowed list against the far end's.
7. Limits. An LACP bundle can have up to 16 member ports, of which at most eight are active and up to eight are hot-standby (flag H). A ninth link that sits in standby is by design, not a fault. Hot-standby means ready to join the bundle if an active member fails. A separate minimum-links setting can require a minimum number of healthy members before the channel is used; below it the flag is M.

Typical fixes

interface GigabitEthernet1/0/47
 switchport mode trunk
 channel-group 1 mode active
interface GigabitEthernet1/0/48
 switchport mode trunk
 channel-group 1 mode active
interface Port-channel1
 switchport mode trunk
 switchport trunk allowed vlan 10,20,30,40

The same channel-group 1 mode active (or passive on one side) on the far switch, with the same trunk settings, makes show etherchannel summary show the members with flag P.

Members on different physical switches

Plain EtherChannel assumes every member goes to the same partner device, so two separate switches look like two different partners and the bundle cannot form across them. There are two working designs:

• A switch stack, where several physical switches are managed as one. Cisco documents that EtherChannels can span members of one stack, so one port on each stack member can be in the same channel.
• vPC (virtual port channel) on Cisco Nexus, a Nexus-specific and more advanced option in which links to two Nexus switches appear as one port channel to the third device. The checks above remain the core for any bundle. Failure modes: a Type 1 configuration mismatch between the two peers can stop the vPC or its member ports from coming up (check show vpc, show vpc brief and show vpc consistency-parameters), the peer link carries synchronisation and the peer-keepalive link detects a dead peer. LACP in active mode is the usual choice on the vPC member interfaces.

On two ordinary independent switches, use separate links with spanning tree or routed uplinks instead, and do not try to bundle.

Pitfalls

• Changing one member's trunk settings after the bundle is up can leave that member suspended while the others stay bundled, so the summary shows both P and s.
• Leaving on on one side while the other side runs LACP is a common reason a bundle stays down, because on only forms a channel against another on.
• A bundle with fewer healthy members than minimum links is flagged M, not P.

Direct answer

At 50,000 concurrent users across regions, treat authentication and tunnel termination as two separately-scaled problems: federate and shard the identity layer so no single authentication service is a global bottleneck, terminate tunnels at regional points of presence close to each user rather than backhauling everyone to one location, and pick the tunnel protocol based on per-session overhead and how gracefully it handles a client moving networks, favoring WireGuard (or a WireGuard-based concentrator) over classic Internet Protocol security (IPsec)/Internet Key Exchange (IKE) for this profile, with a Transport Layer Security (TLS)-based fallback for restrictive networks.

Structured elaboration

flowchart TB
    U[Remote users] --> DNS[Geo DNS or anycast broker]
    DNS --> R1[Region A concentrators]
    DNS --> R2[Region B concentrators]
    R1 --> AUTH1[Region A auth front end]
    R2 --> AUTH2[Region B auth front end]
    AUTH1 --> IDP[Central identity provider]
    AUTH2 --> IDP
    R1 --> APPS[Internal resources]
    R2 --> APPS

1. Protocol choice. IPsec/IKE is standard and broadly supported, but its per-session negotiation state and rekey overhead add up at very large concurrent-session counts, and it recovers slowly when a client's network changes (Wi-Fi to cellular). A TLS-based VPN benefits from looking like ordinary HTTPS through almost any firewall, but still carries a full TCP-plus-TLS stack's overhead per session. WireGuard keeps minimal per-connection state (a peer is just a public key mapped to allowed IP ranges, with nothing to renegotiate), runs over UDP, and updates a peer's known address automatically when a valid packet arrives from a new source, which handles client roaming gracefully. Recommendation: WireGuard for the bulk of traffic, with a TLS-based VPN fallback for legacy clients or networks that only permit outbound HTTPS.
2. Authentication architecture, sharding and federation. A single centralized authentication service handling all 50,000 users' logins and periodic re-authentication would be both a bottleneck and a single point of failure. Instead, deploy regional authentication front ends that validate short-lived tokens locally, issued by a central identity provider through a federated protocol such as OpenID Connect against the corporate identity system, so only the relatively infrequent token-issuance step reaches the central service, while the frequent per-connection check is a local signature verification, not a network round trip. Shard session state by region so a regional outage affects only users natively assigned there.
3. Session brokering and load balancing. A lightweight broker, DNS-based geolocation routing or anycast, directs each client to its nearest regional concentrator cluster. Within a region, a load balancer distributes new connections using a metric that reflects real capacity (concurrent-session count and crypto throughput), since a concentrator's limit is sessions and crypto work, not raw request rate.
4. Regional ingress and egress placement. Terminate tunnels close to users to minimize setup and ongoing latency. Egress placement is a separate decision driven by where internal resources actually live: if those resources are centralized, each regional point of presence needs a fast backbone path back to them, which is exactly where the split-tunnel decision below has the largest impact.
5. Split tunneling versus forced tunneling. Split tunneling routes only traffic destined for internal resources through the VPN and lets everything else, general browsing, video calls, other software-as-a-service traffic, go directly out the client's own connection. Forced tunneling routes all client traffic through the VPN and out corporate egress regardless of destination. Split tunneling dramatically reduces the bandwidth and latency load on the VPN infrastructure, at 50,000 concurrent users this is often the difference between a feasible and infeasible egress capacity requirement, but it means corporate controls like web filtering, data-loss-prevention inspection and centralized logging never see that split-off traffic, a real visibility trade-off, not just a performance one. Forced tunneling preserves full visibility but multiplies concentrator and egress bandwidth by however much non-corporate traffic each client generates, usually the larger cost driver at this scale. A common middle ground is split tunneling with a defined, audited exception list.
6. Key management. WireGuard's per-peer static keypairs (or IPsec certificates) need a provisioning and rotation pipeline tied to the same identity system as authentication: issue a new keypair when a device is enrolled through mobile device management, revoke it promptly when a device or user is deprovisioned. Regional concentrators need a fast path to learn about revocations, since a compromised or terminated-employee device staying valid for hours across every region is a meaningful exposure window.
7. NAT and edge constraints. WireGuard and IPsec both need UDP to reach the concentrator, which some restrictive corporate, hotel or airport networks block or throttle. A fallback path over TCP port 443 needs to exist for clients on those networks, at some cost to the primary protocol's throughput and latency advantages.
8. Client performance considerations. Keepalive intervals need tuning against mobile battery life (frequent keepalives keep NAT mappings alive but drain battery faster) versus how quickly a dead session is detected and failed over. WireGuard's cheap, stateless handshake is an advantage here: a client can drop and rejoin cheaply compared to a heavier IKE renegotiation.
9. Telemetry for large-scale troubleshooting. Per-region dashboards of concurrent sessions, tunnel-setup latency (P50/P95/P99, the 50th/95th/99th percentile), and concentrator resource utilization; per-user session history recording which regional point of presence, connect and disconnect times, and disconnect cause (client-initiated, keepalive timeout, server-side eviction); and aggregate authentication latency and failure rate split by region, so a regional identity-provider or network issue shows up as an isolated regional anomaly rather than a confusing global one.

Worked example

50,000 concurrent users spread evenly across 4 regions averages 12,500 sessions per region. If each concentrator instance is rated for a conservative 3,000 concurrent WireGuard sessions (a modest figure given WireGuard's small per-session footprint), a region needs

$\left\lceil \frac{12{,}500}{3{,}000} \right\rceil = 5 \text{ instances at steady state}$

plus at least one more for N+1 redundancy, six instances per region, twenty-four total, a number the capacity-planning and autoscaling policy should be built around directly rather than discovered after an outage.

Trade-offs and pitfalls

WireGuard's lightweight scaling and roaming behavior come at the cost of some of the enterprise tooling maturity (granular per-application policy, specific compliance certifications) that established IPsec or SSL-VPN (the older industry name for a TLS-based VPN) products have accumulated, worth weighing explicitly for a highly regulated industry. Sharding authentication by region improves resilience but means a traveling user either needs to reauthenticate against the new region's front end or the token-validation public keys need to already be replicated everywhere, an easy detail to miss until someone travels and gets locked out.

Direct answer
The TCP header carries, at minimum, a sequence number and acknowledgment number (for tracking and confirming data), the SYN/ACK/FIN/RST control flags (for connection setup and teardown), a window size (for flow control), and a set of options including MSS (Maximum Segment Size), window scale, SACK-permitted (Selective Acknowledgment), and timestamps (all negotiated at the handshake). If you could only check a few during a performance incident, window size and the options negotiated at the handshake (MSS, window scale, SACK) are the highest-value first checks, since they directly bound how efficiently the connection CAN perform, before even looking at anything dynamic.

Structured elaboration

• Sequence number: identifies the position, in bytes, of this segment's data within the overall byte stream; every byte sent gets a sequence number.
• Acknowledgment number: when the ACK flag is set, indicates the NEXT byte the receiver expects, effectively confirming everything before that point has arrived.
• Flags (SYN/ACK/FIN/RST): SYN initiates a connection, ACK confirms received data (present on nearly every segment after the handshake), FIN requests a graceful close, RST aborts the connection immediately.
• Window size: the receiver's advertised available buffer space (subject to the negotiated window SCALE factor from the handshake), the mechanism behind flow control.
• Options (MSS, window scale, SACK-permitted, timestamps): negotiated ONLY in the SYN/SYN-ACK exchange and fixed for the connection's lifetime; MSS caps the largest single segment, window scale extends the effective window size beyond the raw 16-bit field, SACK-permitted enables selective (rather than only cumulative) acknowledgment, and timestamps support accurate RTT measurement and protect against stale, wrapped sequence numbers.

Worked example
Triaging a performance incident with limited time, check the negotiated OPTIONS first: if window scale never negotiated successfully (visible by comparing the SYN and SYN-ACK), the connection is capped at a 64KB window for its ENTIRE lifetime regardless of anything else, a hard, structural ceiling worth ruling out before looking at anything dynamic. Then check the CURRENT window size value on live segments (has it collapsed to something small, suggesting a flow-control-limited receiver) alongside the flags (any unexpected RSTs indicating the connection is being torn down and re-established repeatedly, itself a red flag). Sequence and acknowledgment numbers matter most for confirming specific loss/retransmission behavior (comparing them across segments), a more detailed, second-pass check once the higher-level structural questions (options, window, flags) have been ruled out.

Trade-offs & pitfalls
It's easy to over-focus on sequence and acknowledgment numbers first because they feel like "the real data" of TCP's bookkeeping, but for a FIRST-PASS performance triage, the options negotiated once at the handshake (which structurally CAP what the connection can ever achieve) and the live window size (which shows whether that cap is even being approached) are higher-leverage checks, they answer "is there a hard ceiling here" before you spend time analyzing moment-to-moment sequence-level behavior.

Direct answer

Bufferbloat is excess latency caused by queues that are too deep at a bottleneck link. When something keeps sending faster than the link can drain (a bulk TCP transfer), the buffer fills, packets wait in it, and every other flow sharing that queue (voice, video, a web click) waits too. Because TCP loss-based congestion control (the sender keeps speeding up until a packet is lost, and only then slows down) only slows down when it sees a drop, a huge buffer delays the drop signal and lets the queue stay full. The fix is to put the queue where you control it and keep it short: shape traffic slightly below the bottleneck rate and use an active queue management (AQM) algorithm, which drops or marks packets early so the queue stays short, with flow isolation, which gives each flow its own queue so a bulk transfer cannot delay a small flow, such as FQ-CoDel or CAKE.

How it hurts latency and jitter

The delay a packet adds at a full queue is the queue size divided by the link rate:

$\text{queue delay} = \frac{\text{buffer bytes} \times 8}{\text{link rate (bit/s)}}$

Worked example: a 20 Mbit/s uplink with a 1,000,000-byte buffer. A full buffer adds 1,000,000 x 8 / 20,000,000 = 0.4 seconds, 400 ms, on top of a 20 ms base round-trip time (RTT). The bandwidth-delay product (BDP), the amount of data in flight needed to keep the link busy, is 20,000,000 x 0.02 / 8 = 50,000 bytes. The buffer is 1,000,000 / 50,000 = 20 times the BDP, so far more queue than needed to keep the link full.

Jitter (variation in delay) comes from the queue occupancy changing: when a transfer starts the queue grows toward 400 ms, when TCP backs off it drains, and a voice packet arriving at each moment sees a different wait. An idle link shows 20 ms steadily and a busy one swings between 20 ms and about 420 ms.

Detecting it in production

1. Compare latency idle versus loaded. Run continuous active probes (ping or an ICMP/TCP probe) through the suspect link and plot RTT against link utilization. Bufferbloat shows as RTT that climbs with utilization and falls back when it drops, with little or no packet loss until the buffer overflows.
2. Run a controlled test. Flent's RRUL test (Realtime Response Under Load, the standard Bufferbloat project chart showing download and upload speeds plus latency) loads the link in both directions while measuring latency: flent rrul -p all_scaled -l 60 -H <netserver-address> -o result.png. It requires netperf (a network benchmarking tool) and its netserver (the listening program) on the far side. A simpler version is to ping across the link during a large transfer and compare with an idle ping.
3. Read the queue itself. On a Linux router, tc -s qdisc show dev eth0 prints the backlog (bytes and packets waiting) and drops for the queue discipline (qdisc, the rule set that decides how packets wait in a queue and leave it). A persistently large backlog on the egress interface (the outbound side) during complaints is direct evidence. On other platforms use queue depth or occupancy counters if the device exports them through SNMP or streaming telemetry.
4. Correlate with complaints. Calls choppy exactly when a backup or software update runs is the signature. For example, plot ping RTT beside link utilization: RTT sits near 20 ms all morning, jumps to several hundred milliseconds at 09:05 when a backup job starts, and drops back at 09:20 when it finishes. That matching shape is the evidence, and the calls that broke up during those 15 minutes are the complaints it explains.

What the queue looks like in tc output

The worked example can be reproduced in a Linux container (debian:12 with NET_ADMIN) on its loopback interface, which has no real bottleneck, so the script creates one: the interface MTU set to 1500, a token-bucket shaper (tbf, which releases packets at a fixed rate) at 20 Mbit/s with a 1,000,000-byte limit, and a 15 s TCP iperf3 transfer in the background, with ping before and during the transfer.

#!/usr/bin/env bash
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq && apt-get install -y -qq iproute2 iperf3 iputils-ping
ip link set lo mtu 1500
echo "# idle"
ping -c 5 -i 0.2 127.0.0.1 | tail -1
tc qdisc replace dev lo root tbf rate 20mbit burst 4kb limit 1000000
iperf3 -s -D
sleep 1
iperf3 -c 127.0.0.1 -t 15 > /dev/null &
sleep 5
echo "# loaded"
tc -s qdisc show dev lo
ping -c 5 -i 0.2 127.0.0.1 | tail -1
wait

Save it as bloat-demo.sh and run it with docker run --rm --cap-add NET_ADMIN -v "$PWD":/w -w /w debian:12 bash bloat-demo.sh. Counters and round-trip times differ on every run; the printout below is one run, and two other runs showed backlogs of 84,462 and 143,706 bytes and average loaded RTTs of about 98 and 100 ms.

This is the printed output of such a run:

# idle
rtt min/avg/max/mdev = 0.014/0.036/0.050/0.013 ms
# loaded
qdisc tbf 8028: root refcnt 2 rate 20Mbit burst 4Kb lat 398ms 
 Sent 12157962 bytes 11874 pkt (dropped 0, overlimits 9895 requeues 0) 
 backlog 130406b 111p requeues 0
rtt min/avg/max/mdev = 90.140/110.183/136.692/21.185 ms

Reading it line by line. rate 20Mbit is the drain speed, and lat 398ms is the worst-case wait the 1,000,000-byte limit allows, matching the 400 ms worked above (1,000,000 x 8 / 20,000,000). Sent is the cumulative traffic that has left the queue (about 12.2 MB, which is what 20 Mbit/s, 2.5 MB/s, delivers in the roughly 5 seconds since the transfer started), and dropped counts packets thrown away because the queue was full (zero here, because loopback applies back-pressure to the sender before the queue overflows). backlog 130406b 111p is the part that diagnoses bufferbloat: 130,406 bytes in 111 packets are waiting right now, which at 20 Mbit/s is 130,406 x 8 / 20,000,000 = 0.052 s, about 52 ms of delay added for every packet behind them. A healthy queue shows a backlog near zero. The two ping lines confirm it from outside: idle RTT is 0.036 ms and loaded RTT is about 110 ms, a rise of over 100 ms. A ping crosses this loopback queue twice (request and reply both use it), so roughly two passes of the 52 ms backlog, about 104 ms, close to the measured 110 ms; the backlog changes from moment to moment. The backlog stays below the 1,000,000-byte limit here because a single local socket is held back by the host before the queue fills; on a real router with many senders the queue can reach the limit and the 400 ms of the worked example.

Mitigation

1. Make your device the bottleneck. The queue only helps if it forms where your AQM runs. Shape egress to a rate a little below the real link rate (18 Mbit/s, 90 percent, on the 20 Mbit/s uplink above is a starting point, tuned by testing): the modem or provider buffer then never fills.
2. Use AQM with flow isolation. FQ-CoDel (RFC 8290) gives each flow its own queue (1,024 by default) and drops or ECN-marks packets (ECN, explicit congestion notification: the router sets a bit in the packet header so the sender slows down without a drop) when queue delay stays above the target (5 ms default, the delay it is willing to tolerate) for an interval (100 ms default, how long the delay must persist before it acts). At 20 Mbit/s, a 5 ms standing queue (a queue that stays non-empty instead of draining between bursts) is 0.005 x 20,000,000 / 8 = 12,500 bytes, against 1,000,000 before. CAKE (Common Applications Kept Enhanced) combines the shaper, the AQM and flow isolation in one queue discipline.
3. Commands on a Linux router (iproute2):

# FQ-CoDel only (defaults: target 5ms, interval 100ms, 1024 flows)
tc qdisc replace dev eth0 root fq_codel

# Or CAKE with a shaper at 90 percent of the 20 Mbit/s uplink
tc qdisc replace dev eth0 root cake bandwidth 18mbit

4. Enable ECN on hosts, so congestion is signalled without drops where both ends support it: the bufferbloat project's Linux tips list net.ipv4.tcp_ecn=1.
5. Download direction. The queue is upstream of you and you cannot shorten it from your side by shaping egress. Shape ingress (traffic arriving at your router) at the router nearest the bottleneck, or ask the provider. Shaping ingress works because your router deliberately drops or delays arriving packets once they exceed a rate just below the real link speed, so the senders slow down, the arrival rate stays below what the provider's link can drain, and the provider's queue never fills.
6. Verify. Rerun the loaded test: RTT under load should rise by only a few milliseconds above idle instead of the hundreds of milliseconds in the example, and throughput should stay near the shaped rate.

Trade-offs and pitfalls

• Shaping below line rate gives up some capacity (2 of 20 Mbit/s here). That is the price of a controlled queue.
• Simply shrinking buffers to the BDP is crude: it can drop bursts and hurt throughput on high-bandwidth paths. AQM adapts the queue to delay instead of a fixed size.
• Bufferbloat is a property of the bottleneck. Fixing a switch that is not the bottleneck changes nothing, so locate the slowest hop first.
• Data center fabrics trade buffer depth against incast (many servers answering one receiver at the same instant, which briefly overflows a switch queue) in a different way, and the link-level fix above does not carry over unchanged.

Direct answer

A public web application that accepts file uploads and processes them in serverless functions maps cleanly onto STRIDE (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege), and the highest-priority findings concentrate specifically on Tampering and Elevation of privilege, since an untrusted file is, by definition, attacker-controlled content reaching a processing function, which is exactly the shape of threat those two categories describe; the mitigations a solutions architect should recommend are architectural (network and identity boundaries), not code-level fixes the architecture review itself cannot verify.

Structured elaboration

Spoofing. An attacker impersonates a legitimate user to upload a file under someone else's identity, or spoofs the upload event itself to trigger processing without a genuine upload having occurred. Likelihood: medium (requires either a stolen credential or a flaw in the upload-authorization flow); impact: medium (primarily an attribution and audit-trail problem, unless combined with a Tampering finding). Mitigation: strong, short-lived upload-authorization tokens (a pre-signed URL scoped to one specific object key and a short expiration) rather than a broadly-reusable upload credential, and event-source validation in the processing function confirming the triggering event genuinely originated from the expected storage location, not an event a caller crafted directly.

Tampering. The uploaded file's content or metadata (filename, declared content type) is attacker-controlled and used unsafely by the processing function, the highest-priority finding in this threat model. Likelihood: high (this is the architecture's primary attacker-reachable surface); impact: high (can range from a processing function crash to remote code execution, depending on how the function parses the file). Mitigation: validate file type by content inspection, not by trusting the client-declared type or the filename's extension; never construct a file path, a shell command, or a downstream query using the filename or any other attacker-controlled metadata without strict validation first; and run the actual file-parsing logic in an isolated, minimally-privileged execution context.

Repudiation. Without sufficient logging, neither the platform nor the uploading user can later prove or disprove that a specific upload occurred, or what a processing function did with it. Likelihood: high if logging is not deliberately designed in; impact: low to medium on its own, but it compounds every other finding's investigability. Mitigation: log the upload event, the authenticated uploader's identity, and every stage of processing with enough detail to reconstruct what happened to a specific file, shipped to a centralized, tamper-resistant log destination.

Information disclosure. A processing function with an execution role broader than its actual function requires can read more data (other users' uploaded files, unrelated internal resources) than the specific upload it was invoked for; separately, an error message or a debug log accidentally including file content could leak sensitive data uploaded by one user to an operator with no legitimate need to see it. Likelihood: medium; impact: high, since this is a direct data-exposure path. Mitigation: per-function least-privilege execution roles scoped to only the specific object the triggering event names, and structured logging that explicitly excludes file content from log output.

Denial of service. A maliciously crafted or oversized file exhausts the processing function's memory, execution time, or downstream storage, or a flood of upload requests exhausts the platform's processing capacity. Likelihood: medium; impact: medium (availability, not data exposure). Mitigation: enforce a maximum file size before the file is even fully accepted, set function-level timeout and memory limits appropriate to legitimate file sizes, and rate-limit the upload endpoint itself.

Elevation of privilege. A compromised processing function (through a successfully exploited Tampering vulnerability) uses its own execution role to reach further than the immediate file it was invoked to process, the second-highest-priority finding, since it is the direct consequence of a successful Tampering attack turning into broader account access. Likelihood: medium (requires a prior successful Tampering exploit as the precondition); impact: high (turns a single-file compromise into a broader account compromise). Mitigation: the same per-function least-privilege role scoping named under Information disclosure, which is the single control doing the most work across both categories.

Prioritization by likelihood and impact

Tampering is prioritized highest (high likelihood, high impact, and the entry point every other high-impact finding in this model depends on). Elevation of privilege is prioritized second, specifically because it is what determines how bad a successful Tampering exploit actually becomes, the multiplier effect named in the elaboration above. Information disclosure and Denial of service follow as independently medium-to-high priority findings. Spoofing and Repudiation, while genuine findings, are lower standalone priority, since their impact is largely contingent on or compounds one of the other categories rather than being independently severe.

Architectural mitigations a solutions architect should recommend

Least-privilege, per-function execution roles (the single highest-leverage architectural control here, addressing both Information disclosure and Elevation of privilege at once); content-based file-type validation happening in a dedicated, isolated validation step before any business-logic processing touches the file; short-lived, narrowly-scoped upload authorization; centralized, tamper-resistant logging covering the full upload-to-processing lifecycle; and explicit size, timeout, and rate limits enforced at the platform edge, not left to the processing function's own default behavior.

Worked example

A file-sharing application's processing function extracts metadata from uploaded documents and stores the results in a database. An attacker uploads a file with a crafted filename containing a path-traversal sequence, exploiting the processing function's unsanitized use of that filename to write its output somewhere outside the intended location, a direct Tampering exploit. Because the function's execution role happens to be scoped broadly (shared across several processing functions "for simplicity"), the attacker's crafted output path lands in a location the function's role can write to, but that a properly-scoped, per-function role would not have permitted, turning a single Tampering finding into an Elevation-of-privilege finding as well. The architectural fix recommended is not a single patch to this one function's filename handling (a code-level fix outside this review's own scope, though also necessary), but the broader architectural correction: per-function role scoping across every processing function in the pipeline, so the next Tampering vulnerability discovered in a different function does not have the same broader-than-necessary blast radius this one did.

Trade-offs and pitfalls

• A threat model that stops at listing STRIDE categories independently, without tracing how a Tampering finding becomes an Elevation-of-privilege finding once it succeeds, misses the compounding relationship that actually determines real-world severity, exactly what the worked example demonstrates directly.
• A solutions architect's recommendations need to stay at the architectural level (role scoping, isolation boundaries, platform-level limits) rather than prescribing a specific code fix for a specific function, since the architectural review's own scope and expertise is the system's structure, not auditing every function's internal code; the worked example's filename-handling bug still needs a code fix, but the review's own deliverable is the broader per-function-role-scoping recommendation that limits the next such bug's impact too.
• Repudiation and Spoofing are genuinely lower standalone priority, and that ranking can be mistaken for "not worth fixing," when actually their value is specifically in supporting investigation of the higher-priority findings; without adequate logging (addressing Repudiation), an actual Tampering exploit in production is far harder to detect and investigate after the fact, even though Repudiation itself was ranked lower.
• A shared execution role "for simplicity" across multiple processing functions, as in the worked example, is a common, well-intentioned shortcut that directly converts what should be an isolated, single-function compromise into an account-wide risk; the cost of per-function role authoring is real but is precisely what the highest-priority finding in this model depends on to stay contained.

Direct answer

A BGP community is a 32-bit tag attached to a route as an optional transitive attribute (RFC 1997): a label that travels with the route from network to network, so routers that do not know what it means still pass it along. By convention it is written AA:NN, where the first two octets (an octet is 8 bits, so 16 bits each) are an autonomous system number (AS, a network under one administration) and the last two octets mean whatever that AS says they mean. Operators use communities to carry policy: a customer tags a route and the provider acts on the tag (set local preference, prepend, limit propagation, blackhole). To prepend is to add your own AS number to the route's AS path extra times so the path looks longer and less attractive; to blackhole is to deliberately drop traffic for an address. A peer is a router at the other end of a BGP session. Well-known values are defined by standards: the three RFC 1997 values (NO_EXPORT, NO_ADVERTISE, NO_EXPORT_SUBCONFED) must be honored by every compliant router, while later registered values such as BLACKHOLE (RFC 7999) and GRACEFUL_SHUTDOWN (RFC 8326) only take effect where the receiving operator has chosen to act on them.

Well-known communities

Provider-defined communities

Each provider publishes its own table; the values below are an illustrative scheme from a made-up provider using AS 64500, which is in the range reserved for documentation (RFC 5398). They are not any real provider's values, so a real design reads the provider's published list.

An RFC 1997 well-known community such as NO_EXPORT drives policy without any prior agreement between the two networks, because every compliant router honors it. Example: a customer wants a provider to carry its prefix to the provider's own routers but not hand it to other networks. The customer attaches NO_EXPORT on the session to the provider (a plain set community no-export in the outbound route-map, together with neighbor ... send-community), and the provider's routers keep the route inside the provider's AS.

To trace one provider-defined tag: the customer (AS 64511) sets 64500:80 on its route, the tag crosses the session, the provider's inbound policy (the route-map below) matches it and sets local preference 80, and on the provider's router the entry shows it (illustrative output; older IOS releases print the community as one decimal number unless ip bgp-community new-format is set):

BGP routing table entry for 198.51.100.0/24, version 7
  64511
    192.0.2.2 from 192.0.2.2 (192.0.2.2)
      Origin IGP, metric 0, localpref 80, valid, external, best
      Community: 64500:80

The localpref 80 and the Community: 64500:80 lines are the proof that the tag arrived and the policy fired. If the Community line is missing, the customer's session is not sending communities.

Operators also use their own communities internally to tag where a route was learned (customer, peer, transit, region) so export policy can be written as "send only customer-tagged routes to peers".

Configuration (Cisco IOS)

Syntax confirmed in Cisco's BGP documentation. By default no communities attribute is sent to a neighbor, so neighbor send-community is required (the optional keyword picks standard, extended or both; standard is the default):

ip community-list standard BACKUP permit 64500:80
!
route-map FROM-CUSTOMER permit 10
 match community BACKUP
 set local-preference 80
route-map FROM-CUSTOMER permit 20
 set local-preference 100
!
router bgp 64500
 neighbor 192.0.2.2 remote-as 64511
 neighbor 192.0.2.2 route-map FROM-CUSTOMER in

Line by line: the ip community-list names the tag to look for (64500:80); in the route-map, sequence 10 matches routes carrying that tag and lowers their local preference to 80, and sequence 20 catches everything else at 100 (without it, routes that fail the match would be dropped); the neighbor lines bind the policy to the session for routes arriving (in).

On the customer side (AS 64511), set community 64500:80 additive in an outbound route-map adds the tag without erasing communities already on the route; without additive it replaces them. Pair it with neighbor ... send-community on the session.

Pitfalls

• Forgetting send-community, so the tag never leaves the router and the policy silently does nothing.
• Replacing instead of adding (additive omitted) and wiping a tag another team set.
• Trusting communities from customers blindly: a provider should filter which communities a customer may set, since a customer able to set 64500:9999 or the blackhole value could do harm.
• Large communities (RFC 8092) hold three 4-octet values, for networks with 4-octet AS numbers that do not fit in the two-octet AA field.

Direct answer

Use NAT64 (a stateful translator from IPv6 to IPv4) together with DNS64 (a DNS feature that invents AAAA records for IPv4-only names). The IPv6-only client asks for a name, DNS64 returns a synthesized IPv6 address that embeds the server's IPv4 address inside a translation prefix, the client sends to that address, the network routes it to the NAT64 gateway, and the gateway turns it into an IPv4 packet to the real server.

Where the pieces sit

• DNS64 lives in the recursive resolvers that the IPv6-only clients are configured to use. It queries for AAAA, and if there is none but there is an A record, it builds the AAAA.
• NAT64 gateway (a router, firewall or dedicated appliance) sits at the boundary between the IPv6-only network and the legacy IPv4 network. For internal legacy servers that is the data centre or server-farm edge; for Internet destinations it is the Internet edge. It needs a route for the translation prefix, a pool of IPv4 addresses to use as the source, and enough capacity, so deploy at least two, with ECMP (equal-cost multipath, the router sharing traffic across equal routes) or anycast (the same address announced from both gateways so traffic goes to the nearest one that is up).

How an IPv4 destination is represented

The synthesized address is a prefix plus the 32-bit IPv4 address (RFC 6052). Two prefix choices:

• The well-known prefix 64:ff9b::/96. Example: 192.0.2.33 becomes 64:ff9b::c000:221 (each IPv4 octet in hex: 192 = c0, 0 = 00, 2 = 02, 33 = 21, written as the two 16-bit groups c000 and 0221). RFC 6052 says this prefix MUST NOT represent non-global IPv4 addresses such as RFC 1918 ones, and translators must drop such packets.
• A network-specific prefix from your own space, for example 2001:db8:64::/96. RFC 6052 requires one when the IPv4 servers are private, which internal legacy servers are. Example: 10.20.30.40 becomes 2001:db8:64::a14:1e28 (10 = 0a, 20 = 14, 30 = 1e, 40 = 28, giving the groups 0a14 and 1e28).
  So for internal RFC 1918 servers you must use the network-specific prefix, route it to the NAT64 gateway, and keep the well-known prefix for global Internet destinations.

Check: a minimal BIND (the open-source DNS server) configuration with the line dns64 2001:db8:64::/96 { clients { any; }; mapped { any; }; }; inside its options block (the statement is valid only in options or a view) and a zone containing erp IN A 10.20.30.40 and partner IN A 192.0.2.33 answered an AAAA query for erp with 2001:db8:64::a14:1e28 and for partner with 2001:db8:64::c000:221, matching the arithmetic above. Reading the line: dns64 2001:db8:64::/96 is the prefix the synthesized addresses are built from; clients { any; } says which askers receive synthesized answers (here everyone; in production restrict it to your IPv6-only networks); mapped { any; } says which IPv4 addresses may be turned into IPv6 ones (any); an optional exclude list names IPv6 prefixes whose AAAA records are ignored as if the name had no AAAA, so the server synthesizes from the A record instead (RFC 6147 section 5.1.4). In a test run, a name with ex IN A 192.0.2.50 and ex IN AAAA 2001:db8:99::1 and exclude { 2001:db8:99::/48; } returned the synthesized 2001:db8:64::c000:232 rather than the real AAAA.

Logging

Legacy servers see only the NAT64 gateway's IPv4 pool address, not the client. To attribute a connection you need the gateway's session logs: IPv6 client address, IPv4 source address and port, destination, and timestamps. Many clients share a pool address and ports get reused, so a timestamp-accurate log is required. Size and ship these logs to the SIEM (security information and event management system) and set retention to match your incident-response needs. Server-side access logs will list the pool address, which weakens per-user auditing and per-source rate limiting.

DNSSEC

DNSSEC adds a digital signature to DNS records so a validating resolver can prove an answer was not altered. The signature covers the real A record. DNS64 invents an AAAA record that no zone ever signed, so a client that validates DNSSEC itself sees an unsigned, altered answer and rejects it as bogus (the lookup fails with SERVFAIL). Two flags in the query matter: DO (DNSSEC OK, a flag in the EDNS0 OPT pseudo-record rather than the DNS header) means the client wants the DNSSEC data, and CD (checking disabled, a bit in the DNS header) means the client asks the resolver not to validate because the client will. RFC 6147 says that when both are set the DNS64 must not synthesize at all, so the resolver returns the answer unchanged: the client's AAAA query gets the real (empty) AAAA response and never a synthesized address, and the client would have to perform the synthesis itself. The workable design is a validating DNS64 resolver: it checks the signature on the A record itself, then synthesizes the AAAA, and clients rely on that resolver (a non-validating client sets neither bit and cannot tell). If you need end-host validation, the host must do DNS64 itself, or the names must get real AAAA records.

Hard-coded IPv4 addresses

No DNS lookup means no synthesized address, so these applications fail on an IPv6-only client. Options:

• Fix the application to use a name (the durable fix).
• Run 464XLAT (RFC 6877): a CLAT (customer-side translator) on the client gives the application a local IPv4 address and translates its packets to IPv6 toward the NAT64 gateway, which acts as the PLAT (provider-side translator) and turns them into IPv4. The application believes it is on IPv4.
• Keep those clients dual-stack.
  Protocols that carry addresses in the payload (some FTP and SIP modes) also need an application-layer gateway (ALG, a translator component that rewrites addresses inside the data, not just in the packet header) on the translator.

Trade-offs and pitfalls

• Recommend the network-specific prefix, two gateways, a validating DNS64, and the 464XLAT or dual-stack exceptions for literal-IP apps.
• NAT64 is stateful, so it is a capacity, logging and single-point-of-failure concern, and in the general case flows are initiated only from the IPv6 side (RFC 6146 lets an administrator add static mappings for IPv4-initiated access).
• Inventory literal-IP applications (flow records or a pilot VLAN) before cutting anyone over.

Direct answer

Terminate each ISP (internet service provider) on its own edge router (edge-a and edge-b), run eBGP (external Border Gateway Protocol, the protocol that exchanges routes between different organizations) to both with your own AS (autonomous system: a network under one routing policy, identified by a number) number and your own address block, and run iBGP (the same protocol between your own routers) between the two edge routers. Outbound you choose the exit with LOCAL_PREF (a value only your AS sees, higher wins). Inbound you can only influence what other networks decide: advertise your block to both providers, make ISP-A the attractive path with AS-path prepending (adding extra copies of your AS number to the route's AS path, the list of networks a route has crossed, so it looks longer) toward ISP-B and, where the provider supports it, provider communities. Before trusting it to survive a failure, verify that the two providers really fail independently (physical path, upstream transit, filtering practice), because the BGP configuration cannot create diversity that is not there. The core of the design is three decisions: LOCAL_PREF for outbound, announcements plus prepending for inbound, and checking provider independence; the rest below supports those.

Outbound: you decide

RFC 4271 evaluates a route's degree of preference (LOCAL_PREF) first and only then compares AS_PATH length, so a higher LOCAL_PREF on routes learned from ISP-A makes ISP-A the exit for every destination regardless of path length. Set it on the inbound policy for ISP-A's session and share it over iBGP so both edge routers agree. If ISP-A fails, its routes disappear and ISP-B's lower-preference routes take over without any other change.

What to receive. A default route (a catch-all entry meaning send anything you have no better route for here) plus a few specifics from each provider is cheap and sufficient if you only need failover. Full tables (a route to every network on the internet, about 1.08 million IPv4 entries alone as of October 2026 according to bgp.potaroo.net, plus the IPv6 table) let you choose exits per destination, at the cost of router memory and operational care. Pick default-plus-partial unless you have a performance reason to steer individual destinations. With a default-only design the BGP session can stay up while the provider's upstream is broken (the default route is still announced), so add active probing of an external target through each link and lower the preference or withdraw announcements when the probe fails.

Inbound: you influence, others decide

Illustrative setup: AS 64500 (a number from the 64496 to 64511 documentation range, RFC 5398), the block 203.0.112.0/23 (illustrative), providers AS 64501 (ISP-A, preferred) and AS 64502 (ISP-B).

The /24s are the longest prefixes that will propagate: RFC 7454 notes IPv4 prefixes longer than /24 (and IPv6 longer than /48) are generally neither announced nor accepted, so splitting below /24 does not work. A remote network that hears both announcements compares AS paths. Through ISP-A it sees 64501 64500 (2 hops). Through ISP-B it sees 64502 64500 64500 64500 64500 (5 hops, because 64500 was added 3 extra times). With everything else equal, the shorter path through ISP-A wins. Keep the aggregate announced to both so that if ISP-A fails, traffic for the whole block still arrives through ISP-B.

To spread inbound load instead of preferring one provider, prepend the first /24 toward ISP-B and the second /24 toward ISP-A. Each provider then attracts roughly half of the networks that hear both.

Why prepending is weak. Remote ISPs apply their own LOCAL_PREF, and a route from a customer (a network that pays them) usually beats one from a peer (a network they exchange traffic with for free), so a network that is a direct customer of ISP-B will choose ISP-B whatever your prepending says. If you need more control, ask each provider whether it offers communities (BGP tags, RFC 1997) that lower the preference of your route inside its network or restrict where it is announced. The community meanings are provider-defined, so read the provider's published list, and test each one. MED (multi-exit discriminator) only helps choose between several links to the same provider, so it does not help here.

Do not become a transit network (a network that carries other networks' traffic between providers). Announce only your own prefixes outbound to each provider (filter anything learned from ISP-A before it goes to ISP-B), or a failure elsewhere can send ISP traffic through your edge.

Capacity and failure behavior

If peak traffic is 700 Mbps on two 1 Gbps circuits, one circuit failing leaves the other at 70%, which survives. At a 1.2 Gbps peak the one left would need to carry 120% of its capacity and drop traffic, so size each circuit for the full peak (or have a QoS policy that sheds bulk traffic first). Run BFD (bidirectional forwarding detection, a fast liveness check) on the sessions if the providers support it, because BGP alone detects a silent failure only when the hold timer (the time without a keepalive message after which a peer is declared dead) expires.

What to verify about the providers

1. Physical diversity. Ask each for its fiber route into your building, separate building entrances, different conduit, different carrier hotel. Two providers in one trench is one failure.
2. Upstream diversity. Look at how each provider reaches the rest of the internet (a looking glass, a public web page or server where an operator lets you query its routers, or a public BGP data service, shows the AS paths). If both buy transit from the same upstream, an incident there takes both down.
3. Routing hygiene. Confirm both will accept your prefixes. If your block is provider-assigned (lent to you from ISP-A's address space, not owned by you), ISP-B will not announce it without a letter of authorization (LOA) from ISP-A saying you may use it. Separately, RPKI (resource public key infrastructure) lets the address holder publish signed records saying which AS may announce a prefix. Register a ROA (route origin authorization, an RPKI object stating which AS may originate a prefix) for the /23 with maximum length 24. Under RFC 6811, a /24 covered by a ROA that does not match it, for example a ROA with maximum length 23, is Invalid and a provider that drops RPKI-invalid routes will drop your /24s.
4. Limits and filters. Ask what they accept (prefix lengths, max-prefix per session; RFC 7454 recommends a limit on routes accepted from a peer) and whether they honor TE communities.
5. Capacity and DDoS handling. Whether the upstream capacity is oversubscribed and whether they offer scrubbing.
6. Operations. Maintenance notices, escalation contacts, BFD support, SLA credits.
7. Test it. Shut each session during a maintenance window and measure recovery time for traffic in both directions.

Pitfalls

• Announcing only /24s without the aggregate, then losing both when one provider filters.
• Assuming prepend 3 means inbound traffic will follow. Measure with flow data after the change.
• Both circuits entering through the same wall.
• A default-only design with no probe.