Summary / approach
Systematically verify configuration from L3→L7 and control plane: confirm CIDR/routing, ACLs/security groups, instance OS/net stack, and provider-side issues. Work top-down and collect packet traces.
VPC & AZ fundamentals
- Confirm correct VPC and peering/transit gateway attachments between AZs; ensure peering is active and routes propagated.
- Verify overlapping CIDRs are not causing blackholes.
Route tables
- Check route tables for subnets in AZ-a and AZ-b: explicit routes to destination CIDRs, next-hops (IGW, NAT, VGW, TGW, peering).
- For peering/TGW: ensure no missing/filtered route propagation and no route priority conflicts.
- Validate route table association per subnet (not default).
Subnet CIDRs
- Ensure target private IPs are inside expected subnet CIDR.
- Verify no overlapping or misassigned secondary CIDRs on ENIs.
NACLs
- Inspect stateless NACLs on both source and destination subnets:
- Confirm explicit allow entries for source/dest ports and ephemeral ranges.
- Check rule ordering and explicit denies.
- Validate both inbound and outbound directions.
Security Groups
- Verify SGs on destination instances allow traffic from source CIDR/SG (for AWS, allow SG as source).
- Check ephemeral port ranges for return traffic and stateful behavior.
- Confirm no deny lists in cloud provider WAF/Network Firewall.
Instance kernel / network stack
- On source and destination:
- Check iptables/nftables for DROP rules, conntrack limits, and NAT rules.
- Check sysctl (net.ipv4.ip_forward, rp_filter, net.core.rmem/wmem).
- Inspect MTU mismatches (ping -M do -s) — look for fragmentation/drop across tunnels (VPN/TGW) or jumbo frames on ENIs.
- Check interface statistics (ifconfig/ip -s link) for RX/TX errors, drops.
- Review conntrack table size and TIME_WAIT exhaustion.
Diagnostics / packet capture
- Use tcpdump/pcap on both ends and on transit appliances; capture SYN, SYN-ACK, ICMP unreachable.
- Trace with traceroute/mtr (ICMP/TCP) to determine hop where flows stop.
- Use cloud provider VPC flow logs / NSG flow logs to see accept/reject and 5-tuple records.
Cloud control-plane & provider services
- Check provider status pages for regional/AZ incidents.
- For managed routing (Transit Gateway, VPC peering, Azure vNets): verify attachment state, route propagation, ACLs at the gateway.
- Inspect cloud audit logs for recent config changes (route, ACLs, security group updates, instance replacement).
- If using ENI failover, confirm secondary ENIs’ attachment and MAC/IP assignments.
Operational checks & remediation
- Reproduce with a bastion in same AZ as failing source and same AZ as working source to isolate AZ boundary.
- Temporarily open NACLs/SGs to narrow fault, then reapply least-privilege.
- If MTU implicated, lower MTU on interfaces or enable TCP MSS clamping on gateway.
- Restart networking services, flush iptables/conntrack if safe.
When to escalate
- If flow logs show accepted but packets drop mid-path — escalate to provider with timestamps, flow logs, packet captures, and traceroute output.
