Cloud Networking and VPC Design Questions

Direct answer

Security groups operate at the instance level (attached to the network interface) and are stateful: allow an inbound request and the matching response is automatically permitted back out, with no separate outbound rule needed. Network ACLs (NACLs) operate at the subnet level and are stateless: you must explicitly allow both directions of every conversation, including the ephemeral high-numbered ports used for return traffic, or replies get silently dropped. In a well-designed multi-tier application both layers are typically in play at once, security groups doing the fine-grained, per-tier access control and NACLs providing a coarser, second layer that holds even if a security group is ever misconfigured.

Structured elaboration

Why stateful vs stateless matters in practice, concretely. A client outside the VPC opens a connection to a web server: the request arrives from an ephemeral, randomly assigned high-numbered source port on the client side, destined for port 443 on the server, and the server's reply goes from port 443 back to that same ephemeral port on the client. A security group only needs one inbound rule (allow port 443 from the client's range) because it tracks the connection's state and automatically permits the matching reply out, no matter what port the reply uses. A NACL has no such memory: the inbound rule allowing port 443 says nothing about the outbound reply, so the NACL also needs an explicit outbound rule allowing traffic to the entire ephemeral port range (typically 1024 to 65535) back to the client, or every reply will be silently dropped at the subnet boundary even though the security group and the application are both configured correctly. This ephemeral-port gap is the single most common NACL misconfiguration: everything looks right, connections still fail, and the cause is invisible unless you specifically know to check for the missing return-traffic rule.

When to use each for a multi-tier application. Security groups do essentially all of the meaningful access control day to day: web tier's security group allows inbound 443 from the internet, application tier's security group allows inbound only from the web tier's security group, database tier's security group allows inbound only from the application tier's security group. NACLs are usually left at a permissive default in this pattern, and are reached for deliberately, not by default, when you need a control that survives a security-group mistake: for example, a subnet-wide rule blocking a specific IP range known to be malicious regardless of which instance or security group it targets, or a hard compliance boundary (say, "the database subnet must never accept inbound traffic from outside the VPC's own CIDR (its assigned IP address range, e.g. 10.0.0.0/16), full stop") that you want enforced even if someone someday attaches an overly permissive security group to something in that subnet by mistake.

Worked example

A case where both are genuinely required, and why. A financial services company runs its database tier in a private subnet and, separately from its normal tier-to-tier security groups, has a compliance requirement that the database subnet must categorically reject any traffic from outside the VPC's address range, independent of any application-level security group configuration, because a security group misconfiguration must never be sufficient on its own to expose the database externally. They implement this as a custom NACL on the database subnet with an explicit rule denying inbound traffic from any source outside the VPC's CIDR, positioned before (a lower rule number than) a broader allow rule for the VPC's own range, plus the normal outbound ephemeral-port allow rule for return traffic to the application tier. Independently, the database's security group allows inbound only from the application tier's specific security group on the database port, which is the layer actually doing meaningful per-tier access control day to day. The two layers answer different questions: the security group asks "is this specific source explicitly permitted," while the NACL asks "does this traffic even belong in this subnet at all," and having both means a mistake in one (an overly broad security-group rule accidentally added during a migration, for instance) does not by itself defeat the other.

Trade-offs and pitfalls

The most common mistake is trying to use NACLs for fine-grained, frequently changing access control, such as per-application-team rules: because NACL rules are evaluated in strict numeric order and typically capped at a modest number per direction, this quickly becomes an unmanageable, easy-to-misorder rule list, which is precisely the job security groups are built for instead. The second most common mistake, already covered above, is forgetting the ephemeral-port outbound rule when customizing a NACL, which silently breaks return traffic for entirely correctly configured applications and security groups. Finally, teams sometimes treat NACLs as unnecessary given that security groups exist, missing that the value of a NACL is specifically that it is a separate, subnet-wide layer that does not depend on any individual resource's security-group configuration being correct, which is exactly the property that makes it worth the extra maintenance for a genuinely hard boundary like the database example above.

Direct Answer

Give each workload type its own subnet tier, sized for how it actually consumes IP addresses, since Kubernetes pods and serverless function instances are far hungrier than a fixed virtual machine fleet, route every tier through a per-Availability-Zone NAT gateway rather than one shared gateway, and reach managed AWS services through private endpoints so cross-Availability-Zone traffic reflects a genuine availability need rather than a topology accident.

Subnet Layout by Workload Type

Virtual machines: a straightforward private subnet per Availability Zone (AZ), sized for the fleet's known, roughly fixed instance count.

Kubernetes: with the standard Amazon VPC Container Network Interface (CNI) plugin, pods draw IP addresses from the same VPC CIDR block, so this tier has to be sized for the worst case, node count multiplied by the maximum pods per node, which is usually far larger than the virtual machine tier and the single most common VPC sizing mistake with a managed Kubernetes service: a subnet that looked generous for virtual machines runs out of addresses quickly once real pod density hits it. Give Kubernetes its own dedicated, deliberately larger block, not a slice sized like the virtual machine tier.

Serverless functions: a function attached to a VPC does not get its own elastic network interface per invocation. AWS Lambda uses a shared Hyperplane elastic network interface model: Lambda creates one Hyperplane ENI per distinct combination of subnets and security groups a function uses, and every function that shares that same subnet-and-security-group combination reuses the same ENI rather than each getting its own. Each Hyperplane ENI supports up to 65,000 concurrent connections; only once that ceiling is crossed does Lambda provision an additional ENI for that combination, scaling automatically with traffic and concurrency. This still consumes IP addresses from the tier's subnet, just at a far lower rate than a one-ENI-per-execution model: a deployment with many distinct subnet-and-security-group combinations (for example, one security group per function for fine-grained isolation) or with enough total concurrency to repeatedly cross the per-ENI connection ceiling can still exhaust a subnet sized only for the virtual machine tier's small, fixed instance count. Size this tier around the number of distinct subnet-and-security-group combinations you expect and their aggregate connection volume, not the raw invocation count; undersizing it now surfaces as Lambda failing to allocate a new Hyperplane ENI, an opaque IP-exhaustion error in the subnet rather than an intuitive capacity message.

NAT and Outbound Egress

Deploy one NAT gateway per Availability Zone, with every tier's subnets in that zone routing to their own zone's NAT gateway, so no workload's egress path requires crossing zones just to reach the internet. This is a cost fix (cross-zone NAT traffic is billed as inter-AZ data transfer on top of NAT processing charges) and a reliability fix, since one zone's NAT gateway failing no longer takes down every zone's egress.

Private Endpoints for Managed Services

Gateway VPC endpoints for object storage and managed database services, and Interface VPC endpoints (AWS PrivateLink) for other managed services, a container registry for image pulls, a secrets manager, log ingestion, and for the serverless platform's own control-plane calls, let all three workload types reach managed services over the AWS backbone instead of through NAT or the public internet. This both avoids unnecessary cross-zone or NAT traffic and removes public exposure for that traffic, which is exactly what the question asks each workload type to get without the two costs it names.

Topology-Aware Routing for Kubernetes

Kubernetes' built-in topology-aware routing hints prefer sending a service's traffic to an endpoint in the same zone as the caller when a healthy same-zone endpoint exists, reducing cross-zone pod-to-pod traffic for calls that stay inside the cluster. This complements the per-zone NAT fix, which only addresses traffic leaving the VPC: one fix covers intra-cluster east-west traffic, the other covers traffic leaving to the internet or to AWS services, and a complete design needs both.

This answer deliberately does not cover service discovery mechanics, how a service actually resolves another service's address, since that is a distinct competency from subnet and routing design and belongs to a different discussion; the focus here is IP addressing, routing, and endpoint placement.

Avoiding Public Exposure

Virtual machine and Kubernetes worker node subnets should have no route to an internet gateway at all, private subnets only, with egress solely through NAT or private endpoints. Any genuinely public-facing tier, an internet-facing load balancer, lives in dedicated public subnets, never mixed into the same subnet as the workloads themselves, so a routing mistake cannot accidentally expose a pod or a function's network interface directly to the internet.

Worked Example

CIDR math for a VPC sized to fit all three tiers without collision, verified with Python's ipaddress module rather than asserted:

Running these twelve subnets through ipaddress confirms every one is a valid subnet of the /16, confirms none of the twelve overlap with any other, and confirms the sizing logic: at 4,091 usable addresses and a common default of 110 maximum pods per node, the Kubernetes tier's per-zone subnet supports

roughly 37 nodes per zone before running out of pod addresses, which is the number to check against the actual planned node count for that zone rather than assuming the block is "big enough." The exact maximum-pods-per-node value depends on instance type and CNI configuration, so this sizing logic, not the specific 37, is what should be recomputed for a real cluster.

Trade-offs and Pitfalls

Sizing the Kubernetes subnet the same as the virtual machine tier "because it's just another compute tier" is the single most common mistake here; pod density means it needs to be substantially larger from day one, since resizing a live VPC's CIDR allocation later is disruptive.

A NAT gateway per zone raises the fixed hourly cost versus one shared gateway; for a very small deployment this may not be worth it, but for anything running production Kubernetes or serverless traffic at real volume, the cost-and-reliability case for per-zone NAT usually wins.

Topology-aware routing only helps traffic that stays inside the cluster; it does nothing for the NAT and endpoint design that governs traffic leaving the VPC, so treat the two as complementary fixes, not substitutes for each other.

Direct answer

A route table is an ordered set of rules, one per subnet association, that tells the network where to send traffic based on its destination IP address. Every route table gets one route it cannot delete, the "local" route for the VPC's own address range, which is why resources in the same VPC can always reach each other by default regardless of what else is configured. When more than one route could match a given destination, the network uses longest-prefix match: the most specific (largest prefix length) route wins, not the first one listed or the most recently added one.

Structured elaboration

Association. Each subnet is associated with exactly one route table (a route table can serve multiple subnets, but not the reverse). If a subnet has no explicit association, it uses the VPC's main route table by default. This is a common source of surprise: a newly created subnet silently inherits whatever the main route table says, which may not be what you intended for it.

The local route. The moment you create a VPC with, say, a 10.0.0.0/16 CIDR (Classless Inter-Domain Routing, the notation for writing an IP address range and its size) block, every route table in that VPC gets an implicit 10.0.0.0/16 -> local route that cannot be edited or removed. This is what makes intra-VPC communication work without any explicit configuration and is also why you cannot use a VPC's own address range for something else internally without addressing conflicts.

Longest-prefix match (route precedence). When a packet's destination matches more than one route in the table, the router picks the route with the longest (most specific) prefix, independent of the order the routes were created in. For example, if a table has both 0.0.0.0/0 -> internet gateway (catch-all, prefix length 0) and 10.0.0.0/16 -> local (prefix length 16), traffic to an address inside 10.0.0.0/16 matches the more specific /16 route even though the /0 route also technically matches everything. This is also how you can carve exceptions into a broad NAT-bound default route: add a more specific route for one destination CIDR that should instead go over a VPN or peering connection, and it wins over the general 0.0.0.0/0 route without touching that catch-all rule.

Route propagation from virtual gateways (BGP). When a Virtual Private Gateway or Transit Gateway is attached to a VPC and used with a Border Gateway Protocol (BGP) session over a VPN or a dedicated circuit, the on-premises router advertises its reachable networks over that BGP session, and if you enable route propagation on the route table, those advertised networks are added to the route table automatically instead of you hand-entering static routes. This matters operationally: if the on-premises network adds a new subnet, propagation means your VPC route table updates itself the next time BGP re-converges, with no manual change needed on the cloud side.

Overlapping or conflicting routes from multiple external networks. If you connect to two external networks (say, two data centers, or a data center and a partner network over separate VPN connections) and their advertised CIDR blocks overlap, or if a statically entered route and a BGP-propagated route both cover the same destination, you have to resolve the ambiguity explicitly, because longest-prefix match only helps when the prefixes actually differ in specificity:

• If one advertised range is a strict subset of the other, longest-prefix match naturally sends traffic for the subset to whichever path advertised the more specific range, so no conflict actually exists even though it looks like one on paper.
• If the two ranges genuinely overlap at the same specificity (both advertise the same /24, for instance), most cloud route tables will simply reject adding the second, identical-specificity route, or, on platforms that support route priority, you set an explicit static route with a fixed destination as the tie-breaker. The durable fix, though, is address planning: two networks that both intend to be reachable from the same VPC should never have been assigned overlapping CIDR blocks in the first place, since even where the platform lets you work around it, only one of the two networks is actually reachable at overlapping addresses at any moment.
• For BGP-specific tie-breaking (both paths are valid but you want a preference, not an outright conflict), use attributes like AS-path prepending (making a path look longer and therefore less preferred) or a local-preference / route-priority setting supported by the platform, rather than fighting it with static routes that will silently go stale.

Worked example

A VPC with CIDR 10.0.0.0/16, an Internet Gateway (IGW), and a Virtual Private Gateway (VGW) with route propagation enabled to an on-premises network at 192.168.0.0/16:

Here, the manually entered /24 route deliberately overrides part of the broader propagated /16 route for one subnet, a legitimate and common pattern for routing one sensitive on-premises subnet over a dedicated path while everything else in that address range uses the general one.

Internet Gateway vs Virtual Private Gateway, since they are easy to conflate: an Internet Gateway is a horizontally scaled, highly available AWS-managed component that gives a VPC a path to and from the public internet; it has no concept of BGP or on-premises networks. A Virtual Private Gateway is the VPN/Direct Connect-side endpoint on the AWS side of a hybrid connection; it terminates VPN tunnels or a Direct Connect (AWS's dedicated, private network circuit, as opposed to a VPN's tunnel over the public internet) virtual interface and is what participates in BGP route propagation with an on-premises router. They serve two entirely different paths (public internet vs private hybrid connectivity) and a VPC commonly has both at once, each referenced by different routes in the same route table.

Trade-offs and pitfalls

The most common mistake is assuming route order in a table or console listing matters, then getting confused when a "later" broad route does not override an earlier specific one; only prefix length decides precedence, never entry order. A second pitfall is enabling route propagation without auditing what gets propagated: an on-premises network that advertises a default route (0.0.0.0/0) over BGP will silently start competing with your Internet Gateway's own default route for precedence (resolved by longest-prefix match rules that still apply, but the practical effect, if the propagated route wins for some traffic class, can be egress traffic unexpectedly hairpinning through on-premises). Finally, when connecting to multiple external networks, resolve address-space overlap at design time through CIDR planning, not at incident time through static-route workarounds; a static override fixes today's symptom and quietly becomes tomorrow's undocumented special case.

Direct Answer

Keep a continuous, cheap sampled baseline running at all times using flow sampling and Berkeley Packet Filter (BPF)-based filters, so there is always some signal, then automatically trigger a short, full-fidelity Traffic Mirroring burst to an ephemeral capture target only when an anomaly detector fires, so full packet detail is available exactly when it is needed without the storage and processing cost of capturing everything all the time.

The Always-On Baseline: Sampling

Continuous, low-overhead flow sampling, VPC Flow Logs at minimum, or sampled flow export from an appliance or mesh that supports it, runs at all times. This is what the anomaly detector actually watches: a cheap, always-available signal for "something changed" without the cost of full packet capture running around the clock.

Narrowing What Gets Captured: BPF-Based Filtering

Whenever a capture is running, the continuous baseline or a triggered burst, a BPF filter, the same filter language standard packet-capture tools use and expressible as a Traffic Mirroring filter, narrows capture to exactly the flows implicated by the anomaly: a specific five-tuple (the combination of source IP, destination IP, source port, destination port, and protocol that identifies one connection), a specific port range, a specific instance's network interface, rather than the whole host's traffic. This keeps both the processing cost of capture and the resulting file volume proportional to the actual problem.

The Triggered Mechanism: Traffic Mirroring

On anomaly detection, programmatically create a Traffic Mirroring session, source the implicated elastic network interface, filter the narrowed rule from the previous step, target an ephemeral capture instance, via the API, rather than keeping a mirror session running permanently. This reuses the same primitive used for manual debugging, just automated and scoped to a detected event instead of a human-initiated one.

Ephemeral Capture Targets

The capture instance, or fleet, launches on demand from a pre-baked image specifically for the capture window and terminates afterward, rather than running a capture fleet continuously for an event that might happen rarely. This is the core cost lever that makes "high-throughput, short window" capture affordable, since the capture instance is paid for only during the anomaly, not continuously.

Automation to Trigger Captures on Detected Anomalies

An anomaly detector, an alarm on a flow-log-derived metric, or a custom detector watching for a retransmit-rate or error-rate spike, invokes an orchestrated workflow (useful here specifically because the process has several sequential, stateful steps rather than a single action) that: launches the ephemeral capture target; creates the Traffic Mirroring session pointed at it; waits a bounded window, long enough to catch a few cycles of the anomaly, short enough to bound cost, for example five to fifteen minutes; deletes the mirror session; triggers upload of the resulting capture files to object storage; and terminates the ephemeral target.

Securing Captured Data

Packet captures can contain unencrypted payloads or session tokens, so treat the destination storage as sensitive by default: default server-side encryption with a dedicated key rather than a shared default key, so access can be scoped and audited independently; a bucket policy that denies access outside a specific incident-response role; and object versioning or a write-once policy if a capture might become evidence for a security investigation that must not be alterable.

Automating the Lifecycle: Start, Stop, Store, Purge

The same orchestrated workflow owns start and stop. Storage goes to the encrypted, access-restricted bucket described above. Purge is the step most manual pipelines forget: a lifecycle rule that expires, or transitions to cold storage and then expires, raw capture files after a short, deliberately chosen retention window, long enough to complete the investigation, short enough to bound both cost and the exposure of holding sensitive captured payloads, kept separate from any sanitized findings an analyst chooses to retain longer.

Worked Example

flowchart LR
  A[VPC Flow Logs baseline] --> B[Anomaly alarm]
  B -->|fires| C[Orchestrated workflow]
  C --> D[Launch ephemeral capture instance]
  C --> E[Create Traffic Mirroring session: source=order-service ENI, filter=narrowed BPF rule]
  D --> F[Bounded capture window, e.g. 10 min]
  E --> F
  F --> G[Delete mirror session]
  F --> H[Upload capture to encrypted storage]
  H --> I[Terminate ephemeral capture instance]
  H --> J[Lifecycle rule purges raw capture after retention window]

A retransmit-rate anomaly on an order-service instance's network interface fires the alarm. The workflow launches a small ephemeral capture instance and a Traffic Mirroring session scoped only to that one interface, filtered to its TCP traffic rather than the whole subnet, captures for a bounded ten-minute window (long enough to catch several retransmit cycles without an open-ended cost commitment), tears down the mirror session and the instance, and lands the encrypted capture in storage, where a lifecycle rule purges it automatically after, say, fourteen days unless an analyst has already pulled it into a longer-lived investigation record.

Trade-offs and Pitfalls

Setting the anomaly threshold too sensitive triggers frequent, needless capture cycles, reintroducing exactly the cost problem this design exists to avoid; tune the detector against the flow-log baseline's normal variance first, do not guess a threshold.

Forgetting the purge step is the most common gap: a pipeline that automates start and stop but leaves storage manual tends to accumulate sensitive captures indefinitely, the opposite of the short-window, minimal-footprint goal.

An ephemeral, on-demand capture target adds a launch delay between anomaly detection and capture start, so the very first moment of a fast, transient anomaly might be missed; the always-on flow-sampling baseline is what covers that gap, since it does not depend on any launch.

A BPF or mirroring filter that is too broad reintroduces both cost and sensitive-data exposure; too narrow and it can filter out the very packets that explain the anomaly. Scope it from the same signal that triggered the detector, do not guess it independently.

Direct Answer

Path MTU Discovery (PMTUD) relies on a router along the path sending back an ICMP (Internet Control Message Protocol, used for network diagnostic and error messages rather than application data) "fragmentation needed" message when a packet marked "don't fragment" is too big for the next link's maximum transmission unit (MTU). IPsec's own encryption overhead shrinks the usable MTU inside a VPN tunnel, and if any device in the path blocks that ICMP message, the sender never learns to shrink its packets: the connection black-holes rather than simply running slower, which is what makes this failure mode so confusing to diagnose.

The Mechanics

A sender's TCP stack picks a segment size assuming an end-to-end MTU, typically 1500 bytes on Ethernet. When a packet marked with the Don't Fragment (DF) flag hits a link whose MTU is smaller, a compliant router must drop it and reply with an ICMP Destination Unreachable message, type 3 code 4 specifically, "fragmentation needed", telling the sender the smaller MTU to use. The sender's stack then shrinks future segments accordingly. That is PMTUD working as designed.

Where IPsec Breaks the Assumption

An IPsec tunnel adds encapsulation overhead, the Encapsulating Security Payload (ESP) header and trailer, plus NAT-Traversal encapsulation if that is in use, so the usable MTU inside the tunnel is smaller than the outer link's 1500 bytes, by an amount that depends on the specific encryption and hashing algorithm. AWS Site-to-Site VPN publishes the achievable tunnel MTU and TCP maximum segment size (MSS) for common algorithm combinations:

The exact byte values shift if AWS changes supported ciphers, so the mechanism, encryption overhead shrinks the usable MTU by an algorithm-dependent amount, is the durable knowledge here; the table is current as of this writing and worth re-checking against AWS's own documentation before hardcoding a value.

Effects of the DF Bit and ICMP Blocking

If the encapsulated, now-larger packet exceeds the tunnel's real MTU and still carries the DF flag, a standards-compliant router must drop it and send the ICMP fragmentation-needed message rather than silently fragmenting it. Many corporate firewalls, some low-end branch routers, and overly strict security rules block all ICMP as a blanket security measure. When that message never reaches the sender, the sender keeps emitting oversized DF packets that keep getting silently dropped: this is PMTUD black-holing. It presents as connections that hang specifically for larger payloads, a big file transfer or a TLS handshake carrying a large certificate chain, while a basic ping, a small ICMP echo packet, succeeds, which is the classic tell that misleads people into ruling out MTU as the cause.

Fragmentation Behavior

AWS's recommended handling is to fragment the original IP packet before encryption, rather than fragmenting the already-encrypted ESP packet afterward, because Site-to-Site VPN reassembles fragments on the AWS side before forwarding, and pre-encryption fragmentation avoids the extra overhead of encrypting each fragment separately.

Mitigation Strategies

Set the customer gateway device's MTU and MSS to the values in the table above for the actual algorithm in use, rather than assuming a generic number and hoping it works. TCP MSS clamping, rewriting the MSS option in the TCP handshake to the safe value for that algorithm, is the most robust fix for TCP traffic because it fixes the segment size at connection setup instead of depending on an ICMP message ever arriving, which sidesteps the ICMP-blocking failure mode entirely for TCP. It does nothing for UDP-based protocols, which still depend on real PMTUD or application-level handling.

For Direct Connect, the same class of problem can appear if intermediate customer-side network equipment, or a nested IPsec tunnel added for encryption, reduces the effective MTU below what the application assumes. Direct Connect supports jumbo frames (9001 bytes) on some virtual interface types, so moving from a 1500-byte assumption to a 9001-capable path can surface the mirror-image bug if only one end is configured for jumbo frames.

Explicitly allow ICMP type 3 code 4 through every security group, network ACL, and any customer-side firewall in the path, rather than blanket-denying ICMP.

Monitoring Indicators of PMTUD Failure

TCP retransmission timeouts and stalls correlated specifically with larger payload sizes or TLS handshakes rather than all traffic; packet captures showing a DF-flagged packet followed by no response and a retransmit rather than an ICMP reply; interface counters showing ICMP messages sent on one end but not received on the other; and, pragmatically, a support pattern where small requests succeed and large ones hang.

Worked Example

An on-premises server, MTU 1500, DF set by default on most modern TCP stacks, sends a TLS ClientHello with a large certificate chain over a Site-to-Site VPN using AES-GCM-16 with NAT-Traversal enabled, MTU 1438 and MSS 1398 per the table above. If the customer gateway's own MSS clamp is still set to a pre-VPN default of 1460, built for a plain 1500-byte path with no IPsec overhead subtracted, the resulting encrypted packet exceeds 1438 bytes, carries DF, and a security appliance somewhere in the path blocks the return ICMP message. The handshake stalls specifically on that oversized packet, while a 64-byte ping succeeds, matching the diagnostic pattern described above. The fix is not "the VPN is broken"; it is reconfiguring the customer gateway's MSS clamp from 1460 down to 1398, the value the table specifies for this exact algorithm and NAT-Traversal combination, so the sender never emits an oversized segment in the first place.

Trade-offs and Pitfalls

Blanket-blocking ICMP as a security hardening step is a common but actively harmful default; the correct fix is allowing the specific fragmentation-needed message, not all ICMP.

Copying an MTU or MSS value from a different tunnel's configuration is a recurring mistake; the byte budget is specific to the exact encryption, hashing, and NAT-Traversal combination in use, and changing the cipher suite later without revisiting MSS silently reintroduces the bug.

MSS clamping only helps TCP. A workload relying on UDP-based protocols needs real end-to-end PMTUD, meaning the ICMP message must actually get through, or application-level fragmentation handling instead.

Jumbo frames over Direct Connect are a separate lever from IPsec MSS tuning, not a substitute for it; confirm which virtual interface type and provider actually support 9001-byte frames before assuming a Direct Connect path benefits.