Cloud Networking and VPC Design Questions

Direct answer

Security groups operate at the instance level (attached to the network interface) and are stateful: allow an inbound request and the matching response is automatically permitted back out, with no separate outbound rule needed. Network ACLs (NACLs) operate at the subnet level and are stateless: you must explicitly allow both directions of every conversation, including the ephemeral high-numbered ports used for return traffic, or replies get silently dropped. In a well-designed multi-tier application both layers are typically in play at once, security groups doing the fine-grained, per-tier access control and NACLs providing a coarser, second layer that holds even if a security group is ever misconfigured.

Structured elaboration

Why stateful vs stateless matters in practice, concretely. A client outside the VPC opens a connection to a web server: the request arrives from an ephemeral, randomly assigned high-numbered source port on the client side, destined for port 443 on the server, and the server's reply goes from port 443 back to that same ephemeral port on the client. A security group only needs one inbound rule (allow port 443 from the client's range) because it tracks the connection's state and automatically permits the matching reply out, no matter what port the reply uses. A NACL has no such memory: the inbound rule allowing port 443 says nothing about the outbound reply, so the NACL also needs an explicit outbound rule allowing traffic to the entire ephemeral port range (typically 1024 to 65535) back to the client, or every reply will be silently dropped at the subnet boundary even though the security group and the application are both configured correctly. This ephemeral-port gap is the single most common NACL misconfiguration: everything looks right, connections still fail, and the cause is invisible unless you specifically know to check for the missing return-traffic rule.

When to use each for a multi-tier application. Security groups do essentially all of the meaningful access control day to day: web tier's security group allows inbound 443 from the internet, application tier's security group allows inbound only from the web tier's security group, database tier's security group allows inbound only from the application tier's security group. NACLs are usually left at a permissive default in this pattern, and are reached for deliberately, not by default, when you need a control that survives a security-group mistake: for example, a subnet-wide rule blocking a specific IP range known to be malicious regardless of which instance or security group it targets, or a hard compliance boundary (say, "the database subnet must never accept inbound traffic from outside the VPC's own CIDR (its assigned IP address range, e.g. 10.0.0.0/16), full stop") that you want enforced even if someone someday attaches an overly permissive security group to something in that subnet by mistake.

Worked example

A case where both are genuinely required, and why. A financial services company runs its database tier in a private subnet and, separately from its normal tier-to-tier security groups, has a compliance requirement that the database subnet must categorically reject any traffic from outside the VPC's address range, independent of any application-level security group configuration, because a security group misconfiguration must never be sufficient on its own to expose the database externally. They implement this as a custom NACL on the database subnet with an explicit rule denying inbound traffic from any source outside the VPC's CIDR, positioned before (a lower rule number than) a broader allow rule for the VPC's own range, plus the normal outbound ephemeral-port allow rule for return traffic to the application tier. Independently, the database's security group allows inbound only from the application tier's specific security group on the database port, which is the layer actually doing meaningful per-tier access control day to day. The two layers answer different questions: the security group asks "is this specific source explicitly permitted," while the NACL asks "does this traffic even belong in this subnet at all," and having both means a mistake in one (an overly broad security-group rule accidentally added during a migration, for instance) does not by itself defeat the other.

Trade-offs and pitfalls

The most common mistake is trying to use NACLs for fine-grained, frequently changing access control, such as per-application-team rules: because NACL rules are evaluated in strict numeric order and typically capped at a modest number per direction, this quickly becomes an unmanageable, easy-to-misorder rule list, which is precisely the job security groups are built for instead. The second most common mistake, already covered above, is forgetting the ephemeral-port outbound rule when customizing a NACL, which silently breaks return traffic for entirely correctly configured applications and security groups. Finally, teams sometimes treat NACLs as unnecessary given that security groups exist, missing that the value of a NACL is specifically that it is a separate, subnet-wide layer that does not depend on any individual resource's security-group configuration being correct, which is exactly the property that makes it worth the extra maintenance for a genuinely hard boundary like the database example above.

Direct answer

The public/private subnet split for a multi-tier app puts only the resources that must accept unsolicited inbound internet traffic (the web tier) in a public subnet, and everything else (application, database) in private subnets that are unreachable from the internet by construction. Security is improved because the blast radius (how far an attacker who breaches one tier can then reach) of an internet-facing compromise stops at the first tier: even with a perfect security-group rule set, only the route table decides whether an internet path exists at all, and the private tiers simply have none. Manageability improves because each tier gets its own routing and firewall boundary, so a change to one tier's exposure cannot silently affect another.

Structured elaboration

Per-tier breakdown:

Route tables, concretely. Each subnet associates with exactly one route table (though one route table can serve several subnets). The public subnet's table has a route for the VPC's own CIDR (added automatically, cannot be deleted) plus 0.0.0.0/0 -> igw-xxxx. Each private subnet's table has the local VPC route plus 0.0.0.0/0 -> nat-xxxx if it needs outbound internet, pointed at the NAT gateway that lives in a public subnet (a NAT gateway needs a public subnet and its own Elastic IP to reach the internet on behalf of everyone routing through it).

Security group vs NACL in this pattern, briefly (the full comparison is its own topic, but the shape that matters here): security groups are stateful and attached to the instance or its network interface, so you write one inbound rule and return traffic is automatically allowed back out. NACLs are stateless and attached to the subnet, evaluated before the security group, so you must explicitly allow both the inbound request and the ephemeral-port outbound return traffic, or connections will hang. Most teams run security groups as the primary control and leave NACLs at their permissive default unless they have a specific reason (blocking a known-bad IP range at the subnet boundary, or enforcing a hard compliance boundary that must hold even if every security group is misconfigured).

Worked example

A two-Availability-Zone (AZ) checklist for a 3-tier app, since anything less than two AZs is not actually highly available:

1. One public subnet per AZ (e.g. 10.0.0.0/24 in AZ-a, 10.0.1.0/24 in AZ-b) holding the load balancer and, if used, a bastion or jump host.
2. One private application subnet per AZ (10.0.10.0/24, 10.0.11.0/24).
3. One private database subnet per AZ (10.0.20.0/24, 10.0.21.0/24), often placed in a dedicated "DB subnet group" so the managed database service only ever launches replicas into subnets you intended for it.
4. One NAT gateway per AZ (not one shared NAT gateway for the whole VPC): if the only NAT gateway lives in AZ-a and AZ-a fails, AZ-b's private resources lose outbound internet access even though their own AZ is healthy. Each AZ's private route tables point only at their own AZ's NAT gateway.
5. Security groups chained tier to tier as in the table above: load-balancer security group accepts the internet, web/app security group accepts only the load-balancer's security group, database security group accepts only the app security group.

Secure administrator access, without a public jump box. A classic weak point in this pattern is putting a bastion host in the public subnet with SSH open to a corporate IP range, which is one more standing internet-facing attack surface to patch and monitor. The current recommended pattern (AWS Systems Manager Session Manager, or Azure Bastion / GCP Identity-Aware Proxy on the other clouds) removes that: the administrator authenticates through the cloud provider's identity and access management (IAM), and the agent already running on the instance opens an outbound-only connection to the Session Manager service, so no inbound port needs to be open on the instance at all, not even from a "trusted" IP range. That also gets you centralized, immutable session logging for free, which a bare SSH bastion does not.

Trade-offs and pitfalls

The single most common mistake in this pattern is a shared, single-AZ NAT gateway "to save cost," which quietly turns a multi-AZ design into one with a single point of failure for all outbound internet access; it also concentrates all cross-AZ private-subnet egress traffic onto one gateway and one AZ's data-transfer path. A second pitfall is treating "private subnet" as sufficient security on its own and skipping the security-group chaining: a private subnet stops internet ingress, but without tier-to-tier security group rules, anything else already inside the VPC (a compromised instance in the same VPC, or an overly broad peering connection) could still reach the database directly. Finally, teams sometimes forget that NACLs are stateless when they add a "temporary" outbound-only NACL rule and wonder why responses never arrive: without the matching ephemeral-port inbound rule, the return leg of the connection is dropped at the subnet boundary even though the security group would have allowed it.

Direct answer

For connecting an on-premises data center to the cloud, VPN over the internet is the cheapest and fastest to stand up but inherits the public internet's variable latency and has no cloud-provider service-level agreement (SLA) for the path outside the cloud; a dedicated private link (AWS Direct Connect or Azure ExpressRoute) gives consistent, high-bandwidth, low-latency connectivity backed by an SLA but costs more and takes longer to provision; third-party SD-WAN (software-defined wide area network) sits above either underlying transport and adds centralized policy, application-aware routing, and often automatic failover across multiple links, at the cost of another vendor and management layer. The right choice depends on what the workload actually needs: sporadic, low-volume, latency-tolerant traffic fits VPN; sustained, latency-sensitive, or compliance-bound traffic justifies a dedicated connection; and a multi-site organization managing many locations' connectivity centrally often layers SD-WAN over one or both.

Structured elaboration

Worked example

Migration framing. A retailer doing a lift-and-shift of its on-premises data center to the cloud initially connects over a Site-to-Site VPN to start moving non-critical workloads immediately, since the VPN can be live the same day and the migration timeline cannot wait weeks for a physical cross-connect. In parallel, they order a Direct Connect circuit at a colocation facility, which takes on the order of weeks to provision. Once Direct Connect is live, they configure Border Gateway Protocol (BGP) so the dedicated circuit is preferred for all cloud-bound traffic (achieved by advertising a shorter AS-path or a better local-preference value on the Direct Connect side) with the VPN kept active as an automatic failover path, giving the migration both a fast start and, once ready, a durable dual-path design rather than a one-time VPN that later has to be torn out and replaced. For cross-region resilience during the migration window, they also plan bandwidth headroom on both paths so a regional failover does not saturate whichever link absorbs the shifted traffic.

Cost-driven recommendation for a scheduled bulk-transfer workload. A different team runs a nightly batch job transferring a large, predictable volume of data (say, tens of terabytes) from an on-premises system to cloud storage during a fixed overnight window. Because the volume is large and the timing is predictable and recurring (not sporadic), the data-transfer economics favor Direct Connect as the primary path, sized to comfortably clear the nightly volume inside the batch window, with an IPsec VPN kept purely as a failover for the rare case the dedicated circuit is down during that window. Sizing the VPN failover path only needs to cover the batch's minimum acceptable throughput, not match Direct Connect's full capacity, since it exists to avoid a missed run, not to be the steady-state path.

Reliability comparison, concretely. An internet-path VPN carries no AWS or Azure commitment for the portion of the trip between the customer's router and the cloud provider's edge; if a transit provider somewhere on the public internet degrades, there is no SLA to invoke against the cloud provider for that. A Direct Connect connection, deployed as multiple connections terminating on separate devices across more than one physical location (the resiliency model AWS recommends for production), carries an SLA and a defined mean-time-to-repair (MTTR) commitment for the dedicated portion of the path; a single, non-redundant Direct Connect connection, by contrast, carries no SLA at all, which is a common and costly misunderstanding.

Trade-offs and pitfalls

The most common mistake is ordering a single Direct Connect connection and treating it as production-ready simply because it is "dedicated": without a second connection on a separate device, ideally at a second location, there is no SLA and no protection against a single device or facility failure, which defeats much of the point of choosing Direct Connect over VPN in the first place. A second pitfall is under-provisioning the VPN failover path's bandwidth on the theory that it will "rarely" be used, then discovering during an actual Direct Connect outage that the VPN cannot carry production traffic at an acceptable level either. Finally, teams evaluating SD-WAN sometimes expect it to fix underlying transport problems it cannot fix. SD-WAN can steer traffic intelligently across the links you give it and add centralized policy, but it does not manufacture bandwidth or eliminate the latency characteristics of a genuinely poor underlying link.

Direct answer

A NAT (Network Address Translation) Gateway is a fully managed, highly available service that lets resources in a private subnet reach the internet without being reachable from it; a NAT instance is a regular virtual machine running NAT software that you provision, patch, and scale yourself. For the overwhelming majority of workloads, NAT Gateway is the right default because it removes an entire category of operational work and a single point of failure for a small, predictable price. NAT instances remain the right call only in a narrow set of cases where you need capabilities a managed NAT Gateway does not expose, most commonly acting as a bastion or as a traffic-inspection point at the same time as doing NAT.

Structured elaboration

Why NAT Gateway is the default. It removes patching and capacity planning from your team's workload entirely, and its availability model (redundant within the AZ, no single instance to fail) is materially better than a single EC2 instance doing the same job, for a cost that is predictable and, at real production traffic volumes, usually smaller than the engineering time spent babysitting a NAT instance.

When a NAT instance is the right call. NAT Gateway is purpose-built to do exactly one thing (translate addresses) and nothing else: you cannot install a package on it, run a proxy alongside it, or use it as a jump host. A NAT instance is a regular virtual machine, so it can also run a forward proxy with custom filtering rules, host an intrusion-detection agent that inspects outbound traffic, or double as a bastion in a tightly cost-constrained environment where a second dedicated instance is not justified. The other legitimate case is throwaway or development environments where the cost difference (a small, possibly spot-priced instance vs. the NAT Gateway's hourly charge) actually matters and the availability guarantees do not.

Worked example

A realistic scenario for recommending a NAT instance: a small managed-services provider runs a low-traffic staging environment for each of dozens of clients, each in its own VPC (Virtual Private Cloud), torn down and rebuilt weekly. Traffic per environment rarely exceeds a few megabits, uptime requirements are minimal since it is not production, and the team wants a lightweight egress-filtering proxy running on the same box to log which external domains staging environments call out to, for a security review, without standing up a separate proxy fleet. Here, a small burstable instance type running as a NAT instance with a simple forward-proxy process alongside it is a reasonable, deliberate choice: the traffic volume makes the per-gigabyte savings meaningful at that scale multiplied across dozens of environments, the availability trade-off is acceptable for non-production, and the combined NAT-plus-proxy role is exactly the kind of thing a managed NAT Gateway cannot do. The same team's production VPCs, serving live customer traffic, still use NAT Gateway, because none of those justifications (cost sensitivity at low volume, needing a general-purpose instance) apply there.

Trade-offs and pitfalls

The most common mistake is choosing a NAT instance purely to save money on a production workload without pricing in the operational cost: patching it, monitoring its health, building failover logic, and right-sizing it as traffic grows are all now your team's job, and a single AZ's NAT instance going down (or getting saturated) takes down that AZ's outbound internet access with no automatic recovery. A second pitfall, when a NAT instance is chosen, is forgetting to disable the instance's source/destination address check, without which it cannot forward traffic on behalf of other instances at all. Finally, teams sometimes assume NAT Gateway's bandwidth is a hard ceiling that will bottleneck them; in practice it scales automatically to a high throughput ceiling, so bandwidth is rarely the actual reason to reach for a NAT instance, capability (running other software on the same box) almost always is.

Direct answer

A route table is an ordered set of rules, one per subnet association, that tells the network where to send traffic based on its destination IP address. Every route table gets one route it cannot delete, the "local" route for the VPC's own address range, which is why resources in the same VPC can always reach each other by default regardless of what else is configured. When more than one route could match a given destination, the network uses longest-prefix match: the most specific (largest prefix length) route wins, not the first one listed or the most recently added one.

Structured elaboration

Association. Each subnet is associated with exactly one route table (a route table can serve multiple subnets, but not the reverse). If a subnet has no explicit association, it uses the VPC's main route table by default. This is a common source of surprise: a newly created subnet silently inherits whatever the main route table says, which may not be what you intended for it.

The local route. The moment you create a VPC with, say, a 10.0.0.0/16 CIDR (Classless Inter-Domain Routing, the notation for writing an IP address range and its size) block, every route table in that VPC gets an implicit 10.0.0.0/16 -> local route that cannot be edited or removed. This is what makes intra-VPC communication work without any explicit configuration and is also why you cannot use a VPC's own address range for something else internally without addressing conflicts.

Longest-prefix match (route precedence). When a packet's destination matches more than one route in the table, the router picks the route with the longest (most specific) prefix, independent of the order the routes were created in. For example, if a table has both 0.0.0.0/0 -> internet gateway (catch-all, prefix length 0) and 10.0.0.0/16 -> local (prefix length 16), traffic to an address inside 10.0.0.0/16 matches the more specific /16 route even though the /0 route also technically matches everything. This is also how you can carve exceptions into a broad NAT-bound default route: add a more specific route for one destination CIDR that should instead go over a VPN or peering connection, and it wins over the general 0.0.0.0/0 route without touching that catch-all rule.

Route propagation from virtual gateways (BGP). When a Virtual Private Gateway or Transit Gateway is attached to a VPC and used with a Border Gateway Protocol (BGP) session over a VPN or a dedicated circuit, the on-premises router advertises its reachable networks over that BGP session, and if you enable route propagation on the route table, those advertised networks are added to the route table automatically instead of you hand-entering static routes. This matters operationally: if the on-premises network adds a new subnet, propagation means your VPC route table updates itself the next time BGP re-converges, with no manual change needed on the cloud side.

Overlapping or conflicting routes from multiple external networks. If you connect to two external networks (say, two data centers, or a data center and a partner network over separate VPN connections) and their advertised CIDR blocks overlap, or if a statically entered route and a BGP-propagated route both cover the same destination, you have to resolve the ambiguity explicitly, because longest-prefix match only helps when the prefixes actually differ in specificity:

• If one advertised range is a strict subset of the other, longest-prefix match naturally sends traffic for the subset to whichever path advertised the more specific range, so no conflict actually exists even though it looks like one on paper.
• If the two ranges genuinely overlap at the same specificity (both advertise the same /24, for instance), most cloud route tables will simply reject adding the second, identical-specificity route, or, on platforms that support route priority, you set an explicit static route with a fixed destination as the tie-breaker. The durable fix, though, is address planning: two networks that both intend to be reachable from the same VPC should never have been assigned overlapping CIDR blocks in the first place, since even where the platform lets you work around it, only one of the two networks is actually reachable at overlapping addresses at any moment.
• For BGP-specific tie-breaking (both paths are valid but you want a preference, not an outright conflict), use attributes like AS-path prepending (making a path look longer and therefore less preferred) or a local-preference / route-priority setting supported by the platform, rather than fighting it with static routes that will silently go stale.

Worked example

A VPC with CIDR 10.0.0.0/16, an Internet Gateway (IGW), and a Virtual Private Gateway (VGW) with route propagation enabled to an on-premises network at 192.168.0.0/16:

Here, the manually entered /24 route deliberately overrides part of the broader propagated /16 route for one subnet, a legitimate and common pattern for routing one sensitive on-premises subnet over a dedicated path while everything else in that address range uses the general one.

Internet Gateway vs Virtual Private Gateway, since they are easy to conflate: an Internet Gateway is a horizontally scaled, highly available AWS-managed component that gives a VPC a path to and from the public internet; it has no concept of BGP or on-premises networks. A Virtual Private Gateway is the VPN/Direct Connect-side endpoint on the AWS side of a hybrid connection; it terminates VPN tunnels or a Direct Connect (AWS's dedicated, private network circuit, as opposed to a VPN's tunnel over the public internet) virtual interface and is what participates in BGP route propagation with an on-premises router. They serve two entirely different paths (public internet vs private hybrid connectivity) and a VPC commonly has both at once, each referenced by different routes in the same route table.

Trade-offs and pitfalls

The most common mistake is assuming route order in a table or console listing matters, then getting confused when a "later" broad route does not override an earlier specific one; only prefix length decides precedence, never entry order. A second pitfall is enabling route propagation without auditing what gets propagated: an on-premises network that advertises a default route (0.0.0.0/0) over BGP will silently start competing with your Internet Gateway's own default route for precedence (resolved by longest-prefix match rules that still apply, but the practical effect, if the propagated route wins for some traffic class, can be egress traffic unexpectedly hairpinning through on-premises). Finally, when connecting to multiple external networks, resolve address-space overlap at design time through CIDR planning, not at incident time through static-route workarounds; a static override fixes today's symptom and quietly becomes tomorrow's undocumented special case.