AWS Core Services and Architecture Questions

gp2, gp3, io1, and io2 (including io2 Block Express) trade off baseline performance, how independently you can provision input/output operations per second (IOPS) and throughput, and cost. gp2 ties IOPS to volume size, gp3 decouples IOPS and throughput from size and lets you provision each independently at a lower cost, and the io family (io1, io2, io2 Block Express) is built for the highest, most consistent IOPS with the strongest durability guarantee, at a real cost premium. For a steady, high-IOPS online transaction processing (OLTP) database, the practical decision is usually gp3 up to its provisioning ceiling for cost efficiency, and io2 (or io2 Block Express on a Nitro-based instance, an instance on AWS's Nitro virtualization platform, which most current-generation types use) once the workload needs IOPS beyond what gp3 offers or needs io2's higher durability target.

Comparison

These are AWS's current published ceilings, and AWS has raised them more than once over the product's life (gp3's original limits, for example, were lower than the 80,000 IOPS / 2,000 MiB/s shown here), so re-check the live EBS documentation before sizing a production volume rather than trusting a remembered number.

Choosing type, size, and provisioned IOPS for a steady high-IOPS OLTP database

1. Establish the real requirement first: sustained IOPS (not peak), required throughput, a latency target (commonly sub-millisecond to low single-digit milliseconds for OLTP), required capacity, and how strict the durability requirement is.
2. If the sustained IOPS requirement fits within gp3's provisioning ceiling and the workload does not need io2's higher durability target, gp3 is the more cost-efficient choice: provision IOPS and throughput to match the measured requirement rather than over-provisioning by default, since both are billed independently of volume size.
3. If the requirement exceeds gp3's ceiling, or the workload is business-critical enough that io2's stronger durability target matters, move to io2, and to io2 Block Express specifically once IOPS or throughput needs exceed io2's non-Block-Express ceiling and the instance is Nitro-based.
4. Size the volume for capacity, not just performance. On gp2, capacity and IOPS were coupled, so undersizing hurt performance; on gp3 and io2 they are decoupled, so size for actual data growth and headroom, and provision IOPS and throughput as a separate decision.
5. Validate against the attached instance, not just the volume: provisioned IOPS are only fully usable if the EC2 instance's own network and storage bandwidth can actually deliver them, so check the instance type's EBS throughput limit alongside the volume's, and prefer a current Nitro-based instance family, which is required to reach the highest gp3 and io2 Block Express tiers anyway.

Tuning and troubleshooting at the instance level (depth beyond what most interviews on this comparison require)

The gp3-vs-io2 decision above is the core answer; the checks below are for when a provisioned number isn't showing up in practice and the conversation goes deeper into instance-level troubleshooting.

• Nitro-based instances are a prerequisite, not an optimization, for the top IOPS and throughput tiers on gp3 and io2 Block Express; an older, non-Nitro instance type simply cannot reach those ceilings regardless of what is provisioned on the volume.
• The Elastic Network Adapter (ENA) driver affects network throughput, which matters because EBS traffic on a Nitro-based instance travels over the network fabric to the storage layer; an outdated or misconfigured ENA driver on the guest operating system can cap effective throughput well below what the instance and volume are both capable of.
• Operating-system tuning (I/O scheduler choice, queue depth, filesystem mount options) affects how much of the volume's provisioned IOPS the application can actually realize; the default scheduler and queue depth on an older Linux image is sometimes tuned for spinning disks, not high-IOPS solid-state storage.
• Troubleshooting EBS I/O wait: a high %iowait in iostat alongside a healthy CloudWatch VolumeQueueLength usually points at the instance side (driver, scheduler, or a bandwidth ceiling on the instance type); a high CloudWatch VolumeQueueLength alongside low instance CPU usually points at the volume being under-provisioned for the actual demand.

Worked example

A database needs a sustained 10,000 IOPS and 300 MiB/s of throughput on a 2 tebibyte (TiB) volume, for an OLTP workload with a normal, not extreme, durability requirement. Both gp3 and io2 can provide this comfortably, since gp3's ceiling on a Nitro-based instance is well above it. Choosing gp3, provisioned at exactly 10,000 IOPS and 300 MiB/s, costs the per-GiB charge plus the provisioned-IOPS and provisioned-throughput charges above baseline; choosing io2 for the same numbers costs more per IOPS for a durability guarantee this workload does not need, so gp3 is the right call here. Now change the scenario: the workload is write-heavy and needs 50,000 sustained IOPS, still on a Nitro-based instance. That is still within gp3's published ceiling on paper, but at that volume of sustained writes the stronger durability target of io2 (or io2 Block Express, if throughput needs also exceed io2's own ceiling) becomes the more defensible choice for a database this business depends on, even at the higher per-IOPS cost, because the cost of io2 is small next to the cost of a durability incident on a workload sustaining that much write traffic.

Trade-offs and pitfalls

• Do not size purely on peak IOPS observed during a rare spike; provisioning for sustained need with headroom is cheaper and usually sufficient, since bursts of a few seconds rarely need dedicated provisioned capacity.
• gp2's IOPS-scales-with-size model means undersizing a gp2 volume silently caps performance; this is one of the reasons gp3 (decoupled) is now the default recommendation over gp2 for most new workloads.
• Provisioned IOPS on the volume are capped by what the attached EC2 instance type can actually deliver; provisioning far more IOPS than the instance's own bandwidth ceiling wastes money.
• io2 Block Express's highest tier requires a Nitro-based instance; verify the instance family before assuming the ceiling applies, and check the ENA driver and operating-system I/O tuning before concluding a performance shortfall is the volume's fault rather than the instance's.

Layer the controls rather than relying on one: deny public access by default at both the account and bucket level, prefer IAM (Identity and Access Management) and bucket policies over ACLs (Access Control Lists) for access control, encrypt at rest (SSE-S3: Server-Side Encryption with Amazon S3-managed keys, as a baseline; SSE-KMS: Server-Side Encryption with AWS Key Management Service keys, with an auditable key policy for sensitive data), require TLS (Transport Layer Security) on every request, and for another account that needs read access, grant a scoped IAM role and bucket-policy combination rather than making anything public.

Block Public Access

Enable S3 Block Public Access at both the account level and the bucket level. Account-level coverage matters because bucket-level settings alone leave every other bucket in the account exposed to a future misconfiguration. Avoid ACLs entirely for new designs; they're the legacy mechanism and bucket policies/IAM give more precise, auditable control.

Bucket policies vs. IAM policies

IAM policies are principal-scoped: attach them to the users/roles that should have access. Bucket policies are resource-scoped and are the right tool for cross-account access or for conditions that must apply regardless of which principal is making the request, such as denying non-TLS traffic. In practice, sensitive buckets use both: IAM policies for your own account's principals, and a bucket policy carrying explicit deny statements (non-TLS, non-org principals) as a backstop.

Server-side encryption options

• SSE-S3: S3-managed keys, no extra setup, a reasonable baseline for non-sensitive data.
• SSE-KMS: keys managed in KMS, with a separate key policy controlling who can decrypt, and CloudTrail logging of every KMS usage. This is the right choice for sensitive data because it adds a second, independently auditable permission system on top of S3's own.
• SSE-C: the client supplies and transmits the encryption key on every request. It's operationally fragile (the key isn't stored by AWS at all) and rarely worth choosing over SSE-KMS.

Enforce TLS

Add a bucket policy statement that denies any request where aws:SecureTransport is false, so plaintext HTTP requests are rejected outright rather than relying on clients to always use HTTPS.

Cross-account read without making the bucket public

The pattern is: the consuming account gets an IAM role scoped to exactly what it needs, and the bucket policy grants that specific role (not the whole account, and never *) read access to a specific prefix.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountReadViaRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::222222222222:role/PartnerReadRole"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::acme-shared-reports/exports/*",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "222222222222"
        }
      }
    },
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::acme-shared-reports",
        "arn:aws:s3:::acme-shared-reports/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

If the objects are SSE-KMS encrypted, the bucket policy alone isn't enough: the KMS key policy also has to grant kms:Decrypt to that same role, because S3 permissions and KMS permissions are two separate systems that both have to say yes.

Serving both private documents and public static assets from the same platform

Don't try to solve this with one bucket and per-object ACL exceptions to Block Public Access; a single misconfigured object exposes data. Use two buckets: a private bucket for sensitive documents, kept fully behind Block Public Access and accessed only via presigned URLs or app-tier logic, and a separate public-assets bucket served through CloudFront using Origin Access Control (OAC), the current recommended mechanism (replacing the older Origin Access Identity, OAI) for letting only CloudFront read a bucket that itself still blocks all direct public access. "Public" delivery then flows through CloudFront's edge, not through a bucket that's actually open to the internet.

Trade-offs and pitfalls

• Forgetting the KMS side of a cross-account grant is the single most common cause of "the bucket policy says allow, but I still get Access Denied" for SSE-KMS objects.
• Using OAI on a new CloudFront-to-S3 setup instead of OAC is legacy practice; OAC is the current recommendation and integrates better with SSE-KMS.
• Bucket-level Block Public Access without the account-level setting leaves other buckets in the account exposed to future mistakes.
• SSE-C's client-managed-key model is rarely worth the operational fragility versus SSE-KMS for anything beyond a narrow compliance requirement that specifically mandates it.

Direct answer

Virtual Private Cloud (VPC) Peering, AWS Transit Gateway (TGW), and AWS PrivateLink solve different connectivity problems, and none is a drop-in replacement for another. A VPC is your own isolated network inside AWS, and these three services differ in how they connect one VPC to another. Peering gives you a direct, non-transitive join between exactly two VPCs. Transit Gateway is a regional (and, via peering, inter-region) routing hub that lets many VPCs, VPNs, and Direct Connect attachments reach each other transitively through shared route tables. PrivateLink exposes a single application endpoint from one VPC into another without joining their networks at all, so it sidesteps both the transitive-routing and the overlapping-CIDR problems that Peering and TGW have. Reach for Transit Gateway once you have more than a handful of VPCs that all need to talk to each other or need hybrid on-premises connectivity; use PrivateLink when you only need to expose one service, not a whole network.

Structured elaboration

Why transitive routing and overlapping CIDRs are the deciding factors

• Peering's transitivity gap means that at organizational scale you either accept a full mesh, every pair of VPCs that needs to talk gets its own peering connection, or you funnel everything through one or two hub VPCs using proxies, both of which get operationally painful past a small VPC count.
• Transit Gateway solves transitivity because every attachment (VPC, VPN, Direct Connect) associates with a TGW route table; routes propagate automatically, and you can segment traffic with multiple route tables, for example a shared-services route table versus a production-isolation route table, rather than one flat mesh.
• Neither Peering nor TGW tolerates overlapping IP ranges on the VPCs you are trying to route between, because the whole point is extending IP routing across the VPC boundary. If two business units both built their VPCs on the same CIDR block by accident, you cannot peer or TGW-attach them without re-addressing one side or inserting NAT in between.
• PrivateLink avoids the CIDR problem entirely because it never routes between the two VPCs' address spaces. The consumer VPC gets an elastic network interface (ENI) with a private IP from its own subnet; that ENI forwards to the provider's load balancer behind the scenes. Overlapping CIDRs on either side are irrelevant because there is no VPC-to-VPC route being created, just a consumer-to-endpoint hop.

At scale: hundreds of VPCs and inter-region TGW peering

When an organization has hundreds of VPCs across many accounts, the standard pattern is a hub-and-spoke Transit Gateway per region, shared to member accounts via AWS Resource Access Manager (RAM) so each account attaches its own VPC without owning the TGW. For multi-region footprints, you connect the per-region Transit Gateways with TGW peering attachments. This gives inter-region connectivity between the two TGWs, but it is still not fully automatic: each TGW's route table has to explicitly propagate routes across the peering attachment, and traffic through a TGW peering attachment is bound by that attachment's own bandwidth, so you plan route propagation and bandwidth deliberately rather than assuming that peering the TGWs alone makes everything route.

Worked example

If you tried to fully connect 20 VPCs with VPC Peering alone, so that every VPC can reach every other VPC directly, you would need one peering connection per pair:

$\binom{20}{2} = \frac{20 \times 19}{2} = 190\ \text{peering connections}$

Each of those 190 connections needs its own route table entries on both sides and its own security-group hygiene. With Transit Gateway, the same 20 VPCs need 20 attachments total, one per VPC, and route propagation handles the rest. That difference, quadratic in VPC count versus linear in VPC count, is the concrete reason TGW becomes the default once you are past a handful of VPCs.

Trade-offs and pitfalls

• Transit Gateway is not free complexity-wise: it charges per attachment-hour and per GB processed, and a flat "everyone can reach everyone" TGW route table is itself a security anti-pattern; use separate route tables to segment, for example isolating a security or inspection VPC that other traffic must pass through.
• PrivateLink's per-AZ endpoint pricing and the fact that it only exposes what the provider puts behind the load balancer make it the wrong tool if the consumer genuinely needs broad network reachability, multiple ports, or arbitrary protocols, rather than a single well-known service.
• A common mistake is reaching for Peering because it is the simplest term you know, then discovering months later that the mesh has become unmanageable; if you can already picture needing a fourth or fifth VPC in the conversation, start with Transit Gateway.
• TGW peering across regions does not auto-propagate routes by default; each side's route table needs an explicit route to the peered TGW's CIDRs, so "we peered the TGWs" is not the same as "everything now routes," verify route propagation on both ends.

Direct answer

S3 is object storage: you PUT and GET whole objects over HTTP, there's no file system underneath, and it's built for massive scale and durability rather than low latency. EBS (Elastic Block Store) is block storage: a virtual disk attached to a single EC2 instance, the same abstraction as a local SSD, used for anything that needs a real file system and fast random I/O. EFS (Elastic File System) is a managed NFS (Network File System): POSIX file semantics, but shared, so many instances or containers can mount and read/write the same file system at once. For a three-tier app: S3 for user uploads, EBS for the database, and EFS if application binaries or shared assets need to be visible to multiple instances at once, EBS if they don't.

Structured elaboration

Container and multi-instance workloads: this is where the block-vs-file distinction shows up sharply. An EBS volume is fundamentally single-attach: one EC2 instance, or one pod through the EBS CSI (Container Storage Interface) driver, can write to it at a time (ReadWriteOnce). EFS, mounted through the EFS CSI driver, supports many pods reading and writing the same file system concurrently (ReadWriteMany), which is why it's the standard choice for shared config, shared uploads directories, or CI/build-runner workspaces across a container fleet, and EBS is the standard choice for a single stateful pod's own local-disk-equivalent storage.

EFS throughput modes: Bursting throughput scales with how much data you have stored, backed by burst credits, fine for spiky-but-moderate workloads. Provisioned throughput lets you set a fixed throughput independent of storage size, for consistently high-throughput needs. Elastic throughput automatically scales up and down with your actual workload without you provisioning anything, and is the default recommendation today for unpredictable or spiky access patterns, since it removes the need to guess a provisioned number or babysit burst-credit balances.

Worked example

For a typical three-tier app (web/app tier, storage for uploads, database):

• User uploads (photos, documents, exports): S3. They're immutable-ish blobs accessed by key, don't need POSIX semantics, and S3's per-request pricing and lifecycle tiers fit "write once, read occasionally" access well.
• Application binaries: depends on deployment model. If instances are deployed from a container image or AMI (Amazon Machine Image), binaries usually don't need separate persistent storage at all. If they do need to live outside the boot volume and be shared across a fleet, EFS. If each instance just needs its own local copy on its own boot/root volume, that's already EBS by default (every EC2 instance's root volume is EBS-backed).
• Database: EBS, provisioned for consistent low-latency IOPS, whether that's the storage backing a self-managed database on EC2 or, more commonly for a new build, the storage layer underneath a managed RDS instance (RDS itself is EBS-backed, just abstracted away from you).

Trade-offs & pitfalls

A common mistake is choosing EFS by default for "shared storage" without checking whether shared write access is actually needed, EFS costs more per GB than S3 at scale and adds NFS-level latency that a single-attach EBS volume wouldn't have, so it should be a deliberate choice, not a default. Another is forgetting EBS volumes are AZ-locked: an instance can't attach a volume from a different AZ, which matters directly for the high-availability design of the database layer. Finally, S3 is not a drop-in file system, if application code assumes it can open, seek, and append to a file in place, S3 will not behave like that; EFS or EBS is the right answer whenever true in-place file mutation is required.

Direct answer

Placement groups let you influence how EC2 places instances on underlying hardware, trading network performance against fault isolation. Cluster packs instances close together on the same high-bandwidth network segment inside one Availability Zone (AZ) for the lowest latency and highest throughput. Spread puts a small number of instances on physically distinct hardware to minimize correlated failure. Partition groups instances into logical partitions, each on its own set of racks, so a single rack failure only takes out one partition, which is how large distributed systems like Kafka or Cassandra stay available.

Structured elaboration

A few rules apply across all three: an instance can belong to only one placement group at a time, placement groups cannot be merged, you can't launch a Dedicated Host into a placement group, and you can't launch a Spot Instance configured to stop or hibernate on interruption into one.

Worked example

For a tightly coupled HPC job needing very low node-to-node latency (say an 8-node MPI simulation), I'd launch a cluster placement group with all 8 instances of the same instance type in a single launch request, in one AZ, using an enhanced-networking-capable instance family; within a cluster placement group, enhanced-networking instances get up to 10 Gbps for single-flow traffic versus 5 Gbps for instances outside one. For a Kafka cluster that needs rack-fault isolation across many more brokers than a spread group's 7-per-AZ cap allows, I'd use a partition placement group instead, with up to 7 partitions in the AZ; Kafka's own rack-awareness can consume the partition topology AWS exposes so a rack failure only affects the brokers in one partition.

Trade-offs & pitfalls

• Cluster placement groups trade fault tolerance for latency: everything sits in one AZ, so an AZ-level event takes the whole group down together. Pair it with cross-AZ checkpointing or keep the control plane elsewhere.
• Adding instances to a cluster group later, or mixing instance types within it, raises the odds of an insufficient-capacity error, since AWS is trying to keep everything on the same network segment.
• You cannot convert a group's strategy after creation and cannot merge two groups; if you outgrow the 7-per-AZ spread limit, the fix is multiple spread groups, which gives no guarantee of spread between the groups themselves.
• You can move a stopped instance into, out of, or between placement groups, but only while it's stopped, so this isn't a live-migration tool.
• Capacity Reservations behave differently per strategy: they don't reserve capacity in spread or partition groups at all, only in cluster groups (via a group-scoped On-Demand Capacity Reservation), which catches people who assume reservations work the same way everywhere.