AWS Core Services and Architecture Questions

Direct answer

S3 is object storage: you PUT and GET whole objects over HTTP, there's no file system underneath, and it's built for massive scale and durability rather than low latency. EBS (Elastic Block Store) is block storage: a virtual disk attached to a single EC2 instance, the same abstraction as a local SSD, used for anything that needs a real file system and fast random I/O. EFS (Elastic File System) is a managed NFS (Network File System): POSIX file semantics, but shared, so many instances or containers can mount and read/write the same file system at once. For a three-tier app: S3 for user uploads, EBS for the database, and EFS if application binaries or shared assets need to be visible to multiple instances at once, EBS if they don't.

Structured elaboration

Container and multi-instance workloads: this is where the block-vs-file distinction shows up sharply. An EBS volume is fundamentally single-attach: one EC2 instance, or one pod through the EBS CSI (Container Storage Interface) driver, can write to it at a time (ReadWriteOnce). EFS, mounted through the EFS CSI driver, supports many pods reading and writing the same file system concurrently (ReadWriteMany), which is why it's the standard choice for shared config, shared uploads directories, or CI/build-runner workspaces across a container fleet, and EBS is the standard choice for a single stateful pod's own local-disk-equivalent storage.

EFS throughput modes: Bursting throughput scales with how much data you have stored, backed by burst credits, fine for spiky-but-moderate workloads. Provisioned throughput lets you set a fixed throughput independent of storage size, for consistently high-throughput needs. Elastic throughput automatically scales up and down with your actual workload without you provisioning anything, and is the default recommendation today for unpredictable or spiky access patterns, since it removes the need to guess a provisioned number or babysit burst-credit balances.

Worked example

For a typical three-tier app (web/app tier, storage for uploads, database):

• User uploads (photos, documents, exports): S3. They're immutable-ish blobs accessed by key, don't need POSIX semantics, and S3's per-request pricing and lifecycle tiers fit "write once, read occasionally" access well.
• Application binaries: depends on deployment model. If instances are deployed from a container image or AMI (Amazon Machine Image), binaries usually don't need separate persistent storage at all. If they do need to live outside the boot volume and be shared across a fleet, EFS. If each instance just needs its own local copy on its own boot/root volume, that's already EBS by default (every EC2 instance's root volume is EBS-backed).
• Database: EBS, provisioned for consistent low-latency IOPS, whether that's the storage backing a self-managed database on EC2 or, more commonly for a new build, the storage layer underneath a managed RDS instance (RDS itself is EBS-backed, just abstracted away from you).

Trade-offs & pitfalls

A common mistake is choosing EFS by default for "shared storage" without checking whether shared write access is actually needed, EFS costs more per GB than S3 at scale and adds NFS-level latency that a single-attach EBS volume wouldn't have, so it should be a deliberate choice, not a default. Another is forgetting EBS volumes are AZ-locked: an instance can't attach a volume from a different AZ, which matters directly for the high-availability design of the database layer. Finally, S3 is not a drop-in file system, if application code assumes it can open, seek, and append to a file in place, S3 will not behave like that; EFS or EBS is the right answer whenever true in-place file mutation is required.

Direct answer

If DynamoDB is write-throttling despite aggregate provisioned or on-demand capacity looking sufficient, the near-universal cause is a hot partition: a small number of partition-key values absorbing a disproportionate share of writes, so one physical partition's own throughput ceiling is exceeded even though the table-wide capacity has headroom. The fix is almost never "add more capacity"; it's spreading the writes across more partition-key values, and the fix isn't done until it's validated with a load test that reproduces the actual traffic shape that caused the incident.

Structured elaboration

1. Confirm it's a hot partition, not a global capacity shortfall. In CloudWatch, compare ThrottledRequests/WriteThrottleEvents against ConsumedWriteCapacityUnits at the table level; if consumed capacity is well below what's provisioned (or below the on-demand ceiling) while throttling is happening, capacity isn't the bottleneck, distribution is.
2. Find the hot key(s). Turn on CloudWatch Contributor Insights for the table (or, if not already enabled, instrument application-level logging of partition-key values on write) to surface the top N keys by request count. Look specifically for low-cardinality keys, monotonically increasing keys (like a timestamp used directly), or a single shared "global" key that many writers hit.
3. Apply short-term mitigation to stop the bleeding. Client-side exponential backoff with jitter, and route incoming writes through a durable buffer (SQS or Kinesis) drained at a controlled rate, so the application doesn't cascade retries into more throttling.
4. Redesign the partition key for the long term. Two complementary techniques:
   • Write sharding (prefix/suffix): append a shard suffix (a small hash or random number) to a hot key so a single logical entity's writes spread across several physical partitions.
   • Time-bucketing: for a shared or "global" key, fold a time bucket (hour, or a finer grain under load) into the partition key so writes naturally spread as new buckets open, rather than piling onto one everlasting key.
5. Decide on-demand vs provisioned deliberately, not reflexively. On-demand absorbs unpredictable spikes without you managing capacity, but it doesn't fix a genuinely skewed key design, a heavily hot key will still throttle on-demand once its write rate exceeds a single partition's ceiling; provisioned with a corrected key design is more predictable and usually cheaper for sustained, forecastable load.
6. Validate with a load test before calling it fixed, using a request generator whose partition-key distribution matches what Contributor Insights showed for the actual incident (not uniformly random synthetic keys, which would hide the exact skew that caused the problem), and confirm both that ConsumedWriteCapacityUnits is now spread evenly across the new key space and that ThrottledRequests stays at zero under that replayed load.

Worked example

Before: an IoT ingestion table uses DEVICE#<deviceId> as the partition key. A handful of devices (or, worse, a shared counter table using one literal key like METRICS#GLOBAL) send far more writes than the rest, so all of their writes land on the same physical partition regardless of the table's total provisioned throughput.

After, write sharding: the partition key becomes DEVICE#<deviceId>#<shard>, where shard is computed as hash(deviceId + writeTimestamp) mod N for some shard count N chosen so that a single device's peak write rate, divided across N shards, comfortably fits under one partition's throughput ceiling. Reads that need "everything for device X" now issue N parallel Query calls (one per shard value) and merge the results, or are served from a Global Secondary Index if a consolidated view is required.

After, time-bucketing (for the shared-counter case): the partition key becomes METRICS#2026-07-20-14 (an hour bucket) instead of a single unchanging METRICS#GLOBAL key, so writes spread across time as new hourly buckets open automatically. The current hour's bucket can still be hot within that hour, which is exactly why the two techniques are usually combined: bucket by time, and shard within the current bucket.

Validation: replay the write pattern captured from the incident (same relative request rate and same key skew observed via Contributor Insights) against a staging table with the new key design, and confirm ThrottledRequests is zero and per-shard ConsumedWriteCapacityUnits is roughly even, before rolling the schema change to production, and again at expected future peak, not just historical peak, since traffic grows.

Trade-offs and pitfalls

• Sharding adds read-side complexity: a query that used to be one Query call becomes N parallel calls (or requires a GSI you now have to keep in sync), which is real ongoing cost, not a one-time redesign expense.
• DynamoDB's adaptive capacity helps absorb transient hot keys automatically, but it's reactive and bounded; it is not a substitute for correcting a sustained, structurally skewed key design, and relying on it alone will resurface the same incident at higher scale.
• Switching to on-demand can mask the symptom for many workloads, but a key that's skewed enough to exceed a single partition's ceiling will still throttle under on-demand; treat on-demand as a cost and predictability trade-off, not a fix for the key design.
• A load test that uses uniformly random synthetic keys instead of the actual observed skew will pass cleanly and still let the same incident recur in production, because it never reproduces the condition that caused the original throttling.

Direct answer

A strongly consistent read always returns the most recently committed value (served from the leader replica), an eventually consistent read may return slightly stale data (typically caught up within a second) but can be served from any replica. In provisioned capacity terms, a strongly consistent read of a 4 KB item costs 1 Read Capacity Unit (RCU); an eventually consistent read of the same item costs half that, 0.5 RCU, because DynamoDB can spread the work across replicas. For a low-latency, high-read leaderboard, use eventually consistent reads for the general (global) view and reserve strong consistency only for the one case that needs it: a user checking their own just-submitted score.

Structured elaboration

Consistency and cost

Leaderboard design

• Source of truth: a DynamoDB table keyed by (leaderboardId, userId) storing each player's current score, updated via an atomic UpdateItem with an ADD (or conditional set) expression so concurrent score updates don't lose writes.
• Fast read path: a cache in front of DynamoDB for the hot "top N" and "my rank" queries, since even a 0.5 RCU eventually consistent read doesn't beat an in-memory sorted structure for sub-millisecond top-N retrieval at high QPS. A managed in-memory cache holding a sorted-set-like structure per leaderboard (score to userId) serves top-N and rank lookups directly from memory.
• Write path: score update writes to DynamoDB first (source of truth), then updates the cache synchronously where feasible, or asynchronously via DynamoDB Streams if you want the write path decoupled from cache availability. Streams-based propagation also gives you a natural retry/replay mechanism if the cache falls behind.
• Consistency split: global "top 100" and "leaderboard around me" reads hit the cache (effectively eventually consistent, and fine for that use case). A player's own current score/rank, right after they submit it, can use a strongly consistent GetItem against DynamoDB so the UI never shows a stale "your score" the instant after they played.

Worked example

A mobile game leaderboard serving 5,000 reads/sec and a much smaller volume of score updates: the "top 100" view and "nearby ranks" view are read from the cache, so DynamoDB only sees the write volume (score updates) plus the low-volume strongly consistent "my current score" reads, not the full 5,000 reads/sec. This keeps provisioned RCU (or on-demand request cost) proportional to writes and to the smaller strongly consistent read slice, not to total leaderboard traffic. If the cache and DynamoDB briefly disagree (cache slightly behind after a burst of writes), that's acceptable for "top 100" but not for "my score," which is exactly why that one query path bypasses the cache and reads DynamoDB directly with strong consistency.

Trade-offs and pitfalls

• Don't default every read to strongly consistent "to be safe": it doubles RCU cost and caps throughput at what the leader replica can serve, which is the wrong trade-off for a leaderboard where 99% of reads are the shared, cacheable top-N view.
• A cache-first design needs an explicit staleness bound (a short time-to-live or streams-driven refresh) or players will see visibly wrong ranks after a burst of updates; decide and document how stale "eventually" is allowed to be.
• Rank computation itself doesn't come from RCU/WCU semantics; a sorted-set-style cache gives ordered rank cheaply, whereas computing rank from DynamoDB alone would need a full scan or a separate rank-tracking scheme, so the cache isn't just a latency optimization here, it's doing work DynamoDB isn't shaped to do efficiently.
• A common pitfall is forgetting to also protect the write path: high-frequency score updates on the same userId are fine (single partition per user is bounded), but a very "hot" leaderboard with an extreme write rate across many users still needs the table's partition key chosen so writes spread across partitions, not concentrated by a poorly chosen key.

Direct answer

An Auto Scaling Group (ASG) continuously compares actual healthy capacity against a desired capacity number and launches or terminates EC2 instances from a Launch Template to close the gap. Three subsystems make this work: the Launch Template defines what to launch, health checks decide which instances actually count toward capacity, and scaling policies decide what the desired capacity should be right now.

Structured elaboration

• Launch Template as source of truth: the ASG references a specific Launch Template and version (or $Latest) for the Amazon Machine Image (AMI), instance type(s), network, an AWS Identity and Access Management (IAM) role, and user data. A MixedInstancesPolicy lets one group launch several instance types or purchase options from the same template.
• Health checks: EC2 status checks confirm the instance itself is healthy; if the group is attached to a load balancer or target group, Elastic Load Balancing (ELB)/target-group health checks confirm the application is actually serving traffic, which EC2-only checks can't see. A configurable health check grace period stops the group from replacing an instance that's still booting.
• Lifecycle hooks: pause an instance in a Pending:Wait or Terminating:Wait state so custom code (Lambda, AWS Systems Manager (SSM) Automation, or a script polling the group) can run before the instance is marked in service (register with a service mesh, warm a cache) or before it's terminated (drain connections, flush in-flight work), then explicitly signal CONTINUE or ABANDON.
• Warm pools: keep a stopped or pre-initialized pool of instances outside the desired-capacity count, so a scale-out event can resume a warm instance instead of paying full cold-boot latency.
• Scaling policy types: target tracking keeps a metric such as average CPU or an Application Load Balancer's request count per target at a chosen value, similar to a thermostat; step scaling makes discrete capacity changes sized to how far a CloudWatch alarm has been breached; scheduled scaling changes capacity at known times for predictable patterns.

flowchart LR
    CW[CloudWatch metric] --> SP[Scaling policy]
    SCHED[Schedule] --> SP
    SP --> DC[Desired capacity]
    DC --> ASG[Auto Scaling Group]
    LT[Launch template] --> ASG
    ASG --> LH1[Lifecycle hook launch]
    LH1 --> INS[Instance in service]
    INS --> HC{Health check}
    HC -->|healthy| DC
    HC -->|unhealthy| LH2[Lifecycle hook terminate]
    LH2 --> TERM[Instance terminated]
    TERM --> ASG

Worked example

A web service sits behind an Application Load Balancer. A scheduled scaling action sets desired capacity to 4 instances at 07:00, before the known daily traffic ramp, and back to 2 at 20:00. On top of that, a target-tracking policy on the ALB's request-count-per-target metric, targeted at 500 requests per target, absorbs variance the schedule doesn't predict: if a spike pushes the per-target rate above 500, the group adds instances until it settles back down. When a new instance launches, a lifecycle hook holds it in Pending:Wait while an SSM Automation document installs the current config bundle and confirms a local health endpoint returns 200, then calls complete-lifecycle-action with CONTINUE; only then does the ALB health check start counting it toward capacity.

Trade-offs & pitfalls

• Scheduled scaling is cheap and precise for known patterns but blind to anything unplanned. Target tracking is the safety net but reacts on the lag of its metric window; for a sharp step-function spike, step scaling's fixed-size jumps per alarm breach can respond faster than target tracking's smoother, proportional adjustments.
• A health check grace period that's too short causes flapping, killing instances mid-boot before the app ever gets a chance to pass a check; too long, and a genuinely broken instance stays in service longer than it should.
• A lifecycle hook whose custom code never reports back (a Lambda crashes, a heartbeat isn't sent) holds the instance in its wait state until the hook's timeout, then the configured default action fires; forgetting to set that default action explicitly is a common gap.
• Combining ELB health checks with EC2 status checks is usually right for anything behind a load balancer, since EC2-only checks won't catch an instance that's up but whose application has hung.

Direct answer

CloudFront is AWS's content delivery network (CDN): a global network of edge locations (points of presence, or POPs) that caches and serves content close to the requester. You put it in front of S3, an ALB (Application Load Balancer), or API Gateway for the same core reason in each case: it terminates TLS and serves cacheable responses from an edge location instead of the origin, which cuts latency, cuts load and cost on the origin, and gives you one place to attach security controls like AWS WAF (Web Application Firewall) and origin failover.

Structured elaboration

Why each origin benefits

• S3: caches static assets at the edge, reduces GET request cost and cross-region data transfer from the bucket, and, combined with Origin Access Control (OAC, the current recommended mechanism, the older Origin Access Identity/OAI still works but OAC is what AWS now recommends for new setups), lets you keep the bucket itself fully private while still serving it publicly through CloudFront.
• ALB: caches static or semi-static responses from an otherwise dynamic app, hides the ALB's origin details from the public internet, and reduces backend load for anything cacheable.
• API Gateway: caches idempotent, repeatable API responses at the edge, cutting both latency and the compute cost of re-running the same request behind the API.

Caching behavior and headers: CloudFront respects the origin's Cache-Control and Expires headers by default, and a distribution's cache behavior settings (minimum, default, and maximum TTL) can override or clamp what the origin sends. Cache-Control: no-store or no-cache should be used for anything sensitive or per-user. The Vary header, or explicitly configuring which headers/cookies/query strings are part of the cache key, matters whenever the response actually differs by those values, otherwise you either get incorrect cache hits (serving one user's response to another) or an accidentally low hit ratio from over-including things in the cache key.

Invalidation strategies

• Prefer asset versioning (a fingerprinted filename or a version query string baked in at build time) for anything you control the deploy pipeline for: a new version is simply a cache miss the first time it's requested, so there's nothing to invalidate.
• Use the CloudFront invalidation API for ad hoc, unplanned purges, a hotfix to an HTML page, for example, keeping in mind that invalidation is not instantaneous across all edge locations and has its own cost profile at high volume.
• Use a short TTL instead of frequent invalidation when a resource changes often and brief staleness is acceptable.

Origin failover: configure an origin group with a primary and a secondary (failover) origin. CloudFront health-checks the primary and automatically routes to the secondary on defined failure conditions (5xx responses, timeouts), which is a straightforward way to get basic availability protection, for example, primary origin an ALB, secondary origin a static S3 fallback page, without building custom failover logic.

Private content: signed URLs (per-object, short-lived access) and signed cookies (grant access to a whole set of paths for a session) are how you gate content behind CloudFront, using a trusted key group, the current recommended mechanism for managing the signing keys. (The older approach, CloudFront key pairs tied to the AWS account root user, still exists but requires root-level access to manage and is not the current best practice; trusted key groups can be managed by any sufficiently permissioned IAM identity.)

Edge customization: CloudFront Functions (lightweight JavaScript, for high-volume, low-latency tasks like header rewrites, redirects, or cache-key normalization) and Lambda@Edge (heavier logic, like verifying a signed JWT or fetching a secret) both let you run code at the edge as part of the request/response cycle, without a round trip to the origin.

Worked example

A documentation site behind CloudFront, with an S3 origin: static assets (JS/CSS bundles) are given fingerprinted filenames and a long max-age (effectively "cache forever," since a new deploy produces new filenames). The HTML pages that reference those bundles get a short TTL, so a deploy is visible within that TTL without needing any invalidation call at all. If a critical typo fix needs to go out immediately, an invalidation targeting just that one HTML path clears it from all edge locations without waiting out the TTL, a deliberate exception to the versioning-first strategy rather than the default way of handling every change.

Trade-offs & pitfalls

Forwarding every header, cookie, and query string into the cache key "to be safe" is a common mistake that quietly collapses your hit ratio to nearly nothing, because almost every request becomes unique. Relying on frequent invalidations as the primary cache-busting strategy instead of versioned asset names is more expensive and slower to propagate than it needs to be. And leaving the origin (an S3 bucket or an ALB) directly reachable from the internet defeats a large part of the reason to use CloudFront in the first place, OAC (for S3) or restricting the ALB/security group to only CloudFront's traffic is what actually enforces "requests must come through the CDN."