AWS Core Services and Architecture Questions

Direct answer

Vertical scaling, a bigger instance, is the fastest lever and the right first move for a short-term burst: no application changes, the same consistency model, done in one resize operation, but it hits a hardware ceiling and its cost tends to grow faster than linearly as you move up instance classes. Horizontal scaling, read replicas for read-heavy load, sharding for write or storage volume that outgrows a single node, scales further but each comes with real complexity: replicas introduce replication lag, a consistency trade-off, and sharding requires real application changes, routing logic, and usually giving up cross-shard transactions and joins. The right sequence for advising a customer is almost always: profile and tune first, then vertical scale for immediate relief, then add replicas for reads, and only reach for sharding once you have evidence that a single primary's write or storage capacity, not just read capacity, is the actual constraint.

Structured elaboration

Vertical scaling

• Strengths: fastest to implement, no application or schema changes, preserves single-node ACID semantics exactly as before.
• Limits: bounded by the largest instance class available for the engine; cost tends to grow faster than the resources you gain as you move to higher-end classes; some concurrency-bound bottlenecks, a single hot lock or a single-threaded portion of the workload, do not improve much from more CPU or RAM.
• Best for: workloads that are genuinely CPU or I/O bound on a single primary, where the dataset comfortably fits on one node.

Horizontal: read replicas

• Strengths: offloads read traffic from the primary, a relatively small application change, route reads to replica endpoints, and can also improve read availability.
• Limits: replication is asynchronous, so replicas run some lag behind the primary; reads from a replica are not guaranteed to reflect the very latest write. Does not help write throughput or storage volume at all, since writes still go through the primary.
• Best for: a read-heavy ratio where the reads that can tolerate slight staleness are the actual bottleneck.

Horizontal: sharding

• Strengths: the only option here that scales write throughput and total storage roughly linearly with node count, since each shard is an independent primary.
• Limits: real application-level complexity, you need a shard key and routing logic, cross-shard joins and multi-row transactions become hard or impossible, and resharding later, if your key choice turns out wrong, is a major operation.
• Best for: sustained growth in write volume or total data size that a single primary genuinely cannot hold or serve, not as a first response to a traffic burst.

Decision framework

1. Always first: profile before scaling anything, slow-query log, execution plans on the hot queries, missing indexes, connection pooling, and a cache in front of the hottest reads. A lot of "we need to scale" turns out to be "we need an index."
2. Short-term, bursty spike: vertical scale, temporarily if the burst is predictable, or as a stopgap while investigating.
3. Sustained read-heavy growth: add read replicas, with monitoring on replication lag so you know how stale a read from a replica actually is.
4. Sustained write or storage growth beyond one node: that is the actual sharding trigger, design the shard key deliberately rather than reaching for it reflexively.
5. Combine, do not choose once: a mature system typically runs vertical sizing plus caching plus read replicas together, and only adds sharding when it has outgrown that combination on the write side specifically.

Worked example

A customer's primary handles 2,000 write transactions per second and is CPU-bound at 90% utilization during bursts, with reads currently mixed in on the same instance at a 4:1 read-to-write ratio. Reads are $4/(4+1) = 80\%$ of total query volume. Moving those reads to two read replicas removes that 80% share from the primary, leaving it handling its 2,000 writes per second plus whatever residual read share it still serves locally. If CPU utilization were proportional to query volume, dropping 80% of the query load would be expected to bring the primary's read-driven load down to roughly:

$90\% \times 20\% = 18\%$

of its prior read-driven utilization, plus whatever the writes alone cost. This is the kind of back-of-envelope math worth doing with a customer's real read-to-write ratio before committing to replicas, because if writes, not reads, turn out to be the actual CPU driver, replicas will not fix the bottleneck, and vertical scaling or sharding is the real lever.

Trade-offs and pitfalls

• Read replicas do not help if the bottleneck is actually writes or lock contention; a customer who describes "high latency during bursts" without first confirming the read-to-write split can end up adding replicas that do not move the needle at all.
• Vertical scaling has a real ceiling, and its cost curve is worth showing the customer explicitly, larger instance classes are not linearly priced against the resources they add, so "just get a bigger box" stops being the answer past a certain point regardless of budget.
• Sharding should be the last resort in this framework specifically because of what it costs in application complexity: cross-shard transactions and joins are genuinely hard, and an ill-chosen shard key can leave you with hot shards, defeating the purpose.
• Always validate the read-to-write ratio and lock-contention picture with real data before recommending a scaling direction; guessing the ratio is the single most common way this advice goes wrong.

Direct answer

Start by correlating timing: pull API Gateway's CloudWatch metrics (ThrottleCount, 4XXError, Count, IntegrationLatency) for the affected stage/method alongside the backend's own error and saturation metrics for the same window. If ThrottleCount rises while the backend's incoming request volume stays flat or drops, API Gateway is throttling before requests even reach the backend. If API Gateway's integration request count keeps climbing but the backend itself is returning its own 429s (or its logs show connection/thread-pool rejections), the backend is the bottleneck and API Gateway is just passing the failure through.

Structured elaboration

Where API Gateway throttling can be applied (and in what precedence order)

API Gateway evaluates throttling in this order, and the tightest applicable limit wins:

1. Per-client (usage plan / API key) throttling for that stage
2. Per-method throttling set on the stage
3. Account-level, per-Region throttling (a default ceiling across all your APIs in that Region)
4. AWS's own Regional throttling ceiling (fixed, not customer-configurable)

API Gateway enforces its limits with a token-bucket algorithm: the rate is how fast tokens refill (steady-state requests/sec), the burst is the bucket's capacity (how many requests can go through instantly before the steady rate takes over). A spike that's short but sharp can exhaust the burst allowance and start getting 429s even if the sustained rate is well within the configured limit.

Diagnosis checklist

• Check whether the traffic is using API keys tied to a usage plan; if so, check that plan's rate/burst settings, since a per-key limit can throttle one client's traffic pattern while the API as a whole has headroom.
• Check stage-level and method-level throttling overrides, since a specific high-traffic method (e.g., a search endpoint) may have a tighter override than the API's general limit.
• Trace a sample of 429 requests end-to-end (distributed tracing, if enabled) to see definitively whether the response originated at API Gateway or was proxied from the backend.
• If a Web Application Firewall (WAF) or Application Load Balancer (ALB) sits in front of API Gateway, check its rate-based rules too; a 429-shaped response can originate there instead of at API Gateway itself.

Worked example

Suppose CloudWatch shows ThrottleCount spiking to several hundred per minute during the traffic spike, while IntegrationLatency and the backend's own request-count metric stay essentially flat during the same window. That pattern says the requests never reached the backend: API Gateway rejected them at the edge because the spike's burst exceeded the configured token-bucket capacity, most likely the account-level or usage-plan burst limit rather than a per-method override (if this were a per-method override, only that one method's traffic would show elevated ThrottleCount while others stayed clean). The fix in this case is at the API Gateway layer: request an account-level limit increase if the API's aggregate legitimate traffic genuinely needs a higher regional ceiling, and/or raise the usage plan's burst allowance for the affected client so short, legitimate spikes don't get rejected before reaching the backend.

If instead ThrottleCount stayed low but the backend's own error logs show its connection pool or thread pool rejecting work and returning its own 429/503, the fix is downstream: autoscale the backend on a leading indicator (queue depth, concurrency, or request latency, not just CPU), and consider adding a buffering layer (a queue) in front of the backend so a burst is absorbed and drained at a steady rate instead of hitting the backend's fixed capacity directly.

Trade-offs and pitfalls

• Raising API Gateway's limits without also confirming the backend can absorb the resulting traffic just moves the 429s downstream, where they show up as backend errors instead, which is a worse failure mode to debug.
• Clients need to implement backoff (respecting the Retry-After behavior implied by 429s, with jitter) regardless of which layer is throttling; without it, a retry storm from many clients simultaneously retrying can turn a brief spike into a sustained overload.
• Usage-plan quotas and rate/burst limits are, per AWS's own framing, "best-effort" targets, not hard guarantees; don't design a system that assumes a configured limit is a precise ceiling under all conditions.
• A capacity playbook (documented steps to raise usage-plan limits, request an account-level increase, and scale the backend) written before an incident saves real time during one; deciding these steps live during a spike is slower and more error-prone than following a rehearsed plan.

Direct answer
The Performance Efficiency pillar is fundamentally about matching resource allocation to actual demand and noticing when that match drifts, so the metrics that matter split into three groups: what the user experiences (latency, error rate), what the infrastructure is actually doing with the resources it has (CPU/memory utilization, cache hit ratio), and how fast the system adapts when demand changes (autoscaling reaction time, queue depth). Track percentiles, not averages, for anything user-facing, an average can look healthy while a meaningful fraction of real users have a bad experience.

Structured elaboration

Worked example
A service is provisioned for a 200 req/sec average. P50 latency looks fine at 80ms, but P99 is 1,800ms. Pulling CPU utilization alongside it shows an average of 35%, which rules out plain under-provisioning; pulling queue depth shows periodic spikes to 500+ queued requests during short bursts, correlating with the P99 spikes. That combination, low average utilization, high tail latency, spiky queue depth, points at a burst-handling efficiency problem (an undersized connection pool, or too-conservative autoscaling reaction time), not a steady-state capacity problem, which needs a different fix than just adding baseline capacity. (This is a diagnostic walkthrough of hypothetical, stated metric values, not a measured production incident.)

Trade-offs & pitfalls

• Tracking too many metrics without tying each one to a decision (scale, cache, re-architect) produces dashboard noise, not efficiency; every metric on this list should map to a specific action taken when it crosses a threshold.
• Alerting on averages instead of percentiles routinely misses real user pain; but alerting on P99 alone, with no P50 for context, can over-react to a small number of genuinely unusual requests, such as first-time cold-cache users, track both.
• Utilization metrics without a workload-shape baseline are ambiguous: 80% CPU is healthy for a batch job and a warning sign for a latency-sensitive API at peak, so calibrate the "efficient" threshold per workload, not as one number applied everywhere.

Direct answer

A strongly consistent read always returns the most recently committed value (served from the leader replica), an eventually consistent read may return slightly stale data (typically caught up within a second) but can be served from any replica. In provisioned capacity terms, a strongly consistent read of a 4 KB item costs 1 Read Capacity Unit (RCU); an eventually consistent read of the same item costs half that, 0.5 RCU, because DynamoDB can spread the work across replicas. For a low-latency, high-read leaderboard, use eventually consistent reads for the general (global) view and reserve strong consistency only for the one case that needs it: a user checking their own just-submitted score.

Structured elaboration

Consistency and cost

Leaderboard design

• Source of truth: a DynamoDB table keyed by (leaderboardId, userId) storing each player's current score, updated via an atomic UpdateItem with an ADD (or conditional set) expression so concurrent score updates don't lose writes.
• Fast read path: a cache in front of DynamoDB for the hot "top N" and "my rank" queries, since even a 0.5 RCU eventually consistent read doesn't beat an in-memory sorted structure for sub-millisecond top-N retrieval at high QPS. A managed in-memory cache holding a sorted-set-like structure per leaderboard (score to userId) serves top-N and rank lookups directly from memory.
• Write path: score update writes to DynamoDB first (source of truth), then updates the cache synchronously where feasible, or asynchronously via DynamoDB Streams if you want the write path decoupled from cache availability. Streams-based propagation also gives you a natural retry/replay mechanism if the cache falls behind.
• Consistency split: global "top 100" and "leaderboard around me" reads hit the cache (effectively eventually consistent, and fine for that use case). A player's own current score/rank, right after they submit it, can use a strongly consistent GetItem against DynamoDB so the UI never shows a stale "your score" the instant after they played.

Worked example

A mobile game leaderboard serving 5,000 reads/sec and a much smaller volume of score updates: the "top 100" view and "nearby ranks" view are read from the cache, so DynamoDB only sees the write volume (score updates) plus the low-volume strongly consistent "my current score" reads, not the full 5,000 reads/sec. This keeps provisioned RCU (or on-demand request cost) proportional to writes and to the smaller strongly consistent read slice, not to total leaderboard traffic. If the cache and DynamoDB briefly disagree (cache slightly behind after a burst of writes), that's acceptable for "top 100" but not for "my score," which is exactly why that one query path bypasses the cache and reads DynamoDB directly with strong consistency.

Trade-offs and pitfalls

• Don't default every read to strongly consistent "to be safe": it doubles RCU cost and caps throughput at what the leader replica can serve, which is the wrong trade-off for a leaderboard where 99% of reads are the shared, cacheable top-N view.
• A cache-first design needs an explicit staleness bound (a short time-to-live or streams-driven refresh) or players will see visibly wrong ranks after a burst of updates; decide and document how stale "eventually" is allowed to be.
• Rank computation itself doesn't come from RCU/WCU semantics; a sorted-set-style cache gives ordered rank cheaply, whereas computing rank from DynamoDB alone would need a full scan or a separate rank-tracking scheme, so the cache isn't just a latency optimization here, it's doing work DynamoDB isn't shaped to do efficiently.
• A common pitfall is forgetting to also protect the write path: high-frequency score updates on the same userId are fine (single partition per user is bounded), but a very "hot" leaderboard with an extreme write rate across many users still needs the table's partition key chosen so writes spread across partitions, not concentrated by a poorly chosen key.

Direct answer

Layer three kinds of control. Preventive: make accidental or unauthorized mass deletion structurally hard, using least-privilege AWS Identity and Access Management (IAM) plus a Service Control Policy (SCP) deny on destructive S3 calls, Versioning and Object Lock on critical buckets, and cross-account replication so no single compromised account can destroy both copies. Detective: near-real-time alerting on delete-heavy API activity via CloudTrail data events and EventBridge, not just periodic audits. Recovery: a tested runbook that restores from the least-effort, most-trustworthy source available, a separate-account replica first, versioned objects second, archived or Glacier restores last.

Structured elaboration

• Preventive
  • Least-privilege IAM plus an AWS Organizations SCP that denies s3:DeleteBucket and bulk s3:DeleteObject* for every principal except a small, separately audited break-glass role; an SCP bounds the maximum permission even a misconfigured IAM policy can grant.
  • Enable Versioning on every bucket that matters; it's a prerequisite for everything below it.
  • Object Lock (governance or compliance mode) on data that must survive even a compromised admin credential. Two things worth getting right: Object Lock can only be enabled on a bucket that already has Versioning turned on, and once Object Lock is enabled on a bucket, you can never disable it or suspend that bucket's Versioning again. It's a one-way architectural decision, not something to flip mid-incident.
  • Cross-account, ideally cross-region, replication to a separate, hardened account whose own SCPs deny delete entirely. This is the strongest recovery guarantee because it survives even total compromise of the source account.
• Detective
  • Turn on CloudTrail data events for S3 (object-level, not just management events) and route them to a centralized, separate-account log destination so the log itself can't be deleted by whatever compromised the source account.
  • An EventBridge rule matching bulk delete API calls triggers a Lambda that counts deletions per principal and bucket in a short window and pages on-call above a threshold, catching a mass delete in minutes rather than when someone notices missing data.
  • Daily S3 Inventory plus a scheduled query (Athena or similar) catches slower-burn deletion patterns that wouldn't trip a real-time threshold.
• Recovery runbook, in order
  1. Isolate: revoke or disable the credentials or role that caused the deletion immediately, and pause replication so a still-running bad process can't propagate deletes to the recovery copy.
  2. Scope: use CloudTrail data events plus S3 Inventory to enumerate exactly which keys and version IDs were affected.
  3. Restore, cheapest-safest source first: copy from the cross-account replica if one exists; otherwise remove delete markers or copy prior versions back using S3 Batch Operations driven from a manifest, not a one-by-one script, for anything Versioning protected; for archived data, initiate Glacier restores prioritized by what's needed soonest.
  4. Validate: reconcile object counts and checksums against S3 Inventory before declaring the incident resolved.
  5. Post-incident: rotate any credentials involved, close the IAM or SCP gap that allowed it, and schedule a recurring restore drill, quarterly is a reasonable cadence, so the runbook is proven before it's needed for real.

Worked example

A CI job's IAM role is accidentally granted broad S3 permissions, and a bad script run deletes 50,000 objects across a data-lake bucket in under a minute. Because Versioning was on, the deletes are only delete markers. The EventBridge rule watching bulk-delete volume per principal pages on-call within minutes of the burst starting, with the exact detection latency being a tunable alarm threshold, not a promised SLA. The team disables the CI role's credentials, confirms via CloudTrail that exactly one principal and one time window are involved, and runs an S3 Batch Operations job from a manifest of affected keys, pulled from ListObjectVersions, to remove the delete markers, restoring the objects without needing the cross-account replica at all. The replica and Object Lock exist for the rarer, worse case where deletes are permanent.

Trade-offs & pitfalls

• Object Lock in Compliance mode is the strongest guarantee available but is irreversible for the configured retention period, including against your own root account; it needs to be a deliberate decision aligned with actual legal or retention requirements, not a default flipped on everywhere "to be safe."
• Enabling Object Lock is itself a one-way door at the bucket level, easy to miss since most S3 settings are freely reversible. Treat it as a design decision made once per bucket, not an incident-response lever.
• SCP-based denies are powerful, but a deny written too broadly can also block legitimate lifecycle expiration and other benign delete operations; scope the deny to the specific bulk and delete-bucket actions and exempt the automation that's supposed to run lifecycle cleanup.
• A cross-account replica is the strongest recovery path only if that account's own permissions are actually locked down; a replica account with the same broad access as the source doesn't add real protection against a credential compromised with access to both.