AWS Core Services and Architecture Questions

Pick the family by matching it to the workload's dominant bottleneck (CPU, memory, storage I/O, or accelerator), not by habit. Every family shares the same underlying platform built on the Nitro System (AWS's custom hardware and hypervisor-offload system) but is engineered with a different vCPU:memory:network ratio, and within a family, bigger sizes get proportionally more network and EBS bandwidth, not just more vCPUs. The family letter plus a processor suffix (i = Intel, a = AMD, g = Graviton/ARM) tells you both the ratio profile and the CPU architecture you're compiling or shipping containers for.

The core families

Generation numbers are a moving target: AWS ships a new generation within each family roughly every 1-2 years, and by mid-2026 the general-purpose line had already moved from m7-series through m8-series to a Graviton5-based m9g. Treat the digit as something to look up at decision time against AWS's current instance-type listing, not something to hardcode from memory.

Burstable (T-series) economics

T-series instances (t3, t4g) are a variant of general purpose worth calling out separately: each size gets a "baseline" CPU percentage per vCPU and earns CPU credits while running below that baseline. Spending above baseline burns credits; if the balance hits zero, performance is throttled back to baseline unless the instance is in "unlimited" mode, which lets it burst past its credit balance at an additional per-vCPU-hour charge. That makes T-series excellent for spiky-but-mostly-idle workloads (dev boxes, low-traffic services) and a bad fit for anything with sustained CPU demand, where the credit model just becomes a slower way to pay compute-optimized prices.

Graviton (ARM) angle

Most families now ship a Graviton (ARM64) variant of the same ratio profile. AWS markets Graviton as offering meaningfully better price-performance than the equivalent x86 size for compatible workloads, so it's usually worth trying first for anything you control the build for. The catch is compatibility, not performance: you need ARM64-native binaries or multi-arch container images, and some closed-source agents, drivers, or legacy dependencies still don't ship ARM builds, which is the main reason teams stay on Intel/AMD for a given fleet.

Applying it to the three scenarios

• CPU-bound batch job: compute-optimized (c-family). The workload is paying for vCPUs, not RAM, so a general-purpose instance of the same vCPU count wastes money on memory you don't use.
• Memory-heavy cache: memory-optimized (r-family, or x-family for extreme RAM-per-vCPU). You're sizing for GiB of hot data, and memory-optimized gets you that RAM at a lower $/GiB than scaling a general-purpose instance up to the same memory footprint.
• GPU training workload: accelerated computing (p or g family, or Trainium for training specifically on AWS's own silicon). Sizing here is driven by GPU count/VRAM and the framework's requirements, not by vCPU count at all.

Worked example

Say a nightly batch job must CPU-encode 10,000 independent video segments, each needing about 20 vCPU-seconds of work.

Running that entirely in parallel on a single compute-optimized instance with 16 vCPUs (a c6i.4xlarge-class size):

A general-purpose instance with the same 16 vCPUs finishes in roughly the same wall-clock time, since the job is purely CPU-bound and doesn't touch the extra memory, but it costs more per finished job because you're paying for RAM the workload never uses. That's the concrete case for "match the family to the bottleneck" rather than defaulting to general purpose.

Trade-offs and pitfalls

• Defaulting to general-purpose because it's the "safe" choice hides real savings when a workload has one obvious bottleneck.
• Storage-optimized local NVMe is ephemeral: it's wiped on stop/terminate, so it's a performance tier, not a durability tier. Pair it with replication or S3-backed persistence for anything that must survive an instance loss.
• T-series credit exhaustion under sustained load either throttles performance or silently bills unlimited-mode overage. Both are worth catching before they surprise you in an incident or a bill.
• Accelerated-computing capacity can be scarce in a given Availability Zone; latency-sensitive GPU workloads sometimes need On-Demand Capacity Reservations to guarantee availability.
• Assuming Graviton is a free performance win: recompiling, rebuilding multi-arch images, and re-validating third-party agents is real migration work, not a toggle.

Choose based on where you want operational ownership to sit and how the workload's traffic and duration shape matches each model's scaling and cold-start profile: Lambda for event-driven work with the lowest operational overhead, containers (ECS/EKS, short for Elastic Container Service / Elastic Kubernetes Service) for portable services that need runtime control with moderate operational overhead, and EC2 when you need full control, specialized hardware, or have steady high utilization that favors owning the box.

Comparison

A detail that matters inside "containers": ECS and EKS can run on the Fargate launch type (AWS manages the underlying compute, closer to Lambda's operational profile) or the EC2 launch type (you manage a fleet of worker nodes yourself, closer to EC2's operational profile). "Containers" isn't one operational overhead level, it's a spectrum depending on that choice.

It's usually a portfolio, not a single pick

A real platform rarely runs on just one of these. A common shape is a stateless web API on ECS/Fargate for steady, latency-sensitive traffic; event-driven glue and scheduled/cron-like batch jobs on Lambda, where the workload is naturally bursty and short; and EC2 reserved for anything needing specialized hardware or licensing. Treating the choice as "pick one for the whole platform" usually means overpaying somewhere.

Worked example

For a stateless API handling an average of 50 requests/second, where each request takes about 100 ms of server-side compute, Little's Law gives the expected number of requests in flight at any instant:

For Lambda, 5 concurrent executions is trivial against the default account concurrency limit, and cost is purely pay-per-invocation. For ECS/Fargate, that same average load might run comfortably on 2-3 small tasks behind an ALB, sized with headroom above the 5-request average to absorb bursts. For EC2, an Auto Scaling Group would be sized similarly, but the team additionally owns AMI (Amazon Machine Image) patching, OS-level scaling policy tuning, and instance health management that the other two options abstract away.

Trade-offs and pitfalls

• Lambda's per-invocation pricing is favorable for bursty or low-average-utilization traffic, but a steady, high-volume workload can end up costing more on Lambda than on right-sized, reserved EC2/container capacity: "serverless is always cheaper" is a common false assumption.
• Even Fargate-backed ECS/EKS carries more deploy-unit and networking surface area (task definitions, service discovery, target groups) than Lambda's simpler single-function deployment model.
• EC2 gives full control but that control is also the operational burden: nothing patches, scales, or fails over unless you build it.
• Conflating "containers" as one operational tier hides the real Fargate-vs-EC2-launch-type decision, which changes the operational overhead comparison significantly.

Direct answer

Put RDS Proxy between Lambda and the database rather than letting each Lambda execution environment open its own connection: Proxy pools and multiplexes a large number of client-facing connections onto a much smaller, stable set of physical database connections, absorbs the connection-storm problem that comes from Lambda's concurrency model, and manages AWS Identity and Access Management (IAM)-auth token rotation for you. Warm-invocation connection reuse, caching a client handle in the Lambda execution environment's global scope, still matters and is complementary, not a substitute: it avoids re-establishing the Proxy-facing connection on every invocation of an already-warm container, while Proxy is what keeps the database itself from seeing thousands of physical connections when Lambda scales out concurrently.

Structured elaboration

The problem RDS Proxy solves

A relational database has a hard ceiling on max_connections, in the low thousands at most, set by instance memory. Lambda can scale to hundreds or thousands of concurrent execution environments in seconds, and if each one opens its own direct connection to the database, you exhaust max_connections almost immediately under a burst, regardless of how efficient the query itself is. RDS Proxy sits in front of the database and:

• Pools and multiplexes: many Lambda-side logical connections share a much smaller pool of physical connections held open to the database, since most connections spend most of their time idle between queries.
• Handles auth centrally: works with IAM database authentication so Lambda does not need database credentials embedded in its code or environment; Proxy manages short-lived credential exchange with the database on the app's behalf.
• Smooths failover: for Aurora, Proxy keeps client-facing connections open across a failover event instead of every client immediately seeing a connection error, reconnecting transparently on the Proxy side.

Configuration knobs that matter

• MaxConnectionsPercent: the percentage of the target database's max_connections this Proxy, or a specific target group, is allowed to use, leaving headroom for other consumers of the same database.
• MaxIdleConnectionsPercent: how many of those connections Proxy is willing to keep idle in the pool versus closing back down.
• ConnectionBorrowTimeout: how long a client waits for the Proxy to hand it a connection before failing; this is your effective backpressure signal when the pool is genuinely saturated.
• Session pinning: certain session-level operations, temp tables, session variables, some transaction patterns, force Proxy to pin a client to one specific physical connection for the rest of that session, which defeats multiplexing for that client. Know which patterns in your queries trigger pinning and avoid them where you can, since they reduce Proxy's actual pooling benefit.

Where warm-invocation reuse still helps

Within a single warm Lambda execution environment, cache the database client as a module-level variable rather than re-creating it on every invocation. This avoids repeating the TLS handshake and Proxy-side connection setup for every invocation of an already-warm container; it does not replace Proxy, because a cold start or a newly spun-up concurrent execution environment still needs a fresh connection, and that is exactly the storm Proxy exists to absorb.

What to do when the pool is genuinely exhausted

• Return a clear rejection with backoff guidance rather than letting the request hang until Lambda's own timeout.
• Buffer non-latency-sensitive writes through a queue and drain them into the database at a controlled rate instead of writing synchronously from every Lambda invocation.
• Keep queries short and transactions small; a long-held transaction ties up a pooled connection and reduces how many other invocations Proxy can serve from the same physical pool.

Worked example

A database sized for a max_connections value of 1,000 fronts a Lambda function that can burst to 2,000 concurrent invocations. Configure the Proxy's target group with MaxConnectionsPercent at 80, leaving 20% headroom for other clients such as an admin console or a batch job:

$1000 \times 0.80 = 800\ \text{connections available to this Proxy}$

With 2,000 concurrent Lambda invocations each needing a connection only for the brief duration of a query, Proxy multiplexes them across those 800 physical connections rather than needing 2,000 physical connections; invocations that arrive while all 800 are briefly in use wait up to the borrow timeout for one to free up, rather than the database itself ever seeing more than 800 connections.

Trade-offs and pitfalls

• RDS Proxy adds a small amount of latency per query, an extra network hop, and its own hourly cost; for very low-concurrency workloads, plain connection reuse in a warm Lambda might be enough and Proxy is unnecessary overhead.
• Session pinning is the most common way teams get less benefit from Proxy than they expected; if your ORM or query patterns rely heavily on session state, audit which patterns trigger pinning.
• Do not rely on Proxy alone to make an unbounded-concurrency Lambda safe for the database; borrow-timeout failures under sustained overload are still failures, just failures the database itself did not see. You still need to address the actual concurrency-to-capacity mismatch, reserved concurrency limits, queue-based buffering, or a bigger database.
• Provisioned Concurrency reduces cold starts, and therefore how often a fresh connection has to be established, but costs money for capacity held ready; it is a traffic-shaping choice independent of whether you are using Proxy, do not treat it as a pooling mechanism by itself.

Direct answer
When traffic exceeds the account's or function's AWS Lambda concurrency ceiling, Lambda throttles (rejects) the excess invocations, and for a synchronous Amazon API Gateway-to-Lambda integration, API Gateway treats that rejected invocation as an internal error and returns a generic HTTP 500 to the caller, not a 429. The 429 status is a separate mechanism entirely: API Gateway's own account/stage-level rate-and-burst throttle, which can trip even when Lambda still has spare concurrency, and vice versa. Left alone, either failure mode just pushes the overload problem onto the client. The actual protection strategy is to smooth the burst with a durable buffer before it ever reaches Lambda, reserve capacity so a critical path can't be starved by a noisy one, and put an explicit circuit breaker in front of the database so a Lambda-side retry storm can't do to the database what the traffic spike just did to Lambda.

Structured elaboration

1. Absorb the spike before it becomes a database problem

• Insert Amazon Simple Queue Service (SQS) between API Gateway and Lambda for anything that doesn't need a synchronous response (an API Gateway-to-SQS direct integration, or an asynchronous Lambda invocation with an SQS-backed event source). SQS durably buffers the burst; the Lambda event-source mapping's batch size and its own concurrency setting decouple ingestion rate from processing rate, so downstream systems only ever see the rate you choose to drain the queue at.
• For the strictly synchronous path, where the client needs an immediate response, buffering isn't available; the mitigation there is capacity planning (below) plus a fast, honest 429 rather than a request that hangs until it fails anyway.

2. Reserve concurrency instead of sharing one undifferentiated pool

• Set reserved concurrency on the critical function(s) so a burst on a lower-priority function (an analytics webhook, say) can never consume all of the account's concurrency and starve checkout.
• Add Provisioned Concurrency (with Application Auto Scaling tracking a utilization metric) for the critical path if cold starts under sudden load are also a problem, since reserved concurrency alone doesn't pre-warm execution environments.

3. Protect the database explicitly, don't assume Lambda's own throttling is enough

• Put Amazon RDS (Relational Database Service) Proxy in front of a relational database so a burst of concurrent Lambda executions doesn't each open a new connection and exhaust the database's connection limit; RDS Proxy pools and multiplexes connections across Lambda's ephemeral execution environments.
• Add an explicit circuit breaker in the handler: track a recent database error/latency rate (a small shared counter in DynamoDB or ElastiCache) and short-circuit to a cached or degraded response once the error rate crosses a threshold, instead of letting every concurrent invocation retry against an already-struggling database.

4. Make the failure mode graceful for the client, using the code that's actually returned

• A Lambda concurrency throttle behind this synchronous integration doesn't hand the client a distinct "throttled" signal: per AWS's own Lambda developer guide, API Gateway treats a rejected Lambda invocation as an internal error and returns a generic HTTP 500 ("API Gateway treats all invocation and function errors as internal errors. If the Lambda API rejects the invocation request, API Gateway returns a 500 error code."). That's separate from API Gateway's own account/stage-level rate-and-burst throttle, a token-bucket limit enforced before the request ever reaches Lambda, which does return 429 Too Many Requests.
• Because a bare 500 doesn't tell the client why the call failed, don't build client retry logic that assumes every 500 is safely retryable. Instead, correlate a 500 spike with the Lambda Throttles and ConcurrentExecutions CloudWatch metrics to confirm it's a concurrency throttle, and apply exponential backoff with jitter against that signal rather than waiting for a 429 a concurrency throttle will never send.
• When it's genuinely API Gateway's own rate/burst limit tripping, the client contract is simpler: back off and resubmit on 429, since API Gateway is explicit about the cause.
• Configure a Dead Letter Queue (DLQ) or Lambda's on-failure destination for the asynchronous path so throttled or failed events aren't silently dropped and can be replayed once capacity recovers.

Worked example
Suppose the account's default unreserved Lambda concurrency budget is around 1,000 (the standard default; raise it via a quota-increase request if genuinely needed), and a spike pushes demand to 4,000 concurrent invocations. Without protection, roughly 3,000 of those invocations throttle; because this is a synchronous integration, API Gateway returns each of those to the caller as a generic 500 (not a 429, that code belongs to API Gateway's own separate rate/burst throttle), and whichever ones do get through hammer the database with up to that many concurrent connection attempts. With the design above: reserved concurrency guarantees checkout 400 of that 1,000 budget regardless of what else is spiking; SQS in front of the non-synchronous paths absorbs the remaining 3,600 invocations' worth of work and drains it at a controlled rate; RDS Proxy caps the effective connection count to the database well below what 400 raw Lambda executions would otherwise open concurrently. (This is a scenario walkthrough using stated, illustrative numbers to reason through the design, not a measured result.)

Trade-offs & pitfalls

• SQS buffering trades latency for durability: anything routed through a queue is no longer synchronous, so it only works for requests where the client doesn't need an immediate response.
• Reserved concurrency for one function subtracts from the account-wide pool available to everything else; over-reserving can itself cause other functions to throttle prematurely, so size it from real traffic data, not a round number.
• RDS Proxy protects the database's connection count, not its CPU or IOPS capacity under load; pair it with the circuit breaker rather than treating it as a complete fix.
• A circuit breaker that trips too aggressively degrades the user experience during a load spike the database could actually have handled; tune the threshold against real load-test data, not a guess.

flowchart LR
    C[Client] --> AG[API Gateway]
    AG -->|sync, reserved concurrency| L1[Critical Lambda]
    AG -->|async / SQS integration| Q[(SQS buffer)]
    Q --> L2[Worker Lambda, controlled batch size]
    L1 --> RP[RDS Proxy]
    L2 --> RP
    RP --> DB[(Database)]
    L1 -.circuit-breaker check.-> CB[(Shared error-rate counter)]

Direct answer

Start by correlating timing: pull API Gateway's CloudWatch metrics (ThrottleCount, 4XXError, Count, IntegrationLatency) for the affected stage/method alongside the backend's own error and saturation metrics for the same window. If ThrottleCount rises while the backend's incoming request volume stays flat or drops, API Gateway is throttling before requests even reach the backend. If API Gateway's integration request count keeps climbing but the backend itself is returning its own 429s (or its logs show connection/thread-pool rejections), the backend is the bottleneck and API Gateway is just passing the failure through.

Structured elaboration

Where API Gateway throttling can be applied (and in what precedence order)

API Gateway evaluates throttling in this order, and the tightest applicable limit wins:

1. Per-client (usage plan / API key) throttling for that stage
2. Per-method throttling set on the stage
3. Account-level, per-Region throttling (a default ceiling across all your APIs in that Region)
4. AWS's own Regional throttling ceiling (fixed, not customer-configurable)

API Gateway enforces its limits with a token-bucket algorithm: the rate is how fast tokens refill (steady-state requests/sec), the burst is the bucket's capacity (how many requests can go through instantly before the steady rate takes over). A spike that's short but sharp can exhaust the burst allowance and start getting 429s even if the sustained rate is well within the configured limit.

Diagnosis checklist

• Check whether the traffic is using API keys tied to a usage plan; if so, check that plan's rate/burst settings, since a per-key limit can throttle one client's traffic pattern while the API as a whole has headroom.
• Check stage-level and method-level throttling overrides, since a specific high-traffic method (e.g., a search endpoint) may have a tighter override than the API's general limit.
• Trace a sample of 429 requests end-to-end (distributed tracing, if enabled) to see definitively whether the response originated at API Gateway or was proxied from the backend.
• If a Web Application Firewall (WAF) or Application Load Balancer (ALB) sits in front of API Gateway, check its rate-based rules too; a 429-shaped response can originate there instead of at API Gateway itself.

Worked example

Suppose CloudWatch shows ThrottleCount spiking to several hundred per minute during the traffic spike, while IntegrationLatency and the backend's own request-count metric stay essentially flat during the same window. That pattern says the requests never reached the backend: API Gateway rejected them at the edge because the spike's burst exceeded the configured token-bucket capacity, most likely the account-level or usage-plan burst limit rather than a per-method override (if this were a per-method override, only that one method's traffic would show elevated ThrottleCount while others stayed clean). The fix in this case is at the API Gateway layer: request an account-level limit increase if the API's aggregate legitimate traffic genuinely needs a higher regional ceiling, and/or raise the usage plan's burst allowance for the affected client so short, legitimate spikes don't get rejected before reaching the backend.

If instead ThrottleCount stayed low but the backend's own error logs show its connection pool or thread pool rejecting work and returning its own 429/503, the fix is downstream: autoscale the backend on a leading indicator (queue depth, concurrency, or request latency, not just CPU), and consider adding a buffering layer (a queue) in front of the backend so a burst is absorbed and drained at a steady rate instead of hitting the backend's fixed capacity directly.

Trade-offs and pitfalls

• Raising API Gateway's limits without also confirming the backend can absorb the resulting traffic just moves the 429s downstream, where they show up as backend errors instead, which is a worse failure mode to debug.
• Clients need to implement backoff (respecting the Retry-After behavior implied by 429s, with jitter) regardless of which layer is throttling; without it, a retry storm from many clients simultaneously retrying can turn a brief spike into a sustained overload.
• Usage-plan quotas and rate/burst limits are, per AWS's own framing, "best-effort" targets, not hard guarantees; don't design a system that assumes a configured limit is a precise ceiling under all conditions.
• A capacity playbook (documented steps to raise usage-plan limits, request an account-level increase, and scale the backend) written before an incident saves real time during one; deciding these steps live during a spike is slower and more error-prone than following a rehearsed plan.