AWS Core Services and Architecture Questions

Direct answer

Start by correlating timing: pull API Gateway's CloudWatch metrics (ThrottleCount, 4XXError, Count, IntegrationLatency) for the affected stage/method alongside the backend's own error and saturation metrics for the same window. If ThrottleCount rises while the backend's incoming request volume stays flat or drops, API Gateway is throttling before requests even reach the backend. If API Gateway's integration request count keeps climbing but the backend itself is returning its own 429s (or its logs show connection/thread-pool rejections), the backend is the bottleneck and API Gateway is just passing the failure through.

Structured elaboration

Where API Gateway throttling can be applied (and in what precedence order)

API Gateway evaluates throttling in this order, and the tightest applicable limit wins:

1. Per-client (usage plan / API key) throttling for that stage
2. Per-method throttling set on the stage
3. Account-level, per-Region throttling (a default ceiling across all your APIs in that Region)
4. AWS's own Regional throttling ceiling (fixed, not customer-configurable)

API Gateway enforces its limits with a token-bucket algorithm: the rate is how fast tokens refill (steady-state requests/sec), the burst is the bucket's capacity (how many requests can go through instantly before the steady rate takes over). A spike that's short but sharp can exhaust the burst allowance and start getting 429s even if the sustained rate is well within the configured limit.

Diagnosis checklist

• Check whether the traffic is using API keys tied to a usage plan; if so, check that plan's rate/burst settings, since a per-key limit can throttle one client's traffic pattern while the API as a whole has headroom.
• Check stage-level and method-level throttling overrides, since a specific high-traffic method (e.g., a search endpoint) may have a tighter override than the API's general limit.
• Trace a sample of 429 requests end-to-end (distributed tracing, if enabled) to see definitively whether the response originated at API Gateway or was proxied from the backend.
• If a Web Application Firewall (WAF) or Application Load Balancer (ALB) sits in front of API Gateway, check its rate-based rules too; a 429-shaped response can originate there instead of at API Gateway itself.

Worked example

Suppose CloudWatch shows ThrottleCount spiking to several hundred per minute during the traffic spike, while IntegrationLatency and the backend's own request-count metric stay essentially flat during the same window. That pattern says the requests never reached the backend: API Gateway rejected them at the edge because the spike's burst exceeded the configured token-bucket capacity, most likely the account-level or usage-plan burst limit rather than a per-method override (if this were a per-method override, only that one method's traffic would show elevated ThrottleCount while others stayed clean). The fix in this case is at the API Gateway layer: request an account-level limit increase if the API's aggregate legitimate traffic genuinely needs a higher regional ceiling, and/or raise the usage plan's burst allowance for the affected client so short, legitimate spikes don't get rejected before reaching the backend.

If instead ThrottleCount stayed low but the backend's own error logs show its connection pool or thread pool rejecting work and returning its own 429/503, the fix is downstream: autoscale the backend on a leading indicator (queue depth, concurrency, or request latency, not just CPU), and consider adding a buffering layer (a queue) in front of the backend so a burst is absorbed and drained at a steady rate instead of hitting the backend's fixed capacity directly.

Trade-offs and pitfalls

• Raising API Gateway's limits without also confirming the backend can absorb the resulting traffic just moves the 429s downstream, where they show up as backend errors instead, which is a worse failure mode to debug.
• Clients need to implement backoff (respecting the Retry-After behavior implied by 429s, with jitter) regardless of which layer is throttling; without it, a retry storm from many clients simultaneously retrying can turn a brief spike into a sustained overload.
• Usage-plan quotas and rate/burst limits are, per AWS's own framing, "best-effort" targets, not hard guarantees; don't design a system that assumes a configured limit is a precise ceiling under all conditions.
• A capacity playbook (documented steps to raise usage-plan limits, request an account-level increase, and scale the backend) written before an incident saves real time during one; deciding these steps live during a spike is slower and more error-prone than following a rehearsed plan.

Choose based on where you want operational ownership to sit and how the workload's traffic and duration shape matches each model's scaling and cold-start profile: Lambda for event-driven work with the lowest operational overhead, containers (ECS/EKS, short for Elastic Container Service / Elastic Kubernetes Service) for portable services that need runtime control with moderate operational overhead, and EC2 when you need full control, specialized hardware, or have steady high utilization that favors owning the box.

Comparison

A detail that matters inside "containers": ECS and EKS can run on the Fargate launch type (AWS manages the underlying compute, closer to Lambda's operational profile) or the EC2 launch type (you manage a fleet of worker nodes yourself, closer to EC2's operational profile). "Containers" isn't one operational overhead level, it's a spectrum depending on that choice.

It's usually a portfolio, not a single pick

A real platform rarely runs on just one of these. A common shape is a stateless web API on ECS/Fargate for steady, latency-sensitive traffic; event-driven glue and scheduled/cron-like batch jobs on Lambda, where the workload is naturally bursty and short; and EC2 reserved for anything needing specialized hardware or licensing. Treating the choice as "pick one for the whole platform" usually means overpaying somewhere.

Worked example

For a stateless API handling an average of 50 requests/second, where each request takes about 100 ms of server-side compute, Little's Law gives the expected number of requests in flight at any instant:

For Lambda, 5 concurrent executions is trivial against the default account concurrency limit, and cost is purely pay-per-invocation. For ECS/Fargate, that same average load might run comfortably on 2-3 small tasks behind an ALB, sized with headroom above the 5-request average to absorb bursts. For EC2, an Auto Scaling Group would be sized similarly, but the team additionally owns AMI (Amazon Machine Image) patching, OS-level scaling policy tuning, and instance health management that the other two options abstract away.

Trade-offs and pitfalls

• Lambda's per-invocation pricing is favorable for bursty or low-average-utilization traffic, but a steady, high-volume workload can end up costing more on Lambda than on right-sized, reserved EC2/container capacity: "serverless is always cheaper" is a common false assumption.
• Even Fargate-backed ECS/EKS carries more deploy-unit and networking surface area (task definitions, service discovery, target groups) than Lambda's simpler single-function deployment model.
• EC2 gives full control but that control is also the operational burden: nothing patches, scales, or fails over unless you build it.
• Conflating "containers" as one operational tier hides the real Fargate-vs-EC2-launch-type decision, which changes the operational overhead comparison significantly.

Direct answer

Wire an EventBridge rule to match GuardDuty findings by type and severity, routing matched events to a Lambda "triage" function and to Security Hub for tracking. The triage Lambda enriches the finding (instance tags, attached AWS Identity and Access Management (IAM) role, Virtual Private Cloud (VPC) context, your own isolated network in AWS) and invokes a Systems Manager (SSM) Automation document to contain the instance, typically by attaching a quarantine security group that blocks egress while preserving SSM connectivity, rather than terminating it outright, so evidence isn't destroyed. Every role in that chain (EventBridge target, the triage Lambda, the SSM Automation execution role) gets the minimum permissions to do exactly its one step, nothing broader, since the automation itself is now a privileged actor that needs to be as trustworthy as any human responder.

Structured elaboration

flowchart LR
    GD["GuardDuty finding
(suspicious outbound traffic)"] --> EB["EventBridge rule
matches finding type/severity"]
    EB --> TR["Triage Lambda
enrich: tags, IAM role, VPC, recent CloudTrail"]
    TR --> SH["Security Hub
finding updated, workflow status set"]
    TR --> SSM["SSM Automation document
runs on the instance via SSM Agent"]
    SSM --> QSG["Attach quarantine security group
(block egress, allow SSM only)"]
    SSM --> SNAP["EBS snapshot +
CloudTrail/VPC Flow Log export to S3"]
    QSG --> NOTIFY["SNS to on-call / ticket"]
    SNAP --> NOTIFY
    NOTIFY --> APPROVE{"Analyst
approval"}
    APPROVE -->|approve remediation| CLEAN["Rotate credentials,
rebuild instance, restore SG"]
    APPROVE -->|false positive| REVERT["Remove quarantine SG,
close finding"]

Detection and routing: the EventBridge rule's event pattern filters on the GuardDuty finding's type and severity fields so only findings above your response threshold trigger automation; lower-severity findings can route to Security Hub for visibility without triggering containment.

Containment: attaching a quarantine security group (deny all egress except to SSM endpoints) is generally preferred over immediately terminating the instance, because termination destroys the evidence you need for the next step. The same pattern applies whether the finding is outbound network activity or, in a related variant, an EC2 instance showing suspicious PutObject patterns against S3 (a data-exfiltration signature): the response is still containment-first, not termination-first.

Forensic data collection: capture an Elastic Block Store (EBS) snapshot of the instance's volumes before any remediation, and export the artifacts that let an analyst reconstruct what happened: CloudTrail (API-level activity, including who/what made the suspicious calls), S3 access logs (for the exfiltration-to-S3 variant, showing which objects were read or written and from where), and VPC Flow Logs (network-level record of the suspicious connections GuardDuty flagged). Store all of it in a locked-down forensic S3 bucket with versioning and access logging of its own.

Credential handling: if the finding suggests the instance's IAM role or an associated credential may be compromised (either variant: outbound C2-style traffic or S3 exfiltration), revoke or rotate the exposed credentials as an explicit containment step, not just a follow-up cleanup task; a still-valid credential is an ongoing exposure even after the instance itself is quarantined.

Cross-account scoping: in a multi-account setup, the automation should be able to act on the affected account's resources without requiring standing, account-wide privileges. Use a dedicated automation/security-tooling account whose Lambda and SSM roles assume narrowly-scoped, temporary cross-account roles into the affected account, limited to the specific containment actions (attach SG, run SSM document, create snapshot) and denied everything else, including account-wide administrative actions.

Least-privilege for the automation itself: separate the Lambda execution role (read GuardDuty/EC2/SSM metadata, start SSM Automation executions, write to the forensic S3 prefix, update Security Hub) from the SSM Automation role (act only on the specific, tag- or ARN-scoped target instance, where an ARN, Amazon Resource Name, is AWS's unique identifier for a specific resource). Neither role should hold broad iam:*, ec2:*, or s3:* permissions; scope by resource ARN and, where possible, by condition keys tied to the finding's specific instance.

Reporting and prevention (mapped to Well-Architected pillars): close the loop with a written incident report and feed findings back into architecture, framed against four Well-Architected concerns:

• Detection: was the GuardDuty finding type/severity threshold right, or did this near-miss a lower bucket that wouldn't have triggered automation?
• Least privilege: did the compromised instance's IAM role have more access than the workload needed, and should its policy be tightened?
• Automation: did the playbook run cleanly end-to-end, or were there manual steps that should be automated for the next incident?
• Governance: are the account boundaries, tagging, and approval gates around this automation documented and enforced, so this response pattern is repeatable and auditable, not tribal knowledge?

Worked example

GuardDuty fires Backdoor:EC2/C&CActivity.B (suspicious outbound to a known command-and-control IP) for instance i-0123456789. EventBridge routes it to the triage Lambda, which tags the finding as IN_PROGRESS in Security Hub and invokes the SSM Automation document. The document attaches the quarantine security group, takes an EBS snapshot, exports the instance's last hour of CloudTrail and VPC Flow Log entries to the forensic bucket, and revokes the instance profile's temporary credentials by detaching and reissuing under a new role. An analyst is paged via SNS, reviews the captured evidence, confirms it's a genuine compromise (not a false positive from a legitimate but newly-added external integration), and approves the remediation branch: the instance is rebuilt from a known-good AMI rather than "cleaned in place," and the incident report goes into the next architecture review under the four pillars above.

Trade-offs and pitfalls

• Fully automated, unapproved termination is tempting for speed but destroys evidence and risks taking down a false positive in production; a human-approval gate before destructive remediation (not before containment) is the right balance for most environments.
• If the automation's own IAM role is over-privileged, an attacker who compromises the automation path (rather than the original instance) inherits broad account access, which is a worse outcome than the original finding; this is why the automation's roles get the same least-privilege scrutiny as the workload they're protecting.
• Snapshotting and log export take time and add cost; prioritize capturing the volatile, hard-to-recover evidence (memory-adjacent artifacts, recent flow logs) before slower steps, and don't let evidence collection delay containment (attaching the quarantine SG) which is the step that actually stops ongoing damage.
• Test the playbook against synthetic findings in a non-production account; a badly-scoped SSM document run for the first time against a real incident is a poor place to discover it also quarantines the bastion host used to investigate it.

A cold start is the latency added when Lambda has no warm execution environment available and must create one from scratch before running the handler: pull and unpack the deployment package, start the language runtime, run any extensions, and execute module-level/global initialization code. A warm invocation skips all of that and only runs the handler. The main mitigations are Provisioned Concurrency (or SnapStart, where supported) to remove the cold path entirely, and shrinking what has to happen during init.

What drives cold-start latency

• Package size: a larger deployment artifact takes longer to fetch and unpack before the runtime can even start.
• Runtime choice: interpreted/lighter runtimes (Node.js, Python, Go via provided.al2023) generally start faster than managed runtimes with heavier startup (Java's JVM warm-up and classloading, .NET's CLR init), unless a snapshot-based mechanism is used to skip that cost.
• VPC networking: Lambda's shared Hyperplane ENI (Elastic Network Interface) model removed most of the old per-invocation ENI-creation penalty, but VPC-attached functions can still add first-attach latency and are worth measuring rather than assuming are cheap.
• Memory allocation: CPU scales with configured memory, so a low-memory function can have a slower init in addition to slower execution.
• Init code: expensive global-scope work (constructing SDK clients, opening DB connections, loading a large in-memory model) all runs during the cold path.

Mitigation strategies

1. Provisioned Concurrency: pre-initializes a set number of execution environments so invocations land warm. It eliminates cold starts for that reserved headroom but is billed as standing capacity, so it needs to be paired with Application Auto Scaling (scheduled or target-tracking) to track the real traffic shape, or it either overpays for idle capacity or still cold-starts above the provisioned count.
2. SnapStart: snapshots an already-initialized execution environment and restores from that snapshot on subsequent cold starts, avoiding repeated init work. It launched for Java and AWS has since extended it to additional managed runtimes; check current per-runtime support before depending on it for a given language, and be aware that anything generated at init time that must be unique per environment (connection handles, random seeds, unique IDs) needs to be explicitly re-initialized after a snapshot restore, not just reused.
3. Shrink package and init cost: trim dependencies, avoid bundling unused code, and move rarely-used imports out of the module's top level so they're only loaded when actually needed.
4. Right-size memory: because CPU is tied to memory, raising memory can shorten both init and execution time, sometimes lowering total cost despite the higher per-ms rate. This has to be measured per function, not assumed.
5. Minimize unnecessary VPC attachment: only attach a function to a VPC when it needs to reach VPC-only resources (e.g., RDS), and use RDS Proxy or VPC endpoints so the shared ENI cost is amortized rather than paid fresh per function.
6. Function decomposition, with a caveat: splitting one large multi-purpose function into smaller single-purpose functions reduces each function's package and init size, but it also multiplies the number of cold-start-prone entry points and adds inter-function invocation latency and orchestration complexity. It's a genuine trade-off, not a free win.

Measuring cold-start impact before committing to a fix

Lambda reports an Init Duration in the REPORT log line (and in traces from X-Ray, AWS's request-tracing tool), which is the only reliable way to confirm a latency spike is actually a cold start rather than something else in the request path. To quantify a candidate mitigation, run a controlled comparison: deploy two versions of the same function (for example, Provisioned Concurrency on vs. off, or Node.js vs. Java for the same logic), drive synthetic traffic that's guaranteed to exceed the current warm pool so cold environments are forced on each run, and compare P50/P99 Init Duration and total duration between the versions. That turns "we think this will help" into a measured decision for the specific workload, instead of applying folklore.

Trade-offs and pitfalls

• Provisioned Concurrency without autoscaling either overpays for idle capacity or fails to cover unscheduled bursts; it needs to track the actual traffic shape.
• SnapStart's runtime coverage and its "regenerate anything unique post-restore" requirement are easy to get wrong the first time; verify both before relying on it.
• Decomposing a function purely to shrink cold starts trades fewer, bigger cold starts for more, smaller ones plus added orchestration.
• Optimizing cold starts on a function that's mostly warm under real traffic is wasted effort; measure the actual cold-start rate for the workload before spending engineering time on it.

Direct answer

The two usual suspects are Availability Zone (AZ) crossing traffic between services that happen to land in different AZs, and Network Address Translation (NAT) Gateway data-processing charges when private-subnet resources reach the internet or other AWS services through a NAT Gateway rather than a direct path. Both are metered per gigabyte and both are easy to accumulate without noticing, because nothing about the traffic looks wrong, it's just routed the expensive way.

What's causing it

• Cross-AZ chattiness: a load balancer, service mesh, or just an Auto Scaling Group (ASG) spreading instances evenly across AZs means calls between two tiers of a service frequently cross AZ boundaries. Each direction of that hop is billed as inter-AZ data transfer, even though the two AZs are in the same region.
• NAT Gateway egress: resources in private subnets that reach S3, DynamoDB, or the public internet through a NAT Gateway pay a per-GB data processing charge on top of the underlying transfer. If the NAT Gateway itself sits in one AZ and the calling resources are spread across several, that traffic pays both the NAT processing charge and a cross-AZ hop to reach it.

Two straightforward fixes

1. Add Virtual Private Cloud (VPC) Gateway Endpoints for S3 and DynamoDB. These are free (no hourly or per-GB endpoint charge) and keep that traffic on the AWS backbone entirely, bypassing the NAT Gateway path altogether for two of the most common egress destinations. This is usually the highest-leverage, lowest-effort fix.
2. Give every AZ its own NAT Gateway instead of routing all private subnets through one shared NAT Gateway in a single AZ. This removes the extra cross-AZ hop to reach the NAT device; each AZ's traffic exits locally. It costs more in NAT Gateway hourly charges but usually still nets out cheaper than the cross-AZ transfer it eliminates, and it also removes an AZ-level single point of failure.

Worked example

A service handling 50 GB/day of outbound calls to an external API, routed through a single NAT Gateway in one AZ, with half of the calling instances in a different AZ: 25 GB/day of that traffic pays both the NAT Gateway data processing rate and a cross-AZ transfer rate, stacking two per-GB charges on the same bytes. Giving each AZ its own NAT Gateway removes the cross-AZ leg entirely for that half, leaving only the (unavoidable) NAT processing charge. This is a design-target illustration, not a measured bill; actual savings depend on current per-GB rates, which vary by region and change over time, and should be checked in AWS Cost Explorer rather than assumed.

Trade-offs and pitfalls

• Per-AZ NAT Gateways cost more in fixed hourly charges than one shared gateway; the fix only pays for itself once the eliminated cross-AZ transfer volume exceeds that fixed cost, which is worth checking with Cost Explorer before rolling it out everywhere.
• VPC Gateway Endpoints only cover S3 and DynamoDB. Other AWS services need Interface Endpoints (AWS PrivateLink-backed), which do carry an hourly and per-GB charge, so they're a narrower cost win, not a blanket replacement for NAT.
• Don't chase AZ-local placement so hard that it undermines the multi-AZ redundancy the architecture depends on for availability; the fix is about routing waste, not about collapsing back to a single AZ.