AWS Core Services and Architecture Questions
Amazon Web Services' core service catalog and how the pieces compose into a working system: EC2, Lambda, S3, VPC, IAM, RDS, and the managed-service ecosystem. Covers service selection within AWS, common reference architectures, the AWS Well-Architected Framework pillars, and operational patterns specific to the platform. For provider-agnostic compute or storage trade-offs, see the cross-cloud entries.
You detect a suspicious IAM assume-role event, for example a role assumed from an unusual region or IP. Walk through how you'd detect this, contain it (revoke or limit the session, tighten the trust policy), and make sure the automation that does this can't itself be abused.
Sample Answer
Direct answer
Treat this as a security incident, not a policy edit: use CloudTrail-derived signals, correlated through GuardDuty and AWS Config, to detect and score the anomalous AssumeRole call, contain it by cutting off that specific session's future permissions (not by editing the role's trust policy, which only affects future AssumeRole calls), and lock the containment automation itself down so a false positive or a compromised detector can't turn into its own incident.
Structured elaboration
Detect
- CloudTrail records every
AssumeRole/AssumeRoleWithWebIdentity/AssumeRoleWithSAMLcall with source IP, region, and user agent. Route these events through an EventBridge rule into a triage Lambda or Step Functions state machine. - GuardDuty (Amazon's threat-detection service) ingests those CloudTrail management events plus its own threat intelligence and flags IAM / AWS Security Token Service (STS) anomalies: impossible-travel geography, an unfamiliar network for that principal, or an API call the principal has never made before. Treat a GuardDuty finding as one scored input, not the sole trigger.
- AWS Config runs as the compliance/drift layer in parallel: it snapshots the role's trust policy and permission boundary over time, so you can tell whether this AssumeRole succeeded because of a recent, possibly unauthorized trust-policy change, versus a genuine anomaly against a policy that hasn't moved.
Contain (the session, not the role)
- You cannot revoke an already-issued STS session token directly. AWS's actual mechanism: IAM's "Revoke active sessions" action attaches an inline policy named
AWSRevokeOlderSessionsto the role, denying all actions to any session whoseaws:TokenIssueTimepredates the revoke timestamp (with roughly 30 seconds of clock-skew tolerance), while leaving brand-new AssumeRole calls unaffected. - For a narrower blast radius than "deny the whole role," attach a Deny statement keyed on
aws:TokenIssueTimecombined withaws:PrincipalArnoraws:SourceIdentity, so only the flagged session is cut off, not every legitimate caller of that role. - Editing the trust policy is a separate control: it stops future AssumeRole calls, it does nothing to a session already issued. Don't rely on it as the primary containment step.
- Escalate on high confidence: rotate the credentials of the upstream calling principal (IAM user, federation provider, or CI system), since the assumed-role session is a symptom, not the root cause.
Keep the automation from being abused
- Run the detector/containment workflow from a dedicated account, with a role scoped to "attach exactly this deny statement to exactly the flagged role ARN" and nothing broader: no
iam:CreateRole, no wildcardiam:AttachRolePolicy, noiam:PassRole. - Session tags on the automation's own assumed role record who or what triggered each action, so every containment step is attributable in CloudTrail.
- Stage the response: a narrow, low-confidence "soft deny" (block only the riskiest actions) can auto-execute; a full
AWSRevokeOlderSessions-style lockout requires a human-approval step (a Step Functions callback, for example) unless confidence crosses a high threshold. This bounds the automation's own worst-case blast radius, a false positive can't turn into a self-inflicted denial-of-service on a production role.
Worked example
Role data-sync-role is normally assumed only from a corporate CIDR range in us-east-1. CloudTrail shows an AssumeRole call for that role from an unfamiliar region with a new user agent string. GuardDuty correlates this with its anomaly detection and raises a finding; AWS Config confirms the role's trust policy hasn't changed recently, ruling out an authorized-but-undocumented change. The EventBridge rule fires the containment Lambda, which attaches a Deny statement scoped to aws:TokenIssueTime before now and aws:SourceIdentity matching that specific session. A follow-up CloudTrail query confirms subsequent calls under that session now fail with AccessDenied, while other legitimate sessions on the same role continue working. A ticket opens for a human to decide whether the upstream credential (the CI system or federated user that called AssumeRole) also needs rotation.
Trade-offs & pitfalls
- Staged containment avoids an overly broad deny becoming a self-inflicted denial-of-service on a legitimate, high-traffic role; don't wire auto-hard-deny until the anomaly-scoring signal is tuned.
AWSRevokeOlderSessionsdenies the entire role by default, not just the flagged session, unless you narrow the condition withaws:PrincipalArn/aws:SourceIdentity: decide up front whether that blast radius is acceptable for the specific role.- The automation account's own permissions are themselves an attack surface: scope it to specific role ARNs, never a wildcard resource.
- Store every generated deny policy and its triggering event in an immutable log so a later incident review can reconstruct exactly what containment did and when.
You have Lambda functions that need to query a relational database under high concurrency. How would you handle connection pooling, and what role does RDS Proxy play versus reusing connections across warm invocations?
Sample Answer
Direct answer
Put RDS Proxy between Lambda and the database rather than letting each Lambda execution environment open its own connection: Proxy pools and multiplexes a large number of client-facing connections onto a much smaller, stable set of physical database connections, absorbs the connection-storm problem that comes from Lambda's concurrency model, and manages AWS Identity and Access Management (IAM)-auth token rotation for you. Warm-invocation connection reuse, caching a client handle in the Lambda execution environment's global scope, still matters and is complementary, not a substitute: it avoids re-establishing the Proxy-facing connection on every invocation of an already-warm container, while Proxy is what keeps the database itself from seeing thousands of physical connections when Lambda scales out concurrently.
Structured elaboration
The problem RDS Proxy solves
A relational database has a hard ceiling on max_connections, in the low thousands at most, set by instance memory. Lambda can scale to hundreds or thousands of concurrent execution environments in seconds, and if each one opens its own direct connection to the database, you exhaust max_connections almost immediately under a burst, regardless of how efficient the query itself is. RDS Proxy sits in front of the database and:
- Pools and multiplexes: many Lambda-side logical connections share a much smaller pool of physical connections held open to the database, since most connections spend most of their time idle between queries.
- Handles auth centrally: works with IAM database authentication so Lambda does not need database credentials embedded in its code or environment; Proxy manages short-lived credential exchange with the database on the app's behalf.
- Smooths failover: for Aurora, Proxy keeps client-facing connections open across a failover event instead of every client immediately seeing a connection error, reconnecting transparently on the Proxy side.
Configuration knobs that matter
- MaxConnectionsPercent: the percentage of the target database's
max_connectionsthis Proxy, or a specific target group, is allowed to use, leaving headroom for other consumers of the same database. - MaxIdleConnectionsPercent: how many of those connections Proxy is willing to keep idle in the pool versus closing back down.
- ConnectionBorrowTimeout: how long a client waits for the Proxy to hand it a connection before failing; this is your effective backpressure signal when the pool is genuinely saturated.
- Session pinning: certain session-level operations, temp tables, session variables, some transaction patterns, force Proxy to pin a client to one specific physical connection for the rest of that session, which defeats multiplexing for that client. Know which patterns in your queries trigger pinning and avoid them where you can, since they reduce Proxy's actual pooling benefit.
Where warm-invocation reuse still helps
Within a single warm Lambda execution environment, cache the database client as a module-level variable rather than re-creating it on every invocation. This avoids repeating the TLS handshake and Proxy-side connection setup for every invocation of an already-warm container; it does not replace Proxy, because a cold start or a newly spun-up concurrent execution environment still needs a fresh connection, and that is exactly the storm Proxy exists to absorb.
What to do when the pool is genuinely exhausted
- Return a clear rejection with backoff guidance rather than letting the request hang until Lambda's own timeout.
- Buffer non-latency-sensitive writes through a queue and drain them into the database at a controlled rate instead of writing synchronously from every Lambda invocation.
- Keep queries short and transactions small; a long-held transaction ties up a pooled connection and reduces how many other invocations Proxy can serve from the same physical pool.
Worked example
A database sized for a max_connections value of 1,000 fronts a Lambda function that can burst to 2,000 concurrent invocations. Configure the Proxy's target group with MaxConnectionsPercent at 80, leaving 20% headroom for other clients such as an admin console or a batch job:
1000×0.80=800 connections available to this Proxy
With 2,000 concurrent Lambda invocations each needing a connection only for the brief duration of a query, Proxy multiplexes them across those 800 physical connections rather than needing 2,000 physical connections; invocations that arrive while all 800 are briefly in use wait up to the borrow timeout for one to free up, rather than the database itself ever seeing more than 800 connections.
Trade-offs and pitfalls
- RDS Proxy adds a small amount of latency per query, an extra network hop, and its own hourly cost; for very low-concurrency workloads, plain connection reuse in a warm Lambda might be enough and Proxy is unnecessary overhead.
- Session pinning is the most common way teams get less benefit from Proxy than they expected; if your ORM or query patterns rely heavily on session state, audit which patterns trigger pinning.
- Do not rely on Proxy alone to make an unbounded-concurrency Lambda safe for the database; borrow-timeout failures under sustained overload are still failures, just failures the database itself did not see. You still need to address the actual concurrency-to-capacity mismatch, reserved concurrency limits, queue-based buffering, or a bigger database.
- Provisioned Concurrency reduces cold starts, and therefore how often a fresh connection has to be established, but costs money for capacity held ready; it is a traffic-shaping choice independent of whether you are using Proxy, do not treat it as a pooling mechanism by itself.
You notice increased end-to-end request latency for a microservice. Walk through the diagnostic steps using CloudWatch metrics, ALB metrics, X-Ray, and logs: which would you check first, and what patterns tell you infrastructure versus application problem?
Sample Answer
Work from the outside in: start with Application Load Balancer (ALB) metrics since they show whether the problem sits between the client and your service or inside it, then use AWS X-Ray to find which downstream call within the request is actually slow, then drop into logs at that specific span to find the root cause. Amazon CloudWatch instance-level metrics answer one narrow question: is the underlying compute starved of resources. The pattern that separates infrastructure from application problems is whether the slowness correlates with resource saturation (CPU, memory, disk queue) or with a specific downstream call showing up consistently in X-Ray traces.
Diagnostic order and what each layer tells you
- ALB metrics first:
TargetResponseTime(time the backend took to respond),HTTPCode_Target_5XX_Count,HealthyHostCount, andRequestCountPerTarget(average request load per target in the target group; ALB has no client-facing queue-depth metric the way Classic Load Balancers do). RisingTargetResponseTimewith a steadyHealthyHostCountpoints at the application; a risingRequestCountPerTargetalongside a droppingHealthyHostCountpoints at capacity, the same targets absorbing more load because fewer of them are healthy. - CloudWatch instance and service metrics: CPU utilization, memory (if the CloudWatch agent publishes it), disk queue length, and status check failures. Sustained CPU above roughly 70 to 80 percent, a growing disk queue, or a failed status check is a real infrastructure signal; normal-looking infrastructure metrics next to a slow ALB response time is a strong sign the problem lives in application code or a downstream dependency, not the host.
- X-Ray traces: the service map and per-trace latency breakdown show which segment of the request is actually slow, an internal computation, a database call, or a third-party API. A long span on a downstream call, a slow SQL query, a saturated connection pool, is an application or dependency problem even though it shows up as backend latency at the ALB.
- Logs: once X-Ray points at a specific segment, application logs, database slow-query logs, and container standard output around that timestamp and trace ID confirm the actual cause, a stack trace, a lock wait, a garbage-collection pause, or a connection pool timeout.
Patterns that separate infrastructure from application
- Infrastructure: CPU, memory, or disk saturation across many hosts, failing status checks, or
RequestCountPerTargetrising alongside a drop inHealthyHostCount, all independent of what any individual trace shows. - Application: normal host-level metrics, but
TargetResponseTimeand X-Ray both point at a specific span (a database call, a downstream API, a lock), and logs at that timestamp show exceptions, long garbage-collection pauses, or connection pool exhaustion. - Mixed, and easy to misdiagnose: Auto Scaling Group cooldown delays, or a database hitting its own connection or throughput limit, look infrastructure-shaped on a dashboard but are actually caused by an upstream traffic pattern or an application-side connection leak.
Tying this to service-level objectives and alerting
Set alarm thresholds on the same metrics used above, tied to an actual service-level objective (SLO), for example 99.9 percent of requests under 500 milliseconds, rather than an arbitrary round number, so an alarm firing means the error budget (the allowed amount of failure or excess latency for the period) is actually at risk. Track the error budget's burn rate rather than paging on every threshold breach, since a two-minute blip that recovers on its own should not wake anyone up, while a sustained burn that will exhaust the monthly budget in a few hours should.
Worked example
Checkout latency creeps from a 99th-percentile of 200 milliseconds to 1.2 seconds over 20 minutes. ALB metrics show TargetResponseTime climbing while HealthyHostCount stays flat and RequestCountPerTarget stays flat too, ruling out a capacity problem. CloudWatch shows CPU at 25 percent across all targets, ruling out host saturation. X-Ray's service map shows the slow span is consistently a call to the inventory service, with its own latency climbing in the trace timeline. Logs on the inventory service around that window show a specific slow-query pattern: a full table scan on an unindexed column that only becomes slow once the table crossed a size threshold. That is a database and application problem end to end, not infrastructure, and the fix, adding an index, has nothing to do with scaling anything.
Trade-offs and pitfalls
- Jumping straight to logs before narrowing scope with ALB and CloudWatch metrics wastes time searching a haystack. The metrics exist specifically to tell you where to look before you start reading logs.
- X-Ray requires sampling and instrumentation to already be in place before the incident. If tracing is not enabled, or the sampling rate is too low to capture the slow requests, you lose the fastest path to the root cause and fall back to correlating logs by timestamp, which is slower and noisier.
- Alarming on raw metric thresholds instead of an SLO-tied burn rate produces either too many pages (threshold too tight) or missed real degradations (threshold too loose). Revisit thresholds against actual historical percentiles, not a guess.
- A downstream dependency showing up as "your" latency in X-Ray is still your incident to manage even though the root cause sits in a service you do not own. Know the escalation path to that team before you need it.
How would you provide secure shell-like access to EC2 instances (and Kubernetes nodes) without exposing SSH to the internet or distributing SSH keys? Walk through using instance profiles and Systems Manager Session Manager, plus how you'd audit sessions.
Sample Answer
Direct answer
Route all shell access through AWS Systems Manager (SSM) Session Manager instead of SSH: give each instance, or Kubernetes node, an AWS Identity and Access Management (IAM) instance profile with just enough permission to act as an SSM managed node, close SSH in the security groups entirely, and grant human or CI identities ssm:StartSession scoped to specific instances and specific SSM documents. Session content is encrypted with a customer managed KMS key and can be streamed to CloudWatch Logs or S3 for audit, while the session API calls themselves, StartSession and TerminateSession, are already logged by CloudTrail as standard management events, no separate configuration needed for that part.
Structured elaboration
Building blocks
- Instance profile, not SSH keys. The instance needs the SSM Agent, preinstalled on most current AMIs, and an IAM role attached as an instance profile with just enough permission to register as a managed node. No key pair, no distributed private keys.
- Network path. Route SSM traffic through Virtual Private Cloud (VPC) interface endpoints, private connections into your own isolated AWS network, for the
ssm,ssmmessages, andec2messagesservices so sessions never need a public IP or NAT egress; a NAT path also works if you would rather keep egress simple, the endpoints are just the tighter-network option. - IAM authorization, scoped per document. Session Manager exposes different capabilities as different SSM documents: an interactive shell is the RunShell document, port forwarding is a separate pair of documents, and file transfer uses separate plugins. Your IAM policy grants
ssm:StartSessionagainst a Resource list of specific document Amazon Resource Names (ARNs), AWS's unique identifier strings for a resource, plus specific instance ARNs or a tag condition, so a user who is only granted the RunShell document ARN cannot start a port-forwarding session at all, there is no separate switch to remember, the capability simply is not in their policy.
{
"Effect": "Allow",
"Action": "ssm:StartSession",
"Resource": [
"arn:aws:ec2:us-east-1:111122223333:instance/i-0123456789abcdef0",
"arn:aws:ssm:us-east-1:111122223333:document/SSM-SessionManagerRunShell"
]
}
- Encryption and logging. In Session Manager preferences, turn on KMS encryption for session data, which grants the caller a data-key permission on the chosen key, and configure logging to CloudWatch Logs and S3, so session transcripts, what was typed and returned, are encrypted at rest and retained per your policy.
- Auditing, two distinct layers. CloudTrail records the control-plane calls, StartSession, TerminateSession, who, when, against which instance, as ordinary management events, logged by default with no extra CloudTrail configuration required. Separately, the session transcript itself, the actual command input and output, only exists if you turned on the CloudWatch Logs or S3 streaming in step 4; that is a distinct feature from CloudTrail and is what you would review to see what a user actually typed.
Worked example
An engineer needs to debug a production instance outside business hours. With this setup: they authenticate normally, their role's policy allows ssm:StartSession scoped to instances tagged for that environment and only the RunShell document, they run the start-session command, get a shell with no SSH port ever having been open, and every keystroke of that session streams, KMS-encrypted, to a CloudWatch Logs group with a retention policy your security team can search later. If that same engineer tried to start a port-forwarding session against the same instance, the call would fail with an IAM authorization error, because their policy's Resource list never included that document's ARN; port forwarding was never granted, not merely disabled by a switch.
Trade-offs and pitfalls
- This removes the bastion host's attack surface and the SSH key distribution problem entirely, but it makes your IAM hygiene the new perimeter: an overly broad
ssm:StartSessionpolicy, wildcarded documents or instances, reintroduces exactly the blast radius you were trying to avoid. - If the SSM Agent itself is unhealthy or the instance cannot reach its required endpoints, Session Manager access is unavailable; you need a documented emergency path, such as the EC2 serial console or a break-glass bastion kept cold, rather than discovering the gap during an actual incident.
- Session logging is opt-in per Session Manager preference, verify it is actually configured and encrypted rather than assuming it is; an unconfigured environment gives you the access-control benefit of SSM without the audit trail.
- IAM session policies can additionally restrict actions taken inside a session, but detecting and killing a session that looks compromised in real time still requires alerting on the logging stream or CloudTrail, SSM will not flag anomalous behavior for you on its own.
Design a layered network security posture using Security Groups, NACLs, AWS WAF, AWS Shield, and Firewall Manager. Where does each control belong, and what are the common misconfigurations to avoid when centralizing enforcement across multiple accounts?
Sample Answer
A layered AWS network security posture applies controls at increasingly broad scopes: Security Groups filter traffic per resource, Network Access Control Lists (NACLs) add a coarse subnet-level backstop, AWS Web Application Firewall (WAF) filters application-layer requests at the edge, AWS Shield absorbs Distributed Denial of Service (DDoS) attacks, and AWS Firewall Manager centrally enforces all of the above across every account in an AWS Organization. No single control is sufficient alone: Security Groups stop most unwanted traffic, but WAF and Shield exist specifically because Security Groups cannot see inside a Hypertext Transfer Protocol (HTTP) request or absorb a flood, and Firewall Manager exists because per-account discipline stops scaling past a handful of accounts.
Where each control belongs
flowchart TB
Internet["Internet traffic"] --> Shield["AWS Shield Standard/Advanced\nedge DDoS absorption"]
Shield --> Edge["CloudFront / Application Load Balancer"]
Edge --> WAF["AWS WAF\napplication-layer filtering, rate-based rules"]
WAF --> NACL["Subnet NACL\nstateless, coarse allow/deny"]
NACL --> SG["Security Group\nstateful, per-resource least privilege"]
SG --> App["Application tier: EC2 / ECS / Lambda"]
App --> DBSG["Database-tier Security Group"]
DBSG --> DB["RDS / database"]
FM["AWS Firewall Manager"] -.enforces policy on.-> WAF
FM -.enforces policy on.-> Shield
FM -.enforces policy on.-> SG
Org["AWS Organizations + Service Control Policies"] -.governs.-> FM
Control-by-control role
- Security Groups (host/resource): a stateful firewall attached to Elastic Network Interfaces (ENIs), Amazon Elastic Compute Cloud (EC2) instances, and load balancers. Reference other security groups by ID instead of Classless Inter-Domain Routing (CIDR) ranges wherever possible, so a rule survives an IP address change.
- NACLs (subnet): stateless, evaluated in rule-number order, and applied to everything in the subnet. Use them as a blunt secondary backstop, for example blocking a known-bad CIDR range, rather than the primary control, since being stateless means you must explicitly allow the return traffic too.
- AWS WAF (application layer): sits in front of CloudFront or an Application Load Balancer and inspects the actual HTTP request. Use managed rule groups for common attack patterns plus rate-based rules to throttle a client hammering an endpoint. This layer also matters for serverless front doors: an API Gateway or Lambda function URL behind WAF gets the same rate-based throttling against a flood of malformed or maliciously injected events trying to drive up invocation cost, protection a Security Group cannot provide since Lambda has no traditional network ingress to filter.
- AWS Shield: Shield Standard is free and automatically protects every CloudFront distribution and Application Load Balancer against common network and transport-layer DDoS traffic. Shield Advanced adds continuous access to the DDoS Response Team, cost protection against scaling charges caused by an attack, and broader detection, worth the added cost for internet-facing production endpoints where downtime or an attack-driven bill spike is unacceptable.
- AWS Firewall Manager: centralizes WAF rule sets, Shield Advanced protections, security group policies, and DNS Firewall rules across every account in an AWS Organization from one security admin account, so a new account automatically inherits the baseline instead of relying on someone to configure it by hand.
Common misconfigurations
- Security Groups open to the entire internet on administrative or database ports, usually left over from initial setup and never tightened.
- Treating the NACL as the primary control: because NACLs are stateless, forgetting the ephemeral-port return-traffic rule silently breaks legitimate connections, and teams then "fix" it by opening the NACL wide, defeating its purpose.
- Applying WAF to the Application Load Balancer but exposing a second entry point, like a CloudFront distribution serving a static site or a direct storage-service website endpoint, that bypasses WAF entirely.
- Enabling Shield Advanced generally but not on the specific resources that need it, then discovering during an attack that the DDoS Response Team engagement and cost protection only apply to protected resources.
- Rolling out Firewall Manager policies without first mapping which accounts and organizational units are in scope, which either misses new accounts or breaks a legitimate account that needed an exception.
Worked example
A public checkout API sits behind CloudFront and an Application Load Balancer, backed by ECS tasks and an RDS database, across twenty AWS accounts in one Organization. Shield Standard covers CloudFront and the load balancer automatically. Shield Advanced is added because a DDoS-driven CloudFront bill spike during a past sale directly cost the business money. WAF sits on the CloudFront distribution with a managed rule group plus a rate-based rule capping any single IP address at a fixed request rate over a five-minute window. The load balancer's security group only allows inbound traffic from CloudFront's managed prefix list, not the open internet. The ECS task security group only allows inbound traffic from the load balancer's security group by reference, and the RDS security group only allows inbound traffic from the ECS task security group. Firewall Manager, run from the security account, pushes the same WAF rule set and Shield Advanced protection to every account tagged "production" in the Organization, so the twentieth account gets the same baseline as the first without a manual step.
Trade-offs and pitfalls
- Defense in depth adds real operational complexity: every layer is one more place a legitimate request can be blocked and one more thing to debug. Automate the rollout, through Firewall Manager, CloudFormation StackSets, or an infrastructure-as-code module, rather than hand-configuring each account, or the layers drift out of sync.
- WAF on CloudFront gives global, edge-level protection and lower origin latency; WAF on the Application Load Balancer is regional and sees traffic after it has already passed the content delivery network. Decide deliberately rather than bolting WAF onto whichever resource is convenient.
- Shield Advanced's cost protection only applies to resources it is actively protecting, so scoping it correctly matters as much as enabling it.
- Centralizing enforcement through Firewall Manager and Organizations Service Control Policies is powerful, but an overly broad policy can block legitimate exceptions, like a development account intentionally testing without WAF. Pair centralization with a documented, auditable exception process.
Unlock Full Question Bank
Get access to all AWS Core Services and Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.