AWS Core Services and Architecture Questions
Amazon Web Services' core service catalog and how the pieces compose into a working system: EC2, Lambda, S3, VPC, IAM, RDS, and the managed-service ecosystem. Covers service selection within AWS, common reference architectures, the AWS Well-Architected Framework pillars, and operational patterns specific to the platform. For provider-agnostic compute or storage trade-offs, see the cross-cloud entries.
You detect a suspicious IAM assume-role event, for example a role assumed from an unusual region or IP. Walk through how you'd detect this, contain it (revoke or limit the session, tighten the trust policy), and make sure the automation that does this can't itself be abused.
Sample Answer
Direct answer
Treat this as a security incident, not a policy edit: use CloudTrail-derived signals, correlated through GuardDuty and AWS Config, to detect and score the anomalous AssumeRole call, contain it by cutting off that specific session's future permissions (not by editing the role's trust policy, which only affects future AssumeRole calls), and lock the containment automation itself down so a false positive or a compromised detector can't turn into its own incident.
Structured elaboration
Detect
- CloudTrail records every
AssumeRole/AssumeRoleWithWebIdentity/AssumeRoleWithSAMLcall with source IP, region, and user agent. Route these events through an EventBridge rule into a triage Lambda or Step Functions state machine. - GuardDuty (Amazon's threat-detection service) ingests those CloudTrail management events plus its own threat intelligence and flags IAM / AWS Security Token Service (STS) anomalies: impossible-travel geography, an unfamiliar network for that principal, or an API call the principal has never made before. Treat a GuardDuty finding as one scored input, not the sole trigger.
- AWS Config runs as the compliance/drift layer in parallel: it snapshots the role's trust policy and permission boundary over time, so you can tell whether this AssumeRole succeeded because of a recent, possibly unauthorized trust-policy change, versus a genuine anomaly against a policy that hasn't moved.
Contain (the session, not the role)
- You cannot revoke an already-issued STS session token directly. AWS's actual mechanism: IAM's "Revoke active sessions" action attaches an inline policy named
AWSRevokeOlderSessionsto the role, denying all actions to any session whoseaws:TokenIssueTimepredates the revoke timestamp (with roughly 30 seconds of clock-skew tolerance), while leaving brand-new AssumeRole calls unaffected. - For a narrower blast radius than "deny the whole role," attach a Deny statement keyed on
aws:TokenIssueTimecombined withaws:PrincipalArnoraws:SourceIdentity, so only the flagged session is cut off, not every legitimate caller of that role. - Editing the trust policy is a separate control: it stops future AssumeRole calls, it does nothing to a session already issued. Don't rely on it as the primary containment step.
- Escalate on high confidence: rotate the credentials of the upstream calling principal (IAM user, federation provider, or CI system), since the assumed-role session is a symptom, not the root cause.
Keep the automation from being abused
- Run the detector/containment workflow from a dedicated account, with a role scoped to "attach exactly this deny statement to exactly the flagged role ARN" and nothing broader: no
iam:CreateRole, no wildcardiam:AttachRolePolicy, noiam:PassRole. - Session tags on the automation's own assumed role record who or what triggered each action, so every containment step is attributable in CloudTrail.
- Stage the response: a narrow, low-confidence "soft deny" (block only the riskiest actions) can auto-execute; a full
AWSRevokeOlderSessions-style lockout requires a human-approval step (a Step Functions callback, for example) unless confidence crosses a high threshold. This bounds the automation's own worst-case blast radius, a false positive can't turn into a self-inflicted denial-of-service on a production role.
Worked example
Role data-sync-role is normally assumed only from a corporate CIDR range in us-east-1. CloudTrail shows an AssumeRole call for that role from an unfamiliar region with a new user agent string. GuardDuty correlates this with its anomaly detection and raises a finding; AWS Config confirms the role's trust policy hasn't changed recently, ruling out an authorized-but-undocumented change. The EventBridge rule fires the containment Lambda, which attaches a Deny statement scoped to aws:TokenIssueTime before now and aws:SourceIdentity matching that specific session. A follow-up CloudTrail query confirms subsequent calls under that session now fail with AccessDenied, while other legitimate sessions on the same role continue working. A ticket opens for a human to decide whether the upstream credential (the CI system or federated user that called AssumeRole) also needs rotation.
Trade-offs & pitfalls
- Staged containment avoids an overly broad deny becoming a self-inflicted denial-of-service on a legitimate, high-traffic role; don't wire auto-hard-deny until the anomaly-scoring signal is tuned.
AWSRevokeOlderSessionsdenies the entire role by default, not just the flagged session, unless you narrow the condition withaws:PrincipalArn/aws:SourceIdentity: decide up front whether that blast radius is acceptable for the specific role.- The automation account's own permissions are themselves an attack surface: scope it to specific role ARNs, never a wildcard resource.
- Store every generated deny policy and its triggering event in an immutable log so a later incident review can reconstruct exactly what containment did and when.
How would you design preventive and detective controls to avoid accidental mass-deletion of S3 objects or buckets, and what's your recovery plan if a mass delete happens anyway?
Sample Answer
Direct answer
Layer three kinds of control. Preventive: make accidental or unauthorized mass deletion structurally hard, using least-privilege AWS Identity and Access Management (IAM) plus a Service Control Policy (SCP) deny on destructive S3 calls, Versioning and Object Lock on critical buckets, and cross-account replication so no single compromised account can destroy both copies. Detective: near-real-time alerting on delete-heavy API activity via CloudTrail data events and EventBridge, not just periodic audits. Recovery: a tested runbook that restores from the least-effort, most-trustworthy source available, a separate-account replica first, versioned objects second, archived or Glacier restores last.
Structured elaboration
- Preventive
- Least-privilege IAM plus an AWS Organizations SCP that denies
s3:DeleteBucketand bulks3:DeleteObject*for every principal except a small, separately audited break-glass role; an SCP bounds the maximum permission even a misconfigured IAM policy can grant. - Enable Versioning on every bucket that matters; it's a prerequisite for everything below it.
- Object Lock (governance or compliance mode) on data that must survive even a compromised admin credential. Two things worth getting right: Object Lock can only be enabled on a bucket that already has Versioning turned on, and once Object Lock is enabled on a bucket, you can never disable it or suspend that bucket's Versioning again. It's a one-way architectural decision, not something to flip mid-incident.
- Cross-account, ideally cross-region, replication to a separate, hardened account whose own SCPs deny delete entirely. This is the strongest recovery guarantee because it survives even total compromise of the source account.
- Least-privilege IAM plus an AWS Organizations SCP that denies
- Detective
- Turn on CloudTrail data events for S3 (object-level, not just management events) and route them to a centralized, separate-account log destination so the log itself can't be deleted by whatever compromised the source account.
- An EventBridge rule matching bulk delete API calls triggers a Lambda that counts deletions per principal and bucket in a short window and pages on-call above a threshold, catching a mass delete in minutes rather than when someone notices missing data.
- Daily S3 Inventory plus a scheduled query (Athena or similar) catches slower-burn deletion patterns that wouldn't trip a real-time threshold.
- Recovery runbook, in order
- Isolate: revoke or disable the credentials or role that caused the deletion immediately, and pause replication so a still-running bad process can't propagate deletes to the recovery copy.
- Scope: use CloudTrail data events plus S3 Inventory to enumerate exactly which keys and version IDs were affected.
- Restore, cheapest-safest source first: copy from the cross-account replica if one exists; otherwise remove delete markers or copy prior versions back using S3 Batch Operations driven from a manifest, not a one-by-one script, for anything Versioning protected; for archived data, initiate Glacier restores prioritized by what's needed soonest.
- Validate: reconcile object counts and checksums against S3 Inventory before declaring the incident resolved.
- Post-incident: rotate any credentials involved, close the IAM or SCP gap that allowed it, and schedule a recurring restore drill, quarterly is a reasonable cadence, so the runbook is proven before it's needed for real.
Worked example
A CI job's IAM role is accidentally granted broad S3 permissions, and a bad script run deletes 50,000 objects across a data-lake bucket in under a minute. Because Versioning was on, the deletes are only delete markers. The EventBridge rule watching bulk-delete volume per principal pages on-call within minutes of the burst starting, with the exact detection latency being a tunable alarm threshold, not a promised SLA. The team disables the CI role's credentials, confirms via CloudTrail that exactly one principal and one time window are involved, and runs an S3 Batch Operations job from a manifest of affected keys, pulled from ListObjectVersions, to remove the delete markers, restoring the objects without needing the cross-account replica at all. The replica and Object Lock exist for the rarer, worse case where deletes are permanent.
Trade-offs & pitfalls
- Object Lock in Compliance mode is the strongest guarantee available but is irreversible for the configured retention period, including against your own root account; it needs to be a deliberate decision aligned with actual legal or retention requirements, not a default flipped on everywhere "to be safe."
- Enabling Object Lock is itself a one-way door at the bucket level, easy to miss since most S3 settings are freely reversible. Treat it as a design decision made once per bucket, not an incident-response lever.
- SCP-based denies are powerful, but a deny written too broadly can also block legitimate lifecycle expiration and other benign delete operations; scope the deny to the specific bulk and delete-bucket actions and exempt the automation that's supposed to run lifecycle cleanup.
- A cross-account replica is the strongest recovery path only if that account's own permissions are actually locked down; a replica account with the same broad access as the source doesn't add real protection against a credential compromised with access to both.
A DynamoDB table is write-throttling even though provisioned capacity looks sufficient. Walk through your investigation: partition-key analysis, hot-partition detection, adaptive capacity, and the mitigations you'd apply (including whether to switch to on-demand).
Sample Answer
Direct answer
If DynamoDB is write-throttling despite aggregate provisioned or on-demand capacity looking sufficient, the near-universal cause is a hot partition: a small number of partition-key values absorbing a disproportionate share of writes, so one physical partition's own throughput ceiling is exceeded even though the table-wide capacity has headroom. The fix is almost never "add more capacity"; it's spreading the writes across more partition-key values, and the fix isn't done until it's validated with a load test that reproduces the actual traffic shape that caused the incident.
Structured elaboration
- Confirm it's a hot partition, not a global capacity shortfall. In CloudWatch, compare
ThrottledRequests/WriteThrottleEventsagainstConsumedWriteCapacityUnitsat the table level; if consumed capacity is well below what's provisioned (or below the on-demand ceiling) while throttling is happening, capacity isn't the bottleneck, distribution is. - Find the hot key(s). Turn on CloudWatch Contributor Insights for the table (or, if not already enabled, instrument application-level logging of partition-key values on write) to surface the top N keys by request count. Look specifically for low-cardinality keys, monotonically increasing keys (like a timestamp used directly), or a single shared "global" key that many writers hit.
- Apply short-term mitigation to stop the bleeding. Client-side exponential backoff with jitter, and route incoming writes through a durable buffer (SQS or Kinesis) drained at a controlled rate, so the application doesn't cascade retries into more throttling.
- Redesign the partition key for the long term. Two complementary techniques:
- Write sharding (prefix/suffix): append a shard suffix (a small hash or random number) to a hot key so a single logical entity's writes spread across several physical partitions.
- Time-bucketing: for a shared or "global" key, fold a time bucket (hour, or a finer grain under load) into the partition key so writes naturally spread as new buckets open, rather than piling onto one everlasting key.
- Decide on-demand vs provisioned deliberately, not reflexively. On-demand absorbs unpredictable spikes without you managing capacity, but it doesn't fix a genuinely skewed key design, a heavily hot key will still throttle on-demand once its write rate exceeds a single partition's ceiling; provisioned with a corrected key design is more predictable and usually cheaper for sustained, forecastable load.
- Validate with a load test before calling it fixed, using a request generator whose partition-key distribution matches what Contributor Insights showed for the actual incident (not uniformly random synthetic keys, which would hide the exact skew that caused the problem), and confirm both that
ConsumedWriteCapacityUnitsis now spread evenly across the new key space and thatThrottledRequestsstays at zero under that replayed load.
Worked example
Before: an IoT ingestion table uses DEVICE#<deviceId> as the partition key. A handful of devices (or, worse, a shared counter table using one literal key like METRICS#GLOBAL) send far more writes than the rest, so all of their writes land on the same physical partition regardless of the table's total provisioned throughput.
After, write sharding: the partition key becomes DEVICE#<deviceId>#<shard>, where shard is computed as hash(deviceId + writeTimestamp) mod N for some shard count N chosen so that a single device's peak write rate, divided across N shards, comfortably fits under one partition's throughput ceiling. Reads that need "everything for device X" now issue N parallel Query calls (one per shard value) and merge the results, or are served from a Global Secondary Index if a consolidated view is required.
After, time-bucketing (for the shared-counter case): the partition key becomes METRICS#2026-07-20-14 (an hour bucket) instead of a single unchanging METRICS#GLOBAL key, so writes spread across time as new hourly buckets open automatically. The current hour's bucket can still be hot within that hour, which is exactly why the two techniques are usually combined: bucket by time, and shard within the current bucket.
Validation: replay the write pattern captured from the incident (same relative request rate and same key skew observed via Contributor Insights) against a staging table with the new key design, and confirm ThrottledRequests is zero and per-shard ConsumedWriteCapacityUnits is roughly even, before rolling the schema change to production, and again at expected future peak, not just historical peak, since traffic grows.
Trade-offs and pitfalls
- Sharding adds read-side complexity: a query that used to be one
Querycall becomesNparallel calls (or requires a GSI you now have to keep in sync), which is real ongoing cost, not a one-time redesign expense. - DynamoDB's adaptive capacity helps absorb transient hot keys automatically, but it's reactive and bounded; it is not a substitute for correcting a sustained, structurally skewed key design, and relying on it alone will resurface the same incident at higher scale.
- Switching to on-demand can mask the symptom for many workloads, but a key that's skewed enough to exceed a single partition's ceiling will still throttle under on-demand; treat on-demand as a cost and predictability trade-off, not a fix for the key design.
- A load test that uses uniformly random synthetic keys instead of the actual observed skew will pass cleanly and still let the same incident recur in production, because it never reproduces the condition that caused the original throttling.
What are EC2 placement groups (cluster, spread, partition) and when would you use each? What are the trade-offs and limitations?
Sample Answer
Direct answer
Placement groups let you influence how EC2 places instances on underlying hardware, trading network performance against fault isolation. Cluster packs instances close together on the same high-bandwidth network segment inside one Availability Zone (AZ) for the lowest latency and highest throughput. Spread puts a small number of instances on physically distinct hardware to minimize correlated failure. Partition groups instances into logical partitions, each on its own set of racks, so a single rack failure only takes out one partition, which is how large distributed systems like Kafka or Cassandra stay available.
Structured elaboration
| Strategy | What it does | AZ scope | Instance limit | Best for |
|---|---|---|---|---|
| Cluster | Packs instances on one high-bisection-bandwidth network segment | Single AZ only (can span peered VPCs in the same Region, not AZs) | No fixed cap, but mixing instance types or adding instances later increases the chance of an insufficient-capacity error | Tightly-coupled HPC (high performance computing) / MPI (Message Passing Interface) workloads needing low latency and high throughput between nodes |
| Spread | Places each instance on physically distinct hardware (its own rack) | A single rack-level spread group can span multiple AZs in the same Region | Max 7 running instances per AZ per group | A small number of critical instances you want maximally isolated from each other (e.g. a handful of database primaries/replicas) |
| Partition | Divides the group into logical partitions, each on its own racks | A single partition group can span multiple AZs in the same Region | Max 7 partitions per AZ; instance count per partition limited only by account limits (Dedicated Instances cap at 2 partitions) | Large distributed, rack-aware systems (HDFS, Cassandra, Kafka) |
A few rules apply across all three: an instance can belong to only one placement group at a time, placement groups cannot be merged, you can't launch a Dedicated Host into a placement group, and you can't launch a Spot Instance configured to stop or hibernate on interruption into one.
Worked example
For a tightly coupled HPC job needing very low node-to-node latency (say an 8-node MPI simulation), I'd launch a cluster placement group with all 8 instances of the same instance type in a single launch request, in one AZ, using an enhanced-networking-capable instance family; within a cluster placement group, enhanced-networking instances get up to 10 Gbps for single-flow traffic versus 5 Gbps for instances outside one. For a Kafka cluster that needs rack-fault isolation across many more brokers than a spread group's 7-per-AZ cap allows, I'd use a partition placement group instead, with up to 7 partitions in the AZ; Kafka's own rack-awareness can consume the partition topology AWS exposes so a rack failure only affects the brokers in one partition.
Trade-offs & pitfalls
- Cluster placement groups trade fault tolerance for latency: everything sits in one AZ, so an AZ-level event takes the whole group down together. Pair it with cross-AZ checkpointing or keep the control plane elsewhere.
- Adding instances to a cluster group later, or mixing instance types within it, raises the odds of an insufficient-capacity error, since AWS is trying to keep everything on the same network segment.
- You cannot convert a group's strategy after creation and cannot merge two groups; if you outgrow the 7-per-AZ spread limit, the fix is multiple spread groups, which gives no guarantee of spread between the groups themselves.
- You can move a stopped instance into, out of, or between placement groups, but only while it's stopped, so this isn't a live-migration tool.
- Capacity Reservations behave differently per strategy: they don't reserve capacity in spread or partition groups at all, only in cluster groups (via a group-scoped On-Demand Capacity Reservation), which catches people who assume reservations work the same way everywhere.
You notice increased end-to-end request latency for a microservice. Walk through the diagnostic steps using CloudWatch metrics, ALB metrics, X-Ray, and logs: which would you check first, and what patterns tell you infrastructure versus application problem?
Sample Answer
Work from the outside in: start with Application Load Balancer (ALB) metrics since they show whether the problem sits between the client and your service or inside it, then use AWS X-Ray to find which downstream call within the request is actually slow, then drop into logs at that specific span to find the root cause. Amazon CloudWatch instance-level metrics answer one narrow question: is the underlying compute starved of resources. The pattern that separates infrastructure from application problems is whether the slowness correlates with resource saturation (CPU, memory, disk queue) or with a specific downstream call showing up consistently in X-Ray traces.
Diagnostic order and what each layer tells you
- ALB metrics first:
TargetResponseTime(time the backend took to respond),HTTPCode_Target_5XX_Count,HealthyHostCount, andRequestCountPerTarget(average request load per target in the target group; ALB has no client-facing queue-depth metric the way Classic Load Balancers do). RisingTargetResponseTimewith a steadyHealthyHostCountpoints at the application; a risingRequestCountPerTargetalongside a droppingHealthyHostCountpoints at capacity, the same targets absorbing more load because fewer of them are healthy. - CloudWatch instance and service metrics: CPU utilization, memory (if the CloudWatch agent publishes it), disk queue length, and status check failures. Sustained CPU above roughly 70 to 80 percent, a growing disk queue, or a failed status check is a real infrastructure signal; normal-looking infrastructure metrics next to a slow ALB response time is a strong sign the problem lives in application code or a downstream dependency, not the host.
- X-Ray traces: the service map and per-trace latency breakdown show which segment of the request is actually slow, an internal computation, a database call, or a third-party API. A long span on a downstream call, a slow SQL query, a saturated connection pool, is an application or dependency problem even though it shows up as backend latency at the ALB.
- Logs: once X-Ray points at a specific segment, application logs, database slow-query logs, and container standard output around that timestamp and trace ID confirm the actual cause, a stack trace, a lock wait, a garbage-collection pause, or a connection pool timeout.
Patterns that separate infrastructure from application
- Infrastructure: CPU, memory, or disk saturation across many hosts, failing status checks, or
RequestCountPerTargetrising alongside a drop inHealthyHostCount, all independent of what any individual trace shows. - Application: normal host-level metrics, but
TargetResponseTimeand X-Ray both point at a specific span (a database call, a downstream API, a lock), and logs at that timestamp show exceptions, long garbage-collection pauses, or connection pool exhaustion. - Mixed, and easy to misdiagnose: Auto Scaling Group cooldown delays, or a database hitting its own connection or throughput limit, look infrastructure-shaped on a dashboard but are actually caused by an upstream traffic pattern or an application-side connection leak.
Tying this to service-level objectives and alerting
Set alarm thresholds on the same metrics used above, tied to an actual service-level objective (SLO), for example 99.9 percent of requests under 500 milliseconds, rather than an arbitrary round number, so an alarm firing means the error budget (the allowed amount of failure or excess latency for the period) is actually at risk. Track the error budget's burn rate rather than paging on every threshold breach, since a two-minute blip that recovers on its own should not wake anyone up, while a sustained burn that will exhaust the monthly budget in a few hours should.
Worked example
Checkout latency creeps from a 99th-percentile of 200 milliseconds to 1.2 seconds over 20 minutes. ALB metrics show TargetResponseTime climbing while HealthyHostCount stays flat and RequestCountPerTarget stays flat too, ruling out a capacity problem. CloudWatch shows CPU at 25 percent across all targets, ruling out host saturation. X-Ray's service map shows the slow span is consistently a call to the inventory service, with its own latency climbing in the trace timeline. Logs on the inventory service around that window show a specific slow-query pattern: a full table scan on an unindexed column that only becomes slow once the table crossed a size threshold. That is a database and application problem end to end, not infrastructure, and the fix, adding an index, has nothing to do with scaling anything.
Trade-offs and pitfalls
- Jumping straight to logs before narrowing scope with ALB and CloudWatch metrics wastes time searching a haystack. The metrics exist specifically to tell you where to look before you start reading logs.
- X-Ray requires sampling and instrumentation to already be in place before the incident. If tracing is not enabled, or the sampling rate is too low to capture the slow requests, you lose the fastest path to the root cause and fall back to correlating logs by timestamp, which is slower and noisier.
- Alarming on raw metric thresholds instead of an SLO-tied burn rate produces either too many pages (threshold too tight) or missed real degradations (threshold too loose). Revisit thresholds against actual historical percentiles, not a guess.
- A downstream dependency showing up as "your" latency in X-Ray is still your incident to manage even though the root cause sits in a service you do not own. Know the escalation path to that team before you need it.
Unlock Full Question Bank
Get access to all AWS Core Services and Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.