AWS Core Services and Architecture Questions

Direct answer

Enforce least privilege operationally, not just by writing tight policies once. Default every non-human principal to a role assumed through STS (AWS Security Token Service) rather than a long-lived IAM user, cap what any role can ever have with permission boundaries (and Service Control Policies at the AWS Organizations level), require MFA for all human console/CLI access, rotate what long-lived credentials remain, and continuously audit actual usage against granted permissions with IAM Access Analyzer and CloudTrail so drift gets caught automatically instead of at the next annual review.

Structured elaboration

Users vs roles

• Reserve IAM users for the rare cases where a human genuinely needs a persistent identity (and even then, prefer federated access through IAM Identity Center where possible).
• Everything else (services, cross-account access, CI/CD) should assume a role. Use instance/task/execution roles for compute, and cross-account IAM roles for service-to-service or partner access.

Temporary credentials (STS)

• Prefer sts:AssumeRole / sts:AssumeRoleWithWebIdentity for all non-root access so credentials expire on their own and the blast radius of a leak is bounded by the session duration.
• Set a short MaxSessionDuration on sensitive roles, and use session tags to carry request context into CloudTrail for later correlation.

Permission boundaries and SCPs

• A permission boundary is a managed policy attached to a user or role that sets the maximum permissions any policy attached to that principal can ever grant, no matter what gets attached later. It never grants anything by itself.
• An SCP, applied at an AWS Organizations account or organizational unit, works the same way at the account level: it restricts what can ever be allowed within that account or OU, and also never grants permissions on its own.
• Together they form two independent ceilings: use SCPs to guardrail entire accounts (deny iam:CreateUser, deny leaving a set of approved regions), and permission boundaries to guardrail individual automation or developer roles within an account.

MFA and root protection

• Require MFA for all human IAM and IAM Identity Center sign-ins, enforced with the aws:MultiFactorAuthPresent condition key on sensitive actions.
• Remove the AWS account root user's access keys entirely, enable MFA on the root user, and treat root sign-in as a rare, logged, break-glass action rather than a routine one.

Rotation and hygiene

• Where long-lived credentials must exist at all (a legacy integration that can't assume a role), rotate them on a defined cadence and automate the deletion of keys that IAM Access Analyzer or Access Advisor shows as unused.

Continuous audit

• Use CloudTrail as the ground truth of what actually happened, IAM Access Analyzer to surface external/cross-account access and generate least-privilege policy suggestions from observed activity, and aws iam simulate-principal-policy to test whether a proposed policy actually allows or denies a specific action before you ship it.
• Treat any principal whose granted permissions are visibly wider than its observed usage (via Access Advisor's last-accessed data) as a finding to remediate, not a one-time cleanup.

Worked example

A permission boundary that caps a developer role to a small, named set of services regardless of what gets attached to it later, paired with a CLI check that the boundary actually blocks an out-of-scope action:

// Permission boundary attached to DeveloperRole
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:*", "dynamodb:*", "logs:*", "cloudwatch:*"],
    "Resource": "*"
  }]
}

Even if someone later attaches an AdministratorAccess-style policy to DeveloperRole by mistake, the boundary still caps effective permissions to the four services listed. You can verify this before it ever gets tested in production:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111122223333:role/DeveloperRole \
  --action-names ec2:TerminateInstances \
  --resource-arns "*"
# Expected result: implicitDeny, because ec2 is outside the boundary

Trade-offs and pitfalls

• SCPs and permission boundaries only restrict; a very common confusion is expecting either one to grant access. If a principal has no explicit Allow in its identity-based (or resource-based) policy, an SCP or boundary that permits the action changes nothing.
• An overly strict SCP can silently break AWS itself if it blocks iam:CreateServiceLinkedRole, which several AWS services need in order to function; scope denies carefully rather than denying broad IAM namespaces wholesale.
• MFA enforced via a condition key applies to how the session was established, not to every subsequent API call inside that session, so pair MFA requirements with short session durations rather than treating MFA alone as a per-call control.
• Continuous auditing only works if CloudTrail is actually enabled organization-wide and retained; a control that exists on paper but isn't wired into monitoring gives a false sense of coverage.

Direct answer

At-least-once delivery means SQS can redeliver a message the consumer already processed (after a crash, a timeout, or a retry), so exactly-once processing has to be built at the application layer: give every event an idempotency key, and before doing real work, atomically claim that key in a durable store using a conditional write. If the claim fails because the key already exists, you already processed it (or are processing it), so skip the work and return the stored result instead of redoing it.

Structured elaboration

Idempotency key and dedup store

• Every event needs a unique idempotencyId: client-supplied, or derived deterministically from the event's content (a hash of stable fields) if the producer doesn't supply one.
• The dedup store exposes one operation conceptually: TryClaim(idempotencyId) -> {claimed: true} or {claimed: false, existingStatus, existingResult}.
• DynamoDB is the natural fit: PutItem with ConditionExpression = attribute_not_exists(idempotencyId) is atomic, so exactly one concurrent caller wins the claim even under retries or duplicate deliveries. Store status (PROCESSING / SUCCESS / FAILED), the result (or a pointer to it, e.g., an S3 key, if it's large), and a Time To Live (TTL) attribute, a timestamp DynamoDB uses to auto-delete the item once it's passed, so old dedup records expire automatically once the duplicate-arrival window has passed.

Handling in-flight duplicates (the crash case)

• If a worker claims the key, crashes before finishing, and a duplicate delivery arrives, a naive "exists = skip" check would permanently skip a message that never actually completed. Store a leaseExpiresAt alongside PROCESSING; a second worker seeing status = PROCESSING with an expired lease can attempt a conditional update (status = PROCESSING and leaseExpiresAt < now) to take over, rather than skip forever.
• On success, UpdateItem sets status = SUCCESS and stores the result; a duplicate delivery arriving after that point reads the stored result and returns it without reprocessing, which is what makes this "exactly-once" from the application's point of view even though SQS delivery itself stays at-least-once.

Latency and storage cost

• The dedup check adds one conditional write (and, on a cache hit, one read) per event, on top of the actual processing work; keep the TTL only as long as the realistic duplicate-arrival window requires (retention_window = expected max duplicate delay + any window you need for after-the-fact auditing), since a longer TTL means more stored items and more storage cost, not more correctness.
• Combine this with Lambda's SQS partial batch response feature so that within a batch of, say, 10 messages, only the ones that actually failed (or weren't yet claimed) get redelivered, not the whole batch; that keeps duplicate volume down at the source instead of relying entirely on the dedup store to absorb it.

Worked example

An order-created event arrives, gets processed, and the worker updates order status and sends a confirmation email, but the Lambda invocation times out right after the email send and before it deletes the SQS message. SQS redelivers the same event after the visibility timeout. Without dedup, the customer gets a second confirmation email and, depending on the downstream logic, a duplicate status update. With the design above: the redelivered event's TryClaim finds status = SUCCESS already recorded (the first attempt did finish the real work, it just didn't get to acknowledge SQS), so the handler short-circuits, returns immediately, and deletes the message without resending the email. If instead the first attempt had crashed before finishing (say, the process died before sending the email), the claim would show PROCESSING with an expired lease, and the redelivered event would correctly retry the actual work rather than being incorrectly skipped.

Trade-offs and pitfalls

• A cache like Redis (SET key NX EX ttl) is faster than DynamoDB for the claim check, but durability requires replication/persistence configuration, and a cache eviction under memory pressure can silently reopen the dedup window; DynamoDB's durability makes it the safer default unless the latency budget specifically requires sub-millisecond claims.
• The lease-expiry takeover logic is the part teams most often skip, and it's exactly the part that causes silent message loss: a worker that dies mid-processing leaves a dangling PROCESSING record that, without lease expiry, permanently blocks all future retries of that event.
• Storing the full result in the dedup record is convenient for idempotent responses but can get expensive for large payloads; store a reference (S3 key, or the ID of the already-created downstream resource) instead of the payload itself when results are non-trivial in size.
• Idempotency at the dedup-store layer doesn't make downstream side effects idempotent for free: if "processing" includes calling a third-party API that isn't itself idempotent, the dedup key only prevents your handler from re-running, not from having already made two calls if the crash happened after the external call but before the claim was marked SUCCESS; that residual risk needs its own mitigation (idempotency keys passed to the downstream API, where supported).

EC2 instance store is physically attached Non-Volatile Memory express (NVMe) storage with the lowest possible latency and highest raw input/output operations per second (IOPS), but it is ephemeral: the data is gone the moment the instance stops, terminates, or the underlying hardware fails. Amazon Elastic Block Store (EBS), in the gp3 or io2 families, is network-attached, durable, persists independently of the instance lifecycle, and supports snapshots, but has to cross the network to reach the instance, which adds latency EBS cannot fully eliminate no matter how much performance is provisioned. Reach for instance store when the data is disposable and the workload needs raw local speed; reach for EBS when the data has to survive the instance.

Why the trade-off exists

• Instance store lives on the physical host, so there is no network hop between the instance and the disk; that is the entire source of its latency and IOPS advantage. The cost of that physical attachment is that the data cannot follow the instance if it stops, is replaced, or the host fails, and it is not covered by EBS snapshots.
• EBS is a separate, replicated storage service the instance attaches to over the network. That network hop is why EBS latency, even on the fastest provisioned-IOPS tiers, will not match local NVMe, but it is also why EBS data survives a stop and restart, supports point-in-time snapshots, and can be detached and reattached to a different instance.

When ephemeral instance-store NVMe is the right call

• Pure scratch space: a shuffle buffer, a temporary cache, or intermediate build or processing files that are cheap to regenerate if lost.
• The absolute lowest latency and highest throughput matters more than persistence, such as staging data locally right before a compute-heavy job processes it.
• Cost-sensitive, high-throughput workloads, where paying for durable, replicated EBS capacity for data you are going to discard anyway is wasted spend.
• Not every instance type includes instance store; it depends on the specific instance family, so this is a constraint to check before designing around it, not an option available everywhere.

When to prefer gp3 or io2 EBS instead

• Anything that has to survive a stop, restart, or instance replacement: databases, application state, or any artifact you cannot afford to regenerate.
• Anything that needs a snapshot for backup, disaster recovery, or cloning to another instance or AWS Region.
• Predictable, provisioned performance with a durability guarantee: gp3 for general steady performance, io2 when the workload needs the highest sustained IOPS with the strongest durability target.

Worked example

A batch analytics job on EC2 shuffles several hundred gigabytes of intermediate data during processing, then writes only the final aggregated result. The intermediate shuffle data goes on instance-store NVMe attached to the instance, since it needs the fastest possible local I/O and is fully disposable if the instance is replaced mid-job, the job would simply restart from its last checkpoint. The final aggregated result, and any checkpoint needed to resume the job, is written to a gp3 EBS volume (or Amazon S3) specifically because that data has to survive even if the instance disappears. Using EBS for the disposable shuffle data would add unnecessary network latency and cost for no durability benefit; using instance store for the final result would risk losing it on any instance interruption, including a Spot Instance reclaim.

Trade-offs and pitfalls

• The most common mistake is treating instance store as free durable storage because it is fast; it has no snapshot mechanism and no protection against instance interruption, which matters more on Spot Instances, where interruption is expected, not exceptional.
• Not every instance family offers instance store, and the ones that do vary in how much local storage they provide, so this decision is partly constrained by which instance types are already in use, not purely a preference.
• Striping multiple instance-store devices together increases throughput but also increases the blast radius of a single device or host failure; know that trade-off before applying it to anything that is not fully disposable.
• A hybrid pattern, using instance store for scratch and periodically checkpointing the durable result to EBS or S3, gets most of the speed benefit without the durability risk, and is the common real-world answer rather than picking one exclusively.

Direct answer

EC2 offers four main purchasing models, and they trade commitment for discount. On-Demand: pay by the hour or second with zero commitment, the most expensive per unit but the most flexible. Reserved Instances: commit to a 1- or 3-year term for a significant discount over On-Demand. Savings Plans: commit to a dollar-per-hour spend for 1 or 3 years, similar discount depth to Reserved Instances but more flexible about which instances the commitment applies to. Spot Instances: bid for AWS's spare capacity at a steep discount, with the risk that AWS can reclaim that capacity with short notice.

Structured elaboration

Why the discount ordering exists: On-Demand carries no commitment risk for AWS, so it costs the most. Reserved Instances and Savings Plans both trade a term commitment for a discount of similar depth; the real choice between them is flexibility (Savings Plans can shift across instance families and even services) versus the narrow but sometimes marginally deeper discount of a Standard Reserved Instance. Spot sits apart from that trade-off entirely: the discount reflects unused capacity risk, not a commitment, which is why it can be steeper than either of the commitment-based options.

Worked example

A fleet mixing Spot for cost, without losing availability, is the standard pattern: run an Auto Scaling group or Spot Fleet with mixed instance types across multiple AZs, using a capacity-optimized allocation strategy so the fleet favors instance pools AWS is less likely to reclaim. Diversifying across several instance types and AZs matters because a single instance type in a single AZ is a much easier target for a capacity reclaim than a fleet spread across many pools. On top of that, workloads should implement graceful shutdown handling for the two-minute Spot interruption notice delivered via instance metadata, checkpointing progress to S3 or a database so an interrupted job resumes rather than restarts from zero, and the fleet should keep a small baseline of On-Demand or Reserved capacity to absorb critical, latency-sensitive traffic that can't tolerate any interruption at all.

Trade-offs & pitfalls

The most common mistake is defaulting to On-Demand for a genuinely steady-state workload, teams under-invest in Reserved Instances or Savings Plans out of a (often outdated) fear of losing flexibility, when Savings Plans in particular were built specifically to remove that rigidity. The opposite mistake is putting a stateful, interruption-sensitive workload on Spot to chase the discount, if the workload can't checkpoint or gracefully hand off in-flight work, an interruption becomes a real outage, not just a cost optimization gone slightly wrong. And treating a Reserved Instance purchase as "set and forget" is a trap too, instance family needs shift over a 1- to 3-year term, which is exactly why Convertible RIs and Savings Plans exist as more flexible alternatives to a Standard RI locked to one configuration.

Direct answer
Operational Excellence means running operations with the same engineering discipline you would apply to the system itself, so operational quality improves continuously instead of depending on any one person's vigilance. AWS's current Well-Architected Framework expresses that idea as eight design principles, listed just below. A real interview conversation rewards going deep on one principle rather than spreading equally thin across all eight, so this answer carries "make frequent, small, reversible changes" all the way through people, process, and automation.

The eight Operational Excellence design principles

• Organize teams around business outcomes: the operating model, ownership, and incentives are structured around the business results a team is accountable for, not around technology silos.
• Implement observability for actionable insights: instrument workloads for performance, reliability, cost, and health, and use that telemetry to act, not just to populate a dashboard.
• Safely automate where possible: define workloads and their operations as code and automate them in response to events, with guardrails such as rate limits, error thresholds, and approvals bounding what the automation can do (this is where the older "perform operations as code" principle now lives).
• Make frequent, small, reversible changes: ship changes at a size and cadence where, if something breaks, reverting is a viable first response instead of a multi-hour rollback exercise. This is the principle this answer goes deep on below.
• Refine operations procedures frequently: treat runbooks as living documents, updated from every incident, not written once at launch.
• Anticipate failure: design and test for component failure before it happens, via chaos testing or game days, instead of discovering failure modes in production.
• Learn from all operational events and metrics: every operational event, not only incidents, feeds back into architecture and process decisions.
• Use managed services: prefer AWS-managed services to reduce the undifferentiated operational burden of running the underlying infrastructure yourself.

(Earlier versions of the framework listed five; AWS has since split and added principles, most visibly folding the older "perform operations as code" principle into the broader "safely automate where possible.")

Structured elaboration: make frequent, small, reversible changes, from people to automation

• Why it's a design principle, not just a deploy-cadence preference: a large, infrequent change bundles many independent risks into one deploy, so when something breaks, both the blast radius and the diagnosis space are large. A small, frequent, reversible change isolates risk to roughly one thing at a time and makes "revert" a viable first response instead of a multi-hour rollback exercise.
• People: engineers own shipping the smallest change that delivers value, not the largest change that's convenient to bundle; release ownership sits with the team making the change, not a separate release-management function, so the person best positioned to revert quickly is the person who shipped it.
• Process: every change ships behind a feature flag or a canary release policy (shift a small percentage of traffic, watch error-rate and latency service level objectives (SLOs) for a fixed window, then proceed or auto-rollback); change-approval overhead scales with blast radius, a config-flag flip needs less ceremony than a schema migration, rather than one uniform change-advisory-board gate for everything.
• Automation: a CI/CD pipeline runs the canary policy automatically; an automated rollback triggers off a monitoring alarm crossing the SLO threshold during the canary window; infrastructure changes are expressed as versioned infrastructure as code, so "revert" means "redeploy the previous known-good version," not "someone remembers what they clicked."

Worked example
A team changes how a checkout service calculates shipping cost. Old way: one large release bundling the new calculation, a UI change, and a database migration, deployed to all traffic at once; when shipping costs come out wrong for a subset of orders, working out which of the three bundled changes caused it, and rolling all three back, takes hours. New way, applying the principle: the database migration ships first and alone, with no user-visible change; the calculation change ships next behind a feature flag, to a small percentage of traffic for a fixed window while a monitoring alarm watches the "shipping cost equals expected value" business metric; only after that canary is clean does it roll out fully; the UI change ships last, independently. If the calculation canary trips the alarm, the automated rollback disables the flag for everyone within the alarm's evaluation window, isolating the failure to exactly the one change that caused it.

Trade-offs & pitfalls

• Small, frequent changes require release infrastructure (feature flagging, canary tooling, fast rollback) up front; a team without that tooling feels the "frequent" part as pure overhead until it's built.
• "Reversible" isn't automatic for schema changes or data migrations: design the migration to be additive and backward-compatible in the same change, so the earlier release remains loadable, rather than assuming reversibility comes for free.
• Not every change decomposes cleanly: an atomic external-contract change, like a hard-cutover API version deprecation, doesn't split into small reversible steps, and the principle's job there is to minimize how often such changes happen, not force-fit them into a canary flow that doesn't fit.