AWS Core Services and Architecture Questions

Direct answer
Granting Account B's analytics role permission to decrypt objects encrypted with an AWS Key Management Service (KMS) key owned by Account A needs two separate, both-mandatory authorizations: the KMS key's own resource-based key policy must name the Account B principal as allowed to use the key, and Account B's IAM identity-based policy must grant that principal kms:Decrypt (plus the S3 read permission). Neither alone is enough. KMS key policies are a deliberate exception to the usual "identity policy OR resource policy" evaluation logic: the key policy is the ultimate gatekeeper for the key, even for principals inside the key's own account, and cross-account calls additionally require the calling account's own IAM policy to allow the action.

Structured elaboration

Why both are required

• For most resources (S3 buckets, for example), the default IAM logic is "allow if EITHER the identity policy OR the resource policy allows it, and nothing explicitly denies it." KMS keys don't follow that default: unless the key policy contains the standard "enable IAM policies" statement (principal = the key-owning account's root, action = kms:*), an allow in an IAM policy is never sufficient by itself, even for a principal in the same account as the key. This stops an IAM admin from silently granting key access without the key owner explicitly signing off in the key policy.
• Cross-account access adds a second, independent requirement: the calling account's own IAM policy must also explicitly allow the action, because a principal can never exceed what its own account's identity policy authorizes, no matter how permissive an external resource policy is.
• Net effect: cross-account KMS access is the AND of three checks: the key policy allows Account B's principal, Account B's IAM policy allows that principal to call KMS, and (since this flows through S3) the S3 bucket policy doesn't carry an explicit Deny that overrides both allows.

Worked example

KMS key policy statement, added by Account A:

{
  "Sid": "AllowAccountBAnalyticsDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::222222222222:role/AnalyticsDecryptRole" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" }
  }
}

IAM policy attached to the role in Account B:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::analytics-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt"],
      "Resource": "arn:aws:kms:us-east-1:111111111111:key/<key-id>"
    }
  ]
}

Trace of one GetObject call: the role assumes in Account B, calls s3:GetObject on the object; S3 internally calls kms:Decrypt using the caller's credentials; KMS checks (1) does the key policy allow this principal, (2) does the principal's own IAM policy allow kms:Decrypt on this key ARN. Both are true here, so decrypt succeeds and S3 returns the object. Flip either one to false and the call fails at the KMS step with AccessDeniedException, even though s3:GetObject itself was permitted.

flowchart LR
    A[Account B: AnalyticsDecryptRole] -->|s3:GetObject| B[S3 bucket, Account A]
    B -->|kms:Decrypt on caller's behalf| C[KMS key, Account A]
    C --> D{Key policy allows Account B principal?}
    D -- No --> X[AccessDenied]
    D -- Yes --> E{Account B IAM policy allows kms:Decrypt?}
    E -- No --> X
    E -- Yes --> F[Decrypt succeeds, object returned]

Trade-offs & pitfalls

• Updating only the IAM policy (skipping the key policy) fails at the KMS step regardless of correct IAM permissions - this trips people up because s3:GetObject otherwise "looks" permitted right up until decryption.
• Updating only the key policy (skipping IAM) also fails: a permissive key policy never bypasses the calling account's own IAM boundary.
• Resource: "*" in the key policy statement grants every KMS action the statement lists across the account boundary, not just decrypt; scope the Action list tightly and keep the kms:ViaService condition so the grant can't be used to call KMS directly outside of S3 operations.
• For access that's meant to be temporary or per-job rather than standing, a KMS grant (kms:CreateGrant) is often a better fit than widening the key policy, since a grant can be revoked individually without editing the key policy.

Direct answer
The security delta between Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS) in a regulated environment is mostly about where AWS-native controls plug in, not which platform is inherently more secure: ECS grants AWS permissions to workloads via a task role (one IAM role per task definition) and integrates natively with Amazon Elastic Container Registry (ECR) image scanning; EKS gives you the same image-provenance controls plus Kubernetes-native admission control, and grants AWS permissions to pods via IAM Roles for Service Accounts (IRSA) or the newer EKS Pod Identity feature, both scoped to a Kubernetes service account rather than a whole node. The bigger axis for a regulated, kernel-module-requiring workload is actually compute platform, not orchestrator: AWS Fargate and AWS Lambda run on AWS-managed microVMs with no exposed kernel to load a module into, so a workload that genuinely needs a custom kernel module has to run on self-managed Amazon EC2 (with or without ECS/EKS layered on top), never on Fargate or Lambda.

Structured elaboration

1. Image provenance / supply chain (same practice on both platforms)

• Enable ECR scan-on-push so every pushed image gets a vulnerability scan before it's eligible to run, and require image signing (for example, via Cosign) plus a generated Software Bill of Materials (SBOM) in the CI pipeline; gate deployment on both a passing scan and a valid signature, not just whatever tag exists in ECR.
• EKS adds a natural enforcement point ECS lacks: a Kubernetes admission controller (Open Policy Agent Gatekeeper or Kyverno) can reject any pod spec referencing an unsigned or unscanned image at admission time, inside the cluster. ECS has no equivalent native admission hook, so that gate has to live entirely in CI/CD, before anyone calls RegisterTaskDefinition or UpdateService.

2. Granting AWS permissions to workloads

• ECS task role: one IAM role per task definition; every container in that task shares the same role's permissions (task-level granularity).
• EKS IRSA: maps a Kubernetes service account to an IAM role via the cluster's OIDC identity provider; a pod using that service account gets credentials scoped to exactly that role, at pod granularity, without relying on node-level IAM permissions.
• EKS Pod Identity: AWS's newer, simpler alternative to IRSA that removes the OIDC-provider setup step, associating an IAM role with a service account directly through EKS, with credentials delivered by a Pod Identity Agent DaemonSet on each node. It grants the same pod-scoped access as IRSA; new EKS deployments should default to Pod Identity, but existing IRSA setups don't need to migrate for a security benefit, both remain supported.
• Both models beat relying on the worker node's own EC2 instance role in a regulated environment, because a node-level role is implicitly reachable by every pod scheduled on that node unless Instance Metadata Service (IMDS) access is explicitly locked down.

3. Compute-platform isolation, the axis that decides "can this even run here"

4. Runtime hardening once you're on EC2-backed compute (ECS-on-EC2 or EKS-on-EC2): seccomp profiles, dropped Linux capabilities, read-only root filesystems where the workload allows it, and behavioral runtime monitoring to catch container-escape attempts, since a shared kernel means an escape reaches every other workload on that host, not just its own task or pod.

Worked example
A regulated workload needs a custom eBPF-based network kernel module for deep packet inspection. Working through the table: Lambda and both Fargate options are eliminated immediately on the kernel-module requirement alone. That leaves EC2 self-managed, ECS-on-EC2, or EKS-on-EC2. Because the organization already runs Kubernetes-native policy tooling and wants IRSA/Pod-Identity-scoped AWS access per workload rather than one task role per definition, EKS-on-EC2 with a dedicated, tainted node group (so only this workload schedules there, bounding the shared-kernel blast radius) is the concrete choice, with ECR scan-on-push plus signature verification enforced at admission by Gatekeeper before any pod referencing that image can schedule.

Trade-offs & pitfalls

• Don't conflate "EKS supports IRSA/Pod Identity" with "EKS is more secure than ECS" - in a regulated environment, the decision that matters most is almost always the compute-platform row (Fargate/Lambda's managed isolation versus EC2's operational burden), not the orchestrator.
• A dedicated node group with a taint only limits scheduling; it doesn't stop a privileged container from reaching the underlying kernel. Combine it with seccomp/capability drops, it isn't a substitute for them.
• IRSA still works and is well understood; treat migrating existing IRSA workloads to Pod Identity as a "prefer for new work" recommendation, not an urgent deprecation.
• Enforcing image signing only in CI (ECS's situation) is bypassable by anyone with ecs:RegisterTaskDefinition permission who skips the pipeline; EKS's admission-controller gate is the stronger control precisely because it's enforced at the cluster, not only at the pipeline.

Direct answer

AWS Key Management Service (KMS) is the managed service that creates, stores, and controls cryptographic keys. AWS now calls these KMS keys (you'll still hear the older term CMK, customer master key, used interchangeably). A KMS key is either symmetric (one key used for both encrypt and decrypt, the default for almost everything) or asymmetric (a public/private key pair, used for signing or for cases where encryption has to happen outside KMS). Separately, a key can be AWS managed (AWS creates and rotates it for you, you get no policy control) or customer managed (you own the key policy, rotation schedule, and lifecycle). Access to a key is governed first by its key policy, a resource policy attached directly to the key, and only within what that policy allows can AWS Identity and Access Management (IAM) policies grant identities permission to use it.

Structured elaboration

Key types

• Symmetric keys: AES-256, used for the vast majority of workloads (S3 SSE-KMS, EBS, RDS, EFS). KMS never lets the symmetric key material leave the service; you always call KMS to encrypt or decrypt.
• Asymmetric keys: RSA or elliptic-curve pairs, used for digital signatures or when a public key must be shared with a system outside AWS. Slower and not meant for bulk data.
• Neither key type rotates the same way: automatic annual rotation is available for customer managed symmetric keys as an opt-in setting; asymmetric and HMAC keys are rotated manually, by creating a new key and repointing an alias.

Key ownership tiers

Key policies vs IAM policies: the key policy is the resource-level gate and is evaluated first; if it doesn't allow an action, no IAM policy can override that. Most customer managed keys use a key policy that delegates day-to-day permission management to IAM (a policy statement enabling the account's IAM policies), so you don't have to touch the key policy for every new role, only for cross-account or unusual grants.

Envelope encryption: services like S3, EBS, and RDS don't send your bulk data to KMS. They call GenerateDataKey to get a plaintext data key plus an encrypted copy, encrypt the data locally with the plaintext key, discard the plaintext key, and store only the encrypted data key alongside the ciphertext. This is why KMS-backed encryption scales to large objects without becoming a network bottleneck.

Grants: a lighter-weight, revocable way to hand a specific principal or service permission to use a key for defined operations, without editing the key policy. Grants are how many AWS service integrations, and cross-account or cross-service delegation, work under the hood.

A distinct, easily-confused option: SSE-C. S3 also supports server-side encryption with customer-provided keys (SSE-C), where you supply your own raw AES-256 key on every request and AWS never stores it at all. That is a different model from a customer managed KMS key: with SSE-C, if you lose the key, AWS cannot help you recover the data, and there's no KMS audit trail because KMS isn't involved.

Encryption options across storage services

Cross-account access: the key's owning account must add a key policy statement naming the external account or role ARN (Amazon Resource Name, AWS's unique identifier string for a resource) and the allowed actions (e.g., kms:Decrypt), and the calling account's IAM policy must separately allow that same action on the key's ARN. Both sides have to agree; missing either one fails closed.

Multi-account separation of duties: a common pattern is a dedicated security/key-management account that owns all customer managed keys. The key policy grants a narrow "key user" role to workload accounts (encrypt/decrypt only) and a separate "key administrator" role, kept out of the workload accounts entirely, for managing the key policy and lifecycle. That way no single workload-account operator can both encrypt data and change who's allowed to decrypt it, and CloudTrail in the central security account gives one place to audit every key use across the organization.

Data residency: KMS keys are regional by default, the key material never leaves the region it was created in. AWS also offers multi-Region keys, a set of replica keys sharing the same key material across regions, purely to support disaster-recovery re-encryption without touching ciphertext. If the requirement is data residency or sovereignty rather than DR, that's a reason to deliberately avoid multi-Region keys and keep a single regional key, since the whole point of a residency requirement is that the key material must not exist anywhere else.

Worked example

Account A (a workload account) needs to decrypt S3 objects encrypted with a customer managed key that lives in Account B (the security account). Both sides need an explicit statement. The key policy in Account B must include something like:

{
  "Sid": "AllowWorkloadAccountDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111111111111:role/workload-read-role" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*"
}

And the IAM policy attached to workload-read-role in Account A must independently allow:

{
  "Effect": "Allow",
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "arn:aws:kms:us-east-1:222222222222:key/EXAMPLE-KEY-ID"
}

If either statement is missing, the decrypt call fails, which is exactly the separation-of-duties property the multi-account pattern is designed to give you: Account A's own admins cannot grant themselves that access unilaterally.

Trade-offs & pitfalls

The most common trap is forgetting that key policy and IAM policy are both required, an IAM policy alone is not sufficient, which shows up as confusing access-denied errors when only one side was updated. A second common decision point is one shared key versus one key per workload or data classification: a single customer managed key is simpler to audit but concentrates blast radius (a compromised principal with decrypt access can read everything protected by that key), while per-workload keys shrink blast radius at the cost of more policies to manage and, at very high request volumes, more KMS API calls that can approach service request-rate limits. Deleting a customer managed key is intentionally hard to reverse quickly, KMS enforces a waiting period before actual deletion, specifically because losing the key means losing every object it protects; that pending window is a safety net, not a formality to route around. Finally, using an asymmetric key for bulk data encryption is a real anti-pattern seen in the field, it's slower and not what asymmetric keys are designed for; reach for envelope encryption with a symmetric key instead.

Direct answer. Use envelope encryption: Lambda calls KMS GenerateDataKey against a customer managed KMS key (AWS Key Management Service; the key that owns the encryption policy) owned by the security or data account, encrypts the payload locally with the returned plaintext data key, discards the plaintext key immediately, and stores only the ciphertext data key next to the encrypted record in DynamoDB or RDS (Relational Database Service). The analytics account never receives a standing IAM (Identity and Access Management) permission on the KMS key itself. Instead it is issued short lived, narrowly scoped KMS grants that permit Decrypt on demand, so the trust boundary lives at the grant's lifecycle rather than at a broad key policy or cross account IAM role.

Why envelope encryption instead of calling KMS directly

KMS Encrypt/Decrypt on a symmetric key caps the plaintext at 4,096 bytes (4 KB), so calling KMS directly per record does not work once payloads exceed that size, and even for small payloads it puts a network round trip to KMS on the request's critical path for every read and write. Envelope encryption removes both problems: KMS is only called once per record to mint a data key, the actual AES-GCM encrypt/decrypt of the payload happens locally at whatever size the payload is, and only the small ciphertext data key (not the payload) ever touches KMS again on decrypt.

End to end flow

flowchart LR
    Client[External client] -->|HTTPS| APIGW[API Gateway]
    APIGW --> Lambda[Ingress Lambda]
    Lambda -->|GenerateDataKey| KMS[(KMS key\nsecurity account)]
    KMS -->|plaintext key + ciphertext key| Lambda
    Lambda -->|AES-GCM encrypt payload| Store[(DynamoDB / RDS\nciphertext + ciphertext key)]
    Analytics[Analytics account job] -->|assume role, use grant| KMS
    KMS -->|Decrypt ciphertext key| Analytics
    Analytics -->|decrypt payload locally| Store
    Broker[Grant broker service] -->|CreateGrant / RetireGrant| KMS

1. Ingress. API Gateway invokes the Lambda function. The Lambda calls kms:GenerateDataKey on the KMS key (AWS's current term for what older documentation and many engineers still call a Customer Master Key, or CMK; both names refer to the same customer managed key resource). KMS returns a plaintext data key and its ciphertext copy in one call.
2. Local encryption. The Lambda uses the plaintext data key to AES-GCM encrypt the record, then zeroes the plaintext key from memory. Only ciphertext payload and ciphertext data key are written to DynamoDB or RDS. No plaintext key is ever persisted or logged.
3. Cross account read. When the analytics account's job starts, a grant broker (a small Lambda or Step Functions task running in the security account) calls kms:CreateGrant, naming the analytics job's IAM role as GranteePrincipal, restricting Operations to Decrypt (and DescribeKey if the client needs to introspect key metadata), and setting a RetiringPrincipal so the broker itself can retire the grant when the job ends.
4. Decrypt. The analytics job reads the ciphertext data key from the record, calls kms:Decrypt (authorized purely by the grant, no key policy edit required), gets back the plaintext data key, and decrypts the payload locally.
5. Teardown. The broker calls RetireGrant (or RevokeGrant from the security account) once the job window closes. Because a grant is a distinct, listable object (ListGrants), it can be issued and torn down per job without ever touching the KMS key's key policy or IAM policies, which is what keeps the blast radius narrow: a leaked analytics role credential is scoped to whatever grants are currently outstanding, not to the key itself.

aws kms create-grant \
  --key-id arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
  --grantee-principal arn:aws:iam::444455556666:role/AnalyticsJobRole \
  --retiring-principal arn:aws:iam::111122223333:role/GrantBrokerRole \
  --operations Decrypt DescribeKey \
  --name analytics-job-2026-07-20

Classification, retention, and audit trail for the data itself

Envelope encryption controls who can read the data; it says nothing about how long it should exist or who touched it when, and those are the questions a reviewer for personally identifiable information (PII) will ask next:

• Classification. Tag each field or record at write time (a classification attribute such as pii-high, pii-low, internal) so downstream consumers, including the analytics job, can filter on sensitivity before deciding whether a grant is even appropriate for that data slice, and so the key policy or grant constraints can be scoped per classification tier rather than treating all rows uniformly.
• Retention. Attach a retention rule to the classification: DynamoDB Time To Live (TTL, an attribute that expires and deletes items automatically) or an RDS scheduled purge job for the raw ciphertext, separate from any retention period required for the audit trail itself. Encryption is not a substitute for deletion; PII that no longer has a business purpose should not still be sitting around merely because it is encrypted.
• Audit trail. Every GenerateDataKey, Decrypt, CreateGrant, and RetireGrant call emits an AWS CloudTrail event, logged in the security account by default since KMS is the resource owner. Centralize these events (CloudTrail organization trail or a dedicated log account) and alert on anomalies: a Decrypt volume spike, a Decrypt from a role or region that never normally calls it, or a grant that outlives its expected job window. Pair this with an application level audit table (job id, grant id, requester, record ids touched) so "who read record X and when" is answerable without reconstructing it from raw CloudTrail JSON.

Trade-offs and pitfalls

• Grants vs key policy cross account statements. A permanent key policy statement granting the analytics account kms:Decrypt is simpler to set up but is always-on: a compromised analytics credential can decrypt anything, anytime. Grants cost more operational machinery (a broker, a lifecycle to manage) but bound the exposure window to the job's actual runtime.
• Data key caching. Calling GenerateDataKey per record at high write volume adds KMS request cost and can approach per key request quotas under bursty traffic; a data key caching library (encrypt many records under one data key, rotate the data key periodically) trades a slightly larger blast radius per data key for far fewer KMS calls. This is a real trade-off to name explicitly, not a free win.
• Symmetric vs asymmetric KMS key. Symmetric is the right default here for performance and because grants and envelope encryption are built around it; asymmetric keys exist for signing or when the decrypting party must never call KMS with AWS credentials, which is not this scenario.
• Forgetting to revoke. A grant with no explicit TTL and no reliable retire step is functionally a standing permission with extra steps. The broker retiring (or revoking) the grant is not optional cleanup, it is the entire point of using a grant instead of a policy statement.
• Encryption context. Passing an EncryptionContext (a non-secret key-value pair, for example {"table": "patient-records"}) on both GenerateDataKey and the later Decrypt adds authenticated context that must match exactly, which stops a ciphertext data key from one table being replayed against a different table's decrypt path.

Direct answer

Enforce least privilege operationally, not just by writing tight policies once. Default every non-human principal to a role assumed through STS (AWS Security Token Service) rather than a long-lived IAM user, cap what any role can ever have with permission boundaries (and Service Control Policies at the AWS Organizations level), require MFA for all human console/CLI access, rotate what long-lived credentials remain, and continuously audit actual usage against granted permissions with IAM Access Analyzer and CloudTrail so drift gets caught automatically instead of at the next annual review.

Structured elaboration

Users vs roles

• Reserve IAM users for the rare cases where a human genuinely needs a persistent identity (and even then, prefer federated access through IAM Identity Center where possible).
• Everything else (services, cross-account access, CI/CD) should assume a role. Use instance/task/execution roles for compute, and cross-account IAM roles for service-to-service or partner access.

Temporary credentials (STS)

• Prefer sts:AssumeRole / sts:AssumeRoleWithWebIdentity for all non-root access so credentials expire on their own and the blast radius of a leak is bounded by the session duration.
• Set a short MaxSessionDuration on sensitive roles, and use session tags to carry request context into CloudTrail for later correlation.

Permission boundaries and SCPs

• A permission boundary is a managed policy attached to a user or role that sets the maximum permissions any policy attached to that principal can ever grant, no matter what gets attached later. It never grants anything by itself.
• An SCP, applied at an AWS Organizations account or organizational unit, works the same way at the account level: it restricts what can ever be allowed within that account or OU, and also never grants permissions on its own.
• Together they form two independent ceilings: use SCPs to guardrail entire accounts (deny iam:CreateUser, deny leaving a set of approved regions), and permission boundaries to guardrail individual automation or developer roles within an account.

MFA and root protection

• Require MFA for all human IAM and IAM Identity Center sign-ins, enforced with the aws:MultiFactorAuthPresent condition key on sensitive actions.
• Remove the AWS account root user's access keys entirely, enable MFA on the root user, and treat root sign-in as a rare, logged, break-glass action rather than a routine one.

Rotation and hygiene

• Where long-lived credentials must exist at all (a legacy integration that can't assume a role), rotate them on a defined cadence and automate the deletion of keys that IAM Access Analyzer or Access Advisor shows as unused.

Continuous audit

• Use CloudTrail as the ground truth of what actually happened, IAM Access Analyzer to surface external/cross-account access and generate least-privilege policy suggestions from observed activity, and aws iam simulate-principal-policy to test whether a proposed policy actually allows or denies a specific action before you ship it.
• Treat any principal whose granted permissions are visibly wider than its observed usage (via Access Advisor's last-accessed data) as a finding to remediate, not a one-time cleanup.

Worked example

A permission boundary that caps a developer role to a small, named set of services regardless of what gets attached to it later, paired with a CLI check that the boundary actually blocks an out-of-scope action:

// Permission boundary attached to DeveloperRole
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:*", "dynamodb:*", "logs:*", "cloudwatch:*"],
    "Resource": "*"
  }]
}

Even if someone later attaches an AdministratorAccess-style policy to DeveloperRole by mistake, the boundary still caps effective permissions to the four services listed. You can verify this before it ever gets tested in production:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111122223333:role/DeveloperRole \
  --action-names ec2:TerminateInstances \
  --resource-arns "*"
# Expected result: implicitDeny, because ec2 is outside the boundary

Trade-offs and pitfalls

• SCPs and permission boundaries only restrict; a very common confusion is expecting either one to grant access. If a principal has no explicit Allow in its identity-based (or resource-based) policy, an SCP or boundary that permits the action changes nothing.
• An overly strict SCP can silently break AWS itself if it blocks iam:CreateServiceLinkedRole, which several AWS services need in order to function; scope denies carefully rather than denying broad IAM namespaces wholesale.
• MFA enforced via a condition key applies to how the session was established, not to every subsequent API call inside that session, so pair MFA requirements with short session durations rather than treating MFA alone as a per-call control.
• Continuous auditing only works if CloudTrail is actually enabled organization-wide and retained; a control that exists on paper but isn't wired into monitoring gives a false sense of coverage.