Amazon Staff Cloud Architect Interview Preparation Guide

Situation & Goals (one line)
Design an enterprise policy and technical system to detect, notify, approve, and automatically retire unused cloud resources with zero/low business disruption and strong auditability.

Detection heuristics (metrics & thresholds)

• Compute (VM/VMSS): CPU < 2% and network I/O < 1 MB/day for 30 consecutive days.
• Block storage: Read/Write ops < 100/day and last snapshot/access > 30 days.
• Load balancers/NAT: No backend healthy checks and 0 connections for 14 days.
• Databases: Connections = 0 and read/write < threshold for 30 days; last backup retention verified.
• IAM keys: unused for 90 days.
  Thresholds configurable by resource tag (env, criticality).

Exemption & approval rules

• Auto-exempt resources with tags: owner, business-critical=true, retention-window, legal-hold.
• Owners can request temporary exemption via ticketing system (TTL'd).
• Default path: detection → owner notification → 7-day soft-quarantine → owner approval to retain.
• Escalation: if no response, escalate to app SLA owner, then to infra steward before retirement.

Notification workflow

• Multi-channel: email + Slack + ITSM ticket (Jira/ServiceNow) with actionable buttons: “Retain 30d”, “Schedule Retirement”, “Mark Exempt”.
• Notifications include resource metadata, last activity, cost impact, rollback playbook link.

Automated retirement & rollback

• Two-phase: 1) Quarantine (isolate network, snapshot/backups, change billing tag) 2) Delete after snapshot + approval window.
• Pre-retirement: create immutable snapshot/s3 export, export infra-as-code (Terraform state), mark in audit log.
• Rollback: restore from snapshot or redeploy from IaC; automated runbook tested weekly; SLA: RTO target defined per class.

Audit trails & compliance

• Immutable audit logs (CloudTrail/Stackdriver + centralized SIEM) capturing detection, notifications, approvals, snapshots, and deletion events.
• Retention of logs and snapshots per policy; signed attestations stored in compliance repo.

Measuring effectiveness & safety

• KPIs: % cost reclaimed, false-positive rate (incidents attributed to retirement), mean time to recover (MTTR) after rollbacks, % resources in quarantine vs retired.
• Safety controls: conservative thresholds, owner-first flows, required snapshot before delete, canary rollouts (pilot on non-prod), automated unit tests for rollback.
• Continuous tuning: weekly review of false-positives, A/B test threshold changes on a subset, and quarterly policy review with stakeholders.

Implementation stack (example)

• Detection: Cloud native metrics (CloudWatch/Stackdriver) + CDM/Cost APIs + custom Lambda/Cloud Function analyzer.
• Orchestration: Step Functions/Workflows to manage notifications, approvals, snapshots, deletion.
• UI & Tickets: ServiceNow/Jira integration, Slack/email.
• IaC integration: Terraform/CloudFormation for restore.
• Observability: SIEM + dashboards (Grafana) for KPIs.

This design balances automation and human control to optimize costs while minimizing accidental outages.

Approach summary
Design a centralized, multi-cloud observability backbone that uses regional collectors to meet data residency, tiered storage for cost, and open standards (OTLP/OTEL, Prometheus, OpenTelemetry traces, OTLP/HTTP/gRPC). Use buffering and fan-in to central processing with tenant-aware pipelines, enforce sampling/aggregation, and provide RBAC + encryption.

Ingestion & buffering

• Regional agent/collector per cloud region (OTel Collector) -> validates, tags (region, tenant, environment), applies local PII redaction.
• Buffering: local disk + cloud-native queues (AWS Kinesis / Azure Event Hubs / GCP Pub/Sub) per region for durability and backpressure.
• Fan-in to central processing via managed Kafka or MSK/Confluent for cross-region aggregation when policy allows.

Processing & pipelines

• Streaming jobs (Flink/Beam/Kafka Streams) for enrichment, sampling, metric rollups, and index routing.
• Traces go to Tempo/Jaeger backend; spans sampled and probabilistically retained; error/high-latency traces forced retention.
• Metrics to Cortex/Thanos with local Prometheus scraping; push via remote_write; use pre-aggregation for high-cardinality metrics.
• Logs to OpenSearch/Elasticsearch for hot indexing + object store (S3/Blob/GS) for cold tier.

Storage tiers & retention

• Hot (0–7 days): fully indexed OpenSearch + blockstore for fast queries.
• Warm (7–90 days): reduced replicas, sparse indexing, lower-cost nodes.
• Cold (90–365+ days): compressed parquet/ndjson in object storage with query via Athena/OpenSearch frozen indices.
• Traces: Hot (14 days indexed), Cold (90+ days in object storage, sampled).
• Metrics: High-res (1m) 15–30 days, downsampled 5m/1h up to 365 days.

Indexing & query patterns

• Index by tenant, region, service, timeframe; use time-based indices and ILM.
• Use inverted indices for logs; metrics in TSDB; traces indexed by trace_id, service, error tag.
• Support cross-region queries only when policy allows; otherwise, query regional data plane and federate results.

Alerting & SLO enforcement

• SLOs defined in central catalog (backed in GitOps). Export SLOs to Prometheus recording rules and Cortex/Thanos for evaluation.
• Alertmanager with dedup and silence rules; route alerts by team and severity using webhooks/PagerDuty.
• Automated remediation runbooks and rate-limited alerting to reduce noise.

Multi-tenant access controls

• Tenant isolation via index/namespace separation, RBAC in UI (Grafana/OpenSearch Dashboards), and API gateways.
• Encryption at rest + in transit; key management via KMS per region/tenant as required.
• Tenant quotas (ingest rate, storage) enforced at collector and broker layers.

Cost & scale controls

• Ingest-side controls: sampling, adaptive sampling for traces, aggregation and metric rollups, client-side batching.
• Storage lifecycle policies, compression, and use of object storage for long tail.
• Prefer managed services where ops-cost outweighs infra-cost; use spot/ephemeral nodes for batch/analytics.
• Autoscale ingestion/processing with backpressure into queues; implement circuit breakers and hard quotas to prevent runaway costs.
• Regular chargeback/showback, cost alerts, and usage caps per tenant.

Trade-offs

• Stronger residency and isolation increases duplication and cost; mitigate via federated query and selective cross-region replication.
• Open-source stack reduces licensing but increases operational overhead; balance with managed offerings for scale.

This architecture balances compliance, performance and cost by enforcing policies at the edge, tiering storage, and controlling cardinality and retention.

Overview — goal
I would evaluate vendors with a governance checklist that turns qualitative controls into measurable scores so the board can compare risk, compliance fit, and operational maturity.

Checklist (grouped)

• Security (25%)
  • Data encryption at rest/in transit; KMS control + HSM support
  • Identity: MFA, SSO, SCIM, least-privilege IAM
  • Network controls: VPC, private endpoints, WAF, DDoS mitigation
  • Security tooling: CSPM, secrets manager, logging, EDR integration
• Compliance & Audit (20%)
  • Certifications: SOC2, ISO27001, PCI-DSS, regional (e.g., FINRA, GDPR)
  • Audit logs retention, exportability, e-discovery support
  • Data residency controls and contractual audit rights
• Global-region support & Resiliency (15%)
  • Regions, AZs, sovereign clouds, cross-region replication, latency SLAs
  • Disaster recovery patterns, runbooks, demonstrated RTO/RPO
• Platform Services Maturity (15%)
  • Managed DBs, K8s, serverless, CI/CD, monitoring — maturity and SLAs
  • Marketplace & partner ecosystem, documented best practices
• SLA / SLT & Support (15%)
  • Financial SLA terms, uptime %, penalty structure, support tiers, escalation paths
  • Maintenance windows, change notification lead times
• Vendor-lock-in Risk (10%)
  • Open standards support, data export tooling, service portability, API stability

Scoring approach

• For each sub-item score 0–5 (0 = none, 5 = enterprise-best-practice). Aggregate weighted average per category.
• Define pass/fail thresholds: >4.5 = Preferred, 3.5–4.5 = Acceptable with mitigations, <3.5 = High risk.

Weighting rationale

• Security & Compliance highest because financial services are regulated and breach impact is critical.
• Global resiliency and platform maturity drive availability and long-term operability.
• SLAs and lock-in get lower weight numerically but are decisive in negotiation and migration planning.

Deliverable
A spreadsheet with weighted scoring, evidence links, mitigations per low score, and a recommended vendor with an implementation risk register.

Direct answer

Start with showback, not chargeback: showback means teams see an itemized bill with no money actually moving, chargeback means it debits their real budget, and you should only flip that switch once the allocation logic is trusted. For shared infrastructure, allocate in a strict order of preference: direct attribution wherever a resource can be tied to one owner, proportional allocation by measured usage where several teams share a resource, and a pooled or even-split fallback only for the genuinely unattributable remainder, shrinking that fallback bucket over time as tagging improves. Build the dispute process and an anti-gaming control before you turn chargeback on, because that's what determines whether teams treat the bill as legitimate rather than as something to game or ignore.

Structured elaboration

Allocation rule hierarchy

Dispute-resolution flow

flowchart TD
    A[Usage events] --> B[Normalize and enrich with owner, cost center, tags]
    B --> C[Apply allocation rules and rate card]
    C --> D[Generate bill line items]
    D --> E[Publish provisional showback dashboard]
    E --> F{Dispute filed?}
    F -->|No| G[Finalize invoice]
    F -->|Yes| H[Dispute workflow: review usage events and allocation]
    H --> I[Issue correction: credit or debit memo]
    I --> G

Every line item should be clickable back to the underlying usage events and the allocation method that produced it, an SLA (service-level agreement) of acknowledging a dispute within 48 hours and resolving within about two weeks, and every correction recorded in an immutable audit log referencing the original line item, so a dispute doesn't quietly change history.

Anti-gaming controls

• Mandatory tagging enforced at provisioning time (a resource can't be created without an owner tag); the enforcement mechanism itself, whether that's a policy-as-code check wired into CI/CD (continuous integration/continuous delivery), belongs to your infrastructure and platform engineering practice, not FinOps, but FinOps owns defining what the rule requires.
• Anomaly detection on sudden usage surges, tag mismatches, or unusual cross-team resource moves, since a team gaming the system to dodge its bill often shows up as one of these patterns.
• Threshold-based approval: allocations that jump sharply month over month require sign-off before they're finalized, catching both genuine spikes and manipulation.
• Periodic recomputation from raw immutable usage events, so a team can't quietly benefit from a stale or manually-edited allocation record.

Getting engineering and finance to adopt it

Run showback for at least one full billing cycle before any money moves, so teams can question and fix their own numbers without a budget consequence attached. Get finance and engineering leadership to co-sign the rate card and allocation methodology up front, not after teams start disputing bills, and revisit that rate card on a fixed cadence (quarterly is reasonable) so it doesn't quietly drift from actual infrastructure cost and become a fight at renewal.

Variants this same hierarchy covers

For a multinational organization invoicing in multiple currencies, add an FX (foreign exchange) conversion step using a rate locked for the billing period, so a team's bill doesn't move purely because of currency swings that have nothing to do with its usage. For a shared, multi-tenant machine learning platform, the same direct-attribution-first, proportional-fallback hierarchy applies at the experiment level: GPU-hours (graphics processing unit hours) per training run are usually directly attributable, while shared orchestration and platform overhead gets pooled and split proportionally, same as any other shared service.

Worked example

Suppose a shared platform costs $60,000 this month and three teams' measured usage (in vCPU-hours) was Team A at 500,000, Team B at 300,000, and Team C at 200,000, for a total of 1,000,000 vCPU-hours. Proportional allocation gives:

$\text{Allocation}_i = \text{SharedCost} \times \frac{\text{Usage}_i}{\sum_j \text{Usage}_j}$

• Team A: $60{,}000 \times 500{,}000/1{,}000{,}000 = \$30{,}000$
• Team B: $60{,}000 \times 300{,}000/1{,}000{,}000 = \$18{,}000$
• Team C: $60{,}000 \times 200{,}000/1{,}000{,}000 = \$12{,}000$

Team C disputes its $12,000 line item, claiming its actual usage was closer to 150,000 vCPU-hours because of a metering gap during a deployment window. The dispute workflow pulls the raw usage events for Team C for that period, finds a five-hour metering outage that undercounted roughly 40,000 vCPU-hours of Team B's usage instead (a shared node was mislabeled), corrects the input usage figures, and reruns the same proportional formula. The correction is posted as a credit to Team C and a debit to Team B on the next invoice, both referencing the original line item and the metering-outage ticket in the audit log, not silently edited into the historical record.

Trade-offs and pitfalls

• Strict direct attribution reduces disputes but increases tagging friction; teams will push back on the overhead unless provisioning tools make tagging closer to free.
• The even-split fallback looks fair but actively incentivizes not tagging, since an untagged resource costs less per unit than one directly and expensively attributed; cap how much cost can flow through that bucket and drive it down over time rather than treating it as a permanent category.
• Turning on chargeback before the dispute SLA and audit trail exist erodes trust immediately, and trust lost in the first billing cycle is expensive to rebuild.
• A rate card set once and never revisited drifts from reality, so what started as a reasonable allocation methodology becomes a recurring argument at each budget cycle instead of a settled mechanism.

Direct answer

Once you scale past a handful of instances, in-memory session state stops working: sticky sessions (routing every request from a user to the same backend instance) keep working but make the fleet fragile and hard to rebalance, while externalizing session state to a shared store (Redis, DynamoDB, or similar) makes every instance stateless at the cost of a network hop and a new piece of infrastructure to run. For most cloud-native, multi-region systems the externalized store wins once you account for failover and elastic scaling, but the right choice depends on session size, read/write rate, and how much latency and operational cost you can absorb.

Structured elaboration

Sticky sessions (load-balancer affinity)
The load balancer hashes a client identifier (cookie or IP) to consistently route it to one backend instance, which keeps the session data in that instance's local memory. This is the load-balancer mechanism; the algorithm and health-check details behind it belong to load-balancing design, not to this decision. What matters here is the state-management consequence: the instance becomes a single point of failure for every session pinned to it.

Externalized session store
The application writes session data to a shared store instead of local memory, so any instance can serve any request. This is what makes the fleet stateless and lets you autoscale, blue/green deploy, and route traffic freely.

Trade-off table

Two problems specific to externalizing state that a senior candidate should raise

1. Data locality/regulatory constraints on where the store lives. Once session state leaves the process and lives in a shared store, where that store is physically located becomes a compliance question in a way it never was for sticky sessions. If you operate in a jurisdiction with a data-residency requirement (for example, keeping EU user data on EU infrastructure), the session store's region placement is now something you have to design around explicitly: either run per-region session stores keyed so a user's session never leaves their region, or make sure the store you pick supports region-pinned partitions. This is a design constraint, not an afterthought bolted on after the store is chosen.
2. Token revocation/invalidation once you drop server-side sticky sessions. With sticky sessions, "logging a user out" or forcibly invalidating a session is trivial: the state lives in one place and you delete it there. Once you move to a stateless architecture backed by client-held tokens (JWTs, JSON Web Tokens), you lose that for free, because a JWT is self-contained and valid until it expires, regardless of what the server thinks. You need an explicit revocation mechanism: keep a server-side allow/deny list (defeats some of the point of a stateless token, but is often still cheaper than full session storage), keep JWT lifetimes short and pair them with a refresh-token that IS checked against the store on each renewal, or store only a session identifier in the token and look up the actual session state (and its revocation flag) in the externalized store on every request. Whichever you pick, decide it up front: it is the first thing that breaks in incident response ("how do we kill this user's access right now") if you didn't.

Worked example

A mid-size SaaS product runs 40 stateless application instances behind a load balancer in two regions (US and EU) for latency and the EU data-residency requirement above. Session state (user id, cart contents, feature flags, roughly 2 KB per session) is stored in a regional Redis cluster with cross-region replication disabled by design, so an EU user's session data never crosses into the US store. Each app instance reads the session on request start and writes it back on request end; a local, short-time-to-live (TTL) cache in front of Redis (5 seconds) absorbs the read traffic for users making several requests per second, so the added latency in the common case is effectively the local cache lookup, not a Redis round trip. Auth uses short-lived JWTs (a few minutes) plus a longer-lived refresh token that is checked against the Redis-backed session record on every refresh; revoking a user (support ticket, password reset, or a fraud flag) sets a revoked flag on that record, so their access token stops renewing once it expires and, for the small number of write-sensitive endpoints, is checked directly against the record on every call. This gives the team elastic autoscaling and blue/green deploys without cold-session churn, at the cost of running and monitoring two regional Redis clusters instead of zero.

Trade-offs & pitfalls

• Reflexively "always externalize" is a mistake for small, low-scale, or lift-and-shift systems where the operational cost of running a session store isn't yet justified; sticky sessions with a short-TTL fallback can be the pragmatic starting point.
• Under-provisioning the session store's replication is the most common failure: teams externalize state assuming it fixed failover, then discover the store itself is a new single point of failure because they ran it as a single node.
• Storing large or unbounded session payloads in the shared store (shopping-cart history, uploaded-file references) rather than keeping sessions small and pushing bulk data to its own store is a frequent scaling mistake that shows up as store memory pressure and slow serialization, not as an obvious session bug.
• Treating token revocation as solved by "just make JWTs short-lived" without also handling the case where a token must be killed immediately (compromised account) is a common gap; short expiry reduces the blast radius but does not give you an immediate kill switch on its own.

Overview / goal
Design a secure, scalable internal DNS using Route 53 Private Hosted Zones (PHZ) across multiple AWS accounts/VPCs to support environment separation (prod/stage) and reliable service discovery for ephemeral endpoints.

Split-horizon DNS

• Use split-horizon to expose different records externally vs internally:
  • Public zone: example.com (Route 53 public PHZ) for internet-facing services.
  • Private zones: example.internal or env-specific private zones for internal resolution.
• Internally, clients resolve internal names to private IPs/NLBs; externally, names either don’t exist or resolve to public endpoints.

PHZ association across accounts/VPCs

• Create PHZ in a central networking/account (or per environment).
• Share PHZ with other accounts via AWS RAM (recommended for Organizations) or use cross-account AssociateVPCWithHostedZone API with appropriate IAM role/trust.
• For multi-region, create PHZ per region or use VPC-peering/Route 53 Resolver rules for cross-region resolution. Always restrict associations to specific VPC ARNs.

Naming conventions

• Use predictable, hierarchical names: <service>.<env>.svc.example.internal
  • Examples: payments.prod.svc.example.internal, payments.stage.svc.example.internal
• Alternatively: <service>.<env>.example.internal (shorter if no other svc subdomain needed)
• Keep env explicit to avoid accidental cross-env dependencies; enforce with templates and IaC.

Service discovery for ephemeral endpoints

• Use AWS Cloud Map (API + DNS) or ECS/Route 53 integration to register/deregister instances automatically.
• Use SRV or A records registered by the orchestrator; prefer alias records to NLBs for stable endpoints when possible.
• Configure low TTLs (e.g., 30s) for truly ephemeral tasks and client-side caching/backoff to avoid thundering herd.
• Combine health checks and Route 53 failover or weighted records for blue/green or rolling deploys.

Operational & security considerations

• Control association via IAM and RAM; log queries with Route 53 Query Logging to centralized S3/CloudWatch.
• Use Resolver endpoints and rules for cross-account/private-to-on-prem resolution.
• Enforce naming and zone ownership via org SCPs and IaC modules (Terraform/CloudFormation).

This approach gives clear environment separation, secure cross-account resolution, and robust discovery for ephemeral services.

Direct answer

Give concrete, structural changes tied to the specific root causes of the original project, not vague platitudes like "communicate more," and be ready to say which of those changes you've actually applied since.

Structured elaboration

Specificity bar
"I'd test more" is a weak answer. "I'd add a data-quality gate before the dashboard build starts" is a strong one. Name the mechanism, not the sentiment.

Categories to draw from
Technical or architecture choices, process or tooling, and stakeholder alignment (definitions, cadence). A strong answer usually touches more than one category, which shows you diagnosed broadly instead of reaching for the easiest lesson.

One question, several framings
This question covers the same underlying move whether it's asked as "what would you do differently," "how would you redesign this system today," or "what changed after you got critical feedback": name the retrospective insight and the concrete change it produced.

Close the loop
State whether you've actually applied the change since. This is what separates a rehearsed lesson from a real one.

Worked example

Original project: an analytics dashboard project where attribution gaps and inconsistent metric definitions surfaced only after launch.

Technical change: build a documented, versioned data model with defined event names and IDs up front, instead of ad hoc joins across sources that let downstream numbers drift out of sync.

Process change: add automated data-quality checks (null, duplicate, schema-drift checks) before any dashboard ships, instead of discovering issues after stakeholders start using the numbers.

Stakeholder change: run a metric-definition alignment session at the start of the project (what counts as a conversion, what attribution window applies) instead of assuming shared understanding.

Applied since: I now start every analytics project with a one-page data contract that stakeholders review before any building starts, which is a direct result of this project.

Trade-offs & pitfalls

• A generic lesson that could apply to any project signals you haven't actually diagnosed root causes.
• Naming only a technical fix and ignoring the process or communication cause (or the reverse), when the original failure had more than one cause.
• Claiming a change you've never actually implemented since; interviewers often ask directly whether it stuck.

Overview (6‑month timeline)
Month 0–1: assess maturity, define baseline controls. Months 2–3: pilot with 3 teams (low/med/high maturity). Months 4–5: phased rollout to remaining teams. Month 6: enforcement and metrics gate; continuous improvement.

Onboarding & Training

• Role-based curriculum: executive briefing, architects deep-dive, devs/practitioners hands-on labs.
• Delivery: 2 half-day instructor-led workshops + self-paced modules (Well‑Architected Pillars, cost ops, security, reliability, observability).
• Hands-on: 1-week “Well‑Architected sprint” per team with checklist and remediation playbooks.

Tooling & Automation

• Deploy AWS Well‑Architected Tool + guardrails: AWS Config rules, SCPs, IAM guardrails, cost allocation tags.
• Templates: IaC (Terraform/CloudFormation) starter patterns embedded with best practices.
• Integrations: CI/CD scans, Security Hub findings mapped to W‑A pillars.

Enforcement & Governance

• Tiered policy: advisory (pilot), recommended (after month 3), mandatory (month 6).
• Gate: PR/CD pipeline checks and monthly architecture review board sign-off for production launches.
• Exceptions process: time-boxed risk acceptance with mitigation and owner.

Support Model

• Central Cloud Center of Excellence (CCoE): 2 architects, 1 SRE, 1 security SME as concierge.
• Office hours, Slack channel, templates, remediation squads for critical findings.

KPIs (measurable)

• % of teams completing training within 2 months.
• % workloads reviewed by Well‑Architected Tool. Target 90% by month 6.
• Average number of critical/major findings per workload (decrease 50% vs baseline).
• Time to remediate critical findings (target <30 days).
• % of infra covered by IaC templates and guardrails (target 80%).

This plan balances education, automation, and governance with measurable gates and a support model to accelerate adoption while minimizing disruption.

Direct answer: A HIPAA-sensitive migration plan needs the same technical rigor as any near-zero-downtime migration, PLUS explicit preservation of specific HIPAA controls throughout: encryption/key management, access segregation, comprehensive audit logging, a Business Associate Agreement (BAA) with the cloud provider covering the specific services used, and PHI-specific data discovery to make sure nothing gets missed or exposed during the move.

Structured elaboration. Data encryption key management: PHI needs to stay encrypted at rest and in transit throughout the migration, with keys managed under a defined lifecycle (generation, rotation, access-logging); using the cloud provider's KMS (Key Management Service) with customer-managed keys (rather than provider-managed defaults) gives the organization more direct control and a clearer audit story for keys protecting PHI specifically. Access segregation: role-based access to PHI in the new environment needs to be at least as strict as on-prem, re-provisioned deliberately (least privilege) rather than carried forward broadly for migration convenience; a common risk during migration is a temporarily over-permissioned migration-tooling service account that isn't cleaned up afterward. Logging and audit trails: HIPAA requires audit logging of PHI access; ensure continuity of this logging through the cutover (no gap), and that the new environment's logging captures the same level of detail (who accessed what PHI, when) as the on-prem system did. Business Associate Agreement implications: confirm a BAA is in place with the cloud provider covering EVERY specific service being used to store or process PHI (not just a general BAA that may not cover a newly-adopted service), since using a service outside the BAA's scope for PHI is itself a compliance violation regardless of how well the migration is executed technically. PCI/PHI data discovery: this scenario is specifically HIPAA-sensitive systems storing PHI, with no stated cardholder-data (PCI) component, so the discovery effort described below is scoped to PHI; if the same systems also processed payment-card data, PCI-scoped discovery would additionally need to locate primary account numbers, CVV data, and any cardholder-data-environment boundary the PHI discovery process alone would not surface, since PCI and HIPAA define overlapping but distinct categories of sensitive data with different handling rules. For the PHI side: run explicit discovery to confirm exactly where PHI lives (including in unexpected places: backups, logs, temp files, or a reporting database that wasn't originally scoped as PHI-containing but received copies of PHI fields), since a HIPAA migration plan that's scoped only to the "obvious" PHI systems risks leaving an in-scope copy behind un-migrated or, worse, exposed during the transition. Validation steps to ensure controls are intact: post-migration, explicitly re-test that encryption, access segregation, and audit logging are all functioning correctly in the new environment (not assumed from configuration alone), ideally including a walkthrough with compliance/security stakeholders. Minimizing downtime while meeting regulatory requirements: use the same near-zero-downtime techniques (change-data-capture (CDC)-based replication, controlled cutover) as any other database migration, layered with the HIPAA-specific controls above rather than treating compliance as a separate, sequential phase after the technical migration.

Worked example. Discovery phase specifically searches for PHI beyond the primary clinical database: a reporting/analytics database that periodically pulls patient-identifiable fields for internal dashboards, and a log aggregation system that may have inadvertently captured PHI in request logs; both get brought into the migration's compliance scope even though neither was the "main" system originally being discussed. The BAA is confirmed to cover the specific managed database and KMS services being used, not just a general cloud-provider BAA that predates those service selections.

Trade-offs & pitfalls. Scoping HIPAA migration planning only to the obviously clinical systems, without a genuine PHI-discovery pass across the broader estate (reporting databases, logs, backups), is the most common way this kind of migration leaves an unmigrated or inadequately-protected copy of PHI behind, which is a compliance risk independent of how well the PRIMARY system's migration was executed.

Summary / goals
Design a cost‑optimized, global API at 100k RPS with ~50ms read and 200ms write targets by combining edge delivery, multi‑region read scaling, partitioned writes with region affinity, read caches, and tunable consistency.

High-level architecture

• Global API GW (CloudFront/Global Accelerator + regional API Gateways) for TLS termination, DDoS protection, & routing.
• Edge CDN for static/JSON cacheable responses.
• Regional API clusters (autoscaled compute) in N active regions.
• Regional in‑memory caches (Redis/Memcached) + global cache invalidation via pub/sub.
• Global datastore: geo‑aware DB (e.g., DynamoDB Global Tables, CockroachDB, or Spanner) with per‑entity partitioning.

Data partitioning

• Partition by customer/tenant or hash(key) to distribute keys evenly.
• Add region affinity: place primary write shard in nearest region for customer (reduces cross‑region writes).
• For hot keys, use deterministic sharding and moveable shards (rehash or split).

Caching strategy

• Edge CDN (TTL + stale‑while‑revalidate) for fully cacheable reads.
• Regional read‑through cache (Redis) for low latency ~<5ms. Populate caches on miss; use write‑through or invalidate via pub/sub.
• Client caching (ETag/Last‑Modified) and conditional requests to reduce origin load.
• Cache tiering: CDN -> regional cache -> DB.

Multi‑region replication & consistency

• Active‑active reads: replicate asynchronously to all regions for read scale.
• Writes: route to shard’s home region (primary) to avoid cross‑region consensus; asynchronously replicate to other regions.
• Consistency model:
  • Default: eventual consistency for global reads (fast, cheap).
  • Strong or read‑after‑write for critical endpoints: session tokens + sticky routing to primary region or use quorum reads (R + W > N) where supported.
  • Conflict resolution: last‑write‑wins for simple cases; application merge or CRDTs for complex state.

Latency mapping

• CDN edge hits: <20ms global.
• Regional cache hit: ~<5–10ms.
• Regional DB read (local replica): ~20–50ms.
• Writes (local primary + async replication): <200ms target; cross‑region writes avoided for most traffic.

Cost controls

• Autoscaling with CPU/memory/RPS metrics; scale down aggressively for idle regions.
• Use serverless (Lambda/FaaS) + API Gateway for highly variable workloads; provisioned capacity for steady baseline.
• Choose mixed instance purchasing: reserved/savings plans for baseline, spot for noncritical workers.
• Cache hit‑rate SLAs and cost per RPS monitoring; tune TTLs to meet hit targets.
• Rate limiting, API tiers and quotas to protect backend.
• Monitoring + budget alerts, cost allocation tags, periodic data lifecycle (cold storage, TTL on cold partitions).
• Evaluate per‑region write placement to trade latency vs storage/replication cost.

Tradeoffs

• Strong consistency everywhere increases cost and latency (consensus across regions) — use selective strong paths.
• More regions = lower latency but higher replication cost; choose regions by traffic/POPs.

This design meets 100k RPS by maximizing cache and edge hits, localizing writes via partitioning/affinity, and exposing tunable consistency to balance latency, correctness and cost.