Amazon Web Services Architecture and Operations Questions
Advanced knowledge of Amazon Web Services platform services, architectural patterns, operational best practices, and trade offs. Candidates should be able to justify compute choices such as Amazon Elastic Compute Cloud instance types, instance sizing and performance tuning, and Auto Scaling strategies; storage and durability decisions including Amazon Simple Storage Service storage classes, versioning, lifecycle management, replication and archival strategies; database patterns such as Amazon Relational Database Service with multi availability zone deployments, read replicas and failover behavior, and Amazon DynamoDB capacity modes and throughput trade offs; networking design including Amazon Virtual Private Cloud topology, subnet and routing strategies, peering, gateway and interface endpoints, and network security controls; infrastructure as code and deployment patterns using Amazon CloudFormation including stack management and automated rollbacks; serverless and event driven design such as Amazon Web Services Lambda concurrency and cold start considerations and integration with Amazon API Gateway; content delivery and caching with Amazon CloudFront and Amazon ElastiCache including cache invalidation and expiry strategies; service specific operational concerns such as rate limiting, backup and restore, monitoring, logging, alerting and incident response; and cross cutting concerns including identity and access governance, cost optimization, disaster recovery planning and testing, and automation. Interview focus is on design reasoning, anticipating failure modes, scaling strategies, performance tuning, observability and automation, and provider specific operational practices.
MediumTechnical
71 practiced
You need to limit egress data transfer costs from multiple VPCs to the internet. Propose network and architectural changes within AWS to reduce cross-AZ egress, NAT gateway costs, and optimize use of VPC endpoints where appropriate.
Sample Answer
Goal: minimize cross‑AZ data transfer and NAT gateway egress costs while preserving functionality and security.Proposal (clear, actionable):1. Align resources with AZ-local NATs- Provision one NAT Gateway per AZ and ensure subnets/instances in that AZ use the AZ’s NAT via subnet route tables. This removes cross‑AZ hairpinning (eliminates inter‑AZ data transfer + NAT processing on another AZ).2. Reduce NAT usage with VPC Endpoints- Use Gateway VPC Endpoints for S3 and DynamoDB (free data transfer for those services).- Use Interface Endpoints (AWS PrivateLink) for other AWS APIs (Secrets Manager, SSM, ECR, KMS where supported) so traffic stays on AWS network and avoids NAT/IGW.- Enable “private DNS” for interface endpoints so SDK calls resolve to the endpoint automatically.3. Centralized egress with Transit Gateway (if multi‑VPC)- Attach VPCs to a Transit Gateway with per‑AZ attachments or use regional TGW and place NAT Gateways in each spoke/attachment AZ. Avoid a single centralized NAT in a different AZ to prevent cross‑AZ charges.- Consider TGW Connect/Edge for centralized inspection only if it’s co‑located per AZ.4. IPv6 & Egress-only IGW- Where possible enable IPv6 and use an egress‑only IGW for outbound IPv6 to cut NAT charges (no NAT for IPv6).5. Caching, batching, and direct uploads- For S3 uploads/downloads, have instances upload directly to S3 via SDK + endpoint, or use presigned URLs from backend so clients don’t traverse NAT.- Batch frequent small calls to reduce per‑byte processing overhead.6. Networking hygiene & routing- Use route table segregation per AZ.- Use prefix lists and security groups to restrict endpoint usage.- Monitor with VPC Flow Logs and Cost Explorer / CloudWatch to validate savings.Tradeoffs & rationale:- More NATs = higher hourly NAT cost but often lower data transfer/processing cost due to eliminated cross‑AZ charges — calculate breakeven using your traffic profile.- Interface endpoints have hourly + per‑GB costs; compare against NAT egress + cross‑AZ costs for high‑volume APIs.- Transit Gateway simplifies topology but adds TGW data charges; ensure per‑AZ placement to avoid cross‑AZ egress.Measurement:- Before/after: track NAT GB processed, inter‑AZ data transfer, endpoint traffic, and monthly cost. Start with high‑volume VPCs/services (S3, ECR) and iterate.
HardBehavioral
84 practiced
Behavioral: Describe a time you were on-call for a cloud outage. Explain the steps you took to identify the root cause, how you communicated with stakeholders, what automated mitigations or runbooks you used or created, and what you changed afterwards to prevent recurrence.
Sample Answer
Situation: Last year I was primary on-call when our SaaS product experienced a major outage—API error rates spiked to 40% and page load latency tripled for ~30 minutes during peak traffic.Task: As the on-call engineer I needed to quickly identify root cause, restore service, keep stakeholders informed, and implement mitigations to prevent recurrence.Action:- Triage & identification: - Immediately acknowledged PagerDuty alert and opened an incident in our channel. - Checked Grafana/CloudWatch dashboards and request traces in X-Ray; errors were concentrated on the payments-service pods. Logs showed repeated 502s and upstream timeouts. - Ran kubectl to inspect pod statuses; discovered a surge of OOMKills correlated with a recent config change to the JVM heap and a new dependency that increased memory usage. - Reproduced locally with a load profile and confirmed memory leak in the new dependency.- Mitigation & runbook use: - Executed our emergency runbook: rolled back the last deployment via Helm to the previous stable revision and scaled replicas + enabled node autoscaling to absorb load. - Applied a circuit-breaker feature flag to throttle non-critical API endpoints to protect core payment flows. - Communicated status every 15 minutes: concise updates in Slack incident channel, formal email to Product and Support with ETA, and an incident badge in customer status page.- Post-incident fixes: - Created automated checks in CI to run memory profiling on that dependency and a load-test gate before deploys. - Wrote a new runbook step for memory-leak symptoms (metrics to inspect, kubectl commands, rollback commands) and automated a rollback script callable from CI. - Added alert thresholds for OOM events and pod restart spikes to reduce MTTR. - Led a postmortem with blameless analysis, documenting root cause (unsafe dependency + config change), timeline, and action items.Result: Service restored within 28 minutes after rollback. MTTR for similar incidents fell from ~45m to ~18m after adding alerts and automating rollback. The CI gate prevented the problematic deployment from reaching production again. I learned to prioritize rapid mitigation and clear stakeholder communication, and to bake automated safety nets into the deployment pipeline.
HardTechnical
86 practiced
Design a disaster recovery testing plan for a mission-critical system on AWS. Define objectives (RPO/RTO verification), test types (table-top, partial failover, full failover), automation to run tests (CloudFormation/Runbooks), rollback plans, and evidence collection for audits.
Sample Answer
Requirements & objectives:- Verify the system can meet defined RPO (max data loss) and RTO (time to recovery) for the mission-critical workload (e.g., RPO=5 min, RTO=30 min).- Provide repeatable, auditable tests with automated orchestration and safe rollback.- Produce evidence for compliance audits.High-level approach:1. Define test matrix: environments (non-prod/prod-like), scope (service-level, region-level), frequency (quarterly full, monthly partial, weekly smoke/table-top).2. Test types: - Table-top: runbooks reviewed with stakeholders; validate decision flow, communications, escalation paths. - Partial failover: fail individual components (AZ, DB read replica promotion, ELB detach, subnet isolation) to validate graceful degradation and RPO/RTO per-component. - Full failover (planned): switch traffic to DR region/account, promote replicas, reconfigure DNS/ALBs, validate end-to-end transactions.3. Automation: - CloudFormation/Terraform templates for DR stacks; parameterized to deploy DR infra identical to prod. - Step Functions / Systems Manager Automation / Lambda runbooks to execute sequence: scale down prod, promote replicas, update Route53 health checks, run smoke-tests. - CI pipeline job to run DR test automation in non-prod on schedule; GitOps for runbook and infra changes.4. Rollback plans: - Pre-defined automated rollback playbook: re-point DNS back to prod weighted records, demote promoted DBs (if supported) or re-seed from backups, failback steps with versioned AMIs and infrastructure templates. - Safety gates: manual approvals for production-impacting steps; canary traffic and staged cutover.5. Evidence & audit artifacts: - Time-stamped logs (CloudWatch Logs, S3 exports) of every automated step. - Metrics snapshots: RPO measurement (last write timestamp vs recovered), RTO measurement (timestamp recovery completed). - Screenshots/recordings of validation tests, smoke-test results, API traces, and perf/latency dashboards. - Signed runbook execution report with actors, timestamps, decisions, and post-mortem.6. Validation & success criteria: - Automated checks: data consistency tests, end-to-end transaction tests, latency and error-rate thresholds. - RPO/RTO pass/fail recorded; any failure creates JIRA incident and postmortem.7. Governance & safety: - Role-based access (IAM), separate DR account for failover, tagging, feature flags to prevent accidental full failover. - Pre-test checklist: backups verified, stakeholders notified, maintenance windows scheduled.8. Continuous improvement: - Post-test retro: update runbooks, fix automation gaps, track trend of RTO/RPO improvements.Example sequence for a full failover test (automated):- Snapshot DB and confirm WAL replication position.- Deploy DR infra via CloudFormation.- Promote read-replica to primary (or restore snapshot).- Update Route53 weighted records to shift traffic (10%→100%) with health checks.- Run smoke-tests; measure RTO from start to green.- If criteria met, run rollback to prod; else execute rollback playbook and open incident.This plan balances automation for repeatability with manual controls for safety, provides measurable RTO/RPO verification, and collects auditable evidence for compliance.
EasyTechnical
65 practiced
Describe the differences between Auto Scaling policies: target-tracking, step scaling, and scheduled scaling. Give an example scenario where target-tracking is superior and one where step-scaling is required.
Sample Answer
Target-tracking, step scaling, and scheduled scaling are three ways to control Auto Scaling behavior:- Target-tracking: You pick a metric target (e.g., keep average CPU at 50%). The autoscaler continuously adjusts capacity to drive the metric toward the target using a control loop. Good for simple, self-tuning scaling with predictable SLAs and cost sensitivity.- Step scaling: You define thresholds and corresponding scaling steps (e.g., CPU > 70% → +2 instances; CPU > 90% → +5). Actions execute when alarms fire. Good for aggressive, discrete responses to large or sudden load changes.- Scheduled scaling: You set capacity changes at specific times (e.g., add 3 instances at 9:00 AM every weekday). Useful for known, predictable traffic patterns.When target-tracking is superior:- Scenario: A web service with fairly smooth, variable traffic where you want to maintain a consistent response-time SLA while minimizing cost. Target-tracking automatically and smoothly adjusts capacity to hit the metric target (e.g., average CPU or request-per-target), reducing oscillation and management overhead.When step-scaling is required:- Scenario: A batch-job service that occasionally receives large spikes (e.g., data ingestion bursts) where you must add many instances quickly when a high threshold is crossed. Step-scaling lets you map severity to discrete scaling jumps and ensures a fast, deterministic response (e.g., +10 instances when queue length > 10k), which target-tracking’s gradual adjustments might not achieve fast enough.Notes:- Combine policies: common to use target-tracking for baseline and step-scaling for emergency spikes.- Monitor CloudWatch metrics, cooldowns, and scale-in protection to avoid flapping and ensure availability while controlling cost.
HardSystem Design
80 practiced
Design a multi-region active-active architecture for a global consumer service with RTO < 2 minutes and RPO < 1 minute. Discuss data replication (Aurora Global DB, DynamoDB Global Tables), DNS strategies (Route 53), conflict resolution, and how you'd test failover.
Sample Answer
Requirements & constraints:- Functional: global reads/writes with low latency- Non-functional: RTO < 2 min, RPO < 1 min, active-active multi-regionHigh-level architecture:- Each region runs stateless frontends (ALB/NLB + autoscaling), API service, and local caches (Redis/Memcached).- Data tier: - For relational transactional data: primary-write in one region using Amazon Aurora with Global DB replicating to secondary regions (fast physical replication). - For user-profile/low-latency key-value data: DynamoDB Global Tables (multi-master) with local read/write copies per region.- Global ingress: Route 53 (latency-based or geo + health checks) and optionally AWS Global Accelerator for deterministic anycast routing and fast failover.Data replication strategy and RPO/RTO rationale:- Aurora Global DB: replicates storage asynchronously across regions with typical replication lag < few seconds. To meet RPO <1m, ensure cross-region replication monitoring and use fast redo apply settings; keep WAL shipping and instance-level backtrack policies tuned. Because Aurora Global DB is primary-secondary (only primary accepts writes), active-active at relational level requires sharding by write-region or promoting a secondary on failover.- DynamoDB Global Tables: native multi-master replication with typically single-digit second convergence. This supports active-active writes in all regions and helps meet RTO/RPO for key-value data.Conflict resolution:- DynamoDB Global Tables: use application-level deterministic resolution strategies: - Last-writer-wins based on synchronized timestamps (use server-side monotonic version tokens or Lamport timestamps) OR - CRDTs / merge functions for composable data (counters/lists) - Use conditional writes and version attributes (optimistic concurrency) for critical items.- Aurora (if multi-primary attempted): avoid cross-region concurrent writes. Preferred pattern: designate a primary writer for relational data; for true active-active, partition user-data such that each region owns a shard/tenant to avoid conflicts. On failover, promote secondary and reconcile via application-level reconciliation for any outstanding uncommitted transactions.DNS & routing:- Route 53 latency-based or geo-routing to send users to nearest healthy region. Use Route 53 health checks (and ALB/NLB endpoints) with low TTL (30s) to reach RTO <2m.- Use weighted routing for staged failover tests (shift 10% → 50% → 100%) and fail-safe blue/green switches.- Consider AWS Global Accelerator to reduce DNS TTL dependency and provide near-instant failover by BGP anycast endpoints.Failover & testing plan:- Automated health checks and runbooks that: - Detect region failure → Route53 health fails → Route53 shifts traffic within TTL window (< 30s) plus client retries to meet RTO. - For Aurora: automated promotion playbook (promote global secondary to primary). Test promotion time regularly; automate via scripts + CloudWatch Events. - For DynamoDB: since multi-master, ensure clients retry on conditional write errors and verify convergence.- Testing methodology: - Canary and staged traffic-shift drills (weighted Route53 shifts). - Chaos experiments: simulate AZ and full-region outages (stop EC2/ALB or blackhole network). - DB drills: simulate Aurora primary loss and measure promotion time and data loss; verify RPO by checking WAL/apply lag. - Recovery runbooks and postmortems: check observability (CloudWatch, X-Ray), verify data consistency, run reconciliation jobs for conflicts.- Rollback & reconciliation: have idempotent replay queues (Kinesis/SQS) that buffer cross-region writes for reconciliation if needed.Observability & SLA operations:- Track replication lag metrics (Aurora replica lag, DynamoDB replication metrics), API error rates, latency, and Route53 health.- Automated alerts if replication lag > 30s or promotion exceeds target.- Practice DR drills quarterly and measure RTO/RPO to validate SLOs.Trade-offs:- Aurora Global DB: simpler strong-consistency model but not truly multi-master; use for relational single-writer patterns or sharded active-active.- DynamoDB Global Tables: true active-active but eventual consistency—requires app-level conflict resolution.- Adding Global Accelerator increases cost but lowers failover unpredictability vs DNS-only.This design meets RPO <1m by using near-real-time replication and monitoring; meets RTO <2m via low TTL DNS/Global Accelerator, automated health checks, scripted DB promotion, and staged failover testing.
Unlock Full Question Bank
Get access to hundreds of Amazon Web Services Architecture and Operations interview questions and detailed answers.
