Amazon Web Services Architecture and Operations Questions
Advanced knowledge of Amazon Web Services platform services, architectural patterns, operational best practices, and trade offs. Candidates should be able to justify compute choices such as Amazon Elastic Compute Cloud instance types, instance sizing and performance tuning, and Auto Scaling strategies; storage and durability decisions including Amazon Simple Storage Service storage classes, versioning, lifecycle management, replication and archival strategies; database patterns such as Amazon Relational Database Service with multi availability zone deployments, read replicas and failover behavior, and Amazon DynamoDB capacity modes and throughput trade offs; networking design including Amazon Virtual Private Cloud topology, subnet and routing strategies, peering, gateway and interface endpoints, and network security controls; infrastructure as code and deployment patterns using Amazon CloudFormation including stack management and automated rollbacks; serverless and event driven design such as Amazon Web Services Lambda concurrency and cold start considerations and integration with Amazon API Gateway; content delivery and caching with Amazon CloudFront and Amazon ElastiCache including cache invalidation and expiry strategies; service specific operational concerns such as rate limiting, backup and restore, monitoring, logging, alerting and incident response; and cross cutting concerns including identity and access governance, cost optimization, disaster recovery planning and testing, and automation. Interview focus is on design reasoning, anticipating failure modes, scaling strategies, performance tuning, observability and automation, and provider specific operational practices.
HardTechnical
69 practiced
Explain a safe CloudFormation change-management strategy for stacks that include stateful resources (RDS, EBS, S3). Discuss nested stacks, Change Sets, resource replacement implications, stack policies, automated backup steps before updates, and manual approval gates for production updates.
Sample Answer
Situation: Managing CloudFormation updates for stacks that contain stateful resources requires a cautious, repeatable change-management strategy to avoid data loss and downtime.Strategy summary (step-by-step):1. Design with nested stacks and least-surface mutations- Split infra into logical nested stacks: networking, stateless compute, stateful resources (RDS, EBS, S3). This limits blast radius so changes to app stacks rarely touch stateful stacks.2. Use Change Sets for safe preview- Always create a Change Set (aws cloudformation create-change-set) and review the resource-level actions (Modify, Replace, Delete).- Reject any Change Set that contains unexpected Replace/Delete for stateful resources.3. Understand replacement implications- Replacing RDS/EBS typically means data loss unless using snapshots/replication. Treat "Replacement" as blocking by default for production. For EBS, replacement detaches and can reattach but may require filesystem work. For RDS, replacement => new instance/endpoint.4. Apply stack policies and guardrails- Attach a stack policy to prevent updates or replacements of critical logical IDs (e.g., RDS instance, S3 bucket). Use CloudFormation Guard or AWS Config rules to enforce patterns.5. Automated pre-update backups- Before applying changes that touch stateful stacks, run automated snapshot workflows: - RDS: create automated snapshot with aws rds create-db-snapshot and verify completion. - EBS: create volume snapshots (create-snapshot) after freezing IO if needed (fs freeze or use application quiesce). - S3: versioning enabled + lifecycle/archive to backup bucket objects; for destructive changes, replicate to a backup bucket.- Store snapshot IDs and validate recovery drill scripts.6. Manual approval gates for production- Require a manual approval step (e.g., CodePipeline manual approval or an approver in your CI/CD) if Change Set includes any Replace/Delete affecting stateful resources.- Approval checklist: Change Set diff, snapshot IDs, expected downtime, rollback plan, monitoring runbook.7. Deployment workflow example- Dev/stage: automated apply-on-merge using CI with simulated Change Set review.- Prod: (a) CI creates Change Set and posts summary to PR/Slack; (b) automated backup job runs and records snapshot IDs; (c) on-success, a human approves; (d) deploy executes aws cloudformation execute-change-set; (e) post-deploy validation and health checks.8. Rollback and recovery- Keep tested rollback playbooks referencing snapshots and DNS/endpoint recovery steps. Automate smoke tests and synthetic monitoring to validate.Trade-offs and best practices- More nested stacks = finer isolation but increases orchestration complexity.- Stack policies + manual approvals increase safety at cost of deployment speed; tune for emergency fast paths.- Regularly test restores from snapshots to ensure backups are reliable.This approach balances automation with safety: preview changes, prevent accidental replacements via policies, take automated backups, and require human approval for any production-impacting stateful changes.
EasyTechnical
87 practiced
What causes AWS Lambda cold starts? Explain warm vs cold start behavior, factors that influence cold-start latency (package size, runtime choice, VPC ENI creation), and list at least three practical mitigation strategies for latency-sensitive services.
Sample Answer
Cold starts occur when AWS Lambda needs to provision a new execution environment (a fresh container + runtime) to handle an invocation because no warm instance is available. That initialization includes downloading the function code, starting the language runtime, running any global/static initialization code, and (if configured) attaching network interfaces for VPC access.Warm vs cold start:- Warm start: invocation reused an existing initialized environment; latency is just handler execution.- Cold start: includes overhead to create and initialize the container and runtime, adding tens to hundreds (or more) of milliseconds.Factors influencing cold-start latency:- Package size: larger deployment packages increase container pull/extract time.- Runtime choice: heavier runtimes (Java/.NET) have longer JVM/CLR startup and classloading; interpreted runtimes (Node/Python) typically start faster.- VPC ENI creation: older Lambda VPC model needed creating elastic network interfaces, which added significant latency (improved by AWS ENI reuse and Hyperplane but still a factor).- Initialization code: expensive global imports, SDK clients, or DB connections add time.- Memory allocation: memory correlates with CPU; low memory can yield slower startup.Mitigation strategies (latency-sensitive services):1. Provisioned Concurrency: keep pre-initialized environments to eliminate cold starts.2. Reduce package/init cost: slim dependencies, use Lambda layers, lazy-init heavy clients inside handler.3. Choose faster runtimes / increase memory: move to Node/Python or allocate more memory to speed startup.4. Avoid VPC cold-start penalty: use RDS Proxy, VPC endpoints, or place resources in public subnets/managed networking to reduce ENI overhead.5. Warmers and traffic shaping: keep a small steady invocation rate or scheduled pings (less reliable than Provisioned Concurrency).6. Observability: instrument cold-start metrics (init duration) and alert on P99 latency so decisions are data-driven.As an SRE, combine measures: monitor cold-start impact, use Provisioned Concurrency for critical paths, optimize code, and validate cost vs latency trade-offs.
HardSystem Design
68 practiced
Design a rate-limiting system for a multi-tenant API serving thousands of API keys that must be fair and scalable. Compare token-bucket vs leaky-bucket algorithms, centralized vs distributed enforcement, and evaluate using API Gateway usage plans versus a custom Redis/ElastiCache-based solution. Include handling of bursts, global limits, and DDoS protection strategies.
Sample Answer
Requirements & constraints:- Per-API-key & per-tenant fairness, thousands of keys, low latency (<10–50ms), support bursts, global (across regions) and per-region limits, scalable, resilient to DDoS, cost-aware.High-level architecture (hybrid):- Edge WAF + CDN (AWS Shield Advanced + CloudFront) for coarse DDoS mitigation and blocking bad IPs.- API Gateway / ALB at edge for routing + optional usage-plan enforcement for simple quotas.- Fast path rate-limit checks in a distributed cache (ElastiCache Redis cluster) with local caching and fallback to a token-sync service for global counts.- Metrics & alerting: Prometheus/CloudWatch, anomaly detection + auto-throttling rules.Token-bucket vs Leaky-bucket:- Token bucket: allows bursts up to bucket size, replenishes tokens at rate R — ideal for per-key burstiness and fairness.- Leaky bucket: enforces steady output rate; simpler smoothing but penalizes legitimate bursts.- Choice: token-bucket per key with min-burst and sustained-rate guarantees.Centralized vs Distributed enforcement:- Centralized (single DB/service): simpler consistency but single point of failure and latency bottleneck.- Distributed (per-edge/region caches + global sync): scales and lowers latency; main challenge is global consistency for global limits.- Choice: distributed enforcement with Redis cluster per region + periodic/global counters in a strongly consistent datastore for hard global caps.Redis-based implementation (recommended for flexibility):- Store per-key token bucket state: tokens, last_ts, refill_rate, burst_size as a Redis hash.- Use an atomic Lua script to refill and consume tokens to avoid race conditions.- For global limits: maintain a separate global counter (sharded) with approximate consistency (Redis INCR + TTL) and enforce hard stops via a centralized coordinator when thresholds near.- Local caching: edge nodes maintain an in-memory local token-bucket with small burst window; sync to Redis to replenish to avoid thundering herd.API Gateway usage plans vs Custom Redis:- API Gateway usage plans: - Pros: Managed, low ops, integrates with API keys, simple quotas & throttling. - Cons: Less flexible (no multi-dimensional limits easily), per-account scaling limits, limited global enforcement across regions, cost at scale.- Custom Redis: - Pros: Flexible policies (per-tenant, per-endpoint), token-bucket semantics, prioritized keys, dynamic policy updates, better horizontal scalability. - Cons: Operational overhead, must manage Redis scaling, durability, and latency.Handling bursts:- Token-bucket with configurable burst_size per tier; allow local in-memory burst; track debt using Redis to prevent sustained abuse.- Graceful degradation: prioritize high-SLA tenants, queue lower-priority requests, return Retry-After headers.Global limits:- Implement two-tier checks: fast local/regional token-bucket, then approximate global counters (sharded Redis or CRDT counters) for hard global caps. For strict global caps use a strongly-consistent service (DynamoDB conditional writes) for rare enforcement points.DDoS protection strategies:- Use AWS Shield + WAF + CloudFront rate-based rules to drop offending IPs before hitting API.- Apply IP reputation scores and progressive throttling (per-IP then per-key).- Implement challenge (CAPTCHA) or connection rate limits for suspicious clients.- Auto-scale Redis/worker pools and enable circuit breakers to return 503s when backend under pressure.Operational concerns & metrics:- Key metrics: requests/sec per-key, token consume rate, throttles, 429s, latency, Redis ops/latency.- Alerting on sudden throttle spikes, Redis latency, memory saturation.- Testing: chaos/injection tests, load tests with synthetic keys, simulate burst & DDoS.Trade-offs summary:- Use API Gateway for simple cases or early stage. For multi-tenant fairness, complex policies, and scale, implement Redis-backed token-bucket with regional caches, global counters, and WAF/Shield at the edge.
EasyTechnical
73 practiced
Explain the differences and relationships between CloudWatch Metrics, CloudWatch Logs, and CloudWatch Alarms. When should you prefer metrics over log-based metrics, how do metric filters work, and how do dashboards and alarms fit into an SRE monitoring strategy?
Sample Answer
CloudWatch Metrics, Logs, and Alarms serve complementary roles:- CloudWatch Metrics: time-series numerical data (CPU, latency, custom counters). Efficient for high-cardinality, aggregated, low-latency queries and long-term retention. Use for SLO/SLA/SLI calculations and real-time dashboards.- CloudWatch Logs: raw log lines (application, audit, stack traces). Rich detail for debugging, post-incident forensics, and extracting new signals.- CloudWatch Alarms: evaluate metrics (built-in or derived from logs) against thresholds over windows, triggering actions (SNS, Lambda, Auto Scaling).When to prefer metrics vs log-based metrics:- Prefer native metrics when you can emit structured numeric values directly from your app/infra (lower cost, lower latency, easier aggregation).- Use log-based metrics when you cannot change producers, need to derive counts from existing logs (e.g., error patterns), or when extracting new signals during investigation.How metric filters work:- A metric filter scans ingested log groups for patterns (string, JSON, or regex). Matches increment a CloudWatch metric (count, value) with optional dimensions. Filters run as logs arrive, creating time-series metrics usable by alarms and dashboards.Dashboards and alarms in an SRE strategy:- Dashboards: present SLIs, key health indicators, and capacity metrics for on-call and runbooks. Use layered dashboards (overview -> service -> host/endpoint).- Alarms: encode alerting rules from SLO error budgets and operational thresholds. Tune for noise reduction (multi-period/aggregation, severity tiers, suppression windows) and route via escalation policies.Together: emit clean metrics, derive where necessary from logs, visualize on dashboards for situational awareness, and use alarms tied to SLOs for actionable, low-noise alerting.
HardSystem Design
91 practiced
Design a global, highly-available architecture for an e-commerce platform with requirements: RTO <5 minutes, RPO <1 minute, 100M monthly users, and regional compliance constraints. Discuss active-active vs active-passive, use of Aurora Global Database or multi-region replication, Route53 routing/health checks, S3 replication, and testing strategies.
Sample Answer
Requirements & constraints:- RTO < 5m, RPO < 1m, 100M monthly users (peak regional traffic), data residency per region.- Strong consistency for payments/orders, eventual for catalogs/media.High-level architecture (multi-region):- Global control plane + regional data plane. Deploy identical stacks in 3+ regions (primary region + 2 replicas) behind local ALBs/EKS clusters for microservices. Use CDN (CloudFront) for static assets and API edge caching.- DB: Use Amazon Aurora Global Database for low-latency cross-region read replicas and fast secondary promotion. Keep writes sharded by region where compliance requires (regional primary clusters) and use change-data-capture to replicate to global view when allowed.- Object storage: S3 with CRR (Cross-Region Replication) to satisfy residency + disaster recovery; use replication rules for buckets that must remain in-region and lifecycle policies.- Routing: Route 53 with latency-based routing + health checks and weighted failover. Use DNS TTLs short (30s) and Route 53 health checks backed by synthetic transactions (not just TCP) to detect app-level failures.- Messaging/async: Multi-region Kafka (MSK) or regional SNS+SQS with durable replication via MirrorMaker/replication. Use idempotent consumers.Active-active vs active-passive trade-offs:- Active-active pros: lower read/write latency, automatic local failover, capacity spread. Cons: complexity around multi-master conflicts, cross-region consensus, compliance complications.- Active-passive (primary-write, secondary-read): simpler consistency, easier to meet RPO/RTO with Aurora Global DB promotion. Use active-passive where orders/payments require strict consistency; consider read-active for catalogs/features.Aurora Global DB vs multi-region replication:- Aurora Global: near-instant cross-region replication (typically <1s for redo logs), fast secondary promotion (minutes), managed conflict-free for primary-secondary topologies — good fit for RPO<1m and RTO<5m for regional failover.- If full multi-master required, consider third-party multi-master DB (Cassandra/CockroachDB) but weigh complexity and transactional limitations.Operational details:- CI/CD: Blue/green deployments with cross-region canaries. Use GitOps to keep region parity.- Observability: Centralized metrics (CloudWatch/Prometheus remote write), distributed tracing (X-Ray/Jaeger), centralized logging (Kinesis->Elasticsearch/S3). Synthetic transaction probes from multiple regions.- Failover plan: Automated runbooks to promote Aurora secondary, update Route53 weights, reconciliation jobs for eventual consistency.S3 replication and compliance:- Use S3 CRR with object-lock and encryption keys scoped by region (KMS multi-region keys or separate CMKs per region for legal separation). Tag buckets for compliance and automate lifecycle.Testing strategies (SRE-focused):- Chaos Engineering: Region kill tests, AZ failures, network partitioning, and degrading DB replication latency to validate RTO/RPO.- DR drills: Quarterly full failover rehearsals with runbook timing measurement. Test both automated and manual promotion paths.- Load tests: Synthetic traffic simulating 100M monthly distribution with peak regional spikes. Test cold-start capacity.- Observability tests: Inject tracing/metrics errors to ensure alerting and runbooks trigger.- Game days: Cross-functional incident simulations including compliance/legal sign-off steps.SLOs & runbooks:- Define SLOs (availability, latency percentiles, error budgets). Automate rollback and alerting thresholds. Maintain playbooks for promotion, DNS change, data reconciliation, and post-mortem.Why this meets requirements:- Aurora Global + Route53 short TTLs + automated promotion gives RPO <<1m and RTO <5m for DB failover path. Multi-region stacks plus S3 CRR and CDN ensure data locality and high availability while allowing active-passive consistency where needed. Regular chaos/DR testing validates real-world guarantees.
Unlock Full Question Bank
Get access to hundreds of Amazon Web Services Architecture and Operations interview questions and detailed answers.
