Covers the core compute models and how to evaluate them for application workloads. Candidates should be able to explain virtual machines where operators manage the operating system and runtime and which provide maximum control but require operating system patching capacity planning and infrastructure maintenance. Candidates should understand container technologies and container orchestration patterns that enable packaging portability and efficient resource use for microservices while introducing operational concerns around orchestration networking storage and deployment. The topic includes managed platform as a service offerings that abstract runtime and deployment responsibilities to reduce infrastructure management at the cost of lower level control and customization. It also covers serverless function models that provide event driven automatic scaling and pay per execution billing while presenting constraints such as cold start latency execution time limits and challenges for long running or highly stateful workloads. Candidates should know instance type selection and resource profiles such as general purpose compute optimized and memory optimized options; autoscaling strategies and performance and cost trade offs; startup latency and cold start implications; state management and persistence patterns; monitoring and observability complexity; security and operational responsibilities; and how team expertise application architecture and cost considerations influence the best choice of compute option for a given workload.
EasyTechnical
60 practiced
Explain the differences between general purpose, compute optimized, memory optimized, and storage optimized instance types. For each type provide example workload profiles, key metrics you would monitor when sizing (CPU utilization, memory usage, disk IOPS, network bandwidth), and when to prefer vertical scaling versus horizontal scaling.
Sample Answer
General purpose: balanced CPU, memory, network. Use for web servers, small databases, dev/test, application servers.- Workload: moderate concurrency, predictable per-request cost.- Metrics to monitor: CPU utilization, memory usage, network throughput, latency.- Scaling: horizontal preferred for stateless services (add instances); vertical if single-process stateful or licensing limits.Compute optimized: high vCPU-to-memory ratio for CPU-bound tasks.- Workload: batch compute, high-performance web API, real-time analytics, video transcoding.- Metrics: sustained CPU utilization, CPU steal, context-switch rate, network bandwidth (if distributed), temperature/throttling.- Scaling: vertical to get faster single-thread performance when single-thread limited; horizontal for parallelizable workloads.Memory optimized: high RAM per vCPU for in-memory caches, databases, analytics.- Workload: Redis/Memcached, in-memory DBs, large JVM heaps, OLAP queries.- Metrics: memory utilization, swap activity, GC pause times (for JVM), disk I/O (for persistence), CPU for query processing.- Scaling: vertical when dataset must fit in one node’s memory; horizontal (sharding/replication) for scale-out and fault tolerance.Storage optimized: high local disk throughput/IOPS and low latency (sometimes NVMe).- Workload: transactional databases (WAL-heavy), data warehousing, log processing, big data nodes.- Metrics: disk IOPS, disk latency, throughput (MB/s), queue depth, CPU waiting on I/O, network for replication.- Scaling: vertical to get more IOPS/throughput per node; horizontal for partitioning data and increasing aggregate IOPS and resiliency.General guidance on choosing:- Measure the dominant bottleneck (CPU, RAM, I/O, network) and pick instance class accordingly.- Prefer vertical scaling when single-process constraints, licensing, or latency-sensitive single-node performance matter. Prefer horizontal scaling for availability, fault tolerance, and linear throughput growth when the workload can be partitioned.- Always combine right-sizing with autoscaling policies, monitoring (alerts on the key metrics above), and load/perf testing to validate choices.
MediumTechnical
52 practiced
A stateful application on Kubernetes needs high IOPS and low latency storage across multiple availability zones. Compare options such as local NVMe SSDs, cloud block storage via CSI drivers, managed replicated block storage, and managed distributed file systems. Discuss performance, failover, data locality, snapshot and backup trade offs and operational implications.
Sample Answer
Situation: A stateful app on Kubernetes requires high IOPS and low latency across multiple AZs. Below I compare four common storage approaches and give guidance on trade-offs and recommended patterns.Local NVMe SSDs (node-local)- Performance: Best latency and highest IOPS (direct-attached NVMe).- Failover: Pod relocation requires rebuild/replication; data lost if node fails unless app-level replication exists.- Data locality: Excellent — schedule pods on same node for locality; Kubernetes topology-aware scheduling helps.- Snapshots/Backup: Limited — snapshots must be handled by app or sidecar; node failure complicates backups.- Ops: High operational burden (stateful set affinity, manual replication, cluster autoscaling constraints).Cloud block storage via CSI (single-AZ volumes, e.g., EBS, Persistent Disk)- Performance: Good, but limited by network and single-AZ attachment; some providers offer high-performance IOPS tiers.- Failover: Fast detach/attach but cross-AZ attach is usually not supported — requires pod restart in target AZ and volume reattach.- Data locality: You must schedule pods in same AZ as volume to avoid cross-AZ latency.- Snapshots/Backup: Native snapshot/restore APIs (fast, incremental) — good for backups.- Ops: Lower ops than local NVMe; predictable, integrates with Kubernetes via CSI.Managed replicated block storage (zone-replicated block, e.g., multi-AZ volumes)- Performance: Lower latency than cross-AZ network volumes because replication is handled by provider; IOPS may be lower than local NVMe but acceptable.- Failover: Transparent — provider replicates across AZs so instance failover doesn't lose data.- Data locality: Reads/writes may still traverse network; some providers route to nearest replica.- Snapshots/Backup: Usually supported; consistency depends on provider features.- Ops: Simplifies HA; cost and vendor lock-in higher.Managed distributed file systems (clustered storage like Ceph, EFS/Filestore)- Performance: Variable — distributed systems add network overhead; some (Ceph on NVMe) can approach block-level IOPS with tuning.- Failover: Built for durability and multi-AZ replication; pods can access data from any AZ.- Data locality: Less locality; performance relies on network and data placement policies.- Snapshots/Backup: Many provide snapshots and replication; complexity in ensuring application-consistent backups.- Ops: Operationally heavy if self-managed; managed offerings reduce burden but still require tuning and monitoring.Recommendation pattern (Solutions Architect view)- For highest IOPS/lowest latency and tolerable single-node risk: use local NVMe with synchronous app-level replication (or StatefulSet with pod anti-affinity) and a backup pipeline to object storage.- For multi-AZ high availability with simpler ops: prefer managed replicated block storage or managed distributed FS from cloud provider that supports cross-AZ replication; accept slightly higher latency.- For balanced performance and native snapshot/backup: use cloud block CSI with AZ-affinity and architect application to be AZ-aware (replicate across AZs at app level) — gives good snapshots and moderate ops cost.- Always: benchmark with representative workload, design for AZ-aware scheduling, implement application-consistent backups (pre-freeze or filesystem quiesce), monitor latency/IOPS, and include failover runbooks.Trade-offs summary- Latency/IOPS: local NVMe > replicated block > distributed FS (tunable).- Failover/HA: distributed/replicated offerings > single-AZ block > local NVMe (unless app does replication).- Operational complexity: local NVMe and self-managed distributed FS highest; managed services lowest.- Snapshots/backup: easiest with cloud block and managed systems; harder with local-only solutions.Pick based on priorities: if low-latency transactional workload is primary and budget allows, use local NVMe plus app replication. If multi-AZ seamless HA and simpler ops are primary, choose managed replicated block or managed distributed FS and optimize for network latency.
HardSystem Design
75 practiced
Design a compute layer that can run a stateless service across AWS and Azure to reduce vendor lock in while enabling centralized deployment controls. Describe IaC choices, CI/CD pipeline strategy, secrets and key management across clouds, networking and service discovery considerations, and whether a service mesh or platform abstraction layer is appropriate.
Sample Answer
Requirements & constraints:- Run stateless service in both AWS and Azure for resilience and reduced vendor lock-in.- Centralized deployment controls, consistent IaC, secure secrets, low-latency service discovery, operability.- Prefer managed services where possible to reduce O&M.High-level architecture:- Deploy service as containers on managed Kubernetes (EKS in AWS, AKS in Azure) or as Fargate/App Service if Kubernetes is too heavy — I recommend Kubernetes for parity and portability.- Central Git repo + GitOps controller (Argo CD) in a management account/tenant that pushes manifests to cluster-specific Argo instances.- Edge: cloud-native API Gateway per cloud (ALB/NLB+Ingress in AWS, Application Gateway/Front Door in Azure) and optionally a global traffic manager (Route53 + Azure Traffic Manager or a DNS-based global load balancer).IaC choices:- Terraform (provider ecosystem + modules) with Terragrunt for environment orchestration OR Pulumi if you prefer imperative languages. Use one repository for reusable modules and per-cloud overlays.- Use a module pattern: core infra (VPC/VNet, subnets, IAM/AD), cluster module (EKS/AKS), platform services (CI, monitoring, vault).- Keep cloud-specific resources in small provider-specific modules; keep service manifests cloud-agnostic.CI/CD pipeline strategy:- Central pipeline (GitHub Actions / Azure DevOps / GitLab) triggers on PR merge to main: - Step 1: Terraform plan/apply per cloud (run in separate jobs, with approval gates). - Step 2: Build container image, push to multi-region registries (ECR and ACR) or a centralized registry (e.g., Docker Hub, GHCR). - Step 3: ArgoCD sync or kubectl apply to target clusters. Use feature flags and canary deployments via Argo Rollouts.- Enforce policy with pre-deploy checks (Conftest/OPA) and signing of artifacts.Secrets & key management:- Use HashiCorp Vault as the multi-cloud secrets control plane (self-managed HA or HashiCorp Cloud Platform). Vault uses cloud KMS for auto-unseal (AWS KMS, Azure Key Vault).- Short-lived dynamic secrets for DB credentials, and use Kubernetes CSI driver or Vault Agent Injector for app injection.- Store long-term KMS keys in each cloud (AWS KMS, Azure Key Vault) for envelope encryption where necessary; maintain key lifecycle policies and cross-account/tenant RBAC.Networking & service discovery:- Each cloud: private VPC/VNet, private subnets for workloads, bastion + managed NAT.- Use Private Link / Private Endpoint for managed services (RDS, Azure SQL, managed caches).- Service discovery: prefer Kubernetes DNS within clusters for intra-cluster. For cross-cluster discovery, use a service registry (Consul) or multi-cluster DNS with ExternalDNS + central DNS zone records. For cross-region/global traffic, use DNS-based weighted routing with health checks.- Peering/VPN: avoid cross-cloud network sprawl; design independence per cloud, rely on DNS + global load balancing rather than persistent cross-cloud L2 paths.Service mesh vs platform abstraction:- For a stateless service: avoid heavy multi-cloud service mesh initially. Start with platform abstraction: Kubernetes + consistent observability (Prometheus + Grafana, OpenTelemetry) and standardized ingress/egress.- If need for cross-cluster mTLS, traffic routing, or fine-grained telemetry, adopt a mesh (Linkerd lighter than Istio) deployed per-cluster with a multi-cluster control plane (consul or service-mesh-hub). Trade-offs: meshes add complexity, latency, and operational cost.Observability & operations:- Centralized logging (Fluentd/Fluent Bit to cloud-native or a central SIEM), distributed tracing (OpenTelemetry exporter to a single tracing back-end or per-cloud compatible endpoints), metrics aggregation (Prometheus federation).- Alerting & runbooks in central platform.Security & governance:- IAM mapping to corporate identity (AWS SSO / Azure AD) and least privilege.- Policy as code (OPA/Gatekeeper) enforced in Kubernetes and Terraform plans.- Central auditing via cloud-native logging exported to a central store.Key trade-offs:- Kubernetes gives portability at cost of cluster ops; serverless per-cloud reduces ops but increases lock-in.- Vault centralizes secrets but is another operational component; cloud-native secrets are simpler but increase vendor coupling.- Service mesh provides advanced traffic controls but is heavy; use only when telemetry/security/routing needs justify it.Why this design:- Terraform + Kubernetes + GitOps gives consistent deployments and centralized controls while enabling cloud independence.- Vault + cloud KMS provides secure, auditable secrets management across clouds.- Start with lightweight abstraction and add mesh only when cross-cluster features are required.
MediumTechnical
57 practiced
A customer runs both machine learning model training and online inference. Training uses many GPUs for throughput whereas inference requires low latency per request. Compare compute options such as GPU instances, Kubernetes with GPU node pools, managed inference services, and edge inference. Recommend configurations that balance cost, latency, and operational complexity for both workloads.
Sample Answer
Situation: Customer needs high-throughput GPU-based training and low‑latency online inference. Below I compare options and give recommended configurations balancing cost, latency, and ops complexity.Key comparisons and trade-offs- GPU instances (pay-as-you-go / reserved / spot): Highest control and predictable performance for both training and inference. Good for large distributed training (A100/V100) using MPI / Horovod. Cost-effective with spot/preemptible for non-critical training; medium ops burden (provisioning, monitoring).- Kubernetes with GPU node pools: Flexible, supports mixed workloads, autoscaling, and multi-tenant clusters. Use GPU node pools (A100 for training pool, T4 for inference pool) and tools like Karpenter/GKE Autopilot for scale. Higher ops complexity (drivers, device plugins) but better utilization and CI/CD integration.- Managed inference services (SageMaker Endpoint, Vertex AI, AWS Lambda + Elastic Inference, KServe/Triton on managed clusters): Lowest ops overhead, built-in autoscaling, A/B rollout, metrics. Slightly higher unit cost but best for predictable, SLA-backed low-latency inference. Supports model optimizations (ONNX/TensorRT).- Edge inference (Jetson, Coral, mobile): Best for ultra-low latency and bandwidth savings; requires model optimization (quantization, pruning) and additional deployment complexity and lifecycle management.Recommended architecture1. Training (throughput-first, cost-sensitive)- Use batch GPU instances or a Kubernetes training node pool with A100s.- Use spot/preemptible instances for non-critical epochs; maintain a smaller on‑demand master node.- Distributed training with NCCL + Horovod or PyTorch DDP; use high-bandwidth networking (RDMA/ENIs).- Store checkpoints on S3/Blob and use object lifecycle policies.2. Online inference (latency-first)- For strict SLA (<50ms), use managed inference endpoints with GPU T4/RTX or CPU with optimized models depending on model size. Example: AWS SageMaker real-time endpoints or Vertex AI with autoscaling and provisioned concurrency.- For variable traffic: use managed autoscaling + warm pools or KServe with NVIDIA Triton on Kubernetes inference node pool (T4s) for lower cost and custom routing.- Techniques: model quantization, batching at micro-batch sizes (1-8), model sharding, using TensorRT/ONNXRuntime, response caching, CDN for static content.3. Edge / hybrid- If users need sub-10ms on-device latency or offline capability, deploy quantized models to Jetson/Coral or mobile SDKs. Centralized telemetry for model/version control.Operational recommendations- Separate clusters/node pools for training vs inference to avoid noisy neighbors.- Use IaC (Terraform) and GitOps for reproducible deployments.- Monitoring: Prometheus/Grafana, NVIDIA DCGM, request latency SLOs, autoscale policies based on p99 latency and GPU utilization.- Cost controls: spot for training, rightsizing inference instances, scheduled scaling for predictable low-traffic periods.Example config- Training: 8x A100 (p3dn/p4d) spot-backed nodes, 1 on-demand coordinator, 100 Gbps networking, S3 checkpointing.- Inference (strict latency): Managed endpoint on T4 with provisioned concurrency 2x expected qps, autoscale to handle bursts; model compiled with TensorRT.- Edge: 1-2 GB quantized model on Jetson Nano/Orin or mobile with 8-bit int.Conclusion: Use specialized GPU instances or k8s GPU pools for training (spot for cost), managed inference endpoints or Triton on a dedicated inference pool for low ops and latency, and edge deployments for ultra-low latency or offline scenarios. Separate infrastructures, optimize models, and use autoscaling and spot strategies to balance cost, latency, and complexity.
MediumSystem Design
74 practiced
Design a compute strategy for an e commerce platform expected to handle 10,000 requests per second at peak with seasonal spikes, PCI compliance for card processing, and a mix of stateful and stateless services. The client has a small operations team. Propose which components to host on VMs, containers, managed PaaS, or serverless, and justify your choices across scale, compliance, cost, and operational burden.
Sample Answer
Requirements (clarify): 10k RPS peak with seasonal spikes, PCI scope for card processing, mixed stateful (sessions, cart, payment state) and stateless services, small ops team.High-level architecture:- Edge: CDN (CloudFront/Cloudflare) + WAF for DDoS/OWASP rules.- Ingress/API layer: API Gateway / Load Balancer.- Stateless microservices: containerized (Kubernetes/EKS or managed Fargate).- Stateful services: managed databases (RDS/Aurora) and managed Redis (ElastiCache).- Payments/PCI: use a PCI-certified 3rd-party gateway (Stripe/Adyen) with tokenization; minimal card handling in-scope.- Background jobs: serverless functions (Lambda) or managed queue workers for bursts.- Batch/stateful workers: VMs for long-running stateful processes if needed.- Observability/ops: managed monitoring (CloudWatch/DataDog), autoscaling, IaC (Terraform).Component mapping & justification:- CDN + WAF (Managed PaaS): scales automatically, reduces origin load and attack surface — low ops.- API Gateway (Managed PaaS): rate-limiting, auth, TLS termination — simplifies security and scaling.- Stateless services (Containers on managed k8s or Fargate): containers give fast deploy, horizontal autoscale for 10k RPS; using managed k8s reduces ops burden; Fargate if ops team is very small.- Stateful DBs & Redis (Managed PaaS RDS/ElastiCache/Aurora): automated backups, Multi-AZ, read replicas — required for reliability and lowers maintenance.- Payment processing (Serverless + Third-party): keep all card data out of your infrastructure; use serverless functions to orchestrate token exchange and webhooks — minimizes PCI scope and cost.- Background tasks (Serverless): event-driven bursts minimize cost during low traffic, scale instantly for spikes.- Long-lived/stateful workers (VMs): use VMs for legacy, heavy in-memory processes needing sticky state; otherwise prefer refactor to containers/managed services.- CI/CD & IaC (Managed services + containers): automated pipelines, blue/green deploys.Trade-offs:- Cost: Serverless + managed PaaS reduces ops cost but may be higher per-request at sustained high load; containers on managed k8s + autoscaling are cost-efficient at sustained load.- Compliance: Using third-party PCI provider plus strict network segmentation (private subnets, HSM/ KMS for keys), logging, and managed DBs reduces PCI burden.- Operational burden: Favor managed PaaS and serverless to keep ops light; reserve VMs for unavoidable stateful legacy components.Scaling strategy:- Autoscale pods/functions by request/queue depth; use horizontal DB read replicas and partitioning; use CDN and caching layers to reduce backend RPS.- Load testing and chaos experiments before seasonal peaks.Security & compliance notes:- Keep card data off-platform; if any card data handled, ensure dedicated PCI environment, network isolation, encryption at rest/in transit, strict logging/retention, and use a compliant deployment checklist.This mix balances scale, cost, compliance, and minimal ops — containers (managed) for core stateless services, managed PaaS for stateful infrastructure, serverless for bursty/background tasks, and VMs only for legacy/stateful processes that cannot be containerized.
Unlock Full Question Bank
Get access to hundreds of Compute Options and Trade Offs interview questions and detailed answers.
