Cloud Infrastructure Knowledge (AWS/GCP/Azure) Questions
Have working knowledge of at least one major cloud platform: common services (EC2/Compute Engine, RDS/Cloud SQL, S3/Cloud Storage, Load Balancers, VPCs, networking), typical failure modes, and how to troubleshoot within that platform. Understand concepts like availability zones, regions, and cross-region failover.
HardTechnical
39 practiced
Design a zero-downtime schema migration strategy for a 5 TB production Postgres table in a cloud-managed database. Discuss techniques like backfilling, shadow tables, online schema change tools, phased deployments, and how you would validate correctness and performance during the process.
Sample Answer
Requirements & constraints:- 5 TB Postgres in cloud-managed service (limited superuser), zero downtime SLA, minimal CPU/IO impact, ability to rollback, validate correctness & performance.High-level approach:1. Phased, online migration using shadow table + dual-writes + controlled backfill, with feature-flagged cutover and monitoring.Steps:- Prep: add no-blocking changes first (new nullable columns, indexes concurrently). Create a shadow table with desired schema (create table like main plus changes).- Dual-write: deploy application change to write to both main and shadow (via library/DB proxy). If app change not possible, use logical replication (pglogical/pg_recvlogical) to replicate changes into shadow.- Backfill: run parallelized, throttled backfill jobs that copy historical rows from main -> shadow in ID ranges using ORDER BY primary key with batched transactions. Use COPY/INSERT with pg_copy or COPY (to minimize WAL) if allowed. Rate-limit with pg_sleep or a job scheduler to control IO.- Keep applying ongoing WAL/replicated changes until lag is zero.Validation:- Row-level checksums/hashes (md5 of important columns) per batch; compare counts and sample queries. Use checksum tables and reconcile diffs.- Query performance: run representative production read queries against shadow (via traffic shadowing or read-replica) under load testing to compare latency and IO.- Monitor: CPU, IO, WAL size, replication lag, query latency, error rates, SLOs.Cutover:- Switch reads to shadow using feature flag / read-route change, monitor for errors for a short canary window.- Promote shadow to primary: in cloud-managed DB, validate approach: if rename/simple swap not possible, create a transactional swap (create new table then rename tables in a short maintenance transaction). If rename takes too long, update app routing to point to new DB endpoint (preferred).- Stop dual-write, ensure no drift.Rollback:- If issues, flip reads back and resume dual-writes to main; mark shadow for investigation. Keep final cutover reversible for a short period.Tools & techniques:- CONCURRENTLY index creation, logical replication (pglogical), pg_repack for bloat, pg_dump/pg_restore for non-production, pt-online-schema-change-like approach for Postgres (gh-ost concepts), cloud-native features (read replicas, replica promotion).- Use orchestration: Airflow/Argo for backfill batches, Prometheus/Grafana + alerts for metrics.Trade-offs:- Dual-write complexity vs replication lag risk. Shadow table + replication reduces app changes but needs robust reconcile.- Backfill speed vs impact: prefer slower safe copy.Key validations:- Automated checksums per batch, query-level SLA comparison, end-to-end functional tests, run chaos tests for partial failures. Document rollback steps and keep stakeholders notified.
MediumSystem Design
32 practiced
Design a centralized logging pipeline for logs from 1,000 VMs/pods using cloud-native logging (or a hosted ELK/BigQuery solution). Include log collection, parsing/indexing, retention, alerting, cost controls, and how to handle spikes in ingestion without data loss.
Sample Answer
Requirements & constraints:- Collect logs from 1,000 VMs/pods, low-loss ingestion, searchable within minutes, retention policy (hot/cold), alerting (SLO/breach), cost control, handle ingestion spikes.High-level architecture:- Agents -> Ingestion buffer/stream -> Parser/Indexer -> Storage (hot/archival) -> Alerting & Query UI- Example stack (hosted): Fluentd/Fluent Bit agents -> Kafka / Pub/Sub -> Log-processing workers -> Elastic Cloud OR BigQuery for long-term + Kibana / Looker for UI -> Alertmanager / Cloud MonitoringComponents & responsibilities:1. Collection:- Lightweight Fluent Bit on pods/VMs: forwards to local forwarder with TLS, metadata enrichments (pod, cluster, labels).- Backpressure via batch size, retry, jitter.2. Ingestion buffer:- Kafka or Cloud Pub/Sub as durable buffer to absorb spikes and enable consumer scaling. Set retention window (e.g., 7 days) to allow reprocessing.3. Parsing & indexing:- Stream processors (Logstash/Fluentd/Kafka Streams/Dataflow) to parse JSON, grok, apply schema, drop noisy fields.- For ELK: index to hot Elasticsearch nodes for recent data; for BigQuery: write streaming inserts into partitioned tables (date/ingest-time) and use columnar storage for cost-effective queries.4. Retention & lifecycle:- Hot tier: 7–30 days in ES hot nodes (SSD). Warm/cold: move older indices to cheaper storage (S3/GCS) and use searchable snapshots.- BigQuery: partitioned tables + expiration policy; export raw compressed logs to GCS for long-term cold archive.5. Alerting & SLOs:- Alerting on error-rate, ingestion lag, agent failures via Prometheus/Cloud Monitoring -> Alertmanager -> PagerDuty.- Alerts on Kafka/Topic lag, ES indexing latency, BigQuery streaming errors.- Create log-based metrics (e.g., 5xx per minute) and SLOs with error budget.6. Cost controls:- Sampling/ingest filtering at agent or stream stage; route verbose debug logs to separate low-priority topic with lower retention.- Field removal and compression before storage.- Use aggregated metrics/dashboards instead of querying raw logs frequently.- Use lifecycle policies to freeze/freeze-thaw indices and move to object storage.7. Handling spikes/no data loss:- Durable buffer (Kafka/PubSub) sized to hold expected burst (hours). Backpressure agents: if buffer full, agents persist to local disk (Fluent Bit supports buffering) and retry.- Autoscale consumers for parsing/indexing; use rate-limiting to ES/BigQuery to avoid overload; use bulk writers with retries and exponential backoff.- Graceful degradation: route excess to cold storage (GCS) for later batch ingestion.Trade-offs:- ELK gives low-latency search; expensive at scale. BigQuery cheaper for large volumes, slower for ad-hoc searches.- Buffering increases latency but prevents loss.Operational considerations:- SLOs for ingestion latency and availability, runbooks for agent/consumer failures, periodic cost reviews, synthetic traffic to test bursts. Regularly tune parsing and retention policies based on ROI per log source.
HardTechnical
57 practiced
Design a cost-aware autoscaler that mixes spot/preemptible instances with on-demand to run batch workloads with low cost but acceptable availability. Discuss capacity estimation, preemption handling, checkpointing, graceful degradation, and trade-offs in SLOs.
Sample Answer
Requirements & constraints:- Batch workload: throughput target X jobs/hour, max job latency T_max, acceptable failure rate p_fail_budget, cost minimization primary.- Cloud supports preemptible/spot with termination notice (e.g., 30s) + on-demand instances.High-level design:- Maintain two pools: On-demand baseline (stable capacity) + Spot fleet (elastic, cheaper).- Orchestrator (e.g., Kubernetes with node pools & Karpenter or autoscaling controller) schedules jobs across pools based on priority and checkpointability.Capacity estimation:- Model historical spot availability and preemption rate per instance type/zone. Compute expected usable time E[U] and effective capacity = instances * E[U]/job_duration.- Choose on-demand baseline = capacity needed to meet SLO when spot availability is minimal. Baseline = ceil((required_throughput * (1 - spot_contribution_target)) / single_node_throughput).- Spot target scaled by predicted availability and cost: target_spot_capacity = (total_capacity_needed - baseline) * safety_factor (e.g., 0.7) to account for volatility.- Continuously update model with monitoring (eviction_rate, time-to-eviction, bid price) and feed into autoscaler.Preemption handling:- Use provider termination notices to drain tasks: on-notice handler signals orchestrator to stop scheduling new work and trigger graceful shutdown/checkpoint.- Implement fast-resume: for short jobs < notice window, attempt to finish; else checkpoint.- Maintain a small "standby" on-demand reserve that autoscaler can quickly scale up (or keep warm instances) to replace lost spot capacity for critical jobs.Checkpointing strategy:- Classify jobs: - Stateless/short (< notice window): run on spot preferentially. - Long-running/checkpointable: run on spot but require periodic checkpoints. - Non-checkpointable/high-priority: run on baseline on-demand.- Adaptive checkpoint frequency: balance recomputation cost vs checkpoint overhead. Use expected time-to-preemption T_pre; choose interval t such that expected wasted work ≈ checkpoint cost: solve minimizing (checkpoint_cost/t + lost_work_probability * avg_work_lost).- Use incremental, application-level checkpoints stored in durable object store (S3/GCS) and keep metadata in a fast DB for resume.Graceful degradation:- Priority queueing: critical SLO jobs served from baseline; best-effort jobs from spot pool.- Backpressure: if spot preemptions spike, autoscaler reduces acceptance of best-effort jobs, throttles retries, and increases on-demand spin-up for critical fraction.- Progressive degradation: reduce concurrency per job class, shift low-priority batch to deferred time windows.Trade-offs & SLOs:- Cost vs availability: larger spot fraction lowers cost but increases variance and job completion latency. Tune spot_fraction to meet p_fail_budget and latency percentiles.- Define SLOs per job class (p99 latency, success_rate). Keep hard SLOs on on-demand capacity; soft SLOs use spot.- Use error budget for spot-driven retries: when error budget is low, autoscaler increases on-demand baseline.Operational considerations:- Monitoring: eviction rate, time-to-evict distribution, job restart rate, checkpoint frequency, cost per completed job.- Alerts: rising eviction rate, queue backlog, autoscaler failing to meet capacity.- Chaos testing: simulate mass preemptions, verify checkpoint/resume and autoscaler responsiveness.- Cost controls: budget alerts, per-job cost attribution.Example technologies: Kubernetes node pools (on-demand + spot), Karpenter/Cluster Autoscaler, Spot Fleet/Instance Groups, termination-notice DaemonSet, durable storage (S3/GCS), Redis for job state, Prometheus + Grafana for metrics.Result:- This hybrid autoscaler maintains low baseline cost for critical SLOs while exploiting spot capacity for throughput. Continuous feedback from eviction metrics and adaptive checkpoint scheduling balances cost vs acceptable availability.
EasyTechnical
29 practiced
Explain the difference between a cloud region and an availability zone (AZ). As an SRE, describe how each affects system design decisions for failure isolation, latency, cost, and data-residency across AWS, GCP, and Azure. Give concrete examples of when to choose single-region-with-multiple-AZs versus multi-region deployments and the trade-offs involved.
Sample Answer
Region vs AZ — short definitions:- Region: a geographically isolated area (e.g., us-east-1, europe-west1, westeurope). Contains multiple AZs; used for data residency, disaster domain separation, and latency to users in that geography.- Availability Zone (AZ): an independent failure domain within a region (separate data center or cluster with independent power/network). Examples: us-east-1a, europe-west1-b.How each affects SRE decisions:- Failure isolation: AZs protect against rack/DC-level faults. Design: run redundant instances across 2–3 AZs, use cross-AZ load balancing and distributed storage with synchronous or asynchronous replication. Regions protect against large-scale outages (power grid, region-wide network); multi-region is required for regional disasters.- Latency: AZs inside a region have low inter-AZ latency (ms); prefer multi-AZ for typical HA. Cross-region latency is higher (tens to hundreds ms); use multi-region only when needed for geo-proximity or DR.- Cost: AZ-redundancy costs less (data transfer inside region often cheaper); multi-region adds cross-region replication costs, duplicate infrastructure, and higher operational overhead.- Data residency/compliance: Regions determine where data resides. For GDPR or local regulations pick region accordingly; multi-region may be constrained by legal rules.Cloud nuances:- AWS/GCP/Azure all offer multiple AZs per region; naming and exact isolation vary but design principles are the same. Watch for “soft” AZ mappings in some regions (e.g., smaller cloud regions with fewer AZs).When to choose:- Single-region, multiple-AZs: Use when you need high availability and low latency for a primary user base, and can accept region-level outages. Example: customer-facing web app serving a single continent with RPO minutes via multi-AZ DB replicas.- Multi-region: Use when you need resilience to region failure, global low-latency presence, or legal residency across regions. Example: global SaaS with active-active reads in US and EU, writes routed to nearest region with cross-region async replication and conflict resolution.Trade-offs summary:- Multi-AZ: lower cost, simpler, fast failover, protects against most hardware/DC failures but not region-wide disasters.- Multi-region: stronger disaster protection and geo-performance but higher cost, complexity (consistency, DNS failover, replication), and operational overhead.Practical tip: start multi-AZ; add multi-region when SLOs require regional independence or legal/latency constraints demand it.
HardTechnical
29 practiced
Explain options for cross-region replication of PostgreSQL in cloud environments (managed read replicas, logical replication, WAL shipping, patroni/PGPool, Aurora Global DB). For each option describe RPO/RTO expectations, typical failure modes, and a plan to promote a replica to primary with minimal data loss.
Sample Answer
Situation: As an SRE designing cross-region PostgreSQL DR/replication, I evaluate options by expected RPO/RTO, common failure modes, and concrete promotion steps to minimize data loss. Below are practical summaries and promotion playbooks.1) Managed read replicas (cloud-managed physical streaming replicas)- RPO/RTO: RPO typically seconds–minutes (depends on network), RTO minutes–tens of minutes.- Failure modes: network partition, replication lag, replica crash, cloud control plane issues.- Promotion plan: 1) Verify replica is caught up (pg_stat_replication/pg_last_wal_receive_lsn vs primary LSN). 2) Stop replica recovery (cloud API promote or pg_ctl promote). 3) Reconfigure DNS/Load Balancer and update application write endpoints. 4) Re-point other replicas to new primary or rebuild. Expect minimal data loss if replica was synced; if lag exists, accept last-wins within RPO.2) Logical replication (pglogical/pg_recvlogical/pgoutput)- RPO/RTO: RPO depends on apply throughput; usually seconds–minutes; RTO minutes–hours for large schema changes.- Failure modes: replication slot bloat, DDL incompatibilities, row filtering bugs, replication replay lag.- Promotion plan: 1) Ensure subscriber has applied all WALs (pg_stat_subscription). 2) Stop writes to old primary or pause; ensure consistent cut-over. 3) Reconfigure sequences, constraints, and any circular replication. 4) Flip DNS and promote subscriber to primary role. Expect small window of in-flight transactions — plan brief write freeze for zero-loss.3) WAL shipping (file-based archive + restore)- RPO/RTO: RPO minutes–hours depending on shipping frequency; RTO often hours (restore + replay).- Failure modes: missing archive files, long restore time, corrupted WAL, storage exhaustion.- Promotion plan: 1) Restore base backup, apply archived WALs to target until desired LSN. 2) Remove recovery.conf/standby settings to allow writes. 3) Start DB and switch traffic. Data loss equals any WALs not archived/transferred prior to outage; aim to ship WALs continuously and monitor archive lag.4) Patroni/PGPool (HA orchestration + leader election)- RPO/RTO: RPO seconds (depends on synchronous commit config); RTO seconds–minutes for leader election.- Failure modes: split-brain if quorum lost, etcd/consul failure, misconfigured sync settings causing accepted writes lost.- Promotion plan: 1) Use orchestrator to elect leader (automatic). 2) Confirm replica LSN and apply state. 3) Ensure fencing of old primary (prevent split-brain). 4) Reconfigure clients via service discovery. For minimal loss, enable synchronous replication for critical replicas and robust quorum.5) Aurora Global DB (cloud proprietary cross-region)- RPO/RTO: RPO typically under a second to seconds; RTO low (minutes) because secondaries are read-only replicas with fast failover mechanisms.- Failure modes: global control-plane outage, cross-region replication lag under heavy write storms, regional cloud incidents.- Promotion plan: 1) Use provider failover API to promote secondary region to writable. 2) Validate LSN/cluster consistency and reconfigure endpoints. 3) Re-point replicas and clients. Very low data loss if design fits provider SLAs.Operational best practices (applies to all):- Define SLOs and accept trade-offs: synchronous for strict RPO, asynchronous for performance/cost.- Continuous monitoring: replication lag, WAL archive queue, replication slots, control-plane health.- Automated runbooks &ingress switch: tested Terraform/Ansible playbooks to promote and reconfigure DNS/LBs.- Regular DR drills: simulate failover, measure RTO/RPO, validate apps under read-only and promoted states.- Fencing and split-brain prevention: use quorum, STONITH equivalents, or cloud provider safeguards.This approach balances realistic RTO/RPO expectations with explicit promotion steps and operational mitigations to minimize data loss.
Unlock Full Question Bank
Get access to hundreds of Cloud Infrastructure Knowledge (AWS/GCP/Azure) interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.