Cloud Platform Fundamentals Questions

Comprehensive understanding of core public cloud services and the primary trade offs when selecting among them across major providers such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Candidates should know compute options including virtual machines, managed compute, containers and serverless functions; storage types including object, block and file storage and lifecycle and archival strategies; managed database offerings for relational, non relational, and data warehouse workloads; networking fundamentals including virtual private networks, subnets, routing, load balancing, content delivery networks, and private connectivity; messaging and integration services such as message queues and event streaming; identity and access management and secrets management; monitoring, logging, and observability; autoscaling, elasticity, high availability, and basic disaster recovery patterns; and cost and pricing considerations. The topic also covers the trade offs between managed services and self managed infrastructure in terms of consistency, latency, cost, operational overhead, and durability, and the ability to map common workload requirements to the right service categories.

EasyTechnical

39 practiced

Explain differences and typical use cases for a relational managed database (e.g., RDS, Cloud SQL), a NoSQL/document store (e.g., DynamoDB, Firestore), and a cloud data warehouse (e.g., Redshift, BigQuery). Give examples of data-engineering tasks suited to each.

HardTechnical

40 practiced

Write pseudocode or describe in detail an approach to achieve exactly-once processing semantics for a streaming ETL pipeline using Kafka (or a cloud pub/sub) and Spark Structured Streaming or similar. Cover ingestion, idempotency, checkpointing, and sink semantics.

Sample Answer

Approach summary:- Use Spark Structured Streaming with Kafka (or cloud pub/sub) as source.- Rely on Spark checkpointing for exactly-once processing semantics, and employ idempotent/transactional sink semantics or external transactional coordination when sink lacks native transactions.- Maintain per-partition offsets and a durable, atomic commit of output + offsets.

Pseudocode (Python-like) — exactly-once via foreachBatch + idempotent upserts and offset-stash:

python

from pyspark.sql import SparkSession

spark = SparkSession.builder \
  .appName("exactly_once_etl") \
  .getOrCreate()

# 1) Read from Kafka with starting offsets and checkpointing
df = spark.readStream.format("kafka") \
  .option("kafka.bootstrap.servers", "k:9092") \
  .option("subscribe", "input_topic") \
  .option("startingOffsets", "earliest") \
  .load()

# 2) Parse and add dedup key (e.g., event_id) and event time
events = df.selectExpr("CAST(value AS STRING) as json") \
  .selectExpr("from_json(json, schema) as data") \
  .select("data.*") \
  .withColumn("event_id", ...) \
  .withColumn("event_ts", ...)

# 3) Use watermark + dropDuplicates to bound state and dedupe within time window
clean = events.withWatermark("event_ts", "1 hour") \
  .dropDuplicates(["event_id"])

# 4) foreachBatch: write idempotently and atomically persist processed offsets
def foreach_batch(batch_df, batch_id):
    # compute max offsets per partition from Kafka metadata (or pass offsets)
    offsets = extract_offsets(batch_df)  # map topic-partition -> offset
    # write batch to sink using idempotent upsert (primary key = event_id)
    # Example: write to transactional DB using MERGE/UPSERT
    batch_df.write \
      .format("jdbc") \
      .option("url", jdbc_url) \
      .option("dbtable", "target_table") \
      .option("isolationLevel","SERIALIZABLE") \
      .mode("append_or_upsert") \
      .save()
    # atomically persist offsets in a durable store (e.g., offsets table, ZK, or transactional DB)
    persist_offsets_atomically(offsets, batch_id)

clean.writeStream \
  .foreachBatch(foreach_batch) \
  .option("checkpointLocation", "/mnt/checkpoints/exactly_once_etl") \
  .start()

Key concepts and reasoning:- Checkpointing: Spark stores progress and state to checkpointLocation so re-start resumes without reprocessing already-committed micro-batches.- Deduplication: dropDuplicates + watermark avoids duplicate event delivery within bounded lateness; required because upstream can redeliver.- Idempotent sink: prefer sinks supporting idempotent writes or transactions: - Databases: use MERGE/UPSERT by primary key (event_id) or transactional batch + persist offsets in same DB transaction. - Kafka sink: use Kafka producer transactions (exactly-once through Spark Kafka sink supports transactions when enabled) so writes are atomic with committed offsets. - Object storage: write atomic partitioned files with unique filenames (use deterministic path including offsets/batch_id), then commit manifest.- Atomic offset commit: Only mark offsets consumed after the output write is durably committed. Implement via: - Using sink transactions that also commit consumer offsets (Kafka transactions), OR - Persist offsets in same DB transaction as upsert, OR - Two-phase commit: write to staging, then atomically move/commit + offsets.- Checkpointing + idempotency: Checkpointing prevents Spark from reprocessing the same micro-batch but cannot guarantee external sink atomicity by itself; combine with idempotent sink ops.

Edge cases & trade-offs:- Long-running dedupe state: choose watermark size carefully to balance correctness vs state size.- Exactly-once for non-transactional sinks requires careful idempotent designs and atomic offset persistence.- Latency vs consistency: larger atomic operations (transactions) can increase latency.- Failure during commit: design retry logic to be idempotent; store batch_id and offsets so downstream can detect incomplete commits.

This pattern gives practical exactly-once semantics: ensure atomicity between output write and offset commit, dedupe upstream duplicates, and rely on Spark checkpointing for progress recovery.

MediumTechnical

41 practiced

Your team is deciding between a managed OLTP database (Cloud SQL/RDS) and self-managed PostgreSQL on Kubernetes for a latency-sensitive transactional workload. Evaluate trade-offs for maintenance, backups, extensions (e.g., PostGIS), failover, and expected operational effort.

Sample Answer

Start by clarifying constraints: SLOs for latency (p99), RPS, dataset size, required extensions (PostGIS), compliance, and team’s DBA/K8s skill level. Given that, here’s a concise comparison.

Managed OLTP (Cloud SQL / RDS)- Maintenance: Provider handles OS/DB patching, minor version upgrades, and automated tuning options — low team overhead.- Backups: Automated snapshots, point-in-time recovery (PITR), retention policies configurable; easy restores but with provider-defined performance/consistency guarantees.- Extensions: Many managed offerings support common extensions (PostGIS on Cloud SQL/RDS) but some niche extensions or custom shared libraries may be restricted.- Failover: Built-in multi-AZ high-availability with automated failover and DNS switchover; predictable but with failover windows and potential connection hiccups.- Operational effort: Low. Focus on schema, query tuning, monitoring. Good for teams prioritizing reliability and fast time-to-market.

Self-managed PostgreSQL on Kubernetes- Maintenance: Full control — you manage OS, PG versions, tuning, and container lifecycle. Higher risk and complexity (storage, I/O tuning).- Backups: You must implement backup tooling (pg_basebackup, WAL shipping, obj store snapshots) and ensure consistent restores across PVs and clusters.- Extensions: Maximum flexibility — any extension or custom build is possible.- Failover: Achievable with Patroni/replication and k8s operators, but orchestrating reliable leader election and fast failover with low latency is complex; storage performance across nodes matters.- Operational effort: High. Requires on-call DB expertise, careful storage/IO planning, testing of failover/restore, and SRE practices.

Recommendation (Data-engineer lens):- If team lacks strong DB/K8s ops and you need low operational overhead with predictable HA and managed backups, choose managed Cloud SQL/RDS.- If you need non-standard extensions, custom tuning for extreme low-p99 latency, or full control over storage/replication, and you have experienced DBAs/SREs, choose self-managed on K8s but budget for higher SRE effort and rigorous runbooks.- Hybrid: Use managed for primary OLTP and a self-managed read/analytics replica if specific extensions or specialized tuning are needed for downstream workloads.

EasyTechnical

43 practiced

Describe the trade-offs between using managed services (e.g., managed Kafka, managed databases, serverless ETL) versus self-managing on VMs or Kubernetes. Focus on operational overhead, consistency guarantees, latency, cost, and version/control for data engineering workloads.

Sample Answer

Managed vs self-managed is a trade-off between operational burden and control — for data engineering you weigh reliability, latency, cost, and ability to evolve pipelines.

Operational overhead- Managed: low — provider handles provisioning, patching, HA, scaling and most runbooks (e.g., Confluent Cloud, RDS). Team focuses on pipelines and schema.- Self-managed: high — you run clusters (VMs/K8s), upgrades, backups, capacity planning, DR. Requires on-call and SRE practices.

Consistency guarantees- Managed: typically strong SLAs and tested replication/topologies; providers expose configs but may hide internals. Good for standard consistency models (exactly-once or at-least-once via managed Kafka connectors).- Self-managed: full control to implement custom replication, stronger isolation or tuned consistency, but correctness depends on your ops expertise.

Latency- Managed: often optimized and geo-replicated; network egress and multi-tenant throttling can add tail latency. Serverless ETL can introduce cold-starts.- Self-managed: can achieve lowest-latency by colocating services, using tuned JVM/OS/kernel settings, and dedicated networks — but only if you invest in tuning.

Cost- Managed: predictable OPEX, higher per-unit price but saves staffing costs. Good for small teams or variable load.- Self-managed: higher fixed costs (instances, licenses, staff), can be cheaper at scale if you fully utilize resources but risk overprovisioning.

Version/control- Managed: limited version choices and slower upgrades; safer defaults but less flexibility for custom builds or bleeding-edge features.- Self-managed: full control over versions, configs, and extensions (custom connectors, JVM args), enabling advanced optimizations or compliance needs.

Recommendation (practical)- Start with managed for ingestion, Kafka, and ETL to move fast and reduce ops risk.- Move to self-managed only if you need fine-grained control (extreme latency SLAs, custom plugins, cost at massive scale) and can support the ops burden.- Hybrid: use managed core services, self-manage performance-critical or proprietary components on K8s/VMs. Monitor costs, SLAs, and runbooks to revisit choices periodically.

HardTechnical

42 practiced

Describe how you would lead a cross-functional migration from VM-based ETL jobs to a serverless architecture (e.g., managed serverless functions and managed dataflow). Include stakeholder engagement, rollout plan, risk mitigation, indicators of success, and training for the team.

Unlock Full Question Bank

Get access to hundreds of Cloud Platform Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.