Big Data Technologies Apache Spark Questions

Understand Apache Spark fundamentals: RDDs, DataFrames, and Datasets. Know Spark SQL for data processing. Understand distributed processing concepts: partitioning, shuffling, and task execution. Know when to use Spark for data processing tasks and its advantages over traditional SQL.

MediumTechnical

49 practiced

You notice severe skew on the join key when joining an events table to a users table, where a small set of user_ids account for most events. Propose a strategy to detect skew and mitigate its impact on joins and aggregations in Spark, including salting, broadcasting hot keys, and sample-based approaches.

Sample Answer

Detecting skew- Sample key frequency: run a lightweight job that computes counts per user_id on a sample (or use approximate methods like approx_count_distinct or DataFrame.sample + groupBy.count). Visualize top-k; if top 0.1–1% account for >50% of rows, it's skewed.- Use Spark UI / metrics: check task durations and shuffle read/write sizes; tasks handling hot keys will be much slower and larger.

Mitigation strategy (practical, incremental)1. Short-circuit small table: if users table is small, broadcast it. - Use broadcast(users) for user table < broadcast threshold (spark.sql.autoBroadcastJoinThreshold). - Benefit: eliminates shuffle.

2. Handle hot keys explicitly - Identify “hot” user_ids from sample/top-k detection. - For joins: split logic: process non-hot keys with normal join; process hot keys with special treatment. - For hot keys, either: a) Broadcast the small set of hot user rows and perform filtering + map-side join, or b) Replicate user rows for hot keys to multiple partitions and partition events by salted key.

3. Salting (for many-hot events side) - Add a salt column to events for hot keys: salt = randInt(0, N-1) where N tuned (e.g., 10–100). - Expand user rows for those hot keys into N duplicates with matching salt values. - Join on (user_id, salt). This spreads load across reducers. - Remember to remove salt and aggregate results after join.

4. Sample-based and hybrid approaches - Use a sample-run to pick N and identify hot keys; adapt N proportional to frequency. - For aggregations: compute aggregates for non-hot keys normally; for hot keys compute with salted partitions and then combine.

Implementation considerations- Choose N to balance parallelism vs. overhead. Validate with smaller runs.- Maintain determinism of salt for reproducibility (hash-based salt on event_id or row_index).- Monitor Spark UI for task skew after change; add fallback (increase N or broadcast) if still skewed.- Ensure idempotence and correctness when duplicating user rows and de-salting results.

Expected outcome- Reduced max task time and more uniform shuffle sizes, faster joins and aggregations. Use automated detection to periodically refresh hot-key list.

EasyTechnical

86 practiced

Compare Spark SQL with a traditional single-node SQL engine like Postgres for analytics workloads. Discuss differences in distributed query execution, latency vs throughput trade-offs, concurrency, handling of large datasets, and when Spark SQL provides advantages over a traditional SQL database.

Sample Answer

Spark SQL vs Postgres (single-node) — concise comparison for analytics workloads:

Distributed query execution- Spark SQL: Executes queries across a cluster using RDD/DataFrame DAGs; shuffles, stages, and task parallelism enable processing huge datasets in parallel. Optimizer: Catalyst + Tungsten for physical planning and vectorized execution. Good for embarrassingly parallel or wide transformations.- Postgres: Single-node planner/executor; all work happens on one machine CPU/RAM/disk. Excellent for OLTP and single-node OLAP on moderate-sized data.

Latency vs throughput trade-offs- Spark SQL: Designed for high throughput and batch/mini-batch processing. High task startup and shuffle overhead → higher query latency (seconds to minutes) but excellent throughput for large scans.- Postgres: Low latency queries (milliseconds to seconds) for indexed lookups and small aggregations. Throughput limited by single-node resources.

Concurrency- Spark SQL: Concurrency via many executors/tasks, but heavyweight per-query resource usage; many concurrent interactive queries can contend for cluster resources unless using multi-tenant scheduling (YARN/Kubernetes, Fair Scheduler) and query isolation.- Postgres: Handles many small concurrent connections well (connection pooling required), but heavy concurrent analytical queries can saturate CPU/IO on the single host.

Handling large datasets- Spark SQL: Can process TB–PB data stored across distributed storage (HDFS, S3). Scales horizontally by adding nodes; tolerates node failures.- Postgres: Limited by node capacity; for very large datasets you need sharding, partitioning, or a distributed SQL layer.

When Spark SQL provides advantages- Very large-scale ETL, batch transforms, feature engineering across TBs of data.- Complex distributed joins, wide transformations, iterative ML pipelines integrated with Spark ML.- Data lake processing directly on S3/HDFS, schema-on-read, and integration with Hadoop ecosystem.- When you need horizontal scalability and fault tolerance over raw low-latency point queries.

When Postgres is better- Low-latency interactive queries, transactional workloads, small/medium analytical datasets, and when strong SQL compliance, ACID, and mature tooling are priorities.

As a data engineer, choose based on data size, latency needs, concurrency patterns, and operational complexity—often both coexist: Spark for heavy batch/ETL and Postgres (or columnar warehouse) for low-latency analytics and serving.

MediumTechnical

90 practiced

Explain checkpointing versus write-ahead logs (WAL) in Spark Streaming contexts. What does each persist, when is each used, and how do they help with recovery? Describe how Structured Streaming uses checkpoints and what you must include in a reliable checkpoint directory.

Sample Answer

Checkpointing and write-ahead logs (WAL) are complementary fault-tolerance mechanisms in Spark Streaming.

WAL (Write-Ahead Log)- What it persists: raw input data or received records before processing (e.g., received Kafka/RDD records for DStreams).- When used: enabled when you need end-to-end exactly-once or at-least-once semantics for sources that don’t provide offsets reliably. For older DStream APIs, Spark can write received data to a durable log (HDFS/S3) before processing.- Recovery role: on executor/driver failure, WAL is replayed to re-create lost input data so processing can resume without data loss or duplication (depending on semantics).

Checkpointing- What it persists: metadata about the streaming application — graph, state (for stateful ops like updateStateByKey or mapWithState), and offsets/commit info. In Structured Streaming, it stores query progress, committed sink offsets, and state store snapshots.- When used: to recover driver failures, reconstruct the streaming query and restore state; required for stateful operations and for exactly-once sinks in Structured Streaming.- Recovery role: on restart, Spark reads checkpoint metadata and state to resume processing exactly where it left off (avoids re-running earlier batches).

Structured Streaming specifics- Structured Streaming uses a checkpoint directory for: query metadata (logical plan, offsets, progress logs), committed sink information, and RocksDB-backed state directories (for stateful operators). It implements atomic commit of output via metadata and uses checkpointed offsets to avoid duplicates.- Reliable checkpoint directory must be on a durable, distributed filesystem (HDFS, S3 with consistent semantics, or cloud storage with strong consistency). Include: checkpoint.parquet or metadata logs, offsets/progress logs, and the state store files. The directory must be stable (same path) across restarts and accessible by the application principal. For S3 use EMRFS/consistent S3 or use filesystem layering to avoid eventual-consistency pitfalls.

Best practices- Put checkpoints and WAL on durable, low-latency storage with strong consistency.- Keep checkpoints small: avoid writing large external artifacts into the checkpoint path.- Regularly back up/monitor checkpoint liveness and permissions.- For Kafka sources prefer using Kafka offsets + checkpointing (Structured Streaming’s Kafka source commits offsets atomically) instead of custom WAL where possible.

EasyTechnical

43 practiced

Explain the differences between RDDs, DataFrames, and Datasets in Apache Spark. For each abstraction describe: API characteristics (typed vs untyped), serialization and optimizer support, performance trade-offs, common use cases in production ETL, and language support differences (Scala/Java/Python). Give examples of when you would pick one over the others.

Sample Answer

RDDs, DataFrames, and Datasets in Spark serve different abstraction levels. Here's a concise comparison focused on a Data Engineer's ETL needs.

API characteristics- RDD: Low-level, untyped (actually weakly typed — returns JVM objects), functional API (map/filter). No schema.- DataFrame: High-level, untyped (Row-based), tabular with schema (like SQL table).- Dataset: High-level, typed (Dataset[T]) in Scala/Java; combines type-safety with a schema.

Serialization & optimizer support- RDD: Uses Java/Scala serialization (or Kryo if configured). No Catalyst optimizer or Tungsten codegen — user controls transformations.- DataFrame: Uses Catalyst optimizer and Tungsten physical optimizations, efficient off-heap binary format, whole-stage codegen.- Dataset: Also benefits from Catalyst/Tungsten; in Scala/Java keeps encoder/decoder (Encoders) to convert between JVM objects and Spark's binary format.

Performance trade-offs- RDD: Flexible but slow and memory-inefficient for SQL-like workloads; higher GC/serialization overhead.- DataFrame: Best performance for SQL/ETL due to optimizer, vectorized I/O, reduced allocation.- Dataset: Near DataFrame performance for supported types; typed operations may incur encoder overhead for complex objects.

Common production ETL use cases- RDD: Custom partitioning, complex low-level transforms, legacy code, or when you must control data movement precisely.- DataFrame: Bulk ETL, joins, aggregations, SQL transforms, reading/writing Parquet/ORC — typical production pipelines.- Dataset: ETL where compile-time type safety matters (Scala flows), complex domain objects with some benefit of optimizer.

Language support differences- Scala/Java: Full support for RDD/DataFrame/Dataset; Dataset[T] is idiomatic in Scala/Java.- Python: RDD and DataFrame available; no typed Dataset API — PySpark DataFrame is untyped and still benefits from Catalyst.- Spark SQL: accessible in all languages via DataFrame API.

When to pick which- Choose DataFrame for most ETL: best performance, interoperability, concise SQL-like operations.- Choose Dataset (Scala/Java) when you want type safety and case-class transforms without sacrificing optimizer benefits.- Choose RDD only for very low-level control, custom partitioners, or when working with operations not expressible with DataFrame/Dataset.

Example: reading millions of Parquet files, cleaning, joining, and writing partitioned Parquet → DataFrame. Mapping domain objects and performing type-checked transformations in a Scala pipeline → Dataset[MyCaseClass]. Implementing a custom shuffle/partition algorithm for a specialized algorithm → RDD.

MediumTechnical

50 practiced

Write PySpark code that performs an efficient join between a large DataFrame users_df (millions of rows) and a small dimension DataFrame countries_df (hundreds of rows) by broadcasting the small table. Provide the code and explain how to verify that broadcast join happened using explain() or the Spark UI.

Sample Answer

Approach: explicitly broadcast the small dimension so Spark uses a broadcast hash join (avoids shuffle of the large users_df). Use pyspark.sql.functions.broadcast and confirm with df.explain(True) or by inspecting the Spark UI’s SQL / Stage details for a BroadcastHashJoin and a small broadcast size.

python

from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast

spark = SparkSession.builder.appName("broadcast-join-example").getOrCreate()

# assume users_df (millions) and countries_df (hundreds) already loaded
# ensure key columns have same type
users_df = users_df.withColumn("country_code", users_df["country_code"].cast("string"))
countries_df = countries_df.withColumn("country_code", countries_df["country_code"].cast("string"))

# explicit broadcast of small dimension
joined = users_df.join(broadcast(countries_df), on="country_code", how="left")

# trigger job and show a few rows
joined.select("user_id", "country_code", "country_name").limit(10).show()

How to verify:- Call joined.explain(True) and look for a physical plan entry like "BroadcastHashJoin" (or "BroadcastNestedLoopJoin" for certain conditions). You should also see a Broadcast hint with size (e.g., BroadcastExchange).- In Spark UI (SQL tab / Job details) open the executed query, inspect the physical plan and stages: one stage will show a BroadcastExchange (small table) and the join stage will be a BroadcastHashJoin. Also check Spark driver/executor logs for broadcast size and that no large shuffle occurred.

Notes & best practices:- If automatic broadcast is enabled, Spark may broadcast small tables if under spark.sql.autoBroadcastJoinThreshold (default 10MB). You can set spark.conf.set("spark.sql.autoBroadcastJoinThreshold", <bytes>) to tune.- Make sure the small table truly fits in driver/executor memory; otherwise broadcasting can OOM. Validate countries_df.count() and approximate size before broadcasting.

Unlock Full Question Bank

Get access to hundreds of Big Data Technologies Apache Spark interview questions and detailed answers.

Join thousands of developers preparing for their dream job.