Big Data Technologies Stack Questions

Overview of big data tooling and platforms used for data ingestion, processing, and analytics at scale. Includes frameworks and platforms such as Apache Spark, Hadoop ecosystem components (HDFS, MapReduce, YARN), data lake architectures, streaming and batch processing, and cloud-based data platforms. Covers data processing paradigms, distributed storage and compute, data quality, and best practices for building robust data pipelines and analytics infrastructure.

HardTechnical

42 practiced

Technical domain-specific (streaming): Design stateful stream joins at scale (500k events/sec) which require time-windowed joins and late-data handling. Describe state storage choices (RocksDB vs in-memory), state TTL configuration, checkpointing strategy, scaling (keyed partitions), and methods to bound state size while maintaining correctness.

Sample Answer

Situation & goal: Build time-windowed, stateful joins at 500k events/sec with late-data handling while keeping state bounded and correct.

Design summary / choices- State backend: Use RocksDB for production at this throughput. Reason: durable on-heap footprint, spill-to-disk for large keyed state, and better GC behavior. In-memory (heap) is only for low-latency tiny-state use (<< memory budget) or testing.- Keying & partitioning: Key-streams by join key and use the cluster’s keyed partitions (Flink parallelism tuned so each task handles ~hundreds of thousands of keys). Ensure key distribution via hash / salt if skewed.- Window semantics & late data: Use event-time tumbling/sliding windows with watermarks. Configure allowedLateness (e.g., 1–5 minutes depending on SLA) and side outputs for “too-late” events for separate handling (audit/reprocess).- State TTL & bounding: Configure state TTL slightly longer than allowedLateness + max out-of-order. Use per-key window state eviction: clear window state when watermark > window_end + allowedLateness. For absolute bounding, combine TTL with compaction: store aggregated/compact representations (counts, sketches) rather than raw lists. For heavy keys, use bloom filters or top-k caches to avoid retaining full detail.- Checkpointing & durability: Enable RocksDB incremental snapshots + async checkpointing. Checkpoint frequency balances RPO and overhead (e.g., 10–30s). Persist checkpoints to durable object store (S3/GCS) and enable externalized checkpoints/savepoints for fast recovery.- Scaling & rebalancing: Use consistent hashing and rescale with savepoints to avoid full rebuilds. For hot keys, implement key-splitting (sharded composite key) and then merge results.- Bounding techniques while keeping correctness: - Watermark-driven eviction ensures correctness for in-order and bounded-lateness streams. - For unbounded lateness, keep late events in a separate path with compensating updates (retractions) and idempotent writes downstream. - Pre-aggregate window inputs to reduce per-key state. - Backpressure / autoscaling: autoscale processing parallelism based on input rate metrics; use adaptive batching or async I/O to smooth spikes.

Why this balances trade-offs- RocksDB + incremental checkpoints keeps memory stable and provides durability; TTL + watermark-driven eviction bounds state with provable semantics for allowed lateness. Sharding and pre-aggregation reduce per-key pressure; side outputs and retractions preserve correctness for extreme late arrivals.

Example parameters to start with (tunable): checkpoint every 15s, RocksDB incremental, allowedLateness = 2 min, TTL = allowedLateness + 5 min, parallelism sized for ~50–100k events/sec per task, monitor heap / RocksDB SST sizes and GC.

HardTechnical

42 practiced

Design an online feature store for low-latency model inference. Describe feature ingestion (streaming and batch), storage choices for online serving (Redis, DynamoDB, RocksDB), freshness SLAs, TTL policies, consistency and atomic updates, and rollback strategies when features are corrected or retrained models require different feature versions.

Sample Answer

Situation: You need an online feature store that serves low-latency features for inference while supporting both streaming and batch ingestion, strong freshness SLAs, safe atomic updates, and clean rollback/versioning.

Design summary- Ingestion - Streaming: ingest via Kafka (or Kinesis) using CDC or event producers. Use stream processors (Flink/Beam) to validate, enrich, compute windowed aggregates, and write to both online store and cold warehouse. Exactly-once processing (Kafka + Flink checkpointing) for correctness. - Batch: periodic Spark jobs compute features from warehouse (Parquet/Delta) and write to online store via bulk-write APIs with versioned payloads and watermarks.- Online storage choices (trade-offs) - Redis (clustered, memory-first): ultra low latency (sub-ms), best for hot joins; use Redis Cluster + Redis Streams for change propagation; use DAX or Redis persistence for recovery. - DynamoDB: single-digit-ms, fully managed, scales for large cardinality, supports conditional writes and TTL; use DAX for caching. - RocksDB (embedded in local inference node): lowest tail latency when co-located; use on-disk LSM with caching — good for very large state per host. - Pattern: tiered approach — Redis for hot keys, DynamoDB for medium, RocksDB local cache for per-host cold reads.- Freshness SLAs & TTLs - Define per-feature SLA (e.g., realtime features < 1s, nearline < 1–5m, historical > 1h). - Store last_updated timestamp with each record; enforce TTL policies: soft TTL (mark stale, route to fallback) and hard TTL (delete after configured retention).- Consistency & atomic updates - Store features as versioned tuples: key = entity_id, feature_family::version. Each write includes (version, updated_at). - Use conditional/compare-and-set semantics where supported (DynamoDB conditional writes, Redis Lua scripts) to ensure atomic multi-feature updates. - For multi-key transactions, use two-phase commit patterns sparingly or write an atomic digest/version entry that aggregates component versions; consumers read digest to ensure consistency.- Rollback & feature correction - Immutable feature versions: never overwrite previous versions; create new version id. Inference can select version via feature-serving config. - For urgent corrections: write a corrected version and advance a global feature version pointer atomically (single key update). This enables instantaneous flip-over. - Backfill: batch recompute corrected features and write versioned entries. Use staged deployment (shadow traffic, canary) where model reads new version flag. - If need revert: flip global pointer back to prior version (atomic update), or route a percentage of traffic. - Auditing & lineage: keep provenance metadata (source offsets, job id) to trace corrections.Operational considerations- Monitoring: freshness metrics (percent within SLA), tail latency, write success, consumer error rates.- Recovery: resilient stream replay using Kafka offsets and idempotent writes.- Security: encryption at rest/in transit, IAM roles per pipeline.This design balances latency, scale, correctness, and operational safety while enabling fast rollbacks and controlled feature evolution.

MediumBehavioral

44 practiced

Behavioral: Tell me about a time you resolved a production data incident where downstream analytics were producing incorrect results. Use the STAR format: describe the situation, the tasks you owned, concrete actions you took (triage, rollback, remediation), how you communicated with stakeholders, and what you changed to prevent recurrence.

Sample Answer

Situation: At my previous company, a nightly Spark job that populated the analytics warehouse failed silently for one partition, causing dashboards and downstream ML features to show stale or incorrect values for ~6% of customers the next morning. Stakeholders (analytics, product, and a revenue ops manager) alerted us after noticing KPI drift.

Task: As the on-call data engineer owner of that pipeline, I had to triage impact, stop further bad data propagation, restore correct data, communicate status, and implement fixes to prevent recurrence.

Action:- Triage: Checked Airflow logs and Spark driver logs; discovered a schema change in the upstream event stream (new nullable field shifted CSV column positions). I reproduced the failure locally using the same partition and confirmed missing rows were dropped by our parsing step.- Containment/Rollback: Paused dependent consumption jobs and marked the affected nightly partition as "do not consume" in the warehouse to avoid analysts using bad data.- Remediation: Reprocessed the affected partition after updating the parser to handle the new schema robustly (explicit column mapping and fallback for unknown columns). Validated row counts and key metric reconciliation against raw event store.- Communication: Sent an incident summary to stakeholders within 60 minutes (impact, actions taken, ETA). Held a short sync with analytics and product to confirm remediation and provide timelines. Posted updates in Slack and closed the loop when reconciliation validated.- Postmortem & Preventive: Wrote a postmortem documenting root cause and added automated schema-change alerts, stricter input validation (fail-fast on unexpected schema unless whitelisted), and a Canary run that compares record counts and checksums before allowing downstream consumption.

Result: Restored correct analytics within 5 hours, prevented inaccurate decisions, reduced recurrence risk—no similar incidents in the following 9 months. The schema-alerting and canary checks caught two later upstream changes before they impacted production.

MediumTechnical

66 practiced

Technical coding (PySpark): Implement sessionization using the PySpark DataFrame API. Given schema (user_id STRING, event_ts TIMESTAMP, event_type STRING), group events into sessions where 30 minutes of inactivity defines a new session. Output: (user_id, session_id, session_start, session_end, event_count). Provide code, explain correctness, and discuss complexity and state requirements.

Sample Answer

Approach: For each user sort events by event_ts, compute the time gap to the previous event. If gap > 30 minutes (1800 seconds) or previous is null, start a new session. Use a running sum of session-start flags to assign a session number, then aggregate per (user, session_number) to get session_start, session_end, and event_count.

python

from pyspark.sql import functions as F
from pyspark.sql.window import Window

# input: df with columns (user_id STRING, event_ts TIMESTAMP, event_type STRING)
# 1. order events per user and find gap to previous event
w = Window.partitionBy("user_id").orderBy("event_ts")
df2 = (df
  .withColumn("prev_ts", F.lag("event_ts").over(w))
  .withColumn("gap_secs", F.when(F.col("prev_ts").isNull(), None)
                          .otherwise(F.unix_timestamp("event_ts") - F.unix_timestamp("prev_ts")))
  .withColumn("new_session", F.when((F.col("prev_ts").isNull()) | (F.col("gap_secs") > 1800), 1).otherwise(0))
)

# 2. assign session number by cumulative sum of new_session per user
w2 = Window.partitionBy("user_id").orderBy("event_ts").rowsBetween(Window.unboundedPreceding, 0)
df3 = df2.withColumn("session_num", F.sum("new_session").over(w2))

# 3. aggregate per user + session_num
out = (df3.groupBy("user_id", "session_num")
       .agg(
         F.min("event_ts").alias("session_start"),
         F.max("event_ts").alias("session_end"),
         F.count("*").alias("event_count")
       )
       .withColumn("session_id", F.concat_ws("_", F.col("user_id"), F.col("session_num")))
       .select("user_id", "session_id", "session_start", "session_end", "event_count")
)

Why correct:- lag+gap identifies boundaries where inactivity > 30min.- cumulative sum turns boundaries into discrete session ids in event order.- final aggregation yields correct start/end/count per session.

Complexity and state:- Requires ordering events per user: global shuffle by user_id then sort within partitions. Cost ~ O(N log M) where M average events per user; overall network + sort dominates.- Space: needs per-partition buffers for sorting and window state (prev_ts and cumulative sum). Memory/state scales with largest user partition; for extremely skewed users consider salting or map-side pre-aggregation.- Works in event-time; for streaming or late data you'd need watermarking and incremental state management (Spark Structured Streaming with mapGroupsWithState or session window API).

MediumTechnical

35 practiced

Compare Lambda and Kappa architectures for combining batch and streaming processing. Explain operational overhead, code duplication, latency vs correctness trade-offs, and a practical migration path from a Lambda architecture (separate batch and serving layers) to a Kappa architecture (stream-first).

Sample Answer

Lambda and Kappa are two patterns for combining batch and stream processing.

Definition & core idea:- Lambda: two paths — a batch layer (accurate, recomputable) and a real-time speed layer plus a serving layer that merges results. Batch provides correctness; stream provides low-latency updates.- Kappa: single stream-processing pipeline that reprocesses history (replay) to achieve correctness; no separate batch codebase.

Operational overhead:- Lambda: higher operational cost—maintain two pipelines, two scheduling systems, separate storage for batch outputs and serving views, and reconciliation logic. More infra, monitoring, and runbooks.- Kappa: simpler infra (one pipeline, one execution engine), fewer systems to monitor, but relies on robust stream processing that supports replay and state snapshots.

Code duplication:- Lambda: significant duplication—transformations implemented twice (batch and streaming) often in different frameworks (Spark batch vs Flink/Storm). This increases bugs and maintenance.- Kappa: single code path reduces duplication; transforms are implemented once in the stream job and applied to both live and replayed data.

Latency vs correctness trade-offs:- Lambda: low latency for fresh data with stream layer; eventual correctness once batch catches up. Complexity in merging results and handling retractions/late data.- Kappa: aims for both: low-latency processing for live events and correctness via replay. Requires idempotency, event ordering handling, durable event log, and ability to rebuild state within acceptable time.

Practical migration path (Lambda → Kappa):1. Audit transforms: catalog all batch and stream transforms; find commonalities.2. Implement streaming-first transforms: refactor batch logic into streaming-compatible operators (idempotent, event-time aware).3. Introduce durable event log (e.g., Kafka) as canonical source; ensure all producers write to it.4. Add replay capability: ensure stream engine can replay from Kafka and rebuild state (checkpointing, compacted topics).5. Run dual-mode: run new Kappa streaming job consuming live stream and replay historical range; continue batch for backfill only.6. Validate: compare outputs (hashes, counts) between batch serving and stream-derived views; fix divergences.7. Cut over serving layer to stream views; decommission batch jobs progressively (keep batch for rare backfills).8. Harden operational controls: add monitoring for lag, consumer offsets, end-to-end tests, and automated rebuild procedures.

When to choose:- Keep Lambda if batch jobs are cheap, transformations are complex and not stream-friendly, or replay time is impractical.- Choose Kappa when low maintenance, unified code, fast recovery via replay, and event log-driven architecture are feasible.

Key practical notes:- Ensure exactly-once or idempotent semantics; handle schema evolution; maintain compacted topics for state bootstrap; benchmark rebuild time to ensure SLAs.

Unlock Full Question Bank

Get access to hundreds of Big Data Technologies Stack interview questions and detailed answers.

Join thousands of developers preparing for their dream job.