Data Engineering & Analytics Infrastructure Topics
Data pipeline design, ETL/ELT processes, streaming architectures, data warehousing infrastructure, analytics platform design, and real-time data processing. Covers event-driven systems, batch and streaming trade-offs, data quality and governance at scale, schema design for analytics, and infrastructure for big data processing. Distinct from Data Science & Analytics (which focuses on statistical analysis and insights) and from Cloud & Infrastructure (platform-focused rather than data-flow focused).
Experimentation Platforms and Infrastructure
Infrastructure for A/B testing and experimentation: assignment/bucketing, metric pipelines for experiments, guardrail and variance-reduction plumbing, and experiment result storage. Covers building the platform that powers trustworthy online experiments at scale. Distinct from the statistics of experiment analysis.
Data Reliability and Fault Tolerance
Designing pipelines that survive failures: retries, idempotency, checkpointing, exactly-once semantics, dead-letter handling, and recovery/replay. Covers reasoning about partial failures, poison messages, and consistency guarantees under faults. The resilience angle distinct from monitoring (detecting and alerting on a failure) and from Workflow Orchestration and Scheduling (the DAG/scheduler mechanics that decide whether and when a task runs again, including backfills and dependency management): this topic owns whether the data itself stays correct, not lost, not duplicated, not corrupted, when a process is retried or replayed.
Data Transformation and Processing Logic
Implementing transformation logic: joins, aggregations, deduplication, pivoting/reshaping, and business-rule application over datasets. Covers writing correct and maintainable transformation code, handling edge cases in the transform layer, and preparing data for downstream consumption. Focuses on the logic of turning raw data into analytics-ready outputs.
Data Governance, Contracts, and Classification
Governing data at scale: data contracts between producers and consumers, schema evolution/compatibility, data classification and sensitivity tagging, access control, and lineage/cataloging. Covers policy, ownership, and compliance-driven controls over data. The governance layer over the technical stack.
Stream Processing and Event Streaming
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
Batch, Streaming, and Real-Time Serving Trade-offs
Reasoning about when to use batch, micro-batch, or continuous streaming and how to serve low-latency analytics: latency, cost, complexity, and correctness trade-offs; lambda vs kappa architectures; and reprocessing semantics. Covers real-time aggregation, freshness vs consistency trade-offs, and reconciling streaming results with batch ground truth, including geospatial and high-throughput real-time workloads under eventual consistency. The data-systems judgment topic for choosing and reconciling batch versus real-time approaches, distinct from the hands-on streaming transport itself.
Data Pipeline Architecture and Design
End-to-end design of data pipelines: source-to-sink flow, staging layers, idempotency, backfills, and reprocessing. Covers choosing between batch and streaming stages, decoupling ingestion from transformation, and designing for evolvability. The foundational systems-design skill for a data engineering interview.