Data Engineering & Analytics Infrastructure Topics
Data pipeline design, ETL/ELT processes, streaming architectures, data warehousing infrastructure, analytics platform design, and real-time data processing. Covers event-driven systems, batch and streaming trade-offs, data quality and governance at scale, schema design for analytics, and infrastructure for big data processing. Distinct from Data Science & Analytics (which focuses on statistical analysis and insights) and from Cloud & Infrastructure (platform-focused rather than data-flow focused).
Experimentation Platforms and Infrastructure
Infrastructure for A/B testing and experimentation: assignment/bucketing, metric pipelines for experiments, guardrail and variance-reduction plumbing, and experiment result storage. Covers building the platform that powers trustworthy online experiments at scale. Distinct from the statistics of experiment analysis.
Data Reliability and Fault Tolerance
Designing pipelines that survive failures: retries, idempotency, checkpointing, exactly-once semantics, dead-letter handling, and recovery/replay. Covers reasoning about partial failures, poison messages, and consistency guarantees under faults. The resilience angle distinct from monitoring (detecting and alerting on a failure) and from Workflow Orchestration and Scheduling (the DAG/scheduler mechanics that decide whether and when a task runs again, including backfills and dependency management): this topic owns whether the data itself stays correct, not lost, not duplicated, not corrupted, when a process is retried or replayed.
Data Warehousing and Data Lakes
Architecture of warehouses, data lakes, and lakehouses: storage-compute separation, medallion/zoned layouts, and when each is appropriate. Covers governance of a lake, table formats, and the trade-offs between warehouse-first and lake-first analytics stacks. A core infrastructure-design topic for analytics platforms.
ML Feature Pipelines and Feature Stores
Data infrastructure for machine learning: feature pipelines, feature stores, online/offline consistency, training-serving skew, and data preparation for models. Covers building reliable feature platforms and preventing leakage in the data path feeding models. The data-engineering-for-ML topic.
Data Transformation and Processing Logic
Implementing transformation logic: joins, aggregations, deduplication, pivoting/reshaping, and business-rule application over datasets. Covers writing correct and maintainable transformation code, handling edge cases in the transform layer, and preparing data for downstream consumption. Focuses on the logic of turning raw data into analytics-ready outputs.
Storage Formats, Partitioning, and Serialization
Physical storage layout for analytics data. Covers columnar and row format internals and their trade-offs (Parquet row groups and column chunks, ORC stripes, Avro), serialization choices, compression codec and column-encoding selection, partitioning and clustering or bucketing strategy including partition-key choice and pruning, the small-file problem and compaction, file-size and row-group tuning, and open table formats (Iceberg, Delta Lake, Hudi) at the layout and metadata level: ACID commits, snapshots and time travel, manifest and metadata-tree structure, schema and partition evolution, and the catalog that tracks them. The scope is how data is laid out on disk and what that layout makes cheap or expensive. Tuning the queries that read it, and choosing which managed platform to run it on, are covered separately.
ETL and ELT Design Patterns
Trade-offs between extract-transform-load and extract-load-transform strategies, where transformation logic should live, and when to push compute into the warehouse. Covers incremental vs full loads, change-data-capture, slowly changing dimensions handling in the load path, and tooling (dbt-style transformation layers). Focuses on the processing-strategy decision rather than a specific vendor.
Data Pipeline Monitoring and Observability
Observing pipeline health: freshness, volume, schema, and distribution monitoring; lineage; alerting; and data-downtime detection. Covers instrumenting pipelines, defining SLAs/SLOs for data, and observability tooling. The operational-visibility discipline for data platforms.
Data Quality and Validation
Ensuring correctness and trust in data: validation rules, constraints, completeness/accuracy/timeliness checks, and quality frameworks. Covers designing validation into pipelines, quality gates before publishing, and handling edge cases and real-world dirty data. Central to any data engineering or analytics role.