Data Engineering & Analytics Infrastructure Topics
Data pipeline design, ETL/ELT processes, streaming architectures, data warehousing infrastructure, analytics platform design, and real-time data processing. Covers event-driven systems, batch and streaming trade-offs, data quality and governance at scale, schema design for analytics, and infrastructure for big data processing. Distinct from Data Science & Analytics (which focuses on statistical analysis and insights) and from Cloud & Infrastructure (platform-focused rather than data-flow focused).
Data Reliability and Fault Tolerance
Designing pipelines that survive failures: retries, idempotency, checkpointing, exactly-once semantics, dead-letter handling, and recovery/replay. Covers reasoning about partial failures, poison messages, and consistency guarantees under faults. The resilience angle distinct from monitoring (detecting and alerting on a failure) and from Workflow Orchestration and Scheduling (the DAG/scheduler mechanics that decide whether and when a task runs again, including backfills and dependency management): this topic owns whether the data itself stays correct, not lost, not duplicated, not corrupted, when a process is retried or replayed.
Data Warehousing and Data Lakes
Architecture of warehouses, data lakes, and lakehouses: storage-compute separation, medallion/zoned layouts, and when each is appropriate. Covers governance of a lake, table formats, and the trade-offs between warehouse-first and lake-first analytics stacks. A core infrastructure-design topic for analytics platforms.
ML Feature Pipelines and Feature Stores
Data infrastructure for machine learning: feature pipelines, feature stores, online/offline consistency, training-serving skew, and data preparation for models. Covers building reliable feature platforms and preventing leakage in the data path feeding models. The data-engineering-for-ML topic.
Data Ingestion and Source System Integration
Getting data out of heterogeneous source systems and landing it reliably: APIs, operational databases, file drops, webhooks, message queues and third-party SaaS. Covers connector selection and design (managed platforms versus Debezium, DMS or Kafka Connect versus building your own), pull versus push and polling versus webhook patterns, incremental extraction and high-watermark strategy including what to do when a source offers no native change capture, authentication and credential rotation against third-party APIs, source-side rate limits and quotas, schema drift and contract breakage at the source boundary, backfill and replay of history, ingestion-time data-quality gates, reconciliation after a source outage, and negotiating with source-system owners. The scope stops at the boundary: once data has landed, transforming it, the architecture of the pipeline that carries it, stream-processing mechanics, and pipeline monitoring are all covered separately.
Cloud Data Platforms and Managed Services
Evaluating and choosing among managed cloud data platform PRODUCTS: cloud data warehouses (Snowflake, BigQuery, Redshift, Synapse) as vendor options, the storage-and-compute-separation model as a purchasing and operating decision, serverless versus provisioned compute models, warehouse and streaming-service sizing and capacity planning, concurrency and workload management as a platform operating concern, pricing-model comparison and platform-level cost trade-offs, vendor lock-in and portability, platform-to-platform migration, and the recurring managed-versus-self-managed decision applied to warehouses, databases, streaming, and ETL/orchestration services. Focuses on platform SELECTION and operation as a product, not designing the ingestion pipelines, ETL transform patterns, or streaming processing logic that run on top of a chosen platform, and not a single vendor's certification trivia.
Dimensional Modeling and Schema Design
Designing analytical schemas: star and snowflake schemas, fact and dimension tables, grain selection, slowly changing dimensions, and normalization vs denormalization for analytics. Covers modeling business requirements into queryable structures and schema design for large datasets. The data-modeling backbone of BI and warehouse work.
Data Governance, Contracts, and Classification
Governing data at scale: data contracts between producers and consumers, schema evolution/compatibility, data classification and sensitivity tagging, access control, and lineage/cataloging. Covers policy, ownership, and compliance-driven controls over data. The governance layer over the technical stack.
Workflow Orchestration and Scheduling
Orchestrating multi-step data workflows with DAG schedulers (Airflow, Dagster, and similar tools): dependency management between tasks, scheduling strategies (cron-based, sensor and trigger patterns, event-driven runs), and backfills or catch-up runs for time-partitioned data. Covers task-level retries and idempotent task design, so a scheduler can safely re-run a failed step, plus SLA tracking and alerting when a run is late or missing. The core concern is coordination: given a set of dependent tasks that must run in some order on some schedule, how do you trigger, sequence, and re-run them reliably. This is distinct from whether the data itself stays correct across a failure (exactly-once processing, deduplication, checkpointing, and dead-letter handling for corrupted or poison messages, which is a data-consistency concern) and from how a specific compute engine executes a task internally (Spark or Hadoop mechanics). The operational glue of a data platform: getting the right task to run at the right time, in the right order, with visibility into failures.
Stream Processing and Event Streaming
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.