Data Engineering & Analytics Infrastructure Topics
Data pipeline design, ETL/ELT processes, streaming architectures, data warehousing infrastructure, analytics platform design, and real-time data processing. Covers event-driven systems, batch and streaming trade-offs, data quality and governance at scale, schema design for analytics, and infrastructure for big data processing. Distinct from Data Science & Analytics (which focuses on statistical analysis and insights) and from Cloud & Infrastructure (platform-focused rather than data-flow focused).
Data Reliability and Fault Tolerance
Designing pipelines that survive failures: retries, idempotency, checkpointing, exactly-once semantics, dead-letter handling, and recovery/replay. Covers reasoning about partial failures, poison messages, and consistency guarantees under faults. The resilience angle distinct from monitoring (detecting and alerting on a failure) and from Workflow Orchestration and Scheduling (the DAG/scheduler mechanics that decide whether and when a task runs again, including backfills and dependency management): this topic owns whether the data itself stays correct, not lost, not duplicated, not corrupted, when a process is retried or replayed.
Data Pipeline Monitoring and Observability
Observing pipeline health: freshness, volume, schema, and distribution monitoring; lineage; alerting; and data-downtime detection. Covers instrumenting pipelines, defining SLAs/SLOs for data, and observability tooling. The operational-visibility discipline for data platforms.
Workflow Orchestration and Scheduling
Orchestrating multi-step data workflows with DAG schedulers (Airflow, Dagster, and similar tools): dependency management between tasks, scheduling strategies (cron-based, sensor and trigger patterns, event-driven runs), and backfills or catch-up runs for time-partitioned data. Covers task-level retries and idempotent task design, so a scheduler can safely re-run a failed step, plus SLA tracking and alerting when a run is late or missing. The core concern is coordination: given a set of dependent tasks that must run in some order on some schedule, how do you trigger, sequence, and re-run them reliably. This is distinct from whether the data itself stays correct across a failure (exactly-once processing, deduplication, checkpointing, and dead-letter handling for corrupted or poison messages, which is a data-consistency concern) and from how a specific compute engine executes a task internally (Spark or Hadoop mechanics). The operational glue of a data platform: getting the right task to run at the right time, in the right order, with visibility into failures.
Stream Processing and Event Streaming
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
Batch, Streaming, and Real-Time Serving Trade-offs
Reasoning about when to use batch, micro-batch, or continuous streaming and how to serve low-latency analytics: latency, cost, complexity, and correctness trade-offs; lambda vs kappa architectures; and reprocessing semantics. Covers real-time aggregation, freshness vs consistency trade-offs, and reconciling streaming results with batch ground truth, including geospatial and high-throughput real-time workloads under eventual consistency. The data-systems judgment topic for choosing and reconciling batch versus real-time approaches, distinct from the hands-on streaming transport itself.
Data Pipeline Scalability and Performance
Diagnosing and fixing throughput, cost, and capacity problems in data pipelines: batch ETL/ELT jobs and streaming ingestion that must handle growing data volume within cost and SLA constraints. Covers bottleneck diagnosis (I/O-bound versus CPU-bound stages, profiling a slow pipeline); partition and key design as a load-distribution DECISION for a processing job (choosing keys, avoiding and mitigating hot partitions); recognizing when a transformation forces an expensive shuffle or join and choosing a mitigation strategy at the decision level, not engine-internal tuning; capacity planning anchored in real data volume (cluster, storage, and network sizing from an event rate and payload profile); and throughput techniques including incremental processing, result caching and materialization, storage-format and compaction choices for throughput, and cost-efficient scaling (autoscaling policy, spot/reserved capacity trade-offs). Also covers multi-tenant resource governance for shared data-processing infrastructure (quotas, isolation, throughput enforcement). Distinct from distributed-data-processing-with-spark-and-hadoop (engine-internal execution mechanics of a specific framework), data-reliability-and-fault-tolerance (correctness and consistency guarantees under failure, e.g. exactly-once semantics and checkpointing), workflow-orchestration-and-scheduling (DAG/dependency coordination), and general application or service performance tuning (web request latency, generic distributed-systems architecture with no data-pipeline framing).