Data Engineering & Analytics Infrastructure Topics
Data pipeline design, ETL/ELT processes, streaming architectures, data warehousing infrastructure, analytics platform design, and real-time data processing. Covers event-driven systems, batch and streaming trade-offs, data quality and governance at scale, schema design for analytics, and infrastructure for big data processing. Distinct from Data Science & Analytics (which focuses on statistical analysis and insights) and from Cloud & Infrastructure (platform-focused rather than data-flow focused).
Data Warehousing and Data Lakes
Architecture of warehouses, data lakes, and lakehouses: storage-compute separation, medallion/zoned layouts, and when each is appropriate. Covers governance of a lake, table formats, and the trade-offs between warehouse-first and lake-first analytics stacks. A core infrastructure-design topic for analytics platforms.
Cloud Data Platforms and Managed Services
Evaluating and choosing among managed cloud data platform PRODUCTS: cloud data warehouses (Snowflake, BigQuery, Redshift, Synapse) as vendor options, the storage-and-compute-separation model as a purchasing and operating decision, serverless versus provisioned compute models, warehouse and streaming-service sizing and capacity planning, concurrency and workload management as a platform operating concern, pricing-model comparison and platform-level cost trade-offs, vendor lock-in and portability, platform-to-platform migration, and the recurring managed-versus-self-managed decision applied to warehouses, databases, streaming, and ETL/orchestration services. Focuses on platform SELECTION and operation as a product, not designing the ingestion pipelines, ETL transform patterns, or streaming processing logic that run on top of a chosen platform, and not a single vendor's certification trivia.
Stream Processing and Event Streaming
Building on event-streaming platforms: Kafka and message queues, event sourcing, partitioning, consumer groups, exactly-once vs at-least-once delivery, and windowing. Covers handling late and out-of-order events, watermarks, and stateful stream operators. The core skill for real-time data engineering.
Batch, Streaming, and Real-Time Serving Trade-offs
Reasoning about when to use batch, micro-batch, or continuous streaming and how to serve low-latency analytics: latency, cost, complexity, and correctness trade-offs; lambda vs kappa architectures; and reprocessing semantics. Covers real-time aggregation, freshness vs consistency trade-offs, and reconciling streaming results with batch ground truth, including geospatial and high-throughput real-time workloads under eventual consistency. The data-systems judgment topic for choosing and reconciling batch versus real-time approaches, distinct from the hands-on streaming transport itself.
Data Pipeline Scalability and Performance
Diagnosing and fixing throughput, cost, and capacity problems in data pipelines: batch ETL/ELT jobs and streaming ingestion that must handle growing data volume within cost and SLA constraints. Covers bottleneck diagnosis (I/O-bound versus CPU-bound stages, profiling a slow pipeline); partition and key design as a load-distribution DECISION for a processing job (choosing keys, avoiding and mitigating hot partitions); recognizing when a transformation forces an expensive shuffle or join and choosing a mitigation strategy at the decision level, not engine-internal tuning; capacity planning anchored in real data volume (cluster, storage, and network sizing from an event rate and payload profile); and throughput techniques including incremental processing, result caching and materialization, storage-format and compaction choices for throughput, and cost-efficient scaling (autoscaling policy, spot/reserved capacity trade-offs). Also covers multi-tenant resource governance for shared data-processing infrastructure (quotas, isolation, throughput enforcement). Distinct from distributed-data-processing-with-spark-and-hadoop (engine-internal execution mechanics of a specific framework), data-reliability-and-fault-tolerance (correctness and consistency guarantees under failure, e.g. exactly-once semantics and checkpointing), workflow-orchestration-and-scheduling (DAG/dependency coordination), and general application or service performance tuning (web request latency, generic distributed-systems architecture with no data-pipeline framing).