Requirements

Pipeline Design & Architecture

▸ Architect and manage scalable ETL/ELT pipelines using Medallion Architecture (Bronze → Silver → Gold)

▸ Design ADF pipelines, data models, and schemas optimised for performance and maintainability

▸ Build near real-time and batch ingestion solutions across structured and semi-structured sources

Data Governance & Quality

▸ Implement data quality checks, validation frameworks, and lineage tracking across pipeline layers

▸ Enforce security controls, access policies, and compliance requirements on ADLS Gen 2 and Azure SQL

▸ Monitor pipeline SLAs, diagnose performance bottlenecks, and own incident resolution

Engineering Excellence

▸ Drive CI/CD adoption for data pipelines using Azure DevOps or equivalent tooling

▸ Define and enforce coding standards, peer-review practices, and documentation norms

▸ Contribute to Al agent development initiatives as a data platform subject-matter expert

Collaboration & Mentorship

▸ Partner with data scientists, analysts, and product managers to translate requirements into reliable data products

▸ Mentor junior engineers through code reviews, design discussions, and structured knowledge sharing

▸ Participate in Agile ceremonies and contribute to sprint planning, backlog refinement, and technical estimation

EXPERIENCE & REQUIREMENTS

Must-Have

▸ 5-6 years of hands-on Data Engineering experience in production environments

▸ Deep expertise in Azure ecosystem — ADLS Gen 2, ADF, Azure SQL Database, and Delta Lake

▸ Strong command of PySpark, Python, and SQL for large-scale data transformation

▸ Proven track record building and operating Medallion Architecture (Bronze/Silver/Gold) on Azure

▸ Experience designing data warehouses and data lake solutions at enterprise scale

▸ Familiarity with CI/CD pipelines, version control (Git), and DevOps practices in a data context

Nice-to-Have

▸ Exposure to AWS or GCP in addition to Azure

▸ Experience supporting or building Al/ML data pipelines and feature stores

▸ Knowledge of streaming frameworks — Kafka, Event Hubs, or Spark Structured Streaming

▸ Azure certifications (DP-203, AZ-900 or equivalent)