Lead Data Engineer is responsible for designing, building, and maintaining the data infrastructure that enables organizations to collect, store, process, and analyze large volumes of data. They create the data pipelines and systems that ensure data flows smoothly and reliably from various sources to databases, analytics platforms, and applications.

Build and maintain data pipelines for ingesting, transforming, and delivering data. Design and manage data lakes, warehouses, and storage systems for analytics. Develop optimized data models that support reporting and business insights. Ensure data quality, data validation, and governance across all data systems. Collaborate with analysts, data scientists, and business teams to deliver reliable datasets. Use cloud and big data technologies (Azure/AWS/GCP, Spark, Databricks) for scalable processing. Monitor, troubleshoot, and optimize pipelines for performance and cost efficiency. Implement DevOps practices like CI/CD, version control, and automation. Apply strong security practices including access control and compliance. Document data flows, models, and processes to maintain clarity and best practices.

Technical Skills Data pipelines & ETL/ELT: Building automated data workflows using tools like ADF, Airflow, dbt. Programming: Strong SQL and Python (often Spark/PySpark). Big Data: Experience with Spark, Databricks, and distributed processing. Cloud Platforms: Azure, AWS, or GCP data services (Data Lake, Redshift, BigQuery, etc.). Data Storage & Modelling: Data lakes, warehouses, Dimensional modelling, Delta Lake. Databases: SQL (PostgreSQL, SQL Server) + NoSQL (MongoDB, DynamoDB). DevOps & Automation CI/CD, Git, Docker, Terraform Monitoring and pipeline reliability Data Governance Data quality, metadata management, security, and compliance Soft Skills Problem solving, cross-team collaboration, clear communication, documentation.