InterviewStack.io LogoInterviewStack.io
Browse more Data Engineer jobs

Data Engineer

Io Tech Solutions Limited

Hong Kong, Hong Kong SAR, Hong Kong1 month ago
100 views52 saves12 applies

Prepare for this role


Job Type

full time

Description

About the role:


  • You are pioneering and innovative and want to be part of the cutting-edge and disruptive crypto-currency world
  • You are eager to learn new knowledge in both financial and technical fields
  • You thrive in a non-hierarchical organization with a casual working environment
  • You enjoy solving complex distributed systems challenges and optimizing streaming data pipelines
  • You value comprehensive documentation and collaborative problem-solving

As a Data Engineer you will:



  • Build real-time data pipelines using Apache Flink (PyFlink) to process high-volume logs through EC2 Vector MSK PyFlink S3 ClickHouse architecture
  • Design stream processing systems with watermark strategies, window operations, exactly-once semantics, and state management for critical data
  • Deploy and manage AWS infrastructure including Managed Flink, S3 data lakes, MSK, IAM, and CloudWatch monitoring with optimized partitioning strategies
  • Optimize performance and troubleshoot SQL queries (Flink/ClickHouse), production issues, data skew, and build Grafana dashboards for pipeline monitoring
  • Implement data quality frameworks with schema evolution, validation strategies, and translate business requirements into scalable data solutions
  • Manage DevOps processes including JAR dependencies, Docker containerization, Kubernetes deployments, and comprehensive documentation

Qualifications:



  • University degree in Computer Science, Software Engineering or related disciplines
  • Apache Flink/PyFlink with watermarks, state management, and window operations
  • Python (pandas, polars, boto3) and SQL (Flink SQL, ClickHouse)
  • Kafka/AWS MSK and message streaming concepts
  • AWS services: Managed Flink, S3, MSK, CloudWatch, IAM
  • Docker, Kubernetes, and containerization fundamentals
  • Grafana monitoring and production troubleshooting experience

This job is found at InterviewStack.io

Skills

distributed systemsdata pipelinesapacheflinkec2s3clickhouseawsiamcloudwatchmonitoringsqlgrafanadashboardsdockercontainerizationkubernetespythonpandaspolarskafkadata quality