The candidate should have extensive experience in designing and developing scalable data pipelines and real-time data processing solutions. As a key member of the team, the Senior Data Engineer will play a critical role in building end-to-end data workflows, supporting machine learning model deployment, and driving MLOps practices in a fast-paced, agile environment. Strong expertise in Apache Kafka, Apache Flink, AWS SageMaker, and Terraform is essential. Additional experience with infrastructure automation and CI/CD for ML models is a significant advantage.Key ResponsibilitiesDesign, develop, and maintain high-performance ETL and real-time data pipelines using Apache Kafka and Apache Flink.Build scalable and automated MLOps pipelines for training, validation, and deployment of models using AWS SageMaker and associated services.Implement and manage Infrastructure as Code (IaC) using Terraform to provision and manage AWS environments.Collaborate with data scientists, ML engineers, and DevOps teams to streamline model deployment workflows and ensure reliable production delivery.Optimize data storage and retrieval strategies for large-scale structured and unstructured datasets.Develop data transformation logic and integrate data from various internal and external sources into data lakes and warehouses.Monitor, troubleshoot, and enhance performance of data systems in a cloud-native, fast-evolving production setup.Ensure adherence to data governance, privacy, and security standards across all data handling activities.Document data engineering solutions and workflows to facilitate cross-functional understanding and ongoing maintenance.
