Data Engineer (PySpark & Generative AI)

Role Overview

Build scalable data pipelines and platforms with Python, PySpark, and cloud data services to support analytics, reporting, and AI/ML initiatives.

Key Responsibilities

Design, build, and maintain scalable ETL/ELT pipelines for batch and real-time data.

Ingest data from structured and unstructured sources into cloud data lakes and warehouses.

Build data models and warehouses for analytics and reporting, and tune SQL for performance.

Implement data validation, cleansing, transformation, and quality checks.

Work with data scientists, analysts, and application teams, following Git, CI/CD, testing, and monitoring practices.

Must Have Skills

5 - 8 years of data engineering, with strong Python programming.

Hands-on PySpark and Apache Spark on large-scale distributed datasets.

Advanced SQL (query optimization and performance tuning), plus data warehousing and dimensional modeling.

Strong experience with Generative AI and Large Language Models (LLMs).

AWS, Azure, or GCP data services, with Git, CI/CD, and Agile/Scrum.

Nice to Have

Apache Airflow or similar orchestration

Kafka and real-time streaming

Databricks, Delta Lake, Apache Iceberg, or Hudi

Docker, Kubernetes, and Terraform

Data governance and metadata management .