AWS Databricks Data Engineer Databricks | PySpark | Spark SQL | AWS Glue

Specialist - Data Engineering

5-7 years

Job Summary

Seeking an experienced Specialist to design build and optimize scalable data pipelines using AWS Databricks and related AWS services

Develop and maintain ETLELT pipelines leveraging Databricks with PySpark and Spark SQL on AWS Design and implement data ingestion frameworks integrating multiple sources such as AWS S3 relational databases and APIs Build and manage Delta Lake tables ensuring ACID compliance schema evolution and time travel capabilities Optimize data storage and processing workflows to ensure high performance and scalability Collaborate with crossfunctional teams to deliver analyticsready datasets aligned with business requirements Implement data validation reconciliation and quality assurance mechanisms within pipelines Ensure secure costefficient and compliant deployment of data solutions following AWS best practices Troubleshoot production issues and perform root cause analysis to drive longterm fixes

Roles and Responsibilities

Lead development and optimization of ETLELT pipelines using Databricks on AWS

Manage Databricks jobs workflows clusters and scheduling to ensure effective operation

Migrate existing data workloads from legacy platforms eg Teradata Hive Talend Glue to Databricks Lakehouse

Tune Spark configurations cluster sizing and caching strategies to optimize workload performance and resource utilization

Integrate Databricks with AWS Glue AWS EMR and other AWS services to support endtoend data processing

Implement and maintain data quality validation and monitoring frameworks

Collaborate with data architects analysts and stakeholders to design scalable data lakehouse architectures

Ensure compliance with AWS security standards including IAM roles policies and data encryption

Participate in version control and CICD processes to support continuous delivery and deployment

Skills

Mandatory Skills : AWS EMR, AWS Glue, AWS S3, Databricks .