Lead Azure Databricks Developer ( 9+ Years) (India)

Azure Databricks Developer

Location: India

Work Mode: 100% Remote

Shift: EST Time Zone

Experience: 910+ Years

Engagement: Contract

Job Overview

We are seeking an experienced Azure Databricks Developer with strong hands-on expertise in Azure Data Engineering, Databricks, PySpark, Spark SQL, Delta Lake, Azure Data Factory (ADF), and ADLS Gen2 .

The successful candidate will be responsible for designing, developing, optimizing, and supporting scalable enterprise-grade data engineering solutions using modern Lakehouse Architecture . The role requires strong experience in ETL/ELT development, incremental processing, CDC, Spark performance optimization, and Azure cloud services.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark .
• Build and orchestrate robust ETL/ELT workflows using Azure Data Factory (ADF) .
• Develop ingestion and transformation pipelines for structured and semi-structured data using ADLS Gen2, Delta Lake, and Spark SQL .
• Implement Medallion Architecture (Bronze, Silver, Gold) and modern Lakehouse design patterns.
• Develop incremental data processing, CDC, MERGE operations, and SCD Type 1/Type 2 implementations.
• Optimize Spark workloads using partitioning, caching, file-size optimization, and other performance-tuning techniques.
• Configure and manage Databricks Jobs/Workflows, clusters, and workspace environments .
• Implement data quality checks, validation, reconciliation, error handling, monitoring, and alerting processes.
• Work with Unity Catalog for data governance, lineage, RBAC, and secure data access.
• Integrate Databricks with ADLS, Azure SQL, APIs, JDBC sources, and external systems .
• Develop and maintain CI/CD pipelines using Azure DevOps, Git, and YAML .
• Implement Infrastructure-as-Code using Terraform, ARM, or Bicep .
• Troubleshoot production data pipelines, perform root-cause analysis, and ensure SLA compliance.
• Monitor and optimize workloads for performance, scalability, reliability, and cost efficiency .
• Collaborate with architects, BI teams, data analysts, developers, and business stakeholders in an Agile/Scrum environment .

Required Skills & Qualifications

• 910+ years of experience in Data Engineering / Azure Cloud Data Development.
• Strong hands-on experience with:
• Azure Databricks
• PySpark
• Spark SQL
• Delta Lake
• Azure Data Factory (ADF)
• ADLS Gen2
• SQL
• Strong understanding of Lakehouse Architecture and distributed data processing .
• Proven experience developing enterprise-scale ETL/ELT pipelines .
• Hands-on experience with Medallion Architecture, CDC, MERGE, and SCD Type 1/Type 2 .
• Strong knowledge of Apache Spark performance tuning and optimization .
• Experience with Azure DevOps, Git, and YAML-based CI/CD pipelines .
• Knowledge of Terraform, ARM, or Bicep for Infrastructure-as-Code.
• Solid SQL development and query performance-tuning capabilities.
• Experience working in Agile/Scrum environments .
• Strong troubleshooting and production-support experience.

Preferred Skills

• Experience with Databricks platform administration , including clusters, jobs, workflows, and workspace configuration.
• Hands-on experience with Unity Catalog, RBAC, governance, and data lineage .
• Experience integrating Databricks with APIs, JDBC sources, Azure SQL, and external systems .
• Knowledge of monitoring and observability tools such as Datadog or Grafana .
• Experience with Azure and Databricks cost optimization .
• Experience supporting production-grade data platforms and resolving critical data pipeline issues.

Candidate Profile The ideal candidate should be:

• Strongly hands-on with Azure Databricks and PySpark .
• Experienced in designing and supporting enterprise-scale data pipelines .
• Strong in Spark optimization, SQL, Delta Lake, and Lakehouse architecture .
• Comfortable working in an EST time-zone shift .
• Capable of independently troubleshooting production issues and meeting defined SLAs.
• Comfortable collaborating with cross-functional technical and business teams.
• Experienced in working in fast-paced Agile/Scrum environments .

Technology Stack Azure Databricks | PySpark | Spark SQL | Delta Lake | Azure Data Factory | ADLS Gen2 | Azure SQL | SQL | Unity Catalog | Azure DevOps | Git | YAML | Terraform | ARM | Bicep | Lakehouse Architecture | CDC | SCD Type 1/2

Engagement Details

ParameterDetails Role :- Azure Databricks Developer

Experience 910+ Years Location India Work Mode 100% Remote Shift EST Engagement Contract Joining Preference Immediate Joiners Preferred .