Data Engineer Azure Databricks|46 YOE| Immediate Joiner

Company Description Codians.ai is a technology platform focused on IT outsourcing, talent hiring, and transparent reviews of IT companies and professionals. The company connects businesses with vetted IT experts, including developers, designers, and consultants, to support a wide range of project needs.

Through a user-friendly interface and a large pool of trusted professionals, Codians.ai simplifies project outsourcing and hiring decisions. The platform emphasizes transparency, ratings, and feedback to help organizations find reliable partners while enabling IT professionals to showcase their skills and build their reputation. Codians.ai is committed to fostering an innovative, efficient, and quality-driven community in the IT industry.

Key Responsibilities

Design and build scalable

batch and real-time data pipelines

using Azure Databricks, PySpark, Python, SQL, and Delta Lake.

Implement

Medallion Architecture (Bronze/Silver/Gold)

for scalable and maintainable data platforms.

Develop governed data products using

Databricks Unity Catalog

.

Build real-time and streaming solutions using

Databricks Structured Streaming and Azure Event Hubs

.

Work extensively with

ADLS Gen2, Azure Key Vault, Azure DevOps, and Git

.

Configure and manage

Databricks Workflows/Jobs

for data pipeline orchestration.

Own

CI/CD pipelines, deployment automation, release processes, monitoring, and production troubleshooting

.

Optimize Databricks workloads and data pipelines for

performance, scalability, reliability, and cost efficiency

.

Build analytical datasets and dashboards using suitable

BI and visualization tools

.

Develop reusable data products capable of supporting

analytics, machine learning, GenAI, RAG, and agentic applications

.

Use

AI coding assistants and AI agents

for development, testing, documentation, debugging, troubleshooting, and engineering automation.

Implement appropriate cloud security, access-control, data-governance, and operational practices.

Collaborate with architects, analysts, business stakeholders, and engineering teams to deliver production-ready solutions.

Take ownership of production issues and ensure reliable operation of deployed data solutions.

Mandatory Skills & Experience

46 years

of hands-on experience in Data Engineering, Data Platforms, or related roles.

Strong hands-on expertise in

Azure Databricks

.

Strong programming skills in

Python and PySpark

.

Strong

SQL

skills, including complex data transformation and optimization.

Hands-on experience with

Delta Lake

.

Strong understanding and implementation experience with

Medallion Architecture Bronze, Silver, and Gold layers

.

Experience developing both

batch and real-time/streaming data pipelines

.

Hands-on experience with

Databricks Unity Catalog

.

Experience with

Databricks Workflows/Jobs

.

Hands-on experience with

Structured Streaming

.

Experience working with

Azure Event Hubs

.

Hands-on experience with

ADLS Gen2

.

Experience with

Azure Key Vault

and cloud security practices.

Experience with

Azure DevOps and Git

.

Good understanding of

CI/CD, deployment automation, monitoring, and production operations

.

Hands-on experience with at least one

BI/dashboarding or data visualization tool

.

Experience troubleshooting and optimizing data pipelines and Databricks workloads.

Strong problem-solving skills with an ownership-driven approach.

Comfortable using

AI coding assistants/agents

and willing to adopt emerging AI-native engineering practices.

Good to Have

Databricks and/or Microsoft Azure certifications

.

Experience with

Lakeflow / Delta Live Tables (DLT)

.

Knowledge of

Terraform or other Infrastructure-as-Code (IaC) tools

.

Experience with

Apache Kafka

or similar event-streaming platforms.

Experience working with

Databricks on AWS or GCP

in addition to Azure.

Exposure to

Generative AI, RAG, LLM-based solutions, or agentic applications

.

Experience developing data platforms that support ML/AI workloads.

Domain experience in

Telecom, Media, Advertising, or AdTech

. .