M

Data Pipeline

Multi Recruit Verified employer

Actively hiring · Posted 6d ago · JEV2611FA0F61DFF1

Salary₹20,00,000 – ₹25,00,000 per year
Experience5–7 yrs
LocationMumbai, Maharashtra
Work modeHybrid · Full time
Openings1

Job description

Key responsibility is to design and develop a data pipeline including the architecture, prototyping, and development of data extraction, transformation/processing, cleansing/standardizing, and loading in Data Warehouse at real-time/near the real-time frequency. Source data can be structured, semi-structured, and/or unstructured format.

Provide technical expertise to design efficient data ingestion solutions to consolidate data from RDBMS, APIs, Messaging queues, weblogs, images, audios, documents, etc of  Enterprise Applications, SAAS applications, external 3rd party sites or APIs, etc through ETL/ELT, API integrations, Change Data Capture, Robotic Process Automation, Custom Python/Java Coding, etc

Development of complex data transformation using Talend (BigData edition), Python/Java transformation in Talend, SQL/Python/Java UDXs, AWS S3, etc to load in OLAP Data Warehouse in Structured/Semi-structured form

Development of data model and creating transformation logic to populate models for faster data consumption with simple SQL.

Implementing automated Audit & Quality assurance checks in Data Pipeline

Document & maintain data lineage to enable data governance

Coordination with BIU, IT, and other stakeholders to provide best-in-class data pipeline solutions, exposing data via APIs, loading in down streams, No-SQL Databases, etc

Requirements

Programming experience using Python / Java, to create functions / UDX

Extensive technical experience with SQL on RDBMS (Oracle/MySQL/Postgresql etc) including code optimization techniques

Strong ETL/ELT skillset using Talend BigData Edition. Experience in Talend CDC & MDM functionality will be an advantage.

Experience & expertise in implementing complex data pipelines, including semi-structured & unstructured data processing

Expertise to design efficient data ingestion solutions to consolidate data from RDBMS, APIs, Messaging queues, weblogs, images, audios, documents, etc of  Enterprise Applications, SAAS applications, external 3rd party sites or APIs, etc through ETL/ELT, API integrations, Change Data Capture, Robotic Process Automation, Custom Python/Java Coding, etc

Good understanding & working experience in OLAP Data Warehousing solutions (Redshift, Synapse, Snowflake, Teradata, Vertica, etc) and cloud-native Data Lake (S3, ADLS, BigQuery, etc) solutions

Familiarity with AWS tool stack for Storage & Processing. Able to recommend the right tools/solutions available to address a technical problem

Good knowledge of database performance and tuning, troubleshooting, query optimization, and tuning

Good analytical skills with the ability to synthesize data to design and deliver meaningful information

Good knowledge of Design, Development & Performance tuning of 3NF/Flat/Hybrid Data Model

Know-how on any No-SQL DB (DynamoDB, MongoDB, CosmosDB, etc) will be an advantage.

Ability to understand business functionality, processes, and flows

Good combination of technical and interpersonal skills with strong written and verbal communication; detail-oriented with the ability to work independently

Functional knowledge

Data Governance & Quality Assurance

Distributed computing

Linux

Data structures and algorithm

Unstructured Data Processing

Key skills

How to apply

  1. Sign in or create your free profile — takes about two minutes.
  2. Apply with one tap — your profile is shared directly with Multi Recruit.
  3. Track your application and get updates by email and WhatsApp.
Apply now
You may also like

Similar jobs

More jobs
Data Pipeline₹20,00,000 – ₹25,00,000 /year · Mumbai
Apply