Job description
Key responsibility is to design and develop a data pipeline including the architecture, prototyping, and development of data extraction, transformation/processing, cleansing/standardizing, and loading in Data Warehouse at real-time/near the real-time frequency. Source data can be structured, semi-structured, and/or unstructured format.
Provide technical expertise to design efficient data ingestion solutions to consolidate data from RDBMS, APIs, Messaging queues, weblogs, images, audios, documents, etc of Enterprise Applications, SAAS applications, external 3rd party sites or APIs, etc through ETL/ELT, API integrations, Change Data Capture, Robotic Process Automation, Custom Python/Java Coding, etc
Development of complex data transformation using Talend (BigData edition), Python/Java transformation in Talend, SQL/Python/Java UDXs, AWS S3, etc to load in OLAP Data Warehouse in Structured/Semi-structured form
Development of data model and creating transformation logic to populate models for faster data consumption with simple SQL.
Implementing automated Audit & Quality assurance checks in Data Pipeline
Document & maintain data lineage to enable data governance
Coordination with BIU, IT, and other stakeholders to provide best-in-class data pipeline solutions, exposing data via APIs, loading in down streams, No-SQL Databases, etc
Requirements
Programming experience using Python / Java, to create functions / UDX
Extensive technical experience with SQL on RDBMS (Oracle/MySQL/Postgresql etc) including code optimization techniques
Strong ETL/ELT skillset using Talend BigData Edition. Experience in Talend CDC & MDM functionality will be an advantage.
Experience & expertise in implementing complex data pipelines, including semi-structured & unstructured data processing
Expertise to design efficient data ingestion solutions to consolidate data from RDBMS, APIs, Messaging queues, weblogs, images, audios, documents, etc of Enterprise Applications, SAAS applications, external 3rd party sites or APIs, etc through ETL/ELT, API integrations, Change Data Capture, Robotic Process Automation, Custom Python/Java Coding, etc
Good understanding & working experience in OLAP Data Warehousing solutions (Redshift, Synapse, Snowflake, Teradata, Vertica, etc) and cloud-native Data Lake (S3, ADLS, BigQuery, etc) solutions
Familiarity with AWS tool stack for Storage & Processing. Able to recommend the right tools/solutions available to address a technical problem
Good knowledge of database performance and tuning, troubleshooting, query optimization, and tuning
Good analytical skills with the ability to synthesize data to design and deliver meaningful information
Good knowledge of Design, Development & Performance tuning of 3NF/Flat/Hybrid Data Model
Know-how on any No-SQL DB (DynamoDB, MongoDB, CosmosDB, etc) will be an advantage.
Ability to understand business functionality, processes, and flows
Good combination of technical and interpersonal skills with strong written and verbal communication; detail-oriented with the ability to work independently
Functional knowledge
Data Governance & Quality Assurance
Distributed computing
Linux
Data structures and algorithm
Unstructured Data Processing
Key skills
How to apply
- Sign in or create your free profile — takes about two minutes.
- Apply with one tap — your profile is shared directly with Multi Recruit.
- Track your application and get updates by email and WhatsApp.