Data Engineer

Responsibilities:

Must be able to write quality code and build secure, highly available systems.

Assemble large, complex datasets that meet functional / non-functional business requirements.

Identify, design, and implement internal process improvements: automating manual processes, optimizing datadelivery, re-designing infrastructure for greater scalability, etc with the guidance.

Create datatools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.

Monitoring performance and advising any necessary infrastructure changes.

Defining dataretention policies.

Implementing the ETL process and optimal data pipeline architecture

Build analytics tools that utilize the datapipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.

Create design documents that describe the functionality, capacity, architecture, and process.

Develop, test, and implement datasolutions based on finalized design documents.

Work with dataand analytics experts to strive for greater functionality in our data

Proactively identify potential production issues and recommend and implement solutions

Skillsets:

Good understanding of optimal extraction, transformation, and loading of datafrom a wide variety of data sources using SQL and AWS ‘big data’ technologies.

Proficient understanding of distributed computing principles

Experience in working with batch processing/ real-time systems using various open-source technologies like NoSQL, Spark, Pig, Hive, Apache Airflow.

Implemented complex projects dealing with the considerable datasize (PB).

Optimization techniques (performance, scalability, monitoring, etc.)

Experience with integration of datafrom multiple data sources

Experience with NoSQL databases, such as HBase, Cassandra, MongoDB, etc.,

Knowledge of various ETL techniques and frameworks, such as Flume

Experience with various messaging systems, such as Kafka or RabbitMQ

Good understanding of Lambda Architecture, along with its advantages and drawbacks

Creation of DAGs for dataengineering

Expert at Python /Scala programming, especially for dataengineering/ ETL purposes