Architect - Analytics / K8s

Job Description

We are looking for an experienced engineer with superb technical skills. Primarily be responsible for architecting and building large scale data pipelines that delivers AI and Analytical solutions to our customers. The right candidate will enthusiastically take ownership in developing and managing a continuously improving, robust, scalable software solutions.

Although your primary responsibilities will be around back-end work, we prize individuals who are willing to step in and contribute to other areas including automation, tooling, and management applications. Experience with or desire to learn Machine Learning a plus.

Skills

Bachelors/Masters/Phd in CS or equivalent industry experience

Demonstrated expertise of building and shipping cloud native applications

5+ years of industry experience in administering (including setting up, managing, monitoring) data processing pipelines (both streaming and batch) using frameworks such as Kafka Streams, Py Spark, and streaming databases like druid or equivalent like Hive

Strong industry expertise with containerization technologies including kubernetes (EKS/AKS), Kubeflow

Experience with cloud platform services such as AWS, Azure or GCP especially with EKS, Managed Kafka

5+ Industry experience in python

Experience with popular modern web frameworks such as Spring boot, Play framework, or Django

Experience with scripting languages. Python experience highly desirable. Experience in API development using Swagger

Implementing automated testing platforms and unit tests

Proficient understanding of code versioning tools, such as Git

Familiarity with continuous integration, Jenkins

Responsibilities

Architect, Design and Implement Large scale data processing pipelines using Kafka Streams, PySpark, Fluentd and Druid

Create custom Operators for Kubernetes, Kubeflow

Develop data ingestion processes and ETLs

Assist in dev ops operations

Design and Implement APIs

Identify performance bottlenecks and bugs, and devise solutions to these problems

Help maintain code quality, organization, and documentation

Communicate with stakeholders regarding various aspects of solution.

Mentor team members on best practices