Software Architect - Analytics / Kubernetes

5+ years of industry experience in administering (including setting up, managing, monitoring) data processing pipelines (both streaming and batch) using frameworks such as Kafka Streams, Py Spark, and streaming databases like druid or equivalent like Hive

Strong industry expertise with containerization technologies including kubernetes (EKS/AKS), Kubeflow

Experience with cloud platform services such as AWS, Azure or GCP especially with EKS, Managed Kafka

5+ Industry experience in python

Experience with popular modern web frameworks such as Spring boot, Play framework, or Django

Experience with scripting languages. Python experience highly desirable. Experience in API development using Swagger

Implementing automated testing platforms and unit tests

Proficient understanding of code versioning tools, such as Git

Familiarity with continuous integration, Jenkins

Responsibilities

Architect, Design and Implement Large scale data processing pipelines using Kafka Streams, PySpark, Fluentd and Druid

Create custom Operators for Kubernetes, Kubeflow

Develop data ingestion processes and ETLs

Assist in dev ops operations

Design and Implement APIs

Identify performance bottlenecks and bugs, and devise solutions to these problems

Help maintain code quality, organization, and documentation

Communicate with stakeholders regarding various aspects of solution.

Mentor team members on best practices