Data Engineer - Python, Apache, Spark

We are a Silicon Valley based start-up, established in 2019 and are recognized as experts in building products and providing R&DandSoftware Development services in wide range of leading-edge technologies such as LTE, 5G,Cloud Services (Public -AWS,AZURE,GCP,Private Openstack) and Kubernetes. Ithasa highly scalable and secured 5G Packet Core Network,orchestrated by ML powered Kubernetes platform,which can be deployed in various multi cloud modealong with a test tool.Headquartered in San Jose, California, we have our R&D centre in Sector V, Salt Lake Kolkata.

Requirements:

Overall 3 to 5 years of experience in designing and implementing complex large scale Software.

Good in Python is must.

Experience in Apache Spark, Scala, Java and Delta Lake

Experience in designing and implementing templated ETL/ELT data pipelines

Expert level experience in Data Pipeline Orchestrationusing Apache Airflow for large scale production deployment

Experience in visualizing data from various tasks in the data pipeline using Apache Zeppelin/Plotly or any other visualization library.

Log management and log monitoring using ELK/Grafana

Git Hub Integration

Technology Stack: Apache Spark, Apache Airflow, Python, AWS, EC2, S3, Kubernetes, ELK, Grafana , Apache Arrow, Java .