Build a scalable, reliable, operable and performant big data platform for both streaming and batch analytics
Design and implement data aggregation, cleansing and transformation layers
Skills :
Around 4+ years of hands-on experience designing and operating large data platforms
Experience in Big data Ingestion, Transformation and stream/batch processing technologies using Apache NiFi, Apache Kafka, Kafka Connect (Confluent), Sqoop, Spark, Storm, Hive etc;
Experience in designing and building streaming data platforms in Lambda, Kappa architectures
Should have working experience in one of NoSQL, OLAP data stores like Druid, Cassandra, Elasticsearch, Pinot etc;
Experience in one of data warehousing tools like RedShift, BigQuery, Azure SQL Data Warehouse
Exposure to other Data Ingestion, Data Lake and querying frameworks like Marmaray, Kylo, Drill, Presto
Experience in designing and consuming microservices
Exposure to security and governance tools like Apache Ranger, Apache Atlas
Any contributions to open source projects a plus
Experience in performance benchmarks will be a plus