Mid Data Engineer GCP & Spark

Mid Data Engineer GCP & Spark

58 years of experience in Data Engineering, Big Data Engineering, or related roles.

Role Overview

We are seeking a highly skilled Senior Data Engineer to design, develop, and support scalable data pipelines and large-scale data processing solutions within the Google Cloud Platform (GCP) ecosystem. The ideal candidate will have strong hands-on experience with Spark, BigQuery, Dataproc, SQL, and cloud-based data engineering practices.

This role requires a strong understanding of end-to-end data pipeline architecture, system design, debugging, performance optimization, and CI/CD practices. The engineer will work closely with data engineering, analytics, and platform teams to build reliable, scalable, and high-performing data solutions.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Apache Spark (PySpark or Scala).Build and optimize batch and distributed data processing workflows on GCP.Work extensively with BigQuery for data transformation, analytics, validation, and reporting.Develop and maintain data workflows using GCP services such as Dataproc, BigQuery, and Cloud Storage.Design end-to-end data pipeline architectures with a focus on scalability, reliability, and performance.Write efficient and optimized SQL queries for large-scale datasets.Troubleshoot, debug, and resolve complex data processing and performance issues.Implement CI/CD best practices for data engineering workflows.Collaborate with cross-functional teams to gather requirements and deliver high-quality data solutions.Participate in code reviews, testing, deployment, and production support activities.Monitor pipeline performance and identify opportunities for optimization and continuous improvement.Required Skills & Qualifications

Strong hands-on experience with Apache Spark (PySpark and/or Scala).Strong SQL skills with experience working on large-scale datasets.Hands-on experience with Google Cloud Platform (GCP).Practical experience with BigQuery and data warehouse concepts.Experience working with Dataproc or similar distributed data processing platforms.Strong understanding of data pipelines and end-to-end workflow design.Experience designing scalable and reliable data processing solutions.Strong debugging, troubleshooting, and problem-solving skills.Experience with Git-based development workflows.Good understanding of CI/CD practices and deployment processes.Experience with Linux/Unix environments and shell scripting.Preferred Qualifications

Experience with workflow orchestration tools such as Airflow, Cloud Composer, Luigi, or equivalent.Experience with Kafka or other messaging/streaming technologies.Knowledge of monitoring, logging, and observability practices.Experience with Python for automation and data engineering tasks.Exposure to data governance, data quality, and metadata management frameworks.Soft Skills

Strong analytical and problem-solving abilities.Excellent communication and collaboration skills.Ability to work effectively in cross-functional teams.Proactive mindset with a focus on operational excellence and continuous improvement.Strong ownership and accountability in delivering reliable data solutions. .