Data Engineer

Job Summary We are looking for an experienced Data Engineer to design, build, and maintain a scalable, metadata-driven data ingestion and transformation framework on Google Cloud Platform (GCP). The role will be responsible for developing configuration-driven ingestion pipelines, onboarding multiple source systems, and implementing Raw, Bronze, Silver, and Gold data layers. The Data Engineer will use technologies including BigQuery, Cloud Composer/Apache Airflow, Python, Dataform, Datastream, Pub/Sub, Dataflow, Cloud Run, and Cloud Storage to deliver reliable and reusable data solutions. The ideal candidate will have strong experience in metadata-driven pipeline development, CDC, advanced SQL, Python, data quality, reconciliation, automated testing, and CI/CD. The engineer will work closely with Data Governance and other platform stakeholders to ensure that data pipelines are scalable, governed, auditable, testable, and production-ready. Responsibilities - Design and build a metadata-driven and configuration-driven ingestion framework using reusable pipeline templates. - Design, develop, and maintain the metadata configuration model covering source systems, source objects, load configurations, execution parameters, run logs, audit information, and reprocessing. - Build dynamic Cloud Composer / Apache Airflow DAGs that generate and execute ingestion workflows based on metadata configurations. - Develop reusable ingestion patterns and pipeline generators instead of building separate pipelines for individual source tables. - Onboard multiple source systems, including relational databases, SAP, REST APIs, files, streaming sources, and other enterprise platforms. - Implement data ingestion into Raw and Bronze layers following defined lakehouse architecture and engineering standards. - Implement source-to-target reconciliation to ensure completeness and accuracy of ingested data. - Implement robust operational controls, including error handling, retries, quarantine processing, alerting, monitoring, and audit logging. - Develop replay-by-batch and reprocessing capabilities to support failed loads, historical reloads, and controlled data recovery. - Support batch, incremental, streaming, and Change Data Capture (CDC) ingestion patterns. - Implement Bronze-layer processing, including data typing, cleansing, schema validation, and schema enforcement. - Implement data de-duplication using appropriate source keys, primary keys, or business keys. - Implement soft-delete representation and appropriate handling of deleted source records. - Implement CDC change-history materialization and maintain historical changes where required. - Develop appropriate BigQuery partitioning, clustering, MERGE, and incremental processing strategies. - Perform data compaction and other performance optimization activities where required. - Design and develop Silver and Gold transformation models using Dataform. - Develop reusable transformation components, macros, dependencies, and incremental transformation strategies. - Implement assertions and automated tests for transformation models and ensure no untested transformation reaches production. - Implement in-pipeline data quality checks, validation rules, and automated promotion gates. - Work closely with the Data Governance Consultant to incorporate data quality, governance, lineage, audit, and control requirements into the engineering framework. - Prevent data that fails critical quality requirements from being promoted to downstream layers. - Implement geospatial data ingestion, including GeoJSON processing and conversion to BigQuery GEOGRAPHY. - Ensure appropriate preservation and handling of Spatial Reference System (SRS) information for geospatial datasets. - Implement ingestion and management of semi-structured and unstructured data using Google Cloud Storage and BigQuery object tables. - Follow Git-based development practices, including branching, pull/merge requests, peer reviews, and code reviews. - Develop automated tests as a standard part of pipeline and transformation development. - Integrate data pipelines and Dataform transformations with CI/CD processes. - Document each source-system onboarding, including configuration, mappings, dependencies, reconciliation, operational procedures, and troubleshooting guidance. - Develop and maintain operational runbooks and onboarding documentation that enable ESNAD teams to independently onboard additional data sources. Must-Have Skills - Minimum 5+ years of professional Data Engineering experience. - Minimum 2+ years of hands-on experience building data pipelines on Google Cloud Platform (GCP). - Strong hands-on experience with Google Cloud Platform data engineering services. - Expert-level SQL skills, including complex analytical SQL development. - Strong hands-on experience with Google BigQuery, including data modeling and large-volume data processing. - Strong Python programming skills for data engineering, automation, API integration, validation, and framework development. - Strong hands-on experience with Cloud Composer / Apache Airflow. - Proven experience developing dynamic Airflow DAGs. - Experience developing metadata/configuration-driven DAG generation. - Experience with Airflow sensors, custom operators, task dependencies, scheduling, retries, and failure management. - Proven experience designing and implementing metadata-driven or configuration-driven ingestion frameworks. - Experience developing pipeline generators and reusable ingestion templates, rather than only building individual pipelines for individual tables. - Experience designing metadata configurations for source systems, source objects, load parameters, runtime configurations, audit logs, and reprocessing. - Hands-on experience with Google Cloud Datastream or an equivalent log-based CDC technology. - Strong understanding of CDC patterns, including inserts, updates, deletes, soft deletes, incremental loads, and historical change management. - Experience implementing CDC recovery, reconciliation, replay, and reprocessing patterns. - Hands-on experience with Dataform or dbt. - Strong understanding of transformation model structures, macros, reusable logic, assertions, automated tests, dependencies, and incremental strategies. - Experience developing Raw, Bronze, Silver, and Gold data layers or equivalent medallion/lakehouse architecture. - Experience implementing source-to-target data reconciliation and validation. - Experience implementing exception handling, quarantine processes, retry mechanisms, operational monitoring, and alerting. - Working knowledge of Google Cloud Pub/Sub and event-driven or streaming ingestion patterns. - Working knowledge of Dataflow / Apache Beam for data processing. - Working knowledge of Cloud Run and/or Cloud Data Fusion. - Strong understanding of relational database extraction and database-engine-specific CDC constraints. - Working knowledge of SAP data extraction patterns, including ODP, SLT, or certified SAP connectors. - Experience with REST API ingestion, including authentication, pagination, error handling, and incremental extraction. - Experience with file-based data ingestion and file manifest validation. - Experience implementing automated data quality checks, validation rules, and pipeline quality gates. - Strong knowledge of BigQuery partitioning, clustering, MERGE operations, de-duplication, schema enforcement, and incremental processing. - Strong understanding of data pipeline monitoring, auditability, traceability, and operational support. - Strong experience with Git-based development and version control. - Experience working with peer reviews, code reviews, and controlled development workflows. - Experience writing unit, integration, data quality, or pipeline tests as a standard engineering practice. - Experience implementing or working with CI/CD pipelines for data engineering workloads. - Ability to produce high-quality technical documentation, source onboarding documentation, and operational runbooks. Good-to-Have Skills - Google Cloud Professional Data Engineer certification. - Google Cloud Associate Cloud Engineer certification. - dbt Analytics Engineering Certification. - Advanced hands-on experience with Google Cloud Datastream and enterprise-scale CDC implementations. - Advanced experience with Dataflow / Apache Beam for batch and streaming workloads. - Advanced experience with Pub/Sub and event-driven data architectures. - Advanced BigQuery performance optimization and cost-optimization experience. - Experience designing and implementing enterprise-scale lakehouse or medallion architectures. - Experience designing reusable enterprise data ingestion frameworks and platform accelerators. - Advanced hands-on experience with Dataform, including reusable components, assertions, incremental models, and CI/CD. - Experience designing enterprise-level automated data quality frameworks. - Experience with data governance, metadata management, data lineage, data cataloging, auditability, and traceability. - Strong hands-on experience with SAP ODP, SAP SLT, or certified SAP extraction connectors. - Experience implementing SAP CDC and incremental extraction patterns. - Experience processing GeoJSON and BigQuery GEOGRAPHY data. - Knowledge of geospatial data engineering and Spatial Reference Systems (SRS). - Experience handling semi-structured and unstructured data at enterprise scale. - Experience with Google Cloud Storage and BigQuery object tables. - Experience implementing CI/CD pipelines for Dataform, dbt, Airflow, and other data engineering workloads. - Experience with automated deployments, environment promotion, testing gates, and release-management practices. - Knowledge of cloud security, IAM, service accounts, secrets management, and secure data pipeline design. - Experience implementing monitoring, observability, logging, alerting, and operational dashboards for enterprise data pipelines. - Experience working in large enterprise data transformation or cloud migration programs. - Experience working closely with Data Governance, Data Architecture, Security, DevOps, and Business teams. - Experience mentoring junior and mid-level Data Engineers and establishing engineering standards. - For a Senior Data Engineer / Technical Lead, 8+ years of overall Data Engineering experience is preferred, along with demonstrated technical leadership and architecture ownership. Originally posted on Himalayas