Jr. and Sr. Data Engineer retail

Background We're looking for a Junior and Senior Data Engineer for our retail client to work on a foundational customer data product that consolidates and harmonises sales and payment transactional data from multiple source systems. It covers the sales transaction lifecycle from the initial payment attempt through order creation and handover to the fulfilment system. The product integrates online and in-store sales, payments and gift-card transactions into a consistent and accessible source of data. It enables analysts and other data consumers to understand customer behaviour, monitor business performance and develop analytical products without spending unnecessary time finding, combining and cleaning data. Built according to data-mesh principles, the product supports domain-driven ownership and federated governance. The product focuses on data accuracy, consistency, accessibility and usability, providing trusted, ready-for-consumption data through BigQuery. It also supports the wider modernisation journey by providing harmonised sales and payment data to analytical products and business teams. Assignment The consultant will join the team as a Senior Data Engineer and contribute to the design, development and operation of a foundational data product that consolidates and harmonises customer sales, payment, order and gift-card transactional data. The assignment includes building reliable batch, streaming and near-real-time data pipelines; developing reusable data models; improving data quality, monitoring and resilience; and supporting the secure and accessible delivery of trusted data to analysts and other consumers. As a senior team member, the consultant will help guide architectural decisions and strengthen engineering practices. The consultant will bring practical Databricks and lakehouse expertise, including Apache Spark, Delta Lake, Unity Catalog and Structured Streaming, while working with and supporting the current GCP-based technology stack. Day-to-Day Tasks - Design, build, test and operate batch, streaming and near-real-time data pipelines. - Ingest and harmonise sales, payment, order and gift-card data from multiple source systems. - Develop transformations and production data models using SQL, Python, BigQuery and dbt. - Apply Databricks and Spark experience to scalable processing, streaming, governance and lakehouse-related design decisions. - Implement validation, reconciliation, monitoring and automated data-quality controls. - Build resilient processing with error handling, recovery mechanisms and dead-letter queues. - Monitor data freshness, pipeline reliability, performance and agreed service levels. - Protect sensitive and personally identifiable information through secure data modelling and access controls. - Contribute to automated testing, CI/CD, code reviews and infrastructure automation. - Improve metadata, lineage, documentation and the experience of data consumers. - Mentor colleagues and collaborate with source teams, analysts, product colleagues, privacy specialists and security stakeholders. Requirements - Senior-level experience designing, building and operating production data platforms and business-critical data products. - Strong hands-on Databricks experience, including Apache Spark, Delta Lake, Unity Catalog, Structured Streaming, Databricks Workflows and lakehouse architecture. - Strong Python and advanced SQL skills. - Experience with distributed data processing and scalable batch and streaming pipelines. - Strong data-modelling skills and experience developing reliable, reusable production data models. - Experience with data-quality validation, reconciliation, monitoring and alerting. - Knowledge of event-driven architecture, change data capture, schema evolution and resilient error-handling patterns. - Experience with Git, automated testing, CI/CD and modern software-engineering practices. - Knowledge of data security, IAM, privacy, PII handling, metadata and data lineage. - Strong communication skills and the ability to guide architectural decisions, mentor colleagues and collaborate with technical and business stakeholders. Desired - Experience with GCP, particularly BigQuery, Dataflow, Pub/Sub, Cloud Run and Cloud Storage. - Experience with dbt, Apache Beam, Terraform and GitHub Actions. - Experience working with sales, payments, orders or other high-volume transactional data. - Understanding of data-mesh principles, domain ownership and federated governance. - Experience in a product-oriented Agile or DevOps environment. Top 3 Priorities - Extensive hands-on Databricks expertise, particularly Apache Spark, Delta Lake, Unity Catalog, Structured Streaming and lakehouse architecture. - Senior-level experience designing and operating reliable, scalable batch and streaming data pipelines for business-critical transactional data. - Strong technical leadership, including architectural decision-making, mentoring and clear collaboration with technical and business stakeholders. Let op: vacaturefraude Helaas komt vacaturefraude steeds vaker voor. We waarschuwen je voor mogelijke misleiding: * Wij zullen nooit via WhatsApp of in een videogesprek vragen om jouw persoonlijke gegevens (zoals een kopie van je ID, bankgegevens of BSN). * Twijfel je over de echtheid van een vacature of contactpersoon? Neem dan altijd rechtstreeks contact met ons op via de officiële contactgegevens op onze website. Important: job fraud Unfortunately, job fraud is becoming more common. Beware of such scams: * We will never ask for personal information (such as a copy of your ID, bank details, or social security number) via WhatsApp or during a video call. * If you're unsure whether a vacancy or contact person is legitimate, please reach out to us directly using the official contact details on our website.