Senior Data Engineer

Job Title: Senior Data Engineer Salary: £70k - £80k Location: Manchester/Lytham St Annes (Hybrid working pattern) Introduction Evolve Energy is scaling rapidly and modernising on Microsoft Azure. We are building a trusted, governed data foundation spanning half-hourly meter reads, industry flows, billing, payments and wholesale positions. As our hands-on Senior Data Engineer, you will help architects and technical leads select the platform, build it’s ingestion, modelling, quality and governance patterns, and set standards for reliable growth. Role Purpose The Senior Data Engineer will design, build and operate Evolve Energy’s core data platform, turning raw, high-volume industry data into reliable, well-modelled and documented data products. Combining hands-on engineering with architectural input, you will work with architects, engineers, analysts, product and operations to deliver early value while keeping the platform sustainable over the long term. They will also set standards, review work and build the team’s data capability. Duties and Responsibilities Build the foundations: - Design ingestion, storage (e.g. lakehouse), modelling and serving; balance capability, cost and operability - Apply database administration experience across SQL, NoSQL and time-series databases, alongside knowledge of open table formats such as Apache Iceberg, to build the platform’s foundations - Build batch and streaming ingestion for high-volume time-series, half-hourly meter and MHHS settlement data - Unify industry flows, settlement, customer, billing, wholesale and trading data in a consistent model - Create documented, reusable models for MPANs/MPRNs, supply periods, tariffs, consumption and settlement - Build in security, scalability, resilience, cost efficiency and capacity for growth Make data trustworthy - Automate quality checks, testing and reconciliation to catch missing reads, late flows and duplicates before use - Establish lineage, metadata and cataloguing so data is easy to find and trace - Protect personal and commercially sensitive data with proportionate controls supporting UK GDPR and regulation - Monitor pipeline freshness, volume and quality, with alerts and runbooks - Own the platform end to end: build it, run it and support it in operation Deliver effectively - Use version control, code review, automated testing and CI/CD for data pipelines and infrastructure - Use infrastructure-as-code for repeatable environments and safe changes - Deliver the greenfield platform incrementally, putting usable data in people’s hands early - Flag risks and dependencies early, especially third-party industry systems and legacy sources - Use metrics to identify delivery bottlenecks and improve pipeline performance and cost efficiency Use AI responsibly - Provide the data quality, context and governance needed to support AI/ML applications in forecasting, anomaly detection and customer insight - Use AI for pipeline development, testing, documentation, profiling and issue investigation - Review and test AI-assisted work, retain accountability and help others assess outputs critically Solve the right problems - Link engineering decisions to business value, customer outcomes, compliance, risk and cost to serve - Prioritise impact on settlement, billing accuracy, forecasting and trading over technical preference - Work with data users to design products around their practical needs - Baseline performance and cost; plan capacity for growing half-hourly settlement volumes Raise the standard - Define and follow standards for modelling, naming, testing, documentation and pipeline design - Create reusable patterns, templates and architectural decision records - Review work constructively and support engineers and analysts new to data engineering - Share platform and energy-domain knowledge across the team Personal skills and experience Technical background - Expert SQL and strong Python for clean, tested, production-quality code - Hands-on experience in relational database administration, covering schema design, indexing, query tuning, backup and recovery, access management, data integrity and maintenance - Experience delivering data platforms commercially on Azure, ideally using Microsoft Fabric, Azure Databricks, Synapse or Data Factory - Ingestion, storage, processing and integration of structured and unstructured data - Hands-on delivery of raw-to-cleansed/refined data layers and curated fact/dimension tables for analytics - Experience modelling data for operational and analytical use, using dimensional modelling, Data Vault or a lakehouse approach, together with experience of Apache Iceberg or Delta Lake table formats - Spark or equivalent distributed processing for large, high-frequency time-series datasets - Batch and streaming/event-driven ingestion, including late data, idempotency, replays and schema changes - Practical data quality testing with dbt tests, Great Expectations or equivalent - Experience implementing CI/CD for data platforms using Azure DevOps or equivalent, and infrastructure-as-code using Bicep or Terraform - Data platform architecture design or evaluation, with clear explanations of trade-offs Leadership and influence - Evidence of setting engineering standards adopted by others, beyond delivering pipelines - Clear communication of architecture trade-offs, risks and assumptions to technical and business audiences - Measurable improvements to data quality, reliability or delivery - Experience reviewing engineers’ work and developing their judgement and technical skills - Confidence shaping data priorities with product, operations, finance and trading stakeholders - Pragmatism that balances long-term quality with simpler, fit-for-purpose solutions The Ideal Candidate - Knowledge of UK energy data, including MHHS, DTC flows, settlement, MPAN/MPRN or smart metering, or a willingness to learn the sector - A background in delivering greenfield data platforms, including platform selection - Practical skills in using data governance and cataloguing tools such as Microsoft Purview or Unity Catalog - Experience working with time-series, data at scale, or data supporting forecasting and trading - Hands-on involvement in migrating legacy or monolithic databases to modern platforms - Practical evidence of Azure or data platform capability matters more than relevant certification - Strong problem-solving skills, with a focus on business outcomes when tackling complex data challenges - You own data quality from ingestion to end users - Clear communication and documentation skills, including the ability to explain complex data and record decisions - Sound judgement, using evidence to make decisions amid ambiguity and revisiting them as circumstances change - You enjoy setting standards that help others succeed - A pragmatic approach to raising standards without unnecessary process or perfectionism