Data Mapping and Research
• Map BM units from Elexon to their corresponding power plants, substations, and fuel types — combining API data, public registers, and manual research
• Map substations to ETYS zones and grid supply points
• Build and maintain reference/master datasets that link identifiers across disparate sources (Elexon, National Grid ESO, TEC register, etc.)
• Document mappings, assumptions, and known limitations clearly for downstream users
Data Reconciliation and Consistency
• Reconcile legacy data formats with current formats (e.g., historical operational data stored in different schemas or granularities)
• Ensure consistency between different Elexon message types — understand the market data structure well enough to know why BOALF, BOD, and DISBSAD might not perfectly align and how to handle it
• Investigate discrepancies between data sources and determine authoritative values
Data Cleaning and Quality
• Clean time-series data: detect outliers (price spikes, meter errors), fill gaps appropriately, resolve overlapping or duplicate timestamps
• Develop reusable Python-based cleaning routines that can be applied across datasets
• Understand why data quality issues occur (settlement reruns, late submissions, format changes) not just patch them
Pipeline Development (Supporting the Above)
• Write and maintain Python data grabbers for energy market APIs
• Build dbt models to transform raw data into clean, analysis-ready datasets
• Orchestrate workflows via GitHub Actions
• Design PostgreSQL schemas that reflect your understanding of the domain