Data Infrastructure and Pipeline Development
Design and build moderately complex data systems, including data models, integration frameworks, and storage solutions. Architect robust pipelines using advanced ETL/ELT techniques, data streaming, CDC, and message queues. Implement scalable solutions using tools such as Databricks, including Lakehouse architecture, Delta Lake storage patterns, and production-grade batch and streaming pipelines. Govern schema design standards and data integrations across multiple source systems.
Performance Optimisation and DataOps
Implement DataOps practices including automated deployments, pipeline monitoring, and data observability. Optimise system performance to handle growing data volumes, proactively address technical debt, and apply tools such as Databricks to improve governance and operational efficiency.
Leadership and Collaboration
Break down complex problems into well-defined tasks, coordinate across teams, and lead architecture reviews. Mentor junior engineers, influence team culture, and help resolve disagreements constructively. Develop plans to achieve key milestones and ensure delivery aligns with GovTech’s strategic objectives.
Data Governance and Quality
Navigate data privacy, governance, and regulatory requirements. Maintain data quality, integrity, security, and compliance across all data systems. Oversee documentation of system architecture and technical specifications.