Build and maintain ETL/ELT pipelines in Microsoft Fabric Notebooks (Python/PySpark) ingesting data from multiple source systems via REST and SOAP APIs
•
Implement Bronze layer landing (raw data, no transformation), Silver layer cleansing/typing/deduplication, and Gold layer aggression for business analytics
•
Design and build custom API connectors with OAuth 2.0/Bearer token authentication, incremental sync, pagination handling, rate-limit/retry logic and error recovery
•
Configure and manage Fabric Workspaces across Dev/Test/Production environments using Fabric Deployment Pipelines and Git integration for CI/CD
•
Build and maintain Power BI semantic models (star/snowflake schemas) supporting operational reporting and analytics dashboards
•
Implement row-level security (RLS) using Azure AD RBAC, warehouse-level DAX filters, and dynamic RLS platforms
•
Set up monitoring, alerting, and telemetry using Azure Monitor, Log Analytics, and Application insights to track pipeline health and data freshness
•
Manage API credentials and secrets via Azure Key Vault with automated rotation policies
•
Embed within the client’s development process, working directly with business stakeholders to gather requirements and co-develop solutions
•
Contribute to data governance: data dictionary creation, lineage tracking, and documentation of transformation logic across all pipeline stages
Requirements
•
5+ years experience in data engineering or a similar role in a commercial environment
•
Hands-on experience with Microsoft Fabric (Lakehouses, Notebooks, Data Pipelines, OneLake, SQL Analytics Endpoints) or equivalent depth in Azure Synapse Analytics
•
Advanced SQL for data transformation, performance tuning, and Delta Lake table management
•
Proficient in Python/PySpark for data processing, API integration, and pipeline automation
•
Proven experience building custom API connectors (REST, SOAP/XML) with OAuth 2.0, pagination, rate limiting, and incremental sync patterns
•
Strong understanding of Medallion Architecture, data lakehouse concepts, and Delta Lake (merge, upsert, soft-delete handling, time travel)
•
Experience with dimensional modelling (star/snowflake schemas) and Power BI semantic model development
•
Working knowledge of Azure services: Key Vault, Azure Monitor, Log Analytics, Azure AD/Entra ID
•
Experience translating business requirements into dimensional models iteratively (not just building pre-designed schemas)
•
Comfort operating autonomously with minimal supervision in a client-embedded model
•
Experience implementing row-level security in Power BI (DAX-based RLS, dynamic security models)
•
Familiarity with CI/CD in a Fabric context: Deployment Pipelines, Git integration, environment promotion workflows
•
Experience with semi-structured data formats (JSON, XML) and handling schema evolution across pipeline layers
Preferred
•
Prior work in retail, wholesale, distribution, or fresh produce / supply chain data environments
•
Experience with ERP data extraction and replication tools in a Fabric or Azure context
•
Knowledge of dbt, Power Query, or similar transformation frameworks
•
Relevant Azure or Microsoft certifications (DP-600, DP-203, PL-300)
•
Openness to AI coding assistants (GitHub Copilot, Cursor, Claude) as part of your development workflow
What We Offer
•
Competitive salary with mentorship and career growth opportunities
•
Hands-on work with cloud-first technologies (Microsoft Fabric, Azure, Power BI)
•
Long-term embedded role with a single client, providing deep domain expertise and continuity
•
Fast-paced, innovative environment where AI-augmented development is the norm