• We are seeking a highly experienced Data Engineer with over four years of expertise in modern data stack technologies, specifically focusing on Azure Databricks and the Medallion Architecture encompassing the Bronze, Silver, and Gold processing layers.
• Proven experience in designing, developing, and optimizing scalable, cloud-based data pipelines is essential.
• Pipeline Development & Orchestration: Responsible for creating, deploying, and maintaining production-grade data pipelines utilizing Azure Databricks, PySpark, and Azure Data Factory (ADF).
• Medallion Framework Management: Accountable for developing and managing the comprehensive Lakehouse pipeline architecture across the Bronze, Silver, and Gold layers.
• Delta Lake Optimization: Expertise in applying advanced Delta Lake features such as Z-Ordering, liquid clustering, vacuuming, and partitioning techniques to minimize query costs and improve read/write efficiency.
• Infrastructure as Code & CI/CD: Collaborate in managing and deploying cloud infrastructure using Terraform, Git, and Azure DevOps, automating the deployment of data pipelines.
• Data Governance & Security: Implement enterprise-level data governance frameworks, enforce access controls, and manage data lineage tracking utilizing Unity Catalog.
• Performance Tuning: Monitor, analyze, and address performance bottlenecks in distributed Spark clusters by optimizing cluster sizing, memory allocation, and execution plans.
• Proficiency in English, both spoken and written, is required for effective communication.
• Working hours are from 8:00 AM to 5:00 PM PST.