Sr Data Engineer (Databricks)
LocationBengaluru, Karnataka, India
Typefull-time
SenioritySenior
Experience7–8 yrs
DepartmentBharat
Company size1,001–5,000 people
PostedOct 2, 2026 · 1w ago
Verified live1d ago
At a glanceSummarised by Seekless from the posting.
Lead the design, architecture, and optimization of enterprise-scale data pipelines using Azure Databricks for ACDP projects
7-8+ years of expertise to mentor teams, ensure scalability, and deliver mission-critical data solutions
Architect and implement end-to-end data pipelines with Azure Databricks (notebooks, jobs, clusters) for ingestion, ETL/ELT, and complex transformations, incorporating Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for ACID compliance, versioning, schema evolution, and data quality
Provide hands-on leadership in Databricks Spark (PySpark, Scala, SQL), Delta Live Tables, Structured Streaming, and Unity Catalog for advanced governance, security, and multi-workspace management
Integrate with Azure ecosystem including ADLS Gen2, Azure SQL Database, Serverless/Dedicated SQL Pools; orchestrate via Databricks workflows and hybrid ADF triggers for high-volume ACDP workloads
Design and deploy CI/CD pipelines using Azure DevOps, Bicep for infrastructure-as code, and advanced automation for Databricks workspaces, emphasizing zero downtime deployments and cost optimization
Champion data quality, validation, cleansing with Databricks expectations/constraints, and compliance via RBAC, encryption, and Purview; mentor on best practices for Photon engine, auto-scaling, spot instances, and performance tuning
Collaborate with senior analysts, data scientists, and executives to translate complex business requirements into reusable Databricks components, fostering AI/ML integration
Drive monitoring, troubleshooting, and optimization for petabyte-scale processing, including unstructured data handling, data modeling, and streaming from Kafka/Event Hubs
7-8+ years in data engineering, with deep Azure Databricks expertise for big data processing in high-demand environments
Advanced proficiency in Python/PySpark, SQL, Scala; strong data architecture and distributed systems experience
Proven track record in cloud-native solutions (AZ-900+ preferred), CI/CD, and mentoring junior engineers
Excellent communication for stakeholder alignment and documentation
Power BI, Power Automate, Azure ML, enterprise governance tools
Skills
Azure Databricks
PySpark
Scala
SQL
Delta Lake
Delta Live Tables
Structured Streaming
Unity Catalog
ADLS Gen2
Azure SQL Database
Serverless/Dedicated SQL Pools
Azure Data Factory
Azure DevOps
Bicep
Power BI
Power Automate
Azure ML
Python
Kafka
Event Hubs
Position Overview:
•
Lead the design, architecture, and optimization of enterprise-scale data pipelines using Azure Databricks for ACDP projects, driving innovation in DATA & AI initiatives.
•
Leverage 7-8 years of expertise to mentor teams, ensure scalability, and deliver mission-critical data solutions.
Key Responsibilities:
•
Architect and implement end-to-end data pipelines with Azure Databricks (notebooks, jobs, clusters) for ingestion, ETL/ELT, and complex transformations, incorporating Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for ACID compliance, versioning, schema evolution, and data quality.
•
Provide hands-on leadership in Databricks Spark (PySpark, Scala, SQL), Delta Live Tables, Structured Streaming, and Unity Catalog for advanced governance, security, and multi-workspace management.
•
Integrate with Azure ecosystem including ADLS Gen2, Azure SQL Database, Serverless/Dedicated SQL Pools; orchestrate via Databricks workflows and hybrid ADF triggers for high-volume ACDP workloads.
•
Design and deploy CI/CD pipelines using Azure DevOps, Bicep for infrastructure-as code, and advanced automation for Databricks workspaces, emphasizing zero downtime deployments and cost optimization.
•
Champion data quality, validation, cleansing with Databricks expectations/constraints, and compliance via RBAC, encryption, and Purview; mentor on best practices for Photon engine, auto-scaling, spot instances, and performance tuning
•
Collaborate with senior analysts, data scientists, and executives to translate complex business requirements into reusable Databricks components, fostering AI/ML integration.
•
Drive monitoring, troubleshooting, and optimization for petabyte-scale processing, including unstructured data handling, data modeling, and streaming from Kafka/Event Hubs
Requirements
Required Qualifications:
•
7-8+ years in data engineering, with deep Azure Databricks expertise for big data processing in high-demand environments
•
Advanced proficiency in Python/PySpark, SQL, Scala; strong data architecture and distributed systems experience
•
Proven track record in cloud-native solutions (AZ-900+ preferred), CI/CD, and mentoring junior engineers
•
Excellent communication for stakeholder alignment and documentation
Preferred: Power BI, Power Automate, Azure ML, enterprise governance tools.