Role:
• Drive modernization from legacy and on-prem systems to modern, cloud-native, and hybrid data platforms.
• Architect and lead the development of a Multi-Agent ETL Platform for batch and event streaming, integrating AI agents to autonomously manage ETL tasks such as data discovery, schema mapping, and error resolution.
• Define and implement data ingestion, transformation, and delivery pipelines using scalable frameworks (e.g., Apache Airflow, Nifi, dbt, Spark, Kafka, or Dagster).
• Leverage LLMs, and agent frameworks (e.g., LangChain, CrewAI, AutoGen) to automate pipeline management and monitoring.
• Ensure robust data governance, cataloging, versioning, and lineage tracking across the ETL platform.
• Define project roadmaps, KPIs, and performance metrics for platform efficiency and data reliability.
• Establish and enforce best practices in data quality, CI/CD for data pipelines, and observability.
• Collaborate closely with cross-functional teams (Data Science, Analytics, and Application Development) to understand requirements and deliver efficient data ingestion and processing workflows.
• Establish and enforce best practices, automation standards, and monitoring frameworks to ensure the platform’s reliability, scalability, and security.
• Build relationships and communicate effectively with internal and external stakeholders, including senior executives, to influence data-driven strategies and decisions.
• Continuously engage and improve teams’ performance by conducting recurring meetings, knowing your people, managing career development, and understanding who is at risk.
• Oversee deployment, monitoring, and scaling of ETL and agent workloads across multi cloud environments
• Continuously improve platform performance, cost efficiency, and automation maturity.