Find jobs
Pricing
Free Resume Checker
Blog
Sign in
Sign up
FIND JOBSPRICINGFREE RESUME CHECKERBLOGSIGN INSIGN UP
LI
Lilly

Test Automation Lead – AI Testing

LocationIndia, Hyderabad
Typefull-time
SeniorityLead
Experience7+ yrs
Company size10,000+ people
PostedAug 14, 2026 · 1mo ago
Verified live1h ago
At a glanceSummarised by Seekless from the posting.
Must have31
Bachelor's or Master's degree in Computer Science, Data Science, Information Technology, or a related field
7+ years of test automation experience, including at least 2 years focused on AI/ML testing
Proven track record in designing and executing automated test suites for complex, distributed, or AI-based systems
Strong programming and problem-solving skills with experience in Python, JS/TS, or similar languages
Deep expertise in test automation frameworks and strategies, with ability to extend to AI/ML models
Hands-on expertise with AI evaluation methodologies, model benchmarking, and quality metrics
Strong understanding of AI testing methodologies such as model validation, bias detection, explainability testing, and performance benchmarking
Ability to design end-to-end automated testing solutions for AI-enabled systems involving APIs, UI, data, and model outputs
Proven experience collaborating with data scientists, ML engineers, and software developers to ensure test coverage and model reliability
Deep understanding of LLMs, Generative AI, RAG, and Agentic AI architectures
Hands-on experience in AI evaluation frameworks (RAGAS, DeepEval, LLM-as-a-Judge, LangFuse)
Expertise in Prompt Validation, Prompt Injection Testing and Adversarial Testing (Red Teaming)
Knowledge of Data Validation, Data Quality, Data Drift, and Training/Inference Pipeline Testing
Deep hands-on experience with a broad range of test automation tools: UI/functional (Playwright, Cypress), API (Postman), Performance (k6, JMeter)
Strong understanding of supervised/unsupervised learning
Experience with ML platforms and tools and cloud AI services
Knowledge of DevOps practices and CI/CD tools like Jenkins, GitHub Actions, or Azure DevOps
Excellent analytical, debugging, and communication skills with a passion for quality and innovation
Ability to define, implement, and own test strategies across enterprise-level projects and programs
Lead and mentor global and cross-functional QA teams, promoting best practices and continuous improvement
Collaborate with senior leadership, product owners, architects, and development teams to ensure quality
Drive implementation of test automation strategies and continuous testing practices
Identify test process improvement opportunities and lead initiatives to enhance QA&E maturity
Oversee risk identification, mitigation planning, and resolution related to QA deliverables
Excellent communication and stakeholder management skills
Exposure to Power BI, Splunk, or other monitoring/reporting tools
Champion an automation-first mindset in test execution and continuous delivery strategies
Provide expert-level consultation on AI and automation tools, frameworks, and best-fit solutions
Align people, tools, and processes to support rapid, reusable, and scalable automation solutions
Continuously evaluate emerging tools and technologies to ensure Lilly remains at forefront of quality engineering innovation
Strong understanding of Agile methodologies such as Scrum, Kanban, and SAFe
Nice to have3
Experience with AI quality tools such as DeepChecks, Great Expectations, DataFold
Understanding of MLOps and automated model lifecycle management
Certification in AI/ML, Test Automation, or Cloud Technologies (e.g., AWS Certified Machine Learning, ISTQB Advanced Test Automation Engineer)
Skills
Python
JS/TS
Playwright
Cypress
Postman
k6
JMeter
Jenkins
GitHub Actions
Azure DevOps
Power BI
Splunk
RAGAS
DeepEval
LangFuse
DeepChecks
Great Expectations
DataFold
About the company
At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.
About the role
Are you passionate about leveraging technology to automate and transform business processes?
Do you have a deep understanding of AI and test automation and a desire to drive innovation in enterprise systems?
If so, the Tech Enablement and Automation team at Tech@Lilly is looking for a Lead Consultant in Test Automation to help scale our automation capabilities and quality practices across the organization.
As a Lead consultant in Test Automation, you will be responsible for driving the quality and reliability of automation solutions across the enterprise. In this role, you will work closely with Engineers and cross-functional teams throughout the full application lifecycle—ensuring that systems are thoroughly tested, resilient, and aligned with business needs from development through to production.
Are you innovative and curious about new technology that can make business processes easy and more efficient? Come join our team!
What You’ll Be Doing:
As a Lead Consultant you will play a pivotal role in shaping and advancing the company’s enterprise-wide quality assurance and engineering strategy for AI powered systems and solutions. You will leverage both your technical expertise and leadership capabilities to oversee the delivery of scalable, high-quality AI and automation solutions. Your responsibilities will include driving resolution of complex testing challenges, leading the implementation of critical enhancements, and spearheading modernization initiatives to ensure our AI and automation frameworks and practices are future-ready and aligned with business objectives.
What You Will Bring:
•
Deep expertise in test automation frameworks and strategies, with the ability to extend them to validate AI and ML models, data pipelines, and intelligent applications.
•
Hands-on expertise with AI evaluation methodologies, model benchmarking, and quality metrics
•
A strong understanding of AI testing methodologies such as model validation, bias detection, explainability testing, and performance benchmarking.
•
Ability to design end-to-end automated testing solutions for AI-enabled systems involving APIs, UI, data, and model outputs.
•
Proven experience collaborating with data scientists, ML engineers, and software developers to ensure test coverage and model reliability.
How You Will Succeed:
•
Lead with influence by promoting a culture of engineering excellence**, responsible AI**, and continuous improvement.
•
Develop and maintain automation frameworks tailored for testing AI/ML workflows, including model training, inference, and integration.
•
Create and execute automated regression, performance, and data validation tests to ensure the stability and quality of AI-driven systems.
•
Implement AI-specific test approaches such as synthetic data generation, result explainability validation, and continuous model monitoring.
•
Drive innovation by identifying opportunities to embed automation and intelligence into the testing lifecycle using GenAI tools and frameworks.
•
Partner cross-functionally to define and track AI quality metrics (accuracy, precision, recall, fairness) and integrate them into CI/CD pipelines.
What You Should Bring:
•
Deep understanding of LLMs, Generative AI, RAG, and Agentic AI architectures
•
Hands-on experience in AI evaluation frameworks (RAGAS, DeepEval, LLM-as-a-Judge, LangFuse)
•
Expertise in Prompt Validation, Prompt Injection Testing and Adversarial Testing (Red Teaming)
•
Knowledge of Data Validation, Data Quality, Data Drift, and Training/Inference Pipeline Testing
•
Deep hands-on experience with a broad range of test automation tools and technologies,
•
UI and functional testing: Playwright, Cypress
•
API testing: Postman
•
Performance testing: k6, JMeter
•
Strong understanding of supervised/unsupervised learning.
•
Experience with ML platforms and tools and cloud AI services.
•
Knowledge of DevOps practices and CI/CD tools like Jenkins, GitHub Actions, or Azure DevOps.
•
Excellent analytical, debugging, and communication skills with a passion for quality and innovation.
•
Define, implement, and own test strategies across enterprise-level projects and programs.
•
Lead and mentor global and cross-functional QA teams, promoting best practices and continuous improvement.
•
Collaborate with senior leadership, product owners, architects, and development teams to ensure quality is built into all stages of the development lifecycle.
•
Drive the implementation of test automation strategies and continuous testing practices.
•
Identify test process improvement opportunities and lead initiatives to enhance QA&E maturity and efficiency
•
Oversee risk identification, mitigation planning, and resolution related to QA deliverables.
•
Excellent communication, and stakeholder management skills.
•
Exposure to Power BI, Splunk, or other monitoring/reporting tools for real-time visibility into test execution and system behaviour.
•
Champion an “automation-first” mindset in test execution and continuous delivery strategies.
•
Provide expert-level consultation on AI and automation tools, frameworks, and best-fit solutions tailored to business and technology needs.
•
Align people, tools, and processes to support rapid, reusable, and scalable automation solutions across the enterprise.
•
Continuously evaluate emerging tools and technologies to ensure Lilly remains at the forefront of quality engineering innovation.
•
Strong understanding of Agile methodologies such as Scrum, Kanban, and SAFe.
Basic Qualifications:
•
Bachelor’s or Master’s degree in Computer Science, Data Science, Information Technology, or a related field.
•
7+ years of test automation experience, including at least 2 years focused on AI/ML testing.
•
Proven track record in designing and executing automated test suites for complex, distributed, or AI-based systems.
•
Strong programming and problem-solving skills with experience in Python, JS/TS, or similar languages.
Preferred Skills:
•
Experience with AI quality tools such as DeepChecks, Great Expectations, DataFold.
•
Understanding of MLOps and automated model lifecycle management.
•
Certification in AI/ML, Test Automation, or Cloud Technologies (e.g., AWS Certified Machine Learning, ISTQB Advanced Test Automation Engineer).
Will this position support sponsorship:
No
Travel percentage for this position:
0-10%
Legal
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.
#WeAreLilly
More roles at Lilly
Senior Device Assembly and Packaging Expert – MSAT (m/f/d)Sr. Manager Lab ScientistMSAT Process Specialist / Expert – Formulation and Filling (m/f/d)Cardiometabolic Medical AdvisorManager, AI Transformation ANZ
Seekless
Seek less: one search across companies' own career pages, instead of a dozen job boards.
Product
Find jobsCompaniesPricingFree Resume Checker
Company
AboutBlog
Legal
PrivacyTermsCookies
© 2026 SEEKLESS. ALL RIGHTS RESERVED.BUILT WITH CARE