Find jobs
Pricing
Free Resume Checker
Sign in
Sign up
FIND JOBSPRICINGFREE RESUME CHECKERSIGN INSIGN UP
TH
The University of Sydney

Manager, AI Evaluation

SalaryA$152k
LocationForest Lodge
Typefull-time
SeniorityLead
Company size10,000+ people
First seenOct 7, 2026 · 4d ago
Verified live1d ago
At a glanceSummarised by Seekless from the posting.
Must have6
postgraduate qualifications in a relevant field, or an equivalent combination of education and experience
demonstrated experience in data science, machine learning, natural language processing and AI solutions within complex organisational environments
strong knowledge of generative AI, large language models (LLMs), agentic AI frameworks and modern data science methodologies
highly developed analytical, problem-solving and data analysis skills, with the ability to interpret complex data, identify patterns and support evidence-based decision-making
experience developing evaluation, monitoring and training frameworks, with strong attention to detail and a commitment to continuous improvement
excellent communication, planning and stakeholder engagement skills, with experience working collaboratively in Agile and cross-functional teams within large or complex organisations
Eligibility1
unrestricted work rights in Australia for the duration of this employment
Skills
Python
generative AI
large language models (LLMs)
agentic AI
About the role
•
Full-time continuing opportunity
•
drive the evaluation and quality assurance of generative AI solutions, ensuring robust governance, performance and continuous improvement across the AI Hub
•
Base Salary $152,045+ 17% superannuation
About the opportunity
As the Manager, AI Evaluation, you will lead the assessment, testing and governance of generative AI solutions, ensuring they are safe, effective and fit for deployment and ongoing use. You will establish and oversee a consistent evaluation framework across the AI Hub, driving pre-release testing, governance approvals, performance measurement and continuous monitoring of AI services in production. Working closely with product, delivery and technical teams, as well as key stakeholders across ICT, Legal, Privacy and data stewardship functions, you will play a critical role in embedding best-practice evaluation processes, identifying risks, and ensuring AI solutions continue to deliver reliable and high-quality outcomes across the University.
Responsibilities
Your key responsibilities will be to:
•
lead the development and continuous improvement of AI evaluation frameworks, methodologies and standards to ensure consistent, rigorous assessment across the AI Hub portfolio
•
design, curate and govern evaluation datasets, metrics and testing approaches that measure performance, reliability, safety and quality of AI solutions
•
oversee pre-release evaluation, testing and assurance activities, providing independent recommendations and evidence to support governance approvals and deployment decisions
•
manage ongoing monitoring, regression testing and performance evaluation of production AI services, identifying risks, drift and opportunities for improvement
•
partner with technical teams, data stewards, governance bodies and key stakeholders to ensure effective evaluation practices, data integrity and compliance requirements are met
•
provide leadership, expert advice and capability development in AI evaluation, communicating findings and driving continuous improvement across the AI Hub and broader university community
About you
•
postgraduate qualifications in a relevant field, or an equivalent combination of education and experience
•
demonstrated experience in data science, machine learning, natural language processing and AI solutions within complex organisational environments
•
strong knowledge of generative AI, large language models (LLMs), agentic AI frameworks and modern data science methodologies
•
highly developed analytical, problem-solving and data analysis skills, with the ability to interpret complex data, identify patterns and support evidence-based decision-making
•
experience developing evaluation, monitoring and training frameworks, with strong attention to detail and a commitment to continuous improvement
•
excellent communication, planning and stakeholder engagement skills, with experience working collaboratively in Agile and cross-functional teams within large or complex organisations
Sponsorship / work rights for Australia
You must have unrestricted work rights in Australia for the duration of this employment to be eligible to apply. Visa sponsorship is not available for this appointment.
Pre-employment checks and declarations
Your employment is conditional upon the successful completion of all pre-employment or background checks required for the role in terms satisfactory to the University. Also, to meet the University’s obligations under the National Higher Education Code to Prevent and Eliminate Gender-Based Violence you will be asked to declare if you have been investigated for, or found to engaged in, sexual harm or gender-based violence in the course of previous employment or in a legal process. Similarly, your ongoing employment is conditional upon the satisfactory maintenance of all relevant clearances and background check requirements. If you do not meet these conditions, the University may take any necessary step, including the termination of your employment.
EEO statement
About the company
At the University of Sydney, our shared values are trust, accountability and excellence and we strive to be a place where everyone can thrive. We are committed to creating a University community that thrives through diversity and reflects the wider community that we serve. We deliver on this through our commitment to diversity and inclusion, evidenced by our people and culture programs, as well as key strategies to increase participation and support the careers of Aboriginal and Torres Strait Islander People, women, people living with a disability, people from culturally and linguistically diverse backgrounds, and those who identify as LGBTQIA+. We welcome applications from candidates from all backgrounds.
We are proud to be recognised as an Australian Workplace Equality Index (AWEI) Platinum Employer. Find out more about our work on diversity and inclusion.
How to apply
Applications (including a cover letter, CV, and any additional supporting documentation) can be submitted via the Apply button at the top of the page.
For employees of the University or contingent workers, please login into your Workday account and navigate to the Career icon on your Dashboard. Click on USYD Find Jobs and apply.
For a confidential discussion about the role, or if you require reasonable adjustment or any documents in alternate formats, please contact Victoria Ford-Rogers Recruitment Operations by email to [email protected]
© The University of Sydney
The University reserves the right not to proceed with any appointment.
Click to view the Position Description for this role.
Applications Close
Friday 16 October 2026 11:59 PM
More roles at The University of Sydney
IT Support TechnicianAdministrative AssistantPostdoctoral Research Fellow in Soil Security for Cotton Farming SystemsPostdoctoral Research Associate - ImmunologyLecturer in Primary English
Seekless
Seek less: one search across companies' own career pages, instead of a dozen job boards.
Product
Find jobsCompaniesPricingFree Resume CheckerBlog
Legal
AboutPrivacyTermsCookies
© 2026 SEEKLESS. ALL RIGHTS RESERVED.BUILT WITH CARE