•Evaluate large language models and AI services for accuracy, reliability, safety, latency, and business suitability on finance use cases.
•Design structured evaluation frameworks, test cases, and benchmarking methodologies.
•Conduct prompt testing, retrieval validation, and failure-mode analysis.
•Implement quality guardrails, safety filters, and monitoring for LLM applications handling sensitive financial data.
System and Backend Development
•Build and maintain backend services and APIs that integrate LLMs or AI workflows with finance systems such as Oracle Fusion.
•Architect scalable systems supporting chat interfaces, retrieval pipelines, classification tools, or finance workflow automation.
•Implement solid software engineering practices: testing, versioning, error handling, observability, and performance optimization.
•Ensure robust integration with internal systems, data services, and production infrastructure.
AI Application Engineering
•Work on features powered by LLMs such as RAG systems, finance copilots, document intelligence for invoices/contracts, and intelligent automation.
•Implement embeddings, document retrieval layers, vector search, caching, and fallback logic.
•Collaborate with platform teams on deployment, API management, and resource optimization.
Operations and Reliability
•Monitor AI features in production and proactively address model drift, latency issues, and failure patterns.
•Maintain evaluation logs, experiment results, and version control for AI workflows.
•Work with DevOps to manage CI/CD pipelines, container deployment, and runtime environments.
Collaboration and Delivery
•Partner with product owners, Finance teams, and engineering teams to convert requirements into reliable AI solutions.
•Provide technical guidance on feasibility, architecture choices, and operational trade-offs.
•Produce clear documentation on workflows, system design, evaluation methods, and application behavior.