Skills
AI/ML evaluation
Automated testing
Metrics and benchmarking
About the Role
Join our team to help ensure the quality and dependability of our AI-powered features. You will design evaluation frameworks, build automated test harnesses, and define measurable standards so we can verify model and system performance before releases impact customers.
Responsibilities
- Design and maintain evaluation frameworks that assess accuracy, reliability, and regression behavior for AI capabilities
- Build automated test harnesses that run on real accounting data to detect failures early
- Define metrics and benchmarks that provide clear, quantitative insight into performance
- Create developer-friendly tooling that helps engineers write, run, and interpret evaluation tasks in their normal workflow
- Partner with Applied AI and Agent Engineering teams to establish “good enough” release criteria and verification processes
Requirements
- Experience designing evaluation or testing frameworks for AI/ML systems
- Strong background in automated testing, metrics, and benchmark design
- Ability to work with real-world datasets and build reliable evaluation pipelines
- Experience collaborating across engineering teams to improve quality and release confidence
Benefits
- Opportunity to directly improve customer-facing AI quality through rigorous evaluation
- Collaborative environment with Applied AI and Agent Engineering teams
- Tools and processes that streamline evaluation into everyday engineering workflows
This job was curated and enriched by FindJobHere to help your search. Apply on the original website: jobs.ashbyhq.com.
Learn how it works.