Senior Software Engineer, AI Evals
Sentry · San Francisco, CA · Engineering
About this role
Sentry is hiring a senior-level Software Engineer based in San Francisco, CA. The posting calls out experience with Python, TypeScript, LLMs, Testing. Compensation is listed at $240,000–$280,000 per year.
- Role
- Software Engineer
- Function
- software engineering
- Level
- senior
- Track
- Individual contributor
- Employment
- Full-time
- Location
- San Francisco, CA
- Department
- Engineering
- Posted
- Jan 28, 2026
More roles at Sentry
Job description
from Sentry careersAbout Sentry
Software runs the world and the pace is faster than ever. Sentry helps developers fix errors and performance issues before users notice, so teams can spend less time firefighting and more time building.
Trusted by 100,000+ organizations, Sentry is today’s application monitoring standard and our team is building its AI-native future.
About the role
As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems. This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale. You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence.
In this role you will
Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
This is an excerpt. Read the full job description on Sentry careers →