The challenge
Ordinal’s support agent answered 30,000 questions a day. When it went wrong, the team read logs for two days to find out why.
The approach
Every plan, tool call and answer became a trace. Four judges scored live output, and a rule dialled the new prompt back when accuracy dropped.
The result
The runbook shrank to four rules. Reviews take twenty minutes, and nobody replays an incident by hand any more.