evals turn a demo into a product

Most LLM apps ship with no evals, so quality is whatever got spot-checked. That bit me twice: I claimed a failure rate I later couldn’t reproduce, and I shipped a prompt typo because nothing tested the prompt’s behaviour.

So the tone translator got an eval harness: a golden set where every case is a real past failure, run through the actual shipping prompt and graded by a stronger model against a rubric. Single scores are noise, since output is non-deterministic. What I trust is the delta between runs: change the prompt, run before and after, watch which failures move. It stays out of CI on purpose (real API calls, and noisy); I run it by hand before any prompt or model change.

Then I automated the part I’d forget: an agent reads real usage, judges it, and proposes new test cases for the failures, which I approve. Quality became something I measure instead of something I hope.