Evals: the unglamorous thing that separates pilots from production

A pilot demos beautifully. Production breaks in week three. The difference is almost always the eval harness — the test set, the metrics, and the regression suite that tells you whether the new prompt is better than the old one.

Build it before you build the agent. Start with 30 hand-labeled examples, three success metrics, and a one-line script that runs the suite. Re-run it on every change. Add to it every time something breaks in production.

Without evals, you cannot ship. With them, you can ship every week.