← Glossary / 86 of 100
n. / Quality and evaluation
Evals
Definition
Evals are repeatable tests that score an AI system's outputs against criteria, such as correctness, tone or safe tool use. They are the AI version of automated tests.
In more detail
Because outputs vary, evals return scores instead of simple pass or fail, and they run on a fixed set of inputs so results can be compared over time. Scores can come from exact checks in code, from a judge model, or from human reviewers. Run them before shipping a change and on a sample of live traffic afterwards.
Example
You run 150 saved questions through a new prompt. 138 score as correct, up from 129 for the old prompt, so you ship it.
