← Glossary / 90 of 100
n. / Quality and evaluation
Offline evaluation
Definition
Offline evaluation tests a change on a saved dataset before it reaches users. Nothing live is affected, and the same inputs can be run again for a fair comparison.
In more detail
It is the right place to compare prompts, models and settings, because only one thing changes at a time. Its weakness is that the dataset is only a sample of reality, so pair it with online evaluation. Gate releases on it: a change that lowers the score does not ship.
Example
Before switching to a cheaper model, you run the golden dataset through both. The cheaper model scores two points lower and costs 60% less, so you decide it is acceptable.
