← Glossary / 89 of 100
n. / Quality and evaluation
Online evaluation
Definition
Online evaluation scores real production runs as they happen, or shortly after, instead of testing a fixed set before release. It shows how the agent performs on actual users' requests.
In more detail
You usually score a sample, using automatic checks or a judge model, and track the results over time and by release. It catches problems that a test set misses, such as new kinds of questions, and it feeds the datasets you use for offline testing. Scoring every run can be costly, so sample.
Example
Every tenth production run is scored by a judge model for helpfulness. After a release the average drops from 4.3 to 3.6 and an alert fires.
