← Glossary / 93 of 100
n. / Quality and evaluation
Benchmark
Definition
A benchmark is a standard set of tasks used to compare models or systems on the same test. Public benchmarks measure general ability, not how well a model fits your product.
In more detail
Public scores are a rough guide for picking candidates, but models can be tuned to them and your tasks differ. A benchmark built from your own traces and labeled answers is a better judge of what matters to you. Always run candidates on the same inputs and settings.
Example
Two models score within one point of each other on a public leaderboard, but on your 100 invoice-extraction cases one gets 94% right and the other 81%.
