← Glossary / 79 of 100
n. / Cost and performance
Throughput
Definition
Throughput is how much work a system completes in a given time, such as runs per minute or tokens per second. It is about volume, while latency is about the speed of a single request.
In more detail
A system can have good latency at low traffic and fall over at high traffic when it hits provider rate limits or runs out of workers. Track throughput next to error rate and latency so you can see the load level at which things start to degrade.
Example
Your agent handles 40 runs per minute with an 8 s P95. At 120 runs per minute, rate limit errors appear and P95 climbs to 30 s.
Related terms
