← Glossary / 77 of 100
n. / Cost and performance
Latency
Definition
Latency is the time between sending a request and getting the response. For an AI agent it is the sum of every model call, tool call and wait inside the run, so it is usually far longer than a single model call.
In more detail
Model latency grows with output length, and agent latency grows with the number of loop passes and sequential tool calls. Measure it per span, not just per run, so you can tell whether time goes to the model, to retrieval or to a slow external API. Report percentiles instead of averages.
Example
A run takes 14 s: 9 s across three model calls, 3 s in one slow search tool and 2 s of other work. Speeding up the search tool has the biggest effect.
