← Glossary / 85 of 100
n. / Cost and performance
Streaming
Definition
Streaming sends a model's reply to the user in pieces as it is generated, instead of waiting for the whole answer. The total time is the same, but the first words appear much sooner.
In more detail
It makes an application feel faster, and the metric to watch is time to first token. For tracing, a streamed call is one span that lasts until the final token, so record the first-token time as an event or attribute. Errors can happen midway, after some text has already been shown, which needs deliberate handling.
Example
A 12-second answer starts appearing after 0.8 s. The user starts reading immediately instead of staring at a spinner.
