n. / Models and tokens
Prompt caching
Definition
Prompt caching lets a model provider reuse the already processed beginning of a prompt across calls, so repeated input is cheaper and faster. It only helps when the start of the prompt is identical from call to call.
In more detail
To benefit, put stable content first, such as the system prompt and tool definitions, and variable content last. Providers report cached tokens separately in usage, so tracing it shows whether your prompt layout is actually getting cache hits. A small change near the top of the prompt can silently break the cache.
Example
An agent's 3,000-token instructions never change. After the first call, later calls read them from cache at a lower price, and the trace shows 3,000 cached input tokens per step.
Related terms
