How to Track LLM Token Usage Per Request, Run and Release

At a glance

One call's input is built from six sources. Only some of them grow while a run is in progress, and those are the ones to watch.

This guide is for engineers who run LLM applications or agents and need to know where their tokens go. By the end you will know where token counts come from, how to record them on every call, how to roll them up to the level you make decisions at, and how to turn a pile of numbers into a short list of things to fix.

A token count is not interesting by itself. It becomes useful when you can ask three questions about it: which request used it, which part of the request used it, and what changed since last week. Those questions decide how the data should be recorded, so the article is organized around them.

The operating model: Capture, Attribute, Roll up, Act

1. Capture the provider's usage numbers on every model call, exactly as reported.
2. Attribute each count to a run, a step, a model, a prompt version and a release.
3. Roll up to the level of decision: per run, per user task, per release.
4. Act on the few spans that dominate, with a before and after comparison.

Know what a token count really is

A token is a chunk of text the model reads or writes. Providers meter usage in tokens, and the bill follows. Two facts shape everything else.

First, the provider's number is the source of truth. Every major provider returns a usage object with each response, with the number of input tokens and output tokens it counted. A local tokenizer can estimate a count before you send a request, which is useful for budgeting prompts, but the estimate can differ from the provider's number, especially for special tokens, images and tool schemas. Record what the provider reports. Use local estimates only for decisions made before the call.

Second, input and output are separate meters. Output tokens are usually priced higher than input tokens, and they are also slower to produce, because the model generates them one at a time. A change that adds a hundred tokens to every output can cost more than one that adds three hundred to every input. Never store only a combined total.

Cached and reasoning tokens

Providers add detail beyond the two basic counts. Some report how many input tokens were served from a prompt cache and how many were written to it, and price those differently. Some report tokens spent on internal reasoning. The names and the arithmetic differ between providers: for example, Anthropic's total input tokens include uncached input plus cache reads and cache writes, and the breakdown is reported separately.

The safe rule is to keep the full usage object as returned, in addition to the headline input and output counts. That way a future price calculation can use fields you did not think to promote today. Tracify's provider wrappers follow this rule: usage is stored exactly as the provider returned it, including cache breakdowns in metadata.

Field note: Do not compute "total tokens" by adding fields you have not verified. Different providers count cached and reasoning tokens in different ways. Store the raw usage and do the arithmetic in one place that you can test.

Capture usage on every model call

There are two practical ways to capture token counts: wrap the provider client, or record them yourself on a span.

Option 1: wrap the provider client

If you call OpenAI or Anthropic directly, the least work is to wrap the client once. Supported calls are recorded with model, input and output, latency, token usage, time to first token for streams, and provider errors. The wrapper keeps the same response types and streaming behavior, and telemetry failures never replace the provider's result or error.

import OpenAI from "openai";
import Anthropic from "@anthropic-ai/sdk";
import { wrapOpenAI, wrapAnthropic } from "tracify-sdk";

const openai = wrapOpenAI(new OpenAI());
const anthropic = wrapAnthropic(new Anthropic());

// Existing calls are recorded with their usage.
const reply = await openai.responses.create({
  model: process.env.OPENAI_MODEL!,
  input: "Summarize this ticket in two sentences.",
});

The setup is described in the provider integrations guide, including the Python equivalents.

Option 2: record usage on a span yourself

If you use a different provider, a gateway or your own model server, record the numbers on a span where you make the call.

import { span } from "tracify-sdk";

const text = await span(
  "chat",
  async (step) => {
    const reply = await callModel(question);
    step.set({
      model: "your-model-id",
      inputTokens: reply.usage.input,
      outputTokens: reply.usage.output,
    });
    return reply.text;
  },
  { type: "llm", input: question },
);

Here callModel and the shape of reply.usage stand in for your own client. The only requirement is that the numbers come from the provider's response, not from a guess.

Streaming and early stops

With streaming, usage often arrives at the end, in the last event, and some APIs only send it if you ask. If your stream ends early because the user cancelled, you may get partial or no usage. Two habits help. First, iterate the stream to completion or close it properly, so the wrapper can record the partial output once. Second, treat a missing count as unknown, not zero. A zero hides a real gap in your data and pulls your averages down.

When the provider reports no usage, Tracify keeps the count unknown instead of inventing a number. Your dashboards then show the gap, which is the honest result.

Attribute tokens to the right owner

A raw token count attached to nothing is almost useless. Attribution is what makes it actionable.

Link every call to a run

An agent makes many model calls to finish one task. If each call is stored on its own, you can say how many tokens the model used overall, but not how many a task used. Group calls under a run so that you can sum them. In the TypeScript SDK, calls made inside traceAgent or span join the current run under the current step, and a standalone call outside any run gets its own completed root so that unrelated requests never share a run.

Record the version of what produced the call

Four labels make comparisons possible later:

• Model id, because models differ in verbosity and price.
• Prompt version, so a prompt change shows up as a clear step in the data.
• Release or deployment, so a code change shows up the same way.
• Environment, so tests do not pollute production numbers.

Add them when you record the span, not later. Adding labels after the fact means joining across systems and guessing.

Keep the structure of the run

A run is a tree: a root, a few steps, and model and tool calls under them. Keeping the parent links tells you whether tokens went to planning, tool use or the final answer. The page on what is in a trace lists the fields that carry this structure.

Roll up to the level where you decide

Per-call numbers feed three roll-ups. Each answers a different question.

Prefer medians and the 90th percentile to averages. Token use per run is heavily skewed: most runs are short, a few are very long, and the long ones often cost the most. An average hides both facts. A median shows the typical run; the 90th percentile shows the tail.

Compare like with like

When you compare two releases, compare runs of the same kind. A release that handles easier requests will look cheaper even if its prompts got longer. Segment by task type, or at least by number of turns, before drawing a conclusion.

Find where the tokens go

With attributed data, finding waste becomes a small number of repeatable questions.

Which source grows during a run?

Look at input tokens by position in the run: the first call, the middle, the last. If input grows steadily, history or tool results are accumulating. The usual fixes are to summarize old turns, trim large tool payloads before they re-enter the prompt, and drop content the model no longer needs. Context window management goes deeper on these techniques.

Which tool result is oversized?

A tool that returns a whole document, a long list of search hits or a verbose JSON blob puts all of it into the next call's input, and every later call. Sort tool spans by output size. A single tool that dominates is a good target: return fewer fields, paginate, or return a summary with a handle to fetch more.

Which prompt carries dead weight?

System prompts and tool definitions are sent on every call. A paragraph of rarely used instructions costs its length multiplied by every call you make. Count how often each part of the prompt affects behavior, and move rarely used guidance behind a tool or a conditional.

Which loop is running too long?

Count model calls per run. Runs with many more calls than the median are often loops: an agent retrying a failing tool, re-asking the model to fix a malformed output, or searching again with nearly the same query. A cap on turns, plus a clear failure message from the tool, usually removes most of the tail.

Set thresholds and alerts that mean something

Tracking only pays off if someone notices a change. Alerts on raw token totals are noisy, because totals rise with traffic. Three signals are better.

1. Tokens per run, median. A shift means the work per task changed.
2. Tokens per run, 90th percentile. A shift means the tail grew, usually a loop or an oversized payload.
3. Share of calls with unknown usage. A jump means instrumentation broke or a provider stopped reporting.

Pick a threshold by looking at normal variation over two or three weeks, then alert when a signal leaves that band for long enough to matter. An hour-long spike from a flaky tool is worth a note; a six-hour shift in the median is worth waking someone for only if it is expensive.

Decision rule: Fix the largest source of growth first, and only call a change a win when the median and the 90th percentile both move the right way.

Operating note: Keep a "budget per run" value for each task type in version control next to the prompt. When a change pushes the median over the budget, the review conversation becomes concrete.

Turn tokens into money, carefully

Tokens are facts. Dollars are tokens multiplied by prices, and prices are yours: they depend on your provider, model, contract and cache behavior. Record tokens first, always. Add a cost calculation next, using your own rate card.

With wrapped clients, you supply a function once:

// Your own rates in USD per token. These numbers are placeholders.
const rates: Record<string, { input: number; output: number }> = {
  "your-model-id": { input: 0.000001, output: 0.000004 },
};

const openai = wrapOpenAI(new OpenAI(), {
  calculateCost: ({ model, inputTokens, outputTokens }) =>
    rates[model].input * inputTokens + rates[model].output * outputTokens,
});

Tracify does not guess prices. If you do not provide a calculation, cost stays at zero and the span is marked as having unknown cost, so a zero never passes for a real number. For cached tokens, use the usage object the provider returned, because cached input is priced differently from fresh input. If you want to plan before you have traces, the free cost estimator turns token assumptions into a monthly range, and the cost per task guide shows how to include retries and failures.

Protect privacy while you measure

Token tracking does not require storing prompts. If your inputs or outputs contain personal or confidential data, you can capture usage without capturing content. The provider wrappers accept options to turn off input and output capture, so you keep the token counts, latency and errors without storing text.

• Redaction removes obvious secrets such as bearer tokens, API keys and passwords from captured text, but it is not a data-loss-prevention system. Do not rely on it for personal data.
• Cap and sample content capture in production if volumes are high.
• Restrict who can read raw prompts, even when token counts are widely visible.

Security note: Aggregated token counts are low risk. Raw prompts and outputs are not. Separate the two in your access rules.

Verify that your counts are right

Four hops, no guessing: provider usage becomes span fields, spans sum into runs, and runs feed cost and release views.

Every hop in that path is a place where numbers can be lost, duplicated or mislabeled. A provider reply can arrive without usage, a span can be dropped under load, a run can be split in two by a missing parent link, and a roll-up can double count a retry. Verification is the habit of checking each hop against something independent, on a schedule, rather than after a surprise.

Before you trust a dashboard, check it against something independent.

Duplicates deserve special attention. If you use a framework's model callbacks and also wrap the provider client, the same call may be recorded twice, which doubles your counts. Use one source for each model call.

Work through an example

Consider a research assistant that answers questions by searching, reading and writing. After a release, its median tokens per run rise from 14,000 to 19,500, while the answer quality scores stay flat.

Grouping by step type shows planning and answering barely moved, but "tool follow-up" input grew by 60 percent. Sorting tool spans by output size shows one search tool now returns full page text where it used to return snippets, after a change meant to improve quality. The next model call and every call after it carry that text.

The fix is to return snippets plus an identifier, and let the model fetch the full page only when it decides it needs to. After the change, the median returns to roughly 14,500 tokens and quality scores hold. The value of tracking was not the dashboard. It was the ability to say, within an hour, which step changed and what it cost.

Next steps

If you are starting from nothing, wrap your provider client or add step.set to your model spans, run twenty realistic tasks, and look at the median and the longest run. That alone usually reveals the first fix. If cost is the real concern, read how to reduce LLM costs without hurting quality before you cut anything, so that savings do not turn into regressions.

Checklist for reliable token tracking

• [ ] Every model call records input tokens, output tokens and model id from the provider's response.
• [ ] The full usage object is stored, including cache and reasoning fields where reported.
• [ ] Missing usage is stored as unknown, never as zero.
• [ ] Calls are grouped under runs and keep their parent step.
• [ ] Prompt version, release and environment are recorded.
• [ ] Dashboards show median and 90th percentile per run, not only totals.
• [ ] A coverage check and a duplicate check run regularly.
• [ ] Cost uses your own rate card and is labeled unknown when absent.
• [ ] Content capture is limited to what your privacy rules allow.

Tokens are cheap to record and expensive to ignore. Start with the smallest version of this list, and tighten it each time a surprise teaches you what you were missing.

Frequently asked questions