Product / LLM calls
Model behavior you can measure.
Every generation with its model, prompt, tokens, latency, time to first token and cost. Compare calls inside the real workflow instead of judging only the final answer.
- Tokens
- In and out
- Latency
- + first token
- Cost
- Per call
Compare generations
Judge the call, not just the answer.
Open the trace and select the model span responsible for a decision or final response.
Inspect payload, token mix, latency, cost, retry state, and downstream effects together.
Use a versioned dataset and evaluator suite before promoting a prompt or model change.
Pick the generation that made the decision, not the final answer.
Who it is for
Which generation changed the result?
Teams routing across models, tuning prompts, or diagnosing quality and performance at generation level.
Instrument it
Tag the model. Keep the tokens.
Set the model and token counts on the span and they roll up into the run total, next to latency and cost.
$ npm install tracify-sdk
import { span, traceAgent } from "tracify-sdk";
export const answer = traceAgent(async (question: string) => {
return span("chat", async (step) => {
const reply = await callModel(question);
step.set({ model: "claude-sonnet-5-5", inputTokens: reply.usage.input, outputTokens: reply.usage.output });
return reply.text;
}, { type: "llm", input: question });
});Find the call that changed quality, speed or cost.
Model attribution
Record the model used on each span so routing and fallback behavior can be verified against the actual trace.
Token accounting
Keep input and output token counts with each generation and aggregate them into the run total.
Latency and TTFT
Separate total call latency from time to first token when streaming responsiveness matters.
Prompt and release context
Use trace metadata to compare model behavior across prompts, environments, and releases.
Questions people ask.
What is an LLM call trace?
An LLM call trace records the model used, its input and output, token counts, cost, latency, and time to first token for a single model generation inside an agent run.
What is time to first token (TTFT)?
TTFT is the delay between sending a request to a model and receiving the first streamed token back. It is tracked separately from total latency because it determines perceived responsiveness.
How do I compare models for an AI agent?
Compare models on the same trace metadata: token mix, latency, TTFT, cost, and quality, ideally over a versioned dataset with repeatable evaluators before promoting a change.
How do I know which model call caused a bad response?
Open the trace and select the model span responsible for the decision, then inspect its payload, token mix, and downstream effects together.
Does token counting include both input and output tokens?
Yes. Input and output token counts are recorded per generation and aggregated into the run total so cost and context usage stay accountable.
See your own runs like this.