Reference / definitions
Short, direct definitions for the terms used across agent tracing, LLM observability and evaluation, written to be quoted, not just read.
Flip through / 01 of 100
Start with the picture
One trace / five spans
Yellow = the step that failed
AGENT
The root. One run from request to final answer. Everything below hangs off it.
100 terms
Agent drift is a gradual change in an AI agent's behavior or quality over time, caused by model updates, prompt edits, changing data or shifting user inputs.
Drift is caught by tracking scores, cost and failure rates over time and alerting when they move away from a baseline.
What is Agent drift?Agent evaluation is the practice of scoring AI agent behavior against versioned datasets using deterministic or model-based evaluators, instead of relying on manual spot checks.
Evaluations can run offline, over a fixed dataset before a release, or online, scoring live production traces continuously. Both modes attach a repeatable score to a specific version of the agent, prompt, or model being tested.
What is Agent evaluation?An agent handoff is when one agent passes a task, along with context, to another agent that takes over. What gets passed on, and what gets left out, decides whether the second agent can succeed.
Handoffs are a frequent failure point because the receiving agent only knows what it is told. Missing facts, an over-long dump of history, or a vague instruction all cause errors that look like the second agent's fault. A trace that links both agents shows exactly what crossed the boundary.
What is Agent handoff?The agent loop is the repeating cycle in which an agent calls the model, runs any tools it requests, adds the results to the context, and calls the model again until it produces a final answer or hits a limit.
Every pass adds tokens and time, and a loop with no stopping rule can run until it hits a budget. Good agents have a maximum number of steps, a spend limit and a way to notice repeated identical calls. In a trace, one pass of the loop appears as a model span followed by its tool spans.
What is Agent loop?Agent memory is information an agent keeps beyond a single model call, either within a run (short-term) or across runs and sessions (long-term). Models do not remember anything on their own, so memory is something your code stores and puts back into the prompt.
Short-term memory is usually the message history. Long-term memory is typically saved notes, preferences or facts retrieved when relevant. The risks are stale or wrong memories and memories that leak between users, so record what was retrieved on each run and keep each user's data separate.
What is Agent memory?Agent regression testing is re-running a fixed set of cases after a prompt, model or tool change to confirm that behavior that used to work still works.
Because agent output varies run to run, regression tests compare scores and pass rates across many cases rather than exact strings.
What is Agent regression testing?An agent run is one complete execution of an AI agent from a single trigger, such as a user request, to its final result, made up of all the spans recorded along the way.
Runs are the rows of a tracing table: each has a status, duration, token count and cost that roll up from its spans.
What is Agent run?An agentic workflow is a multi-step process in which a language model helps decide what happens, as opposed to a fixed pipeline where code decides every step. Many are a mix, with some fixed stages and some model-chosen ones.
Fixed steps are cheaper, faster and easier to test. Model-chosen steps handle open-ended tasks. A common pattern is to keep the overall flow in code and let the model decide only inside a step, such as which tool to call. Tracing the whole workflow as one run shows where the model's freedom helps and where it causes failures.
What is Agentic workflow?An AI agent is a program that uses a language model to decide its own next steps, calling tools and reading the results until it reaches a goal or gives up. It differs from a single model call because it loops and acts.
The model is the decision maker, and the tools are how it affects the world: searching, reading files, calling APIs, writing data. Because the path is not fixed in advance, two runs of the same request can take different routes, which is why you need a trace of each run to see what actually happened.
What is AI agent?AI agent observability is the ability to see what an AI agent did and why, by recording every model call, tool call, decision, cost and error of each run so failures can be found and fixed.
It extends LLM observability to multi-step agents: a run can loop, branch, call tools and hand work to other agents, so the useful unit is the whole trace, not a single request.
What is AI agent observability?An AI agent trace is the complete recorded execution path of a single agent run, from the initial request through every model call, tool call, and retry, down to the final output.
A trace is made of spans: individual operations such as one model generation or one tool call, each with its own timing, inputs, outputs, and status. Traces let engineers move from a bad final answer to the exact operation that produced it.
What is AI agent trace?AI agent tracking is monitoring what your agents do in production: which runs happen, what each step costs, which runs fail, and how behavior changes between releases.
Tracking covers runs, tool calls, tokens, cost and quality scores over time. It is the monitoring side of AI agent observability, while traces give the detail needed to debug a single run.
What is AI agent tracking?The AI engineering lifecycle is the loop from observing production agent behavior, to curating datasets from that behavior, to running experiments, to releasing and monitoring the result.
The lifecycle closes the gap between production evidence and the next release: a trace or user signal becomes a dataset case, the case evaluates a candidate change, and the release is then monitored against the same evaluation criteria.
What is AI engineering lifecycle?AI pipeline tracing is following a request through every stage of an AI workflow, such as retrieval, prompt building, model calls, tools and post-processing, as one connected trace.
Pipelines fail between stages as often as inside them. A single trace shows where time and cost go, which stage returned bad data, and how an early mistake carried into the final answer.
What is AI pipeline tracing?An alert rule is a condition that notifies you when a metric crosses a threshold, such as the error rate rising above 5% or daily cost passing a set amount.
Good rules are tied to something you will act on and use a window long enough to ignore single blips. Too many noisy alerts get ignored, so start with a few that matter: failure rate, cost spikes and latency. Each alert should link to the runs that caused it.
What is Alert rule?Anomaly detection finds values that differ sharply from what is normal for a metric, without you setting a fixed threshold for each one.
It compares current behavior with a baseline, often adjusted for time of day and week, and flags large departures. That helps with metrics that vary naturally, like traffic. It can still be noisy, so use it to point you at runs worth inspecting, and keep simple threshold alerts for the limits that must never be crossed.
What is Anomaly detection?Auto-instrumentation is tracing that is switched on by a library or an import instead of by writing span code yourself. It patches known clients, such as model provider SDKs, and records their calls for you.
It is the fastest way to get a first trace, and it captures details such as model, tokens and latency without extra code. The limit is that it only knows about libraries it supports, so your own functions and custom tools still need manual spans to show up with sensible names.
What is Auto-instrumentation?A benchmark is a standard set of tasks used to compare models or systems on the same test. Public benchmarks measure general ability, not how well a model fits your product.
Public scores are a rough guide for picking candidates, but models can be tuned to them and your tasks differ. A benchmark built from your own traces and labeled answers is a better judge of what matters to you. Always run candidates on the same inputs and settings.
What is Benchmark?Chain of thought is having a model work through a problem in written intermediate steps before giving a final answer. It often improves results on tasks with several steps, such as math or planning.
You can ask for it in the prompt, or use a reasoning model that does it on its own. It costs extra output tokens and time, and the written reasoning is not always a faithful account of how the answer was reached, so treat it as a useful aid for debugging rather than proof.
What is Chain of thought?Chunking is splitting documents into smaller pieces before embedding and storing them, so retrieval can return just the relevant part instead of a whole file.
Chunks that are too small lose meaning, and chunks that are too large dilute it and waste tokens. Splitting along natural boundaries such as headings and paragraphs, with a little overlap, usually works better than cutting at a fixed length. Changing your chunking changes retrieval quality, so re-run your evaluation set after any change.
What is Chunking?Context engineering is deciding what information goes into the model's context at each step of an agent: instructions, tools, retrieved facts, memory and past results. The aim is the smallest set that lets the model do the step well.
In an agent, the context changes on every loop. Too little leaves the model guessing; too much costs money, slows each call and buries the useful part. Typical techniques are trimming old tool results, summarizing history, and retrieving only what the current step needs. Traces show exactly what was in context when a step went wrong.
What is Context engineering?The context window is the maximum amount of text, measured in tokens, that a language model can read and write in a single call, including the prompt, the conversation history, tool results and the reply.
Agents that keep appending tool output can fill the window, which raises cost and latency and can push important instructions out of the model's attention.
What is Context window?Context window overflow happens when the input to a model is larger than its context window. The provider returns an error, or your code silently cuts text to fit and loses information.
In long agent runs it appears slowly: each step adds messages and tool output until a call fails or, worse, an old instruction falls out of view and the agent starts behaving differently. Alert on input size approaching the limit and on truncation in your own code, not just on hard errors.
What is Context window overflow?A cost budget is a limit on spend, per run, per user or per time period, with a defined action when it is reached, such as stopping the run, switching to a cheaper model or sending an alert.
Agents can loop, so a per-run cap protects you from one bad request becoming a large bill. Account-level budgets catch slower leaks, like a prompt change that doubled input size. Decide in advance whether hitting a budget should stop, degrade or only notify.
What is Cost budget?Cost per run is the total money spent on one agent run, found by adding up the cost of every model call and paid tool call inside it. It is the number that tells you whether a feature is affordable at scale.
Averages mislead because cost is uneven: most runs are cheap and a few looping runs are expensive. Look at the median, the 95th percentile and the most expensive runs, and break cost down by model and step. Pair it with a quality score, since the cheapest run is not the best one.
What is Cost per run?Data retention is how long you keep stored data, such as traces and prompts, before it is deleted. It balances debugging needs, cost and privacy obligations.
Longer retention helps you investigate old incidents and build datasets, but it also stores more sensitive text for longer. Decide a period per kind of data, and make sure you can delete a specific user's data when asked. Where the data is stored, and in which region, also matters for compliance.
What is Data retention?Distributed tracing follows one request across several services, processes or machines and joins the steps into a single trace. It works by passing a trace ID and parent span ID along with each call.
AI systems are distributed more often than they look: a web app calls an agent service, which calls a vector database, a model provider and several tools. Without distributed tracing you get a separate fragment from each service. With it, one timeline shows the full path and where the delay or failure began.
What is Distributed tracing?Embeddings are lists of numbers that represent the meaning of a piece of text, so texts with similar meaning have similar numbers. They power semantic search and retrieval.
An embedding model turns a document chunk or a question into a vector. Searching for the closest vectors finds relevant text even when the words differ. The same model must be used for both documents and queries, and changing the model means re-embedding everything.
What is Embeddings?Error rate is the share of runs or calls that fail out of all of them in a period, for example 2% of tool calls returning an error.
Track it separately for model calls, tool calls and whole runs, since a failed tool call may be recovered by a retry while a failed run reaches the user. Also look at quiet failures, such as runs that finish with status success but give a wrong answer, which only evaluation catches.
What is Error rate?Evals are repeatable tests that score an AI system's outputs against criteria, such as correctness, tone or safe tool use. They are the AI version of automated tests.
Because outputs vary, evals return scores instead of simple pass or fail, and they run on a fixed set of inputs so results can be compared over time. Scores can come from exact checks in code, from a judge model, or from human reviewers. Run them before shipping a change and on a sample of live traffic afterwards.
What is Evals?An evaluation dataset is a versioned set of inputs, and often expected outputs, used to test an AI agent before a release, usually built from real production traces and known failures.
Turning a bad production trace into a dataset case means that failure is checked on every future change.
What is Evaluation dataset?Few-shot prompting is including a handful of worked examples in the prompt so the model copies their pattern. Zero-shot means no examples, and one-shot means a single example.
Examples are often the strongest way to fix a format or tone problem. Choose varied, correct examples, keep them short, and watch the token cost since they are sent on every call. Poor or inconsistent examples teach the wrong pattern as reliably as good ones teach the right one.
What is Few-shot prompting?Fine-tuning is training an existing model a little further on your own examples so it follows a style, format or task more reliably. It changes the model itself, while prompting only changes its input.
It is worth considering when you have many good examples and prompting alone is too long, costly or inconsistent. It is not a good way to add facts that change often; retrieval handles that better. Good traces and reviewed datasets are the raw material for a fine-tuning set, and an evaluation set tells you whether it helped.
What is Fine-tuning?A foundation model is a large model trained on broad data that can be adapted to many tasks, usually through prompting or fine-tuning. Large language models are the best-known kind, but the term also covers image and audio models.
Most teams do not train a foundation model. They choose one from a provider, then shape its behavior with prompts, retrieved context and tools. That choice matters for cost, speed and quality, which is why comparing models on your own traces beats trusting a public benchmark.
What is Foundation model?Function calling is the model provider feature that lets a model return a structured request to run one of your named functions with arguments that match a schema. It is the mechanism behind most tool use.
You declare functions with names, descriptions and JSON schemas for their parameters. The model returns the chosen function and arguments instead of plain text. Clear names and descriptions matter a lot, since the model picks tools using only that text. Validate arguments before running anything with side effects.
What is Function calling?A golden dataset is a small, carefully reviewed set of inputs with trusted expected answers, used as the reference for judging changes. Quality matters more than size.
Build it from real traces, including the failures you have already seen, and have a person confirm each expected answer. Keep it stable so scores are comparable between versions, and add new cases when new failures appear. A golden dataset that never changes slowly stops representing real traffic, so review it regularly.
What is Golden dataset?Ground truth is the known correct answer or label for an input, used as the standard when scoring a model's output.
For some tasks it is clear, like the right category or the correct number. For open-ended tasks there may be many good answers, so ground truth becomes a list of required facts or a rubric instead of one string. Labels made by people are not perfect either, so check agreement between reviewers on a sample.
What is Ground truth?Grounding means tying a model's answer to specific source material, such as retrieved documents or tool results, so each claim can be checked against evidence.
A grounded answer cites or quotes its source and says so when the source does not contain the answer. It is the main defense against hallucination in retrieval systems. You can score it automatically by checking whether each claim is supported by the retrieved text, then review the failures by hand.
What is Grounding?Guardrails are checks placed around a model or agent that block, change or flag inputs and outputs that break your rules, such as unsafe content, personal data, off-topic requests or invalid tool arguments.
They can be simple code, like a schema check or an allow-list of tools, or a second model that classifies content. They work best as layers: validate inputs, restrict what tools can do, and check outputs. Log every time a guardrail fires, since the count and the reasons are a useful quality signal.
What is Guardrails?A hallucination is a confident answer from a language model that is not supported by its input or by reality, such as an invented fact, citation or tool result.
In agents, hallucinations often appear as made-up tool arguments or claims that a step succeeded when it did not, which traces make visible.
What is Hallucination?Human feedback is ratings, corrections or comments from people on an AI system's outputs, from end users or from internal reviewers.
A thumbs up or down from users is cheap but noisy. Expert review is slower and more reliable. Both are most useful when attached to the trace of the run they refer to, so you can see why an answer was rated badly. Reviewed examples become evaluation data and show whether automatic judges agree with people.
What is Human feedback?Human-in-the-loop means a person reviews, approves or corrects an agent's work at defined points, such as before it sends money or deletes data, or after it produces an answer a reviewer samples.
It is the practical answer to actions that are costly to undo. You decide which steps need approval, which can run alone, and how reviewers' labels flow back into your evaluation data. Reviewer decisions are valuable data: each correction shows where the agent falls short.
What is Human-in-the-loop?Input tokens are the tokens you send to a model in a call: the system prompt, conversation history, retrieved documents, tool definitions and the user message. They are usually billed at a lower rate than output tokens.
In agents the input grows with every step, because each loop resends the earlier messages and tool results. That makes input tokens the biggest cost driver in long runs. Prompt caching, shorter history and tighter retrieval are the main ways to reduce them.
What is Input tokens?Instrumentation is the code that records spans, attributes and errors from your application so they can be sent to an observability tool. Without it, nothing is traced.
You can instrument manually, by wrapping a step in a span, or automatically, by letting a library patch your model and framework clients. Manual spans give you control over names and attributes; automatic ones give wide coverage quickly. Most teams start automatic and add manual spans around their own business logic.
What is Instrumentation?A jailbreak is an attempt to make a model ignore its safety rules or instructions, usually through carefully worded input such as role-play or hidden instructions.
It is related to prompt injection but aimed at the model's own rules. No prompt fully blocks it, so defense is layered: input and output checks, limited tool permissions and monitoring. Keep the traces of suspected attempts, since they become test cases for later versions.
What is Jailbreak?A large language model is a neural network trained on very large amounts of text to predict the next token, which lets it write, summarize, translate, reason about text and call tools. Most AI agents use an LLM as the part that decides what to do next.
An LLM takes tokens in and produces tokens out, one at a time. It has no memory between calls beyond what you put in the prompt, and its output can vary from call to call. These properties are why LLM applications need tracing: you have to see the exact input a model received to understand its output.
What is Large language model (LLM)?Latency is the time between sending a request and getting the response. For an AI agent it is the sum of every model call, tool call and wait inside the run, so it is usually far longer than a single model call.
Model latency grows with output length, and agent latency grows with the number of loop passes and sequential tool calls. Measure it per span, not just per run, so you can tell whether time goes to the model, to retrieval or to a slow external API. Report percentiles instead of averages.
What is Latency?LLM cost tracking is measuring what each model call and each agent run costs, from input and output tokens multiplied by the model's price, and attributing it to a project, user, feature or release.
Cost per run and cost per task matter more than the monthly bill alone, because a retry loop or an oversized prompt shows up as one expensive run long before it moves the total.
What is LLM cost tracking?LLM observability is the practice of recording traces, model calls, tool calls, cost, latency, and errors for LLM-powered applications so their behavior can be inspected and debugged in production.
Unlike traditional application logging, LLM observability keeps prompts, model outputs, token usage, and quality evidence attached to the run that produced them, since a single user-visible answer can depend on several chained model and tool calls.
What is LLM observability?LLM tracing is the technique of recording the sequence of model and tool calls behind a single AI agent response so it can be replayed and inspected step by step.
LLM tracing typically builds on distributed tracing concepts (spans, parent-child relationships) extended with AI-specific fields: tokens, cost, model identity, prompt and completion payloads, and time to first token.
What is LLM tracing?LLM-as-a-judge is an evaluation method where one language model scores another model's or agent's output against a rubric, such as correctness, helpfulness or safety.
It scales far beyond manual review, but judges have biases, so teams validate them against a human-labeled sample before trusting the scores.
What is LLM-as-a-judge?Max tokens is a setting that caps how many tokens a model may generate in one reply. If the reply hits the cap, it stops mid-sentence and the call reports a length stop reason.
It protects you from runaway cost and long waits, but a cap set too low truncates answers and can break structured output, leaving invalid JSON. Watch for runs that end because of length, since they are a common hidden cause of bad outputs and failed parsing.
What is Max tokens?The Model Context Protocol is an open standard that lets AI applications connect to external tools and data sources through a common interface of servers and clients.
Each MCP tool invocation is a tool call, so tracing MCP servers shows which tools an agent used and how they behaved.
What is Model Context Protocol (MCP)?Model fallback is automatically switching to a backup model or provider when the first choice fails, times out or is rate limited, so the user still gets an answer.
Fallbacks improve availability but change behavior: a different model may follow instructions differently or cost more. Record which model actually answered on every run, and test the fallback path on purpose, since it is the code that runs least often.
What is Model fallback?A multi-agent system is an application where several AI agents, each with its own role and tools, cooperate by passing tasks and results to each other.
Failures can hide in the handoffs between agents, so observability needs to follow work across agents inside one trace.
What is Multi-agent system?Offline evaluation tests a change on a saved dataset before it reaches users. Nothing live is affected, and the same inputs can be run again for a fair comparison.
It is the right place to compare prompts, models and settings, because only one thing changes at a time. Its weakness is that the dataset is only a sample of reality, so pair it with online evaluation. Gate releases on it: a change that lowers the score does not ship.
What is Offline evaluation?Online evaluation scores real production runs as they happen, or shortly after, instead of testing a fixed set before release. It shows how the agent performs on actual users' requests.
You usually score a sample, using automatic checks or a judge model, and track the results over time and by release. It catches problems that a test set misses, such as new kinds of questions, and it feeds the datasets you use for offline testing. Scoring every run can be costly, so sample.
What is Online evaluation?OpenTelemetry is an open standard for producing and sending traces, metrics and logs, and it is widely used to send LLM and agent traces to an observability backend over the OTLP protocol.
Using OpenTelemetry means instrumentation is not tied to one vendor, so a team can change backends without rewriting its code.
What is OpenTelemetry (OTel)?An orchestrator is the part of an agent system that decides which agent or tool runs next, passes work between them and combines their results. It can be ordinary code or an agent itself.
In a multi-agent setup the orchestrator is where routing mistakes happen: sending a task to the wrong specialist, losing context in the handoff, or running steps in sequence that could run in parallel. Tracing the orchestrator as the parent span makes each worker's contribution visible underneath it.
What is Orchestrator?Output tokens are the tokens a model generates in its reply. They cost more per token than input tokens and are produced one at a time, so they dominate how long a response takes.
Latency grows with the number of output tokens, which is why asking for shorter answers or structured output often speeds up an agent. Models that reason before answering can also spend many output tokens on reasoning you never see, so check the usage numbers rather than the visible text.
What is Output tokens?P95 latency is the response time that 95% of requests are faster than, so only the slowest 5% take longer. It shows what slow-but-normal requests feel like better than an average does.
Averages hide the tail. If most runs take 3 s and one in twenty takes 40 s, the average looks fine but users feel the slow ones. Watching P50, P95 and P99 together tells you the typical experience and the worst realistic one, and a rising P95 is often the first sign of a provider slowdown or a looping agent.
What is P95 latency?A parent span is a span that contains other spans, called its children, because the work it measures started them. The parent-child links are what turn a flat list of spans into a tree you can read from top to bottom.
In an agent run the root span usually covers the whole request. Its children are model calls and tool calls, and a tool call can have its own children, such as a database query. A parent's duration covers its children, so a slow parent with fast children points to time spent in the parent's own code.
What is Parent span?PII redaction is removing or masking personal data, such as names, emails, phone numbers and card numbers, from text before it is stored or sent on.
Traces contain prompts and outputs, which often include personal details. Redacting before storage reduces risk and helps meet privacy rules. Pattern matching catches formats like emails and card numbers, but names and free-text details are harder, so decide what you must remove and check the results on samples.
What is PII redaction?A planner agent breaks a large goal into smaller steps and hands them to other agents or tools to carry out. It focuses on deciding what to do, not on doing it.
Planning first can make long tasks more reliable, but plans can be wrong or go stale as results come in, so many systems re-plan after each step. Compare the plan with what actually ran: a gap between the two is a common source of silent failures.
What is Planner agent?A prompt is the full input given to a language model for one call: instructions, examples, retrieved context, conversation history and the user's request. The model's reply depends entirely on it.
What the user typed is only a small part of the prompt. Most of it is assembled by your code, which is why bugs often hide in prompt building rather than in the model. Seeing the final prompt exactly as the model received it is the first thing to check in any bad answer.
What is Prompt?Prompt caching lets a model provider reuse the already processed beginning of a prompt across calls, so repeated input is cheaper and faster. It only helps when the start of the prompt is identical from call to call.
To benefit, put stable content first, such as the system prompt and tool definitions, and variable content last. Providers report cached tokens separately in usage, so tracing it shows whether your prompt layout is actually getting cache hits. A small change near the top of the prompt can silently break the cache.
What is Prompt caching?Prompt engineering is the practice of writing and testing prompts to get reliable results from a model. It covers instructions, examples, output formats and how context is arranged.
Good prompt engineering is closer to testing than writing: you change one thing, run it on a fixed set of inputs, and compare scores. Without traces and an evaluation set, it turns into guessing. As agents grow, the work widens into context engineering, which decides what goes into the prompt at each step.
What is Prompt engineering?Prompt injection is an attack where text in a user message, web page or document tricks a language model into ignoring its instructions or taking an unintended action.
Agents with tools are most at risk, since a successful injection can trigger real actions. Tracing which input led to which tool call helps investigate incidents.
What is Prompt injection?A prompt regression is a drop in quality caused by changing a prompt: something that used to work now fails, even if the new wording looks better.
Prompts are fragile because small edits shift behavior in unrelated cases. The defense is a saved set of cases that you run before and after every edit, with the prompt version recorded on each run. When a score falls, the version history shows what changed.
What is Prompt regression?A prompt template is a prompt with named placeholders that your code fills in at run time, such as the customer name or retrieved documents. It keeps the wording fixed while the data changes.
Templates make prompts reusable and testable, and they let you version the wording separately from the code that fills it. The usual bugs are missing values, which leave a literal placeholder in the prompt, and unescaped user text that changes the meaning of the template.
What is Prompt template?Prompt versioning is keeping a numbered history of each prompt so you know which wording produced which result and can roll back a change that made things worse.
A one-word edit can change quality, cost and tool use. Recording the prompt version on every run lets you compare versions on real traffic and tie a regression to the change that caused it. Combine it with an evaluation set so each new version is scored before it ships.
What is Prompt versioning?Retrieval-augmented generation is a pattern where your system first looks up relevant documents and then gives them to the model along with the question, so the answer is based on that material instead of the model's memory alone.
It lets a model answer from private or recent data without retraining. Quality depends on two parts: whether retrieval found the right passages, and whether the model used them faithfully. When an answer is wrong, a trace tells you which part failed, because it shows the retrieved text next to the final reply.
What is RAG (retrieval-augmented generation)?A rate limit is a cap a provider sets on how many requests or tokens you may use in a period. When you exceed it, calls are rejected, usually with an HTTP 429 error.
Limits are often per minute, per model and per account tier. Agents hit them easily because one user request can trigger many model calls at once. Handle them with retries that wait, queues that smooth bursts and fallbacks to another model, and watch the count of 429 errors in traces.
What is Rate limit?ReAct, short for reason and act, is an agent pattern in which the model alternates between writing a short thought about what to do and taking an action with a tool, then reads the observation before the next thought.
It was an early, influential recipe for tool-using agents and still underlies many frameworks, though modern models usually do the reasoning internally or through native tool calling. Seeing the thought, action and observation of each cycle in a trace makes it easy to find the step where the reasoning went off course.
What is ReAct pattern?A reasoning model is a language model that spends extra tokens thinking through a problem before it gives an answer. It tends to do better on multi-step tasks, at the price of more latency and cost.
The thinking tokens are billed as output tokens even when the provider hides the text. That makes usage numbers in traces important: a short visible answer can hide a large bill. Many providers let you set how much effort the model spends, so you can use a lower setting for simple steps and a higher one for hard ones.
What is Reasoning model?Reranking is a second pass that reorders the results of a first search using a more careful relevance model, so the best passages come first and weak ones are dropped.
The first search is fast but rough, so you fetch more candidates than you need, perhaps 30. A reranker scores each candidate against the question and you keep the top few. This often improves answers more than changing the embedding model, at the cost of extra latency you can see in the trace.
What is Reranking?Retry with backoff means trying a failed call again after waiting, and waiting longer after each failure, often with a little random jitter. It handles temporary errors without hammering a service that is struggling.
Retry only errors that can succeed later, such as timeouts, rate limits and server errors, and set a maximum number of attempts. Never retry blindly on actions that are not safe to repeat, like creating a payment, unless the call carries an idempotency key. Record each attempt in the trace so hidden retries do not hide real cost and delay.
What is Retry with backoff?Self-hosted observability means running your tracing and monitoring tools on infrastructure you control instead of using a vendor's cloud. Your trace data stays inside your own environment.
Teams choose it for data control, compliance or cost at high volume. The trade-off is operating the system yourself: storage, upgrades, backups and scaling. Check which parts of a tool can be self-hosted and which still depend on a managed service.
What is Self-hosted observability?Semantic search finds text by meaning rather than by exact words, using embeddings to compare a query with documents. A search for "car won't start" can find a page about a dead battery.
It handles synonyms and paraphrases well but can miss exact identifiers like error codes or product names, so many systems combine it with keyword search and rerank the merged results. Always check results on your own questions; similarity scores are not a guarantee of relevance.
What is Semantic search?Session tracing groups the traces of one conversation or user session together, so you can follow a multi-turn exchange instead of looking at each message in isolation.
A chat agent often fails because of something said three turns earlier. Tagging every run with a session ID lets you open the whole conversation in order, see how context grew, and find the turn where the agent went wrong. It also lets you measure things per conversation, such as total cost or number of turns to resolve.
What is Session tracing?A span is a single recorded operation inside a trace, such as one model call, one tool call, or one retrieval step, with its own start time, duration, inputs, outputs, and status.
Spans nest inside a parent trace, forming a timeline or graph. Parallel work, handoffs between agents, and repeated operations all show up as sibling or child spans under the same run.
What is Span?Span attributes are key-value pairs attached to a span that describe what happened, such as the model name, token counts, tool name or user plan. They are what you filter, group and chart by once traces are stored.
Good attributes are specific and low in variety, like model, environment and release. Avoid putting whole prompts or personal data in attributes that are used for filtering; keep large payloads as span input and output instead. OpenTelemetry defines standard attribute names for generative AI so different tools agree on what they mean.
What is Span attributes?A span event is a timestamped note recorded inside a span, marking something that happened at a moment in time rather than over a duration. Typical examples are an exception, a retry, or the arrival of the first streamed token.
Events keep a span readable. Instead of creating a tiny span for every small moment, you add an event to the span that was running. Exceptions are the most useful case, since the error message and stack trace stay attached to the exact step that failed.
What is Span event?Streaming sends a model's reply to the user in pieces as it is generated, instead of waiting for the whole answer. The total time is the same, but the first words appear much sooner.
It makes an application feel faster, and the metric to watch is time to first token. For tracing, a streamed call is one span that lasts until the final token, so record the first-token time as an event or attribute. Errors can happen midway, after some text has already been shown, which needs deliberate handling.
What is Streaming?Structured output is a model reply that follows a fixed format, usually JSON that matches a schema, so your code can read it without guessing. Many providers can enforce the schema while the model generates.
Free text is hard to parse reliably. A schema gives you named fields, types and allowed values, and turns most formatting failures into a validation error you can count. Truncation, refusals and a schema that is too loose are the remaining failure modes to watch in traces.
What is Structured output?A system prompt is the standing set of instructions sent at the start of every call that tells the model its role, rules, tone and what tools it may use. The user does not see it.
It is where you set boundaries, such as what to refuse and how to format answers. Because it is repeated on every step, it adds to input tokens and is a good candidate for prompt caching. It also is not a security boundary: a determined user can sometimes override or extract it, so enforce important rules in code as well.
What is System prompt?Temperature is a setting that controls how random a model's output is. Low values make answers more predictable and repeatable, and high values make them more varied.
For extraction, classification and tool calling, a low temperature is usually the safer choice. For brainstorming or creative writing, higher values help. Even at zero, outputs are not guaranteed to be identical, and some reasoning models fix the setting for you, so record it as an attribute when you compare runs.
What is Temperature?Throughput is how much work a system completes in a given time, such as runs per minute or tokens per second. It is about volume, while latency is about the speed of a single request.
A system can have good latency at low traffic and fall over at high traffic when it hits provider rate limits or runs out of workers. Track throughput next to error rate and latency so you can see the load level at which things start to degrade.
What is Throughput?Time to first token is the delay between sending a request to a language model and receiving the first streamed token of its response.
TTFT is tracked separately from total call latency because it drives perceived responsiveness in streaming interfaces, even when total generation time is unchanged.
What is Time to first token (TTFT)?A token is the unit of text a language model reads and writes, usually a word piece of about four characters of English. Model providers count and bill usage in tokens, and context limits are measured in them.
Common words are often one token, while rare words, numbers and code split into several. Other languages can use more tokens for the same meaning. Because price and speed both scale with tokens, knowing how many a call used is the starting point for cost control.
What is Token?Token usage is the number of input (prompt) and output (completion) tokens a language model processes for a call, which determines both its cost and how much of the context window it uses.
Cached input tokens are usually billed at a lower rate, so good tracking records input, cached and output tokens separately.
What is Token usage?A tool call is an action an AI agent takes outside the language model itself, such as a search, database query, browser action, or API request, recorded with its arguments, result, latency, and failure state.
Tool calls are attached to the reasoning span that selected them, which makes it possible to tell whether a bad outcome came from the agent choosing the wrong tool or from the tool itself failing.
What is Tool call?Tool call tracking is recording every tool an AI agent calls, with its arguments, result, duration and errors, so you can see which tools are slow, fail often or are called with the wrong inputs.
Agents often fail at the tool step: a wrong argument, a timeout or a stale result. Tracking tool calls as spans inside the run links each call to the model decision that triggered it and to the cost of the run.
What is Tool call tracking?Tool use is a model's ability to ask for an external function to be run, such as a search, calculation or API call, and then continue with the result. The model asks; your code actually runs the tool.
Each tool is described to the model by name, purpose and parameters. The model replies with the tool it wants and the arguments, your code executes it, and the result goes back into the next call. Most agent failures involve this exchange: the wrong tool, wrong arguments or a result the model misreads.
What is Tool use?Trace context propagation is passing the trace ID and current span ID from one service to the next, usually in an HTTP header, so the next service can continue the same trace instead of starting a new one.
The common standard is the W3C traceparent header. The caller writes it on the outgoing request and the receiver reads it before starting its first span. If a hop drops the header, the trace splits in two, which is the most frequent reason a distributed trace looks incomplete. Queues and background jobs need the context stored in the message.
What is Trace context propagation?A trace ID is the unique identifier shared by every span in one trace, so a backend can gather all the steps of a single run back together. It is usually a random 128-bit value written as 32 hexadecimal characters.
Each span carries the trace ID of the run it belongs to, plus its own span ID. When spans arrive out of order, or from different services, the trace ID is what lets them be grouped into one timeline. Putting the trace ID in your application logs and error reports is the quickest way to jump from a user complaint to the exact run.
What is Trace ID?Trace sampling is recording only a share of all traces, chosen by a rule, to keep storage and cost down. Head sampling decides when a trace starts; tail sampling decides after it finishes, so it can keep every failure.
Sampling is a trade-off between cost and visibility. A fixed percentage is simple but can drop the one run that failed. Teams often keep all errors and all slow runs and sample the healthy rest. Whatever you choose, record the sampling rate so counts and totals can be scaled back up honestly.
What is Trace sampling?A trace viewer is a tool that shows one AI agent run as a timeline of spans, so you can open each model call, tool call, retry and error in order and see what the agent did and why it failed.
A good trace viewer shows inputs, outputs, timing, tokens and cost for every step, and lets you jump from a bad answer to the first step that went wrong. Tracify's trace viewer does this for every run sent through the SDK or OpenTelemetry.
What is Trace viewer?A trace waterfall is a chart that lays out the spans of one trace as horizontal bars on a shared time axis, indented by parent and child. It shows at a glance what ran in sequence, what ran in parallel and where the time went.
Long bars are slow steps, gaps between bars are waiting or untraced work, and stacked bars that start together ran in parallel. Reading a waterfall is usually the first step when a run is slow, because it separates a slow model call from a slow tool, or from a chain of sequential calls that could run at the same time.
What is Trace waterfall?A vector database stores embeddings and quickly finds the ones closest to a query embedding. It is the usual storage layer behind semantic search and retrieval-augmented generation.
You index document chunks with their embeddings and metadata, then search by similarity with optional filters such as customer or date. Latency, index freshness and filters are the practical concerns. Trace each search as its own span with the query, number of results and scores, since weak scores are an early sign of poor retrieval.
What is Vector database?Open a real trace and watch these terms move.
