AI Agent Evaluation in Production: A Practical Debugging Framework

At a glance

If your AI agent is already serving users, offline benchmark scores are no longer enough. The question is not only whether a model can produce a good answer. It is whether the complete system chose an acceptable plan, called the right tools, handled uncertainty, respected permissions, and completed the user’s job at a sustainable cost.

This guide is for engineers and operators who need to make that decision from production evidence. You will learn a framework called TRACE: target, record, assess, classify, and evolve. It is a decision framework, not a promise that every agent can be reduced to one score. It deliberately does not cover model training, legal certification, or a vendor-specific evaluator API.

Why production evaluation is different

An offline test usually starts with a fixed input and checks an output. A production agent has a changing context: user permissions, retrieved documents, tool availability, model routing, retries, timeouts, and side effects. The same prompt can therefore produce materially different executions.

That difference creates a common trap. A team sees a lower answer score and edits the system prompt. The next sample improves, but the actual failure was stale retrieval or a tool returning an incomplete record. Another team sees a high judge score and ignores that the agent made three unnecessary calls and exposed an internal identifier in its explanation. Both teams are optimizing a symptom.

The AI agent evaluation practical guide is useful for designing datasets and evaluators. This article starts later in the lifecycle: the run has happened, a user or monitor has raised a question, and you need to decide what evidence justifies a change.

The TRACE operating model

TRACE turns “the agent felt wrong” into a sequence of evidence checks.

The order matters. A classification made before the trace is read is usually a guess. An evolution step without a regression set is an uncontrolled experiment in production.

Define target before score

Write the task contract in observable terms. “Helpful” is too vague. For a support agent, a stronger contract might be: identify the correct account, retrieve the current policy, answer with a supported explanation, avoid unauthorized changes, and ask for a human when confidence is insufficient.

Separate hard constraints from preferences. A fabricated policy citation is a release blocker even if the prose is fluent. A response that takes one extra second may be acceptable if it prevents a wrong account update. Quality dimensions need an order of precedence before they can guide a release.

Record the whole execution

The useful unit is the run, but the useful diagnosis is usually a span inside that run. Record a stable run identifier, task type, environment, model route, prompt version, retrieval context, tool arguments and results, retries, latency, token usage, estimated cost, final output, and failure state. Redact secrets and minimize personal data before retention.

Assess quality without hiding failures

Use several narrow checks instead of one composite score. A task can pass correctness while failing authorization. It can pass relevance while violating a format contract. It can be safe but too slow for the user’s workflow.

A practical evaluation record contains a task label, a rubric version, individual dimensions, evidence, and a disposition. The disposition should be one of pass, review, retry, or fail. Keep the explanation short but concrete: “selected invoice_lookup when the request required subscription_lookup” is more actionable than “agent quality: 0.42.”

Use judges as reviewers, not authorities

LLM-as-a-judge evaluation can scale review, especially for relevance, completeness, and style. It is weaker when the answer depends on a private fact, exact arithmetic, authorization, or a side effect. For those dimensions, prefer deterministic assertions or human review.

The LLM-as-a-judge evaluation guide explains rubric design in more detail. In production, also sample judge disagreements. A stable high score with frequent user corrections is a monitoring failure, not evidence that users are wrong.

Classify the failure by its first divergence

Four categories cover many agent incidents. They are not mutually exclusive, but the first divergence gives you the smallest useful place to start.

Prompt failures

Prompt fixes are appropriate when the trace shows the agent had the required facts and permissions but ignored a clear instruction. Test the smallest change possible. Pin the prompt version, replay a representative set, and look for regressions in refusals, tool selection, verbosity, and cost.

Do not use a prompt to compensate for missing validation. “Always use the exact account ID” does not make an unvalidated account ID safe. The application or tool boundary should enforce constraints that must never be violated.

Tool failures

Tool problems often masquerade as reasoning failures. A tool can silently return an empty list, change its field names, apply a timezone differently, or accept a syntactically valid but semantically dangerous argument. Instrument tool inputs and outputs with a redacted schema-aware representation, and record whether the result was authoritative, partial, or stale.

Illustrative TypeScript for a narrow tool boundary:

type ToolResult =
  | { ok: true; source: "authoritative" | "cached"; records: number }
  | { ok: false; kind: "timeout" | "invalid_input" | "permission" };

function mayContinue(result: ToolResult): boolean {
  return result.ok && result.source === "authoritative" && result.records > 0;
}

This is illustrative rather than a drop-in Tracify SDK call. The important behavior is explicit: a cached or empty result is not silently treated as proof. Add tests for malformed arguments, partial results, retries, and permission boundaries.

Retrieval and context failures

Retrieval quality is part of agent quality. Measure whether the needed source was retrieved, whether it was current, whether contradictory sources were present, and whether the final answer cited or reflected the evidence. A larger context window does not guarantee better context; irrelevant passages can make the decision less reliable.

The RAG evaluation guide gives a deeper evaluation model. For a production incident, begin with one failed trace: compare retrieved chunks, rank, metadata filters, truncation, and the answer’s factual claims.

Carry one incident through the framework

Imagine an internal operations agent asked: “Find the latest failed payment for customer 1842 and draft a message; do not refund anything.” The agent retrieves a stale payment record, calls a refund-capable tool, retries after a timeout, and drafts a confident message with the wrong date.

The visible symptom is a bad message. TRACE finds several boundaries:

1. The target included “latest” and “do not refund.”
2. The record shows the retrieval filter used an old account snapshot.
3. The assessment fails freshness and side-effect safety, even before prose quality.
4. The first divergence is retrieval freshness; tool selection is a second risk.
5. The evolution should fix the account filter, make refund tools unavailable to this task, and add a regression case.

The safest fix may be a permission change, not a better prompt.

The counterfactual is useful: replay the same request with the correct current record and a read-only tool set. If the agent now drafts the correct message, the prompt was not the primary cause. If it still misstates the date, inspect the synthesis step next.

Decision rule: If a failure could have caused an unauthorized side effect, treat it as a safety incident even when the final answer looks correct.

Turn evaluations into release gates

An evaluation matters when it changes an operating decision. Define thresholds per task family rather than one universal number. For a low-risk summarizer, a small sample may support a canary. For an agent that changes billing state, correctness, authorization, and auditability may all be hard gates.

Use a fixed evaluation set for regression and a sampled production set for drift. The first tells you whether a change broke known behavior. The second tells you whether reality is moving outside the set.

Build the feedback loop

The feedback loop should have an owner and a cadence. On every release, run the fixed regression set. Daily or weekly, review sampled production traces by task type. After an incident, add the trace-derived case to the regression set only after removing secrets and irrelevant personal data.

A production evaluator should produce an action, not just a number.

Track changes as hypotheses: “Adding a freshness filter will reduce stale-answer failures without increasing latency beyond the task budget.” A hypothesis makes the expected trade-off explicit. It also prevents a dashboard from becoming a graveyard of unexplained scores.

Watch for drift

Drift can appear in inputs, tools, retrieved data, user expectations, or model behavior. Segment metrics by task, model route, environment, tenant class, and prompt version. Overall averages can improve while a high-value task degrades.

Alert on changes that have an operational response. A useful alert names the segment, comparison window, threshold, and owner. “Quality down” is not an alert; “read-only billing lookup correctness below its release floor for two consecutive windows” is.

Field note: Store the evaluator version with every result. Otherwise a later rubric edit can make an old score look comparable to a new one when it is not.

Make rollback boring

Version prompts, tools, routing rules, evaluators, and retrieval configuration independently where practical. Keep the previous known-good configuration available. A rollback should reduce exposure without deleting traces needed for diagnosis.

Make the evidence legible to the team

Evaluation only compounds when another person can understand it later. Give every run a stable link or identifier, but make the surrounding record useful even if the dashboard is unavailable. The record should answer five questions: what task was attempted, which version ran, what evidence was available, where the first divergence occurred, and what action followed.

Keep evaluator explanations tied to observable evidence. For a factuality check, identify the source passage used by the rubric. For a tool check, retain the normalized operation name and validation result rather than a raw secret-bearing payload. For a latency check, show the critical path and retry count. This makes review faster and reduces arguments about whether a score “feels right.”

Assign ownership by failure class

Ownership prevents every incident from becoming a model-team problem. Prompt and policy regressions usually need the application or agent owner. Tool contract failures belong with the integration owner. Retrieval freshness belongs with the data or search owner. Unsafe side effects need a security or product decision, not just a better evaluator.

The owner is responsible for the next experiment, not necessarily for the original cause. Record that distinction so incident review stays factual.

Segmented evidence beats a flattering global average.

A failed evaluation is useful when it narrows the next experiment.

Keep hard safety failures separate from soft quality preferences.

Review the smallest span that changed the user outcome.

Treat review time as an engineering constraint too: a trace that takes minutes to interpret will not support a healthy operating cadence.

That discipline keeps evaluation connected to shipping decisions, incident learning, and measurable user outcomes over time.

Keep a change journal

A lightweight journal connects evaluation movement to engineering work. Record the hypothesis, changed artifact, affected task segments, baseline, result, and decision. Include negative results. A prompt change that improved summaries but increased unsupported claims is valuable evidence when it is preserved.

This journal also protects against false causality. If a model route, retrieval index, and system prompt all changed together, an improved score does not tell you which change helped. Prefer one meaningful variable per experiment unless the changes are inseparable for safety.

Security and privacy boundaries

Evaluation data is often more sensitive than ordinary logs because it combines user requests, retrieved documents, tool arguments, and outputs. Redact credentials, tokens, payment details, personal identifiers, and private document content according to your retention policy. Use tenant-aware access controls and ensure evaluators cannot invoke production tools.

Do not send raw production traces to an external judge without reviewing the data path and authorization. Store the minimum evidence needed for the rubric. For high-impact actions, preserve an auditable human approval record rather than treating a judge score as authorization.

The interaction at the top of this article is a deterministic teaching scenario, not a live connection to a project. Its result does not inspect your production data and should not be used as a release decision without your own evidence.

What changes the recommendation

The TRACE framework is intentionally conservative, but the right threshold depends on the task. A creative writing agent may prioritize user preference and style. A research agent may need citation coverage and source freshness. A customer-support agent may prioritize identity, policy accuracy, and escalation. A workflow agent that can mutate records needs the strongest tool and human-review gates.

Do not compare teams by a single “agent quality” score when their tasks have different risk. Compare the dimensions that matter to each job, and report uncertainty when the sample is small. A ten-point improvement from ten examples is a lead, not a fact about the system.

A practical production checklist

• [ ] Define the user outcome and hard constraints for each task family.
• [ ] Capture the run and span evidence needed to reconstruct decisions.
• [ ] Evaluate correctness, safety, reliability, cost, and observability separately.
• [ ] Locate the first divergence before changing a prompt or model.
• [ ] Test one fix against a fixed regression set and sampled production traces.
• [ ] Segment results by task, route, version, environment, and risk.
• [ ] Redact sensitive data and isolate evaluators from production side effects.
• [ ] Assign an owner, release threshold, and rollback path.
• [ ] Add confirmed incidents to the regression set.

Start with one important task, not every agent behavior at once. Pick ten to twenty representative traces, label the first divergence, and make one reversible change. The goal is not a perfect score. It is a system where each score points to evidence and each evidence-backed change can be tested.

Frequently asked questions