AI Agent Evaluation Datasets: How to Build Test Cases From Real User Interactions

At a glance

This guide is for engineers and applied-AI practitioners who already have an evaluation framework — metrics, scoring, a judge or human review process — but an evaluation dataset that is thin, stale, or invented. It answers one question directly: build an AI agent evaluation dataset by sourcing cases from five production signal types — user-reported failures, low human-review scores, expensive or retried runs, tool-call errors, and random sampling — then sanitize sensitive fields, tag each case by failure category, and version the dataset so results stay comparable release over release. The rest of this article works through that process end to end: where to pull cases from, how to structure them, how to sanitize them responsibly, and how to keep the set representative as the agent changes.

This is a construction guide, not a scoring guide. If you need help choosing metrics or a judge model, see AI evaluation metrics and the broader evaluation practice — this piece assumes you already have somewhere to run cases and focuses only on where the cases themselves should come from.

The Five-Signal Sourcing Model

Every useful evaluation case starts as a real event, not a guess. The Five-Signal Sourcing Model names the five places a representative case can come from: a user reported the run as wrong, a human reviewer scored it low, the run was unusually expensive or had to retry, a tool call failed, or the run was picked by plain random sampling to keep the set honest about what "normal" looks like. Each signal produces a different kind of failure evidence, and a dataset built from only one of them will be blind to the others.

What makes an evaluation dataset useful, not just large

A dataset with hundreds of cases that all resemble each other is not more useful than one with forty — it's the same case repeated with different words. Usefulness comes from coverage of what actually goes wrong, not from raw count. A set built entirely by imagining what might break tends to cluster around the failures the author already knows about and misses everything else: the tool an integration partner changed last month, the phrasing a specific user segment uses, the retry loop that only shows up under load.

Two properties matter more than size:

• It reflects the shape of production traffic, including the failures that are rare but expensive and the ones that are common but easy to miss in aggregate metrics.
• It stays attached to evidence. Every case should trace back to something that actually happened — a flagged run, a low score, a failed call — so a reviewer can ask "why is this case here?" and get a real answer.

A dataset authored purely from imagination isn't wrong to have — synthetic cases genuinely help with edge and adversarial coverage a production system hasn't produced yet — but it cannot be the whole dataset. It has no way to catch a regression in something the author didn't think to imagine, which is exactly what happened in the scenario at the top of this article.

There's also a quieter cost to an imagined dataset beyond missed coverage: it tends to test the agent against the easy version of a hard case. Someone writing a test case from memory of "what tool-call failures look like" will usually write a clean, obviously-wrong example, because that's what's memorable. A tool-call error sourced directly from a trace, by contrast, comes with the actual malformed arguments, the actual error message, and the actual (often subtly wrong, not obviously wrong) final response the agent produced. That subtlety is frequently the whole point — the failures worth testing for are the ones a reviewer could plausibly miss, not the ones that would be caught by casual inspection anyway.

Five sources of real signal to mine

User-reported failures

Anywhere a user can flag a run, correct an answer, or escalate to a human, that action produces a labeled failure with no extra work. These are the highest-value cases because they came from the person the agent is supposed to serve, not from an internal reviewer's interpretation of correctness. Pull the flagged run's input, the agent's actual output, and — where available — what the user expected or corrected it to.

The tradeoff: user reports are biased toward failures visible to the user. A wrong tool call that produced a plausible-looking answer will never be user-reported even though it's a real defect. This signal alone under-represents silent failures, which is why it needs the other four.

Low-scoring human reviews

If part of your evaluation process already includes human review — spot-checking a sample of runs against a rubric — the cases that scored below your acceptance threshold are ready-made evaluation candidates. Unlike user reports, these catch failures the user might not have noticed: a subtly ungrounded citation, a tone violation, a technically correct but unhelpful answer.

Because a reviewer applied a rubric, these cases usually come with structured reasoning already attached (the score, the criterion that failed, the reviewer's note), which maps directly into the test-case schema below with almost no extra annotation work.

Expensive or retried runs

A run that cost several times the typical amount for its task, or needed multiple tool-call retries to finish, is evidence of a reasoning or efficiency problem even when the final answer looked fine. These cases are easy to miss because they don't fail outright — they just cost more than they should, which means they usually don't show up in a "wrong answer" review at all.

Sourcing this signal means querying cost and retry-count outliers directly from your trace store rather than waiting for a human or user to notice — nothing about an expensive-but-successful run makes it visible any other way.

Tool-call errors

Any run where a tool call returned an error, an unexpected shape, or was invoked with invalid arguments is a mechanical, unambiguous failure signal — no judgment call required to decide it belongs in the dataset. These cases are valuable because they reproduce cleanly: the input, the tool call attempted, and the error returned are all present in the trace already.

Tool-call errors also tend to cluster around integration changes — an API that changed its response shape, a rate limit that started triggering, a parameter that became required. A dataset with good tool-error coverage catches exactly the kind of regression described in this article's opening scenario.

Trade-off: Sourcing every case from failure signals produces a dataset that's excellent at catching regressions and poor at telling you whether ordinary traffic is still healthy. A dataset skewed entirely toward failure cases can pass every regression check while a slow, non-failing quality drift goes completely undetected — which is exactly what random sampling below is for.

Representative random sampling

The first four signals are all failure-seeking by design, which means a dataset built only from them will overrepresent failure and underrepresent everything that's working normally. Random sampling — pulling a fixed percentage of unflagged, ordinary runs on a regular cadence — keeps the dataset honest about what typical traffic actually looks like, and gives you a baseline to detect regressions in behavior that hasn't failed yet but has started to drift.

Without this signal, a team can end up over-optimizing for the failure modes that were easiest to notice while a real regression grows quietly in the "normal" 95% of traffic nobody is sampling.

A test-case schema

Once a case is sourced, it needs a consistent shape so it can be filtered, versioned, and compared across releases. The following schema keeps enough information to trace every case back to its source and category without requiring a redesign every time a new signal type gets added.

Treat expected_behavior as a description of an acceptable outcome, not a single golden string — an evaluator or judge comparing against an exact match will fail correct answers phrased differently. source_signal records where the case came from and never changes, while category records what it's testing for and can be corrected as understanding improves.

Sanitizing production data before it becomes a test case

Sourcing a case from production does not mean the raw trace goes into the dataset as-is. Every case should pass through sanitization before it's stored — the standard here is a starting engineering practice, not a compliance guarantee, and if your production data includes anything sensitive, involve your own legal or privacy function before deciding what's acceptable to retain.

A working sanitization checklist before a case is marked sanitized:

• [ ] Direct identifiers (names, emails, phone numbers, account IDs) are removed or replaced with stable placeholders.
• [ ] Free-text fields are scanned for identifiers that weren't in a structured field — a user pasting their own email into a chat message, for example.
• [ ] Authentication material, API keys, and session tokens are stripped from tool-call arguments and responses.
• [ ] Internal system details unrelated to the test (internal hostnames, unrelated customer names in a retrieved document) are redacted.
• [ ] The sanitized input still reproduces the original failure — over-redaction that removes the actual trigger condition produces a useless case.
• [ ] A second reviewer confirms the sanitized version before sanitization_status moves from pending-review to sanitized.
• [ ] Cases that can't be safely sanitized without destroying the failure signal are rebuilt as a synthetic case instead of stored raw.

That last point matters: if sanitizing a case would strip out the exact detail that made it fail, don't force it into the dataset raw. Rebuild the same failure shape with fabricated details instead — you lose the case's status as "verified real" but keep the coverage, and mark sanitization_status as synthetic so nobody mistakes it for a cleared production case later.

Positive, negative, edge, and adversarial cases

The five-signal model sources real failures well, but a dataset built only from what already happened will always lag behind what could happen. Round it out with four case types:

• Positive cases — runs that succeeded and should keep succeeding. These come mostly from random sampling and protect against regressions in behavior that currently works.
• Negative cases — the failures sourced from the four failure-seeking signals above.
• Edge cases — inputs at the boundary of what the agent is expected to handle: unusually long inputs, ambiguous requests, missing context. Real traffic eventually produces these, but synthetic authoring can add coverage before it does.
• Adversarial cases — inputs designed to probe for policy violations, prompt injection, or unsafe tool use. These are rarely present in early production traffic in meaningful volume, which makes synthetic authoring appropriate and often necessary here specifically.

Synthetic cases are not a lesser substitute for real ones — they're the right tool for edge and adversarial coverage that production traffic hasn't generated yet. Real traces anchor the dataset to what the agent actually encounters; synthetic cases extend it to what it hasn't encountered but should be able to handle. A mature dataset needs both.

Representativeness: how much is enough, and of what

There's no fixed number of cases that makes a dataset "done." The useful question isn't "how many cases" but whether every known failure category has more than one example from more than one source. A category with exactly one case can't distinguish a fixed regression from a coincidence; a category with zero cases means that failure class is currently invisible to evaluation entirely, no matter how large the total dataset looks.

Track representativeness with two simple checks rather than a target count:

1. Category coverage — list every known failure category and confirm each has cases from more than one source signal where possible. A dataset built this way pairs naturally with regression testing, since a category with real coverage is exactly what a regression suite needs to catch a reintroduced failure.
2. Source-signal balance — periodically check the proportion of cases from each of the five signals. If 90% of the dataset is user-reported failures, silent failures picked up by tool-call errors or expensive-run sampling are underrepresented, regardless of total size.

Grow the dataset over time as new categories appear rather than trying to front-load a complete set before running a single evaluation. A thirty-case dataset that gets ten new, well-sourced cases every release cycle will out-catch a five-hundred-case dataset that was written once and never touched again.

Representativeness also has a failure mode in the other direction: overcorrecting toward failure coverage until the dataset no longer resembles production at all. If every case in the set is a documented failure, an evaluation run that scores well on it tells you the agent handles known past failures — it says nothing about whether the agent still performs well on the ordinary requests that make up most of its traffic. That's what the random-sampling signal protects against, and it's worth deliberately checking, not just assuming the other four signals will stay balanced on their own. A quarterly review of the source-signal mix — literally counting how many stored cases came from each of the five sources — catches this drift early, before an evaluation score becomes a measure of failure-avoidance rather than of overall quality.

Versioning and ownership

Evaluation scores are only meaningful relative to a fixed dataset. If cases are added, removed, or reclassified between two evaluation runs, a score change might reflect a real behavior change, a dataset change, or both — and there's no way to tell which without knowing what changed. Version the dataset the same way you'd version a schema or an API: bump the version whenever cases are added, removed, or have their category or expected_behavior changed, and record which version an evaluation run was scored against.

Ownership works best when it's split into two roles: someone who owns agent quality end to end owns the dataset's structure, thresholds, and what counts as "good enough" coverage, while anyone on the team who spots a new failure pattern should be able to propose a case — a support engineer noticing a repeated user complaint, a reviewer flagging a pattern in low-scoring runs, an on-call engineer who traced an incident to a specific tool-call shape. Restricting case contribution to one team turns the dataset into a reflection of what that team happens to notice, which reproduces the same blind-spot problem as writing every case from imagination in the first place.

From trace to test case: a worked example

Here's a raw trace excerpt — the kind a tool-call-error query against a trace store would surface — moving through sourcing, sanitization, and categorization into a stored test case.

Raw trace (before):

A support agent received the message "my account is jane.doe@example.com, can you cancel my renewal and refund the last charge of $214.50?" The agent called a cancel_subscription tool with {"account_email": "jane.doe@example.com", "refund_amount": 214.50}, and the tool returned an error because the refund endpoint required a transaction_id the agent never looked up. The agent's final response incorrectly told the user the refund had been processed.

What sourcing and sanitization do to it:

• The direct identifier (jane.doe@example.com) is replaced with a stable placeholder.
• The specific dollar amount is generalized, since the exact figure isn't what caused the failure — the missing transaction_id lookup is.
• The tool-call error and the agent's incorrect final claim are preserved exactly, since that's the failure signal the case exists to catch.
• A second reviewer confirms the sanitized version still reproduces the original failure before it's marked sanitized.

Sanitized, categorized test case (after) — illustrative:

{
  "input": "my account is {{user_email}}, can you cancel my renewal and refund the last charge?",
  "expected_behavior": "Looks up the required transaction_id before calling the refund tool, or tells the user the refund could not be confirmed rather than claiming it succeeded.",
  "category": "tool-argument-validity",
  "source_signal": "tool-call-error",
  "sanitization_status": "sanitized",
  "version": "2026.09.1"
}

That single case now does something the original 40 hand-written cases in this article's opening scenario never could: it directly tests whether the agent still looks up a transaction_id before calling the refund tool, because that's the exact mechanism that broke in production, not a guess about what might break.

Evaluation-Case Builder

Pick a source signal below to see, using one fixed illustrative trace, which fields typically get extracted into a stored test case and which get redacted or dropped before it's saved.

The same mapping, as a static reference if you're skimming rather than clicking through each option:

Turning the five signals into a repeatable pipeline

None of the five signals are useful as a one-time export. A dataset sourced once during a sprint and never revisited decays the same way the 40 hand-written cases in this article's opening scenario did — it freezes a snapshot of what the agent got wrong at one point in time and stops updating as the agent, its tools, and its traffic change underneath it. Treat sourcing as a recurring job, not a project.

A minimal version of that job has four stages, run on a fixed cadence (weekly is a reasonable starting point for most teams):

1. Query each signal source. User reports usually come from a support or feedback table; low review scores from wherever your human-review process stores results; expensive/retried runs and tool-call errors from your trace store, filtered by cost, retry count, or error status; random samples from a fixed-percentage pull of unflagged runs.
2. Deduplicate against the existing dataset. Two support tickets about the same underlying failure shape shouldn't become two nearly identical cases — check for near-duplicate inputs and expected behaviors before adding a new one.
3. Route new candidates to sanitization review. Nothing produced by step one goes directly into the dataset; it lands in a pending-review queue first, as described in the checklist above.
4. Tag category and source signal, then version. Once a reviewer clears a case, it gets its category, keeps its source_signal, and the dataset version is bumped to reflect the addition.

The following illustrative pseudocode shows the shape of the query stage for one signal — tool-call errors — against a trace store. It is not a runnable snippet against any specific Tracify API; it's meant to show the filtering logic a real query would need, which is the part that transfers across trace stores.

# Illustrative — shape of a tool-call-error sourcing query, not a runnable API call.
candidates = trace_store.query(
  time_window="last_7_days",
  filter={
    "tool_call.status": "error",
    "run.already_in_dataset": False,
  },
  fields=["input", "tool_call.name", "tool_call.arguments", "tool_call.error", "final_response"],
  limit=200,
)

The other four signals follow the same shape with different filters: user_report.flagged == true for user reports, review.score < threshold for low review scores, run.cost > p95_cost OR run.retry_count > 1 for expensive/retried runs, and a fixed-percentage random selector with no filter at all for representative sampling. Keeping all five queries in one place is what makes source-signal balance something you can actually check instead of something you assume, rather than one team's ad hoc export and another team's separate spreadsheet.

Automating the query stage doesn't mean automating the whole pipeline. Sanitization review and category assignment should stay a human step for as long as a case might contain sensitive data or require judgment about what it's actually testing — automating sourcing removes the tedious part (finding candidates) without removing the part that needs a person (deciding what's safe to keep and what it means).

Protect the sourcing boundary

Sourcing a dataset from production traffic touches real user data by definition, so the sourcing pipeline itself needs boundaries. Whoever or whatever pulls traces for dataset candidates should have read-only access scoped to what sourcing needs, not general production database access. Sanitization should happen before a case is stored anywhere outside the production system it came from — not after, and not "eventually." And the sanitization_status field exists specifically so a case can never silently move from pending-review to being used in an evaluation run without a second person confirming it's clean; treat a case stuck at pending-review as unusable, not as usable-with-caveats.

None of this replaces your organization's actual data-handling policy — it's the minimum structure that keeps a dataset-sourcing pipeline from becoming an unreviewed second copy of production data sitting in a less-protected place.

Limitations of a trace-sourced dataset

Sourcing from production solves the "authored from imagination" problem, but it introduces limitations of its own worth naming rather than glossing over.

It lags behind new failure modes by definition. A source signal can only surface a failure after it has already happened to a real user or run at least once. For a genuinely new integration, a newly launched agent, or a capability nobody has used in production yet, there is no trace history to mine — that gap is exactly where synthetic and hand-authored edge/adversarial cases carry more of the weight, at least until real traffic accumulates.

It can encode a period of bad behavior as "expected." If a case is sourced during a stretch where the agent was already misbehaving in a way nobody caught yet, and the reviewer marking expected_behavior isn't careful, the dataset can end up asserting the wrong thing as correct. This is why the sanitization checklist's second-reviewer step matters for more than privacy — a second reviewer is also a check against writing down a plausible-sounding but incorrect expected_behavior.

It's only as good as what gets instrumented. A signal you don't query for is a signal you'll never source from, no matter how often it happens in production. If your trace store doesn't record retry counts, the "expensive or retried runs" signal is invisible to you regardless of how real the underlying problem is — sourcing quality is downstream of tracing quality, not independent of it.

It needs a workflow-specific lens for retrieval-heavy agents. The five signals apply the same way to a retrieval-augmented agent, but the categories they produce often need RAG-specific evaluation to interpret correctly — a low-scoring review on a retrieval-heavy run may point to a retrieval problem rather than a generation problem, and the case's category should reflect that distinction rather than lumping every groundedness failure together.

It can't fully replace domain expertise. A domain expert reviewing failure categories will sometimes recognize a failure shape the five signals haven't surfaced yet — a regulatory nuance, a competitor's behavior users expect to be matched, an edge case specific to one customer segment. Sourcing from traces is the foundation, not a reason to stop asking people who understand the domain what's missing.

None of these limitations argue against trace-sourced datasets — they argue for treating sourcing as one input to a dataset that also includes synthetic and expert-authored cases, reviewed periodically rather than assembled once and trusted indefinitely.

Make the sourcing decision

The team in this article's opening scenario didn't need more hand-written cases — they needed a pipeline that pulled from what the agent was actually doing. The next tool-call regression that ships will be caught by a case sourced from an error just like it, not by the fortieth case someone imagined months before it happened.

Leave the reader with the next move

• [ ] A query exists for each of the five signals, run on a regular cadence, not as a one-time export.
• [ ] Every stored case has passed the sanitization checklist and carries a sanitization_status.
• [ ] Known failure categories each have more than one case, sourced from more than one signal where possible.
• [ ] The dataset version is bumped whenever cases are added, removed, or reclassified, and evaluation runs record which version they scored against.
• [ ] Ownership is clear: one role owns dataset structure and thresholds; anyone can propose a case.

Frequently asked questions