Product / Evaluation engine
Quality you can repeat.
Build datasets from real traces, run code checks or model judges, and score live traffic with the same evaluators. Know a change is better before it ships, and notice when it stops being true.
- Datasets
- Versioned
- Evaluators
- Typed
- Monitors
- Live
Replace spot checks
Test the behavior, then watch it live.
Choose examples and expected outcomes that represent the product promise or risk being tested.
Apply versioned evaluators, compare aggregate results, and open individual trace-linked explanations.
Sample live traffic, collect user or human feedback, and alert when a score distribution crosses its threshold.
Examples and expected outcomes that match the promise you ship.
What you get
Replace gut-feel spot checks with a record you can rerun.
Versioned datasets
01Collect inputs, expected outputs, metadata, and selected production traces into controlled test sets.
Typed evaluators
02Use code checks or model judges and persist numeric, boolean, categorical, or text results with explanations.
Offline and online runs
03Run suites over datasets and attach live scores to traces or spans using the same evaluator versions.
Review and monitors
04Queue failed evidence for annotation and alert on failure rate, averages, counts, or categorical rates over a defined window.
Instrument it
Score a trace in one line.
Attach a typed score or end-user feedback to any run. The same score shows up in suites, monitors and the review queue.
$ npm install tracify-sdk
import { TracifyClient } from "tracify-sdk";
const client = new TracifyClient({ apiKey: process.env.TRACIFY_API_KEY, projectId: "your-project-id" });
client.score(runId, "answer_quality", 0.92, { dataType: "numeric" });
client.feedback(runId, true, { kind: "thumb", comment: "Helpful answer" });
Who it is for
A change is only better if you can show it.
Teams validating prompt, model, retrieval, policy, and workflow changes before and after release.
Questions people ask.
What is AI agent evaluation?
AI agent evaluation is the practice of running versioned datasets through deterministic or model-based evaluators to score agent behavior repeatably, instead of relying on subjective spot checks.
What is the difference between offline and online evaluation?
Offline evaluation runs a suite of evaluators over a fixed dataset before release. Online evaluation attaches live scores to production traces or spans using the same evaluator versions.
What is a code evaluator versus a model-based evaluator?
A code evaluator applies a deterministic check, such as a schema or string match. A model-based evaluator uses another model as a judge and returns a numeric, boolean, categorical, or text result with an explanation.
How do I build a dataset for agent evaluation?
Collect representative inputs, expected outputs, metadata, and selected production traces into a versioned dataset that reflects the product behavior or risk being tested.
Can I monitor evaluation scores in production?
Yes. Sampled live traffic and human or user feedback can be scored continuously, with alerts triggered when a score distribution crosses a defined threshold.
See your own runs like this.