Use case / RAG apps
See which chunk the answer came from.
A RAG app can sound sure while using the wrong document. Trace the query, the chunks and the answer together, and a bad source is a visible step.
- Chunks
- Per answer
- Ranking
- Visible
- Grounding
- Checked
The failure to catch
A fluent answer built on the wrong document.
Trace retrieval, ranking and generation so a wrong answer can be traced to the chunk, query or prompt that caused it.
Retrieval record
Which chunk the answer came from.
The run starts with the question so retrieval can be judged against it.
See the chunks the model was handed, with their scores.
A wrong chunk is a flagged step, not a mystery in the final text.
The answer is stored next to the context that produced it.
Show or log the source so a reviewer can check it in seconds.
- 01ok
Question embedded
“What is the refund window for annual plans?”
- 02ok
Top 8 chunks fetched
Scores from 0.82 down to 0.61.
- 03off-topic
Off-topic chunk ranked
A 2022 pricing page ranks above the policy.
- 04ok
Answer generated
The reply quotes the older page.
- 05ok
Source shown
The chunk used is stored next to the answer.
What Tracify watches.
Retrieved chunks
Recorded on every run, so it can be searched, compared and alerted on.
Rank and score
Recorded on every run, so it can be searched, compared and alerted on.
Context size
Recorded on every run, so it can be searched, compared and alerted on.
Answer grounding
Recorded on every run, so it can be searched, compared and alerted on.
Instrument it
Make retrieval a step you can read.
Record the retrieval as its own span and the trace shows which chunks the model was given, next to the answer it wrote.
$ npm install tracify-sdk
import { span, traceAgent } from "tracify-sdk";
export const answer = traceAgent(async (question: string) => {
const chunks = await span("retrieve", () => vectorSearch(question, 8), { type: "retrieval", input: question });
return span("generate", () => writeAnswer(question, chunks), { type: "llm" });
}, { environment: "production" });