Voice AI Agent Observability: Tracing Latency, Turn-Taking, and Failures in Real-Time Voice Agents

A support line replaces its IVR menu with a voice agent. The demo sounds great: fast, natural, barely a pause between the caller's question and the agent's answer. Three weeks into production, a caller reports that the agent "wouldn't shut up" — it kept talking over her while she tried to correct a wrong account number. The team pulls the trace. The run completed. The transcript is accurate. The LLM call succeeded. Every span is green.

Nothing in that trace explains what the caller experienced, because nothing in that trace recorded when the user started speaking relative to when the agent's audio actually stopped. That gap — between what a text-agent trace captures and what a spoken conversation needs — is the subject of this guide.

At a glance

This guide is for engineers building or operating a real-time voice agent — a phone-based support line, a voice assistant, or any system where a person talks and the agent talks back with an audible pause in between. It teaches a trace model for one voice turn — not one LLM call — and the metrics that expose failures a text-agent dashboard cannot see: dead air, talk-over, mispronunciation, and silent misrecognition. A voice agent can pass every text-agent health check and still be unusable on a phone call, because the pipeline has stages, timing relationships, and interruption behavior that a request/response trace was never built to hold.

The guide does not cover choosing a specific ASR or TTS vendor, telephony/SIP transport engineering, or codec-level audio processing — those are implementation details that sit underneath the trace model, not part of it. For the general trace and span vocabulary this guide extends, see the AI agent trace schema; for propagating that vocabulary across services, see LLM tracing explained.

The CRISP turn model

Name the five stages every voice turn passes through and you have a framework the rest of the article reuses: Capture → Recognize → Interpret → Synthesize → Play, or CRISP. Barge-in is not a sixth stage — it is a condition that can interrupt any of the other five, which is why it needs its own cross-cutting trace fields rather than a single span.

Decision rule: If a trace cannot answer "which of the five CRISP stages consumed the time, and was barge-in active during any of them," it is not yet a voice trace — it is a text trace with an audio label.

Why text-agent tracing misses voice failures

A text or chat agent trace is built around a request-response unit: input arrives, the agent reasons, output returns. The AI agent trace schema models this well — a run contains traces, traces contain spans, spans contain events. That model still applies to voice, but a voice turn adds two properties a chat turn does not have: it is streaming at every stage, and it is interruptible at every stage. A chat message either exists or it doesn't; a spoken utterance can be half-heard, half-spoken, or spoken over.

Three consequences follow. First, latency is not one number — the caller does not experience "the LLM call took 700ms," they experience the accumulated silence from the moment they stopped talking to the moment sound starts coming back, which is the sum of every CRISP stage plus network delivery. LLM latency optimization covers reducing the model's own contribution, but a voice agent needs the full chain measured, not just the model span.

Second, a "successful" span can still be a conversational failure. The ASR can return a grammatically valid, confidently scored transcript that is simply wrong — a misheard account number, a misheard "not" that flips a yes/no answer. The TTS can synthesize the correct text and mispronounce the one word that mattered, such as a drug name or a dollar amount. None of these register as errors in a conventional trace because nothing threw an exception.

Third, the agent's own audio output is a live, cancelable process, not a returned value. A chat agent's response is complete the moment it's generated. A voice agent's response is still being played for as long as several seconds after generation, and during that window the user can interrupt it. If a trace only records when synthesis finished, not when playback actually stopped, it cannot explain a barge-in failure at all.

Field note: Teams that adapt an existing text-agent trace schema to voice by adding a single duration_ms field to the TTS span consistently miss barge-in failures, because that field describes synthesis time, not the playback interval during which the user could have interrupted and the system failed to notice.

Stage 1: Capture — audio in, VAD, and endpointing

Voice activity detection sets the clock for everything downstream

Capture is where the pipeline decides the user has finished a thought and it is the agent's turn. A voice-activity detector (VAD) watches the incoming audio stream for speech versus silence, and an endpointing policy decides how much trailing silence counts as "done talking" versus "still thinking." This single threshold — commonly 300–800ms of silence — sets the floor for how quickly the agent can respond and the ceiling for how often it cuts someone off mid-sentence.

Trace endpoint_detected_at relative to the last frame of detected speech, and record the endpointing policy's configured silence threshold as span metadata. Without that pairing, a support engineer investigating "the agent interrupted me" cannot tell whether the threshold was too aggressive for this call or whether the caller genuinely paused for a long time.

What to trace at capture

Capture should emit, at minimum: stream start time, first-speech-detected time, endpoint-detected time, the configured silence threshold, and an audio-quality signal such as estimated signal-to-noise ratio. That last field matters more than it looks — a noisy line (car, warehouse, crowded room) degrades every downstream stage, and without a capture-stage quality signal, a high ASR word-error-rate later in the turn looks like an ASR problem when it is actually a capture-stage problem.

Stage 2: Recognize — streaming ASR and word-error rate

Partial transcripts are not the same signal as final transcripts

Modern streaming ASR emits a sequence of partial transcripts that get revised as more audio arrives, followed by one final transcript once the endpoint fires. A common instrumentation mistake is logging only the final transcript. Partial transcripts matter for two reasons: some turn-taking logic uses partial transcript stability to predict an interruption is safe to act on, and a large gap between an early partial and the eventual final transcript is itself a signal that the ASR model is struggling with this audio — background noise, cross-talk, or an unfamiliar accent.

Trace every partial transcript as an event on the recognize span with its confidence score and the wall-clock time it arrived, then trace the final transcript separately with asr_final_at. This lets a later investigation replay exactly what the system "knew" at each moment, not just what it eventually decided.

Measuring word-error-rate on real, noisy audio

Word-error-rate (WER) — the edit distance between the recognized transcript and a verified reference, divided by reference length — is the standard ASR quality metric, but it is only useful in production if it is measured on audio that resembles production audio: phone-line compression, background noise, regional accents, code-switching, and interruption artifacts. A WER benchmark measured on clean, single-speaker studio audio tells you almost nothing about a customer calling from a car with the window down.

Build a small held-out sample of real production audio, hand-correct the transcripts, and track WER against that sample by cohort — by language, by detected accent cluster, by reported audio-quality bucket. The comparison method in the practical AI agent evaluation guide — matching baseline and candidate on the same task set — applies directly here: compare ASR model versions on the same held-out audio set rather than trusting an aggregate WER number from a vendor's own benchmark.

Field note: A WER of 6% sounds low until you notice it is concentrated entirely in one cohort — callers with a specific regional accent — because the training data underrepresented that accent. An aggregate WER hides exactly the failure a segmented WER reveals.

Stage 3: Interpret — turn-taking decisions, not just answers

The LLM's job in a voice agent is bigger than generating a response

In a text agent, the model's job is to produce a good answer. In a voice agent, the model (or a lightweight turn-taking policy layered around it) has a second job: deciding whether the user has actually finished speaking, whether to speak now, wait, or ask a clarifying question, and whether an in-progress user utterance is a real interruption or a filler sound ("um," a cough, a side conversation picked up by the microphone). Getting the content right and getting the timing right are separate problems, and a trace needs to capture both.

Record llm_first_token_at — time-to-first-token — separately from total generation time, because the first stage of TTS can often begin as soon as the first sentence of the response streams out, and time-to-first-token is what determines how much of the overall turn latency budget the interpret stage consumes. Also record the explicit turn_decision the system made: respond now, wait for more input, or yield the floor because the user appears to still be talking. Silent failures cluster here — the model decides to respond while the endpoint detector was actually wrong about the user finishing, producing exactly the "talked over the interruption" failure from the opening scenario.

End-of-turn detection latency

End-of-turn detection latency is the time between the true end of the user's utterance and the moment the system commits to a turn decision. It is distinct from ASR finalization latency, because a system can finalize a transcript quickly and still delay its turn decision waiting for a confidence threshold or a secondary confirmation pass. Measure it directly rather than assuming it equals ASR latency; the two drift apart under load, and that drift is often the first sign of a capacity problem before p95 latency alerts fire.

Stage 4: Synthesize — TTS streaming and the silent mispronunciation

Time-to-first-audio-byte is the metric that determines whether the agent feels alive

Time-to-first-audio-byte (TTFAB) measures from the moment the interpret stage produces enough text to synthesize through to the first audio byte the playback stage receives. Streaming TTS can start synthesizing and delivering audio before the full response text exists, which is the difference between a caller hearing the agent start speaking in 400ms versus waiting the full 1,500ms for the complete response to generate and then synthesize. If your TTS integration waits for the complete LLM response before starting synthesis, TTFAB and total turn latency become the same number, and that is usually the single largest latency win available in an existing pipeline.

Silent failures: audio that is valid but conversationally wrong

The most dangerous TTS failures never raise an error. The synthesis succeeds, the audio plays, the caller hears something — and that something is wrong in a way no health check catches. A confirmation number gets mispronounced digit by digit incorrectly. A medication name gets an emphasis pattern that makes it unintelligible. A price gets read in the wrong currency's cadence. These are conversationally wrong, technically valid outputs, and they are the voice-agent equivalent of a text agent that returns a well-formatted but factually incorrect answer.

Trace the exact text string sent to the TTS engine alongside the synthesized audio's duration and any engine-reported pronunciation warnings or phoneme substitutions. For a defined list of critical terms — product names, medical terms, monetary amounts, confirmation codes — maintain a pronunciation regression set the same way you would maintain a text-agent regression suite, and replay it against each TTS model or voice version before rollout.

Stage 5: Play — playback and the barge-in race

The interruption race condition

Barge-in is fundamentally a race: the system must detect that the user has started speaking, decide that this is a genuine interruption rather than noise, and stop or duck the agent's outgoing audio — all while audio may already be buffered several hundred milliseconds ahead in a playback queue, a jitter buffer, or a telephony provider's own pipeline. The gap between "system detected the interruption" and "audio actually stopped" is the barge-in response time, and it is frequently invisible in a trace that only logs when synthesis completed, not when playback stopped.

Trace three separate timestamps for every turn where barge-in is possible: bargein_candidate_at (the moment speech was first detected in the user channel while the agent was still playing audio), bargein_confirmed_at (the moment the system decided this was a real interruption, not noise), and playback_stopped_at (the moment agent audio actually ceased). The interval between the first and the third is the caller's talk-over experience; the interval between the first and the second isolates detection latency from cancellation latency, and those two intervals point to different fixes.

Security note: Never treat a barge-in event as implicit reauthorization for a sensitive action already in progress — for example, do not let an interruption mid-confirmation silently commit a payment or account change that was still being read back. Barge-in should pause the flow, not fast-forward it.

A worked example: the agent talks over an interruption

Return to the opening scenario. The caller says the agent spoke over her while she tried to correct a wrong account number. The team pulls the CRISP trace instead of the plain LLM/TTS trace, and gets a very different picture.

The evidence separates two candidate root causes that looked identical from the outside. The 180ms detection-to-confirmation gap is a VAD sensitivity tuning problem — the barge-in classifier required more confirming audio than necessary before accepting the interruption as genuine. The additional 160ms cancellation delay is a different problem entirely: a downstream audio buffer was already queued with roughly one sentence of synthesized speech, and canceling playback meant draining that buffer rather than stopping instantly. Fixing the classifier threshold would not have fixed the buffering delay, and fixing the buffer size alone would not have fixed classifier sensitivity. A trace that only recorded "barge-in happened, response was delayed" would have sent the team chasing the wrong half of a 340ms problem.

The voice observability metrics that matter

A useful voice-agent dashboard tracks a small set of metrics that map directly to the CRISP stages, not a generic latency-and-error-rate pair borrowed from a web service dashboard.

Track p95, not only mean, for every latency metric in this table. A voice agent that is fast on average but has a fat tail of 4-second turns is worse for the caller than one that is uniformly 1.8 seconds, because the caller cannot tell in the moment whether a long pause means the agent is thinking or has failed silently.

Diagnosing common voice pipeline failure patterns

Silence storms and dead air

A "silence storm" is a run of turns where the endpoint detector repeatedly waits the full silence threshold without a good reason — often because a noise floor changed (a door opened, traffic noise increased) and the VAD's sensitivity no longer matches the room. The symptom is dead air the caller experiences as the agent "not responding." Trace the capture-stage signal-to-noise estimate alongside endpoint latency; a spike in silence-to-endpoint duration that correlates with a drop in estimated SNR points at the environment, not the model.

Echo and self-interruption

If the agent's own outgoing audio leaks into the microphone input — common on speakerphone or with imperfect echo cancellation — the barge-in detector can classify the agent's own voice as a user interruption, causing the agent to stop mid-sentence for no external reason. Trace whether a detected interruption correlates in time and frequency with the agent's own currently-playing audio; a high correlation points to echo, not a genuine barge-in, and should route to an acoustic-echo-cancellation investigation rather than a turn-taking policy change.

Language and accent drift in ASR

Cohort-level WER regressions often show up gradually as a caller population shifts — a new market opens, a marketing campaign shifts the caller mix, a holiday period brings in less-frequent callers with different speech patterns. Without cohort-segmented tracking, this drift is invisible until aggregate quality metrics move enough to trigger an alert, by which point the affected cohort has had a degraded experience for weeks.

Instrumenting a voice-turn trace schema

Extending a standard trace/span model for voice means adding stage-specific timestamps and turn-level fields without abandoning the run/trace/span/event shape that already works for text agents. The schema below is illustrative — adapt field names to your existing trace schema rather than adopting this one wholesale.

// Illustrative: extend an existing trace/span schema with voice-turn fields.
// Verify field names against your production schema before using this shape.
interface VoiceTurnSpan {
  turnId: string;
  callId: string;

  // Stage 1 — Capture
  speechStartedAt: number;
  endpointDetectedAt: number;
  silenceThresholdMs: number;
  estimatedSnrDb?: number;

  // Stage 2 — Recognize
  asrPartials: Array<{ text: string; confidence: number; atMs: number }>;
  asrFinalAt: number;
  asrFinalText: string;
  asrWerEstimate?: number;

  // Stage 3 — Interpret
  llmFirstTokenAt: number;
  llmResponseCompleteAt: number;
  turnDecision: "respond" | "wait" | "yield";

  // Stage 4 — Synthesize
  ttsFirstByteAt: number;
  ttsSourceText: string;
  pronunciationWarnings?: string[];

  // Stage 5 — Play, including barge-in (cross-cutting, optional)
  playbackStartedAt: number;
  playbackStoppedAt?: number;
  bargeIn?: {
    candidateAt: number;
    confirmedAt: number;
    playbackStoppedAt: number;
  };
}

What belongs on a span versus an event

Keep per-stage timestamps and summary fields — asrFinalAt, ttsFirstByteAt, turnDecision — as span attributes, since they describe one occurrence per turn. Keep the sequence of ASR partial transcripts and any barge-in candidate detections as ordered events within the span, since there can be zero, one, or several of each per turn and an investigator needs to replay their order. This mirrors the general run/span/event separation in the AI agent trace schema: spans describe a bounded unit of work, events describe things that happened during it.

Protect the audio privacy boundary

What must never be logged

Voice agents create a category of sensitive data a text agent does not: raw audio recordings of a caller's actual voice, which can contain account numbers, medical information, or other regulated content spoken aloud, plus biometric characteristics of the voice itself. Treat full-call audio recordings the way you would treat unencrypted credentials — access-controlled, encrypted at rest, and never routed into a general-purpose logging or trace-search system where any engineer with dashboard access can retrieve them. Trace fields should carry timestamps, durations, confidence scores, and stage outcomes; they should not carry raw audio bytes or unredacted transcripts by default.

Security note: If a debugging workflow genuinely requires audio playback, gate it behind a separate, audited access path with a narrower retention window than your general trace store — not the same trace UI an on-call engineer opens for a routine latency investigation.

Retention boundaries for recordings

Define a retention window for raw audio that is shorter than your trace-metadata retention window, and make the difference explicit to the team: trace metadata (timestamps, WER, turn decisions) can reasonably persist as long as other observability data, but raw audio and full transcripts carry materially higher risk per day retained. Where regulation applies to the call type — healthcare, financial account verification — retention and consent requirements should be set by your compliance function, not inferred from what the observability tooling happens to store by default.

Trade-offs: streaming complexity versus observability overhead

Every additional trace field on the hot path of a real-time voice pipeline has a cost: serialization time, network payload size, and — if instrumentation itself blocks the audio pipeline even briefly — a direct latency regression in the metric you are trying to protect. This is a sharper version of a trade-off every observability system faces, but voice makes it acute because the pipeline is latency-sensitive at the level of individual milliseconds, not the hundreds-of-milliseconds tolerance a typical API endpoint has.

The practical resolution is asynchronous, buffered instrumentation: write trace events to a local buffer and flush them off the hot path, never synchronously call out to a tracing backend before releasing an audio frame or a token. Sample full partial-transcript event streams for a subset of calls rather than every call, while keeping stage-summary timestamps (endpoint, ASR final, first token, TTFAB, barge-in) on every call — those summary fields are cheap and carry most of the diagnostic value. LLM latency optimization covers the same buffered-instrumentation principle for text agents; the tolerance for getting it wrong is simply much smaller in a voice pipeline where a 50ms instrumentation stall is audible as a stutter.

Decision rule: Sample expensive, high-volume trace data (raw partial transcripts, full audio snapshots); never sample the cheap, high-value summary timestamps that make a turn diagnosable. If a budget cut has to happen, cut volume before cutting coverage.

Ship it: a practical instrumentation checklist

• [ ] Every turn records the five CRISP-stage summary timestamps: endpoint detected, ASR final, LLM first token, TTS first byte, playback stopped.
• [ ] Barge-in turns additionally record candidate, confirmed, and playback-stopped timestamps as three distinct fields, not one collapsed "interrupted" flag.
• [ ] Word-error-rate is tracked by cohort (language, accent cluster, reported audio quality), not only as a single aggregate number.
• [ ] TTS source text and any pronunciation warnings are traced alongside audio duration, so a mispronunciation can be investigated without needing to re-listen to the call.
• [ ] Raw audio and unredacted transcripts sit behind a narrower-retention, access-controlled path separate from general trace search.
• [ ] Trace writes are buffered and flushed off the real-time audio path, never synchronously blocking a frame or token.
• [ ] p95 turn latency, not just mean, is on the primary dashboard, with a per-stage breakdown available from the same view.
• [ ] At least one barge-in scenario and one noisy-audio scenario are covered by a repeatable test call before each release.

A voice trace earns its keep the first time it turns "the agent felt off" into a specific stage and a specific millisecond gap — that is the bar to hold it to, not whether every field in the schema above is populated on day one.

Carry this checklist into your release process the same way LLM tracing explained treats propagation correctness: as a gate before rollout, not a retrospective exercise after the first complaint arrives. Start with the five summary timestamps, add barge-in and cohort-level ASR tracking next, and expand only when a real investigation needs a field the current schema does not have.

Frequently asked questions