When the Agent Breaks at 3am: An Incident Response Guide for AI Systems

At a glance

This guide is for the engineer holding the pager for an LLM agent in production. It helps you run an incident from the first alert to a written postmortem so you can leave with a severity ladder, a triage order, and containment moves you can adopt this week.

Agent incidents feel different from the outages most on-call rotations were designed for. A database is either up or down. An agent can be up, responsive, well within its latency budget, returning HTTP 200 on every request — and quietly wrong for four hours. The service health dashboard stays green the entire time. That mismatch is the whole problem, and most of this article is about closing it.

The boundary is worth stating up front. This is not a guide to evaluating model quality before release, choosing a model, or designing a retry policy from scratch. It assumes you already ship an agent, already collect traces, and now need to operate the thing when it misbehaves at an inconvenient hour.

The operating model

Everything below hangs off a six-stage loop. It is deliberately close to the incident loop your infrastructure team already runs, because the last thing an on-call engineer needs at 3am is a second, unfamiliar process. What changes is the content of each stage, not its order.

The two stages teams skip most often are Establish and Evaluate. Skipping Establish turns the incident into a debate about whether it was an incident. Skipping Evaluate ships a fix validated on exactly one prompt, by one engineer, at 4am.

What actually counts as an agent incident

Start by fixing the definition, because a vague one produces both alert fatigue and missed outages.

An agent incident is any production behavior where the agent's output or actions diverge from its intended contract in a way a user would care about. That covers three families of failure, and only one of them looks like a traditional outage.

Availability failures are the familiar kind. The model provider returns 503s, the vector store times out, the ingest pipeline backs up. Your existing monitoring probably catches these already, because they produce error rates and latency spikes that look like every other service failure.

Behavioral failures are the hard ones. The agent returns a fluent, confident, well-formatted answer that is wrong. It calls the refund tool when it should have called the lookup tool. It loops on a task it cannot complete, spending real money on each iteration. Every request succeeds. Every span returns a 200. Nothing in a standard service dashboard moves.

Boundary failures are the expensive ones. The agent takes an action it should not have been able to take: emails a customer, writes to a record, triggers a downstream workflow, leaks context into a tool argument. These are rare and they dominate your risk profile, because they are the only category where the damage continues after you stop the agent.

A useful test: if the agent stopped right now, would the harm stop? For availability failures, yes. For behavioral failures, mostly yes. For boundary failures, no — the email is already sent. That asymmetry is what should drive your severity ladder, and it is why an incident where the agent worked perfectly and called a tool it should not have is more serious than one where it fell over entirely.

Field note: The most dangerous agent incident is not the one that throws errors. It is the one where every span is green and the answers are wrong. If your paging strategy only fires on exceptions, you have instrumented for the least costly failure family.

Detection signals that are worth a page

Detection is where most agent on-call rotations go wrong, in one of two directions. Either nothing pages until a customer complains, or everything pages and the rotation learns to ignore the channel within a month.

The rule that resolves both: page on signals that correlate with user harm, and only on those. Everything else is a dashboard, a weekly review, or a ticket.

Four signals that earn a page

Task completion rate by intent. The single most useful agent signal, and the one most teams do not have. For each category of user request, what fraction of runs reach a terminal success state rather than an error, an abandonment, or a handoff to a human? A drop here means users are not getting what they came for, regardless of what the error rate says. Alert on a sustained drop against a rolling baseline, segmented by intent — a global rate hides a total failure on one intent behind healthy volume on nine others.

Loop depth and step count distribution. Agents that cannot make progress do not usually crash. They iterate. A run that normally takes four steps taking nineteen is a strong signal that something in the tool layer or the prompt has changed underneath it. Watch the 95th percentile of steps per run, not the mean, because the mean stays comfortable while the tail eats the budget.

Cost per successful task. Not cost per run, and not total spend. Total spend rising can mean growth. Cost per successful task rising means you are paying more for the same outcome, which is the economic signature of a retry loop, a context bloat regression, or a silent fallback to a more expensive model. Teams working through reducing LLM costs without hurting quality usually end up with this metric anyway; it doubles as an incident signal.

Tool error rate, segmented by tool. Aggregate tool error rate is nearly useless — one flaky tool called rarely will never move it. Per-tool rates catch the thing that actually happens: one integration's schema changed, the agent now gets a validation error every time, and it retries into the same wall.

Signals that should never page

The distinction is not about how bad the thing is. It is about whether waking a human at 3am changes the outcome. If the answer is no, it is not a page.

Set thresholds against a baseline, not a constant

Absolute thresholds age badly for agents, because normal shifts with every prompt change, model version, and seasonal change in what users ask. A completion rate of 82% might be excellent for one intent and alarming for another.

Use a rolling baseline — the same metric over the trailing seven or fourteen days, segmented the same way — and alert on the deviation. Then add a floor for the cases where slow drift matters: if completion rate on any intent falls below an absolute contractual minimum, page regardless of what the baseline says. Drift and cliffs are different failures and need different detectors.

A severity ladder built on blast radius

Severity is the stage teams most want to skip and most regret skipping. Without it, every incident is argued at the level of vibes, and the loudest participant sets the response.

For agents, severity should be a function of two things: how reversible the agent's action was, and how many users are affected. Not error count. Not how alarming the logs look.

Promote a level when the action is irreversible. Demote only when a fallback is proven to carry the user through.

Reversibility deserves the top weight because it determines whether your containment options still matter. If the agent has only produced text on a screen, stopping it resolves the exposure. If it has written to systems of record or communicated externally, stopping it is merely the first of several steps, and the remaining steps involve people rather than deploys.

Writing the severity down changes the incident

The act of writing severity into the incident channel does three things at once. It sets the expected response time, so nobody has to ask. It names an owner, so decisions have a single point of accountability. And it creates the record you will need later, when someone asks how long the agent was misbehaving before anyone noticed.

Do it in one line, within the first few minutes:

SEV2 — support agent returning wrong refund policy on EU accounts since ~02:10 UTC. Owner: @oncall. Contained by disabling refund tool at 03:12.

That line is worth more than an hour of unstructured discussion. It is also the seed of the postmortem.

Contain before you diagnose

The strongest single habit an agent on-call rotation can build: containment precedes diagnosis, always. Curiosity is the enemy at 3am. The trace is fascinating and it will still be there in twenty minutes.

Containment for agents has a specific advantage over most infrastructure incidents. Because agents are composed of separable capabilities, you usually have options between "everything is fine" and "the product is down."

The containment ladder, cheapest first

Cap the loop. If the failure signature is iteration without progress, the smallest effective move is lowering the maximum step count or tightening the budget ceiling. Runs terminate instead of spiraling, users get a clear failure instead of a hang, and spend stops climbing. This is often a config change rather than a deploy.

Disable one tool. If a single tool is the failing surface, removing it from the agent's available set is far less invasive than taking the agent offline. A well-designed agent degrades: it tells the user it cannot complete that specific action rather than failing everything. If yours does not degrade — if removing one tool causes cascading confusion — that is a finding for the postmortem.

Force the safe path. Route the affected intent to a deterministic fallback: a scripted flow, a template response, a human queue. Slower and less capable, but bounded and predictable. For a support agent, "I'm connecting you with someone who can help" is a much better outcome than a confident wrong answer about refunds.

Roll back the last change. Prompt, model version, tool schema, retrieval index, or configuration. Cheap when changes are versioned, painful when they are not.

Take the agent offline. The last resort, and genuinely correct for SEV1 boundary failures where each additional run risks another irreversible action.

Decision rule: If the agent can still take an irreversible action and you do not yet know why it is misbehaving, revoke the capability first and investigate second. Capability revocation is reversible. The action the agent is about to take is not.

Triage in trace order

Once harm has stopped, you can be curious. The most efficient way to spend that curiosity is to read one failing trace in execution order and find the first place reality diverged from the intended path.

This is the discipline that separates a twenty-minute diagnosis from a three-hour one. Under pressure the instinct is to look at the last thing that happened — the error, the timeout, the exceeded budget. But the last event in an agent run is usually a symptom several steps downstream of the actual fault. The retry loop in this article's opening image is not a retry problem; retries are what a well-behaved agent does when a tool keeps failing. The fault is whatever made the tool fail the first time.

Read in this order

One: the user input and the resolved system prompt. Not the prompt template — the fully rendered prompt for this specific run, with variables interpolated and retrieved context inserted. A surprising share of "the model got worse" incidents are a template variable resolving to an empty string.

Two: the retrieval or context assembly step. What did the agent actually have to work with? Wrong documents, stale documents, or truncated context produce confident wrong answers that look exactly like model failures. Check what was retrieved and what survived the token budget.

Three: the first tool call. Compare intended tool against called tool, and inspect the arguments. Tool selection errors and argument construction errors are different faults with different fixes — one is usually a prompt or tool-description problem, the other a schema or validation problem.

Four: the first divergence. The first step where the run left the intended path. Everything after this point is consequence. Mark it, and resist the pull to keep scrolling.

This ordering is the operational core of debugging AI agents in production, applied under time pressure. During an incident you are not doing root-cause analysis in full — you are finding the smallest change that restores correct behavior.

Compare against a known-good trace

The fastest diagnostic available to you is a diff. Pull a successful run of the same intent from before the incident window and put it beside the failing one. Structural differences — an extra step, a missing retrieval, a different tool, a shorter context — usually point at the fault within a minute or two.

This only works if you retain enough trace history to have a known-good example from before the change. Retention is not a cost line item; it is the thing that makes 3am tractable.

# Illustrative: find the first divergent step between a known-good run
# and a failing one. Works on any span list ordered by start time.

def first_divergence(good_spans, bad_spans):
    """Return (index, reason) for the first meaningful difference."""
    for i in range(max(len(good_spans), len(bad_spans))):
        if i >= len(bad_spans):
            return i, "failing run ended early"
        if i >= len(good_spans):
            return i, f"extra step: {bad_spans[i]['name']}"

        good, bad = good_spans[i], bad_spans[i]

        if good["name"] != bad["name"]:
            return i, f"different step: expected {good['name']}, got {bad['name']}"
        if bad.get("status") == "error" and good.get("status") != "error":
            return i, f"{bad['name']} failed: {bad.get('error_type')}"
        if good["name"] == "retrieval" and not bad.get("attributes", {}).get("doc_count"):
            return i, "retrieval returned no documents"

    return None, "no structural divergence - inspect payloads, not shape"

The fallback in that last line matters. When two runs have identical structure and different outcomes, the fault is in content — prompt rendering, retrieved text, model nondeterminism — not in control flow. That is a different investigation, and knowing which one you are in saves real time.

Choosing what to roll back

Rollback is where agent incidents diverge most sharply from service incidents. In a normal service you roll back a deploy. An agent has at least five independently mutable surfaces, and only some of them are in your deploy pipeline.

The uncomfortable row is model version. If you call a floating alias rather than a pinned version, agent behavior can change with no deploy, no commit, and no entry in your change log — which means the first triage question ("what changed?") returns nothing and sends you looking in the wrong place. Pin versions and upgrade deliberately; it converts an entire class of unattributable incidents into scheduled work.

The correlate for prompts is version control with a deployment history you can query during an incident. If your on-call engineer cannot answer "what was the prompt at 02:10?" in under a minute, prompt changes will remain the hardest incidents you run. The practices in prompt versioning in production exist substantially for this moment.

When rollback is the wrong move

Rolling back is not free and it is not always correct. Three cases where it makes things worse:

The change was correct and exposed an existing bug. A prompt improvement that made the agent more willing to use a tool did not break the tool — it revealed that the tool was already broken for a subset of inputs. Rolling back hides the fault until the next attempt.

The failing surface is not yours. If an upstream tool's schema changed, rolling back your prompt does nothing. Containment — disabling that tool, forcing the safe path — is the real move, and the fix belongs to a conversation with the owning team.

You cannot identify a change. If nothing changed on any surface, the fault is probably a shift in input distribution: a new customer segment, a marketing campaign, a seasonal pattern, or an adversarial user probing the agent. Rolling back a change you did not make wastes the window in which you could be reading traces.

Field note: "What changed?" is the right first question and a trap when the honest answer is "nothing we control." Give the question a two-minute budget. If no change surfaces, switch to characterizing what is different about the inputs, not the system.

The security boundary during an incident

Incidents are exactly when security discipline degrades, because the fastest path to understanding is usually the one that violates it. Two rules are worth holding even at 3am.

Traces contain user data. Prompts and tool arguments carry whatever the user typed, plus whatever retrieval pulled — which can include other customers' records. Pasting a raw trace into a shared incident channel, a ticket, or a third-party tool is a data exposure that outlives the incident. Redact before sharing, and prefer linking to the trace in a system with access controls over copying its contents.

Credentials do not get widened to speed up debugging. The instinct to grant the agent broader permissions "temporarily" so you can reproduce the failure is how a SEV2 becomes a SEV1. Reproduce in a sandbox with synthetic data. If the failure only reproduces with production permissions, that itself is a finding.

There is also a category of incident where the agent's behavior is the attack rather than a bug. If tool arguments contain instructions that did not come from your prompt, or retrieved documents contain text aimed at the model rather than the reader, you are looking at a prompt injection attempt, not a regression. The containment move is different: revoke the capability, preserve the evidence, and treat it as a security incident with the people that implies — not a rollback.

Postmortems for nondeterministic systems

The standard postmortem template assumes a deterministic system: a specific change caused a specific failure, and preventing that change prevents recurrence. Agents break that assumption in a way that produces bad postmortems if you do not adjust.

Three adjustments matter.

Replace "root cause" with "contributing conditions." An agent failure is usually a conjunction: an ambiguous prompt, plus an unusual input, plus a tool that returns an unhelpful error, plus a retry policy that masks it. Removing any one might have prevented this instance. Listing one as the cause guarantees the next variant surprises you.

Add a detection gap section. For agent incidents, the most valuable question is rarely "why did it break?" It is "why did it take ninety minutes to notice?" Write down the actual time from first bad response to first human awareness, and what signal would have shortened it. Over a few incidents this section, not the fix list, is what improves on-call life.

Convert the incident into a test case. Every agent incident produces something more durable than a fix: a real, failing input. Add it to the evaluation set with the expected behavior. This is the mechanism that stops the same failure returning under a different phrasing three months later, and it is the direct link between incident response and regression testing for agents.

A worked example

An internal support agent starts telling EU customers the wrong refund window. No errors, no latency change, no alert. A support lead notices at 09:40 and asks in Slack.

The weak column produces one action item that nobody owns. The useful column produces four: pin and log index changes, segment completion rate by region, add a relevance floor with an explicit "I don't know" path, and add the EU refund question to the evaluation set.

What changes the recommendation

This whole model assumes traces exist, are searchable, and are retained past the incident window. If your agent is instrumented with logs only, the triage order above is unusable — you cannot diff runs you cannot reconstruct. Fix instrumentation before adopting the process; the process is downstream of the data. Teams starting from that position should treat agent monitoring as the prerequisite, not this guide.

The model also assumes the agent has meaningful capabilities. A read-only agent that only produces text has no boundary failures, which collapses the severity ladder to two levels and makes "take it offline" a reasonable first move rather than a last one. Match the ceremony to the blast radius.

The on-call checklist

Keep this where the pager takes you, not in a wiki nobody opens under pressure.

When paged:

• [ ] Can the agent still take irreversible actions? If yes, revoke the capability before anything else.
• [ ] Write one line: severity, symptom, start time, owner.
• [ ] Contain with the cheapest sufficient move: cap the loop, disable a tool, force the safe path, roll back, or stop.
• [ ] Confirm harm stopped climbing before starting to diagnose.

When diagnosing:

• [ ] Pull one failing trace and one known-good trace of the same intent.
• [ ] Read in order: resolved prompt, retrieval, first tool call, first divergence.
• [ ] Check all five mutable surfaces for changes: prompt, model version, tool schema, index, config.
• [ ] If no change surfaced in two minutes, characterize the inputs instead.

Before closing:

• [ ] The fix passed on the replayed failing traces, not just on a fresh manual attempt.
• [ ] The failing input is in the evaluation set with its expected behavior.
• [ ] The detection gap is written down with a time-to-detect number.
• [ ] Every temporary containment measure is either reverted or converted into a permanent control with an owner.

That last item is the one that quietly accumulates. Disabled tools and lowered loop caps are supposed to be temporary, and six months later nobody remembers why the agent cannot issue refunds. Put an expiry on every containment change at the moment you make it.

The next action worth taking is smaller than it sounds: pick your highest-volume intent and add one segmented completion-rate alert against a rolling baseline. Most teams discover within a week that their agent has been failing a specific intent for longer than they would have guessed — and that discovery, made on a Tuesday afternoon, is much cheaper than making it at 3am.

Frequently asked questions