AI Agent Security: A Practical Production Checklist

At a glance

This diagram is a topic-specific decision aid: connect the article’s context to evidence before taking the next action.

This guide is for engineers shipping tool-using AI agents. It helps you turn vague security concerns into enforceable boundaries. It covers runtime controls and evidence, not a complete compliance program or a substitute for threat modeling.

The security framework

Use Input → Policy → Tool → Evidence:

1. Input: classify user, retrieved, and tool-returned content as untrusted by default.
2. Policy: decide what the agent may propose, read, or execute.
3. Tool: validate arguments and scope credentials to the smallest useful permission.
4. Evidence: record the decision, tool call, result, and approval path.

Start with the consequence

Security controls should be strongest where an incorrect action is hardest to reverse. Reading a public document, sending an email, changing an account, and issuing a refund should not share one permission level. The prompt injection defense guide explains why instructions found inside data cannot be trusted merely because a model saw them.

Make tools narrow and explicit

Prefer several small tools with typed arguments over one general-purpose executor. Validate identifiers, ranges, destinations, and authorization in application code. The model can suggest an action; it should not be the final authority that grants itself access.

type RefundRequest = { orderId: string; amountCents: number };

function authorizeRefund(request: RefundRequest, actorId: string) {
  if (request.amountCents <= 0 || request.amountCents > 5000) throw new Error("approval required");
  return { actorId, request };
}

The example is illustrative: adapt the policy to your authorization system and test boundary cases. Trace the proposed call separately from the executed call so an investigator can tell what the model wanted from what the system allowed.

Protect data across the run

Redact secrets before telemetry leaves the process. Limit retention for prompts, retrieved documents, and tool payloads. Keep tenant, user, and authorization context attached to the parent run so a later review can detect cross-boundary access. Human-in-the-loop design helps define where a person must become part of the decision.

Release checklist

• List every tool, credential, data source, and irreversible action.
• Test prompt injection, confused-deputy, replay, and excessive-permission cases.
• Log proposed, approved, rejected, and executed actions separately.
• Redact secrets and define retention before enabling production traces.
• Give every high-consequence alert an owner and rollback path.

The release reviewer should sign off on the boundary, not merely the implementation. Record the reviewed workflow, tools, data classes, fixtures, observed results, unresolved risks, and rollback owner. That record is the durable proof that the team considered security before increasing autonomy, and it gives the next reviewer a concrete baseline instead of another informal promise. Keep the decision attached to the release and revisit it after real usage. Compare the evidence with the baseline before widening permissions and after incidents.

Recommended next reads

Continue with prompt injection defense, structured outputs for AI agents, and AI agent reliability.

Threat model the complete execution path

The security boundary is not only the prompt. An agent crosses several boundaries during one run: the user supplies an objective, retrieval supplies documents, a model proposes a plan, tools read or mutate systems, and an orchestration layer may retry or continue the work later. Each transition can change authority or introduce data that looks like an instruction.

Identify assets and principals

List the data and actions that would matter if exposed, changed, or delayed. Include customer records, credentials, internal instructions, retrieved documents, model responses, tool arguments, approval records, and trace payloads. Then list principals separately: end user, service account, reviewer, model provider, tool provider, and operator. A service account that can read every tenant is not made safe by a prompt that says “only read the current tenant.”

Write abuse cases as workflows

An abuse case should describe an entry point, a trust transition, an action, and an observable consequence. For example: a user uploads a document containing an instruction to export all invoices; retrieval places that text in context; the model proposes an export tool call; the application accepts the destination from the model; the result leaves the tenant boundary. This is more useful than a generic statement that the system is vulnerable to prompt injection because it tells the team where to add a control and what trace should prove it worked.

Separate data from authority

Retrieved text, web pages, emails, tool results, and user-provided files are data. They may contain useful facts, but they do not automatically gain authority to change the task or policy. Put trust labels next to those inputs and keep system policy outside the model-controlled context where possible. When an agent must reason over untrusted instructions, require the application to validate the proposed action against an independent policy before execution.

Design permissions as a matrix

A useful permission matrix has more than allow and deny. Include read, propose, prepare, approve, execute, and administer. The same tool may be safe for one tenant, record type, amount, or environment and unsafe for another. Encode those dimensions in the authorization decision rather than asking the model to remember them.

This matrix also makes incident review faster. If a trace shows an execution that the matrix forbids, the problem is an authorization defect. If the execution was allowed but the model chose the wrong record, the problem is selection or validation. Those are different fixes and should not be collapsed into “the AI made a mistake.”

Validate arguments before execution

Schema validation is necessary but not sufficient. A value can have the right type and still violate policy. Validate ownership, ranges, destination, state transitions, and idempotency. Reject unknown fields instead of silently ignoring them, because ignored fields can hide an attempt to widen the action.

function validateUpdate(input: { tenantId: string; recordId: string; status: string }, session: Session) {
  if (input.tenantId !== session.tenantId) throw new Error("tenant boundary");
  if (!['open', 'pending', 'closed'].includes(input.status)) throw new Error("invalid transition");
  return { ...input, actorId: session.userId };
}

Keep the authorization result and the normalized arguments in the trace. Do not record credentials or unrestricted payloads merely because they are useful during development. Redact before export, not only in the dashboard.

Test the controls adversarially

Security tests should cover direct attacks and ordinary edge cases. Include instructions hidden in retrieved content, a tool result that asks for a new tool call, a user who changes tenant context mid-run, replayed approval tokens, duplicate events, overlong arguments, malformed JSON, expired sessions, and a provider timeout after a side effect may have happened. For each case, assert both the user-visible behavior and the evidence left for the operator.

Use a failure disposition

Every rejected action should have a disposition such as blocked by policy, missing approval, invalid argument, expired authorization, or dependency failure. A generic error makes dashboards quiet but makes learning difficult. Store the policy version and the rule that fired so a future reviewer can determine whether the control worked as intended.

Operate security as a feedback loop

Review high-consequence traces on a regular cadence. Look for repeated rejected actions, near misses, unusually broad reads, permission changes, and workflows that repeatedly reach approval. Turn recurring mechanisms into regression fixtures. If a control is frequently bypassed through a manual process, that is evidence about the workflow design—not a reason to remove the control without understanding the risk.

Security metrics that support decisions

Useful measures include blocked high-risk actions, approval rate by action class, authorization failures by tenant, redaction failures, time to investigate, and confirmed policy violations. Avoid treating a lower block rate as automatically good. A control can appear successful because it stopped seeing the traffic it was meant to detect, or because teams routed work around it.

Final implementation sequence

Start with one consequential workflow. Draw its trust boundaries, define the permission matrix, validate one tool in application code, and capture a redacted trace from proposal through result. Add adversarial fixtures before adding more tools. Then stage the workflow with a narrow cohort and a reversible response. This sequence gives the team evidence about the control before the automation becomes difficult to unwind.

Build an asset and data inventory

Security work becomes concrete when the team can answer what the agent can see, change, and cause. Inventory prompt templates, retrieved sources, conversation state, tool credentials, generated files, trace payloads, caches, queues, and exports. For each asset, record its owner, sensitivity, tenant scope, retention rule, and permitted destinations. Include derived data: embeddings, summaries, classifications, and evaluator records can reveal sensitive facts even when the original document is not present.

Classify content by trust and sensitivity

Trust and sensitivity are different dimensions. A public web page may be low sensitivity but still untrusted as an instruction source. A private customer record may be trusted as data for one workflow while being forbidden from another. Carry both labels through retrieval and tool preparation. The model should not be asked to infer authorization from prose; the application should derive the allowed scope from the session and policy.

Review telemetry as a data product

Traces can contain prompts, retrieved passages, tool arguments, identifiers, and error messages. Decide which fields are necessary for an investigation, which can be hashed or summarized, and which must never leave the process. Apply redaction before export and test the redactor with secrets in unusual formats, nested JSON, Unicode, and error strings. Restrict trace access by tenant and role, and record access to high-sensitivity evidence.

Design the authorization path

The authorization path should be understandable without reading the model response. Resolve the current principal, tenant, resource, operation, policy version, and approval state in application code. Then validate the model’s proposed arguments against that decision. If the request changes during a long-running workflow, reauthorize at the point of execution rather than trusting a decision made at the beginning.

Avoid confused deputies

An agent service often has more authority than the user who initiated a run. That makes it a potential confused deputy: it may use its own credentials to perform an action the user could not perform directly. Pass the user and tenant context into every authorization check. Do not allow a tool to accept an arbitrary tenant or account identifier from model output. Derive it from authenticated context and reject mismatches.

Handle delegated authority explicitly

When a person approves an action, define the exact scope of that approval. Include the action class, normalized arguments, resource, expiration, policy version, and workflow ID. Recheck these values before execution. An approval for “send the prepared reply to this recipient” must not become authority to send a revised message to a different recipient after the state changes.

Defend against prompt injection by design

Prompt injection is not only a malicious sentence in a user prompt. It can arrive through a retrieved document, web page, email, tool result, image transcription, or memory record. Mark untrusted content as data, keep policy outside the data channel when possible, and require an independent application check before a consequential action. The prompt injection defense guide gives examples of the trust-boundary reasoning to apply.

Test indirect injection

Create fixtures where a source document tells the agent to ignore policy, reveal hidden context, call an unrelated tool, or export information. Include benign documents that contain quoted instructions so the system does not learn to reject all instructional text. The expected result should specify not only that the action is blocked, but also what safe explanation, trace label, and owner notification occur.

Protect the tool-result boundary

Tool output is often treated as more trustworthy than retrieved text because it came from an internal function. That assumption is unsafe when the tool reflects user-controlled content, external APIs, or a compromised dependency. Validate its schema, origin, and allowed fields. A tool result may provide evidence for a decision without gaining authority to redefine the task or request another privileged operation.

Secure memory and state

Long-term memory introduces persistence and often outlives the session that created it. Define what may be remembered, who may read it, how a user can correct or remove it, and how stale or conflicting memories are resolved. Store provenance and confidence with each memory item. Do not let a generated summary become an unquestioned fact simply because it was written by an earlier run.

Prevent cross-session leakage

Test a new session with similar names, shared devices, changed memberships, and a different tenant. Verify that retrieval filters memory by the current authorization context before ranking or prompt assembly. Clear caches when identity or tenant context changes. A response that does not quote the private memory can still leak it through a suggestion, ranking, or refusal explanation.

Secure files, browsers, and external systems

Agents that browse, download, upload, or manipulate files need controls beyond text validation. Restrict network destinations, file paths, MIME types, archive expansion, and maximum sizes. Scan or quarantine content before it becomes available to another tool. For browser agents, isolate profiles and downloads, prevent navigation to unapproved destinations, and require confirmation before external communication or account changes.

Make external actions reversible

Prefer drafts, previews, dry runs, and staged changes. When an action cannot be reversed, make the approval and evidence requirements stronger. For updates, record the before and after representation when safe. For messages, retain the approved content, recipient, and send result. For deletions, require an explicit policy and a recovery plan. “The tool returned success” is not enough to prove the user’s intended effect occurred.

Protect the model and provider boundary

Decide which data may be sent to each model provider, region, or hosting environment. Enforce the rule in routing code, not only documentation. Record provider and model identifiers without copying sensitive prompts into broad-access logs. If a provider returns a request ID or safety classification, retain it where it helps incident response, but do not treat provider metadata as a replacement for application authorization.

Manage provider failures safely

Fallbacks can change data residency, model behavior, tool selection, and cost. Define which workflows may use each fallback and what happens when no approved provider is available. Test a provider timeout after a request may have been accepted. Reconcile side effects before retrying and make the degraded state visible to the user and operator.

Establish security-focused release gates

A security gate should evaluate changes that alter authority, data flow, model routing, prompt instructions, tool schemas, memory, retrieval filters, or trace retention. Require a threat-model update for material boundary changes. Run the adversarial fixture set and compare blocked, allowed, escalated, and misclassified cases with the baseline. Review high-consequence disagreements manually before widening the cohort.

Define stop conditions

Write the condition that pauses a rollout before it begins. Examples include a confirmed cross-tenant result, an unredacted secret in exported telemetry, an unauthorized external action, a rise in high-risk approval bypasses, or an inability to reconstruct the affected run. The stop condition should name the owner, immediate containment action, communication path, and evidence needed to resume.

Run incident response from the trace

When a security event occurs, preserve the parent run and related child spans before changing retention or access. Identify the principal, data scope, policy version, proposed action, authorization decision, execution result, and downstream recipients. Contain first: revoke or narrow credentials, pause the workflow, disable a tool, or isolate a tenant as appropriate. Then reconstruct the timeline and distinguish exposure, attempted action, completed action, and user impact.

Learn without expanding exposure

Incident analysis often creates more copies of the sensitive data than the original event. Use redacted excerpts, identifiers, hashes, and controlled access. Keep the incident record separate from broad product telemetry when the content is especially sensitive. Record the mechanism and the corrective control so the lesson can become a regression test without republishing the original secret or customer content.

Assign ownership across the system

Security is shared across product, application, platform, data, and operations teams, but each signal still needs one accountable owner. Name who reviews authorization failures, who owns redaction, who can pause a tool, who approves provider changes, and who maintains adversarial fixtures. Review ownership after reorganizations and major workflow changes. An alert without a reachable owner is not a control; it is a delayed discovery.

Use a measured security cadence

Daily review may focus on active high-risk failures and unacknowledged approvals. Weekly review can examine blocked actions, near misses, redaction tests, and changes in tool usage. Monthly review can revisit permissions, retention, provider routing, memory scope, and the threat model. Keep the cadence proportional to impact. The goal is to make evidence and accountability routine before an incident forces attention.

Final security review questions

Before enabling a new agent capability, ask: What is the worst plausible action? Which input can influence it? Where is the independent authorization check? What happens if the model, tool, provider, or queue lies or fails? Can the team reconstruct one representative run without exposing unnecessary data? Who can stop the capability? What regression case proves the boundary? If any answer is unclear, narrow the scope and collect evidence before granting more autonomy.

A worked review sequence

Imagine a support agent that can search an account, draft a reply, change a shipping address, and issue a small credit. Begin by classifying each operation. Searching the current account may be a low-risk read if the application derives the account from the session. Drafting is a proposal and should not send anything. Changing an address is a write that affects future delivery and needs field-level validation. Issuing credit is a financial action whose amount, order, and approval policy must be checked independently.

Next, follow an ordinary run and a deliberately hostile run. In the ordinary run, the user asks for a delivery update, the agent retrieves the order, proposes a response, and stops because no change is needed. In the hostile run, a retrieved note contains an instruction to issue a credit and reveal internal policy. The expected trace should show the note as untrusted content, the attempted policy transition as blocked, and the user-facing answer limited to the delivery question. If the trace only shows a final refusal, the team has not yet proved which control prevented the action.

Then test a race. The user approves an address change, but the address changes in the account before the worker executes. The worker must recheck ownership, current state, normalized arguments, and approval expiry. If it cannot prove that the approved action still applies, it should stop and request a new decision. This example demonstrates why an approval record, execution record, and final result are separate events rather than one boolean field on a workflow.

Document residual risk

No control eliminates all risk. State what remains: provider compromise, incorrect classification, unavailable reviewers, stale policy, compromised credentials, incomplete redaction, or a user misunderstanding a confirmation. Residual risk should have an owner and a monitoring plan. If the team accepts it for a limited cohort, record the cohort, duration, evidence threshold, and decision date. This makes a temporary compromise distinguishable from an undocumented permanent exposure.

Revisit after capability changes

Adding a new tool, memory source, browser permission, provider, or automation path can invalidate the original threat model even when the prompt does not change. Trigger a focused review for each new trust boundary. Compare the new permission matrix with the previous one and update the adversarial suite. Do not assume that a capability is safe because its first examples are read-only; future prompt or tool changes may compose it with a high-consequence action.

Security is part of product quality

Users experience security through predictable boundaries: the agent explains what it can do, asks before consequential actions, keeps information in the right account, and fails safely when evidence is incomplete. Operators experience it through traces that expose the decision path without creating a second data leak. Engineers experience it through fixtures, policy checks, and a release process that makes the safe behavior easy to preserve. Those are not separate concerns. They are the practical meaning of a secure AI agent in production.

Make the safe path observable

Record successful policy checks as well as blocked actions. Without successful examples, an operator cannot tell whether a control is narrowly enforcing the intended boundary or simply rejecting all work. Sample approved actions by risk class and compare their normalized arguments with the eventual execution. This gives the team a baseline for detecting drift in permissions, routing, and tool behavior.

Review near misses

A near miss is an action that was almost allowed outside its intended scope, even if a later control stopped it. Near misses may include a cross-tenant identifier rejected by validation, a replayed approval caught by expiry, or an untrusted document that changed the model’s proposed plan but did not reach execution. Review them without blaming the individual component. The system succeeded because multiple layers existed; the lesson is to understand which layer was necessary and whether the earlier layer can become clearer.

Close the loop with users

When an agent refuses or pauses, the explanation should tell the user what safe next step is available without revealing internal policy or sensitive data. A useful response might request missing authorization, ask the user to confirm a bounded action, or route the case to a named support process. Track whether users abandon, retry, or complete the workflow after the boundary. Security that consistently creates confusing dead ends will encourage unsafe workarounds, while a clear boundary can improve both trust and completion.

Keep evidence proportional

The best security trace is not the largest trace. It is the smallest authorized record that answers what happened, why it was allowed or blocked, which policy applied, and what needs to happen next. Start with identifiers, decisions, versions, and outcomes. Add sensitive content only through controlled access when the investigation requires it. This reduces exposure while preserving the causal story needed for engineering and incident response.

Check the control during ordinary operation

Do not reserve security review for incidents. Sample ordinary runs and confirm that the principal, tenant, policy, tool, and outcome fields agree. Check that low-risk work is not being routed through an overly broad credential and that approval queues contain enough context to make decisions. Ordinary samples reveal quiet drift: a new integration may widen a scope, a new field may bypass redaction, or a fallback may quietly use a provider with a different data policy.

Make changes easy to reverse

For a new tool, begin with a dry-run response that shows the normalized action without executing it. For a new provider, route only synthetic or approved low-sensitivity cases. For a new memory source, keep it read-only and attach provenance. Define the evidence that allows each boundary to widen and the evidence that causes an immediate rollback. Reversibility gives the team room to learn while keeping an unsafe assumption from becoming a permanent permission.

The release reviewer should sign off on the boundary, not merely the implementation. Record the reviewed workflow, tools, data classes, fixtures, observed results, unresolved risks, and rollback owner. That record is the durable proof that the team considered security before increasing autonomy, and it gives the next reviewer a concrete baseline instead of another informal promise. Keep the decision attached to the release and revisit it after real usage. Compare the evidence with the baseline before widening permissions and after incidents.

Decision rule: Treat every model output, retrieved document, tool argument, and browser result as untrusted until the application validates its scope and authority.

Frequently asked questions

Build the threat model around actions

AI agent security starts with what the system can cause, not with a generic list of model risks. Inventory data, identities, tools, external systems, durable memory, and irreversible actions. For each asset, identify who may request access, what the model can propose, what the application must authorize, and how an operator can contain misuse. This produces controls tied to consequences instead of a checklist detached from the workflow.

Separate instruction from authority

User input, retrieved documents, tool responses, browser content, and memory can all contain instructions. Treat them as untrusted data. The model may summarize or propose, but application policy decides which resources and actions are allowed. Validate tenant, actor, resource, arguments, and current state at the tool boundary. A prompt that says “ignore previous instructions” must not change the authorization path.

Control the confused deputy

An agent often has more access than the user who initiated a task. Propagate identity and tenant context to every retrieval and tool call. Use least-privilege credentials, narrow resource scopes, allowlists for destinations, and explicit approval for high-impact actions. Test a user who asks for another tenant’s record, a document that contains malicious instructions, a revoked session, and a tool argument that changes the target after approval.

Protect memory and browser paths

Memory should have a purpose, owner, retention rule, deletion path, and access policy. Do not write untrusted claims into durable memory without validation. Browser agents need visible origin checks, destination restrictions, confirmation for submissions, and protection against page content impersonating system instructions. Capture enough evidence to investigate without storing secrets or unnecessary personal data.

Prepare containment and evidence

Security controls are incomplete without a response path. Log policy decisions, denied actions, credential scope, tool arguments after redaction, approval state, and resulting changes. Define how to revoke a token, pause a workflow, quarantine an input source, invalidate memory, and identify affected runs. Rehearse the path before launch and keep one representative adversarial case in the regression set.

Security rule: The model can recommend an action, but only the application can authorize it at the current side-effect boundary.