Context Window Management for Long-Running Agents

An agent that runs for five turns can get away with stuffing everything into the prompt. An agent that runs for two hundred turns cannot. Somewhere between those two runs, unbounded context stops being a convenience and starts being the reason the agent forgets its own instructions, calls the wrong tool, or simply errors out with a context-length exception in front of a user.

At a glance

This guide is for engineers building agents that run long conversations or loops. It helps you allocate a fixed context window across system prompt, tools, history, and scratch memory, then compact or offload before hitting the limit so you can leave with a budget you can measure, enforce, and observe in production.

The context budget framework

Use Allocate → Measure → Compact → Observe as the decision path:

1. Allocate: give system prompt, tools, history, and scratch memory each a share of the window before the run starts.
2. Measure: count tokens per component on every turn, not just at the end.
3. Compact: trim, summarize, or offload the components that grow, before they push out the ones that do not.
4. Observe: track budget usage across a run and alert before a hard failure, not after.

Decision rule: Every component of the prompt should have an owner and a ceiling. If a component has no ceiling, it will eventually consume the whole window.

The framework does not claim one summarization strategy or one context size is universally correct. It gives a repeatable way to reason about a resource that is finite no matter which model you use.

Why context is a budget, not a convenience

A model's context window is a hard resource constraint, the same way memory or disk space is for a regular service. The difference is that most teams do not treat it that way. A prompt template grows a field here, a tool response gets pasted in whole there, and the conversation history is simply appended to on every turn. None of these decisions look dangerous in isolation. Together, over a long-running agent, they are the single most common cause of degraded quality that has nothing to do with the model itself.

Four components compete for the same window in almost every agent architecture:

• System prompt — instructions, persona, output format, and safety rules. Usually fixed size per release.
• Tool definitions — names, descriptions, and parameter schemas for every tool the agent can call. Grows with the toolset, not with the conversation.
• Conversation history — the running transcript of user turns, agent turns, and tool calls/results. Grows every turn unless something intervenes.
• Scratch or working memory — intermediate reasoning, plans, retrieved snippets, or notes the agent keeps for itself during a task. Can spike sharply mid-task and shrink again.

Treat these four as line items in a budget rather than as an undifferentiated blob of "the prompt." A budget makes the trade-off explicit: giving conversation history more room means giving something else less room, and every model has a hard ceiling where the trade stops being optional.

Why bigger context windows do not solve this

It is tempting to treat a larger context window as a fix. A 200K or 1M token window buys time, not immunity. Two problems persist regardless of window size:

1. Cost and latency scale with tokens sent, not just tokens available. A 150,000-token prompt costs and takes longer than a 15,000-token prompt, whether or not the window technically supports it.
2. Context rot is a quality problem, not just a capacity problem. Research and practical evaluation both show that model attention to any single fact degrades as irrelevant tokens accumulate around it, well before the hard token limit is reached. A model can technically fit a fact in context and still fail to use it correctly if it is buried under thousands of tokens of noise.

A bigger context window changes when you run out of room. It does not change whether a crowded prompt degrades answer quality before that point.

Build the operating model: a token ledger

Before writing any trimming logic, define the ledger you are trying to keep balanced. A token ledger names every component, its current size, its ceiling, and what happens when it is over budget.

The percentages are a starting point, not a rule. A coding agent with forty tool definitions might spend a much larger share on tools than a customer support agent with five. The point of writing the ledger down is that every team can see where their own budget actually goes, rather than assuming history is the only variable that matters.

Count tokens before you trim anything

You cannot manage a budget you cannot measure. Token counting has to happen with the same tokenizer the target model uses, on the actual serialized payload you plan to send — including tool-call JSON, role markers, and any framework overhead, not just the visible text.

// Illustrative: adapt the tokenizer import to your provider's SDK.
import { encoding_for_model } from "@dqbd/tiktoken";

type ContextComponent = "system" | "tools" | "history" | "scratch";

type TokenBudget = {
  limit: number;
  used: Record<ContextComponent, number>;
};

const encoder = encoding_for_model("gpt-4o");

function countTokens(text: string): number {
  return encoder.encode(text).length;
}

export function buildBudget(
  limit: number,
  parts: Record<ContextComponent, string>,
): TokenBudget {
  const used = Object.fromEntries(
    (Object.keys(parts) as ContextComponent[]).map((key) => [key, countTokens(parts[key])]),
  ) as Record<ContextComponent, number>;

  return { limit, used };
}

export function remainingBudget(budget: TokenBudget): number {
  const total = Object.values(budget.used).reduce((sum, tokens) => sum + tokens, 0);
  return budget.limit - total;
}

This is deliberately small. The important habit is calling buildBudget on every turn, not just once at the start of a run, since history and scratch memory are the two components most likely to move.

Field note: Reserve headroom for the model's own output. A budget that fills the entire window with input leaves no room for a long response, which then gets truncated mid-answer — a failure that looks like a model bug but is actually a budget bug.

Enforce the budget before the call, not after

A token counter that only reports usage after a call fails has already cost you the latency and the money for that call. Enforce the budget before constructing the request, with a clear policy for what happens when a component would push the total over the limit.

type EnforcementResult =
  | { action: "proceed"; remaining: number }
  | { action: "compact"; overBy: number; component: ContextComponent }
  | { action: "escalate"; reason: string };

const RESPONSE_RESERVE = 2_000; // tokens reserved for the model's own output

export function enforceBudget(budget: TokenBudget): EnforcementResult {
  const remaining = remainingBudget(budget) - RESPONSE_RESERVE;
  if (remaining >= 0) return { action: "proceed", remaining };

  // Compact the largest compactable component first; history usually wins.
  const compactable: ContextComponent[] = ["history", "scratch", "tools"];
  const target = compactable.reduce((largest, key) =>
    budget.used[key] > budget.used[largest] ? key : largest,
  );

  if (budget.used[target] === 0) {
    return { action: "escalate", reason: "No compactable component has room to give; the system prompt alone is over budget." };
  }

  return { action: "compact", overBy: Math.abs(remaining), component: target };
}

The escalate branch matters as much as the compact branch. If the system prompt and tool definitions alone exceed the budget, no amount of history trimming fixes the run — that is a configuration problem, and the agent should stop and report it rather than silently truncating instructions.

Trim and summarize: three strategies compared

Once you know which component is over budget, you need a compaction strategy. Three approaches cover most production cases, and they trade information loss against cost and latency differently.

Sliding window

Keep only the most recent N turns verbatim and drop everything older. This is the cheapest option: no extra model calls, no summarization latency, and a predictable token count.

type Turn = { role: "user" | "assistant" | "tool"; content: string };

export function slidingWindow(history: Turn[], keepTurns: number): Turn[] {
  return history.slice(-keepTurns);
}

The failure mode is obvious: anything outside the window is gone. A user's stated constraint from turn two is invisible to the agent at turn fifty. Sliding windows work well for short-lived, stateless-ish tasks (a single Q&A exchange, a bounded tool-calling loop) and work poorly for anything that depends on early commitments staying in force.

Rolling summarization

Periodically replace older turns with a running summary, keeping only the most recent turns verbatim alongside it. Each compaction pass folds the previous summary and the turns being dropped into a new, updated summary.

type SummarizeFn = (priorSummary: string, turnsToFold: Turn[]) => Promise<string>;

export async function rollingSummarize(
  history: Turn[],
  keepVerbatim: number,
  priorSummary: string,
  summarize: SummarizeFn,
): Promise<{ summary: string; recent: Turn[] }> {
  if (history.length <= keepVerbatim) {
    return { summary: priorSummary, recent: history };
  }

  const cutoff = history.length - keepVerbatim;
  const toFold = history.slice(0, cutoff);
  const recent = history.slice(cutoff);
  const summary = await summarize(priorSummary, toFold);
  return { summary, recent };
}

Rolling summarization preserves the gist of the whole run in bounded space, which is a large improvement over a sliding window for anything that spans many turns. The cost is an extra model call per compaction pass, added latency at the moment of compaction, and gradual information loss — a fact mentioned once, twenty turns ago, and folded into three successive summaries may not survive with its original precision.

Hierarchical summarization

Keep summaries at multiple granularities instead of collapsing everything into one running paragraph: a per-subtask summary, a per-phase summary, and a whole-run summary, each retained and updated independently. When the agent needs detail, it can look at the finer-grained summary for the relevant phase instead of a single blurred account of the entire run.

type SummaryLevel = { label: string; spanTurns: [number, number]; summary: string };

export type HierarchicalMemory = {
  levels: SummaryLevel[]; // ordered fine-to-coarse
};

export function selectRelevantSummary(
  memory: HierarchicalMemory,
  turnIndex: number,
): SummaryLevel | undefined {
  return memory.levels.find(
    (level) => turnIndex >= level.spanTurns[0] && turnIndex <= level.spanTurns[1],
  );
}

This preserves more usable detail than a single rolling summary at a similar total token cost, but it is the most complex to implement and requires deciding, at query time, which level of the hierarchy is relevant — a retrieval decision layered on top of a summarization decision.

When to offload instead of compacting

Compaction assumes the information belongs in context at all, just in a smaller form. Often it does not. If a fact is stable, structured, and retrievable by key, it usually belongs in external memory or retrieval rather than in the running transcript.

A useful test: would you be comfortable if this fact were dropped from context and re-fetched only when needed? If yes, offload it. If the agent needs to reason over it continuously across many turns, summarization is more appropriate than retrieval. AI agent memory design covers the scope, provenance, and correction rules that make offloaded memory safe to reuse — the short version here is that a memory record needs a source and an expiry, not just a vector embedding.

Retrieval also changes the shape of the cost curve. Instead of paying for the same growing history on every turn, you pay a smaller, closer-to-fixed cost to fetch only what a specific turn needs. That trade only pays off if retrieval quality is good enough that the agent does not end up re-asking for the same fact repeatedly, which is itself worth measuring rather than assuming.

Summarization keeps information in context in a compressed form. Offloading removes it from context and makes the agent ask for it back. Choose based on how often the agent needs the fact, not how big it is.

Try the budget yourself

Adjust the sliders below to see how the four components interact against a chosen context window. The bar chart and remaining-budget figure update immediately; nothing here calls a model or stores data beyond this page.

Try the presets: a support agent on an 8K window runs out of room almost immediately once history passes a handful of turns, which is exactly why short-window deployments need aggressive sliding windows or frequent summarization. A coding agent on a 1M window can carry much more history verbatim, but the cost and context-rot arguments above still apply well before the hard limit — a full window is rarely the cheapest or highest-quality choice available.

Watch context rot, not just token count

Token count tells you whether a request will succeed. It does not tell you whether the model is actually using the tokens you sent. Context rot — the gradual degradation of a model's ability to use information buried deep in a long context — shows up as a quality regression long before a context-length error does.

Practical signals worth tracking alongside raw token counts:

• Instruction adherence over the run. Does the agent still follow formatting or safety rules stated in the system prompt by turn fifty, or has it drifted?
• Repeated questions. An agent re-asking something the user already answered is a strong signal that the relevant fact fell out of the effective context, even if it is technically still present in the raw transcript.
• Tool selection accuracy over time. If tool definitions are getting crowded out or buried, the agent may start choosing the wrong tool or omitting required parameters more often as a run progresses.
• Position of critical facts. Facts placed in the middle of a long context are used less reliably than facts placed near the beginning or the end. If a compaction strategy buries the system prompt in the middle of a reconstructed context, expect adherence to drop even though nothing was technically deleted.

None of these signals require a new tool if you already trace agent runs — they require deciding to look at turn-level context composition, not just the final answer, when investigating a quality regression. Optimizing LLM latency covers the related cost and speed side of trimming context, since a smaller, well-chosen context is usually also a faster one.

Compare the meaningful trade-offs

Decision rule: If the agent's next decision depends on a fact staying continuously "in mind" across many turns, keep it in context in compressed form. If the agent only needs the fact occasionally and can ask for it by key, offload it.

Protect the security boundary

Context management touches the same trust boundary as any other input to the model. A summarization pass that folds tool output into a running summary can also fold in untrusted content from a retrieved page or a user-controlled field, and that content persists across every future turn that includes the summary. Treat summarization input the same way you would treat any other untrusted text before letting it influence future tool selection or reasoning — this connects directly to the injection concerns in prompt injection defense.

Scratch memory deserves the same scrutiny. If an agent writes intermediate reasoning to a scratch buffer that later gets folded into a summary or exposed to a different tool call, make sure it does not silently carry secrets, credentials, or user PII further into the run than intended. A token-budget system that only asks "how big is this" and never asks "what is in this" is incomplete.

Observability: budget usage over a run

A budget you cannot see in production is a budget you are guessing about. Emit budget usage as structured span data on every turn, not just a final total, so an operator can reconstruct exactly when a run crossed a threshold.

type BudgetEvent = {
  runId: string;
  turnIndex: number;
  used: Record<ContextComponent, number>;
  limit: number;
  action: "proceed" | "compact" | "escalate";
};

export function recordBudgetEvent(event: BudgetEvent) {
  // Illustrative: send to your tracing pipeline as a child span on the parent run.
  const total = Object.values(event.used).reduce((sum, tokens) => sum + tokens, 0);
  const pctUsed = (total / event.limit) * 100;

  return {
    ...event,
    totalTokens: total,
    pctUsed: Number(pctUsed.toFixed(1)),
    nearLimit: pctUsed >= 85,
  };
}

Alert before the limit, not at it. A threshold around 80–85% of the window gives the run room to trigger a compaction pass or an offload before the next turn would otherwise fail outright. Route the alert to whatever already owns agent reliability signals, alongside the guidance in AI agent monitoring, so a budget breach shows up next to retries, latency, and cost rather than in an isolated dashboard nobody checks.

Two dashboards are worth building once budget events exist: a per-run timeline showing how the four components moved turn over turn, and a fleet-wide view showing what fraction of runs cross the near-limit threshold before completing. The second view tells you whether your default budget allocation is actually sized for how the agent behaves in production, rather than for how it behaved in a demo with three turns.

Read the lower half visually

The pattern above repeats in almost every long-running agent: system prompt and tool definitions hold roughly flat, history climbs until something intervenes, and scratch memory spikes during active reasoning and drops once a subtask closes. Design the budget around this shape, not around the size of the very first turn, since the first turn is the one place where every agent looks healthy regardless of its long-run compaction strategy.

What changes the recommendation

A few conditions shift the right answer away from the defaults above:

• Very short-lived tasks (single Q&A, one bounded tool call) rarely need summarization at all — a sliding window or even no trimming is fine if the run completes in a handful of turns.
• Regulatory or audit requirements may require keeping a verbatim transcript somewhere, even if the model only ever sees a compacted version. Separate "what the model sees" from "what gets stored" explicitly.
• Multi-agent handoffs change the unit of compaction: a summary meant for the next agent in a pipeline needs different content than a summary meant to keep the same agent oriented, since the next agent has no memory of the earlier turns at all.
• Tool-heavy agents may find tool definitions, not history, are the dominant budget item — in that case, load tools dynamically based on the current subtask instead of summarizing history more aggressively.

Make the operating decision

Every agent's real turn-count distribution should drive this decision, not intuition about how long conversations "usually" get. Pull turn counts from a week of production traces before choosing a compaction strategy, and revisit the choice whenever the toolset, model, or typical task shape changes materially.

Leave the reader with the next move

• [ ] Every context component (system, tools, history, scratch) has a named owner and a ceiling.
• [ ] Token counting runs on every turn with the actual serialized payload, not just visible text.
• [ ] A compaction strategy is chosen and matched to how long the agent actually runs in production.
• [ ] Summarization and scratch-memory content are treated as untrusted input, not assumed-safe internal state.
• [ ] Budget usage is traced per turn and alerts fire before the hard limit, not after a failure.

Start with the token ledger and the enforcement check above — they are the cheapest changes and the ones most likely to catch a budget problem before a user sees a truncated answer or a repeated question. Add summarization or retrieval offload only once traces show which component is actually driving the overrun in your own agent.

Frequently asked questions