The AI infrastructure stack: What each layer actually does

At a glance

If you are the engineer who just got asked to "own AI infra" for a product team, the job is rarely one system. It is a stack of eight layers that each solve one narrow problem, and a request in production touches most of them before it becomes a response. This article names each layer, explains what it does and does not do, and shows where they hand off to each other. It does not rank vendors or tell you which specific product to buy in each layer — the boundary here is architecture, not procurement.

The operating model

Think of the stack as a request's path, not a diagram of teams or budgets. Every layer either prepares data, produces a model output, enforces a rule, measures quality, or records what happened. Reading the stack this way keeps you from buying a tool for a layer you don't actually have — a team running one well-tested prompt against one hosted model does not need a full orchestration graph yet.

Start with the data the system reads

Every layer above data pipelines assumes the data underneath it is trustworthy. That assumption is usually the first one that breaks.

What a data pipeline for AI actually moves

A data pipeline in this context is not a generic ETL job. It moves three distinct kinds of material, and conflating them is a common early mistake:

1. Source content — documents, tickets, transcripts, code, or product data that will be chunked and embedded for retrieval.
2. Fine-tuning or preference data — labeled examples used to adapt a model's behavior, usually much smaller and higher-scrutiny than source content.
3. Evaluation and eval-set data — held-out examples with known-good answers, used to score changes before and after they ship.

Treating these as one undifferentiated "AI data" bucket causes a specific failure: teams accidentally let evaluation examples leak into the retrieval index, which makes evaluation scores look better than production reality because the system is retrieving the answer key.

Field note: Keep the eval set in a separate store with separate access control from the retrieval index, even if both start as the same three CSV files. The first time someone "just quickly" copies an eval question into the knowledge base to fix a support ticket, your evaluation numbers stop meaning anything, and nobody notices for months.

Freshness is the property that decays

Unlike a traditional application database, a retrieval index or fine-tuning dataset does not throw an error when it goes stale — it just quietly returns worse answers. The pipeline layer's real job is keeping three properties current: freshness (how old is the source), completeness (did the last sync actually finish), and provenance (can you trace an answer back to the document it came from). A pipeline that only handles ingestion and skips freshness, completeness, and provenance checks will look fine in a demo and fail six weeks into production.

Give the model something to retrieve

Retrieval is the layer most teams add first after a bare model call, because it is the fastest way to make a general-purpose model sound like it knows your product.

What a vector store is actually for

A vector store indexes embeddings — numeric representations of text, code, or other content — so that a query can find semantically similar chunks quickly, even when the words don't match exactly. This solves keyword search's biggest weakness: a user asking "why did my payment fail" should find a document titled "declined transaction troubleshooting" even though no words overlap.

The deeper technical trade-offs — chunk size, hybrid search, re-ranking, and index architecture — are covered in more depth in a dedicated guide to picking and operating a vector database; this article treats retrieval as one layer in the larger request path rather than repeating that detail.

Retrieval augmented generation is not automatically grounded

A retrieval step reduces hallucination risk, but it does not eliminate it. The model still has to read the retrieved chunks and decide what to say, and it can misread, over-generalize, or ignore retrieved context entirely if the prompt does not force it to cite what it used. Retrieval gives the model better raw material; it does not force the model to use it correctly. That gap is exactly what the evaluation and guardrail layers exist to catch.

Run inference: the model serving layer

Model serving is the layer people usually mean when they say "the model." It is the component that actually runs inference and returns tokens, whether that's a hosted API call or a self-hosted GPU cluster.

Two structurally different serving paths

Most production systems end up using both: a managed API for iteration and burst capacity, and self-hosted or reserved capacity for the steady-state workload that dominates the bill. A gateway that routes requests across providers is the component that usually sits in front of this choice, so the application code doesn't have to hardcode a single vendor's client library.

The two knobs that actually move latency

Serving-layer latency comes from two places: time to first token (how long before anything streams back) and tokens per second once streaming starts. Batching improves throughput and cost but can hurt time-to-first-token under load, because a request sometimes waits for a batch window to fill. Streaming responses hide total latency from the user's perception but do not reduce it — a slow model streamed one token at a time still finishes slowly, it just feels less broken while it does.

Sequence the calls: orchestration

Orchestration is the layer that decides what happens next — which model to call, which tool to invoke, whether to retry, and in what order. For a single prompt-in, answer-out system, orchestration can be nearly invisible. For an agent that calls tools, checks results, and decides on a next action, orchestration is the layer that turns a language model into something that can actually complete multi-step work.

What orchestration owns

• Sequencing: deciding the order of model calls, tool calls, and retrieval steps.
• State: carrying context, intermediate results, and conversation history across steps.
• Retries and fallbacks: what happens when a tool call times out or a model returns an error.
• Branching: choosing between paths based on intermediate output, such as escalating to a human when confidence is low.

A minimal, illustrative retry policy looks like this. It is not a full orchestration framework — it shows the shape of the decision an orchestration layer has to make on every tool call.

// Illustrative: a bounded retry policy for one tool call inside an agent step.
type ToolOutcome = "success" | "timeout" | "error";

interface RetryDecision {
  action: "return" | "retry" | "escalate";
  attemptsUsed: number;
}

export function decideToolRetry(
  outcome: ToolOutcome,
  attemptsUsed: number,
  maxAttempts = 2
): RetryDecision {
  if (outcome === "success") {
    return { action: "return", attemptsUsed };
  }
  if (attemptsUsed >= maxAttempts) {
    return { action: "escalate", attemptsUsed };
  }
  return { action: "retry", attemptsUsed: attemptsUsed + 1 };
}

The important detail this snippet makes concrete: a retry policy needs a hard ceiling (maxAttempts) and an explicit terminal action (escalate) other than silently failing. An agent that retries a failing tool call indefinitely is not resilient — it is a cost and latency incident waiting to happen. Multi-agent orchestration patterns go further into how this sequencing changes once more than one agent is coordinating work, including handoffs and shared state.

Orchestration frameworks versus hand-rolled control flow

Teams often ask whether they need an orchestration framework at all. A useful rule: if your system has more than two sequential decision points that depend on model output (not just fixed steps), a framework's state management and retry primitives usually save more engineering time than they cost in abstraction overhead. Below that threshold, a plain function calling a model and a tool in sequence is often clearer and easier to debug than a framework's abstractions.

Enforce the rules: guardrails

Guardrails are the layer that decides whether an input or output is allowed to proceed, independent of whether it's a good answer. A response can be well-written, fluent, and still be a policy violation — leaking a system prompt, recommending an off-label medical use, or authorizing a refund above a threshold without approval.

Where guardrails sit in the request path

Guardrails typically run at two points: input guardrails before the model sees the request (blocking prompt injection attempts, PII in the wrong context, or out-of-scope requests), and output guardrails before the response reaches the user or before a tool call executes (blocking unsafe content, enforcing structured-output schemas, or requiring approval for high-risk actions). A dedicated guide to agent guardrails covers the layered model of input, context, output, authorization, limits, approval, and audit controls in more depth than fits here.

Decision rule: If an action is irreversible (sending money, deleting data, sending an external email), the guardrail layer should require explicit approval or a stricter threshold than an action that's easy to undo. Reversibility, not just "risk," is the variable that should set the bar.

Guardrails are not a substitute for evaluation

A guardrail answers "was this allowed?" An evaluator answers "was this good?" A response can pass every guardrail and still be a bad answer — technically permitted, factually wrong, or unhelpfully verbose. Teams that only invest in guardrails and skip evaluation end up shipping systems that are safe but mediocre, and they usually don't find out until user complaints pile up, because guardrails don't measure quality at all.

Measure quality: evaluation

Evaluation is the layer that scores output against a rubric, a labeled dataset, or another model acting as a judge. It runs in two different modes that are easy to conflate.

Offline evaluation versus online evaluation

Offline evaluation is what makes a prompt or model change a reviewable pull request instead of a guess — you can compare a candidate's score against the current production baseline on the same dataset before merging. Online evaluation is what tells you the offline dataset stopped representing reality, which it always eventually does as usage patterns shift. A deeper walkthrough of the specific metrics worth tracking is useful once you're past the "do we have any evaluation at all" stage this article addresses.

A framework doesn't replace judgment

Automated scoring — including LLM-as-judge — is a proxy for what you actually care about, not the goal itself. A rising eval score with flat or falling user satisfaction is a sign the eval set stopped matching production traffic, not a sign the product improved. Treat evaluation scores as a leading indicator to investigate, not a number to optimize in isolation.

Record what happened: observability

Observability is the layer that captures the full path of a request — every model call, tool call, retrieval step, guardrail decision, and evaluation score — as one connected trace you can replay after the fact. It is the layer this article's interaction above demonstrates: the retry, the guardrail check, and the eventual score are only useful together if you can see them as one sequence, not as separate logs in separate systems.

Why generic APM does not cover this layer

Traditional application performance monitoring instruments requests, database calls, and exceptions well, but it has no native concept of a prompt, a token, a retrieved chunk, or a model's confidence. AI observability adds trace schemas built around those concepts: which prompt version ran, which chunks were retrieved and whether they were used, what the model actually returned before any post-processing, and what the evaluator scored it. Without that schema, debugging a bad agent response means reconstructing the request from scattered logs — assuming every layer even logged it.

Ship and scale: deployment

Deployment is the layer that versions, releases, and scales the whole system — not just the model, but the prompts, the orchestration logic, the guardrail rules, and the retrieval index together, since a mismatch between any two of them can break behavior.

What changes when the "artifact" isn't just code

A traditional deployment ships one artifact: compiled code. An AI system ships at least four things that can each drift independently: application code, the model or model version, the prompt, and the retrieval index. A rollback plan that only reverts code and forgets the prompt version that shipped alongside it will "roll back" into a broken combination. Rollback strategies for agent deployments cover this specific problem — versioning prompts and models together with code — in more detail than this overview layer can.

Scaling has a queueing problem, not just a compute problem

Scaling an AI system is not purely "add more GPUs." Because inference has meaningfully higher and more variable latency than a typical API call, request queues build up differently — a burst of traffic can create a backlog that a stateless web service would never notice, because each request holds a connection open for seconds instead of milliseconds. Capacity planning for this layer has to account for concurrent in-flight requests, not just requests per second.

Choose a serving path for a real workload

Work through one example to see how the layers interact instead of staying abstract. A team is building an internal support agent: moderate volume (a few thousand requests a day), latency-sensitive (agents expect sub-3-second first response), and handling some regulated customer data.

Walking the decision

Given moderate, variable volume, this workload fits a managed API behind a gateway, with orchestration and guardrails owned by the application rather than the provider, and observability wired in from day one so the team has evidence — not guesses — when they revisit the self-hosting question at higher volume. That's a gateway-plus-routing path, not because it's the default answer, but because the volume and latency profile pointed there.

Protect the security boundary across layers

Security in this stack is not one layer's job — it is a set of boundaries that span several layers, and treating it as only the guardrail layer's problem leaves gaps.

• Data pipelines: control what source data is eligible for retrieval, and audit for accidental inclusion of secrets, credentials, or eval answer keys in ingested content.
• Model serving: control which callers can invoke which models, and ensure API keys or credentials are never embedded in prompts sent to a third-party provider.
• Orchestration: scope tool permissions per agent and per action — a tool that can read a customer record should not automatically be able to write to it.
• Guardrails: enforce approval requirements for irreversible actions, and treat prompt injection from retrieved or tool-returned content as an untrusted input, not just user text.
• Observability: store traces with the same access controls as the underlying data — a trace of a support conversation contains the same sensitive content the conversation did.

None of these layers alone constitutes "AI security." A system with excellent guardrails but no permission scoping in orchestration can still let a compromised tool call do more damage than the guardrail layer was designed to catch.

Where the layers hand off

The failure mode worth naming explicitly: most production incidents in AI systems happen at the handoff between two layers, not inside a single layer. A retrieval result that's technically correct but stale hands off to a model that has no way to know it's stale. A model output that violates policy hands off to a guardrail that only checks for a narrower category of violation. An orchestration retry hands off to a serving layer that wasn't told the retry was happening and double-bills the request.

Field note: When debugging a production incident in one of these systems, resist the urge to inspect the layer that produced the visibly wrong output first. Start one layer upstream — the input it received — and one layer downstream — what happened to its output next. The defect is more often in a handoff than in the layer that surfaced the symptom.

This is also the strongest argument for the observability layer existing at all: it is the only layer whose entire job is making handoffs visible, rather than making one layer's own behavior visible in isolation.

Make the build decision

The practical sequencing most teams land on: ship observability and basic guardrails alongside the very first version, even a single-prompt one. Add retrieval when answers need to be grounded in your own data. Add orchestration when the task needs more than one model or tool call to complete. Add self-hosted serving only when volume and cost data — from the observability layer you already built — justify the operational cost of running it.

Leave the reader with the next move

• [ ] Every layer in your system is named, not implicit — you can say which component owns retries, which owns policy, and which owns quality scoring.
• [ ] Observability captures a full request end to end, across model calls, tool calls, and guardrail decisions, in one connected trace.
• [ ] Prompts, models, and retrieval index versions are tracked together, not just application code.
• [ ] Irreversible actions require an explicit approval or authorization step, not just a guardrail check.
• [ ] Evaluation runs offline before release and is sampled in production, not treated as a one-time launch gate.

If you can't check most of these boxes today, the fastest-leverage next step is usually observability, not a new orchestration framework — it's the layer that turns every future incident into evidence instead of a guess.

Frequently asked questions