AI Agents, for people building on top of an LLM
What actually turns a language model into something that plans, acts, and adapts — the loop underneath the word "agent," when it's worth the cost, and where it goes wrong.
The line is control flow, not intelligence
Anthropic's own framing draws the distinction cleanly: a workflow is a system where the LLM and tools are orchestrated through predefined code paths — you, the developer, decided the steps and their order in advance. An agent is a system where the LLM dynamically directs its own process and tool use, deciding how to accomplish a task as it goes, rather than following a path someone hardcoded.
The mechanism that makes this possible is a loop: the model gets a goal, plans a next step, takes an action (usually a tool call), observes the real result of that action, and decides whether to continue, adjust course, or stop. Anthropic states it plainly — an autonomous agent is essentially "LLMs using tools based on environmental feedback in a loop." The qualitative shift from a single LLM call is that the number of steps and which tools get used aren't fixed in advance — they're decided at runtime, based on what the model actually observes, not by code written ahead of time.
This is also the guide's first piece of practical guidance, stated up front rather than saved for the end: reach for an agent specifically when a task is genuinely open-ended and you can't predict the number of steps in advance. When you can enumerate the steps ahead of time, a hardcoded workflow is more reliable, cheaper, and easier to debug than letting a model improvise the same steps.
ReAct, and what tool calling actually looks like underneath
ReAct (Reasoning + Acting, Yao et al. 2023) is the pattern nearly every modern tool-using agent descends from. Instead of reasoning silently or acting blindly, the model interleaves the two explicitly: Thought → Action → Observation → Thought → Action → Observation, repeating until done. Reasoning-only chains (plain chain-of-thought) have no mechanism to check facts against the outside world, so errors and fabrications propagate silently. Acting-only loops have no explicit plan-tracking, so they can't gracefully recover when a step fails. Interleaving both lets the model track intent, revise its plan when a step doesn't work, and stay grounded in real tool outputs rather than its own unchecked assumptions — in the original paper's benchmarks, this improved task success rates by double-digit percentage points over either approach alone.
What that looks like mechanically: the model is given a list of tool definitions — a name, a natural-language description (this is what the model actually uses to decide when to reach for it), and a schema describing the expected arguments. At generation time, the model decides whether it needs a tool, and if so emits a structured call — a tool name plus JSON arguments — instead of a plain-text answer. Your code (not the model) actually executes that call, and the result gets appended back into the conversation as a new message for the model to read on its next turn. The loop is: send prompt + tools → receive a tool call → execute it → return the result → call the model again — repeating until it emits plain text instead of another tool call. This is the engineering-level realization of the abstract Thought/Action/Observation cycle above.
Beyond one step at a time
Pure ReAct calls the model at every single step, which is reliable but slow and expensive for long tasks. Several alternative planning strategies trade that reactivity for speed, cost, or better upfront reasoning.
| Strategy | How it differs from step-by-step ReAct |
|---|---|
| Plan-and-execute | A capable model produces a full multi-step plan upfront; execution of each step can hand off to lighter models or deterministic code without re-consulting the planner every time |
| Re-planning | After execution, the agent reassesses — if the plan didn't work, it generates a follow-up plan rather than rigidly finishing the original one |
| ReWOO | Decouples planning from execution using variable substitution, so a full plan can reference not-yet-known outputs of earlier steps — cutting redundant model calls |
| LLMCompiler | Represents the plan as a dependency graph and executes independent branches in parallel rather than serially — reported up to a 3.6× speedup |
| Tree-of-thought / search-based | Explores multiple candidate reasoning paths and backtracks from unpromising ones — far more model calls, but useful when the best path genuinely isn't obvious upfront |
None of these replace ReAct's core insight — grounding reasoning in real observations — they just change when the model gets consulted, trading reactivity for cost and speed.
What the agent remembers, and for how long
Short-term memory is scoped to a single run — the conversation history, scratchpad, and accumulated tool outputs within one execution. It vanishes when that session ends; it's effectively "whatever fits in the current context window." Long-term memory persists across sessions, letting an agent recall facts, prior outcomes, or preferences from entirely separate interactions — typically implemented as records in a namespaced store (often vector-database-backed for semantic recall, sometimes plain key-value for facts that need exact retrieval rather than similarity search).
This remains one of the genuinely open problems in agent design — not because storage is hard, but because three questions don't have clean answers yet: what is actually worth persisting (not every interaction should become permanent memory), when to write or update it, and how to guarantee retrieval actually surfaces the relevant memory at the right moment instead of drowning it in irrelevant entries. The stakes are concrete, not theoretical: Anthropic's own production multi-agent research system explicitly saves its plan to external memory before continuing, specifically so a long-running task doesn't lose its own strategy once the conversation crosses roughly 200,000 tokens and starts getting truncated — for long-horizon agentic work, memory is a correctness requirement, not a nice-to-have.
When more agents beat one better agent
The dominant multi-agent pattern is orchestrator/worker: a lead agent breaks a task into sub-questions and spawns subagents to explore each in parallel — each with its own separate context window, its own tool calls, returning a compressed summary rather than raw output for the orchestrator to synthesize. A related pattern is debate/critique (Anthropic calls this "evaluator-optimizer"): one model generates a candidate, a second critiques it, looping until the output clears a quality bar — useful specifically when there's a clear standard a critic can check against.
Anthropic's own reported result is a useful calibration point: a multi-agent system (one lead model plus parallel subagents) outperformed a single strong agent by 90.2% on their internal research benchmark — but specifically on breadth-first tasks that decompose into independent, parallelizable directions. Multi-agent systems effectively multiply usable context by giving each subagent a fresh window; their own analysis found token usage alone explained roughly 80% of the performance variance — the gains come substantially from parallel context capacity, not just "more brains thinking about it."
The cost side is just as concrete: multi-agent systems use roughly 15× the tokens of a single chat interaction (agentic loops alone already run around 4×). Vague task assignment causes subagents to duplicate the same searches instead of actually dividing the work; reconciling conflicting decisions written by different subagents to shared state is difficult after the fact; and even identical starting conditions can produce different, individually-valid-looking trajectories, which makes multi-agent runs inherently harder to reproduce and debug than a single agent's. This pattern is a good fit for loosely-coupled, parallelizable work — not for tasks with tight interdependency between steps, where the coordination overhead outweighs the benefit.
What exists, and what problem each one targets
| Framework | What it's built to solve |
|---|---|
| LangGraph | Graph-based orchestration — agents as explicit state machines, for developers who want fine-grained control over control flow that a plain "agent loop" abstraction hides |
| CrewAI | Role-based "crews" of agents (role, goal, backstory) collaborating via sequential or hierarchical processes — a higher-level, less code-heavy abstraction for standing up agent teams quickly |
| AutoGen / AG2 | Multi-agent conversation patterns — agents (and optionally a human) converse to jointly solve a task; AG2 is the community-governed continuation after AutoGen's project split |
| OpenAI Agents SDK | A lightweight, code-first SDK with typed primitives — Agents, Tools, Handoffs (explicit control transfer between specialists), Guardrails, Sessions, and Tracing — avoiding hand-rolled control loops without forcing a rigid architecture |
| Anthropic's own guidance | Not a framework — published patterns (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer) meant to be implemented directly with plain API calls and standard code, deliberately favoring simple composable patterns over a dedicated abstraction |
Small mistakes compound fast in a loop
Because each step's observation feeds directly into the next decision, an agent's failure modes are mostly about how errors propagate through a sequence, not about any single step being wrong in isolation.
| Failure mode | What's actually happening |
|---|---|
| Infinite / repetitive loops | The agent retries a failing action or oscillates between a few unproductive steps, because the observation doesn't give it enough signal to recognize the approach itself is wrong |
| Tool misuse | The model invents a tool that doesn't exist, calls a real one with malformed arguments, or applies it to a task it wasn't designed for |
| Error compounding | A small mistake early in the trajectory sends the agent down an entirely different path — identical starting conditions can still produce divergent, individually-plausible-but-wrong runs |
| Context window bloat | Accumulating tool outputs and observations both cost tokens and risk truncating the original instructions or plan out of context |
| Over-planning / under-planning | Too little decomposition misses steps and leaves gaps; too much wastes tokens and latency on a task that didn't need it — early versions of Anthropic's own research system reportedly spawned excessive subagents for simple queries |
Anthropic's own mitigation for tool misuse is a design principle, not a model fix: make the tool itself harder to call incorrectly (their example — requiring absolute file paths instead of relative ones eliminated a whole category of tool-call errors in their coding-agent benchmark).
The loop that everything else plugs into
An agent is the layer that puts the LLM in motion — but it still needs somewhere to reach for tools and data (MCP, next in this series), and some way to know whether its trajectories are actually any good.
MCP (Layer 05)
Tool calling here is deliberately generic — how a tool gets registered, discovered, and reached at scale, across many different tools and providers, is what the MCP guide covers next.
Evals (Layer 07)
Basic agent quality checks look at trajectory (did it follow a sensible path), tool-call correctness, and task completion — the Evals guide covers the full methodology for turning that into a repeatable suite.
Deep dives on MCP and Guardrails are next — back to the hub to check what's published.
Start with a workflow, not an agent
- Start with the simplest thing that could work — a single well-prompted LLM call, or a fixed workflow. For a large share of applications, optimizing single calls with retrieval and good examples is genuinely enough.
- Add agentic autonomy only when the step count can't be predicted in advance — that unpredictability is the actual justification for the added latency and cost, not novelty.
- Design tools to be hard to misuse, not just documented well — constrain arguments, prefer unambiguous formats, and treat tool-call errors as a design problem before a prompting problem.
- Reach for multi-agent orchestration only for genuinely parallelizable, loosely-coupled work — the 15× token cost needs a matching quality payoff, and tightly-coupled tasks don't parallelize well regardless.
- Give every subagent explicit boundaries — an objective, an output format, and clear task scope — vague delegation is the most common cause of duplicated or gap-ridden multi-agent work.
- Log full trajectories, not just final outputs, from day one — trace-level visibility is what makes a stuck loop or a compounding error diagnosable at all.
Before you build an agent
- Confirm the task is genuinely open-ended — if you can enumerate the steps ahead of time, build a workflow instead and save the added cost and unpredictability.
- Write tool descriptions as carefully as you'd write documentation for a teammate — the model uses that description, not your intent, to decide when to call it.
- Decide upfront what gets persisted to long-term memory versus what's disposable scratch state for a single run.
- Set a hard step-count or token ceiling before shipping — a mechanism to stop a stuck loop is cheaper to build now than to debug in production.
- Plan for trace-level observability from the start — trajectory, tool-call correctness, and task completion (§07) are what the Evals guide builds a real suite around.
- Prototype single-agent first, even if you expect to need multiple agents eventually — it's the cheaper baseline to prove the task needs the complexity at all.
Success with agents isn't building the most sophisticated system — it's building the system whose complexity actually matches what the task requires.
What this guide draws on
Read in full and used to shape the structure and claims of this guide.
| Source | Publisher | Used for |
|---|---|---|
| Building Effective AI Agents | Anthropic | Workflow-vs-agent distinction, the core agent loop, composable workflow patterns, tool design, "start simple" guidance (§00, §05, §08) |
| ReAct: Synergizing Reasoning and Acting in Language Models | Yao et al., ICLR 2023 | The original interleaved reasoning + acting pattern and its benchmark results (§01) |
| Function Calling Guide | OpenAI | Tool/function calling mechanics — schemas, argument generation, the request/response loop (§01) |
| Plan-and-Execute Agents | LangChain | Plan-and-execute architecture, re-planning, ReWOO, and LLMCompiler (§02) |
| Long-term memory | LangChain Docs | Short-term vs. long-term memory architecture and the open problems in what/when to persist (§03) |
| How we built our multi-agent research system | Anthropic | Orchestrator/worker architecture, the 90.2% result, token-cost multipliers, coordination failure modes (§03, §04, §06) |
| How and when to build multi-agent systems | LangChain | When specialization beats a generalist agent, context-sharing and delegation failure modes (§04) |
| Agents SDK Guide | OpenAI | Agents SDK primitives — Agents, Tools, Handoffs, Guardrails, Sessions, Tracing (§05) |
| AI agent evaluation: trajectory, tool calls, and task completion | Langfuse | Basic agent evaluation dimensions and why a trace, not a single completion, is the right unit of observability (§07) |
Compiled 2026. Specific benchmark figures (the 90.2% and 15× numbers especially) come from the cited sources' own test setups and internal benchmarks — treat them as illustrative of direction and rough magnitude, not universal guarantees.