The Zoo of AI Agent Loops

Over the New Year holidays, while most people were recovering from celebrations, I fell into a rabbit hole. I'd been curious about AI agent loops for a while - how they actually work under the hood, which patterns hold up in practice - and the break seemed like the perfect time to finally dig in. What started as "let me read a few articles" turned into three weeks of reading around 260 articles across 35 sources.

Blog posts from companies like Anthropic, Google Research, LangChain, Cognition, and Letta. Deep technical posts from engineers like Lilian Weng, Simon Willison, Chip Huyen, and Andrej Karpathy. Research papers, benchmark analyses, production post-mortems.

This article is my attempt to synthesize what I learned. Not a survey paper, not a tutorial. Just an honest reflection on where things actually stand with agent loops in early 2026.

The autonomous agent loop: LLM decides actions, gets feedback from the environment, and repeats

The core agent loop - deceptively simple. Source: Anthropic

Part 1: What's Out There

Let's start with the basics. When people say "agent loop," they mean the core cycle an AI agent runs: perceive the world, think about what to do, take an action, observe the result, and repeat. Simple concept. But the variations on this theme are wild.

After going through everything, I've identified about 7 distinct categories that keep showing up. Let me walk through each one - what it is, when it works, and when it falls apart.

ReAct: The Default

ReAct (Reason + Act) is what most people mean when they say "agent." The idea, originally proposed by Shunyu Yao, is straightforward: the model generates a thought ("I need to find the user's order history"), then takes an action (calls a tool), gets an observation (the tool result), and loops until it's done.

HuggingFace's smolagents puts it elegantly in pseudocode:

while llm_should_continue(memory):
    action = llm_get_next_action(memory)
    observations = execute_action(action)
    memory += [action, observations]

That's it. That's the loop. And honestly, it's remarkable how far this simple pattern gets you.

The reasoning traces make the agent's behavior interpretable - you can literally read what it was "thinking" at each step. As Cameron Wolfe writes, "the ability to think during action is critically important" because the agent can adapt to what it actually finds, rather than committing to a plan upfront.

But ReAct has a brutal weakness: it's sequential. Every single step requires a full LLM call, which is slow. Cognition's team found this so painful that they built SWE-grep - a specialized model that runs up to 8 parallel tool calls in at most 4 rounds, specifically to avoid the sequential bottleneck of standard ReAct search loops.

And then there's the compound error problem, which I'll dig into more later. For now, the short version: each step has some probability of error, and those probabilities multiply. Chip Huyen's math is sobering - 95% accuracy per step drops to 60% over 10 steps and 0.6% over 100 steps.

Plan-and-Execute: Think First, Act Later

Four fundamental agent control flow patterns: sequential, parallel, conditional, loop

The four fundamental control flows. Source: Chip Huyen

Instead of reasoning one step at a time, the agent creates a full plan upfront, then executes each step. The planner and executor can even use different models - a powerful one for planning, a fast one for execution.

The BabyAGI lineage is instructive here. Yohei Nakajima's original BabyAGI used three separate agents (executor, task generator, prioritizer) in an infinite loop. But by the time he built BabyCatAGI, he'd shifted to proactive planning - a single agent creates the full plan as JSON upfront. The result? "Radically faster execution." Proactive planning beats reactive replanning.

LangChain formalized this as the Plan-and-Execute pattern, and Anthropic recommends it when "the task is well-understood" and can be decomposed clearly.

The problem? Plans go stale. Information you discover at step 3 might invalidate step 5. Cognition found that Devin "doesn't handle changing requirements mid-task" well - and that's essentially what happens when reality doesn't match the plan.

Reflection: Learning from Your Own Mistakes

In a reflection loop, the agent looks back at what it did, evaluates the result, and tries again with the benefit of that self-critique. Shunyu Yao's Reflexion hit 91% on HumanEval (a coding benchmark) through this iterative self-improvement approach.

Cognition's "closing the loop" approach is a beautiful real-world example: one agent writes code, another reviews it, the first one auto-fixes the review comments, CI validates. They describe it as "a system, not a set of tools." The downside? It "massively increased internal token spend" - but the tradeoff was worth it because bugs in PRs dropped dramatically.

The meta-lesson from Nakajima's BabyFoxAGI is interesting too - the FOXY method has agents save a "final reflection" after each task and retrieve relevant past reflections when starting new ones. It's self-improvement through accumulated experience.

But here's the catch: without strong verification, self-improvement can lead to self-destruction. Nakajima's NeurIPS 2025 survey puts it clearly: "successful self-improvement requires a balance between persistent skill representation, quality feedback, and strict verification to prevent degradation."

Tree of Thoughts: Exploring Multiple Paths

Tree of Thoughts, also from Shunyu Yao, lets the agent generate multiple possible continuations at each step, evaluate them, and pick the best one - with the ability to backtrack if a path turns out to be a dead end.

This is powerful for tasks with high uncertainty. Cameron Wolfe's analysis of reasoning models shows that models like OpenAI's o1/o3 and DeepSeek-R1 naturally exhibit "tree-search-like behavior with backtracking." Google's Deep Think uses iterative verification cycles - generating solution candidates, checking them, finding flaws, and revising. It reached Gold-medal level on the International Math Olympiad.

The downside is cost. Exploring multiple branches is exponentially expensive, and you need good heuristics to prune the tree. Even Deep Think "stabilized at ~38%" on PhD-level math problems despite massive compute.

State Machines: Structure Over Freedom

Instead of letting the LLM decide what to do next freely, you define explicit states and transitions. The LLM controls which transition to take, but the set of possible transitions is fixed.

Pydantic AI's Graph system implements this beautifully - nodes contain logic, return types define edges, union types enable conditional branching. It's a real finite state machine where type safety catches invalid transitions at compile time.

LlamaIndex makes a compelling argument for when to use structure: "predictable processes, critical business logic, error-prone decisions." And for when to let agents be free: "unstructured inputs, edge cases, novel situations."

But Dust's team warns about the failure mode: "The workflow that started as five nodes becomes 50 nodes as edge cases accumulate." Real knowledge work is non-linear, and trying to capture it in a state machine can lead to an unmaintainable mess.

Generator-Critic: Making Then Judging

Evaluator-optimizer workflow: generator creates output, evaluator provides feedback in a loop

The generator-critic loop. Source: Anthropic

One agent generates, another evaluates. Keep iterating until the critic is satisfied. Anthropic lists this as a core workflow pattern: "one LLM generates a response, another provides feedback in a loop."

Google's ADK documentation describes Generator and Critic as one of their 8 recommended multi-agent patterns. And the AI Co-Scientist system takes this to an extreme: 6 specialized agents (Generation, Reflection, Ranking, Evolution, Proximity, Meta-review) coordinate through Elo-rated scientific debate.

The problem? If your critic is unreliable, the loop doesn't converge. Langfuse found that "automated evaluators detect what regressed but poorly explain why." And Eugene Yan's analysis of LLM-as-Judge shows systematic biases: position bias, authority bias, confirmation bias. Your critic might just be confidently wrong.

Multi-Agent: Divide and Conquer

This is where things get really interesting - and really nuanced.

The idea: a central orchestrator distributes subtasks to specialized worker agents, each with their own context window and expertise. Manus's "Wide Research" is a great example - each sub-agent is "a fully capable, general-purpose Manus instance" with a dedicated cloud VM, solving the context window problem by giving each agent a fresh context.

Five agent architectures compared: single-agent, independent, centralized, decentralized, and hybrid

Five canonical multi-agent architectures - from simple to complex. Source: Google Research

But Google Research dropped what I think is the most important finding in this entire space: multi-agent is NOT always better. After testing 180 agent configurations on 4 benchmarks, they found:

Parallelizable tasks (like financial analysis): +81% improvement
Sequential tasks (like planning): -70% degradation
Independent agents amplified errors by 17.2x; centralized systems reduced this to 4.4x

They even built a predictive model with 87% accuracy for choosing between single-agent and multi-agent architectures. The message is clear: think about your task structure before reaching for multi-agent.

And as Anthropic notes, agentic systems "trade latency and cost for better task performance" - you should only add complexity when it demonstrably improves outcomes.

The Hybrid Reality

What struck me most across all this reading is that the best practitioners don't pick one loop type. They compose them.

Harrison Chase describes this as a spectrum: hardcoded logic → single LLM call → chain of LLM calls → router → state machine → autonomous agent. The right answer isn't to go to the far right of this spectrum - it's to go exactly as far right as your problem requires, and not one step further.

Sierra uses 15+ different models - frontier, open-weight, and proprietary - each matched to a different subtask. Their insight: "you can't build a good agent from a single building block."

And Nakajima makes a counterintuitive point: "the most valuable tasks for automation are expensive and frequent, so you don't want to dynamically generate a task list - you want a hardcoded agent." Sometimes the best loop is no loop at all.

Part 2: The Hard Walls Nobody Has Climbed

Now for the uncomfortable part. There are fundamental problems in agent loops that remain unsolved - not "we need more engineering" unsolved, but "we don't even know if the current paradigm can solve this" unsolved.

Compound Errors: The Math Doesn't Lie

This is the elephant in the room. Chip Huyen laid out the arithmetic:

95% accuracy per step → 60% over 10 steps → 0.6% over 100 steps

Think about that. Even if each individual step is quite good, a 100-step task is essentially guaranteed to fail. And real-world agent tasks often involve dozens to hundreds of steps.

Every solution to this - reflection, verification loops, critic agents - adds more steps, each with their own error probability. You're fighting fire with fire. This is one of the fundamental problems of agents, and it doesn't have a clean answer.

Even Google's Deep Think, with all of DeepMind's resources, "stabilized at ~38%" on PhD-level math despite massive iteration. More steps don't always help.

Context Rot: The Slow Poison

As an agent works, its context window fills up with previous actions, observations, tool outputs, and error messages. The more it works, the worse it gets. This is context rot, and it's insidious.

Peak Ji at Manus observed that "models start fabricating data around the 8th-9th element" when processing items sequentially. Bigger context windows don't help because of the "lost in the middle" problem - models attend poorly to information in the middle of long contexts.

Cognition's team discovered something fascinating they call "context anxiety": when approaching the context window limit, the model starts proactively summarizing its work - but the summaries aren't complete enough, and this premature compression causes information loss. They tried "enabling 1M token beta but capping usage at 200k" as a workaround.

The current best approaches all feel like patches, not solutions:

Manus uses the filesystem as extended memory - a constantly-updated todo.md file pushes the global plan into the model's "recent attention zone"
Jason Liu argues that grep beats embeddings for many retrieval tasks - simpler tools can be more reliable than complex ones
Anthropic's Claude Code uses compaction - summarizing old context when the window fills up
Multi-agent systems give each agent a fresh context window, essentially sidestepping the problem (Manus Wide Research)

But none of these truly solve the fundamental issue: long-running agents gradually lose coherence.

Memory: The Unsolved Frontier

How should an agent remember things? What should it keep? What should it forget? When should it update?

The Berkeley Function Calling Leaderboard (BFCL V4) tested three memory architectures across 5 domains:

Recursive Summarization: best overall at 67.74%, but loses details
Vector Store: up to 63.87%, good for semantic search, but hallucination-prone
Key-Value Store: up to 53.55%, precise but inflexible

None of them cracked 70%. And there's a devastating failure mode: "aggressive deletion" - models delete previously stored information when new "urgent" data arrives, destroying long-term coherence.

Letta (formerly MemGPT) has the most ambitious approach - an OS-inspired memory hierarchy with core memory, recall memory, and archival storage, plus "sleep-time compute" where agents consolidate memories during idle time. Shunyu Yao's CoALA framework proposes separating procedural, semantic, and episodic memory. These are promising directions, but they're still experimental.

Dust made a deliberately conservative choice: user-scoped memory, full transparency (all memories inspectable by users), and opt-in activation. They specifically decided that NOT all agents should have memory - a code review agent without memory applies "consistent standards across all PRs," while one with memory might develop biases.

Ambiguity: The Kryptonite

Agents are shockingly bad at handling ambiguous instructions.

The OpenEnv evaluation from Meta and Hugging Face found that agents hit ~90% success with explicit identifiers but crashed to ~40% with natural language descriptions of the same tasks. That's a 50-percentage-point drop just from making instructions less precise.

Cognition's annual review of Devin says it directly: "like most junior engineers, [Devin] does best with clear requirements." Ambiguous specs, changing requirements, the need to ask clarifying questions - these remain beyond current agent capabilities.

Karpathy calls it "jagged intelligence" - a model brilliantly solves a PhD-level problem, then fails on something trivial. You can never quite predict where the gaps will be.

Reliability at Scale: The 1-in-10,000 Problem

Sierra put it starkly:

At scale, a one-in-ten-thousand hallucination rate is a daily occurrence.

Enterprise needs ~99.9% reliability (E2B). Current agents are nowhere close. Sierra's approach is layers of defense - supervisor agents ("Jiminy Crickets"), input filtering, output validation, post-conversation analysis - but this is essentially accepting the problem and building guardrails around it, not solving it.

Format Sensitivity: Fragile by Nature

This one surprised me. BFCL V4's prompt variation study tested 200 tasks across 26 format variations - things like changing JSON to XML, switching documentation formats, rephrasing. Some models were mostly stable. Others completely broke: "watt-tool-70B outputs Python instead of JSON, CoALM-70B drops to near-zero with certain tags."

Minor formatting changes causing catastrophic failures. For production systems, this is terrifying. Manus addresses this through "structured variability" - deliberately varying action patterns to prevent the agent from getting stuck in repetitive loops. A creative workaround, but the underlying fragility remains.

Part 3: What Actually Works (and What Doesn't)

Theory is one thing. What happens when you deploy these loops on real tasks?

Where Agents Shine

Coding with clear specs. This is the sweet spot. Devin hit a 67% merge rate (up from 34% a year ago) and runs 4x faster than human developers for tasks with clear requirements. Reflexion achieved 91% on HumanEval through iterative self-correction. HuggingFace showed that code agents significantly outperform JSON-based tool calling - code gives you composability, loops, conditionals, all the things that make complex actions expressible.

Parallelizable analysis. Google's 180-configuration study found an 81% improvement on tasks like financial analysis where work can be split across independent agents. Dust's @research agent cites 14 sources versus 5 for standard models. An open-source deep research agent produced comparable results to OpenAI's for $0.71 and 3.5 minutes.

Document processing with known structure. LlamaIndex's Agentic Document Workflows - Parse, Retrieve, Reason, Act - works well when the pipeline stages are well-defined. Dust found that agents "spontaneously invented syntax for navigating data," creating file:path/ notation on their own.

Code review and auto-fix loops. Cognition's write → review → autofix cycle "radically reduced bugs in PRs." The key insight: this is a closed loop with a clear quality signal (review comments). When the agent can verify its own output, everything works better.

Customer support (bounded decision spaces). Sierra uses 15+ models, matching the right one to each subtask. Nakajima is right: "the most valuable tasks for automation are expensive and frequent, so you want a hardcoded agent." Customer support with known policies is exactly this - high volume, clear rules, bounded outcomes.

Where Agents Struggle

Sequential tasks with dependencies. That Google Research number bears repeating: -70% degradation on planning tasks with multi-agent systems. Independent agents amplified errors by 17.2x. Chip Huyen's compound error math explains why: 100 sequential steps at 95% accuracy = 0.6% chance of getting everything right.

Vague or changing requirements. Devin "does best with clear requirements" and "can't do mentoring or stakeholder management." The OpenEnv benchmark showed a crash from ~90% to ~40% just from making instructions less explicit.

Tasks requiring long-term coherence. Manus found models "start fabricating data around the 8th-9th element." BFCL V4's memory benchmarks top out at 67.74% - nearly a third of memory operations fail. Models "aggressively delete" previously stored information when new data arrives.

Tasks requiring implicit prerequisites. BFCL V3 found three critical failure patterns: models can't recognize implicit prerequisites (checking fuel level before refueling), have poor awareness of current system state, and unnecessarily overcomplicate things (re-authenticating an already-authorized API).

GUI and browser automation. E2B's computer-use agent is compared by its creator to "a 5-6 year old child" - it struggles with multi-step planning, focus, and recognizing UI states. This is still early days.

Part 4: How Do We Even Measure This?

The metrics landscape for agent loops is, frankly, a mess. But some useful frameworks are emerging.

pass@k vs pass^k: The Reliability Distinction

Anthropic makes a crucial distinction:

pass@k: probability that at least 1 of k attempts succeeds. This measures the "ceiling" - how good can the agent get?
pass^k: probability that k consecutive attempts all succeed. This measures reliability - how dependable is it?

A system with pass@5 = 90% and pass^5 = 30% is an agent that can do the job but often doesn't. For production, pass^k matters far more.

SWE-Bench: The Coding Standard

SWE-Bench has become the standard for coding agents - real GitHub issues that need real fixes. But LangChain's experience reveals a troubling truth: they jumped from 30th to 5th place on Terminal-Bench just by changing the harness (the system around the model), not the model itself.

This means: what you're measuring is the harness as much as the agent. Two identical models with different prompt templates, tool configurations, and retry logic will score very differently. Keep this in mind when reading any benchmark results.

BFCL: Function Calling Under the Microscope

The Berkeley Function Calling Leaderboard is probably the most rigorous evaluation of tool-use specifically. It evolved from V1 (2000 question-function pairs) through V2 (live dataset to prevent contamination) and V3 (multi-turn scenarios) to V4 (agentic tasks: web search, memory management, prompt sensitivity).

Key finding from BFCL: even leading models have "substantial gaps in agentic reasoning when using tools." Open-source models match frontier models on simple function calls but fall far behind on complex parallel scenarios.

The Harness Problem

This deserves its own section because it's so important. Harrison Chase at LangChain put it bluntly: "harness engineering matters more than model selection." The system design around the model - prompt templates, tool definitions, retry logic, context management - can make more difference than which model you use.

Cognition took this to heart: their SWE-1.5 model, the agent harness (Cascade framework), and the inference system were all "optimized simultaneously, not in isolation." Training used end-to-end RL in real coding environments with three types of grading: unit tests, human quality rubrics, and browser-based agent testing.

This means any benchmark comparison between agents is comparing entire systems, not just loop architectures. And most benchmark papers don't control for this.

Business Metrics: What Actually Matters

For real-world deployment, some simpler metrics turn out to be more useful than any benchmark:

Merge rate (Cognition): Devin's 67% merge rate is a direct measure of "does this code actually get used?"
Bounded error rates (Sierra): Instead of aiming for perfection, define acceptable error rates by severity tier
Tokens per task: Agentic systems trade latency and cost for performance (Anthropic). Is the quality improvement worth the cost increase?
Time to completion: Cognition's SWE-1.5 reduced Kubernetes editing from ~20s to less than 5s - entering the "flow window" where developers don't lose context

Hamel Husain argues that without your own evaluations tied to your specific business metrics, all public benchmarks are essentially useless. And Langfuse frames evaluation as "a journey, not a destination" - you need ongoing, iterative measurement, not one-time benchmarking.

Part 5: What I Actually Took Away From All This

After reading 260 articles, here's what I think actually matters:

1. More autonomy is not always better

This was the single biggest surprise. I went in assuming the field was marching steadily toward more autonomous agents. Instead, the most successful practitioners are going the other way - toward carefully calibrated autonomy.

Harrison Chase's cognitive architecture spectrum is the best mental model I found: go exactly as far toward autonomy as your problem requires, and not one step further. Anthropic says it too: "start with workflows, graduate to agents only when needed."

Google's finding that multi-agent systems degraded performance by 70% on sequential tasks should give everyone pause. More agents, more loops, more autonomy can make things worse.

2. Context engineering > prompt engineering

The industry has shifted from "how do I write a better prompt?" to "how do I control what information is in the context window at each step?" Anthropic calls this "context engineering." Manus devotes an entire blog post to 6 practical principles, including KV-cache optimization, attention management through "retelling" (constantly updating a todo.md to push the plan into recent attention), and keeping errors in context.

LlamaIndex identified 9 elements of agent context: system prompts, user input, short-term memory, long-term memory, knowledge base, tool definitions, tool responses, structured outputs, and global state. Getting each of these right - and in the right combination for each step - is the real engineering challenge.

3. The harness matters more than the model

LangChain's jump from 30th to 5th place by changing only the harness is the most striking data point here. Cognition co-optimizes model, harness, and inference together. The agent loop itself is just one piece of a much larger system.

This means: if you're building an agent and it's not working well, don't immediately reach for a better model. Look at your system prompts, tool definitions, error handling, context management, and retry logic first.

4. The feedback loop is everything

Dust argues that "the missing piece isn't smarter models" but closing the feedback loop between agent performance and configuration updates. Agents should analyze their conversations, generate improvement suggestions, and update their instructions - with human review.

Cognition's write-review-autofix loop shows the same principle: the tighter and faster your feedback cycle, the better the results. HuggingFace recommends that every tool should log everything useful through print() - "how easy would it be for me, if I was dumb and using this tool for the first time?" Better information flow to the LLM trumps better prompting.

5. Simple tools often beat complex ones

LlamaIndex's benchmark showed that filesystem tools beat RAG on correctness (8.4 vs 6.4) and relevance (9.6 vs 8.0), though RAG was faster. Jason Liu argues grep beats embeddings for many use cases. Jerry Liu says "files are all you need" - agents with basic file system tools plus a code interpreter can dynamically navigate context of any complexity.

The lesson: don't over-engineer your tool stack. Start with simple, reliable tools and add complexity only when the data proves you need it.

6. We're still in the "it depends" era

There is no best agent loop. There is no best architecture. Google proved that the optimal architecture depends on task characteristics. Berkeley's Agent Arena found that the best agent "depends heavily on the domain." Sierra uses 15+ models because "models that are good at reasoning significantly degrade when asked for a quick response."

This is frustrating if you want a simple answer. But it's also honest. The field is too young and the problem space is too varied for universal solutions.

What's Next

If I had to bet on what will matter most in the next 12 months, based on everything I've read:

Context engineering will become a discipline. Not just prompt engineering with a fancy name, but a systematic approach to managing what information enters the LLM context window at each step. Manus, Anthropic, and LlamaIndex are all converging on this.

Memory architectures will get serious. Letta's sleep-time compute, Berkeley's memory benchmarks, Dust's opt-in approach - memory is moving from afterthought to first-class architectural concern.

Specialized fast models will proliferate. Cognition's SWE-1.5 at 950 tok/s and SWE-grep at 2800+ tok/s show that task-specific models, co-optimized with their harnesses, dramatically outperform general-purpose models on specific subtasks.

Evaluation will remain the bottleneck. Hamel Husain is right: without good evals, you're flying blind. Langfuse showed that "averages hide outliers" and automated evaluators detect the "what" but not the "why." We need better evaluation tools, methods, and practices.

But maybe the most important takeaway is the one Nakajima articulates: the future isn't fully autonomous agents doing everything. It's a progression - AI assistant → AI reviewer → full automation with human oversight - where each step earns trust through demonstrated reliability.

We're somewhere between step one and step two. And that's okay.

Sources

This article is based on research across 35 sources (20 companies/teams, 15 individual engineers). Here's a selection of the most cited sources: