Agentic loop deep-dive

In the A4 flagship journey you built your first agent — a ReAct loop with tools. It worked. You got confident and gave it harder tasks. Sometimes it succeeded brilliantly. Sometimes it went in circles for 20 iterations and gave up. Sometimes it confidently returned a wrong answer.

You searched for "agent architectures" and drowned in acronyms: ReAct, Reflexion, ToT, GoT, LATS, MRKL, HuggingGPT, AutoGPT, BabyAGI, CAMEL, MetaGPT, ChatDev. Papers keep publishing new ones. You started to feel that agents are more art than engineering.

They're not. 95% of production agents in 2025 are one of four canonical loops. The other 200+ named variants are decorations. Once you understand these four — and the specific failure mode each one addresses — you can defend every agent-architecture decision with math instead of hype.

The 4 canonical loops — the only diagram you need

Rule: start at Loop 1 (ReAct). Only climb the ladder when you have EVIDENCE the simpler loop fails a specific class of task. Never pick Loop 4 (multi-agent) because "it seems cool" — the coordination cost is real and empirical results are mixed.

Let me walk each loop — the mechanism, the failure mode it addresses, and the cost/quality math.

Loop 1: ReAct — the baseline every agent starts here

Paper: Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (Oct 2022, arxiv.org/abs/2210.03629).

Mechanism: three-step loop. Thought (LLM plans) → Action (calls a tool) → Observation (sees the result) → repeat until Final Answer.

Powers: Cursor, Claude Code, GitHub Copilot Workspace, ChatGPT's browsing mode, Perplexity's answer engine. If you look at any 2025 agent product and squint, you see ReAct.

When it wins:

• Task is inherently sequential (read a file → search for a term → analyze → answer)
• Each step provides useful information for the next
• Task is well-scoped (max ~10-20 iterations is enough)
• Cost sensitivity is high (this is the cheapest loop)

When it fails:

• Task requires backtracking — realizing step 3 was wrong and undoing it
• Task requires exploration — trying multiple approaches in parallel
• Task requires verification — checking if the answer is actually right before returning

Cost model: each iteration = 1 LLM call + 1 tool call. A 5-iteration task costs 5 × input tokens (growing linearly as context accumulates) + 5 × output tokens + 5 × tool cost. Typical 5-step ReAct on GPT-4o: $0.05-0.20 per task.

Loop 2: Reflexion — ReAct with self-critique

Paper: Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning" (Mar 2023, arxiv.org/abs/2303.11366).

Mechanism: run ReAct to completion. Then evaluate the output (either automatically — did tests pass? does the answer parse? — or via an LLM judge). If failed: reflect verbally on the trajectory (what went wrong?), add the reflection to the next attempt's context, run ReAct again.

text
    ══════════ REFLEXION LOOP ══════════

    Attempt 1: ReAct trajectory → Answer
                          ↓
    Evaluator: PASS?
       ↓ FAIL
    Reflection: "I chose the wrong file to read in step 2.
                 Next time, search by symbol name first."
                          ↓
    Attempt 2: ReAct with reflection prepended
                          ↓
    Evaluator: PASS?
       ↓ FAIL after N attempts → return best attempt
       ↓ PASS → return answer

When it wins:

• Task has a verifiable success criterion (tests pass, code compiles, output matches schema, judge rates ≥4/5)
• The failure signal is informative (you can tell WHY it failed, not just THAT it failed)
• Cost budget allows 3-5 attempts

When it fails:

• No cheap verifier exists (subjective quality tasks with no ground truth)
• Failures are catastrophic (agent broke prod on attempt 1 — no time for reflection)
• Reflections don't correlate with actual fixes (model is bad at introspection on this task)

Cost model: ~3-5× ReAct cost. A task that succeeds in 3 Reflexion attempts costs ~3x. A task that always succeeds first try is exactly ReAct cost.

Empirical results (from the paper): on HumanEval (Python coding), Reflexion boosted GPT-4 from 80% → 91% pass rate. On ALFWorld (agent benchmark), 75% → 97%.

Loop 3: Tree-of-Thought (ToT) — parallel exploration with pruning

Paper: Yao et al., "Tree of Thoughts: Deliberate Problem Solving with LLMs" (May 2023, arxiv.org/abs/2305.10601).

Mechanism: at each step, generate K candidate thoughts instead of one. Evaluate each candidate (LLM as evaluator, or task-specific heuristic). Keep the top-B branches ("beam width"). Expand each surviving branch by another K candidates. Repeat to depth D. Return the best terminal leaf.

When it wins:

• Task requires exploration (planning, math word problems, logic puzzles, creative writing where multiple angles help)
• There's a good evaluator for intermediate states (not just the final answer)
• Cost budget tolerates 10-30x baseline

When it fails:

• Linear tasks — exploring branches wastes cost with no benefit
• No good intermediate evaluator — you can't prune reliably, so exploration blows up cost
• Task doesn't decompose into thought-sized steps (whole-thing-at-once tasks)

Cost model: with branching factor K=3, beam width B=3, depth D=4 → ~3^4 = 81 LLM calls worst case, ~3×4=12 with aggressive pruning. Typical ToT run costs $1-5 vs ReAct's $0.10.

Empirical results (from paper): on Game of 24, ToT solved 74% vs GPT-4 CoT's 4%. Massive lift on tasks where exploration is essential.

Note: LATS (Language Agent Tree Search, Zhou et al. 2023) generalizes ToT with Monte Carlo Tree Search. Even more expensive but strong on longer tasks.

Loop 4: Multi-agent — coordinator + specialists

Papers: many — CAMEL (Li et al. 2023), MetaGPT (Hong et al. 2023), AutoGen (Wu et al. 2023), Anthropic's "Building Effective Agents" (Dec 2024). Each proposes slightly different orchestration patterns.

Two dominant shapes:

Shape A — Orchestrator + workers (Anthropic's default recommendation):

text
    ══════════ ORCHESTRATOR + WORKERS ══════════

    User task
       ↓
    Orchestrator agent (planning + delegation)
       ↓
    Decomposes into sub-tasks
       ↓
    Dispatches to specialist workers:
      - Research worker (tools: web search, RAG)
      - Code worker (tools: file edit, tests, git)
      - Analysis worker (tools: python REPL, plotting)
       ↓
    Each worker returns result (or requests clarification)
       ↓
    Orchestrator synthesizes final answer

Shape B — Debate / peer-to-peer:

text
    ══════════ PEER DEBATE ══════════

    Task → Two or more agents with different prompts / models
       ↓
    Each proposes an answer
       ↓
    Each critiques others answers
       ↓
    Round 2: agents revise given critiques
       ↓
    Convergence: winner picked by judge OR final round consensus

When multi-agent wins:

• Task requires genuinely different capabilities (code + docs + data analysis) that don't compose in one prompt
• Coordination is cheap relative to specialization gain
• Task benefits from debate (safety-critical or high-stakes decisions)

When multi-agent fails:

• Agents pass excessive context back and forth (blows cost + latency)
• Coordinator drifts (loses track of overall goal across delegations)
• The task could have been one bigger ReAct with more tools

Anthropic's guidance (from "Building Effective Agents" Dec 2024): prefer single agent with more tools over multi-agent orchestration. Multi-agent shines only when the specialization gain > coordination cost, which is rarer than it seems.

Cost model: 2-10x baseline depending on message-passing depth. Cursor's multi-agent mode with 3 agents runs ~4x baseline cost for a comparable task.

The evolution ladder — how a real product picks

The right move for most teams: sit at V2 (ReAct + selective Reflexion) for a long time. Only climb to V3/V4 when specific task classes fail and cheaper interventions (better tools, better prompts) don't fix them.

The 4 loops compared — quick table

The one that trips everyone up: multi-agent quality can go NEGATIVE compared to a single agent with more tools. Debate loops sometimes converge on wrong-but-agreed answers. Orchestrators sometimes lose the plot. This is why Anthropic explicitly recommends "single agent + more tools" as the default and multi-agent as the exception.

The most common mistakes

1. "We built multi-agent from day 1 because it seemed powerful." You spent 3 months on orchestration. A single ReAct with the same tools would have been shipped in a week and probably scored higher. Start simple.

2. "We use Reflexion but our evaluator is another LLM call with no ground truth." Reflexion needs a RELIABLE verifier. If the verifier itself is noisy, Reflexion amplifies errors.

3. "We use Tree-of-Thought on every query." ToT costs 10-30x. Use it only on queries where exploration is essential (planning, math). Route by intent.

4. "Our agent went in circles for 20 iterations." Add a MAX_ITERATIONS safety limit + a loop-detection heuristic (same tool + same args twice = fail fast). ReAct without a stopping condition is a bill generator.

5. "We chose the loop before we chose the eval." Backwards. Define the eval — success rate, cost budget, latency budget — THEN pick the loop that hits those numbers. Otherwise you're guessing.

The L4 → L7 agent architecture ladder

• L4 (starter): ReAct single-agent, 5-10 hand-picked tools, max_iterations=10. Ships fast. Handles 60-70% of production tasks well.

• L5 (production): ReAct + task-specific Reflexion (only re-runs on verifiable failures) + strong tool design + observability (every tool call logged). Standard for consumer AI products.

• L6 (advanced): router at the entry point classifies tasks; simple tasks go to ReAct, hard reasoning tasks go to Tree-of-Thought, verification-heavy tasks go to Reflexion. Per-task cost budget. Cursor, Devin operate here.

• L7 (frontier): learned agent policies (RL-trained), multi-agent for genuine capability specialization, self-modifying prompts, continuous fine-tuning from user feedback. Anthropic's Claude Code and Google's project Astra live here.

What's next in this journey:

• Chapter 1: ReAct implementation from scratch — max_iterations, loop detection, state persistence, tool result formatting
• Chapter 2: Reflexion in production — evaluators, reflection prompts, cost gating
• Chapter 3: Tree-of-Thought implementation — beam search, evaluator design, when it beats CoT
• Chapter 4: Multi-agent patterns — LangGraph state machines, Anthropic's Building Effective Agents recipes
• Chapter 5: Agent evaluation — success rate, cost per task, latency, tool-call efficiency
• Chapter 6: Safety envelopes — human-in-loop, permission systems, sandboxed execution

Sources cited in this chapter:

• Yao et al. "ReAct: Synergizing Reasoning and Acting" (Oct 2022): arxiv.org/abs/2210.03629
• Shinn et al. "Reflexion: Language Agents with Verbal Reinforcement Learning" (Mar 2023): arxiv.org/abs/2303.11366
• Yao et al. "Tree of Thoughts" (May 2023): arxiv.org/abs/2305.10601
• Zhou et al. "Language Agent Tree Search (LATS)" (Oct 2023): arxiv.org/abs/2310.04406
• Li et al. "CAMEL: Communicative Agents" (Mar 2023): arxiv.org/abs/2303.17760
• Hong et al. "MetaGPT" (Aug 2023): arxiv.org/abs/2308.00352
• Wu et al. "AutoGen" (Aug 2023): arxiv.org/abs/2308.08155
• Anthropic "Building Effective Agents" (Dec 2024): anthropic.com/research/building-effective-agents — must-read
• LangGraph (production agent framework): langchain-ai.github.io/langgraph
• Cognition AI (Devin) engineering blog: cognition.ai/blog