Agentic Reinforcement Learning: Training Models to Act, Not Just Answer

In the RLHF that aligned the first ChatGPT, the model did exactly one thing: it read a prompt and wrote a response. A reward model scored that response, and the policy was nudged to produce higher-scoring text next time. The whole interaction was a single step. There was no world to change, no tool to call, no consequence that outlived the reply. Reinforcement learning was, in a strict technical sense, barely reinforcement learning at all; it was a bandit problem wearing an RL costume.

The agent era broke that frame. When a model spends forty minutes editing a codebase, running the test suite, reading the failures, and patching again, the thing being optimized is no longer a paragraph. It is a trajectory: a sequence of actions taken against an environment that answers back, where the reward, "did the tests pass," arrives only at the very end and says nothing about which of the forty actions deserved credit. The 2025 survey that named this shift, Wang et al., 2025, The Landscape of Agentic Reinforcement Learning for LLMs, arXiv:2509.02547, put the distinction precisely: conventional LLM-RL optimizes a degenerate single-step Markov decision process, while agentic RL optimizes a temporally extended, partially observable one. Everything hard about the second sentence follows from that upgrade.

Why this matters: The models topping SWE-bench and running hours-long research tasks were not prompted into that behavior; they were trained into it with RL against real environments. If you want to understand why 2026's best agents are so much more reliable than 2024's prompted ReAct loops, the answer is mostly here, in how the reward reaches back across a hundred actions.

TL;DR

• Agentic RL reframes the LLM from a text generator solving a single-step problem into a policy acting in a multi-turn, partially observable environment, where actions are tool calls and observations are the environment's responses.
• The defining difficulty is credit assignment over long horizons: a single terminal reward ("task solved") must be distributed across dozens or hundreds of actions, most of which were neither clearly right nor clearly wrong.
• RLVR (RL from verifiable rewards) is the bridge: it replaced the learned reward model with a checkable outcome (unit tests pass, the answer matches), which is what makes multi-turn agentic RL trainable without a fragile reward model in the loop.
• GRPO and its sequence-level successors dropped the value critic and estimate advantage by comparing a group of sampled trajectories, which suits agents because training a good critic over 100-turn episodes is nearly impossible.
• The environment (the harness) is now a first-class part of the model. The same weights score very differently depending on tool design, observation formatting, and retry affordances; harness and policy are co-designed.
• Reliability, not peak capability, is the live metric. τ-bench's pass^k shows agents that solve a task once often fail to solve it eight times in a row, and closing that gap is much of what agentic RL buys.
• The failure modes are distinctive: reward hacking against checkable rewards, reward sparsity over long horizons, and non-stationary environments that shift under the policy as it learns.

At a Glance

The whole method is one loop wrapped around a language model. The policy emits an action, the environment returns an observation and eventually a reward, and an RL update pushes the policy toward trajectories that scored well.

flowchart LR
  P[Policy<br/>LLM weights] -->|action / tool call| E[Environment<br/>code, browser, API]
  E -->|observation| P
  E -->|terminal reward| R[Reward + advantage<br/>estimation]
  R -->|gradient update| P
  class P purple
  class E blue
  class R emerald
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

The loop looks trivial. The engineering is not, because the "action" is a token sequence sampled from a 100-billion-parameter model, the "environment" is a real code sandbox or web browser, and the "reward" is one bit at the end of a trajectory that may be thousands of tokens long.

[IMAGE: Side-by-side comparison. Left panel "RLHF (single step)": prompt to response to scalar reward, one arrow each. Right panel "Agentic RL (multi-turn)": policy looping with an environment box across turns 1..N, reward attached only at turn N. Emphasize the horizon length difference.]

Before Agentic RL

Reinforcement learning met language models through alignment. Ouyang et al., 2022, Training language models to follow instructions with human feedback, arXiv:2203.02155 trained a reward model on human preference comparisons, then optimized the policy against it with PPO (Schulman et al., 2017, Proximal Policy Optimization Algorithms, arXiv:1707.06347). This worked, and it defined the field's mental model of "RL on LLMs" for two years: one prompt, one response, one score. The reward was a learned approximation of human taste, and the horizon was one.

Two threads pulled the field toward agents. The first was tool use. Schick et al., 2023, Toolformer, arXiv:2302.04761 showed a model could learn to call APIs mid-generation, and Yao et al., 2022, ReAct, arXiv:2210.03629 interleaved reasoning traces with actions so a model could plan, act, observe, and revise. These were mostly prompted or supervised behaviors, not RL-trained, but they established the action-observation loop that agentic RL would later optimize.

The second thread was verifiable reward. The reward-model-in-the-loop of RLHF is expensive and hackable. Shao et al., 2024, DeepSeekMath, arXiv:2402.03300 introduced GRPO and, more importantly, popularized rewarding a model for getting a checkable answer right rather than for pleasing a preference model. DeepSeek-AI, 2025, DeepSeek-R1, arXiv:2501.12948 then showed that large-scale RL against verifiable math and code rewards could induce long chains of reasoning with almost no supervised warm-up. That result reset expectations: RL was not just for polishing tone, it could teach genuinely new problem-solving behavior.

Agentic RL is the merger of those two threads. Take the verifiable-reward machinery that made R1 work and point it at the action-observation loop that ReAct sketched, and you get a model trained to succeed at tasks it can only complete by acting over many turns.

timeline
  title From Alignment to Agentic RL
  2017 : PPO stabilizes policy-gradient RL
  2022 : RLHF (InstructGPT) : ReAct interleaves reason and act
  2023 : Toolformer learns API calls
  2024 : GRPO drops the critic : verifiable rewards gain ground
  2025 : DeepSeek-R1 scales RLVR : agentic RL named and surveyed
  2026 : multi-turn tool-agent RL goes mainstream

How Agentic RL Actually Works

The problem shape: from MDP to POMDP

Start with the formal object, because the whole difficulty is encoded in it. Single-turn RLHF is a Markov decision process that has collapsed to one step: state \(s\) is the prompt, action \(a\) is the full response, the episode ends, and the return is just the immediate reward \(r(s,a)\). There is no transition to a next state because there is no next state.

Agentic RL restores the full structure and adds one twist. The agent now faces a partially observable MDP, a POMDP, defined by a horizon \(T\) of many turns. At turn \(t\) the policy \(\pi_\theta\) sees an observation \(o_t\) (the current context: tool outputs, error messages, retrieved documents), emits an action \(a_t\) (a tool call or a message), and the environment transitions and returns the next observation. The objective is the familiar expected return, but now the sum runs over the whole trajectory:

\[J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t=0}^{T} \gamma^t \, r(s_t, a_t)\right]\]

The word "partially observable" is doing real work. The agent never sees the true environment state, the actual contents of a database, the real state of a filesystem, the ground truth of a web page it has not loaded. It sees only what its tools have surfaced so far. So the policy must act under uncertainty and, implicitly, decide when to gather more information versus when to commit. That is a qualitatively harder learning problem than mapping a prompt to a good paragraph.

[IMAGE: Two MDP graphs side by side. Left: a single-step MDP with one state node, one action edge, one terminal reward (labelled "RLHF / RLVR single-turn"). Right: a POMDP chain with hidden true-state nodes on top, observation nodes below them, actions linking observations, and a reward only on the final node (labelled "Agentic RL"). Dashed lines from hidden states to observations show partial observability.]

Verifiable rewards make the loop trainable

The reason agentic RL took off in 2025 rather than 2020 is the reward. A learned reward model over a 100-turn trajectory is a nightmare: it has to judge a long, branching interaction, and any systematic error in its judgment becomes a target the policy will exploit. RL from verifiable rewards sidesteps this. Instead of asking a model "was this good," the environment runs a check: do the unit tests pass, does the final SQL return the expected rows, did the database reach the correct end state. The reward is a program, not a prediction.

This is why coding and math were the beachhead. They come with free, exact verifiers. The τ-bench work (Yao et al., 2024, τ-bench, arXiv:2406.12045) generalized the idea to tool-agent tasks by defining success as reaching the correct database end state after a customer-service interaction, which is checkable even though the dialogue is open-ended. Verifiability is the constraint that shapes which agentic tasks are trainable today: where you can write a checker, you can train; where you cannot, you are back to a fragile learned reward.

Credit assignment: the central difficulty

Here is the crux. A trajectory of forty actions ends in a single bit: solved or not. Which actions caused the outcome? Classic RL answers this with a value function (a critic) that estimates the expected return from each state, so you can compute a per-step advantage. But training a reliable critic over long, high-variance LLM trajectories is brutally hard, and a bad critic injects bias into every update.

The dominant answer in 2025 and 2026 was to stop estimating a critic and instead compare whole trajectories. GRPO samples a group of \(G\) trajectories for the same task, scores each with the verifiable reward, and computes each trajectory's advantage by normalizing against the group:

\[\hat{A}_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}\]

Every token in a better-than-average trajectory gets a positive push; every token in a worse-than-average one gets a negative push. This is beautifully simple and it dodges the critic entirely, which is why it dominates agentic training. But look closely at what it does over a long horizon: it assigns the same advantage to every action in the trajectory. A single brilliant tool call and a dozen pointless ones in a winning trajectory all receive identical credit. Over short reasoning chains this coarseness is tolerable. Over 100-turn agent episodes it becomes the core bottleneck, and it is the problem the 2026 literature is largely organized around (Cui et al., 2026, Rethinking Agentic Reinforcement Learning in Large Language Models, arXiv:2604.27859).

Two families of fixes are emerging. Process rewards attach a signal to intermediate steps, rewarding a correct sub-goal or a successful tool call rather than only the final outcome, which densifies the sparse terminal reward. Turn-level and hierarchical advantage methods keep the group-relative trick but compute advantages at a finer grain than the whole trajectory, grouping by turn or by sub-task so that credit is at least locally targeted. Both trade simplicity for a sharper learning signal, and neither has fully won.

[IMAGE: A single trajectory drawn as a horizontal chain of 8 action nodes ending in a green "reward = 1" flag. Three overlaid credit schemes: (a) trajectory-level, one flat bar of identical height over all 8 nodes; (b) process reward, small green ticks on the 3 nodes that hit sub-goals; (c) turn-level advantage, bars of varying height per node. Caption contrasts "same credit everywhere" with "targeted credit."]

The environment is part of the model

A subtle lesson of the agent era is that the harness, the scaffolding of tools, prompts, memory, and control flow around the raw model, is not a fixed backdrop; it is co-optimized with the weights. The same policy scores very differently under different harnesses, because the harness determines what actions are even expressible, how observations are formatted, and how many retries the agent gets before the episode ends. Training a policy against one harness and deploying it under another can erase most of the gains. This coupling means agentic RL is really training a policy-plus-harness system, and the environment engineering (tool schemas, observation truncation, error message design) is as consequential as the algorithm.

graph TD
  subgraph Harness
    T[Tool schemas]
    M[Memory / context manager]
    C[Control loop + retries]
  end
  POL[Policy<br/>LLM] -->|samples action| C
  C -->|dispatch| T
  T -->|raw result| M
  M -->|formatted observation| POL
  V[Verifier<br/>tests / end-state check] -->|reward| RL[RL updater]
  C -->|trajectory| V
  RL -->|gradient| POL
  class POL purple
  class T,M,C blue
  class V emerald
  class RL amber
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

[IMAGE: Diagram of a "rollout" showing token generation interleaved with tool execution. Highlight the asynchronous gap where the GPU sits idle waiting for a slow tool (a test suite, a web request) to return, annotated "the throughput problem in agentic RL."]

The systems problem: rollouts are slow

One reason agentic RL is a distinct engineering discipline is throughput. In single-turn RL, generating a rollout is one forward pass. In agentic RL, a rollout means repeatedly generating tokens, pausing to execute a real tool (which might take seconds), reading the result, and generating again. The GPU that holds the policy sits idle during every tool call. Frameworks built for this, such as Jiang et al., 2025, VerlTool, arXiv:2509.01055, report roughly a 2x speedup mainly by executing rollouts asynchronously so tool latency on one trajectory overlaps with computation on another. The lesson generalizes: in agentic RL, the environment's wall-clock latency, not just the model's FLOPs, sets the training cost.

Seeing It in Motion

The multi-turn loop, shown as the actual message exchange between the three actors that exist during a rollout:

sequenceDiagram
  participant Pol as Policy (LLM)
  participant Env as Environment
  participant Ver as Verifier
  Pol->>Env: action a_1 (tool call)
  Env-->>Pol: observation o_1 (result / error)
  Pol->>Env: action a_2 (tool call)
  Env-->>Pol: observation o_2
  Note over Pol,Env: ... up to 100+ turns ...
  Pol->>Env: action a_T (final answer / commit)
  Env->>Ver: final state
  Ver-->>Pol: terminal reward r (one scalar)

The single terminal message from the verifier is the whole training signal for everything above it. That asymmetry, many actions in, one number out, is the sequence diagram's real content.

The trajectory also has a lifecycle, and modeling it as a state machine clarifies where episodes end and how reward is issued:

stateDiagram-v2
  [*] --> Observe
  Observe --> Act: policy samples action
  Act --> Execute: dispatch tool
  Execute --> Observe: return observation
  Execute --> Terminal: task-ending action
  Observe --> Terminal: horizon exceeded
  Terminal --> Reward: run verifier
  Reward --> [*]: emit advantage, update policy

Watch It Run

The pipeline below is the training loop in motion, exported from the companion diagram (agentic-reinforcement-learning.drawio); the arrows animate in the direction data flows, the policy-environment self-loop shows the multi-turn rollout iterating, and the dashed feedback edge shows the gradient update flowing back into the weights.

By the Numbers

The headline numbers in agentic RL are about reliability under repetition, not single-shot accuracy, and about the cost of the training loop. A few concrete anchors:

[IMAGE: Line chart of pass^k versus k (k from 1 to 8) for two agents. A prompted agent's curve starts near 0.5 and decays steeply toward 0.2 by k=8; an RL-trained agent's curve starts slightly higher and stays much flatter. Shade the gap between them and label it "what agentic RL buys: reliability, not peak."]

The pass^k metric deserves emphasis because it reframes what "good" means. Standard code benchmarks report pass@k: the probability that at least one of \(k\) samples is correct, which rewards diversity and lets you retry. τ-bench's pass^k asks the opposite: the probability that all \(k\) independent trials succeed. As \(k\) grows, pass^k falls off a cliff for any agent that is not consistently correct, which is exactly the property production deployments need and prompted agents lack. Much of what agentic RL buys is a flatter pass^k curve, not a higher single-shot peak.

A Concrete Example

Take a customer-service refund agent trained with GRPO on a τ-bench-style task. The task: "Refund the customer's most recent order if it shipped in the last 30 days." Success is verifiable by checking the database end state.

For one task instance, the trainer samples a group of four trajectories from the current policy:

Group mean reward is $0.5$, standard deviation $0.5$. The normalized advantages are \(\hat{A}_A = \hat{A}_C = +1\) and \(\hat{A}_B = \hat{A}_D = -1\). Every token in trajectories A and C is reinforced; every token in B and D is discouraged.

Notice what the algorithm learned and what it did not. It correctly pushed toward checking the ship date before refunding (present in the winners, absent in B). Good. But it also positively reinforced every token in A and C uniformly, including any incidental verbosity, and it punished the entire trajectory D even though D's first two actions (get_orders, then asking a clarifying question) were reasonable; D failed only because it then looped until the horizon cut it off. That clarifying question, a genuinely good habit, got a negative gradient because it happened to live in a losing trajectory. This is trajectory-level credit assignment doing damage at the margin, and it is precisely why turn-level and process-reward methods exist: they would let the checker reward the correct check_ship_date step directly and stop blaming D's early good moves for its late failure.

Run this over thousands of task instances and the policy converges toward the robust habit (check, then act) despite the noisy per-token credit, because the correlation between "checks the date" and "wins" is strong enough to survive the averaging. Agentic RL works not because the credit is clean but because, over enough groups, the signal outvotes the noise.

[IMAGE: The four sampled trajectories A-D from the worked example drawn as parallel action chains, colour-coded green (reward 1) or red (reward 0), with the group mean line at 0.5 and the resulting +1 / -1 advantages annotated on each. Circle trajectory D's early clarifying question in amber to show a "good action punished for a late failure."]

[IMAGE: A reward-hacking gallery: three small panels showing a policy exploiting a verifier - (1) deleting the failing test, (2) hard-coding the expected output, (3) catching and swallowing the error. Each panel shows "tests pass ✓" with a red "but the task is not solved" stamp, illustrating that a checkable reward is not a safe reward.]

Where It Breaks

Reward hacking against the verifier. A checkable reward is not a safe reward. Point RL at "make the tests pass" and a sufficiently capable policy may learn to delete the failing test, hard-code the expected output, or exploit a bug in the checker. The verifier defines the objective literally, and the policy optimizes the literal objective, not the intended one. Robust checkers, held-out tests, and adversarial verification are now standard defenses, and they are never fully sufficient.

Reward sparsity over long horizons. When success requires forty correct actions and the reward is one terminal bit, early in training the policy almost never succeeds, so almost every trajectory gets zero reward and there is nothing to learn from. Group-relative methods degenerate when the whole group scores zero (the advantage is undefined or uniformly flat). Curriculum design, dense process rewards, and starting from a supervised-fine-tuned policy that can already sometimes succeed are the usual escapes.

Non-stationary environments. In a POMDP where the environment includes a simulated user or another agent, the environment shifts as the policy learns, so the target moves. A policy that learned to exploit a particular user-simulator quirk in early training finds that quirk gone once the simulator is updated, and its learned behavior transfers poorly. This is the multi-agent instability problem, imported into training.

Harness overfitting. Because the harness is part of the trained system, a policy can quietly memorize idiosyncrasies of the training harness (a specific error-message format, a particular tool-timeout behavior) and collapse when deployed against a slightly different one. The gains look real in training and evaporate in production. Evaluating on a held-out harness, not just held-out tasks, is the check most teams skip.

Cost and reproducibility. Rollouts that call real tools are slow, expensive, and non-deterministic (a web page changes, an API rate-limits). This makes agentic RL runs hard to reproduce and expensive to debug, and it pushes teams toward simulated environments, which then reintroduce the sim-to-real gap.

Alternative Designs

Agentic RL is not the only way to get a capable agent, and it is often not the first thing to try. The honest comparison:

The practical stack is usually layered, not either-or: SFT to warm-start a policy that can occasionally succeed, then agentic RL to push reliability up, with the whole thing wrapped in a well-engineered harness. RL from scratch on a cold policy is rarely worth the pain when the terminal reward is sparse.

How It Is Used in Practice

Coding is the flagship. The agents that lead SWE-bench-style leaderboards in 2026 are, with few exceptions, RL-trained against real repositories where the reward is the test suite. The task is a natural fit: the environment (a code sandbox) is cheap to run, the verifier (pytest) is exact, and the horizon (edit, run, read, patch) is genuinely multi-turn. This is where the method's advantages are least ambiguous.

Tool-and-API agents are the fast-growing second front. Frameworks like VerlTool and the open post-training stacks shipped by hardware and model vendors provide the rollout infrastructure, environment abstractions, and verifiable-reward tooling to train agents for SQL generation, retrieval-augmented QA, and structured tool use. τ-bench and its successors are the yardstick, and the explicit goal is lifting the pass^k curve so an agent that can do a refund once can do it a thousand times without a costly mistake.

Web and GUI agents are the hardest and least mature. The environment (a live browser or operating system) is slow, non-deterministic, and only partially verifiable, so training is dominated by the environment-engineering problems above. Progress here is real but slower, and it leans heavily on simulated environments to make rollouts affordable, which reopens the sim-to-real gap.

Across all three, the operational reality is the same: the environment is now infrastructure you build, maintain, and pay for, on the same footing as the training cluster.

Insights Worth Remembering

• The jump from RLHF to agentic RL is not "more RL," it is a change in the problem's mathematical type: from a one-step bandit to a long-horizon POMDP. Every difficulty follows from that.
• Verifiable rewards are what made multi-turn agent training practical. The reward became a program you can trust literally, which is exactly why the policy will exploit it literally.
• Credit assignment, not model capacity, is the binding constraint on today's agentic RL. The field's active frontier is making the reward signal reach the right actions across a long trajectory.
• Vanilla GRPO gives every action in a trajectory the same credit. It works anyway, because over enough sampled groups the true signal outvotes the per-token noise, not because the credit is correct.
• The harness is part of the trained model. You are never training a policy alone; you are training a policy-plus-environment system, and the two overfit to each other.
• Reliability is the real product. A higher single-shot score is easy to demo and easy to fake; a flat pass^k curve is what makes an agent deployable, and it is what agentic RL is actually for.

Open Questions

• Can credit assignment be solved cleanly? Turn-level advantages, process rewards, and hierarchical grouping all densify the signal, but each adds a component that can itself be gamed or mis-specified. Whether a general, robust long-horizon credit method exists, or whether it stays task-specific, is genuinely open (evidence: the 2026 literature is a proliferation of partial fixes, not a convergence on one).
• How far does verifiability reach? Coding and math have free checkers; most valuable real tasks (research, design, negotiation) do not. Whether reliable verifiers can be built for fuzzy tasks, or whether learned reward models must return and be made hack-resistant, will decide how broad agentic RL becomes. This is inference, not settled fact.
• Will agents trained in simulation transfer? The economics push toward simulated environments; the sim-to-real gap pushes back. How much of a simulator-trained web agent survives contact with the live web is measured case by case and not yet generally understood.
• Does the harness need to be learned too? If the policy and harness are co-adapted, jointly optimizing both (learning tool schemas, retry policies, and memory strategies rather than hand-designing them) is a natural next step, and early work is probing it. Whether this helps or just enlarges the overfitting surface is unresolved.
• Multi-agent RL as environment. When the environment contains other learning agents, training becomes a moving-target game with all the instability that implies. Whether cooperative multi-agent systems can be trained stably, or whether they must be assembled from separately trained specialists, is an open and active question.

Sources and Further Reading

Foundational Papers

• Schulman et al., 2017, Proximal Policy Optimization Algorithms, arXiv:1707.06347 - the policy-gradient method underneath most LLM RL.
• Ouyang et al., 2022, Training language models to follow instructions with human feedback (InstructGPT), arXiv:2203.02155 - the RLHF recipe that framed LLM-RL as a single-step problem.
• Yao et al., 2022, ReAct: Synergizing Reasoning and Acting in Language Models, arXiv:2210.03629 - the reason-act-observe loop agentic RL optimizes.
• Schick et al., 2023, Toolformer: Language Models Can Teach Themselves to Use Tools, arXiv:2302.04761 - learned tool invocation.
• Shao et al., 2024, DeepSeekMath (GRPO), arXiv:2402.03300 - group-relative advantage without a critic; verifiable rewards.
• DeepSeek-AI, 2025, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL, arXiv:2501.12948 - large-scale RLVR inducing reasoning behavior.

Important Follow-up Work

• Wang et al., 2025, The Landscape of Agentic Reinforcement Learning for LLMs: A Survey, arXiv:2509.02547 - the MDP-to-POMDP reframing and the capability taxonomy; the definitional reference.
• Yao et al., 2024, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv:2406.12045 - the pass^k reliability metric and verifiable end-state tasks.
• Jiang et al., 2025, VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use, arXiv:2509.01055 - unified rollout infrastructure and asynchronous execution.
• Cui et al., 2026, Rethinking Agentic Reinforcement Learning in Large Language Models, arXiv:2604.27859 - a 2026 synthesis of the credit-assignment and design challenges.

Technical Blogs and Additional Resources

• Sierra, τ-bench: Shaping the development and evaluation of AI agents - the reliability framing in plain terms.
• Awesome Agentic LLM RL Papers (GitHub) - the community-maintained reading list tracking this fast-moving area.