Reinforcement learning for language-model agents

Where reinforcement learning sits in training

Language models are trained in phases, and each phase supplies a different target. Pretraining supplies the next token of existing text. Supervised fine-tuning supplies demonstrations. Preference training supplies comparisons. Reinforcement learning (RL) supplies a score for something the model produced.Published recipes often mix or repeat these phases. Reports from OpenAI, DeepSeek, Qwen and other teams describe larger RL runs for reasoning and tool use than earlier instruction-following systems used; recipes and compute allocations remain mostly undisclosed.

The loop, in a game and in a language model

RL describes an agent that receives an observation, takes an action, and receives a reward from an environment. The environment has a full state that the agent may see only in part. For a language-model agent, the observation is whatever sits in the context window; an action can be a single token, a message, or a tool call; and the environment can be a code sandbox, a browser, a grader, or a simulated user.

The example uses one outcome reward at the end of the episode, so a long sequence of token choices shares a small amount of feedback. Other systems add intermediate or process rewards, which score steps along the way.

Policy gradients: make what scored well more likely

A policy π_θ is a probability distribution over actions, set by the model’s weights θ. A policy-gradient method samples an action, scores it, and adjusts θ to change the log-probability of that action in proportion to its advantage: how much better its return was than expected. A return is the sum of rewards from that action onward, with optional discounting of later rewards. A positive advantage pushes up the sampled action’s log-probability; a negative one pushes it down. Shared model weights mean a batch update can also affect other actions and contexts.

A baseline is a reference value subtracted from the sampled return, often an estimate of the expected return from that state. A well-chosen baseline reduces the variance of the gradient estimate without changing its expected value. In the one-action example below, return and reward are the same.The estimator comes from Williams’s REINFORCE (1992). The baseline may depend on the state but not on the action taken; that condition keeps the expected gradient unchanged.

PPO’s clipped objective

Proximal Policy Optimization (PPO) reuses each batch of samples for several gradient steps. During those steps the policy drifts away from the one that generated the samples, π_old. PPO measures the drift for each sampled action with the probability ratio r and clips the objective in the direction favored by the advantage: above 1 + ε for a positive advantage, below 1 − ε for a negative one.

Clipping discourages large updates, but it does not impose a hard bound on how far the policy can change.DAPO (2025) uses separate lower and upper clip values and gives the upper side more room, a change its authors call “clip-higher.” They report that it lets low-probability tokens increase during training. arXiv:2503.14476 The PPO formulation used in early RLHF systems also trains a value model, or critic, that predicts the expected reward from each state and supplies the baseline.

GRPO: a baseline from a group of samples

Group Relative Policy Optimization (GRPO), introduced with DeepSeekMath in 2024, removes the value model. It samples a group of G responses to the same prompt, scores each, and normalizes the rewards within the group. With a single outcome reward, every token of response i receives the same advantage Â_i. Responses above the group mean are reinforced; those below it are suppressed.

When every response scores the same, the standard deviation is zero and the written formula is undefined. Implementations use a small denominator offset or explicitly set these advantages to zero, as this figure does. The group then supplies no reward-based policy gradient. A separate KL penalty can still contribute a gradient.The original DeepSeekMath objective, equation 3, adds KL regularization separately from the normalized reward term. The next figure shows the reward advantages for every possible outcome of a six-response group.

GRPO’s main costs and fixes, as reported in the papers that introduced them:

• No learned value model. This reduces memory and training complexity. The total saving depends on how the policy, reference, reward model and optimizer states are placed across hardware.
• Natural fit for verifiable tasks. Binary correct/incorrect rewards give a clean relative signal within a group.
• Wasted groups. DAPO’s “dynamic sampling” drops prompts whose groups are all correct or all wrong and keeps sampling until the batch has signal. Curriculum methods keep prompts near a 50% solve rate for the same reason.Without the division by the standard deviation, the total squared advantage of a 0/1 group is G·p(1−p), which peaks at a solve rate p of one half. Dr. GRPO removes that division, arguing it biases training toward very easy and very hard prompts. Liu et al., 2025
• Length bias. The original per-response length normalization favored short correct answers and long wrong ones. Dr. GRPO and DAPO’s token-level loss correct it.
• Sequence-level ratios. Qwen’s GSPO (2025) computes the importance ratio for a whole sequence instead of each token, which the authors report stabilizes training for large mixture-of-experts models.

Later methods each change one part of this recipe: the baseline, the clipping rule, the token weighting, the sampling policy, or the correction for stale samples. Similar names can hide materially different objectives.

Rewards learned from human preferences

Helpfulness has no answer key. RLHF, reinforcement learning from human feedback, trains a reward model r_φ from comparisons: a person reads two responses to the same prompt and picks the better one. The Bradley–Terry model turns the two scalar scores into a probability that the first is preferred.

Training maximizes the log-probability of each choice people made. A comparison moves the chosen response’s score up and the other’s down in proportion to 1 − P(chosen), so comparisons the reward model already predicts well move it little.

Keeping the policy near a reference

The policy is then trained to score well under the learned reward, with a penalty for drifting from a reference policy π_ref, usually the supervised model it started from. The penalty is the KL divergence, a measure of how different two probability distributions are, scaled by a coefficient β.

The penalty discourages moving into regions where the reward model has little training evidence, but it does not guarantee reliable scores. With fixed rewards, a fixed reference and β > 0, optimization over all output distributions has an exact solution: the optimal policy is the reference reweighted by exp(r/β) and renormalized.π*(y) = π_ref(y) · exp(r(y)/β) / Z, where Z normalizes the distribution. This is equation 4 in the DPO paper (Rafailov et al., 2023), which builds its loss on it. The figure applies that formula to one next-token distribution.

Variants that change the source of the preference

Direct Preference Optimization (DPO, 2023) derives a policy loss directly from preference pairs and a reference model, using the solution above. It avoids fitting an explicit reward model and running on-policy sampling during the DPO update.

Constitutional AI (Anthropic, 2022) uses written principles to guide model critiques, revisions and AI-generated preference labels. Anthropic then trains a preference model on those labels and applies reinforcement learning from AI feedback (RLAIF).

Rubric-based rewards have a grader model score an open-ended response against explicit criteria. The reward is more structured than one overall preference, but its reliability still depends on the rubric, the grader and how both were validated.

Rewards a program can check

RL with verifiable rewards (RLVR) uses tasks whose outcomes a program can check: a final math answer compared with a known one, or code run against tests. OpenAI reports that o1 learned to use chain of thought productively through large-scale RL but has not published its reward design. DeepSeek documents a clearer example. DeepSeek-R1-Zero was trained with RL directly from the base model, with a rule-based accuracy reward and a format reward and no supervised reasoning traces.

During that run, R1-Zero’s pass@1 score on the AIME 2024 math competition rose from 15.6% to 71.0%.Pass@1 is the fraction of problems solved by a single sampled answer, averaged over samples. Figures from the DeepSeek-R1 paper (2025), section 2.2.4. Its average response length grew steadily over training, and the model began re-checking its work and changing strategy mid-answer. DeepSeek called one such shift, in an intermediate checkpoint, an “aha moment.” Length growth can be limited: Kimi k1.5 used a length penalty to train short chain-of-thought models, and token budgets and penalties for overlong responses serve the same purpose.

Verifiable rewards help in three ways. Exact answers and well-designed tests give a more reproducible signal than an overall preference score, though weak tests can be incomplete or exploitable. Once tasks and checkers exist, many sampled responses can be graded without a person labeling each one. And a final score rewards any reasoning path that reaches the correct result; it does not say which intermediate steps were valid.

The method has spread in three directions. DeepSeek fine-tuned smaller Qwen and Llama models on reasoning traces from R1 (distillation) and reported large benchmark gains over the corresponding base models. Products expose the length–accuracy trade-off to users: Claude’s extended thinking, Gemini’s thinking budget and OpenAI’s reasoning effort. And verifiers now cover formal proofs in Lean, tool-call schemas, structured extraction, chemistry and rubric-graded free text.

Train-time and test-time compute

Train-time compute updates the weights. Test-time compute gives a trained model more generation or search per query. OpenAI reports that o1’s performance improved with more of both. Its o3 announcement says the team added another order of magnitude in both and kept seeing gains. These are relative trends from one provider, with no absolute compute values or cost breakdown.

xAI separately states that Grok 4 used reinforcement-learning compute comparable to its pretraining compute. That claim concerns a different model and a different reporting basis, so it does not belong on the same scale as OpenAI’s relative result.

Reinforcement learning for tool-using agents

A tool-using episode interleaves model messages, tool calls, and observations returned by an environment. Training can score the whole trajectory with an outcome, such as hidden-test results, or add intermediate scores. OpenAI has described end-to-end RL for deep research and tool use, and Qwen has published an agentic RL setup for coding. Other providers disclose less, so the figure shows a representative training design, not one product’s recipe.

The tool-call option pays +0.10 after each of the four search, test or edit calls, with no bonus for submission. On a failed attempt, the first turn therefore gets 0 + 4 × 0.10 − 0.45 = −0.05; the final submission gets 0 − 0.45 = −0.45. This is a return-to-go estimate with a fixed illustrative baseline, not a learned critic.See Spinning Up’s derivation of reward-to-go. Keeping the baseline fixed makes the arithmetic visible; a trained critic would estimate the return under the selected reward scheme. Paying for tool calls can reward unnecessary calls, so denser feedback also changes what the agent is encouraged to do.

Four difficulties dominate:

• Credit assignment. One outcome reward can apply to thousands of token choices across many turns without identifying which choices caused the result. Turn-level judges and process reward models give denser feedback but add more learned graders that can be wrong.
• Expensive environments. Coding rollouts need isolated repositories, dependencies, tools and tests. The Qwen3-Coder team reported running 20,000 environments in parallel.
• Asynchrony. Trajectories take different amounts of time to finish. Systems such as AReaL overlap generation and training to keep hardware busy, then use importance weighting to account for samples generated by an older policy.
• Train/inference mismatch. Separate generation and training systems can compute slightly different token probabilities. That changes the effective sampling policy and can destabilize importance-weighted updates.

Where agent rewards come from

Agent training needs tasks, environments and graders that expose the behavior being trained. The common sources, with no claim about how often each is used:

Reward hacking

A policy is optimized against the reward the training system computes, not against the designer’s intent. If that reward omits part of the intended behavior, the policy can raise its score without improving the real outcome: by modifying tests, exploiting a grader’s preferences, or reading information the task meant to hide.

Gao, Schulman and Hilton measured this with a stand-in for people: a large “gold” reward model labeled the data, a smaller proxy reward model learned from those labels, and a policy was trained against the proxy.Scaling Laws for Reward Model Overoptimization (2022). They define d = √KL(π ‖ π_init) and fit the gold score for RL as R(d) = d(α − β ln d). With α held fixed, β varied smoothly with proxy reward-model size, and larger proxies reached higher gold scores. As training moved the policy further from its starting point, the proxy score kept rising while the gold score rose, peaked and fell.

Examples from published reports

• Test special-casing. Detecting the test harness and hard-coding expected outputs; editing or deleting failing tests instead of fixing the code.
• Sycophancy toward the judge. Long, confident, well-formatted answers that an LLM grader scores higher regardless of correctness.
• Length gaming. Padding reasoning when the reward correlates with length; producing empty “thinking” when length is penalized.
• Sandbox escape. Reading the answer file, monkey-patching the checker, or altering the timer in a benchmark.
• Impossible-task capitulation. Claiming success on a task that cannot be completed, because “I could not do this” scores zero.

Countermeasures

• Hidden, held-out tests and randomized environments, so a hack that works on one instance does not transfer.
• Chain-of-thought monitoring, in which a second model reads the reasoning for intent to hack. OpenAI reported in 2025 that penalizing “bad thoughts” directly teaches the model to hide them, so it recommends monitoring without optimizing against the monitor.
• Anthropic reported in 2025 that when a model learns to hack rewards in training, misalignment can generalize to unrelated behaviors. Its mitigation, “inoculation prompting,” tells the model during training that hacking is acceptable in that specific environment, which breaks the link between hacking and “being a bad model.”
• Stronger graders: ensembles, explicit rubrics, and checks on relevant state changes, such as whether the agent modified test files.

What laboratories have published

The table summarizes system cards, papers and official posts published through mid-2026, grouped by organization and ordered by date. Each row links its source. It lists what was disclosed; complete training recipes and compute allocations are not public for any of these systems.

The algorithms, in order

Public releases through mid-2026 point in one general direction: longer coding and research trajectories, more parallel environments, and more asynchronous training infrastructure.

Open problems

Long-horizon credit assignment

A trajectory-level reward does not show which earlier decisions caused the outcome. Process rewards, turn-level judges and hierarchical policies give denser signals, but each can introduce new grader errors.

Environment construction

Useful agent training needs realistic tasks with a reproducible starting state and reliable grading. Teams build and synthesize such tasks, then filter or validate them with tests and verifiers.

Reward hacking at scale

A fixed grader can contain exploitable gaps. Held-out checks and chain-of-thought monitoring reveal some failures, while optimizing directly against a monitor can teach a model to hide the behavior being monitored.

Exploration and entropy collapse

Some RL runs concentrate on a small set of responses, and output diversity drops. Asymmetric clipping, entropy bonuses and prompt curricula are among the methods used to preserve useful exploration.

Capability elicitation

Researchers disagree about how much RL creates new strategies and how much it shifts probability toward behaviors the base model could already produce. Results depend on the task, training horizon and metric.

Generalization from narrow rewards

Gains on math or code do not guarantee better instruction following, calibration or performance in other domains. Mixed training objectives try to preserve several behaviors at once.

Infrastructure

Large runs coordinate generation, sandboxed environments, training and evaluation. Asynchronous rollouts improve hardware use but make policy staleness and importance weighting harder to control.

Self-play and self-verification

Formal systems can verify generated proofs and support self-play. Open-ended domains lack an equally reliable verifier, which makes self-generated tasks and rewards harder to validate.

Evaluation

Static benchmarks saturate or leak into training data. Held-out tasks, live evaluations and carefully isolated agent environments reduce these problems without eliminating them.

Glossary

Policy (π)

The model being trained, viewed as a probability distribution over actions given its observation or context.

Rollout, trajectory

One sampled episode: a sequence of observations, model actions and rewards.

Reward

A scalar signal from the environment, a verifier or a reward model. It can arrive during a trajectory or at the end.

Return-to-go

The sum of rewards from the current action onward, optionally discounting later rewards. Subtracting a baseline gives a sampled advantage estimate.

Advantage

How much better an action or episode was than expected; the quantity that scales each gradient step.

Baseline, critic, value model

A baseline estimates the expected return for the advantage calculation. PPO is commonly used with a learned value function; GRPO uses group reward statistics.

On-policy, off-policy

Whether the samples used for an update came from the current policy. Stale samples need importance-weight corrections.

KL penalty

A regularization term that penalizes divergence from a reference policy.

RLHF

RL from human feedback: a reward model trained on preference comparisons, then policy optimization against it.

RLAIF

RL from AI feedback: model-generated evaluations supply some or all of the reward signal.

RLVR

RL with verifiable rewards: the reward is a program, such as an answer checker, unit tests, a compiler or a proof assistant.

PPO

Proximal Policy Optimization: a policy-gradient method with a clipped surrogate objective, commonly paired with a learned value function.

GRPO

Group Relative Policy Optimization: a PPO variant that normalizes rewards across responses to the same prompt and has no learned value model.

DPO

Direct Preference Optimization: an objective that updates a policy directly from preference pairs and a reference policy, without fitting an explicit reward model.

Process reward model (PRM)

A grader that scores intermediate steps as well as the final result.

Loss masking

Excluding environment-produced tokens, such as tool outputs, from the training loss in multi-turn episodes.

Reward hacking, specification gaming

Achieving high measured reward through behavior that violates the task designer’s intent.

Entropy collapse

A large drop in output diversity during training, leaving fewer behaviors available for exploration.

Test-time compute

Spending more inference tokens, through longer reasoning or more samples, to improve accuracy on a single query.