2025 was a breakout year for RL-based methods in LLM training. For a while, 'RL for LLMs' basically meant RLHF, largely following the ChatGPT-replication journey and its alignment-focused pipeline. Then OpenAI's o1 and DeepSeek-R1 helped prove a different point: on tasks with verifiable answers, RL can seriously boost an LLM's reasoning capability, not just polish its vibes. Around the same time, RL also became a tool for shaping agentic behavior, especially the ability to reliably call tools and execute multi-turn workflows.

With those trends converging, the community spent a lot of 2025 with discussions and experiments on RL algorithms tailored to token-level LLM training, especially with GRPO being the main attraction. This two-part post is my 'dont forget this' summary of what came out in this year. Part 1 lays out the basics of off-policy policy gradients, how GRPO emerged, and why it started to dominate PPO in practice. Part 2 surveys the many training tweaks and variants people proposed and debated throughout the year.

We’ll start with the token-level MDP framing and off-policy policy gradients, since that’s the common spine behind PPO, GRPO, and most of their descendants.

Token-Level MDPs

Training LLMs with RL has stemmed from the success of RLHF ^{1 2}, which optimizes LLMs using human feedback. These methods typically model the LLM as a policy in a token-level Markov decision process (MDP). Let the LLM be a stochastic policy $\pi_\theta$, parameterized by $\theta$, that maps token histories to distributions over the vocabulary. The state space $\mathcal{S}$ consists of all possible token sequences, while the action space $\mathcal{A}$ is the vocabulary itself.

Given a prompt dataset $\mathcal{D}$, a prompt $x \in \mathcal{D}$ initializes the episode. The model then generates a response autoregressively,

where generation terminates when $y_T$ is the end-of-sequence (EOS) token ¹.

At time step $t$, the state $s_t = (x, y_{\leq t}) \in \mathcal{S}$ is defined as the prompt concatenated with the partial response, and the action $a_t = y_{t+1} \in \mathcal{A}$ corresponds to selecting the next token. Upon taking action $a_t$, the agent transitions deterministically to $s_{t+1} = (x, y_{\leq t+1})$ and receives a reward $r(s_t, a_t) = r(x, y_{\leq t+1})$, where $r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ is a heuristic reward function or a learned reward model.

In practice, the reward structure is typically sparse, with $r(x, y_{\leq t}) = 0$ for $t < T$ and a single terminal reward $r(x, y_{\leq T})$ at the end of the episode. To facilitate learning, we may also design intermediate rewards (also called process rewards) for $t < T$, yet this is an active area of research.

The overall learning objective is to maximize the expected discounted sum of rewards under the policy:

Policy Gradient Methods

A popular approach to optimize the policy is to use policy gradient methods. The key idea is to compute an estimate of the gradient of the objective with respect to the policy parameters $\theta$, and then perform gradient ascent. As an LLM itself is a parameterized stochastic policy, it is natural to apply policy gradient methods to LLM training. Using the policy gradient theorem, we can derive the gradient estimator as follows. (For more details, see Section 13.1 of Sutton & Barto's book or this nice blog post by Lilian Weng!)

This is the basic idea of REINFORCE ³. Here, $A$ is an advantage function, which may be any function in the following form:

As stated, $\hat{Q}$ must be an estimator of the state action-value function $Q^{\pi_\theta} : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$. More specifically, it must satify:

$b$ is a baseline, which may be any function that does not depend on the action $y_t$. The purpose of the baseline is to help reduce the variance of the gradient estimator without introducing bias. A near-optimal minimizer of variance is the state-value function $V^{\pi_\theta} : \mathcal{S} \to \mathbb{R}$, defined as:

At this point, one may imagine directly using $Q^{\pi_\theta}$ and $V^{\pi_\theta}$ to define the advantage function. However, both functions are unknown and must be estimated from samples. To this end, the actor-critic method trains alongside a value network $V_\phi$, parameterized by $\phi$, to approximate $V^{\pi_\theta}$. As $Q^{\pi_\theta}$ can be expressed in terms of $V^{\pi_\theta}$, we may then use $V_\phi$ to construct various estimators for the advantage function. Some common choices include:

• Baselined MC return: low bias, high variance $$A^{\textsf{MC}}(x, y_{\leq t}) = \sum_{t'=t}^{\lvert y \rvert}\gamma^{t' - t}r(x, y_{\leq t'}) - V_\phi(x, y_{<t})$$
• TD residual: high bias, low variance $$A^{\textsf{TD}}(x, y_{\leq t}) = r(x, y_{\leq t}) + \gamma V_\phi(x, y_{< t + 1}) - V_\phi(x, y_{<t})$$
• $\boldsymbol{n}$-step TD residual: discrete control between bias and variance using a hyperparameter $n \in \{1, 2, \ldots, \lvert y \rvert - t + 1\}$; note that $A^{\textsf{TD}} = A^{\textsf{TD}(1)}$ and $A^{\textsf{MC}} = A^{\textsf{TD}(\lvert y \rvert - t + 1)}$ $$A^{\textsf{TD}(n)}(x, y_{\leq t}) = \sum_{t'=t}^{t+n-1}\gamma^{t' - t}r(x, y_{\leq t'}) + \gamma^{n} V_\phi(x, y_{< t+n}) - V_\phi(x, y_{<t})$$
• Generalized Advantage Estimation (GAE) ⁴: continuous control between bias and variance using a hyperparameter $\lambda \in [0, 1]$; note that $A^{\textsf{TD}} = A^{\textsf{GAE}(0)}$ and $A^{\textsf{MC}} = A^{\textsf{GAE}(1)}$ $$\begin{aligned} A^{\textsf{GAE}(\lambda)}(x, y_{\leq t}) &= (1 - \lambda)\sum_{n=1}^{\lvert y \rvert - t}\lambda^{n-1}A^{\textsf{TD}(n)}(x, y_{\leq t}) + \lambda^{\lvert y \rvert - t}A^{\textsf{TD}(\lvert y \rvert - t + 1)}(x, y_{\leq t}) \\ &= \sum_{t'=t}^{\lvert y \rvert - 1}(\gamma\lambda)^{t' - t}A^{\textsf{TD}}(x, y_{\leq t'}) \end{aligned}$$

(For more details and insights behind advantage estimation choices and GAE, see this blog post by Daniel Seita.)

The policy network $\pi_\theta$ and the value network $V_\phi$ are typically trained alternately. The value network is trained to minimize the mean squared error between its predictions and the bootstrapped returns (which is equivalent to the sum of the advantage estimate and predicted value), while the policy network is updated using the policy gradient estimator with the chosen advantage function.

Off-Policy Policy Gradient Methods

For LLMs, or even in other RL environments, generating on-policy trajectories can be computationally expensive and time-consuming. This is especially true for LLMs as it requires running the LLM to generate responses for each prompt in the dataset. To increase efficiency, we may consider generating multiple responses in parallel for multiple prompts, more than the amount we would use for a single model update. However, this introduces a distribution mismatch between the current policy $\pi_\theta$ and the behavior policy $\pi_0$ used to generate the responses. Thus, we need to adjust the policy gradient estimator by incorporating importance sampling ratios. This is where off-policy policy gradient methods come into play. Using importance sampling, we can rewrite the policy gradient estimator as follows:

The importance sampling ratios are inserted in a somewhat unnatural way: we maintain on-policy samples $(y_1, y_2, \ldots, y_T)$ generated from the current policy $\pi_\theta$, but each per-token expectation is taken over an off-policy sample $y_t'$ sampled from the behavior policy $\pi_0$, conditioned on the on-policy history $(x, y_{<t})$.

After this adjustment, we may apply an approximation $(*)$ that replaces the on-policy history with off-policy history entirely. This is valid only when $\pi_\theta \approx \pi_0$, so that the distributions are close enough for the difference in histories to become negligible. This leads to a fully off-policy estimator, where all tokens are sampled from the behavior policy $\pi_0$. Also, in general, importance sampling estimators become inaccurate when the sampling distribution is too far from the nominal distribution. This is another reason why we need $\pi_\theta \approx \pi_0$ for the approximation to hold well in practice.

Therefore, assuming $\pi_\theta \approx \pi_0$ is kept throughout training, we can express the overall objective function as:

Practical Implementation

The overall training procedure for off-policy policy gradient methods can be summarized as follows:

(Here, $[\![\cdot]\!]$ denotes the stop-gradient operator, which is equivalent to torch.Tensor.detach().)

The value network is a model that takes in an arbitrary token sequence and outputs a scalar value. In practice, it is often implemented as an LLM with a scalar head on top. If we are using a reward model for reward computation (as in RLHF), this is very similar to the functionality of the reward model itself. For this reason, it is common to initialize the value network with the reward model weights in this case. ^{1 5} If we do not have a reward model and instead use a heuristic reward function, we usually initialize the value network as same as the policy network.

Many modern RL-LLM frameworks (e.g., TRL, verl, slime) implement their pipelines based on the above structure. Often, they also incorporate various techniques to speed up training and improve memory efficiency. For instance, rollouts may be accelerated using vLLM or SGLang. Training may be optimized using parallelism techniques such as FSDP or Megatron-LM.

Meanwhile, as modern LLMs generate long reasoning traces and this behavior especially gets amplified under RL, rollouts may remain heavy and time-consuming even with aggressive optimizations. As a result, the community has been actively working on approaches to increase asynchrony, for example by performing asynchronous rollouts and asynchronous reward computation if necessary. The Prime Intellect team has been making notable progress in this direction (PRIME-RL, Verifiers).

Proximal Policy Optimization (PPO)

Now, to ensure that the assumption $\pi_\theta \approx \pi_0$ holds during training, we may need to take special care. One popular approach is to use proximal policy optimization (PPO) ⁶, which is widely adopted in RL for LLMs due to its simplicity and effectiveness. The core idea of PPO is to prevent large policy updates by constraining the change in policy at each update step. To this end, we may add trust-region constraints. The original idea comes from trust region policy optimization (TRPO) ⁷, and PPO further simplifies the implementation as follows:

1. Use a clipped objective as a proxy for trust-region constraints. Formally, we define the clipping function

   $$\mathcal{C}_\epsilon(\rho, A) \coloneqq \min\Bigl(\rho A, \mathrm{clip}\bigl(\rho, 1 - \epsilon, 1 + \epsilon\bigr)A\Bigr).$$

   $\mathcal{C}_\epsilon$ places an upper bound on $\rho$ when $A$ is positive, and a lower bound on $\rho$ when $A$ is negative. Intuitively, if we consider $\rho$ as the importance sampling ratio, this prevents excessively large updates that may arise from distribution mismatch. Using this, we replace the per-token contribution to the policy objective by

   $$\frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_0(y_t \mid x, y_{<t})}A(x, y_{\leq t}) \gets \mathcal{C}_\epsilon\left(\frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_0(y_t \mid x, y_{<t})}, A(x, y_{\leq t})\right).$$

2. Maintain the initial policy $\pi_{\textsf{ref}}$ as a reference policy to prevent policy collapse. Then, add a KL-penalty to the per-token reward by:

   $$r(x, y_{\leq t}) \gets r(x, y_{\leq t}) - \beta\log\frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_{\textsf{ref}}(y_t \mid x, y_{<t})}.$$

   Here, $\beta > 0$ is a hyperparameter that controls the strength of the penalty. After this adjustment, update the advantages accordingly, using the chosen advantage function (GAE is the most common choice with PPO). Let $A^{\textsf{KL}}$ denote the updated advantage function, including the KL-penalty.

Finally, the resulting PPO objective becomes:

In the early days of RL for LLMs, PPO was the go-to algorithm. This was especially true during the RLHF era, where RL was primarily used for human alignment, as exemplified by ChatGPT and its many replications. During this period, relatively little attention was paid to advancing RL algorithms themselves. Most of the effort went into building better reward models, mitigating overoptimization, figuring out how to collect preference data at scale, etc. (which eventually led us to the RL vs DPO debate and beyond — here are some great articles that summarize the progress back then).

Group Relative Policy Optimization (GRPO)

However, things started to change after the success of o1 and R1. RL for LLMs began to matter again, not just for alignment, but as a way to incentivize general reasoning capabilites. A big reason this worked was that these models were trained on tasks with clear, verifiable answers. When rewards are strictly verifiable, many of the classic RLHF problems that come from reward modeling (reward hacking, weird incentives, misalignment) largely disappear. Now that these are less of a bottleneck, attention has shifted back to the RL algorithms, with the goal of maximizing scalability and stability. DeepSeek's solution to this problem was group relative policy optimization (GRPO) ⁸, which builds upon PPO with several key modifications to take better advantage of verifiable rewards in reasoning tasks.

GRPO assumes completely sequence-level rewards ², which was always the case in RL for LLMs anyway. The key difference is that instead of treating the reward as given at the last token only, we treat it as given for the entire generated sequence. Accordingly, we calculate advantages at the sequence level, and broadcast the same advantage to every token in the sequence. Furthermore, GRPO assumes a grouped sampling scheme, meaning that for each prompt $x$, we generate a group of $G$ responses $\mathbf{y} = \left(y^{(1)}, y^{(2)}, \ldots, y^{(G)}\right)$ using the behavior policy $\pi_0$. The key idea is to use relative rewards within the group to define a more informative advantage function. Specifically, GRPO introduces the following components:

1. Define the advantage function $A^{\textsf{GRPO}}$ using normalized rewards within the group:

   $$A^{\textsf{GRPO}}\left(x, y^{(j)}\right) = \frac{r\left(x, y^{(j)}\right) - \mathrm{mean}\Bigl(\left\{r\left(x, y^{(1)}\right), \ldots, r\left(x, y^{(G)}\right)\right\}\Bigr)}{\mathrm{std}\Bigl(\left\{r\left(x, y^{(1)}\right), \ldots, r\left(x, y^{(G)}\right)\right\}\Bigr) + \epsilon},$$

   and broadcast it to all tokens in $y^{(j)}$. The standard advantage function $Q^{\pi_\theta} - V^{\pi_\theta}$ is a measure of better-than-average performance. Similarly, $A^{\textsf{GRPO}}$ measures how much better a response is compared to other responses in the same group. Here, $\epsilon > 0$ is a small constant added for numerical stability.

2. As we do not have per-token rewards, we cannot directly apply the KL-penalty as in PPO. Instead, add KL-penalty to the per-token objective using the k3 approximator ⁹:

   $$k_3\left(\xi; p, q\right) \coloneqq \frac{q(\xi)}{p(\xi)} - \log\frac{q(\xi)}{p(\xi)} - 1, \quad \textsf{where} \;\; \xi \sim p.$$

   This is meant to approximate the KL-divergence $D_{\mathrm{KL}}(p \,\|\, q)$. In GRPO, we use $\xi = y_t^{(j)}$ with $p = \pi_\theta\left(\cdot \Bigm\vert x, y^{(j)}_{<t}\right)$ and $q = \pi_{\textsf{ref}}\left(\cdot \Bigm\vert x, y^{(j)}_{<t}\right)$. You may notice some mismatch in the sampling distribution here, as $y_t^{(j)}$ is sampled from the behavior policy $\pi_0$ instead of the current policy $\pi_\theta$. I will discuss this subtlety in the next post.

3. Apply sequence-level length normalization by dividing the per-token contributions by the sequence length $\lvert y^{(j)} \rvert$. This is slightly unintuitive, as the idea of average reward is typically discussed in infinite-horizon MDPs. There has been much debate on this length normalization term in the sense of loss aggregation, which I will also discuss in the next post. For now, just note that this is part of the original GRPO formulation.

Applying all these components, the overall GRPO objective becomes:

Why GRPO?

Following DeepSeek’s breakout success, GRPO rapidly gained attention and, in many discussions, overtook PPO as the method of choice. Yet PPO isn’t dead. In fact, it still works surprisingly well. However, it’s hard to ignore that much of the recent progress in RL for LLMs has been built on top of GRPO-style methods rather than on PPO itself. This shift isn’t just a hype; GRPO appears to offer a some practical advantages that make it an appealing choice for large-scale LLM training.

The most obvious advantage of GRPO is its computational simplicity. GRPO eliminates the need for a learned value function in the advantage estimate, which means there is no separate value network $V_\phi$ to train or maintain. This reduces both compute cost and system complexity, making it easier to implement and scale.

But more interestingly, recent empirical results suggest that removing or weakening value learning may actually improve performance, not just efficiency. A notable trend is that even when PPO is used, practitioners increasingly push the GAE parameter $\lambda$ toward 1.0, effectively reducing reliance on the value function. This behavior is reflected in results from the recent DeepSeek-R1 report published in Nature ¹⁰, which shows a clear performance ordering:

This matches a growing consensus that value learning is hard in RL for LLMs. Researchers like John Schulman or Ross Taylor have both commented on this publicly. While the exact cause is still unclear, a common hypothesis is that bias in the learned value function harms optimization more than variance reduction helps. In other words, the errors introduced by imperfect value estimates may outweigh the benefits they provide in stabilizing policy updates.

Work such as VC-PPO ¹¹ offers more concrete analysis of this issue. Two key takeaways stand out. First, value network initialization matters enormously, and explicit value-pretraining under a fixed initial policy may help. Second, variance reduction may not be important enough in value learning as it is in policy learning. This has led to proposals like decoupled GAE, where different $\lambda$ values are used for policy updates and value updates. Even with these refinements, however, value learning remains brittle and sensitive to design choices.

GRPO avoids these issues altogether. With no value network, there’s no value bias, no delicate initialization, and no need to tune bias-variance tradeoffs. That simplicity, combined with strong empirical results, explains why GRPO has become the dominant foundation for recent RL-LLM work.

References