In the previous post, we introduced off-policy policy gradient methods that are widely used for LLM training, especially GRPO. In this post, we will discuss several tweaks and improvements to GRPO that have been proposed in recent literature.

Some fixes are specific to GRPO, while others are more general and can be applied to other off-policy policy gradient methods as well. Throughout this post, to make everything simple, we will build upon the following objective:

This is the basic off-policy policy gradient objective in the token-level MDP setup. Adding appropriate clipping, KL penalty, group sampling and averaging, or length normalization to this objective will give us PPO or GRPO.

Also, recall that the clipping function $\mathcal{C}_\epsilon$ is defined as:

Mitigating Unintended Off-Policiness

The first group of tweaks are about mitigating unintended off-policiness in the training pipeline. Though our baseline formulation already includes the IS ratios to correct for the off-policiness, there still exist some sources of off-policiness that people haven't paid much attention to, but can actually cause significant issues in training. These sources of off-policiness can lead to high variance and instability in training, and therefore we need to be careful about them.

Training-Inference Mismatch

One major source of off-policiness is the mismatch between training and inference policies. Most existing LLM-RL frameworks use highly optimized inference engines (e.g., vLLM or SGLang) for collecting rollouts, while using different training policies (e.g., FSDP or Megatron-LM) for computing the losses and gradients. These inference engines employ various optimization techniques such as speculative decoding, low-precision computation, or batch-variant CUDA kernels, to achieve high throughput and low latency. In contrast, the training polices prioritize numerical stability and ease of gradient computation, and therefore may not have the same optimizations as the inference engines. This leads to a mismatch between the training and inference policies, which can cause significant off-policiness in the training process.

To be more specific, in the training pipeline, we are usually doing the following three steps:

1. training a policy $\pi_\theta^{\textsf{train}}$ with FSDP or Megatron-LM etc.,
2. collecting online rollouts with optimized an inference engine $\pi_0^{\textsf{rollout}}$ like vLLM or SGLang etc.,
3. but computing the off-policy logprobs from $\pi_0^{\textsf{train}}$, the training policy at the beginning of each iteration.

As a result, what we are actually computing (LHS) and what we should be computing (RHS) become different:

Therefore, we need to build our algorithms upon the RHS. However, naively applying clipping to the RHS may cause issues, especially regarding batch size. When we apply clipping to the RHS, the IS ratios become batch-variant, because the denominator $\pi_0^{\textsf{rollout}}$ is different for different batches. This can lead to significant batch size sensitivity in training, as the effective learning rate can vary significantly across batches due to the varying IS ratios. This was first observed by Hilton et al. (2022), and they propose a simple solution to this issue by decoupling the IS ratios into two parts: one for the rollout-train mismatch, and the other for the policy update. The first part is not clipped, while the second part is clipped:

This way, we can mitigate the batch size sensitivity issue caused by clipping, while still correcting for the off-policiness caused by the training-inference mismatch.

This fix was first thoroughly discussed by Yao et al. (2025) ¹. In their work, they also find that applying truncated importance sampling (TIS) to the first part works the best:

Routing Replay for MoE Models

For MoE models, the policy is not determined only by the dense parameters, but also by the router decisions. It is useful to write the policy as depending explicitly on the routing masks:

The usual token-level IS ratio is then implicitly

The problem is that $m_\theta^{\textsf{train}}$ and $m_0^{\textsf{train}}$ may activate different experts for the same token. Worse, the rollout was actually sampled using an inference engine, so the training ratio may be comparing probabilities computed under routing masks that differ from the masks used to generate the data. In other words, the ratio is pretending to compare policies, while the router quietly swaps the subnetworks underneath it. This is the main MoE-specific instability that motivates Routing Replay ^2,3.

Routing Replay fixes this by caching a routing mask $\bar m$ and using the same mask when computing the numerator and denominator of the IS ratio. Then the training objective (without fixing the training-inference mismatch for logprob computation) becomes:

There are two natural choices of $\bar m$. Vanilla Routing Replay (R2) ² replays the old-policy routing mask computed in the training engine:

This mainly reduces policy-staleness effects, because both the old and current policies are evaluated using the old training-engine routing pattern.

Rollout Routing Replay (R3) ³ instead replays the routing mask used by the inference engine during rollout:

This directly aligns the training forward pass with the rollout computation. Implementation-wise, R3 does not freeze the router logits completely. It reuses only the binary routing mask from inference, but still computes the gate weights from the training router logits. Let $z_{\theta,\ell,t}(e)$ and $m_{\theta,\ell,t}(e)$ each denote the router logits and expert mask for token $t$ at layer $\ell$ and expert $e$. Then the replayed gate weights are given by:

Then the replayed MoE output is

where $\mathcal{E}_{\theta,\ell,e}$ is the expert function for expert $e$ at layer $\ell$ and $h_{\ell,t}$ is the input to the router at layer $\ell$ and token $t$. So the discrete expert choice is replayed, but gradients can still flow through the training router logits inside the selected expert set. This avoids changing the activated sparse network between rollout and training, without completely removing router learning.

Routing Replay creates bias in the IS ratios, since the target policy is no longer the naturally routed policy, but rather the replay-routed policy. However, this routing gives a much more stable and meaningful IS ratio. In practice, this tradeoff is often worthwhile for MoE RL, especially under off-policy mini-batch reuse, where routing changes can otherwise make token-level ratios extremely noisy ⁴.

Biased KL estimator in GRPO

Recall that the original GRPO paper uses the k3 approximator for the KL-penalty term, which is given as:

This is an unbiased estimator for an approximation of $D_{\mathrm{KL}}(p \,\|\, q)$ near $p = q$. However, if we sample $\xi$ from a different distribution $b$, then this estimator becomes biased. To correct for this bias, we can use importance sampling to get an unbiased estimator ⁵:

In our case, we should use $\xi = y_t^{(j)}$ with $p = \pi_\theta\left(\cdot \Bigm\vert x, y^{(j)}_{<t}\right)$, $q = \pi_{\textsf{ref}}\left(\cdot \Bigm\vert x, y^{(j)}_{<t}\right)$, and $b = \pi_0\left(\cdot \Bigm\vert x, y^{(j)}_{<t}\right)$. Notably, DeepSeek-V3.2 adopts this fix as well.

Moreover, this IS-corrected, off-policy k3 estimator is also the correct choice at gradient-level. That is, even if we were using the original k3 estimator where $b \approx p$ and the differences are negligible, the value estimates would be unbiased, but the gradients would not be. If we take the gradient of $D_{\mathrm{KL}}(p \,\|\, q)$ with respect to the parameters of $p$, we should be careful since there are two contributions: $p$ appears both as the sampling distribution and inside the log-ratio. This leads to a common pitfall where if the estimates use on-policy samples, the gradients are not guaranteed to be unbiased, even if the values are. ⁶ For instance, the correct gradient of $D_{\mathrm{KL}}(p \,\|\, q)$ is given by:

but if we use the k3 estimator with on-policy samples, the gradient is given by:

which is a wrong estimator. So the k3 estimator is unbiased as a value, but differentiating it as a loss would give us incorrect objectives. In fact, this is actually an unbiased estimator for the gradient of the reverse KL $D_{\mathrm{KL}}(q \,\|\, p)$:

In contrast, differentiating the IS-corrected k3 estimator gives us an IS-corrected estimator of the correct gradient.

Hence, we can observe that using the IS-corrected k3 estimator is important for getting the correct estimates, both in terms of values and gradients.

Loss Aggregation

The second group of tweaks are about how to aggregate the token-level objectives into a sequence-level objective. GRPO's loss aggregation can introduce some unintended biases and issues in training, which can be mitigated by using different ways of aggregating the token-level objectives.

Token-Level Objectives

Recall that when GRPO aggregates the token-level objectives, it introduces an unintuitive length normalization $1/|y^{(j)}|$:

However, to match the original off-policy policy gradient and PPO objectives, the inner expectation term should simply be an average of the PPO objectives over the group. Therefore, the correct way to aggregate the loss should exclude the length normalization, as proposed in Dr. GRPO ⁷:

The authors of Dr. GRPO additionally argue that without this fix, GRPO is implicitly biased towards short, correct and long, incorrect responses. This is not desirable as the whole concept of test-time scaling is to generate long, correct responses, meanwhile without wasting tokens. This also aligns with some early observations of DeepSeek-R1, that the model tends to generate excessively long responses with low-quality, repetitive patterns.

A similar fix is also proposed by DAPO ⁸, which gives us a slightly different objective, where we normalize by the total number of tokens in the group, instead of normalizing each response by its own length. This way, we can still maintain the length normalization, while avoiding the bias on response lengths. The objective of DAPO is given by:

Though DAPO is still biased as it includes the length normalization, modern implementations of GRPO mostly use DAPO-style loss aggregation as default. This is probably because without length normalization, the magnitude of the loss can vary significantly across responses with different lengths, which can lead to instability in training. DAPO's way of normalizing by the total number of tokens in the group can mitigate this issue while still avoiding the bias on response lengths.

Sequence-Level Objectives

However, in GRPO-style advantage estimation, it is actually more natural to take a sequence-level view. Currently we are calculating advantages at sequence-level, but simply broadcasting those advantages to all tokens, equally. This becomes very weird, because the GRPO advantage is a better-than-average in the sequence-level, not token-level. In fact, even if it were not GRPO, any sequence-level advantage estimator would have the same issue. Therefore, when we are using sequence-level advantages, it is more natural to view the entire sequence as a single decision and compute the IS ratios at sequence-level as well. This is essentially treating LLM generation as a bandit problem, not a token-level MDP.

With this sequence-level view, the off-policy policy gradient objective should be:

However, naively applying clipping and computing gradients on this sequence-level objective is not a good idea, because the IS ratios have very high variance and large numerical range at sequence-level. Therefore, we need to be careful about how to apply clipping and how to compute gradients. ¹

GSPO ² proposes a practical way to apply clipping at sequence-level, which is to first add length normalization, then apply sequence-level clipping on the length-normalized IS ratios. The objective of GSPO is given by:

Ignoring the clipping, this results in using the geometric mean of the token-level IS ratios as the sequence-level IS ratio. Note that the original GRPO objective without clipping is equivalent to using the arithmetic mean of the token-level IS ratios as the sequence-level IS ratio. Hence, GSPO can also be viewed as a variant of GRPO with a different way of aggregating the token-level IS ratios.

But in some scenarios, we may want to use a mixture of sequence-level and token-level advantages, such as certain multi-turn dialogue settings where we want to have turn-level advantages as well. The authors of GSPO introduce a token-level variant of GSPO as well, which is called GSPO-Token, where we can apply clipping at token-level and use token-level advantages.

We can derive it by first computing the gradient of GSPO without clipping:

Then, applying clipping to the IS ratios gives us the objective of GSPO-Token:

Since we have more fine-grained control over the IS ratios at token-level, it is safer to add customized token-level advantages as well, instead of using the same sequence-level advantage for all tokens. ²

But notably, a subsequent work ⁴ from the same authors shows that the sequence-level objective is in fact already very close to the token-level objective (where we broadcast the sequence-level advantage to all tokens), so using the token-level objective with sequence-level advantages is not really a problem. Using the first-order approximation $\prod_{t=1}^{T}(1 + \delta_t) \approx 1 + \sum_{t=1}^{T}\delta_t$ when $\delta_t$'s are small, we have:

The symbol $\overset{\nabla_\theta}{\equiv}$ means that the two objectives have the same gradient with respect to $\theta$, and therefore are equivalent for optimization. Hence, we can see that the original GRPO-style token-level objective is already a very good approximation of the correct sequence-level objective, at gradient-level. Therefore, we can just stick to the original GRPO-style token-level objective, without worrying too much about issues regarding interpretation of the advantages.

Clipping only the IS Ratios

The last group of tweaks are about how to apply clipping to the IS ratios. While PPO and GRPO originally apply clipping to the entire product of IS ratios and advantages, some recent algorithms consider applying clipping only to the IS ratios instead. This idea is motivated especially in the context of LLM training, and is currently considered as a promising direction for improving the stability and performance of LLM-RL algorithms.

CISPO

This idea was proposed in CISPO as part of the training recipe for MiniMax-M1 ⁹. In standard PPO or GRPO-style objectives, clipping is applied to the full update term. When the ratio goes outside the clipping range, the clipped branch can produce zero gradient with respect to the token log-probability. This is intentional from a trust-region perspective, but it can be problematic in LLM reasoning training.

The issue is especially visible for rare but important tokens. Reasoning models often improve by increasing the probability of tokens that trigger reflection or correction, such as Wait, Hmm, Actually, or Aha. These tokens may have very low probability under the old policy, so their IS ratios can become large as soon as the current policy starts assigning them more probability. Under ordinary PPO/GRPO clipping, these updates may be clipped away, which means exactly the tokens we want the model to learn from can stop receiving useful gradients. Humanity invented a training signal and then carefully deleted it. Very elegant.

To avoid this, CISPO starts from the off-policy token-level policy gradient:

CISPO then clips the IS weight itself, while keeping the log-probability term differentiable:

The important difference is that clipping no longer removes the token-level learning signal. The clipped ratio only controls the scale of the update, while the gradient still flows through $\log \pi_\theta(y_t \mid x,y_{<t})$. Thus, CISPO behaves more like a variance-controlled policy gradient estimator than a pessimistic PPO-style lower-bound objective.

Truncated Importance Sampling (TIS)

A closely related variant is truncated importance sampling (TIS) ¹⁰. Instead of clipping both sides of the ratio, we can only truncate the upper tail:

This keeps the same CISPO-style stop-gradient structure, but focuses only on preventing extremely large ratios from dominating the update. Intuitively, this is useful when the main danger is not that a token has become too unlikely under the current policy, but that a small number of tokens have huge ratios and therefore create high-variance gradients. This style of loss is found to be very effective and stable in scaling up LLM-RL training, according to recent studies.

So far, importance sampling has been applied at the token level. However, token-level correction does not fully correct the mismatch between the old rollout distribution and the current policy distribution. Therefore, it is also natural to consider applying importance sampling at sequence-level. This gives:

Then adding truncation to the sequence-level IS ratio gives the sequence-level TIS objective:

Masking Sequences

Masked Importance Sampling (MIS)

TIS still gives some weight to sequences with very large ratios. If $\rho(y \mid x) \gg C$, the sequence is still used with weight $C$. This is reasonable if high-ratio samples are merely rare but valid. However, in LLM-RL, extremely large ratios may also indicate that the sample is outside the trustworthy overlap between the rollout policy and the training policy. This can happen due to stale rollouts, training-inference mismatch, or simply because long-horizon generation found a bizarre corner of the distribution.

This motivates masked importance sampling (MIS) ¹⁰. Instead of softly clipping high-ratio sequences, MIS rejects them entirely. The strict sequence-level IS version is

This can be interpreted as a hard trust region. If the sequence ratio is too large, then the sample is treated as unreliable and removed from the update. Compared with Seq-TIS, this sacrifices sample efficiency, since rejected samples provide no gradient. The benefit is robustness, since if high-ratio samples are actually OOD or corrupted by training-inference mismatch, clipping them to $C$ still lets them affect the update, while masking removes them completely.

Geometric Sequence Masking

In practice, one may also use a mask-only version of Seq-MIS as a safety wrapper around another objective:

This is no longer a full IS estimator, but rather a practical filtering heuristic. We first check whether the sequence is too far off-policy, and only then apply the base loss. However, a problem with sequence-level masking is that the raw sequence ratio is length-dependent, as it is a product of token-level ratios. Even if each per-token ratio is only slightly larger than $1$, the product can become very large for long responses. This means that a fixed threshold $C$ can systematically reject long reasoning chains, even when the average per-token mismatch is small. For reasoning models, this is a fairly terrible failure mode, as the model can be punished not because the reasoning trajectory is bad, but because it is long.

Geometric sequence masking ¹⁰ fixes this by normalizing the log-ratio by sequence length:

Intuitively, Geo-Mask filters out sequences whose average per-token log IS ratio is too extreme. Unlike Seq-MIS, it does not penalize a sequence merely for being long. A 200-token answer and a 20,000-token answer are judged by the same average per-token divergence criterion, which is much more appropriate for long-CoT or agentic, reasoning RL.

Importantly, Geo-Mask by itself is only a filtering rule. To recover a valid IS estimator, the mask should be placed in front of an objective that already contains a proper IS correction. In that case, the mask defines which samples are trusted, while the underlying IS estimator still determines how accepted samples are weighted. In fact, DeepSeek-V3.2 applies a this mask in front of GRPO, though only using an upper constraint to remove sequences whose sampling policy and training policy have diverged too much. The original authors of the RL-collapse analysis similarly suggest combining geometric sequence masking with Tok-TIS.

Dynamic Sampling

Dynamic Sampling is another practical trick proposed by DAPO ⁸. In GRPO, if all responses in a group receive the same reward, then all group-relative advantages become zero. For binary correctness rewards, this happens when the group accuracy is either $0$ or $1$. These prompts therefore contribute no useful policy-gradient signal, but still occupy batch space, because naturally we must pay compute to learn nothing.

DAPO handles this by over-sampling prompts and filtering out groups whose accuracy is exactly $0$ or $1$. Equivalently, only groups with at least one correct and at least one incorrect response are kept:

This keeps the number of prompts with nonzero relative advantages more consistent across batches. Although it requires sampling extra groups, DAPO reports that it can still improve wall-clock efficiency because training needs fewer effective update steps.

Misc

Clip-Higher

Clip-Higher is a simple asymmetric clipping strategy from DAPO ⁸. Instead of using the same clipping range on both sides, it uses a larger upper clipping threshold:

The motivation is that the upper clipping bound can make it difficult to increase the probability of low-probability but high-reward tokens. In long-CoT RL, these low-probability tokens may correspond to exploratory reasoning patterns, so clipping them too aggressively can reduce diversity and lead to entropy collapse. By increasing only the upper clipping range, Clip-Higher gives the policy more room to upweight useful exploratory tokens, while still keeping the lower clipping range relatively tight for stability.

Removing STD normalization

Another small but important change is to remove the standard-deviation normalization in GRPO advantages, as proposed by Dr. GRPO ⁷. Standard GRPO computes group-relative advantages by subtracting the group mean and dividing by the group standard deviation, but Dr. GRPO removes the standard-deviation term:

The intuition is that dividing by the group standard deviation changes the effective scale of the update depending on the reward distribution inside each sampled group. This can introduce another source of optimization bias, especially when rewards are sparse or nearly binary. Removing the STD normalization makes the advantage closer to a plain reward-minus-baseline estimator, which is simpler and more interpretable. DeepSeek-V3.2 also adopts this style of advantage normalization.

Remove the KL Penalty

Another common simplification in modern LLM RL is to remove the explicit KL penalty against a frozen reference policy. The motivation is that KL regularization plays a different role in RLHF and reasoning RL. In RLHF, which was the beginning of most LLM RL work, the reward model is learned from data near the reference policy, so staying close to the reference helps avoid reward-model extrapolation. However, as modern LLM RL has shifted towards verifier-based reasoning, the reward is often more robust to distribution shift, so the KL penalty may be less necessary. Moreover, long-CoT reasoning may require the policy to move far from the initial model, producing longer traces, new reflection patterns, and different exploration behavior. A frozen reference can therefore become overly restrictive.

Numerous recent works, including DAPO ⁸, Dr. GRPO ⁷, or Open-Reasoner-Zero ¹¹ all report that removing the KL penalty gives the best stability and final performance. This no-KL choice has also become common in frontier open models or training frameworks.

However, removing KL entirely is not the only reasonable choice. ProRL ¹² argues that for prolonged RL, especially when starting from a strong distilled checkpoint, some KL control can help maintain entropy and stability. Instead of using a permanently frozen reference, ProRL periodically resets the reference policy to a recent online-policy checkpoint:

This gives a moving trust region. The policy is still discouraged from making abrupt updates, but it is not permanently tied to an outdated reference model.

This can also be seen as a mirror-descent-style update ¹³:

where $\eta = 1/\beta$ in the original KL penalty formulation. This is a more principled way to derive the moving-reference KL penalty, which can be viewed as a trust-region-style update that encourages the policy to stay close to its recent self, rather than an outdated reference.

To conclude, removing KL gives the policy maximum freedom and is often effective for verifier-based reasoning RL, while moving/adaptive KL methods try to preserve stability without over-constraining exploration. The best choice may depend on the specific task, reward model, and training setup, so it is worth experimenting with different KL strategies.

References