11.7 Latest Research Advances (2025–2026)

📖 "From DeepSeek-R1 making the cover of Nature to DAPO/VAPO setting new reasoning benchmarks, Agentic-RL is moving from the laboratory to engineering practice at an astonishing pace. This section will give you a panoramic view of the cutting-edge research in this field."

⏰ Timeliness Note: The content of this section is updated to March 20, 2026. Since this field is developing extremely rapidly, readers are advised to supplement with open-source projects like Awesome-RL-Reasoning-Recipes for the latest developments.

7.1 Overview: The Paradigm Shift from RLHF to Reasoning RL

The past two years (2025–2026) have been years of explosive development in the field of large model reinforcement learning. Marked by DeepSeek-R1 making the cover of Nature, RL training of LLMs has leaped from an auxiliary role in "aligning human preferences" (RLHF) to the core technology for stimulating model reasoning capabilities. We can use a timeline to overview the key milestones:

2024.09  OpenAI o1 released, first demonstrates potential of "test-time compute scaling"
2025.01  DeepSeek-R1 released, pure RL training stimulates autonomous reasoning, uses GRPO algorithm
2025.01  Kimi k1.5 released, 128K long-context RL training, Long2Short distillation technique
2025.02  QwQ-32B released, demonstrates reasoning RL training effects at medium scale
2025.03  DAPO open-sourced, proposes reproducible large-scale RL training solution
2025.04  VAPO released, value-augmented PPO framework, AIME 2024 reaches 60.4
2025.06  OpenAI o3 released, reasoning capability further leaps
2025.07  GSPO proposed (Qwen team), sequence-level policy optimization stabilizes MoE training, trains Qwen3
2025.08  Self-Aligned Reward (SAR) proposed, uses perplexity signals to address overthinking
2025.10  PURE framework released, min-form credit assignment solves reward hacking
2025.12  Co-rewarding (ICLR 2026) proposes self-supervised RL learning scheme
2026.01  RLVR new paradigm: efficient RL method based on problem decomposition
2026.02  DRQA dynamic reasoning quota allocation, token cost reduced by 31%
2026.03  CoRLHF proposes cooperative policy-reward joint optimization

These works can be summarized into the following core research directions:

Let's dive into the important papers in each direction one by one.

7.2 Reasoning Models: Pure RL Training Stimulates Autonomous Reasoning

7.2.1 DeepSeek-R1: The Nature Cover Breakthrough

Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (Nature, 2025) [1]

DeepSeek-R1 is the most milestone-significant work in this field. Its core finding is:

Through RL training alone (without manually annotated reasoning chains), models can autonomously emerge advanced cognitive capabilities such as multi-step reasoning, self-reflection, and dynamic strategy adjustment.

Core Technical Points

1. GRPO Algorithm: Uses Group Relative Policy Optimization (see Section 11.5), optimizes policy through within-group response competition, avoids expensive Critic networks, total training cost approximately $294,000.

2. Multi-phase Training Framework:

   • R1-Zero Phase: Uses only result correctness as reward (Verifiable Reward RL, RLVR), no SFT data used. The model spontaneously emerges "Aha moments" — learning to self-reflect and correct errors during the reasoning process.
   • R1 Phase: Building on R1-Zero, incorporates a small amount of high-quality SFT data and human preference alignment to improve comprehensive capabilities.

3. Verifiable Rewards (RLVR): Reward signals come from automatically verifiable tasks (such as final answers to math problems), no manual annotation needed.

Key Experimental Results

• Achieves SOTA on 21 benchmarks including MMLU, AIME 2024, LiveCodeBench
• R1-Zero demonstrates the possibility of "learning to reason from scratch" — reasoning chain length spontaneously grows during RL training
• Maintains strong reasoning capability after distillation to 7B/14B small models

Why Is It Important?

DeepSeek-R1 proves two key arguments:

1. RL can stimulate latent reasoning capabilities from pre-training — these capabilities are difficult to fully release through SFT or prompt engineering
2. Reasoning capabilities can "emerge" in a pure RL environment — without relying on manually annotated reasoning chains as demonstrations

7.2.2 Kimi k1.5: Breakthrough in Long-Context RL

Paper: Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025) [2]

Kimi k1.5, developed by the Moonshot AI team, makes unique contributions in several areas:

Core Innovations

1. 128K Long-Context RL Training: Extends the RL training context window from the traditional 4K-8K to 128K tokens, improving training efficiency through Partial Rollout Reuse.

2. Simplified RL Framework: Abandons Monte Carlo Tree Search (MCTS) and value functions, directly optimizes the model through improved Online Mirror Descent, greatly reducing computational burden.

3. Long2Short Distillation Technique: "Compresses" long-context reasoning capability into short-context models. Specifically:

   • First train strong reasoning capability in long-context settings
   • Then through knowledge distillation, teach short-context models to "refine" reasoning

Key Results

• Surpasses GPT-4o by 550% on short tasks like LiveCodeBench
• Long2Short technique proves that long-chain reasoning capability can be compressed without significant loss
• First demonstrates feasibility of RL training with 128K context window

7.2.3 QwQ-32B: Reasoning RL at Medium Scale

Paper: QwQ: Reflect and Question to Understand the World (Alibaba, 2025) [3]

QwQ-32B is a medium-scale reasoning model released by Alibaba's Tongyi team. Its significance lies in proving that models at the 32B parameter scale can also obtain strong reasoning capabilities through RL training.

Technical Characteristics

• Based on Qwen2.5-32B for RL training
• Approaches DeepSeek-R1's performance on mathematical reasoning tasks
• Training cost far lower than 670B-scale models

Why Is It Important?

QwQ proves that reasoning RL is not "exclusive to large models" — medium-scale models can also achieve significant reasoning capability improvements through appropriate RL training. This has great practical value for resource-constrained teams and edge deployment scenarios.

7.2.4 OpenAI o1/o3: Test-Time Compute Scaling

Models: OpenAI o1 (2024.09) / OpenAI o3 (2025.06) [4]

Although OpenAI has not published complete technical reports, the o1 and o3 series models have had a profound impact on the industry:

Core Concept: Test-Time Compute Scaling

Traditional Scaling Laws focus on training-time compute scaling (larger models + more data). The o1/o3 series proposes another dimension:

Investing more computation at inference time (longer thinking chains, more search/verification) can also continuously improve model capabilities.

This means there are two complementary scaling paths:

1. Training-time scaling: increase model size, increase data
2. Inference-time scaling: increase reasoning steps, verification loops

Impact on the Field

• Spawned the new category of "reasoning models"
• Drove the development of RL algorithms for reasoning tasks like GRPO, DAPO, VAPO
• Triggered attention to "reasoning efficiency" — the Overthinking problem emerged

7.3 RL Algorithm Improvements: Making Large Model RL Training More Stable and Efficient

7.3.1 DAPO: Large-Scale Reproducible RL Training

Paper: DAPO: An Open-Source LLM Reinforcement Learning System at Scale (2025) [5]

DAPO (Decoupled Clip and Dynamic Sampling PPO), proposed by ByteDance's Seed team, has the core goal of solving the reproducibility problem of large-scale RL training.

Core Techniques

1. Decoupled Clipping: Traditional PPO uses symmetric clipping $\epsilon$;DAPOseparatestheupperandlowerclippingboundaries:-$\epsilon_{\text{high}}$(larger):encouragesexplorationofgoodresponses-$\epsilon_{\text{low}}$ (smaller): strictly suppresses bad responses

   This asymmetric design lets the model "boldly explore good behavior" while "conservatively suppressing bad behavior."

2. Dynamic Sampling: Dynamically adjusts the number of samples per question based on training progress:

   • Early training: more sampling, increase exploration
   • Late training: less sampling, fine-grained optimization

3. Token-Level Policy Constraint: Applies KL constraints at the token level rather than sequence level, more precisely controlling policy drift.

Open-Source Contribution

DAPO fully open-sources training code and datasets (based on Qwen2.5-32B), making it one of the most reproducible large-scale RL training solutions currently available.

7.3.2 VAPO: Value-Augmented PPO

Paper: VAPO: Efficient and Reliable RL Framework for Advanced Reasoning Tasks (ByteDance Seed, 2025) [6]

VAPO (Value-based Augmented PPO) is a follow-up to DAPO, specifically targeting challenges in long-chain reasoning tasks.

Core Problems

In long-chain reasoning (such as mathematical proofs, complex programming), RL training faces three major challenges:

1. Value model bias: Critic network's value estimation for long sequences is inaccurate
2. Heterogeneous sequence lengths: Response lengths within the same batch vary greatly
3. Sparse rewards: Only the final answer has a reward signal

Core Techniques

1. Value Pretraining: Uses Monte Carlo returns to pretrain the Critic network, reducing initialization bias.

2. Decoupled GAE:

   • Uses $\lambda_V = 1.0$forthevaluenetwork(lowbias,highvariance)-Uses$\lambda_P = 0.95$forthepolicynetwork(balancedbiasandvariance)3.\ast \ast Length-AdaptiveGAE\ast \ast :Dynamicallyadjusts$\lambda$basedonsequencelength:$$\lambda =1-\frac{1}{0.05\cdot l}$$Where$l$isthesequencelength.Longsequencesuselarger$\lambda$(reducebias),shortsequencesusesmaller$\lambda$(reducevariance).4.\ast \ast Clip-HigherExploration\ast \ast :Usesasymmetricclipping$\epsilon_{\text{high}} = 0.28$,$\epsilon_{\text{low}} = 0.2$, encouraging diverse sampling.

Key Results

VAPO surpasses DeepSeek-R1-Zero (671B) using only Qwen-32B and 5,000 training steps, with completely collapse-free training.

7.3.3 GRPO Variants and Improvements

Since DeepSeek-R1 proposed GRPO, multiple papers have improved upon it:

Among these, GSPO is the most practically impactful improvement — it has already been used by Alibaba's Qwen team to train the Qwen3 series models. For detailed principles and implementation of GSPO, see Section 11.5's GSPO chapter.

7.4 Reward Design: How to Tell the Model What Good Reasoning Is?

The reward function is the "soul" of RL training. In 2025–2026, three important directions emerged in reward design.

7.4.1 Self-Aligned Reward (SAR): Leveraging Model Internal Signals

Paper: Self-Aligned Reward: Towards Effective and Efficient Reasoners (UIUC & Amazon AWS, 2025) [7]

Core Idea

SAR's core insight is: differences in the model's internal perplexity (PPL) can serve as high-quality reward signals.

Specifically, SAR computes the perplexity difference under two conditions:

$$r_{\text{SAR}}(y|x)=\frac{\text{PPL}(y)-\text{PPL}(y|x)}{\text{PPL}(y)}$$Where:-$\text{PPL}(y|x)$:perplexityofgeneratingresponse$y$givenquestion$x$-$\text{PPL}(y)$:perplexityoftreatingresponse$y$ as independent text

Intuitive explanation:

• High SAR: response highly depends on the question (targeted, concise response)
• Low SAR: response weakly associated with the question (possibly verbose, generic content)

Why Is It Effective?

1. No external reward model needed: leverages the model's own language modeling capability
2. Fine-grained scoring: can distinguish "correct and concise" vs "correct but verbose"
3. Cross-task generalization: trained on math data, also effective on non-math tasks like logical reasoning

Experimental Results

Across 4 base models and 7 datasets:

• Average accuracy improvement of 4%
• Output length reduced by 30%

7.4.2 Co-rewarding: Self-Supervised RL Learning

Paper: Co-rewarding: Self-Supervised RL for LLM Reasoning (ICLR 2026) [8]

Core Problem

Self-rewarding RL (letting the model score itself) is prone to training collapse — the model learns to generate responses that are "easy to give itself high scores" rather than "truly good."

Solution

Co-rewarding introduces complementary supervision signals:

1. Generate paraphrased versions of the same question
2. Use responses to paraphrased questions as auxiliary evaluation for original question responses
3. Evaluations in both directions mutually constrain each other, preventing collapse

Key Results

• Performance improvement of 12.9% on reasoning tasks (without ground truth labels)
• Training process significantly more stable

7.4.3 CoRLHF: Cooperative Policy-Reward Joint Optimization

Paper: CoRLHF: Reinforcement Learning from Human Feedback with Cooperative Policy-Reward Optimization (Expert Systems with Applications, 2026) [9]

Core Innovation

Traditional RLHF has two steps: first train the reward model, then use the reward model to train the policy. This causes a distribution mismatch problem — the data distribution seen during reward model training is inconsistent with the data distribution generated during policy optimization.

CoRLHF merges policy optimization and reward model optimization into one iterative process:

1. Policy generates new data
2. Reward model updates on new data
3. Policy optimizes on updated rewards
4. Iterative loop

This approach bridges RLHF and RLAIF, maintaining alignment quality while reducing dependence on human feedback.

7.4.4 Endogenous Reward: LLM as a Built-in Reward Model

Paper: Related work by Zhi-Hua Zhou's team (Nanjing University, 2025) [10]

Disruptive Finding

This research finds: LLM's next-token prediction capability itself contains a general reward function (Endogenous Reward).

That is, the language model distribution learned during pre-training has already implicitly encoded the judgment capability of "what is good output," without needing to additionally train a reward model.

Practical Significance

• Reduces one component (reward model) in the RLHF pipeline
• Reduces the risk of error accumulation
• Surpasses traditional reward models on multiple alignment benchmarks

7.5 Overthinking and Reasoning Efficiency

With the popularization of reasoning models, a new problem has emerged: Overthinking — models generate verbose reasoning chains even for simple problems, wasting computational resources and potentially reducing accuracy.

7.5.1 Problem Analysis: Why Do Reasoning Models "Think Too Much"?

The root of overthinking lies in the reward structure of RLVR (RL with Verifiable Rewards):

As long as the final answer is correct, regardless of how long or redundant the reasoning process is, the model receives the same reward.

This leads to two problems:

1. Reward inflation: standard RL's summation-form credit assignment makes models prefer generating more steps
2. Undifferentiated incentives: cannot distinguish "concisely correct" from "verbosely correct"

7.5.2 PURE: Min-Form Credit Assignment

Paper: Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning (2025) [11]

Core Insight

Traditional RL defines trajectory value as the sum of future rewards:

$$V_{\text{sum}}(s_t)={\sum }_{k=t}^T{\gamma }^{k-t}{r}_k$$PUREproposesreplacingthesumwiththe\ast \ast minimum\ast \ast :$$V_{\text{min}}(s_t)=\min (r_t,r_{t+1},\dots ,r_T)$$

Intuition: the strength of a reasoning chain depends on its weakest link.

Implementation

PURE converts process rewards to new rewards through a temperature parameter $T$, making standard RL algorithms (PPO/GRPO)'s summation formula mathematically equivalent to taking the minimum — no need to modify the underlying algorithm, only reward preprocessing needed.

Experimental Results

• Sum-form training collapses almost immediately
• Min-form training improves stably
• Sample efficiency improved 2-3×

7.5.3 DRQA: Dynamic Reasoning Quota Allocation

Paper: DRQA: Dynamic Reasoning Quota Allocation for Controlling Overthinking in Reasoning Large Language Models (2026) [12]

Core Observation

An interesting finding: when models batch process multiple questions (rather than processing one by one), total output length significantly shortens — models seem to implicitly distinguish problem difficulty and "compress" reasoning for simple problems.

Method

1. Build preference data:

   • Reasoning chains generated individually (verbose version)
   • Reasoning chains generated in batches (refined version)
   • Annotate preferences by correctness and conciseness

2. Use GRPO to train models to simultaneously optimize logical correctness and reasoning conciseness

Results

• Reasoning token cost reduced by 31%
• Accuracy actually improves
• Shortens most for simple problems, maintains sufficient reasoning for complex problems

7.5.4 DEER: Dynamic Early Exit in Reasoning

Paper: Dynamic Early Exit in Reasoning Models (DEER) (2026) [13]

DEER is a training-free inference-time optimization method:

1. Monitors model confidence in real-time during reasoning
2. Triggers early exit when model is highly confident about the current answer
3. Simple problems end quickly, complex problems continue thinking

Results

• Reasoning chain length shortened by 19.1%–80.1%
• Accuracy improved by 0.3%–5.0%
• No additional training needed, plug-and-play

7.5.5 Method Comparison

7.6 RLVR: Reinforcement Learning with Verifiable Rewards

RLVR (Reinforcement Learning with Verifiable Rewards) is one of the hottest research directions in 2025–2026, and is also the key to DeepSeek-R1's success.

7.6.1 What Is RLVR?

Unlike traditional RLHF which relies on manually annotated preference data, RLVR uses automatically verifiable signals as rewards:

7.6.2 RLVR Problems and Improvements

Problem Decomposition Framework (Renmin University & ByteDance, 2026) [14]:

Traditional RLVR only gives rewards at the final answer (sparse rewards), causing credit assignment difficulties in long-chain reasoning. This work proposes the Decomposer-Reasoner Framework:

1. Decomposer: decomposes complex problems into sub-problems
2. Reasoner: solves sub-problems step by step
3. Dense rewards: each sub-problem solution has a verifiable reward

This approach converts sparse rewards to dense rewards, significantly improving the exploration efficiency of RL training.

7.7 RL Training for Agentic Tasks

Most of the above discussion is about RL training for reasoning tasks (math, code). A more cutting-edge direction is applying RL to truly Agentic tasks — scenarios requiring tool calling, environment interaction, and multi-step decision making.

7.7.1 AgentPRM: Process Reward Models for Agent Evaluation

In multi-turn Agent tasks (such as web navigation, API calls), evaluating only the final result is insufficient — the quality of each decision step needs to be evaluated. AgentPRM introduces Process Reward Models to evaluate the Agent's intermediate decisions.

7.7.2 R³L: Reflect-then-Retry RL

R³L (Reflect-then-Retry RL) targets failure recovery in Agent tasks:

1. When the Agent fails, generate language feedback diagnosing the error cause
2. Restart from the failure point, using feedback to avoid repeating mistakes
3. Greatly reduces rollout cost

7.7.3 DeepSWE: RL Training for Software Engineering Agents

DeepSeek team's DeepSWE demonstrates that RL-trained software engineering Agents can match closed-source models' SWE-bench performance, proving RL's potential in complex Agentic tasks.

7.8 Open Challenges and Future Directions

Despite rapid progress, the field still faces many open challenges:

7.8.1 Reward Hacking

Models may find loopholes in reward functions to "cheat" rather than truly improving capabilities. For example:

• Generating long text that "looks like reasoning" but is actually nonsense
• Using formatting tricks (such as specific keywords) to get high rewards
• Learning to "self-deceive" in self-evaluation

7.8.2 Training Stability

Large model RL training is still not stable enough:

• KL divergence management: excessive policy drift causes catastrophic forgetting
• Reward scale: inconsistent scales across different reward dimensions
• Data diversity: diversity of training data directly affects exploration quality

7.8.3 Generalization Capability

Current RL-trained reasoning capabilities are mainly validated in math and code domains; generalization to the following areas still needs exploration:

• Open-domain reasoning (scientific reasoning, commonsense reasoning)
• Multimodal reasoning (vision-language, video understanding)
• Cross-lingual reasoning

7.8.4 Efficiency and Cost

RL training computational costs are still high:

• Large amounts of rollout sampling
• Multiple models (Policy, Reference, possibly Critic) simultaneously in GPU memory
• Memory and time overhead of long-sequence reasoning

7.8.5 Future Outlook

Based on current research trends, we expect the following directions to become hot topics:

7.9 Paper List

The following are the main papers covered in this section, organized by topic:

Reasoning Models

RL Algorithms

Reward Design

Reasoning Efficiency

7.10 Recommended Reading Path

If you are new to this field, it is recommended to read in the following order:

Beginner path:
1. DeepSeek-R1 paper (understand core ideas of RLVR + GRPO)
   ↓
2. GSPO paper (understand advantages of sequence-level optimization over token-level)
   ↓
3. DAPO paper + code (hands-on reproduction of large model RL training)
   ↓
4. VAPO paper (understand role of value function in long-chain reasoning)
   ↓
5. SAR / PURE papers (understand reward design and overthinking problems)
   ↓
6. Kimi k1.5 / QwQ (understand different teams' technical approaches)

If you are interested in specific topics:

• Want to train reasoning models → Focus on DeepSeek-R1 + GSPO + DAPO + VAPO
• Want to design reward functions → Focus on SAR + PURE + Co-rewarding
• Want to optimize reasoning efficiency → Focus on DRQA + DEER + PURE
• Want to do Agent RL → Focus on DeepSWE + AgentPRM + R³L
• Want to train MoE models → Focus on GSPO + DAPO

Section Summary

In 2025–2026, the Agentic-RL field underwent a fundamental transformation from "alignment auxiliary tool" to "core capability stimulation engine." Several key trends are worth noting:

1. RL from auxiliary to core: RL is no longer just used for "alignment," but for stimulating latent reasoning capabilities from pre-training
2. Algorithms from complex to practical: from PPO's four-model architecture to GRPO's two-model architecture, then to GSPO's sequence-level optimization and VAPO's value-augmented solution, training is becoming increasingly efficient and stable
3. Rewards from external to internal: from manual annotation to verifiable rewards to model internal signals, reward design is becoming increasingly self-consistent
4. Focus from "stronger" to "more efficient": the overthinking problem has spawned a series of reasoning efficiency optimization solutions

These advances are gradually making the vision of "letting models learn autonomously through practice" a reality.