11.1 What Is Agentic-RL

The Paradigm Shift from "Prompt Engineering" to "Training Optimization"

In previous chapters of this book, our core method for building Agents was Prompt Engineering + tool calling: carefully writing system prompts, defining tool interfaces, and having general-purpose large models play specific roles to complete tasks. This approach has a low development barrier and is fast to develop, but has a fundamental bottleneck:

The Agent's capability ceiling = the base model's general capability ceiling.

No matter how cleverly the prompts are designed, if the base model has systematic deficiencies in certain types of reasoning (such as multi-step mathematical reasoning, complex code repair, long-horizon planning), the Agent's performance cannot break through this ceiling.

Agentic-RL (Agentic Reinforcement Learning) proposes a fundamentally different approach: rather than guiding model behavior through prompts at inference time, use reinforcement learning signals during training to let the model autonomously learn high-quality Agent strategies. The core insight of this paradigm comes from experimental findings in DeepSeek-R1 [3] — pure RL training can cause reasoning chains to emerge in models that humans never explicitly taught.

Systematic Comparison of the Two Paradigms

When Should You Choose Agentic-RL?

Agentic-RL is not a silver bullet. The following is a decision framework based on practical experience:

Scenarios suitable for investing in Agentic-RL:

• ✅ Tasks have objectively verifiable evaluation criteria (code test pass rate, mathematical answer correctness, API call success rate)
• ✅ Tasks are high-frequency and repetitive, so training costs can be amortized over long-term benefits
• ✅ The current base model has systematic, improvable deficiencies on the task
• ✅ Sufficient training data is available or can be generated automatically

Scenarios not suitable for Agentic-RL:

• ❌ One-time, low-frequency open-ended tasks (insufficient ROI)
• ❌ Tasks that cannot be objectively quantified for evaluation (e.g., open-ended creative writing)
• ❌ Base model + prompt approach has already reached an acceptable level
• ❌ Lack of GPU compute resources (GRPO training for a 7B model requires at least 1× A100 40GB)

Theoretical Foundation of Agentic-RL: The MDP Framework

Markov Decision Process Modeling

The theoretical foundation of Agentic-RL is the Markov Decision Process (MDP) [1]. The Agent's task execution process is formalized as a finite-horizon MDP:

$$\mathcal{M}=⟨\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\gamma ⟩$$

The correspondence of each element to the Agent scenario is as follows:

The training objective is to maximize expected cumulative reward:

$${\theta }^{\ast }=\arg \underset{\theta }{\max }{\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\gamma }^t{r}_t\right]$$

Term-by-term explanation:

• ${\theta }^{\ast }$: optimal model parameters — the target we want to find through training
• $\arg {\max }_{\theta }$: among all possible parameters $\theta$, find the one that maximizes the objective function
• ${\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[\cdot ]$: expectation operator — since model generation is stochastic (temperature > 0), the trajectory $\tau$ generated for the same problem differs each time; we optimize the average performance across all possible trajectories, not the result of any specific generation
• ${\sum }_{t=0}^T{\gamma }^t{r}_t$: discounted cumulative reward — the immediate reward $r_t$ at each step of the trajectory is weighted and summed by the time discount ${\gamma }^t$; when $\gamma <1$, near-term rewards have greater weight than distant rewards (in Agentic-RL, $\gamma =1.0$ is typically used, i.e., no discount, because we care about the final task completion quality rather than temporal differences in intermediate steps)
• $\tau =(s_0,a_0,s_1,a_1,\dots ,s_T)$: a complete interaction trajectory, recording the complete state-action sequence from initial state to terminal state

Intuitive understanding: the meaning of this objective function is — adjust model parameters $\theta$ so that the model can obtain as high a cumulative reward as possible on average when facing various tasks. This is consistent with the intuition of human learning: through extensive practice (sampling trajectories), continuously adjust the strategy (update parameters) to continuously improve average performance.

Formal Description of the Agent Interaction Loop

Six Core Capability Dimensions

After Agentic-RL training, models can achieve systematic improvements in the following six dimensions [2]:

Two-Phase Training Paradigm

Contemporary mainstream Agentic-RL training follows the SFT → RL two-phase paradigm [3], with each phase playing an irreplaceable role.

Phase 1: SFT (Supervised Fine-Tuning) — Policy Initialization

Core objective: adjust the base model's policy distribution ${\pi }_0$ to an initial policy ${\pi }_{SFT}$ with basic Agent behavior formats.

The training objective in the SFT phase is to maximize log-likelihood:

$${\mathcal{L}}_{SFT}(\theta )=-{\mathbb{E}}_{(x,y^{\ast })\sim {\mathcal{D}}_{SFT}}\left[\log {\pi }_{\theta }(y^{\ast }\mid x)\right]$$

Term-by-term explanation:

• $-{\mathbb{E}}_{(x,y^{\ast })\sim {\mathcal{D}}_{SFT}}$: negative sign + expectation symbol, indicating averaging over all samples $(x,y^{\ast })$ in the supervised fine-tuning dataset ${\mathcal{D}}_{SFT}$.
  • $x$: model input (e.g., user instructions, prompts).
  • $y^{\ast }$: the corresponding human-annotated ideal output (i.e., the "ground truth answer").
  • ${\mathcal{D}}_{SFT}$: the collected instruction-answer pair dataset.
• ${\mathcal{D}}_{SFT}=\{(x^{(i)},y^{\ast (i)}){\}}_{i=1}^N$: a high-quality Agent interaction trajectory dataset, where each sample contains input context $x$ (system prompt + user question) and expert demonstration output $y^{\ast }$ (including reasoning process and tool calls)
• $\log {\pi }_{\theta }(y^{\ast }\mid x)$: the log probability of the model generating the expert demonstration sequence $y^{\ast }$ given input $x$. The larger this value (closer to 0), the more the model considers the expert demonstration to be "reasonable output"; adding a negative sign converts it to a loss, and minimizing the loss maximizes the log probability
• In practice, $\log {\pi }_{\theta }(y^{\ast }\mid x)$ is expanded via autoregressive decomposition into a sum of per-token log probabilities: ${\sum }_{t=1}^{|y^{\ast }|}\log {\pi }_{\theta }(y_t^{\ast }\mid x,y_{<t}^{\ast })$, i.e., the generation probability of each token is conditioned on all preceding tokens

Intuitive understanding: the essence of SFT is "imitation learning" — show the model a large number of expert demonstrations, letting the model learn "given this input, what would an expert output?" This phase is similar to "copying calligraphy" — first imitate correct behavioral patterns to establish format norms and basic capabilities.

Training data example:
Input x: "Calculate the result of 1234 × 5678"
Expected output y*:
  <think>
  This requires precise integer multiplication. Should use the calculator tool to ensure accuracy.
  </think>
  <tool_call>calculator(expression="1234 * 5678")</tool_call>

Phase 2: RL (Reinforcement Learning) — Policy Optimization

Core objective: building on ${\pi }_{SFT}$, use reward signals to guide the policy toward ${\pi }^{\ast }$, breaking through the quality ceiling of SFT data.

The training objective in the RL phase is to maximize expected reward while constraining the policy from drifting too far via KL divergence:

$${\mathcal{L}}_{RL}(\theta )=-{\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[R(\tau )\right]+\beta \cdot D_{KL}({\pi }_{\theta }\Vert {\pi }_{SFT})$$

Term-by-term explanation:

• $-{\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[R(\tau )]$: policy loss term — maximize the expected reward of trajectories sampled by the current policy ${\pi }_{\theta }$. The negative sign converts maximization to minimization (gradient descent convention)
• $\beta \cdot D_{KL}({\pi }_{\theta }\Vert {\pi }_{SFT})$: KL divergence penalty term — $D_{KL}$ measures the "distributional distance" between the current policy ${\pi }_{\theta }$ and the SFT initial policy ${\pi }_{SFT}$. When the two distributions are identical, $D_{KL}=0$; the greater the difference, the larger $D_{KL}$. The coefficient $\beta$ controls penalty strength: larger $\beta$ makes the policy more conservative (reluctant to deviate from the SFT model); smaller $\beta$ makes the policy more aggressive (may produce reward hacking behavior).

  💡 What is KL divergence? If you're not yet familiar with KL divergence, we recommend reading Appendix E: KL Divergence Explained in this book, which includes a complete introduction from intuitive understanding to mathematical definition. Simply put, $D_{KL}(P\Vert Q)$ answers the question: "If the true distribution is $P$, how much information is lost on average by approximating it with distribution $Q$?"

• The tension between the two terms: the policy loss term encourages the model to "boldly explore" for higher rewards, while the KL penalty constrains the model to "not go too far" to maintain language quality. $\beta$ is the balance point between these two forces

SFT can only bring the model to the quality ceiling of the training data. The RL phase uses the reward function to tell the model "what constitutes a good result," letting the model autonomously explore solution paths that exceed the demonstration data — this is precisely the key mechanism by which DeepSeek-R1 [3] can develop "self-reflection" and "long-chain reasoning" capabilities.

What SFT learned:  "See a math problem → call the calculator"
What RL learned:   "Analyze problem structure → judge whether step-by-step is needed → 
                    select optimal tool combination → verify intermediate results → 
                    proactively backtrack when errors are found"

Why Are Both Phases Indispensable?

📌 Engineering Practice Notes

Data quality in the SFT phase is more critical than quantity. Research from LIMA [4] shows that 1,000 carefully selected high-quality data points often outperform 10,000 noisy data points. Practical recommendations:

• SFT data scale: 500–2,000 manually verified Agent interaction trajectories
• RL compute cost: approximately 3–10× the SFT phase (due to online sampling requirements)
• Validation strategy: first validate the training pipeline's correctness on a small 7B model, then scale to larger models
• When does SFT "graduate"? The goal of SFT is not to "do the best" but to "do well enough" — loss converged + format correctness rate ≥ 90% + output still has diversity. Over-SFT seriously damages RL effectiveness, see Section 11.2's SFT Graduation Criteria

Representative Works and Empirical Results

These works collectively validate the effectiveness of the Agentic-RL paradigm: through reinforcement learning training, models can develop reasoning strategies that never appeared in the training data — something pure SFT methods cannot achieve.

In the next section, we will start with SFT + LoRA and introduce in detail the principles and implementation of the first-phase supervised fine-tuning.

References

[1] SUTTON R S, BARTO A G. Reinforcement Learning: An Introduction[M]. 2nd ed. Cambridge: MIT Press, 2018.

[2] XI Z, CHEN W, GUO X, et al. The rise and potential of large language model based agents: A survey[R]. arXiv preprint arXiv:2309.07864, 2023.

[3] DEEPSEEK AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning[R]. arXiv preprint arXiv:2501.12948, 2025.

[4] ZHOU C, LIU P, XU P, et al. LIMA: Less is more for alignment[C]//Advances in Neural Information Processing Systems (NeurIPS). 2023.

[5] DEEPSEEK AI. DeepSWE: An open agentic SWE model that matches the performance of closed-source models[R]. 2025.

[6] OPENAI. Learning to reason with LLMs[EB/OL]. 2024. https://openai.com/index/learning-to-reason-with-llms.

[7] YANG A, YANG B, HUI B, et al. Qwen2.5 technical report[R]. arXiv preprint arXiv:2412.15115, 2024.