← 返回 2026-04-15

重新审视大语言模型的在策略蒸馏:现象、机制与配方 Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, Ning Ding 📅 2026-04-14 👍 114 2026-07-13 08:36
LLM post-training knowledge distillation on-policy distillation reinforcement learning reverse KL divergence thinking-pattern consistency
arXiv HuggingFace PDF

系统解构OPD成败:思维模式一致性加新知识,overlap token才是梯度真正作用区。

前置知识

知识蒸馏(Knowledge Distillation, KD)

通过让小模型(学生)模仿大模型(老师)的输出分布来传递知识。标准做法是最小化两者的KL散度 $D_{\mathrm{KL}}(\pi_T \| \pi_\theta)$,其中 $\pi_T$ 和 $\pi_\theta$ 分别是老师和学生在某状态下的下一词分布。Off-policy版本用老师生成的固定序列训练,存在曝光偏差(exposure bias);On-policy版本(OPD)则让学生自己采样再求token级监督,缓解了分布失配问题。Reverse KL与forward KL的关键区别在于前者mode-seeking,会让学生只覆盖teacher的某个高概率模式。

整篇文章的研究对象就是KD的一种新形式(OPD),需要理解为什么off-policy KD有局限、为什么要用学生自己的rollout、以及reverse KL的mode-seeking特性如何影响蒸馏效果。

On-Policy Distillation (OPD)

在策略蒸馏。给定prompt $x$,学生 $\pi_\theta$ 自回归采样得到 $\hat y = (\hat y_1, \ldots, \hat y_T)$,然后在每个位置 $t$ 让老师和学生在学生前缀 $\hat y_{<t}$ 上同时给出下一词分布 $p_t = \pi_\theta(\cdot|x,\hat y_{<t})$ 与 $q_t = \pi_T(\cdot|x,\hat y_{<t})$,最小化序列级reverse KL: $\mathcal{L}_{\mathrm{OPD}} = \mathbb{E}_{x,\hat y} \sum_{t=1}^{T} D_{\mathrm{KL}}(p_t \| q_t)$。文章区分了sampled-token、full-vocabulary、top-$k$ 三种监督粒度。

Qwen3、MiMo、GLM-5 等工业级模型都依赖OPD,理解其成败条件对设计post-training pipeline至关重要。

GRPO (Group Relative Policy Optimization)

一种基于组内归一化优势的强化学习算法,常用于LLM的RL post-training。它对同一prompt采样一组回答,用组内相对奖励代替绝对奖励,降低方差。文章用GRPO把Qwen3-4B-Base变成Qwen3-4B-Base-GRPO,作为与原Qwen3-4B做思维模式对比的teacher;还用JustRL-DeepSeek-1.5B、Qwen3-4B-Non-Thinking-RL-Math、Skywork-OR1-Math-7B等RL后模型演示"RL能注入新知识"这一核心论点。

理解teacher是否经过RL对思维模式的塑造,是本文核心实验设计的关键。文章反复用GRPO/RL后训练产生的teacher(如Qwen3-4B-Base-GRPO、Skywork-OR1-Math-7B)与未RL teacher做对比,以验证"RL给teacher注入新知识"这一论点。

Reverse KL 与 mode-seeking

KL散度不对称。$D_{\mathrm{KL}}(P \| Q)$(reverse KL)要求 $P=0$ 时 $Q$ 也必须为0,因此优化 $P$ 趋向于只覆盖 $Q$ 的某个高概率模式,不会把概率铺开到 $Q$ 认为不太可能的token上。文章多处利用这个特性:成功OPD中overlap ratio稳步从72%升至91%以上,正是因为reverse KL的mode-seeking会进一步把学生概率集中到overlap token上,从而把竞争token挤出top-$k$集合,形成自我强化循环。

文章把OPD成功的核心机制归因于reverse KL的mode-seeking在overlap token上形成自我强化的正反馈循环。

Top-$k$ 概率集合与overlap ratio

Top-$k$ 集合指每个位置上学生(或老师)概率最高的 $k$ 个token。Overlap ratio定义为两个top-$k$集合交集大小占 $k$ 的比例: $M_{\mathrm{overlap}} = \mathbb{E}_t \frac{|S_t^{(p)} \cap S_t^{(q)}|}{k}$。文章证明有效OPD中该指标会从72%稳步升至91%以上,而失败的run从一开始就停滞。

overlap ratio是贯穿全文最核心的动态监控指标,直接对应OPD的"思考模式是否对齐"。成功OPD中它从72%稳步升至91%以上,失败run从一开始就停滞,几乎可以作为训练是否有效的早期预警信号。

研究动机

On-Policy Distillation已成为LLM后训练的核心技术,Qwen3、MiMo、GLM-5等工业级模型都在post-training中采用OPD并报告了显著增益,Thinking Machines Lab也独立复现了Qwen3的OPD配方。但实践中存在一个突出且反直觉的失败模式:一个更强的teacher可能完全无法提升学生,而一个更弱的teacher反而能从较低的初始对齐开始取得明显进步。文章以Qwen3-4B-Base-GRPO与Qwen3-4B(Non-thinking)做对比时,后者尽管在benchmark上得分相近甚至更高,但因为初始overlap ratio更高(0.69 vs 0.66)且思维模式更接近学生,蒸馏效果反而更好,AIME 2024准确率0.242 vs 0.212的差距在200步训练内持续存在,说明早期思维模式错位造成了无法弥补的蒸馏损失。已有的知识蒸馏文献大多把关注点放在off-policy蒸馏上,对OPD何时失败、token级监督究竟在做什么,几乎没有系统分析。

本文的目标是本文目标分三层:第一,给出OPD成功与失败的判据,即什么样的teacher才能真正驱动学生,把"teacher越强越好"的隐含假设替换为可验证的两条条件;第二,从token层面解释OPD训练动力学,通过overlap ratio、overlap-token advantage、entropy gap三个指标追踪训练过程,揭示梯度信号到底作用在哪些token上;第三,基于上述分析,提出在配置失败时如何"救"OPD的可操作recipe(包括off-policy冷启动与teacher-aligned prompt),并讨论OPD长期可扩展性的边界条件。

与已有工作不同的是，不同于以往工作中"teacher越强越好"的默认假设,文章的核心切入角度是把OPD的成败拆解成两个互相独立、缺一不可的条件:(i) 思维模式一致性(thinking-pattern consistency),即学生和teacher在高概率token集合上的overlap ratio;(ii) 新知识(higher scores ≠ new knowledge),teacher必须拥有学生在训练中未见过的新能力,否则即便分数更高也不会提供新梯度信号。文章通过"反向蒸馏"实验(用JustRL-1.5B做学生,R1-Distill-1.5B做teacher,即把模型往它自己RL之前的checkpoint蒸馏)来同时验证这两个条件,这是以往工作中缺失的、受控的、跨家族的对比设置。

核心方法

整体思路分四步:现象学(Phenomenology)→ 机制(Mechanism)→ 配方(Recipe)→ 代价讨论(Discussion)。先在受控对比实验中找到决定OPD成败的两条规律,然后通过动态指标(overlap ratio、overlap-token advantage、entropy gap)追踪训练过程,把成败归因到具体的token集合,再据此提出在失败情形下可以"修补"的两个策略,最后讨论OPD的密监督在长轨迹下是否还可靠。文章采用统一的评估设置:用DAPO-Math-17K训练,在AIME 2024、AIME 2025、AMC 2023上以avg@16(temperature 0.7、top-$p$ 0.95、最大长度31744)为主指标。技术路线是"先经验现象 → 再数学化指标 → 再做消融与干预 → 再讨论边界"。

核心创新点在于:(1) 首次把OPD的成败归因到overlap token集合而非整个词表,证明overlap token承载了97%-99%的概率质量,且只优化这些token与全top-$k$效果相当;(2) 区分了"benchmark分数高"和"拥有新知识"——同一家族的7B模型相对1.5B只是同一分布的不同尺度拟合,从学生视角看其局部目标分布几乎相同,因此分数再高也不会带来新梯度信号;(3) 揭示了reverse KL的mode-seeking在overlap集合上形成自我强化循环:某token一旦进入共享高概率区,reverse KL会进一步把学生概率集中到它身上,从而把它推出竞争位置,使overlap区域不断扩大,这就是成功OPD中overlap ratio从72%稳步升至91%的根本机制。本质区别于已有工作把OPD当成"KL目标 = SFT + RL"的黑盒使用,本文给出了"为什么"的解释和"何时用、怎么用"的指南。

方法步骤详情

方法分四步。(1)形式化三种OPD变体:学生 $\pi_\theta$ 在prompt $x$ 上采样 $\hat y$,位置 $t$ 上算 $p_t$ 与 $q_t$。Sampled-token用单token算 $\ell_t^{\mathrm{sample}}=\log p_t(\hat y_t)-\log q_t(\hat y_t)$;Full-vocabulary对整词表算 $D_{\mathrm{KL}}(p_t \| q_t)$,内存 $O(BTM)$;Top-$k$在学生top-$k$集合归一化后算KL,是折中方案。(2)三组指标:Overlap ratio $M_{\mathrm{overlap}}$、Overlap-token advantage $M_{\mathrm{adv}}$、Entropy gap $\Delta H_t=|H(q_t)-H(p_t)|$。(3)受控对比+反向蒸馏验证两条件。(4)消融把Top-$k$拆为overlap与non-overlap分别优化,提出off-policy冷启动与teacher-aligned prompts两条干预策略。

技术新颖性

技术新颖性体现在三个层面。第一,实验设计层面:同时考虑teacher的分数、teacher与学生的思维模式、teacher是否拥有新知识三因素,并用"反向蒸馏"这种受控设置把三者解耦,这是首次对OPD的系统性受控研究。第二,机制分析层面:用overlap ratio、overlap-token advantage、entropy gap三个指标追踪训练过程,并证明97%-99%的概率质量集中在overlap token上,从而把OPD的优化目标精确地定位到一个小子集上,这是此前没有的工作。第三,几何层面的洞察:虽然失败teacher(R1-Distill-7B)的全局reward分布与正确性相关性与成功teacher接近(AUROC 0.75 vs 0.73),但其per-token优势在序列内是各向异性的,聚合后梯度会相互抵消,导致梯度范数反而更小,这揭示了"全局信息量大"不等于"局部可被利用"这一非平凡结论。

Successful vs. failing OPD with the same student (R1-Distill-1.5B) and two teachers. Top: avg@16 accuracy on three benchmarks. Dashed lines indicate teacher performance. Bottom: three dynamics over training.
Figure 6: Successful vs. failing OPD with the same student (R1-Distill-1.5B) and two teachers. Top: avg@16 accuracy on three benchmarks. Dashed lines indicate teacher performance. Bottom: three dynamics over training.
Ablation on the optimization support in Top-k OPD. Overlap Top-k matches Student Top-k, while Non-Overlap Top-k is substantially weaker.
Figure 7: Ablation on the optimization support in Top-k OPD. Overlap Top-k matches Student Top-k, while Non-Overlap Top-k is substantially weaker.
Effect of off-policy cold start before OPD, using fixed teacher Qwen3-4B (Non-thinking). The two curves correspond to OPD from Qwen3-1.7B-SFT and Qwen3-1.7B-Base.
Figure 8: Effect of off-policy cold start before OPD, using fixed teacher Qwen3-4B (Non-thinking). The two curves correspond to OPD from Qwen3-1.7B-SFT and Qwen3-1.7B-Base.
Effect of prompt template alignment. The teacher-aligned template yields higher accuracy and overlap growth throughout training.
Figure 9: Effect of prompt template alignment. The teacher-aligned template yields higher accuracy and overlap growth throughout training.
Effect of prompt content alignment. The teacher-aligned prompt contents yield stronger performance, with higher mass concentration on shared tokens and notably lower entropy.
Figure 10: Effect of prompt content alignment. The teacher-aligned prompt contents yield stronger performance, with higher mass concentration on shared tokens and notably lower entropy.

实验结果

五条核心发现:(1)思维模式一致性关键——Qwen3家族GRPO teacher初始overlap 0.69 vs Non-thinking 0.66,蒸馏AIME 2024达0.242 vs 0.212,差距200步持续存在;(2)高分≠新知识——反向蒸馏把学生拉回RL前;Skywork(RL后)gap recovery 16.9%远高于R1-Distill-7B的5.3%,Qwen家族Qwen3-4B-RL-Math 58.6% vs 15.6%;(3)成功OPD核心机制是overlap token渐进对齐——overlap ratio从72%稳步升至91%以上,$M_{\mathrm{adv}}$趋向0,$\Delta H_t$单调收窄;(4)只优化overlap token即与全top-$k$性能相当——Overlap Top-$k$在AIME 2024约0.40,Non-Overlap仅约0.30;(5)OPD密监督存在长度天花板——3K-7K最优,10K-15K后期overlap ratio陡降,teacher续写增益从1K前缀+0.37单调下降到16K前缀+0.02。

Effect of the support size k in Top-k OPD. All numbers are reported as avg@16.
Figure 15: Effect of the support size k in Top-k OPD. All numbers are reported as avg@16.
Training dynamics under different support sizes k for Top-k OPD.
Figure 16: Training dynamics under different support sizes k for Top-k OPD.
查看结构化数据
任务指标本文基线提升
AIME 2024 (avg@16) 准确率 0.454 (sampled-token) / 0.446 (Top-1) / 0.473 (Top-4) / 0.458 (Top-16) / 0.463 (Top-64) Qwen3-4B Non-thinking 蒸馏 Qwen3-1.7B-Base 仅OPD基线约 0.10 (200步后) Top-$k$ 不同 $k$ 之间差异小于0.03,sampled-token 与 Top-4 相当,Top-1 明显最差;off-policy冷启动将最终准确率从基线的约0.10提升到约0.125 (AIME 2024)
AIME 2025 (avg@16) 准确率 0.327 (sampled-token) / 0.310 (Top-1) / 0.331 (Top-4) / 0.338 (Top-16) / 0.338 (Top-64) 纯OPD基线约 0.06 (200步) $k \geq 4$ 后无明显提升,sampled-token 表现不弱于Top-$k$
AMC 2023 (avg@16) 准确率 0.782 (sampled-token) / 0.772 (Top-1) / 0.793 (Top-4) / 0.791 (Top-16) / 0.785 (Top-64) 纯OPD基线约 0.30 $k=4$ 略高于其他;Top-1 仍为最差
DeepSeek家族 gap recovery rate (Acc_after - Acc_before)/(Acc_teacher - Acc_before) Skywork-OR1-Math-7B(RL后)= 16.9%;R1-Distill-7B(同管线)= 5.3% 同家族未RL teacher RL后训练teacher使恢复率绝对提升11.6个百分点,证明新知识带来可迁移信号
Qwen家族 gap recovery rate (Acc_after - Acc_before)/(Acc_teacher - Acc_before) Qwen3-4B-Non-Thinking-RL-Math(RL后)= 58.6%;Qwen3-4B Non-thinking= 15.6% 同管线未RL teacher 绝对提升43个百分点,与DeepSeek家族趋势一致

局限与改进

作者明确承认的局限:全部实验局限在数学推理(AIME 2024/2025、AMC 2023),能否推广到代码、开放域、agentic任务尚未验证。"新知识"判断依赖teacher是否经额外RL,但RL与预训练数据差异的贡献难以解耦,跨家族对比又被tokenizer、架构差异污染。长轨迹下reward质量系统性退化,学生前缀超16K后teacher续写增益几乎消失(+0.02),当前OPD难扩展到超长CoT或agentic设置;失败teacher的全局AUROC(0.75)与成功teacher(0.73)相当,作者只给"各向异性per-token优势相消"的假设,未直接验证梯度方向结构。我的观察:"score vs new knowledge"主要靠反向蒸馏这一个别实验,跨家族是否复现需更多证据;文章没定量分析"多大overlap才能让OPD成功",这个阈值是工程师最关心的。

独立分析的弱点

三个独立弱点。第一,所有teacher-student对同家族/同管线,R1-Distill-1.5B与R1-Distill-7B共享太多预训练数据,反向蒸馏证明两者从学生视角几乎不可区分;若换tokenizer或预训练差异更大的对,思维模式一致性重要性是否仍成立未知——改进方向是补充跨家族(Qwen与LLaMA)受控对比。第二,off-policy冷启动效果虽明显,但没回答"多少SFT数据足够""SFT与OPD学习率如何衔接"等实操问题;off-policy冷启动相当于teacher assistant,Mirzadeh 2020已有类似思路但本文未引用对比——改进方向是给出SFT数据量与学习率扫描,并和teacher assistant方案head-to-head。第三,长horizon退化部分仅考察teacher续写增益衰减和训练曲线,未从理论上分析reverse KL为何在长轨迹出现反向传播式不稳定——改进方向是分段KL或curriculum,短片段用OPD、长片段切换到GRPO/PPO,并监控teacher confidence随深度的变化。

未来方向

作者第8节给出四个方向:(1)拓展到数学之外——验证思维模式一致性和overlap token机制在代码、开放问答、多轮对话场景是否同样成立;(2)隔离预训练数据影响——设计共享tokenizer且架构相同、仅预训练语料不同的teacher-student对,目前跨家族对比被tokenizer/架构差异污染,真正受控的预训练消实验代价过高;(3)self-distillation动力学——此时思维模式天然一致,关键变量变成"特权信息"(真值答案、执行反馈等);(4)长horizon与agentic场景——结合token级密监督和outcome级稀疏奖励的混合目标,以及curriculum式训练horizon扩展。基于文章成果,两个延伸方向:把overlap ratio做成训练实时监控指标,下降超阈值时自动触发冷启动或prompt切换;研究overlap token的几何结构(如per-token log-ratio分布),把"全局可分但局部难用"转化为可验证的优化器改进(如per-token advantage clipping、按entropy gap加权的梯度)。

复现评估

复现评估整体较好。代码仓库 https://github.com/thunlp/OPD 已开源。所有模型来自公开仓库:Qwen3系列、DeepSeek-R1-Distill-Qwen-1.5B/7B、Skywork-OR1-Math-7B、JustRL-DeepSeek-1.5B。训练数据DAPO-Math-17K、DeepMath的57K子集、OpenThoughts3-1.2M的math子集均为公开;评估用AIME 2024/2025、AMC 2023,作者明确采样设置(temperature 0.7、top-$p$ 0.95、每题16次、max length 31744)和avg@16指标。算力方面,1.5B-7B级LLM可在单节点8卡A100/H100完成全部200-600步训练(冷启动SFT 200K样本需多卡),复现难度中等偏低;主要门槛是teacher GPU推理开销,top-$k$ OPD需teacher返回前$k$个logits,full-vocabulary OPD还需完整词表logits,内存更高($O(BTM)$)。代码开源后中等ML团队能在数天内复现主实验。

论文图表

Overview of our paper. JustRL-1.5B is obtained by applying RL to DeepSeek-Distill-1.5B (DS-1.5B), and Skywork-OR1-Math-7B (SW-7B) by applying RL to DeepSeek-Distill-7B (DS-7B).
Figure 1: Overview of our paper. JustRL-1.5B is obtained by applying RL to DeepSeek-Distill-1.5B (DS-1.5B), and Skywork-OR1-Math-7B (SW-7B) by applying RL to DeepSeek-Distill-7B (DS-7B).

全文概览图,分三个子图总结核心发现:左图展示Student-Teacher-Gap Recovery,DeepSeek家族中经过RL的Skywork-OR1-Math-7B做teacher时gap恢复率为16.9%,而未RL的R1-Distill-7B只有5.3%;Qwen家族中Qwen3-4B-RL-Math恢复率58.6%,远高于未RL的15.6%。中图展示反向蒸馏(Weak→Strong),用JustRL-1.5B做学生、R1-Distill-1.5B或R1-Distill-7B做teacher都会让其回到RL前水平,Acc: DS-7B > JustRL-1.5B >> DS-1.5B。右图展示Overlap Token Ratio对训练的贡献:在JustRL-1.5B→DS-1.5B成功run中,overlap token贡献了主要梯度和优势。

这张图是整篇论文的roadmap,把现象(两条件)、机制(overlap token)和验证(反向蒸馏)三个核心叙事在视觉上一次性讲清,理解这张图就理解了论文70%的内容。

OPD from two teachers with different thinking patterns into the same student (Qwen3-1.7B-Base). The GRPO teacher achieves stronger performance (left) and higher initial overlap ratio (right), suggesting that thinking pattern compatibility governs OPD effectiveness.
Figure 2: OPD from two teachers with different thinking patterns into the same student (Qwen3-1.7B-Base). The GRPO teacher achieves stronger performance (left) and higher initial overlap ratio (right), suggesting that thinking pattern compatibility governs OPD effectiveness.

固定学生Qwen3-1.7B-Base,左图用Qwen3-4B-Base-GRPO做teacher,右图用Qwen3-4B(Non-thinking)做teacher,纵轴分别是平均验证准确率(avg@16)和overlap ratio,横轴是训练步数0-200。两个teacher在benchmark上得分相近(0.700 vs 0.599),但GRPO teacher的初始overlap ratio明显更高(约0.69 vs 约0.66),并且蒸馏后在AIME 2024上达到0.242,显著高于Non-thinking teacher的0.212。

这是thinking-pattern consistency条件的核心证据,说明即使teacher分数接近,初始overlap ratio决定OPD最终天花板。

Comparison of OPD performance with and without additional teacher RL post-training across two model families. Left: DeepSeek family. Right: Qwen family. Post-trained teachers yield substantially stronger gains and higher teacher-student gap recovery rate.
Figure 4: Comparison of OPD performance with and without additional teacher RL post-training across two model families. Left: DeepSeek family. Right: Qwen family. Post-trained teachers yield substantially stronger gains and higher teacher-student gap recovery rate.

分两个家族对比。左图DeepSeek家族:R1-Distill-1.5B学生配两个teacher,Skywork-OR1-Math-7B(RL后)在avg@16上明显高于R1-Distill-7B(同管线未RL),gap recovery rate为16.9% vs 5.3%,初始overlap ratio分别为71.5%和74.7%。右图Qwen家族:Qwen3-1.7B Non-thinking学生配Qwen3-4B-RL-Math vs Qwen3-4B Non-thinking,前者recovery rate 58.6%,后者仅15.6%,初始overlap 70.3% vs 75.7%。两组实验都呈现"未RL teacher即使初始overlap更高,也无法传递新知识"的规律。

这是"新知识不等于高分"条件的最强证据,说明必须用teacher的RL后训练给学生注入新能力,否则即便分数更高也无济于事。

Reverse distillation with JustRL-1.5B as the student and two same-family teachers (R1-Distill-1.5B and R1-Distill-7B). Both runs cause the student to regress to approximately the same level despite R1-Distill-7B scoring higher than JustRL-1.5B, indicating that OPD training dynamics are governed by thinking pattern rather than benchmark performance.
Figure 5: Reverse distillation with JustRL-1.5B as the student and two same-family teachers (R1-Distill-1.5B and R1-Distill-7B). Both runs cause the student to regress to approximately the same level despite R1-Distill-7B scoring higher than JustRL-1.5B, indicating that OPD training dynamics are governed by thinking pattern rather than benchmark performance.

学生JustRL-1.5B(已RL),teacher分别为其RL前的R1-Distill-1.5B(更弱)与同家族的R1-Distill-7B(更强,benchmark更高)。两条曲线在AIME 2024/2025/AMC 2023上都几乎重合,都把学生拉回到RL前水平(约0.20/0.27/0.62),而teacher各自的准确率分别为JustRL-1.5B约0.70/0.66/0.84、R1-Distill-7B约0.78/0.66/0.83。

反向蒸馏同时验证"思维模式决定OPD"和"分数不等于新知识"两个条件,是最具说服力的单一实验。

(a) Validation accuracy on three benchmarks under different response lengths. (b) Accuracy gain from teacher continuation under different student prefix truncation lengths.
Figure 11: (a) Validation accuracy on three benchmarks under different response lengths. (b) Accuracy gain from teacher continuation under different student prefix truncation lengths.

图(a):横轴为响应长度档位0.5K/1K/3K/7K/10K/15K,纵轴为avg@16,AIME 2024/2025/AMC 2023均呈现倒U型,3K-7K最优(分别约0.45/0.30/0.75),15K下降到约0.30/0.18/0.60。图(b):横轴为截断前缀长度1K/4K/8K/16K,纵轴为teacher续写相对学生的增益,从1K前缀的+0.37单调下降到16K前缀的+0.02。

这是讨论章节的核心证据,证明OPD密监督存在"长度甜区",长horizon下teacher信号不可靠,直接影响OPD能否扩展到agentic场景。

Training dynamics under different maximum response lengths for OPD.
Figure 12: Training dynamics under different maximum response lengths for OPD.

横轴训练步数0-260,六条曲线对应0.5K/1K/3K/7K/10K/15K最大响应长度。Overlap ratio:3K-7K稳定升至0.90以上,15K在step 200后从0.88骤降至0.78。Student entropy:3K-7K始终在2.0左右,10K-15K在step 200后出现尖峰,15K甚至冲到3.0。Grad norm:短长度梯度平稳,15K在训练后期出现明显尖峰,说明长轨迹下优化出现不稳定。

补充图11(a),把长度扫描的动态曲线展开,展示不稳定性是何时何地开始的,呼应图13的entropy heatmap。

Average student entropy across decoding positions during OPD training with 15K max response length, measured on student-generated trajectories from Step 180 to Step 250.
Figure 13: Average student entropy across decoding positions during OPD training with 15K max response length, measured on student-generated trajectories from Step 180 to Step 250.

热力图:纵轴是训练步数180-250,横轴是输出位置0-15K,颜色越深代表entropy越高。可以清楚看到高entropy区域从Step 180的右端(大于12K位置)开始出现,逐步向左(前部位置)蔓延,到Step 250时整个序列后半段都已进入高entropy区,呈现"从后往前扩散"的传播模式。

证明长轨迹下训练不稳定的源头在末端token,反向传播到早期位置,这是解释"为什么长CoT失败"的关键可视化。

Sequence mean reward distributions for correct and incorrect student rollouts. Both teachers assign higher reward to correct rollouts with comparable AUROC (0.73 and 0.75).
Figure 14: Sequence mean reward distributions for correct and incorrect student rollouts. Both teachers assign higher reward to correct rollouts with comparable AUROC (0.73 and 0.75).

两幅直方图。左:teacher为JustRL-1.5B(成功),正确rollouts(N=2828)sequence mean reward分布偏向右侧(-0.05附近),错误rollouts(N=1451)偏向左侧(-0.15附近),AUROC=0.7333。右:teacher为R1-Distill-7B(失败),两组分布形状几乎相同,AUROC=0.7511,说明全局上失败的teacher仍然能区分对错。

这是讨论章节最反直觉的发现:失败teacher的全局信号并不比成功teacher差,问题不在信号质量而在局部优化几何,这一证据支撑"全局可分但局部难用"的核心结论。