← 返回 2026-10-08

长上下文混合模型力学 1.1：从混合注意力到混合位置编码 Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

Xiaoran Liu, Ziwei He, Xipeng Qiu 📅 2026-10-07 👍 17 2026-11-11 18:30
位置编码 混合架构 滑动窗口注意力 线性注意力 长上下文 长度外推
arXiv HuggingFace PDF

从位置偏置视角解释混合架构为何擅长长上下文，并提出滑动窗口线性注意力实现16倍免训练外推。

前置知识

RoPE（旋转位置编码）

把 query/key 向量按 token 位置旋转特定角度，使注意力分数只依赖两个 token 的相对位置。rotary base（本文默认 10000）决定旋转的频率谱：base 越大，位置信息能覆盖越长的距离。长上下文续训时通常把 base 从 10000 放大到 500000，并配合 NTK 式插值来支持更长序列。

RoPE 是本文所有对比的基线位置编码；SWA 扩窗时的 base 缩放律、RoPE-NoPE 混合、以及'位置偏置注意力'这一定义都围绕 RoPE 展开。

NoPE（无位置编码注意力）

完全不施加显式位置编码的 full attention 层。依靠因果掩码（每个 token 只能看见过去）与 softmax 的不对称性，模型仍能隐式获得位置信号。近年工作（Jamba、SWAN-GPT、RNoPE）发现混合模型中的 full attention 层可以去掉 RoPE，并获得更好的免训练长度外推。

论文的核心命题是：混合模型的收益来自'NoPE + 其他位置偏置'的混合位置组合，NoPE 头承担粗粒度全局聚合与高命中率定位。

SWA（滑动窗口注意力）

限制每个 token 只能注意最近 $w$ 个 token 的注意力（本文 $w=128$，跟随 GPT-OSS/Mimo-v2.5）。计算量与显存不随序列长度增长，是固定内存复杂度的高效注意力；代价是天然丢失窗口外的长程依赖，只能靠与其混合的其他模块补偿。

SWA 混合是全文两大研究对象之一：它在长度外推中最强，但在 32k 续训后反而被反超（跷跷板效应），窗口大小的两难（短窗口疲倦/长窗口懒惰）是关键发现。

线性注意力与门控变体 GLA/GDN

把 softmax attention 改写为带固定大小递归状态的 RNN 形式：训练可并行、推理显存 $O(1)$。GLA（Gated Linear Attention）用数据依赖的门控标量 $\alpha_i$ 逐位衰减历史状态；GDN（Gated DeltaNet）在其上加入 delta rule 增量修正状态。本文把这种门控视为一种隐式的、数据依赖的位置偏置。

GLA/GDN 混合是另一研究对象：训练长度内拟合强但外推弱（无免费午餐效应），论文为其提出滑动窗口线性注意力（SWLA）外推方案。

长度外推 vs 上下文扩展

长度外推（length extrapolation）指模型只在短上下文（本文 4k×50B tokens）预训练后，不经再训练直接推理更长序列（8k–64k）；上下文扩展（context extension）指用长文档语料（LongWanjuan，32k×5B tokens）继续预训练来扩展可用上下文。两者的赢家可能不同——这是跷跷板效应的舞台。

全文所有实验都按这两个场景分别报告，混淆两者就无法理解 SWA 与 LA 混合'各有千秋'的结论。

注意力熵与 top-k 命中率

注意力熵衡量某个头的注意力分布均匀程度，论文按 $\log$(上下文长度) 归一化到 [0,1]；top-k 命中率指输入末尾 query 的 top-$k$ 注意力分打中待检索位置的概率（在 RULER SK1 的 4k 提示上测，$k$ 从 1 扫到 4k）。两者构成'熵-命中率散点图'，每个点是一个注意力头，用于功能画像。

这套诊断工具是潮汐效应的证据来源：NoPE 头聚集在高命中率高熵带状区，RoPE/GLA/GDN 头聚集在低熵扇形区，边界随混合比移动。

RULER/NIAH 与 BABILong

RULER 是长上下文检索基准，其中 NIAH-SK1/SK2/SK3 是单针、双针、三针'大海捞针'任务，是混合模型论文最常报告的指标；BABILong 把 bAbI 问答故事嵌入超长干扰文本，考查状态追踪与多跳推理。两者在 0k–64k 多个长度上评测，报告 4k 内（in-domain）与 4k 外（OOD）平均分。

跷跷板效应、无免费午餐效应、16 倍外推等核心结论全部以 RULER+BABILong 的分数变化来量化，例如 SK1@64k 从 0.0 提升到 100.0。

NTK 插值与 log 缩放外推

NTK-aware 插值通过放大 rotary base、压缩位置间隔，使训练在 4k 的 RoPE 模型能推理更长序列；Dynamic NTK 按推理长度动态调整。log 缩放（log-scaled attention）则用于 NoPE 注意力：随上下文增长按对数放大 attention logit 的尺度，防止注意力熵随长度无界增大。两者可叠加使用。

本文的 EME 外推策略 = 位置偏置注意力加滑动窗口 + NoPE 加 log 缩放，并证明其优于传统 Dynamic NTK 插值。

研究动机

主流开源 LLM 的架构正从纯 full-attention 转向混合架构：Qwen-3.5、Kimi-K3、OLMo-Hybrid 在 full softmax attention 中混入 GDN/KDA 线性注意力，Gemma4、Mimo-v2 混入滑动窗口注意力（SWA），Jamba、GPT-OSS、MiniMax-01 等也已落地。动机很直接：agentic 时代的百万 token 上下文让纯 full attention 代价高得不可接受，而任何单一高效注意力又不可靠。但已有工作大多停留在'哪个模型更好'的现象层比较（下游评测、scaling 曲线、系统效率），缺乏机理解释：不同注意力模块在同一模型内到底如何分工协作？为什么 SWA 混合在长度外推上占优、LA 混合在长上下文续训后反而更强？论文在 4k 短上下文预训练（50B tokens）后直接外推到 64k、以及 32k 长上下文续训（5B tokens）后发现，两类混合模型的表现排序会反转，这一跷跷板现象此前从未被系统观察和解释，而它直接关系到'给长上下文模型选哪种混合架构、续训时该怎么调'的实际决策。

本文的目标是本文是《Mechanics of Long-Context Hybrid Models》系列的 Part 1.1，目标是建立一套'训练—评测—机理观察—闭环验证'的分析方法，回答两个问题：混合模型为什么在长上下文有效？如何设计得更有效？具体做法是在 376M/776M/1B 三个规模（另加 3B 验证）上，系统比较 RoPE full attention、RoPE-NoPE 混合位置 full attention、SWA-NoPE 混合与 GLA/GDN-NoPE 混合，覆盖层混合（LH）与头混合（HH）两种范式（默认 3:1、SWA 窗口 128、MHA），在语言建模（PG19 PPL/LongPPL）、检索（RULER）与状态追踪（BABILong）三类能力上，同时考察两个场景：4k 预训练后的免训练长度外推，与 32k×5B tokens 长上下文续训后的上下文扩展。最终把观察到的规律凝练成 6 条 Takeaway 与两个可复用配方：EME 外推策略（滑动窗口+log 缩放）和 SWA 续训修复配方（扩窗+LongCE）。

与已有工作不同的是，独特切入点是'位置偏置'视角：作者把 RoPE attention、SWA、门控线性注意力（GLA/GDN 的门控可视为数据依赖的位置偏置）统一看作'带位置偏置的注意力'，把 NoPE full attention 看作'无位置偏置的粗粒度全局聚合器'，指出混合模型的收益本质上来自'混合位置'（hybrid position）而非笼统的'混合注意力'。此前工作要么从理论上估计 3:1 的混合比例（Wang et al.），要么指出检索主要由 full attention 承担（Wu et al.），但没有人用 top-k 命中率、检索头/流式头比例、归一化注意力熵等微观指标刻画两类注意力在同一个模型内的功能分区，更没有人把这个分区反过来指导外推算法设计。本文提出的滑动窗口线性注意力（SWLA）正是'位置偏置注意力外推时应限制相对位置范围'这一原理的自然推论，把 iRoPE/SWAN-GPT 式的 RoPE-NoPE 混合外推策略第一次推广到了线性注意力混合，打破了 SWA 与 LA 的传统边界。

核心方法

整体思路是'受控对比→机理归因→配方设计→闭环验证'。实验矩阵为：注意力组合（RoPE、RoPE-NoPE、SWA/GLA/GDN 与 NoPE 或 RoPE 混合）×两种范式（层混合 LH/头混合 HH）×四个规模（376M/776M/1B/3B）。两阶段训练：先在 DCLM-Baseline-1.0 上以 4k 上下文预训练 50B tokens（Llama 3 tokenizer、BF16、DeepSpeed+FlashAttention2+FlashLinearAttention、batch 1M tokens），再用 LongWanjuan 以 32k 上下文续训 5B tokens（RoPE base 升至 500000）。评测用 PG19 PPL/LongPPL、RULER、BABILong，附录补充 LongBench 与通用短上下文任务。机理工具箱包括：混合比消融（3:1/1:1/1:3）、高斯噪声扰动、top-k 命中率、检索头比例、按 $\log$(上下文长度) 归一化的注意力熵、熵-命中率散点图、$W_Q/W_K$ 参数变化率，以及把注意力分布推广到线性注意力的'扩展注意力分布'。

核心创新是把混合模型的收益归结为'混合位置'并提炼出三条机理定律。(1) 潮汐效应（Takeaway 4）：在熵-命中率图上，NoPE 头占据从中上部延伸到右下的带状区（高命中率高熵、粗粒度聚合），位置偏置注意力（RoPE/GLA/GDN）占据左下角向外的扇形区（低熵、负责降噪），边界随混合比移动；有效协作依赖'少数高命中率 NoPE 头 + 多数低熵位置偏置头'。这统一解释了为什么 3:1 优于 1:3、为什么给位置偏置头加高斯噪声比给 NoPE 头加噪声伤害更大。(2) 跷跷板效应（Takeaway 2/5）：SWA-NoPE 外推最强，但 32k 续训后被 LA-NoPE 反超，根源是短上下文学习陷阱——$\|W'_Q-W_Q\|_2/\|W_Q\|_2$ 显示 SWA 分支在续训中获得异常大的更新，而固定 128 的窗口装不下长程依赖，导致长上下文位置 PPL 不降反升；扩窗至 2048 并叠加 LongCE 即可修复。(3) 马太效应（Takeaway 6）：位置偏置注意力无论 RoPE 还是门控 LA，外推时都应限制相对位置范围（前者用 SWA、后者用 SWLA），NoPE 则用 log 缩放增强全局感知。

方法步骤详情

方法分四步。第一步，受控对比：固定 3:1 混合比、SWA 窗口 128、MHA（GDN 不支持 GQA），完成 4k×50B 预训练与 32k×5B 续训，绘制损失/PPL 与 RULER/BABILong 曲线（Figures 1–9），确立 Takeaway 1–3；并验证 RoPE-NoPE 混合外推（RoPE 截 4k 窗口 + NoPE log 缩放）优于 Dynamic NTK。第二步，机理归因：在头混合上做比例消融与噪声实验；用 4k SK1 提示测 top-$k$ 命中率、检索头比例与归一化注意力熵，画出熵-命中率散点图（Figures 12–13、21–22）；对无 softmax 分布的线性注意力，把注意力权重重解释为 $\partial o_t/\partial v_s$ 雅可比矩阵的谱范数，得到'扩展注意力熵'。第三步，SWA 修复：续训时把窗口从 128 扩到 2048（同步调大 rotary base），叠加 LongCE 长上下文损失，并用后 SWA-NoPE-LH 追平 GDN-NoPE-LH。第四步，SWLA 外推：不改 FLA kernel，用窗口 4096、步长 1024 分块并行——对 $[0,4095]$、$[1024,5119]$、$[2048,6143]\dots$ 各块并行调用原始 GLA/GDN 算子再拼接有效段；配合 NoPE 头 log 缩放（合称 EME）实现 4k→64k 的 16 倍免训练外推。

技术新颖性

技术新颖性有四点。第一，概念重构：首次把 SWA/RoPE/门控线性注意力与 NoPE 的各种组合统一命名为'混合位置'，把 Jamba、GPT-OSS、Qwen、Kimi 等大量经验性混合设计纳入同一解释框架，并证明即使在纯 full attention 模型中，RoPE-NoPE 混合+混合外推也优于 Dynamic NTK 插值。第二，测量学创新：为没有 softmax 分布的线性注意力定义'扩展注意力分布/熵'（雅可比谱范数归一化），使 RoPE/SWA/GLA/GDN/NoPE 五种模块能在同一张熵-命中率图上比较，这是可复用的分析工具，且不修改任何注意力计算本身。第三，新现象的系统命名与验证：短上下文学习陷阱、短窗口疲倦/长窗口懒惰、马太效应均为首次命名，其中跷跷板效应在 1e-4–1e-3 四个学习率与 YOCO 式层布局下均稳定复现。第四，方法极简且通用：SWLA 不改任何 kernel，仅靠分块并行即可实现，与具体 LA 变体解耦，理论上兼容任意带门的线性注意力，并把'长程衰减'（如 Lightning Attention）这一特性放大为显式的滑动窗口约束。

Comparison of training loss, RULER, and BABILong among different hybrid ratios in head-wise hybrid models.
Figure 10: Comparison of training loss, RULER, and BABILong among different hybrid ratios in head-wise hybrid models.
Results of noise experiments on RULER and BABILong benchmark. Adding Gaussian noise to RoPE full attention or SWA/GLA/GDN degrades long-context performance more severely than the same perturbation applied to NoPE full attention in the corresponding hybrid models.
Figure 11: Results of noise experiments on RULER and BABILong benchmark. Adding Gaussian noise to RoPE full attention or SWA/GLA/GDN degrades long-context performance more severely than the same perturbation applied to NoPE full attention in the corresponding hybrid models.
The results of attention analysis within 1:1 RoPE-NoPE-HH and among RoPE-NoPE-HH with different hybrid ratio. NoPE full attention shows a stronger tendency for coarse global information localization, given the higher attention score hit rate and larger attention entropy.
Figure 12: The results of attention analysis within 1:1 RoPE-NoPE-HH and among RoPE-NoPE-HH with different hybrid ratio. NoPE full attention shows a stronger tendency for coarse global information localization, given the higher attention score hit rate and larger attention entropy.
The results of attention analysis within 1:1 RoPE-NoPE-LH and among RoPE-NoPE-LH with different hybrid ratios. The tidal effect of hybrid position is still applicable in RoPE-NoPE layer-wise hybrid models.
Figure 13: The results of attention analysis within 1:1 RoPE-NoPE-LH and among RoPE-NoPE-LH with different hybrid ratios. The tidal effect of hybrid position is still applicable in RoPE-NoPE layer-wise hybrid models.
Perplexity curves of different hybrid models under direct extrapolation, log-scale NoPE extrapolation (Log), and long-context continual pretraining (LC-CPT). The perplexity curves of SWA-NoPE-LH show a remarkable increasing trend as the context length grows, different from other hybrid models.
Figure 14: Perplexity curves of different hybrid models under direct extrapolation, log-scale NoPE extrapolation (Log), and long-context continual pretraining (LC-CPT). The perplexity curves of SWA-NoPE-LH show a remarkable increasing trend as the context length grows, different from other hybrid models.
Comparison of the change rates of the $W_Q$ and $W_K$ matrices in attention modules across different hybrid models before and after long-context training.
Figure 15: Comparison of the change rates of the $W_Q$ and $W_K$ matrices in attention modules across different hybrid models before and after long-context training.
Comparison of long-context perplexity and performance of SWA-NoPE-LH with different window sizes (wsz) in long-context continual pretraining (CPT).
Figure 16: Comparison of long-context perplexity and performance of SWA-NoPE-LH with different window sizes (wsz) in long-context continual pretraining (CPT).
Comparison of long-context perplexity and performance of SWA-NoPE-LH with different window sizes (wsz) in short-context pretraining.
Figure 17: Comparison of long-context perplexity and performance of SWA-NoPE-LH with different window sizes (wsz) in short-context pretraining.
Tendency of long-context perplexity and performance as the window size of SWA-NoPE-LH increases in long-context pretraining. Larger window sizes show better context extension, implying short-window weariness.
Figure 18: Tendency of long-context perplexity and performance as the window size of SWA-NoPE-LH increases in long-context pretraining. Larger window sizes show better context extension, implying short-window weariness.
Tendency of long-context perplexity and performance as the window size of SWA-NoPE-LH increases in short-context pretraining. Shorter window sizes show better length extrapolation, implying long-window laziness.
Figure 19: Tendency of long-context perplexity and performance as the window size of SWA-NoPE-LH increases in short-context pretraining. Shorter window sizes show better length extrapolation, implying long-window laziness.
Comparison of long-context performance of SWA-NoPE-LH after long-context continual pretraining enhanced by LongCE and extended window size. Both clearly weaken the seesaw effect and also contribute to SWA-RoPE-LH.
Figure 20: Comparison of long-context performance of SWA-NoPE-LH after long-context continual pretraining enhanced by LongCE and extended window size. Both clearly weaken the seesaw effect and also contribute to SWA-RoPE-LH.
Attention analysis results among 3:1 RoPE-NoPE, SWA-NoPE, GLA-NoPE, and GDN-NoPE hybrid models, with extended attention entropy and hit rate.
Figure 21: Attention analysis results among 3:1 RoPE-NoPE, SWA-NoPE, GLA-NoPE, and GDN-NoPE hybrid models, with extended attention entropy and hit rate.
Attention analysis results among GLA-NoPE-HH and GDN-NoPE-HH with different hybrid ratios. The tidal effect of hybrid position is still applicable in GLA-NoPE and GDN-NoPE head-wise hybrid models.
Figure 22: Attention analysis results among GLA-NoPE-HH and GDN-NoPE-HH with different hybrid ratios. The tidal effect of hybrid position is still applicable in GLA-NoPE and GDN-NoPE head-wise hybrid models.

实验结果

(1) 短上下文预训练后（Figures 1–4）：混合模型的 loss/PPL 均低于纯 RoPE，SWA/GLA/GDN-NoPE 在 4k 内外的 RULER+BABILong 平均分全面领先；混合外推显著优于 Dynamic NTK（Figure 5：1B 上 SWA-4k-NoPE-LH+Log 全平均 43.1 vs RoPE+NTK+Log 24.3）。(2) 32k 续训后（Figures 6–9）：外推最强的 SWA-NoPE-LH 被 LA-NoPE 反超（跷跷板），其 PPL 曲线在长上下文位置不降反升（Figure 14），$W_Q/W_K$ 变化率显示 SWA 分支被异常大量更新（Figure 15）；该效应对学习率与层布局稳健。(3) 机理：3:1 比例大多最优，NoPE 越多外推越差；扰动位置偏置头比扰动 NoPE 头伤害更大；top-64 命中率上 NoPE 与 RoPE 差距最大；GLA/GDN 熵随长度下降，NoPE 上升，SWA 几乎不变。(4) 窗口实验（Tables 13–15）：续训时 2048 窗口最优，短预训练时 128 外推最好；wsz2k+LongCE 把 776M SWA-NoPE-LH 全平均从 29.4 提到 37.4、3B 从 45.9 到 53.7。(5) SWLA/EME（Table 1）：776M GLA-NoPE-LH 的 SK1@64k 从 0.0 提到 100.0、SK2 0→59、SK3 0→64；GDN-NoPE-LH SK2@64k 达 90；1B/3B 的 RULER 全平均普遍提升 8–11 分。(6) 效率（Figures 26–27）：混合模型 TTFT/TPOT/显存/吞吐全面优于 full attention。

Extrapolation comparison of the 776M GLA/GDN-NoPE hybrid model on the NIAH-SK1, SK2, and SK3 tasks.
Table 1: Extrapolation comparison of the 776M GLA/GDN-NoPE hybrid model on the NIAH-SK1, SK2, and SK3 tasks.
Model configurations for 376M/776M/1B/3B models in Appendix A.
Table 2: Model configurations for 376M/776M/1B/3B models in Appendix A.
Short-context performance of layer-wise hybrid models after short-context pretraining.
Table 3: Short-context performance of layer-wise hybrid models after short-context pretraining.
Short-context performance of head-wise hybrid models after short-context pretraining.
Table 4: Short-context performance of head-wise hybrid models after short-context pretraining.
Short-context performance of layer-wise hybrid models after long-context continual pretraining.
Table 5: Short-context performance of layer-wise hybrid models after long-context continual pretraining.
Short-context performance of head-wise hybrid models after long-context continual pretraining.
Table 6: Short-context performance of head-wise hybrid models after long-context continual pretraining.
Long-context performance of 376M and 776M layer-wise hybrid models after short-context pretraining.
Table 7: Long-context performance of 376M and 776M layer-wise hybrid models after short-context pretraining.
Long-context performance of 1B and 3B layer-wise hybrid models after short-context pretraining.
Table 8: Long-context performance of 1B and 3B layer-wise hybrid models after short-context pretraining.
Long-context performance of 376M and 776M head-wise hybrid models after short-context pretraining.
Table 9: Long-context performance of 376M and 776M head-wise hybrid models after short-context pretraining.
Long-context performance of 1B and 3B head-wise hybrid models after short-context pretraining.
Table 10: Long-context performance of 1B and 3B head-wise hybrid models after short-context pretraining.
Long-context performance of layer-wise hybrid models after long-context continual pretraining.
Table 11: Long-context performance of layer-wise hybrid models after long-context continual pretraining.
Long-context performance of head-wise hybrid models after long-context continual pretraining.
Table 12: Long-context performance of head-wise hybrid models after long-context continual pretraining.
Long-context performance of SWA-NoPE-LH under long-context continual pretraining with larger window sizes (wsz). Larger window sizes exhibit better long-context performance, implying the Short-Window Weariness.
Table 13: Long-context performance of SWA-NoPE-LH under long-context continual pretraining with larger window sizes (wsz). Larger window sizes exhibit better long-context performance, implying the Short-Window Weariness.
Long-context performance of SWA-NoPE-LH under short-context pretraining with larger window sizes (wsz). Shorter window sizes exhibit better length extrapolation, implying the Long-Window Laziness.
Table 14: Long-context performance of SWA-NoPE-LH under short-context pretraining with larger window sizes (wsz). Shorter window sizes exhibit better length extrapolation, implying the Long-Window Laziness.
Long-context performance of SWA layer-wise hybrid models after long-context continual pretraining with extended window sizes (wsz) and LongCE enhancement, both relieving the short-context learning trap effectively.
Table 15: Long-context performance of SWA layer-wise hybrid models after long-context continual pretraining with extended window sizes (wsz) and LongCE enhancement, both relieving the short-context learning trap effectively.
Extrapolation comparison of the GLA/GDN-NoPE hybrid model on the NIAH-SK1, SK2, and SK3 tasks.
Table 16: Extrapolation comparison of the GLA/GDN-NoPE hybrid model on the NIAH-SK1, SK2, and SK3 tasks.
Long-context performance of head-wise hybrids with different hybrid ratios after short-context pretraining.
Table 17: Long-context performance of head-wise hybrids with different hybrid ratios after short-context pretraining.
LongBench performance of 376M and 776M layer-wise hybrid models after short-context pretraining.
Table 18: LongBench performance of 376M and 776M layer-wise hybrid models after short-context pretraining.
LongBench performance of 1B and 3B layer-wise hybrid models after short-context pretraining.
Table 19: LongBench performance of 1B and 3B layer-wise hybrid models after short-context pretraining.
LongBench performance of 376M and 776M head-wise hybrid models after short-context pretraining.
Table 20: LongBench performance of 376M and 776M head-wise hybrid models after short-context pretraining.
LongBench performance of 1B and 3B head-wise hybrid models after short-context pretraining.
Table 21: LongBench performance of 1B and 3B head-wise hybrid models after short-context pretraining.
LongBench performance of layer-wise hybrid models after long-context continual pretraining.
Table 22: LongBench performance of layer-wise hybrid models after long-context continual pretraining.
LongBench performance of head-wise hybrid models after long-context continual pretraining.
Table 23: LongBench performance of head-wise hybrid models after long-context continual pretraining.
Long-context performance of layer-wise hybrid models under the YOCO-like layout.
Table 24: Long-context performance of layer-wise hybrid models under the YOCO-like layout.
Long-context performance of SWA-NoPE-LH with smaller rotary bases, from 10000 to 100, under direct or log-scaled extrapolation.
Table 25: Long-context performance of SWA-NoPE-LH with smaller rotary bases, from 10000 to 100, under direct or log-scaled extrapolation.
Comparison of training loss, perplexity (PPL), and LongPPL among different layer-wise hybrid models after 4k context length pretraining. SWA/GLA/GDN-NoPE hybrids achieve stronger long-context language modeling.
Figure 1: Comparison of training loss, perplexity (PPL), and LongPPL among different layer-wise hybrid models after 4k context length pretraining. SWA/GLA/GDN-NoPE hybrids achieve stronger long-context language modeling.
Comparison of long-context performance under RULER and BABILong benchmarks among different layer-wise hybrid models after 4k context length pretraining.
Figure 2: Comparison of long-context performance under RULER and BABILong benchmarks among different layer-wise hybrid models after 4k context length pretraining.
Comparison of training loss, perplexity (PPL), and LongPPL among different head-wise hybrid models after 4k context length pretraining.
Figure 3: Comparison of training loss, perplexity (PPL), and LongPPL among different head-wise hybrid models after 4k context length pretraining.
Comparison of long-context performance under RULER and BABILong benchmarks among different head-wise hybrid models after short-context pretraining.
Figure 4: Comparison of long-context performance under RULER and BABILong benchmarks among different head-wise hybrid models after short-context pretraining.
Comparison of long-context performance between RoPE-only and RoPE-NoPE hybrid full-attention models. Both layer-wise and head-wise RoPE-NoPE hybrids have better long-context performance and can achieve effective length extrapolation directly or under log-scaled NoPE extrapolation.
Figure 5: Comparison of long-context performance between RoPE-only and RoPE-NoPE hybrid full-attention models. Both layer-wise and head-wise RoPE-NoPE hybrids have better long-context performance and can achieve effective length extrapolation directly or under log-scaled NoPE extrapolation.
Comparison of long-context performance under RULER and BABILong benchmarks among different layer-wise hybrid models after 32k context length continual pretraining.
Figure 6: Comparison of long-context performance under RULER and BABILong benchmarks among different layer-wise hybrid models after 32k context length continual pretraining.
Comparison of long-context performance under RULER and BABILong benchmarks among different head-wise hybrid models after 32k context length continual pretraining.
Figure 7: Comparison of long-context performance under RULER and BABILong benchmarks among different head-wise hybrid models after 32k context length continual pretraining.
Comparison of the long-context performance of the SWA/GLA/GDN-NoPE layer-wise hybrid models under log-scale NoPE extrapolation after short-context pretraining (left) and after long-context continual pretraining (right). The results of SWA hybrids exhibit a remarkable seesaw effect.
Figure 8: Comparison of the long-context performance of the SWA/GLA/GDN-NoPE layer-wise hybrid models under log-scale NoPE extrapolation after short-context pretraining (left) and after long-context continual pretraining (right). The results of SWA hybrids exhibit a remarkable seesaw effect.
Comparison of the long-context performance of the SWA/GLA/GDN-NoPE head-wise hybrid models under log-scale NoPE extrapolation after short-context pretraining (left) and after long-context continual pretraining (right).
Figure 9: Comparison of the long-context performance of the SWA/GLA/GDN-NoPE head-wise hybrid models under log-scale NoPE extrapolation after short-context pretraining (left) and after long-context continual pretraining (right).
Comparison of length extrapolation in GLA-NoPE hybrid models. Applying a sliding window for GLA and a log scale for NoPE attention achieves the best length extrapolation performance.
Figure 23: Comparison of length extrapolation in GLA-NoPE hybrid models. Applying a sliding window for GLA and a log scale for NoPE attention achieves the best length extrapolation performance.
Comparison of length extrapolation in GDN-NoPE hybrid models. Still, applying a sliding window for GDN and a log scale for NoPE attention achieves the best length extrapolation performance in most cases.
Figure 24: Comparison of length extrapolation in GDN-NoPE hybrid models. Still, applying a sliding window for GDN and a log scale for NoPE attention achieves the best length extrapolation performance in most cases.
Comparison of EME in RoPE-NoPE, GLA-NoPE, and GDN-NoPE hybrids as well as SWA-NoPE hybrids.
Figure 25: Comparison of EME in RoPE-NoPE, GLA-NoPE, and GDN-NoPE hybrids as well as SWA-NoPE hybrids.
A brief comparison between the traditional full attention model and hybrid models on training efficiency.
Figure 27: A brief comparison between the traditional full attention model and hybrid models on training efficiency.
Comparison of the LongBench performance of the SWA/GLA/GDN-NoPE hybrid models under log-scale NoPE extrapolation (Short-Log) and context extension (Long). The Seesaw Effect still exists.
Figure 28: Comparison of the LongBench performance of the SWA/GLA/GDN-NoPE hybrid models under log-scale NoPE extrapolation (Short-Log) and context extension (Long). The Seesaw Effect still exists.
Comparison of LongBench performance in RoPE-NoPE hybrid models. Applying a sliding-window RoPE attention and a log-scale NoPE attention achieves the best length extrapolation. The Matthew Effect still works.
Figure 29: Comparison of LongBench performance in RoPE-NoPE hybrid models. Applying a sliding-window RoPE attention and a log-scale NoPE attention achieves the best length extrapolation. The Matthew Effect still works.
Comparison of LongBench performance in GLA-NoPE hybrid models. Applying a sliding-window GLA and a log-scale NoPE attention achieves the best length extrapolation. The Matthew Effect still works.
Figure 30: Comparison of LongBench performance in GLA-NoPE hybrid models. Applying a sliding-window GLA and a log-scale NoPE attention achieves the best length extrapolation. The Matthew Effect still works.
Comparison of LongBench performance in GDN-NoPE hybrid models. Applying a sliding-window GDN and a log-scale NoPE attention achieves the best length extrapolation. The Matthew Effect still works.
Figure 31: Comparison of LongBench performance in GDN-NoPE hybrid models. Applying a sliding-window GDN and a log-scale NoPE attention achieves the best length extrapolation. The Matthew Effect still works.
查看结构化数据
任务指标本文基线提升
NIAH-SK1（RULER 大海捞针，4k 训练→64k 免训练推理） 准确率 (%) GLA-NoPE-LH 776M + SWLA&Log（EME）：100.0@64k GLA-NoPE-LH 776M 不加处理：0.0@64k 0→100，实现 16 倍免训练长度外推
NIAH-SK2（双针检索，776M 层混合） 准确率 (%) GDN-NoPE-LH + EME：90.0@64k GDN-NoPE-LH：0.0@64k +90 个百分点
NIAH-SK3（三针检索，776M 层混合） 准确率 (%) GLA-NoPE-LH + EME：64.0@64k GLA-NoPE-LH：0.0@64k +64 个百分点
RULER+BABILong 全平均（1B 层混合，4k 预训练后直接外推） 平均分 GLA-NoPE-LH + EME：40.2（64k 处 37.5） GLA-NoPE-LH：26.6（64k 处 1.5） 全平均 +13.6 分；64k 处 +36 个百分点
SWA-NoPE-LH 上下文扩展（776M，32k 续训） RULER+BABILong 全平均 续训窗口 2k + LongCE：37.4（1B 42.7，3B 53.7） 默认 128 窗口续训：29.4（1B 37.6，3B 45.9） 776M +8.0 分，修复跷跷板效应、追平 GDN-NoPE-LH
长度外推方法对比（1B 层混合，4k 预训练） RULER+BABILong 全平均 SWA-4k-NoPE-LH + Log（EME）：43.1 RoPE full attention + Dynamic NTK + Log：24.3 +18.8 分，混合位置外推大幅优于传统插值
短上下文通用能力（2k 内，Open LLM Leaderboard 12 项） 平均分 776M GDN-NoPE-LH：41.3 776M RoPE full attention：40.3 持平略优，混合不损失通用能力

局限与改进

作者承认三点：验证范围局限于预训练阶段，未覆盖 post-training 与多模态；架构范围未含稀疏注意力混合与压缩注意力混合；机理范围未分析 sink bias、partial RoPE、门控变体与短卷积。我的补充：(1) 最大规模仅 3B、训练量 50B+5B tokens，结论向 10B+/数 T tokens 外推未知，且论文自己指出短上下文学习陷阱随模型增大更明显，说明该现象可能与规模强耦合；(2) SWLA 推理需窗口 4096/步长 1024 的分块重算，其真实开销未与原生 LA 对齐比较，效率图测的是不加 SWLA 的模型；(3) 主评测是合成检索与 BABILong，LongBench 上小模型平均分仅约 8–25，对真实长文档任务指导有限；(4) 马太效应在 YOCO 式布局下收益明显减弱（Figure 34），说明结论对层布局敏感，但缺少'何时失效'的判据；(5) 扩窗需把 rotary base 抬到 $4\times10^8$ 量级，降低位置判别力，Table 25 显示 base 从 10000 降到 100 反而更好但无一致趋势，最优（训练长度, 窗口, base）调度器仍是开放问题。

独立分析的弱点

独立分析的五点弱点。(1) 短上下文学习陷阱的因果链依赖 $W_Q/W_K$ 变化率与 PPL 曲线等间接证据，缺少干预实验：若续训时冻结 SWA 分支的 $W_Q, W_K$，跷跷板是否消失？分支冻结/梯度掩码消融能直接验证因果。(2) SWLA 的窗口 4096/步长 1024 为手工设定，未做敏感性扫描，也未解释为何窗口须与训练长度对齐；应系统研究（窗口、步长、训练长度、门控衰减参数）的联合缩放律。(3) 命中率实验只在 SK1 大海捞针上做，多跳推理、聚合、状态追踪任务中的头级功能分工可能完全不同，应按任务类型分别刻画分区。(4) 潮汐效应只给出定性分区，与'3:1 最优'之间没有定量模型，无法回答'给定算力与任务分布，最优 NoPE 头比例是多少'；可把熵-命中率分布做成可微目标，让每层每头的位置偏置分配可学习。(5) SWLA 的分块近似改变了注意力计算图（每个输出只聚合窗口内历史），与完整线性递归的偏差、对长程状态漂移的影响未被量化，对需要精确状态追踪的任务可能引入系统性误差。

未来方向

作者提出的方向：把力学分析扩展到 post-training 与视觉-语言多模态；覆盖稀疏注意力混合与压缩注意力混合；分析 sink bias、partial RoPE、门控变体、短卷积等位置设计；为 SWA 混合开发（训练长度, 窗口, rotary base）最优调度器（Table 25 显示小 base 总体更好但无一致趋势）；厘清 GLA 与 GDN 的差异是否源于 delta rule（GDN 在熵-命中率图上占据部分高熵高命中区而 GLA 没有）。基于本文成果可延伸：(1) 把 SWLA 从推理时分块近似内化到训练阶段，设计原生'硬窗口线性注意力'kernel；(2) 用潮汐效应指导动态/可学习混合比例，让模型自动调整每层 NoPE 头数量，替代固定 3:1；(3) 把扩窗+LongCE 固化为长上下文续训标准配方，并与 MoE、量化等正交技术叠加验证；(4) 在 agentic 百万 token 场景检验 EME 的极限外推倍数；(5) 把扩展注意力熵用作训练监控信号，在续训早期检测短上下文学习陷阱并自动触发扩窗。

复现评估

代码已在 GitHub 开源（OpenMOSS/Hybrid-Mechanics），训练栈为 DeepSpeed + FlashAttention2 + FlashLinearAttention，数据全部公开（DCLM-Baseline-1.0、LongWanjuan、PG19、RULER、BABILong、LongBench），tokenizer 用 Llama 3。算力需求中等：4k 预训练 50B tokens 用 8×H200（3B 用 16×H200），32k 续训仅 5B tokens，单次训练成本可控，学术界可负担。但结论建立在很大的消融矩阵上（4 规模 × 约 20 种配置 × 两阶段训练），完整复现全部图表需数十次训练；建议沿关键路径低成本验证：先做 376M 层混合系列（跷跷板+扩窗+LongCE），再做 776M 的 SWLA/EME 外推（Table 1）。不确定点：LongCE 公式需参考其原始文献 [43]；SWLA 分块近似与 FLA kernel 的接口细节需读代码；评测协议文中已交代清楚。总体难度：中等——单点实验可复现，全矩阵复现昂贵。

论文图表

Figure 26: A brief comparison between the traditional full attention model and hybrid models on inference efficiency.

376M–3B、1k–64k 上下文上的 TTFT、TPOT、峰值显存与 KV/状态缓存对比：混合模型全面占优，且规模与长度越大优势越明显；GLA/GDN 的 FP32 状态缓存与 SWA-128 FP16 缓存相当；头混合 SWA 因 QKV 拼接峰值显存与 full attention 持平。

补齐效率维度，说明混合架构的收益不仅是质量，还有部署成本；也揭示了头混合 SWA 的工程细节差异。

Comparison of long-context training performance between SWA-NoPE-LH, GLA-NoPE-LH, and GDN-NoPE-LH under different learning rates, where 0 learning rate means the performance before long-context continual pretraining.
Figure 32: Comparison of long-context training performance between SWA-NoPE-LH, GLA-NoPE-LH, and GDN-NoPE-LH under different learning rates, where 0 learning rate means the performance before long-context continual pretraining.

在 1e-4/2e-4/5e-4/1e-3 四个峰值学习率下重复续训对比（376M/776M）：无论学习率如何，续训后 SWA-NoPE-LH 都稳定落后于 LA-NoPE-LH。

排除'跷跷板是学习率伪影'的质疑，是核心现象鲁棒性的重要控制实验。

Comparison of the long-context performance of the SWA/GLA/GDN-NoPE layer-wise hybrid models in YOCO-like layout under log-scale NoPE extrapolation and context extension. The Seesaw Effect still exists.
Figure 33: Comparison of the long-context performance of the SWA/GLA/GDN-NoPE layer-wise hybrid models in YOCO-like layout under log-scale NoPE extrapolation and context extension. The Seesaw Effect still exists.

在 YOCO 式布局（高效注意力层全部放在低层）下重复跷跷板实验：现象依旧存在。

证明跷跷板效应不依赖 3:1 交错布局，是现象普适性的第二层验证。

Performance of enhanced SWA-NoPE-LH context extension and RoPE/GLA/GDN-NoPE-LH extrapolation based on the Matthew Effect in layer-wise hybrid models in YOCO-like layout.
Figure 34: Performance of enhanced SWA-NoPE-LH context extension and RoPE/GLA/GDN-NoPE-LH extrapolation based on the Matthew Effect in layer-wise hybrid models in YOCO-like layout.

在 YOCO 布局下测试本文两个配方的可迁移性：扩窗+LongCE 仍有效，但基于马太效应的 RoPE/GLA/GDN-NoPE 外推收益明显减弱。

诚实地划定了结论的边界：层布局会改变马太效应的有效性，提示读者不要过度外推。

The entropy-hit-rate diagram of RoPE-based full attention models and SWA/GLA/GDN-RoPE hybrid models.
Figure 35: The entropy-hit-rate diagram of RoPE-based full attention models and SWA/GLA/GDN-RoPE hybrid models.

展示与 RoPE（而非 NoPE）混合的模型（含头混合版本）的熵-命中率图：RoPE 失去左下低熵区，低熵区被 SWA/LA 占据，RoPE 反而呈现类似 NoPE 的带状分布。

挑战'带状区是 NoPE 专属'的简单解读，推动作者追问功能分区的本质归属，展示了严谨的自我质疑。

Figure 36: The entropy-hit-rate diagram of NoPE/RoPE-based full attention models and RoPE-NoPE-HH models.

纯 NoPE 模型的训练极不稳定（376M loss 在 5B/10B/15B tokens 时为 3.56/5.92/6.49，776M 在 15B 时达 7.20），其熵-命中率结构在训练中崩溃；RoPE 模型与 RoPE-NoPE 混合则早早形成稳定分布。

回答'带状分布是 NoPE 本性还是混合环境产物'：纯 NoPE 无法存活，故带状区应理解为 NoPE 在混合模型中的属性，同时解释了为什么 NoPE 只能当少数派。

相似论文推荐

重新思考混合架构中高效注意力的角色 🎯 81.3%
混合线性注意力的正确做法：超长上下文的高效蒸馏与有效架构 🎯 79.5%
MiniCPM-SALA：混合稀疏与线性注意力实现高效长上下文建模 🎯 74.8%
HydraHead：从头部级功能异构性到专业化注意力混合 🎯 74.0%
混合线性注意力大语言模型中的巨激活：全注意力前尖峰与尖峰间平台 🎯 71.8%