长上下文强化学习的数据配方：超越奖励工程 Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Data recipe improves long-context reasoning
前置知识
Long-Context Reasoning
The ability of large language models to locate relevant evidence, integrate scattered information, and perform complex reasoning when processing ultra-long inputs (e.g., 128K+ tokens). This is particularly important when modern LLMs are deployed as autonomous agents, as agents encounter large amounts of web pages, code repositories, and action histories where task-relevant information is often scattered across lengthy inputs.
This is the core research object of this paper. Understanding this concept is foundational to understanding the problem definition and solution approach.
GRPO (Group Relative Policy Optimization)
A reinforcement learning algorithm that samples a group of G outputs from the old policy for each prompt x, scores them with rewards, and calculates the group-relative advantage for each output. The final objective function maximizes the clipped surrogate loss across all tokens in all outputs.
This is the RL algorithm used in this paper. Understanding how it works helps understand the training setup and why effective long-context RL can be achieved.
Needle-in-a-Haystack
A long-context test task that embeds a short key piece of information (needle) in a large amount of irrelevant text (haystack) and requires the model to retrieve that key information. This task has become a basic benchmark for evaluating long-context capability but is largely saturated on modern long-context models.
This is the starting point for retrieval task design in this paper. Understanding the saturation state of this benchmark helps understand why more complex retrieval tasks (FuzzyNeedle and MultiNeedle) need to be designed.
研究动机
Existing long-context reinforcement learning methods focus primarily on reward engineering, attempting to guide models through more sophisticated reward signals. However, these methods face two core problems. First, the outcome-only reward of standard RLVR is too sparse to effectively guide models in locating evidence over long inputs, leading to plateaued contextual recall and shortcut reasoning that bypasses grounding. Second, high-quality long-context training data is extremely scarce, with existing synthetic data pipelines either narrow in task coverage or kept closed-source. This causes models to perform poorly in real-world agent tasks that require processing large amounts of web pages, code repositories, and action histories.
本文的目标是The goal is to revisit the long-context RL problem from a data-centric perspective and demonstrate that a simple yet effective data recipe, even when paired with a minimal outcome-based GRPO setup, suffices to significantly improve long-context reasoning capability. Specifically, the authors aim to demonstrate that by carefully constructing training data covering three complementary core abilities, consistent improvements can be achieved across seven long-context benchmarks, and these improvements can transfer to agent tasks.
与已有工作不同的是，The unique angle of this paper is abandoning complex reward engineering and focusing instead on data recipe construction. This contrasts with existing work: algorithm-centric methods focus on improving the RL training process (reward design, optimization algorithms), while data-centric methods construct long-context RL training sets that are either limited in scale and diversity or confined to a single synthetic task format. This paper proposes a more unified recipe that simultaneously exercises three complementary abilities of long-context reasoning, without special reward engineering or staged RL training.
核心方法
The core idea is that long-context reasoning can be decomposed into three complementary core abilities: retrieval, multi-evidence synthesis, and reasoning. By constructing sufficiently diverse training data to exercise all three abilities simultaneously, paired with a minimal outcome-based GRPO setup, it suffices to significantly improve long-context reasoning performance. The data recipe contains 8 datasets totaling approximately 14K examples, distributed across three task categories. Training uses GRPO with G=8 samples per prompt, temperature 1.0, top-p=0.95, input length capped at 64K tokens, and generation length capped at 16K tokens. For training stability, task-balanced sampling and task-level advantage normalization are applied.
The core innovation is a data-centric perspective proposing a unified training mixture that simultaneously exercises three complementary abilities of long-context reasoning. The essential difference from existing methods is that existing work focuses either on reward design (algorithm-centric) or constructs training sets with limited scale or single task format (data-centric). This paper contributes a larger-scale training mixture (14K examples) with more comprehensive task coverage, demonstrating that such a simple data recipe is effective without complex reward engineering. Another key innovation is recognizing the importance of task balancing, applying task-balanced sampling and task-level advantage normalization to mitigate task competition in multi-task RL.
方法步骤详情
The method consists of two main steps. Step 1 constructs the long-context training data mixture. For each of three task categories, the authors constructed datasets: retrieval includes FuzzyNeedle and MultiNeedle; multi-evidence synthesis includes CrossEntity, WebSearch, MultiQuery, KeyChain, and LongDocQA; reasoning includes LongMath. All data undergoes two-stage post-processing: discarding samples exceeding 64K tokens, then rolling out each question 4 times with Qwen3-30B-A3B-Thinking-2507 and retaining only samples of moderate difficulty (neither all-correct nor all-wrong). Step 2 is minimal long-context RL setup using GRPO with word-level recall reward for most datasets and DeepSeek-V3.2 as judge for CrossEntity and LongMath.
技术新颖性
Technical novelty is reflected in three aspects: first, proposing a systematic taxonomy of long-context reasoning abilities decomposed into retrieval, multi-evidence synthesis, and reasoning, providing a clear analytical framework for future research; second, constructing 8 carefully designed datasets, each targeting specific weaknesses of long-context LLMs, with data scale (14K examples) significantly larger than existing long-context RL datasets; third, demonstrating the effectiveness of data recipe itself without complex reward engineering or staged training, contrasting with the current reward-engineering-dominated research paradigm. Additionally, ablation experiments demonstrate the complementarity of three task categories and validate transferability to agent tasks.
实验结果
The core finding is that the data recipe achieves consistent improvements across all three models. On Qwen3-4B-Thinking-2507, the average score across seven benchmarks improved from 53.42 to 60.63 (+7.21); on Qwen3-8B-128K, from 49.93 to 53.10 (+3.17); on Qwen3-30B-A3B-Thinking-2507, from 61.82 to 68.23 (+6.41). Significant gains appear on reasoning-intensive benchmarks: for Qwen3-4B-Thinking-2507, improvements of +7.0, +10.5, and +8.6 on LBv2, AA-LCR, and DocFinQA respectively; for Qwen3-30B-A3B-Thinking-2507, +4.8, +7.3, and +10.5 respectively. Ablation experiments show the reasoning category contributes most (+3.46 average alone), retrieval+reasoning combination works best (+4.06 average), but full mixture is optimal (+4.16 average), proving three task categories are complementary. Removing task balancing causes performance drop (57.58 to 56.05, -1.53), indicating multi-task long-context RL still suffers from task competition. Adding process reward does not improve (57.58 to 56.29, -1.29). Transfer experiments on AgentCPM-Explore show after 50-step training, GAIA Pass@3 improved by +4.8, BrowseComp by +7.0, and long-context average by +5.54.
查看结构化数据
| 任务 | 指标 | 本文 | 基线 | 提升 |
|---|---|---|---|---|
| LongBench v2 QA (average) | Accuracy | 82.52 (4B) | Qwen3-4B-Thinking-2507: 77.55 | +4.97 |
| AA-LCR | Avg@4 | 48.37 (4B) | Qwen3-4B-Thinking-2507: 39.80 | +8.57 |
| LongReason | Accuracy | 82.12 (4B) | Qwen3-4B-Thinking-2507: 69.50 | +12.62 |
| DocFinQA | Accuracy | 45.75 (4B) | Qwen3-4B-Thinking-2507: 35.25 | +10.50 |
| GAIA | Pass@3 | 70.9 (50 steps) | AgentCPM-Explore: 66.1 | +4.8 |
| BrowseComp | Pass@3 | 34.0 (50 steps) | AgentCPM-Explore: 27.00 | +7.0 |
局限与改进
Limitations acknowledged by authors include: experiments limited to Qwen3 family up to 30B-A3B-Thinking scale, not validated on larger models from other families due to computational constraints; all RL training constrained to inputs under 64K tokens, extension to longer context lengths left for future work; most datasets in the mixture are synthetic, while effective for improving long-context reasoning, may not fully match real-world long-context input distributions; agent task transfer study is preliminary, limited to one agent-tuned model and two benchmarks, broader relationship between long-context reasoning and agent performance remains to be explored. Observed additional limitations include: task balancing techniques still show task competition (1.53 drop after removal), indicating need for better multi-task RL strategies; LLM-as-judge process reward introduces noise rather than improvement, suggesting that without ground truth evidence, additional process-level supervision may be harmful; although 64K training generalizes to longer contexts (+2.1 at 512K+), gains diminish beyond training distribution, suggesting potential benefits of longer-context training.
独立分析的弱点
Independent analysis weaknesses include: the data recipe is primarily based on synthetic data, which may not fully match real-world agent task distributions such as web searching and GUI automation, potentially leading to smaller real-world improvements than benchmark results. The static single-turn QA format may not adequately simulate the long-term memory and context accumulation characteristics of multi-turn dialogue required by agent tasks. Task balancing techniques, while effective, still show performance drop, lacking more fine-grained control over task difficulty and data quality. Improvement directions include: constructing training data on real-world agent trajectories, developing dynamic task balancing strategies, and exploring long-context RL methods for multi-turn dialogue scenarios.
未来方向
Future work proposed by authors includes validating the recipe on larger models from other families and extending RL training to longer context lengths. Extendable directions based on paper results include: validating transferability to broader real-world agent tasks such as software engineering and GUI automation; developing more sophisticated multi-task RL strategies beyond simple task-balanced sampling, potentially including task difficulty adaptive sampling or curriculum learning; exploring automated data recipe construction methods to reduce manual design and validation effort; studying synergy between long-context RL training and other training paradigms such as pre-training and instruction fine-tuning; analyzing impact of data recipe on different model architectures and architecture-specific optimization strategies.
复现评估
The paper promises to release datasets following original data source licenses. Experiments use Miles framework for RL training with relatively complete configuration details. Main challenges are high computational requirements and need for careful multi-task balancing tuning. Reproduction difficulty is medium-high, requiring large-scale GPU resources and RL training experience.
论文图表
