← 返回 2026-10-02

ActiveSaddler：面向智能体Harness优化的自动化课程学习 ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization

Sungho Park, Wonjoong Kim, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Victor Rühle 📅 2026-10-01 👍 53 2026-11-11 18:30
Harness优化 LLM智能体 多臂老虎机 探索-利用权衡 自动课程学习
arXiv HuggingFace PDF

将训练场景选择建模为非平稳多臂老虎机，让优化课程与harness共同演化

前置知识

Agent Harness（智能体线束）

包裹在LLM外层的工程框架，决定模型如何被提示（system prompt、ReAct循环）、能调用哪些工具及其接口、执行过程如何被控制监控与纠错（如executor hook、重试回滚、超时处理）。模型权重完全不变，仅调整harness就能显著改变智能体在多步长程任务上的可靠性，因此Anthropic、OpenAI、LangChain等都强调harness工程。

本文的优化对象不是模型而是harness：底座优化器AutoSaddler会自动诊断执行轨迹缺陷并生成补丁修改prompt、工具接口与运行时控制逻辑，理解harness的构成才能理解优化器在改什么、课程层在为哪些更新提供证据。

自动Harness优化（AHO）

让优化器自动迭代改进harness的闭环流程：在训练场景上执行当前harness得到轨迹与对错结果，从失败中诊断缺陷，提出候选补丁，再用开发集做回归检查与候选筛选，保留更优harness进入下一轮。代表方法有GEPA（反思式prompt进化+Pareto搜索）、Meta-Harness（端到端优化）和本文的底座AutoSaddler（从执行轨迹做持久化更新）。

ActiveSaddler不发明新优化器，而是插入AHO循环接管“每轮喂给优化器哪些训练场景”这一决策；论文的对比基线与train/dev/test实验协议都建立在AHO的标准流程之上。

多臂老虎机与非平稳bandit

序贯决策模型：每轮从若干“臂”中选一个拉动并获得随机收益，目标是在有限次数内最大化累计收益。非平稳设定下臂价值随时间漂移，需要持续重估；无限多臂变体（如UCB-AIR）还要求在已知臂与“可能存在但未观测到的新臂”之间分配尝试次数。

ActiveSaddler把“选哪类失败来优化”形式化为非平稳bandit：每个臂对应一个失败模式，收益是修复该弱点带来的性能提升；臂优先器用softmax分布随机选臂，探索控制器处理未观测新臂问题，消融还与UCB-AIR调度直接对比。

课程学习与自动课程学习

Bengio等2009年提出：不随机均匀地喂训练数据，而是按难度或价值调度训练样本/任务/环境。自动化版本（Graves 2017、TSCL/Matiisen 2017）用可观测的学习者信号（loss下降、胜任力进步、TD误差）估计每个任务的“学习进展”，据此动态排课，常被建模为bandit问题。

本文把这一思想迁移到harness优化，但指出关键差异：这里没有loss式的目标级信号表明某弱点是否仍值得干预，必须由LLM从优化历史推断，这正是失败模式臂与学习进展分数$\phi$设计的出发点。

Pass@1

智能体评测常用指标：智能体对每个任务独立执行一次，成功即得分，通过率即Pass@1。本文报告的是同一优化后harness在测试集上独立执行三次的均值±标准差，衡量最终交付harness的单次成功率。

论文所有主表（Table 1、4、5、6）都用测试集Pass@1比较方法，GAIA2与TB2两条主结论（59.8% vs 55.4%、80.0% vs 72.5%）均以该指标表述。

探索-利用权衡（exploration-exploitation）

在信息不全的序贯决策中，“利用”指把资源投向当前已知有价值的选项，“探索”指尝试信息不足的选项以发现可能更优的选择。两者相对价值随状态变化：已知选项仍有油水时偏向利用，被榨干时偏向探索。

ActiveSaddler的探索控制器每轮在DRAW（执行未见场景、可能发现新失败模式）与PULL（重访已知失败模式继续修复）之间二选一；消融显示去掉它GAIA2/TB2各掉4.5/8.3个百分点。

研究动机

自动harness优化方法（GEPA、Meta-Harness、AutoSaddler等）几乎都把注意力放在“如何根据执行反馈更新harness”上，而驱动更新的训练场景通常在优化开始前就已定死：要么用全量训练集，要么用预先排好的mini-batch顺序。这在预算受限时造成两类浪费。其一，harness演化后各场景的剩余价值剧变——已修复的场景继续占用rollout几乎没有收益，而未解决的失败却可能过早被跳过；数据显示固定顺序下AutoSaddler训练期观察到的失败最终只有21.7%（GAIA2）/33.3%（TB2）被转化为成功（Figure 3b）。其二，“重访已知失败”与“执行未见场景以发现新失败”的相对价值也随优化进程漂移，固定时间表无法自适应。在本文的紧凑预算下（$K=10\times(|D_{train}|+|D_{dev}|)$，GAIA2共1400次、TB2共490次任务rollout，每轮最多3个场景），每一次错配都是实打实的损失。

本文的目标是本文要把“为每次harness更新选哪些训练场景”这个被忽视的维度，形式化为预算受限离线优化中的自动化课程学习问题：把当前harness视为“学习者”，课程策略$\pi$依据历史信息$I_{<t}$选择下一批训练场景$X_t\subseteq D_{train}$，在所有方法共享相同rollout预算$K$的前提下最大化按开发集选出的最终harness的期望性能，$\pi^*\in\arg\max_\pi \mathbb{E}[J(\hat{\theta}_\pi)]$。具体目标拆成三个可操作组件：把反复出现的失败抽象为可复用的优化目标（失败模式臂）、估计继续攻每个目标的潜在收益（臂优先器）、在重访已知弱点与探索未见场景之间动态分配机会（探索控制器），让课程与harness共同演化。

与已有工作不同的是，自动课程学习已有成熟传统（competence/learning progress优先级、TSCL的EMA评分、教师-学生任务选择），但作者指出直接照搬行不通：harness优化中课程目标与进展都是隐式的——一个训练场景并不固定对应某个弱点（同一场景在不同harness下暴露不同失败），同一弱点又横跨多个场景，且被诊断出的弱点没有任何loss式信号说明它是否仍未解决、是否值得再干预，只能从当前harness、历史补丁与执行记录推断。与近邻工作的切入也不同：CoEvolve面向模型权重训练的任务生成；Task-CoEvolve与HarnessLens都在“候选修改已提出之后”的评估侧选任务，而ActiveSaddler作用于训练证据生成侧，贯穿优化全程地反复决定重访哪个已知弱点、何时探索新弱点。

核心方法

直觉上可以把harness优化器看成一个学生：执行轨迹中的失败是“知识漏洞”，预算有限时应优先复习剩余油水最多的漏洞，同时定期接触新题以发现尚未暴露的漏洞。技术路线上，ActiveSaddler把场景选择建模为非平稳多臂老虎机，臂=失败模式（failure pattern），在线实例化。每轮迭代$t$的输入是当前harness $H_t$、历史$I_{<t}$、已实例化臂集$A_t$与未见场景池$U_t$。探索控制器先做二元决策$d_t\in\{\text{unseen},\text{arm}\}$：选unseen（DRAW）就从$U_t$无放回采样每轮至多3个场景；选arm（PULL）则由臂优先器给每臂打学习进展分$\phi_t(a)\in[0,1]$，按$\text{softmax}(\phi/\tau_{sel})$分布采样一个臂并从其支持场景集构造批次。所选场景$X_t$在$H_t$上执行得到记录$E_t$，交给固定优化器$\mathcal{O}$（AutoSaddler的诊断-打补丁-反思-演化流程）产出$H_{t+1}$；失败模式提取器再把失败证据抽象成症状并归一化到现有臂或新建臂。循环往复，课程与harness共同演化，预算耗尽后按dev集表现选出最终harness。

核心创新是“优化目标本身被动态构建并持续重估”，与已有方法的本质区别有三。第一，臂的表示：以症状级失败模式为臂，而非类别或单个场景——类别臂太粗，不同弱点共享一个优化状态，部分场景的成功会掩盖其他未解决弱点（案例tnxtee属于time类臂，该类一次批次通过后severity从0.62骤降到0.12，tnxtee始终未被采样）；场景臂太细，同一弱点碎成75个独立臂（超过失败模式35个上限的两倍多），预算内难获重访。第二，优先级自适应：LLM综合severity/fixability/breadth/side-effect四轴估计$\phi$并在每次选择时重算，使预算流向“当前仍有学习潜力”的弱点——已全通过的68.6%的臂只分到26.2%的拉力概率。第三，探索时机自适应：LLM按当前优化状态逐轮决定DRAW/PULL，而非固定每5轮探索一次。三者共同把“喂什么数据”提升为与“怎么更新”并列的优化维度。

方法步骤详情

算法1每轮迭代五步。初始化：$H_0$为基准harness，$A_0=\emptyset$，$U_0=D_{train}$，循环至预算$K$耗尽。(1)探索决策：边界情形强制DRAW（无臂）或PULL（无未现场景），否则由LLM会话$d_t=g_{explore}(H_t,A_t,U_t,I_{<t})$二选一。(2)构造批次：DRAW从$U_t$无放回采样$X_t$（每轮≤3场景）并移出池；PULL按四轴(severity+fixability+breadth+(1−side_effect))/4为每臂打学习进展分$\phi_t(a)\in[0,1]$，以softmax分布（$\tau_{sel}=0.15$）采样臂并从其支持场景集构造$X_t$。(3)执行与优化：执行$H_t$得$E_t$；AutoSaddler诊断失败、生成补丁候选$H'_t$并同批复验，改善则上dev做回归/泛化检查，反思会话更新EvoDAG后演化出$H_{t+1}$。(4)模式提取：对每个失败独立抽取症状（不看现有臂以免锚定），经$g_{norm}$三路归一化（匹配/组合/新建），把(harness,trace,scenario)元组连根因经pattern CLI标注到臂。(5)状态更新；全部场景看过且每臂访问后，旧成功场景重入$U_t$以捕获静默回归。结束后按dev选出最终harness。

技术新颖性

技术新颖性可从四组对照看清。对经典自动课程（Graves 2017的loss信号、TSCL的EMA）：本文的臂价值无法从标量信号计算，需LLM读取harness代码、补丁意图、dev影响与执行历史做四轴判断，且臂空间在线增长、目标随harness漂移，属于非平稳+臂可生长的bandit。对AHO同行（GEPA、Meta-Harness、AutoSaddler）：完全不动更新机制，只换证据来源，因此正交可插拔——挂到GEPA上也有+3.0pp。对自适应任务选择近邻（Task-CoEvolve、HarnessLens）：那些方法在评估侧筛选任务，本文在训练侧决定证据生成，解决不同环节的问题。对非LLM替代：同框架下把$\phi$换成EMA残差失败率掉3.1pp、把探索控制器换成UCB-AIR计数调度掉4.2pp，说明LLM的状态理解对两个关键决策都不可替代。工程上，Pattern Registry加pattern CLI（带每会话capability manifest的读写权限控制）让不断增长的臂历史可被按需检索而非塞进prompt，这是让LLM课程器可行的关键基础设施。

Overview of ActiveSaddler. At each iteration, the curriculum chooses between exploring unseen scenarios and revisiting an instantiated arm, optimizes the harness on the selected scenario batch, and updates its state based on the resulting execution and optimization outcomes.
Figure 2: Overview of ActiveSaddler. At each iteration, the curriculum chooses between exploring unseen scenarios and revisiting an instantiated arm, optimizes the harness on the selected scenario batch, and updates its state based on the resulting execution and optimization outcomes.
Adaptive exploration over the optimization trajectory. The policy chooses DRAW when discovery is more valuable and PULL when promising repair opportunities remain.
Figure 4: Adaptive exploration over the optimization trajectory. The policy chooses DRAW when discovery is more valuable and PULL when promising repair opportunities remain.
Case study on tnxtee (GAIA2), where a failure-pattern arm instantiated from the initial failure enables an unsuccessful repair to be retargeted and refined in a subsequent iteration.
Figure 6: Case study on tnxtee (GAIA2), where a failure-pattern arm instantiated from the initial failure enables an unsuccessful repair to be retargeted and refined in a subsequent iteration.
Case study on 4bytkx (GAIA2), where retargeting the same failure-pattern arm corrects an executor hook that was initially inserted too late in the execution flow.
Figure 7: Case study on 4bytkx (GAIA2), where retargeting the same failure-pattern arm corrects an executor hook that was initially inserted too late in the execution flow.
Case study of a resolved weakness (P3 of GAIA2). After an accepted patch at Iteration 5, the arm's priority drops sharply. A later pull finds all supporting scenarios successful and produces no further patch, after which the arm remains at low priority.
Figure 9: Case study of a resolved weakness (P3 of GAIA2). After an accepted patch at Iteration 5, the arm's priority drops sharply. A later pull finds all supporting scenarios successful and produces no further patch, after which the arm remains at low priority.
Case study of a recurring weakness (P8 of GAIA2). Early pulls produce no patch because the associated scenarios succeed when re-executed, yet the arm remains highly prioritized. The same failure pattern later appears in additional scenarios, providing new evidence that the weakness remains active; a subsequent pull produces an accepted patch.
Figure 10: Case study of a recurring weakness (P8 of GAIA2). Early pulls produce no patch because the associated scenarios succeed when re-executed, yet the arm remains highly prioritized. The same failure pattern later appears in additional scenarios, providing new evidence that the weakness remains active; a subsequent pull produces an accepted patch.
Arm Scoring Prompt: the agent estimates a learning-progress score $\phi_t(a)$ for every instantiated failure-pattern arm by considering severity, fixability, breadth, and side-effect risk. The resulting scores define the stochastic prioritization of known arms.
Figure 15: Arm Scoring Prompt: the agent estimates a learning-progress score $\phi_t(a)$ for every instantiated failure-pattern arm by considering severity, fixability, breadth, and side-effect risk. The resulting scores define the stochastic prioritization of known arms.

实验结果

主结果（Table 1）：GAIA2测试集（3个未见Universe共300场景，OOD划分）上ActiveSaddler达59.8±1.0 Pass@1，比同优化器固定顺序的AutoSaddler（55.4±1.2）高4.4pp，也超GEPA与Meta-Harness（均54.2）；TB2（40任务）达80.0±2.5，比AutoSaddler（72.5）高7.5pp、比手动Terminus-KIRA（69.2）高10.8pp，对精度排序固定课程也占优（+3.9/+4.1pp与+9.2/+6.7pp）。消融：换类别臂/场景臂GAIA2掉3.0/3.6pp、TB2掉10.8/6.7pp；去臂优先器掉4.5/7.5pp；去探索控制器掉4.5/8.3pp。泛化：挂到GEPA上54.2%→57.2%（+3.0pp）；独立复跑61.6%仍超全部固定基线（Table 4）。机制：失败命中率52.9%/60.4%高于类别臂36.4%/30.5%、场景臂17.6%/36.6%（Figure 8）；失败→成功转化率50.0%/75.0%均最高（Figure 3b）；DRAW率44%→20%，PULL时未解决臂为DRAW的2.0×/6.9×、固定表仅0.8×/1.3×（Figure 13）。成本：每补丁优化器开销${'$'}7.87→${'$'}11.44，但GAIA2到58.5% dev仅${'$'}298（AutoSaddler ${'$'}1,360）、TB2到78.9%仅${'$'}128（AutoSaddler ${'$'}220）；GAIA2全程发现35个失败模式，已全通过的68.6%臂仅分得26.2%拉力概率。

Test Pass@1 on GAIA2 and Terminal-Bench 2.0 (mean ± std. over three test-time executions). Parentheses show task counts; bold indicates the best result.
Table 1: Test Pass@1 on GAIA2 and Terminal-Bench 2.0 (mean ± std. over three test-time executions). Parentheses show task counts; bold indicates the best result.
Commands exposed by the pattern CLI, grouped into read operations for scoped views of the arm population and write operations for per-session registry updates. Each session receives only the subset of commands in its capability manifest.
Table 2: Commands exposed by the pattern CLI, grouped into read operations for scoped views of the arm population and write operations for per-session registry updates. Each session receives only the subset of commands in its capability manifest.
Data splits across the two benchmarks. For GAIA2, we evaluate generalization under distribution shift by partitioning the benchmark such that train, development, and test sets contain tasks from disjoint Universes (personas).
Table 3: Data splits across the two benchmarks. For GAIA2, we evaluate generalization under distribution shift by partitioning the benchmark such that train, development, and test sets contain tasks from disjoint Universes (personas).
Robustness to optimization stochasticity. Pass@1 on three held-out GAIA2 universes for harnesses obtained from two independent optimization runs of AutoSaddler, its difficulty-based fixed curricula, and ActiveSaddler.
Table 4: Robustness to optimization stochasticity. Pass@1 on three held-out GAIA2 universes for harnesses obtained from two independent optimization runs of AutoSaddler, its difficulty-based fixed curricula, and ActiveSaddler.
Comparison with alternative curriculum strategies. Pass@1 on three held-out GAIA2 universes when replacing either the Arm Prioritizer or Exploration Controller in ActiveSaddler with an alternative strategy while keeping the remaining curriculum components unchanged.
Table 5: Comparison with alternative curriculum strategies. Pass@1 on three held-out GAIA2 universes when replacing either the Arm Prioritizer or Exploration Controller in ActiveSaddler with an alternative strategy while keeping the remaining curriculum components unchanged.
Applying ActiveSaddler to GEPA. Pass@1 on three held-out GAIA2 universes for the default agent, GEPA, and GEPA augmented with the ActiveSaddler curriculum.
Table 6: Applying ActiveSaddler to GEPA. Pass@1 on three held-out GAIA2 universes for the default agent, GEPA, and GEPA augmented with the ActiveSaddler curriculum.
Optimizer-side cost per generated patch. Runtime, monetary cost, LLM calls, and token usage are averaged per generated patch.
Table 7: Optimizer-side cost per generated patch. Runtime, monetary cost, LLM calls, and token usage are averaged per generated patch.
Average task-agent evaluation cost per rollout.
Table 8: Average task-agent evaluation cost per rollout.
Failure patterns induced by ActiveSaddler on GAIA2
Table 9: Failure patterns induced by ActiveSaddler on GAIA2
Failure patterns induced by ActiveSaddler on Terminal-Bench 2.0
Table 10: Failure patterns induced by ActiveSaddler on Terminal-Bench 2.0
End-to-end optimization efficiency. Best-so-far development accuracy against cumulative monetary cost, LLM calls, input tokens, and output tokens on GAIA2 (top) and Terminal-Bench 2.0 (bottom).
Figure 5: End-to-end optimization efficiency. Best-so-far development accuracy against cumulative monetary cost, LLM calls, input tokens, and output tokens on GAIA2 (top) and Terminal-Bench 2.0 (bottom).
Failure hit rate by arm definition. Fraction of optimization iterations whose sampled mini-batch contains at least one scenario with an unresolved failure.
Figure 8: Failure hit rate by arm definition. Fraction of optimization iterations whose sampled mini-batch contains at least one scenario with an unresolved failure.
Evolution of pull-probability mass across failure-pattern arms. The ten arms with the highest average pull probability over the run are shown individually, while the remaining 25 arms are aggregated into Other arms.
Figure 11: Evolution of pull-probability mass across failure-pattern arms. The ten arms with the highest average pull probability over the run are shown individually, while the remaining 25 arms are aggregated into Other arms.
Pull-probability trajectories for all failure-pattern arms. Rows show all 35 discovered arms ordered by instantiation time, and columns correspond to the 34 iterations in which arm selection is performed.
Figure 12: Pull-probability trajectories for all failure-pattern arms. Rows show all 35 discovered arms ordered by instantiation time, and columns correspond to the 34 iterations in which arm selection is performed.
Dependence of exploration decisions on unresolved weaknesses. Ratio of the average number of unresolved arms at PULL to DRAW decisions. Values above one indicate more unresolved weaknesses at PULL, while values near one indicate little difference between the two decision types.
Figure 13: Dependence of exploration decisions on unresolved weaknesses. Ratio of the average number of unresolved arms at PULL to DRAW decisions. Values above one indicate more unresolved weaknesses at PULL, while values near one indicate little difference between the two decision types.
查看结构化数据
任务指标本文基线提升
GAIA2 测试集（3个未见Universe，300场景，OOD划分） Pass@1（3次执行均值±标准差） ActiveSaddler 59.8 ± 1.0 AutoSaddler（同优化器、随机固定顺序）55.4 ± 1.2；GEPA 54.2 ± 2.2；Meta-Harness 54.2 ± 1.2；默认智能体 53.6 ± 1.1 +4.4 pp（对同优化器固定顺序），+5.6 pp（对默认智能体）
Terminal-Bench 2.0 测试集（40任务） Pass@1（3次执行均值±标准差） ActiveSaddler 80.0 ± 2.5 AutoSaddler 72.5 ± 0.0；场景精度序 73.3 ± 1.4；Terminus-KIRA（手动）69.2 ± 3.8；Terminus 2（手动）64.2 ± 2.9 +7.5 pp（对同优化器固定顺序），较最强手动harness +10.8 pp
臂表示消融（两基准） 测试 Pass@1 失败模式臂：GAIA2 59.8、TB2 80.0 类别臂：GAIA2 56.8、TB2 69.2；场景臂：GAIA2 56.2、TB2 73.3 GAIA2 +3.0/+3.6 pp，TB2 +10.8/+6.7 pp，失败模式粒度是收益主要来源之一
跨优化器泛化（Appendix E） GAIA2测试 Pass@1 GEPA + ActiveSaddler 57.2 ± 0.4 GEPA 54.2 ± 2.2 +3.0 pp，证明课程层可插拔、不限于AutoSaddler
端到端成本效率（Figure 5，含优化器+任务agent开销） 达到指定dev精度所需累计成本 GAIA2：58.5%@${'$'}298、61.5%@${'$'}730、63.1%@${'$'}1,178；TB2：78.9%@${'$'}128、84.2%@${'$'}181 GAIA2：AutoSaddler需${'$'}1,360才到58.5%、类别序${'$'}673；场景序花${'$'}1,698仅56.9%。TB2：AutoSaddler需${'$'}220、场景序需${'$'}363才到78.9%；类别序花${'$'}548仅73.7% 同等精度成本约为基线的1/2至1/5，抵消了课程层每补丁约45%的优化器开销（${'$'}7.87→${'$'}11.44）

局限与改进

作者在附录L明确承认：全部实验在基准环境中用公开/合成数据完成，不针对生产部署；方法不含独立安全过滤器，也没有检测reward hacking或specification gaming的机制，任务级提升不代表优化后harness安全合规；未评估命令执行、越权、敏感文件访问等安全风险，也未触及真实trace中的隐私问题。我自己的观察补充如下：(1)单模型依赖——任务智能体与优化器都用gpt-5.5，课程层三个组件本身依赖强模型推理（优化侧xhigh reasoning effort），换弱模型时LLM评分与探索决策质量如何保持未验证；(2)规模有限——TB2总共89任务（测试仅40），GAIA2训练+dev只有140场景，TB2上AutoSaddler报72.5±0.0的三次执行零方差略显异常，结论外推需谨慎；(3)每轮≤3场景、K=10倍全集的预算是为对齐AutoSaddler协议，更大batch或更长预算下课程收益是否保持未知；(4)LLM臂评分是主观cardinal分数，单次会话即定分，无校准或评分者间一致性检验；(5)固定课程基线只实现了按精度排序这一族，未与更多课程策略（按失败多样性、按Universe覆盖等）比较；(6)课程层使优化器侧成本上升约45%，在极小预算下可能得不偿失，论文未给出盈亏平衡点。

独立分析的弱点

独立分析几点弱点并给出改进方向。(1)LLM评分噪声：$\phi$由单次arm-scoring会话产出，四轴判断主观且不可复现，一次误判（如把间歇性通过误判为已修复）会连续多轮挤占预算；可改用多个独立会话打分取中位数，或给分数附置信度并在低置信时强制探索性重访。(2)失败模式提取无质量度量：symptom-extract/normalize全靠prompt约束（勿过合并/勿过拆分），没有自动指标衡量模式原子性，错误归并会让一个臂的评分被异质弱点污染；可引入embedding聚类辅助校验或对臂做定期分裂检测。(3)探索决策二值且无预算意识：控制器不感知剩余预算与各未解决臂的期望修复成本，后期可能过度PULL低产臂；可把期望剩余价值/成本比纳入决策，或在临近预算耗尽时切换为贪心。(4)回归再发现规则保守：只有“全部场景看过且每臂访问过”后才重放旧成功场景，长优化中静默回归可能潜伏很久；可按harness补丁的影响面（blast radius）定向重放高风险场景。(5)成本：课程层三个LLM会话（extract/score/explore）推高优化器开销约45%，小预算团队难承受；可用小模型跑score/explore、仅extract用大模型。(6)对优化器的接口假设：$F_t$依赖底层优化器暴露的失败与诊断证据，嫁接到不提供诊断的搜索式优化器（进化/贝叶斯）需重新设计证据接口。

未来方向

作者明确提出“自动化课程学习是harness优化的新关键维度”，并留出若干延伸。(1)在线/测试时结合：把课程层移植到Continual Harness、TTHE等在线演化设定，在部署数据流上实时决定重访与探索。(2)与权重共演化：CoEvolve、Co-Harness等联合优化harness与模型权重时，弱点定义与修复手段都变了，失败模式臂如何跨越“harness补丁”与“微调数据”两种修复通道值得研究。(3)服务搜索式AHO：HARBOR、HGM等进化/贝叶斯搜索目前也用固定评估批次，课程层原则上可直接嫁接。(4)跨任务弱点记忆：GAIA2发现的35个模式（如“trace-less超时”“表面smoke测试即定稿”）与TB2的15个模式有明显跨任务共性，构建可迁移复用的失败模式库是自然下一步。(5)理论与校准：把LLM评分视为学习进展的先验估计，分析这种LLM-in-the-loop bandit的regret与样本效率，并校准四轴分数。(6)多目标课程：把安全回归、成本约束、specification gaming风险作为独立约束或臂纳入课程决策，弥补作者承认的安全盲区。(7)主动场景合成：不只从$D_{train}$中选，还参考CoEvolve式生成针对高价值失败模式的新训练场景，把课程从“选择”扩展到“创造”。

复现评估

开源与材料：代码将发布于https://aka.ms/ActiveSaddler-website，附录材料完整——Algorithm 1/2伪代码、数据划分（GAIA2按Universe OOD划分：训练75+开发65、测试为Universe 21/22/27共300场景；TB2随机30/19/40）、超参（$K=10\times(|D_{train}|+|D_{dev}|)$、$\tau_{sel}=0.15$、每轮≤3场景、gpt-5.5任务侧medium/优化侧xhigh）、pattern CLI命令（Table 2）、三个组件完整prompt（Figures 14-16）与失败模式清单（Tables 9-10），匿名仓库另含SKILL.md；两基准均公开。算力成本：无需训练GPU但需gpt-5.5 API，端到端约${'$'}1,178-${'$'}1,838（GAIA2）、${'$'}181-${'$'}548（TB2），每补丁优化器900-1600秒。难度：方法本质是LLM会话编排、无需炼丹，但工程集成较重（Copilot SDK、EvoDAG、CLI权限），LLM非确定性使两次运行差1.8pp，复现相同数字较难，复现定性结论属中等难度。

论文图表

Figure 1: Optimization efficiency across benchmarks. Best-so-far development accuracy versus cumulative task-agent rollouts for ActiveSaddler and baseline methods.

两个子图分别画GAIA2与TB2上最佳开发集精度随累计任务rollout数（GAIA2至1400、TB2至490）的曲线；ActiveSaddler曲线全程位于GEPA、Meta-Harness、AutoSaddler及其固定顺序变体之上，GAIA2最高约63%、TB2最高约84%。

这是论文效率主张的核心证据：在完全相同的rollout预算下，ActiveSaddler不仅终点更高且全程领先，直观展示课程自适应对昂贵rollout的利用率提升。

Figure 3: Effect of adaptive arm scoring. Adaptive scoring dynamically reallocates optimization priority and yields more effective repair of observed failures.

(a)GAIA2上各臂PULL概率质量随迭代持续漂移的热力图（非固定子集）；(b)失败→成功转化率柱状图：ActiveSaddler在GAIA2达50.0%、TB2达75.0%，高于w/o Arm Prioritizer（30.0%/36.4%）与AutoSaddler（21.7%/33.3%）。

直接回答“自适应打分为什么有用”：预算确实随证据重新分配，且观察到的失败被更有效地修复。

Figure 14: Failure Pattern Extraction Prompt: the agent independently abstracts failed execution evidence into candidate symptoms and then normalizes them against the existing arm set by linking to existing atomic patterns, composing multiple patterns when needed, or instantiating new failure-pattern arms.

失败模式提取的完整prompt：强制先对每个失败独立做symptom-extract（不参考现有臂以避免锚定偏差），再用symptom-normalize做三路归一化（同/组合/新），通过pattern register与pattern tag写注册表，并强调根因只作证据、模式须原子化。

臂的质量决定整个课程的上限，这份prompt揭示了防止过合并/过拆分的具体工程手段，对复现至关重要。

Figure 16: Unseen Scenario Exploration Prompt: the agent decides whether to continue optimizing a known failure-pattern arm or explore scenarios that may reveal new failure patterns by weighing the value of unresolved known targets against the remaining value of further exploration.

探索控制器的完整prompt：PULL（修已知模式）与DRAW（探未见场景）的二元决策；给出两个思考透镜（是否有明显值得修的已知模式、失败面覆盖完整度），明确禁止仅凭|P_t|、|U_t|等单一统计量机械决策，要求用pattern decide记录带理由的决策。

展示LLM探索决策如何避免退化为公式化规则，是理解第三个组件设计与Appendix D中UCB-AIR对照实验的基础。

相似论文推荐

AutoSaddler：基于智能体执行轨迹的持久化自动Harness优化框架 🎯 83.0%
Continual Harness：面向自我提升基础智能体的在线自适应框架 🎯 73.7%
ParetoSaddler：帕累托引导的智能体框架质量与成本联合优化 🎯 71.7%
你将构建的最后一个 Harness：自动化 Agent 工程的双层演化框架 🎯 71.0%
Task-CoEvolve：基于自适应验证任务选择的高效Harness优化 🎯 71.0%