← 返回 2026-05-01

最后一份人类撰写的论文:面向智能体的原生研究产物 The Last Human-Written Paper: Agent-Native Research Artifacts

Jiachen Liu, Jiaxin Pei, Jintao Huang, Chenglei Si, Ao Qu, Xiangru Tang, Runyu Lu, Lichang Chen, Xiaoyan Bai, Haizhong Zheng, Carl Chen, Zhiyang Chen, Haojie Ye, Yujuan Fu, Zexue He, Zijian Jin, Zhenyu Zhang, Shangquan Sun, Maestro Harmon, John Dianzhuo Wang, Jianqiao Zeng, Jiachen Sun, Mingyuan Wu, Baoyu Zhou, Chenyu You, Shijian Lu, Yiming Qiu, Fan Lai, Yuan Yuan, Yao Li, Junyuan Hong, Ruihao Zhu, Beidi Chen, Alex Pentland, Ang Chen, Mosharaf Chowdhury, Zechen Zhang 📅 2026-04-29 👍 24 2026-07-13 08:36
AI科研智能体 LLM智能体评估 协议设计 可复现性 机器可读科学 知识表示
arXiv HuggingFace PDF

提出ARA协议:把论文从叙事文档重构为四层机器可执行知识包,提升智能体复现能力。

前置知识

叙事论文的信息损失(Storytelling Tax)

指科学出版在将分支探索过程压缩成线性故事时,系统性丢弃失败实验、被拒假设和试错经验的现象。科研本身是分支、回溯、积累失败知识再收敛的树状过程,但论文只保留最终的成功路径,造成可复现工程师无法继承试错信息。

本文的核心问题定义之一,读者需要理解'论文不是研究产物本身,而是其有损编译视图'这一前提,才能明白ARA为何要从'文档'转向'知识包'的设计哲学。

工程化税(Engineering Tax)

指评审通过的论文(prose)与可执行代码(repo)之间存在的隐性知识鸿沟——超参数调优技巧、隐式假设、环境细节等Polanyi意义上的'默会知识'(tacit knowledge),它们只在导师/同事的口耳相传中存在,PaperBench分析显示45.4%的复现要求在PDF中完全缺失,代码开发类低至37.3%。

这是本文要解决的第二个结构性问题,理解'机器可读'必须达到何种精度(执行而不只是令人信服)才能体会ARA四层分离(认知/物理/探索/证据)的必要性。

AI原生的科研工作流(AI-native research)

指研究者与通用编程智能体(如Claude Code)通过长会话协作完成假设构思、文献调研、写代码调实验、分析结果与撰写论文的完整生命周期的研究范式。其结构性后果是:整条研究轨迹首次以'born-digital, born-textual'的机器可读文本形式存在,这为全程自动捕获研究过程提供了历史首次的可行时机。

这是Live Research Manager的设计基础——会话本身就是结构化知识的免费原材料,只需一个静默运行的后台过程把散乱的对话'结晶'为合规的ARA。

探索图(Exploration Graph)/有向无环研究图

一种把科研决策轨迹存储为嵌套YAML树的数据结构,节点类型分为五类(question、decision、experiment、dead_end、pivot),通过also_depends_on字段编码收敛点。本质是'科研的git log',允许智能体直接遍历分支,并保留每条死路(hypothesis、failure mode、lesson)以供后续研究继承。

ARA中与叙事论文差异最大的层(Trace层),理解它如何把被舍弃的负结果变为可查询的结构化监督信号,是理解本文'失败知识可机器化'主张的关键。

证据层隔离与机器验证的可复现性(Seal protocol)

论文提出的ARA Seal是一种三档机器可验证凭证:Level 1秒级确定性检查结构(schema与跨层引用);Level 2由Rigor Auditor在六维锚定rubric上对论证严谨性打1–5分;Level 3在沙盒内执行定向复现并把预期数值掩码为[X]%防止智能体抄答案。证据层与实验逻辑层分离,使得训练环境可控、伪造可被审计。

理解Seal才能明白ARA如何把'代码可获得(reproducibility-as-norm)'升级为'结构化本身可机器验证(reproducibility-as-artifact-property)',也是理解作者如何抵御对'人类评审带宽有限'担忧的关键。

研究动机

现有学术出版体系把分支探索过程压缩为线性叙事,产生两种结构性的'税':(1)Storytelling Tax——论文系统性丢弃失败实验和被弃假设,作者对METR eval-analysis-public数据集(24,008次智能体运行、21个前沿模型)的分析显示,失败运行占RE-Bench总成本的90.2%、token数的59.2%,中位失败-成功token比达113倍,后续智能体必须独立重新发现每一个死胡同;(2)Engineering Tax——论文为说服评审者而写,代码库为执行而写,中间存在大量隐性知识,作者对PaperBench 8,921条专家标注复现要求中只有45.4%在源PDF中完整说明(代码开发类仅37.3%),缺失超参数一项就占全部缺口的26.2%。这种结构在过去可被容忍,因为读者始终是带宽受限的人类,而当AI智能体成为论文的主要消费者时,这两种税直接转化为智能体的失败模式,LLM在SWE-bench上实现新贡献率低于40%、EXP-Bench端到端实验成功率仅0.5%。

本文的目标是本文的具体目标是提出并落地一种Agent-Native Research Artifact(ARA)协议,把研究产物从人类阅读的叙事文档重构为智能体可执行的机器可读知识包,并构建配套的三套支撑机制(Live Research Manager、ARA Compiler、ARA-Native Review System),使得一份产出的'论文'等价于一份能被前沿智能体从零开始无人工干预复现的活体知识对象。

与已有工作不同的是，本文的独特切入角度是把科研知识的'组织形式'(organization)而非'内容'(content)当作主问题。既有工作(FAIR数据原则、RO-Crate归档包、Nanopublications原子声明、AGENTS.md代码文档、MLflow/W&B实验跟踪、OpenAlex/S2ORC语料库)覆盖了单个维度但彼此不互通,Table 5显示即使同时使用PDF+GitHub+Tracker,五个维度中至多结构性覆盖两个。本文把认知层、物理层、探索层、证据层四者绑定在同一个文件本体下并通过forensic bindings使跨层追溯成为机器可执行的事,并在三档Seal验证、(Human+AI)^2研究网络、基于Git式diff的论文演化上提供端到端的工程实现。

核心方法

ARA的整体思路可以一句话概括:把研究的对象从'文档'提升为'可被智能体'操作'的文件系统本体',论文退化为其上的编译视图。技术路线分四层并行:认知层(/logic)负责结构化科学叙事——problem.md、solution/、claims.md(每条带Falsification criteria与Proof指针)、experiments.md(验证计划)、related_work.md(机器可执行的依赖图,而非被动引用);物理层(/src)按贡献类型自适应为kernel mode(只导出核心模块带类型化I/O签名,环境脚手架让智能体按需重生)或repository mode(保留完整仓库并附index.md映射);探索层(/trace)以YAML DAG保存完整决策、死端、pivot三类失败节点;证据层(/evidence)只存原始数值与日志(不混入逻辑),并以三层Seal(结构/论证/执行)担保整包可被机器验证。配套的三套机制进一步完成'出生—编译—评审'全生命周期:Live Research Manager在每次研究会话边界静默运行Context Harvester→Event Router→Maturity Tracker三阶段管道,按七类事件类型(decision、experiment、dead_end、pivot、claim、heuristic、observation)把对话内容结晶进对应层;ARA Compiler以单一智能体skill引导自顶向下编译(语义解构→认知映射→物理落地→探索图抽取),由Level 1在环内feedback驱动2–3轮迭代;ARA-Native Review按CI/CD思路分三阶段(分钟级概念验证、小时-天级实证验证、天-周级人类判断),把机械校验交给机器、把'意义/新颖性/品味'留给人。

核心创新点是显式地把'研究轨迹'从文本文档里抽离出来作为一个独立的、机器可查询的层,并用跨层forensic bindings把它和代码、证据、声明绑定起来,使得'循一个声明追溯到一串代码行、再顺着代码回溯到原始假设'成为单步操作。和已有方法的本质区别在于:既不是把论文转成Markdown(只解包内容不解包结构),也不是把代码贴上自动生成的文档(自动化的是注释而非知识),更不是把失败日志dump成JSON(只是丢弃变成了无序堆积)。ARA把研究当作'活的、带版本控制的、拥有死亡轨迹的实体'——一个'科研的git',并且首次让Storytelling Tax成为可被机器审计的违规(因为死端会被自动结构化保留,任何'删光失败只留成功'的论文会在Seal Level 2里被Rigor Auditor检测为'orphan experiment'或'rebutted-branch leak')。

方法步骤详情

完整方法流程分为两条并行路径。第一条是出生即合规路径:研究者在AI-native会话中开展正常研究,Live Research Manager按P1(静默、与框架无关,任何编程智能体加载skill规范即可)、P2(忠实认识论血缘,每条事件标注user/ai-suggested/ai-executed/user-revised四档provenance,ai-suggested必须经研究者确认才升级)、P3(完整分支探索保留,版本控制使每个milestone生成可导航快照)三条原则工作——在会话结束时进入三阶段:Context Harvester扫描整段会话记录(对话、工具输出、实验结果、代码diff)抽取研究相关动作;Event Router按Table 1的七类事件类型(classification)和provenance标签写到对应层(trace事件入/trace/,claim/heuristic入/logic/,无法分类的入/staging/);Maturity Tracker审查staging area,把已积累足够证据的observation提升为正式claim/heuristic/config,pivot发生时跨层propagation修订相关条目但保留原始理由,整体通过双时间尺度(连续追加trace事件,周期性milestone结晶)保证'原始→凝固→结构化'的渐进成熟,在新会话开始时读取artifact形成结构化briefing闭合回路。第二条是历史回溯路径:对遗留PDF+repo使用ARA Compiler,采用universal input→canonical output原则接受任意输入组合,以Semantic Deconstruction先剥离叙事框架只保留事实稠密表述,Cognitive Mapping填充/logic并对每个claim加上proof pointer,Physical Grounding生成/src(stubs在repo可用时替换为真实实现并做code-paper reconciliation回写heuristic),Exploration Graph Extraction重建含dead-end节点的嵌套DAG;过程采用generate→validate→fix循环,Seal Level 1(schema一致性、跨层引用解析、必填字段完整)作为环内诊断驱动定向修复,典型2–3轮收敛;遇到评估rubric等外部专家标注时按source-aware routing分配到最契合的层,并在有多份同领域已编译ARA时执行collective inference召回共有启发式但显式打上collective_inference标签以区分陈述知识与推断知识。

技术新颖性

技术新颖性体现在五个层面。其一,Schema层首次把研究产物的四个本质不同但互补的知识维度(推理/代码/过程/证据)显式物化为文件系统路径,通过YAML frontmatter + 500 token内triageable的PAPER.md manifest让智能体按需加载(progressive disclosure),避开单一文档扁平化的不可逆损耗。其二,Process保留层把'叙事舍弃的死端'转译为带hypothesis/failure mode/lesson三字段的dead_end节点,使失败知识首次成为机器可查询监督信号,这把'负结果期刊'和'注册报告'这些过去因额外负担而失败的思路(negative-result journals/registered reports)折叠进了零成本的流水线副产品。其三,验证机制把可复现性从'代码可获得'升级为'凭证可验证',Seal Level 1/2/3分别对应秒级确定性、分钟级rubric-anchored、小时级沙盒执行,并且通过在证据层与实验逻辑层之间引入物理隔离(支持训练环境/防伪抄答案),让'reproducibility as artifact property'具备工程意义。其四,Review流水线借鉴CI/CD的Stage 1/2/3分阶段gate,Level 2的Rigor Auditor在六维锚定rubric上对每条claim做evidence relevance/falsifiability quality/methodological rigor等可审计打分,把人类评审带宽从机械校验解放到significance/novelty/taste。其五,生态层((Human+AI)^2网络)首次让论文可fork/diff/merge,paper artifact具备Git式lineage、parent declaration与structured diff,推动科研贡献在artifact粒度而非句子粒度上复合(compound)。

The ARA directory structure. Each file's role is annotated inline; layer badges mark the four top-level divisions.
Figure 4: The ARA directory structure. Each file's role is annotated inline; layer badges mark the four top-level divisions.
Cross-layer structure of a real ARA. Claims in /logic link to code in /src and evidence in /evidence via forensic bindings. The Exploration Graph (bottom center) captures the research DAG with dead-end nodes (marked ×) that preserve failure modes and lessons.
Figure 5: Cross-layer structure of a real ARA. Claims in /logic link to code in /src and evidence in /evidence via forensic bindings. The Exploration Graph (bottom center) captures the research DAG with dead-end nodes (marked ×) that preserve failure modes and lessons.
The Live Research Manager operates at session boundaries: a three-stage pipeline (Context Harvester → Event Router → Maturity Tracker) distills each researcher-agent conversation into typed events that accumulate across ARA layers over time.
Figure 6: The Live Research Manager operates at session boundaries: a three-stage pipeline (Context Harvester → Event Router → Maturity Tracker) distills each researcher-agent conversation into typed events that accumulate across ARA layers over time.
The ARA Compiler accepts any combination of research sources and guides a coding agent through four stages of top-down artifact compilation, iterating 2–3× with in-loop ARA Seal Level 1 validation until the output conforms to the protocol.
Figure 7: The ARA Compiler accepts any combination of research sources and guides a coding agent through four stages of top-down artifact compilation, iterating 2–3× with in-loop ARA Seal Level 1 validation until the output conforms to the protocol.
The ARA Seal is a three-level verification credential. Each level tests a progressively stronger property of the artifact, escalating in cost and breadth: structural integrity (seconds, deterministic), argumentative rigor (minutes, rubric-anchored agent), and execution reproducibility (hours to days, sandboxed coding agent).
Figure 8: The ARA Seal is a three-level verification credential. Each level tests a progressively stronger property of the artifact, escalating in cost and breadth: structural integrity (seconds, deterministic), argumentative rigor (minutes, rubric-anchored agent), and execution reproducibility (hours to days, sandboxed coding agent).
Three-stage ARA-native review pipeline. Stages 1–2 invoke the ARA Seal levels of Figure 8 to resolve mechanical and rigor issues before human reviewers engage, redirecting expert attention to novelty and significance.
Figure 9: Three-stage ARA-native review pipeline. Stages 1–2 invoke the ARA Seal levels of Figure 8 to resolve mechanical and rigor issues before human reviewers engage, redirecting expert attention to novelty and significance.
The (Human+AI)2 research network. Each researcher works through a research agent that interfaces with a shared ARA network via /submit, /retrieve, and /fork; agents may also collaborate directly.
Figure 10: The (Human+AI)2 research network. Each researcher works through a research agent that interfaces with a shared ARA network via /submit, /retrieve, and /fork; agents may also collaborate directly.

实验结果

核心发现按三层评估逐层呈现。理解层(Table 3)在450对(目标,格式,问题)三元组上ARA以93.7% vs 72.4%的总体准确率击败PDF+GitHub基线(+21.3%),三个类别按隔离机制分别获得+A类(表层忠实性)+14.8%且token低12%(PAPER.md的layer index把线性扫描转为定向查找)、B类(超参/环境/预处理细节)+24.8%在相当token预算下(src/configs/把知识集中到单文件)、C类(死端/替代/教训,仅MALT轨迹可得)+65.7%(基线被迫放弃大部分问题,58K vs 139K token),跨类别机制解构清晰;token随问题深度在ARA上单调增长(显式61K / 分散96K / 隐式失败153K),在基线上几乎扁平(83–118K),证明ARA支持progressive disclosure。复现层(§7.3,Figure 11)在15篇PaperBench全文复现任务(150子任务、1,743条要求,易50/中49/难51)上ARA达难度加权成功率64.4% vs 57.4%基线(8胜5平2负),差距按难度单调扩大:+4.9%(易)/+5.6%(中)/+8.5%(难),与PDF欠规格化最严重的复杂度区间一一对应;最大单篇增益来自fre、mechanistic-understanding、pinn等多阶段流水管线论文(fre任务中ARA agent用PyTorch重写了原JAX实现,在1.8GB GPU上完成全部17个模型训练并通过所有中难子任务,而基线agent被困JAX环境,预算耗尽时仅完成3次训练)。扩展层(§7.4,Figure 12)在5个RE-Bench任务上ARA在所有5个任务上都更早产出有用的首个动作(例如rust_codecontests在t=9min读heuristic H12后立刻动手手写Rust库,而基线做六小时prompt-engineering到t=395min才想到同样的few_shots/缓存),但Sonnet 4.6下最终best score仅在3/5任务领先(rust_codecontests、nanogpt_chat_rl、fix_embedding),另两个(triton_cumsum、restricted_mlm)被基线反超——前者靠基线发明的int8输入压缩(trace未记录),后者靠基线把单一架构ConvMLMDilated深度跑满8小时,而ARA agent按trace忠实实现了所有heuristic命名的替代架构(ReLU-attention、MLPMixer等)却都被Sonnet 4.6的优化卡住;关键的公平性证据是同一对比在更弱的Sonnet 4.5上完全反转(ARA在两任务上分别0.27 vs 0.64 和 0.73 vs 1.03胜出),说明trace的价值随着'trace文档覆盖范围'与'智能体可自行发现范围'之间的差距变化。复审系统层(§7.5,Table 4)Rigor Auditor在5种注入×23 ARA的mutation benchmark上对三档高危注入(fabricated claim / rebutted-branch leak / over-claim)达到100%检出率,对missing falsification 91%,但对orphan experiment系统性漏检22%——作者识别出两类LLM-as-judge系统偏差(grade inflation: 17/23 ARA上均值被刚好抬到Accept阈值;finding-score decoupling: 22/23 正确flag但对应维度分数不下降),并据此提出'LLM应该只生成findings而由deterministic verdict合成总评'的修正。

Research event types and structured payloads.
Table 1: Research event types and structured payloads.
Benchmark characteristics. PaperBench supplies configuration depth via expert rubrics; RE-Bench supplies trajectory depth via MALT failure traces.
Table 2: Benchmark characteristics. PaperBench supplies configuration depth via expert rubrics; RE-Bench supplies trajectory depth via MALT failure traces.
Understanding evaluation: accuracy and per-question token usage across 450 paired outcomes. ARA wins at every category and every benchmark; the per-category mechanism is unpacked in Appendix E.4.
Table 3: Understanding evaluation: accuracy and per-question token usage across 450 paired outcomes. ARA wins at every category and every benchmark; the per-category mechanism is unpacked in Appendix E.4.
Rigor Auditor effectiveness on the mutation benchmark (23 ARAs × 5 injection types). The auditor catches all high-severity structural anomalies but exhibits a systematic blind spot on orphan experiments.
Table 4: Rigor Auditor effectiveness on the mutation benchmark (23 ARAs × 5 injection types). The auditor catches all high-severity structural anomalies but exhibits a systematic blind spot on orphan experiments.
Dimensional coverage of existing research artifacts. Each row is a requirement for agent-native research (§1). Existing tools cover at most two dimensions structurally; ARA covers all five with explicit cross-layer bindings.
Table 5: Dimensional coverage of existing research artifacts. Each row is a requirement for agent-native research (§1). Existing tools cover at most two dimensions structurally; ARA covers all five with explicit cross-layer bindings.
Reproduction information gap across 23 PaperBench papers.
Table 8: Reproduction information gap across 23 PaperBench papers.
Aggregate reproduction success rates across all 15 papers, stratified by difficulty. The ARA advantage widens monotonically with difficulty (+4.9% easy, +5.6% medium, +8.5% hard), tracking the tiers where reproduction depends most heavily on configuration content the PDF underspecifies.
Figure 11: Aggregate reproduction success rates across all 15 papers, stratified by difficulty. The ARA advantage widens monotonically with difficulty (+4.9% easy, +5.6% medium, +8.5% hard), tracking the tiers where reproduction depends most heavily on configuration content the PDF underspecifies.
Extension trajectories on five RE-Bench tasks under Claude Sonnet 4.6. One task per column: top row is score-vs-wall-clock-time, bottom row is score-vs-cumulative-cost; the y-axis is shared down each column. Faint markers are individual scoring attempts, solid lines are the best-so-far envelope, and stars mark the best attempt; dotted grey lines mark each task's RE-Bench reference.
Figure 12: Extension trajectories on five RE-Bench tasks under Claude Sonnet 4.6. One task per column: top row is score-vs-wall-clock-time, bottom row is score-vs-cumulative-cost; the y-axis is shared down each column. Faint markers are individual scoring attempts, solid lines are the best-so-far envelope, and stars mark the best attempt; dotted grey lines mark each task's RE-Bench reference.
Per-paper ARA − baseline delta (percentage points).
Figure 13: Per-paper ARA − baseline delta (percentage points).
查看结构化数据
任务指标本文基线提升
PaperBench Understanding (450 paired Q&A across 30 targets) 三元(yes/partial/no)judge评定的准确率 93.7% (Category A 95.6%, B 92.6%, C 81.4%) 72.4% (Category A 80.8%, B 67.8%, C 15.7%) — PDF+GitHub (15/23篇有repo) +21.3个百分点;token/question 114.0K vs 109.1K(基本持平,实际在表层题上ARA少12%)
PaperBench Reproduction (15 papers, 150 subtasks, 1,743 requirements, easy/medium/hard = 50/49/51) 难度加权(1:2:3)成功率 64.4% (Per-difficulty: 85.1% / 62.9% / 54.5%) 57.4% (Per-difficulty: 80.2% / 57.3% / 46.0%) — PDF + companion GitHub +7.0个百分点全局;按难度 +4.9% / +5.6% / +8.5%,win/tie/loss = 8/5/2
PaperBench 信息完备率审计(8,921条专家复现要求,对23篇ICML 2024论文的PDF做配对分类) 按10类信息缺口的占比分布 ARA封装后100% ARA通过Seal Level 1,Cat.A理解准确率95.6%意味着所有源信息都被检索到 45.4% 整体完整;代码开发类仅37.3%;最大单项缺失'缺失超参数'占26.2%,其次'描述模糊'21.9%和'仅给交叉引用'13.4% 通过把工程税显式化进物理层(/src/configs/)和逻辑层(heuristic),目标100%消除PDF系统性欠规格化
RE-Bench Extension (5 tasks: triton_cumsum, rust_codecontests, nanogpt_chat_rl, fix_embedding, restricted_mlm; 8h+ $50 cap/run; Sonnet 4.6) best score across invocations vs reference;wall-clock/API-cost轨迹 3/5 tasks 终胜(rust_codecontests / nanogpt_chat_rl / fix_embedding);5/5在首个有用动作的时间上都更早(例rust t=9min vs t=395min,triton t=11min 0.47分 vs t=37min 0分) Sonnet 4.6下2/5终胜(triton_cumsum依靠int8 kernel重设计;restricted_mlm靠单架构8小时深耕);Sonnet 4.5下ARA在两任务都领先(0.27 vs 0.64;0.73 vs 1.03) 效应符号依赖模型-带宽差距:trace优势随智能体独立发现能力的提升而衰减
Rigor Auditor mutation benchmark (23 ARAs × 5 注入类型=115个oracle) 按target entity匹配的检出率 fabricated claim 23/23 (100%), rebutted-branch leak 23/23 (100%), over-claim 23/23 (100%), missing falsification 21/23 (91%), orphan experiment 5/23 (22%), 整体95/115 (82.6%) 无对照基线(基准即oracle注入本身),但作者对artifact-blind paper-coded discrepancy LLM检测率<46%(Baumgärtner & Gurevych, 2026)做对比 结构化artifact后检出率从亚50%提升到82.6%整体,且高危类100%

局限与改进

作者在§10显式承认三类边界。其一,评估覆盖仅限机器学习——ARA四层结构对齐算法/架构/训练流程,是否推广到物理执行的实验科学或以形式证明为主的理论学科未经验证,Cognitive与Evidence层域无歧,Physical Layer与Exploration Graph在湿实验/纯理论语境下需大幅改造;此外人类标注benchmark由熟悉ARA格式与目标论文的标注员构建,在不熟悉或小众领域性能可能偏移。其二,fidelity天花板由源监督上限决定,Compiler只能忠实重建PDF已有的内容,当论文本身就省略了实验细节/环境规格/消融结果时任何提取方法都无法恢复;Live Research Manager可弥补但前提是全程有编程智能体在场,对于非AI-native工作流,Compiler产出的artifact仍继承PDF的省略。其三,生产部署前置条件未实现:作者承认Seal的对抗鲁棒性与隐私保证尚属启发性目标,当前系统缺乏沙盒执行、内容级异常检测、Exploration Graph粒度访问控制;另schema演化需要稳定的迁移路径(自动重写、长期校验器可用性、弃用策略),目前只在minor修订上验证了forward/backward兼容纪律。

独立分析的弱点

独立分析的弱点主要在以下三方面。第一,扩展层实验中Sonnet 4.6反超暴露了trace契约的'过拟合于中位智能体'风险——当智能体能力足够高时,死端heuristic变成行动清单而非参考意见,会抑制agent对trace外优秀解的创造性探索(Figure 12中restricted_mlm按trace穷举所有heuristic命名的替代架构但全部失败,基线单一架构深度挖掘反而赢),改进方向是为trace节点附加model-class provenance并在评测时按当前智能体类别动态打折。第二,Even-though文件本体结构清晰,L3 执行验证当前的'小型数据+少量epoch+t toy config'的方向性复现可能在高度依赖超参数组合的论文上产生误导,基线agent可能因toy规模改变行为而无法暴露原问题(PaperBench数据已显示易题天花板),改进方向是Seal Level 3应允许venue声明最小可接受复现强度(如'至少N条与原文同分布的随机种子')。第三,Rigor Auditor只在结构性artifact-anchored evaluation上达到82.6%总体检出且orphan experiment漏检78%,证明Level 2仍严重依赖LLM-as-judge路径,这给出一条低成本但高ROI的改进方向——把orphan detection(枚举每个experiment并校验Verifies target在claim列表中)从非确定性LLM流程移到Level 1的确定性结构检查,而非继续容忍LLM judge's grade inflation(17/23 ARA整体均值被抬到Accept门槛,即便findings正确flag也不下调分数)。此外fabrication风险在25次盲审运行中仍有1次ARA + 2次基线发生,说明evidence-layer隔离的'expected masking'虽然在简单数值上有效,但对分布/曲线类结果尚不充分,应加入统计一致性测试。

未来方向

作者明确将未来工作按三时段分层。近时段,最迫切的缺口是artifact durability——像代码仓库会因依赖腐化而失效,ARA也面临相同问题但目前无人维护;自然延伸是lineage机制使每个ARA声明parent artifacts并以structured diff表达贡献,既降低构造成本(作者只specify delta)又降低验证成本(评审与agent只复查新增),再叠加self-maintaining ecosystem(消费ARA的agent检测并修复陈旧依赖,把每次消费变成一次维护)。中时段,跨artifact对齐会自然形成查询式科学知识图,把文献综述转译为子图查询,让评审agent验证reported baselines确实匹配被引ARA所记录,并暴露'在某处被声明成功而在另一处被记录失败'的方法轨迹冲突;在此之上,review进化为非一次性accept,而是基于replications与counter-evidence上下浮动的claim-confidence surface,从而把人类专家注意力全部分配到仅有人才能回答的novelty/significance/taste问题。远时段,跨学科集体记忆是开放问题——Cognitive与Evidence两层域无关,但Physical Layer与Exploration Graph以可迭代的计算机实验为前提,在湿实验科学/纯数学/形式证明领域需要实质性改造,如果跨学科适配成功ARA将成为天然的知识迁移基底,使一个领域的失败在另一个领域通过图遍历立即可用。基于本文成果还可以衍生的工作包括:把build/reference/Seal三个产物接入arXiv以触发现实出版流程、用ARA做RL训练环境(任务在logic/、奖励在evidence/、监督在trace/)、以及把/trace做成可视化'科研git blame'以提升同行评审的解释性。

复现评估

复现性评估整体偏正面。代码与skill资源完全开源:Live Research Manager、ARA Compiler、Rigor Auditor都以Anthropic agent skill规范提供,仓库github.com/AmberLJC/Agent-Native-Research-Artifact。数据来源为PaperBench(23篇ICML 2024论文、8,921条专家标注)和RE-Bench(METR MALT 24,008条运行记录),均为公开benchmark,无需独立获取。算力要求温和:理解层实验基于Claude Sonnet 4.6子agent与Opus 4.6 judge,每目标15题共450三元组,基本无GPU需求;复现层每篇14–20M token预算按缓存10%折扣;扩展层每任务8h wall clock + $50 API spend cap;评审计23 ARA × 5 injection的mutation benchmark量级有限。算力瓶颈主要在复现层——完整跑完15篇需要长时段(数十小时级)的Claude编程智能体调用,小团队可参考Appendix F/G复用harness但要承担相应API成本。复现难度主要为(i)Seal Level 3需要可控算力+完整环境(原作者提供附录级harness规范与score.sh模板)、(ii)完整结论链需要作者透明说明MALT轨迹的beat-reference fairness filter实现细节、Grok/Claude之间的裁决rubric与人类原始标注对比——这些细节作者确实分散放在App. B/C/G/H里,但跨附录拼读成本较高。整体而言,理解层与评审计对小团队友好,复现层与扩展层需要中等规模的智能体API budget,但所有模块都已released为可独立调用skill。

论文图表

Publishing compiles a rich research object into a lossy narrative (left); ARA preserves the original as a high-fidelity, agent-executable knowledge package (right).
Figure 1: Publishing compiles a rich research object into a lossy narrative (left); ARA preserves the original as a high-fidelity, agent-executable knowledge package (right).

左半边用一个标'Research Object'的圆形被箭头压平成标'Paper'的扁长方形,暗示研究对象被压缩丢信息;右半边把同样的Research Object直接输出为标'Agent-Native Research Artifact'的圆形,并伴随展开的exploration tree与cross-layer bindings,展示零损耗结构化输出。

整篇论文的核心视觉宣言——把论文定位为'科研对象的编译视图'而非产物本身,理解这张图才能抓住ARA的哲学起点。

The Storytelling Tax: research proceeds as a branching tree with dead ends (left), but publication compiles it into a linear narrative (right), discarding all failure knowledge.
Figure 2: The Storytelling Tax: research proceeds as a branching tree with dead ends (left), but publication compiles it into a linear narrative (right), discarding all failure knowledge.

左画一棵含多个dead-end叶子的研究决策树;右把它压成一条带X号dead-end但被擦除的直线,标注discarded loss。

把Storytelling Tax这一抽象成本可视化,让读者直观看到叙事压缩舍弃的失败知识,这是Live Research Manager必须捕获的部分。

The reproduction information gap across 8,921 PaperBench requirements. (a) PDFs systematically under-specify code development tasks. (b) The three largest gap types are precisely the categories ARA's structured layers address.
Figure 3: The reproduction information gap across 8,921 PaperBench requirements. (a) PDFs systematically under-specify code development tasks. (b) The three largest gap types are precisely the categories ARA's structured layers address.

(a)三组柱状图按'Code Development(3,942)/Code Execution(4,355)/Result Analysis(624)/Overall(8,921)'分组展示'Sufficient/Partial/Absent'三档比例,代码开发类sufficient仅37.3%、overall 45.4%;(b)饼图列出10类gap type分布,缺失超参数26.2%、描述模糊21.9%、仅交叉引用13.4%排前三。

这是Engineering Tax的实证锚——为后续评估理解层93.7%与复现层64.4%提供必要的下限因果链,也是Seal Level 1/2存在的动机。

Table 9: Distribution of information gap types across 8,921 requirements.

列出10类信息gap type的细粒度分布:Missing hyperparameter 26.2%、Vague description 21.9%、Cross-reference only 13.4%、Missing codebase detail 10.9%、Missing baseline detail 10.8%、Missing URL 5.5%、Figure only 5.5%、Missing detail 5.1%、Implicit assumption 4.1%、Ambiguous specification 1.5%,前两类gap合计近半数,直接对应ARA要重点解决的heuristic与config层。

为ARA的每一层应该捕获什么类型的隐性知识提供设计清单,使方法层与动机层之间的因果链可追溯。

Table 10: Cost of below-reference exploration across 24,008 agent runs.

逐run分解失败成本:24,008个运行中有90.2%的美元成本(59.2% token)花在未能达成reference的探索上;失败-成功token比中位数高达113×,意味着每个失败运行平均消耗113倍成功运行的token,直观展示Storytelling Tax的算力代价。

为扩展层实验(§7.4)提供经济学锚——保留失败轨迹的ROI直接来自'避免每次新agent重做所有死端'。

Table 11: Per-paper reproduction success rates (%) by difficulty level.

15篇×3难度(easy/medium/hard)的完整矩阵,如fre的medium列ARA显著领先(基线agent困在JAX环境最终只完成3次训练,ARA用PyTorch重新实现完成全部17模型),mechanistic-understanding/pinn同样领先于多阶段训练管线论文;唯一清晰基线胜场为self-expansion(ARA agent发生一次被盲judge捕获的fabrication),narrow ties见adaptive-pruning与rice(其companion repo部分弥补了PDF信息缺口)。

把复现层'8胜5平2负'的全局结论下沉到具体任务,识别哪些论文受ARA帮助最大,以及'fabrication率从2降至1'等小但是关键的反例。

Table 6: Taxonomy of reproduction-critical information in PaperBench rubrics.

10类复现关键信息的分类表(C1证据/数据、C2实验设置、C3超参/计算、C4代码细节…),按3,050条leaf requirement统计频率,提供从原benchmark需求到ARA对应层的映射参考。

作为整个ARA schema设计的方法学锚——每层捕获什么、应当如何结构化字段都来源于这份对PaperBench专家rubric的解构。