wuyang
793 lines
38 KiB
HTML
793 lines
38 KiB
HTML
<!doctype html>
|
||
<html lang="zh-CN">
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||
<meta name="theme-color" content="#f6f7f5">
|
||
<meta
|
||
name="description"
|
||
content="37 篇全文证据下，Agent 如何证明任务完成，验证器怎样误判，以及失败后如何恢复。"
|
||
>
|
||
<title>Agent 到底凭什么说“完成了” · K1412 Research</title>
|
||
<link
|
||
rel="icon"
|
||
href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' fill='%23131715'/%3E%3Cpath d='M8 8h16v3H11v10h13v3H8z' fill='%23fff'/%3E%3Ccircle cx='21' cy='16' r='3' fill='%23008779'/%3E%3C/svg%3E"
|
||
>
|
||
<link rel="stylesheet" href="/assets/research.css?v=20260728-01">
|
||
<script src="/assets/evidence-data.js?v=20260728-01" defer></script>
|
||
<script src="/assets/research.js?v=20260728-01" defer></script>
|
||
</head>
|
||
<body>
|
||
<a class="skip-link" href="#main">跳到正文</a>
|
||
<div class="reading-progress" aria-hidden="true">
|
||
<span id="reading-progress"></span>
|
||
</div>
|
||
|
||
<header class="site-header">
|
||
<a class="brand" href="#overview" aria-label="回到研究文档开头">
|
||
<span class="brand-mark" aria-hidden="true">K</span>
|
||
<span>
|
||
<strong>K1412 Research</strong>
|
||
<small>Agent Completion · Note 01</small>
|
||
</span>
|
||
</a>
|
||
|
||
<nav class="primary-nav" aria-label="文档章节">
|
||
<a class="is-active" href="#overview">结论</a>
|
||
<a href="#capabilities">能力</a>
|
||
<a href="#model">模型</a>
|
||
<a href="#arguments">论证</a>
|
||
<a href="#progress">进展</a>
|
||
<a href="#evidence">证据</a>
|
||
<a href="#unknowns">未知</a>
|
||
</nav>
|
||
|
||
<div class="header-actions">
|
||
<a href="/evaluation-v1.html">旧评测方案</a>
|
||
<a
|
||
href="https://git.k1412.top/wuyang/agent"
|
||
target="_blank"
|
||
rel="noreferrer"
|
||
>源仓库</a>
|
||
</div>
|
||
</header>
|
||
|
||
<main id="main">
|
||
<section class="cover section-shell" id="overview" data-section>
|
||
<div class="cover-copy">
|
||
<p class="eyebrow">FULL-TEXT RESEARCH AUDIT · 2026.07.27</p>
|
||
<h1>Agent 到底凭什么说<br>“完成了”</h1>
|
||
<p class="lead">
|
||
完成不是 Agent 的一句话，也不是 benchmark 里的一个绿色勾。
|
||
它是关于外部世界的证据合同：目标效果发生了、禁止效果没有发生、
|
||
必要过程没有被绕过，证明不到的部分仍被诚实保留。
|
||
</p>
|
||
<div class="cover-actions">
|
||
<a class="primary-link" href="#conclusions">先读八条结论</a>
|
||
<a class="text-link" href="#evidence">直接核查证据</a>
|
||
</div>
|
||
</div>
|
||
|
||
<figure class="completion-figure" aria-labelledby="completion-figure-title">
|
||
<figcaption id="completion-figure-title">
|
||
<span>一条可信的完成声明</span>
|
||
<strong>Completion claim</strong>
|
||
</figcaption>
|
||
<div class="claim-line claim-positive">
|
||
<span class="claim-index">01</span>
|
||
<div><strong>目标效果</strong><small>required effects</small></div>
|
||
<span class="claim-state">已观察</span>
|
||
</div>
|
||
<div class="claim-line claim-negative">
|
||
<span class="claim-index">02</span>
|
||
<div><strong>禁止副作用</strong><small>forbidden effects</small></div>
|
||
<span class="claim-state">未发现</span>
|
||
</div>
|
||
<div class="claim-line claim-process">
|
||
<span class="claim-index">03</span>
|
||
<div><strong>必要过程</strong><small>procedure / evidence</small></div>
|
||
<span class="claim-state">可追溯</span>
|
||
</div>
|
||
<div class="claim-line claim-unknown">
|
||
<span class="claim-index">04</span>
|
||
<div><strong>剩余未知</strong><small>residual uncertainty</small></div>
|
||
<span class="claim-state">已声明</span>
|
||
</div>
|
||
<p>缺少任意一层，都只能说“部分可证”，不能用自信语气补齐。</p>
|
||
</figure>
|
||
|
||
<dl class="audit-strip" aria-label="研究证据边界">
|
||
<div><dt>全文池</dt><dd>40</dd><small>问题驱动，不是标题投票</small></div>
|
||
<div><dt>证据账本</dt><dd>37</dd><small>核过结果、消融或限制</small></div>
|
||
<div><dt>核心证据</dt><dd>26</dd><small>直接承担一般结论</small></div>
|
||
<div><dt>自有跑分</dt><dd>0</dd><small>本页是研究，不是 benchmark</small></div>
|
||
</dl>
|
||
</section>
|
||
|
||
<section class="conclusion-index section-shell" id="conclusions" data-section>
|
||
<header class="section-heading">
|
||
<p class="section-kicker">01 · 先读结论</p>
|
||
<h2>八条可以带走的判断</h2>
|
||
<p>先建立判断框架，再进入论文、数字和例外。</p>
|
||
</header>
|
||
|
||
<ol class="conclusion-list">
|
||
<li>
|
||
<a href="#finding-1">
|
||
<span>01</span>
|
||
<strong>完成是外部证据，不是最终文本</strong>
|
||
<small>Agent 自述、测试通过和状态成功都只覆盖一部分。</small>
|
||
</a>
|
||
</li>
|
||
<li>
|
||
<a href="#capabilities">
|
||
<span>02</span>
|
||
<strong>Agent 能力不能压成一个总分</strong>
|
||
<small>结果、过程、协作、恢复和可靠性是不同能力。</small>
|
||
</a>
|
||
</li>
|
||
<li>
|
||
<a href="#finding-3">
|
||
<span>03</span>
|
||
<strong>不存在一个通用 verifier</strong>
|
||
<small>规则、轨迹 judge、模型 judge 和人各有盲区。</small>
|
||
</a>
|
||
</li>
|
||
<li>
|
||
<a href="#finding-4">
|
||
<span>04</span>
|
||
<strong>行动者不能批准自己</strong>
|
||
<small>独立环境、权限和重放证据比更多 reasoning 更关键。</small>
|
||
</a>
|
||
</li>
|
||
<li>
|
||
<a href="#finding-5">
|
||
<span>05</span>
|
||
<strong>验证效果，不要写死实现路径</strong>
|
||
<small>等价路径应被允许，必要程序仍必须被证明。</small>
|
||
</a>
|
||
</li>
|
||
<li>
|
||
<a href="#finding-6">
|
||
<span>06</span>
|
||
<strong>验证失败不等于会恢复</strong>
|
||
<small>发现、定位、选路、恢复和复验会分别失败。</small>
|
||
</a>
|
||
</li>
|
||
<li>
|
||
<a href="#finding-7">
|
||
<span>07</span>
|
||
<strong>没有固定的恢复阶梯</strong>
|
||
<small>重试、修复、重规划、回滚和升级依赖当前状态。</small>
|
||
</a>
|
||
</li>
|
||
<li>
|
||
<a href="#finding-8">
|
||
<span>08</span>
|
||
<strong>更多轮次可能主动破坏结果</strong>
|
||
<small>没有新证据的 verify-repair loop 只会稳定偏差。</small>
|
||
</a>
|
||
</li>
|
||
</ol>
|
||
</section>
|
||
|
||
<section class="capability-section section-shell" id="capabilities" data-section>
|
||
<header class="section-heading split-heading">
|
||
<div>
|
||
<p class="section-kicker">02 · 能力拆解</p>
|
||
<h2>大家究竟在评价 Agent 的什么</h2>
|
||
</div>
|
||
<p>
|
||
“题有没有做完”只是结果位。真正可诊断的评估，需要知道哪一段能力发生了变化。
|
||
</p>
|
||
</header>
|
||
|
||
<div class="capability-grid">
|
||
<article>
|
||
<span class="capability-code">A / INTENT</span>
|
||
<h3>理解与约束</h3>
|
||
<ul>
|
||
<li><strong>目标理解</strong><span>做的是用户真正要的事吗</span></li>
|
||
<li><strong>限制识别</strong><span>权限、范围、预算是否被保留</span></li>
|
||
<li><strong>必要澄清</strong><span>意图不完整时是否停止猜测</span></li>
|
||
</ul>
|
||
</article>
|
||
<article>
|
||
<span class="capability-code">B / ACTION</span>
|
||
<h3>执行与环境</h3>
|
||
<ul>
|
||
<li><strong>路径选择</strong><span>计划是否可达、代价是否合理</span></li>
|
||
<li><strong>工具契约</strong><span>工具、参数、顺序是否正确</span></li>
|
||
<li><strong>状态改变</strong><span>外部世界是否真的发生变化</span></li>
|
||
</ul>
|
||
</article>
|
||
<article>
|
||
<span class="capability-code">C / CONTROL</span>
|
||
<h3>过程与副作用</h3>
|
||
<ul>
|
||
<li><strong>副作用控制</strong><span>是否只改变允许的对象</span></li>
|
||
<li><strong>程序合规</strong><span>确认、授权、读取是否被绕过</span></li>
|
||
<li><strong>协作能力</strong><span>能否指导用户或其他 Agent</span></li>
|
||
</ul>
|
||
</article>
|
||
<article>
|
||
<span class="capability-code">D / RECOVERY</span>
|
||
<h3>失败与恢复</h3>
|
||
<ul>
|
||
<li><strong>失败检测</strong><span>是否知道当前结果不可信</span></li>
|
||
<li><strong>根因定位</strong><span>哪一步让任务注定失败</span></li>
|
||
<li><strong>恢复执行</strong><span>新动作是否真的恢复目标状态</span></li>
|
||
</ul>
|
||
</article>
|
||
<article>
|
||
<span class="capability-code">E / TRUST</span>
|
||
<h3>收尾与可靠性</h3>
|
||
<ul>
|
||
<li><strong>诚实收尾</strong><span>最终声明是否与证据一致</span></li>
|
||
<li><strong>重复可靠</strong><span>多次运行是否仍然成立</span></li>
|
||
<li><strong>效率边界</strong><span>用了多少时间、调用和权限</span></li>
|
||
</ul>
|
||
</article>
|
||
</div>
|
||
|
||
<blockquote class="research-question">
|
||
<span>更严格的问题</span>
|
||
<p>
|
||
在任务、环境、工具、权限和预算固定时，这个系统在哪一种能力上比基线更好；
|
||
它带来了哪些收益、回归和新的不可判定项？
|
||
</p>
|
||
</blockquote>
|
||
</section>
|
||
|
||
<section class="model-section section-shell" id="model" data-section>
|
||
<header class="section-heading split-heading">
|
||
<div>
|
||
<p class="section-kicker">03 · 研究模型</p>
|
||
<h2>完成证明与失败恢复，是两条相连但不同的链</h2>
|
||
</div>
|
||
<p>验证器负责拒绝没有证据的成功；恢复系统负责决定拒绝之后怎样继续。</p>
|
||
</header>
|
||
|
||
<div class="two-chain-model">
|
||
<article class="chain-panel">
|
||
<div class="chain-label">A · CLOSURE / 完成证明</div>
|
||
<ol>
|
||
<li><span>01</span><div><strong>Effect</strong><small>目标效果是否发生</small></div></li>
|
||
<li><span>02</span><div><strong>Invariant</strong><small>禁止副作用是否出现</small></div></li>
|
||
<li><span>03</span><div><strong>Procedure</strong><small>必要过程是否合规</small></div></li>
|
||
<li><span>04</span><div><strong>Residual</strong><small>剩余语义与未知是否声明</small></div></li>
|
||
</ol>
|
||
<div class="chain-outcome">
|
||
<span>结果状态</span>
|
||
<strong>PASS · FAIL · AMBIGUOUS · UNVERIFIABLE</strong>
|
||
</div>
|
||
</article>
|
||
|
||
<article class="chain-panel recovery-panel">
|
||
<div class="chain-label">B · RECOVERY / 失败恢复</div>
|
||
<ol>
|
||
<li><span>01</span><div><strong>Detect</strong><small>发现当前结果不可信</small></div></li>
|
||
<li><span>02</span><div><strong>Localize</strong><small>定位责任步骤和最后可信状态</small></div></li>
|
||
<li><span>03</span><div><strong>Route</strong><small>选择重试、修复、重规划或升级</small></div></li>
|
||
<li><span>04</span><div><strong>Restore</strong><small>执行恢复并处理已有副作用</small></div></li>
|
||
<li><span>05</span><div><strong>Re-verify</strong><small>从独立状态重新证明</small></div></li>
|
||
</ol>
|
||
<div class="chain-outcome warning">
|
||
<span>关键边界</span>
|
||
<strong>检出失败，只是把“假成功”变成“明确失败”</strong>
|
||
</div>
|
||
</article>
|
||
</div>
|
||
|
||
<div class="verifier-roles">
|
||
<div>
|
||
<strong>确定性状态检查</strong>
|
||
<p>负责机器可观察的效果和禁止副作用。精确、可复现，但合同昂贵且覆盖有限。</p>
|
||
</div>
|
||
<div>
|
||
<strong>轨迹与程序规则</strong>
|
||
<p>负责确认、授权、顺序和禁止动作。能看过程，但过严时会拒绝合法路径。</p>
|
||
</div>
|
||
<div>
|
||
<strong>LLM / Agent judge</strong>
|
||
<p>负责剩余语义、等价路径和部分进展。适合辅助判断，不适合独立发布认证。</p>
|
||
</div>
|
||
<div>
|
||
<strong>人工复核</strong>
|
||
<p>负责高风险、冲突、不可观察和价值判断。不是无限资源，应由明确异常触发。</p>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="arguments-section section-shell" id="arguments" data-section>
|
||
<header class="section-heading split-heading">
|
||
<div>
|
||
<p class="section-kicker">04 · 逐条论证</p>
|
||
<h2>结论为什么成立，又在哪里停止</h2>
|
||
</div>
|
||
<p>每一条都同时保留直接证据和不能外推的边界。</p>
|
||
</header>
|
||
|
||
<article class="finding" id="finding-1">
|
||
<div class="finding-number">01</div>
|
||
<div class="finding-body">
|
||
<p class="finding-label">COMPLETION</p>
|
||
<h3>完成是证据合同，不是最终文本</h3>
|
||
<p class="finding-thesis">
|
||
Agent 的语言只能报告完成，不能构成完成。可相信的对象是外部状态、禁止副作用、
|
||
必要过程和仍未被观察的部分。
|
||
</p>
|
||
<div class="evidence-points">
|
||
<div><strong>45% / 48%</strong><span>tau2 airline / retail 失败轨迹中的假完成</span></div>
|
||
<div><strong>75.8%</strong><span>AppWorld 明确声称完成的失败轨迹子集</span></div>
|
||
<div><strong>7.8%</strong><span>已通过 SWE-bench 的补丁被额外开发者测试直接判错</span></div>
|
||
</div>
|
||
<details>
|
||
<summary>证据与边界</summary>
|
||
<div class="detail-copy">
|
||
<p>
|
||
tau-bench、AppWorld 和 OSWorld 把完成落到数据库、文件或桌面状态；
|
||
AppWorld 还检查允许集合之外是否出现额外变化。Procedure-Aware Evaluation
|
||
又证明，状态成功仍可能绕过必要确认和策略。
|
||
</p>
|
||
<p>
|
||
这些比例来自不同子集，不能直接互比。结论不是“形式化一切”，而是系统只能在
|
||
证据覆盖范围内声称完成；开放任务需要保留 <code>AMBIGUOUS</code> 和
|
||
<code>UNVERIFIABLE</code>。
|
||
</p>
|
||
</div>
|
||
</details>
|
||
</div>
|
||
</article>
|
||
|
||
<article class="finding" id="finding-2">
|
||
<div class="finding-number">02</div>
|
||
<div class="finding-body">
|
||
<p class="finding-label">PROTOCOL</p>
|
||
<h3>评估对象是整个运行协议，不只是模型</h3>
|
||
<p class="finding-thesis">
|
||
一个 Agent 分数实际衡量的是模型、prompt、orchestrator、工具、环境、用户模拟器、
|
||
verifier、预算和版本的乘积。
|
||
</p>
|
||
<pre class="formula"><code>model × prompt × orchestrator × tools × environment
|
||
× user/simulator × verifier × budget × version</code></pre>
|
||
<details>
|
||
<summary>证据与边界</summary>
|
||
<div class="detail-copy">
|
||
<p>
|
||
DynamicMCPBench 的 126 次 live 重放中，只有 36% 返回完全相同，33% 漂移，
|
||
32% 已损坏。AgentLens 的 provider 工具参数解析故障，也会被误读成模型能力差。
|
||
</p>
|
||
<p>
|
||
因此每个结果都要携带模型版本、prompt、工具、环境快照、validator、预算和种子。
|
||
没有 manifest，跨版本分数没有稳定含义。
|
||
</p>
|
||
</div>
|
||
</details>
|
||
</div>
|
||
</article>
|
||
|
||
<article class="finding" id="finding-3">
|
||
<div class="finding-number">03</div>
|
||
<div class="finding-body">
|
||
<p class="finding-label">VERIFIER</p>
|
||
<h3>不存在一个通用 verifier</h3>
|
||
<p class="finding-thesis">
|
||
规则判分更容易漏报，模型 judge 更容易误报。可靠方案不是让两者投票，
|
||
而是让它们承担不同责任。
|
||
</p>
|
||
<div class="contrast-table" role="table" aria-label="不同验证器的误差">
|
||
<div role="row">
|
||
<span role="cell">官方规则 evaluator</span>
|
||
<strong role="cell">83.8% precision</strong>
|
||
<strong role="cell">55.9% recall</strong>
|
||
</div>
|
||
<div role="row">
|
||
<span role="cell">最佳 LLM judge</span>
|
||
<strong role="cell">< 70% precision</strong>
|
||
<strong role="cell">更高 recall</strong>
|
||
</div>
|
||
</div>
|
||
<details>
|
||
<summary>证据与边界</summary>
|
||
<div class="detail-copy">
|
||
<p>
|
||
AgentRewardBench 给出了 precision/recall 对照；AJ-Bench 中让 judge 使用工具
|
||
虽然明显改善判断，不同任务域 false-positive rate 仍为 8.77%-56.60%。
|
||
AgentLens 的两个 judge 在 23% 比较上选择不同赢家。
|
||
</p>
|
||
<p>
|
||
LLM judge 可以处理规则未覆盖的语义等价，也可以提出异常；它不应单独把一个
|
||
确定性失败降级为成功。
|
||
</p>
|
||
</div>
|
||
</details>
|
||
</div>
|
||
</article>
|
||
|
||
<article class="finding" id="finding-4">
|
||
<div class="finding-number">04</div>
|
||
<div class="finding-body">
|
||
<p class="finding-label">INDEPENDENCE</p>
|
||
<h3>verifier 必须与 actor 分权</h3>
|
||
<p class="finding-thesis">
|
||
同一个 Agent 同时行动、解释和批准自己，会形成最危险的确认偏差闭环。
|
||
verifier 应拥有 actor 无法伪造的状态、权限和重放环境。
|
||
</p>
|
||
<div class="before-after">
|
||
<div><small>同模型自评</small><strong>10 / 27</strong><span>真实 XSS 报告</span></div>
|
||
<div class="arrow" aria-hidden="true">→</div>
|
||
<div><small>隔离重放 + 角色分离</small><strong>30 / 30</strong><span>作者报告真阳性</span></div>
|
||
</div>
|
||
<details>
|
||
<summary>证据与边界</summary>
|
||
<div class="detail-copy">
|
||
<p>
|
||
RECEIPT 的消融说明，关键变化来自浏览器判决、隔离、可重放 PoC 和角色分离，
|
||
不是让 judge 再思考一轮。
|
||
</p>
|
||
<p>
|
||
这是单模型、白盒 XSS 和特定 harness 的结果，不能当通用精度保证；它支持的是
|
||
分权原则，而不是那组绝对分数。
|
||
</p>
|
||
</div>
|
||
</details>
|
||
</div>
|
||
</article>
|
||
|
||
<article class="finding" id="finding-5">
|
||
<div class="finding-number">05</div>
|
||
<div class="finding-body">
|
||
<p class="finding-label">EQUIVALENCE</p>
|
||
<h3>验证效果，不要要求复制 gold path</h3>
|
||
<p class="finding-thesis">
|
||
实现路径原则上可以等价；授权、确认、来源和不可逆操作如果属于任务义务，
|
||
则必须被证明。两者不能混为一谈。
|
||
</p>
|
||
<div class="path-model">
|
||
<div>
|
||
<small>可以变化</small>
|
||
<strong>implementation path</strong>
|
||
<span>工具选择、步骤拆分、等价实现</span>
|
||
</div>
|
||
<div>
|
||
<small>不能绕过</small>
|
||
<strong>causal / procedural obligation</strong>
|
||
<span>授权、确认、来源、不可逆动作</span>
|
||
</div>
|
||
</div>
|
||
<details>
|
||
<summary>证据与边界</summary>
|
||
<div class="detail-copy">
|
||
<p>
|
||
AppWorld 用状态测试允许多个合法实现；DynamicMCPBench 采用 path-agnostic
|
||
effect scoring，并发现 reference answer 完全正确的比例只有 79%。
|
||
ToolSandbox 则用 milestone DAG 与 minefield 同时保留弹性和禁止动作。
|
||
</p>
|
||
</div>
|
||
</details>
|
||
</div>
|
||
</article>
|
||
|
||
<article class="finding" id="finding-6">
|
||
<div class="finding-number">06</div>
|
||
<div class="finding-body">
|
||
<p class="finding-label">RECOVERY</p>
|
||
<h3>验证失败不等于会恢复</h3>
|
||
<p class="finding-thesis">
|
||
发现错误、定位原因、选择响应、执行恢复、再次证明，是五个不同的能力。
|
||
只展示最终分数，会隐藏增益究竟来自诊断、更多尝试还是更强 fallback。
|
||
</p>
|
||
<div class="evidence-points">
|
||
<div><strong>17.58%</strong><span>ToolMaze 隐式永久故障的平均恢复率，即使提供 hint</span></div>
|
||
<div><strong>28.8%</strong><span>AgentDebugX 严格 agent + exact-step 根因定位</span></div>
|
||
<div><strong>13 / 73</strong><span>定位后在 GAIA 失败集上的一次重跑恢复</span></div>
|
||
</div>
|
||
<details>
|
||
<summary>证据与边界</summary>
|
||
<div class="detail-copy">
|
||
<p>
|
||
Hell or High Water 保证存在短替代路径，模型仍下降约 22-30 个成功率点。
|
||
R2Act 中根因服务定位可达 91.4%-99.7%，恢复动作有效率却只有 36.8%-60.3%。
|
||
</p>
|
||
<p>
|
||
AgentDebugX 的 GAIA 对照比较的是完整 recipe，且可见参考答案，不能把全部增益
|
||
归因于定位方法。
|
||
</p>
|
||
</div>
|
||
</details>
|
||
</div>
|
||
</article>
|
||
|
||
<article class="finding" id="finding-7">
|
||
<div class="finding-number">07</div>
|
||
<div class="finding-body">
|
||
<p class="finding-label">ROUTING</p>
|
||
<h3>恢复策略必须依赖失败状态</h3>
|
||
<p class="finding-thesis">
|
||
重试适合瞬时且幂等的错误；局部修复适合已定位错误；重规划适合路径失效；
|
||
回滚处理已有副作用；升级和停止处理权限、能力与不可逆风险。
|
||
</p>
|
||
<div class="route-table">
|
||
<div><strong>retry</strong><span>超时、限流、偶发工具错误</span><small>动作必须幂等</small></div>
|
||
<div><strong>repair</strong><span>错误局部且责任步骤可信</span><small>保留仍有价值的状态</small></div>
|
||
<div><strong>replan</strong><span>原路径失效但目标仍可达</span><small>先处理已有副作用</small></div>
|
||
<div><strong>rollback</strong><span>变更可撤销或补偿</span><small>依赖完整变更账本</small></div>
|
||
<div><strong>escalate</strong><span>权限、能力、意图或证据不足</span><small>交给人或更强系统</small></div>
|
||
<div><strong>stop</strong><span>风险不可逆、预算耗尽</span><small>诚实保留未完成</small></div>
|
||
</div>
|
||
<details>
|
||
<summary>证据与边界</summary>
|
||
<div class="detail-copy">
|
||
<p>
|
||
CodeRescue 的固定 reflect / replan / escalate 恢复率分别为
|
||
27.5% / 45.3% / 68.6%，学习 router 为 81.7%。可恢复失败中，
|
||
28% 只能由便宜策略解决，45% 只能升级解决，27% 两者均可。
|
||
</p>
|
||
<p>
|
||
这是代码任务、特定模型和单次路由结果；它支持“策略依赖状态”，
|
||
不证明该 router 可以直接进入生产。
|
||
</p>
|
||
</div>
|
||
</details>
|
||
</div>
|
||
</article>
|
||
|
||
<article class="finding" id="finding-8">
|
||
<div class="finding-number">08</div>
|
||
<div class="finding-body">
|
||
<p class="finding-label">STOPPING</p>
|
||
<h3>没有新证据的恢复循环会开始破坏结果</h3>
|
||
<p class="finding-thesis">
|
||
重复询问同一个有偏 judge，只会得到更稳定的偏差。只有新增独立证据或改变恢复策略，
|
||
下一轮才有信息价值。
|
||
</p>
|
||
<div class="before-after repair-risk">
|
||
<div><small>初始正确率</small><strong>0.700</strong><span>stress 设置</span></div>
|
||
<div class="arrow" aria-hidden="true">→</div>
|
||
<div class="negative-result"><small>固定五轮修复</small><strong>0.116</strong><span>正确计划被反复改坏</span></div>
|
||
<div class="arrow" aria-hidden="true">→</div>
|
||
<div><small>带停止规则</small><strong>0.722</strong><span>在该 stress 条件下</span></div>
|
||
</div>
|
||
<details>
|
||
<summary>证据与边界</summary>
|
||
<div class="detail-copy">
|
||
<p>
|
||
VRR-Stop 的巨大差值来自刻意制造的 verifier/repair mismatch，不能解释成一般收益。
|
||
但 Reflexion、CRITIC 和 AgentLTL 也分别观察到无测试反思、错误修改和强制终止回归。
|
||
</p>
|
||
<p>
|
||
停止条件至少要问：有没有新独立证据、失败类型是否变化、状态是否改善、
|
||
下一步是否可逆、预期收益是否仍大于成本和风险。
|
||
</p>
|
||
</div>
|
||
</details>
|
||
</div>
|
||
</article>
|
||
</section>
|
||
|
||
<section class="progress-section section-shell" id="progress" data-section>
|
||
<header class="section-heading split-heading">
|
||
<div>
|
||
<p class="section-kicker">05 · 发展脉络</p>
|
||
<h2>最近的进步，主要是把失败说清楚</h2>
|
||
</div>
|
||
<p>这不是一条成功率持续上涨的曲线，而是研究对象逐步被拆开。</p>
|
||
</header>
|
||
|
||
<div class="timeline">
|
||
<article>
|
||
<time>2023</time>
|
||
<h3>让模型根据反馈自我修正</h3>
|
||
<p>Reflexion、CRITIC、自调试证明外部执行反馈有用。</p>
|
||
<small>仍会误改正确答案，自我反馈很弱。</small>
|
||
</article>
|
||
<article>
|
||
<time>2024</time>
|
||
<h3>检查 Agent 是否真的改变环境</h3>
|
||
<p>WebArena、SWE-bench、tau-bench、AppWorld、OSWorld 引入状态、轨迹与 `pass^k`。</p>
|
||
<small>evaluator 昂贵、不完备，过程和副作用覆盖不足。</small>
|
||
</article>
|
||
<article>
|
||
<time>2025</time>
|
||
<h3>开始审计 evaluator 和故障恢复</h3>
|
||
<p>AgentRewardBench、SWE 测试审计、tau2、AgentDebug、替代路径 benchmark。</p>
|
||
<small>judge 假阳性高，恢复仍与一般求解能力混在一起。</small>
|
||
</article>
|
||
<article>
|
||
<time>2026</time>
|
||
<h3>验证过程、隔离证据、路由并停止</h3>
|
||
<p>PAE、AgentLTL、RECEIPT、ToolMaze、CodeRescue、VRR-Stop、DynamicMCPBench。</p>
|
||
<small>多数仍是单域 preprint，真实回滚和开放任务没有解决。</small>
|
||
</article>
|
||
</div>
|
||
|
||
<div class="progress-verdict">
|
||
<div>
|
||
<span>已经进步</span>
|
||
<strong>测量与分解</strong>
|
||
<p>假完成、程序违规、副作用、根因定位、恢复路由和停止条件开始被单独测量。</p>
|
||
</div>
|
||
<div>
|
||
<span>局部有效</span>
|
||
<strong>可判定门禁与独立重放</strong>
|
||
<p>对高价值规则做前置 gate，对高风险结果做隔离重放，有直接对照支持。</p>
|
||
</div>
|
||
<div>
|
||
<span>尚未证明</span>
|
||
<strong>通用可靠 Agent</strong>
|
||
<p>隐式语义故障、开放目标、真实服务漂移和不可逆副作用仍缺强证据。</p>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="synthesis-section section-shell" id="synthesis" data-section>
|
||
<header class="section-heading split-heading">
|
||
<div>
|
||
<p class="section-kicker">06 · 系统推论</p>
|
||
<h2>证据共同约束出的候选系统</h2>
|
||
</div>
|
||
<p class="inference-note">
|
||
这是跨论文综合出的推论，不是任何一篇论文已经证明的架构，也不是本项目已完成的实现。
|
||
</p>
|
||
</header>
|
||
|
||
<ol class="system-flow">
|
||
<li><span>01</span><div><strong>Task Contract</strong><p>效果、禁止项、必要过程、权限、预算、可逆性与可观察边界。</p></div></li>
|
||
<li><span>02</span><div><strong>Mutation Gate</strong><p>写操作执行前检查授权、对象、参数、幂等、回滚和确认。</p></div></li>
|
||
<li><span>03</span><div><strong>Trace + Change Ledger</strong><p>保存原始返回、状态快照、变更、错误、证据版本和恢复分支。</p></div></li>
|
||
<li><span>04</span><div><strong>Independent Closure</strong><p>效果、禁止副作用、程序和剩余语义依次判断。</p></div></li>
|
||
<li><span>05</span><div><strong>Failure Object</strong><p>违反条款、期望与观察、最后可信状态、变更、重试性和缺失证据。</p></div></li>
|
||
<li><span>06</span><div><strong>Recovery Router</strong><p>选择 retry、repair、replan、rollback、clarify、escalate 或 stop。</p></div></li>
|
||
<li><span>07</span><div><strong>Completion Certificate</strong><p>说明改变了什么、通过哪些检查、什么未检查、还剩什么不确定。</p></div></li>
|
||
</ol>
|
||
|
||
<div class="certificate-preview" aria-label="完成证书示意">
|
||
<div class="certificate-header">
|
||
<span>COMPLETION CERTIFICATE / 示例结构</span>
|
||
<strong>PARTIALLY VERIFIED</strong>
|
||
</div>
|
||
<dl>
|
||
<div><dt>Observed change</dt><dd>目标状态已由独立读取确认</dd></div>
|
||
<div><dt>Passed checks</dt><dd>效果 4/4 · 禁止副作用 3/3 · 程序 2/2</dd></div>
|
||
<div><dt>Not checked</dt><dd>外部第三方系统的最终同步状态</dd></div>
|
||
<div><dt>Residual risk</dt><dd>结果可能在服务刷新后漂移</dd></div>
|
||
<div><dt>Recovery history</dt><dd>1 次瞬时工具重试，无写入回滚</dd></div>
|
||
</dl>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="evidence-section section-shell" id="evidence" data-section>
|
||
<header class="section-heading split-heading">
|
||
<div>
|
||
<p class="section-kicker">07 · 证据账本</p>
|
||
<h2>37 篇全文，按支持范围而不是标题排列</h2>
|
||
</div>
|
||
<p>
|
||
每条记录同时保留直接数字、能支持的判断和不能外推的部分。
|
||
`context` 只作历史背景。
|
||
</p>
|
||
</header>
|
||
|
||
<div class="evidence-tools">
|
||
<label class="evidence-search" for="evidence-search">
|
||
<span>搜索证据</span>
|
||
<input
|
||
id="evidence-search"
|
||
type="search"
|
||
placeholder="论文、数字、失败、judge、回滚…"
|
||
autocomplete="off"
|
||
>
|
||
</label>
|
||
|
||
<div class="filter-group" aria-label="证据级别">
|
||
<span>级别</span>
|
||
<div class="segmented">
|
||
<button class="is-active" type="button" data-depth="all" aria-pressed="true">全部</button>
|
||
<button type="button" data-depth="core" aria-pressed="false">核心</button>
|
||
<button type="button" data-depth="support" aria-pressed="false">旁证</button>
|
||
<button type="button" data-depth="context" aria-pressed="false">背景</button>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="filter-group" aria-label="证据主题">
|
||
<span>主题</span>
|
||
<div class="segmented theme-filter">
|
||
<button class="is-active" type="button" data-theme="all" aria-pressed="true">全部</button>
|
||
<button type="button" data-theme="verification" aria-pressed="false">完成证明</button>
|
||
<button type="button" data-theme="protocol" aria-pressed="false">协议审计</button>
|
||
<button type="button" data-theme="recovery" aria-pressed="false">失败恢复</button>
|
||
<button type="button" data-theme="reliability" aria-pressed="false">风险可靠性</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="evidence-summary">
|
||
<p id="evidence-count">载入证据账本…</p>
|
||
<button type="button" id="expand-evidence">展开当前结果</button>
|
||
</div>
|
||
|
||
<div class="evidence-list" id="evidence-list" aria-live="polite"></div>
|
||
<p class="empty-evidence" id="empty-evidence" hidden>没有符合当前条件的证据。</p>
|
||
</section>
|
||
|
||
<section class="unknown-section section-shell" id="unknowns" data-section>
|
||
<header class="section-heading split-heading">
|
||
<div>
|
||
<p class="section-kicker">08 · 未知与预测</p>
|
||
<h2>研究结束在这里，不用架构图填补空白</h2>
|
||
</div>
|
||
<p>下面的问题仍没有被解决；四条预测以后可以被实验反驳。</p>
|
||
</header>
|
||
|
||
<div class="unknown-layout">
|
||
<div>
|
||
<h3>仍然不知道</h3>
|
||
<ol class="unknown-list">
|
||
<li><span>01</span><p>开放任务如何低成本写合同，而不把维护成本推给人。</p></li>
|
||
<li><span>02</span><p>怎样发现合同中没有预先列出的未知副作用。</p></li>
|
||
<li><span>03</span><p>怎样判断替代路径真的语义等价，而不是不安全捷径。</p></li>
|
||
<li><span>04</span><p>数据库、消息、付款和外部 API 如何真实回滚或补偿。</p></li>
|
||
<li><span>05</span><p>长轨迹根因如何可靠定位，最新严格准确率仍低于 40%。</p></li>
|
||
<li><span>06</span><p>verifier 在分布漂移和被优化攻击后怎样保持可信。</p></li>
|
||
<li><span>07</span><p>用户满意、部分帮助与客观效果如何放在同一合同里。</p></li>
|
||
<li><span>08</span><p>恢复策略能否跨代码、客服、安全、桌面和真实业务泛化。</p></li>
|
||
</ol>
|
||
</div>
|
||
|
||
<div>
|
||
<h3>可反驳的预测</h3>
|
||
<ol class="prediction-list">
|
||
<li>
|
||
<span>P1</span>
|
||
<p><strong>mutation gate + effect check</strong> 会减少假完成，但过宽 gate 会增加拒绝。</p>
|
||
</li>
|
||
<li>
|
||
<span>P2</span>
|
||
<p>基于 failure object 的路由会在永久故障和隐式语义故障上优于固定 retry/replan。</p>
|
||
</li>
|
||
<li>
|
||
<span>P3</span>
|
||
<p>独立 verifier 的最大收益会体现在 false-success 下降，而不一定是平均成功率上升。</p>
|
||
</li>
|
||
<li>
|
||
<span>P4</span>
|
||
<p>当恢复轮次不产生新独立证据时，边际收益会迅速归零并开始制造回归。</p>
|
||
</li>
|
||
</ol>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
</main>
|
||
|
||
<footer class="site-footer">
|
||
<div>
|
||
<strong>K1412 Agent Research</strong>
|
||
<p>
|
||
网页负责阅读，事实源仍是
|
||
<a
|
||
href="https://git.k1412.top/wuyang/agent/src/branch/main/research/completion-verification/findings.md"
|
||
target="_blank"
|
||
rel="noreferrer"
|
||
>findings.md</a>
|
||
与
|
||
<a
|
||
href="https://git.k1412.top/wuyang/agent/src/branch/main/research/completion-verification/evidence-ledger.md"
|
||
target="_blank"
|
||
rel="noreferrer"
|
||
>evidence-ledger.md</a>。
|
||
</p>
|
||
</div>
|
||
<div class="footer-meta">
|
||
<span>40 full texts</span>
|
||
<span>37 audited</span>
|
||
<span>0 self-run benchmarks</span>
|
||
</div>
|
||
</footer>
|
||
|
||
<noscript>
|
||
<p class="noscript-note">证据筛选需要 JavaScript；正文内容不受影响。</p>
|
||
</noscript>
|
||
</body>
|
||
</html>
|