wuyang
354 lines
16 KiB
HTML
354 lines
16 KiB
HTML
<!doctype html>
|
||
<html lang="zh-CN">
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||
<meta name="theme-color" content="#f3f4ef">
|
||
<meta
|
||
name="description"
|
||
content="K1412 Agent 评估方案：用任务契约、独立判分和配对回归决定一次 Agent 改动是否应该上线。"
|
||
>
|
||
<title>K1412 Agent 评估</title>
|
||
<link
|
||
rel="icon"
|
||
href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' fill='%23121713'/%3E%3Ctext x='16' y='22' text-anchor='middle' font-family='serif' font-size='18' fill='white'%3EK%3C/text%3E%3C/svg%3E"
|
||
>
|
||
<link rel="stylesheet" href="/assets/styles.css">
|
||
<script src="/assets/app.js" defer></script>
|
||
</head>
|
||
<body>
|
||
<a class="skip-link" href="#main">跳到正文</a>
|
||
|
||
<header class="topbar">
|
||
<a class="brand" href="#conclusion" aria-label="K1412 Agent 评估首页">
|
||
<span class="brand-mark" aria-hidden="true">K</span>
|
||
<span>
|
||
<strong>Agent 评估</strong>
|
||
<small>K1412 Research</small>
|
||
</span>
|
||
</a>
|
||
|
||
<nav class="topnav" aria-label="页面导航">
|
||
<a class="is-active" href="#conclusion">结论</a>
|
||
<a href="#decision">决策</a>
|
||
<a href="#layers">层次</a>
|
||
<a href="#tasks">任务</a>
|
||
<a href="#artifacts">产物</a>
|
||
</nav>
|
||
|
||
<a
|
||
class="source-link"
|
||
href="https://git.k1412.top/wuyang/agent"
|
||
target="_blank"
|
||
rel="noreferrer"
|
||
>源仓库</a>
|
||
</header>
|
||
|
||
<main id="main">
|
||
<section class="report-intro" id="conclusion" data-section>
|
||
<div class="section-number">01 / 结论</div>
|
||
<div class="intro-layout">
|
||
<div class="intro-copy">
|
||
<p class="eyebrow">K1412 Agent Evaluation v1 · 2026.07.27</p>
|
||
<h1>不评总分，只评一次改动是否该上线。</h1>
|
||
<p class="lead">
|
||
固定模型、任务、工作区和预算，只改变一个 Agent 行为假设。
|
||
用独立验证回答它改善了什么、破坏了什么，以及证据是否足以进入生产。
|
||
</p>
|
||
</div>
|
||
|
||
<aside class="evidence-boundary" aria-label="证据边界">
|
||
<p class="boundary-label">当前证据边界</p>
|
||
<p>
|
||
方案来自对 <code>zk-data-agent@e2e7a7b</code> 真实运行事件、
|
||
隔离工作区和完成门禁的检查。
|
||
</p>
|
||
<p class="boundary-warning">
|
||
尚未运行真实模型对比。最新论文只是摘要级线索。
|
||
</p>
|
||
</aside>
|
||
</div>
|
||
|
||
<dl class="headline-stats" aria-label="方案摘要">
|
||
<div>
|
||
<dt>任务契约</dt>
|
||
<dd>10</dd>
|
||
<small>跨产物、恢复、上下文、安全与委派</small>
|
||
</div>
|
||
<div>
|
||
<dt>评测层次</dt>
|
||
<dd>6</dd>
|
||
<small>从确定性单测到生产价值抽样</small>
|
||
</div>
|
||
<div>
|
||
<dt>当前可跑</dt>
|
||
<dd>2</dd>
|
||
<small>其余任务明确标注 runner 缺口</small>
|
||
</div>
|
||
<div>
|
||
<dt>真实对比</dt>
|
||
<dd>0</dd>
|
||
<small>不把设计契约伪装成实验结论</small>
|
||
</div>
|
||
</dl>
|
||
|
||
<div class="method-rail" aria-label="评估闭环">
|
||
<div><span>01</span><strong>问题</strong><small>一次只改变一个假设</small></div>
|
||
<div><span>02</span><strong>任务</strong><small>固定夹具、预算和模型</small></div>
|
||
<div><span>03</span><strong>运行</strong><small>配对、重复、随机顺序</small></div>
|
||
<div><span>04</span><strong>判分</strong><small>独立 Validator 与轨迹规则</small></div>
|
||
<div><span>05</span><strong>决定</strong><small>推广、继续或拒绝</small></div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="decision-section" id="decision" data-section>
|
||
<div class="section-heading">
|
||
<div>
|
||
<div class="section-number">02 / 发布决策</div>
|
||
<h2>先看回归，再谈平均提升</h2>
|
||
</div>
|
||
<p>
|
||
下面是方法演算，不是真实跑分。它使用与仓库评分器一致的配对逻辑和统计口径。
|
||
</p>
|
||
</div>
|
||
|
||
<div class="decision-workbench">
|
||
<form class="simulation-controls" id="simulation-form">
|
||
<fieldset>
|
||
<legend>配对运行结果</legend>
|
||
|
||
<label class="range-row" for="gains">
|
||
<span><strong>新增成功</strong><small>基线失败，候选成功</small></span>
|
||
<input id="gains" name="gains" type="range" min="0" max="20" value="6">
|
||
<output for="gains">6</output>
|
||
</label>
|
||
|
||
<label class="range-row" for="regressions">
|
||
<span><strong>回归失败</strong><small>基线成功，候选失败</small></span>
|
||
<input id="regressions" name="regressions" type="range" min="0" max="20" value="1">
|
||
<output for="regressions">1</output>
|
||
</label>
|
||
|
||
<label class="range-row" for="both-pass">
|
||
<span><strong>双方成功</strong><small>两种版本都通过</small></span>
|
||
<input id="both-pass" name="bothPass" type="range" min="0" max="30" value="18">
|
||
<output for="both-pass">18</output>
|
||
</label>
|
||
|
||
<label class="range-row" for="both-fail">
|
||
<span><strong>双方失败</strong><small>两种版本都未通过</small></span>
|
||
<input id="both-fail" name="bothFail" type="range" min="0" max="30" value="5">
|
||
<output for="both-fail">5</output>
|
||
</label>
|
||
</fieldset>
|
||
|
||
<fieldset class="gate-controls">
|
||
<legend>硬门禁与效率</legend>
|
||
<label>
|
||
<span>安全违规</span>
|
||
<input id="safety" name="safety" type="number" min="0" max="99" value="0">
|
||
</label>
|
||
<label>
|
||
<span>新增假完成</span>
|
||
<input id="false-completion" name="falseCompletion" type="number" min="0" max="99" value="0">
|
||
</label>
|
||
<label class="latency-control" for="latency">
|
||
<span>候选延迟变化</span>
|
||
<input id="latency" name="latency" type="range" min="-50" max="50" value="-12">
|
||
<output for="latency">-12%</output>
|
||
</label>
|
||
</fieldset>
|
||
</form>
|
||
|
||
<div class="decision-output" aria-live="polite">
|
||
<div class="decision-status">
|
||
<span class="status-dot" aria-hidden="true"></span>
|
||
<div>
|
||
<small>机械建议</small>
|
||
<h3 id="decision-label">发布候选</h3>
|
||
</div>
|
||
<span class="sample-count" id="sample-count">30 次配对</span>
|
||
</div>
|
||
|
||
<p class="decision-reason" id="decision-reason">
|
||
新增成功多于回归，且没有安全或假完成阻断项。进入轨迹复核，而不是直接发布。
|
||
</p>
|
||
|
||
<div class="outcome-matrix" aria-label="基线与候选结果矩阵">
|
||
<div class="matrix-corner">基线 \ 候选</div>
|
||
<div class="matrix-axis">失败</div>
|
||
<div class="matrix-axis">成功</div>
|
||
<div class="matrix-axis matrix-row-label">失败</div>
|
||
<div class="matrix-cell neutral"><span>双方失败</span><strong id="matrix-both-fail">5</strong></div>
|
||
<div class="matrix-cell positive"><span>新增成功</span><strong id="matrix-gain">6</strong></div>
|
||
<div class="matrix-axis matrix-row-label">成功</div>
|
||
<div class="matrix-cell negative"><span>回归失败</span><strong id="matrix-regression">1</strong></div>
|
||
<div class="matrix-cell stable"><span>双方成功</span><strong id="matrix-both-pass">18</strong></div>
|
||
</div>
|
||
|
||
<div class="rate-comparison">
|
||
<div>
|
||
<span>基线成功率</span>
|
||
<strong id="baseline-rate">63.3%</strong>
|
||
<small id="baseline-ci">95% CI 45.5%–78.1%</small>
|
||
</div>
|
||
<div>
|
||
<span>候选成功率</span>
|
||
<strong id="variant-rate">80.0%</strong>
|
||
<small id="variant-ci">95% CI 62.7%–90.5%</small>
|
||
</div>
|
||
<div>
|
||
<span>配对差异</span>
|
||
<strong id="net-gain">+5</strong>
|
||
<small id="mcnemar-p">McNemar p = 0.125</small>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="blocker-line" id="blocker-line">
|
||
<strong>硬门禁通过</strong>
|
||
<span>安全违规 0 · 新增假完成 0</span>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="layer-section" id="layers" data-section>
|
||
<div class="section-heading">
|
||
<div>
|
||
<div class="section-number">03 / 评测层次</div>
|
||
<h2>高层成功不能掩盖底层回归</h2>
|
||
</div>
|
||
<p>真实模型评测建立在单测、隔离和伪模型链路之上，不替代它们。</p>
|
||
</div>
|
||
|
||
<div class="layer-tabs" role="tablist" aria-label="评测层次">
|
||
<button class="is-active" role="tab" aria-selected="true" data-layer="0">L0 单元</button>
|
||
<button role="tab" aria-selected="false" data-layer="1">L1 Docker</button>
|
||
<button role="tab" aria-selected="false" data-layer="2">L2 E2E</button>
|
||
<button role="tab" aria-selected="false" data-layer="3">L3 行为</button>
|
||
<button role="tab" aria-selected="false" data-layer="4">L4 压力</button>
|
||
<button role="tab" aria-selected="false" data-layer="5">L5 生产</button>
|
||
</div>
|
||
|
||
<article class="layer-detail" id="layer-detail" tabindex="0">
|
||
<div class="layer-code" id="layer-code">L0</div>
|
||
<div>
|
||
<p class="layer-purpose" id="layer-purpose">单元与策略门禁</p>
|
||
<h3 id="layer-title">先证明规则本身没有坏</h3>
|
||
<p id="layer-description">
|
||
检查参数规范化、完成证据、权限和事件顺序。每次提交运行，不调用真实模型。
|
||
</p>
|
||
</div>
|
||
<dl>
|
||
<div><dt>模型</dt><dd id="layer-model">不调用</dd></div>
|
||
<div><dt>节奏</dt><dd id="layer-cadence">每次提交</dd></div>
|
||
<div><dt>失败动作</dt><dd id="layer-action">阻止进入高层评测</dd></div>
|
||
</dl>
|
||
</article>
|
||
</section>
|
||
|
||
<section class="task-section" id="tasks" data-section>
|
||
<div class="section-heading">
|
||
<div>
|
||
<div class="section-number">04 / 任务契约</div>
|
||
<h2>任务描述不是真值</h2>
|
||
</div>
|
||
<p>最终状态、命令结果和副作用 Validator 才决定任务是否真的完成。</p>
|
||
</div>
|
||
|
||
<div class="task-toolbar">
|
||
<label class="search-field" for="task-search">
|
||
<span class="visually-hidden">搜索任务</span>
|
||
<input id="task-search" type="search" placeholder="搜索任务、能力或场景">
|
||
</label>
|
||
<div class="segmented" role="group" aria-label="任务状态筛选">
|
||
<button class="is-active" data-filter="all">全部 <span>10</span></button>
|
||
<button data-filter="current">当前可跑 <span>2</span></button>
|
||
<button data-filter="needed">需补能力 <span>8</span></button>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="task-list-header" aria-hidden="true">
|
||
<span>任务</span>
|
||
<span>能力家族</span>
|
||
<span>Runner 状态</span>
|
||
<span></span>
|
||
</div>
|
||
<div class="task-list" id="task-list"></div>
|
||
<p class="empty-state" id="task-empty" hidden>没有匹配的任务。</p>
|
||
</section>
|
||
|
||
<section class="artifact-section" id="artifacts" data-section>
|
||
<div class="section-heading">
|
||
<div>
|
||
<div class="section-number">05 / 相关产物</div>
|
||
<h2>结论、契约、工具与边界均可追溯</h2>
|
||
</div>
|
||
<p>网页是阅读入口，Git 仓库仍是事实源。</p>
|
||
</div>
|
||
|
||
<div class="artifact-list">
|
||
<a href="https://git.k1412.top/wuyang/agent/src/branch/main/research/evaluation/k1412-agent-evaluation-v1.md" target="_blank" rel="noreferrer">
|
||
<span class="artifact-index">A1</span>
|
||
<span><strong>中文评估方案</strong><small>问题、任务、层次、判分、指标、协议和发布门禁</small></span>
|
||
<em>设计契约</em>
|
||
</a>
|
||
<a href="https://git.k1412.top/wuyang/agent/src/branch/main/data/evaluation/k1412-agent-eval-task-contracts-v1.json" target="_blank" rel="noreferrer">
|
||
<span class="artifact-index">A2</span>
|
||
<span><strong>10 个任务契约</strong><small>机器可读的目标、完成条件、禁区、预算和 Runner 缺口</small></span>
|
||
<em>JSON</em>
|
||
</a>
|
||
<a href="https://git.k1412.top/wuyang/agent/src/branch/main/tools/evaluation/score_agent_runs.py" target="_blank" rel="noreferrer">
|
||
<span class="artifact-index">A3</span>
|
||
<span><strong>运行结果评分器</strong><small>汇总成功、假完成、回归、区间、配对检验、延迟与 Token</small></span>
|
||
<em>Python</em>
|
||
</a>
|
||
<a href="https://git.k1412.top/wuyang/agent/src/branch/main/experiments/agent-evaluation/2026-07-27-k1412-agent-evaluation-design.md" target="_blank" rel="noreferrer">
|
||
<span class="artifact-index">A4</span>
|
||
<span><strong>实验记录</strong><small>已经确认的事实、设计假设、验证结果和不可外推边界</small></span>
|
||
<em>记录</em>
|
||
</a>
|
||
<a href="https://git.k1412.top/wuyang/agent/src/branch/main/papers/corpus-summary-2026-07-27.md" target="_blank" rel="noreferrer">
|
||
<span class="artifact-index">A5</span>
|
||
<span><strong>论文增量摘要</strong><small>249 个候选、157 个高相关、151 篇新增及评测研究信号</small></span>
|
||
<em>语料</em>
|
||
</a>
|
||
<a href="https://lab.k1412.top/" target="_blank" rel="noreferrer">
|
||
<span class="artifact-index">A6</span>
|
||
<span><strong>Agent 论文库</strong><small>1125 篇论文的搜索、主题探索和原始条目入口</small></span>
|
||
<em>在线</em>
|
||
</a>
|
||
</div>
|
||
|
||
<div class="command-line">
|
||
<code>python3 tools/evaluation/score_agent_runs.py results.jsonl --baseline v3 --variant v4 --format markdown</code>
|
||
<button id="copy-command" type="button">复制命令</button>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="implementation-section" id="implementation">
|
||
<div class="section-number">06 / 下一步</div>
|
||
<div class="implementation-layout">
|
||
<div>
|
||
<p class="eyebrow">Implementation order</p>
|
||
<h2>先让两项任务真实可判，再扩大任务集。</h2>
|
||
</div>
|
||
<ol>
|
||
<li><span>01</span><p><strong>抽出 LiveEvalRunner</strong>从现有单次脚本提取可复用运行器。</p></li>
|
||
<li><span>02</span><p><strong>接入独立 Validator</strong>先覆盖脚本报告与数据分析两个任务。</p></li>
|
||
<li><span>03</span><p><strong>导出 Result JSONL</strong>固定模型、Prompt、工具、镜像、预算与版本。</p></li>
|
||
<li><span>04</span><p><strong>运行配对 Smoke</strong>确定性层全部通过后才调用真实模型。</p></li>
|
||
<li><span>05</span><p><strong>复核回归轨迹</strong>检查 gains、regressions、both-fail 与假完成。</p></li>
|
||
</ol>
|
||
</div>
|
||
</section>
|
||
</main>
|
||
|
||
<footer>
|
||
<span>K1412 Agent Evaluation</span>
|
||
<span>方案基线 <code>df475e8</code></span>
|
||
<span>数据更新 2026.07.27</span>
|
||
</footer>
|
||
</body>
|
||
</html>
|