feat: add evaluation and safety chapter
This commit is contained in:
wuyang
@@ -0,0 +1,642 @@
|
||||
---
|
||||
const protocolPresets = [
|
||||
["greedy", "一次 Greedy", "1 次、无搜索", 1, 1],
|
||||
["sample", "Mean pass@1", "多次采样取平均", 64, 1],
|
||||
["search", "Pass@k", "有 verifier 的搜索", 64, 8],
|
||||
["reliable", "Pass^k", "连续成功可靠性", 5, 5],
|
||||
];
|
||||
---
|
||||
|
||||
<figure class="eval-lab" data-evaluation-lab>
|
||||
<div class="eval-lab-head">
|
||||
<div>
|
||||
<p>INTERACTIVE / MEASUREMENT WORKBENCH</p>
|
||||
<h3>把榜单拆回四台测量仪</h3>
|
||||
</div>
|
||||
<p>
|
||||
所有输出都是公开公式上的教学模型，用来观察协议变量怎样改变结论；它们不是任何真实模型的复跑成绩，
|
||||
也不把相关性写成因果。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="eval-tabs" role="tablist" aria-label="选择评测实验">
|
||||
<button type="button" role="tab" data-eval-tab="metric" aria-selected="true">
|
||||
<span>01</span><b>指标与校准</b><small>PPL · pass@k · abstain</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-eval-tab="judge" aria-selected="false" tabindex="-1">
|
||||
<span>02</span><b>Judge 与 Arena</b><small>order · length · uncertainty</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-eval-tab="contamination" aria-selected="false" tabindex="-1">
|
||||
<span>03</span><b>污染与动态题</b><small>leak · detect · refresh</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-eval-tab="system" aria-selected="false" tabindex="-1">
|
||||
<span>04</span><b>系统与安全边界</b><small>harness · budget · threat</small>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<section class="eval-view" data-eval-panel="metric">
|
||||
<div class="eval-panel-intro">
|
||||
<div><span>WORKBENCH 01 / METRIC</span><h4>同一个“80%”，可能回答四个不同问题</h4></div>
|
||||
<p>先看一次成功率，再分别计算“至少一次”“全部成功”和选择性回答；同时观察 tokenizer 对 PPL 的影响。</p>
|
||||
</div>
|
||||
|
||||
<div class="protocol-presets" role="group" aria-label="选择采样协议">
|
||||
{protocolPresets.map(([id, name, note], index) => (
|
||||
<button type="button" data-protocol-preset={id} class={index === 0 ? "active" : ""}>
|
||||
<b>{name}</b><small>{note}</small>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="eval-controls">
|
||||
<label>
|
||||
<span>单次成功率 <output data-metric-p-label>80%</output></span>
|
||||
<input data-metric-p type="range" min="5" max="99" value="80" />
|
||||
</label>
|
||||
<label>
|
||||
<span>采样池 n</span>
|
||||
<input data-metric-n type="number" min="1" max="256" value="1" />
|
||||
</label>
|
||||
<label>
|
||||
<span>允许尝试 k</span>
|
||||
<input data-metric-k type="number" min="1" max="64" value="1" />
|
||||
</label>
|
||||
<label>
|
||||
<span>平均置信度 <output data-confidence-label>88%</output></span>
|
||||
<input data-confidence type="range" min="5" max="99" value="88" />
|
||||
</label>
|
||||
<label>
|
||||
<span>拒答阈值 <output data-threshold-label>60%</output></span>
|
||||
<input data-threshold type="range" min="0" max="95" value="60" />
|
||||
</label>
|
||||
<label>
|
||||
<span>每字节 NLL</span>
|
||||
<input data-byte-nll type="number" min=".01" max="8" step=".01" value=".82" />
|
||||
</label>
|
||||
<label>
|
||||
<span>平均 bytes / token</span>
|
||||
<select data-token-bytes>
|
||||
<option value="2">2.0（细粒度）</option>
|
||||
<option value="3.4" selected>3.4（中等）</option>
|
||||
<option value="5.5">5.5（粗粒度）</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="eval-formula">
|
||||
<span>THREE DIFFERENT QUESTIONS</span>
|
||||
<code>at least one = 1 − (1 − p)<sup>k</sup> · all succeed = p<sup>k</sup> · ECE<sub>toy</sub> = |confidence − accuracy|</code>
|
||||
<p>pass@k 的严格无偏估计还需要每题 n 个样本中通过数 c；这里用独立同分布近似建立方向直觉。</p>
|
||||
</div>
|
||||
|
||||
<div class="eval-metrics five">
|
||||
<article><span>ONE TRY</span><b data-one-try>80.0%</b><p>一次随机尝试成功率</p></article>
|
||||
<article><span>AT LEAST ONE / k</span><b data-at-least>80.0%</b><p>有 verifier 时的搜索价值</p></article>
|
||||
<article><span>ALL k SUCCEED</span><b data-all-succeed>80.0%</b><p>连续任务的可靠性压力</p></article>
|
||||
<article><span>TOY ECE</span><b data-toy-ece>8.0 pp</b><p>单桶校准差，仅作直觉</p></article>
|
||||
<article class="dark"><span>SELECTIVE VIEW</span><b data-selective>80% / 100%</b><p data-selective-note>准确率 / 覆盖率</p></article>
|
||||
</div>
|
||||
|
||||
<div class="ppl-compare">
|
||||
<div>
|
||||
<span>SAME BYTE LOSS</span><b data-bpb>1.18 bits / byte</b>
|
||||
<p>按原始字节归一，跨 tokenizer 的单位更稳定。</p>
|
||||
</div>
|
||||
<div>
|
||||
<span>TOKENIZER A / PPL</span><b data-ppl-a>16.3</b>
|
||||
<p>细粒度 token：每 token 承载的字节更少。</p>
|
||||
</div>
|
||||
<div>
|
||||
<span>TOKENIZER B / PPL</span><b data-ppl-b>91.0</b>
|
||||
<p>粗粒度 token：即使 byte loss 相同，PPL 也会变。</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="boundary-note">
|
||||
<b>结论边界</b>
|
||||
<p data-metric-explain>一次成功率和多次搜索成功率目前相同，因为 k=1；把 k 调大后，搜索分会升，连续可靠性会降。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="eval-view" data-eval-panel="judge" hidden>
|
||||
<div class="eval-panel-intro">
|
||||
<div><span>WORKBENCH 02 / JUDGE</span><h4>Judge 不是尺子，它也是一个会偏的模型</h4></div>
|
||||
<p>用可见混杂做压力测试：回答顺序、长度差、自我偏好、Judge 能力与票数怎样改变胜率和不确定性。</p>
|
||||
</div>
|
||||
|
||||
<div class="judge-arena">
|
||||
<article class="answer-card answer-a">
|
||||
<span>ANSWER A</span>
|
||||
<h5>短而直接</h5>
|
||||
<p>先给结论，再列证据边界；不重复问题。</p>
|
||||
</article>
|
||||
<div aria-hidden="true">VS</div>
|
||||
<article class="answer-card answer-b">
|
||||
<span>ANSWER B</span>
|
||||
<h5>长而铺陈</h5>
|
||||
<p>提供更多背景、分点、例子与总结；可能更完整，也可能只是更长。</p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="eval-controls">
|
||||
<label>
|
||||
<span>A 的真实任务质量 <output data-quality-label>54%</output></span>
|
||||
<input data-quality type="range" min="10" max="90" value="54" />
|
||||
</label>
|
||||
<label>
|
||||
<span>B 比 A 长</span>
|
||||
<select data-length-gap>
|
||||
<option value="-40">短 40%</option>
|
||||
<option value="0">一样长</option>
|
||||
<option value="40" selected>长 40%</option>
|
||||
<option value="100">长 100%</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>长度偏好 <output data-length-bias-label>18%</output></span>
|
||||
<input data-length-bias type="range" min="0" max="40" value="18" />
|
||||
</label>
|
||||
<label>
|
||||
<span>位置偏好 <output data-position-label>8%</output></span>
|
||||
<input data-position-bias type="range" min="-30" max="30" value="8" />
|
||||
</label>
|
||||
<label>
|
||||
<span>展示顺序</span>
|
||||
<select data-order>
|
||||
<option value="a-first" selected>A 在前</option>
|
||||
<option value="b-first">B 在前</option>
|
||||
<option value="randomized">随机交换并平均</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>Judge 可靠度 <output data-judge-skill-label>82%</output></span>
|
||||
<input data-judge-skill type="range" min="50" max="99" value="82" />
|
||||
</label>
|
||||
<label>
|
||||
<span>自我偏好 <output data-self-label>5%</output></span>
|
||||
<input data-self-bias type="range" min="0" max="25" value="5" />
|
||||
</label>
|
||||
<label>
|
||||
<span>票数</span>
|
||||
<select data-votes>
|
||||
<option value="25">25</option>
|
||||
<option value="100" selected>100</option>
|
||||
<option value="1000">1,000</option>
|
||||
<option value="10000">10,000</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="eval-metrics four">
|
||||
<article><span>RAW A WIN</span><b data-raw-win>49.6%</b><p>混入顺序、长度、自偏后的观察值</p></article>
|
||||
<article><span>LENGTH-CONTROLLED</span><b data-adjusted-win>56.8%</b><p>移除教学模型中的长度项</p></article>
|
||||
<article><span>ORDER FLIP</span><b data-order-flip>16.0 pp</b><p>交换先后可造成的差</p></article>
|
||||
<article class="dark"><span>95% INTERVAL</span><b data-judge-ci>39.8–59.4%</b><p>二项近似；真实 Arena 图更复杂</p></article>
|
||||
</div>
|
||||
|
||||
<div class="rank-rail">
|
||||
<div><span>0%</span><span>50%</span><span>100%</span></div>
|
||||
<div class="rank-track"><i data-rank-raw style="width:49.6%"></i><b data-rank-adjusted style="left:56.8%"></b></div>
|
||||
<p><i></i> 观察胜率 <b></b> 长度控制后的估计</p>
|
||||
</div>
|
||||
|
||||
<div class="boundary-note">
|
||||
<b>元评测提醒</b>
|
||||
<p data-judge-explain>当前协议里，较长的 B 获得长度红利；交换顺序或控制长度，排名可能翻转。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="eval-view" data-eval-panel="contamination" hidden>
|
||||
<div class="eval-panel-intro">
|
||||
<div><span>WORKBENCH 03 / DATA</span><h4>“没搜到 n-gram”不能推出“没有污染”</h4></div>
|
||||
<p>把五层泄漏、检测能力、题目刷新与历史锚点分开，观察新鲜度和纵向可比性的冲突。</p>
|
||||
</div>
|
||||
|
||||
<div class="leak-stack">
|
||||
<label><input data-leak="verbatim" type="checkbox" checked /><span>01</span><b>原文泄漏</b><small>测试题文本进入训练</small></label>
|
||||
<label><input data-leak="answer" type="checkbox" checked /><span>02</span><b>答案泄漏</b><small>题—答、解析或单测进入训练</small></label>
|
||||
<label><input data-leak="format" type="checkbox" /><span>03</span><b>格式泄漏</b><small>模板、选项或 verbalizer 重复</small></label>
|
||||
<label><input data-leak="semantic" type="checkbox" /><span>04</span><b>语义泄漏</b><small>改写题、同事实或同算法</small></label>
|
||||
<label><input data-leak="temporal" type="checkbox" /><span>05</span><b>时间泄漏</b><small>发布后进入 SFT / RL / 合成数据</small></label>
|
||||
</div>
|
||||
|
||||
<div class="eval-controls">
|
||||
<label>
|
||||
<span>受影响题目 <output data-leak-rate-label>18%</output></span>
|
||||
<input data-leak-rate type="range" min="0" max="80" value="18" />
|
||||
</label>
|
||||
<label>
|
||||
<span>污染带来的表观增益 <output data-leak-boost-label>25 pp</output></span>
|
||||
<input data-leak-boost type="range" min="0" max="60" value="25" />
|
||||
</label>
|
||||
<label>
|
||||
<span>n-gram 检测阈值</span>
|
||||
<select data-ngram>
|
||||
<option value="8">8-gram（宽松）</option>
|
||||
<option value="13" selected>13-gram（中等）</option>
|
||||
<option value="30">30-gram（严格匹配）</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>每月刷新题目 <output data-refresh-label>30%</output></span>
|
||||
<input data-refresh type="range" min="0" max="100" value="30" />
|
||||
</label>
|
||||
<label>
|
||||
<span>固定锚点占比 <output data-anchor-label>40%</output></span>
|
||||
<input data-anchor type="range" min="0" max="100" value="40" />
|
||||
</label>
|
||||
<label>
|
||||
<span>标签错误率 <output data-label-error-label>4%</output></span>
|
||||
<input data-label-error type="range" min="0" max="20" value="4" />
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="eval-metrics five">
|
||||
<article><span>TRUE CLEAN</span><b data-true-clean>82%</b><p>教学设定中未受泄漏影响的题</p></article>
|
||||
<article><span>N-GRAM DETECTED</span><b data-detected>23%</b><p>五层污染中可被当前扫描发现的比例</p></article>
|
||||
<article><span>HIDDEN LEAK</span><b data-hidden-leak>13.9%</b><p>存在但未被当前方法捕获</p></article>
|
||||
<article><span>FRESHNESS</span><b data-freshness>51%</b><p>刷新速度与未泄漏比例的教学合成</p></article>
|
||||
<article class="dark"><span>COMPARABILITY</span><b data-comparability>70%</b><p>固定锚点越多，跨月纵向比较越稳</p></article>
|
||||
</div>
|
||||
|
||||
<div class="freshness-matrix">
|
||||
<div><span>公开静态集</span><b>可复跑强</b><small>污染与饱和风险随时间上升</small></div>
|
||||
<div><span>纯动态集</span><b>新鲜度强</b><small>月份间难度可能漂移</small></div>
|
||||
<div class="recommended"><span>版本化动态 + 锚点</span><b>同时留两本账</b><small>报告版本、刷新率和 anchor 表现</small></div>
|
||||
</div>
|
||||
|
||||
<div class="boundary-note">
|
||||
<b>检测边界</b>
|
||||
<p data-contamination-explain>当前只启用原文与答案泄漏；n-gram 对原文较敏感，对答案代码与语义改写仍不充分。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="eval-view" data-eval-panel="system" hidden>
|
||||
<div class="eval-panel-intro">
|
||||
<div><span>WORKBENCH 04 / SYSTEM</span><h4>Agent 分数属于“模型 × Harness × 环境”，安全分也属于威胁模型</h4></div>
|
||||
<p>把模型、脚手架、工具、预算、Verifier、产品安全壳和攻击强度放到同一张成本—能力—风险图。</p>
|
||||
</div>
|
||||
|
||||
<div class="system-equation">
|
||||
<div><span>MODEL</span><b>策略与表征</b></div><i>×</i>
|
||||
<div><span>HARNESS</span><b>循环与上下文</b></div><i>×</i>
|
||||
<div><span>TOOLS</span><b>权限与反馈</b></div><i>×</i>
|
||||
<div><span>ENV</span><b>版本与状态</b></div><i>×</i>
|
||||
<div><span>VERIFIER</span><b>成功定义</b></div>
|
||||
</div>
|
||||
|
||||
<div class="eval-controls">
|
||||
<label>
|
||||
<span>模型一次任务能力 <output data-model-label>58%</output></span>
|
||||
<input data-model type="range" min="5" max="95" value="58" />
|
||||
</label>
|
||||
<label>
|
||||
<span>Harness 增益 <output data-harness-label>12 pp</output></span>
|
||||
<input data-harness type="range" min="-20" max="35" value="12" />
|
||||
</label>
|
||||
<label>
|
||||
<span>工具可用性 <output data-tools-label>92%</output></span>
|
||||
<input data-tools type="range" min="20" max="100" value="92" />
|
||||
</label>
|
||||
<label>
|
||||
<span>最大重试</span>
|
||||
<select data-retries>
|
||||
<option value="1">1 次</option>
|
||||
<option value="4" selected>4 次</option>
|
||||
<option value="16">16 次</option>
|
||||
<option value="64">64 次</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>单次 Token 预算</span>
|
||||
<select data-budget>
|
||||
<option value="4">4K</option>
|
||||
<option value="32" selected>32K</option>
|
||||
<option value="128">128K</option>
|
||||
<option value="512">512K</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>Verifier 覆盖 <output data-verifier-label>80%</output></span>
|
||||
<input data-verifier type="range" min="20" max="100" value="80" />
|
||||
</label>
|
||||
<label>
|
||||
<span>产品安全壳强度 <output data-wrapper-label>72%</output></span>
|
||||
<input data-wrapper type="range" min="0" max="100" value="72" />
|
||||
</label>
|
||||
<label>
|
||||
<span>攻击者预算 <output data-attack-label>45%</output></span>
|
||||
<input data-attack type="range" min="0" max="100" value="45" />
|
||||
</label>
|
||||
<label>
|
||||
<span>无害请求敏感度 <output data-benign-label>24%</output></span>
|
||||
<input data-benign type="range" min="0" max="100" value="24" />
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="eval-metrics five">
|
||||
<article><span>MODEL ONLY</span><b data-model-only>58.0%</b><p>不含 Harness / 工具 / 多试</p></article>
|
||||
<article><span>SYSTEM SUCCESS</span><b data-system-success>92.9%</b><p>教学独立近似，不能当真实复跑</p></article>
|
||||
<article><span>COST INDEX</span><b data-cost>128</b><p>重试 × Token 预算的相对成本</p></article>
|
||||
<article><span>UNSAFE COMPLIANCE</span><b data-unsafe>9.2%</b><p>在当前攻击与安全壳设定下</p></article>
|
||||
<article class="dark"><span>OVER-REFUSAL</span><b data-overrefusal>17.3%</b><p>安全壳也会损失无害可用性</p></article>
|
||||
</div>
|
||||
|
||||
<div class="frontier-plot" aria-label="能力、成本与风险教学图">
|
||||
<div class="axis y">能力 ↑</div>
|
||||
<div class="axis x">成本 →</div>
|
||||
<div class="risk-zone">高风险区</div>
|
||||
<i data-frontier-point style="left:58%;bottom:70%"><span>当前系统</span></i>
|
||||
</div>
|
||||
|
||||
<div class="boundary-note">
|
||||
<b>归因规则</b>
|
||||
<p data-system-explain>四次尝试和 32K 预算把系统成功率推高，但应同时披露成本；安全壳压低危险服从，也带来过拒。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<figcaption>
|
||||
<b>怎么使用：</b>先改变一个变量，口述“我究竟换了模型、测量协议、系统预算还是威胁模型”；
|
||||
如果一句比较没有这些字段，就先把结论降级为“在该公开设置下的观察值”。
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
<script>
|
||||
const lab = document.querySelector<HTMLElement>("[data-evaluation-lab]");
|
||||
|
||||
if (lab && lab.dataset.bound !== "true") {
|
||||
lab.dataset.bound = "true";
|
||||
const tabs = [...lab.querySelectorAll<HTMLButtonElement>("[data-eval-tab]")];
|
||||
const panels = [...lab.querySelectorAll<HTMLElement>("[data-eval-panel]")];
|
||||
|
||||
const activate = (id: string) => {
|
||||
tabs.forEach((tab) => {
|
||||
const active = tab.dataset.evalTab === id;
|
||||
tab.setAttribute("aria-selected", String(active));
|
||||
tab.tabIndex = active ? 0 : -1;
|
||||
});
|
||||
panels.forEach((panel) => { panel.hidden = panel.dataset.evalPanel !== id; });
|
||||
};
|
||||
|
||||
tabs.forEach((tab, index) => {
|
||||
tab.addEventListener("click", () => activate(tab.dataset.evalTab ?? "metric"));
|
||||
tab.addEventListener("keydown", (event) => {
|
||||
if (!["ArrowLeft", "ArrowRight", "Home", "End"].includes(event.key)) return;
|
||||
event.preventDefault();
|
||||
let next = index;
|
||||
if (event.key === "ArrowRight") next = (index + 1) % tabs.length;
|
||||
if (event.key === "ArrowLeft") next = (index - 1 + tabs.length) % tabs.length;
|
||||
if (event.key === "Home") next = 0;
|
||||
if (event.key === "End") next = tabs.length - 1;
|
||||
tabs[next].focus();
|
||||
activate(tabs[next].dataset.evalTab ?? "metric");
|
||||
});
|
||||
});
|
||||
|
||||
const q = <T extends HTMLElement>(selector: string) => lab.querySelector<T>(selector);
|
||||
const value = (selector: string) => Number((q<HTMLInputElement | HTMLSelectElement>(selector))?.value ?? 0);
|
||||
const set = (selector: string, text: string) => {
|
||||
const node = q<HTMLElement>(selector);
|
||||
if (node) node.textContent = text;
|
||||
};
|
||||
const clamp = (number: number, min = 0, max = 1) => Math.max(min, Math.min(max, number));
|
||||
|
||||
const updateMetric = () => {
|
||||
const p = value("[data-metric-p]") / 100;
|
||||
const n = Math.max(1, value("[data-metric-n]"));
|
||||
const k = Math.max(1, Math.min(value("[data-metric-k]"), n));
|
||||
const confidence = value("[data-confidence]") / 100;
|
||||
const threshold = value("[data-threshold]") / 100;
|
||||
const byteNll = value("[data-byte-nll]");
|
||||
const tokenBytes = value("[data-token-bytes]");
|
||||
const atLeast = 1 - Math.pow(1 - p, k);
|
||||
const all = Math.pow(p, k);
|
||||
const ece = Math.abs(confidence - p);
|
||||
const coverage = clamp(1 - threshold * .72);
|
||||
const selective = clamp(p + threshold * (1 - p) * .42);
|
||||
const bpb = byteNll / Math.log(2);
|
||||
const pplA = Math.exp(byteNll * 2);
|
||||
const pplB = Math.exp(byteNll * tokenBytes);
|
||||
|
||||
set("[data-metric-p-label]", `${Math.round(p * 100)}%`);
|
||||
set("[data-confidence-label]", `${Math.round(confidence * 100)}%`);
|
||||
set("[data-threshold-label]", `${Math.round(threshold * 100)}%`);
|
||||
set("[data-one-try]", `${(p * 100).toFixed(1)}%`);
|
||||
set("[data-at-least]", `${(atLeast * 100).toFixed(1)}%`);
|
||||
set("[data-all-succeed]", `${(all * 100).toFixed(1)}%`);
|
||||
set("[data-toy-ece]", `${(ece * 100).toFixed(1)} pp`);
|
||||
set("[data-selective]", `${Math.round(selective * 100)}% / ${Math.round(coverage * 100)}%`);
|
||||
set("[data-bpb]", `${bpb.toFixed(2)} bits / byte`);
|
||||
set("[data-ppl-a]", pplA.toFixed(1));
|
||||
set("[data-ppl-b]", pplB.toFixed(1));
|
||||
set("[data-metric-explain]", k === 1
|
||||
? "k=1 时，一次成功、至少一次成功和全部成功是同一个问题；先把 k 调大。"
|
||||
: `把 k 提到 ${k} 后，“至少一次”升到 ${(atLeast * 100).toFixed(1)}%，但“连续全部成功”降到 ${(all * 100).toFixed(1)}%。这就是搜索能力和可靠性不能混报。`);
|
||||
};
|
||||
|
||||
lab.querySelectorAll("[data-protocol-preset]").forEach((node) => {
|
||||
node.addEventListener("click", () => {
|
||||
lab.querySelectorAll("[data-protocol-preset]").forEach((item) => item.classList.remove("active"));
|
||||
node.classList.add("active");
|
||||
const id = (node as HTMLElement).dataset.protocolPreset;
|
||||
const settings: Record<string, [number, number]> = {
|
||||
greedy: [1, 1], sample: [64, 1], search: [64, 8], reliable: [5, 5],
|
||||
};
|
||||
const [n, k] = settings[id ?? "greedy"];
|
||||
const nInput = q<HTMLInputElement>("[data-metric-n]");
|
||||
const kInput = q<HTMLInputElement>("[data-metric-k]");
|
||||
if (nInput) nInput.value = String(n);
|
||||
if (kInput) kInput.value = String(k);
|
||||
updateMetric();
|
||||
});
|
||||
});
|
||||
lab.querySelectorAll("[data-eval-panel='metric'] input,[data-eval-panel='metric'] select")
|
||||
.forEach((node) => node.addEventListener("input", updateMetric));
|
||||
|
||||
const updateJudge = () => {
|
||||
const quality = value("[data-quality]") / 100;
|
||||
const gap = value("[data-length-gap]") / 100;
|
||||
const lengthBias = value("[data-length-bias]") / 100;
|
||||
const position = value("[data-position-bias]") / 100;
|
||||
const order = q<HTMLSelectElement>("[data-order]")?.value ?? "a-first";
|
||||
const skill = value("[data-judge-skill]") / 100;
|
||||
const selfBias = value("[data-self-bias]") / 100;
|
||||
const votes = Math.max(1, value("[data-votes]"));
|
||||
const orderTerm = order === "randomized" ? 0 : position * (order === "a-first" ? 1 : -1);
|
||||
const lengthTerm = -gap * lengthBias;
|
||||
const signal = (quality - .5) * (skill * 1.5) + orderTerm + lengthTerm + selfBias * .25;
|
||||
const raw = clamp(.5 + signal);
|
||||
const adjusted = clamp(raw - lengthTerm);
|
||||
const se = Math.sqrt(raw * (1 - raw) / votes);
|
||||
const low = clamp(raw - 1.96 * se);
|
||||
const high = clamp(raw + 1.96 * se);
|
||||
|
||||
set("[data-quality-label]", `${Math.round(quality * 100)}%`);
|
||||
set("[data-length-bias-label]", `${Math.round(lengthBias * 100)}%`);
|
||||
set("[data-position-label]", `${Math.round(position * 100)}%`);
|
||||
set("[data-judge-skill-label]", `${Math.round(skill * 100)}%`);
|
||||
set("[data-self-label]", `${Math.round(selfBias * 100)}%`);
|
||||
set("[data-raw-win]", `${(raw * 100).toFixed(1)}%`);
|
||||
set("[data-adjusted-win]", `${(adjusted * 100).toFixed(1)}%`);
|
||||
set("[data-order-flip]", `${(Math.abs(position) * 200).toFixed(1)} pp`);
|
||||
set("[data-judge-ci]", `${(low * 100).toFixed(1)}–${(high * 100).toFixed(1)}%`);
|
||||
const rawBar = q<HTMLElement>("[data-rank-raw]");
|
||||
const adjustedMark = q<HTMLElement>("[data-rank-adjusted]");
|
||||
if (rawBar) rawBar.style.width = `${raw * 100}%`;
|
||||
if (adjustedMark) adjustedMark.style.left = `${adjusted * 100}%`;
|
||||
set("[data-judge-explain]", order === "randomized"
|
||||
? `随机交换顺序已抵消教学模型里的位置项；控制长度后，A 的估计胜率从 ${(raw * 100).toFixed(1)}% 变为 ${(adjusted * 100).toFixed(1)}%。`
|
||||
: `当前展示顺序贡献 ${(orderTerm * 100).toFixed(1)} pp，长度项贡献 ${(lengthTerm * 100).toFixed(1)} pp；交换顺序或控制长度都可能改变结论。`);
|
||||
};
|
||||
lab.querySelectorAll("[data-eval-panel='judge'] input,[data-eval-panel='judge'] select")
|
||||
.forEach((node) => node.addEventListener("input", updateJudge));
|
||||
|
||||
const detectability: Record<string, number> = {
|
||||
verbatim: .9, answer: .46, format: .16, semantic: .05, temporal: .08,
|
||||
};
|
||||
const updateContamination = () => {
|
||||
const active = [...lab.querySelectorAll<HTMLInputElement>("[data-leak]:checked")]
|
||||
.map((input) => input.dataset.leak ?? "");
|
||||
const leakRate = value("[data-leak-rate]") / 100;
|
||||
const boost = value("[data-leak-boost]");
|
||||
const ngram = value("[data-ngram]");
|
||||
const refresh = value("[data-refresh]") / 100;
|
||||
const anchor = value("[data-anchor]") / 100;
|
||||
const labelError = value("[data-label-error]") / 100;
|
||||
const avgDetect = active.length
|
||||
? active.reduce((sum, id) => sum + (detectability[id] ?? 0), 0) / active.length
|
||||
: 0;
|
||||
const thresholdFactor = clamp(1.18 - ngram / 50, .4, 1.05);
|
||||
const detected = clamp(avgDetect * thresholdFactor);
|
||||
const hidden = leakRate * (1 - detected);
|
||||
const freshness = clamp((1 - leakRate) * .35 + refresh * .65 - labelError * .4);
|
||||
const comparability = clamp(anchor * .75 + (1 - refresh) * .25);
|
||||
|
||||
set("[data-leak-rate-label]", `${Math.round(leakRate * 100)}%`);
|
||||
set("[data-leak-boost-label]", `${Math.round(boost)} pp`);
|
||||
set("[data-refresh-label]", `${Math.round(refresh * 100)}%`);
|
||||
set("[data-anchor-label]", `${Math.round(anchor * 100)}%`);
|
||||
set("[data-label-error-label]", `${Math.round(labelError * 100)}%`);
|
||||
set("[data-true-clean]", `${Math.round((1 - leakRate) * 100)}%`);
|
||||
set("[data-detected]", `${Math.round(detected * 100)}%`);
|
||||
set("[data-hidden-leak]", `${(hidden * 100).toFixed(1)}%`);
|
||||
set("[data-freshness]", `${Math.round(freshness * 100)}%`);
|
||||
set("[data-comparability]", `${Math.round(comparability * 100)}%`);
|
||||
const names: Record<string, string> = {
|
||||
verbatim: "原文", answer: "答案", format: "格式", semantic: "语义", temporal: "时间",
|
||||
};
|
||||
set("[data-contamination-explain]", active.length
|
||||
? `已启用${active.map((id) => names[id]).join("、")}泄漏。当前 n-gram 扫描平均只能捕获约 ${Math.round(detected * 100)}%；“未命中”不能排除剩余层级。`
|
||||
: "尚未启用污染层；这只代表教学设定为零，不代表现实训练数据已经被证明干净。");
|
||||
};
|
||||
lab.querySelectorAll("[data-eval-panel='contamination'] input,[data-eval-panel='contamination'] select")
|
||||
.forEach((node) => node.addEventListener("input", updateContamination));
|
||||
|
||||
const updateSystem = () => {
|
||||
const model = value("[data-model]") / 100;
|
||||
const harness = value("[data-harness]") / 100;
|
||||
const tools = value("[data-tools]") / 100;
|
||||
const retries = Math.max(1, value("[data-retries]"));
|
||||
const budget = value("[data-budget]");
|
||||
const verifier = value("[data-verifier]") / 100;
|
||||
const wrapper = value("[data-wrapper]") / 100;
|
||||
const attack = value("[data-attack]") / 100;
|
||||
const benign = value("[data-benign]") / 100;
|
||||
const perTry = clamp((model + harness) * tools);
|
||||
const searched = 1 - Math.pow(1 - perTry, retries);
|
||||
const observable = clamp(searched * (.72 + verifier * .28));
|
||||
const unsafe = clamp((model * .35 + attack * .65) * Math.pow(1 - wrapper, 1.35));
|
||||
const overrefusal = clamp(wrapper * benign);
|
||||
const cost = retries * budget;
|
||||
|
||||
set("[data-model-label]", `${Math.round(model * 100)}%`);
|
||||
set("[data-harness-label]", `${Math.round(harness * 100)} pp`);
|
||||
set("[data-tools-label]", `${Math.round(tools * 100)}%`);
|
||||
set("[data-verifier-label]", `${Math.round(verifier * 100)}%`);
|
||||
set("[data-wrapper-label]", `${Math.round(wrapper * 100)}%`);
|
||||
set("[data-attack-label]", `${Math.round(attack * 100)}%`);
|
||||
set("[data-benign-label]", `${Math.round(benign * 100)}%`);
|
||||
set("[data-model-only]", `${(model * 100).toFixed(1)}%`);
|
||||
set("[data-system-success]", `${(observable * 100).toFixed(1)}%`);
|
||||
set("[data-cost]", String(cost));
|
||||
set("[data-unsafe]", `${(unsafe * 100).toFixed(1)}%`);
|
||||
set("[data-overrefusal]", `${(overrefusal * 100).toFixed(1)}%`);
|
||||
const point = q<HTMLElement>("[data-frontier-point]");
|
||||
if (point) {
|
||||
point.style.left = `${clamp(Math.log2(cost + 1) / 11) * 100}%`;
|
||||
point.style.bottom = `${observable * 80 + 5}%`;
|
||||
}
|
||||
set("[data-system-explain]", `${retries} 次尝试与 ${budget}K 单次预算把系统成功率估到 ${(observable * 100).toFixed(1)}%，相对成本为 ${cost}；安全壳同时留下 ${(unsafe * 100).toFixed(1)}% 危险服从与 ${(overrefusal * 100).toFixed(1)}% 过拒。`);
|
||||
};
|
||||
lab.querySelectorAll("[data-eval-panel='system'] input,[data-eval-panel='system'] select")
|
||||
.forEach((node) => node.addEventListener("input", updateSystem));
|
||||
|
||||
updateMetric();
|
||||
updateJudge();
|
||||
updateContamination();
|
||||
updateSystem();
|
||||
}
|
||||
</script>
|
||||
|
||||
<style>
|
||||
.eval-lab{margin:42px 0 0;border:1px solid rgba(201,139,85,.38);background:#edf0e9;color:#13242b;box-shadow:0 22px 70px rgba(4,18,22,.15);overflow:hidden}
|
||||
.eval-lab-head{display:grid;grid-template-columns:1.05fr .95fr;gap:28px;padding:30px;background:#0d252c;color:#eef0e8;border-bottom:1px solid rgba(255,255,255,.13)}
|
||||
.eval-lab-head p{margin:0;color:#aebdbb;font-size:.84rem;line-height:1.65}
|
||||
.eval-lab-head>div p{color:#d69357;font:700 .66rem/1.2 var(--font-mono);letter-spacing:.12em}
|
||||
.eval-lab-head h3{margin:8px 0 0;color:#fff;font-size:clamp(1.4rem,3vw,2.15rem);letter-spacing:-.04em}
|
||||
.eval-tabs{display:grid;grid-template-columns:repeat(4,1fr);background:#17323a;border-bottom:1px solid #385058}
|
||||
.eval-tabs button{min-width:0;overflow:hidden;padding:17px 16px;border:0;border-right:1px solid #385058;background:transparent;color:#a8b7b5;text-align:left;cursor:pointer}
|
||||
.eval-tabs button:last-child{border-right:0}
|
||||
.eval-tabs button[aria-selected=true]{background:#edf0e9;color:#13242b;box-shadow:inset 0 4px #d08b50}
|
||||
.eval-tabs span,.eval-tabs small{display:block;font:600 .58rem/1.3 var(--font-mono);letter-spacing:.08em}
|
||||
.eval-tabs b{display:block;margin:5px 0;font-size:.9rem}
|
||||
.eval-tabs small{overflow:hidden;opacity:.68;white-space:nowrap;text-overflow:ellipsis}
|
||||
.eval-view{padding:30px}
|
||||
.eval-panel-intro{display:grid;grid-template-columns:1fr 1fr;gap:28px;align-items:end;padding-bottom:22px;border-bottom:1px solid #bdc8c3}
|
||||
.eval-panel-intro span{font:700 .61rem/1 var(--font-mono);letter-spacing:.1em;color:#a75d2e}
|
||||
.eval-panel-intro h4{margin:7px 0 0;font-size:clamp(1.25rem,2.4vw,1.8rem);letter-spacing:-.035em}
|
||||
.eval-panel-intro p{margin:0;color:#53676a;font-size:.88rem;line-height:1.7}
|
||||
.protocol-presets{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:22px 0}
|
||||
.protocol-presets button{padding:13px;border:1px solid #b5c2be;background:#f7f7f1;color:#24383d;text-align:left;cursor:pointer}
|
||||
.protocol-presets button.active{border-color:#c6783f;background:#fff1e4;box-shadow:inset 3px 0 #c6783f}
|
||||
.protocol-presets b,.protocol-presets small{display:block}.protocol-presets b{font-size:.8rem}.protocol-presets small{margin-top:4px;color:#69797a;font-size:.68rem}
|
||||
.eval-controls{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:10px;margin:22px 0}
|
||||
.eval-controls label{display:flex;min-width:0;flex-direction:column;justify-content:space-between;gap:9px;padding:12px;border:1px solid #c0cac6;background:#f9faf5}
|
||||
.eval-controls label>span{min-height:2.1em;font:700 .66rem/1.45 var(--font-mono);letter-spacing:.025em;color:#485b5e}
|
||||
.eval-controls output{color:#a95d2d}
|
||||
.eval-controls input,.eval-controls select{width:100%;box-sizing:border-box;accent-color:#b96934}
|
||||
.eval-controls input[type=number],.eval-controls select{height:34px;padding:0 8px;border:1px solid #aebbb7;background:#fff;color:#17292e}
|
||||
.eval-formula{margin:20px 0;padding:17px 19px;border-left:4px solid #c57b43;background:#fff7ec}
|
||||
.eval-formula span{font:700 .6rem/1 var(--font-mono);letter-spacing:.1em;color:#a65c2c}
|
||||
.eval-formula code{display:block;margin:10px 0;color:#16333b;white-space:normal;font-size:.88rem}
|
||||
.eval-formula p{margin:0;color:#617172;font-size:.75rem}
|
||||
.eval-metrics{display:grid;gap:8px;margin:20px 0}.eval-metrics.five{grid-template-columns:repeat(5,1fr)}.eval-metrics.four{grid-template-columns:repeat(4,1fr)}
|
||||
.eval-metrics article{min-width:0;padding:16px;border:1px solid #bbc7c3;background:#f8f9f4}
|
||||
.eval-metrics span{font:700 .58rem/1 var(--font-mono);letter-spacing:.08em;color:#6c7d7d}
|
||||
.eval-metrics b{display:block;margin:9px 0;color:#1b3339;font-size:1.22rem;white-space:nowrap}
|
||||
.eval-metrics p{margin:0;color:#69797a;font-size:.68rem;line-height:1.45}
|
||||
.eval-metrics .dark{background:#173239;color:#fff;border-color:#173239}.eval-metrics .dark span,.eval-metrics .dark p{color:#b7c5c2}.eval-metrics .dark b{color:#f0b77e}
|
||||
.ppl-compare{display:grid;grid-template-columns:repeat(3,1fr);gap:8px}
|
||||
.ppl-compare>div{padding:16px;background:#dfe6e1}.ppl-compare span{font:700 .58rem/1 var(--font-mono);color:#607271}.ppl-compare b{display:block;margin:7px 0;font-size:1rem}.ppl-compare p{margin:0;color:#647576;font-size:.7rem;line-height:1.45}
|
||||
.boundary-note{display:grid;grid-template-columns:130px 1fr;gap:18px;margin-top:20px;padding:16px 18px;border-top:1px solid #aab9b5;border-bottom:1px solid #aab9b5}
|
||||
.boundary-note b{font:700 .67rem/1.4 var(--font-mono);letter-spacing:.08em;color:#a55c2d}.boundary-note p{margin:0;color:#40585c;font-size:.8rem;line-height:1.65}
|
||||
.judge-arena{display:grid;grid-template-columns:1fr auto 1fr;gap:12px;align-items:center;margin:22px 0}.judge-arena>div{font:800 .7rem/1 var(--font-mono);color:#a26337}
|
||||
.answer-card{padding:17px;border:1px solid #bbc5c1;background:#f9faf5}.answer-card span{font:700 .6rem/1 var(--font-mono);color:#9f5f34}.answer-card h5{margin:7px 0;font-size:1rem}.answer-card p{margin:0;color:#617375;font-size:.76rem}
|
||||
.rank-rail{margin:22px 0}.rank-rail>div:first-child{display:flex;justify-content:space-between;color:#70807f;font:600 .58rem/1 var(--font-mono)}
|
||||
.rank-track{position:relative;height:15px;margin:7px 0;background:linear-gradient(90deg,#d6ded9 49.5%,#8ea09e 50%,#d6ded9 50.5%)}
|
||||
.rank-track i{position:absolute;inset:0 auto 0 0;background:#bf7541;transition:width .2s}.rank-track b{position:absolute;top:-5px;width:3px;height:25px;background:#15343c;transform:translateX(-1px)}
|
||||
.rank-rail p{margin:9px 0 0;color:#627473;font-size:.68rem}.rank-rail p i,.rank-rail p b{display:inline-block;width:12px;height:4px;background:#bf7541}.rank-rail p b{width:3px;height:12px;background:#15343c}
|
||||
.leak-stack{display:grid;grid-template-columns:repeat(5,1fr);gap:7px;margin:22px 0}.leak-stack label{position:relative;padding:15px 12px;border:1px solid #bec9c5;background:#f8f9f4;cursor:pointer}
|
||||
.leak-stack input{position:absolute;right:10px;top:10px;accent-color:#b86d39}.leak-stack span,.leak-stack b,.leak-stack small{display:block}.leak-stack span{font:700 .58rem/1 var(--font-mono);color:#a55d31}.leak-stack b{margin:9px 0 5px;font-size:.78rem}.leak-stack small{color:#6c7b7c;font-size:.65rem;line-height:1.4}
|
||||
.leak-stack label:has(input:checked){background:#fff0e2;border-color:#c77c45;box-shadow:inset 0 3px #c77c45}
|
||||
.freshness-matrix{display:grid;grid-template-columns:repeat(3,1fr);gap:8px}.freshness-matrix>div{padding:16px;background:#e0e6e2;border-top:3px solid #95a5a2}.freshness-matrix .recommended{background:#173239;color:white;border-color:#d18b50}
|
||||
.freshness-matrix span,.freshness-matrix b,.freshness-matrix small{display:block}.freshness-matrix span{font:700 .58rem/1 var(--font-mono);letter-spacing:.05em}.freshness-matrix b{margin:8px 0;font-size:.85rem}.freshness-matrix small{font-size:.67rem;line-height:1.4;opacity:.72}
|
||||
.system-equation{display:flex;gap:7px;align-items:stretch;margin:22px 0}.system-equation>div{flex:1;padding:14px 10px;background:#dfe6e2;text-align:center}.system-equation i{align-self:center;color:#ae6a3a;font-style:normal}.system-equation span,.system-equation b{display:block}.system-equation span{font:700 .56rem/1 var(--font-mono);color:#6b7e7c}.system-equation b{margin-top:6px;font-size:.68rem}
|
||||
.frontier-plot{position:relative;height:210px;margin:22px 0 0;border-left:1px solid #7f9290;border-bottom:1px solid #7f9290;background:linear-gradient(135deg,rgba(67,119,107,.12),transparent 55%),repeating-linear-gradient(0deg,transparent 0 41px,rgba(30,67,72,.08) 41px 42px),repeating-linear-gradient(90deg,transparent 0 19.8%,rgba(30,67,72,.08) 19.8% 20%)}
|
||||
.frontier-plot .axis{position:absolute;color:#627675;font:700 .58rem/1 var(--font-mono)}.frontier-plot .axis.y{left:6px;top:7px}.frontier-plot .axis.x{right:7px;bottom:7px}.frontier-plot .risk-zone{position:absolute;right:10px;top:10px;color:#b55e35;font:700 .58rem/1 var(--font-mono)}
|
||||
.frontier-plot>i{position:absolute;width:15px;height:15px;border:3px solid #fff;border-radius:50%;background:#c66f37;box-shadow:0 0 0 2px #173239;transform:translate(-50%,50%);transition:.2s}
|
||||
.frontier-plot>i span{position:absolute;left:12px;bottom:13px;width:80px;color:#173239;font:700 .6rem/1 var(--font-mono);font-style:normal}
|
||||
.eval-lab figcaption{padding:18px 30px;background:#d9e1dc;color:#536866;font-size:.75rem;line-height:1.65;border-top:1px solid #bdc8c3}
|
||||
@media(max-width:900px){.eval-lab-head,.eval-panel-intro{grid-template-columns:1fr}.eval-tabs{grid-template-columns:repeat(2,1fr)}.eval-controls{grid-template-columns:repeat(2,1fr)}.eval-metrics.five{grid-template-columns:repeat(2,1fr)}.leak-stack{grid-template-columns:repeat(2,1fr)}.system-equation{display:grid;grid-template-columns:repeat(5,1fr)}.system-equation i{display:none}}
|
||||
@media(max-width:620px){.eval-view{padding:20px 14px}.eval-lab-head{padding:23px 18px}.eval-tabs button{padding:14px 10px}.eval-tabs small{white-space:normal}.protocol-presets,.eval-controls,.eval-metrics.five,.eval-metrics.four,.ppl-compare,.freshness-matrix{grid-template-columns:1fr}.judge-arena{grid-template-columns:1fr}.judge-arena>div{text-align:center}.leak-stack{grid-template-columns:1fr 1fr}.boundary-note{grid-template-columns:1fr}.system-equation{grid-template-columns:1fr 1fr}.eval-lab figcaption{padding:16px 18px}}
|
||||
</style>
|
||||
Reference in New Issue
Block a user