feat: add evaluation and safety chapter
This commit is contained in:
wuyang
+17
-4
@@ -23,6 +23,7 @@
|
||||
| 大规模训练系统 | 完成首版 | 71% | 真实集群 traces、故障案例与精确 topology 配置 |
|
||||
| 推理服务与低成本部署 | 完成首版 | 78% | 真实 GPU kernel / workload traces、功耗与跨框架复现 |
|
||||
| 数值精度、优化器与稳定性 | 完成首版 | 75% | 真实 kernel 吞吐、长程训练 traces 与逐图论文精读 |
|
||||
| 评测、安全与“到底强不强” | 完成首版 | 79% | 真实 cross-harness 复跑、Judge 元评测与动态污染案例 |
|
||||
| 引用与事实检查 | 进行中 | 57% | 自动化外链复查与来源等级扩展 |
|
||||
| 开源仓库 | 已完成首版 | 100% | 持续提交研究与网站迭代 |
|
||||
| k1412 部署 | 已完成首版 | 100% | 每轮发布保留不可变镜像与回滚点 |
|
||||
@@ -32,14 +33,14 @@
|
||||
- [x] 建立长期任务目标与阶段计划。
|
||||
- [x] 确认 K3 官方 47 页技术报告与官方模型仓库。
|
||||
- [x] 提取技术报告目录与 151 条参考来源，建立本地只读研究缓存。
|
||||
- [x] 从报告反推出 16 个专题与三条贯穿案例。
|
||||
- [x] 从报告反推出 17 个专题与三条贯穿案例。
|
||||
- [x] 提炼参考网站的编辑设计语言。
|
||||
- [x] 确认 `git.k1412.top` 为 Gitea/Forgejo 兼容服务且本机 HTTPS 凭据可用于既有仓库。
|
||||
- [x] 使用 Grok CLI 检索并形成约 95 篇一手论文的补充路线，主代理已回查关键来源。
|
||||
- [x] 完成 400 篇关键论文索引，覆盖 15 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成 450 篇关键论文索引，覆盖 15 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务与数值优化十五篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、语言模型前史四联实验、Transformer 四联实验、DeepSeek 谱系、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent 与原生多模态专题各四页签等四十七个原创交互视图。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十六篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、语言模型前史四联实验、Transformer 四联实验、DeepSeek 谱系、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等五十一个原创交互视图。
|
||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||
@@ -125,9 +126,16 @@
|
||||
- [x] 推理服务真实 Chrome 断言通过：MHA OOM、chunked prefill 降低 stall、慢网络反噬 P/D、低验收率负加速、缓存故障重算、平均准入伤害短请求、键盘 tabs 与 390px 移动端均正确响应。
|
||||
- [x] Astro 类型检查、生产构建、19 个页面、962 个站内引用和 16 个跨页锚点通过；推理服务页面桌面 / 移动端无文档级横向溢出。
|
||||
- [x] 推理服务首版以源提交 `db2b7d3`、不可变镜像 `20260729T003946Z-db2b7d3` 发布；NAS、VPS/Tailscale、NPM、DNS、HTTPS、证书、门户、公开 Forgejo 与十二套生产 Chrome 回归全链路通过。
|
||||
- [x] 启动评测与安全专题：用二十二张账拆开构念、指标、Prompt、采样、Harness、环境、Judge、污染、动态题、成本、威胁模型与披露。
|
||||
- [x] 使用 Grok Headless 生成 12 个“协议陷阱”教学问题；全部保留在未核验 leads，正式事实回查一手论文与官方报告。
|
||||
- [x] 本地缓存并校验 50 份新增 PDF/TXT，建立 1389 行正式研究账本、80 节点阅读链和 DeepSeek/K3 协议谱系。
|
||||
- [x] 完成评测安全首版：33 个正文目录、二十二张测量账，以及指标校准—Judge 偏差—污染动态题—系统安全边界四联实验。
|
||||
- [x] 论文库新增 BLEU、GLUE、HumanEval、MT-Bench、LiveBench、HarmBench、InjecAgent、AILuminate 等 50 个节点，从 400 篇扩充至 450 篇。
|
||||
- [x] Astro 类型检查、生产构建、20 个页面、1053 个站内引用和 15 个跨页锚点通过；十三套专题 Chrome 回归、键盘 tabs 与 390px 移动端无异常。
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
||||
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||
- [ ] 原生多模态二轮：真实视觉 Token traces、跨分辨率 / connector 消融、OCR 与视觉 Agent 安全失败案例。
|
||||
- [ ] Agent 二轮：真实环境 traces、cross-harness ablation、Agent RL 训练曲线与提示注入案例。
|
||||
@@ -218,6 +226,11 @@
|
||||
| 2026-07-29 | 论文库扩充到 400 篇 | 新增 45 个内存管理、调度、缓存、P/D 解耦、量化、推测解码、kernel 与真实 workload 节点 |
|
||||
| 2026-07-29 | 推理服务首版用四个独立实验闭环 | 显存、阶段干扰、推测验收和集群状态分开建模；作者报告、教学估算与 benchmark 永久分级 |
|
||||
| 2026-07-29 | 推理服务首版用不可变镜像 `20260729T003946Z-db2b7d3` 发布 | OCI digest `sha256:06374603…fe752`；复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo |
|
||||
| 2026-07-29 | 评测安全按二十二张测量账组织 | 构念、样本、指标、Prompt、采样、Harness、裁判、污染、成本与威胁模型不再压成一个榜单分 |
|
||||
| 2026-07-29 | Grok 评测线索与正式证据永久分离 | 12 个协议陷阱只负责教学查漏；50 份新增一手全文与 DeepSeek/K3 官方报告承载正文事实 |
|
||||
| 2026-07-29 | DeepSeek 与 K3 评测谱系按协议字段重建 | R1 的非零温多采样 pass@1、V4 effort / context / tool budget 与 K3 §6 task-specific harness 分层说明 |
|
||||
| 2026-07-29 | 论文库扩充到 450 篇 | 新增 50 个指标、套件、动态基准、Judge、代码 Verifier、安全与 Agent 风险节点 |
|
||||
| 2026-07-29 | 评测安全首版用四个独立实验闭环 | 指标/校准、Judge/Arena、污染/动态题、系统/安全边界分开建模，不合成伪“模型总分” |
|
||||
|
||||
## 未决问题
|
||||
|
||||
|
||||
@@ -17,9 +17,9 @@
|
||||
- 持续进度：[PROGRESS.md](./PROGRESS.md)
|
||||
- 证据与写作规范：[research/METHODOLOGY.md](./research/METHODOLOGY.md)
|
||||
|
||||
当前里程碑包含 16 专题学习地图、400 篇关键论文索引、Kimi K3 完整导读，
|
||||
语言模型前史、Transformer 基础、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务与数值优化深度专题，
|
||||
以及 47 个覆盖核心机制的原创交互视图。
|
||||
当前里程碑包含 17 专题学习地图、450 篇关键论文索引、Kimi K3 完整导读，
|
||||
语言模型前史、Transformer 基础、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
||||
以及 51 个覆盖核心机制的原创交互视图。
|
||||
其余专题按进度账本持续扩建。
|
||||
|
||||
## 本地开发
|
||||
|
||||
+8
-1
@@ -15,7 +15,7 @@
|
||||
- **L2 论文**：能读关键公式、消融实验和训练配置。
|
||||
- **L3 工程**：理解并行、通信、显存、精度与服务系统。
|
||||
|
||||
## 16 个专题
|
||||
## 17 个专题
|
||||
|
||||
### 00. 导航：先看懂一张大模型地图
|
||||
|
||||
@@ -132,6 +132,13 @@ EAGLE-3 draft、cache-aware affinity 与 token-budget admission。配套显存
|
||||
|
||||
困惑度到 MMLU/GPQA/HLE，SWE-bench、OSWorld、BrowseComp；污染、harness、工具预算、LLM-as-a-judge、选择性报告与网络安全边界。
|
||||
|
||||
首版已完成：以构念、单位、任务分布、指标、Prompt、解码、采样、Harness、环境、裁判、聚合、
|
||||
不确定性、污染、题质、动态性、Arena、成本、子群、事实性、威胁模型、安全边界与披露二十二张账，
|
||||
串起 2002–2026 的 80 个一手节点。正文从 PPL / BLEU / ROUGE、MMLU 与 HELM，走到 pass@k /
|
||||
pass^k、校准、动态基准、代码 Verifier、LLM Judge、Arena、Agent 最终状态与安全威胁模型；
|
||||
重点逐项复原 DeepSeek LLM→Math→V2→V3→R1→V3.2→V4 与 Kimi K3 §6 的评测协议，并提供
|
||||
指标校准、Judge 偏差、污染动态题、系统成本与安全边界四个独立实验。
|
||||
|
||||
## 三条贯穿式案例
|
||||
|
||||
1. **Kimi K3 解剖**：把上述全部专题重新汇总到一张架构与训练系统图。
|
||||
|
||||
+2
-1
@@ -21,7 +21,8 @@
|
||||
"check:alignment-browser": "node scripts/check-alignment-browser.mjs",
|
||||
"check:agents-browser": "node scripts/check-agents-browser.mjs",
|
||||
"check:multimodal-browser": "node scripts/check-multimodal-browser.mjs",
|
||||
"check:inference-serving-browser": "node scripts/check-inference-serving-browser.mjs"
|
||||
"check:inference-serving-browser": "node scripts/check-inference-serving-browser.mjs",
|
||||
"check:evaluation-browser": "node scripts/check-evaluation-browser.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"@astrojs/sitemap": "3.7.3",
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
# 评测与安全：Grok 候选线索账
|
||||
|
||||
> 状态：**未核验候选，不可作为正文证据。**
|
||||
>
|
||||
> 生成日期：2026-07-29
|
||||
>
|
||||
> 调用约束：`--no-plan --verbatim --no-subagents --disable-web-search`
|
||||
>
|
||||
> 使用纪律：Grok 只负责扩大候选召回和提出教学问题；论文年份、标题、指标、数值、
|
||||
> 因果解释与 K3 / DeepSeek 细节必须回到 P0 论文、技术报告或 P1 作者官方实现核验。
|
||||
|
||||
---
|
||||
|
||||
## 0. 为什么保留这份文件
|
||||
|
||||
这份文件不是“第二套参考文献”，而是防止研究过程失忆的候选池。它同时保留：
|
||||
|
||||
1. Grok 建议调查但尚未核验的节点；
|
||||
2. 已经进入一手核验队列的候选；
|
||||
3. 可能被拒绝、合并或降级为旁注的线索；
|
||||
4. 一组适合做成视觉与交互实验的协议比较问题。
|
||||
|
||||
正式证据账见 `EVALUATION_SAFETY_RESEARCH.md`。
|
||||
|
||||
---
|
||||
|
||||
## 1. 二十条候选主线
|
||||
|
||||
| # | 候选主线 | Grok 提出的核心矛盾 | 核验状态 |
|
||||
|---|---|---|---|
|
||||
| 1 | 困惑度与 next-token loss | 语言分布拟合不等于任务能力 | 已进入一手核验 |
|
||||
| 2 | 静态基准饱和 | 难度、区分度与题目质量会随模型进步失效 | 已进入一手核验 |
|
||||
| 3 | 数据污染 | 文本不重合不等于语义、答案或格式没有泄漏 | 已进入一手核验 |
|
||||
| 4 | 动态评测 | 更新题目能减轻污染，但会改变跨时间可比性 | 已进入一手核验 |
|
||||
| 5 | 代码执行 | 单元测试是强于文字相似度的 verifier，但测试也可能不充分 | 已进入一手核验 |
|
||||
| 6 | 数学验证 | exact match、数值检查、形式化证明不是同一强度 | 已进入一手核验 |
|
||||
| 7 | 人类偏好 | 偏好受用户群、抽样、顺序与呈现影响 | 已进入一手核验 |
|
||||
| 8 | LLM-as-a-Judge | 位置、长度、自我偏好与能力上限会进入分数 | 已进入一手核验 |
|
||||
| 9 | Arena / Elo | 排名是比较图与统计模型的产物，不是绝对能力刻度 | 已进入一手核验 |
|
||||
| 10 | 长上下文 | 声明窗口不等于有效理解长度 | 已进入一手核验 |
|
||||
| 11 | 多模态 | 总分可能掩盖 OCR、感知、知识与推理瓶颈 | 已进入一手核验 |
|
||||
| 12 | Agent / harness | 分数同时属于模型、脚手架、工具、环境和预算 | 已进入一手核验 |
|
||||
| 13 | 校准与弃答 | 准确率不回答“何时应该相信或拒答” | 已进入一手核验 |
|
||||
| 14 | 红队 | 红队发现什么取决于参与者、时间、访问权和目标 | 已进入一手核验 |
|
||||
| 15 | Jailbreak | “没有拒绝”不等于输出真的有害、具体且可用 | 已进入一手核验 |
|
||||
| 16 | Prompt injection | Agent 读取不可信数据后，权限与控制流成为评测对象 | 已进入一手核验 |
|
||||
| 17 | 网络安全双重用途 | 解题能力、真实攻击能力与部署风险必须分层 | 已进入一手核验 |
|
||||
| 18 | System / model card | 评测披露本身是可审计对象 | 已进入一手核验 |
|
||||
| 19 | 成本归一化 | 同一分数可能用了不同 token、工具调用和重复采样预算 | 已进入一手核验 |
|
||||
| 20 | 跨语言、文化与公平 | 翻译题不自动成为目标文化的有效测量 | 已进入一手核验 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 候选节点池
|
||||
|
||||
### 2.1 测量与综合基准
|
||||
|
||||
- BLEU、ROUGE；
|
||||
- GLUE、SuperGLUE；
|
||||
- MMLU、BIG-bench、BIG-Bench Hard、HELM；
|
||||
- MMLU-Redux、MMLU-Pro、GPQA、Humanity’s Last Exam；
|
||||
- PALOMA、LiveBench；
|
||||
- Dynabench、Dynaboard；
|
||||
- Global MMLU。
|
||||
|
||||
### 2.2 校准、事实性与不确定性
|
||||
|
||||
- On Calibration of Modern Neural Networks；
|
||||
- Language Models (Mostly) Know What They Know；
|
||||
- Semantic Uncertainty；
|
||||
- TruthfulQA；
|
||||
- SelfCheckGPT；
|
||||
- FActScore；
|
||||
- HaluEval；
|
||||
- conformal abstention 候选。
|
||||
|
||||
### 2.3 开放生成、偏好与 Judge
|
||||
|
||||
- MT-Bench / Chatbot Arena；
|
||||
- Chatbot Arena 正式平台论文；
|
||||
- AlpacaEval 与 length-controlled AlpacaEval；
|
||||
- G-Eval；
|
||||
- LLMBar；
|
||||
- Arena-Hard / BenchBuilder；
|
||||
- RewardBench；
|
||||
- CoBBLEr、JudgeBench 等元评测候选。
|
||||
|
||||
### 2.4 代码、数学、上下文与 Agent
|
||||
|
||||
- HumanEval、DS-1000、EvalPlus、LiveCodeBench；
|
||||
- SWE-bench、SWE-bench Verified、SWE-bench Live / Pro；
|
||||
- FrontierMath；
|
||||
- LongBench、RULER；
|
||||
- AgentBench、WebArena、OSWorld；
|
||||
- τ-bench、BrowseComp、Terminal-Bench；
|
||||
- AgentDojo、InjecAgent、Agent Security Bench、Cybench。
|
||||
|
||||
### 2.5 安全、拒答与红队
|
||||
|
||||
- RealToxicityPrompts、StereoSet、CrowS-Pairs、BBQ、ToxiGen；
|
||||
- Red Teaming Language Models with Language Models；
|
||||
- Red Teaming Language Models to Reduce Harms；
|
||||
- GCG adversarial suffix；
|
||||
- Jailbroken；
|
||||
- XSTest、Do-Not-Answer；
|
||||
- HarmBench、StrongREJECT、JailbreakBench、WildGuard；
|
||||
- Instruction Hierarchy；
|
||||
- AILuminate；
|
||||
- Red-Teaming for Generative AI。
|
||||
|
||||
---
|
||||
|
||||
## 3. Grok 提出的十二张“协议护照”
|
||||
|
||||
下列内容只作为教学设计候选。每一项都必须在正式材料中用一手来源重新填写。
|
||||
|
||||
| # | 审计问题 | 容易产生的误读 | 必须披露的证据 | 候选视觉隐喻 |
|
||||
|---|---|---|---|---|
|
||||
| G1 | 同名 benchmark 是否用了相同题面、shots、示例与答案抽取？ | 分数更高就是模型更强 | 完整 prompt、shots、示例选择、CoT、parser | 同名考试，不同答题纸 |
|
||||
| G2 | 是 greedy、单次采样、pass@k、best-of-k 还是多数票？ | “90%”就是日常一次回答有 90% 正确率 | temperature、top-p、样本数、聚合、seed | 一次考试对十次取最好 |
|
||||
| G3 | max tokens 与 reasoning effort 是否相同？ | 架构更聪明，而不是测试时算得更多 | token cap、effort 档、停止规则、平均输出 | 草稿纸不限量的选手 |
|
||||
| G4 | 是裸模型，还是带代码、搜索、计算器与 Agent loop？ | 工具增强分数等于裸模型能力 | harness、工具清单、sandbox、重试、baseline 工具 | 开卷计算器对闭卷口试 |
|
||||
| G5 | 谁判对：字符串、程序、单测、人还是 LLM Judge？ | 不同裁判仍产生可移植的同一准确率 | judge 模型与 prompt、verifier、human audit | 不同终点传感器 |
|
||||
| G6 | 评的是裸 checkpoint 还是带风控的产品 API？ | 产品分数可同时代表模型能力与安全 | system prompt、过滤器、过拒率、wrapper ablation | 装有限速器的赛车 |
|
||||
| G7 | 长上下文是否被截断、摘要、检索或重新打包？ | 长上下文榜等于纯上下文理解 | window、截断方向、RAG、chunk、实际可见 token | 有人读全文，有人只看目录 |
|
||||
| G8 | benchmark 版本、split、commit 和日期是否一致？ | 名字相同就是同一套题 | hash、日期、decontamination、private holdout | 不同年份的马拉松赛道 |
|
||||
| G9 | 分数用了多少成本、延迟、token 与工具调用？ | 最高准确率自然是最佳生产选择 | 输入/输出 token、价格、P50/P95、硬件、重复采样成本 | 烧完整个车队油量的冠军 |
|
||||
| G10 | 数字来自作者自报、第三方重跑还是营销图？ | 出现在榜单上的数字同样可审计 | raw logs、config、seed、代码 commit、provenance | 实验室标签对自印标签 |
|
||||
| G11 | base、chat、reasoner 是否被同一协议公平地激发？ | 代际或架构独自解释全部差距 | 训练阶段、chat template、system role、CoT、默认解码 | 短跑、马拉松与徒步同榜 |
|
||||
| G12 | 拒答在当前任务里算失败、成功还是剔除？ | helpfulness 与 safety 可以压成单轴 | 拒答 rubric、过拒套件、政策版本、人工复核 | 门打不开是故障还是防盗 |
|
||||
|
||||
Grok 给出的候选总句：
|
||||
|
||||
```text
|
||||
leaderboard entry
|
||||
≠ intrinsic model property
|
||||
= model × prompt × sample budget × tools × judge × wrapper
|
||||
× data vintage × cost
|
||||
```
|
||||
|
||||
这句话仍需在正式正文中改写为“测量模型”，而不是引用 Grok。
|
||||
|
||||
---
|
||||
|
||||
## 4. 候选教学实验
|
||||
|
||||
### Lab A：协议护照
|
||||
|
||||
给两份公开技术报告，不看模型名，只填写：
|
||||
|
||||
```text
|
||||
dataset / split / prompt / shots / decode / samples
|
||||
/ budget / harness / tools / judge / verifier / provenance
|
||||
```
|
||||
|
||||
看字段是否足以支持横向比较。
|
||||
|
||||
### Lab B：Judge 偏差
|
||||
|
||||
保持答案语义不变，改变：
|
||||
|
||||
- 顺序；
|
||||
- 长度；
|
||||
- 风格；
|
||||
- 是否来自与 Judge 同族的模型；
|
||||
- rubric；
|
||||
|
||||
观察“偏好”怎样变化。
|
||||
|
||||
### Lab C：污染与动态基准
|
||||
|
||||
模拟五种污染：
|
||||
|
||||
- 原文；
|
||||
- 答案；
|
||||
- 格式；
|
||||
- 语义改写；
|
||||
- 时间泄漏；
|
||||
|
||||
再比较静态、私有、动态和可执行 verifier 四种补救。
|
||||
|
||||
### Lab D：系统分数
|
||||
|
||||
固定模型，改变：
|
||||
|
||||
- harness；
|
||||
- 工具；
|
||||
-步数；
|
||||
- 上下文；
|
||||
- 重试；
|
||||
- verifier；
|
||||
- 成本；
|
||||
|
||||
让学生看到 Agent 分数为何不能只归给模型。
|
||||
|
||||
---
|
||||
|
||||
## 5. 拒绝或降级规则
|
||||
|
||||
以下候选即使最终看起来“有趣”，也不直接进入正文结论：
|
||||
|
||||
1. 找不到 P0 / P1 原始来源的榜单数字；
|
||||
2. 只在聚合博客中出现的 benchmark 描述；
|
||||
3. 没有 threat model 的 jailbreak 成功率；
|
||||
4. 没有 Judge prompt / model 的自动偏好分；
|
||||
5. 没有 harness、工具和环境版本的 Agent 分数；
|
||||
6. 没有样本数与解码配置的 reasoning 分数；
|
||||
7. 把“未发现危险行为”写成“模型安全”；
|
||||
8. 把 n-gram 无重合写成“绝无污染”；
|
||||
9. 把作者单一设置下的相关性外推为 Judge 普遍可靠；
|
||||
10. 把安全 wrapper 的产品表现归因于裸模型。
|
||||
|
||||
---
|
||||
|
||||
## 6. 当前核验去向
|
||||
|
||||
- 测量与历史：BLEU、ROUGE、GLUE、MMLU、BIG-bench、HELM；
|
||||
- 基准修复：MMLU-Redux、MMLU-Pro、HLE、EvalPlus、SWE-bench Verified；
|
||||
- 污染与动态：PALOMA、FreshQA、LiveBench、LiveCodeBench、SWE-bench Live；
|
||||
- Judge 与 Arena：MT-Bench、Chatbot Arena、LLMBar、length-controlled AlpacaEval；
|
||||
- 校准：Guo et al.、P(IK) / P(True)、semantic uncertainty；
|
||||
- 安全：XSTest、HarmBench、StrongREJECT、JailbreakBench、AILuminate；
|
||||
- Agent 安全：InjecAgent、AgentDojo、Cybench、Agent Security Bench；
|
||||
- 锚点报告：DeepSeek LLM / Math / V2 / V3 / R1 / V3.2 / V4 与 Kimi K3 §6。
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -66,3 +66,17 @@ DeepSeek-VL/VL2、Janus、OCR 三分支，Kimi 三代 MoonViT、十六张问题
|
||||
计入全文证据。十八本独立账、62 节点阅读链、DeepSeek 与 Kimi 服务谱系、十五组视觉合同和
|
||||
四实验合同见 `../INFERENCE_SERVING_RESEARCH.md`。Grok Headless 的 70 个候选只保存在
|
||||
`../INFERENCE_SERVING_GROK_LEADS.md`，不能越过一手来源核验进入正文。
|
||||
|
||||
评测与安全首轮缓存位于 `evaluation-safety/`（不提交 PDF/TXT）：
|
||||
|
||||
- 指标与套件：BLEU、ROUGE、GLUE、SuperGLUE、MMLU-Pro / Redux、PALOMA 与 RULER；
|
||||
- 动态与代码：Dynabench、FreshLLMs、LiveBench、HumanEval、EvalPlus、LiveCodeBench 与 SWE-bench Live；
|
||||
- 裁判与偏好：MT-Bench、G-Eval、LLMBar、Length-Controlled AlpacaEval、Arena-Hard 与 Chatbot Arena；
|
||||
- 安全与 Agent：RealToxicity、ToxiGen、GCG、HarmBench、StrongREJECT、JailbreakBench、InjecAgent、
|
||||
Cybench、WildGuard 与 AILuminate；
|
||||
- DeepSeek LLM/Math/V2/V3/R1/V3.2/V4 与 Kimi K3 复用既有模型报告缓存。
|
||||
|
||||
本轮完整校验 50 份新增 PDF/TXT；Global MMLU 通过 canonical arXiv 导出入口取得，PDF 可读性警告
|
||||
未影响文本核验。二十二张测量账、80 节点阅读链、DeepSeek/K3 协议谱系、四实验合同与证据边界见
|
||||
`../EVALUATION_SAFETY_RESEARCH.md`。Grok Headless 仅生成教学问题和候选线索，永久隔离在
|
||||
`../EVALUATION_SAFETY_GROK_LEADS.md`，不能作为正式事实来源。
|
||||
|
||||
@@ -217,7 +217,7 @@ if (!overview.title.includes("可靠行动")) failures.push("章节标题异常"
|
||||
if (overview.sections !== 28 || overview.tocLinks !== 28) failures.push("章节/目录数量异常");
|
||||
if (overview.paperLinks !== 52) failures.push("正式论文链不是 52 个节点");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.navLinks !== 18 || home.navLinks !== 18 || mobile.mobileLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||
if (overview.navLinks !== 19 || home.navLinks !== 19 || mobile.mobileLinks !== 19) failures.push("全站导航未同步评测专题");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||
if (loop.initial.finalState !== "UNVERIFIED" || loop.directSchema.finalState !== "FAILED") failures.push("控制循环终局状态异常");
|
||||
if (!loop.directSchema.observation.includes("ERROR schema") || loop.directSchema.recovery !== "FRAGILE") failures.push("Direct/schema 故障传播异常");
|
||||
@@ -228,8 +228,8 @@ if (numeric(reliability.initial.passAt) <= numeric(reliability.k2.passAt) || num
|
||||
if (reliability.nonIdempotent.sideRisk === "LOW") failures.push("非幂等写操作风险没有提升");
|
||||
if (numeric(rl.wait.utilization) >= numeric(rl.full.utilization) || numeric(rl.wait.lostWork) <= numeric(rl.full.lostWork)) failures.push("wait-all 长尾/重算方向异常");
|
||||
if (!rl.wait.takeaway.includes("wait-all") || rl.keyboardSelected !== "rl" || rl.keyboardVisible !== "rl") failures.push("长程 RL 解释或键盘导航异常");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||
if (home.paperCount !== "400" || papers.total !== 400 || !papers.hasAgentFilter || papers.agentVisible < 52) failures.push("论文库 Agent 标签或论文总数异常");
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") failures.push("首页评测首发入口异常");
|
||||
if (home.paperCount !== "450" || papers.total !== 450 || !papers.hasAgentFilter || papers.agentVisible < 52) failures.push("论文库 Agent 标签或论文总数异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
|
||||
@@ -216,7 +216,7 @@ if (!overview.title.includes("真正与人协作")) failures.push("章节标题
|
||||
if (overview.sections !== 22 || overview.tocLinks !== 22) failures.push("章节/目录数量异常");
|
||||
if (overview.paperLinks !== 44) failures.push("正式论文链不是 44 个节点");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.navLinks !== 18 || home.navLinks !== 18 || mobile.mobileLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||
if (overview.navLinks !== 19 || home.navLinks !== 19 || mobile.mobileLinks !== 19) failures.push("全站导航未同步评测专题");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||
if (sft.initial.active !== "6 / 10" || !sft.initial.lossStates.slice(0, 4).every((value) => value === "MASKED")) failures.push("SFT response-only mask 异常");
|
||||
if (sft.unsafeAll.active !== "10 / 10" || numeric(sft.unsafeAll.nll) <= numeric(sft.initial.nll) || !sft.unsafeAll.reading.includes("错误")) failures.push("SFT 全序列/坏示范交互异常");
|
||||
@@ -226,8 +226,8 @@ if (!update.steps[0].includes("Fixed preference")) failures.push("DPO 更新流
|
||||
if (!recipe.family.includes("Multi-effort") || !recipe.regime.includes("9 RL experts") || !recipe.constraints.includes("verbosity")) failures.push("K3 配方合同异常");
|
||||
if (!recipe.path.some((step) => step.includes("3 domains × 3 efforts")) || !recipe.path.some((step) => step.includes("MOPD"))) failures.push("K3 配方路径异常");
|
||||
if (recipe.keyboardSelected !== "recipe" || recipe.keyboardVisible !== "recipe") failures.push("实验 tab 键盘导航异常");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||
if (home.paperCount !== "400" || papers.total !== 400 || !papers.hasAlignmentFilter || papers.alignmentVisible < 35) failures.push("论文库后训练标签或论文总数异常");
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") failures.push("首页评测首发入口异常");
|
||||
if (home.paperCount !== "450" || papers.total !== 450 || !papers.hasAlignmentFilter || papers.alignmentVisible < 35) failures.push("论文库后训练标签或论文总数异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
|
||||
@@ -230,15 +230,15 @@ if (transform.keyboard.selected !== "transform" || transform.keyboard.visible !=
|
||||
if (layout.articleSections !== 18 || layout.paperLinks !== 31 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||
failures.push("章节、论文或实验数量异常");
|
||||
}
|
||||
if (layout.navLinks !== 18 || mobile.mobileLinks !== 18 || home.navLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||
if (layout.navLinks !== 19 || mobile.mobileLinks !== 19 || home.navLinks !== 19) failures.push("全站导航未同步评测专题");
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") {
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") {
|
||||
failures.push("首页 Transformer 新章入口异常");
|
||||
}
|
||||
if (home.paperCount !== "400") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||
if (!papers.hasDataFilter || papers.total !== 400 || papers.visible < 25) failures.push("论文库数据标签或论文总数异常");
|
||||
if (home.paperCount !== "450") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||
if (!papers.hasDataFilter || papers.total !== 450 || papers.visible < 25) failures.push("论文库数据标签或论文总数异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
const cdpPort = process.env.CDP_PORT ?? "9227";
|
||||
const baseUrl = process.env.SITE_URL ?? "http://127.0.0.1:4327";
|
||||
const pages = await fetch(`http://127.0.0.1:${cdpPort}/json/list`).then((response) => response.json());
|
||||
const page = pages.find((entry) => entry.type === "page");
|
||||
if (!page) throw new Error(`CDP ${cdpPort} 没有可用页面`);
|
||||
|
||||
const socket = new WebSocket(page.webSocketDebuggerUrl);
|
||||
await new Promise((resolve, reject) => {
|
||||
socket.addEventListener("open", resolve, { once: true });
|
||||
socket.addEventListener("error", reject, { once: true });
|
||||
});
|
||||
|
||||
let nextId = 0;
|
||||
const pending = new Map();
|
||||
const exceptions = [];
|
||||
socket.addEventListener("message", (event) => {
|
||||
const message = JSON.parse(event.data);
|
||||
if (message.id && pending.has(message.id)) {
|
||||
const { resolve, reject } = pending.get(message.id);
|
||||
pending.delete(message.id);
|
||||
if (message.error) reject(new Error(message.error.message));
|
||||
else resolve(message.result);
|
||||
}
|
||||
if (message.method === "Runtime.exceptionThrown") {
|
||||
exceptions.push(message.params.exceptionDetails.exception?.description ?? message.params.exceptionDetails.text);
|
||||
}
|
||||
});
|
||||
|
||||
const command = (method, params = {}) => new Promise((resolve, reject) => {
|
||||
const id = ++nextId;
|
||||
pending.set(id, { resolve, reject });
|
||||
socket.send(JSON.stringify({ id, method, params }));
|
||||
});
|
||||
const pause = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||
const evaluate = async (expression) => {
|
||||
const result = await command("Runtime.evaluate", { expression, returnByValue: true, awaitPromise: true });
|
||||
if (result.exceptionDetails) throw new Error(result.exceptionDetails.exception?.description ?? result.exceptionDetails.text);
|
||||
return result.result.value;
|
||||
};
|
||||
const navigate = async (path) => {
|
||||
await command("Page.navigate", { url: `${baseUrl}${path}` });
|
||||
for (let attempt = 0; attempt < 70; attempt += 1) {
|
||||
await pause(100);
|
||||
if (await evaluate("document.readyState === 'complete'")) return;
|
||||
}
|
||||
throw new Error(`${path} 加载超时`);
|
||||
};
|
||||
const screenshot = async (path) => {
|
||||
const result = await command("Page.captureScreenshot", { format: "png", captureBeyondViewport: false });
|
||||
writeFileSync(path, Buffer.from(result.data, "base64"));
|
||||
};
|
||||
|
||||
await command("Page.enable");
|
||||
await command("Runtime.enable");
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 1440,
|
||||
height: 1100,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: false,
|
||||
});
|
||||
|
||||
await navigate("/evaluation/");
|
||||
await screenshot("/tmp/llm-atlas-evaluation-desktop.png");
|
||||
|
||||
const overview = await evaluate(`(() => ({
|
||||
title: document.querySelector("h1")?.textContent.trim(),
|
||||
sections: document.querySelectorAll(".article-section").length,
|
||||
tocLinks: document.querySelectorAll(".side-rail a").length,
|
||||
paperLinks: document.querySelectorAll(".paper-chain a").length,
|
||||
ledgers: document.querySelectorAll(".eval-ledgers > article").length,
|
||||
labTabs: document.querySelectorAll("[data-eval-tab]").length,
|
||||
labPanels: document.querySelectorAll("[data-eval-panel]").length,
|
||||
navLinks: document.querySelectorAll(".top-nav a").length,
|
||||
activeNav: document.querySelector('.top-nav a[aria-current="page"]')?.textContent.trim(),
|
||||
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||
}))()`);
|
||||
|
||||
const metric = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-evaluation-lab]");
|
||||
const read = () => ({
|
||||
one: root.querySelector("[data-one-try]").textContent.trim(),
|
||||
atLeast: root.querySelector("[data-at-least]").textContent.trim(),
|
||||
all: root.querySelector("[data-all-succeed]").textContent.trim(),
|
||||
ece: root.querySelector("[data-toy-ece]").textContent.trim(),
|
||||
bpb: root.querySelector("[data-bpb]").textContent.trim(),
|
||||
pplA: root.querySelector("[data-ppl-a]").textContent.trim(),
|
||||
pplB: root.querySelector("[data-ppl-b]").textContent.trim(),
|
||||
visible: root.querySelector("[data-eval-panel]:not([hidden])").dataset.evalPanel,
|
||||
explain: root.querySelector("[data-metric-explain]").textContent.trim(),
|
||||
});
|
||||
const initial = read();
|
||||
root.querySelector('[data-protocol-preset="search"]').click();
|
||||
const search = read();
|
||||
root.querySelector('[data-protocol-preset="reliable"]').click();
|
||||
const reliable = read();
|
||||
return { initial, search, reliable };
|
||||
})()`);
|
||||
|
||||
const judge = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-evaluation-lab]");
|
||||
root.querySelector('[data-eval-tab="judge"]').click();
|
||||
const read = () => ({
|
||||
raw: root.querySelector("[data-raw-win]").textContent.trim(),
|
||||
adjusted: root.querySelector("[data-adjusted-win]").textContent.trim(),
|
||||
flip: root.querySelector("[data-order-flip]").textContent.trim(),
|
||||
ci: root.querySelector("[data-judge-ci]").textContent.trim(),
|
||||
explain: root.querySelector("[data-judge-explain]").textContent.trim(),
|
||||
});
|
||||
const initial = read();
|
||||
const gap = root.querySelector("[data-length-gap]");
|
||||
const bias = root.querySelector("[data-length-bias]");
|
||||
const order = root.querySelector("[data-order]");
|
||||
const votes = root.querySelector("[data-votes]");
|
||||
gap.value = "100";
|
||||
gap.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
bias.value = "40";
|
||||
bias.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const biased = read();
|
||||
order.value = "randomized";
|
||||
order.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
votes.value = "10000";
|
||||
votes.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const controlled = read();
|
||||
return { initial, biased, controlled };
|
||||
})()`);
|
||||
|
||||
const contamination = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-evaluation-lab]");
|
||||
root.querySelector('[data-eval-tab="contamination"]').click();
|
||||
const read = () => ({
|
||||
clean: root.querySelector("[data-true-clean]").textContent.trim(),
|
||||
detected: root.querySelector("[data-detected]").textContent.trim(),
|
||||
hidden: root.querySelector("[data-hidden-leak]").textContent.trim(),
|
||||
fresh: root.querySelector("[data-freshness]").textContent.trim(),
|
||||
comparable: root.querySelector("[data-comparability]").textContent.trim(),
|
||||
explain: root.querySelector("[data-contamination-explain]").textContent.trim(),
|
||||
});
|
||||
const initial = read();
|
||||
root.querySelectorAll("[data-leak]").forEach((input) => {
|
||||
input.checked = input.dataset.leak === "semantic";
|
||||
input.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
});
|
||||
const ngram = root.querySelector("[data-ngram]");
|
||||
ngram.value = "30";
|
||||
ngram.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const semantic = read();
|
||||
const refresh = root.querySelector("[data-refresh]");
|
||||
const anchor = root.querySelector("[data-anchor]");
|
||||
refresh.value = "90";
|
||||
refresh.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
anchor.value = "10";
|
||||
anchor.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const dynamic = read();
|
||||
return { initial, semantic, dynamic };
|
||||
})()`);
|
||||
|
||||
const system = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-evaluation-lab]");
|
||||
root.querySelector('[data-eval-tab="system"]').click();
|
||||
const read = () => ({
|
||||
model: root.querySelector("[data-model-only]").textContent.trim(),
|
||||
success: root.querySelector("[data-system-success]").textContent.trim(),
|
||||
cost: root.querySelector("[data-cost]").textContent.trim(),
|
||||
unsafe: root.querySelector("[data-unsafe]").textContent.trim(),
|
||||
overrefusal: root.querySelector("[data-overrefusal]").textContent.trim(),
|
||||
explain: root.querySelector("[data-system-explain]").textContent.trim(),
|
||||
});
|
||||
const initial = read();
|
||||
const retries = root.querySelector("[data-retries]");
|
||||
const budget = root.querySelector("[data-budget]");
|
||||
retries.value = "1";
|
||||
retries.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
budget.value = "4";
|
||||
budget.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const cheap = read();
|
||||
const wrapper = root.querySelector("[data-wrapper]");
|
||||
wrapper.value = "100";
|
||||
wrapper.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const locked = read();
|
||||
const firstTab = root.querySelector('[data-eval-tab="metric"]');
|
||||
firstTab.focus();
|
||||
firstTab.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowRight", bubbles: true }));
|
||||
return {
|
||||
initial,
|
||||
cheap,
|
||||
locked,
|
||||
keyboardSelected: root.querySelector('[data-eval-tab][aria-selected="true"]').dataset.evalTab,
|
||||
keyboardVisible: root.querySelector("[data-eval-panel]:not([hidden])").dataset.evalPanel,
|
||||
};
|
||||
})()`);
|
||||
|
||||
await evaluate(`document.querySelector("[data-evaluation-lab]").scrollIntoView({ block: "start", behavior: "instant" })`);
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-evaluation-lab-desktop.png");
|
||||
|
||||
await navigate("/");
|
||||
const home = await evaluate(`(() => ({
|
||||
releaseCards: document.querySelectorAll(".release-card").length,
|
||||
firstRelease: document.querySelector(".release-card h2").textContent.trim(),
|
||||
firstHref: document.querySelector(".release-card").getAttribute("href"),
|
||||
paperCount: document.querySelector(".hero-stats div:nth-child(3) b").textContent.trim(),
|
||||
topicCount: document.querySelector(".hero-stats div:nth-child(1) b").textContent.trim(),
|
||||
navLinks: document.querySelectorAll(".top-nav a").length,
|
||||
}))()`);
|
||||
|
||||
await navigate("/papers/");
|
||||
const papers = await evaluate(`(() => {
|
||||
const button = [...document.querySelectorAll("[data-filter]")].find((node) => node.textContent.trim() === "评测");
|
||||
button?.click();
|
||||
return {
|
||||
total: document.querySelectorAll("[data-paper]").length,
|
||||
visible: document.querySelectorAll("[data-paper]:not([hidden])").length,
|
||||
hasFilter: Boolean(button),
|
||||
};
|
||||
})()`);
|
||||
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 390,
|
||||
height: 844,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: true,
|
||||
});
|
||||
await navigate("/evaluation/");
|
||||
const mobile = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-evaluation-lab]");
|
||||
root.scrollIntoView({ block: "start", behavior: "instant" });
|
||||
const toggle = document.querySelector("#menu-toggle");
|
||||
toggle?.click();
|
||||
return {
|
||||
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||
menuVisible: getComputedStyle(toggle).display !== "none",
|
||||
menuOpen: toggle.getAttribute("aria-expanded"),
|
||||
mobileLinks: document.querySelectorAll("#mobile-nav a").length,
|
||||
tabs: root.querySelectorAll("[data-eval-tab]").length,
|
||||
offenders: [...document.querySelectorAll("body *")]
|
||||
.filter((node) => !node.closest(".aggregate-example, .repair-table, .k3-protocol-table, .paper-chain"))
|
||||
.filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1)
|
||||
.slice(0, 12)
|
||||
.map((node) => ({
|
||||
tag: node.tagName,
|
||||
className: typeof node.className === "string" ? node.className : "",
|
||||
right: Math.round(node.getBoundingClientRect().right),
|
||||
width: Math.round(node.getBoundingClientRect().width),
|
||||
})),
|
||||
};
|
||||
})()`);
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-evaluation-mobile.png");
|
||||
|
||||
const report = { overview, metric, judge, contamination, system, home, papers, mobile, exceptions };
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
|
||||
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
||||
const intervalWidth = (text) => {
|
||||
const values = text.match(/[0-9.]+/g)?.map(Number) ?? [];
|
||||
return values.length >= 2 ? values[1] - values[0] : Number.NaN;
|
||||
};
|
||||
const failures = [];
|
||||
if (!overview.title.includes("粗体数字")) failures.push("章节标题异常");
|
||||
if (overview.sections !== 33 || overview.tocLinks !== 33) failures.push("章节 / 目录数量异常");
|
||||
if (overview.paperLinks !== 80) failures.push("正式论文链不是 80 个节点");
|
||||
if (overview.ledgers !== 22) failures.push("二十二张评测账结构异常");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.navLinks !== 19 || home.navLinks !== 19 || mobile.mobileLinks !== 19 || overview.activeNav !== "评测安全") failures.push("全站导航未同步评测专题");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
||||
if (metric.initial.visible !== "metric" || numeric(metric.initial.one) !== 80 || numeric(metric.initial.atLeast) !== 80 || numeric(metric.initial.all) !== 80) failures.push("指标实验初始值异常");
|
||||
if (numeric(metric.search.atLeast) <= 99 || numeric(metric.search.all) >= 30 || !metric.search.explain.includes("搜索")) failures.push("pass@k / pass^k 方向没有分开");
|
||||
if (Math.abs(numeric(metric.reliable.all) - 32.8) > .2) failures.push("连续五次可靠性计算异常");
|
||||
if (numeric(metric.initial.pplB) <= numeric(metric.initial.pplA) || !metric.initial.bpb.includes("bits")) failures.push("tokenizer / BPB 教学对照异常");
|
||||
if (numeric(judge.biased.raw) >= numeric(judge.biased.adjusted)) failures.push("长度偏好没有压低短回答的观察胜率");
|
||||
if (!judge.controlled.explain.includes("随机交换") || intervalWidth(judge.controlled.ci) >= intervalWidth(judge.initial.ci)) failures.push("顺序控制或票数区间异常");
|
||||
if (numeric(contamination.semantic.detected) >= 10 || numeric(contamination.semantic.hidden) <= 15 || !contamination.semantic.explain.includes("语义")) failures.push("语义污染没有暴露 n-gram 检测盲区");
|
||||
if (numeric(contamination.dynamic.fresh) <= numeric(contamination.initial.fresh) || numeric(contamination.dynamic.comparable) >= numeric(contamination.initial.comparable)) failures.push("动态刷新与可比性权衡异常");
|
||||
if (numeric(system.initial.success) <= numeric(system.initial.model) || numeric(system.initial.cost) !== 128) failures.push("Harness / 重试系统成功率或成本异常");
|
||||
if (numeric(system.cheap.success) >= numeric(system.initial.success) || numeric(system.cheap.cost) !== 4) failures.push("低预算没有降低成功率 / 成本");
|
||||
if (numeric(system.locked.unsafe) !== 0 || numeric(system.locked.overrefusal) <= numeric(system.initial.overrefusal)) failures.push("安全壳没有展现危险服从 / 过拒权衡");
|
||||
if (system.keyboardSelected !== "judge" || system.keyboardVisible !== "judge") failures.push("实验键盘 tab 导航异常");
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") failures.push("首页评测首发入口异常");
|
||||
if (home.paperCount !== "450" || home.topicCount !== "17" || papers.total !== 450 || !papers.hasFilter || papers.visible < 80) failures.push("首页 / 论文库评测索引异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
if (failures.length) {
|
||||
console.error(`\nFAIL\n- ${failures.join("\n- ")}`);
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
console.log("\nPASS evaluation browser regression");
|
||||
}
|
||||
|
||||
socket.close();
|
||||
@@ -251,7 +251,7 @@ if (overview.sections !== 31 || overview.tocLinks !== 31) failures.push("章节/
|
||||
if (overview.paperLinks !== 62) failures.push("正式论文链不是 62 个节点");
|
||||
if (overview.ledgers !== 18) failures.push("十八本服务账结构异常");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.navLinks !== 18 || home.navLinks !== 18 || mobile.mobileLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||
if (overview.navLinks !== 19 || home.navLinks !== 19 || mobile.mobileLinks !== 19) failures.push("全站导航未同步评测专题");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||
if (!memory.initial.weight.includes("32.6") || !memory.initial.perToken.includes("128") || memory.initial.visible !== "memory") failures.push("GQA 默认显存账异常");
|
||||
if (!memory.exploded.status.includes("OOM") || !memory.exploded.fit.includes("over")) failures.push("极端 MHA 配置没有触发 OOM");
|
||||
@@ -264,8 +264,8 @@ if (!fleet.k3.avoided.includes("320K") || fleet.k3.shortSlo !== "PROTECTED") fai
|
||||
if (!fleet.failed.state.includes("SECONDARY RE-PREFILL") || !fleet.failed.recompute.includes("FAILED PRIMARY")) failures.push("缓存故障没有触发原子失效后的重算");
|
||||
if (fleet.bursty.shortSlo !== "VIOLATED") failures.push("平均并发阈值没有暴露长请求突发");
|
||||
if (fleet.keyboardSelected !== "phase" || fleet.keyboardVisible !== "phase") failures.push("实验键盘 tab 导航异常");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||
if (home.paperCount !== "400" || papers.total !== 400 || !papers.hasFilter || papers.visible !== 45) failures.push("论文库推理服务标签或总数异常");
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") failures.push("首页评测首发入口异常");
|
||||
if (home.paperCount !== "450" || papers.total !== 450 || !papers.hasFilter || papers.visible !== 45) failures.push("论文库推理服务标签或总数异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
@@ -177,7 +177,7 @@ if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentO
|
||||
}
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||
if (!mobile.menuVisible) failures.push("移动端菜单按钮未显示");
|
||||
if (home.releaseCards !== 13) failures.push(`首页新章卡数量异常：${home.releaseCards}`);
|
||||
if (home.releaseCards !== 14) failures.push(`首页新章卡数量异常：${home.releaseCards}`);
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -234,7 +234,7 @@ if (!overview.title.includes("第一类输入")) failures.push("章节标题异
|
||||
if (overview.sections !== 30 || overview.tocLinks !== 30) failures.push("章节/目录数量异常");
|
||||
if (overview.paperLinks !== 55) failures.push("正式论文链不是 55 个节点");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.navLinks !== 18 || home.navLinks !== 18 || mobile.mobileLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||
if (overview.navLinks !== 19 || home.navLinks !== 19 || mobile.mobileLinks !== 19) failures.push("全站导航未同步评测专题");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||
if (numeric(tokens.initial.patches) !== 5476 || numeric(tokens.initial.visual) !== 1369 || tokens.initial.visiblePanel !== "tokens") failures.push("视觉 Token 初始计算异常");
|
||||
if (!tokens.overloaded.status.includes("超预算") || numeric(tokens.overloaded.share) <= 100) failures.push("超大视频没有触发上下文超预算");
|
||||
@@ -246,8 +246,8 @@ if (ocr.unreported.status !== "OUT OF EVIDENCE" || ocr.unreported.accuracy !== "
|
||||
if (loop.toolsStart.state !== "OPEN" || loop.toolsEnd.state !== "VERIFIED" || loop.toolsEnd.evidence !== "97%" || loop.toolsEnd.tools !== "3") failures.push("vision-in-the-loop 终局异常");
|
||||
if (loop.cotEnd.state !== "FAILED" || !loop.cotEnd.takeaway.includes("不能凭空增加")) failures.push("文字 CoT 与新观察没有分开");
|
||||
if (loop.keyboardSelected !== "connector" || loop.keyboardVisible !== "connector") failures.push("实验键盘 tab 导航异常");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||
if (home.paperCount !== "400" || papers.total !== 400 || !papers.hasFilter || papers.multimodalVisible < 59) failures.push("论文库多模态标签或总数异常");
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") failures.push("首页评测首发入口异常");
|
||||
if (home.paperCount !== "450" || papers.total !== 450 || !papers.hasFilter || papers.multimodalVisible < 59) failures.push("论文库多模态标签或总数异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
@@ -273,15 +273,15 @@ if (stability.keyboard.selected !== "stability" || stability.keyboard.visible !=
|
||||
if (layout.articleSections !== 19 || layout.paperLinks !== 36 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||
failures.push("章节、论文或实验数量异常");
|
||||
}
|
||||
if (layout.navLinks !== 18 || mobile.mobileLinks !== 18 || home.navLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||
if (layout.navLinks !== 19 || mobile.mobileLinks !== 19 || home.navLinks !== 19) failures.push("全站导航未同步评测专题");
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") {
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") {
|
||||
failures.push("首页 Transformer 新章入口异常");
|
||||
}
|
||||
if (home.paperCount !== "400") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||
if (!papers.hasOptimizerFilter || papers.total !== 400 || papers.visible < 8) failures.push("论文库优化器标签或论文总数异常");
|
||||
if (home.paperCount !== "450") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||
if (!papers.hasOptimizerFilter || papers.total !== 450 || papers.visible < 8) failures.push("论文库优化器标签或论文总数异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -289,7 +289,7 @@ if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentO
|
||||
}
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化")) failures.push("首页推理服务新章入口异常");
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单")) failures.push("首页评测新章入口异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -269,14 +269,14 @@ if (emergence.paths.some((length) => length < 500)) failures.push("涌现多指
|
||||
if (layout.articleSections !== 16 || layout.paperLinks !== 29 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||
failures.push("章节、论文或实验数量异常");
|
||||
}
|
||||
if (layout.navLinks !== 18 || mobile.mobileLinks !== 18 || home.navLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||
if (layout.navLinks !== 19 || mobile.mobileLinks !== 19 || home.navLinks !== 19) failures.push("全站导航未同步评测专题");
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") {
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") {
|
||||
failures.push("首页 Transformer 新章入口异常");
|
||||
}
|
||||
if (home.paperCount !== "400") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||
if (home.paperCount !== "450") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -233,7 +233,7 @@ if (layout.articleSections !== 16 || layout.paperLinks !== 37 || layout.labTabs
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化")) failures.push("首页推理服务新章入口异常");
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单")) failures.push("首页评测新章入口异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -172,7 +172,7 @@ const home = await evaluate(`(() => ({
|
||||
releaseCards: document.querySelectorAll(".release-card").length,
|
||||
firstRelease: document.querySelector(".release-card h2").textContent,
|
||||
firstHref: document.querySelector(".release-card").getAttribute("href"),
|
||||
paperCount: [...document.querySelectorAll(".hero-stats b")].map((node) => node.textContent.trim()).find((value) => value === "400"),
|
||||
paperCount: [...document.querySelectorAll(".hero-stats b")].map((node) => node.textContent.trim()).find((value) => value === "450"),
|
||||
}))()`);
|
||||
|
||||
await navigate("/papers/");
|
||||
@@ -236,8 +236,8 @@ if (block.family.trim() !== "Hybrid MoE" || !block.kv.includes("3 KDA : 1 Gated
|
||||
if (!block.path.some((step) => step.includes("KDA × 3")) || !block.note.includes("AttnRes")) failures.push("K3 Block 路径异常");
|
||||
if (block.context.trim() !== "128K" || numeric(block.mha) !== 400 || numeric(block.kda) !== 1) failures.push("KV 成本缩放异常");
|
||||
if (block.keyboardSelected !== "block" || block.keyboardVisible !== "block") failures.push("实验 tab 键盘导航异常");
|
||||
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||
if (home.paperCount !== "400" || papers.total !== 400 || papers.transformerVisible < 30) failures.push("论文库或首页论文数量异常");
|
||||
if (home.releaseCards !== 14 || !home.firstRelease.includes("榜单不是体检单") || home.firstHref !== "/evaluation/") failures.push("首页评测首发入口异常");
|
||||
if (home.paperCount !== "450" || papers.total !== 450 || papers.transformerVisible < 30) failures.push("论文库或首页论文数量异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
|
||||
@@ -0,0 +1,642 @@
|
||||
---
|
||||
const protocolPresets = [
|
||||
["greedy", "一次 Greedy", "1 次、无搜索", 1, 1],
|
||||
["sample", "Mean pass@1", "多次采样取平均", 64, 1],
|
||||
["search", "Pass@k", "有 verifier 的搜索", 64, 8],
|
||||
["reliable", "Pass^k", "连续成功可靠性", 5, 5],
|
||||
];
|
||||
---
|
||||
|
||||
<figure class="eval-lab" data-evaluation-lab>
|
||||
<div class="eval-lab-head">
|
||||
<div>
|
||||
<p>INTERACTIVE / MEASUREMENT WORKBENCH</p>
|
||||
<h3>把榜单拆回四台测量仪</h3>
|
||||
</div>
|
||||
<p>
|
||||
所有输出都是公开公式上的教学模型，用来观察协议变量怎样改变结论；它们不是任何真实模型的复跑成绩，
|
||||
也不把相关性写成因果。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="eval-tabs" role="tablist" aria-label="选择评测实验">
|
||||
<button type="button" role="tab" data-eval-tab="metric" aria-selected="true">
|
||||
<span>01</span><b>指标与校准</b><small>PPL · pass@k · abstain</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-eval-tab="judge" aria-selected="false" tabindex="-1">
|
||||
<span>02</span><b>Judge 与 Arena</b><small>order · length · uncertainty</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-eval-tab="contamination" aria-selected="false" tabindex="-1">
|
||||
<span>03</span><b>污染与动态题</b><small>leak · detect · refresh</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-eval-tab="system" aria-selected="false" tabindex="-1">
|
||||
<span>04</span><b>系统与安全边界</b><small>harness · budget · threat</small>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<section class="eval-view" data-eval-panel="metric">
|
||||
<div class="eval-panel-intro">
|
||||
<div><span>WORKBENCH 01 / METRIC</span><h4>同一个“80%”，可能回答四个不同问题</h4></div>
|
||||
<p>先看一次成功率，再分别计算“至少一次”“全部成功”和选择性回答；同时观察 tokenizer 对 PPL 的影响。</p>
|
||||
</div>
|
||||
|
||||
<div class="protocol-presets" role="group" aria-label="选择采样协议">
|
||||
{protocolPresets.map(([id, name, note], index) => (
|
||||
<button type="button" data-protocol-preset={id} class={index === 0 ? "active" : ""}>
|
||||
<b>{name}</b><small>{note}</small>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="eval-controls">
|
||||
<label>
|
||||
<span>单次成功率 <output data-metric-p-label>80%</output></span>
|
||||
<input data-metric-p type="range" min="5" max="99" value="80" />
|
||||
</label>
|
||||
<label>
|
||||
<span>采样池 n</span>
|
||||
<input data-metric-n type="number" min="1" max="256" value="1" />
|
||||
</label>
|
||||
<label>
|
||||
<span>允许尝试 k</span>
|
||||
<input data-metric-k type="number" min="1" max="64" value="1" />
|
||||
</label>
|
||||
<label>
|
||||
<span>平均置信度 <output data-confidence-label>88%</output></span>
|
||||
<input data-confidence type="range" min="5" max="99" value="88" />
|
||||
</label>
|
||||
<label>
|
||||
<span>拒答阈值 <output data-threshold-label>60%</output></span>
|
||||
<input data-threshold type="range" min="0" max="95" value="60" />
|
||||
</label>
|
||||
<label>
|
||||
<span>每字节 NLL</span>
|
||||
<input data-byte-nll type="number" min=".01" max="8" step=".01" value=".82" />
|
||||
</label>
|
||||
<label>
|
||||
<span>平均 bytes / token</span>
|
||||
<select data-token-bytes>
|
||||
<option value="2">2.0（细粒度）</option>
|
||||
<option value="3.4" selected>3.4（中等）</option>
|
||||
<option value="5.5">5.5（粗粒度）</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="eval-formula">
|
||||
<span>THREE DIFFERENT QUESTIONS</span>
|
||||
<code>at least one = 1 − (1 − p)<sup>k</sup> · all succeed = p<sup>k</sup> · ECE<sub>toy</sub> = |confidence − accuracy|</code>
|
||||
<p>pass@k 的严格无偏估计还需要每题 n 个样本中通过数 c；这里用独立同分布近似建立方向直觉。</p>
|
||||
</div>
|
||||
|
||||
<div class="eval-metrics five">
|
||||
<article><span>ONE TRY</span><b data-one-try>80.0%</b><p>一次随机尝试成功率</p></article>
|
||||
<article><span>AT LEAST ONE / k</span><b data-at-least>80.0%</b><p>有 verifier 时的搜索价值</p></article>
|
||||
<article><span>ALL k SUCCEED</span><b data-all-succeed>80.0%</b><p>连续任务的可靠性压力</p></article>
|
||||
<article><span>TOY ECE</span><b data-toy-ece>8.0 pp</b><p>单桶校准差，仅作直觉</p></article>
|
||||
<article class="dark"><span>SELECTIVE VIEW</span><b data-selective>80% / 100%</b><p data-selective-note>准确率 / 覆盖率</p></article>
|
||||
</div>
|
||||
|
||||
<div class="ppl-compare">
|
||||
<div>
|
||||
<span>SAME BYTE LOSS</span><b data-bpb>1.18 bits / byte</b>
|
||||
<p>按原始字节归一，跨 tokenizer 的单位更稳定。</p>
|
||||
</div>
|
||||
<div>
|
||||
<span>TOKENIZER A / PPL</span><b data-ppl-a>16.3</b>
|
||||
<p>细粒度 token：每 token 承载的字节更少。</p>
|
||||
</div>
|
||||
<div>
|
||||
<span>TOKENIZER B / PPL</span><b data-ppl-b>91.0</b>
|
||||
<p>粗粒度 token：即使 byte loss 相同，PPL 也会变。</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="boundary-note">
|
||||
<b>结论边界</b>
|
||||
<p data-metric-explain>一次成功率和多次搜索成功率目前相同，因为 k=1；把 k 调大后，搜索分会升，连续可靠性会降。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="eval-view" data-eval-panel="judge" hidden>
|
||||
<div class="eval-panel-intro">
|
||||
<div><span>WORKBENCH 02 / JUDGE</span><h4>Judge 不是尺子，它也是一个会偏的模型</h4></div>
|
||||
<p>用可见混杂做压力测试：回答顺序、长度差、自我偏好、Judge 能力与票数怎样改变胜率和不确定性。</p>
|
||||
</div>
|
||||
|
||||
<div class="judge-arena">
|
||||
<article class="answer-card answer-a">
|
||||
<span>ANSWER A</span>
|
||||
<h5>短而直接</h5>
|
||||
<p>先给结论，再列证据边界；不重复问题。</p>
|
||||
</article>
|
||||
<div aria-hidden="true">VS</div>
|
||||
<article class="answer-card answer-b">
|
||||
<span>ANSWER B</span>
|
||||
<h5>长而铺陈</h5>
|
||||
<p>提供更多背景、分点、例子与总结；可能更完整，也可能只是更长。</p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="eval-controls">
|
||||
<label>
|
||||
<span>A 的真实任务质量 <output data-quality-label>54%</output></span>
|
||||
<input data-quality type="range" min="10" max="90" value="54" />
|
||||
</label>
|
||||
<label>
|
||||
<span>B 比 A 长</span>
|
||||
<select data-length-gap>
|
||||
<option value="-40">短 40%</option>
|
||||
<option value="0">一样长</option>
|
||||
<option value="40" selected>长 40%</option>
|
||||
<option value="100">长 100%</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>长度偏好 <output data-length-bias-label>18%</output></span>
|
||||
<input data-length-bias type="range" min="0" max="40" value="18" />
|
||||
</label>
|
||||
<label>
|
||||
<span>位置偏好 <output data-position-label>8%</output></span>
|
||||
<input data-position-bias type="range" min="-30" max="30" value="8" />
|
||||
</label>
|
||||
<label>
|
||||
<span>展示顺序</span>
|
||||
<select data-order>
|
||||
<option value="a-first" selected>A 在前</option>
|
||||
<option value="b-first">B 在前</option>
|
||||
<option value="randomized">随机交换并平均</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>Judge 可靠度 <output data-judge-skill-label>82%</output></span>
|
||||
<input data-judge-skill type="range" min="50" max="99" value="82" />
|
||||
</label>
|
||||
<label>
|
||||
<span>自我偏好 <output data-self-label>5%</output></span>
|
||||
<input data-self-bias type="range" min="0" max="25" value="5" />
|
||||
</label>
|
||||
<label>
|
||||
<span>票数</span>
|
||||
<select data-votes>
|
||||
<option value="25">25</option>
|
||||
<option value="100" selected>100</option>
|
||||
<option value="1000">1,000</option>
|
||||
<option value="10000">10,000</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="eval-metrics four">
|
||||
<article><span>RAW A WIN</span><b data-raw-win>49.6%</b><p>混入顺序、长度、自偏后的观察值</p></article>
|
||||
<article><span>LENGTH-CONTROLLED</span><b data-adjusted-win>56.8%</b><p>移除教学模型中的长度项</p></article>
|
||||
<article><span>ORDER FLIP</span><b data-order-flip>16.0 pp</b><p>交换先后可造成的差</p></article>
|
||||
<article class="dark"><span>95% INTERVAL</span><b data-judge-ci>39.8–59.4%</b><p>二项近似；真实 Arena 图更复杂</p></article>
|
||||
</div>
|
||||
|
||||
<div class="rank-rail">
|
||||
<div><span>0%</span><span>50%</span><span>100%</span></div>
|
||||
<div class="rank-track"><i data-rank-raw style="width:49.6%"></i><b data-rank-adjusted style="left:56.8%"></b></div>
|
||||
<p><i></i> 观察胜率 <b></b> 长度控制后的估计</p>
|
||||
</div>
|
||||
|
||||
<div class="boundary-note">
|
||||
<b>元评测提醒</b>
|
||||
<p data-judge-explain>当前协议里，较长的 B 获得长度红利；交换顺序或控制长度，排名可能翻转。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="eval-view" data-eval-panel="contamination" hidden>
|
||||
<div class="eval-panel-intro">
|
||||
<div><span>WORKBENCH 03 / DATA</span><h4>“没搜到 n-gram”不能推出“没有污染”</h4></div>
|
||||
<p>把五层泄漏、检测能力、题目刷新与历史锚点分开，观察新鲜度和纵向可比性的冲突。</p>
|
||||
</div>
|
||||
|
||||
<div class="leak-stack">
|
||||
<label><input data-leak="verbatim" type="checkbox" checked /><span>01</span><b>原文泄漏</b><small>测试题文本进入训练</small></label>
|
||||
<label><input data-leak="answer" type="checkbox" checked /><span>02</span><b>答案泄漏</b><small>题—答、解析或单测进入训练</small></label>
|
||||
<label><input data-leak="format" type="checkbox" /><span>03</span><b>格式泄漏</b><small>模板、选项或 verbalizer 重复</small></label>
|
||||
<label><input data-leak="semantic" type="checkbox" /><span>04</span><b>语义泄漏</b><small>改写题、同事实或同算法</small></label>
|
||||
<label><input data-leak="temporal" type="checkbox" /><span>05</span><b>时间泄漏</b><small>发布后进入 SFT / RL / 合成数据</small></label>
|
||||
</div>
|
||||
|
||||
<div class="eval-controls">
|
||||
<label>
|
||||
<span>受影响题目 <output data-leak-rate-label>18%</output></span>
|
||||
<input data-leak-rate type="range" min="0" max="80" value="18" />
|
||||
</label>
|
||||
<label>
|
||||
<span>污染带来的表观增益 <output data-leak-boost-label>25 pp</output></span>
|
||||
<input data-leak-boost type="range" min="0" max="60" value="25" />
|
||||
</label>
|
||||
<label>
|
||||
<span>n-gram 检测阈值</span>
|
||||
<select data-ngram>
|
||||
<option value="8">8-gram（宽松）</option>
|
||||
<option value="13" selected>13-gram（中等）</option>
|
||||
<option value="30">30-gram（严格匹配）</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>每月刷新题目 <output data-refresh-label>30%</output></span>
|
||||
<input data-refresh type="range" min="0" max="100" value="30" />
|
||||
</label>
|
||||
<label>
|
||||
<span>固定锚点占比 <output data-anchor-label>40%</output></span>
|
||||
<input data-anchor type="range" min="0" max="100" value="40" />
|
||||
</label>
|
||||
<label>
|
||||
<span>标签错误率 <output data-label-error-label>4%</output></span>
|
||||
<input data-label-error type="range" min="0" max="20" value="4" />
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="eval-metrics five">
|
||||
<article><span>TRUE CLEAN</span><b data-true-clean>82%</b><p>教学设定中未受泄漏影响的题</p></article>
|
||||
<article><span>N-GRAM DETECTED</span><b data-detected>23%</b><p>五层污染中可被当前扫描发现的比例</p></article>
|
||||
<article><span>HIDDEN LEAK</span><b data-hidden-leak>13.9%</b><p>存在但未被当前方法捕获</p></article>
|
||||
<article><span>FRESHNESS</span><b data-freshness>51%</b><p>刷新速度与未泄漏比例的教学合成</p></article>
|
||||
<article class="dark"><span>COMPARABILITY</span><b data-comparability>70%</b><p>固定锚点越多，跨月纵向比较越稳</p></article>
|
||||
</div>
|
||||
|
||||
<div class="freshness-matrix">
|
||||
<div><span>公开静态集</span><b>可复跑强</b><small>污染与饱和风险随时间上升</small></div>
|
||||
<div><span>纯动态集</span><b>新鲜度强</b><small>月份间难度可能漂移</small></div>
|
||||
<div class="recommended"><span>版本化动态 + 锚点</span><b>同时留两本账</b><small>报告版本、刷新率和 anchor 表现</small></div>
|
||||
</div>
|
||||
|
||||
<div class="boundary-note">
|
||||
<b>检测边界</b>
|
||||
<p data-contamination-explain>当前只启用原文与答案泄漏；n-gram 对原文较敏感，对答案代码与语义改写仍不充分。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="eval-view" data-eval-panel="system" hidden>
|
||||
<div class="eval-panel-intro">
|
||||
<div><span>WORKBENCH 04 / SYSTEM</span><h4>Agent 分数属于“模型 × Harness × 环境”，安全分也属于威胁模型</h4></div>
|
||||
<p>把模型、脚手架、工具、预算、Verifier、产品安全壳和攻击强度放到同一张成本—能力—风险图。</p>
|
||||
</div>
|
||||
|
||||
<div class="system-equation">
|
||||
<div><span>MODEL</span><b>策略与表征</b></div><i>×</i>
|
||||
<div><span>HARNESS</span><b>循环与上下文</b></div><i>×</i>
|
||||
<div><span>TOOLS</span><b>权限与反馈</b></div><i>×</i>
|
||||
<div><span>ENV</span><b>版本与状态</b></div><i>×</i>
|
||||
<div><span>VERIFIER</span><b>成功定义</b></div>
|
||||
</div>
|
||||
|
||||
<div class="eval-controls">
|
||||
<label>
|
||||
<span>模型一次任务能力 <output data-model-label>58%</output></span>
|
||||
<input data-model type="range" min="5" max="95" value="58" />
|
||||
</label>
|
||||
<label>
|
||||
<span>Harness 增益 <output data-harness-label>12 pp</output></span>
|
||||
<input data-harness type="range" min="-20" max="35" value="12" />
|
||||
</label>
|
||||
<label>
|
||||
<span>工具可用性 <output data-tools-label>92%</output></span>
|
||||
<input data-tools type="range" min="20" max="100" value="92" />
|
||||
</label>
|
||||
<label>
|
||||
<span>最大重试</span>
|
||||
<select data-retries>
|
||||
<option value="1">1 次</option>
|
||||
<option value="4" selected>4 次</option>
|
||||
<option value="16">16 次</option>
|
||||
<option value="64">64 次</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>单次 Token 预算</span>
|
||||
<select data-budget>
|
||||
<option value="4">4K</option>
|
||||
<option value="32" selected>32K</option>
|
||||
<option value="128">128K</option>
|
||||
<option value="512">512K</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>Verifier 覆盖 <output data-verifier-label>80%</output></span>
|
||||
<input data-verifier type="range" min="20" max="100" value="80" />
|
||||
</label>
|
||||
<label>
|
||||
<span>产品安全壳强度 <output data-wrapper-label>72%</output></span>
|
||||
<input data-wrapper type="range" min="0" max="100" value="72" />
|
||||
</label>
|
||||
<label>
|
||||
<span>攻击者预算 <output data-attack-label>45%</output></span>
|
||||
<input data-attack type="range" min="0" max="100" value="45" />
|
||||
</label>
|
||||
<label>
|
||||
<span>无害请求敏感度 <output data-benign-label>24%</output></span>
|
||||
<input data-benign type="range" min="0" max="100" value="24" />
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="eval-metrics five">
|
||||
<article><span>MODEL ONLY</span><b data-model-only>58.0%</b><p>不含 Harness / 工具 / 多试</p></article>
|
||||
<article><span>SYSTEM SUCCESS</span><b data-system-success>92.9%</b><p>教学独立近似，不能当真实复跑</p></article>
|
||||
<article><span>COST INDEX</span><b data-cost>128</b><p>重试 × Token 预算的相对成本</p></article>
|
||||
<article><span>UNSAFE COMPLIANCE</span><b data-unsafe>9.2%</b><p>在当前攻击与安全壳设定下</p></article>
|
||||
<article class="dark"><span>OVER-REFUSAL</span><b data-overrefusal>17.3%</b><p>安全壳也会损失无害可用性</p></article>
|
||||
</div>
|
||||
|
||||
<div class="frontier-plot" aria-label="能力、成本与风险教学图">
|
||||
<div class="axis y">能力 ↑</div>
|
||||
<div class="axis x">成本 →</div>
|
||||
<div class="risk-zone">高风险区</div>
|
||||
<i data-frontier-point style="left:58%;bottom:70%"><span>当前系统</span></i>
|
||||
</div>
|
||||
|
||||
<div class="boundary-note">
|
||||
<b>归因规则</b>
|
||||
<p data-system-explain>四次尝试和 32K 预算把系统成功率推高，但应同时披露成本；安全壳压低危险服从，也带来过拒。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<figcaption>
|
||||
<b>怎么使用：</b>先改变一个变量，口述“我究竟换了模型、测量协议、系统预算还是威胁模型”；
|
||||
如果一句比较没有这些字段，就先把结论降级为“在该公开设置下的观察值”。
|
||||
</figcaption>
|
||||
</figure>
|
||||
|
||||
<script>
|
||||
const lab = document.querySelector<HTMLElement>("[data-evaluation-lab]");
|
||||
|
||||
if (lab && lab.dataset.bound !== "true") {
|
||||
lab.dataset.bound = "true";
|
||||
const tabs = [...lab.querySelectorAll<HTMLButtonElement>("[data-eval-tab]")];
|
||||
const panels = [...lab.querySelectorAll<HTMLElement>("[data-eval-panel]")];
|
||||
|
||||
const activate = (id: string) => {
|
||||
tabs.forEach((tab) => {
|
||||
const active = tab.dataset.evalTab === id;
|
||||
tab.setAttribute("aria-selected", String(active));
|
||||
tab.tabIndex = active ? 0 : -1;
|
||||
});
|
||||
panels.forEach((panel) => { panel.hidden = panel.dataset.evalPanel !== id; });
|
||||
};
|
||||
|
||||
tabs.forEach((tab, index) => {
|
||||
tab.addEventListener("click", () => activate(tab.dataset.evalTab ?? "metric"));
|
||||
tab.addEventListener("keydown", (event) => {
|
||||
if (!["ArrowLeft", "ArrowRight", "Home", "End"].includes(event.key)) return;
|
||||
event.preventDefault();
|
||||
let next = index;
|
||||
if (event.key === "ArrowRight") next = (index + 1) % tabs.length;
|
||||
if (event.key === "ArrowLeft") next = (index - 1 + tabs.length) % tabs.length;
|
||||
if (event.key === "Home") next = 0;
|
||||
if (event.key === "End") next = tabs.length - 1;
|
||||
tabs[next].focus();
|
||||
activate(tabs[next].dataset.evalTab ?? "metric");
|
||||
});
|
||||
});
|
||||
|
||||
const q = <T extends HTMLElement>(selector: string) => lab.querySelector<T>(selector);
|
||||
const value = (selector: string) => Number((q<HTMLInputElement | HTMLSelectElement>(selector))?.value ?? 0);
|
||||
const set = (selector: string, text: string) => {
|
||||
const node = q<HTMLElement>(selector);
|
||||
if (node) node.textContent = text;
|
||||
};
|
||||
const clamp = (number: number, min = 0, max = 1) => Math.max(min, Math.min(max, number));
|
||||
|
||||
const updateMetric = () => {
|
||||
const p = value("[data-metric-p]") / 100;
|
||||
const n = Math.max(1, value("[data-metric-n]"));
|
||||
const k = Math.max(1, Math.min(value("[data-metric-k]"), n));
|
||||
const confidence = value("[data-confidence]") / 100;
|
||||
const threshold = value("[data-threshold]") / 100;
|
||||
const byteNll = value("[data-byte-nll]");
|
||||
const tokenBytes = value("[data-token-bytes]");
|
||||
const atLeast = 1 - Math.pow(1 - p, k);
|
||||
const all = Math.pow(p, k);
|
||||
const ece = Math.abs(confidence - p);
|
||||
const coverage = clamp(1 - threshold * .72);
|
||||
const selective = clamp(p + threshold * (1 - p) * .42);
|
||||
const bpb = byteNll / Math.log(2);
|
||||
const pplA = Math.exp(byteNll * 2);
|
||||
const pplB = Math.exp(byteNll * tokenBytes);
|
||||
|
||||
set("[data-metric-p-label]", `${Math.round(p * 100)}%`);
|
||||
set("[data-confidence-label]", `${Math.round(confidence * 100)}%`);
|
||||
set("[data-threshold-label]", `${Math.round(threshold * 100)}%`);
|
||||
set("[data-one-try]", `${(p * 100).toFixed(1)}%`);
|
||||
set("[data-at-least]", `${(atLeast * 100).toFixed(1)}%`);
|
||||
set("[data-all-succeed]", `${(all * 100).toFixed(1)}%`);
|
||||
set("[data-toy-ece]", `${(ece * 100).toFixed(1)} pp`);
|
||||
set("[data-selective]", `${Math.round(selective * 100)}% / ${Math.round(coverage * 100)}%`);
|
||||
set("[data-bpb]", `${bpb.toFixed(2)} bits / byte`);
|
||||
set("[data-ppl-a]", pplA.toFixed(1));
|
||||
set("[data-ppl-b]", pplB.toFixed(1));
|
||||
set("[data-metric-explain]", k === 1
|
||||
? "k=1 时，一次成功、至少一次成功和全部成功是同一个问题；先把 k 调大。"
|
||||
: `把 k 提到 ${k} 后，“至少一次”升到 ${(atLeast * 100).toFixed(1)}%，但“连续全部成功”降到 ${(all * 100).toFixed(1)}%。这就是搜索能力和可靠性不能混报。`);
|
||||
};
|
||||
|
||||
lab.querySelectorAll("[data-protocol-preset]").forEach((node) => {
|
||||
node.addEventListener("click", () => {
|
||||
lab.querySelectorAll("[data-protocol-preset]").forEach((item) => item.classList.remove("active"));
|
||||
node.classList.add("active");
|
||||
const id = (node as HTMLElement).dataset.protocolPreset;
|
||||
const settings: Record<string, [number, number]> = {
|
||||
greedy: [1, 1], sample: [64, 1], search: [64, 8], reliable: [5, 5],
|
||||
};
|
||||
const [n, k] = settings[id ?? "greedy"];
|
||||
const nInput = q<HTMLInputElement>("[data-metric-n]");
|
||||
const kInput = q<HTMLInputElement>("[data-metric-k]");
|
||||
if (nInput) nInput.value = String(n);
|
||||
if (kInput) kInput.value = String(k);
|
||||
updateMetric();
|
||||
});
|
||||
});
|
||||
lab.querySelectorAll("[data-eval-panel='metric'] input,[data-eval-panel='metric'] select")
|
||||
.forEach((node) => node.addEventListener("input", updateMetric));
|
||||
|
||||
const updateJudge = () => {
|
||||
const quality = value("[data-quality]") / 100;
|
||||
const gap = value("[data-length-gap]") / 100;
|
||||
const lengthBias = value("[data-length-bias]") / 100;
|
||||
const position = value("[data-position-bias]") / 100;
|
||||
const order = q<HTMLSelectElement>("[data-order]")?.value ?? "a-first";
|
||||
const skill = value("[data-judge-skill]") / 100;
|
||||
const selfBias = value("[data-self-bias]") / 100;
|
||||
const votes = Math.max(1, value("[data-votes]"));
|
||||
const orderTerm = order === "randomized" ? 0 : position * (order === "a-first" ? 1 : -1);
|
||||
const lengthTerm = -gap * lengthBias;
|
||||
const signal = (quality - .5) * (skill * 1.5) + orderTerm + lengthTerm + selfBias * .25;
|
||||
const raw = clamp(.5 + signal);
|
||||
const adjusted = clamp(raw - lengthTerm);
|
||||
const se = Math.sqrt(raw * (1 - raw) / votes);
|
||||
const low = clamp(raw - 1.96 * se);
|
||||
const high = clamp(raw + 1.96 * se);
|
||||
|
||||
set("[data-quality-label]", `${Math.round(quality * 100)}%`);
|
||||
set("[data-length-bias-label]", `${Math.round(lengthBias * 100)}%`);
|
||||
set("[data-position-label]", `${Math.round(position * 100)}%`);
|
||||
set("[data-judge-skill-label]", `${Math.round(skill * 100)}%`);
|
||||
set("[data-self-label]", `${Math.round(selfBias * 100)}%`);
|
||||
set("[data-raw-win]", `${(raw * 100).toFixed(1)}%`);
|
||||
set("[data-adjusted-win]", `${(adjusted * 100).toFixed(1)}%`);
|
||||
set("[data-order-flip]", `${(Math.abs(position) * 200).toFixed(1)} pp`);
|
||||
set("[data-judge-ci]", `${(low * 100).toFixed(1)}–${(high * 100).toFixed(1)}%`);
|
||||
const rawBar = q<HTMLElement>("[data-rank-raw]");
|
||||
const adjustedMark = q<HTMLElement>("[data-rank-adjusted]");
|
||||
if (rawBar) rawBar.style.width = `${raw * 100}%`;
|
||||
if (adjustedMark) adjustedMark.style.left = `${adjusted * 100}%`;
|
||||
set("[data-judge-explain]", order === "randomized"
|
||||
? `随机交换顺序已抵消教学模型里的位置项；控制长度后，A 的估计胜率从 ${(raw * 100).toFixed(1)}% 变为 ${(adjusted * 100).toFixed(1)}%。`
|
||||
: `当前展示顺序贡献 ${(orderTerm * 100).toFixed(1)} pp，长度项贡献 ${(lengthTerm * 100).toFixed(1)} pp；交换顺序或控制长度都可能改变结论。`);
|
||||
};
|
||||
lab.querySelectorAll("[data-eval-panel='judge'] input,[data-eval-panel='judge'] select")
|
||||
.forEach((node) => node.addEventListener("input", updateJudge));
|
||||
|
||||
const detectability: Record<string, number> = {
|
||||
verbatim: .9, answer: .46, format: .16, semantic: .05, temporal: .08,
|
||||
};
|
||||
const updateContamination = () => {
|
||||
const active = [...lab.querySelectorAll<HTMLInputElement>("[data-leak]:checked")]
|
||||
.map((input) => input.dataset.leak ?? "");
|
||||
const leakRate = value("[data-leak-rate]") / 100;
|
||||
const boost = value("[data-leak-boost]");
|
||||
const ngram = value("[data-ngram]");
|
||||
const refresh = value("[data-refresh]") / 100;
|
||||
const anchor = value("[data-anchor]") / 100;
|
||||
const labelError = value("[data-label-error]") / 100;
|
||||
const avgDetect = active.length
|
||||
? active.reduce((sum, id) => sum + (detectability[id] ?? 0), 0) / active.length
|
||||
: 0;
|
||||
const thresholdFactor = clamp(1.18 - ngram / 50, .4, 1.05);
|
||||
const detected = clamp(avgDetect * thresholdFactor);
|
||||
const hidden = leakRate * (1 - detected);
|
||||
const freshness = clamp((1 - leakRate) * .35 + refresh * .65 - labelError * .4);
|
||||
const comparability = clamp(anchor * .75 + (1 - refresh) * .25);
|
||||
|
||||
set("[data-leak-rate-label]", `${Math.round(leakRate * 100)}%`);
|
||||
set("[data-leak-boost-label]", `${Math.round(boost)} pp`);
|
||||
set("[data-refresh-label]", `${Math.round(refresh * 100)}%`);
|
||||
set("[data-anchor-label]", `${Math.round(anchor * 100)}%`);
|
||||
set("[data-label-error-label]", `${Math.round(labelError * 100)}%`);
|
||||
set("[data-true-clean]", `${Math.round((1 - leakRate) * 100)}%`);
|
||||
set("[data-detected]", `${Math.round(detected * 100)}%`);
|
||||
set("[data-hidden-leak]", `${(hidden * 100).toFixed(1)}%`);
|
||||
set("[data-freshness]", `${Math.round(freshness * 100)}%`);
|
||||
set("[data-comparability]", `${Math.round(comparability * 100)}%`);
|
||||
const names: Record<string, string> = {
|
||||
verbatim: "原文", answer: "答案", format: "格式", semantic: "语义", temporal: "时间",
|
||||
};
|
||||
set("[data-contamination-explain]", active.length
|
||||
? `已启用${active.map((id) => names[id]).join("、")}泄漏。当前 n-gram 扫描平均只能捕获约 ${Math.round(detected * 100)}%；“未命中”不能排除剩余层级。`
|
||||
: "尚未启用污染层；这只代表教学设定为零，不代表现实训练数据已经被证明干净。");
|
||||
};
|
||||
lab.querySelectorAll("[data-eval-panel='contamination'] input,[data-eval-panel='contamination'] select")
|
||||
.forEach((node) => node.addEventListener("input", updateContamination));
|
||||
|
||||
const updateSystem = () => {
|
||||
const model = value("[data-model]") / 100;
|
||||
const harness = value("[data-harness]") / 100;
|
||||
const tools = value("[data-tools]") / 100;
|
||||
const retries = Math.max(1, value("[data-retries]"));
|
||||
const budget = value("[data-budget]");
|
||||
const verifier = value("[data-verifier]") / 100;
|
||||
const wrapper = value("[data-wrapper]") / 100;
|
||||
const attack = value("[data-attack]") / 100;
|
||||
const benign = value("[data-benign]") / 100;
|
||||
const perTry = clamp((model + harness) * tools);
|
||||
const searched = 1 - Math.pow(1 - perTry, retries);
|
||||
const observable = clamp(searched * (.72 + verifier * .28));
|
||||
const unsafe = clamp((model * .35 + attack * .65) * Math.pow(1 - wrapper, 1.35));
|
||||
const overrefusal = clamp(wrapper * benign);
|
||||
const cost = retries * budget;
|
||||
|
||||
set("[data-model-label]", `${Math.round(model * 100)}%`);
|
||||
set("[data-harness-label]", `${Math.round(harness * 100)} pp`);
|
||||
set("[data-tools-label]", `${Math.round(tools * 100)}%`);
|
||||
set("[data-verifier-label]", `${Math.round(verifier * 100)}%`);
|
||||
set("[data-wrapper-label]", `${Math.round(wrapper * 100)}%`);
|
||||
set("[data-attack-label]", `${Math.round(attack * 100)}%`);
|
||||
set("[data-benign-label]", `${Math.round(benign * 100)}%`);
|
||||
set("[data-model-only]", `${(model * 100).toFixed(1)}%`);
|
||||
set("[data-system-success]", `${(observable * 100).toFixed(1)}%`);
|
||||
set("[data-cost]", String(cost));
|
||||
set("[data-unsafe]", `${(unsafe * 100).toFixed(1)}%`);
|
||||
set("[data-overrefusal]", `${(overrefusal * 100).toFixed(1)}%`);
|
||||
const point = q<HTMLElement>("[data-frontier-point]");
|
||||
if (point) {
|
||||
point.style.left = `${clamp(Math.log2(cost + 1) / 11) * 100}%`;
|
||||
point.style.bottom = `${observable * 80 + 5}%`;
|
||||
}
|
||||
set("[data-system-explain]", `${retries} 次尝试与 ${budget}K 单次预算把系统成功率估到 ${(observable * 100).toFixed(1)}%，相对成本为 ${cost}；安全壳同时留下 ${(unsafe * 100).toFixed(1)}% 危险服从与 ${(overrefusal * 100).toFixed(1)}% 过拒。`);
|
||||
};
|
||||
lab.querySelectorAll("[data-eval-panel='system'] input,[data-eval-panel='system'] select")
|
||||
.forEach((node) => node.addEventListener("input", updateSystem));
|
||||
|
||||
updateMetric();
|
||||
updateJudge();
|
||||
updateContamination();
|
||||
updateSystem();
|
||||
}
|
||||
</script>
|
||||
|
||||
<style>
|
||||
.eval-lab{margin:42px 0 0;border:1px solid rgba(201,139,85,.38);background:#edf0e9;color:#13242b;box-shadow:0 22px 70px rgba(4,18,22,.15);overflow:hidden}
|
||||
.eval-lab-head{display:grid;grid-template-columns:1.05fr .95fr;gap:28px;padding:30px;background:#0d252c;color:#eef0e8;border-bottom:1px solid rgba(255,255,255,.13)}
|
||||
.eval-lab-head p{margin:0;color:#aebdbb;font-size:.84rem;line-height:1.65}
|
||||
.eval-lab-head>div p{color:#d69357;font:700 .66rem/1.2 var(--font-mono);letter-spacing:.12em}
|
||||
.eval-lab-head h3{margin:8px 0 0;color:#fff;font-size:clamp(1.4rem,3vw,2.15rem);letter-spacing:-.04em}
|
||||
.eval-tabs{display:grid;grid-template-columns:repeat(4,1fr);background:#17323a;border-bottom:1px solid #385058}
|
||||
.eval-tabs button{min-width:0;overflow:hidden;padding:17px 16px;border:0;border-right:1px solid #385058;background:transparent;color:#a8b7b5;text-align:left;cursor:pointer}
|
||||
.eval-tabs button:last-child{border-right:0}
|
||||
.eval-tabs button[aria-selected=true]{background:#edf0e9;color:#13242b;box-shadow:inset 0 4px #d08b50}
|
||||
.eval-tabs span,.eval-tabs small{display:block;font:600 .58rem/1.3 var(--font-mono);letter-spacing:.08em}
|
||||
.eval-tabs b{display:block;margin:5px 0;font-size:.9rem}
|
||||
.eval-tabs small{overflow:hidden;opacity:.68;white-space:nowrap;text-overflow:ellipsis}
|
||||
.eval-view{padding:30px}
|
||||
.eval-panel-intro{display:grid;grid-template-columns:1fr 1fr;gap:28px;align-items:end;padding-bottom:22px;border-bottom:1px solid #bdc8c3}
|
||||
.eval-panel-intro span{font:700 .61rem/1 var(--font-mono);letter-spacing:.1em;color:#a75d2e}
|
||||
.eval-panel-intro h4{margin:7px 0 0;font-size:clamp(1.25rem,2.4vw,1.8rem);letter-spacing:-.035em}
|
||||
.eval-panel-intro p{margin:0;color:#53676a;font-size:.88rem;line-height:1.7}
|
||||
.protocol-presets{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:22px 0}
|
||||
.protocol-presets button{padding:13px;border:1px solid #b5c2be;background:#f7f7f1;color:#24383d;text-align:left;cursor:pointer}
|
||||
.protocol-presets button.active{border-color:#c6783f;background:#fff1e4;box-shadow:inset 3px 0 #c6783f}
|
||||
.protocol-presets b,.protocol-presets small{display:block}.protocol-presets b{font-size:.8rem}.protocol-presets small{margin-top:4px;color:#69797a;font-size:.68rem}
|
||||
.eval-controls{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:10px;margin:22px 0}
|
||||
.eval-controls label{display:flex;min-width:0;flex-direction:column;justify-content:space-between;gap:9px;padding:12px;border:1px solid #c0cac6;background:#f9faf5}
|
||||
.eval-controls label>span{min-height:2.1em;font:700 .66rem/1.45 var(--font-mono);letter-spacing:.025em;color:#485b5e}
|
||||
.eval-controls output{color:#a95d2d}
|
||||
.eval-controls input,.eval-controls select{width:100%;box-sizing:border-box;accent-color:#b96934}
|
||||
.eval-controls input[type=number],.eval-controls select{height:34px;padding:0 8px;border:1px solid #aebbb7;background:#fff;color:#17292e}
|
||||
.eval-formula{margin:20px 0;padding:17px 19px;border-left:4px solid #c57b43;background:#fff7ec}
|
||||
.eval-formula span{font:700 .6rem/1 var(--font-mono);letter-spacing:.1em;color:#a65c2c}
|
||||
.eval-formula code{display:block;margin:10px 0;color:#16333b;white-space:normal;font-size:.88rem}
|
||||
.eval-formula p{margin:0;color:#617172;font-size:.75rem}
|
||||
.eval-metrics{display:grid;gap:8px;margin:20px 0}.eval-metrics.five{grid-template-columns:repeat(5,1fr)}.eval-metrics.four{grid-template-columns:repeat(4,1fr)}
|
||||
.eval-metrics article{min-width:0;padding:16px;border:1px solid #bbc7c3;background:#f8f9f4}
|
||||
.eval-metrics span{font:700 .58rem/1 var(--font-mono);letter-spacing:.08em;color:#6c7d7d}
|
||||
.eval-metrics b{display:block;margin:9px 0;color:#1b3339;font-size:1.22rem;white-space:nowrap}
|
||||
.eval-metrics p{margin:0;color:#69797a;font-size:.68rem;line-height:1.45}
|
||||
.eval-metrics .dark{background:#173239;color:#fff;border-color:#173239}.eval-metrics .dark span,.eval-metrics .dark p{color:#b7c5c2}.eval-metrics .dark b{color:#f0b77e}
|
||||
.ppl-compare{display:grid;grid-template-columns:repeat(3,1fr);gap:8px}
|
||||
.ppl-compare>div{padding:16px;background:#dfe6e1}.ppl-compare span{font:700 .58rem/1 var(--font-mono);color:#607271}.ppl-compare b{display:block;margin:7px 0;font-size:1rem}.ppl-compare p{margin:0;color:#647576;font-size:.7rem;line-height:1.45}
|
||||
.boundary-note{display:grid;grid-template-columns:130px 1fr;gap:18px;margin-top:20px;padding:16px 18px;border-top:1px solid #aab9b5;border-bottom:1px solid #aab9b5}
|
||||
.boundary-note b{font:700 .67rem/1.4 var(--font-mono);letter-spacing:.08em;color:#a55c2d}.boundary-note p{margin:0;color:#40585c;font-size:.8rem;line-height:1.65}
|
||||
.judge-arena{display:grid;grid-template-columns:1fr auto 1fr;gap:12px;align-items:center;margin:22px 0}.judge-arena>div{font:800 .7rem/1 var(--font-mono);color:#a26337}
|
||||
.answer-card{padding:17px;border:1px solid #bbc5c1;background:#f9faf5}.answer-card span{font:700 .6rem/1 var(--font-mono);color:#9f5f34}.answer-card h5{margin:7px 0;font-size:1rem}.answer-card p{margin:0;color:#617375;font-size:.76rem}
|
||||
.rank-rail{margin:22px 0}.rank-rail>div:first-child{display:flex;justify-content:space-between;color:#70807f;font:600 .58rem/1 var(--font-mono)}
|
||||
.rank-track{position:relative;height:15px;margin:7px 0;background:linear-gradient(90deg,#d6ded9 49.5%,#8ea09e 50%,#d6ded9 50.5%)}
|
||||
.rank-track i{position:absolute;inset:0 auto 0 0;background:#bf7541;transition:width .2s}.rank-track b{position:absolute;top:-5px;width:3px;height:25px;background:#15343c;transform:translateX(-1px)}
|
||||
.rank-rail p{margin:9px 0 0;color:#627473;font-size:.68rem}.rank-rail p i,.rank-rail p b{display:inline-block;width:12px;height:4px;background:#bf7541}.rank-rail p b{width:3px;height:12px;background:#15343c}
|
||||
.leak-stack{display:grid;grid-template-columns:repeat(5,1fr);gap:7px;margin:22px 0}.leak-stack label{position:relative;padding:15px 12px;border:1px solid #bec9c5;background:#f8f9f4;cursor:pointer}
|
||||
.leak-stack input{position:absolute;right:10px;top:10px;accent-color:#b86d39}.leak-stack span,.leak-stack b,.leak-stack small{display:block}.leak-stack span{font:700 .58rem/1 var(--font-mono);color:#a55d31}.leak-stack b{margin:9px 0 5px;font-size:.78rem}.leak-stack small{color:#6c7b7c;font-size:.65rem;line-height:1.4}
|
||||
.leak-stack label:has(input:checked){background:#fff0e2;border-color:#c77c45;box-shadow:inset 0 3px #c77c45}
|
||||
.freshness-matrix{display:grid;grid-template-columns:repeat(3,1fr);gap:8px}.freshness-matrix>div{padding:16px;background:#e0e6e2;border-top:3px solid #95a5a2}.freshness-matrix .recommended{background:#173239;color:white;border-color:#d18b50}
|
||||
.freshness-matrix span,.freshness-matrix b,.freshness-matrix small{display:block}.freshness-matrix span{font:700 .58rem/1 var(--font-mono);letter-spacing:.05em}.freshness-matrix b{margin:8px 0;font-size:.85rem}.freshness-matrix small{font-size:.67rem;line-height:1.4;opacity:.72}
|
||||
.system-equation{display:flex;gap:7px;align-items:stretch;margin:22px 0}.system-equation>div{flex:1;padding:14px 10px;background:#dfe6e2;text-align:center}.system-equation i{align-self:center;color:#ae6a3a;font-style:normal}.system-equation span,.system-equation b{display:block}.system-equation span{font:700 .56rem/1 var(--font-mono);color:#6b7e7c}.system-equation b{margin-top:6px;font-size:.68rem}
|
||||
.frontier-plot{position:relative;height:210px;margin:22px 0 0;border-left:1px solid #7f9290;border-bottom:1px solid #7f9290;background:linear-gradient(135deg,rgba(67,119,107,.12),transparent 55%),repeating-linear-gradient(0deg,transparent 0 41px,rgba(30,67,72,.08) 41px 42px),repeating-linear-gradient(90deg,transparent 0 19.8%,rgba(30,67,72,.08) 19.8% 20%)}
|
||||
.frontier-plot .axis{position:absolute;color:#627675;font:700 .58rem/1 var(--font-mono)}.frontier-plot .axis.y{left:6px;top:7px}.frontier-plot .axis.x{right:7px;bottom:7px}.frontier-plot .risk-zone{position:absolute;right:10px;top:10px;color:#b55e35;font:700 .58rem/1 var(--font-mono)}
|
||||
.frontier-plot>i{position:absolute;width:15px;height:15px;border:3px solid #fff;border-radius:50%;background:#c66f37;box-shadow:0 0 0 2px #173239;transform:translate(-50%,50%);transition:.2s}
|
||||
.frontier-plot>i span{position:absolute;left:12px;bottom:13px;width:80px;color:#173239;font:700 .6rem/1 var(--font-mono);font-style:normal}
|
||||
.eval-lab figcaption{padding:18px 30px;background:#d9e1dc;color:#536866;font-size:.75rem;line-height:1.65;border-top:1px solid #bdc8c3}
|
||||
@media(max-width:900px){.eval-lab-head,.eval-panel-intro{grid-template-columns:1fr}.eval-tabs{grid-template-columns:repeat(2,1fr)}.eval-controls{grid-template-columns:repeat(2,1fr)}.eval-metrics.five{grid-template-columns:repeat(2,1fr)}.leak-stack{grid-template-columns:repeat(2,1fr)}.system-equation{display:grid;grid-template-columns:repeat(5,1fr)}.system-equation i{display:none}}
|
||||
@media(max-width:620px){.eval-view{padding:20px 14px}.eval-lab-head{padding:23px 18px}.eval-tabs button{padding:14px 10px}.eval-tabs small{white-space:normal}.protocol-presets,.eval-controls,.eval-metrics.five,.eval-metrics.four,.ppl-compare,.freshness-matrix{grid-template-columns:1fr}.judge-arena{grid-template-columns:1fr}.judge-arena>div{text-align:center}.leak-stack{grid-template-columns:1fr 1fr}.boundary-note{grid-template-columns:1fr}.system-equation{grid-template-columns:1fr 1fr}.eval-lab figcaption{padding:16px 18px}}
|
||||
</style>
|
||||
@@ -21,6 +21,7 @@ const items = [
|
||||
{ id: "training-systems", href: "/training-systems/", label: "训练系统" },
|
||||
{ id: "systems-inference", href: "/systems/inference/", label: "推理服务" },
|
||||
{ id: "numerics", href: "/systems/numerics/", label: "数值" },
|
||||
{ id: "evaluation", href: "/evaluation/", label: "评测安全" },
|
||||
{ id: "papers", href: "/papers/", label: "论文库" },
|
||||
{ id: "progress", href: "/progress/", label: "进度" },
|
||||
];
|
||||
|
||||
@@ -216,12 +216,12 @@ export const chapters: Chapter[] = [
|
||||
title: "评测、安全与“到底强不强”",
|
||||
kicker: "EVALUATION",
|
||||
question: "一个榜单分数，究竟测到了模型、脚手架还是预算？",
|
||||
summary: "从语言建模指标走到知识、代码、Agent 和多模态评测，识别污染、harness 与选择性报告。",
|
||||
status: "queued",
|
||||
progress: 13,
|
||||
papers: 24,
|
||||
summary: "用二十二张测量账拆开任务、指标、Prompt、采样、Harness、裁判、污染、成本与威胁模型，并逐项复原 DeepSeek 与 K3 的评测协议。",
|
||||
status: "published",
|
||||
progress: 79,
|
||||
papers: 80,
|
||||
prerequisites: ["04", "10", "12"],
|
||||
highlights: ["基准演化", "Harness", "安全边界"],
|
||||
highlights: ["二十二张测量账", "DeepSeek / K3 协议谱系", "四联实验"],
|
||||
},
|
||||
];
|
||||
|
||||
|
||||
@@ -3254,6 +3254,406 @@ export const papers: Paper[] = [
|
||||
contribution: "NVIDIA 官方 LLM 推理运行时，整合低精度 kernel、并行、KV Cache 与动态批处理。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2002,
|
||||
title: "BLEU: a Method for Automatic Evaluation of Machine Translation",
|
||||
url: "https://aclanthology.org/P02-1040/",
|
||||
topics: ["评测"],
|
||||
contribution: "用修正 n-gram precision 与长度惩罚构造可重复的机器翻译代理指标，也留下表面重叠不等于语义质量的边界。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2004,
|
||||
title: "ROUGE: A Package for Automatic Evaluation of Summaries",
|
||||
url: "https://aclanthology.org/W04-1013/",
|
||||
topics: ["评测"],
|
||||
contribution: "以参考摘要的 n-gram、最长公共子序列等召回式重叠形成摘要自动评测工具族。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2017,
|
||||
title: "On Calibration of Modern Neural Networks",
|
||||
url: "https://proceedings.mlr.press/v70/guo17a.html",
|
||||
topics: ["评测"],
|
||||
contribution: "系统刻画现代神经网络的置信度失准，并以 temperature scaling 提供简单有效的后校准基线。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2018,
|
||||
title: "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding",
|
||||
url: "https://arxiv.org/abs/1804.07461",
|
||||
topics: ["评测"],
|
||||
contribution: "把九类语言理解任务、统一接口与诊断集汇成套件，推动跨任务可比的通用表示评测。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2019,
|
||||
title: "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems",
|
||||
url: "https://arxiv.org/abs/1905.00537",
|
||||
topics: ["评测"],
|
||||
contribution: "在 GLUE 接近饱和后引入更难任务、软件工具和公开排行榜，展示基准需要随能力更新。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2020,
|
||||
title: "RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models",
|
||||
url: "https://arxiv.org/abs/2009.11462",
|
||||
topics: ["评测"],
|
||||
contribution: "用来自开放网络的自然提示测量语言模型延续毒性，揭示静态语料分布下的退化风险。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2020,
|
||||
title: "StereoSet: Measuring stereotypical bias in pretrained language models",
|
||||
url: "https://arxiv.org/abs/2004.09456",
|
||||
topics: ["评测"],
|
||||
contribution: "用句内与篇章级任务同时考察刻板关联和语言建模能力，避免只靠去偏得分奖励无意义输出。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2020,
|
||||
title: "CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models",
|
||||
url: "https://arxiv.org/abs/2010.00133",
|
||||
topics: ["评测"],
|
||||
contribution: "以最小句对比较刻板与反刻板文本的模型偏好，覆盖九类美国社会偏见。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "Evaluating Large Language Models Trained on Code",
|
||||
url: "https://arxiv.org/abs/2107.03374",
|
||||
topics: ["评测", "推理"],
|
||||
contribution: "发布 HumanEval，并以执行测试和无偏 pass@k 估计把代码评测从字符串相似度推进到功能正确性。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "TruthfulQA: Measuring How Models Mimic Human Falsehoods",
|
||||
url: "https://arxiv.org/abs/2109.07958",
|
||||
topics: ["评测"],
|
||||
contribution: "用人类常见误解设计问答集，区分信息性与真实性，测试模型是否复述训练分布中的流行谬误。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "Dynabench: Rethinking Benchmarking in NLP",
|
||||
url: "https://arxiv.org/abs/2104.14337",
|
||||
topics: ["评测"],
|
||||
contribution: "以 human-and-model-in-the-loop 持续收集当前模型会错的样本，把动态造题纳入基准生命周期。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "BBQ: A Hand-Built Bias Benchmark for Question Answering",
|
||||
url: "https://arxiv.org/abs/2110.08193",
|
||||
topics: ["评测"],
|
||||
contribution: "在歧义与消歧两类上下文中测试社会偏见，区分缺少证据时的刻板猜测和明确证据下的准确性。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Language Models (Mostly) Know What They Know",
|
||||
url: "https://arxiv.org/abs/2207.05221",
|
||||
topics: ["评测"],
|
||||
contribution: "研究语言模型对自己答案正确性的 P(True) 与 P(IK) 估计，显示自知能力可测但跨任务校准仍困难。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection",
|
||||
url: "https://arxiv.org/abs/2203.09509",
|
||||
topics: ["评测"],
|
||||
contribution: "用模型生成并由人类筛选隐含仇恨与中性句，扩大毒性分类和生成安全评测的覆盖。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Red Teaming Language Models with Language Models",
|
||||
url: "https://arxiv.org/abs/2202.03286",
|
||||
topics: ["评测"],
|
||||
contribution: "用攻击语言模型自动生成测试输入，扩展人工红队覆盖并探索训练攻击生成器的路线。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned",
|
||||
url: "https://arxiv.org/abs/2209.07858",
|
||||
topics: ["评测"],
|
||||
contribution: "报告面向对话模型的大规模人工红队方法、扩展行为与数据使用经验，强调失败发现和缓解闭环。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them",
|
||||
url: "https://arxiv.org/abs/2210.09261",
|
||||
topics: ["评测", "推理"],
|
||||
contribution: "从 BIG-bench 中筛出模型与人类差距明显的 23 个难任务，形成 BIG-Bench Hard 与 CoT 诊断坐标。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation",
|
||||
url: "https://arxiv.org/abs/2211.11501",
|
||||
topics: ["评测", "推理"],
|
||||
contribution: "以真实数据科学问题和多条件测试扩展代码生成评测，覆盖七个常用 Python 库。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena",
|
||||
url: "https://arxiv.org/abs/2306.05685",
|
||||
topics: ["评测"],
|
||||
contribution: "发布多轮 MT-Bench 与 Chatbot Arena，并系统讨论位置、冗长、自增强等 LLM Judge 偏差。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment",
|
||||
url: "https://arxiv.org/abs/2303.16634",
|
||||
topics: ["评测"],
|
||||
contribution: "以任务说明、评价标准和 CoT 作为 Judge 提示，在摘要和对话质量上研究与人类判断的一致性。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "LLMBar: An Open-Source Benchmark for Instruction-Following Evaluation",
|
||||
url: "https://arxiv.org/abs/2310.07641",
|
||||
topics: ["评测"],
|
||||
contribution: "用遵循指令但可能不华丽的回答对抗风格诱饵，专门元评测自动 evaluator 是否真正服从 rubric。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation",
|
||||
url: "https://arxiv.org/abs/2305.01210",
|
||||
topics: ["评测", "推理"],
|
||||
contribution: "以 EvalPlus 自动扩展 HumanEval 与 MBPP 测试，揭示弱单测会高估正确率并改变模型排序。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation",
|
||||
url: "https://arxiv.org/abs/2305.14251",
|
||||
topics: ["评测"],
|
||||
contribution: "把长文本拆成原子事实并逐项验证来源支持，形成细粒度事实精确率。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models",
|
||||
url: "https://arxiv.org/abs/2303.08896",
|
||||
topics: ["评测"],
|
||||
contribution: "比较同一模型多次采样的一致性，在无白盒概率和外部知识库时检测可能幻觉。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models",
|
||||
url: "https://arxiv.org/abs/2305.11747",
|
||||
topics: ["评测"],
|
||||
contribution: "构造大规模生成与人工标注的幻觉样本，覆盖问答、对话和摘要。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Universal and Transferable Adversarial Attacks on Aligned Language Models",
|
||||
url: "https://arxiv.org/abs/2307.15043",
|
||||
topics: ["评测"],
|
||||
contribution: "以贪心坐标梯度搜索通用后缀，展示对齐模型的白盒越狱可迁移性与自动化攻击面。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Jailbroken: How Does LLM Safety Training Fail?",
|
||||
url: "https://arxiv.org/abs/2307.02483",
|
||||
topics: ["评测"],
|
||||
contribution: "从竞争目标与错配泛化解释安全训练失效，并系统整理多类越狱策略。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models",
|
||||
url: "https://arxiv.org/abs/2308.01263",
|
||||
topics: ["评测"],
|
||||
contribution: "用看似敏感但实际无害的提示测量过度拒答，使安全评测同时约束有害服从和无害可用性。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs",
|
||||
url: "https://arxiv.org/abs/2308.13387",
|
||||
topics: ["评测"],
|
||||
contribution: "以五类风险和细分场景评估模型对不应直接回答请求的安全防护行为。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation",
|
||||
url: "https://arxiv.org/abs/2310.03214",
|
||||
topics: ["评测"],
|
||||
contribution: "发布可定期刷新的 FreshQA，覆盖快速变化、慢变化、不变与错误前提问题，测试知识时效与搜索增强。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation",
|
||||
url: "https://arxiv.org/abs/2302.09664",
|
||||
topics: ["评测"],
|
||||
contribution: "将语义等价回答聚类后估计语义熵，避免把同义表述差异误当成知识不确定性。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference",
|
||||
url: "https://arxiv.org/abs/2403.04132",
|
||||
topics: ["评测"],
|
||||
contribution: "把匿名随机对战、众包投票和统计排名组成开放平台，让真实用户偏好成为动态测量信号。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators",
|
||||
url: "https://arxiv.org/abs/2404.04475",
|
||||
topics: ["评测"],
|
||||
contribution: "显式控制回答长度对自动偏好分的混杂，展示 Judge 排名可被可见风格变量显著改变。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline",
|
||||
url: "https://arxiv.org/abs/2406.11939",
|
||||
topics: ["评测"],
|
||||
contribution: "从真实用户问题中筛选高区分度难题，形成更稳定的开放回答自动比较集与构建流水线。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "LiveBench: A Challenging, Contamination-Free LLM Benchmark",
|
||||
url: "https://arxiv.org/abs/2406.19314",
|
||||
topics: ["评测"],
|
||||
contribution: "从近期来源持续更新、采用客观自动判分并限制主观 Judge，探索新鲜度与可比性的平衡。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code",
|
||||
url: "https://arxiv.org/abs/2403.07974",
|
||||
topics: ["评测", "推理"],
|
||||
contribution: "以按时间发布的新竞赛题持续测试代码生成、执行、修复和自测，降低静态公开题污染。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "RULER: What's the Real Context Size of Your Long-Context Language Models?",
|
||||
url: "https://arxiv.org/abs/2404.06654",
|
||||
topics: ["评测", "长上下文"],
|
||||
contribution: "用多 needle、变量追踪与聚合任务诊断声明窗口内的真实有效上下文，而不只测单针检索。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal",
|
||||
url: "https://arxiv.org/abs/2402.04249",
|
||||
topics: ["评测"],
|
||||
contribution: "统一行为、攻击、目标模型与分类器，系统比较自动红队和稳健拒答。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "StrongREJECT: Nailing the Effectiveness of Jailbreak Attacks with Human Labeling",
|
||||
url: "https://arxiv.org/abs/2402.10260",
|
||||
topics: ["评测"],
|
||||
contribution: "区分绕过拒绝与真正提供有效有害信息，用人工标注校准 jailbreak 实效评分。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models",
|
||||
url: "https://arxiv.org/abs/2404.01318",
|
||||
topics: ["评测"],
|
||||
contribution: "标准化越狱威胁模型、行为数据、攻击提交与评测流程，改善跨攻击比较的可复现性。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents",
|
||||
url: "https://arxiv.org/abs/2403.02691",
|
||||
topics: ["评测", "Agent"],
|
||||
contribution: "把恶意指令藏入 Agent 工具返回的不可信内容，测量间接 Prompt 注入与工具副作用。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models",
|
||||
url: "https://arxiv.org/abs/2408.08926",
|
||||
topics: ["评测", "Agent"],
|
||||
contribution: "用专业网络安全任务、子任务和可执行环境评测语言模型在真实攻击链中的能力与风险。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions",
|
||||
url: "https://arxiv.org/abs/2404.13208",
|
||||
topics: ["评测", "后训练", "Agent"],
|
||||
contribution: "把 system、user 与不可信第三方指令的优先级显式化并训练，连接提示注入防御与行为评测。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation",
|
||||
url: "https://arxiv.org/abs/2412.03304",
|
||||
topics: ["评测"],
|
||||
contribution: "重构并翻译多学科题，分析文化敏感内容与机器翻译伪影，扩展多语言评测的效度边界。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI",
|
||||
url: "https://arxiv.org/abs/2411.04872",
|
||||
topics: ["评测", "推理"],
|
||||
contribution: "由专家创作高难、可验证、未公开答案的数学问题，延伸前沿推理能力的区分度。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs",
|
||||
url: "https://arxiv.org/abs/2406.18495",
|
||||
topics: ["评测"],
|
||||
contribution: "以统一数据和分类模型同时识别有害提示、有害回答与拒答，支持开放安全评测与审核。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark",
|
||||
url: "https://arxiv.org/abs/2406.01574",
|
||||
topics: ["评测"],
|
||||
contribution: "增加推理题、扩展到十个选项并清理噪声，缓解 MMLU 饱和与提示敏感。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Are We Done with MMLU?",
|
||||
url: "https://arxiv.org/abs/2406.04127",
|
||||
topics: ["评测"],
|
||||
contribution: "重新标注 MMLU 全集并分析错误、歧义与题目质量，说明静态权威基准也需要系统维护。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons",
|
||||
url: "https://arxiv.org/abs/2503.05731",
|
||||
topics: ["评测"],
|
||||
contribution: "以标准化危险类别和等级报告通用聊天模型风险，同时明确通过测试不能证明系统安全。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "SWE-bench Goes Live!",
|
||||
url: "https://arxiv.org/abs/2505.23419",
|
||||
topics: ["评测", "Agent"],
|
||||
contribution: "从新近 GitHub issue 持续构建可执行软件任务，以版本化动态集降低仓库级代码评测污染。",
|
||||
verified: true,
|
||||
},
|
||||
];
|
||||
|
||||
export const paperTopics: PaperTopic[] = [
|
||||
|
||||
@@ -0,0 +1,974 @@
|
||||
---
|
||||
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||
import EvaluationLab from "@/components/EvaluationLab.astro";
|
||||
|
||||
const toc = [
|
||||
["00", "compass", "先拆成二十二张账"],
|
||||
["01", "measurement", "分数是一份测量协议"],
|
||||
["02", "history", "七次历史转向"],
|
||||
["03", "token-metrics", "PPL、BLEU 与 ROUGE"],
|
||||
["04", "suites", "从 GLUE 到 HELM"],
|
||||
["05", "lifecycle", "基准也有生命周期"],
|
||||
["06", "prompt", "Prompt 与答案格式"],
|
||||
["07", "sampling", "pass@k 与 pass^k"],
|
||||
["08", "calibration", "校准与选择性回答"],
|
||||
["09", "contamination", "污染的五个层级"],
|
||||
["10", "dynamic", "动态题与历史锚点"],
|
||||
["11", "verifier", "代码 Verifier"],
|
||||
["12", "swe", "SWE-bench 的维修史"],
|
||||
["13", "long-multi", "长上下文与多模态"],
|
||||
["14", "agent-unit", "Agent 的测量单位"],
|
||||
["15", "final-state", "最终状态与可靠性"],
|
||||
["16", "human", "人类评测"],
|
||||
["17", "judge", "LLM-as-a-Judge"],
|
||||
["18", "arena", "Arena、排名与区间"],
|
||||
["19", "aggregation", "聚合与子群"],
|
||||
["20", "cost", "成本归一化前沿"],
|
||||
["21", "safety-object", "安全到底在测谁"],
|
||||
["22", "threat", "先写威胁模型"],
|
||||
["23", "safety-lineage", "安全评测谱系"],
|
||||
["24", "jailbreak", "越狱成功不等于有害有效"],
|
||||
["25", "agent-security", "注入、权限与网络安全"],
|
||||
["26", "overrefusal", "拒答与过度拒答"],
|
||||
["27", "deepseek", "DeepSeek 评测谱系"],
|
||||
["28", "r1", "R1：采样与安全壳"],
|
||||
["29", "k3", "K3 §6 逐字段复原"],
|
||||
["30", "lab", "四联交互实验"],
|
||||
["31", "audit", "一张可复用审计卡"],
|
||||
["↳", "papers", "80 个关键节点"],
|
||||
];
|
||||
|
||||
const ledgers = [
|
||||
["Q1 / CONSTRUCT", "构念", "到底想测知识、推理、行动、偏好还是风险？", "指标名不是能力本身；先写出目标能力的可观察定义。"],
|
||||
["Q2 / UNIT", "测量单位", "token、题、回答、episode 还是完整系统？", "不同单位不能因都叫百分比就直接平均。"],
|
||||
["Q3 / DISTRIBUTION", "任务分布", "样本代表哪些用户、领域、语言、难度和时间？", "测试集只是现实分布的一份带偏样本。"],
|
||||
["Q4 / METRIC", "指标", "loss、accuracy、F1、win rate、resolved 各丢掉什么？", "每个指标都把一部分目标写进了分母。"],
|
||||
["Q5 / PROMPT", "提示", "shots、CoT、模板和答案抽取是否一致？", "同名 benchmark 不会自动生成相同协议。"],
|
||||
["Q6 / DECODING", "解码", "temperature、top-p、长度与 seed 怎样进入成绩？", "解码是被测系统的一部分，不是脚注。"],
|
||||
["Q7 / SAMPLING", "重复采样", "pass@1、pass@k、cons@k 与 pass^k 测什么？", "搜索能力、投票能力和可靠性是三个方向。"],
|
||||
["Q8 / HARNESS", "脚手架", "system prompt、工具和上下文管理贡献多少？", "Agent 分数不能全部归到 checkpoint。"],
|
||||
["Q9 / VERSION", "环境版本", "数据、容器、网站、依赖和 verifier 是哪个快照？", "动态环境会漂移，复跑必须带日期。"],
|
||||
["Q10 / JUDGE", "裁判", "exact match、单测、人类和 LLM Judge 谁定义好？", "裁判也有误差、偏好和覆盖盲区。"],
|
||||
["Q11 / AGGREGATE", "聚合", "macro、micro、权重和缺失项怎样处理？", "总分永远包含价值判断。"],
|
||||
["Q12 / UNCERTAINTY", "不确定性", "样本数、方差和区间足以支持排名吗？", "0.1 分差不等于确定的能力差。"],
|
||||
["Q13 / LEAK", "污染", "原文、答案、格式、语义与时间泄漏怎样分？", "n-gram 无重合只能排除一小类泄漏。"],
|
||||
["Q14 / QUALITY", "饱和与错误", "高分是能力强、题太易还是标签错？", "基准不是发布后永远正确的法规。"],
|
||||
["Q15 / FRESH", "动态性", "怎样保持新鲜，又保留跨时间可比？", "纯动态题也会引入难度漂移。"],
|
||||
["Q16 / PREFERENCE", "偏好与 Arena", "谁在投票、看见什么、被怎样配对？", "Elo 是相对排序，不是绝对智力单位。"],
|
||||
["Q17 / COST", "成本", "用了多少 token、工具、重试、延迟与金钱？", "最高准确率未必是可部署的最优点。"],
|
||||
["Q18 / ROBUST", "鲁棒与子群", "换语言、提示、长度或扰动后是否稳定？", "平均值不能代表每个用户群。"],
|
||||
["Q19 / FACT", "事实性", "长回答中的原子事实被哪些来源支持？", "流畅、自洽与有引用都不自动等于正确。"],
|
||||
["Q20 / THREAT", "威胁模型", "攻击者能改什么、访问几次、目标是什么？", "不同攻击权限下的 ASR 不能直接横比。"],
|
||||
["Q21 / BOUNDARY", "安全边界", "危险服从与无害过拒怎样同时测？", "什么都拒绝不等于安全又有用。"],
|
||||
["Q22 / DISCLOSURE", "披露与审计", "配置、输出、代码、版本和来源能否复核？", "一张排行榜不能替代完整实验记录。"],
|
||||
];
|
||||
|
||||
const waves = [
|
||||
["2002–04", "先用便宜代理加速迭代", "BLEU · ROUGE", "参考重叠让翻译与摘要可大规模回归，但表面相似不等于语义、事实和风格质量。"],
|
||||
["2018–20", "把通用性做成多任务套件", "GLUE · SuperGLUE · MMLU", "同一模型跨任务比较成为可能；任务权重、格式敏感和公开题污染开始显现。"],
|
||||
["2021–22", "整体性、动态性与风险进场", "Dynabench · BIG-bench · HELM", "evaluation 被重新写成场景、适配、指标与报告的组合。"],
|
||||
["2021–24", "可执行结果重新成为硬裁判", "HumanEval · DS-1000 · EvalPlus · LiveCodeBench", "代码和数学能用测试或 verifier 判分，但测试覆盖和规格质量仍是测量边界。"],
|
||||
["2023–24", "开放回答交给偏好与模型裁判", "MT-Bench · Arena · G-Eval · AlpacaEval", "没有唯一参考答案的问题可比较了，同时引入顺序、长度、自偏与 Judge 能力。"],
|
||||
["2023–26", "测量单位从回答变成系统", "SWE-bench · OSWorld · τ-bench · BrowseComp", "Harness、工具、环境、上下文、重试和最终状态共同决定 Agent 成绩。"],
|
||||
["2020–26", "安全从内容标签走向产品威胁模型", "RealToxicity · GCG · HarmBench · InjecAgent · AILuminate", "发现失败能证明漏洞，没发现失败却不能证明系统安全。"],
|
||||
];
|
||||
|
||||
const metricCards = [
|
||||
["NLL / PPL", "模型给测试序列多大概率？", "适合同 tokenizer、同分布的语言建模比较；不直接测指令、事实、工具或安全。"],
|
||||
["BLEU / ROUGE", "输出和参考表面上多相似？", "便宜、确定、适合回归；多种合理答案、语义与事实会被压扁。"],
|
||||
["ACCURACY / EM", "最终标签或字符串是否匹配？", "清晰但依赖答案 parser、选项顺序、标签质量和格式策略。"],
|
||||
["PASS@K", "k 次尝试里至少一次通过吗？", "衡量有 verifier 搜索的潜力，不代表默认单次体验。"],
|
||||
["WIN RATE", "在给定展示下谁更受偏好？", "依赖题分布、裁判人群、顺序、长度、风格和对手集合。"],
|
||||
["RESOLVED", "环境最终状态是否满足目标？", "更接近真实行动，但强依赖 Harness、权限、版本、预算与 verifier。"],
|
||||
];
|
||||
|
||||
const contaminationLayers = [
|
||||
["01", "原文污染", "测试题、上下文或 benchmark 文件原样出现", "exact / fuzzy n-gram 能发现一部分"],
|
||||
["02", "答案污染", "题—答配对、解析器、隐藏单测或参考补丁出现", "只搜题面会漏掉"],
|
||||
["03", "格式污染", "模板、选项位置、verbalizer 或固定答案模式出现", "文本相似度通常不充分"],
|
||||
["04", "语义污染", "改写题、同一事实、算法或等价证明出现", "需要语义、来源与人工审计"],
|
||||
["05", "时间污染", "发布后进入继续预训练、SFT、RL 或合成数据", "需要数据谱系与时间戳"],
|
||||
];
|
||||
|
||||
const deepseekRows = [
|
||||
["DeepSeek LLM / 2024", "公开集 + held-out + safety；base/chat 使用不同 shots", "证明“同一张表”内部也可能有不同 prompting；需要逐列读脚注。"],
|
||||
["DeepSeekMath / 2024", "Maj@K 提升而 Pass@K 未同步提升", "多数投票变好不等于候选覆盖变广；采样指标必须分开。"],
|
||||
["DeepSeek-V2 / 2024", "语言、代码、数学与开放对话分协议评测", "架构收益要和训练 token、激活参数、上下文及对话协议共同解释。"],
|
||||
["DeepSeek-V3 / 2024", "Base、Chat、推理、代码与安全分表", "一个 checkpoint 的单步能力与产品包装后的行为不是同一对象。"],
|
||||
["DeepSeek-R1 / 2025", "非零温多次采样的 mean pass@1；不同任务 k 不同", "表中的 pass@1 不是一次 greedy；采样池和温度属于核心结论。"],
|
||||
["DeepSeek-V3.2 / 2025", "长上下文与 reasoning effort 继续分档", "窗口声明、有效利用和测试时计算预算必须一起看。"],
|
||||
["DeepSeek-V4 / 2026", "Nonthink / High / Max；8K / 128K / 384K；Agent 可到 500 tools/steps", "所谓模型分数已显式成为 effort、context 与 Harness 条件下的系统曲线。"],
|
||||
];
|
||||
|
||||
const k3Protocol = [
|
||||
["MODEL", "Kimi K3 Thinking", "明确 checkpoint / 产品形态；不可和不同 wrapper 混称“K3”。"],
|
||||
["EFFORT", "reasoning effort = max", "比较对象必须使用同等测试时计算，或把它作为成本维度。"],
|
||||
["SAMPLING", "temperature = 1.0", "无工具单步题 top-p=.95，Agent 任务 top-p=1；不是默认 greedy。"],
|
||||
["TOOLS", "HLE 同时报无工具 / 有工具", "43.5 与 56.0 回答的是两个系统设置，不是同一个模型常数。"],
|
||||
["HARNESS", "任务采用各自脚手架", "SWE、终端、浏览、GUI 与自动化任务必须分别披露 harness。"],
|
||||
["SWE", "DeepSWE v1.1；官方 mini-SWE-agent 另报 67.3", "同模型换脚手架就是一次重要消融，不应把差异藏掉。"],
|
||||
["ENV", "H20 校准 SWE-Marathon；FrontierSWE 于 2026-07-16 重算", "硬件、环境和评测日期都是结果的一部分。"],
|
||||
["JUDGE", "部分任务使用 Gemini 3.1 Pro 裁判", "Judge 版本、prompt、rubric 与抽样复核决定开放回答效度。"],
|
||||
["DATA", "AutomationBench 600 个公开任务；内部集频繁刷新", "公开与内部、静态与动态必须用不同证据标签。"],
|
||||
["COST", "引用第三方排名与成本时保留来源和时点", "外部榜单是带日期的观察，不是永久属性。"],
|
||||
];
|
||||
|
||||
const auditRows = [
|
||||
["对象", "模型 checkpoint、产品 wrapper、Harness 与完整系统分别叫什么？"],
|
||||
["构念", "要测的能力或风险能否写成可观察行为？有哪些替代解释？"],
|
||||
["样本", "来源、版本、时间、语言、领域、难度、子群和排除规则是什么？"],
|
||||
["Prompt", "system/chat template、shots、CoT、答案格式与 parser 是否完整公开？"],
|
||||
["解码", "temperature、top-p、长度、samples、seed、effort、timeout 和 invalid policy 是什么？"],
|
||||
["工具", "工具 schema、权限、上下文管理、网络、重试、并行 rollout 与缓存怎样设？"],
|
||||
["环境", "容器、仓库、网站、依赖、数据快照与重置逻辑能否复现？"],
|
||||
["裁判", "单测、formal verifier、人类或 LLM Judge 的覆盖、偏差和版本是什么？"],
|
||||
["统计", "样本量、点估计、区间、paired test、多重比较与缺失项怎样处理？"],
|
||||
["污染", "原文、答案、格式、语义、时间五层分别有哪些证据？"],
|
||||
["可靠性", "一次成功、至少一次成功、全部重复成功和尾部失败是否分报？"],
|
||||
["安全", "威胁模型、攻击预算、有害有效性、无害回答率和过拒是否同表？"],
|
||||
["成本", "输入/输出 token、思考预算、工具、延迟、金钱和能耗是否有共同分母？"],
|
||||
["边界", "作者自报、第三方复跑、内部评测、教学推导和线上动态值是否分层？"],
|
||||
];
|
||||
|
||||
const paperChain = [
|
||||
["1951", "Perplexity / Prediction and Entropy", "https://doi.org/10.1002/j.1538-7305.1951.tb01366.x", "以序列预测和熵建立语言模型内在评测直觉。"],
|
||||
["2002", "BLEU", "https://aclanthology.org/P02-1040/", "用 n-gram precision 与长度惩罚代理翻译质量。"],
|
||||
["2004", "ROUGE", "https://aclanthology.org/W04-1013/", "以召回式参考重叠代理摘要质量。"],
|
||||
["2017", "On Calibration", "https://proceedings.mlr.press/v70/guo17a.html", "把准确率与置信度可靠性拆成两件事。"],
|
||||
["2018", "GLUE", "https://arxiv.org/abs/1804.07461", "统一多任务语言理解评测接口。"],
|
||||
["2019", "SuperGLUE", "https://arxiv.org/abs/1905.00537", "饱和后以更难任务更新测量范围。"],
|
||||
["2020", "MMLU", "https://arxiv.org/abs/2009.03300", "用多学科选择题扩大知识与问题求解覆盖。"],
|
||||
["2020", "RealToxicityPrompts", "https://arxiv.org/abs/2009.11462", "以自然提示测模型的毒性延续。"],
|
||||
["2020", "StereoSet", "https://arxiv.org/abs/2004.09456", "联合测刻板偏好与语言建模能力。"],
|
||||
["2020", "CrowS-Pairs", "https://arxiv.org/abs/2010.00133", "用最小句对测社会刻板关联。"],
|
||||
["2021", "HumanEval", "https://arxiv.org/abs/2107.03374", "执行测试与无偏 pass@k 进入代码评测。"],
|
||||
["2021", "TruthfulQA", "https://arxiv.org/abs/2109.07958", "测试模型是否复述人类常见谬误。"],
|
||||
["2021", "Dynabench", "https://arxiv.org/abs/2104.14337", "用人机闭环持续寻找当前模型难例。"],
|
||||
["2021", "BBQ", "https://arxiv.org/abs/2110.08193", "在歧义与消歧上下文中分别测偏见。"],
|
||||
["2022", "BIG-bench", "https://arxiv.org/abs/2206.04615", "大规模协作汇集数百种能力任务。"],
|
||||
["2022", "BIG-Bench Hard", "https://arxiv.org/abs/2210.09261", "从套件中筛选前沿模型仍困难的任务。"],
|
||||
["2022", "HELM", "https://arxiv.org/abs/2211.09110", "用场景、适配、指标与透明报告定义整体性评测。"],
|
||||
["2022", "Language Models Mostly Know", "https://arxiv.org/abs/2207.05221", "研究模型能否估计自身知识与答案正确性。"],
|
||||
["2022", "ToxiGen", "https://arxiv.org/abs/2203.09509", "扩展隐含仇恨与中性文本的对抗覆盖。"],
|
||||
["2022", "Red Teaming with LMs", "https://arxiv.org/abs/2202.03286", "用语言模型自动生成红队输入。"],
|
||||
["2022", "Red Teaming to Reduce Harms", "https://arxiv.org/abs/2209.07858", "总结大规模人工红队和缓解闭环。"],
|
||||
["2022", "DS-1000", "https://arxiv.org/abs/2211.11501", "用真实数据科学问题与多条件测试扩展代码评测。"],
|
||||
["2023", "PALOMA", "https://arxiv.org/abs/2312.10523", "以域、格式、去污染和 bits-per-byte 改善 LM 可比性。"],
|
||||
["2023", "Training Data Contamination", "https://arxiv.org/abs/2311.04850", "系统调查预训练数据与评测集重叠的影响。"],
|
||||
["2023", "Semantic Uncertainty", "https://arxiv.org/abs/2302.09664", "按语义等价类估计生成不确定性。"],
|
||||
["2023", "MT-Bench / LLM Judge", "https://arxiv.org/abs/2306.05685", "开放回答自动裁判与偏差成为中心问题。"],
|
||||
["2023", "G-Eval", "https://arxiv.org/abs/2303.16634", "以 rubric 和推理提示提升 NLG 自动评价。"],
|
||||
["2023", "LLMBar", "https://arxiv.org/abs/2310.07641", "用风格诱饵元评测 evaluator 的指令遵循。"],
|
||||
["2023", "EvalPlus", "https://arxiv.org/abs/2305.01210", "扩充代码测试覆盖并暴露原测试漏错。"],
|
||||
["2023", "FActScore", "https://arxiv.org/abs/2305.14251", "把长回答拆为原子事实逐项核验。"],
|
||||
["2023", "SelfCheckGPT", "https://arxiv.org/abs/2303.08896", "用黑盒多次采样不一致检测幻觉。"],
|
||||
["2023", "HaluEval", "https://arxiv.org/abs/2305.11747", "构造问答、对话和摘要幻觉数据。"],
|
||||
["2023", "GCG", "https://arxiv.org/abs/2307.15043", "自动搜索通用可迁移对抗后缀。"],
|
||||
["2023", "Jailbroken", "https://arxiv.org/abs/2307.02483", "从竞争目标和泛化解释安全训练失效。"],
|
||||
["2023", "XSTest", "https://arxiv.org/abs/2308.01263", "专门测看似敏感但应正常回答的无害请求。"],
|
||||
["2023", "Do-Not-Answer", "https://arxiv.org/abs/2308.13387", "细分不应直接回答的风险场景。"],
|
||||
["2023", "FreshLLMs / FreshQA", "https://arxiv.org/abs/2310.03214", "用持续更新问题测知识时效与搜索增强。"],
|
||||
["2023", "LongBench", "https://arxiv.org/abs/2308.14508", "以双语多任务评估长上下文能力。"],
|
||||
["2023", "GPQA", "https://arxiv.org/abs/2311.12022", "专家级、Google-proof 的研究生科学问答。"],
|
||||
["2023", "RewardBench", "https://arxiv.org/abs/2403.13787", "对偏好与奖励模型进行多类别压力测试。"],
|
||||
["2023", "AgentBench", "https://arxiv.org/abs/2308.03688", "跨八种交互环境评测 LLM Agent。"],
|
||||
["2023", "SWE-bench", "https://arxiv.org/abs/2310.06770", "以真实 GitHub issue 和仓库状态测软件修复。"],
|
||||
["2023", "OSWorld", "https://arxiv.org/abs/2404.07972", "在真实计算机环境中测多模态 GUI Agent。"],
|
||||
["2024", "Chatbot Arena", "https://arxiv.org/abs/2403.04132", "把匿名人类偏好、配对图和动态排名平台化。"],
|
||||
["2024", "Length-Controlled AlpacaEval", "https://arxiv.org/abs/2404.04475", "控制长度混杂后重新估计自动偏好。"],
|
||||
["2024", "Arena-Hard", "https://arxiv.org/abs/2406.11939", "从真实流量筛高区分度开放问题。"],
|
||||
["2024", "LiveBench", "https://arxiv.org/abs/2406.19314", "用近期来源、客观判分和月度更新降低污染。"],
|
||||
["2024", "LiveCodeBench", "https://arxiv.org/abs/2403.07974", "以新竞赛题和时间切分构建代码动态评测。"],
|
||||
["2024", "RULER", "https://arxiv.org/abs/2404.06654", "用检索、追踪和聚合诊断真实有效上下文。"],
|
||||
["2024", "HarmBench", "https://arxiv.org/abs/2402.04249", "标准化自动红队和稳健拒答框架。"],
|
||||
["2024", "StrongREJECT", "https://arxiv.org/abs/2402.10260", "区分绕过拒绝与真正有效有害回答。"],
|
||||
["2024", "JailbreakBench", "https://arxiv.org/abs/2404.01318", "统一越狱威胁模型、行为、攻击与提交。"],
|
||||
["2024", "InjecAgent", "https://arxiv.org/abs/2403.02691", "评测工具返回中的间接 Prompt 注入。"],
|
||||
["2024", "AgentDojo", "https://arxiv.org/abs/2406.13352", "在动态工具环境中联合测效用与注入安全。"],
|
||||
["2024", "Cybench", "https://arxiv.org/abs/2408.08926", "在可执行网络安全环境中测能力与风险。"],
|
||||
["2024", "Agent Security Bench", "https://arxiv.org/abs/2410.02644", "从攻击与防御维度系统评测 LLM Agent 安全。"],
|
||||
["2024", "Instruction Hierarchy", "https://arxiv.org/abs/2404.13208", "训练模型区分 privileged 与不可信指令。"],
|
||||
["2024", "Global MMLU", "https://arxiv.org/abs/2412.03304", "审计多语言翻译与文化偏差。"],
|
||||
["2024", "FrontierMath", "https://arxiv.org/abs/2411.04872", "用专家原创可验证难题延伸数学区分度。"],
|
||||
["2024", "WildGuard", "https://arxiv.org/abs/2406.18495", "统一有害提示、回答与拒答的开放审核。"],
|
||||
["2024", "MMLU-Pro", "https://arxiv.org/abs/2406.01574", "以更多选项、推理题和清理缓解饱和。"],
|
||||
["2024", "MMLU-Redux", "https://arxiv.org/abs/2406.04127", "重标全集并揭示错误与歧义。"],
|
||||
["2024", "τ-bench", "https://arxiv.org/abs/2406.12045", "在工具与用户交互中测 Agent 状态一致性。"],
|
||||
["2024", "OCRBench", "https://arxiv.org/abs/2305.07895", "细分多模态模型的 OCR 感知能力。"],
|
||||
["2023", "MMBench", "https://arxiv.org/abs/2307.06281", "用能力矩阵和循环评测诊断多模态理解。"],
|
||||
["2023", "MMMU", "https://arxiv.org/abs/2311.16502", "用大学级多学科视觉问题测专家知识与推理。"],
|
||||
["2024", "Video-MME", "https://arxiv.org/abs/2405.21075", "跨视频时长和模态信息测视频理解。"],
|
||||
["2025", "Humanity's Last Exam", "https://arxiv.org/abs/2501.14249", "用专家原创高难题继续拓展知识推理上限。"],
|
||||
["2025", "AILuminate", "https://arxiv.org/abs/2503.05731", "标准化风险类别，同时声明通过不等于安全。"],
|
||||
["2025", "SWE-bench Live", "https://arxiv.org/abs/2505.23419", "持续从新 issue 构建版本化可执行任务。"],
|
||||
["2025", "BrowseComp", "https://arxiv.org/abs/2504.12516", "用难检索、可验证问题测浏览 Agent。"],
|
||||
["2024", "DeepSeek LLM", "https://arxiv.org/abs/2401.02954", "分开 base/chat、shots、开放集与安全评测。"],
|
||||
["2024", "DeepSeekMath", "https://arxiv.org/abs/2402.03300", "用 Maj@K 与 Pass@K 的分离揭示采样协议差异。"],
|
||||
["2024", "DeepSeek-V2", "https://arxiv.org/abs/2405.04434", "在架构、训练与多类评测协议中报告效率—能力。"],
|
||||
["2024", "DeepSeek-V3", "https://arxiv.org/abs/2412.19437", "将 base、chat、推理、代码与安全分层评测。"],
|
||||
["2025", "DeepSeek-R1", "https://arxiv.org/abs/2501.12948", "公开非零温多采样 pass@1 和安全包装差异。"],
|
||||
["2025", "DeepSeek-V3.2", "https://arxiv.org/abs/2512.02556", "按长上下文与推理设置继续展开能力曲线。"],
|
||||
["2026", "DeepSeek-V4", "https://arxiv.org/abs/2606.19348", "把 effort、context、工具数和步数写进前沿 Agent 协议。"],
|
||||
["2025", "Kimi k1.5", "https://arxiv.org/abs/2501.12599", "把长上下文强化学习与测试时扩展放入同一评测框架。"],
|
||||
["2026", "Kimi K3", "https://arxiv.org/abs/2607.24653", "以 max effort、任务特定 Harness、工具和动态环境报告前沿系统结果。"],
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="评测、安全与“到底强不强”：从 PPL 到 Kimi K3"
|
||||
description="用二十二张测量账拆开 LLM 指标、Prompt、采样、污染、LLM Judge、Agent Harness、安全威胁模型，并逐字段复原 DeepSeek 与 Kimi K3 评测协议。"
|
||||
section="evaluation"
|
||||
>
|
||||
<header class="page-hero evaluation-hero">
|
||||
<div class="page-hero-inner">
|
||||
<div>
|
||||
<p class="eyebrow"><span>EVALUATION / 15</span> SCORE · PROTOCOL · THREAT MODEL</p>
|
||||
<h1>一个模型到底强不强，<br />不能只看那一个粗体数字</h1>
|
||||
<p class="lead">
|
||||
从 perplexity、BLEU 和 MMLU 走到代码 Verifier、LLM Judge、Arena、动态基准、
|
||||
Agent 最终状态与安全威胁模型；最后逐字段复原 DeepSeek-R1 与 Kimi K3 的评测协议，
|
||||
学会把“谁更强”改写成一条有限、可复核、带成本的结论。
|
||||
</p>
|
||||
</div>
|
||||
<dl class="page-facts">
|
||||
<div><dt>LEVEL</dt><dd>L0 直觉 → L3 审计</dd></div>
|
||||
<div><dt>LEDGERS</dt><dd>22 张测量账</dd></div>
|
||||
<div><dt>NODES</dt><dd>80 个一手节点</dd></div>
|
||||
<div><dt>LAB</dt><dd>4 台互动测量仪</dd></div>
|
||||
<div><dt>TIME</dt><dd>约 300–420 分钟</dd></div>
|
||||
<div><dt>VERIFIED</dt><dd>2026-07-29</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="report-shell">
|
||||
<aside class="side-rail" aria-label="本页目录">
|
||||
<p>CONTENTS / 33</p>
|
||||
<ol>
|
||||
{toc.map(([number, id, label]) => <li><a href={`#${id}`}><span>{number}</span>{label}</a></li>)}
|
||||
</ol>
|
||||
<div class="rail-note">
|
||||
<b>先记住一句</b>
|
||||
分数不是模型固有属性，而是模型在一份任务、协议、预算、环境与裁判合同下留下的观察值。
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<article class="article evaluation-article">
|
||||
<section class="article-section" id="compass">
|
||||
<p class="eyebrow"><span>00</span> TWENTY-TWO LEDGERS</p>
|
||||
<h2>看到排行榜，先把一个数字拆成二十二张账</h2>
|
||||
<p class="lede">
|
||||
“A 比 B 高 3 分”把测量对象、样本、提示、采样、工具、裁判、统计与成本压成了一个标量。
|
||||
这句话可能有用，但只有把下面二十二张账补齐，才知道它允许多强的结论。
|
||||
</p>
|
||||
|
||||
<div class="prerequisite-grid">
|
||||
<a href="/scaling/"><span>PREREQUISITE / 04</span><b>Scaling Laws</b><p>先区分能力曲线、计算预算和外推边界。</p></a>
|
||||
<a href="/post-training/alignment/"><span>PREREQUISITE / 10</span><b>指令与偏好</b><p>理解 base model、对齐行为与产品包装不是同一对象。</p></a>
|
||||
<a href="/agents/"><span>PREREQUISITE / 12</span><b>工具与长程 Agent</b><p>Agent 成绩属于模型、Harness、工具和环境的乘积。</p></a>
|
||||
</div>
|
||||
|
||||
<div class="ledger-grid eval-ledgers">
|
||||
{ledgers.map(([code, title, question, answer]) => (
|
||||
<article><span>{code}</span><h3>{title}</h3><b>{question}</b><p>{answer}</p></article>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="one-line">
|
||||
<span>ONE-SENTENCE MODEL</span>
|
||||
<p><b>评测是一项测量工程：先定义想测什么，再固定任务分布、系统边界、资源预算与裁判，最后用不确定性和证据等级限制结论。</b></p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="measurement">
|
||||
<p class="eyebrow"><span>01</span> MEASUREMENT PROTOCOL</p>
|
||||
<h2>Benchmark 不是一张题单，而是一份十二字段合同</h2>
|
||||
<p>
|
||||
在物理测量里，我们不会只写“温度 25”，却省略摄氏、测量位置、时间和仪器。
|
||||
LLM 榜单同样需要单位。最小的完整表达不是 <code>score(model)</code>，而是：
|
||||
</p>
|
||||
<div class="measurement-equation">
|
||||
<span>OBSERVED SCORE</span>
|
||||
<code>
|
||||
measure(task distribution, dataset/split, model/wrapper, prompt/shots,
|
||||
decoding/samples, effort/tools, harness/context, environment/verifier,
|
||||
judge/rubric, aggregation/uncertainty, cost)
|
||||
</code>
|
||||
</div>
|
||||
<div class="validity-triptych">
|
||||
<article><span>01</span><h3>构念效度</h3><p>我们说要测“推理”，题目是否只靠记忆或格式线索就能做对？</p><b>题目真的测到了目标吗？</b></article>
|
||||
<article><span>02</span><h3>内部效度</h3><p>差异是否被污染、Prompt、Harness、Judge 或样本预算劫持？</p><b>分数真由所声称的变量造成吗？</b></article>
|
||||
<article><span>03</span><h3>外部效度</h3><p>实验室高分能否迁移到真实用户、语言、工具、时间与风险环境？</p><b>结论能走出测试集多远？</b></article>
|
||||
</div>
|
||||
<div class="protocol-passport">
|
||||
<div><span>弱写法</span><b>“K3 在 HLE 得 56，所以知识推理是 56。”</b></div>
|
||||
<div><span>可审计写法</span><b>“K3 报告在 max effort、给定工具与采样协议下作者自报 HLE 56.0；无工具设置为 43.5，二者测的是不同系统条件。”</b></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="history">
|
||||
<p class="eyebrow"><span>02</span> SEVEN SHIFTS</p>
|
||||
<h2>评测史不是“题越来越难”，而是测量边界不断外扩</h2>
|
||||
<p>
|
||||
每一代 benchmark 都修复了旧盲区，也创造了新盲区。自动代理便宜，却离真实偏好远；
|
||||
人类和 LLM Judge 能看开放回答，却带来人群、风格与裁判偏差；Agent 更接近真实工作，
|
||||
却把整个运行系统带进了分数。
|
||||
</p>
|
||||
<div class="wave-timeline">
|
||||
{waves.map(([year, title, nodes, note], index) => (
|
||||
<article><div>{String(index + 1).padStart(2, "0")}</div><time>{year}</time><section><h3>{title}</h3><b>{nodes}</b><p>{note}</p></section></article>
|
||||
))}
|
||||
</div>
|
||||
<div class="direction-map">
|
||||
<div><span>便宜、固定、可重复</span><b>参考重叠 → 静态题集 → 自动 Verifier</b></div>
|
||||
<i>同时推进，而非线性替代</i>
|
||||
<div><span>真实、开放、系统化</span><b>人类偏好 → 动态环境 → 长程 Agent</b></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="token-metrics">
|
||||
<p class="eyebrow"><span>03</span> METRICS ARE QUESTIONS</p>
|
||||
<h2>PPL、BLEU 与 ROUGE 都很重要，但它们从未等于“整体能力”</h2>
|
||||
<div class="formula-board">
|
||||
<div><span>NEGATIVE LOG-LIKELIHOOD</span><code>NLL = −Σ<sub>t</sub> log p(x<sub>t</sub> | x<sub><t</sub>)</code></div>
|
||||
<div><span>CROSS-ENTROPY</span><code>H = NLL / T</code></div>
|
||||
<div><span>PERPLEXITY</span><code>PPL = exp(H)</code></div>
|
||||
<div><span>TOKENIZER-ROBUST UNIT</span><code>BPB = NLL / (ln 2 × bytes)</code></div>
|
||||
</div>
|
||||
<p>
|
||||
PPL 的直觉是模型在每一步面对的“等效平均分支数”。它只对同一 tokenizer、同一文本预处理、
|
||||
同一测试分布具有直接可比性。把一句话切成 6 个 token 或 12 个 token，平均单位已经变了；
|
||||
因此 PALOMA 等工作在 tokenizer 不同时使用 bits per byte。
|
||||
</p>
|
||||
<div class="metric-grid eval-six">
|
||||
{metricCards.map(([name, question, boundary]) => <article><span>{name}</span><h3>{question}</h3><p>{boundary}</p></article>)}
|
||||
</div>
|
||||
<div class="concept-callout">
|
||||
<span>没有“最先进指标”</span>
|
||||
<p>BLEU / ROUGE 仍适合确定性回归；PPL 仍适合语言建模诊断。错误不是使用代理，而是忘记代理和目标之间的距离。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="suites">
|
||||
<p class="eyebrow"><span>04</span> BENCHMARK SUITES</p>
|
||||
<h2>把很多任务放到一张表，不会自动得到“通用智能”</h2>
|
||||
<div class="suite-lineage">
|
||||
<article><time>2018</time><h3>GLUE</h3><p>统一九种 NLU 任务、接口与诊断集，让通用表示可比较。</p><small>新盲区：很快饱和。</small></article>
|
||||
<article><time>2019</time><h3>SuperGLUE</h3><p>替换为更难任务，并把人类基线、排行榜和工具一并发布。</p><small>新盲区：静态公开题仍被生态吸收。</small></article>
|
||||
<article><time>2020</time><h3>MMLU</h3><p>57 个学科把知识与解题能力扩到高中、大学和专业考试。</p><small>新盲区：选择题格式、题质与文化分布。</small></article>
|
||||
<article><time>2022</time><h3>BIG-bench</h3><p>社区汇集数百种任务，寻找规模带来的涌现与失败。</p><small>新盲区：任务异质，聚合解释困难。</small></article>
|
||||
<article><time>2022</time><h3>HELM</h3><p>把场景、适配、准确、校准、鲁棒、公平、毒性与效率写进透明报告。</p><small>新盲区：覆盖越全，协议维护成本越高。</small></article>
|
||||
</div>
|
||||
<p>
|
||||
套件总分至少藏着三次选择：选哪些任务、每个任务如何归一、每个任务占多大权重。
|
||||
因此应先看每个子项与最差切片，再看总分。总分可以用于导航，不能替代诊断。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="lifecycle">
|
||||
<p class="eyebrow"><span>05</span> BENCHMARK LIFECYCLE</p>
|
||||
<h2>一个基准会出生、流行、饱和、被污染、修复，也可能退休</h2>
|
||||
<div class="lifecycle-loop">
|
||||
<div><span>01</span><b>设计</b><small>定义构念、任务、样本与裁判</small></div><i>→</i>
|
||||
<div><span>02</span><b>发布</b><small>代码、数据与 leaderboard 加速采用</small></div><i>→</i>
|
||||
<div><span>03</span><b>优化</b><small>模型与 Prompt 围绕公开协议进步</small></div><i>→</i>
|
||||
<div><span>04</span><b>失真</b><small>饱和、污染、标签错与生态博弈</small></div><i>→</i>
|
||||
<div><span>05</span><b>维护</b><small>重标、加难、动态刷新或退役</small></div>
|
||||
</div>
|
||||
<div class="repair-table">
|
||||
<div class="head"><span>故障</span><span>看起来像</span><span>维修方法</span></div>
|
||||
<div><b>题目错误</b><p>能力强的系统也被判错</p><p>专家重标、保留争议、多正确答案</p></div>
|
||||
<div><b>饱和</b><p>前沿模型分差趋近噪声</p><p>增加难度和区分度，不只增加冷知识</p></div>
|
||||
<div><b>污染</b><p>公开题变成训练材料</p><p>时间切分、私有题、动态刷新、数据谱系</p></div>
|
||||
<div><b>环境腐烂</b><p>依赖、网站或容器无法重跑</p><p>镜像、版本、定期重建与可执行快照</p></div>
|
||||
<div><b>目标错位</b><p>高分不再代表真实用户价值</p><p>重新定义构念，加入现场与纵向效度</p></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="prompt">
|
||||
<p class="eyebrow"><span>06</span> PROMPT IS PART OF THE TEST</p>
|
||||
<h2>改一个模板，可能既改变模型“理解了什么”，也改变 parser“看见了什么”</h2>
|
||||
<div class="prompt-stack">
|
||||
<div><span>01</span><b>System prompt</b><p>角色、安全、工具和输出原则。</p></div>
|
||||
<div><span>02</span><b>Chat template</b><p>special tokens、role 边界和 assistant 起始。</p></div>
|
||||
<div><span>03</span><b>Shots</b><p>示例数量、身份、顺序与是否包含推理。</p></div>
|
||||
<div><span>04</span><b>Task instruction</b><p>direct answer、CoT、结构化格式与语言。</p></div>
|
||||
<div><span>05</span><b>Parser</b><p>从自然输出抽取选项、数字、代码或动作。</p></div>
|
||||
</div>
|
||||
<div class="example-duel">
|
||||
<article><span>协议 A</span><code>Answer with A, B, C, or D only.</code><p>测得更接近选项选择，但可能压制 reasoning model 的自然轨迹。</p></article>
|
||||
<article><span>协议 B</span><code>Think step by step, then put \boxed{"{answer}"}.</code><p>给了额外计算与格式要求；parser 失败和长输出上限会进入分数。</p></article>
|
||||
</div>
|
||||
<p>
|
||||
DeepSeek LLM 明确给出 base 与 chat 在部分任务上的 shots 差异；DeepSeek-R1 又因为 few-shot CoT
|
||||
可能伤害 reasoning model 而采用若干 zero-shot 协议。这里没有一个抽象的“绝对公平 Prompt”：
|
||||
正确做法是公开、固定，并用多模板敏感性分析说明结论是否稳健。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="sampling">
|
||||
<p class="eyebrow"><span>07</span> SAMPLING CHANGES THE QUESTION</p>
|
||||
<h2>pass@1、pass@k、cons@k、best-of-N 与 pass^k 不能互换</h2>
|
||||
<div class="sampling-grid">
|
||||
<article><span>ONE-SHOT</span><h3>Greedy / 一次运行</h3><p>默认产品体验：不给搜索和选择器，只看一次能否成功。</p></article>
|
||||
<article><span>MEAN PASS@1</span><h3>采样分布下一次成功</h3><p>对每题多次采样后取正确率均值，能比一次随机运行更稳。</p></article>
|
||||
<article><span>PASS@K</span><h3>至少一个候选通过</h3><p>适合有单测或 verifier 的搜索系统；k 越大，成本和成功率都增。</p></article>
|
||||
<article><span>CONS@K</span><h3>多数答案正确</h3><p>自一致依赖正确轨迹能汇聚到同一答案；不等于候选覆盖。</p></article>
|
||||
<article><span>BEST-OF-N</span><h3>选择器挑中最好答案</h3><p>总成绩同时测生成器和 reward / Judge 的排序能力。</p></article>
|
||||
<article><span>PASS^K</span><h3>连续 k 次全部成功</h3><p>测可靠性而非探索；k 增大时分数通常下降。</p></article>
|
||||
</div>
|
||||
<div class="formula-board two">
|
||||
<div><span>HUMANEVAL UNBIASED ESTIMATOR</span><code>pass@k = 1 − C(n−c, k) / C(n, k)</code></div>
|
||||
<div><span>INDEPENDENCE INTUITION</span><code>at least one = 1−(1−p)<sup>k</sup> · all = p<sup>k</sup></code></div>
|
||||
</div>
|
||||
<div class="numeric-story">
|
||||
<b>一次成功率 80% 的 Agent</b>
|
||||
<div><span>5 次至少一次成功</span><strong>99.97%</strong></div>
|
||||
<div><span>连续 5 次全部成功</span><strong>32.77%</strong></div>
|
||||
<p>同一个 p、同一个 k，两个看似都叫“5 次”的指标却给出相反故事。真实运行还可能相关，所以要报告经验重复分布。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="calibration">
|
||||
<p class="eyebrow"><span>08</span> CALIBRATION & SELECTIVE RISK</p>
|
||||
<h2>答对多少，和“知道自己什么时候会错”，是两种能力</h2>
|
||||
<div class="calibration-visual">
|
||||
<div class="cal-chart">
|
||||
<span>实际正确率</span>
|
||||
<div class="diag"></div>
|
||||
<i style="left:20%;bottom:17%"></i><i style="left:45%;bottom:33%"></i><i class="bad" style="left:72%;bottom:48%"></i><i class="bad" style="left:90%;bottom:61%"></i>
|
||||
<small>预测置信度 →</small>
|
||||
</div>
|
||||
<div>
|
||||
<h3>完美校准</h3>
|
||||
<code>P(correct | confidence=q) = q</code>
|
||||
<p>声称 80% 置信的答案，长期应约有 80% 正确。高准确率不自动校准；校准好也不自动准确率高。</p>
|
||||
<ul>
|
||||
<li>reliability diagram 看局部偏差；</li>
|
||||
<li>ECE 把置信桶误差压成一个代理；</li>
|
||||
<li>Brier score 同时惩罚概率误差；</li>
|
||||
<li>risk–coverage 曲线观察拒答后的质量；</li>
|
||||
<li>semantic entropy 合并同义答案再估不确定性。</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
<p>
|
||||
Kadavath 等人的结果说明，在其任务和格式中，更大模型能给出有希望的自评估；
|
||||
但 P(IK) 跨任务仍难校准，不能被简写为“模型普遍知道自己什么时候错”。
|
||||
真正的产品问题往往不是强迫每题都答，而是：在多少覆盖率下，把错误风险压到可接受范围？
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="contamination">
|
||||
<p class="eyebrow"><span>09</span> FIVE LEAKAGE LAYERS</p>
|
||||
<h2>污染不是一个开关，也不能靠一次字符串搜索结案</h2>
|
||||
<div class="contamination-stack">
|
||||
{contaminationLayers.map(([number, title, meaning, detect]) => (
|
||||
<article><span>{number}</span><div><h3>{title}</h3><p>{meaning}</p></div><b>{detect}</b></article>
|
||||
))}
|
||||
</div>
|
||||
<p>
|
||||
“干净子集分数低于污染子集”也不能独自证明记忆带来提升，因为两组题可能难度不同。
|
||||
最低限度要同时披露匹配定义、覆盖率、人工抽检、来源时间、效应量与不确定性。
|
||||
DeepSeek-R1 明确承认 n-gram 去污染无法阻止测试集改写，这是一条应被保留的诚实边界。
|
||||
</p>
|
||||
<div class="failure-callout">
|
||||
<b>常见错误推理</b>
|
||||
<code>no exact overlap ⇒ no contamination</code>
|
||||
<p>正确结论只能是：“在当前语料覆盖、归一化与匹配阈值下，未发现这种表面重叠。”</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="dynamic">
|
||||
<p class="eyebrow"><span>10</span> FRESHNESS × COMPARABILITY</p>
|
||||
<h2>动态题解决“见过”，却可能失去“同一把尺”</h2>
|
||||
<div class="tradeoff-grid">
|
||||
<article><span>STATIC PUBLIC</span><h3>固定公开题</h3><p>任何模型都能复跑，解释清晰；但会被训练生态吸收，且容易饱和。</p><b>纵向可比强 · 新鲜度弱</b></article>
|
||||
<article><span>PRIVATE SNAPSHOT</span><h3>私有时间切片</h3><p>泄漏风险较低；外部读者无法看题，错误与构念效度更难审计。</p><b>新鲜度中 · 透明度弱</b></article>
|
||||
<article><span>FULLY DYNAMIC</span><h3>持续生成 / 收集</h3><p>能追新知识和新失败；不同月份题目难度、用户构成与环境都会漂移。</p><b>新鲜度强 · 纵向可比弱</b></article>
|
||||
<article class="recommended"><span>VERSIONED + ANCHOR</span><h3>版本化动态题 + 固定锚点</h3><p>按月更新主体，同时保留一组受控 anchor；分报当期能力和历史曲线。</p><b>两本账，不强行压成一个分数</b></article>
|
||||
</div>
|
||||
<p>
|
||||
Dynabench 用人机闭环造当前难例；FreshQA 刷新快速变化知识；LiveBench 用近期来源与客观判分按月更新；
|
||||
LiveCodeBench 和 SWE-bench Live 利用题目或 issue 的时间切分。它们没有“终结污染”，而是把数据时间和版本
|
||||
提升为一等协议字段。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="verifier">
|
||||
<p class="eyebrow"><span>11</span> EXECUTABLE VERIFIERS</p>
|
||||
<h2>代码能运行是巨大进步，但“通过当前测试”仍不等于“完全正确”</h2>
|
||||
<div class="verifier-ladder">
|
||||
<div><span>L0</span><b>字符串 / AST</b><p>便宜，容易漏掉语义错误。</p></div><i>→</i>
|
||||
<div><span>L1</span><b>示例单测</b><p>能执行，但覆盖路径有限。</p></div><i>→</i>
|
||||
<div><span>L2</span><b>属性 / 变异测试</b><p>扩大输入空间与边界条件。</p></div><i>→</i>
|
||||
<div><span>L3</span><b>独立实现 / 隐藏测试</b><p>更强，仍依赖规格正确。</p></div><i>→</i>
|
||||
<div><span>L4</span><b>形式化证明器</b><p>对形式规格最硬；规格本身仍需正确。</p></div>
|
||||
</div>
|
||||
<p>
|
||||
EvalPlus 将 HumanEval / MBPP 的测试大幅扩充，并发现许多原测试未捕获的错误，模型排序也会改变。
|
||||
DS-1000 又把库 API、数值约束、禁止操作等多条件写进判分。Verifier 的真正优势不是“绝不会错”，
|
||||
而是它的判分程序更明确、更容易审计和改进。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="swe">
|
||||
<p class="eyebrow"><span>12</span> A BENCHMARK REPAIRS ITSELF</p>
|
||||
<h2>SWE-bench 是理解“基准生命周期”最好的完整案例</h2>
|
||||
<div class="swe-story">
|
||||
<article><time>2023</time><h3>SWE-bench</h3><p>从真实 GitHub issue、仓库与合并补丁构造软件工程任务；把评测单位从函数改为仓库状态。</p></article>
|
||||
<article><time>2024</time><h3>SWE-bench Verified</h3><p>专业开发者检查任务是否可解、规格是否明确、FAIL_TO_PASS 测试是否公平，保留 500 个高质量样本。</p></article>
|
||||
<article><time>2025</time><h3>SWE-bench Live</h3><p>持续从新 issue 构建可执行任务，用时间版本降低静态题被训练吸收的风险。</p></article>
|
||||
<article><time>2026</time><h3>公开退役信号</h3><p>OpenAI 官方说明 Verified 已越来越受污染，转而推荐更难、更新、覆盖不同语言的 SWE-bench Pro。</p></article>
|
||||
</div>
|
||||
<div class="concept-callout">
|
||||
<span>不是“旧成绩作废”</span>
|
||||
<p>旧分数仍描述当时协议下的历史观察；错误是把它当成 2026 年仍等效、仍干净、仍能区分前沿能力的当前证据。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="long-multi">
|
||||
<p class="eyebrow"><span>13</span> DIAGNOSTIC CHAINS</p>
|
||||
<h2>1M 窗口和一个多模态总分，都必须拆成瓶颈链</h2>
|
||||
<div class="dual-chain">
|
||||
<article>
|
||||
<span>LONG CONTEXT</span>
|
||||
<h3>声明窗口 ≠ 有效窗口</h3>
|
||||
<div><b>检索</b><i>→</i><b>位置鲁棒</b><i>→</i><b>多跳整合</b><i>→</i><b>长输出</b></div>
|
||||
<p>Needle 主要测找到一段信息；RULER 增加多 needle、变量追踪与聚合；仍要报告 tokenizer、位置、截断、输出与长度曲线。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>MULTIMODAL</span>
|
||||
<h3>总分 ≠ 每个模态环节都强</h3>
|
||||
<div><b>感知 / OCR</b><i>→</i><b>Grounding</b><i>→</i><b>融合</b><i>→</i><b>推理 / 行动</b></div>
|
||||
<p>OCRBench、MMBench、MMMU、Video-MME 的输入与构念不同；图像尺寸、tile、帧采样、工具和 Judge 必须披露。</p>
|
||||
</article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="agent-unit">
|
||||
<p class="eyebrow"><span>14</span> MODEL IS NOT THE AGENT</p>
|
||||
<h2>Agent 的最小测量单位，是“模型—脚手架—工具—环境”</h2>
|
||||
<div class="agent-equation">
|
||||
<div><span>MODEL</span><b>生成策略</b><p>checkpoint、effort、解码</p></div><i>×</i>
|
||||
<div><span>HARNESS</span><b>控制循环</b><p>prompt、规划、上下文、恢复</p></div><i>×</i>
|
||||
<div><span>TOOLS</span><b>行动契约</b><p>schema、权限、错误反馈</p></div><i>×</i>
|
||||
<div><span>ENVIRONMENT</span><b>世界状态</b><p>版本、网络、账户、随机性</p></div><i>×</i>
|
||||
<div><span>VERIFIER</span><b>成功定义</b><p>最终状态与禁止副作用</p></div>
|
||||
</div>
|
||||
<p>
|
||||
同一个模型换 system prompt、文件编辑器、浏览器工具、上下文压缩策略、重试次数或容器镜像，
|
||||
就是另一个被测系统。公平比较可以固定 Harness 测模型，也可以允许每家最优 Harness 测产品上限；
|
||||
但两种赛道必须命名不同，不能混入同一列后只写模型名。
|
||||
</p>
|
||||
<div class="agent-protocol-min">
|
||||
{["model ID", "system prompt", "harness commit", "tool schema / permission", "environment image / date", "context policy", "max steps / tokens / wall time", "retry / reset", "parallel rollouts", "verifier", "cost", "failure taxonomy"].map((item, index) => <span><b>{String(index + 1).padStart(2, "0")}</b>{item}</span>)}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="final-state">
|
||||
<p class="eyebrow"><span>15</span> VERIFY OUTCOMES, NOT THEATER</p>
|
||||
<h2>轨迹像专家，不等于事情真的完成了</h2>
|
||||
<div class="state-compare">
|
||||
<article class="weak"><span>弱裁判</span><h3>读轨迹打分</h3><p>“分析得很合理”“命令看起来正确”“步骤接近 gold trajectory”。</p><b>容易奖励表演、冗长和唯一范例路径。</b></article>
|
||||
<article class="strong"><span>更强裁判</span><h3>验证最终状态</h3><p>仓库测试通过、文件存在、订单状态正确、禁止副作用未发生。</p><b>允许多条有效路径，但 verifier 仍需覆盖目标。</b></article>
|
||||
</div>
|
||||
<div class="reliability-chain">
|
||||
<span>单步动作 99%</span><i>×</i><span>100 个关键步骤</span><i>=</i><strong>约 36.6% 全程无错</strong>
|
||||
</div>
|
||||
<p>
|
||||
这是教学独立近似，不是通用 Agent 定律；现实错误会相关，恢复机制也会修复。
|
||||
但它揭示一个方向：长程任务应报告完成率、失败阶段、重试后成功、连续重复可靠性和尾部成本，
|
||||
而不仅是平均一步准确率。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="human">
|
||||
<p class="eyebrow"><span>16</span> HUMAN EVALUATION</p>
|
||||
<h2>“人类偏好”不是一个天然、统一、无噪声的真值</h2>
|
||||
<div class="human-card">
|
||||
<div>
|
||||
<h3>至少回答十个问题</h3>
|
||||
<ol>
|
||||
<li>目标用户是谁，标注者是谁？</li>
|
||||
<li>需要什么专业资格与语言文化背景？</li>
|
||||
<li>如何培训、校准并支付标注者？</li>
|
||||
<li>模型身份是否盲化，顺序是否随机？</li>
|
||||
<li>允许 tie、both bad、不可判断吗？</li>
|
||||
<li>rubric 是整体偏好还是逐维评分？</li>
|
||||
<li>每项由几人评，分歧是否保留？</li>
|
||||
<li>一致性和置信区间怎样计算？</li>
|
||||
<li>展示长度、格式和引用是否一致？</li>
|
||||
<li>伦理、隐私与有害内容暴露怎样处理？</li>
|
||||
</ol>
|
||||
</div>
|
||||
<blockquote>
|
||||
<span>正确的表述</span>
|
||||
“在这组提示、这类目标用户、这套展示与 rubric 下，A 被该标注者群体偏好。”
|
||||
<small>不是：“人类证明 A 客观更智能。”</small>
|
||||
</blockquote>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="judge">
|
||||
<p class="eyebrow"><span>17</span> JUDGE THE JUDGE</p>
|
||||
<h2>LLM-as-a-Judge 解决了规模，却没有消灭测量误差</h2>
|
||||
<div class="judge-bias-grid">
|
||||
<article><span>POSITION</span><h3>位置偏好</h3><p>相同答案交换 A/B 顺序，结论会不会翻转？</p><b>随机交换、双向评、报告 flip rate</b></article>
|
||||
<article><span>VERBOSITY</span><h3>长度偏好</h3><p>更长回答获得更多分，是信息更全还是风格红利？</p><b>长度控制、配对短长反例</b></article>
|
||||
<article><span>SELF</span><h3>自我偏好</h3><p>Judge 是否偏爱与自身家族或风格相近的回答？</p><b>多 Judge、盲化、跨家族元评测</b></article>
|
||||
<article><span>CAPABILITY</span><h3>能力上限</h3><p>Judge 自己不会数学、代码或视觉时，如何判别人？</p><b>参考答案、工具、专家抽检、分任务裁判</b></article>
|
||||
<article><span>RUBRIC</span><h3>标准漂移</h3><p>“helpful”是否偷带风格、安全或篇幅偏好？</p><b>逐维 rubric、锚点样例、版本冻结</b></article>
|
||||
<article><span>STOCHASTICITY</span><h3>采样与版本</h3><p>同一 Judge 多次结果和线上模型版本是否稳定？</p><b>重复采样、日期、模型 ID、解析失败率</b></article>
|
||||
</div>
|
||||
<p>
|
||||
MT-Bench / Chatbot Arena 系统讨论了位置、冗长和自增强偏差；G-Eval 说明 rubric 与推理提示
|
||||
可以提高与人类的一致性，也指出偏好 LLM 文本的可能；LLMBar 则用“真正遵循指令但风格不讨巧”
|
||||
的答案专门测试 evaluator。结论很清楚：Judge 本身必须有 benchmark。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="arena">
|
||||
<p class="eyebrow"><span>18</span> ARENA IS A RELATIVE GRAPH</p>
|
||||
<h2>Elo / Bradley–Terry 是比较图上的相对位置，不是“智力点数”</h2>
|
||||
<div class="arena-diagram">
|
||||
<div class="users"><span>真实用户提示</span><b>题目分布</b></div>
|
||||
<div class="matches"><i>A ↔ B</i><i>B ↔ C</i><i>A ↔ D</i><i>C ↔ D</i><span>匿名配对 + 投票</span></div>
|
||||
<div class="model"><span>统计模型</span><b>相对强度 + 区间</b></div>
|
||||
<div class="leaderboard"><span>排行榜</span><b>某个时间窗的排序</b></div>
|
||||
</div>
|
||||
<div class="arena-checks">
|
||||
{["用户问题分布", "配对与曝光策略", "投票者 / 专家一致性", "tie 与 style 控制", "模型精确版本", "投票时间窗", "bootstrap 区间", "分类榜与总榜", "异常票清理", "成本与长度"].map((item, index) => <span><b>{String(index + 1).padStart(2, "0")}</b>{item}</span>)}
|
||||
</div>
|
||||
<p>
|
||||
Arena 的价值是把真实开放问题和相对偏好带进评测，不是把它神化成无偏真值。
|
||||
当两个模型区间重叠、题目类别不同或版本更新时，应说“当前数据不足以确定排序”，
|
||||
而不是强行用小数位制造确定性。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="aggregation">
|
||||
<p class="eyebrow"><span>19</span> AVERAGES HIDE VALUES</p>
|
||||
<h2>平均数不仅压缩数据，也压缩了“谁重要”的价值判断</h2>
|
||||
<div class="aggregate-example">
|
||||
<div class="score-head"><span>模型</span><span>英语</span><span>中文</span><span>低资源语言</span><span>总体 micro</span><span>macro</span></div>
|
||||
<div><b>A</b><span>92</span><span>78</span><span class="low">31</span><strong>88</strong><strong>67</strong></div>
|
||||
<div><b>B</b><span>84</span><span>81</span><span>72</span><strong>82</strong><strong>79</strong></div>
|
||||
</div>
|
||||
<p>
|
||||
如果样本 80% 是英语，micro average 会让 A 看起来领先；给每种语言相同权重的 macro average
|
||||
则可能让 B 领先。两者都不是“数学上错误”，而是回答了不同分布下的问题。公平、安全和文化评测尤其要：
|
||||
</p>
|
||||
<ul class="editorial-list">
|
||||
<li>同时报总体、关键子群、最差组与组间差；</li>
|
||||
<li>公开权重、缺失项、拒答和无效输出处理；</li>
|
||||
<li>避免用一个总分抵消某个群体的严重失败；</li>
|
||||
<li>对多指标、多子组和多次提交处理选择性报告与多重比较。</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="cost">
|
||||
<p class="eyebrow"><span>20</span> ACCURACY × COST × LATENCY</p>
|
||||
<h2>测试时扩展时代，不报预算的“最高分”越来越难解释</h2>
|
||||
<div class="cost-frontier">
|
||||
<div class="cost-axis y">任务成功 ↑</div><div class="cost-axis x">Token / 工具 / 延迟 / ¥ →</div>
|
||||
<svg viewBox="0 0 800 300" role="img" aria-label="成本能力帕累托前沿示意">
|
||||
<path d="M50 252 C170 205, 250 150, 360 112 S610 55, 750 40" fill="none" stroke="#b86e3c" stroke-width="4"/>
|
||||
<path d="M50 252 L750 252 M50 252 L50 28" fill="none" stroke="#6b807f" stroke-width="2"/>
|
||||
<g fill="#17363e"><circle cx="145" cy="215" r="8"/><circle cx="300" cy="136" r="8"/><circle cx="475" cy="88" r="8"/><circle cx="690" cy="48" r="8"/></g>
|
||||
<g font-size="17" fill="#17363e"><text x="105" y="195">Nonthink</text><text x="265" y="116">High</text><text x="445" y="68">Max</text><text x="620" y="88">Max + tools</text></g>
|
||||
<circle cx="590" cy="170" r="9" fill="#a6b5b1"/><text x="610" y="176" font-size="16" fill="#647977">被支配：更贵且更弱</text>
|
||||
</svg>
|
||||
</div>
|
||||
<div class="cost-denominators">
|
||||
{["输入 token", "隐藏思考 / 输出 token", "并行 samples", "工具调用与网络", "最大步数 / 重试", "wall-clock latency", "金钱 / 能耗", "满足 SLO 的成功任务"].map((item) => <span>{item}</span>)}
|
||||
</div>
|
||||
<p>
|
||||
合理比较不一定强迫所有系统同预算。可以报告固定预算下的能力，也可以画完整 Pareto frontier：
|
||||
在每一个成本点上，谁提供最高成功率？DeepSeek-V4 的 Nonthink / High / Max，以及 K3 的 max effort，
|
||||
都说明“模型能力”正在变成一条由测试时资源参数化的曲线。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="safety-object">
|
||||
<p class="eyebrow"><span>21</span> WHAT IS THE SAFETY OBJECT?</p>
|
||||
<h2>基础模型、对齐行为、产品防线与部署环境，必须分四层测</h2>
|
||||
<div class="safety-layers">
|
||||
<article><span>L0</span><h3>Base model</h3><p>在给定上下文下会生成什么能力和内容？</p><b>模型权重本体</b></article>
|
||||
<article><span>L1</span><h3>Aligned behavior</h3><p>SFT / RL 后如何遵循、拒绝、校准和解释？</p><b>checkpoint 行为</b></article>
|
||||
<article><span>L2</span><h3>Product wrapper</h3><p>system prompt、分类器、过滤、监控和速率限制怎样改变行为？</p><b>产品防线</b></article>
|
||||
<article><span>L3</span><h3>Deployment</h3><p>工具权限、数据、用户、网络、日志与人工升级路径是什么？</p><b>真实风险系统</b></article>
|
||||
</div>
|
||||
<p>
|
||||
在裸模型上做攻击可以测权重层的稳健性；在产品 API 上做攻击可以测防御总和。两者都有效，
|
||||
但不能互相冒充。DeepSeek-R1 报告中安全评测所用 wrapper 与裸 reasoning checkpoint 的差异，
|
||||
正是为什么要先命名对象。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="threat">
|
||||
<p class="eyebrow"><span>22</span> THREAT MODEL FIRST</p>
|
||||
<h2>在报攻击成功率之前，先回答攻击者是谁</h2>
|
||||
<div class="threat-grid">
|
||||
<article><span>KNOWLEDGE</span><h3>知道什么？</h3><p>黑盒、logprobs、梯度、权重、system prompt、过滤器规则。</p></article>
|
||||
<article><span>CONTROL</span><h3>能改什么？</h3><p>用户文本、图片、网页内容、工具返回、记忆、文件、环境状态。</p></article>
|
||||
<article><span>BUDGET</span><h3>可试多少次？</h3><p>单次、固定 queries、自适应搜索、多人长期攻击。</p></article>
|
||||
<article><span>GOAL</span><h3>想达到什么？</h3><p>非拒绝、有效有害知识、越权动作、泄露秘密、持久化控制。</p></article>
|
||||
<article><span>ACCESS</span><h3>拥有什么权限？</h3><p>只聊天、上传文件、联网、执行代码、调用高风险工具。</p></article>
|
||||
<article><span>SUCCESS</span><h3>怎样才算成功？</h3><p>Judge 标签、人工效用、真实副作用、最终状态或损失上界。</p></article>
|
||||
</div>
|
||||
<div class="threat-passport">
|
||||
<b>最小攻击护照</b>
|
||||
<code>target × attacker knowledge × controllable channel × query budget × adaptation × success criterion × defense stack × date</code>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="safety-lineage">
|
||||
<p class="eyebrow"><span>23</span> SAFETY GENEALOGY</p>
|
||||
<h2>安全评测从“内容像不像毒性”走到“系统是否真的被攻破”</h2>
|
||||
<div class="safety-timeline">
|
||||
<article><time>2020–21</time><span>CONTENT & BIAS</span><h3>RealToxicity · StereoSet · CrowS · BBQ</h3><p>测生成内容、刻板关联与歧义场景偏差。</p></article>
|
||||
<article><time>2022</time><span>RED TEAMING</span><h3>LM-generated attacks · human red teams</h3><p>主动寻找模型当前会暴露的有害失败。</p></article>
|
||||
<article><time>2023</time><span>JAILBREAK & OVER-REFUSAL</span><h3>GCG · Jailbroken · XSTest · Do-Not-Answer</h3><p>同时暴露绕过安全与错误拒绝无害请求。</p></article>
|
||||
<article><time>2024</time><span>STANDARDIZATION</span><h3>HarmBench · StrongREJECT · JailbreakBench · WildGuard</h3><p>统一行为、攻击、分类器与有害有效性。</p></article>
|
||||
<article><time>2024–25</time><span>AGENT & PRODUCT</span><h3>InjecAgent · AgentDojo · Cybench · AILuminate</h3><p>把不可信内容、工具权限、环境行动和标准风险等级带进来。</p></article>
|
||||
</div>
|
||||
<div class="negative-predictive">
|
||||
<span>AILUMINATE 的重要边界</span>
|
||||
<h3>发现失败，可以证明存在漏洞；没有发现失败，不能证明系统安全。</h3>
|
||||
<p>安全测试具有强负向预测力，但覆盖永远有限。一个好等级只描述当前测试范围内的观察。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="jailbreak">
|
||||
<p class="eyebrow"><span>24</span> NON-REFUSAL IS NOT HARM</p>
|
||||
<h2>越狱“成功”至少有四道门</h2>
|
||||
<div class="jailbreak-funnel">
|
||||
<div><span>01</span><b>模型没有拒绝</b><small>refusal bypass</small></div><i>→</i>
|
||||
<div><span>02</span><b>内容与目标相关</b><small>relevance</small></div><i>→</i>
|
||||
<div><span>03</span><b>信息具体、可用</b><small>utility</small></div><i>→</i>
|
||||
<div><span>04</span><b>造成能力或现实影响</b><small>impact under threat model</small></div>
|
||||
</div>
|
||||
<p>
|
||||
一些攻击让模型输出很长、看似顺从，却只有空洞警告或错误信息。若分类器只看“是否拒绝”，
|
||||
就会把这些当成功。StrongREJECT 的核心贡献正是用人工标注关注有害回答的相关性与可用性。
|
||||
因此最低限度同时报：
|
||||
</p>
|
||||
<div class="safety-metrics">
|
||||
<span>unsafe compliance</span><span>harmful utility</span><span>robust refusal</span><span>benign answer</span><span>over-refusal</span><span>attack queries / cost</span>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="agent-security">
|
||||
<p class="eyebrow"><span>25</span> UNTRUSTED DATA BECOMES INSTRUCTION</p>
|
||||
<h2>Agent 安全的核心变化：模型开始读取外部世界，并有权限改变它</h2>
|
||||
<div class="injection-flow">
|
||||
<div><span>USER GOAL</span><b>“总结邮件并生成日程”</b></div><i>→</i>
|
||||
<div class="untrusted"><span>TOOL RESULT</span><b>邮件正文藏有恶意指令</b></div><i>→</i>
|
||||
<div><span>MODEL</span><b>误把数据当高优先级命令</b></div><i>→</i>
|
||||
<div class="danger"><span>TOOL ACTION</span><b>外发数据 / 越权修改</b></div>
|
||||
</div>
|
||||
<div class="security-cases">
|
||||
<article><h3>InjecAgent</h3><p>把恶意指令嵌入工具返回，系统化测间接 Prompt 注入。</p></article>
|
||||
<article><h3>AgentDojo</h3><p>在动态工具环境中同时测实用任务效用与攻击成功。</p></article>
|
||||
<article><h3>Cybench</h3><p>用可执行网络安全任务和子任务诊断真实攻击链能力。</p></article>
|
||||
<article><h3>Instruction Hierarchy</h3><p>把 system / user / third-party 指令优先级显式训练与评测。</p></article>
|
||||
</div>
|
||||
<p>
|
||||
这里的防线不是“让模型更会拒绝”一句话：还包括最小权限、数据—指令分离、动作确认、
|
||||
工具参数验证、秘密隔离、网络策略、审计日志、速率限制和人工升级。安全评测必须覆盖整条链。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="overrefusal">
|
||||
<p class="eyebrow"><span>26</span> SAFETY × UTILITY FRONTIER</p>
|
||||
<h2>拒绝越多并不等于越安全：无害可用性必须进入同一张图</h2>
|
||||
<div class="refusal-matrix">
|
||||
<div class="blank"></div><div class="head">模型回答</div><div class="head">模型拒绝</div>
|
||||
<div class="head">有害请求</div><div class="bad"><b>危险服从</b><span>需要压低</span></div><div class="good"><b>正确拒绝</b><span>需要提高</span></div>
|
||||
<div class="head">无害请求</div><div class="good"><b>正常帮助</b><span>需要保留</span></div><div class="bad"><b>过度拒答</b><span>XSTest 关注</span></div>
|
||||
</div>
|
||||
<p>
|
||||
安全阈值移动会在危险服从和过度拒答之间形成前沿。更好的系统不是简单把阈值调高，
|
||||
而是提升区分能力、澄清意图、提供安全替代、按权限分级，并对不确定高风险动作升级给人。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="deepseek">
|
||||
<p class="eyebrow"><span>27</span> DEEPSEEK PROTOCOL LINEAGE</p>
|
||||
<h2>沿 DeepSeek 七篇报告，看“模型分数”一步步变成“系统曲线”</h2>
|
||||
<div class="protocol-lineage">
|
||||
{deepseekRows.map(([model, protocol, lesson]) => (
|
||||
<article><span>{model}</span><h3>{protocol}</h3><p>{lesson}</p></article>
|
||||
))}
|
||||
</div>
|
||||
<p>
|
||||
DeepSeek 特别适合作为评测教学主线，因为它同时公开了 dense、MoE、数学、reasoning、长上下文和
|
||||
Agent 演化。最值得学的不是记住某张表，而是观察协议怎样随着能力形态改变：
|
||||
shots 分开、Maj@K / Pass@K 分开、采样 pass@1、effort 分档、上下文分档，最后把工具数和步数写进报告。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="r1">
|
||||
<p class="eyebrow"><span>28</span> DEEPSEEK-R1 CASE STUDY</p>
|
||||
<h2>R1 的“pass@1”不是一次 greedy；安全评测也不是裸模型行为</h2>
|
||||
<div class="r1-protocol">
|
||||
<article><span>WHY SAMPLE?</span><h3>Greedy 对长推理出现重复与 checkpoint 变异</h3><p>报告改用 temperature 0.6、top-p 0.95 多次采样，再取样本正确率均值。</p></article>
|
||||
<article><span>HOW MANY?</span><h3>不同 benchmark 的样本预算不同</h3><p>AIME / GPQA 64；MATH / Codeforces 16；LiveCodeBench 8。k 不是无关脚注。</p></article>
|
||||
<article><span>PROMPT</span><h3>部分任务采用 zero-shot</h3><p>作者指出 few-shot CoT 可能损伤 reasoning model，因此调整协议；比较时必须披露适配。</p></article>
|
||||
<article><span>AGENT HARNESS</span><h3>SWE-bench 使用 Agentless</h3><p>成绩同时属于模型与固定脚手架；换搜索、编辑和上下文管理会改变系统。</p></article>
|
||||
<article><span>SAFETY WRAPPER</span><h3>安全表中的对象含额外包装</h3><p>不能把产品安全行为直接归因到裸 R1 checkpoint。</p></article>
|
||||
<article><span>CONTAMINATION</span><h3>n-gram 无法阻止改写污染</h3><p>作者边界本身就是重要证据，避免把检测写成“已证明无泄漏”。</p></article>
|
||||
</div>
|
||||
<div class="r1-equation">
|
||||
<span>R1 TABLE CELL</span>
|
||||
<b>checkpoint × zero/few-shot × T=.6 × top-p=.95 × samples/task × parser/verifier × wrapper</b>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="k3">
|
||||
<p class="eyebrow"><span>29</span> KIMI K3 §6</p>
|
||||
<h2>逐字段复原 K3 的公开评测：它报告的已经是多种系统设置</h2>
|
||||
<div class="k3-protocol-table">
|
||||
<div class="head"><span>字段</span><span>报告设置</span><span>怎样正确解读</span></div>
|
||||
{k3Protocol.map(([field, setting, meaning]) => <div><b>{field}</b><p>{setting}</p><p>{meaning}</p></div>)}
|
||||
</div>
|
||||
<div class="author-reported">
|
||||
<div>
|
||||
<span>AUTHOR-REPORTED / PROTOCOL-BOUND</span>
|
||||
<h3>公开结果只能连同协议引用</h3>
|
||||
<p>下列是 K3 技术报告在其公开设置下自报的点估计，不是本站复跑，也不是脱离 Harness 与预算的模型常数。</p>
|
||||
</div>
|
||||
<div class="reported-grid">
|
||||
<span><b>93.5</b>GPQA</span>
|
||||
<span><b>43.5 / 56.0</b>HLE 无 / 有工具</span>
|
||||
<span><b>67.5</b>DeepSWE</span>
|
||||
<span><b>81.2</b>FrontierSWE</span>
|
||||
<span><b>42.0</b>SWE-Marathon</span>
|
||||
<span><b>91.2</b>BrowseComp</span>
|
||||
<span><b>84.8</b>OSWorld Verified</span>
|
||||
<span><b>58.3</b>OSWorld 2.0</span>
|
||||
</div>
|
||||
</div>
|
||||
<p>
|
||||
最有教学价值的不是哪一项第一，而是报告主动暴露了 Harness 敏感性：DeepSWE v1.1 与官方
|
||||
mini-SWE-agent 可分别观察；Terminal-Bench 会跨 Harness 报告；BrowseComp 的上下文策略、
|
||||
SWE-Marathon 的硬件校准、FrontierSWE 的重算日期都进入了分数。这正是未来评测报告应该走的方向。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="lab">
|
||||
<p class="eyebrow"><span>30</span> INTERACTIVE LAB</p>
|
||||
<h2>亲手改一次协议，比背十张排行榜更有用</h2>
|
||||
<p>
|
||||
四台测量仪分别对应本章最容易混淆的四组变量。每次只改一个控件，然后先说出
|
||||
“我改变的是模型、协议、系统、裁判还是威胁模型”，再观察结果。
|
||||
</p>
|
||||
<EvaluationLab />
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="audit">
|
||||
<p class="eyebrow"><span>31</span> REUSABLE AUDIT CARD</p>
|
||||
<h2>以后看任何模型报告，先填完这十四行</h2>
|
||||
<div class="audit-table">
|
||||
{auditRows.map(([field, question], index) => <article><span>{String(index + 1).padStart(2, "0")}</span><b>{field}</b><p>{question}</p></article>)}
|
||||
</div>
|
||||
<div class="conclusion-ladder">
|
||||
<article><span>L0 / 描述</span><p>“报告在协议 P 上自报分数 S。”</p></article>
|
||||
<article><span>L1 / 有限比较</span><p>“在关键字段相同的 P 上，A 的点估计高于 B，但区间 / 成本为……”</p></article>
|
||||
<article><span>L2 / 稳健比较</span><p>“跨 Prompt、seed、Harness 或子群敏感性分析后，差异仍稳定。”</p></article>
|
||||
<article><span>L3 / 外部结论</span><p>“在目标用户与真实环境的纵向验证中，差异迁移到产品结果。”</p></article>
|
||||
</div>
|
||||
<div class="one-line final">
|
||||
<span>FINAL RULE</span>
|
||||
<p><b>最可信的评测不是拥有最多小数位，而是让读者知道：测了什么、没测什么、花了多少、谁来判、结论能走多远。</b></p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section paper-section" id="papers">
|
||||
<p class="eyebrow"><span>↳</span> PRIMARY-SOURCE CHAIN</p>
|
||||
<h2>80 个关键节点：从概率代理到 Kimi K3 系统评测</h2>
|
||||
<p>
|
||||
这条链只收录本章实际使用的一手论文、官方技术报告或作者入口。每个节点先回答“它把哪条测量边界向外推了一步”，
|
||||
再决定是否进入精读；全站论文库现已扩充至 450 篇。
|
||||
</p>
|
||||
<div class="paper-chain evaluation-paper-chain">
|
||||
{paperChain.map(([year, title, url, note], index) => (
|
||||
<a href={url} target="_blank" rel="noreferrer">
|
||||
<span>{String(index + 1).padStart(2, "0")} · {year}</span><b>{title}</b><p>{note}</p>
|
||||
</a>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.evaluation-hero{background:
|
||||
radial-gradient(circle at 79% 28%,rgba(203,127,68,.22),transparent 26%),
|
||||
linear-gradient(115deg,#0a2229 0 61%,#17353b 61% 100%)}
|
||||
.evaluation-hero h1{max-width:980px;color:#f7f3eb}
|
||||
.evaluation-hero .lead{color:#bdccce}
|
||||
.evaluation-hero .page-facts{border-color:rgba(255,255,255,.2)}
|
||||
.evaluation-hero .page-facts div{border-color:rgba(255,255,255,.14)}
|
||||
.evaluation-hero .page-facts dt{color:#91a9af}
|
||||
.evaluation-hero .page-facts dd{color:#edf1ef}
|
||||
.evaluation-hero .page-facts div:nth-child(2),.evaluation-hero .page-facts div:nth-child(4){background:rgba(194,111,54,.12)}
|
||||
.evaluation-article code{font-family:var(--font-mono);font-size:.9em}
|
||||
.eval-ledgers{grid-template-columns:repeat(2,minmax(0,1fr))}
|
||||
.measurement-equation{margin:25px 0;padding:22px 24px;background:#102a31;color:white;border-left:5px solid #cf8247}
|
||||
.measurement-equation span{display:block;margin-bottom:10px;color:#d79762;font:700 .62rem/1 var(--font-mono);letter-spacing:.11em}
|
||||
.measurement-equation code{display:block;color:#e6ece8;white-space:normal;line-height:1.8}
|
||||
.validity-triptych{display:grid;grid-template-columns:repeat(3,1fr);gap:10px;margin:25px 0}
|
||||
.validity-triptych article{padding:22px;background:#e6ebe6;border-top:4px solid #77918d}
|
||||
.validity-triptych span{font:700 .65rem/1 var(--font-mono);color:#b66a36}.validity-triptych h3{margin:8px 0;font-size:1.05rem}.validity-triptych p{font-size:.82rem}.validity-triptych b{display:block;color:#37565a;font-size:.78rem}
|
||||
.protocol-passport{display:grid;grid-template-columns:1fr 2fr;gap:1px;background:#b7c3bf;border:1px solid #b7c3bf}
|
||||
.protocol-passport div{padding:18px;background:#f5f6f1}.protocol-passport div:first-child{background:#eee4db}.protocol-passport span,.protocol-passport b{display:block}.protocol-passport span{font:700 .6rem/1 var(--font-mono);color:#a25d31}.protocol-passport b{margin-top:9px;font-size:.82rem;line-height:1.55}
|
||||
.direction-map{display:grid;grid-template-columns:1fr auto 1fr;gap:15px;align-items:center;margin:26px 0;padding:22px;border:1px solid #aebdb8;background:#e7ece7}
|
||||
.direction-map span,.direction-map b{display:block}.direction-map span{font:700 .6rem/1 var(--font-mono);color:#a65d30}.direction-map b{margin-top:8px;font-size:.88rem}.direction-map i{color:#738581;font:600 .64rem/1.5 var(--font-mono);font-style:normal;text-align:center}
|
||||
.formula-board{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:24px 0}.formula-board.two{grid-template-columns:repeat(2,1fr)}
|
||||
.formula-board>div{padding:17px;background:#142f36;color:#f0f2ec}.formula-board span{display:block;color:#d4945d;font:700 .57rem/1.3 var(--font-mono);letter-spacing:.06em}.formula-board code{display:block;margin-top:9px;color:#fff;white-space:normal}
|
||||
.eval-six{grid-template-columns:repeat(3,1fr)}.eval-six article{min-height:145px}
|
||||
.concept-callout{display:grid;grid-template-columns:170px 1fr;gap:20px;margin:25px 0;padding:19px 22px;border-top:2px solid #be7441;border-bottom:1px solid #aebcb8}
|
||||
.concept-callout span{font:700 .68rem/1.5 var(--font-mono);color:#a65a2b}.concept-callout p{margin:0}
|
||||
.suite-lineage{display:grid;grid-template-columns:repeat(5,1fr);gap:8px;margin:25px 0}.suite-lineage article{padding:17px;background:#e4eae5;border-top:4px solid #758e8a}.suite-lineage time{font:700 .6rem/1 var(--font-mono);color:#ac6232}.suite-lineage h3{margin:8px 0}.suite-lineage p{font-size:.76rem}.suite-lineage small{color:#a15c32;font-size:.68rem}
|
||||
.lifecycle-loop{display:flex;gap:7px;align-items:stretch;margin:24px 0}.lifecycle-loop div{flex:1;padding:16px 11px;background:#e1e7e3}.lifecycle-loop i{align-self:center;color:#a75f32;font-style:normal}.lifecycle-loop span,.lifecycle-loop b,.lifecycle-loop small{display:block}.lifecycle-loop span{font:700 .6rem/1 var(--font-mono);color:#a65f33}.lifecycle-loop b{margin:7px 0}.lifecycle-loop small{font-size:.65rem;color:#647674}
|
||||
.repair-table,.k3-protocol-table{margin:24px 0;border:1px solid #aebcb8}.repair-table>div,.k3-protocol-table>div{display:grid;grid-template-columns:.75fr 1.2fr 1.5fr;border-bottom:1px solid #c0cac6}.repair-table>div:last-child,.k3-protocol-table>div:last-child{border-bottom:0}.repair-table span,.repair-table b,.repair-table p,.k3-protocol-table span,.k3-protocol-table b,.k3-protocol-table p{margin:0;padding:12px}.repair-table .head,.k3-protocol-table .head{background:#173239;color:white;font:700 .62rem/1 var(--font-mono)}.repair-table>div:not(.head):nth-child(odd),.k3-protocol-table>div:not(.head):nth-child(odd){background:#e8ece8}.repair-table p,.k3-protocol-table p{font-size:.77rem}
|
||||
.prompt-stack{display:grid;grid-template-columns:repeat(5,1fr);gap:8px;margin:24px 0}.prompt-stack>div{padding:16px;background:#173239;color:white}.prompt-stack span{font:700 .58rem/1 var(--font-mono);color:#d58d54}.prompt-stack b{display:block;margin:8px 0;font-size:.8rem}.prompt-stack p{margin:0;color:#b8c6c3;font-size:.68rem}
|
||||
.example-duel{display:grid;grid-template-columns:1fr 1fr;gap:10px;margin:22px 0}.example-duel article{padding:20px;border:1px solid #b5c2be;background:#eef1ec}.example-duel span{font:700 .6rem/1 var(--font-mono);color:#a75d2e}.example-duel code{display:block;margin:10px 0;padding:10px;background:white}.example-duel p{font-size:.78rem}
|
||||
.sampling-grid,.judge-bias-grid,.threat-grid,.r1-protocol{display:grid;grid-template-columns:repeat(3,1fr);gap:9px;margin:24px 0}.sampling-grid article,.judge-bias-grid article,.threat-grid article,.r1-protocol article{padding:18px;border:1px solid #b9c4c0;background:#eef1ec}.sampling-grid span,.judge-bias-grid span,.threat-grid span,.r1-protocol span{font:700 .58rem/1 var(--font-mono);color:#a75e31}.sampling-grid h3,.judge-bias-grid h3,.threat-grid h3,.r1-protocol h3{margin:8px 0;font-size:.94rem}.sampling-grid p,.judge-bias-grid p,.threat-grid p,.r1-protocol p{font-size:.75rem}.judge-bias-grid b{font-size:.68rem;color:#3b5a5d}
|
||||
.numeric-story{display:grid;grid-template-columns:1.2fr 1fr 1fr;gap:8px;align-items:stretch;margin:24px 0}.numeric-story>b,.numeric-story>div{padding:17px;background:#dfe6e1}.numeric-story div span,.numeric-story div strong{display:block}.numeric-story div span{font:700 .58rem/1 var(--font-mono);color:#657876}.numeric-story div strong{margin-top:8px;font-size:1.2rem;color:#b9632f}.numeric-story p{grid-column:1/-1;margin:0;font-size:.78rem}
|
||||
.calibration-visual{display:grid;grid-template-columns:.8fr 1.2fr;gap:28px;margin:24px 0}.cal-chart{position:relative;min-height:300px;border-left:2px solid #6f8481;border-bottom:2px solid #6f8481;background:repeating-linear-gradient(0deg,transparent 0 19.5%,rgba(50,86,87,.09) 19.5% 20%),repeating-linear-gradient(90deg,transparent 0 19.5%,rgba(50,86,87,.09) 19.5% 20%)}.cal-chart .diag{position:absolute;left:0;bottom:0;width:141%;border-top:2px dashed #79908c;transform:rotate(-45deg);transform-origin:left}.cal-chart>span{position:absolute;left:8px;top:8px;font:700 .58rem/1 var(--font-mono)}.cal-chart>small{position:absolute;right:6px;bottom:8px}.cal-chart>i{position:absolute;width:14px;height:14px;border-radius:50%;background:#477c73;transform:translate(-50%,50%)}.cal-chart>i.bad{background:#c16c39}.calibration-visual h3{margin-top:0}.calibration-visual code{display:block;padding:14px;background:#173239;color:white}.calibration-visual li{margin:.4em 0}
|
||||
.contamination-stack{margin:24px 0}.contamination-stack article{display:grid;grid-template-columns:55px 1.4fr 1fr;gap:18px;align-items:center;padding:16px;border-bottom:1px solid #b7c3bf}.contamination-stack article:nth-child(odd){background:#e7ece7}.contamination-stack span{font:700 .65rem/1 var(--font-mono);color:#ad6232}.contamination-stack h3,.contamination-stack p{margin:0}.contamination-stack h3{font-size:.9rem}.contamination-stack p,.contamination-stack b{font-size:.74rem}.contamination-stack b{color:#496366}
|
||||
.failure-callout{margin:23px 0;padding:20px;background:#312524;color:white;border-left:5px solid #d27841}.failure-callout b,.failure-callout code,.failure-callout p{display:block}.failure-callout b{color:#e4a06a;font:700 .62rem/1 var(--font-mono)}.failure-callout code{margin:10px 0;font-size:1.1rem}.failure-callout p{margin:0;color:#d9d3cc}
|
||||
.tradeoff-grid{display:grid;grid-template-columns:repeat(2,1fr);gap:9px;margin:24px 0}.tradeoff-grid article{padding:20px;background:#e6ebe7;border-top:4px solid #78918d}.tradeoff-grid article.recommended{background:#173239;color:white;border-color:#d18850}.tradeoff-grid span{font:700 .58rem/1 var(--font-mono);color:#ae6435}.tradeoff-grid h3{margin:8px 0}.tradeoff-grid p{font-size:.78rem}.tradeoff-grid b{font-size:.72rem;color:#456267}.tradeoff-grid .recommended b{color:#dca16e}
|
||||
.verifier-ladder{display:flex;gap:7px;align-items:stretch;margin:24px 0}.verifier-ladder>div{flex:1;padding:16px 10px;background:#e2e8e3}.verifier-ladder>i{align-self:center;color:#ad6537;font-style:normal}.verifier-ladder span{font:700 .58rem/1 var(--font-mono);color:#ac6233}.verifier-ladder b{display:block;margin:7px 0;font-size:.76rem}.verifier-ladder p{margin:0;font-size:.66rem}
|
||||
.swe-story,.safety-timeline{display:grid;grid-template-columns:repeat(4,1fr);gap:9px;margin:24px 0}.swe-story article,.safety-timeline article{padding:19px;background:#e4eae5;border-top:4px solid #78908d}.swe-story time,.safety-timeline time{font:700 .63rem/1 var(--font-mono);color:#aa6030}.swe-story h3,.safety-timeline h3{margin:9px 0}.swe-story p,.safety-timeline p{font-size:.75rem}
|
||||
.dual-chain{display:grid;grid-template-columns:1fr 1fr;gap:12px;margin:24px 0}.dual-chain article{padding:22px;background:#173239;color:white}.dual-chain>article>span{font:700 .58rem/1 var(--font-mono);color:#d38b52}.dual-chain h3{margin:9px 0}.dual-chain article>div{display:flex;gap:7px;align-items:center;margin:18px 0}.dual-chain article>div b{padding:8px;background:#26464d;font-size:.68rem}.dual-chain i{color:#d38b52;font-style:normal}.dual-chain p{color:#bac8c5;font-size:.76rem}
|
||||
.agent-equation{display:flex;gap:7px;align-items:stretch;margin:24px 0}.agent-equation>div{flex:1;padding:16px 10px;background:#e2e8e3;text-align:center}.agent-equation>i{align-self:center;color:#a96033;font-style:normal}.agent-equation span{font:700 .57rem/1 var(--font-mono);color:#a96033}.agent-equation b{display:block;margin:8px 0;font-size:.76rem}.agent-equation p{margin:0;font-size:.64rem}
|
||||
.agent-protocol-min,.arena-checks,.cost-denominators,.safety-metrics{display:grid;grid-template-columns:repeat(4,1fr);gap:7px;margin:22px 0}.agent-protocol-min span,.arena-checks span,.cost-denominators span,.safety-metrics span{padding:11px;background:#e3e9e4;font-size:.7rem}.agent-protocol-min b,.arena-checks b{margin-right:8px;color:#a85e30;font:700 .58rem/1 var(--font-mono)}
|
||||
.state-compare{display:grid;grid-template-columns:1fr 1fr;gap:10px;margin:24px 0}.state-compare article{padding:22px;border:1px solid #b4c1bd}.state-compare .weak{background:#eee6df}.state-compare .strong{background:#dfeae4}.state-compare span{font:700 .6rem/1 var(--font-mono);color:#a55e32}.state-compare h3{margin:8px 0}.state-compare b{font-size:.73rem}
|
||||
.reliability-chain{display:flex;gap:13px;align-items:center;justify-content:center;margin:25px 0;padding:22px;background:#173239;color:white}.reliability-chain span{font-size:.8rem}.reliability-chain i{color:#d28b53;font-style:normal}.reliability-chain strong{color:#f0b378}
|
||||
.human-card{display:grid;grid-template-columns:1.3fr .7fr;gap:12px;margin:24px 0}.human-card>div,.human-card blockquote{margin:0;padding:22px;background:#e4eae5}.human-card blockquote{display:flex;flex-direction:column;justify-content:center;background:#173239;color:white;font-size:1rem;line-height:1.7}.human-card blockquote span{color:#dc9760;font:700 .6rem/1 var(--font-mono)}.human-card blockquote small{margin-top:15px;color:#afc0bc}.human-card ol{columns:2;padding-left:22px}.human-card li{margin:.45em 0;font-size:.75rem}
|
||||
.arena-diagram{display:grid;grid-template-columns:.8fr 1.4fr .8fr .8fr;gap:9px;align-items:center;margin:24px 0}.arena-diagram>div{padding:19px;background:#e2e8e3}.arena-diagram span,.arena-diagram b{display:block}.arena-diagram span{font:700 .57rem/1 var(--font-mono);color:#a55e31}.arena-diagram b{margin-top:8px;font-size:.76rem}.arena-diagram .matches{display:grid;grid-template-columns:1fr 1fr;gap:5px;background:#173239;color:white}.arena-diagram .matches i{padding:7px;background:#27464d;font-style:normal;text-align:center}.arena-diagram .matches span{grid-column:1/-1;color:#d9935b}
|
||||
.aggregate-example{margin:24px 0;border:1px solid #aebdb8}.aggregate-example>div{display:grid;grid-template-columns:1fr repeat(5,.8fr);text-align:center}.aggregate-example>div>*{padding:12px}.aggregate-example .score-head{background:#173239;color:white;font:700 .6rem/1 var(--font-mono)}.aggregate-example>div:not(.score-head):nth-child(odd){background:#e4eae5}.aggregate-example .low{color:#b64f2a;font-weight:700}.aggregate-example strong{color:#a95d2d}
|
||||
.editorial-list li{margin:.5em 0}
|
||||
.cost-frontier{position:relative;margin:24px 0;padding:20px 25px 30px;background:#e5ebe6}.cost-frontier svg{width:100%;height:auto}.cost-axis{position:absolute;font:700 .58rem/1 var(--font-mono);color:#5d7371}.cost-axis.y{left:29px;top:11px}.cost-axis.x{right:29px;bottom:11px}
|
||||
.safety-layers{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:24px 0}.safety-layers article{padding:18px;background:#e2e8e3;border-top:4px solid #758e8a}.safety-layers span{font:700 .6rem/1 var(--font-mono);color:#aa6031}.safety-layers h3{margin:8px 0}.safety-layers p{font-size:.74rem}.safety-layers b{font-size:.68rem;color:#466164}
|
||||
.threat-passport,.r1-equation{margin:22px 0;padding:20px;background:#173239;color:white}.threat-passport b,.threat-passport code,.r1-equation span,.r1-equation b{display:block}.threat-passport b,.r1-equation span{color:#d89057;font:700 .6rem/1 var(--font-mono)}.threat-passport code,.r1-equation b{margin-top:10px;color:white;white-space:normal}
|
||||
.safety-timeline{grid-template-columns:repeat(5,1fr)}.safety-timeline span{font:700 .55rem/1.4 var(--font-mono);color:#a65f32}.safety-timeline h3{font-size:.82rem}
|
||||
.negative-predictive{margin:24px 0;padding:25px;background:#342625;color:white;border-left:6px solid #d37b43}.negative-predictive span{font:700 .6rem/1 var(--font-mono);color:#e19a63}.negative-predictive h3{margin:10px 0}.negative-predictive p{margin:0;color:#d9d1ca}
|
||||
.jailbreak-funnel{display:flex;gap:7px;align-items:stretch;margin:24px 0}.jailbreak-funnel>div{flex:1;padding:17px 11px;background:#e3e9e4}.jailbreak-funnel>i{align-self:center;color:#a96234;font-style:normal}.jailbreak-funnel span,.jailbreak-funnel b,.jailbreak-funnel small{display:block}.jailbreak-funnel span{font:700 .58rem/1 var(--font-mono);color:#a75e31}.jailbreak-funnel b{margin:8px 0;font-size:.76rem}.jailbreak-funnel small{color:#647674;font-size:.64rem}
|
||||
.injection-flow{display:flex;gap:7px;align-items:stretch;margin:24px 0}.injection-flow>div{flex:1;padding:17px;background:#e2e8e3}.injection-flow>i{align-self:center;color:#aa6133;font-style:normal}.injection-flow .untrusted{background:#f0dfd2}.injection-flow .danger{background:#392826;color:white}.injection-flow span,.injection-flow b{display:block}.injection-flow span{font:700 .57rem/1 var(--font-mono);color:#aa6133}.injection-flow b{margin-top:8px;font-size:.72rem}
|
||||
.security-cases{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:22px 0}.security-cases article{padding:17px;background:#e4eae5}.security-cases h3{margin:0 0 8px}.security-cases p{font-size:.72rem}
|
||||
.refusal-matrix{display:grid;grid-template-columns:.8fr 1fr 1fr;gap:2px;margin:24px 0;background:#aebcb8;border:1px solid #aebcb8}.refusal-matrix>div{padding:18px;background:#f4f5f0}.refusal-matrix .head{background:#173239;color:white;font:700 .65rem/1 var(--font-mono)}.refusal-matrix .blank{background:#173239}.refusal-matrix .good{background:#dce9e1}.refusal-matrix .bad{background:#efe0d5}.refusal-matrix b,.refusal-matrix span{display:block}.refusal-matrix span{margin-top:6px;font-size:.68rem;color:#647572}
|
||||
.protocol-lineage{margin:24px 0;border-left:2px solid #8fa29e}.protocol-lineage article{position:relative;margin-left:18px;padding:15px 18px;border-bottom:1px solid #b9c4c0}.protocol-lineage article:before{content:"";position:absolute;left:-24px;top:21px;width:10px;height:10px;border-radius:50%;background:#bd7040}.protocol-lineage span{font:700 .59rem/1 var(--font-mono);color:#a95e30}.protocol-lineage h3{margin:7px 0;font-size:.88rem}.protocol-lineage p{margin:0;font-size:.75rem}
|
||||
.author-reported{display:grid;grid-template-columns:.8fr 1.2fr;gap:12px;margin:24px 0;padding:22px;background:#173239;color:white}.author-reported span{font:700 .58rem/1.4 var(--font-mono);color:#d7925a}.author-reported h3{margin:9px 0}.author-reported p{color:#b9c8c4;font-size:.74rem}.reported-grid{display:grid;grid-template-columns:repeat(2,1fr);gap:7px}.reported-grid span{padding:10px;background:#27464d;color:#cad5d2}.reported-grid b{display:block;color:#f0b177;font-size:1rem}
|
||||
.audit-table{display:grid;grid-template-columns:repeat(2,1fr);gap:8px;margin:24px 0}.audit-table article{display:grid;grid-template-columns:35px 85px 1fr;gap:8px;padding:14px;background:#e4eae5}.audit-table span{font:700 .58rem/1.5 var(--font-mono);color:#a75e31}.audit-table b{font-size:.76rem}.audit-table p{margin:0;font-size:.73rem}
|
||||
.conclusion-ladder{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:24px 0}.conclusion-ladder article{padding:17px;background:#e2e8e3;border-top:4px solid #869b97}.conclusion-ladder span{font:700 .58rem/1 var(--font-mono);color:#a95f31}.conclusion-ladder p{font-size:.72rem}.one-line.final{background:#392825}
|
||||
.evaluation-paper-chain{grid-template-columns:repeat(2,minmax(0,1fr))}
|
||||
@media(max-width:1050px){.suite-lineage,.prompt-stack,.safety-timeline{grid-template-columns:repeat(3,1fr)}.sampling-grid,.judge-bias-grid,.threat-grid,.r1-protocol{grid-template-columns:repeat(2,1fr)}.swe-story{grid-template-columns:repeat(2,1fr)}.agent-equation{display:grid;grid-template-columns:repeat(5,1fr)}.agent-equation>i{display:none}}
|
||||
@media(max-width:760px){.eval-ledgers,.validity-triptych,.eval-six,.formula-board,.formula-board.two,.tradeoff-grid,.dual-chain,.human-card,.author-reported,.audit-table,.evaluation-paper-chain{grid-template-columns:1fr}.suite-lineage,.prompt-stack,.safety-timeline,.safety-layers,.security-cases,.conclusion-ladder{grid-template-columns:repeat(2,1fr)}.protocol-passport,.concept-callout,.calibration-visual{grid-template-columns:1fr}.lifecycle-loop,.verifier-ladder,.jailbreak-funnel,.injection-flow{display:grid;grid-template-columns:1fr 1fr}.lifecycle-loop>i,.verifier-ladder>i,.jailbreak-funnel>i,.injection-flow>i{display:none}.agent-equation{grid-template-columns:repeat(2,1fr)}.agent-protocol-min,.arena-checks,.cost-denominators,.safety-metrics{grid-template-columns:repeat(2,1fr)}.arena-diagram{grid-template-columns:1fr 1fr}.numeric-story{grid-template-columns:1fr}.numeric-story p{grid-column:auto}.contamination-stack article{grid-template-columns:42px 1fr}.contamination-stack b{grid-column:2}.audit-table article{grid-template-columns:30px 70px 1fr}}
|
||||
@media(max-width:520px){.suite-lineage,.sampling-grid,.judge-bias-grid,.threat-grid,.r1-protocol,.swe-story,.safety-timeline,.safety-layers,.security-cases,.conclusion-ladder,.agent-equation,.reported-grid{grid-template-columns:1fr}.direction-map,.state-compare{grid-template-columns:1fr}.direction-map i{transform:rotate(90deg)}.example-duel{grid-template-columns:1fr}.reliability-chain{display:grid;text-align:center}.aggregate-example{overflow:auto}.aggregate-example>div{min-width:620px}.repair-table,.k3-protocol-table{overflow:auto}.repair-table>div,.k3-protocol-table>div{min-width:620px}.refusal-matrix{font-size:.78rem}.author-reported{padding:17px}.audit-table article{grid-template-columns:28px 1fr}.audit-table p{grid-column:2}.human-card ol{columns:1}}
|
||||
</style>
|
||||
</BaseLayout>
|
||||
+21
-3
@@ -19,6 +19,7 @@ const routes: Record<string, string> = {
|
||||
"training-systems": "/training-systems/",
|
||||
"systems/inference": "/systems/inference/",
|
||||
"systems/numerics": "/systems/numerics/",
|
||||
evaluation: "/evaluation/",
|
||||
};
|
||||
|
||||
const paths = [
|
||||
@@ -99,6 +100,7 @@ const paths = [
|
||||
<a class="button" href="/training-systems/">训练系统专题</a>
|
||||
<a class="button" href="/systems/inference/">推理服务专题</a>
|
||||
<a class="button" href="/systems/numerics/">数值与优化专题</a>
|
||||
<a class="button" href="/evaluation/">评测与安全专题</a>
|
||||
</div>
|
||||
</div>
|
||||
<aside class="hero-aside" aria-label="项目统计">
|
||||
@@ -106,9 +108,9 @@ const paths = [
|
||||
<strong>29 · 07 · 2026</strong>
|
||||
<p>首版持续建设中，所有动态结论带日期</p>
|
||||
<div class="hero-stats">
|
||||
<div><b>16</b><span>核心专题</span></div>
|
||||
<div><b>17</b><span>核心专题</span></div>
|
||||
<div><b>151</b><span>K3 报告来源</span></div>
|
||||
<div><b>400</b><span>关键论文索引</span></div>
|
||||
<div><b>450</b><span>关键论文索引</span></div>
|
||||
<div><b>47p</b><span>K3 技术报告</span></div>
|
||||
</div>
|
||||
</aside>
|
||||
@@ -122,6 +124,22 @@ const paths = [
|
||||
|
||||
<section class="section compact release-section" id="new-chapters">
|
||||
<div class="release-grid">
|
||||
<a class="release-card evaluation-release" href="/evaluation/">
|
||||
<div>
|
||||
<p class="eyebrow"><span>NEW / CHAPTER 15</span> SCORE · PROTOCOL · THREAT MODEL</p>
|
||||
<h2>榜单不是体检单：一个分数同时测到了模型、协议、脚手架、预算与裁判</h2>
|
||||
<p>
|
||||
用二十二张测量账从 PPL、MMLU、pass@k、校准与污染，走到 LLM Judge、Arena、
|
||||
Agent 最终状态和安全威胁模型；逐字段复原 DeepSeek-R1 与 Kimi K3 的评测协议。
|
||||
</p>
|
||||
</div>
|
||||
<dl>
|
||||
<div><dt>LINEAGE</dt><dd>2002 → 2026</dd></div>
|
||||
<div><dt>NODES</dt><dd>80 个一手节点</dd></div>
|
||||
<div><dt>LAB</dt><dd>指标 · Judge · 污染 · 系统边界</dd></div>
|
||||
</dl>
|
||||
<span class="release-arrow" aria-hidden="true">从一个分数进入完整测量系统 →</span>
|
||||
</a>
|
||||
<a class="release-card inference-release" href="/systems/inference/">
|
||||
<div>
|
||||
<p class="eyebrow"><span>NEW / CHAPTER 14</span> MEMORY · TIME · FLEET</p>
|
||||
@@ -425,7 +443,7 @@ const paths = [
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>02</span> CURRICULUM MAP</p>
|
||||
<h2>16 个专题，拼成一张完整技术地图</h2>
|
||||
<h2>17 个专题，拼成一张完整技术地图</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
进度条表示内容成熟度，不代表相关领域的重要性。首版先打通全站骨架，再逐章扩写到论文与工程层。
|
||||
|
||||
@@ -24,6 +24,7 @@ const workstreams = [
|
||||
{ label: "大规模训练系统", value: 71, next: "补真实集群 traces、故障案例与精确 topology 配置" },
|
||||
{ label: "推理服务与低成本部署", value: 78, next: "补真实 GPU kernel / workload traces、功耗与跨框架复现" },
|
||||
{ label: "数值精度、优化器与稳定性", value: 75, next: "加入真实 kernel 吞吐、长程训练 traces 与逐图论文精读" },
|
||||
{ label: "评测、安全与“到底强不强”", value: 79, next: "补真实 cross-harness 复跑、Judge 元评测与动态污染案例" },
|
||||
{ label: "引用与事实检查", value: 57, next: "自动化外链复查与来源等级扩展" },
|
||||
{ label: "开源与部署", value: 100, next: "每轮保留不可变镜像、提交与回滚点" },
|
||||
];
|
||||
@@ -48,7 +49,7 @@ const workstreams = [
|
||||
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
||||
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
||||
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 08:34 CST</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 09:24 CST</dd></div>
|
||||
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
@@ -58,7 +59,7 @@ const workstreams = [
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>01</span> WORKSTREAMS</p>
|
||||
<h2>十九条工作流同时推进，但不混淆“有页面”和“已核验”</h2>
|
||||
<h2>二十条工作流同时推进，但不混淆“有页面”和“已核验”</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
内容首版优先打通全局脉络；随后每轮迭代选择一个专题推进到论文/工程层，并做独立事实复核。
|
||||
@@ -93,10 +94,10 @@ const workstreams = [
|
||||
<div class="check-grid">
|
||||
<article><span>✓</span><h3>研究目标已持久化</h3><p>总体路线、完成标准、来源等级与进度账本已写入项目。</p></article>
|
||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||
<article><span>✓</span><h3>16 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||
<article><span>✓</span><h3>四十七个原创交互视图</h3><p>K3、语言模型前史、Transformer、DeepSeek、长上下文、MoE、推理、Agent、多模态，以及训练系统、推理服务、Scaling、数据工程、数值和 Alignment 专题。</p></article>
|
||||
<article><span>✓</span><h3>十五篇首版长文</h3><p>K3、语言模型前史、Transformer、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务与数值优化专题。</p></article>
|
||||
<article><span>✓</span><h3>五十一个原创交互视图</h3><p>K3、语言模型前史、Transformer、DeepSeek、长上下文、MoE、推理、Agent、多模态，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>十六篇首版长文</h3><p>K3、语言模型前史、Transformer、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||
<article><span>✓</span><h3>Scaling Laws 深度专题</h3><p>九张账、29 个一手节点、DeepSeek/Kimi 双谱系与曲面—部署—复用—涌现四联实验。</p></article>
|
||||
@@ -110,7 +111,8 @@ const workstreams = [
|
||||
<article><span>✓</span><h3>工具使用与长程 Agent 深度专题</h3><p>十四张账、52 个一手节点、DeepSeek/Kimi Agent 双谱系，以及循环—工具契约—可靠性—长程 RL 四联实验。</p></article>
|
||||
<article><span>✓</span><h3>原生多模态深度专题</h3><p>十六张账、55 个一手节点、DeepSeek 三分支、Kimi 三代 MoonViT，以及 Token—连接器—光学压缩—视觉闭环四联实验。</p></article>
|
||||
<article><span>✓</span><h3>推理服务与低成本部署深度专题</h3><p>十八本账、62 个一手节点、DeepSeek V2→V4 与 Mooncake→K3 双谱系，以及显存—阶段—推测—集群四联实验。</p></article>
|
||||
<article><span>✓</span><h3>400 篇关键论文索引</h3><p>新增 vLLM、SGLang、DistServe、Sarathi、FlashInfer、EAGLE-3、DeepGEMM、FlashMLA 等 45 个推理服务节点。</p></article>
|
||||
<article><span>✓</span><h3>评测、安全与“到底强不强”深度专题</h3><p>二十二张账、80 个一手节点、DeepSeek/K3 评测协议谱系，以及指标—Judge—污染—系统安全四联实验。</p></article>
|
||||
<article><span>✓</span><h3>450 篇关键论文索引</h3><p>新增 BLEU、GLUE、HumanEval、MT-Bench、LiveBench、HarmBench、InjecAgent、AILuminate 等 50 个评测安全节点。</p></article>
|
||||
<article><span>✓</span><h3>公开仓库与自托管发布</h3><p>源码公开到 git.k1412.top，网站由不可变镜像、Compose Manager 与 HTTPS 交付。</p></article>
|
||||
</div>
|
||||
</section>
|
||||
@@ -138,6 +140,7 @@ const workstreams = [
|
||||
<div><span>P1</span><strong>Agent 二轮深化</strong><p>真实环境 traces → cross-harness ablation → Agent RL 曲线与提示注入案例</p><em>运行证据 + 安全案例库</em></div>
|
||||
<div><span>P1</span><strong>原生多模态二轮</strong><p>真实视觉 Token traces → 跨分辨率 / connector 消融 → OCR 与视觉 Agent 安全失败案例</p><em>运行证据 + 逐图笔记</em></div>
|
||||
<div><span>P1</span><strong>推理服务二轮</strong><p>真实 GPU kernel / workload traces → 功耗与成本 → 跨 vLLM / SGLang / TensorRT-LLM 复现</p><em>可复现实测 + 成本账</em></div>
|
||||
<div><span>P1</span><strong>评测安全二轮</strong><p>真实 cross-harness / pass@k 复跑 → Judge 元评测 → 动态污染与过拒案例</p><em>可复现实验 + 协议审计</em></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user