feat: publish reasoning deep dive
This commit is contained in:
wuyang
+18
-9
@@ -1,16 +1,17 @@
|
||||
# 持续进度
|
||||
|
||||
最后更新：2026-07-28
|
||||
最后更新：2026-07-29
|
||||
|
||||
## 总体状态
|
||||
|
||||
| 工作流 | 状态 | 完成度 | 下一检查点 |
|
||||
|---|---:|---:|---|
|
||||
| 研究框架与规范 | 进行中 | 76% | 给 130 篇索引补充逐篇精读层级 |
|
||||
| 研究框架与规范 | 进行中 | 82% | 给推理专题补逐篇图表/实验精读层级 |
|
||||
| 网站设计系统 | 进行中 | 89% | 打印样式与更多通用可视化组件 |
|
||||
| Kimi K3 深读 | 进行中 | 55% | 扩写 scaling / infra 逐图笔记 |
|
||||
| Kimi K3 深读 | 进行中 | 64% | 扩写 scaling / pre-training / infra 逐图笔记 |
|
||||
| Transformer 基础 | 进行中 | 52% | 矩阵形状动画与手算练习 |
|
||||
| DeepSeek 专题 | 进行中 | 61% | GRPO 完整公式与训练轨迹推导 |
|
||||
| DeepSeek 专题 | 进行中 | 71% | 补 R1 / DAPO 的逐图训练轨迹与复现对照 |
|
||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||
| 稀疏计算与 MoE | 完成首版 | 74% | 真实负载 traces 与专家特化案例 |
|
||||
| 长上下文专题 | 完成首版 | 72% | 真实模型配置、内核细节与失败案例 |
|
||||
| 引用与事实检查 | 进行中 | 54% | 自动化外链复查与来源等级扩展 |
|
||||
@@ -26,14 +27,18 @@
|
||||
- [x] 提炼参考网站的编辑设计语言。
|
||||
- [x] 确认 `git.k1412.top` 为 Gitea/Forgejo 兼容服务且本机 HTTPS 凭据可用于既有仓库。
|
||||
- [x] 使用 Grok CLI 检索并形成约 95 篇一手论文的补充路线，主代理已回查关键来源。
|
||||
- [x] 完成首批 130 篇关键论文索引，覆盖 12 个专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成 146 篇关键论文索引，覆盖 12 个专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、Transformer 基础、DeepSeek 谱系、长上下文与 MoE 五篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、Self-Attention 实验、DeepSeek 谱系、长上下文成本与 MoE 路由实验室五张原创交互图。
|
||||
- [x] 完成 K3、Transformer 基础、DeepSeek 谱系、长上下文、MoE 与推理六篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、Self-Attention、DeepSeek 谱系、长上下文、MoE 路由与推理专题三页签等八个原创交互视图。
|
||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||
- [x] 完成 MoE 首版：六张账、19 篇一手论文、完整 DeepSeek/K3 主线与路由—容量—通信实验室。
|
||||
- [x] 为推理专题缓存并核验 23 份一手论文，另复用 K3 原报告，建立 24 份来源的研究账本。
|
||||
- [x] 明确 CoT / verifier / test-time compute、PPO → GRPO、DeepSeek-R1 与 Kimi k1.5 → K3 四条主线。
|
||||
- [x] 分清 K3 partial rollout 的 off-policy 稳定化与 MOPD 的 student on-policy 逐 Token 蒸馏。
|
||||
- [x] 完成推理首版：八张账、30 篇一手论文链、DeepSeek/Kimi 双主线与预算—GRPO—MOPD 三页签实验室。
|
||||
- [x] Astro 类型检查、生产构建、9 个内部路由和桌面/移动端视觉检查通过。
|
||||
- [x] 创建 `wuyang/llm-atlas` 公开仓库，匿名 API 确认 `private: false`。
|
||||
- [x] 本地生产镜像通过健康检查与全部 9 个页面路由烟雾测试。
|
||||
@@ -41,7 +46,8 @@
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] 推理模型与测试时扩展：CoT、verifier、GRPO、R1、Kimi k1.5 与 K3 MOPD。
|
||||
- [ ] 大规模训练系统：ZeRO / Megatron → Expert/Context Parallel → DualPipe / MoonEP。
|
||||
- [ ] 推理专题二轮：真实 pass@k 曲线、PRM 失败案例与逐篇图表精读。
|
||||
- [ ] 长上下文专题的真实模型配置对比、内核细节与失败案例二轮深化。
|
||||
- [ ] MoE 专题的真实集群 traces、专家特化案例与二轮外部证据。
|
||||
|
||||
@@ -53,13 +59,16 @@
|
||||
| 2026-07-28 | K3 作为“汇流点”，不是课程起点 | 初学者可以先学基础，高阶读者可以从 K3 反向跳转 |
|
||||
| 2026-07-28 | 优先重绘论文图并标明“简化/改绘” | 图可缩放、可交互，也减少脱离上下文复制论文图片 |
|
||||
| 2026-07-28 | Grok 只用于线索扩展与交叉检查 | 正文事实必须回到论文、官方仓库或正式文档 |
|
||||
| 2026-07-28 | 首批论文库收录 130 篇，按问题与专题多标签组织 | 论文库承担发现入口，专题正文承担深度精读与机制复核 |
|
||||
| 2026-07-28 | 首批论文库按问题与专题多标签组织，推理研究后扩充至 146 篇 | 论文库承担发现入口，专题正文承担深度精读与机制复核 |
|
||||
| 2026-07-28 | 源码公开到 `git.k1412.top/wuyang/llm-atlas` | 路线、进度、研究方法和内容变更均可追踪 |
|
||||
| 2026-07-28 | 站点使用不可变镜像与 Compose Manager 部署 | 每次发布保留明确版本、健康检查和回滚点 |
|
||||
| 2026-07-28 | 长上下文按计算、缓存、位置、状态容量、系统五张账单组织 | 避免把 FlashAttention、位置外推和“记住更久”混成同一个问题 |
|
||||
| 2026-07-28 | 交互缓存数字统一标记为教学估算 | 展示增长规律，不冒充任一模型的真实线上显存基准 |
|
||||
| 2026-07-28 | MoE 按六张账组织，路由算法与集群执行分开核算 | 避免用“稀疏所以便宜”跳过容量、负载、通信与权重读取 |
|
||||
| 2026-07-28 | “aux-loss-free”保留论文真实边界 | 区分 selection bias、mixture weight、z-loss 与 V3 的极小 sequence-wise loss |
|
||||
| 2026-07-29 | 推理专题按结果、覆盖、选择、过程、预算、优化、分布、系统八张账组织 | 避免把 pass@1、pass@k、搜索收益、RL 能力增长与系统吞吐混成“会思考” |
|
||||
| 2026-07-29 | K3 partial rollout RL 与 MOPD 在正文中强制并排 | 前者显式容忍跨迭代 stale trajectory，后者由当前 student 采样并接收稠密 teacher signal |
|
||||
| 2026-07-29 | 推理首版用 30 篇一手论文和三页签实验闭环 | 分开演示预算分配、GRPO 聚合偏置和 K3 九教师 MOPD，不合成伪“总分” |
|
||||
|
||||
## 未决问题
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
- 持续进度：[PROGRESS.md](./PROGRESS.md)
|
||||
- 证据与写作规范：[research/METHODOLOGY.md](./research/METHODOLOGY.md)
|
||||
|
||||
当前里程碑包含 16 专题学习地图、130 篇关键论文索引、Kimi K3 完整导读、
|
||||
当前里程碑包含 16 专题学习地图、146 篇关键论文索引、Kimi K3 完整导读、
|
||||
Transformer 基础、DeepSeek 技术谱系、长上下文与 MoE 深度专题，以及五张原创交互可视化。
|
||||
其余专题按进度账本持续扩建。
|
||||
|
||||
|
||||
+2
-1
@@ -10,7 +10,8 @@
|
||||
"preview": "astro preview --host 0.0.0.0",
|
||||
"check": "astro check",
|
||||
"check:site": "node scripts/check-site.mjs",
|
||||
"check:moe-browser": "node scripts/check-moe-browser.mjs"
|
||||
"check:moe-browser": "node scripts/check-moe-browser.mjs",
|
||||
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"@astrojs/sitemap": "3.7.3",
|
||||
|
||||
@@ -0,0 +1,898 @@
|
||||
# 推理模型与测试时扩展研究账本
|
||||
|
||||
状态：原始论文核验中，正文尚未发布
|
||||
研究截止：2026-07-28
|
||||
本轮本地缓存：23 份一手 PDF / 文本，另复用 Kimi K3 官方技术报告
|
||||
线索发现：K3 references、论文引用网络、作者/机构页面、Grok CLI
|
||||
结论依据：原论文正文、附录、官方技术报告；Grok 只用于扩大检索覆盖
|
||||
|
||||
## 0. 这一章真正要回答什么
|
||||
|
||||
“模型会推理”不是一个单一指标。至少要把下面八张账分开：
|
||||
|
||||
1. **结果账**：第一次回答是否正确，通常看 pass@1。
|
||||
2. **覆盖账**：多采样以后，至少有一个正确答案的概率，通常看 pass@k。
|
||||
3. **选择账**：如果正确答案已经在候选里，投票、ORM、PRM 或 verifier 能否选出来。
|
||||
4. **过程账**：中间步骤是否有效、可检查，以及写出来的 CoT 是否真是模型决策的因果解释。
|
||||
5. **预算账**：额外计算花在更长的单条轨迹、更多并行样本、树搜索，还是工具调用。
|
||||
6. **优化账**：PPO、GRPO、DAPO、Dr.GRPO 与 Kimi 的目标函数究竟改变了哪个梯度。
|
||||
7. **分布账**：训练是在发现新解法，还是把已有正确解法的概率质量推到前面，同时牺牲多样性。
|
||||
8. **系统账**：长尾 rollout、KV 状态、沙箱、验证器和教师 prefill 让训练吞吐付出什么代价。
|
||||
|
||||
本章核心问题不是“哪篇论文赢了 benchmark”，而是：
|
||||
|
||||
> 给定一个模型、一个问题和一笔有限预算，
|
||||
> 如何产生候选、分配计算、评价过程、选择答案，并知道能力到底来自哪里？
|
||||
|
||||
## 1. 贯穿全章的三条因果链
|
||||
|
||||
### 1.1 从“写步骤”到“分配测试时计算”
|
||||
|
||||
```text
|
||||
直接回答
|
||||
→ Chain-of-Thought：把计算摊到更多串行 Token
|
||||
→ Self-Consistency：对多条 CoT 做并行采样与投票
|
||||
→ ORM / PRM：不只投票，而是学习判断结果或中间步骤
|
||||
→ Tree of Thoughts：主动展开、评价、回溯
|
||||
→ ReAct：把内部推理与外部行动/观察交错
|
||||
→ compute-optimal allocation：按问题难度在串行、并行、搜索之间分预算
|
||||
→ reasoning effort：训练一个模型在 low / high / max 下改变推理预算
|
||||
```
|
||||
|
||||
### 1.2 从“人类偏好 RL”到“可验证奖励 RL”
|
||||
|
||||
```text
|
||||
PPO：actor + critic + clipped surrogate
|
||||
→ RLHF：偏好奖励模型给标量奖励
|
||||
→ DeepSeekMath GRPO：同题成组采样，用组内相对奖励代替 critic
|
||||
→ DeepSeek-R1-Zero：规则奖励直接驱动长 CoT
|
||||
→ DeepSeek-R1：cold start + reasoning RL + rejection/SFT + general RL
|
||||
→ DAPO：Clip-Higher、Dynamic Sampling、token-level loss、overlong shaping
|
||||
→ Dr.GRPO：去掉长度与组标准差归一化，揭示原始 GRPO 的两类偏置
|
||||
→ RLVR 边界争论：pass@1 上升是否伴随大 k 覆盖下降
|
||||
```
|
||||
|
||||
### 1.3 Kimi 的长轨迹与能力整合线
|
||||
|
||||
```text
|
||||
Kimi k1.5
|
||||
128K RL context + partial rollout + online mirror-descent surrogate
|
||||
+ long2short（merge / RS / DPO / RL）
|
||||
→ Kimi K2
|
||||
verifiable rewards gym + self-critique rubric reward
|
||||
+ budget control + PTX + temperature decay
|
||||
→ Kimi K2.5
|
||||
token-level off-policy clipping + Toggle 预算训练
|
||||
+ unified agentic RL + 100K concurrent tasks
|
||||
→ Kimi K3
|
||||
3 domains × 3 reasoning efforts = 9 RL teachers
|
||||
+ extreme off-policy partial rollout stabilization
|
||||
+ Multi-Teacher On-Policy Distillation
|
||||
+ million-token agentic RL / AgentENV
|
||||
```
|
||||
|
||||
## 2. 指标词典：先统一“正确率”口径
|
||||
|
||||
### 2.1 pass@1
|
||||
|
||||
从指定采样策略抽一个答案时正确的概率。很多论文为了降低方差，会对同一题生成多次，再使用无偏估计汇总成 pass@1；它不一定等于“贪心解码一次”的准确率。
|
||||
|
||||
必须同时记录：
|
||||
|
||||
- temperature / top-p；
|
||||
- 最大输出长度；
|
||||
- 是否有工具；
|
||||
- prompt template；
|
||||
- 重复运行次数；
|
||||
- 是否使用 grader / verifier。
|
||||
|
||||
### 2.2 pass@k
|
||||
|
||||
在 `k` 次候选中至少出现一个正确答案的概率。若一共采样 `n` 个，其中 `c` 个正确，无偏估计为：
|
||||
|
||||
```text
|
||||
pass@k = 1 - C(n-c, k) / C(n, k)
|
||||
```
|
||||
|
||||
它近似回答“这个分布里还覆盖着正确解法吗”，但不是能力的完美真值：
|
||||
|
||||
- 有限 `k` 只能看到有限尾部；
|
||||
- temperature 会改变覆盖；
|
||||
- prompt 与答案解析器会改变 `c`；
|
||||
- 高 pass@k 不代表实际系统能找到并选中正确答案。
|
||||
|
||||
### 2.3 majority@k / consensus@k
|
||||
|
||||
采样 `k` 条推理，把最终答案归一化后多数投票。它依赖“正确答案形成稳定簇”的假设：
|
||||
|
||||
- 多样错误彼此抵消时很有效；
|
||||
- 同一种系统性错误占多数时会一起错；
|
||||
- 自由文本任务难以稳定抽取等价答案。
|
||||
|
||||
### 2.4 best-of-N
|
||||
|
||||
先生成 `N` 个候选，再由 verifier / reward model 选择最高分答案。最终成功率拆成两部分：
|
||||
|
||||
```text
|
||||
候选覆盖：正确答案有没有生成
|
||||
×
|
||||
选择可靠性：评分器有没有把正确答案排到前面
|
||||
```
|
||||
|
||||
因此 best-of-N 的提升不能全部归功于生成模型，也不能只报告 verifier 分数而不报告候选覆盖。
|
||||
|
||||
### 2.5 训练时计算与测试时计算
|
||||
|
||||
| 口径 | 花在哪里 | 常见混淆 |
|
||||
|---|---|---|
|
||||
| 预训练计算 | 更多数据、参数、训练 Token | 与 inference scaling 不是同一条轴 |
|
||||
| 后训练计算 | SFT、RL rollout、reward、teacher prefill | 训练变贵不代表单次推理变贵 |
|
||||
| 串行测试时计算 | 更长 CoT、反思、修订 | Token 多不必然更正确 |
|
||||
| 并行测试时计算 | 多采样、self-consistency、best-of-N | 可并行但总 Token 大 |
|
||||
| 搜索计算 | 分支、评价、回溯 | verifier 错误会被搜索放大 |
|
||||
| 工具计算 | 搜索、代码执行、环境交互 | 延迟和成本不只来自模型 Token |
|
||||
|
||||
## 3. 八张教学账
|
||||
|
||||
### 3.1 结果账：它最后答对了吗
|
||||
|
||||
最适合规则可验证任务，例如数学最终答案、单元测试、棋局合法性。优势是奖励便宜且客观；局限是：
|
||||
|
||||
- 最终答案对，不保证过程可靠；
|
||||
- 奖励函数有漏洞时，模型会学会利用漏洞；
|
||||
- 二元奖励在极难或极易题上几乎不给区分信号；
|
||||
- 开放式写作、研究与复杂代理任务没有唯一 verifier。
|
||||
|
||||
### 3.2 覆盖账：正确路径还在分布里吗
|
||||
|
||||
pass@1 上升可能来自两种完全不同的变化：
|
||||
|
||||
```text
|
||||
A. 真正扩展：新增原先几乎不会出现的解法模式
|
||||
B. 分布锐化：把原先低概率的正确模式推到更高概率
|
||||
```
|
||||
|
||||
DeepSeekMath 已在 2024 年报告：RL 提升 Maj@K，却没有改善 Pass@K；作者谨慎解释为输出分布更稳健、正确答案从 Top-K 被推向前面，而不是已经证明基础能力扩展。
|
||||
|
||||
2025 年的 RLVR capacity-limit 工作把这一问题扩大到多个数学、代码和视觉设置：当前测试中的 RLVR 模型常在小 `k` 更强，但基础模型会在大 `k` 追上或超过。它是对一组当前算法的实证边界，不是“RL 永远不能创造能力”的定理。
|
||||
|
||||
### 3.3 选择账：如何从候选里找到对的
|
||||
|
||||
选择器的主要分支：
|
||||
|
||||
1. **答案投票**：不训练 verifier，只看答案频次。
|
||||
2. **Outcome Reward Model（ORM）**：整条解答得到一个分数。
|
||||
3. **Process Reward Model（PRM）**：每一步得到分数，再聚合。
|
||||
4. **规则 verifier**：执行代码、比较数学答案、检查约束。
|
||||
5. **Generative Reward Model（GRM）**：先生成评价过程/rubric，再给分。
|
||||
|
||||
选择器不是免费 oracle。它可能：
|
||||
|
||||
- 偏爱格式、长度或表面完整性；
|
||||
- 在分布外候选上失准；
|
||||
- 把局部错误一路传播到树搜索；
|
||||
- 被策略模型共同训练后产生共谋式 reward hacking。
|
||||
|
||||
### 3.4 过程账：步骤好看、有效、忠实是三回事
|
||||
|
||||
必须分开：
|
||||
|
||||
- **有效性**：这一步是否有助于得到正确答案；
|
||||
- **可验证性**：外部规则或 PRM 能否判断这一步；
|
||||
- **可读性**：人是否容易理解；
|
||||
- **忠实性**：公开写出的步骤是否真反映模型产生答案的因果过程。
|
||||
|
||||
Turpin 等人的偏置提示实验显示，模型能给出听起来合理但隐瞒偏置影响的解释；“正确/流畅的 CoT”不能自动当作模型内部因果证词。这不是说所有 CoT 都不可信，而是说明公开推理文本必须接受独立 faithfulness 检查。
|
||||
|
||||
### 3.5 预算账：四种测试时扩展不是一回事
|
||||
|
||||
#### 串行深度
|
||||
|
||||
同一条轨迹继续写、检查、修订。适合一个局部错误可以被后续发现的题；风险是错误前提越写越深。
|
||||
|
||||
#### 并行宽度
|
||||
|
||||
从同一问题采样多个独立候选。适合模型有多种可行路径、错误较分散的题；风险是样本高度相关，新增候选边际收益迅速下降。
|
||||
|
||||
#### 显式搜索
|
||||
|
||||
在“状态—候选思路—评价—回溯”树上分配预算。适合步骤可局部评价的问题；风险是搜索控制和 verifier 本身消耗大量计算。
|
||||
|
||||
#### 工具/环境
|
||||
|
||||
让模型通过搜索、代码执行、浏览器或软件环境获得新观察。它不只是“想更久”，而是改变信息集。对 agentic RL 来说，这也是最可能超越单轮静态提示能力边界的路径。
|
||||
|
||||
Snell 等人的 compute-optimal 工作在专门训练的 PaLM 2 模型与 MATH 设置中表明：
|
||||
|
||||
- 最优策略依题目难度和模型而变；
|
||||
- 其最优分配在特定实验中可用约 4 倍更少计算超过 best-of-N；
|
||||
- 在一些 FLOPs 匹配条件下，小模型可超过约 14 倍大的模型；
|
||||
- 但最难题常从当前测试时计算中获益很少，继续预训练更有效。
|
||||
|
||||
这些数字不能外推成“推理计算总能替代参数规模”。
|
||||
|
||||
### 3.6 优化账：每个目标函数到底改了什么
|
||||
|
||||
#### PPO
|
||||
|
||||
PPO 使用 actor 产生动作、critic 估计价值/优势，并通过 clipped surrogate 限制单次策略更新。用于 LLM RLHF 时还常加入 reference KL、reward model 和 GAE。它的工程代价包括：
|
||||
|
||||
- 额外价值网络或 value head；
|
||||
- rollout、reference、reward、critic 多模型协调；
|
||||
- 长序列信用分配；
|
||||
- 训练/推理引擎概率不一致。
|
||||
|
||||
#### DeepSeekMath GRPO
|
||||
|
||||
对同一问题 `q`，从旧策略采样一组 `G` 个输出。用组内奖励均值作为 baseline，并以组内标准差归一化：
|
||||
|
||||
```text
|
||||
A_i = (r_i - mean(r_1...r_G)) / std(r_1...r_G)
|
||||
```
|
||||
|
||||
Outcome supervision 时，同一输出的每个 token 共用 `A_i`。原始 DeepSeekMath GRPO：
|
||||
|
||||
- 不训练 critic；
|
||||
- 对每个回答先做 token 平均，再对组平均；
|
||||
- 把 KL 直接加入目标，而不是先从 reward 中扣；
|
||||
- 使用论文给出的正值 KL 估计器；
|
||||
- 只对“组内有奖励差异”的问题产生相对信号。
|
||||
|
||||
DeepSeekMath-RL 的报告设置包括：约 144K 个 GSM8K/MATH CoT 问题、每题 64 个输出、最大 1024 tokens、KL 系数 0.04。该模型在报告中 MATH 从 46.8 提升到 51.7，64-sample self-consistency 为 60.9；只能按论文设置解释。
|
||||
|
||||
#### DAPO
|
||||
|
||||
DAPO 不是简单“把 GRPO 再跑大一点”，而是修复实际训练中的四个断点：
|
||||
|
||||
1. **Clip-Higher**：上下裁剪范围解耦，给低概率 token 更大上升空间，缓解熵坍缩。
|
||||
2. **Dynamic Sampling**：过滤组内全对或全错的 prompt，因为相对优势全为零。
|
||||
3. **Token-level Policy Gradient Loss**：整批 token 等权聚合，改变长短回答的梯度权重。
|
||||
4. **Overlong Reward Shaping**：在长度上限附近平滑惩罚，避免硬截断制造噪声。
|
||||
|
||||
DAPO 在其 Qwen2.5-32B Base、数据、系统和评测设置中达到 AIME 2024 平均 50；论文对比的 naive GRPO 为 30、引用的 R1-Zero-Qwen-32B 为 47。由于复现数据与系统并不完全相同，不能写成无条件“DAPO 超过 R1”。
|
||||
|
||||
#### Dr.GRPO
|
||||
|
||||
Dr.GRPO 指出原始 GRPO 的两个潜在偏置：
|
||||
|
||||
1. **response-level length bias**：每条回答除以自己的长度，会让短正确答案获得更强正梯度、长错误答案获得更弱负梯度。
|
||||
2. **question-level difficulty bias**：再除以组内 reward 标准差，会使某些难度的题获得不同权重。
|
||||
|
||||
其改法是去掉回答长度与组标准差归一化，用固定全局最大 token 数作为实现中的分母。论文还展示 DeepSeek-V3 Base 在 RL 前已经会出现 “aha”/反思式表达，因此不能仅凭训练后出现 “wait/aha” 就断言 RL 从零发明了反思。
|
||||
|
||||
这是一个有明确假设和实验范围的批判，不代表所有 GRPO 结果无效。
|
||||
|
||||
#### Kimi k1.5 的 mirror-descent surrogate
|
||||
|
||||
k1.5 从迭代参考策略采样 `k` 条回答，使用组均值奖励 baseline，并优化“奖励 − 与迭代参考策略的 KL 正则”对应的 surrogate。论文强调：
|
||||
|
||||
- 不使用 value network；
|
||||
- 每轮重置优化器；
|
||||
- 允许使用 off-policy 数据；
|
||||
- 128K context 与 partial rollout 让一条超长轨迹跨训练迭代；
|
||||
- 局部错误之后仍可能恢复，因此不在本文框架里依赖 PRM/value/MCTS。
|
||||
|
||||
“不依赖 PRM/MCTS”只描述 k1.5 的选择，不能泛化为搜索与过程监督无效。
|
||||
|
||||
#### Kimi K2 的统一 RL
|
||||
|
||||
K2 延续 k1.5 的组相对奖励与 squared log-ratio 正则，并扩展任务/奖励：
|
||||
|
||||
- 数学、STEM、逻辑、代码、软件工程、复杂指令等 verifiable gym；
|
||||
- 对开放任务使用 self-critique rubric reward；
|
||||
- critic 用可验证任务的 on-policy rollout 持续校准；
|
||||
- 分任务 token budget，超预算截断并惩罚；
|
||||
- 高质量 PTX loss 防遗忘；
|
||||
- temperature decay 从探索过渡到稳定利用。
|
||||
|
||||
K2 的“self-critique”不是模型随口说“我觉得好”，而是 pairwise 比较，结合 core、prescriptive 和人工 rubric；它仍可能受 judge 偏差与 reward hacking 影响。
|
||||
|
||||
#### Kimi K2.5 的 token-level off-policy clipping
|
||||
|
||||
K2.5 对每个 token 的新旧策略比率做 `[α, β]` 区间裁剪。正文明确说明：
|
||||
|
||||
- 区间内正常计算 policy gradient；
|
||||
- 区间外梯度屏蔽；
|
||||
- 只依据 log-ratio 是否越界；
|
||||
- 不像标准 PPO 那样根据 advantage 正负决定截哪一边；
|
||||
- 目的是约束训练引擎与推理引擎差异放大的 off-policy drift。
|
||||
|
||||
K2.5 同时提出 **Toggle**：
|
||||
|
||||
- Phase 0：当同题平均正确率超过阈值时，才按正确样本长度分位数施加预算；
|
||||
- Phase 1：恢复最大长度，继续学习利用更多推理计算；
|
||||
- 两阶段每 `m` 轮交替。
|
||||
|
||||
直觉是避免只做“越短越好”以后失去向上扩展预算的能力。
|
||||
|
||||
### 3.7 分布账：pass@1 上升不等于能力空间变大
|
||||
|
||||
教学时固定画两张分布：
|
||||
|
||||
```text
|
||||
基础模型：正确模式概率低，但模式覆盖较宽
|
||||
RL 模型：正确模式概率高，但部分低概率模式消失
|
||||
```
|
||||
|
||||
需要同时问：
|
||||
|
||||
- pass@1 是否上升；
|
||||
- pass@k 曲线在多大 `k` 后交叉；
|
||||
- entropy 是否下降；
|
||||
- 相同 temperature 还是 entropy-matched temperature；
|
||||
- 训练 prompt 和测试 prompt 是否相同；
|
||||
- distillation 是否从更强教师引入了基础模型原本没有的模式。
|
||||
|
||||
RLVR capacity-limit 论文观察到，蒸馏模型的 pass@k 可以超过基础模型，并据此把“从更强教师转入新模式”与“当前 on-policy RL 对已有模式重加权”区分开。但这仍依赖有限采样与具体教师。
|
||||
|
||||
### 3.8 系统账：长推理不是只把 max_tokens 改大
|
||||
|
||||
长轨迹训练带来的系统问题：
|
||||
|
||||
- 一批 rollout 中少数极长样本拖住全部 GPU；
|
||||
- 长序列 KV cache 占用巨大；
|
||||
- 工具/浏览器/沙箱在模型思考时空闲，在环境运行时 GPU 又可能空闲；
|
||||
- 策略更新后，未完成轨迹变成 stale/off-policy；
|
||||
- 训练引擎与高吞吐推理引擎可能给出不同 log-prob；
|
||||
- verifier、teacher prefill 和环境执行形成新的流水线。
|
||||
|
||||
Kimi 的演化正好提供一条系统线：
|
||||
|
||||
- k1.5：partial rollout、replay buffer、长轨迹分段；
|
||||
- K2.5：每个 agent task 是异步 coroutine，Rollout Manager 支持最多 100K concurrent tasks；
|
||||
- K3：几百张 GPU 上的 co-located RL、外部 CPU DRAM KV pool、NVMe 状态卸载、自动节流与 AgentENV。
|
||||
|
||||
## 4. DeepSeek 高亮主线
|
||||
|
||||
### 4.1 DeepSeekMath：GRPO 先是一种“去 critic”的工程/统计选择
|
||||
|
||||
不能把 GRPO 缩成一句“PPO 不要 value model”。它同时改变了：
|
||||
|
||||
- baseline：同题组均值；
|
||||
- advantage normalization：组内标准差；
|
||||
- loss aggregation：先回答内 token 平均；
|
||||
- KL 放置与估计；
|
||||
- 可学习 prompt：全对/全错组没有相对信号。
|
||||
|
||||
还要把论文自己的保守结论放在正文：RL 提升 Maj@K 而非 Pass@K，作者当时已没有把它夸张成必然创造新能力。
|
||||
|
||||
### 4.2 R1-Zero：规则奖励能让长 CoT 自组织，但现象不等于机制证明
|
||||
|
||||
R1-Zero：
|
||||
|
||||
- 从 DeepSeek-V3 Base 直接开始 RL，没有先做 reasoning SFT；
|
||||
- 使用 GRPO；
|
||||
- 使用规则 accuracy reward 与 format reward；
|
||||
- 不使用神经 ORM/PRM 作为 reasoning reward，理由包括 reward hacking 与复杂训练管线；
|
||||
- 随训练出现更长回答、反思、验证和被称为 “aha moment” 的行为；
|
||||
- 同时存在可读性差、语言混合等问题。
|
||||
|
||||
应把“观察到长推理/反思”写成训练现象，而不是断言某个单一 token 或表达就是能力涌现的因果标志。Dr.GRPO 对基础模型的检查尤其提醒这一点。
|
||||
|
||||
### 4.3 R1：完整管线不是“纯 RL”
|
||||
|
||||
R1 的公开管线：
|
||||
|
||||
```text
|
||||
少量高质量 cold-start reasoning data
|
||||
→ reasoning-oriented RL
|
||||
→ rejection sampling
|
||||
→ reasoning + non-reasoning SFT
|
||||
→ broader RL（helpfulness / safety 等）
|
||||
```
|
||||
|
||||
边界：
|
||||
|
||||
- R1-Zero 才是“base 直接 RL”的主要实验；
|
||||
- R1 使用 SFT、rejection sampling 和多阶段 RL；
|
||||
- reasoning 阶段偏向规则奖励，后续 general RL 仍有 reward models；
|
||||
- 扩展版报告记载 model preference reward 只在最后约 400 steps 使用，过久会 reward hack；
|
||||
- 报告对 AIME/GPQA 常采样 64 次、MATH/Codeforces 16 次，再估计 pass@1；consensus 另列，不能把两者混用。
|
||||
|
||||
### 4.4 R1 蒸馏：小模型得到的是强教师轨迹
|
||||
|
||||
六个 1.5B–70B distilled 模型使用约 800K 由 R1 产生/筛选的数据进行 SFT，本身没有再做该报告中的 RL。附录对比显示在小模型上蒸馏更有效，但报告同时认为 RL 仍是继续突破更强能力的必要方向。
|
||||
|
||||
所以：
|
||||
|
||||
- “小模型只靠 SFT 就会推理”缺了强教师数据来源；
|
||||
- “蒸馏证明 RL 不需要”也不成立，因为教师本身来自更重的训练管线。
|
||||
|
||||
### 4.5 DAPO 与 Dr.GRPO：从复现失败反推算法细节
|
||||
|
||||
这两篇论文最适合作为“研究如何进步”的案例：
|
||||
|
||||
```text
|
||||
R1 给出强结果与简要 GRPO 配方
|
||||
→ naive reproduction 明显落后
|
||||
→ DAPO 暴露熵、无信号组、长序列聚合、截断四个工程断点
|
||||
→ Dr.GRPO 再追问长度增长和 aha 是否可能来自目标函数偏置/基础模型
|
||||
```
|
||||
|
||||
这不是互相推翻，而是把“RL 有效”拆成更可审计的机制。
|
||||
|
||||
## 5. Kimi 高亮主线
|
||||
|
||||
### 5.1 Kimi k1.5：把 RL 的序列长度本身当作 scaling axis
|
||||
|
||||
关键配置/结论：
|
||||
|
||||
- 最大 RL context 128K；
|
||||
- long-CoT 报告 AIME 77.5、MATH 500 96.2、Codeforces 94th percentile、MathVista 74.9；
|
||||
- partial rollout 让长回答跨迭代继续；
|
||||
- replay buffer 保存旧片段，当前轮只对可用部分做 on-policy 计算，并可把旧片段排除出 loss；
|
||||
- 长度惩罚在初期能力增长后再 warm up。
|
||||
|
||||
数字只能按论文的模型、采样与评测协议使用。
|
||||
|
||||
### 5.2 long2short 不是一种蒸馏算法
|
||||
|
||||
k1.5 的 long2short 是四类方法：
|
||||
|
||||
1. **weight averaging**：合并长 CoT 与短模型；
|
||||
2. **shortest rejection sampling**：每题采样 8 条，选最短正确答案；
|
||||
3. **DPO**：最短正确为正样本，较长错误或超过 1.5 倍的正确回答为负样本；
|
||||
4. **long2short RL**：加强长度惩罚并降低最大 rollout 长度。
|
||||
|
||||
报告中 long2short RL 在 AIME 2024 的 8 次运行 pass@1 为 60.8、平均约 3,272 tokens。它说明可以优化“正确率—长度”前沿，不说明短思维总能保留全部长思维能力。
|
||||
|
||||
### 5.3 K2：把可验证推理扩展到一般任务
|
||||
|
||||
K2 的新意不是继续拉长数学 CoT，而是建立从规则奖励到 subjective rubric 的闭环：
|
||||
|
||||
```text
|
||||
可验证任务的 on-policy rollout
|
||||
→ 用客观信号持续校准 critic
|
||||
→ critic 对开放任务做 pairwise rubric evaluation
|
||||
→ actor 学习更一般的偏好
|
||||
```
|
||||
|
||||
这是一种把 verifier 能力迁移到开放任务 judge 的尝试；开放任务奖励仍不是客观真值。
|
||||
|
||||
### 5.4 K2.5：训练效率与推理预算要一起控制
|
||||
|
||||
K2.5 的两项关键桥梁：
|
||||
|
||||
- token-level probability-ratio clipping（用于约束 log-ratio drift），处理大规模异步 rollout 的 off-policy 偏移；
|
||||
- Toggle 在 budget-limited 与 standard scaling 间交替，保留模型继续使用更多推理计算的能力。
|
||||
|
||||
其 unified agentic RL 还把 text、vision、parallel-agent RL 放进同一异步环境，说明“reasoning model”已从单轮数学走向多模态、工具与并行代理控制。
|
||||
|
||||
### 5.5 K3：九个专家怎样合成一个可控模型
|
||||
|
||||
K3 post-training 有三阶段：
|
||||
|
||||
```text
|
||||
Stage 1: SFT
|
||||
Stage 2: domain × reasoning effort RL
|
||||
Stage 3: Multi-Teacher On-Policy Distillation
|
||||
```
|
||||
|
||||
Stage 2 的三个领域：
|
||||
|
||||
1. general：经验、视觉、推理、faithfulness、search、knowledge work；
|
||||
2. general agent：long-horizon assistant、deep research、paragraph writing；
|
||||
3. coding agent：SWE、coding experience、kernel、web development。
|
||||
|
||||
每个领域再训练 `low / high / max` 三档 reasoning effort，共 9 个教师。
|
||||
|
||||
#### reasoning effort RL
|
||||
|
||||
对每题先估计初始预算 `b0(x)`。如果回答长度 `T(y)` 超过 `τ · b0(x)`，任务奖励被覆盖为 `-1`：
|
||||
|
||||
```text
|
||||
max effort：先用较大的 τ 学会充分计算
|
||||
→ high / low：逐步 anneal τ，压缩可用预算
|
||||
```
|
||||
|
||||
它不是推理时简单截断，而是在训练时改变“超预算回答”的奖励。
|
||||
|
||||
#### Agentic GRM
|
||||
|
||||
K3 要求生成式奖励模型遵循：
|
||||
|
||||
```text
|
||||
outcome → rubric → score → scorepad
|
||||
```
|
||||
|
||||
为缓解模型通过冗长回答骗取分数，候选长度超过阈值 `σ · l0` 时会在二元比较中自动失败。这里控制的是 judge 的 verbosity 偏好，不等于 reasoning effort 的长度奖励。
|
||||
|
||||
## 6. K3 最容易混淆的两种训练
|
||||
|
||||
### 6.1 partial rollout RL：允许长轨迹跨迭代，显式处理 stale/off-policy
|
||||
|
||||
对 `N` 个 prompt、每题 `K` 条轨迹：
|
||||
|
||||
1. 并行生成 `N×K` 条 rollout；
|
||||
2. 当其中 `λNK` 条完成，就暂停 generation；
|
||||
3. 已完成轨迹进入本轮 policy optimization；
|
||||
4. 未完成轨迹排队，后续迭代恢复；
|
||||
5. 一条超长轨迹因此可能横跨多个 policy 版本。
|
||||
|
||||
K3 使用 per-token regularization，把更新限制在局部邻域，以容忍这种“极端 off-policy”状态。这里解决的是长尾吞吐与旧策略轨迹问题。
|
||||
|
||||
### 6.2 MOPD：学生 on-policy，教师给逐 token 稠密信号
|
||||
|
||||
对领域 `d`、effort `e`：
|
||||
|
||||
1. 学生 `πθ` 自己生成轨迹；
|
||||
2. 路由到对应冻结教师 `πteacher(d,e)`；
|
||||
3. 教师对学生已经走过的前缀做 prefill；
|
||||
4. 每个学生采样 token 得到教师—学生 log-prob 差；
|
||||
5. clip 后作为 dense token reward / advantage。
|
||||
|
||||
K3 报告中的形式可简写为：
|
||||
|
||||
```text
|
||||
r_opd,t =
|
||||
clip(
|
||||
stop_gradient[
|
||||
log π_teacher(y_t | e, x, y_<t)
|
||||
- log π_student(y_t | e, x, y_<t)
|
||||
],
|
||||
-R_max,
|
||||
R_max
|
||||
)
|
||||
```
|
||||
|
||||
独立 MOPD 论文把它推导为 student-to-teacher reverse KL 的 policy-gradient 实现：
|
||||
|
||||
```text
|
||||
A_MOPD,t =
|
||||
stop_gradient[
|
||||
log π_teacher(y_t | x, y_<t)
|
||||
- log π_student(y_t | x, y_<t)
|
||||
]
|
||||
```
|
||||
|
||||
再做双边 advantage clipping。
|
||||
|
||||
关键区别：
|
||||
|
||||
| 机制 | 谁生成轨迹 | 为什么需要稳定化 | 信号密度 |
|
||||
|---|---|---|---|
|
||||
| K3 partial rollout RL | 跨多个旧/新 policy 的未完成任务 | 轨迹 stale、极端 off-policy | 通常 outcome / GRM 为主 |
|
||||
| K3 MOPD | 当前 student | student 与 teacher 分布差异 | 每个 token 都有 teacher signal |
|
||||
|
||||
不得把 “partial rollout 是 off-policy” 和 “MOPD 是 on-policy” 写成矛盾；它们是不同阶段/目标。
|
||||
|
||||
### 6.3 为什么强调 same-origin teacher
|
||||
|
||||
独立 MOPD 论文验证：
|
||||
|
||||
- 每个领域教师从同一 SFT checkpoint 经过领域 RL 得到；
|
||||
- student 也从该 SFT checkpoint 初始化；
|
||||
- teacher/student 分布接近时，policy-gradient 和 top-k 形式都较稳定；
|
||||
- 换成绝对能力更强但分布更远的外部教师，训练可能不稳定，entropy 收缩。
|
||||
|
||||
“教师越强越好”因此不成立；教师与学生的分布距离也是关键变量。
|
||||
|
||||
### 6.4 为什么 K3 没有采用更细的 top-k 教师分布
|
||||
|
||||
独立 MOPD 给出两种实现：
|
||||
|
||||
- 只使用学生实际采样 token 的 policy-gradient 形式；
|
||||
- 传输教师 top-k token 分布的低方差形式。
|
||||
|
||||
K3 报告称在其设置中，更细粒度 top-k distillation 没有改善收敛或最终性能。必须写成“在 K3 的同源教师与具体基础设施设置中”，不能推广成 top-k 蒸馏普遍无用。
|
||||
|
||||
## 7. K3 百万 Token agentic RL 的系统闭环
|
||||
|
||||
### 7.1 co-located RL
|
||||
|
||||
K3 在几百张 GPU 规模做 co-located training。策略训练与 rollout 共享设备，配套：
|
||||
|
||||
- partial rollout 减少长尾等待；
|
||||
- KV 写回外部 CPU DRAM pool；
|
||||
- KDA recurrent state 与 MLA KV cache 统一生命周期；
|
||||
- train state 在阶段间卸载到 NVMe；
|
||||
- 根据 active/queued requests 与 KV utilization 自动 throttle；
|
||||
- reference / non-policy 权重可从 CPU 流入 policy FP32 gradient buffer。
|
||||
|
||||
这些机制回答的是“如何让百万 Token agent trajectory 实际跑得动”，不是新推理算法本身。
|
||||
|
||||
### 7.2 AgentENV
|
||||
|
||||
K3 的沙箱系统使用 Firecracker microVM，支持：
|
||||
|
||||
- pause / resume；
|
||||
- incremental checkpoint；
|
||||
- fork / snapshot；
|
||||
- 暂停时不消耗 memory/CPU；
|
||||
- checkpoint 最低 133 ms、resume 最低 49 ms；
|
||||
- 真实负载最高 6.5× memory overcommit。
|
||||
|
||||
K3 报告称训练与评测共创建 51,219,741 个 sandboxes、跨 1,505,678 个 images。它们是该项目报告的规模数据，不是通用性能保证。
|
||||
|
||||
AgentENV 已开源：
|
||||
|
||||
- https://github.com/kvcache-ai/AgentENV
|
||||
|
||||
## 8. 过程监督与验证器主线
|
||||
|
||||
### 8.1 Uesato 2022：outcome/process 不是简单二选一
|
||||
|
||||
在 GSM8K 设置中，该工作比较 outcome-based 与 process-based feedback，并指出若追求低 trace error，通常需要过程反馈，或一个能近似过程反馈的 reward model。
|
||||
|
||||
不能外推为“所有任务 PRM 都优于 ORM”；生成模型、标注预算和搜索规模都不同。
|
||||
|
||||
### 8.2 PRM800K / Let’s Verify Step by Step
|
||||
|
||||
关键事实：
|
||||
|
||||
- 约 800K step-level labels；
|
||||
- 来自约 75K solutions、12K problems；
|
||||
- 在代表性 500 道 MATH 子集上，process-supervised reward model 配合 best-of-1860 选择达到 78.2%；
|
||||
- 生成器与 reward model 来自内部 GPT-4-base 系列；
|
||||
- 4,500 道 MATH test problems 用于训练，只评估剩余 500；
|
||||
- 主要研究 reward-model reliability 与 best-of-N，不是 generator RL。
|
||||
|
||||
网站不能写“PRM 解出 78.2% MATH”，必须保留：
|
||||
|
||||
```text
|
||||
500-problem subset + best-of-1860 + reward-model selection
|
||||
```
|
||||
|
||||
### 8.3 verifier 与搜索的反馈回路
|
||||
|
||||
搜索不是自动纠错器：
|
||||
|
||||
```text
|
||||
生成器提出候选
|
||||
→ verifier 给局部排序
|
||||
→ 搜索把高分分支分配更多预算
|
||||
→ verifier 的系统性偏差被反复放大
|
||||
```
|
||||
|
||||
因此互动实验要允许单独调：
|
||||
|
||||
- 基础候选正确率；
|
||||
- 候选相关性/多样性；
|
||||
- verifier true-positive / false-positive；
|
||||
- branching factor；
|
||||
- depth；
|
||||
- 总 token budget。
|
||||
|
||||
## 9. s1：极少数据、预算强制与蒸馏边界
|
||||
|
||||
s1 的关键设置：
|
||||
|
||||
- 从候选池筛出 1,000 个问题组成 s1K；
|
||||
- 共约 4.7M tokens；
|
||||
- traces 来自 Gemini 2.0 Flash Thinking，后续 s1.1 改用 R1；
|
||||
- SFT Qwen2.5-32B-Instruct；
|
||||
- 训练约 7 个 H100 GPU-hours；
|
||||
- 用 “Wait” 继续生成或强制结束，实现 budget forcing。
|
||||
|
||||
论文中的 AIME 2024 无 budget forcing 为 50，特定扩展运行约到 56.7/57。教学边界：
|
||||
|
||||
- 这是强教师轨迹蒸馏，不是 1,000 个原始问题让基础模型无中生有；
|
||||
- 59K 全量数据未超过精选 1K，说明 difficulty/diversity/quality 很重要；
|
||||
- “超过 o1-preview”是特定 benchmark、prompt、预算与评测协议下的比较。
|
||||
|
||||
## 10. 首版互动实验设计
|
||||
|
||||
### 10.1 Reasoning Budget Lab（主交互）
|
||||
|
||||
固定一笔总预算，例如 16K / 64K / 256K tokens，让读者分给四种策略：
|
||||
|
||||
1. 单轨串行深度；
|
||||
2. 并行采样数量；
|
||||
3. verifier/search；
|
||||
4. 工具调用。
|
||||
|
||||
输入：
|
||||
|
||||
- base pass@1；
|
||||
- 独立样本相关性；
|
||||
- 深度收益曲线；
|
||||
- verifier 准确率；
|
||||
- search branching/depth；
|
||||
- tool success / latency；
|
||||
- reasoning effort；
|
||||
- 总 token/latency budget。
|
||||
|
||||
输出：
|
||||
|
||||
- 估计 pass@1；
|
||||
- pass@k / coverage；
|
||||
- majority@k；
|
||||
- verifier-selected success；
|
||||
- 串行关键路径 latency；
|
||||
- 总 token；
|
||||
- “能力扩展”与“选择改进”的分解。
|
||||
|
||||
所有数值必须标注为 deterministic teaching simulation，不得伪装成模型 benchmark。
|
||||
|
||||
预设：
|
||||
|
||||
- Direct / CoT；
|
||||
- Self-Consistency；
|
||||
- Best-of-N + ORM；
|
||||
- PRM search；
|
||||
- DeepSeek-R1 风格；
|
||||
- Kimi k1.5 long-CoT；
|
||||
- K3 low / high / max；
|
||||
- tool-using agent。
|
||||
|
||||
### 10.2 GRPO Bias Lab（第二交互或主交互第二页签）
|
||||
|
||||
给定同题 8 条 rollout：
|
||||
|
||||
- reward；
|
||||
- token length；
|
||||
- old/new token probability ratio；
|
||||
- 是否 overlong；
|
||||
- 是否全对/全错组。
|
||||
|
||||
并排显示：
|
||||
|
||||
- DeepSeekMath GRPO；
|
||||
- DAPO token-level aggregation；
|
||||
- Dr.GRPO；
|
||||
- K2.5 log-ratio gradient mask。
|
||||
|
||||
读者可以直接看到：
|
||||
|
||||
- 组内全同奖励为何梯度为零；
|
||||
- response-length normalization 如何改变长短样本权重；
|
||||
- std normalization 如何改变题目难度权重；
|
||||
- ratio 越界时 K2.5 如何屏蔽 token；
|
||||
- hard overlong penalty 与 soft shaping 的差别。
|
||||
|
||||
### 10.3 MOPD Capability Mixer
|
||||
|
||||
画 3×3 教师矩阵：
|
||||
|
||||
```text
|
||||
low high max
|
||||
general ● ● ●
|
||||
agent ● ● ●
|
||||
coding ● ● ●
|
||||
```
|
||||
|
||||
选择 prompt domain / effort 后：
|
||||
|
||||
- student rollout；
|
||||
- router 选择 teacher；
|
||||
- 每 token 显示 student prob、teacher prob、clipped advantage；
|
||||
- 可切换 same-origin / distant teacher；
|
||||
- 显示 entropy、teacher-student KL 和稳定性提示。
|
||||
|
||||
## 11. 正文可视化清单
|
||||
|
||||
首版至少需要：
|
||||
|
||||
1. 八账总览：结果、覆盖、选择、过程、预算、优化、分布、系统。
|
||||
2. 四种 test-time compute：串行、并行、搜索、工具。
|
||||
3. pass@1 / pass@k / majority@k / best-of-N 的同一候选池示意。
|
||||
4. CoT → self-consistency → verifier → search → agent 时间线。
|
||||
5. PPO → GRPO → DAPO / Dr.GRPO 公式差异图。
|
||||
6. DeepSeekMath → R1-Zero → R1 → distill 因果链。
|
||||
7. “能力扩展 vs 概率重排”的双分布图。
|
||||
8. PRM 的逐步评分与 best-of-N 选择图。
|
||||
9. Kimi k1.5 partial rollout 的跨迭代时间线。
|
||||
10. k1.5 long2short 四方法对照。
|
||||
11. K2 verifiable gym → critic → subjective rubric 闭环。
|
||||
12. K2.5 Toggle 两阶段预算训练。
|
||||
13. K3 3 domains × 3 efforts → MOPD 的九教师矩阵。
|
||||
14. K3 partial rollout RL 与 MOPD 并排图。
|
||||
15. K3 co-located RL、外部 KV pool、AgentENV 流水线。
|
||||
16. CoT correctness / plausibility / faithfulness 三圆图。
|
||||
|
||||
## 12. 本地一手材料
|
||||
|
||||
PDF 与文本只作研究缓存，受 `.gitignore` 排除；公开仓库提交本账本与 canonical URL。
|
||||
|
||||
| ID | 来源 | 页数 | 本章用途 |
|
||||
|---|---|---:|---|
|
||||
| `1707.06347` | Proximal Policy Optimization Algorithms | 12 | PPO 起点 |
|
||||
| `2110.14168` | Training Verifiers to Solve Math Word Problems | 22 | GSM8K、verifier、采样 |
|
||||
| `2201.11903` | Chain-of-Thought Prompting | 43 | 串行推理起点 |
|
||||
| `2203.11171` | Self-Consistency Improves Chain of Thought Reasoning | 24 | 并行采样与投票 |
|
||||
| `2205.10625` | Least-to-Most Prompting | 61 | 分解式推理 |
|
||||
| `2210.03629` | ReAct | 33 | 推理与行动交错 |
|
||||
| `2211.14275` | Solving Math Word Problems With Process- and Outcome-Based Feedback | 29 | PRM/ORM 早期比较 |
|
||||
| `2305.04388` | Language Models Don’t Always Say What They Think | 32 | CoT faithfulness |
|
||||
| `2305.10601` | Tree of Thoughts | 14 | 显式搜索 |
|
||||
| `2305.20050` | Let’s Verify Step by Step | 29 | PRM800K / best-of-1860 |
|
||||
| `2402.03300` | DeepSeekMath | 30 | GRPO |
|
||||
| `2408.03314` | Scaling LLM Test-Time Compute Optimally | 37 | compute-optimal allocation |
|
||||
| `2411.15124` | Tülu 3 | 82 | RLVR / 开放 post-training 配方 |
|
||||
| `2412.16720` | OpenAI o1 System Card | 51 | reasoning model 评测/安全边界 |
|
||||
| `2501.12599` | Kimi k1.5 | 25 | long-CoT、partial rollout、long2short |
|
||||
| `2501.12948` | DeepSeek-R1 | 86 | R1-Zero、R1、distillation |
|
||||
| `2501.19393` | s1 | 46 | budget forcing、精选蒸馏数据 |
|
||||
| `2503.14476` | DAPO | 16 | GRPO 复现与四项修正 |
|
||||
| `2503.20783` | Understanding R1-Zero-Like Training: A Critical Perspective | 20 | Dr.GRPO、长度/难度偏置 |
|
||||
| `2504.13837` | Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? | 31 | RLVR coverage 边界 |
|
||||
| `2507.20534` | Kimi K2 | 32 | verifiable + self-critique RL |
|
||||
| `2602.02276` | Kimi K2.5 | 30 | token clipping、Toggle、agentic RL |
|
||||
| `2606.30406` | MOPD | 15 | multi-teacher on-policy distillation |
|
||||
| `2607.24653` | Kimi K3 | 47 | reasoning effort、MOPD、agentic infra |
|
||||
|
||||
首版时间线另补 6 个桥接节点。其 canonical 页面、标题与摘要已核对；后续二轮再加入本地逐页笔记：
|
||||
|
||||
| ID | 来源 | 首版用途 |
|
||||
|---|---|---|
|
||||
| `2203.14465` | STaR | 迭代生成、筛选与训练成功 rationale |
|
||||
| `2205.11916` | Large Language Models are Zero-Shot Reasoners | 从 few-shot CoT 到统一零样本触发 |
|
||||
| `2206.14858` | Minerva | 技术内容继续训练与定量推理 |
|
||||
| `2211.12588` | Program of Thoughts | 把语言推理与外部数值执行分开 |
|
||||
| `2303.17651` | Self-Refine | 无额外训练的反馈—修订串行扩展 |
|
||||
| `2403.09629` | Quiet-STaR | 从任务 CoT 走向一般文本内部 rationale |
|
||||
|
||||
## 13. 一手来源 canonical URLs
|
||||
|
||||
- PPO — https://arxiv.org/abs/1707.06347
|
||||
- GSM8K / Verifiers — https://arxiv.org/abs/2110.14168
|
||||
- Chain-of-Thought — https://arxiv.org/abs/2201.11903
|
||||
- Self-Consistency — https://arxiv.org/abs/2203.11171
|
||||
- STaR — https://arxiv.org/abs/2203.14465
|
||||
- Least-to-Most — https://arxiv.org/abs/2205.10625
|
||||
- Zero-Shot Reasoners — https://arxiv.org/abs/2205.11916
|
||||
- Minerva — https://arxiv.org/abs/2206.14858
|
||||
- ReAct — https://arxiv.org/abs/2210.03629
|
||||
- Process vs Outcome Feedback — https://arxiv.org/abs/2211.14275
|
||||
- Program of Thoughts — https://arxiv.org/abs/2211.12588
|
||||
- Self-Refine — https://arxiv.org/abs/2303.17651
|
||||
- CoT Unfaithfulness — https://arxiv.org/abs/2305.04388
|
||||
- Tree of Thoughts — https://arxiv.org/abs/2305.10601
|
||||
- Let’s Verify Step by Step — https://arxiv.org/abs/2305.20050
|
||||
- DeepSeekMath — https://arxiv.org/abs/2402.03300
|
||||
- Quiet-STaR — https://arxiv.org/abs/2403.09629
|
||||
- Scaling LLM Test-Time Compute Optimally — https://arxiv.org/abs/2408.03314
|
||||
- Tülu 3 — https://arxiv.org/abs/2411.15124
|
||||
- OpenAI o1 System Card — https://arxiv.org/abs/2412.16720
|
||||
- Kimi k1.5 — https://arxiv.org/abs/2501.12599
|
||||
- DeepSeek-R1 — https://arxiv.org/abs/2501.12948
|
||||
- s1 — https://arxiv.org/abs/2501.19393
|
||||
- DAPO — https://arxiv.org/abs/2503.14476
|
||||
- Dr.GRPO — https://arxiv.org/abs/2503.20783
|
||||
- RLVR Capacity Limits — https://arxiv.org/abs/2504.13837
|
||||
- Kimi K2 — https://arxiv.org/abs/2507.20534
|
||||
- Kimi K2.5 — https://arxiv.org/abs/2602.02276
|
||||
- MOPD — https://arxiv.org/abs/2606.30406
|
||||
- Kimi K3 — https://arxiv.org/abs/2607.24653
|
||||
- Thinking Machines: On-Policy Distillation — https://thinkingmachines.ai/blog/on-policy-distillation/
|
||||
- AgentENV — https://github.com/kvcache-ai/AgentENV
|
||||
|
||||
## 14. 高风险表述检查表
|
||||
|
||||
- [x] 不把 pass@1、pass@k、majority@k、best-of-N 混写。
|
||||
- [x] 不把训练计算、串行推理、并行采样、搜索与工具成本混写。
|
||||
- [x] 不把可读 CoT 当作忠实因果解释。
|
||||
- [x] 不把 PRM800K 的 78.2% 写成单次 MATH 准确率。
|
||||
- [x] 不把 R1 完整管线写成“纯 RL、零 SFT”。
|
||||
- [x] 不把 R1-Zero 的 “aha” 当作 RL 从零创造能力的单一证据。
|
||||
- [x] 不把 R1 distilled 小模型写成只用 1K/800K 原始题自主学会推理。
|
||||
- [x] 不把 DAPO 的同 base 对比写成完全同数据/同系统的严格赛跑。
|
||||
- [x] 不把 Dr.GRPO 写成已经推翻所有 GRPO。
|
||||
- [x] 不把当前 RLVR pass@k 结果写成关于 RL 的不可能定理。
|
||||
- [x] 不把 s1 的 1K 写成没有强教师轨迹。
|
||||
- [x] 不把 k1.5 不使用 PRM/MCTS 写成这些方法普遍无效。
|
||||
- [x] 不把 K2 self-critique reward 当作客观 verifier。
|
||||
- [x] 不把 K2.5 probability-ratio clipping / log-ratio drift 约束当作标准 PPO clipping。
|
||||
- [x] 不把 K3 partial rollout 的 off-policy 稳定化与 MOPD 的 on-policy student rollout 混写。
|
||||
- [x] 不把更强但远分布教师写成 MOPD 必然更好。
|
||||
- [x] 不把 K3 报告中的系统规模数字写成第三方复现结果。
|
||||
- [x] 所有 benchmark 数字同时带模型、采样、选择与数据边界。
|
||||
|
||||
## 15. 首版正文完成闸门
|
||||
|
||||
- [x] 把八张账做成一张可导航总图。
|
||||
- [x] 建立不少于 30 篇的关键论文时间线，并逐项核对 canonical URL。
|
||||
- [x] 完整解释 CoT、self-consistency、verifier、PRM、search、tool use 的桥接关系。
|
||||
- [x] 用统一候选池讲清 pass@1 / pass@k / consensus / best-of-N。
|
||||
- [x] 画出 PPO → GRPO → DAPO / Dr.GRPO 的目标函数差异。
|
||||
- [x] 单独写 DeepSeekMath → R1-Zero → R1 → distillation 高亮主线。
|
||||
- [x] 单独写 Kimi k1.5 → K2 → K2.5 → K3 高亮主线。
|
||||
- [x] 并排画 K3 partial rollout RL 与 MOPD，明确 off-policy / on-policy。
|
||||
- [x] 讲清 reasoning effort、budget forcing、long2short 与 Toggle 的差异。
|
||||
- [x] 讲清 RLVR pass@1 / pass@k 能力边界争论及其局限。
|
||||
- [x] 加入 CoT faithfulness 的独立警示与实验。
|
||||
- [x] 完成 Reasoning Budget Lab。
|
||||
- [x] 完成 GRPO Bias Lab 或等价交互页签。
|
||||
- [x] 完成 MOPD 九教师可视化。
|
||||
- [x] 论文链全部指向一手来源。
|
||||
- [ ] 类型、链接、anchor、交互、桌面/移动端、容器与生产 Chrome 检查通过。
|
||||
@@ -177,7 +177,7 @@ if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentO
|
||||
}
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||
if (!mobile.menuVisible) failures.push("移动端菜单按钮未显示");
|
||||
if (home.releaseCards !== 2) failures.push(`首页新章卡数量异常：${home.releaseCards}`);
|
||||
if (home.releaseCards !== 3) failures.push(`首页新章卡数量异常：${home.releaseCards}`);
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -0,0 +1,299 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
const cdpPort = process.env.CDP_PORT ?? "9224";
|
||||
const baseUrl = process.env.SITE_URL ?? "http://127.0.0.1:4323";
|
||||
const pages = await fetch(`http://127.0.0.1:${cdpPort}/json/list`).then((response) => response.json());
|
||||
const page = pages.find((entry) => entry.type === "page");
|
||||
if (!page) throw new Error(`CDP ${cdpPort} 没有可用页面`);
|
||||
|
||||
const socket = new WebSocket(page.webSocketDebuggerUrl);
|
||||
await new Promise((resolve, reject) => {
|
||||
socket.addEventListener("open", resolve, { once: true });
|
||||
socket.addEventListener("error", reject, { once: true });
|
||||
});
|
||||
|
||||
let nextId = 0;
|
||||
const pending = new Map();
|
||||
const exceptions = [];
|
||||
socket.addEventListener("message", (event) => {
|
||||
const message = JSON.parse(event.data);
|
||||
if (message.id && pending.has(message.id)) {
|
||||
const { resolve, reject } = pending.get(message.id);
|
||||
pending.delete(message.id);
|
||||
if (message.error) reject(new Error(message.error.message));
|
||||
else resolve(message.result);
|
||||
}
|
||||
if (message.method === "Runtime.exceptionThrown") {
|
||||
exceptions.push(message.params.exceptionDetails.text);
|
||||
}
|
||||
});
|
||||
|
||||
const command = (method, params = {}) => new Promise((resolve, reject) => {
|
||||
const id = ++nextId;
|
||||
pending.set(id, { resolve, reject });
|
||||
socket.send(JSON.stringify({ id, method, params }));
|
||||
});
|
||||
const pause = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||
const evaluate = async (expression) => {
|
||||
const result = await command("Runtime.evaluate", {
|
||||
expression,
|
||||
returnByValue: true,
|
||||
awaitPromise: true,
|
||||
});
|
||||
if (result.exceptionDetails) throw new Error(result.exceptionDetails.text);
|
||||
return result.result.value;
|
||||
};
|
||||
const navigate = async (path) => {
|
||||
await command("Page.navigate", { url: `${baseUrl}${path}` });
|
||||
for (let attempt = 0; attempt < 40; attempt += 1) {
|
||||
await pause(100);
|
||||
if (await evaluate("document.readyState === 'complete'")) return;
|
||||
}
|
||||
throw new Error(`${path} 加载超时`);
|
||||
};
|
||||
const screenshot = async (path, full = false) => {
|
||||
const params = { format: "png", captureBeyondViewport: full };
|
||||
if (full) {
|
||||
const metrics = await command("Page.getLayoutMetrics");
|
||||
params.clip = {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: metrics.cssContentSize.width,
|
||||
height: metrics.cssContentSize.height,
|
||||
scale: 1,
|
||||
};
|
||||
}
|
||||
const result = await command("Page.captureScreenshot", params);
|
||||
writeFileSync(path, Buffer.from(result.data, "base64"));
|
||||
};
|
||||
|
||||
await command("Page.enable");
|
||||
await command("Runtime.enable");
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 1440,
|
||||
height: 1100,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: false,
|
||||
});
|
||||
await navigate("/reasoning/");
|
||||
await screenshot("/tmp/llm-atlas-reasoning-desktop.png");
|
||||
|
||||
const budget = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-reasoning-lab]");
|
||||
const correlation = root.querySelector("[data-correlation]");
|
||||
correlation.value = "0";
|
||||
correlation.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const independent = {
|
||||
coverage: root.querySelector("[data-metric-coverage]").textContent,
|
||||
effective: root.querySelector("[data-effective]").textContent,
|
||||
};
|
||||
correlation.value = "95";
|
||||
correlation.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const correlated = {
|
||||
coverage: root.querySelector("[data-metric-coverage]").textContent,
|
||||
effective: root.querySelector("[data-effective]").textContent,
|
||||
};
|
||||
root.querySelector('[data-budget-preset="agent"]').click();
|
||||
return {
|
||||
independent,
|
||||
correlated,
|
||||
preset: root.querySelector("[data-budget-name]").textContent,
|
||||
buys: root.querySelector("[data-budget-buys]").textContent,
|
||||
metrics: root.querySelectorAll(".budget-metrics > article").length,
|
||||
};
|
||||
})()`);
|
||||
|
||||
const grpo = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-reasoning-lab]");
|
||||
root.querySelector('[data-lab-tab="grpo"]').click();
|
||||
root.querySelector('[data-rollout-profile="all-right"]').click();
|
||||
const allRight = {
|
||||
signal: root.querySelector("[data-group-signal]").textContent,
|
||||
note: root.querySelector("[data-group-note]").textContent,
|
||||
};
|
||||
root.querySelector('[data-rollout-profile="rare"]').click();
|
||||
const ratio = root.querySelector("[data-ratio-window]");
|
||||
ratio.value = "10";
|
||||
ratio.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
return {
|
||||
allRight,
|
||||
rareSignal: root.querySelector("[data-group-signal]").textContent,
|
||||
masks: root.querySelector("[data-mask-count]").textContent,
|
||||
rows: root.querySelectorAll("[data-gradient-row]").length,
|
||||
};
|
||||
})()`);
|
||||
|
||||
const mopd = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-reasoning-lab]");
|
||||
root.querySelector('[data-lab-tab="mopd"]').click();
|
||||
root.querySelector('[data-teacher="coding-max"]').click();
|
||||
const distance = root.querySelector("[data-distance]");
|
||||
const clip = root.querySelector("[data-clip]");
|
||||
distance.value = "90";
|
||||
clip.value = "10";
|
||||
distance.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
clip.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
return {
|
||||
teacher: root.querySelector("[data-teacher-name]").textContent,
|
||||
warning: root.querySelector("[data-mopd-warning]").textContent,
|
||||
clipped: root.querySelector("[data-mopd-clipped]").textContent,
|
||||
density: [...root.querySelectorAll(".mopd-metrics > article b")].at(-1).textContent,
|
||||
tokens: root.querySelectorAll("[data-mopd-token]").length,
|
||||
};
|
||||
})()`);
|
||||
|
||||
const layout = await evaluate(`(() => {
|
||||
const nav = document.querySelector(".top-nav");
|
||||
const meta = document.querySelector(".header-meta");
|
||||
const viewport = document.documentElement.clientWidth;
|
||||
const overflowers = [...document.querySelectorAll("*")]
|
||||
.map((node) => {
|
||||
const rect = node.getBoundingClientRect();
|
||||
return {
|
||||
tag: node.tagName,
|
||||
className: typeof node.className === "string" ? node.className : "",
|
||||
parentClass: typeof node.parentElement?.className === "string" ? node.parentElement.className : "",
|
||||
text: (node.textContent ?? "").trim().replace(/\\s+/g, " ").slice(0, 70),
|
||||
left: Number(rect.left.toFixed(1)),
|
||||
right: Number(rect.right.toFixed(1)),
|
||||
width: Number(rect.width.toFixed(1)),
|
||||
scrollWidth: node.scrollWidth,
|
||||
};
|
||||
})
|
||||
.filter((item) => item.right > viewport + 1 || item.left < -1)
|
||||
.sort((a, b) => Math.max(b.right - viewport, -b.left) - Math.max(a.right - viewport, -a.left))
|
||||
.slice(0, 12);
|
||||
const scrollNodes = [...document.querySelectorAll("*")]
|
||||
.map((node) => ({
|
||||
tag: node.tagName,
|
||||
className: typeof node.className === "string" ? node.className : "",
|
||||
parentClass: typeof node.parentElement?.className === "string" ? node.parentElement.className : "",
|
||||
delta: node.scrollWidth - node.clientWidth,
|
||||
clientWidth: node.clientWidth,
|
||||
scrollWidth: node.scrollWidth,
|
||||
overflowX: getComputedStyle(node).overflowX,
|
||||
}))
|
||||
.filter((item) => item.delta > 1)
|
||||
.sort((a, b) => b.delta - a.delta)
|
||||
.slice(0, 12);
|
||||
return {
|
||||
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||
navGap: Number((meta.getBoundingClientRect().left - nav.getBoundingClientRect().right).toFixed(1)),
|
||||
navLinks: document.querySelectorAll(".top-nav a").length,
|
||||
articleSections: document.querySelectorAll(".article-section").length,
|
||||
paperLinks: document.querySelectorAll(".paper-chain a").length,
|
||||
overflowers,
|
||||
};
|
||||
})()`);
|
||||
|
||||
await evaluate(`(() => {
|
||||
document.documentElement.style.scrollBehavior = "auto";
|
||||
document.querySelector("[data-reasoning-lab]").scrollIntoView({ block: "start", behavior: "instant" });
|
||||
})()`);
|
||||
await pause(150);
|
||||
await screenshot("/tmp/llm-atlas-reasoning-lab-desktop.png");
|
||||
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 390,
|
||||
height: 844,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: true,
|
||||
});
|
||||
await navigate("/reasoning/");
|
||||
await screenshot("/tmp/llm-atlas-reasoning-mobile-closed.png");
|
||||
const mobile = await evaluate(`(() => {
|
||||
const toggle = document.querySelector("#menu-toggle");
|
||||
toggle.click();
|
||||
const viewport = document.documentElement.clientWidth;
|
||||
const overflowers = [...document.querySelectorAll("*")]
|
||||
.map((node) => {
|
||||
const rect = node.getBoundingClientRect();
|
||||
return {
|
||||
tag: node.tagName,
|
||||
className: typeof node.className === "string" ? node.className : "",
|
||||
parentClass: typeof node.parentElement?.className === "string" ? node.parentElement.className : "",
|
||||
text: (node.textContent ?? "").trim().replace(/\\s+/g, " ").slice(0, 70),
|
||||
left: Number(rect.left.toFixed(1)),
|
||||
right: Number(rect.right.toFixed(1)),
|
||||
width: Number(rect.width.toFixed(1)),
|
||||
scrollWidth: node.scrollWidth,
|
||||
};
|
||||
})
|
||||
.filter((item) => item.right > viewport + 1 || item.left < -1)
|
||||
.sort((a, b) => Math.max(b.right - viewport, -b.left) - Math.max(a.right - viewport, -a.left))
|
||||
.slice(0, 12);
|
||||
const scrollNodes = [...document.querySelectorAll("*")]
|
||||
.map((node) => ({
|
||||
tag: node.tagName,
|
||||
className: typeof node.className === "string" ? node.className : "",
|
||||
parentClass: typeof node.parentElement?.className === "string" ? node.parentElement.className : "",
|
||||
delta: node.scrollWidth - node.clientWidth,
|
||||
clientWidth: node.clientWidth,
|
||||
scrollWidth: node.scrollWidth,
|
||||
overflowX: getComputedStyle(node).overflowX,
|
||||
}))
|
||||
.filter((item) => item.delta > 1)
|
||||
.sort((a, b) => b.delta - a.delta)
|
||||
.slice(0, 12);
|
||||
return {
|
||||
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||
menuVisible: getComputedStyle(toggle).display !== "none",
|
||||
menuOpen: toggle.getAttribute("aria-expanded"),
|
||||
title: document.querySelector("h1").innerText,
|
||||
overflowers,
|
||||
scrollNodes,
|
||||
};
|
||||
})()`);
|
||||
await screenshot("/tmp/llm-atlas-reasoning-mobile.png");
|
||||
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 1440,
|
||||
height: 1100,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: false,
|
||||
});
|
||||
await navigate("/");
|
||||
await evaluate("scrollTo(0, 0)");
|
||||
const home = await evaluate(`({
|
||||
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||
releaseCards: document.querySelectorAll(".release-card").length,
|
||||
firstRelease: document.querySelector(".release-card h2").textContent,
|
||||
navLinks: document.querySelectorAll(".top-nav a").length,
|
||||
})`);
|
||||
await evaluate(`document.querySelector("#new-chapters").scrollIntoView({ block: "start", behavior: "instant" })`);
|
||||
await pause(100);
|
||||
await screenshot("/tmp/llm-atlas-home-reasoning-release.png");
|
||||
|
||||
const report = { budget, grpo, mopd, layout, mobile, home, exceptions };
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
|
||||
const numeric = (value) => Number.parseFloat(value);
|
||||
const failures = [];
|
||||
if (numeric(budget.independent.coverage) <= numeric(budget.correlated.coverage)) {
|
||||
failures.push("候选相关性没有降低 coverage");
|
||||
}
|
||||
if (numeric(budget.independent.effective) <= numeric(budget.correlated.effective)) {
|
||||
failures.push("候选相关性没有降低有效独立样本");
|
||||
}
|
||||
if (!budget.preset.includes("TOOL AGENT") || !budget.buys.includes("新观察")) failures.push("Tool Agent 预设未生效");
|
||||
if (budget.metrics !== 4) failures.push(`预算指标数量异常：${budget.metrics}`);
|
||||
if (!grpo.allRight.signal.includes("零") || !grpo.allRight.note.includes("Dynamic Sampling")) {
|
||||
failures.push("全对组零优势解释缺失");
|
||||
}
|
||||
if (grpo.rows !== 8 || numeric(grpo.masks) < 1) failures.push("GRPO 行数或 K2.5 mask 异常");
|
||||
if (!mopd.teacher.includes("CODING · MAX") || !mopd.warning.includes("高风险")) failures.push("MOPD 远教师预设未生效");
|
||||
if (mopd.tokens !== 12 || mopd.density.trim() !== "12 / 12") failures.push("MOPD 稠密 token 信号异常");
|
||||
if (layout.articleSections !== 15 || layout.paperLinks !== 30) failures.push("章节或论文链数量异常");
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) {
|
||||
failures.push("页面存在横向溢出");
|
||||
}
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 3 || !home.firstRelease.includes("多想一会儿")) failures.push("首页推理新章入口异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
if (failures.length) {
|
||||
failures.forEach((failure) => console.error(`- ${failure}`));
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -0,0 +1,974 @@
|
||||
---
|
||||
const budgetPresets = [
|
||||
{ id: "direct", label: "Direct", serial: 0.15, parallel: 1, verifier: 0, search: 0, tool: 0 },
|
||||
{ id: "cot", label: "Long CoT", serial: 0.82, parallel: 1, verifier: 0, search: 0, tool: 0 },
|
||||
{ id: "sc", label: "Self-Consistency", serial: 0.22, parallel: 12, verifier: 0.45, search: 0, tool: 0 },
|
||||
{ id: "bon", label: "Best-of-N", serial: 0.18, parallel: 16, verifier: 0.82, search: 0, tool: 0 },
|
||||
{ id: "search", label: "PRM Search", serial: 0.28, parallel: 8, verifier: 0.9, search: 0.62, tool: 0 },
|
||||
{ id: "agent", label: "Tool Agent", serial: 0.38, parallel: 4, verifier: 0.76, search: 0.2, tool: 0.68 },
|
||||
{ id: "k3max", label: "K3 · MAX", serial: 0.7, parallel: 6, verifier: 0.84, search: 0.35, tool: 0.52 },
|
||||
];
|
||||
|
||||
const rolloutProfiles = [
|
||||
{
|
||||
id: "mixed",
|
||||
label: "5 对 / 3 错",
|
||||
rewards: [1, 0, 1, 1, 0, 1, 0, 1],
|
||||
lengths: [640, 2860, 980, 1740, 4280, 1220, 3560, 760],
|
||||
ratios: [1.02, 0.89, 1.11, 1.04, 1.36, 0.96, 0.72, 1.08],
|
||||
},
|
||||
{
|
||||
id: "all-right",
|
||||
label: "全对组",
|
||||
rewards: [1, 1, 1, 1, 1, 1, 1, 1],
|
||||
lengths: [620, 880, 1340, 1760, 2110, 2640, 3180, 4020],
|
||||
ratios: [1.02, 1.06, 0.96, 1.12, 0.91, 1.2, 0.84, 1.28],
|
||||
},
|
||||
{
|
||||
id: "rare",
|
||||
label: "1 对 / 7 错",
|
||||
rewards: [0, 0, 0, 1, 0, 0, 0, 0],
|
||||
lengths: [760, 1260, 2060, 3480, 4420, 2860, 5180, 1640],
|
||||
ratios: [0.94, 1.08, 0.82, 1.14, 1.42, 0.7, 1.52, 0.88],
|
||||
},
|
||||
];
|
||||
|
||||
const mopdDomains = [
|
||||
["general-low", "GENERAL", "LOW"],
|
||||
["general-high", "GENERAL", "HIGH"],
|
||||
["general-max", "GENERAL", "MAX"],
|
||||
["agent-low", "AGENT", "LOW"],
|
||||
["agent-high", "AGENT", "HIGH"],
|
||||
["agent-max", "AGENT", "MAX"],
|
||||
["coding-low", "CODING", "LOW"],
|
||||
["coding-high", "CODING", "HIGH"],
|
||||
["coding-max", "CODING", "MAX"],
|
||||
];
|
||||
|
||||
const tokenLabels = ["先", "拆", "约束", "，", "再", "验证", "关键", "步骤", "，", "最后", "调用", "工具"];
|
||||
---
|
||||
|
||||
<section class="reasoning-lab" data-reasoning-lab>
|
||||
<div class="lab-header">
|
||||
<div>
|
||||
<p>INTERACTIVE / REASONING COMPUTE LAB</p>
|
||||
<h3>“多想一会儿”至少有三套完全不同的账</h3>
|
||||
</div>
|
||||
<p>
|
||||
下面所有概率都是确定性教学模拟，不是任何真实模型跑分。它用同一界面拆开测试时预算、GRPO 梯度和 K3 MOPD，
|
||||
让概念差异可以被操作，而不是只靠背术语。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="mode-tabs" role="tablist" aria-label="选择推理实验">
|
||||
<button type="button" role="tab" data-lab-tab="budget" aria-selected="true">
|
||||
<span>01</span><b>预算怎样分</b><small>serial / parallel / search / tools</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-lab-tab="grpo" aria-selected="false">
|
||||
<span>02</span><b>梯度怎样变</b><small>GRPO / DAPO / Dr.GRPO / K2.5</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-lab-tab="mopd" aria-selected="false">
|
||||
<span>03</span><b>九教师怎样合</b><small>K3 multi-effort MOPD</small>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="lab-view" data-lab-view="budget">
|
||||
<div class="view-intro">
|
||||
<div>
|
||||
<p class="panel-kicker">TEST-TIME COMPUTE</p>
|
||||
<h3>固定总 Token，不同策略买到的不是同一种计算</h3>
|
||||
</div>
|
||||
<p>
|
||||
串行深度增加单条轨迹可做的工作；并行宽度增加覆盖；verifier 负责选择；工具改变模型能看到的信息。
|
||||
把四者只写成 “thinking tokens” 会掩盖真正瓶颈。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="budget-presets" role="group" aria-label="预算策略预设">
|
||||
{budgetPresets.map((preset) => (
|
||||
<button
|
||||
type="button"
|
||||
data-budget-preset={preset.id}
|
||||
data-serial={preset.serial}
|
||||
data-parallel={preset.parallel}
|
||||
data-verifier={preset.verifier}
|
||||
data-search={preset.search}
|
||||
data-tool={preset.tool}
|
||||
aria-pressed={preset.id === "sc" ? "true" : "false"}
|
||||
>
|
||||
{preset.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="budget-workbench">
|
||||
<div class="budget-controls">
|
||||
<label>
|
||||
<span>总预算 <output data-budget-output>64K tokens</output></span>
|
||||
<input data-budget type="range" min="16" max="256" value="64" step="16" />
|
||||
<small>16K <i></i> 256K</small>
|
||||
</label>
|
||||
<label>
|
||||
<span>基础单次正确率 <output data-base-output>34%</output></span>
|
||||
<input data-base type="range" min="8" max="72" value="34" step="2" />
|
||||
<small>弱基础模型 <i></i> 强基础模型</small>
|
||||
</label>
|
||||
<label>
|
||||
<span>候选相关性 <output data-correlation-output>55%</output></span>
|
||||
<input data-correlation type="range" min="0" max="95" value="55" step="5" />
|
||||
<small>独立探索 <i></i> 重复同类错误</small>
|
||||
</label>
|
||||
<label>
|
||||
<span>Verifier 可靠度 <output data-verifier-output>45%</output></span>
|
||||
<input data-verifier type="range" min="0" max="98" value="45" step="2" />
|
||||
<small>近似随机 <i></i> 接近 oracle</small>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="allocation-panel">
|
||||
<div class="panel-label"><span>BUDGET ALLOCATION</span><b data-budget-name>SELF-CONSISTENCY</b></div>
|
||||
<div class="allocation-bar" aria-label="测试时计算预算分配">
|
||||
<i data-alloc="serial"><span>串行</span></i>
|
||||
<i data-alloc="parallel"><span>并行</span></i>
|
||||
<i data-alloc="search"><span>搜索</span></i>
|
||||
<i data-alloc="tool"><span>工具</span></i>
|
||||
</div>
|
||||
<div class="allocation-readout">
|
||||
<div><span>单轨长度</span><b data-serial-tokens>7.0K</b></div>
|
||||
<div><span>并行候选</span><b data-candidates>9</b></div>
|
||||
<div><span>有效独立样本</span><b data-effective>4.6</b></div>
|
||||
<div><span>关键路径</span><b data-latency>14.1s</b></div>
|
||||
</div>
|
||||
<div class="trajectory-strip" aria-label="并行推理轨迹示意">
|
||||
{Array.from({ length: 16 }, (_, index) => (
|
||||
<i data-trajectory={index}><span></span><b></b></i>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="metric-grid budget-metrics">
|
||||
<article><span>SINGLE TRAJECTORY</span><b data-metric-single>47.2%</b><p>串行思考后的单条成功率</p></article>
|
||||
<article><span>COVERAGE / PASS@K</span><b data-metric-coverage>91.3%</b><p>候选里至少出现一个正确解</p></article>
|
||||
<article><span>MAJORITY@K</span><b data-metric-majority>42.8%</b><p>不训练选择器的答案投票</p></article>
|
||||
<article><span>VERIFIER SELECTED</span><b data-metric-selected>69.5%</b><p>覆盖 × 选择可靠度的合成结果</p></article>
|
||||
</div>
|
||||
|
||||
<div class="interpretation">
|
||||
<div><span>现在主要买到</span><b data-budget-buys>覆盖，而不是更深的单条推理</b></div>
|
||||
<p data-budget-note>
|
||||
候选相关性会让 nominal N 高估真实探索宽度；继续加样本以前，先问它们是否只是重复同一种错误。
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="lab-view" data-lab-view="grpo" hidden>
|
||||
<div class="view-intro">
|
||||
<div>
|
||||
<p class="panel-kicker">POLICY OPTIMIZATION</p>
|
||||
<h3>同一组 rollout，四种聚合方法会把力用在不同地方</h3>
|
||||
</div>
|
||||
<p>
|
||||
每行是一条同题回答。条形图不是完整训练梯度，而是把 reward、长度归一化和 ratio mask
|
||||
压缩成可比较的教学权重，专门暴露“为什么实现细节会改变训练行为”。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="grpo-toolbar">
|
||||
<div role="group" aria-label="选择 rollout 奖励分布">
|
||||
{rolloutProfiles.map((profile) => (
|
||||
<button
|
||||
type="button"
|
||||
data-rollout-profile={profile.id}
|
||||
data-rewards={profile.rewards.join(",")}
|
||||
data-lengths={profile.lengths.join(",")}
|
||||
data-ratios={profile.ratios.join(",")}
|
||||
aria-pressed={profile.id === "mixed" ? "true" : "false"}
|
||||
>
|
||||
{profile.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<label>
|
||||
<span>K2.5 ratio 区间 <output data-ratio-output>0.80–1.25</output></span>
|
||||
<input data-ratio-window type="range" min="10" max="45" value="25" step="5" />
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="gradient-table">
|
||||
<div class="gradient-head">
|
||||
<span>ROLLOUT</span><span>REWARD</span><span>LENGTH</span><span>GRPO</span><span>DAPO</span><span>Dr.GRPO</span><span>K2.5 MASK</span>
|
||||
</div>
|
||||
{Array.from({ length: 8 }, (_, index) => (
|
||||
<div class="gradient-row" data-gradient-row={index}>
|
||||
<span><b>y{index + 1}</b><small data-ratio-label>ratio 1.00</small></span>
|
||||
<span data-reward>1</span>
|
||||
<span data-length>640</span>
|
||||
<span><i data-gradient="grpo"><b></b></i><em data-gradient-label="grpo">+0.00</em></span>
|
||||
<span><i data-gradient="dapo"><b></b></i><em data-gradient-label="dapo">+0.00</em></span>
|
||||
<span><i data-gradient="dr"><b></b></i><em data-gradient-label="dr">+0.00</em></span>
|
||||
<span data-mask><i></i><b>KEEP</b></span>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="grpo-summary">
|
||||
<article>
|
||||
<span>GROUP SIGNAL</span>
|
||||
<b data-group-signal>有相对奖励</b>
|
||||
<p data-group-note>同题组里既有对也有错，组均值 baseline 能产生正负 advantage。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>LENGTH EFFECT</span>
|
||||
<b data-length-effect>GRPO 偏向短正确</b>
|
||||
<p>回答内 token 平均会改变长短样本的总贡献；DAPO 与 Dr.GRPO 选择不同的聚合口径。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>OFF-POLICY TOKENS</span>
|
||||
<b data-mask-count>2 / 8 被屏蔽</b>
|
||||
<p>K2.5 只看 token ratio 是否越界；这里不模拟标准 PPO 的 advantage-aware clipping。</p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="formula-comparison">
|
||||
<article><span>GRPO</span><b>(r − μ) / σ，再按回答长度平均</b><p>去 critic，但引入组标准差和 response length 两层归一化。</p></article>
|
||||
<article><span>DAPO</span><b>全 batch token-level 聚合</b><p>配合 dynamic sampling、Clip-Higher 和 overlong shaping。</p></article>
|
||||
<article><span>Dr.GRPO</span><b>r − μ，固定全局长度分母</b><p>移除论文指出的长度与题目难度归一化偏置。</p></article>
|
||||
<article><span>K2.5</span><b>ratio ∉ [α,β] → gradient mask</b><p>用于约束训练—推理 mismatch 放大的 off-policy drift。</p></article>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="lab-view" data-lab-view="mopd" hidden>
|
||||
<div class="view-intro">
|
||||
<div>
|
||||
<p class="panel-kicker">K3 / MULTI-TEACHER ON-POLICY DISTILLATION</p>
|
||||
<h3>不是把九个模型参数平均，而是让学生在自己的路上逐 Token 问老师</h3>
|
||||
</div>
|
||||
<p>
|
||||
先选 prompt 的领域和 reasoning effort；student 生成当前轨迹，再由对应冻结 teacher 对相同前缀打分。
|
||||
这与跨迭代 partial rollout 的 off-policy RL 是两套不同机制。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="mopd-grid">
|
||||
<div class="teacher-matrix">
|
||||
<div class="matrix-head"><span>DOMAIN ↓ / EFFORT →</span><b>LOW</b><b>HIGH</b><b>MAX</b></div>
|
||||
<span class="row-name">GENERAL</span>
|
||||
<span class="row-name">AGENT</span>
|
||||
<span class="row-name">CODING</span>
|
||||
{mopdDomains.map(([id, domain, effort]) => (
|
||||
<button
|
||||
type="button"
|
||||
data-teacher={id}
|
||||
data-domain={domain}
|
||||
data-effort={effort}
|
||||
aria-pressed={id === "agent-high" ? "true" : "false"}
|
||||
>
|
||||
<i></i><b>{effort}</b><small>{domain}</small>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="mopd-controls">
|
||||
<div>
|
||||
<span>ROUTED TEACHER</span>
|
||||
<b data-teacher-name>AGENT · HIGH</b>
|
||||
<p data-teacher-story>长程助手、deep research 与段落写作教师；在 high effort 下兼顾探索与预算。</p>
|
||||
</div>
|
||||
<label>
|
||||
<span>Teacher–student 分布距离 <output data-distance-output>28%</output></span>
|
||||
<input data-distance type="range" min="5" max="90" value="28" step="1" />
|
||||
<small>同源且接近 <i></i> 更强但远分布</small>
|
||||
</label>
|
||||
<label>
|
||||
<span>Advantage clip ± <output data-clip-output>2.5</output></span>
|
||||
<input data-clip type="range" min="10" max="50" value="25" step="5" />
|
||||
<small>更强裁剪 <i></i> 更少裁剪</small>
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="token-prefill">
|
||||
<div class="panel-label"><span>STUDENT ROLLOUT → TEACHER PREFILL</span><b>DENSE TOKEN SIGNAL</b></div>
|
||||
<div class="token-flow">
|
||||
{tokenLabels.map((token, index) => (
|
||||
<div data-mopd-token={index}>
|
||||
<b>{token}</b>
|
||||
<span><i data-student-prob></i><small data-student-label>S .42</small></span>
|
||||
<span><i data-teacher-prob></i><small data-teacher-label>T .61</small></span>
|
||||
<em data-token-advantage>+0.37</em>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
<div class="prob-legend"><span><i class="student"></i>student probability</span><span><i class="teacher"></i>teacher probability</span><span>Â = clip(log T − log S)</span></div>
|
||||
</div>
|
||||
|
||||
<div class="metric-grid mopd-metrics">
|
||||
<article><span>MEAN REVERSE-KL PROXY</span><b data-mopd-kl>0.081</b><p>越远不代表越好，分布错配会增加训练压力</p></article>
|
||||
<article><span>CLIPPED TOKENS</span><b data-mopd-clipped>0 / 12</b><p>超出 ±Amax 的 token 不再继续放大 advantage</p></article>
|
||||
<article><span>ENTROPY PROXY</span><b data-mopd-entropy>0.79</b><p>远分布教师可能让 student 向少数模式收缩</p></article>
|
||||
<article><span>SIGNAL DENSITY</span><b>12 / 12</b><p>不同于只在轨迹结束时给一个 outcome reward</p></article>
|
||||
</div>
|
||||
|
||||
<div class="mopd-pipeline">
|
||||
<article><span>01</span><b>Student samples</b><p>当前学生在自己的推理分布上生成。</p></article>
|
||||
<i>→</i>
|
||||
<article><span>02</span><b>Prompt routes</b><p>按领域与 effort 选择 9 个教师之一。</p></article>
|
||||
<i>→</i>
|
||||
<article><span>03</span><b>Teacher prefills</b><p>教师读取学生前缀，返回逐 token 概率。</p></article>
|
||||
<i>→</i>
|
||||
<article><span>04</span><b>Student updates</b><p>clipped log-prob 差进入 policy-gradient。</p></article>
|
||||
</div>
|
||||
|
||||
<div class="interpretation">
|
||||
<div><span>稳定性提示</span><b data-mopd-warning>同源教师分布接近，信号强且仍可控</b></div>
|
||||
<p>
|
||||
独立 MOPD 论文发现，绝对能力更强但分布更远的外部教师不一定更好；这里用距离滑杆显示这种风险，不把它当作真实训练预测器。
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="lab-footnote">
|
||||
<b>如何读这套实验</b>
|
||||
<p>
|
||||
预算页只讲概率结构；梯度页只讲聚合方向；MOPD 页只讲 teacher signal。三页故意不合并成一个“总分”，
|
||||
因为真实 reasoning system 的能力、选择、训练稳定性与运行成本没有可诚实相加的单一单位。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<script is:inline>
|
||||
(() => {
|
||||
const root = document.querySelector("[data-reasoning-lab]");
|
||||
if (!root) return;
|
||||
|
||||
const clamp = (value, min = 0, max = 1) => Math.min(max, Math.max(min, value));
|
||||
const percent = (value) => `${(100 * clamp(value)).toFixed(1)}%`;
|
||||
const setText = (selector, value) => {
|
||||
const node = root.querySelector(selector);
|
||||
if (node) node.textContent = value;
|
||||
};
|
||||
|
||||
const tabs = [...root.querySelectorAll("[data-lab-tab]")];
|
||||
const views = [...root.querySelectorAll("[data-lab-view]")];
|
||||
tabs.forEach((tab) => tab.addEventListener("click", () => {
|
||||
const id = tab.dataset.labTab;
|
||||
tabs.forEach((item) => item.setAttribute("aria-selected", String(item === tab)));
|
||||
views.forEach((view) => { view.hidden = view.dataset.labView !== id; });
|
||||
}));
|
||||
|
||||
const budgetInput = root.querySelector("[data-budget]");
|
||||
const baseInput = root.querySelector("[data-base]");
|
||||
const correlationInput = root.querySelector("[data-correlation]");
|
||||
const verifierInput = root.querySelector("[data-verifier]");
|
||||
const budgetButtons = [...root.querySelectorAll("[data-budget-preset]")];
|
||||
let budgetPreset = budgetButtons.find((button) => button.getAttribute("aria-pressed") === "true");
|
||||
|
||||
const chooseBudgetPreset = (button) => {
|
||||
budgetPreset = button;
|
||||
budgetButtons.forEach((item) => item.setAttribute("aria-pressed", String(item === button)));
|
||||
if (verifierInput) verifierInput.value = String(Math.round(Number(button.dataset.verifier) * 100));
|
||||
updateBudget();
|
||||
};
|
||||
|
||||
const combination = (n, k) => {
|
||||
if (k < 0 || k > n) return 0;
|
||||
let result = 1;
|
||||
for (let i = 1; i <= Math.min(k, n - k); i += 1) result = result * (n - i + 1) / i;
|
||||
return result;
|
||||
};
|
||||
|
||||
const majorityProbability = (n, p) => {
|
||||
const size = Math.max(1, Math.min(31, Math.round(n)));
|
||||
const threshold = Math.floor(size / 2) + 1;
|
||||
let total = 0;
|
||||
for (let k = threshold; k <= size; k += 1) {
|
||||
total += combination(size, k) * (p ** k) * ((1 - p) ** (size - k));
|
||||
}
|
||||
return clamp(total);
|
||||
};
|
||||
|
||||
function updateBudget() {
|
||||
if (!budgetPreset || !budgetInput || !baseInput || !correlationInput || !verifierInput) return;
|
||||
const budget = Number(budgetInput.value) * 1024;
|
||||
const base = Number(baseInput.value) / 100;
|
||||
const correlation = Number(correlationInput.value) / 100;
|
||||
const verifier = Number(verifierInput.value) / 100;
|
||||
const serialShare = Number(budgetPreset.dataset.serial);
|
||||
const requestedParallel = Number(budgetPreset.dataset.parallel);
|
||||
const search = Number(budgetPreset.dataset.search);
|
||||
const tool = Number(budgetPreset.dataset.tool);
|
||||
const overheadShare = clamp(search * 0.24 + tool * 0.2, 0, 0.42);
|
||||
const available = budget * (1 - overheadShare);
|
||||
const candidates = Math.max(1, Math.min(32, Math.round(requestedParallel * (budget / 65536) ** 0.48)));
|
||||
const serialTokens = Math.max(768, available * serialShare / Math.max(1, candidates * 0.32 + 0.68));
|
||||
const serialGain = 0.25 * (1 - Math.exp(-serialTokens / 11500));
|
||||
const searchGain = search * verifier * 0.12;
|
||||
const toolGain = tool * (1 - base) * 0.2;
|
||||
const single = clamp(base + (1 - base) * serialGain + searchGain + toolGain, 0.01, 0.96);
|
||||
const effective = 1 + (candidates - 1) * ((1 - correlation) ** 0.78);
|
||||
const coverage = clamp(1 - ((1 - single) ** effective));
|
||||
const majority = majorityProbability(effective, single);
|
||||
const selectorLift = verifier * (0.42 + 0.45 * search) + tool * 0.08;
|
||||
const selected = clamp(single + (coverage - single) * selectorLift);
|
||||
const waves = Math.ceil(candidates / Math.max(1, 8 - Math.round(tool * 3)));
|
||||
const latency = serialTokens / 720 + waves * (0.32 + tool * 1.8) + search * 2.4;
|
||||
|
||||
setText("[data-budget-output]", `${Math.round(budget / 1024)}K tokens`);
|
||||
setText("[data-base-output]", `${Math.round(base * 100)}%`);
|
||||
setText("[data-correlation-output]", `${Math.round(correlation * 100)}%`);
|
||||
setText("[data-verifier-output]", `${Math.round(verifier * 100)}%`);
|
||||
setText("[data-budget-name]", budgetPreset.textContent.trim().toUpperCase());
|
||||
setText("[data-serial-tokens]", `${(serialTokens / 1024).toFixed(1)}K`);
|
||||
setText("[data-candidates]", String(candidates));
|
||||
setText("[data-effective]", effective.toFixed(1));
|
||||
setText("[data-latency]", `${latency.toFixed(1)}s`);
|
||||
setText("[data-metric-single]", percent(single));
|
||||
setText("[data-metric-coverage]", percent(coverage));
|
||||
setText("[data-metric-majority]", percent(majority));
|
||||
setText("[data-metric-selected]", percent(selected));
|
||||
|
||||
const allocations = {
|
||||
serial: serialShare,
|
||||
parallel: clamp(1 - serialShare - overheadShare, 0.05, 0.8),
|
||||
search: search * 0.24,
|
||||
tool: tool * 0.2,
|
||||
};
|
||||
const allocationTotal = Object.values(allocations).reduce((sum, value) => sum + value, 0);
|
||||
Object.entries(allocations).forEach(([key, value]) => {
|
||||
const node = root.querySelector(`[data-alloc="${key}"]`);
|
||||
if (node) node.style.width = `${100 * value / allocationTotal}%`;
|
||||
});
|
||||
|
||||
[...root.querySelectorAll("[data-trajectory]")].forEach((node, index) => {
|
||||
const active = index < candidates;
|
||||
node.hidden = !active;
|
||||
if (!active) return;
|
||||
const deterministic = ((index * 37 + Math.round(single * 100)) % 100) / 100;
|
||||
const correct = deterministic < single;
|
||||
node.dataset.correct = String(correct);
|
||||
node.style.setProperty("--length", `${44 + ((index * 29 + Math.round(serialTokens / 200)) % 54)}%`);
|
||||
});
|
||||
|
||||
let buys = "单条轨迹深度";
|
||||
let note = "预算主要沿一条串行轨迹展开；如果前提错误，继续写更久也可能只是把错误推得更深。";
|
||||
if (tool > 0.45) {
|
||||
buys = "新观察与环境反馈";
|
||||
note = "工具调用改变信息集，但环境延迟、失败恢复和结果验证不会被模型 Token 账自动覆盖。";
|
||||
} else if (search > 0.45) {
|
||||
buys = "由 verifier 引导的分支探索";
|
||||
note = "搜索会重复使用 verifier；系统性评分偏差可能随分支扩展被放大。";
|
||||
} else if (candidates >= 5) {
|
||||
buys = "候选覆盖，而不是更深的单条推理";
|
||||
note = "候选相关性会让 nominal N 高估真实探索宽度；继续加样本以前，先问它们是否只是重复同一种错误。";
|
||||
}
|
||||
setText("[data-budget-buys]", buys);
|
||||
setText("[data-budget-note]", note);
|
||||
}
|
||||
|
||||
budgetButtons.forEach((button) => button.addEventListener("click", () => chooseBudgetPreset(button)));
|
||||
[budgetInput, baseInput, correlationInput, verifierInput].forEach((input) => input?.addEventListener("input", updateBudget));
|
||||
|
||||
const profileButtons = [...root.querySelectorAll("[data-rollout-profile]")];
|
||||
const ratioInput = root.querySelector("[data-ratio-window]");
|
||||
let profileButton = profileButtons.find((button) => button.getAttribute("aria-pressed") === "true");
|
||||
|
||||
const signed = (value) => `${value >= 0 ? "+" : ""}${value.toFixed(2)}`;
|
||||
|
||||
function updateGradients() {
|
||||
if (!profileButton || !ratioInput) return;
|
||||
const rewards = profileButton.dataset.rewards.split(",").map(Number);
|
||||
const lengths = profileButton.dataset.lengths.split(",").map(Number);
|
||||
const ratios = profileButton.dataset.ratios.split(",").map(Number);
|
||||
const mean = rewards.reduce((sum, value) => sum + value, 0) / rewards.length;
|
||||
const variance = rewards.reduce((sum, value) => sum + ((value - mean) ** 2), 0) / rewards.length;
|
||||
const std = Math.sqrt(variance);
|
||||
const meanLength = lengths.reduce((sum, value) => sum + value, 0) / lengths.length;
|
||||
const width = Number(ratioInput.value) / 100;
|
||||
const low = 1 - width;
|
||||
const high = 1 + width;
|
||||
let masked = 0;
|
||||
|
||||
rewards.forEach((reward, index) => {
|
||||
const row = root.querySelector(`[data-gradient-row="${index}"]`);
|
||||
if (!row) return;
|
||||
const centered = reward - mean;
|
||||
const normalized = std > 1e-8 ? centered / std : 0;
|
||||
const grpo = normalized * meanLength / lengths[index];
|
||||
const dapo = normalized * lengths[index] / meanLength;
|
||||
const dr = centered;
|
||||
const values = { grpo, dapo, dr };
|
||||
|
||||
row.querySelector("[data-reward]").textContent = reward ? "✓ 1" : "× 0";
|
||||
row.querySelector("[data-reward]").dataset.correct = String(Boolean(reward));
|
||||
row.querySelector("[data-length]").textContent = `${(lengths[index] / 1000).toFixed(2)}K`;
|
||||
row.querySelector("[data-ratio-label]").textContent = `ratio ${ratios[index].toFixed(2)}`;
|
||||
Object.entries(values).forEach(([key, value]) => {
|
||||
const bar = row.querySelector(`[data-gradient="${key}"]`);
|
||||
const label = row.querySelector(`[data-gradient-label="${key}"]`);
|
||||
bar.dataset.sign = value >= 0 ? "positive" : "negative";
|
||||
bar.querySelector("b").style.width = `${Math.min(100, Math.abs(value) * 52)}%`;
|
||||
label.textContent = signed(value);
|
||||
});
|
||||
|
||||
const keep = ratios[index] >= low && ratios[index] <= high;
|
||||
const mask = row.querySelector("[data-mask]");
|
||||
mask.dataset.keep = String(keep);
|
||||
mask.querySelector("b").textContent = keep ? "KEEP" : "MASK";
|
||||
if (!keep) masked += 1;
|
||||
});
|
||||
|
||||
const hasSignal = std > 1e-8;
|
||||
setText("[data-ratio-output]", `${low.toFixed(2)}–${high.toFixed(2)}`);
|
||||
setText("[data-group-signal]", hasSignal ? "有相对奖励" : "零相对优势");
|
||||
setText(
|
||||
"[data-group-note]",
|
||||
hasSignal
|
||||
? "同题组里既有对也有错，组均值 baseline 能产生正负 advantage。"
|
||||
: "组内 reward 完全相同，减去组均值后全部为零；DAPO 的 Dynamic Sampling 会过滤这类 prompt。",
|
||||
);
|
||||
setText("[data-length-effect]", hasSignal ? "三种聚合给长短回答不同权重" : "没有 reward 差异，长度也救不回信号");
|
||||
setText("[data-mask-count]", `${masked} / 8 被屏蔽`);
|
||||
}
|
||||
|
||||
profileButtons.forEach((button) => button.addEventListener("click", () => {
|
||||
profileButton = button;
|
||||
profileButtons.forEach((item) => item.setAttribute("aria-pressed", String(item === button)));
|
||||
updateGradients();
|
||||
}));
|
||||
ratioInput?.addEventListener("input", updateGradients);
|
||||
|
||||
const teacherButtons = [...root.querySelectorAll("[data-teacher]")];
|
||||
const distanceInput = root.querySelector("[data-distance]");
|
||||
const clipInput = root.querySelector("[data-clip]");
|
||||
let teacherButton = teacherButtons.find((button) => button.getAttribute("aria-pressed") === "true");
|
||||
|
||||
const teacherStories = {
|
||||
GENERAL: "经验、视觉、推理、faithfulness、search 与 knowledge work 教师。",
|
||||
AGENT: "长程助手、deep research 与段落写作教师。",
|
||||
CODING: "SWE、coding experience、kernel 与 web development 教师。",
|
||||
};
|
||||
|
||||
function updateMopd() {
|
||||
if (!teacherButton || !distanceInput || !clipInput) return;
|
||||
const domain = teacherButton.dataset.domain;
|
||||
const effort = teacherButton.dataset.effort;
|
||||
const distance = Number(distanceInput.value) / 100;
|
||||
const clip = Number(clipInput.value) / 10;
|
||||
const effortFactor = { LOW: 0.82, HIGH: 1, MAX: 1.18 }[effort];
|
||||
const domainOffset = { GENERAL: 0.03, AGENT: 0.08, CODING: -0.02 }[domain];
|
||||
let clipped = 0;
|
||||
let kl = 0;
|
||||
let entropy = 0;
|
||||
|
||||
setText("[data-teacher-name]", `${domain} · ${effort}`);
|
||||
setText("[data-teacher-story]", `${teacherStories[domain]}${effort === "LOW" ? "低 effort 强调预算效率。" : effort === "MAX" ? "最大 effort 允许最长的探索预算。" : "high effort 在探索与预算之间折中。"}`);
|
||||
setText("[data-distance-output]", `${Math.round(distance * 100)}%`);
|
||||
setText("[data-clip-output]", clip.toFixed(1));
|
||||
|
||||
[...root.querySelectorAll("[data-mopd-token]")].forEach((node, index) => {
|
||||
const wave = Math.sin(index * 1.71 + domainOffset * 11) * 0.11;
|
||||
const student = clamp(0.28 + ((index * 17 + effort.length * 9) % 29) / 100 + wave, 0.08, 0.78);
|
||||
const direction = Math.sin(index * 0.93 + domainOffset * 7) > -0.15 ? 1 : -1;
|
||||
const teacher = clamp(student * Math.exp(direction * distance * effortFactor * (0.9 + (index % 4) * 0.24)), 0.02, 0.94);
|
||||
const rawAdvantage = Math.log(teacher) - Math.log(student);
|
||||
const advantage = clamp(rawAdvantage, -clip, clip);
|
||||
if (Math.abs(rawAdvantage) > clip) clipped += 1;
|
||||
kl += Math.abs(rawAdvantage) * student;
|
||||
entropy += -(student * Math.log(student) + (1 - student) * Math.log(1 - student));
|
||||
|
||||
node.querySelector("[data-student-prob]").style.width = `${student * 100}%`;
|
||||
node.querySelector("[data-teacher-prob]").style.width = `${teacher * 100}%`;
|
||||
node.querySelector("[data-student-label]").textContent = `S ${student.toFixed(2)}`;
|
||||
node.querySelector("[data-teacher-label]").textContent = `T ${teacher.toFixed(2)}`;
|
||||
const advantageNode = node.querySelector("[data-token-advantage]");
|
||||
advantageNode.textContent = signed(advantage);
|
||||
advantageNode.dataset.sign = advantage >= 0 ? "positive" : "negative";
|
||||
});
|
||||
|
||||
kl /= 12;
|
||||
entropy = entropy / 12 * (1 - Math.max(0, distance - 0.45) * 0.62);
|
||||
setText("[data-mopd-kl]", kl.toFixed(3));
|
||||
setText("[data-mopd-clipped]", `${clipped} / 12`);
|
||||
setText("[data-mopd-entropy]", entropy.toFixed(2));
|
||||
|
||||
let warning = "同源教师分布接近，信号强且仍可控";
|
||||
if (distance > 0.68) warning = "教师虽可能更强，但分布距离已进入高风险区";
|
||||
else if (distance > 0.42) warning = "分布差异开始放大 punitive gradient 与熵收缩风险";
|
||||
setText("[data-mopd-warning]", warning);
|
||||
}
|
||||
|
||||
teacherButtons.forEach((button) => button.addEventListener("click", () => {
|
||||
teacherButton = button;
|
||||
teacherButtons.forEach((item) => item.setAttribute("aria-pressed", String(item === button)));
|
||||
updateMopd();
|
||||
}));
|
||||
[distanceInput, clipInput].forEach((input) => input?.addEventListener("input", updateMopd));
|
||||
|
||||
updateBudget();
|
||||
updateGradients();
|
||||
updateMopd();
|
||||
})();
|
||||
</script>
|
||||
|
||||
<style>
|
||||
.reasoning-lab {
|
||||
color: var(--ink);
|
||||
background:
|
||||
linear-gradient(135deg, color-mix(in srgb, var(--paper) 92%, var(--blue-pale)), var(--paper) 48%),
|
||||
var(--paper);
|
||||
border: 1px solid var(--line-strong);
|
||||
box-shadow: 0 28px 80px rgb(35 45 60 / 10%);
|
||||
}
|
||||
|
||||
.lab-header,
|
||||
.view-intro {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1.1fr) minmax(320px, 0.9fr);
|
||||
gap: 48px;
|
||||
align-items: end;
|
||||
padding: 34px 38px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.lab-header > div > p,
|
||||
.panel-kicker,
|
||||
.panel-label span,
|
||||
.teacher-matrix span,
|
||||
.mopd-controls > div > span {
|
||||
color: var(--copper);
|
||||
font: 0.64rem/1.3 var(--mono);
|
||||
letter-spacing: 0.11em;
|
||||
}
|
||||
|
||||
.lab-header h3,
|
||||
.view-intro h3 {
|
||||
margin-top: 11px;
|
||||
max-width: 760px;
|
||||
font-size: clamp(1.35rem, 2.5vw, 2.25rem);
|
||||
line-height: 1.12;
|
||||
}
|
||||
|
||||
.lab-header > p,
|
||||
.view-intro > p {
|
||||
color: var(--muted);
|
||||
font-size: 0.78rem;
|
||||
line-height: 1.85;
|
||||
}
|
||||
|
||||
.mode-tabs {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
border-bottom: 1px solid var(--line-strong);
|
||||
}
|
||||
|
||||
.mode-tabs button {
|
||||
display: grid;
|
||||
grid-template-columns: 36px 1fr;
|
||||
gap: 5px 12px;
|
||||
padding: 20px 24px;
|
||||
color: var(--muted);
|
||||
text-align: left;
|
||||
background: transparent;
|
||||
border: 0;
|
||||
border-right: 1px solid var(--line);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.mode-tabs button:last-child { border-right: 0; }
|
||||
.mode-tabs button[aria-selected="true"] {
|
||||
color: var(--ink);
|
||||
background: var(--blue-pale);
|
||||
box-shadow: inset 0 -3px 0 var(--blue);
|
||||
}
|
||||
.mode-tabs span { grid-row: 1 / 3; color: var(--copper); font: 0.68rem/1 var(--mono); }
|
||||
.mode-tabs b { font-size: 0.88rem; }
|
||||
.mode-tabs small { font: 0.58rem/1.4 var(--mono); }
|
||||
|
||||
.lab-view[hidden] { display: none; }
|
||||
.view-intro { padding-block: 30px; }
|
||||
.view-intro h3 { font-size: clamp(1.25rem, 2vw, 1.75rem); }
|
||||
|
||||
.budget-presets,
|
||||
.grpo-toolbar {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px;
|
||||
padding: 22px 38px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.budget-presets button,
|
||||
.grpo-toolbar button {
|
||||
padding: 10px 13px;
|
||||
color: var(--muted);
|
||||
font: 0.64rem/1 var(--mono);
|
||||
background: var(--paper);
|
||||
border: 1px solid var(--line-strong);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.budget-presets button[aria-pressed="true"],
|
||||
.grpo-toolbar button[aria-pressed="true"] {
|
||||
color: white;
|
||||
background: var(--blue);
|
||||
border-color: var(--blue);
|
||||
}
|
||||
|
||||
.budget-workbench {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(260px, 0.72fr) minmax(0, 1.28fr);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.budget-controls {
|
||||
display: grid;
|
||||
gap: 24px;
|
||||
padding: 28px 38px;
|
||||
border-right: 1px solid var(--line);
|
||||
}
|
||||
|
||||
label > span {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
gap: 18px;
|
||||
margin-bottom: 12px;
|
||||
font-size: 0.72rem;
|
||||
}
|
||||
|
||||
label output { color: var(--copper); font: 0.65rem/1 var(--mono); }
|
||||
input[type="range"] { width: 100%; accent-color: var(--blue); }
|
||||
label > small {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
margin-top: 8px;
|
||||
color: var(--muted);
|
||||
font: 0.55rem/1 var(--mono);
|
||||
}
|
||||
label > small i { flex: 1; height: 1px; background: var(--line-strong); }
|
||||
|
||||
.allocation-panel { padding: 28px 38px; }
|
||||
.panel-label { display: flex; justify-content: space-between; gap: 24px; }
|
||||
.panel-label b { font: 0.68rem/1 var(--mono); }
|
||||
.allocation-bar { display: flex; height: 76px; margin-top: 23px; overflow: hidden; background: var(--warm-gray); }
|
||||
.allocation-bar i {
|
||||
display: flex;
|
||||
align-items: flex-end;
|
||||
min-width: 3px;
|
||||
padding: 10px;
|
||||
color: white;
|
||||
font-style: normal;
|
||||
transition: width 240ms ease;
|
||||
}
|
||||
.allocation-bar i:nth-child(1) { background: var(--navy); }
|
||||
.allocation-bar i:nth-child(2) { background: var(--blue); }
|
||||
.allocation-bar i:nth-child(3) { background: var(--copper); }
|
||||
.allocation-bar i:nth-child(4) { background: var(--sage); }
|
||||
.allocation-bar span { font: 0.56rem/1 var(--mono); }
|
||||
|
||||
.allocation-readout {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
margin-top: 20px;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
.allocation-readout div { padding: 14px; border-right: 1px solid var(--line); border-bottom: 1px solid var(--line); }
|
||||
.allocation-readout span { display: block; color: var(--muted); font-size: 0.6rem; }
|
||||
.allocation-readout b { display: block; margin-top: 8px; font: 0.78rem/1 var(--mono); }
|
||||
|
||||
.trajectory-strip { display: flex; align-items: end; gap: 5px; height: 82px; margin-top: 22px; }
|
||||
.trajectory-strip i { position: relative; flex: 1; height: var(--length); min-width: 3px; background: color-mix(in srgb, var(--blue) 18%, var(--paper)); }
|
||||
.trajectory-strip i::after { content: ""; position: absolute; inset: auto 0 0; height: 5px; background: var(--blue); }
|
||||
.trajectory-strip i[data-correct="true"]::after { background: var(--sage); }
|
||||
|
||||
.metric-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
.metric-grid article { min-height: 150px; padding: 24px; border-right: 1px solid var(--line); }
|
||||
.metric-grid article:last-child { border-right: 0; }
|
||||
.metric-grid span { color: var(--copper); font: 0.56rem/1.3 var(--mono); }
|
||||
.metric-grid b { display: block; margin: 15px 0 10px; font: 1.28rem/1 var(--mono); }
|
||||
.metric-grid p { color: var(--muted); font-size: 0.64rem; line-height: 1.6; }
|
||||
|
||||
.interpretation {
|
||||
display: grid;
|
||||
grid-template-columns: 0.75fr 1.25fr;
|
||||
gap: 36px;
|
||||
padding: 24px 38px;
|
||||
background: color-mix(in srgb, var(--blue-pale) 62%, var(--paper));
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
.interpretation span { display: block; color: var(--copper); font: 0.58rem/1 var(--mono); }
|
||||
.interpretation b { display: block; margin-top: 9px; font-size: 0.86rem; }
|
||||
.interpretation p { color: var(--muted); font-size: 0.72rem; line-height: 1.75; }
|
||||
|
||||
.grpo-toolbar { justify-content: space-between; align-items: center; }
|
||||
.grpo-toolbar > div { display: flex; gap: 8px; }
|
||||
.grpo-toolbar label { width: min(330px, 100%); }
|
||||
|
||||
.gradient-table { padding: 24px 38px 32px; overflow-x: auto; border-bottom: 1px solid var(--line); }
|
||||
.gradient-head,
|
||||
.gradient-row {
|
||||
display: grid;
|
||||
grid-template-columns: 98px 72px 74px repeat(3, minmax(105px, 1fr)) 92px;
|
||||
min-width: 880px;
|
||||
}
|
||||
.gradient-head { color: var(--muted); font: 0.55rem/1 var(--mono); border-bottom: 1px solid var(--line-strong); }
|
||||
.gradient-head span { padding: 10px 8px; }
|
||||
.gradient-row { align-items: center; min-height: 62px; border-bottom: 1px solid var(--line); }
|
||||
.gradient-row > span { padding: 8px; font-size: 0.65rem; }
|
||||
.gradient-row > span:first-child b { font: 0.74rem/1 var(--mono); }
|
||||
.gradient-row > span:first-child small { display: block; margin-top: 5px; color: var(--muted); font: 0.5rem/1 var(--mono); }
|
||||
[data-reward][data-correct="true"] { color: var(--sage); }
|
||||
[data-reward][data-correct="false"] { color: var(--red); }
|
||||
[data-gradient] { display: block; width: 70%; height: 6px; background: var(--warm-gray); }
|
||||
[data-gradient] b { display: block; height: 100%; background: var(--sage); transition: width 180ms ease; }
|
||||
[data-gradient][data-sign="negative"] b { background: var(--red); }
|
||||
[data-gradient-label] { margin-left: 7px; color: var(--muted); font: 0.5rem/1 var(--mono); }
|
||||
[data-mask] { display: flex; align-items: center; gap: 7px; font: 0.54rem/1 var(--mono); }
|
||||
[data-mask] i { width: 8px; height: 8px; border-radius: 50%; background: var(--sage); }
|
||||
[data-mask][data-keep="false"] { color: var(--red); }
|
||||
[data-mask][data-keep="false"] i { background: var(--red); }
|
||||
|
||||
.grpo-summary,
|
||||
.formula-comparison {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
.grpo-summary article,
|
||||
.formula-comparison article { padding: 25px 30px; border-right: 1px solid var(--line); }
|
||||
.grpo-summary article:last-child,
|
||||
.formula-comparison article:last-child { border-right: 0; }
|
||||
.grpo-summary span,
|
||||
.formula-comparison span { color: var(--copper); font: 0.56rem/1 var(--mono); }
|
||||
.grpo-summary b,
|
||||
.formula-comparison b { display: block; margin: 12px 0; font-size: 0.82rem; }
|
||||
.grpo-summary p,
|
||||
.formula-comparison p { color: var(--muted); font-size: 0.66rem; line-height: 1.7; }
|
||||
.formula-comparison { grid-template-columns: repeat(4, 1fr); background: var(--warm-gray); }
|
||||
|
||||
.mopd-grid { display: grid; grid-template-columns: 1.2fr 0.8fr; border-bottom: 1px solid var(--line); }
|
||||
.teacher-matrix {
|
||||
position: relative;
|
||||
display: grid;
|
||||
grid-template-columns: 90px repeat(3, 1fr);
|
||||
grid-template-rows: 42px repeat(3, 100px);
|
||||
gap: 1px;
|
||||
padding: 28px 38px;
|
||||
background: var(--line);
|
||||
border-right: 1px solid var(--line);
|
||||
}
|
||||
.matrix-head { display: contents; }
|
||||
.matrix-head span,
|
||||
.matrix-head b,
|
||||
.row-name { display: grid; place-items: center; background: var(--paper); }
|
||||
.matrix-head b { font: 0.58rem/1 var(--mono); }
|
||||
.row-name { color: var(--muted) !important; writing-mode: vertical-rl; font-size: 0.54rem !important; }
|
||||
.row-name:nth-of-type(1) { grid-column: 1; grid-row: 2; }
|
||||
.row-name:nth-of-type(2) { grid-column: 1; grid-row: 3; }
|
||||
.row-name:nth-of-type(3) { grid-column: 1; grid-row: 4; }
|
||||
.teacher-matrix button {
|
||||
position: relative;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
gap: 5px;
|
||||
color: var(--muted);
|
||||
background: var(--paper);
|
||||
border: 0;
|
||||
cursor: pointer;
|
||||
}
|
||||
.teacher-matrix button i { width: 13px; height: 13px; border: 1px solid var(--blue); border-radius: 50%; }
|
||||
.teacher-matrix button b { font: 0.66rem/1 var(--mono); }
|
||||
.teacher-matrix button small { font: 0.48rem/1 var(--mono); }
|
||||
.teacher-matrix button[aria-pressed="true"] { color: white; background: var(--blue); }
|
||||
.teacher-matrix button[aria-pressed="true"] i { background: white; border-color: white; }
|
||||
.teacher-matrix button:nth-of-type(1) { grid-column: 2; grid-row: 2; }
|
||||
.teacher-matrix button:nth-of-type(2) { grid-column: 3; grid-row: 2; }
|
||||
.teacher-matrix button:nth-of-type(3) { grid-column: 4; grid-row: 2; }
|
||||
.teacher-matrix button:nth-of-type(4) { grid-column: 2; grid-row: 3; }
|
||||
.teacher-matrix button:nth-of-type(5) { grid-column: 3; grid-row: 3; }
|
||||
.teacher-matrix button:nth-of-type(6) { grid-column: 4; grid-row: 3; }
|
||||
.teacher-matrix button:nth-of-type(7) { grid-column: 2; grid-row: 4; }
|
||||
.teacher-matrix button:nth-of-type(8) { grid-column: 3; grid-row: 4; }
|
||||
.teacher-matrix button:nth-of-type(9) { grid-column: 4; grid-row: 4; }
|
||||
|
||||
.mopd-controls { display: grid; align-content: center; gap: 30px; padding: 34px 38px; }
|
||||
.mopd-controls > div > b { display: block; margin: 11px 0; font-size: 1.15rem; }
|
||||
.mopd-controls > div > p { color: var(--muted); font-size: 0.7rem; line-height: 1.7; }
|
||||
|
||||
.token-prefill { padding: 28px 38px; border-bottom: 1px solid var(--line); overflow-x: auto; }
|
||||
.token-flow { display: grid; grid-template-columns: repeat(12, minmax(68px, 1fr)); gap: 8px; min-width: 900px; margin-top: 25px; }
|
||||
.token-flow > div { padding: 12px 9px; background: var(--warm-gray); }
|
||||
.token-flow > div > b { display: block; min-height: 30px; font-size: 0.7rem; text-align: center; }
|
||||
.token-flow > div > span { position: relative; display: block; height: 7px; margin-top: 8px; background: var(--paper); }
|
||||
.token-flow > div > span i { display: block; height: 100%; background: var(--blue); }
|
||||
.token-flow > div > span:nth-of-type(2) i { background: var(--copper); }
|
||||
.token-flow small { position: absolute; top: 10px; left: 0; color: var(--muted); font: 0.45rem/1 var(--mono); }
|
||||
.token-flow em { display: block; margin-top: 22px; color: var(--sage); font: 0.55rem/1 var(--mono); font-style: normal; text-align: center; }
|
||||
.token-flow em[data-sign="negative"] { color: var(--red); }
|
||||
.prob-legend { display: flex; gap: 20px; margin-top: 24px; color: var(--muted); font: 0.54rem/1 var(--mono); }
|
||||
.prob-legend span { display: flex; align-items: center; gap: 7px; }
|
||||
.prob-legend i { width: 14px; height: 5px; background: var(--blue); }
|
||||
.prob-legend i.teacher { background: var(--copper); }
|
||||
|
||||
.mopd-pipeline { display: grid; grid-template-columns: 1fr 30px 1fr 30px 1fr 30px 1fr; align-items: stretch; border-bottom: 1px solid var(--line); }
|
||||
.mopd-pipeline article { padding: 24px; }
|
||||
.mopd-pipeline > i { display: grid; place-items: center; color: var(--copper); font-style: normal; }
|
||||
.mopd-pipeline span { color: var(--copper); font: 0.56rem/1 var(--mono); }
|
||||
.mopd-pipeline b { display: block; margin: 10px 0; font-size: 0.78rem; }
|
||||
.mopd-pipeline p { color: var(--muted); font-size: 0.64rem; line-height: 1.55; }
|
||||
|
||||
.lab-footnote { display: grid; grid-template-columns: 180px 1fr; gap: 28px; padding: 24px 38px; background: var(--navy); color: white; }
|
||||
.lab-footnote b { font-size: 0.8rem; }
|
||||
.lab-footnote p { color: rgb(255 255 255 / 70%); font-size: 0.7rem; line-height: 1.7; }
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.lab-header,
|
||||
.view-intro,
|
||||
.budget-workbench,
|
||||
.mopd-grid,
|
||||
.interpretation { grid-template-columns: 1fr; }
|
||||
.mode-tabs { grid-template-columns: 1fr; }
|
||||
.mode-tabs button { border-right: 0; border-bottom: 1px solid var(--line); }
|
||||
.budget-controls,
|
||||
.teacher-matrix { border-right: 0; border-bottom: 1px solid var(--line); }
|
||||
.metric-grid,
|
||||
.grpo-summary,
|
||||
.formula-comparison { grid-template-columns: repeat(2, 1fr); }
|
||||
.metric-grid article:nth-child(2),
|
||||
.grpo-summary article:nth-child(2),
|
||||
.formula-comparison article:nth-child(2) { border-right: 0; }
|
||||
.mopd-pipeline { grid-template-columns: 1fr; }
|
||||
.mopd-pipeline > i { transform: rotate(90deg); min-height: 26px; }
|
||||
}
|
||||
|
||||
@media (max-width: 620px) {
|
||||
.lab-header,
|
||||
.view-intro,
|
||||
.budget-controls,
|
||||
.allocation-panel,
|
||||
.gradient-table,
|
||||
.mopd-controls,
|
||||
.token-prefill { padding-inline: 20px; }
|
||||
.budget-presets,
|
||||
.grpo-toolbar { padding-inline: 20px; }
|
||||
.allocation-readout,
|
||||
.metric-grid,
|
||||
.grpo-summary,
|
||||
.formula-comparison { grid-template-columns: 1fr 1fr; }
|
||||
.allocation-readout div:nth-child(2),
|
||||
.metric-grid article:nth-child(2n),
|
||||
.grpo-summary article:nth-child(2),
|
||||
.formula-comparison article:nth-child(2n) { border-right: 0; }
|
||||
.teacher-matrix { grid-template-columns: 60px repeat(3, 1fr); padding-inline: 20px; }
|
||||
.teacher-matrix button small { display: none; }
|
||||
.mopd-pipeline article { padding: 20px; }
|
||||
.lab-footnote { grid-template-columns: 1fr; padding-inline: 20px; }
|
||||
.prob-legend { flex-wrap: wrap; }
|
||||
}
|
||||
</style>
|
||||
@@ -7,6 +7,7 @@
|
||||
<a href="/roadmap/">学习地图</a>
|
||||
<a href="/moe/">MoE 专题</a>
|
||||
<a href="/long-context/">长上下文专题</a>
|
||||
<a href="/reasoning/">推理专题</a>
|
||||
<a href="/progress/">研究进度</a>
|
||||
<a href="https://git.k1412.top/wuyang/llm-atlas" rel="noreferrer">开放源码</a>
|
||||
<a href="https://github.com/MoonshotAI/Kimi-K3" rel="noreferrer">K3 官方报告</a>
|
||||
|
||||
@@ -12,6 +12,7 @@ const items = [
|
||||
{ id: "foundations", href: "/foundations/", label: "基础原理" },
|
||||
{ id: "moe", href: "/moe/", label: "MoE" },
|
||||
{ id: "long-context", href: "/long-context/", label: "长上下文" },
|
||||
{ id: "reasoning", href: "/reasoning/", label: "推理" },
|
||||
{ id: "papers", href: "/papers/", label: "论文库" },
|
||||
{ id: "progress", href: "/progress/", label: "进度" },
|
||||
];
|
||||
|
||||
@@ -165,9 +165,9 @@ export const chapters: Chapter[] = [
|
||||
kicker: "REASONING",
|
||||
question: "模型如何学会多想一会儿，并检查自己的答案？",
|
||||
summary: "从 CoT、搜索与验证器，到 GRPO、DeepSeek-R1、Kimi k1.5 和 multi-effort RL。",
|
||||
status: "researching",
|
||||
progress: 25,
|
||||
papers: 21,
|
||||
status: "published",
|
||||
progress: 76,
|
||||
papers: 30,
|
||||
prerequisites: ["10"],
|
||||
highlights: ["GRPO", "R1-Zero", "On-policy 蒸馏"],
|
||||
},
|
||||
|
||||
+133
-5
@@ -760,6 +760,30 @@ export const papers: Paper[] = [
|
||||
contribution: "用中间推理示例显著提升大模型复杂任务表现。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Large Language Models are Zero-Shot Reasoners",
|
||||
url: "https://arxiv.org/abs/2205.11916",
|
||||
topics: ["推理"],
|
||||
contribution: "用统一的 step-by-step 触发语句，在不提供 few-shot rationale 时激发多任务零样本 CoT。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Solving Quantitative Reasoning Problems with Language Models",
|
||||
url: "https://arxiv.org/abs/2206.14858",
|
||||
topics: ["推理", "Scaling"],
|
||||
contribution: "Minerva 以技术内容继续训练语言模型，系统推进数学与科学定量推理。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "Training Verifiers to Solve Math Word Problems",
|
||||
url: "https://arxiv.org/abs/2110.14168",
|
||||
topics: ["推理", "后训练"],
|
||||
contribution: "建立 GSM8K，并系统展示多采样后由学习式 verifier 选择答案的收益。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Self-Consistency Improves Chain of Thought Reasoning in Language Models",
|
||||
@@ -768,6 +792,14 @@ export const papers: Paper[] = [
|
||||
contribution: "采样多条推理路径并对最终答案聚合。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Least-to-Most Prompting Enables Complex Reasoning in Large Language Models",
|
||||
url: "https://arxiv.org/abs/2205.10625",
|
||||
topics: ["推理"],
|
||||
contribution: "先把难题分解成子问题，再按顺序利用已解结果组合答案。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "STaR: Bootstrapping Reasoning With Reasoning",
|
||||
@@ -776,6 +808,30 @@ export const papers: Paper[] = [
|
||||
contribution: "迭代生成、筛选并训练成功 rationale。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Solving Math Word Problems With Process- and Outcome-Based Feedback",
|
||||
url: "https://arxiv.org/abs/2211.14275",
|
||||
topics: ["推理", "后训练"],
|
||||
contribution: "在 GSM8K 中比较过程与结果反馈，揭示最终正确率和推理轨迹错误率的不同需求。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks",
|
||||
url: "https://arxiv.org/abs/2211.12588",
|
||||
topics: ["推理", "Agent"],
|
||||
contribution: "让模型用程序表达推理，把数值计算交给外部执行器。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Self-Refine: Iterative Refinement with Self-Feedback",
|
||||
url: "https://arxiv.org/abs/2303.17651",
|
||||
topics: ["推理"],
|
||||
contribution: "由同一模型循环生成、反馈与修订，在不追加训练的情况下扩展串行测试时计算。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Tree of Thoughts: Deliberate Problem Solving with Large Language Models",
|
||||
@@ -784,12 +840,20 @@ export const papers: Paper[] = [
|
||||
contribution: "显式搜索多个 thought 分支并评估中间状态。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting",
|
||||
url: "https://arxiv.org/abs/2305.04388",
|
||||
topics: ["推理", "评测"],
|
||||
contribution: "用偏置提示实验表明，流畅的公开 CoT 可能合理化答案而不忠实披露影响因素。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Let's Verify Step by Step",
|
||||
url: "https://arxiv.org/abs/2305.20050",
|
||||
topics: ["推理", "后训练"],
|
||||
contribution: "过程奖励模型在数学推理中优于只看最终答案。",
|
||||
contribution: "发布 PRM800K；在 500 题 MATH 子集的 best-of-1860 选择中验证过程监督优势。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
@@ -809,12 +873,36 @@ export const papers: Paper[] = [
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters",
|
||||
url: "https://arxiv.org/abs/2408.03314",
|
||||
topics: ["推理", "Scaling"],
|
||||
contribution: "按模型与题目难度在修订、并行采样和 PRM 搜索之间分配测试时计算。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
|
||||
url: "https://arxiv.org/abs/2411.15124",
|
||||
topics: ["后训练", "推理"],
|
||||
contribution: "开放从数据策展、SFT、偏好学习到 RLVR 的完整 post-training 配方。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "OpenAI o1 System Card",
|
||||
url: "https://arxiv.org/abs/2412.16720",
|
||||
topics: ["推理", "评测"],
|
||||
contribution: "记录 reasoning model 的能力、安全评测与测试时推理边界。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning",
|
||||
url: "https://arxiv.org/abs/2501.12948",
|
||||
topics: ["推理", "后训练"],
|
||||
contribution: "R1-Zero 纯 RL 涌现推理；R1 加冷启动、多阶段训练与蒸馏。",
|
||||
contribution: "R1-Zero 从 base 直接做规则奖励 RL；R1 再加入冷启动、SFT、多阶段 RL 与蒸馏。",
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
@@ -823,10 +911,50 @@ export const papers: Paper[] = [
|
||||
title: "Kimi k1.5: Scaling Reinforcement Learning with LLMs",
|
||||
url: "https://arxiv.org/abs/2501.12599",
|
||||
topics: ["推理", "后训练"],
|
||||
contribution: "扩展长 CoT 强化学习和测试时计算。",
|
||||
contribution: "用 128K RL context、partial rollout 与 long2short 扩展长 CoT 和测试时计算。",
|
||||
spotlight: "Kimi",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "s1: Simple test-time scaling",
|
||||
url: "https://arxiv.org/abs/2501.19393",
|
||||
topics: ["推理", "后训练"],
|
||||
contribution: "从强教师精选 1K 道推理题蒸馏，并用 budget forcing 控制思考长度。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "DAPO: An Open-Source LLM Reinforcement Learning System at Scale",
|
||||
url: "https://arxiv.org/abs/2503.14476",
|
||||
topics: ["推理", "后训练"],
|
||||
contribution: "用 Clip-Higher、Dynamic Sampling、token-level loss 与 overlong shaping 稳定长 CoT RL。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "Understanding R1-Zero-Like Training: A Critical Perspective",
|
||||
url: "https://arxiv.org/abs/2503.20783",
|
||||
topics: ["推理", "后训练"],
|
||||
contribution: "提出 Dr.GRPO，分析 response-length 与 question-difficulty 两种优化偏置。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?",
|
||||
url: "https://arxiv.org/abs/2504.13837",
|
||||
topics: ["推理", "后训练", "评测"],
|
||||
contribution: "用 pass@k 检查当前 RLVR 是扩展解法覆盖，还是主要重排已有正确路径。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2026,
|
||||
title: "MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training",
|
||||
url: "https://arxiv.org/abs/2606.30406",
|
||||
topics: ["推理", "后训练", "Agent"],
|
||||
contribution: "让学生在自己的 rollout 上接收同源领域教师的稠密逐 Token 信号，整合多项 RL 能力。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "WebGPT: Browser-assisted Question-Answering with Human Feedback",
|
||||
@@ -927,8 +1055,8 @@ export const papers: Paper[] = [
|
||||
year: 2025,
|
||||
title: "Kimi K2: Open Agentic Intelligence",
|
||||
url: "https://arxiv.org/abs/2507.20534",
|
||||
topics: ["MoE", "Agent", "后训练"],
|
||||
contribution: "开放 1T MoE Agent 模型，强调工具调用数据合成与 joint RL。",
|
||||
topics: ["MoE", "Agent", "后训练", "推理"],
|
||||
contribution: "开放 1T MoE Agent 模型，以 verifiable gym 与 self-critique rubric 做 joint RL。",
|
||||
spotlight: "Kimi",
|
||||
verified: true,
|
||||
},
|
||||
|
||||
@@ -313,6 +313,7 @@ const toc = [
|
||||
增加推理计算可以转化为能力；不是“输出越长越聪明”。
|
||||
</p>
|
||||
</div>
|
||||
<a class="button primary" href="/reasoning/#deepseek">进入推理专题：从 DeepSeekMath、R1 到 DAPO / Dr.GRPO 的完整推导 →</a>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="v32">
|
||||
|
||||
+32
-1
@@ -9,6 +9,7 @@ const routes: Record<string, string> = {
|
||||
foundations: "/foundations/",
|
||||
moe: "/moe/",
|
||||
"long-context": "/long-context/",
|
||||
reasoning: "/reasoning/",
|
||||
};
|
||||
|
||||
const paths = [
|
||||
@@ -78,6 +79,7 @@ const paths = [
|
||||
<a class="button" href="/deepseek/">DeepSeek 专题</a>
|
||||
<a class="button" href="/moe/">MoE 专题</a>
|
||||
<a class="button" href="/long-context/">长上下文专题</a>
|
||||
<a class="button" href="/reasoning/">推理专题</a>
|
||||
</div>
|
||||
</div>
|
||||
<aside class="hero-aside" aria-label="项目统计">
|
||||
@@ -87,7 +89,7 @@ const paths = [
|
||||
<div class="hero-stats">
|
||||
<div><b>16</b><span>核心专题</span></div>
|
||||
<div><b>151</b><span>K3 报告来源</span></div>
|
||||
<div><b>130</b><span>关键论文索引</span></div>
|
||||
<div><b>146</b><span>关键论文索引</span></div>
|
||||
<div><b>47p</b><span>K3 技术报告</span></div>
|
||||
</div>
|
||||
</aside>
|
||||
@@ -101,6 +103,22 @@ const paths = [
|
||||
|
||||
<section class="section compact release-section" id="new-chapters">
|
||||
<div class="release-grid">
|
||||
<a class="release-card reasoning-release" href="/reasoning/">
|
||||
<div>
|
||||
<p class="eyebrow"><span>NEW / CHAPTER 11</span> REASONING & TEST-TIME SCALING</p>
|
||||
<h2>“多想一会儿”，到底把计算花到了哪里？</h2>
|
||||
<p>
|
||||
把推理拆成结果、覆盖、选择、过程、预算、优化、分布与系统八张账，
|
||||
从 CoT、verifier 与 GRPO 一路走到 DeepSeek-R1、Kimi k1.5 与 K3 MOPD。
|
||||
</p>
|
||||
</div>
|
||||
<dl>
|
||||
<div><dt>LINEAGE</dt><dd>2021 → 2026</dd></div>
|
||||
<div><dt>PAPERS</dt><dd>30 篇一手来源</dd></div>
|
||||
<div><dt>LAB</dt><dd>预算 · GRPO · 九教师</dd></div>
|
||||
</dl>
|
||||
<span class="release-arrow" aria-hidden="true">进入推理专题 →</span>
|
||||
</a>
|
||||
<a class="release-card moe-release" href="/moe/">
|
||||
<div>
|
||||
<p class="eyebrow"><span>NEW / CHAPTER 06</span> SPARSE EXPERTS</p>
|
||||
@@ -366,6 +384,15 @@ const paths = [
|
||||
transition: transform 180ms ease, border-color 180ms ease;
|
||||
}
|
||||
|
||||
.reasoning-release {
|
||||
grid-column: 1 / -1;
|
||||
min-height: 510px;
|
||||
background:
|
||||
radial-gradient(circle at 82% 18%, rgba(56, 91, 128, 0.17), transparent 30%),
|
||||
radial-gradient(circle at 63% 72%, rgba(150, 93, 58, 0.11), transparent 28%),
|
||||
var(--paper-raised);
|
||||
}
|
||||
|
||||
.release-card:hover {
|
||||
transform: translateY(-3px);
|
||||
border-color: var(--copper);
|
||||
@@ -428,6 +455,10 @@ const paths = [
|
||||
padding-bottom: 76px;
|
||||
}
|
||||
|
||||
.reasoning-release {
|
||||
grid-column: auto;
|
||||
}
|
||||
|
||||
.release-card dl {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
@@ -378,6 +378,7 @@ const toc = [
|
||||
On-policy distillation 让学生自己生成当前前缀，再在这个前缀上比较教师和学生对下一个 Token 的概率，
|
||||
形成稠密 reward。这样训练分布更贴近学生真正会访问的状态，也能自然结合 partial rollout。
|
||||
</p>
|
||||
<a class="button primary" href="/reasoning/#k3">进入推理专题：并排理解 partial rollout、reasoning effort 与 MOPD →</a>
|
||||
|
||||
<h3>部署约束直接进入后训练</h3>
|
||||
<p>
|
||||
|
||||
@@ -683,7 +683,7 @@ const paperChain = [
|
||||
<div class="next-links">
|
||||
<a class="button primary" href="/k3/#moe">回到 K3：在整机架构中定位 Stable LatentMoE →</a>
|
||||
<a class="button" href="/deepseek/">DeepSeek 完整论文谱系</a>
|
||||
<a class="button" href="/papers/">搜索全部 130 篇论文</a>
|
||||
<a class="button" href="/papers/">搜索全部 146 篇论文</a>
|
||||
</div>
|
||||
</section>
|
||||
</article>
|
||||
|
||||
@@ -7,11 +7,12 @@ const published = chapters.filter((chapter) => chapter.status === "published").l
|
||||
const researching = chapters.filter((chapter) => ["researching", "drafting"].includes(chapter.status)).length;
|
||||
|
||||
const workstreams = [
|
||||
{ label: "研究框架与规范", value: 76, next: "给 130 篇索引补充逐篇精读层级" },
|
||||
{ label: "研究框架与规范", value: 82, next: "给推理专题补逐篇图表/实验精读层级" },
|
||||
{ label: "网站设计系统", value: 89, next: "打印样式与更多通用可视化组件" },
|
||||
{ label: "Kimi K3 深读", value: 55, next: "扩写 scaling / infra 逐图笔记" },
|
||||
{ label: "Kimi K3 深读", value: 64, next: "扩写 scaling / pre-training / infra 逐图笔记" },
|
||||
{ label: "Transformer 基础", value: 52, next: "加入矩阵形状动画与手算练习" },
|
||||
{ label: "DeepSeek 专题", value: 61, next: "GRPO 完整公式与训练轨迹推导" },
|
||||
{ label: "DeepSeek 专题", value: 71, next: "补 R1 / DAPO 的逐图训练轨迹与复现对照" },
|
||||
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
||||
{ label: "稀疏计算与 MoE", value: 74, next: "补充真实集群 traces 与专家特化案例" },
|
||||
{ label: "长上下文专题", value: 72, next: "加入更多论文逐图笔记与真实模型配置对比" },
|
||||
{ label: "引用与事实检查", value: 54, next: "自动化外链复查与来源等级扩展" },
|
||||
@@ -38,7 +39,7 @@ const workstreams = [
|
||||
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
||||
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
||||
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-28 23:24 CST</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 00:38 CST</dd></div>
|
||||
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
@@ -48,7 +49,7 @@ const workstreams = [
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>01</span> WORKSTREAMS</p>
|
||||
<h2>九条工作流同时推进，但不混淆“有页面”和“已核验”</h2>
|
||||
<h2>十条工作流同时推进，但不混淆“有页面”和“已核验”</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
内容首版优先打通全局脉络；随后每轮迭代选择一个专题推进到论文/工程层，并做独立事实复核。
|
||||
@@ -85,11 +86,12 @@ const workstreams = [
|
||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||
<article><span>✓</span><h3>16 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||
<article><span>✓</span><h3>五张原创交互图</h3><p>K3 三轴架构、Self-Attention Query、DeepSeek 谱系、长上下文成本与 MoE 路由实验室。</p></article>
|
||||
<article><span>✓</span><h3>五篇首版长文</h3><p>K3 导读、Transformer 基础、DeepSeek 谱系、长上下文与 MoE 专题。</p></article>
|
||||
<article><span>✓</span><h3>八个原创交互视图</h3><p>K3、注意力、DeepSeek、长上下文、MoE，以及推理预算/GRPO/MOPD 三页签实验。</p></article>
|
||||
<article><span>✓</span><h3>六篇首版长文</h3><p>K3 导读、Transformer 基础、DeepSeek 谱系、长上下文、MoE 与推理专题。</p></article>
|
||||
<article><span>✓</span><h3>长上下文深度专题</h3><p>五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。</p></article>
|
||||
<article><span>✓</span><h3>MoE 深度专题</h3><p>六张账、19 篇一手论文、DeepSeek/K3 主线与路由—容量—通信交互实验室。</p></article>
|
||||
<article><span>✓</span><h3>130 篇关键论文索引</h3><p>覆盖 12 个专题，支持全文搜索、标签筛选与 Kimi/DeepSeek 聚光主线。</p></article>
|
||||
<article><span>✓</span><h3>推理深度专题</h3><p>八张账、30 篇一手论文链、DeepSeek/Kimi 双主线与三页签互动实验室。</p></article>
|
||||
<article><span>✓</span><h3>146 篇关键论文索引</h3><p>覆盖 12 个专题，支持全文搜索、标签筛选与 Kimi/DeepSeek 聚光主线。</p></article>
|
||||
<article><span>✓</span><h3>公开仓库与自托管发布</h3><p>源码公开到 git.k1412.top，网站由不可变镜像、Compose Manager 与 HTTPS 交付。</p></article>
|
||||
</div>
|
||||
</section>
|
||||
@@ -104,10 +106,10 @@ const workstreams = [
|
||||
</div>
|
||||
<div class="queue-table">
|
||||
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
||||
<div><span>P0</span><strong>推理模型与测试时扩展</strong><p>CoT → verifier → GRPO → R1 → k1.5 → K3 MOPD</p><em>奖励/预算交互图</em></div>
|
||||
<div><span>P0</span><strong>大规模训练系统</strong><p>ZeRO / Megatron → Expert/Context Parallel → DualPipe / MoonEP</p><em>显存与通信计算器</em></div>
|
||||
<div><span>P1</span><strong>长上下文二轮深化</strong><p>真实模型配置 → 内核细节 → 长上下文评测与失败案例</p><em>配置比较器 + 逐图论文笔记</em></div>
|
||||
<div><span>P1</span><strong>MoE 二轮深化</strong><p>真实负载 traces → 专家特化可解释性 → 共享专家语义</p><em>案例库 + 集群证据</em></div>
|
||||
<div><span>P1</span><strong>大规模训练系统</strong><p>ZeRO/Megatron → Expert/Context Parallel → DualPipe/MoonEP</p><em>显存与通信计算器</em></div>
|
||||
<div><span>P1</span><strong>推理二轮深化</strong><p>真实 pass@k 曲线 → PRM 失败案例 → 逐篇图表精读</p><em>案例库 + 真实 traces</em></div>
|
||||
<div><span>P2</span><strong>原生多模态</strong><p>ViT/CLIP → connector VLM → Kimi-VL/MoonViT-V2</p><em>视觉 Token 流程图</em></div>
|
||||
</div>
|
||||
</section>
|
||||
@@ -147,6 +149,7 @@ const workstreams = [
|
||||
<div><time>2026-07-28</time><b>优先原创重绘</b><p>架构图做成可缩放 SVG/HTML，明确简化与来源。</p></div>
|
||||
<div><time>2026-07-28</time><b>双重开放许可</b><p>代码 MIT，原创文字与图 CC BY-SA 4.0。</p></div>
|
||||
<div><time>2026-07-28</time><b>自托管交付</b><p>源码公开到 git.k1412.top，网站部署到 k1412 私有基础设施。</p></div>
|
||||
<div><time>2026-07-29</time><b>推理按八张账组织</b><p>把答案、覆盖、选择、过程、预算、优化、分布与系统证据分开核算。</p></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -128,7 +128,9 @@
|
||||
.reading-progress {
|
||||
position: fixed;
|
||||
z-index: 100;
|
||||
inset: 0 0 auto;
|
||||
inset: 0 auto auto 0;
|
||||
width: 100vw;
|
||||
max-width: 100%;
|
||||
height: 3px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user