feat: publish native multimodal chapter
This commit is contained in:
+18
-3
@@ -17,6 +17,7 @@
|
||||
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
||||
| 原生多模态 | 完成首版 | 76% | 真实视觉 Token traces、跨分辨率消融、OCR 失败案例与视觉 Agent 安全轨迹 |
|
||||
| 稀疏计算与 MoE | 完成首版 | 74% | 真实负载 traces 与专家特化案例 |
|
||||
| 长上下文专题 | 完成首版 | 72% | 真实模型配置、内核细节与失败案例 |
|
||||
| 大规模训练系统 | 完成首版 | 71% | 真实集群 traces、故障案例与精确 topology 配置 |
|
||||
@@ -34,10 +35,10 @@
|
||||
- [x] 提炼参考网站的编辑设计语言。
|
||||
- [x] 确认 `git.k1412.top` 为 Gitea/Forgejo 兼容服务且本机 HTTPS 凭据可用于既有仓库。
|
||||
- [x] 使用 Grok CLI 检索并形成约 95 篇一手论文的补充路线,主代理已回查关键来源。
|
||||
- [x] 完成 314 篇关键论文索引,覆盖 14 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成 355 篇关键论文索引,覆盖 14 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、训练系统与数值优化十三篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、语言模型前史四联实验、Transformer 四联实验、DeepSeek 谱系、长上下文、MoE 路由、推理三页签,以及训练系统、Scaling、数据工程、数值、Alignment 与 Agent 专题各四页签等三十九个原创交互视图。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统与数值优化十四篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、语言模型前史四联实验、Transformer 四联实验、DeepSeek 谱系、长上下文、MoE 路由、推理三页签,以及训练系统、Scaling、数据工程、数值、Alignment、Agent 与原生多模态专题各四页签等四十三个原创交互视图。
|
||||
- [x] 完成长上下文首版:五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文,并建立长上下文研究账本。
|
||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文,并建立 MoE 研究账本。
|
||||
@@ -108,9 +109,17 @@
|
||||
- [x] Agent 真实 Chrome 断言通过:Direct/schema 故障传播、final-state verifier、pass@k / pass^k、非幂等副作用、wait-all 长尾、键盘 tabs 与 390px 移动端均正确响应。
|
||||
- [x] Astro 类型检查、生产构建、17 个页面、792 个站内引用和 18 个跨页锚点通过;Agent 与既有九套专题本地 Chrome 全量回归无异常。
|
||||
- [x] Agent 首版以源提交 `25982cb`、不可变镜像 `20260728T225618Z-25982cb` 发布;NAS、VPS/Tailscale、NPM、DNS、HTTPS、证书、门户、公开 Forgejo 与十套生产 Chrome 回归全链路通过。
|
||||
- [x] 启动原生多模态专题:用十六张问题账拆开视觉语义空间、connector、Token 预算、OCR、视频、统一理解生成、RL、幻觉、评测、Agent 与系统。
|
||||
- [x] 使用 Grok Headless 扩展七波、55+ 候选主线;218 行候选账与 1362 行正式一手证据账永久分离。
|
||||
- [x] 本地完整校验 13 份多模态论文 PDF,并通过官方 arXiv HTML、作者仓库与既有 K2.5/K3 报告核验其余核心节点。
|
||||
- [x] 完成原生多模态首版:30 个正文目录、DeepSeek-VL/VL2—Janus—OCR 三分支、Kimi 三代 MoonViT 与四联交互实验。
|
||||
- [x] 论文库新增 ResNet、ALIGN、NaViT、DeepSeek-VL2、Janus、DeepSeek-OCR、Vision-R1 等 41 个节点,从 314 篇扩充至 355 篇。
|
||||
- [x] 原生多模态真实 Chrome 断言通过:视觉 Token 超预算、K3 五维原生训练、OCR 报告 / 插值 / 证据外边界、vision-in-the-loop、键盘 tabs 与 390px 移动端均正确响应。
|
||||
- [x] Astro 类型检查、生产构建、18 个页面、875 个站内引用和 17 个跨页锚点通过;多模态页面桌面 / 移动端无文档级横向溢出。
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] 原生多模态二轮:真实视觉 Token traces、跨分辨率 / connector 消融、OCR 与视觉 Agent 安全失败案例。
|
||||
- [ ] Agent 二轮:真实环境 traces、cross-harness ablation、Agent RL 训练曲线与提示注入案例。
|
||||
- [ ] Transformer 二轮:多头电路逐图、Pre/Post-LN 真实 traces、Flash/KV kernel 与模型配置对照。
|
||||
- [ ] 语言模型前史二轮:Kneser–Ney / LSTM / Bahdanau 逐图精读、真实小语料复现与 tokenizer 公平性案例。
|
||||
@@ -186,6 +195,12 @@
|
||||
| 2026-07-29 | Grok Agent 线索与正式证据永久分离 | 候选只负责查漏;公式、数字和系统结论必须回到一手论文、正式会议版本或官方报告 |
|
||||
| 2026-07-29 | 论文库扩充到 314 篇 | 新增 34 个环境、工具、Web/SWE/桌面 Agent、可靠性、安全与 Agent RL 节点 |
|
||||
| 2026-07-29 | Agent 首版用不可变镜像 `20260728T225618Z-25982cb` 发布 | OCI digest `sha256:a781ad92…ba52`;复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo |
|
||||
| 2026-07-29 | 原生多模态按五层管道与十六张账组织 | 像素、视觉塔、压缩 / connector、主干与输出 / 工具闭环分开定位;“原生”再拆成数据、目标、优化、输入输出与 Agent 五维 |
|
||||
| 2026-07-29 | Grok 多模态召回与正式证据永久分离 | 218 行候选账只做查漏;1362 行正式账本只接受一手论文、官方报告与可计算公式 |
|
||||
| 2026-07-29 | DeepSeek 多模态永久保留三分支 | VL/VL2 的理解、Janus 的统一生成、OCR 的光学上下文压缩不画成错误单向代际谱系 |
|
||||
| 2026-07-29 | Kimi 三代 MoonViT 按训练制度重建 | Kimi-VL 的 SigLIP + caption / 对齐,K2.5 的共享图像视频 / zero-vision SFT,K3 的从头共同 NTP 分开说明 |
|
||||
| 2026-07-29 | 光学压缩实验显式分三级证据 | DeepSeek-OCR 报告锚点、中间教学插值与超过已核范围的“不外推”在交互中使用不同状态 |
|
||||
| 2026-07-29 | 论文库扩充到 355 篇 | 新增 41 个视觉表示、连接器、分辨率、OCR、视频、统一生成、评测与视觉 RL 节点 |
|
||||
|
||||
## 未决问题
|
||||
|
||||
|
||||
@@ -17,9 +17,9 @@
|
||||
- 持续进度:[PROGRESS.md](./PROGRESS.md)
|
||||
- 证据与写作规范:[research/METHODOLOGY.md](./research/METHODOLOGY.md)
|
||||
|
||||
当前里程碑包含 16 专题学习地图、314 篇关键论文索引、Kimi K3 完整导读,
|
||||
当前里程碑包含 16 专题学习地图、355 篇关键论文索引、Kimi K3 完整导读,
|
||||
语言模型前史、Transformer 基础、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、训练系统与数值优化深度专题,
|
||||
以及 39 个覆盖核心机制的原创交互视图。
|
||||
以及 43 个覆盖核心机制的原创交互视图。
|
||||
其余专题按进度账本持续扩建。
|
||||
|
||||
## 本地开发
|
||||
|
||||
@@ -108,6 +108,14 @@ Kimi K2→K2.5→K3 的 white-box harness、AET、AgentENV 和百万 Token Agent
|
||||
|
||||
ViT/CLIP → Flamingo/BLIP-2/LLaVA → 原生多模态与视频;Kimi-VL、MoonViT-V2 和“视觉进入同一主干”的意义。
|
||||
|
||||
首版已完成:以语义空间、连接器、视觉 Token、OCR / 文档、视频、融合、理解 / 生成、SFT、RL、
|
||||
Agent、幻觉、安全、评测、系统、MoE 与产品谱系十六张账,串起 2012–2026 的 55 个一手节点。
|
||||
正文把像素、视觉塔、压缩 / projector、主干与输出 / 工具拆成五层;重点讲解 DeepSeek-VL/VL2、
|
||||
Janus、DeepSeek-OCR 三条不同分支,以及 Kimi-VL → K2.5 → K3 从 SigLIP 初始化和独立对齐,
|
||||
走到 MoonViT-V2 从头共同 next-token prediction 的训练制度变化。配套视觉 Token、connector /
|
||||
native、光学上下文压缩与 vision-in-the-loop 四个独立实验,所有作者报告、教学插值和证据外区域
|
||||
明确分级。
|
||||
|
||||
### 14. 推理服务与低成本部署
|
||||
|
||||
KV Cache、PagedAttention/vLLM、连续批处理、推测解码、Prefill/Decode 解耦、前缀缓存、集群调度;Mooncake 与 K3 KDA-aware serving。
|
||||
|
||||
+2
-1
@@ -19,7 +19,8 @@
|
||||
"check:language-model-history-browser": "node scripts/check-language-model-history-browser.mjs",
|
||||
"check:transformer-browser": "node scripts/check-transformer-browser.mjs",
|
||||
"check:alignment-browser": "node scripts/check-alignment-browser.mjs",
|
||||
"check:agents-browser": "node scripts/check-agents-browser.mjs"
|
||||
"check:agents-browser": "node scripts/check-agents-browser.mjs",
|
||||
"check:multimodal-browser": "node scripts/check-multimodal-browser.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"@astrojs/sitemap": "3.7.3",
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
# 原生多模态专题 · Grok 候选召回
|
||||
|
||||
> 生成方式:Grok CLI Headless(`--no-subagents --no-memory`)
|
||||
> 日期:2026-07-29
|
||||
> 角色:只负责扩大候选池、暴露易混边界和提供可视化灵感,不作为事实来源。进入正式课程的技术主张、数字和历史定位,必须回到论文、技术报告或官方代码仓库核验。
|
||||
|
||||
## 1. Grok 建议的十六张问题账
|
||||
|
||||
| 编号 | 问题账 | 核心问题 |
|
||||
|---|---|---|
|
||||
| Q1 | Visual semantic space | 图像怎样获得可与语言比较的语义表示? |
|
||||
| Q2 | Connector | 视觉特征怎样接进语言模型,谁负责跨模态翻译? |
|
||||
| Q3 | Resolution & token budget | 高分辨率细节为何会迅速吃光上下文与算力? |
|
||||
| Q4 | OCR & documents | 场景文字、文档、表格与公式为何比自然图像更难? |
|
||||
| Q5 | Video | 时间维度怎样进入模型,帧数与视觉 Token 怎样压缩? |
|
||||
| Q6 | Fusion timing | early fusion、cross-attention、late fusion 分别交换了什么? |
|
||||
| Q7 | Understanding vs generation | 图像理解与图像生成能否共用表示和主干? |
|
||||
| Q8 | Multimodal SFT | 指令数据怎样把视觉表征变成可对话能力? |
|
||||
| Q9 | Multimodal RL | 奖励怎样越过文字输出,回到视觉观察与动作? |
|
||||
| Q10 | Agent & tools | 模型怎样主动裁剪、放大、写代码、截图并继续观察? |
|
||||
| Q11 | Hallucination | 模型为何会相信语言先验而忽略像素证据? |
|
||||
| Q12 | Safety | 图像中的提示注入、隐私与工具副作用怎样进入威胁模型? |
|
||||
| Q13 | Evaluation | 感知、OCR、知识、推理、视频和 Agent 应怎样分开测? |
|
||||
| Q14 | Training systems | 图像尺寸、动态 Token 与数据混合怎样改变并行和吞吐? |
|
||||
| Q15 | Sparse MoE + vision | 视觉 Token 会怎样影响路由负载、专家专化与稳定性? |
|
||||
| Q16 | DeepSeek vs Kimi | 两条产品线分别把什么问题当成了主矛盾? |
|
||||
|
||||
这些问题不会直接变成十六段论文罗列。正式章节会先建立一条统一的数据流,再让每张问题账回答“瓶颈为什么出现、哪篇论文改变了设计空间、代价是什么”。
|
||||
|
||||
## 2. Grok 召回的七个历史波次
|
||||
|
||||
| 波次 | 时间 | 候选主题 | 进入主线前要核验的边界 |
|
||||
|---|---|---|---|
|
||||
| W1 | 2012–2017 | AlexNet、ResNet、检测与区域特征 | 这是视觉表示前史,不应伪装成 LLM |
|
||||
| W2 | 2021–2023 | CLIP、ALIGN、LiT、SigLIP | 对齐表征不等于能开放式对话 |
|
||||
| W3 | 2021–2023 | Frozen、Flamingo、BLIP-2、LLaVA | “冻结”了谁、训练了谁必须逐篇说明 |
|
||||
| W4 | 2023–2025 | NaViT、动态切图、OCR、文档、视频 | 更高分辨率的真实代价是视觉 Token 与系统方差 |
|
||||
| W5 | 2023–2025 | Chameleon、Transfusion、SOLO、统一自回归 | 统一主干、统一序列、统一目标是三个不同命题 |
|
||||
| W6 | 2024–2025 | DeepSeek-VL/VL2、Janus、Qwen、InternVL、Kimi-VL | 产品线之间不能只用总参数或榜单横比 |
|
||||
| W7 | 2025–2026 | 视觉推理 RL、视觉工具、GUI Agent、K2.5、K3 | 结果必须记录工具、harness、步数与验证器 |
|
||||
|
||||
## 3. Grok 召回的候选论文池
|
||||
|
||||
### 视觉表示与语言监督
|
||||
|
||||
- AlexNet
|
||||
- ResNet
|
||||
- Vision Transformer
|
||||
- CLIP
|
||||
- ALIGN
|
||||
- LiT
|
||||
- SigLIP
|
||||
- NaViT
|
||||
|
||||
### 连接器与冻结式 VLM
|
||||
|
||||
- Frozen
|
||||
- Flamingo
|
||||
- BLIP
|
||||
- BLIP-2
|
||||
- MiniGPT-4
|
||||
- LLaVA
|
||||
- LLaVA-1.5
|
||||
- InstructBLIP
|
||||
- Kosmos-2
|
||||
|
||||
### 高分辨率、OCR、文档与视频
|
||||
|
||||
- Qwen-VL / Qwen2-VL / Qwen2.5-VL
|
||||
- InternVL / InternVL 1.5 / InternVL 2.5
|
||||
- LLaVA-NeXT / LLaVA-OneVision
|
||||
- Video-LLaVA
|
||||
- LLaMA-VID
|
||||
- DeepSeek-VL
|
||||
- DeepSeek-VL2
|
||||
- DeepSeek-OCR
|
||||
- DeepSeek-OCR2
|
||||
- Kimi-VL
|
||||
|
||||
### 统一理解与生成
|
||||
|
||||
- Chameleon
|
||||
- Transfusion
|
||||
- SOLO
|
||||
- Janus
|
||||
- JanusFlow
|
||||
- Janus-Pro
|
||||
|
||||
### 多模态推理、工具与 Agent
|
||||
|
||||
- Visual Sketchpad
|
||||
- Vision-R1
|
||||
- VTool-R1
|
||||
- ToolsRL
|
||||
- OSWorld
|
||||
- VisualWebArena
|
||||
- Kimi K2.5
|
||||
- Kimi K3
|
||||
|
||||
### 评测、幻觉与诊断
|
||||
|
||||
- MMBench
|
||||
- MMMU
|
||||
- MathVista
|
||||
- OCRBench
|
||||
- POPE
|
||||
- HallusionBench
|
||||
- Video-MME
|
||||
|
||||
## 4. DeepSeek 候选分支:不要画成一条直线
|
||||
|
||||
Grok 提醒应把 DeepSeek 多模态工作拆成三条问题导向不同的分支:
|
||||
|
||||
1. **理解分支:DeepSeek-VL → DeepSeek-VL2**
|
||||
- 关注真实世界高分辨率输入、动态切图、视觉—语言竞争和稀疏语言骨干;
|
||||
- 核验重点:两代视觉编码器、切图策略、像素重排、激活参数量。
|
||||
2. **统一生成分支:Janus → JanusFlow → Janus-Pro**
|
||||
- 关注理解和生成是否应该共享视觉编码路径;
|
||||
- 核验重点:解耦的是编码器还是主干,生成目标是离散自回归还是 rectified flow。
|
||||
3. **光学压缩分支:DeepSeek-OCR → DeepSeek-OCR2**
|
||||
- 关注把长文本上下文先压到图像,再以较少视觉 Token 解码;
|
||||
- 核验重点:DeepEncoder 结构、压缩率定义、官方报告的准确率边界、OCR2 的 token 重排。
|
||||
|
||||
正式课程不会用“DeepSeek 一路演进到 OCR”这种线性叙事,因为三条分支优化的是不同目标。
|
||||
|
||||
## 5. Kimi 候选分支:MoonViT 的三次训练制度变化
|
||||
|
||||
Grok 给出的候选主线:
|
||||
|
||||
```text
|
||||
Kimi-VL
|
||||
SigLIP 初始化
|
||||
+ 对比学习 / caption 视觉预训练
|
||||
+ MoonViT / NaViT 动态分辨率
|
||||
↓
|
||||
Kimi K2.5
|
||||
MoonViT-3D
|
||||
+ 图像 / 视频共享参数
|
||||
+ caption NTP,不再使用对比损失
|
||||
+ text-only SFT 激活视觉工具
|
||||
↓
|
||||
Kimi K3
|
||||
MoonViT-V2 从头训练
|
||||
+ 与语言主干共同做 NTP
|
||||
+ 无事后视觉对齐阶段
|
||||
+ vision-in-the-loop Agent
|
||||
```
|
||||
|
||||
这一主线已被选为正式章节的锚点,但每个框中的初始化、损失和训练阶段仍以 Kimi-VL、K2.5、K3 三份官方报告逐项核验。
|
||||
|
||||
## 6. 候选可视化
|
||||
|
||||
1. 像素 → patch → 视觉 Token → 压缩 → 上下文占比的水流图;
|
||||
2. 224²、1024²、3584² 在不同 patch size 下的 Token 爆炸对比;
|
||||
3. CLIP batch 内成对矩阵与 SigLIP 独立 sigmoid 对比;
|
||||
4. Frozen / Flamingo / BLIP-2 / LLaVA 连接器横截面;
|
||||
5. 固定缩放、动态切图、NaViT packing 的分辨率策略对比;
|
||||
6. image / video 的空间注意力与时间注意力分解;
|
||||
7. 理解、生成、工具行动三种“输出空间”;
|
||||
8. Janus 的双视觉编码路径与共享自回归主干;
|
||||
9. DeepSeek-VL → VL2 的高分辨率路线;
|
||||
10. DeepSeek-OCR 的“文字 Token → 页面图像 → 视觉 Token”压缩漏斗;
|
||||
11. DeepSeek-OCR2 的语义 token 重排;
|
||||
12. Kimi 三代 MoonViT 训练制度时间线;
|
||||
13. K3 原生多模态的数据—训练—Agent 闭环;
|
||||
14. 多模态幻觉的“语言先验压过视觉证据”跷跷板;
|
||||
15. 感知、OCR、推理、视频、Agent 五类评测矩阵。
|
||||
|
||||
## 7. 候选交互实验
|
||||
|
||||
### Lab A · 视觉 Token 预算
|
||||
|
||||
调节分辨率、patch size、帧数、时间池化、2×2 pixel shuffle 与上下文长度,观察原始 patch 数、压缩后 Token 数和上下文占比。
|
||||
|
||||
### Lab B · 连接器与训练阶段
|
||||
|
||||
切换 Flamingo、BLIP-2、LLaVA、Kimi-VL、K2.5、K3,观察哪些模块冻结、哪些目标生效、训练制度怎样变化。
|
||||
|
||||
### Lab C · 光学上下文压缩
|
||||
|
||||
把文档文本 Token 数、页面数和视觉 Token 数转成压缩率。DeepSeek-OCR 的曲线只以“论文报告值”显示;插值区域必须明确标成教学近似。
|
||||
|
||||
### Lab D · Vision-in-the-loop
|
||||
|
||||
比较直接 VQA、文字思维链和工具闭环:裁剪、放大、OCR、Python 图像处理、截图和最终验证分别改变什么观察。
|
||||
|
||||
## 8. Grok 暴露的二十个易错点
|
||||
|
||||
1. 视觉编码器不等于多模态模型;
|
||||
2. CLIP 的共享语义空间不等于生成式语言能力;
|
||||
3. projector 小不代表视觉信息传输没有瓶颈;
|
||||
4. “冻结 LLM”与“冻结视觉塔”不是同一个训练制度;
|
||||
5. cross-attention 与把视觉 Token 直接塞进序列不是同一种融合;
|
||||
6. native resolution 不等于没有 resize 或 patch 化;
|
||||
7. 动态切图会引入重复区域与 tile 顺序问题;
|
||||
8. 更高像素不必然产生更多有效信息;
|
||||
9. 视频不是“很多独立图片”的简单堆叠;
|
||||
10. 时间池化会省 Token,也会损失短暂事件;
|
||||
11. 统一主干不等于理解与生成共用同一个视觉编码器;
|
||||
12. 统一序列不等于统一损失;
|
||||
13. “原生多模态”没有唯一行业定义;
|
||||
14. K3 原生训练仍然保留 MoonViT-V2 与投影层;
|
||||
15. DeepSeek-VL2 的总参数不能与激活参数混用;
|
||||
16. DeepSeek-OCR 的压缩率和准确率是论文设定下的报告值;
|
||||
17. 多模态 RL 的提升不能自动归因于视觉推理;
|
||||
18. 工具调用提升可能来自 harness,而非裸模型;
|
||||
19. benchmark 总分会掩盖感知、OCR、知识与推理瓶颈;
|
||||
20. 图像中的间接提示注入会把视觉输入变成安全边界。
|
||||
|
||||
## 9. 正式筛选规则
|
||||
|
||||
- 核心论文必须改变一种表示、接口、训练目标、系统策略或评测方法;
|
||||
- 每个数字绑定模型版本、输入设置、训练或评测条件;
|
||||
- 官方报告数字写成“作者报告”,不写成本站复测;
|
||||
- 多模态 Agent 结果同时记录视觉观察、工具、harness、步数和 verifier;
|
||||
- 章节主线控制在约 50–60 个节点,外围候选进入论文图谱;
|
||||
- DeepSeek 与 Kimi 获得独立聚光灯,但不牺牲从 CLIP、连接器、高分辨率到原生训练的完整历史;
|
||||
- “原生”必须拆成原生数据混合、原生训练目标、原生主干优化、原生输入输出和原生 Agent 闭环五个可核验维度。
|
||||
File diff suppressed because it is too large
Load Diff
@@ -43,3 +43,14 @@ Agent 首轮缓存位于 `agents/`(不提交 PDF/TXT):
|
||||
复用其他专题缓存。十四张问题账、52 节点论文链、四个交互实验合同与证据边界见
|
||||
`../AGENTS_RESEARCH.md`;Grok Headless 只负责扩展召回,未核验候选永久隔离在
|
||||
`../AGENTS_GROK_LEADS.md`。
|
||||
|
||||
原生多模态首轮缓存位于 `multimodal/`(不提交 PDF/TXT):
|
||||
|
||||
- ResNet、ViT、ALIGN、CLIP、LiT、BLIP、SigLIP 与 NaViT;
|
||||
- DeepSeek-VL2、Janus-Pro、DeepSeek-OCR、DeepSeek-OCR2 与 Kimi-VL;
|
||||
- Kimi K2.5 与 K3 复用 `reasoning/`、`kimi-k3/` 中的官方技术报告缓存。
|
||||
|
||||
本轮本地完整校验 13 份 PDF,并通过官方 arXiv HTML、作者仓库和既有 K2.5/K3 报告核验其余核心节点。
|
||||
DeepSeek-VL/VL2、Janus、OCR 三分支,Kimi 三代 MoonViT、十六张问题账、55 节点正文链与四实验合同见
|
||||
`../MULTIMODAL_RESEARCH.md`;Grok Headless 候选召回只保存在
|
||||
`../MULTIMODAL_GROK_LEADS.md`,不得作为正式事实来源。
|
||||
|
||||
@@ -217,7 +217,7 @@ if (!overview.title.includes("可靠行动")) failures.push("章节标题异常"
|
||||
if (overview.sections !== 28 || overview.tocLinks !== 28) failures.push("章节/目录数量异常");
|
||||
if (overview.paperLinks !== 52) failures.push("正式论文链不是 52 个节点");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.navLinks !== 16 || home.navLinks !== 16 || mobile.mobileLinks !== 16) failures.push("全站导航未同步 Agent 专题");
|
||||
if (overview.navLinks !== 17 || home.navLinks !== 17 || mobile.mobileLinks !== 17) failures.push("全站导航未同步 Agent 专题");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||
if (loop.initial.finalState !== "UNVERIFIED" || loop.directSchema.finalState !== "FAILED") failures.push("控制循环终局状态异常");
|
||||
if (!loop.directSchema.observation.includes("ERROR schema") || loop.directSchema.recovery !== "FRAGILE") failures.push("Direct/schema 故障传播异常");
|
||||
@@ -228,8 +228,8 @@ if (numeric(reliability.initial.passAt) <= numeric(reliability.k2.passAt) || num
|
||||
if (reliability.nonIdempotent.sideRisk === "LOW") failures.push("非幂等写操作风险没有提升");
|
||||
if (numeric(rl.wait.utilization) >= numeric(rl.full.utilization) || numeric(rl.wait.lostWork) <= numeric(rl.full.lostWork)) failures.push("wait-all 长尾/重算方向异常");
|
||||
if (!rl.wait.takeaway.includes("wait-all") || rl.keyboardSelected !== "rl" || rl.keyboardVisible !== "rl") failures.push("长程 RL 解释或键盘导航异常");
|
||||
if (home.releaseCards !== 11 || !home.firstRelease.includes("Agent 不是一个循环") || home.firstHref !== "/agents/") failures.push("首页 Agent 首发入口异常");
|
||||
if (home.paperCount !== "314" || papers.total !== 314 || !papers.hasAgentFilter || papers.agentVisible < 52) failures.push("论文库 Agent 标签或论文总数异常");
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") failures.push("首页 Agent 首发入口异常");
|
||||
if (home.paperCount !== "355" || papers.total !== 355 || !papers.hasAgentFilter || papers.agentVisible < 52) failures.push("论文库 Agent 标签或论文总数异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (exceptions.length) failures.push(`浏览器异常:${exceptions.join(" | ")}`);
|
||||
|
||||
|
||||
@@ -216,7 +216,7 @@ if (!overview.title.includes("真正与人协作")) failures.push("章节标题
|
||||
if (overview.sections !== 22 || overview.tocLinks !== 22) failures.push("章节/目录数量异常");
|
||||
if (overview.paperLinks !== 44) failures.push("正式论文链不是 44 个节点");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.navLinks !== 16 || home.navLinks !== 16 || mobile.mobileLinks !== 16) failures.push("全站导航未同步后训练专题");
|
||||
if (overview.navLinks !== 17 || home.navLinks !== 17 || mobile.mobileLinks !== 17) failures.push("全站导航未同步后训练专题");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||
if (sft.initial.active !== "6 / 10" || !sft.initial.lossStates.slice(0, 4).every((value) => value === "MASKED")) failures.push("SFT response-only mask 异常");
|
||||
if (sft.unsafeAll.active !== "10 / 10" || numeric(sft.unsafeAll.nll) <= numeric(sft.initial.nll) || !sft.unsafeAll.reading.includes("错误")) failures.push("SFT 全序列/坏示范交互异常");
|
||||
@@ -226,8 +226,8 @@ if (!update.steps[0].includes("Fixed preference")) failures.push("DPO 更新流
|
||||
if (!recipe.family.includes("Multi-effort") || !recipe.regime.includes("9 RL experts") || !recipe.constraints.includes("verbosity")) failures.push("K3 配方合同异常");
|
||||
if (!recipe.path.some((step) => step.includes("3 domains × 3 efforts")) || !recipe.path.some((step) => step.includes("MOPD"))) failures.push("K3 配方路径异常");
|
||||
if (recipe.keyboardSelected !== "recipe" || recipe.keyboardVisible !== "recipe") failures.push("实验 tab 键盘导航异常");
|
||||
if (home.releaseCards !== 11 || !home.firstRelease.includes("Agent 不是一个循环") || home.firstHref !== "/agents/") failures.push("首页 Alignment 首发入口异常");
|
||||
if (home.paperCount !== "314" || papers.total !== 314 || !papers.hasAlignmentFilter || papers.alignmentVisible < 35) failures.push("论文库后训练标签或论文总数异常");
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") failures.push("首页 Alignment 首发入口异常");
|
||||
if (home.paperCount !== "355" || papers.total !== 355 || !papers.hasAlignmentFilter || papers.alignmentVisible < 35) failures.push("论文库后训练标签或论文总数异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (exceptions.length) failures.push(`浏览器异常:${exceptions.join(" | ")}`);
|
||||
|
||||
|
||||
@@ -230,15 +230,15 @@ if (transform.keyboard.selected !== "transform" || transform.keyboard.visible !=
|
||||
if (layout.articleSections !== 18 || layout.paperLinks !== 31 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||
failures.push("章节、论文或实验数量异常");
|
||||
}
|
||||
if (layout.navLinks !== 16 || mobile.mobileLinks !== 16 || home.navLinks !== 16) failures.push("全站导航未同步数值专题");
|
||||
if (layout.navLinks !== 17 || mobile.mobileLinks !== 17 || home.navLinks !== 17) failures.push("全站导航未同步数值专题");
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞:${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 11 || !home.firstRelease.includes("Agent 不是一个循环") || home.firstHref !== "/agents/") {
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") {
|
||||
failures.push("首页 Transformer 新章入口异常");
|
||||
}
|
||||
if (home.paperCount !== "314") failures.push(`首页论文总数异常:${home.paperCount}`);
|
||||
if (!papers.hasDataFilter || papers.total !== 314 || papers.visible < 25) failures.push("论文库数据标签或论文总数异常");
|
||||
if (home.paperCount !== "355") failures.push(`首页论文总数异常:${home.paperCount}`);
|
||||
if (!papers.hasDataFilter || papers.total !== 355 || papers.visible < 25) failures.push("论文库数据标签或论文总数异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常:${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -177,7 +177,7 @@ if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentO
|
||||
}
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞:${layout.navGap}px`);
|
||||
if (!mobile.menuVisible) failures.push("移动端菜单按钮未显示");
|
||||
if (home.releaseCards !== 11) failures.push(`首页新章卡数量异常:${home.releaseCards}`);
|
||||
if (home.releaseCards !== 12) failures.push(`首页新章卡数量异常:${home.releaseCards}`);
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常:${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
|
||||
const cdpPort = process.env.CDP_PORT ?? "9225";
|
||||
const baseUrl = process.env.SITE_URL ?? "http://127.0.0.1:4323";
|
||||
const pages = await fetch(`http://127.0.0.1:${cdpPort}/json/list`).then((response) => response.json());
|
||||
const page = pages.find((entry) => entry.type === "page");
|
||||
if (!page) throw new Error(`CDP ${cdpPort} 没有可用页面`);
|
||||
|
||||
const socket = new WebSocket(page.webSocketDebuggerUrl);
|
||||
await new Promise((resolve, reject) => {
|
||||
socket.addEventListener("open", resolve, { once: true });
|
||||
socket.addEventListener("error", reject, { once: true });
|
||||
});
|
||||
|
||||
let nextId = 0;
|
||||
const pending = new Map();
|
||||
const exceptions = [];
|
||||
socket.addEventListener("message", (event) => {
|
||||
const message = JSON.parse(event.data);
|
||||
if (message.id && pending.has(message.id)) {
|
||||
const { resolve, reject } = pending.get(message.id);
|
||||
pending.delete(message.id);
|
||||
if (message.error) reject(new Error(message.error.message));
|
||||
else resolve(message.result);
|
||||
}
|
||||
if (message.method === "Runtime.exceptionThrown") {
|
||||
exceptions.push(message.params.exceptionDetails.exception?.description ?? message.params.exceptionDetails.text);
|
||||
}
|
||||
});
|
||||
|
||||
const command = (method, params = {}) => new Promise((resolve, reject) => {
|
||||
const id = ++nextId;
|
||||
pending.set(id, { resolve, reject });
|
||||
socket.send(JSON.stringify({ id, method, params }));
|
||||
});
|
||||
const pause = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||
const evaluate = async (expression) => {
|
||||
const result = await command("Runtime.evaluate", { expression, returnByValue: true, awaitPromise: true });
|
||||
if (result.exceptionDetails) throw new Error(result.exceptionDetails.exception?.description ?? result.exceptionDetails.text);
|
||||
return result.result.value;
|
||||
};
|
||||
const navigate = async (path) => {
|
||||
await command("Page.navigate", { url: `${baseUrl}${path}` });
|
||||
for (let attempt = 0; attempt < 70; attempt += 1) {
|
||||
await pause(100);
|
||||
if (await evaluate("document.readyState === 'complete'")) return;
|
||||
}
|
||||
throw new Error(`${path} 加载超时`);
|
||||
};
|
||||
const screenshot = async (path) => {
|
||||
const result = await command("Page.captureScreenshot", {
|
||||
format: "png",
|
||||
captureBeyondViewport: false,
|
||||
});
|
||||
writeFileSync(path, Buffer.from(result.data, "base64"));
|
||||
};
|
||||
|
||||
await command("Page.enable");
|
||||
await command("Runtime.enable");
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 1440,
|
||||
height: 1100,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: false,
|
||||
});
|
||||
await navigate("/multimodal/");
|
||||
await screenshot("/tmp/llm-atlas-multimodal-desktop.png");
|
||||
|
||||
const overview = await evaluate(`(() => ({
|
||||
title: document.querySelector("h1")?.textContent.trim(),
|
||||
sections: document.querySelectorAll(".article-section").length,
|
||||
tocLinks: document.querySelectorAll(".side-rail a").length,
|
||||
paperLinks: document.querySelectorAll(".paper-chain a").length,
|
||||
labTabs: document.querySelectorAll("[data-mm-tab]").length,
|
||||
labPanels: document.querySelectorAll("[data-mm-panel]").length,
|
||||
navLinks: document.querySelectorAll(".top-nav a").length,
|
||||
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||
}))()`);
|
||||
|
||||
const tokens = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-mm-lab]");
|
||||
const read = () => ({
|
||||
patches: root.querySelector("[data-token-patches]").textContent.trim(),
|
||||
visual: root.querySelector("[data-token-visual]").textContent.trim(),
|
||||
share: root.querySelector("[data-token-share]").textContent.trim(),
|
||||
status: root.querySelector("[data-token-status]").textContent.trim(),
|
||||
visiblePanel: root.querySelector("[data-mm-panel]:not([hidden])").dataset.mmPanel,
|
||||
});
|
||||
const initial = read();
|
||||
const resolution = root.querySelector("[data-token-resolution]");
|
||||
const frames = root.querySelector("[data-token-frames]");
|
||||
const context = root.querySelector("[data-token-context]");
|
||||
resolution.value = "7";
|
||||
resolution.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
frames.value = "32";
|
||||
frames.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
context.value = "8192";
|
||||
context.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
root.querySelector('[data-spatial="1"]').click();
|
||||
const overloaded = read();
|
||||
return { initial, overloaded };
|
||||
})()`);
|
||||
|
||||
const connector = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-mm-lab]");
|
||||
root.querySelector('[data-mm-tab="connector"]').click();
|
||||
const read = () => ({
|
||||
visual: root.querySelector("[data-connector-visual]").textContent.trim(),
|
||||
alignment: root.querySelector("[data-connector-alignment]").textContent.trim(),
|
||||
score: root.querySelector("[data-native-score]").textContent.trim(),
|
||||
activeNative: root.querySelectorAll("[data-native-item].active").length,
|
||||
visiblePanel: root.querySelector("[data-mm-panel]:not([hidden])").dataset.mmPanel,
|
||||
});
|
||||
root.querySelector('[data-connector="k3"]').click();
|
||||
const k3 = read();
|
||||
root.querySelector('[data-connector="kimivl"]').click();
|
||||
const kimiVl = read();
|
||||
return { k3, kimiVl };
|
||||
})()`);
|
||||
|
||||
const ocr = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-mm-lab]");
|
||||
root.querySelector('[data-mm-tab="ocr"]').click();
|
||||
const curve = root.querySelector("[data-ocr-curve]");
|
||||
const read = () => ({
|
||||
ratio: root.querySelector("[data-ocr-ratio]").textContent.trim(),
|
||||
accuracy: root.querySelector("[data-ocr-accuracy]").textContent.trim(),
|
||||
evidence: root.querySelector("[data-ocr-evidence]").textContent.trim(),
|
||||
status: root.querySelector("[data-ocr-status]").textContent.trim(),
|
||||
});
|
||||
const boundary = read();
|
||||
curve.value = "15";
|
||||
curve.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const interpolated = read();
|
||||
curve.value = "23";
|
||||
curve.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
const unreported = read();
|
||||
return { boundary, interpolated, unreported };
|
||||
})()`);
|
||||
|
||||
const loop = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-mm-lab]");
|
||||
root.querySelector('[data-mm-tab="loop"]').click();
|
||||
const read = () => ({
|
||||
evidence: root.querySelector("[data-loop-evidence]").textContent.trim(),
|
||||
tools: root.querySelector("[data-loop-tools]").textContent.trim(),
|
||||
state: root.querySelector("[data-loop-state]").textContent.trim(),
|
||||
type: root.querySelector("[data-loop-type]").textContent.trim(),
|
||||
takeaway: root.querySelector("[data-loop-takeaway]").textContent.trim(),
|
||||
});
|
||||
const toolsStart = read();
|
||||
const next = root.querySelector("[data-loop-next]");
|
||||
for (let index = 0; index < 4; index += 1) next.click();
|
||||
const toolsEnd = read();
|
||||
root.querySelector('[data-loop-mode="cot"]').click();
|
||||
next.click();
|
||||
next.click();
|
||||
const cotEnd = read();
|
||||
const firstTab = root.querySelector('[data-mm-tab="tokens"]');
|
||||
firstTab.focus();
|
||||
firstTab.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowRight", bubbles: true }));
|
||||
return {
|
||||
toolsStart,
|
||||
toolsEnd,
|
||||
cotEnd,
|
||||
keyboardSelected: root.querySelector('[data-mm-tab][aria-selected="true"]').dataset.mmTab,
|
||||
keyboardVisible: root.querySelector("[data-mm-panel]:not([hidden])").dataset.mmPanel,
|
||||
};
|
||||
})()`);
|
||||
|
||||
await evaluate(`document.querySelector("[data-mm-lab]").scrollIntoView({ block: "start", behavior: "instant" })`);
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-multimodal-lab-desktop.png");
|
||||
|
||||
await navigate("/");
|
||||
const home = await evaluate(`(() => ({
|
||||
releaseCards: document.querySelectorAll(".release-card").length,
|
||||
firstRelease: document.querySelector(".release-card h2").textContent.trim(),
|
||||
firstHref: document.querySelector(".release-card").getAttribute("href"),
|
||||
paperCount: document.querySelector(".hero-stats div:nth-child(3) b").textContent.trim(),
|
||||
navLinks: document.querySelectorAll(".top-nav a").length,
|
||||
}))()`);
|
||||
|
||||
await navigate("/papers/");
|
||||
const papers = await evaluate(`(() => {
|
||||
const button = [...document.querySelectorAll("[data-filter]")].find((node) => node.textContent.trim() === "多模态");
|
||||
button?.click();
|
||||
return {
|
||||
total: document.querySelectorAll("[data-paper]").length,
|
||||
multimodalVisible: document.querySelectorAll("[data-paper]:not([hidden])").length,
|
||||
hasFilter: Boolean(button),
|
||||
};
|
||||
})()`);
|
||||
|
||||
await command("Emulation.setDeviceMetricsOverride", {
|
||||
width: 390,
|
||||
height: 844,
|
||||
deviceScaleFactor: 1,
|
||||
mobile: true,
|
||||
});
|
||||
await navigate("/multimodal/");
|
||||
const mobile = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-mm-lab]");
|
||||
root.scrollIntoView({ block: "start", behavior: "instant" });
|
||||
const toggle = document.querySelector("#menu-toggle");
|
||||
toggle?.click();
|
||||
return {
|
||||
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||
menuVisible: getComputedStyle(toggle).display !== "none",
|
||||
menuOpen: toggle.getAttribute("aria-expanded"),
|
||||
mobileLinks: document.querySelectorAll("#mobile-nav a").length,
|
||||
tabs: root.querySelectorAll("[data-mm-tab]").length,
|
||||
offenders: [...document.querySelectorAll("body *")]
|
||||
.filter((node) => !node.closest(".eval-matrix"))
|
||||
.filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1)
|
||||
.slice(0, 12)
|
||||
.map((node) => ({
|
||||
tag: node.tagName,
|
||||
className: typeof node.className === "string" ? node.className : "",
|
||||
right: Math.round(node.getBoundingClientRect().right),
|
||||
width: Math.round(node.getBoundingClientRect().width),
|
||||
})),
|
||||
};
|
||||
})()`);
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-multimodal-mobile.png");
|
||||
|
||||
const report = { overview, tokens, connector, ocr, loop, home, papers, mobile, exceptions };
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
|
||||
const numeric = (value) => Number.parseFloat(value.replaceAll(",", ""));
|
||||
const failures = [];
|
||||
if (!overview.title.includes("第一类输入")) failures.push("章节标题异常");
|
||||
if (overview.sections !== 30 || overview.tocLinks !== 30) failures.push("章节/目录数量异常");
|
||||
if (overview.paperLinks !== 55) failures.push("正式论文链不是 55 个节点");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.navLinks !== 17 || home.navLinks !== 17 || mobile.mobileLinks !== 17) failures.push("全站导航未同步多模态专题");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||
if (numeric(tokens.initial.patches) !== 5476 || numeric(tokens.initial.visual) !== 1369 || tokens.initial.visiblePanel !== "tokens") failures.push("视觉 Token 初始计算异常");
|
||||
if (!tokens.overloaded.status.includes("超预算") || numeric(tokens.overloaded.share) <= 100) failures.push("超大视频没有触发上下文超预算");
|
||||
if (!connector.k3.visual.includes("从头") || !connector.k3.alignment.includes("无 post-hoc") || connector.k3.score !== "5 / 5" || connector.k3.activeNative !== 5) failures.push("K3 训练制度预设异常");
|
||||
if (connector.kimiVl.score !== "4 / 5" || connector.kimiVl.activeNative !== 4 || connector.k3.visiblePanel !== "connector") failures.push("Kimi-VL 原生五维预设异常");
|
||||
if (ocr.boundary.status !== "BOUNDARY" || Math.abs(numeric(ocr.boundary.ratio) - 7.81) > 0.01) failures.push("OCR 初始压缩边界异常");
|
||||
if (ocr.interpolated.status !== "TEACHING INTERPOLATION" || !ocr.interpolated.evidence.includes("教学插值")) failures.push("OCR 教学插值没有显式标记");
|
||||
if (ocr.unreported.status !== "OUT OF EVIDENCE" || ocr.unreported.accuracy !== "未报告") failures.push("OCR 超证据区仍在外推");
|
||||
if (loop.toolsStart.state !== "OPEN" || loop.toolsEnd.state !== "VERIFIED" || loop.toolsEnd.evidence !== "97%" || loop.toolsEnd.tools !== "3") failures.push("vision-in-the-loop 终局异常");
|
||||
if (loop.cotEnd.state !== "FAILED" || !loop.cotEnd.takeaway.includes("不能凭空增加")) failures.push("文字 CoT 与新观察没有分开");
|
||||
if (loop.keyboardSelected !== "connector" || loop.keyboardVisible !== "connector") failures.push("实验键盘 tab 导航异常");
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") failures.push("首页多模态首发入口异常");
|
||||
if (home.paperCount !== "355" || papers.total !== 355 || !papers.hasFilter || papers.multimodalVisible < 59) failures.push("论文库多模态标签或总数异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (mobile.offenders.length) failures.push(`移动端越界元素:${JSON.stringify(mobile.offenders)}`);
|
||||
if (exceptions.length) failures.push(`浏览器异常:${exceptions.join(" | ")}`);
|
||||
|
||||
if (failures.length) {
|
||||
console.error(`\nFAIL\n- ${failures.join("\n- ")}`);
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
console.log("\nPASS multimodal browser regression");
|
||||
}
|
||||
|
||||
socket.close();
|
||||
@@ -273,15 +273,15 @@ if (stability.keyboard.selected !== "stability" || stability.keyboard.visible !=
|
||||
if (layout.articleSections !== 19 || layout.paperLinks !== 36 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||
failures.push("章节、论文或实验数量异常");
|
||||
}
|
||||
if (layout.navLinks !== 16 || mobile.mobileLinks !== 16 || home.navLinks !== 16) failures.push("全站导航未同步数值专题");
|
||||
if (layout.navLinks !== 17 || mobile.mobileLinks !== 17 || home.navLinks !== 17) failures.push("全站导航未同步数值专题");
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞:${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 11 || !home.firstRelease.includes("Agent 不是一个循环") || home.firstHref !== "/agents/") {
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") {
|
||||
failures.push("首页 Transformer 新章入口异常");
|
||||
}
|
||||
if (home.paperCount !== "314") failures.push(`首页论文总数异常:${home.paperCount}`);
|
||||
if (!papers.hasOptimizerFilter || papers.total !== 314 || papers.visible < 8) failures.push("论文库优化器标签或论文总数异常");
|
||||
if (home.paperCount !== "355") failures.push(`首页论文总数异常:${home.paperCount}`);
|
||||
if (!papers.hasOptimizerFilter || papers.total !== 355 || papers.visible < 8) failures.push("论文库优化器标签或论文总数异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常:${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -289,7 +289,7 @@ if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentO
|
||||
}
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞:${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 11 || !home.firstRelease.includes("Agent 不是一个循环")) failures.push("首页 Transformer 新章入口异常");
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态")) failures.push("首页 Transformer 新章入口异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常:${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -269,14 +269,14 @@ if (emergence.paths.some((length) => length < 500)) failures.push("涌现多指
|
||||
if (layout.articleSections !== 16 || layout.paperLinks !== 29 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||
failures.push("章节、论文或实验数量异常");
|
||||
}
|
||||
if (layout.navLinks !== 16 || mobile.mobileLinks !== 16 || home.navLinks !== 16) failures.push("全站导航未同步数值专题");
|
||||
if (layout.navLinks !== 17 || mobile.mobileLinks !== 17 || home.navLinks !== 17) failures.push("全站导航未同步数值专题");
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞:${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 11 || !home.firstRelease.includes("Agent 不是一个循环") || home.firstHref !== "/agents/") {
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") {
|
||||
failures.push("首页 Transformer 新章入口异常");
|
||||
}
|
||||
if (home.paperCount !== "314") failures.push(`首页论文总数异常:${home.paperCount}`);
|
||||
if (home.paperCount !== "355") failures.push(`首页论文总数异常:${home.paperCount}`);
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常:${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -233,7 +233,7 @@ if (layout.articleSections !== 16 || layout.paperLinks !== 37 || layout.labTabs
|
||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞:${layout.navGap}px`);
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||
if (home.releaseCards !== 11 || !home.firstRelease.includes("Agent 不是一个循环")) failures.push("首页 Transformer 新章入口异常");
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态")) failures.push("首页 Transformer 新章入口异常");
|
||||
if (exceptions.length) failures.push(`浏览器脚本异常:${exceptions.join("; ")}`);
|
||||
|
||||
socket.close();
|
||||
|
||||
@@ -172,7 +172,7 @@ const home = await evaluate(`(() => ({
|
||||
releaseCards: document.querySelectorAll(".release-card").length,
|
||||
firstRelease: document.querySelector(".release-card h2").textContent,
|
||||
firstHref: document.querySelector(".release-card").getAttribute("href"),
|
||||
paperCount: [...document.querySelectorAll(".hero-stats b")].map((node) => node.textContent.trim()).find((value) => value === "314"),
|
||||
paperCount: [...document.querySelectorAll(".hero-stats b")].map((node) => node.textContent.trim()).find((value) => value === "355"),
|
||||
}))()`);
|
||||
|
||||
await navigate("/papers/");
|
||||
@@ -236,8 +236,8 @@ if (block.family.trim() !== "Hybrid MoE" || !block.kv.includes("3 KDA : 1 Gated
|
||||
if (!block.path.some((step) => step.includes("KDA × 3")) || !block.note.includes("AttnRes")) failures.push("K3 Block 路径异常");
|
||||
if (block.context.trim() !== "128K" || numeric(block.mha) !== 400 || numeric(block.kda) !== 1) failures.push("KV 成本缩放异常");
|
||||
if (block.keyboardSelected !== "block" || block.keyboardVisible !== "block") failures.push("实验 tab 键盘导航异常");
|
||||
if (home.releaseCards !== 11 || !home.firstRelease.includes("Agent 不是一个循环") || home.firstHref !== "/agents/") failures.push("首页 Transformer 首发入口异常");
|
||||
if (home.paperCount !== "314" || papers.total !== 314 || papers.transformerVisible < 30) failures.push("论文库或首页论文数量异常");
|
||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") failures.push("首页 Transformer 首发入口异常");
|
||||
if (home.paperCount !== "355" || papers.total !== 355 || papers.transformerVisible < 30) failures.push("论文库或首页论文数量异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||
if (exceptions.length) failures.push(`浏览器异常:${exceptions.join(" | ")}`);
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -17,6 +17,7 @@ const items = [
|
||||
{ id: "alignment", href: "/post-training/alignment/", label: "后训练" },
|
||||
{ id: "reasoning", href: "/reasoning/", label: "推理" },
|
||||
{ id: "agents", href: "/agents/", label: "Agent" },
|
||||
{ id: "multimodal", href: "/multimodal/", label: "多模态" },
|
||||
{ id: "training-systems", href: "/training-systems/", label: "训练系统" },
|
||||
{ id: "numerics", href: "/systems/numerics/", label: "数值" },
|
||||
{ id: "papers", href: "/papers/", label: "论文库" },
|
||||
|
||||
@@ -190,12 +190,12 @@ export const chapters: Chapter[] = [
|
||||
title: "原生多模态",
|
||||
kicker: "MULTIMODAL",
|
||||
question: "图像是外接插件,还是和文字一样的第一类输入?",
|
||||
summary: "从 ViT/CLIP、连接器式 VLM 走到 Kimi-VL、MoonViT-V2 与统一主干。",
|
||||
status: "queued",
|
||||
progress: 11,
|
||||
papers: 19,
|
||||
summary: "用十六张账串起视觉 Token、CLIP、连接器、高分辨率、OCR、视频、统一理解生成,并重点拆解 DeepSeek 三分支与 Kimi 三代 MoonViT。",
|
||||
status: "published",
|
||||
progress: 76,
|
||||
papers: 55,
|
||||
prerequisites: ["02"],
|
||||
highlights: ["ViT", "视觉 Token", "MoonViT-V2"],
|
||||
highlights: ["DeepSeek 三分支", "Kimi 三代 MoonViT", "四联实验"],
|
||||
},
|
||||
{
|
||||
number: "14",
|
||||
|
||||
@@ -2416,6 +2416,22 @@ export const papers: Paper[] = [
|
||||
contribution: "面向 Agentic AI 的 Firecracker microVM 环境,支持 pause、resume、fork、snapshot 与高密度运行。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2012,
|
||||
title: "ImageNet Classification with Deep Convolutional Neural Networks",
|
||||
url: "https://proceedings.neurips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks",
|
||||
topics: ["多模态"],
|
||||
contribution: "AlexNet 以 GPU、大规模卷积网络与 ImageNet 监督推动可扩展视觉表示。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2015,
|
||||
title: "Deep Residual Learning for Image Recognition",
|
||||
url: "https://arxiv.org/abs/1512.03385",
|
||||
topics: ["多模态", "Transformer"],
|
||||
contribution: "残差连接让更深视觉骨干稳定优化,成为多模态视觉塔的重要前史。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2020,
|
||||
title: "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale",
|
||||
@@ -2424,6 +2440,14 @@ export const papers: Paper[] = [
|
||||
contribution: "ViT 把图像 patch 视为 Token 输入 Transformer。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision",
|
||||
url: "https://arxiv.org/abs/2102.05918",
|
||||
topics: ["多模态", "数据", "Scaling"],
|
||||
contribution: "ALIGN 用超大规模噪声 web 图文对训练双编码器,展示数据规模路线。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "Learning Transferable Visual Models From Natural Language Supervision",
|
||||
@@ -2432,6 +2456,30 @@ export const papers: Paper[] = [
|
||||
contribution: "CLIP 以海量图文对比学习建立可迁移视觉语义空间。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "Multimodal Few-Shot Learning with Frozen Language Models",
|
||||
url: "https://arxiv.org/abs/2106.13884",
|
||||
topics: ["多模态"],
|
||||
contribution: "Frozen 用视觉编码器生成连续前缀,连接冻结语言模型并激活多模态 in-context learning。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2021,
|
||||
title: "LiT: Zero-Shot Transfer with Locked-image Text Tuning",
|
||||
url: "https://arxiv.org/abs/2111.07991",
|
||||
topics: ["多模态", "后训练"],
|
||||
contribution: "锁定强图像塔,只训练文本侧去匹配既有视觉空间。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation",
|
||||
url: "https://arxiv.org/abs/2201.12086",
|
||||
topics: ["多模态", "数据"],
|
||||
contribution: "用 captioner 生成描述、filter 清理噪声,同时服务视觉理解与生成。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2022,
|
||||
title: "Flamingo: a Visual Language Model for Few-Shot Learning",
|
||||
@@ -2448,6 +2496,134 @@ export const papers: Paper[] = [
|
||||
contribution: "用轻量 Q-Former 桥接冻结视觉编码器与 LLM。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models",
|
||||
url: "https://arxiv.org/abs/2304.10592",
|
||||
topics: ["多模态", "后训练"],
|
||||
contribution: "用少量高质量视觉对话对齐冻结视觉编码器与冻结语言模型。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning",
|
||||
url: "https://arxiv.org/abs/2305.06500",
|
||||
topics: ["多模态", "后训练"],
|
||||
contribution: "让 Q-Former 感知指令,并系统研究跨任务视觉指令泛化。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "PaLI-X: On Scaling up a Multilingual Vision and Language Model",
|
||||
url: "https://arxiv.org/abs/2305.18565",
|
||||
topics: ["多模态", "Scaling"],
|
||||
contribution: "从组件规模与多语言任务混合两轴扩展视觉—语言模型。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models",
|
||||
url: "https://arxiv.org/abs/2305.07895",
|
||||
topics: ["多模态", "评测"],
|
||||
contribution: "联合评测文字识别、场景文字 VQA、文档 VQA、KIE 与手写公式。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Evaluating Object Hallucination in Large Vision-Language Models",
|
||||
url: "https://arxiv.org/abs/2305.10355",
|
||||
topics: ["多模态", "评测"],
|
||||
contribution: "POPE 用轮询式对象存在性问题更稳定地诊断视觉幻觉。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Kosmos-2: Grounding Multimodal Large Language Models to the World",
|
||||
url: "https://arxiv.org/abs/2306.14824",
|
||||
topics: ["多模态"],
|
||||
contribution: "用 location token 和 grounded image-text pairs 把文本指称绑定到视觉区域。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution",
|
||||
url: "https://arxiv.org/abs/2307.06304",
|
||||
topics: ["多模态", "训练系统"],
|
||||
contribution: "在固定 Token 预算内打包不同分辨率与长宽比图像的 patch 序列。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "MMBench: Is Your Multi-modal Model an All-around Player?",
|
||||
url: "https://arxiv.org/abs/2307.06281",
|
||||
topics: ["多模态", "评测"],
|
||||
contribution: "用双语多选、能力分层与 CircularEval 建立综合 VLM 评测。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond",
|
||||
url: "https://arxiv.org/abs/2308.12966",
|
||||
topics: ["多模态"],
|
||||
contribution: "把开放视觉对话、OCR 与 grounding 能力整合进 Qwen 视觉语言路线。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Improved Baselines with Visual Instruction Tuning",
|
||||
url: "https://arxiv.org/abs/2310.03744",
|
||||
topics: ["多模态", "后训练"],
|
||||
contribution: "LLaVA-1.5 用两层 MLP、更多任务数据与清晰训练配方强化简单基线。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts",
|
||||
url: "https://arxiv.org/abs/2310.02255",
|
||||
topics: ["多模态", "推理", "评测"],
|
||||
contribution: "用图表、几何、函数与视觉数学题联合考察感知和多步推理。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion",
|
||||
url: "https://arxiv.org/abs/2310.14566",
|
||||
topics: ["多模态", "评测"],
|
||||
contribution: "用控制组问题区分语言幻觉、视觉错觉、回答倾向与逻辑一致性。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Video-LLaVA: Learning United Visual Representation by Alignment Before Projection",
|
||||
url: "https://arxiv.org/abs/2311.10122",
|
||||
topics: ["多模态"],
|
||||
contribution: "在投影前对齐图像和视频表示,探索共同视觉—语言空间。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI",
|
||||
url: "https://arxiv.org/abs/2311.16502",
|
||||
topics: ["多模态", "推理", "评测"],
|
||||
contribution: "以 11.5K 大学级问题覆盖六大学科、183 子领域与 30 种图像类型。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models",
|
||||
url: "https://arxiv.org/abs/2311.17043",
|
||||
topics: ["多模态", "长上下文"],
|
||||
contribution: "用内容 Token 与上下文 Token 压缩长视频视觉序列。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback",
|
||||
url: "https://arxiv.org/abs/2312.00849",
|
||||
topics: ["多模态", "后训练", "评测"],
|
||||
contribution: "以 segment-level 人工纠错和 dense DPO 对齐视觉忠实度。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2023,
|
||||
title: "Visual Instruction Tuning",
|
||||
@@ -2464,6 +2640,55 @@ export const papers: Paper[] = [
|
||||
contribution: "SigLIP 以成对 sigmoid loss 简化大规模图文对比。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "DeepSeek-VL: Towards Real-World Vision-Language Understanding",
|
||||
url: "https://arxiv.org/abs/2403.05525",
|
||||
topics: ["多模态", "数据"],
|
||||
contribution: "面向截图、PDF、OCR 与图表,用混合视觉编码器处理高分辨率并管理模态竞争。",
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites",
|
||||
url: "https://arxiv.org/abs/2404.16821",
|
||||
topics: ["多模态", "Scaling"],
|
||||
contribution: "InternVL 1.5 结合 6B 视觉塔、最多 40 个动态 tile 与高质量双语数据。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Chameleon: Mixed-Modal Early-Fusion Foundation Models",
|
||||
url: "https://arxiv.org/abs/2405.09818",
|
||||
topics: ["多模态", "Transformer"],
|
||||
contribution: "把图像与文字离散成可交错 Token,用 early-fusion 自回归主干统一建模。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis",
|
||||
url: "https://arxiv.org/abs/2405.21075",
|
||||
topics: ["多模态", "评测"],
|
||||
contribution: "用 900 段短中长视频、字幕和音频条件系统评估视频理解。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models",
|
||||
url: "https://arxiv.org/abs/2406.09403",
|
||||
topics: ["多模态", "推理", "Agent"],
|
||||
contribution: "让模型画线、框选并调用视觉工具,以新视觉产物继续推理。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "SOLO: A Single Transformer for Scalable Vision-Language Modeling",
|
||||
url: "https://arxiv.org/abs/2407.06438",
|
||||
topics: ["多模态", "Transformer", "Scaling"],
|
||||
contribution: "探索单一 Transformer 视觉—语言架构与可稳定训练的开放配方。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "LLaVA-OneVision: Easy Visual Task Transfer",
|
||||
@@ -2472,6 +2697,82 @@ export const papers: Paper[] = [
|
||||
contribution: "统一单图、多图与视频的视觉指令迁移。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model",
|
||||
url: "https://arxiv.org/abs/2408.11039",
|
||||
topics: ["多模态", "Transformer"],
|
||||
contribution: "在同一 Transformer 上对文字做 NTP、对连续图像表示做 diffusion。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution",
|
||||
url: "https://arxiv.org/abs/2409.12191",
|
||||
topics: ["多模态", "长上下文"],
|
||||
contribution: "引入动态视觉 Token、M-RoPE 与统一图像 / 视频输入。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation",
|
||||
url: "https://arxiv.org/abs/2410.13848",
|
||||
topics: ["多模态"],
|
||||
contribution: "解耦理解与生成的视觉编码路径,同时共享一个自回归主干。",
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation",
|
||||
url: "https://arxiv.org/abs/2411.07975",
|
||||
topics: ["多模态"],
|
||||
contribution: "把语言自回归与图像 rectified flow 放进同一框架,并对齐解耦视觉表示。",
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling",
|
||||
url: "https://arxiv.org/abs/2412.05271",
|
||||
topics: ["多模态", "Scaling", "推理"],
|
||||
contribution: "InternVL 2.5 从模型、数据与测试时配置三个轴研究多模态扩展。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2024,
|
||||
title: "DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding",
|
||||
url: "https://arxiv.org/abs/2412.10302",
|
||||
topics: ["多模态", "MoE", "长上下文"],
|
||||
contribution: "用动态 tile、2×2 pixel shuffle、MLP adaptor、DeepSeekMoE 与 MLA 处理高分辨率输入。",
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling",
|
||||
url: "https://arxiv.org/abs/2501.17811",
|
||||
topics: ["多模态", "Scaling"],
|
||||
contribution: "延续 Janus 解耦视觉路径,改进训练策略、数据规模与模型规模。",
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "Qwen2.5-VL Technical Report",
|
||||
url: "https://arxiv.org/abs/2502.13923",
|
||||
topics: ["多模态", "Agent"],
|
||||
contribution: "从头训练动态分辨率 ViT,引入绝对时间编码并强化文档和视觉 Agent。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models",
|
||||
url: "https://arxiv.org/abs/2503.06749",
|
||||
topics: ["多模态", "推理", "后训练"],
|
||||
contribution: "探索用大规模强化学习激活多模态长思维与视觉推理。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "Kimi-VL Technical Report",
|
||||
@@ -2481,6 +2782,40 @@ export const papers: Paper[] = [
|
||||
spotlight: "Kimi",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use",
|
||||
url: "https://arxiv.org/abs/2505.19255",
|
||||
topics: ["多模态", "推理", "Agent", "后训练"],
|
||||
contribution: "用可验证奖励训练模型自主选择视觉工具并从新图像观察继续推理。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2025,
|
||||
title: "DeepSeek-OCR: Contexts Optical Compression",
|
||||
url: "https://arxiv.org/abs/2510.18234",
|
||||
topics: ["多模态", "长上下文", "数据"],
|
||||
contribution: "用 DeepEncoder 将二维页面压成少量视觉 Token,研究光学长上下文压缩。",
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2026,
|
||||
title: "DeepSeek-OCR 2: Visual Causal Flow",
|
||||
url: "https://arxiv.org/abs/2601.20552",
|
||||
topics: ["多模态", "长上下文"],
|
||||
contribution: "DeepEncoder V2 根据图像语义动态重排视觉 Token,再交给语言解码器。",
|
||||
spotlight: "DeepSeek",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2026,
|
||||
title: "ToolsRL: Reward Is All Tool Learning Needs",
|
||||
url: "https://arxiv.org/abs/2604.19945",
|
||||
topics: ["多模态", "推理", "Agent", "后训练"],
|
||||
contribution: "以强化学习研究视觉工具选择、调用与多步观察闭环。",
|
||||
verified: true,
|
||||
},
|
||||
{
|
||||
year: 2020,
|
||||
title: "Measuring Massive Multitask Language Understanding",
|
||||
|
||||
+19
-1
@@ -15,6 +15,7 @@ const routes: Record<string, string> = {
|
||||
"post-training/alignment": "/post-training/alignment/",
|
||||
reasoning: "/reasoning/",
|
||||
agents: "/agents/",
|
||||
multimodal: "/multimodal/",
|
||||
"training-systems": "/training-systems/",
|
||||
"systems/numerics": "/systems/numerics/",
|
||||
};
|
||||
@@ -93,6 +94,7 @@ const paths = [
|
||||
<a class="button" href="/post-training/alignment/">后训练与偏好专题</a>
|
||||
<a class="button" href="/reasoning/">推理专题</a>
|
||||
<a class="button" href="/agents/">Agent 专题</a>
|
||||
<a class="button" href="/multimodal/">原生多模态专题</a>
|
||||
<a class="button" href="/training-systems/">训练系统专题</a>
|
||||
<a class="button" href="/systems/numerics/">数值与优化专题</a>
|
||||
</div>
|
||||
@@ -104,7 +106,7 @@ const paths = [
|
||||
<div class="hero-stats">
|
||||
<div><b>16</b><span>核心专题</span></div>
|
||||
<div><b>151</b><span>K3 报告来源</span></div>
|
||||
<div><b>314</b><span>关键论文索引</span></div>
|
||||
<div><b>355</b><span>关键论文索引</span></div>
|
||||
<div><b>47p</b><span>K3 技术报告</span></div>
|
||||
</div>
|
||||
</aside>
|
||||
@@ -118,6 +120,22 @@ const paths = [
|
||||
|
||||
<section class="section compact release-section" id="new-chapters">
|
||||
<div class="release-grid">
|
||||
<a class="release-card multimodal-release" href="/multimodal/">
|
||||
<div>
|
||||
<p class="eyebrow"><span>NEW / CHAPTER 13</span> PIXELS · TOKENS · NATIVE MULTIMODALITY</p>
|
||||
<h2>原生多模态不等于移除视觉塔:真正改变的是数据、目标、优化与行为闭环</h2>
|
||||
<p>
|
||||
用语义空间、连接器、分辨率、OCR、视频、训练、幻觉、评测与 Agent 十六张账,
|
||||
从 ViT、CLIP 和 LLaVA 走到 DeepSeek-VL / Janus / OCR 三分支,以及 Kimi 三代 MoonViT 与 K3 从头共同 NTP。
|
||||
</p>
|
||||
</div>
|
||||
<dl>
|
||||
<div><dt>LINEAGE</dt><dd>2012 → 2026</dd></div>
|
||||
<div><dt>NODES</dt><dd>55 个一手节点</dd></div>
|
||||
<div><dt>LAB</dt><dd>Token · 连接器 · 光学压缩 · 视觉闭环</dd></div>
|
||||
</dl>
|
||||
<span class="release-arrow" aria-hidden="true">从像素进入可验证视觉行动 →</span>
|
||||
</a>
|
||||
<a class="release-card agent-release" href="/agents/">
|
||||
<div>
|
||||
<p class="eyebrow"><span>NEW / CHAPTER 12</span> AGENTS · TOOL USE · ENVIRONMENTS</p>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -18,6 +18,7 @@ const workstreams = [
|
||||
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
||||
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
||||
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
||||
{ label: "原生多模态", value: 76, next: "补真实视觉 Token traces、跨分辨率消融、OCR 失败案例与视觉 Agent 安全轨迹" },
|
||||
{ label: "稀疏计算与 MoE", value: 74, next: "补充真实集群 traces 与专家特化案例" },
|
||||
{ label: "长上下文专题", value: 72, next: "加入更多论文逐图笔记与真实模型配置对比" },
|
||||
{ label: "大规模训练系统", value: 71, next: "补真实集群 traces、故障案例与精确 topology 配置" },
|
||||
@@ -46,7 +47,7 @@ const workstreams = [
|
||||
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
||||
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
||||
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 06:56 CST</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 07:46 CST</dd></div>
|
||||
<div><dt>MODE</dt><dd>持续迭代,不锁死版本</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
@@ -56,7 +57,7 @@ const workstreams = [
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>01</span> WORKSTREAMS</p>
|
||||
<h2>十七条工作流同时推进,但不混淆“有页面”和“已核验”</h2>
|
||||
<h2>十八条工作流同时推进,但不混淆“有页面”和“已核验”</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
内容首版优先打通全局脉络;随后每轮迭代选择一个专题推进到论文/工程层,并做独立事实复核。
|
||||
@@ -93,8 +94,8 @@ const workstreams = [
|
||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||
<article><span>✓</span><h3>16 专题知识图</h3><p>从语言模型基础到评测安全,包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||
<article><span>✓</span><h3>三十九个原创交互视图</h3><p>K3、语言模型前史、Transformer、DeepSeek、长上下文、MoE、推理、Agent,以及训练系统、Scaling、数据工程、数值和 Alignment 专题。</p></article>
|
||||
<article><span>✓</span><h3>十三篇首版长文</h3><p>K3、语言模型前史、Transformer、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、训练系统与数值优化专题。</p></article>
|
||||
<article><span>✓</span><h3>四十三个原创交互视图</h3><p>K3、语言模型前史、Transformer、DeepSeek、长上下文、MoE、推理、Agent、多模态,以及训练系统、Scaling、数据工程、数值和 Alignment 专题。</p></article>
|
||||
<article><span>✓</span><h3>十四篇首版长文</h3><p>K3、语言模型前史、Transformer、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统与数值优化专题。</p></article>
|
||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||
<article><span>✓</span><h3>Scaling Laws 深度专题</h3><p>九张账、29 个一手节点、DeepSeek/Kimi 双谱系与曲面—部署—复用—涌现四联实验。</p></article>
|
||||
@@ -106,7 +107,8 @@ const workstreams = [
|
||||
<article><span>✓</span><h3>数值、优化器与稳定性深度专题</h3><p>十张账、36 个一手节点、K2/K3 与 DeepSeek-V3/V4 双谱系,以及格式—状态—更新—失稳四联实验。</p></article>
|
||||
<article><span>✓</span><h3>指令微调与人类偏好深度专题</h3><p>十二张账、44 个一手节点、DeepSeek/Kimi 后训练双谱系,以及 SFT—RM—PPO/DPO—配方四联实验。</p></article>
|
||||
<article><span>✓</span><h3>工具使用与长程 Agent 深度专题</h3><p>十四张账、52 个一手节点、DeepSeek/Kimi Agent 双谱系,以及循环—工具契约—可靠性—长程 RL 四联实验。</p></article>
|
||||
<article><span>✓</span><h3>314 篇关键论文索引</h3><p>新增 TextWorld、WebArena、ToolSandbox、AgentDojo、RAGEN、Agent Lightning、AgentENV 等 34 个 Agent 节点。</p></article>
|
||||
<article><span>✓</span><h3>原生多模态深度专题</h3><p>十六张账、55 个一手节点、DeepSeek 三分支、Kimi 三代 MoonViT,以及 Token—连接器—光学压缩—视觉闭环四联实验。</p></article>
|
||||
<article><span>✓</span><h3>355 篇关键论文索引</h3><p>新增 ResNet、ALIGN、NaViT、DeepSeek-VL2、Janus、DeepSeek-OCR、Vision-R1 等 41 个多模态节点。</p></article>
|
||||
<article><span>✓</span><h3>公开仓库与自托管发布</h3><p>源码公开到 git.k1412.top,网站由不可变镜像、Compose Manager 与 HTTPS 交付。</p></article>
|
||||
</div>
|
||||
</section>
|
||||
@@ -132,7 +134,7 @@ const workstreams = [
|
||||
<div><span>P1</span><strong>推理二轮深化</strong><p>真实 pass@k 曲线 → PRM 失败案例 → 逐篇图表精读</p><em>案例库 + 真实 traces</em></div>
|
||||
<div><span>P1</span><strong>Alignment 二轮深化</strong><p>真实偏好分歧 → RM 长度偏置 → PPO/DPO 小模型复现</p><em>数据案例 + 可复现实验</em></div>
|
||||
<div><span>P1</span><strong>Agent 二轮深化</strong><p>真实环境 traces → cross-harness ablation → Agent RL 曲线与提示注入案例</p><em>运行证据 + 安全案例库</em></div>
|
||||
<div><span>P2</span><strong>原生多模态</strong><p>ViT/CLIP → connector VLM → Kimi-VL/MoonViT-V2</p><em>视觉 Token 流程图</em></div>
|
||||
<div><span>P1</span><strong>原生多模态二轮</strong><p>真实视觉 Token traces → 跨分辨率 / connector 消融 → OCR 与视觉 Agent 安全失败案例</p><em>运行证据 + 逐图笔记</em></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -184,6 +186,9 @@ const workstreams = [
|
||||
<div><time>2026-07-29</time><b>Attention 权重与因果解释永久分离</b><p>热力图可描述中间权重和提出假设;因果结论必须补消融、patching 或反事实干预。</p></div>
|
||||
<div><time>2026-07-29</time><b>Agent 按十四张账组织</b><p>模型、harness、工具契约、环境、评测器、可靠性与安全不再被压成一个 Agent 分数。</p></div>
|
||||
<div><time>2026-07-29</time><b>Agent 完成必须回到 final state</b><p>function schema、工具执行、业务状态、策略合规与 pass^k 分层评价;模型自报完成不作为证据。</p></div>
|
||||
<div><time>2026-07-29</time><b>原生多模态按五层管道与十六张账组织</b><p>像素、视觉塔、压缩 / connector、主干与输出 / 工具闭环分开定位;“原生”再拆成数据、目标、优化、输入输出与 Agent 五维。</p></div>
|
||||
<div><time>2026-07-29</time><b>DeepSeek 多模态永久保留三分支</b><p>VL/VL2 的理解、Janus 的统一生成、OCR 的光学压缩不画成错误的单向代际谱系。</p></div>
|
||||
<div><time>2026-07-29</time><b>光学压缩实验分开报告值、教学插值与证据外区域</b><p>DeepSeek-OCR 的 <10× / 20× 锚点标成作者报告;中间只做显式教学插值,超过范围不外推。</p></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user