feat: launch LLM Atlas research course
This commit is contained in:
wuyang
@@ -0,0 +1,6 @@
|
||||
node_modules
|
||||
dist
|
||||
.astro
|
||||
.git
|
||||
research/sources
|
||||
*.log
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
node_modules/
|
||||
dist/
|
||||
.astro/
|
||||
.DS_Store
|
||||
*.log
|
||||
|
||||
# Local primary-source cache. The public repository stores canonical URLs,
|
||||
# checksums and research notes instead of redistributing downloaded papers.
|
||||
research/sources/**/*.pdf
|
||||
research/sources/**/*.txt
|
||||
research/sources/**/*.html
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
FROM node:22-alpine AS build
|
||||
WORKDIR /app
|
||||
COPY --chmod=0644 package.json package-lock.json ./
|
||||
RUN npm ci
|
||||
COPY --chmod=0644 . .
|
||||
RUN npm run build
|
||||
|
||||
FROM nginx:1.29-alpine
|
||||
COPY --chmod=0644 deploy/nginx.conf /etc/nginx/conf.d/default.conf
|
||||
COPY --chmod=0644 --from=build /app/dist/ /usr/share/nginx/html/
|
||||
EXPOSE 8080
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD wget -q -O /dev/null http://127.0.0.1:8080/healthz || exit 1
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 LLM Atlas contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
# 持续进度
|
||||
|
||||
最后更新：2026-07-28
|
||||
|
||||
## 总体状态
|
||||
|
||||
| 工作流 | 状态 | 完成度 | 下一检查点 |
|
||||
|---|---:|---:|---|
|
||||
| 研究框架与规范 | 进行中 | 72% | 给 125 篇索引补充逐篇精读层级 |
|
||||
| 网站设计系统 | 进行中 | 86% | 打印样式与更多通用可视化组件 |
|
||||
| Kimi K3 深读 | 进行中 | 55% | 扩写 scaling / infra 逐图笔记 |
|
||||
| Transformer 基础 | 进行中 | 52% | 矩阵形状动画与手算练习 |
|
||||
| DeepSeek 专题 | 进行中 | 54% | MLA 与 GRPO 完整公式推导 |
|
||||
| 引用与事实检查 | 进行中 | 48% | 自动化外链复查与来源等级扩展 |
|
||||
| 开源仓库 | 待开始 | 0% | 首次提交并推送到 git.k1412.top |
|
||||
| k1412 部署 | 待开始 | 0% | 首个公开预览与 HTTPS 验证 |
|
||||
|
||||
## 已完成
|
||||
|
||||
- [x] 建立长期任务目标与阶段计划。
|
||||
- [x] 确认 K3 官方 47 页技术报告与官方模型仓库。
|
||||
- [x] 提取技术报告目录与 151 条参考来源，建立本地只读研究缓存。
|
||||
- [x] 从报告反推出 16 个专题与三条贯穿案例。
|
||||
- [x] 提炼参考网站的编辑设计语言。
|
||||
- [x] 确认 `git.k1412.top` 为 Gitea/Forgejo 兼容服务且本机 HTTPS 凭据可用于既有仓库。
|
||||
- [x] 使用 Grok CLI 检索并形成约 95 篇一手论文的补充路线，主代理已回查关键来源。
|
||||
- [x] 完成首批 125 篇关键论文索引，覆盖 12 个专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、Transformer 基础与 DeepSeek 三篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、Self-Attention 实验与 DeepSeek 谱系三张原创交互图。
|
||||
- [x] Astro 类型检查、生产构建、7 个内部路由和桌面/移动端视觉检查通过。
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] 开源仓库首次提交与远端公开验证。
|
||||
- [ ] 不可变容器镜像、NAS Compose Manager 部署与 HTTPS 验证。
|
||||
- [ ] 长上下文 / 高效注意力专题深度正文。
|
||||
- [ ] MoE 路由模拟器与通信成本账本。
|
||||
|
||||
## 研究账本
|
||||
|
||||
| 日期 | 决策/发现 | 影响 |
|
||||
|---|---|---|
|
||||
| 2026-07-28 | 网站命名为 **LLM Atlas / 大模型技术全景** | 既能容纳 K3 深读，也能承载完整 LLM 课程 |
|
||||
| 2026-07-28 | K3 作为“汇流点”，不是课程起点 | 初学者可以先学基础，高阶读者可以从 K3 反向跳转 |
|
||||
| 2026-07-28 | 优先重绘论文图并标明“简化/改绘” | 图可缩放、可交互，也减少脱离上下文复制论文图片 |
|
||||
| 2026-07-28 | Grok 只用于线索扩展与交叉检查 | 正文事实必须回到论文、官方仓库或正式文档 |
|
||||
| 2026-07-28 | 首批论文库收录 125 篇，按问题与专题多标签组织 | 论文库承担发现入口，专题正文承担深度精读与机制复核 |
|
||||
|
||||
## 未决问题
|
||||
|
||||
- K3 报告给出了多项公开组件的整合方式；独立论文与 K3 内部最终实现之间仍需逐项对照。
|
||||
- 公开技术报告不会披露完整数据配比和训练细节；网站会把“已知”“合理推断”“未知”分开。
|
||||
- 2026 年模型与榜单变化快，所有动态比较都必须带研究截止日期。
|
||||
@@ -0,0 +1,41 @@
|
||||
# LLM Atlas
|
||||
|
||||
一套以 Kimi K3 技术报告为锚点、从第一性原理重新梳理大语言模型技术发展的中文开放课程。
|
||||
|
||||
项目不把论文按年份堆成目录，而是持续回答四个问题：
|
||||
|
||||
1. 当时真正卡住研究者的问题是什么？
|
||||
2. 旧方法为什么不够？
|
||||
3. 关键论文改变了哪个假设或工程瓶颈？
|
||||
4. 这条思路如何汇入今天的 Kimi K3、DeepSeek 与前沿 Agent 系统？
|
||||
|
||||
## 当前交付
|
||||
|
||||
- 网站：`https://llm-atlas.k1412.top/`（首次发布后生效）
|
||||
- 开源仓库：`https://git.k1412.top/wuyang/llm-atlas`（首次推送后生效）
|
||||
- 研究路线：[ROADMAP.md](./ROADMAP.md)
|
||||
- 持续进度：[PROGRESS.md](./PROGRESS.md)
|
||||
- 证据与写作规范：[research/METHODOLOGY.md](./research/METHODOLOGY.md)
|
||||
|
||||
首个里程碑包含 16 专题学习地图、125 篇关键论文索引、Kimi K3 完整导读、
|
||||
Transformer 基础、DeepSeek 技术谱系，以及三张原创交互可视化。其余专题按进度账本持续扩建。
|
||||
|
||||
## 本地开发
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
生产构建与检查：
|
||||
|
||||
```bash
|
||||
npm run check
|
||||
npm run build
|
||||
```
|
||||
|
||||
## 内容与代码许可
|
||||
|
||||
- 网站代码采用 MIT License。
|
||||
- 原创文字与重绘图采用 [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/deed.zh-hans)。
|
||||
- 被引用论文、图表、模型与商标仍归各自权利人所有；项目优先链接一手来源，并明确标注改绘与推断。
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
# LLM Atlas 总体研究地图
|
||||
|
||||
最后更新：2026-07-28
|
||||
|
||||
## 组织原则
|
||||
|
||||
全站采用“问题链”而不是“名词表”：
|
||||
|
||||
> 现象与直觉 → 最小数学模型 → 旧方案瓶颈 → 关键论文 → 架构图/实验 → 与后继工作的桥 → K3/DeepSeek 中的落点 → 局限与开放问题
|
||||
|
||||
难度分为四层：
|
||||
|
||||
- **L0 直觉**：不要求线性代数，先建立可视化心智模型。
|
||||
- **L1 机制**：理解张量形状、训练目标与计算流程。
|
||||
- **L2 论文**：能读关键公式、消融实验和训练配置。
|
||||
- **L3 工程**：理解并行、通信、显存、精度与服务系统。
|
||||
|
||||
## 16 个专题
|
||||
|
||||
### 00. 导航：先看懂一张大模型地图
|
||||
|
||||
学习依赖、术语、读论文方法、证据等级、K3 的全局坐标。
|
||||
|
||||
### 01. 语言模型从哪里来
|
||||
|
||||
N-gram → 神经概率语言模型 → 分布式词表示 → RNN/LSTM → Seq2Seq。解释“预测下一个词”为何最终能长成通用模型。
|
||||
|
||||
### 02. 注意力与 Transformer
|
||||
|
||||
Bahdanau Attention → Self-Attention → Transformer → decoder-only GPT。逐项拆解 Q/K/V、因果掩码、多头、残差、归一化、FFN。
|
||||
|
||||
### 03. 表示层、位置与残差高速公路
|
||||
|
||||
BPE/SentencePiece、绝对/相对位置、RoPE/ALiBi、LayerNorm/RMSNorm、Pre-LN、SwiGLU，以及从 ResNet 到 K3 Attention Residuals 的深度信息流。
|
||||
|
||||
### 04. Scaling Laws：规模为什么有效
|
||||
|
||||
GPT 系列 → Kaplan scaling laws → Chinchilla compute-optimal → 数据质量与重复 → 推理时计算。区分参数、激活参数、训练 FLOPs 与能力。
|
||||
|
||||
### 05. 数据工程与预训练配方
|
||||
|
||||
采集、清洗、去重、质量分类、数据混合、课程学习、合成数据、污染控制；对公开报告中“没有说”的部分也明确标记。
|
||||
|
||||
### 06. 稀疏计算与 MoE
|
||||
|
||||
Conditional Computation → Sparsely-Gated MoE → GShard/Switch → DeepSeekMoE → LatentMoE → K3 Stable LatentMoE。重点解释路由、专家特化、负载均衡与通信。
|
||||
|
||||
### 07. 长上下文与高效注意力
|
||||
|
||||
稀疏注意力、线性注意力、FlashAttention、MQA/GQA、MLA、状态空间模型、Delta Rule、Kimi Linear/KDA、混合注意力与 1M 上下文。
|
||||
|
||||
### 08. 大规模训练系统
|
||||
|
||||
数据/张量/流水线/序列/上下文/专家并行，ZeRO，Megatron，通信重叠，容错；对比 DeepSeek DualPipe/DeepEP 与 K3 MoonEP。
|
||||
|
||||
### 09. 数值精度、优化器与稳定性
|
||||
|
||||
Adam/AdamW、Adafactor、μP、Muon；FP16/BF16/FP8/MXFP4、量化感知训练、缩放与误差。重点讲 DeepSeek-V3 FP8 与 K3 per-head Muon/MXFP4。
|
||||
|
||||
### 10. 指令微调与人类偏好
|
||||
|
||||
Instruction Tuning、SFT、RLHF/PPO、RLAIF、Constitutional AI、DPO 及其后续。解释 base model 如何变成可协作助手。
|
||||
|
||||
### 11. 推理模型与测试时扩展
|
||||
|
||||
CoT、自洽性、搜索、验证器、过程奖励、GRPO、DeepSeekMath、DeepSeek-R1/R1-Zero、Kimi k1.5、multi-effort RL 与 on-policy distillation。
|
||||
|
||||
### 12. 工具使用与长程 Agent
|
||||
|
||||
WebGPT、Toolformer、ReAct、Reflexion、代码 Agent、Computer Use、环境奖励、可验证任务、沙箱、百万 Token 轨迹与 K3 Agentic RL。
|
||||
|
||||
### 13. 原生多模态
|
||||
|
||||
ViT/CLIP → Flamingo/BLIP-2/LLaVA → 原生多模态与视频；Kimi-VL、MoonViT-V2 和“视觉进入同一主干”的意义。
|
||||
|
||||
### 14. 推理服务与低成本部署
|
||||
|
||||
KV Cache、PagedAttention/vLLM、连续批处理、推测解码、Prefill/Decode 解耦、前缀缓存、集群调度；Mooncake 与 K3 KDA-aware serving。
|
||||
|
||||
### 15. 评测、安全与“到底强不强”
|
||||
|
||||
困惑度到 MMLU/GPQA/HLE，SWE-bench、OSWorld、BrowseComp；污染、harness、工具预算、LLM-as-a-judge、选择性报告与网络安全边界。
|
||||
|
||||
## 三条贯穿式案例
|
||||
|
||||
1. **Kimi K3 解剖**：把上述全部专题重新汇总到一张架构与训练系统图。
|
||||
2. **DeepSeek 技术谱系**：DeepSeek LLM → DeepSeekMoE → V2/MLA → V3/FP8/MTP/DualPipe → Math/GRPO → R1 → V3.2/DSA → V4 长上下文。
|
||||
3. **“一个 Token 的旅行”**：从文本分词，经注意力、MoE、GPU 集群、后训练，再到线上推理与工具调用。
|
||||
|
||||
## 完成标准
|
||||
|
||||
每个专题至少包含：
|
||||
|
||||
- 1 条清晰问题链；
|
||||
- 8–20 篇一手论文；
|
||||
- 2 个以上可缩放架构图；
|
||||
- 1 个交互演示或逐步动画；
|
||||
- 1 个“常见误解”区；
|
||||
- 1 个 K3 与 DeepSeek 对照落点；
|
||||
- 所有外部事实的可点击来源、研究截止日期与证据等级；
|
||||
- 独立技术复核与链接检查。
|
||||
@@ -0,0 +1,9 @@
|
||||
import { defineConfig } from "astro/config";
|
||||
import sitemap from "@astrojs/sitemap";
|
||||
|
||||
export default defineConfig({
|
||||
site: "https://llm-atlas.k1412.top",
|
||||
output: "static",
|
||||
trailingSlash: "always",
|
||||
integrations: [sitemap()]
|
||||
});
|
||||
@@ -0,0 +1,26 @@
|
||||
server {
|
||||
listen 8080;
|
||||
listen [::]:8080;
|
||||
server_name _;
|
||||
root /usr/share/nginx/html;
|
||||
index index.html;
|
||||
|
||||
charset utf-8;
|
||||
server_tokens off;
|
||||
|
||||
location = /healthz {
|
||||
access_log off;
|
||||
default_type text/plain;
|
||||
return 200 "ok\n";
|
||||
}
|
||||
|
||||
location ~* \.(?:css|js|svg|png|jpg|jpeg|webp|woff2)$ {
|
||||
expires 7d;
|
||||
add_header Cache-Control "public, max-age=604800, immutable";
|
||||
try_files $uri =404;
|
||||
}
|
||||
|
||||
location / {
|
||||
try_files $uri $uri/ $uri/index.html =404;
|
||||
}
|
||||
}
|
||||
Generated
+4933
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"name": "llm-atlas",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"description": "从语言模型基础到 Kimi K3 的中文技术脉络与可视化课程",
|
||||
"scripts": {
|
||||
"dev": "astro dev --host 0.0.0.0",
|
||||
"build": "astro build",
|
||||
"preview": "astro preview --host 0.0.0.0",
|
||||
"check": "astro check"
|
||||
},
|
||||
"dependencies": {
|
||||
"@astrojs/sitemap": "3.7.3",
|
||||
"astro": "7.1.4"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@astrojs/check": "0.9.6",
|
||||
"typescript": "5.9.3"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">
|
||||
<rect width="64" height="64" rx="14" fill="#27364a"/>
|
||||
<path d="M15 18h5v23h13v5H15zm22 0h12v5h-7v6h6v5h-6v7h7v5H37z" fill="#f2eee7"/>
|
||||
<circle cx="51" cy="16" r="4" fill="#b86f4b"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 255 B |
@@ -0,0 +1,4 @@
|
||||
User-agent: *
|
||||
Allow: /
|
||||
|
||||
Sitemap: https://llm-atlas.k1412.top/sitemap-index.xml
|
||||
@@ -0,0 +1,535 @@
|
||||
# LLM Atlas — Grok 辅助检索路线（K3-Anchored）
|
||||
|
||||
> 生成来源：本机 Grok CLI Headless 模式，2026-07-28。
|
||||
> 用途：扩展论文线索与检查遗漏，不直接作为正文证据。主代理需回到每篇一手来源核验后，才能把结论写入网站。
|
||||
|
||||
**Site:** Chinese educational site *LLM Atlas*
|
||||
**Anchor primary source:** Kimi Team, *Kimi K3: Open Frontier Intelligence*, arXiv:[2607.24653](https://arxiv.org/abs/2607.24653) (2026-07) · DOI:[10.48550/arXiv.2607.24653](https://doi.org/10.48550/arXiv.2607.24653)
|
||||
**Companion official sources:** [Kimi Linear](https://arxiv.org/abs/2510.26692) · [Attention Residuals](https://arxiv.org/abs/2603.15031) · [Kimi K2](https://arxiv.org/abs/2507.20534) · [Kimi K2.5](https://arxiv.org/abs/2602.02276) · [Kimi-VL / MoonViT](https://arxiv.org/abs/2504.07491) · [Muon is Scalable](https://arxiv.org/abs/2502.16982)
|
||||
|
||||
**Scope rules**
|
||||
- Landmark *primary* papers only; year | title | canonical arXiv/DOI | problem solved | bridge to next idea.
|
||||
- ≥80 papers; DeepSeek lineage given dense coverage; post-V3.2 only if **official** DeepSeek release.
|
||||
- Uncertain / underspecified claims marked **`[UNCERTAIN]`**.
|
||||
- No chapter prose — notes + tables only.
|
||||
|
||||
**Paper count (this doc):** ~95 listed entries (some multi-cited across modules).
|
||||
|
||||
---
|
||||
|
||||
## 0. K3 Spec Snapshot (from official report)
|
||||
|
||||
| Spec | Value (official abstract / Table 1) |
|
||||
|---|---|
|
||||
| Total / activated params | 2.8T MoE / **104B** activated |
|
||||
| Context | **1M** tokens |
|
||||
| Attention pattern | Hybrid **3 KDA : 1 Gated MLA** per block; final layer Gated MLA; **69 KDA + 24 MLA** |
|
||||
| Depth connectivity | **Block Attention Residuals** (blocks of 12 layers) |
|
||||
| FFN | **Stable LatentMoE**: **16 of 896** routed experts (+ shared experts) |
|
||||
| Vision | **MoonViT-V2** (~401M), trained **from scratch** with NTP |
|
||||
| Optimizer | **Per-Head Muon** (+ weight clipping) |
|
||||
| Positional | **NoPE** on MLA; position via KDA decay/gating |
|
||||
| Deploy quant (post-train QAT) | MoE expert weights **MXFP4**, activations **MXFP8** |
|
||||
| Scaling claim | ~**2.5×** overall scaling efficiency vs Kimi K2 **`[claim from authors]`** |
|
||||
| Post-train pillars | Multi-effort RL · Multi-Teacher On-Policy Distillation (MOPD) · agent environments · 1M agentic RL |
|
||||
|
||||
---
|
||||
|
||||
## 1. K3 Ingredient → Research Ancestry Map
|
||||
|
||||
| K3 ingredient | Direct primary | Immediate ancestry | Module |
|
||||
|---|---|---|---|
|
||||
| **KDA / Kimi Linear** | [2510.26692](https://arxiv.org/abs/2510.26692) | Gated DeltaNet → DeltaNet → GLA → Linear Transformer | M04–M05 |
|
||||
| **Gated MLA** | K3 §2.1.2 + DeepSeek-V2 MLA [2405.04434](https://arxiv.org/abs/2405.04434) | MQA/GQA → KV compression → latent KV | M03, M05 |
|
||||
| **Attention Residuals** | [2603.15031](https://arxiv.org/abs/2603.15031) | ResNet residuals → PreNorm dilution problem | M02, M06 |
|
||||
| **Stable LatentMoE** | K3 §2.3 + LatentMoE [2601.18089](https://arxiv.org/abs/2601.18089) | Switch/GShard → DeepSeekMoE → aux-loss-free routing → QB | M07, M09 |
|
||||
| **MoonViT-V2** | K3 §2.4; prior MoonViT in Kimi-VL [2504.07491](https://arxiv.org/abs/2504.07491) | ViT → native-res VL encoders; K3 drops SigLIP init | M14 |
|
||||
| **Per-head Muon** | K3 §2.5; Muon scale-up [2502.16982](https://arxiv.org/abs/2502.16982); MuonClip in K2 [2507.20534](https://arxiv.org/abs/2507.20534) | AdamW → matrix orthogonalization optimizers | M08 |
|
||||
| **Scaling / data** | K3 §3; Kaplan/Hoffmann; K2 data rephrase | Scaling laws + domain mix + rephrase | M08 |
|
||||
| **Long context (1M)** | K3 §3 progressive extension + KDA CP | RoPE/YaRN → Ring/Ulysses → NoPE+KDA | M11 |
|
||||
| **Multi-effort RL** | K3 §4 effort ∈ {low, high, max} | o1-style test-time compute; k1.5 | M13 |
|
||||
| **Multi-teacher on-policy distillation (MOPD)** | K3 §4.1.3; Thinking Machines on-policy distill | R1 distillation; on-policy KD | M13 |
|
||||
| **Agent environments** | K3 §4.2 (pluggable harness, AET, knowledge graph) | Tool-use / SWE / OSWorld lineage | M14 |
|
||||
| **KDA systems** | K3 §5.1 FlashKDA, KDA Context Parallelism | FLA / DeltaNet CP | M11, M15 |
|
||||
| **MoonEP** | K3 §5 (perfect balance EP; vs DeepEP) | DeepEP / expert parallel | M11, M15 |
|
||||
| **1M agentic RL** | K3 §5 partial rollouts, external KV, resumable sandboxes | Long-horizon RL infra | M14–M15 |
|
||||
| **MXFP4** | K3 QAT §4.1.4; OCP MX [2310.10537](https://arxiv.org/abs/2310.10537) | FP8 training → microscaling | M10 |
|
||||
|
||||
---
|
||||
|
||||
## 2. Module Graph & Prerequisites
|
||||
|
||||
```
|
||||
M01 Foundations ──► M02 Residual/Norm/Pos
|
||||
│ │
|
||||
▼ ▼
|
||||
M03 Softmax KV path M04 Linear/Delta/SSM path
|
||||
│ │
|
||||
└────────┬───────────┘
|
||||
▼
|
||||
M05 Hybrid Attention (KDA+Gated MLA, NoPE)
|
||||
│
|
||||
▼
|
||||
M06 Depth Connectivity (AttnRes)
|
||||
│
|
||||
M07 Sparse MoE width (Stable LatentMoE)
|
||||
│
|
||||
M08 Scale / Data / Muon
|
||||
│
|
||||
M09 DeepSeek full stack (MLA→V3→R1→V3.2→V4)
|
||||
│
|
||||
┌───────────┼───────────┐
|
||||
▼ ▼ ▼
|
||||
M10 Quant M11 Long-ctx M12 Alignment (PPO/DPO)
|
||||
│ systems │
|
||||
└───────────┬───────────┘
|
||||
▼
|
||||
M13 Reasoning RL + MOPD + multi-effort
|
||||
│
|
||||
▼
|
||||
M14 Agents + Vision (MoonViT-V2)
|
||||
│
|
||||
▼
|
||||
M15 K3 Capstone Systems (MoonEP, 1M RL, MXFP4 serve)
|
||||
```
|
||||
|
||||
**Suggested learning order:** M01→M02→(M03∥M04)→M05→M06→M07→M08→M09→(M10∥M11∥M12)→M13→M14→M15.
|
||||
|
||||
---
|
||||
|
||||
## M01 — Foundations: Sequence Models & Softmax Attention
|
||||
|
||||
**Prerequisites:** undergrad ML, basic seq modeling
|
||||
**Why for K3:** defines the quadratic attention baseline that KDA/MLA hybridize.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2014 | Sequence to Sequence Learning with Neural Networks | https://arxiv.org/abs/1409.3215 | End-to-end neural transduction without alignments | Need better alignment / attention |
|
||||
| 2015 | Neural Machine Translation by Jointly Learning to Align and Translate | https://arxiv.org/abs/1409.0473 | Soft attention over encoder states | Attention as differentiable lookup |
|
||||
| 2017 | Attention Is All You Need | https://arxiv.org/abs/1706.03762 | Drop recurrence; multi-head self-attention + PE | Canonical Transformer stack |
|
||||
| 2018 | BERT: Pre-training of Deep Bidirectional Transformers | https://arxiv.org/abs/1810.04805 | Bidirectional pretrain for understanding | Pretrain–finetune paradigm |
|
||||
| 2018 | Improving Language Understanding by Generative Pre-Training (GPT) | https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf | Decoder-only LM pretrain | GPT lineage |
|
||||
| 2019 | Language Models are Unsupervised Multitask Learners (GPT-2) | https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf | Zero-shot transfer via scale | Scaling narrative |
|
||||
| 2020 | Language Models are Few-Shot Learners (GPT-3) | https://arxiv.org/abs/2005.14165 | In-context learning at 175B | ICL as emergent interface |
|
||||
|
||||
---
|
||||
|
||||
## M02 — Residual Stacks, Normalization, Position
|
||||
|
||||
**Prerequisites:** M01
|
||||
**Why for K3:** PreNorm residual dilution motivates **AttnRes**; NoPE choice; RMSNorm in MoonViT-V2.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2015 | Deep Residual Learning for Image Recognition | https://arxiv.org/abs/1512.03385 | Train very deep nets via identity skip | Residual highway becomes default |
|
||||
| 2016 | Layer Normalization | https://arxiv.org/abs/1607.06450 | Stabilize RNN/Transformer activations | LN → PreNorm stacks |
|
||||
| 2019 | Root Mean Square Layer Normalization | https://arxiv.org/abs/1910.07467 | Cheaper, stable norm without mean centering | Modern LLM default (incl. MoonViT-V2) |
|
||||
| 2021 | RoFormer: Enhanced Transformer with Rotary Position Embedding | https://arxiv.org/abs/2104.09864 | Relative PE via rotations | Dominant PE until long-ctx hacks |
|
||||
| 2023 | YaRN: Efficient Context Window Extension of LLMs | https://arxiv.org/abs/2309.00071 | Extend RoPE LMs beyond train length | Positional interpolation family |
|
||||
| 2024 | The NoPE Hypothesis (and related NoPE studies) **`[select primary carefully; see also K3 NoPE practice]`** | https://arxiv.org/abs/2404.12224 | When explicit PE is unnecessary | K3: NoPE on MLA; position via KDA |
|
||||
|
||||
**K3 note:** MLA layers use **NoPE**; positional signal comes from KDA’s channel-wise decay/gating (K3 §2.1, §3).
|
||||
|
||||
---
|
||||
|
||||
## M03 — Softmax Attention Efficiency & KV Compression → MLA
|
||||
|
||||
**Prerequisites:** M01–M02
|
||||
**Why for K3:** **Gated MLA** is the global-attention “anchor” layer in the 3:1 hybrid.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2019 | Fast Transformer Decoding: One Write-Head is All You Need (MQA) | https://arxiv.org/abs/1911.02150 | Share KV across heads → smaller cache | Multi-query attention |
|
||||
| 2023 | GQA: Training Generalized Multi-Query Transformer Models | https://arxiv.org/abs/2305.13245 | Interpolate MHA↔MQA | Production GQA default |
|
||||
| 2022 | FlashAttention: Fast and Memory-Efficient Exact Attention | https://arxiv.org/abs/2205.14135 | IO-aware exact attention | Hardware-aware attention kernels |
|
||||
| 2023 | FlashAttention-2 | https://arxiv.org/abs/2307.08691 | Higher occupancy / better parallelism | Training/prefill baseline |
|
||||
| 2023 | LongNet: Scaling Transformers to 1B Tokens (dilated attn) | https://arxiv.org/abs/2307.02486 | Sparse patterns for extreme length | Sparse attention design space |
|
||||
| 2024 | DeepSeek-V2: Strong Economical Efficient MoE LM (**MLA**) | https://arxiv.org/abs/2405.04434 | Compress KV into latent vector (MLA) | Latent cache for huge MoEs |
|
||||
| 2024 | DeepSeek-V3 Technical Report (MLA at 671B) | https://arxiv.org/abs/2412.19437 | Scale MLA + DeepSeekMoE production-grade | Gated MLA inherits MLA |
|
||||
| 2025 | DeepSeek-V3.2 (**DSA** on MLA) | https://arxiv.org/abs/2512.02556 | Fine-grained sparse attention + lightning indexer | Sparse selection over latent KV |
|
||||
|
||||
**K3 Gated MLA (official):** ungated MLA output gated by full-rank \(W_g\); gate matches full-rank KDA output gate (K3 §2.1.2).
|
||||
|
||||
---
|
||||
|
||||
## M04 — Linear Attention, Delta Rule, SSMs → KDA Ancestry
|
||||
|
||||
**Prerequisites:** M01
|
||||
**Why for K3:** **KDA** = gated delta-rule recurrence with finer channel-wise forget + full-rank output gate.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2020 | Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention | https://arxiv.org/abs/2006.16236 | Kernelize attention → linear time / RNN form | Linear Transformer baseline |
|
||||
| 2021 | Linear Transformers Are Secretly Fast Weight Programmers | https://arxiv.org/abs/2102.11174 | Delta-rule / fast weights view of linear attn | **Delta rule** update idea |
|
||||
| 2023 | Mamba: Linear-Time Sequence Modeling with Selective SSMs | https://arxiv.org/abs/2312.00752 | Input-dependent SSM selection | Modern linear-time competitor |
|
||||
| 2024 | Gated Linear Attention Transformers (GLA) | https://arxiv.org/abs/2312.06635 | Data-dependent gates + FlashLinearAttention | Hardware-efficient gated linear attn |
|
||||
| 2024 | Parallelizing Linear Transformers with the Delta Rule (DeltaNet) | https://arxiv.org/abs/2406.06484 | Parallel train delta-rule linear transformers | Chunkwise delta training |
|
||||
| 2024 | Transformers are SSMs / Mamba-2 (SSD) | https://arxiv.org/abs/2405.21060 | Unify Transformer & SSM via structured duality | State-space dual algorithms |
|
||||
| 2025 | Gated Delta Networks: Improving Mamba2 with Delta Rule | https://arxiv.org/abs/2412.06464 | Combine **gating + delta rule** | Direct parent of **KDA** |
|
||||
| 2025 | Kimi Linear: Expressive Efficient Attention (**KDA**) | https://arxiv.org/abs/2510.26692 | Channel-wise fine-grained KDA + hybrid with MLA | Productionized in K3 (with K3-specific gates) |
|
||||
|
||||
**K3 KDA deltas vs Kimi Linear (official §2.1.1):** lower-bounded log-decay (tile-stable chunkwise form); **full-rank** output gate (vs low-rank in Linear).
|
||||
|
||||
---
|
||||
|
||||
## M05 — Hybrid Attention Architectures (KDA + Gated MLA + NoPE)
|
||||
|
||||
**Prerequisites:** M03, M04
|
||||
**Why for K3:** core sequence mixer: **3×KDA + 1×Gated MLA**, final Gated MLA.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2023 | Retentive Network (RetNet) | https://arxiv.org/abs/2307.08621 | Multi-scale retention recurrence | Hybrid linear/global motivation |
|
||||
| 2024 | Griffin: Mixing Gated Linear Recurrences with Local Attention | https://arxiv.org/abs/2402.19427 | Mix local attention + linear recurrence | Hybrid design pattern |
|
||||
| 2024 | Jamba: Hybrid Transformer-Mamba | https://arxiv.org/abs/2403.19887 | Production hybrid MoE | Hybrid at scale |
|
||||
| 2024 | DeltaNet hybrids (sliding / global layers) *in* DeltaNet paper | https://arxiv.org/abs/2406.06484 | Sparse global layers restore quality | 3:1-style interleaving idea |
|
||||
| 2025 | Kimi Linear (3:1 KDA:MLA hybrid) | https://arxiv.org/abs/2510.26692 | Match/beat full MLA quality at lower KV | Direct K3 attention recipe |
|
||||
| 2026 | Kimi K3 (Gated MLA + refined KDA) | https://arxiv.org/abs/2607.24653 | Scale hybrid to 2.8T / 1M ctx | Capstone hybrid |
|
||||
|
||||
**Reading target:** derive why periodic full latent attention repairs linear-state information bottlenecks.
|
||||
|
||||
---
|
||||
|
||||
## M06 — Depth-wise Connectivity: PreNorm Dilution → Attention Residuals
|
||||
|
||||
**Prerequisites:** M02, M05
|
||||
**Why for K3:** **Block AttnRes** is the depth information-flow upgrade.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2015 | Deep Residual Learning (again as residual baseline) | https://arxiv.org/abs/1512.03385 | Identity mapping depth | Fixed unit residual weights |
|
||||
| 2016 | Highway Networks | https://arxiv.org/abs/1505.00387 | Learned gates on residual paths | Content-dependent depth flow |
|
||||
| 2017 | DenseNet | https://arxiv.org/abs/1608.06993 | Concatenate all prior feature maps | Dense cross-layer reuse |
|
||||
| 2021 | DeepNet: Stabilizing Extremely Deep Transformers | https://arxiv.org/abs/2203.00555 | Scale residuals for 1000-layer Transformers | Deep PreNorm issues |
|
||||
| 2026 | Attention Residuals (AttnRes / Block AttnRes) | https://arxiv.org/abs/2603.15031 | Softmax over prior layer outputs; block variant for memory | **K3 uses Block AttnRes** (S≈12) |
|
||||
| 2026 | DeepSeek-V4 mHC (manifold-constrained hyper-connections) *official later residual variant* | https://arxiv.org/abs/2606.19348 | Alternative residual upgrade at 1M-ctx MoE | Parallel residual research thread |
|
||||
|
||||
**K3 detail:** pseudo-query \(q_l=w_l\); Block AttnRes reduces overhead \(O(Ld)\to O(Nd)\).
|
||||
|
||||
---
|
||||
|
||||
## M07 — Sparse Width: MoE → DeepSeekMoE → LatentMoE → Stable LatentMoE
|
||||
|
||||
**Prerequisites:** M01, M08 (scaling intuition helps)
|
||||
**Why for K3:** **Stable LatentMoE** width path (16/896 + Quantile Balancing).
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2017 | Outrageously Large Neural Networks (Sparsely-Gated MoE) | https://arxiv.org/abs/1701.06538 | Conditional computation at scale | MoE modern start |
|
||||
| 2020 | GShard: Scaling Giant Models with Conditional Computation | https://arxiv.org/abs/2006.16668 | Automatic sharding + MoE at TPU scale | Systems MoE |
|
||||
| 2021 | Switch Transformers | https://arxiv.org/abs/2101.03961 | Simplify to top-1 routing; trillion-param | Load-balance auxiliaries |
|
||||
| 2022 | ST-MoE | https://arxiv.org/abs/2202.08906 | Stable training recipes for sparse models | Stability folklore |
|
||||
| 2024 | DeepSeekMoE: Towards Ultimate Expert Specialization | https://arxiv.org/abs/2401.06066 | Fine-grained experts + shared experts | DeepSeek MoE DNA |
|
||||
| 2024 | Mixtral of Experts | https://arxiv.org/abs/2401.04088 | Strong open sparse LM | Sparse quality reference |
|
||||
| 2024 | DeepSeek-V2 (DeepSeekMoE @ production) | https://arxiv.org/abs/2405.04434 | Economical training with sparse FFN | MLA+MoE combo |
|
||||
| 2024 | DeepSeek-V3 (aux-loss-free routing) | https://arxiv.org/abs/2412.19437 | Bias-based load balance without aux loss | Aux-loss-free lineage |
|
||||
| 2026 | LatentMoE: Optimal Accuracy per FLOP/Parameter | https://arxiv.org/abs/2601.18089 | Route in latent cheaper expert path | Namesake of **LatentMoE** |
|
||||
| 2026 | Kimi K3 Stable LatentMoE + **Quantile Balancing** | https://arxiv.org/abs/2607.24653 | Stabilize extreme sparsity (896 experts, k=16) | Production K3 FFN |
|
||||
|
||||
**K3 QB (official):** expert bias from margin quantiles via global histogram all-reduce (not token-side exact quantiles).
|
||||
**Also cite:** Jianlin Su blog *Travels in MoE* (load balance via optimal assignment) — referenced by K3 as [111] (Chinese primary blog, not arXiv).
|
||||
|
||||
---
|
||||
|
||||
## M08 — Scaling Laws, Data Recipes, Optimizers (→ Per-Head Muon)
|
||||
|
||||
**Prerequisites:** M01
|
||||
**Why for K3:** 2.5× scaling-efficiency claim; Per-Head Muon; cosine > WSD in their search.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2020 | Scaling Laws for Neural Language Models (Kaplan et al.) | https://arxiv.org/abs/2001.08361 | Power-law loss vs N, D, C | Classical scaling |
|
||||
| 2022 | Training Compute-Optimal LLMs (Chinchilla / Hoffmann) | https://arxiv.org/abs/2203.15556 | Tokens–params balance | TPP retuning practice |
|
||||
| 2014 | Adam: A Method for Stochastic Optimization | https://arxiv.org/abs/1412.6980 | Adaptive moments | Default baseline |
|
||||
| 2017 | Decoupled Weight Decay Regularization (AdamW) | https://arxiv.org/abs/1711.05101 | Correct weight decay | Pre-Muon default |
|
||||
| 2024 | Muon: An Optimizer for Hidden Layers (Keller Jordan et al.) | https://kellerjordan.github.io/posts/muon/ | Newton–Schulz orthogonalized momentum | Matrix-wise updates |
|
||||
| 2025 | Muon is Scalable for LLM Training (Moonshot / Moonlight) | https://arxiv.org/abs/2502.16982 | Weight decay + update scale → Muon at LLM scale | Moonshot optimizer stack |
|
||||
| 2025 | Kimi K2 (MuonClip / QK-Clip) | https://arxiv.org/abs/2507.20534 | Stabilize attention logits under Muon | Pretrain 15.5T zero spike claim |
|
||||
| 2026 | Kimi K3 Per-Head Muon | https://arxiv.org/abs/2607.24653 | Orthogonalize QKV **per head** for balanced head scales | K3 default optimizer |
|
||||
|
||||
**Data / schedule landmarks**
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2020 | Exploring the Limits of Transfer Learning (T5) | https://arxiv.org/abs/1910.10683 | Unified text-to-text + C4 data | Data quality culture |
|
||||
| 2022 | Training Compute-Optimal… (data side of Chinchilla) | https://arxiv.org/abs/2203.15556 | More tokens for given compute | Overtrain regimes |
|
||||
| 2024 | DeepSeek LLM: Scaling Open-Source LMs with Longtermism | https://arxiv.org/abs/2401.02954 | Open scaling-law study + 67B | DeepSeek program start |
|
||||
| 2025 | MiniCPM / WSD schedule discussion | https://arxiv.org/abs/2404.06395 | Warmup–Stable–Decay alternative | K3 finds **cosine better** after independent HPO |
|
||||
| 2025 | Kimi K2 data rephrasing recipe | https://arxiv.org/abs/2507.20534 | Knowledge/math rephrase + fidelity checks | Inherited by K3 §3.1 |
|
||||
|
||||
---
|
||||
|
||||
## M09 — DeepSeek Lineage (Dense → MoE → MLA → V3 → R1 → V3.2 → V4)
|
||||
|
||||
**Prerequisites:** M03, M07, M08, M12–M13 for post-train pieces
|
||||
**Policy:** include only **official** DeepSeek papers/reports.
|
||||
|
||||
### 9.1 Pretrain architecture stack
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2024 | DeepSeek LLM: Scaling Open-Source Language Models with Longtermism | https://arxiv.org/abs/2401.02954 | Open dense 7B/67B + scaling study | Foundation |
|
||||
| 2024 | DeepSeekMoE: Ultimate Expert Specialization | https://arxiv.org/abs/2401.06066 | Fine-grained + shared experts | Sparse specialization |
|
||||
| 2024 | DeepSeek-V2: MLA + DeepSeekMoE | https://arxiv.org/abs/2405.04434 | KV latent compression + economical MoE | **MLA birth** |
|
||||
| 2024 | DeepSeek-V3 Technical Report | https://arxiv.org/abs/2412.19437 | 671B/37B act; **FP8**; **MTP**; **aux-loss-free routing**; **DualPipe** | Flagship pretrain stack |
|
||||
| 2025 | DeepSeek-V3.2: DSA + scalable RL + agentic synthesis | https://arxiv.org/abs/2512.02556 | **DeepSeek Sparse Attention**; agent post-train scale | Long-ctx efficiency |
|
||||
| 2026 | DeepSeek-V4: Million-Token Context Intelligence | https://arxiv.org/abs/2606.19348 | CSA+HCA hybrid attn; **mHC**; Muon; 1M ctx Pro/Flash | Official later 1M-ctx peer to K3 |
|
||||
|
||||
### 9.2 Math, RL, reasoning distillation
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2024 | DeepSeekMath (+ **GRPO**) | https://arxiv.org/abs/2402.03300 | Math continual pretrain + critic-free group RL | **GRPO** algorithm |
|
||||
| 2025 | DeepSeek-R1: Incentivizing Reasoning via RL | https://arxiv.org/abs/2501.12948 | **R1-Zero** pure RL; **R1** multi-stage; **distillation** to dense 1.5B–70B | Reasoning RL template |
|
||||
| 2025 | DeepSeek-R1 Nature version (same line) | https://doi.org/10.1038/s41586-025-09422-z | Peer-reviewed presentation of R1 | Archival citation |
|
||||
|
||||
### 9.3 V3 systems keywords → primary sections
|
||||
|
||||
| Keyword | Where (primary) | Problem solved | Bridge |
|
||||
|---|---|---|---|
|
||||
| **FP8 mixed precision** | V3 report §3.3 | Low-precision train at 671B | → MX / QAT serving |
|
||||
| **MTP** (multi-token prediction) | V3 report | Stronger pretrain objective / draft head | Speculative / denser signal |
|
||||
| **Aux-loss-free routing** | V3 report | Balance without aux loss interference | → K3 QB different mechanism |
|
||||
| **DualPipe** | V3 report §3.2.1 | Overlap PP bubbles + MoE dispatch/combine | → MoonEP alternative EP design |
|
||||
| **DSA** | V3.2 report | Sparse token selection under MLA | Contrast with KDA hybrid |
|
||||
| **DeepEP** | https://github.com/deepseek-ai/DeepEP | Expert-parallel comm library | K3 MoonEP builds on / contrasts |
|
||||
|
||||
**Atlas teaching note:** treat DeepSeek as the *dense reference stack* for MLA/MoE/FP8/RL; treat K3 as the *hybrid linear + AttnRes + Stable LatentMoE* fork at larger sparsity and 1M agentic RL.
|
||||
|
||||
---
|
||||
|
||||
## M10 — Low-Precision Training & Deployment (FP8 → MXFP4)
|
||||
|
||||
**Prerequisites:** M08–M09
|
||||
**Why for K3:** post-train **QAT** with **MXFP4 weights / MXFP8 activations** on MoE experts.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2018 | Quantization and Training of Neural Networks for Integer-Arithmetic-Only Inference | https://arxiv.org/abs/1712.05877 | QAT for int inference | QAT paradigm |
|
||||
| 2020 | Training with Quantization Noise | https://arxiv.org/abs/2004.07320 | Noise injection for robust quant | QAT variants |
|
||||
| 2022 | FP8 Formats for Deep Learning (Micikevicius et al.) | https://arxiv.org/abs/2209.05433 | FP8 train/infer formats | Hardware FP8 |
|
||||
| 2023 | Microscaling Data Formats for Deep Learning (**MX / MXFP4**) | https://arxiv.org/abs/2310.10537 | Block scales + narrow element types | **OCP MX** family |
|
||||
| 2024 | DeepSeek-V3 FP8 training framework | https://arxiv.org/abs/2412.19437 | Validate FP8 at extreme MoE scale | Pretrain low-prec |
|
||||
| 2026 | Kimi K3 MXFP4 QAT from SFT onward | https://arxiv.org/abs/2607.24653 | Deploy-time memory for 2.8T experts | Serving recipe |
|
||||
|
||||
**OCP primary (non-arXiv):** OCP Microscaling Formats (MX) Specification v1.0 — https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf
|
||||
|
||||
---
|
||||
|
||||
## M11 — Long Context: Algorithms & Parallelism (→ KDA CP, 1M)
|
||||
|
||||
**Prerequisites:** M03–M05
|
||||
**Why for K3:** progressive 8K→64K→256K→**1M**; **KDA Context Parallelism**; state-aware prefix cache.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2023 | ALiBi | https://arxiv.org/abs/2108.12409 | Length extrapolation via linear biases | PE-free-ish long ctx |
|
||||
| 2023 | RoPE / YaRN (see M02) | https://arxiv.org/abs/2309.00071 | Extend rotary models | Common production path |
|
||||
| 2023 | Ring Attention with Blockwise Transformers | https://arxiv.org/abs/2310.01889 | Sequence parallel for near-infinite context | Distributed long attn |
|
||||
| 2023 | DeepSpeed Ulysses | https://arxiv.org/abs/2309.14509 | Sequence parallelism system opts | Cluster long-ctx train |
|
||||
| 2024 | Linear Attention Sequence Parallelism (LASP) | https://arxiv.org/abs/2404.02882 | SP specialized for linear attn | Linear-SP ancestors |
|
||||
| 2025 | LASP-2 / hybrid linear SP | https://arxiv.org/abs/2502.07864 **`[verify id if citing; K3 cites related]`** | Hybrid linear+softmax SP | Closer to KDA hybrids |
|
||||
| 2025 | Context Parallelism for DeltaNet (Wang) | https://yywangcs.notion.site/DeltaNet-2a9fc9f5d8058013a498f34e0b25bd52 | CP for delta recurrence | Direct ancestor of **KCP** |
|
||||
| 2026 | Kimi K3 KDA Context Parallelism + FlashKDA | https://arxiv.org/abs/2607.24653 · FlashKDA https://github.com/MoonshotAI/FlashKDA | Correct state transport under delta update \(M_t S_{t-1}\) | 1M train/prefill |
|
||||
|
||||
**K3 progressive extension (official):** pretrain grows 8K→64K; cooldown 256K→1M; needle/synthetic scatter tasks force full-window use.
|
||||
|
||||
---
|
||||
|
||||
## M12 — Alignment Foundations (RLHF / PPO / DPO)
|
||||
|
||||
**Prerequisites:** M01
|
||||
**Why for K3:** base of modern post-train before GRPO / multi-effort / MOPD.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2017 | Proximal Policy Optimization Algorithms | https://arxiv.org/abs/1707.06347 | Stable policy gradient updates | RL workhorse |
|
||||
| 2022 | Training Language Models to Follow Instructions with Human Feedback (InstructGPT) | https://arxiv.org/abs/2203.02155 | RLHF pipeline for LLMs | Industry alignment template |
|
||||
| 2022 | Constitutional AI | https://arxiv.org/abs/2212.08073 | Principle-based AI feedback | RLAIF direction |
|
||||
| 2023 | Direct Preference Optimization (DPO) | https://arxiv.org/abs/2305.18290 | Preference learning without RL loop | Offline preference |
|
||||
| 2023 | Llama 2: Open Foundation and Fine-Tuned Chat Models | https://arxiv.org/abs/2307.09288 | Open RLHF stack details | Open alignment recipes |
|
||||
| 2024 | SimPO / ORPO family (optional shortlist) | https://arxiv.org/abs/2405.14734 | Simpler preference objectives | Alt to DPO |
|
||||
|
||||
---
|
||||
|
||||
## M13 — Reasoning RL, Multi-Effort, On-Policy Distillation
|
||||
|
||||
**Prerequisites:** M12, M09
|
||||
**Why for K3:** multi-domain RL × effort levels; **MOPD** consolidation.
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2022 | Chain-of-Thought Prompting | https://arxiv.org/abs/2201.11903 | Elicit intermediate reasoning | Test-time reasoning culture |
|
||||
| 2023 | Let’s Verify Step by Step (process reward) | https://arxiv.org/abs/2305.20050 | Process vs outcome supervision | PRM path |
|
||||
| 2024 | DeepSeekMath / **GRPO** | https://arxiv.org/abs/2402.03300 | Group-relative baseline, no critic | Memory-efficient RL |
|
||||
| 2024 | OpenAI o1 announcement / “Learning to Reason with LLMs” | https://openai.com/index/learning-to-reason-with-llms/ | Scale RL + test-time compute | Multi-effort ancestor **`[blog primary; not full paper]`** |
|
||||
| 2025 | Kimi k1.5: Scaling RL with LLMs | https://arxiv.org/abs/2501.12599 | Long-CoT RL scaling (Moonshot) | Kimi reasoning line |
|
||||
| 2025 | DeepSeek-R1 / R1-Zero / distillation | https://arxiv.org/abs/2501.12948 | Pure RL emergence; then multi-stage + distill | Distill reasoning patterns |
|
||||
| 2025 | On-policy distillation (Thinking Machines Lab) | https://thinkingmachines.ai/blog/on-policy-distillation/ **`[blog; K3 cites as Connectionism note]`** | Student on-policy w.r.t teacher | **MOPD** conceptual parent |
|
||||
| 2026 | MiMo-V2-Flash Technical Report (multi-teacher mention lineage) | https://arxiv.org/abs/2601.02780 | Related multi-teacher post-train | Peer system cited by K3 |
|
||||
| 2026 | Kimi K3 multi-effort RL + **MOPD** | https://arxiv.org/abs/2607.24653 | Domain experts {general, agentic, coding} × {low, high, max} → single policy | Capstone post-train |
|
||||
|
||||
**K3 MOPD (official):** on-policy student samples; multi-teacher logits/rewards consolidate specialists; top-k distill ablations showed no clear gain.
|
||||
|
||||
---
|
||||
|
||||
## M14 — Agents, Environments, Multimodal (MoonViT-V2)
|
||||
|
||||
**Prerequisites:** M13, M05
|
||||
**Why for K3:** agent harness modularization; AET; native vision-in-the-loop; MoonViT-V2 from scratch.
|
||||
|
||||
### 14.1 Agent / tool / coding environments
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2023 | Toolformer | https://arxiv.org/abs/2302.04761 | Self-supervised tool API calls | Tool-use pretrain |
|
||||
| 2023 | ReAct | https://arxiv.org/abs/2210.03629 | Interleave reason + act | Agent loop pattern |
|
||||
| 2024 | SWE-bench | https://arxiv.org/abs/2310.06770 | Real GitHub issue resolution | Coding agent benchmark |
|
||||
| 2024 | OSWorld | https://arxiv.org/abs/2404.07972 | Computer-use agents | GUI agent env |
|
||||
| 2025 | Terminal-Bench | https://arxiv.org/abs/2502.14045 **`[confirm version K3 cites]`** | Hard CLI agent tasks | Terminal agents |
|
||||
| 2025 | BrowseComp | https://arxiv.org/abs/2504.12516 | Browsing agents | Web agents |
|
||||
| 2025 | Kimi K2 agentic data synthesis + joint RL | https://arxiv.org/abs/2507.20534 | Large synthetic tool trajectories | Kimi agent stack |
|
||||
| 2026 | Kimi K2.5 Visual Agentic Intelligence | https://arxiv.org/abs/2602.02276 | Multimodal agents / swarm **`[swarm details in K2.5]`** | Vision agents |
|
||||
| 2026 | Kimi K3 agent envs (pluggable harness, AET, knowledge graph) | https://arxiv.org/abs/2607.24653 | Avoid harness overfitting; long-horizon verifiers | Production agent RL data |
|
||||
|
||||
### 14.2 Vision encoders → MoonViT-V2
|
||||
|
||||
| Year | Title | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2020 | An Image is Worth 16x16 Words (ViT) | https://arxiv.org/abs/2010.11929 | Transformer vision backbone | ViT era |
|
||||
| 2021 | Learning Transferable Visual Models From Natural Language Supervision (CLIP) | https://arxiv.org/abs/2103.00020 | Contrastive vision–language pretrain | SigLIP-style inits |
|
||||
| 2023 | SigLIP | https://arxiv.org/abs/2303.15343 | Sigmoid contrastive VL | Prior MoonViT init |
|
||||
| 2024 | LLaVA-OneVision | https://arxiv.org/abs/2408.03326 | Unified single/multi-image/video tasks | Native-res VL practice |
|
||||
| 2025 | Kimi-VL Technical Report (**MoonViT**) | https://arxiv.org/abs/2504.07491 | Native-resolution MoonViT + MoE LM | Moonshot VL stack |
|
||||
| 2026 | Kimi K3 **MoonViT-V2** from-scratch NTP | https://arxiv.org/abs/2607.24653 | Drop SigLIP init for stability; match quality | K3 vision path |
|
||||
|
||||
**K3 vision claim (official):** MoonViT-V2 ≈0.4B, 27 layers, RMSNorm, bias-free; shared image/video; 2×2 pixel-shuffle; up to 3584² in 1M context.
|
||||
|
||||
---
|
||||
|
||||
## M15 — Capstone Systems: KDA Kernels, MoonEP, 1M Agentic RL, Serve
|
||||
|
||||
**Prerequisites:** M05–M07, M10–M11, M13–M14
|
||||
**Why for K3:** makes 2.8T hybrid + 1M agentic RL trainable and deployable.
|
||||
|
||||
| Year | Title / artifact | URL | Problem solved | Bridge to next |
|
||||
|---|---|---|---|---|
|
||||
| 2019 | Triton: Intermediate Language for Tiled NN Kernels | https://www.eecs.harvard.edu/~htk/publication/2019-mapl-tillet-kung-cox.pdf | Productive GPU kernels | FLA / custom attn |
|
||||
| 2021 | GPipe | https://arxiv.org/abs/1811.06965 | Pipeline parallel | PP lineage |
|
||||
| 2021 | Megatron-LM / efficient large-scale training | https://arxiv.org/abs/2104.04473 | 3D parallelism | Pretrain systems baseline |
|
||||
| 2020 | ZeRO | https://arxiv.org/abs/1910.02054 | Shard optimizer states | Memory for huge models |
|
||||
| 2024 | FLA: Flash Linear Attention library | https://github.com/fla-org/flash-linear-attention | Kernel zoo for linear/delta | KDA ops host |
|
||||
| 2025 | DeepEP | https://github.com/deepseek-ai/DeepEP | Expert-parallel communication | MoonEP contrast |
|
||||
| 2025 | DualPipe (in V3 report) | https://arxiv.org/abs/2412.19437 | Overlap MoE+PP | Pipeline bubbles |
|
||||
| 2026 | FlashKDA | https://github.com/MoonshotAI/FlashKDA | CUTLASS chunkwise KDA | Train/prefill speed |
|
||||
| 2026 | Kimi K3 **MoonEP** | https://arxiv.org/abs/2607.24653 | Perfect balance EP, static shapes, zero-copy, bounded redundant experts | 2.8T MoE train |
|
||||
| 2026 | Kimi K3 1M agentic RL co-located system | https://arxiv.org/abs/2607.24653 | Partial rollouts, external KV retention, resumable sandboxes | Long-horizon RL |
|
||||
| 2026 | Kimi K3 state-aware KDA prefix caching + MXFP4 serve | https://arxiv.org/abs/2607.24653 | Decode across hybrid state + quant experts | Production deployment |
|
||||
|
||||
---
|
||||
|
||||
## 3. Module → K3 Ingredient Checklist
|
||||
|
||||
| Module | Delivers toward K3 |
|
||||
|---|---|
|
||||
| M01–M02 | Transformer residual + PE literacy |
|
||||
| M03 | MLA / Gated MLA |
|
||||
| M04–M05 | KDA + hybrid 3:1 |
|
||||
| M06 | Attention Residuals |
|
||||
| M07 | Stable LatentMoE + QB vs aux-loss-free |
|
||||
| M08 | Scaling 2.5× story + Per-Head Muon |
|
||||
| M09 | DeepSeek dense comparator (MLA, FP8, MTP, DualPipe, GRPO, R1, DSA, V4) |
|
||||
| M10 | MXFP4/MXFP8 QAT |
|
||||
| M11 | 1M context + KCP |
|
||||
| M12–M13 | Multi-effort RL + MOPD |
|
||||
| M14 | Agent envs + MoonViT-V2 |
|
||||
| M15 | MoonEP + 1M agentic RL systems |
|
||||
|
||||
---
|
||||
|
||||
## 4. DeepSeek Coverage Matrix (Atlas “strong track”)
|
||||
|
||||
| Topic | Primary | Module | Must-read density |
|
||||
|---|---|---|---|
|
||||
| DeepSeek LLM | 2401.02954 | M08/M09 | high |
|
||||
| DeepSeekMoE | 2401.06066 | M07/M09 | high |
|
||||
| V2 / MLA | 2405.04434 | M03/M09 | **core** |
|
||||
| Math / GRPO | 2402.03300 | M13/M09 | **core** |
|
||||
| V3 FP8 / MTP / aux-free / DualPipe | 2412.19437 | M09/M10/M15 | **core** |
|
||||
| R1 / R1-Zero / distill | 2501.12948 (+ Nature DOI) | M13/M09 | **core** |
|
||||
| V3.2 / DSA | 2512.02556 | M03/M09 | high |
|
||||
| V4 (official later) | 2606.19348 | M06/M09/M11 | high (1M peer) |
|
||||
| DeepEP | GitHub deepseek-ai/DeepEP | M15 | systems |
|
||||
|
||||
---
|
||||
|
||||
## 5. Uncertain / Careful Claims
|
||||
|
||||
| Claim | Status |
|
||||
|---|---|
|
||||
| K3 **2.5×** scaling efficiency vs K2 | Author-reported fit on OOD val curves (Fig. 7); **not independent replication** |
|
||||
| Benchmarks vs Claude Fable 5 / GPT-5.6 Sol | Author suite; **external harness variance** possible |
|
||||
| “First open 3T-class” marketing phrasing | Product language; verify total-param definitions (shared vs routed, embeddings, vision) |
|
||||
| Exact MoonEP algorithm vs DeepEP | Described in K3 §5; **no standalone MoonEP paper** at time of roadmap |
|
||||
| FlashKDA paper | Primarily **code artifact** + K3 description; limited standalone theory paper |
|
||||
| LASP-2 arXiv number if used in syllabus | Double-check id before publishing student links |
|
||||
| On-policy distillation “Connectionism” note | Blog-level primary; treat as concept source, not peer-reviewed algorithm paper |
|
||||
| DeepSeek-V4 CSA/HCA vs K3 KDA | Parallel 1M-ctx designs; **do not equate** without side-by-side study |
|
||||
| MoonViT-V2 “matches SigLIP init quality” | Author ablation claim (Fig. 6 + text) |
|
||||
| Multi-effort exact reward formulas | Partially specified (thinking-token thresholds); full reward suite may be incomplete in report |
|
||||
|
||||
---
|
||||
|
||||
## 6. Compact Reading Paths (for site UX)
|
||||
|
||||
### Path A — “Understand K3 architecture in 12 papers”
|
||||
1. Attention Is All You Need
|
||||
2. FlashAttention-2
|
||||
3. DeepSeek-V2 (MLA)
|
||||
4. Gated DeltaNet
|
||||
5. Kimi Linear (KDA)
|
||||
6. Attention Residuals
|
||||
7. DeepSeekMoE
|
||||
8. LatentMoE
|
||||
9. Muon is Scalable
|
||||
10. Microscaling (MXFP4)
|
||||
11. DeepSeek-V3 (systems)
|
||||
12. **Kimi K3**
|
||||
|
||||
### Path B — “DeepSeek full stack (official only)”
|
||||
DeepSeek LLM → DeepSeekMoE → DeepSeekMath/GRPO → V2/MLA → V3 → R1 → V3.2/DSA → V4
|
||||
|
||||
### Path C — “Post-train / agents”
|
||||
InstructGPT → DPO → GRPO → R1 → k1.5 → K2 → K2.5 → K3 (MOPD + multi-effort + 1M agentic RL)
|
||||
|
||||
---
|
||||
|
||||
## 7. Suggested Atlas Module Metadata (CMS fields)
|
||||
|
||||
```yaml
|
||||
module_id: M05
|
||||
title_zh: 混合注意力：KDA 与 Gated MLA
|
||||
prerequisites: [M03, M04]
|
||||
k3_tags: [KDA, GatedMLA, NoPE, hybrid-3to1]
|
||||
primary_papers: 6
|
||||
estimated_hours: 8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Source Log (what this roadmap used)
|
||||
|
||||
| Source | Role |
|
||||
|---|---|
|
||||
| arXiv:2607.24653 K3 PDF text extract | Architecture, post-train, systems, bibliography |
|
||||
| arXiv abstracts for DeepSeek V2/V3/V3.2/V4, R1, Math, MoE, LLM | Official lineage |
|
||||
| arXiv:2510.26692 Kimi Linear | KDA definition |
|
||||
| arXiv:2603.15031 AttnRes | Depth residual redesign |
|
||||
| arXiv:2502.16982 / 2507.20534 | Muon / K2 |
|
||||
| arXiv:2504.07491 / 2602.02276 | MoonViT / K2.5 |
|
||||
| arXiv:2310.10537 + OCP MX | MXFP4 |
|
||||
| arXiv:2601.18089 LatentMoE | Latent MoE naming/ancestry |
|
||||
|
||||
---
|
||||
|
||||
*End of structured research notes — LLM Atlas / K3-anchored roadmap.*
|
||||
@@ -0,0 +1,46 @@
|
||||
# 研究、引用与写作规范
|
||||
|
||||
## 来源优先级
|
||||
|
||||
1. **P0 一手论文**：arXiv、会议论文、期刊论文、作者正式技术报告。
|
||||
2. **P1 一手实现**：作者官方仓库、模型卡、训练/评测代码。
|
||||
3. **P2 官方说明**：实验室技术博客、产品文档、系统卡。
|
||||
4. **P3 独立复现**：有方法、代码和数据的第三方复现或评测。
|
||||
5. **P4 二手解释**：只用于发现线索，不承载关键事实。
|
||||
|
||||
正文关键结论原则上至少有一个 P0/P1 来源。模型发布当日尚无论文时，可以临时使用 P2，并清楚标注。
|
||||
|
||||
## 三类句子
|
||||
|
||||
- **事实**：论文明确报告的架构、数据、实验或结论，可直接引用。
|
||||
- **解释**：为了教学而做的类比、拆解和重绘，标为“直觉解释”。
|
||||
- **推断**：跨论文比较或报告未直接说明的因果判断，标为“我们的推断”并写出依据。
|
||||
|
||||
## 论文卡字段
|
||||
|
||||
- 稳定 ID、标题、作者/机构、年份；
|
||||
- canonical URL、arXiv/DOI、发表状态；
|
||||
- 所属专题、先修概念、后继工作；
|
||||
- 它解决的问题、核心机制、关键证据；
|
||||
- 一句话直觉、关键公式、图表索引；
|
||||
- 限制、复现状态、与 K3/DeepSeek 的关系；
|
||||
- 核验状态、核验日期、核验人/工具。
|
||||
|
||||
## 图表规范
|
||||
|
||||
- 优先原创重绘 SVG/HTML，可缩放并支持键盘阅读。
|
||||
- 图题必须说明“原创示意”“依据某图改绘”或“原图引用”。
|
||||
- 简化图不能伪装成模型的逐算子精确实现。
|
||||
- 图中颜色始终保持语义一致：蓝色表示信息流，铜色表示稀疏选择，绿色表示训练/验证反馈，紫色表示系统状态。
|
||||
|
||||
## 质量闸门
|
||||
|
||||
章节从草稿到发布依次通过：
|
||||
|
||||
1. 结构检查：问题链是否完整；
|
||||
2. 事实检查：数字、版本、时间、公式；
|
||||
3. 来源检查：链接可达、引用贴近结论；
|
||||
4. 教学检查：术语首次出现有解释，图能独立读懂；
|
||||
5. 对照检查：K3/DeepSeek 映射是否准确；
|
||||
6. 可访问性检查：键盘、对比度、移动端、减少动画；
|
||||
7. 构建与链接检查。
|
||||
@@ -0,0 +1,12 @@
|
||||
# 本地来源缓存
|
||||
|
||||
这里用于研究期间缓存论文 PDF、网页快照与文本抽取。二进制和全文缓存默认被 `.gitignore` 排除；公开仓库只提交 canonical URL、校验信息、研究笔记和允许再分发的原创内容。
|
||||
|
||||
当前锚点：
|
||||
|
||||
- Kimi K3 Technical Report
|
||||
<https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf>
|
||||
- Kimi K3 official repository
|
||||
<https://github.com/MoonshotAI/Kimi-K3>
|
||||
- arXiv:2607.24653
|
||||
<https://arxiv.org/abs/2607.24653>
|
||||
@@ -0,0 +1,412 @@
|
||||
---
|
||||
const axes = [
|
||||
{
|
||||
id: "token",
|
||||
label: "序列 / Token",
|
||||
title: "KDA × Gated MLA：让一百万 Token 既高效流动，也保留全局精确交互",
|
||||
plain: "三层 KDA 像持续更新的“工作记忆”，每四层插入一层全局 MLA，像定期把整本笔记摊开重看。",
|
||||
points: ["3 个 KDA 层 + 1 个 Gated MLA 层为一组", "主干共 69 层 KDA、24 层 Gated MLA", "KDA 线性扩展；MLA 周期性补足全局两两交互"],
|
||||
source: "K3 §2.1",
|
||||
},
|
||||
{
|
||||
id: "depth",
|
||||
label: "深度 / Layer",
|
||||
title: "Attention Residuals：每一层不只接住上一步，而是有选择地回看更早层",
|
||||
plain: "普通残差像接力棒，只能拿到累加后的结果；AttnRes 更像档案索引，可以挑选哪一层的中间表示最有用。",
|
||||
points: ["学习 pseudo-query，计算跨层注意力权重", "覆盖 embedding、当前 block 与先前 block", "目标是改善 93 层网络中的信息与梯度流"],
|
||||
source: "K3 §2.2",
|
||||
},
|
||||
{
|
||||
id: "channel",
|
||||
label: "宽度 / Expert",
|
||||
title: "Stable LatentMoE：896 位专家里，每个 Token 只请 16 位",
|
||||
plain: "模型把“知识容量”和“本次计算量”拆开：专家库很大，但每次只激活最匹配的一小组。",
|
||||
points: ["2.8T 总参数，约 104B 激活参数", "896 个 routed experts，另有 2 个 shared experts", "Normalized LatentMoE、SiTU-GLU、Quantile Balancing 稳住极稀疏路由"],
|
||||
source: "K3 §2.3",
|
||||
},
|
||||
{
|
||||
id: "vision",
|
||||
label: "视觉 / Input",
|
||||
title: "MoonViT-V2：图像不是外部 OCR 结果，而是进入同一主干的视觉 Token",
|
||||
plain: "视觉编码器先把图像压成一串向量，再由轻量投影器把它们放进和文字相同的表示空间。",
|
||||
points: ["401M 参数视觉编码器", "训练中联合文本、图像与视频数据", "支持视觉反馈闭环：看截图、改代码、再次验证"],
|
||||
source: "K3 §2.4",
|
||||
},
|
||||
{
|
||||
id: "system",
|
||||
label: "系统 / Scale",
|
||||
title: "算法—系统协同：模型结构必须能在真实集群上被训练、强化学习和服务",
|
||||
plain: "一个公式只有在 GPU 内核、跨卡通信、显存和调度上都跑得通，才真正成为 2.8T 模型的一部分。",
|
||||
points: ["FlashKDA 与 KDA Context Parallelism", "MoonEP 的平衡专家并行与零拷贝通信", "长上下文 RL 的外置 KV Cache、可恢复 microVM 沙箱与部分 rollout"],
|
||||
source: "K3 §5",
|
||||
},
|
||||
];
|
||||
---
|
||||
|
||||
<figure class="architecture-explorer" data-architecture>
|
||||
<div class="architecture-stage" aria-label="Kimi K3 简化架构图">
|
||||
<svg viewBox="0 0 920 510" role="img" aria-labelledby="k3-arch-title k3-arch-desc">
|
||||
<title id="k3-arch-title">Kimi K3 三维信息流简化图</title>
|
||||
<desc id="k3-arch-desc">文字与图像进入模型后，依次通过由 KDA、Gated MLA 和 Stable LatentMoE 组成的模块；Attention Residuals 连接不同深度的表示。</desc>
|
||||
<defs>
|
||||
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse">
|
||||
<path d="M 0 0 L 10 5 L 0 10 z" fill="currentColor"></path>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<g class="arch-flow">
|
||||
<path d="M82 254H170M310 254H367M522 254H579M734 254H832" />
|
||||
<path class="depth-link" d="M230 207C310 68 633 62 669 205" />
|
||||
</g>
|
||||
|
||||
<g class="arch-node input-text" data-axis="token">
|
||||
<rect x="20" y="215" width="62" height="78" rx="8" />
|
||||
<text x="51" y="246">文</text>
|
||||
<text x="51" y="269">字</text>
|
||||
</g>
|
||||
|
||||
<g class="arch-node input-vision" data-axis="vision">
|
||||
<rect x="20" y="323" width="62" height="78" rx="8" />
|
||||
<circle cx="42" cy="346" r="7" />
|
||||
<path d="M30 386l16-20 11 11 15-18v27z" />
|
||||
<text x="51" y="421">MoonViT</text>
|
||||
</g>
|
||||
|
||||
<g class="arch-node embed" data-axis="vision token">
|
||||
<rect x="170" y="204" width="140" height="100" rx="8" />
|
||||
<text x="240" y="244">Embedding</text>
|
||||
<text class="sub" x="240" y="269">共享表示空间</text>
|
||||
<path d="M82 362H128V278H170" />
|
||||
</g>
|
||||
|
||||
<g class="arch-node attention kda" data-axis="token system">
|
||||
<rect x="367" y="191" width="155" height="126" rx="8" />
|
||||
<text x="444" y="233">KDA</text>
|
||||
<text class="sub" x="444" y="258">线性工作记忆</text>
|
||||
<text class="tag" x="444" y="292">× 3</text>
|
||||
</g>
|
||||
|
||||
<g class="arch-node attention mla" data-axis="token">
|
||||
<rect x="579" y="191" width="155" height="126" rx="8" />
|
||||
<text x="656" y="233">Gated MLA</text>
|
||||
<text class="sub" x="656" y="258">压缩的全局注意力</text>
|
||||
<text class="tag" x="656" y="292">× 1</text>
|
||||
</g>
|
||||
|
||||
<g class="arch-node output" data-axis="token">
|
||||
<rect x="832" y="215" width="68" height="78" rx="8" />
|
||||
<text x="866" y="248">下一个</text>
|
||||
<text x="866" y="270">Token</text>
|
||||
</g>
|
||||
|
||||
<g class="arch-node moe moe-a" data-axis="channel system">
|
||||
<rect x="367" y="350" width="155" height="92" rx="8" />
|
||||
<text x="444" y="385">Stable LatentMoE</text>
|
||||
<text class="sub" x="444" y="410">896 → 16 experts</text>
|
||||
<path d="M444 317V350" />
|
||||
</g>
|
||||
|
||||
<g class="arch-node moe moe-b" data-axis="channel system">
|
||||
<rect x="579" y="350" width="155" height="92" rx="8" />
|
||||
<text x="656" y="385">Stable LatentMoE</text>
|
||||
<text class="sub" x="656" y="410">896 → 16 experts</text>
|
||||
<path d="M656 317V350" />
|
||||
</g>
|
||||
|
||||
<g class="attn-res-label" data-axis="depth">
|
||||
<rect x="389" y="55" width="309" height="54" rx="27" />
|
||||
<text x="544" y="88">Attention Residuals · 跨层选择</text>
|
||||
</g>
|
||||
|
||||
<g class="system-base" data-axis="system">
|
||||
<rect x="170" y="470" width="564" height="24" rx="12" />
|
||||
<text x="452" y="486">FlashKDA · MoonEP · 1M RL · Prefix Cache · Fleet Scheduling</text>
|
||||
</g>
|
||||
</svg>
|
||||
<p class="figure-caption"><b>图 01</b> Kimi K3 架构的教学化简图。它强调 token、depth、channel 三个信息流维度，不代表逐算子实现；依据 K3 Technical Report Figure 2 与 §2 重绘。</p>
|
||||
</div>
|
||||
|
||||
<div class="architecture-controls" role="tablist" aria-label="选择信息流维度">
|
||||
{axes.map((axis, index) => (
|
||||
<button
|
||||
type="button"
|
||||
role="tab"
|
||||
id={`axis-${axis.id}`}
|
||||
aria-controls={`axis-panel-${axis.id}`}
|
||||
aria-selected={index === 0 ? "true" : "false"}
|
||||
data-axis-button={axis.id}
|
||||
>
|
||||
<span>{String(index + 1).padStart(2, "0")}</span>
|
||||
{axis.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="architecture-details">
|
||||
{axes.map((axis, index) => (
|
||||
<section
|
||||
id={`axis-panel-${axis.id}`}
|
||||
role="tabpanel"
|
||||
aria-labelledby={`axis-${axis.id}`}
|
||||
hidden={index !== 0}
|
||||
data-axis-panel={axis.id}
|
||||
>
|
||||
<span class="micro-label">{axis.source}</span>
|
||||
<h3>{axis.title}</h3>
|
||||
<div class="plain-language">
|
||||
<b>先用直觉说</b>
|
||||
<p>{axis.plain}</p>
|
||||
</div>
|
||||
<ul>
|
||||
{axis.points.map((point) => <li>{point}</li>)}
|
||||
</ul>
|
||||
</section>
|
||||
))}
|
||||
</div>
|
||||
</figure>
|
||||
|
||||
<script>
|
||||
document.querySelectorAll<HTMLElement>("[data-architecture]").forEach((root) => {
|
||||
const buttons = [...root.querySelectorAll<HTMLButtonElement>("[data-axis-button]")];
|
||||
const panels = [...root.querySelectorAll<HTMLElement>("[data-axis-panel]")];
|
||||
const nodes = [...root.querySelectorAll<SVGGElement>("[data-axis]")];
|
||||
|
||||
const select = (axis: string) => {
|
||||
buttons.forEach((button) => {
|
||||
const active = button.dataset.axisButton === axis;
|
||||
button.setAttribute("aria-selected", String(active));
|
||||
button.tabIndex = active ? 0 : -1;
|
||||
});
|
||||
panels.forEach((panel) => {
|
||||
panel.hidden = panel.dataset.axisPanel !== axis;
|
||||
});
|
||||
nodes.forEach((node) => {
|
||||
const active = node.dataset.axis?.split(" ").includes(axis);
|
||||
node.toggleAttribute("data-highlight", Boolean(active));
|
||||
});
|
||||
};
|
||||
|
||||
buttons.forEach((button, index) => {
|
||||
button.addEventListener("click", () => select(button.dataset.axisButton ?? "token"));
|
||||
button.addEventListener("keydown", (event) => {
|
||||
if (!["ArrowLeft", "ArrowRight"].includes(event.key)) return;
|
||||
event.preventDefault();
|
||||
const delta = event.key === "ArrowRight" ? 1 : -1;
|
||||
const next = buttons[(index + delta + buttons.length) % buttons.length];
|
||||
next.focus();
|
||||
select(next.dataset.axisButton ?? "token");
|
||||
});
|
||||
});
|
||||
select("token");
|
||||
});
|
||||
</script>
|
||||
|
||||
<style>
|
||||
.architecture-explorer {
|
||||
max-width: 980px;
|
||||
margin: 38px 0;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.architecture-stage {
|
||||
padding: 30px 30px 18px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
svg {
|
||||
width: 100%;
|
||||
height: auto;
|
||||
overflow: visible;
|
||||
color: var(--muted-light);
|
||||
}
|
||||
|
||||
.arch-flow path {
|
||||
fill: none;
|
||||
stroke: currentColor;
|
||||
stroke-width: 1.5;
|
||||
marker-end: url(#arrow);
|
||||
}
|
||||
|
||||
.arch-flow .depth-link {
|
||||
stroke-dasharray: 6 6;
|
||||
}
|
||||
|
||||
.arch-node rect,
|
||||
.attn-res-label rect,
|
||||
.system-base rect {
|
||||
fill: var(--paper);
|
||||
stroke: var(--line-strong);
|
||||
stroke-width: 1.2;
|
||||
transition: fill 160ms ease, stroke 160ms ease, opacity 160ms ease;
|
||||
}
|
||||
|
||||
.arch-node text,
|
||||
.attn-res-label text,
|
||||
.system-base text {
|
||||
fill: var(--ink);
|
||||
font-family: var(--display);
|
||||
font-size: 15px;
|
||||
font-weight: 650;
|
||||
text-anchor: middle;
|
||||
}
|
||||
|
||||
.arch-node .sub {
|
||||
fill: var(--muted);
|
||||
font-size: 11px;
|
||||
font-weight: 450;
|
||||
}
|
||||
|
||||
.arch-node .tag {
|
||||
fill: var(--copper);
|
||||
font-family: var(--mono);
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.arch-node path {
|
||||
fill: none;
|
||||
stroke: var(--line-strong);
|
||||
stroke-width: 1.4;
|
||||
}
|
||||
|
||||
.input-vision circle {
|
||||
fill: var(--copper);
|
||||
}
|
||||
|
||||
.input-vision > path:first-of-type {
|
||||
fill: var(--sky-pale);
|
||||
stroke: var(--sky);
|
||||
}
|
||||
|
||||
.input-vision text {
|
||||
fill: var(--muted);
|
||||
font-family: var(--mono);
|
||||
font-size: 8px;
|
||||
}
|
||||
|
||||
[data-axis] {
|
||||
opacity: 0.42;
|
||||
transition: opacity 160ms ease;
|
||||
}
|
||||
|
||||
[data-axis][data-highlight] {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
[data-highlight].attention rect {
|
||||
fill: var(--sky-pale);
|
||||
stroke: var(--sky);
|
||||
}
|
||||
|
||||
[data-highlight].moe rect {
|
||||
fill: var(--copper-pale);
|
||||
stroke: var(--copper);
|
||||
}
|
||||
|
||||
[data-highlight].attn-res-label rect {
|
||||
fill: var(--violet-pale);
|
||||
stroke: var(--violet);
|
||||
}
|
||||
|
||||
[data-highlight].input-vision rect {
|
||||
fill: var(--sage-pale);
|
||||
stroke: var(--sage);
|
||||
}
|
||||
|
||||
[data-highlight].system-base rect {
|
||||
fill: var(--sage-pale);
|
||||
stroke: var(--sage);
|
||||
}
|
||||
|
||||
.attn-res-label text,
|
||||
.system-base text {
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.figure-caption {
|
||||
margin-top: 12px;
|
||||
color: var(--muted);
|
||||
font-size: 0.72rem;
|
||||
line-height: 1.65;
|
||||
}
|
||||
|
||||
.architecture-controls {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(5, minmax(0, 1fr));
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.architecture-controls button {
|
||||
display: grid;
|
||||
gap: 8px;
|
||||
min-height: 72px;
|
||||
padding: 13px;
|
||||
border: 0;
|
||||
border-right: 1px solid var(--line);
|
||||
background: transparent;
|
||||
color: var(--muted);
|
||||
text-align: left;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.architecture-controls button:last-child {
|
||||
border-right: 0;
|
||||
}
|
||||
|
||||
.architecture-controls button span {
|
||||
color: var(--muted-light);
|
||||
font: 0.61rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.architecture-controls button[aria-selected="true"] {
|
||||
background: var(--ink);
|
||||
color: var(--paper-raised);
|
||||
}
|
||||
|
||||
.architecture-controls button[aria-selected="true"] span {
|
||||
color: var(--copper-pale);
|
||||
}
|
||||
|
||||
.architecture-details {
|
||||
min-height: 355px;
|
||||
padding: 32px;
|
||||
}
|
||||
|
||||
.architecture-details h3 {
|
||||
max-width: 800px;
|
||||
margin: 15px 0 22px;
|
||||
font-size: clamp(1.4rem, 2.5vw, 2rem);
|
||||
}
|
||||
|
||||
.architecture-details ul {
|
||||
display: grid;
|
||||
gap: 9px;
|
||||
margin: 22px 0 0;
|
||||
padding-left: 1.2rem;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
@media (max-width: 720px) {
|
||||
.architecture-stage {
|
||||
padding: 16px 10px;
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
svg {
|
||||
min-width: 720px;
|
||||
}
|
||||
|
||||
.architecture-controls {
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
}
|
||||
|
||||
.architecture-controls button {
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.architecture-details {
|
||||
min-height: 440px;
|
||||
padding: 23px;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
const tokens = ["小猫", "坐在", "柔软的", "垫子", "上"];
|
||||
const weights = [
|
||||
[0.44, 0.14, 0.08, 0.26, 0.08],
|
||||
[0.31, 0.24, 0.08, 0.25, 0.12],
|
||||
[0.07, 0.08, 0.26, 0.51, 0.08],
|
||||
[0.25, 0.09, 0.32, 0.27, 0.07],
|
||||
[0.07, 0.15, 0.05, 0.55, 0.18],
|
||||
];
|
||||
---
|
||||
|
||||
<figure class="attention-lab" data-attention-lab data-weights={JSON.stringify(weights)}>
|
||||
<div class="lab-header">
|
||||
<div>
|
||||
<span class="micro-label">INTERACTIVE / SELF-ATTENTION</span>
|
||||
<h3>点一个词，看它“回头看”谁</h3>
|
||||
</div>
|
||||
<p>示意权重不是训练模型的真实输出；它只帮助理解一次注意力查询的流程。</p>
|
||||
</div>
|
||||
|
||||
<div class="token-row" role="tablist" aria-label="选择作为 Query 的词">
|
||||
{tokens.map((token, index) => (
|
||||
<button
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected={index === 4 ? "true" : "false"}
|
||||
data-query={index}
|
||||
>
|
||||
<span>{index + 1}</span>{token}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="attention-result">
|
||||
<div class="query-card">
|
||||
<span>QUERY / 我正在理解</span>
|
||||
<strong data-query-label>上</strong>
|
||||
<p>Query 是当前词提出的问题：“为了更新我的表示，我应该从哪些词取信息？”</p>
|
||||
</div>
|
||||
<div class="weight-list" aria-live="polite">
|
||||
{tokens.map((token, index) => (
|
||||
<div class="weight-item">
|
||||
<span>{token}</span>
|
||||
<div><i data-bar={index} style={`width:${weights[4][index] * 100}%`}></i></div>
|
||||
<b data-value={index}>{Math.round(weights[4][index] * 100)}%</b>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<figcaption><b>图 02</b> 单个注意力头的教学示意。真正的模型会在每一层、每个头上并行进行类似计算，权重由 QKᵀ 经缩放和 softmax 得到。</figcaption>
|
||||
</figure>
|
||||
|
||||
<script>
|
||||
document.querySelectorAll<HTMLElement>("[data-attention-lab]").forEach((root) => {
|
||||
const matrix = JSON.parse(root.dataset.weights ?? "[]") as number[][];
|
||||
const labels = ["小猫", "坐在", "柔软的", "垫子", "上"];
|
||||
const buttons = [...root.querySelectorAll<HTMLButtonElement>("[data-query]")];
|
||||
const queryLabel = root.querySelector<HTMLElement>("[data-query-label]");
|
||||
|
||||
const select = (index: number) => {
|
||||
buttons.forEach((button, buttonIndex) => {
|
||||
button.setAttribute("aria-selected", String(buttonIndex === index));
|
||||
});
|
||||
if (queryLabel) queryLabel.textContent = labels[index];
|
||||
matrix[index]?.forEach((weight, targetIndex) => {
|
||||
const bar = root.querySelector<HTMLElement>(`[data-bar="${targetIndex}"]`);
|
||||
const value = root.querySelector<HTMLElement>(`[data-value="${targetIndex}"]`);
|
||||
if (bar) bar.style.width = `${weight * 100}%`;
|
||||
if (value) value.textContent = `${Math.round(weight * 100)}%`;
|
||||
});
|
||||
};
|
||||
|
||||
buttons.forEach((button) => {
|
||||
button.addEventListener("click", () => select(Number(button.dataset.query ?? 0)));
|
||||
});
|
||||
});
|
||||
</script>
|
||||
|
||||
<style>
|
||||
.attention-lab {
|
||||
max-width: 900px;
|
||||
margin: 38px 0;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.lab-header {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 270px;
|
||||
gap: 34px;
|
||||
align-items: end;
|
||||
padding: 30px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.lab-header h3 {
|
||||
margin: 12px 0 0;
|
||||
font-size: 1.65rem;
|
||||
}
|
||||
|
||||
.lab-header p {
|
||||
color: var(--muted);
|
||||
font-size: 0.74rem;
|
||||
line-height: 1.7;
|
||||
}
|
||||
|
||||
.token-row {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(5, minmax(0, 1fr));
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.token-row button {
|
||||
display: grid;
|
||||
gap: 9px;
|
||||
min-height: 78px;
|
||||
padding: 14px;
|
||||
border: 0;
|
||||
border-right: 1px solid var(--line);
|
||||
background: transparent;
|
||||
color: var(--muted);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.token-row button:last-child {
|
||||
border-right: 0;
|
||||
}
|
||||
|
||||
.token-row button span {
|
||||
color: var(--muted-light);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.token-row button[aria-selected="true"] {
|
||||
background: var(--ink);
|
||||
color: var(--paper-raised);
|
||||
}
|
||||
|
||||
.attention-result {
|
||||
display: grid;
|
||||
grid-template-columns: 240px 1fr;
|
||||
}
|
||||
|
||||
.query-card {
|
||||
padding: 30px;
|
||||
background: var(--paper-deep);
|
||||
}
|
||||
|
||||
.query-card span {
|
||||
color: var(--muted);
|
||||
font: 0.62rem/1.5 var(--mono);
|
||||
}
|
||||
|
||||
.query-card strong {
|
||||
display: block;
|
||||
margin: 22px 0;
|
||||
color: var(--copper);
|
||||
font-size: 2.6rem;
|
||||
}
|
||||
|
||||
.query-card p {
|
||||
color: var(--muted);
|
||||
font-size: 0.75rem;
|
||||
line-height: 1.75;
|
||||
}
|
||||
|
||||
.weight-list {
|
||||
display: grid;
|
||||
gap: 15px;
|
||||
padding: 30px;
|
||||
}
|
||||
|
||||
.weight-item {
|
||||
display: grid;
|
||||
grid-template-columns: 70px 1fr 42px;
|
||||
gap: 14px;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.weight-item > span,
|
||||
.weight-item > b {
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
|
||||
.weight-item > b {
|
||||
color: var(--muted);
|
||||
font-family: var(--mono);
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
.weight-item div {
|
||||
height: 10px;
|
||||
overflow: hidden;
|
||||
background: var(--paper-deep);
|
||||
}
|
||||
|
||||
.weight-item i {
|
||||
display: block;
|
||||
height: 100%;
|
||||
background: linear-gradient(90deg, var(--sky), var(--copper));
|
||||
transition: width 280ms ease;
|
||||
}
|
||||
|
||||
figcaption {
|
||||
padding: 14px 30px;
|
||||
border-top: 1px solid var(--line);
|
||||
color: var(--muted);
|
||||
font-size: 0.72rem;
|
||||
line-height: 1.65;
|
||||
}
|
||||
|
||||
@media (max-width: 720px) {
|
||||
.lab-header,
|
||||
.attention-result {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.token-row {
|
||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||||
}
|
||||
|
||||
.token-row button {
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
const milestones = [
|
||||
{
|
||||
year: "2024.01",
|
||||
model: "DeepSeek LLM",
|
||||
idea: "公开尺度规律与中英双语预训练，建立 7B / 67B dense 基线。",
|
||||
bridge: "先弄清规模、数据与训练配方，再做稀疏化。",
|
||||
url: "https://arxiv.org/abs/2401.02954",
|
||||
},
|
||||
{
|
||||
year: "2024.01",
|
||||
model: "DeepSeekMoE",
|
||||
idea: "细粒度专家分割 + shared experts，让专家更专、公共知识不必重复。",
|
||||
bridge: "把容量扩张和每 Token 计算量分开。",
|
||||
url: "https://arxiv.org/abs/2401.06066",
|
||||
},
|
||||
{
|
||||
year: "2024.05",
|
||||
model: "DeepSeek-V2",
|
||||
idea: "MLA 压缩 KV Cache；DeepSeekMoE 扩张稀疏容量。",
|
||||
bridge: "训练经济性之外，开始直接优化推理内存与吞吐。",
|
||||
url: "https://arxiv.org/abs/2405.04434",
|
||||
},
|
||||
{
|
||||
year: "2024.12",
|
||||
model: "DeepSeek-V3",
|
||||
idea: "671B-A37B、FP8 训练、无辅助损失负载均衡、MTP 与 DualPipe。",
|
||||
bridge: "模型算法、数值格式与集群通信共同设计。",
|
||||
url: "https://arxiv.org/abs/2412.19437",
|
||||
},
|
||||
{
|
||||
year: "2025.01",
|
||||
model: "DeepSeek-R1",
|
||||
idea: "R1-Zero 展示纯大规模 RL 可涌现推理；R1 用冷启动数据修复可读性与稳定性。",
|
||||
bridge: "从“模仿答案”转向用可验证奖励塑造推理策略。",
|
||||
url: "https://arxiv.org/abs/2501.12948",
|
||||
},
|
||||
{
|
||||
year: "2025.12",
|
||||
model: "DeepSeek-V3.2",
|
||||
idea: "DeepSeek Sparse Attention 降低长上下文成本，并统一 thinking 与 tool use。",
|
||||
bridge: "把推理模型推进长上下文 Agent 场景。",
|
||||
url: "https://arxiv.org/abs/2512.02556",
|
||||
},
|
||||
{
|
||||
year: "2026.06",
|
||||
model: "DeepSeek-V4",
|
||||
idea: "围绕百万 Token 上下文效率继续扩展，成为 K3 报告直接比较的开放前沿之一。",
|
||||
bridge: "长上下文不再只是位置外推，而是注意力、训练与服务的全系统问题。",
|
||||
url: "https://arxiv.org/abs/2606.19348",
|
||||
},
|
||||
];
|
||||
---
|
||||
|
||||
<div class="deepseek-lineage">
|
||||
{milestones.map((item, index) => (
|
||||
<a href={item.url} class="lineage-row" rel="noreferrer">
|
||||
<div class="lineage-time">
|
||||
<span>{item.year}</span>
|
||||
<i aria-hidden="true"></i>
|
||||
</div>
|
||||
<div class="lineage-main">
|
||||
<span>DS / {String(index + 1).padStart(2, "0")}</span>
|
||||
<h3>{item.model}</h3>
|
||||
<p>{item.idea}</p>
|
||||
</div>
|
||||
<div class="lineage-bridge">
|
||||
<span>通向下一步</span>
|
||||
<p>{item.bridge}</p>
|
||||
</div>
|
||||
</a>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.deepseek-lineage {
|
||||
max-width: 960px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.lineage-row {
|
||||
display: grid;
|
||||
grid-template-columns: 110px minmax(250px, 1fr) minmax(230px, 0.75fr);
|
||||
gap: 28px;
|
||||
padding: 28px 8px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
text-decoration: none;
|
||||
transition: background-color 160ms ease;
|
||||
}
|
||||
|
||||
.lineage-row:hover {
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.lineage-time {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 12px;
|
||||
gap: 10px;
|
||||
color: var(--copper);
|
||||
font: 0.7rem/1.5 var(--mono);
|
||||
}
|
||||
|
||||
.lineage-time i {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
margin-top: 3px;
|
||||
border: 2px solid var(--copper);
|
||||
border-radius: 50%;
|
||||
background: var(--paper);
|
||||
}
|
||||
|
||||
.lineage-main > span,
|
||||
.lineage-bridge > span {
|
||||
color: var(--muted-light);
|
||||
font: 0.6rem/1 var(--mono);
|
||||
letter-spacing: 0.1em;
|
||||
}
|
||||
|
||||
.lineage-main h3 {
|
||||
margin: 10px 0 11px;
|
||||
font-size: 1.25rem;
|
||||
}
|
||||
|
||||
.lineage-main p,
|
||||
.lineage-bridge p {
|
||||
color: var(--muted);
|
||||
font-size: 0.8rem;
|
||||
line-height: 1.7;
|
||||
}
|
||||
|
||||
.lineage-bridge {
|
||||
padding-left: 24px;
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.lineage-bridge p {
|
||||
margin-top: 10px;
|
||||
}
|
||||
|
||||
@media (max-width: 720px) {
|
||||
.lineage-row {
|
||||
grid-template-columns: 75px 1fr;
|
||||
}
|
||||
|
||||
.lineage-bridge {
|
||||
grid-column: 2;
|
||||
padding-left: 0;
|
||||
border-left: 0;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,13 @@
|
||||
<footer class="site-footer">
|
||||
<div>
|
||||
<a class="footer-mark" href="/">L/A</a>
|
||||
<p>从第一性原理到 Kimi K3：一套持续更新、证据可追溯的中文 LLM 技术课程。</p>
|
||||
</div>
|
||||
<div class="footer-links">
|
||||
<a href="/roadmap/">学习地图</a>
|
||||
<a href="/progress/">研究进度</a>
|
||||
<a href="https://git.k1412.top/wuyang/llm-atlas" rel="noreferrer">开放源码</a>
|
||||
<a href="https://github.com/MoonshotAI/Kimi-K3" rel="noreferrer">K3 官方报告</a>
|
||||
</div>
|
||||
<p class="footer-legal">代码 MIT · 原创内容 CC BY-SA 4.0 · 研究截止 2026-07-28</p>
|
||||
</footer>
|
||||
@@ -0,0 +1,50 @@
|
||||
---
|
||||
interface Props {
|
||||
active?: string;
|
||||
}
|
||||
|
||||
const { active = "" } = Astro.props;
|
||||
const items = [
|
||||
{ id: "home", href: "/", label: "首页" },
|
||||
{ id: "roadmap", href: "/roadmap/", label: "学习地图" },
|
||||
{ id: "k3", href: "/k3/", label: "K3 解剖" },
|
||||
{ id: "deepseek", href: "/deepseek/", label: "DeepSeek" },
|
||||
{ id: "foundations", href: "/foundations/", label: "基础原理" },
|
||||
{ id: "papers", href: "/papers/", label: "论文库" },
|
||||
{ id: "progress", href: "/progress/", label: "进度" },
|
||||
];
|
||||
---
|
||||
|
||||
<header class="site-header" id="top">
|
||||
<a class="wordmark" href="/" aria-label="LLM Atlas 首页">
|
||||
<span class="wordmark-mark">L/A</span>
|
||||
<span class="wordmark-copy">
|
||||
<b>LLM ATLAS</b>
|
||||
<small>FROM FIRST PRINCIPLES TO KIMI K3</small>
|
||||
</span>
|
||||
</a>
|
||||
<nav class="top-nav" aria-label="主要导航">
|
||||
{items.map((item) => (
|
||||
<a href={item.href} aria-current={active === item.id ? "page" : undefined}>{item.label}</a>
|
||||
))}
|
||||
</nav>
|
||||
<div class="header-meta">
|
||||
<span>RESEARCH · 2026</span>
|
||||
<button id="menu-toggle" class="menu-toggle" type="button" aria-expanded="false" aria-controls="mobile-nav">目录</button>
|
||||
</div>
|
||||
<nav id="mobile-nav" class="mobile-nav" aria-label="移动导航">
|
||||
{items.map((item) => (
|
||||
<a href={item.href} aria-current={active === item.id ? "page" : undefined}>{item.label}</a>
|
||||
))}
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<script>
|
||||
const toggle = document.querySelector<HTMLButtonElement>("#menu-toggle");
|
||||
const nav = document.querySelector<HTMLElement>("#mobile-nav");
|
||||
toggle?.addEventListener("click", () => {
|
||||
const open = toggle.getAttribute("aria-expanded") === "true";
|
||||
toggle.setAttribute("aria-expanded", String(!open));
|
||||
nav?.toggleAttribute("data-open", !open);
|
||||
});
|
||||
</script>
|
||||
@@ -0,0 +1,233 @@
|
||||
export type ChapterStatus = "published" | "drafting" | "researching" | "queued";
|
||||
|
||||
export interface Chapter {
|
||||
number: string;
|
||||
slug: string;
|
||||
title: string;
|
||||
kicker: string;
|
||||
question: string;
|
||||
summary: string;
|
||||
status: ChapterStatus;
|
||||
progress: number;
|
||||
papers: number;
|
||||
prerequisites: string[];
|
||||
highlights: string[];
|
||||
}
|
||||
|
||||
export const chapters: Chapter[] = [
|
||||
{
|
||||
number: "00",
|
||||
slug: "roadmap",
|
||||
title: "先看懂一张大模型地图",
|
||||
kicker: "ORIENTATION",
|
||||
question: "面对几百个术语，我们究竟先学什么？",
|
||||
summary: "建立问题链、依赖关系、论文阅读方法和证据等级；把 K3 放回完整技术地图。",
|
||||
status: "published",
|
||||
progress: 72,
|
||||
papers: 12,
|
||||
prerequisites: [],
|
||||
highlights: ["四层难度", "一张依赖图", "阅读路径"],
|
||||
},
|
||||
{
|
||||
number: "01",
|
||||
slug: "foundations/language-models",
|
||||
title: "语言模型从哪里来",
|
||||
kicker: "LANGUAGE MODELING",
|
||||
question: "预测下一个 Token，为什么能产生通用能力？",
|
||||
summary: "从 N-gram、神经概率语言模型、词向量一路走到 Seq2Seq，理解 Transformer 出现前的瓶颈。",
|
||||
status: "researching",
|
||||
progress: 24,
|
||||
papers: 10,
|
||||
prerequisites: [],
|
||||
highlights: ["概率分解", "分布式表示", "序列瓶颈"],
|
||||
},
|
||||
{
|
||||
number: "02",
|
||||
slug: "foundations",
|
||||
title: "注意力与 Transformer",
|
||||
kicker: "TRANSFORMER",
|
||||
question: "一句话里的每个词，怎样直接找到真正相关的词？",
|
||||
summary: "用可操作的小例子拆开 Q、K、V、自注意力、多头、因果掩码、残差与前馈网络。",
|
||||
status: "published",
|
||||
progress: 45,
|
||||
papers: 12,
|
||||
prerequisites: ["01"],
|
||||
highlights: ["Q / K / V", "交互实验", "张量形状"],
|
||||
},
|
||||
{
|
||||
number: "03",
|
||||
slug: "architecture/representation",
|
||||
title: "表示、位置与残差高速公路",
|
||||
kicker: "REPRESENTATION",
|
||||
question: "模型如何知道词序，又如何让信息穿过上百层？",
|
||||
summary: "从分词、位置编码、归一化与激活函数，走到深层网络的信息流和 Attention Residuals。",
|
||||
status: "researching",
|
||||
progress: 18,
|
||||
papers: 16,
|
||||
prerequisites: ["02"],
|
||||
highlights: ["RoPE", "RMSNorm", "AttnRes"],
|
||||
},
|
||||
{
|
||||
number: "04",
|
||||
slug: "scaling",
|
||||
title: "Scaling Laws：规模为什么有效",
|
||||
kicker: "SCALING",
|
||||
question: "多大模型、多少数据、多少计算才是划算的？",
|
||||
summary: "区分参数量、激活参数、训练计算和推理预算；理解 Kaplan、Chinchilla 与 K3 的双轴扩展。",
|
||||
status: "researching",
|
||||
progress: 22,
|
||||
papers: 11,
|
||||
prerequisites: ["01", "02"],
|
||||
highlights: ["幂律", "计算最优", "测试时扩展"],
|
||||
},
|
||||
{
|
||||
number: "05",
|
||||
slug: "pretraining/data",
|
||||
title: "数据工程与预训练配方",
|
||||
kicker: "DATA",
|
||||
question: "更多网页为什么不等于更好的模型？",
|
||||
summary: "追踪采集、过滤、去重、混合、课程、合成数据与污染控制，并明确公开报告的知识边界。",
|
||||
status: "queued",
|
||||
progress: 8,
|
||||
papers: 14,
|
||||
prerequisites: ["01", "04"],
|
||||
highlights: ["数据质量", "混合策略", "污染"],
|
||||
},
|
||||
{
|
||||
number: "06",
|
||||
slug: "architecture/moe",
|
||||
title: "稀疏计算与 MoE",
|
||||
kicker: "SPARSE EXPERTS",
|
||||
question: "怎样让模型装下更多知识，却不让每个 Token 都付全部算力？",
|
||||
summary: "从条件计算到 DeepSeekMoE、LatentMoE 与 K3 Stable LatentMoE，解释路由、特化和负载均衡。",
|
||||
status: "researching",
|
||||
progress: 28,
|
||||
papers: 17,
|
||||
prerequisites: ["02", "04"],
|
||||
highlights: ["专家路由", "细粒度专家", "896 选 16"],
|
||||
},
|
||||
{
|
||||
number: "07",
|
||||
slug: "architecture/long-context",
|
||||
title: "长上下文与高效注意力",
|
||||
kicker: "LONG CONTEXT",
|
||||
question: "从 8K 到 1M Token，真正昂贵的是什么？",
|
||||
summary: "比较稀疏/线性注意力、FlashAttention、MLA、状态空间模型、Delta Rule、KDA 与混合注意力。",
|
||||
status: "researching",
|
||||
progress: 31,
|
||||
papers: 24,
|
||||
prerequisites: ["02", "03"],
|
||||
highlights: ["KV Cache", "MLA", "KDA"],
|
||||
},
|
||||
{
|
||||
number: "08",
|
||||
slug: "systems/training",
|
||||
title: "大规模训练系统",
|
||||
kicker: "DISTRIBUTED TRAINING",
|
||||
question: "一个 2.8T 模型如何摊到成千上万张卡上？",
|
||||
summary: "从 ZeRO、Megatron 和五类并行，走到 DeepSeek DualPipe/DeepEP 与 K3 MoonEP。",
|
||||
status: "queued",
|
||||
progress: 12,
|
||||
papers: 18,
|
||||
prerequisites: ["04", "06"],
|
||||
highlights: ["并行维度", "通信重叠", "MoonEP"],
|
||||
},
|
||||
{
|
||||
number: "09",
|
||||
slug: "systems/numerics",
|
||||
title: "数值精度、优化器与稳定性",
|
||||
kicker: "OPTIMIZATION",
|
||||
question: "为什么少几个比特能省巨资，也可能让训练瞬间崩掉？",
|
||||
summary: "解释 AdamW、μP、Muon、BF16/FP8/MXFP4 与量化感知训练的数值直觉。",
|
||||
status: "queued",
|
||||
progress: 13,
|
||||
papers: 16,
|
||||
prerequisites: ["02", "08"],
|
||||
highlights: ["FP8", "Muon", "MXFP4"],
|
||||
},
|
||||
{
|
||||
number: "10",
|
||||
slug: "post-training/alignment",
|
||||
title: "指令微调与人类偏好",
|
||||
kicker: "ALIGNMENT",
|
||||
question: "会续写的 base model，怎样变成愿意协作的助手？",
|
||||
summary: "从 instruction tuning、SFT、RLHF/PPO 到 DPO 与 RLAIF，区分能力学习和行为塑形。",
|
||||
status: "queued",
|
||||
progress: 10,
|
||||
papers: 18,
|
||||
prerequisites: ["01", "02"],
|
||||
highlights: ["SFT", "奖励模型", "偏好优化"],
|
||||
},
|
||||
{
|
||||
number: "11",
|
||||
slug: "reasoning",
|
||||
title: "推理模型与测试时扩展",
|
||||
kicker: "REASONING",
|
||||
question: "模型如何学会多想一会儿，并检查自己的答案？",
|
||||
summary: "从 CoT、搜索与验证器，到 GRPO、DeepSeek-R1、Kimi k1.5 和 multi-effort RL。",
|
||||
status: "researching",
|
||||
progress: 25,
|
||||
papers: 21,
|
||||
prerequisites: ["10"],
|
||||
highlights: ["GRPO", "R1-Zero", "On-policy 蒸馏"],
|
||||
},
|
||||
{
|
||||
number: "12",
|
||||
slug: "agents",
|
||||
title: "工具使用与长程 Agent",
|
||||
kicker: "AGENTS",
|
||||
question: "生成一段文字，怎样变成持续数小时的可靠行动？",
|
||||
summary: "讨论工具协议、行动—观察循环、环境奖励、沙箱、可验证任务与百万 Token 轨迹。",
|
||||
status: "queued",
|
||||
progress: 14,
|
||||
papers: 22,
|
||||
prerequisites: ["10", "11"],
|
||||
highlights: ["ReAct", "沙箱", "长轨迹 RL"],
|
||||
},
|
||||
{
|
||||
number: "13",
|
||||
slug: "multimodal",
|
||||
title: "原生多模态",
|
||||
kicker: "MULTIMODAL",
|
||||
question: "图像是外接插件，还是和文字一样的第一类输入？",
|
||||
summary: "从 ViT/CLIP、连接器式 VLM 走到 Kimi-VL、MoonViT-V2 与统一主干。",
|
||||
status: "queued",
|
||||
progress: 11,
|
||||
papers: 19,
|
||||
prerequisites: ["02"],
|
||||
highlights: ["ViT", "视觉 Token", "MoonViT-V2"],
|
||||
},
|
||||
{
|
||||
number: "14",
|
||||
slug: "systems/inference",
|
||||
title: "推理服务与低成本部署",
|
||||
kicker: "INFERENCE",
|
||||
question: "模型训练完以后，怎样让千万人用得起？",
|
||||
summary: "拆解 KV Cache、PagedAttention、批处理、推测解码、PD 解耦、前缀缓存与集群调度。",
|
||||
status: "queued",
|
||||
progress: 12,
|
||||
papers: 20,
|
||||
prerequisites: ["07", "08", "09"],
|
||||
highlights: ["vLLM", "Mooncake", "KDA 缓存"],
|
||||
},
|
||||
{
|
||||
number: "15",
|
||||
slug: "evaluation",
|
||||
title: "评测、安全与“到底强不强”",
|
||||
kicker: "EVALUATION",
|
||||
question: "一个榜单分数，究竟测到了模型、脚手架还是预算？",
|
||||
summary: "从语言建模指标走到知识、代码、Agent 和多模态评测，识别污染、harness 与选择性报告。",
|
||||
status: "queued",
|
||||
progress: 13,
|
||||
papers: 24,
|
||||
prerequisites: ["04", "10", "12"],
|
||||
highlights: ["基准演化", "Harness", "安全边界"],
|
||||
},
|
||||
];
|
||||
|
||||
export const statusLabel: Record<ChapterStatus, string> = {
|
||||
published: "首版可读",
|
||||
drafting: "写作中",
|
||||
researching: "研究中",
|
||||
queued: "待展开",
|
||||
};
|
||||
+1058
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,55 @@
|
||||
---
|
||||
import SiteHeader from "@/components/SiteHeader.astro";
|
||||
import SiteFooter from "@/components/SiteFooter.astro";
|
||||
import "@/styles/global.css";
|
||||
|
||||
interface Props {
|
||||
title: string;
|
||||
description: string;
|
||||
section?: string;
|
||||
}
|
||||
|
||||
const {
|
||||
title,
|
||||
description,
|
||||
section = "",
|
||||
} = Astro.props;
|
||||
|
||||
const pageTitle = title === "LLM Atlas" ? title : `${title}｜LLM Atlas`;
|
||||
---
|
||||
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<meta name="theme-color" content="#27364a" />
|
||||
<meta name="description" content={description} />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:title" content={pageTitle} />
|
||||
<meta property="og:description" content={description} />
|
||||
<meta property="og:locale" content="zh_CN" />
|
||||
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
|
||||
<title>{pageTitle}</title>
|
||||
</head>
|
||||
<body data-section={section}>
|
||||
<div class="reading-progress" aria-hidden="true"><span id="reading-progress"></span></div>
|
||||
<SiteHeader active={section} />
|
||||
<main>
|
||||
<slot />
|
||||
</main>
|
||||
<SiteFooter />
|
||||
<script is:inline>
|
||||
const bar = document.querySelector("#reading-progress");
|
||||
const updateProgress = () => {
|
||||
if (!bar) return;
|
||||
const root = document.documentElement;
|
||||
const distance = root.scrollHeight - root.clientHeight;
|
||||
const ratio = distance > 0 ? root.scrollTop / distance : 0;
|
||||
bar.style.transform = `scaleX(${Math.min(1, Math.max(0, ratio))})`;
|
||||
};
|
||||
addEventListener("scroll", updateProgress, { passive: true });
|
||||
updateProgress();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,815 @@
|
||||
---
|
||||
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||
import DeepSeekLineage from "@/components/DeepSeekLineage.astro";
|
||||
|
||||
const toc = [
|
||||
["00", "lineage", "先看完整论文谱系"],
|
||||
["01", "dense", "LLM：先建立 Dense 基线"],
|
||||
["02", "moe", "DeepSeekMoE：专家特化"],
|
||||
["03", "mla", "V2：MLA 压缩 KV Cache"],
|
||||
["04", "v3", "V3：算法—系统协同"],
|
||||
["05", "grpo", "DeepSeekMath：GRPO"],
|
||||
["06", "r1", "R1：RL 涌现推理"],
|
||||
["07", "v32", "V3.2：稀疏注意力与 Agent"],
|
||||
["08", "v4", "V4：百万上下文"],
|
||||
["09", "k3", "DeepSeek 怎样流入 K3"],
|
||||
["↳", "papers", "精读顺序与来源"],
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="DeepSeek 论文谱系：从 MoE、MLA 到 R1 与 V4"
|
||||
description="沿 DeepSeek LLM、DeepSeekMoE、V2、V3、DeepSeekMath、R1、V3.2 和 V4 的关键论文，理解 MoE、MLA、FP8、GRPO 与长上下文。"
|
||||
section="deepseek"
|
||||
>
|
||||
<header class="page-hero deepseek-hero">
|
||||
<div class="page-hero-inner">
|
||||
<div>
|
||||
<p class="eyebrow"><span>SPOTLIGHT / DEEPSEEK</span> ALGORITHM × SYSTEM</p>
|
||||
<h1>一条少见的、环环相扣的<br />开放论文主线</h1>
|
||||
<p class="lead">
|
||||
DeepSeek 的价值不只在某个模型分数，而在持续公开“为什么这样设计”：
|
||||
容量贵，就做细粒度 MoE；KV Cache 贵，就做 MLA；训练贵，就做 FP8 与通信重叠；
|
||||
推理难，就把可验证奖励规模化。
|
||||
</p>
|
||||
</div>
|
||||
<dl class="page-facts">
|
||||
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
||||
<div><dt>CORE</dt><dd>MoE · MLA · FP8 · GRPO</dd></div>
|
||||
<div><dt>LINE</dt><dd>Dense → Sparse → Reasoning</dd></div>
|
||||
<div><dt>K3 LINK</dt><dd>MLA · MoE · Muon · 1M</dd></div>
|
||||
<div><dt>STATUS</dt><dd>重点专题 · 首版</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="report-shell">
|
||||
<aside class="side-rail" aria-label="本页目录">
|
||||
<p>CONTENTS</p>
|
||||
<ol>
|
||||
{toc.map(([number, id, label]) => (
|
||||
<li><a href={`#${id}`}><span>{number}</span>{label}</a></li>
|
||||
))}
|
||||
</ol>
|
||||
<div class="rail-note">
|
||||
<b>本页选择</b>
|
||||
聚焦通用模型主线；Coder、Prover、VL/OCR、Engram 与硬件论文将在对应专题展开。
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<article class="article">
|
||||
<section class="article-section" id="lineage">
|
||||
<p class="eyebrow"><span>00</span> THE LINEAGE</p>
|
||||
<h2>每一代都在偿还上一代最昂贵的账单</h2>
|
||||
<p class="lede">
|
||||
把 DeepSeek 看成模型名字序列会很乱；把它看成“容量、缓存、训练、推理、长上下文”五张账单，
|
||||
技术演进就清楚了。
|
||||
</p>
|
||||
<DeepSeekLineage />
|
||||
<div class="thesis">
|
||||
<span class="micro-label">核心观察</span>
|
||||
<p>
|
||||
DeepSeek 的强项是把模型结构和硬件约束写在同一张设计图里。MLA 不只是新 attention，
|
||||
它直接针对服务时 KV Cache；FP8 不只是少用几个比特，它要求累加精度、缩放和通信共同配合；
|
||||
GRPO 不只是 PPO 变体，它直接移除同规模 critic 的显存负担。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="dense">
|
||||
<p class="eyebrow"><span>01</span> DEEPSEEK LLM</p>
|
||||
<h2>先用 Dense 模型建立基线：规模、数据和双语能力</h2>
|
||||
<p>
|
||||
2024 年初的 <a href="https://arxiv.org/abs/2401.02954">DeepSeek LLM</a> 发布 7B 与 67B dense 模型，
|
||||
在约 2T 中英文 Token 上预训练。它的重要性不在今天看来并不夸张的参数量，而在于建立后续研究的可比基线：
|
||||
tokenizer、数据配方、训练稳定性、中文评测和 scaling behavior 有了统一起点。
|
||||
</p>
|
||||
<h3>为什么先做 Dense 很重要</h3>
|
||||
<p>
|
||||
MoE 同时改变总参数、激活参数、路由、通信和数据分配。没有 dense 基线，很难判断收益来自“更多容量”
|
||||
还是来自别的训练差异。DeepSeek LLM 还训练小规模模型拟合 scaling laws，
|
||||
再用这些规律选择 67B 的超参数；这条“先小规模试验，再外推大模型”的方法后来在 V3/K3 都持续出现。
|
||||
</p>
|
||||
<div class="plain-language">
|
||||
<b>把这一代当作实验坐标系</b>
|
||||
<p>
|
||||
DeepSeek LLM 回答的是“如果我们先不引入稀疏专家和低秩缓存，一套扎实的中英 dense Transformer 能到哪里？”
|
||||
后面的论文才有明确的对照物。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="moe">
|
||||
<p class="eyebrow"><span>02</span> DEEPSEEKMOE</p>
|
||||
<h2>专家越大不一定越专：把一个大专家拆成许多细粒度专家</h2>
|
||||
<p>
|
||||
传统 MoE 往往把 FFN 分成少数大专家，每个 Token 选 1–2 个。问题是一个大专家仍可能同时处理许多无关知识，
|
||||
专家之间也会重复学习公共能力。<a href="https://arxiv.org/abs/2401.06066">DeepSeekMoE</a>
|
||||
提出两项互补策略。
|
||||
</p>
|
||||
<div class="moe-compare">
|
||||
<article>
|
||||
<span>CONVENTIONAL MOE</span>
|
||||
<h3>少数大专家</h3>
|
||||
<div class="expert-pool large"><i>E1</i><i>E2</i><i>E3</i><i>E4</i></div>
|
||||
<p>每个专家覆盖面广，公共知识在多个专家中重复；可组合的专业分工有限。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>DEEPSEEKMOE</span>
|
||||
<h3>细粒度 routed + shared</h3>
|
||||
<div class="expert-pool fine"><i>S</i><i>e1</i><i>e2</i><i>e3</i><i>e4</i><i>e5</i><i>e6</i><i>e7</i></div>
|
||||
<p>shared expert 吸收公共知识；更多小 routed experts 可以按 Token 组合出细分能力。</p>
|
||||
</article>
|
||||
</div>
|
||||
<h3>Fine-Grained Expert Segmentation</h3>
|
||||
<p>
|
||||
在保持单 Token 激活计算近似不变时，把专家 FFN 切得更小，同时选择更多个小专家。
|
||||
组合数显著增加：同样的 Token 可以同时调用“代码语法”“Python 库”“矩阵计算”等几个子专长，
|
||||
不必把它们硬塞进一个笼统的“代码专家”。
|
||||
</p>
|
||||
<h3>Shared Expert Isolation</h3>
|
||||
<p>
|
||||
某些变换几乎每个 Token 都需要。如果全部交给 routed experts，多个专家会重复学习。
|
||||
DeepSeekMoE 把 shared experts 始终激活，承担公共知识；路由专家获得更强动力去形成差异化专长。
|
||||
这套组织直接进入 DeepSeek-V2/V3，也成为 Kimi K3 Stable LatentMoE 的结构祖先。
|
||||
</p>
|
||||
<div class="warning-note">
|
||||
<b>MoE 的代价从 FLOPs 转移到了通信</b>
|
||||
<p>
|
||||
Token 必须被发送到拥有对应专家的设备。专家越细、选择越多，all-to-all、负载波动和权重读取越可能成为瓶颈。
|
||||
因此模型侧路由与系统侧 Expert Parallel 从来不能分开看。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="mla">
|
||||
<p class="eyebrow"><span>03</span> DEEPSEEK-V2 / MLA</p>
|
||||
<h2>训练只付一次参数成本，KV Cache 却在每个请求、每个 Token 上重复付费</h2>
|
||||
<p>
|
||||
<a href="https://arxiv.org/abs/2405.04434">DeepSeek-V2</a> 是整条谱系的关键转折：
|
||||
236B 总参数、21B 激活参数、128K 上下文，把 DeepSeekMoE 用于训练经济性，
|
||||
再用 Multi-head Latent Attention（MLA）直接攻击推理服务的内存瓶颈。
|
||||
</p>
|
||||
<div class="stat-strip">
|
||||
<div><b>−42.5%</b><span>训练成本</span></div>
|
||||
<div><b>−93.3%</b><span>KV Cache</span></div>
|
||||
<div><b>5.76×</b><span>最大生成吞吐</span></div>
|
||||
<p>均为 V2 报告相对 DeepSeek 67B 的特定配置结果，不应外推为任意硬件上的固定倍数。</p>
|
||||
</div>
|
||||
|
||||
<h3>标准 MHA 为什么会让 KV Cache 爆长</h3>
|
||||
<p>
|
||||
自回归生成每一步都要查询历史 Token。历史的 K/V 不变，因此缓存起来避免重算。
|
||||
但 MHA 为每层、每个 Token、每个 head 保存 K 和 V；batch、序列和层数一大，缓存会吃掉大量显存，
|
||||
直接限制并发和长上下文。
|
||||
</p>
|
||||
<div class="mla-visual" role="img" aria-label="标准多头注意力与 MLA 缓存差异">
|
||||
<div class="mha-side">
|
||||
<span>MHA CACHE / EACH TOKEN</span>
|
||||
<div class="heads"><i>K₁</i><i>V₁</i><i>K₂</i><i>V₂</i><i>K₃</i><i>V₃</i><i>K₄</i><i>V₄</i></div>
|
||||
<p>每个 head 各存一份 K/V。</p>
|
||||
</div>
|
||||
<strong>→</strong>
|
||||
<div class="latent-side">
|
||||
<span>MLA CACHE / EACH TOKEN</span>
|
||||
<div class="latent"><i>cᴷⱽ</i></div>
|
||||
<p>先存低维 joint latent，计算时再上投影。</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="formula">
|
||||
cₜᴷⱽ = Wᴰᴷⱽhₜ kₜᶜ = Wᵁᴷcₜᴷⱽ vₜᶜ = Wᵁⱽcₜᴷⱽ
|
||||
<small>
|
||||
省略 decoupled RoPE key、各 head 展开与权重吸收细节。核心是 K/V 先联合低秩压缩，缓存 latent 而非完整多头表示。
|
||||
</small>
|
||||
</div>
|
||||
<h3>为什么 MLA 比简单 MQA/GQA 更有野心</h3>
|
||||
<p>
|
||||
MQA 让所有 Query heads 共用一组 K/V，GQA 让一组 Query heads 共用 K/V，缓存更小但容量可能下降。
|
||||
MLA 用低秩 latent 保留可恢复的内容子空间，并把 RoPE 相关部分分离，追求接近 MHA 的表达力与更小缓存。
|
||||
它后来被 Kimi K2/K2.5 采用，并在 K3 中作为周期性全局注意力保留。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="v3">
|
||||
<p class="eyebrow"><span>04</span> DEEPSEEK-V3</p>
|
||||
<h2>V3 的亮点不是一个技巧，而是四层协同</h2>
|
||||
<p>
|
||||
<a href="https://arxiv.org/abs/2412.19437">DeepSeek-V3</a> 扩到 671B 总参数、37B 激活参数，
|
||||
在 14.8T Token 上预训练。报告给出的完整训练用量约 2.788M H800 GPU hours，并称整个训练没有不可恢复的 loss spike 或回滚。
|
||||
这组数字之所以引人注目，是因为它背后同时改动模型、目标、数值和系统。
|
||||
</p>
|
||||
<div class="four-layer">
|
||||
<article><span>MODEL</span><h3>MLA + DeepSeekMoE</h3><p>沿用 V2 验证过的低缓存 attention 与细粒度专家。</p></article>
|
||||
<article><span>ROUTING</span><h3>Aux-loss-free balance</h3><p>用动态 expert bias 调整负载，减少辅助平衡损失对主目标的干扰。</p></article>
|
||||
<article><span>OBJECTIVE</span><h3>Multi-Token Prediction</h3><p>训练时预测多个未来 Token，增加训练信号，并可转化为推测解码草稿能力。</p></article>
|
||||
<article><span>SYSTEM</span><h3>FP8 + DualPipe</h3><p>低精度训练、计算通信重叠、跨节点专家并行共同压低成本。</p></article>
|
||||
</div>
|
||||
|
||||
<h3>无辅助损失负载均衡</h3>
|
||||
<p>
|
||||
MoE 必须避免少数专家过载。传统做法加入 load-balancing auxiliary loss，
|
||||
但它与语言建模主目标可能冲突：为了均匀而把 Token 送给次优专家。V3 为每个专家维护 bias，
|
||||
根据近期负载上调冷门专家、下调热门专家；bias 只影响路由选择，不直接进入最终门控权重，
|
||||
从而把“系统要均衡”和“模型要准确”更松地解耦。
|
||||
</p>
|
||||
|
||||
<h3>FP8 训练真正难在哪</h3>
|
||||
<p>
|
||||
FP8 动态范围和有效精度有限，不能简单把所有 tensor 强转为 8 位。V3 使用细粒度量化、较高精度累加、
|
||||
在线缩放和特定敏感模块的高精度保留；同时让低精度通信减少跨卡带宽。
|
||||
论文最值得看的不是“首次大规模 FP8”宣传，而是哪些路径降精度、哪些路径绝不降，以及误差怎样被控制。
|
||||
</p>
|
||||
|
||||
<h3>DualPipe：流水线的空泡是一笔真金白银</h3>
|
||||
<p>
|
||||
Pipeline Parallel 把层切到不同 stage；朴素调度会让设备在前向/反向依赖之间等待。
|
||||
DualPipe 从流水线两端同时喂入 micro-batch，并重叠前向、反向与专家通信，尽量把空泡藏在计算后面。
|
||||
它与 DeepEP 的高吞吐/低延迟 all-to-all 一起，把 MoE 理论稀疏性变成真实集群效率。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="grpo">
|
||||
<p class="eyebrow"><span>05</span> DEEPSEEKMATH / GRPO</p>
|
||||
<h2>GRPO：不用再养一个和策略模型同样昂贵的 Critic</h2>
|
||||
<p>
|
||||
<a href="https://arxiv.org/abs/2402.03300">DeepSeekMath</a> 不只是数学模型论文。
|
||||
它在 7B 模型和 120B 数学相关 Token 上验证数据工程，同时提出 Group Relative Policy Optimization，
|
||||
为后来 DeepSeek-V2 对齐和 R1 大规模推理 RL 铺路。
|
||||
</p>
|
||||
<h3>PPO 的显存账单</h3>
|
||||
<p>
|
||||
经典 RLHF/PPO 常同时保留 policy、reference、reward model 和 value/critic model。
|
||||
对大模型而言，critic 往往与 policy 同量级。GRPO 对同一道题采样一组答案，
|
||||
用组内奖励的均值和标准差构造相对优势，省去单独 value model。
|
||||
</p>
|
||||
<div class="grpo-visual">
|
||||
<div class="prompt"><span>PROMPT</span><b>证明 / 求解一道题</b></div>
|
||||
<i>sample group</i>
|
||||
<div class="answers">
|
||||
<div><b>y₁</b><span>reward 0</span></div>
|
||||
<div><b>y₂</b><span>reward 1</span></div>
|
||||
<div><b>y₃</b><span>reward 0.7</span></div>
|
||||
<div><b>y₄</b><span>reward 0.2</span></div>
|
||||
</div>
|
||||
<i>normalize</i>
|
||||
<div class="advantage"><span>RELATIVE ADVANTAGE</span><b>高于组均值的轨迹被鼓励</b></div>
|
||||
</div>
|
||||
<div class="formula">
|
||||
Âᵢ = (rᵢ − mean(r₁…rᴳ)) / (std(r₁…rᴳ) + ε)
|
||||
<small>GRPO 完整目标仍包含 clipped policy ratio 与 KL 约束；这里只展示“组相对优势”这一核心直觉。</small>
|
||||
</div>
|
||||
<h3>组相对不是免费午餐</h3>
|
||||
<p>
|
||||
一道题要采样多条答案，rollout 成本仍然很高；奖励若不可验证或存在偏差，组内比较也会放大奖励漏洞。
|
||||
当一组奖励全相同，归一优势几乎不给学习信号。R1 的成功因此同时依赖可验证数学/代码奖励、足够多样的采样和大规模基础设施。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="r1">
|
||||
<p class="eyebrow"><span>06</span> DEEPSEEK-R1</p>
|
||||
<h2>R1-Zero 最重要的实验：先不教推理格式，只给可验证结果奖励</h2>
|
||||
<p class="lede">
|
||||
<a href="https://arxiv.org/abs/2501.12948">DeepSeek-R1</a> 的历史意义，
|
||||
是把“推理可以通过大规模 RL 从强 base model 中被激发”做成公开、可研究的系统证据。
|
||||
</p>
|
||||
<h3>R1-Zero 做了什么</h3>
|
||||
<p>
|
||||
从 DeepSeek-V3 Base 出发，不先做推理 SFT，直接以 GRPO 进行大规模 RL。
|
||||
奖励以答案正确性为主，并加入格式奖励。训练中出现更长推理、反思、回溯和自我验证等行为；
|
||||
论文把某些突然延长思考的轨迹称为 “aha moment”。
|
||||
</p>
|
||||
<h3>它也明确暴露了纯 RL 的问题</h3>
|
||||
<p>
|
||||
R1-Zero 会重复、可读性差、混合语言。奖励只关心最终正确时，模型没有充分动力照顾人类阅读体验。
|
||||
正式 R1 因而先加入少量高质量 cold-start CoT 数据，再做 reasoning-oriented RL；
|
||||
随后用 rejection sampling 产生 SFT 数据，混入写作、事实问答等非推理任务，再进行第二阶段 RL 兼顾帮助性与安全。
|
||||
</p>
|
||||
<div class="r1-pipeline">
|
||||
<div><span>BASE</span><b>DeepSeek-V3 Base</b><small>强预训练基础</small></div>
|
||||
<i>→</i>
|
||||
<div><span>COLD START</span><b>少量可读 CoT</b><small>稳定格式与语言</small></div>
|
||||
<i>→</i>
|
||||
<div class="hot"><span>REASONING RL</span><b>可验证奖励 + GRPO</b><small>强化求解策略</small></div>
|
||||
<i>→</i>
|
||||
<div><span>SFT MIX</span><b>拒绝采样 + 通用数据</b><small>恢复广泛任务</small></div>
|
||||
<i>→</i>
|
||||
<div><span>FINAL RL</span><b>帮助性与安全</b><small>统一模型</small></div>
|
||||
</div>
|
||||
<h3>Distillation 的关键发现</h3>
|
||||
<p>
|
||||
团队用 R1 生成的推理样本微调 Qwen/Llama dense 模型，发布 1.5B–70B 蒸馏版本。
|
||||
这说明小模型不一定要自己承担完整的探索式 RL 成本，可以模仿强 reasoning teacher 的轨迹；
|
||||
但蒸馏得到的是 teacher 数据分布上的能力，不等于小模型内部复现了同样的 RL 发现过程。
|
||||
</p>
|
||||
<div class="warning-note">
|
||||
<b>CoT 变长不等于推理一定更好</b>
|
||||
<p>
|
||||
长轨迹可能包含有效搜索，也可能是重复与绕路。R1 证明的是在可验证任务和适当训练下，
|
||||
增加推理计算可以转化为能力；不是“输出越长越聪明”。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="v32">
|
||||
<p class="eyebrow"><span>07</span> DEEPSEEK-V3.2</p>
|
||||
<h2>从会推理到会在长上下文里使用工具</h2>
|
||||
<p>
|
||||
<a href="https://arxiv.org/abs/2512.02556">DeepSeek-V3.2</a> 把三条线合并：
|
||||
DeepSeek Sparse Attention（DSA）降低长上下文成本，更大规模的 RL 提升 reasoning，
|
||||
Agentic task synthesis 则把 reasoning 放进工具交互轨迹。
|
||||
</p>
|
||||
<h3>DSA 的两阶段直觉</h3>
|
||||
<p>
|
||||
完整注意力让每个 Query 和全部历史交互。DSA 先用轻量、可学习的 indexer 给历史 Token 评分，
|
||||
选出小部分候选，再让主 attention 只在候选上做高容量计算。
|
||||
关键不只是 top-k，而是 indexer 也在训练中学习“什么值得看”；这比固定窗口更能适应内容相关的远距离依赖。
|
||||
</p>
|
||||
<div class="sparse-visual">
|
||||
<div class="history">
|
||||
{Array.from({ length: 18 }, (_, index) => <i class:list={{ picked: [1, 5, 6, 12, 16].includes(index) }}>{index + 1}</i>)}
|
||||
</div>
|
||||
<span>learned indexer → top-k</span>
|
||||
<div class="picked"><i>2</i><i>6</i><i>7</i><i>13</i><i>17</i></div>
|
||||
<span>main attention</span>
|
||||
<b>Query</b>
|
||||
</div>
|
||||
<p>
|
||||
Agent 方面，V3.2 的任务合成管线系统地产生复杂、交互式工具任务，让思考与 tool use 交织。
|
||||
这与 K3 的 white-box harness、知识图谱任务合成和可验证环境形成同期对照：前沿模型竞争正在从静态题库转向训练环境。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="v4">
|
||||
<p class="eyebrow"><span>08</span> DEEPSEEK-V4</p>
|
||||
<h2>百万上下文从“支持”变成一套专门架构</h2>
|
||||
<p>
|
||||
2026 年的 <a href="https://arxiv.org/abs/2606.19348">DeepSeek-V4 preview</a> 包含
|
||||
1.6T-A49B 的 Pro 与 284B-A13B 的 Flash，均支持 1M Token。
|
||||
它使用 Compressed Sparse Attention（CSA）与 Heavily Compressed Attention（HCA）的混合注意力，
|
||||
Manifold-Constrained Hyper-Connections（mHC）改善深度残差，并采用 Muon 优化器。
|
||||
</p>
|
||||
<div class="stat-strip v4">
|
||||
<div><b>1.6T / 49B</b><span>V4-Pro total / active</span></div>
|
||||
<div><b>284B / 13B</b><span>V4-Flash total / active</span></div>
|
||||
<div><b>>32T</b><span>预训练 Token</span></div>
|
||||
<p>V4 官方报告摘要数据；模型是 preview 版本，后续版本需按研究截止日重新核验。</p>
|
||||
</div>
|
||||
<p>
|
||||
报告称在 1M 上下文下，V4-Pro 的单 Token 推理 FLOPs 是 V3.2 的 27%，KV Cache 是 10%。
|
||||
这说明长上下文竞争已从单一位置外推转向混合注意力、压缩缓存、残差、优化器和后训练的系统协同。
|
||||
它也解释 K3 报告为何把 V4 列为同代开放基础模型对照。
|
||||
</p>
|
||||
<div class="warning-note">
|
||||
<b>V4 与 K3 不是同一条注意力路线</b>
|
||||
<p>
|
||||
V4 用 CSA/HCA 混合压缩与稀疏注意力；K3 用 3:1 KDA/Gated MLA 混合线性递归与全局注意力。
|
||||
两者目标相近——降低百万上下文成本并保留能力——但状态表示、检索方式和系统内核不同。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="k3">
|
||||
<p class="eyebrow"><span>09</span> INTO KIMI K3</p>
|
||||
<h2>DeepSeek 的哪些思想直接流入 K3，哪些只是同期呼应</h2>
|
||||
<div class="mapping-table">
|
||||
<div class="head"><b>DeepSeek 线索</b><b>K3 中的落点</b><b>关系</b></div>
|
||||
<div><span>DeepSeekMoE shared + routed experts</span><span>Stable LatentMoE 保留 shared/routed 组织</span><em>直接结构祖先</em></div>
|
||||
<div><span>V2 Multi-head Latent Attention</span><span>每四层一次 Gated MLA，全局注意力</span><em>明确采用并改造</em></div>
|
||||
<div><span>V3 Auxiliary-loss-free balancing</span><span>Quantile Balancing 应对近千专家</span><em>同一问题的新方案</em></div>
|
||||
<div><span>V3 FP8 / 低精度协同</span><span>专家 MXFP4 权重、MXFP8 激活与 QAT</span><em>更低精度的延伸</em></div>
|
||||
<div><span>DeepSeekMath / R1 的 GRPO 与 RL</span><span>多 domain、多 effort RL + MOPD</span><em>共享测试时扩展范式</em></div>
|
||||
<div><span>V4 Muon 与深度残差创新</span><span>Per-Head Muon + Attention Residuals</span><em>同期不同设计</em></div>
|
||||
<div><span>V3.2/V4 长上下文</span><span>KDA + Gated MLA + KDA system co-design</span><em>同目标、不同路线</em></div>
|
||||
</div>
|
||||
<p>
|
||||
这正是为什么 DeepSeek 值得在 LLM Atlas 中作为贯穿案例：它不是 K3 的“对手名单”之一，
|
||||
而是 K3 架构里多条思想的公开祖先与同代参照。读懂 V2 的 MLA 和 DeepSeekMoE，
|
||||
K3 的一半架构会突然变得熟悉。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="papers">
|
||||
<p class="eyebrow"><span>↳</span> READING ORDER</p>
|
||||
<h2>建议精读顺序：不要直接从 R1 开始</h2>
|
||||
<div class="paper-chain">
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2401.02954"><time>01</time><b>DeepSeek LLM</b><p>建立 dense、数据与 scaling 基线；精读架构与 scaling law 部分。</p></a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2401.06066"><time>02</time><b>DeepSeekMoE</b><p>精读 fine-grained segmentation、shared expert isolation 与消融。</p></a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2405.04434"><time>03</time><b>DeepSeek-V2</b><p>精读 MLA 推导、RoPE 解耦、权重吸收和 KV Cache 对比。</p></a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2412.19437"><time>04</time><b>DeepSeek-V3</b><p>分四遍读：架构、FP8、DualPipe/通信、MTP 与后训练。</p></a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2402.03300"><time>05</time><b>DeepSeekMath</b><p>先理解 PPO，再推导 GRPO 的组相对优势与 KL 项。</p></a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2501.12948"><time>06</time><b>DeepSeek-R1</b><p>对照 R1-Zero 与 R1 pipeline，区分 RL 涌现、冷启动与蒸馏。</p></a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2512.02556"><time>07</time><b>DeepSeek-V3.2</b><p>把稀疏 attention 与 agentic task synthesis 放在一起读。</p></a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2606.19348"><time>08</time><b>DeepSeek-V4</b><p>从百万上下文成本倒推 CSA/HCA、mHC 与 Muon。</p></a>
|
||||
</div>
|
||||
<div class="hero-actions">
|
||||
<a class="button primary" href="/k3/">回到 K3：看这些技术怎样重新组合 →</a>
|
||||
<a class="button" href="/roadmap/">完整课程地图</a>
|
||||
</div>
|
||||
</section>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.deepseek-hero::before {
|
||||
background:
|
||||
radial-gradient(circle at 43% 47%, rgba(107, 98, 145, 0.16), transparent 31%),
|
||||
radial-gradient(circle at 65% 38%, rgba(84, 124, 116, 0.13), transparent 24%);
|
||||
}
|
||||
|
||||
.moe-compare,
|
||||
.four-layer {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
max-width: 920px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.moe-compare article,
|
||||
.four-layer article {
|
||||
min-height: 260px;
|
||||
padding: 26px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.moe-compare article > span,
|
||||
.four-layer span {
|
||||
color: var(--copper);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
letter-spacing: 0.09em;
|
||||
}
|
||||
|
||||
.moe-compare h3,
|
||||
.four-layer h3 {
|
||||
margin: 24px 0 14px;
|
||||
font-size: 1.15rem;
|
||||
}
|
||||
|
||||
.moe-compare p,
|
||||
.four-layer p {
|
||||
color: var(--muted);
|
||||
font-size: 0.78rem;
|
||||
line-height: 1.72;
|
||||
}
|
||||
|
||||
.expert-pool {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 7px;
|
||||
margin: 22px 0;
|
||||
}
|
||||
|
||||
.expert-pool i {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 46px;
|
||||
height: 46px;
|
||||
border: 1px solid var(--line-strong);
|
||||
border-radius: 6px;
|
||||
color: var(--muted);
|
||||
font: 0.68rem/1 var(--mono);
|
||||
font-style: normal;
|
||||
}
|
||||
|
||||
.expert-pool.large i {
|
||||
width: 66px;
|
||||
height: 66px;
|
||||
background: var(--sky-pale);
|
||||
}
|
||||
|
||||
.expert-pool.fine i {
|
||||
background: var(--copper-pale);
|
||||
}
|
||||
|
||||
.expert-pool.fine i:first-child {
|
||||
border-color: var(--sage);
|
||||
background: var(--sage-pale);
|
||||
color: var(--sage);
|
||||
}
|
||||
|
||||
.stat-strip {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||||
max-width: 860px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.stat-strip > div {
|
||||
min-height: 130px;
|
||||
padding: 25px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.stat-strip b,
|
||||
.stat-strip span {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.stat-strip b {
|
||||
color: var(--copper);
|
||||
font: 700 1.4rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.stat-strip span {
|
||||
margin-top: 16px;
|
||||
color: var(--muted);
|
||||
font-size: 0.74rem;
|
||||
}
|
||||
|
||||
.stat-strip > p {
|
||||
grid-column: 1 / -1;
|
||||
padding: 12px 18px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
color: var(--muted);
|
||||
font-size: 0.68rem;
|
||||
}
|
||||
|
||||
.mla-visual {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 50px 1fr;
|
||||
gap: 18px;
|
||||
align-items: center;
|
||||
max-width: 820px;
|
||||
margin: 34px 0;
|
||||
}
|
||||
|
||||
.mla-visual > div {
|
||||
min-height: 210px;
|
||||
padding: 26px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.mla-visual > strong {
|
||||
color: var(--muted-light);
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.mla-visual span,
|
||||
.grpo-visual span,
|
||||
.r1-pipeline span {
|
||||
color: var(--muted);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.heads {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
gap: 6px;
|
||||
margin: 27px 0;
|
||||
}
|
||||
|
||||
.heads i,
|
||||
.latent i {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
height: 46px;
|
||||
border: 1px solid var(--sky);
|
||||
background: var(--sky-pale);
|
||||
color: var(--sky);
|
||||
font: 0.67rem/1 var(--mono);
|
||||
font-style: normal;
|
||||
}
|
||||
|
||||
.latent {
|
||||
margin: 27px 0;
|
||||
}
|
||||
|
||||
.latent i {
|
||||
width: 74px;
|
||||
border-color: var(--copper);
|
||||
background: var(--copper-pale);
|
||||
color: var(--copper);
|
||||
}
|
||||
|
||||
.mla-visual p {
|
||||
color: var(--muted);
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
|
||||
.four-layer {
|
||||
grid-template-columns: repeat(4, minmax(0, 1fr));
|
||||
}
|
||||
|
||||
.four-layer article {
|
||||
min-height: 230px;
|
||||
padding: 22px;
|
||||
}
|
||||
|
||||
.grpo-visual {
|
||||
display: grid;
|
||||
grid-template-columns: 160px 70px 1fr 70px 220px;
|
||||
gap: 10px;
|
||||
align-items: center;
|
||||
max-width: 920px;
|
||||
margin: 34px 0;
|
||||
}
|
||||
|
||||
.grpo-visual > div:not(.answers) {
|
||||
min-height: 125px;
|
||||
padding: 18px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.grpo-visual > i {
|
||||
color: var(--muted);
|
||||
font: 0.6rem/1.4 var(--mono);
|
||||
font-style: normal;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.grpo-visual b {
|
||||
display: block;
|
||||
margin-top: 23px;
|
||||
font-size: 0.82rem;
|
||||
}
|
||||
|
||||
.answers {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
gap: 7px;
|
||||
}
|
||||
|
||||
.answers div {
|
||||
min-height: 110px;
|
||||
padding: 14px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.answers span {
|
||||
display: block;
|
||||
margin-top: 24px;
|
||||
color: var(--muted);
|
||||
font-size: 0.58rem;
|
||||
}
|
||||
|
||||
.r1-pipeline {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, minmax(110px, 1fr) 22px) minmax(110px, 1fr);
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
max-width: 940px;
|
||||
margin: 34px 0;
|
||||
}
|
||||
|
||||
.r1-pipeline > div {
|
||||
min-height: 130px;
|
||||
padding: 17px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.r1-pipeline > div.hot {
|
||||
border-color: var(--copper);
|
||||
background: var(--copper-pale);
|
||||
}
|
||||
|
||||
.r1-pipeline > i {
|
||||
color: var(--muted-light);
|
||||
font-style: normal;
|
||||
}
|
||||
|
||||
.r1-pipeline b,
|
||||
.r1-pipeline small {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.r1-pipeline b {
|
||||
margin-top: 20px;
|
||||
font-size: 0.78rem;
|
||||
}
|
||||
|
||||
.r1-pipeline small {
|
||||
margin-top: 8px;
|
||||
color: var(--muted);
|
||||
font-size: 0.62rem;
|
||||
}
|
||||
|
||||
.sparse-visual {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 120px auto 100px 60px;
|
||||
gap: 15px;
|
||||
align-items: center;
|
||||
max-width: 910px;
|
||||
margin: 34px 0;
|
||||
padding: 28px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.history,
|
||||
.picked {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 5px;
|
||||
}
|
||||
|
||||
.history i,
|
||||
.picked i,
|
||||
.sparse-visual > b {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
border: 1px solid var(--line);
|
||||
color: var(--muted-light);
|
||||
font: 0.58rem/1 var(--mono);
|
||||
font-style: normal;
|
||||
}
|
||||
|
||||
.history i.picked,
|
||||
.picked i {
|
||||
border-color: var(--copper);
|
||||
background: var(--copper-pale);
|
||||
color: var(--copper);
|
||||
}
|
||||
|
||||
.sparse-visual > span {
|
||||
color: var(--muted);
|
||||
font: 0.58rem/1.4 var(--mono);
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.sparse-visual > b {
|
||||
width: 56px;
|
||||
height: 56px;
|
||||
border-color: var(--sky);
|
||||
border-radius: 50%;
|
||||
background: var(--sky-pale);
|
||||
color: var(--sky);
|
||||
}
|
||||
|
||||
.mapping-table {
|
||||
display: grid;
|
||||
max-width: 940px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.mapping-table > div {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1.2fr 0.7fr;
|
||||
}
|
||||
|
||||
.mapping-table > div > * {
|
||||
padding: 15px 17px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
font-size: 0.75rem;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
.mapping-table .head {
|
||||
background: var(--ink);
|
||||
color: var(--paper-raised);
|
||||
}
|
||||
|
||||
.mapping-table .head b {
|
||||
font: 0.62rem/1.5 var(--mono);
|
||||
letter-spacing: 0.08em;
|
||||
}
|
||||
|
||||
.mapping-table em {
|
||||
color: var(--copper);
|
||||
font-style: normal;
|
||||
}
|
||||
|
||||
@media (max-width: 820px) {
|
||||
.moe-compare,
|
||||
.four-layer,
|
||||
.stat-strip {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.stat-strip > p {
|
||||
grid-column: auto;
|
||||
}
|
||||
|
||||
.mla-visual,
|
||||
.grpo-visual,
|
||||
.r1-pipeline,
|
||||
.sparse-visual {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.mla-visual > strong,
|
||||
.r1-pipeline > i {
|
||||
transform: rotate(90deg);
|
||||
}
|
||||
|
||||
.mapping-table > div {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</BaseLayout>
|
||||
@@ -0,0 +1,583 @@
|
||||
---
|
||||
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||
import AttentionLab from "@/components/AttentionLab.astro";
|
||||
|
||||
const toc = [
|
||||
["00", "goal", "这一章要建立什么"],
|
||||
["01", "next-token", "预测下一个 Token"],
|
||||
["02", "vector", "词怎样变成向量"],
|
||||
["03", "bottleneck", "RNN 的顺序瓶颈"],
|
||||
["04", "attention", "注意力的出现"],
|
||||
["05", "qkv", "Q / K / V"],
|
||||
["06", "block", "Transformer Block"],
|
||||
["07", "training", "训练与生成"],
|
||||
["08", "myths", "常见误解"],
|
||||
["↳", "papers", "关键论文链"],
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="注意力与 Transformer：从零建立直觉"
|
||||
description="从预测下一个 Token、向量表示与 Seq2Seq 瓶颈开始，用交互图逐步理解 Q、K、V、自注意力、因果掩码和 Transformer Block。"
|
||||
section="foundations"
|
||||
>
|
||||
<header class="page-hero">
|
||||
<div class="page-hero-inner">
|
||||
<div>
|
||||
<p class="eyebrow"><span>FOUNDATIONS / 02</span> TRANSFORMER</p>
|
||||
<h1>注意力，不是让模型<br />“更专心”那么简单</h1>
|
||||
<p class="lead">
|
||||
它把序列建模从“一步一步传话”改成“每个位置直接检索相关位置”。
|
||||
这一变化解决了长距离依赖，也把训练变成高度并行的矩阵计算。
|
||||
</p>
|
||||
</div>
|
||||
<dl class="page-facts">
|
||||
<div><dt>LEVEL</dt><dd>L0 直觉 → L2 公式</dd></div>
|
||||
<div><dt>PREREQ</dt><dd>只需基础概率概念</dd></div>
|
||||
<div><dt>CORE</dt><dd>Q · K · V · Softmax</dd></div>
|
||||
<div><dt>TIME</dt><dd>约 55 分钟</dd></div>
|
||||
<div><dt>STATUS</dt><dd>首版可读 · 继续补图</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="report-shell">
|
||||
<aside class="side-rail" aria-label="本页目录">
|
||||
<p>CONTENTS</p>
|
||||
<ol>
|
||||
{toc.map(([number, id, label]) => (
|
||||
<li><a href={`#${id}`}><span>{number}</span>{label}</a></li>
|
||||
))}
|
||||
</ol>
|
||||
<div class="rail-note">
|
||||
<b>阅读建议</b>
|
||||
第一次先跳过公式里的维度，只追踪“谁在向谁取信息”；第二遍再看矩阵形状。
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<article class="article">
|
||||
<section class="article-section" id="goal">
|
||||
<p class="eyebrow"><span>00</span> MENTAL MODEL</p>
|
||||
<h2>读完以后，你应该能画出一个 Token 的完整旅程</h2>
|
||||
<p class="lede">
|
||||
文本先被切成 Token，Token 变成向量；自注意力让每个位置从其他位置收集信息，
|
||||
前馈网络在每个位置内部加工；残差和归一化让几十上百层能够稳定堆叠。
|
||||
</p>
|
||||
<div class="token-journey" role="img" aria-label="一个 Token 通过 Transformer 的旅程">
|
||||
<div><span>01</span><b>Tokenizer</b><small>文字 → 离散 ID</small></div><i>→</i>
|
||||
<div><span>02</span><b>Embedding</b><small>ID → 连续向量</small></div><i>→</i>
|
||||
<div class="accent"><span>03</span><b>Attention</b><small>位置之间交流</small></div><i>→</i>
|
||||
<div><span>04</span><b>FFN / MoE</b><small>位置内部加工</small></div><i>→</i>
|
||||
<div><span>05</span><b>LM Head</b><small>下一 Token 概率</small></div>
|
||||
</div>
|
||||
<p>
|
||||
K3 的复杂架构仍然没有离开这张图：KDA 与 MLA 替换“位置之间交流”的具体机制，
|
||||
Stable LatentMoE 替换 FFN，Attention Residuals 改变层与层之间的传递，MoonViT 把图像也转成可进入主干的表示。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="next-token">
|
||||
<p class="eyebrow"><span>01</span> LANGUAGE MODELING</p>
|
||||
<h2>大模型最底层的训练题：根据前文猜下一个 Token</h2>
|
||||
<p>
|
||||
一段文本的联合概率可以用链式法则拆开：
|
||||
第一个 Token 的概率 × 在第一个已知时第二个的概率 × 在前两个已知时第三个的概率……。
|
||||
语言模型只要不断学会这些条件概率，就能给整段文本分配概率。
|
||||
</p>
|
||||
<div class="formula">
|
||||
P(x₁, x₂, …, xₙ) = ∏ₜ P(xₜ | x₁, …, xₜ₋₁)
|
||||
<small>这只是概率链式法则，本身不规定用 N-gram、RNN 还是 Transformer 计算条件概率。</small>
|
||||
</div>
|
||||
<p>
|
||||
训练时，我们已知句子后面的正确 Token，因此可以把一段文本整体右移一位做标签，所有位置并行产生损失。
|
||||
模型对正确 Token 给出的概率越低，负对数损失越大。
|
||||
</p>
|
||||
<div class="formula">
|
||||
L = − Σₜ log Pθ(xₜ | x<ₜ)
|
||||
<small>交叉熵训练不会直接写入“事实库”或“推理规则”；这些结构是模型为降低大量预测误差而在参数中形成的。</small>
|
||||
</div>
|
||||
<h3>为什么这样简单的任务会学到很多东西</h3>
|
||||
<p>
|
||||
要准确补全“牛顿在 1687 年出版了……”，模型需要吸收事实；要补全一段代码，需要跟踪变量和语法；
|
||||
要补全论证，需要建模前提与结论。下一个 Token 预测没有显式要求“理解”，但数据中的各种规律都会降低预测损失。
|
||||
模型容量和数据足够大时，许多规律共享同一套内部表示，因此出现广泛迁移。
|
||||
</p>
|
||||
<div class="warning-note">
|
||||
<b>但不要反过度解释</b>
|
||||
<p>
|
||||
预测目标能诱导知识与能力，不保证事实永远准确、推理过程忠实或目标与人类一致。
|
||||
这正是检索、后训练、验证器和安全对齐仍然必要的原因。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="vector">
|
||||
<p class="eyebrow"><span>02</span> DISTRIBUTED REPRESENTATION</p>
|
||||
<h2>Token ID 没有意义；Embedding 才是模型能计算的语义坐标</h2>
|
||||
<p>
|
||||
Tokenizer 把文本分成词、子词或字节片段，并映射为整数 ID。ID 只是目录编号：
|
||||
42 比 17 大，不代表第 42 个 Token 语义更强。Embedding table 根据 ID 查出一个可训练向量，
|
||||
后续网络只处理这些连续数值。
|
||||
</p>
|
||||
<div class="embedding-demo">
|
||||
<div class="word"><span>TOKEN</span><b>小猫</b><small>ID = 4821</small></div>
|
||||
<i>lookup</i>
|
||||
<div class="vector"><span>EMBEDDING</span><code>[0.18, −0.42, 0.07, …, 0.31]</code><small>d 个可学习维度</small></div>
|
||||
</div>
|
||||
<p>
|
||||
“分布式表示”的要点是：一个概念不放在单个神经元里，而分散在许多维度；
|
||||
一个维度也参与表示许多概念。相似用法的 Token 在训练中往往形成相近或具有稳定关系的向量。
|
||||
Bengio 2003 的神经概率语言模型和 2013 年 Word2Vec，是这条历史线的关键节点。
|
||||
</p>
|
||||
<h3>同一个词在不同句子里怎么办</h3>
|
||||
<p>
|
||||
初始 embedding 对同一个 Token 固定不变，但经过 self-attention 后会变成 contextual representation。
|
||||
“苹果发布手机”和“吃一个苹果”里的“苹果”起点相同，周围 Token 通过注意力写入不同上下文后，深层表示会分开。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="bottleneck">
|
||||
<p class="eyebrow"><span>03</span> SEQUENTIAL BOTTLENECK</p>
|
||||
<h2>Transformer 之前：信息像传话游戏一样逐步穿过 RNN</h2>
|
||||
<p>
|
||||
RNN 每读一个 Token，就把旧隐藏状态和新输入合成新状态。理论上最后状态可以包含全部历史；
|
||||
实际上，遥远信息必须经过许多次变换，梯度也必须沿时间反向穿过这些步骤。
|
||||
LSTM/GRU 用门控缓解遗忘与梯度问题，但训练仍然按时间顺序，难以把序列位置全部摊到 GPU 上并行。
|
||||
</p>
|
||||
<div class="sequence-compare">
|
||||
<div>
|
||||
<span>RNN / 顺序传递</span>
|
||||
<div class="rnn-line"><b>x₁</b><i>→</i><b>h₁</b><i>→</i><b>h₂</b><i>→</i><b>h₃</b><i>→</i><b>h₄</b></div>
|
||||
<p>x₁ 的信息要到 h₄，必须走过每一个中间状态。</p>
|
||||
</div>
|
||||
<div>
|
||||
<span>SELF-ATTENTION / 直接检索</span>
|
||||
<div class="attn-lines"><b>x₁</b><b>x₂</b><b>x₃</b><b>x₄</b><strong>↘ ↓ ↙</strong><em>x₄′</em></div>
|
||||
<p>x₄ 可以在一层内直接读取 x₁、x₂、x₃。</p>
|
||||
</div>
|
||||
</div>
|
||||
<p>
|
||||
2014 年 Seq2Seq 用 encoder 把源句压进固定向量，再由 decoder 生成目标句。长句表现受限时，
|
||||
Bahdanau Attention 让 decoder 每一步不再只依赖最后一个 encoder state，而能对全部 encoder states 加权读取。
|
||||
注意力最初是一座跨 encoder—decoder 的桥；Transformer 的突破是让桥本身成为主体。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="attention">
|
||||
<p class="eyebrow"><span>04</span> ATTENTION</p>
|
||||
<h2>注意力本质上是一次可学习的“软检索”</h2>
|
||||
<p>
|
||||
每个位置提出一个 Query；所有位置提供 Key 和 Value。Query 与 Key 的匹配度决定从相应 Value 读取多少信息。
|
||||
因为权重是连续概率，而不是只选择一个位置，所以称为“软”检索。
|
||||
</p>
|
||||
<AttentionLab />
|
||||
<h3>为什么相似度之后要除以 √dₖ</h3>
|
||||
<p>
|
||||
若 Q、K 的每个分量方差相近，维度越高，点积的典型幅值越大。softmax 输入过大时会接近 one-hot，
|
||||
梯度变得很小。除以 √dₖ 把数值尺度拉回较稳定范围，这就是 Scaled Dot-Product Attention 的“scaled”。
|
||||
</p>
|
||||
<div class="formula">
|
||||
Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V
|
||||
<small>先得到每个 Query 对所有 Key 的相似度，经 softmax 变成和为 1 的权重，再对 Value 做加权和。</small>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="qkv">
|
||||
<p class="eyebrow"><span>05</span> QUERY · KEY · VALUE</p>
|
||||
<h2>Q、K、V 不是三份词义，而是三个可学习角色</h2>
|
||||
<p>
|
||||
同一个输入向量 <em>x</em> 乘三个不同矩阵，得到 Q、K、V。模型可以让“用于匹配的特征”和“匹配后真正搬运的内容”分开。
|
||||
例如代词位置的 Query 可以强调性别/单复数线索，名词位置的 Key 暴露这些线索，
|
||||
Value 则携带更丰富的实体表示。
|
||||
</p>
|
||||
<div class="qkv-grid">
|
||||
<article><span>Q / QUERY</span><h3>我在找什么？</h3><p>当前位置为了更新自己，需要哪类上下文信号。</p></article>
|
||||
<article><span>K / KEY</span><h3>我能被怎样找到？</h3><p>每个来源位置公开哪些匹配特征。</p></article>
|
||||
<article><span>V / VALUE</span><h3>找到后带走什么？</h3><p>一旦获得权重，真正汇入目标位置的信息内容。</p></article>
|
||||
</div>
|
||||
<h3>Multi-Head 的意义</h3>
|
||||
<p>
|
||||
单个注意力分布必须把多种关系挤在一起。Multi-Head 将表示投影到多个较小子空间：
|
||||
不同 head 可以学习局部搭配、指代、结构边界、长距离依赖等不同模式，最后拼接并线性投影。
|
||||
“某个头一定是语法头”并没有保证；head 的功能是训练中形成的，常常混合且可替代。
|
||||
</p>
|
||||
<h3>Causal Mask 为什么不可少</h3>
|
||||
<p>
|
||||
训练 decoder-only 模型时，位置 t 的输入序列里同时存在后续 Token。若不遮住未来位置，
|
||||
模型会直接偷看答案。Causal mask 把 QKᵀ 右上三角设为负无穷，
|
||||
softmax 后未来权重变成 0；这样所有位置仍可并行训练，却遵守生成时只能看过去的约束。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="block">
|
||||
<p class="eyebrow"><span>06</span> TRANSFORMER BLOCK</p>
|
||||
<h2>Attention 只负责交流；FFN 才负责每个位置内部加工</h2>
|
||||
<p>
|
||||
一个现代 decoder block 通常由两类子层组成：Attention 在位置之间混合信息；
|
||||
FFN 对每个位置独立应用相同的非线性变换。残差让子层学习“在现有表示上改多少”，
|
||||
Normalization 则控制数值尺度。
|
||||
</p>
|
||||
<div class="block-diagram" role="img" aria-label="现代 Pre-Norm Transformer Block">
|
||||
<div class="stream"><span>x</span><i>──────────────┐</i></div>
|
||||
<div><span>01</span><b>RMSNorm</b><small>统一输入尺度</small></div>
|
||||
<div class="accent"><span>02</span><b>Causal Attention</b><small>位置之间交流</small></div>
|
||||
<div class="merge"><b>＋</b><small>残差合并</small></div>
|
||||
<div class="stream"><span>h</span><i>──────────────┐</i></div>
|
||||
<div><span>03</span><b>RMSNorm</b><small>再次稳定尺度</small></div>
|
||||
<div class="warm"><span>04</span><b>SwiGLU FFN / MoE</b><small>位置内部加工</small></div>
|
||||
<div class="merge"><b>＋</b><small>残差合并</small></div>
|
||||
</div>
|
||||
<h3>Pre-Norm 与 Post-Norm</h3>
|
||||
<p>
|
||||
原始 Transformer 在残差相加后做 LayerNorm（Post-Norm）。更深模型常把 Norm 放到子层之前（Pre-Norm），
|
||||
让主残差路径更接近恒等映射，梯度传播更稳定。许多现代模型使用 RMSNorm，省去均值中心化，
|
||||
并用 SwiGLU 替代简单 ReLU FFN。K3 进一步把 FFN 替换为 Stable LatentMoE，并用 AttnRes 重做跨深度信息流。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="training">
|
||||
<p class="eyebrow"><span>07</span> TRAINING & GENERATION</p>
|
||||
<h2>训练能并行看完整答案；生成却必须一次走一步</h2>
|
||||
<div class="train-infer">
|
||||
<article>
|
||||
<span>TRAINING / TEACHER FORCING</span>
|
||||
<h3>一段文本同时产生许多训练题</h3>
|
||||
<p>输入“今 天 天 气”，标签右移为“天 天 气 好”。因果 mask 阻止偷看，四个位置可并行算损失。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>INFERENCE / AUTOREGRESSIVE</span>
|
||||
<h3>新 Token 不存在，必须逐个生成</h3>
|
||||
<p>先产生第一个，再把它放回输入产生第二个。KV Cache 避免每一步重算全部旧 Token 的 K/V。</p>
|
||||
</article>
|
||||
</div>
|
||||
<p>
|
||||
这个差异解释了为什么训练吞吐和线上生成速度是两套系统问题，也解释了 K3 为什么同时关心
|
||||
KDA state、MLA KV Cache、推测解码 draft model 和 fleet scheduling。
|
||||
</p>
|
||||
<h3>Logits 怎样变成一个 Token</h3>
|
||||
<p>
|
||||
最后一层表示经 LM head 投影到词表大小，得到每个 Token 的 logits。temperature 调整分布尖锐程度，
|
||||
top-p/top-k 截断候选，再采样或取最大值。它们改变选择策略，不会给模型增加知识，也不能修复错误推理。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="myths">
|
||||
<p class="eyebrow"><span>08</span> COMMON MISCONCEPTIONS</p>
|
||||
<h2>六个最容易带进后续论文的误解</h2>
|
||||
<div class="myth-list">
|
||||
<article><span>误解 01</span><h3>Attention 权重就是模型解释</h3><p>权重只描述某层某头的 value 混合比例；它不必等价于最终预测的因果解释。</p></article>
|
||||
<article><span>误解 02</span><h3>上下文越长就记得越好</h3><p>窗口是容量上限；检索、位置偏差、干扰和训练分布决定模型是否会用。</p></article>
|
||||
<article><span>误解 03</span><h3>参数越多，每次计算一定越贵</h3><p>Dense 模型大致如此；MoE 可让总参数远大于激活参数，但引入路由和通信成本。</p></article>
|
||||
<article><span>误解 04</span><h3>Tokenizer 只影响输入长度</h3><p>它还决定模型看到的基本单位，影响跨语言效率、数字/代码规律与词表输出。</p></article>
|
||||
<article><span>误解 05</span><h3>模型在数据库里查答案</h3><p>预训练知识分布在参数计算中；显式检索需要外接索引、工具或上下文。</p></article>
|
||||
<article><span>误解 06</span><h3>CoT 是模型真实内心记录</h3><p>生成的推理文本可用于求解和检查，但不保证完整、忠实地暴露内部因果过程。</p></article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="papers">
|
||||
<p class="eyebrow"><span>↳</span> LANDMARK PAPERS</p>
|
||||
<h2>从问题到 Transformer 的关键论文链</h2>
|
||||
<div class="paper-chain">
|
||||
<a class="paper-row" href="https://www.jmlr.org/papers/v3/bengio03a.html">
|
||||
<time>2003</time><b>A Neural Probabilistic Language Model</b><p>用可学习分布式词表示和神经网络估计语言概率，突破纯计数泛化。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/1301.3781">
|
||||
<time>2013</time><b>Efficient Estimation of Word Representations in Vector Space</b><p>Word2Vec 让大规模词向量训练变得简单高效。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/1409.3215">
|
||||
<time>2014</time><b>Sequence to Sequence Learning with Neural Networks</b><p>Encoder—decoder 统一可变长序列映射，也暴露固定向量瓶颈。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/1409.0473">
|
||||
<time>2014</time><b>Neural Machine Translation by Jointly Learning to Align and Translate</b><p>Decoder 每一步软选择 encoder 状态，注意力成为跨序列桥梁。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/1706.03762">
|
||||
<time>2017</time><b>Attention Is All You Need</b><p>用 self-attention 和 FFN 取代循环，打开大规模并行训练路径。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf">
|
||||
<time>2018</time><b>Improving Language Understanding by Generative Pre-Training</b><p>Decoder Transformer 先做通用语言建模，再迁移到下游任务。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/1810.04805">
|
||||
<time>2018</time><b>BERT</b><p>双向 masked language modeling 证明大规模预训练表示的通用价值。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2005.14165">
|
||||
<time>2020</time><b>Language Models are Few-Shot Learners</b><p>GPT-3 展示规模扩大后上下文学习成为重要交互范式。</p>
|
||||
</a>
|
||||
</div>
|
||||
<div class="hero-actions">
|
||||
<a class="button primary" href="/k3/#kda">下一步：KDA 如何改造注意力 →</a>
|
||||
<a class="button" href="/roadmap/">回到课程地图</a>
|
||||
</div>
|
||||
</section>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.token-journey {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, minmax(120px, 1fr) 24px) minmax(120px, 1fr);
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
max-width: 930px;
|
||||
margin: 34px 0;
|
||||
}
|
||||
|
||||
.token-journey > div {
|
||||
min-height: 130px;
|
||||
padding: 18px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.token-journey > div.accent {
|
||||
border-color: var(--sky);
|
||||
background: var(--sky-pale);
|
||||
}
|
||||
|
||||
.token-journey span,
|
||||
.embedding-demo span,
|
||||
.sequence-compare > div > span,
|
||||
.qkv-grid span,
|
||||
.block-diagram span,
|
||||
.train-infer span,
|
||||
.myth-list span {
|
||||
color: var(--copper);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
letter-spacing: 0.08em;
|
||||
}
|
||||
|
||||
.token-journey b,
|
||||
.token-journey small {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.token-journey b {
|
||||
margin-top: 20px;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.token-journey small {
|
||||
margin-top: 8px;
|
||||
color: var(--muted);
|
||||
font-size: 0.66rem;
|
||||
}
|
||||
|
||||
.token-journey > i {
|
||||
color: var(--muted-light);
|
||||
font-style: normal;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.embedding-demo {
|
||||
display: grid;
|
||||
grid-template-columns: 170px 60px minmax(260px, 1fr);
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
max-width: 760px;
|
||||
margin: 34px 0;
|
||||
}
|
||||
|
||||
.embedding-demo > div {
|
||||
padding: 24px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.embedding-demo > i {
|
||||
color: var(--muted);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.embedding-demo b,
|
||||
.embedding-demo code,
|
||||
.embedding-demo small {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.embedding-demo b,
|
||||
.embedding-demo code {
|
||||
margin: 16px 0 7px;
|
||||
}
|
||||
|
||||
.embedding-demo code {
|
||||
color: var(--ink);
|
||||
font-size: 0.82rem;
|
||||
}
|
||||
|
||||
.embedding-demo small {
|
||||
color: var(--muted);
|
||||
font-size: 0.7rem;
|
||||
}
|
||||
|
||||
.sequence-compare,
|
||||
.train-infer {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
max-width: 900px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.sequence-compare > div,
|
||||
.train-infer > article {
|
||||
min-height: 230px;
|
||||
padding: 26px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.rnn-line,
|
||||
.attn-lines {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
min-height: 78px;
|
||||
margin: 24px 0;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.rnn-line b,
|
||||
.attn-lines b,
|
||||
.attn-lines em {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 38px;
|
||||
height: 38px;
|
||||
border: 1px solid var(--line-strong);
|
||||
border-radius: 50%;
|
||||
font: 0.72rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.attn-lines {
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.attn-lines strong {
|
||||
position: absolute;
|
||||
right: 8px;
|
||||
bottom: 4px;
|
||||
color: var(--copper);
|
||||
font: 1.3rem/1 var(--mono);
|
||||
letter-spacing: 0.3em;
|
||||
}
|
||||
|
||||
.attn-lines em {
|
||||
border-color: var(--copper);
|
||||
background: var(--copper-pale);
|
||||
color: var(--copper);
|
||||
font-style: normal;
|
||||
}
|
||||
|
||||
.sequence-compare p,
|
||||
.train-infer p {
|
||||
color: var(--muted);
|
||||
font-size: 0.78rem;
|
||||
line-height: 1.7;
|
||||
}
|
||||
|
||||
.qkv-grid,
|
||||
.myth-list {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||||
max-width: 900px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.qkv-grid article,
|
||||
.myth-list article {
|
||||
min-height: 190px;
|
||||
padding: 24px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.qkv-grid h3,
|
||||
.myth-list h3 {
|
||||
margin: 24px 0 12px;
|
||||
font-size: 1.05rem;
|
||||
}
|
||||
|
||||
.qkv-grid p,
|
||||
.myth-list p {
|
||||
color: var(--muted);
|
||||
font-size: 0.78rem;
|
||||
line-height: 1.72;
|
||||
}
|
||||
|
||||
.block-diagram {
|
||||
display: grid;
|
||||
max-width: 660px;
|
||||
gap: 10px;
|
||||
margin: 34px 0;
|
||||
}
|
||||
|
||||
.block-diagram > div:not(.stream) {
|
||||
min-height: 82px;
|
||||
padding: 17px 22px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.block-diagram > .accent {
|
||||
border-color: var(--sky) !important;
|
||||
background: var(--sky-pale) !important;
|
||||
}
|
||||
|
||||
.block-diagram > .warm {
|
||||
border-color: var(--copper) !important;
|
||||
background: var(--copper-pale) !important;
|
||||
}
|
||||
|
||||
.block-diagram > .merge {
|
||||
display: flex;
|
||||
min-height: 45px !important;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
background: var(--paper-deep) !important;
|
||||
}
|
||||
|
||||
.block-diagram b,
|
||||
.block-diagram small {
|
||||
display: block;
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
.block-diagram small {
|
||||
color: var(--muted);
|
||||
font-size: 0.7rem;
|
||||
}
|
||||
|
||||
.block-diagram .stream {
|
||||
display: grid;
|
||||
grid-template-columns: 30px 1fr;
|
||||
color: var(--muted);
|
||||
font: 0.67rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.train-infer h3 {
|
||||
margin: 26px 0 13px;
|
||||
font-size: 1.2rem;
|
||||
}
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.token-journey {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.token-journey > i {
|
||||
transform: rotate(90deg);
|
||||
}
|
||||
|
||||
.embedding-demo,
|
||||
.sequence-compare,
|
||||
.train-infer,
|
||||
.qkv-grid,
|
||||
.myth-list {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</BaseLayout>
|
||||
@@ -0,0 +1,298 @@
|
||||
---
|
||||
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||
import ArchitectureExplorer from "@/components/ArchitectureExplorer.astro";
|
||||
import DeepSeekLineage from "@/components/DeepSeekLineage.astro";
|
||||
import { chapters, statusLabel } from "@/data/chapters";
|
||||
|
||||
const routes: Record<string, string> = {
|
||||
roadmap: "/roadmap/",
|
||||
foundations: "/foundations/",
|
||||
};
|
||||
|
||||
const paths = [
|
||||
{
|
||||
id: "beginner",
|
||||
number: "01",
|
||||
label: "零基础",
|
||||
title: "从“下一个词”开始，不先背 Transformer 公式",
|
||||
text: "先建立 Token、概率、向量和训练目标的直觉，再通过可操作的注意力实验进入架构。",
|
||||
steps: ["01 语言模型从哪里来", "02 注意力与 Transformer", "03 表示、位置与残差", "04 Scaling Laws"],
|
||||
time: "8–12 小时",
|
||||
target: "能独立阅读主流 LLM 架构图",
|
||||
},
|
||||
{
|
||||
id: "k3",
|
||||
number: "02",
|
||||
label: "K3 反向拆解",
|
||||
title: "先看全貌，再沿组件回到每条技术祖先",
|
||||
text: "适合已经用过大模型、想迅速读懂 K3 报告的人。每个组件都能跳回其历史专题。",
|
||||
steps: ["K3 三维信息流", "KDA 与 MLA", "Stable LatentMoE", "1M Agentic RL 与系统"],
|
||||
time: "6–10 小时",
|
||||
target: "能逐节解释 K3 技术报告",
|
||||
},
|
||||
{
|
||||
id: "deepseek",
|
||||
number: "03",
|
||||
label: "DeepSeek 主线",
|
||||
title: "沿一家公司，看算法—系统协同如何一步步形成",
|
||||
text: "从 DeepSeek LLM 的 dense 基线，到 MoE、MLA、FP8、GRPO、R1、稀疏注意力与百万上下文。",
|
||||
steps: ["DeepSeekMoE", "V2 / MLA", "V3 / FP8 / DualPipe", "Math / GRPO / R1", "V3.2 / V4"],
|
||||
time: "7–11 小时",
|
||||
target: "理解 DeepSeek 论文间的因果关系",
|
||||
},
|
||||
{
|
||||
id: "systems",
|
||||
number: "04",
|
||||
label: "系统工程",
|
||||
title: "跟着一个 Token 穿过 GPU、网络、缓存与服务集群",
|
||||
text: "适合工程背景读者，从计算与内存账本出发理解并行、低精度、通信和推理服务。",
|
||||
steps: ["08 大规模训练", "09 数值与优化", "14 推理服务", "K3 §5 Infrastructure"],
|
||||
time: "10–16 小时",
|
||||
target: "看懂大模型系统报告与性能数字",
|
||||
},
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="LLM Atlas"
|
||||
description="以 Kimi K3 技术报告为锚点，从第一性原理梳理大语言模型的技术发展、关键论文、DeepSeek 谱系与系统工程。"
|
||||
section="home"
|
||||
>
|
||||
<section class="hero">
|
||||
<div class="hero-inner">
|
||||
<div class="hero-copy">
|
||||
<p class="eyebrow"><span>OPEN COURSE / 2026</span> LARGE LANGUAGE MODELS</p>
|
||||
<h1>
|
||||
大模型技术全景
|
||||
<em>从“预测下一个词”到 Kimi K3 的 2.8T 开放前沿</em>
|
||||
</h1>
|
||||
<p class="hero-deck">
|
||||
这是一套按问题脉络组织的中文开放课程。我们从最小直觉出发，追踪每篇关键论文究竟解决了什么，
|
||||
再把注意力、MoE、规模化训练、多模态、推理强化学习与 Agent 系统重新汇入 Kimi K3。
|
||||
</p>
|
||||
<div class="hero-actions">
|
||||
<a class="button primary" href="/roadmap/">选择学习路径 <span aria-hidden="true">↓</span></a>
|
||||
<a class="button" href="/k3/">直接解剖 K3</a>
|
||||
<a class="button" href="/deepseek/">DeepSeek 专题</a>
|
||||
</div>
|
||||
</div>
|
||||
<aside class="hero-aside" aria-label="项目统计">
|
||||
<span>RESEARCH CUTOFF</span>
|
||||
<strong>28 · 07 · 2026</strong>
|
||||
<p>首版持续建设中，所有动态结论带日期</p>
|
||||
<div class="hero-stats">
|
||||
<div><b>16</b><span>核心专题</span></div>
|
||||
<div><b>151</b><span>K3 报告来源</span></div>
|
||||
<div><b>125</b><span>关键论文索引</span></div>
|
||||
<div><b>47p</b><span>K3 技术报告</span></div>
|
||||
</div>
|
||||
</aside>
|
||||
</div>
|
||||
<div class="scope-strip">
|
||||
<span>研究范围</span>
|
||||
<p>表示学习 · Transformer · Scaling · MoE · 长上下文 · 训练系统 · 后训练 · 推理 · Agent · 多模态 · 服务与评测</p>
|
||||
<a href="/progress/">查看实时进度 →</a>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="why">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>00</span> WHY THIS ATLAS</p>
|
||||
<h2>不是论文清单，而是一条能走通的理解路径</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
大模型知识最难的地方不是资料少，而是资料之间的桥断了：初学解释省略公式，论文省略历史，
|
||||
系统报告又默认你已经理解模型。LLM Atlas 专门补这些桥。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="thesis">
|
||||
<span class="micro-label">课程主张</span>
|
||||
<p>
|
||||
真正理解一项技术，需要同时回答：它解决什么瓶颈、核心机制如何工作、证据是否支持、
|
||||
工程代价是什么，以及下一篇论文为什么自然会出现。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="feature-grid">
|
||||
<article class="feature-card">
|
||||
<span class="card-number">01 / PROBLEM</span>
|
||||
<h3>从问题开始</h3>
|
||||
<p>先看到旧方法撞上的墙，再引入新名词。KDA、MLA、MoE 都不再是凭空掉下来的缩写。</p>
|
||||
</article>
|
||||
<article class="feature-card">
|
||||
<span class="card-number">02 / LAYERS</span>
|
||||
<h3>四层解释</h3>
|
||||
<p>同一概念同时提供直觉、机制、论文和工程四层入口，允许读者在适合自己的深度停留。</p>
|
||||
</article>
|
||||
<article class="feature-card">
|
||||
<span class="card-number">03 / VISUAL</span>
|
||||
<h3>图必须能帮助推理</h3>
|
||||
<p>优先重绘可缩放、可交互的架构图；每种颜色固定含义，简化之处明确标出。</p>
|
||||
</article>
|
||||
<article class="feature-card">
|
||||
<span class="card-number">04 / EVIDENCE</span>
|
||||
<h3>事实、解释、推断分开</h3>
|
||||
<p>关键结论回到论文和官方实现。榜单写明 harness、工具预算与截止日期，未知就明确说未知。</p>
|
||||
</article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="paths">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>01</span> LEARNING PATHS</p>
|
||||
<h2>你不必从第一页顺序读到最后</h2>
|
||||
</div>
|
||||
<p class="section-lead">四条路径共享同一张知识图。点击路径切换，章节之间的先修关系始终可追踪。</p>
|
||||
</div>
|
||||
|
||||
<div class="pathways" data-pathways>
|
||||
<div class="path-tabs" role="tablist" aria-label="学习路径">
|
||||
{paths.map((path, index) => (
|
||||
<button
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected={index === 0 ? "true" : "false"}
|
||||
data-path-button={path.id}
|
||||
>
|
||||
<span>{path.number}</span>{path.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<div>
|
||||
{paths.map((path, index) => (
|
||||
<div class="path-panel" data-path-panel={path.id} data-active={index === 0 ? "" : undefined}>
|
||||
<div class="path-copy">
|
||||
<span class="micro-label">{path.label}</span>
|
||||
<h3>{path.title}</h3>
|
||||
<p>{path.text}</p>
|
||||
<ol>
|
||||
{path.steps.map((step) => <li>{step}</li>)}
|
||||
</ol>
|
||||
</div>
|
||||
<aside class="path-meta">
|
||||
<div><span>ESTIMATED TIME</span><b>{path.time}</b></div>
|
||||
<div><span>LEARNING OUTCOME</span><b>{path.target}</b></div>
|
||||
<div><span>READING MODE</span><b>直觉 → 机制 → 论文 → 工程</b></div>
|
||||
</aside>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="chapters">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>02</span> CURRICULUM MAP</p>
|
||||
<h2>16 个专题，拼成一张完整技术地图</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
进度条表示内容成熟度，不代表相关领域的重要性。首版先打通全站骨架，再逐章扩写到论文与工程层。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="chapter-grid">
|
||||
{chapters.map((chapter) => {
|
||||
const href = routes[chapter.slug] ?? `/roadmap/#chapter-${chapter.number}`;
|
||||
return (
|
||||
<a class="chapter-card" href={href}>
|
||||
<div class="chapter-top">
|
||||
<span class="chapter-number">{chapter.number}</span>
|
||||
<span class={`status ${chapter.status}`}>{statusLabel[chapter.status]}</span>
|
||||
</div>
|
||||
<h3>{chapter.title}</h3>
|
||||
<p>{chapter.question}</p>
|
||||
<div class="chapter-bottom">
|
||||
<div class="progress-track" aria-label={`完成 ${chapter.progress}%`}>
|
||||
<span style={`width:${chapter.progress}%`}></span>
|
||||
</div>
|
||||
<div class="chapter-meta">
|
||||
<span>{chapter.papers} 篇核心论文</span>
|
||||
<span>{chapter.progress}%</span>
|
||||
</div>
|
||||
</div>
|
||||
</a>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="k3-map">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>03</span> KIMI K3 AS THE CONFLUENCE</p>
|
||||
<h2>K3 把三种信息流和一套庞大系统放进同一模型</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
K3 报告最好的阅读钥匙，是把架构看成 token、depth、channel 三个方向的信息流；
|
||||
视觉输入与训练/服务基础设施则包住这三条轴。
|
||||
</p>
|
||||
</div>
|
||||
<ArchitectureExplorer />
|
||||
<a class="button primary" href="/k3/">进入完整 K3 技术报告导读 →</a>
|
||||
</section>
|
||||
|
||||
<section class="section" id="deepseek-spotlight">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>04</span> DEEPSEEK SPOTLIGHT</p>
|
||||
<h2>DeepSeek：一条极适合学习“算法—系统协同”的论文主线</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
从细粒度 MoE、MLA 和 FP8，到 GRPO 与 R1，DeepSeek 的每篇报告都在解决上一代留下的明确约束。
|
||||
这里会给予它比普通模型谱系更细的篇幅。
|
||||
</p>
|
||||
</div>
|
||||
<DeepSeekLineage />
|
||||
<a class="button primary" href="/deepseek/">阅读 DeepSeek 专题 →</a>
|
||||
</section>
|
||||
|
||||
<section class="section compact" id="method">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>05</span> OPEN RESEARCH</p>
|
||||
<h2>课程本身也是一项开放研究工程</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
网站源码、路线、进度和研究规范全部公开。Grok 用来扩展检索线索，最终结论由一手论文和官方实现核验。
|
||||
</p>
|
||||
</div>
|
||||
<div class="method-grid">
|
||||
<article class="method-card">
|
||||
<span class="card-number">P0</span>
|
||||
<h3>一手论文</h3>
|
||||
<p>arXiv、会议、期刊与作者正式技术报告，承载核心机制和实验数字。</p>
|
||||
</article>
|
||||
<article class="method-card">
|
||||
<span class="card-number">P1</span>
|
||||
<h3>官方实现</h3>
|
||||
<p>仓库、模型卡、训练与评测代码，用于确认论文描述如何落到真实系统。</p>
|
||||
</article>
|
||||
<article class="method-card">
|
||||
<span class="card-number">P2</span>
|
||||
<h3>官方说明</h3>
|
||||
<p>实验室博客、系统卡和产品文档；尚无论文的新模型会临时使用并标明状态。</p>
|
||||
</article>
|
||||
<article class="method-card">
|
||||
<span class="card-number">P3+</span>
|
||||
<h3>独立复现</h3>
|
||||
<p>第三方结果用于检验外部有效性；二手文章只发现线索，不承载关键事实。</p>
|
||||
</article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<script>
|
||||
document.querySelectorAll<HTMLElement>("[data-pathways]").forEach((root) => {
|
||||
const buttons = [...root.querySelectorAll<HTMLButtonElement>("[data-path-button]")];
|
||||
const panels = [...root.querySelectorAll<HTMLElement>("[data-path-panel]")];
|
||||
const select = (id: string) => {
|
||||
buttons.forEach((button) => button.setAttribute("aria-selected", String(button.dataset.pathButton === id)));
|
||||
panels.forEach((panel) => panel.toggleAttribute("data-active", panel.dataset.pathPanel === id));
|
||||
};
|
||||
buttons.forEach((button) => button.addEventListener("click", () => select(button.dataset.pathButton ?? "beginner")));
|
||||
});
|
||||
</script>
|
||||
</BaseLayout>
|
||||
@@ -0,0 +1,710 @@
|
||||
---
|
||||
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||
import ArchitectureExplorer from "@/components/ArchitectureExplorer.astro";
|
||||
|
||||
const toc = [
|
||||
["00", "orientation", "先建立阅读坐标"],
|
||||
["01", "architecture", "一张图看完整架构"],
|
||||
["02", "kda", "KDA：线性工作记忆"],
|
||||
["03", "mla", "Gated MLA：全局回看"],
|
||||
["04", "attnres", "Attention Residuals"],
|
||||
["05", "moe", "Stable LatentMoE"],
|
||||
["06", "vision", "原生视觉与优化器"],
|
||||
["07", "pretrain", "预训练与长上下文"],
|
||||
["08", "posttrain", "后训练与推理强度"],
|
||||
["09", "agents", "Agentic RL 环境"],
|
||||
["10", "infra", "2.8T / 1M 基础设施"],
|
||||
["11", "evaluation", "评测与局限"],
|
||||
["↳", "sources", "论文链与一手来源"],
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="Kimi K3 技术报告完整导读"
|
||||
description="从三维信息流、KDA、Gated MLA、Attention Residuals、Stable LatentMoE 到 1M Agentic RL 与系统基础设施，逐步读懂 Kimi K3 技术报告。"
|
||||
section="k3"
|
||||
>
|
||||
<header class="page-hero">
|
||||
<div class="page-hero-inner">
|
||||
<div>
|
||||
<p class="eyebrow"><span>ANCHOR REPORT / 01</span> KIMI K3</p>
|
||||
<h1>把 47 页 K3 报告<br />读成一条因果链</h1>
|
||||
<p class="lead">
|
||||
不从 2.8 万亿这个最大数字开始，而从模型同时面对的四个瓶颈开始：
|
||||
序列太长、网络太深、容量太大、Agent 轨迹太久。K3 的每个新组件都可以放回这四个问题。
|
||||
</p>
|
||||
</div>
|
||||
<dl class="page-facts">
|
||||
<div><dt>REPORT</dt><dd>47 页 · 151 条参考来源</dd></div>
|
||||
<div><dt>MODEL</dt><dd>2.8T total / 104B active</dd></div>
|
||||
<div><dt>CONTEXT</dt><dd>1,048,576 tokens</dd></div>
|
||||
<div><dt>ARCH</dt><dd>69 KDA + 24 Gated MLA</dd></div>
|
||||
<div><dt>STATUS</dt><dd>首版导读 · 持续扩写</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="report-shell">
|
||||
<aside class="side-rail" aria-label="本页目录">
|
||||
<p>CONTENTS</p>
|
||||
<ol>
|
||||
{toc.map(([number, id, label]) => (
|
||||
<li><a href={`#${id}`}><span>{number}</span>{label}</a></li>
|
||||
))}
|
||||
</ol>
|
||||
<div class="rail-note">
|
||||
<b>证据状态</b>
|
||||
本页架构和训练事实来自 K3 官方技术报告；直觉类比与简图为本站原创解释。
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<article class="article">
|
||||
<section class="article-section" id="orientation">
|
||||
<p class="eyebrow"><span>00</span> READING ORIENTATION</p>
|
||||
<h2>先别急着记缩写：K3 在扩展三种信息流</h2>
|
||||
<p class="lede">
|
||||
K3 报告自己的组织方式非常漂亮：沿序列长度、网络深度和模型宽度扩展信息流。
|
||||
这比“又加了哪些模块”更接近真正的设计逻辑。
|
||||
</p>
|
||||
<div class="concept-grid">
|
||||
<article>
|
||||
<span>SEQUENCE / TOKEN</span>
|
||||
<h3>一句话内部怎样交流</h3>
|
||||
<p>KDA 负责低成本持续记忆，周期性的 Gated MLA 负责完整全局交互。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>DEPTH / LAYER</span>
|
||||
<h3>信息怎样穿过 93 层</h3>
|
||||
<p>Attention Residuals 让当前层有选择地读取早期层，而不只是接收统一累加结果。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>WIDTH / EXPERT</span>
|
||||
<h3>容量怎样远大于计算量</h3>
|
||||
<p>Stable LatentMoE 建立 896 个路由专家，每个 Token 只激活 16 个。</p>
|
||||
</article>
|
||||
</div>
|
||||
<p>
|
||||
第四个问题藏在模型外部：这些结构要在真实硬件上完成预训练、百万 Token 强化学习和线上服务。
|
||||
所以报告第 5 章不是附录，而是 K3 设计的一半。没有 FlashKDA、专家并行、外置 KV Cache、
|
||||
可恢复沙箱和集群调度，前面的架构只是一张昂贵的蓝图。
|
||||
</p>
|
||||
<div class="plain-language">
|
||||
<b>一句话版本</b>
|
||||
<p>
|
||||
K3 想让信息在“很长的时间、很深的网络、很宽的专家库”里都能流动，同时让整个系统仍然能被训练和部署。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="architecture">
|
||||
<p class="eyebrow"><span>01</span> ARCHITECTURE OVERVIEW</p>
|
||||
<h2>一张图看完 K3：先看流向，再看数字</h2>
|
||||
<ArchitectureExplorer />
|
||||
|
||||
<h3>关键规格应该怎样读</h3>
|
||||
<div class="spec-grid">
|
||||
<div><span>总参数</span><b>2.8T</b><p>表示模型装下的总容量，不等于每个 Token 都计算全部参数。</p></div>
|
||||
<div><span>激活参数</span><b>104B</b><p>单个 Token 前向时实际经过的参数规模，仍然非常大。</p></div>
|
||||
<div><span>层数</span><b>93</b><p>其中 1 层 dense；注意力由 69 KDA 与 24 Gated MLA 组成。</p></div>
|
||||
<div><span>隐藏维度</span><b>7168</b><p>主干表示宽度；LatentMoE 内部路由空间压到 3584。</p></div>
|
||||
<div><span>专家</span><b>896 → 16</b><p>另有 2 个 shared experts，给通用变换保留稳定路径。</p></div>
|
||||
<div><span>上下文</span><b>1,048,576</b><p>约一百万 Token；能放下不等于不需要上下文管理。</p></div>
|
||||
<div><span>视觉编码器</span><b>401M</b><p>MoonViT-V2 把图像和视频编码到共享表示空间。</p></div>
|
||||
<div><span>部署精度</span><b>MXFP4 / 8</b><p>专家权重 MXFP4、专家激活 MXFP8，非专家模块保留更高精度。</p></div>
|
||||
</div>
|
||||
<div class="warning-note">
|
||||
<b>常见误解：2.8T 不是“每次都算 2.8T”</b>
|
||||
<p>
|
||||
MoE 的核心正是把参数容量和每 Token 计算拆开。2.8T 描述可用参数总量；104B 才更接近一次前向的激活规模。
|
||||
但 104B 依然远高于许多 dense 模型，所以“稀疏”不等于“能在普通显卡轻松运行”。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="kda">
|
||||
<p class="eyebrow"><span>02</span> KIMI DELTA ATTENTION</p>
|
||||
<h2>KDA：不保存所有配对，而是维护一份会更新的工作记忆</h2>
|
||||
<p class="lede">
|
||||
标准注意力会让每个新 Token 和许多旧 Token 直接比较。KDA 改成维护一个固定形状的状态：
|
||||
新信息到来时，先有选择地遗忘，再用“纠错式写入”更新这份状态。
|
||||
</p>
|
||||
|
||||
<h3>从标准注意力的账单说起</h3>
|
||||
<p>
|
||||
长度为 <em>n</em> 的序列，如果做完整 self-attention，需要形成近似 <em>n × n</em> 的交互。
|
||||
在一百万 Token 处，哪怕使用 FlashAttention 避免把整张矩阵写回显存，计算量仍按平方增长。
|
||||
线性注意力的基本想法是先把历史压进状态 <em>S</em>，读取时只让 Query 查询状态。
|
||||
</p>
|
||||
|
||||
<div class="formula">
|
||||
Sₜ = (I − βₜkₜkₜᵀ) · Diag(αₜ) · Sₜ₋₁ + βₜkₜvₜᵀ<br />
|
||||
õₜ = Sₜᵀqₜ
|
||||
<small>
|
||||
教学化书写，符号对应 K3 Report Eq. 1。α 控制各通道保留多少旧状态；β 控制本次写入强度；
|
||||
delta rule 先擦除当前 key 已有的预测，再写入新的 value。
|
||||
</small>
|
||||
</div>
|
||||
|
||||
<h3>为什么叫 Delta Rule</h3>
|
||||
<p>
|
||||
如果直接做 <code>S ← S + kvᵀ</code>，同一个 key 反复出现会不断累加，状态容易被重复信息污染。
|
||||
Delta Rule 先问“现有状态对这个 key 已经会输出什么”，只写入真实 value 与旧预测之间的差值。
|
||||
它像修改文档：不是每次把整篇内容追加到末尾，而是找到对应位置做差量更新。
|
||||
</p>
|
||||
|
||||
<h3>K3 相比 Kimi Linear 改了什么</h3>
|
||||
<ol>
|
||||
<li>
|
||||
<strong>逐通道遗忘门。</strong>α 不是一个标量，而是 key channel 级别的保留率；不同维度可以拥有不同记忆时长。
|
||||
</li>
|
||||
<li>
|
||||
<strong>把 log-decay 下界固定为 −5。</strong>Kimi Linear 的衰减映射下界无穷，会让 chunk 内累计衰减的倒数爆大；
|
||||
K3 将其限制在可由 BF16 表示的范围，从而让对角 tile 也能直接使用 Tensor Core 的稠密矩阵乘。
|
||||
</li>
|
||||
<li>
|
||||
<strong>全秩、输入相关的输出门。</strong>读取状态后，模型可以对每个输入动态决定哪些输出通道放行。
|
||||
</li>
|
||||
<li>
|
||||
<strong>块内并行、块间递归。</strong>序列分成 chunk：chunk 之间传状态，chunk 内改写成并行矩阵计算，兼顾训练吞吐与线性推理。
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<div class="evidence-note">
|
||||
<b>论文证据怎样看</b>
|
||||
<p>
|
||||
K3 报告把整体约 2.5× scaling efficiency 改善归因于 KDA、AttnRes、Stable LatentMoE 与训练/数据配方的组合，
|
||||
不能把 2.5× 单独归到 KDA。KDA 的独立机制与消融需要同时阅读
|
||||
<a href="https://arxiv.org/abs/2510.26692">Kimi Linear</a> 和 K3 §2.1。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="mla">
|
||||
<p class="eyebrow"><span>03</span> GATED MLA</p>
|
||||
<h2>周期性 MLA：工作记忆之外，仍要把全局摊开来看</h2>
|
||||
<p>
|
||||
纯递归状态的优势是成本低，弱点是历史被压进固定大小状态后，精确回看某个遥远 Token 会更难。
|
||||
K3 没有把完整注意力全部删除，而是采用 <strong>3 层 KDA + 1 层 Gated MLA</strong> 的混合模式，
|
||||
并保证主干最后一层也是 Gated MLA。
|
||||
</p>
|
||||
|
||||
<h3>MLA 在压缩什么</h3>
|
||||
<p>
|
||||
Multi-head Latent Attention 由 <a href="https://arxiv.org/abs/2405.04434">DeepSeek-V2</a> 引入。
|
||||
标准多头注意力会为每个历史 Token 缓存多头的 K 和 V；MLA 先把它们压到低维 latent <em>cₜ</em>，
|
||||
服务时缓存 <em>cₜ</em>，计算注意力时再通过投影恢复各头所需内容。它保留全局 token-to-token 交互，
|
||||
同时显著缩小 KV Cache。
|
||||
</p>
|
||||
|
||||
<div class="formula">
|
||||
cₜ = Wᶜxₜ → 缓存 cₜ → 按需上投影为各头的 K / V
|
||||
<small>这是结构直觉，不是 MLA 完整公式。旋转位置编码的解耦与吸收技巧会在“长上下文”专题单独推导。</small>
|
||||
</div>
|
||||
|
||||
<h3>为什么 K3 的 MLA 不再使用位置编码</h3>
|
||||
<p>
|
||||
K3 在全部 MLA 层采用 NoPE：Query 和 Key 不显式加入位置编码。位置与近因信息主要由穿插的 KDA 提供，
|
||||
MLA 专注于全局内容匹配。这样扩展上下文时，也不必重新调 RoPE base 或应用 YaRN。
|
||||
这是混合架构的一个重要分工：<strong>KDA 提供位置敏感的持续混合，MLA 提供不受限的全局内容交互。</strong>
|
||||
</p>
|
||||
<p>
|
||||
K3 还给 MLA 加入与 KDA 对齐的输入相关全秩输出门，并在训练时将 attention output 保持为 FP32，
|
||||
以纠正 FlashAttention 中有偏的舍入误差；代价是更大的片上存储，报告为此重新安排了 kernel 中的 tile 缓冲重叠。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="attnres">
|
||||
<p class="eyebrow"><span>04</span> ATTENTION RESIDUALS</p>
|
||||
<h2>把注意力从“时间方向”旋转到“深度方向”</h2>
|
||||
<p class="lede">
|
||||
Transformer 用注意力解决了 RNN 必须把全部历史压进一个时间状态的问题。
|
||||
K3 提问：普通 residual 是否又把所有早期层压进了一个深度状态？
|
||||
</p>
|
||||
<p>
|
||||
标准残差不断做 <code>hₗ₊₁ = hₗ + Fₗ(hₗ)</code>。它很利于梯度传播，但越早的特征会在统一累加中混在一起。
|
||||
Full AttnRes 为每一层设置可学习 pseudo-query，对 embedding 和所有先前层输出计算权重，再按权重组合。
|
||||
这里的“Query”不是当前 Token 内容，而是“第 l 层通常希望从哪些深度取信息”的可学习参数。
|
||||
</p>
|
||||
|
||||
<div class="formula">
|
||||
αᵢ→ₗ = exp(qₗᵀ · RMSNorm(kᵢ)) / Σⱼ exp(qₗᵀ · RMSNorm(kⱼ))<br />
|
||||
hₗ = Σᵢ αᵢ→ₗ · vᵢ
|
||||
<small>
|
||||
对应 K3 Report Eq. 8–9 的简化展示。RMSNorm 防止幅值大的层仅凭数值尺度垄断权重。
|
||||
</small>
|
||||
</div>
|
||||
|
||||
<h3>为什么又要做 Block AttnRes</h3>
|
||||
<p>
|
||||
Full AttnRes 的层数平方计算在不足 100 层时并不离谱，真正麻烦是保留所有层输出带来的内存与流水线跨 stage 通信。
|
||||
K3 把层分成 block：block 内先求和，跨 block 才做完整深度注意力。报告称经验上约 8 个 block 可保留大部分收益；
|
||||
K3 使用 12 层一个 block，加上 embedding 来源，总计形成 9 个深度来源组。
|
||||
</p>
|
||||
<div class="warning-note">
|
||||
<b>当前证据边界</b>
|
||||
<p>
|
||||
Kimi Team 已公开 <a href="https://arxiv.org/abs/2603.15031">Attention Residuals 独立预印本</a>，
|
||||
在 48B-total / 3B-active 模型上用 1.4T Token 做了 scaling 与消融；K3 则证明它能进入 2.8T 系统。
|
||||
但它仍是 2026 年的新方法，第三方复现和跨架构外部有效性需要持续积累。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="moe">
|
||||
<p class="eyebrow"><span>05</span> STABLE LATENTMOE</p>
|
||||
<h2>896 选 16：极稀疏带来容量，也放大不稳定</h2>
|
||||
<p>
|
||||
传统 MoE 让被选中的每个专家都处理完整 <em>d</em> 维 Token。选更多专家时，不只计算变多，
|
||||
Token 跨设备发送的数据和专家权重读取也随 routing multiplicity 增长。
|
||||
LatentMoE 把通用的 full-width 路径留给 shared experts，把 routed experts 放进较窄 latent space：
|
||||
K3 主干宽度 7168，路由 latent 维度 3584。
|
||||
</p>
|
||||
|
||||
<div class="moe-flow" role="img" aria-label="Stable LatentMoE 信息流示意">
|
||||
<div><span>7168d</span><b>Token x</b><small>完整表示</small></div>
|
||||
<i>→</i>
|
||||
<div class="shared"><span>FULL</span><b>2 Shared</b><small>公共变换</small></div>
|
||||
<div class="split-label">同时</div>
|
||||
<div><span>W↓</span><b>压到 3584d</b><small>latent route</small></div>
|
||||
<i>→</i>
|
||||
<div class="experts"><span>TOP-K</span><b>896 选 16</b><small>专门知识</small></div>
|
||||
<i>→</i>
|
||||
<div><span>W↑</span><b>回到 7168d</b><small>合并输出</small></div>
|
||||
</div>
|
||||
|
||||
<h3>Stable 具体在稳定什么</h3>
|
||||
<p>报告指出极端稀疏度会放大两个失败模式：routed path 连续多次矩阵乘导致内部激活爆炸；近千专家的负载难以靠旧的无辅助损失 bias update 稳定平衡。K3 加入三项修复：</p>
|
||||
<ol>
|
||||
<li><strong>Normalized LatentMoE：</strong>在上投影之前加入 RMSNorm，阻止 routed aggregate 的尺度失控。</li>
|
||||
<li><strong>SiTU-GLU：</strong>用有界的 tanh 分支近似 SwiGLU 的近原点行为，同时限制大正输入的输出幅值。</li>
|
||||
<li><strong>Quantile Balancing：</strong>不只盯平均负载，而用路由分数的分位统计更新专家 bias，适应近千专家下更复杂的分布。</li>
|
||||
</ol>
|
||||
<p>
|
||||
这条技术线与 DeepSeekMoE 紧密相连：shared experts 保存公共知识，细粒度 routed experts 促进专业化。
|
||||
K3 再借 LatentMoE 让“选 16 个专家”的通信和权重读取可承受，并为极端稀疏补上稳定性机制。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="vision">
|
||||
<p class="eyebrow"><span>06</span> NATIVE VISION & MUON</p>
|
||||
<h2>视觉是第一类输入；优化器也按注意力头重新分组</h2>
|
||||
<h3>MoonViT-V2 的角色</h3>
|
||||
<p>
|
||||
401M 参数 MoonViT-V2 将图片与视频编码成视觉特征，轻量 projector 把它们映射到语言主干的 embedding space。
|
||||
“原生”最重要的含义不是“能看图”，而是视觉数据在预训练阶段就进入统一模型；后训练中的 Agent 还能把截图、
|
||||
图表、裁剪/缩放后的新图片当作连续 observation，形成看—行动—再看的闭环。
|
||||
</p>
|
||||
<p>
|
||||
报告描述了渐进式多模态训练：先固定语言模型训练视觉组件，再逐步解冻主干；多模态编码器优化则涉及动态分辨率、
|
||||
packing 与避免视觉计算造成流水线 bubble。完整视觉谱系会从 ViT、CLIP、Flamingo、BLIP-2、LLaVA 讲到 Kimi-VL。
|
||||
</p>
|
||||
|
||||
<h3>Per-Head Muon 为什么出现</h3>
|
||||
<p>
|
||||
Muon 在矩阵更新上做正交化，目标是比逐元素 Adam 更好地控制隐藏层更新。K3 采用 per-head 分组：
|
||||
对 Q/K/V 等多头投影，按 head 分开应用 Muon，而不是把整块矩阵当作一个整体。
|
||||
直觉上，每个 head 是相对独立的子空间，按 head 归一更新可减少大矩阵不同部分相互干扰。
|
||||
</p>
|
||||
<div class="warning-note">
|
||||
<b>不要把优化器效果和架构效果混为一谈</b>
|
||||
<p>
|
||||
报告把 Per-Head Muon 放进统一训练配方，但整体 2.5× scaling efficiency 并不是单项优化器消融结论。
|
||||
Muon 的通用可扩展性应另外阅读 <a href="https://arxiv.org/abs/2502.16982">Muon is Scalable for LLM Training</a>。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="pretrain">
|
||||
<p class="eyebrow"><span>07</span> PRE-TRAINING</p>
|
||||
<h2>预训练：扩展的不只是参数，还有数据、长度和数值配方</h2>
|
||||
<p>
|
||||
K3 报告将模型能力的提升明确归因于架构、数据和训练 recipe 的共同作用，但没有公开足以复现的完整语料配比。
|
||||
这需要区分两件事：我们可以准确解释训练阶段和公开机制，却不能根据少量描述虚构完整数据集。
|
||||
</p>
|
||||
|
||||
<h3>Scaling Law 在这里怎样用</h3>
|
||||
<p>
|
||||
团队先在较小规模训练一系列模型，拟合 loss 与参数、数据和计算之间的关系，再用它比较架构候选与估计 2.8T 模型的预算。
|
||||
K3 所称约 2.5× overall scaling efficiency，是相对 K2 的经验缩放效率：在相同计算下取得更低 loss，或达到同等 loss 所需计算更少。
|
||||
它不是“推理速度提高 2.5×”，也不是所有下游任务统一提升 2.5×。
|
||||
</p>
|
||||
|
||||
<h3>长上下文不是把配置里的 32K 改成 1M</h3>
|
||||
<p>
|
||||
上下文扩展需要长度 curriculum、长文档数据、并行策略和稳定训练。K3 的 NoPE MLA 避免 RoPE 外推参数；
|
||||
KDA 的递归/块并行结构降低长序列成本；系统侧再用 KDA Context Parallelism 分摊状态与 chunk。
|
||||
最后还要在 post-training 里真正让模型经历长轨迹，否则“窗口能装下”不等于“模型会有效使用”。
|
||||
</p>
|
||||
<div class="plain-language">
|
||||
<b>窗口容量 vs. 使用能力</b>
|
||||
<p>
|
||||
能把一百万 Token 放进输入，像图书馆允许你搬进一百万字；能否跨文档找到证据、维持任务状态并避免遗忘，
|
||||
才是在考你是否真的读懂。K3 同时用架构、长程 RL 和上下文管理训练这件事。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="posttrain">
|
||||
<p class="eyebrow"><span>08</span> POST-TRAINING</p>
|
||||
<h2>九位“专家老师”，最后蒸馏成一个可调思考强度的模型</h2>
|
||||
<p>
|
||||
K3 的 post-training 覆盖 SFT 与 RL，RL 按三类 domain 与三种 reasoning effort 组织：
|
||||
general reasoning/knowledge、agentic、coding × low/high/max，形成九个专业策略。
|
||||
然后用 Multi-Teacher On-Policy Distillation（MOPD）把它们整合进统一学生模型。
|
||||
</p>
|
||||
|
||||
<div class="teacher-grid">
|
||||
<div class="corner">DOMAIN × EFFORT</div>
|
||||
<b>LOW</b><b>HIGH</b><b>MAX</b>
|
||||
<strong>REASONING</strong><span>Rₗ</span><span>Rₕ</span><span>Rₘ</span>
|
||||
<strong>AGENTIC</strong><span>Aₗ</span><span>Aₕ</span><span>Aₘ</span>
|
||||
<strong>CODING</strong><span>Cₗ</span><span>Cₕ</span><span>Cₘ</span>
|
||||
</div>
|
||||
|
||||
<h3>为什么要训练不同 reasoning effort</h3>
|
||||
<p>
|
||||
“更久思考”会提高部分难题表现，也会增加延迟、成本和过度思考。K3 通过预算控制参数训练 max/high/low 专家：
|
||||
先在较大预算下追求能力，再逐步退火得到更短策略。不同 domain 的预算调整由人工参与配置，
|
||||
因为一道数学题、一次网页研究和一个代码仓库任务的合理轨迹长度并不相同。
|
||||
</p>
|
||||
|
||||
<h3>On-policy distillation 为什么比离线模仿更适合长轨迹</h3>
|
||||
<p>
|
||||
普通蒸馏常让学生模仿教师已经生成的固定答案；学生一旦在真实生成中走到不同前缀，教师数据就失去覆盖。
|
||||
On-policy distillation 让学生自己生成当前前缀，再在这个前缀上比较教师和学生对下一个 Token 的概率，
|
||||
形成稠密 reward。这样训练分布更贴近学生真正会访问的状态，也能自然结合 partial rollout。
|
||||
</p>
|
||||
|
||||
<h3>部署约束直接进入后训练</h3>
|
||||
<p>
|
||||
K3 从 SFT 开始对 MoE expert weights 使用 MXFP4 QAT，expert activations 使用 MXFP8；
|
||||
RL rollout 和训练采用相同量化方案，减少训练—推理失配。预训练中的 MTP 层随后被微调为 EAGLE-3 风格 draft model，
|
||||
用低/中/高层 AttnRes 特征预测候选 Token，并直接优化与无损推测采样接受率相关的 LK loss。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="agents">
|
||||
<p class="eyebrow"><span>09</span> AGENTIC RL</p>
|
||||
<h2>Agent 能力不是只靠“更聪明”，而是靠环境提供可学习的反馈</h2>
|
||||
<p>
|
||||
K3 报告最值得细读的部分之一，是它把 Agent 后训练写成环境工程：工具、system prompt、context management、
|
||||
skills、memory、subagents 和 harness 都成为可组合变量。训练时动态实例化 Kimi Code、Claude Code、Codex、
|
||||
OpenClaw、Hermes 等风格，避免模型过拟合一套固定工具 schema。
|
||||
</p>
|
||||
|
||||
<h3>六类环境回答六种失败模式</h3>
|
||||
<div class="environment-list">
|
||||
<article><span>01</span><h4>可验证搜索与专业工作</h4><p>多步检索、投行、数据分析、法律交付物；奖励落在证据和最终成果。</p></article>
|
||||
<article><span>02</span><h4>视觉推理</h4><p>模型在隔离 Python 环境里裁剪、放大、计算并把新图片作为 observation。</p></article>
|
||||
<article><span>03</span><h4>GPU Kernel 优化</h4><p>先过数值正确性，再比较专家实现与硬件 roofline，并检测缓存/降精度等作弊。</p></article>
|
||||
<article><span>04</span><h4>长期个人助理</h4><p>用 Gmail、Notion、Slack 等 mock app 构造跨多日事件；单次可达数千工具调用。</p></article>
|
||||
<article><span>05</span><h4>Autonomous Execution</h4><p>只给初始状态、目标、约束、工具和 verifier，不提供参考轨迹；奖励最终环境状态。</p></article>
|
||||
<article><span>06</span><h4>Web 开发</h4><p>容器内构建网页、游戏、3D、可视化；功能测试、结构/像素相似与模型检查共同评分。</p></article>
|
||||
</div>
|
||||
|
||||
<h3>为什么 verifier 比“模型说完成了”更重要</h3>
|
||||
<p>
|
||||
长任务最常见的失败之一，是 Agent 输出一段令人信服的总结，却没有真正改变目标状态。
|
||||
K3 的 AET 把 reward 绑定到独立 verifier 读取的环境结果；public verifier 提供诊断，
|
||||
hidden verifier 检查保留场景，并限制提交预算以减少 reward hacking。
|
||||
</p>
|
||||
<div class="plain-language">
|
||||
<b>这条主线会贯穿 Agent 专题</b>
|
||||
<p>
|
||||
可靠 Agent 的学习单位不是“一条漂亮回答”，而是状态明确、动作可执行、反馈可验证的一整段轨迹。
|
||||
Harness 多样化、持久环境和独立 verifier，分别处理接口过拟合、短视行为和自我宣告成功。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="infra">
|
||||
<p class="eyebrow"><span>10</span> INFRASTRUCTURE</p>
|
||||
<h2>2.8T 参数与 1M 轨迹，迫使系统重新设计状态放在哪里</h2>
|
||||
<p>
|
||||
K3 基础设施可以按三类状态理解：模型状态（参数、优化器）、序列状态（KDA state、KV Cache）、
|
||||
环境状态（代码仓库、应用、microVM）。三类状态的寿命和移动成本完全不同。
|
||||
</p>
|
||||
|
||||
<div class="system-stack">
|
||||
<article><span>ONLINE SERVING</span><h3>请求级状态</h3><p>KDA-aware prefix cache、专用 kernel、cache/budget-aware fleet scheduling，把共享前缀和不同思考预算纳入调度。</p></article>
|
||||
<article><span>1M AGENTIC RL</span><h3>轨迹级状态</h3><p>partial rollout、外置 KV retention、adaptive throttling 与可恢复 microVM，避免每次更新都丢掉漫长轨迹。</p></article>
|
||||
<article><span>3T PRE-TRAIN</span><h3>训练级状态</h3><p>MoonEP 用静态计算形状、平衡专家执行与 zero-copy communication，配合内存优化和多模态 encoder 调度。</p></article>
|
||||
<article><span>KDA CO-DESIGN</span><h3>算子级状态</h3><p>针对不同序列 regime 的 fused kernel、KDA Context Parallelism 与状态感知前缀缓存。</p></article>
|
||||
</div>
|
||||
|
||||
<h3>MoonEP 在解决什么</h3>
|
||||
<p>
|
||||
MoE 的理论 FLOPs 很漂亮，真实系统却可能被“这个专家突然收到太多 Token”拖垮。
|
||||
跨卡 all-to-all 通信、专家权重读取和不规则 shape 都会制造等待。K3 报告称 MoonEP 追求 perfectly balanced expert execution，
|
||||
使用静态计算形状和零拷贝通信把路由后的 Token 送到对应专家。这里的“平衡”是系统执行层结果，不等于路由概率天然均匀；
|
||||
它和模型侧 Quantile Balancing 是互补层次。
|
||||
</p>
|
||||
|
||||
<h3>为什么长程 RL 需要保留外部状态</h3>
|
||||
<p>
|
||||
传统 RL rollout 常在模型更新后重新生成。若一条轨迹已经调用工具几百次、积累几十万 Token，
|
||||
重新生成会浪费巨大。K3 将 KV Cache 放到外部、允许 partial rollout 暂停并续接；
|
||||
同时把工具环境放进可恢复 microVM，使代码、文件和应用状态与语言上下文一起跨训练 step 存续。
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="evaluation">
|
||||
<p class="eyebrow"><span>11</span> EVALUATION & LIMITS</p>
|
||||
<h2>K3 很强，但报告也明确说它总体仍落后最强闭源模型</h2>
|
||||
<p>
|
||||
官方报告的总体表述很克制：K3 在其评测套件中领先被比较的其他开放与部分闭源模型，
|
||||
但整体仍落后 Claude Fable 5 与 GPT-5.6 Sol。理解这句话，比挑一个 K3 第一名的榜单更重要。
|
||||
</p>
|
||||
|
||||
<h3>评测数字至少要带四个脚注</h3>
|
||||
<ol>
|
||||
<li><strong>思考预算：</strong>K3 主结果使用 reasoning effort=max，其他模型也尽量用 max/xhigh；成本和延迟不是相同维度。</li>
|
||||
<li><strong>Harness：</strong>代码与 Agent 分数是“模型 + Codex/Kimi Code/Claude Code 等脚手架”的系统结果。</li>
|
||||
<li><strong>工具增强：</strong>HLE、视觉数学等同时报告不用工具/用工具，不能把两列混为纯模型能力。</li>
|
||||
<li><strong>Fallback / Guard：</strong>某些闭源模型出现 fallback、拒答或 cyber guard，可能显著影响特定任务分数。</li>
|
||||
</ol>
|
||||
|
||||
<p>
|
||||
例如 BrowseComp 使用 300K Token 触发上下文压缩时 K3 报告 91.2；完整 1M 窗口、不做上下文管理时是 90.4。
|
||||
这反而给出重要工程信号：更大的窗口不自动消灭 context management，适时压缩可能更有效。
|
||||
</p>
|
||||
|
||||
<div class="warning-note">
|
||||
<b>我们当前还不能知道的事</b>
|
||||
<p>
|
||||
报告没有给出足以独立复现的完整数据配比、总训练 token 数、全部集群规模与训练成本；
|
||||
新架构也缺少广泛第三方复现。网站会持续加入开放权重评测和独立复现，但不会用参数量或单榜第一替代综合判断。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="article-section" id="sources">
|
||||
<p class="eyebrow"><span>↳</span> PRIMARY SOURCES</p>
|
||||
<h2>读完本页以后，下一步回到这些一手材料</h2>
|
||||
<div class="paper-chain">
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2607.24653">
|
||||
<time>2026</time><b>Kimi K3: Open Frontier Intelligence</b><p>本页锚点；架构、预训练、后训练、系统和评测的完整一手报告。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2603.15031">
|
||||
<time>2026</time><b>Attention Residuals</b><p>跨深度选择的独立预印本、Block AttnRes 系统设计与 scaling 消融。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2510.26692">
|
||||
<time>2025</time><b>Kimi Linear</b><p>KDA 的表达能力、chunkwise 算法与 hybrid linear attention 祖先。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2405.04434">
|
||||
<time>2024</time><b>DeepSeek-V2</b><p>MLA 与 DeepSeekMoE 的系统性引入，K3 周期性全局注意力的关键来源。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2401.06066">
|
||||
<time>2024</time><b>DeepSeekMoE</b><p>shared experts 与细粒度专家特化，Stable LatentMoE 的组织祖先。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2601.18089">
|
||||
<time>2026</time><b>LatentMoE</b><p>把 full model width 与 routed expert width 分离，扩张专家数量与激活数。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2501.12599">
|
||||
<time>2025</time><b>Kimi k1.5</b><p>大规模强化学习、长 CoT 与推理时扩展的 Kimi 前代主线。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2507.20534">
|
||||
<time>2025</time><b>Kimi K2</b><p>开放 Agentic Intelligence、MoE 主干和工具使用的直接前代。</p>
|
||||
</a>
|
||||
<a class="paper-row" href="https://arxiv.org/abs/2602.02276">
|
||||
<time>2026</time><b>Kimi K2.5</b><p>视觉 Agent、并行 Agent Swarm 与 K3 多模态/Agent 后训练的前代。</p>
|
||||
</a>
|
||||
</div>
|
||||
<div class="hero-actions">
|
||||
<a class="button primary" href="https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf">打开官方 PDF</a>
|
||||
<a class="button" href="https://github.com/MoonshotAI/Kimi-K3">官方代码与权重说明</a>
|
||||
<a class="button" href="/roadmap/">回到完整学习地图</a>
|
||||
</div>
|
||||
</section>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.spec-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, minmax(0, 1fr));
|
||||
max-width: 930px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.spec-grid > div {
|
||||
min-height: 180px;
|
||||
padding: 22px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.spec-grid span {
|
||||
color: var(--muted);
|
||||
font: 0.61rem/1 var(--mono);
|
||||
letter-spacing: 0.1em;
|
||||
}
|
||||
|
||||
.spec-grid b {
|
||||
display: block;
|
||||
margin: 22px 0 14px;
|
||||
color: var(--copper);
|
||||
font: 700 1.25rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.spec-grid p {
|
||||
color: var(--muted);
|
||||
font-size: 0.74rem;
|
||||
line-height: 1.65;
|
||||
}
|
||||
|
||||
.moe-flow {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(110px, 1fr) 24px) minmax(120px, 1fr);
|
||||
gap: 10px;
|
||||
align-items: center;
|
||||
max-width: 930px;
|
||||
margin: 34px 0;
|
||||
}
|
||||
|
||||
.moe-flow > div:not(.split-label) {
|
||||
min-height: 125px;
|
||||
padding: 18px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.moe-flow .shared {
|
||||
border-color: rgba(85, 122, 114, 0.4) !important;
|
||||
background: var(--sage-pale) !important;
|
||||
}
|
||||
|
||||
.moe-flow .experts {
|
||||
border-color: rgba(173, 100, 69, 0.4) !important;
|
||||
background: var(--copper-pale) !important;
|
||||
}
|
||||
|
||||
.moe-flow span {
|
||||
color: var(--muted);
|
||||
font: 0.61rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.moe-flow b,
|
||||
.moe-flow small {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.moe-flow b {
|
||||
margin-top: 20px;
|
||||
font-size: 0.93rem;
|
||||
}
|
||||
|
||||
.moe-flow small {
|
||||
margin-top: 7px;
|
||||
color: var(--muted);
|
||||
font-size: 0.66rem;
|
||||
}
|
||||
|
||||
.moe-flow > i {
|
||||
color: var(--muted-light);
|
||||
font-style: normal;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.split-label {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.teacher-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 150px repeat(3, minmax(90px, 1fr));
|
||||
max-width: 760px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.teacher-grid > * {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
min-height: 72px;
|
||||
padding: 12px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.teacher-grid .corner,
|
||||
.teacher-grid > b {
|
||||
background: var(--paper-deep);
|
||||
color: var(--muted);
|
||||
font: 0.62rem/1.4 var(--mono);
|
||||
}
|
||||
|
||||
.teacher-grid > strong {
|
||||
justify-items: start;
|
||||
font: 0.68rem/1.3 var(--mono);
|
||||
}
|
||||
|
||||
.teacher-grid > span {
|
||||
background: var(--paper-raised);
|
||||
color: var(--copper);
|
||||
font: 700 0.8rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.environment-list,
|
||||
.system-stack {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
max-width: 930px;
|
||||
margin: 34px 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.environment-list article,
|
||||
.system-stack article {
|
||||
min-height: 190px;
|
||||
padding: 24px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.environment-list span,
|
||||
.system-stack span {
|
||||
color: var(--copper);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
letter-spacing: 0.1em;
|
||||
}
|
||||
|
||||
.environment-list h4,
|
||||
.system-stack h3 {
|
||||
margin: 24px 0 12px;
|
||||
font-size: 1.05rem;
|
||||
}
|
||||
|
||||
.environment-list p,
|
||||
.system-stack p {
|
||||
color: var(--muted);
|
||||
font-size: 0.79rem;
|
||||
line-height: 1.72;
|
||||
}
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.spec-grid,
|
||||
.environment-list,
|
||||
.system-stack {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.moe-flow {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.moe-flow > i {
|
||||
transform: rotate(90deg);
|
||||
}
|
||||
|
||||
.teacher-grid {
|
||||
grid-template-columns: 100px repeat(3, minmax(64px, 1fr));
|
||||
font-size: 0.72rem;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</BaseLayout>
|
||||
@@ -0,0 +1,636 @@
|
||||
---
|
||||
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||
import { papers, paperTopics } from "@/data/papers";
|
||||
|
||||
const sortedPapers = [...papers].sort((a, b) => b.year - a.year || a.title.localeCompare(b.title));
|
||||
const verifiedCount = papers.filter((paper) => paper.verified).length;
|
||||
const spotlightCount = papers.filter((paper) => paper.spotlight).length;
|
||||
const firstYear = Math.min(...papers.map((paper) => paper.year));
|
||||
const lastYear = Math.max(...papers.map((paper) => paper.year));
|
||||
const filters = [
|
||||
{ id: "all", label: "全部" },
|
||||
...paperTopics.map((topic) => ({ id: topic, label: topic })),
|
||||
{ id: "Kimi", label: "Kimi" },
|
||||
{ id: "DeepSeek", label: "DeepSeek" },
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="关键论文索引"
|
||||
description="LLM Atlas 持续核验的关键论文索引：按技术专题、Kimi 与 DeepSeek 主线筛选，并用一句话解释每篇论文改变了什么。"
|
||||
section="papers"
|
||||
>
|
||||
<header class="papers-hero">
|
||||
<div class="papers-hero-copy">
|
||||
<p class="eyebrow"><span>LIBRARY / 01</span> PRIMARY SOURCES</p>
|
||||
<h1>不是论文坟场，<br />而是一张演化地图</h1>
|
||||
<p class="lead">
|
||||
每一篇只回答两个问题：它解决了上一代路线的什么困难，又把什么新问题留给了后来者。
|
||||
你可以按专题筛选，也可以沿 Kimi / DeepSeek 两条聚光主线回看技术汇流。
|
||||
</p>
|
||||
</div>
|
||||
<dl class="papers-facts">
|
||||
<div><dt>INDEXED</dt><dd>{papers.length} 篇</dd></div>
|
||||
<div><dt>SPAN</dt><dd>{firstYear}—{lastYear}</dd></div>
|
||||
<div><dt>LINK CHECKED</dt><dd>{verifiedCount} 篇</dd></div>
|
||||
<div><dt>SPOTLIGHT</dt><dd>{spotlightCount} 篇</dd></div>
|
||||
</dl>
|
||||
</header>
|
||||
|
||||
<section class="library-shell" aria-labelledby="library-title">
|
||||
<div class="library-intro">
|
||||
<div>
|
||||
<p class="eyebrow"><span>01</span> CURATED INDEX</p>
|
||||
<h2 id="library-title">从问题出发，再去读论文</h2>
|
||||
</div>
|
||||
<p>
|
||||
“主链接已核验”只代表题名与权威入口经过检查，不代表我们复现了全部实验。
|
||||
深度精读、公式推导与架构图会逐章进入专题正文；这里先负责帮你找到正确入口。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="library-tools">
|
||||
<label class="paper-search">
|
||||
<span>SEARCH</span>
|
||||
<input id="paper-search" type="search" placeholder="搜索标题、贡献、年份或路线…" autocomplete="off" />
|
||||
<kbd>/</kbd>
|
||||
</label>
|
||||
<div class="paper-filters" role="group" aria-label="按技术专题筛选">
|
||||
{filters.map((filter, index) => (
|
||||
<button
|
||||
type="button"
|
||||
class="paper-filter"
|
||||
data-filter={filter.id}
|
||||
aria-pressed={index === 0 ? "true" : "false"}
|
||||
>
|
||||
{filter.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<div class="library-result" aria-live="polite">
|
||||
<span id="result-count">{papers.length}</span>
|
||||
<small>篇论文可见</small>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="paper-list" id="paper-list">
|
||||
{sortedPapers.map((paper, index) => {
|
||||
const searchable = [
|
||||
paper.year,
|
||||
paper.title,
|
||||
paper.contribution,
|
||||
...paper.topics,
|
||||
paper.spotlight ?? "",
|
||||
].join(" ").toLocaleLowerCase("zh-CN");
|
||||
const filterTokens = [...paper.topics, paper.spotlight].filter(Boolean).join("|");
|
||||
|
||||
return (
|
||||
<article
|
||||
class="paper-card"
|
||||
data-paper
|
||||
data-search={searchable}
|
||||
data-filters={filterTokens}
|
||||
>
|
||||
<div class="paper-sequence">{String(index + 1).padStart(3, "0")}</div>
|
||||
<time>{paper.year}</time>
|
||||
<div class="paper-main">
|
||||
<div class="paper-title-line">
|
||||
<h3>
|
||||
<a href={paper.url} target="_blank" rel="noreferrer">{paper.title}</a>
|
||||
</h3>
|
||||
{paper.spotlight && <span class:list={["spotlight", paper.spotlight.toLowerCase()]}>{paper.spotlight}</span>}
|
||||
</div>
|
||||
<p>{paper.contribution}</p>
|
||||
<div class="paper-tags">
|
||||
{paper.topics.map((topic) => <span>{topic}</span>)}
|
||||
{paper.verified && <span class="verified">主链接已核验</span>}
|
||||
</div>
|
||||
</div>
|
||||
<a class="paper-open" href={paper.url} target="_blank" rel="noreferrer" aria-label={`打开论文：${paper.title}`}>↗</a>
|
||||
</article>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
|
||||
<div id="paper-empty" class="paper-empty" hidden>
|
||||
<b>没有找到匹配论文</b>
|
||||
<p>试试更短的关键词，或切换回“全部”。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="reading-note">
|
||||
<p class="eyebrow"><span>02</span> HOW TO READ</p>
|
||||
<h2>别从摘要一路硬啃到附录</h2>
|
||||
<div class="reading-steps">
|
||||
<article><span>01</span><h3>先找旧瓶颈</h3><p>作者认为上一代方法哪里太慢、太贵、不稳定或不够通用？</p></article>
|
||||
<article><span>02</span><h3>再看核心替换</h3><p>是哪一个数据流、损失函数或系统边界被重新设计？先画图，再看公式。</p></article>
|
||||
<article><span>03</span><h3>盯住公平对照</h3><p>参数量、激活量、训练 token、硬件和推理预算是否真的可比？</p></article>
|
||||
<article><span>04</span><h3>最后追后继者</h3><p>真正重要的想法，会被下一篇论文复用、修正，或暴露新的代价。</p></article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<script>
|
||||
const input = document.querySelector<HTMLInputElement>("#paper-search");
|
||||
const filterButtons = [...document.querySelectorAll<HTMLButtonElement>(".paper-filter")];
|
||||
const cards = [...document.querySelectorAll<HTMLElement>("[data-paper]")];
|
||||
const count = document.querySelector<HTMLElement>("#result-count");
|
||||
const empty = document.querySelector<HTMLElement>("#paper-empty");
|
||||
let activeFilter = "all";
|
||||
|
||||
const normalize = (value: string) => value.trim().toLocaleLowerCase("zh-CN");
|
||||
|
||||
const applyFilters = () => {
|
||||
const query = normalize(input?.value ?? "");
|
||||
let visible = 0;
|
||||
|
||||
cards.forEach((card) => {
|
||||
const matchesQuery = !query || (card.dataset.search ?? "").includes(query);
|
||||
const tokens = (card.dataset.filters ?? "").split("|");
|
||||
const matchesFilter = activeFilter === "all" || tokens.includes(activeFilter);
|
||||
const show = matchesQuery && matchesFilter;
|
||||
card.hidden = !show;
|
||||
if (show) visible += 1;
|
||||
});
|
||||
|
||||
if (count) count.textContent = String(visible);
|
||||
if (empty) empty.hidden = visible !== 0;
|
||||
};
|
||||
|
||||
input?.addEventListener("input", applyFilters);
|
||||
|
||||
filterButtons.forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
activeFilter = button.dataset.filter ?? "all";
|
||||
filterButtons.forEach((item) => item.setAttribute("aria-pressed", String(item === button)));
|
||||
applyFilters();
|
||||
});
|
||||
});
|
||||
|
||||
document.addEventListener("keydown", (event) => {
|
||||
if (event.key === "/" && document.activeElement !== input) {
|
||||
event.preventDefault();
|
||||
input?.focus();
|
||||
}
|
||||
if (event.key === "Escape" && document.activeElement === input) {
|
||||
if (input) input.value = "";
|
||||
input?.blur();
|
||||
applyFilters();
|
||||
}
|
||||
});
|
||||
</script>
|
||||
</BaseLayout>
|
||||
|
||||
<style>
|
||||
.papers-hero {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1.65fr) minmax(300px, 0.65fr);
|
||||
gap: clamp(48px, 8vw, 140px);
|
||||
width: var(--page);
|
||||
margin: 0 auto;
|
||||
padding: clamp(96px, 12vw, 180px) 0 clamp(76px, 9vw, 128px);
|
||||
}
|
||||
|
||||
.papers-hero h1 {
|
||||
max-width: 1040px;
|
||||
margin-top: 26px;
|
||||
font-size: clamp(3.3rem, 6.4vw, 7.25rem);
|
||||
}
|
||||
|
||||
.papers-hero .lead {
|
||||
max-width: 770px;
|
||||
margin-top: 38px;
|
||||
color: var(--muted);
|
||||
font-family: var(--serif);
|
||||
font-size: clamp(1.05rem, 1.45vw, 1.32rem);
|
||||
line-height: 1.9;
|
||||
}
|
||||
|
||||
.papers-facts {
|
||||
align-self: end;
|
||||
border-top: 1px solid var(--line-strong);
|
||||
}
|
||||
|
||||
.papers-facts div {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
justify-content: space-between;
|
||||
gap: 24px;
|
||||
padding: 18px 0;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.papers-facts dt,
|
||||
.papers-facts dd {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.papers-facts dt {
|
||||
color: var(--muted);
|
||||
font: 0.66rem/1 var(--mono);
|
||||
letter-spacing: 0.12em;
|
||||
}
|
||||
|
||||
.papers-facts dd {
|
||||
font-family: var(--serif);
|
||||
font-size: 1.1rem;
|
||||
}
|
||||
|
||||
.library-shell {
|
||||
padding: clamp(78px, 9vw, 130px) max(36px, calc((100vw - 1380px) / 2));
|
||||
background: var(--paper-raised);
|
||||
border-block: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.library-intro {
|
||||
display: grid;
|
||||
grid-template-columns: 1.25fr 0.75fr;
|
||||
gap: 80px;
|
||||
align-items: end;
|
||||
margin-bottom: 58px;
|
||||
}
|
||||
|
||||
.library-intro h2,
|
||||
.reading-note h2 {
|
||||
margin-top: 18px;
|
||||
}
|
||||
|
||||
.library-intro > p {
|
||||
max-width: 560px;
|
||||
color: var(--muted);
|
||||
font-family: var(--serif);
|
||||
font-size: 1.03rem;
|
||||
}
|
||||
|
||||
.library-tools {
|
||||
position: sticky;
|
||||
z-index: 20;
|
||||
top: 72px;
|
||||
display: grid;
|
||||
grid-template-columns: minmax(230px, 0.55fr) minmax(0, 1.45fr) auto;
|
||||
gap: 20px;
|
||||
align-items: center;
|
||||
margin-inline: -20px;
|
||||
padding: 20px;
|
||||
border: 1px solid var(--line);
|
||||
background: color-mix(in srgb, var(--paper-raised) 94%, transparent);
|
||||
backdrop-filter: blur(18px);
|
||||
}
|
||||
|
||||
.paper-search {
|
||||
display: grid;
|
||||
grid-template-columns: auto 1fr auto;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
min-height: 44px;
|
||||
padding: 0 14px;
|
||||
border: 1px solid var(--line-strong);
|
||||
background: var(--paper);
|
||||
}
|
||||
|
||||
.paper-search span {
|
||||
color: var(--muted);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
letter-spacing: 0.12em;
|
||||
}
|
||||
|
||||
.paper-search input {
|
||||
min-width: 0;
|
||||
border: 0;
|
||||
outline: 0;
|
||||
background: transparent;
|
||||
color: var(--ink);
|
||||
font-size: 0.83rem;
|
||||
}
|
||||
|
||||
.paper-search kbd {
|
||||
padding: 2px 7px;
|
||||
border: 1px solid var(--line);
|
||||
color: var(--muted);
|
||||
font-size: 0.68rem;
|
||||
}
|
||||
|
||||
.paper-filters {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 7px;
|
||||
}
|
||||
|
||||
.paper-filter {
|
||||
padding: 7px 11px;
|
||||
border: 1px solid var(--line);
|
||||
background: transparent;
|
||||
color: var(--muted);
|
||||
cursor: pointer;
|
||||
font-size: 0.73rem;
|
||||
}
|
||||
|
||||
.paper-filter:hover,
|
||||
.paper-filter[aria-pressed="true"] {
|
||||
border-color: var(--ink);
|
||||
background: var(--ink);
|
||||
color: var(--paper-raised);
|
||||
}
|
||||
|
||||
.library-result {
|
||||
display: grid;
|
||||
min-width: 72px;
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
.library-result span {
|
||||
font: 600 1.25rem/1 var(--serif);
|
||||
}
|
||||
|
||||
.library-result small {
|
||||
margin-top: 5px;
|
||||
color: var(--muted);
|
||||
font-size: 0.62rem;
|
||||
}
|
||||
|
||||
.paper-list {
|
||||
margin-top: 28px;
|
||||
border-top: 1px solid var(--line-strong);
|
||||
}
|
||||
|
||||
.paper-card {
|
||||
display: grid;
|
||||
grid-template-columns: 48px 76px minmax(0, 1fr) 42px;
|
||||
gap: 22px;
|
||||
align-items: start;
|
||||
padding: 25px 8px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
transition: background 180ms ease, transform 180ms ease;
|
||||
}
|
||||
|
||||
.paper-card:hover {
|
||||
background: var(--paper);
|
||||
transform: translateX(4px);
|
||||
}
|
||||
|
||||
.paper-card[hidden] {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.paper-sequence {
|
||||
padding-top: 5px;
|
||||
color: var(--muted-light);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
letter-spacing: 0.08em;
|
||||
}
|
||||
|
||||
.paper-card time {
|
||||
padding-top: 1px;
|
||||
color: var(--copper);
|
||||
font: 500 0.9rem/1.3 var(--mono);
|
||||
}
|
||||
|
||||
.paper-title-line {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 11px;
|
||||
}
|
||||
|
||||
.paper-title-line h3 {
|
||||
font-family: var(--serif);
|
||||
font-size: clamp(1.02rem, 1.25vw, 1.22rem);
|
||||
font-weight: 600;
|
||||
letter-spacing: -0.012em;
|
||||
}
|
||||
|
||||
.paper-title-line a {
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.paper-title-line a:hover {
|
||||
color: var(--copper);
|
||||
}
|
||||
|
||||
.spotlight {
|
||||
flex: 0 0 auto;
|
||||
padding: 3px 7px;
|
||||
border: 1px solid var(--line);
|
||||
font: 0.58rem/1 var(--mono);
|
||||
letter-spacing: 0.06em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.spotlight.kimi {
|
||||
border-color: rgba(109, 102, 135, 0.35);
|
||||
background: var(--violet-pale);
|
||||
color: var(--violet);
|
||||
}
|
||||
|
||||
.spotlight.deepseek {
|
||||
border-color: rgba(98, 127, 153, 0.35);
|
||||
background: var(--sky-pale);
|
||||
color: var(--sky);
|
||||
}
|
||||
|
||||
.paper-main > p {
|
||||
max-width: 850px;
|
||||
margin-top: 8px;
|
||||
color: var(--muted);
|
||||
font-size: 0.86rem;
|
||||
line-height: 1.62;
|
||||
}
|
||||
|
||||
.paper-tags {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 7px;
|
||||
margin-top: 13px;
|
||||
}
|
||||
|
||||
.paper-tags span {
|
||||
padding: 3px 8px;
|
||||
border: 1px solid var(--line);
|
||||
color: var(--muted);
|
||||
font: 0.6rem/1.3 var(--mono);
|
||||
}
|
||||
|
||||
.paper-tags .verified {
|
||||
border-color: rgba(85, 122, 114, 0.28);
|
||||
background: var(--sage-pale);
|
||||
color: var(--sage);
|
||||
}
|
||||
|
||||
.paper-open {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 38px;
|
||||
aspect-ratio: 1;
|
||||
border: 1px solid var(--line);
|
||||
color: var(--muted);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.paper-open:hover {
|
||||
border-color: var(--ink);
|
||||
background: var(--ink);
|
||||
color: var(--paper-raised);
|
||||
}
|
||||
|
||||
.paper-empty {
|
||||
padding: 90px 20px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.paper-empty b {
|
||||
font-family: var(--serif);
|
||||
font-size: 1.35rem;
|
||||
}
|
||||
|
||||
.paper-empty p {
|
||||
margin-top: 8px;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.reading-note {
|
||||
width: var(--page);
|
||||
margin: 0 auto;
|
||||
padding: clamp(90px, 11vw, 160px) 0;
|
||||
}
|
||||
|
||||
.reading-steps {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
margin-top: 60px;
|
||||
border-block: 1px solid var(--line-strong);
|
||||
}
|
||||
|
||||
.reading-steps article {
|
||||
min-height: 240px;
|
||||
padding: 30px 26px;
|
||||
border-right: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.reading-steps article:first-child {
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.reading-steps span {
|
||||
color: var(--copper);
|
||||
font: 0.68rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.reading-steps h3 {
|
||||
margin-top: 54px;
|
||||
}
|
||||
|
||||
.reading-steps p {
|
||||
margin-top: 14px;
|
||||
color: var(--muted);
|
||||
font-size: 0.88rem;
|
||||
}
|
||||
|
||||
@media (max-width: 1100px) {
|
||||
.papers-hero {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.papers-facts {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, 1fr);
|
||||
max-width: 720px;
|
||||
}
|
||||
|
||||
.papers-facts div:nth-child(odd) {
|
||||
padding-right: 28px;
|
||||
border-right: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.papers-facts div:nth-child(even) {
|
||||
padding-left: 28px;
|
||||
}
|
||||
|
||||
.library-tools {
|
||||
grid-template-columns: 1fr auto;
|
||||
}
|
||||
|
||||
.paper-filters {
|
||||
grid-column: 1 / -1;
|
||||
grid-row: 2;
|
||||
}
|
||||
|
||||
.reading-steps {
|
||||
grid-template-columns: repeat(2, 1fr);
|
||||
}
|
||||
|
||||
.reading-steps article:nth-child(3) {
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.papers-hero {
|
||||
width: calc(100vw - 40px);
|
||||
padding-top: 72px;
|
||||
}
|
||||
|
||||
.papers-hero h1 {
|
||||
font-size: clamp(2.8rem, 14vw, 4.5rem);
|
||||
}
|
||||
|
||||
.papers-facts {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.papers-facts div:nth-child(n) {
|
||||
padding-inline: 0;
|
||||
border-right: 0;
|
||||
}
|
||||
|
||||
.library-shell {
|
||||
padding-inline: 20px;
|
||||
}
|
||||
|
||||
.library-intro {
|
||||
grid-template-columns: 1fr;
|
||||
gap: 28px;
|
||||
}
|
||||
|
||||
.library-tools {
|
||||
top: 62px;
|
||||
grid-template-columns: 1fr auto;
|
||||
margin-inline: 0;
|
||||
padding: 14px;
|
||||
}
|
||||
|
||||
.paper-filter {
|
||||
padding: 6px 9px;
|
||||
}
|
||||
|
||||
.paper-card {
|
||||
grid-template-columns: 54px minmax(0, 1fr) 36px;
|
||||
gap: 12px;
|
||||
padding-block: 22px;
|
||||
}
|
||||
|
||||
.paper-sequence {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.paper-title-line {
|
||||
align-items: flex-start;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.paper-open {
|
||||
width: 34px;
|
||||
}
|
||||
|
||||
.reading-note {
|
||||
width: calc(100vw - 40px);
|
||||
}
|
||||
|
||||
.reading-steps {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.reading-steps article:nth-child(n) {
|
||||
min-height: 210px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,372 @@
|
||||
---
|
||||
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||
import { chapters } from "@/data/chapters";
|
||||
|
||||
const average = Math.round(chapters.reduce((sum, chapter) => sum + chapter.progress, 0) / chapters.length);
|
||||
const published = chapters.filter((chapter) => chapter.status === "published").length;
|
||||
const researching = chapters.filter((chapter) => ["researching", "drafting"].includes(chapter.status)).length;
|
||||
|
||||
const workstreams = [
|
||||
{ label: "研究框架与规范", value: 72, next: "给 125 篇索引补充逐篇精读层级" },
|
||||
{ label: "网站设计系统", value: 86, next: "打印样式与更多通用可视化组件" },
|
||||
{ label: "Kimi K3 深读", value: 55, next: "扩写 scaling / infra 逐图笔记" },
|
||||
{ label: "Transformer 基础", value: 52, next: "加入矩阵形状动画与手算练习" },
|
||||
{ label: "DeepSeek 专题", value: 54, next: "MLA 与 GRPO 完整公式推导" },
|
||||
{ label: "引用与事实检查", value: 48, next: "自动化外链复查与来源等级扩展" },
|
||||
{ label: "开源与部署", value: 18, next: "首次提交、远端仓库与 HTTPS" },
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="持续建设进度"
|
||||
description="LLM Atlas 的研究、写作、可视化、引用核验、开源仓库与 k1412 部署实时进度。"
|
||||
section="progress"
|
||||
>
|
||||
<header class="page-hero">
|
||||
<div class="page-hero-inner">
|
||||
<div>
|
||||
<p class="eyebrow"><span>PUBLIC LEDGER</span> BUILD IN THE OPEN</p>
|
||||
<h1>庞大工作需要一份<br />公开、可检查的账本</h1>
|
||||
<p class="lead">
|
||||
这里不把“正在研究”包装成“已经完成”。每个专题显示成熟度、下一检查点和证据边界；
|
||||
本地 PROGRESS.md 与网站同步维护。
|
||||
</p>
|
||||
</div>
|
||||
<dl class="page-facts">
|
||||
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
||||
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
||||
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-28 21:55 CST</dd></div>
|
||||
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<section class="section" id="workstreams">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>01</span> WORKSTREAMS</p>
|
||||
<h2>七条工作流同时推进，但不混淆“有页面”和“已核验”</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
内容首版优先打通全局脉络；随后每轮迭代选择一个专题推进到论文/工程层，并做独立事实复核。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="stream-list">
|
||||
{workstreams.map((stream, index) => (
|
||||
<article>
|
||||
<span>{String(index + 1).padStart(2, "0")}</span>
|
||||
<div>
|
||||
<h3>{stream.label}</h3>
|
||||
<p>下一检查点：{stream.next}</p>
|
||||
</div>
|
||||
<div class="stream-progress">
|
||||
<b>{stream.value}%</b>
|
||||
<div class="progress-track"><span style={`width:${stream.value}%`}></span></div>
|
||||
</div>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="done">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>02</span> COMPLETED THIS ITERATION</p>
|
||||
<h2>第一轮已经落地什么</h2>
|
||||
</div>
|
||||
<p class="section-lead">下列项目都能在仓库或网站中直接检查，不是计划项。</p>
|
||||
</div>
|
||||
<div class="check-grid">
|
||||
<article><span>✓</span><h3>研究目标已持久化</h3><p>总体路线、完成标准、来源等级与进度账本已写入项目。</p></article>
|
||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||
<article><span>✓</span><h3>16 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||
<article><span>✓</span><h3>三张原创交互图</h3><p>K3 三轴架构、Self-Attention Query 实验与 DeepSeek 技术谱系。</p></article>
|
||||
<article><span>✓</span><h3>三篇首版长文</h3><p>K3 完整导读、Transformer 基础与 DeepSeek 论文谱系。</p></article>
|
||||
<article><span>✓</span><h3>125 篇关键论文索引</h3><p>覆盖 12 个专题，支持全文搜索、标签筛选与 Kimi/DeepSeek 聚光主线。</p></article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="queue">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>03</span> NEXT QUEUE</p>
|
||||
<h2>下一批按“能闭环的专题”推进</h2>
|
||||
</div>
|
||||
<p class="section-lead">优先级由 K3 依赖度、初学者断点和论文可验证性共同决定。</p>
|
||||
</div>
|
||||
<div class="queue-table">
|
||||
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
||||
<div><span>P0</span><strong>长上下文与高效注意力</strong><p>FlashAttention → MLA → Delta Rule → Kimi Linear/KDA</p><em>6 张图 + 20 篇论文</em></div>
|
||||
<div><span>P0</span><strong>稀疏计算与 MoE</strong><p>Switch → DeepSeekMoE → LatentMoE → Stable LatentMoE</p><em>路由模拟器 + 通信账本</em></div>
|
||||
<div><span>P1</span><strong>推理模型与测试时扩展</strong><p>CoT → verifier → GRPO → R1 → k1.5 → K3 MOPD</p><em>奖励/预算交互图</em></div>
|
||||
<div><span>P1</span><strong>大规模训练系统</strong><p>ZeRO/Megatron → Expert/Context Parallel → DualPipe/MoonEP</p><em>显存与通信计算器</em></div>
|
||||
<div><span>P2</span><strong>原生多模态</strong><p>ViT/CLIP → connector VLM → Kimi-VL/MoonViT-V2</p><em>视觉 Token 流程图</em></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="method">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>04</span> QUALITY CONTROL</p>
|
||||
<h2>每条结论怎样进入网站</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
Grok CLI 用于扩大检索覆盖和找遗漏；任何进入正文的机制、数字与时间仍回到原论文、官方仓库或正式文档。
|
||||
</p>
|
||||
</div>
|
||||
<div class="evidence-pipeline">
|
||||
<article><span>01 / DISCOVER</span><h3>发现线索</h3><p>K3 references、引用网络、Grok/web 检索、作者仓库。</p></article>
|
||||
<i>→</i>
|
||||
<article><span>02 / VERIFY</span><h3>打开一手来源</h3><p>核对标题、版本、公式、表格、实验设置和限制。</p></article>
|
||||
<i>→</i>
|
||||
<article><span>03 / EXPLAIN</span><h3>写四层解释</h3><p>直觉、机制、论文证据、工程代价；简化处显式标注。</p></article>
|
||||
<i>→</i>
|
||||
<article><span>04 / REVIEW</span><h3>交叉检查</h3><p>链接、数字、先修、图文一致性、移动端与可访问性。</p></article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section compact" id="ledger">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>05</span> DECISION LEDGER</p>
|
||||
<h2>重要决策不会只留在对话里</h2>
|
||||
</div>
|
||||
<p class="section-lead">更完整的机器可读进度和研究记录位于开源仓库的 ROADMAP.md、PROGRESS.md 与 research/。</p>
|
||||
</div>
|
||||
<div class="decision-list">
|
||||
<div><time>2026-07-28</time><b>命名为 LLM Atlas</b><p>K3 是锚点，范围覆盖完整 LLM 技术史。</p></div>
|
||||
<div><time>2026-07-28</time><b>按问题链组织</b><p>现象 → 旧瓶颈 → 关键论文 → 后继桥梁 → K3/DeepSeek 落点。</p></div>
|
||||
<div><time>2026-07-28</time><b>优先原创重绘</b><p>架构图做成可缩放 SVG/HTML，明确简化与来源。</p></div>
|
||||
<div><time>2026-07-28</time><b>双重开放许可</b><p>代码 MIT，原创文字与图 CC BY-SA 4.0。</p></div>
|
||||
<div><time>2026-07-28</time><b>自托管交付</b><p>源码公开到 git.k1412.top，网站部署到 k1412 私有基础设施。</p></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<style>
|
||||
.stream-list,
|
||||
.decision-list {
|
||||
max-width: 1050px;
|
||||
border-top: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.stream-list article {
|
||||
display: grid;
|
||||
grid-template-columns: 55px 1fr 220px;
|
||||
gap: 24px;
|
||||
align-items: center;
|
||||
min-height: 115px;
|
||||
padding: 20px 8px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.stream-list article > span {
|
||||
color: var(--copper);
|
||||
font: 0.67rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.stream-list h3 {
|
||||
font-size: 1.05rem;
|
||||
}
|
||||
|
||||
.stream-list p {
|
||||
margin-top: 8px;
|
||||
color: var(--muted);
|
||||
font-size: 0.74rem;
|
||||
}
|
||||
|
||||
.stream-progress {
|
||||
display: grid;
|
||||
grid-template-columns: 46px 1fr;
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.stream-progress b {
|
||||
color: var(--muted);
|
||||
font: 0.68rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.check-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||||
max-width: 1100px;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.check-grid article {
|
||||
min-height: 220px;
|
||||
padding: 26px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.check-grid span {
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
border-radius: 50%;
|
||||
background: var(--sage-pale);
|
||||
color: var(--sage);
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
|
||||
.check-grid h3 {
|
||||
margin: 32px 0 12px;
|
||||
font-size: 1.08rem;
|
||||
}
|
||||
|
||||
.check-grid p {
|
||||
color: var(--muted);
|
||||
font-size: 0.78rem;
|
||||
line-height: 1.72;
|
||||
}
|
||||
|
||||
.queue-table {
|
||||
display: grid;
|
||||
max-width: 1100px;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.queue-table > div {
|
||||
display: grid;
|
||||
grid-template-columns: 90px 210px 1fr 230px;
|
||||
}
|
||||
|
||||
.queue-table > div > * {
|
||||
padding: 17px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
font-size: 0.77rem;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
.queue-table .head {
|
||||
background: var(--ink);
|
||||
color: var(--paper-raised);
|
||||
}
|
||||
|
||||
.queue-table .head b {
|
||||
font: 0.61rem/1.5 var(--mono);
|
||||
letter-spacing: 0.08em;
|
||||
}
|
||||
|
||||
.queue-table span {
|
||||
color: var(--copper);
|
||||
font-family: var(--mono);
|
||||
}
|
||||
|
||||
.queue-table p {
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.queue-table em {
|
||||
color: var(--sage);
|
||||
font-style: normal;
|
||||
}
|
||||
|
||||
.evidence-pipeline {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, minmax(150px, 1fr) 24px) minmax(150px, 1fr);
|
||||
gap: 10px;
|
||||
align-items: center;
|
||||
max-width: 1100px;
|
||||
}
|
||||
|
||||
.evidence-pipeline article {
|
||||
min-height: 210px;
|
||||
padding: 24px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.evidence-pipeline article > span {
|
||||
color: var(--copper);
|
||||
font: 0.58rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.evidence-pipeline h3 {
|
||||
margin: 32px 0 12px;
|
||||
font-size: 1.05rem;
|
||||
}
|
||||
|
||||
.evidence-pipeline p {
|
||||
color: var(--muted);
|
||||
font-size: 0.75rem;
|
||||
line-height: 1.7;
|
||||
}
|
||||
|
||||
.evidence-pipeline > i {
|
||||
color: var(--muted-light);
|
||||
font-style: normal;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.decision-list > div {
|
||||
display: grid;
|
||||
grid-template-columns: 120px 230px 1fr;
|
||||
gap: 26px;
|
||||
padding: 20px 8px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.decision-list time {
|
||||
color: var(--copper);
|
||||
font: 0.66rem/1.6 var(--mono);
|
||||
}
|
||||
|
||||
.decision-list b {
|
||||
font-size: 0.88rem;
|
||||
}
|
||||
|
||||
.decision-list p {
|
||||
color: var(--muted);
|
||||
font-size: 0.78rem;
|
||||
}
|
||||
|
||||
@media (max-width: 800px) {
|
||||
.stream-list article,
|
||||
.decision-list > div {
|
||||
grid-template-columns: 45px 1fr;
|
||||
}
|
||||
|
||||
.stream-progress {
|
||||
grid-column: 2;
|
||||
}
|
||||
|
||||
.check-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.queue-table > div {
|
||||
grid-template-columns: 68px 1fr;
|
||||
}
|
||||
|
||||
.queue-table > div > p,
|
||||
.queue-table > div > em {
|
||||
grid-column: 2;
|
||||
}
|
||||
|
||||
.queue-table .head b:nth-child(n + 3) {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.evidence-pipeline {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.evidence-pipeline > i {
|
||||
transform: rotate(90deg);
|
||||
}
|
||||
|
||||
.decision-list p {
|
||||
grid-column: 2;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</BaseLayout>
|
||||
@@ -0,0 +1,468 @@
|
||||
---
|
||||
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||
import { chapters, statusLabel } from "@/data/chapters";
|
||||
|
||||
const totalPapers = chapters.reduce((sum, chapter) => sum + chapter.papers, 0);
|
||||
const averageProgress = Math.round(chapters.reduce((sum, chapter) => sum + chapter.progress, 0) / chapters.length);
|
||||
|
||||
const stages = [
|
||||
{ label: "A · 起点", items: ["00", "01"] },
|
||||
{ label: "B · 核心架构", items: ["02", "03", "04", "05"] },
|
||||
{ label: "C · 规模化", items: ["06", "07", "08", "09"] },
|
||||
{ label: "D · 行为与能力", items: ["10", "11", "12", "13"] },
|
||||
{ label: "E · 落地与判断", items: ["14", "15"] },
|
||||
];
|
||||
---
|
||||
|
||||
<BaseLayout
|
||||
title="完整学习地图"
|
||||
description="LLM Atlas 的 16 个专题、先修依赖、关键问题、论文规模、完成状态与四条推荐学习路径。"
|
||||
section="roadmap"
|
||||
>
|
||||
<header class="page-hero">
|
||||
<div class="page-hero-inner">
|
||||
<div>
|
||||
<p class="eyebrow"><span>CURRICULUM / 00</span> LEARNING GRAPH</p>
|
||||
<h1>先知道自己在哪里，<br />再决定往哪里深入</h1>
|
||||
<p class="lead">
|
||||
16 个专题不是一条必须顺序走完的直线。它们组成有依赖的图：
|
||||
Transformer 是共同地基，MoE/长上下文/训练系统相互牵引，后训练再通向推理与 Agent。
|
||||
</p>
|
||||
</div>
|
||||
<dl class="page-facts">
|
||||
<div><dt>MODULES</dt><dd>16 个专题</dd></div>
|
||||
<div><dt>PAPER SLOTS</dt><dd>{totalPapers} 篇核心论文位</dd></div>
|
||||
<div><dt>PATHS</dt><dd>4 条推荐路径</dd></div>
|
||||
<div><dt>PROGRESS</dt><dd>首版平均 {averageProgress}%</dd></div>
|
||||
<div><dt>DEPTH</dt><dd>L0 直觉 → L3 工程</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<section class="section" id="graph">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>01</span> DEPENDENCY GRAPH</p>
|
||||
<h2>从左到右，是知识依赖，不是历史年份</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
一篇论文可能跨多个专题。例如 DeepSeek-V3 同时属于 MoE、训练系统、低精度和后训练；
|
||||
K3 则是几乎全部主线的汇流点。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="learning-graph" role="img" aria-label="LLM Atlas 专题依赖图">
|
||||
{stages.map((stage, stageIndex) => (
|
||||
<div class="graph-stage">
|
||||
<span>{stage.label}</span>
|
||||
<div>
|
||||
{stage.items.map((number) => {
|
||||
const chapter = chapters.find((item) => item.number === number)!;
|
||||
return (
|
||||
<a href={`#chapter-${number}`} class:list={{ hot: ["02", "06", "07", "11"].includes(number) }}>
|
||||
<b>{number}</b>
|
||||
<p>{chapter.title}</p>
|
||||
</a>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
{stageIndex < stages.length - 1 && <i aria-hidden="true">→</i>}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div class="graph-legend">
|
||||
<div><i class="base"></i><span>普通节点：专题章节</span></div>
|
||||
<div><i class="hot"></i><span>关键枢纽：被多条路径依赖</span></div>
|
||||
<div><b>→</b><span>建议先修方向</span></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="paths">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>02</span> FOUR ROUTES</p>
|
||||
<h2>四条路径，四种不同的“读懂”</h2>
|
||||
</div>
|
||||
<p class="section-lead">路线可以交叉。每条路线的终点不是记住术语，而是具备一项可验证的阅读或设计能力。</p>
|
||||
</div>
|
||||
|
||||
<div class="route-grid">
|
||||
<article>
|
||||
<span>ROUTE A · BEGINNER</span>
|
||||
<h3>从零建立架构直觉</h3>
|
||||
<p>01 → 02 → 03 → 04 → 10</p>
|
||||
<ol>
|
||||
<li>解释 next-token objective</li>
|
||||
<li>手算一次 self-attention</li>
|
||||
<li>画出 decoder block</li>
|
||||
<li>区分 pretrain 与 post-train</li>
|
||||
</ol>
|
||||
</article>
|
||||
<article>
|
||||
<span>ROUTE B · K3 REVERSE</span>
|
||||
<h3>从 K3 反向拆组件</h3>
|
||||
<p>K3 → 07 → 06 → 03 → 11 → 08/14</p>
|
||||
<ol>
|
||||
<li>解释三维信息流</li>
|
||||
<li>比较 KDA 与 MLA</li>
|
||||
<li>解释 2.8T / 104B</li>
|
||||
<li>读懂 1M Agentic RL 系统</li>
|
||||
</ol>
|
||||
</article>
|
||||
<article>
|
||||
<span>ROUTE C · DEEPSEEK</span>
|
||||
<h3>沿论文看算法—系统协同</h3>
|
||||
<p>04 → 06 → 07 → 09 → 11 → 12</p>
|
||||
<ol>
|
||||
<li>推导 MLA 缓存压缩</li>
|
||||
<li>理解 FP8 scaling</li>
|
||||
<li>区分 PPO 与 GRPO</li>
|
||||
<li>对照 V4 与 K3 长上下文</li>
|
||||
</ol>
|
||||
</article>
|
||||
<article>
|
||||
<span>ROUTE D · SYSTEMS</span>
|
||||
<h3>跟一个 Token 穿过真实集群</h3>
|
||||
<p>02 → 04 → 08 → 09 → 14 → 15</p>
|
||||
<ol>
|
||||
<li>做显存与 FLOPs 账本</li>
|
||||
<li>选择并行切分</li>
|
||||
<li>理解 KV Cache 与调度</li>
|
||||
<li>设计评测 harness</li>
|
||||
</ol>
|
||||
</article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section" id="modules">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>03</span> ALL MODULES</p>
|
||||
<h2>每个专题都有问题、先修、论文和完成标准</h2>
|
||||
</div>
|
||||
<p class="section-lead">
|
||||
“研究中”表示资料已建立但正文未完成；“首版可读”表示已有连贯解释，不代表内容已经停止迭代。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="module-list">
|
||||
{chapters.map((chapter) => (
|
||||
<article id={`chapter-${chapter.number}`} class="module-row">
|
||||
<div class="module-index">
|
||||
<b>{chapter.number}</b>
|
||||
<span>{chapter.kicker}</span>
|
||||
</div>
|
||||
<div class="module-main">
|
||||
<div class="module-heading">
|
||||
<h3>{chapter.title}</h3>
|
||||
<span class={`status ${chapter.status}`}>{statusLabel[chapter.status]}</span>
|
||||
</div>
|
||||
<p class="question">{chapter.question}</p>
|
||||
<p>{chapter.summary}</p>
|
||||
<div class="module-tags">
|
||||
{chapter.highlights.map((item) => <span>{item}</span>)}
|
||||
</div>
|
||||
</div>
|
||||
<aside>
|
||||
<div><span>PREREQUISITES</span><b>{chapter.prerequisites.length ? chapter.prerequisites.join(" · ") : "无"}</b></div>
|
||||
<div><span>CORE PAPERS</span><b>{chapter.papers}</b></div>
|
||||
<div><span>PROGRESS</span><b>{chapter.progress}%</b></div>
|
||||
<div class="progress-track"><span style={`width:${chapter.progress}%`}></span></div>
|
||||
</aside>
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="section compact" id="standard">
|
||||
<div class="section-heading">
|
||||
<div>
|
||||
<p class="eyebrow"><span>04</span> DEFINITION OF DONE</p>
|
||||
<h2>什么时候一个专题才算真正完成</h2>
|
||||
</div>
|
||||
<p class="section-lead">完成不是“字数够长”。每个专题必须通过结构、事实、来源、教学、对照和可访问性六道闸门。</p>
|
||||
</div>
|
||||
<div class="feature-grid">
|
||||
<article class="feature-card"><span class="card-number">01</span><h3>问题链</h3><p>旧方法为何失败、新方法改变什么、怎样通向下一篇论文。</p></article>
|
||||
<article class="feature-card"><span class="card-number">02</span><h3>一手证据</h3><p>8–20 篇核心论文，数字与结论链接到原始来源。</p></article>
|
||||
<article class="feature-card"><span class="card-number">03</span><h3>视觉推导</h3><p>至少两张可缩放图和一个交互实验或逐步动画。</p></article>
|
||||
<article class="feature-card"><span class="card-number">04</span><h3>贯穿对照</h3><p>明确回答这项技术在 K3 与 DeepSeek 中怎样出现。</p></article>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<style>
|
||||
.learning-graph {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(5, minmax(0, 1fr));
|
||||
max-width: 1200px;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.graph-stage {
|
||||
position: relative;
|
||||
min-height: 440px;
|
||||
padding: 22px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.graph-stage > span {
|
||||
color: var(--muted);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
letter-spacing: 0.1em;
|
||||
}
|
||||
|
||||
.graph-stage > div {
|
||||
display: grid;
|
||||
gap: 12px;
|
||||
margin-top: 27px;
|
||||
}
|
||||
|
||||
.graph-stage a {
|
||||
min-height: 74px;
|
||||
padding: 13px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--paper-raised);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.graph-stage a.hot {
|
||||
border-color: rgba(173, 100, 69, 0.5);
|
||||
background: var(--copper-pale);
|
||||
}
|
||||
|
||||
.graph-stage a b {
|
||||
color: var(--copper);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.graph-stage a p {
|
||||
margin-top: 9px;
|
||||
font-size: 0.75rem;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.graph-stage > i {
|
||||
position: absolute;
|
||||
z-index: 2;
|
||||
top: 50%;
|
||||
right: -12px;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 50%;
|
||||
background: var(--paper);
|
||||
color: var(--muted);
|
||||
font-style: normal;
|
||||
}
|
||||
|
||||
.graph-legend {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 24px;
|
||||
margin-top: 18px;
|
||||
}
|
||||
|
||||
.graph-legend div {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
color: var(--muted);
|
||||
font-size: 0.7rem;
|
||||
}
|
||||
|
||||
.graph-legend i {
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
border: 1px solid var(--line-strong);
|
||||
background: var(--paper-raised);
|
||||
}
|
||||
|
||||
.graph-legend i.hot {
|
||||
border-color: var(--copper);
|
||||
background: var(--copper-pale);
|
||||
}
|
||||
|
||||
.route-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
max-width: 1100px;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.route-grid article {
|
||||
min-height: 330px;
|
||||
padding: 30px;
|
||||
border-right: 1px solid var(--line);
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.route-grid article > span,
|
||||
.module-index span,
|
||||
.module-row aside span {
|
||||
color: var(--copper);
|
||||
font: 0.62rem/1 var(--mono);
|
||||
letter-spacing: 0.1em;
|
||||
}
|
||||
|
||||
.route-grid h3 {
|
||||
margin: 30px 0 15px;
|
||||
font-size: 1.4rem;
|
||||
}
|
||||
|
||||
.route-grid article > p {
|
||||
color: var(--muted);
|
||||
font: 0.7rem/1.6 var(--mono);
|
||||
}
|
||||
|
||||
.route-grid ol {
|
||||
display: grid;
|
||||
gap: 10px;
|
||||
margin: 25px 0 0;
|
||||
padding-left: 1.2rem;
|
||||
color: var(--muted);
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.module-list {
|
||||
max-width: 1150px;
|
||||
border-top: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.module-row {
|
||||
display: grid;
|
||||
grid-template-columns: 120px minmax(0, 1fr) 180px;
|
||||
gap: 32px;
|
||||
padding: 34px 8px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
scroll-margin-top: 96px;
|
||||
}
|
||||
|
||||
.module-index b {
|
||||
display: block;
|
||||
margin-bottom: 18px;
|
||||
color: var(--copper);
|
||||
font: 700 1.1rem/1 var(--mono);
|
||||
}
|
||||
|
||||
.module-index span {
|
||||
color: var(--muted-light);
|
||||
font-size: 0.55rem;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.module-heading {
|
||||
display: flex;
|
||||
align-items: start;
|
||||
justify-content: space-between;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
.module-heading h3 {
|
||||
font-size: 1.4rem;
|
||||
}
|
||||
|
||||
.module-main > p {
|
||||
max-width: 700px;
|
||||
margin-top: 12px;
|
||||
color: var(--muted);
|
||||
font-size: 0.8rem;
|
||||
line-height: 1.72;
|
||||
}
|
||||
|
||||
.module-main > p.question {
|
||||
color: var(--ink);
|
||||
font-family: var(--serif);
|
||||
font-size: 1rem;
|
||||
}
|
||||
|
||||
.module-tags {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 7px;
|
||||
margin-top: 18px;
|
||||
}
|
||||
|
||||
.module-tags span {
|
||||
padding: 5px 8px;
|
||||
border: 1px solid var(--line);
|
||||
color: var(--muted);
|
||||
font-size: 0.64rem;
|
||||
}
|
||||
|
||||
.module-row aside {
|
||||
display: grid;
|
||||
gap: 14px;
|
||||
align-content: start;
|
||||
padding-left: 22px;
|
||||
border-left: 1px solid var(--line);
|
||||
}
|
||||
|
||||
.module-row aside div:not(.progress-track) {
|
||||
display: grid;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.module-row aside span {
|
||||
color: var(--muted-light);
|
||||
font-size: 0.54rem;
|
||||
}
|
||||
|
||||
.module-row aside b {
|
||||
font-size: 0.76rem;
|
||||
font-weight: 550;
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.learning-graph {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.graph-stage {
|
||||
min-height: auto;
|
||||
}
|
||||
|
||||
.graph-stage > div {
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
}
|
||||
|
||||
.graph-stage > i {
|
||||
top: auto;
|
||||
right: 50%;
|
||||
bottom: -12px;
|
||||
transform: rotate(90deg);
|
||||
}
|
||||
|
||||
.module-row {
|
||||
grid-template-columns: 70px 1fr;
|
||||
}
|
||||
|
||||
.module-row aside {
|
||||
grid-column: 2;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
padding: 20px 0 0;
|
||||
border-top: 1px solid var(--line);
|
||||
border-left: 0;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 650px) {
|
||||
.route-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.module-row {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.module-row aside {
|
||||
grid-column: auto;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</BaseLayout>
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"extends": "astro/tsconfigs/strict",
|
||||
"compilerOptions": {
|
||||
"baseUrl": ".",
|
||||
"paths": {
|
||||
"@/*": ["src/*"]
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user