feat: add inference serving chapter
This commit is contained in:
wuyang
+16
-3
@@ -21,6 +21,7 @@
|
|||||||
| 稀疏计算与 MoE | 完成首版 | 74% | 真实负载 traces 与专家特化案例 |
|
| 稀疏计算与 MoE | 完成首版 | 74% | 真实负载 traces 与专家特化案例 |
|
||||||
| 长上下文专题 | 完成首版 | 72% | 真实模型配置、内核细节与失败案例 |
|
| 长上下文专题 | 完成首版 | 72% | 真实模型配置、内核细节与失败案例 |
|
||||||
| 大规模训练系统 | 完成首版 | 71% | 真实集群 traces、故障案例与精确 topology 配置 |
|
| 大规模训练系统 | 完成首版 | 71% | 真实集群 traces、故障案例与精确 topology 配置 |
|
||||||
|
| 推理服务与低成本部署 | 完成首版 | 78% | 真实 GPU kernel / workload traces、功耗与跨框架复现 |
|
||||||
| 数值精度、优化器与稳定性 | 完成首版 | 75% | 真实 kernel 吞吐、长程训练 traces 与逐图论文精读 |
|
| 数值精度、优化器与稳定性 | 完成首版 | 75% | 真实 kernel 吞吐、长程训练 traces 与逐图论文精读 |
|
||||||
| 引用与事实检查 | 进行中 | 57% | 自动化外链复查与来源等级扩展 |
|
| 引用与事实检查 | 进行中 | 57% | 自动化外链复查与来源等级扩展 |
|
||||||
| 开源仓库 | 已完成首版 | 100% | 持续提交研究与网站迭代 |
|
| 开源仓库 | 已完成首版 | 100% | 持续提交研究与网站迭代 |
|
||||||
@@ -35,10 +36,10 @@
|
|||||||
- [x] 提炼参考网站的编辑设计语言。
|
- [x] 提炼参考网站的编辑设计语言。
|
||||||
- [x] 确认 `git.k1412.top` 为 Gitea/Forgejo 兼容服务且本机 HTTPS 凭据可用于既有仓库。
|
- [x] 确认 `git.k1412.top` 为 Gitea/Forgejo 兼容服务且本机 HTTPS 凭据可用于既有仓库。
|
||||||
- [x] 使用 Grok CLI 检索并形成约 95 篇一手论文的补充路线，主代理已回查关键来源。
|
- [x] 使用 Grok CLI 检索并形成约 95 篇一手论文的补充路线，主代理已回查关键来源。
|
||||||
- [x] 完成 355 篇关键论文索引，覆盖 14 个标签专题与 Kimi/DeepSeek 聚光主线。
|
- [x] 完成 400 篇关键论文索引，覆盖 15 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||||
- [x] 完成 K3、语言模型前史、Transformer 基础、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统与数值优化十四篇首版长文。
|
- [x] 完成 K3、语言模型前史、Transformer 基础、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务与数值优化十五篇首版长文。
|
||||||
- [x] 完成 K3 三轴架构、语言模型前史四联实验、Transformer 四联实验、DeepSeek 谱系、长上下文、MoE 路由、推理三页签，以及训练系统、Scaling、数据工程、数值、Alignment、Agent 与原生多模态专题各四页签等四十三个原创交互视图。
|
- [x] 完成 K3 三轴架构、语言模型前史四联实验、Transformer 四联实验、DeepSeek 谱系、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent 与原生多模态专题各四页签等四十七个原创交互视图。
|
||||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||||
@@ -117,9 +118,16 @@
|
|||||||
- [x] 原生多模态真实 Chrome 断言通过：视觉 Token 超预算、K3 五维原生训练、OCR 报告 / 插值 / 证据外边界、vision-in-the-loop、键盘 tabs 与 390px 移动端均正确响应。
|
- [x] 原生多模态真实 Chrome 断言通过：视觉 Token 超预算、K3 五维原生训练、OCR 报告 / 插值 / 证据外边界、vision-in-the-loop、键盘 tabs 与 390px 移动端均正确响应。
|
||||||
- [x] Astro 类型检查、生产构建、18 个页面、875 个站内引用和 17 个跨页锚点通过；多模态页面桌面 / 移动端无文档级横向溢出。
|
- [x] Astro 类型检查、生产构建、18 个页面、875 个站内引用和 17 个跨页锚点通过；多模态页面桌面 / 移动端无文档级横向溢出。
|
||||||
- [x] 原生多模态首版以源提交 `9f56732`、不可变镜像 `20260728T235248Z-9f56732` 发布；NAS、VPS/Tailscale、NPM、DNS、HTTPS、证书、门户、公开 Forgejo 与十一套生产 Chrome 回归全链路通过。
|
- [x] 原生多模态首版以源提交 `9f56732`、不可变镜像 `20260728T235248Z-9f56732` 发布；NAS、VPS/Tailscale、NPM、DNS、HTTPS、证书、门户、公开 Forgejo 与十一套生产 Chrome 回归全链路通过。
|
||||||
|
- [x] 启动推理服务与低成本部署专题：用十八本账拆开请求形状、SLO、权重、增长状态、内存分配、阶段、缓存、Kernel、推测、并行、网络、路由、故障与经济性。
|
||||||
|
- [x] 使用 Grok Headless 扩展 70 个候选节点；正式账本回查 26 份完整 PDF/TXT、官方会议页与 DeepSeek/Kimi 报告，候选与证据永久分离。
|
||||||
|
- [x] 完成推理服务首版：31 个正文目录、62 个一手节点、DeepSeek V2→V4 与 Mooncake→K3 双谱系，以及显存—阶段—推测—集群四联实验。
|
||||||
|
- [x] 论文库新增「推理服务」标签与 vLLM、SGLang、DistServe、Sarathi、FlashInfer、EAGLE-3、DeepGEMM、FlashMLA 等 45 个节点，从 355 篇扩充至 400 篇。
|
||||||
|
- [x] 推理服务真实 Chrome 断言通过：MHA OOM、chunked prefill 降低 stall、慢网络反噬 P/D、低验收率负加速、缓存故障重算、平均准入伤害短请求、键盘 tabs 与 390px 移动端均正确响应。
|
||||||
|
- [x] Astro 类型检查、生产构建、19 个页面、962 个站内引用和 16 个跨页锚点通过；推理服务页面桌面 / 移动端无文档级横向溢出。
|
||||||
|
|
||||||
## 正在进行
|
## 正在进行
|
||||||
|
|
||||||
|
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||||
- [ ] 原生多模态二轮：真实视觉 Token traces、跨分辨率 / connector 消融、OCR 与视觉 Agent 安全失败案例。
|
- [ ] 原生多模态二轮：真实视觉 Token traces、跨分辨率 / connector 消融、OCR 与视觉 Agent 安全失败案例。
|
||||||
- [ ] Agent 二轮：真实环境 traces、cross-harness ablation、Agent RL 训练曲线与提示注入案例。
|
- [ ] Agent 二轮：真实环境 traces、cross-harness ablation、Agent RL 训练曲线与提示注入案例。
|
||||||
- [ ] Transformer 二轮：多头电路逐图、Pre/Post-LN 真实 traces、Flash/KV kernel 与模型配置对照。
|
- [ ] Transformer 二轮：多头电路逐图、Pre/Post-LN 真实 traces、Flash/KV kernel 与模型配置对照。
|
||||||
@@ -203,6 +211,11 @@
|
|||||||
| 2026-07-29 | 光学压缩实验显式分三级证据 | DeepSeek-OCR 报告锚点、中间教学插值与超过已核范围的“不外推”在交互中使用不同状态 |
|
| 2026-07-29 | 光学压缩实验显式分三级证据 | DeepSeek-OCR 报告锚点、中间教学插值与超过已核范围的“不外推”在交互中使用不同状态 |
|
||||||
| 2026-07-29 | 论文库扩充到 355 篇 | 新增 41 个视觉表示、连接器、分辨率、OCR、视频、统一生成、评测与视觉 RL 节点 |
|
| 2026-07-29 | 论文库扩充到 355 篇 | 新增 41 个视觉表示、连接器、分辨率、OCR、视频、统一生成、评测与视觉 RL 节点 |
|
||||||
| 2026-07-29 | 原生多模态首版用不可变镜像 `20260728T235248Z-9f56732` 发布 | OCI digest `sha256:b9c66e4e…07955`；复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo |
|
| 2026-07-29 | 原生多模态首版用不可变镜像 `20260728T235248Z-9f56732` 发布 | OCI digest `sha256:b9c66e4e…07955`；复用 `12010→8080`、NPM host 31 / cert 41、门户 order 180 与公开 Forgejo |
|
||||||
|
| 2026-07-29 | 推理服务拆成十八本彼此独立的账 | 权重、增长状态、分配、阶段、batch、cache、kernel、推测、量化、网络、路由、故障与经济性不再压成一个 tokens/s |
|
||||||
|
| 2026-07-29 | Grok 推理服务召回与正式证据永久分离 | 70 个候选节点只负责查漏；26 份完整 PDF/TXT、会议页、官方仓库和模型报告承载正文事实 |
|
||||||
|
| 2026-07-29 | DeepSeek 与 Kimi 服务谱系按状态对象重建 | MLA→V4 异构状态和 Mooncake→KDA→K3 混合缓存分开讲，精确 KV 公式不套到异构状态上 |
|
||||||
|
| 2026-07-29 | 论文库扩充到 400 篇 | 新增 45 个内存管理、调度、缓存、P/D 解耦、量化、推测解码、kernel 与真实 workload 节点 |
|
||||||
|
| 2026-07-29 | 推理服务首版用四个独立实验闭环 | 显存、阶段干扰、推测验收和集群状态分开建模；作者报告、教学估算与 benchmark 永久分级 |
|
||||||
|
|
||||||
## 未决问题
|
## 未决问题
|
||||||
|
|
||||||
|
|||||||
@@ -17,9 +17,9 @@
|
|||||||
- 持续进度：[PROGRESS.md](./PROGRESS.md)
|
- 持续进度：[PROGRESS.md](./PROGRESS.md)
|
||||||
- 证据与写作规范：[research/METHODOLOGY.md](./research/METHODOLOGY.md)
|
- 证据与写作规范：[research/METHODOLOGY.md](./research/METHODOLOGY.md)
|
||||||
|
|
||||||
当前里程碑包含 16 专题学习地图、355 篇关键论文索引、Kimi K3 完整导读，
|
当前里程碑包含 16 专题学习地图、400 篇关键论文索引、Kimi K3 完整导读，
|
||||||
语言模型前史、Transformer 基础、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、训练系统与数值优化深度专题，
|
语言模型前史、Transformer 基础、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务与数值优化深度专题，
|
||||||
以及 43 个覆盖核心机制的原创交互视图。
|
以及 47 个覆盖核心机制的原创交互视图。
|
||||||
其余专题按进度账本持续扩建。
|
其余专题按进度账本持续扩建。
|
||||||
|
|
||||||
## 本地开发
|
## 本地开发
|
||||||
|
|||||||
@@ -120,6 +120,14 @@ native、光学上下文压缩与 vision-in-the-loop 四个独立实验，所有
|
|||||||
|
|
||||||
KV Cache、PagedAttention/vLLM、连续批处理、推测解码、Prefill/Decode 解耦、前缀缓存、集群调度；Mooncake 与 K3 KDA-aware serving。
|
KV Cache、PagedAttention/vLLM、连续批处理、推测解码、Prefill/Decode 解耦、前缀缓存、集群调度；Mooncake 与 K3 KDA-aware serving。
|
||||||
|
|
||||||
|
首版已完成：以请求形状、SLO、权重、增长状态、分配、Prefill、Decode、Batch、缓存、Kernel、
|
||||||
|
推测、量化、并行、MoE、网络、路由、故障与经济性十八本账，串起 2019–2026 的 62 个一手节点。
|
||||||
|
正文明确区分吞吐与 SLO goodput、标准 KV 公式与 MLA/KDA 状态、PagedAttention 的内存收益与
|
||||||
|
Transformer FLOPs、作者报告值与教学估算；重点追踪 DeepSeek-V2 MLA → V3 FP8 / MoE 部署 →
|
||||||
|
V3.2 → V4 异构状态，以及 Mooncake → Kimi Linear → K3 的 69 KDA + 24 Gated MLA、page cache、
|
||||||
|
EAGLE-3 draft、cache-aware affinity 与 token-budget admission。配套显存与 KV、Prefill/Decode、
|
||||||
|
推测解码、缓存与集群四个独立实验。
|
||||||
|
|
||||||
### 15. 评测、安全与“到底强不强”
|
### 15. 评测、安全与“到底强不强”
|
||||||
|
|
||||||
困惑度到 MMLU/GPQA/HLE，SWE-bench、OSWorld、BrowseComp；污染、harness、工具预算、LLM-as-a-judge、选择性报告与网络安全边界。
|
困惑度到 MMLU/GPQA/HLE，SWE-bench、OSWorld、BrowseComp；污染、harness、工具预算、LLM-as-a-judge、选择性报告与网络安全边界。
|
||||||
|
|||||||
+2
-1
@@ -20,7 +20,8 @@
|
|||||||
"check:transformer-browser": "node scripts/check-transformer-browser.mjs",
|
"check:transformer-browser": "node scripts/check-transformer-browser.mjs",
|
||||||
"check:alignment-browser": "node scripts/check-alignment-browser.mjs",
|
"check:alignment-browser": "node scripts/check-alignment-browser.mjs",
|
||||||
"check:agents-browser": "node scripts/check-agents-browser.mjs",
|
"check:agents-browser": "node scripts/check-agents-browser.mjs",
|
||||||
"check:multimodal-browser": "node scripts/check-multimodal-browser.mjs"
|
"check:multimodal-browser": "node scripts/check-multimodal-browser.mjs",
|
||||||
|
"check:inference-serving-browser": "node scripts/check-inference-serving-browser.mjs"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@astrojs/sitemap": "3.7.3",
|
"@astrojs/sitemap": "3.7.3",
|
||||||
|
|||||||
@@ -0,0 +1,217 @@
|
|||||||
|
# 推理服务与低成本部署：Grok 候选召回账
|
||||||
|
|
||||||
|
> 状态：`UNVERIFIED CANDIDATE LEADS`
|
||||||
|
>
|
||||||
|
> 生成方式：2026-07-29 使用本机 Grok CLI Headless 做候选召回。
|
||||||
|
>
|
||||||
|
> 使用边界：本文件只负责查漏，不承载正文事实、公式或数字。标题、年份、URL、归属和性能数字
|
||||||
|
> 必须由主代理回到论文、会议正式页、作者技术报告或官方仓库逐项核验。无法定位的一律淘汰，
|
||||||
|
> 不用“可能存在”的占位节点填满时间线。
|
||||||
|
|
||||||
|
## 1. Grok 收到的检索合同
|
||||||
|
|
||||||
|
围绕 2019–2026 的 LLM inference / serving，候选必须尽量覆盖：
|
||||||
|
|
||||||
|
1. KV Cache 容量公式、分页、碎片、共享与迁移；
|
||||||
|
2. request / iteration / continuous batching；
|
||||||
|
3. prefill 与 decode 的算力—带宽差异；
|
||||||
|
4. chunked prefill 与 tail latency；
|
||||||
|
5. FlashAttention、FlashDecoding、FlashInfer 等内核；
|
||||||
|
6. prefill–decode disaggregation；
|
||||||
|
7. prefix / radix cache、RAG 非前缀复用与外部缓存池；
|
||||||
|
8. speculative sampling、树验证、Medusa、EAGLE 与 MTP；
|
||||||
|
9. 权重、激活、KV Cache 量化与真实内核收益；
|
||||||
|
10. tensor / pipeline / expert parallel 与 MoE serving；
|
||||||
|
11. 调度、SLO、goodput、尾延迟、公平和过载；
|
||||||
|
12. DeepSeek MLA / DSA / FP4 与服务部署；
|
||||||
|
13. Mooncake、Kimi K2/K2.5/K3、KDA-aware cache；
|
||||||
|
14. 性能模拟、负载轨迹、能耗与成本口径。
|
||||||
|
|
||||||
|
## 2. 十八张候选问题账
|
||||||
|
|
||||||
|
| # | 账本 | 核心问题 |
|
||||||
|
|---|---|---|
|
||||||
|
| Q1 | 指标 | TTFT、TPOT、E2E latency、throughput、goodput 各测什么？ |
|
||||||
|
| Q2 | 权重 | 模型权重、量化 scale 与运行时 workspace 占多少显存？ |
|
||||||
|
| Q3 | KV | 每 token、每层、每请求的 KV Cache 怎样增长？ |
|
||||||
|
| Q4 | 碎片 | 预留容量与真实 token 为什么差这么多？ |
|
||||||
|
| Q5 | 阶段 | prefill 与 decode 为什么像两种不同工作负载？ |
|
||||||
|
| Q6 | 批处理 | static、iteration-level、continuous batching 的调度粒度有何不同？ |
|
||||||
|
| Q7 | 干扰 | 长 prefill 为什么会让正在 decode 的请求“卡一下”？ |
|
||||||
|
| Q8 | 内核 | FLOPs 少为什么仍可能慢？HBM、SRAM、融合与 launch 如何记账？ |
|
||||||
|
| Q9 | 前缀 | 哪些 KV 可以合法复用？位置、掩码和前序上下文怎样限制复用？ |
|
||||||
|
| Q10 | 分离 | PD 分离省下什么，又新增哪些 KV 运输与排队成本？ |
|
||||||
|
| Q11 | 推测 | 接受率、draft 成本与验证并行度怎样决定净加速？ |
|
||||||
|
| Q12 | 量化 | W/A/KV 分别量化到几位，速度与质量是否真的同时改善？ |
|
||||||
|
| Q13 | 并行 | TP、PP、DP、EP 应怎样映射到 prefill 与 decode？ |
|
||||||
|
| Q14 | MoE | 激活参数少，为何权重带宽、all-to-all 与专家倾斜仍昂贵？ |
|
||||||
|
| Q15 | 调度 | 平均 tokens/s、P99、fairness 与 SLO 为什么会互相冲突？ |
|
||||||
|
| Q16 | 集群 | 请求迁移、缓存亲和、故障切换和扩缩容怎样共同决策？ |
|
||||||
|
| Q17 | 成本 | `$ / 1M input tokens` 与 `$ / 1M output tokens` 为什么不能混算？ |
|
||||||
|
| Q18 | 谱系 | DeepSeek 与 Kimi 在模型结构、内核、缓存和 fleet 各改了哪一层？ |
|
||||||
|
|
||||||
|
## 3. Grok 候选节点
|
||||||
|
|
||||||
|
### 3.1 KV、分页与压缩
|
||||||
|
|
||||||
|
- PagedAttention / vLLM — `2309.06180`
|
||||||
|
- H2O — `2306.14048`
|
||||||
|
- StreamingLLM — `2309.17453`
|
||||||
|
- KVQuant — `2401.18079`
|
||||||
|
- KIVI — `2402.02750`
|
||||||
|
- SnapKV — `2404.14469`
|
||||||
|
- PyramidKV — `2406.02069`
|
||||||
|
- Quest — `2406.10774`
|
||||||
|
- InfiniGen — `2406.19707`
|
||||||
|
|
||||||
|
### 3.2 批处理与调度
|
||||||
|
|
||||||
|
- Clockwork — USENIX OSDI 2020
|
||||||
|
- Orca — USENIX OSDI 2022
|
||||||
|
- AlpaServe — `2302.11665`
|
||||||
|
- FastServe — `2305.05920`
|
||||||
|
- vLLM — `2309.06180`
|
||||||
|
- Fairness in Serving Large Language Models — `2401.00588`
|
||||||
|
- Sarathi-Serve — `2403.02310`
|
||||||
|
- Llumnix — `2406.03243`
|
||||||
|
- Preble — `2407.00023`
|
||||||
|
|
||||||
|
### 3.3 注意力与解码内核
|
||||||
|
|
||||||
|
- FlashAttention — `2205.14135`
|
||||||
|
- FlashAttention-2 — `2307.08691`
|
||||||
|
- FlashDecoding — Stanford CRFM 官方技术说明
|
||||||
|
- FlashDecoding++ — `2311.01282`
|
||||||
|
- FlashAttention-3 — `2407.08608`
|
||||||
|
- FlashInfer — `2501.01005`
|
||||||
|
- DeepSeek FlashMLA — 官方仓库
|
||||||
|
- DeepSeek DeepGEMM — 官方仓库
|
||||||
|
- Moonshot FlashKDA — 官方仓库
|
||||||
|
|
||||||
|
### 3.4 PD 分离与 KV 运输
|
||||||
|
|
||||||
|
- Splitwise — `2311.18677`
|
||||||
|
- DistServe — `2401.09670`
|
||||||
|
- DéjàVu — `2403.01876`
|
||||||
|
- MemServe — `2406.17565`
|
||||||
|
- Mooncake — `2407.00079`
|
||||||
|
- P/D-Serve — `2408.08147`
|
||||||
|
|
||||||
|
### 3.5 前缀和上下文复用
|
||||||
|
|
||||||
|
- Prompt Cache — `2311.04934`
|
||||||
|
- SGLang / RadixAttention — `2312.07104`
|
||||||
|
- Hydragen — `2402.05099`
|
||||||
|
- ChunkAttention — `2402.15220`
|
||||||
|
- CacheGen — `2310.07240`
|
||||||
|
- CacheBlend — `2405.16444`
|
||||||
|
- Preble — `2407.00023`
|
||||||
|
- MemServe — `2406.17565`
|
||||||
|
|
||||||
|
### 3.6 推测解码
|
||||||
|
|
||||||
|
- Fast Inference via Speculative Decoding — `2211.17192`
|
||||||
|
- Speculative Sampling — `2302.01318`
|
||||||
|
- SpecInfer — `2305.09781`
|
||||||
|
- REST — `2311.08252`
|
||||||
|
- Medusa — `2401.10774`
|
||||||
|
- EAGLE — `2401.15077`
|
||||||
|
- Lookahead Decoding — `2402.02057`
|
||||||
|
- EAGLE-2 — `2406.16858`
|
||||||
|
- LayerSkip — `2404.16710`
|
||||||
|
- EAGLE-3 — `2503.01840`
|
||||||
|
- DeepSeek-V3 MTP、K3 MTP→EAGLE-3 draft
|
||||||
|
|
||||||
|
### 3.7 量化
|
||||||
|
|
||||||
|
- LLM.int8 — `2208.07339`
|
||||||
|
- GPTQ — `2210.17323`
|
||||||
|
- SmoothQuant — `2211.10438`
|
||||||
|
- AWQ — `2306.00978`
|
||||||
|
- KVQuant — `2401.18079`
|
||||||
|
- KIVI — `2402.02750`
|
||||||
|
- QuaRot — `2404.00456`
|
||||||
|
- QServe — `2405.04532`
|
||||||
|
- DeepSeek-V4 FP4 QAT、K3 MXFP4/MXFP8 post-training
|
||||||
|
|
||||||
|
### 3.8 分布式与 MoE
|
||||||
|
|
||||||
|
- DeepSpeed-MoE — `2201.05596`
|
||||||
|
- FasterMoE — `2202.09368`
|
||||||
|
- Petals — `2209.01188`
|
||||||
|
- MegaBlocks — `2211.15841`
|
||||||
|
- FlexGen — `2303.06865`
|
||||||
|
- DeepEP — DeepSeek 官方仓库
|
||||||
|
- TensorRT-LLM、TGI、vLLM、SGLang — 官方 runtime 节点
|
||||||
|
|
||||||
|
### 3.9 DeepSeek / Kimi
|
||||||
|
|
||||||
|
- DeepSeek-V2 — `2405.04434`：MLA 与 DeepSeekMoE；
|
||||||
|
- DeepSeek-V3 — `2412.19437`：PD 分离、冗余专家、MTP；
|
||||||
|
- DeepSeek-V3.2 — `2512.02556`：DSA 的 prefill / decode 成本；
|
||||||
|
- DeepSeek-V4 — `2606.19348`：异构 cache、on-disk prefix cache、FP4；
|
||||||
|
- Mooncake — `2407.00079`：Kimi 生产 serving 平台；
|
||||||
|
- Kimi K2 — `2507.20534`：MLA / MoE 结构对服务成本的约束；
|
||||||
|
- Kimi Linear — `2510.26692`：KDA 的固定状态路线；
|
||||||
|
- Kimi K2.5 — `2602.02276`：多模态与 Agent 流量背景；
|
||||||
|
- Kimi K3 — `2607.24653`：KDA-aware cache、KDA decode replay、预算式准入。
|
||||||
|
|
||||||
|
### 3.10 负载与评测
|
||||||
|
|
||||||
|
- BurstGPT — `2401.17644`
|
||||||
|
- Vidur — `2405.05465`
|
||||||
|
- ShareGPT workload traces
|
||||||
|
- MLPerf Inference / GenAI
|
||||||
|
- LLMPerf / GenAI-Perf 官方负载生成器
|
||||||
|
|
||||||
|
## 4. Grok 输出中已识别的错误与淘汰项
|
||||||
|
|
||||||
|
以下不进入正式账本，除非后来找到一手来源：
|
||||||
|
|
||||||
|
- “MegaScale-Infer / MegaScale-MoE”无准确标题和 URL 的占位节点；
|
||||||
|
- “UGache”无可确认论文；
|
||||||
|
- 把 FastChat / Chatbot Arena 当 serving 性能基准；
|
||||||
|
- 把 Kimi k1.5 的推理能力报告当成生产 serving 论文；
|
||||||
|
- 把 Megatron-LM 训练论文直接当 MoE serving 证据；
|
||||||
|
- 把某个框架 README 的当前功能倒灌为原始论文贡献；
|
||||||
|
- 把所有 KV pruning / eviction 近似都写成无损；
|
||||||
|
- 把任何作者报告 speedup 当跨硬件、跨 workload 的常数。
|
||||||
|
|
||||||
|
## 5. 四个候选交互实验
|
||||||
|
|
||||||
|
1. **显存与 KV 账本**：MHA / GQA / MLA / KDA hybrid、精度、上下文、并发共同决定容量。
|
||||||
|
2. **批处理与阶段调度**：static / continuous / chunked / PD 对 TTFT、TPOT 和 goodput 的影响。
|
||||||
|
3. **推测接受率墙**：draft 长度、接受率、draft 成本和 target verification 成本共同决定 speedup。
|
||||||
|
4. **缓存与 fleet**：前缀命中、缓存亲和、故障副本和长短请求预算共同决定尾延迟。
|
||||||
|
|
||||||
|
## 6. 二十个必须纠正的误解
|
||||||
|
|
||||||
|
1. 显存能装下权重，就能稳定服务。
|
||||||
|
2. 推理速度等于峰值 FLOPs。
|
||||||
|
3. prefill 和 decode 是同一种 kernel 工作负载。
|
||||||
|
4. batch 越大，所有用户都越快。
|
||||||
|
5. continuous batching 会自动解决队头阻塞。
|
||||||
|
6. PagedAttention 减少了模型的理论 FLOPs。
|
||||||
|
7. KV Cache 只与上下文长度有关，与 batch、层数和 KV heads 无关。
|
||||||
|
8. GQA、MLA、KDA 都只是同一种“压缩 KV”。
|
||||||
|
9. 前缀缓存可以复用任意相同文本块。
|
||||||
|
10. cache hit rate 是模型固有指标。
|
||||||
|
11. PD 分离一定比共置更便宜。
|
||||||
|
12. KV 迁移只需考虑带宽，不需考虑排队与拓扑。
|
||||||
|
13. 量化位数越低一定越快。
|
||||||
|
14. weight-only 量化会同比减少 KV Cache。
|
||||||
|
15. speculative decoding 保证 2–3×。
|
||||||
|
16. draft 越小越有利。
|
||||||
|
17. MoE 激活参数少，所以在线服务天然便宜。
|
||||||
|
18. 平均 latency 好就代表用户体验好。
|
||||||
|
19. tokens/s、requests/s、goodput 可以互换。
|
||||||
|
20. 官方技术报告的部署数字可直接当成跨系统可复现基准。
|
||||||
|
|
||||||
|
## 7. 转正式账本的硬闸门
|
||||||
|
|
||||||
|
- 每个正文节点必须有 canonical primary URL；
|
||||||
|
- 关键公式必须从架构维度推导或来自论文正文；
|
||||||
|
- 作者报告数字必须带模型、硬件、负载与对照边界；
|
||||||
|
- 教学模拟值必须标“解析模型 / 教学假设”，不得伪装成实测；
|
||||||
|
- DeepSeek 与 Kimi 必须按“模型结构—单机内核—缓存—集群调度”四层分别画；
|
||||||
|
- “推理”能力、test-time compute 与“推理服务”不得混为一章。
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -54,3 +54,15 @@ Agent 首轮缓存位于 `agents/`（不提交 PDF/TXT）：
|
|||||||
DeepSeek-VL/VL2、Janus、OCR 三分支，Kimi 三代 MoonViT、十六张问题账、55 节点正文链与四实验合同见
|
DeepSeek-VL/VL2、Janus、OCR 三分支，Kimi 三代 MoonViT、十六张问题账、55 节点正文链与四实验合同见
|
||||||
`../MULTIMODAL_RESEARCH.md`；Grok Headless 候选召回只保存在
|
`../MULTIMODAL_RESEARCH.md`；Grok Headless 候选召回只保存在
|
||||||
`../MULTIMODAL_GROK_LEADS.md`，不得作为正式事实来源。
|
`../MULTIMODAL_GROK_LEADS.md`，不得作为正式事实来源。
|
||||||
|
|
||||||
|
推理服务首轮缓存位于 `inference-serving/`（不提交 PDF/TXT）：
|
||||||
|
|
||||||
|
- Clockwork、Orca、vLLM / PagedAttention、SGLang、Sarathi、Llumnix、Splitwise 与 DistServe；
|
||||||
|
- speculative decoding / sampling、SpecInfer、Medusa、EAGLE / EAGLE-2 / EAGLE-3；
|
||||||
|
- GPTQ、SmoothQuant、AWQ、KVQuant、KIVI、QServe、FlashInfer 与 Mooncake；
|
||||||
|
- DeepSeek-V2/V3/V3.2/V4、Kimi Linear/K3 与 DeepGEMM/FlashMLA/DeepEP 复用既有报告或官方仓库缓存。
|
||||||
|
|
||||||
|
本轮完整校验 26 份新增 PDF/TXT；AlpaServe 只以 canonical arXiv 页面定位，未把不完整本地下载
|
||||||
|
计入全文证据。十八本独立账、62 节点阅读链、DeepSeek 与 Kimi 服务谱系、十五组视觉合同和
|
||||||
|
四实验合同见 `../INFERENCE_SERVING_RESEARCH.md`。Grok Headless 的 70 个候选只保存在
|
||||||
|
`../INFERENCE_SERVING_GROK_LEADS.md`，不能越过一手来源核验进入正文。
|
||||||
|
|||||||
@@ -217,7 +217,7 @@ if (!overview.title.includes("可靠行动")) failures.push("章节标题异常"
|
|||||||
if (overview.sections !== 28 || overview.tocLinks !== 28) failures.push("章节/目录数量异常");
|
if (overview.sections !== 28 || overview.tocLinks !== 28) failures.push("章节/目录数量异常");
|
||||||
if (overview.paperLinks !== 52) failures.push("正式论文链不是 52 个节点");
|
if (overview.paperLinks !== 52) failures.push("正式论文链不是 52 个节点");
|
||||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||||
if (overview.navLinks !== 17 || home.navLinks !== 17 || mobile.mobileLinks !== 17) failures.push("全站导航未同步 Agent 专题");
|
if (overview.navLinks !== 18 || home.navLinks !== 18 || mobile.mobileLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||||
if (loop.initial.finalState !== "UNVERIFIED" || loop.directSchema.finalState !== "FAILED") failures.push("控制循环终局状态异常");
|
if (loop.initial.finalState !== "UNVERIFIED" || loop.directSchema.finalState !== "FAILED") failures.push("控制循环终局状态异常");
|
||||||
if (!loop.directSchema.observation.includes("ERROR schema") || loop.directSchema.recovery !== "FRAGILE") failures.push("Direct/schema 故障传播异常");
|
if (!loop.directSchema.observation.includes("ERROR schema") || loop.directSchema.recovery !== "FRAGILE") failures.push("Direct/schema 故障传播异常");
|
||||||
@@ -228,8 +228,8 @@ if (numeric(reliability.initial.passAt) <= numeric(reliability.k2.passAt) || num
|
|||||||
if (reliability.nonIdempotent.sideRisk === "LOW") failures.push("非幂等写操作风险没有提升");
|
if (reliability.nonIdempotent.sideRisk === "LOW") failures.push("非幂等写操作风险没有提升");
|
||||||
if (numeric(rl.wait.utilization) >= numeric(rl.full.utilization) || numeric(rl.wait.lostWork) <= numeric(rl.full.lostWork)) failures.push("wait-all 长尾/重算方向异常");
|
if (numeric(rl.wait.utilization) >= numeric(rl.full.utilization) || numeric(rl.wait.lostWork) <= numeric(rl.full.lostWork)) failures.push("wait-all 长尾/重算方向异常");
|
||||||
if (!rl.wait.takeaway.includes("wait-all") || rl.keyboardSelected !== "rl" || rl.keyboardVisible !== "rl") failures.push("长程 RL 解释或键盘导航异常");
|
if (!rl.wait.takeaway.includes("wait-all") || rl.keyboardSelected !== "rl" || rl.keyboardVisible !== "rl") failures.push("长程 RL 解释或键盘导航异常");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") failures.push("首页 Agent 首发入口异常");
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||||
if (home.paperCount !== "355" || papers.total !== 355 || !papers.hasAgentFilter || papers.agentVisible < 52) failures.push("论文库 Agent 标签或论文总数异常");
|
if (home.paperCount !== "400" || papers.total !== 400 || !papers.hasAgentFilter || papers.agentVisible < 52) failures.push("论文库 Agent 标签或论文总数异常");
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||||
|
|
||||||
|
|||||||
@@ -216,7 +216,7 @@ if (!overview.title.includes("真正与人协作")) failures.push("章节标题
|
|||||||
if (overview.sections !== 22 || overview.tocLinks !== 22) failures.push("章节/目录数量异常");
|
if (overview.sections !== 22 || overview.tocLinks !== 22) failures.push("章节/目录数量异常");
|
||||||
if (overview.paperLinks !== 44) failures.push("正式论文链不是 44 个节点");
|
if (overview.paperLinks !== 44) failures.push("正式论文链不是 44 个节点");
|
||||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||||
if (overview.navLinks !== 17 || home.navLinks !== 17 || mobile.mobileLinks !== 17) failures.push("全站导航未同步后训练专题");
|
if (overview.navLinks !== 18 || home.navLinks !== 18 || mobile.mobileLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||||
if (sft.initial.active !== "6 / 10" || !sft.initial.lossStates.slice(0, 4).every((value) => value === "MASKED")) failures.push("SFT response-only mask 异常");
|
if (sft.initial.active !== "6 / 10" || !sft.initial.lossStates.slice(0, 4).every((value) => value === "MASKED")) failures.push("SFT response-only mask 异常");
|
||||||
if (sft.unsafeAll.active !== "10 / 10" || numeric(sft.unsafeAll.nll) <= numeric(sft.initial.nll) || !sft.unsafeAll.reading.includes("错误")) failures.push("SFT 全序列/坏示范交互异常");
|
if (sft.unsafeAll.active !== "10 / 10" || numeric(sft.unsafeAll.nll) <= numeric(sft.initial.nll) || !sft.unsafeAll.reading.includes("错误")) failures.push("SFT 全序列/坏示范交互异常");
|
||||||
@@ -226,8 +226,8 @@ if (!update.steps[0].includes("Fixed preference")) failures.push("DPO 更新流
|
|||||||
if (!recipe.family.includes("Multi-effort") || !recipe.regime.includes("9 RL experts") || !recipe.constraints.includes("verbosity")) failures.push("K3 配方合同异常");
|
if (!recipe.family.includes("Multi-effort") || !recipe.regime.includes("9 RL experts") || !recipe.constraints.includes("verbosity")) failures.push("K3 配方合同异常");
|
||||||
if (!recipe.path.some((step) => step.includes("3 domains × 3 efforts")) || !recipe.path.some((step) => step.includes("MOPD"))) failures.push("K3 配方路径异常");
|
if (!recipe.path.some((step) => step.includes("3 domains × 3 efforts")) || !recipe.path.some((step) => step.includes("MOPD"))) failures.push("K3 配方路径异常");
|
||||||
if (recipe.keyboardSelected !== "recipe" || recipe.keyboardVisible !== "recipe") failures.push("实验 tab 键盘导航异常");
|
if (recipe.keyboardSelected !== "recipe" || recipe.keyboardVisible !== "recipe") failures.push("实验 tab 键盘导航异常");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") failures.push("首页 Alignment 首发入口异常");
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||||
if (home.paperCount !== "355" || papers.total !== 355 || !papers.hasAlignmentFilter || papers.alignmentVisible < 35) failures.push("论文库后训练标签或论文总数异常");
|
if (home.paperCount !== "400" || papers.total !== 400 || !papers.hasAlignmentFilter || papers.alignmentVisible < 35) failures.push("论文库后训练标签或论文总数异常");
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||||
|
|
||||||
|
|||||||
@@ -230,15 +230,15 @@ if (transform.keyboard.selected !== "transform" || transform.keyboard.visible !=
|
|||||||
if (layout.articleSections !== 18 || layout.paperLinks !== 31 || layout.labTabs !== 4 || layout.views !== 4) {
|
if (layout.articleSections !== 18 || layout.paperLinks !== 31 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||||
failures.push("章节、论文或实验数量异常");
|
failures.push("章节、论文或实验数量异常");
|
||||||
}
|
}
|
||||||
if (layout.navLinks !== 17 || mobile.mobileLinks !== 17 || home.navLinks !== 17) failures.push("全站导航未同步数值专题");
|
if (layout.navLinks !== 18 || mobile.mobileLinks !== 18 || home.navLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") {
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") {
|
||||||
failures.push("首页 Transformer 新章入口异常");
|
failures.push("首页 Transformer 新章入口异常");
|
||||||
}
|
}
|
||||||
if (home.paperCount !== "355") failures.push(`首页论文总数异常：${home.paperCount}`);
|
if (home.paperCount !== "400") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||||
if (!papers.hasDataFilter || papers.total !== 355 || papers.visible < 25) failures.push("论文库数据标签或论文总数异常");
|
if (!papers.hasDataFilter || papers.total !== 400 || papers.visible < 25) failures.push("论文库数据标签或论文总数异常");
|
||||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||||
|
|
||||||
socket.close();
|
socket.close();
|
||||||
|
|||||||
@@ -0,0 +1,280 @@
|
|||||||
|
import { writeFileSync } from "node:fs";
|
||||||
|
|
||||||
|
const cdpPort = process.env.CDP_PORT ?? "9227";
|
||||||
|
const baseUrl = process.env.SITE_URL ?? "http://127.0.0.1:4327";
|
||||||
|
const pages = await fetch(`http://127.0.0.1:${cdpPort}/json/list`).then((response) => response.json());
|
||||||
|
const page = pages.find((entry) => entry.type === "page");
|
||||||
|
if (!page) throw new Error(`CDP ${cdpPort} 没有可用页面`);
|
||||||
|
|
||||||
|
const socket = new WebSocket(page.webSocketDebuggerUrl);
|
||||||
|
await new Promise((resolve, reject) => {
|
||||||
|
socket.addEventListener("open", resolve, { once: true });
|
||||||
|
socket.addEventListener("error", reject, { once: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
let nextId = 0;
|
||||||
|
const pending = new Map();
|
||||||
|
const exceptions = [];
|
||||||
|
socket.addEventListener("message", (event) => {
|
||||||
|
const message = JSON.parse(event.data);
|
||||||
|
if (message.id && pending.has(message.id)) {
|
||||||
|
const { resolve, reject } = pending.get(message.id);
|
||||||
|
pending.delete(message.id);
|
||||||
|
if (message.error) reject(new Error(message.error.message));
|
||||||
|
else resolve(message.result);
|
||||||
|
}
|
||||||
|
if (message.method === "Runtime.exceptionThrown") {
|
||||||
|
exceptions.push(message.params.exceptionDetails.exception?.description ?? message.params.exceptionDetails.text);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const command = (method, params = {}) => new Promise((resolve, reject) => {
|
||||||
|
const id = ++nextId;
|
||||||
|
pending.set(id, { resolve, reject });
|
||||||
|
socket.send(JSON.stringify({ id, method, params }));
|
||||||
|
});
|
||||||
|
const pause = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||||
|
const evaluate = async (expression) => {
|
||||||
|
const result = await command("Runtime.evaluate", { expression, returnByValue: true, awaitPromise: true });
|
||||||
|
if (result.exceptionDetails) throw new Error(result.exceptionDetails.exception?.description ?? result.exceptionDetails.text);
|
||||||
|
return result.result.value;
|
||||||
|
};
|
||||||
|
const navigate = async (path) => {
|
||||||
|
await command("Page.navigate", { url: `${baseUrl}${path}` });
|
||||||
|
for (let attempt = 0; attempt < 70; attempt += 1) {
|
||||||
|
await pause(100);
|
||||||
|
if (await evaluate("document.readyState === 'complete'")) return;
|
||||||
|
}
|
||||||
|
throw new Error(`${path} 加载超时`);
|
||||||
|
};
|
||||||
|
const screenshot = async (path) => {
|
||||||
|
const result = await command("Page.captureScreenshot", { format: "png", captureBeyondViewport: false });
|
||||||
|
writeFileSync(path, Buffer.from(result.data, "base64"));
|
||||||
|
};
|
||||||
|
|
||||||
|
await command("Page.enable");
|
||||||
|
await command("Runtime.enable");
|
||||||
|
await command("Emulation.setDeviceMetricsOverride", {
|
||||||
|
width: 1440,
|
||||||
|
height: 1100,
|
||||||
|
deviceScaleFactor: 1,
|
||||||
|
mobile: false,
|
||||||
|
});
|
||||||
|
await navigate("/systems/inference/");
|
||||||
|
await screenshot("/tmp/llm-atlas-inference-desktop.png");
|
||||||
|
|
||||||
|
const overview = await evaluate(`(() => ({
|
||||||
|
title: document.querySelector("h1")?.textContent.trim(),
|
||||||
|
sections: document.querySelectorAll(".article-section").length,
|
||||||
|
tocLinks: document.querySelectorAll(".side-rail a").length,
|
||||||
|
paperLinks: document.querySelectorAll(".paper-chain a").length,
|
||||||
|
labTabs: document.querySelectorAll("[data-is-tab]").length,
|
||||||
|
labPanels: document.querySelectorAll("[data-is-panel]").length,
|
||||||
|
ledgers: document.querySelectorAll(".ledger-grid > article").length,
|
||||||
|
navLinks: document.querySelectorAll(".top-nav a").length,
|
||||||
|
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||||
|
}))()`);
|
||||||
|
|
||||||
|
const memory = await evaluate(`(() => {
|
||||||
|
const root = document.querySelector("[data-inference-lab]");
|
||||||
|
const read = () => ({
|
||||||
|
weight: root.querySelector("[data-memory-weight]").textContent.trim(),
|
||||||
|
perToken: root.querySelector("[data-memory-kv-token]").textContent.trim(),
|
||||||
|
request: root.querySelector("[data-memory-request]").textContent.trim(),
|
||||||
|
active: root.querySelector("[data-memory-active]").textContent.trim(),
|
||||||
|
fit: root.querySelector("[data-memory-fit]").textContent.trim(),
|
||||||
|
status: root.querySelector("[data-memory-status]").textContent.trim(),
|
||||||
|
note: root.querySelector("[data-memory-note]").textContent.trim(),
|
||||||
|
visible: root.querySelector("[data-is-panel]:not([hidden])").dataset.isPanel,
|
||||||
|
});
|
||||||
|
const initial = read();
|
||||||
|
root.querySelector('[data-memory-preset="mha"]').click();
|
||||||
|
const params = root.querySelector("[data-memory-params]");
|
||||||
|
const context = root.querySelector("[data-memory-context]");
|
||||||
|
const concurrency = root.querySelector("[data-memory-concurrency]");
|
||||||
|
params.value = "671";
|
||||||
|
params.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
context.value = "131072";
|
||||||
|
context.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
concurrency.value = "32";
|
||||||
|
concurrency.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
const exploded = read();
|
||||||
|
root.querySelector('[data-memory-preset="k3"]').click();
|
||||||
|
const k3 = read();
|
||||||
|
return { initial, exploded, k3 };
|
||||||
|
})()`);
|
||||||
|
|
||||||
|
const phase = await evaluate(`(() => {
|
||||||
|
const root = document.querySelector("[data-inference-lab]");
|
||||||
|
root.querySelector('[data-is-tab="phase"]').click();
|
||||||
|
const read = () => ({
|
||||||
|
ttft: root.querySelector("[data-phase-ttft]").textContent.trim(),
|
||||||
|
tpot: root.querySelector("[data-phase-tpot]").textContent.trim(),
|
||||||
|
goodput: root.querySelector("[data-phase-goodput]").textContent.trim(),
|
||||||
|
stall: root.querySelector("[data-phase-stall]").textContent.trim(),
|
||||||
|
bottleneck: root.querySelector("[data-phase-bottleneck]").textContent.trim(),
|
||||||
|
status: root.querySelector("[data-phase-status]").textContent.trim(),
|
||||||
|
});
|
||||||
|
const staticBatch = read();
|
||||||
|
root.querySelector('[data-phase-strategy="chunked"]').click();
|
||||||
|
const chunked = read();
|
||||||
|
root.querySelector('[data-phase-strategy="pd"]').click();
|
||||||
|
const network = root.querySelector("[data-phase-network]");
|
||||||
|
network.value = "25";
|
||||||
|
network.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
const slowPd = read();
|
||||||
|
return { staticBatch, chunked, slowPd };
|
||||||
|
})()`);
|
||||||
|
|
||||||
|
const speculative = await evaluate(`(() => {
|
||||||
|
const root = document.querySelector("[data-inference-lab]");
|
||||||
|
root.querySelector('[data-is-tab="speculative"]').click();
|
||||||
|
const read = () => ({
|
||||||
|
expected: root.querySelector("[data-spec-expected]").textContent.trim(),
|
||||||
|
speedup: root.querySelector("[data-spec-speedup]").textContent.trim(),
|
||||||
|
waste: root.querySelector("[data-spec-waste]").textContent.trim(),
|
||||||
|
status: root.querySelector("[data-spec-status]").textContent.trim(),
|
||||||
|
evidence: root.querySelector("[data-spec-evidence]").textContent.trim(),
|
||||||
|
tokens: root.querySelectorAll("[data-spec-strip] i").length,
|
||||||
|
});
|
||||||
|
const k3 = read();
|
||||||
|
const acceptance = root.querySelector("[data-spec-a]");
|
||||||
|
acceptance.value = "20";
|
||||||
|
acceptance.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
const lowAcceptance = read();
|
||||||
|
return { k3, lowAcceptance };
|
||||||
|
})()`);
|
||||||
|
|
||||||
|
const fleet = await evaluate(`(() => {
|
||||||
|
const root = document.querySelector("[data-inference-lab]");
|
||||||
|
root.querySelector('[data-is-tab="fleet"]').click();
|
||||||
|
root.querySelector("[data-fleet-k3]").click();
|
||||||
|
const read = () => ({
|
||||||
|
avoided: root.querySelector("[data-fleet-avoided]").textContent.trim(),
|
||||||
|
skew: root.querySelector("[data-fleet-skew]").textContent.trim(),
|
||||||
|
recompute: root.querySelector("[data-fleet-recompute]").textContent.trim(),
|
||||||
|
admitted: root.querySelector("[data-fleet-admitted]").textContent.trim(),
|
||||||
|
rejected: root.querySelector("[data-fleet-rejected]").textContent.trim(),
|
||||||
|
shortSlo: root.querySelector("[data-fleet-short-slo]").textContent.trim(),
|
||||||
|
state: root.querySelector("[data-fleet-state]").textContent.trim(),
|
||||||
|
});
|
||||||
|
const k3 = read();
|
||||||
|
const failure = root.querySelector("[data-fleet-failure]");
|
||||||
|
failure.checked = true;
|
||||||
|
failure.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
const failed = read();
|
||||||
|
failure.checked = false;
|
||||||
|
failure.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
const admission = root.querySelector("[data-fleet-admission]");
|
||||||
|
const burst = root.querySelector("[data-fleet-burst]");
|
||||||
|
admission.value = "average";
|
||||||
|
admission.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
burst.value = "100";
|
||||||
|
burst.dispatchEvent(new Event("input", { bubbles: true }));
|
||||||
|
const bursty = read();
|
||||||
|
const firstTab = root.querySelector('[data-is-tab="memory"]');
|
||||||
|
firstTab.focus();
|
||||||
|
firstTab.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowRight", bubbles: true }));
|
||||||
|
return {
|
||||||
|
k3,
|
||||||
|
failed,
|
||||||
|
bursty,
|
||||||
|
keyboardSelected: root.querySelector('[data-is-tab][aria-selected="true"]').dataset.isTab,
|
||||||
|
keyboardVisible: root.querySelector("[data-is-panel]:not([hidden])").dataset.isPanel,
|
||||||
|
};
|
||||||
|
})()`);
|
||||||
|
|
||||||
|
await evaluate(`document.querySelector("[data-inference-lab]").scrollIntoView({ block: "start", behavior: "instant" })`);
|
||||||
|
await pause(180);
|
||||||
|
await screenshot("/tmp/llm-atlas-inference-lab-desktop.png");
|
||||||
|
|
||||||
|
await navigate("/");
|
||||||
|
const home = await evaluate(`(() => ({
|
||||||
|
releaseCards: document.querySelectorAll(".release-card").length,
|
||||||
|
firstRelease: document.querySelector(".release-card h2").textContent.trim(),
|
||||||
|
firstHref: document.querySelector(".release-card").getAttribute("href"),
|
||||||
|
paperCount: document.querySelector(".hero-stats div:nth-child(3) b").textContent.trim(),
|
||||||
|
navLinks: document.querySelectorAll(".top-nav a").length,
|
||||||
|
}))()`);
|
||||||
|
|
||||||
|
await navigate("/papers/");
|
||||||
|
const papers = await evaluate(`(() => {
|
||||||
|
const button = [...document.querySelectorAll("[data-filter]")].find((node) => node.textContent.trim() === "推理服务");
|
||||||
|
button?.click();
|
||||||
|
return {
|
||||||
|
total: document.querySelectorAll("[data-paper]").length,
|
||||||
|
visible: document.querySelectorAll("[data-paper]:not([hidden])").length,
|
||||||
|
hasFilter: Boolean(button),
|
||||||
|
};
|
||||||
|
})()`);
|
||||||
|
|
||||||
|
await command("Emulation.setDeviceMetricsOverride", {
|
||||||
|
width: 390,
|
||||||
|
height: 844,
|
||||||
|
deviceScaleFactor: 1,
|
||||||
|
mobile: true,
|
||||||
|
});
|
||||||
|
await navigate("/systems/inference/");
|
||||||
|
const mobile = await evaluate(`(() => {
|
||||||
|
const root = document.querySelector("[data-inference-lab]");
|
||||||
|
root.scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
|
const toggle = document.querySelector("#menu-toggle");
|
||||||
|
toggle?.click();
|
||||||
|
return {
|
||||||
|
documentOverflow: document.documentElement.scrollWidth - document.documentElement.clientWidth,
|
||||||
|
menuVisible: getComputedStyle(toggle).display !== "none",
|
||||||
|
menuOpen: toggle.getAttribute("aria-expanded"),
|
||||||
|
mobileLinks: document.querySelectorAll("#mobile-nav a").length,
|
||||||
|
tabs: root.querySelectorAll("[data-is-tab]").length,
|
||||||
|
offenders: [...document.querySelectorAll("body *")]
|
||||||
|
.filter((node) => !node.closest(".chunk-schedule, .formula, .paper-chain, .scheduler-board"))
|
||||||
|
.filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1)
|
||||||
|
.slice(0, 12)
|
||||||
|
.map((node) => ({
|
||||||
|
tag: node.tagName,
|
||||||
|
className: typeof node.className === "string" ? node.className : "",
|
||||||
|
right: Math.round(node.getBoundingClientRect().right),
|
||||||
|
width: Math.round(node.getBoundingClientRect().width),
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
})()`);
|
||||||
|
await pause(180);
|
||||||
|
await screenshot("/tmp/llm-atlas-inference-mobile.png");
|
||||||
|
|
||||||
|
const report = { overview, memory, phase, speculative, fleet, home, papers, mobile, exceptions };
|
||||||
|
console.log(JSON.stringify(report, null, 2));
|
||||||
|
|
||||||
|
const numeric = (value) => Number.parseFloat(value.replaceAll(",", ""));
|
||||||
|
const failures = [];
|
||||||
|
if (!overview.title.includes("千万人用得起")) failures.push("章节标题异常");
|
||||||
|
if (overview.sections !== 31 || overview.tocLinks !== 31) failures.push("章节/目录数量异常");
|
||||||
|
if (overview.paperLinks !== 62) failures.push("正式论文链不是 62 个节点");
|
||||||
|
if (overview.ledgers !== 18) failures.push("十八本服务账结构异常");
|
||||||
|
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||||
|
if (overview.navLinks !== 18 || home.navLinks !== 18 || mobile.mobileLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||||
|
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||||
|
if (!memory.initial.weight.includes("32.6") || !memory.initial.perToken.includes("128") || memory.initial.visible !== "memory") failures.push("GQA 默认显存账异常");
|
||||||
|
if (!memory.exploded.status.includes("OOM") || !memory.exploded.fit.includes("over")) failures.push("极端 MHA 配置没有触发 OOM");
|
||||||
|
if (!memory.k3.note.includes("69") || !memory.k3.note.includes("24") || !memory.k3.note.includes("未伪造")) failures.push("K3 混合状态证据边界异常");
|
||||||
|
if (numeric(phase.chunked.stall) >= numeric(phase.staticBatch.stall) || numeric(phase.chunked.tpot) >= numeric(phase.staticBatch.tpot)) failures.push("Chunked prefill 没有降低教学 stall / TPOT");
|
||||||
|
if (!phase.slowPd.bottleneck.includes("慢网络") || !phase.slowPd.bottleneck.includes("KV")) failures.push("慢网络没有暴露 P/D 状态传输瓶颈");
|
||||||
|
if (numeric(speculative.k3.speedup) <= 1 || speculative.k3.tokens !== 8 || !speculative.k3.evidence.includes("7 步")) failures.push("K3 7-step 推测预设异常");
|
||||||
|
if (speculative.lowAcceptance.status !== "NO GAIN" || numeric(speculative.lowAcceptance.speedup) >= 1) failures.push("低验收率没有触发负加速");
|
||||||
|
if (!fleet.k3.avoided.includes("320K") || fleet.k3.shortSlo !== "PROTECTED") failures.push("K3 coding trace 缓存 / 准入预设异常");
|
||||||
|
if (!fleet.failed.state.includes("SECONDARY RE-PREFILL") || !fleet.failed.recompute.includes("FAILED PRIMARY")) failures.push("缓存故障没有触发原子失效后的重算");
|
||||||
|
if (fleet.bursty.shortSlo !== "VIOLATED") failures.push("平均并发阈值没有暴露长请求突发");
|
||||||
|
if (fleet.keyboardSelected !== "phase" || fleet.keyboardVisible !== "phase") failures.push("实验键盘 tab 导航异常");
|
||||||
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||||
|
if (home.paperCount !== "400" || papers.total !== 400 || !papers.hasFilter || papers.visible !== 45) failures.push("论文库推理服务标签或总数异常");
|
||||||
|
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||||
|
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||||
|
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||||
|
|
||||||
|
if (failures.length) {
|
||||||
|
console.error(`\nFAIL\n- ${failures.join("\n- ")}`);
|
||||||
|
process.exitCode = 1;
|
||||||
|
} else {
|
||||||
|
console.log("\nPASS inference serving browser regression");
|
||||||
|
}
|
||||||
|
|
||||||
|
socket.close();
|
||||||
@@ -177,7 +177,7 @@ if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentO
|
|||||||
}
|
}
|
||||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||||
if (!mobile.menuVisible) failures.push("移动端菜单按钮未显示");
|
if (!mobile.menuVisible) failures.push("移动端菜单按钮未显示");
|
||||||
if (home.releaseCards !== 12) failures.push(`首页新章卡数量异常：${home.releaseCards}`);
|
if (home.releaseCards !== 13) failures.push(`首页新章卡数量异常：${home.releaseCards}`);
|
||||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||||
|
|
||||||
socket.close();
|
socket.close();
|
||||||
|
|||||||
@@ -234,7 +234,7 @@ if (!overview.title.includes("第一类输入")) failures.push("章节标题异
|
|||||||
if (overview.sections !== 30 || overview.tocLinks !== 30) failures.push("章节/目录数量异常");
|
if (overview.sections !== 30 || overview.tocLinks !== 30) failures.push("章节/目录数量异常");
|
||||||
if (overview.paperLinks !== 55) failures.push("正式论文链不是 55 个节点");
|
if (overview.paperLinks !== 55) failures.push("正式论文链不是 55 个节点");
|
||||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||||
if (overview.navLinks !== 17 || home.navLinks !== 17 || mobile.mobileLinks !== 17) failures.push("全站导航未同步多模态专题");
|
if (overview.navLinks !== 18 || home.navLinks !== 18 || mobile.mobileLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在横向溢出");
|
||||||
if (numeric(tokens.initial.patches) !== 5476 || numeric(tokens.initial.visual) !== 1369 || tokens.initial.visiblePanel !== "tokens") failures.push("视觉 Token 初始计算异常");
|
if (numeric(tokens.initial.patches) !== 5476 || numeric(tokens.initial.visual) !== 1369 || tokens.initial.visiblePanel !== "tokens") failures.push("视觉 Token 初始计算异常");
|
||||||
if (!tokens.overloaded.status.includes("超预算") || numeric(tokens.overloaded.share) <= 100) failures.push("超大视频没有触发上下文超预算");
|
if (!tokens.overloaded.status.includes("超预算") || numeric(tokens.overloaded.share) <= 100) failures.push("超大视频没有触发上下文超预算");
|
||||||
@@ -246,8 +246,8 @@ if (ocr.unreported.status !== "OUT OF EVIDENCE" || ocr.unreported.accuracy !== "
|
|||||||
if (loop.toolsStart.state !== "OPEN" || loop.toolsEnd.state !== "VERIFIED" || loop.toolsEnd.evidence !== "97%" || loop.toolsEnd.tools !== "3") failures.push("vision-in-the-loop 终局异常");
|
if (loop.toolsStart.state !== "OPEN" || loop.toolsEnd.state !== "VERIFIED" || loop.toolsEnd.evidence !== "97%" || loop.toolsEnd.tools !== "3") failures.push("vision-in-the-loop 终局异常");
|
||||||
if (loop.cotEnd.state !== "FAILED" || !loop.cotEnd.takeaway.includes("不能凭空增加")) failures.push("文字 CoT 与新观察没有分开");
|
if (loop.cotEnd.state !== "FAILED" || !loop.cotEnd.takeaway.includes("不能凭空增加")) failures.push("文字 CoT 与新观察没有分开");
|
||||||
if (loop.keyboardSelected !== "connector" || loop.keyboardVisible !== "connector") failures.push("实验键盘 tab 导航异常");
|
if (loop.keyboardSelected !== "connector" || loop.keyboardVisible !== "connector") failures.push("实验键盘 tab 导航异常");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") failures.push("首页多模态首发入口异常");
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||||
if (home.paperCount !== "355" || papers.total !== 355 || !papers.hasFilter || papers.multimodalVisible < 59) failures.push("论文库多模态标签或总数异常");
|
if (home.paperCount !== "400" || papers.total !== 400 || !papers.hasFilter || papers.multimodalVisible < 59) failures.push("论文库多模态标签或总数异常");
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||||
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||||
|
|||||||
@@ -273,15 +273,15 @@ if (stability.keyboard.selected !== "stability" || stability.keyboard.visible !=
|
|||||||
if (layout.articleSections !== 19 || layout.paperLinks !== 36 || layout.labTabs !== 4 || layout.views !== 4) {
|
if (layout.articleSections !== 19 || layout.paperLinks !== 36 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||||
failures.push("章节、论文或实验数量异常");
|
failures.push("章节、论文或实验数量异常");
|
||||||
}
|
}
|
||||||
if (layout.navLinks !== 17 || mobile.mobileLinks !== 17 || home.navLinks !== 17) failures.push("全站导航未同步数值专题");
|
if (layout.navLinks !== 18 || mobile.mobileLinks !== 18 || home.navLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") {
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") {
|
||||||
failures.push("首页 Transformer 新章入口异常");
|
failures.push("首页 Transformer 新章入口异常");
|
||||||
}
|
}
|
||||||
if (home.paperCount !== "355") failures.push(`首页论文总数异常：${home.paperCount}`);
|
if (home.paperCount !== "400") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||||
if (!papers.hasOptimizerFilter || papers.total !== 355 || papers.visible < 8) failures.push("论文库优化器标签或论文总数异常");
|
if (!papers.hasOptimizerFilter || papers.total !== 400 || papers.visible < 8) failures.push("论文库优化器标签或论文总数异常");
|
||||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||||
|
|
||||||
socket.close();
|
socket.close();
|
||||||
|
|||||||
@@ -289,7 +289,7 @@ if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentO
|
|||||||
}
|
}
|
||||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态")) failures.push("首页 Transformer 新章入口异常");
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化")) failures.push("首页推理服务新章入口异常");
|
||||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||||
|
|
||||||
socket.close();
|
socket.close();
|
||||||
|
|||||||
@@ -269,14 +269,14 @@ if (emergence.paths.some((length) => length < 500)) failures.push("涌现多指
|
|||||||
if (layout.articleSections !== 16 || layout.paperLinks !== 29 || layout.labTabs !== 4 || layout.views !== 4) {
|
if (layout.articleSections !== 16 || layout.paperLinks !== 29 || layout.labTabs !== 4 || layout.views !== 4) {
|
||||||
failures.push("章节、论文或实验数量异常");
|
failures.push("章节、论文或实验数量异常");
|
||||||
}
|
}
|
||||||
if (layout.navLinks !== 17 || mobile.mobileLinks !== 17 || home.navLinks !== 17) failures.push("全站导航未同步数值专题");
|
if (layout.navLinks !== 18 || mobile.mobileLinks !== 18 || home.navLinks !== 18) failures.push("全站导航未同步推理服务专题");
|
||||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") {
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") {
|
||||||
failures.push("首页 Transformer 新章入口异常");
|
failures.push("首页 Transformer 新章入口异常");
|
||||||
}
|
}
|
||||||
if (home.paperCount !== "355") failures.push(`首页论文总数异常：${home.paperCount}`);
|
if (home.paperCount !== "400") failures.push(`首页论文总数异常：${home.paperCount}`);
|
||||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||||
|
|
||||||
socket.close();
|
socket.close();
|
||||||
|
|||||||
@@ -233,7 +233,7 @@ if (layout.articleSections !== 16 || layout.paperLinks !== 37 || layout.labTabs
|
|||||||
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
if (layout.documentOverflow > 0 || mobile.documentOverflow > 0 || home.documentOverflow > 0) failures.push("页面存在横向溢出");
|
||||||
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
if (layout.navGap < 0) failures.push(`桌面导航碰撞：${layout.navGap}px`);
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true") failures.push("移动端菜单不可用");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态")) failures.push("首页 Transformer 新章入口异常");
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化")) failures.push("首页推理服务新章入口异常");
|
||||||
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
if (exceptions.length) failures.push(`浏览器脚本异常：${exceptions.join("; ")}`);
|
||||||
|
|
||||||
socket.close();
|
socket.close();
|
||||||
|
|||||||
@@ -172,7 +172,7 @@ const home = await evaluate(`(() => ({
|
|||||||
releaseCards: document.querySelectorAll(".release-card").length,
|
releaseCards: document.querySelectorAll(".release-card").length,
|
||||||
firstRelease: document.querySelector(".release-card h2").textContent,
|
firstRelease: document.querySelector(".release-card h2").textContent,
|
||||||
firstHref: document.querySelector(".release-card").getAttribute("href"),
|
firstHref: document.querySelector(".release-card").getAttribute("href"),
|
||||||
paperCount: [...document.querySelectorAll(".hero-stats b")].map((node) => node.textContent.trim()).find((value) => value === "355"),
|
paperCount: [...document.querySelectorAll(".hero-stats b")].map((node) => node.textContent.trim()).find((value) => value === "400"),
|
||||||
}))()`);
|
}))()`);
|
||||||
|
|
||||||
await navigate("/papers/");
|
await navigate("/papers/");
|
||||||
@@ -236,8 +236,8 @@ if (block.family.trim() !== "Hybrid MoE" || !block.kv.includes("3 KDA : 1 Gated
|
|||||||
if (!block.path.some((step) => step.includes("KDA × 3")) || !block.note.includes("AttnRes")) failures.push("K3 Block 路径异常");
|
if (!block.path.some((step) => step.includes("KDA × 3")) || !block.note.includes("AttnRes")) failures.push("K3 Block 路径异常");
|
||||||
if (block.context.trim() !== "128K" || numeric(block.mha) !== 400 || numeric(block.kda) !== 1) failures.push("KV 成本缩放异常");
|
if (block.context.trim() !== "128K" || numeric(block.mha) !== 400 || numeric(block.kda) !== 1) failures.push("KV 成本缩放异常");
|
||||||
if (block.keyboardSelected !== "block" || block.keyboardVisible !== "block") failures.push("实验 tab 键盘导航异常");
|
if (block.keyboardSelected !== "block" || block.keyboardVisible !== "block") failures.push("实验 tab 键盘导航异常");
|
||||||
if (home.releaseCards !== 12 || !home.firstRelease.includes("原生多模态") || home.firstHref !== "/multimodal/") failures.push("首页 Transformer 首发入口异常");
|
if (home.releaseCards !== 13 || !home.firstRelease.includes("推理优化") || home.firstHref !== "/systems/inference/") failures.push("首页推理服务首发入口异常");
|
||||||
if (home.paperCount !== "355" || papers.total !== 355 || papers.transformerVisible < 30) failures.push("论文库或首页论文数量异常");
|
if (home.paperCount !== "400" || papers.total !== 400 || papers.transformerVisible < 30) failures.push("论文库或首页论文数量异常");
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4) failures.push("移动端导航或实验异常");
|
||||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,998 @@
|
|||||||
|
---
|
||||||
|
const memoryPresets = [
|
||||||
|
{
|
||||||
|
id: "mha",
|
||||||
|
name: "MHA",
|
||||||
|
subtitle: "每个 Q 头保存一组 K/V",
|
||||||
|
params: 70,
|
||||||
|
weightBits: 16,
|
||||||
|
layers: 80,
|
||||||
|
kvHeads: 64,
|
||||||
|
headDim: 128,
|
||||||
|
kvBits: 16,
|
||||||
|
note: "标准 MHA 教学配置：KV 随 batch、序列长度和层数线性增长。",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "gqa",
|
||||||
|
name: "GQA",
|
||||||
|
subtitle: "多个 Q 头共享 K/V",
|
||||||
|
params: 70,
|
||||||
|
weightBits: 4,
|
||||||
|
layers: 32,
|
||||||
|
kvHeads: 8,
|
||||||
|
headDim: 128,
|
||||||
|
kvBits: 16,
|
||||||
|
note: "默认教学配置，不对应某个具体 checkpoint；用于观察减少 KV heads 的直接收益。",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "mla",
|
||||||
|
name: "MLA",
|
||||||
|
subtitle: "缓存低维 latent",
|
||||||
|
params: 236,
|
||||||
|
weightBits: 8,
|
||||||
|
layers: 60,
|
||||||
|
kvHeads: 1,
|
||||||
|
headDim: 576,
|
||||||
|
kvBits: 16,
|
||||||
|
note: "把 MLA 的压缩 latent 写成“等效 1 个 576 维 KV 头”只为量纲教学，不是复刻 DeepSeek-V2 权重布局。",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "k3",
|
||||||
|
name: "K3 HYBRID",
|
||||||
|
subtitle: "69 KDA + 24 MLA",
|
||||||
|
params: 1000,
|
||||||
|
weightBits: 4,
|
||||||
|
layers: 24,
|
||||||
|
kvHeads: 1,
|
||||||
|
headDim: 512,
|
||||||
|
kvBits: 8,
|
||||||
|
note: "K3 报告公开了 69 个 KDA 与 24 个 Gated MLA 层；这里仅估算会随上下文增长的 MLA latent，KDA 固定状态未伪造为精确字节。",
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
const phaseStrategies = [
|
||||||
|
["static", "静态批处理", "等最长请求"],
|
||||||
|
["continuous", "连续批处理", "完成即补位"],
|
||||||
|
["chunked", "Chunked prefill", "切片交错 decode"],
|
||||||
|
["pd", "P/D 解耦", "分开扩容与调度"],
|
||||||
|
];
|
||||||
|
|
||||||
|
const speculativePresets = [
|
||||||
|
["vanilla", "独立小模型", 5, 0.62, 0.12, 1.08],
|
||||||
|
["medusa", "Medusa heads", 5, 0.72, 0.05, 1.14],
|
||||||
|
["eagle3", "EAGLE-3", 6, 0.79, 0.07, 1.13],
|
||||||
|
["k3", "K3 / 7-step", 7, 0.82, 0.08, 1.15],
|
||||||
|
];
|
||||||
|
---
|
||||||
|
|
||||||
|
<figure
|
||||||
|
class="is-lab"
|
||||||
|
data-inference-lab
|
||||||
|
data-memory-presets={JSON.stringify(memoryPresets)}
|
||||||
|
data-spec-presets={JSON.stringify(speculativePresets)}
|
||||||
|
>
|
||||||
|
<div class="lab-head">
|
||||||
|
<div>
|
||||||
|
<p>INTERACTIVE / SERVING CONTROL ROOM</p>
|
||||||
|
<h3>别只背系统名：亲手管四本推理账</h3>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
每个实验都公开公式、假设与证据边界。它们是可复算的教学模型，不是 GPU
|
||||||
|
benchmark，也不会把论文中的特定集群结果外推成普遍性能。
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="lab-tabs" role="tablist" aria-label="选择推理服务实验">
|
||||||
|
<button type="button" role="tab" data-is-tab="memory" aria-selected="true">
|
||||||
|
<span>01</span><b>显存与 KV 账</b><small>weights / state / pages</small>
|
||||||
|
</button>
|
||||||
|
<button type="button" role="tab" data-is-tab="phase" aria-selected="false" tabindex="-1">
|
||||||
|
<span>02</span><b>Prefill / Decode</b><small>batching / chunk / P-D</small>
|
||||||
|
</button>
|
||||||
|
<button type="button" role="tab" data-is-tab="speculative" aria-selected="false" tabindex="-1">
|
||||||
|
<span>03</span><b>推测解码</b><small>acceptance / draft / verify</small>
|
||||||
|
</button>
|
||||||
|
<button type="button" role="tab" data-is-tab="fleet" aria-selected="false" tabindex="-1">
|
||||||
|
<span>04</span><b>缓存与集群</b><small>affinity / budget / failure</small>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<section class="lab-panel" data-is-panel="memory">
|
||||||
|
<div class="panel-intro">
|
||||||
|
<div><span>LEDGER 01 / MEMORY</span><h4>80 GB 到底装了什么？</h4></div>
|
||||||
|
<p>先付权重，再付随请求增长的状态；分页只能减少浪费，不能让真实 KV 消失。</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="preset-row" role="group" aria-label="选择注意力状态预设">
|
||||||
|
{memoryPresets.map((preset, index) => (
|
||||||
|
<button type="button" data-memory-preset={preset.id} class={index === 1 ? "active" : ""}>
|
||||||
|
<b>{preset.name}</b><small>{preset.subtitle}</small>
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="control-grid memory-controls">
|
||||||
|
<label>
|
||||||
|
<span>参数量</span>
|
||||||
|
<select data-memory-params>
|
||||||
|
<option value="8">8B</option>
|
||||||
|
<option value="70" selected>70B</option>
|
||||||
|
<option value="236">236B</option>
|
||||||
|
<option value="671">671B</option>
|
||||||
|
<option value="1000">1T</option>
|
||||||
|
<option value="3000">3T</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>权重精度</span>
|
||||||
|
<select data-memory-weight-bits>
|
||||||
|
<option value="16">16-bit</option>
|
||||||
|
<option value="8">8-bit</option>
|
||||||
|
<option value="4" selected>4-bit</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>注意力层数</span>
|
||||||
|
<input data-memory-layers type="number" min="1" max="160" value="32" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>KV heads</span>
|
||||||
|
<input data-memory-kv-heads type="number" min="1" max="128" value="8" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>每头维度</span>
|
||||||
|
<input data-memory-head-dim type="number" min="32" max="1024" step="32" value="128" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>KV 精度</span>
|
||||||
|
<select data-memory-kv-bits>
|
||||||
|
<option value="16" selected>16-bit</option>
|
||||||
|
<option value="8">8-bit</option>
|
||||||
|
<option value="4">4-bit</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>上下文</span>
|
||||||
|
<select data-memory-context>
|
||||||
|
<option value="8192">8K</option>
|
||||||
|
<option value="32760" selected>32,760</option>
|
||||||
|
<option value="131072">128K</option>
|
||||||
|
<option value="400000">400K</option>
|
||||||
|
<option value="1000000">1M</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>并发请求</span>
|
||||||
|
<input data-memory-concurrency type="number" min="1" max="256" value="8" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>分页 block</span>
|
||||||
|
<select data-memory-block>
|
||||||
|
<option value="8">8 tokens</option>
|
||||||
|
<option value="16" selected>16 tokens</option>
|
||||||
|
<option value="64">64 tokens</option>
|
||||||
|
<option value="512">512 tokens</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>单卡 HBM</span>
|
||||||
|
<select data-memory-hbm>
|
||||||
|
<option value="48">48 GiB</option>
|
||||||
|
<option value="80" selected>80 GiB</option>
|
||||||
|
<option value="96">96 GiB</option>
|
||||||
|
<option value="141">141 GiB</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="formula-card">
|
||||||
|
<span>EXACT FOR MHA / MQA / GQA</span>
|
||||||
|
<code>KVBytes = B × T × L × 2 × H<sub>kv</sub> × D<sub>h</sub> × bytes</code>
|
||||||
|
<p>“2”来自 K 与 V。MLA / KDA 的状态结构不同，必须另立口径，不能只改一个名字。</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="metric-grid five">
|
||||||
|
<article>
|
||||||
|
<span>WEIGHTS</span><b data-memory-weight>32.60 GiB</b><p>未计 scale、元数据与 runtime workspace</p>
|
||||||
|
</article>
|
||||||
|
<article>
|
||||||
|
<span>KV / TOKEN</span><b data-memory-kv-token>128.00 KiB</b><p>单请求每新增一个位置</p>
|
||||||
|
</article>
|
||||||
|
<article>
|
||||||
|
<span>ONE REQUEST</span><b data-memory-request>4.00 GiB</b><p>指定上下文长度的增长状态</p>
|
||||||
|
</article>
|
||||||
|
<article>
|
||||||
|
<span>ACTIVE SET</span><b data-memory-active>64.60 GiB</b><p>权重 + 全部活动请求 KV</p>
|
||||||
|
</article>
|
||||||
|
<article class="dark">
|
||||||
|
<span>FIT / ONE GPU</span><b data-memory-fit>15.40 GiB left</b><p data-memory-status>能装下，但还没给 kernel workspace 留余量</p>
|
||||||
|
</article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="readout-row">
|
||||||
|
<div>
|
||||||
|
<span>PAGE TAIL WASTE</span>
|
||||||
|
<b data-memory-waste>0 B</b>
|
||||||
|
<p>这里只有最后一个 block 的内部浪费；PagedAttention 还会改善非连续分配与共享。</p>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<span>EVIDENCE BOUNDARY</span>
|
||||||
|
<b data-memory-note>默认 GQA 是量纲教学配置</b>
|
||||||
|
<p>选 K3 会切换为“增长状态等效估算”，同时保留固定 KDA 状态未公开的边界。</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="lab-panel" data-is-panel="phase" hidden>
|
||||||
|
<div class="panel-intro">
|
||||||
|
<div><span>LEDGER 02 / TIME</span><h4>首字慢，和字间慢，不是同一种慢</h4></div>
|
||||||
|
<p>观察调度策略怎样改变 TTFT、TPOT、吞吐与 goodput；均为相对教学模拟。</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="preset-row strategy-row" role="group" aria-label="选择调度策略">
|
||||||
|
{phaseStrategies.map(([id, name, note], index) => (
|
||||||
|
<button type="button" data-phase-strategy={id} class={index === 0 ? "active" : ""}>
|
||||||
|
<b>{name}</b><small>{note}</small>
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="control-grid phase-controls">
|
||||||
|
<label>
|
||||||
|
<span>长请求占比 <output data-phase-mix-label>60%</output></span>
|
||||||
|
<input data-phase-mix type="range" min="0" max="100" value="60" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>平均 prompt</span>
|
||||||
|
<select data-phase-prompt>
|
||||||
|
<option value="1024">1K</option>
|
||||||
|
<option value="8192" selected>8K</option>
|
||||||
|
<option value="32768">32K</option>
|
||||||
|
<option value="131072">128K</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>平均输出</span>
|
||||||
|
<select data-phase-output>
|
||||||
|
<option value="128">128</option>
|
||||||
|
<option value="512" selected>512</option>
|
||||||
|
<option value="2048">2K</option>
|
||||||
|
<option value="8192">8K</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>并发</span>
|
||||||
|
<input data-phase-concurrency type="range" min="4" max="128" step="4" value="32" />
|
||||||
|
<output data-phase-concurrency-label>32 requests</output>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>Prefill chunk</span>
|
||||||
|
<select data-phase-chunk>
|
||||||
|
<option value="512">512</option>
|
||||||
|
<option value="1024" selected>1K</option>
|
||||||
|
<option value="4096">4K</option>
|
||||||
|
<option value="16384">16K</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>P→D 网络</span>
|
||||||
|
<select data-phase-network>
|
||||||
|
<option value="400" selected>400 Gb/s</option>
|
||||||
|
<option value="100">100 Gb/s</option>
|
||||||
|
<option value="25">25 Gb/s</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="phase-pipeline" aria-label="Prefill 与 Decode 服务流水线">
|
||||||
|
<article>
|
||||||
|
<span>COMPUTE-HEAVY REGIME</span>
|
||||||
|
<b>PREFILL</b>
|
||||||
|
<i><em data-phase-prefill-meter></em></i>
|
||||||
|
<p>一次处理许多 prompt Token；长输入会推迟第一次输出。</p>
|
||||||
|
</article>
|
||||||
|
<u data-phase-link>同一队列互相阻塞</u>
|
||||||
|
<article>
|
||||||
|
<span>BANDWIDTH-HEAVY REGIME</span>
|
||||||
|
<b>DECODE</b>
|
||||||
|
<i><em data-phase-decode-meter></em></i>
|
||||||
|
<p>每步只生成少量 Token，却反复读取权重与历史状态。</p>
|
||||||
|
</article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="metric-grid five">
|
||||||
|
<article><span>TTFT P95</span><b data-phase-ttft>—</b><p>排队 + prefill + 调度</p></article>
|
||||||
|
<article><span>TPOT P95</span><b data-phase-tpot>—</b><p>相邻输出 Token 间隔</p></article>
|
||||||
|
<article><span>THROUGHPUT</span><b data-phase-throughput>—</b><p>完成的 output tokens/s</p></article>
|
||||||
|
<article><span>GOODPUT</span><b data-phase-goodput>—</b><p>同时满足 TTFT / TPOT SLO</p></article>
|
||||||
|
<article class="dark"><span>DECODE STALL</span><b data-phase-stall>—</b><p data-phase-status>—</p></article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="readout-row">
|
||||||
|
<div><span>BOTTLENECK</span><b data-phase-bottleneck>—</b><p>最优策略会随工作负载和网络变化，不存在脱离 SLO 的单一冠军。</p></div>
|
||||||
|
<div><span>MODEL STATUS</span><b>ANALYTIC TOY / NOT A BENCHMARK</b><p data-phase-evidence>以静态批为归一化基线，展示方向性取舍。</p></div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="lab-panel" data-is-panel="speculative" hidden>
|
||||||
|
<div class="panel-intro">
|
||||||
|
<div><span>LEDGER 03 / SPECULATION</span><h4>草稿越长，为什么可能反而越慢？</h4></div>
|
||||||
|
<p>速度取决于验收率、草稿成本和并行验证成本；“一次猜七步”不是免费得到七个 Token。</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="preset-row" role="group" aria-label="选择推测解码预设">
|
||||||
|
{speculativePresets.map(([id, name], index) => (
|
||||||
|
<button type="button" data-spec-preset={id} class={index === 3 ? "active" : ""}>
|
||||||
|
<b>{name}</b><small>{id === "k3" ? "KDA rollback + feature fusion" : "教学预设"}</small>
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="control-grid spec-controls">
|
||||||
|
<label>
|
||||||
|
<span>草稿长度 k <output data-spec-k-label>7</output></span>
|
||||||
|
<input data-spec-k type="range" min="1" max="12" value="7" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>单步验收率 a <output data-spec-a-label>0.82</output></span>
|
||||||
|
<input data-spec-a type="range" min="5" max="98" value="82" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>每步草稿成本 <output data-spec-draft-label>0.08×</output></span>
|
||||||
|
<input data-spec-draft type="range" min="1" max="40" value="8" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>并行验证成本 <output data-spec-verify-label>1.15×</output></span>
|
||||||
|
<input data-spec-verify type="range" min="100" max="220" value="115" />
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="spec-strip" data-spec-strip aria-label="草稿 Token 验收示意"></div>
|
||||||
|
|
||||||
|
<div class="formula-card">
|
||||||
|
<span>INDEPENDENT-ACCEPTANCE TOY</span>
|
||||||
|
<code data-spec-formula>E[tokens] = 1 + a + … + aᵏ</code>
|
||||||
|
<p>精确采样算法使用目标分布与草稿分布的重叠质量；独立常数 a 只是帮助建立直觉。</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="metric-grid four">
|
||||||
|
<article><span>EXPECTED OUTPUT</span><b data-spec-expected>—</b><p>一次 target verification 的期望产出</p></article>
|
||||||
|
<article><span>SPEEDUP</span><b data-spec-speedup>—</b><p>相对逐 Token target 解码</p></article>
|
||||||
|
<article><span>WASTED DRAFT</span><b data-spec-waste>—</b><p>算过但未被接受的候选</p></article>
|
||||||
|
<article class="dark"><span>VERDICT</span><b data-spec-status>—</b><p>低验收率会把草稿计算变成纯开销</p></article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="readout-row single">
|
||||||
|
<div>
|
||||||
|
<span>EVIDENCE BOUNDARY</span>
|
||||||
|
<b data-spec-evidence>K3 报告：预训练 MTP → EAGLE-3 风格 draft，展开 7 步</b>
|
||||||
|
<p>K3 的 KDA rollback 会缓存投影输入并重放已接受状态；这里没有把回滚成本假装成零。</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="lab-panel" data-is-panel="fleet" hidden>
|
||||||
|
<div class="panel-intro">
|
||||||
|
<div><span>LEDGER 04 / FLEET</span><h4>缓存命中以后，系统问题才刚开始</h4></div>
|
||||||
|
<p>亲和路由提高复用，却可能制造热点；预算准入保护短请求，却会主动拒绝一部分长请求。</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="fleet-presets">
|
||||||
|
<button type="button" data-fleet-k3 class="active">
|
||||||
|
<b>K3 coding trace</b><small>400K cached prefix + 4K increment</small>
|
||||||
|
</button>
|
||||||
|
<label class="failure-toggle">
|
||||||
|
<input data-fleet-failure type="checkbox" />
|
||||||
|
<span><b>注入主缓存故障</b><small>观察 pin / copy / invalidation 后的回源</small></span>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="control-grid fleet-controls">
|
||||||
|
<label>
|
||||||
|
<span>可复用前缀</span>
|
||||||
|
<select data-fleet-prefix>
|
||||||
|
<option value="8192">8K</option>
|
||||||
|
<option value="131072">128K</option>
|
||||||
|
<option value="400000" selected>400K</option>
|
||||||
|
<option value="1000000">1M</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>本轮新增 Token</span>
|
||||||
|
<select data-fleet-increment>
|
||||||
|
<option value="512">512</option>
|
||||||
|
<option value="4096" selected>4K</option>
|
||||||
|
<option value="32768">32K</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>有效命中率 <output data-fleet-hit-label>80%</output></span>
|
||||||
|
<input data-fleet-hit type="range" min="0" max="100" value="80" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>服务集群</span>
|
||||||
|
<input data-fleet-clusters type="range" min="2" max="16" value="8" />
|
||||||
|
<output data-fleet-clusters-label>8 clusters</output>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>长请求突发 <output data-fleet-burst-label>60%</output></span>
|
||||||
|
<input data-fleet-burst type="range" min="0" max="100" value="60" />
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>路由</span>
|
||||||
|
<select data-fleet-routing>
|
||||||
|
<option value="round">Round-robin</option>
|
||||||
|
<option value="affinity" selected>Cache-aware affinity</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>准入</span>
|
||||||
|
<select data-fleet-admission>
|
||||||
|
<option value="average">平均并发阈值</option>
|
||||||
|
<option value="budget" selected>Token budget</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="fleet-map" aria-label="缓存亲和路由示意">
|
||||||
|
<div class="request-stack">
|
||||||
|
<span>REQUESTS</span>
|
||||||
|
<i></i><i></i><i></i><i></i><i></i>
|
||||||
|
</div>
|
||||||
|
<u data-fleet-route-label>CACHE KEY + LOAD</u>
|
||||||
|
<div class="cluster-stack" data-fleet-cluster-stack></div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="metric-grid four">
|
||||||
|
<article><span>AVOIDED PREFILL</span><b data-fleet-avoided>—</b><p>命中缓存而无需重新计算的 Token</p></article>
|
||||||
|
<article><span>LOAD SKEW</span><b data-fleet-skew>—</b><p>亲和性收益对应的热点代价</p></article>
|
||||||
|
<article><span>ADMISSION</span><b><em data-fleet-admitted>—</em> / <em data-fleet-rejected>—</em></b><p>admitted / rejected per 100</p></article>
|
||||||
|
<article class="dark"><span>SHORT-REQ SLO</span><b data-fleet-short-slo>—</b><p data-fleet-state>—</p></article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="readout-row">
|
||||||
|
<div><span>RECOMPUTE</span><b data-fleet-recompute>—</b><p>故障时不能把已失效 page 当成命中；需要原子失效后重新 prefill。</p></div>
|
||||||
|
<div><span>EVIDENCE BOUNDARY</span><b data-fleet-evidence>K3 报告的典型 coding trace 与缓存语义</b><p>这里的命中率、负载和准入曲线是教学模型，不是 K3 线上集群披露值。</p></div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<figcaption>
|
||||||
|
<b>HOW TO READ</b>
|
||||||
|
先让实验失败，再寻找是哪一本账失衡。显存、延迟、算力、网络和 SLO
|
||||||
|
不能互相替代，也不能用单一“tokens/s”概括。
|
||||||
|
</figcaption>
|
||||||
|
</figure>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
document.querySelectorAll<HTMLElement>("[data-inference-lab]").forEach((root) => {
|
||||||
|
const query = <T extends Element>(selector: string) => root.querySelector<T>(selector);
|
||||||
|
const queryAll = <T extends Element>(selector: string) => [...root.querySelectorAll<T>(selector)];
|
||||||
|
const number = (selector: string) => Number((query<HTMLInputElement | HTMLSelectElement>(selector))?.value ?? 0);
|
||||||
|
const set = (selector: string, value: string) => {
|
||||||
|
const node = query<HTMLElement>(selector);
|
||||||
|
if (node) node.textContent = value;
|
||||||
|
};
|
||||||
|
const format = new Intl.NumberFormat("en-US", { maximumFractionDigits: 2 });
|
||||||
|
const formatTokens = (value: number) =>
|
||||||
|
value >= 1_000_000 ? `${format.format(value / 1_000_000)}M` : value >= 1_000 ? `${format.format(value / 1_000)}K` : format.format(value);
|
||||||
|
const formatBytes = (bytes: number) => {
|
||||||
|
const gib = bytes / 2 ** 30;
|
||||||
|
if (gib >= 0.1) return `${format.format(gib)} GiB`;
|
||||||
|
const mib = bytes / 2 ** 20;
|
||||||
|
if (mib >= 1) return `${format.format(mib)} MiB`;
|
||||||
|
return `${format.format(bytes / 2 ** 10)} KiB`;
|
||||||
|
};
|
||||||
|
|
||||||
|
const tabs = queryAll<HTMLButtonElement>("[data-is-tab]");
|
||||||
|
const activateTab = (id: string) => {
|
||||||
|
tabs.forEach((button) => {
|
||||||
|
const selected = button.dataset.isTab === id;
|
||||||
|
button.setAttribute("aria-selected", String(selected));
|
||||||
|
button.tabIndex = selected ? 0 : -1;
|
||||||
|
});
|
||||||
|
queryAll<HTMLElement>("[data-is-panel]").forEach((panel) => {
|
||||||
|
panel.hidden = panel.dataset.isPanel !== id;
|
||||||
|
});
|
||||||
|
};
|
||||||
|
tabs.forEach((button, index) => {
|
||||||
|
button.addEventListener("click", () => activateTab(button.dataset.isTab ?? "memory"));
|
||||||
|
button.addEventListener("keydown", (event) => {
|
||||||
|
if (!["ArrowLeft", "ArrowRight", "Home", "End"].includes(event.key)) return;
|
||||||
|
event.preventDefault();
|
||||||
|
let next = index;
|
||||||
|
if (event.key === "ArrowLeft") next = (index - 1 + tabs.length) % tabs.length;
|
||||||
|
if (event.key === "ArrowRight") next = (index + 1) % tabs.length;
|
||||||
|
if (event.key === "Home") next = 0;
|
||||||
|
if (event.key === "End") next = tabs.length - 1;
|
||||||
|
tabs[next]?.focus();
|
||||||
|
activateTab(tabs[next]?.dataset.isTab ?? "memory");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
const memoryPresets = JSON.parse(root.dataset.memoryPresets ?? "[]");
|
||||||
|
let memoryMode = "gqa";
|
||||||
|
const updateMemory = () => {
|
||||||
|
const params = number("[data-memory-params]") * 1e9;
|
||||||
|
const weightBits = number("[data-memory-weight-bits]");
|
||||||
|
const layers = number("[data-memory-layers]");
|
||||||
|
const heads = number("[data-memory-kv-heads]");
|
||||||
|
const dim = number("[data-memory-head-dim]");
|
||||||
|
const kvBits = number("[data-memory-kv-bits]");
|
||||||
|
const context = number("[data-memory-context]");
|
||||||
|
const concurrency = number("[data-memory-concurrency]");
|
||||||
|
const block = number("[data-memory-block]");
|
||||||
|
const hbm = number("[data-memory-hbm]");
|
||||||
|
const weights = params * weightBits / 8;
|
||||||
|
const kvPerToken = layers * 2 * heads * dim * kvBits / 8;
|
||||||
|
const request = kvPerToken * context;
|
||||||
|
const active = weights + request * concurrency;
|
||||||
|
const capacity = hbm * 2 ** 30;
|
||||||
|
const remainder = capacity - active;
|
||||||
|
const padded = Math.ceil(context / block) * block;
|
||||||
|
const waste = (padded - context) * kvPerToken * concurrency;
|
||||||
|
set("[data-memory-weight]", formatBytes(weights));
|
||||||
|
set("[data-memory-kv-token]", formatBytes(kvPerToken));
|
||||||
|
set("[data-memory-request]", formatBytes(request));
|
||||||
|
set("[data-memory-active]", formatBytes(active));
|
||||||
|
set("[data-memory-waste]", waste === 0 ? "0 B" : formatBytes(waste));
|
||||||
|
set("[data-memory-fit]", remainder >= 0 ? `${formatBytes(remainder)} left` : `${formatBytes(Math.abs(remainder))} over`);
|
||||||
|
set("[data-memory-status]", remainder >= capacity * 0.15
|
||||||
|
? "能装下；仍需为激活、kernel 与通信留空间"
|
||||||
|
: remainder >= 0
|
||||||
|
? "边缘可装：真实 runtime 很可能把它推过界"
|
||||||
|
: "OOM：必须切分权重、压缩状态或减少活动 Token");
|
||||||
|
const preset = memoryPresets.find((item: { id: string }) => item.id === memoryMode);
|
||||||
|
set("[data-memory-note]", preset?.note ?? "量纲教学配置");
|
||||||
|
const dark = query<HTMLElement>("[data-memory-fit]")?.parentElement;
|
||||||
|
dark?.classList.toggle("danger", remainder < 0);
|
||||||
|
};
|
||||||
|
queryAll<HTMLInputElement | HTMLSelectElement>(".memory-controls input, .memory-controls select")
|
||||||
|
.forEach((input) => input.addEventListener("input", updateMemory));
|
||||||
|
queryAll<HTMLButtonElement>("[data-memory-preset]").forEach((button) => {
|
||||||
|
button.addEventListener("click", () => {
|
||||||
|
const preset = memoryPresets.find((item: { id: string }) => item.id === button.dataset.memoryPreset);
|
||||||
|
if (!preset) return;
|
||||||
|
memoryMode = preset.id;
|
||||||
|
queryAll<HTMLButtonElement>("[data-memory-preset]").forEach((item) => item.classList.toggle("active", item === button));
|
||||||
|
const fields: [string, keyof typeof preset][] = [
|
||||||
|
["[data-memory-params]", "params"],
|
||||||
|
["[data-memory-weight-bits]", "weightBits"],
|
||||||
|
["[data-memory-layers]", "layers"],
|
||||||
|
["[data-memory-kv-heads]", "kvHeads"],
|
||||||
|
["[data-memory-head-dim]", "headDim"],
|
||||||
|
["[data-memory-kv-bits]", "kvBits"],
|
||||||
|
];
|
||||||
|
fields.forEach(([selector, key]) => {
|
||||||
|
const input = query<HTMLInputElement | HTMLSelectElement>(selector);
|
||||||
|
if (input) input.value = String(preset[key]);
|
||||||
|
});
|
||||||
|
updateMemory();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
updateMemory();
|
||||||
|
|
||||||
|
let phaseStrategy = "static";
|
||||||
|
const updatePhase = () => {
|
||||||
|
const mix = number("[data-phase-mix]") / 100;
|
||||||
|
const prompt = number("[data-phase-prompt]");
|
||||||
|
const output = number("[data-phase-output]");
|
||||||
|
const concurrency = number("[data-phase-concurrency]");
|
||||||
|
const chunk = number("[data-phase-chunk]");
|
||||||
|
const network = number("[data-phase-network]");
|
||||||
|
set("[data-phase-mix-label]", `${Math.round(mix * 100)}%`);
|
||||||
|
set("[data-phase-concurrency-label]", `${concurrency} requests`);
|
||||||
|
const promptScale = Math.sqrt(prompt / 1024);
|
||||||
|
const outputScale = Math.log2(output / 128 + 1);
|
||||||
|
let ttft = 220 * promptScale * (1 + mix * 1.7) * (1 + concurrency / 180);
|
||||||
|
let tpot = 24 * outputScale * (1 + mix * 2.1) * (1 + concurrency / 150);
|
||||||
|
let throughput = 1500 * Math.sqrt(concurrency / 16) / (1 + mix * 0.55);
|
||||||
|
let stall = 16 + mix * 61;
|
||||||
|
let transfer = 0;
|
||||||
|
if (phaseStrategy === "continuous") {
|
||||||
|
ttft *= 0.76; tpot *= 0.65; throughput *= 1.42; stall *= 0.58;
|
||||||
|
}
|
||||||
|
if (phaseStrategy === "chunked") {
|
||||||
|
const chunkFactor = Math.max(0.72, Math.min(1.22, Math.sqrt(chunk / 1024)));
|
||||||
|
ttft *= 0.84 * chunkFactor;
|
||||||
|
tpot *= 0.42 + 0.11 * chunkFactor;
|
||||||
|
throughput *= 1.48 / Math.max(0.9, chunkFactor);
|
||||||
|
stall *= 0.31 * chunkFactor;
|
||||||
|
}
|
||||||
|
if (phaseStrategy === "pd") {
|
||||||
|
transfer = (400 / network - 1) * 65 + prompt / 8192 * (400 / network) * 18;
|
||||||
|
ttft = ttft * 0.58 + Math.max(8, transfer);
|
||||||
|
tpot = tpot * 0.43 + Math.max(0, transfer / 28);
|
||||||
|
throughput *= network >= 100 ? 1.58 : 0.78;
|
||||||
|
stall = stall * 0.23 + Math.max(0, transfer / 5);
|
||||||
|
}
|
||||||
|
const ttftSlo = 1800;
|
||||||
|
const tpotSlo = 120;
|
||||||
|
const sloFactor = Math.max(0, Math.min(1, (ttftSlo / ttft) * (tpotSlo / tpot)));
|
||||||
|
const goodput = throughput * Math.min(1, sloFactor);
|
||||||
|
set("[data-phase-ttft]", `${Math.round(ttft)} ms`);
|
||||||
|
set("[data-phase-tpot]", `${format.format(tpot)} ms`);
|
||||||
|
set("[data-phase-throughput]", `${Math.round(throughput).toLocaleString()} tok/s`);
|
||||||
|
set("[data-phase-goodput]", `${Math.round(goodput).toLocaleString()} tok/s`);
|
||||||
|
set("[data-phase-stall]", `${Math.min(99, Math.round(stall))}%`);
|
||||||
|
set("[data-phase-status]", ttft <= ttftSlo && tpot <= tpotSlo ? "SLO protected in this toy load" : "SLO miss: throughput is not goodput");
|
||||||
|
const bottleneck = phaseStrategy === "pd" && network <= 25
|
||||||
|
? "KV transfer：慢网络吃掉 P/D 解耦收益"
|
||||||
|
: phaseStrategy === "static" && mix > 0.35
|
||||||
|
? "Head-of-line blocking：长 prefill 卡住 decode"
|
||||||
|
: mix > 0.65
|
||||||
|
? "长 prompt 的 compute 与排队"
|
||||||
|
: "Decode memory bandwidth 与调度间隙";
|
||||||
|
set("[data-phase-bottleneck]", bottleneck);
|
||||||
|
set("[data-phase-link]", phaseStrategy === "pd" ? `KV TRANSFER / ${network} Gb/s` : phaseStrategy === "chunked" ? `INTERLEAVE / ${chunk} TOKENS` : "SHARED SCHEDULER");
|
||||||
|
set("[data-phase-evidence]", phaseStrategy === "pd"
|
||||||
|
? "Splitwise、DistServe、Mooncake 说明 P/D 可独立扩容，但网络和 KV 传输必须入账。"
|
||||||
|
: phaseStrategy === "chunked"
|
||||||
|
? "Sarathi 用 chunked prefill 缓和 prefill 对 decode 的干扰；切片大小仍有取舍。"
|
||||||
|
: "Clockwork、Orca、vLLM / PagedAttention 提供调度与连续批处理的历史坐标。");
|
||||||
|
const prefillMeter = query<HTMLElement>("[data-phase-prefill-meter]");
|
||||||
|
const decodeMeter = query<HTMLElement>("[data-phase-decode-meter]");
|
||||||
|
if (prefillMeter) prefillMeter.style.width = `${Math.min(100, 25 + promptScale * 13 + mix * 30)}%`;
|
||||||
|
if (decodeMeter) decodeMeter.style.width = `${Math.min(100, 25 + outputScale * 14 + concurrency / 3)}%`;
|
||||||
|
};
|
||||||
|
queryAll<HTMLButtonElement>("[data-phase-strategy]").forEach((button) => {
|
||||||
|
button.addEventListener("click", () => {
|
||||||
|
phaseStrategy = button.dataset.phaseStrategy ?? "static";
|
||||||
|
queryAll<HTMLButtonElement>("[data-phase-strategy]").forEach((item) => item.classList.toggle("active", item === button));
|
||||||
|
updatePhase();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
queryAll<HTMLInputElement | HTMLSelectElement>(".phase-controls input, .phase-controls select")
|
||||||
|
.forEach((input) => input.addEventListener("input", updatePhase));
|
||||||
|
updatePhase();
|
||||||
|
|
||||||
|
const specPresets = JSON.parse(root.dataset.specPresets ?? "[]");
|
||||||
|
let specMode = "k3";
|
||||||
|
const updateSpec = () => {
|
||||||
|
const k = number("[data-spec-k]");
|
||||||
|
const a = number("[data-spec-a]") / 100;
|
||||||
|
const draftCost = number("[data-spec-draft]") / 100;
|
||||||
|
const verifyCost = number("[data-spec-verify]") / 100;
|
||||||
|
set("[data-spec-k-label]", String(k));
|
||||||
|
set("[data-spec-a-label]", a.toFixed(2));
|
||||||
|
set("[data-spec-draft-label]", `${draftCost.toFixed(2)}×`);
|
||||||
|
set("[data-spec-verify-label]", `${verifyCost.toFixed(2)}×`);
|
||||||
|
const expected = a === 1 ? k + 1 : (1 - a ** (k + 1)) / (1 - a);
|
||||||
|
const acceptedDraft = expected - 1;
|
||||||
|
const time = k * draftCost + verifyCost;
|
||||||
|
const speedup = expected / time;
|
||||||
|
const waste = Math.max(0, k - acceptedDraft);
|
||||||
|
set("[data-spec-expected]", `${expected.toFixed(2)} tokens`);
|
||||||
|
set("[data-spec-speedup]", `${speedup.toFixed(2)}×`);
|
||||||
|
set("[data-spec-waste]", `${waste.toFixed(2)} / ${k}`);
|
||||||
|
set("[data-spec-status]", speedup >= 1.8 ? "STRONG TOY GAIN" : speedup >= 1 ? "MARGINAL GAIN" : "NO GAIN");
|
||||||
|
set("[data-spec-formula]", `E = (1 − ${a.toFixed(2)}^${k + 1}) / (1 − ${a.toFixed(2)})`);
|
||||||
|
set("[data-spec-evidence]", specMode === "k3"
|
||||||
|
? "K3 报告：MTP 初始化 EAGLE-3 风格 draft，融合第 1、4、最后 AttnRes，并展开 7 步；数值结果仍是本实验假设。"
|
||||||
|
: "Leviathan / Chen 给出保分布的推测采样主线；Medusa 与 EAGLE 系列改变草稿来源。");
|
||||||
|
const strip = query<HTMLElement>("[data-spec-strip]");
|
||||||
|
if (strip) {
|
||||||
|
strip.innerHTML = "";
|
||||||
|
for (let index = 0; index < k; index += 1) {
|
||||||
|
const item = document.createElement("i");
|
||||||
|
const probability = a ** (index + 1);
|
||||||
|
item.style.setProperty("--accept", String(probability));
|
||||||
|
item.classList.toggle("likely", probability >= 0.5);
|
||||||
|
item.innerHTML = `<span>T+${index + 1}</span><b>${Math.round(probability * 100)}%</b>`;
|
||||||
|
strip.append(item);
|
||||||
|
}
|
||||||
|
const bonus = document.createElement("i");
|
||||||
|
bonus.className = "bonus";
|
||||||
|
bonus.innerHTML = "<span>BONUS</span><b>target</b>";
|
||||||
|
strip.append(bonus);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
queryAll<HTMLButtonElement>("[data-spec-preset]").forEach((button) => {
|
||||||
|
button.addEventListener("click", () => {
|
||||||
|
const preset = specPresets.find((item: [string]) => item[0] === button.dataset.specPreset);
|
||||||
|
if (!preset) return;
|
||||||
|
specMode = preset[0];
|
||||||
|
queryAll<HTMLButtonElement>("[data-spec-preset]").forEach((item) => item.classList.toggle("active", item === button));
|
||||||
|
const values = [
|
||||||
|
["[data-spec-k]", preset[2]],
|
||||||
|
["[data-spec-a]", Math.round(preset[3] * 100)],
|
||||||
|
["[data-spec-draft]", Math.round(preset[4] * 100)],
|
||||||
|
["[data-spec-verify]", Math.round(preset[5] * 100)],
|
||||||
|
];
|
||||||
|
values.forEach(([selector, value]) => {
|
||||||
|
const input = query<HTMLInputElement>(String(selector));
|
||||||
|
if (input) input.value = String(value);
|
||||||
|
});
|
||||||
|
updateSpec();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
queryAll<HTMLInputElement>(".spec-controls input").forEach((input) => input.addEventListener("input", updateSpec));
|
||||||
|
updateSpec();
|
||||||
|
|
||||||
|
const updateFleet = () => {
|
||||||
|
const prefix = number("[data-fleet-prefix]");
|
||||||
|
const increment = number("[data-fleet-increment]");
|
||||||
|
const hit = number("[data-fleet-hit]") / 100;
|
||||||
|
const clusters = number("[data-fleet-clusters]");
|
||||||
|
const burst = number("[data-fleet-burst]") / 100;
|
||||||
|
const routing = query<HTMLSelectElement>("[data-fleet-routing]")?.value ?? "affinity";
|
||||||
|
const admission = query<HTMLSelectElement>("[data-fleet-admission]")?.value ?? "budget";
|
||||||
|
const failure = query<HTMLInputElement>("[data-fleet-failure]")?.checked ?? false;
|
||||||
|
set("[data-fleet-hit-label]", `${Math.round(hit * 100)}%`);
|
||||||
|
set("[data-fleet-clusters-label]", `${clusters} clusters`);
|
||||||
|
set("[data-fleet-burst-label]", `${Math.round(burst * 100)}%`);
|
||||||
|
const effectiveHit = failure ? hit * 0.18 : hit;
|
||||||
|
const affinityBonus = routing === "affinity" ? 1 : 0.56;
|
||||||
|
const avoided = prefix * effectiveHit * affinityBonus;
|
||||||
|
const skew = routing === "affinity" ? 1 + hit * 1.45 + burst * 0.55 : 1 + burst * 0.18;
|
||||||
|
let rejected = admission === "budget"
|
||||||
|
? Math.round(Math.max(0, burst * 42 + (prefix + increment) / 40000 - clusters * 2.3))
|
||||||
|
: Math.round(Math.max(0, burst * 8 - clusters * 0.25));
|
||||||
|
rejected = Math.min(72, rejected);
|
||||||
|
const admitted = 100 - rejected;
|
||||||
|
const protectedSlo = admission === "budget"
|
||||||
|
? burst < 0.93 || rejected > 18
|
||||||
|
: burst < 0.48 && prefix < 200000;
|
||||||
|
const recompute = failure ? prefix * hit * 0.82 : prefix * (1 - hit) * 0.08;
|
||||||
|
set("[data-fleet-avoided]", `${formatTokens(avoided)} tokens`);
|
||||||
|
set("[data-fleet-skew]", `${skew.toFixed(2)}× hottest`);
|
||||||
|
set("[data-fleet-admitted]", String(admitted));
|
||||||
|
set("[data-fleet-rejected]", String(rejected));
|
||||||
|
set("[data-fleet-short-slo]", protectedSlo ? "PROTECTED" : "VIOLATED");
|
||||||
|
set("[data-fleet-state]", failure ? "SECONDARY RE-PREFILL after atomic invalidation" : protectedSlo ? "budget keeps short requests moving" : "long burst consumes short-request budget");
|
||||||
|
set("[data-fleet-recompute]", failure ? `${formatTokens(recompute)} tokens / FAILED PRIMARY` : `${formatTokens(recompute)} background tokens`);
|
||||||
|
set("[data-fleet-route-label]", routing === "affinity" ? "CACHE KEY + LOAD" : "ROUND ROBIN");
|
||||||
|
set("[data-fleet-evidence]", failure
|
||||||
|
? "K3 报告描述 pin / copy / atomic invalidation；故障命中不能继续被当作有效状态。"
|
||||||
|
: "K3 报告给出 cache-aware affinity 与 budget admission；图中的比例是教学假设。");
|
||||||
|
const stack = query<HTMLElement>("[data-fleet-cluster-stack]");
|
||||||
|
if (stack) {
|
||||||
|
stack.innerHTML = "";
|
||||||
|
const visible = Math.min(8, clusters);
|
||||||
|
for (let index = 0; index < visible; index += 1) {
|
||||||
|
const item = document.createElement("i");
|
||||||
|
const load = routing === "affinity" && index === 1 ? Math.min(98, 38 * skew) : 28 + ((index * 17) % 35);
|
||||||
|
item.style.setProperty("--load", `${load}%`);
|
||||||
|
item.classList.toggle("failed", failure && index === 1);
|
||||||
|
item.innerHTML = `<span>C${index + 1}</span><b>${failure && index === 1 ? "X" : Math.round(load) + "%"}</b>`;
|
||||||
|
stack.append(item);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
query<HTMLElement>("[data-fleet-short-slo]")?.parentElement?.classList.toggle("danger", !protectedSlo);
|
||||||
|
};
|
||||||
|
queryAll<HTMLInputElement | HTMLSelectElement>(".fleet-controls input, .fleet-controls select, [data-fleet-failure]")
|
||||||
|
.forEach((input) => input.addEventListener("input", updateFleet));
|
||||||
|
query<HTMLButtonElement>("[data-fleet-k3]")?.addEventListener("click", () => {
|
||||||
|
const values: [string, string][] = [
|
||||||
|
["[data-fleet-prefix]", "400000"],
|
||||||
|
["[data-fleet-increment]", "4096"],
|
||||||
|
["[data-fleet-hit]", "80"],
|
||||||
|
["[data-fleet-clusters]", "8"],
|
||||||
|
["[data-fleet-burst]", "60"],
|
||||||
|
["[data-fleet-routing]", "affinity"],
|
||||||
|
["[data-fleet-admission]", "budget"],
|
||||||
|
];
|
||||||
|
values.forEach(([selector, value]) => {
|
||||||
|
const input = query<HTMLInputElement | HTMLSelectElement>(selector);
|
||||||
|
if (input) input.value = value;
|
||||||
|
});
|
||||||
|
const failure = query<HTMLInputElement>("[data-fleet-failure]");
|
||||||
|
if (failure) failure.checked = false;
|
||||||
|
updateFleet();
|
||||||
|
});
|
||||||
|
updateFleet();
|
||||||
|
});
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<style>
|
||||||
|
.is-lab {
|
||||||
|
overflow: hidden;
|
||||||
|
margin: 42px 0;
|
||||||
|
border: 1px solid var(--line-strong);
|
||||||
|
background: var(--paper-raised);
|
||||||
|
box-shadow: 0 28px 70px rgba(28, 41, 59, .08);
|
||||||
|
}
|
||||||
|
.is-lab button, .is-lab select, .is-lab input { font: inherit; }
|
||||||
|
.lab-head {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1.12fr .88fr;
|
||||||
|
gap: 48px;
|
||||||
|
padding: 30px;
|
||||||
|
border-bottom: 1px solid var(--line);
|
||||||
|
background:
|
||||||
|
radial-gradient(circle at 13% 0, rgba(92, 145, 136, .22), transparent 310px),
|
||||||
|
#172b39;
|
||||||
|
color: white;
|
||||||
|
}
|
||||||
|
.lab-head > div > p,
|
||||||
|
.panel-intro span,
|
||||||
|
.metric-grid span,
|
||||||
|
.formula-card span,
|
||||||
|
.readout-row span,
|
||||||
|
.phase-pipeline span,
|
||||||
|
.fleet-map > div > span {
|
||||||
|
color: #e3a279;
|
||||||
|
font: .55rem/1.35 var(--mono);
|
||||||
|
letter-spacing: .13em;
|
||||||
|
}
|
||||||
|
.lab-head h3 { margin-top: 10px; color: white; font: 650 clamp(1.4rem, 2.5vw, 2.3rem)/1.08 var(--serif); }
|
||||||
|
.lab-head > p { align-self: end; color: #c4d0d6; font-size: .71rem; line-height: 1.72; }
|
||||||
|
.lab-tabs { display: grid; grid-template-columns: repeat(4, 1fr); border-bottom: 1px solid var(--line); background: var(--paper-deep); }
|
||||||
|
.lab-tabs button {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: auto 1fr;
|
||||||
|
gap: 4px 10px;
|
||||||
|
min-width: 0;
|
||||||
|
padding: 18px;
|
||||||
|
border: 0;
|
||||||
|
border-right: 1px solid var(--line);
|
||||||
|
background: transparent;
|
||||||
|
color: var(--ink);
|
||||||
|
text-align: left;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.lab-tabs button:last-child { border-right: 0; }
|
||||||
|
.lab-tabs button[aria-selected="true"] { background: var(--paper-raised); box-shadow: inset 0 -3px var(--copper); }
|
||||||
|
.lab-tabs span { grid-row: 1 / 3; color: var(--copper); font: .54rem var(--mono); }
|
||||||
|
.lab-tabs b { font-size: .68rem; }
|
||||||
|
.lab-tabs small { overflow: hidden; color: var(--muted); font: .47rem var(--mono); text-overflow: ellipsis; white-space: nowrap; }
|
||||||
|
.lab-panel { padding: 30px; }
|
||||||
|
.panel-intro { display: grid; grid-template-columns: 1fr 1fr; gap: 38px; align-items: end; }
|
||||||
|
.panel-intro h4 { margin-top: 8px; font: 650 clamp(1.15rem, 2vw, 1.68rem)/1.16 var(--serif); }
|
||||||
|
.panel-intro > p { color: var(--muted); font-size: .7rem; }
|
||||||
|
.preset-row { display: grid; grid-template-columns: repeat(4, 1fr); gap: 8px; margin-top: 24px; }
|
||||||
|
.preset-row button, .fleet-presets > button {
|
||||||
|
display: grid;
|
||||||
|
gap: 5px;
|
||||||
|
padding: 13px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
background: transparent;
|
||||||
|
color: var(--ink);
|
||||||
|
text-align: left;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.preset-row button.active, .fleet-presets > button.active { border-color: var(--sage); background: var(--sage-pale); box-shadow: inset 0 -2px var(--sage); }
|
||||||
|
.preset-row b, .fleet-presets b { font-size: .63rem; }
|
||||||
|
.preset-row small, .fleet-presets small { color: var(--muted); font: .48rem/1.4 var(--mono); }
|
||||||
|
.control-grid {
|
||||||
|
display: grid;
|
||||||
|
gap: 12px;
|
||||||
|
margin-top: 14px;
|
||||||
|
padding: 18px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
background: #f2eee6;
|
||||||
|
}
|
||||||
|
.memory-controls { grid-template-columns: repeat(5, 1fr); }
|
||||||
|
.phase-controls { grid-template-columns: repeat(3, 1fr); }
|
||||||
|
.spec-controls { grid-template-columns: repeat(4, 1fr); }
|
||||||
|
.fleet-controls { grid-template-columns: repeat(4, 1fr); }
|
||||||
|
.control-grid label {
|
||||||
|
display: grid;
|
||||||
|
align-content: space-between;
|
||||||
|
gap: 8px;
|
||||||
|
min-width: 0;
|
||||||
|
color: var(--muted);
|
||||||
|
font: .54rem var(--mono);
|
||||||
|
}
|
||||||
|
.control-grid output { color: var(--ink); font-weight: 700; }
|
||||||
|
.control-grid input, .control-grid select {
|
||||||
|
width: 100%;
|
||||||
|
min-width: 0;
|
||||||
|
padding: 7px;
|
||||||
|
border: 1px solid var(--line-strong);
|
||||||
|
border-radius: 2px;
|
||||||
|
background: var(--paper-raised);
|
||||||
|
color: var(--ink);
|
||||||
|
}
|
||||||
|
.control-grid input[type="range"] { padding: 0; accent-color: var(--copper); }
|
||||||
|
.formula-card {
|
||||||
|
margin-top: 16px;
|
||||||
|
padding: 20px;
|
||||||
|
border-left: 4px solid var(--copper);
|
||||||
|
background: #213746;
|
||||||
|
color: white;
|
||||||
|
}
|
||||||
|
.formula-card code { display: block; overflow-x: auto; margin: 10px 0 7px; color: white; font: 650 clamp(.72rem, 1.5vw, 1rem)/1.5 var(--mono); white-space: nowrap; }
|
||||||
|
.formula-card p { color: #bdcbd2; font-size: .57rem; }
|
||||||
|
.metric-grid {
|
||||||
|
display: grid;
|
||||||
|
gap: 1px;
|
||||||
|
margin-top: 18px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
background: var(--line);
|
||||||
|
}
|
||||||
|
.metric-grid.five { grid-template-columns: repeat(5, 1fr); }
|
||||||
|
.metric-grid.four { grid-template-columns: repeat(4, 1fr); }
|
||||||
|
.metric-grid article { min-width: 0; padding: 17px; background: var(--paper-raised); }
|
||||||
|
.metric-grid article.dark { background: #213746; color: white; }
|
||||||
|
.metric-grid article.dark b { color: white; }
|
||||||
|
.metric-grid article.danger { background: #733c35; }
|
||||||
|
.metric-grid b { display: block; overflow-wrap: anywhere; margin: 9px 0 6px; color: var(--ink-strong); font: 650 clamp(.83rem, 1.7vw, 1.14rem)/1.15 var(--serif); }
|
||||||
|
.metric-grid b em { color: inherit; font-style: normal; }
|
||||||
|
.metric-grid p { color: var(--muted); font-size: .51rem; line-height: 1.48; }
|
||||||
|
.metric-grid article.dark p { color: #bdcbd2; }
|
||||||
|
.readout-row { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; margin-top: 1px; border: 1px solid var(--line); background: var(--line); }
|
||||||
|
.readout-row.single { grid-template-columns: 1fr; }
|
||||||
|
.readout-row > div { padding: 17px; background: var(--paper-raised); }
|
||||||
|
.readout-row b { display: block; margin: 7px 0; font-size: .66rem; }
|
||||||
|
.readout-row p { color: var(--muted); font-size: .56rem; }
|
||||||
|
.phase-pipeline {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1fr auto 1fr;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
margin-top: 18px;
|
||||||
|
padding: 22px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
background: #213746;
|
||||||
|
}
|
||||||
|
.phase-pipeline article { min-height: 150px; padding: 18px; background: rgba(255,255,255,.055); color: white; }
|
||||||
|
.phase-pipeline b { display: block; margin: 14px 0 9px; color: white; font: 650 1.1rem var(--serif); }
|
||||||
|
.phase-pipeline p { color: #bdcbd2; font-size: .55rem; }
|
||||||
|
.phase-pipeline i { display: block; height: 8px; margin: 14px 0; background: rgba(255,255,255,.12); }
|
||||||
|
.phase-pipeline em { display: block; width: 50%; height: 100%; background: #e3a279; transition: width .2s ease; }
|
||||||
|
.phase-pipeline u { max-width: 120px; color: #e3a279; font: .49rem/1.5 var(--mono); text-align: center; text-decoration: none; }
|
||||||
|
.spec-strip { display: flex; gap: 5px; overflow-x: auto; margin-top: 18px; padding: 18px; border: 1px solid var(--line); background: #f2eee6; }
|
||||||
|
.spec-strip i { display: grid; flex: 1 0 60px; place-content: center; min-height: 82px; border: 1px dashed var(--line-strong); background: color-mix(in srgb, var(--sage-pale) calc(var(--accept) * 100%), white); font-style: normal; text-align: center; }
|
||||||
|
.spec-strip i.likely { border-style: solid; border-color: var(--sage); }
|
||||||
|
.spec-strip i.bonus { border-style: solid; border-color: var(--copper); background: var(--copper-pale); }
|
||||||
|
.spec-strip span { color: var(--muted); font: .47rem var(--mono); }
|
||||||
|
.spec-strip b { margin-top: 6px; font-size: .62rem; }
|
||||||
|
.fleet-presets { display: grid; grid-template-columns: 1fr 1fr; gap: 9px; margin-top: 24px; }
|
||||||
|
.failure-toggle { display: flex; align-items: center; gap: 12px; padding: 13px; border: 1px solid var(--line); cursor: pointer; }
|
||||||
|
.failure-toggle input { width: 18px; height: 18px; accent-color: #8b493f; }
|
||||||
|
.failure-toggle span { display: grid; gap: 5px; }
|
||||||
|
.fleet-map {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: .55fr auto 1.45fr;
|
||||||
|
align-items: center;
|
||||||
|
gap: 16px;
|
||||||
|
margin-top: 18px;
|
||||||
|
padding: 22px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
background: #213746;
|
||||||
|
color: white;
|
||||||
|
}
|
||||||
|
.request-stack { display: grid; grid-template-columns: repeat(5, 1fr); gap: 5px; }
|
||||||
|
.request-stack > span { grid-column: 1 / -1; }
|
||||||
|
.request-stack i { height: 46px; background: #e3a279; opacity: .55; }
|
||||||
|
.fleet-map > u { max-width: 100px; color: #e3a279; font: .49rem/1.5 var(--mono); text-align: center; text-decoration: none; }
|
||||||
|
.cluster-stack { display: grid; grid-template-columns: repeat(4, 1fr); gap: 7px; }
|
||||||
|
.cluster-stack i {
|
||||||
|
position: relative;
|
||||||
|
display: grid;
|
||||||
|
place-content: center;
|
||||||
|
min-height: 70px;
|
||||||
|
overflow: hidden;
|
||||||
|
border: 1px solid #597180;
|
||||||
|
font-style: normal;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
.cluster-stack i::before { position: absolute; right: 0; bottom: 0; left: 0; height: var(--load); background: rgba(86, 139, 129, .35); content: ""; }
|
||||||
|
.cluster-stack i.failed { border-color: #e5796b; background: rgba(165,66,57,.4); }
|
||||||
|
.cluster-stack span, .cluster-stack b { position: relative; z-index: 1; }
|
||||||
|
.cluster-stack span { color: #b8c7cf; font: .46rem var(--mono); }
|
||||||
|
.cluster-stack b { margin-top: 4px; color: white; font-size: .6rem; }
|
||||||
|
.is-lab figcaption { padding: 16px 30px; border-top: 1px solid var(--line); color: var(--muted); font-size: .59rem; }
|
||||||
|
.is-lab figcaption b { margin-right: 8px; color: var(--copper); font-family: var(--mono); }
|
||||||
|
@media (max-width: 980px) {
|
||||||
|
.lab-head, .panel-intro { grid-template-columns: 1fr; }
|
||||||
|
.memory-controls { grid-template-columns: repeat(3, 1fr); }
|
||||||
|
.phase-controls, .fleet-controls { grid-template-columns: repeat(2, 1fr); }
|
||||||
|
.metric-grid.five { grid-template-columns: repeat(3, 1fr); }
|
||||||
|
}
|
||||||
|
@media (max-width: 720px) {
|
||||||
|
.lab-tabs, .preset-row, .spec-controls, .metric-grid.four, .metric-grid.five { grid-template-columns: repeat(2, 1fr); }
|
||||||
|
.phase-pipeline, .fleet-map { grid-template-columns: 1fr; }
|
||||||
|
.phase-pipeline u, .fleet-map > u { max-width: none; }
|
||||||
|
.cluster-stack { grid-template-columns: repeat(4, 1fr); }
|
||||||
|
}
|
||||||
|
@media (max-width: 520px) {
|
||||||
|
.lab-head, .lab-panel { padding: 20px; }
|
||||||
|
.lab-tabs, .preset-row, .control-grid, .metric-grid.four, .metric-grid.five, .readout-row, .fleet-presets { grid-template-columns: 1fr; }
|
||||||
|
.lab-tabs button { border-right: 0; border-bottom: 1px solid var(--line); }
|
||||||
|
.cluster-stack { grid-template-columns: repeat(2, 1fr); }
|
||||||
|
}
|
||||||
|
@media (prefers-reduced-motion: reduce) {
|
||||||
|
.phase-pipeline em { transition: none; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
@@ -19,6 +19,7 @@ const items = [
|
|||||||
{ id: "agents", href: "/agents/", label: "Agent" },
|
{ id: "agents", href: "/agents/", label: "Agent" },
|
||||||
{ id: "multimodal", href: "/multimodal/", label: "多模态" },
|
{ id: "multimodal", href: "/multimodal/", label: "多模态" },
|
||||||
{ id: "training-systems", href: "/training-systems/", label: "训练系统" },
|
{ id: "training-systems", href: "/training-systems/", label: "训练系统" },
|
||||||
|
{ id: "systems-inference", href: "/systems/inference/", label: "推理服务" },
|
||||||
{ id: "numerics", href: "/systems/numerics/", label: "数值" },
|
{ id: "numerics", href: "/systems/numerics/", label: "数值" },
|
||||||
{ id: "papers", href: "/papers/", label: "论文库" },
|
{ id: "papers", href: "/papers/", label: "论文库" },
|
||||||
{ id: "progress", href: "/progress/", label: "进度" },
|
{ id: "progress", href: "/progress/", label: "进度" },
|
||||||
|
|||||||
@@ -203,12 +203,12 @@ export const chapters: Chapter[] = [
|
|||||||
title: "推理服务与低成本部署",
|
title: "推理服务与低成本部署",
|
||||||
kicker: "INFERENCE",
|
kicker: "INFERENCE",
|
||||||
question: "模型训练完以后，怎样让千万人用得起？",
|
question: "模型训练完以后，怎样让千万人用得起？",
|
||||||
summary: "拆解 KV Cache、PagedAttention、批处理、推测解码、PD 解耦、前缀缓存与集群调度。",
|
summary: "用十八本账拆开显存、KV Cache、PagedAttention、连续批处理、P/D 解耦、量化、推测解码与集群调度，并重点追踪 DeepSeek V2→V4 与 Mooncake→K3。",
|
||||||
status: "queued",
|
status: "published",
|
||||||
progress: 12,
|
progress: 78,
|
||||||
papers: 20,
|
papers: 62,
|
||||||
prerequisites: ["07", "08", "09"],
|
prerequisites: ["07", "08", "09"],
|
||||||
highlights: ["vLLM", "Mooncake", "KDA 缓存"],
|
highlights: ["十八本服务账", "DeepSeek / Kimi 谱系", "四联实验"],
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
number: "15",
|
number: "15",
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ export type PaperTopic =
|
|||||||
| "推理"
|
| "推理"
|
||||||
| "Agent"
|
| "Agent"
|
||||||
| "多模态"
|
| "多模态"
|
||||||
|
| "推理服务"
|
||||||
| "评测";
|
| "评测";
|
||||||
|
|
||||||
export interface Paper {
|
export interface Paper {
|
||||||
@@ -2891,6 +2892,368 @@ export const papers: Paper[] = [
|
|||||||
spotlight: "Kimi",
|
spotlight: "Kimi",
|
||||||
verified: true,
|
verified: true,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
year: 2020,
|
||||||
|
title: "Clockwork: Predictable and Efficient Deep Learning Inference",
|
||||||
|
url: "https://www.usenix.org/conference/osdi20/presentation/gujarati",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "用可预测执行、集中式调度和显式模型缓存控制 DNN 服务尾延迟。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2022,
|
||||||
|
title: "ZeRO-Inference: Democratizing Massive Model Inference",
|
||||||
|
url: "https://arxiv.org/abs/2207.00032",
|
||||||
|
topics: ["推理服务", "训练系统", "低精度"],
|
||||||
|
contribution: "把巨量权重卸载到 CPU / NVMe，并通过分层预取和并行减少 GPU 容量门槛。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2022,
|
||||||
|
title: "Orca: A Distributed Serving System for Transformer-Based Generative Models",
|
||||||
|
url: "https://www.usenix.org/conference/osdi22/presentation/yu",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "以 iteration-level scheduling 和 selective batching 奠定连续批处理主线。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2022,
|
||||||
|
title: "Petals: Collaborative Inference and Fine-tuning of Large Models",
|
||||||
|
url: "https://arxiv.org/abs/2209.01188",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "让参与者跨互联网协作托管模型层，暴露异构、不可靠网络下的服务边界。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2022,
|
||||||
|
title: "Fast Inference from Transformers via Speculative Decoding",
|
||||||
|
url: "https://arxiv.org/abs/2211.17192",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "用便宜 draft model 提议多个 Token，再由目标模型并行验收且保持目标分布。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "Accelerating Large Language Model Decoding with Speculative Sampling",
|
||||||
|
url: "https://arxiv.org/abs/2302.01318",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "独立建立保分布的推测采样算法，减少昂贵模型的串行调用轮数。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU",
|
||||||
|
url: "https://arxiv.org/abs/2303.06865",
|
||||||
|
topics: ["推理服务", "训练系统", "低精度"],
|
||||||
|
contribution: "联合 GPU、CPU 与磁盘 offload，并以搜索策略优化吞吐导向的单卡大模型推理。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving",
|
||||||
|
url: "https://arxiv.org/abs/2302.11665",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "把统计复用、模型并行和模型放置结合，用于突发式大模型工作负载。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "FastServe: Fast Distributed Inference Serving for Large Language Models",
|
||||||
|
url: "https://arxiv.org/abs/2305.05920",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "借鉴多级反馈队列和抢占，降低自回归请求的 head-of-line blocking 与尾延迟。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification",
|
||||||
|
url: "https://arxiv.org/abs/2305.09781",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "用树状候选和并行验证扩展推测解码，使一次目标调用覆盖多条草稿分支。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "Efficient Memory Management for Large Language Model Serving with PagedAttention",
|
||||||
|
url: "https://arxiv.org/abs/2309.06180",
|
||||||
|
topics: ["推理服务", "长上下文", "训练系统"],
|
||||||
|
contribution: "vLLM 把 KV Cache 分页，支持非连续分配、共享与 copy-on-write。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving",
|
||||||
|
url: "https://arxiv.org/abs/2310.07240",
|
||||||
|
topics: ["推理服务", "长上下文", "低精度"],
|
||||||
|
contribution: "压缩并流式传输 KV Cache，以降低远端上下文加载延迟。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "SGLang: Efficient Execution of Structured Language Model Programs",
|
||||||
|
url: "https://arxiv.org/abs/2312.07104",
|
||||||
|
topics: ["推理服务", "Agent", "长上下文"],
|
||||||
|
contribution: "用 RadixAttention 自动复用共享前缀，并为结构化生成提供语言与运行时。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "Splitwise: Efficient Generative LLM Inference Using Phase Splitting",
|
||||||
|
url: "https://arxiv.org/abs/2311.18677",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "将 prompt processing 与 token generation 拆到适配的硬件池并独立调度。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "Prompt Cache: Modular Attention Reuse for Low-Latency Inference",
|
||||||
|
url: "https://arxiv.org/abs/2311.04934",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "用模块化 prompt schema 标注可复用片段并缓存中间注意力状态。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving",
|
||||||
|
url: "https://arxiv.org/abs/2401.09670",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "以满足 TTFT / TPOT SLO 的 goodput 为目标，分离并独立扩容 prefill 与 decode。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads",
|
||||||
|
url: "https://arxiv.org/abs/2401.10774",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "在目标模型上增加多个 decoding heads，构造树状候选并减少独立 draft 成本。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty",
|
||||||
|
url: "https://arxiv.org/abs/2401.15077",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "在 feature 层自回归预测并结合已知 Token，提升草稿准确率与推测解码速度。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization",
|
||||||
|
url: "https://arxiv.org/abs/2401.18079",
|
||||||
|
topics: ["推理服务", "低精度", "长上下文"],
|
||||||
|
contribution: "按通道量化 key、按 token 量化 value，并分离 outlier 以压缩长上下文 KV。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache",
|
||||||
|
url: "https://arxiv.org/abs/2402.02750",
|
||||||
|
topics: ["推理服务", "低精度", "长上下文"],
|
||||||
|
contribution: "利用 key / value 不同分布采用非对称 2-bit 量化，并保留近期高精度 residual。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "Hydragen: High-Throughput LLM Inference with Shared Prefixes",
|
||||||
|
url: "https://arxiv.org/abs/2402.05099",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "将共享前缀与独有后缀注意力拆开，跨请求批量复用前缀计算。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition",
|
||||||
|
url: "https://arxiv.org/abs/2402.15220",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "以前缀树组织共享 KV chunks，并用两阶段分区提高内存与计算复用。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve",
|
||||||
|
url: "https://arxiv.org/abs/2403.02310",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "用 chunked prefill 与 stall-free scheduling 缓和长 prefill 对 decode 的干扰。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving",
|
||||||
|
url: "https://arxiv.org/abs/2405.04532",
|
||||||
|
topics: ["推理服务", "低精度"],
|
||||||
|
contribution: "联合 W4A8KV4、数据布局和硬件友好 kernel 优化端到端服务。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion",
|
||||||
|
url: "https://arxiv.org/abs/2405.16444",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "融合独立缓存的文本块，并选择性重算跨块依赖以接近完整 prefill。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "Llumnix: Dynamic Scheduling for Large Language Model Serving",
|
||||||
|
url: "https://arxiv.org/abs/2406.03243",
|
||||||
|
topics: ["推理服务", "训练系统"],
|
||||||
|
contribution: "用跨实例实时迁移重平衡请求、处理碎片并支持主动容错。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool",
|
||||||
|
url: "https://arxiv.org/abs/2406.17565",
|
||||||
|
topics: ["推理服务", "长上下文", "训练系统"],
|
||||||
|
contribution: "把独立 KV 缓存池连接到弹性 prefill / decode 服务，形成存算分离架构。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "Preble: Efficient Distributed Prompt Scheduling for LLM Serving",
|
||||||
|
url: "https://arxiv.org/abs/2407.00023",
|
||||||
|
topics: ["推理服务", "长上下文", "训练系统"],
|
||||||
|
contribution: "在分布式 prompt 服务中联合前缀复用、实例负载与路由决策。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision",
|
||||||
|
url: "https://arxiv.org/abs/2407.08608",
|
||||||
|
topics: ["推理服务", "低精度", "长上下文"],
|
||||||
|
contribution: "面向 Hopper 用异步、warp specialization 与低精度流水进一步优化精确注意力。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees",
|
||||||
|
url: "https://arxiv.org/abs/2406.16858",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "根据上下文置信度动态调整草稿树，而不是固定候选拓扑。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2025,
|
||||||
|
title: "FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving",
|
||||||
|
url: "https://arxiv.org/abs/2501.01005",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "提供面向多种 KV 布局、batch 形状与生成阶段的可组合 attention / sampling kernel。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2025,
|
||||||
|
title: "EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test",
|
||||||
|
url: "https://arxiv.org/abs/2503.01840",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "直接预测 Token，并融合低、中、高层特征构造更强的推测草稿。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2025,
|
||||||
|
title: "DeepGEMM",
|
||||||
|
url: "https://github.com/deepseek-ai/DeepGEMM",
|
||||||
|
topics: ["推理服务", "低精度", "训练系统"],
|
||||||
|
contribution: "DeepSeek 官方开源 FP8 与低精度 GEMM kernel 库，覆盖 dense 与 MoE 形状。",
|
||||||
|
spotlight: "DeepSeek",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2025,
|
||||||
|
title: "FlashMLA",
|
||||||
|
url: "https://github.com/deepseek-ai/FlashMLA",
|
||||||
|
topics: ["推理服务", "长上下文", "训练系统"],
|
||||||
|
contribution: "DeepSeek 官方面向 Multi-head Latent Attention prefill / decode 的高效 kernel。",
|
||||||
|
spotlight: "DeepSeek",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "REST: Retrieval-Based Speculative Decoding",
|
||||||
|
url: "https://arxiv.org/abs/2311.08252",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "从历史语料检索连续 Token 构造免训练草稿，再由目标模型并行验证。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "Lookahead Decoding: Parallel Decoding without a Draft Model or Data Store",
|
||||||
|
url: "https://arxiv.org/abs/2402.02057",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "用 Jacobi 迭代并行发现并验证 n-gram，无需独立草稿模型。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding",
|
||||||
|
url: "https://arxiv.org/abs/2404.16710",
|
||||||
|
topics: ["推理服务", "推理"],
|
||||||
|
contribution: "训练早退层作为同一模型的草稿，再由剩余层验证，形成自推测解码。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models",
|
||||||
|
url: "https://arxiv.org/abs/2306.14048",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "识别并保留 attention heavy hitters，在有限预算下淘汰其余 KV。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "StreamingLLM: Efficient Streaming Language Models with Attention Sinks",
|
||||||
|
url: "https://arxiv.org/abs/2309.17453",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "保留初始 attention sinks 与近期窗口，使有限缓存支持持续流式文本。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "SnapKV: LLM Knows What You are Looking for Before Generation",
|
||||||
|
url: "https://arxiv.org/abs/2404.14469",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "用 prompt 末端 observation window 选择各头重要位置，压缩长 prompt KV。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling",
|
||||||
|
url: "https://arxiv.org/abs/2406.02069",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "依据层间信息聚合现象，为不同层分配递减的 KV 容量预算。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management",
|
||||||
|
url: "https://arxiv.org/abs/2406.19707",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "提前预测关键 KV，并从 CPU 内存选择性加载以减少 GPU 状态容量。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference",
|
||||||
|
url: "https://arxiv.org/abs/2406.10774",
|
||||||
|
topics: ["推理服务", "长上下文"],
|
||||||
|
contribution: "利用 page 级统计按 query 选择相关 KV pages，减少长上下文 decode 读取。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2024,
|
||||||
|
title: "BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems",
|
||||||
|
url: "https://arxiv.org/abs/2401.17644",
|
||||||
|
topics: ["推理服务", "训练系统", "评测"],
|
||||||
|
contribution: "公开真实服务 trace，揭示突发、长度和模型混合对容量规划的影响。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
year: 2023,
|
||||||
|
title: "TensorRT-LLM",
|
||||||
|
url: "https://github.com/NVIDIA/TensorRT-LLM",
|
||||||
|
topics: ["推理服务", "低精度", "训练系统"],
|
||||||
|
contribution: "NVIDIA 官方 LLM 推理运行时，整合低精度 kernel、并行、KV Cache 与动态批处理。",
|
||||||
|
verified: true,
|
||||||
|
},
|
||||||
];
|
];
|
||||||
|
|
||||||
export const paperTopics: PaperTopic[] = [
|
export const paperTopics: PaperTopic[] = [
|
||||||
@@ -2907,5 +3270,6 @@ export const paperTopics: PaperTopic[] = [
|
|||||||
"推理",
|
"推理",
|
||||||
"Agent",
|
"Agent",
|
||||||
"多模态",
|
"多模态",
|
||||||
|
"推理服务",
|
||||||
"评测",
|
"评测",
|
||||||
];
|
];
|
||||||
|
|||||||
+29
-1
@@ -17,6 +17,7 @@ const routes: Record<string, string> = {
|
|||||||
agents: "/agents/",
|
agents: "/agents/",
|
||||||
multimodal: "/multimodal/",
|
multimodal: "/multimodal/",
|
||||||
"training-systems": "/training-systems/",
|
"training-systems": "/training-systems/",
|
||||||
|
"systems/inference": "/systems/inference/",
|
||||||
"systems/numerics": "/systems/numerics/",
|
"systems/numerics": "/systems/numerics/",
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -96,6 +97,7 @@ const paths = [
|
|||||||
<a class="button" href="/agents/">Agent 专题</a>
|
<a class="button" href="/agents/">Agent 专题</a>
|
||||||
<a class="button" href="/multimodal/">原生多模态专题</a>
|
<a class="button" href="/multimodal/">原生多模态专题</a>
|
||||||
<a class="button" href="/training-systems/">训练系统专题</a>
|
<a class="button" href="/training-systems/">训练系统专题</a>
|
||||||
|
<a class="button" href="/systems/inference/">推理服务专题</a>
|
||||||
<a class="button" href="/systems/numerics/">数值与优化专题</a>
|
<a class="button" href="/systems/numerics/">数值与优化专题</a>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -106,7 +108,7 @@ const paths = [
|
|||||||
<div class="hero-stats">
|
<div class="hero-stats">
|
||||||
<div><b>16</b><span>核心专题</span></div>
|
<div><b>16</b><span>核心专题</span></div>
|
||||||
<div><b>151</b><span>K3 报告来源</span></div>
|
<div><b>151</b><span>K3 报告来源</span></div>
|
||||||
<div><b>355</b><span>关键论文索引</span></div>
|
<div><b>400</b><span>关键论文索引</span></div>
|
||||||
<div><b>47p</b><span>K3 技术报告</span></div>
|
<div><b>47p</b><span>K3 技术报告</span></div>
|
||||||
</div>
|
</div>
|
||||||
</aside>
|
</aside>
|
||||||
@@ -120,6 +122,22 @@ const paths = [
|
|||||||
|
|
||||||
<section class="section compact release-section" id="new-chapters">
|
<section class="section compact release-section" id="new-chapters">
|
||||||
<div class="release-grid">
|
<div class="release-grid">
|
||||||
|
<a class="release-card inference-release" href="/systems/inference/">
|
||||||
|
<div>
|
||||||
|
<p class="eyebrow"><span>NEW / CHAPTER 14</span> MEMORY · TIME · FLEET</p>
|
||||||
|
<h2>推理优化不是一句“更快”：显存、首字、字间、网络与 SLO 是五本不同的账</h2>
|
||||||
|
<p>
|
||||||
|
用十八本账从 KV Cache、PagedAttention、连续批处理、chunked prefill、量化和推测解码，
|
||||||
|
走到 DeepSeek V2→V4 的异构状态谱系，以及 Mooncake→Kimi K3 的混合缓存、亲和路由与预算准入。
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<dl>
|
||||||
|
<div><dt>LINEAGE</dt><dd>2019 → 2026</dd></div>
|
||||||
|
<div><dt>NODES</dt><dd>62 个一手节点</dd></div>
|
||||||
|
<div><dt>LAB</dt><dd>显存 · P/D · 推测 · 集群</dd></div>
|
||||||
|
</dl>
|
||||||
|
<span class="release-arrow" aria-hidden="true">从一个 Token 进入完整服务集群 →</span>
|
||||||
|
</a>
|
||||||
<a class="release-card multimodal-release" href="/multimodal/">
|
<a class="release-card multimodal-release" href="/multimodal/">
|
||||||
<div>
|
<div>
|
||||||
<p class="eyebrow"><span>NEW / CHAPTER 13</span> PIXELS · TOKENS · NATIVE MULTIMODALITY</p>
|
<p class="eyebrow"><span>NEW / CHAPTER 13</span> PIXELS · TOKENS · NATIVE MULTIMODALITY</p>
|
||||||
@@ -545,6 +563,7 @@ const paths = [
|
|||||||
transition: transform 180ms ease, border-color 180ms ease;
|
transition: transform 180ms ease, border-color 180ms ease;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.inference-release,
|
||||||
.agent-release,
|
.agent-release,
|
||||||
.alignment-release,
|
.alignment-release,
|
||||||
.transformer-release,
|
.transformer-release,
|
||||||
@@ -558,6 +577,14 @@ const paths = [
|
|||||||
min-height: 510px;
|
min-height: 510px;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.inference-release {
|
||||||
|
background:
|
||||||
|
radial-gradient(circle at 82% 18%, rgba(35, 86, 84, 0.22), transparent 31%),
|
||||||
|
radial-gradient(circle at 60% 76%, rgba(159, 91, 52, 0.13), transparent 27%),
|
||||||
|
repeating-linear-gradient(90deg, transparent 0 66px, rgba(35, 86, 84, 0.045) 66px 67px),
|
||||||
|
var(--paper-raised);
|
||||||
|
}
|
||||||
|
|
||||||
.agent-release {
|
.agent-release {
|
||||||
background:
|
background:
|
||||||
radial-gradient(circle at 82% 18%, rgba(56, 91, 128, 0.2), transparent 30%),
|
radial-gradient(circle at 82% 18%, rgba(56, 91, 128, 0.2), transparent 30%),
|
||||||
@@ -688,6 +715,7 @@ const paths = [
|
|||||||
padding-bottom: 76px;
|
padding-bottom: 76px;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.inference-release,
|
||||||
.alignment-release,
|
.alignment-release,
|
||||||
.transformer-release,
|
.transformer-release,
|
||||||
.foundation-release,
|
.foundation-release,
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ const workstreams = [
|
|||||||
{ label: "稀疏计算与 MoE", value: 74, next: "补充真实集群 traces 与专家特化案例" },
|
{ label: "稀疏计算与 MoE", value: 74, next: "补充真实集群 traces 与专家特化案例" },
|
||||||
{ label: "长上下文专题", value: 72, next: "加入更多论文逐图笔记与真实模型配置对比" },
|
{ label: "长上下文专题", value: 72, next: "加入更多论文逐图笔记与真实模型配置对比" },
|
||||||
{ label: "大规模训练系统", value: 71, next: "补真实集群 traces、故障案例与精确 topology 配置" },
|
{ label: "大规模训练系统", value: 71, next: "补真实集群 traces、故障案例与精确 topology 配置" },
|
||||||
|
{ label: "推理服务与低成本部署", value: 78, next: "补真实 GPU kernel / workload traces、功耗与跨框架复现" },
|
||||||
{ label: "数值精度、优化器与稳定性", value: 75, next: "加入真实 kernel 吞吐、长程训练 traces 与逐图论文精读" },
|
{ label: "数值精度、优化器与稳定性", value: 75, next: "加入真实 kernel 吞吐、长程训练 traces 与逐图论文精读" },
|
||||||
{ label: "引用与事实检查", value: 57, next: "自动化外链复查与来源等级扩展" },
|
{ label: "引用与事实检查", value: 57, next: "自动化外链复查与来源等级扩展" },
|
||||||
{ label: "开源与部署", value: 100, next: "每轮保留不可变镜像、提交与回滚点" },
|
{ label: "开源与部署", value: 100, next: "每轮保留不可变镜像、提交与回滚点" },
|
||||||
@@ -47,7 +48,7 @@ const workstreams = [
|
|||||||
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
||||||
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
||||||
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
||||||
<div><dt>UPDATED</dt><dd>2026-07-29 07:46 CST</dd></div>
|
<div><dt>UPDATED</dt><dd>2026-07-29 08:34 CST</dd></div>
|
||||||
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
|
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
|
||||||
</dl>
|
</dl>
|
||||||
</div>
|
</div>
|
||||||
@@ -57,7 +58,7 @@ const workstreams = [
|
|||||||
<div class="section-heading">
|
<div class="section-heading">
|
||||||
<div>
|
<div>
|
||||||
<p class="eyebrow"><span>01</span> WORKSTREAMS</p>
|
<p class="eyebrow"><span>01</span> WORKSTREAMS</p>
|
||||||
<h2>十八条工作流同时推进，但不混淆“有页面”和“已核验”</h2>
|
<h2>十九条工作流同时推进，但不混淆“有页面”和“已核验”</h2>
|
||||||
</div>
|
</div>
|
||||||
<p class="section-lead">
|
<p class="section-lead">
|
||||||
内容首版优先打通全局脉络；随后每轮迭代选择一个专题推进到论文/工程层，并做独立事实复核。
|
内容首版优先打通全局脉络；随后每轮迭代选择一个专题推进到论文/工程层，并做独立事实复核。
|
||||||
@@ -94,8 +95,8 @@ const workstreams = [
|
|||||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||||
<article><span>✓</span><h3>16 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
<article><span>✓</span><h3>16 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||||
<article><span>✓</span><h3>四十三个原创交互视图</h3><p>K3、语言模型前史、Transformer、DeepSeek、长上下文、MoE、推理、Agent、多模态，以及训练系统、Scaling、数据工程、数值和 Alignment 专题。</p></article>
|
<article><span>✓</span><h3>四十七个原创交互视图</h3><p>K3、语言模型前史、Transformer、DeepSeek、长上下文、MoE、推理、Agent、多模态，以及训练系统、推理服务、Scaling、数据工程、数值和 Alignment 专题。</p></article>
|
||||||
<article><span>✓</span><h3>十四篇首版长文</h3><p>K3、语言模型前史、Transformer、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统与数值优化专题。</p></article>
|
<article><span>✓</span><h3>十五篇首版长文</h3><p>K3、语言模型前史、Transformer、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务与数值优化专题。</p></article>
|
||||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>Scaling Laws 深度专题</h3><p>九张账、29 个一手节点、DeepSeek/Kimi 双谱系与曲面—部署—复用—涌现四联实验。</p></article>
|
<article><span>✓</span><h3>Scaling Laws 深度专题</h3><p>九张账、29 个一手节点、DeepSeek/Kimi 双谱系与曲面—部署—复用—涌现四联实验。</p></article>
|
||||||
@@ -108,7 +109,8 @@ const workstreams = [
|
|||||||
<article><span>✓</span><h3>指令微调与人类偏好深度专题</h3><p>十二张账、44 个一手节点、DeepSeek/Kimi 后训练双谱系，以及 SFT—RM—PPO/DPO—配方四联实验。</p></article>
|
<article><span>✓</span><h3>指令微调与人类偏好深度专题</h3><p>十二张账、44 个一手节点、DeepSeek/Kimi 后训练双谱系，以及 SFT—RM—PPO/DPO—配方四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>工具使用与长程 Agent 深度专题</h3><p>十四张账、52 个一手节点、DeepSeek/Kimi Agent 双谱系，以及循环—工具契约—可靠性—长程 RL 四联实验。</p></article>
|
<article><span>✓</span><h3>工具使用与长程 Agent 深度专题</h3><p>十四张账、52 个一手节点、DeepSeek/Kimi Agent 双谱系，以及循环—工具契约—可靠性—长程 RL 四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>原生多模态深度专题</h3><p>十六张账、55 个一手节点、DeepSeek 三分支、Kimi 三代 MoonViT，以及 Token—连接器—光学压缩—视觉闭环四联实验。</p></article>
|
<article><span>✓</span><h3>原生多模态深度专题</h3><p>十六张账、55 个一手节点、DeepSeek 三分支、Kimi 三代 MoonViT，以及 Token—连接器—光学压缩—视觉闭环四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>355 篇关键论文索引</h3><p>新增 ResNet、ALIGN、NaViT、DeepSeek-VL2、Janus、DeepSeek-OCR、Vision-R1 等 41 个多模态节点。</p></article>
|
<article><span>✓</span><h3>推理服务与低成本部署深度专题</h3><p>十八本账、62 个一手节点、DeepSeek V2→V4 与 Mooncake→K3 双谱系，以及显存—阶段—推测—集群四联实验。</p></article>
|
||||||
|
<article><span>✓</span><h3>400 篇关键论文索引</h3><p>新增 vLLM、SGLang、DistServe、Sarathi、FlashInfer、EAGLE-3、DeepGEMM、FlashMLA 等 45 个推理服务节点。</p></article>
|
||||||
<article><span>✓</span><h3>公开仓库与自托管发布</h3><p>源码公开到 git.k1412.top，网站由不可变镜像、Compose Manager 与 HTTPS 交付。</p></article>
|
<article><span>✓</span><h3>公开仓库与自托管发布</h3><p>源码公开到 git.k1412.top，网站由不可变镜像、Compose Manager 与 HTTPS 交付。</p></article>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
@@ -135,6 +137,7 @@ const workstreams = [
|
|||||||
<div><span>P1</span><strong>Alignment 二轮深化</strong><p>真实偏好分歧 → RM 长度偏置 → PPO/DPO 小模型复现</p><em>数据案例 + 可复现实验</em></div>
|
<div><span>P1</span><strong>Alignment 二轮深化</strong><p>真实偏好分歧 → RM 长度偏置 → PPO/DPO 小模型复现</p><em>数据案例 + 可复现实验</em></div>
|
||||||
<div><span>P1</span><strong>Agent 二轮深化</strong><p>真实环境 traces → cross-harness ablation → Agent RL 曲线与提示注入案例</p><em>运行证据 + 安全案例库</em></div>
|
<div><span>P1</span><strong>Agent 二轮深化</strong><p>真实环境 traces → cross-harness ablation → Agent RL 曲线与提示注入案例</p><em>运行证据 + 安全案例库</em></div>
|
||||||
<div><span>P1</span><strong>原生多模态二轮</strong><p>真实视觉 Token traces → 跨分辨率 / connector 消融 → OCR 与视觉 Agent 安全失败案例</p><em>运行证据 + 逐图笔记</em></div>
|
<div><span>P1</span><strong>原生多模态二轮</strong><p>真实视觉 Token traces → 跨分辨率 / connector 消融 → OCR 与视觉 Agent 安全失败案例</p><em>运行证据 + 逐图笔记</em></div>
|
||||||
|
<div><span>P1</span><strong>推理服务二轮</strong><p>真实 GPU kernel / workload traces → 功耗与成本 → 跨 vLLM / SGLang / TensorRT-LLM 复现</p><em>可复现实测 + 成本账</em></div>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
@@ -189,6 +192,8 @@ const workstreams = [
|
|||||||
<div><time>2026-07-29</time><b>原生多模态按五层管道与十六张账组织</b><p>像素、视觉塔、压缩 / connector、主干与输出 / 工具闭环分开定位；“原生”再拆成数据、目标、优化、输入输出与 Agent 五维。</p></div>
|
<div><time>2026-07-29</time><b>原生多模态按五层管道与十六张账组织</b><p>像素、视觉塔、压缩 / connector、主干与输出 / 工具闭环分开定位；“原生”再拆成数据、目标、优化、输入输出与 Agent 五维。</p></div>
|
||||||
<div><time>2026-07-29</time><b>DeepSeek 多模态永久保留三分支</b><p>VL/VL2 的理解、Janus 的统一生成、OCR 的光学压缩不画成错误的单向代际谱系。</p></div>
|
<div><time>2026-07-29</time><b>DeepSeek 多模态永久保留三分支</b><p>VL/VL2 的理解、Janus 的统一生成、OCR 的光学压缩不画成错误的单向代际谱系。</p></div>
|
||||||
<div><time>2026-07-29</time><b>光学压缩实验分开报告值、教学插值与证据外区域</b><p>DeepSeek-OCR 的 <10× / 20× 锚点标成作者报告；中间只做显式教学插值，超过范围不外推。</p></div>
|
<div><time>2026-07-29</time><b>光学压缩实验分开报告值、教学插值与证据外区域</b><p>DeepSeek-OCR 的 <10× / 20× 锚点标成作者报告；中间只做显式教学插值，超过范围不外推。</p></div>
|
||||||
|
<div><time>2026-07-29</time><b>推理服务按十八本账组织</b><p>权重、增长状态、分配、阶段、batch、cache、kernel、推测、网络、路由、故障与经济性不再压成单一 tokens/s。</p></div>
|
||||||
|
<div><time>2026-07-29</time><b>DeepSeek 与 Kimi 服务谱系按状态对象重建</b><p>MLA→V4 异构状态与 Mooncake→KDA→K3 混合缓存分开说明；作者报告、精确公式和教学估算使用不同标签。</p></div>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,981 @@
|
|||||||
|
---
|
||||||
|
import BaseLayout from "@/layouts/BaseLayout.astro";
|
||||||
|
import InferenceServingLab from "@/components/InferenceServingLab.astro";
|
||||||
|
|
||||||
|
const toc = [
|
||||||
|
["00", "compass", "先拆成十八本账"],
|
||||||
|
["01", "stack", "一条请求的全栈"],
|
||||||
|
["02", "metrics", "延迟、吞吐与 Goodput"],
|
||||||
|
["03", "clock", "请求时钟"],
|
||||||
|
["04", "weights", "权重显存账"],
|
||||||
|
["05", "kv", "KV Cache 数学"],
|
||||||
|
["06", "states", "MHA / GQA / MLA / KDA"],
|
||||||
|
["07", "paging", "PagedAttention"],
|
||||||
|
["08", "prefill", "Prefill 阶段"],
|
||||||
|
["09", "decode", "Decode 阶段"],
|
||||||
|
["10", "batching", "连续批处理"],
|
||||||
|
["11", "chunked", "Chunked prefill"],
|
||||||
|
["12", "kernels", "Kernel 与算子"],
|
||||||
|
["13", "prefix", "前缀缓存"],
|
||||||
|
["14", "pd", "Prefill / Decode 解耦"],
|
||||||
|
["15", "quant", "量化不是一种开关"],
|
||||||
|
["16", "speculative", "推测解码"],
|
||||||
|
["17", "parallel", "并行与放置"],
|
||||||
|
["18", "moe", "MoE 推理"],
|
||||||
|
["19", "slo", "调度、SLO 与 Goodput"],
|
||||||
|
["20", "fleet", "集群与故障"],
|
||||||
|
["21", "deepseek-map", "DeepSeek 服务谱系"],
|
||||||
|
["22", "deepseek-v23", "V2 → V3"],
|
||||||
|
["23", "deepseek-v4", "V3.2 → V4"],
|
||||||
|
["24", "kimi-map", "Kimi 服务谱系"],
|
||||||
|
["25", "mooncake", "Mooncake"],
|
||||||
|
["26", "k3-cache", "K3 混合状态缓存"],
|
||||||
|
["27", "k3-spec", "K3 推测与准入"],
|
||||||
|
["28", "lab", "四联交互实验"],
|
||||||
|
["29", "checklist", "审计一套推理系统"],
|
||||||
|
["↳", "papers", "62 个关键节点"],
|
||||||
|
];
|
||||||
|
|
||||||
|
const ledgers = [
|
||||||
|
["Q1 / PRODUCT", "请求形状", "输入、输出、并发与复用到底长什么样？", "聊天、代码 Agent、离线批处理和 1M context 的最优系统不相同。"],
|
||||||
|
["Q2 / SLO", "体验契约", "用户在等首字，还是在等完整答案？", "TTFT、TPOT、E2E、deadline 与可用性要分别记录。"],
|
||||||
|
["Q3 / WEIGHT", "权重", "模型本体怎样装进设备？", "精度、分片、复制、offload、专家放置与 runtime workspace。"],
|
||||||
|
["Q4 / STATE", "增长状态", "每多一个 Token，要多存什么？", "标准 KV、压缩 latent、线性注意力状态与混合缓存。"],
|
||||||
|
["Q5 / ALLOC", "内存分配", "空间够，为什么仍会 OOM？", "连续块、内部碎片、生命周期、共享与 copy-on-write。"],
|
||||||
|
["Q6 / PREFILL", "预填充", "长 prompt 怎样变成可复用状态？", "通常算力密集；长度、chunk、缓存命中与排队共同决定首字。"],
|
||||||
|
["Q7 / DECODE", "逐步生成", "一个 Token 为什么也很贵？", "通常受权重与状态读带宽限制；batch 能摊权重，却增加竞争。"],
|
||||||
|
["Q8 / BATCH", "批处理", "不同长度请求怎样同车？", "静态、连续、iteration-level 与 token-budget scheduling。"],
|
||||||
|
["Q9 / CACHE", "复用", "哪些旧计算可以安全命中？", "前缀哈希、page 对齐、版本、租户隔离、淘汰与失效。"],
|
||||||
|
["Q10 / KERNEL", "算子", "理论 FLOPs 为何不是实测延迟？", "IO、融合、shape、量化布局、通信与硬件利用率。"],
|
||||||
|
["Q11 / SPEC", "推测", "能否用便宜猜测减少串行步数？", "验收率、draft 成本、并行验证与回滚必须同账。"],
|
||||||
|
["Q12 / QUANT", "精度", "少几个 bit 省在哪里，又伤在哪里？", "权重、激活、KV 与通信不是同一个量化对象。"],
|
||||||
|
["Q13 / PARALLEL", "并行", "一份请求跨多少设备？", "TP、PP、DP、EP、CP 的通信、冗余与尾延迟不同。"],
|
||||||
|
["Q14 / MOE", "专家", "激活参数少，为何部署仍然难？", "所有权重仍需放置，路由造成 all-to-all、热点与小 batch GEMM。"],
|
||||||
|
["Q15 / NETWORK", "网络", "拆开阶段后，要搬多少状态？", "P/D 解耦、远端缓存和专家通信把网络变成一等资源。"],
|
||||||
|
["Q16 / ROUTE", "路由", "送到空闲实例，还是有缓存的实例？", "负载、缓存亲和、数据局部性与租户边界需要共同优化。"],
|
||||||
|
["Q17 / FAILURE", "故障", "缓存或 worker 消失时，状态还可信吗？", "pin、复制、原子失效、重算、降级和幂等必须定义。"],
|
||||||
|
["Q18 / ECON", "经济", "便宜究竟按什么分母？", "每 Token 成本之外还要看 SLO goodput、利用率、功耗与拒绝率。"],
|
||||||
|
];
|
||||||
|
|
||||||
|
const waves = [
|
||||||
|
["2019–20", "先把单请求变得可预测", "MQA · Clockwork", "减少增长状态；用可预测执行与集中调度替代黑盒式服务。"],
|
||||||
|
["2022", "批处理进入 iteration 粒度", "Orca · ZeRO-Inference · Petals", "请求完成即可补位；权重 offload 与分布式协作拓宽部署边界。"],
|
||||||
|
["2022–23", "少走串行步数、少搬字节", "Speculative Decoding · FlexGen · GPTQ · SmoothQuant", "推测、offload 与量化成为三条互补主线。"],
|
||||||
|
["2023", "KV Cache 成为显式系统对象", "vLLM · PagedAttention · SGLang · Prompt Cache", "page、前缀树和结构化语言运行时把状态复用提升为核心抽象。"],
|
||||||
|
["2024", "按阶段、上下文与 SLO 拆服务", "Splitwise · DistServe · Sarathi · Mooncake", "prefill、decode、缓存与网络开始独立扩容和调度。"],
|
||||||
|
["2024–25", "Kernel、压缩与推测共同优化", "FlashInfer · QServe · KVQuant · EAGLE-3", "性能来自算法、数据布局、精度和运行时的协同，不是单点技巧。"],
|
||||||
|
["2024–26", "开放前沿模型反向定义系统", "DeepSeek V2→V4 · Kimi K2→K3", "MLA/KDA、MoE、FP8/FP4、分离服务与混合状态缓存一起设计。"],
|
||||||
|
];
|
||||||
|
|
||||||
|
const paperChain = [
|
||||||
|
["2019", "Fast Transformer Decoding / MQA", "https://arxiv.org/abs/1911.02150", "让多个 query heads 共享 K/V，直接减少 decode 状态读取。"],
|
||||||
|
["2020", "Megatron-LM", "https://arxiv.org/abs/1909.08053", "模型内张量并行成为超大 Transformer 推理放置的基础坐标。"],
|
||||||
|
["2020", "Clockwork", "https://www.usenix.org/conference/osdi20/presentation/gujarati", "以可预测执行、集中调度和模型缓存控制服务延迟。"],
|
||||||
|
["2021", "ZeRO", "https://arxiv.org/abs/1910.02054", "分片状态的思想延伸到超大模型训练与服务。"],
|
||||||
|
["2021", "DeepSpeed-MoE", "https://arxiv.org/abs/2201.05596", "把稀疏专家推理的并行、通信与延迟作为联合问题。"],
|
||||||
|
["2022", "FlashAttention", "https://arxiv.org/abs/2205.14135", "用 IO-aware tiling 减少 HBM 往返，而不是近似注意力。"],
|
||||||
|
["2022", "ZeRO-Inference", "https://arxiv.org/abs/2207.00032", "把巨量权重从 GPU 卸载并以带宽感知方式分层取用。"],
|
||||||
|
["2022", "Orca", "https://www.usenix.org/conference/osdi22/presentation/yu", "iteration-level scheduling 与 selective batching 形成连续批处理主线。"],
|
||||||
|
["2022", "Petals", "https://arxiv.org/abs/2209.01188", "跨互联网协作服务大模型，展示异构和不可靠网络边界。"],
|
||||||
|
["2022", "LLM.int8()", "https://arxiv.org/abs/2208.07339", "混合精度处理 outlier，让超大模型 8-bit 推理更稳健。"],
|
||||||
|
["2022", "GPTQ", "https://arxiv.org/abs/2210.17323", "逐层二阶近似的 post-training weight quantization。"],
|
||||||
|
["2022", "SmoothQuant", "https://arxiv.org/abs/2211.10438", "把激活离群难度迁移到权重，实现 W8A8。"],
|
||||||
|
["2022", "Speculative Decoding", "https://arxiv.org/abs/2211.17192", "用便宜草稿与目标模型验收，保持目标分布不变。"],
|
||||||
|
["2023", "Speculative Sampling", "https://arxiv.org/abs/2302.01318", "独立建立保分布的推测采样与验收机制。"],
|
||||||
|
["2023", "AlpaServe", "https://arxiv.org/abs/2302.11665", "把统计复用与模型并行放置用于突发式服务。"],
|
||||||
|
["2023", "FlexGen", "https://arxiv.org/abs/2303.06865", "GPU、CPU、磁盘三层 offload 与线性规划式策略搜索。"],
|
||||||
|
["2023", "FastServe", "https://arxiv.org/abs/2305.05920", "以抢占和跳级队列降低 autoregressive 服务尾延迟。"],
|
||||||
|
["2023", "SpecInfer", "https://arxiv.org/abs/2305.09781", "树状草稿与并行验证拓展推测解码。"],
|
||||||
|
["2023", "AWQ", "https://arxiv.org/abs/2306.00978", "保护少量显著权重的激活感知 weight-only 量化。"],
|
||||||
|
["2023", "H2O", "https://arxiv.org/abs/2306.14048", "以 heavy-hitter oracle 选择性保留 KV。"],
|
||||||
|
["2023", "FlashAttention-2", "https://arxiv.org/abs/2307.08691", "改善 work partitioning，减少非矩阵乘法开销。"],
|
||||||
|
["2023", "vLLM / PagedAttention", "https://arxiv.org/abs/2309.06180", "把 KV Cache 分页，支持近零外部碎片与共享。"],
|
||||||
|
["2023", "StreamingLLM", "https://arxiv.org/abs/2309.17453", "用 attention sinks 保持有限缓存下的流式长序列。"],
|
||||||
|
["2023", "CacheGen", "https://arxiv.org/abs/2310.07240", "压缩 KV 的网络传输以加速上下文加载。"],
|
||||||
|
["2023", "Prompt Cache", "https://arxiv.org/abs/2311.04934", "用模块化 prompt schema 复用中间状态。"],
|
||||||
|
["2023", "REST", "https://arxiv.org/abs/2311.08252", "从检索到的历史 n-gram 构造免训练草稿。"],
|
||||||
|
["2023", "Splitwise", "https://arxiv.org/abs/2311.18677", "拆分 prompt 与 token generation 阶段，按资源特征放置。"],
|
||||||
|
["2023", "SGLang", "https://arxiv.org/abs/2312.07104", "RadixAttention 与结构化生成运行时复用共享前缀。"],
|
||||||
|
["2024", "DistServe", "https://arxiv.org/abs/2401.09670", "以 goodput 为目标分离 prefill / decode 并独立扩容。"],
|
||||||
|
["2024", "Medusa", "https://arxiv.org/abs/2401.10774", "在目标模型上增加多步预测 heads，减少独立 draft 开销。"],
|
||||||
|
["2024", "EAGLE", "https://arxiv.org/abs/2401.15077", "在 feature 层自回归预测并结合 token，提高草稿准确率。"],
|
||||||
|
["2024", "KVQuant", "https://arxiv.org/abs/2401.18079", "按通道 key、按 token value 与 outlier 分离量化 KV。"],
|
||||||
|
["2024", "BurstGPT", "https://arxiv.org/abs/2401.17644", "真实服务 trace 揭示突发、长度与模型混合的工作负载。"],
|
||||||
|
["2024", "KIVI", "https://arxiv.org/abs/2402.02750", "非对称 2-bit KV 量化并保留近期 residual。"],
|
||||||
|
["2024", "Lookahead Decoding", "https://arxiv.org/abs/2402.02057", "用 Jacobi 迭代并行发现、验证 n-gram，无需 draft model。"],
|
||||||
|
["2024", "Hydragen", "https://arxiv.org/abs/2402.05099", "共享前缀请求分解注意力，提升跨序列复用。"],
|
||||||
|
["2024", "ChunkAttention", "https://arxiv.org/abs/2402.15220", "以前缀树组织 KV chunk 并共享计算。"],
|
||||||
|
["2024", "Sarathi-Serve", "https://arxiv.org/abs/2403.02310", "chunked prefill 与 stall-free batching 缓和阶段干扰。"],
|
||||||
|
["2024", "LayerSkip", "https://arxiv.org/abs/2404.16710", "训练早退层作为自推测草稿，并由后续层验证。"],
|
||||||
|
["2024", "SnapKV", "https://arxiv.org/abs/2404.14469", "从 observation window 选择重要位置压缩长 prompt KV。"],
|
||||||
|
["2024", "DeepSeek-V2", "https://arxiv.org/abs/2405.04434", "MLA 将增长 KV 压到 latent 表示，DeepSeekMoE 降低激活计算。"],
|
||||||
|
["2024", "QServe", "https://arxiv.org/abs/2405.04532", "W4A8KV4 与硬件友好 kernel 的联合推理系统。"],
|
||||||
|
["2024", "CacheBlend", "https://arxiv.org/abs/2405.16444", "融合独立缓存的文本块，并选择性重算跨块依赖。"],
|
||||||
|
["2024", "PyramidKV", "https://arxiv.org/abs/2406.02069", "按层分配递减 KV 预算。"],
|
||||||
|
["2024", "Llumnix", "https://arxiv.org/abs/2406.03243", "跨实例实时迁移请求以重平衡和避障。"],
|
||||||
|
["2024", "Quest", "https://arxiv.org/abs/2406.10774", "以 page 级查询感知稀疏选择减少长上下文 decode 读量。"],
|
||||||
|
["2024", "EAGLE-2", "https://arxiv.org/abs/2406.16858", "用上下文置信度动态调整草稿树。"],
|
||||||
|
["2024", "MemServe", "https://arxiv.org/abs/2406.17565", "把分离 KV 缓存池与弹性服务连接成存算分离架构。"],
|
||||||
|
["2024", "InfiniGen", "https://arxiv.org/abs/2406.19707", "提前预测关键 KV 并从 CPU 选择性加载。"],
|
||||||
|
["2024", "Preble", "https://arxiv.org/abs/2407.00023", "在分布式 prompt 服务中联合前缀复用与负载。"],
|
||||||
|
["2024", "Mooncake", "https://arxiv.org/abs/2407.00079", "以 KVCache-centric disaggregated architecture 服务长上下文。"],
|
||||||
|
["2024", "FlashAttention-3", "https://arxiv.org/abs/2407.08608", "面向 Hopper 异步、低精度与 warp specialization 优化 attention。"],
|
||||||
|
["2024", "DeepSeek-V3", "https://arxiv.org/abs/2412.19437", "FP8、MLA、MoE 与无损负载均衡共同改变部署形状。"],
|
||||||
|
["2025", "FlashInfer", "https://arxiv.org/abs/2501.01005", "为多形态 LLM serving 提供可组合 attention / sampling kernel。"],
|
||||||
|
["2025", "EAGLE-3", "https://arxiv.org/abs/2503.01840", "直接预测 token，并融合低中高层特征增强草稿。"],
|
||||||
|
["2025", "DeepEP", "https://github.com/deepseek-ai/DeepEP", "DeepSeek 官方 MoE dispatch / combine 通信库。"],
|
||||||
|
["2025", "DeepGEMM", "https://github.com/deepseek-ai/DeepGEMM", "DeepSeek 官方 FP8 / low-precision GEMM kernel 库。"],
|
||||||
|
["2025", "FlashMLA", "https://github.com/deepseek-ai/FlashMLA", "DeepSeek 官方面向 MLA decode / prefill 的高效 kernel。"],
|
||||||
|
["2025", "Kimi Linear", "https://arxiv.org/abs/2510.26692", "KDA 以门控 Delta Rule 形成固定大小 recurrent state。"],
|
||||||
|
["2025", "DeepSeek-V3.2", "https://arxiv.org/abs/2512.02556", "稀疏注意力、长上下文与推理训练继续压低服务成本。"],
|
||||||
|
["2026", "DeepSeek-V4", "https://arxiv.org/abs/2606.19348", "CSA/HCA/SWA 与异构状态缓存把上下文状态进一步分层。"],
|
||||||
|
["2026", "Kimi K3", "https://arxiv.org/abs/2607.24653", "69 KDA + 24 Gated MLA、page cache、EAGLE-3 draft 与预算准入联合设计。"],
|
||||||
|
];
|
||||||
|
|
||||||
|
const audits = [
|
||||||
|
["工作负载", "prompt / output 长度分布、并发、突发、前缀复用和租户隔离是否来自真实 trace？"],
|
||||||
|
["SLO", "TTFT、TPOT、E2E、deadline、可用性和拒绝率是否分别定义了分位数？"],
|
||||||
|
["分母", "报告的是峰值吞吐、完成吞吐，还是满足 SLO 的 goodput？"],
|
||||||
|
["权重", "参数量、精度、scale / metadata、workspace、复制和 offload 是否都入账？"],
|
||||||
|
["状态", "MHA/GQA/MLA/KDA 的状态公式、精度、page 与回滚口径是否明确？"],
|
||||||
|
["调度", "静态 / 连续 batch、token budget、chunk、抢占和 fairness 如何交互？"],
|
||||||
|
["缓存", "key、page 边界、版本、命中有效长度、淘汰、租户隔离与失效怎样定义？"],
|
||||||
|
["拆分", "P/D 分离后的 KV 传输字节、网络、放置、重试和失效成本是否入账？"],
|
||||||
|
["量化", "W/A/KV/通信分别多少 bit？校准、QAT、fallback 与任务退化怎样测？"],
|
||||||
|
["推测", "草稿长度、验收率、draft / verify / rollback 成本和分布正确性是否同时披露？"],
|
||||||
|
["MoE", "总权重放置、激活专家、all-to-all、热点、冗余专家与小 batch GEMM 是否入账？"],
|
||||||
|
["故障", "worker、缓存、网络和路由故障下，是重试、重算、降级还是拒绝？状态是否原子失效？"],
|
||||||
|
["可比性", "硬件、软件版本、精度、模型、请求分布、SLO 与功耗是否相同？"],
|
||||||
|
["边界", "作者报告值、教学估算、内部线上数与可复现实测是否用不同标签？"],
|
||||||
|
];
|
||||||
|
---
|
||||||
|
|
||||||
|
<BaseLayout
|
||||||
|
title="推理服务与低成本部署：从 KV Cache 到 DeepSeek V4 与 Kimi K3"
|
||||||
|
description="从显存、KV Cache、PagedAttention、连续批处理、prefill/decode、量化、推测解码与集群调度，完整走到 DeepSeek V2–V4 和 Mooncake–Kimi K3 的推理系统。"
|
||||||
|
section="systems-inference"
|
||||||
|
>
|
||||||
|
<header class="page-hero inference-hero">
|
||||||
|
<div class="page-hero-inner">
|
||||||
|
<div>
|
||||||
|
<p class="eyebrow"><span>INFERENCE / 14</span> MEMORY · TIME · FLEET</p>
|
||||||
|
<h1>模型训练完以后，<br />怎样让千万人用得起？</h1>
|
||||||
|
<p class="lead">
|
||||||
|
从一条请求的权重与 KV 字节账出发，穿过 PagedAttention、连续批处理、
|
||||||
|
chunked prefill、P/D 解耦、量化和推测解码；再重点追踪 DeepSeek
|
||||||
|
从 MLA 到 V4 异构缓存，以及 Kimi 从 Mooncake 到 K3 混合状态服务的完整谱系。
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<dl class="page-facts">
|
||||||
|
<div><dt>LEVEL</dt><dd>L0 直觉 → L3 集群</dd></div>
|
||||||
|
<div><dt>LEDGERS</dt><dd>18 本独立账</dd></div>
|
||||||
|
<div><dt>NODES</dt><dd>62 个一手节点</dd></div>
|
||||||
|
<div><dt>LAB</dt><dd>4 个交互实验</dd></div>
|
||||||
|
<div><dt>TIME</dt><dd>约 260–340 分钟</dd></div>
|
||||||
|
<div><dt>VERIFIED</dt><dd>2026-07-29</dd></div>
|
||||||
|
</dl>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<div class="report-shell">
|
||||||
|
<aside class="side-rail" aria-label="本页目录">
|
||||||
|
<p>CONTENTS / 31</p>
|
||||||
|
<ol>
|
||||||
|
{toc.map(([number, id, label]) => <li><a href={`#${id}`}><span>{number}</span>{label}</a></li>)}
|
||||||
|
</ol>
|
||||||
|
<div class="rail-note">
|
||||||
|
<b>先记住一句</b>
|
||||||
|
推理优化不是让模型“少算一点”这么简单，而是在权重、增长状态、算力、带宽、网络与体验契约之间重新分配成本。
|
||||||
|
</div>
|
||||||
|
</aside>
|
||||||
|
|
||||||
|
<article class="article">
|
||||||
|
<section class="article-section" id="compass">
|
||||||
|
<p class="eyebrow"><span>00</span> EIGHTEEN LEDGERS</p>
|
||||||
|
<h2>不要先问“哪个框架最快”，先问哪本账正在爆</h2>
|
||||||
|
<p class="lede">
|
||||||
|
同一个模型可以在离线批处理里吞吐极高，却在聊天服务中首字很慢；可以靠缓存服务
|
||||||
|
400K 代码前缀，也可能因为亲和路由把一台机器压成热点。只有把十八本账分开，系统名才有意义。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div class="prerequisite-grid">
|
||||||
|
<a href="/long-context/"><span>PREREQUISITE / 07</span><b>长上下文与高效注意力</b><p>先理解 KV、MLA、KDA 和上下文状态。</p></a>
|
||||||
|
<a href="/training-systems/"><span>PREREQUISITE / 08</span><b>大规模训练系统</b><p>并行、通信和 MoE 放置在推理端仍然存在。</p></a>
|
||||||
|
<a href="/systems/numerics/"><span>PREREQUISITE / 09</span><b>数值精度与稳定性</b><p>量化节省的是哪类字节，要从数值格式开始。</p></a>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="ledger-grid">
|
||||||
|
{ledgers.map(([code, title, question, answer]) => (
|
||||||
|
<article><span>{code}</span><h3>{title}</h3><b>{question}</b><p>{answer}</p></article>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="one-line">
|
||||||
|
<span>ONE-SENTENCE MODEL</span>
|
||||||
|
<p><b>推理服务是一间状态工厂：prefill 把 prompt 制造成可复用状态，decode 反复读取权重与状态产生下一个 Token，调度器则在 SLO 前提下决定谁先占用哪种资源。</b></p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="wave-timeline">
|
||||||
|
{waves.map(([year, title, nodes, note], index) => (
|
||||||
|
<article><div>{String(index + 1).padStart(2, "0")}</div><time>{year}</time><section><h3>{title}</h3><b>{nodes}</b><p>{note}</p></section></article>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="stack">
|
||||||
|
<p class="eyebrow"><span>01</span> REQUEST STACK</p>
|
||||||
|
<h2>用户看到一个回答，机房里经过七层</h2>
|
||||||
|
<p>
|
||||||
|
API 网关只负责把请求送进来。真正决定成本的是：怎样排队、是否命中前缀、由谁做
|
||||||
|
prefill、状态放在哪里、谁做 decode、每步调用哪些 kernel，以及流式输出能否按时送回。
|
||||||
|
</p>
|
||||||
|
<div class="request-stack">
|
||||||
|
{[
|
||||||
|
["01", "GATEWAY", "鉴权、限流、模型 / 租户选择", "失败不是 GPU 慢，而是入口排队。"],
|
||||||
|
["02", "SCHEDULER", "SLO、token budget、优先级", "决定请求何时成为 active。"],
|
||||||
|
["03", "CACHE INDEX", "前缀 hash、page、版本与租户", "命中必须同时语义有效与状态可用。"],
|
||||||
|
["04", "PREFILL", "批量处理 prompt，生成状态", "常见情形偏 compute-bound，但不是定律。"],
|
||||||
|
["05", "STATE FABRIC", "HBM / DRAM / SSD / remote KV", "容量、带宽、复制和失效都入账。"],
|
||||||
|
["06", "DECODE", "逐步采样、draft、verify", "常见情形偏 memory-bandwidth-bound。"],
|
||||||
|
["07", "STREAM", "首字、字间与最终状态", "用户体验由最慢分位数而非均值决定。"],
|
||||||
|
].map(([n, title, body, foot]) => (
|
||||||
|
<article><span>{n}</span><b>{title}</b><p>{body}</p><small>{foot}</small></article>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
<div class="boundary-grid">
|
||||||
|
<article><b>模型 FLOPs</b><i>≠</i><p>线上延迟</p></article>
|
||||||
|
<article><b>GPU 利用率</b><i>≠</i><p>用户满意</p></article>
|
||||||
|
<article><b>Cache hit</b><i>≠</i><p>有效复用</p></article>
|
||||||
|
<article><b>吞吐最高</b><i>≠</i><p>单位 goodput 最便宜</p></article>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="metrics">
|
||||||
|
<p class="eyebrow"><span>02</span> METRICS</p>
|
||||||
|
<h2>把“快”拆成五个彼此冲突的指标</h2>
|
||||||
|
<div class="metric-defs">
|
||||||
|
<article><span>TTFT</span><h3>Time To First Token</h3><p>入口排队、cache lookup、prefill、首轮调度与网络返回之和。长 prompt 首先打在这里。</p></article>
|
||||||
|
<article><span>TPOT / ITL</span><h3>Time Per Output Token</h3><p>流式输出相邻 Token 的间隔。decode 抖动会让回答“卡顿”。</p></article>
|
||||||
|
<article><span>E2E</span><h3>End-to-End Latency</h3><p>从请求进入到完整输出结束，受输出长度强烈影响。</p></article>
|
||||||
|
<article><span>THROUGHPUT</span><h3>Tokens / second</h3><p>系统总产出；若大量请求违约，峰值吞吐依然可能很漂亮。</p></article>
|
||||||
|
<article><span>GOODPUT</span><h3>SLO-qualified throughput</h3><p>同时满足 TTFT / TPOT 等约束的完成量，才可用于容量与成本决策。</p></article>
|
||||||
|
</div>
|
||||||
|
<div class="formula">
|
||||||
|
Goodput = Σ completed work × 𝟙[all SLOs satisfied] / time
|
||||||
|
<small>SLO 必须带分位数与请求类。例如“短聊天 TTFT P95 < 1 s、TPOT P99 < 80 ms”；只写平均延迟不足以复现。</small>
|
||||||
|
</div>
|
||||||
|
<div class="plain-language"><b>PLAIN LANGUAGE</b><p>餐厅一小时做 300 道菜是吞吐；其中 280 道在承诺时间内送到，才是 goodput。为了第 301 道菜让所有桌都迟到，不叫优化。</p></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="clock">
|
||||||
|
<p class="eyebrow"><span>03</span> REQUEST CLOCK</p>
|
||||||
|
<h2>一次生成包含两种完全不同的计算节奏</h2>
|
||||||
|
<div class="request-clock">
|
||||||
|
<div class="queue"><span>QUEUE</span><b>排队</b><i style="--w: 13%"></i></div>
|
||||||
|
<div class="prefill"><span>PREFILL</span><b>处理整个 prompt</b><i style="--w: 36%"></i></div>
|
||||||
|
<div class="first"><span>TTFT</span><b>首字返回</b><i style="--w: 4%"></i></div>
|
||||||
|
<div class="decode"><span>DECODE × N</span><b>读状态 → 生成 → 更新</b><i style="--w: 47%"></i></div>
|
||||||
|
</div>
|
||||||
|
<div class="phase-compare">
|
||||||
|
<article><span>PREFILL</span><h3>一次并行处理很多输入位置</h3><ul><li>矩阵更大，通常更容易吃满算力</li><li>长 prompt 延长 TTFT</li><li>产物是后续可复用的状态</li><li>可切成 chunk 与 decode 交错</li></ul></article>
|
||||||
|
<article><span>DECODE</span><h3>每一步只有新位置，循环很多次</h3><ul><li>反复读取权重与历史状态</li><li>batch 增大能摊权重读取</li><li>TPOT 决定流式体验</li><li>推测解码试图减少串行轮数</li></ul></article>
|
||||||
|
</div>
|
||||||
|
<div class="warning-note"><b>REGIME, NOT LAW</b><p>“prefill 算力受限、decode 带宽受限”是常见硬件与 shape 下的工作区间，不是所有模型、batch、长度与 kernel 都成立的物理定律。</p></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="weights">
|
||||||
|
<p class="eyebrow"><span>04</span> WEIGHT MEMORY</p>
|
||||||
|
<h2>权重是进门票，不是全部显存</h2>
|
||||||
|
<div class="formula">Weight bytes ≈ parameter count × bits / 8<small>真实部署还要加量化 scale / zero-point、embedding / head 特殊精度、临时 workspace、通信 buffer、allocator reserve 与框架开销。</small></div>
|
||||||
|
<div class="byte-bars">
|
||||||
|
<article><span>70B / BF16</span><i style="--size: 100%"></i><b>约 130.4 GiB</b><p>单卡 80 GiB 放不下权重本体。</p></article>
|
||||||
|
<article><span>70B / INT8</span><i style="--size: 50%"></i><b>约 65.2 GiB</b><p>看似可放，留给 KV 与 workspace 的余量很窄。</p></article>
|
||||||
|
<article><span>70B / INT4</span><i style="--size: 25%"></i><b>约 32.6 GiB</b><p>容量改善显著，但速度取决于 kernel 与硬件路径。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
当模型跨卡时，还要选择复制还是分片。复制提高数据并行容量，却让每个副本承担全部权重；
|
||||||
|
tensor parallel 分片矩阵，却在层内频繁通信；pipeline parallel 减少单卡权重，却引入阶段气泡和请求微批约束。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="kv">
|
||||||
|
<p class="eyebrow"><span>05</span> KV CACHE MATH</p>
|
||||||
|
<h2>上下文不是抽象长度，它是一笔逐 Token 增长的字节债</h2>
|
||||||
|
<div class="formula">
|
||||||
|
KVBytes = B × T × L × 2 × H<sub>kv</sub> × D<sub>h</sub> × bytes
|
||||||
|
<small>B 并发请求，T 已缓存位置，L 层数，2 表示 K 与 V，Hkv 为 KV heads，Dh 为每头维度。这个式子精确适用于标准 MHA / MQA / GQA 口径。</small>
|
||||||
|
</div>
|
||||||
|
<div class="kv-factor">
|
||||||
|
{[
|
||||||
|
["B", "并发", "请求数翻倍，状态近似翻倍"],
|
||||||
|
["T", "长度", "上下文翻倍，增长状态翻倍"],
|
||||||
|
["L", "层数", "每个注意力层各存一份"],
|
||||||
|
["2", "K + V", "两组历史张量"],
|
||||||
|
["Hkv", "KV heads", "MQA / GQA 直接减少此项"],
|
||||||
|
["Dh", "头维度", "每个位置每个头的宽度"],
|
||||||
|
["bytes", "精度", "FP16=2，INT8=1，4-bit≈0.5"],
|
||||||
|
].map(([symbol, title, note]) => <article><b>{symbol}</b><span>{title}</span><p>{note}</p></article>)}
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
最常见的错误是只算一个请求，或把模型参数精度当成 KV 精度。W4 不自动等于 KV4；
|
||||||
|
量化权重后留下的空间，也可能很快被长上下文和高并发吃完。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="states">
|
||||||
|
<p class="eyebrow"><span>06</span> STATE ARCHITECTURES</p>
|
||||||
|
<h2>“KV Cache”已经不是一种统一形状</h2>
|
||||||
|
<div class="state-compare">
|
||||||
|
<article><span>MHA</span><div class="head-dots many">{Array.from({length: 16}).map(() => <i></i>)}</div><h3>每个 Q 头一组 K/V</h3><p>表达直接，增长状态最大；decode 需要读更多历史字节。</p><small>状态 ∝ Hq × T</small></article>
|
||||||
|
<article><span>GQA / MQA</span><div class="head-dots few">{Array.from({length: 4}).map(() => <i></i>)}</div><h3>多个 Q 头共享 K/V</h3><p>以更少 KV heads 换容量与带宽，成为服务友好设计。</p><small>状态 ∝ Hkv × T，Hkv ≪ Hq</small></article>
|
||||||
|
<article class="deepseek"><span>DEEPSEEK MLA</span><div class="latent-box"><i></i></div><h3>缓存低维 latent</h3><p>通过低秩压缩与解耦 RoPE，避免保存完整每头 K/V。</p><small>仍随 T 增长，但每位置更小</small></article>
|
||||||
|
<article class="kimi"><span>KIMI KDA</span><div class="state-box"><i></i><i></i><i></i></div><h3>固定 recurrent state</h3><p>线性注意力用门控 Delta Rule 更新有限状态；局部精确回忆仍需混合 MLA。</p><small>核心 recurrent state 不随 T 线性增长</small></article>
|
||||||
|
</div>
|
||||||
|
<div class="warning-note"><b>DO NOT SUBSTITUTE FORMULAS</b><p>不能把 MLA 的 latent 或 KDA 的 recurrent matrix 硬塞进标准 Hkv 公式并称为精确值。先确认状态对象，再谈字节。</p></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="paging">
|
||||||
|
<p class="eyebrow"><span>07</span> PAGEDATTENTION</p>
|
||||||
|
<h2>vLLM 的关键不是“少算注意力”，而是让状态不必连续居住</h2>
|
||||||
|
<div class="paging-visual">
|
||||||
|
<section><span>CONTIGUOUS ALLOCATION</span><div class="memory-row"><i class="used w6"></i><i class="hole w2"></i><i class="used w4"></i><i class="hole w4"></i></div><p>为最大长度预留、不同寿命请求离开后留下洞；连续扩容困难。</p></section>
|
||||||
|
<u>→ PAGE TABLE →</u>
|
||||||
|
<section><span>PAGED KV</span><div class="page-row">{Array.from({length: 12}).map((_, i) => <i class:list={{used: i < 9, shared: i === 2 || i === 3}}></i>)}</div><p>逻辑 block 映射到非连续物理 page；前缀可引用共享 page，写入时 copy-on-write。</p></section>
|
||||||
|
</div>
|
||||||
|
<div class="three-gains">
|
||||||
|
<article><b>按需分配</b><p>请求每增长一个 block 才取新 page，不必按最大长度预留。</p></article>
|
||||||
|
<article><b>生命周期管理</b><p>请求结束可逐 page 回收，避免必须寻找大连续区域。</p></article>
|
||||||
|
<article><b>共享</b><p>并行采样和共同前缀可以引用同一物理 page。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
PagedAttention 解决的是内存管理和共享，不会降低 Transformer 本身的理论 FLOPs；
|
||||||
|
“近零浪费”也不等于绝对零，最后一个 block 仍有内部碎片，page table 和 kernel 也有开销。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="prefill">
|
||||||
|
<p class="eyebrow"><span>08</span> PREFILL</p>
|
||||||
|
<h2>长 prompt 是一项可排队、可缓存、可切片的生产任务</h2>
|
||||||
|
<div class="prefill-flow">
|
||||||
|
<article><span>INPUT</span><b>32K prompt</b><div class="token-line"></div><p>长度、模态与 padding 决定实际输入。</p></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><span>FORWARD</span><b>宽矩阵计算</b><div class="matrix"></div><p>每层同时处理大量位置，通常更容易利用算力。</p></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><span>OUTPUT</span><b>Layer states</b><div class="page-row mini">{Array.from({length: 8}).map(() => <i class="used"></i>)}</div><p>状态写入 HBM 或缓存层，供 decode 和后续请求读取。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
首字延迟不是单纯的 prefill kernel 时间：请求可能在 admission、batch 形成、cache lookup、
|
||||||
|
page 分配与队列中停留。长 prompt 若一次占满调度迭代，还会让已经在流式输出的请求停止前进。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="decode">
|
||||||
|
<p class="eyebrow"><span>09</span> DECODE</p>
|
||||||
|
<h2>每次只写一个新 Token，却要反复搬动整个模型与历史</h2>
|
||||||
|
<div class="decode-loop">
|
||||||
|
<div><span>01</span><b>读取权重</b><p>batch 内请求共同摊一次矩阵权重访问。</p></div><u>→</u>
|
||||||
|
<div><span>02</span><b>读取历史状态</b><p>attention 访问此前 K/V 或其他 recurrent state。</p></div><u>→</u>
|
||||||
|
<div><span>03</span><b>采样一个 Token</b><p>logits、约束、采样与停止条件。</p></div><u>↺</u>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
decode 的“算术强度”常较低：每个新位置对应的计算量有限，却需读大量字节。增加 batch
|
||||||
|
可以提升权重重用，但 batch 过大又会增加排队、KV 容量和 TPOT。系统优化目标因此是一个带 SLO 的工作点，而非无限扩大 batch。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="batching">
|
||||||
|
<p class="eyebrow"><span>10</span> CONTINUOUS BATCHING</p>
|
||||||
|
<h2>静态 batch 等最慢者；连续 batch 在每一步补位</h2>
|
||||||
|
<div class="batch-compare">
|
||||||
|
<article><span>STATIC</span><h3>整批一起开始，一起结束</h3><div class="batch-grid static"><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i></div><p>短请求完成后留下空槽，直到最长请求结束。</p></article>
|
||||||
|
<article><span>ORCA / ITERATION-LEVEL</span><h3>每次 decode 迭代重新组批</h3><div class="batch-grid dynamic"><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i><i></i></div><p>完成即补新请求，显著提高 slot 利用率；调度开销和公平性仍需管理。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
continuous batching 不是把所有请求简单堆在一起。一个服务迭代可能包含 decode token、
|
||||||
|
新请求 prefill、被抢占请求恢复和 cache transfer；真正的调度单位越来越接近“token budget + 状态资源”。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="chunked">
|
||||||
|
<p class="eyebrow"><span>11</span> CHUNKED PREFILL</p>
|
||||||
|
<h2>把 128K prompt 切开，给流式请求留出呼吸</h2>
|
||||||
|
<div class="chunk-schedule">
|
||||||
|
<span>ITERATION</span><b>1</b><b>2</b><b>3</b><b>4</b><b>5</b><b>6</b>
|
||||||
|
<span>MONOLITHIC</span><i class="prefill span4">PREFILL 128K</i><i class="decode">D</i><i class="decode">D</i>
|
||||||
|
<span>CHUNKED</span><i class="prefill">P 1</i><i class="decode">D</i><i class="prefill">P 2</i><i class="decode">D</i><i class="prefill">P 3</i><i class="decode">D</i>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
Sarathi-Serve 的核心直觉是把大 prefill 切成 chunk，与 decode 组合成更均匀的迭代。
|
||||||
|
chunk 太大仍会 stall，太小则增加调度和 kernel 开销；它改善阶段干扰，不保证所有 workload 都降低 TTFT。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="kernels">
|
||||||
|
<p class="eyebrow"><span>12</span> KERNELS</p>
|
||||||
|
<h2>算法写下 FLOPs，kernel 决定字节怎么走</h2>
|
||||||
|
<div class="kernel-stack">
|
||||||
|
{[
|
||||||
|
["ALGORITHM", "Attention / MLA / MoE / quantization", "定义数学工作与可利用结构。"],
|
||||||
|
["LAYOUT", "page、tile、group、scale、sparsity", "决定能否连续访问、复用和向量化。"],
|
||||||
|
["KERNEL", "FlashAttention · FlashInfer · FlashMLA · DeepGEMM", "融合操作，安排 warp、共享内存与异步流水。"],
|
||||||
|
["RUNTIME", "shape dispatch、graph、batch、stream", "为不同长度和并发选择实际实现。"],
|
||||||
|
["HARDWARE", "HBM、tensor core、network", "最终受容量、带宽、指令和拓扑约束。"],
|
||||||
|
].map(([title, nodes, note]) => <article><span>{title}</span><b>{nodes}</b><p>{note}</p></article>)}
|
||||||
|
</div>
|
||||||
|
<div class="deepseek-callout"><span>DEEPSEEK OPEN KERNELS</span><h3>FlashMLA · DeepGEMM · DeepEP</h3><p>分别覆盖 MLA attention、低精度 GEMM 与 MoE dispatch/combine。开源 kernel 让“模型架构如何要求系统实现”变得可检查，而不只是论文里的吞吐数字。</p></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="prefix">
|
||||||
|
<p class="eyebrow"><span>13</span> PREFIX CACHE</p>
|
||||||
|
<h2>命中不是一个布尔值，而是一段有效状态的证明</h2>
|
||||||
|
<div class="prefix-visual">
|
||||||
|
<div><span>ROOT</span><i></i></div>
|
||||||
|
<u>system prompt</u>
|
||||||
|
<div><span>SHARED 8K</span><i></i></div>
|
||||||
|
<section><u>repo A</u><div><span>A / 120K</span><i></i></div></section>
|
||||||
|
<section><u>repo B</u><div><span>B / 86K</span><i></i></div></section>
|
||||||
|
</div>
|
||||||
|
<div class="cache-contract">
|
||||||
|
<article><b>Key</b><p>模型 / adapter / tokenizer / 精度 / 内容 hash / 租户。</p></article>
|
||||||
|
<article><b>Extent</b><p>到底命中多少有效 Token，不是“有相似 prompt”。</p></article>
|
||||||
|
<article><b>Placement</b><p>page 在哪台 worker、哪层内存，搬运是否比重算便宜。</p></article>
|
||||||
|
<article><b>Validity</b><p>版本、过期、故障、写入和 copy-on-write 的原子语义。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>缓存节省 prefill compute，却消耗容量、索引、网络与路由自由度。SGLang 的 RadixAttention、Prompt Cache、Hydragen、ChunkAttention、Preble 分别从程序结构、共享计算和分布式调度推进这条主线。</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="pd">
|
||||||
|
<p class="eyebrow"><span>14</span> PREFILL / DECODE DISAGGREGATION</p>
|
||||||
|
<h2>把两种节奏拆开扩容，但必须付状态搬运费</h2>
|
||||||
|
<div class="pd-visual">
|
||||||
|
<section><span>PREFILL POOL</span><div class="gpu-row"><i></i><i></i><i></i><i></i></div><b>compute-oriented</b><p>长 prompt、宽矩阵、cache creation</p></section>
|
||||||
|
<u><b>KV / STATE FABRIC</b><small>bytes · bandwidth · queue · retry</small></u>
|
||||||
|
<section><span>DECODE POOL</span><div class="gpu-row decode"><i></i><i></i><i></i><i></i><i></i><i></i></div><b>bandwidth-oriented</b><p>持续小步、batch、streaming SLO</p></section>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
Splitwise 把 prompt 与 token generation 放到适配的机器；DistServe 以 goodput 与独立扩容为中心；
|
||||||
|
Mooncake 更进一步把 KVCache 作为存算分离系统的中心。拆分消除资源干扰，却新增 KV transfer、跨池排队和故障协调。
|
||||||
|
</p>
|
||||||
|
<div class="formula">Disaggregate only if: interference saved > state transfer + extra queueing + failure overhead<small>不是看到 prefill/decode 特征不同就自动应该拆。短 prompt、慢网络或小规模服务可能不划算。</small></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="quant">
|
||||||
|
<p class="eyebrow"><span>15</span> QUANTIZATION</p>
|
||||||
|
<h2>“4-bit 模型”至少漏掉三本精度账</h2>
|
||||||
|
<div class="quant-grid">
|
||||||
|
<article><span>W</span><h3>Weights</h3><b>GPTQ · AWQ · weight-only</b><p>省权重容量和读取；是否加速取决于反量化融合与低精度 GEMM。</p></article>
|
||||||
|
<article><span>A</span><h3>Activations</h3><b>SmoothQuant · W8A8 · FP8</b><p>离群值、累加精度和校准影响 tensor core 路径。</p></article>
|
||||||
|
<article><span>KV</span><h3>Growing state</h3><b>KVQuant · KIVI · QServe</b><p>长上下文直接获益；key/value、近期 / 远期状态可采用不同策略。</p></article>
|
||||||
|
<article><span>NET</span><h3>Communication</h3><b>dispatch · cache transfer</b><p>量化网络载荷可能省带宽，却增加转换和误差边界。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
DeepSeek-V3 将 FP8 训练 / 推理和 MLA、MoE 一起设计；K3 从 SFT 进入 MXFP4/MXFP8 QAT，
|
||||||
|
并保持非专家模块更高精度。QAT 是让模型适应目标数值路径，不等于所有层都用同一格式。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="speculative">
|
||||||
|
<p class="eyebrow"><span>16</span> SPECULATIVE DECODING</p>
|
||||||
|
<h2>目标模型一次验多步，减少最昂贵的串行轮数</h2>
|
||||||
|
<div class="spec-flow">
|
||||||
|
<article><span>DRAFT</span><div class="draft-tokens"><i>A</i><i>B</i><i>C</i><i>D</i><i>E</i></div><p>小模型、额外 heads、feature predictor、检索或自推测产生候选。</p></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><span>VERIFY IN PARALLEL</span><div class="verify-line"><i></i><i></i><i></i><i></i><i></i></div><p>目标模型并行计算候选位置，并按算法逐项验收。</p></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><span>COMMIT</span><div class="draft-tokens accepted"><i>A</i><i>B</i><i>C</i><i>×</i></div><p>提交被接受前缀，再从目标分布纠正；状态必须可回滚。</p></article>
|
||||||
|
</div>
|
||||||
|
<div class="formula">
|
||||||
|
a = Σ<sub>x</sub> min(p(x), q(x)) ·
|
||||||
|
E[tokens] = 1 + a + … + a<sup>k</sup>
|
||||||
|
<small>左式是目标分布 p 与草稿 q 的重叠质量；右式是假设每步独立、验收率固定为 a 的教学期望，不是所有实现的实测加速。</small>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
Vanilla speculative sampling 用独立小模型；Medusa 在主模型上加多步 heads；EAGLE 在 feature
|
||||||
|
层预测，EAGLE-2 动态构树，EAGLE-3 融合多层特征。速度取决于验收率、草稿成本、验证 shape 和状态回滚，草稿越长并不总越快。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="parallel">
|
||||||
|
<p class="eyebrow"><span>17</span> PARALLELISM</p>
|
||||||
|
<h2>放不下是一类问题，放得下但通信太多是另一类</h2>
|
||||||
|
<div class="parallel-grid">
|
||||||
|
<article><span>DP</span><b>Data Parallel</b><p>完整副本服务不同请求。扩吞吐自然，但每份权重都占容量。</p></article>
|
||||||
|
<article><span>TP</span><b>Tensor Parallel</b><p>层内矩阵跨卡，减少单卡权重；每层都有 collective。</p></article>
|
||||||
|
<article><span>PP</span><b>Pipeline Parallel</b><p>不同层跨阶段，降低单卡容量；气泡和逐 Token 流水复杂。</p></article>
|
||||||
|
<article><span>EP</span><b>Expert Parallel</b><p>MoE 专家跨设备，激活按路由 all-to-all。</p></article>
|
||||||
|
<article><span>CP / SP</span><b>Context / Sequence</b><p>长序列状态或计算跨设备，交换边界与聚合结果。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>服务并行还要考虑请求级容错：TP 组中一张卡失败可能使整个 replica 失效；DP 副本则较易摘除。最少 GPU 数、最优 GPU 数和可容错 GPU 数不是同一个答案。</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="moe">
|
||||||
|
<p class="eyebrow"><span>18</span> MOE INFERENCE</p>
|
||||||
|
<h2>激活参数少，不等于只需要存激活专家</h2>
|
||||||
|
<div class="moe-serving">
|
||||||
|
<div class="tokens">{Array.from({length: 12}).map((_, i) => <i>{i + 1}</i>)}</div>
|
||||||
|
<u>ROUTER</u>
|
||||||
|
<div class="experts">{Array.from({length: 8}).map((_, i) => <i class:list={{hot: i === 2 || i === 5}}>E{i + 1}</i>)}</div>
|
||||||
|
</div>
|
||||||
|
<div class="moe-debts">
|
||||||
|
<article><b>总权重</b><p>全部专家仍需放在 GPU、主存或分层存储中。</p></article>
|
||||||
|
<article><b>路由通信</b><p>Token dispatch / combine 带来 all-to-all 与拓扑敏感性。</p></article>
|
||||||
|
<article><b>热点</b><p>输入分布让少数专家拥挤；均值负载掩盖尾部。</p></article>
|
||||||
|
<article><b>小 GEMM</b><p>单专家 batch 过小，理论低 FLOPs 仍可能 memory-bound。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>DeepSeek-V3 的部署报告把 prefill 与 decode 设成完全不同的 EP 规模，并使用冗余专家；DeepEP 则为 dispatch / combine 提供高吞吐与低延迟路径。模型路由和网络拓扑必须共同设计。</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="slo">
|
||||||
|
<p class="eyebrow"><span>19</span> SCHEDULING & SLO</p>
|
||||||
|
<h2>调度器不是把 GPU 塞满，而是在过载时决定谁不被伤害</h2>
|
||||||
|
<div class="scheduler-board">
|
||||||
|
<header><span>REQUEST</span><span>PROMPT</span><span>OUTPUT</span><span>DEADLINE</span><span>DECISION</span></header>
|
||||||
|
<div><b>chat-31</b><span>1K</span><span>200</span><span>TTFT 800ms</span><em class="go">ADMIT</em></div>
|
||||||
|
<div><b>agent-08</b><span>400K hit</span><span>8K</span><span>TPOT 90ms</span><em class="go">AFFINITY</em></div>
|
||||||
|
<div><b>batch-92</b><span>128K</span><span>32K</span><span>30 min</span><em>DEFER</em></div>
|
||||||
|
<div><b>agent-77</b><span>1M miss</span><span>16K</span><span>TTFT 5s</span><em class="stop">REJECT</em></div>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
平均并发阈值看不见请求长度：1 个 1M 请求和 1 个 1K 请求都被计为“1”。
|
||||||
|
token-budget admission 将预期 prefill、KV 和 decode 工作纳入预算；过载时拒绝或延后长任务，可能提高短请求 goodput。
|
||||||
|
</p>
|
||||||
|
<div class="warning-note"><b>FAIRNESS IS A PRODUCT CHOICE</b><p>保护短请求、优先付费租户、保证老请求不饿死、为长 Agent 保留配额，都是不同策略；不能只用系统吞吐替用户做决定。</p></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="fleet">
|
||||||
|
<p class="eyebrow"><span>20</span> FLEET & FAILURE</p>
|
||||||
|
<h2>单机优化完成后，路由、热点和故障成为主问题</h2>
|
||||||
|
<div class="fleet-triangle">
|
||||||
|
<article><span>LOAD</span><b>送到最空闲实例</b><p>减少排队，却可能放弃已有 400K 前缀。</p></article>
|
||||||
|
<article><span>LOCALITY</span><b>送到有缓存实例</b><p>避免 prefill，却可能把热门 repo 挤到单点。</p></article>
|
||||||
|
<article><span>RESILIENCE</span><b>复制与可重算</b><p>多副本增加成本；不复制则故障时重算和违约。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
Llumnix 通过请求迁移重平衡实例；Preble 联合考虑前缀复用与负载；Mooncake / MemServe
|
||||||
|
把状态放进独立缓存层。真实系统还必须定义 pin、复制中读写、版本升级、原子失效和 secondary re-prefill。
|
||||||
|
</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section deepseek-section" id="deepseek-map">
|
||||||
|
<p class="eyebrow"><span>21</span> DEEPSEEK LINEAGE</p>
|
||||||
|
<h2>DeepSeek 的低成本服务不是一项技巧，而是四代共同设计</h2>
|
||||||
|
<div class="lineage-map">
|
||||||
|
<article><time>2024 / V2</time><h3>MLA + DeepSeekMoE</h3><p>从模型结构上减少每 Token KV 和激活计算。</p><span>STATE ARCHITECTURE</span></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><time>2024 / V3</time><h3>FP8 + system co-design</h3><p>训练、MoE 路由、专家通信与 P/D 部署一起设计。</p><span>FULL STACK</span></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><time>2025 / V3.2</time><h3>Sparse attention</h3><p>长上下文继续压缩 attention 工作，服务与推理训练协同。</p><span>LONG CONTEXT</span></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><time>2026 / V4</time><h3>Heterogeneous state</h3><p>CSA / HCA / SWA 把增长状态按功能与介质分层。</p><span>STATE HIERARCHY</span></article>
|
||||||
|
</div>
|
||||||
|
<div class="deepseek-callout"><span>READING KEY</span><h3>架构降成本 × 数值降字节 × kernel 提利用率 × 服务分阶段</h3><p>只复制 MLA 或只换 FP8，都不等于复制 DeepSeek 的整体经济性。</p></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section deepseek-section" id="deepseek-v23">
|
||||||
|
<p class="eyebrow"><span>22</span> DEEPSEEK V2 → V3</p>
|
||||||
|
<h2>先压每 Token 状态，再把 MoE 推理摊到两种集群</h2>
|
||||||
|
<div class="v23-grid">
|
||||||
|
<article><span>V2 / MLA</span><h3>低秩 latent 代替完整每头 K/V</h3><p>把增长缓存从“每个 KV 头的完整向量”压缩成共享 latent，并把 RoPE 部分解耦。</p></article>
|
||||||
|
<article><span>V2 / MOE</span><h3>细粒度专家与 shared experts</h3><p>总容量增加而每 Token 只激活部分专家；服务端承担专家放置与通信。</p></article>
|
||||||
|
<article><span>V3 / NUMERICS</span><h3>FP8 与累加控制</h3><p>减少权重、激活与通信字节，并以配方和 kernel 守住稳定性。</p></article>
|
||||||
|
<article><span>V3 / ROUTING</span><h3>无 auxiliary-loss 负载均衡</h3><p>模型训练中的路由平衡直接影响线上专家热点与硬件利用率。</p></article>
|
||||||
|
</div>
|
||||||
|
<div class="deployment-report">
|
||||||
|
<header><span>AUTHOR-REPORTED DEPLOYMENT SHAPE</span><b>DeepSeek-V3</b></header>
|
||||||
|
<section><article><b>PREFILL</b><strong>最少 4 nodes / 32 GPUs</strong><p>TP4 · SP / DP8 · EP32 · 32 redundant experts</p></article><article><b>DECODE</b><strong>最少 40 nodes / 320 GPUs</strong><p>TP4 · SP / DP80 · EP320；单专家 batch 通常 ≤256，偏 memory-bound</p></article></section>
|
||||||
|
<footer>这是论文报告的部署形状与局限，不是运行 V3 的普遍最低硬件要求，也不能直接外推成本倍数。</footer>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section deepseek-section" id="deepseek-v4">
|
||||||
|
<p class="eyebrow"><span>23</span> DEEPSEEK V3.2 → V4</p>
|
||||||
|
<h2>从压缩 KV 走到异构状态缓存</h2>
|
||||||
|
<div class="v4-state">
|
||||||
|
<article><span>CSA</span><b>Compressed Sparse Attention</b><p>只选择与当前 query 相关的一部分历史。</p><i class="sparse"></i></article>
|
||||||
|
<article><span>HCA</span><b>Hierarchical / compressed state</b><p>用不同粒度保留全局记忆与可检索摘要。</p><i class="hierarchy"></i></article>
|
||||||
|
<article><span>SWA</span><b>Sliding Window Attention</b><p>为局部精确依赖保留有限窗口。</p><i class="window"></i></article>
|
||||||
|
<article><span>STORAGE</span><b>HBM → host → disk</b><p>冷状态可下沉，容量变大但访问和失败路径更复杂。</p><i class="storage"></i></article>
|
||||||
|
</div>
|
||||||
|
<div class="reported-values">
|
||||||
|
<span>AUTHOR'S 1M-CONTEXT ESTIMATES / VS V3.2</span>
|
||||||
|
<article><b>V4-Pro</b><strong>27% FLOPs · 10% KV</strong></article>
|
||||||
|
<article><b>V4-Flash</b><strong>10% FLOPs · 7% KV</strong></article>
|
||||||
|
<p>这些是报告内特定配置的估算比例，不是任意工作负载的端到端延迟或成本倍数。磁盘缓存将容量收益换成 IO、预取与尾延迟风险。</p>
|
||||||
|
</div>
|
||||||
|
<p>V4 还报告了组件级 FP4 selector：特定选择器实验中 2× 加速、99.7% recall。它描述一个组件，不应写成“V4 整体 2× 且精度 99.7%”。</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section kimi-section" id="kimi-map">
|
||||||
|
<p class="eyebrow"><span>24</span> KIMI LINEAGE</p>
|
||||||
|
<h2>Kimi 的主线是把超长 Agent 历史变成可管理的状态系统</h2>
|
||||||
|
<div class="lineage-map kimi-lineage">
|
||||||
|
<article><time>2024 / MOONCAKE</time><h3>KVCache-centric</h3><p>prefill、decode 与缓存池分离，围绕长上下文复用组织系统。</p><span>DISAGGREGATION</span></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><time>2025 / KIMI K2</time><h3>Agentic workload</h3><p>长工具轨迹与代码前缀让缓存、调度和稳定流式输出更重要。</p><span>WORKLOAD</span></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><time>2025 / KIMI LINEAR</time><h3>KDA state</h3><p>以 Gated Delta Rule 形成固定 recurrent state，降低长度增长债。</p><span>ARCHITECTURE</span></article>
|
||||||
|
<u>→</u>
|
||||||
|
<article><time>2026 / K3</time><h3>Hybrid serving</h3><p>KDA + Gated MLA、page cache、speculation、affinity 与 admission 联动。</p><span>FULL STACK</span></article>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section kimi-section" id="mooncake">
|
||||||
|
<p class="eyebrow"><span>25</span> MOONCAKE</p>
|
||||||
|
<h2>把 KVCache 从 GPU 附属品提升为分布式基础设施</h2>
|
||||||
|
<div class="mooncake-visual">
|
||||||
|
<section><span>PREFILL CLUSTERS</span><div class="gpu-row"><i></i><i></i><i></i></div><p>生成 KV，并写入分布式缓存。</p></section>
|
||||||
|
<u>↕</u>
|
||||||
|
<section class="cache"><span>CONTEXT CACHE</span><div class="cache-layers"><i>HBM</i><i>DRAM</i><i>SSD</i></div><p>以带宽、容量和热度管理状态。</p></section>
|
||||||
|
<u>↕</u>
|
||||||
|
<section><span>DECODE CLUSTERS</span><div class="gpu-row decode"><i></i><i></i><i></i><i></i></div><p>读取状态并持续生成。</p></section>
|
||||||
|
</div>
|
||||||
|
<div class="reported-values mooncake-values">
|
||||||
|
<span>AUTHOR-REPORTED / KEEP THE DENOMINATOR</span>
|
||||||
|
<article><b>特定模拟场景</b><strong>最高 525% throughput improvement</strong></article>
|
||||||
|
<article><b>真实工作负载</b><strong>在 SLO 下多服务 75% requests</strong></article>
|
||||||
|
<p>两者分母和条件不同，不能合并成“Mooncake 普遍 5.25×”。论文的核心贡献是体系结构与调度方法，不是一个脱离 workload 的营销倍数。</p>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section kimi-section" id="k3-cache">
|
||||||
|
<p class="eyebrow"><span>26</span> K3 HYBRID STATE CACHE</p>
|
||||||
|
<h2>69 个 KDA 固定状态，与 24 个 MLA 增长缓存一起管理</h2>
|
||||||
|
<div class="k3-block">
|
||||||
|
<div class="kda-layers"><span>× 3 KDA</span><i></i><i></i><i></i></div>
|
||||||
|
<u>+</u>
|
||||||
|
<div class="mla-layer"><span>× 1 GATED MLA</span><i></i></div>
|
||||||
|
<u>REPEAT</u>
|
||||||
|
<div class="totals"><b>69 KDA</b><b>24 MLA</b><small>报告配置另列 96 attention heads</small></div>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
KDA 用固定 recurrent state 处理远距离历史，MLA 保留随上下文增长的精确注意力缓存。
|
||||||
|
因为两者生命周期与回滚语义不同，K3 不能只复用传统 KV page manager。
|
||||||
|
</p>
|
||||||
|
<div class="page-semantics">
|
||||||
|
<article><span>PHYSICAL PAGE</span><b>6,144 tokens</b><p>报告示例中的物理管理粒度。</p></article>
|
||||||
|
<article><span>HASH BLOCK</span><b>512 tokens</b><p>更细粒度索引内容匹配。</p></article>
|
||||||
|
<article><span>MATCH</span><b>2,800 tokens</b><p>内容匹配长度不自动等于有效状态长度。</p></article>
|
||||||
|
<article class="accent"><span>VALID HIT</span><b>2,560 tokens</b><p>按 hash block 对齐后可安全复用的示例。</p></article>
|
||||||
|
</div>
|
||||||
|
<p>报告还定义并发 pin、copy 与原子 invalidation：复制中不能回收源页，失效后不能继续向路由器宣称命中。这里的难点是分布式状态一致性，不只是 hash 查表。</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section kimi-section" id="k3-spec">
|
||||||
|
<p class="eyebrow"><span>27</span> K3 SPECULATION & ADMISSION</p>
|
||||||
|
<h2>草稿、回滚、缓存亲和与预算准入共同守住长 Agent</h2>
|
||||||
|
<div class="k3-spec-grid">
|
||||||
|
<article><span>DRAFT</span><h3>MTP → EAGLE-3 style</h3><p>预训练 MTP 初始化 draft；feature fusion 读取第 1、第 4 和最终 AttnRes，训练时展开 7 步。</p></article>
|
||||||
|
<article><span>OBJECTIVE</span><h3>Loss with acceptance mass</h3><p><code>L<sub>LK</sub> = −log Σ min(p, q)</code> 直接鼓励草稿分布与目标分布重叠。</p></article>
|
||||||
|
<article><span>ROLLBACK</span><h3>Replay accepted state</h3><p>KDA 不为每条草稿复制完整 recurrent state；缓存投影输入，按被接受前缀重放更新。</p></article>
|
||||||
|
<article><span>FLEET</span><h3>Affinity + budget</h3><p>优先路由到已有状态的实例，同时用 token budget 防止超长任务淹没短请求。</p></article>
|
||||||
|
</div>
|
||||||
|
<div class="coding-trace"><span>TYPICAL REPORT EXAMPLE</span><b>400K cached code prefix</b><u>+</u><b>4K new increment</b><p>缓存价值来自迭代式 Agent 反复使用巨型 repo 状态；如果路由丢失亲和性，就会为很小增量重复付出巨大 prefill。</p></div>
|
||||||
|
<p>K3 的 QAT 从 SFT 阶段进入 MXFP4 / MXFP8，非专家模块保留更高精度。这里再次说明：模型架构、draft、数值路径、cache manager 与 admission policy 是一套系统，而非五个独立“加速插件”。</p>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="lab">
|
||||||
|
<p class="eyebrow"><span>28</span> INTERACTIVE LAB</p>
|
||||||
|
<h2>现在让系统亲手失败一次</h2>
|
||||||
|
<p>四个实验分别管理容量、阶段干扰、串行轮数与集群状态。先使用默认 K3 / 长上下文场景，再故意把每项推到失败区。</p>
|
||||||
|
<InferenceServingLab />
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="checklist">
|
||||||
|
<p class="eyebrow"><span>29</span> AUDIT CHECKLIST</p>
|
||||||
|
<h2>看到任何“吞吐提升 X 倍”，先跑完这十四问</h2>
|
||||||
|
<div class="audit-list">
|
||||||
|
{audits.map(([title, question], index) => <article><span>{String(index + 1).padStart(2, "0")}</span><b>{title}</b><p>{question}</p></article>)}
|
||||||
|
</div>
|
||||||
|
<div class="three-levels">
|
||||||
|
<article><span>REPORTED</span><b>作者报告值</b><p>保留硬件、模型、工作负载、SLO 和分母，不跨场景外推。</p></article>
|
||||||
|
<article><span>REPRODUCED</span><b>独立复现值</b><p>公开脚本、commit、环境与误差，才可作为可比较证据。</p></article>
|
||||||
|
<article><span>ESTIMATED</span><b>教学 / 容量估算</b><p>公开公式与假设，用于量纲推理，不冒充 benchmark。</p></article>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="article-section" id="papers">
|
||||||
|
<p class="eyebrow"><span>↳</span> PRIMARY-SOURCE CHAIN</p>
|
||||||
|
<h2>62 个节点：从 MQA 到 Kimi K3</h2>
|
||||||
|
<p>每个链接都指向论文、会议页或官方实现。历史顺序用于建立因果脉络，不代表后出的系统在所有工作负载上都更优。</p>
|
||||||
|
<div class="paper-chain">
|
||||||
|
{paperChain.map(([year, title, url, note]) => (
|
||||||
|
<a href={url} target="_blank" rel="noreferrer" data-paper-link>
|
||||||
|
<time>{year}</time><div><b>{title}</b><p>{note}</p></div><span>↗</span>
|
||||||
|
</a>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
</article>
|
||||||
|
</div>
|
||||||
|
</BaseLayout>
|
||||||
|
|
||||||
|
<style>
|
||||||
|
.inference-hero {
|
||||||
|
background:
|
||||||
|
linear-gradient(115deg, rgba(18, 39, 52, .98), rgba(30, 62, 66, .94)),
|
||||||
|
repeating-linear-gradient(90deg, transparent 0 58px, rgba(255,255,255,.035) 58px 59px);
|
||||||
|
}
|
||||||
|
.inference-hero h1 { max-width: 920px; color: #f7f3eb; }
|
||||||
|
.inference-hero .lead { color: #c0ced3; }
|
||||||
|
.inference-hero .page-facts { border-color: rgba(255,255,255,.2); }
|
||||||
|
.inference-hero .page-facts div { border-color: rgba(255,255,255,.14); }
|
||||||
|
.inference-hero .page-facts dt { color: #97adb5; }
|
||||||
|
.inference-hero .page-facts dd { color: #edf1ef; }
|
||||||
|
.prerequisite-grid { display: grid; grid-template-columns: repeat(3, 1fr); gap: 8px; margin: 28px 0; }
|
||||||
|
.prerequisite-grid a { padding: 18px; border: 1px solid var(--line); background: var(--paper); color: inherit; text-decoration: none; }
|
||||||
|
.prerequisite-grid span, .ledger-grid span, .one-line span, .request-stack > article > span, .metric-defs > article > span, .phase-compare span, .byte-bars span, .state-compare span, .paging-visual span, .prefill-flow span, .batch-compare span, .kernel-stack span, .pd-visual span, .quant-grid > article > span, .spec-flow span, .parallel-grid span, .lineage-map time, .lineage-map article > span, .deepseek-callout span, .v23-grid span, .deployment-report header span, .v4-state span, .reported-values > span, .mooncake-visual span, .k3-block span, .page-semantics span, .k3-spec-grid span, .coding-trace > span, .three-levels span {
|
||||||
|
color: var(--copper);
|
||||||
|
font: .55rem/1.4 var(--mono);
|
||||||
|
letter-spacing: .1em;
|
||||||
|
}
|
||||||
|
.prerequisite-grid b { display: block; margin: 8px 0; font-size: .67rem; }
|
||||||
|
.prerequisite-grid p { color: var(--muted); font-size: .56rem; }
|
||||||
|
.ledger-grid { display: grid; grid-template-columns: repeat(3, 1fr); gap: 1px; margin-top: 24px; border: 1px solid var(--line); background: var(--line); }
|
||||||
|
.ledger-grid article { min-height: 190px; padding: 20px; background: var(--paper-raised); }
|
||||||
|
.ledger-grid h3 { margin: 9px 0 14px; font: 650 1rem var(--serif); }
|
||||||
|
.ledger-grid b { display: block; margin-bottom: 8px; font-size: .63rem; }
|
||||||
|
.ledger-grid p { color: var(--muted); font-size: .57rem; }
|
||||||
|
.one-line { margin-top: 18px; padding: 22px; border-left: 4px solid var(--copper); background: var(--copper-pale); }
|
||||||
|
.one-line p { margin-top: 8px; font: 650 1rem/1.55 var(--serif); }
|
||||||
|
.wave-timeline { margin-top: 24px; border-top: 1px solid var(--line); }
|
||||||
|
.wave-timeline article { display: grid; grid-template-columns: 42px 92px 1fr; gap: 15px; padding: 18px 0; border-bottom: 1px solid var(--line); }
|
||||||
|
.wave-timeline > article > div { color: var(--copper); font: .55rem var(--mono); }
|
||||||
|
.wave-timeline time { color: var(--muted); font: .55rem var(--mono); }
|
||||||
|
.wave-timeline h3 { font: 650 1rem var(--serif); }
|
||||||
|
.wave-timeline section b { display: block; margin: 6px 0; font: .58rem var(--mono); }
|
||||||
|
.wave-timeline section p { color: var(--muted); font-size: .59rem; }
|
||||||
|
.request-stack { display: grid; grid-template-columns: repeat(7, 1fr); gap: 7px; margin-top: 24px; }
|
||||||
|
.request-stack article { min-width: 0; padding: 17px; border-top: 4px solid var(--sage); background: var(--paper); }
|
||||||
|
.request-stack b { display: block; margin: 12px 0; font: 650 .68rem var(--serif); }
|
||||||
|
.request-stack p { min-height: 78px; font-size: .55rem; }
|
||||||
|
.request-stack small { color: var(--muted); font-size: .49rem; }
|
||||||
|
.boundary-grid { display: grid; grid-template-columns: repeat(4, 1fr); gap: 1px; margin-top: 18px; border: 1px solid var(--line); background: var(--line); }
|
||||||
|
.boundary-grid article { display: grid; grid-template-columns: 1fr auto 1fr; align-items: center; gap: 8px; padding: 16px; background: var(--paper-raised); text-align: center; }
|
||||||
|
.boundary-grid b, .boundary-grid p { font-size: .58rem; }
|
||||||
|
.boundary-grid i { color: var(--copper); font-style: normal; }
|
||||||
|
.metric-defs { display: grid; grid-template-columns: repeat(5, 1fr); gap: 8px; margin-top: 24px; }
|
||||||
|
.metric-defs article { padding: 18px; border: 1px solid var(--line); background: var(--paper); }
|
||||||
|
.metric-defs h3 { margin: 10px 0; font: 650 .8rem var(--serif); }
|
||||||
|
.metric-defs p { color: var(--muted); font-size: .56rem; }
|
||||||
|
.formula { overflow-x: auto; margin: 24px 0; padding: 22px; border-left: 4px solid var(--copper); background: #213746; color: white; font: 650 clamp(.8rem, 1.8vw, 1.16rem)/1.6 var(--mono); }
|
||||||
|
.formula small { display: block; margin-top: 10px; color: #bdcbd2; font: .55rem/1.6 var(--sans); }
|
||||||
|
.plain-language, .warning-note { margin: 18px 0; padding: 18px; background: var(--sage-pale); }
|
||||||
|
.plain-language b, .warning-note b { color: var(--sage); font: .55rem var(--mono); }
|
||||||
|
.plain-language p, .warning-note p { margin-top: 7px; font-size: .61rem; }
|
||||||
|
.warning-note { border-left: 3px solid var(--copper); background: var(--copper-pale); }
|
||||||
|
.warning-note b { color: var(--copper); }
|
||||||
|
.request-clock { display: grid; grid-template-columns: 13fr 36fr 4fr 47fr; gap: 3px; margin-top: 24px; }
|
||||||
|
.request-clock > div { min-width: 0; padding: 16px 10px; background: var(--paper); }
|
||||||
|
.request-clock span { color: var(--copper); font: .49rem var(--mono); }
|
||||||
|
.request-clock b { display: block; overflow: hidden; margin: 8px 0; font-size: .57rem; text-overflow: ellipsis; white-space: nowrap; }
|
||||||
|
.request-clock i { display: block; width: var(--w); min-width: 12px; height: 7px; background: var(--sage); }
|
||||||
|
.request-clock .first { background: var(--copper-pale); }
|
||||||
|
.phase-compare, .batch-compare { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; margin-top: 18px; }
|
||||||
|
.phase-compare article, .batch-compare article { padding: 22px; border: 1px solid var(--line); }
|
||||||
|
.phase-compare h3, .batch-compare h3 { margin: 10px 0; font: 650 1rem var(--serif); }
|
||||||
|
.phase-compare li { margin: 6px 0; color: var(--muted); font-size: .58rem; }
|
||||||
|
.byte-bars { display: grid; gap: 12px; margin-top: 20px; }
|
||||||
|
.byte-bars article { display: grid; grid-template-columns: 100px 1fr 120px; align-items: center; gap: 12px; padding: 13px; border: 1px solid var(--line); }
|
||||||
|
.byte-bars i { width: var(--size); height: 16px; background: var(--sage); }
|
||||||
|
.byte-bars b { font: 650 .68rem var(--mono); }
|
||||||
|
.byte-bars p { grid-column: 2 / -1; color: var(--muted); font-size: .52rem; }
|
||||||
|
.kv-factor { display: grid; grid-template-columns: repeat(7, 1fr); gap: 7px; }
|
||||||
|
.kv-factor article { padding: 15px; border: 1px solid var(--line); text-align: center; }
|
||||||
|
.kv-factor b { display: block; color: var(--copper); font: 650 1.2rem var(--serif); }
|
||||||
|
.kv-factor span { font-size: .55rem; }
|
||||||
|
.kv-factor p { margin-top: 8px; color: var(--muted); font-size: .49rem; }
|
||||||
|
.state-compare { display: grid; grid-template-columns: repeat(4, 1fr); gap: 8px; margin-top: 24px; }
|
||||||
|
.state-compare article { padding: 20px; border: 1px solid var(--line); background: var(--paper); }
|
||||||
|
.state-compare article.deepseek { border-top: 4px solid #6e718e; }
|
||||||
|
.state-compare article.kimi { border-top: 4px solid var(--copper); }
|
||||||
|
.state-compare h3 { margin: 14px 0 8px; font: 650 .85rem var(--serif); }
|
||||||
|
.state-compare p { min-height: 84px; color: var(--muted); font-size: .55rem; }
|
||||||
|
.state-compare small { font: .49rem var(--mono); }
|
||||||
|
.head-dots { display: grid; grid-template-columns: repeat(8, 1fr); gap: 3px; margin-top: 18px; }
|
||||||
|
.head-dots i { aspect-ratio: 1; background: var(--sage); }
|
||||||
|
.head-dots.few i { grid-column: span 2; }
|
||||||
|
.latent-box { display: grid; place-content: center; height: 48px; margin-top: 18px; border: 1px dashed var(--line-strong); }
|
||||||
|
.latent-box i { width: 65px; height: 16px; background: #6e718e; }
|
||||||
|
.state-box { display: grid; grid-template-columns: repeat(3, 1fr); gap: 4px; height: 48px; margin-top: 18px; }
|
||||||
|
.state-box i { background: var(--copper); opacity: .7; }
|
||||||
|
.paging-visual, .pd-visual, .mooncake-visual { display: grid; grid-template-columns: 1fr auto 1fr; align-items: center; gap: 16px; margin-top: 24px; padding: 22px; border: 1px solid var(--line); background: #213746; color: white; }
|
||||||
|
.paging-visual section, .pd-visual section, .mooncake-visual section { min-height: 150px; padding: 18px; background: rgba(255,255,255,.055); }
|
||||||
|
.paging-visual p, .pd-visual p, .mooncake-visual p { color: #bdcbd2; font-size: .55rem; }
|
||||||
|
.paging-visual > u, .pd-visual > u, .mooncake-visual > u { color: #e3a279; font: .5rem/1.5 var(--mono); text-align: center; text-decoration: none; }
|
||||||
|
.memory-row, .page-row { display: flex; gap: 3px; margin: 20px 0; }
|
||||||
|
.memory-row i { height: 24px; }
|
||||||
|
.memory-row .used { background: var(--sage); }.memory-row .hole { background: #65737b; opacity: .4; }
|
||||||
|
.memory-row .w6 { flex: 6 }.memory-row .w4 { flex: 4 }.memory-row .w2 { flex: 2 }
|
||||||
|
.page-row i { flex: 1; height: 24px; border: 1px solid #657782; }
|
||||||
|
.page-row i.used { background: var(--sage); }.page-row i.shared { background: var(--copper); }
|
||||||
|
.three-gains, .cache-contract, .moe-debts { display: grid; grid-template-columns: repeat(3, 1fr); gap: 8px; margin-top: 16px; }
|
||||||
|
.three-gains article, .cache-contract article, .moe-debts article { padding: 16px; border: 1px solid var(--line); }
|
||||||
|
.three-gains b, .cache-contract b, .moe-debts b { font-size: .65rem; }
|
||||||
|
.three-gains p, .cache-contract p, .moe-debts p { margin-top: 7px; color: var(--muted); font-size: .54rem; }
|
||||||
|
.prefill-flow, .spec-flow { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr; align-items: center; gap: 10px; margin-top: 24px; }
|
||||||
|
.prefill-flow article, .spec-flow article { min-height: 180px; padding: 20px; border: 1px solid var(--line); background: var(--paper); }
|
||||||
|
.prefill-flow > u, .spec-flow > u { color: var(--copper); text-decoration: none; }
|
||||||
|
.prefill-flow b { display: block; margin: 12px 0; font: 650 1rem var(--serif); }
|
||||||
|
.prefill-flow p, .spec-flow p { color: var(--muted); font-size: .55rem; }
|
||||||
|
.token-line { height: 20px; margin: 18px 0; background: repeating-linear-gradient(90deg, var(--sage) 0 4px, transparent 4px 7px); }
|
||||||
|
.matrix { height: 65px; margin: 18px 0; background: repeating-linear-gradient(0deg, transparent 0 6px, rgba(87,130,121,.5) 6px 9px), repeating-linear-gradient(90deg, transparent 0 6px, rgba(87,130,121,.4) 6px 9px); }
|
||||||
|
.page-row.mini i { min-width: 10px; }
|
||||||
|
.decode-loop { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto; align-items: center; gap: 8px; margin: 24px 0; padding: 22px; background: #213746; color: white; }
|
||||||
|
.decode-loop div { min-height: 120px; padding: 18px; background: rgba(255,255,255,.055); }
|
||||||
|
.decode-loop span { color: #e3a279; font: .53rem var(--mono); }
|
||||||
|
.decode-loop b { display: block; margin: 14px 0; color: white; }
|
||||||
|
.decode-loop p { color: #bdcbd2; font-size: .54rem; }
|
||||||
|
.decode-loop u { color: #e3a279; text-decoration: none; }
|
||||||
|
.batch-grid { display: grid; grid-template-columns: repeat(6, 1fr); gap: 4px; margin: 20px 0; }
|
||||||
|
.batch-grid i { height: 28px; background: var(--sage); }
|
||||||
|
.batch-grid.static i:nth-child(5n), .batch-grid.static i:nth-child(7n) { opacity: .14; }
|
||||||
|
.batch-grid.dynamic i:nth-child(4n) { background: var(--copper); }
|
||||||
|
.batch-compare p { color: var(--muted); font-size: .56rem; }
|
||||||
|
.chunk-schedule { display: grid; grid-template-columns: 90px repeat(6, 1fr); gap: 4px; align-items: center; margin: 24px 0; }
|
||||||
|
.chunk-schedule > span, .chunk-schedule > b { padding: 9px; font: .5rem var(--mono); text-align: center; }
|
||||||
|
.chunk-schedule i { display: grid; place-content: center; height: 42px; font: .5rem var(--mono); font-style: normal; }
|
||||||
|
.chunk-schedule .prefill { background: var(--copper-pale); }.chunk-schedule .decode { background: var(--sage-pale); }
|
||||||
|
.chunk-schedule .span4 { grid-column: span 4; }
|
||||||
|
.kernel-stack { display: grid; grid-template-columns: repeat(5, 1fr); gap: 1px; margin-top: 24px; border: 1px solid var(--line); background: var(--line); }
|
||||||
|
.kernel-stack article { padding: 20px; background: var(--paper-raised); }
|
||||||
|
.kernel-stack b { display: block; min-height: 45px; margin: 12px 0; font-size: .64rem; }
|
||||||
|
.kernel-stack p { color: var(--muted); font-size: .54rem; }
|
||||||
|
.deepseek-callout { margin-top: 18px; padding: 22px; border: 1px solid #77778e; background: #eeeef3; }
|
||||||
|
.deepseek-callout h3 { margin: 9px 0; font: 650 1.1rem var(--serif); }
|
||||||
|
.deepseek-callout p { color: var(--muted); font-size: .59rem; }
|
||||||
|
.prefix-visual { display: flex; align-items: center; justify-content: center; gap: 6px; overflow-x: auto; margin: 24px 0; padding: 24px; background: #213746; color: white; }
|
||||||
|
.prefix-visual > div, .prefix-visual section > div { padding: 14px; border: 1px solid #657782; text-align: center; }
|
||||||
|
.prefix-visual span { color: white; font: .5rem var(--mono); }
|
||||||
|
.prefix-visual i { display: block; width: 54px; height: 12px; margin-top: 9px; background: var(--sage); }
|
||||||
|
.prefix-visual u { color: #e3a279; font: .47rem var(--mono); text-decoration: none; }
|
||||||
|
.prefix-visual section { display: grid; gap: 9px; }
|
||||||
|
.cache-contract { grid-template-columns: repeat(4, 1fr); }
|
||||||
|
.pd-visual > u { display: grid; gap: 5px; }
|
||||||
|
.pd-visual > u b { color: #e3a279; }.pd-visual > u small { color: #aebfc7; }
|
||||||
|
.gpu-row { display: flex; gap: 6px; margin: 25px 0; }
|
||||||
|
.gpu-row i { flex: 1; height: 45px; border: 1px solid #739286; background: rgba(87,139,128,.35); }
|
||||||
|
.gpu-row.decode i { height: 31px; background: rgba(218,150,112,.32); }
|
||||||
|
.quant-grid, .parallel-grid, .v23-grid, .k3-spec-grid { display: grid; grid-template-columns: repeat(4, 1fr); gap: 8px; margin-top: 24px; }
|
||||||
|
.quant-grid article, .parallel-grid article, .v23-grid article, .k3-spec-grid article { padding: 19px; border: 1px solid var(--line); }
|
||||||
|
.quant-grid h3, .k3-spec-grid h3 { margin: 9px 0; font: 650 .9rem var(--serif); }
|
||||||
|
.quant-grid b { display: block; min-height: 42px; margin: 9px 0; font-size: .6rem; }
|
||||||
|
.quant-grid p, .parallel-grid p, .v23-grid p, .k3-spec-grid p { color: var(--muted); font-size: .55rem; }
|
||||||
|
.draft-tokens, .verify-line { display: flex; gap: 4px; margin: 20px 0; }
|
||||||
|
.draft-tokens i { display: grid; flex: 1; place-content: center; height: 35px; background: var(--copper-pale); font: .55rem var(--mono); font-style: normal; }
|
||||||
|
.draft-tokens.accepted i:not(:last-child) { background: var(--sage-pale); }
|
||||||
|
.verify-line i { flex: 1; height: 35px; border: 2px solid var(--sage); }
|
||||||
|
.parallel-grid { grid-template-columns: repeat(5, 1fr); }
|
||||||
|
.parallel-grid span { display: inline-grid; place-content: center; width: 34px; height: 34px; border-radius: 50%; background: var(--sage); color: white; font: .52rem var(--mono); }
|
||||||
|
.parallel-grid b { display: block; margin: 12px 0; font-size: .64rem; }
|
||||||
|
.moe-serving { display: grid; grid-template-columns: 1fr auto 1.3fr; align-items: center; gap: 16px; margin: 24px 0; padding: 24px; background: #213746; }
|
||||||
|
.moe-serving .tokens, .moe-serving .experts { display: grid; grid-template-columns: repeat(4, 1fr); gap: 5px; }
|
||||||
|
.moe-serving i { display: grid; place-content: center; height: 42px; background: rgba(87,139,128,.45); color: white; font: .5rem var(--mono); font-style: normal; }
|
||||||
|
.moe-serving .experts i { border: 1px solid #657782; background: transparent; }
|
||||||
|
.moe-serving .experts i.hot { border-color: #e3a279; background: rgba(218,150,112,.3); }
|
||||||
|
.moe-serving > u { color: #e3a279; font: .5rem var(--mono); text-decoration: none; }
|
||||||
|
.moe-debts { grid-template-columns: repeat(4, 1fr); }
|
||||||
|
.scheduler-board { margin: 24px 0; border: 1px solid var(--line); }
|
||||||
|
.scheduler-board header, .scheduler-board > div { display: grid; grid-template-columns: 1fr .8fr .8fr 1.2fr .8fr; align-items: center; gap: 8px; padding: 12px; border-bottom: 1px solid var(--line); }
|
||||||
|
.scheduler-board header { background: #213746; color: white; font: .5rem var(--mono); }
|
||||||
|
.scheduler-board > div span { color: var(--muted); font-size: .56rem; }
|
||||||
|
.scheduler-board b { font-size: .58rem; }
|
||||||
|
.scheduler-board em { padding: 6px; background: var(--line); font: .49rem var(--mono); font-style: normal; text-align: center; }
|
||||||
|
.scheduler-board em.go { background: var(--sage-pale); color: var(--sage); }.scheduler-board em.stop { background: #f0d9d4; color: #8a3f36; }
|
||||||
|
.fleet-triangle { display: grid; grid-template-columns: repeat(3, 1fr); gap: 8px; margin: 24px 0; }
|
||||||
|
.fleet-triangle article { min-height: 170px; padding: 20px; border-top: 4px solid var(--copper); background: var(--paper); }
|
||||||
|
.fleet-triangle span { color: var(--copper); font: .53rem var(--mono); }
|
||||||
|
.fleet-triangle b { display: block; margin: 14px 0; font: 650 .85rem var(--serif); }
|
||||||
|
.fleet-triangle p { color: var(--muted); font-size: .56rem; }
|
||||||
|
.deepseek-section, .kimi-section { position: relative; }
|
||||||
|
.deepseek-section::before, .kimi-section::before { position: absolute; top: 0; right: 0; width: 58px; height: 4px; background: #73738c; content: ""; }
|
||||||
|
.kimi-section::before { background: var(--copper); }
|
||||||
|
.lineage-map { display: grid; grid-template-columns: 1fr auto 1fr auto 1fr auto 1fr; align-items: stretch; gap: 7px; margin-top: 24px; }
|
||||||
|
.lineage-map article { padding: 19px; border: 1px solid var(--line); background: #eeeef3; }
|
||||||
|
.lineage-map > u { align-self: center; color: var(--copper); text-decoration: none; }
|
||||||
|
.lineage-map h3 { min-height: 46px; margin: 11px 0; font: 650 .86rem var(--serif); }
|
||||||
|
.lineage-map p { min-height: 86px; color: var(--muted); font-size: .54rem; }
|
||||||
|
.lineage-map article > span { display: block; margin-top: 12px; }
|
||||||
|
.kimi-lineage article { background: var(--copper-pale); }
|
||||||
|
.v23-grid article, .k3-spec-grid article { border-top: 3px solid #73738c; }
|
||||||
|
.v23-grid h3 { min-height: 56px; margin: 10px 0; font: 650 .83rem var(--serif); }
|
||||||
|
.deployment-report { margin-top: 20px; border: 1px solid #73738c; }
|
||||||
|
.deployment-report header { display: flex; justify-content: space-between; padding: 16px; background: #34364b; color: white; }
|
||||||
|
.deployment-report header b { color: white; }
|
||||||
|
.deployment-report section { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; background: var(--line); }
|
||||||
|
.deployment-report article { padding: 20px; background: var(--paper-raised); }
|
||||||
|
.deployment-report article b { color: var(--copper); font: .54rem var(--mono); }
|
||||||
|
.deployment-report strong { display: block; margin: 9px 0; font: 650 1rem var(--serif); }
|
||||||
|
.deployment-report p, .deployment-report footer { color: var(--muted); font-size: .55rem; }
|
||||||
|
.deployment-report footer { padding: 15px; border-top: 1px solid var(--line); }
|
||||||
|
.v4-state { display: grid; grid-template-columns: repeat(4, 1fr); gap: 8px; margin-top: 24px; }
|
||||||
|
.v4-state article { min-height: 220px; padding: 19px; border: 1px solid var(--line); }
|
||||||
|
.v4-state b { display: block; min-height: 42px; margin: 10px 0; font-size: .63rem; }
|
||||||
|
.v4-state p { min-height: 77px; color: var(--muted); font-size: .54rem; }
|
||||||
|
.v4-state i { display: block; height: 35px; margin-top: 15px; }
|
||||||
|
.v4-state .sparse { background: repeating-linear-gradient(90deg, var(--sage) 0 6px, transparent 6px 18px); }
|
||||||
|
.v4-state .hierarchy { background: linear-gradient(90deg, var(--sage) 0 25%, transparent 25% 30%, var(--sage) 30% 70%, transparent 70% 75%, var(--sage) 75%); opacity: .7; }
|
||||||
|
.v4-state .window { background: linear-gradient(90deg, transparent 0 60%, var(--copper) 60%); }
|
||||||
|
.v4-state .storage { background: linear-gradient(#557f76 0 33%, #7f988f 33% 66%, #b99a79 66%); }
|
||||||
|
.reported-values { margin-top: 18px; padding: 20px; border: 1px solid #73738c; background: #eeeef3; }
|
||||||
|
.reported-values { display: grid; grid-template-columns: 1fr 1fr; gap: 9px; }
|
||||||
|
.reported-values > span, .reported-values > p { grid-column: 1 / -1; }
|
||||||
|
.reported-values article { padding: 16px; background: rgba(255,255,255,.6); }
|
||||||
|
.reported-values b { display: block; font-size: .59rem; }.reported-values strong { display: block; margin-top: 8px; font: 650 1rem var(--serif); }
|
||||||
|
.reported-values p { color: var(--muted); font-size: .55rem; }
|
||||||
|
.mooncake-visual { grid-template-columns: 1fr auto 1fr auto 1fr; }
|
||||||
|
.cache-layers { display: grid; gap: 5px; margin: 18px 0; }
|
||||||
|
.cache-layers i { padding: 8px; background: rgba(87,139,128,.35); font: .5rem var(--mono); font-style: normal; }
|
||||||
|
.mooncake-values { background: var(--copper-pale); border-color: var(--copper); }
|
||||||
|
.k3-block { display: grid; grid-template-columns: 1fr auto 1fr auto 1.2fr; align-items: center; gap: 14px; margin: 24px 0; padding: 24px; background: #213746; color: white; }
|
||||||
|
.k3-block > div { padding: 18px; border: 1px solid #657782; }
|
||||||
|
.k3-block > u { color: #e3a279; font: .5rem var(--mono); text-decoration: none; }
|
||||||
|
.kda-layers i { display: block; height: 13px; margin-top: 6px; background: var(--copper); opacity: .7; }
|
||||||
|
.mla-layer i { height: 52px; display: block; margin-top: 10px; background: #72768e; }
|
||||||
|
.totals { display: grid; gap: 8px; }.totals b { color: white; font: 650 .9rem var(--serif); }.totals small { color: #bdcbd2; font-size: .5rem; }
|
||||||
|
.page-semantics { display: grid; grid-template-columns: repeat(4, 1fr); gap: 1px; border: 1px solid var(--line); background: var(--line); }
|
||||||
|
.page-semantics article { padding: 18px; background: var(--paper-raised); }
|
||||||
|
.page-semantics article.accent { background: var(--copper-pale); }
|
||||||
|
.page-semantics b { display: block; margin: 9px 0; font: 650 1rem var(--serif); }
|
||||||
|
.page-semantics p { color: var(--muted); font-size: .54rem; }
|
||||||
|
.coding-trace { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; margin-top: 18px; padding: 20px; background: var(--copper-pale); }
|
||||||
|
.coding-trace > span { width: 100%; }
|
||||||
|
.coding-trace b { font: 650 1rem var(--serif); }.coding-trace u { color: var(--copper); text-decoration: none; }.coding-trace p { flex: 1 0 100%; color: var(--muted); font-size: .56rem; }
|
||||||
|
.audit-list { display: grid; grid-template-columns: 1fr 1fr; gap: 1px; margin-top: 24px; border: 1px solid var(--line); background: var(--line); }
|
||||||
|
.audit-list article { display: grid; grid-template-columns: 34px 90px 1fr; gap: 12px; align-items: start; padding: 16px; background: var(--paper-raised); }
|
||||||
|
.audit-list span { color: var(--copper); font: .53rem var(--mono); }.audit-list b { font-size: .61rem; }.audit-list p { color: var(--muted); font-size: .55rem; }
|
||||||
|
.three-levels { display: grid; grid-template-columns: repeat(3, 1fr); gap: 8px; margin-top: 18px; }
|
||||||
|
.three-levels article { padding: 18px; border: 1px solid var(--line); }.three-levels b { display: block; margin: 9px 0; font-size: .65rem; }.three-levels p { color: var(--muted); font-size: .55rem; }
|
||||||
|
.paper-chain { margin-top: 22px; border-top: 1px solid var(--line); }
|
||||||
|
.paper-chain a { display: grid; grid-template-columns: 62px 1fr auto; gap: 14px; align-items: start; padding: 15px 4px; border-bottom: 1px solid var(--line); color: inherit; text-decoration: none; }
|
||||||
|
.paper-chain a:hover { background: var(--paper); }
|
||||||
|
.paper-chain time { color: var(--copper); font: .53rem var(--mono); }
|
||||||
|
.paper-chain b { font-size: .63rem; }.paper-chain p { margin-top: 4px; color: var(--muted); font-size: .54rem; }.paper-chain a > span { color: var(--copper); }
|
||||||
|
@media (max-width: 1050px) {
|
||||||
|
.request-stack { grid-template-columns: repeat(4, 1fr); }
|
||||||
|
.metric-defs, .kernel-stack, .parallel-grid { grid-template-columns: repeat(3, 1fr); }
|
||||||
|
.lineage-map { grid-template-columns: 1fr 1fr; }
|
||||||
|
.lineage-map > u { display: none; }
|
||||||
|
}
|
||||||
|
@media (max-width: 820px) {
|
||||||
|
.prerequisite-grid, .ledger-grid, .metric-defs, .state-compare, .quant-grid, .parallel-grid, .v23-grid, .v4-state, .k3-spec-grid { grid-template-columns: repeat(2, 1fr); }
|
||||||
|
.request-stack { grid-template-columns: repeat(2, 1fr); }
|
||||||
|
.boundary-grid, .kv-factor, .page-semantics { grid-template-columns: repeat(2, 1fr); }
|
||||||
|
.paging-visual, .prefill-flow, .spec-flow, .pd-visual, .mooncake-visual, .k3-block { grid-template-columns: 1fr; }
|
||||||
|
.paging-visual > u, .prefill-flow > u, .spec-flow > u, .pd-visual > u, .mooncake-visual > u, .k3-block > u { transform: rotate(90deg); justify-self: center; }
|
||||||
|
.mooncake-visual { grid-template-columns: 1fr; }
|
||||||
|
.three-gains, .cache-contract, .moe-debts { grid-template-columns: repeat(2, 1fr); }
|
||||||
|
.audit-list { grid-template-columns: 1fr; }
|
||||||
|
}
|
||||||
|
@media (max-width: 560px) {
|
||||||
|
.prerequisite-grid, .ledger-grid, .request-stack, .boundary-grid, .metric-defs, .phase-compare, .state-compare, .three-gains, .cache-contract, .quant-grid, .parallel-grid, .moe-debts, .fleet-triangle, .lineage-map, .v23-grid, .deployment-report section, .v4-state, .reported-values, .page-semantics, .k3-spec-grid, .three-levels { grid-template-columns: 1fr; }
|
||||||
|
.request-clock { grid-template-columns: 1fr; }
|
||||||
|
.byte-bars article { grid-template-columns: 1fr; }.byte-bars p { grid-column: auto; }
|
||||||
|
.kv-factor { grid-template-columns: repeat(2, 1fr); }
|
||||||
|
.decode-loop { grid-template-columns: 1fr; }.decode-loop u { transform: rotate(90deg); justify-self: center; }
|
||||||
|
.chunk-schedule { grid-template-columns: 72px repeat(6, minmax(52px, 1fr)); overflow-x: auto; }
|
||||||
|
.scheduler-board { overflow-x: auto; }.scheduler-board header, .scheduler-board > div { min-width: 620px; }
|
||||||
|
.moe-serving { grid-template-columns: 1fr; }.moe-serving > u { text-align: center; }
|
||||||
|
.reported-values > span, .reported-values > p { grid-column: auto; }
|
||||||
|
.audit-list article { grid-template-columns: 30px 1fr; }.audit-list p { grid-column: 2; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
Reference in New Issue
Block a user