feat: execute DeepSeek absorbed MLA cache
This commit is contained in:
wuyang
+9
-3
@@ -14,7 +14,7 @@
|
|||||||
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
||||||
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
||||||
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
||||||
| DeepSeek 专题 | 三轮实证进行中 | 87% | 真实 latent-cache kernel、更大样本负载、FP8/pipeline 与 R1-like RL 复现 |
|
| DeepSeek 专题 | 三轮实证进行中 | 89% | SM90 FlashMLA 优化 kernel、更大样本负载、FP8/pipeline 与 R1-like RL 复现 |
|
||||||
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
||||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||||
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
||||||
@@ -41,7 +41,7 @@
|
|||||||
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||||
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
||||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 八联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等七十五个原创交互视图。
|
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 九联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等七十六个原创交互视图。
|
||||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||||
@@ -174,11 +174,15 @@
|
|||||||
- [x] 独立复跑 31/31 exact：来源、配置、tokenization、逐层 hidden hashes、MLA shapes、aggregate loads 与全部 token routes 完全一致；计时明确不比较。
|
- [x] 独立复跑 31/31 exact：来源、配置、tokenization、逐层 hidden hashes、MLA shapes、aggregate loads 与全部 token routes 完全一致；计时明确不比较。
|
||||||
- [x] DeepSeek 新四视图本地真实 Chrome 回归通过：逐 token 路由、覆盖/均衡分账、latent/HF eager cache 计算、27 层执行断面、键盘 tabs 与 390px 移动端均无异常。
|
- [x] DeepSeek 新四视图本地真实 Chrome 回归通过：逐 token 路由、覆盖/均衡分账、latent/HF eager cache 计算、27 层执行断面、键盘 tabs 与 390px 移动端均无异常。
|
||||||
- [x] DeepSeek 三轮首个真实权重执行里程碑以源提交 `e864205`、不可变镜像 `20260729T062034Z-e864205` 发布；OCI digest `sha256:c3c37631…91addc`，NAS、VPS/Tailscale、NPM、DNS、HTTPS、证书、门户与十六套生产 Chrome 回归全链路通过；保留 `20260729T053852Z-2ef846f` 回滚。
|
- [x] DeepSeek 三轮首个真实权重执行里程碑以源提交 `e864205`、不可变镜像 `20260729T062034Z-e864205` 发布；OCI digest `sha256:c3c37631…91addc`，NAS、VPS/Tailscale、NPM、DNS、HTTPS、证书、门户与十六套生产 Chrome 回归全链路通过；保留 `20260729T053852Z-2ef846f` 回滚。
|
||||||
|
- [x] DeepSeek 真实吸收式缓存闭环：同一官方 V2-Lite layer-1 权重与真实 hidden state 依次执行 HF eager、V3 naive 与 V3 absorb；26-token BF16 active cache 从 266,240 B 降至 29,952 B，实际比率 8.8889× / 88.75%。
|
||||||
|
- [x] 权重吸收正确性闸门通过：HF eager↔V3 naive BF16 max |Δ| 0.0012207，V3 naive↔absorb BF16 max |Δ| 0.00390625；同一 BF16 权重提升到 FP32 后 naive↔absorb max |Δ| 1.19e-7，所有输出 finite。
|
||||||
|
- [x] 独立吸收实验复跑 byte-exact：两份完整 JSON SHA-256 均为 `6b4c714a…e63d`；实验脚本、机器可读结果与正式研究账本进入开源树。
|
||||||
|
- [x] FlashMLA 架构边界闭合：固定 revision `15f13e50…a7772c6` 与 CUTLASS submodule；官方矩阵、gencode 与 dense runtime guard 只覆盖 SM90/SM100，本机 SM120 不宣称优化 kernel 已执行；两次隔离构建失败位置如实入账。
|
||||||
|
|
||||||
## 正在进行
|
## 正在进行
|
||||||
|
|
||||||
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
||||||
- [ ] DeepSeek 三轮下一闸门：真实 latent-cache kernel、更大样本 expert-load 置信区间、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
- [ ] DeepSeek 三轮下一闸门：在官方支持的 SM90 环境执行 FlashMLA 优化 kernel、更大样本 expert-load 置信区间、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||||
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
||||||
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
||||||
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||||
@@ -298,6 +302,8 @@
|
|||||||
| 2026-07-29 | MLA 算法状态与框架物化格式分两张账 | V2-Lite latent 576 元素与 HF eager 5,120 元素同时报告；88.75% 不冒充 V2 论文 93.3% |
|
| 2026-07-29 | MLA 算法状态与框架物化格式分两张账 | V2-Lite latent 576 元素与 HF eager 5,120 元素同时报告；88.75% 不冒充 V2 论文 93.3% |
|
||||||
| 2026-07-29 | 专家覆盖、负载均衡与语义永久分开 | 60–64/64 used 不推出 balanced；CV/Gini/effective 同报；expert ID 不跨层连线或命名 |
|
| 2026-07-29 | 专家覆盖、负载均衡与语义永久分开 | 60–64/64 used 不推出 balanced；CV/Gini/effective 同报；expert ID 不跨层连线或命名 |
|
||||||
| 2026-07-29 | DeepSeek 三轮首个真实权重版本以 `20260729T062034Z-e864205` 发布 | OCI digest `sha256:c3c37631…91addc`；复用 `12010→8080`、NPM 31 / cert 41、门户 order 180；保留 `20260729T053852Z-2ef846f` 回滚 |
|
| 2026-07-29 | DeepSeek 三轮首个真实权重版本以 `20260729T062034Z-e864205` 发布 | OCI digest `sha256:c3c37631…91addc`；复用 `12010→8080`、NPM 31 / cert 41、门户 order 180；保留 `20260729T053852Z-2ef846f` 回滚 |
|
||||||
|
| 2026-07-29 | MLA 576 元素缓存从算法合同升级为真实执行 | 官方 V3 naive / absorb 使用同一 V2-Lite 权重与 hidden state；26-token active cache 266,240 B→29,952 B，FP32 代数审计 max \|Δ\| 1.19e-7 |
|
||||||
|
| 2026-07-29 | reference absorb 与 FlashMLA optimized kernel 永久分身份 | 前者已在 RTX 5090 执行；pinned FlashMLA 只列 SM90/SM100 并只生成 sm_90a/sm_100f，本机 SM120 明确标为未支持/未执行 |
|
||||||
| 2026-07-29 | K3 二轮按 32 张对象账与完整报告顺序重建 | total/active、2.5×、KDA state、深度来源、专家路由、视觉目标、轨迹、缓存与评测协议不再压成一页组件摘要 |
|
| 2026-07-29 | K3 二轮按 32 张对象账与完整报告顺序重建 | total/active、2.5×、KDA state、深度来源、专家路由、视觉目标、轨迹、缓存与评测协议不再压成一页组件摘要 |
|
||||||
| 2026-07-29 | K3 原生视觉事实回到 §2.4 / §3.3 核验 | 删除“先冻结语言模型再解冻”旧表述；明确 MoonViT-V2 从头训练，视觉/文本从开始共同 NTP |
|
| 2026-07-29 | K3 原生视觉事实回到 §2.4 / §3.3 核验 | 删除“先冻结语言模型再解冻”旧表述；明确 MoonViT-V2 从头训练，视觉/文本从开始共同 NTP |
|
||||||
| 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项；作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 |
|
| 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项；作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 |
|
||||||
|
|||||||
@@ -19,7 +19,7 @@
|
|||||||
|
|
||||||
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
||||||
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
||||||
以及 75 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
以及 76 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||||
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
||||||
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
||||||
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
||||||
@@ -28,10 +28,13 @@
|
|||||||
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
||||||
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
||||||
DeepSeek 三轮专题以 24 张问题账、10 次技术转向、
|
DeepSeek 三轮专题以 24 张问题账、10 次技术转向、
|
||||||
8 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
9 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||||
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
||||||
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑。详见
|
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
|
||||||
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md)。
|
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
|
||||||
|
`1.19e-7`。FlashMLA 的 SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
|
||||||
|
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
|
||||||
|
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)。
|
||||||
其余专题按进度账本持续扩建。
|
其余专题按进度账本持续扩建。
|
||||||
|
|
||||||
## 本地开发
|
## 本地开发
|
||||||
|
|||||||
@@ -0,0 +1,661 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Execute DeepSeek MLA's naive and weight-absorbed cache paths with real weights.
|
||||||
|
|
||||||
|
The probe deliberately separates three things:
|
||||||
|
|
||||||
|
1. DeepSeek-V2-Lite's official Hugging Face eager attention;
|
||||||
|
2. DeepSeek-V3's official pure-PyTorch ``naive`` and ``absorb`` reference paths;
|
||||||
|
3. FlashMLA's optimized-kernel architecture boundary.
|
||||||
|
|
||||||
|
The first official checkpoint shard contains the embedding, layer 0, and all
|
||||||
|
layer-1 attention weights. We execute layer 0, normalize the real layer-1 input,
|
||||||
|
map the unchanged V2-Lite attention tensors into the pinned V3 reference class,
|
||||||
|
then perform a 25-token prefill plus one-token incremental decode.
|
||||||
|
|
||||||
|
This is a correctness and cache-layout probe, not a serving benchmark.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import copy
|
||||||
|
import gc
|
||||||
|
import hashlib
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import platform
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import types
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import torch.nn.functional as F
|
||||||
|
from safetensors import safe_open
|
||||||
|
from transformers import AutoTokenizer, DynamicCache
|
||||||
|
|
||||||
|
|
||||||
|
HF_REVISION = "604d5664dddd88a0433dbae533b7fe9472482de0"
|
||||||
|
V3_REVISION = "9b4e9788e4a3a731f7567338ed15d3ec549ce03b"
|
||||||
|
FLASHMLA_REVISION = "15f13e5030374295491c5ce31b02d7e63a7772c6"
|
||||||
|
CUTLASS_REVISION = "147f5673d0c1c3dcf66f78d677fd647e4a020219"
|
||||||
|
PROMPT = "用通俗的语言解释，为什么稀疏专家模型可以拥有很多参数，但每个 token 只使用其中一小部分。"
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--artifact-dir", type=Path, required=True)
|
||||||
|
parser.add_argument("--v3-repo", type=Path, required=True)
|
||||||
|
parser.add_argument("--flashmla-repo", type=Path, required=True)
|
||||||
|
parser.add_argument("--output", type=Path, required=True)
|
||||||
|
parser.add_argument("--device", default="cuda")
|
||||||
|
parser.add_argument("--cache-slots", type=int, default=32)
|
||||||
|
parser.add_argument("--captured-at", default=None)
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def sha256(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as handle:
|
||||||
|
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||||
|
digest.update(block)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def git_revision(root: Path) -> str:
|
||||||
|
return subprocess.check_output(
|
||||||
|
["git", "-C", str(root), "rev-parse", "HEAD"], text=True
|
||||||
|
).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def tensor_bytes(tensor: torch.Tensor) -> int:
|
||||||
|
return tensor.numel() * tensor.element_size()
|
||||||
|
|
||||||
|
|
||||||
|
def compare(left: torch.Tensor, right: torch.Tensor) -> dict[str, Any]:
|
||||||
|
if left.shape != right.shape:
|
||||||
|
raise ValueError(f"shape mismatch: {left.shape} != {right.shape}")
|
||||||
|
delta = (left.float() - right.float()).abs().flatten()
|
||||||
|
return {
|
||||||
|
"shape": list(left.shape),
|
||||||
|
"finite": bool(torch.isfinite(left).all() and torch.isfinite(right).all()),
|
||||||
|
"exact_fraction": (left == right).float().mean().item(),
|
||||||
|
"max_abs": delta.max().item(),
|
||||||
|
"mean_abs": delta.mean().item(),
|
||||||
|
"p50_abs": torch.quantile(delta, 0.50).item(),
|
||||||
|
"p90_abs": torch.quantile(delta, 0.90).item(),
|
||||||
|
"p99_abs": torch.quantile(delta, 0.99).item(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_v2_modules(root: Path) -> tuple[Any, Any]:
|
||||||
|
package_name = "deepseek_v2_lite_absorb_official"
|
||||||
|
package = types.ModuleType(package_name)
|
||||||
|
package.__path__ = [str(root)]
|
||||||
|
sys.modules[package_name] = package
|
||||||
|
loaded = {}
|
||||||
|
for leaf in ("configuration_deepseek", "modeling_deepseek"):
|
||||||
|
name = f"{package_name}.{leaf}"
|
||||||
|
spec = importlib.util.spec_from_file_location(name, root / f"{leaf}.py")
|
||||||
|
if spec is None or spec.loader is None:
|
||||||
|
raise RuntimeError(f"cannot import official V2-Lite module: {leaf}")
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
sys.modules[name] = module
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
loaded[leaf] = module
|
||||||
|
return loaded["configuration_deepseek"], loaded["modeling_deepseek"]
|
||||||
|
|
||||||
|
|
||||||
|
def load_v3_module(root: Path) -> Any:
|
||||||
|
inference = root / "inference"
|
||||||
|
sys.path.insert(0, str(inference))
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"deepseek_v3_absorb_official", inference / "model.py"
|
||||||
|
)
|
||||||
|
if spec is None or spec.loader is None:
|
||||||
|
raise RuntimeError("cannot import official DeepSeek-V3 inference/model.py")
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
sys.modules[spec.name] = module
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def read_shard_state(
|
||||||
|
shard: Path, prefixes: tuple[str, ...], exact: tuple[str, ...] = ()
|
||||||
|
) -> dict[str, torch.Tensor]:
|
||||||
|
with safe_open(shard, framework="pt", device="cpu") as handle:
|
||||||
|
return {
|
||||||
|
key: handle.get_tensor(key)
|
||||||
|
for key in handle.keys()
|
||||||
|
if key in exact or any(key.startswith(prefix) for prefix in prefixes)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_real_layer1_input(
|
||||||
|
shard: Path,
|
||||||
|
modeling: Any,
|
||||||
|
config: Any,
|
||||||
|
tokenizer: Any,
|
||||||
|
device: torch.device,
|
||||||
|
) -> tuple[torch.Tensor, dict[str, torch.Tensor], list[int]]:
|
||||||
|
encoded = tokenizer(PROMPT, add_special_tokens=True, return_tensors="pt")
|
||||||
|
input_ids = encoded.input_ids
|
||||||
|
token_ids = input_ids[0].tolist()
|
||||||
|
if len(token_ids) != 26:
|
||||||
|
raise RuntimeError(f"prompt contract changed: expected 26 tokens, got {len(token_ids)}")
|
||||||
|
|
||||||
|
state = read_shard_state(
|
||||||
|
shard,
|
||||||
|
("model.layers.0.", "model.layers.1.self_attn."),
|
||||||
|
("model.embed_tokens.weight", "model.layers.1.input_layernorm.weight"),
|
||||||
|
)
|
||||||
|
embedding = state.pop("model.embed_tokens.weight")
|
||||||
|
hidden = F.embedding(input_ids, embedding).to(device)
|
||||||
|
|
||||||
|
layer0_prefix = "model.layers.0."
|
||||||
|
layer0_state = {
|
||||||
|
key[len(layer0_prefix) :]: value
|
||||||
|
for key, value in state.items()
|
||||||
|
if key.startswith(layer0_prefix)
|
||||||
|
}
|
||||||
|
with torch.device("meta"):
|
||||||
|
layer0 = modeling.DeepseekV2DecoderLayer(config, 0)
|
||||||
|
layer0.to_empty(device="cpu")
|
||||||
|
layer0.load_state_dict(layer0_state, strict=True, assign=True)
|
||||||
|
layer0 = layer0.to(device=device, dtype=torch.bfloat16).eval()
|
||||||
|
|
||||||
|
sequence = input_ids.shape[1]
|
||||||
|
position_ids = torch.arange(sequence, device=device).unsqueeze(0)
|
||||||
|
attention_mask = torch.ones_like(input_ids, device=device)
|
||||||
|
causal_mask = modeling._prepare_4d_causal_attention_mask(
|
||||||
|
attention_mask,
|
||||||
|
input_ids.shape,
|
||||||
|
hidden,
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
with torch.inference_mode():
|
||||||
|
hidden = layer0(
|
||||||
|
hidden,
|
||||||
|
attention_mask=causal_mask,
|
||||||
|
position_ids=position_ids,
|
||||||
|
past_key_value=DynamicCache(),
|
||||||
|
use_cache=True,
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
layer1_norm = modeling.DeepseekV2RMSNorm(
|
||||||
|
config.hidden_size, eps=config.rms_norm_eps
|
||||||
|
).to(device=device, dtype=torch.bfloat16)
|
||||||
|
with torch.no_grad():
|
||||||
|
layer1_norm.weight.copy_(
|
||||||
|
state["model.layers.1.input_layernorm.weight"].to(
|
||||||
|
device=device, dtype=torch.bfloat16
|
||||||
|
)
|
||||||
|
)
|
||||||
|
with torch.inference_mode():
|
||||||
|
normalized = layer1_norm(hidden)
|
||||||
|
|
||||||
|
attention_prefix = "model.layers.1.self_attn."
|
||||||
|
attention_state = {
|
||||||
|
key[len(attention_prefix) :]: value
|
||||||
|
for key, value in state.items()
|
||||||
|
if key.startswith(attention_prefix)
|
||||||
|
}
|
||||||
|
del layer0, layer0_state, embedding, hidden, layer1_norm, state
|
||||||
|
gc.collect()
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
return normalized, attention_state, token_ids
|
||||||
|
|
||||||
|
|
||||||
|
def v3_args(v3: Any, config: Any, cache_slots: int) -> Any:
|
||||||
|
rope = config.rope_scaling
|
||||||
|
args = v3.ModelArgs(
|
||||||
|
max_batch_size=1,
|
||||||
|
max_seq_len=cache_slots,
|
||||||
|
dtype="bf16",
|
||||||
|
vocab_size=config.vocab_size,
|
||||||
|
dim=config.hidden_size,
|
||||||
|
n_layers=config.num_hidden_layers,
|
||||||
|
n_heads=config.num_attention_heads,
|
||||||
|
q_lora_rank=0,
|
||||||
|
kv_lora_rank=config.kv_lora_rank,
|
||||||
|
qk_nope_head_dim=config.qk_nope_head_dim,
|
||||||
|
qk_rope_head_dim=config.qk_rope_head_dim,
|
||||||
|
v_head_dim=config.v_head_dim,
|
||||||
|
original_seq_len=rope["original_max_position_embeddings"],
|
||||||
|
rope_theta=config.rope_theta,
|
||||||
|
rope_factor=rope["factor"],
|
||||||
|
beta_fast=rope["beta_fast"],
|
||||||
|
beta_slow=rope["beta_slow"],
|
||||||
|
mscale=rope["mscale"],
|
||||||
|
)
|
||||||
|
# The official class uses max_seq_len both to configure YaRN and to allocate
|
||||||
|
# cache buffers. Keep the workstation cache small, but retain the model's
|
||||||
|
# official 163,840-position YaRN contract for frequencies and scaling.
|
||||||
|
args.rope_reference_max_seq_len = config.max_position_embeddings
|
||||||
|
return args
|
||||||
|
|
||||||
|
|
||||||
|
def copy_v2_attention_weights(
|
||||||
|
module: Any,
|
||||||
|
state: dict[str, torch.Tensor],
|
||||||
|
device: torch.device,
|
||||||
|
dtype: torch.dtype,
|
||||||
|
) -> None:
|
||||||
|
mapping = {
|
||||||
|
"wq.weight": "q_proj.weight",
|
||||||
|
"wkv_a.weight": "kv_a_proj_with_mqa.weight",
|
||||||
|
"kv_norm.weight": "kv_a_layernorm.weight",
|
||||||
|
"wkv_b.weight": "kv_b_proj.weight",
|
||||||
|
"wo.weight": "o_proj.weight",
|
||||||
|
}
|
||||||
|
target = dict(module.named_parameters())
|
||||||
|
with torch.no_grad():
|
||||||
|
for destination, source in mapping.items():
|
||||||
|
target[destination].copy_(state[source].to(device=device, dtype=dtype))
|
||||||
|
|
||||||
|
|
||||||
|
def execute_v3_path(
|
||||||
|
v3: Any,
|
||||||
|
args: Any,
|
||||||
|
state: dict[str, torch.Tensor],
|
||||||
|
normalized: torch.Tensor,
|
||||||
|
device: torch.device,
|
||||||
|
dtype: torch.dtype,
|
||||||
|
implementation: str,
|
||||||
|
) -> tuple[torch.Tensor, dict[str, Any]]:
|
||||||
|
v3.attn_impl = implementation
|
||||||
|
v3.Linear.dtype = dtype
|
||||||
|
module = v3.MLA(args).to(device=device, dtype=dtype).eval()
|
||||||
|
copy_v2_attention_weights(module, state, device, dtype)
|
||||||
|
frequency_args = copy.copy(args)
|
||||||
|
frequency_args.max_seq_len = args.rope_reference_max_seq_len
|
||||||
|
frequencies = v3.precompute_freqs_cis(frequency_args).to(device)
|
||||||
|
if frequency_args.max_seq_len > args.original_seq_len:
|
||||||
|
mscale = 0.1 * args.mscale * math.log(args.rope_factor) + 1.0
|
||||||
|
module.softmax_scale = module.qk_head_dim ** -0.5 * mscale * mscale
|
||||||
|
prefill = normalized[:, :-1].to(dtype)
|
||||||
|
decode = normalized[:, -1:].to(dtype)
|
||||||
|
prefill_length = prefill.shape[1]
|
||||||
|
mask = torch.full(
|
||||||
|
(prefill_length, prefill_length),
|
||||||
|
float("-inf"),
|
||||||
|
device=device,
|
||||||
|
dtype=torch.float32,
|
||||||
|
).triu_(1)
|
||||||
|
with torch.inference_mode():
|
||||||
|
module(prefill, 0, frequencies[:prefill_length], mask)
|
||||||
|
output = module(
|
||||||
|
decode,
|
||||||
|
prefill_length,
|
||||||
|
frequencies[prefill_length : prefill_length + 1],
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
|
||||||
|
if implementation == "naive":
|
||||||
|
active = {
|
||||||
|
"key": {
|
||||||
|
"shape": [1, prefill_length + 1, args.n_heads, args.qk_nope_head_dim + args.qk_rope_head_dim],
|
||||||
|
"bytes": (prefill_length + 1)
|
||||||
|
* args.n_heads
|
||||||
|
* (args.qk_nope_head_dim + args.qk_rope_head_dim)
|
||||||
|
* torch.empty((), dtype=dtype).element_size(),
|
||||||
|
},
|
||||||
|
"value": {
|
||||||
|
"shape": [1, prefill_length + 1, args.n_heads, args.v_head_dim],
|
||||||
|
"bytes": (prefill_length + 1)
|
||||||
|
* args.n_heads
|
||||||
|
* args.v_head_dim
|
||||||
|
* torch.empty((), dtype=dtype).element_size(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
allocated = {
|
||||||
|
"key_shape": list(module.k_cache.shape),
|
||||||
|
"key_bytes": tensor_bytes(module.k_cache),
|
||||||
|
"value_shape": list(module.v_cache.shape),
|
||||||
|
"value_bytes": tensor_bytes(module.v_cache),
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
active = {
|
||||||
|
"latent": {
|
||||||
|
"shape": [1, prefill_length + 1, args.kv_lora_rank],
|
||||||
|
"bytes": (prefill_length + 1)
|
||||||
|
* args.kv_lora_rank
|
||||||
|
* torch.empty((), dtype=dtype).element_size(),
|
||||||
|
},
|
||||||
|
"rope": {
|
||||||
|
"shape": [1, prefill_length + 1, args.qk_rope_head_dim],
|
||||||
|
"bytes": (prefill_length + 1)
|
||||||
|
* args.qk_rope_head_dim
|
||||||
|
* torch.empty((), dtype=dtype).element_size(),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
allocated = {
|
||||||
|
"latent_shape": list(module.kv_cache.shape),
|
||||||
|
"latent_bytes": tensor_bytes(module.kv_cache),
|
||||||
|
"rope_shape": list(module.pe_cache.shape),
|
||||||
|
"rope_bytes": tensor_bytes(module.pe_cache),
|
||||||
|
}
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"implementation": implementation,
|
||||||
|
"dtype": str(dtype).replace("torch.", ""),
|
||||||
|
"active_cache": active,
|
||||||
|
"active_cache_bytes": sum(item["bytes"] for item in active.values()),
|
||||||
|
"allocated_cache": allocated,
|
||||||
|
"allocated_cache_bytes": sum(
|
||||||
|
value for key, value in allocated.items() if key.endswith("_bytes")
|
||||||
|
),
|
||||||
|
"decode_finite": bool(torch.isfinite(output).all()),
|
||||||
|
}
|
||||||
|
output = output.detach().cpu()
|
||||||
|
del module, frequencies, prefill, decode, mask
|
||||||
|
gc.collect()
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
return output, result
|
||||||
|
|
||||||
|
|
||||||
|
def execute_hf_eager(
|
||||||
|
modeling: Any,
|
||||||
|
config: Any,
|
||||||
|
state: dict[str, torch.Tensor],
|
||||||
|
normalized: torch.Tensor,
|
||||||
|
device: torch.device,
|
||||||
|
) -> tuple[torch.Tensor, dict[str, Any]]:
|
||||||
|
# A standalone DynamicCache is zero-indexed; the weights still come from
|
||||||
|
# decoder layer 1, while this isolated attention module occupies cache slot 0.
|
||||||
|
attention = modeling.DeepseekV2Attention(config, layer_idx=0)
|
||||||
|
attention.load_state_dict(state, strict=True, assign=True)
|
||||||
|
attention = attention.to(device=device, dtype=torch.bfloat16).eval()
|
||||||
|
prefill = normalized[:, :-1]
|
||||||
|
decode = normalized[:, -1:]
|
||||||
|
prefill_length = prefill.shape[1]
|
||||||
|
prefill_mask = modeling._prepare_4d_causal_attention_mask(
|
||||||
|
torch.ones((1, prefill_length), dtype=torch.long, device=device),
|
||||||
|
(1, prefill_length),
|
||||||
|
prefill,
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
cache = DynamicCache()
|
||||||
|
with torch.inference_mode():
|
||||||
|
attention(
|
||||||
|
prefill,
|
||||||
|
attention_mask=prefill_mask,
|
||||||
|
position_ids=torch.arange(prefill_length, device=device).unsqueeze(0),
|
||||||
|
past_key_value=cache,
|
||||||
|
use_cache=True,
|
||||||
|
)
|
||||||
|
output = attention(
|
||||||
|
decode,
|
||||||
|
attention_mask=torch.zeros(
|
||||||
|
(1, 1, 1, prefill_length + 1),
|
||||||
|
device=device,
|
||||||
|
dtype=decode.dtype,
|
||||||
|
),
|
||||||
|
position_ids=torch.tensor([[prefill_length]], device=device),
|
||||||
|
past_key_value=cache,
|
||||||
|
use_cache=True,
|
||||||
|
)[0]
|
||||||
|
key = cache.key_cache[0]
|
||||||
|
value = cache.value_cache[0]
|
||||||
|
result = {
|
||||||
|
"implementation": "huggingface_eager",
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"key_shape": list(key.shape),
|
||||||
|
"value_shape": list(value.shape),
|
||||||
|
"active_cache_bytes": tensor_bytes(key) + tensor_bytes(value),
|
||||||
|
"decode_finite": bool(torch.isfinite(output).all()),
|
||||||
|
}
|
||||||
|
output = output.detach().cpu()
|
||||||
|
del attention, cache, key, value, prefill, decode, prefill_mask
|
||||||
|
gc.collect()
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
return output, result
|
||||||
|
|
||||||
|
|
||||||
|
def nvidia_smi() -> dict[str, str]:
|
||||||
|
fields = ["name", "driver_version", "memory.total", "compute_cap"]
|
||||||
|
output = subprocess.check_output(
|
||||||
|
[
|
||||||
|
"nvidia-smi",
|
||||||
|
f"--query-gpu={','.join(fields)}",
|
||||||
|
"--format=csv,noheader,nounits",
|
||||||
|
],
|
||||||
|
text=True,
|
||||||
|
).strip()
|
||||||
|
return dict(zip(fields, [item.strip() for item in output.split(",")], strict=True))
|
||||||
|
|
||||||
|
|
||||||
|
def flashmla_boundary(root: Path, device: torch.device) -> dict[str, Any]:
|
||||||
|
readme = (root / "README.md").read_text()
|
||||||
|
setup = (root / "setup.py").read_text()
|
||||||
|
dense_header = (root / "csrc/api/dense_decode.h").read_text()
|
||||||
|
required = [
|
||||||
|
"| Dense Decoding | SM90 | MQA | BF16 |",
|
||||||
|
"| Sparse Decoding | SM90 & SM100 | MQA | FP8",
|
||||||
|
'"arch=compute_100f,code=sm_100f"',
|
||||||
|
'"arch=compute_90a,code=sm_90a"',
|
||||||
|
"Dense decode MLA is only supported on SM90a architecture",
|
||||||
|
]
|
||||||
|
combined = "\n".join((readme, setup, dense_header))
|
||||||
|
missing = [needle for needle in required if needle not in combined]
|
||||||
|
if missing:
|
||||||
|
raise RuntimeError(f"FlashMLA source contract changed: {missing}")
|
||||||
|
capability = torch.cuda.get_device_capability(device)
|
||||||
|
return {
|
||||||
|
"official_revision": FLASHMLA_REVISION,
|
||||||
|
"cutlass_revision": CUTLASS_REVISION,
|
||||||
|
"official_support_matrix": [
|
||||||
|
{"kernel": "dense_decode", "architectures": ["SM90"], "mode": "MQA", "cache": "BF16"},
|
||||||
|
{"kernel": "sparse_decode", "architectures": ["SM90", "SM100"], "mode": "MQA", "cache": "FP8"},
|
||||||
|
{"kernel": "dense_prefill", "architectures": ["SM100"], "mode": "MHA", "cache": None},
|
||||||
|
{"kernel": "sparse_prefill", "architectures": ["SM90", "SM100"], "mode": "MQA", "cache": None},
|
||||||
|
],
|
||||||
|
"compiled_gencode": ["sm_100f", "sm_90a"],
|
||||||
|
"dense_decode_runtime_guard": "Dense decode MLA is only supported on SM90a architecture",
|
||||||
|
"local_architecture": f"SM{capability[0]}{capability[1]}",
|
||||||
|
"optimized_kernel_executed": False,
|
||||||
|
"why_not": "The pinned official source does not list or generate SM120 kernels; dense decode additionally rejects non-SM90a devices.",
|
||||||
|
"build_attempts": [
|
||||||
|
{
|
||||||
|
"environment": "host CUDA 12.8 / g++ 15.2",
|
||||||
|
"result": "failed before compilation",
|
||||||
|
"evidence": "CUDA 12.8 rejects host compiler versions newer than g++ 13; g++-13 is not installed.",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"environment": "isolated CUDA 13.0.2 / torch 2.11.0+cu130 / g++ 13.3",
|
||||||
|
"result": "failed during official extension compilation",
|
||||||
|
"evidence": "csrc/api.cpp could not find cuda/std/utility; no wheel or runtime kernel was produced.",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"source_sha256": {
|
||||||
|
"readme": sha256(root / "README.md"),
|
||||||
|
"setup": sha256(root / "setup.py"),
|
||||||
|
"dense_decode_header": sha256(root / "csrc/api/dense_decode.h"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
args = parse_args()
|
||||||
|
artifact_root = args.artifact_dir.resolve()
|
||||||
|
v3_root = args.v3_repo.resolve()
|
||||||
|
flashmla_root = args.flashmla_repo.resolve()
|
||||||
|
shard = artifact_root / "model-00001-of-000004.safetensors"
|
||||||
|
required = [
|
||||||
|
shard,
|
||||||
|
artifact_root / "config.json",
|
||||||
|
artifact_root / "configuration_deepseek.py",
|
||||||
|
artifact_root / "modeling_deepseek.py",
|
||||||
|
artifact_root / "tokenizer.json",
|
||||||
|
v3_root / "inference/model.py",
|
||||||
|
flashmla_root / "README.md",
|
||||||
|
flashmla_root / "setup.py",
|
||||||
|
flashmla_root / "csrc/api/dense_decode.h",
|
||||||
|
]
|
||||||
|
missing = [str(path) for path in required if not path.exists()]
|
||||||
|
if missing:
|
||||||
|
raise FileNotFoundError(f"missing required official artifacts: {missing}")
|
||||||
|
if git_revision(v3_root) != V3_REVISION:
|
||||||
|
raise RuntimeError("DeepSeek-V3 revision does not match the probe contract")
|
||||||
|
if git_revision(flashmla_root) != FLASHMLA_REVISION:
|
||||||
|
raise RuntimeError("FlashMLA revision does not match the probe contract")
|
||||||
|
if args.cache_slots < 26:
|
||||||
|
raise ValueError("cache-slots must be at least 26")
|
||||||
|
device = torch.device(args.device)
|
||||||
|
if device.type == "cuda" and not torch.cuda.is_available():
|
||||||
|
raise RuntimeError("CUDA requested but unavailable")
|
||||||
|
|
||||||
|
torch.manual_seed(0)
|
||||||
|
configuration, modeling = load_v2_modules(artifact_root)
|
||||||
|
config = configuration.DeepseekV2Config.from_pretrained(artifact_root)
|
||||||
|
config._attn_implementation = "eager"
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(
|
||||||
|
artifact_root,
|
||||||
|
trust_remote_code=True,
|
||||||
|
local_files_only=True,
|
||||||
|
)
|
||||||
|
v3 = load_v3_module(v3_root)
|
||||||
|
|
||||||
|
normalized, attention_state, token_ids = load_real_layer1_input(
|
||||||
|
shard, modeling, config, tokenizer, device
|
||||||
|
)
|
||||||
|
hf_output, hf_result = execute_hf_eager(
|
||||||
|
modeling, config, attention_state, normalized, device
|
||||||
|
)
|
||||||
|
model_args = v3_args(v3, config, args.cache_slots)
|
||||||
|
naive_bf16, naive_result = execute_v3_path(
|
||||||
|
v3,
|
||||||
|
model_args,
|
||||||
|
attention_state,
|
||||||
|
normalized,
|
||||||
|
device,
|
||||||
|
torch.bfloat16,
|
||||||
|
"naive",
|
||||||
|
)
|
||||||
|
absorb_bf16, absorb_result = execute_v3_path(
|
||||||
|
v3,
|
||||||
|
model_args,
|
||||||
|
attention_state,
|
||||||
|
normalized,
|
||||||
|
device,
|
||||||
|
torch.bfloat16,
|
||||||
|
"absorb",
|
||||||
|
)
|
||||||
|
naive_fp32, naive_fp32_result = execute_v3_path(
|
||||||
|
v3,
|
||||||
|
model_args,
|
||||||
|
attention_state,
|
||||||
|
normalized,
|
||||||
|
device,
|
||||||
|
torch.float32,
|
||||||
|
"naive",
|
||||||
|
)
|
||||||
|
absorb_fp32, absorb_fp32_result = execute_v3_path(
|
||||||
|
v3,
|
||||||
|
model_args,
|
||||||
|
attention_state,
|
||||||
|
normalized,
|
||||||
|
device,
|
||||||
|
torch.float32,
|
||||||
|
"absorb",
|
||||||
|
)
|
||||||
|
|
||||||
|
naive_bytes = naive_result["active_cache_bytes"]
|
||||||
|
absorb_bytes = absorb_result["active_cache_bytes"]
|
||||||
|
result = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"captured_at": args.captured_at or datetime.now(timezone.utc).isoformat(),
|
||||||
|
"evidence_identity": "X / official V2-Lite weights executed through official HF eager and V3 naive/absorb reference paths",
|
||||||
|
"boundary": {
|
||||||
|
"correctness_probe": True,
|
||||||
|
"serving_benchmark": False,
|
||||||
|
"full_model_generation": False,
|
||||||
|
"optimized_flashmla_kernel_executed": False,
|
||||||
|
"scope": "one real layer-1 attention input; 25-token prefill plus one-token decode",
|
||||||
|
},
|
||||||
|
"provenance": {
|
||||||
|
"huggingface_model": "deepseek-ai/DeepSeek-V2-Lite",
|
||||||
|
"huggingface_revision": HF_REVISION,
|
||||||
|
"deepseek_v3_repository": "deepseek-ai/DeepSeek-V3",
|
||||||
|
"deepseek_v3_revision": V3_REVISION,
|
||||||
|
"flashmla_repository": "deepseek-ai/FlashMLA",
|
||||||
|
"flashmla_revision": FLASHMLA_REVISION,
|
||||||
|
"sha256": {
|
||||||
|
"v2_config": sha256(artifact_root / "config.json"),
|
||||||
|
"v2_modeling": sha256(artifact_root / "modeling_deepseek.py"),
|
||||||
|
"v2_shard_1": sha256(shard),
|
||||||
|
"v3_model": sha256(v3_root / "inference/model.py"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"environment": {
|
||||||
|
"python": platform.python_version(),
|
||||||
|
"platform": platform.platform(),
|
||||||
|
"libc": list(platform.libc_ver()),
|
||||||
|
"torch": torch.__version__,
|
||||||
|
"torch_cuda": torch.version.cuda,
|
||||||
|
"transformers": __import__("transformers").__version__,
|
||||||
|
"safetensors": __import__("safetensors").__version__,
|
||||||
|
"device": str(device),
|
||||||
|
"cuda_capability": list(torch.cuda.get_device_capability(device)),
|
||||||
|
"nvidia_smi": nvidia_smi(),
|
||||||
|
},
|
||||||
|
"input": {
|
||||||
|
"prompt": PROMPT,
|
||||||
|
"token_ids": token_ids,
|
||||||
|
"tokens": tokenizer.convert_ids_to_tokens(token_ids),
|
||||||
|
"sequence_tokens": len(token_ids),
|
||||||
|
"prefill_tokens": len(token_ids) - 1,
|
||||||
|
"decode_tokens": 1,
|
||||||
|
"layer0_executed": True,
|
||||||
|
"layer1_input_normalized": True,
|
||||||
|
},
|
||||||
|
"configuration": {
|
||||||
|
"hidden": config.hidden_size,
|
||||||
|
"attention_heads": config.num_attention_heads,
|
||||||
|
"qk_nope_head_dim": config.qk_nope_head_dim,
|
||||||
|
"qk_rope_head_dim": config.qk_rope_head_dim,
|
||||||
|
"v_head_dim": config.v_head_dim,
|
||||||
|
"kv_lora_rank": config.kv_lora_rank,
|
||||||
|
"cache_slots": args.cache_slots,
|
||||||
|
},
|
||||||
|
"execution": {
|
||||||
|
"huggingface_eager": hf_result,
|
||||||
|
"v3_naive_bfloat16": naive_result,
|
||||||
|
"v3_absorb_bfloat16": absorb_result,
|
||||||
|
"v3_naive_float32": naive_fp32_result,
|
||||||
|
"v3_absorb_float32": absorb_fp32_result,
|
||||||
|
},
|
||||||
|
"correctness": {
|
||||||
|
"hf_eager_vs_v3_naive_bfloat16": compare(hf_output, naive_bf16),
|
||||||
|
"v3_naive_vs_absorb_bfloat16": compare(naive_bf16, absorb_bf16),
|
||||||
|
"hf_eager_vs_v3_absorb_bfloat16": compare(hf_output, absorb_bf16),
|
||||||
|
"v3_naive_vs_absorb_float32": compare(naive_fp32, absorb_fp32),
|
||||||
|
},
|
||||||
|
"cache_accounting": {
|
||||||
|
"active_tokens": len(token_ids),
|
||||||
|
"dtype": "BF16",
|
||||||
|
"naive_active_bytes": naive_bytes,
|
||||||
|
"absorb_active_bytes": absorb_bytes,
|
||||||
|
"naive_elements_per_token": config.num_attention_heads
|
||||||
|
* (
|
||||||
|
config.qk_nope_head_dim
|
||||||
|
+ config.qk_rope_head_dim
|
||||||
|
+ config.v_head_dim
|
||||||
|
),
|
||||||
|
"absorb_elements_per_token": config.kv_lora_rank
|
||||||
|
+ config.qk_rope_head_dim,
|
||||||
|
"naive_over_absorb_ratio": naive_bytes / absorb_bytes,
|
||||||
|
"absorb_reduction_vs_naive": 1 - absorb_bytes / naive_bytes,
|
||||||
|
},
|
||||||
|
"flashmla_boundary": flashmla_boundary(flashmla_root, device),
|
||||||
|
}
|
||||||
|
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
args.output.write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n")
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,420 @@
|
|||||||
|
# DeepSeek MLA 吸收式缓存：真实权重执行与 FlashMLA 架构边界
|
||||||
|
|
||||||
|
> 状态：DeepSeek 第三轮第二里程碑
|
||||||
|
> 证据等级：`O` 官方源码 / `X` 本机真实执行 / `D` 确定推导 / `U` 未覆盖
|
||||||
|
> 结论边界：正确性与缓存格式实验，不是服务吞吐 benchmark
|
||||||
|
|
||||||
|
## 0. 先给结论
|
||||||
|
|
||||||
|
这轮补上了上一次实验最重要的空白：
|
||||||
|
|
||||||
|
- 上一次已经观察到 V2-Lite 的真实压缩投影是 `[4, 27, 576]`；
|
||||||
|
- 但 Hugging Face 官方 eager 路径实际缓存展开后的
|
||||||
|
`K [4,16,27,192]` 与 `V [4,16,27,128]`；
|
||||||
|
- 这一次把同一组真实 V2-Lite layer-1 权重映射到 DeepSeek-V3 官方
|
||||||
|
`inference/model.py`，真正执行了 `attn_impl="absorb"`；
|
||||||
|
- 25-token prefill + 1-token decode 后，官方 buffer 中实际存在：
|
||||||
|
- `kv_cache [1,32,512]`，其中 26-token active slice 为 `26,624 B`；
|
||||||
|
- `pe_cache [1,32,64]`，其中 26-token active slice 为 `3,328 B`；
|
||||||
|
- active 总计 `29,952 B`。
|
||||||
|
- 同一官方参考类的 naive 路径实际 active cache 为：
|
||||||
|
- `k_cache`：`159,744 B`；
|
||||||
|
- `v_cache`：`106,496 B`；
|
||||||
|
- active 总计 `266,240 B`。
|
||||||
|
- 因而在这个 V2-Lite 配置、BF16、同一 26-token 输入下：
|
||||||
|
|
||||||
|
```text
|
||||||
|
naive / absorb = 266,240 / 29,952 = 8.888888…
|
||||||
|
reduction = 1 - 29,952 / 266,240 = 88.75%
|
||||||
|
```
|
||||||
|
|
||||||
|
这不再只是 shape 算术：两组官方 cache buffers 都已在 RTX 5090 上实际分配、写入并用于
|
||||||
|
incremental decode。
|
||||||
|
|
||||||
|
但是另一条边界同样重要：
|
||||||
|
|
||||||
|
- DeepSeek-V3 的官方纯 PyTorch `absorb` 参考路径：**已执行**；
|
||||||
|
- DeepSeek FlashMLA 优化 kernel：**未执行**；
|
||||||
|
- 原因不是把环境问题猜成性能结论，而是 pinned FlashMLA 官方支持矩阵与源码只覆盖
|
||||||
|
SM90 / SM100，本机 RTX 5090 是 SM120。
|
||||||
|
|
||||||
|
## 1. 可复现身份
|
||||||
|
|
||||||
|
### 1.1 模型与源码
|
||||||
|
|
||||||
|
| 工件 | 固定身份 |
|
||||||
|
|---|---|
|
||||||
|
| Hugging Face 模型 | `deepseek-ai/DeepSeek-V2-Lite` |
|
||||||
|
| HF revision | `604d5664dddd88a0433dbae533b7fe9472482de0` |
|
||||||
|
| DeepSeek-V3 仓库 | `deepseek-ai/DeepSeek-V3` |
|
||||||
|
| V3 revision | `9b4e9788e4a3a731f7567338ed15d3ec549ce03b` |
|
||||||
|
| FlashMLA 仓库 | `deepseek-ai/FlashMLA` |
|
||||||
|
| FlashMLA revision | `15f13e5030374295491c5ce31b02d7e63a7772c6` |
|
||||||
|
| FlashMLA CUTLASS submodule | `147f5673d0c1c3dcf66f78d677fd647e4a020219` |
|
||||||
|
|
||||||
|
官方入口：
|
||||||
|
|
||||||
|
- [DeepSeek-V2-Lite 模型](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite)
|
||||||
|
- [DeepSeek-V2 仓库](https://github.com/deepseek-ai/DeepSeek-V2)
|
||||||
|
- [DeepSeek-V2 技术报告](https://arxiv.org/abs/2405.04434)
|
||||||
|
- [DeepSeek-V3 inference/model.py](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py)
|
||||||
|
- [FlashMLA](https://github.com/deepseek-ai/FlashMLA)
|
||||||
|
|
||||||
|
### 1.2 本机环境
|
||||||
|
|
||||||
|
| 字段 | 值 |
|
||||||
|
|---|---|
|
||||||
|
| GPU | NVIDIA GeForce RTX 5090 |
|
||||||
|
| compute capability | `12.0` / SM120 |
|
||||||
|
| driver | `595.84` |
|
||||||
|
| torch | `2.11.0+cu128` |
|
||||||
|
| CUDA runtime | `12.8` |
|
||||||
|
| Transformers | 隔离的 `4.41.2` |
|
||||||
|
| checkpoint dtype | BF16 |
|
||||||
|
|
||||||
|
V2-Lite 的 2024 官方 remote code 不能直接在当前 Transformers 5.5 中导入，因此延续上一轮合同：
|
||||||
|
官方模型源码不打补丁，只把依赖固定到隔离目录中的 Transformers 4.41.2。
|
||||||
|
|
||||||
|
### 1.3 输出
|
||||||
|
|
||||||
|
- 主结果：`src/data/deepseek-v2-lite-absorb.json`
|
||||||
|
- 独立复跑：`src/data/deepseek-v2-lite-absorb-repro.json`
|
||||||
|
- 两份完整 JSON SHA-256：
|
||||||
|
`6b4c714a470baa6d6b04982cd289e693ec863398b5def4e6df541cb10525e63d`
|
||||||
|
- `cmp`：逐字节一致
|
||||||
|
|
||||||
|
实验没有计时字段，因此不需要用“排除 timing”掩盖不同；两次输出完整 byte-exact。
|
||||||
|
|
||||||
|
## 2. 为什么权重可以“吸收”
|
||||||
|
|
||||||
|
### 2.1 普通展开路径
|
||||||
|
|
||||||
|
令每个历史 token 的 512 维压缩状态为 `cₜ`。忽略位置分支时，每个 attention head 的
|
||||||
|
key/value 可以写成：
|
||||||
|
|
||||||
|
```text
|
||||||
|
kₜ = Wᴷ cₜ
|
||||||
|
vₜ = Wⱽ cₜ
|
||||||
|
```
|
||||||
|
|
||||||
|
naive 实现先把所有历史 `cₜ` 展开成每个 head 的 `kₜ` 和 `vₜ`，再缓存展开结果。V2-Lite
|
||||||
|
有 16 个 head：
|
||||||
|
|
||||||
|
```text
|
||||||
|
key = 16 × (128 no-RoPE + 64 RoPE) = 3,072 elements/token
|
||||||
|
value = 16 × 128 = 2,048 elements/token
|
||||||
|
total = 5,120 elements/token/layer
|
||||||
|
```
|
||||||
|
|
||||||
|
问题不是这些张量算错了，而是同一个 512 维内容状态被投影并复制到各个 head 后才进入缓存。
|
||||||
|
|
||||||
|
### 2.2 把 key 投影搬到 query
|
||||||
|
|
||||||
|
注意力分数里的 no-RoPE 部分：
|
||||||
|
|
||||||
|
```text
|
||||||
|
qᵀkₜ
|
||||||
|
= qᵀ(Wᴷcₜ)
|
||||||
|
= (Wᴷᵀq)ᵀcₜ
|
||||||
|
```
|
||||||
|
|
||||||
|
矩阵乘法结合律允许先把 query 变成 `Wᴷᵀq`，再直接与历史 latent `cₜ` 点积。于是历史状态不必
|
||||||
|
缓存每个 head 展开后的 `Wᴷcₜ`。
|
||||||
|
|
||||||
|
这里的“吸收”不是删掉参数：
|
||||||
|
|
||||||
|
- `Wᴷ` 仍然存在；
|
||||||
|
- 它从“写 cache 前作用在每个历史 token 上”，搬到“读 cache 时作用在当前 query 上”；
|
||||||
|
- 权重的作用位置改变，数学函数在精确算术中不变。
|
||||||
|
|
||||||
|
### 2.3 把 value 投影搬到 attention 后
|
||||||
|
|
||||||
|
attention 的 value 聚合：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Σₜ pₜvₜ
|
||||||
|
= Σₜ pₜ(Wⱽcₜ)
|
||||||
|
= Wⱽ(Σₜ pₜcₜ)
|
||||||
|
```
|
||||||
|
|
||||||
|
因此可以先用注意力概率 `pₜ` 对 512 维 latent 做加权和，再把聚合结果通过 `Wⱽ` 展开到各个
|
||||||
|
head 的 value 空间。
|
||||||
|
|
||||||
|
naive 顺序：
|
||||||
|
|
||||||
|
```text
|
||||||
|
每个历史 cₜ → Wⱽ → 展开 vₜ → cache → 按 pₜ 加权
|
||||||
|
```
|
||||||
|
|
||||||
|
absorb 顺序：
|
||||||
|
|
||||||
|
```text
|
||||||
|
每个历史 cₜ → cache → 按 pₜ 加权 → Wⱽ → 展开输出
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.4 为什么 64 维 RoPE key 仍要缓存
|
||||||
|
|
||||||
|
RoPE 是位置相关变换。历史 token `t` 的 key 旋转角度依赖位置 `t`，不能像固定线性投影那样无条件
|
||||||
|
搬到当前 query 一侧。因此 V2-Lite 的可缓存状态不是只有 512：
|
||||||
|
|
||||||
|
```text
|
||||||
|
512 KV latent + 64 decoupled RoPE key = 576 elements/token/layer
|
||||||
|
```
|
||||||
|
|
||||||
|
这也是网站从第一轮起始终把“512 latent”与“576 完整缓存合同”分开的原因。
|
||||||
|
|
||||||
|
## 3. 实验怎样避免偷换输入
|
||||||
|
|
||||||
|
### 3.1 输入不是随机 hidden state
|
||||||
|
|
||||||
|
固定中文 prompt：
|
||||||
|
|
||||||
|
```text
|
||||||
|
用通俗的语言解释，为什么稀疏专家模型可以拥有很多参数，但每个 token 只使用其中一小部分。
|
||||||
|
```
|
||||||
|
|
||||||
|
官方 tokenizer 得到 26 个 token。执行链是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
官方 tokenizer
|
||||||
|
→ 官方 embedding 权重
|
||||||
|
→ 官方 V2-Lite decoder layer 0 完整 forward
|
||||||
|
→ 官方 layer-1 input RMSNorm
|
||||||
|
→ 真实 layer-1 attention input
|
||||||
|
```
|
||||||
|
|
||||||
|
因此 attention 的输入来自 checkpoint 执行轨迹，不是为了让误差更小而构造的随机数。
|
||||||
|
|
||||||
|
### 3.2 为什么只测 layer 1 attention
|
||||||
|
|
||||||
|
第一 checkpoint 分片完整包含：
|
||||||
|
|
||||||
|
- embedding；
|
||||||
|
- layer 0 全部参数；
|
||||||
|
- layer 1 全部 attention 参数与 input RMSNorm。
|
||||||
|
|
||||||
|
本里程碑问的是“吸收路径能否用真实权重兑现 576 元素缓存”，不需要再次运行 MoE。只隔离 layer-1
|
||||||
|
attention，可以同时做到：
|
||||||
|
|
||||||
|
- 输入由真实前层产生；
|
||||||
|
- attention 权重全部来自官方 checkpoint；
|
||||||
|
- naive 与 absorb 使用完全相同的输入和参数；
|
||||||
|
- 不把未下载的完整模型写成已复现。
|
||||||
|
|
||||||
|
### 3.3 权重怎样映射
|
||||||
|
|
||||||
|
V2-Lite HF remote code 与 V3 官方 inference reference 的张量语义一致：
|
||||||
|
|
||||||
|
| V2-Lite HF | V3 reference |
|
||||||
|
|---|---|
|
||||||
|
| `q_proj.weight` | `wq.weight` |
|
||||||
|
| `kv_a_proj_with_mqa.weight` | `wkv_a.weight` |
|
||||||
|
| `kv_a_layernorm.weight` | `kv_norm.weight` |
|
||||||
|
| `kv_b_proj.weight` | `wkv_b.weight` |
|
||||||
|
| `o_proj.weight` | `wo.weight` |
|
||||||
|
|
||||||
|
没有转置 checkpoint、没有重新训练、没有拟合中间 adapter。
|
||||||
|
|
||||||
|
### 3.4 三条 BF16 路径
|
||||||
|
|
||||||
|
同一 26-token 输入切成：
|
||||||
|
|
||||||
|
- prefill：前 25 token；
|
||||||
|
- incremental decode：最后 1 token。
|
||||||
|
|
||||||
|
执行：
|
||||||
|
|
||||||
|
1. 官方 V2-Lite Hugging Face eager attention；
|
||||||
|
2. pinned V3 官方 `attn_impl="naive"`；
|
||||||
|
3. pinned V3 官方 `attn_impl="absorb"`。
|
||||||
|
|
||||||
|
HF eager 与 V3 naive 的对照先验证“权重映射 + YaRN 配置 + causal/incremental 路径”没有发生大幅漂移；
|
||||||
|
V3 naive 与 V3 absorb 的对照再隔离权重吸收本身。
|
||||||
|
|
||||||
|
## 4. 真实 cache 观测
|
||||||
|
|
||||||
|
### 4.1 BF16 active slice
|
||||||
|
|
||||||
|
| 路径 | 实际 shape | bytes |
|
||||||
|
|---|---|---:|
|
||||||
|
| V3 naive key | `[1,26,16,192]` | 159,744 |
|
||||||
|
| V3 naive value | `[1,26,16,128]` | 106,496 |
|
||||||
|
| **V3 naive total** | 26 × 5,120 BF16 | **266,240** |
|
||||||
|
| V3 absorb latent | `[1,26,512]` | 26,624 |
|
||||||
|
| V3 absorb RoPE | `[1,26,64]` | 3,328 |
|
||||||
|
| **V3 absorb total** | 26 × 576 BF16 | **29,952** |
|
||||||
|
|
||||||
|
### 4.2 32-slot 实际 buffer
|
||||||
|
|
||||||
|
| 路径 | 分配 shape | allocated bytes |
|
||||||
|
|---|---|---:|
|
||||||
|
| V3 naive key | `[1,32,16,192]` | 196,608 |
|
||||||
|
| V3 naive value | `[1,32,16,128]` | 131,072 |
|
||||||
|
| **V3 naive total** | | **327,680** |
|
||||||
|
| V3 absorb latent | `[1,32,512]` | 32,768 |
|
||||||
|
| V3 absorb RoPE | `[1,32,64]` | 4,096 |
|
||||||
|
| **V3 absorb total** | | **36,864** |
|
||||||
|
|
||||||
|
active 与 allocated 分开记录，是为了不把 32-slot 容器的尾部空位算成 26-token 状态。
|
||||||
|
|
||||||
|
## 5. 数值正确性
|
||||||
|
|
||||||
|
所有输出 shape 都是 `[1,1,2048]`，全部 finite。
|
||||||
|
|
||||||
|
### 5.1 HF eager 与 V3 naive
|
||||||
|
|
||||||
|
| 指标 | BF16 |
|
||||||
|
|---|---:|
|
||||||
|
| max absolute error | 0.001220703125 |
|
||||||
|
| mean absolute error | 0.0000888268 |
|
||||||
|
| p99 absolute error | 0.0003823861 |
|
||||||
|
| exact fraction | 0.284668 |
|
||||||
|
|
||||||
|
两份官方实现的张量布局、einsum/matmul 顺序不同，BF16 不能要求逐元素完全相等。这个对照的作用是检查
|
||||||
|
映射与配置；它不是“证明两个代码库 bit-exact”。
|
||||||
|
|
||||||
|
### 5.2 V3 naive 与 V3 absorb
|
||||||
|
|
||||||
|
| 指标 | BF16 |
|
||||||
|
|---|---:|
|
||||||
|
| max absolute error | 0.00390625 |
|
||||||
|
| mean absolute error | 0.0002703113 |
|
||||||
|
| p99 absolute error | 0.0009236149 |
|
||||||
|
| exact fraction | 0.100098 |
|
||||||
|
|
||||||
|
absorb 改变了矩阵乘法结合顺序；有限精度下，先乘什么会改变舍入位置。因此 BF16 输出接近但不
|
||||||
|
bit-exact，符合预期。
|
||||||
|
|
||||||
|
### 5.3 FP32 代数审计
|
||||||
|
|
||||||
|
为了区分“公式不等价”和“BF16 舍入顺序”，实验把同一组 BF16 checkpoint 权重与真实输入提升为 FP32，
|
||||||
|
重新执行 V3 naive/absorb：
|
||||||
|
|
||||||
|
| 指标 | FP32 |
|
||||||
|
|---|---:|
|
||||||
|
| max absolute error | `1.1920929e-7` |
|
||||||
|
| mean absolute error | `9.4644861e-9` |
|
||||||
|
| p99 absolute error | `3.1446120e-8` |
|
||||||
|
|
||||||
|
这不是声称存在 FP32 checkpoint，也不是模型精度评测。它是一个代数隔离实验：精度提高后，两种结合
|
||||||
|
顺序的误差下降到 FP32 舍入量级。
|
||||||
|
|
||||||
|
## 6. FlashMLA：为什么没有在 RTX 5090 上宣称成功
|
||||||
|
|
||||||
|
### 6.1 官方支持矩阵
|
||||||
|
|
||||||
|
pinned FlashMLA README：
|
||||||
|
|
||||||
|
| kernel | GPU architecture | mode | KV cache |
|
||||||
|
|---|---|---|---|
|
||||||
|
| dense decoding | SM90 | MQA | BF16 |
|
||||||
|
| sparse decoding | SM90 & SM100 | MQA | FP8 |
|
||||||
|
| dense prefill | SM100 | MHA | — |
|
||||||
|
| sparse prefill | SM90 & SM100 | MQA | — |
|
||||||
|
|
||||||
|
README 对 MQA mode 的定义是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
head_dim_k = 576
|
||||||
|
head_dim_v = 512
|
||||||
|
```
|
||||||
|
|
||||||
|
这里的 `head_dim_v=512` 是吸收后的 latent 聚合空间，不应与 naive 路径每个 query head 的
|
||||||
|
`v_head_dim=128` 混为一谈。
|
||||||
|
|
||||||
|
### 6.2 编译目标与运行时 guard
|
||||||
|
|
||||||
|
pinned `setup.py` 只加入：
|
||||||
|
|
||||||
|
```text
|
||||||
|
arch=compute_100f,code=sm_100f
|
||||||
|
arch=compute_90a,code=sm_90a
|
||||||
|
```
|
||||||
|
|
||||||
|
`csrc/api/dense_decode.h` 还显式检查：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Dense decode MLA is only supported on SM90a architecture
|
||||||
|
```
|
||||||
|
|
||||||
|
本机是 SM120。因此即使 CUDA 工具链能把扩展编译完成，也不能从这份官方源码推导出 SM120 dense
|
||||||
|
decode 支持。
|
||||||
|
|
||||||
|
### 6.3 两次构建尝试
|
||||||
|
|
||||||
|
| 环境 | 结果 | 停止位置 |
|
||||||
|
|---|---|---|
|
||||||
|
| host CUDA 12.8 + g++ 15.2 | 失败 | CUDA 12.8 在正式编译前拒绝过新的 host compiler；本机没有 g++-13 |
|
||||||
|
| 隔离 CUDA 13.0.2 + torch 2.11.0+cu130 + g++ 13.3 | 失败 | 官方 extension 编译 `csrc/api.cpp` 时找不到 `cuda/std/utility` |
|
||||||
|
|
||||||
|
第二次构建中的其它 CUDA translation units 继续输出日志，是 Ninja 并行任务已经启动的结果；最终
|
||||||
|
`ninja: build stopped: subcommand failed`，没有 wheel。
|
||||||
|
|
||||||
|
正确的实验状态是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
official V3 pure-PyTorch absorb reference: EXECUTED
|
||||||
|
official FlashMLA optimized kernel on SM120: NOT EXECUTED / NOT LISTED AS SUPPORTED
|
||||||
|
```
|
||||||
|
|
||||||
|
不能把前者的成功写成后者的成功，也不能把后者的未支持写成“5090 做不了 MLA”。
|
||||||
|
|
||||||
|
## 7. 与 V2 报告 93.3% 的关系
|
||||||
|
|
||||||
|
本实验得到的 `88.75%` 是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
V2-Lite 576-element latent contract
|
||||||
|
vs
|
||||||
|
V2-Lite 5,120-element HF/V3-naive expanded cache
|
||||||
|
```
|
||||||
|
|
||||||
|
V2 技术报告的 `93.3%` 属于论文中的完整 V2 模型、论文基线与论文配置。两者回答的口径不同。
|
||||||
|
|
||||||
|
可以说：
|
||||||
|
|
||||||
|
- V2-Lite 本地吸收缓存相对本地展开格式减少 88.75%；
|
||||||
|
- 这轮已经从 shape 推导升级为真实 buffer 与 decode 执行；
|
||||||
|
- 结果支持 MLA 权重吸收机制。
|
||||||
|
|
||||||
|
不能说：
|
||||||
|
|
||||||
|
- 本实验“复现了论文 93.3%”；
|
||||||
|
- 88.75% 推翻了 93.3%；
|
||||||
|
- reference 实现的缓存降幅就是生产吞吐提升；
|
||||||
|
- SM120 已运行 FlashMLA。
|
||||||
|
|
||||||
|
## 8. 复现命令
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=/tmp/deepseek-v2-lite-pydeps.6ZJAH3 \
|
||||||
|
/home/wuyang/.pyenv/versions/navi-router-cu128/bin/python \
|
||||||
|
experiments/deepseek/v2_lite_absorb_probe.py \
|
||||||
|
--artifact-dir /tmp/deepseek-v2-lite-artifacts.OEjfce \
|
||||||
|
--v3-repo /tmp/llm-atlas-deepseek-v3-official \
|
||||||
|
--flashmla-repo /tmp/llm-atlas-flashmla-official \
|
||||||
|
--output src/data/deepseek-v2-lite-absorb.json \
|
||||||
|
--captured-at 2026-07-29T06:35:00Z
|
||||||
|
```
|
||||||
|
|
||||||
|
第二次改写 `--output` 到
|
||||||
|
`src/data/deepseek-v2-lite-absorb-repro.json`，然后：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sha256sum \
|
||||||
|
src/data/deepseek-v2-lite-absorb.json \
|
||||||
|
src/data/deepseek-v2-lite-absorb-repro.json
|
||||||
|
|
||||||
|
cmp -s \
|
||||||
|
src/data/deepseek-v2-lite-absorb.json \
|
||||||
|
src/data/deepseek-v2-lite-absorb-repro.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. 下一轮应该做什么
|
||||||
|
|
||||||
|
1. 在官方明确支持的 SM90/Hopper 环境执行 FlashMLA dense decode correctness test；
|
||||||
|
2. 把同一组 latent cache 接入 serving runtime，分开测 prefill、decode、batch 与 context；
|
||||||
|
3. 同时记录吞吐、TTFT、TPOT、峰值显存和数值误差，避免只报一个“快多少”；
|
||||||
|
4. 扩展到多个 layer 与多个 prompt，检查误差如何累计；
|
||||||
|
5. 若 FlashMLA 官方未来加入 SM120，固定新 revision 后重开支持矩阵审计，不能把未来支持倒写到本次记录。
|
||||||
|
|
||||||
@@ -276,6 +276,29 @@ const artifactCache = await evaluate(`(() => {
|
|||||||
return { trace, million };
|
return { trace, million };
|
||||||
})()`);
|
})()`);
|
||||||
|
|
||||||
|
const artifactAbsorb = await evaluate(`(() => {
|
||||||
|
const root = document.querySelector("[data-dsv2-lab]");
|
||||||
|
root.querySelector('[data-artifact-tab="absorb"]').click();
|
||||||
|
return {
|
||||||
|
panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
|
||||||
|
algebra: root.querySelectorAll(".absorb-algebra article").length,
|
||||||
|
naive: root.querySelector(".absorb-cache-flow .naive header b").textContent.trim(),
|
||||||
|
absorbed: root.querySelector(".absorb-cache-flow .absorbed header b").textContent.trim(),
|
||||||
|
metrics: [...root.querySelectorAll(".absorb-metrics article b")].map((node) => node.textContent.trim()),
|
||||||
|
precisionRows: root.querySelectorAll(".precision-lens > div").length,
|
||||||
|
matrixRows: root.querySelectorAll(".kernel-matrix > div").length,
|
||||||
|
localUnsupported: root.querySelectorAll(".kernel-matrix i.no").length,
|
||||||
|
executionCards: root.querySelectorAll(".execution-split article").length,
|
||||||
|
boundary: root.querySelector('[data-artifact-panel="absorb"] .artifact-boundary').textContent.replaceAll(/\\s+/g, " ").trim(),
|
||||||
|
};
|
||||||
|
})()`);
|
||||||
|
await evaluate(`(() => {
|
||||||
|
document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
|
window.scrollBy(0, -82);
|
||||||
|
})()`);
|
||||||
|
await pause(180);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-absorb-desktop.png");
|
||||||
|
|
||||||
const artifactEvidence = await evaluate(`(() => {
|
const artifactEvidence = await evaluate(`(() => {
|
||||||
const root = document.querySelector("[data-dsv2-lab]");
|
const root = document.querySelector("[data-dsv2-lab]");
|
||||||
root.querySelector('[data-artifact-tab="evidence"]').click();
|
root.querySelector('[data-artifact-tab="evidence"]').click();
|
||||||
@@ -361,13 +384,15 @@ const mobile = await evaluate(`(() => {
|
|||||||
})()`);
|
})()`);
|
||||||
await evaluate(`(() => {
|
await evaluate(`(() => {
|
||||||
document.querySelector("#menu-toggle")?.click();
|
document.querySelector("#menu-toggle")?.click();
|
||||||
document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
|
const artifact = document.querySelector("[data-dsv2-lab]");
|
||||||
|
artifact.querySelector('[data-artifact-tab="absorb"]').click();
|
||||||
|
artifact.scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
window.scrollBy(0, -82);
|
window.scrollBy(0, -82);
|
||||||
})()`);
|
})()`);
|
||||||
await pause(180);
|
await pause(180);
|
||||||
await screenshot("/tmp/llm-atlas-deepseek-artifact-mobile.png");
|
await screenshot("/tmp/llm-atlas-deepseek-artifact-mobile.png");
|
||||||
|
|
||||||
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactEvidence, home, papers, mobile, exceptions };
|
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactEvidence, home, papers, mobile, exceptions };
|
||||||
console.log(JSON.stringify(report, null, 2));
|
console.log(JSON.stringify(report, null, 2));
|
||||||
|
|
||||||
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
||||||
@@ -377,7 +402,7 @@ if (overview.sections !== 26 || overview.tocLinks !== 26) failures.push("二十
|
|||||||
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
|
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
|
||||||
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
|
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
|
||||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||||
if (overview.artifactTabs !== 4 || overview.artifactPanels !== 4 || overview.artifactLayers !== 27) failures.push("真实权重四联实验结构异常");
|
if (overview.artifactTabs !== 5 || overview.artifactPanels !== 5 || overview.artifactLayers !== 27) failures.push("真实权重五联实验结构异常");
|
||||||
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
|
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
|
||||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
||||||
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
|
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
|
||||||
@@ -401,12 +426,15 @@ if (artifactLoad.layer1.panel !== "load" || artifactLoad.layer1.used !== "63" ||
|
|||||||
if (artifactLoad.layer2.used !== "64" || artifactLoad.layer2.cv !== "0.549" || artifactLoad.layer4.used !== "62" || artifactLoad.layer4.gini !== "0.417") failures.push("跨层负载统计切换异常");
|
if (artifactLoad.layer2.used !== "64" || artifactLoad.layer2.cv !== "0.549" || artifactLoad.layer4.used !== "62" || artifactLoad.layer4.gini !== "0.417") failures.push("跨层负载统计切换异常");
|
||||||
if (artifactCache.trace.panel !== "cache" || artifactCache.trace.latent !== "850.50 KiB" || artifactCache.trace.eager !== "7.38 MiB" || artifactCache.trace.ratio !== "8.89×" || artifactCache.trace.reduction !== "88.75%") failures.push("V2-Lite trace 缓存实现账异常");
|
if (artifactCache.trace.panel !== "cache" || artifactCache.trace.latent !== "850.50 KiB" || artifactCache.trace.eager !== "7.38 MiB" || artifactCache.trace.ratio !== "8.89×" || artifactCache.trace.reduction !== "88.75%") failures.push("V2-Lite trace 缓存实现账异常");
|
||||||
if (!artifactCache.million.latent.includes("GiB") || !artifactCache.million.eager.includes("TiB")) failures.push("V2-Lite 百万 Token 缓存外推异常");
|
if (!artifactCache.million.latent.includes("GiB") || !artifactCache.million.eager.includes("TiB")) failures.push("V2-Lite 百万 Token 缓存外推异常");
|
||||||
|
if (artifactAbsorb.panel !== "absorb" || artifactAbsorb.algebra !== 2 || artifactAbsorb.naive !== "260.00 KiB" || artifactAbsorb.absorbed !== "29.25 KiB") failures.push("真实 absorb 缓存执行账异常");
|
||||||
|
if (artifactAbsorb.metrics[0] !== "8.8889×" || artifactAbsorb.metrics[1] !== "0.00390625" || artifactAbsorb.metrics[2] !== "1.19e-7" || artifactAbsorb.metrics[3] !== "BYTE-EXACT") failures.push("absorb 数值正确性或复跑闸门异常");
|
||||||
|
if (artifactAbsorb.precisionRows !== 3 || artifactAbsorb.matrixRows !== 5 || artifactAbsorb.localUnsupported !== 4 || artifactAbsorb.executionCards !== 3 || !artifactAbsorb.boundary.includes("不是 FlashMLA 性能")) failures.push("FlashMLA SM120 边界结构异常");
|
||||||
if (artifactEvidence.panel !== "evidence" || artifactEvidence.layers !== 27 || artifactEvidence.executed !== 7 || artifactEvidence.split !== 1 || artifactEvidence.unloaded !== 19 || artifactEvidence.exact !== "31 / 31") failures.push("真实工件执行边界或复跑闸门异常");
|
if (artifactEvidence.panel !== "evidence" || artifactEvidence.layers !== 27 || artifactEvidence.executed !== 7 || artifactEvidence.split !== 1 || artifactEvidence.unloaded !== 19 || artifactEvidence.exact !== "31 / 31") failures.push("真实工件执行边界或复跑闸门异常");
|
||||||
if (!artifactEvidence.dependency.includes("Transformers 5.5") || !artifactEvidence.dependency.includes("4.41.2") || !artifactEvidence.boundary.includes("完整 27 层生成")) failures.push("依赖版本或未覆盖边界异常");
|
if (!artifactEvidence.dependency.includes("Transformers 5.5") || !artifactEvidence.dependency.includes("4.41.2") || !artifactEvidence.boundary.includes("完整 27 层生成")) failures.push("依赖版本或未覆盖边界异常");
|
||||||
if (artifactEvidence.keyboardSelected !== "load" || artifactEvidence.keyboardVisible !== "load") failures.push("真实工件实验键盘 tab 导航异常");
|
if (artifactEvidence.keyboardSelected !== "load" || artifactEvidence.keyboardVisible !== "load") failures.push("真实工件实验键盘 tab 导航异常");
|
||||||
if (home.releaseCards !== 17 || !home.firstRelease.includes("47 页不再压成摘要") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 DeepSeek 首发入口或论文数异常");
|
if (home.releaseCards !== 17 || !home.firstRelease.includes("47 页不再压成摘要") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 DeepSeek 首发入口或论文数异常");
|
||||||
if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常");
|
if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常");
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 4 || mobile.artifactHeatCells !== 64) failures.push("移动端导航或实验异常");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 5 || mobile.artifactHeatCells !== 64) failures.push("移动端导航或实验异常");
|
||||||
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||||
|
|
||||||
|
|||||||
@@ -1,9 +1,18 @@
|
|||||||
---
|
---
|
||||||
import rawTrace from "@/data/deepseek-v2-lite-trace.json";
|
import rawTrace from "@/data/deepseek-v2-lite-trace.json";
|
||||||
import rawRepro from "@/data/deepseek-v2-lite-trace-repro.json";
|
import rawRepro from "@/data/deepseek-v2-lite-trace-repro.json";
|
||||||
|
import rawAbsorb from "@/data/deepseek-v2-lite-absorb.json";
|
||||||
|
import rawAbsorbRepro from "@/data/deepseek-v2-lite-absorb-repro.json";
|
||||||
|
|
||||||
const trace = rawTrace as any;
|
const trace = rawTrace as any;
|
||||||
const repro = rawRepro as any;
|
const repro = rawRepro as any;
|
||||||
|
const absorb = rawAbsorb as any;
|
||||||
|
const absorbRepro = rawAbsorbRepro as any;
|
||||||
|
const absorbExact = JSON.stringify(absorb) === JSON.stringify(absorbRepro);
|
||||||
|
const bytes = (value: number) => value >= 1024
|
||||||
|
? `${(value / 1024).toFixed(2)} KiB`
|
||||||
|
: `${value.toLocaleString()} B`;
|
||||||
|
const scientific = (value: number) => value.toExponential(2);
|
||||||
const compact = {
|
const compact = {
|
||||||
prompts: trace.prompts.map((prompt: any) => ({
|
prompts: trace.prompts.map((prompt: any) => ({
|
||||||
id: prompt.id,
|
id: prompt.id,
|
||||||
@@ -46,7 +55,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
</div>
|
</div>
|
||||||
<p>
|
<p>
|
||||||
固定官方 revision、tokenizer、模型代码和 BF16 第一分片；RTX 5090 连续执行 layer 0–6，
|
固定官方 revision、tokenizer、模型代码和 BF16 第一分片；RTX 5090 连续执行 layer 0–6，
|
||||||
捕获真实 MLA 状态与 3,240 次 routed-expert 选择。所有结论都带证据身份与停止线。
|
捕获 3,240 次真实路由，并让 layer-1 权重继续走入官方吸收式 cache。所有结论都带证据身份与停止线。
|
||||||
</p>
|
</p>
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
@@ -67,8 +76,11 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
<button type="button" role="tab" data-artifact-tab="cache" aria-selected="false" tabindex="-1">
|
<button type="button" role="tab" data-artifact-tab="cache" aria-selected="false" tabindex="-1">
|
||||||
<span>03</span><b>缓存实现账</b><small>latent vs HF eager</small>
|
<span>03</span><b>缓存实现账</b><small>latent vs HF eager</small>
|
||||||
</button>
|
</button>
|
||||||
|
<button type="button" role="tab" data-artifact-tab="absorb" aria-selected="false" tabindex="-1">
|
||||||
|
<span>04</span><b>吸收式执行</b><small>real cache · SM120</small>
|
||||||
|
</button>
|
||||||
<button type="button" role="tab" data-artifact-tab="evidence" aria-selected="false" tabindex="-1">
|
<button type="button" role="tab" data-artifact-tab="evidence" aria-selected="false" tabindex="-1">
|
||||||
<span>04</span><b>证据断面</b><small>revision · shards · rerun</small>
|
<span>05</span><b>证据断面</b><small>revision · shards · rerun</small>
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -243,7 +255,114 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
|
|
||||||
<div class="artifact-boundary">
|
<div class="artifact-boundary">
|
||||||
<b>IMPLEMENTATION CONTRACT</b>
|
<b>IMPLEMENTATION CONTRACT</b>
|
||||||
<p>算法上“可以缓存 576 元素”与本次框架“实际缓存 5,120 元素”同时为真；生产吞吐还需要真实 latent-cache kernel 与 serving runtime。</p>
|
<p>算法上“可以缓存 576 元素”与 HF eager“实际缓存 5,120 元素”同时为真；下一页继续执行官方 absorb 参考路径，生产吞吐仍需要受支持的优化 kernel 与 serving runtime。</p>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="artifact-panel" data-artifact-panel="absorb" hidden>
|
||||||
|
<div class="panel-lead">
|
||||||
|
<div><span>X + O / WEIGHT ABSORPTION</span><h4>这一次 576 元素真的进入了缓存</h4></div>
|
||||||
|
<p>
|
||||||
|
同一组真实 V2-Lite layer-1 权重、同一条 26-token hidden-state 轨迹：
|
||||||
|
先用 V3 官方 naive 路径展开 K/V，再用官方 absorb 路径做 25-token prefill + 1-token decode。
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="absorb-algebra" aria-label="MLA 权重吸收的两条代数变换">
|
||||||
|
<article>
|
||||||
|
<span>SCORE / 把 Wᴷ 搬到 query</span>
|
||||||
|
<code>qᵀ(Wᴷc) = (Wᴷᵀq)ᵀc</code>
|
||||||
|
<p>不再为每个历史 token、每个 head 存展开后的 no-RoPE key；query 先变成 512 维，再与 latent <i>c</i> 相乘。</p>
|
||||||
|
</article>
|
||||||
|
<article>
|
||||||
|
<span>VALUE / 把 Wⱽ 搬到 attention 后</span>
|
||||||
|
<code>Σ pₜ(Wⱽcₜ) = Wⱽ(Σ pₜcₜ)</code>
|
||||||
|
<p>先对 512 维 latent 做加权和，再展开成各 head 的 value。含位置的 64 维 RoPE key 不能这样吸收，必须单独缓存。</p>
|
||||||
|
</article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="absorb-cache-flow">
|
||||||
|
<article class="naive">
|
||||||
|
<header><span>X / V3 NAIVE · BF16</span><b>{bytes(absorb.cache_accounting.naive_active_bytes)}</b></header>
|
||||||
|
<div>
|
||||||
|
<code>K [1, 26, 16, 192]</code>
|
||||||
|
<code>V [1, 26, 16, 128]</code>
|
||||||
|
</div>
|
||||||
|
<p>26 × 5,120 元素；与 HF eager 是同一种展开状态。</p>
|
||||||
|
</article>
|
||||||
|
<i aria-hidden="true">→</i>
|
||||||
|
<article class="absorbed">
|
||||||
|
<header><span>X / V3 ABSORB · BF16</span><b>{bytes(absorb.cache_accounting.absorb_active_bytes)}</b></header>
|
||||||
|
<div>
|
||||||
|
<code>latent [1, 26, 512]</code>
|
||||||
|
<code>RoPE [1, 26, 64]</code>
|
||||||
|
</div>
|
||||||
|
<p>26 × 576 元素；两个官方 buffer 的真实 active slice。</p>
|
||||||
|
</article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="absorb-metrics">
|
||||||
|
<article><span>REAL CACHE RATIO</span><b>{absorb.cache_accounting.naive_over_absorb_ratio.toFixed(4)}×</b><p>展开 K/V ÷ latent + RoPE</p></article>
|
||||||
|
<article><span>BF16 MAX |Δ|</span><b>{absorb.correctness.v3_naive_vs_absorb_bfloat16.max_abs}</b><p>官方 V3 naive ↔ absorb decode</p></article>
|
||||||
|
<article><span>FP32 MAX |Δ|</span><b>{scientific(absorb.correctness.v3_naive_vs_absorb_float32.max_abs)}</b><p>同一 BF16 权重转 FP32 做代数审计</p></article>
|
||||||
|
<article class="exact"><span>INDEPENDENT RERUN</span><b>{absorbExact ? "BYTE-EXACT" : "MISMATCH"}</b><p>完整 JSON SHA-256 6b4c714a…e63d</p></article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="precision-lens">
|
||||||
|
<div>
|
||||||
|
<span>HF EAGER ↔ V3 NAIVE / BF16</span>
|
||||||
|
<i><em style={`--error:${absorb.correctness.hf_eager_vs_v3_naive_bfloat16.max_abs / .004}`}></em></i>
|
||||||
|
<b>max {absorb.correctness.hf_eager_vs_v3_naive_bfloat16.max_abs}</b>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<span>V3 NAIVE ↔ ABSORB / BF16</span>
|
||||||
|
<i><em style={`--error:${absorb.correctness.v3_naive_vs_absorb_bfloat16.max_abs / .004}`}></em></i>
|
||||||
|
<b>max {absorb.correctness.v3_naive_vs_absorb_bfloat16.max_abs}</b>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<span>V3 NAIVE ↔ ABSORB / FP32</span>
|
||||||
|
<i><em style={`--error:${absorb.correctness.v3_naive_vs_absorb_float32.max_abs / .004}`}></em></i>
|
||||||
|
<b>max {scientific(absorb.correctness.v3_naive_vs_absorb_float32.max_abs)}</b>
|
||||||
|
</div>
|
||||||
|
<p>
|
||||||
|
三组输出均 finite。FP32 不是新的 checkpoint 精度，而是把同一组 BF16 权重提升后，
|
||||||
|
隔离矩阵乘法顺序造成的舍入误差。
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="kernel-contract">
|
||||||
|
<div>
|
||||||
|
<span>OFFICIAL FLASHMLA / PINNED {absorb.flashmla_boundary.official_revision.slice(0, 8)}</span>
|
||||||
|
<h5>算法路径已执行，不等于优化 kernel 已执行</h5>
|
||||||
|
<p>官方支持矩阵与编译目标只覆盖 SM90 / SM100；本机 RTX 5090 是 SM120。</p>
|
||||||
|
</div>
|
||||||
|
<div class="kernel-matrix" role="table" aria-label="FlashMLA 官方架构支持矩阵">
|
||||||
|
<div class="head" role="row"><b>KERNEL</b><b>SM90</b><b>SM100</b><b>LOCAL SM120</b></div>
|
||||||
|
<div role="row"><span>dense decode</span><i class="yes">支持</i><i>—</i><i class="no">未支持</i></div>
|
||||||
|
<div role="row"><span>sparse decode</span><i class="yes">支持</i><i class="yes">支持</i><i class="no">未支持</i></div>
|
||||||
|
<div role="row"><span>dense prefill</span><i>—</i><i class="yes">支持</i><i class="no">未支持</i></div>
|
||||||
|
<div role="row"><span>sparse prefill</span><i class="yes">支持</i><i class="yes">支持</i><i class="no">未支持</i></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="execution-split">
|
||||||
|
<article><span>EXECUTED</span><b>官方 V3 pure-PyTorch absorb</b><p>真实权重、真实 latent / RoPE buffers、真实 incremental decode。</p></article>
|
||||||
|
<article><span>BUILD ATTEMPTS</span><b>2 个环境，均未产出 wheel</b><p>host 编译器边界；隔离 CUDA 13 环境缺少 <code>cuda/std/utility</code>。</p></article>
|
||||||
|
<article class="blocked"><span>NOT EXECUTED</span><b>FlashMLA optimized kernel</b><p>源码只生成 <code>sm_90a</code> / <code>sm_100f</code>，dense decode 运行时还拒绝非 SM90a。</p></article>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="evidence-links">
|
||||||
|
<a href="https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py" rel="noreferrer">官方 V3 absorb 参考实现 ↗</a>
|
||||||
|
<a href="https://github.com/deepseek-ai/FlashMLA" rel="noreferrer">官方 FlashMLA 支持矩阵 ↗</a>
|
||||||
|
<a href="https://arxiv.org/abs/2405.04434" rel="noreferrer">DeepSeek-V2 MLA 报告 ↗</a>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="artifact-boundary">
|
||||||
|
<b>CORRECT CLAIM</b>
|
||||||
|
<p>
|
||||||
|
本实验复现的是“真实权重 + 官方参考实现中的压缩缓存与 decode 等价性”，不是 FlashMLA 性能、
|
||||||
|
生产 serving 吞吐或 V2 论文完整模型相对 MHA 的 93.3% 缓存降幅。
|
||||||
|
</p>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
@@ -328,15 +447,16 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
|
|
||||||
<div class="artifact-boundary">
|
<div class="artifact-boundary">
|
||||||
<b>U / STILL OPEN</b>
|
<b>U / STILL OPEN</b>
|
||||||
<p>完整 27 层生成、真实 latent-cache kernel、生产服务、训练负载、FP8/pipeline 与 R1-like 训练 trace 仍未覆盖。</p>
|
<p>完整 27 层生成、受支持硬件上的 FlashMLA 优化 kernel、生产服务、训练负载、FP8/pipeline 与 R1-like 训练 trace 仍未覆盖。</p>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<figcaption>
|
<figcaption>
|
||||||
<span>可复现入口</span>
|
<span>可复现入口</span>
|
||||||
<code>experiments/deepseek/v2_lite_trace.py</code> ·
|
<code>experiments/deepseek/v2_lite_trace.py</code> ·
|
||||||
|
<code>experiments/deepseek/v2_lite_absorb_probe.py</code> ·
|
||||||
<code>src/data/deepseek-v2-lite-trace.json</code> ·
|
<code>src/data/deepseek-v2-lite-trace.json</code> ·
|
||||||
<code>research/DEEPSEEK_V2_LITE_TRACE.md</code>
|
<code>research/DEEPSEEK_MLA_ABSORB_AUDIT.md</code>
|
||||||
</figcaption>
|
</figcaption>
|
||||||
|
|
||||||
<script is:inline type="application/json" data-dsv2-trace set:html={compactJson}></script>
|
<script is:inline type="application/json" data-dsv2-trace set:html={compactJson}></script>
|
||||||
@@ -575,6 +695,11 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.jaccard-block span,
|
.jaccard-block span,
|
||||||
.observed-cache span,
|
.observed-cache span,
|
||||||
.artifact-identity span,
|
.artifact-identity span,
|
||||||
|
.absorb-algebra span,
|
||||||
|
.absorb-cache-flow span,
|
||||||
|
.absorb-metrics span,
|
||||||
|
.kernel-contract span,
|
||||||
|
.execution-split span,
|
||||||
.repro-gate span,
|
.repro-gate span,
|
||||||
.dependency-split span,
|
.dependency-split span,
|
||||||
.checksum-grid span {
|
.checksum-grid span {
|
||||||
@@ -631,7 +756,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.artifact-status b { color: var(--ink); font-size: .72rem; }
|
.artifact-status b { color: var(--ink); font-size: .72rem; }
|
||||||
.artifact-tabs {
|
.artifact-tabs {
|
||||||
display: grid;
|
display: grid;
|
||||||
grid-template-columns: repeat(4, 1fr);
|
grid-template-columns: repeat(5, 1fr);
|
||||||
background: var(--ink);
|
background: var(--ink);
|
||||||
}
|
}
|
||||||
.artifact-tabs button {
|
.artifact-tabs button {
|
||||||
@@ -792,6 +917,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.route-metrics,
|
.route-metrics,
|
||||||
.cache-ratio,
|
.cache-ratio,
|
||||||
.load-lessons,
|
.load-lessons,
|
||||||
|
.absorb-metrics,
|
||||||
.artifact-identity,
|
.artifact-identity,
|
||||||
.checksum-grid {
|
.checksum-grid {
|
||||||
display: grid;
|
display: grid;
|
||||||
@@ -802,6 +928,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.route-metrics article,
|
.route-metrics article,
|
||||||
.cache-ratio article,
|
.cache-ratio article,
|
||||||
.load-lessons article,
|
.load-lessons article,
|
||||||
|
.absorb-metrics article,
|
||||||
.artifact-identity article,
|
.artifact-identity article,
|
||||||
.checksum-grid article {
|
.checksum-grid article {
|
||||||
padding: .8rem;
|
padding: .8rem;
|
||||||
@@ -810,20 +937,24 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.route-metrics article:last-child,
|
.route-metrics article:last-child,
|
||||||
.cache-ratio article:last-child,
|
.cache-ratio article:last-child,
|
||||||
.load-lessons article:last-child,
|
.load-lessons article:last-child,
|
||||||
|
.absorb-metrics article:last-child,
|
||||||
.artifact-identity article:last-child,
|
.artifact-identity article:last-child,
|
||||||
.checksum-grid article:last-child { border-right: 0; }
|
.checksum-grid article:last-child { border-right: 0; }
|
||||||
.route-metrics span,
|
.route-metrics span,
|
||||||
.cache-ratio span,
|
.cache-ratio span,
|
||||||
.load-lessons span {
|
.load-lessons span,
|
||||||
|
.absorb-metrics span {
|
||||||
display: block;
|
display: block;
|
||||||
color: var(--blue);
|
color: var(--blue);
|
||||||
font: 700 .61rem/1 var(--font-mono);
|
font: 700 .61rem/1 var(--font-mono);
|
||||||
}
|
}
|
||||||
.route-metrics b,
|
.route-metrics b,
|
||||||
.cache-ratio b { display: block; margin-top: .35rem; font: 750 1.25rem/1 var(--font-display); }
|
.cache-ratio b,
|
||||||
|
.absorb-metrics b { display: block; margin-top: .35rem; font: 750 1.25rem/1 var(--font-display); }
|
||||||
.route-metrics p,
|
.route-metrics p,
|
||||||
.cache-ratio p,
|
.cache-ratio p,
|
||||||
.load-lessons p { margin: .3rem 0 0; color: rgba(32,32,39,.56); font-size: .65rem; line-height: 1.45; }
|
.load-lessons p,
|
||||||
|
.absorb-metrics p { margin: .3rem 0 0; color: rgba(32,32,39,.56); font-size: .65rem; line-height: 1.45; }
|
||||||
.artifact-boundary {
|
.artifact-boundary {
|
||||||
display: grid;
|
display: grid;
|
||||||
grid-template-columns: 11rem 1fr;
|
grid-template-columns: 11rem 1fr;
|
||||||
@@ -950,6 +1081,127 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.cache-ledger p { margin: .35rem 0 0; color: rgba(32,32,39,.58); font-size: .68rem; line-height: 1.45; }
|
.cache-ledger p { margin: .35rem 0 0; color: rgba(32,32,39,.58); font-size: .68rem; line-height: 1.45; }
|
||||||
.cache-ratio { grid-template-columns: repeat(3, 1fr); }
|
.cache-ratio { grid-template-columns: repeat(3, 1fr); }
|
||||||
.cache-ratio .warning { background: rgba(186,118,44,.09); }
|
.cache-ratio .warning { background: rgba(186,118,44,.09); }
|
||||||
|
.absorb-algebra {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1fr 1fr;
|
||||||
|
gap: .8rem;
|
||||||
|
}
|
||||||
|
.absorb-algebra article {
|
||||||
|
padding: 1rem;
|
||||||
|
border: 1px solid rgba(32,32,39,.15);
|
||||||
|
background: #fffdf8;
|
||||||
|
}
|
||||||
|
.absorb-algebra code {
|
||||||
|
display: block;
|
||||||
|
margin: .75rem 0;
|
||||||
|
color: var(--ink);
|
||||||
|
font: 750 clamp(.88rem, 1.7vw, 1.2rem)/1.2 var(--font-mono);
|
||||||
|
}
|
||||||
|
.absorb-algebra p,
|
||||||
|
.absorb-cache-flow p,
|
||||||
|
.kernel-contract p,
|
||||||
|
.execution-split p {
|
||||||
|
margin: 0;
|
||||||
|
color: rgba(32,32,39,.62);
|
||||||
|
font-size: .69rem;
|
||||||
|
line-height: 1.55;
|
||||||
|
}
|
||||||
|
.absorb-cache-flow {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1fr auto 1fr;
|
||||||
|
gap: .8rem;
|
||||||
|
align-items: center;
|
||||||
|
margin-top: .8rem;
|
||||||
|
}
|
||||||
|
.absorb-cache-flow > i { color: var(--teal); font: 800 1.5rem/1 var(--font-display); }
|
||||||
|
.absorb-cache-flow article {
|
||||||
|
padding: 1rem;
|
||||||
|
border: 1px solid rgba(32,32,39,.15);
|
||||||
|
}
|
||||||
|
.absorb-cache-flow article.naive { background: rgba(161,77,77,.07); }
|
||||||
|
.absorb-cache-flow article.absorbed { background: rgba(57,120,110,.09); }
|
||||||
|
.absorb-cache-flow header { display: flex; justify-content: space-between; gap: 1rem; align-items: center; }
|
||||||
|
.absorb-cache-flow header b { font: 760 1.25rem/1 var(--font-display); }
|
||||||
|
.absorb-cache-flow div { display: flex; flex-wrap: wrap; gap: .4rem; margin: .75rem 0; }
|
||||||
|
.absorb-cache-flow code {
|
||||||
|
padding: .35rem .45rem;
|
||||||
|
background: rgba(32,32,39,.08);
|
||||||
|
font-size: .66rem;
|
||||||
|
}
|
||||||
|
.absorb-metrics { grid-template-columns: repeat(4, 1fr); }
|
||||||
|
.absorb-metrics article.exact { background: rgba(57,120,110,.1); }
|
||||||
|
.precision-lens {
|
||||||
|
display: grid;
|
||||||
|
gap: .55rem;
|
||||||
|
margin-top: .8rem;
|
||||||
|
padding: 1rem;
|
||||||
|
background: var(--ink);
|
||||||
|
color: white;
|
||||||
|
}
|
||||||
|
.precision-lens > div {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: minmax(13rem, 1.3fr) 1fr minmax(8rem, auto);
|
||||||
|
gap: .8rem;
|
||||||
|
align-items: center;
|
||||||
|
}
|
||||||
|
.precision-lens span,
|
||||||
|
.precision-lens b { font: 650 .62rem/1.3 var(--font-mono); }
|
||||||
|
.precision-lens b { text-align: right; }
|
||||||
|
.precision-lens i { height: 7px; overflow: hidden; background: rgba(255,255,255,.13); }
|
||||||
|
.precision-lens em {
|
||||||
|
display: block;
|
||||||
|
width: max(calc(var(--error) * 100%), 2px);
|
||||||
|
max-width: 100%;
|
||||||
|
height: 100%;
|
||||||
|
background: #b7bce5;
|
||||||
|
}
|
||||||
|
.precision-lens > div:last-of-type em { background: #91cfc1; }
|
||||||
|
.precision-lens > p {
|
||||||
|
margin: .25rem 0 0;
|
||||||
|
padding-top: .7rem;
|
||||||
|
border-top: 1px solid rgba(255,255,255,.15);
|
||||||
|
color: rgba(255,255,255,.58);
|
||||||
|
font-size: .66rem;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
.kernel-contract {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: .7fr 1.3fr;
|
||||||
|
gap: 1rem;
|
||||||
|
margin-top: .8rem;
|
||||||
|
padding: 1rem;
|
||||||
|
border: 1px solid rgba(32,32,39,.15);
|
||||||
|
background: #e6dfd3;
|
||||||
|
}
|
||||||
|
.kernel-contract h5 { margin: .45rem 0; font: 720 1rem/1.15 var(--font-display); }
|
||||||
|
.kernel-matrix { overflow-x: auto; border: 1px solid rgba(32,32,39,.14); background: #fffdf8; }
|
||||||
|
.kernel-matrix > div {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1.4fr repeat(3, minmax(5rem, .7fr));
|
||||||
|
min-width: 520px;
|
||||||
|
}
|
||||||
|
.kernel-matrix b,
|
||||||
|
.kernel-matrix span,
|
||||||
|
.kernel-matrix i {
|
||||||
|
padding: .55rem;
|
||||||
|
border-right: 1px solid rgba(32,32,39,.1);
|
||||||
|
border-bottom: 1px solid rgba(32,32,39,.1);
|
||||||
|
font: 650 .61rem/1.2 var(--font-mono);
|
||||||
|
}
|
||||||
|
.kernel-matrix .head { background: var(--ink); color: white; }
|
||||||
|
.kernel-matrix i { color: rgba(32,32,39,.45); font-style: normal; text-align: center; }
|
||||||
|
.kernel-matrix i.yes { color: var(--teal); background: rgba(57,120,110,.08); }
|
||||||
|
.kernel-matrix i.no { color: var(--red); background: rgba(161,77,77,.08); }
|
||||||
|
.execution-split {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(3, 1fr);
|
||||||
|
margin-top: .8rem;
|
||||||
|
border: 1px solid rgba(32,32,39,.14);
|
||||||
|
}
|
||||||
|
.execution-split article { padding: .9rem; border-right: 1px solid rgba(32,32,39,.12); }
|
||||||
|
.execution-split article:last-child { border-right: 0; }
|
||||||
|
.execution-split article.blocked { background: rgba(161,77,77,.08); }
|
||||||
|
.execution-split b { display: block; margin: .4rem 0; font-size: .77rem; }
|
||||||
.artifact-identity { grid-template-columns: repeat(3, 1fr); }
|
.artifact-identity { grid-template-columns: repeat(3, 1fr); }
|
||||||
.artifact-identity article { display: grid; gap: .35rem; }
|
.artifact-identity article { display: grid; gap: .35rem; }
|
||||||
.artifact-identity b { font-size: .78rem; }
|
.artifact-identity b { font-size: .78rem; }
|
||||||
@@ -1030,7 +1282,8 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.panel-lead,
|
.panel-lead,
|
||||||
.heat-head,
|
.heat-head,
|
||||||
.aggregate-card,
|
.aggregate-card,
|
||||||
.jaccard-block { grid-template-columns: 1fr; }
|
.jaccard-block,
|
||||||
|
.kernel-contract { grid-template-columns: 1fr; }
|
||||||
.artifact-status { grid-template-columns: 1fr 1fr; }
|
.artifact-status { grid-template-columns: 1fr 1fr; }
|
||||||
.artifact-tabs { grid-template-columns: 1fr 1fr; }
|
.artifact-tabs { grid-template-columns: 1fr 1fr; }
|
||||||
.route-controls { grid-template-columns: 1fr 1fr; }
|
.route-controls { grid-template-columns: 1fr 1fr; }
|
||||||
@@ -1039,7 +1292,8 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.route-stage > i { transform: rotate(90deg); justify-self: center; }
|
.route-stage > i { transform: rotate(90deg); justify-self: center; }
|
||||||
.cache-controls { grid-template-columns: 1fr 1fr; }
|
.cache-controls { grid-template-columns: 1fr 1fr; }
|
||||||
.route-metrics,
|
.route-metrics,
|
||||||
.checksum-grid { grid-template-columns: 1fr 1fr; }
|
.checksum-grid,
|
||||||
|
.absorb-metrics { grid-template-columns: 1fr 1fr; }
|
||||||
.layer-evidence { grid-template-columns: repeat(9, 1fr); }
|
.layer-evidence { grid-template-columns: repeat(9, 1fr); }
|
||||||
.repro-gate { grid-template-columns: 1fr 1fr; }
|
.repro-gate { grid-template-columns: 1fr 1fr; }
|
||||||
.repro-gate > p { grid-column: 1 / -1; padding: .8rem 0 0; border-left: 0; border-top: 1px solid rgba(255,255,255,.18); }
|
.repro-gate > p { grid-column: 1 / -1; padding: .8rem 0 0; border-left: 0; border-top: 1px solid rgba(255,255,255,.18); }
|
||||||
@@ -1062,20 +1316,28 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
|||||||
.route-metrics,
|
.route-metrics,
|
||||||
.cache-ratio,
|
.cache-ratio,
|
||||||
.load-lessons,
|
.load-lessons,
|
||||||
|
.absorb-metrics,
|
||||||
.artifact-identity,
|
.artifact-identity,
|
||||||
.checksum-grid,
|
.checksum-grid,
|
||||||
.cache-ledger { grid-template-columns: 1fr; }
|
.cache-ledger { grid-template-columns: 1fr; }
|
||||||
.route-metrics article,
|
.route-metrics article,
|
||||||
.cache-ratio article,
|
.cache-ratio article,
|
||||||
.load-lessons article,
|
.load-lessons article,
|
||||||
|
.absorb-metrics article,
|
||||||
.artifact-identity article,
|
.artifact-identity article,
|
||||||
.checksum-grid article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
|
.checksum-grid article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
|
||||||
.artifact-boundary { grid-template-columns: 1fr; }
|
.artifact-boundary { grid-template-columns: 1fr; }
|
||||||
.load-dials { grid-template-columns: 1fr; }
|
.load-dials { grid-template-columns: 1fr; }
|
||||||
.observed-cache,
|
.observed-cache,
|
||||||
.dependency-split { grid-template-columns: 1fr; }
|
.dependency-split,
|
||||||
|
.absorb-algebra,
|
||||||
|
.absorb-cache-flow,
|
||||||
|
.execution-split { grid-template-columns: 1fr; }
|
||||||
.observed-cache > i,
|
.observed-cache > i,
|
||||||
.dependency-split > i { transform: rotate(90deg); justify-self: center; }
|
.dependency-split > i { transform: rotate(90deg); justify-self: center; }
|
||||||
|
.absorb-cache-flow > i { transform: rotate(90deg); justify-self: center; }
|
||||||
|
.precision-lens > div { grid-template-columns: 1fr; }
|
||||||
|
.precision-lens b { text-align: left; }
|
||||||
.layer-evidence { grid-template-columns: repeat(7, 1fr); }
|
.layer-evidence { grid-template-columns: repeat(7, 1fr); }
|
||||||
.repro-gate { grid-template-columns: 1fr; }
|
.repro-gate { grid-template-columns: 1fr; }
|
||||||
.repro-gate > p { grid-column: auto; }
|
.repro-gate > p { grid-column: auto; }
|
||||||
|
|||||||
@@ -0,0 +1,439 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"captured_at": "2026-07-29T06:35:00Z",
|
||||||
|
"evidence_identity": "X / official V2-Lite weights executed through official HF eager and V3 naive/absorb reference paths",
|
||||||
|
"boundary": {
|
||||||
|
"correctness_probe": true,
|
||||||
|
"serving_benchmark": false,
|
||||||
|
"full_model_generation": false,
|
||||||
|
"optimized_flashmla_kernel_executed": false,
|
||||||
|
"scope": "one real layer-1 attention input; 25-token prefill plus one-token decode"
|
||||||
|
},
|
||||||
|
"provenance": {
|
||||||
|
"huggingface_model": "deepseek-ai/DeepSeek-V2-Lite",
|
||||||
|
"huggingface_revision": "604d5664dddd88a0433dbae533b7fe9472482de0",
|
||||||
|
"deepseek_v3_repository": "deepseek-ai/DeepSeek-V3",
|
||||||
|
"deepseek_v3_revision": "9b4e9788e4a3a731f7567338ed15d3ec549ce03b",
|
||||||
|
"flashmla_repository": "deepseek-ai/FlashMLA",
|
||||||
|
"flashmla_revision": "15f13e5030374295491c5ce31b02d7e63a7772c6",
|
||||||
|
"sha256": {
|
||||||
|
"v2_config": "f346286b0f1c8b044252fd54cb4fa78b9fab6472a6e8bebb9edfe03d414ea03d",
|
||||||
|
"v2_modeling": "7d8e5221095286eea991137760893fd7ba52727c0b4ebf48ec09e8bc56b45b9c",
|
||||||
|
"v2_shard_1": "0d7e9f39bde40111a4c0f390b87497dce4565cf578d916395e6b2c7851f1e8da",
|
||||||
|
"v3_model": "fdb7995bc234badbf0011b5af8cf391be94343abc51305428cb9a04452116f02"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"environment": {
|
||||||
|
"python": "3.10.14",
|
||||||
|
"platform": "Linux-7.0.0-28-generic-x86_64-with-glibc2.43",
|
||||||
|
"libc": [
|
||||||
|
"glibc",
|
||||||
|
"2.43"
|
||||||
|
],
|
||||||
|
"torch": "2.11.0+cu128",
|
||||||
|
"torch_cuda": "12.8",
|
||||||
|
"transformers": "4.41.2",
|
||||||
|
"safetensors": "0.8.0",
|
||||||
|
"device": "cuda",
|
||||||
|
"cuda_capability": [
|
||||||
|
12,
|
||||||
|
0
|
||||||
|
],
|
||||||
|
"nvidia_smi": {
|
||||||
|
"name": "NVIDIA GeForce RTX 5090",
|
||||||
|
"driver_version": "595.84",
|
||||||
|
"memory.total": "32607",
|
||||||
|
"compute_cap": "12.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"input": {
|
||||||
|
"prompt": "用通俗的语言解释，为什么稀疏专家模型可以拥有很多参数，但每个 token 只使用其中一小部分。",
|
||||||
|
"token_ids": [
|
||||||
|
100000,
|
||||||
|
1082,
|
||||||
|
83983,
|
||||||
|
91476,
|
||||||
|
17882,
|
||||||
|
19304,
|
||||||
|
10285,
|
||||||
|
21844,
|
||||||
|
20923,
|
||||||
|
10900,
|
||||||
|
39322,
|
||||||
|
1876,
|
||||||
|
9476,
|
||||||
|
3599,
|
||||||
|
21273,
|
||||||
|
19304,
|
||||||
|
1705,
|
||||||
|
13712,
|
||||||
|
10728,
|
||||||
|
207,
|
||||||
|
2141,
|
||||||
|
5118,
|
||||||
|
6802,
|
||||||
|
51905,
|
||||||
|
5781,
|
||||||
|
398
|
||||||
|
],
|
||||||
|
"tokens": [
|
||||||
|
"<｜begin▁of▁sentence｜>",
|
||||||
|
"çĶ¨",
|
||||||
|
"éĢļä¿Ĺ",
|
||||||
|
"çļĦè¯Ńè¨Ģ",
|
||||||
|
"解éĩĬ",
|
||||||
|
"ï¼Į",
|
||||||
|
"为ä»Ģä¹Ī",
|
||||||
|
"ç¨Ģ",
|
||||||
|
"çĸı",
|
||||||
|
"ä¸ĵ家",
|
||||||
|
"模åŀĭ",
|
||||||
|
"åı¯ä»¥",
|
||||||
|
"æĭ¥æľī",
|
||||||
|
"å¾Īå¤ļ",
|
||||||
|
"åıĤæķ°",
|
||||||
|
"ï¼Į",
|
||||||
|
"ä½Ĩ",
|
||||||
|
"æ¯ı个",
|
||||||
|
"Ġtoken",
|
||||||
|
"Ġ",
|
||||||
|
"åıª",
|
||||||
|
"使çĶ¨",
|
||||||
|
"åħ¶ä¸Ń",
|
||||||
|
"ä¸Ģå°ı",
|
||||||
|
"éĥ¨åĪĨ",
|
||||||
|
"ãĢĤ"
|
||||||
|
],
|
||||||
|
"sequence_tokens": 26,
|
||||||
|
"prefill_tokens": 25,
|
||||||
|
"decode_tokens": 1,
|
||||||
|
"layer0_executed": true,
|
||||||
|
"layer1_input_normalized": true
|
||||||
|
},
|
||||||
|
"configuration": {
|
||||||
|
"hidden": 2048,
|
||||||
|
"attention_heads": 16,
|
||||||
|
"qk_nope_head_dim": 128,
|
||||||
|
"qk_rope_head_dim": 64,
|
||||||
|
"v_head_dim": 128,
|
||||||
|
"kv_lora_rank": 512,
|
||||||
|
"cache_slots": 32
|
||||||
|
},
|
||||||
|
"execution": {
|
||||||
|
"huggingface_eager": {
|
||||||
|
"implementation": "huggingface_eager",
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"key_shape": [
|
||||||
|
1,
|
||||||
|
16,
|
||||||
|
26,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"value_shape": [
|
||||||
|
1,
|
||||||
|
16,
|
||||||
|
26,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"active_cache_bytes": 266240,
|
||||||
|
"decode_finite": true
|
||||||
|
},
|
||||||
|
"v3_naive_bfloat16": {
|
||||||
|
"implementation": "naive",
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"active_cache": {
|
||||||
|
"key": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
16,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"bytes": 159744
|
||||||
|
},
|
||||||
|
"value": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
16,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"bytes": 106496
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"active_cache_bytes": 266240,
|
||||||
|
"allocated_cache": {
|
||||||
|
"key_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
16,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"key_bytes": 196608,
|
||||||
|
"value_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
16,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"value_bytes": 131072
|
||||||
|
},
|
||||||
|
"allocated_cache_bytes": 327680,
|
||||||
|
"decode_finite": true
|
||||||
|
},
|
||||||
|
"v3_absorb_bfloat16": {
|
||||||
|
"implementation": "absorb",
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"active_cache": {
|
||||||
|
"latent": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
512
|
||||||
|
],
|
||||||
|
"bytes": 26624
|
||||||
|
},
|
||||||
|
"rope": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"bytes": 3328
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"active_cache_bytes": 29952,
|
||||||
|
"allocated_cache": {
|
||||||
|
"latent_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
512
|
||||||
|
],
|
||||||
|
"latent_bytes": 32768,
|
||||||
|
"rope_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"rope_bytes": 4096
|
||||||
|
},
|
||||||
|
"allocated_cache_bytes": 36864,
|
||||||
|
"decode_finite": true
|
||||||
|
},
|
||||||
|
"v3_naive_float32": {
|
||||||
|
"implementation": "naive",
|
||||||
|
"dtype": "float32",
|
||||||
|
"active_cache": {
|
||||||
|
"key": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
16,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"bytes": 319488
|
||||||
|
},
|
||||||
|
"value": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
16,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"bytes": 212992
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"active_cache_bytes": 532480,
|
||||||
|
"allocated_cache": {
|
||||||
|
"key_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
16,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"key_bytes": 393216,
|
||||||
|
"value_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
16,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"value_bytes": 262144
|
||||||
|
},
|
||||||
|
"allocated_cache_bytes": 655360,
|
||||||
|
"decode_finite": true
|
||||||
|
},
|
||||||
|
"v3_absorb_float32": {
|
||||||
|
"implementation": "absorb",
|
||||||
|
"dtype": "float32",
|
||||||
|
"active_cache": {
|
||||||
|
"latent": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
512
|
||||||
|
],
|
||||||
|
"bytes": 53248
|
||||||
|
},
|
||||||
|
"rope": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"bytes": 6656
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"active_cache_bytes": 59904,
|
||||||
|
"allocated_cache": {
|
||||||
|
"latent_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
512
|
||||||
|
],
|
||||||
|
"latent_bytes": 65536,
|
||||||
|
"rope_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"rope_bytes": 8192
|
||||||
|
},
|
||||||
|
"allocated_cache_bytes": 73728,
|
||||||
|
"decode_finite": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"correctness": {
|
||||||
|
"hf_eager_vs_v3_naive_bfloat16": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
2048
|
||||||
|
],
|
||||||
|
"finite": true,
|
||||||
|
"exact_fraction": 0.28466796875,
|
||||||
|
"max_abs": 0.001220703125,
|
||||||
|
"mean_abs": 8.882676775101572e-05,
|
||||||
|
"p50_abs": 6.103515625e-05,
|
||||||
|
"p90_abs": 0.000244140625,
|
||||||
|
"p99_abs": 0.00038238614797592163
|
||||||
|
},
|
||||||
|
"v3_naive_vs_absorb_bfloat16": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
2048
|
||||||
|
],
|
||||||
|
"finite": true,
|
||||||
|
"exact_fraction": 0.10009765625,
|
||||||
|
"max_abs": 0.00390625,
|
||||||
|
"mean_abs": 0.00027031131321564317,
|
||||||
|
"p50_abs": 0.000244140625,
|
||||||
|
"p90_abs": 0.00054931640625,
|
||||||
|
"p99_abs": 0.0009236149489879608
|
||||||
|
},
|
||||||
|
"hf_eager_vs_v3_absorb_bfloat16": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
2048
|
||||||
|
],
|
||||||
|
"finite": true,
|
||||||
|
"exact_fraction": 0.1044921875,
|
||||||
|
"max_abs": 0.00390625,
|
||||||
|
"mean_abs": 0.00027483145822770894,
|
||||||
|
"p50_abs": 0.000244140625,
|
||||||
|
"p90_abs": 0.00054931640625,
|
||||||
|
"p99_abs": 0.0009765625
|
||||||
|
},
|
||||||
|
"v3_naive_vs_absorb_float32": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
2048
|
||||||
|
],
|
||||||
|
"finite": true,
|
||||||
|
"exact_fraction": 0.0556640625,
|
||||||
|
"max_abs": 1.1920928955078125e-07,
|
||||||
|
"mean_abs": 9.464486083743395e-09,
|
||||||
|
"p50_abs": 7.450580596923828e-09,
|
||||||
|
"p90_abs": 1.955777406692505e-08,
|
||||||
|
"p99_abs": 3.144612037431216e-08
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cache_accounting": {
|
||||||
|
"active_tokens": 26,
|
||||||
|
"dtype": "BF16",
|
||||||
|
"naive_active_bytes": 266240,
|
||||||
|
"absorb_active_bytes": 29952,
|
||||||
|
"naive_elements_per_token": 5120,
|
||||||
|
"absorb_elements_per_token": 576,
|
||||||
|
"naive_over_absorb_ratio": 8.88888888888889,
|
||||||
|
"absorb_reduction_vs_naive": 0.8875
|
||||||
|
},
|
||||||
|
"flashmla_boundary": {
|
||||||
|
"official_revision": "15f13e5030374295491c5ce31b02d7e63a7772c6",
|
||||||
|
"cutlass_revision": "147f5673d0c1c3dcf66f78d677fd647e4a020219",
|
||||||
|
"official_support_matrix": [
|
||||||
|
{
|
||||||
|
"kernel": "dense_decode",
|
||||||
|
"architectures": [
|
||||||
|
"SM90"
|
||||||
|
],
|
||||||
|
"mode": "MQA",
|
||||||
|
"cache": "BF16"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kernel": "sparse_decode",
|
||||||
|
"architectures": [
|
||||||
|
"SM90",
|
||||||
|
"SM100"
|
||||||
|
],
|
||||||
|
"mode": "MQA",
|
||||||
|
"cache": "FP8"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kernel": "dense_prefill",
|
||||||
|
"architectures": [
|
||||||
|
"SM100"
|
||||||
|
],
|
||||||
|
"mode": "MHA",
|
||||||
|
"cache": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kernel": "sparse_prefill",
|
||||||
|
"architectures": [
|
||||||
|
"SM90",
|
||||||
|
"SM100"
|
||||||
|
],
|
||||||
|
"mode": "MQA",
|
||||||
|
"cache": null
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"compiled_gencode": [
|
||||||
|
"sm_100f",
|
||||||
|
"sm_90a"
|
||||||
|
],
|
||||||
|
"dense_decode_runtime_guard": "Dense decode MLA is only supported on SM90a architecture",
|
||||||
|
"local_architecture": "SM120",
|
||||||
|
"optimized_kernel_executed": false,
|
||||||
|
"why_not": "The pinned official source does not list or generate SM120 kernels; dense decode additionally rejects non-SM90a devices.",
|
||||||
|
"build_attempts": [
|
||||||
|
{
|
||||||
|
"environment": "host CUDA 12.8 / g++ 15.2",
|
||||||
|
"result": "failed before compilation",
|
||||||
|
"evidence": "CUDA 12.8 rejects host compiler versions newer than g++ 13; g++-13 is not installed."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"environment": "isolated CUDA 13.0.2 / torch 2.11.0+cu130 / g++ 13.3",
|
||||||
|
"result": "failed during official extension compilation",
|
||||||
|
"evidence": "csrc/api.cpp could not find cuda/std/utility; no wheel or runtime kernel was produced."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source_sha256": {
|
||||||
|
"readme": "46c001c53800f47148918f4fc2969102492dd5164f710fc015db1622f6f9929a",
|
||||||
|
"setup": "b493b648e4f718e50dd323775834ab53703b155aee6f639e7c812b9e8a2ce48d",
|
||||||
|
"dense_decode_header": "f3b012556eca4c8469bc99f6e299785793e6e6fe4aef090789501696e68e860f"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,439 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"captured_at": "2026-07-29T06:35:00Z",
|
||||||
|
"evidence_identity": "X / official V2-Lite weights executed through official HF eager and V3 naive/absorb reference paths",
|
||||||
|
"boundary": {
|
||||||
|
"correctness_probe": true,
|
||||||
|
"serving_benchmark": false,
|
||||||
|
"full_model_generation": false,
|
||||||
|
"optimized_flashmla_kernel_executed": false,
|
||||||
|
"scope": "one real layer-1 attention input; 25-token prefill plus one-token decode"
|
||||||
|
},
|
||||||
|
"provenance": {
|
||||||
|
"huggingface_model": "deepseek-ai/DeepSeek-V2-Lite",
|
||||||
|
"huggingface_revision": "604d5664dddd88a0433dbae533b7fe9472482de0",
|
||||||
|
"deepseek_v3_repository": "deepseek-ai/DeepSeek-V3",
|
||||||
|
"deepseek_v3_revision": "9b4e9788e4a3a731f7567338ed15d3ec549ce03b",
|
||||||
|
"flashmla_repository": "deepseek-ai/FlashMLA",
|
||||||
|
"flashmla_revision": "15f13e5030374295491c5ce31b02d7e63a7772c6",
|
||||||
|
"sha256": {
|
||||||
|
"v2_config": "f346286b0f1c8b044252fd54cb4fa78b9fab6472a6e8bebb9edfe03d414ea03d",
|
||||||
|
"v2_modeling": "7d8e5221095286eea991137760893fd7ba52727c0b4ebf48ec09e8bc56b45b9c",
|
||||||
|
"v2_shard_1": "0d7e9f39bde40111a4c0f390b87497dce4565cf578d916395e6b2c7851f1e8da",
|
||||||
|
"v3_model": "fdb7995bc234badbf0011b5af8cf391be94343abc51305428cb9a04452116f02"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"environment": {
|
||||||
|
"python": "3.10.14",
|
||||||
|
"platform": "Linux-7.0.0-28-generic-x86_64-with-glibc2.43",
|
||||||
|
"libc": [
|
||||||
|
"glibc",
|
||||||
|
"2.43"
|
||||||
|
],
|
||||||
|
"torch": "2.11.0+cu128",
|
||||||
|
"torch_cuda": "12.8",
|
||||||
|
"transformers": "4.41.2",
|
||||||
|
"safetensors": "0.8.0",
|
||||||
|
"device": "cuda",
|
||||||
|
"cuda_capability": [
|
||||||
|
12,
|
||||||
|
0
|
||||||
|
],
|
||||||
|
"nvidia_smi": {
|
||||||
|
"name": "NVIDIA GeForce RTX 5090",
|
||||||
|
"driver_version": "595.84",
|
||||||
|
"memory.total": "32607",
|
||||||
|
"compute_cap": "12.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"input": {
|
||||||
|
"prompt": "用通俗的语言解释，为什么稀疏专家模型可以拥有很多参数，但每个 token 只使用其中一小部分。",
|
||||||
|
"token_ids": [
|
||||||
|
100000,
|
||||||
|
1082,
|
||||||
|
83983,
|
||||||
|
91476,
|
||||||
|
17882,
|
||||||
|
19304,
|
||||||
|
10285,
|
||||||
|
21844,
|
||||||
|
20923,
|
||||||
|
10900,
|
||||||
|
39322,
|
||||||
|
1876,
|
||||||
|
9476,
|
||||||
|
3599,
|
||||||
|
21273,
|
||||||
|
19304,
|
||||||
|
1705,
|
||||||
|
13712,
|
||||||
|
10728,
|
||||||
|
207,
|
||||||
|
2141,
|
||||||
|
5118,
|
||||||
|
6802,
|
||||||
|
51905,
|
||||||
|
5781,
|
||||||
|
398
|
||||||
|
],
|
||||||
|
"tokens": [
|
||||||
|
"<｜begin▁of▁sentence｜>",
|
||||||
|
"çĶ¨",
|
||||||
|
"éĢļä¿Ĺ",
|
||||||
|
"çļĦè¯Ńè¨Ģ",
|
||||||
|
"解éĩĬ",
|
||||||
|
"ï¼Į",
|
||||||
|
"为ä»Ģä¹Ī",
|
||||||
|
"ç¨Ģ",
|
||||||
|
"çĸı",
|
||||||
|
"ä¸ĵ家",
|
||||||
|
"模åŀĭ",
|
||||||
|
"åı¯ä»¥",
|
||||||
|
"æĭ¥æľī",
|
||||||
|
"å¾Īå¤ļ",
|
||||||
|
"åıĤæķ°",
|
||||||
|
"ï¼Į",
|
||||||
|
"ä½Ĩ",
|
||||||
|
"æ¯ı个",
|
||||||
|
"Ġtoken",
|
||||||
|
"Ġ",
|
||||||
|
"åıª",
|
||||||
|
"使çĶ¨",
|
||||||
|
"åħ¶ä¸Ń",
|
||||||
|
"ä¸Ģå°ı",
|
||||||
|
"éĥ¨åĪĨ",
|
||||||
|
"ãĢĤ"
|
||||||
|
],
|
||||||
|
"sequence_tokens": 26,
|
||||||
|
"prefill_tokens": 25,
|
||||||
|
"decode_tokens": 1,
|
||||||
|
"layer0_executed": true,
|
||||||
|
"layer1_input_normalized": true
|
||||||
|
},
|
||||||
|
"configuration": {
|
||||||
|
"hidden": 2048,
|
||||||
|
"attention_heads": 16,
|
||||||
|
"qk_nope_head_dim": 128,
|
||||||
|
"qk_rope_head_dim": 64,
|
||||||
|
"v_head_dim": 128,
|
||||||
|
"kv_lora_rank": 512,
|
||||||
|
"cache_slots": 32
|
||||||
|
},
|
||||||
|
"execution": {
|
||||||
|
"huggingface_eager": {
|
||||||
|
"implementation": "huggingface_eager",
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"key_shape": [
|
||||||
|
1,
|
||||||
|
16,
|
||||||
|
26,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"value_shape": [
|
||||||
|
1,
|
||||||
|
16,
|
||||||
|
26,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"active_cache_bytes": 266240,
|
||||||
|
"decode_finite": true
|
||||||
|
},
|
||||||
|
"v3_naive_bfloat16": {
|
||||||
|
"implementation": "naive",
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"active_cache": {
|
||||||
|
"key": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
16,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"bytes": 159744
|
||||||
|
},
|
||||||
|
"value": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
16,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"bytes": 106496
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"active_cache_bytes": 266240,
|
||||||
|
"allocated_cache": {
|
||||||
|
"key_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
16,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"key_bytes": 196608,
|
||||||
|
"value_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
16,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"value_bytes": 131072
|
||||||
|
},
|
||||||
|
"allocated_cache_bytes": 327680,
|
||||||
|
"decode_finite": true
|
||||||
|
},
|
||||||
|
"v3_absorb_bfloat16": {
|
||||||
|
"implementation": "absorb",
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"active_cache": {
|
||||||
|
"latent": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
512
|
||||||
|
],
|
||||||
|
"bytes": 26624
|
||||||
|
},
|
||||||
|
"rope": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"bytes": 3328
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"active_cache_bytes": 29952,
|
||||||
|
"allocated_cache": {
|
||||||
|
"latent_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
512
|
||||||
|
],
|
||||||
|
"latent_bytes": 32768,
|
||||||
|
"rope_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"rope_bytes": 4096
|
||||||
|
},
|
||||||
|
"allocated_cache_bytes": 36864,
|
||||||
|
"decode_finite": true
|
||||||
|
},
|
||||||
|
"v3_naive_float32": {
|
||||||
|
"implementation": "naive",
|
||||||
|
"dtype": "float32",
|
||||||
|
"active_cache": {
|
||||||
|
"key": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
16,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"bytes": 319488
|
||||||
|
},
|
||||||
|
"value": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
16,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"bytes": 212992
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"active_cache_bytes": 532480,
|
||||||
|
"allocated_cache": {
|
||||||
|
"key_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
16,
|
||||||
|
192
|
||||||
|
],
|
||||||
|
"key_bytes": 393216,
|
||||||
|
"value_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
16,
|
||||||
|
128
|
||||||
|
],
|
||||||
|
"value_bytes": 262144
|
||||||
|
},
|
||||||
|
"allocated_cache_bytes": 655360,
|
||||||
|
"decode_finite": true
|
||||||
|
},
|
||||||
|
"v3_absorb_float32": {
|
||||||
|
"implementation": "absorb",
|
||||||
|
"dtype": "float32",
|
||||||
|
"active_cache": {
|
||||||
|
"latent": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
512
|
||||||
|
],
|
||||||
|
"bytes": 53248
|
||||||
|
},
|
||||||
|
"rope": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
26,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"bytes": 6656
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"active_cache_bytes": 59904,
|
||||||
|
"allocated_cache": {
|
||||||
|
"latent_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
512
|
||||||
|
],
|
||||||
|
"latent_bytes": 65536,
|
||||||
|
"rope_shape": [
|
||||||
|
1,
|
||||||
|
32,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"rope_bytes": 8192
|
||||||
|
},
|
||||||
|
"allocated_cache_bytes": 73728,
|
||||||
|
"decode_finite": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"correctness": {
|
||||||
|
"hf_eager_vs_v3_naive_bfloat16": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
2048
|
||||||
|
],
|
||||||
|
"finite": true,
|
||||||
|
"exact_fraction": 0.28466796875,
|
||||||
|
"max_abs": 0.001220703125,
|
||||||
|
"mean_abs": 8.882676775101572e-05,
|
||||||
|
"p50_abs": 6.103515625e-05,
|
||||||
|
"p90_abs": 0.000244140625,
|
||||||
|
"p99_abs": 0.00038238614797592163
|
||||||
|
},
|
||||||
|
"v3_naive_vs_absorb_bfloat16": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
2048
|
||||||
|
],
|
||||||
|
"finite": true,
|
||||||
|
"exact_fraction": 0.10009765625,
|
||||||
|
"max_abs": 0.00390625,
|
||||||
|
"mean_abs": 0.00027031131321564317,
|
||||||
|
"p50_abs": 0.000244140625,
|
||||||
|
"p90_abs": 0.00054931640625,
|
||||||
|
"p99_abs": 0.0009236149489879608
|
||||||
|
},
|
||||||
|
"hf_eager_vs_v3_absorb_bfloat16": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
2048
|
||||||
|
],
|
||||||
|
"finite": true,
|
||||||
|
"exact_fraction": 0.1044921875,
|
||||||
|
"max_abs": 0.00390625,
|
||||||
|
"mean_abs": 0.00027483145822770894,
|
||||||
|
"p50_abs": 0.000244140625,
|
||||||
|
"p90_abs": 0.00054931640625,
|
||||||
|
"p99_abs": 0.0009765625
|
||||||
|
},
|
||||||
|
"v3_naive_vs_absorb_float32": {
|
||||||
|
"shape": [
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
2048
|
||||||
|
],
|
||||||
|
"finite": true,
|
||||||
|
"exact_fraction": 0.0556640625,
|
||||||
|
"max_abs": 1.1920928955078125e-07,
|
||||||
|
"mean_abs": 9.464486083743395e-09,
|
||||||
|
"p50_abs": 7.450580596923828e-09,
|
||||||
|
"p90_abs": 1.955777406692505e-08,
|
||||||
|
"p99_abs": 3.144612037431216e-08
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cache_accounting": {
|
||||||
|
"active_tokens": 26,
|
||||||
|
"dtype": "BF16",
|
||||||
|
"naive_active_bytes": 266240,
|
||||||
|
"absorb_active_bytes": 29952,
|
||||||
|
"naive_elements_per_token": 5120,
|
||||||
|
"absorb_elements_per_token": 576,
|
||||||
|
"naive_over_absorb_ratio": 8.88888888888889,
|
||||||
|
"absorb_reduction_vs_naive": 0.8875
|
||||||
|
},
|
||||||
|
"flashmla_boundary": {
|
||||||
|
"official_revision": "15f13e5030374295491c5ce31b02d7e63a7772c6",
|
||||||
|
"cutlass_revision": "147f5673d0c1c3dcf66f78d677fd647e4a020219",
|
||||||
|
"official_support_matrix": [
|
||||||
|
{
|
||||||
|
"kernel": "dense_decode",
|
||||||
|
"architectures": [
|
||||||
|
"SM90"
|
||||||
|
],
|
||||||
|
"mode": "MQA",
|
||||||
|
"cache": "BF16"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kernel": "sparse_decode",
|
||||||
|
"architectures": [
|
||||||
|
"SM90",
|
||||||
|
"SM100"
|
||||||
|
],
|
||||||
|
"mode": "MQA",
|
||||||
|
"cache": "FP8"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kernel": "dense_prefill",
|
||||||
|
"architectures": [
|
||||||
|
"SM100"
|
||||||
|
],
|
||||||
|
"mode": "MHA",
|
||||||
|
"cache": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kernel": "sparse_prefill",
|
||||||
|
"architectures": [
|
||||||
|
"SM90",
|
||||||
|
"SM100"
|
||||||
|
],
|
||||||
|
"mode": "MQA",
|
||||||
|
"cache": null
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"compiled_gencode": [
|
||||||
|
"sm_100f",
|
||||||
|
"sm_90a"
|
||||||
|
],
|
||||||
|
"dense_decode_runtime_guard": "Dense decode MLA is only supported on SM90a architecture",
|
||||||
|
"local_architecture": "SM120",
|
||||||
|
"optimized_kernel_executed": false,
|
||||||
|
"why_not": "The pinned official source does not list or generate SM120 kernels; dense decode additionally rejects non-SM90a devices.",
|
||||||
|
"build_attempts": [
|
||||||
|
{
|
||||||
|
"environment": "host CUDA 12.8 / g++ 15.2",
|
||||||
|
"result": "failed before compilation",
|
||||||
|
"evidence": "CUDA 12.8 rejects host compiler versions newer than g++ 13; g++-13 is not installed."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"environment": "isolated CUDA 13.0.2 / torch 2.11.0+cu130 / g++ 13.3",
|
||||||
|
"result": "failed during official extension compilation",
|
||||||
|
"evidence": "csrc/api.cpp could not find cuda/std/utility; no wheel or runtime kernel was produced."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source_sha256": {
|
||||||
|
"readme": "46c001c53800f47148918f4fc2969102492dd5164f710fc015db1622f6f9929a",
|
||||||
|
"setup": "b493b648e4f718e50dd323775834ab53703b155aee6f639e7c812b9e8a2ce48d",
|
||||||
|
"dense_decode_header": "f3b012556eca4c8469bc99f6e299785793e6e6fe4aef090789501696e68e860f"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -37,7 +37,7 @@ const toc = [
|
|||||||
|
|
||||||
<BaseLayout
|
<BaseLayout
|
||||||
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
|
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
|
||||||
description="用二十四张问题账、十次技术转向、八个交互实验、真实 V2-Lite 权重 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
description="用二十四张问题账、十次技术转向、九个交互实验、真实 V2-Lite 权重与吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||||
section="deepseek"
|
section="deepseek"
|
||||||
>
|
>
|
||||||
<header class="page-hero deepseek-hero">
|
<header class="page-hero deepseek-hero">
|
||||||
@@ -55,7 +55,7 @@ const toc = [
|
|||||||
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
||||||
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
|
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
|
||||||
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
|
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
|
||||||
<div><dt>LABS</dt><dd>8 个可操作实验</dd></div>
|
<div><dt>LABS</dt><dd>9 个可操作实验</dd></div>
|
||||||
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
|
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
|
||||||
<div><dt>STATUS</dt><dd>三轮 · 真实权重执行</dd></div>
|
<div><dt>STATUS</dt><dd>三轮 · 真实权重执行</dd></div>
|
||||||
</dl>
|
</dl>
|
||||||
@@ -768,14 +768,14 @@ const toc = [
|
|||||||
<p class="eyebrow"><span>22</span> OFFICIAL WEIGHTS / EXECUTED</p>
|
<p class="eyebrow"><span>22</span> OFFICIAL WEIGHTS / EXECUTED</p>
|
||||||
<h2>从“MLA 与 MoE 的概念”再往前一步：让官方 V2-Lite 权重真的跑起来</h2>
|
<h2>从“MLA 与 MoE 的概念”再往前一步：让官方 V2-Lite 权重真的跑起来</h2>
|
||||||
<p class="lede">
|
<p class="lede">
|
||||||
前面的四联实验负责建立公式与角色合同；下面的四联工件实验固定官方 revision、tokenizer、
|
前面的四联实验负责建立公式与角色合同；下面的五联工件实验固定官方 revision、tokenizer、
|
||||||
模型代码和 checkpoint 第一分片，在 RTX 5090 上连续执行 layer 0–6。它把真实观测、shape 推导、
|
模型代码和 checkpoint 第一分片，在 RTX 5090 上连续执行 layer 0–6。它把真实观测、shape 推导、
|
||||||
实现差距和未覆盖范围放在同一张证据图里。
|
吸收式 latent cache、实现差距和未覆盖范围放在同一张证据图里。
|
||||||
</p>
|
</p>
|
||||||
<div class="artifact-callout">
|
<div class="artifact-callout">
|
||||||
<article><span>X / FORWARD</span><b>7 / 27 layers</b><p>1 个 dense 层 + 6 个 MoE 层；layer 7 因跨分片停止。</p></article>
|
<article><span>X / FORWARD</span><b>7 / 27 layers</b><p>1 个 dense 层 + 6 个 MoE 层；layer 7 因跨分片停止。</p></article>
|
||||||
<article><span>X / ROUTES</span><b>3,240</b><p>90 个有效 token × 6 层 × top-6 routed experts。</p></article>
|
<article><span>X / ROUTES</span><b>3,240</b><p>90 个有效 token × 6 层 × top-6 routed experts。</p></article>
|
||||||
<article><span>X + D / CACHE</span><b>576 ↔ 5,120</b><p>latent 合同与 HF eager 实际展开元素，两张账同时保留。</p></article>
|
<article><span>X / ABSORB CACHE</span><b>266,240 → 29,952 B</b><p>同一真实 layer-1 权重的 naive / absorb active buffers。</p></article>
|
||||||
<article><span>X / RERUN</span><b>31 / 31 exact</b><p>hidden hashes、MLA shapes、loads 与全部 token routes。</p></article>
|
<article><span>X / RERUN</span><b>31 / 31 exact</b><p>hidden hashes、MLA shapes、loads 与全部 token routes。</p></article>
|
||||||
</div>
|
</div>
|
||||||
<DeepSeekArtifactLab />
|
<DeepSeekArtifactLab />
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ const workstreams = [
|
|||||||
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
|
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
|
||||||
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
|
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
|
||||||
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
|
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
|
||||||
{ label: "DeepSeek 专题", value: 87, next: "真实 latent-cache kernel、更大样本负载、FP8/pipeline 与 R1-like RL 复现" },
|
{ label: "DeepSeek 专题", value: 89, next: "SM90 FlashMLA 优化 kernel、更大样本负载、FP8/pipeline 与 R1-like RL 复现" },
|
||||||
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
||||||
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
||||||
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
||||||
@@ -97,12 +97,12 @@ const workstreams = [
|
|||||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||||
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||||
<article><span>✓</span><h3>七十五个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与四联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
<article><span>✓</span><h3>七十六个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与五联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||||
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
||||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
|
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、3,240 次真实专家选择、MLA/eager cache 实现账与 31/31 exact 复跑四联实验。</p></article>
|
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、3,240 次真实专家选择、MLA/eager cache 实现账、31/31 exact 复跑，以及官方 V3 absorb 的 576 元素真实缓存五联实验。</p></article>
|
||||||
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
|
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
|
||||||
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
|
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
|
||||||
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
|
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
|
||||||
@@ -134,7 +134,7 @@ const workstreams = [
|
|||||||
<div class="queue-table">
|
<div class="queue-table">
|
||||||
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
||||||
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
|
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
|
||||||
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>真实 latent-cache kernel / 更大负载样本 → FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>SM90 FlashMLA 优化 kernel / 更大负载样本 → FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||||
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
|
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
|
||||||
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
|
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
|
||||||
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
|
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
|
||||||
|
|||||||
Reference in New Issue
Block a user