feat: add bootstrapped DeepSeek routing corpus
This commit is contained in:
wuyang
+10
-3
@@ -14,7 +14,7 @@
|
||||
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
||||
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
||||
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
||||
| DeepSeek 专题 | 三轮实证进行中 | 89% | SM90 FlashMLA 优化 kernel、更大样本负载、FP8/pipeline 与 R1-like RL 复现 |
|
||||
| DeepSeek 专题 | 三轮实证进行中 | 91% | SM90 FlashMLA kernel、完整 27 层、长度匹配对照、FP8/pipeline 与 R1-like RL 复现 |
|
||||
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
||||
@@ -41,7 +41,7 @@
|
||||
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 九联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等七十六个原创交互视图。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等七十七个原创交互视图。
|
||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||
@@ -179,11 +179,15 @@
|
||||
- [x] 独立吸收实验复跑 byte-exact：两份完整 JSON SHA-256 均为 `6b4c714a…e63d`；实验脚本、机器可读结果与正式研究账本进入开源树。
|
||||
- [x] FlashMLA 架构边界闭合：固定 revision `15f13e50…a7772c6` 与 CUTLASS submodule；官方矩阵、gencode 与 dense runtime guard 只覆盖 SM90/SM100，本机 SM120 不宣称优化 kernel 已执行；两次隔离构建失败位置如实入账。
|
||||
- [x] DeepSeek 吸收式缓存里程碑以源提交 `9ba26da`、不可变镜像 `20260729T065822Z-9ba26da` 发布；OCI digest `sha256:0184d7e8…905659`，复用 NAS `12010→8080`、NPM host 31 / cert 41、门户 `LLM ATLAS / projects / 180`，十六套生产 Chrome 回归全通过；保留 `20260729T062034Z-e864205` 回滚。
|
||||
- [x] DeepSeek 路由探针从 4 条手写 prompt 扩展为 4 个公开域：WikiText-2、TNEWS、HumanEval、GSM8K 各 32 条；固定 revision、原始文件 SHA-256、哈希选样 salt、source ID、tokenizer 与右截断合同。
|
||||
- [x] RTX 5090 执行 128 条 prompt、8,460 个有效 token；layer 1–6 每层产生 50,760 次真实 top-6 选择，六层共 304,560 次，逐 prompt 的 64 维 integer load 全量保留。
|
||||
- [x] 统计单位从 token 修正为 prompt：域内有放回重采样 2,000 次，固定 seed `20260729`，同时报告 token-weighted / prompt-balanced 的 CV、Gini、effective experts、top share、64 专家份额与两两 JSD 95% percentile 区间。
|
||||
- [x] 多域路由正式运行与独立重跑 byte-exact：两份 1.6 MiB JSON SHA-256 均为 `4678a1d1…a09e4`；六联交互实验可切换层与聚合口径，并永久注明不是训练分布、线上流量、专家语义或显著性检验。
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
||||
- [ ] DeepSeek 三轮下一闸门：在官方支持的 SM90 环境执行 FlashMLA 优化 kernel、更大样本 expert-load 置信区间、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] DeepSeek 三轮下一闸门：在官方支持的 SM90 环境执行 FlashMLA 优化 kernel；扩到完整 27 层并做长度/tokenization 匹配对照，再推进 FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
||||
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
||||
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||
@@ -306,6 +310,9 @@
|
||||
| 2026-07-29 | MLA 576 元素缓存从算法合同升级为真实执行 | 官方 V3 naive / absorb 使用同一 V2-Lite 权重与 hidden state；26-token active cache 266,240 B→29,952 B，FP32 代数审计 max \|Δ\| 1.19e-7 |
|
||||
| 2026-07-29 | reference absorb 与 FlashMLA optimized kernel 永久分身份 | 前者已在 RTX 5090 执行；pinned FlashMLA 只列 SM90/SM100 并只生成 sm_90a/sm_100f，本机 SM120 明确标为未支持/未执行 |
|
||||
| 2026-07-29 | DeepSeek 吸收式缓存版本以 `20260729T065822Z-9ba26da` 发布 | OCI digest `sha256:0184d7e8…905659`；复用 `12010→8080`、NPM 31 / cert 41、门户 order 180；十六套生产 Chrome 回归通过，保留 `20260729T062034Z-e864205` 回滚 |
|
||||
| 2026-07-29 | 大样本路由使用四个可重建公开域 | WikiText-2 / TNEWS / HumanEval / GSM8K 各 32 条；只用 text / sentence / prompt / question，答案不输入、代码不执行 |
|
||||
| 2026-07-29 | 路由区间以 prompt 而非 token 为抽样单位 | 同 prompt token 有共同前缀与主题，不能伪装成独立样本；域内 2,000 次 bootstrap，同时保留 token 加权与 prompt 等权 |
|
||||
| 2026-07-29 | 路由分布差异与 expert semantics 永久分开 | Layer 4 中文、Layer 5/6 数学的集中度与域间 JSD 只描述 128-prompt 探针；不命名 expert，不冒充训练/线上总体或显著性检验 |
|
||||
| 2026-07-29 | K3 二轮按 32 张对象账与完整报告顺序重建 | total/active、2.5×、KDA state、深度来源、专家路由、视觉目标、轨迹、缓存与评测协议不再压成一页组件摘要 |
|
||||
| 2026-07-29 | K3 原生视觉事实回到 §2.4 / §3.3 核验 | 删除“先冻结语言模型再解冻”旧表述；明确 MoonViT-V2 从头训练，视觉/文本从开始共同 NTP |
|
||||
| 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项；作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 |
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
||||
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
||||
以及 76 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
以及 77 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
||||
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
||||
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
||||
@@ -28,13 +28,16 @@
|
||||
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
||||
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
||||
DeepSeek 三轮专题以 24 张问题账、10 次技术转向、
|
||||
9 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
10 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
||||
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
|
||||
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
|
||||
`1.19e-7`。FlashMLA 的 SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
|
||||
`1.19e-7`；再以 WikiText-2、TNEWS、HumanEval、GSM8K 各 32 条固定样本执行 304,560 次
|
||||
真实 top-6 路由，用 2,000 次 prompt-level 分层 bootstrap 同报点估计与 95% 区间，独立重跑
|
||||
整份 JSON byte-exact。FlashMLA 的 SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
|
||||
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
|
||||
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)。
|
||||
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
|
||||
[DEEPSEEK_ROUTING_CORPUS_AUDIT.md](./research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md)。
|
||||
其余专题按进度账本持续扩建。
|
||||
|
||||
## 本地开发
|
||||
|
||||
@@ -7,6 +7,17 @@ server {
|
||||
|
||||
charset utf-8;
|
||||
server_tokens off;
|
||||
gzip on;
|
||||
gzip_comp_level 6;
|
||||
gzip_min_length 1024;
|
||||
gzip_vary on;
|
||||
gzip_types
|
||||
text/plain
|
||||
text/css
|
||||
application/javascript
|
||||
application/json
|
||||
application/xml
|
||||
image/svg+xml;
|
||||
|
||||
location = /healthz {
|
||||
access_log off;
|
||||
|
||||
@@ -54,3 +54,49 @@ python experiments/deepseek/compare_v2_lite_traces.py \
|
||||
--second /path/to/trace-2.json \
|
||||
--output src/data/deepseek-v2-lite-trace-repro.json
|
||||
```
|
||||
|
||||
## Fixed public routing corpus
|
||||
|
||||
`v2_lite_routing_corpus.py` keeps the same official layer 0–6 execution boundary
|
||||
but replaces the four authored prompts with 128 source-addressable public
|
||||
prompts:
|
||||
|
||||
- 32 WikiText-2 raw validation passages;
|
||||
- 32 CLUE TNEWS public-test sentences;
|
||||
- 32 OpenAI HumanEval prompts, without solutions/tests or code execution;
|
||||
- 32 OpenAI GSM8K test questions, without answers.
|
||||
|
||||
Selection is the ascending SHA-256 rank of a fixed salt, domain, and source ID.
|
||||
Inputs are truncated to 96 DeepSeek tokens. The six MoE layers therefore produce
|
||||
304,560 actual top-6 routed-expert selections over 8,460 valid tokens.
|
||||
|
||||
The output includes both token-weighted and prompt-balanced distributions. Its
|
||||
95% intervals use 2,000 prompt-level bootstrap resamples within each domain,
|
||||
rather than treating correlated tokens as independent observations.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||||
python -B experiments/deepseek/v2_lite_routing_corpus.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-corpus.json \
|
||||
--per-domain 32 \
|
||||
--max-tokens 96 \
|
||||
--batch-size 16 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T07:45:00+00:00
|
||||
```
|
||||
|
||||
The committed independent rerun is byte-exact. Both JSON files have SHA-256:
|
||||
|
||||
```text
|
||||
4678a1d15395de93ffba757598cc3642bf35e9e07f71d82ddd87c27fc38a09e4
|
||||
```
|
||||
|
||||
See `research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md` for corpus revisions and hashes,
|
||||
metric definitions, interval semantics, results, and claim boundaries.
|
||||
|
||||
@@ -0,0 +1,819 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a fixed, multi-domain DeepSeek-V2-Lite routing corpus.
|
||||
|
||||
The probe executes the official embedding and decoder layers 0 through 6 from
|
||||
the first official BF16 safetensors shard. Layers 1 through 6 are MoE layers.
|
||||
Their real top-6 routed-expert IDs are aggregated at prompt level before any
|
||||
statistics are computed.
|
||||
|
||||
Sampling is deterministic and source-addressable. Confidence intervals use a
|
||||
prompt-level, within-domain bootstrap; tokens from the same prompt are never
|
||||
treated as independent observations.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gc
|
||||
import gzip
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import platform
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import types
|
||||
from datetime import datetime, timezone
|
||||
from itertools import combinations
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from safetensors import safe_open
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
|
||||
DOMAIN_ORDER = ("english", "chinese", "code", "math")
|
||||
DOMAIN_LABELS = {
|
||||
"english": "English encyclopedia",
|
||||
"chinese": "中文新闻",
|
||||
"code": "Python code",
|
||||
"math": "Grade-school math",
|
||||
}
|
||||
MIN_TOKENS = {
|
||||
"english": 24,
|
||||
"chinese": 8,
|
||||
"code": 24,
|
||||
"math": 16,
|
||||
}
|
||||
SAMPLE_SALT = "llm-atlas-deepseek-routing-v1"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--artifact-dir", type=Path, required=True)
|
||||
parser.add_argument("--human-eval", type=Path, required=True)
|
||||
parser.add_argument("--gsm8k", type=Path, required=True)
|
||||
parser.add_argument("--tnews", type=Path, required=True)
|
||||
parser.add_argument("--tnews-archive", type=Path, required=True)
|
||||
parser.add_argument("--wikitext", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--per-domain", type=int, default=32)
|
||||
parser.add_argument("--max-tokens", type=int, default=96)
|
||||
parser.add_argument("--batch-size", type=int, default=16)
|
||||
parser.add_argument("--layers", type=int, default=7)
|
||||
parser.add_argument("--bootstrap", type=int, default=2000)
|
||||
parser.add_argument("--seed", type=int, default=20260729)
|
||||
parser.add_argument("--device", default="cuda")
|
||||
parser.add_argument("--captured-at", default=None)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def text_sha256(value: str) -> str:
|
||||
return hashlib.sha256(value.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def git_revision(path: Path) -> str | None:
|
||||
current = path.resolve()
|
||||
for candidate in (current, *current.parents):
|
||||
if (candidate / ".git").exists():
|
||||
return subprocess.check_output(
|
||||
["git", "-C", str(candidate), "rev-parse", "HEAD"],
|
||||
text=True,
|
||||
).strip()
|
||||
return None
|
||||
|
||||
|
||||
def scoped_seed(seed: int, scope: str) -> int:
|
||||
payload = f"{seed}:{scope}".encode()
|
||||
return int.from_bytes(hashlib.sha256(payload).digest()[:8], "big")
|
||||
|
||||
|
||||
def load_official_modules(root: Path) -> tuple[Any, Any]:
|
||||
package_name = "deepseek_v2_lite_official_routing_corpus"
|
||||
package = types.ModuleType(package_name)
|
||||
package.__path__ = [str(root)]
|
||||
sys.modules[package_name] = package
|
||||
|
||||
loaded = {}
|
||||
for leaf in ("configuration_deepseek", "modeling_deepseek"):
|
||||
name = f"{package_name}.{leaf}"
|
||||
spec = importlib.util.spec_from_file_location(name, root / f"{leaf}.py")
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"cannot load official module: {leaf}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[name] = module
|
||||
spec.loader.exec_module(module)
|
||||
loaded[leaf] = module
|
||||
return loaded["configuration_deepseek"], loaded["modeling_deepseek"]
|
||||
|
||||
|
||||
def read_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
with path.open(encoding="utf-8") as handle:
|
||||
return [json.loads(line) for line in handle if line.strip()]
|
||||
|
||||
|
||||
def load_candidates(args: argparse.Namespace) -> dict[str, list[dict[str, str]]]:
|
||||
with gzip.open(args.human_eval, "rt", encoding="utf-8") as handle:
|
||||
human_eval = [json.loads(line) for line in handle if line.strip()]
|
||||
gsm8k = read_jsonl(args.gsm8k)
|
||||
tnews = read_jsonl(args.tnews)
|
||||
wikitext = pq.read_table(args.wikitext, columns=["text"]).column("text").to_pylist()
|
||||
|
||||
return {
|
||||
"english": [
|
||||
{
|
||||
"id": f"wikitext2/raw-validation/{index:04d}",
|
||||
"text": str(text).strip(),
|
||||
}
|
||||
for index, text in enumerate(wikitext)
|
||||
if str(text).strip()
|
||||
and not re.fullmatch(r"=+\s*.*?\s*=+", str(text).strip())
|
||||
],
|
||||
"chinese": [
|
||||
{
|
||||
"id": f"tnews/test/{row['id']}",
|
||||
"text": str(row["sentence"]).strip(),
|
||||
}
|
||||
for row in tnews
|
||||
if str(row.get("sentence", "")).strip()
|
||||
],
|
||||
"code": [
|
||||
{
|
||||
"id": str(row["task_id"]),
|
||||
"text": str(row["prompt"]).rstrip(),
|
||||
}
|
||||
for row in human_eval
|
||||
if str(row.get("prompt", "")).strip()
|
||||
],
|
||||
"math": [
|
||||
{
|
||||
"id": f"gsm8k/test/{index:04d}",
|
||||
"text": str(row["question"]).strip(),
|
||||
}
|
||||
for index, row in enumerate(gsm8k)
|
||||
if str(row.get("question", "")).strip()
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def select_corpus(
|
||||
tokenizer: Any,
|
||||
candidates: dict[str, list[dict[str, str]]],
|
||||
per_domain: int,
|
||||
max_tokens: int,
|
||||
) -> tuple[list[dict[str, Any]], dict[str, dict[str, int]]]:
|
||||
selected: list[dict[str, Any]] = []
|
||||
counts: dict[str, dict[str, int]] = {}
|
||||
for domain in DOMAIN_ORDER:
|
||||
rows = candidates[domain]
|
||||
texts = [row["text"] for row in rows]
|
||||
encoded: list[list[int]] = []
|
||||
for start in range(0, len(texts), 512):
|
||||
result = tokenizer(
|
||||
texts[start : start + 512],
|
||||
add_special_tokens=True,
|
||||
truncation=True,
|
||||
max_length=max_tokens,
|
||||
padding=False,
|
||||
)
|
||||
encoded.extend(result.input_ids)
|
||||
|
||||
eligible = []
|
||||
for row, token_ids in zip(rows, encoded, strict=True):
|
||||
if len(token_ids) < MIN_TOKENS[domain]:
|
||||
continue
|
||||
rank = hashlib.sha256(
|
||||
f"{SAMPLE_SALT}|{domain}|{row['id']}".encode()
|
||||
).hexdigest()
|
||||
eligible.append(
|
||||
{
|
||||
"id": row["id"],
|
||||
"domain": domain,
|
||||
"label": DOMAIN_LABELS[domain],
|
||||
"text": row["text"],
|
||||
"text_sha256": text_sha256(row["text"]),
|
||||
"characters": len(row["text"]),
|
||||
"token_ids": token_ids,
|
||||
"tokens": len(token_ids),
|
||||
"selection_rank": rank,
|
||||
}
|
||||
)
|
||||
eligible.sort(key=lambda row: (row["selection_rank"], row["id"]))
|
||||
if len(eligible) < per_domain:
|
||||
raise RuntimeError(
|
||||
f"{domain} has only {len(eligible)} eligible prompts; need {per_domain}"
|
||||
)
|
||||
domain_selection = eligible[:per_domain]
|
||||
for within_domain_index, row in enumerate(domain_selection):
|
||||
row["within_domain_index"] = within_domain_index
|
||||
selected.extend(domain_selection)
|
||||
counts[domain] = {
|
||||
"candidate_records_after_text_filter": len(rows),
|
||||
"eligible_records": len(eligible),
|
||||
"selected_records": len(domain_selection),
|
||||
"valid_tokens": sum(row["tokens"] for row in domain_selection),
|
||||
}
|
||||
return selected, counts
|
||||
|
||||
|
||||
def make_batches(
|
||||
samples: list[dict[str, Any]],
|
||||
batch_size: int,
|
||||
pad_token_id: int,
|
||||
) -> list[dict[str, Any]]:
|
||||
ordered = sorted(
|
||||
samples,
|
||||
key=lambda row: (row["tokens"], row["domain"], row["id"]),
|
||||
)
|
||||
batches = []
|
||||
for start in range(0, len(ordered), batch_size):
|
||||
rows = ordered[start : start + batch_size]
|
||||
sequence = max(row["tokens"] for row in rows)
|
||||
input_ids = torch.full(
|
||||
(len(rows), sequence),
|
||||
pad_token_id,
|
||||
dtype=torch.long,
|
||||
)
|
||||
attention_mask = torch.zeros((len(rows), sequence), dtype=torch.long)
|
||||
for index, row in enumerate(rows):
|
||||
length = row["tokens"]
|
||||
input_ids[index, :length] = torch.tensor(row["token_ids"])
|
||||
attention_mask[index, :length] = 1
|
||||
batches.append(
|
||||
{
|
||||
"samples": rows,
|
||||
"input_ids": input_ids,
|
||||
"attention_mask": attention_mask,
|
||||
"padded_sequence": sequence,
|
||||
}
|
||||
)
|
||||
return batches
|
||||
|
||||
|
||||
def distribution(loads: np.ndarray, mode: str) -> np.ndarray:
|
||||
if mode == "token_weighted":
|
||||
values = loads.sum(axis=0, dtype=np.float64)
|
||||
return values / values.sum()
|
||||
if mode == "prompt_balanced":
|
||||
prompt_distributions = loads / loads.sum(axis=1, keepdims=True)
|
||||
values = prompt_distributions.mean(axis=0, dtype=np.float64)
|
||||
return values / values.sum()
|
||||
raise ValueError(mode)
|
||||
|
||||
|
||||
def metric_vector(distributions: np.ndarray) -> dict[str, np.ndarray]:
|
||||
values = np.atleast_2d(distributions).astype(np.float64, copy=False)
|
||||
expert_count = values.shape[1]
|
||||
means = values.mean(axis=1)
|
||||
cv = values.std(axis=1) / means
|
||||
ordered = np.sort(values, axis=1)
|
||||
indices = np.arange(1, expert_count + 1, dtype=np.float64)
|
||||
gini = (
|
||||
((2 * indices - expert_count - 1) * ordered).sum(axis=1)
|
||||
/ (expert_count * ordered.sum(axis=1))
|
||||
)
|
||||
log_values = np.zeros_like(values)
|
||||
np.log(values, out=log_values, where=values > 0)
|
||||
entropy = -(values * log_values).sum(axis=1)
|
||||
return {
|
||||
"cv": cv,
|
||||
"gini": gini,
|
||||
"effective_experts": np.exp(entropy),
|
||||
"top_expert_share": values.max(axis=1),
|
||||
"used_experts": (values > 0).sum(axis=1).astype(np.float64),
|
||||
}
|
||||
|
||||
|
||||
def interval(values: np.ndarray) -> list[float]:
|
||||
low, high = np.quantile(values, [0.025, 0.975], axis=0)
|
||||
if np.ndim(low) == 0:
|
||||
return [float(low), float(high)]
|
||||
return np.stack([low, high], axis=-1).tolist()
|
||||
|
||||
|
||||
def bootstrap_domain(
|
||||
loads: np.ndarray,
|
||||
mode: str,
|
||||
replicates: int,
|
||||
seed: int,
|
||||
scope: str,
|
||||
) -> dict[str, Any]:
|
||||
point_distribution = distribution(loads, mode)
|
||||
prompt_count = loads.shape[0]
|
||||
rng = np.random.default_rng(scoped_seed(seed, scope))
|
||||
sampled = rng.integers(
|
||||
0,
|
||||
prompt_count,
|
||||
size=(replicates, prompt_count),
|
||||
endpoint=False,
|
||||
)
|
||||
if mode == "token_weighted":
|
||||
bootstrap_loads = loads[sampled].sum(axis=1, dtype=np.float64)
|
||||
bootstrap_distributions = bootstrap_loads / bootstrap_loads.sum(
|
||||
axis=1, keepdims=True
|
||||
)
|
||||
else:
|
||||
prompt_distributions = loads / loads.sum(axis=1, keepdims=True)
|
||||
bootstrap_distributions = prompt_distributions[sampled].mean(axis=1)
|
||||
bootstrap_distributions /= bootstrap_distributions.sum(axis=1, keepdims=True)
|
||||
|
||||
point_metrics = metric_vector(point_distribution)
|
||||
bootstrap_metrics = metric_vector(bootstrap_distributions)
|
||||
metrics = {
|
||||
name: {
|
||||
"point": float(point_metrics[name][0]),
|
||||
"ci95": interval(bootstrap_metrics[name]),
|
||||
}
|
||||
for name in point_metrics
|
||||
}
|
||||
order = np.argsort(point_distribution)[::-1][:8]
|
||||
return {
|
||||
"distribution": point_distribution.tolist(),
|
||||
"expert_share_ci95": interval(bootstrap_distributions),
|
||||
"metrics": metrics,
|
||||
"top_experts": [
|
||||
{
|
||||
"expert": int(expert),
|
||||
"share": float(point_distribution[expert]),
|
||||
"ci95": interval(bootstrap_distributions[:, expert]),
|
||||
}
|
||||
for expert in order
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def js_divergence(left: np.ndarray, right: np.ndarray) -> np.ndarray:
|
||||
left = np.atleast_2d(left).astype(np.float64, copy=False)
|
||||
right = np.atleast_2d(right).astype(np.float64, copy=False)
|
||||
midpoint = (left + right) / 2
|
||||
left_log_ratio = np.zeros_like(left)
|
||||
right_log_ratio = np.zeros_like(right)
|
||||
left_ratio = np.ones_like(left)
|
||||
right_ratio = np.ones_like(right)
|
||||
np.divide(left, midpoint, out=left_ratio, where=left > 0)
|
||||
np.divide(right, midpoint, out=right_ratio, where=right > 0)
|
||||
np.log(left_ratio, out=left_log_ratio, where=left > 0)
|
||||
np.log(right_ratio, out=right_log_ratio, where=right > 0)
|
||||
return 0.5 * (
|
||||
(left * left_log_ratio).sum(axis=1)
|
||||
+ (right * right_log_ratio).sum(axis=1)
|
||||
)
|
||||
|
||||
|
||||
def bootstrap_pair(
|
||||
left: np.ndarray,
|
||||
right: np.ndarray,
|
||||
mode: str,
|
||||
replicates: int,
|
||||
seed: int,
|
||||
scope: str,
|
||||
) -> dict[str, Any]:
|
||||
left_point = distribution(left, mode)
|
||||
right_point = distribution(right, mode)
|
||||
rng = np.random.default_rng(scoped_seed(seed, scope))
|
||||
left_indices = rng.integers(
|
||||
0, left.shape[0], size=(replicates, left.shape[0]), endpoint=False
|
||||
)
|
||||
right_indices = rng.integers(
|
||||
0, right.shape[0], size=(replicates, right.shape[0]), endpoint=False
|
||||
)
|
||||
|
||||
if mode == "token_weighted":
|
||||
left_boot = left[left_indices].sum(axis=1, dtype=np.float64)
|
||||
right_boot = right[right_indices].sum(axis=1, dtype=np.float64)
|
||||
left_boot /= left_boot.sum(axis=1, keepdims=True)
|
||||
right_boot /= right_boot.sum(axis=1, keepdims=True)
|
||||
else:
|
||||
left_prompt = left / left.sum(axis=1, keepdims=True)
|
||||
right_prompt = right / right.sum(axis=1, keepdims=True)
|
||||
left_boot = left_prompt[left_indices].mean(axis=1)
|
||||
right_boot = right_prompt[right_indices].mean(axis=1)
|
||||
left_boot /= left_boot.sum(axis=1, keepdims=True)
|
||||
right_boot /= right_boot.sum(axis=1, keepdims=True)
|
||||
|
||||
bootstrap_jsd = js_divergence(left_boot, right_boot)
|
||||
return {
|
||||
"point": float(js_divergence(left_point, right_point)[0]),
|
||||
"ci95": interval(bootstrap_jsd),
|
||||
"unit": "nats",
|
||||
"upper_bound": math.log(2),
|
||||
}
|
||||
|
||||
|
||||
def layer_statistics(
|
||||
prompt_rows: list[dict[str, Any]],
|
||||
replicates: int,
|
||||
seed: int,
|
||||
layer_index: int,
|
||||
) -> dict[str, Any]:
|
||||
loads_by_domain = {
|
||||
domain: np.asarray(
|
||||
[row["load"] for row in prompt_rows if row["domain"] == domain],
|
||||
dtype=np.int64,
|
||||
)
|
||||
for domain in DOMAIN_ORDER
|
||||
}
|
||||
modes: dict[str, Any] = {}
|
||||
for mode in ("token_weighted", "prompt_balanced"):
|
||||
domains = {
|
||||
domain: bootstrap_domain(
|
||||
loads_by_domain[domain],
|
||||
mode,
|
||||
replicates,
|
||||
seed,
|
||||
f"layer={layer_index}|mode={mode}|domain={domain}",
|
||||
)
|
||||
for domain in DOMAIN_ORDER
|
||||
}
|
||||
pairs = []
|
||||
for left, right in combinations(DOMAIN_ORDER, 2):
|
||||
pairs.append(
|
||||
{
|
||||
"left": left,
|
||||
"right": right,
|
||||
"js_divergence": bootstrap_pair(
|
||||
loads_by_domain[left],
|
||||
loads_by_domain[right],
|
||||
mode,
|
||||
replicates,
|
||||
seed,
|
||||
f"layer={layer_index}|mode={mode}|pair={left}:{right}",
|
||||
),
|
||||
}
|
||||
)
|
||||
modes[mode] = {"domains": domains, "pairs": pairs}
|
||||
return modes
|
||||
|
||||
|
||||
def gpu_identity(device: torch.device) -> dict[str, Any] | None:
|
||||
if device.type != "cuda":
|
||||
return None
|
||||
index = device.index or 0
|
||||
properties = torch.cuda.get_device_properties(index)
|
||||
driver = subprocess.check_output(
|
||||
[
|
||||
"nvidia-smi",
|
||||
"--query-gpu=driver_version",
|
||||
"--format=csv,noheader,nounits",
|
||||
f"--id={index}",
|
||||
],
|
||||
text=True,
|
||||
).strip()
|
||||
return {
|
||||
"name": properties.name,
|
||||
"total_memory_bytes": properties.total_memory,
|
||||
"compute_capability": list(torch.cuda.get_device_capability(index)),
|
||||
"driver": driver,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
root = args.artifact_dir.resolve()
|
||||
shard = root / "model-00001-of-000004.safetensors"
|
||||
required = [
|
||||
root / "config.json",
|
||||
root / "configuration_deepseek.py",
|
||||
root / "modeling_deepseek.py",
|
||||
root / "model.safetensors.index.json",
|
||||
root / "tokenizer.json",
|
||||
shard,
|
||||
args.human_eval,
|
||||
args.gsm8k,
|
||||
args.tnews,
|
||||
args.tnews_archive,
|
||||
args.wikitext,
|
||||
]
|
||||
missing = [str(path) for path in required if not path.exists()]
|
||||
if missing:
|
||||
raise FileNotFoundError(f"missing artifacts: {missing}")
|
||||
if args.device.startswith("cuda") and not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA requested but unavailable")
|
||||
if not 2 <= args.layers <= 7:
|
||||
raise ValueError("need layer 0 plus at least one MoE layer; shard ends at layer 6")
|
||||
if args.per_domain < 2:
|
||||
raise ValueError("per-domain sample must be at least two")
|
||||
if args.bootstrap < 100:
|
||||
raise ValueError("bootstrap replicates must be at least 100")
|
||||
|
||||
torch.manual_seed(args.seed)
|
||||
torch.cuda.manual_seed_all(args.seed)
|
||||
torch.backends.cuda.matmul.allow_tf32 = False
|
||||
|
||||
configuration, modeling = load_official_modules(root)
|
||||
config = configuration.DeepseekV2Config.from_pretrained(root)
|
||||
config._attn_implementation = "eager"
|
||||
tokenizer = AutoTokenizer.from_pretrained(
|
||||
root,
|
||||
trust_remote_code=True,
|
||||
local_files_only=True,
|
||||
)
|
||||
if tokenizer.pad_token_id is None:
|
||||
tokenizer.pad_token = tokenizer.eos_token
|
||||
tokenizer.padding_side = "right"
|
||||
|
||||
candidates = load_candidates(args)
|
||||
samples, corpus_counts = select_corpus(
|
||||
tokenizer,
|
||||
candidates,
|
||||
args.per_domain,
|
||||
args.max_tokens,
|
||||
)
|
||||
batches = make_batches(samples, args.batch_size, tokenizer.pad_token_id)
|
||||
device = torch.device(args.device)
|
||||
|
||||
with safe_open(shard, framework="pt", device="cpu") as handle:
|
||||
embedding = handle.get_tensor("model.embed_tokens.weight")
|
||||
for batch in batches:
|
||||
batch["hidden"] = F.embedding(batch["input_ids"], embedding)
|
||||
del embedding
|
||||
|
||||
layer_results = []
|
||||
for layer_index in range(args.layers):
|
||||
prefix = f"model.layers.{layer_index}."
|
||||
with safe_open(shard, framework="pt", device="cpu") as handle:
|
||||
state = {
|
||||
key[len(prefix) :]: handle.get_tensor(key)
|
||||
for key in handle.keys()
|
||||
if key.startswith(prefix)
|
||||
}
|
||||
state_numel = sum(value.numel() for value in state.values())
|
||||
state_bytes = sum(value.numel() * value.element_size() for value in state.values())
|
||||
with torch.device("meta"):
|
||||
layer = modeling.DeepseekV2DecoderLayer(config, layer_index)
|
||||
layer.to_empty(device="cpu")
|
||||
layer.load_state_dict(state, strict=True, assign=True)
|
||||
del state
|
||||
layer = layer.to(device=device, dtype=torch.bfloat16).eval()
|
||||
|
||||
prompt_rows = []
|
||||
next_hidden = []
|
||||
for batch in batches:
|
||||
hidden = batch["hidden"].to(device)
|
||||
attention_mask = batch["attention_mask"].to(device)
|
||||
sequence = batch["padded_sequence"]
|
||||
position_ids = torch.arange(sequence, device=device).unsqueeze(0)
|
||||
causal_mask = modeling._prepare_4d_causal_attention_mask(
|
||||
attention_mask,
|
||||
batch["input_ids"].shape,
|
||||
hidden,
|
||||
0,
|
||||
)
|
||||
captures: dict[str, torch.Tensor] = {}
|
||||
hook = None
|
||||
if layer_index > 0:
|
||||
|
||||
def capture_gate(_module: Any, _inputs: Any, output: Any) -> None:
|
||||
captures["topk_ids"] = output[0].detach().cpu()
|
||||
|
||||
hook = layer.mlp.gate.register_forward_hook(capture_gate)
|
||||
|
||||
with torch.inference_mode():
|
||||
outputs = layer(
|
||||
hidden,
|
||||
attention_mask=causal_mask,
|
||||
position_ids=position_ids,
|
||||
past_key_value=None,
|
||||
use_cache=False,
|
||||
)
|
||||
next_hidden.append(outputs[0].cpu())
|
||||
if hook is not None:
|
||||
hook.remove()
|
||||
topk_ids = captures["topk_ids"].view(
|
||||
len(batch["samples"]),
|
||||
sequence,
|
||||
config.num_experts_per_tok,
|
||||
)
|
||||
for sample_index, sample in enumerate(batch["samples"]):
|
||||
length = sample["tokens"]
|
||||
load = torch.bincount(
|
||||
topk_ids[sample_index, :length].flatten(),
|
||||
minlength=config.n_routed_experts,
|
||||
)
|
||||
prompt_rows.append(
|
||||
{
|
||||
"id": sample["id"],
|
||||
"domain": sample["domain"],
|
||||
"tokens": length,
|
||||
"routes": int(load.sum()),
|
||||
"load": load.tolist(),
|
||||
}
|
||||
)
|
||||
del hidden, attention_mask, causal_mask, outputs
|
||||
|
||||
for batch, hidden in zip(batches, next_hidden, strict=True):
|
||||
batch["hidden"] = hidden
|
||||
|
||||
result: dict[str, Any] = {
|
||||
"layer": layer_index,
|
||||
"ffn": "dense" if layer_index == 0 else "moe",
|
||||
"state_numel": state_numel,
|
||||
"state_bytes": state_bytes,
|
||||
}
|
||||
if layer_index > 0:
|
||||
prompt_rows.sort(
|
||||
key=lambda row: (
|
||||
DOMAIN_ORDER.index(row["domain"]),
|
||||
row["id"],
|
||||
)
|
||||
)
|
||||
result["valid_tokens"] = sum(row["tokens"] for row in prompt_rows)
|
||||
result["routes"] = sum(row["routes"] for row in prompt_rows)
|
||||
result["prompts"] = prompt_rows
|
||||
result["statistics"] = layer_statistics(
|
||||
prompt_rows,
|
||||
args.bootstrap,
|
||||
args.seed,
|
||||
layer_index,
|
||||
)
|
||||
layer_results.append(result)
|
||||
|
||||
del layer, next_hidden
|
||||
gc.collect()
|
||||
if device.type == "cuda":
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
captured_at = args.captured_at or datetime.now(timezone.utc).isoformat()
|
||||
index = json.loads((root / "model.safetensors.index.json").read_text())
|
||||
selected_identity = [
|
||||
{
|
||||
"id": sample["id"],
|
||||
"domain": sample["domain"],
|
||||
"within_domain_index": sample["within_domain_index"],
|
||||
"selection_rank": sample["selection_rank"],
|
||||
"text_sha256": sample["text_sha256"],
|
||||
"characters": sample["characters"],
|
||||
"tokens": sample["tokens"],
|
||||
}
|
||||
for sample in samples
|
||||
]
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"captured_at": captured_at,
|
||||
"evidence_identity": "X / official BF16 weights, fixed public corpus, local truncated forward",
|
||||
"boundary": {
|
||||
"model": "DeepSeek-V2-Lite base",
|
||||
"executed_layers": list(range(args.layers)),
|
||||
"measured_moe_layers": list(range(1, args.layers)),
|
||||
"total_model_layers": config.num_hidden_layers,
|
||||
"full_model_generation": False,
|
||||
"training_or_online_load": False,
|
||||
"expert_semantics_inferred": False,
|
||||
"population": (
|
||||
f"{len(samples)} fixed public prompts across four domains; "
|
||||
"not training data, online traffic, or a task-performance benchmark"
|
||||
),
|
||||
"code_execution": False,
|
||||
"answers_used": False,
|
||||
},
|
||||
"provenance": {
|
||||
"model": {
|
||||
"huggingface_model": "deepseek-ai/DeepSeek-V2-Lite",
|
||||
"huggingface_revision": "604d5664dddd88a0433dbae533b7fe9472482de0",
|
||||
"sha256": {
|
||||
"config": sha256(root / "config.json"),
|
||||
"modeling_code": sha256(root / "modeling_deepseek.py"),
|
||||
"tokenizer": sha256(root / "tokenizer.json"),
|
||||
"index": sha256(root / "model.safetensors.index.json"),
|
||||
"shard_1": sha256(shard),
|
||||
},
|
||||
"checkpoint_tensor_bytes": index["metadata"]["total_size"],
|
||||
"shard_1_bytes": shard.stat().st_size,
|
||||
},
|
||||
"corpora": {
|
||||
"english": {
|
||||
"name": "WikiText-2 raw validation",
|
||||
"url": "https://huggingface.co/datasets/Salesforce/wikitext",
|
||||
"revision": "b08601e04326c79dfdd32d625aee71d232d685c3",
|
||||
"file_sha256": sha256(args.wikitext),
|
||||
"field_used": "text",
|
||||
},
|
||||
"chinese": {
|
||||
"name": "CLUE TNEWS public test",
|
||||
"url": "https://github.com/CLUEbenchmark/CLUE",
|
||||
"download_url": "https://storage.googleapis.com/cluebenchmark/tasks/tnews_public.zip",
|
||||
"revision": "9e61ddd3659ddb57ed82b4d0ba0a8613dfb55a2e",
|
||||
"archive_sha256": sha256(args.tnews_archive),
|
||||
"file_sha256": sha256(args.tnews),
|
||||
"field_used": "sentence",
|
||||
},
|
||||
"code": {
|
||||
"name": "OpenAI HumanEval",
|
||||
"url": "https://github.com/openai/human-eval",
|
||||
"revision": git_revision(args.human_eval),
|
||||
"file_sha256": sha256(args.human_eval),
|
||||
"field_used": "prompt",
|
||||
},
|
||||
"math": {
|
||||
"name": "OpenAI GSM8K test",
|
||||
"url": "https://github.com/openai/grade-school-math",
|
||||
"revision": git_revision(args.gsm8k),
|
||||
"file_sha256": sha256(args.gsm8k),
|
||||
"field_used": "question",
|
||||
},
|
||||
},
|
||||
},
|
||||
"environment": {
|
||||
"python": platform.python_version(),
|
||||
"platform": platform.platform(),
|
||||
"torch": torch.__version__,
|
||||
"torch_cuda": torch.version.cuda,
|
||||
"transformers": __import__("transformers").__version__,
|
||||
"safetensors": __import__("safetensors").__version__,
|
||||
"numpy": np.__version__,
|
||||
"pyarrow": __import__("pyarrow").__version__,
|
||||
"device": str(device),
|
||||
"gpu": gpu_identity(device),
|
||||
"matmul_allow_tf32": torch.backends.cuda.matmul.allow_tf32,
|
||||
},
|
||||
"configuration": {
|
||||
"layers": config.num_hidden_layers,
|
||||
"hidden": config.hidden_size,
|
||||
"routed_experts": config.n_routed_experts,
|
||||
"active_routed_experts": config.num_experts_per_tok,
|
||||
"shared_experts": config.n_shared_experts,
|
||||
"first_dense_layers": config.first_k_dense_replace,
|
||||
"router_scoring": config.scoring_func,
|
||||
"router_topk_method": config.topk_method,
|
||||
"normalize_selected_weights": config.norm_topk_prob,
|
||||
},
|
||||
"corpus_contract": {
|
||||
"domains": list(DOMAIN_ORDER),
|
||||
"domain_labels": DOMAIN_LABELS,
|
||||
"sample_salt": SAMPLE_SALT,
|
||||
"selection": "ascending SHA256(salt|domain|source_id), then source_id",
|
||||
"per_domain": args.per_domain,
|
||||
"max_tokens": args.max_tokens,
|
||||
"minimum_tokens": MIN_TOKENS,
|
||||
"special_tokens": True,
|
||||
"truncation": "right",
|
||||
"counts": corpus_counts,
|
||||
"selected": selected_identity,
|
||||
},
|
||||
"inference_contract": {
|
||||
"batch_size": args.batch_size,
|
||||
"batches": len(batches),
|
||||
"batch_order": "ascending token count, then domain, then source id",
|
||||
"attention": "official eager",
|
||||
"dtype": "BF16",
|
||||
"total_prompts": len(samples),
|
||||
"valid_tokens": sum(sample["tokens"] for sample in samples),
|
||||
"routes_per_moe_layer": sum(sample["tokens"] for sample in samples)
|
||||
* config.num_experts_per_tok,
|
||||
},
|
||||
"statistical_contract": {
|
||||
"resampling_unit": "prompt",
|
||||
"strata": "domain",
|
||||
"replicates": args.bootstrap,
|
||||
"seed": args.seed,
|
||||
"interval": "95% percentile bootstrap",
|
||||
"modes": {
|
||||
"token_weighted": "sum route counts, so longer prompts contribute more",
|
||||
"prompt_balanced": "normalize each prompt first, then give every prompt equal weight",
|
||||
},
|
||||
"metrics": {
|
||||
"cv": "population standard deviation across 64 expert shares divided by their mean",
|
||||
"gini": "Gini coefficient across 64 expert shares",
|
||||
"effective_experts": "exp(Shannon entropy) in nats",
|
||||
"top_expert_share": "largest routed-expert share",
|
||||
"used_experts": "experts with nonzero share in the resample",
|
||||
"js_divergence": "Jensen-Shannon divergence in nats; bounded by ln(2)",
|
||||
},
|
||||
"multiple_comparison_correction": False,
|
||||
},
|
||||
"layers": layer_results,
|
||||
}
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"prompts": len(samples),
|
||||
"tokens": result["inference_contract"]["valid_tokens"],
|
||||
"routes_per_moe_layer": result["inference_contract"][
|
||||
"routes_per_moe_layer"
|
||||
],
|
||||
"measured_moe_layers": args.layers - 1,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,386 @@
|
||||
# DeepSeek-V2-Lite 多域路由语料审计
|
||||
|
||||
> 状态：第三轮真实权重执行 / 固定公开语料 / prompt-level bootstrap
|
||||
> 捕获时间：2026-07-29T07:45:00+00:00
|
||||
> 证据身份：X（本机真实执行）+ O（官方工件）+ S（统计摘要）
|
||||
> 输出：
|
||||
> `src/data/deepseek-v2-lite-routing-corpus.json` 与独立重跑
|
||||
> `src/data/deepseek-v2-lite-routing-corpus-repro.json`
|
||||
|
||||
## 1. 这轮到底补了什么
|
||||
|
||||
上一轮 `v2_lite_trace.py` 的任务是做“显微镜”：4 条手写 prompt、90 个有效
|
||||
token，逐 token 保存 top-6 expert ID 与 gate weight。它适合解释 router 实际输出
|
||||
什么，却不适合回答“换一批文本，这个现象还在不在”。
|
||||
|
||||
本轮把问题改成：
|
||||
|
||||
1. 固定四种可公开重建的文本域；
|
||||
2. 每域固定 32 条，共 128 条 prompt；
|
||||
3. 对官方 DeepSeek-V2-Lite 的 embedding 与 layer 0–6 做连续 BF16 forward；
|
||||
4. 在 layer 1–6 捕获每个有效 token 的真实 top-6 routed-expert ID；
|
||||
5. 先聚合到 prompt，再以 prompt 为单位做 2,000 次分层 bootstrap；
|
||||
6. 同时报告点估计、95% 区间、两种聚合口径和明确停止线。
|
||||
|
||||
正式运行包含：
|
||||
|
||||
| 数量 | 值 |
|
||||
| --- | ---: |
|
||||
| prompt | 128 |
|
||||
| 有效 token | 8,460 |
|
||||
| 每个 MoE 层的 routed-expert 选择 | 50,760 |
|
||||
| 实测 MoE 层 | 6 |
|
||||
| routed-expert 选择总数 | **304,560** |
|
||||
| routed experts / layer | 64 |
|
||||
| active routed experts / token | 6 |
|
||||
| bootstrap replicates | 2,000 |
|
||||
|
||||
这里的 304,560 是 `8,460 tokens × top-6 × 6 MoE layers`，不是模型参数量、
|
||||
训练 token 数或线上请求量。
|
||||
|
||||
## 2. 语料合同
|
||||
|
||||
### 2.1 四个公开来源
|
||||
|
||||
| 域 | 公开来源 | 只使用的字段 | 固定 revision / 文件 SHA-256 |
|
||||
| --- | --- | --- | --- |
|
||||
| 英文百科 | [Salesforce WikiText-2 raw validation](https://huggingface.co/datasets/Salesforce/wikitext/tree/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-2-raw-v1) | `text` | revision `b08601e…685c3`; parquet `204929b7…11c4c` |
|
||||
| 中文新闻 | [CLUE TNEWS](https://github.com/CLUEbenchmark/CLUE/tree/9e61ddd3659ddb57ed82b4d0ba0a8613dfb55a2e) | `sentence` | repo `9e61ddd…55a2e`; archive `77c476e7…33462`; test `74f19932…a0ca8` |
|
||||
| Python 代码 | [OpenAI HumanEval](https://github.com/openai/human-eval/tree/6d43fb980f9fee3c892a914eda09951f772ad10d) | `prompt` | revision `6d43fb9…ad10d`; gzip `b796127e…979ef` |
|
||||
| 小学数学 | [OpenAI GSM8K](https://github.com/openai/grade-school-math/tree/3101c7d5072418e28b9008a6636bde82a006892c) | `question` | revision `3101c7d…6892c`; test JSONL `3730d312…d14` |
|
||||
|
||||
边界：
|
||||
|
||||
- HumanEval 的 `canonical_solution`、`test` 没有进入模型，也没有执行代码；
|
||||
- GSM8K 的 `answer` 没有进入模型；
|
||||
- TNEWS 的标签和关键词没有进入模型；
|
||||
- WikiText 只取 raw validation 的正文候选，不连接其他 split；
|
||||
- 这四个来源被当作“探针语料”，不是任务正确率评测。
|
||||
|
||||
### 2.2 确定性选样
|
||||
|
||||
固定 salt：
|
||||
|
||||
```text
|
||||
llm-atlas-deepseek-routing-v1
|
||||
```
|
||||
|
||||
每域候选按下式升序排列，再取前 32 条：
|
||||
|
||||
```text
|
||||
SHA256(salt | domain | source_id), then source_id
|
||||
```
|
||||
|
||||
这比 `random.sample` 多保留了两层可复现性：
|
||||
|
||||
- 不依赖 Python / NumPy 随机实现；
|
||||
- 每条输出都保留 source ID、selection rank、原文 SHA-256、字符数与 token 数，
|
||||
可以从固定源文件重新定位。
|
||||
|
||||
文本使用官方 DeepSeek-V2-Lite tokenizer，加入 special tokens，从右侧截到 96
|
||||
tokens。最低长度过滤为：
|
||||
|
||||
| 域 | 最低 token | 文本过滤后的候选 | 长度合格 | 选中 | 选中 token |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 英文百科 | 24 | 1,841 | 1,655 | 32 | 2,882 |
|
||||
| 中文新闻 | 8 | 10,000 | 9,415 | 32 | 573 |
|
||||
| Python 代码 | 24 | 164 | 164 | 32 | 2,983 |
|
||||
| 小学数学 | 16 | 1,319 | 1,319 | 32 | 2,022 |
|
||||
|
||||
英文与代码经常触到 96-token 截止线；中文新闻标题明显更短。因此后续不能只给
|
||||
一份“把所有 token 倒进桶里”的汇总。
|
||||
|
||||
## 3. 模型与执行边界
|
||||
|
||||
### 3.1 官方工件
|
||||
|
||||
```text
|
||||
deepseek-ai/DeepSeek-V2-Lite
|
||||
revision 604d5664dddd88a0433dbae533b7fe9472482de0
|
||||
```
|
||||
|
||||
关键 SHA-256：
|
||||
|
||||
| 工件 | SHA-256 |
|
||||
| --- | --- |
|
||||
| `config.json` | `f346286b0f1c8b044252fd54cb4fa78b9fab6472a6e8bebb9edfe03d414ea03d` |
|
||||
| `modeling_deepseek.py` | `7d8e5221095286eea991137760893fd7ba52727c0b4ebf48ec09e8bc56b45b9c` |
|
||||
| `tokenizer.json` | `41f3bf64213da8c012d8bd0871a58a1fdf70463e8f08f110ddbb1082f529f669` |
|
||||
| checkpoint index | `d2cdb2f325f6682cf3ad1ad2526a9f979d857390b579380c0331d975136e0acf` |
|
||||
| shard 1 | `0d7e9f39bde40111a4c0f390b87497dce4565cf578d916395e6b2c7851f1e8da` |
|
||||
|
||||
第一分片完整包含 embedding 与 decoder layer 0–6；layer 7 跨分片，因此继续遵守
|
||||
原有停止线。Layer 0 是 dense FFN，layer 1–6 是本次统计的六个 MoE 层。
|
||||
|
||||
### 3.2 批处理不会把 padding 算进统计
|
||||
|
||||
128 条样本按 token 数、domain、source ID 稳定排序，batch size 为 16。每个 batch
|
||||
右侧 padding，并使用官方 eager attention 与 causal/padding mask。
|
||||
|
||||
gate hook 会看到 padded positions 的输出，但统计时只切：
|
||||
|
||||
```text
|
||||
topk_ids[sample, :valid_length, :]
|
||||
```
|
||||
|
||||
所以每条 prompt 的 route 总数必须严格等于：
|
||||
|
||||
```text
|
||||
valid_tokens × 6
|
||||
```
|
||||
|
||||
六层均通过这个不变量检查；每层都是 50,760 routes。
|
||||
|
||||
### 3.3 未执行的对象
|
||||
|
||||
- 未下载或执行 layer 7–26；
|
||||
- 未做完整模型生成；
|
||||
- 未测回答正确率；
|
||||
- 未测训练期间 router load；
|
||||
- 未测线上 serving 流量；
|
||||
- 未执行 FlashMLA 优化 kernel；
|
||||
- 未给 expert ID 赋语义名称。
|
||||
|
||||
## 4. 为什么 bootstrap 的单位必须是 prompt
|
||||
|
||||
一个 prompt 中相邻 token：
|
||||
|
||||
- 共享主题、语言与格式；
|
||||
- 共享相同前缀 hidden state；
|
||||
- 在 causal attention 下相互依赖；
|
||||
- 不是独立同分布样本。
|
||||
|
||||
如果把 8,460 个 token 当成 8,460 个独立样本，区间会虚假变窄。这里先为每个
|
||||
prompt 保存 64 维 route-count vector，再在每个 domain 的 32 条 prompt 内有放回
|
||||
抽取 32 条。四个 domain 分层进行，固定：
|
||||
|
||||
```text
|
||||
replicates = 2000
|
||||
seed = 20260729
|
||||
interval = percentile [2.5%, 97.5%]
|
||||
```
|
||||
|
||||
它回答的是：
|
||||
|
||||
> 如果仍从这个固定探针来源与选样合同附近换一批 prompt，统计量会怎样波动？
|
||||
|
||||
它不回答：
|
||||
|
||||
> 真实训练分布或线上流量的总体参数是什么？
|
||||
|
||||
## 5. 两种聚合口径
|
||||
|
||||
### 5.1 Token weighted
|
||||
|
||||
先汇总一个 domain 内所有 prompt 的 route counts，再归一：
|
||||
|
||||
```text
|
||||
p_e = Σ_i count(i,e) / Σ_i Σ_e count(i,e)
|
||||
```
|
||||
|
||||
长 prompt 贡献更多，适合回答“本批实际 token 流向哪里”。
|
||||
|
||||
### 5.2 Prompt balanced
|
||||
|
||||
先把每条 prompt 的 64 维分布归一，再让 32 条 prompt 等权：
|
||||
|
||||
```text
|
||||
p_e = mean_i [count(i,e) / Σ_e count(i,e)]
|
||||
```
|
||||
|
||||
适合跨格式比较，因为一条短新闻标题与一条被截到 96 token 的代码 prompt 各有一票。
|
||||
网站默认展示这一口径，并允许切换。
|
||||
|
||||
在本次四域内部，CV 的两种口径最大绝对差为 0.027（Layer 1 数学）。这说明主要
|
||||
层级图景不是某一条特别长 prompt 单独制造的；但两套数字仍完整保留，不能据此假设
|
||||
未来语料也不敏感。
|
||||
|
||||
## 6. 指标是什么
|
||||
|
||||
对当前层、当前 domain 的 64 维 expert share `p`：
|
||||
|
||||
### CV
|
||||
|
||||
```text
|
||||
CV = population_std(p) / mean(p)
|
||||
```
|
||||
|
||||
越大表示 share 越分散。它不是训练损失，也不直接等于“不均衡惩罚”。
|
||||
|
||||
### Gini
|
||||
|
||||
64 个 share 的 Gini coefficient。0 表示完全均匀；越大表示质量集中到更少 expert。
|
||||
|
||||
### Effective experts
|
||||
|
||||
```text
|
||||
H(p) = -Σ p_e log p_e
|
||||
effective = exp(H)
|
||||
```
|
||||
|
||||
完全均匀时为 64；越小表示“按熵折算”的有效 expert 数越少。
|
||||
|
||||
### Top-expert share
|
||||
|
||||
64 个 share 的最大值。完全均匀基线为 `1/64 = 1.5625%`。
|
||||
|
||||
### Jensen–Shannon divergence
|
||||
|
||||
同层两个 domain 分布 `P,Q`：
|
||||
|
||||
```text
|
||||
M = (P + Q) / 2
|
||||
JSD(P,Q) = 1/2 KL(P||M) + 1/2 KL(Q||M)
|
||||
```
|
||||
|
||||
使用自然对数，范围 `[0, ln(2)]`。0 表示两条 64 维分布完全相同。
|
||||
|
||||
## 7. 主要结果
|
||||
|
||||
以下均为 **prompt-balanced** 点估计；括号内是 effective experts。网站可切换到
|
||||
token-weighted 并查看完整 95% 区间。
|
||||
|
||||
| layer | 英文百科 CV（effective） | 中文新闻 CV（effective） | 代码 CV（effective） | 数学 CV（effective） |
|
||||
| ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 0.439（59.0） | 0.754（53.3） | 0.566（55.4） | 0.613（56.3） |
|
||||
| 2 | 0.363（60.2） | 0.460（58.4） | 0.400（59.1） | 0.403（59.0） |
|
||||
| 3 | 0.398（59.2） | 0.469（57.9） | 0.440（57.9） | 0.525（57.0） |
|
||||
| 4 | 0.424（58.6） | **0.910（47.2）** | 0.610（54.0） | 0.527（56.0） |
|
||||
| 5 | 0.418（58.5） | 0.551（55.0） | 0.524（56.0） | **0.848（49.7）** |
|
||||
| 6 | 0.496（56.7） | 0.594（53.1） | 0.556（54.7） | **0.770（50.7）** |
|
||||
|
||||
### 7.1 Layer 2 在四域都相对平
|
||||
|
||||
Layer 2 的 prompt-balanced CV 为 0.363–0.460，effective experts 为
|
||||
58.4–60.2，是六个被测 MoE 层中四域共同最平的一段。
|
||||
|
||||
“相对平”只指这 64 维 route-count distribution；不等于 expert 权重相同、计算量
|
||||
完全相同或训练全局均衡。
|
||||
|
||||
### 7.2 Layer 4 的中文新闻更集中
|
||||
|
||||
Layer 4 中文新闻：
|
||||
|
||||
```text
|
||||
CV = 0.910, 95% CI [0.857, 0.993]
|
||||
effective experts = 47.2, 95% CI [44.4, 48.5]
|
||||
top share = E29 9.06%, 95% CI [8.41%, 9.78%]
|
||||
```
|
||||
|
||||
同层英文百科 CV 为 0.424、代码为 0.610、数学为 0.527。正确表述是：
|
||||
|
||||
> 在这 32 条 TNEWS 句子、这个 tokenizer、这个 layer 的固定探针中，route counts
|
||||
> 比另外三域更集中。
|
||||
|
||||
错误表述是：
|
||||
|
||||
> E29 是“中文专家”。
|
||||
|
||||
### 7.3 Layer 5/6 的数学更集中
|
||||
|
||||
Layer 5 数学：
|
||||
|
||||
```text
|
||||
CV = 0.848, 95% CI [0.794, 0.911]
|
||||
effective experts = 49.7, 95% CI [48.1, 50.7]
|
||||
top share = E14 8.04%, 95% CI [7.38%, 8.64%]
|
||||
```
|
||||
|
||||
Layer 6 数学：
|
||||
|
||||
```text
|
||||
CV = 0.770, 95% CI [0.728, 0.826]
|
||||
effective experts = 50.7, 95% CI [49.3, 51.5]
|
||||
top share = E28 7.54%, 95% CI [6.79%, 8.26%]
|
||||
```
|
||||
|
||||
E14 与 E28 位于不同层，是不同参数；不能连成一个跨层“数学专家轨迹”。
|
||||
|
||||
### 7.4 同层域间 JSD
|
||||
|
||||
每层最大的 prompt-balanced pair：
|
||||
|
||||
| layer | 最大 pair | JSD point | 95% bootstrap interval |
|
||||
| ---: | --- | ---: | ---: |
|
||||
| 1 | 中文新闻 ↔ 代码 | 0.0587 | [0.0549, 0.0703] |
|
||||
| 2 | 中文新闻 ↔ 代码 | 0.0345 | [0.0306, 0.0466] |
|
||||
| 3 | 中文新闻 ↔ 代码 | 0.0569 | [0.0519, 0.0713] |
|
||||
| 4 | **中文新闻 ↔ 代码** | **0.1502** | **[0.1376, 0.1715]** |
|
||||
| 5 | 中文新闻 ↔ 数学 | 0.1113 | [0.0990, 0.1368] |
|
||||
| 6 | 中文新闻 ↔ 代码 | 0.1161 | [0.1054, 0.1351] |
|
||||
|
||||
Layer 4 的中文新闻↔代码是本探针最大的同层距离，但仍只有理论上界
|
||||
`ln(2)≈0.693` 的约 21.7%。
|
||||
|
||||
必须特别注意：普通 percentile bootstrap 是围绕经验分布重采样，不是“两个总体完全
|
||||
相同”的 null bootstrap。区间不含 0 不能被偷换成经过校正的显著性检验。
|
||||
|
||||
## 8. 可复现性
|
||||
|
||||
正式运行和独立重跑的完整 JSON：
|
||||
|
||||
```text
|
||||
4678a1d15395de93ffba757598cc3642bf35e9e07f71d82ddd87c27fc38a09e4
|
||||
```
|
||||
|
||||
两个文件 byte-for-byte identical。确定性范围包括：
|
||||
|
||||
- 128 条选样身份、token 数与原文 SHA-256；
|
||||
- 六层、每条 prompt 的 64 维 integer route counts；
|
||||
- 两种聚合口径；
|
||||
- 2,000 次 bootstrap 导出的所有 metric / expert-share / JSD 区间；
|
||||
- 模型、语料、环境与统计合同。
|
||||
|
||||
没有记录 wall-clock timing，因此复跑一致性没有把 GPU warm-up、频率或系统噪声混进
|
||||
证据闸门。
|
||||
|
||||
### 8.1 运行入口
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||||
python -B experiments/deepseek/v2_lite_routing_corpus.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-corpus.json \
|
||||
--per-domain 32 \
|
||||
--max-tokens 96 \
|
||||
--batch-size 16 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T07:45:00+00:00
|
||||
```
|
||||
|
||||
官方 2024 remote code 在本机 Transformers 5.5 上会因已移除的
|
||||
`is_torch_fx_available` 失败；与前两轮相同，本实验使用隔离的 Transformers 4.41.2
|
||||
依赖，并把官方模型代码作为只读本地 package 导入，不给源码打补丁。
|
||||
|
||||
## 9. 仍然不能说什么
|
||||
|
||||
本实验没有授权以下结论：
|
||||
|
||||
- “DeepSeek 训练时专家全局负载就是这样”；
|
||||
- “线上中文请求一定路由到 E29”；
|
||||
- “E48 是代码专家、E14 是数学专家”；
|
||||
- “某个 domain 的任务性能更好”；
|
||||
- “router 的分布差异来自语义，而不是长度、tokenization、格式或共享前缀”；
|
||||
- “六个早期 MoE 层代表完整 27 层”；
|
||||
- “bootstrap CI 已经完成假设检验或多重比较控制”。
|
||||
|
||||
更稳妥的结论只有：
|
||||
|
||||
> 在固定、可重建的 128 条公开 prompt 上，DeepSeek-V2-Lite 的六个早期 MoE 层
|
||||
> 确实呈现可测量的层间与域间 route-count 差异；prompt-level bootstrap 让我们能同时
|
||||
> 看见差异和换 prompt 时的波动，但不能把参数索引直接解释成语义角色。
|
||||
|
||||
## 10. 下一步
|
||||
|
||||
1. 在受支持的 SM90 / SM100 环境执行 FlashMLA 优化 kernel，而不只运行算法参考路径；
|
||||
2. 扩展到完整 layer 0–26，需要下载并校验其余 checkpoint shards；
|
||||
3. 加入长度匹配与 tokenizer-fragmentation 对照，拆开 domain 和长度因素；
|
||||
4. 预注册更少的主要比较，或使用 permutation / null bootstrap 与多重比较校正；
|
||||
5. 研究 shared expert 与 routed expert 的激活规模，而不只看 routed IDs；
|
||||
6. 若讨论性能，另建生成质量与吞吐合同，不与本路由描述实验混在一起。
|
||||
@@ -299,6 +299,35 @@ await evaluate(`(() => {
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-absorb-desktop.png");
|
||||
|
||||
const artifactCorpus = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-dsv2-lab]");
|
||||
root.querySelector('[data-artifact-tab="corpus"]').click();
|
||||
const read = () => ({
|
||||
panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
|
||||
rows: root.querySelectorAll("[data-corpus-domain-rows] > div").length,
|
||||
heatRows: root.querySelectorAll("[data-corpus-heatmap] > div").length,
|
||||
heatCells: root.querySelectorAll("[data-corpus-heatmap] > div > span").length,
|
||||
jsdCells: root.querySelectorAll("[data-corpus-jsd] > *").length,
|
||||
highest: root.querySelector("[data-corpus-highest-cv]").textContent.trim(),
|
||||
largest: root.querySelector("[data-corpus-largest-jsd]").textContent.trim(),
|
||||
heatTitle: root.querySelector("[data-corpus-heat-title]").textContent.trim(),
|
||||
exact: root.querySelector(".corpus-ledger .exact b").textContent.trim(),
|
||||
modeNote: root.querySelector("[data-corpus-mode-note]").textContent.trim(),
|
||||
});
|
||||
const layer1 = read();
|
||||
root.querySelector('[data-corpus-layer="4"]').click();
|
||||
const layer4 = read();
|
||||
root.querySelector('[data-corpus-mode="token_weighted"]').click();
|
||||
const tokenWeighted = read();
|
||||
return { layer1, layer4, tokenWeighted };
|
||||
})()`);
|
||||
await evaluate(`(() => {
|
||||
document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
|
||||
window.scrollBy(0, -82);
|
||||
})()`);
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-corpus-desktop.png");
|
||||
|
||||
const artifactEvidence = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-dsv2-lab]");
|
||||
root.querySelector('[data-artifact-tab="evidence"]').click();
|
||||
@@ -385,14 +414,14 @@ const mobile = await evaluate(`(() => {
|
||||
await evaluate(`(() => {
|
||||
document.querySelector("#menu-toggle")?.click();
|
||||
const artifact = document.querySelector("[data-dsv2-lab]");
|
||||
artifact.querySelector('[data-artifact-tab="absorb"]').click();
|
||||
artifact.querySelector('[data-artifact-tab="corpus"]').click();
|
||||
artifact.scrollIntoView({ block: "start", behavior: "instant" });
|
||||
window.scrollBy(0, -82);
|
||||
})()`);
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-artifact-mobile.png");
|
||||
await screenshot("/tmp/llm-atlas-deepseek-corpus-mobile.png");
|
||||
|
||||
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactEvidence, home, papers, mobile, exceptions };
|
||||
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactCorpus, artifactEvidence, home, papers, mobile, exceptions };
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
|
||||
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
||||
@@ -402,7 +431,7 @@ if (overview.sections !== 26 || overview.tocLinks !== 26) failures.push("二十
|
||||
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
|
||||
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.artifactTabs !== 5 || overview.artifactPanels !== 5 || overview.artifactLayers !== 27) failures.push("真实权重五联实验结构异常");
|
||||
if (overview.artifactTabs !== 6 || overview.artifactPanels !== 6 || overview.artifactLayers !== 27) failures.push("真实权重六联实验结构异常");
|
||||
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
||||
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
|
||||
@@ -429,12 +458,16 @@ if (!artifactCache.million.latent.includes("GiB") || !artifactCache.million.eage
|
||||
if (artifactAbsorb.panel !== "absorb" || artifactAbsorb.algebra !== 2 || artifactAbsorb.naive !== "260.00 KiB" || artifactAbsorb.absorbed !== "29.25 KiB") failures.push("真实 absorb 缓存执行账异常");
|
||||
if (artifactAbsorb.metrics[0] !== "8.8889×" || artifactAbsorb.metrics[1] !== "0.00390625" || artifactAbsorb.metrics[2] !== "1.19e-7" || artifactAbsorb.metrics[3] !== "BYTE-EXACT") failures.push("absorb 数值正确性或复跑闸门异常");
|
||||
if (artifactAbsorb.precisionRows !== 3 || artifactAbsorb.matrixRows !== 5 || artifactAbsorb.localUnsupported !== 4 || artifactAbsorb.executionCards !== 3 || !artifactAbsorb.boundary.includes("不是 FlashMLA 性能")) failures.push("FlashMLA SM120 边界结构异常");
|
||||
if (artifactCorpus.layer1.panel !== "corpus" || artifactCorpus.layer1.rows !== 4 || artifactCorpus.layer1.heatRows !== 4 || artifactCorpus.layer1.heatCells !== 256 || artifactCorpus.layer1.jsdCells !== 25) failures.push("128 样本路由区间结构异常");
|
||||
if (!artifactCorpus.layer1.highest.includes("中文新闻 · 0.754") || !artifactCorpus.layer1.largest.includes("中文新闻 ↔ Python 代码 · 0.059") || artifactCorpus.layer1.exact !== "BYTE-EXACT") failures.push("Layer 1 多域统计或复跑闸门异常");
|
||||
if (!artifactCorpus.layer4.highest.includes("中文新闻 · 0.910") || !artifactCorpus.layer4.largest.includes("中文新闻 ↔ Python 代码 · 0.150") || !artifactCorpus.layer4.heatTitle.includes("layer 4")) failures.push("Layer 4 多域路由切换异常");
|
||||
if (!artifactCorpus.tokenWeighted.heatTitle.includes("按 token 加权") || !artifactCorpus.tokenWeighted.modeNote.includes("长 prompt 权重更高")) failures.push("公开语料聚合口径切换异常");
|
||||
if (artifactEvidence.panel !== "evidence" || artifactEvidence.layers !== 27 || artifactEvidence.executed !== 7 || artifactEvidence.split !== 1 || artifactEvidence.unloaded !== 19 || artifactEvidence.exact !== "31 / 31") failures.push("真实工件执行边界或复跑闸门异常");
|
||||
if (!artifactEvidence.dependency.includes("Transformers 5.5") || !artifactEvidence.dependency.includes("4.41.2") || !artifactEvidence.boundary.includes("完整 27 层生成")) failures.push("依赖版本或未覆盖边界异常");
|
||||
if (artifactEvidence.keyboardSelected !== "load" || artifactEvidence.keyboardVisible !== "load") failures.push("真实工件实验键盘 tab 导航异常");
|
||||
if (home.releaseCards !== 17 || !home.firstRelease.includes("47 页不再压成摘要") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 DeepSeek 首发入口或论文数异常");
|
||||
if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 5 || mobile.artifactHeatCells !== 64) failures.push("移动端导航或实验异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 6 || mobile.artifactHeatCells !== 64) failures.push("移动端导航或实验异常");
|
||||
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
|
||||
@@ -3,12 +3,17 @@ import rawTrace from "@/data/deepseek-v2-lite-trace.json";
|
||||
import rawRepro from "@/data/deepseek-v2-lite-trace-repro.json";
|
||||
import rawAbsorb from "@/data/deepseek-v2-lite-absorb.json";
|
||||
import rawAbsorbRepro from "@/data/deepseek-v2-lite-absorb-repro.json";
|
||||
import rawCorpus from "@/data/deepseek-v2-lite-routing-corpus.json";
|
||||
import rawCorpusRepro from "@/data/deepseek-v2-lite-routing-corpus-repro.json";
|
||||
|
||||
const trace = rawTrace as any;
|
||||
const repro = rawRepro as any;
|
||||
const absorb = rawAbsorb as any;
|
||||
const absorbRepro = rawAbsorbRepro as any;
|
||||
const corpus = rawCorpus as any;
|
||||
const corpusRepro = rawCorpusRepro as any;
|
||||
const absorbExact = JSON.stringify(absorb) === JSON.stringify(absorbRepro);
|
||||
const corpusExact = JSON.stringify(corpus) === JSON.stringify(corpusRepro);
|
||||
const bytes = (value: number) => value >= 1024
|
||||
? `${(value / 1024).toFixed(2)} KiB`
|
||||
: `${value.toLocaleString()} B`;
|
||||
@@ -44,6 +49,19 @@ const compact = {
|
||||
},
|
||||
};
|
||||
const compactJson = JSON.stringify(compact).replaceAll("<", "\\u003c");
|
||||
const corpusCompact = {
|
||||
domains: corpus.corpus_contract.domains,
|
||||
labels: corpus.corpus_contract.domain_labels,
|
||||
counts: corpus.corpus_contract.counts,
|
||||
inference: corpus.inference_contract,
|
||||
statistics: corpus.statistical_contract,
|
||||
layers: corpus.layers.slice(1).map((layer: any) => ({
|
||||
layer: layer.layer,
|
||||
routes: layer.routes,
|
||||
modes: layer.statistics,
|
||||
})),
|
||||
};
|
||||
const corpusCompactJson = JSON.stringify(corpusCompact).replaceAll("<", "\\u003c");
|
||||
const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoint_tensor_bytes;
|
||||
---
|
||||
|
||||
@@ -55,7 +73,8 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
</div>
|
||||
<p>
|
||||
固定官方 revision、tokenizer、模型代码和 BF16 第一分片；RTX 5090 连续执行 layer 0–6，
|
||||
捕获 3,240 次真实路由，并让 layer-1 权重继续走入官方吸收式 cache。所有结论都带证据身份与停止线。
|
||||
从 3,240 次 token 显微轨迹扩到 304,560 次公开语料路由，并让 layer-1 权重继续走入官方吸收式 cache。
|
||||
所有结论都带证据身份与停止线。
|
||||
</p>
|
||||
</header>
|
||||
|
||||
@@ -79,8 +98,11 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
<button type="button" role="tab" data-artifact-tab="absorb" aria-selected="false" tabindex="-1">
|
||||
<span>04</span><b>吸收式执行</b><small>real cache · SM120</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-artifact-tab="corpus" aria-selected="false" tabindex="-1">
|
||||
<span>05</span><b>128 样本区间</b><small>4 domains · bootstrap</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-artifact-tab="evidence" aria-selected="false" tabindex="-1">
|
||||
<span>05</span><b>证据断面</b><small>revision · shards · rerun</small>
|
||||
<span>06</span><b>证据断面</b><small>revision · shards · rerun</small>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
@@ -366,6 +388,94 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="artifact-panel" data-artifact-panel="corpus" hidden>
|
||||
<div class="panel-lead">
|
||||
<div><span>X + S / FIXED PUBLIC CORPUS</span><h4>从四条示例，走到 128 条可重建样本与区间</h4></div>
|
||||
<p>
|
||||
WikiText-2、TNEWS、HumanEval、GSM8K 各取 32 条；固定哈希选样、最多 96 tokens。
|
||||
每个区间都重采样 prompt，而不是把同一 prompt 里的 token 假装成独立样本。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="corpus-ledger">
|
||||
<article><span>PROMPTS</span><b>{corpus.inference_contract.total_prompts}</b><p>4 domains × 32</p></article>
|
||||
<article><span>VALID TOKENS</span><b>{corpus.inference_contract.valid_tokens.toLocaleString()}</b><p>答案未使用，代码未执行</p></article>
|
||||
<article><span>REAL ROUTES</span><b>{(corpus.inference_contract.routes_per_moe_layer * 6).toLocaleString()}</b><p>50,760 / layer × 6 MoE layers</p></article>
|
||||
<article><span>BOOTSTRAP</span><b>{corpus.statistical_contract.replicates.toLocaleString()}</b><p>prompt-level / domain-stratified</p></article>
|
||||
<article class="exact"><span>INDEPENDENT RERUN</span><b>{corpusExact ? "BYTE-EXACT" : "MISMATCH"}</b><p>SHA-256 4678a1d1…a09e4</p></article>
|
||||
</div>
|
||||
|
||||
<div class="corpus-controls">
|
||||
<div>
|
||||
<span>MOE LAYER</span>
|
||||
<div class="layer-switch corpus-layer-switch" role="group" aria-label="选择公开语料路由层">
|
||||
{[1, 2, 3, 4, 5, 6].map((layer) => (
|
||||
<button type="button" data-corpus-layer={layer} class={layer === 1 ? "active" : ""}>L{layer}</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<span>AGGREGATION</span>
|
||||
<div class="corpus-mode-switch" role="group" aria-label="选择公开语料聚合口径">
|
||||
<button type="button" data-corpus-mode="prompt_balanced" aria-pressed="true">每条 prompt 等权</button>
|
||||
<button type="button" data-corpus-mode="token_weighted" aria-pressed="false">按 token 加权</button>
|
||||
</div>
|
||||
</div>
|
||||
<p data-corpus-mode-note>
|
||||
先把每条 prompt 的 64 维路由分布归一，再平均；短中文标题与长代码 prompt 各有一票。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="corpus-domain-table" role="table" aria-label="四域专家负载与 bootstrap 区间">
|
||||
<div class="head" role="row">
|
||||
<b role="columnheader">DOMAIN / TOKENS</b>
|
||||
<b role="columnheader">CV · 95% CI</b>
|
||||
<b role="columnheader">EFFECTIVE · 95% CI</b>
|
||||
<b role="columnheader">TOP EXPERT · SHARE</b>
|
||||
</div>
|
||||
<div data-corpus-domain-rows></div>
|
||||
</div>
|
||||
|
||||
<div class="corpus-heat-head">
|
||||
<div><span>64 EXPERT SHARES</span><b data-corpus-heat-title>layer 1 · 每条 prompt 等权</b></div>
|
||||
<p>每一行独立着色；悬停查看 expert ID、份额与 95% 区间。同号 expert 只在当前层内有意义。</p>
|
||||
</div>
|
||||
<div class="corpus-heat-scroll">
|
||||
<div class="corpus-heatmap" data-corpus-heatmap aria-label="四个语料域的 64 专家份额热图"></div>
|
||||
</div>
|
||||
|
||||
<div class="corpus-comparison">
|
||||
<div>
|
||||
<span>PAIRWISE DISTANCE / JENSEN–SHANNON · NATS</span>
|
||||
<h5>同层四域分布距离</h5>
|
||||
<p>0 表示两条分布相同，理论上界 ln(2)≈0.693；这里展示点估计与 prompt bootstrap 区间。</p>
|
||||
</div>
|
||||
<div class="corpus-jsd" data-corpus-jsd role="table" aria-label="四域 Jensen-Shannon divergence 矩阵"></div>
|
||||
</div>
|
||||
|
||||
<div class="corpus-findings">
|
||||
<article><span>CURRENT HIGHEST CV</span><b data-corpus-highest-cv></b><p data-corpus-highest-cv-ci></p></article>
|
||||
<article><span>CURRENT LARGEST JSD</span><b data-corpus-largest-jsd></b><p data-corpus-largest-jsd-ci></p></article>
|
||||
<article><span>WHAT CHANGED</span><b>区间替代单点印象</b><p>看到“不同”之后，继续问 prompt 换一批时波动多大。</p></article>
|
||||
</div>
|
||||
|
||||
<div class="evidence-links">
|
||||
<a href="https://huggingface.co/datasets/Salesforce/wikitext" rel="noreferrer">WikiText-2 数据卡 ↗</a>
|
||||
<a href="https://github.com/CLUEbenchmark/CLUE" rel="noreferrer">CLUE / TNEWS ↗</a>
|
||||
<a href="https://github.com/openai/human-eval" rel="noreferrer">OpenAI HumanEval ↗</a>
|
||||
<a href="https://github.com/openai/grade-school-math" rel="noreferrer">OpenAI GSM8K ↗</a>
|
||||
</div>
|
||||
|
||||
<div class="artifact-boundary">
|
||||
<b>DESCRIPTIVE, NOT SEMANTICS</b>
|
||||
<p>
|
||||
这是固定 128 条公开 prompt 上的前六个 MoE 层，不是训练分布或线上流量。
|
||||
bootstrap 区间描述本探针换 prompt 的稳定性，不是零差异假设检验；没有多重比较校正，
|
||||
也不能把 E29、E48 等参数索引命名成“中文专家”或“代码专家”。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="artifact-panel" data-artifact-panel="evidence" hidden>
|
||||
<div class="panel-lead">
|
||||
<div><span>O + X / EVIDENCE SLICE</span><h4>为什么执行到 layer 6 就停，而不是把“部分下载”写成“完整复现”</h4></div>
|
||||
@@ -455,11 +565,12 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
<span>可复现入口</span>
|
||||
<code>experiments/deepseek/v2_lite_trace.py</code> ·
|
||||
<code>experiments/deepseek/v2_lite_absorb_probe.py</code> ·
|
||||
<code>src/data/deepseek-v2-lite-trace.json</code> ·
|
||||
<code>research/DEEPSEEK_MLA_ABSORB_AUDIT.md</code>
|
||||
<code>experiments/deepseek/v2_lite_routing_corpus.py</code> ·
|
||||
<code>research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md</code>
|
||||
</figcaption>
|
||||
|
||||
<script is:inline type="application/json" data-dsv2-trace set:html={compactJson}></script>
|
||||
<script is:inline type="application/json" data-dsv2-corpus set:html={corpusCompactJson}></script>
|
||||
</figure>
|
||||
|
||||
<script>
|
||||
@@ -471,8 +582,10 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
if (node) node.textContent = value;
|
||||
};
|
||||
const payloadNode = one<HTMLScriptElement>("[data-dsv2-trace]");
|
||||
if (!payloadNode?.textContent) return;
|
||||
const corpusNode = one<HTMLScriptElement>("[data-dsv2-corpus]");
|
||||
if (!payloadNode?.textContent || !corpusNode?.textContent) return;
|
||||
const data = JSON.parse(payloadNode.textContent);
|
||||
const corpusData = JSON.parse(corpusNode.textContent);
|
||||
|
||||
const tabs = all<HTMLButtonElement>("[data-artifact-tab]");
|
||||
const panels = all<HTMLElement>("[data-artifact-panel]");
|
||||
@@ -659,6 +772,173 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
control.addEventListener("change", renderCache);
|
||||
});
|
||||
renderCache();
|
||||
|
||||
const corpusLabels: Record<string, string> = {
|
||||
english: "英文百科",
|
||||
chinese: "中文新闻",
|
||||
code: "Python 代码",
|
||||
math: "小学数学",
|
||||
};
|
||||
const corpusLayer = (layer: number) => corpusData.layers.find((item: any) => item.layer === layer);
|
||||
let corpusLayerNumber = 1;
|
||||
let corpusMode = "prompt_balanced";
|
||||
const formatCi = (ci: number[], digits = 3) => `[${ci[0].toFixed(digits)}, ${ci[1].toFixed(digits)}]`;
|
||||
const renderCorpus = () => {
|
||||
const layer = corpusLayer(corpusLayerNumber);
|
||||
const mode = layer.modes[corpusMode];
|
||||
all<HTMLButtonElement>("[data-corpus-layer]").forEach((button) => {
|
||||
button.classList.toggle("active", Number(button.dataset.corpusLayer) === corpusLayerNumber);
|
||||
});
|
||||
all<HTMLButtonElement>("[data-corpus-mode]").forEach((button) => {
|
||||
button.setAttribute("aria-pressed", String(button.dataset.corpusMode === corpusMode));
|
||||
});
|
||||
set(
|
||||
"[data-corpus-mode-note]",
|
||||
corpusMode === "prompt_balanced"
|
||||
? "先把每条 prompt 的 64 维路由分布归一，再平均；短中文标题与长代码 prompt 各有一票。"
|
||||
: "直接汇总全部 token 的路由计数；长 prompt 权重更高，适合回答“本批 token 实际流向哪里”。",
|
||||
);
|
||||
set(
|
||||
"[data-corpus-heat-title]",
|
||||
`layer ${corpusLayerNumber} · ${corpusMode === "prompt_balanced" ? "每条 prompt 等权" : "按 token 加权"}`,
|
||||
);
|
||||
|
||||
const rows = one<HTMLElement>("[data-corpus-domain-rows]");
|
||||
if (rows) {
|
||||
rows.replaceChildren(...corpusData.domains.map((domain: string) => {
|
||||
const result = mode.domains[domain];
|
||||
const metrics = result.metrics;
|
||||
const top = result.top_experts[0];
|
||||
const row = document.createElement("div");
|
||||
row.setAttribute("role", "row");
|
||||
|
||||
const identity = document.createElement("span");
|
||||
const identityLabel = document.createElement("b");
|
||||
const identityMeta = document.createElement("small");
|
||||
identityLabel.textContent = corpusLabels[domain];
|
||||
identityMeta.textContent = `32 prompts · ${corpusData.counts[domain].valid_tokens.toLocaleString()} tokens`;
|
||||
identity.append(identityLabel, identityMeta);
|
||||
|
||||
const cv = document.createElement("span");
|
||||
const cvPoint = document.createElement("b");
|
||||
const cvCi = document.createElement("small");
|
||||
cvPoint.textContent = metrics.cv.point.toFixed(3);
|
||||
cvCi.textContent = formatCi(metrics.cv.ci95);
|
||||
cv.append(cvPoint, cvCi);
|
||||
|
||||
const effective = document.createElement("span");
|
||||
const effectivePoint = document.createElement("b");
|
||||
const effectiveCi = document.createElement("small");
|
||||
effectivePoint.textContent = metrics.effective_experts.point.toFixed(1);
|
||||
effectiveCi.textContent = formatCi(metrics.effective_experts.ci95, 1);
|
||||
effective.append(effectivePoint, effectiveCi);
|
||||
|
||||
const topExpert = document.createElement("span");
|
||||
const topLabel = document.createElement("b");
|
||||
const topCi = document.createElement("small");
|
||||
topLabel.textContent = `E${top.expert} · ${(top.share * 100).toFixed(2)}%`;
|
||||
topCi.textContent = `${(top.ci95[0] * 100).toFixed(2)}–${(top.ci95[1] * 100).toFixed(2)}%`;
|
||||
topExpert.append(topLabel, topCi);
|
||||
row.append(identity, cv, effective, topExpert);
|
||||
return row;
|
||||
}));
|
||||
}
|
||||
|
||||
const heatmap = one<HTMLElement>("[data-corpus-heatmap]");
|
||||
if (heatmap) {
|
||||
heatmap.replaceChildren(...corpusData.domains.map((domain: string) => {
|
||||
const result = mode.domains[domain];
|
||||
const max = Math.max(...result.distribution);
|
||||
const row = document.createElement("div");
|
||||
const label = document.createElement("b");
|
||||
label.textContent = corpusLabels[domain];
|
||||
row.append(label);
|
||||
result.distribution.forEach((share: number, expert: number) => {
|
||||
const cell = document.createElement("span");
|
||||
const ci = result.expert_share_ci95[expert];
|
||||
cell.style.setProperty("--share", String(share / max));
|
||||
cell.title = `${corpusLabels[domain]} · L${corpusLayerNumber} · E${expert} · ${(share * 100).toFixed(2)}% · 95% CI ${(ci[0] * 100).toFixed(2)}–${(ci[1] * 100).toFixed(2)}%`;
|
||||
cell.setAttribute("aria-label", cell.title);
|
||||
row.append(cell);
|
||||
});
|
||||
return row;
|
||||
}));
|
||||
}
|
||||
|
||||
const jsd = one<HTMLElement>("[data-corpus-jsd]");
|
||||
const pairFor = (left: string, right: string) => mode.pairs.find((pair: any) =>
|
||||
(pair.left === left && pair.right === right) || (pair.left === right && pair.right === left)
|
||||
);
|
||||
if (jsd) {
|
||||
const cells: HTMLElement[] = [];
|
||||
const corner = document.createElement("b");
|
||||
corner.textContent = "DOMAIN";
|
||||
cells.push(corner);
|
||||
corpusData.domains.forEach((domain: string) => {
|
||||
const header = document.createElement("b");
|
||||
header.textContent = corpusLabels[domain];
|
||||
cells.push(header);
|
||||
});
|
||||
corpusData.domains.forEach((left: string) => {
|
||||
const header = document.createElement("b");
|
||||
header.textContent = corpusLabels[left];
|
||||
cells.push(header);
|
||||
corpusData.domains.forEach((right: string) => {
|
||||
const cell = document.createElement("span");
|
||||
if (left === right) {
|
||||
cell.className = "diagonal";
|
||||
cell.textContent = "0";
|
||||
} else {
|
||||
const result = pairFor(left, right).js_divergence;
|
||||
const point = document.createElement("b");
|
||||
const ci = document.createElement("small");
|
||||
point.textContent = result.point.toFixed(3);
|
||||
ci.textContent = formatCi(result.ci95);
|
||||
cell.append(point, ci);
|
||||
}
|
||||
cells.push(cell);
|
||||
});
|
||||
});
|
||||
jsd.replaceChildren(...cells);
|
||||
}
|
||||
|
||||
const highest = corpusData.domains
|
||||
.map((domain: string) => ({ domain, value: mode.domains[domain].metrics.cv }))
|
||||
.sort((left: any, right: any) => right.value.point - left.value.point)[0];
|
||||
const largest = [...mode.pairs]
|
||||
.sort((left: any, right: any) =>
|
||||
right.js_divergence.point - left.js_divergence.point
|
||||
)[0];
|
||||
set(
|
||||
"[data-corpus-highest-cv]",
|
||||
`${corpusLabels[highest.domain]} · ${highest.value.point.toFixed(3)}`,
|
||||
);
|
||||
set(
|
||||
"[data-corpus-highest-cv-ci]",
|
||||
`95% CI ${formatCi(highest.value.ci95)} · L${corpusLayerNumber}`,
|
||||
);
|
||||
set(
|
||||
"[data-corpus-largest-jsd]",
|
||||
`${corpusLabels[largest.left]} ↔ ${corpusLabels[largest.right]} · ${largest.js_divergence.point.toFixed(3)}`,
|
||||
);
|
||||
set(
|
||||
"[data-corpus-largest-jsd-ci]",
|
||||
`95% CI ${formatCi(largest.js_divergence.ci95)} · nats`,
|
||||
);
|
||||
};
|
||||
all<HTMLButtonElement>("[data-corpus-layer]").forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
corpusLayerNumber = Number(button.dataset.corpusLayer);
|
||||
renderCorpus();
|
||||
});
|
||||
});
|
||||
all<HTMLButtonElement>("[data-corpus-mode]").forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
corpusMode = button.dataset.corpusMode ?? "prompt_balanced";
|
||||
renderCorpus();
|
||||
});
|
||||
});
|
||||
renderCorpus();
|
||||
});
|
||||
</script>
|
||||
|
||||
@@ -702,7 +982,12 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.execution-split span,
|
||||
.repro-gate span,
|
||||
.dependency-split span,
|
||||
.checksum-grid span {
|
||||
.checksum-grid span,
|
||||
.corpus-ledger span,
|
||||
.corpus-controls > div > span,
|
||||
.corpus-heat-head span,
|
||||
.corpus-comparison span,
|
||||
.corpus-findings span {
|
||||
margin: 0;
|
||||
color: var(--blue);
|
||||
font: 700 .69rem/1.3 var(--font-mono);
|
||||
@@ -756,7 +1041,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.artifact-status b { color: var(--ink); font-size: .72rem; }
|
||||
.artifact-tabs {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(5, 1fr);
|
||||
grid-template-columns: repeat(6, 1fr);
|
||||
background: var(--ink);
|
||||
}
|
||||
.artifact-tabs button {
|
||||
@@ -1052,6 +1337,212 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.jaccard-block :global([data-load-jaccard] article b) { font: 700 .62rem/1 var(--font-mono); text-align: right; }
|
||||
.load-lessons { grid-template-columns: repeat(3, 1fr); }
|
||||
.load-lessons b { display: block; margin-top: .35rem; font-size: .77rem; }
|
||||
.corpus-ledger {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(5, 1fr);
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
}
|
||||
.corpus-ledger article {
|
||||
padding: .85rem;
|
||||
border-right: 1px solid rgba(32,32,39,.12);
|
||||
}
|
||||
.corpus-ledger article:last-child { border-right: 0; }
|
||||
.corpus-ledger article.exact { background: rgba(57,120,110,.1); }
|
||||
.corpus-ledger b {
|
||||
display: block;
|
||||
margin-top: .4rem;
|
||||
font: 750 1.2rem/1 var(--font-display);
|
||||
}
|
||||
.corpus-ledger p {
|
||||
margin: .3rem 0 0;
|
||||
color: rgba(32,32,39,.56);
|
||||
font-size: .63rem;
|
||||
line-height: 1.4;
|
||||
}
|
||||
.corpus-controls {
|
||||
display: grid;
|
||||
grid-template-columns: auto 1.1fr minmax(15rem, 1.3fr);
|
||||
gap: 1rem;
|
||||
align-items: end;
|
||||
margin-top: .8rem;
|
||||
padding: .85rem;
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
background: #e8e2d7;
|
||||
}
|
||||
.corpus-controls > div { display: grid; gap: .45rem; }
|
||||
.corpus-controls .layer-switch { margin: 0; }
|
||||
.corpus-mode-switch { display: flex; }
|
||||
.corpus-mode-switch button {
|
||||
padding: .58rem .75rem;
|
||||
border: 1px solid rgba(32,32,39,.22);
|
||||
background: #fffdf8;
|
||||
color: var(--ink);
|
||||
font: 650 .67rem/1 var(--font-mono);
|
||||
cursor: pointer;
|
||||
}
|
||||
.corpus-mode-switch button + button { border-left: 0; }
|
||||
.corpus-mode-switch button[aria-pressed="true"] {
|
||||
border-color: var(--blue);
|
||||
background: var(--blue);
|
||||
color: white;
|
||||
}
|
||||
.corpus-controls > p {
|
||||
margin: 0;
|
||||
color: rgba(32,32,39,.62);
|
||||
font-size: .69rem;
|
||||
line-height: 1.5;
|
||||
}
|
||||
.corpus-domain-table {
|
||||
margin-top: .8rem;
|
||||
border: 1px solid rgba(32,32,39,.15);
|
||||
overflow-x: auto;
|
||||
background: #fffdf8;
|
||||
}
|
||||
.corpus-domain-table .head,
|
||||
.corpus-domain-table :global([data-corpus-domain-rows] > div) {
|
||||
display: grid;
|
||||
grid-template-columns: 1.25fr repeat(3, 1fr);
|
||||
min-width: 680px;
|
||||
}
|
||||
.corpus-domain-table .head { background: var(--ink); color: white; }
|
||||
.corpus-domain-table .head b,
|
||||
.corpus-domain-table :global([data-corpus-domain-rows] > div > span) {
|
||||
display: grid;
|
||||
gap: .25rem;
|
||||
padding: .7rem;
|
||||
border-right: 1px solid rgba(32,32,39,.1);
|
||||
border-bottom: 1px solid rgba(32,32,39,.1);
|
||||
}
|
||||
.corpus-domain-table .head b {
|
||||
font: 650 .62rem/1.2 var(--font-mono);
|
||||
border-color: rgba(255,255,255,.12);
|
||||
}
|
||||
.corpus-domain-table :global([data-corpus-domain-rows] b) {
|
||||
font: 720 .76rem/1.2 var(--font-mono);
|
||||
}
|
||||
.corpus-domain-table :global([data-corpus-domain-rows] small) {
|
||||
color: rgba(32,32,39,.52);
|
||||
font: .61rem/1.25 var(--font-mono);
|
||||
}
|
||||
.corpus-heat-head {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1.4fr;
|
||||
gap: 1rem;
|
||||
align-items: end;
|
||||
margin: 1rem 0 .55rem;
|
||||
}
|
||||
.corpus-heat-head > div { display: grid; gap: .25rem; }
|
||||
.corpus-heat-head b { font-size: .78rem; }
|
||||
.corpus-heat-head p {
|
||||
margin: 0;
|
||||
color: rgba(32,32,39,.58);
|
||||
font-size: .67rem;
|
||||
line-height: 1.45;
|
||||
text-align: right;
|
||||
}
|
||||
.corpus-heat-scroll {
|
||||
overflow-x: auto;
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
background: #ddd6ca;
|
||||
}
|
||||
.corpus-heatmap { min-width: 880px; }
|
||||
.corpus-heatmap > :global(div) {
|
||||
display: grid;
|
||||
grid-template-columns: 6.5rem repeat(64, minmax(8px, 1fr));
|
||||
gap: 2px;
|
||||
padding: 3px;
|
||||
border-bottom: 1px solid rgba(32,32,39,.12);
|
||||
}
|
||||
.corpus-heatmap > :global(div:last-child) { border-bottom: 0; }
|
||||
.corpus-heatmap > :global(div > b) {
|
||||
position: sticky;
|
||||
left: 0;
|
||||
z-index: 1;
|
||||
display: grid;
|
||||
align-items: center;
|
||||
padding: .45rem;
|
||||
background: #eee8de;
|
||||
font: 700 .61rem/1.1 var(--font-mono);
|
||||
}
|
||||
.corpus-heatmap > :global(div > span) {
|
||||
min-height: 1.55rem;
|
||||
background: color-mix(in srgb, var(--blue) calc(var(--share) * 88%), #f2ede4);
|
||||
cursor: help;
|
||||
}
|
||||
.corpus-heatmap > :global(div > span:hover) {
|
||||
outline: 2px solid var(--amber);
|
||||
z-index: 2;
|
||||
}
|
||||
.corpus-comparison {
|
||||
display: grid;
|
||||
grid-template-columns: .65fr 1.35fr;
|
||||
gap: 1rem;
|
||||
margin-top: .8rem;
|
||||
padding: 1rem;
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
background: #e8e2d7;
|
||||
}
|
||||
.corpus-comparison h5 {
|
||||
margin: .4rem 0;
|
||||
font: 720 1rem/1.15 var(--font-display);
|
||||
}
|
||||
.corpus-comparison p {
|
||||
margin: 0;
|
||||
color: rgba(32,32,39,.58);
|
||||
font-size: .68rem;
|
||||
line-height: 1.5;
|
||||
}
|
||||
.corpus-jsd {
|
||||
display: grid;
|
||||
grid-template-columns: 6rem repeat(4, minmax(5.5rem, 1fr));
|
||||
overflow-x: auto;
|
||||
border: 1px solid rgba(32,32,39,.13);
|
||||
background: #fffdf8;
|
||||
}
|
||||
.corpus-jsd > :global(b),
|
||||
.corpus-jsd > :global(span) {
|
||||
display: grid;
|
||||
align-content: center;
|
||||
gap: .2rem;
|
||||
min-height: 3.1rem;
|
||||
padding: .45rem;
|
||||
border-right: 1px solid rgba(32,32,39,.1);
|
||||
border-bottom: 1px solid rgba(32,32,39,.1);
|
||||
font: 650 .59rem/1.2 var(--font-mono);
|
||||
}
|
||||
.corpus-jsd > :global(b) { background: var(--ink); color: white; }
|
||||
.corpus-jsd > :global(span > b) { font-size: .72rem; }
|
||||
.corpus-jsd > :global(span > small) {
|
||||
color: rgba(32,32,39,.48);
|
||||
font-size: .52rem;
|
||||
}
|
||||
.corpus-jsd > :global(span.diagonal) {
|
||||
place-items: center;
|
||||
background: rgba(57,120,110,.08);
|
||||
color: var(--teal);
|
||||
}
|
||||
.corpus-findings {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
margin-top: .8rem;
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
}
|
||||
.corpus-findings article {
|
||||
padding: .85rem;
|
||||
border-right: 1px solid rgba(32,32,39,.12);
|
||||
}
|
||||
.corpus-findings article:last-child { border-right: 0; }
|
||||
.corpus-findings b {
|
||||
display: block;
|
||||
margin-top: .4rem;
|
||||
font-size: .78rem;
|
||||
}
|
||||
.corpus-findings p {
|
||||
margin: .3rem 0 0;
|
||||
color: rgba(32,32,39,.56);
|
||||
font-size: .65rem;
|
||||
line-height: 1.45;
|
||||
}
|
||||
.observed-cache {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr auto 1.25fr;
|
||||
@@ -1283,7 +1774,10 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.heat-head,
|
||||
.aggregate-card,
|
||||
.jaccard-block,
|
||||
.kernel-contract { grid-template-columns: 1fr; }
|
||||
.kernel-contract,
|
||||
.corpus-controls,
|
||||
.corpus-heat-head,
|
||||
.corpus-comparison { grid-template-columns: 1fr; }
|
||||
.artifact-status { grid-template-columns: 1fr 1fr; }
|
||||
.artifact-tabs { grid-template-columns: 1fr 1fr; }
|
||||
.route-controls { grid-template-columns: 1fr 1fr; }
|
||||
@@ -1294,6 +1788,8 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.route-metrics,
|
||||
.checksum-grid,
|
||||
.absorb-metrics { grid-template-columns: 1fr 1fr; }
|
||||
.corpus-ledger { grid-template-columns: repeat(3, 1fr); }
|
||||
.corpus-heat-head p { text-align: left; }
|
||||
.layer-evidence { grid-template-columns: repeat(9, 1fr); }
|
||||
.repro-gate { grid-template-columns: 1fr 1fr; }
|
||||
.repro-gate > p { grid-column: 1 / -1; padding: .8rem 0 0; border-left: 0; border-top: 1px solid rgba(255,255,255,.18); }
|
||||
@@ -1319,13 +1815,19 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.absorb-metrics,
|
||||
.artifact-identity,
|
||||
.checksum-grid,
|
||||
.cache-ledger { grid-template-columns: 1fr; }
|
||||
.cache-ledger,
|
||||
.corpus-ledger,
|
||||
.corpus-findings { grid-template-columns: 1fr; }
|
||||
.route-metrics article,
|
||||
.cache-ratio article,
|
||||
.load-lessons article,
|
||||
.absorb-metrics article,
|
||||
.artifact-identity article,
|
||||
.checksum-grid article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
|
||||
.corpus-ledger article,
|
||||
.corpus-findings article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
|
||||
.corpus-mode-switch { display: grid; grid-template-columns: 1fr; }
|
||||
.corpus-mode-switch button + button { border-left: 1px solid rgba(32,32,39,.22); border-top: 0; }
|
||||
.artifact-boundary { grid-template-columns: 1fr; }
|
||||
.load-dials { grid-template-columns: 1fr; }
|
||||
.observed-cache,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -37,7 +37,7 @@ const toc = [
|
||||
|
||||
<BaseLayout
|
||||
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
|
||||
description="用二十四张问题账、十次技术转向、九个交互实验、真实 V2-Lite 权重与吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||
description="用二十四张问题账、十次技术转向、十个交互实验、真实 V2-Lite 权重、公开语料路由区间与吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||
section="deepseek"
|
||||
>
|
||||
<header class="page-hero deepseek-hero">
|
||||
@@ -55,7 +55,7 @@ const toc = [
|
||||
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
||||
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
|
||||
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
|
||||
<div><dt>LABS</dt><dd>9 个可操作实验</dd></div>
|
||||
<div><dt>LABS</dt><dd>10 个可操作实验</dd></div>
|
||||
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
|
||||
<div><dt>STATUS</dt><dd>三轮 · 真实权重执行</dd></div>
|
||||
</dl>
|
||||
@@ -768,15 +768,15 @@ const toc = [
|
||||
<p class="eyebrow"><span>22</span> OFFICIAL WEIGHTS / EXECUTED</p>
|
||||
<h2>从“MLA 与 MoE 的概念”再往前一步：让官方 V2-Lite 权重真的跑起来</h2>
|
||||
<p class="lede">
|
||||
前面的四联实验负责建立公式与角色合同；下面的五联工件实验固定官方 revision、tokenizer、
|
||||
前面的四联实验负责建立公式与角色合同；下面的六联工件实验固定官方 revision、tokenizer、
|
||||
模型代码和 checkpoint 第一分片，在 RTX 5090 上连续执行 layer 0–6。它把真实观测、shape 推导、
|
||||
吸收式 latent cache、实现差距和未覆盖范围放在同一张证据图里。
|
||||
吸收式 latent cache、128 条公开语料的路由区间、实现差距和未覆盖范围放在同一张证据图里。
|
||||
</p>
|
||||
<div class="artifact-callout">
|
||||
<article><span>X / FORWARD</span><b>7 / 27 layers</b><p>1 个 dense 层 + 6 个 MoE 层；layer 7 因跨分片停止。</p></article>
|
||||
<article><span>X / ROUTES</span><b>3,240</b><p>90 个有效 token × 6 层 × top-6 routed experts。</p></article>
|
||||
<article><span>X / ROUTES</span><b>304,560</b><p>128 条公开 prompt、8,460 token、6 个 MoE 层的真实 top-6 选择。</p></article>
|
||||
<article><span>X / ABSORB CACHE</span><b>266,240 → 29,952 B</b><p>同一真实 layer-1 权重的 naive / absorb active buffers。</p></article>
|
||||
<article><span>X / RERUN</span><b>31 / 31 exact</b><p>hidden hashes、MLA shapes、loads 与全部 token routes。</p></article>
|
||||
<article><span>X / RERUN</span><b>BYTE-EXACT</b><p>固定选样、逐 prompt loads 与 2,000 次 bootstrap 摘要完整复跑。</p></article>
|
||||
</div>
|
||||
<DeepSeekArtifactLab />
|
||||
</section>
|
||||
|
||||
@@ -15,7 +15,7 @@ const workstreams = [
|
||||
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
|
||||
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
|
||||
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
|
||||
{ label: "DeepSeek 专题", value: 89, next: "SM90 FlashMLA 优化 kernel、更大样本负载、FP8/pipeline 与 R1-like RL 复现" },
|
||||
{ label: "DeepSeek 专题", value: 91, next: "SM90 FlashMLA kernel、完整 27 层、长度匹配对照、FP8/pipeline 与 R1-like RL 复现" },
|
||||
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
||||
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
||||
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
||||
@@ -50,7 +50,7 @@ const workstreams = [
|
||||
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
||||
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
||||
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 14:15 CST</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 15:25 CST</dd></div>
|
||||
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
@@ -97,12 +97,12 @@ const workstreams = [
|
||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||
<article><span>✓</span><h3>七十六个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与五联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>七十七个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与六联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
|
||||
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、3,240 次真实专家选择、MLA/eager cache 实现账、31/31 exact 复跑，以及官方 V3 absorb 的 576 元素真实缓存五联实验。</p></article>
|
||||
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、官方 V3 absorb 的 576 元素真实缓存，以及四域 128 条固定公开 prompt、304,560 次真实路由、2,000 次 prompt bootstrap 与 byte-exact 独立重跑的六联实验。</p></article>
|
||||
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
|
||||
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
|
||||
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
|
||||
@@ -134,7 +134,7 @@ const workstreams = [
|
||||
<div class="queue-table">
|
||||
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
||||
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
|
||||
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>SM90 FlashMLA 优化 kernel / 更大负载样本 → FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>SM90 FlashMLA kernel / 完整 27 层 / 长度匹配对照 → FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
|
||||
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
|
||||
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
|
||||
|
||||
Reference in New Issue
Block a user