feat: add DeepSeek equal-length history control
This commit is contained in:
wuyang
+12
-3
@@ -14,7 +14,7 @@
|
||||
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
||||
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
||||
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
||||
| DeepSeek 专题 | 三轮实证进行中 | 94% | SM90 FlashMLA kernel、完整 27 层、词元边界 / 距离 / EOS / 角色正交控制、FP8/pipeline 与 R1-like RL 复现 |
|
||||
| DeepSeek 专题 | 三轮实证进行中 | 95% | SM90 FlashMLA kernel、完整 27 层、EOS / 角色 / 多 filler / 内容正交控制、FP8/pipeline 与 R1-like RL 复现 |
|
||||
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
||||
@@ -41,7 +41,7 @@
|
||||
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十二联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等七十九个原创交互视图。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十三联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十个原创交互视图。
|
||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||
@@ -203,11 +203,17 @@
|
||||
- [x] 消息历史正式运行与独立复跑 SHA-256 均为 `5765fbf8…c1fb`，约 28 MiB JSON byte-exact；网站第八个真实工件页签联动 layer、scope、aggregation 与 system/few-shot/interaction depth map。
|
||||
- [x] 消息历史里程碑本地闸门通过：69 个 Astro 文件零诊断、21 个页面、1,151 个站内引用、12 个跨页锚点零失败，十六套真实 Chrome 回归全部通过，桌面与 390px 移动端无文档级横向溢出。
|
||||
- [x] DeepSeek 消息历史因子里程碑以源提交 `efe3ffc`、不可变镜像 `20260729T100729Z-efe3ffc` 发布；OCI digest `sha256:e2ea1c35…77e2f`，NAS / VPS / NPM / DNS / TLS / HTTP2 / gzip / 门户与十六套生产 Chrome 回归全通过；保留 `20260729T092426Z-b0acc8b` 回滚。
|
||||
- [x] 等长历史控制把 system 开/关与 none / repeated-token filler / fixed demo 组成 2×3；filler 与 demo 在两条 system 水平都逐条精确 +17 tokens，并保持相同角色、assistant EOS、目标位置与六格 batch。
|
||||
- [x] RTX 5090 执行 768 个输入变体、38,236 个输入 token，新增 1,376,496 次真实 top-6 路由，使公开语料累计达到 3,112,848 次；六格各有相同的 2,874 个精确对齐目标 token。
|
||||
- [x] 目标内容 system-edge TV 呈 none `.0738` → filler `.0378` → demo `.0188` 两级阶梯；none→filler 与 filler→demo 都在 24 / 24 个 layer×domain 下降，且 24 / 24 source-paired 区间完全低于零。
|
||||
- [x] filler 不冒充“无语义”或纯距离：`x` 是学习过的重复 token；CV edge 方向不统一，完整输入只有 20/24、23/24 点估计下降，主结论严格限定为精确目标内容 route-distribution TV。
|
||||
- [x] 六格正式运行与独立复跑 SHA-256 均为 `423a095d…e648e`，约 41 MiB JSON byte-exact；网站第九个真实工件页签联动 layer、scope、aggregation 与两个 TV 台阶 depth map。
|
||||
- [x] 等长历史控制本地闸门通过：69 个 Astro 文件零诊断、21 个页面、1,151 个站内引用、12 个跨页锚点零失败，十六套真实 Chrome 回归全部通过，9 页签桌面端与 390px 移动端均无文档级横向溢出。
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
||||
- [ ] DeepSeek 三轮下一闸门：在官方支持的 SM90 环境执行 FlashMLA 优化 kernel；扩到完整 27 层并继续拆分词元边界、历史距离、EOS、角色与示例内容，再推进 FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] DeepSeek 三轮下一闸门：在官方支持的 SM90 环境执行 FlashMLA 优化 kernel；扩到完整 27 层并继续拆分 EOS、角色、多 filler、示例内容与 batch shape，再推进 FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
||||
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
||||
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||
@@ -346,6 +352,9 @@
|
||||
| 2026-07-29 | one-shot 后的 system-edge TV 下降只命名为历史缓冲模式 | 24 / 24 格下降与逐 token set/Jaccard 同向；示例语义、距离、EOS、角色转换和固定文本尚未拆开，因此不写成 few-shot 因果语义 |
|
||||
| 2026-07-29 | 因子分布 effect 使用 half-L1 magnitude 而非普通 TV | 主效应和 interaction 是带符号 expert-share 向量；只有四条实际条件边继续使用标准 TV / JSD |
|
||||
| 2026-07-29 | DeepSeek 消息历史因子里程碑以 `20260729T100729Z-efe3ffc` 发布 | OCI digest `sha256:e2ea1c35…77e2f`；复用 NAS 12010→8080、NPM 31 / cert 41、门户 order 180；十六套生产 Chrome 回归通过，保留上一不可变镜像回滚 |
|
||||
| 2026-07-29 | repeated-token filler 与 demo 逐条精确等长 | 两者共享 user/assistant 角色、assistant EOS、目标位置与 batch shape；`x` 仍是学习过的 token，因此不命名为无语义或纯距离 |
|
||||
| 2026-07-29 | 等长历史的两个 TV 台阶分开报告 | none→filler 回答“历史结构是否足以复现缓冲”；filler→demo 回答固定协议字段后的文本替换；两者都不升级为能力或示例正确性 |
|
||||
| 2026-07-29 | batch contract 是 BF16 路由复现的一部分 | 跨实验 512/512 token-ID 合同 exact，但矩阵形状改变会让深层临界 gate hash 分化；正式结论只使用同一次六格 batch 内对比 |
|
||||
| 2026-07-29 | K3 二轮按 32 张对象账与完整报告顺序重建 | total/active、2.5×、KDA state、深度来源、专家路由、视觉目标、轨迹、缓存与评测协议不再压成一页组件摘要 |
|
||||
| 2026-07-29 | K3 原生视觉事实回到 §2.4 / §3.3 核验 | 删除“先冻结语言模型再解冻”旧表述；明确 MoonViT-V2 从头训练，视觉/文本从开始共同 NTP |
|
||||
| 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项；作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 |
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
||||
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
||||
以及 79 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
以及 80 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
||||
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
||||
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
||||
@@ -28,7 +28,7 @@
|
||||
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
||||
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
||||
DeepSeek 三轮专题以 24 张问题账、10 次技术转向、
|
||||
12 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
13 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
||||
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
|
||||
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
|
||||
@@ -43,14 +43,18 @@ naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两
|
||||
不能改写过去路由。随后又把 system 与 one-shot 组成固定 16 / 17-token 正交增量的
|
||||
2×2 实验，新增 865,440 次真实路由；在目标内容上，加入 one-shot 历史后 system-edge
|
||||
TV 在 24 / 24 个 layer×domain 中都下降，均值从 `0.073` 降至 `0.019`，但这一模式不被
|
||||
越界解释为示例语义或能力提升。当前累计 1,736,352 次公开语料路由。FlashMLA 的
|
||||
越界解释为示例语义或能力提升。最新一步再加入与原 one-shot 精确等长的重复词元 filler，
|
||||
新增 1,376,496 次真实路由：system-edge TV 呈 `0.0738→0.0378→0.0188`
|
||||
的 none→filler→demo 两级阶梯，两个台阶都在 24 / 24 格下降且 paired 区间完全低于零；
|
||||
但 filler 仍是学习过的 token，不能冒充纯距离因果。当前累计 3,112,848 次公开语料路由。FlashMLA 的
|
||||
SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
|
||||
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
|
||||
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
|
||||
[DEEPSEEK_ROUTING_CORPUS_AUDIT.md](./research/DEEPSEEK_ROUTING_CORPUS_AUDIT.md)、
|
||||
[DEEPSEEK_ROUTING_LENGTH_CONTROL_AUDIT.md](./research/DEEPSEEK_ROUTING_LENGTH_CONTROL_AUDIT.md) 与
|
||||
[DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md](./research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md)、
|
||||
[DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md)。
|
||||
[DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md)、
|
||||
[DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md)。
|
||||
其余专题按进度账本持续扩建。
|
||||
|
||||
## 本地开发
|
||||
|
||||
@@ -264,3 +264,58 @@ See `research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md` for the factorial
|
||||
definition, all CV intervals, the 24/24 system-edge TV attenuation pattern,
|
||||
per-token stability, and the strict boundary between a message-history effect
|
||||
and a role-semantic claim.
|
||||
|
||||
## Equal-length filler history control
|
||||
|
||||
`v2_lite_routing_history_distance_control.py` extends the preceding 2×2 probe
|
||||
to six cells:
|
||||
|
||||
```text
|
||||
system off/on × no history / repeated-token filler / fixed one-shot
|
||||
```
|
||||
|
||||
The filler turn is:
|
||||
|
||||
```text
|
||||
User: x x x x x x x x x
|
||||
Assistant: x
|
||||
```
|
||||
|
||||
Under the pinned official template, both filler and one-shot add exactly 17
|
||||
tokens per source on both system levels. They therefore share the same
|
||||
user/assistant roles, assistant EOS, target position, generation prompt, and
|
||||
batch shape. The filler is deliberately called low-information rather than
|
||||
semantics-free: repeated `x` tokens remain learned inputs.
|
||||
|
||||
```bash
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||
python -B experiments/deepseek/v2_lite_routing_history_distance_control.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-history-distance-control.json \
|
||||
--per-domain 32 \
|
||||
--content-tokens 23 \
|
||||
--batch-prompts 5 \
|
||||
--layers 7 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T10:17:00+00:00
|
||||
```
|
||||
|
||||
The six cells add 1,376,496 real top-6 route selections. The committed run and
|
||||
independent rerun are byte-exact:
|
||||
|
||||
```text
|
||||
423a095d92738d4bc6c2ace2afa9b35627e61811458c7883dca7b44aee4e648e
|
||||
```
|
||||
|
||||
For exact target content under prompt-balanced aggregation, mean system-edge
|
||||
TV is `0.0738 → 0.0378 → 0.0188` for none → filler → demo. Both steps decrease
|
||||
in 24/24 layer×domain cells, with all 24 paired intervals below zero. See
|
||||
`research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md` for the full
|
||||
table, token-level route stability, BF16 batch-shape boundary, literature
|
||||
context, and non-claims.
|
||||
|
||||
@@ -0,0 +1,552 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a 2 x 3 message-history control on DeepSeek-V2-Lite.
|
||||
|
||||
The system factor is off/on. The history factor has three levels:
|
||||
|
||||
none: no completed turn before the target user message
|
||||
filler: a repeated-token user/assistant turn
|
||||
demo: the fixed meaningful user/assistant turn from the prior probe
|
||||
|
||||
The filler and demo histories add exactly the same number of official-template
|
||||
tokens and keep the same user role, assistant role, assistant EOS, target
|
||||
position, and generation prompt. This tests whether the prior attenuation of
|
||||
the system edge requires the specific demo text. It does not identify a pure
|
||||
distance effect because filler token identity and the added history remain
|
||||
coupled.
|
||||
|
||||
The layer forward path is intentionally reused from the preceding audited
|
||||
probe. This file owns the six-cell condition renderer, the shared-bootstrap
|
||||
2 x 3 statistics, the control metadata, and the finalized result contract.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
import v2_lite_routing_history_factorial_probe as base
|
||||
|
||||
|
||||
SYSTEM_MESSAGE = base.SYSTEM_MESSAGE
|
||||
DEMO_USER = base.DEMO_USER
|
||||
DEMO_ASSISTANT = base.DEMO_ASSISTANT
|
||||
FILLER_USER = "x x x x x x x x x"
|
||||
FILLER_ASSISTANT = "x"
|
||||
|
||||
HISTORY_LEVELS = ("none", "filler", "demo")
|
||||
CONDITIONS = (
|
||||
"s0_none",
|
||||
"s1_none",
|
||||
"s0_filler",
|
||||
"s1_filler",
|
||||
"s0_demo",
|
||||
"s1_demo",
|
||||
)
|
||||
FACTORS = {
|
||||
"s0_none": {"system": 0, "history": "none"},
|
||||
"s1_none": {"system": 1, "history": "none"},
|
||||
"s0_filler": {"system": 0, "history": "filler"},
|
||||
"s1_filler": {"system": 1, "history": "filler"},
|
||||
"s0_demo": {"system": 0, "history": "demo"},
|
||||
"s1_demo": {"system": 1, "history": "demo"},
|
||||
}
|
||||
SYSTEM_CELLS = {
|
||||
"none": ("s0_none", "s1_none"),
|
||||
"filler": ("s0_filler", "s1_filler"),
|
||||
"demo": ("s0_demo", "s1_demo"),
|
||||
}
|
||||
COMPARISONS = (
|
||||
("system_none", "s0_none", "s1_none"),
|
||||
("system_filler", "s0_filler", "s1_filler"),
|
||||
("system_demo", "s0_demo", "s1_demo"),
|
||||
("filler_at_s0", "s0_none", "s0_filler"),
|
||||
("filler_at_s1", "s1_none", "s1_filler"),
|
||||
("demo_at_s0", "s0_none", "s0_demo"),
|
||||
("demo_at_s1", "s1_none", "s1_demo"),
|
||||
("demo_vs_filler_s0", "s0_filler", "s0_demo"),
|
||||
("demo_vs_filler_s1", "s1_filler", "s1_demo"),
|
||||
)
|
||||
ALIGNMENT_COMPARISONS = (
|
||||
COMPARISONS[0],
|
||||
COMPARISONS[1],
|
||||
COMPARISONS[2],
|
||||
COMPARISONS[7],
|
||||
COMPARISONS[8],
|
||||
)
|
||||
SYSTEM_EDGE_CONTRASTS = {
|
||||
"filler_minus_none": ("none", "filler"),
|
||||
"demo_minus_none": ("none", "demo"),
|
||||
"demo_minus_filler": ("filler", "demo"),
|
||||
}
|
||||
|
||||
|
||||
def condition_messages(
|
||||
content: str,
|
||||
condition: str,
|
||||
) -> list[dict[str, str]]:
|
||||
factors = FACTORS[condition]
|
||||
messages: list[dict[str, str]] = []
|
||||
if factors["system"]:
|
||||
messages.append({"role": "system", "content": SYSTEM_MESSAGE})
|
||||
if factors["history"] == "filler":
|
||||
messages.extend(
|
||||
[
|
||||
{"role": "user", "content": FILLER_USER},
|
||||
{"role": "assistant", "content": FILLER_ASSISTANT},
|
||||
]
|
||||
)
|
||||
elif factors["history"] == "demo":
|
||||
messages.extend(
|
||||
[
|
||||
{"role": "user", "content": DEMO_USER},
|
||||
{"role": "assistant", "content": DEMO_ASSISTANT},
|
||||
]
|
||||
)
|
||||
messages.append({"role": "user", "content": content})
|
||||
return messages
|
||||
|
||||
|
||||
def history_control_domain(
|
||||
loads: dict[str, np.ndarray],
|
||||
mode: str,
|
||||
replicates: int,
|
||||
seed: int,
|
||||
scope: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Compute all six cells with one shared source-bootstrap matrix."""
|
||||
shapes = {value.shape for value in loads.values()}
|
||||
if len(shapes) != 1:
|
||||
raise ValueError(f"history-control shape mismatch: {sorted(shapes)}")
|
||||
rows = next(iter(loads.values())).shape[0]
|
||||
rng = np.random.default_rng(base.scoped_seed(seed, scope))
|
||||
sampled = rng.integers(
|
||||
0,
|
||||
rows,
|
||||
size=(replicates, rows),
|
||||
endpoint=False,
|
||||
)
|
||||
point = {
|
||||
condition: base.distribution(value, mode)
|
||||
for condition, value in loads.items()
|
||||
}
|
||||
boot = {
|
||||
condition: base.bootstrap_distributions(value, mode, sampled)
|
||||
for condition, value in loads.items()
|
||||
}
|
||||
point_metrics = {
|
||||
condition: base.metric_vector(value)
|
||||
for condition, value in point.items()
|
||||
}
|
||||
boot_metrics = {
|
||||
condition: base.metric_vector(value)
|
||||
for condition, value in boot.items()
|
||||
}
|
||||
|
||||
def system_edges(
|
||||
values: dict[str, np.ndarray],
|
||||
) -> dict[str, np.ndarray]:
|
||||
return {
|
||||
history: values[after] - values[before]
|
||||
for history, (before, after) in SYSTEM_CELLS.items()
|
||||
}
|
||||
|
||||
def edge_contrasts(
|
||||
edges: dict[str, np.ndarray],
|
||||
) -> dict[str, np.ndarray]:
|
||||
return {
|
||||
name: edges[after] - edges[before]
|
||||
for name, (before, after) in SYSTEM_EDGE_CONTRASTS.items()
|
||||
}
|
||||
|
||||
metric_system_edges: dict[str, Any] = {}
|
||||
metric_system_edge_contrasts: dict[str, Any] = {}
|
||||
for metric in point_metrics["s0_none"]:
|
||||
point_edges = system_edges(
|
||||
{
|
||||
condition: point_metrics[condition][metric]
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
)
|
||||
boot_edges = system_edges(
|
||||
{
|
||||
condition: boot_metrics[condition][metric]
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
)
|
||||
metric_system_edges[metric] = {
|
||||
history: {
|
||||
"point": float(point_edges[history][0]),
|
||||
"ci95": base.interval(boot_edges[history]),
|
||||
}
|
||||
for history in HISTORY_LEVELS
|
||||
}
|
||||
point_contrasts = edge_contrasts(point_edges)
|
||||
boot_contrasts = edge_contrasts(boot_edges)
|
||||
metric_system_edge_contrasts[metric] = {
|
||||
name: {
|
||||
"point": float(point_contrasts[name][0]),
|
||||
"ci95": base.interval(boot_contrasts[name]),
|
||||
}
|
||||
for name in SYSTEM_EDGE_CONTRASTS
|
||||
}
|
||||
|
||||
point_vectors = system_edges(point)
|
||||
boot_vectors = system_edges(boot)
|
||||
point_vector_contrasts = edge_contrasts(point_vectors)
|
||||
boot_vector_contrasts = edge_contrasts(boot_vectors)
|
||||
distribution_system_edge_contrasts = {}
|
||||
for name in SYSTEM_EDGE_CONTRASTS:
|
||||
point_magnitude = 0.5 * np.abs(
|
||||
point_vector_contrasts[name]
|
||||
).sum()
|
||||
boot_magnitude = 0.5 * np.abs(
|
||||
boot_vector_contrasts[name]
|
||||
).sum(axis=1)
|
||||
distribution_system_edge_contrasts[name] = {
|
||||
"half_l1_magnitude": {
|
||||
"point": float(point_magnitude),
|
||||
"ci95": base.interval(boot_magnitude),
|
||||
},
|
||||
"expert_share_difference_in_system_edges": (
|
||||
point_vector_contrasts[name].tolist()
|
||||
),
|
||||
"expert_share_difference_in_system_edges_ci95": (
|
||||
base.interval(boot_vector_contrasts[name])
|
||||
),
|
||||
}
|
||||
|
||||
system_edge_distances: dict[str, Any] = {}
|
||||
point_tv: dict[str, float] = {}
|
||||
boot_tv: dict[str, np.ndarray] = {}
|
||||
point_jsd: dict[str, float] = {}
|
||||
boot_jsd: dict[str, np.ndarray] = {}
|
||||
for history, (before, after) in SYSTEM_CELLS.items():
|
||||
point_delta = point[after] - point[before]
|
||||
point_tv[history] = float(0.5 * np.abs(point_delta).sum())
|
||||
boot_tv[history] = 0.5 * np.abs(
|
||||
boot[after] - boot[before]
|
||||
).sum(axis=1)
|
||||
point_jsd[history] = float(
|
||||
base.js_divergence(point[before], point[after])[0]
|
||||
)
|
||||
boot_jsd[history] = base.js_divergence(
|
||||
boot[before],
|
||||
boot[after],
|
||||
)
|
||||
system_edge_distances[history] = {
|
||||
"total_variation": {
|
||||
"point": point_tv[history],
|
||||
"ci95": base.interval(boot_tv[history]),
|
||||
},
|
||||
"js_divergence": {
|
||||
"point": point_jsd[history],
|
||||
"ci95": base.interval(boot_jsd[history]),
|
||||
"unit": "nats",
|
||||
},
|
||||
}
|
||||
|
||||
system_edge_distance_contrasts = {}
|
||||
for name, (before, after) in SYSTEM_EDGE_CONTRASTS.items():
|
||||
system_edge_distance_contrasts[name] = {
|
||||
"total_variation_delta": {
|
||||
"point": point_tv[after] - point_tv[before],
|
||||
"ci95": base.interval(boot_tv[after] - boot_tv[before]),
|
||||
},
|
||||
"js_divergence_delta": {
|
||||
"point": point_jsd[after] - point_jsd[before],
|
||||
"ci95": base.interval(boot_jsd[after] - boot_jsd[before]),
|
||||
"unit": "nats",
|
||||
},
|
||||
}
|
||||
|
||||
lexical_replacement = {}
|
||||
for system in (0, 1):
|
||||
before = f"s{system}_filler"
|
||||
after = f"s{system}_demo"
|
||||
point_delta = point[after] - point[before]
|
||||
tv_boot = 0.5 * np.abs(
|
||||
boot[after] - boot[before]
|
||||
).sum(axis=1)
|
||||
jsd_boot = base.js_divergence(boot[before], boot[after])
|
||||
lexical_replacement[f"at_s{system}"] = {
|
||||
"total_variation": {
|
||||
"point": float(0.5 * np.abs(point_delta).sum()),
|
||||
"ci95": base.interval(tv_boot),
|
||||
},
|
||||
"js_divergence": {
|
||||
"point": float(
|
||||
base.js_divergence(point[before], point[after])[0]
|
||||
),
|
||||
"ci95": base.interval(jsd_boot),
|
||||
"unit": "nats",
|
||||
},
|
||||
}
|
||||
|
||||
return {
|
||||
"metric_system_edges": metric_system_edges,
|
||||
"metric_system_edge_contrasts": metric_system_edge_contrasts,
|
||||
"system_edge_distances": system_edge_distances,
|
||||
"system_edge_distance_contrasts": (
|
||||
system_edge_distance_contrasts
|
||||
),
|
||||
"distribution_system_edge_contrasts": (
|
||||
distribution_system_edge_contrasts
|
||||
),
|
||||
"lexical_replacement": lexical_replacement,
|
||||
}
|
||||
|
||||
|
||||
def layer_statistics(
|
||||
prompt_rows: list[dict[str, Any]],
|
||||
replicates: int,
|
||||
seed: int,
|
||||
layer_index: int,
|
||||
) -> dict[str, Any]:
|
||||
scopes = {}
|
||||
for load_scope, load_key in (
|
||||
("full_input", "full_load"),
|
||||
("target_content", "content_load"),
|
||||
):
|
||||
modes = {}
|
||||
for mode in ("token_weighted", "prompt_balanced"):
|
||||
conditions: dict[str, Any] = {}
|
||||
domain_loads: dict[str, dict[str, np.ndarray]] = {}
|
||||
for domain in base.DOMAIN_ORDER:
|
||||
rows = [
|
||||
row for row in prompt_rows
|
||||
if row["domain"] == domain
|
||||
]
|
||||
domain_loads[domain] = {
|
||||
condition: np.asarray(
|
||||
[
|
||||
row["conditions"][condition][load_key]
|
||||
for row in rows
|
||||
],
|
||||
dtype=np.int64,
|
||||
)
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
for condition in CONDITIONS:
|
||||
conditions.setdefault(condition, {})[domain] = (
|
||||
base.bootstrap_domain(
|
||||
domain_loads[domain][condition],
|
||||
mode,
|
||||
replicates,
|
||||
seed,
|
||||
(
|
||||
f"layer={layer_index}|scope={load_scope}|"
|
||||
f"mode={mode}|condition={condition}|"
|
||||
f"domain={domain}"
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
comparisons: dict[str, Any] = {}
|
||||
for comparison, before, after in COMPARISONS:
|
||||
comparisons[comparison] = {}
|
||||
for domain in base.DOMAIN_ORDER:
|
||||
comparisons[comparison][domain] = base.paired_domain(
|
||||
domain_loads[domain][before],
|
||||
domain_loads[domain][after],
|
||||
mode,
|
||||
replicates,
|
||||
seed,
|
||||
(
|
||||
f"layer={layer_index}|scope={load_scope}|"
|
||||
f"mode={mode}|comparison={comparison}|"
|
||||
f"domain={domain}"
|
||||
),
|
||||
)
|
||||
|
||||
control = {
|
||||
domain: history_control_domain(
|
||||
domain_loads[domain],
|
||||
mode,
|
||||
replicates,
|
||||
seed,
|
||||
(
|
||||
f"layer={layer_index}|scope={load_scope}|"
|
||||
f"mode={mode}|history_control|domain={domain}"
|
||||
),
|
||||
)
|
||||
for domain in base.DOMAIN_ORDER
|
||||
}
|
||||
modes[mode] = {
|
||||
"conditions": conditions,
|
||||
"comparisons": comparisons,
|
||||
"history_control": control,
|
||||
}
|
||||
scopes[load_scope] = {"modes": modes}
|
||||
return scopes
|
||||
|
||||
|
||||
def install_control_contract() -> None:
|
||||
"""Install the six-cell renderer/statistics into the audited runner."""
|
||||
base.CONDITIONS = CONDITIONS
|
||||
base.FACTORS = FACTORS
|
||||
base.COMPARISONS = COMPARISONS
|
||||
base.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
|
||||
base.condition_messages = condition_messages
|
||||
base.layer_statistics = layer_statistics
|
||||
|
||||
|
||||
def output_path_from_argv() -> Path:
|
||||
try:
|
||||
return Path(sys.argv[sys.argv.index("--output") + 1])
|
||||
except (ValueError, IndexError) as error:
|
||||
raise ValueError("--output is required") from error
|
||||
|
||||
|
||||
def finalize_result(path: Path) -> dict[str, Any]:
|
||||
result = json.loads(path.read_text(encoding="utf-8"))
|
||||
result["evidence_identity"] = (
|
||||
"X / official BF16 weights, official tokenizer chat template, "
|
||||
"paired 2x3 history-distance control on local truncated forward"
|
||||
)
|
||||
boundary = result["boundary"]
|
||||
boundary.pop("factorial_claim", None)
|
||||
boundary.update(
|
||||
{
|
||||
"history_control_claim": (
|
||||
"filler and demo histories have identical official-template "
|
||||
"token increments, roles, assistant EOS, and target position; "
|
||||
"their text identities differ"
|
||||
),
|
||||
"filler_is_semantics_free": False,
|
||||
"pure_distance_isolated": False,
|
||||
"lexical_replacement_control": True,
|
||||
"causal_boundary": (
|
||||
"none-to-filler still couples added history, repeated filler "
|
||||
"tokens, and distance; filler-to-demo isolates replacement of "
|
||||
"the fixed history text only within this protocol"
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
old_contract = result.pop("message_history_contract")
|
||||
selected = result["corpus_contract"]["selected"]
|
||||
increments = {}
|
||||
for system in (0, 1):
|
||||
base_condition = f"s{system}_none"
|
||||
for history in ("filler", "demo"):
|
||||
condition = f"s{system}_{history}"
|
||||
deltas = [
|
||||
row["conditions"][condition]["tokens"]
|
||||
- row["conditions"][base_condition]["tokens"]
|
||||
for row in selected
|
||||
]
|
||||
increments[f"{condition}_minus_{base_condition}"] = {
|
||||
"min": min(deltas),
|
||||
"max": max(deltas),
|
||||
"all_equal": len(set(deltas)) == 1,
|
||||
}
|
||||
result["history_control_contract"] = {
|
||||
"chat_template_revision": base.MODEL_REVISION,
|
||||
"chat_template": old_contract["chat_template"],
|
||||
"chat_template_sha256": old_contract["chat_template_sha256"],
|
||||
"system_message": SYSTEM_MESSAGE,
|
||||
"system_message_sha256": base.text_sha256(SYSTEM_MESSAGE),
|
||||
"demo_user": DEMO_USER,
|
||||
"demo_user_sha256": base.text_sha256(DEMO_USER),
|
||||
"demo_assistant": DEMO_ASSISTANT,
|
||||
"demo_assistant_sha256": base.text_sha256(DEMO_ASSISTANT),
|
||||
"filler_user": FILLER_USER,
|
||||
"filler_user_sha256": base.text_sha256(FILLER_USER),
|
||||
"filler_assistant": FILLER_ASSISTANT,
|
||||
"filler_assistant_sha256": base.text_sha256(FILLER_ASSISTANT),
|
||||
"target_role": "user",
|
||||
"add_generation_prompt": True,
|
||||
"conditions": FACTORS,
|
||||
"comparisons": [
|
||||
{"name": name, "before": before, "after": after}
|
||||
for name, before, after in COMPARISONS
|
||||
],
|
||||
"system_edge_contrasts": {
|
||||
name: {
|
||||
"before_history": before,
|
||||
"after_history": after,
|
||||
"definition": (
|
||||
f"system edge at {after} minus system edge at {before}"
|
||||
),
|
||||
}
|
||||
for name, (before, after) in SYSTEM_EDGE_CONTRASTS.items()
|
||||
},
|
||||
"token_increment_validation": increments,
|
||||
"scope_split": {
|
||||
"full_input": (
|
||||
"all rendered BOS, system/history, target, newlines, EOS, "
|
||||
"and generation-prompt tokens"
|
||||
),
|
||||
"target_content": (
|
||||
"exact intersection of (relative character span, token ID) "
|
||||
"inside target user content across all six conditions"
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
inference = result["inference_contract"]
|
||||
inference["batch_grouping"] = (
|
||||
"all six control variants of one source prompt execute in the same "
|
||||
"right-padded batch"
|
||||
)
|
||||
statistical = result["statistical_contract"]
|
||||
statistical["paired_indices"] = (
|
||||
"one sampled source-prompt index matrix is reused across all six "
|
||||
"cells for every history-control contrast within each "
|
||||
"domain/layer/scope/mode"
|
||||
)
|
||||
statistical.pop("interaction_distribution_magnitude", None)
|
||||
statistical["system_edge_contrast_distribution_magnitude"] = (
|
||||
"0.5 * L1 norm of the signed difference between two system-edge "
|
||||
"expert-share vectors; this is not labeled standard TV"
|
||||
)
|
||||
result["schema_version"] = 2
|
||||
path.write_text(
|
||||
json.dumps(result, indent=2, ensure_ascii=False) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
install_control_contract()
|
||||
output = output_path_from_argv()
|
||||
with open(os.devnull, "w", encoding="utf-8") as sink:
|
||||
with contextlib.redirect_stdout(sink):
|
||||
base.main()
|
||||
result = finalize_result(output)
|
||||
payload = output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(output),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"bytes": len(payload),
|
||||
"source_prompts": result["inference_contract"][
|
||||
"total_source_prompts"
|
||||
],
|
||||
"prompt_variants": result["inference_contract"][
|
||||
"total_prompt_variants"
|
||||
],
|
||||
"input_tokens_by_condition": result[
|
||||
"inference_contract"
|
||||
]["input_tokens_by_condition"],
|
||||
"total_routes": result["inference_contract"][
|
||||
"total_routes_all_conditions_all_moe_layers"
|
||||
],
|
||||
},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,678 @@
|
||||
# DeepSeek-V2-Lite 消息历史距离控制审计
|
||||
|
||||
> 状态：真实官方权重执行（X）<br />
|
||||
> 模型：`deepseek-ai/DeepSeek-V2-Lite`<br />
|
||||
> revision：`604d5664dddd88a0433dbae533b7fe9472482de0`<br />
|
||||
> 执行边界：layer 0–6；观测 MoE layer 1–6<br />
|
||||
> 样本：WikiText-2 / TNEWS / HumanEval / GSM8K 各 32 条<br />
|
||||
> 正式运行与独立复跑：byte-exact<br />
|
||||
> 完整 JSON SHA-256：`423a095d92738d4bc6c2ace2afa9b35627e61811458c7883dca7b44aee4e648e`
|
||||
|
||||
## 0. 一句话先说结论
|
||||
|
||||
上一轮发现：
|
||||
|
||||
> system 后面先经过一轮 one-shot 历史，再到目标 user 时，<br />
|
||||
> system 对目标 token 专家路由的影响明显变小。
|
||||
|
||||
但那轮把五件事绑在一起：
|
||||
|
||||
1. system 离目标更远；
|
||||
2. 中间多了一段文本；
|
||||
3. 多了一次 user → assistant 角色转换；
|
||||
4. 多了 assistant EOS；
|
||||
5. 中间文本恰好是一个可读示例。
|
||||
|
||||
本轮加入一个**重复词元 filler 历史**：
|
||||
|
||||
```text
|
||||
User: x x x x x x x x x
|
||||
Assistant: x + EOS
|
||||
```
|
||||
|
||||
它与原 one-shot 在官方模板中都精确增加 17 tokens，并保持完全相同的：
|
||||
|
||||
- user / assistant 角色；
|
||||
- assistant EOS；
|
||||
- 目标 user 的绝对 token 位置；
|
||||
- generation prompt；
|
||||
- padded batch 形状。
|
||||
|
||||
目标内容上的 system-edge TV 均值变为：
|
||||
|
||||
```text
|
||||
无历史 filler 历史 原 one-shot
|
||||
0.0738 → 0.0378 → 0.0188
|
||||
```
|
||||
|
||||
而且两个台阶都在 **24 / 24 个 layer × domain** 中下降，
|
||||
各自的 source-paired 95% 区间也都是 **24 / 24 完全低于零**。
|
||||
|
||||
最稳妥的解释是：
|
||||
|
||||
> 在这组固定协议中，不需要一个可读的正确示例，
|
||||
> 仅加入等长的重复词元历史就能复现约一半的 system-edge 缓冲；
|
||||
> 换回原 one-shot 文本后还会再下降约一半。
|
||||
|
||||
这仍然不是“纯距离因果效应”：
|
||||
|
||||
- filler token 本身不是空气；
|
||||
- filler 仍然改变了历史长度与隐藏状态；
|
||||
- 原 one-shot 与 filler 的词元身份不同；
|
||||
- 本轮没有生成答案，也没有测能力。
|
||||
|
||||
## 1. 为什么不是直接说“few-shot 让路由稳定”
|
||||
|
||||
“few-shot 让路由稳定”至少混合了三个问题：
|
||||
|
||||
1. **结构问题**：前面是否已有一个完成的 user / assistant turn？
|
||||
2. **位置问题**：system 离目标有多远？
|
||||
3. **内容问题**：中间那轮到底说了什么？
|
||||
|
||||
如果只比较：
|
||||
|
||||
```text
|
||||
system → target
|
||||
system → demo → target
|
||||
```
|
||||
|
||||
看到差异后无法知道是哪一项造成的。
|
||||
|
||||
本轮不是一步解决所有问题，而是插入一个中间台阶：
|
||||
|
||||
```text
|
||||
无历史 → 等长 filler → 原 one-shot
|
||||
```
|
||||
|
||||
这样至少可以问：
|
||||
|
||||
- 没有可读示例时，历史结构是否已经足以产生缓冲？
|
||||
- 在长度、角色、EOS 与位置相同时，替换具体文本还有多少额外变化？
|
||||
|
||||
## 2. 六格实验是什么
|
||||
|
||||
两个因子：
|
||||
|
||||
- `S`：system message 是否存在；
|
||||
- `H`：历史为 none / filler / demo。
|
||||
|
||||
| 条件 | system | 历史 | 官方模板结构 |
|
||||
|---|---:|---|---|
|
||||
| `S0 / NONE` | 0 | none | target user → `Assistant:` |
|
||||
| `S1 / NONE` | 1 | none | system → target user → `Assistant:` |
|
||||
| `S0 / FILLER` | 0 | filler | repeated-token user → assistant + EOS → target |
|
||||
| `S1 / FILLER` | 1 | filler | system → repeated-token turn → target |
|
||||
| `S0 / DEMO` | 0 | demo | fixed demo user → assistant + EOS → target |
|
||||
| `S1 / DEMO` | 1 | demo | system → fixed demo turn → target |
|
||||
|
||||
固定文本：
|
||||
|
||||
```text
|
||||
system
|
||||
Answer accurately and concisely. 请准确、简洁地回答。
|
||||
|
||||
demo user
|
||||
Reply with OK. 只回复 OK。
|
||||
|
||||
demo assistant
|
||||
OK
|
||||
|
||||
filler user
|
||||
x x x x x x x x x
|
||||
|
||||
filler assistant
|
||||
x
|
||||
```
|
||||
|
||||
六格全部使用：
|
||||
|
||||
```python
|
||||
tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
```
|
||||
|
||||
没有手写一个“看起来像官方”的模板。
|
||||
|
||||
## 3. “等长”到底等在哪里
|
||||
|
||||
128 条 source 上，逐条验证：
|
||||
|
||||
| 对比 | 每条 token 增量 | min | max | 是否全部相等 |
|
||||
|---|---:|---:|---:|---|
|
||||
| S0 filler − S0 none | +17 | 17 | 17 | 是 |
|
||||
| S0 demo − S0 none | +17 | 17 | 17 | 是 |
|
||||
| S1 filler − S1 none | +17 | 17 | 17 | 是 |
|
||||
| S1 demo − S1 none | +17 | 17 | 17 | 是 |
|
||||
|
||||
system 也继续保持上一轮的正交增量：
|
||||
|
||||
```text
|
||||
S1 − S0 = 每条 +16 tokens
|
||||
```
|
||||
|
||||
因此：
|
||||
|
||||
- filler 与 demo 的目标位置相同；
|
||||
- filler 与 demo 的角色前缀数量相同；
|
||||
- filler 与 demo 都有一个 assistant EOS；
|
||||
- 两者差别集中在 history content token identity。
|
||||
|
||||
### 3.1 为什么不叫“无语义 filler”
|
||||
|
||||
`x` 是一个真实、经过训练的 token。
|
||||
|
||||
九个 `x` 也可能触发：
|
||||
|
||||
- 重复模式检测；
|
||||
- induction-like matching；
|
||||
- 特定 token embedding；
|
||||
- 特定 attention / MoE 路由。
|
||||
|
||||
所以这里使用：
|
||||
|
||||
> **低信息重复词元 filler**
|
||||
|
||||
而不使用：
|
||||
|
||||
> “无语义”“空白上下文”“纯距离”
|
||||
|
||||
## 4. 样本与上一轮完全一致
|
||||
|
||||
继续复用：
|
||||
|
||||
| 领域 | 数据 | 条数 | 输入字段 |
|
||||
|---|---|---:|---|
|
||||
| 英文百科 | WikiText-2 raw validation | 32 | `text` |
|
||||
| 中文新闻 | CLUE TNEWS public test | 32 | `sentence` |
|
||||
| Python 代码 | OpenAI HumanEval | 32 | `prompt` |
|
||||
| 小学数学 | OpenAI GSM8K test | 32 | `question` |
|
||||
|
||||
固定：
|
||||
|
||||
- 相同 source IDs；
|
||||
- 相同 source text SHA-256；
|
||||
- 相同 canonical content SHA-256；
|
||||
- 相同 23-token canonical prefix；
|
||||
- 相同 sample salt；
|
||||
- 答案不输入；
|
||||
- 代码不执行。
|
||||
|
||||
本轮 128 个 `(source ID, content SHA-256)` 与模板实验、2×2 历史实验
|
||||
逐项一致。
|
||||
|
||||
## 5. 执行账本
|
||||
|
||||
| 项目 | 数值 |
|
||||
|---|---:|
|
||||
| source prompts | 128 |
|
||||
| 六格输入变体 | 768 |
|
||||
| 输入 tokens | 38,236 |
|
||||
| 六格共同目标交集 | 每格 2,874 |
|
||||
| 执行层 | layer 0–6 |
|
||||
| 观测 MoE 层 | layer 1–6 |
|
||||
| 每 token routed experts | 6 |
|
||||
| 新增真实 top-6 路由 | 1,376,496 |
|
||||
| 公开语料累计真实路由 | 3,112,848 |
|
||||
| bootstrap | 每域 2,000 次 |
|
||||
|
||||
分条件：
|
||||
|
||||
| 条件 | 输入 tokens | 六层真实路由 |
|
||||
|---|---:|---:|
|
||||
| S0 none | 3,898 | 140,328 |
|
||||
| S1 none | 5,946 | 214,056 |
|
||||
| S0 filler | 6,074 | 218,664 |
|
||||
| S1 filler | 8,122 | 292,392 |
|
||||
| S0 demo | 6,074 | 218,664 |
|
||||
| S1 demo | 8,122 | 292,392 |
|
||||
|
||||
注意：路由次数等于真实输入 tokens × 6 experts × 6 MoE layers，
|
||||
不是模型参数量，也不是训练 token 数。
|
||||
|
||||
## 6. 为什么仍然要精确对齐目标 token
|
||||
|
||||
不同历史会改变目标前面的文本，进而可能改变：
|
||||
|
||||
- 目标第一个 token 的边界切分；
|
||||
- wrapper token 数量；
|
||||
- token 的绝对位置。
|
||||
|
||||
本轮继续用三元组：
|
||||
|
||||
```text
|
||||
(相对目标字符跨度 start, end, token ID)
|
||||
```
|
||||
|
||||
只有六个条件都存在的三元组才进入 `target_content`。
|
||||
|
||||
结果：
|
||||
|
||||
```text
|
||||
S0 none 2,874
|
||||
S1 none 2,874
|
||||
S0 filler 2,874
|
||||
S1 filler 2,874
|
||||
S0 demo 2,874
|
||||
S1 demo 2,874
|
||||
```
|
||||
|
||||
因此目标内容比较没有混入：
|
||||
|
||||
- `User:`；
|
||||
- `Assistant:`；
|
||||
- system；
|
||||
- filler / demo；
|
||||
- EOS；
|
||||
- 边界重切分 token。
|
||||
|
||||
## 7. 统计单位与三个 system edges
|
||||
|
||||
对每个历史水平 \(h\)，定义：
|
||||
|
||||
```text
|
||||
p(0,h) = system 关闭时的 64-expert route-share
|
||||
p(1,h) = system 开启时的 64-expert route-share
|
||||
```
|
||||
|
||||
system edge 的标准 Total Variation：
|
||||
|
||||
\[
|
||||
TV_h = \frac{1}{2}\lVert p(1,h)-p(0,h)\rVert_1
|
||||
\]
|
||||
|
||||
三个真实边：
|
||||
|
||||
```text
|
||||
TV_none
|
||||
TV_filler
|
||||
TV_demo
|
||||
```
|
||||
|
||||
再比较：
|
||||
|
||||
```text
|
||||
filler − none
|
||||
demo − none
|
||||
demo − filler
|
||||
```
|
||||
|
||||
每个 domain / layer / scope / aggregation 内：
|
||||
|
||||
1. 以 source prompt 为单位有放回重采样；
|
||||
2. 2,000 次；
|
||||
3. 六格共用同一组 sampled source indices；
|
||||
4. 先同时重建六格分布；
|
||||
5. 再计算三条 system edge 与 edge 差。
|
||||
|
||||
因此 token 没有被伪装成独立样本。
|
||||
|
||||
## 8. 主结果：两个台阶都是 24 / 24
|
||||
|
||||
以下均为：
|
||||
|
||||
```text
|
||||
target_content / prompt_balanced
|
||||
```
|
||||
|
||||
| 层 | 域 | none TV | filler TV | demo TV | filler−none 95% CI | demo−filler 95% CI |
|
||||
|---:|---|---:|---:|---:|---|---|
|
||||
| 1 | 英文 | .1055 | .0559 | .0236 | -.0496 [-.0565,-.0406] | -.0323 [-.0402,-.0274] |
|
||||
| 1 | 中文 | .0778 | .0276 | .0166 | -.0501 [-.0600,-.0410] | -.0111 [-.0161,-.0063] |
|
||||
| 1 | 代码 | .1123 | .0571 | .0243 | -.0552 [-.0623,-.0480] | -.0328 [-.0397,-.0272] |
|
||||
| 1 | 数学 | .1328 | .0791 | .0250 | -.0537 [-.0632,-.0457] | -.0541 [-.0638,-.0428] |
|
||||
| 2 | 英文 | .0621 | .0300 | .0153 | -.0321 [-.0395,-.0244] | -.0147 [-.0211,-.0087] |
|
||||
| 2 | 中文 | .0420 | .0204 | .0145 | -.0216 [-.0286,-.0172] | -.0059 [-.0104,-.0026] |
|
||||
| 2 | 代码 | .0586 | .0329 | .0166 | -.0257 [-.0322,-.0197] | -.0163 [-.0209,-.0099] |
|
||||
| 2 | 数学 | .0734 | .0399 | .0195 | -.0335 [-.0420,-.0246] | -.0204 [-.0276,-.0152] |
|
||||
| 3 | 英文 | .0574 | .0317 | .0184 | -.0257 [-.0342,-.0191] | -.0133 [-.0204,-.0080] |
|
||||
| 3 | 中文 | .0351 | .0220 | .0131 | -.0131 [-.0212,-.0079] | -.0089 [-.0143,-.0053] |
|
||||
| 3 | 代码 | .0654 | .0398 | .0130 | -.0256 [-.0331,-.0187] | -.0267 [-.0315,-.0220] |
|
||||
| 3 | 数学 | .0485 | .0345 | .0164 | -.0140 [-.0233,-.0094] | -.0181 [-.0225,-.0133] |
|
||||
| 4 | 英文 | .0834 | .0338 | .0198 | -.0496 [-.0591,-.0432] | -.0139 [-.0211,-.0073] |
|
||||
| 4 | 中文 | .0714 | .0286 | .0150 | -.0428 [-.0539,-.0353] | -.0136 [-.0186,-.0082] |
|
||||
| 4 | 代码 | .0750 | .0445 | .0148 | -.0305 [-.0381,-.0242] | -.0297 [-.0349,-.0241] |
|
||||
| 4 | 数学 | .1079 | .0511 | .0232 | -.0569 [-.0672,-.0473] | -.0279 [-.0345,-.0229] |
|
||||
| 5 | 英文 | .0666 | .0338 | .0222 | -.0328 [-.0420,-.0220] | -.0116 [-.0218,-.0062] |
|
||||
| 5 | 中文 | .0520 | .0269 | .0161 | -.0250 [-.0364,-.0182] | -.0108 [-.0156,-.0061] |
|
||||
| 5 | 代码 | .0549 | .0386 | .0184 | -.0163 [-.0233,-.0104] | -.0202 [-.0249,-.0161] |
|
||||
| 5 | 数学 | .0728 | .0449 | .0229 | -.0279 [-.0376,-.0192] | -.0221 [-.0313,-.0162] |
|
||||
| 6 | 英文 | .0761 | .0312 | .0208 | -.0449 [-.0542,-.0373] | -.0104 [-.0160,-.0036] |
|
||||
| 6 | 中文 | .0578 | .0256 | .0181 | -.0322 [-.0432,-.0247] | -.0074 [-.0131,-.0038] |
|
||||
| 6 | 代码 | .0731 | .0343 | .0164 | -.0388 [-.0489,-.0292] | -.0179 [-.0244,-.0131] |
|
||||
| 6 | 数学 | .1093 | .0422 | .0270 | -.0671 [-.0768,-.0562] | -.0153 [-.0227,-.0083] |
|
||||
|
||||
汇总：
|
||||
|
||||
| 问题 | 点估计下降 | 95% 区间完全低于零 |
|
||||
|---|---:|---:|
|
||||
| filler 是否小于 none | 24 / 24 | 24 / 24 |
|
||||
| demo 是否小于 filler | 24 / 24 | 24 / 24 |
|
||||
|
||||
均值：
|
||||
|
||||
```text
|
||||
none .0738
|
||||
filler .0378 相对 none 约下降 49%
|
||||
demo .0188 相对 filler 约再下降 50%
|
||||
相对 none 约下降 75%
|
||||
```
|
||||
|
||||
这不是 24 个独立总体显著性检验，也没有多重比较校正。
|
||||
它是固定 cohort 上 24 个分层描述区间。
|
||||
|
||||
## 9. 六层都出现相同阶梯
|
||||
|
||||
四域平均：
|
||||
|
||||
| 层 | none | filler | demo |
|
||||
|---:|---:|---:|---:|
|
||||
| 1 | .1071 | .0549 | .0224 |
|
||||
| 2 | .0590 | .0308 | .0165 |
|
||||
| 3 | .0516 | .0320 | .0152 |
|
||||
| 4 | .0844 | .0395 | .0182 |
|
||||
| 5 | .0616 | .0361 | .0199 |
|
||||
| 6 | .0791 | .0333 | .0206 |
|
||||
|
||||
四域分别跨层平均：
|
||||
|
||||
| 域 | none | filler | demo |
|
||||
|---|---:|---:|---:|
|
||||
| 英文 | .0752 | .0361 | .0200 |
|
||||
| 中文 | .0560 | .0252 | .0156 |
|
||||
| 代码 | .0732 | .0412 | .0173 |
|
||||
| 数学 | .0908 | .0486 | .0223 |
|
||||
|
||||
不能据此给 expert 命名，也不能推出数学“更依赖”某个专家。
|
||||
|
||||
## 10. 逐 token 路由集合也同向
|
||||
|
||||
下表把 128 条 prompt 的精确对齐目标 token 汇总。
|
||||
|
||||
每格：
|
||||
|
||||
```text
|
||||
top-6 set exact / mean Jaccard
|
||||
```
|
||||
|
||||
| 层 | system / none | system / filler | system / demo |
|
||||
|---:|---|---|---|
|
||||
| 1 | .316 / .756 | .565 / .864 | .790 / .939 |
|
||||
| 2 | .459 / .820 | .693 / .904 | .815 / .946 |
|
||||
| 3 | .479 / .831 | .683 / .901 | .826 / .950 |
|
||||
| 4 | .390 / .784 | .674 / .894 | .800 / .942 |
|
||||
| 5 | .438 / .799 | .657 / .887 | .789 / .939 |
|
||||
| 6 | .387 / .770 | .645 / .886 | .769 / .932 |
|
||||
|
||||
也就是说，aggregate route-share TV 的下降不是只有汇总分布看得到；
|
||||
同一个目标 token 的 top-6 expert 集合也逐级更相似。
|
||||
|
||||
但“路由集合更相似”仍不等于“答案更好”。
|
||||
|
||||
## 11. 固定长度后，具体文本仍然重要
|
||||
|
||||
filler → demo 只替换 history content，角色、EOS、长度和目标位置不变。
|
||||
|
||||
目标内容上的平均 filler↔demo TV：
|
||||
|
||||
```text
|
||||
system 关闭：.0357
|
||||
system 开启：.0145
|
||||
```
|
||||
|
||||
在 24 / 24 个 layer × domain 中：
|
||||
|
||||
```text
|
||||
system 开启时的 filler↔demo TV < system 关闭时
|
||||
```
|
||||
|
||||
这显示的是一个双向交互模式：
|
||||
|
||||
- history 类型改变 system edge；
|
||||
- system 的存在也改变 filler / demo 之间的距离。
|
||||
|
||||
它不证明：
|
||||
|
||||
- 正确示例比错误示例好；
|
||||
- model 理解了 `Reply with OK`；
|
||||
- system 与 demo 在功能上互相替代；
|
||||
- 路由距离与任务性能单调相关。
|
||||
|
||||
## 12. 为什么 CV 没有 24 / 24
|
||||
|
||||
TV 衡量整个 64-expert 分布的距离。
|
||||
|
||||
CV 只把分布压成一个“负载离散程度”标量：
|
||||
|
||||
- 分布可以大幅重排，但 CV 几乎不变；
|
||||
- 分布可以朝不同 expert 移动，CV 方向相反；
|
||||
- 两个分布 TV 很小，也可能跨过 CV 的局部斜率。
|
||||
|
||||
绝对 system-edge CV 变化：
|
||||
|
||||
| 对比 | \|CV edge\| 下降 |
|
||||
|---|---:|
|
||||
| none → filler | 18 / 24 |
|
||||
| filler → demo | 19 / 24 |
|
||||
|
||||
CV edge contrast 的区间方向也不统一：
|
||||
|
||||
| 对比 | 完全负 | 完全正 | 跨零 |
|
||||
|---|---:|---:|---:|
|
||||
| filler − none | 4 | 9 | 11 |
|
||||
| demo − filler | 8 | 4 | 12 |
|
||||
|
||||
所以本轮主结论严格写成：
|
||||
|
||||
> **system-edge route-distribution TV 下降**
|
||||
|
||||
而不是：
|
||||
|
||||
> “专家负载一定更均衡”
|
||||
|
||||
## 13. 完整输入与目标内容必须分开
|
||||
|
||||
目标内容：
|
||||
|
||||
```text
|
||||
none .0738 → filler .0378 → demo .0188
|
||||
24/24 → 24/24
|
||||
```
|
||||
|
||||
完整输入：
|
||||
|
||||
```text
|
||||
none .1582 → filler .1390 → demo .1131
|
||||
20/24 → 23/24
|
||||
```
|
||||
|
||||
完整输入还把以下 token 计入：
|
||||
|
||||
- system；
|
||||
- filler / demo；
|
||||
- `User:`；
|
||||
- `Assistant:`；
|
||||
- EOS；
|
||||
- generation prompt。
|
||||
|
||||
它回答的是：
|
||||
|
||||
> 整个协议流量的 route-share 如何变化？
|
||||
|
||||
目标内容回答的是：
|
||||
|
||||
> 同一段目标文本在不同历史下如何被条件化？
|
||||
|
||||
两者不能互相替代。
|
||||
|
||||
## 14. BF16 batch shape 是一个真实边界
|
||||
|
||||
新六格中，旧四格的输入合同仍可与上一轮逐项核对：
|
||||
|
||||
```text
|
||||
512 / 512 token-ID SHA-256 exact
|
||||
```
|
||||
|
||||
但上一轮每批：
|
||||
|
||||
```text
|
||||
8 sources × 4 variants = 32 rows
|
||||
```
|
||||
|
||||
本轮每批：
|
||||
|
||||
```text
|
||||
5 sources × 6 variants = 30 rows
|
||||
```
|
||||
|
||||
虽然相同条件的 token IDs 完全一致，BF16 layer forward 的矩阵形状不同。
|
||||
跨两次实验比较：
|
||||
|
||||
```text
|
||||
1,062 / 3,072 prompt-layer-condition route hashes exact
|
||||
1,414 / 3,072 integer load-vector pairs exact
|
||||
```
|
||||
|
||||
Layer 1 仍有：
|
||||
|
||||
```text
|
||||
481 / 512 route hashes exact
|
||||
```
|
||||
|
||||
但细小 BF16 差异向深层传播后，整段 hash 很容易不再相同。
|
||||
|
||||
这意味着：
|
||||
|
||||
1. 相同 token IDs 不保证不同 batch shape 下所有临界 top-k gate 完全相同；
|
||||
2. 本轮不能把旧实验与新实验的点估计差异全部解释为文本处理；
|
||||
3. 本轮正式对比只使用**同一次六格运行内部**的条件；
|
||||
4. 六格在每个 source 内进入同一 padded batch；
|
||||
5. 正式运行自身的独立复跑仍然 byte-exact。
|
||||
|
||||
这是为什么“可复现”不仅要固定 seed，还要记录 batch contract。
|
||||
|
||||
## 15. 与已有论文的关系
|
||||
|
||||
### 15.1 demonstration 的作用不只是真实标签
|
||||
|
||||
[Min et al., EMNLP 2022](https://aclanthology.org/2022.emnlp-main.759/)
|
||||
在一组分类与多选 ICL 实验中发现，label correctness 并非所有收益的唯一来源，
|
||||
输入分布、label space 与 sequence format 也会贡献表现。
|
||||
|
||||
本轮与其问题意识相近：
|
||||
|
||||
> 把“示例内容”与“示例格式 / 历史结构”分开。
|
||||
|
||||
但本轮只测 DeepSeek-V2-Lite 的局部 MoE 路由，不复现其任务性能结论。
|
||||
|
||||
### 15.2 重复结构可能被模型机制利用
|
||||
|
||||
[Olsson et al., 2022](https://arxiv.org/abs/2209.11895)
|
||||
讨论 induction heads 与序列内重复模式的关系；其在小型 attention-only 模型中提供
|
||||
强因果证据，在更大含 MLP 模型中主要是相关证据。
|
||||
|
||||
因此九个重复 `x` 不能被当作“什么都没有”。
|
||||
它恰好可能是一种很强的结构信号。
|
||||
|
||||
### 15.3 位置会影响模型如何使用上下文
|
||||
|
||||
[Liu et al., TACL 2024](https://aclanthology.org/2024.tacl-1.9/)
|
||||
展示了长上下文任务中的位置敏感性。
|
||||
|
||||
本轮只移动 17 个模板 tokens，且没有任务输出，
|
||||
不能直接套用“lost in the middle”结论；它只说明位置 / 历史距离值得单独控制。
|
||||
|
||||
### 15.4 irrelevant context 可能改变行为
|
||||
|
||||
[Shi et al., ICML 2023](https://proceedings.mlr.press/v202/shi23a.html)
|
||||
研究了无关上下文对任务表现的干扰。
|
||||
|
||||
本轮没有测答案，因此 filler 导致路由变化不等于 distractor 导致能力下降。
|
||||
|
||||
## 16. 独立复跑
|
||||
|
||||
正式结果：
|
||||
|
||||
```text
|
||||
src/data/deepseek-v2-lite-routing-history-distance-control.json
|
||||
```
|
||||
|
||||
独立复跑：
|
||||
|
||||
```text
|
||||
src/data/deepseek-v2-lite-routing-history-distance-control-repro.json
|
||||
```
|
||||
|
||||
两者：
|
||||
|
||||
```text
|
||||
42,885,795 bytes
|
||||
SHA-256
|
||||
423a095d92738d4bc6c2ace2afa9b35627e61811458c7883dca7b44aee4e648e
|
||||
```
|
||||
|
||||
脚本：
|
||||
|
||||
```text
|
||||
experiments/deepseek/v2_lite_routing_history_distance_control.py
|
||||
```
|
||||
|
||||
关键执行参数：
|
||||
|
||||
```bash
|
||||
python -B experiments/deepseek/v2_lite_routing_history_distance_control.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--output src/data/deepseek-v2-lite-routing-history-distance-control.json \
|
||||
--per-domain 32 \
|
||||
--content-tokens 23 \
|
||||
--batch-prompts 5 \
|
||||
--layers 7 \
|
||||
--bootstrap 2000 \
|
||||
--seed 20260729 \
|
||||
--captured-at 2026-07-29T10:17:00+00:00
|
||||
```
|
||||
|
||||
## 17. 现在能说什么
|
||||
|
||||
- filler 与 demo 在官方模板下精确等长；
|
||||
- 两者拥有相同角色、EOS 与目标位置；
|
||||
- 重复词元 filler 已复现明显的 system-edge TV 缓冲；
|
||||
- 从 filler 换为原 one-shot 后，system-edge TV 还会继续下降；
|
||||
- 两个下降在 24 / 24 个 layer × domain 中同向；
|
||||
- 逐 token top-6 set / Jaccard 与 aggregate TV 同向；
|
||||
- 六格正式运行可以独立 byte-exact 复跑；
|
||||
- batch shape 会影响深层临界 gate identity，必须记录。
|
||||
|
||||
## 18. 现在不能说什么
|
||||
|
||||
- filler 是无语义的；
|
||||
- 本轮已经隔离纯距离效应；
|
||||
- 原 one-shot 的“正确性”造成额外下降；
|
||||
- 更小的 route TV 代表更好的答案；
|
||||
- 更稳定的 expert set 代表更强能力；
|
||||
- CV 一定下降或负载一定更均衡；
|
||||
- 结果代表完整 27 层；
|
||||
- 结果代表线上多轮对话；
|
||||
- 24 个区间是多重校正后的总体显著性结论。
|
||||
|
||||
## 19. 下一步
|
||||
|
||||
1. **EOS 控制**：尽可能固定可见 token 文本，只切换 assistant EOS / turn closure；
|
||||
2. **角色控制**：相同 token 序列放入 user / assistant / system 的不同角色；
|
||||
3. **内容控制**：固定长度，比较正确、无关、冲突与重复示例；
|
||||
4. **多 filler 控制**：使用多个 token identity 与随机种子，避免把 `x` 当作总体；
|
||||
5. **batch-shape 控制**：固定矩阵形状并加入显式 dummy-row contract；
|
||||
6. **行为联结**：生成答案，用独立任务指标检查 route TV 与能力是否相关；
|
||||
7. **完整模型**：获得其余 shards 后扩展到全部 27 层。
|
||||
|
||||
在这些控制完成之前，本轮结论保持为：
|
||||
|
||||
> 固定 DeepSeek-V2-Lite、固定公开 cohort、固定官方模板与固定六格 batch 下的
|
||||
> **消息历史结构 / 文本替换路由实证**。
|
||||
@@ -418,7 +418,7 @@ const artifactHistory = await evaluate(`(() => {
|
||||
allEffects: node.querySelector(":scope > small").textContent.trim(),
|
||||
edges: node.querySelector("em").textContent.trim(),
|
||||
})),
|
||||
bufferSummary: [...root.querySelectorAll(".history-buffer-summary article b")].map((node) => node.textContent.trim()),
|
||||
bufferSummary: [...root.querySelectorAll('[data-artifact-panel="history"] .history-buffer-summary article b')].map((node) => node.textContent.trim()),
|
||||
bufferCards: [...root.querySelectorAll("[data-history-buffer-grid] > article")].map((node) => ({
|
||||
label: node.querySelector("span").textContent.trim(),
|
||||
tv: node.querySelector("b").textContent.trim(),
|
||||
@@ -461,6 +461,64 @@ await evaluate(`(() => {
|
||||
await pause(120);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-history-results-desktop.png");
|
||||
|
||||
const artifactDistance = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-dsv2-lab]");
|
||||
root.querySelector('[data-artifact-tab="distance"]').click();
|
||||
const read = () => ({
|
||||
panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
|
||||
matrixRows: root.querySelectorAll(".distance-row:not(.head)").length,
|
||||
matrixCells: root.querySelectorAll(".distance-row:not(.head) > article").length,
|
||||
domainCards: root.querySelectorAll("[data-distance-domain-grid] > article").length,
|
||||
domains: [...root.querySelectorAll("[data-distance-domain-grid] > article")].map((node) => ({
|
||||
label: node.querySelector(":scope > span").textContent.trim(),
|
||||
values: [...node.querySelectorAll(".distance-tv-ladder i")].map((cell) => ({
|
||||
name: cell.querySelector("small").textContent.trim(),
|
||||
value: cell.querySelector("b").textContent.trim(),
|
||||
})),
|
||||
effect: node.querySelector("strong").textContent.trim(),
|
||||
className: node.querySelector("strong").className,
|
||||
ci: node.querySelector(":scope > p").textContent.trim(),
|
||||
stability: node.querySelector(":scope > small").textContent.trim(),
|
||||
lexical: node.querySelector(":scope > em").textContent.trim(),
|
||||
cv: node.querySelector(":scope > u").textContent.trim(),
|
||||
})),
|
||||
summary: [...root.querySelectorAll(".distance-summary article b")].map((node) => node.textContent.trim()),
|
||||
depthRows: root.querySelectorAll("[data-distance-depth-map] > div").length,
|
||||
depthCells: root.querySelectorAll("[data-distance-depth-map] > div > span").length,
|
||||
depthTitle: root.querySelector("[data-distance-depth-title]").textContent.trim(),
|
||||
exact: root.querySelector(".distance-ledger .exact b").textContent.trim(),
|
||||
note: root.querySelector("[data-distance-note]").textContent.trim(),
|
||||
activeLayer: root.querySelector("[data-distance-layer].active").textContent.trim(),
|
||||
activeScope: root.querySelector('[data-distance-scope][aria-pressed="true"]').dataset.distanceScope,
|
||||
activeMode: root.querySelector('[data-distance-mode][aria-pressed="true"]').dataset.distanceMode,
|
||||
activeContrast: root.querySelector('[data-distance-contrast][aria-pressed="true"]').dataset.distanceContrast,
|
||||
});
|
||||
const layer1Filler = read();
|
||||
root.querySelector('[data-distance-layer="4"]').click();
|
||||
const layer4Filler = read();
|
||||
root.querySelector('[data-distance-contrast="demo_minus_filler"]').click();
|
||||
const layer4Demo = read();
|
||||
root.querySelector('[data-distance-scope="full_input"]').click();
|
||||
const layer4FullDemo = read();
|
||||
root.querySelector('[data-distance-mode="token_weighted"]').click();
|
||||
const layer4FullToken = read();
|
||||
root.querySelector('[data-distance-scope="target_content"]').click();
|
||||
root.querySelector('[data-distance-mode="prompt_balanced"]').click();
|
||||
root.querySelector('[data-distance-contrast="filler_minus_none"]').click();
|
||||
return { layer1Filler, layer4Filler, layer4Demo, layer4FullDemo, layer4FullToken, restored: read() };
|
||||
})()`);
|
||||
await evaluate(`(() => {
|
||||
document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
|
||||
window.scrollBy(0, -82);
|
||||
})()`);
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-distance-desktop.png");
|
||||
await evaluate(`(() => {
|
||||
document.querySelector(".distance-domain-grid").scrollIntoView({ block: "center", behavior: "instant" });
|
||||
})()`);
|
||||
await pause(120);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-distance-results-desktop.png");
|
||||
|
||||
const artifactEvidence = await evaluate(`(() => {
|
||||
const root = document.querySelector("[data-dsv2-lab]");
|
||||
root.querySelector('[data-artifact-tab="evidence"]').click();
|
||||
@@ -545,6 +603,12 @@ const mobile = await evaluate(`(() => {
|
||||
historyEffects: artifact.querySelectorAll("[data-history-effect]").length,
|
||||
historyDomainCards: artifact.querySelectorAll("[data-history-domain-grid] > article").length,
|
||||
historyDepthCells: artifact.querySelectorAll("[data-history-depth-map] > div > span").length,
|
||||
distanceLayers: artifact.querySelectorAll("[data-distance-layer]").length,
|
||||
distanceScopes: artifact.querySelectorAll("[data-distance-scope]").length,
|
||||
distanceModes: artifact.querySelectorAll("[data-distance-mode]").length,
|
||||
distanceContrasts: artifact.querySelectorAll("[data-distance-contrast]").length,
|
||||
distanceDomainCards: artifact.querySelectorAll("[data-distance-domain-grid] > article").length,
|
||||
distanceDepthCells: artifact.querySelectorAll("[data-distance-depth-map] > div > span").length,
|
||||
offenders: [...document.querySelectorAll("body *")]
|
||||
.filter((node) => !node.closest(".paper-chain, .advantage-table, .precision-table, .mapping-table, [data-deepseek-lab], [data-dsv2-lab]"))
|
||||
.filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1)
|
||||
@@ -600,8 +664,22 @@ await evaluate(`(() => {
|
||||
})()`);
|
||||
await pause(120);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-history-results-mobile.png");
|
||||
await evaluate(`(() => {
|
||||
const artifact = document.querySelector("[data-dsv2-lab]");
|
||||
artifact.querySelector('[data-artifact-tab="distance"]').click();
|
||||
artifact.scrollIntoView({ block: "start", behavior: "instant" });
|
||||
window.scrollBy(0, -70);
|
||||
})()`);
|
||||
await pause(180);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-distance-mobile.png");
|
||||
await evaluate(`(() => {
|
||||
document.querySelector(".distance-domain-grid").scrollIntoView({ block: "start", behavior: "instant" });
|
||||
window.scrollBy(0, -72);
|
||||
})()`);
|
||||
await pause(120);
|
||||
await screenshot("/tmp/llm-atlas-deepseek-distance-results-mobile.png");
|
||||
|
||||
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactCorpus, artifactTemplate, artifactHistory, artifactEvidence, home, papers, mobile, exceptions };
|
||||
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactCorpus, artifactTemplate, artifactHistory, artifactDistance, artifactEvidence, home, papers, mobile, exceptions };
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
|
||||
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
||||
@@ -611,8 +689,8 @@ if (overview.sections !== 26 || overview.tocLinks !== 26) failures.push("二十
|
||||
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
|
||||
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
|
||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||
if (overview.artifactTabs !== 8 || overview.artifactPanels !== 8 || overview.artifactLayers !== 27) failures.push("真实权重八联实验结构异常");
|
||||
if (overview.heroLabs !== "12 个可操作实验") failures.push("DeepSeek 实验总数账异常");
|
||||
if (overview.artifactTabs !== 9 || overview.artifactPanels !== 9 || overview.artifactLayers !== 27) failures.push("真实权重九联实验结构异常");
|
||||
if (overview.heroLabs !== "13 个可操作实验") failures.push("DeepSeek 实验总数账异常");
|
||||
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
|
||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
||||
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
|
||||
@@ -659,12 +737,19 @@ if (artifactHistory.layer4Interaction.domains[1].cells.map((cell) => cell.value)
|
||||
if (artifactHistory.layer4System.domains[0].effect !== "SYSTEM MAIN · Δ -0.082" || artifactHistory.layer4System.activeEffect !== "system_main" || !artifactHistory.layer4System.depthTitle.includes("System main")) failures.push("消息历史 effect 切换异常");
|
||||
if (artifactHistory.layer4FullSystem.domains[0].effect !== "SYSTEM MAIN · Δ -0.057" || artifactHistory.layer4FullSystem.activeScope !== "full_input" || !artifactHistory.layer4FullSystem.note.includes("完整输入")) failures.push("消息历史完整输入 scope 异常");
|
||||
if (artifactHistory.layer4FullToken.activeMode !== "token_weighted" || artifactHistory.restored.activeScope !== "target_content" || artifactHistory.restored.activeMode !== "prompt_balanced" || artifactHistory.restored.activeEffect !== "interaction") failures.push("消息历史聚合口径或恢复状态异常");
|
||||
if (artifactDistance.layer1Filler.panel !== "distance" || artifactDistance.layer1Filler.matrixRows !== 2 || artifactDistance.layer1Filler.matrixCells !== 6 || artifactDistance.layer1Filler.domainCards !== 4 || artifactDistance.layer1Filler.depthRows !== 4 || artifactDistance.layer1Filler.depthCells !== 24 || artifactDistance.layer1Filler.exact !== "BYTE-EXACT") failures.push("等长历史 2×3 结构或独立复跑闸门异常");
|
||||
if (artifactDistance.layer1Filler.domains[0].values.map((cell) => cell.value).join("/") !== "0.106/0.056/0.024" || artifactDistance.layer1Filler.domains[0].effect !== "FILLER − NONE · ΔTV -0.050" || artifactDistance.layer1Filler.domains[1].effect !== "FILLER − NONE · ΔTV -0.050") failures.push("L1 等长 filler 阶梯统计异常");
|
||||
if (artifactDistance.layer1Filler.summary.join("|") !== "24 / 24 ↓|24 / 24 ↓|.074 → .038 → .019|NOT PURE DISTANCE" || !artifactDistance.layer1Filler.domains[0].stability.includes("target top-6 set exact") || !artifactDistance.layer1Filler.domains[0].lexical.includes("filler↔demo TV")) failures.push("等长历史双台阶总账或逐 token 稳定性异常");
|
||||
if (artifactDistance.layer4Filler.domains[1].values.map((cell) => cell.value).join("/") !== "0.071/0.029/0.015" || artifactDistance.layer4Filler.domains[1].effect !== "FILLER − NONE · ΔTV -0.043") failures.push("L4 中文 filler 阶梯异常");
|
||||
if (artifactDistance.layer4Demo.domains[1].effect !== "DEMO − FILLER · ΔTV -0.014" || artifactDistance.layer4Demo.activeContrast !== "demo_minus_filler" || !artifactDistance.layer4Demo.depthTitle.includes("文本替换")) failures.push("等长历史文本替换 contrast 异常");
|
||||
if (artifactDistance.layer4FullDemo.domains[1].effect !== "DEMO − FILLER · ΔTV -0.023" || artifactDistance.layer4FullDemo.activeScope !== "full_input" || !artifactDistance.layer4FullDemo.note.includes("完整输入")) failures.push("等长历史完整输入 scope 异常");
|
||||
if (artifactDistance.layer4FullToken.activeMode !== "token_weighted" || artifactDistance.restored.activeScope !== "target_content" || artifactDistance.restored.activeMode !== "prompt_balanced" || artifactDistance.restored.activeContrast !== "filler_minus_none") failures.push("等长历史聚合口径或恢复状态异常");
|
||||
if (artifactEvidence.panel !== "evidence" || artifactEvidence.layers !== 27 || artifactEvidence.executed !== 7 || artifactEvidence.split !== 1 || artifactEvidence.unloaded !== 19 || artifactEvidence.exact !== "31 / 31") failures.push("真实工件执行边界或复跑闸门异常");
|
||||
if (!artifactEvidence.dependency.includes("Transformers 5.5") || !artifactEvidence.dependency.includes("4.41.2") || !artifactEvidence.boundary.includes("完整 27 层生成")) failures.push("依赖版本或未覆盖边界异常");
|
||||
if (artifactEvidence.keyboardSelected !== "load" || artifactEvidence.keyboardVisible !== "load") failures.push("真实工件实验键盘 tab 导航异常");
|
||||
if (home.releaseCards !== 17 || !home.firstRelease.includes("47 页不再压成摘要") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 DeepSeek 首发入口或论文数异常");
|
||||
if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 8 || mobile.artifactHeatCells !== 64 || mobile.corpusCohorts !== 3 || mobile.lengthDeltaCards !== 4 || mobile.templateLayers !== 6 || mobile.templateScopes !== 2 || mobile.templateModes !== 2 || mobile.templateDomainCards !== 4 || mobile.templateDepthCells !== 24 || mobile.historyLayers !== 6 || mobile.historyScopes !== 2 || mobile.historyModes !== 2 || mobile.historyEffects !== 3 || mobile.historyDomainCards !== 4 || mobile.historyDepthCells !== 24) failures.push("移动端导航或实验异常");
|
||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 9 || mobile.artifactHeatCells !== 64 || mobile.corpusCohorts !== 3 || mobile.lengthDeltaCards !== 4 || mobile.templateLayers !== 6 || mobile.templateScopes !== 2 || mobile.templateModes !== 2 || mobile.templateDomainCards !== 4 || mobile.templateDepthCells !== 24 || mobile.historyLayers !== 6 || mobile.historyScopes !== 2 || mobile.historyModes !== 2 || mobile.historyEffects !== 3 || mobile.historyDomainCards !== 4 || mobile.historyDepthCells !== 24 || mobile.distanceLayers !== 6 || mobile.distanceScopes !== 2 || mobile.distanceModes !== 2 || mobile.distanceContrasts !== 2 || mobile.distanceDomainCards !== 4 || mobile.distanceDepthCells !== 24) failures.push("移动端导航或实验异常");
|
||||
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||
|
||||
|
||||
@@ -14,6 +14,8 @@ import rawTemplate from "@/data/deepseek-v2-lite-routing-template.json";
|
||||
import rawTemplateRepro from "@/data/deepseek-v2-lite-routing-template-repro.json";
|
||||
import rawHistory from "@/data/deepseek-v2-lite-routing-history-factorial.json";
|
||||
import rawHistoryRepro from "@/data/deepseek-v2-lite-routing-history-factorial-repro.json";
|
||||
import rawDistance from "@/data/deepseek-v2-lite-routing-history-distance-control.json";
|
||||
import rawDistanceRepro from "@/data/deepseek-v2-lite-routing-history-distance-control-repro.json";
|
||||
|
||||
const trace = rawTrace as any;
|
||||
const repro = rawRepro as any;
|
||||
@@ -30,12 +32,15 @@ const template = rawTemplate as any;
|
||||
const templateRepro = rawTemplateRepro as any;
|
||||
const history = rawHistory as any;
|
||||
const historyRepro = rawHistoryRepro as any;
|
||||
const distance = rawDistance as any;
|
||||
const distanceRepro = rawDistanceRepro as any;
|
||||
const absorbExact = JSON.stringify(absorb) === JSON.stringify(absorbRepro);
|
||||
const corpusExact = JSON.stringify(corpus) === JSON.stringify(corpusRepro);
|
||||
const matched16Exact = JSON.stringify(matched16) === JSON.stringify(matched16Repro);
|
||||
const matched24Exact = JSON.stringify(matched24) === JSON.stringify(matched24Repro);
|
||||
const templateExact = JSON.stringify(template) === JSON.stringify(templateRepro);
|
||||
const historyExact = JSON.stringify(history) === JSON.stringify(historyRepro);
|
||||
const distanceExact = JSON.stringify(distance) === JSON.stringify(distanceRepro);
|
||||
const bytes = (value: number) => value >= 1024
|
||||
? `${(value / 1024).toFixed(2)} KiB`
|
||||
: `${value.toLocaleString()} B`;
|
||||
@@ -244,6 +249,86 @@ const historyCompact = {
|
||||
})),
|
||||
};
|
||||
const historyCompactJson = JSON.stringify(historyCompact).replaceAll("<", "\\u003c");
|
||||
const distanceEdges = [
|
||||
"system_none",
|
||||
"system_filler",
|
||||
"system_demo",
|
||||
"demo_vs_filler_s0",
|
||||
"demo_vs_filler_s1",
|
||||
];
|
||||
const aggregateDistanceAlignment = (layer: any, domain: string, edge: string) => {
|
||||
const rows = layer.prompts
|
||||
.filter((prompt: any) => prompt.domain === domain)
|
||||
.map((prompt: any) => prompt.alignments[edge]);
|
||||
const aligned = rows.reduce((sum: number, row: any) => sum + row.aligned_tokens, 0);
|
||||
const setExact = rows.reduce((sum: number, row: any) => sum + row.set_topk_exact, 0);
|
||||
const orderedExact = rows.reduce((sum: number, row: any) => sum + row.ordered_topk_exact, 0);
|
||||
const weightedJaccard = rows.reduce(
|
||||
(sum: number, row: any) => sum + row.mean_jaccard * row.aligned_tokens,
|
||||
0,
|
||||
);
|
||||
return {
|
||||
aligned,
|
||||
setExactRate: setExact / aligned,
|
||||
orderedExactRate: orderedExact / aligned,
|
||||
meanJaccard: weightedJaccard / aligned,
|
||||
};
|
||||
};
|
||||
const distanceCompact = {
|
||||
domains: distance.corpus_contract.domains,
|
||||
labels: distance.corpus_contract.domain_labels,
|
||||
inference: distance.inference_contract,
|
||||
messages: {
|
||||
system: distance.history_control_contract.system_message,
|
||||
demoUser: distance.history_control_contract.demo_user,
|
||||
demoAssistant: distance.history_control_contract.demo_assistant,
|
||||
fillerUser: distance.history_control_contract.filler_user,
|
||||
fillerAssistant: distance.history_control_contract.filler_assistant,
|
||||
},
|
||||
exact: distanceExact,
|
||||
layers: distance.layers.slice(1).map((layer: any) => ({
|
||||
layer: layer.layer,
|
||||
alignment: Object.fromEntries(
|
||||
distance.corpus_contract.domains.map((domain: string) => [
|
||||
domain,
|
||||
Object.fromEntries(
|
||||
distanceEdges.map((edge) => [
|
||||
edge,
|
||||
aggregateDistanceAlignment(layer, domain, edge),
|
||||
]),
|
||||
),
|
||||
]),
|
||||
),
|
||||
scopes: Object.fromEntries(
|
||||
["target_content", "full_input"].map((scope) => [
|
||||
scope,
|
||||
{
|
||||
modes: Object.fromEntries(
|
||||
["prompt_balanced", "token_weighted"].map((mode) => {
|
||||
const control = layer.statistics[scope].modes[mode].history_control;
|
||||
return [
|
||||
mode,
|
||||
Object.fromEntries(
|
||||
distance.corpus_contract.domains.map((domain: string) => [
|
||||
domain,
|
||||
{
|
||||
distances: control[domain].system_edge_distances,
|
||||
contrasts: control[domain].system_edge_distance_contrasts,
|
||||
cvEdges: control[domain].metric_system_edges.cv,
|
||||
cvContrasts: control[domain].metric_system_edge_contrasts.cv,
|
||||
lexical: control[domain].lexical_replacement,
|
||||
},
|
||||
]),
|
||||
),
|
||||
];
|
||||
}),
|
||||
),
|
||||
},
|
||||
]),
|
||||
),
|
||||
})),
|
||||
};
|
||||
const distanceCompactJson = JSON.stringify(distanceCompact).replaceAll("<", "\\u003c");
|
||||
const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoint_tensor_bytes;
|
||||
---
|
||||
|
||||
@@ -255,7 +340,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
</div>
|
||||
<p>
|
||||
固定官方 revision、tokenizer、模型代码和 BF16 第一分片；RTX 5090 连续执行 layer 0–6，
|
||||
从 3,240 次 token 显微轨迹扩到 1,736,352 次公开语料路由，并让 layer-1 权重继续走入官方吸收式 cache。
|
||||
从 3,240 次 token 显微轨迹扩到 3,112,848 次公开语料路由，并让 layer-1 权重继续走入官方吸收式 cache。
|
||||
所有结论都带证据身份与停止线。
|
||||
</p>
|
||||
</header>
|
||||
@@ -289,8 +374,11 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
<button type="button" role="tab" data-artifact-tab="history" aria-selected="false" tabindex="-1">
|
||||
<span>07</span><b>消息历史 2×2</b><small>system × one-shot</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-artifact-tab="distance" aria-selected="false" tabindex="-1">
|
||||
<span>08</span><b>等长历史控制</b><small>none → filler → demo</small>
|
||||
</button>
|
||||
<button type="button" role="tab" data-artifact-tab="evidence" aria-selected="false" tabindex="-1">
|
||||
<span>08</span><b>证据断面</b><small>revision · shards · rerun</small>
|
||||
<span>09</span><b>证据断面</b><small>revision · shards · rerun</small>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
@@ -950,6 +1038,132 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="artifact-panel" data-artifact-panel="distance" hidden>
|
||||
<div class="panel-lead">
|
||||
<div><span>X / EQUAL-LENGTH HISTORY CONTROL</span><h4>把“历史缓冲”再拆一层：可读示例并不是唯一来源</h4></div>
|
||||
<p>
|
||||
system 开 / 关分别搭配无历史、17-token 重复词元 filler 与 17-token 原 one-shot。
|
||||
filler / demo 的角色、EOS、目标位置与 batch 形状完全相同，只替换历史文本。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="history-ledger distance-ledger">
|
||||
<article><span>SOURCE PROMPTS</span><b>128</b><p>四个公开域各 32 条；同一 cohort</p></article>
|
||||
<article><span>2×3 VARIANTS</span><b>768</b><p>system 0/1 × none/filler/demo</p></article>
|
||||
<article><span>INPUT TOKENS</span><b>38,236</b><p>六格完整官方模板输入</p></article>
|
||||
<article><span>REAL ROUTES</span><b>1,376,496</b><p>六格 × 前六个 MoE 层</p></article>
|
||||
<article><span>ALIGNED TARGET</span><b>2,874 × 6</b><p>相同字符跨度与 token ID</p></article>
|
||||
<article class="exact"><span>INDEPENDENT RERUN</span><b>{distanceExact ? "BYTE-EXACT" : "MISMATCH"}</b><p>完整 JSON SHA-256 423a095d…e648e</p></article>
|
||||
</div>
|
||||
|
||||
<div class="distance-protocol" aria-label="System 与三种消息历史的二乘三控制矩阵">
|
||||
<div class="distance-row head">
|
||||
<span>HISTORY →</span>
|
||||
<b>NONE · 无历史</b>
|
||||
<b>FILLER · 重复词元</b>
|
||||
<b>DEMO · 原 one-shot</b>
|
||||
</div>
|
||||
<div class="distance-row">
|
||||
<strong>S0<br /><small>无 SYSTEM</small></strong>
|
||||
<article><span>S0 / NONE</span><div><i>USER TARGET</i><em>Assistant:</em></div><p>目标前没有已完成 turn。</p></article>
|
||||
<article><span>S0 / FILLER</span><div><b>USER · x×9</b><b>ASSISTANT · x · EOS</b><i>USER TARGET</i></div><p>相对 none 固定 +17 tokens。</p></article>
|
||||
<article><span>S0 / DEMO</span><div><u>USER · Reply with OK</u><u>ASSISTANT · OK · EOS</u><i>USER TARGET</i></div><p>同样固定 +17 tokens。</p></article>
|
||||
</div>
|
||||
<div class="distance-row">
|
||||
<strong>S1<br /><small>固定 SYSTEM</small></strong>
|
||||
<article><span>S1 / NONE</span><div><mark>SYSTEM</mark><i>USER TARGET</i><em>Assistant:</em></div><p>system 相对 S0 固定 +16 tokens。</p></article>
|
||||
<article><span>S1 / FILLER</span><div><mark>SYSTEM</mark><b>FILLER TURN · EOS</b><i>USER TARGET</i></div><p>与 S1 demo 的目标位置相同。</p></article>
|
||||
<article><span>S1 / DEMO</span><div><mark>SYSTEM</mark><u>DEMO TURN · EOS</u><i>USER TARGET</i></div><p>只把 filler 文本替换为原示例。</p></article>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="distance-controls">
|
||||
<div>
|
||||
<span>MOE LAYER</span>
|
||||
<div class="layer-switch distance-layer-switch" role="group" aria-label="选择等长历史控制层">
|
||||
{[1, 2, 3, 4, 5, 6].map((layer) => (
|
||||
<button type="button" data-distance-layer={layer} class={layer === 1 ? "active" : ""}>L{layer}</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<span>MEASUREMENT SCOPE</span>
|
||||
<div class="distance-scope-switch" role="group" aria-label="选择等长历史统计范围">
|
||||
<button type="button" data-distance-scope="target_content" aria-pressed="true">目标内容</button>
|
||||
<button type="button" data-distance-scope="full_input" aria-pressed="false">完整输入</button>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<span>AGGREGATION</span>
|
||||
<div class="distance-mode-switch" role="group" aria-label="选择等长历史聚合口径">
|
||||
<button type="button" data-distance-mode="prompt_balanced" aria-pressed="true">prompt 等权</button>
|
||||
<button type="button" data-distance-mode="token_weighted" aria-pressed="false">token 加权</button>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<span>DEPTH MAP CONTRAST</span>
|
||||
<div class="distance-contrast-switch" role="group" aria-label="选择等长历史阶梯">
|
||||
<button type="button" data-distance-contrast="filler_minus_none" aria-pressed="true">Filler − None</button>
|
||||
<button type="button" data-distance-contrast="demo_minus_filler" aria-pressed="false">Demo − Filler</button>
|
||||
</div>
|
||||
</div>
|
||||
<p data-distance-note>
|
||||
目标内容：六格只保留相同相对字符跨度与相同 token ID；TV 是两条真实 system edge 的分布距离。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="distance-domain-grid" data-distance-domain-grid></div>
|
||||
|
||||
<div class="history-buffer-summary distance-summary">
|
||||
<article><span>STRUCTURE STEP</span><b>24 / 24 ↓</b><p>none → filler 的 system-edge TV 全部下降，区间也全部低于零。</p></article>
|
||||
<article><span>TEXT STEP</span><b>24 / 24 ↓</b><p>filler → demo 再次全部下降，区间同样全部低于零。</p></article>
|
||||
<article><span>MEAN TARGET TV</span><b>.074 → .038 → .019</b><p>重复历史复现约一半缓冲；原示例再贡献一个台阶。</p></article>
|
||||
<article><span>BOUNDARY</span><b>NOT PURE DISTANCE</b><p>`x` 仍是学习过的 token；没有生成答案，也没有测能力。</p></article>
|
||||
</div>
|
||||
|
||||
<div class="distance-depth">
|
||||
<div>
|
||||
<span>DEPTH MAP / Δ SYSTEM-EDGE TV</span>
|
||||
<h5 data-distance-depth-title>Filler − None：等长历史结构的第一个台阶</h5>
|
||||
<p>绿色表示 selected history 让 system-edge TV 下降；每格都来自六格共享 source-bootstrap。</p>
|
||||
</div>
|
||||
<div data-distance-depth-map></div>
|
||||
</div>
|
||||
|
||||
<div class="distance-interpretation">
|
||||
<article>
|
||||
<span>WHAT FILLER CONTROLS</span>
|
||||
<b>角色 · EOS · 长度 · 目标位置</b>
|
||||
<p>filler 与 demo 在这些协议字段上完全相同，因此可比较固定文本替换。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>WHAT IT DOES NOT CONTROL</span>
|
||||
<b>纯距离 · token identity · repetition</b>
|
||||
<p>none → filler 仍同时加入历史与重复 token，不能命名为纯位置因果效应。</p>
|
||||
</article>
|
||||
<article>
|
||||
<span>NUMERICAL CONTRACT</span>
|
||||
<b>BF16 batch shape matters</b>
|
||||
<p>跨实验 token IDs 虽 exact，深层 gate hash 可因矩阵形状改变；正式结论只做六格组内比较。</p>
|
||||
</article>
|
||||
</div>
|
||||
|
||||
<div class="evidence-links">
|
||||
<a href="https://aclanthology.org/2022.emnlp-main.759/" rel="noreferrer">Min et al. · demonstration format ↗</a>
|
||||
<a href="https://arxiv.org/abs/2209.11895" rel="noreferrer">Induction Heads ↗</a>
|
||||
<a href="https://aclanthology.org/2024.tacl-1.9/" rel="noreferrer">Lost in the Middle ↗</a>
|
||||
<a href="https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite" rel="noreferrer">官方 V2-Lite ↗</a>
|
||||
</div>
|
||||
|
||||
<div class="artifact-boundary">
|
||||
<b>HISTORY STRUCTURE + TEXT REPLACEMENT, NOT CAPABILITY</b>
|
||||
<p>
|
||||
filler 已说明“可读正确示例”不是缓冲出现的必要条件，但 `x` 不是无语义空气；
|
||||
demo 的额外台阶也不能归因给示例正确性。下一步仍需 EOS、角色、多 filler 与行为指标控制。
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="artifact-panel" data-artifact-panel="evidence" hidden>
|
||||
<div class="panel-lead">
|
||||
<div><span>O + X / EVIDENCE SLICE</span><h4>为什么执行到 layer 6 就停，而不是把“部分下载”写成“完整复现”</h4></div>
|
||||
@@ -1045,13 +1259,16 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
<code>experiments/deepseek/v2_lite_routing_template_probe.py</code> ·
|
||||
<code>research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md</code> ·
|
||||
<code>experiments/deepseek/v2_lite_routing_history_factorial_probe.py</code> ·
|
||||
<code>research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md</code>
|
||||
<code>research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md</code> ·
|
||||
<code>experiments/deepseek/v2_lite_routing_history_distance_control.py</code> ·
|
||||
<code>research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md</code>
|
||||
</figcaption>
|
||||
|
||||
<script is:inline type="application/json" data-dsv2-trace set:html={compactJson}></script>
|
||||
<script is:inline type="application/json" data-dsv2-corpus set:html={corpusCompactJson}></script>
|
||||
<script is:inline type="application/json" data-dsv2-template set:html={templateCompactJson}></script>
|
||||
<script is:inline type="application/json" data-dsv2-history set:html={historyCompactJson}></script>
|
||||
<script is:inline type="application/json" data-dsv2-distance set:html={distanceCompactJson}></script>
|
||||
</figure>
|
||||
|
||||
<script>
|
||||
@@ -1066,16 +1283,19 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
const corpusNode = one<HTMLScriptElement>("[data-dsv2-corpus]");
|
||||
const templateNode = one<HTMLScriptElement>("[data-dsv2-template]");
|
||||
const historyNode = one<HTMLScriptElement>("[data-dsv2-history]");
|
||||
const distanceNode = one<HTMLScriptElement>("[data-dsv2-distance]");
|
||||
if (
|
||||
!payloadNode?.textContent
|
||||
|| !corpusNode?.textContent
|
||||
|| !templateNode?.textContent
|
||||
|| !historyNode?.textContent
|
||||
|| !distanceNode?.textContent
|
||||
) return;
|
||||
const data = JSON.parse(payloadNode.textContent);
|
||||
const corpusData = JSON.parse(corpusNode.textContent);
|
||||
const templateData = JSON.parse(templateNode.textContent);
|
||||
const historyData = JSON.parse(historyNode.textContent);
|
||||
const distanceData = JSON.parse(distanceNode.textContent);
|
||||
|
||||
const tabs = all<HTMLButtonElement>("[data-artifact-tab]");
|
||||
const panels = all<HTMLElement>("[data-artifact-panel]");
|
||||
@@ -1817,6 +2037,157 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
});
|
||||
});
|
||||
renderHistory();
|
||||
|
||||
let distanceLayerNumber = 1;
|
||||
let distanceScope = "target_content";
|
||||
let distanceMode = "prompt_balanced";
|
||||
let distanceContrast = "filler_minus_none";
|
||||
const distanceContrastLabels: Record<string, string> = {
|
||||
filler_minus_none: "FILLER − NONE",
|
||||
demo_minus_filler: "DEMO − FILLER",
|
||||
};
|
||||
const distanceHistoryLabels: Record<string, string> = {
|
||||
none: "NONE",
|
||||
filler: "FILLER",
|
||||
demo: "DEMO",
|
||||
};
|
||||
const renderDistance = () => {
|
||||
all<HTMLButtonElement>("[data-distance-layer]").forEach((button) => {
|
||||
button.classList.toggle(
|
||||
"active",
|
||||
Number(button.dataset.distanceLayer) === distanceLayerNumber,
|
||||
);
|
||||
});
|
||||
all<HTMLButtonElement>("[data-distance-scope]").forEach((button) => {
|
||||
button.setAttribute(
|
||||
"aria-pressed",
|
||||
String(button.dataset.distanceScope === distanceScope),
|
||||
);
|
||||
});
|
||||
all<HTMLButtonElement>("[data-distance-mode]").forEach((button) => {
|
||||
button.setAttribute(
|
||||
"aria-pressed",
|
||||
String(button.dataset.distanceMode === distanceMode),
|
||||
);
|
||||
});
|
||||
all<HTMLButtonElement>("[data-distance-contrast]").forEach((button) => {
|
||||
button.setAttribute(
|
||||
"aria-pressed",
|
||||
String(button.dataset.distanceContrast === distanceContrast),
|
||||
);
|
||||
});
|
||||
set(
|
||||
"[data-distance-note]",
|
||||
distanceScope === "target_content"
|
||||
? "目标内容：六格只保留相同相对字符跨度与相同 token ID；TV 是两条真实 system edge 的分布距离。"
|
||||
: "完整输入：system、filler/demo、角色、EOS、目标与 Assistant: 全部计入；它回答协议流量而非目标内容本身。",
|
||||
);
|
||||
|
||||
const currentLayer = distanceData.layers.find(
|
||||
(item: any) => item.layer === distanceLayerNumber,
|
||||
);
|
||||
const view = currentLayer.scopes[distanceScope].modes[distanceMode];
|
||||
const grid = one<HTMLElement>("[data-distance-domain-grid]");
|
||||
if (grid) {
|
||||
grid.replaceChildren(...distanceData.domains.map((domain: string) => {
|
||||
const data = view[domain];
|
||||
const card = document.createElement("article");
|
||||
const label = document.createElement("span");
|
||||
const ladder = document.createElement("div");
|
||||
const contrast = data.contrasts[distanceContrast].total_variation_delta;
|
||||
const primary = document.createElement("strong");
|
||||
const ci = document.createElement("p");
|
||||
const stability = document.createElement("small");
|
||||
const lexical = document.createElement("em");
|
||||
const cv = document.createElement("u");
|
||||
label.textContent = corpusLabels[domain];
|
||||
ladder.className = "distance-tv-ladder";
|
||||
["none", "filler", "demo"].forEach((history) => {
|
||||
const cell = document.createElement("i");
|
||||
const name = document.createElement("small");
|
||||
const value = document.createElement("b");
|
||||
name.textContent = distanceHistoryLabels[history];
|
||||
value.textContent = data.distances[history].total_variation.point.toFixed(3);
|
||||
cell.append(name, value);
|
||||
ladder.append(cell);
|
||||
});
|
||||
primary.textContent = `${distanceContrastLabels[distanceContrast]} · ΔTV ${signed(contrast.point)}`;
|
||||
primary.className = deltaClass(contrast.ci95);
|
||||
ci.textContent = `source-paired 95% ${formatSignedCi(contrast.ci95)}`;
|
||||
const noneAlign = currentLayer.alignment[domain].system_none;
|
||||
const fillerAlign = currentLayer.alignment[domain].system_filler;
|
||||
const demoAlign = currentLayer.alignment[domain].system_demo;
|
||||
stability.textContent = `target top-6 set exact ${(noneAlign.setExactRate * 100).toFixed(1)}% → ${(fillerAlign.setExactRate * 100).toFixed(1)}% → ${(demoAlign.setExactRate * 100).toFixed(1)}%`;
|
||||
lexical.textContent = [
|
||||
"filler↔demo TV",
|
||||
`at S0 ${data.lexical.at_s0.total_variation.point.toFixed(3)}`,
|
||||
`at S1 ${data.lexical.at_s1.total_variation.point.toFixed(3)}`,
|
||||
].join(" · ");
|
||||
cv.textContent = [
|
||||
"system ΔCV",
|
||||
...["none", "filler", "demo"].map(
|
||||
(history) => `${distanceHistoryLabels[history]} ${signed(data.cvEdges[history].point)}`,
|
||||
),
|
||||
].join(" · ");
|
||||
card.append(label, ladder, primary, ci, stability, lexical, cv);
|
||||
return card;
|
||||
}));
|
||||
}
|
||||
|
||||
const titles: Record<string, string> = {
|
||||
filler_minus_none: "Filler − None：等长历史结构的第一个台阶",
|
||||
demo_minus_filler: "Demo − Filler：固定协议字段后的文本替换台阶",
|
||||
};
|
||||
set("[data-distance-depth-title]", titles[distanceContrast]);
|
||||
const depth = one<HTMLElement>("[data-distance-depth-map]");
|
||||
if (depth) {
|
||||
depth.replaceChildren(...distanceData.domains.map((domain: string) => {
|
||||
const row = document.createElement("div");
|
||||
const label = document.createElement("b");
|
||||
label.textContent = corpusLabels[domain];
|
||||
row.append(label);
|
||||
distanceData.layers.forEach((layer: any) => {
|
||||
const contrast = layer.scopes[distanceScope].modes[distanceMode]
|
||||
[domain].contrasts[distanceContrast].total_variation_delta;
|
||||
const cell = document.createElement("span");
|
||||
cell.className = deltaClass(contrast.ci95);
|
||||
cell.style.setProperty(
|
||||
"--strength",
|
||||
String(Math.min(1, Math.abs(contrast.point) / 0.07)),
|
||||
);
|
||||
cell.textContent = `L${layer.layer} ${signed(contrast.point)}`;
|
||||
cell.title = `${corpusLabels[domain]} · L${layer.layer} · ${distanceContrastLabels[distanceContrast]} Δ system-edge TV ${signed(contrast.point)} · paired 95% ${formatSignedCi(contrast.ci95)}`;
|
||||
row.append(cell);
|
||||
});
|
||||
return row;
|
||||
}));
|
||||
}
|
||||
};
|
||||
all<HTMLButtonElement>("[data-distance-layer]").forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
distanceLayerNumber = Number(button.dataset.distanceLayer);
|
||||
renderDistance();
|
||||
});
|
||||
});
|
||||
all<HTMLButtonElement>("[data-distance-scope]").forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
distanceScope = button.dataset.distanceScope ?? "target_content";
|
||||
renderDistance();
|
||||
});
|
||||
});
|
||||
all<HTMLButtonElement>("[data-distance-mode]").forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
distanceMode = button.dataset.distanceMode ?? "prompt_balanced";
|
||||
renderDistance();
|
||||
});
|
||||
});
|
||||
all<HTMLButtonElement>("[data-distance-contrast]").forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
distanceContrast = button.dataset.distanceContrast ?? "filler_minus_none";
|
||||
renderDistance();
|
||||
});
|
||||
});
|
||||
renderDistance();
|
||||
});
|
||||
</script>
|
||||
|
||||
@@ -1879,7 +2250,12 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.history-domain-grid > :global(article > span),
|
||||
.history-buffer-summary span,
|
||||
.history-buffer span,
|
||||
.history-depth span {
|
||||
.history-depth span,
|
||||
.distance-protocol span,
|
||||
.distance-controls > div > span,
|
||||
.distance-domain-grid > :global(article > span),
|
||||
.distance-depth span,
|
||||
.distance-interpretation span {
|
||||
margin: 0;
|
||||
color: var(--blue);
|
||||
font: 700 .69rem/1.3 var(--font-mono);
|
||||
@@ -1933,14 +2309,15 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.artifact-status b { color: var(--ink); font-size: .72rem; }
|
||||
.artifact-tabs {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(8, 1fr);
|
||||
grid-template-columns: repeat(9, 1fr);
|
||||
background: var(--ink);
|
||||
}
|
||||
.artifact-tabs button {
|
||||
display: grid;
|
||||
grid-template-columns: 2rem 1fr;
|
||||
gap: .08rem .65rem;
|
||||
padding: 1rem;
|
||||
grid-template-columns: 1.45rem minmax(0, 1fr);
|
||||
gap: .08rem .45rem;
|
||||
min-width: 0;
|
||||
padding: .9rem .6rem;
|
||||
border: 0;
|
||||
border-right: 1px solid rgba(255,255,255,.13);
|
||||
background: transparent;
|
||||
@@ -1956,8 +2333,8 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
color: currentColor;
|
||||
font: 700 .7rem/1 var(--font-mono);
|
||||
}
|
||||
.artifact-tabs b { font-size: .83rem; }
|
||||
.artifact-tabs small { font: .62rem/1.2 var(--font-mono); }
|
||||
.artifact-tabs b { min-width: 0; font-size: .79rem; overflow-wrap: anywhere; }
|
||||
.artifact-tabs small { min-width: 0; font: .58rem/1.2 var(--font-mono); overflow-wrap: anywhere; }
|
||||
.artifact-panel { padding: 1.6rem; }
|
||||
.artifact-panel[hidden] { display: none; }
|
||||
.panel-lead {
|
||||
@@ -3065,6 +3442,255 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
background: rgba(186,118,44,.1);
|
||||
color: var(--ink);
|
||||
}
|
||||
.distance-protocol {
|
||||
margin-top: .8rem;
|
||||
border: 1px solid rgba(32,32,39,.15);
|
||||
overflow-x: auto;
|
||||
background: #fffdf8;
|
||||
}
|
||||
.distance-row {
|
||||
display: grid;
|
||||
grid-template-columns: 7rem repeat(3, minmax(13rem, 1fr));
|
||||
min-width: 890px;
|
||||
}
|
||||
.distance-row.head {
|
||||
background: var(--ink);
|
||||
color: white;
|
||||
}
|
||||
.distance-row.head > * {
|
||||
padding: .65rem .8rem;
|
||||
border-right: 1px solid rgba(255,255,255,.13);
|
||||
}
|
||||
.distance-row.head b { font: 650 .62rem/1.3 var(--font-mono); }
|
||||
.distance-row > strong {
|
||||
display: grid;
|
||||
align-content: center;
|
||||
padding: .8rem;
|
||||
border-right: 1px solid rgba(32,32,39,.12);
|
||||
border-bottom: 1px solid rgba(32,32,39,.12);
|
||||
background: #e5ded2;
|
||||
color: var(--blue);
|
||||
font: 750 .8rem/1.4 var(--font-mono);
|
||||
}
|
||||
.distance-row > strong small {
|
||||
color: rgba(32,32,39,.55);
|
||||
font: .56rem/1.35 var(--font-mono);
|
||||
}
|
||||
.distance-row article {
|
||||
padding: .8rem;
|
||||
border-right: 1px solid rgba(32,32,39,.12);
|
||||
border-bottom: 1px solid rgba(32,32,39,.12);
|
||||
}
|
||||
.distance-row article > div {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: .2rem;
|
||||
margin-top: .55rem;
|
||||
}
|
||||
.distance-row article > div > * {
|
||||
padding: .34rem .4rem;
|
||||
border: 1px solid rgba(32,32,39,.13);
|
||||
font: 650 .53rem/1 var(--font-mono);
|
||||
font-style: normal;
|
||||
text-decoration: none;
|
||||
}
|
||||
.distance-row article mark { background: rgba(161,77,77,.1); color: var(--red); }
|
||||
.distance-row article b { background: rgba(186,118,44,.1); color: var(--amber); }
|
||||
.distance-row article u { background: rgba(98,105,155,.12); color: var(--blue); }
|
||||
.distance-row article i { background: rgba(57,120,110,.11); color: var(--teal); }
|
||||
.distance-row article em { background: rgba(98,105,155,.13); color: var(--blue); }
|
||||
.distance-row article p {
|
||||
margin: .5rem 0 0;
|
||||
color: rgba(32,32,39,.56);
|
||||
font-size: .61rem;
|
||||
line-height: 1.4;
|
||||
}
|
||||
.distance-controls {
|
||||
display: grid;
|
||||
grid-template-columns: auto .9fr .9fr 1.45fr;
|
||||
gap: .8rem;
|
||||
align-items: end;
|
||||
margin-top: .8rem;
|
||||
padding: .85rem;
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
background: #e8e2d7;
|
||||
}
|
||||
.distance-controls > div { display: grid; gap: .45rem; }
|
||||
.distance-controls .layer-switch { margin: 0; }
|
||||
.distance-scope-switch,
|
||||
.distance-mode-switch,
|
||||
.distance-contrast-switch { display: flex; }
|
||||
.distance-scope-switch button,
|
||||
.distance-mode-switch button,
|
||||
.distance-contrast-switch button {
|
||||
padding: .58rem .66rem;
|
||||
border: 1px solid rgba(32,32,39,.22);
|
||||
background: #fffdf8;
|
||||
color: var(--ink);
|
||||
font: 650 .6rem/1 var(--font-mono);
|
||||
cursor: pointer;
|
||||
}
|
||||
.distance-scope-switch button + button,
|
||||
.distance-mode-switch button + button,
|
||||
.distance-contrast-switch button + button { border-left: 0; }
|
||||
.distance-scope-switch button[aria-pressed="true"],
|
||||
.distance-mode-switch button[aria-pressed="true"],
|
||||
.distance-contrast-switch button[aria-pressed="true"] {
|
||||
border-color: var(--blue);
|
||||
background: var(--blue);
|
||||
color: white;
|
||||
}
|
||||
.distance-controls > p {
|
||||
grid-column: 1 / -1;
|
||||
margin: 0;
|
||||
padding-top: .75rem;
|
||||
border-top: 1px solid rgba(32,32,39,.12);
|
||||
color: rgba(32,32,39,.62);
|
||||
font-size: .69rem;
|
||||
line-height: 1.5;
|
||||
}
|
||||
.distance-domain-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
margin-top: .8rem;
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
background: #fffdf8;
|
||||
}
|
||||
.distance-domain-grid > :global(article) {
|
||||
padding: .85rem;
|
||||
border-right: 1px solid rgba(32,32,39,.12);
|
||||
}
|
||||
.distance-domain-grid > :global(article:last-child) { border-right: 0; }
|
||||
.distance-domain-grid :global(.distance-tv-ladder) {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
gap: .22rem;
|
||||
margin-top: .55rem;
|
||||
}
|
||||
.distance-domain-grid :global(.distance-tv-ladder > i) {
|
||||
display: grid;
|
||||
gap: .18rem;
|
||||
padding: .38rem;
|
||||
background: #e8e2d7;
|
||||
font-style: normal;
|
||||
}
|
||||
.distance-domain-grid :global(.distance-tv-ladder small) {
|
||||
color: rgba(32,32,39,.5);
|
||||
font: 650 .49rem/1 var(--font-mono);
|
||||
}
|
||||
.distance-domain-grid :global(.distance-tv-ladder b) {
|
||||
font: 720 .66rem/1 var(--font-mono);
|
||||
}
|
||||
.distance-domain-grid > :global(article > strong) {
|
||||
display: inline-block;
|
||||
margin-top: .48rem;
|
||||
padding: .26rem .38rem;
|
||||
font: 750 .62rem/1 var(--font-mono);
|
||||
}
|
||||
.distance-domain-grid > :global(article > strong.down),
|
||||
.distance-depth :global(span.down) {
|
||||
background: rgba(57,120,110,.13);
|
||||
color: var(--teal);
|
||||
}
|
||||
.distance-domain-grid > :global(article > strong.up),
|
||||
.distance-depth :global(span.up) {
|
||||
background: rgba(161,77,77,.12);
|
||||
color: var(--red);
|
||||
}
|
||||
.distance-domain-grid > :global(article > strong.neutral),
|
||||
.distance-depth :global(span.neutral) {
|
||||
background: rgba(186,118,44,.12);
|
||||
color: var(--amber);
|
||||
}
|
||||
.distance-domain-grid > :global(article > p),
|
||||
.distance-domain-grid > :global(article > small),
|
||||
.distance-domain-grid > :global(article > em),
|
||||
.distance-domain-grid > :global(article > u) {
|
||||
display: block;
|
||||
margin: .38rem 0 0;
|
||||
color: rgba(32,32,39,.57);
|
||||
font: .55rem/1.4 var(--font-mono);
|
||||
font-style: normal;
|
||||
text-decoration: none;
|
||||
}
|
||||
.distance-domain-grid > :global(article > em),
|
||||
.distance-domain-grid > :global(article > u) {
|
||||
padding-top: .35rem;
|
||||
border-top: 1px solid rgba(32,32,39,.1);
|
||||
}
|
||||
.distance-depth {
|
||||
display: grid;
|
||||
grid-template-columns: .52fr 1.48fr;
|
||||
gap: 1rem;
|
||||
margin-top: .8rem;
|
||||
padding: 1rem;
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
}
|
||||
.distance-depth h5 {
|
||||
margin: .4rem 0;
|
||||
font: 720 1rem/1.15 var(--font-display);
|
||||
}
|
||||
.distance-depth p {
|
||||
margin: 0;
|
||||
color: rgba(32,32,39,.58);
|
||||
font-size: .66rem;
|
||||
line-height: 1.5;
|
||||
}
|
||||
.distance-depth > :global([data-distance-depth-map]) {
|
||||
display: grid;
|
||||
gap: .35rem;
|
||||
}
|
||||
.distance-depth :global([data-distance-depth-map] > div) {
|
||||
display: grid;
|
||||
grid-template-columns: 5.5rem repeat(6, 1fr);
|
||||
gap: .25rem;
|
||||
}
|
||||
.distance-depth :global([data-distance-depth-map] > div > b),
|
||||
.distance-depth :global([data-distance-depth-map] > div > span) {
|
||||
display: grid;
|
||||
align-items: center;
|
||||
min-height: 2.2rem;
|
||||
padding: .35rem;
|
||||
font: 650 .55rem/1.2 var(--font-mono);
|
||||
}
|
||||
.distance-depth :global([data-distance-depth-map] > div > b) {
|
||||
color: var(--blue);
|
||||
}
|
||||
.distance-depth :global([data-distance-depth-map] > div > span.down) {
|
||||
background: color-mix(in srgb, var(--teal) calc(var(--strength) * 55%), #eef0e9);
|
||||
color: var(--ink);
|
||||
}
|
||||
.distance-depth :global([data-distance-depth-map] > div > span.up) {
|
||||
background: color-mix(in srgb, var(--red) calc(var(--strength) * 48%), #f3ebe6);
|
||||
color: var(--ink);
|
||||
}
|
||||
.distance-depth :global([data-distance-depth-map] > div > span.neutral) {
|
||||
background: rgba(186,118,44,.1);
|
||||
color: var(--ink);
|
||||
}
|
||||
.distance-interpretation {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
margin-top: .8rem;
|
||||
border: 1px solid rgba(32,32,39,.14);
|
||||
background: #e8e2d7;
|
||||
}
|
||||
.distance-interpretation article {
|
||||
padding: .9rem;
|
||||
border-right: 1px solid rgba(32,32,39,.12);
|
||||
}
|
||||
.distance-interpretation article:last-child { border-right: 0; }
|
||||
.distance-interpretation b {
|
||||
display: block;
|
||||
margin-top: .4rem;
|
||||
font: 730 .78rem/1.2 var(--font-display);
|
||||
}
|
||||
.distance-interpretation p {
|
||||
margin: .4rem 0 0;
|
||||
color: rgba(32,32,39,.58);
|
||||
font-size: .62rem;
|
||||
line-height: 1.45;
|
||||
}
|
||||
.observed-cache {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr auto 1.25fr;
|
||||
@@ -3306,7 +3932,9 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.template-depth,
|
||||
.history-controls,
|
||||
.history-buffer,
|
||||
.history-depth { grid-template-columns: 1fr; }
|
||||
.history-depth,
|
||||
.distance-controls,
|
||||
.distance-depth { grid-template-columns: 1fr; }
|
||||
.artifact-status { grid-template-columns: 1fr 1fr; }
|
||||
.artifact-tabs { grid-template-columns: 1fr 1fr; }
|
||||
.route-controls { grid-template-columns: 1fr 1fr; }
|
||||
@@ -3322,7 +3950,10 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.template-domain-grid { grid-template-columns: 1fr 1fr; }
|
||||
.history-ledger { grid-template-columns: repeat(3, 1fr); }
|
||||
.history-domain-grid,
|
||||
.history-buffer-summary { grid-template-columns: 1fr 1fr; }
|
||||
.history-buffer-summary,
|
||||
.distance-domain-grid { grid-template-columns: 1fr 1fr; }
|
||||
.distance-interpretation { grid-template-columns: 1fr; }
|
||||
.distance-interpretation article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
|
||||
.template-protocol > i { transform: rotate(90deg); justify-self: center; }
|
||||
.length-delta-grid { grid-template-columns: 1fr 1fr; }
|
||||
.corpus-heat-head p { text-align: left; }
|
||||
@@ -3361,7 +3992,9 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.template-negative-control,
|
||||
.history-ledger,
|
||||
.history-domain-grid,
|
||||
.history-buffer-summary { grid-template-columns: 1fr; }
|
||||
.history-buffer-summary,
|
||||
.distance-domain-grid,
|
||||
.distance-interpretation { grid-template-columns: 1fr; }
|
||||
.route-metrics article,
|
||||
.cache-ratio article,
|
||||
.load-lessons article,
|
||||
@@ -3375,21 +4008,29 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|
||||
.template-negative-control article,
|
||||
.history-ledger article,
|
||||
.history-domain-grid > :global(article),
|
||||
.history-buffer-summary article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
|
||||
.history-buffer-summary article,
|
||||
.distance-domain-grid > :global(article),
|
||||
.distance-interpretation article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
|
||||
.corpus-mode-switch,
|
||||
.corpus-cohort-switch,
|
||||
.template-scope-switch,
|
||||
.template-mode-switch,
|
||||
.history-scope-switch,
|
||||
.history-mode-switch,
|
||||
.history-effect-switch { display: grid; grid-template-columns: 1fr; }
|
||||
.history-effect-switch,
|
||||
.distance-scope-switch,
|
||||
.distance-mode-switch,
|
||||
.distance-contrast-switch { display: grid; grid-template-columns: 1fr; }
|
||||
.corpus-mode-switch button + button,
|
||||
.corpus-cohort-switch button + button,
|
||||
.template-scope-switch button + button,
|
||||
.template-mode-switch button + button,
|
||||
.history-scope-switch button + button,
|
||||
.history-mode-switch button + button,
|
||||
.history-effect-switch button + button { border-left: 1px solid rgba(32,32,39,.22); border-top: 0; }
|
||||
.history-effect-switch button + button,
|
||||
.distance-scope-switch button + button,
|
||||
.distance-mode-switch button + button,
|
||||
.distance-contrast-switch button + button { border-left: 1px solid rgba(32,32,39,.22); border-top: 0; }
|
||||
.length-delta-grid > :global(article),
|
||||
.length-pair-summary article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.11); }
|
||||
.artifact-boundary { grid-template-columns: 1fr; }
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -37,7 +37,7 @@ const toc = [
|
||||
|
||||
<BaseLayout
|
||||
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
|
||||
description="用二十四张问题账、十次技术转向、十二个交互实验、真实 V2-Lite 权重、公开语料路由区间、官方模板与消息历史因子实验、吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||
description="用二十四张问题账、十次技术转向、十三个交互实验、真实 V2-Lite 权重、公开语料路由区间、官方模板、消息历史因子与等长 filler 控制、吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||
section="deepseek"
|
||||
>
|
||||
<header class="page-hero deepseek-hero">
|
||||
@@ -55,7 +55,7 @@ const toc = [
|
||||
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
||||
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
|
||||
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
|
||||
<div><dt>LABS</dt><dd>12 个可操作实验</dd></div>
|
||||
<div><dt>LABS</dt><dd>13 个可操作实验</dd></div>
|
||||
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
|
||||
<div><dt>STATUS</dt><dd>三轮 · 真实权重执行</dd></div>
|
||||
</dl>
|
||||
@@ -768,15 +768,15 @@ const toc = [
|
||||
<p class="eyebrow"><span>22</span> OFFICIAL WEIGHTS / EXECUTED</p>
|
||||
<h2>从“MLA 与 MoE 的概念”再往前一步：让官方 V2-Lite 权重真的跑起来</h2>
|
||||
<p class="lede">
|
||||
前面的四联实验负责建立公式与角色合同；下面的八联工件实验固定官方 revision、tokenizer、
|
||||
前面的四联实验负责建立公式与角色合同；下面的九联工件实验固定官方 revision、tokenizer、
|
||||
模型代码和 checkpoint 第一分片，在 RTX 5090 上连续执行 layer 0–6。它把真实观测、shape 推导、
|
||||
吸收式 latent cache、长度对照、官方 chat-template 扰动、实现差距和未覆盖范围放在同一张证据图里。
|
||||
</p>
|
||||
<div class="artifact-callout">
|
||||
<article><span>X / FORWARD</span><b>7 / 27 layers</b><p>1 个 dense 层 + 6 个 MoE 层；layer 7 因跨分片停止。</p></article>
|
||||
<article><span>X / ROUTES</span><b>1,736,352</b><p>三档长度、raw/user/generation 与 system × one-shot 四格的真实 top-6 选择。</p></article>
|
||||
<article><span>X / ROUTES</span><b>3,112,848</b><p>三档长度、模板、system × one-shot 与 none/filler/demo 六格的真实 top-6 选择。</p></article>
|
||||
<article><span>X / ABSORB CACHE</span><b>266,240 → 29,952 B</b><p>同一真实 layer-1 权重的 naive / absorb active buffers。</p></article>
|
||||
<article><span>X / RERUN</span><b>5 / 5 EXACT</b><p>三档长度、官方模板与消息历史因子 trace 均 byte-exact；比较使用 paired prompt bootstrap。</p></article>
|
||||
<article><span>X / RERUN</span><b>6 / 6 EXACT</b><p>三档长度、官方模板、历史因子与等长 filler 控制均 byte-exact；比较使用 paired prompt bootstrap。</p></article>
|
||||
</div>
|
||||
<DeepSeekArtifactLab />
|
||||
</section>
|
||||
|
||||
@@ -15,7 +15,7 @@ const workstreams = [
|
||||
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
|
||||
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
|
||||
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
|
||||
{ label: "DeepSeek 专题", value: 94, next: "SM90 FlashMLA kernel、完整 27 层、词元边界 / 距离 / EOS / 角色正交控制、FP8/pipeline 与 R1-like RL 复现" },
|
||||
{ label: "DeepSeek 专题", value: 95, next: "SM90 FlashMLA kernel、完整 27 层、EOS / 角色 / 多 filler / 内容与 batch-shape 控制、FP8/pipeline 与 R1-like RL 复现" },
|
||||
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
||||
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
||||
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
||||
@@ -50,7 +50,7 @@ const workstreams = [
|
||||
<div><dt>OVERALL</dt><dd>专题平均 {average}%</dd></div>
|
||||
<div><dt>READABLE</dt><dd>{published} 个首版可读专题</dd></div>
|
||||
<div><dt>ACTIVE</dt><dd>{researching} 个研究/写作中</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 18:10 CST</dd></div>
|
||||
<div><dt>UPDATED</dt><dd>2026-07-29 18:55 CST</dd></div>
|
||||
<div><dt>MODE</dt><dd>持续迭代，不锁死版本</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
@@ -97,12 +97,12 @@ const workstreams = [
|
||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||
<article><span>✓</span><h3>七十九个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与八联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>八十个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与九联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
|
||||
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、官方 V3 absorb、长度/模板对照与 system × one-shot 2×2；累计 1,736,352 次真实路由，模板共享前缀 21,852 / 21,852 exact，历史缓冲 TV 在 24 / 24 格下降，五份运行结果均 byte-exact 独立复跑。</p></article>
|
||||
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、官方 V3 absorb、长度/模板、system × one-shot 与等长 filler 控制；累计 3,112,848 次真实路由，none→filler→demo 的两个 TV 台阶均在 24 / 24 格下降，六份运行结果均 byte-exact 独立复跑。</p></article>
|
||||
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
|
||||
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
|
||||
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
|
||||
@@ -134,7 +134,7 @@ const workstreams = [
|
||||
<div class="queue-table">
|
||||
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
||||
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
|
||||
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>SM90 FlashMLA kernel / 完整 27 层 / 词元边界、距离、EOS、角色与示例内容正交控制 → FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>SM90 FlashMLA kernel / 完整 27 层 / EOS、角色、多 filler、示例内容与 batch shape 控制 → FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
|
||||
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
|
||||
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
|
||||
|
||||
Reference in New Issue
Block a user