feat: isolate DeepSeek role marker head token

This commit is contained in:
wuyang
2026-07-29 20:44:43 +08:00
parent 8ef2d33437
commit ea49ed0511
15 changed files with 3676450 additions and 33 deletions
+12 -3
View File
@@ -14,7 +14,7 @@
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
| DeepSeek 专题 | 三轮实证进行中 | 96% | 角色标记与 special-token family、V2-Lite-Chat 行为、完整 27 层与固定 batch-shape 对照,再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL 复现 |
| DeepSeek 专题 | 三轮实证进行中 | 97% | special-token family 与完整两-token 角色块、V2-Lite-Chat 行为、完整 27 层与固定 batch-content 对照,再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL 复现 |
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
@@ -41,7 +41,7 @@
- [x] 完成 486 篇关键论文索引,覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
- [x] 完成可检索、可按专题筛选的论文库页面。
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十四联实验、长上下文、MoE 路由、推理三页签,以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十一个原创交互视图。
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十五联实验、长上下文、MoE 路由、推理三页签,以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十二个原创交互视图。
- [x] 完成长上下文首版:五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文,并建立长上下文研究账本。
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文,并建立 MoE 研究账本。
@@ -217,11 +217,17 @@
- [x] 八格正式运行与独立复跑各 54,254,445 bytes,SHA-256 均为 `9bb93834…b9c37` 且 byte-exact;确定性 compact 生成器把前端载荷降至约 1.2 MiB,同时在产物内钉住两份完整来源 hash。
- [x] 历史边界控制本地闸门通过:70 个 Astro 文件零诊断、21 个页面、1,151 个站内引用、12 个跨页锚点零失败;十页签桌面与 390px 移动端浏览器回归通过,无运行时异常或文档级横向溢出。
- [x] DeepSeek 历史边界单 token 控制以源提交 `f4d4a05`、不可变镜像 `20260729T120248Z-f4d4a05` 发布;OCI digest `sha256:438ca51d…d0cf4`,复用 NAS `12010→8080`、NPM host 31 / cert 41、门户 `LLM ATLAS / projects / 180`,十六套生产 Chrome 回归全通过;保留 `20260729T110301Z-8130e37` 回滚。
- [x] 角色词头单 token 控制沿用同一 128-source / 23-content-token cohort,在 system 0/1 × official / target `User→Assistant` / target `User→x` / suffix `Assistant→User` 八格中固定官方 EOS、冒号、长度、目标位置、mask 与 32-row batch shape;768 / 768 个反事实序列恰好只改一个 input ID。
- [x] RTX 5090 执行 1,024 个输入变体、56,784 个输入 token,新增 2,044,224 次真实 top-6 路由,使公开语料累计达到 7,201,296 次;前置 U→A / U→x direct target TV 均约 `.02–.03`,但 system-edge contrast 在 24 格分别为 12↑12↓ / 13↑11↓,不支持统一调制方向。
- [x] 后置 generation head `Assistant→User` 通过严格 causal negative control:34,488 / 34,488 个目标 ordered top-6 exact,目标 route hash 1,536 / 1,536、integer load 1,536 / 1,536 exact,direct TV / JSD / ΔCV 全为零;完整输入仍能看到 suffix 自身差异。
- [x] 八格正式运行与独立复跑各 53,440,884 bytes,SHA-256 均为 `9dc0e37f…b8caf` 且 byte-exact;确定性 compact 前端载荷为 1,238,687 bytes,并钉住两份完整来源 hash。
- [x] 跨实验 BF16 batch-content 审计:当前 official 与上一轮 EOS 条件 256 / 256 token-ID hashes exact,Layer 1 target route hash 256 / 256 exact,但 L2–L6 随 companion rows 改变而分化;正式结论只使用本轮八格组内比较,不把相同 batch shape 误写成相同数值执行。
- [x] 角色词头里程碑本地闸门通过:71 个 Astro 文件零诊断、21 个页面、1,151 个站内引用、12 个跨页锚点零失败;十一页签桌面与 390px 移动端无运行时异常或文档级横向溢出,十六套真实 Chrome 回归全部通过。
## 正在进行
- [ ] K3 三轮下一闸门:获得真实 token hidden states、expert load 与 cache traces,解释或修订 `A_log [128]` 工件冲突,再做 Figure 3/4/5 数值重绘和独立小模型复现。
- [ ] DeepSeek 三轮下一闸门:拆分 `User:` / `Assistant:` 角色标记与 special-token family,加入 V2-Lite-Chat 生成/行为对照;扩到完整 27 层并固定 batch shape 审计,再推进 SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
- [ ] DeepSeek 三轮下一闸门:控制 special-token family 与完整 `User:` / `Assistant:` 两-token 角色块,加入 V2-Lite-Chat 生成/行为对照;扩到完整 27 层并固定 batch content 审计,再推进 SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
- [ ] 表示、位置与残差二轮:真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
- [ ] 评测安全二轮:真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
- [ ] 推理服务二轮:真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
@@ -368,6 +374,9 @@
| 2026-07-29 | 边界控制主结论只使用精确对齐目标内容 TV | x/句点/换行相对 EOS 的 system edge 更大,但 direct interaction、CV 与完整输入方向更混合;不命名为路由更优、回合理解或能力变化 |
| 2026-07-29 | base checkpoint 与 Chat 行为永久分证据层 | 当前 V2-Lite base 虽使用官方 tokenizer/template 构造协议,却没有 Chat/SFT 行为身份;后续必须另跑 Chat checkpoint 与生成/任务指标 |
| 2026-07-29 | DeepSeek 历史边界控制以 `20260729T120248Z-f4d4a05` 发布 | OCI digest `sha256:438ca51d…d0cf4`;复用 NAS 12010→8080、NPM 31 / cert 41、门户 order 180;十六套生产 Chrome 回归通过,保留上一不可变镜像回滚 |
| 2026-07-29 | 角色词头效应分为 direct replacement 与 system-edge modulation | `User→Assistant/x` 都会直接改变后续路由,但 system-edge contrast 方向混合;不把“有 direct effect”偷换成“统一强化/削弱 system” |
| 2026-07-29 | 目标后的角色词头承担严格 causal negative control | suffix `Assistant→User` 在 34,488 个目标 ordered top-6 上全 exact,目标 TV/JSD/ΔCV 为零;完整输入保留 suffix 自身差异 |
| 2026-07-29 | BF16 复现合同从 batch shape 扩展到 batch content | official token IDs 与 32-row shape 相同仍不足以保证深层 route hash 相同;companion rows 与数值执行路径也进入边界说明 |
| 2026-07-29 | K3 二轮按 32 张对象账与完整报告顺序重建 | total/active、2.5×、KDA state、深度来源、专家路由、视觉目标、轨迹、缓存与评测协议不再压成一页组件摘要 |
| 2026-07-29 | K3 原生视觉事实回到 §2.4 / §3.3 核验 | 删除“先冻结语言模型再解冻”旧表述;明确 MoonViT-V2 从头训练,视觉/文本从开始共同 NTP |
| 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项;作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 |
+10 -4
View File
@@ -19,7 +19,7 @@
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读,
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化,以及评测与安全深度专题,
以及 81 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
以及 82 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
8 个交互实验和 100 个一手/官方节点,完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
第三轮已完成开放工件与首个真实 kernel 里程碑:固定官方模型与 FlashKDA revisions,审计 96 个 checkpoint shards、
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计,并用 4 个新视图
@@ -28,7 +28,7 @@
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
DeepSeek 三轮专题以 24 张问题账、10 次技术转向、
14 个交互实验和 60 个一手/官方节点,串起 Dense、MoE、MLA、V3 协同、R1 与 V4;
15 个交互实验和 60 个一手/官方节点,串起 Dense、MoE、MLA、V3 协同、R1 与 V4;
并固定官方 V2-Lite revision,在 RTX 5090 上连续执行 7/27 层,记录 3,240 次真实专家选择、
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑;进一步用真实 layer-1 权重执行官方 V3
naive/absorb 路径,实际写入 576 元素 latent cache,并以 FP32 将两种结合顺序的最大误差压到
@@ -51,7 +51,12 @@ TV 在 24 / 24 个 layer×domain 中都下降,均值从 `0.073` 降至 `0.019`
同目标位置且相对官方序列恰好只改一个 ID。目标内容的平均 system-edge TV 为
`.0374 / .0539 / .0553 / .0492`(EOS / x / 句点 / 换行);X 与句点在 24 / 24 格高于
EOS,换行为 23 / 24,但 base checkpoint、非法反事实序列和无行为指标的边界被明确保留。
当前累计 5,157,072 次公开语料路由。FlashMLA 的
再往前一步只替换目标 `User:` 或生成 `Assistant:` 的第一个普通 token ID,新增
2,044,224 次真实路由:前置 `User→Assistant` / `User→x` 的 direct target TV 均约
`.02–.03`,说明一个词头 ID 足以条件化后续专家集合;但二者对 system-edge 的 24 格方向
分别为 12↑12↓ / 13↑11↓,不能写成统一的 system 调制。后置 `Assistant→User`
则在 34,488 / 34,488 个目标 ordered top-6 上 exact,direct TV / JSD / ΔCV 全为零,
形成严格 causal suffix 负对照。当前累计 7,201,296 次公开语料路由。FlashMLA 的
SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
@@ -60,7 +65,8 @@ SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
[DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md](./research/DEEPSEEK_ROUTING_TEMPLATE_AUDIT.md)、
[DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md)、
[DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md) 与
[DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md)。
[DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md) 与
[DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md](./research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md)。
其余专题按进度账本持续扩建。
## 本地开发
+54
View File
@@ -371,3 +371,57 @@ See `research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md` for all paired
intervals, per-token route alignment, scope split, BF16 batch-shape audit,
primary literature, and the boundary between an input-ID intervention and a
chat-turn semantic claim.
## Role-marker-head single-token control
`v2_lite_routing_role_marker_head_control.py` keeps the official filler
history, EOS, colon, target span, generation prompt, mask, length, and 32-row
batch shape, while changing one ordinary token ID:
```text
system off/on × official / pre-target User→Assistant /
pre-target User→x / post-target Assistant→User
```
The pinned tokenizer maps `User`, `Assistant`, `:`, and `x` to IDs `5726`,
`77398`, `25`, and `87`. The pre-target controls identify the effect of the
first role-marker token only; the colon remains. The post-target replacement
is a causal suffix negative control. None of the three counterfactuals is a
valid official chat serialization.
```bash
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
python -B experiments/deepseek/v2_lite_routing_role_marker_head_control.py \
--artifact-dir /path/to/deepseek-v2-lite \
--human-eval /path/to/HumanEval.jsonl.gz \
--gsm8k /path/to/gsm8k/test.jsonl \
--tnews /path/to/tnews/test.json \
--tnews-archive /path/to/tnews_public.zip \
--wikitext /path/to/wikitext-validation.parquet \
--output src/data/deepseek-v2-lite-routing-role-marker-head-control.json \
--per-domain 32 \
--content-tokens 23 \
--batch-prompts 4 \
--layers 7 \
--bootstrap 2000 \
--seed 20260729 \
--captured-at 2026-07-29T12:30:00+00:00
```
The eight cells add 2,044,224 real top-6 route selections. All 256
source×system groups preserve length and target position; every one of the
768 counterfactuals differs from its official sequence at exactly one ID. The
committed run and independent rerun are byte-exact:
```text
9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf
```
For exact target content, direct replacement TV is about `.02–.03`, but the
system-edge contrast has mixed direction: `12↑12↓` for User→Assistant and
`13↑11↓` for User→x. The post-target control is exact for all 34,488 aligned
target-token ordered top-6 routes, with zero target TV, JSD, and ΔCV. See
`research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md` for all paired intervals,
depth maps, token-level alignment, cross-experiment BF16 batch-content audit,
primary sources, and the boundary against full role semantics or Chat-model
behavior.
@@ -43,6 +43,7 @@ FILLER_USER = prior.FILLER_USER
FILLER_ASSISTANT = prior.FILLER_ASSISTANT
BOUNDARY_LEVELS = ("eos", "x", "period", "newline")
REFERENCE_LEVEL = "eos"
BOUNDARY_TEXT = {
"x": "x",
"period": ".",
@@ -297,7 +298,7 @@ def boundary_control_domain(
metric_system_edges: dict[str, Any] = {}
metric_system_edge_contrasts: dict[str, Any] = {}
for metric in point_metrics["s0_eos"]:
for metric in point_metrics[f"s0_{REFERENCE_LEVEL}"]:
point_edges = system_edges(
{
condition: point_metrics[condition][metric]
@@ -398,7 +399,7 @@ def boundary_control_domain(
direct_substitutions = {}
for boundary in BOUNDARY_LEVELS:
if boundary == "eos":
if boundary == REFERENCE_LEVEL:
continue
direct_substitutions[boundary] = {}
direct_point_tv: dict[int, float] = {}
@@ -406,7 +407,7 @@ def boundary_control_domain(
direct_point_jsd: dict[int, float] = {}
direct_boot_jsd: dict[int, np.ndarray] = {}
for system in (0, 1):
before = f"s{system}_eos"
before = f"s{system}_{REFERENCE_LEVEL}"
after = f"s{system}_{boundary}"
point_delta = point[after] - point[before]
tv_boot = 0.5 * np.abs(
@@ -0,0 +1,573 @@
#!/usr/bin/env python3
"""Run a 2 x 4 role-marker-head control on DeepSeek-V2-Lite.
Every condition keeps the same repeated-token user/assistant history, the
official assistant EOS boundary, and the same target user content. The system
factor is off/on. The second factor is one of four role-head conditions:
official: target ``User`` and suffix ``Assistant`` stay official
target_assistant: target ``User`` -> ``Assistant`` (one pre-target ID)
target_x: target ``User`` -> ``x`` (one pre-target ID)
suffix_user: suffix ``Assistant`` -> ``User`` (one post-target ID)
The colon token, sequence length, target position, attention mask, official
EOS, and padded batch shape are held fixed. The post-target suffix edit is a
causal-mask negative control: it must not alter earlier target-content routes.
All three edited sequences are explicit token-ID counterfactuals rather than
valid official chat serializations.
The official BF16 forward path and shared source-bootstrap statistics are
reused from the audited boundary-token runner. This file owns the renderer,
role-head contract, result schema, and causal-suffix validation.
"""
from __future__ import annotations
import contextlib
import hashlib
import json
import os
import sys
from pathlib import Path
from typing import Any
import v2_lite_routing_history_boundary_token_control as prior
base = prior.base
SYSTEM_MESSAGE = prior.SYSTEM_MESSAGE
FILLER_USER = prior.FILLER_USER
FILLER_ASSISTANT = prior.FILLER_ASSISTANT
ROLE_LEVELS = (
"official",
"target_assistant",
"target_x",
"suffix_user",
)
REFERENCE_LEVEL = "official"
CONDITIONS = tuple(
f"s{system}_{level}"
for level in ROLE_LEVELS
for system in (0, 1)
)
FACTORS = {
condition: {
"system": int(condition[1]),
"history": "filler",
"assistant_boundary": "official_eos",
"role_head": condition.split("_", 1)[1],
}
for condition in CONDITIONS
}
SYSTEM_CELLS = {
level: (f"s0_{level}", f"s1_{level}")
for level in ROLE_LEVELS
}
SYSTEM_EDGE_CONTRASTS = {
f"{level}_minus_official": (REFERENCE_LEVEL, level)
for level in ROLE_LEVELS
if level != REFERENCE_LEVEL
}
COMPARISONS = tuple(
[
(f"system_{level}", f"s0_{level}", f"s1_{level}")
for level in ROLE_LEVELS
]
+ [
(
f"{level}_at_s{system}",
f"s{system}_{REFERENCE_LEVEL}",
f"s{system}_{level}",
)
for level in ROLE_LEVELS
if level != REFERENCE_LEVEL
for system in (0, 1)
]
)
ALIGNMENT_COMPARISONS = COMPARISONS
RENDER_AUDIT: list[dict[str, Any]] = []
ROLE_TOKEN_IDS: dict[str, int] = {}
def condition_messages(
content: str,
condition: str,
) -> list[dict[str, str]]:
factors = FACTORS[condition]
messages: list[dict[str, str]] = []
if factors["system"]:
messages.append({"role": "system", "content": SYSTEM_MESSAGE})
messages.extend(
[
{"role": "user", "content": FILLER_USER},
{"role": "assistant", "content": FILLER_ASSISTANT},
{"role": "user", "content": content},
]
)
return messages
def role_token_ids(tokenizer: Any) -> dict[str, int]:
texts = {
"user": "User",
"assistant": "Assistant",
"x": "x",
"colon": ":",
"eos": tokenizer.eos_token,
}
ids: dict[str, int] = {}
for name, text in texts.items():
encoded = list(
tokenizer(text, add_special_tokens=False).input_ids
)
if len(encoded) != 1:
raise RuntimeError(
f"{name} role control is not one token: {encoded}"
)
ids[name] = int(encoded[0])
if ids["eos"] != tokenizer.eos_token_id:
raise RuntimeError("encoded EOS differs from tokenizer.eos_token_id")
if any(
ids[name] in tokenizer.all_special_ids
for name in ("user", "assistant", "x", "colon")
):
raise RuntimeError("ordinary role-head control is unexpectedly special")
if len({ids["user"], ids["assistant"], ids["x"]}) != 3:
raise RuntimeError(f"role-head token IDs are not distinct: {ids}")
return ids
def render_role_variant(
tokenizer: Any,
content: str,
condition: str,
) -> dict[str, Any]:
messages = condition_messages(content, condition)
rendered = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
official_ids = list(
tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
)
)
token_ids, offsets = base.tokenize_with_offsets(
tokenizer,
rendered,
add_special_tokens=False,
)
if token_ids != official_ids:
raise RuntimeError(
f"{condition} rendered IDs differ from apply_chat_template"
)
ids = role_token_ids(tokenizer)
ROLE_TOKEN_IDS.update(ids)
content_start = rendered.rfind(content)
if content_start < 0:
raise RuntimeError(f"{condition} target content is absent")
content_end = content_start + len(content)
positions, records, crossing = base.content_positions(
token_ids,
offsets,
content_start,
content_end,
)
if not positions:
raise RuntimeError(f"{condition} has no target-content tokens")
target_first = min(positions)
target_last = max(positions)
target_user_candidates = [
index
for index, token_id in enumerate(official_ids)
if token_id == ids["user"] and index < target_first
]
if len(target_user_candidates) < 2:
raise RuntimeError(
f"{condition} cannot locate filler and target User heads"
)
target_user_position = max(target_user_candidates)
suffix_assistant_candidates = [
index
for index, token_id in enumerate(official_ids)
if token_id == ids["assistant"] and index > target_last
]
if len(suffix_assistant_candidates) != 1:
raise RuntimeError(
f"{condition} expected one suffix Assistant head; "
f"got {suffix_assistant_candidates}"
)
suffix_assistant_position = suffix_assistant_candidates[0]
if official_ids[target_user_position + 1] != ids["colon"]:
raise RuntimeError("target User head is not followed by colon")
if official_ids[suffix_assistant_position + 1] != ids["colon"]:
raise RuntimeError("suffix Assistant head is not followed by colon")
if ids["eos"] not in official_ids[:target_user_position]:
raise RuntimeError("official history EOS is absent before target role")
level = FACTORS[condition]["role_head"]
edit_position: int | None = None
before_id: int | None = None
after_id: int | None = None
edit_side = "none"
if level == "target_assistant":
edit_position = target_user_position
before_id = ids["user"]
after_id = ids["assistant"]
edit_side = "pre_target"
elif level == "target_x":
edit_position = target_user_position
before_id = ids["user"]
after_id = ids["x"]
edit_side = "pre_target"
elif level == "suffix_user":
edit_position = suffix_assistant_position
before_id = ids["assistant"]
after_id = ids["user"]
edit_side = "post_target"
elif level != REFERENCE_LEVEL:
raise RuntimeError(f"unknown role-head level: {level}")
token_ids = list(token_ids)
if edit_position is not None:
if token_ids[edit_position] != before_id:
raise RuntimeError(
f"{condition} edit source ID differs at {edit_position}"
)
token_ids[edit_position] = int(after_id)
differing_ids = sum(
left != right
for left, right in zip(official_ids, token_ids, strict=True)
)
expected_differences = 0 if level == REFERENCE_LEVEL else 1
if differing_ids != expected_differences:
raise RuntimeError(
f"{condition} changed {differing_ids} IDs; "
f"expected {expected_differences}"
)
if edit_side == "pre_target" and edit_position >= target_first:
raise RuntimeError("pre-target edit is not before target content")
if edit_side == "post_target" and edit_position <= target_last:
raise RuntimeError("post-target edit is not after target content")
decoded = tokenizer.decode(
token_ids,
skip_special_tokens=False,
clean_up_tokenization_spaces=False,
)
RENDER_AUDIT.append(
{
"content_sha256": base.text_sha256(content),
"condition": condition,
"role_head": level,
"edit_side": edit_side,
"edit_position": edit_position,
"before_token_id": before_id,
"after_token_id": after_id,
"target_user_position": target_user_position,
"suffix_assistant_position": suffix_assistant_position,
"official_eos_token_id": ids["eos"],
"input_tokens": len(token_ids),
"target_first_position": target_first,
"target_last_position": target_last,
"changed_token_ids_vs_official": differing_ids,
"counterfactual_decoded_sha256": base.text_sha256(decoded),
}
)
return {
"condition": condition,
"messages_sha256": base.canonical_hash(messages),
"rendered_sha256": base.text_sha256(rendered),
"token_ids": token_ids,
"tokens": len(token_ids),
"token_ids_sha256": base.canonical_hash(token_ids),
"content_positions": positions,
"content_records": records,
"content_tokens": len(positions),
"wrapper_tokens": len(token_ids) - len(positions),
"boundary_crossing_tokens": crossing,
}
def layer_statistics(
prompt_rows: list[dict[str, Any]],
replicates: int,
seed: int,
layer_index: int,
) -> dict[str, Any]:
scopes = prior.layer_statistics(
prompt_rows,
replicates,
seed,
layer_index,
)
for scope in scopes.values():
for mode in scope["modes"].values():
mode["role_marker_control"] = mode.pop("boundary_control")
return scopes
def install_control_contract() -> None:
"""Install the eight-cell role renderer and shared statistics."""
prior.BOUNDARY_LEVELS = ROLE_LEVELS
prior.REFERENCE_LEVEL = REFERENCE_LEVEL
prior.CONDITIONS = CONDITIONS
prior.FACTORS = FACTORS
prior.SYSTEM_CELLS = SYSTEM_CELLS
prior.SYSTEM_EDGE_CONTRASTS = SYSTEM_EDGE_CONTRASTS
prior.COMPARISONS = COMPARISONS
prior.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
base.CONDITIONS = CONDITIONS
base.FACTORS = FACTORS
base.COMPARISONS = COMPARISONS
base.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
base.condition_messages = condition_messages
base.render_variant = render_role_variant
base.layer_statistics = layer_statistics
def output_path_from_argv() -> Path:
try:
return Path(sys.argv[sys.argv.index("--output") + 1])
except (ValueError, IndexError) as error:
raise ValueError("--output is required") from error
def render_contract_summary() -> dict[str, Any]:
if not RENDER_AUDIT:
raise RuntimeError("render audit is empty")
by_content: dict[str, list[dict[str, Any]]] = {}
for row in RENDER_AUDIT:
by_content.setdefault(row["content_sha256"], []).append(row)
if len(by_content) != 128 and "--per-domain" not in sys.argv:
raise RuntimeError(
f"expected 128 selected contents; observed {len(by_content)}"
)
equal_lengths = 0
equal_target_positions = 0
official_exact = 0
one_id_replacements = 0
pre_target_edits = 0
post_target_edits = 0
for rows in by_content.values():
for system in (0, 1):
cells = [
row
for row in rows
if FACTORS[row["condition"]]["system"] == system
]
if len(cells) != len(ROLE_LEVELS):
raise RuntimeError(
"render audit lacks one or more role-head cells"
)
if len({row["input_tokens"] for row in cells}) == 1:
equal_lengths += 1
target_spans = {
(
row["target_first_position"],
row["target_last_position"],
)
for row in cells
}
if len(target_spans) == 1:
equal_target_positions += 1
official_exact += sum(
row["role_head"] == REFERENCE_LEVEL
and row["changed_token_ids_vs_official"] == 0
for row in cells
)
one_id_replacements += sum(
row["role_head"] != REFERENCE_LEVEL
and row["changed_token_ids_vs_official"] == 1
for row in cells
)
pre_target_edits += sum(
row["edit_side"] == "pre_target"
for row in cells
)
post_target_edits += sum(
row["edit_side"] == "post_target"
for row in cells
)
groups = 2 * len(by_content)
replacements = 3 * groups
return {
"selected_contents": len(by_content),
"system_groups": groups,
"equal_input_length_groups": equal_lengths,
"equal_target_position_groups": equal_target_positions,
"official_cells_exact": official_exact,
"one_id_counterfactual_cells_exact": one_id_replacements,
"expected_one_id_counterfactual_cells": replacements,
"pre_target_one_id_edits": pre_target_edits,
"post_target_one_id_edits": post_target_edits,
"all_group_lengths_equal": equal_lengths == groups,
"all_target_positions_equal": equal_target_positions == groups,
"all_counterfactuals_change_exactly_one_id": (
one_id_replacements == replacements
),
}
def finalize_result(path: Path) -> dict[str, Any]:
result = json.loads(path.read_text(encoding="utf-8"))
result["schema_version"] = 3
result["evidence_identity"] = (
"X / official BF16 weights and tokenizer; official role heads plus "
"paired one-token pre-target and post-target role-head "
"counterfactuals on local truncated forward"
)
boundary = result["boundary"]
boundary.pop("factorial_claim", None)
boundary.update(
{
"role_marker_head_control": True,
"official_serialization_by_role_head": {
"official": True,
"target_assistant": False,
"target_x": False,
"suffix_user": False,
},
"single_input_id_intervention": True,
"official_assistant_eos_held_fixed": True,
"colon_token_held_fixed": True,
"target_position_held_fixed": True,
"causal_suffix_negative_control": True,
"complete_role_semantics_identified": False,
"task_performance": False,
"causal_boundary": (
"official-to-control comparisons intervene on exactly one "
"role-head input ID in this fixed forward contract; the "
"colon and all other protocol tokens remain, the edited "
"sequences are not official chats, and no answer quality "
"is measured"
),
}
)
old_contract = result.pop("message_history_contract")
render_validation = render_contract_summary()
result["role_marker_head_contract"] = {
"chat_template_revision": base.MODEL_REVISION,
"chat_template": old_contract["chat_template"],
"chat_template_sha256": old_contract["chat_template_sha256"],
"official_sequence": (
"Assistant: {filler} + eos_token + User: {target} + "
"generation-prompt Assistant:"
),
"system_message": SYSTEM_MESSAGE,
"system_message_sha256": base.text_sha256(SYSTEM_MESSAGE),
"filler_user": FILLER_USER,
"filler_user_sha256": base.text_sha256(FILLER_USER),
"filler_assistant": FILLER_ASSISTANT,
"filler_assistant_sha256": base.text_sha256(FILLER_ASSISTANT),
"role_levels": list(ROLE_LEVELS),
"role_token_ids": ROLE_TOKEN_IDS,
"conditions": FACTORS,
"comparisons": [
{"name": name, "before": before, "after": after}
for name, before, after in COMPARISONS
],
"system_edge_contrasts": {
name: {
"before_role_head": before,
"after_role_head": after,
"definition": (
f"system edge at {after} minus system edge at {before}"
),
}
for name, (before, after) in (
SYSTEM_EDGE_CONTRASTS.items()
)
},
"render_validation": render_validation,
"target_role": "user",
"add_generation_prompt": True,
"scope_split": {
"full_input": (
"all official or counterfactually edited BOS, system/history, "
"target, newline, and generation-prompt token IDs"
),
"target_content": (
"exact intersection of (relative character span, token ID) "
"inside target user content across all eight conditions"
),
},
}
inference = result["inference_contract"]
inference["batch_grouping"] = (
"all eight role-head variants of one source prompt execute in the "
"same right-padded batch"
)
statistical = result["statistical_contract"]
statistical["paired_indices"] = (
"one sampled source-prompt index matrix is reused across all eight "
"cells for every role-head contrast within each "
"domain/layer/scope/mode"
)
statistical.pop("interaction_distribution_magnitude", None)
statistical["system_edge_contrast_distribution_magnitude"] = (
"0.5 * L1 norm of the signed difference between two system-edge "
"expert-share vectors; this is not labeled standard TV"
)
path.write_text(
json.dumps(result, indent=2, ensure_ascii=False) + "\n",
encoding="utf-8",
)
return result
def main() -> None:
install_control_contract()
if {"-h", "--help"} & set(sys.argv[1:]):
base.main()
return
output = output_path_from_argv()
with open(os.devnull, "w", encoding="utf-8") as sink:
with contextlib.redirect_stdout(sink):
base.main()
result = finalize_result(output)
payload = output.read_bytes()
print(
json.dumps(
{
"output": str(output),
"sha256": hashlib.sha256(payload).hexdigest(),
"bytes": len(payload),
"source_prompts": result["inference_contract"][
"total_source_prompts"
],
"prompt_variants": result["inference_contract"][
"total_prompt_variants"
],
"input_tokens_by_condition": result[
"inference_contract"
]["input_tokens_by_condition"],
"total_routes": result["inference_contract"][
"total_routes_all_conditions_all_moe_layers"
],
"render_validation": result[
"role_marker_head_contract"
]["render_validation"],
},
indent=2,
ensure_ascii=False,
)
)
if __name__ == "__main__":
main()
+1
View File
@@ -10,6 +10,7 @@
"preview": "astro preview --host 0.0.0.0",
"check": "astro check",
"build:data:deepseek-boundary": "node scripts/build-deepseek-boundary-compact.mjs",
"build:data:deepseek-role": "node scripts/build-deepseek-role-marker-compact.mjs",
"check:site": "node scripts/check-site.mjs",
"check:moe-browser": "node scripts/check-moe-browser.mjs",
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
@@ -0,0 +1,648 @@
# DeepSeek-V2-Lite 角色词头单 Token 控制审计
> 状态:真实官方权重执行(X)<br />
> 模型:`deepseek-ai/DeepSeek-V2-Lite` **base checkpoint**<br />
> revision:`604d5664dddd88a0433dbae533b7fe9472482de0`<br />
> 执行边界:layer 0–6;观测 MoE layer 1–6<br />
> 样本:WikiText-2 / TNEWS / HumanEval / GSM8K 各 32 条<br />
> 正式运行与独立复跑:byte-exact<br />
> 完整 JSON SHA-256:`9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf`
## 0. 一句话先说结论
在固定 DeepSeek-V2-Lite base 权重、固定 128 个公开样本、固定重复词元历史、
固定官方 assistant EOS、固定冒号、固定长度、固定目标位置和固定八格 batch 时:
> 只把目标内容前面的 `User` 词头换成 `Assistant` 或 `x`,会让后续目标
> token 的专家路由发生清晰的直接变化;但是它没有在 24 个 layer×domain
> 中统一放大或削弱 system message 的路由作用。
目标内容、prompt-balanced 口径下,官方序列与两个前置干预之间的直接 TV 为:
```text
system off system on
User → Assistant .0222 .0218
User → x .0263 .0253
```
而四种角色词头条件下,system-edge TV 的 24 格平均为:
```text
官方 User: .03729
目标 User→Assistant .03705
目标 User→x .03754
后置 Assistant→User .03729
```
最后一行不是“小数恰好接近”,而是严格的 causal suffix 负对照:
> 把目标内容后 generation prompt 的 `Assistant` 换成 `User`,在六个 MoE
> 层、两个 system 水平的 34,488 个目标 layer×token 对上 ordered top-6
> 全部相同;目标 load、TV、JSD 与 system edge 也全部精确不变。
最准确的命名是:
> **固定协议中的角色词头单 input-ID 条件化。**
它不是“完整角色语义探针”,更不是 Chat 模型能力实验。
---
## 1. 为什么要从 EOS 继续拆到角色词头
上一轮已经固定重复词元历史、角色包装、长度和目标位置,只把历史 assistant
后的 EOS 换成三个普通单 token。结果说明 EOS identity 不是普通占位符。
但官方序列仍然是:
```text
Assistant: x <EOS> User: TARGET
```
所以即便 EOS 被替换,下一条 `User:` 仍然提供回合入口线索。上一轮不能回答:
1. `User:` 中的角色词头是否会条件化后续目标计算;
2. 这种变化是角色词头的直接作用,还是对 system message 效应的统一调制;
3. 同一个单 ID 干预如果发生在目标之后,目标路由是否按 causal mask 保持不变。
本轮把这三个问题放进同一个八格 batch。
---
## 2. 官方模板中的 `User:` 不是一个 token
固定 revision 的官方模板仍然是:
```jinja
{% if message['role'] == 'user' %}
{{ 'User: ' + message['content'] + '\n\n' }}
{% elif message['role'] == 'assistant' %}
{{ 'Assistant: ' + message['content'] + eos_token }}
{% elif message['role'] == 'system' %}
{{ message['content'] + '\n\n' }}
{% endif %}
```
固定 tokenizer 的真实 tokenization 是:
| 文本 | token IDs | token 数 | special |
|---|---|---:|---:|
| `User` | `[5726]` | 1 | 否 |
| `Assistant` | `[77398]` | 1 | 否 |
| `User:` | `[5726, 25]` | 2 | 否 |
| `Assistant:` | `[77398, 25]` | 2 | 否 |
| `x` | `[87]` | 1 | 否 |
| `<EOS>` | `[100001]` | 1 | 是 |
因此本实验只干预 `User:` / `Assistant:` 的**词头 ID**,冒号 ID `25`
始终保留。
这一区分很重要:
> `User → Assistant` 不是把完整 `User:` 换成完整 `Assistant:` 的字符串实验,
> 而是把两个 token 序列中的第一个 ID 替换掉。
官方模板可在
[固定 tokenizer_config](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite/blob/604d5664dddd88a0433dbae533b7fe9472482de0/tokenizer_config.json)
直接核查。
---
## 3. 四个角色词头水平
两个因子是:
- `S`:system 关闭 / 开启;
- `R`:角色词头控制。
四个 `R` 水平如下:
| 水平 | 单 ID 改动 | 相对目标 | 官方合法序列 |
|---|---|---|---:|
| `official` | 无 | — | 是 |
| `target_assistant` | 目标前 `User 5726 → Assistant 77398` | 前 | 否 |
| `target_x` | 目标前 `User 5726 → x 87` | 前 | 否 |
| `suffix_user` | 目标后 `Assistant 77398 → User 5726` | 后 | 否 |
四种序列可以直观看成:
```text
OFFICIAL
... Assistant: x <EOS> User: TARGET \n\n Assistant:
TARGET_ASSISTANT
... Assistant: x <EOS> Assistant: TARGET \n\n Assistant:
TARGET_X
... Assistant: x <EOS> x: TARGET \n\n Assistant:
SUFFIX_USER
... Assistant: x <EOS> User: TARGET \n\n User:
```
三个干预都在官方 tokenization 完成后修改 ID,因此它们是明确的
counterfactual token sequences,不是官方 `apply_chat_template` 会自然生成的
合法聊天。
---
## 4. 固定了什么
每个 source 的八格共同固定:
- 同一 DeepSeek-V2-Lite base checkpoint;
- 同一 system 文本;
- 同一 repeated-token user / assistant 历史;
- 同一官方 assistant EOS;
- 同一目标文本;
- 同一 canonical 23-token 截断合同;
- 同一冒号 token;
- 同一序列长度;
- 同一目标绝对位置;
- 同一 attention mask;
- 同一 right-padding;
- 同一次 32-row forward 的 batch shape。
正式 renderer 验证:
| 检查 | 结果 |
|---|---:|
| source prompts | 128 |
| system groups | 256 |
| 四格同长度 | 256 / 256 |
| 四格同目标 span | 256 / 256 |
| 官方序列零 ID 改动 | 256 / 256 |
| 三种反事实恰好一 ID 改动 | 768 / 768 |
| 前置单 ID 干预 | 512 |
| 后置单 ID 干预 | 256 |
---
## 5. 为什么必须放一个目标后的角色词头
如果只比较:
```text
User: TARGET
Assistant: TARGET
```
即使看到差异,也只能知道“前面改一个 ID 会影响后面”,却不知道:
- renderer 是否错误对齐目标;
- batch 中的其他条件是否让任意单 ID 改动都污染所有位置;
- causal mask 是否按预期工作;
- route 对齐代码是否把 suffix token 错算进目标。
因此加入:
```text
TARGET ... Assistant:
TARGET ... User:
```
改动发生在所有目标内容 token 之后。对 decoder-only causal mask 而言,过去
位置不能看见未来位置,所以目标路由应精确不变。
这不是统计上的“希望接近零”,而是一条确定性执行闸门。
---
## 6. 统计量仍然分三张账
### 6.1 直接替换距离
在 system 水平 `s` 下:
```text
TV(P(s, official), P(s, role_control))
```
回答:
> 只改这个角色词头,目标内容的聚合专家份额移动了多少?
### 6.2 system edge
角色水平 `r` 下:
```text
TV(P(S0, r), P(S1, r))
```
回答:
> 固定角色词头时,system 开关让目标路由移动了多少?
### 6.3 system-edge contrast
```text
TV_system(role_control) - TV_system(official)
```
回答:
> 替换角色词头是否改变 system effect 的**强度**?
直接替换距离非零,不自动推出 system-edge contrast 也应同方向。
---
## 7. 先看直接作用:前置角色词头确实改变后续路由
目标内容、prompt-balanced 聚合:
| 干预 | S0 平均 direct TV | S1 平均 direct TV | S1−S0 |
|---|---:|---:|---:|
| `User→Assistant` | .022243 | .021804 | -.000439 |
| `User→x` | .026305 | .025339 | -.000966 |
| 后置 `Assistant→User` | 0 | 0 | 0 |
两个前置词头替换都产生约 `.02–.03` 的直接 TV;但 S1−S0 的均值接近零。
逐层平均与逐 token top-6 set exact:
| 层 | U→A direct TV S0 / S1 | U→A set exact S0 / S1 | U→x direct TV S0 / S1 | U→x set exact S0 / S1 | suffix direct TV |
|---:|---:|---:|---:|---:|---:|
| L1 | 0.0267 / 0.0253 | 79.9% / 81.8% | 0.0275 / 0.0278 | 78.2% / 80.5% | 0 / 0 |
| L2 | 0.0187 / 0.0175 | 82.0% / 83.1% | 0.0214 / 0.0240 | 80.2% / 78.8% | 0 / 0 |
| L3 | 0.0175 / 0.0191 | 81.7% / 80.3% | 0.0235 / 0.0231 | 79.5% / 76.7% | 0 / 0 |
| L4 | 0.0209 / 0.0238 | 76.3% / 74.3% | 0.0260 / 0.0249 | 72.2% / 71.4% | 0 / 0 |
| L5 | 0.0229 / 0.0218 | 71.4% / 73.7% | 0.0266 / 0.0251 | 68.3% / 70.4% | 0 / 0 |
| L6 | 0.0267 / 0.0233 | 70.0% / 70.5% | 0.0329 / 0.0270 | 66.4% / 68.4% | 0 / 0 |
随着层数增加,直接替换后的 set exact 整体下降,说明一个前置 ID 的差异会沿
残差流累积;但这仍只是 route-set 稳定性,不是专家语义或任务质量。
---
## 8. 关键反例:system-edge 强度没有统一方向
目标内容、prompt-balanced 口径,24 个 layer×domain 的平均:
```text
official .037290
target_assistant .037045
target_x .037542
suffix_user .037290
```
相对官方的 contrast:
| contrast | 平均 ΔTV | 点估计正 / 负 | CI 负 / 跨零 / 正 |
|---|---:|---:|---:|
| `target_assistant − official` | -.000246 | 12 / 12 | 2 / 19 / 3 |
| `target_x − official` | +.000252 | 13 / 11 | 3 / 19 / 2 |
| `suffix_user − official` | 0 | 0 / 0 | 0 / 24 / 0 |
因此不允许写:
- “Assistant 词头让 system 更强”;
- “x 会屏蔽 system”;
- “角色标记统一提高路由稳定性”。
数据支持的是:
> 前置词头会直接改变后续路由,但它与 system message 的组合关系依赖层与域。
---
## 9. 全部 24 格
下表只使用目标内容、prompt-balanced 与共享 source bootstrap:
| 层 | 域 | 官方 TV | User→Assistant TV | Δ / 95% CI | User→x TV | Δ / 95% CI | suffix TV |
|---:|---|---:|---:|---|---:|---|---:|
| L1 | 英文 | 0.0559 | 0.0585 | +0.0026 / [-0.0033, +0.0068] | 0.0625 | +0.0066 / [-0.0002, +0.0117] | 0.0559 |
| L1 | 中文 | 0.0276 | 0.0251 | -0.0025 / [-0.0054, +0.0016] | 0.0245 | -0.0032 / [-0.0070, +0.0007] | 0.0276 |
| L1 | 代码 | 0.0571 | 0.0574 | +0.0003 / [-0.0027, +0.0059] | 0.0553 | -0.0018 / [-0.0054, +0.0027] | 0.0571 |
| L1 | 数学 | 0.0784 | 0.0769 | -0.0015 / [-0.0070, +0.0045] | 0.0889 | +0.0105 / [+0.0060, +0.0157] | 0.0784 |
| L2 | 英文 | 0.0293 | 0.0298 | +0.0004 / [-0.0036, +0.0049] | 0.0321 | +0.0028 / [-0.0008, +0.0066] | 0.0293 |
| L2 | 中文 | 0.0204 | 0.0188 | -0.0016 / [-0.0045, +0.0025] | 0.0199 | -0.0005 / [-0.0042, +0.0038] | 0.0204 |
| L2 | 代码 | 0.0331 | 0.0377 | +0.0046 / [+0.0005, +0.0082] | 0.0375 | +0.0043 / [+0.0002, +0.0084] | 0.0331 |
| L2 | 数学 | 0.0401 | 0.0364 | -0.0037 / [-0.0090, +0.0016] | 0.0346 | -0.0056 / [-0.0103, -0.0007] | 0.0401 |
| L3 | 英文 | 0.0312 | 0.0360 | +0.0047 / [-0.0009, +0.0076] | 0.0328 | +0.0016 / [-0.0043, +0.0060] | 0.0312 |
| L3 | 中文 | 0.0224 | 0.0188 | -0.0036 / [-0.0075, +0.0011] | 0.0234 | +0.0009 / [-0.0036, +0.0061] | 0.0224 |
| L3 | 代码 | 0.0411 | 0.0464 | +0.0053 / [+0.0012, +0.0098] | 0.0443 | +0.0032 / [-0.0021, +0.0070] | 0.0411 |
| L3 | 数学 | 0.0337 | 0.0343 | +0.0006 / [-0.0041, +0.0050] | 0.0309 | -0.0028 / [-0.0057, +0.0024] | 0.0337 |
| L4 | 英文 | 0.0326 | 0.0326 | -0.0000 / [-0.0057, +0.0052] | 0.0360 | +0.0033 / [-0.0021, +0.0088] | 0.0326 |
| L4 | 中文 | 0.0293 | 0.0250 | -0.0043 / [-0.0085, +0.0013] | 0.0281 | -0.0011 / [-0.0053, +0.0041] | 0.0293 |
| L4 | 代码 | 0.0390 | 0.0356 | -0.0034 / [-0.0079, +0.0030] | 0.0322 | -0.0069 / [-0.0107, -0.0015] | 0.0390 |
| L4 | 数学 | 0.0513 | 0.0449 | -0.0064 / [-0.0127, -0.0012] | 0.0494 | -0.0019 / [-0.0086, +0.0049] | 0.0513 |
| L5 | 英文 | 0.0335 | 0.0366 | +0.0031 / [-0.0036, +0.0083] | 0.0354 | +0.0019 / [-0.0038, +0.0083] | 0.0335 |
| L5 | 中文 | 0.0244 | 0.0254 | +0.0009 / [-0.0036, +0.0062] | 0.0231 | -0.0014 / [-0.0059, +0.0038] | 0.0244 |
| L5 | 代码 | 0.0386 | 0.0283 | -0.0104 / [-0.0144, -0.0036] | 0.0252 | -0.0135 / [-0.0173, -0.0061] | 0.0386 |
| L5 | 数学 | 0.0459 | 0.0424 | -0.0035 / [-0.0081, +0.0015] | 0.0472 | +0.0013 / [-0.0053, +0.0073] | 0.0459 |
| L6 | 英文 | 0.0303 | 0.0321 | +0.0019 / [-0.0038, +0.0083] | 0.0298 | -0.0005 / [-0.0073, +0.0058] | 0.0303 |
| L6 | 中文 | 0.0242 | 0.0281 | +0.0039 / [-0.0020, +0.0091] | 0.0245 | +0.0002 / [-0.0047, +0.0066] | 0.0242 |
| L6 | 代码 | 0.0316 | 0.0395 | +0.0080 / [+0.0031, +0.0127] | 0.0357 | +0.0042 / [-0.0020, +0.0107] | 0.0316 |
| L6 | 数学 | 0.0436 | 0.0424 | -0.0013 / [-0.0067, +0.0047] | 0.0479 | +0.0043 / [-0.0022, +0.0096] | 0.0436 |
仅 5 / 24 个 `User→Assistant` 区间、5 / 24 个 `User→x` 区间完全位于零
的一侧;正负方向同时存在。
---
## 10. causal suffix 负对照的完整账
目标后的 `Assistant→User`:
| 对象 | exact |
|---|---:|
| target ordered top-6 layer×token×system | 34,488 / 34,488 |
| target set top-6 layer×token×system | 34,488 / 34,488 |
| target route hash prompt×layer×system | 1,536 / 1,536 |
| target integer load prompt×layer×system | 1,536 / 1,536 |
| target direct TV | 0 |
| target direct JSD | 0 |
| target system-edge contrast | 0 |
| target ΔCV contrast | 0 |
完整输入则不同:
| 对象 | exact |
|---|---:|
| full route hash | 0 / 1,536 |
| full integer load | 6 / 1,536 |
| mean direct TV at S0 | .012134 |
| mean direct TV at S1 | .008710 |
这正是合理分账:
- 目标内容位于 edit 之前,所以不可见;
- suffix 自身和其后的冒号位于 edit 之后,所以完整输入可变。
---
## 11. 为什么完整输入不能代替目标内容
完整输入的 mean system-edge TV:
```text
official .138929
target_assistant .139118
target_x .140953
suffix_user .139329
```
其中 `target_x − official` 平均为 `+.002024`,19 / 24 点估计为正;
`suffix_user − official` 也不为零。
但完整输入混入:
- 被替换的角色词头;
- 历史包装;
- 目标前后的换行;
- generation prompt;
- suffix 本身。
所以它回答的是:
> 整条协议流量如何改变?
而不是:
> 相同目标内容如何被上下文条件化?
网站必须允许 scope 切换,但主结论只能来自 exact target content。
---
## 12. ΔCV 与 direct interaction 都不支持统一方向
目标内容、prompt-balanced:
| 对象 | 平均 | 点估计正 / 负 | CI 负 / 跨零 / 正 |
|---|---:|---:|---:|
| U→A 的 system-edge ΔCV | +.003138 | 16 / 8 | 4 / 16 / 4 |
| U→x 的 system-edge ΔCV | +.003260 | 15 / 9 | 4 / 15 / 5 |
| U→A direct TV 的 S1−S0 | -.000439 | 11 / 13 | 3 / 20 / 1 |
| U→x direct TV 的 S1−S0 | -.000966 | 10 / 14 | 5 / 17 / 2 |
| suffix 全部上述量 | 0 | 0 / 0 | 0 / 24 / 0 |
因此不写“角色词头让路由更均衡”或“system 会放大角色作用”。
---
## 13. 同 batch shape 仍不等于同 BF16 执行
本轮的官方条件与上一轮 EOS 条件:
- 256 / 256 messages hash 相同;
- 256 / 256 rendered text hash 相同;
- 256 / 256 token-ID hash 相同;
- 256 / 256 长度与目标 token 合同相同;
- batch 都是 32 rows。
但同行的其他反事实从:
```text
EOS / x / period / newline
```
变成:
```text
official / target_assistant / target_x / suffix_user
```
跨实验 official route exact:
| 层 | full hash | target hash | full load | target load |
|---:|---:|---:|---:|---:|
| L1 | 256 / 256 | 256 / 256 | 256 / 256 | 256 / 256 |
| L2 | 60 / 256 | 180 / 256 | 158 / 256 | 208 / 256 |
| L3 | 43 / 256 | 149 / 256 | 99 / 256 | 188 / 256 |
| L4 | 27 / 256 | 102 / 256 | 74 / 256 | 149 / 256 |
| L5 | 13 / 256 | 72 / 256 | 52 / 256 | 124 / 256 |
| L6 | 16 / 256 | 73 / 256 | 49 / 256 | 129 / 256 |
观察到的事实是:
> 第一 MoE 层完全相同;经过不同 companion rows 的 MoE 计算后,深层 BF16
> route hashes 开始分化。
一种合理但尚未直接验证的工程解释是:其他 rows 改变 expert grouping / microbatch
形状或低精度 kernel 数值路径,细小误差再通过深层 gate 放大。当前证据不能唯一定位
是哪一个 kernel。
因此正式结论继续限定为:
> **同一次八格 batch 内的配对对比。**
“相同 batch shape”不足以支持跨实验 route-byte equality;batch 内容也是执行合同。
---
## 14. 与一手文献如何连接
### 14.1 Hugging Face chat templates
[官方文档](https://huggingface.co/docs/transformers/en/chat_templating)明确把
messages 转换为模型实际读取的 token sequence,并强调不同 chat model 使用不同控制
token / 格式。
它支持:
- 必须检查最终 token IDs;
- 不能把统一的 `{role, content}` API 当成统一模型输入。
它不支持:
- DeepSeek 的 `User` ID 必然有跨模型通用语义。
### 14.2 LIMA
[LIMA](https://arxiv.org/abs/2305.11206)在监督数据中用 EOT 标记说话人边界,
说明 speaker / turn serialization 是 alignment 数据合同的一部分。
它支持研究问题的重要性,但不证明本 base checkpoint 的某个 route TV 是 EOT 语义。
### 14.3 URIAL
[The Unlocking Spell on Base LLMs / URIAL](https://proceedings.iclr.cc/paper_files/paper/2024/file/6bcbb4a501dbad0eba1b660c1a55318c-Paper-Conference.pdf)
展示 system prompt、风格化示例与固定格式可以显著改变 base model 的生成行为。
它支持:
- base model 也可能受结构化上下文条件化;
- “base ≠ 完全不响应模板”。
它不支持:
- 把本实验的路由差异命名为 alignment;
- 推出 DeepSeek-V2-Lite-Chat 的行为。
### 14.4 Prompt template 与安全/能力
[Keeping LLMs Aligned After Fine-tuning](https://proceedings.neurips.cc/paper_files/paper/2024/file/d6f034bb216b472fc7d32ec7aff20342-Paper-Conference.pdf)
在多个 chat model 上显示 fine-tuning / inference prompt template 会影响安全行为。
[State of What Art?](https://aclanthology.org/2024.tacl-1.52/)则在 6.5M
实例上展示单一 instruction template 会造成性能与排名脆弱性。
两者支持“模板不能当无关包装”,但测量对象是行为/评测,不是本模型的专家路由机制。
---
## 15. 因果阶梯
当前证据阶梯可以写成:
```text
一级:官方模板精确 tokenization
已完成
二级:同长度单 ID 干预
已完成
三级:目标后 suffix causal 负对照
已完成,34,488 / 34,488 ordered exact
四级:前置角色词头直接改变目标路由
已完成,direct TV ≈ .02–.03
五级:角色词头统一调制 system
证据反对;方向混合、均值近零
六级:完整角色协议
未完成;冒号、换行、EOS、词头组合尚未一起正交
七级:Chat/SFT 行为
未完成
八级:任务能力 / 安全 / 准确率
未完成
```
---
## 16. 不允许外推的结论
本实验不能证明:
- 模型“理解 User 和 Assistant 的社会角色”;
- `Assistant` 词头比 `User` 更强或更弱;
- 角色词头统一放大 system prompt;
- 某个专家是 user / assistant 专家;
- 路由变化等于回答质量变化;
- base checkpoint 与 Chat checkpoint 使用相同角色机制;
- 任意 chat template 都可用一个 token 代表;
- suffix edit 对生成结果没有影响;
- 完整 27 层与当前前六个 MoE 层相同;
- 跨 batch 或跨硬件的 route hash 必须相同。
---
## 17. 可复现运行
脚本:
```text
experiments/deepseek/v2_lite_routing_role_marker_head_control.py
```
正式命令:
```bash
PYTHONPATH=/tmp/deepseek-v2-lite-pydeps.6ZJAH3:/usr/lib/python3/dist-packages \
/home/wuyang/.pyenv/versions/navi-router-cu128/bin/python -B \
experiments/deepseek/v2_lite_routing_role_marker_head_control.py \
--artifact-dir /tmp/deepseek-v2-lite-artifacts.OEjfce \
--human-eval /tmp/llm-atlas-routing-corpus/human-eval/data/HumanEval.jsonl.gz \
--gsm8k /tmp/llm-atlas-routing-corpus/gsm8k/grade_school_math/data/test.jsonl \
--tnews /tmp/llm-atlas-routing-corpus/tnews/test.json \
--tnews-archive /tmp/llm-atlas-routing-corpus/tnews_public.zip \
--wikitext /tmp/llm-atlas-routing-corpus/wikitext-2-raw-v1-validation.parquet \
--output src/data/deepseek-v2-lite-routing-role-marker-head-control.json \
--per-domain 32 \
--content-tokens 23 \
--batch-prompts 4 \
--layers 7 \
--bootstrap 2000 \
--seed 20260729 \
--captured-at 2026-07-29T12:30:00+00:00
```
正式结果:
```text
src/data/deepseek-v2-lite-routing-role-marker-head-control.json
53,440,884 bytes
SHA-256 9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf
```
独立复跑:
```text
src/data/deepseek-v2-lite-routing-role-marker-head-control-repro.json
53,440,884 bytes
SHA-256 9dc0e37fbce6581269428dcfcb84c7a17b466c5239c8171e6741f66d5eeb8caf
```
两份文件 `cmp` byte-exact。
---
## 18. 下一步
角色词头已经拆到单 ID,但完整角色协议仍包含:
```text
角色词头 + 冒号 + 空格/换行 + EOS/EOT + 训练身份
```
下一批优先级:
1. special-token family:BOS / EOS / 普通词头使用同长度家族对照;
2. 完整 `User:` / `Assistant:` 两 token 角色块,用等长两 token 控制;
3. V2-Lite-Chat checkpoint 的同构路由与生成行为;
4. target role swap 后的 next-token logits / generation / task score;
5. 固定整批 companion rows 的 dummy-shape 与 batch-content 对照;
6. 完整 27 层;
7. gate logits / margin,而不只看离散 top-6。
本轮最重要的收获不是一句“角色很重要”,而是一条更精确的分界:
> 一个前置角色词头 ID 足以改变后续路由;但它没有把 system 作用变成统一方向。
> 真正严格的 Chat 角色机制,需要把训练身份、完整控制 token 族与行为指标一起补上。
@@ -0,0 +1,232 @@
import { createHash } from "node:crypto";
import { readFileSync, statSync, writeFileSync } from "node:fs";
import { resolve } from "node:path";
const root = resolve(import.meta.dirname, "..");
const mainPath = resolve(
root,
"src/data/deepseek-v2-lite-routing-role-marker-head-control.json",
);
const reproPath = resolve(
root,
"src/data/deepseek-v2-lite-routing-role-marker-head-control-repro.json",
);
const boundaryPath = resolve(
root,
"src/data/deepseek-v2-lite-routing-history-boundary-token-control.json",
);
const outputPath = resolve(
root,
"src/data/deepseek-v2-lite-routing-role-marker-head-control-compact.json",
);
const sha256 = (path) => createHash("sha256")
.update(readFileSync(path))
.digest("hex");
const mainSha256 = sha256(mainPath);
const reproSha256 = sha256(reproPath);
const mainBytes = statSync(mainPath).size;
const reproBytes = statSync(reproPath).size;
const exact = mainSha256 === reproSha256 && mainBytes === reproBytes;
if (!exact) {
throw new Error("role-marker formal run and rerun are not byte-exact");
}
const role = JSON.parse(readFileSync(mainPath, "utf8"));
const boundary = JSON.parse(readFileSync(boundaryPath, "utf8"));
const roleEdges = [
"system_official",
"system_target_assistant",
"system_target_x",
"system_suffix_user",
"target_assistant_at_s0",
"target_assistant_at_s1",
"target_x_at_s0",
"target_x_at_s1",
"suffix_user_at_s0",
"suffix_user_at_s1",
];
const aggregateAlignment = (layer, domain, edge) => {
const rows = layer.prompts
.filter((prompt) => prompt.domain === domain)
.map((prompt) => prompt.alignments[edge]);
const aligned = rows.reduce(
(sum, row) => sum + row.aligned_tokens,
0,
);
const setExact = rows.reduce(
(sum, row) => sum + row.set_topk_exact,
0,
);
const orderedExact = rows.reduce(
(sum, row) => sum + row.ordered_topk_exact,
0,
);
const weightedJaccard = rows.reduce(
(sum, row) => sum + row.mean_jaccard * row.aligned_tokens,
0,
);
return {
aligned,
setExactRate: setExact / aligned,
orderedExactRate: orderedExact / aligned,
meanJaccard: weightedJaccard / aligned,
};
};
const sourceById = new Map(
boundary.corpus_contract.selected.map((source) => [source.id, source]),
);
const tokenContract = {
compared: 0,
messageHashExact: 0,
renderedHashExact: 0,
tokenIdHashExact: 0,
targetContractExact: 0,
};
for (const source of role.corpus_contract.selected) {
const priorSource = sourceById.get(source.id);
if (!priorSource) throw new Error(`boundary source missing: ${source.id}`);
for (const system of [0, 1]) {
const current = source.conditions[`s${system}_official`];
const previous = priorSource.conditions[`s${system}_eos`];
tokenContract.compared += 1;
tokenContract.messageHashExact += (
current.messages_sha256 === previous.messages_sha256
);
tokenContract.renderedHashExact += (
current.rendered_sha256 === previous.rendered_sha256
);
tokenContract.tokenIdHashExact += (
current.token_ids_sha256 === previous.token_ids_sha256
);
tokenContract.targetContractExact += (
current.tokens === previous.tokens
&& current.content_tokens === previous.content_tokens
&& current.aligned_content_tokens
=== previous.aligned_content_tokens
);
}
}
const crossBatch = role.layers.slice(1).map((layer) => {
const previousLayer = boundary.layers.find(
(candidate) => candidate.layer === layer.layer,
);
const previousById = new Map(
previousLayer.prompts.map((prompt) => [prompt.id, prompt]),
);
const counts = {
compared: 0,
fullRouteHashExact: 0,
targetRouteHashExact: 0,
fullLoadExact: 0,
targetLoadExact: 0,
};
for (const prompt of layer.prompts) {
const previousPrompt = previousById.get(prompt.id);
if (!previousPrompt) {
throw new Error(`boundary prompt missing: ${prompt.id}`);
}
for (const system of [0, 1]) {
const current = prompt.conditions[`s${system}_official`];
const previous = previousPrompt.conditions[`s${system}_eos`];
counts.compared += 1;
counts.fullRouteHashExact += (
current.topk_sha256 === previous.topk_sha256
);
counts.targetRouteHashExact += (
current.content_topk_sha256 === previous.content_topk_sha256
);
counts.fullLoadExact += (
JSON.stringify(current.full_load)
=== JSON.stringify(previous.full_load)
);
counts.targetLoadExact += (
JSON.stringify(current.content_load)
=== JSON.stringify(previous.content_load)
);
}
}
return { layer: layer.layer, ...counts };
});
const compact = {
schemaVersion: 1,
source: {
mainSha256,
reproSha256,
mainBytes,
reproBytes,
exact,
},
domains: role.corpus_contract.domains,
labels: role.corpus_contract.domain_labels,
inference: role.inference_contract,
contract: {
tokenIds: role.role_marker_head_contract.role_token_ids,
validation: role.role_marker_head_contract.render_validation,
official: role.boundary.official_serialization_by_role_head,
tokenContractAgainstBoundaryRun: tokenContract,
},
crossBatch,
layers: role.layers.slice(1).map((layer) => ({
layer: layer.layer,
alignment: Object.fromEntries(
role.corpus_contract.domains.map((domain) => [
domain,
Object.fromEntries(
roleEdges.map((edge) => [
edge,
aggregateAlignment(layer, domain, edge),
]),
),
]),
),
scopes: Object.fromEntries(
["target_content", "full_input"].map((scope) => [
scope,
{
modes: Object.fromEntries(
["prompt_balanced", "token_weighted"].map((mode) => {
const control = (
layer.statistics[scope].modes[mode].role_marker_control
);
return [
mode,
Object.fromEntries(
role.corpus_contract.domains.map((domain) => [
domain,
{
distances: control[domain].system_edge_distances,
contrasts: (
control[domain].system_edge_distance_contrasts
),
cvEdges: control[domain].metric_system_edges.cv,
cvContrasts: (
control[domain].metric_system_edge_contrasts.cv
),
direct: control[domain].direct_substitutions,
},
]),
),
];
}),
),
},
]),
),
})),
};
writeFileSync(
outputPath,
`${JSON.stringify(compact, null, 2)}\n`,
"utf8",
);
process.stdout.write(
`${outputPath}\n${mainSha256}\n${mainBytes} bytes source → `
+ `${statSync(outputPath).size} bytes compact\n`,
);
+98 -4
View File
@@ -581,6 +581,71 @@ await evaluate(`(() => {
await pause(120);
await screenshot("/tmp/llm-atlas-deepseek-boundary-results-desktop.png");
const artifactRole = await evaluate(`(() => {
const root = document.querySelector("[data-dsv2-lab]");
root.querySelector('[data-artifact-tab="role"]').click();
const read = () => ({
panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
levelCards: root.querySelectorAll(".role-level-grid > article").length,
domainCards: root.querySelectorAll("[data-role-domain-grid] > article").length,
domains: [...root.querySelectorAll("[data-role-domain-grid] > article")].map((node) => ({
label: node.querySelector(":scope > span").textContent.trim(),
values: [...node.querySelectorAll(".role-tv-ladder > b")].map((cell) => ({
name: cell.querySelector("small").textContent.trim(),
value: cell.querySelector("strong").textContent.trim(),
})),
effect: node.querySelector(":scope > strong").textContent.trim(),
className: node.querySelector(":scope > strong").className,
ci: node.querySelector(":scope > p").textContent.trim(),
stability: node.querySelector(":scope > small").textContent.trim(),
direct: node.querySelector(":scope > em").textContent.trim(),
interaction: node.querySelector(":scope > u").textContent.trim(),
cv: node.querySelector(":scope > i").textContent.trim(),
})),
summary: [...root.querySelectorAll('[data-artifact-panel="role"] .role-summary article b')].map((node) => node.textContent.trim()),
depthRows: root.querySelectorAll("[data-role-depth-map] > div").length,
depthCells: root.querySelectorAll("[data-role-depth-map] > div > span").length,
depthTitle: root.querySelector("[data-role-depth-title]").textContent.trim(),
exact: root.querySelector(".role-ledger .exact b").textContent.trim(),
causalCards: root.querySelectorAll(".role-causal-ledger > article").length,
batchCells: root.querySelectorAll(".role-batch-audit > div:last-child > span").length,
batchValues: [...root.querySelectorAll(".role-batch-audit > div:last-child > span i")].map((node) => node.textContent.trim()),
note: root.querySelector("[data-role-note]").textContent.trim(),
targetHead: root.querySelector("[data-role-target-head]").textContent.trim(),
suffixHead: root.querySelector("[data-role-suffix-head]").textContent.trim(),
activeLayer: root.querySelector("[data-role-layer].active").textContent.trim(),
activeScope: root.querySelector('[data-role-scope][aria-pressed="true"]').dataset.roleScope,
activeMode: root.querySelector('[data-role-mode][aria-pressed="true"]').dataset.roleMode,
activeContrast: root.querySelector('[data-role-contrast][aria-pressed="true"]').dataset.roleContrast,
});
const layer1Assistant = read();
root.querySelector('[data-role-layer="5"]').click();
root.querySelector('[data-role-contrast="target_x_minus_official"]').click();
const layer5X = read();
root.querySelector('[data-role-contrast="suffix_user_minus_official"]').click();
const layer5Suffix = read();
root.querySelector('[data-role-scope="full_input"]').click();
const layer5FullSuffix = read();
root.querySelector('[data-role-mode="token_weighted"]').click();
const layer5FullToken = read();
root.querySelector('[data-role-layer="1"]').click();
root.querySelector('[data-role-scope="target_content"]').click();
root.querySelector('[data-role-mode="prompt_balanced"]').click();
root.querySelector('[data-role-contrast="target_assistant_minus_official"]').click();
return { layer1Assistant, layer5X, layer5Suffix, layer5FullSuffix, layer5FullToken, restored: read() };
})()`);
await evaluate(`(() => {
document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
window.scrollBy(0, -82);
})()`);
await pause(180);
await screenshot("/tmp/llm-atlas-deepseek-role-desktop.png");
await evaluate(`(() => {
document.querySelector(".role-domain-grid").scrollIntoView({ block: "center", behavior: "instant" });
})()`);
await pause(120);
await screenshot("/tmp/llm-atlas-deepseek-role-results-desktop.png");
const artifactEvidence = await evaluate(`(() => {
const root = document.querySelector("[data-dsv2-lab]");
root.querySelector('[data-artifact-tab="evidence"]').click();
@@ -678,6 +743,13 @@ const mobile = await evaluate(`(() => {
boundaryTokenCards: artifact.querySelectorAll(".boundary-token-grid > article").length,
boundaryDomainCards: artifact.querySelectorAll("[data-boundary-domain-grid] > article").length,
boundaryDepthCells: artifact.querySelectorAll("[data-boundary-depth-map] > div > span").length,
roleLayers: artifact.querySelectorAll("[data-role-layer]").length,
roleScopes: artifact.querySelectorAll("[data-role-scope]").length,
roleModes: artifact.querySelectorAll("[data-role-mode]").length,
roleContrasts: artifact.querySelectorAll("[data-role-contrast]").length,
roleLevelCards: artifact.querySelectorAll(".role-level-grid > article").length,
roleDomainCards: artifact.querySelectorAll("[data-role-domain-grid] > article").length,
roleDepthCells: artifact.querySelectorAll("[data-role-depth-map] > div > span").length,
offenders: [...document.querySelectorAll("body *")]
.filter((node) => !node.closest(".paper-chain, .advantage-table, .precision-table, .mapping-table, [data-deepseek-lab], [data-dsv2-lab]"))
.filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1)
@@ -761,8 +833,22 @@ await evaluate(`(() => {
})()`);
await pause(120);
await screenshot("/tmp/llm-atlas-deepseek-boundary-results-mobile.png");
await evaluate(`(() => {
const artifact = document.querySelector("[data-dsv2-lab]");
artifact.querySelector('[data-artifact-tab="role"]').click();
artifact.scrollIntoView({ block: "start", behavior: "instant" });
window.scrollBy(0, -70);
})()`);
await pause(180);
await screenshot("/tmp/llm-atlas-deepseek-role-mobile.png");
await evaluate(`(() => {
document.querySelector(".role-domain-grid").scrollIntoView({ block: "start", behavior: "instant" });
window.scrollBy(0, -72);
})()`);
await pause(120);
await screenshot("/tmp/llm-atlas-deepseek-role-results-mobile.png");
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactCorpus, artifactTemplate, artifactHistory, artifactDistance, artifactBoundary, artifactEvidence, home, papers, mobile, exceptions };
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactCorpus, artifactTemplate, artifactHistory, artifactDistance, artifactBoundary, artifactRole, artifactEvidence, home, papers, mobile, exceptions };
console.log(JSON.stringify(report, null, 2));
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
@@ -772,8 +858,8 @@ if (overview.sections !== 26 || overview.tocLinks !== 26) failures.push("二十
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
if (overview.artifactTabs !== 10 || overview.artifactPanels !== 10 || overview.artifactLayers !== 27) failures.push("真实权重十联实验结构异常");
if (overview.heroLabs !== "14 个可操作实验") failures.push("DeepSeek 实验总数账异常");
if (overview.artifactTabs !== 11 || overview.artifactPanels !== 11 || overview.artifactLayers !== 27) failures.push("真实权重十一联实验结构异常");
if (overview.heroLabs !== "15 个可操作实验") failures.push("DeepSeek 实验总数账异常");
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
@@ -835,12 +921,20 @@ if (artifactBoundary.layer4Period.domains[2].effect !== "PERIOD − EOS · ΔTV
if (artifactBoundary.layer4Newline.activeContrast !== "newline_minus_eos" || artifactBoundary.layer4Newline.trackToken !== "↵" || !artifactBoundary.layer4Newline.depthTitle.includes("换行")) failures.push("换行边界替换切换异常");
if (artifactBoundary.layer4Full.activeScope !== "full_input" || !artifactBoundary.layer4Full.note.includes("完整输入") || artifactBoundary.layer4Full.domains[0].values[0].value === artifactBoundary.layer4Newline.domains[0].values[0].value) failures.push("边界控制完整输入 scope 异常");
if (artifactBoundary.layer4FullToken.activeMode !== "token_weighted" || artifactBoundary.restored.activeLayer !== "L1" || artifactBoundary.restored.activeScope !== "target_content" || artifactBoundary.restored.activeMode !== "prompt_balanced" || artifactBoundary.restored.activeContrast !== "x_minus_eos") failures.push("边界控制聚合口径或恢复状态异常");
if (artifactRole.layer1Assistant.panel !== "role" || artifactRole.layer1Assistant.levelCards !== 4 || artifactRole.layer1Assistant.domainCards !== 4 || artifactRole.layer1Assistant.depthRows !== 4 || artifactRole.layer1Assistant.depthCells !== 24 || artifactRole.layer1Assistant.exact !== "BYTE-EXACT" || artifactRole.layer1Assistant.causalCards !== 3 || artifactRole.layer1Assistant.batchCells !== 6) failures.push("角色词头单 ID 控制结构或独立复跑闸门异常");
if (artifactRole.layer1Assistant.domains[0].values.map((cell) => cell.value).join("/") !== "0.056/0.059/0.062/0.056" || artifactRole.layer1Assistant.domains[0].effect !== "U→A − OFFICIAL · ΔTV +0.003" || !artifactRole.layer1Assistant.domains[0].ci.includes("-0.003, +0.007")) failures.push("L1 英文角色词头 system-edge 统计异常");
if (!artifactRole.layer1Assistant.domains[0].stability.includes("S0 79.7% · S1 84.6%") || !artifactRole.layer1Assistant.domains[0].direct.includes("S0 0.024 · S1 0.021") || artifactRole.layer1Assistant.summary.join("|") !== ".0222 / .0218|.0263 / .0253|34,488 / 34,488|12↑12↓ / 13↑11↓") failures.push("角色词头直接效应、对齐率或总账异常");
if (artifactRole.layer5X.activeContrast !== "target_x_minus_official" || artifactRole.layer5X.targetHead !== "x" || !artifactRole.layer5X.depthTitle.includes("普通 token")) failures.push("角色词头 U→x 层或 contrast 切换异常");
if (artifactRole.layer5Suffix.domains.some((domain) => domain.effect !== "SUFFIX A→U − OFFICIAL · ΔTV +0.000") || artifactRole.layer5Suffix.targetHead !== "User" || artifactRole.layer5Suffix.suffixHead !== "User" || !artifactRole.layer5Suffix.domains.every((domain) => domain.direct.includes("S0 0.000 · S1 0.000"))) failures.push("后置角色词头 causal suffix 负对照异常");
if (artifactRole.layer5FullSuffix.activeScope !== "full_input" || !artifactRole.layer5FullSuffix.note.includes("suffix 自身") || artifactRole.layer5FullSuffix.domains[0].values[0].value === artifactRole.layer5Suffix.domains[0].values[0].value) failures.push("角色词头完整输入 scope 异常");
if (artifactRole.layer5FullToken.activeMode !== "token_weighted" || artifactRole.restored.activeLayer !== "L1" || artifactRole.restored.activeScope !== "target_content" || artifactRole.restored.activeMode !== "prompt_balanced" || artifactRole.restored.activeContrast !== "target_assistant_minus_official" || artifactRole.restored.targetHead !== "Assistant" || artifactRole.restored.suffixHead !== "Assistant") failures.push("角色词头聚合口径或恢复状态异常");
if (artifactRole.layer1Assistant.batchValues.join("|") !== "256 / 256|180 / 256|149 / 256|102 / 256|72 / 256|73 / 256") failures.push("角色词头 BF16 batch-content 审计异常");
if (artifactEvidence.panel !== "evidence" || artifactEvidence.layers !== 27 || artifactEvidence.executed !== 7 || artifactEvidence.split !== 1 || artifactEvidence.unloaded !== 19 || artifactEvidence.exact !== "31 / 31") failures.push("真实工件执行边界或复跑闸门异常");
if (!artifactEvidence.dependency.includes("Transformers 5.5") || !artifactEvidence.dependency.includes("4.41.2") || !artifactEvidence.boundary.includes("完整 27 层生成")) failures.push("依赖版本或未覆盖边界异常");
if (artifactEvidence.keyboardSelected !== "load" || artifactEvidence.keyboardVisible !== "load") failures.push("真实工件实验键盘 tab 导航异常");
if (home.releaseCards !== 17 || !home.firstRelease.includes("47 页不再压成摘要") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 DeepSeek 首发入口或论文数异常");
if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常");
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 10 || mobile.artifactHeatCells !== 64 || mobile.corpusCohorts !== 3 || mobile.lengthDeltaCards !== 4 || mobile.templateLayers !== 6 || mobile.templateScopes !== 2 || mobile.templateModes !== 2 || mobile.templateDomainCards !== 4 || mobile.templateDepthCells !== 24 || mobile.historyLayers !== 6 || mobile.historyScopes !== 2 || mobile.historyModes !== 2 || mobile.historyEffects !== 3 || mobile.historyDomainCards !== 4 || mobile.historyDepthCells !== 24 || mobile.distanceLayers !== 6 || mobile.distanceScopes !== 2 || mobile.distanceModes !== 2 || mobile.distanceContrasts !== 2 || mobile.distanceDomainCards !== 4 || mobile.distanceDepthCells !== 24 || mobile.boundaryLayers !== 6 || mobile.boundaryScopes !== 2 || mobile.boundaryModes !== 2 || mobile.boundaryContrasts !== 3 || mobile.boundaryTokenCards !== 4 || mobile.boundaryDomainCards !== 4 || mobile.boundaryDepthCells !== 24) failures.push("移动端导航或实验异常");
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 11 || mobile.artifactHeatCells !== 64 || mobile.corpusCohorts !== 3 || mobile.lengthDeltaCards !== 4 || mobile.templateLayers !== 6 || mobile.templateScopes !== 2 || mobile.templateModes !== 2 || mobile.templateDomainCards !== 4 || mobile.templateDepthCells !== 24 || mobile.historyLayers !== 6 || mobile.historyScopes !== 2 || mobile.historyModes !== 2 || mobile.historyEffects !== 3 || mobile.historyDomainCards !== 4 || mobile.historyDepthCells !== 24 || mobile.distanceLayers !== 6 || mobile.distanceScopes !== 2 || mobile.distanceModes !== 2 || mobile.distanceContrasts !== 2 || mobile.distanceDomainCards !== 4 || mobile.distanceDepthCells !== 24 || mobile.boundaryLayers !== 6 || mobile.boundaryScopes !== 2 || mobile.boundaryModes !== 2 || mobile.boundaryContrasts !== 3 || mobile.boundaryTokenCards !== 4 || mobile.boundaryDomainCards !== 4 || mobile.boundaryDepthCells !== 24 || mobile.roleLayers !== 6 || mobile.roleScopes !== 2 || mobile.roleModes !== 2 || mobile.roleContrasts !== 3 || mobile.roleLevelCards !== 4 || mobile.roleDomainCards !== 4 || mobile.roleDepthCells !== 24) failures.push("移动端导航或实验异常");
if (mobile.offenders.length) failures.push(`移动端越界元素:${JSON.stringify(mobile.offenders)}`);
if (exceptions.length) failures.push(`浏览器异常:${exceptions.join(" | ")}`);
+682 -10
View File
@@ -17,6 +17,7 @@ import rawHistoryRepro from "@/data/deepseek-v2-lite-routing-history-factorial-r
import rawDistance from "@/data/deepseek-v2-lite-routing-history-distance-control.json";
import rawDistanceRepro from "@/data/deepseek-v2-lite-routing-history-distance-control-repro.json";
import rawBoundaryCompact from "@/data/deepseek-v2-lite-routing-history-boundary-token-control-compact.json";
import rawRoleCompact from "@/data/deepseek-v2-lite-routing-role-marker-head-control-compact.json";
const trace = rawTrace as any;
const repro = rawRepro as any;
@@ -36,6 +37,7 @@ const historyRepro = rawHistoryRepro as any;
const distance = rawDistance as any;
const distanceRepro = rawDistanceRepro as any;
const boundaryCompact = rawBoundaryCompact as any;
const roleCompact = rawRoleCompact as any;
const absorbExact = JSON.stringify(absorb) === JSON.stringify(absorbRepro);
const corpusExact = JSON.stringify(corpus) === JSON.stringify(corpusRepro);
const matched16Exact = JSON.stringify(matched16) === JSON.stringify(matched16Repro);
@@ -44,6 +46,7 @@ const templateExact = JSON.stringify(template) === JSON.stringify(templateRepro)
const historyExact = JSON.stringify(history) === JSON.stringify(historyRepro);
const distanceExact = JSON.stringify(distance) === JSON.stringify(distanceRepro);
const boundaryExact = boundaryCompact.source.exact;
const roleExact = roleCompact.source.exact;
const bytes = (value: number) => value >= 1024
? `${(value / 1024).toFixed(2)} KiB`
: `${value.toLocaleString()} B`;
@@ -333,6 +336,7 @@ const distanceCompact = {
};
const distanceCompactJson = JSON.stringify(distanceCompact).replaceAll("<", "\\u003c");
const boundaryCompactJson = JSON.stringify(boundaryCompact).replaceAll("<", "\\u003c");
const roleCompactJson = JSON.stringify(roleCompact).replaceAll("<", "\\u003c");
const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoint_tensor_bytes;
---
@@ -344,7 +348,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
</div>
<p>
固定官方 revision、tokenizer、模型代码和 BF16 第一分片;RTX 5090 连续执行 layer 0–6,
从 3,240 次 token 显微轨迹扩到 5,157,072 次公开语料路由,并让 layer-1 权重继续走入官方吸收式 cache。
从 3,240 次 token 显微轨迹扩到 7,201,296 次公开语料路由,并让 layer-1 权重继续走入官方吸收式 cache。
所有结论都带证据身份与停止线。
</p>
</header>
@@ -384,8 +388,11 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
<button type="button" role="tab" data-artifact-tab="boundary" aria-selected="false" tabindex="-1">
<span>09</span><b>边界单词元控制</b><small>EOS ↔ x / . / ↵</small>
</button>
<button type="button" role="tab" data-artifact-tab="role" aria-selected="false" tabindex="-1">
<span>10</span><b>角色词头控制</b><small>User ↔ Assistant / x</small>
</button>
<button type="button" role="tab" data-artifact-tab="evidence" aria-selected="false" tabindex="-1">
<span>10</span><b>证据断面</b><small>revision · shards · rerun</small>
<span>11</span><b>证据断面</b><small>revision · shards · rerun</small>
</button>
</div>
@@ -1312,6 +1319,157 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
</div>
</section>
<section class="artifact-panel" data-artifact-panel="role" hidden>
<div class="panel-lead">
<div><span>X / ROLE-HEAD SINGLE-ID CONTROL</span><h4>只换 `User:` 的第一个 ID:直接作用不等于统一调制 system</h4></div>
<p>
官方 EOS、冒号、长度、目标位置、mask 与 32-row batch 全部不动;
目标后的 generation head 另做 causal suffix 负对照。
</p>
</div>
<div class="history-ledger role-ledger">
<article><span>SOURCE PROMPTS</span><b>128</b><p>四个公开域各 32 条;沿用同一 cohort</p></article>
<article><span>2×4 VARIANTS</span><b>1,024</b><p>system 0/1 × 四种词头水平</p></article>
<article><span>INPUT TOKENS</span><b>56,784</b><p>八格逐组完全同长度</p></article>
<article><span>REAL ROUTES</span><b>2,044,224</b><p>八格 × 前六个 MoE 层</p></article>
<article><span>CAUSAL SUFFIX</span><b>34,488 exact</b><p>目标 layer×token×system ordered top-6</p></article>
<article class="exact"><span>INDEPENDENT RERUN</span><b>{roleExact ? "BYTE-EXACT" : "MISMATCH"}</b><p>完整 JSON SHA-256 9dc0e37f…b8caf</p></article>
</div>
<div class="role-protocol" aria-label="目标角色词头的单 token ID 控制">
<div class="role-track">
<span>FIXED HISTORY + EOS</span>
<b data-role-target-head>User</b><i>:</i><mark>TARGET</mark>
<span>CAUSAL SUFFIX</span>
<b data-role-suffix-head>Assistant</b><i>:</i>
</div>
<div class="role-level-grid">
<article class="official">
<span>OFFICIAL</span><b>User … Assistant</b><code>5726 / 77398</code>
<p>目标入口与 generation prompt 都保持官方词头。</p>
</article>
<article>
<span>PRE-TARGET A</span><b>User → Assistant</b><code>5726 → 77398</code>
<p>只改目标前词头;冒号与官方 EOS 不动。</p>
</article>
<article>
<span>PRE-TARGET B</span><b>User → x</b><code>5726 → 87</code>
<p>普通内容 token 对照;仍只改一个 ID。</p>
</article>
<article class="negative">
<span>POST-TARGET CONTROL</span><b>Assistant → User</b><code>77398 → 5726</code>
<p>发生在目标之后;目标路由必须 causal-exact。</p>
</article>
</div>
<p>
256 / 256 组同长度同目标位置;768 / 768 个反事实相对官方序列恰好一 ID。
三个反事实都不是官方合法 chat。
</p>
</div>
<div class="role-controls">
<div>
<span>MOE LAYER</span>
<div class="layer-switch role-layer-switch" role="group" aria-label="选择角色词头控制层">
{[1, 2, 3, 4, 5, 6].map((layer) => (
<button type="button" data-role-layer={layer} class={layer === 1 ? "active" : ""}>L{layer}</button>
))}
</div>
</div>
<div>
<span>MEASUREMENT SCOPE</span>
<div class="role-scope-switch" role="group" aria-label="选择角色词头统计范围">
<button type="button" data-role-scope="target_content" aria-pressed="true">目标内容</button>
<button type="button" data-role-scope="full_input" aria-pressed="false">完整输入</button>
</div>
</div>
<div>
<span>AGGREGATION</span>
<div class="role-mode-switch" role="group" aria-label="选择角色词头聚合口径">
<button type="button" data-role-mode="prompt_balanced" aria-pressed="true">prompt 等权</button>
<button type="button" data-role-mode="token_weighted" aria-pressed="false">token 加权</button>
</div>
</div>
<div>
<span>CONTROL − OFFICIAL</span>
<div class="role-contrast-switch" role="group" aria-label="选择角色词头对照">
<button type="button" data-role-contrast="target_assistant_minus_official" aria-pressed="true">U→A</button>
<button type="button" data-role-contrast="target_x_minus_official" aria-pressed="false">U→x</button>
<button type="button" data-role-contrast="suffix_user_minus_official" aria-pressed="false">suffix A→U</button>
</div>
</div>
<p data-role-note>
目标内容:正 ΔTV 表示替换词头后 system edge 更大;direct TV 单独回答“这个 ID 本身改了多少路由”。
</p>
</div>
<div class="role-domain-grid" data-role-domain-grid></div>
<div class="history-buffer-summary role-summary">
<article><span>DIRECT U→A</span><b>.0222 / .0218</b><p>S0 / S1 mean target TV;前置词头有直接作用。</p></article>
<article><span>DIRECT U→x</span><b>.0263 / .0253</b><p>S0 / S1;普通 token 对照同样改变后续路由。</p></article>
<article><span>SUFFIX CONTROL</span><b>34,488 / 34,488</b><p>目标 ordered top-6 exact;direct TV / JSD 全为零。</p></article>
<article><span>SYSTEM MODULATION</span><b>12↑12↓ / 13↑11↓</b><p>U→A / U→x 的 24 格方向混合,均值近零。</p></article>
</div>
<div class="role-depth">
<div>
<span>DEPTH MAP / SYSTEM-EDGE CONTRAST</span>
<h5 data-role-depth-title>User→Assistant:system-edge TV 相对官方的变化</h5>
<p>每格用八条件共享 source-bootstrap;红色为更大、绿色为更小,跨零不冒充稳定方向。</p>
</div>
<div data-role-depth-map></div>
</div>
<div class="role-causal-ledger">
<article>
<span>BEFORE TARGET</span><b>direct TV ≈ .02–.03</b>
<p>单个角色词头 ID 足以条件化后续 token 的专家集合。</p>
</article>
<article class="exact">
<span>AFTER TARGET</span><b>ordered top-6 = 100%</b>
<p>六层 × 两个 system 水平全部 causal-exact;对齐与 mask 闸门通过。</p>
</article>
<article>
<span>NOT IDENTIFIED</span><b>完整角色语义</b>
<p>冒号仍在,且 base ≠ Chat/SFT;没有生成、准确率或安全行为。</p>
</article>
</div>
<div class="role-batch-audit">
<div>
<span>BF16 BATCH-CONTENT AUDIT</span>
<h5>同样 32 rows、同样 official token IDs,companion rows 改了,深层 route 仍会分化</h5>
<p>当前 official 与上一轮 EOS 条件的 256 / 256 token-ID hashes 相同;下表是跨实验 official target route-hash exact。</p>
</div>
<div>
{roleCompact.crossBatch.map((row: any) => (
<span class={row.layer === 1 ? "exact" : ""}>
<b>L{row.layer}</b>
<i>{row.targetRouteHashExact} / {row.compared}</i>
</span>
))}
</div>
</div>
<div class="evidence-links">
<a href="https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite/blob/604d5664dddd88a0433dbae533b7fe9472482de0/tokenizer_config.json" rel="noreferrer">固定官方 tokenizer_config ↗</a>
<a href="https://huggingface.co/docs/transformers/en/chat_templating" rel="noreferrer">HF Chat Templates ↗</a>
<a href="https://arxiv.org/abs/2305.11206" rel="noreferrer">LIMA · EOT ↗</a>
<a href="https://proceedings.iclr.cc/paper_files/paper/2024/file/6bcbb4a501dbad0eba1b660c1a55318c-Paper-Conference.pdf" rel="noreferrer">URIAL · base formatting ↗</a>
<a href="https://aclanthology.org/2024.tacl-1.52/" rel="noreferrer">Multi-prompt sensitivity ↗</a>
</div>
<div class="artifact-boundary">
<b>ONE ROLE-HEAD ID, NOT COMPLETE ROLE SEMANTICS</b>
<p>
前置词头会改变后续路由,但没有统一调制 system;后置负对照严格为零。
下一步仍需 special-token family、完整两-token 角色块、Chat 权重与行为指标。
</p>
</div>
</section>
<section class="artifact-panel" data-artifact-panel="evidence" hidden>
<div class="panel-lead">
<div><span>O + X / EVIDENCE SLICE</span><h4>为什么执行到 layer 6 就停,而不是把“部分下载”写成“完整复现”</h4></div>
@@ -1411,7 +1569,9 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
<code>experiments/deepseek/v2_lite_routing_history_distance_control.py</code> ·
<code>research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md</code> ·
<code>experiments/deepseek/v2_lite_routing_history_boundary_token_control.py</code> ·
<code>research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md</code>
<code>research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md</code> ·
<code>experiments/deepseek/v2_lite_routing_role_marker_head_control.py</code> ·
<code>research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md</code>
</figcaption>
<script is:inline type="application/json" data-dsv2-trace set:html={compactJson}></script>
@@ -1420,6 +1580,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
<script is:inline type="application/json" data-dsv2-history set:html={historyCompactJson}></script>
<script is:inline type="application/json" data-dsv2-distance set:html={distanceCompactJson}></script>
<script is:inline type="application/json" data-dsv2-boundary set:html={boundaryCompactJson}></script>
<script is:inline type="application/json" data-dsv2-role set:html={roleCompactJson}></script>
</figure>
<script>
@@ -1436,6 +1597,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
const historyNode = one<HTMLScriptElement>("[data-dsv2-history]");
const distanceNode = one<HTMLScriptElement>("[data-dsv2-distance]");
const boundaryNode = one<HTMLScriptElement>("[data-dsv2-boundary]");
const roleNode = one<HTMLScriptElement>("[data-dsv2-role]");
if (
!payloadNode?.textContent
|| !corpusNode?.textContent
@@ -1443,6 +1605,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
|| !historyNode?.textContent
|| !distanceNode?.textContent
|| !boundaryNode?.textContent
|| !roleNode?.textContent
) return;
const data = JSON.parse(payloadNode.textContent);
const corpusData = JSON.parse(corpusNode.textContent);
@@ -1450,6 +1613,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
const historyData = JSON.parse(historyNode.textContent);
const distanceData = JSON.parse(distanceNode.textContent);
const boundaryData = JSON.parse(boundaryNode.textContent);
const roleData = JSON.parse(roleNode.textContent);
const tabs = all<HTMLButtonElement>("[data-artifact-tab]");
const panels = all<HTMLElement>("[data-artifact-panel]");
@@ -2508,6 +2672,182 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
});
});
renderBoundary();
let roleLayerNumber = 1;
let roleScope = "target_content";
let roleMode = "prompt_balanced";
let roleContrast = "target_assistant_minus_official";
const roleContrastLabels: Record<string, string> = {
target_assistant_minus_official: "U→A − OFFICIAL",
target_x_minus_official: "U→X − OFFICIAL",
suffix_user_minus_official: "SUFFIX A→U − OFFICIAL",
};
const roleLevelLabels: Record<string, string> = {
official: "OFF.",
target_assistant: "U→A",
target_x: "U→x",
suffix_user: "SFX",
};
const roleReplacement = () => roleContrast.replace("_minus_official", "");
const renderRole = () => {
all<HTMLButtonElement>("[data-role-layer]").forEach((button) => {
button.classList.toggle(
"active",
Number(button.dataset.roleLayer) === roleLayerNumber,
);
});
all<HTMLButtonElement>("[data-role-scope]").forEach((button) => {
button.setAttribute(
"aria-pressed",
String(button.dataset.roleScope === roleScope),
);
});
all<HTMLButtonElement>("[data-role-mode]").forEach((button) => {
button.setAttribute(
"aria-pressed",
String(button.dataset.roleMode === roleMode),
);
});
all<HTMLButtonElement>("[data-role-contrast]").forEach((button) => {
button.setAttribute(
"aria-pressed",
String(button.dataset.roleContrast === roleContrast),
);
});
set(
"[data-role-note]",
roleScope === "target_content"
? "目标内容:正 ΔTV 表示替换词头后 system edge 更大;direct TV 单独回答“这个 ID 本身改了多少路由”。"
: "完整输入:被替换词头、历史、目标、换行与 generation prompt 全部计入;suffix 自身因此不再是零。",
);
const replacement = roleReplacement();
set(
"[data-role-target-head]",
replacement === "target_assistant"
? "Assistant"
: replacement === "target_x"
? "x"
: "User",
);
set(
"[data-role-suffix-head]",
replacement === "suffix_user" ? "User" : "Assistant",
);
const currentLayer = roleData.layers.find(
(item: any) => item.layer === roleLayerNumber,
);
const view = currentLayer.scopes[roleScope].modes[roleMode];
const grid = one<HTMLElement>("[data-role-domain-grid]");
if (grid) {
grid.replaceChildren(...roleData.domains.map((domain: string) => {
const data = view[domain];
const contrast = data.contrasts[roleContrast].total_variation_delta;
const direct = data.direct[replacement];
const interaction = direct.s1_minus_s0.total_variation_delta;
const alignmentS0 = currentLayer.alignment[domain][`${replacement}_at_s0`];
const alignmentS1 = currentLayer.alignment[domain][`${replacement}_at_s1`];
const card = document.createElement("article");
const label = document.createElement("span");
const ladder = document.createElement("div");
const primary = document.createElement("strong");
const ci = document.createElement("p");
const stability = document.createElement("small");
const substitution = document.createElement("em");
const interactionLine = document.createElement("u");
const cv = document.createElement("i");
label.textContent = corpusLabels[domain];
ladder.className = "role-tv-ladder";
["official", "target_assistant", "target_x", "suffix_user"].forEach((level) => {
const cell = document.createElement("b");
const name = document.createElement("small");
const value = document.createElement("strong");
name.textContent = roleLevelLabels[level];
value.textContent = data.distances[level].total_variation.point.toFixed(3);
cell.classList.toggle("selected", level === replacement);
cell.classList.toggle("official", level === "official");
cell.classList.toggle("causal", level === "suffix_user");
cell.append(name, value);
ladder.append(cell);
});
primary.textContent = `${roleContrastLabels[roleContrast]} · ΔTV ${signed(contrast.point)}`;
primary.className = deltaClass(contrast.ci95);
ci.textContent = `source-paired 95% ${formatSignedCi(contrast.ci95)}`;
stability.textContent = `direct target top-6 set exact · S0 ${(alignmentS0.setExactRate * 100).toFixed(1)}% · S1 ${(alignmentS1.setExactRate * 100).toFixed(1)}% · J ${alignmentS0.meanJaccard.toFixed(3)} / ${alignmentS1.meanJaccard.toFixed(3)}`;
substitution.textContent = `direct OFFICIAL↔${roleLevelLabels[replacement]} TV · S0 ${direct.at_s0.total_variation.point.toFixed(3)} · S1 ${direct.at_s1.total_variation.point.toFixed(3)}`;
interactionLine.textContent = `direct S1−S0 ${signed(interaction.point)} · 95% ${formatSignedCi(interaction.ci95)}`;
const cvContrast = data.cvContrasts[roleContrast];
cv.textContent = `system-edge ΔCV ${signed(cvContrast.point)} · 95% ${formatSignedCi(cvContrast.ci95)}`;
card.append(
label,
ladder,
primary,
ci,
stability,
substitution,
interactionLine,
cv,
);
return card;
}));
}
const titles: Record<string, string> = {
target_assistant_minus_official: "User→Assistant:system-edge TV 相对官方的变化",
target_x_minus_official: "User→x:普通 token 词头的 system-edge contrast",
suffix_user_minus_official: "suffix Assistant→User:目标内容 causal-exact 负对照",
};
set("[data-role-depth-title]", titles[roleContrast]);
const depth = one<HTMLElement>("[data-role-depth-map]");
if (depth) {
depth.replaceChildren(...roleData.domains.map((domain: string) => {
const row = document.createElement("div");
const label = document.createElement("b");
label.textContent = corpusLabels[domain];
row.append(label);
roleData.layers.forEach((layer: any) => {
const contrast = layer.scopes[roleScope].modes[roleMode]
[domain].contrasts[roleContrast].total_variation_delta;
const cell = document.createElement("span");
cell.className = deltaClass(contrast.ci95);
cell.style.setProperty(
"--strength",
String(Math.min(1, Math.abs(contrast.point) / 0.015)),
);
cell.textContent = `L${layer.layer} ${signed(contrast.point)}`;
cell.title = `${corpusLabels[domain]} · L${layer.layer} · ${roleContrastLabels[roleContrast]} Δ system-edge TV ${signed(contrast.point)} · paired 95% ${formatSignedCi(contrast.ci95)}`;
row.append(cell);
});
return row;
}));
}
};
all<HTMLButtonElement>("[data-role-layer]").forEach((button) => {
button.addEventListener("click", () => {
roleLayerNumber = Number(button.dataset.roleLayer);
renderRole();
});
});
all<HTMLButtonElement>("[data-role-scope]").forEach((button) => {
button.addEventListener("click", () => {
roleScope = button.dataset.roleScope ?? "target_content";
renderRole();
});
});
all<HTMLButtonElement>("[data-role-mode]").forEach((button) => {
button.addEventListener("click", () => {
roleMode = button.dataset.roleMode ?? "prompt_balanced";
renderRole();
});
});
all<HTMLButtonElement>("[data-role-contrast]").forEach((button) => {
button.addEventListener("click", () => {
roleContrast = button.dataset.roleContrast
?? "target_assistant_minus_official";
renderRole();
});
});
renderRole();
});
</script>
@@ -2629,7 +2969,7 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
.artifact-status b { color: var(--ink); font-size: .72rem; }
.artifact-tabs {
display: grid;
grid-template-columns: repeat(10, 1fr);
grid-template-columns: repeat(6, 1fr);
background: var(--ink);
}
.artifact-tabs button {
@@ -4286,6 +4626,316 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
font-size: .62rem;
line-height: 1.45;
}
.role-protocol {
margin-top: .8rem;
padding: 1rem;
border: 1px solid rgba(32,32,39,.15);
background:
linear-gradient(115deg, rgba(57,120,110,.08), transparent 52%),
#fffdf8;
}
.role-track {
display: flex;
align-items: center;
justify-content: center;
flex-wrap: wrap;
gap: .25rem;
padding: .8rem;
background: var(--ink);
color: white;
}
.role-track > * {
padding: .36rem .48rem;
font: 650 .61rem/1 var(--font-mono);
font-style: normal;
}
.role-track span {
color: rgba(255,255,255,.48);
font-size: .52rem;
}
.role-track b { background: rgba(255,255,255,.1); }
.role-track i { color: #d5d8ef; }
.role-track mark {
min-width: 4.6rem;
background: var(--amber);
color: white;
text-align: center;
}
.role-level-grid {
display: grid;
grid-template-columns: repeat(4, 1fr);
margin-top: .55rem;
border: 1px solid rgba(32,32,39,.12);
}
.role-level-grid article {
display: grid;
gap: .34rem;
padding: .75rem;
border-right: 1px solid rgba(32,32,39,.12);
background: #f3eee5;
}
.role-level-grid article:last-child { border-right: 0; }
.role-level-grid article.official { background: rgba(57,120,110,.11); }
.role-level-grid article.negative { background: rgba(98,105,155,.11); }
.role-level-grid span {
color: var(--blue);
font: 700 .52rem/1 var(--font-mono);
}
.role-level-grid b { font: 760 .84rem/1.1 var(--font-display); }
.role-level-grid code {
width: max-content;
padding: .22rem .3rem;
background: rgba(32,32,39,.08);
font-size: .57rem;
}
.role-level-grid p,
.role-protocol > p {
margin: 0;
color: rgba(32,32,39,.58);
font-size: .61rem;
line-height: 1.45;
}
.role-protocol > p {
margin-top: .7rem;
text-align: center;
}
.role-controls {
display: grid;
grid-template-columns: auto .9fr .9fr 1.55fr;
gap: .8rem;
align-items: end;
margin-top: .8rem;
padding: .85rem;
border: 1px solid rgba(32,32,39,.14);
background: #e8e2d7;
}
.role-controls > div { display: grid; gap: .45rem; }
.role-controls .layer-switch { margin: 0; }
.role-scope-switch,
.role-mode-switch,
.role-contrast-switch { display: flex; }
.role-scope-switch button,
.role-mode-switch button,
.role-contrast-switch button {
padding: .58rem .66rem;
border: 1px solid rgba(32,32,39,.22);
background: #fffdf8;
color: var(--ink);
font: 650 .6rem/1 var(--font-mono);
cursor: pointer;
}
.role-scope-switch button + button,
.role-mode-switch button + button,
.role-contrast-switch button + button { border-left: 0; }
.role-scope-switch button[aria-pressed="true"],
.role-mode-switch button[aria-pressed="true"],
.role-contrast-switch button[aria-pressed="true"] {
border-color: var(--blue);
background: var(--blue);
color: white;
}
.role-controls > p {
grid-column: 1 / -1;
margin: 0;
padding-top: .75rem;
border-top: 1px solid rgba(32,32,39,.12);
color: rgba(32,32,39,.62);
font-size: .69rem;
line-height: 1.5;
}
.role-domain-grid {
display: grid;
grid-template-columns: repeat(4, 1fr);
margin-top: .8rem;
border: 1px solid rgba(32,32,39,.14);
background: #fffdf8;
}
.role-domain-grid > :global(article) {
min-width: 0;
padding: .85rem;
border-right: 1px solid rgba(32,32,39,.12);
}
.role-domain-grid > :global(article:last-child) { border-right: 0; }
.role-domain-grid :global(.role-tv-ladder) {
display: grid;
grid-template-columns: repeat(4, 1fr);
gap: .18rem;
margin-top: .55rem;
}
.role-domain-grid :global(.role-tv-ladder > b) {
display: grid;
gap: .18rem;
min-width: 0;
padding: .34rem;
background: #e8e2d7;
}
.role-domain-grid :global(.role-tv-ladder > b.official) {
background: rgba(57,120,110,.12);
}
.role-domain-grid :global(.role-tv-ladder > b.causal) {
background: rgba(98,105,155,.1);
}
.role-domain-grid :global(.role-tv-ladder > b.selected) {
outline: 1px solid var(--blue);
outline-offset: -1px;
}
.role-domain-grid :global(.role-tv-ladder small) {
color: rgba(32,32,39,.5);
font: 650 .42rem/1 var(--font-mono);
}
.role-domain-grid :global(.role-tv-ladder strong) {
white-space: nowrap;
font: 720 .58rem/1 var(--font-mono);
letter-spacing: -.025em;
}
.role-domain-grid > :global(article > strong) {
display: inline-block;
margin-top: .48rem;
padding: .26rem .38rem;
font: 750 .58rem/1.15 var(--font-mono);
}
.role-domain-grid > :global(article > strong.down),
.role-depth :global(span.down) {
background: rgba(57,120,110,.13);
color: var(--teal);
}
.role-domain-grid > :global(article > strong.up),
.role-depth :global(span.up) {
background: rgba(161,77,77,.12);
color: var(--red);
}
.role-domain-grid > :global(article > strong.neutral),
.role-depth :global(span.neutral) {
background: rgba(186,118,44,.12);
color: var(--amber);
}
.role-domain-grid > :global(article > p),
.role-domain-grid > :global(article > small),
.role-domain-grid > :global(article > em),
.role-domain-grid > :global(article > u),
.role-domain-grid > :global(article > i) {
display: block;
margin: .38rem 0 0;
color: rgba(32,32,39,.57);
overflow-wrap: anywhere;
font: .54rem/1.4 var(--font-mono);
font-style: normal;
text-decoration: none;
}
.role-domain-grid > :global(article > em),
.role-domain-grid > :global(article > u),
.role-domain-grid > :global(article > i) {
padding-top: .34rem;
border-top: 1px solid rgba(32,32,39,.1);
}
.role-depth {
display: grid;
grid-template-columns: .52fr 1.48fr;
gap: 1rem;
margin-top: .8rem;
padding: 1rem;
border: 1px solid rgba(32,32,39,.14);
}
.role-depth h5 {
margin: .4rem 0;
font: 720 1rem/1.15 var(--font-display);
}
.role-depth p {
margin: 0;
color: rgba(32,32,39,.58);
font-size: .66rem;
line-height: 1.5;
}
.role-depth > :global([data-role-depth-map]) {
display: grid;
gap: .35rem;
}
.role-depth :global([data-role-depth-map] > div) {
display: grid;
grid-template-columns: 5.5rem repeat(6, 1fr);
gap: .25rem;
}
.role-depth :global([data-role-depth-map] > div > b),
.role-depth :global([data-role-depth-map] > div > span) {
display: grid;
align-items: center;
min-height: 2.2rem;
padding: .35rem;
font: 650 .55rem/1.2 var(--font-mono);
}
.role-depth :global([data-role-depth-map] > div > b) { color: var(--blue); }
.role-depth :global([data-role-depth-map] > div > span.down) {
background: color-mix(in srgb, var(--teal) calc(var(--strength) * 55%), #eef0e9);
color: var(--ink);
}
.role-depth :global([data-role-depth-map] > div > span.up) {
background: color-mix(in srgb, var(--red) calc(var(--strength) * 48%), #f3ebe6);
color: var(--ink);
}
.role-depth :global([data-role-depth-map] > div > span.neutral) {
background: rgba(186,118,44,.1);
color: var(--ink);
}
.role-causal-ledger {
display: grid;
grid-template-columns: repeat(3, 1fr);
margin-top: .8rem;
border: 1px solid rgba(32,32,39,.14);
background: #e8e2d7;
}
.role-causal-ledger article {
padding: .9rem;
border-right: 1px solid rgba(32,32,39,.12);
}
.role-causal-ledger article:last-child { border-right: 0; }
.role-causal-ledger article.exact { background: rgba(57,120,110,.11); }
.role-causal-ledger b {
display: block;
margin-top: .4rem;
font: 730 .78rem/1.2 var(--font-display);
}
.role-causal-ledger p {
margin: .4rem 0 0;
color: rgba(32,32,39,.58);
font-size: .62rem;
line-height: 1.45;
}
.role-batch-audit {
display: grid;
grid-template-columns: 1fr 1.35fr;
gap: .8rem;
margin-top: .8rem;
padding: 1rem;
border: 1px solid rgba(32,32,39,.14);
background: #fffdf8;
}
.role-batch-audit h5 {
margin: .45rem 0;
font: 720 .9rem/1.15 var(--font-display);
}
.role-batch-audit p {
margin: 0;
color: rgba(32,32,39,.58);
font-size: .62rem;
line-height: 1.5;
}
.role-batch-audit > div:last-child {
display: grid;
grid-template-columns: repeat(3, 1fr);
gap: .3rem;
}
.role-batch-audit > div:last-child span {
display: grid;
gap: .2rem;
padding: .55rem;
background: #e8e2d7;
}
.role-batch-audit > div:last-child span.exact {
background: rgba(57,120,110,.14);
}
.role-batch-audit i {
font: normal .55rem/1 var(--font-mono);
}
.observed-cache {
display: grid;
grid-template-columns: 1fr auto 1.25fr;
@@ -4531,7 +5181,10 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
.distance-controls,
.distance-depth,
.boundary-controls,
.boundary-depth { grid-template-columns: 1fr; }
.boundary-depth,
.role-controls,
.role-depth,
.role-batch-audit { grid-template-columns: 1fr; }
.artifact-status { grid-template-columns: 1fr 1fr; }
.artifact-tabs { grid-template-columns: 1fr 1fr; }
.route-controls { grid-template-columns: 1fr 1fr; }
@@ -4550,7 +5203,10 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
.history-buffer-summary,
.distance-domain-grid,
.boundary-domain-grid,
.boundary-token-grid { grid-template-columns: 1fr 1fr; }
.boundary-token-grid,
.role-domain-grid,
.role-level-grid,
.role-causal-ledger { grid-template-columns: 1fr 1fr; }
.distance-interpretation,
.boundary-interpretation { grid-template-columns: 1fr; }
.distance-interpretation article,
@@ -4598,7 +5254,11 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
.distance-interpretation,
.boundary-domain-grid,
.boundary-token-grid,
.boundary-interpretation { grid-template-columns: 1fr; }
.boundary-interpretation,
.role-ledger,
.role-domain-grid,
.role-level-grid,
.role-causal-ledger { grid-template-columns: 1fr; }
.route-metrics article,
.cache-ratio article,
.load-lessons article,
@@ -4617,7 +5277,11 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
.distance-interpretation article,
.boundary-domain-grid > :global(article),
.boundary-token-grid article,
.boundary-interpretation article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
.boundary-interpretation article,
.role-ledger article,
.role-domain-grid > :global(article),
.role-level-grid article,
.role-causal-ledger article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.12); }
.corpus-mode-switch,
.corpus-cohort-switch,
.template-scope-switch,
@@ -4630,7 +5294,10 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
.distance-contrast-switch,
.boundary-scope-switch,
.boundary-mode-switch,
.boundary-contrast-switch { display: grid; grid-template-columns: 1fr; }
.boundary-contrast-switch,
.role-scope-switch,
.role-mode-switch,
.role-contrast-switch { display: grid; grid-template-columns: 1fr; }
.corpus-mode-switch button + button,
.corpus-cohort-switch button + button,
.template-scope-switch button + button,
@@ -4643,7 +5310,10 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
.distance-contrast-switch button + button,
.boundary-scope-switch button + button,
.boundary-mode-switch button + button,
.boundary-contrast-switch button + button { border-left: 1px solid rgba(32,32,39,.22); border-top: 0; }
.boundary-contrast-switch button + button,
.role-scope-switch button + button,
.role-mode-switch button + button,
.role-contrast-switch button + button { border-left: 1px solid rgba(32,32,39,.22); border-top: 0; }
.length-delta-grid > :global(article),
.length-pair-summary article { border-right: 0; border-bottom: 1px solid rgba(32,32,39,.11); }
.artifact-boundary { grid-template-columns: 1fr; }
@@ -4665,6 +5335,8 @@ const shardFraction = trace.provenance.shard_1_bytes / trace.provenance.checkpoi
.history-depth > :global([data-history-depth-map]) { min-width: 620px; }
.boundary-depth { overflow-x: auto; }
.boundary-depth > :global([data-boundary-depth-map]) { min-width: 620px; }
.role-depth { overflow-x: auto; }
.role-depth > :global([data-role-depth-map]) { min-width: 620px; }
.layer-evidence { grid-template-columns: repeat(7, 1fr); }
.repro-gate { grid-template-columns: 1fr; }
.repro-gate > p { grid-column: auto; }
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+5 -5
View File
@@ -37,7 +37,7 @@ const toc = [
<BaseLayout
title="DeepSeek 技术谱系与真实权重深读:从 Dense、MoE、MLA 到 R1 与 V4"
description="用二十四张问题账、十次技术转向、十四个交互实验、真实 V2-Lite 权重、公开语料路由区间、官方模板、消息历史、等长 filler 与单 EOS-ID 边界控制、吸收式缓存 trace 和六十个一手节点,完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
description="用二十四张问题账、十次技术转向、十五个交互实验、真实 V2-Lite 权重、公开语料路由区间、官方模板、消息历史、等长 filler、单 EOS-ID 与角色词头控制、吸收式缓存 trace 和六十个一手节点,完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
section="deepseek"
>
<header class="page-hero deepseek-hero">
@@ -55,7 +55,7 @@ const toc = [
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
<div><dt>LABS</dt><dd>14 个可操作实验</dd></div>
<div><dt>LABS</dt><dd>15 个可操作实验</dd></div>
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
<div><dt>STATUS</dt><dd>三轮 · 真实权重执行</dd></div>
</dl>
@@ -768,15 +768,15 @@ const toc = [
<p class="eyebrow"><span>22</span> OFFICIAL WEIGHTS / EXECUTED</p>
<h2>从“MLA 与 MoE 的概念”再往前一步:让官方 V2-Lite 权重真的跑起来</h2>
<p class="lede">
前面的四联实验负责建立公式与角色合同;下面的十联工件实验固定官方 revision、tokenizer、
前面的四联实验负责建立公式与角色合同;下面的十一联工件实验固定官方 revision、tokenizer、
模型代码和 checkpoint 第一分片,在 RTX 5090 上连续执行 layer 0–6。它把真实观测、shape 推导、
吸收式 latent cache、长度对照、官方 chat-template 扰动、实现差距和未覆盖范围放在同一张证据图里。
</p>
<div class="artifact-callout">
<article><span>X / FORWARD</span><b>7 / 27 layers</b><p>1 个 dense 层 + 6 个 MoE 层;layer 7 因跨分片停止。</p></article>
<article><span>X / ROUTES</span><b>5,157,072</b><p>三档长度、模板、消息历史、等长 filler 与单 EOS-ID 八格控制的真实 top-6 选择。</p></article>
<article><span>X / ROUTES</span><b>7,201,296</b><p>三档长度、模板、消息历史、等长 filler、单 EOS-ID 与角色词头八格控制的真实 top-6 选择。</p></article>
<article><span>X / ABSORB CACHE</span><b>266,240 → 29,952 B</b><p>同一真实 layer-1 权重的 naive / absorb active buffers。</p></article>
<article><span>X / RERUN</span><b>7 / 7 EXACT</b><p>三档长度、官方模板、历史因子、等长 filler 与边界 ID 控制均 byte-exact;比较使用 paired prompt bootstrap。</p></article>
<article><span>X / RERUN</span><b>8 / 8 EXACT</b><p>三档长度、官方模板、历史因子、等长 filler、边界与角色词头控制均 byte-exact;比较使用 paired prompt bootstrap。</p></article>
</div>
<DeepSeekArtifactLab />
</section>
+7 -4
View File
@@ -15,7 +15,7 @@ const workstreams = [
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM,加入真实去重与 mixture traces" },
{ label: "DeepSeek 专题", value: 96, next: "角色标记与 special-token family、V2-Lite-Chat 行为、完整 27 层与固定 batch shape,再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL" },
{ label: "DeepSeek 专题", value: 97, next: "special-token family 与完整两-token 角色块、V2-Lite-Chat 行为、完整 27 层与固定 batch content,再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL" },
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
@@ -97,12 +97,12 @@ const workstreams = [
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全,包含先修依赖和三条贯穿案例。</p></article>
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
<article><span>✓</span><h3>八十一个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验,DeepSeek 四联公式实验与十联真实权重实验,以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
<article><span>✓</span><h3>八十二个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验,DeepSeek 四联公式实验与十一联真实权重实验,以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系,以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上,新增 V2-Lite 7/27 层连续 forward、官方 V3 absorb、长度/模板、system × one-shot、等长 filler 与 EOS 单 token 边界控制;累计 5,157,072 次真实路由。最新八格实验固定长度、位置、角色与 batch,仅替换一个 input ID,两份 54,254,445-byte JSON 的 SHA-256 同为 9bb93834…b9c37。</p></article>
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上,新增 V2-Lite 7/27 层连续 forward、官方 V3 absorb、长度/模板、system × one-shot、等长 filler、EOS 与角色词头单 token 控制;累计 7,201,296 次真实路由。最新八格显示前置词头有约 .02–.03 direct TV,却没有统一 system 调制;后置改动在 34,488 个目标 ordered top-6 上 causal-exact,两份 53,440,884-byte JSON 的 SHA-256 同为 9dc0e37f…b8caf。</p></article>
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链,以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions,审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes;四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel;6/6 官方参考逐元素相等,并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
@@ -134,7 +134,7 @@ const workstreams = [
<div class="queue-table">
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>角色标记 / special-token family / V2-Lite-Chat 行为 → 完整 27 层与固定 batch shape → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>special-token family / 完整两-token 角色块 / V2-Lite-Chat 行为 → 完整 27 层与固定 batch content → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
@@ -220,6 +220,9 @@ const workstreams = [
<div><time>2026-07-29</time><b>三类 cohort 永久分身份</b><p>自然长度回答本批样本如何路由;matched-16 / 24 回答同一 prompt 多看 8 tokens 后如何变化,不把二者混成内容因果。</p></div>
<div><time>2026-07-29</time><b>历史边界用单 ID 替换而非删除</b><p>EOS→x / 句点 / 换行保持长度、目标位置、角色标记、mask 与同一 batch;识别一个输入 ID 的干预,不冒充移除了全部回合结构。</p></div>
<div><time>2026-07-29</time><b>base、Chat 与行为永久分层</b><p>V2-Lite base 的路由 TV 不是回合理解或能力指标;`User:` 仍在,反事实不是官方合法 chat,后续另跑 Chat checkpoint 与生成指标。</p></div>
<div><time>2026-07-29</time><b>角色词头 direct effect 与 system 调制分开</b><p>`User→Assistant/x` 都直接改变后续路由,但 24 格 system-edge 方向混合,不能偷换成统一强化或削弱。</p></div>
<div><time>2026-07-29</time><b>目标后改动承担因果负对照</b><p>suffix `Assistant→User` 在 34,488 个目标 ordered top-6 上全 exact;完整输入仍保留 suffix 自身作用。</p></div>
<div><time>2026-07-29</time><b>BF16 batch content 进入复现合同</b><p>相同 official token IDs 与 32-row shape 不保证深层 route hash 相同;companion rows 的构成也必须记录。</p></div>
</div>
</section>