feat: factor DeepSeek boundary and role blocks
This commit is contained in:
wuyang
+15
-3
@@ -14,7 +14,7 @@
|
|||||||
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
||||||
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
||||||
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
||||||
| DeepSeek 专题 | 三轮实证进行中 | 97% | special-token family 与完整两-token 角色块、V2-Lite-Chat 行为、完整 27 层与固定 batch-content 对照，再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL 复现 |
|
| DeepSeek 专题 | 三轮实证进行中 | 98% | V2-Lite-Chat 生成/行为对照、完整 27 层与固定 batch-content 对照，再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL 复现 |
|
||||||
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
||||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||||
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
||||||
@@ -41,7 +41,7 @@
|
|||||||
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||||
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
||||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十五联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十二个原创交互视图。
|
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 十七联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十四个原创交互视图。
|
||||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||||
@@ -224,11 +224,20 @@
|
|||||||
- [x] 跨实验 BF16 batch-content 审计：当前 official 与上一轮 EOS 条件 256 / 256 token-ID hashes exact，Layer 1 target route hash 256 / 256 exact，但 L2–L6 随 companion rows 改变而分化；正式结论只使用本轮八格组内比较，不把相同 batch shape 误写成相同数值执行。
|
- [x] 跨实验 BF16 batch-content 审计：当前 official 与上一轮 EOS 条件 256 / 256 token-ID hashes exact，Layer 1 target route hash 256 / 256 exact，但 L2–L6 随 companion rows 改变而分化；正式结论只使用本轮八格组内比较，不把相同 batch shape 误写成相同数值执行。
|
||||||
- [x] 角色词头里程碑本地闸门通过：71 个 Astro 文件零诊断、21 个页面、1,151 个站内引用、12 个跨页锚点零失败；十一页签桌面与 390px 移动端无运行时异常或文档级横向溢出，十六套真实 Chrome 回归全部通过。
|
- [x] 角色词头里程碑本地闸门通过：71 个 Astro 文件零诊断、21 个页面、1,151 个站内引用、12 个跨页锚点零失败；十一页签桌面与 390px 移动端无运行时异常或文档级横向溢出，十六套真实 Chrome 回归全部通过。
|
||||||
- [x] DeepSeek 角色词头单 token 控制以源提交 `ea49ed0`、不可变镜像 `20260729T124516Z-ea49ed0` 发布；OCI digest `sha256:fa8910ee…0c0bf`，复用 NAS `12010→8080`、NPM host 31 / cert 41、门户 `LLM ATLAS / projects / 180`，十六套生产 Chrome 回归全通过；保留 `20260729T120248Z-f4d4a05` 回滚。
|
- [x] DeepSeek 角色词头单 token 控制以源提交 `ea49ed0`、不可变镜像 `20260729T124516Z-ea49ed0` 发布；OCI digest `sha256:fa8910ee…0c0bf`，复用 NAS `12010→8080`、NPM host 31 / cert 41、门户 `LLM ATLAS / projects / 180`，十六套生产 Chrome 回归全通过；保留 `20260729T120248Z-f4d4a05` 回滚。
|
||||||
|
- [x] 特殊 token 家族控制核清固定 tokenizer 的完整库存：`all_special_ids=[100000,100001]`，BOS/EOS 为 2 / 2 全枚举，PAD 与 EOS 共用 ID；`x` / 句点只作为两个选定普通对照，不把四 ID 的 2-vs-2 描述性汇总冒充总体类别推断。
|
||||||
|
- [x] 特殊 token 八格固定同一 128-source / 23-content-token cohort、重复词元历史、角色块、长度、目标位置、mask 与 32-row batch；256 / 256 official 零改动、768 / 768 counterfactual 恰好一 ID。目标内容 system-edge TV 均值为 EOS `.037518`、BOS `.048018`、x `.053975`、句点 `.055289`；BOS−EOS 为 22↑2↓，两个普通对照相对 EOS 均为 24↑0↓。
|
||||||
|
- [x] 特殊 token 四 ID 的 selected-ordinary mean − complete-special mean 为 `+.011864`，24 / 24 点估计为正、20 个 paired CI 全正；该数字永久标为四 ID 描述性结果。BOS direct S1−S0、ΔCV 与 full-input 方向更混合，主结论只落在精确对齐目标内容。
|
||||||
|
- [x] 特殊 token 正式运行与独立复跑各 54,440,827 bytes、SHA-256 均为 `c372c1b0…4af5bf`，byte-exact；新增 2,044,224 次真实路由，使累计达到 9,245,520 次。跨上一轮共享三水平 token 合同 768 / 768 exact，L1 route 全 exact，L2–L6 随 companion rows 分化。
|
||||||
|
- [x] 完整角色块把 `User:` / `Assistant:` 还原成 `[head, delimiter]` 两个普通 IDs：在 system 0/1 × head User/Assistant × delimiter `:`/`x` 八格中固定官方 EOS、generation suffix、长度、目标位置与 batch；256 个零改动、512 个单 ID 与 256 个双 ID edit contract 全 exact。
|
||||||
|
- [x] 完整角色块目标内容 system-edge TV 四条件均值为 `.037304/.037015/.037239/.036220`；head main、delimiter main、interaction 分别为 `−.000654/−.000430/−.000729`，绝大多数 paired CI 跨零且出现方向相反的局部单元，不支持统一 system 调制。
|
||||||
|
- [x] 四条直接因子边均非零；`head@x − head@colon` 在 S0/S1 跨 24 格均值为 `−.003673/−.001960`，多数格说明单个 `x` 对照会略微缩小 User-vs-Assistant direct TV，但 system DiD 混合，不泛化成标点或角色语义。
|
||||||
|
- [x] 完整角色块正式运行与独立复跑各 66,975,110 bytes、SHA-256 均为 `a703dddb…7e82`，byte-exact；新增 2,044,224 次真实路由，使累计达到 11,289,744 次。两组 compact 产物分别约 1.4 / 3.3 MiB，累计正式实验集达到 10 / 10 byte-exact。
|
||||||
|
- [x] special family + 完整角色块本地闸门通过：73 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；十三页签桌面与 390px 移动端无运行时异常或文档级横向溢出，十六套真实 Chrome 专题回归全部通过。
|
||||||
|
|
||||||
## 正在进行
|
## 正在进行
|
||||||
|
|
||||||
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
||||||
- [ ] DeepSeek 三轮下一闸门：控制 special-token family 与完整 `User:` / `Assistant:` 两-token 角色块，加入 V2-Lite-Chat 生成/行为对照；扩到完整 27 层并固定 batch content 审计，再推进 SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
- [ ] DeepSeek 三轮下一闸门：加入 V2-Lite-Chat 生成/行为对照；扩到完整 27 层并固定 batch content 审计，再推进 SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||||
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
||||||
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
||||||
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||||
@@ -379,6 +388,9 @@
|
|||||||
| 2026-07-29 | 目标后的角色词头承担严格 causal negative control | suffix `Assistant→User` 在 34,488 个目标 ordered top-6 上全 exact，目标 TV/JSD/ΔCV 为零；完整输入保留 suffix 自身差异 |
|
| 2026-07-29 | 目标后的角色词头承担严格 causal negative control | suffix `Assistant→User` 在 34,488 个目标 ordered top-6 上全 exact，目标 TV/JSD/ΔCV 为零；完整输入保留 suffix 自身差异 |
|
||||||
| 2026-07-29 | BF16 复现合同从 batch shape 扩展到 batch content | official token IDs 与 32-row shape 相同仍不足以保证深层 route hash 相同；companion rows 与数值执行路径也进入边界说明 |
|
| 2026-07-29 | BF16 复现合同从 batch shape 扩展到 batch content | official token IDs 与 32-row shape 相同仍不足以保证深层 route hash 相同；companion rows 与数值执行路径也进入边界说明 |
|
||||||
| 2026-07-29 | DeepSeek 角色词头控制以 `20260729T124516Z-ea49ed0` 发布 | OCI digest `sha256:fa8910ee…0c0bf`；复用 NAS 12010→8080、NPM 31 / cert 41、门户 order 180；十六套生产 Chrome 回归通过，保留上一不可变镜像回滚 |
|
| 2026-07-29 | DeepSeek 角色词头控制以 `20260729T124516Z-ea49ed0` 发布 | OCI digest `sha256:fa8910ee…0c0bf`；复用 NAS 12010→8080、NPM 31 / cert 41、门户 order 180；十六套生产 Chrome 回归通过，保留上一不可变镜像回滚 |
|
||||||
|
| 2026-07-29 | special inventory 与 ordinary controls 永久分统计身份 | BOS/EOS 是固定 tokenizer 的完整 2-ID 库；x/句点只是两个选定普通对照，2-vs-2 只描述四 ID，不推断普通词元总体 |
|
||||||
|
| 2026-07-29 | 角色块按 head × delimiter 分解 | `User:`/`Assistant:` 都是两个普通 IDs；head、delimiter、interaction 与四条 direct edges 分账，不把单位置作用冒充完整角色语义 |
|
||||||
|
| 2026-07-29 | target-content 与 full-input 永久分因果身份 | 精确对齐目标只观察编辑后的下游传播；完整输入包含被编辑 token 自身，因此方向变化不是主结果矛盾 |
|
||||||
| 2026-07-29 | K3 二轮按 32 张对象账与完整报告顺序重建 | total/active、2.5×、KDA state、深度来源、专家路由、视觉目标、轨迹、缓存与评测协议不再压成一页组件摘要 |
|
| 2026-07-29 | K3 二轮按 32 张对象账与完整报告顺序重建 | total/active、2.5×、KDA state、深度来源、专家路由、视觉目标、轨迹、缓存与评测协议不再压成一页组件摘要 |
|
||||||
| 2026-07-29 | K3 原生视觉事实回到 §2.4 / §3.3 核验 | 删除“先冻结语言模型再解冻”旧表述；明确 MoonViT-V2 从头训练，视觉/文本从开始共同 NTP |
|
| 2026-07-29 | K3 原生视觉事实回到 §2.4 / §3.3 核验 | 删除“先冻结语言模型再解冻”旧表述；明确 MoonViT-V2 从头训练，视觉/文本从开始共同 NTP |
|
||||||
| 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项；作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 |
|
| 2026-07-29 | K3 Figure 1–16 / Table 1–5 全部建立课程视觉契约 | 每张图同时写支持范围与不可外推项；作者报告、论文、推导与 toy model 使用 R/P/D/T 标签 |
|
||||||
|
|||||||
@@ -19,7 +19,7 @@
|
|||||||
|
|
||||||
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
||||||
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
||||||
以及 82 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
以及 84 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||||
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
||||||
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
||||||
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
||||||
@@ -28,7 +28,7 @@
|
|||||||
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
||||||
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
||||||
DeepSeek 三轮专题以 24 张问题账、10 次技术转向、
|
DeepSeek 三轮专题以 24 张问题账、10 次技术转向、
|
||||||
15 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
17 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||||
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
||||||
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
|
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
|
||||||
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
|
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
|
||||||
@@ -56,7 +56,9 @@ EOS，换行为 23 / 24，但 base checkpoint、非法反事实序列和无行
|
|||||||
`.02–.03`，说明一个词头 ID 足以条件化后续专家集合；但二者对 system-edge 的 24 格方向
|
`.02–.03`，说明一个词头 ID 足以条件化后续专家集合；但二者对 system-edge 的 24 格方向
|
||||||
分别为 12↑12↓ / 13↑11↓，不能写成统一的 system 调制。后置 `Assistant→User`
|
分别为 12↑12↓ / 13↑11↓，不能写成统一的 system 调制。后置 `Assistant→User`
|
||||||
则在 34,488 / 34,488 个目标 ordered top-6 上 exact，direct TV / JSD / ΔCV 全为零，
|
则在 34,488 / 34,488 个目标 ordered top-6 上 exact，direct TV / JSD / ΔCV 全为零，
|
||||||
形成严格 causal suffix 负对照。当前累计 7,201,296 次公开语料路由。FlashMLA 的
|
形成严格 causal suffix 负对照。进一步穷尽 tokenizer 的 BOS/EOS special inventory，
|
||||||
|
并以 `User/Assistant × :/x` 完成两-token 角色块因子分解；两组实验各新增 2,044,224
|
||||||
|
次真实路由、完整 JSON 独立复跑 byte-exact。当前累计 11,289,744 次公开语料路由。FlashMLA 的
|
||||||
SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
|
SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
|
||||||
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
|
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
|
||||||
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
|
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
|
||||||
@@ -66,7 +68,9 @@ SM90/SM100 官方支持矩阵与本机 SM120 边界单独记账。详见
|
|||||||
[DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md)、
|
[DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_FACTORIAL_AUDIT.md)、
|
||||||
[DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md) 与
|
[DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_DISTANCE_CONTROL_AUDIT.md) 与
|
||||||
[DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md) 与
|
[DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md](./research/DEEPSEEK_ROUTING_HISTORY_BOUNDARY_TOKEN_AUDIT.md) 与
|
||||||
[DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md](./research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md)。
|
[DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md](./research/DEEPSEEK_ROUTING_ROLE_MARKER_HEAD_AUDIT.md)、
|
||||||
|
[DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md](./research/DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md) 与
|
||||||
|
[DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md](./research/DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md)。
|
||||||
其余专题按进度账本持续扩建。
|
其余专题按进度账本持续扩建。
|
||||||
|
|
||||||
## 本地开发
|
## 本地开发
|
||||||
|
|||||||
@@ -425,3 +425,109 @@ target-token ordered top-6 routes, with zero target TV, JSD, and ΔCV. See
|
|||||||
depth maps, token-level alignment, cross-experiment BF16 batch-content audit,
|
depth maps, token-level alignment, cross-experiment BF16 batch-content audit,
|
||||||
primary sources, and the boundary against full role semantics or Chat-model
|
primary sources, and the boundary against full role semantics or Chat-model
|
||||||
behavior.
|
behavior.
|
||||||
|
|
||||||
|
## Special-token family control
|
||||||
|
|
||||||
|
`v2_lite_routing_special_token_family_control.py` keeps the same repeated-token
|
||||||
|
history and changes the completed assistant boundary to four single IDs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
system off/on × EOS / BOS / x / period
|
||||||
|
```
|
||||||
|
|
||||||
|
The pinned tokenizer has exactly two special-token IDs: BOS `100000` and EOS
|
||||||
|
`100001`; PAD aliases EOS. EOS/BOS therefore exhaust the special inventory,
|
||||||
|
while `x` and period are only two selected ordinary controls. The 2-vs-2
|
||||||
|
family summary is descriptive for these four IDs and is not a
|
||||||
|
population-level specialness claim.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||||
|
python -B experiments/deepseek/v2_lite_routing_special_token_family_control.py \
|
||||||
|
--artifact-dir /path/to/deepseek-v2-lite \
|
||||||
|
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||||
|
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||||
|
--tnews /path/to/tnews/test.json \
|
||||||
|
--tnews-archive /path/to/tnews_public.zip \
|
||||||
|
--wikitext /path/to/wikitext-validation.parquet \
|
||||||
|
--output src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||||
|
--per-domain 32 \
|
||||||
|
--content-tokens 23 \
|
||||||
|
--batch-prompts 4 \
|
||||||
|
--layers 7 \
|
||||||
|
--bootstrap 2000 \
|
||||||
|
--seed 20260729 \
|
||||||
|
--captured-at 2026-07-29T12:57:00+00:00
|
||||||
|
```
|
||||||
|
|
||||||
|
The eight cells add 2,044,224 real top-6 route selections. All 256
|
||||||
|
source×system groups preserve length and target position, and all 768
|
||||||
|
counterfactuals change exactly one ID. The committed run and independent
|
||||||
|
rerun are byte-exact:
|
||||||
|
|
||||||
|
```text
|
||||||
|
c372c1b03a8b15f615b54ded5d9257a8fc2cdb7728001735d3d4c1d8534af5bf
|
||||||
|
```
|
||||||
|
|
||||||
|
For exact target content under prompt-balanced aggregation, mean system-edge
|
||||||
|
TV is `.037518 / .048018 / .053975 / .055289` for EOS / BOS / x / period.
|
||||||
|
BOS−EOS is positive in 22/24 layer×domain cells; x−EOS and period−EOS are
|
||||||
|
positive in 24/24. The selected ordinary-control mean exceeds the complete
|
||||||
|
special-inventory mean by `.011864` in these four IDs only. See
|
||||||
|
`research/DEEPSEEK_ROUTING_SPECIAL_TOKEN_FAMILY_AUDIT.md` for all paired
|
||||||
|
intervals, direct edges, scope split, alignment, batch-content audit,
|
||||||
|
literature context, and non-claims.
|
||||||
|
|
||||||
|
## Full two-token role-marker factorial
|
||||||
|
|
||||||
|
`v2_lite_routing_role_marker_block_factorial.py` treats the pre-target
|
||||||
|
two-token marker as two independent factors:
|
||||||
|
|
||||||
|
```text
|
||||||
|
system off/on × head User/Assistant × delimiter colon/x
|
||||||
|
|
||||||
|
User: [5726, 25] Assistant: [77398, 25]
|
||||||
|
User x [5726, 87] Assistant x [77398, 87]
|
||||||
|
```
|
||||||
|
|
||||||
|
The official assistant EOS and post-target generation suffix remain unchanged.
|
||||||
|
The four blocks have zero, one, one, and two edited IDs relative to official
|
||||||
|
`User:`; every edit is verified while length, target position, attention mask,
|
||||||
|
and 32-row batch shape stay fixed.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=/path/to/transformers-4.41.2-deps:/usr/lib/python3/dist-packages \
|
||||||
|
python -B experiments/deepseek/v2_lite_routing_role_marker_block_factorial.py \
|
||||||
|
--artifact-dir /path/to/deepseek-v2-lite \
|
||||||
|
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||||
|
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||||
|
--tnews /path/to/tnews/test.json \
|
||||||
|
--tnews-archive /path/to/tnews_public.zip \
|
||||||
|
--wikitext /path/to/wikitext-validation.parquet \
|
||||||
|
--output src/data/deepseek-v2-lite-routing-role-marker-block-factorial.json \
|
||||||
|
--per-domain 32 \
|
||||||
|
--content-tokens 23 \
|
||||||
|
--batch-prompts 4 \
|
||||||
|
--layers 7 \
|
||||||
|
--bootstrap 2000 \
|
||||||
|
--seed 20260729 \
|
||||||
|
--captured-at 2026-07-29T13:06:00+00:00
|
||||||
|
```
|
||||||
|
|
||||||
|
The eight cells add 2,044,224 real top-6 route selections. The 256 official,
|
||||||
|
512 one-ID, and 256 two-ID cells all pass their exact edit contracts. The
|
||||||
|
committed run and independent rerun are byte-exact:
|
||||||
|
|
||||||
|
```text
|
||||||
|
a703dddb6d04182b1a608c213bfd341cfc32e1801be42c417dca30a505087e82
|
||||||
|
```
|
||||||
|
|
||||||
|
For exact target content under prompt-balanced aggregation, the four mean
|
||||||
|
system-edge TVs are `.037304 / .037015 / .037239 / .036220`. The head,
|
||||||
|
delimiter, and head×delimiter effects are small and mixed across the 24
|
||||||
|
layer×domain cells. Direct head and delimiter edges remain nonzero; replacing
|
||||||
|
colon with the single `x` control modestly reduces User-vs-Assistant direct TV
|
||||||
|
in most cells, without a uniform system modulation. See
|
||||||
|
`research/DEEPSEEK_ROUTING_ROLE_MARKER_BLOCK_AUDIT.md` for the exact factor
|
||||||
|
coding, intervals, direct dependencies, full-input split, route alignment,
|
||||||
|
cross-batch audit, sources, and non-claims.
|
||||||
|
|||||||
@@ -737,6 +737,9 @@ def finalize_result(path: Path) -> dict[str, Any]:
|
|||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
install_control_contract()
|
install_control_contract()
|
||||||
|
if {"-h", "--help"} & set(sys.argv[1:]):
|
||||||
|
base.main()
|
||||||
|
return
|
||||||
output = output_path_from_argv()
|
output = output_path_from_argv()
|
||||||
with open(os.devnull, "w", encoding="utf-8") as sink:
|
with open(os.devnull, "w", encoding="utf-8") as sink:
|
||||||
with contextlib.redirect_stdout(sink):
|
with contextlib.redirect_stdout(sink):
|
||||||
|
|||||||
@@ -0,0 +1,894 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Run a 2 x 2 x 2 target role-marker-block control on DeepSeek-V2-Lite.
|
||||||
|
|
||||||
|
The official target role block is two ordinary token IDs:
|
||||||
|
|
||||||
|
User : -> [5726, 25]
|
||||||
|
|
||||||
|
This experiment crosses system off/on with a complete two-token block:
|
||||||
|
|
||||||
|
role head: User / Assistant
|
||||||
|
delimiter: colon / x
|
||||||
|
|
||||||
|
The four blocks are therefore ``User:``, ``Assistant:``, ``User x``, and
|
||||||
|
``Assistant x``. The official EOS, repeated-token history, target content,
|
||||||
|
generation-prompt ``Assistant:``, sequence length, target position, attention
|
||||||
|
mask, and 32-row padded batch shape are fixed. One cell is an official chat
|
||||||
|
serialization; the other three are explicit pre-target token-ID
|
||||||
|
counterfactuals.
|
||||||
|
|
||||||
|
The design identifies role-head, delimiter, and head-by-delimiter routing
|
||||||
|
effects inside this fixed base-checkpoint forward contract. It does not
|
||||||
|
identify complete role semantics, Chat/SFT behavior, answer quality, or a
|
||||||
|
population-level punctuation effect from one chosen delimiter control.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
import v2_lite_routing_role_marker_head_control as role_prior
|
||||||
|
|
||||||
|
|
||||||
|
boundary_prior = role_prior.prior
|
||||||
|
base = role_prior.base
|
||||||
|
|
||||||
|
SYSTEM_MESSAGE = role_prior.SYSTEM_MESSAGE
|
||||||
|
FILLER_USER = role_prior.FILLER_USER
|
||||||
|
FILLER_ASSISTANT = role_prior.FILLER_ASSISTANT
|
||||||
|
|
||||||
|
BLOCK_LEVELS = (
|
||||||
|
"user_colon",
|
||||||
|
"assistant_colon",
|
||||||
|
"user_x",
|
||||||
|
"assistant_x",
|
||||||
|
)
|
||||||
|
REFERENCE_LEVEL = "user_colon"
|
||||||
|
LEVEL_FACTORS = {
|
||||||
|
"user_colon": {"head": "user", "delimiter": "colon"},
|
||||||
|
"assistant_colon": {"head": "assistant", "delimiter": "colon"},
|
||||||
|
"user_x": {"head": "user", "delimiter": "x"},
|
||||||
|
"assistant_x": {"head": "assistant", "delimiter": "x"},
|
||||||
|
}
|
||||||
|
CONDITIONS = tuple(
|
||||||
|
f"s{system}_{level}"
|
||||||
|
for level in BLOCK_LEVELS
|
||||||
|
for system in (0, 1)
|
||||||
|
)
|
||||||
|
FACTORS = {
|
||||||
|
condition: {
|
||||||
|
"system": int(condition[1]),
|
||||||
|
"history": "filler",
|
||||||
|
"assistant_boundary": "official_eos",
|
||||||
|
"role_block": condition.split("_", 1)[1],
|
||||||
|
**LEVEL_FACTORS[condition.split("_", 1)[1]],
|
||||||
|
}
|
||||||
|
for condition in CONDITIONS
|
||||||
|
}
|
||||||
|
SYSTEM_CELLS = {
|
||||||
|
level: (f"s0_{level}", f"s1_{level}")
|
||||||
|
for level in BLOCK_LEVELS
|
||||||
|
}
|
||||||
|
SYSTEM_EDGE_CONTRASTS = {
|
||||||
|
f"{level}_minus_user_colon": (REFERENCE_LEVEL, level)
|
||||||
|
for level in BLOCK_LEVELS
|
||||||
|
if level != REFERENCE_LEVEL
|
||||||
|
}
|
||||||
|
COMPARISONS = tuple(
|
||||||
|
[
|
||||||
|
(f"system_{level}", f"s0_{level}", f"s1_{level}")
|
||||||
|
for level in BLOCK_LEVELS
|
||||||
|
]
|
||||||
|
+ [
|
||||||
|
(
|
||||||
|
f"{level}_at_s{system}",
|
||||||
|
f"s{system}_{REFERENCE_LEVEL}",
|
||||||
|
f"s{system}_{level}",
|
||||||
|
)
|
||||||
|
for level in BLOCK_LEVELS
|
||||||
|
if level != REFERENCE_LEVEL
|
||||||
|
for system in (0, 1)
|
||||||
|
]
|
||||||
|
+ [
|
||||||
|
(
|
||||||
|
f"head_at_x_s{system}",
|
||||||
|
f"s{system}_user_x",
|
||||||
|
f"s{system}_assistant_x",
|
||||||
|
)
|
||||||
|
for system in (0, 1)
|
||||||
|
]
|
||||||
|
+ [
|
||||||
|
(
|
||||||
|
f"delimiter_at_assistant_s{system}",
|
||||||
|
f"s{system}_assistant_colon",
|
||||||
|
f"s{system}_assistant_x",
|
||||||
|
)
|
||||||
|
for system in (0, 1)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
ALIGNMENT_COMPARISONS = COMPARISONS
|
||||||
|
|
||||||
|
RENDER_AUDIT: list[dict[str, Any]] = []
|
||||||
|
ROLE_TOKEN_IDS: dict[str, int] = {}
|
||||||
|
ORIGINAL_BOUNDARY_DOMAIN = boundary_prior.boundary_control_domain
|
||||||
|
|
||||||
|
|
||||||
|
def condition_messages(
|
||||||
|
content: str,
|
||||||
|
condition: str,
|
||||||
|
) -> list[dict[str, str]]:
|
||||||
|
factors = FACTORS[condition]
|
||||||
|
messages: list[dict[str, str]] = []
|
||||||
|
if factors["system"]:
|
||||||
|
messages.append({"role": "system", "content": SYSTEM_MESSAGE})
|
||||||
|
messages.extend(
|
||||||
|
[
|
||||||
|
{"role": "user", "content": FILLER_USER},
|
||||||
|
{"role": "assistant", "content": FILLER_ASSISTANT},
|
||||||
|
{"role": "user", "content": content},
|
||||||
|
]
|
||||||
|
)
|
||||||
|
return messages
|
||||||
|
|
||||||
|
|
||||||
|
def role_token_ids(tokenizer: Any) -> dict[str, int]:
|
||||||
|
ids = role_prior.role_token_ids(tokenizer)
|
||||||
|
if ids != {
|
||||||
|
"user": 5726,
|
||||||
|
"assistant": 77398,
|
||||||
|
"x": 87,
|
||||||
|
"colon": 25,
|
||||||
|
"eos": 100001,
|
||||||
|
}:
|
||||||
|
raise RuntimeError(f"pinned role token contract changed: {ids}")
|
||||||
|
return ids
|
||||||
|
|
||||||
|
|
||||||
|
def render_role_block_variant(
|
||||||
|
tokenizer: Any,
|
||||||
|
content: str,
|
||||||
|
condition: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
messages = condition_messages(content, condition)
|
||||||
|
rendered = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
tokenize=False,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
)
|
||||||
|
official_ids = list(
|
||||||
|
tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
tokenize=True,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
token_ids, offsets = base.tokenize_with_offsets(
|
||||||
|
tokenizer,
|
||||||
|
rendered,
|
||||||
|
add_special_tokens=False,
|
||||||
|
)
|
||||||
|
if token_ids != official_ids:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"{condition} rendered IDs differ from apply_chat_template"
|
||||||
|
)
|
||||||
|
|
||||||
|
ids = role_token_ids(tokenizer)
|
||||||
|
ROLE_TOKEN_IDS.update(ids)
|
||||||
|
content_start = rendered.rfind(content)
|
||||||
|
if content_start < 0:
|
||||||
|
raise RuntimeError(f"{condition} target content is absent")
|
||||||
|
content_end = content_start + len(content)
|
||||||
|
positions, records, crossing = base.content_positions(
|
||||||
|
token_ids,
|
||||||
|
offsets,
|
||||||
|
content_start,
|
||||||
|
content_end,
|
||||||
|
)
|
||||||
|
if not positions:
|
||||||
|
raise RuntimeError(f"{condition} has no target-content tokens")
|
||||||
|
target_first = min(positions)
|
||||||
|
target_last = max(positions)
|
||||||
|
|
||||||
|
target_user_candidates = [
|
||||||
|
index
|
||||||
|
for index, token_id in enumerate(official_ids)
|
||||||
|
if token_id == ids["user"] and index < target_first
|
||||||
|
]
|
||||||
|
if len(target_user_candidates) < 2:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"{condition} cannot locate filler and target User heads"
|
||||||
|
)
|
||||||
|
target_head_position = max(target_user_candidates)
|
||||||
|
target_delimiter_position = target_head_position + 1
|
||||||
|
if official_ids[target_delimiter_position] != ids["colon"]:
|
||||||
|
raise RuntimeError("target User head is not followed by colon")
|
||||||
|
|
||||||
|
suffix_assistant_candidates = [
|
||||||
|
index
|
||||||
|
for index, token_id in enumerate(official_ids)
|
||||||
|
if token_id == ids["assistant"] and index > target_last
|
||||||
|
]
|
||||||
|
if len(suffix_assistant_candidates) != 1:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"{condition} expected one suffix Assistant head; "
|
||||||
|
f"got {suffix_assistant_candidates}"
|
||||||
|
)
|
||||||
|
suffix_head_position = suffix_assistant_candidates[0]
|
||||||
|
if official_ids[suffix_head_position + 1] != ids["colon"]:
|
||||||
|
raise RuntimeError("suffix Assistant head is not followed by colon")
|
||||||
|
if ids["eos"] not in official_ids[:target_head_position]:
|
||||||
|
raise RuntimeError("official history EOS is absent before target role")
|
||||||
|
|
||||||
|
level = FACTORS[condition]["role_block"]
|
||||||
|
desired_head = ids[LEVEL_FACTORS[level]["head"]]
|
||||||
|
desired_delimiter = ids[LEVEL_FACTORS[level]["delimiter"]]
|
||||||
|
token_ids = list(token_ids)
|
||||||
|
token_ids[target_head_position] = desired_head
|
||||||
|
token_ids[target_delimiter_position] = desired_delimiter
|
||||||
|
changed_positions = [
|
||||||
|
index
|
||||||
|
for index, (left, right) in enumerate(
|
||||||
|
zip(official_ids, token_ids, strict=True)
|
||||||
|
)
|
||||||
|
if left != right
|
||||||
|
]
|
||||||
|
expected_differences = (
|
||||||
|
int(LEVEL_FACTORS[level]["head"] != "user")
|
||||||
|
+ int(LEVEL_FACTORS[level]["delimiter"] != "colon")
|
||||||
|
)
|
||||||
|
if len(changed_positions) != expected_differences:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"{condition} changed {changed_positions}; "
|
||||||
|
f"expected {expected_differences} positions"
|
||||||
|
)
|
||||||
|
if any(position >= target_first for position in changed_positions):
|
||||||
|
raise RuntimeError("role-block edit is not strictly pre-target")
|
||||||
|
|
||||||
|
decoded = tokenizer.decode(
|
||||||
|
token_ids,
|
||||||
|
skip_special_tokens=False,
|
||||||
|
clean_up_tokenization_spaces=False,
|
||||||
|
)
|
||||||
|
RENDER_AUDIT.append(
|
||||||
|
{
|
||||||
|
"content_sha256": base.text_sha256(content),
|
||||||
|
"condition": condition,
|
||||||
|
"role_block": level,
|
||||||
|
"head": LEVEL_FACTORS[level]["head"],
|
||||||
|
"delimiter": LEVEL_FACTORS[level]["delimiter"],
|
||||||
|
"target_head_position": target_head_position,
|
||||||
|
"target_delimiter_position": target_delimiter_position,
|
||||||
|
"suffix_head_position": suffix_head_position,
|
||||||
|
"changed_positions": changed_positions,
|
||||||
|
"changed_token_ids_vs_official": len(changed_positions),
|
||||||
|
"official_eos_token_id": ids["eos"],
|
||||||
|
"input_tokens": len(token_ids),
|
||||||
|
"target_first_position": target_first,
|
||||||
|
"target_last_position": target_last,
|
||||||
|
"counterfactual_decoded_sha256": base.text_sha256(decoded),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"condition": condition,
|
||||||
|
"messages_sha256": base.canonical_hash(messages),
|
||||||
|
"rendered_sha256": base.text_sha256(rendered),
|
||||||
|
"token_ids": token_ids,
|
||||||
|
"tokens": len(token_ids),
|
||||||
|
"token_ids_sha256": base.canonical_hash(token_ids),
|
||||||
|
"content_positions": positions,
|
||||||
|
"content_records": records,
|
||||||
|
"content_tokens": len(positions),
|
||||||
|
"wrapper_tokens": len(token_ids) - len(positions),
|
||||||
|
"boundary_crossing_tokens": crossing,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def factorial_effects(
|
||||||
|
values: dict[str, np.ndarray],
|
||||||
|
) -> dict[str, np.ndarray]:
|
||||||
|
return {
|
||||||
|
"head_main": 0.5 * (
|
||||||
|
values["assistant_colon"]
|
||||||
|
+ values["assistant_x"]
|
||||||
|
- values["user_colon"]
|
||||||
|
- values["user_x"]
|
||||||
|
),
|
||||||
|
"delimiter_main": 0.5 * (
|
||||||
|
values["user_x"]
|
||||||
|
+ values["assistant_x"]
|
||||||
|
- values["user_colon"]
|
||||||
|
- values["assistant_colon"]
|
||||||
|
),
|
||||||
|
"head_by_delimiter": (
|
||||||
|
values["assistant_x"]
|
||||||
|
- values["user_x"]
|
||||||
|
- values["assistant_colon"]
|
||||||
|
+ values["user_colon"]
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def role_block_domain(
|
||||||
|
loads: dict[str, np.ndarray],
|
||||||
|
mode: str,
|
||||||
|
replicates: int,
|
||||||
|
seed: int,
|
||||||
|
scope: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Add paired role-head, delimiter, and interaction statistics."""
|
||||||
|
result = ORIGINAL_BOUNDARY_DOMAIN(
|
||||||
|
loads,
|
||||||
|
mode,
|
||||||
|
replicates,
|
||||||
|
seed,
|
||||||
|
scope,
|
||||||
|
)
|
||||||
|
shapes = {value.shape for value in loads.values()}
|
||||||
|
if len(shapes) != 1:
|
||||||
|
raise ValueError(f"role-block shape mismatch: {sorted(shapes)}")
|
||||||
|
rows = next(iter(loads.values())).shape[0]
|
||||||
|
rng = np.random.default_rng(base.scoped_seed(seed, scope))
|
||||||
|
sampled = rng.integers(
|
||||||
|
0,
|
||||||
|
rows,
|
||||||
|
size=(replicates, rows),
|
||||||
|
endpoint=False,
|
||||||
|
)
|
||||||
|
point = {
|
||||||
|
condition: base.distribution(value, mode)
|
||||||
|
for condition, value in loads.items()
|
||||||
|
}
|
||||||
|
boot = {
|
||||||
|
condition: base.bootstrap_distributions(value, mode, sampled)
|
||||||
|
for condition, value in loads.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
point_system_vectors = {
|
||||||
|
level: point[after] - point[before]
|
||||||
|
for level, (before, after) in SYSTEM_CELLS.items()
|
||||||
|
}
|
||||||
|
boot_system_vectors = {
|
||||||
|
level: boot[after] - boot[before]
|
||||||
|
for level, (before, after) in SYSTEM_CELLS.items()
|
||||||
|
}
|
||||||
|
point_vector_effects = factorial_effects(point_system_vectors)
|
||||||
|
boot_vector_effects = factorial_effects(boot_system_vectors)
|
||||||
|
distribution_effects = {}
|
||||||
|
for name in point_vector_effects:
|
||||||
|
point_magnitude = 0.5 * np.abs(
|
||||||
|
point_vector_effects[name]
|
||||||
|
).sum()
|
||||||
|
boot_magnitude = 0.5 * np.abs(
|
||||||
|
boot_vector_effects[name]
|
||||||
|
).sum(axis=1)
|
||||||
|
distribution_effects[name] = {
|
||||||
|
"half_l1_magnitude": {
|
||||||
|
"point": float(point_magnitude),
|
||||||
|
"ci95": base.interval(boot_magnitude),
|
||||||
|
},
|
||||||
|
"signed_expert_share_effect": (
|
||||||
|
point_vector_effects[name].tolist()
|
||||||
|
),
|
||||||
|
"signed_expert_share_effect_ci95": (
|
||||||
|
base.interval(boot_vector_effects[name])
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
point_tv: dict[str, float] = {}
|
||||||
|
boot_tv: dict[str, np.ndarray] = {}
|
||||||
|
point_jsd: dict[str, float] = {}
|
||||||
|
boot_jsd: dict[str, np.ndarray] = {}
|
||||||
|
for level, (before, after) in SYSTEM_CELLS.items():
|
||||||
|
point_tv[level] = float(
|
||||||
|
0.5 * np.abs(point[after] - point[before]).sum()
|
||||||
|
)
|
||||||
|
boot_tv[level] = 0.5 * np.abs(
|
||||||
|
boot[after] - boot[before]
|
||||||
|
).sum(axis=1)
|
||||||
|
point_jsd[level] = float(
|
||||||
|
base.js_divergence(point[before], point[after])[0]
|
||||||
|
)
|
||||||
|
boot_jsd[level] = base.js_divergence(
|
||||||
|
boot[before],
|
||||||
|
boot[after],
|
||||||
|
)
|
||||||
|
|
||||||
|
def scalar_factorial(
|
||||||
|
point_values: dict[str, float],
|
||||||
|
boot_values: dict[str, np.ndarray],
|
||||||
|
unit: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
point_arrays = {
|
||||||
|
key: np.asarray([value])
|
||||||
|
for key, value in point_values.items()
|
||||||
|
}
|
||||||
|
point_effects = factorial_effects(point_arrays)
|
||||||
|
boot_effects = factorial_effects(boot_values)
|
||||||
|
payload = {
|
||||||
|
name: {
|
||||||
|
"point": float(point_effects[name][0]),
|
||||||
|
"ci95": base.interval(boot_effects[name]),
|
||||||
|
}
|
||||||
|
for name in point_effects
|
||||||
|
}
|
||||||
|
if unit is not None:
|
||||||
|
for value in payload.values():
|
||||||
|
value["unit"] = unit
|
||||||
|
return payload
|
||||||
|
|
||||||
|
metric_effects: dict[str, Any] = {}
|
||||||
|
point_metrics = {
|
||||||
|
condition: base.metric_vector(value)
|
||||||
|
for condition, value in point.items()
|
||||||
|
}
|
||||||
|
boot_metrics = {
|
||||||
|
condition: base.metric_vector(value)
|
||||||
|
for condition, value in boot.items()
|
||||||
|
}
|
||||||
|
for metric in point_metrics[f"s0_{REFERENCE_LEVEL}"]:
|
||||||
|
point_edges = {
|
||||||
|
level: (
|
||||||
|
point_metrics[after][metric]
|
||||||
|
- point_metrics[before][metric]
|
||||||
|
)
|
||||||
|
for level, (before, after) in SYSTEM_CELLS.items()
|
||||||
|
}
|
||||||
|
boot_edges = {
|
||||||
|
level: (
|
||||||
|
boot_metrics[after][metric]
|
||||||
|
- boot_metrics[before][metric]
|
||||||
|
)
|
||||||
|
for level, (before, after) in SYSTEM_CELLS.items()
|
||||||
|
}
|
||||||
|
point_effects = factorial_effects(point_edges)
|
||||||
|
boot_effects = factorial_effects(boot_edges)
|
||||||
|
metric_effects[metric] = {
|
||||||
|
name: {
|
||||||
|
"point": float(point_effects[name][0]),
|
||||||
|
"ci95": base.interval(boot_effects[name]),
|
||||||
|
}
|
||||||
|
for name in point_effects
|
||||||
|
}
|
||||||
|
|
||||||
|
def direct_distance(
|
||||||
|
before_level: str,
|
||||||
|
after_level: str,
|
||||||
|
system: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
before = f"s{system}_{before_level}"
|
||||||
|
after = f"s{system}_{after_level}"
|
||||||
|
point_delta = point[after] - point[before]
|
||||||
|
tv_boot = 0.5 * np.abs(
|
||||||
|
boot[after] - boot[before]
|
||||||
|
).sum(axis=1)
|
||||||
|
jsd_boot = base.js_divergence(boot[before], boot[after])
|
||||||
|
return {
|
||||||
|
"total_variation": {
|
||||||
|
"point": float(0.5 * np.abs(point_delta).sum()),
|
||||||
|
"ci95": base.interval(tv_boot),
|
||||||
|
"_bootstrap": tv_boot,
|
||||||
|
},
|
||||||
|
"js_divergence": {
|
||||||
|
"point": float(
|
||||||
|
base.js_divergence(point[before], point[after])[0]
|
||||||
|
),
|
||||||
|
"ci95": base.interval(jsd_boot),
|
||||||
|
"unit": "nats",
|
||||||
|
"_bootstrap": jsd_boot,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
direct_specs = {
|
||||||
|
"head_at_colon": ("user_colon", "assistant_colon"),
|
||||||
|
"head_at_x": ("user_x", "assistant_x"),
|
||||||
|
"delimiter_at_user": ("user_colon", "user_x"),
|
||||||
|
"delimiter_at_assistant": (
|
||||||
|
"assistant_colon",
|
||||||
|
"assistant_x",
|
||||||
|
),
|
||||||
|
}
|
||||||
|
direct_raw = {
|
||||||
|
name: {
|
||||||
|
system: direct_distance(
|
||||||
|
before_level,
|
||||||
|
after_level,
|
||||||
|
system,
|
||||||
|
)
|
||||||
|
for system in (0, 1)
|
||||||
|
}
|
||||||
|
for name, (before_level, after_level) in direct_specs.items()
|
||||||
|
}
|
||||||
|
direct: dict[str, Any] = {}
|
||||||
|
for name in direct_specs:
|
||||||
|
cells = direct_raw[name]
|
||||||
|
direct[name] = {}
|
||||||
|
for system in (0, 1):
|
||||||
|
direct[name][f"at_s{system}"] = {
|
||||||
|
metric: {
|
||||||
|
key: value
|
||||||
|
for key, value in payload.items()
|
||||||
|
if key != "_bootstrap"
|
||||||
|
}
|
||||||
|
for metric, payload in cells[system].items()
|
||||||
|
}
|
||||||
|
direct[name]["s1_minus_s0"] = {}
|
||||||
|
for metric in ("total_variation", "js_divergence"):
|
||||||
|
before_payload = cells[0][metric]
|
||||||
|
after_payload = cells[1][metric]
|
||||||
|
direct[name]["s1_minus_s0"][
|
||||||
|
f"{metric}_delta"
|
||||||
|
] = {
|
||||||
|
"point": (
|
||||||
|
after_payload["point"] - before_payload["point"]
|
||||||
|
),
|
||||||
|
"ci95": base.interval(
|
||||||
|
after_payload["_bootstrap"]
|
||||||
|
- before_payload["_bootstrap"]
|
||||||
|
),
|
||||||
|
**(
|
||||||
|
{"unit": "nats"}
|
||||||
|
if metric == "js_divergence"
|
||||||
|
else {}
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
dependency_specs = {
|
||||||
|
"delimiter_dependence_of_head_direct": (
|
||||||
|
"head_at_colon",
|
||||||
|
"head_at_x",
|
||||||
|
),
|
||||||
|
"head_dependence_of_delimiter_direct": (
|
||||||
|
"delimiter_at_user",
|
||||||
|
"delimiter_at_assistant",
|
||||||
|
),
|
||||||
|
}
|
||||||
|
dependencies: dict[str, Any] = {}
|
||||||
|
for name, (before_edge, after_edge) in dependency_specs.items():
|
||||||
|
dependencies[name] = {}
|
||||||
|
system_payloads: dict[int, dict[str, Any]] = {}
|
||||||
|
for system in (0, 1):
|
||||||
|
system_payloads[system] = {}
|
||||||
|
for metric in ("total_variation", "js_divergence"):
|
||||||
|
before_payload = direct_raw[before_edge][system][metric]
|
||||||
|
after_payload = direct_raw[after_edge][system][metric]
|
||||||
|
payload = {
|
||||||
|
"point": (
|
||||||
|
after_payload["point"] - before_payload["point"]
|
||||||
|
),
|
||||||
|
"ci95": base.interval(
|
||||||
|
after_payload["_bootstrap"]
|
||||||
|
- before_payload["_bootstrap"]
|
||||||
|
),
|
||||||
|
"_bootstrap": (
|
||||||
|
after_payload["_bootstrap"]
|
||||||
|
- before_payload["_bootstrap"]
|
||||||
|
),
|
||||||
|
}
|
||||||
|
if metric == "js_divergence":
|
||||||
|
payload["unit"] = "nats"
|
||||||
|
system_payloads[system][metric] = payload
|
||||||
|
dependencies[name].setdefault(
|
||||||
|
f"at_s{system}",
|
||||||
|
{},
|
||||||
|
)[f"{metric}_delta"] = {
|
||||||
|
key: value
|
||||||
|
for key, value in payload.items()
|
||||||
|
if key != "_bootstrap"
|
||||||
|
}
|
||||||
|
dependencies[name]["s1_minus_s0"] = {}
|
||||||
|
for metric in ("total_variation", "js_divergence"):
|
||||||
|
before_payload = system_payloads[0][metric]
|
||||||
|
after_payload = system_payloads[1][metric]
|
||||||
|
dependencies[name]["s1_minus_s0"][
|
||||||
|
f"{metric}_difference_in_differences"
|
||||||
|
] = {
|
||||||
|
"point": (
|
||||||
|
after_payload["point"] - before_payload["point"]
|
||||||
|
),
|
||||||
|
"ci95": base.interval(
|
||||||
|
after_payload["_bootstrap"]
|
||||||
|
- before_payload["_bootstrap"]
|
||||||
|
),
|
||||||
|
**(
|
||||||
|
{"unit": "nats"}
|
||||||
|
if metric == "js_divergence"
|
||||||
|
else {}
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
result["role_block_factorial"] = {
|
||||||
|
"factor_coding": {
|
||||||
|
"head_main": (
|
||||||
|
"0.5 * [(Assistant:-User:) + "
|
||||||
|
"(Assistant x-User x)]"
|
||||||
|
),
|
||||||
|
"delimiter_main": (
|
||||||
|
"0.5 * [(User x-User:) + "
|
||||||
|
"(Assistant x-Assistant:)]"
|
||||||
|
),
|
||||||
|
"head_by_delimiter": (
|
||||||
|
"(Assistant x-User x) - (Assistant:-User:)"
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"system_edge_distance_effects": {
|
||||||
|
"total_variation": scalar_factorial(point_tv, boot_tv),
|
||||||
|
"js_divergence": scalar_factorial(
|
||||||
|
point_jsd,
|
||||||
|
boot_jsd,
|
||||||
|
unit="nats",
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"metric_system_edge_effects": metric_effects,
|
||||||
|
"distribution_system_edge_effects": distribution_effects,
|
||||||
|
"direct_factor_edges": direct,
|
||||||
|
"direct_effect_dependencies": dependencies,
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def layer_statistics(
|
||||||
|
prompt_rows: list[dict[str, Any]],
|
||||||
|
replicates: int,
|
||||||
|
seed: int,
|
||||||
|
layer_index: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
scopes = boundary_prior.layer_statistics(
|
||||||
|
prompt_rows,
|
||||||
|
replicates,
|
||||||
|
seed,
|
||||||
|
layer_index,
|
||||||
|
)
|
||||||
|
for scope in scopes.values():
|
||||||
|
for mode in scope["modes"].values():
|
||||||
|
mode["role_block_control"] = mode.pop("boundary_control")
|
||||||
|
return scopes
|
||||||
|
|
||||||
|
|
||||||
|
def install_control_contract() -> None:
|
||||||
|
boundary_prior.BOUNDARY_LEVELS = BLOCK_LEVELS
|
||||||
|
boundary_prior.REFERENCE_LEVEL = REFERENCE_LEVEL
|
||||||
|
boundary_prior.CONDITIONS = CONDITIONS
|
||||||
|
boundary_prior.FACTORS = FACTORS
|
||||||
|
boundary_prior.SYSTEM_CELLS = SYSTEM_CELLS
|
||||||
|
boundary_prior.SYSTEM_EDGE_CONTRASTS = SYSTEM_EDGE_CONTRASTS
|
||||||
|
boundary_prior.COMPARISONS = COMPARISONS
|
||||||
|
boundary_prior.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
|
||||||
|
boundary_prior.boundary_control_domain = role_block_domain
|
||||||
|
|
||||||
|
base.CONDITIONS = CONDITIONS
|
||||||
|
base.FACTORS = FACTORS
|
||||||
|
base.COMPARISONS = COMPARISONS
|
||||||
|
base.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
|
||||||
|
base.condition_messages = condition_messages
|
||||||
|
base.render_variant = render_role_block_variant
|
||||||
|
base.layer_statistics = layer_statistics
|
||||||
|
|
||||||
|
|
||||||
|
def output_path_from_argv() -> Path:
|
||||||
|
try:
|
||||||
|
return Path(sys.argv[sys.argv.index("--output") + 1])
|
||||||
|
except (ValueError, IndexError) as error:
|
||||||
|
raise ValueError("--output is required") from error
|
||||||
|
|
||||||
|
|
||||||
|
def render_contract_summary() -> dict[str, Any]:
|
||||||
|
if not RENDER_AUDIT:
|
||||||
|
raise RuntimeError("render audit is empty")
|
||||||
|
by_content: dict[str, list[dict[str, Any]]] = {}
|
||||||
|
for row in RENDER_AUDIT:
|
||||||
|
by_content.setdefault(row["content_sha256"], []).append(row)
|
||||||
|
if len(by_content) != 128 and "--per-domain" not in sys.argv:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"expected 128 selected contents; observed {len(by_content)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
equal_lengths = 0
|
||||||
|
equal_target_positions = 0
|
||||||
|
exact_by_level = {
|
||||||
|
level: 0
|
||||||
|
for level in BLOCK_LEVELS
|
||||||
|
}
|
||||||
|
for rows in by_content.values():
|
||||||
|
for system in (0, 1):
|
||||||
|
cells = [
|
||||||
|
row
|
||||||
|
for row in rows
|
||||||
|
if FACTORS[row["condition"]]["system"] == system
|
||||||
|
]
|
||||||
|
if len(cells) != len(BLOCK_LEVELS):
|
||||||
|
raise RuntimeError(
|
||||||
|
"render audit lacks one or more role-block cells"
|
||||||
|
)
|
||||||
|
if len({row["input_tokens"] for row in cells}) == 1:
|
||||||
|
equal_lengths += 1
|
||||||
|
target_spans = {
|
||||||
|
(
|
||||||
|
row["target_first_position"],
|
||||||
|
row["target_last_position"],
|
||||||
|
)
|
||||||
|
for row in cells
|
||||||
|
}
|
||||||
|
if len(target_spans) == 1:
|
||||||
|
equal_target_positions += 1
|
||||||
|
for row in cells:
|
||||||
|
expected = (
|
||||||
|
int(row["head"] != "user")
|
||||||
|
+ int(row["delimiter"] != "colon")
|
||||||
|
)
|
||||||
|
exact_by_level[row["role_block"]] += int(
|
||||||
|
row["changed_token_ids_vs_official"] == expected
|
||||||
|
)
|
||||||
|
|
||||||
|
groups = 2 * len(by_content)
|
||||||
|
return {
|
||||||
|
"selected_contents": len(by_content),
|
||||||
|
"system_groups": groups,
|
||||||
|
"equal_input_length_groups": equal_lengths,
|
||||||
|
"equal_target_position_groups": equal_target_positions,
|
||||||
|
"exact_edit_contract_cells_by_level": exact_by_level,
|
||||||
|
"official_zero_id_cells_exact": exact_by_level["user_colon"],
|
||||||
|
"one_id_counterfactual_cells_exact": (
|
||||||
|
exact_by_level["assistant_colon"]
|
||||||
|
+ exact_by_level["user_x"]
|
||||||
|
),
|
||||||
|
"two_id_counterfactual_cells_exact": (
|
||||||
|
exact_by_level["assistant_x"]
|
||||||
|
),
|
||||||
|
"all_group_lengths_equal": equal_lengths == groups,
|
||||||
|
"all_target_positions_equal": equal_target_positions == groups,
|
||||||
|
"all_edit_contracts_exact": all(
|
||||||
|
value == groups
|
||||||
|
for value in exact_by_level.values()
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def finalize_result(path: Path) -> dict[str, Any]:
|
||||||
|
result = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
result["schema_version"] = 4
|
||||||
|
result["evidence_identity"] = (
|
||||||
|
"X / official BF16 weights and pinned tokenizer; target two-ID "
|
||||||
|
"role head by delimiter factorial counterfactuals"
|
||||||
|
)
|
||||||
|
boundary = result["boundary"]
|
||||||
|
boundary.pop("factorial_claim", None)
|
||||||
|
boundary.update(
|
||||||
|
{
|
||||||
|
"role_marker_block_factorial": True,
|
||||||
|
"official_serialization_by_role_block": {
|
||||||
|
"user_colon": True,
|
||||||
|
"assistant_colon": False,
|
||||||
|
"user_x": False,
|
||||||
|
"assistant_x": False,
|
||||||
|
},
|
||||||
|
"official_assistant_eos_held_fixed": True,
|
||||||
|
"generation_prompt_held_fixed": True,
|
||||||
|
"target_position_held_fixed": True,
|
||||||
|
"complete_role_semantics_identified": False,
|
||||||
|
"task_performance": False,
|
||||||
|
"causal_boundary": (
|
||||||
|
"Within this fixed base-checkpoint batch, edits intervene on "
|
||||||
|
"the two target role-block input positions before target "
|
||||||
|
"content. They identify head, one delimiter control, and "
|
||||||
|
"their interaction, not complete role semantics, Chat/SFT "
|
||||||
|
"behavior, or answer quality."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
old_contract = result.pop("message_history_contract")
|
||||||
|
result["role_marker_block_contract"] = {
|
||||||
|
"chat_template_revision": base.MODEL_REVISION,
|
||||||
|
"chat_template": old_contract["chat_template"],
|
||||||
|
"chat_template_sha256": old_contract["chat_template_sha256"],
|
||||||
|
"official_sequence": (
|
||||||
|
"Assistant: {filler} + eos_token + User: {target} + "
|
||||||
|
"generation-prompt Assistant:"
|
||||||
|
),
|
||||||
|
"system_message": SYSTEM_MESSAGE,
|
||||||
|
"system_message_sha256": base.text_sha256(SYSTEM_MESSAGE),
|
||||||
|
"filler_user": FILLER_USER,
|
||||||
|
"filler_user_sha256": base.text_sha256(FILLER_USER),
|
||||||
|
"filler_assistant": FILLER_ASSISTANT,
|
||||||
|
"filler_assistant_sha256": base.text_sha256(FILLER_ASSISTANT),
|
||||||
|
"block_levels": list(BLOCK_LEVELS),
|
||||||
|
"level_factors": LEVEL_FACTORS,
|
||||||
|
"role_token_ids": ROLE_TOKEN_IDS,
|
||||||
|
"official_target_block_ids": [
|
||||||
|
ROLE_TOKEN_IDS["user"],
|
||||||
|
ROLE_TOKEN_IDS["colon"],
|
||||||
|
],
|
||||||
|
"conditions": FACTORS,
|
||||||
|
"comparisons": [
|
||||||
|
{"name": name, "before": before, "after": after}
|
||||||
|
for name, before, after in COMPARISONS
|
||||||
|
],
|
||||||
|
"render_validation": render_contract_summary(),
|
||||||
|
"target_role": "user",
|
||||||
|
"add_generation_prompt": True,
|
||||||
|
"scope_split": {
|
||||||
|
"full_input": (
|
||||||
|
"all official or counterfactually edited BOS, system/history, "
|
||||||
|
"target, newline, and generation-prompt token IDs"
|
||||||
|
),
|
||||||
|
"target_content": (
|
||||||
|
"exact intersection of (relative character span, token ID) "
|
||||||
|
"inside target user content across all eight conditions"
|
||||||
|
),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result["inference_contract"]["batch_grouping"] = (
|
||||||
|
"all eight system x head x delimiter variants of one source prompt "
|
||||||
|
"execute in the same 32-row right-padded batch"
|
||||||
|
)
|
||||||
|
statistical = result["statistical_contract"]
|
||||||
|
statistical["paired_indices"] = (
|
||||||
|
"one sampled source-prompt index matrix is reused across all eight "
|
||||||
|
"cells for every role-block main effect and interaction within each "
|
||||||
|
"domain/layer/scope/mode"
|
||||||
|
)
|
||||||
|
statistical["role_block_factor_coding"] = {
|
||||||
|
"head_main": (
|
||||||
|
"average Assistant-minus-User edge across colon and x"
|
||||||
|
),
|
||||||
|
"delimiter_main": (
|
||||||
|
"average x-minus-colon edge across User and Assistant"
|
||||||
|
),
|
||||||
|
"head_by_delimiter": (
|
||||||
|
"(Assistant x-User x) - (Assistant:-User:)"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(result, indent=2, ensure_ascii=False) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
install_control_contract()
|
||||||
|
if {"-h", "--help"} & set(sys.argv[1:]):
|
||||||
|
base.main()
|
||||||
|
return
|
||||||
|
output = output_path_from_argv()
|
||||||
|
with open(os.devnull, "w", encoding="utf-8") as sink:
|
||||||
|
with contextlib.redirect_stdout(sink):
|
||||||
|
base.main()
|
||||||
|
result = finalize_result(output)
|
||||||
|
payload = output.read_bytes()
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"output": str(output),
|
||||||
|
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||||
|
"bytes": len(payload),
|
||||||
|
"source_prompts": result["inference_contract"][
|
||||||
|
"total_source_prompts"
|
||||||
|
],
|
||||||
|
"prompt_variants": result["inference_contract"][
|
||||||
|
"total_prompt_variants"
|
||||||
|
],
|
||||||
|
"total_routes": result["inference_contract"][
|
||||||
|
"total_routes_all_conditions_all_moe_layers"
|
||||||
|
],
|
||||||
|
"render_validation": result[
|
||||||
|
"role_marker_block_contract"
|
||||||
|
]["render_validation"],
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,375 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Run a pinned special-token-family boundary control on DeepSeek-V2-Lite.
|
||||||
|
|
||||||
|
The official tokenizer at revision 604d5664 has exactly two special token IDs:
|
||||||
|
|
||||||
|
bos: 100000, <｜begin▁of▁sentence｜>
|
||||||
|
eos: 100001, <｜end▁of▁sentence｜> (also used as pad_token)
|
||||||
|
|
||||||
|
This experiment keeps the audited repeated-token history and changes exactly
|
||||||
|
one completed-assistant boundary ID:
|
||||||
|
|
||||||
|
system off/on x official EOS / counterfactual BOS / x / period
|
||||||
|
|
||||||
|
EOS and BOS exhaust the tokenizer's special-token inventory. X and period are
|
||||||
|
ordinary one-token controls. The three counterfactuals are not valid official
|
||||||
|
chat serializations. This design can identify differences among these four
|
||||||
|
pinned IDs inside one fixed BF16 batch contract; it cannot establish a general
|
||||||
|
"specialness" property from two special and two selected ordinary tokens.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
import v2_lite_routing_history_boundary_token_control as prior
|
||||||
|
|
||||||
|
|
||||||
|
base = prior.base
|
||||||
|
|
||||||
|
BOUNDARY_LEVELS = ("eos", "bos", "x", "period")
|
||||||
|
REFERENCE_LEVEL = "eos"
|
||||||
|
BOUNDARY_TEXT = {
|
||||||
|
"x": "x",
|
||||||
|
"period": ".",
|
||||||
|
}
|
||||||
|
CONDITIONS = tuple(
|
||||||
|
f"s{system}_{boundary}"
|
||||||
|
for boundary in BOUNDARY_LEVELS
|
||||||
|
for system in (0, 1)
|
||||||
|
)
|
||||||
|
FACTORS = {
|
||||||
|
condition: {
|
||||||
|
"system": int(condition[1]),
|
||||||
|
"history": "filler",
|
||||||
|
"boundary": condition.split("_", 1)[1],
|
||||||
|
"token_class": (
|
||||||
|
"special"
|
||||||
|
if condition.split("_", 1)[1] in {"eos", "bos"}
|
||||||
|
else "ordinary_control"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
for condition in CONDITIONS
|
||||||
|
}
|
||||||
|
SYSTEM_CELLS = {
|
||||||
|
boundary: (f"s0_{boundary}", f"s1_{boundary}")
|
||||||
|
for boundary in BOUNDARY_LEVELS
|
||||||
|
}
|
||||||
|
SYSTEM_EDGE_CONTRASTS = {
|
||||||
|
f"{boundary}_minus_eos": ("eos", boundary)
|
||||||
|
for boundary in BOUNDARY_LEVELS
|
||||||
|
if boundary != "eos"
|
||||||
|
}
|
||||||
|
COMPARISONS = tuple(
|
||||||
|
[
|
||||||
|
(
|
||||||
|
f"system_{boundary}",
|
||||||
|
f"s0_{boundary}",
|
||||||
|
f"s1_{boundary}",
|
||||||
|
)
|
||||||
|
for boundary in BOUNDARY_LEVELS
|
||||||
|
]
|
||||||
|
+ [
|
||||||
|
(
|
||||||
|
f"{boundary}_at_s{system}",
|
||||||
|
f"s{system}_eos",
|
||||||
|
f"s{system}_{boundary}",
|
||||||
|
)
|
||||||
|
for boundary in BOUNDARY_LEVELS
|
||||||
|
if boundary != "eos"
|
||||||
|
for system in (0, 1)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
ALIGNMENT_COMPARISONS = COMPARISONS
|
||||||
|
ORIGINAL_BOUNDARY_DOMAIN = prior.boundary_control_domain
|
||||||
|
|
||||||
|
|
||||||
|
def special_family_token_ids(tokenizer: Any) -> dict[str, int]:
|
||||||
|
"""Resolve and validate the complete pinned special-token inventory."""
|
||||||
|
if tokenizer.bos_token_id is None or tokenizer.eos_token_id is None:
|
||||||
|
raise RuntimeError("tokenizer must expose both BOS and EOS")
|
||||||
|
special_ids = [int(value) for value in tokenizer.all_special_ids]
|
||||||
|
expected_special = {
|
||||||
|
int(tokenizer.bos_token_id),
|
||||||
|
int(tokenizer.eos_token_id),
|
||||||
|
}
|
||||||
|
if set(special_ids) != expected_special or len(special_ids) != 2:
|
||||||
|
raise RuntimeError(
|
||||||
|
"pinned special-token inventory changed: "
|
||||||
|
f"observed={special_ids} expected={sorted(expected_special)}"
|
||||||
|
)
|
||||||
|
if tokenizer.pad_token_id != tokenizer.eos_token_id:
|
||||||
|
raise RuntimeError(
|
||||||
|
"pinned tokenizer no longer aliases pad_token_id to eos_token_id"
|
||||||
|
)
|
||||||
|
|
||||||
|
ids = {
|
||||||
|
"eos": int(tokenizer.eos_token_id),
|
||||||
|
"bos": int(tokenizer.bos_token_id),
|
||||||
|
}
|
||||||
|
for name, text in BOUNDARY_TEXT.items():
|
||||||
|
encoded = list(
|
||||||
|
tokenizer(text, add_special_tokens=False).input_ids
|
||||||
|
)
|
||||||
|
if len(encoded) != 1:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"{name} boundary control is not one token: {encoded}"
|
||||||
|
)
|
||||||
|
if encoded[0] in expected_special:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"{name} boundary control unexpectedly uses a special token"
|
||||||
|
)
|
||||||
|
ids[name] = int(encoded[0])
|
||||||
|
if len(set(ids.values())) != len(ids):
|
||||||
|
raise RuntimeError(f"boundary token IDs are not distinct: {ids}")
|
||||||
|
return ids
|
||||||
|
|
||||||
|
|
||||||
|
def special_family_domain(
|
||||||
|
loads: dict[str, np.ndarray],
|
||||||
|
mode: str,
|
||||||
|
replicates: int,
|
||||||
|
seed: int,
|
||||||
|
scope: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Add a paired descriptive 2-special versus 2-ordinary summary."""
|
||||||
|
result = ORIGINAL_BOUNDARY_DOMAIN(
|
||||||
|
loads,
|
||||||
|
mode,
|
||||||
|
replicates,
|
||||||
|
seed,
|
||||||
|
scope,
|
||||||
|
)
|
||||||
|
shapes = {value.shape for value in loads.values()}
|
||||||
|
if len(shapes) != 1:
|
||||||
|
raise ValueError(
|
||||||
|
f"special-family shape mismatch: {sorted(shapes)}"
|
||||||
|
)
|
||||||
|
rows = next(iter(loads.values())).shape[0]
|
||||||
|
rng = np.random.default_rng(base.scoped_seed(seed, scope))
|
||||||
|
sampled = rng.integers(
|
||||||
|
0,
|
||||||
|
rows,
|
||||||
|
size=(replicates, rows),
|
||||||
|
endpoint=False,
|
||||||
|
)
|
||||||
|
point = {
|
||||||
|
condition: base.distribution(value, mode)
|
||||||
|
for condition, value in loads.items()
|
||||||
|
}
|
||||||
|
boot = {
|
||||||
|
condition: base.bootstrap_distributions(value, mode, sampled)
|
||||||
|
for condition, value in loads.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
point_tv: dict[str, float] = {}
|
||||||
|
boot_tv: dict[str, np.ndarray] = {}
|
||||||
|
point_jsd: dict[str, float] = {}
|
||||||
|
boot_jsd: dict[str, np.ndarray] = {}
|
||||||
|
for level, (before, after) in SYSTEM_CELLS.items():
|
||||||
|
point_tv[level] = float(
|
||||||
|
0.5 * np.abs(point[after] - point[before]).sum()
|
||||||
|
)
|
||||||
|
boot_tv[level] = 0.5 * np.abs(
|
||||||
|
boot[after] - boot[before]
|
||||||
|
).sum(axis=1)
|
||||||
|
point_jsd[level] = float(
|
||||||
|
base.js_divergence(point[before], point[after])[0]
|
||||||
|
)
|
||||||
|
boot_jsd[level] = base.js_divergence(
|
||||||
|
boot[before],
|
||||||
|
boot[after],
|
||||||
|
)
|
||||||
|
|
||||||
|
def paired_family(
|
||||||
|
point_metric: dict[str, float],
|
||||||
|
boot_metric: dict[str, np.ndarray],
|
||||||
|
unit: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
special_point = 0.5 * (
|
||||||
|
point_metric["eos"] + point_metric["bos"]
|
||||||
|
)
|
||||||
|
ordinary_point = 0.5 * (
|
||||||
|
point_metric["x"] + point_metric["period"]
|
||||||
|
)
|
||||||
|
special_boot = 0.5 * (
|
||||||
|
boot_metric["eos"] + boot_metric["bos"]
|
||||||
|
)
|
||||||
|
ordinary_boot = 0.5 * (
|
||||||
|
boot_metric["x"] + boot_metric["period"]
|
||||||
|
)
|
||||||
|
payload = {
|
||||||
|
"special_mean": {
|
||||||
|
"point": special_point,
|
||||||
|
"ci95": base.interval(special_boot),
|
||||||
|
},
|
||||||
|
"ordinary_control_mean": {
|
||||||
|
"point": ordinary_point,
|
||||||
|
"ci95": base.interval(ordinary_boot),
|
||||||
|
},
|
||||||
|
"ordinary_minus_special": {
|
||||||
|
"point": ordinary_point - special_point,
|
||||||
|
"ci95": base.interval(ordinary_boot - special_boot),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if unit is not None:
|
||||||
|
for value in payload.values():
|
||||||
|
value["unit"] = unit
|
||||||
|
return payload
|
||||||
|
|
||||||
|
result["descriptive_family_summary"] = {
|
||||||
|
"definition": (
|
||||||
|
"mean(EOS,BOS) versus mean(x,period) inside the same shared "
|
||||||
|
"source bootstrap; descriptive for these four IDs only"
|
||||||
|
),
|
||||||
|
"total_variation": paired_family(point_tv, boot_tv),
|
||||||
|
"js_divergence": paired_family(
|
||||||
|
point_jsd,
|
||||||
|
boot_jsd,
|
||||||
|
unit="nats",
|
||||||
|
),
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def install_control_contract() -> None:
|
||||||
|
"""Install four pinned boundary IDs into the audited eight-cell runner."""
|
||||||
|
prior.BOUNDARY_LEVELS = BOUNDARY_LEVELS
|
||||||
|
prior.REFERENCE_LEVEL = REFERENCE_LEVEL
|
||||||
|
prior.BOUNDARY_TEXT = BOUNDARY_TEXT
|
||||||
|
prior.CONDITIONS = CONDITIONS
|
||||||
|
prior.FACTORS = FACTORS
|
||||||
|
prior.SYSTEM_CELLS = SYSTEM_CELLS
|
||||||
|
prior.SYSTEM_EDGE_CONTRASTS = SYSTEM_EDGE_CONTRASTS
|
||||||
|
prior.COMPARISONS = COMPARISONS
|
||||||
|
prior.ALIGNMENT_COMPARISONS = ALIGNMENT_COMPARISONS
|
||||||
|
prior.RENDER_AUDIT.clear()
|
||||||
|
prior.BOUNDARY_TOKEN_IDS.clear()
|
||||||
|
prior.boundary_token_ids = special_family_token_ids
|
||||||
|
prior.boundary_control_domain = special_family_domain
|
||||||
|
prior.install_control_contract()
|
||||||
|
|
||||||
|
|
||||||
|
def output_path_from_argv() -> Path:
|
||||||
|
try:
|
||||||
|
return Path(sys.argv[sys.argv.index("--output") + 1])
|
||||||
|
except (ValueError, IndexError) as error:
|
||||||
|
raise ValueError("--output is required") from error
|
||||||
|
|
||||||
|
|
||||||
|
def finalize_result(path: Path) -> dict[str, Any]:
|
||||||
|
result = prior.finalize_result(path)
|
||||||
|
result["schema_version"] = 4
|
||||||
|
result["evidence_identity"] = (
|
||||||
|
"X / official BF16 weights and pinned tokenizer; official EOS plus "
|
||||||
|
"BOS and two ordinary single-ID boundary counterfactuals"
|
||||||
|
)
|
||||||
|
boundary = result["boundary"]
|
||||||
|
boundary.update(
|
||||||
|
{
|
||||||
|
"special_token_family_control": True,
|
||||||
|
"complete_pinned_special_inventory": True,
|
||||||
|
"specialness_generalized": False,
|
||||||
|
"causal_boundary": (
|
||||||
|
"All replacement contrasts causally intervene on exactly one "
|
||||||
|
"prior boundary input ID inside this run. BOS versus EOS "
|
||||||
|
"exhausts the pinned tokenizer's two special IDs, while x "
|
||||||
|
"and period are selected ordinary controls; four IDs do not "
|
||||||
|
"identify a universal special-token category effect."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
boundary["official_serialization_by_boundary"] = {
|
||||||
|
"eos": True,
|
||||||
|
"bos": False,
|
||||||
|
"x": False,
|
||||||
|
"period": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
old = result.pop("history_boundary_token_contract")
|
||||||
|
result["special_token_family_contract"] = {
|
||||||
|
**old,
|
||||||
|
"boundary_levels": list(BOUNDARY_LEVELS),
|
||||||
|
"boundary_token_ids": dict(prior.BOUNDARY_TOKEN_IDS),
|
||||||
|
"boundary_text_controls": BOUNDARY_TEXT,
|
||||||
|
"conditions": FACTORS,
|
||||||
|
"tokenizer_special_inventory": {
|
||||||
|
"all_special_tokens": [
|
||||||
|
"<｜begin▁of▁sentence｜>",
|
||||||
|
"<｜end▁of▁sentence｜>",
|
||||||
|
],
|
||||||
|
"all_special_ids": [100000, 100001],
|
||||||
|
"bos_token_id": 100000,
|
||||||
|
"eos_token_id": 100001,
|
||||||
|
"pad_token_id": 100001,
|
||||||
|
"pad_aliases_eos": True,
|
||||||
|
"inventory_size": 2,
|
||||||
|
},
|
||||||
|
"class_comparison_boundary": (
|
||||||
|
"EOS/BOS are the complete special inventory, but x/period are "
|
||||||
|
"only two chosen ordinary controls; report individual-ID "
|
||||||
|
"contrasts and a descriptive 2-vs-2 family summary, not a "
|
||||||
|
"population-level specialness claim"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
result["inference_contract"]["batch_grouping"] = (
|
||||||
|
"all eight EOS/BOS/x/period variants of one source prompt execute "
|
||||||
|
"in the same 32-row right-padded batch"
|
||||||
|
)
|
||||||
|
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(result, indent=2, ensure_ascii=False) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
install_control_contract()
|
||||||
|
if {"-h", "--help"} & set(sys.argv[1:]):
|
||||||
|
base.main()
|
||||||
|
return
|
||||||
|
output = output_path_from_argv()
|
||||||
|
with open(os.devnull, "w", encoding="utf-8") as sink:
|
||||||
|
with contextlib.redirect_stdout(sink):
|
||||||
|
base.main()
|
||||||
|
result = finalize_result(output)
|
||||||
|
payload = output.read_bytes()
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"output": str(output),
|
||||||
|
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||||
|
"bytes": len(payload),
|
||||||
|
"source_prompts": result["inference_contract"][
|
||||||
|
"total_source_prompts"
|
||||||
|
],
|
||||||
|
"prompt_variants": result["inference_contract"][
|
||||||
|
"total_prompt_variants"
|
||||||
|
],
|
||||||
|
"total_routes": result["inference_contract"][
|
||||||
|
"total_routes_all_conditions_all_moe_layers"
|
||||||
|
],
|
||||||
|
"token_inventory": result[
|
||||||
|
"special_token_family_contract"
|
||||||
|
]["tokenizer_special_inventory"],
|
||||||
|
"render_validation": result[
|
||||||
|
"special_token_family_contract"
|
||||||
|
]["render_validation"],
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -11,6 +11,8 @@
|
|||||||
"check": "astro check",
|
"check": "astro check",
|
||||||
"build:data:deepseek-boundary": "node scripts/build-deepseek-boundary-compact.mjs",
|
"build:data:deepseek-boundary": "node scripts/build-deepseek-boundary-compact.mjs",
|
||||||
"build:data:deepseek-role": "node scripts/build-deepseek-role-marker-compact.mjs",
|
"build:data:deepseek-role": "node scripts/build-deepseek-role-marker-compact.mjs",
|
||||||
|
"build:data:deepseek-special": "node scripts/build-deepseek-special-token-family-compact.mjs",
|
||||||
|
"build:data:deepseek-role-block": "node scripts/build-deepseek-role-marker-block-compact.mjs",
|
||||||
"check:site": "node scripts/check-site.mjs",
|
"check:site": "node scripts/check-site.mjs",
|
||||||
"check:moe-browser": "node scripts/check-moe-browser.mjs",
|
"check:moe-browser": "node scripts/check-moe-browser.mjs",
|
||||||
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
|
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
|
||||||
|
|||||||
@@ -0,0 +1,486 @@
|
|||||||
|
# DeepSeek-V2-Lite 完整角色块 2×2 因子审计
|
||||||
|
|
||||||
|
> 状态：真实官方权重执行（X）<br />
|
||||||
|
> 模型：`deepseek-ai/DeepSeek-V2-Lite` **base checkpoint**<br />
|
||||||
|
> revision：`604d5664dddd88a0433dbae533b7fe9472482de0`<br />
|
||||||
|
> 执行边界：layer 0–6；观测 MoE layer 1–6<br />
|
||||||
|
> 样本：WikiText-2 / TNEWS / HumanEval / GSM8K 各 32 条<br />
|
||||||
|
> 正式运行与独立复跑：byte-exact<br />
|
||||||
|
> 完整 JSON SHA-256：`a703dddb6d04182b1a608c213bfd341cfc32e1801be42c417dca30a505087e82`
|
||||||
|
|
||||||
|
## 0. 一句话先说结论
|
||||||
|
|
||||||
|
固定 DeepSeek-V2-Lite base 权重、128 个公开样本、历史、EOS、目标长度、
|
||||||
|
目标位置和八格 batch 后，把目标前的完整两-token 角色块拆成：
|
||||||
|
|
||||||
|
```text
|
||||||
|
delimiter
|
||||||
|
: x
|
||||||
|
head User User: User x
|
||||||
|
Assistant Assistant: Assistant x
|
||||||
|
```
|
||||||
|
|
||||||
|
目标内容、prompt-balanced 的 24 个 layer×domain 平均 system-edge TV：
|
||||||
|
|
||||||
|
```text
|
||||||
|
User: .037304
|
||||||
|
Assistant: .037015
|
||||||
|
User x .037239
|
||||||
|
Assistant x .036220
|
||||||
|
```
|
||||||
|
|
||||||
|
对 system-edge 距离做 2×2 因子分解：
|
||||||
|
|
||||||
|
```text
|
||||||
|
head main −.000654 9↑ / 15↓
|
||||||
|
delimiter main −.000430 8↑ / 16↓
|
||||||
|
head×delimiter −.000729 6↑ / 18↓
|
||||||
|
```
|
||||||
|
|
||||||
|
大多数 95% bootstrap CI 跨零，且跨层跨域存在相反方向。因此：
|
||||||
|
|
||||||
|
> `User`/`Assistant` 词头和冒号/`x` 分隔符都会直接条件化后续路由，但没有
|
||||||
|
> 任何一个因子对 system message 的路由作用形成统一、跨层跨域的调制方向。
|
||||||
|
|
||||||
|
更细的一条可见模式是：
|
||||||
|
|
||||||
|
> 在本轮选定的单个分隔符对照中，把冒号换成 `x`，多数格会让
|
||||||
|
> User-vs-Assistant 的直接路由距离稍微缩小。
|
||||||
|
|
||||||
|
但这只是 `:`↔`x` 的 one-control 结果，不能泛化成“标点语义”。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 为什么上一轮角色词头实验还不够
|
||||||
|
|
||||||
|
固定 tokenizer 中：
|
||||||
|
|
||||||
|
```text
|
||||||
|
User: → [5726, 25]
|
||||||
|
Assistant: → [77398, 25]
|
||||||
|
```
|
||||||
|
|
||||||
|
上一轮只替换第一个 ID：
|
||||||
|
|
||||||
|
```text
|
||||||
|
User → Assistant / x
|
||||||
|
```
|
||||||
|
|
||||||
|
而冒号 ID `25` 始终保留。这能识别“词头单 ID 条件化”，却不能回答：
|
||||||
|
|
||||||
|
1. 冒号自身是否参与后续路由；
|
||||||
|
2. 词头作用是否依赖冒号；
|
||||||
|
3. `User:`↔`Assistant:` 的差异能否被称为完整角色块效应。
|
||||||
|
|
||||||
|
本轮把两个 token 位置都纳入因子设计，同时保留所有其他序列合同。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 角色标记并不是 special token
|
||||||
|
|
||||||
|
固定 tokenizer：
|
||||||
|
|
||||||
|
| 文本 | token IDs | token 数 | special |
|
||||||
|
|---|---|---:|---:|
|
||||||
|
| `User` | `[5726]` | 1 | 否 |
|
||||||
|
| `Assistant` | `[77398]` | 1 | 否 |
|
||||||
|
| `:` | `[25]` | 1 | 否 |
|
||||||
|
| `x` | `[87]` | 1 | 否 |
|
||||||
|
| `User:` | `[5726, 25]` | 2 | 否 |
|
||||||
|
| `Assistant:` | `[77398, 25]` | 2 | 否 |
|
||||||
|
| EOS | `[100001]` | 1 | 是 |
|
||||||
|
|
||||||
|
所以“角色标记”是两个普通词元组成的块，而不是一个不可拆的专用 role token。
|
||||||
|
|
||||||
|
这也解释了为什么要避免一句含混的话：
|
||||||
|
|
||||||
|
```text
|
||||||
|
我们替换了角色 token。
|
||||||
|
```
|
||||||
|
|
||||||
|
更准确的是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
我们在固定两-token 角色块中独立操纵词头 ID 与分隔符 ID。
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 2×2×2 设计
|
||||||
|
|
||||||
|
三个二元因子：
|
||||||
|
|
||||||
|
- `S`：system message 关闭 / 开启；
|
||||||
|
- `H`：head 为 `User` / `Assistant`；
|
||||||
|
- `D`：delimiter 为冒号 / `x`。
|
||||||
|
|
||||||
|
四个角色块水平：
|
||||||
|
|
||||||
|
| 水平 | token IDs | head | delimiter | 官方 |
|
||||||
|
|---|---|---|---|---:|
|
||||||
|
| `user_colon` | `[5726, 25]` | User | `:` | 是 |
|
||||||
|
| `assistant_colon` | `[77398, 25]` | Assistant | `:` | 否 |
|
||||||
|
| `user_x` | `[5726, 87]` | User | `x` | 否 |
|
||||||
|
| `assistant_x` | `[77398, 87]` | Assistant | `x` | 否 |
|
||||||
|
|
||||||
|
乘上 system 开/关，共八格：
|
||||||
|
|
||||||
|
```text
|
||||||
|
S0/S1 × {User:, Assistant:, User x, Assistant x}
|
||||||
|
```
|
||||||
|
|
||||||
|
所有修改都发生在目标内容之前。历史 assistant 的官方 EOS 和目标之后的
|
||||||
|
generation suffix `Assistant:` 均保持不变。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 编辑合同与验证闸门
|
||||||
|
|
||||||
|
相对官方 `[User, :]`：
|
||||||
|
|
||||||
|
| 水平 | 改动 token 数 |
|
||||||
|
|---|---:|
|
||||||
|
| `user_colon` | 0 |
|
||||||
|
| `assistant_colon` | 1 |
|
||||||
|
| `user_x` | 1 |
|
||||||
|
| `assistant_x` | 2 |
|
||||||
|
|
||||||
|
正式 renderer 验证：
|
||||||
|
|
||||||
|
| 闸门 | 结果 |
|
||||||
|
|---|---:|
|
||||||
|
| source prompts | 128 |
|
||||||
|
| 输入变体 | 1,024 |
|
||||||
|
| system groups | 256 |
|
||||||
|
| 八格同长度 | 256 / 256 |
|
||||||
|
| 八格同目标位置 | 256 / 256 |
|
||||||
|
| official 零改动 | 256 / 256 |
|
||||||
|
| 单 ID 改动 | 512 / 512 |
|
||||||
|
| 双 ID 改动 | 256 / 256 |
|
||||||
|
| 实际 MoE routes | 2,044,224 |
|
||||||
|
|
||||||
|
每个 source 的八个条件进入同一个 32-row BF16 batch。正式运行与独立复跑
|
||||||
|
都是 66,975,110 bytes，完整 JSON byte-exact。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 因子分解怎样读
|
||||||
|
|
||||||
|
设四种角色块下的 system-edge 距离分别为：
|
||||||
|
|
||||||
|
```text
|
||||||
|
E(U:) E(A:) E(Ux) E(Ax)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.1 head main
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.5 × [(E(A:) − E(U:)) + (E(Ax) − E(Ux))]
|
||||||
|
```
|
||||||
|
|
||||||
|
问：平均跨过两个 delimiter，User→Assistant 是否改变 system edge？
|
||||||
|
|
||||||
|
### 5.2 delimiter main
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.5 × [(E(Ux) − E(U:)) + (E(Ax) − E(A:))]
|
||||||
|
```
|
||||||
|
|
||||||
|
问：平均跨过两个 head，冒号→x 是否改变 system edge？
|
||||||
|
|
||||||
|
### 5.3 head × delimiter
|
||||||
|
|
||||||
|
```text
|
||||||
|
(E(Ax) − E(Ux)) − (E(A:) − E(U:))
|
||||||
|
```
|
||||||
|
|
||||||
|
问：head 的作用是否依赖 delimiter？
|
||||||
|
|
||||||
|
这些是带符号的距离差。它们不是标准 TV，也不是准确率效应。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 主结果：没有统一的 system 调制因子
|
||||||
|
|
||||||
|
主口径：
|
||||||
|
|
||||||
|
```text
|
||||||
|
scope = target_content
|
||||||
|
aggregation = prompt_balanced
|
||||||
|
metric = total variation
|
||||||
|
bootstrap = source-paired, 2,000 resamples
|
||||||
|
seed = 20260729
|
||||||
|
```
|
||||||
|
|
||||||
|
24 格平均：
|
||||||
|
|
||||||
|
| 量 | 平均值 | 点估计方向 | CI 全正 / 全负 / 跨零 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| `E(User:)` | .037304 | — | — |
|
||||||
|
| `E(Assistant:)` | .037015 | — | — |
|
||||||
|
| `E(User x)` | .037239 | — | — |
|
||||||
|
| `E(Assistant x)` | .036220 | — | — |
|
||||||
|
| head main | -.000654 | 9↑ / 15↓ | 3 / 3 / 18 |
|
||||||
|
| delimiter main | -.000430 | 8↑ / 16↓ | 1 / 0 / 23 |
|
||||||
|
| head×delimiter | -.000729 | 6↑ / 18↓ | 1 / 1 / 22 |
|
||||||
|
|
||||||
|
可读结论：
|
||||||
|
|
||||||
|
- 四种角色块的平均 system-edge TV 非常接近；
|
||||||
|
- 三个因子效应相对原始 system edge 很小；
|
||||||
|
- 大多数 CI 跨零；
|
||||||
|
- 个别 layer×domain 出现方向明确但互相相反的局部单元；
|
||||||
|
- 因此不能把某个局部格子升级成统一规律。
|
||||||
|
|
||||||
|
代表性局部单元：
|
||||||
|
|
||||||
|
| 层×域 | 观察 |
|
||||||
|
|---|---|
|
||||||
|
| L2 代码 | head main `+.004`，CI `[+.002, +.007]` |
|
||||||
|
| L3 代码 | head main `+.005`，CI `[+.003, +.008]` |
|
||||||
|
| L4 中文 | head main `-.005`，CI `[-.008, -.000]` |
|
||||||
|
| L4 数学 | head main `-.006`，CI `[-.010, -.003]` |
|
||||||
|
| L5 代码 | interaction `+.014`，CI `[+.005, +.023]` |
|
||||||
|
| L6 中文 | interaction `-.009`，CI `[-.017, -.002]` |
|
||||||
|
| L6 代码 | head main `+.005`，CI `[+.001, +.009]` |
|
||||||
|
|
||||||
|
正负局部单元同时存在，恰好是“不存在统一方向”的证据，而不是噪声需要隐藏。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. CV 也不支持单一均衡故事
|
||||||
|
|
||||||
|
目标内容、prompt-balanced 的 system-edge ΔCV 因子：
|
||||||
|
|
||||||
|
| 因子 | 24 格平均 | 点估计方向 | CI 正 / 负 / 跨零 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| head main | +.001756 | 17↑ / 7↓ | 4 / 3 / 17 |
|
||||||
|
| delimiter main | -.003107 | 10↑ / 14↓ | 1 / 5 / 18 |
|
||||||
|
| head×delimiter | -.001543 | 11↑ / 13↓ | 3 / 2 / 19 |
|
||||||
|
|
||||||
|
CV 是专家负载离散程度，不是“模型质量”：
|
||||||
|
|
||||||
|
- CV 增大不等于能力提高；
|
||||||
|
- CV 减小不等于路由更合理；
|
||||||
|
- system-edge ΔCV 也不等于 absolute CV。
|
||||||
|
|
||||||
|
本轮只能说不同角色块对负载分布变化的调制跨层跨域混合。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 直接作用：两个 token 位置都会条件化后续路由
|
||||||
|
|
||||||
|
跨 24 格平均 direct target TV：
|
||||||
|
|
||||||
|
| 直接边 | S0 | S1 | S1−S0 | 点估计方向 |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| head：`User:`↔`Assistant:` | .022399 | .021933 | -.000466 | 10↑ / 14↓ |
|
||||||
|
| head：`User x`↔`Assistant x` | .018726 | .019973 | +.001248 | 14↑ / 10↓ |
|
||||||
|
| delimiter：`User:`↔`User x` | .026560 | .025679 | -.000881 | 12↑ / 12↓ |
|
||||||
|
| delimiter：`Assistant:`↔`Assistant x` | .024495 | .024949 | +.000454 | 12↑ / 12↓ |
|
||||||
|
|
||||||
|
所有四条 direct edge 的 TV 都清晰非零，说明：
|
||||||
|
|
||||||
|
- head 身份能影响后续目标；
|
||||||
|
- delimiter 身份也能影响后续目标；
|
||||||
|
- 但 system 是否放大它们没有统一方向。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. head 的直接作用依赖 delimiter 吗
|
||||||
|
|
||||||
|
比较：
|
||||||
|
|
||||||
|
```text
|
||||||
|
head@x − head@colon
|
||||||
|
```
|
||||||
|
|
||||||
|
跨 24 格平均：
|
||||||
|
|
||||||
|
| system | Δ direct TV | 点估计方向 | CI 正 / 负 / 跨零 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| S0 | -.003673 | 3↑ / 21↓ | 0 / 8 / 16 |
|
||||||
|
| S1 | -.001960 | 4↑ / 20↓ | 1 / 6 / 17 |
|
||||||
|
| S1−S0 | +.001714 | 13↑ / 11↓ | 4 / 0 / 20 |
|
||||||
|
|
||||||
|
因此最窄的描述是：
|
||||||
|
|
||||||
|
> 在本实验选定的 `:`↔`x` 对照中，冒号换成 `x` 后，
|
||||||
|
> User-vs-Assistant 的直接路由距离在多数 layer×domain 略微减小。
|
||||||
|
|
||||||
|
但 system 对这项依赖的 modulation 仍混合。
|
||||||
|
|
||||||
|
不能说：
|
||||||
|
|
||||||
|
> 冒号编码了角色语义，或标点普遍增强角色区分。
|
||||||
|
|
||||||
|
要支持这类说法，至少还需多个频率与句法匹配的 delimiter，对 Chat/SFT
|
||||||
|
checkpoint 做同样设计，并接入生成行为。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. 完整输入为什么出现更明显的 delimiter 效应
|
||||||
|
|
||||||
|
full-input、prompt-balanced 的平均 system-edge TV：
|
||||||
|
|
||||||
|
```text
|
||||||
|
User: .138962
|
||||||
|
Assistant: .139096
|
||||||
|
User x .141328
|
||||||
|
Assistant x .141220
|
||||||
|
```
|
||||||
|
|
||||||
|
对应因子：
|
||||||
|
|
||||||
|
```text
|
||||||
|
head main +.000013
|
||||||
|
delimiter main +.002246
|
||||||
|
head×delimiter −.000242
|
||||||
|
```
|
||||||
|
|
||||||
|
delimiter main 在 24 格为 20↑ / 4↓，CI 分类 14 正 / 1 负 / 9 跨零。
|
||||||
|
|
||||||
|
但完整输入包含被编辑的 delimiter token 本身，因此它无法单独回答“对后续目标
|
||||||
|
的下游影响”。这就是为什么主结论仍使用精确对齐的 `target_content`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. 路由集合稳定性
|
||||||
|
|
||||||
|
head direct edge 的聚合 target top-6 set exact：
|
||||||
|
|
||||||
|
| 层 | `User:`↔`Assistant:` S0 / S1 | `User x`↔`Assistant x` S0 / S1 |
|
||||||
|
|---:|---:|---:|
|
||||||
|
| 1 | 79.9% / 81.8% | 80.4% / 81.7% |
|
||||||
|
| 2 | 81.9% / 82.8% | 83.3% / 84.4% |
|
||||||
|
| 3 | 81.4% / 79.8% | 85.7% / 83.7% |
|
||||||
|
| 4 | 76.8% / 74.3% | 78.4% / 77.0% |
|
||||||
|
| 5 | 71.8% / 73.8% | 77.0% / 74.8% |
|
||||||
|
| 6 | 69.7% / 70.7% | 77.3% / 76.4% |
|
||||||
|
|
||||||
|
delimiter-at-User 的 set exact 从 L1 的约 83% 下降到 L6 的约 67%。
|
||||||
|
这表示两-token 块的微小改动可以在深度中逐步传播，但不是所有目标 token
|
||||||
|
都改变专家集合。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. batch-content 审计
|
||||||
|
|
||||||
|
本轮的 `User:` / `Assistant:` 两类条件与上一轮角色词头实验共享：
|
||||||
|
|
||||||
|
```text
|
||||||
|
message hash 512 / 512 exact
|
||||||
|
rendered hash 512 / 512 exact
|
||||||
|
token-ID hash 512 / 512 exact
|
||||||
|
target contract 512 / 512 exact
|
||||||
|
```
|
||||||
|
|
||||||
|
跨实验共享条件的 target route hash：
|
||||||
|
|
||||||
|
| 层 | exact |
|
||||||
|
|---:|---:|
|
||||||
|
| 1 | 512 / 512 |
|
||||||
|
| 2 | 368 / 512 |
|
||||||
|
| 3 | 279 / 512 |
|
||||||
|
| 4 | 230 / 512 |
|
||||||
|
| 5 | 178 / 512 |
|
||||||
|
| 6 | 156 / 512 |
|
||||||
|
|
||||||
|
原因不是 token 合同变化，而是 companion rows 从上一轮的其他角色词头条件换成
|
||||||
|
本轮的完整两-token 组合；BF16 深层临界 gate 会因此分化。
|
||||||
|
|
||||||
|
正式推断规则：
|
||||||
|
|
||||||
|
> 只使用同一次八格 forward 内的比较；跨实验 exact 仅作为执行审计，
|
||||||
|
> 不跨 batch 拼接效应值。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. 可复现入口
|
||||||
|
|
||||||
|
正式运行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=/tmp/deepseek-v2-lite-pydeps.6ZJAH3:/usr/lib/python3/dist-packages \
|
||||||
|
/home/wuyang/.pyenv/versions/navi-router-cu128/bin/python \
|
||||||
|
experiments/deepseek/v2_lite_routing_role_marker_block_factorial.py \
|
||||||
|
--artifact-dir /tmp/deepseek-v2-lite-artifacts.OEjfce \
|
||||||
|
--human-eval /tmp/llm-atlas-routing-corpus/human-eval/data/HumanEval.jsonl.gz \
|
||||||
|
--gsm8k /tmp/llm-atlas-routing-corpus/gsm8k/grade_school_math/data/test.jsonl \
|
||||||
|
--tnews /tmp/llm-atlas-routing-corpus/tnews/test.json \
|
||||||
|
--tnews-archive /tmp/llm-atlas-routing-corpus/tnews_public.zip \
|
||||||
|
--wikitext /tmp/llm-atlas-routing-corpus/wikitext-2-raw-v1-validation.parquet \
|
||||||
|
--output src/data/deepseek-v2-lite-routing-role-marker-block-factorial.json \
|
||||||
|
--batch-prompts 4 \
|
||||||
|
--bootstrap 2000 \
|
||||||
|
--seed 20260729
|
||||||
|
```
|
||||||
|
|
||||||
|
独立复跑只改输出文件名。前端 compact：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build:data:deepseek-role-block
|
||||||
|
```
|
||||||
|
|
||||||
|
| 文件 | bytes | SHA-256 |
|
||||||
|
|---|---:|---|
|
||||||
|
| 正式运行 JSON | 66,975,110 | `a703dddb…7e82` |
|
||||||
|
| 独立复跑 JSON | 66,975,110 | `a703dddb…7e82` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. 与 prompt sensitivity 文献的关系
|
||||||
|
|
||||||
|
角色块同时包含词汇选择与 delimiter 选择，因此与 prompt sensitivity 文献中的
|
||||||
|
lexical / syntactic / delimiter perturbation 有直接问题关联：
|
||||||
|
|
||||||
|
- [Lexical Sensitivity of Language Models](https://aclanthology.org/2024.emnlp-main.295/)
|
||||||
|
研究词汇变化对模型表现的影响；
|
||||||
|
- [Prompt Sensitivity Prediction](https://aclanthology.org/2024.naacl-long.325/)
|
||||||
|
讨论 prompt variation 的可预测性；
|
||||||
|
- [PROMPTPRISM](https://aclanthology.org/2026.findings-eacl.61.pdf)
|
||||||
|
把 delimiter、prefix、suffix 与 special-token 修改明确分开；
|
||||||
|
- [URIAL](https://proceedings.iclr.cc/paper_files/paper/2024/file/6bcbb4a501dbad0eba1b660c1a55318c-Paper-Conference.pdf)
|
||||||
|
是理解 base model 如何利用对话格式的背景；
|
||||||
|
- [LIMA](https://arxiv.org/abs/2305.11206)
|
||||||
|
提供回合格式与对齐数据的相关背景。
|
||||||
|
|
||||||
|
这些工作说明“格式细节值得测”，但没有预先决定本实验的路由结果。本实验也
|
||||||
|
没有复现它们的行为指标。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15. 允许说什么，不允许说什么
|
||||||
|
|
||||||
|
可以说：
|
||||||
|
|
||||||
|
- `User:` 与 `Assistant:` 在固定 tokenizer 中都是两个普通 token；
|
||||||
|
- head 与 delimiter 的直接替换都能改变后续目标的专家路由；
|
||||||
|
- `:`→`x` 在多数格略微缩小 head direct TV；
|
||||||
|
- head、delimiter 和 interaction 都没有统一调制 system-edge 的方向；
|
||||||
|
- 本轮 edit、位置、长度、对齐与复跑闸门通过。
|
||||||
|
|
||||||
|
不可以说：
|
||||||
|
|
||||||
|
- 冒号就是角色语义；
|
||||||
|
- `User` / `Assistant` head 对应固定专家专长；
|
||||||
|
- 某种角色块让路由“更好”或负载“更合理”；
|
||||||
|
- `x` 代表全部非标点或全部普通 delimiter；
|
||||||
|
- base checkpoint 的路由等价于 Chat/SFT/RL 行为；
|
||||||
|
- 路由 TV 可以替代生成准确率、指令遵循或安全评测；
|
||||||
|
- 六个 MoE 层的结果可直接外推到完整 27 层。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 16. 下一步
|
||||||
|
|
||||||
|
本轮完成了两-token 角色块的固定协议因子分解。接下来最有价值的两条线是：
|
||||||
|
|
||||||
|
1. 在 `DeepSeek-V2-Lite-Chat` 上复现同一 tokenizer 与同一设计，区分
|
||||||
|
base routing 与对齐后行为；
|
||||||
|
2. 下载完整 checkpoint，执行全部 27 层，检查局部模式是否在更深层反转、
|
||||||
|
消失或增强。
|
||||||
|
|
||||||
|
在这两条完成前，网站会把结论停在：
|
||||||
|
|
||||||
|
> **base 权重、layer 1–6、固定公开语料、固定 batch 的路由机制证据。**
|
||||||
@@ -0,0 +1,513 @@
|
|||||||
|
# DeepSeek-V2-Lite 特殊 Token 家族控制审计
|
||||||
|
|
||||||
|
> 状态：真实官方权重执行（X）<br />
|
||||||
|
> 模型：`deepseek-ai/DeepSeek-V2-Lite` **base checkpoint**<br />
|
||||||
|
> revision：`604d5664dddd88a0433dbae533b7fe9472482de0`<br />
|
||||||
|
> 执行边界：layer 0–6；观测 MoE layer 1–6<br />
|
||||||
|
> 样本：WikiText-2 / TNEWS / HumanEval / GSM8K 各 32 条<br />
|
||||||
|
> 正式运行与独立复跑：byte-exact<br />
|
||||||
|
> 完整 JSON SHA-256：`c372c1b03a8b15f615b54ded5d9257a8fc2cdb7728001735d3d4c1d8534af5bf`
|
||||||
|
|
||||||
|
## 0. 一句话先说结论
|
||||||
|
|
||||||
|
固定 DeepSeek-V2-Lite base 权重、128 个公开样本、重复词元历史、角色块、
|
||||||
|
长度、目标位置和八格 batch 后，只把历史 assistant 后的一个边界 ID 设为
|
||||||
|
`EOS / BOS / x / .`：
|
||||||
|
|
||||||
|
```text
|
||||||
|
目标内容 · prompt-balanced · 24 个 layer×domain 平均 system-edge TV
|
||||||
|
|
||||||
|
EOS .037518
|
||||||
|
BOS .048018
|
||||||
|
x .053975
|
||||||
|
句点 .055289
|
||||||
|
```
|
||||||
|
|
||||||
|
相对官方 EOS：
|
||||||
|
|
||||||
|
```text
|
||||||
|
BOS − EOS +.010500 22↑ / 2↓
|
||||||
|
x − EOS +.016457 24↑ / 0↓
|
||||||
|
句点 − EOS +.017771 24↑ / 0↓
|
||||||
|
```
|
||||||
|
|
||||||
|
这说明：
|
||||||
|
|
||||||
|
> 在本实验的固定协议中，边界的具体 token 身份会改变后续目标内容对 system
|
||||||
|
> 开关的路由距离；BOS 不能简单复现 EOS 的作用。
|
||||||
|
|
||||||
|
但是它**不能**推出：
|
||||||
|
|
||||||
|
> “特殊 token 普遍让 system effect 更小”。
|
||||||
|
|
||||||
|
原因非常具体：固定 tokenizer 的 special inventory 只有 EOS 和 BOS，
|
||||||
|
所以二者确实被完全枚举；但普通词元空间接近十万个 ID，本实验只挑了 `x`
|
||||||
|
和句点两个对照。`mean(x, .) − mean(EOS, BOS) = +.011864` 是四个 ID 内的
|
||||||
|
描述性 2-vs-2 汇总，不是对“普通 token 总体”的抽样推断。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 为什么要继续做这一轮
|
||||||
|
|
||||||
|
上一轮边界实验比较：
|
||||||
|
|
||||||
|
```text
|
||||||
|
EOS ↔ x / 句点 / 换行
|
||||||
|
```
|
||||||
|
|
||||||
|
结果表明，三个普通单 token 替换都能改变后续目标路由，而且多数格子的
|
||||||
|
system-edge TV 大于 EOS 条件。但它仍留下一个关键混淆：
|
||||||
|
|
||||||
|
- EOS 是特殊 token；
|
||||||
|
- 三个替换项都不是特殊 token；
|
||||||
|
- 因此差异可能来自 EOS 的具体身份，也可能来自“specialness”。
|
||||||
|
|
||||||
|
本轮加入 BOS，因为 BOS：
|
||||||
|
|
||||||
|
- 与 EOS 一样是 tokenizer 官方声明的 special token；
|
||||||
|
- 与 EOS 一样只有一个 token ID；
|
||||||
|
- 可以在不改变长度、目标位置和 batch shape 的前提下进行单 ID 替换。
|
||||||
|
|
||||||
|
问题随之变成：
|
||||||
|
|
||||||
|
1. BOS 能否复现 EOS 的 system-edge 路由模式？
|
||||||
|
2. EOS/BOS 与两个选定普通对照的描述性均值是否不同？
|
||||||
|
3. 这种差异是否在直接替换、CV、完整输入等口径下也同向？
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 固定 tokenizer 的完整特殊词元库存
|
||||||
|
|
||||||
|
固定 revision 的真实 tokenizer 合同：
|
||||||
|
|
||||||
|
| 项目 | token / ID |
|
||||||
|
|---|---|
|
||||||
|
| `vocab_size` | `100000` |
|
||||||
|
| tokenizer 总长度 | `100002` |
|
||||||
|
| BOS | `<｜begin▁of▁sentence｜>` / `100000` |
|
||||||
|
| EOS | `<｜end▁of▁sentence｜>` / `100001` |
|
||||||
|
| PAD | `100001`，与 EOS 同 ID |
|
||||||
|
| `all_special_ids` | `[100000, 100001]` |
|
||||||
|
| `all_special_tokens` 数量 | `2` |
|
||||||
|
|
||||||
|
因此：
|
||||||
|
|
||||||
|
```text
|
||||||
|
special inventory = {BOS, EOS}
|
||||||
|
```
|
||||||
|
|
||||||
|
本轮确实穷尽了 special inventory，而不是只挑两个看起来特殊的字符串。
|
||||||
|
|
||||||
|
四个实验 ID：
|
||||||
|
|
||||||
|
| 水平 | token ID | special | 官方历史边界 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| `eos` | `100001` | 是 | 是 |
|
||||||
|
| `bos` | `100000` | 是 | 否，反事实 |
|
||||||
|
| `x` | `87` | 否 | 否，反事实 |
|
||||||
|
| `period` | `13` | 否 | 否，反事实 |
|
||||||
|
|
||||||
|
需要特别注意：
|
||||||
|
|
||||||
|
- PAD 与 EOS 共用 ID，所以不能把 PAD 当作第五个独立 special control；
|
||||||
|
- `x` 和句点不是从普通词元总体随机抽样；
|
||||||
|
- 普通词元还有不同频率、脚本、空白、标点和语义类别，本实验没有覆盖。
|
||||||
|
|
||||||
|
官方合同可在固定 revision 的
|
||||||
|
[tokenizer_config.json](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite/blob/604d5664dddd88a0433dbae533b7fe9472482de0/tokenizer_config.json)
|
||||||
|
中核查。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 2×4 实验设计
|
||||||
|
|
||||||
|
两个因子：
|
||||||
|
|
||||||
|
- `S`：system message 关闭 / 开启；
|
||||||
|
- `B`：边界 ID 为 EOS / BOS / `x` / 句点。
|
||||||
|
|
||||||
|
八格：
|
||||||
|
|
||||||
|
| 条件 | system | 边界 ID | 官方模板序列 |
|
||||||
|
|---|---:|---|---:|
|
||||||
|
| `S0 / EOS` | 0 | `100001` | 是 |
|
||||||
|
| `S1 / EOS` | 1 | `100001` | 是 |
|
||||||
|
| `S0 / BOS` | 0 | `100000` | 否 |
|
||||||
|
| `S1 / BOS` | 1 | `100000` | 否 |
|
||||||
|
| `S0 / X` | 0 | `87` | 否 |
|
||||||
|
| `S1 / X` | 1 | `87` | 否 |
|
||||||
|
| `S0 / PERIOD` | 0 | `13` | 否 |
|
||||||
|
| `S1 / PERIOD` | 1 | `13` | 否 |
|
||||||
|
|
||||||
|
序列局部形态：
|
||||||
|
|
||||||
|
```text
|
||||||
|
... Assistant : x [BOUNDARY_ID] User : TARGET ...
|
||||||
|
```
|
||||||
|
|
||||||
|
所有替换发生在官方 renderer 完成之后。除 EOS 外，其余三格是明确的
|
||||||
|
token-ID counterfactual，不是官方模板会自然渲染出的聊天。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 什么被严格固定
|
||||||
|
|
||||||
|
每个 source 的八格共同固定：
|
||||||
|
|
||||||
|
- 同一 checkpoint、revision、tokenizer 与 remote code；
|
||||||
|
- 同一 system 文本；
|
||||||
|
- 同一重复词元 user/assistant 历史；
|
||||||
|
- 同一 `User:` / `Assistant:` 角色块；
|
||||||
|
- 同一目标文本和 canonical 23-token span；
|
||||||
|
- 同一序列长度；
|
||||||
|
- 同一目标绝对起止位置；
|
||||||
|
- 同一 attention mask 和 right-padding；
|
||||||
|
- 同一个 32-row BF16 forward。
|
||||||
|
|
||||||
|
正式验证：
|
||||||
|
|
||||||
|
| 闸门 | 结果 |
|
||||||
|
|---|---:|
|
||||||
|
| source prompts | 128 |
|
||||||
|
| 输入变体 | 1,024 |
|
||||||
|
| system groups | 256 |
|
||||||
|
| 八格同长度 | 256 / 256 |
|
||||||
|
| 八格同目标位置 | 256 / 256 |
|
||||||
|
| 官方 EOS 零 ID 改动 | 256 / 256 |
|
||||||
|
| 三类反事实恰好一 ID 改动 | 768 / 768 |
|
||||||
|
| 实际 MoE routes | 2,044,224 |
|
||||||
|
|
||||||
|
正式运行与独立复跑都是 54,440,827 bytes，完整 JSON byte-exact。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 统计量回答的是三个不同问题
|
||||||
|
|
||||||
|
### 5.1 system edge
|
||||||
|
|
||||||
|
固定边界 `b`：
|
||||||
|
|
||||||
|
```text
|
||||||
|
TV(P(S0, b), P(S1, b))
|
||||||
|
```
|
||||||
|
|
||||||
|
它问：开关 system 时，目标内容的专家份额移动了多少？
|
||||||
|
|
||||||
|
### 5.2 相对 EOS 的 system-edge contrast
|
||||||
|
|
||||||
|
```text
|
||||||
|
TV_system(b) − TV_system(EOS)
|
||||||
|
```
|
||||||
|
|
||||||
|
它问：把边界 ID 换掉后，system-edge 距离变大还是变小？
|
||||||
|
|
||||||
|
### 5.3 直接替换距离
|
||||||
|
|
||||||
|
固定 system 水平 `s`：
|
||||||
|
|
||||||
|
```text
|
||||||
|
TV(P(s, EOS), P(s, b))
|
||||||
|
```
|
||||||
|
|
||||||
|
它问：只改这个历史 ID，本身让后续目标路由移动了多少？
|
||||||
|
|
||||||
|
直接替换距离非零，不代表 system-edge contrast 必然同方向。这两张账不能相减
|
||||||
|
之后只保留一个顺眼结论。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 主结果：目标内容的 system edge
|
||||||
|
|
||||||
|
主口径是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
scope = target_content
|
||||||
|
aggregation = prompt_balanced
|
||||||
|
metric = total variation
|
||||||
|
bootstrap = source-paired, 2,000 resamples
|
||||||
|
seed = 20260729
|
||||||
|
```
|
||||||
|
|
||||||
|
24 格 system-edge TV：
|
||||||
|
|
||||||
|
| 层 | 域 | EOS | BOS | x | 句点 | BOS−EOS [95% CI] |
|
||||||
|
|---:|---|---:|---:|---:|---:|---:|
|
||||||
|
| 1 | 英文 | .056 | .108 | .084 | .081 | +.052 [.040, .061] |
|
||||||
|
| 1 | 中文 | .028 | .057 | .051 | .052 | +.030 [.022, .037] |
|
||||||
|
| 1 | 代码 | .057 | .071 | .080 | .081 | +.014 [.007, .022] |
|
||||||
|
| 1 | 数学 | .078 | .125 | .107 | .105 | +.047 [.039, .053] |
|
||||||
|
| 2 | 英文 | .030 | .039 | .044 | .043 | +.009 [.002, .017] |
|
||||||
|
| 2 | 中文 | .020 | .026 | .027 | .029 | +.005 [.001, .013] |
|
||||||
|
| 2 | 代码 | .033 | .044 | .054 | .050 | +.011 [.005, .019] |
|
||||||
|
| 2 | 数学 | .041 | .049 | .052 | .055 | +.008 [.003, .016] |
|
||||||
|
| 3 | 英文 | .030 | .042 | .043 | .044 | +.012 [.005, .017] |
|
||||||
|
| 3 | 中文 | .023 | .027 | .032 | .028 | +.004 [.000, .013] |
|
||||||
|
| 3 | 代码 | .041 | .049 | .047 | .044 | +.008 [.002, .014] |
|
||||||
|
| 3 | 数学 | .032 | .039 | .035 | .036 | +.006 [.001, .014] |
|
||||||
|
| 4 | 英文 | .034 | .040 | .050 | .052 | +.006 [.001, .014] |
|
||||||
|
| 4 | 中文 | .030 | .032 | .041 | .042 | +.003 [-.001, .010] |
|
||||||
|
| 4 | 代码 | .041 | .048 | .061 | .075 | +.007 [.001, .015] |
|
||||||
|
| 4 | 数学 | .051 | .057 | .069 | .070 | +.006 [.000, .015] |
|
||||||
|
| 5 | 英文 | .034 | .039 | .050 | .052 | +.006 [-.000, .013] |
|
||||||
|
| 5 | 中文 | .026 | .030 | .037 | .045 | +.004 [.001, .013] |
|
||||||
|
| 5 | 代码 | .039 | .039 | .051 | .051 | +.000 [-.005, .008] |
|
||||||
|
| 5 | 数学 | .044 | .042 | .055 | .053 | -.002 [-.008, .006] |
|
||||||
|
| 6 | 英文 | .031 | .038 | .056 | .060 | +.007 [.001, .014] |
|
||||||
|
| 6 | 中文 | .026 | .029 | .044 | .048 | +.003 [.000, .011] |
|
||||||
|
| 6 | 代码 | .032 | .043 | .061 | .062 | +.011 [.006, .018] |
|
||||||
|
| 6 | 数学 | .045 | .039 | .066 | .070 | -.006 [-.011, .002] |
|
||||||
|
|
||||||
|
把 24 格作为描述性层×域单元汇总：
|
||||||
|
|
||||||
|
| contrast | 平均 ΔTV | 点估计方向 | CI 全正 / 全负 / 跨零 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| BOS−EOS | +.010500 | 22↑ / 2↓ | 19 / 0 / 5 |
|
||||||
|
| x−EOS | +.016457 | 24↑ / 0↓ | 23 / 0 / 1 |
|
||||||
|
| 句点−EOS | +.017771 | 24↑ / 0↓ | 22 / 0 / 2 |
|
||||||
|
|
||||||
|
可读结论：
|
||||||
|
|
||||||
|
- BOS 在多数格比 EOS 产生更大的 system-edge 路由距离；
|
||||||
|
- 但 L5/L6 数学已经出现反向点估计；
|
||||||
|
- `x` 和句点的差异更大、更一致；
|
||||||
|
- 因此“BOS 是特殊 token”并不足以复现 EOS 的模式。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 四 ID 的描述性 family summary
|
||||||
|
|
||||||
|
只在同一 shared-source bootstrap 内计算：
|
||||||
|
|
||||||
|
```text
|
||||||
|
special mean = mean(EOS, BOS)
|
||||||
|
ordinary-control mean = mean(x, period)
|
||||||
|
ordinary − special
|
||||||
|
```
|
||||||
|
|
||||||
|
目标内容、prompt-balanced 的 24 格均值：
|
||||||
|
|
||||||
|
```text
|
||||||
|
special mean .042768
|
||||||
|
ordinary-control mean .054632
|
||||||
|
ordinary − special +.011864
|
||||||
|
```
|
||||||
|
|
||||||
|
24 / 24 点估计为正；95% CI 分类为 20 全正、0 全负、4 跨零。
|
||||||
|
|
||||||
|
正确表述：
|
||||||
|
|
||||||
|
> 在这四个具体 ID 中，两个选定普通对照的 system-edge TV 均值高于完整
|
||||||
|
> special inventory 的均值。
|
||||||
|
|
||||||
|
错误表述：
|
||||||
|
|
||||||
|
> 普通 token 普遍比特殊 token 产生更大的 system effect。
|
||||||
|
|
||||||
|
后一句需要对普通 token 总体定义抽样框、抽取更多频率与类别匹配 ID，并进行
|
||||||
|
层级推断；本轮没有做。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 直接替换没有给出同样简单的故事
|
||||||
|
|
||||||
|
跨 24 格平均 direct target TV：
|
||||||
|
|
||||||
|
| EOS→替换项 | S0 | S1 | S1−S0 | 点估计方向 |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| BOS | .043479 | .040793 | -.002686 | 7↑ / 17↓ |
|
||||||
|
| x | .036428 | .044132 | +.007704 | 20↑ / 4↓ |
|
||||||
|
| 句点 | .036296 | .045606 | +.009310 | 22↑ / 2↓ |
|
||||||
|
|
||||||
|
其中：
|
||||||
|
|
||||||
|
- BOS direct interaction 的 CI 分类为 3 正 / 4 负 / 17 跨零；
|
||||||
|
- `x` 为 10 正 / 0 负 / 14 跨零；
|
||||||
|
- 句点为 12 正 / 0 负 / 12 跨零。
|
||||||
|
|
||||||
|
所以本轮不是“所有替换都被 system 统一放大”。BOS 与两个普通对照在
|
||||||
|
直接作用的 system modulation 上呈现不同方向。
|
||||||
|
|
||||||
|
CV 也没有形成简单类别规律。BOS−EOS 的 system-edge ΔCV 跨 24 格平均
|
||||||
|
`+.009904`，点估计 `13↑ / 11↓`，CI `7 / 3 / 14`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 为什么完整输入只能作稳健性账
|
||||||
|
|
||||||
|
完整输入会把被编辑的边界 token 自己也计入聚合，因此它混合：
|
||||||
|
|
||||||
|
1. 边界 ID 自身的路由；
|
||||||
|
2. 后续角色块；
|
||||||
|
3. 对齐目标内容的下游传播。
|
||||||
|
|
||||||
|
full-input、prompt-balanced 的平均 system-edge TV：
|
||||||
|
|
||||||
|
```text
|
||||||
|
EOS .138988
|
||||||
|
BOS .137253
|
||||||
|
x .141592
|
||||||
|
句点 .140312
|
||||||
|
```
|
||||||
|
|
||||||
|
BOS−EOS 平均为 `-.001735`，点估计 `6↑ / 18↓`；方向与目标内容主结果不同。
|
||||||
|
四 ID family summary 为 `+.002831`，也远小于目标内容的 `+.011864`。
|
||||||
|
|
||||||
|
这不是矛盾，而是统计对象不同。因果主张只落在：
|
||||||
|
|
||||||
|
> **边界 token 之后、精确对齐的目标内容路由。**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. 路由集合稳定性
|
||||||
|
|
||||||
|
以 EOS↔BOS 的直接替换为例，聚合后的 target top-6 set exact：
|
||||||
|
|
||||||
|
| 层 | S0 | S1 | mean Jaccard S0 / S1 |
|
||||||
|
|---:|---:|---:|---:|
|
||||||
|
| 1 | 40.9% | 45.4% | .795 / .817 |
|
||||||
|
| 2 | 52.8% | 53.2% | .851 / .853 |
|
||||||
|
| 3 | 60.4% | 57.4% | .880 / .870 |
|
||||||
|
| 4 | 59.8% | 54.8% | .878 / .862 |
|
||||||
|
| 5 | 59.5% | 58.6% | .875 / .874 |
|
||||||
|
| 6 | 58.9% | 57.9% | .872 / .870 |
|
||||||
|
|
||||||
|
这说明一个历史单 ID 的变化可以传播到后续专家集合，但并非每个 token 都换
|
||||||
|
专家；路由变化是部分、分层且受内容影响的。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. batch-content 审计
|
||||||
|
|
||||||
|
当前实验与上一轮边界实验共享 EOS / `x` / 句点三类条件：
|
||||||
|
|
||||||
|
```text
|
||||||
|
message hash 768 / 768 exact
|
||||||
|
rendered hash 768 / 768 exact
|
||||||
|
token-ID hash 768 / 768 exact
|
||||||
|
target contract 768 / 768 exact
|
||||||
|
```
|
||||||
|
|
||||||
|
然而把同一行放入不同 companion rows 的 32-row BF16 batch 后，深层临界
|
||||||
|
gate 决策不必 byte-exact。把三个共享水平合计：
|
||||||
|
|
||||||
|
| 层 | target route hash exact | target integer load exact |
|
||||||
|
|---:|---:|---:|
|
||||||
|
| 1 | 768 / 768 | 768 / 768 |
|
||||||
|
| 2 | 610 / 768 | 667 / 768 |
|
||||||
|
| 3 | 477 / 768 | 572 / 768 |
|
||||||
|
| 4 | 368 / 768 | 496 / 768 |
|
||||||
|
| 5 | 269 / 768 | 437 / 768 |
|
||||||
|
| 6 | 244 / 768 | 402 / 768 |
|
||||||
|
|
||||||
|
因此：
|
||||||
|
|
||||||
|
> BF16 的 batch content 是执行合同的一部分。正式统计只比较本轮同一个
|
||||||
|
> 八格 batch 内的 source-paired 条件，不跨实验拼接深层数值。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. 可复现入口
|
||||||
|
|
||||||
|
正式运行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PYTHONPATH=/tmp/deepseek-v2-lite-pydeps.6ZJAH3:/usr/lib/python3/dist-packages \
|
||||||
|
/home/wuyang/.pyenv/versions/navi-router-cu128/bin/python \
|
||||||
|
experiments/deepseek/v2_lite_routing_special_token_family_control.py \
|
||||||
|
--artifact-dir /tmp/deepseek-v2-lite-artifacts.OEjfce \
|
||||||
|
--human-eval /tmp/llm-atlas-routing-corpus/human-eval/data/HumanEval.jsonl.gz \
|
||||||
|
--gsm8k /tmp/llm-atlas-routing-corpus/gsm8k/grade_school_math/data/test.jsonl \
|
||||||
|
--tnews /tmp/llm-atlas-routing-corpus/tnews/test.json \
|
||||||
|
--tnews-archive /tmp/llm-atlas-routing-corpus/tnews_public.zip \
|
||||||
|
--wikitext /tmp/llm-atlas-routing-corpus/wikitext-2-raw-v1-validation.parquet \
|
||||||
|
--output src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||||
|
--batch-prompts 4 \
|
||||||
|
--bootstrap 2000 \
|
||||||
|
--seed 20260729
|
||||||
|
```
|
||||||
|
|
||||||
|
独立复跑只改输出文件名。确定性前端 compact：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run build:data:deepseek-special
|
||||||
|
```
|
||||||
|
|
||||||
|
完整工件：
|
||||||
|
|
||||||
|
| 文件 | bytes | SHA-256 |
|
||||||
|
|---|---:|---|
|
||||||
|
| 正式运行 JSON | 54,440,827 | `c372c1b0…4af5bf` |
|
||||||
|
| 独立复跑 JSON | 54,440,827 | `c372c1b0…4af5bf` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. 与已有论文怎样对齐
|
||||||
|
|
||||||
|
外部研究已经反复说明 prompt 的词汇、格式与分隔符变化可能显著影响模型输出：
|
||||||
|
|
||||||
|
- [Lexical Sensitivity of Language Models](https://aclanthology.org/2024.emnlp-main.295/)
|
||||||
|
研究词汇变化对任务表现的影响；
|
||||||
|
- [Prompt Sensitivity Prediction](https://aclanthology.org/2024.naacl-long.325/)
|
||||||
|
讨论模型与任务对 prompt 变化的敏感性；
|
||||||
|
- [PROMPTPRISM](https://aclanthology.org/2026.findings-eacl.61.pdf)
|
||||||
|
在 taxonomy 中明确区分 delimiter、prefix、suffix 与 special-token 修改；
|
||||||
|
- [URIAL](https://proceedings.iclr.cc/paper_files/paper/2024/file/6bcbb4a501dbad0eba1b660c1a55318c-Paper-Conference.pdf)
|
||||||
|
展示 base model 对格式化与对齐样式的利用；
|
||||||
|
- [LIMA](https://arxiv.org/abs/2305.11206)
|
||||||
|
是理解对话格式、回合边界与少量高质量对齐数据的背景材料。
|
||||||
|
|
||||||
|
这些论文提供问题背景，但不替本实验背书。尤其不能把：
|
||||||
|
|
||||||
|
```text
|
||||||
|
路由 TV 变化
|
||||||
|
```
|
||||||
|
|
||||||
|
直接改写成：
|
||||||
|
|
||||||
|
```text
|
||||||
|
准确率、质量、安全性或指令遵循变化
|
||||||
|
```
|
||||||
|
|
||||||
|
本实验没有生成答案，也没有行为评测。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. 允许说什么，不允许说什么
|
||||||
|
|
||||||
|
可以说：
|
||||||
|
|
||||||
|
- 固定 tokenizer 的 special inventory 只有 BOS 和 EOS；
|
||||||
|
- 在固定协议里，BOS 不复现 EOS 的目标 system-edge 路由模式；
|
||||||
|
- `x`/句点相对 EOS 在 24 格中表现出更大的目标 system-edge TV；
|
||||||
|
- 四 ID 的 2-vs-2 family summary 是描述性结果；
|
||||||
|
- 单 ID 边界变化能传播到后续目标的 MoE routing；
|
||||||
|
- 正式运行与独立复跑 byte-exact。
|
||||||
|
|
||||||
|
不可以说：
|
||||||
|
|
||||||
|
- 普通 token 普遍比 special token 更有效；
|
||||||
|
- EOS 让模型更懂回合或让回答更好；
|
||||||
|
- BOS/EOS 具有已识别的专家语义；
|
||||||
|
- 结果适用于所有 27 层、Chat/SFT/RL 权重或线上流量；
|
||||||
|
- 路由差异等于性能差异；
|
||||||
|
- 24 个 layer×domain 单元是 24 个独立随机样本。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15. 下一步
|
||||||
|
|
||||||
|
这一轮解决了 special inventory，却还没有解决角色块内部结构。固定模板中的：
|
||||||
|
|
||||||
|
```text
|
||||||
|
User: = [5726, 25]
|
||||||
|
Assistant: = [77398, 25]
|
||||||
|
```
|
||||||
|
|
||||||
|
都是两个普通 token。所以下一轮采用完整 2×2：
|
||||||
|
|
||||||
|
```text
|
||||||
|
head = User / Assistant
|
||||||
|
delimiter = : / x
|
||||||
|
```
|
||||||
|
|
||||||
|
用同一个 batch 分离词头主效应、分隔符主效应与二者 interaction。
|
||||||
@@ -0,0 +1,289 @@
|
|||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { readFileSync, statSync, writeFileSync } from "node:fs";
|
||||||
|
import { resolve } from "node:path";
|
||||||
|
|
||||||
|
const root = resolve(import.meta.dirname, "..");
|
||||||
|
const mainPath = resolve(
|
||||||
|
root,
|
||||||
|
"src/data/deepseek-v2-lite-routing-role-marker-block-factorial.json",
|
||||||
|
);
|
||||||
|
const reproPath = resolve(
|
||||||
|
root,
|
||||||
|
"src/data/deepseek-v2-lite-routing-role-marker-block-factorial-repro.json",
|
||||||
|
);
|
||||||
|
const roleHeadPath = resolve(
|
||||||
|
root,
|
||||||
|
"src/data/deepseek-v2-lite-routing-role-marker-head-control.json",
|
||||||
|
);
|
||||||
|
const outputPath = resolve(
|
||||||
|
root,
|
||||||
|
"src/data/deepseek-v2-lite-routing-role-marker-block-factorial-compact.json",
|
||||||
|
);
|
||||||
|
|
||||||
|
const sha256 = (path) => createHash("sha256")
|
||||||
|
.update(readFileSync(path))
|
||||||
|
.digest("hex");
|
||||||
|
|
||||||
|
const mainSha256 = sha256(mainPath);
|
||||||
|
const reproSha256 = sha256(reproPath);
|
||||||
|
const mainBytes = statSync(mainPath).size;
|
||||||
|
const reproBytes = statSync(reproPath).size;
|
||||||
|
const exact = mainSha256 === reproSha256 && mainBytes === reproBytes;
|
||||||
|
if (!exact) {
|
||||||
|
throw new Error("role-block formal run and rerun are not byte-exact");
|
||||||
|
}
|
||||||
|
|
||||||
|
const block = JSON.parse(readFileSync(mainPath, "utf8"));
|
||||||
|
const head = JSON.parse(readFileSync(roleHeadPath, "utf8"));
|
||||||
|
const blockEdges = [
|
||||||
|
"system_user_colon",
|
||||||
|
"system_assistant_colon",
|
||||||
|
"system_user_x",
|
||||||
|
"system_assistant_x",
|
||||||
|
"assistant_colon_at_s0",
|
||||||
|
"assistant_colon_at_s1",
|
||||||
|
"user_x_at_s0",
|
||||||
|
"user_x_at_s1",
|
||||||
|
"assistant_x_at_s0",
|
||||||
|
"assistant_x_at_s1",
|
||||||
|
"head_at_x_s0",
|
||||||
|
"head_at_x_s1",
|
||||||
|
"delimiter_at_assistant_s0",
|
||||||
|
"delimiter_at_assistant_s1",
|
||||||
|
];
|
||||||
|
|
||||||
|
const aggregateAlignment = (layer, domain, edge) => {
|
||||||
|
const rows = layer.prompts
|
||||||
|
.filter((prompt) => prompt.domain === domain)
|
||||||
|
.map((prompt) => prompt.alignments[edge]);
|
||||||
|
const aligned = rows.reduce(
|
||||||
|
(sum, row) => sum + row.aligned_tokens,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
const setExact = rows.reduce(
|
||||||
|
(sum, row) => sum + row.set_topk_exact,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
const orderedExact = rows.reduce(
|
||||||
|
(sum, row) => sum + row.ordered_topk_exact,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
const weightedJaccard = rows.reduce(
|
||||||
|
(sum, row) => sum + row.mean_jaccard * row.aligned_tokens,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
return {
|
||||||
|
aligned,
|
||||||
|
setExactRate: setExact / aligned,
|
||||||
|
orderedExactRate: orderedExact / aligned,
|
||||||
|
meanJaccard: weightedJaccard / aligned,
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
const sharedMappings = {
|
||||||
|
user_colon: "official",
|
||||||
|
assistant_colon: "target_assistant",
|
||||||
|
};
|
||||||
|
const sourceById = new Map(
|
||||||
|
head.corpus_contract.selected.map((source) => [source.id, source]),
|
||||||
|
);
|
||||||
|
const tokenContract = {
|
||||||
|
compared: 0,
|
||||||
|
messageHashExact: 0,
|
||||||
|
renderedHashExact: 0,
|
||||||
|
tokenIdHashExact: 0,
|
||||||
|
targetContractExact: 0,
|
||||||
|
};
|
||||||
|
for (const source of block.corpus_contract.selected) {
|
||||||
|
const previousSource = sourceById.get(source.id);
|
||||||
|
if (!previousSource) throw new Error(`role-head source missing: ${source.id}`);
|
||||||
|
for (const system of [0, 1]) {
|
||||||
|
for (const [level, previousLevel] of Object.entries(sharedMappings)) {
|
||||||
|
const current = source.conditions[`s${system}_${level}`];
|
||||||
|
const previous = (
|
||||||
|
previousSource.conditions[`s${system}_${previousLevel}`]
|
||||||
|
);
|
||||||
|
tokenContract.compared += 1;
|
||||||
|
tokenContract.messageHashExact += (
|
||||||
|
current.messages_sha256 === previous.messages_sha256
|
||||||
|
);
|
||||||
|
tokenContract.renderedHashExact += (
|
||||||
|
current.rendered_sha256 === previous.rendered_sha256
|
||||||
|
);
|
||||||
|
tokenContract.tokenIdHashExact += (
|
||||||
|
current.token_ids_sha256 === previous.token_ids_sha256
|
||||||
|
);
|
||||||
|
tokenContract.targetContractExact += (
|
||||||
|
current.tokens === previous.tokens
|
||||||
|
&& current.content_tokens === previous.content_tokens
|
||||||
|
&& current.aligned_content_tokens
|
||||||
|
=== previous.aligned_content_tokens
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const crossBatch = block.layers.slice(1).flatMap((layer) => {
|
||||||
|
const previousLayer = head.layers.find(
|
||||||
|
(candidate) => candidate.layer === layer.layer,
|
||||||
|
);
|
||||||
|
const previousById = new Map(
|
||||||
|
previousLayer.prompts.map((prompt) => [prompt.id, prompt]),
|
||||||
|
);
|
||||||
|
return Object.entries(sharedMappings).map(([level, previousLevel]) => {
|
||||||
|
const counts = {
|
||||||
|
compared: 0,
|
||||||
|
fullRouteHashExact: 0,
|
||||||
|
targetRouteHashExact: 0,
|
||||||
|
fullLoadExact: 0,
|
||||||
|
targetLoadExact: 0,
|
||||||
|
};
|
||||||
|
for (const prompt of layer.prompts) {
|
||||||
|
const previousPrompt = previousById.get(prompt.id);
|
||||||
|
if (!previousPrompt) {
|
||||||
|
throw new Error(`role-head prompt missing: ${prompt.id}`);
|
||||||
|
}
|
||||||
|
for (const system of [0, 1]) {
|
||||||
|
const current = prompt.conditions[`s${system}_${level}`];
|
||||||
|
const previous = (
|
||||||
|
previousPrompt.conditions[`s${system}_${previousLevel}`]
|
||||||
|
);
|
||||||
|
counts.compared += 1;
|
||||||
|
counts.fullRouteHashExact += (
|
||||||
|
current.topk_sha256 === previous.topk_sha256
|
||||||
|
);
|
||||||
|
counts.targetRouteHashExact += (
|
||||||
|
current.content_topk_sha256 === previous.content_topk_sha256
|
||||||
|
);
|
||||||
|
counts.fullLoadExact += (
|
||||||
|
JSON.stringify(current.full_load)
|
||||||
|
=== JSON.stringify(previous.full_load)
|
||||||
|
);
|
||||||
|
counts.targetLoadExact += (
|
||||||
|
JSON.stringify(current.content_load)
|
||||||
|
=== JSON.stringify(previous.content_load)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
layer: layer.layer,
|
||||||
|
level,
|
||||||
|
previousLevel,
|
||||||
|
...counts,
|
||||||
|
};
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
const compact = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
source: {
|
||||||
|
mainSha256,
|
||||||
|
reproSha256,
|
||||||
|
mainBytes,
|
||||||
|
reproBytes,
|
||||||
|
exact,
|
||||||
|
},
|
||||||
|
domains: block.corpus_contract.domains,
|
||||||
|
labels: block.corpus_contract.domain_labels,
|
||||||
|
inference: block.inference_contract,
|
||||||
|
contract: {
|
||||||
|
tokenIds: block.role_marker_block_contract.role_token_ids,
|
||||||
|
officialIds: (
|
||||||
|
block.role_marker_block_contract.official_target_block_ids
|
||||||
|
),
|
||||||
|
levels: block.role_marker_block_contract.level_factors,
|
||||||
|
validation: block.role_marker_block_contract.render_validation,
|
||||||
|
official: block.boundary.official_serialization_by_role_block,
|
||||||
|
tokenContractAgainstRoleHeadRun: tokenContract,
|
||||||
|
},
|
||||||
|
crossBatch,
|
||||||
|
layers: block.layers.slice(1).map((layer) => ({
|
||||||
|
layer: layer.layer,
|
||||||
|
alignment: Object.fromEntries(
|
||||||
|
block.corpus_contract.domains.map((domain) => [
|
||||||
|
domain,
|
||||||
|
Object.fromEntries(
|
||||||
|
blockEdges.map((edge) => [
|
||||||
|
edge,
|
||||||
|
aggregateAlignment(layer, domain, edge),
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
scopes: Object.fromEntries(
|
||||||
|
["target_content", "full_input"].map((scope) => [
|
||||||
|
scope,
|
||||||
|
{
|
||||||
|
modes: Object.fromEntries(
|
||||||
|
["prompt_balanced", "token_weighted"].map((mode) => {
|
||||||
|
const control = (
|
||||||
|
layer.statistics[scope].modes[mode].role_block_control
|
||||||
|
);
|
||||||
|
return [
|
||||||
|
mode,
|
||||||
|
Object.fromEntries(
|
||||||
|
block.corpus_contract.domains.map((domain) => [
|
||||||
|
domain,
|
||||||
|
{
|
||||||
|
distances: control[domain].system_edge_distances,
|
||||||
|
contrasts: (
|
||||||
|
control[domain].system_edge_distance_contrasts
|
||||||
|
),
|
||||||
|
cvEdges: control[domain].metric_system_edges.cv,
|
||||||
|
cvContrasts: (
|
||||||
|
control[domain].metric_system_edge_contrasts.cv
|
||||||
|
),
|
||||||
|
direct: control[domain].direct_substitutions,
|
||||||
|
factorial: {
|
||||||
|
factorCoding: (
|
||||||
|
control[domain].role_block_factorial.factor_coding
|
||||||
|
),
|
||||||
|
systemEdgeDistanceEffects: (
|
||||||
|
control[domain].role_block_factorial
|
||||||
|
.system_edge_distance_effects
|
||||||
|
),
|
||||||
|
metricSystemEdgeEffects: (
|
||||||
|
control[domain].role_block_factorial
|
||||||
|
.metric_system_edge_effects
|
||||||
|
),
|
||||||
|
distributionSystemEdgeMagnitudes: (
|
||||||
|
Object.fromEntries(
|
||||||
|
Object.entries(
|
||||||
|
control[domain].role_block_factorial
|
||||||
|
.distribution_system_edge_effects,
|
||||||
|
).map(([name, value]) => [
|
||||||
|
name,
|
||||||
|
value.half_l1_magnitude,
|
||||||
|
]),
|
||||||
|
)
|
||||||
|
),
|
||||||
|
directFactorEdges: (
|
||||||
|
control[domain].role_block_factorial
|
||||||
|
.direct_factor_edges
|
||||||
|
),
|
||||||
|
directEffectDependencies: (
|
||||||
|
control[domain].role_block_factorial
|
||||||
|
.direct_effect_dependencies
|
||||||
|
),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
];
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
},
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
|
||||||
|
writeFileSync(
|
||||||
|
outputPath,
|
||||||
|
`${JSON.stringify(compact, null, 2)}\n`,
|
||||||
|
"utf8",
|
||||||
|
);
|
||||||
|
process.stdout.write(
|
||||||
|
`${outputPath}\n${mainSha256}\n${mainBytes} bytes source → `
|
||||||
|
+ `${statSync(outputPath).size} bytes compact\n`,
|
||||||
|
);
|
||||||
@@ -0,0 +1,245 @@
|
|||||||
|
import { createHash } from "node:crypto";
|
||||||
|
import { readFileSync, statSync, writeFileSync } from "node:fs";
|
||||||
|
import { resolve } from "node:path";
|
||||||
|
|
||||||
|
const root = resolve(import.meta.dirname, "..");
|
||||||
|
const mainPath = resolve(
|
||||||
|
root,
|
||||||
|
"src/data/deepseek-v2-lite-routing-special-token-family-control.json",
|
||||||
|
);
|
||||||
|
const reproPath = resolve(
|
||||||
|
root,
|
||||||
|
"src/data/deepseek-v2-lite-routing-special-token-family-control-repro.json",
|
||||||
|
);
|
||||||
|
const boundaryPath = resolve(
|
||||||
|
root,
|
||||||
|
"src/data/deepseek-v2-lite-routing-history-boundary-token-control.json",
|
||||||
|
);
|
||||||
|
const outputPath = resolve(
|
||||||
|
root,
|
||||||
|
"src/data/deepseek-v2-lite-routing-special-token-family-control-compact.json",
|
||||||
|
);
|
||||||
|
|
||||||
|
const sha256 = (path) => createHash("sha256")
|
||||||
|
.update(readFileSync(path))
|
||||||
|
.digest("hex");
|
||||||
|
|
||||||
|
const mainSha256 = sha256(mainPath);
|
||||||
|
const reproSha256 = sha256(reproPath);
|
||||||
|
const mainBytes = statSync(mainPath).size;
|
||||||
|
const reproBytes = statSync(reproPath).size;
|
||||||
|
const exact = mainSha256 === reproSha256 && mainBytes === reproBytes;
|
||||||
|
if (!exact) {
|
||||||
|
throw new Error("special-token formal run and rerun are not byte-exact");
|
||||||
|
}
|
||||||
|
|
||||||
|
const family = JSON.parse(readFileSync(mainPath, "utf8"));
|
||||||
|
const boundary = JSON.parse(readFileSync(boundaryPath, "utf8"));
|
||||||
|
const familyEdges = [
|
||||||
|
"system_eos",
|
||||||
|
"system_bos",
|
||||||
|
"system_x",
|
||||||
|
"system_period",
|
||||||
|
"bos_at_s0",
|
||||||
|
"bos_at_s1",
|
||||||
|
"x_at_s0",
|
||||||
|
"x_at_s1",
|
||||||
|
"period_at_s0",
|
||||||
|
"period_at_s1",
|
||||||
|
];
|
||||||
|
|
||||||
|
const aggregateAlignment = (layer, domain, edge) => {
|
||||||
|
const rows = layer.prompts
|
||||||
|
.filter((prompt) => prompt.domain === domain)
|
||||||
|
.map((prompt) => prompt.alignments[edge]);
|
||||||
|
const aligned = rows.reduce(
|
||||||
|
(sum, row) => sum + row.aligned_tokens,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
const setExact = rows.reduce(
|
||||||
|
(sum, row) => sum + row.set_topk_exact,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
const orderedExact = rows.reduce(
|
||||||
|
(sum, row) => sum + row.ordered_topk_exact,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
const weightedJaccard = rows.reduce(
|
||||||
|
(sum, row) => sum + row.mean_jaccard * row.aligned_tokens,
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
return {
|
||||||
|
aligned,
|
||||||
|
setExactRate: setExact / aligned,
|
||||||
|
orderedExactRate: orderedExact / aligned,
|
||||||
|
meanJaccard: weightedJaccard / aligned,
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
const sharedLevels = ["eos", "x", "period"];
|
||||||
|
const sourceById = new Map(
|
||||||
|
boundary.corpus_contract.selected.map((source) => [source.id, source]),
|
||||||
|
);
|
||||||
|
const tokenContract = {
|
||||||
|
compared: 0,
|
||||||
|
messageHashExact: 0,
|
||||||
|
renderedHashExact: 0,
|
||||||
|
tokenIdHashExact: 0,
|
||||||
|
targetContractExact: 0,
|
||||||
|
};
|
||||||
|
for (const source of family.corpus_contract.selected) {
|
||||||
|
const previousSource = sourceById.get(source.id);
|
||||||
|
if (!previousSource) throw new Error(`boundary source missing: ${source.id}`);
|
||||||
|
for (const system of [0, 1]) {
|
||||||
|
for (const level of sharedLevels) {
|
||||||
|
const current = source.conditions[`s${system}_${level}`];
|
||||||
|
const previous = previousSource.conditions[`s${system}_${level}`];
|
||||||
|
tokenContract.compared += 1;
|
||||||
|
tokenContract.messageHashExact += (
|
||||||
|
current.messages_sha256 === previous.messages_sha256
|
||||||
|
);
|
||||||
|
tokenContract.renderedHashExact += (
|
||||||
|
current.rendered_sha256 === previous.rendered_sha256
|
||||||
|
);
|
||||||
|
tokenContract.tokenIdHashExact += (
|
||||||
|
current.token_ids_sha256 === previous.token_ids_sha256
|
||||||
|
);
|
||||||
|
tokenContract.targetContractExact += (
|
||||||
|
current.tokens === previous.tokens
|
||||||
|
&& current.content_tokens === previous.content_tokens
|
||||||
|
&& current.aligned_content_tokens
|
||||||
|
=== previous.aligned_content_tokens
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const crossBatch = family.layers.slice(1).flatMap((layer) => {
|
||||||
|
const previousLayer = boundary.layers.find(
|
||||||
|
(candidate) => candidate.layer === layer.layer,
|
||||||
|
);
|
||||||
|
const previousById = new Map(
|
||||||
|
previousLayer.prompts.map((prompt) => [prompt.id, prompt]),
|
||||||
|
);
|
||||||
|
return sharedLevels.map((level) => {
|
||||||
|
const counts = {
|
||||||
|
compared: 0,
|
||||||
|
fullRouteHashExact: 0,
|
||||||
|
targetRouteHashExact: 0,
|
||||||
|
fullLoadExact: 0,
|
||||||
|
targetLoadExact: 0,
|
||||||
|
};
|
||||||
|
for (const prompt of layer.prompts) {
|
||||||
|
const previousPrompt = previousById.get(prompt.id);
|
||||||
|
if (!previousPrompt) {
|
||||||
|
throw new Error(`boundary prompt missing: ${prompt.id}`);
|
||||||
|
}
|
||||||
|
for (const system of [0, 1]) {
|
||||||
|
const condition = `s${system}_${level}`;
|
||||||
|
const current = prompt.conditions[condition];
|
||||||
|
const previous = previousPrompt.conditions[condition];
|
||||||
|
counts.compared += 1;
|
||||||
|
counts.fullRouteHashExact += (
|
||||||
|
current.topk_sha256 === previous.topk_sha256
|
||||||
|
);
|
||||||
|
counts.targetRouteHashExact += (
|
||||||
|
current.content_topk_sha256 === previous.content_topk_sha256
|
||||||
|
);
|
||||||
|
counts.fullLoadExact += (
|
||||||
|
JSON.stringify(current.full_load)
|
||||||
|
=== JSON.stringify(previous.full_load)
|
||||||
|
);
|
||||||
|
counts.targetLoadExact += (
|
||||||
|
JSON.stringify(current.content_load)
|
||||||
|
=== JSON.stringify(previous.content_load)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { layer: layer.layer, level, ...counts };
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
const compact = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
source: {
|
||||||
|
mainSha256,
|
||||||
|
reproSha256,
|
||||||
|
mainBytes,
|
||||||
|
reproBytes,
|
||||||
|
exact,
|
||||||
|
},
|
||||||
|
domains: family.corpus_contract.domains,
|
||||||
|
labels: family.corpus_contract.domain_labels,
|
||||||
|
inference: family.inference_contract,
|
||||||
|
contract: {
|
||||||
|
tokenIds: family.special_token_family_contract.boundary_token_ids,
|
||||||
|
inventory: (
|
||||||
|
family.special_token_family_contract.tokenizer_special_inventory
|
||||||
|
),
|
||||||
|
validation: family.special_token_family_contract.render_validation,
|
||||||
|
official: family.boundary.official_serialization_by_boundary,
|
||||||
|
classBoundary: (
|
||||||
|
family.special_token_family_contract.class_comparison_boundary
|
||||||
|
),
|
||||||
|
tokenContractAgainstBoundaryRun: tokenContract,
|
||||||
|
},
|
||||||
|
crossBatch,
|
||||||
|
layers: family.layers.slice(1).map((layer) => ({
|
||||||
|
layer: layer.layer,
|
||||||
|
alignment: Object.fromEntries(
|
||||||
|
family.corpus_contract.domains.map((domain) => [
|
||||||
|
domain,
|
||||||
|
Object.fromEntries(
|
||||||
|
familyEdges.map((edge) => [
|
||||||
|
edge,
|
||||||
|
aggregateAlignment(layer, domain, edge),
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
scopes: Object.fromEntries(
|
||||||
|
["target_content", "full_input"].map((scope) => [
|
||||||
|
scope,
|
||||||
|
{
|
||||||
|
modes: Object.fromEntries(
|
||||||
|
["prompt_balanced", "token_weighted"].map((mode) => {
|
||||||
|
const control = (
|
||||||
|
layer.statistics[scope].modes[mode].boundary_control
|
||||||
|
);
|
||||||
|
return [
|
||||||
|
mode,
|
||||||
|
Object.fromEntries(
|
||||||
|
family.corpus_contract.domains.map((domain) => [
|
||||||
|
domain,
|
||||||
|
{
|
||||||
|
distances: control[domain].system_edge_distances,
|
||||||
|
contrasts: (
|
||||||
|
control[domain].system_edge_distance_contrasts
|
||||||
|
),
|
||||||
|
cvEdges: control[domain].metric_system_edges.cv,
|
||||||
|
cvContrasts: (
|
||||||
|
control[domain].metric_system_edge_contrasts.cv
|
||||||
|
),
|
||||||
|
direct: control[domain].direct_substitutions,
|
||||||
|
family: control[domain].descriptive_family_summary,
|
||||||
|
},
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
];
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
},
|
||||||
|
]),
|
||||||
|
),
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
|
||||||
|
writeFileSync(
|
||||||
|
outputPath,
|
||||||
|
`${JSON.stringify(compact, null, 2)}\n`,
|
||||||
|
"utf8",
|
||||||
|
);
|
||||||
|
process.stdout.write(
|
||||||
|
`${outputPath}\n${mainSha256}\n${mainBytes} bytes source → `
|
||||||
|
+ `${statSync(outputPath).size} bytes compact\n`,
|
||||||
|
);
|
||||||
@@ -608,8 +608,8 @@ const artifactRole = await evaluate(`(() => {
|
|||||||
depthTitle: root.querySelector("[data-role-depth-title]").textContent.trim(),
|
depthTitle: root.querySelector("[data-role-depth-title]").textContent.trim(),
|
||||||
exact: root.querySelector(".role-ledger .exact b").textContent.trim(),
|
exact: root.querySelector(".role-ledger .exact b").textContent.trim(),
|
||||||
causalCards: root.querySelectorAll(".role-causal-ledger > article").length,
|
causalCards: root.querySelectorAll(".role-causal-ledger > article").length,
|
||||||
batchCells: root.querySelectorAll(".role-batch-audit > div:last-child > span").length,
|
batchCells: root.querySelectorAll('[data-artifact-panel="role"] .role-batch-audit > div:last-child > span').length,
|
||||||
batchValues: [...root.querySelectorAll(".role-batch-audit > div:last-child > span i")].map((node) => node.textContent.trim()),
|
batchValues: [...root.querySelectorAll('[data-artifact-panel="role"] .role-batch-audit > div:last-child > span i')].map((node) => node.textContent.trim()),
|
||||||
note: root.querySelector("[data-role-note]").textContent.trim(),
|
note: root.querySelector("[data-role-note]").textContent.trim(),
|
||||||
targetHead: root.querySelector("[data-role-target-head]").textContent.trim(),
|
targetHead: root.querySelector("[data-role-target-head]").textContent.trim(),
|
||||||
suffixHead: root.querySelector("[data-role-suffix-head]").textContent.trim(),
|
suffixHead: root.querySelector("[data-role-suffix-head]").textContent.trim(),
|
||||||
@@ -646,6 +646,135 @@ await evaluate(`(() => {
|
|||||||
await pause(120);
|
await pause(120);
|
||||||
await screenshot("/tmp/llm-atlas-deepseek-role-results-desktop.png");
|
await screenshot("/tmp/llm-atlas-deepseek-role-results-desktop.png");
|
||||||
|
|
||||||
|
const artifactSpecial = await evaluate(`(() => {
|
||||||
|
const root = document.querySelector("[data-dsv2-lab]");
|
||||||
|
root.querySelector('[data-artifact-tab="special"]').click();
|
||||||
|
const read = () => ({
|
||||||
|
panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
|
||||||
|
tokenCards: root.querySelectorAll(".special-token-grid > article").length,
|
||||||
|
domainCards: root.querySelectorAll("[data-special-domain-grid] > article").length,
|
||||||
|
domains: [...root.querySelectorAll("[data-special-domain-grid] > article")].map((node) => ({
|
||||||
|
label: node.querySelector(":scope > span").textContent.trim(),
|
||||||
|
values: [...node.querySelectorAll(".special-tv-ladder > b")].map((cell) => ({
|
||||||
|
name: cell.querySelector("small").textContent.trim(),
|
||||||
|
value: cell.querySelector("strong").textContent.trim(),
|
||||||
|
})),
|
||||||
|
effect: node.querySelector(":scope > strong").textContent.trim(),
|
||||||
|
className: node.querySelector(":scope > strong").className,
|
||||||
|
ci: node.querySelector(":scope > p").textContent.trim(),
|
||||||
|
detail: node.querySelector(":scope > em").textContent.trim(),
|
||||||
|
stability: node.querySelector(":scope > small").textContent.trim(),
|
||||||
|
cv: node.querySelector(":scope > i").textContent.trim(),
|
||||||
|
})),
|
||||||
|
summary: [...root.querySelectorAll('[data-artifact-panel="special"] .special-summary article b')].map((node) => node.textContent.trim()),
|
||||||
|
depthRows: root.querySelectorAll("[data-special-depth-map] > div").length,
|
||||||
|
depthCells: root.querySelectorAll("[data-special-depth-map] > div > span").length,
|
||||||
|
depthTitle: root.querySelector("[data-special-depth-title]").textContent.trim(),
|
||||||
|
exact: root.querySelector(".special-ledger .exact b").textContent.trim(),
|
||||||
|
scopeCards: root.querySelectorAll('[data-artifact-panel="special"] .special-scope-ledger > article').length,
|
||||||
|
batchCells: root.querySelectorAll(".special-batch-audit > div:last-child > span").length,
|
||||||
|
batchValues: [...root.querySelectorAll(".special-batch-audit > div:last-child > span i")].map((node) => node.textContent.trim()),
|
||||||
|
note: root.querySelector("[data-special-note]").textContent.trim(),
|
||||||
|
activeLayer: root.querySelector("[data-special-layer].active").textContent.trim(),
|
||||||
|
activeScope: root.querySelector('[data-special-scope][aria-pressed="true"]').dataset.specialScope,
|
||||||
|
activeMode: root.querySelector('[data-special-mode][aria-pressed="true"]').dataset.specialMode,
|
||||||
|
activeContrast: root.querySelector('[data-special-contrast][aria-pressed="true"]').dataset.specialContrast,
|
||||||
|
});
|
||||||
|
const layer1Bos = read();
|
||||||
|
root.querySelector('[data-special-layer="4"]').click();
|
||||||
|
root.querySelector('[data-special-contrast="x_minus_eos"]').click();
|
||||||
|
const layer4X = read();
|
||||||
|
root.querySelector('[data-special-contrast="period_minus_eos"]').click();
|
||||||
|
const layer4Period = read();
|
||||||
|
root.querySelector('[data-special-contrast="family"]').click();
|
||||||
|
const layer4Family = read();
|
||||||
|
root.querySelector('[data-special-scope="full_input"]').click();
|
||||||
|
const layer4FullFamily = read();
|
||||||
|
root.querySelector('[data-special-mode="token_weighted"]').click();
|
||||||
|
const layer4FullToken = read();
|
||||||
|
root.querySelector('[data-special-layer="1"]').click();
|
||||||
|
root.querySelector('[data-special-scope="target_content"]').click();
|
||||||
|
root.querySelector('[data-special-mode="prompt_balanced"]').click();
|
||||||
|
root.querySelector('[data-special-contrast="bos_minus_eos"]').click();
|
||||||
|
return { layer1Bos, layer4X, layer4Period, layer4Family, layer4FullFamily, layer4FullToken, restored: read() };
|
||||||
|
})()`);
|
||||||
|
await evaluate(`(() => {
|
||||||
|
document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
|
window.scrollBy(0, -82);
|
||||||
|
})()`);
|
||||||
|
await pause(180);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-special-desktop.png");
|
||||||
|
await evaluate(`(() => {
|
||||||
|
document.querySelector(".special-domain-grid").scrollIntoView({ block: "center", behavior: "instant" });
|
||||||
|
})()`);
|
||||||
|
await pause(120);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-special-results-desktop.png");
|
||||||
|
|
||||||
|
const artifactRoleBlock = await evaluate(`(() => {
|
||||||
|
const root = document.querySelector("[data-dsv2-lab]");
|
||||||
|
root.querySelector('[data-artifact-tab="role-block"]').click();
|
||||||
|
const read = () => ({
|
||||||
|
panel: root.querySelector("[data-artifact-panel]:not([hidden])").dataset.artifactPanel,
|
||||||
|
matrixCards: root.querySelectorAll(".role-block-matrix > article").length,
|
||||||
|
domainCards: root.querySelectorAll("[data-role-block-domain-grid] > article").length,
|
||||||
|
domains: [...root.querySelectorAll("[data-role-block-domain-grid] > article")].map((node) => ({
|
||||||
|
label: node.querySelector(":scope > span").textContent.trim(),
|
||||||
|
values: [...node.querySelectorAll(".role-block-tv-ladder > b")].map((cell) => ({
|
||||||
|
name: cell.querySelector("small").textContent.trim(),
|
||||||
|
value: cell.querySelector("strong").textContent.trim(),
|
||||||
|
})),
|
||||||
|
effect: node.querySelector(":scope > strong").textContent.trim(),
|
||||||
|
className: node.querySelector(":scope > strong").className,
|
||||||
|
ci: node.querySelector(":scope > p").textContent.trim(),
|
||||||
|
direct: node.querySelector(":scope > em").textContent.trim(),
|
||||||
|
dependency: node.querySelector(":scope > u").textContent.trim(),
|
||||||
|
stability: node.querySelector(":scope > small").textContent.trim(),
|
||||||
|
cv: node.querySelector(":scope > i").textContent.trim(),
|
||||||
|
})),
|
||||||
|
summary: [...root.querySelectorAll('[data-artifact-panel="role-block"] .role-block-summary article b')].map((node) => node.textContent.trim()),
|
||||||
|
directCards: root.querySelectorAll(".role-block-direct > article").length,
|
||||||
|
depthRows: root.querySelectorAll("[data-role-block-depth-map] > div").length,
|
||||||
|
depthCells: root.querySelectorAll("[data-role-block-depth-map] > div > span").length,
|
||||||
|
depthTitle: root.querySelector("[data-role-block-depth-title]").textContent.trim(),
|
||||||
|
exact: root.querySelector(".role-block-ledger .exact b").textContent.trim(),
|
||||||
|
boundaryCards: root.querySelectorAll(".role-block-boundaries > article").length,
|
||||||
|
batchCells: root.querySelectorAll(".role-block-batch-audit > div:last-child > span").length,
|
||||||
|
batchValues: [...root.querySelectorAll(".role-block-batch-audit > div:last-child > span i")].map((node) => node.textContent.trim()),
|
||||||
|
note: root.querySelector("[data-role-block-note]").textContent.trim(),
|
||||||
|
activeLayer: root.querySelector("[data-role-block-layer].active").textContent.trim(),
|
||||||
|
activeScope: root.querySelector('[data-role-block-scope][aria-pressed="true"]').dataset.roleBlockScope,
|
||||||
|
activeMode: root.querySelector('[data-role-block-mode][aria-pressed="true"]').dataset.roleBlockMode,
|
||||||
|
activeEffect: root.querySelector('[data-role-block-effect][aria-pressed="true"]').dataset.roleBlockEffect,
|
||||||
|
});
|
||||||
|
const layer1Head = read();
|
||||||
|
root.querySelector('[data-role-block-layer="4"]').click();
|
||||||
|
root.querySelector('[data-role-block-effect="delimiter_main"]').click();
|
||||||
|
const layer4Delimiter = read();
|
||||||
|
root.querySelector('[data-role-block-layer="5"]').click();
|
||||||
|
root.querySelector('[data-role-block-effect="head_by_delimiter"]').click();
|
||||||
|
const layer5Interaction = read();
|
||||||
|
root.querySelector('[data-role-block-scope="full_input"]').click();
|
||||||
|
const layer5FullInteraction = read();
|
||||||
|
root.querySelector('[data-role-block-mode="token_weighted"]').click();
|
||||||
|
const layer5FullToken = read();
|
||||||
|
root.querySelector('[data-role-block-layer="1"]').click();
|
||||||
|
root.querySelector('[data-role-block-scope="target_content"]').click();
|
||||||
|
root.querySelector('[data-role-block-mode="prompt_balanced"]').click();
|
||||||
|
root.querySelector('[data-role-block-effect="head_main"]').click();
|
||||||
|
return { layer1Head, layer4Delimiter, layer5Interaction, layer5FullInteraction, layer5FullToken, restored: read() };
|
||||||
|
})()`);
|
||||||
|
await evaluate(`(() => {
|
||||||
|
document.querySelector("[data-dsv2-lab]").scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
|
window.scrollBy(0, -82);
|
||||||
|
})()`);
|
||||||
|
await pause(180);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-role-block-desktop.png");
|
||||||
|
await evaluate(`(() => {
|
||||||
|
document.querySelector(".role-block-domain-grid").scrollIntoView({ block: "center", behavior: "instant" });
|
||||||
|
})()`);
|
||||||
|
await pause(120);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-role-block-results-desktop.png");
|
||||||
|
|
||||||
const artifactEvidence = await evaluate(`(() => {
|
const artifactEvidence = await evaluate(`(() => {
|
||||||
const root = document.querySelector("[data-dsv2-lab]");
|
const root = document.querySelector("[data-dsv2-lab]");
|
||||||
root.querySelector('[data-artifact-tab="evidence"]').click();
|
root.querySelector('[data-artifact-tab="evidence"]').click();
|
||||||
@@ -750,6 +879,20 @@ const mobile = await evaluate(`(() => {
|
|||||||
roleLevelCards: artifact.querySelectorAll(".role-level-grid > article").length,
|
roleLevelCards: artifact.querySelectorAll(".role-level-grid > article").length,
|
||||||
roleDomainCards: artifact.querySelectorAll("[data-role-domain-grid] > article").length,
|
roleDomainCards: artifact.querySelectorAll("[data-role-domain-grid] > article").length,
|
||||||
roleDepthCells: artifact.querySelectorAll("[data-role-depth-map] > div > span").length,
|
roleDepthCells: artifact.querySelectorAll("[data-role-depth-map] > div > span").length,
|
||||||
|
specialLayers: artifact.querySelectorAll("[data-special-layer]").length,
|
||||||
|
specialScopes: artifact.querySelectorAll("[data-special-scope]").length,
|
||||||
|
specialModes: artifact.querySelectorAll("[data-special-mode]").length,
|
||||||
|
specialContrasts: artifact.querySelectorAll("[data-special-contrast]").length,
|
||||||
|
specialTokenCards: artifact.querySelectorAll(".special-token-grid > article").length,
|
||||||
|
specialDomainCards: artifact.querySelectorAll("[data-special-domain-grid] > article").length,
|
||||||
|
specialDepthCells: artifact.querySelectorAll("[data-special-depth-map] > div > span").length,
|
||||||
|
roleBlockLayers: artifact.querySelectorAll("[data-role-block-layer]").length,
|
||||||
|
roleBlockScopes: artifact.querySelectorAll("[data-role-block-scope]").length,
|
||||||
|
roleBlockModes: artifact.querySelectorAll("[data-role-block-mode]").length,
|
||||||
|
roleBlockEffects: artifact.querySelectorAll("[data-role-block-effect]").length,
|
||||||
|
roleBlockMatrixCards: artifact.querySelectorAll(".role-block-matrix > article").length,
|
||||||
|
roleBlockDomainCards: artifact.querySelectorAll("[data-role-block-domain-grid] > article").length,
|
||||||
|
roleBlockDepthCells: artifact.querySelectorAll("[data-role-block-depth-map] > div > span").length,
|
||||||
offenders: [...document.querySelectorAll("body *")]
|
offenders: [...document.querySelectorAll("body *")]
|
||||||
.filter((node) => !node.closest(".paper-chain, .advantage-table, .precision-table, .mapping-table, [data-deepseek-lab], [data-dsv2-lab]"))
|
.filter((node) => !node.closest(".paper-chain, .advantage-table, .precision-table, .mapping-table, [data-deepseek-lab], [data-dsv2-lab]"))
|
||||||
.filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1)
|
.filter((node) => node.getBoundingClientRect().right > document.documentElement.clientWidth + 1)
|
||||||
@@ -847,8 +990,36 @@ await evaluate(`(() => {
|
|||||||
})()`);
|
})()`);
|
||||||
await pause(120);
|
await pause(120);
|
||||||
await screenshot("/tmp/llm-atlas-deepseek-role-results-mobile.png");
|
await screenshot("/tmp/llm-atlas-deepseek-role-results-mobile.png");
|
||||||
|
await evaluate(`(() => {
|
||||||
|
const artifact = document.querySelector("[data-dsv2-lab]");
|
||||||
|
artifact.querySelector('[data-artifact-tab="special"]').click();
|
||||||
|
artifact.scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
|
window.scrollBy(0, -70);
|
||||||
|
})()`);
|
||||||
|
await pause(180);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-special-mobile.png");
|
||||||
|
await evaluate(`(() => {
|
||||||
|
document.querySelector(".special-domain-grid").scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
|
window.scrollBy(0, -72);
|
||||||
|
})()`);
|
||||||
|
await pause(120);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-special-results-mobile.png");
|
||||||
|
await evaluate(`(() => {
|
||||||
|
const artifact = document.querySelector("[data-dsv2-lab]");
|
||||||
|
artifact.querySelector('[data-artifact-tab="role-block"]').click();
|
||||||
|
artifact.scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
|
window.scrollBy(0, -70);
|
||||||
|
})()`);
|
||||||
|
await pause(180);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-role-block-mobile.png");
|
||||||
|
await evaluate(`(() => {
|
||||||
|
document.querySelector(".role-block-domain-grid").scrollIntoView({ block: "start", behavior: "instant" });
|
||||||
|
window.scrollBy(0, -72);
|
||||||
|
})()`);
|
||||||
|
await pause(120);
|
||||||
|
await screenshot("/tmp/llm-atlas-deepseek-role-block-results-mobile.png");
|
||||||
|
|
||||||
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactCorpus, artifactTemplate, artifactHistory, artifactDistance, artifactBoundary, artifactRole, artifactEvidence, home, papers, mobile, exceptions };
|
const report = { overview, capacity, cache, codesign, rl, artifactRoute, artifactLoad, artifactCache, artifactAbsorb, artifactCorpus, artifactTemplate, artifactHistory, artifactDistance, artifactBoundary, artifactRole, artifactSpecial, artifactRoleBlock, artifactEvidence, home, papers, mobile, exceptions };
|
||||||
console.log(JSON.stringify(report, null, 2));
|
console.log(JSON.stringify(report, null, 2));
|
||||||
|
|
||||||
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
const numeric = (text) => Number.parseFloat(text.replaceAll(",", ""));
|
||||||
@@ -858,8 +1029,8 @@ if (overview.sections !== 26 || overview.tocLinks !== 26) failures.push("二十
|
|||||||
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
|
if (overview.ledgers !== 24 || overview.waves !== 10) failures.push("二十四张问题账或十次转向结构异常");
|
||||||
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
|
if (overview.paperLinks !== 60 || overview.branches !== 5 || overview.followups !== 1) failures.push("论文链、旁支或公开后续标记异常");
|
||||||
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
if (overview.labTabs !== 4 || overview.labPanels !== 4) failures.push("四联实验结构异常");
|
||||||
if (overview.artifactTabs !== 11 || overview.artifactPanels !== 11 || overview.artifactLayers !== 27) failures.push("真实权重十一联实验结构异常");
|
if (overview.artifactTabs !== 13 || overview.artifactPanels !== 13 || overview.artifactLayers !== 27) failures.push("真实权重十三联实验结构异常");
|
||||||
if (overview.heroLabs !== "15 个可操作实验") failures.push("DeepSeek 实验总数账异常");
|
if (overview.heroLabs !== "17 个可操作实验") failures.push("DeepSeek 实验总数账异常");
|
||||||
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
|
if (overview.navLinks !== 20 || home.navLinks !== 20 || mobile.mobileLinks !== 20 || overview.activeNav !== "DeepSeek") failures.push("全站导航未同步 DeepSeek");
|
||||||
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
if (overview.documentOverflow > 1 || mobile.documentOverflow > 1) failures.push("桌面或移动端存在文档级横向溢出");
|
||||||
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
|
if (capacity.initial.panel !== "capacity" || capacity.initial.total !== "32.1× FFN" || capacity.initial.active !== "1.13× FFN") failures.push("V3 稀疏容量初始账异常");
|
||||||
@@ -929,12 +1100,26 @@ if (artifactRole.layer5Suffix.domains.some((domain) => domain.effect !== "SUFFIX
|
|||||||
if (artifactRole.layer5FullSuffix.activeScope !== "full_input" || !artifactRole.layer5FullSuffix.note.includes("suffix 自身") || artifactRole.layer5FullSuffix.domains[0].values[0].value === artifactRole.layer5Suffix.domains[0].values[0].value) failures.push("角色词头完整输入 scope 异常");
|
if (artifactRole.layer5FullSuffix.activeScope !== "full_input" || !artifactRole.layer5FullSuffix.note.includes("suffix 自身") || artifactRole.layer5FullSuffix.domains[0].values[0].value === artifactRole.layer5Suffix.domains[0].values[0].value) failures.push("角色词头完整输入 scope 异常");
|
||||||
if (artifactRole.layer5FullToken.activeMode !== "token_weighted" || artifactRole.restored.activeLayer !== "L1" || artifactRole.restored.activeScope !== "target_content" || artifactRole.restored.activeMode !== "prompt_balanced" || artifactRole.restored.activeContrast !== "target_assistant_minus_official" || artifactRole.restored.targetHead !== "Assistant" || artifactRole.restored.suffixHead !== "Assistant") failures.push("角色词头聚合口径或恢复状态异常");
|
if (artifactRole.layer5FullToken.activeMode !== "token_weighted" || artifactRole.restored.activeLayer !== "L1" || artifactRole.restored.activeScope !== "target_content" || artifactRole.restored.activeMode !== "prompt_balanced" || artifactRole.restored.activeContrast !== "target_assistant_minus_official" || artifactRole.restored.targetHead !== "Assistant" || artifactRole.restored.suffixHead !== "Assistant") failures.push("角色词头聚合口径或恢复状态异常");
|
||||||
if (artifactRole.layer1Assistant.batchValues.join("|") !== "256 / 256|180 / 256|149 / 256|102 / 256|72 / 256|73 / 256") failures.push("角色词头 BF16 batch-content 审计异常");
|
if (artifactRole.layer1Assistant.batchValues.join("|") !== "256 / 256|180 / 256|149 / 256|102 / 256|72 / 256|73 / 256") failures.push("角色词头 BF16 batch-content 审计异常");
|
||||||
|
if (artifactSpecial.layer1Bos.panel !== "special" || artifactSpecial.layer1Bos.tokenCards !== 4 || artifactSpecial.layer1Bos.domainCards !== 4 || artifactSpecial.layer1Bos.depthRows !== 4 || artifactSpecial.layer1Bos.depthCells !== 24 || artifactSpecial.layer1Bos.exact !== "BYTE-EXACT" || artifactSpecial.layer1Bos.scopeCards !== 3 || artifactSpecial.layer1Bos.batchCells !== 6) failures.push("特殊词元家族控制结构或独立复跑闸门异常");
|
||||||
|
if (artifactSpecial.layer1Bos.domains[0].values.map((cell) => cell.value).join("/") !== "0.056/0.108/0.084/0.081" || artifactSpecial.layer1Bos.domains[0].effect !== "BOS − EOS · ΔTV +0.052" || !artifactSpecial.layer1Bos.domains[0].ci.includes("+0.040, +0.061")) failures.push("L1 英文特殊词元家族统计异常");
|
||||||
|
if (!artifactSpecial.layer1Bos.domains[0].detail.includes("S0 0.087 · S1 0.064") || !artifactSpecial.layer1Bos.domains[0].stability.includes("S0 37.4%") || artifactSpecial.layer1Bos.summary.join("|") !== ".037518|+.010500|24↑ / 24↑|+.011864") failures.push("特殊词元直接效应、对齐率或总账异常");
|
||||||
|
if (artifactSpecial.layer4X.activeContrast !== "x_minus_eos" || !artifactSpecial.layer4X.depthTitle.includes("普通内容 token") || artifactSpecial.layer4Period.activeContrast !== "period_minus_eos" || !artifactSpecial.layer4Period.depthTitle.includes("句点")) failures.push("特殊词元单 ID contrast 切换异常");
|
||||||
|
if (artifactSpecial.layer4Family.activeContrast !== "family" || !artifactSpecial.layer4Family.note.includes("四个 ID") || !artifactSpecial.layer4Family.domains.every((domain) => domain.stability.includes("2-vs-2"))) failures.push("特殊词元描述性 family 汇总异常");
|
||||||
|
if (artifactSpecial.layer4FullFamily.activeScope !== "full_input" || !artifactSpecial.layer4FullFamily.note.includes("完整输入") || artifactSpecial.layer4FullToken.activeMode !== "token_weighted") failures.push("特殊词元完整输入或聚合口径切换异常");
|
||||||
|
if (artifactSpecial.restored.activeLayer !== "L1" || artifactSpecial.restored.activeScope !== "target_content" || artifactSpecial.restored.activeMode !== "prompt_balanced" || artifactSpecial.restored.activeContrast !== "bos_minus_eos") failures.push("特殊词元控制恢复状态异常");
|
||||||
|
if (artifactSpecial.layer1Bos.batchValues.join("|") !== "768 / 768|610 / 768|477 / 768|368 / 768|269 / 768|244 / 768") failures.push("特殊词元 BF16 batch-content 审计异常");
|
||||||
|
if (artifactRoleBlock.layer1Head.panel !== "role-block" || artifactRoleBlock.layer1Head.matrixCards !== 4 || artifactRoleBlock.layer1Head.domainCards !== 4 || artifactRoleBlock.layer1Head.depthRows !== 4 || artifactRoleBlock.layer1Head.depthCells !== 24 || artifactRoleBlock.layer1Head.exact !== "BYTE-EXACT" || artifactRoleBlock.layer1Head.directCards !== 4 || artifactRoleBlock.layer1Head.boundaryCards !== 3 || artifactRoleBlock.layer1Head.batchCells !== 6) failures.push("完整角色块 2×2 结构或独立复跑闸门异常");
|
||||||
|
if (artifactRoleBlock.layer1Head.domains[0].values.map((cell) => cell.value).join("/") !== "0.056/0.059/0.056/0.058" || artifactRoleBlock.layer1Head.domains[0].effect !== "HEAD MAIN · ΔTV +0.002" || !artifactRoleBlock.layer1Head.domains[0].ci.includes("-0.002, +0.004")) failures.push("L1 英文完整角色块 head 统计异常");
|
||||||
|
if (!artifactRoleBlock.layer1Head.domains[0].direct.includes("head @ colon") || !artifactRoleBlock.layer1Head.domains[0].dependency.includes("head @ x") || artifactRoleBlock.layer1Head.summary.join("|") !== ".037304 → .036220|−.000654|−.000430|−.000729") failures.push("完整角色块直接边或总账异常");
|
||||||
|
if (artifactRoleBlock.layer4Delimiter.activeEffect !== "delimiter_main" || !artifactRoleBlock.layer4Delimiter.depthTitle.includes("冒号→x") || artifactRoleBlock.layer5Interaction.activeEffect !== "head_by_delimiter" || !artifactRoleBlock.layer5Interaction.depthTitle.includes("依赖 delimiter") || !artifactRoleBlock.layer5Interaction.domains[0].direct.includes("head@x − head@colon")) failures.push("完整角色块因子或 interaction 切换异常");
|
||||||
|
if (artifactRoleBlock.layer5FullInteraction.activeScope !== "full_input" || artifactRoleBlock.layer5FullToken.activeMode !== "token_weighted" || artifactRoleBlock.restored.activeLayer !== "L1" || artifactRoleBlock.restored.activeScope !== "target_content" || artifactRoleBlock.restored.activeMode !== "prompt_balanced" || artifactRoleBlock.restored.activeEffect !== "head_main") failures.push("完整角色块 scope、聚合口径或恢复状态异常");
|
||||||
|
if (artifactRoleBlock.layer1Head.batchValues.join("|") !== "512 / 512|368 / 512|279 / 512|230 / 512|178 / 512|156 / 512") failures.push("完整角色块 BF16 batch-content 审计异常");
|
||||||
if (artifactEvidence.panel !== "evidence" || artifactEvidence.layers !== 27 || artifactEvidence.executed !== 7 || artifactEvidence.split !== 1 || artifactEvidence.unloaded !== 19 || artifactEvidence.exact !== "31 / 31") failures.push("真实工件执行边界或复跑闸门异常");
|
if (artifactEvidence.panel !== "evidence" || artifactEvidence.layers !== 27 || artifactEvidence.executed !== 7 || artifactEvidence.split !== 1 || artifactEvidence.unloaded !== 19 || artifactEvidence.exact !== "31 / 31") failures.push("真实工件执行边界或复跑闸门异常");
|
||||||
if (!artifactEvidence.dependency.includes("Transformers 5.5") || !artifactEvidence.dependency.includes("4.41.2") || !artifactEvidence.boundary.includes("完整 27 层生成")) failures.push("依赖版本或未覆盖边界异常");
|
if (!artifactEvidence.dependency.includes("Transformers 5.5") || !artifactEvidence.dependency.includes("4.41.2") || !artifactEvidence.boundary.includes("完整 27 层生成")) failures.push("依赖版本或未覆盖边界异常");
|
||||||
if (artifactEvidence.keyboardSelected !== "load" || artifactEvidence.keyboardVisible !== "load") failures.push("真实工件实验键盘 tab 导航异常");
|
if (artifactEvidence.keyboardSelected !== "load" || artifactEvidence.keyboardVisible !== "load") failures.push("真实工件实验键盘 tab 导航异常");
|
||||||
if (home.releaseCards !== 17 || !home.firstRelease.includes("47 页不再压成摘要") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 DeepSeek 首发入口或论文数异常");
|
if (home.releaseCards !== 17 || !home.firstRelease.includes("47 页不再压成摘要") || home.firstHref !== "/k3/" || home.paperCount !== "486") failures.push("首页 DeepSeek 首发入口或论文数异常");
|
||||||
if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常");
|
if (papers.total !== 486 || !papers.hasFilter || papers.visible < 20 || !papers.hasCoder || !papers.hasEngram) failures.push("论文库 DeepSeek 聚光异常");
|
||||||
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 11 || mobile.artifactHeatCells !== 64 || mobile.corpusCohorts !== 3 || mobile.lengthDeltaCards !== 4 || mobile.templateLayers !== 6 || mobile.templateScopes !== 2 || mobile.templateModes !== 2 || mobile.templateDomainCards !== 4 || mobile.templateDepthCells !== 24 || mobile.historyLayers !== 6 || mobile.historyScopes !== 2 || mobile.historyModes !== 2 || mobile.historyEffects !== 3 || mobile.historyDomainCards !== 4 || mobile.historyDepthCells !== 24 || mobile.distanceLayers !== 6 || mobile.distanceScopes !== 2 || mobile.distanceModes !== 2 || mobile.distanceContrasts !== 2 || mobile.distanceDomainCards !== 4 || mobile.distanceDepthCells !== 24 || mobile.boundaryLayers !== 6 || mobile.boundaryScopes !== 2 || mobile.boundaryModes !== 2 || mobile.boundaryContrasts !== 3 || mobile.boundaryTokenCards !== 4 || mobile.boundaryDomainCards !== 4 || mobile.boundaryDepthCells !== 24 || mobile.roleLayers !== 6 || mobile.roleScopes !== 2 || mobile.roleModes !== 2 || mobile.roleContrasts !== 3 || mobile.roleLevelCards !== 4 || mobile.roleDomainCards !== 4 || mobile.roleDepthCells !== 24) failures.push("移动端导航或实验异常");
|
if (!mobile.menuVisible || mobile.menuOpen !== "true" || mobile.tabs !== 4 || mobile.artifactTabs !== 13 || mobile.artifactHeatCells !== 64 || mobile.corpusCohorts !== 3 || mobile.lengthDeltaCards !== 4 || mobile.templateLayers !== 6 || mobile.templateScopes !== 2 || mobile.templateModes !== 2 || mobile.templateDomainCards !== 4 || mobile.templateDepthCells !== 24 || mobile.historyLayers !== 6 || mobile.historyScopes !== 2 || mobile.historyModes !== 2 || mobile.historyEffects !== 3 || mobile.historyDomainCards !== 4 || mobile.historyDepthCells !== 24 || mobile.distanceLayers !== 6 || mobile.distanceScopes !== 2 || mobile.distanceModes !== 2 || mobile.distanceContrasts !== 2 || mobile.distanceDomainCards !== 4 || mobile.distanceDepthCells !== 24 || mobile.boundaryLayers !== 6 || mobile.boundaryScopes !== 2 || mobile.boundaryModes !== 2 || mobile.boundaryContrasts !== 3 || mobile.boundaryTokenCards !== 4 || mobile.boundaryDomainCards !== 4 || mobile.boundaryDepthCells !== 24 || mobile.roleLayers !== 6 || mobile.roleScopes !== 2 || mobile.roleModes !== 2 || mobile.roleContrasts !== 3 || mobile.roleLevelCards !== 4 || mobile.roleDomainCards !== 4 || mobile.roleDepthCells !== 24 || mobile.specialLayers !== 6 || mobile.specialScopes !== 2 || mobile.specialModes !== 2 || mobile.specialContrasts !== 4 || mobile.specialTokenCards !== 4 || mobile.specialDomainCards !== 4 || mobile.specialDepthCells !== 24 || mobile.roleBlockLayers !== 6 || mobile.roleBlockScopes !== 2 || mobile.roleBlockModes !== 2 || mobile.roleBlockEffects !== 3 || mobile.roleBlockMatrixCards !== 4 || mobile.roleBlockDomainCards !== 4 || mobile.roleBlockDepthCells !== 24) failures.push("移动端导航或实验异常");
|
||||||
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
if (mobile.offenders.length) failures.push(`移动端越界元素：${JSON.stringify(mobile.offenders)}`);
|
||||||
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
if (exceptions.length) failures.push(`浏览器异常：${exceptions.join(" | ")}`);
|
||||||
|
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -37,7 +37,7 @@ const toc = [
|
|||||||
|
|
||||||
<BaseLayout
|
<BaseLayout
|
||||||
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
|
title="DeepSeek 技术谱系与真实权重深读：从 Dense、MoE、MLA 到 R1 与 V4"
|
||||||
description="用二十四张问题账、十次技术转向、十五个交互实验、真实 V2-Lite 权重、公开语料路由区间、官方模板、消息历史、等长 filler、单 EOS-ID 与角色词头控制、吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
description="用二十四张问题账、十次技术转向、十七个交互实验、真实 V2-Lite 权重、公开语料路由区间、官方模板、消息历史、等长 filler、特殊词元家族与完整角色块控制、吸收式缓存 trace 和六十个一手节点，完整理解 DeepSeek 的 MoE、MLA、FP8、DualPipe、GRPO、R1、V3.2 与 V4。"
|
||||||
section="deepseek"
|
section="deepseek"
|
||||||
>
|
>
|
||||||
<header class="page-hero deepseek-hero">
|
<header class="page-hero deepseek-hero">
|
||||||
@@ -55,7 +55,7 @@ const toc = [
|
|||||||
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
<div><dt>SPAN</dt><dd>2024.01 → 2026.06</dd></div>
|
||||||
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
|
<div><dt>LEDGERS</dt><dd>24 张问题账</dd></div>
|
||||||
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
|
<div><dt>LINEAGE</dt><dd>10 次技术转向</dd></div>
|
||||||
<div><dt>LABS</dt><dd>15 个可操作实验</dd></div>
|
<div><dt>LABS</dt><dd>17 个可操作实验</dd></div>
|
||||||
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
|
<div><dt>EVIDENCE</dt><dd>60 个一手 / 官方节点</dd></div>
|
||||||
<div><dt>STATUS</dt><dd>三轮 · 真实权重执行</dd></div>
|
<div><dt>STATUS</dt><dd>三轮 · 真实权重执行</dd></div>
|
||||||
</dl>
|
</dl>
|
||||||
@@ -768,15 +768,15 @@ const toc = [
|
|||||||
<p class="eyebrow"><span>22</span> OFFICIAL WEIGHTS / EXECUTED</p>
|
<p class="eyebrow"><span>22</span> OFFICIAL WEIGHTS / EXECUTED</p>
|
||||||
<h2>从“MLA 与 MoE 的概念”再往前一步：让官方 V2-Lite 权重真的跑起来</h2>
|
<h2>从“MLA 与 MoE 的概念”再往前一步：让官方 V2-Lite 权重真的跑起来</h2>
|
||||||
<p class="lede">
|
<p class="lede">
|
||||||
前面的四联实验负责建立公式与角色合同；下面的十一联工件实验固定官方 revision、tokenizer、
|
前面的四联实验负责建立公式与角色合同；下面的十三联工件实验固定官方 revision、tokenizer、
|
||||||
模型代码和 checkpoint 第一分片，在 RTX 5090 上连续执行 layer 0–6。它把真实观测、shape 推导、
|
模型代码和 checkpoint 第一分片，在 RTX 5090 上连续执行 layer 0–6。它把真实观测、shape 推导、
|
||||||
吸收式 latent cache、长度对照、官方 chat-template 扰动、实现差距和未覆盖范围放在同一张证据图里。
|
吸收式 latent cache、长度对照、官方 chat-template 扰动、实现差距和未覆盖范围放在同一张证据图里。
|
||||||
</p>
|
</p>
|
||||||
<div class="artifact-callout">
|
<div class="artifact-callout">
|
||||||
<article><span>X / FORWARD</span><b>7 / 27 layers</b><p>1 个 dense 层 + 6 个 MoE 层；layer 7 因跨分片停止。</p></article>
|
<article><span>X / FORWARD</span><b>7 / 27 layers</b><p>1 个 dense 层 + 6 个 MoE 层；layer 7 因跨分片停止。</p></article>
|
||||||
<article><span>X / ROUTES</span><b>7,201,296</b><p>三档长度、模板、消息历史、等长 filler、单 EOS-ID 与角色词头八格控制的真实 top-6 选择。</p></article>
|
<article><span>X / ROUTES</span><b>11,289,744</b><p>三档长度、模板、消息历史、等长 filler、边界、特殊词元家族与完整角色块八格控制的真实 top-6 选择。</p></article>
|
||||||
<article><span>X / ABSORB CACHE</span><b>266,240 → 29,952 B</b><p>同一真实 layer-1 权重的 naive / absorb active buffers。</p></article>
|
<article><span>X / ABSORB CACHE</span><b>266,240 → 29,952 B</b><p>同一真实 layer-1 权重的 naive / absorb active buffers。</p></article>
|
||||||
<article><span>X / RERUN</span><b>8 / 8 EXACT</b><p>三档长度、官方模板、历史因子、等长 filler、边界与角色词头控制均 byte-exact；比较使用 paired prompt bootstrap。</p></article>
|
<article><span>X / RERUN</span><b>10 / 10 EXACT</b><p>三档长度、官方模板、历史因子、等长 filler、边界、角色词头、special family 与完整角色块均 byte-exact；比较使用 paired prompt bootstrap。</p></article>
|
||||||
</div>
|
</div>
|
||||||
<DeepSeekArtifactLab />
|
<DeepSeekArtifactLab />
|
||||||
</section>
|
</section>
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ const workstreams = [
|
|||||||
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
|
{ label: "表示、位置与残差高速公路", value: 81, next: "加入真实 hidden-state / norm traces、长上下文位置外推复现与更多深层稳定性消融" },
|
||||||
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
|
{ label: "Scaling Laws", value: 74, next: "加入真实拟合复现、置信区间与更多模型族对照" },
|
||||||
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
|
{ label: "数据工程与预训练配方", value: 73, next: "逐图精读 FineWeb / DCLM，加入真实去重与 mixture traces" },
|
||||||
{ label: "DeepSeek 专题", value: 97, next: "special-token family 与完整两-token 角色块、V2-Lite-Chat 行为、完整 27 层与固定 batch content，再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL" },
|
{ label: "DeepSeek 专题", value: 98, next: "V2-Lite-Chat 生成/行为对照、完整 27 层与固定 batch content，再推进 SM90 FlashMLA、FP8/pipeline 与 R1-like RL" },
|
||||||
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
{ label: "指令微调与人类偏好", value: 75, next: "加入真实偏好分歧样本、RM 长度偏置与 PPO/DPO 小模型复现" },
|
||||||
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
{ label: "推理与测试时扩展", value: 76, next: "真实模型采样曲线、PRM 案例与逐篇图表精读" },
|
||||||
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
{ label: "工具使用与长程 Agent", value: 74, next: "补真实环境 traces、cross-harness 对照、Agent RL 训练曲线与安全案例" },
|
||||||
@@ -97,12 +97,12 @@ const workstreams = [
|
|||||||
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
<article><span>✓</span><h3>K3 报告已结构化拆解</h3><p>47 页报告目录、151 条参考来源和架构/后训练/系统主线已经提取。</p></article>
|
||||||
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
<article><span>✓</span><h3>17 专题知识图</h3><p>从语言模型基础到评测安全，包含先修依赖和三条贯穿案例。</p></article>
|
||||||
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
<article><span>✓</span><h3>编辑式网站系统</h3><p>响应式导航、章节模板、侧栏、进度、论文链和证据提示组件。</p></article>
|
||||||
<article><span>✓</span><h3>八十二个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与十一联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
<article><span>✓</span><h3>八十四个原创交互视图</h3><p>K3 三轴图、八联报告实验与四联开放工件实验，DeepSeek 四联公式实验与十三联真实权重实验，以及语言模型前史、Transformer、表示深度、长上下文、MoE、推理、Agent、多模态、训练系统、推理服务、Scaling、数据工程、数值、Alignment 与评测安全专题。</p></article>
|
||||||
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
<article><span>✓</span><h3>十七篇首版长文</h3><p>K3、语言模型前史、Transformer、表示/位置/残差、DeepSeek、Scaling、数据工程、长上下文、MoE、后训练、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全专题。</p></article>
|
||||||
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
<article><span>✓</span><h3>语言模型前史深度专题</h3><p>八张独立问题账、33 个正式节点、20 段长文与概率—向量—记忆—对齐四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
<article><span>✓</span><h3>Transformer 深度专题</h3><p>十张独立问题账、40 个正式节点、21 段正文与 QKV—Mask—多头位置—Block 成本四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
|
<article><span>✓</span><h3>表示、位置与残差高速公路深度专题</h3><p>二十张问题账、66 个一手节点、DeepSeek/Kimi 双谱系，以及 Token—位置—Norm—Residual/FFN 四联实验。</p></article>
|
||||||
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、官方 V3 absorb、长度/模板、system × one-shot、等长 filler、EOS 与角色词头单 token 控制；累计 7,201,296 次真实路由。最新八格显示前置词头有约 .02–.03 direct TV，却没有统一 system 调制；后置改动在 34,488 个目标 ordered top-6 上 causal-exact，两份 53,440,884-byte JSON 的 SHA-256 同为 9dc0e37f…b8caf。</p></article>
|
<article><span>✓</span><h3>DeepSeek 三轮真实权重里程碑</h3><p>在二十四张问题账、十次转向与四联公式实验上，新增 V2-Lite 7/27 层连续 forward、官方 V3 absorb，以及长度、模板、消息历史、边界、角色词头、完整 special inventory 与两-token 角色块控制；累计 11,289,744 次真实路由。最新两组八格各自 byte-exact：BOS 不复现 EOS，四 ID 的 2-vs-2 只作描述；head、delimiter 与 interaction 均无统一 system 调制方向。</p></article>
|
||||||
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
|
<article><span>✓</span><h3>Kimi K3 技术报告二轮深读</h3><p>三十二张问题账、Figure 1–16 / Table 1–5 审计、100 节点阅读链，以及 Delta—Decay—AttnRes—LatentMoE—SiTU—QB—MOPD—Cache 八联实验。</p></article>
|
||||||
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
|
<article><span>✓</span><h3>Kimi K3 三轮开放工件里程碑</h3><p>固定官方 revisions，审计 96 个 shards、497,220 个 tensor entries 与真实 KDA / MLA / MoE / MoonViT shapes；四联实验分开显示层型、tensor anatomy、参数范围和复现边界。</p></article>
|
||||||
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
|
<article><span>✓</span><h3>FlashKDA RTX 5090 执行闸门</h3><p>隔离 CUDA 13.0 / glibc 2.39 编译 sm_120a wheel；6/6 官方参考逐元素相等，并完成 fixed / varlen、三种 state mode 的 1,800 个 CUDA Event samples。</p></article>
|
||||||
@@ -134,7 +134,7 @@ const workstreams = [
|
|||||||
<div class="queue-table">
|
<div class="queue-table">
|
||||||
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
<div class="head"><b>优先级</b><b>专题</b><b>本轮交付</b><b>完成闸门</b></div>
|
||||||
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
|
<div><span>P0</span><strong>K3 三轮</strong><p>开放权重 traces → FlashKDA / AttnRes / MoE 真实行为 → Figure 1–16 数值重绘与独立复现</p><em>运行证据 + 逐图复现</em></div>
|
||||||
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>special-token family / 完整两-token 角色块 / V2-Lite-Chat 行为 → 完整 27 层与固定 batch content → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
<div><span>P0</span><strong>DeepSeek 三轮</strong><p>V2-Lite-Chat 生成/行为对照 → 完整 27 层与固定 batch content → SM90 FlashMLA / FP8 / pipeline traces → R1-like RL 小模型复现</p><em>运行证据 + 独立复现</em></div>
|
||||||
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
|
<div><span>P0</span><strong>Transformer 二轮</strong><p>多头电路逐图 → Pre/Post-LN 真实 traces → Flash/KV 配置与 kernel 对照</p><em>逐图笔记 + 实测边界</em></div>
|
||||||
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
|
<div><span>P0</span><strong>表示、位置与残差二轮</strong><p>真实 hidden-state / norm traces → 长上下文位置外推 → mHC / AttnRes 深层稳定性消融</p><em>可复现实验 + 逐图笔记</em></div>
|
||||||
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
|
<div><span>P0</span><strong>语言模型前史二轮</strong><p>Kneser–Ney / LSTM / Bahdanau 逐图 → 真实小语料复现 → tokenizer 公平性</p><em>可复现实验 + 逐图笔记</em></div>
|
||||||
@@ -223,6 +223,9 @@ const workstreams = [
|
|||||||
<div><time>2026-07-29</time><b>角色词头 direct effect 与 system 调制分开</b><p>`User→Assistant/x` 都直接改变后续路由，但 24 格 system-edge 方向混合，不能偷换成统一强化或削弱。</p></div>
|
<div><time>2026-07-29</time><b>角色词头 direct effect 与 system 调制分开</b><p>`User→Assistant/x` 都直接改变后续路由，但 24 格 system-edge 方向混合，不能偷换成统一强化或削弱。</p></div>
|
||||||
<div><time>2026-07-29</time><b>目标后改动承担因果负对照</b><p>suffix `Assistant→User` 在 34,488 个目标 ordered top-6 上全 exact；完整输入仍保留 suffix 自身作用。</p></div>
|
<div><time>2026-07-29</time><b>目标后改动承担因果负对照</b><p>suffix `Assistant→User` 在 34,488 个目标 ordered top-6 上全 exact；完整输入仍保留 suffix 自身作用。</p></div>
|
||||||
<div><time>2026-07-29</time><b>BF16 batch content 进入复现合同</b><p>相同 official token IDs 与 32-row shape 不保证深层 route hash 相同；companion rows 的构成也必须记录。</p></div>
|
<div><time>2026-07-29</time><b>BF16 batch content 进入复现合同</b><p>相同 official token IDs 与 32-row shape 不保证深层 route hash 相同；companion rows 的构成也必须记录。</p></div>
|
||||||
|
<div><time>2026-07-29</time><b>完整 special inventory 与普通对照分身份</b><p>BOS/EOS 穷尽固定 tokenizer 的两个 special IDs；x/句点只是两个选定普通对照，2-vs-2 只描述四个 ID。</p></div>
|
||||||
|
<div><time>2026-07-29</time><b>两-token 角色块按因子分解</b><p>`User:` / `Assistant:` 都是 head + delimiter 两个普通 IDs；head、delimiter、interaction 与直接边分别记账。</p></div>
|
||||||
|
<div><time>2026-07-29</time><b>下游目标与完整输入分因果身份</b><p>目标内容只看编辑位置后的精确对齐路由；完整输入包含被编辑 token 自身，只作稳健性账。</p></div>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user