Compare commits
36 Commits
756e0e6b18
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 wuyang
|
f8f0712103 | ||
|
wuyang
|
f177fa676d | ||
|
wuyang
|
7ea91caabb | ||
|
wuyang
|
d81aacfc68 | ||
|
wuyang
|
badf498598 | ||
|
wuyang
|
dc8ec30f78 | ||
|
wuyang
|
2dbfd7944a | ||
|
wuyang
|
39a9ad6215 | ||
|
wuyang
|
6911efc6e7 | ||
|
wuyang
|
5e64990a7d | ||
|
wuyang
|
b39378006a | ||
|
wuyang
|
a1a52d4280 | ||
|
wuyang
|
3cbb163715 | ||
|
wuyang
|
e9ca86c21e | ||
|
wuyang
|
54c80fb115 | ||
|
wuyang
|
204f599304 | ||
|
wuyang
|
97ec50a642 | ||
|
wuyang
|
f7670efcdd | ||
|
wuyang
|
26fc824409 | ||
|
wuyang
|
3f7cc1f544 | ||
|
wuyang
|
1675ca54f3 | ||
|
wuyang
|
091a05f0a0 | ||
|
wuyang
|
40f6a1d631 | ||
|
wuyang
|
4ce780dcc9 | ||
|
wuyang
|
d1d9d22bf3 | ||
|
wuyang
|
5f49906be6 | ||
|
wuyang
|
1f20f81939 | ||
|
wuyang
|
f99872039f | ||
|
wuyang
|
e3617532de | ||
|
wuyang
|
9039de1b02 | ||
|
wuyang
|
592599fd59 | ||
|
wuyang
|
975ed3dce2 | ||
|
wuyang
|
f041c15e81 | ||
|
wuyang
|
65a91ae178 | ||
|
wuyang
|
a09e6b14ce | ||
|
wuyang
|
6181af90c8 |
+82
-5
@@ -8,13 +8,13 @@
|
||||
|---|---:|---:|---|
|
||||
| 研究框架与规范 | 进行中 | 83% | Scaling Laws 二轮拟合复现与逐图精读 |
|
||||
| 网站设计系统 | 进行中 | 89% | 打印样式与更多通用可视化组件 |
|
||||
| Kimi K3 深读 | 三轮实证进行中 | 94% | 接入真实 hidden-state / expert-load / cache traces，并重绘报告数值图 |
|
||||
| Kimi K3 深读 | 八轮实证已收敛 | 100% | 稳定维护；真实 forward 等待 `A_log` 官方裁决 |
|
||||
| 语言模型前史 | 完成首版 | 78% | Kneser–Ney、LSTM、Bahdanau 逐图精读与真实小语料复现 |
|
||||
| Transformer 基础 | 完成首版 | 79% | 多头电路、归一化 traces 与真实 kernel / KV 配置 |
|
||||
| 表示、位置与残差高速公路 | 完成首版 | 81% | 真实 hidden-state / norm traces、长上下文位置外推与深层稳定性消融 |
|
||||
| Scaling Laws | 完成首版 | 74% | 真实拟合复现、置信区间与更多模型族对照 |
|
||||
| 数据工程与预训练配方 | 完成首版 | 73% | FineWeb / DCLM 逐图精读、真实去重误伤与 mixture traces |
|
||||
| DeepSeek 专题 | 七轮实证进行中 | 99% | 扩大到可做 task-level bootstrap，加入 per-row RNG，再推进干预式 mediation、SM90 FlashMLA、FP8/pipeline 与 R1-like RL |
|
||||
| DeepSeek 专题 | 八轮实证进行中 | 99% | 推进干预式 mediation、SM90 FlashMLA、FP8/pipeline 与 R1-like RL |
|
||||
| 指令微调与人类偏好 | 完成首版 | 75% | 真实偏好分歧、RM 长度偏置与 PPO/DPO 小模型复现 |
|
||||
| 推理与测试时扩展 | 完成首版 | 76% | 真实模型采样曲线、PRM 案例与逐篇图表精读 |
|
||||
| 工具使用与长程 Agent | 完成首版 | 74% | 真实环境 traces、cross-harness 对照、Agent RL 曲线与安全案例 |
|
||||
@@ -41,7 +41,7 @@
|
||||
- [x] 完成 486 篇关键论文索引，覆盖 16 个标签专题与 Kimi/DeepSeek 聚光主线。
|
||||
- [x] 完成可检索、可按专题筛选的论文库页面。
|
||||
- [x] 完成 K3、语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、Agent、原生多模态、训练系统、推理服务、数值优化与评测安全十七篇首版长文。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验与四联开放工件实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 二十一联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等八十八个原创交互视图。
|
||||
- [x] 完成 K3 三轴架构、八联报告实验、四联开放工件实验、Round 04 / 05 / 06 / 07 / 08 各五联 AttnRes 独立实验、语言模型前史四联实验、Transformer 四联实验、表示深度四联实验、DeepSeek 二十二联实验、长上下文、MoE 路由、推理三页签，以及训练系统、推理服务、Scaling、数据工程、数值、Alignment、Agent、原生多模态与评测安全专题各四页签等一百一十四个原创交互视图。
|
||||
- [x] 完成长上下文首版：五张成本账、26 篇一手论文、10+ 机制图与 8 策略交互实验室。
|
||||
- [x] 核验 FlashAttention、DeepSeek-V2/V3.2/V4、Kimi Linear/K3 等六份论文原文，并建立长上下文研究账本。
|
||||
- [x] 核验 Switch、ST-MoE、DeepSeekMoE、Loss-Free、V3、LatentMoE 与 K3 原文，并建立 MoE 研究账本。
|
||||
@@ -264,11 +264,58 @@
|
||||
- [x] 第二十一个 DeepSeek 交互实验以四页签讲解 source→seed 层级、Math/Code 4×4 task matrix、source-level 方向与均值反例、64 格八字段复现及五段 hash chain；协议、完整审计、探针、独立 evaluator、analysis 与 compact builder 全部落盘。
|
||||
- [x] Round 07 本地闸门通过：83 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；Python 探针/评测/分析编译通过，compact builder 重建 SHA-256 不变。跨来源专项与 DeepSeek 全量真实 Chrome 回归通过，4×4 矩阵、English 反例、键盘页签、桌面/390px 移动端无溢出及零运行时异常均有自动断言。
|
||||
- [x] DeepSeek Round 07 以源提交 `9211333`、不可变镜像 `20260729T195952Z-9211333` 发布；OCI index digest `sha256:6c92883b…5584e2`，复用 NAS `12010→8080`、NPM host 31 / cert 41 与门户 `LLM ATLAS / projects / 180`。容器 healthy、0 次重启，NAS / VPS Tailscale / HTTPS / gzip / immutable assets、跨来源专项与 DeepSeek 全量生产 Chrome 回归通过；保留 `20260729T180228Z-580f696` 回滚。
|
||||
- [x] DeepSeek Round 08 在任何正式输出前冻结 HumanEval / GSM8K 各 32 条任务、四条件、T0 主分析与 T1–T3 诊断，并把共同随机数实现为 SHA-256 派生的显式 uniform tape；四个条件对同一 `u_t` 各自计算 float32 CDF，不再把 generator seed 重置冒充 CRN。
|
||||
- [x] 64 条短 smoke 的 prompt 64/64 exact、同进程重放 32/32 exact、CPU/CUDA RNG 16/16 不变；正式执行 64 tasks × T0 × 4 conditions 与 8 tasks × T1–T3 × 4 conditions，共 352 条输出，343 条 natural EOS、9 条截断、320 个完整 trajectory hashes。
|
||||
- [x] 独立 evaluator 对 256 条 T0 输出逐题评分：Code 四格 pass 为 `16/32 · 16/32 · 15/32 · 12/32`，Math 为 `19/32 · 19/32 · 17/32 · 16/32`；不把 59/128 与 71/128 写成 benchmark accuracy。
|
||||
- [x] 两域分别做 10,000 次选定任务 paired bootstrap，八个正确性 contrast 区间都跨零；长度 system-at-period 在 Code 为 `−130.875 [−178.688,−83.218]`，Math 为 `+25.125 [7.874,44.531]`，揭示任务域方向相反。
|
||||
- [x] 共同随机数对照的 256 / 256 个 contrast 全部共享相同 uniform tape；新进程重放 64 条结果，uniform hashes、token IDs、文本、停止状态与 RNG 等十二字段全部 64 / 64 exact。审计同时公开运行器在生成进程结束前已加载 gold、但 gold 没有进入 prompt/tokenizer/logits/warpers/CDF/tape/selection 的流程偏差。
|
||||
- [x] 第二十二个 DeepSeek 交互实验用五页签讲解显式 CRN sampler、选定任务 bootstrap、32 题浏览器、4×4 tape 诊断与十二字段复现/失败/偏差账；正式协议、manifest、runner、独立 evaluator、分析、复现与完整/compact 数据均已进入开源树。
|
||||
- [x] Round 08 本地闸门通过：88 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；数据合同、DeepSeek 全量历史、Round 06/07/08 三套 sampling 专项真实 Chrome 回归均通过，桌面与 390px 移动端无文档级溢出。
|
||||
- [x] DeepSeek Round 08 以源提交 `975ed3d`、不可变镜像 `20260729T221654Z-975ed3d` 发布；OCI index digest `sha256:759a8446…21452b`，复用 NAS `12010→8080`、NPM host 31 / cert 41 与门户 `LLM ATLAS / projects / 180`。容器 healthy、0 次重启，21/21 公网路由、HTTPS/2、gzip / immutable assets、Round 08 专项与 DeepSeek 全量生产 Chrome 回归通过；保留 `20260729T195952Z-9211333` 回滚。
|
||||
- [x] K3 Round 04 在任何正式结果前冻结 `llm-atlas-k3-attnres-reduced-v1`：WikiText-2 固定 revision 与 byte 拼接、192,000-window SHA-256 日程、16-block / 32-sublayer / width-192 公共主干、Baseline / Full / Block 三结构、三个初始化 seed、2,000-step 预算、主 BPC 判据、RMS / mixer / gradient 诊断与独立 replay 合同全部先落盘。
|
||||
- [x] 九个正式训练格全部完成，共消费 147,456,000 target bytes；同一 seed 的 9,541,824 个公共 core 参数初始化 hash 在三结构间 exact。Full − Baseline 三个 paired delta 为 `−.015996 / −.013207 / −.014498`，Block 为 `−.052660 / −.041349 / −.033388`，分别以均值 `−.014567 / −.042466 BPC` 满足预注册方向支持规则。
|
||||
- [x] 反结果同时进入主审计：核心参数 gradient RMS 的 16-block mean CV 为 Baseline `0.3447`、Full `0.5087`、Block `0.6306`；本 reduced metric 没有复现论文的大模型梯度均匀性叙述，不改换指标掩盖。
|
||||
- [x] 指定 Block / seed 2026073001 进行全新进程 2,000-step replay；manifest、model、optimizer、hashes、evaluations、training history、diagnostic、environment 八字段全部 exact，最终 BPC 同为 `1.947877975922777`；timing 明确不要求 exact。
|
||||
- [x] K3 Round 04 五视图实验室完成：三 seed BPC 曲线、Residual RMS / Block 锯齿、Full / Block depth-weight heatmap、梯度反证、成本/哈希/claim boundary 分开展示；完整 9-run JSON、compact 数据、复现清单、协议、审计、训练与聚合代码进入公开仓库。
|
||||
- [x] Round 04 本地闸门通过：91 个受检文件零诊断/提示，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；冻结数据、AttnRes 专项与 K3 全量真实 Chrome 回归通过，桌面/390px 移动端零文档级溢出、零运行时异常。
|
||||
- [x] K3 Round 04 以功能源提交 `4ce780d`、不可变镜像 `20260729T233142Z-4ce780d` 发布；OCI index digest `sha256:6e89f802…25f582`，复用 NAS `12010→8080`、NPM host 31 / cert 41 与门户 `LLM ATLAS / projects / 180`。容器 healthy、0 次重启，21/21 公网页面、HTTPS/2、gzip / immutable assets、AttnRes 专项与 K3 全量生产 Chrome 回归通过；保留 `20260729T221654Z-975ed3d` 回滚。
|
||||
- [x] K3 Round 05 一手定义审计确认 Figure 5(c) 未公开 gradient tensor、norm、reduction、diagnostic batch、AMP / clipping 时点或统计代码；Round 04 参数梯度与 Round 05 post-MLP output activation gradient 永久分对象记账，不把本站 operationalization 冒充作者实现。
|
||||
- [x] 在任何 formal 输出前冻结 16 / 32 blocks、Baseline / Block、三个 seed、8,000 steps、六个诊断点、CV + 首尾四分位 imbalance 联合判据、FP32 residual accumulator、20-step 双 smoke 与 depth-32 Block 完整 replay；768,000-window schedule SHA-256 为 `5041e09b…f4e`。
|
||||
- [x] 12 个 formal 格全部完成，共 786,432,000 target bytes；公共主干与三个 gate input tensor hashes 在 paired 架构间 exact。Block 的验证 BPC 在 6 / 6 配对中更低，depth-16 / 32 mean delta 为 `−.008938 / −.009872`，但不追加事后 BPC support 阈值。
|
||||
- [x] activation-gradient 结果分裂：首/末四分位 imbalance 在 6 / 6 配对改善，depth-16 / 32 均值为 `+61.0% / +72.0%`；全层 CV 却在 6 / 6 配对恶化，均值相对 reduction 为 `−10.3% / −60.0%`。两个 depth 都按预注册规则判为 `mixed / inconclusive`，总判定 `depth-dependent or inconclusive`。
|
||||
- [x] 绝对 gradient mean 仅为 Baseline 的 `57.4% / 54.4%`；参数 gradient CV 从 `0.416→0.683 / 0.397→0.772`，继续保留反结果。Output RMS 最后/第一层比则由 Baseline `4.59× / 6.08×` 降至 Block `1.17× / 1.89×`。
|
||||
- [x] 指定 depth-32 / Block / seed-2026073001 从初始化完整重训 8,000 steps；排除 run-kind / timing 后冻结字段 compare SHA-256 同为 `46300a45…4817`，model / optimizer state hashes exact。正式/compact/reproduction 物理 SHA-256 为 `ad461cbe…a8d / 5377a5e7…68e3 / aedcde6a…dea6`。
|
||||
- [x] K3 Round 05 五视图实验室完成：论文定义已知/未定义、绝对/归一化深度谱、六 checkpoint 时间轨迹、Output RMS 组节律、activation/parameter/BPC/成本/重放联合账全部可切换；21 个 raw JSON、完整 aggregate、compact、runner、analyzer、协议与审计进入公开树。
|
||||
- [x] Round 05 本地闸门通过：94 个 Astro 文件零诊断/提示，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；冻结数据、新专项、Round 04 与 K3 全量真实 Chrome 回归通过，桌面/390px 移动端零文档级溢出、零运行时异常。
|
||||
- [x] K3 Round 05 以功能提交 `f7670ef`、不可变镜像 `20260730T022610Z-f7670ef` 发布；OCI digest `sha256:02f902c1a64f976c6ed5d8614a826fca487b483f9addf2392637ea3f021cc09b`，NAS healthy / 零重启、21/21 公网路由、HTTPS/2、gzip/immutable 与三套 K3 生产 Chrome 回归全部通过；保留 `20260729T233142Z-4ce780d` 回滚。
|
||||
- [x] K3 Round 06 在正式训练前冻结 `llm-atlas-k3-attnres-spike-path-v1`：固定 Round 05 depth-32 Block、三个 seed、8,000 steps、六个 checkpoint、layer 21–25 目标集合、六个张量位置、四种 confirmatory reduction、三条 same-forward backward rule、20% 材料级阈值与完整 seed-1 replay；研究身份明确为受 Round 05 启发的 targeted follow-up，不冒充 blind discovery。
|
||||
- [x] 三个正式格共消费 196,608,000 target bytes；最终模型、优化器、六个 BPC、全部训练历史与 Round 05 post-MLP 数组逐 seed exact。完整 replay 再消费 65,536,000 bytes，16 组冻结字段全部 exact，compare SHA-256 为 `05396eaf…cbb87`。
|
||||
- [x] 尖峰发展轨迹显示 layer 21–25 不是初始化时固有：三 seed mean contrast 在 step 0 / 100 / 500 为 `0.757 / 0.563 / 0.524×`，到 step 2,000 / 4,000 / 8,000 升为 `2.233 / 2.594 / 2.754×`；seed 3 的最终 `1.881×` 同向但更弱。
|
||||
- [x] 六个张量位置的最终 mean contrast 均超过 1.5，且每个位置 3 / 3 seed 可见；最早采样点 `pre_attention_input` 已可见，但观测顺序不升级为物理起源或注入位置。
|
||||
- [x] 预注册 reduction 家族 12 / 12 格同时满足 contrast ≥ 1.5、top-5 与目标集合至少重叠 3 层、相对 element RMS 的 Spearman ≥ .8；正式结论严格写作 `robust within the preregistered reduction family`。
|
||||
- [x] 全局切断 64 个 depth mixer 与 output mixer 的 softmax/query/key source-gradient paths 后，contrast / peak 平均相对“降幅”为 `−2.00% / −0.84%`，三 seed 均略微上升，不支持材料级降低。将 learned value-backward coefficients 换成 `1/N` 后，contrast / peak 平均下降 `70.20% / 37.03%`，三 seed 都超过 20% 阈值，peak layer 全部移到 layer 2。
|
||||
- [x] mixer 观察关联与反向干预分层：layers 19–28 的 MLP latest weight 对 normalized gradient Pearson / Spearman 为 `.690 / .693`，MLP entropy 为 `−.636 / −.640`；相关性只提供候选机制，全局干预只支持 value-route sensitivity，不冒充局部 mixer attribution。
|
||||
- [x] K3 Round 06 五视图实验室完成：训练轨迹、六位置谱、12 格 reduction 稳健性、same-forward 三规则干预与 mixer 散点/证据阶梯可交互；protocol、scoping、audit、runner、analyzer、raw/aggregate/compact/reproduction 全部进入公开树。
|
||||
- [x] Round 06 本地闸门通过：97 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；Round 04/05/06 三套冻结数据、三套专项、K3 全量与全站 22 套真实 Chrome 回归通过，桌面/390px 移动端零文档级溢出、零 offender、零运行时异常；首页首发卡与陈旧跨页断言同步到 Round 06。
|
||||
- [x] K3 Round 06 以运行源提交 `b393780`、不可变镜像 `20260730T042651Z-b393780` 发布；OCI index digest `sha256:4e4cb2e065526f50c10cf759bab80a5a871177812ca9fa5c9da77c662c81a63a`，NAS healthy / 0 次重启、Compose Manager 标签、VPS→NAS、NPM host 31 / cert 41、DNS、HTTPS/2、首页/K3 公网内容、门户 `LLM ATLAS / projects / 180` 与全站 22 套生产 Chrome 回归全部通过；保留 Round 05 `20260730T022610Z-f7670ef` 与中间 `20260730T042110Z-a1a52d4` 回滚点。
|
||||
- [x] K3 Round 07 在正式输出前冻结 `llm-atlas-k3-attnres-local-path-v1`：固定 Round 06 三个 depth-32 Block 正式格、layer 21–25 spike set、groups 6+7、14 个 exact selector masks、sufficiency / restoration 双向 log-gap score、主门 50%、单组/分支/输出控制与完整 seed-1 replay；Grok 只读对抗审阅提出的六个 blocking protocol 问题全部在冻结前修正。
|
||||
- [x] 三个正式格与 replay 共消费 262,144,000 target bytes；Round 06 parent equivalence、forward identity、65-node census、selector identity / order / uniqueness、负对照与 loss-scale 闸门全部通过。正式最终 BPC 为 `1.7123525941 / 1.7093240656 / 1.7030966813`。
|
||||
- [x] global gap 在两个指标 × 三 seed 的 6 / 6 格通过。groups 6+7 sufficiency 的 mean score 为 contrast `.677`、peak `1.700`，6 / 6 ≥ `.50`；restoration mean 为 contrast `.650`、peak `.380`，contrast 3 / 3 通过而 peak 0 / 3 通过，正式状态固定为 `one_sided_evidence_localization_not_established`。
|
||||
- [x] 次级控制显示 group 7 MLP-only branch 在 sufficiency 方向 6 / 6 通过 material + margin 门；group 6 MLP-only 为 5 / 6，不能宣布 dominance。output-only mean sufficiency 仅 `.131 / .187`、0 / 6 过 50%；all-depth 为 `.949 / 1.145`。所有 score 都是非加性 log-gap 诊断，不写成贡献率。
|
||||
- [x] K3 Round 07 五视图实验室完成：65-node 路径图、14-mask 全矩阵、双向主门、branch/output controls 与 32 层原始谱/replay 审计可交互；protocol、scoping、Grok 结果前审阅、audit、runner、analyzer、packager、四个 raw JSON、aggregate、compact 与 reproduction 全部进入公开树。
|
||||
- [x] Round 07 本地闸门通过：100 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；Round 04/05/06/07 四套冻结数据、四套 K3 专项、K3 全量与全站 23 套真实 Chrome 回归通过。动态生成矩阵的 scoped CSS 退化由截图复查发现并修复；桌面/390px 移动端零文档级溢出、零 offender、零运行时异常。
|
||||
- [x] K3 Round 07 以功能源提交 `dc8ec30`、不可变镜像 `20260730T063811Z-dc8ec30` 发布；OCI index digest `sha256:98d441412408774628195ce23f431d25034bb1590210c11230d884513898e71d`。NAS `12010→8080` healthy / 0 次重启、Compose Manager 标签、VPS→NAS、NPM host 31 / cert 41、DNS、HTTPS/2、gzip、immutable asset、首页/K3 公网内容、门户 `LLM ATLAS / projects / 180` 与全站 23 套生产 Chrome 回归全部通过；保留 Round 06 `20260730T042651Z-b393780` 回滚点。
|
||||
- [x] K3 Round 08 在正式输出前冻结 `llm-atlas-k3-attnres-forward-training-v1`：四个 train-time forward variants × 三 seed × 8,000 steps、同 seed Round 05 historical pairing、step 8,000 的 layers 21–25 contrast / peak 双指标 20% 主门、BPC 每 seed / mean 护栏、描述性 `I67` 与完整 primary replay；不把架构消融写成 pure-forward 因果实验。
|
||||
- [x] 12 formal + 1 replay 全部一次完成，每格 65,536,000 target bytes；新处理总量 851,968,000，历史 references 196,608,000 单列。联合 groups 6+7 的 contrast / peak 六格降幅为 62.1%–77.3% / 32.3%–62.0%，6 / 6 通过；BPC 三 seed 最大 `+.009598`、均值 `+.006578`，质量门 4 / 4 通过。
|
||||
- [x] primary seed-1 replay scientific payload exact，SHA-256 为 `b85563ca…c051`；冻结主状态为 `forward_training_attenuation_established_within_reduced_protocol`。`I67` 的 step-8,000 mean 为 contrast `−.3670`、peak `−.1704`，只保留为跨独立训练的描述性 log residual。
|
||||
- [x] 结果前 Grok 实现审阅指出 smoke-only empty selector 的空 census；矩阵结束后删除 early return、加入 `forward_calls > 0`，修补后的 learned wrapper 实际执行 39 次 forward，父/包装器 15 组科学字段仍 exact。结果后 Grok 只读复算报告 `blocking_errors=0`、status / replay 均确认。
|
||||
- [x] K3 Round 08 五视图实验室完成：forward contract、六 checkpoint 训练轨迹、attenuation+BPC 主门、non-additivity map 与 32 层谱/replay audit 可交互；protocol、scoping、两阶段 Grok 审阅、runner、analyzer、13 raw、aggregate、compact、reproduction 与结果审计全部进入公开树。
|
||||
- [x] Round 08 本地闸门通过：101 个 Astro 文件零诊断，21 个页面、1,151 个站内引用、12 个跨页锚点零失败；Round 04–08 五套冻结数据、五套 K3 专项、K3 全量与全站 24 套真实 Chrome 回归通过。截图复查修复黑色实验室标题对比度；桌面/390px 移动端零文档级溢出、零 offender、零运行时异常。
|
||||
- [x] K3 Round 08 以功能源提交 `f177fa6`、不可变镜像 `20260730T104228Z-f177fa6` 发布；OCI index digest `sha256:0b5b9d9cbf1b4538a37b01727ba1d16be13f991408d3d60e20bc76023b9008bb`。NAS `12010→8080` healthy / 0 次重启、Compose Manager、VPS→NAS、NPM host 31 / cert 41、DNS、HTTPS/2、gzip、首页/K3/进度页、门户 `LLM ATLAS / projects / 180` 与全站 24 套生产 Chrome 回归全部通过；保留 Round 07 `20260730T064331Z-badf498` 回滚点。
|
||||
|
||||
## 正在进行
|
||||
|
||||
- [ ] K3 三轮下一闸门：获得真实 token hidden states、expert load 与 cache traces，解释或修订 `A_log [128]` 工件冲突，再做 Figure 3/4/5 数值重绘和独立小模型复现。
|
||||
- [ ] DeepSeek 七轮下一闸门：扩大到可做 task-level bootstrap 的预注册 source 抽样框，加入 per-row RNG/common-random-number 对照与 failure taxonomy；再推进干预式 mediation、SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] DeepSeek 八轮下一闸门：推进干预式 mediation、SM90 FlashMLA、FP8 / pipeline traces 与 R1-like RL 小模型复现。
|
||||
- [ ] 表示、位置与残差二轮：真实 hidden-state / norm traces、长上下文位置外推复现与 mHC / AttnRes 深层稳定性消融。
|
||||
- [ ] 评测安全二轮：真实 cross-harness / pass@k 复跑、Judge 元评测、动态污染与过拒案例。
|
||||
- [ ] 推理服务二轮：真实 GPU kernel / workload traces、功耗与成本、跨 vLLM / SGLang / TensorRT-LLM 复现。
|
||||
@@ -451,6 +498,36 @@
|
||||
| 2026-07-30 | 域均值必须与 source 方向同时展示 | English 句点 contrast 的均值 −8.5，但 3/4 source 为正；Code task interaction 的 +1 与 −1 在域均值 0 中抵消 |
|
||||
| 2026-07-30 | Round 07 随机分叉与复现同时过闸 | R0/R1 63/64 同格 trajectory 分叉；新进程 R0 的八项预注册字段 64/64 exact，五段 artifact hash chain 闭合 |
|
||||
| 2026-07-30 | DeepSeek Round 07 跨来源采样里程碑发布 | 源提交 `9211333`、镜像 `20260729T195952Z-9211333`、OCI `sha256:6c92883b…5584e2`；生产专项/全量 Chrome、HTTPS 与门户通过，保留 Round 06 回滚点 |
|
||||
| 2026-07-30 | Round 08 用显式 uniform tape 实现真正的共同随机数 | 64 tasks × T0 × 4 conditions，加 8 tasks × 3 诊断 tapes × 4 conditions；352 条正式输出的四条件 CDF 共享逐步 `u_t` |
|
||||
| 2026-07-30 | bootstrap 只描述冻结任务集的敏感性 | HumanEval / GSM8K 分开做 10,000 次 task-paired resamples；正确性带全跨零，不报告总体/seed CI 或 p-value |
|
||||
| 2026-07-30 | 输出长度揭示强任务域交互 | system-at-period 在 Code 为负、Math 为正，两个选定任务带都不跨零；不从长度外推能力 |
|
||||
| 2026-07-30 | Round 08 十二字段重放过闸 | 64/64 exact；uniform hash、完整 token IDs、文本、stop 与 RNG 一并进入复现合同，评分 gold 提前加载的流程偏差公开保留 |
|
||||
| 2026-07-30 | DeepSeek Round 08 任务 bootstrap / CRN 里程碑发布 | 源提交 `975ed3d`、镜像 `20260729T221654Z-975ed3d`、OCI `sha256:759a8446…21452b`；21/21 公网路由与生产专项/全量 Chrome 通过，保留 Round 07 回滚点 |
|
||||
| 2026-07-30 | K3 `A_log [128]` 冲突仍没有官方裁决 | 官方 main 仍期望 96；社区 #144 改为 128，#150 验证 tail zero 后裁为 96，两案都未合并。不把候选 patch 冒充官方 K3 forward |
|
||||
| 2026-07-30 | AttnRes 缩小实验先冻结再训练 | 三结构共享公共主干、初始化、窗口与优化器；三 seed paired BPC 只按预注册 `3/3 same direction + mean≤−.010` 判为本协议内方向支持 |
|
||||
| 2026-07-30 | 主结果与机制反结果同时发布 | Full / Block BPC 方向支持；核心参数 gradient RMS CV 却高于 Baseline，明确写成未复现论文梯度叙述 |
|
||||
| 2026-07-30 | 独立重放按数值合同而非计时合同验收 | Block / seed-1 的八组冻结字段 2,000 steps exact；wall time 受调度影响，不要求或声称 bit-exact |
|
||||
| 2026-07-30 | K3 Round 04 缩小 AttnRes 里程碑发布 | 功能源 `4ce780d`、镜像 `20260729T233142Z-4ce780d`、OCI `sha256:6e89f802…25f582`；21/21 公网页面与生产专项/全量 Chrome 通过，保留 Round 08 回滚点 |
|
||||
| 2026-07-30 | AttnRes 的“梯度”先按公开证据拆对象 | 论文 Figure 5 没有公开唯一 telemetry 合同；参数梯度与 post-MLP activation gradient 不再互相代称 |
|
||||
| 2026-07-30 | “更均匀”拆成首尾失衡与全层 CV | Block 6/6 改善 first/last，却 6/6 恶化 CV；局部尖峰与系统性早层隆起必须分开解释 |
|
||||
| 2026-07-30 | 绝对尺度与归一化形状永久同报 | Block activation-gradient mean 约为 Baseline 54%–57%；不能把更接近 1 的首尾比自动解释为各层信号更强 |
|
||||
| 2026-07-30 | Round 05 完整重放过闸 | depth-32 Block seed-1 从零重训 8,000 steps；全部冻结字段与 model/optimizer state hashes exact，timing 仍单独报告 |
|
||||
| 2026-07-30 | K3 Round 05 梯度尺度里程碑发布 | 功能源 `f7670ef`、镜像 `20260730T022610Z-f7670ef`、OCI `sha256:02f902c1…cc09b`；复用 NAS `12010→8080`、NPM 31 / cert 41、门户 order 180，保留 Round 04 回滚点 |
|
||||
| 2026-07-30 | 尖峰集合是 targeted follow-up，不是盲发现 | layer 21–25 来自 Round 05；Round 06 固定集合后才检查时间、位置、reduction 与路径，避免用同一结果发现并确认 |
|
||||
| 2026-07-30 | “最早可见”与“物理起源”永久分账 | pre-attention input 是六个采样点中最早可见张量；更早 mixer 与反向传播已经作用，不能写成尖峰从这里注入 |
|
||||
| 2026-07-30 | 同一前向只识别 backward-rule sensitivity | learned / detached / uniform 三模式的 logits、loss、activations 与 mixer summaries exact；结果不升级为重训模型或性能结论 |
|
||||
| 2026-07-30 | value-route 降幅不写成因果贡献百分比 | 全局 uniform value-backward 让 contrast 平均下降 70.2%，只支持预注册阈值下的材料级敏感性；不声称 value 路径“解释了 70.2%” |
|
||||
| 2026-07-30 | reduction 稳健性限定在预注册家族 | element RMS、token RMS mean/median/P95 的 12/12 格通过；探索性 reduction 和其他 batch 不被纳入确认性外推 |
|
||||
| 2026-07-30 | K3 Round 06 尖峰路径里程碑发布 | 运行源 `b393780`、镜像 `20260730T042651Z-b393780`、OCI `sha256:4e4cb2e0…1a63a`；21/21 公网页面链路与全站 22 套生产 Chrome 通过，保留 Round 05 与中间 Round 06 回滚点 |
|
||||
| 2026-07-30 | 局部路径必须同时通过 sufficiency 与 restoration | groups 6+7 的 sufficiency 6/6 过 `.50`，restoration 仅 contrast 3/3 通过、peak 0/3 通过；强单侧证据不升级为 localization |
|
||||
| 2026-07-30 | 局部 score 不写成可加贡献率 | 同一 scope 在 learned 与 uniform 背景的响应不同，`S_peak > 1` 与负 interaction residual 都是非加性诊断，不是 170% 贡献或方差分解 |
|
||||
| 2026-07-30 | branch 与 output 控制保持次级证据身份 | group 7 MLP-only 的 6/6 只属于 sufficiency branch gate；group 6 为 5/6，output-only 为 0/6，都不能补救失败的双向主门 |
|
||||
| 2026-07-30 | K3 Round 07 局部路径里程碑发布 | 功能源 `dc8ec30`、镜像 `20260730T063811Z-dc8ec30`、OCI `sha256:98d44141…e71d`；21/21 公网页面链路与全站 23 套生产 Chrome 通过，保留 Round 06 回滚点 |
|
||||
| 2026-07-30 | 训练期前向干预属于架构消融 | selected uniform mixer 同时改变 train/eval forward、natural backward 与后续 updates；不能写成只改 forward 的路径因果 |
|
||||
| 2026-07-30 | Round 08 主门与质量门同时成立 | groups 6+7 六个 attenuation 格 6/6 ≥20%；三 seed ΔBPC mean `+.006578`，4/4 过闸；结论只限固定缩小协议 |
|
||||
| 2026-07-30 | non-additivity 永久保留描述身份 | `I67` 来自三套独立训练，只是 cross-run log residual，不是因果 interaction、Shapley 或贡献率 |
|
||||
| 2026-07-30 | K3 研究线在 Round 08 主动收敛 | 不启动 Round 09；公开 `A_log [128]↔[96]` 冲突继续等待官方裁决，现有五轮实证停在可复现、可回滚的稳定边界 |
|
||||
| 2026-07-30 | K3 Round 08 训练期前向里程碑发布 | 功能源 `f177fa6`、镜像 `20260730T104228Z-f177fa6`、OCI `sha256:0b5b9d9c…008bb`；21/21 公网页面链路与全站 24 套生产 Chrome 通过，保留 Round 07 回滚点 |
|
||||
|
||||
## 未决问题
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
当前里程碑包含 17 专题学习地图、486 篇关键论文索引、Kimi K3 完整导读，
|
||||
语言模型前史、Transformer 基础、表示/位置/残差、DeepSeek 技术谱系、Scaling Laws、数据工程、长上下文、MoE、指令微调与人类偏好、推理、工具使用与长程 Agent、原生多模态、训练系统、推理服务、数值优化，以及评测与安全深度专题，
|
||||
以及 88 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
以及 109 个覆盖核心机制的原创交互视图。K3 二轮导读以 32 张问题账、16 图 / 5 表审计、
|
||||
8 个交互实验和 100 个一手/官方节点，完整覆盖架构、预训练、后训练、系统、评测、案例与附录。
|
||||
第三轮已完成开放工件与首个真实 kernel 里程碑：固定官方模型与 FlashKDA revisions，审计 96 个 checkpoint shards、
|
||||
497,220 个 tensor entries、真实 KDA / MLA / MoE / MoonViT shapes 与小范围参数统计，并用 4 个新视图
|
||||
@@ -27,8 +27,59 @@
|
||||
`sm_120a` wheel，在 RTX 5090 上完成 6/6 官方参考 exact-match 和 K3 fixed / varlen 形状计时。详见
|
||||
[K3_ARTIFACT_AUDIT.md](./research/K3_ARTIFACT_AUDIT.md) 与
|
||||
[checkpoint_probe.py](./experiments/k3/checkpoint_probe.py)、[FlashKDA probe](./experiments/k3/flashkda/)。
|
||||
DeepSeek 七轮专题以 24 张问题账、10 次技术转向、
|
||||
21 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
第四轮不裁剪仍未解释的 `A_log [128]` 去伪造 K3 forward，而是先冻结
|
||||
`llm-atlas-k3-attnres-reduced-v1`，用相同 16-block / 32-sublayer Transformer 主干、
|
||||
相同 WikiText-2 byte windows 与三个共同初始化 seed，从零训练 Baseline、Full AttnRes
|
||||
与 Block AttnRes 共 9 个 2,000-step 正式格。最终 Full / Block 相对 Baseline 的三 seed
|
||||
平均 paired delta 分别为 `−0.01457 / −0.04247 BPC`，都满足预注册的本协议内方向支持规则；
|
||||
但核心参数梯度 RMS 的跨 block CV 为 `0.3447 / 0.5087 / 0.6306`，没有复现论文式
|
||||
梯度均匀性叙述。指定 Block / seed-1 的全新进程 2,000-step replay 在八组数值与哈希字段
|
||||
上全部 exact，计时不要求 exact。完整 9-run JSON、compact payload、复现清单、训练/
|
||||
聚合代码与五视图实验室均进入开源树。详见
|
||||
[K3_ATTNRES_REDUCED_PROTOCOL.md](./research/K3_ATTNRES_REDUCED_PROTOCOL.md)、
|
||||
[K3_ATTNRES_REDUCED_AUDIT.md](./research/K3_ATTNRES_REDUCED_AUDIT.md) 与
|
||||
[AttnRes experiment](./experiments/k3/attnres/)。
|
||||
第五轮先审计 Attention Residuals Figure 5 的公开定义边界，再冻结
|
||||
`llm-atlas-k3-attnres-gradient-scale-v1`：以 post-MLP block output activation gradient
|
||||
为公开 operationalization，把深度扩为 16 / 32 blocks、预算扩为每格 8,000 steps，
|
||||
完成 Baseline / Block × 三 seed 共 12 格、786,432,000 formal target bytes。结果把
|
||||
“更均匀”拆成两个相反方向：Block 在 6 / 6 配对中把首/末四分位失衡改善 56%–81%，
|
||||
却因中后段局部尖峰让全层 CV 在 6 / 6 配对中恶化；两个深度都按预注册联合规则判为
|
||||
mixed / inconclusive。验证 BPC 仍在 6 / 6 配对中更低，但实际 step time 约 2.6×、
|
||||
peak allocated memory 约 2.2×，不冒充同算力优势。指定 depth-32 / Block / seed-1
|
||||
从零重训完整 8,000 steps，全部冻结字段以及 model / optimizer state hashes exact。
|
||||
详见
|
||||
[K3_ATTNRES_GRADIENT_DEFINITION_AUDIT.md](./research/K3_ATTNRES_GRADIENT_DEFINITION_AUDIT.md)、
|
||||
[K3_ATTNRES_GRADIENT_SCALE_AUDIT.md](./research/K3_ATTNRES_GRADIENT_SCALE_AUDIT.md) 与
|
||||
[gradient experiment](./experiments/k3/attnres_gradient/)。
|
||||
第六轮冻结 `llm-atlas-k3-attnres-spike-path-v1`，严格复用 Round 05 的 depth-32 Block
|
||||
训练格，在 optimizer 之外追踪六个 checkpoint、六个 block 内张量位置与四种预注册
|
||||
reduction，并在同一前向值下比较 learned、detached-learned 与 uniform-value-backward
|
||||
三条反向规则。layer 21–25 的 post-MLP contrast 在 step 500 仍仅为三 seed 平均 `0.524×`，
|
||||
到 step 2,000 升至 `2.233×`，说明尖峰在训练中形成；六个位置全部在 3/3 seed 可见，
|
||||
四种 reduction 的 12/12 格同时通过 contrast、top-5 overlap 与 Spearman 闸门。
|
||||
切断全局 softmax/query/key 源梯度后 contrast 反而平均增加约 2.0%；把 learned
|
||||
value-backward coefficients 换成均匀系数后，contrast 平均下降 `70.2%`、peak/mean
|
||||
下降 `37.0%`，三 seed 都达到预注册材料级敏感性阈值。该结果只支持全局反向路径敏感性，
|
||||
不冒充训练变体、因果贡献百分比或局部 mixer 归因。正式三格与完整 replay 共
|
||||
262,144,000 target bytes，Round 05 等价字段与 16 组 replay 字段全部 exact。详见
|
||||
[K3_ATTNRES_SPIKE_PROTOCOL.md](./research/K3_ATTNRES_SPIKE_PROTOCOL.md)、
|
||||
[K3_ATTNRES_SPIKE_AUDIT.md](./research/K3_ATTNRES_SPIKE_AUDIT.md) 与
|
||||
[spike-path experiment](./experiments/k3/attnres_spike/)。
|
||||
第七轮冻结 `llm-atlas-k3-attnres-local-path-v1`，把 Round 06 的全局 value-route
|
||||
sensitivity 收缩为 14 个 exact selector masks，并同时从 learned 背景测 sufficiency、
|
||||
从 all-uniform 背景测 restoration。groups 6+7 的 16 个 depth mixers 在充分性方向
|
||||
两个指标 × 三 seed 的 `6 / 6` 格全部超过 `.50` global log gap；恢复性却只有 contrast
|
||||
`3 / 3` 通过，peak `0 / 3` 通过，mean peak restoration 仅 `.380`。因此冻结结论是
|
||||
`one_sided_evidence_localization_not_established`，不是“定位成功”。group 7 的
|
||||
MLP-only 次级门 6 / 6 通过，但只属于 sufficiency 探索；output-only 为 0 / 6。
|
||||
三个正式格与完整 replay 共处理 262,144,000 target bytes，Round 06 equivalence、
|
||||
14 / 14 forward identity、65-node census、selector 与 canonical replay 全部 exact。
|
||||
详见 [K3_ATTNRES_LOCAL_PATH_PROTOCOL.md](./research/K3_ATTNRES_LOCAL_PATH_PROTOCOL.md)、
|
||||
[K3_ATTNRES_LOCAL_PATH_AUDIT.md](./research/K3_ATTNRES_LOCAL_PATH_AUDIT.md) 与
|
||||
[local-path experiment](./experiments/k3/attnres_local_path/)。
|
||||
DeepSeek 八轮专题以 24 张问题账、10 次技术转向、
|
||||
22 个交互实验和 60 个一手/官方节点，串起 Dense、MoE、MLA、V3 协同、R1 与 V4；
|
||||
并固定官方 V2-Lite revision，在 RTX 5090 上连续执行 7/27 层，记录 3,240 次真实专家选择、
|
||||
MLA/HF eager cache shapes 与 `31/31` exact 独立复跑；进一步用真实 layer-1 权重执行官方 V3
|
||||
naive/absorb 路径，实际写入 576 元素 latent cache，并以 FP32 将两种结合顺序的最大误差压到
|
||||
@@ -85,6 +136,14 @@ Round 07 保持 256 条输出预算不变，改为 16 条预先冻结的 source
|
||||
`16/16 · 8/16 · 12/16 · 16/16`。句点在 11 / 16 条 source 上缩短平均长度，但
|
||||
English 的首条 source 为 `−121`、其余三条为正，证明单题均值会与多数 source
|
||||
方向相反；新进程 R0 的八项合同字段仍为 `64 / 64` exact。
|
||||
Round 08 再把覆盖扩大到 HumanEval / GSM8K 各 32 条冻结任务，并以 SHA-256 显式
|
||||
uniform tape 代替“重置同一 seed”的伪共同随机数：T0 全任务四条件 256 条、额外三条
|
||||
随机带诊断 96 条，共 352 条正式输出，343 条 natural EOS、320 个完整 trajectory hashes。
|
||||
两个域分别做 10,000 次选定任务 paired bootstrap；八个正确性区间全部跨零，但
|
||||
system-at-period 的输出长度在 Code 为 `−130.9 [−178.7,−83.2]`、在 Math 为
|
||||
`+25.1 [7.9,44.5]`，说明干预方向依赖任务域。全新进程对 64 条输出的 uniform hash、
|
||||
token IDs、文本与停止等十二字段逐一重放，`64 / 64` exact；同时公开评分金标准被运行器
|
||||
提前加载、但没有进入 prompt / logits / CDF / tape / selection 的流程偏差。
|
||||
详见
|
||||
[DEEPSEEK_V2_LITE_TRACE.md](./research/DEEPSEEK_V2_LITE_TRACE.md) 与
|
||||
[DEEPSEEK_MLA_ABSORB_AUDIT.md](./research/DEEPSEEK_MLA_ABSORB_AUDIT.md)、
|
||||
@@ -102,7 +161,9 @@ English 的首条 source 为 `−121`、其余三条为正，证明单题均值
|
||||
[DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md](./research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_PROTOCOL.md) 与
|
||||
[DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_SAMPLING_AUDIT.md)，以及
|
||||
[DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md](./research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md) 与
|
||||
[DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md)。
|
||||
[DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md)，以及
|
||||
[DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md](./research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md) 与
|
||||
[DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md](./research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md)。
|
||||
其余专题按进度账本持续扩建。
|
||||
|
||||
## 本地开发
|
||||
|
||||
@@ -853,3 +853,135 @@ See `research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_CROSS_SOURCE_SAMPLING_AUDIT.md` for the
|
||||
preregistered source frame, task matrices, source-direction contrasts,
|
||||
failure identities, hash chain, exact replay scope, and non-claims.
|
||||
|
||||
## Selected-task bootstrap with explicit common random numbers
|
||||
|
||||
`v2_lite_chat_task_bootstrap_crn_probe.py` expands the task frame to all 32
|
||||
frozen HumanEval and 32 frozen GSM8K sources. The primary analysis runs one
|
||||
preregistered tape T0 on every task. Four tasks per domain additionally run
|
||||
T1-T3 as a crossed tape-sensitivity diagnostic:
|
||||
|
||||
```text
|
||||
64 tasks × T0 × 4 conditions = 256 outputs
|
||||
8 diagnostic tasks × 3 additional tapes × 4 conditions = 96 outputs
|
||||
formal total = 352 outputs
|
||||
```
|
||||
|
||||
Unlike the preceding batch-seed-aligned studies, Round 08 constructs a true
|
||||
step-index common-random-number tape. The same SHA-256-derived `u_t` is
|
||||
mapped through each condition's official `.3/.95` temperature/top-p
|
||||
distribution using a float32 token-ID-ordered inverse CDF. It does not call
|
||||
`transformers.generate` or `torch.multinomial`.
|
||||
|
||||
The frozen manifest and all 256 Chat prompt hashes must be built before any
|
||||
model outputs:
|
||||
|
||||
```bash
|
||||
node scripts/build-deepseek-chat-task-bootstrap-manifest.mjs \
|
||||
--reference \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--output \
|
||||
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json
|
||||
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps \
|
||||
python -B scripts/freeze-deepseek-chat-task-bootstrap-prompts.py \
|
||||
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||
--manifest \
|
||||
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
|
||||
--reference-routing-json \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet
|
||||
```
|
||||
|
||||
The formal runner writes one hash-checked journal per source and supports
|
||||
`--resume`:
|
||||
|
||||
```bash
|
||||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
PYTHONPATH=/path/to/transformers-4.41.2-deps:experiments/deepseek \
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_probe.py \
|
||||
--execution-mode formal \
|
||||
--artifact-dir /path/to/deepseek-v2-lite-chat \
|
||||
--manifest \
|
||||
research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json \
|
||||
--reference-routing-json \
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--tnews /path/to/tnews/test.json \
|
||||
--tnews-archive /path/to/tnews_public.zip \
|
||||
--wikitext /path/to/wikitext-validation.parquet \
|
||||
--journal-dir /path/to/formal-journal \
|
||||
--resume \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json
|
||||
```
|
||||
|
||||
Evaluation keeps T0 primary rows separate from T1-T3 diagnostics:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=experiments/deepseek \
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_evaluator.py \
|
||||
--sampling-json \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
|
||||
--human-eval /path/to/HumanEval.jsonl.gz \
|
||||
--gsm8k /path/to/gsm8k/test.jsonl \
|
||||
--sandbox-image \
|
||||
python:3.11-alpine@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4 \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json
|
||||
```
|
||||
|
||||
The analysis resamples the 32 selected tasks 10,000 times within each domain
|
||||
while preserving condition pairing. These are selected-task resampling
|
||||
bands under T0, not benchmark-population or generation-seed confidence
|
||||
intervals:
|
||||
|
||||
```bash
|
||||
python -B \
|
||||
experiments/deepseek/v2_lite_chat_task_bootstrap_crn_analysis.py \
|
||||
--sampling-json \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn.json \
|
||||
--evaluation-json \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-eval.json \
|
||||
--reproduction-json \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-reproduction.json \
|
||||
--output \
|
||||
src/data/deepseek-v2-lite-chat-task-bootstrap-crn-analysis.json
|
||||
```
|
||||
|
||||
The formal grid contains 343/352 natural EOS outputs and 320 unique complete
|
||||
trajectories. T0 correctness totals are Code 59/128 and Math 71/128 across
|
||||
32 tasks × four prompt conditions; they are not standard benchmark scores.
|
||||
All correctness contrast bands cross zero. The clearest length interaction
|
||||
is system-on minus system-off under the period boundary: Code is -130.875
|
||||
tokens with band [-178.688, -83.218], while Math is +25.125 with band
|
||||
[7.874, 44.531]. The preregistered 64-cell cross-process replay is exact in
|
||||
all twelve audited fields.
|
||||
|
||||
```text
|
||||
formal ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809
|
||||
eval 82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab
|
||||
replay 6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508
|
||||
compare 63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5
|
||||
analysis 9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae
|
||||
compact 082e3c56373e5f91d51d88b1430d14ad3b68b0f200e80901171c2fb62a1e57bf
|
||||
```
|
||||
|
||||
Run the complete offline hash and invariant audit with:
|
||||
|
||||
```bash
|
||||
node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs
|
||||
```
|
||||
|
||||
See `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md` and
|
||||
`research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_CRN_AUDIT.md` for the
|
||||
pre-output corrections, explicit sampler, task bootstrap, transition tables,
|
||||
multi-tape matrices, exact replay, reported gold-loading process deviation,
|
||||
and claim boundaries.
|
||||
|
||||
@@ -0,0 +1,620 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build task-paired bootstrap and multi-tape diagnostics for Round 08."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from statistics import mean, median
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1"
|
||||
CONDITIONS = (
|
||||
"s0_eos",
|
||||
"s1_eos",
|
||||
"s0_period",
|
||||
"s1_period",
|
||||
)
|
||||
DOMAINS = ("code", "math")
|
||||
BOOTSTRAP_SEED = 1364512825
|
||||
BOOTSTRAP_RESAMPLES = 10000
|
||||
CONTRASTS = {
|
||||
"period_at_s0": ("s0_eos", "s0_period"),
|
||||
"period_at_s1": ("s1_eos", "s1_period"),
|
||||
"system_at_eos": ("s0_eos", "s1_eos"),
|
||||
"system_at_period": ("s0_period", "s1_period"),
|
||||
}
|
||||
METRICS = (
|
||||
"fixed_budget_success",
|
||||
"strict_complete_success",
|
||||
"natural_eos",
|
||||
"generated_tokens",
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--sampling-json", type=Path, required=True)
|
||||
parser.add_argument("--evaluation-json", type=Path, required=True)
|
||||
parser.add_argument("--reproduction-json", type=Path)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(16 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_hash(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def metric_value(row: dict[str, Any], metric: str) -> float:
|
||||
evaluation = row["task_evaluation"]
|
||||
if metric == "fixed_budget_success":
|
||||
key = (
|
||||
"fixed_budget_tests_pass"
|
||||
if row["domain"] == "code"
|
||||
else "fixed_budget_numeric_exact"
|
||||
)
|
||||
return float(evaluation[key])
|
||||
if metric == "strict_complete_success":
|
||||
key = (
|
||||
"strict_complete_tests_pass"
|
||||
if row["domain"] == "code"
|
||||
else "strict_complete_numeric_exact"
|
||||
)
|
||||
return float(evaluation[key])
|
||||
if metric == "natural_eos":
|
||||
return float(row["hit_eos"])
|
||||
if metric == "generated_tokens":
|
||||
return float(row["generated_tokens"])
|
||||
raise KeyError(metric)
|
||||
|
||||
|
||||
def summarize(values: list[float]) -> dict[str, Any]:
|
||||
return {
|
||||
"count": len(values),
|
||||
"mean": mean(values) if values else None,
|
||||
"median": median(values) if values else None,
|
||||
"min": min(values) if values else None,
|
||||
"max": max(values) if values else None,
|
||||
}
|
||||
|
||||
|
||||
def percentile_band(values: np.ndarray) -> dict[str, float]:
|
||||
quantiles = np.percentile(values, [2.5, 50.0, 97.5])
|
||||
return {
|
||||
"p2_5": float(quantiles[0]),
|
||||
"p50": float(quantiles[1]),
|
||||
"p97_5": float(quantiles[2]),
|
||||
}
|
||||
|
||||
|
||||
def direction_counts(values: list[float]) -> dict[str, int]:
|
||||
epsilon = 1e-12
|
||||
return {
|
||||
"positive": sum(value > epsilon for value in values),
|
||||
"zero": sum(abs(value) <= epsilon for value in values),
|
||||
"negative": sum(value < -epsilon for value in values),
|
||||
}
|
||||
|
||||
|
||||
def success_transition(
|
||||
left: list[float],
|
||||
right: list[float],
|
||||
) -> dict[str, int]:
|
||||
pairs = [(int(a), int(b)) for a, b in zip(left, right, strict=True)]
|
||||
return {
|
||||
"fail_to_fail": sum(a == 0 and b == 0 for a, b in pairs),
|
||||
"fail_to_pass": sum(a == 0 and b == 1 for a, b in pairs),
|
||||
"pass_to_fail": sum(a == 1 and b == 0 for a, b in pairs),
|
||||
"pass_to_pass": sum(a == 1 and b == 1 for a, b in pairs),
|
||||
}
|
||||
|
||||
|
||||
def common_prefix(left: list[int], right: list[int]) -> int:
|
||||
count = 0
|
||||
for left_id, right_id in zip(left, right):
|
||||
if left_id != right_id:
|
||||
break
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def tape_uint64(tape: str, source_id: str, step: int) -> int:
|
||||
payload = (
|
||||
f"{PROTOCOL_ID}\0uniform\0"
|
||||
f"{tape}\0{source_id}\0{step}"
|
||||
).encode()
|
||||
return int.from_bytes(hashlib.sha256(payload).digest()[:8], "big")
|
||||
|
||||
|
||||
def build_indices(
|
||||
sampling: dict[str, Any],
|
||||
evaluation: dict[str, Any],
|
||||
) -> tuple[
|
||||
dict[tuple[str, str, str], dict[str, Any]],
|
||||
dict[tuple[str, str, str], dict[str, Any]],
|
||||
]:
|
||||
sample_index = {
|
||||
(
|
||||
source["id"],
|
||||
run["tape_label"],
|
||||
output["condition"],
|
||||
): output
|
||||
for source in sampling["sources"]
|
||||
for run in source["runs"]
|
||||
for output in run["outputs"]
|
||||
}
|
||||
eval_index = {
|
||||
(
|
||||
row["source_id"],
|
||||
row["tape_label"],
|
||||
row["condition"],
|
||||
): row
|
||||
for row in evaluation["rows"]
|
||||
}
|
||||
if set(sample_index) != set(eval_index):
|
||||
raise RuntimeError("sampling/evaluation cell keys differ")
|
||||
return sample_index, eval_index
|
||||
|
||||
|
||||
def main_analysis(
|
||||
*,
|
||||
sampling: dict[str, Any],
|
||||
sample_index: dict[tuple[str, str, str], dict[str, Any]],
|
||||
eval_index: dict[tuple[str, str, str], dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
result = {}
|
||||
for domain in DOMAINS:
|
||||
sources = [
|
||||
source
|
||||
for source in sampling["sources"]
|
||||
if source["domain"] == domain
|
||||
]
|
||||
sources.sort(key=lambda row: row["within_domain_index"])
|
||||
if len(sources) != 32:
|
||||
raise RuntimeError(f"{domain}: main frame must have 32 tasks")
|
||||
source_ids = [source["id"] for source in sources]
|
||||
rng = np.random.default_rng(BOOTSTRAP_SEED)
|
||||
sampled_indices = rng.integers(
|
||||
0,
|
||||
len(source_ids),
|
||||
size=(BOOTSTRAP_RESAMPLES, len(source_ids)),
|
||||
endpoint=False,
|
||||
)
|
||||
conditions = {}
|
||||
for condition in CONDITIONS:
|
||||
rows = [
|
||||
eval_index[(source_id, "T0", condition)]
|
||||
for source_id in source_ids
|
||||
]
|
||||
conditions[condition] = {
|
||||
metric: summarize(
|
||||
[metric_value(row, metric) for row in rows]
|
||||
)
|
||||
for metric in METRICS
|
||||
} | {
|
||||
"task_outcomes": dict(
|
||||
sorted(
|
||||
Counter(row["task_outcome"] for row in rows).items()
|
||||
)
|
||||
)
|
||||
}
|
||||
contrasts = {}
|
||||
for name, (left_condition, right_condition) in CONTRASTS.items():
|
||||
metrics = {}
|
||||
for metric in METRICS:
|
||||
left = [
|
||||
metric_value(
|
||||
eval_index[(source_id, "T0", left_condition)],
|
||||
metric,
|
||||
)
|
||||
for source_id in source_ids
|
||||
]
|
||||
right = [
|
||||
metric_value(
|
||||
eval_index[(source_id, "T0", right_condition)],
|
||||
metric,
|
||||
)
|
||||
for source_id in source_ids
|
||||
]
|
||||
differences = np.asarray(right) - np.asarray(left)
|
||||
bootstrap_means = differences[
|
||||
sampled_indices
|
||||
].mean(axis=1)
|
||||
metric_result = {
|
||||
"right_minus_left_point": float(differences.mean()),
|
||||
"selected_task_resampling_band": percentile_band(
|
||||
bootstrap_means
|
||||
),
|
||||
"task_differences": summarize(
|
||||
differences.tolist()
|
||||
),
|
||||
"direction_counts": direction_counts(
|
||||
differences.tolist()
|
||||
),
|
||||
"by_source": [
|
||||
{
|
||||
"source_id": source_id,
|
||||
"domain_index": index,
|
||||
"left": float(left[index]),
|
||||
"right": float(right[index]),
|
||||
"right_minus_left": float(
|
||||
differences[index]
|
||||
),
|
||||
}
|
||||
for index, source_id in enumerate(source_ids)
|
||||
],
|
||||
}
|
||||
if metric in {
|
||||
"fixed_budget_success",
|
||||
"strict_complete_success",
|
||||
}:
|
||||
metric_result["transition"] = success_transition(
|
||||
left,
|
||||
right,
|
||||
)
|
||||
metrics[metric] = metric_result
|
||||
|
||||
trajectory_rows = []
|
||||
crn_exact = 0
|
||||
for source_id in source_ids:
|
||||
left_output = sample_index[
|
||||
(source_id, "T0", left_condition)
|
||||
]
|
||||
right_output = sample_index[
|
||||
(source_id, "T0", right_condition)
|
||||
]
|
||||
left_ids = left_output["generated_token_ids"]
|
||||
right_ids = right_output["generated_token_ids"]
|
||||
shared_steps = min(len(left_ids), len(right_ids))
|
||||
expected_left_uniforms = [
|
||||
tape_uint64("T0", source_id, step)
|
||||
for step in range(
|
||||
left_output["uniform_steps_consumed"]
|
||||
)
|
||||
]
|
||||
expected_right_uniforms = [
|
||||
tape_uint64("T0", source_id, step)
|
||||
for step in range(
|
||||
right_output["uniform_steps_consumed"]
|
||||
)
|
||||
]
|
||||
left_uniform_exact = (
|
||||
canonical_hash(expected_left_uniforms)
|
||||
== left_output["uniform_uint64_prefix_sha256"]
|
||||
)
|
||||
right_uniform_exact = (
|
||||
canonical_hash(expected_right_uniforms)
|
||||
== right_output["uniform_uint64_prefix_sha256"]
|
||||
)
|
||||
crn_exact += left_uniform_exact and right_uniform_exact
|
||||
trajectory_rows.append(
|
||||
{
|
||||
"source_id": source_id,
|
||||
"shared_active_steps": shared_steps,
|
||||
"common_prefix_tokens": common_prefix(
|
||||
left_ids,
|
||||
right_ids,
|
||||
),
|
||||
"token_ids_exact": left_ids == right_ids,
|
||||
"shared_uniform_prefix_exact": (
|
||||
left_uniform_exact and right_uniform_exact
|
||||
),
|
||||
}
|
||||
)
|
||||
contrasts[name] = {
|
||||
"left": left_condition,
|
||||
"right": right_condition,
|
||||
"metrics": metrics,
|
||||
"trajectory": {
|
||||
"sources": len(trajectory_rows),
|
||||
"shared_uniform_prefix_exact": crn_exact,
|
||||
"exact_trajectories": sum(
|
||||
row["token_ids_exact"]
|
||||
for row in trajectory_rows
|
||||
),
|
||||
"common_prefix_tokens": summarize(
|
||||
[
|
||||
float(row["common_prefix_tokens"])
|
||||
for row in trajectory_rows
|
||||
]
|
||||
),
|
||||
"rows": trajectory_rows,
|
||||
},
|
||||
}
|
||||
result[domain] = {
|
||||
"tasks": len(source_ids),
|
||||
"source_ids": source_ids,
|
||||
"tape": "T0",
|
||||
"conditions": conditions,
|
||||
"contrasts": contrasts,
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def tape_diagnostic(
|
||||
*,
|
||||
sampling: dict[str, Any],
|
||||
eval_index: dict[tuple[str, str, str], dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
result = {}
|
||||
for domain in DOMAINS:
|
||||
sources = [
|
||||
source
|
||||
for source in sampling["sources"]
|
||||
if source["domain"] == domain and len(source["runs"]) == 4
|
||||
]
|
||||
sources.sort(key=lambda row: row["within_domain_index"])
|
||||
if [row["within_domain_index"] for row in sources] != [
|
||||
0,
|
||||
8,
|
||||
16,
|
||||
24,
|
||||
]:
|
||||
raise RuntimeError(
|
||||
f"{domain}: diagnostic task indices differ"
|
||||
)
|
||||
source_ids = [row["id"] for row in sources]
|
||||
tapes = ["T0", "T1", "T2", "T3"]
|
||||
by_tape_condition = {
|
||||
tape: {
|
||||
condition: {
|
||||
metric: summarize(
|
||||
[
|
||||
metric_value(
|
||||
eval_index[
|
||||
(source_id, tape, condition)
|
||||
],
|
||||
metric,
|
||||
)
|
||||
for source_id in source_ids
|
||||
]
|
||||
)
|
||||
for metric in METRICS
|
||||
}
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
for tape in tapes
|
||||
}
|
||||
contrasts = {}
|
||||
for name, (left_condition, right_condition) in CONTRASTS.items():
|
||||
metric_results = {}
|
||||
for metric in METRICS:
|
||||
matrix = np.asarray(
|
||||
[
|
||||
[
|
||||
metric_value(
|
||||
eval_index[
|
||||
(
|
||||
source_id,
|
||||
tape,
|
||||
right_condition,
|
||||
)
|
||||
],
|
||||
metric,
|
||||
)
|
||||
- metric_value(
|
||||
eval_index[
|
||||
(
|
||||
source_id,
|
||||
tape,
|
||||
left_condition,
|
||||
)
|
||||
],
|
||||
metric,
|
||||
)
|
||||
for tape in tapes
|
||||
]
|
||||
for source_id in source_ids
|
||||
],
|
||||
dtype=float,
|
||||
)
|
||||
tape_means = matrix.mean(axis=0)
|
||||
task_means = matrix.mean(axis=1)
|
||||
task_ranges_within_tape = (
|
||||
matrix.max(axis=0) - matrix.min(axis=0)
|
||||
)
|
||||
tape_ranges_within_task = (
|
||||
matrix.max(axis=1) - matrix.min(axis=1)
|
||||
)
|
||||
metric_results[metric] = {
|
||||
"matrix_task_by_tape": matrix.tolist(),
|
||||
"tape_means": {
|
||||
tape: float(tape_means[index])
|
||||
for index, tape in enumerate(tapes)
|
||||
},
|
||||
"task_means": {
|
||||
source_id: float(task_means[index])
|
||||
for index, source_id in enumerate(source_ids)
|
||||
},
|
||||
"direction_by_tape": {
|
||||
tape: direction_counts(
|
||||
matrix[:, index].tolist()
|
||||
)
|
||||
for index, tape in enumerate(tapes)
|
||||
},
|
||||
"task_range_within_tape": summarize(
|
||||
task_ranges_within_tape.tolist()
|
||||
),
|
||||
"tape_range_within_task": summarize(
|
||||
tape_ranges_within_task.tolist()
|
||||
),
|
||||
"grand_mean_descriptive": float(matrix.mean()),
|
||||
}
|
||||
contrasts[name] = {
|
||||
"left": left_condition,
|
||||
"right": right_condition,
|
||||
"metrics": metric_results,
|
||||
}
|
||||
result[domain] = {
|
||||
"tasks": len(source_ids),
|
||||
"source_ids": source_ids,
|
||||
"tapes": tapes,
|
||||
"by_tape_condition": by_tape_condition,
|
||||
"contrasts": contrasts,
|
||||
"independence_warning": (
|
||||
"The 4 tasks x 4 tapes are crossed repeated measures, "
|
||||
"not 16 independent tasks."
|
||||
),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
for path in (args.sampling_json, args.evaluation_json):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
sampling = json.loads(
|
||||
args.sampling_json.read_text(encoding="utf-8")
|
||||
)
|
||||
evaluation = json.loads(
|
||||
args.evaluation_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if (
|
||||
sampling["protocol_id"] != PROTOCOL_ID
|
||||
or evaluation["protocol_id"] != PROTOCOL_ID
|
||||
):
|
||||
raise RuntimeError("protocol ID differs")
|
||||
if sampling["summary"]["outputs"] != 352:
|
||||
raise RuntimeError("sampling formal output count differs")
|
||||
if len(evaluation["rows"]) != 352:
|
||||
raise RuntimeError("evaluation row count differs")
|
||||
sample_index, eval_index = build_indices(sampling, evaluation)
|
||||
reproduction = None
|
||||
if args.reproduction_json is not None:
|
||||
if not args.reproduction_json.is_file():
|
||||
raise FileNotFoundError(args.reproduction_json)
|
||||
reproduction_payload = json.loads(
|
||||
args.reproduction_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if reproduction_payload["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError("reproduction protocol ID differs")
|
||||
reproduction = {
|
||||
"path": str(args.reproduction_json),
|
||||
"sha256": sha256_file(args.reproduction_json),
|
||||
"summary": reproduction_payload["summary"],
|
||||
}
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"inputs": {
|
||||
"sampling": {
|
||||
"path": str(args.sampling_json),
|
||||
"sha256": sha256_file(args.sampling_json),
|
||||
"content_hash": sampling["content_hash"],
|
||||
},
|
||||
"evaluation": {
|
||||
"path": str(args.evaluation_json),
|
||||
"sha256": sha256_file(args.evaluation_json),
|
||||
"content_hash": evaluation["content_hash"],
|
||||
},
|
||||
"reproduction": reproduction,
|
||||
},
|
||||
"bootstrap_contract": {
|
||||
"resamples": BOOTSTRAP_RESAMPLES,
|
||||
"seed": BOOTSTRAP_SEED,
|
||||
"rng": "numpy.random.default_rng reset per domain",
|
||||
"unit": "selected task",
|
||||
"paired_conditions": True,
|
||||
"tape": "T0",
|
||||
"interval_label": (
|
||||
"selected-task resampling band for the fixed "
|
||||
"32-task frame and T0"
|
||||
),
|
||||
"not": [
|
||||
"benchmark-population confidence interval",
|
||||
"model-ability confidence interval",
|
||||
"generation-seed uncertainty interval",
|
||||
"causal-effect confidence interval",
|
||||
],
|
||||
},
|
||||
"main_t0_selected_task_analysis": main_analysis(
|
||||
sampling=sampling,
|
||||
sample_index=sample_index,
|
||||
eval_index=eval_index,
|
||||
),
|
||||
"multi_tape_diagnostic": tape_diagnostic(
|
||||
sampling=sampling,
|
||||
eval_index=eval_index,
|
||||
),
|
||||
"claim_boundary": [
|
||||
"HumanEval and GSM8K are analyzed separately.",
|
||||
(
|
||||
"T0 selected-task bands describe only this frozen "
|
||||
"32-task frame."
|
||||
),
|
||||
(
|
||||
"T1-T3 are sensitivity diagnostics and are not pooled "
|
||||
"into the T0 primary success rates."
|
||||
),
|
||||
(
|
||||
"Common random numbers align probability quantiles; "
|
||||
"they do not force identical sampled tokens."
|
||||
),
|
||||
"Period prompts are counterfactual, not official-valid chats.",
|
||||
],
|
||||
}
|
||||
result["content_hash"] = canonical_hash(
|
||||
{
|
||||
"bootstrap_contract": result["bootstrap_contract"],
|
||||
"main": result["main_t0_selected_task_analysis"],
|
||||
"diagnostic": result["multi_tape_diagnostic"],
|
||||
}
|
||||
)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.output.read_bytes()
|
||||
compact = {
|
||||
domain: {
|
||||
name: {
|
||||
metric: result[
|
||||
"main_t0_selected_task_analysis"
|
||||
][domain]["contrasts"][name]["metrics"][metric][
|
||||
"right_minus_left_point"
|
||||
]
|
||||
for metric in METRICS
|
||||
}
|
||||
for name in CONTRASTS
|
||||
}
|
||||
for domain in DOMAINS
|
||||
}
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"bytes": len(payload),
|
||||
"main_contrast_points": compact,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,364 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Evaluate Round 08 while keeping T0 and multi-tape diagnostics separate."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from statistics import mean
|
||||
from typing import Any
|
||||
|
||||
import v2_lite_chat_completion_evaluator as completion
|
||||
import v2_lite_chat_sampling_evaluator as shared
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1"
|
||||
CONDITIONS = (
|
||||
"s0_eos",
|
||||
"s1_eos",
|
||||
"s0_period",
|
||||
"s1_period",
|
||||
)
|
||||
DOMAINS = ("code", "math")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--sampling-json", type=Path, required=True)
|
||||
parser.add_argument("--human-eval", type=Path, required=True)
|
||||
parser.add_argument("--gsm8k", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--sandbox-image",
|
||||
required=True,
|
||||
help="Pinned image reference including @sha256 digest.",
|
||||
)
|
||||
parser.add_argument("--timeout-seconds", type=float, default=5.0)
|
||||
parser.add_argument("--skip-code-execution", action="store_true")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def task_outcome(
|
||||
domain: str,
|
||||
evaluation: dict[str, Any],
|
||||
) -> str:
|
||||
if domain == "math":
|
||||
if evaluation["fixed_budget_numeric_exact"]:
|
||||
return "passed"
|
||||
if evaluation["predicted_final"] is None:
|
||||
return "no_numeric_answer"
|
||||
return "wrong_numeric_answer"
|
||||
if not evaluation["contains_entry_point_definition"]:
|
||||
return "extract_failed"
|
||||
if not evaluation["python_ast_parse"]:
|
||||
return "syntax_error"
|
||||
status = evaluation["execution"]["status"]
|
||||
return {
|
||||
"passed": "passed",
|
||||
"timeout": "timeout",
|
||||
"runtime_error": "runtime_error",
|
||||
"assertion_failed": "assertion_failed",
|
||||
"not_run": "not_run",
|
||||
}.get(status, f"execution_{status}")
|
||||
|
||||
|
||||
def success(row: dict[str, Any], *, strict: bool = False) -> int:
|
||||
evaluation = row["task_evaluation"]
|
||||
if row["domain"] == "code":
|
||||
key = (
|
||||
"strict_complete_tests_pass"
|
||||
if strict
|
||||
else "fixed_budget_tests_pass"
|
||||
)
|
||||
else:
|
||||
key = (
|
||||
"strict_complete_numeric_exact"
|
||||
if strict
|
||||
else "fixed_budget_numeric_exact"
|
||||
)
|
||||
return int(evaluation[key])
|
||||
|
||||
|
||||
def compact_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
result = {
|
||||
"outputs": len(rows),
|
||||
"sources": len({row["source_id"] for row in rows}),
|
||||
"tapes": sorted({row["tape_label"] for row in rows}),
|
||||
"natural_eos": sum(row["hit_eos"] for row in rows),
|
||||
"budget_truncated": sum(
|
||||
row["stopped_at_max_new_tokens"] for row in rows
|
||||
),
|
||||
"mean_generated_tokens": (
|
||||
mean(row["generated_tokens"] for row in rows)
|
||||
if rows
|
||||
else None
|
||||
),
|
||||
"fixed_budget_success": sum(success(row) for row in rows),
|
||||
"strict_complete_success": sum(
|
||||
success(row, strict=True) for row in rows
|
||||
),
|
||||
"task_outcomes": dict(
|
||||
sorted(Counter(row["task_outcome"] for row in rows).items())
|
||||
),
|
||||
"completion_classes": dict(
|
||||
sorted(
|
||||
Counter(
|
||||
row["completion_class"] for row in rows
|
||||
).items()
|
||||
)
|
||||
),
|
||||
}
|
||||
result["by_domain"] = {
|
||||
domain: compact_summary_no_recursion(
|
||||
[row for row in rows if row["domain"] == domain]
|
||||
)
|
||||
for domain in DOMAINS
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def compact_summary_no_recursion(
|
||||
rows: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"outputs": len(rows),
|
||||
"sources": len({row["source_id"] for row in rows}),
|
||||
"tapes": sorted({row["tape_label"] for row in rows}),
|
||||
"natural_eos": sum(row["hit_eos"] for row in rows),
|
||||
"budget_truncated": sum(
|
||||
row["stopped_at_max_new_tokens"] for row in rows
|
||||
),
|
||||
"mean_generated_tokens": (
|
||||
mean(row["generated_tokens"] for row in rows)
|
||||
if rows
|
||||
else None
|
||||
),
|
||||
"fixed_budget_success": sum(success(row) for row in rows),
|
||||
"strict_complete_success": sum(
|
||||
success(row, strict=True) for row in rows
|
||||
),
|
||||
"task_outcomes": dict(
|
||||
sorted(Counter(row["task_outcome"] for row in rows).items())
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
for path in (
|
||||
args.sampling_json,
|
||||
args.human_eval,
|
||||
args.gsm8k,
|
||||
):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
if "@sha256:" not in args.sandbox_image:
|
||||
raise ValueError("--sandbox-image must include immutable digest")
|
||||
if args.timeout_seconds <= 0:
|
||||
raise ValueError("--timeout-seconds must be positive")
|
||||
sampling = json.loads(
|
||||
args.sampling_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if sampling["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError("sampling protocol ID differs")
|
||||
if tuple(
|
||||
sampling["seed_contract"]["condition_row_order"]
|
||||
) != CONDITIONS:
|
||||
raise RuntimeError("sampling condition order differs")
|
||||
if sampling["summary"]["outputs"] != 352:
|
||||
raise RuntimeError("formal grid must contain 352 outputs")
|
||||
|
||||
human_eval, gsm8k = completion.load_tasks(
|
||||
args.human_eval,
|
||||
args.gsm8k,
|
||||
)
|
||||
code_cache: dict[
|
||||
tuple[str, str, str],
|
||||
dict[str, Any],
|
||||
] = {}
|
||||
rows = []
|
||||
for source in sampling["sources"]:
|
||||
for run in source["runs"]:
|
||||
for output in run["outputs"]:
|
||||
if source["domain"] == "math":
|
||||
evaluation = completion.evaluate_math(
|
||||
output["text"],
|
||||
output["hit_eos"],
|
||||
gsm8k[source["id"]]["answer"],
|
||||
)
|
||||
cache_hit = False
|
||||
else:
|
||||
evaluation, cache_hit = (
|
||||
shared.evaluate_code_cached(
|
||||
text=output["text"],
|
||||
hit_eos=output["hit_eos"],
|
||||
task=human_eval[source["id"]],
|
||||
image=args.sandbox_image,
|
||||
timeout_seconds=args.timeout_seconds,
|
||||
skip_execution=args.skip_code_execution,
|
||||
cache=code_cache,
|
||||
)
|
||||
)
|
||||
row = {
|
||||
"source_id": source["id"],
|
||||
"domain": source["domain"],
|
||||
"domain_index": source["within_domain_index"],
|
||||
"replicate_index": run["replicate_index"],
|
||||
"replicate_label": run["replicate_label"],
|
||||
"tape_label": run["tape_label"],
|
||||
"base_seed": run["base_seed"],
|
||||
"run_seed": run["run_seed"],
|
||||
"condition": output["condition"],
|
||||
"generated_tokens": output["generated_tokens"],
|
||||
"hit_eos": output["hit_eos"],
|
||||
"stopped_at_max_new_tokens": output[
|
||||
"stopped_at_max_new_tokens"
|
||||
],
|
||||
"prompt_token_ids_sha256": output[
|
||||
"prompt_token_ids_sha256"
|
||||
],
|
||||
"generated_token_ids_sha256": output[
|
||||
"generated_token_ids_sha256"
|
||||
],
|
||||
"text_sha256": output["text_sha256"],
|
||||
"uniform_uint64_prefix_sha256": output[
|
||||
"uniform_uint64_prefix_sha256"
|
||||
],
|
||||
"task_evaluation": evaluation,
|
||||
"completion_class": completion.completion_class(
|
||||
output,
|
||||
evaluation,
|
||||
),
|
||||
"code_execution_cache_hit": cache_hit,
|
||||
}
|
||||
row["task_outcome"] = task_outcome(
|
||||
source["domain"],
|
||||
evaluation,
|
||||
)
|
||||
rows.append(row)
|
||||
|
||||
main_rows = [
|
||||
row for row in rows if row["tape_label"] == "T0"
|
||||
]
|
||||
diagnostic_sources = {
|
||||
source["id"]
|
||||
for source in sampling["sources"]
|
||||
if len(source["runs"]) == 4
|
||||
}
|
||||
diagnostic_rows = [
|
||||
row
|
||||
for row in rows
|
||||
if row["source_id"] in diagnostic_sources
|
||||
]
|
||||
additional_rows = [
|
||||
row for row in rows if row["tape_label"] != "T0"
|
||||
]
|
||||
if (
|
||||
len(main_rows) != 256
|
||||
or len(diagnostic_rows) != 128
|
||||
or len(additional_rows) != 96
|
||||
):
|
||||
raise RuntimeError(
|
||||
"T0 / diagnostic grid counts differ from protocol"
|
||||
)
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"input": {
|
||||
"sampling_path": str(args.sampling_json),
|
||||
"sampling_sha256": completion.sha256_file(
|
||||
args.sampling_json
|
||||
),
|
||||
"sampling_content_hash": sampling["content_hash"],
|
||||
"human_eval_sha256": completion.sha256_file(
|
||||
args.human_eval
|
||||
),
|
||||
"gsm8k_sha256": completion.sha256_file(args.gsm8k),
|
||||
"model_revision": sampling["model"]["revision"],
|
||||
"max_new_tokens": sampling["generation_contract"][
|
||||
"max_new_tokens"
|
||||
],
|
||||
},
|
||||
"sandbox": {
|
||||
"image": args.sandbox_image,
|
||||
"timeout_seconds": args.timeout_seconds,
|
||||
"code_execution_skipped": args.skip_code_execution,
|
||||
"network": "none",
|
||||
"filesystem": "read-only",
|
||||
"user": "65534:65534",
|
||||
"capabilities": "ALL dropped",
|
||||
"no_new_privileges": True,
|
||||
"memory": "256m",
|
||||
"memory_swap": "256m",
|
||||
"pids_limit": 64,
|
||||
"cpus": 0.5,
|
||||
"tmpfs": "/tmp:rw,noexec,nosuid,size=16m",
|
||||
"host_mounts": 0,
|
||||
"unique_code_cache_entries": len(code_cache),
|
||||
},
|
||||
"rows": rows,
|
||||
"summary": {
|
||||
"formal_all_outputs": compact_summary(rows),
|
||||
"main_t0": compact_summary(main_rows),
|
||||
"diagnostic_all_four_tapes": compact_summary(
|
||||
diagnostic_rows
|
||||
),
|
||||
"diagnostic_additional_t1_t3": compact_summary(
|
||||
additional_rows
|
||||
),
|
||||
"main_t0_by_domain_condition": {
|
||||
domain: {
|
||||
condition: compact_summary_no_recursion(
|
||||
[
|
||||
row
|
||||
for row in main_rows
|
||||
if row["domain"] == domain
|
||||
and row["condition"] == condition
|
||||
]
|
||||
)
|
||||
for condition in CONDITIONS
|
||||
}
|
||||
for domain in DOMAINS
|
||||
},
|
||||
},
|
||||
"claim_boundary": [
|
||||
(
|
||||
"Primary summaries use T0 only; T1-T3 are isolated "
|
||||
"multi-tape sensitivity diagnostics."
|
||||
),
|
||||
"HumanEval and GSM8K are never pooled into one ability rate.",
|
||||
(
|
||||
"The frozen 32 tasks per domain are a selected task "
|
||||
"frame, not a full benchmark sample."
|
||||
),
|
||||
"Passing HumanEval tests is functional, not safety, evidence.",
|
||||
"Counterfactual period prompts are not official-valid chats.",
|
||||
],
|
||||
}
|
||||
result["content_hash"] = completion.canonical_hash(rows)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"sha256": completion.sha256_bytes(payload),
|
||||
"bytes": len(payload),
|
||||
"rows": len(rows),
|
||||
"main_t0": result["summary"]["main_t0"],
|
||||
"unique_code_cache_entries": len(code_cache),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,224 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare the preregistered 64-cell Round 08 replay with the formal grid."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-deepseek-chat-task-bootstrap-crn-v1"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--formal-json", type=Path, required=True)
|
||||
parser.add_argument("--rerun-json", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(16 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_hash(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def run_index(
|
||||
payload: dict[str, Any],
|
||||
) -> dict[
|
||||
tuple[str, str, str],
|
||||
tuple[dict[str, Any], dict[str, Any]],
|
||||
]:
|
||||
return {
|
||||
(
|
||||
source["id"],
|
||||
run["tape_label"],
|
||||
output["condition"],
|
||||
): (run, output)
|
||||
for source in payload["sources"]
|
||||
for run in source["runs"]
|
||||
for output in run["outputs"]
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
for path in (args.formal_json, args.rerun_json):
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
formal = json.loads(
|
||||
args.formal_json.read_text(encoding="utf-8")
|
||||
)
|
||||
rerun = json.loads(
|
||||
args.rerun_json.read_text(encoding="utf-8")
|
||||
)
|
||||
if (
|
||||
formal["protocol_id"] != PROTOCOL_ID
|
||||
or rerun["protocol_id"] != PROTOCOL_ID
|
||||
):
|
||||
raise RuntimeError("protocol ID differs")
|
||||
if formal["model"]["revision"] != rerun["model"]["revision"]:
|
||||
raise RuntimeError("model revision differs")
|
||||
if formal["generation_contract"] != rerun["generation_contract"]:
|
||||
raise RuntimeError("generation contract differs")
|
||||
if rerun["summary"]["outputs"] != 64:
|
||||
raise RuntimeError("replay grid must contain 64 outputs")
|
||||
formal_rows = run_index(formal)
|
||||
rerun_rows = run_index(rerun)
|
||||
if any(key[1] != "T0" for key in rerun_rows):
|
||||
raise RuntimeError("replay grid must use T0 only")
|
||||
missing = sorted(set(rerun_rows) - set(formal_rows))
|
||||
if missing:
|
||||
raise RuntimeError(
|
||||
f"{len(missing)} replay cells are absent from formal grid"
|
||||
)
|
||||
|
||||
rows = []
|
||||
for key in sorted(rerun_rows):
|
||||
formal_run, formal_output = formal_rows[key]
|
||||
rerun_run, rerun_output = rerun_rows[key]
|
||||
checks = {
|
||||
"run_seed_exact": (
|
||||
formal_run["run_seed"] == rerun_run["run_seed"]
|
||||
),
|
||||
"prompt_hash_exact": (
|
||||
formal_output["prompt_token_ids_sha256"]
|
||||
== rerun_output["prompt_token_ids_sha256"]
|
||||
),
|
||||
"uniform_run_hash_exact": (
|
||||
formal_run["uniform_uint64_sha256"]
|
||||
== rerun_run["uniform_uint64_sha256"]
|
||||
),
|
||||
"uniform_output_prefix_hash_exact": (
|
||||
formal_output["uniform_uint64_prefix_sha256"]
|
||||
== rerun_output["uniform_uint64_prefix_sha256"]
|
||||
),
|
||||
"uniform_steps_exact": (
|
||||
formal_output["uniform_steps_consumed"]
|
||||
== rerun_output["uniform_steps_consumed"]
|
||||
),
|
||||
"generated_token_ids_exact": (
|
||||
formal_output["generated_token_ids"]
|
||||
== rerun_output["generated_token_ids"]
|
||||
),
|
||||
"decoded_text_exact": (
|
||||
formal_output["text"] == rerun_output["text"]
|
||||
),
|
||||
"eos_state_exact": (
|
||||
formal_output["hit_eos"]
|
||||
== rerun_output["hit_eos"]
|
||||
),
|
||||
"truncation_state_exact": (
|
||||
formal_output["stopped_at_max_new_tokens"]
|
||||
== rerun_output["stopped_at_max_new_tokens"]
|
||||
),
|
||||
"cpu_rng_pre_state_exact": (
|
||||
formal_run["rng_state_before"]["cpu_sha256"]
|
||||
== rerun_run["rng_state_before"]["cpu_sha256"]
|
||||
),
|
||||
"cuda_rng_pre_state_exact": (
|
||||
formal_run["rng_state_before"][
|
||||
"cuda_combined_sha256"
|
||||
]
|
||||
== rerun_run["rng_state_before"][
|
||||
"cuda_combined_sha256"
|
||||
]
|
||||
),
|
||||
"torch_rng_unchanged_exact": (
|
||||
formal_run["torch_rng_unchanged"]
|
||||
and rerun_run["torch_rng_unchanged"]
|
||||
),
|
||||
}
|
||||
rows.append(
|
||||
{
|
||||
"source_id": key[0],
|
||||
"tape_label": key[1],
|
||||
"condition": key[2],
|
||||
**checks,
|
||||
"all_preregistered_fields_exact": all(
|
||||
checks.values()
|
||||
),
|
||||
}
|
||||
)
|
||||
check_names = [
|
||||
key
|
||||
for key in rows[0]
|
||||
if key.endswith("_exact")
|
||||
and key != "all_preregistered_fields_exact"
|
||||
]
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"formal": {
|
||||
"path": str(args.formal_json),
|
||||
"sha256": sha256_file(args.formal_json),
|
||||
"content_hash": formal["content_hash"],
|
||||
},
|
||||
"rerun": {
|
||||
"path": str(args.rerun_json),
|
||||
"sha256": sha256_file(args.rerun_json),
|
||||
"content_hash": rerun["content_hash"],
|
||||
},
|
||||
"rows": rows,
|
||||
"summary": {
|
||||
"cells": len(rows),
|
||||
"all_preregistered_fields_exact": sum(
|
||||
row["all_preregistered_fields_exact"]
|
||||
for row in rows
|
||||
),
|
||||
"by_field": {
|
||||
name: sum(row[name] for row in rows)
|
||||
for name in check_names
|
||||
},
|
||||
},
|
||||
"claim_boundary": [
|
||||
"Only the preregistered 16-source T0 subset is replayed.",
|
||||
(
|
||||
"Exact replay is scoped to the pinned checkpoint, "
|
||||
"software, sampler, precision, and hardware contract."
|
||||
),
|
||||
"Reproduction does not imply tape-invariant trajectories.",
|
||||
],
|
||||
}
|
||||
if result["summary"]["cells"] != 64:
|
||||
raise RuntimeError("reproduction comparison must contain 64 cells")
|
||||
result["content_hash"] = canonical_hash(rows)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
payload = args.output.read_bytes()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"bytes": len(payload),
|
||||
"summary": result["summary"],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,64 @@
|
||||
# Reduced Attention Residuals reproduction
|
||||
|
||||
This directory implements protocol
|
||||
`llm-atlas-k3-attnres-reduced-v1`, frozen in
|
||||
`research/K3_ATTNRES_REDUCED_PROTOCOL.md`.
|
||||
|
||||
The experiment is a reduced independent mechanism probe. It is not a Kimi K3
|
||||
checkpoint forward pass and not a reproduction of the paper-scale training run.
|
||||
|
||||
## Environment
|
||||
|
||||
The pinned execution environment used by this project is:
|
||||
|
||||
```text
|
||||
Python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python
|
||||
PyTorch 2.11.0+cu128
|
||||
GPU NVIDIA GeForce RTX 5090
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8
|
||||
```
|
||||
|
||||
## Build the frozen dataset
|
||||
|
||||
```bash
|
||||
python experiments/k3/attnres/build_dataset.py \
|
||||
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1 \
|
||||
--manifest experiments/k3/attnres/manifest.json
|
||||
```
|
||||
|
||||
## Run one cell
|
||||
|
||||
```bash
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8 \
|
||||
python experiments/k3/attnres/train.py \
|
||||
--architecture baseline \
|
||||
--seed 2026073001 \
|
||||
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1 \
|
||||
--manifest experiments/k3/attnres/manifest.json \
|
||||
--output /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/runs/baseline-2026073001.json
|
||||
```
|
||||
|
||||
Raw parquet and checkpoints stay in the local cache. Frozen manifests, metric
|
||||
JSON, analyses, code, checksums, and a compact website payload enter the public
|
||||
repository.
|
||||
|
||||
## Validate and aggregate the complete study
|
||||
|
||||
After the nine formal cells, the preregistered replay, and the paired smoke runs
|
||||
exist in the cache:
|
||||
|
||||
```bash
|
||||
python experiments/k3/attnres/analyze.py \
|
||||
--formal-dir /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/formal \
|
||||
--smoke-dir /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/smoke \
|
||||
--replay /home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/replay/block-2026073001.json \
|
||||
--manifest experiments/k3/attnres/manifest.json \
|
||||
--output src/data/k3-attnres-reduced.json \
|
||||
--compact-output src/data/k3-attnres-reduced-compact.json \
|
||||
--reproduction-output experiments/k3/attnres/reproduction.json
|
||||
```
|
||||
|
||||
The aggregator fails closed on protocol identity, grid completeness, byte
|
||||
budget, schedule hashes, shared initialization, non-finite metrics, diagnostic
|
||||
shape, smoke mismatch, or formal replay mismatch. Timing is recorded but is not
|
||||
required to replay bit-for-bit.
|
||||
@@ -0,0 +1,536 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate, aggregate, and compact the reduced Attention Residuals study."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-reduced-v1"
|
||||
ARCHITECTURES = ("baseline", "full", "block")
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
REPLAY_FIELDS = (
|
||||
"manifest",
|
||||
"model",
|
||||
"optimizer",
|
||||
"hashes",
|
||||
"evaluations",
|
||||
"training_history",
|
||||
"diagnostic",
|
||||
"environment",
|
||||
)
|
||||
SMOKE_FIELDS = REPLAY_FIELDS
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--formal-dir", type=Path, required=True)
|
||||
parser.add_argument("--smoke-dir", type=Path, required=True)
|
||||
parser.add_argument("--replay", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--compact-output", type=Path, required=True)
|
||||
parser.add_argument("--reproduction-output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def read_json(path: Path) -> dict[str, Any]:
|
||||
return json.loads(path.read_text())
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def write_json(path: Path, value: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def mean(values: Iterable[float]) -> float:
|
||||
return statistics.fmean(values)
|
||||
|
||||
|
||||
def elementwise_summary(rows: list[list[float]]) -> dict[str, list[float]]:
|
||||
length = len(rows[0])
|
||||
if any(len(row) != length for row in rows):
|
||||
raise ValueError("array lengths do not match")
|
||||
return {
|
||||
"mean": [mean(row[index] for row in rows) for index in range(length)],
|
||||
"min": [min(row[index] for row in rows) for index in range(length)],
|
||||
"max": [max(row[index] for row in rows) for index in range(length)],
|
||||
}
|
||||
|
||||
|
||||
def coefficient_of_variation(values: list[float]) -> float:
|
||||
average = mean(values)
|
||||
variance = mean((value - average) ** 2 for value in values)
|
||||
return math.sqrt(variance) / average
|
||||
|
||||
|
||||
def verdict(deltas: list[float]) -> dict[str, Any]:
|
||||
average = mean(deltas)
|
||||
if all(delta < 0 for delta in deltas) and average <= -0.010:
|
||||
label = "directional support in this reduced protocol"
|
||||
elif all(delta > 0 for delta in deltas) and average >= 0.010:
|
||||
label = "directional concern in this reduced protocol"
|
||||
else:
|
||||
label = "inconclusive at this budget"
|
||||
return {
|
||||
"paired_deltas_bpc": deltas,
|
||||
"mean_delta_bpc": average,
|
||||
"min_delta_bpc": min(deltas),
|
||||
"max_delta_bpc": max(deltas),
|
||||
"same_direction": len({delta < 0 for delta in deltas}) == 1,
|
||||
"threshold_bpc": 0.010,
|
||||
"verdict": label,
|
||||
}
|
||||
|
||||
|
||||
def average_depth_weights(runs: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
rows_by_run = [run["diagnostic"]["depth_weights"] for run in runs]
|
||||
layer_count = len(rows_by_run[0])
|
||||
if any(len(rows) != layer_count for rows in rows_by_run):
|
||||
raise ValueError("depth-weight layer counts differ")
|
||||
rows = []
|
||||
max_sources = 0
|
||||
for layer in range(layer_count):
|
||||
source_count = rows_by_run[0][layer]["sources"]
|
||||
if any(rows[layer]["sources"] != source_count for rows in rows_by_run):
|
||||
raise ValueError("source count differs across seeds")
|
||||
weights = [
|
||||
mean(rows_by_run[seed_index][layer]["mean_weights"][source]
|
||||
for seed_index in range(len(runs)))
|
||||
for source in range(source_count)
|
||||
]
|
||||
entropies = [
|
||||
rows_by_run[seed_index][layer]["entropy_mean"]
|
||||
for seed_index in range(len(runs))
|
||||
]
|
||||
rows.append(
|
||||
{
|
||||
"sublayer": layer + 1,
|
||||
"sources": source_count,
|
||||
"mean_weights": weights,
|
||||
"entropy_mean": mean(entropies),
|
||||
"entropy_min": min(entropies),
|
||||
"entropy_max": max(entropies),
|
||||
}
|
||||
)
|
||||
max_sources = max(max_sources, source_count)
|
||||
|
||||
output_rows = [run["diagnostic"]["output_weights"] for run in runs]
|
||||
output_source_count = output_rows[0]["sources"]
|
||||
output_weights = [
|
||||
mean(row["mean_weights"][source] for row in output_rows)
|
||||
for source in range(output_source_count)
|
||||
]
|
||||
return {
|
||||
"rows": rows,
|
||||
"max_sources": max_sources,
|
||||
"output": {
|
||||
"sources": output_source_count,
|
||||
"mean_weights": output_weights,
|
||||
"entropy_mean": mean(row["entropy_mean"] for row in output_rows),
|
||||
"entropy_min": min(row["entropy_mean"] for row in output_rows),
|
||||
"entropy_max": max(row["entropy_mean"] for row in output_rows),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
manifest = read_json(args.manifest)
|
||||
if manifest["protocol_id"] != PROTOCOL_ID:
|
||||
raise ValueError("manifest protocol mismatch")
|
||||
|
||||
runs: dict[tuple[str, int], dict[str, Any]] = {}
|
||||
formal_file_hashes: dict[str, str] = {}
|
||||
for seed in SEEDS:
|
||||
for architecture in ARCHITECTURES:
|
||||
path = args.formal_dir / f"{architecture}-{seed}.json"
|
||||
run = read_json(path)
|
||||
if run["protocol_id"] != PROTOCOL_ID:
|
||||
raise ValueError(f"protocol mismatch: {path}")
|
||||
if run["run_kind"] != "formal":
|
||||
raise ValueError(f"not a formal run: {path}")
|
||||
if run["architecture"] != architecture or run["seed"] != seed:
|
||||
raise ValueError(f"cell identity mismatch: {path}")
|
||||
if run["steps"] != 2000 or run["batch_size"] != 32:
|
||||
raise ValueError(f"formal budget mismatch: {path}")
|
||||
if run["target_bytes_seen"] != 16_384_000:
|
||||
raise ValueError(f"target byte count mismatch: {path}")
|
||||
if run["manifest"]["file_sha256"] != file_sha256(args.manifest):
|
||||
raise ValueError(f"manifest file hash mismatch: {path}")
|
||||
if run["manifest"]["formal_schedule_sha256"] != (
|
||||
manifest["windows"]["formal_schedule_sha256"]
|
||||
):
|
||||
raise ValueError(f"schedule mismatch: {path}")
|
||||
if run["evaluations"][-1]["step"] != 2000:
|
||||
raise ValueError(f"missing final evaluation: {path}")
|
||||
if any(
|
||||
not math.isfinite(value)
|
||||
for evaluation in run["evaluations"]
|
||||
for value in (
|
||||
evaluation["cross_entropy_nats"],
|
||||
evaluation["bits_per_byte"],
|
||||
)
|
||||
):
|
||||
raise ValueError(f"non-finite evaluation: {path}")
|
||||
if len(run["diagnostic"]["layer_input_rms"]) != 32:
|
||||
raise ValueError(f"diagnostic depth mismatch: {path}")
|
||||
if len(run["diagnostic"]["core_parameter_grad_rms_by_block"]) != 16:
|
||||
raise ValueError(f"gradient depth mismatch: {path}")
|
||||
runs[(architecture, seed)] = run
|
||||
formal_file_hashes[path.name] = file_sha256(path)
|
||||
|
||||
common_initial_exact = {}
|
||||
for seed in SEEDS:
|
||||
hashes = {
|
||||
architecture: runs[(architecture, seed)]["hashes"][
|
||||
"initial_common_parameters"
|
||||
]
|
||||
for architecture in ARCHITECTURES
|
||||
}
|
||||
common_initial_exact[str(seed)] = {
|
||||
"hashes": hashes,
|
||||
"exact": len(set(hashes.values())) == 1,
|
||||
}
|
||||
if not common_initial_exact[str(seed)]["exact"]:
|
||||
raise ValueError(f"common initialization mismatch for seed {seed}")
|
||||
|
||||
by_seed = []
|
||||
for seed in SEEDS:
|
||||
values = {
|
||||
architecture: runs[(architecture, seed)]["evaluations"][-1][
|
||||
"bits_per_byte"
|
||||
]
|
||||
for architecture in ARCHITECTURES
|
||||
}
|
||||
by_seed.append(
|
||||
{
|
||||
"seed": seed,
|
||||
"final_bpc": values,
|
||||
"full_minus_baseline": values["full"] - values["baseline"],
|
||||
"block_minus_baseline": values["block"] - values["baseline"],
|
||||
"block_minus_full": values["block"] - values["full"],
|
||||
}
|
||||
)
|
||||
|
||||
final = {
|
||||
"by_seed": by_seed,
|
||||
"means": {
|
||||
architecture: mean(
|
||||
runs[(architecture, seed)]["evaluations"][-1]["bits_per_byte"]
|
||||
for seed in SEEDS
|
||||
)
|
||||
for architecture in ARCHITECTURES
|
||||
},
|
||||
"full_contrast": verdict(
|
||||
[row["full_minus_baseline"] for row in by_seed]
|
||||
),
|
||||
"block_contrast": verdict(
|
||||
[row["block_minus_baseline"] for row in by_seed]
|
||||
),
|
||||
"block_minus_full": {
|
||||
"paired_deltas_bpc": [row["block_minus_full"] for row in by_seed],
|
||||
"mean_delta_bpc": mean(row["block_minus_full"] for row in by_seed),
|
||||
},
|
||||
}
|
||||
|
||||
evaluation_steps = [
|
||||
evaluation["step"] for evaluation in runs[("baseline", SEEDS[0])]["evaluations"]
|
||||
]
|
||||
curves = {}
|
||||
for architecture in ARCHITECTURES:
|
||||
curve = []
|
||||
for index, step in enumerate(evaluation_steps):
|
||||
values = [
|
||||
runs[(architecture, seed)]["evaluations"][index]["bits_per_byte"]
|
||||
for seed in SEEDS
|
||||
]
|
||||
if any(
|
||||
runs[(architecture, seed)]["evaluations"][index]["step"] != step
|
||||
for seed in SEEDS
|
||||
):
|
||||
raise ValueError("evaluation step mismatch")
|
||||
curve.append(
|
||||
{
|
||||
"step": step,
|
||||
"mean_bpc": mean(values),
|
||||
"min_bpc": min(values),
|
||||
"max_bpc": max(values),
|
||||
"by_seed": values,
|
||||
}
|
||||
)
|
||||
curves[architecture] = curve
|
||||
|
||||
timing = {}
|
||||
for architecture in ARCHITECTURES:
|
||||
cells = [runs[(architecture, seed)]["timing"] for seed in SEEDS]
|
||||
timing[architecture] = {
|
||||
"mean_step_ms": mean(cell["mean_ms"] for cell in cells),
|
||||
"median_step_ms": mean(cell["median_ms"] for cell in cells),
|
||||
"p95_step_ms": mean(cell["p95_ms"] for cell in cells),
|
||||
"mean_peak_allocated_bytes": mean(
|
||||
cell["peak_allocated_bytes"] for cell in cells
|
||||
),
|
||||
"mean_peak_reserved_bytes": mean(
|
||||
cell["peak_reserved_bytes"] for cell in cells
|
||||
),
|
||||
"by_seed": cells,
|
||||
}
|
||||
timing["relative_to_baseline"] = {
|
||||
architecture: {
|
||||
"step_time_ratio": timing[architecture]["mean_step_ms"]
|
||||
/ timing["baseline"]["mean_step_ms"],
|
||||
"allocated_memory_ratio": timing[architecture][
|
||||
"mean_peak_allocated_bytes"
|
||||
]
|
||||
/ timing["baseline"]["mean_peak_allocated_bytes"],
|
||||
}
|
||||
for architecture in ("full", "block")
|
||||
}
|
||||
|
||||
parameters = {
|
||||
architecture: runs[(architecture, SEEDS[0])]["model"]["parameters"]
|
||||
for architecture in ARCHITECTURES
|
||||
}
|
||||
parameters["mixer_overhead_fraction_of_baseline"] = (
|
||||
parameters["full"]["mixer"] / parameters["baseline"]["total"]
|
||||
)
|
||||
|
||||
traces = {}
|
||||
gradients = {}
|
||||
for architecture in ARCHITECTURES:
|
||||
architecture_runs = [runs[(architecture, seed)] for seed in SEEDS]
|
||||
traces[architecture] = {
|
||||
key: elementwise_summary(
|
||||
[run["diagnostic"][key] for run in architecture_runs]
|
||||
)
|
||||
for key in (
|
||||
"layer_input_rms",
|
||||
"branch_output_rms",
|
||||
"stream_state_rms",
|
||||
)
|
||||
}
|
||||
gradient_rows = [
|
||||
run["diagnostic"]["core_parameter_grad_rms_by_block"]
|
||||
for run in architecture_runs
|
||||
]
|
||||
gradients[architecture] = {
|
||||
"by_block": elementwise_summary(gradient_rows),
|
||||
"cv_by_seed": [
|
||||
coefficient_of_variation(row) for row in gradient_rows
|
||||
],
|
||||
"mean_cv": mean(coefficient_of_variation(row) for row in gradient_rows),
|
||||
"first_last_ratio_by_seed": [
|
||||
row[0] / row[-1] for row in gradient_rows
|
||||
],
|
||||
"mean_first_last_ratio": mean(row[0] / row[-1] for row in gradient_rows),
|
||||
}
|
||||
|
||||
mixers = {
|
||||
architecture: average_depth_weights(
|
||||
[runs[(architecture, seed)] for seed in SEEDS]
|
||||
)
|
||||
for architecture in ("full", "block")
|
||||
}
|
||||
|
||||
full_branch = traces["full"]["branch_output_rms"]["mean"]
|
||||
largest_index = max(range(len(full_branch)), key=full_branch.__getitem__)
|
||||
# Full output source 0 is the embedding; branch l is source l+1.
|
||||
largest_source_weight = mixers["full"]["output"]["mean_weights"][
|
||||
largest_index + 1
|
||||
]
|
||||
uniform_output_weight = 1 / mixers["full"]["output"]["sources"]
|
||||
posthoc = {
|
||||
"label": "post-hoc descriptive callout; not a preregistered endpoint",
|
||||
"largest_full_branch_sublayer": largest_index + 1,
|
||||
"largest_full_branch_rms": full_branch[largest_index],
|
||||
"corresponding_final_output_weight": largest_source_weight,
|
||||
"uniform_final_output_weight": uniform_output_weight,
|
||||
"weight_over_uniform": largest_source_weight / uniform_output_weight,
|
||||
}
|
||||
|
||||
replay = read_json(args.replay)
|
||||
formal_replay_source = runs[("block", 2026073001)]
|
||||
replay_exact = {
|
||||
field: formal_replay_source[field] == replay[field]
|
||||
for field in REPLAY_FIELDS
|
||||
}
|
||||
if not all(replay_exact.values()):
|
||||
raise ValueError(f"formal replay mismatch: {replay_exact}")
|
||||
|
||||
smoke = {}
|
||||
for architecture in ARCHITECTURES:
|
||||
first_path = args.smoke_dir / f"{architecture}-2026073001-a.json"
|
||||
second_path = args.smoke_dir / f"{architecture}-2026073001-b.json"
|
||||
first = read_json(first_path)
|
||||
second = read_json(second_path)
|
||||
exact = {field: first[field] == second[field] for field in SMOKE_FIELDS}
|
||||
if not all(exact.values()):
|
||||
raise ValueError(f"smoke mismatch for {architecture}: {exact}")
|
||||
smoke[architecture] = {
|
||||
"fields": exact,
|
||||
"all_exact": True,
|
||||
"first_sha256": file_sha256(first_path),
|
||||
"second_sha256": file_sha256(second_path),
|
||||
}
|
||||
|
||||
reproduction = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"manifest_sha256": file_sha256(args.manifest),
|
||||
"formal_files": formal_file_hashes,
|
||||
"common_initial_parameters": common_initial_exact,
|
||||
"smoke": smoke,
|
||||
"formal_replay": {
|
||||
"architecture": "block",
|
||||
"seed": 2026073001,
|
||||
"fields": replay_exact,
|
||||
"all_numeric_and_hash_fields_exact": all(replay_exact.values()),
|
||||
"timing_exact_required": False,
|
||||
"timing_exact_observed": formal_replay_source["timing"] == replay["timing"],
|
||||
"formal_file_sha256": formal_file_hashes[
|
||||
"block-2026073001.json"
|
||||
],
|
||||
"replay_file_sha256": file_sha256(args.replay),
|
||||
},
|
||||
}
|
||||
reproduction["canonical_sha256_without_self"] = canonical_sha256(reproduction)
|
||||
|
||||
analysis = {
|
||||
"final_validation": final,
|
||||
"evaluation_curves": curves,
|
||||
"timing": timing,
|
||||
"parameters": parameters,
|
||||
"traces": traces,
|
||||
"gradients": gradients,
|
||||
"mixers": mixers,
|
||||
"posthoc": posthoc,
|
||||
"interpretation": {
|
||||
"primary": (
|
||||
"Both Full and Block AttnRes satisfy the preregistered "
|
||||
"directional-support rule in this reduced protocol."
|
||||
),
|
||||
"bounded_depth_pattern": (
|
||||
"Block partial-state RMS resets every four residual sublayers; "
|
||||
"the complete 32-point vectors are reported."
|
||||
),
|
||||
"gradient_boundary": (
|
||||
"The preregistered core-parameter gradient RMS is not more "
|
||||
"uniform for AttnRes here; this metric and scale do not reproduce "
|
||||
"the paper's large-model gradient-magnitude result."
|
||||
),
|
||||
"scope": (
|
||||
"Reduced byte-level WikiText-2 mechanism probe; not a K3 "
|
||||
"checkpoint run, paper-scale reproduction, benchmark, or "
|
||||
"same-FLOP comparison."
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
raw = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"manifest": manifest,
|
||||
"provenance": {
|
||||
"manifest_file_sha256": file_sha256(args.manifest),
|
||||
"formal_file_sha256": formal_file_hashes,
|
||||
"reproduction_sha256": reproduction[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
},
|
||||
"formal_runs": [
|
||||
runs[(architecture, seed)]
|
||||
for seed in SEEDS
|
||||
for architecture in ARCHITECTURES
|
||||
],
|
||||
"analysis": analysis,
|
||||
"reproduction": reproduction,
|
||||
}
|
||||
raw["canonical_sha256_without_self"] = canonical_sha256(raw)
|
||||
|
||||
compact = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"dataset": {
|
||||
"repository": manifest["dataset"]["repository"],
|
||||
"revision": manifest["dataset"]["revision"],
|
||||
"train_bytes": manifest["dataset"]["splits"]["train"][
|
||||
"concatenated_bytes"
|
||||
],
|
||||
"schedule_sha256": manifest["windows"]["formal_schedule_sha256"],
|
||||
"validation_sha256": manifest["windows"][
|
||||
"validation_tensor_sha256"
|
||||
],
|
||||
},
|
||||
"grid": {
|
||||
"architectures": list(ARCHITECTURES),
|
||||
"seeds": list(SEEDS),
|
||||
"runs": 9,
|
||||
"steps_per_run": 2000,
|
||||
"target_bytes_per_run": 16_384_000,
|
||||
"target_bytes_total": 9 * 16_384_000,
|
||||
},
|
||||
"final_validation": final,
|
||||
"evaluation_curves": curves,
|
||||
"timing": timing,
|
||||
"parameters": parameters,
|
||||
"traces": traces,
|
||||
"gradients": gradients,
|
||||
"mixers": mixers,
|
||||
"posthoc": posthoc,
|
||||
"interpretation": analysis["interpretation"],
|
||||
"reproduction": reproduction,
|
||||
"source_sha256": raw["canonical_sha256_without_self"],
|
||||
}
|
||||
compact["canonical_sha256_without_self"] = canonical_sha256(compact)
|
||||
|
||||
write_json(args.reproduction_output, reproduction)
|
||||
write_json(args.output, raw)
|
||||
write_json(args.compact_output, compact)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"compact_output": str(args.compact_output),
|
||||
"reproduction_output": str(args.reproduction_output),
|
||||
"raw_sha256": file_sha256(args.output),
|
||||
"compact_sha256": file_sha256(args.compact_output),
|
||||
"reproduction_sha256": file_sha256(args.reproduction_output),
|
||||
"full": final["full_contrast"],
|
||||
"block": final["block_contrast"],
|
||||
"formal_replay": reproduction["formal_replay"],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Download and freeze the byte-level WikiText-2 corpus for the AttnRes study."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pyarrow.parquet as pq
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-reduced-v1"
|
||||
DATASET_REPO = "Salesforce/wikitext"
|
||||
DATASET_REVISION = "b08601e04326c79dfdd32d625aee71d232d685c3"
|
||||
DATASET_VARIANT = "wikitext-2-raw-v1"
|
||||
SPLITS = ("train", "validation", "test")
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
CONTEXT = 256
|
||||
FORMAL_STEPS = 2000
|
||||
FORMAL_BATCH = 32
|
||||
VALIDATION_WINDOWS = 64
|
||||
DIAGNOSTIC_WINDOWS = 16
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--cache-dir", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def bytes_sha256(payload: bytes) -> str:
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def atomic_json(path: Path, value: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def download(url: str, path: Path) -> None:
|
||||
if path.exists():
|
||||
return
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".part")
|
||||
request = urllib.request.Request(
|
||||
url,
|
||||
headers={"User-Agent": "llm-atlas-k3-attnres-reduced/1.0"},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=120) as response:
|
||||
with temporary.open("wb") as output:
|
||||
while block := response.read(1024 * 1024):
|
||||
output.write(block)
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def window_start(label: str, index: int, corpus_length: int, seed: int | None = None) -> int:
|
||||
fields = [PROTOCOL_ID, label]
|
||||
if seed is not None:
|
||||
fields.append(str(seed))
|
||||
fields.append(str(index))
|
||||
payload = "\0".join(fields).encode()
|
||||
value = int.from_bytes(hashlib.sha256(payload).digest()[:8], "big")
|
||||
return value % (corpus_length - (CONTEXT + 1))
|
||||
|
||||
|
||||
def train_window_start(seed: int, step: int, row: int, corpus_length: int) -> int:
|
||||
payload = "\0".join(
|
||||
[PROTOCOL_ID, "train-window", str(seed), str(step), str(row)]
|
||||
).encode()
|
||||
value = int.from_bytes(hashlib.sha256(payload).digest()[:8], "big")
|
||||
return value % (corpus_length - (CONTEXT + 1))
|
||||
|
||||
|
||||
def concatenate_split(parquet_path: Path) -> tuple[bytes, int]:
|
||||
table = pq.read_table(parquet_path, columns=["text"])
|
||||
rows = table.column("text").to_pylist()
|
||||
payload = b"".join(((row or "") + "\n").encode("utf-8") for row in rows)
|
||||
return payload, len(rows)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
args.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
split_manifest: dict[str, Any] = {}
|
||||
split_bytes: dict[str, bytes] = {}
|
||||
for split in SPLITS:
|
||||
relative = f"{DATASET_VARIANT}/{split}-00000-of-00001.parquet"
|
||||
url = (
|
||||
f"https://huggingface.co/datasets/{DATASET_REPO}/resolve/"
|
||||
f"{DATASET_REVISION}/{relative}"
|
||||
)
|
||||
parquet_path = args.cache_dir / f"{split}.parquet"
|
||||
download(url, parquet_path)
|
||||
payload, rows = concatenate_split(parquet_path)
|
||||
binary_path = args.cache_dir / f"{split}.bin"
|
||||
if not binary_path.exists() or binary_path.read_bytes() != payload:
|
||||
temporary = binary_path.with_suffix(".bin.tmp")
|
||||
temporary.write_bytes(payload)
|
||||
os.replace(temporary, binary_path)
|
||||
split_bytes[split] = payload
|
||||
split_manifest[split] = {
|
||||
"source_path": relative,
|
||||
"source_url": url,
|
||||
"parquet_bytes": parquet_path.stat().st_size,
|
||||
"parquet_sha256": file_sha256(parquet_path),
|
||||
"rows": rows,
|
||||
"concatenated_bytes": len(payload),
|
||||
"concatenated_sha256": bytes_sha256(payload),
|
||||
"binary_path": str(binary_path),
|
||||
"binary_sha256": file_sha256(binary_path),
|
||||
}
|
||||
|
||||
train = split_bytes["train"]
|
||||
validation = split_bytes["validation"]
|
||||
|
||||
schedule_digest = hashlib.sha256()
|
||||
schedule_cells = 0
|
||||
for seed in SEEDS:
|
||||
for step in range(1, FORMAL_STEPS + 1):
|
||||
for row in range(FORMAL_BATCH):
|
||||
start = train_window_start(seed, step, row, len(train))
|
||||
schedule_digest.update(start.to_bytes(8, "big"))
|
||||
schedule_cells += 1
|
||||
|
||||
validation_starts = [
|
||||
window_start("validation-window", index, len(validation))
|
||||
for index in range(VALIDATION_WINDOWS)
|
||||
]
|
||||
diagnostic_starts = [
|
||||
window_start("diagnostic-window", index, len(validation))
|
||||
for index in range(DIAGNOSTIC_WINDOWS)
|
||||
]
|
||||
|
||||
def tensor_hash(starts: list[int]) -> str:
|
||||
digest = hashlib.sha256()
|
||||
for start in starts:
|
||||
digest.update(validation[start : start + CONTEXT + 1])
|
||||
return digest.hexdigest()
|
||||
|
||||
manifest = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"status": "frozen-before-model-output",
|
||||
"dataset": {
|
||||
"repository": DATASET_REPO,
|
||||
"revision": DATASET_REVISION,
|
||||
"variant": DATASET_VARIANT,
|
||||
"preprocessing": (
|
||||
"parquet row order; (text or empty string) + LF; UTF-8; "
|
||||
"no normalization; vocabulary is raw bytes 0..255"
|
||||
),
|
||||
"splits": split_manifest,
|
||||
},
|
||||
"windows": {
|
||||
"context": CONTEXT,
|
||||
"target_bytes_per_window": CONTEXT,
|
||||
"seeds": list(SEEDS),
|
||||
"formal_steps": FORMAL_STEPS,
|
||||
"formal_batch": FORMAL_BATCH,
|
||||
"formal_schedule_cells": schedule_cells,
|
||||
"formal_schedule_sha256": schedule_digest.hexdigest(),
|
||||
"validation_starts": validation_starts,
|
||||
"validation_tensor_sha256": tensor_hash(validation_starts),
|
||||
"diagnostic_starts": diagnostic_starts,
|
||||
"diagnostic_tensor_sha256": tensor_hash(diagnostic_starts),
|
||||
},
|
||||
}
|
||||
atomic_json(args.manifest, manifest)
|
||||
print(json.dumps(manifest, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,145 @@
|
||||
{
|
||||
"dataset": {
|
||||
"preprocessing": "parquet row order; (text or empty string) + LF; UTF-8; no normalization; vocabulary is raw bytes 0..255",
|
||||
"repository": "Salesforce/wikitext",
|
||||
"revision": "b08601e04326c79dfdd32d625aee71d232d685c3",
|
||||
"splits": {
|
||||
"test": {
|
||||
"binary_path": "/home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/test.bin",
|
||||
"binary_sha256": "bfe9eb16ab9987fb88bde4ea9a30a00f2a45db01dfc14bad78d05325789c4f12",
|
||||
"concatenated_bytes": 1292014,
|
||||
"concatenated_sha256": "bfe9eb16ab9987fb88bde4ea9a30a00f2a45db01dfc14bad78d05325789c4f12",
|
||||
"parquet_bytes": 732610,
|
||||
"parquet_sha256": "5f1bea067869d04849c0f975a2b29c4ff47d867f484f5010ea5e861eab246d91",
|
||||
"rows": 4358,
|
||||
"source_path": "wikitext-2-raw-v1/test-00000-of-00001.parquet",
|
||||
"source_url": "https://huggingface.co/datasets/Salesforce/wikitext/resolve/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-2-raw-v1/test-00000-of-00001.parquet"
|
||||
},
|
||||
"train": {
|
||||
"binary_path": "/home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/train.bin",
|
||||
"binary_sha256": "0ca7d3e74dbe44564ea5942b85232f1bbcb525c9cd481cd5d28a87ee90e7e9b4",
|
||||
"concatenated_bytes": 10951563,
|
||||
"concatenated_sha256": "0ca7d3e74dbe44564ea5942b85232f1bbcb525c9cd481cd5d28a87ee90e7e9b4",
|
||||
"parquet_bytes": 6357543,
|
||||
"parquet_sha256": "e83889baabc497075506f91975be5fac0d45c5290b6b20582c8cd1e853d0c9f7",
|
||||
"rows": 36718,
|
||||
"source_path": "wikitext-2-raw-v1/train-00000-of-00001.parquet",
|
||||
"source_url": "https://huggingface.co/datasets/Salesforce/wikitext/resolve/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-2-raw-v1/train-00000-of-00001.parquet"
|
||||
},
|
||||
"validation": {
|
||||
"binary_path": "/home/wuyang/.cache/llm-atlas/k3-attnres-reduced-v1/validation.bin",
|
||||
"binary_sha256": "a42356f6a8ff1d25daf25ec9db49e10a537c265581b61c74604bb63231dee719",
|
||||
"concatenated_bytes": 1148008,
|
||||
"concatenated_sha256": "a42356f6a8ff1d25daf25ec9db49e10a537c265581b61c74604bb63231dee719",
|
||||
"parquet_bytes": 657209,
|
||||
"parquet_sha256": "204929b7ff9d6184953f867dedb860e40aa69c078fc1e54b3baaa8fb28511c4c",
|
||||
"rows": 3760,
|
||||
"source_path": "wikitext-2-raw-v1/validation-00000-of-00001.parquet",
|
||||
"source_url": "https://huggingface.co/datasets/Salesforce/wikitext/resolve/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-2-raw-v1/validation-00000-of-00001.parquet"
|
||||
}
|
||||
},
|
||||
"variant": "wikitext-2-raw-v1"
|
||||
},
|
||||
"protocol_id": "llm-atlas-k3-attnres-reduced-v1",
|
||||
"schema_version": 1,
|
||||
"status": "frozen-before-model-output",
|
||||
"windows": {
|
||||
"context": 256,
|
||||
"diagnostic_starts": [
|
||||
611936,
|
||||
81370,
|
||||
284950,
|
||||
884010,
|
||||
436549,
|
||||
319425,
|
||||
664210,
|
||||
2929,
|
||||
1120568,
|
||||
1130204,
|
||||
567111,
|
||||
471404,
|
||||
178798,
|
||||
773999,
|
||||
8739,
|
||||
567335
|
||||
],
|
||||
"diagnostic_tensor_sha256": "d970af9b0c656c9826f369b5fe6e3869a6f6cfeccfa5a922fe94ed1d24b86818",
|
||||
"formal_batch": 32,
|
||||
"formal_schedule_cells": 192000,
|
||||
"formal_schedule_sha256": "81521a70ec61f3717968f160cb711e50c5f52a665a6961538d339360cb695f48",
|
||||
"formal_steps": 2000,
|
||||
"seeds": [
|
||||
2026073001,
|
||||
2026073002,
|
||||
2026073003
|
||||
],
|
||||
"target_bytes_per_window": 256,
|
||||
"validation_starts": [
|
||||
990074,
|
||||
30961,
|
||||
68731,
|
||||
926982,
|
||||
644250,
|
||||
11726,
|
||||
878377,
|
||||
825572,
|
||||
874166,
|
||||
182830,
|
||||
392129,
|
||||
822765,
|
||||
583693,
|
||||
287605,
|
||||
117755,
|
||||
463343,
|
||||
1005482,
|
||||
27886,
|
||||
1107742,
|
||||
1025132,
|
||||
190867,
|
||||
563499,
|
||||
375410,
|
||||
611922,
|
||||
576979,
|
||||
959290,
|
||||
139748,
|
||||
855267,
|
||||
439281,
|
||||
268378,
|
||||
205337,
|
||||
638978,
|
||||
916468,
|
||||
1059268,
|
||||
1098648,
|
||||
863779,
|
||||
1028257,
|
||||
340396,
|
||||
547037,
|
||||
682617,
|
||||
217662,
|
||||
744004,
|
||||
288924,
|
||||
890860,
|
||||
856857,
|
||||
312424,
|
||||
12454,
|
||||
1135148,
|
||||
385677,
|
||||
954097,
|
||||
490766,
|
||||
164873,
|
||||
620587,
|
||||
329982,
|
||||
1014567,
|
||||
236028,
|
||||
356220,
|
||||
603142,
|
||||
879272,
|
||||
511832,
|
||||
472270,
|
||||
823189,
|
||||
653568,
|
||||
159430
|
||||
],
|
||||
"validation_tensor_sha256": "5f71fda757fc75010ed16e7636bc394c69f55b34a3713b3b5a7ef8e03eae3c20"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
{
|
||||
"canonical_sha256_without_self": "89a127e625ccefbd9b749e6ffca4ba72868e217e0c7a24773024bd2fafde427a",
|
||||
"common_initial_parameters": {
|
||||
"2026073001": {
|
||||
"exact": true,
|
||||
"hashes": {
|
||||
"baseline": "af2724a1c34bcfd61d8a8bef402246430898c815e56b6e5c5257949a4eb0e7b1",
|
||||
"block": "af2724a1c34bcfd61d8a8bef402246430898c815e56b6e5c5257949a4eb0e7b1",
|
||||
"full": "af2724a1c34bcfd61d8a8bef402246430898c815e56b6e5c5257949a4eb0e7b1"
|
||||
}
|
||||
},
|
||||
"2026073002": {
|
||||
"exact": true,
|
||||
"hashes": {
|
||||
"baseline": "9fd4f04212ab5cea822f469902d8e80ecc368da329f3c20abacfe6b7a50ed523",
|
||||
"block": "9fd4f04212ab5cea822f469902d8e80ecc368da329f3c20abacfe6b7a50ed523",
|
||||
"full": "9fd4f04212ab5cea822f469902d8e80ecc368da329f3c20abacfe6b7a50ed523"
|
||||
}
|
||||
},
|
||||
"2026073003": {
|
||||
"exact": true,
|
||||
"hashes": {
|
||||
"baseline": "7a565cd353efdb3b95e9b8b1844581082c029991ea18d438b4b18566970c8c61",
|
||||
"block": "7a565cd353efdb3b95e9b8b1844581082c029991ea18d438b4b18566970c8c61",
|
||||
"full": "7a565cd353efdb3b95e9b8b1844581082c029991ea18d438b4b18566970c8c61"
|
||||
}
|
||||
}
|
||||
},
|
||||
"formal_files": {
|
||||
"baseline-2026073001.json": "f5cddecace6a70ee3824f272811d2a6336ab6dc24ad1100b18292f4413692e9a",
|
||||
"baseline-2026073002.json": "4c3b9e1ea2ed6fd3078f1212984327364832a1146bfcbdb3a6f2e61b589bfaa8",
|
||||
"baseline-2026073003.json": "e5ce5a8944dad0c4811f8bf19b4cfec8ce4a9283858fd59cd739dbc37626707b",
|
||||
"block-2026073001.json": "5df870369d9a86ccb4ba4191fbd1d6f3642893dd47a60f8f6d1143006bdfbdaf",
|
||||
"block-2026073002.json": "be1f109a630e27c8469438e33b53806cdaecac2f712af4f885da83f00ef6843b",
|
||||
"block-2026073003.json": "949ec51ca3a219a11f260171da01f737296241536f5e139cf07d444c27efba92",
|
||||
"full-2026073001.json": "648cb25ea98da1868779733da155275e9a16aad5efbbeeced7812c19d75817d5",
|
||||
"full-2026073002.json": "1fa715bbd81a3a04a5aa0ba0fc27feb8883c44a01e9f28e281a309a394447979",
|
||||
"full-2026073003.json": "4e34fa35bbfec460cabc94bce7a08f44fc188d908705ae2e256500f00813b4e3"
|
||||
},
|
||||
"formal_replay": {
|
||||
"all_numeric_and_hash_fields_exact": true,
|
||||
"architecture": "block",
|
||||
"fields": {
|
||||
"diagnostic": true,
|
||||
"environment": true,
|
||||
"evaluations": true,
|
||||
"hashes": true,
|
||||
"manifest": true,
|
||||
"model": true,
|
||||
"optimizer": true,
|
||||
"training_history": true
|
||||
},
|
||||
"formal_file_sha256": "5df870369d9a86ccb4ba4191fbd1d6f3642893dd47a60f8f6d1143006bdfbdaf",
|
||||
"replay_file_sha256": "e74d3323e5fe31378bb8aad7a8efa2fb91995cd7224c158cf03006466cdea2a7",
|
||||
"seed": 2026073001,
|
||||
"timing_exact_observed": false,
|
||||
"timing_exact_required": false
|
||||
},
|
||||
"manifest_sha256": "9778ade5b1c9dd7676d2cdc52b4e4e7ff5ae513cb56c667974e2422702f9dc2b",
|
||||
"protocol_id": "llm-atlas-k3-attnres-reduced-v1",
|
||||
"schema_version": 1,
|
||||
"smoke": {
|
||||
"baseline": {
|
||||
"all_exact": true,
|
||||
"fields": {
|
||||
"diagnostic": true,
|
||||
"environment": true,
|
||||
"evaluations": true,
|
||||
"hashes": true,
|
||||
"manifest": true,
|
||||
"model": true,
|
||||
"optimizer": true,
|
||||
"training_history": true
|
||||
},
|
||||
"first_sha256": "fd8b14f70a6a14978e65f9899b694164ba91753bf82eead436a40837c251e82c",
|
||||
"second_sha256": "fd8b14f70a6a14978e65f9899b694164ba91753bf82eead436a40837c251e82c"
|
||||
},
|
||||
"block": {
|
||||
"all_exact": true,
|
||||
"fields": {
|
||||
"diagnostic": true,
|
||||
"environment": true,
|
||||
"evaluations": true,
|
||||
"hashes": true,
|
||||
"manifest": true,
|
||||
"model": true,
|
||||
"optimizer": true,
|
||||
"training_history": true
|
||||
},
|
||||
"first_sha256": "980a4a534e458199d95b5864b008a81f51b565e05a7b2f24a644b36d2134eccc",
|
||||
"second_sha256": "980a4a534e458199d95b5864b008a81f51b565e05a7b2f24a644b36d2134eccc"
|
||||
},
|
||||
"full": {
|
||||
"all_exact": true,
|
||||
"fields": {
|
||||
"diagnostic": true,
|
||||
"environment": true,
|
||||
"evaluations": true,
|
||||
"hashes": true,
|
||||
"manifest": true,
|
||||
"model": true,
|
||||
"optimizer": true,
|
||||
"training_history": true
|
||||
},
|
||||
"first_sha256": "c6e32b737c7bf7fdc9b647650ebd7c9d2f9f8550f66a5b84f168f16fa7d2eacb",
|
||||
"second_sha256": "c6e32b737c7bf7fdc9b647650ebd7c9d2f9f8550f66a5b84f168f16fa7d2eacb"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,717 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Train one frozen residual variant for the reduced Attention Residuals study."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import statistics
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-reduced-v1"
|
||||
ARCHITECTURES = ("baseline", "full", "block")
|
||||
EXPECTED_SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
EVAL_STEPS = (0, 100, 250, 500, 1000, 1500, 2000)
|
||||
VOCABULARY = 256
|
||||
CONTEXT = 256
|
||||
LAYERS = 16
|
||||
SUBLAYERS = LAYERS * 2
|
||||
BLOCKS = 8
|
||||
SUBLAYERS_PER_BLOCK = SUBLAYERS // BLOCKS
|
||||
D_MODEL = 192
|
||||
HEADS = 6
|
||||
D_HEAD = D_MODEL // HEADS
|
||||
D_FF = 768
|
||||
RMS_EPS = 1e-6
|
||||
PEAK_LR = 3e-4
|
||||
MIN_LR = 3e-5
|
||||
WARMUP_STEPS = 100
|
||||
WEIGHT_DECAY = 0.1
|
||||
BETAS = (0.9, 0.95)
|
||||
ADAM_EPS = 1e-8
|
||||
GRAD_CLIP = 1.0
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--architecture", choices=ARCHITECTURES, required=True)
|
||||
parser.add_argument("--seed", type=int, required=True)
|
||||
parser.add_argument("--steps", type=int, default=2000)
|
||||
parser.add_argument("--batch-size", type=int, default=32)
|
||||
parser.add_argument("--cache-dir", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--validation-windows", type=int, default=64)
|
||||
parser.add_argument("--diagnostic-windows", type=int, default=16)
|
||||
parser.add_argument("--eval-batch-size", type=int, default=8)
|
||||
parser.add_argument("--timing-warmup", type=int, default=20)
|
||||
parser.add_argument("--run-kind", choices=("smoke", "formal", "replay"), default="formal")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def configure_determinism(seed: int) -> None:
|
||||
if os.environ.get("CUBLAS_WORKSPACE_CONFIG") != ":4096:8":
|
||||
raise RuntimeError("CUBLAS_WORKSPACE_CONFIG must be :4096:8 before Python starts")
|
||||
torch.manual_seed(seed)
|
||||
torch.cuda.manual_seed_all(seed)
|
||||
torch.use_deterministic_algorithms(True)
|
||||
torch.backends.cudnn.benchmark = False
|
||||
torch.backends.cudnn.deterministic = True
|
||||
torch.backends.cuda.matmul.allow_tf32 = False
|
||||
torch.backends.cudnn.allow_tf32 = False
|
||||
torch.set_float32_matmul_precision("highest")
|
||||
|
||||
|
||||
def canonical_json_sha256(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def tensor_bytes(tensor: torch.Tensor) -> bytes:
|
||||
value = tensor.detach().cpu().contiguous()
|
||||
header = f"{value.dtype}|{tuple(value.shape)}|".encode()
|
||||
return header + value.view(torch.uint8).numpy().tobytes()
|
||||
|
||||
|
||||
def state_hash(
|
||||
model: nn.Module,
|
||||
*,
|
||||
include_mixers: bool | None,
|
||||
) -> str:
|
||||
digest = hashlib.sha256()
|
||||
for name, tensor in sorted(model.state_dict().items()):
|
||||
is_mixer = name.startswith("mixers.") or name.startswith("output_mixer.")
|
||||
if include_mixers is not None and is_mixer != include_mixers:
|
||||
continue
|
||||
digest.update(name.encode())
|
||||
digest.update(b"\0")
|
||||
digest.update(tensor_bytes(tensor))
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def window_start(seed: int, step: int, row: int, corpus_length: int) -> int:
|
||||
payload = "\0".join(
|
||||
[PROTOCOL_ID, "train-window", str(seed), str(step), str(row)]
|
||||
).encode()
|
||||
value = int.from_bytes(hashlib.sha256(payload).digest()[:8], "big")
|
||||
return value % (corpus_length - (CONTEXT + 1))
|
||||
|
||||
|
||||
class ByteCorpus:
|
||||
def __init__(self, cache_dir: Path, manifest: dict[str, Any], device: torch.device):
|
||||
self.device = device
|
||||
self.train = np.memmap(cache_dir / "train.bin", dtype=np.uint8, mode="r")
|
||||
self.validation = np.memmap(
|
||||
cache_dir / "validation.bin", dtype=np.uint8, mode="r"
|
||||
)
|
||||
self.validation_starts = manifest["windows"]["validation_starts"]
|
||||
self.diagnostic_starts = manifest["windows"]["diagnostic_starts"]
|
||||
|
||||
def training_batch(
|
||||
self, seed: int, step: int, batch_size: int
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
rows = np.empty((batch_size, CONTEXT + 1), dtype=np.int64)
|
||||
for row in range(batch_size):
|
||||
start = window_start(seed, step, row, len(self.train))
|
||||
rows[row] = self.train[start : start + CONTEXT + 1]
|
||||
tensor = torch.from_numpy(rows).to(self.device, non_blocking=False)
|
||||
return tensor[:, :-1], tensor[:, 1:]
|
||||
|
||||
def fixed_batch(
|
||||
self, starts: list[int], begin: int, end: int
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
chosen = starts[begin:end]
|
||||
rows = np.empty((len(chosen), CONTEXT + 1), dtype=np.int64)
|
||||
for row, start in enumerate(chosen):
|
||||
rows[row] = self.validation[start : start + CONTEXT + 1]
|
||||
tensor = torch.from_numpy(rows).to(self.device, non_blocking=False)
|
||||
return tensor[:, :-1], tensor[:, 1:]
|
||||
|
||||
|
||||
class RMSNorm(nn.Module):
|
||||
def __init__(self, dimension: int):
|
||||
super().__init__()
|
||||
self.weight = nn.Parameter(torch.ones(dimension))
|
||||
|
||||
def forward(self, value: torch.Tensor) -> torch.Tensor:
|
||||
normalized = value.float() * torch.rsqrt(
|
||||
value.float().square().mean(dim=-1, keepdim=True) + RMS_EPS
|
||||
)
|
||||
return normalized.to(value.dtype) * self.weight
|
||||
|
||||
|
||||
class CausalAttention(nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.qkv = nn.Linear(D_MODEL, 3 * D_MODEL, bias=False)
|
||||
self.o_proj = nn.Linear(D_MODEL, D_MODEL, bias=False)
|
||||
mask = torch.triu(torch.ones(CONTEXT, CONTEXT, dtype=torch.bool), diagonal=1)
|
||||
self.register_buffer("causal_mask", mask, persistent=False)
|
||||
|
||||
def forward(self, value: torch.Tensor) -> torch.Tensor:
|
||||
batch, sequence, _ = value.shape
|
||||
qkv = self.qkv(value).view(batch, sequence, 3, HEADS, D_HEAD)
|
||||
query, key, content = qkv.unbind(dim=2)
|
||||
query = query.transpose(1, 2)
|
||||
key = key.transpose(1, 2)
|
||||
content = content.transpose(1, 2)
|
||||
scores = torch.matmul(query, key.transpose(-1, -2)).float() / math.sqrt(D_HEAD)
|
||||
scores = scores.masked_fill(
|
||||
self.causal_mask[:sequence, :sequence], float("-inf")
|
||||
)
|
||||
probabilities = torch.softmax(scores, dim=-1).to(query.dtype)
|
||||
mixed = torch.matmul(probabilities, content)
|
||||
mixed = mixed.transpose(1, 2).contiguous().view(batch, sequence, D_MODEL)
|
||||
return self.o_proj(mixed)
|
||||
|
||||
|
||||
class SwiGLU(nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.gate = nn.Linear(D_MODEL, D_FF, bias=False)
|
||||
self.up = nn.Linear(D_MODEL, D_FF, bias=False)
|
||||
self.down = nn.Linear(D_FF, D_MODEL, bias=False)
|
||||
|
||||
def forward(self, value: torch.Tensor) -> torch.Tensor:
|
||||
return self.down(F.silu(self.gate(value)) * self.up(value))
|
||||
|
||||
|
||||
class TransformerBlock(nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.attention_norm = RMSNorm(D_MODEL)
|
||||
self.attention = CausalAttention()
|
||||
self.mlp_norm = RMSNorm(D_MODEL)
|
||||
self.mlp = SwiGLU()
|
||||
|
||||
|
||||
class DepthMixer(nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.query = nn.Parameter(torch.zeros(D_MODEL))
|
||||
self.key_norm = RMSNorm(D_MODEL)
|
||||
|
||||
def forward(
|
||||
self, sources: list[torch.Tensor], capture: bool = False
|
||||
) -> tuple[torch.Tensor, dict[str, Any] | None]:
|
||||
values = torch.stack(sources, dim=0)
|
||||
keys = self.key_norm(values)
|
||||
logits = torch.einsum("d,nbtd->nbt", self.query, keys.float())
|
||||
weights = torch.softmax(logits, dim=0)
|
||||
output = torch.einsum("nbt,nbtd->btd", weights, values.float()).to(
|
||||
values.dtype
|
||||
)
|
||||
if not capture:
|
||||
return output, None
|
||||
entropy = -(weights * torch.log(weights.clamp_min(1e-30))).sum(dim=0)
|
||||
return output, {
|
||||
"mean_weights": weights.mean(dim=(1, 2)).detach().cpu().tolist(),
|
||||
"entropy_mean": entropy.mean().detach().cpu().item(),
|
||||
"sources": len(sources),
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class TraceAccumulator:
|
||||
layer_input_rms: list[float]
|
||||
branch_output_rms: list[float]
|
||||
stream_state_rms: list[float]
|
||||
depth_weights: list[dict[str, Any]]
|
||||
output_weights: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def rms(value: torch.Tensor) -> float:
|
||||
return value.float().square().mean().sqrt().detach().cpu().item()
|
||||
|
||||
|
||||
class ReducedLanguageModel(nn.Module):
|
||||
def __init__(self, architecture: str):
|
||||
super().__init__()
|
||||
self.architecture = architecture
|
||||
self.token_embedding = nn.Embedding(VOCABULARY, D_MODEL)
|
||||
self.position_embedding = nn.Embedding(CONTEXT, D_MODEL)
|
||||
self.blocks = nn.ModuleList([TransformerBlock() for _ in range(LAYERS)])
|
||||
self.final_norm = RMSNorm(D_MODEL)
|
||||
if architecture == "baseline":
|
||||
self.mixers = nn.ModuleList()
|
||||
self.output_mixer = None
|
||||
else:
|
||||
self.mixers = nn.ModuleList([DepthMixer() for _ in range(SUBLAYERS)])
|
||||
self.output_mixer = DepthMixer()
|
||||
self.reset_parameters()
|
||||
|
||||
def reset_parameters(self) -> None:
|
||||
for module in self.modules():
|
||||
if isinstance(module, nn.Embedding):
|
||||
nn.init.normal_(module.weight, mean=0.0, std=0.02)
|
||||
elif isinstance(module, nn.Linear):
|
||||
nn.init.normal_(module.weight, mean=0.0, std=0.02)
|
||||
elif isinstance(module, RMSNorm):
|
||||
nn.init.ones_(module.weight)
|
||||
scaled = 0.02 / math.sqrt(2 * LAYERS)
|
||||
for block in self.blocks:
|
||||
nn.init.normal_(block.attention.o_proj.weight, mean=0.0, std=scaled)
|
||||
nn.init.normal_(block.mlp.down.weight, mean=0.0, std=scaled)
|
||||
for mixer in self.mixers:
|
||||
nn.init.zeros_(mixer.query)
|
||||
nn.init.ones_(mixer.key_norm.weight)
|
||||
if self.output_mixer is not None:
|
||||
nn.init.zeros_(self.output_mixer.query)
|
||||
nn.init.ones_(self.output_mixer.key_norm.weight)
|
||||
|
||||
def embed(self, input_ids: torch.Tensor) -> torch.Tensor:
|
||||
positions = torch.arange(input_ids.shape[1], device=input_ids.device)
|
||||
return self.token_embedding(input_ids) + self.position_embedding(positions)[None]
|
||||
|
||||
def forward(
|
||||
self, input_ids: torch.Tensor, capture: bool = False
|
||||
) -> tuple[torch.Tensor, TraceAccumulator | None]:
|
||||
embedded = self.embed(input_ids)
|
||||
trace = (
|
||||
TraceAccumulator([], [], [], [])
|
||||
if capture
|
||||
else None
|
||||
)
|
||||
|
||||
if self.architecture == "baseline":
|
||||
hidden = embedded
|
||||
for block in self.blocks:
|
||||
attention_input = hidden
|
||||
attention_output = block.attention(block.attention_norm(attention_input))
|
||||
hidden = hidden + attention_output
|
||||
if trace is not None:
|
||||
trace.layer_input_rms.append(rms(attention_input))
|
||||
trace.branch_output_rms.append(rms(attention_output))
|
||||
trace.stream_state_rms.append(rms(hidden))
|
||||
mlp_input = hidden
|
||||
mlp_output = block.mlp(block.mlp_norm(mlp_input))
|
||||
hidden = hidden + mlp_output
|
||||
if trace is not None:
|
||||
trace.layer_input_rms.append(rms(mlp_input))
|
||||
trace.branch_output_rms.append(rms(mlp_output))
|
||||
trace.stream_state_rms.append(rms(hidden))
|
||||
elif self.architecture == "full":
|
||||
sources = [embedded]
|
||||
mixer_index = 0
|
||||
for block in self.blocks:
|
||||
attention_input, weights = self.mixers[mixer_index](sources, capture)
|
||||
mixer_index += 1
|
||||
attention_output = block.attention(block.attention_norm(attention_input))
|
||||
sources.append(attention_output)
|
||||
if trace is not None:
|
||||
trace.layer_input_rms.append(rms(attention_input))
|
||||
trace.branch_output_rms.append(rms(attention_output))
|
||||
trace.stream_state_rms.append(
|
||||
rms(torch.stack(sources, dim=0))
|
||||
)
|
||||
trace.depth_weights.append(weights or {})
|
||||
mlp_input, weights = self.mixers[mixer_index](sources, capture)
|
||||
mixer_index += 1
|
||||
mlp_output = block.mlp(block.mlp_norm(mlp_input))
|
||||
sources.append(mlp_output)
|
||||
if trace is not None:
|
||||
trace.layer_input_rms.append(rms(mlp_input))
|
||||
trace.branch_output_rms.append(rms(mlp_output))
|
||||
trace.stream_state_rms.append(
|
||||
rms(torch.stack(sources, dim=0))
|
||||
)
|
||||
trace.depth_weights.append(weights or {})
|
||||
assert self.output_mixer is not None
|
||||
hidden, output_weights = self.output_mixer(sources, capture)
|
||||
if trace is not None:
|
||||
trace.output_weights = output_weights
|
||||
else:
|
||||
completed = [embedded]
|
||||
partial: torch.Tensor | None = None
|
||||
mixer_index = 0
|
||||
for block in self.blocks:
|
||||
for branch_index in range(2):
|
||||
sources = completed + ([] if partial is None else [partial])
|
||||
branch_input, weights = self.mixers[mixer_index](sources, capture)
|
||||
mixer_index += 1
|
||||
if branch_index == 0:
|
||||
branch_output = block.attention(
|
||||
block.attention_norm(branch_input)
|
||||
)
|
||||
else:
|
||||
branch_output = block.mlp(block.mlp_norm(branch_input))
|
||||
partial = (
|
||||
branch_output if partial is None else partial + branch_output
|
||||
)
|
||||
if trace is not None:
|
||||
trace.layer_input_rms.append(rms(branch_input))
|
||||
trace.branch_output_rms.append(rms(branch_output))
|
||||
trace.stream_state_rms.append(rms(partial))
|
||||
trace.depth_weights.append(weights or {})
|
||||
if mixer_index % SUBLAYERS_PER_BLOCK == 0:
|
||||
completed.append(partial)
|
||||
partial = None
|
||||
assert partial is None
|
||||
assert len(completed) == BLOCKS + 1
|
||||
assert self.output_mixer is not None
|
||||
hidden, output_weights = self.output_mixer(completed, capture)
|
||||
if trace is not None:
|
||||
trace.output_weights = output_weights
|
||||
|
||||
normalized = self.final_norm(hidden)
|
||||
logits = F.linear(normalized, self.token_embedding.weight)
|
||||
return logits, trace
|
||||
|
||||
|
||||
def learning_rate(step: int, total_steps: int) -> float:
|
||||
if step <= WARMUP_STEPS:
|
||||
return PEAK_LR * step / WARMUP_STEPS
|
||||
progress = (step - WARMUP_STEPS) / max(1, total_steps - WARMUP_STEPS)
|
||||
cosine = 0.5 * (1 + math.cos(math.pi * progress))
|
||||
return MIN_LR + (PEAK_LR - MIN_LR) * cosine
|
||||
|
||||
|
||||
def cross_entropy(logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:
|
||||
return F.cross_entropy(
|
||||
logits.float().reshape(-1, VOCABULARY), targets.reshape(-1)
|
||||
)
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def evaluate(
|
||||
model: ReducedLanguageModel,
|
||||
corpus: ByteCorpus,
|
||||
starts: list[int],
|
||||
window_count: int,
|
||||
eval_batch_size: int,
|
||||
) -> dict[str, float]:
|
||||
model.eval()
|
||||
loss_sum = 0.0
|
||||
target_count = 0
|
||||
for begin in range(0, window_count, eval_batch_size):
|
||||
end = min(begin + eval_batch_size, window_count)
|
||||
inputs, targets = corpus.fixed_batch(starts, begin, end)
|
||||
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
||||
logits, _ = model(inputs)
|
||||
loss = F.cross_entropy(
|
||||
logits.float().reshape(-1, VOCABULARY),
|
||||
targets.reshape(-1),
|
||||
reduction="sum",
|
||||
)
|
||||
loss_sum += loss.detach().cpu().item()
|
||||
target_count += targets.numel()
|
||||
nats = loss_sum / target_count
|
||||
return {"cross_entropy_nats": nats, "bits_per_byte": nats / math.log(2)}
|
||||
|
||||
|
||||
def percentile(values: list[float], quantile: float) -> float:
|
||||
return float(np.quantile(np.asarray(values, dtype=np.float64), quantile))
|
||||
|
||||
|
||||
def core_parameter_gradient_rms(model: ReducedLanguageModel) -> list[float]:
|
||||
values = []
|
||||
for block in model.blocks:
|
||||
sum_square = 0.0
|
||||
count = 0
|
||||
for parameter in block.parameters():
|
||||
if parameter.grad is None:
|
||||
continue
|
||||
gradient = parameter.grad.detach().float()
|
||||
sum_square += gradient.square().sum().detach().cpu().item()
|
||||
count += gradient.numel()
|
||||
values.append(math.sqrt(sum_square / count))
|
||||
return values
|
||||
|
||||
|
||||
def diagnostic(
|
||||
model: ReducedLanguageModel,
|
||||
corpus: ByteCorpus,
|
||||
window_count: int,
|
||||
) -> dict[str, Any]:
|
||||
model.eval()
|
||||
model.zero_grad(set_to_none=True)
|
||||
inputs, targets = corpus.fixed_batch(
|
||||
corpus.diagnostic_starts, 0, window_count
|
||||
)
|
||||
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
||||
logits, trace = model(inputs, capture=True)
|
||||
loss = cross_entropy(logits, targets)
|
||||
loss.backward()
|
||||
gradients = core_parameter_gradient_rms(model)
|
||||
assert trace is not None
|
||||
return {
|
||||
"loss_nats": loss.detach().cpu().item(),
|
||||
"bits_per_byte": loss.detach().cpu().item() / math.log(2),
|
||||
"layer_input_rms": trace.layer_input_rms,
|
||||
"branch_output_rms": trace.branch_output_rms,
|
||||
"stream_state_rms": trace.stream_state_rms,
|
||||
"core_parameter_grad_rms_by_block": gradients,
|
||||
"depth_weights": trace.depth_weights,
|
||||
"output_weights": trace.output_weights,
|
||||
}
|
||||
|
||||
|
||||
def parameter_inventory(model: ReducedLanguageModel) -> dict[str, int]:
|
||||
total = sum(parameter.numel() for parameter in model.parameters())
|
||||
mixer = sum(
|
||||
parameter.numel()
|
||||
for name, parameter in model.named_parameters()
|
||||
if name.startswith("mixers.") or name.startswith("output_mixer.")
|
||||
)
|
||||
embedding = model.token_embedding.weight.numel() + model.position_embedding.weight.numel()
|
||||
return {
|
||||
"total": total,
|
||||
"core": total - mixer,
|
||||
"mixer": mixer,
|
||||
"embedding": embedding,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required by the frozen protocol")
|
||||
if args.run_kind != "smoke" and args.seed not in EXPECTED_SEEDS:
|
||||
raise ValueError(f"formal/replay seed is not preregistered: {args.seed}")
|
||||
configure_determinism(args.seed)
|
||||
device = torch.device("cuda")
|
||||
|
||||
manifest = json.loads(args.manifest.read_text())
|
||||
if manifest["protocol_id"] != PROTOCOL_ID:
|
||||
raise ValueError("manifest protocol mismatch")
|
||||
if manifest["dataset"]["revision"] != (
|
||||
"b08601e04326c79dfdd32d625aee71d232d685c3"
|
||||
):
|
||||
raise ValueError("dataset revision mismatch")
|
||||
corpus = ByteCorpus(args.cache_dir, manifest, device)
|
||||
|
||||
model = ReducedLanguageModel(args.architecture).to(device)
|
||||
initial_common_hash = state_hash(model, include_mixers=False)
|
||||
initial_mixer_hash = (
|
||||
state_hash(model, include_mixers=True)
|
||||
if args.architecture != "baseline"
|
||||
else None
|
||||
)
|
||||
inventory = parameter_inventory(model)
|
||||
|
||||
decay_parameters: list[nn.Parameter] = []
|
||||
no_decay_parameters: list[nn.Parameter] = []
|
||||
for parameter in model.parameters():
|
||||
if parameter.ndim >= 2:
|
||||
decay_parameters.append(parameter)
|
||||
else:
|
||||
no_decay_parameters.append(parameter)
|
||||
optimizer = torch.optim.AdamW(
|
||||
[
|
||||
{"params": decay_parameters, "weight_decay": WEIGHT_DECAY},
|
||||
{"params": no_decay_parameters, "weight_decay": 0.0},
|
||||
],
|
||||
lr=PEAK_LR,
|
||||
betas=BETAS,
|
||||
eps=ADAM_EPS,
|
||||
)
|
||||
|
||||
evaluation_steps = sorted(
|
||||
set(step for step in EVAL_STEPS if step <= args.steps) | {0, args.steps}
|
||||
)
|
||||
evaluations = [
|
||||
{
|
||||
"step": 0,
|
||||
**evaluate(
|
||||
model,
|
||||
corpus,
|
||||
corpus.validation_starts,
|
||||
args.validation_windows,
|
||||
args.eval_batch_size,
|
||||
),
|
||||
}
|
||||
]
|
||||
training_history: list[dict[str, float | int]] = []
|
||||
step_times: list[float] = []
|
||||
model.train()
|
||||
|
||||
for step in range(1, args.steps + 1):
|
||||
lr = learning_rate(step, args.steps)
|
||||
for group in optimizer.param_groups:
|
||||
group["lr"] = lr
|
||||
inputs, targets = corpus.training_batch(args.seed, step, args.batch_size)
|
||||
optimizer.zero_grad(set_to_none=True)
|
||||
|
||||
torch.cuda.synchronize()
|
||||
started = time.perf_counter()
|
||||
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
||||
logits, _ = model(inputs)
|
||||
loss = cross_entropy(logits, targets)
|
||||
if not torch.isfinite(loss):
|
||||
raise RuntimeError(f"non-finite loss at step {step}: {loss}")
|
||||
loss.backward()
|
||||
unclipped_norm = torch.nn.utils.clip_grad_norm_(
|
||||
model.parameters(), GRAD_CLIP
|
||||
)
|
||||
optimizer.step()
|
||||
torch.cuda.synchronize()
|
||||
elapsed_ms = (time.perf_counter() - started) * 1000
|
||||
|
||||
if step == args.timing_warmup:
|
||||
torch.cuda.reset_peak_memory_stats()
|
||||
elif step > args.timing_warmup:
|
||||
step_times.append(elapsed_ms)
|
||||
|
||||
if step == 1 or step % 10 == 0 or step == args.steps:
|
||||
training_history.append(
|
||||
{
|
||||
"step": step,
|
||||
"loss_nats": loss.detach().cpu().item(),
|
||||
"bits_per_byte": loss.detach().cpu().item() / math.log(2),
|
||||
"learning_rate": lr,
|
||||
"unclipped_grad_norm": float(unclipped_norm.detach().cpu()),
|
||||
}
|
||||
)
|
||||
|
||||
if step in evaluation_steps and step != 0:
|
||||
evaluations.append(
|
||||
{
|
||||
"step": step,
|
||||
**evaluate(
|
||||
model,
|
||||
corpus,
|
||||
corpus.validation_starts,
|
||||
args.validation_windows,
|
||||
args.eval_batch_size,
|
||||
),
|
||||
}
|
||||
)
|
||||
model.train()
|
||||
|
||||
training_peak_allocated = torch.cuda.max_memory_allocated()
|
||||
training_peak_reserved = torch.cuda.max_memory_reserved()
|
||||
diagnostic_result = diagnostic(model, corpus, args.diagnostic_windows)
|
||||
final_common_hash = state_hash(model, include_mixers=False)
|
||||
final_mixer_hash = (
|
||||
state_hash(model, include_mixers=True)
|
||||
if args.architecture != "baseline"
|
||||
else None
|
||||
)
|
||||
timing = {
|
||||
"warmup_steps_excluded": args.timing_warmup,
|
||||
"measured_steps": len(step_times),
|
||||
"mean_ms": statistics.fmean(step_times) if step_times else None,
|
||||
"median_ms": statistics.median(step_times) if step_times else None,
|
||||
"p95_ms": percentile(step_times, 0.95) if step_times else None,
|
||||
"peak_allocated_bytes": training_peak_allocated,
|
||||
"peak_reserved_bytes": training_peak_reserved,
|
||||
}
|
||||
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"run_kind": args.run_kind,
|
||||
"architecture": args.architecture,
|
||||
"seed": args.seed,
|
||||
"steps": args.steps,
|
||||
"batch_size": args.batch_size,
|
||||
"target_bytes_seen": args.steps * args.batch_size * CONTEXT,
|
||||
"manifest": {
|
||||
"path": str(args.manifest),
|
||||
"file_sha256": file_sha256(args.manifest),
|
||||
"formal_schedule_sha256": manifest["windows"][
|
||||
"formal_schedule_sha256"
|
||||
],
|
||||
"validation_tensor_sha256": manifest["windows"][
|
||||
"validation_tensor_sha256"
|
||||
],
|
||||
"diagnostic_tensor_sha256": manifest["windows"][
|
||||
"diagnostic_tensor_sha256"
|
||||
],
|
||||
},
|
||||
"model": {
|
||||
"layers": LAYERS,
|
||||
"sublayers": SUBLAYERS,
|
||||
"blocks_for_block_attnres": BLOCKS,
|
||||
"sublayers_per_attnres_block": SUBLAYERS_PER_BLOCK,
|
||||
"d_model": D_MODEL,
|
||||
"heads": HEADS,
|
||||
"d_head": D_HEAD,
|
||||
"d_ff": D_FF,
|
||||
"context": CONTEXT,
|
||||
"vocabulary": VOCABULARY,
|
||||
"parameters": inventory,
|
||||
},
|
||||
"optimizer": {
|
||||
"name": "AdamW",
|
||||
"betas": list(BETAS),
|
||||
"epsilon": ADAM_EPS,
|
||||
"weight_decay_ndim_ge_2": WEIGHT_DECAY,
|
||||
"peak_lr": PEAK_LR,
|
||||
"min_lr": MIN_LR,
|
||||
"warmup_steps": WARMUP_STEPS,
|
||||
"grad_clip": GRAD_CLIP,
|
||||
},
|
||||
"hashes": {
|
||||
"initial_common_parameters": initial_common_hash,
|
||||
"initial_mixer_parameters": initial_mixer_hash,
|
||||
"final_common_parameters": final_common_hash,
|
||||
"final_mixer_parameters": final_mixer_hash,
|
||||
},
|
||||
"evaluations": evaluations,
|
||||
"training_history": training_history,
|
||||
"diagnostic": diagnostic_result,
|
||||
"timing": timing,
|
||||
"environment": {
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"cuda": torch.version.cuda,
|
||||
"gpu": torch.cuda.get_device_name(0),
|
||||
"compute_capability": list(torch.cuda.get_device_capability(0)),
|
||||
"cublas_workspace_config": os.environ["CUBLAS_WORKSPACE_CONFIG"],
|
||||
"deterministic_algorithms": torch.are_deterministic_algorithms_enabled(),
|
||||
"autocast": "cuda-bfloat16",
|
||||
"compile": False,
|
||||
},
|
||||
}
|
||||
result["canonical_sha256_without_self"] = canonical_json_sha256(result)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = args.output.with_suffix(args.output.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
os.replace(temporary, args.output)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"architecture": args.architecture,
|
||||
"seed": args.seed,
|
||||
"steps": args.steps,
|
||||
"final_bpc": evaluations[-1]["bits_per_byte"],
|
||||
"initial_common_hash": initial_common_hash,
|
||||
"final_common_hash": final_common_hash,
|
||||
"canonical_sha256": result["canonical_sha256_without_self"],
|
||||
"timing": timing,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,70 @@
|
||||
# Attention Residuals train-time forward intervention
|
||||
|
||||
This directory implements preregistered protocol
|
||||
`llm-atlas-k3-attnres-forward-training-v1`:
|
||||
|
||||
- `research/K3_ATTNRES_FORWARD_TRAINING_SCOPING.md`
|
||||
- `research/K3_ATTNRES_FORWARD_TRAINING_PROTOCOL.md`
|
||||
- `research/K3_ATTNRES_FORWARD_TRAINING_GROK_REVIEW.md`
|
||||
- `research/K3_ATTNRES_FORWARD_TRAINING_IMPLEMENTATION_REVIEW.md`
|
||||
- `research/K3_ATTNRES_FORWARD_TRAINING_AUDIT.md`
|
||||
|
||||
It is a depth-32 reduced Block AttnRes architecture ablation. It is not a
|
||||
Kimi-K3 checkpoint forward pass and does not claim to recover unpublished
|
||||
Figure 5 telemetry.
|
||||
|
||||
## Frozen environment
|
||||
|
||||
```text
|
||||
Python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python
|
||||
PyTorch 2.11.0+cu128
|
||||
GPU NVIDIA GeForce RTX 5090
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8
|
||||
maximum concurrency 2
|
||||
```
|
||||
|
||||
## Pre-result gates
|
||||
|
||||
The checked-in gate artifacts must pass before formal output:
|
||||
|
||||
```bash
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8 \
|
||||
/home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python \
|
||||
experiments/k3/attnres_forward/verify.py step-zero \
|
||||
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
||||
--parent-manifest experiments/k3/attnres_gradient/manifest.json \
|
||||
--study-manifest experiments/k3/attnres_forward/manifest.json \
|
||||
--output experiments/k3/attnres_forward/results/gates/step-zero.json
|
||||
```
|
||||
|
||||
`learned_reference` is smoke-only. Its 20-step result is compared with a
|
||||
fresh parent Round 05 smoke using `verify.py smoke-compare`.
|
||||
|
||||
## Formal matrix
|
||||
|
||||
```bash
|
||||
/home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python \
|
||||
experiments/k3/attnres_forward/run_matrix.py \
|
||||
--python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python \
|
||||
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
||||
--parent-manifest experiments/k3/attnres_gradient/manifest.json \
|
||||
--study-manifest experiments/k3/attnres_forward/manifest.json \
|
||||
--output-dir experiments/k3/attnres_forward/results/raw \
|
||||
--phase all \
|
||||
--concurrency 2
|
||||
```
|
||||
|
||||
This runs 12 formal cells and one full replay. The analyzer reads all cells,
|
||||
the frozen historical paired references, and generates the only authoritative
|
||||
status, interaction map, and website compact artifact.
|
||||
|
||||
The checked-in Round 08 release contains:
|
||||
|
||||
- 13 raw results under `results/raw/`;
|
||||
- `reproduction.json` with the exact primary scientific-payload hash;
|
||||
- aggregate / compact website data under `src/data/`;
|
||||
- a frozen-data checker and real-Chrome five-view regression in `scripts/`.
|
||||
|
||||
The established status is deliberately scoped to this reduced protocol. It is
|
||||
not a real Kimi-K3 checkpoint result or a reproduction of unpublished Figure
|
||||
5(c) telemetry.
|
||||
@@ -0,0 +1,699 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Aggregate and gate preregistered Round 08 forward-training results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-forward-training-v1"
|
||||
PARENT_PROTOCOL_ID = "llm-atlas-k3-attnres-gradient-scale-v1"
|
||||
METRICS = ("spike_contrast", "peak_normalized")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--formal", type=Path, action="append", required=True)
|
||||
parser.add_argument("--replay", type=Path, required=True)
|
||||
parser.add_argument("--reference-dir", type=Path, required=True)
|
||||
parser.add_argument("--aggregate-output", type=Path, required=True)
|
||||
parser.add_argument("--compact-output", type=Path, required=True)
|
||||
parser.add_argument("--reproduction-output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def mean(values: Iterable[float]) -> float:
|
||||
return statistics.fmean(values)
|
||||
|
||||
|
||||
def read_result(path: Path, expected_protocol: str) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text())
|
||||
if value.get("protocol_id") != expected_protocol:
|
||||
raise RuntimeError(f"protocol mismatch: {path}")
|
||||
expected = value.get("canonical_sha256_without_self")
|
||||
payload = {
|
||||
key: item
|
||||
for key, item in value.items()
|
||||
if key != "canonical_sha256_without_self"
|
||||
}
|
||||
if not isinstance(expected, str) or canonical_sha256(payload) != expected:
|
||||
raise RuntimeError(f"canonical self-hash mismatch: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def exactly_one(values: list[dict[str, Any]], step: int) -> dict[str, Any]:
|
||||
matches = [value for value in values if value["step"] == step]
|
||||
if len(matches) != 1:
|
||||
raise RuntimeError(f"step {step} missing or duplicated")
|
||||
return matches[0]
|
||||
|
||||
|
||||
def spectrum_metrics(
|
||||
diagnostic: dict[str, Any],
|
||||
spike_layers: tuple[int, ...],
|
||||
epsilon: float,
|
||||
) -> dict[str, Any]:
|
||||
values = [
|
||||
float(value)
|
||||
for value in diagnostic["activation_grad_rms_by_block"]
|
||||
]
|
||||
if len(values) != 32:
|
||||
raise RuntimeError("activation-gradient spectrum must have 32 layers")
|
||||
if any(not math.isfinite(value) or value <= epsilon for value in values):
|
||||
raise RuntimeError("activation-gradient spectrum is non-finite/non-positive")
|
||||
spike_indices = {layer - 1 for layer in spike_layers}
|
||||
spike_values = [
|
||||
value for index, value in enumerate(values) if index in spike_indices
|
||||
]
|
||||
reference_values = [
|
||||
value for index, value in enumerate(values) if index not in spike_indices
|
||||
]
|
||||
spike_mean = mean(spike_values)
|
||||
reference_mean = mean(reference_values)
|
||||
global_mean = mean(values)
|
||||
contrast = spike_mean / reference_mean
|
||||
peak = max(values) / global_mean
|
||||
if any(
|
||||
not math.isfinite(value) or value <= epsilon
|
||||
for value in (spike_mean, reference_mean, contrast, peak)
|
||||
):
|
||||
raise RuntimeError("derived spike metric is non-finite/non-positive")
|
||||
ordered = sorted(range(32), key=lambda index: (-values[index], index))
|
||||
return {
|
||||
"values": values,
|
||||
"normalized": [value / global_mean for value in values],
|
||||
"spike_mean": spike_mean,
|
||||
"reference_mean": reference_mean,
|
||||
"global_mean": global_mean,
|
||||
"spike_contrast": contrast,
|
||||
"peak_normalized": peak,
|
||||
"peak_layer_1based": ordered[0] + 1,
|
||||
"top_five_layers_1based": [index + 1 for index in ordered[:5]],
|
||||
}
|
||||
|
||||
|
||||
def final_bpc(value: dict[str, Any], step: int) -> float:
|
||||
result = float(exactly_one(value["evaluations"], step)["bits_per_byte"])
|
||||
if not math.isfinite(result):
|
||||
raise RuntimeError("final validation BPC is non-finite")
|
||||
return result
|
||||
|
||||
|
||||
def stable_environment(value: dict[str, Any]) -> dict[str, Any]:
|
||||
keys = (
|
||||
"cublas_workspace_config",
|
||||
"deterministic_algorithms",
|
||||
"autocast",
|
||||
"compile",
|
||||
)
|
||||
return {key: value["environment"][key] for key in keys}
|
||||
|
||||
|
||||
def pairing_checks(
|
||||
run: dict[str, Any], reference: dict[str, Any]
|
||||
) -> dict[str, bool]:
|
||||
manifest_fields = (
|
||||
"formal_schedule_sha256",
|
||||
"validation_tensor_sha256",
|
||||
"diagnostic_tensor_sha256",
|
||||
"input_gate_tensor_hashes",
|
||||
)
|
||||
checks = {
|
||||
"seed": run["seed"] == reference["seed"],
|
||||
"architecture": (
|
||||
run["architecture"] == reference["architecture"] == "block"
|
||||
),
|
||||
"depth": run["depth"] == reference["depth"] == 32,
|
||||
"steps": run["steps"] == reference["steps"] == 8000,
|
||||
"batch_size": run["batch_size"] == reference["batch_size"] == 32,
|
||||
"initial_public_parameters": (
|
||||
run["hashes"]["initial_public_parameters"]
|
||||
== reference["hashes"]["initial_public_parameters"]
|
||||
),
|
||||
"initial_mixer_parameters": (
|
||||
run["hashes"]["initial_mixer_parameters"]
|
||||
== reference["hashes"]["initial_mixer_parameters"]
|
||||
),
|
||||
"model_topology": run["model"] == reference["model"],
|
||||
"optimizer_hyperparameters": (
|
||||
run["optimizer"] == reference["optimizer"]
|
||||
),
|
||||
"scientific_environment": (
|
||||
stable_environment(run) == stable_environment(reference)
|
||||
),
|
||||
}
|
||||
for field in manifest_fields:
|
||||
checks[f"manifest.{field}"] = (
|
||||
run["manifest"][field] == reference["manifest"][field]
|
||||
)
|
||||
return checks
|
||||
|
||||
|
||||
def scientific_replay_payload(value: dict[str, Any]) -> dict[str, Any]:
|
||||
payload = copy.deepcopy(value)
|
||||
for key in (
|
||||
"run_kind",
|
||||
"timing",
|
||||
"canonical_sha256_without_self",
|
||||
"parent_runner_canonical_sha256",
|
||||
):
|
||||
payload.pop(key, None)
|
||||
payload["manifest"].pop("path", None)
|
||||
payload["study_manifest"].pop("path", None)
|
||||
payload["environment"] = stable_environment(value)
|
||||
return payload
|
||||
|
||||
|
||||
def quality_gate(
|
||||
variant_runs: dict[int, dict[str, Any]],
|
||||
references: dict[int, dict[str, Any]],
|
||||
*,
|
||||
step: int,
|
||||
per_seed_maximum: float,
|
||||
mean_maximum: float,
|
||||
) -> dict[str, Any]:
|
||||
per_seed = {}
|
||||
for seed, run in sorted(variant_runs.items()):
|
||||
variant_bpc = final_bpc(run, step)
|
||||
reference_bpc = final_bpc(references[seed], step)
|
||||
delta = variant_bpc - reference_bpc
|
||||
per_seed[str(seed)] = {
|
||||
"variant_bpc": variant_bpc,
|
||||
"reference_bpc": reference_bpc,
|
||||
"delta_bpc": delta,
|
||||
"passed": delta <= per_seed_maximum,
|
||||
}
|
||||
mean_delta = mean(item["delta_bpc"] for item in per_seed.values())
|
||||
per_seed_passed = all(item["passed"] for item in per_seed.values())
|
||||
mean_passed = mean_delta <= mean_maximum
|
||||
return {
|
||||
"passed": per_seed_passed and mean_passed,
|
||||
"passed_checks": (
|
||||
sum(item["passed"] for item in per_seed.values())
|
||||
+ int(mean_passed)
|
||||
),
|
||||
"required_checks": 4,
|
||||
"per_seed_maximum": per_seed_maximum,
|
||||
"mean_maximum": mean_maximum,
|
||||
"mean_delta_bpc": mean_delta,
|
||||
"mean_passed": mean_passed,
|
||||
"per_seed": per_seed,
|
||||
}
|
||||
|
||||
|
||||
def variant_effect(
|
||||
variant: str,
|
||||
runs: dict[int, dict[str, Any]],
|
||||
references: dict[int, dict[str, Any]],
|
||||
metrics_by_cell: dict[tuple[str, int, int], dict[str, Any]],
|
||||
*,
|
||||
step: int,
|
||||
threshold: float,
|
||||
quality: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
cells = []
|
||||
for seed in sorted(runs):
|
||||
candidate = metrics_by_cell[(variant, seed, step)]
|
||||
reference = metrics_by_cell[("learned_reference", seed, step)]
|
||||
for metric in METRICS:
|
||||
reference_value = reference[metric]
|
||||
candidate_value = candidate[metric]
|
||||
relative_drop = (
|
||||
reference_value - candidate_value
|
||||
) / reference_value
|
||||
cells.append(
|
||||
{
|
||||
"seed": seed,
|
||||
"metric": metric,
|
||||
"reference": reference_value,
|
||||
"variant": candidate_value,
|
||||
"relative_drop": relative_drop,
|
||||
"passed": relative_drop >= threshold,
|
||||
}
|
||||
)
|
||||
attenuation_passed = all(cell["passed"] for cell in cells)
|
||||
return {
|
||||
"variant": variant,
|
||||
"threshold": threshold,
|
||||
"passed_cells": sum(cell["passed"] for cell in cells),
|
||||
"required_cells": len(cells),
|
||||
"attenuation_passed": attenuation_passed,
|
||||
"quality": quality,
|
||||
"material_response_passed": (
|
||||
attenuation_passed and quality["passed"]
|
||||
),
|
||||
"cells": cells,
|
||||
}
|
||||
|
||||
|
||||
def interaction_map(
|
||||
metrics_by_cell: dict[tuple[str, int, int], dict[str, Any]],
|
||||
seeds: tuple[int, ...],
|
||||
steps: tuple[int, ...],
|
||||
) -> dict[str, Any]:
|
||||
cells = []
|
||||
for step in steps:
|
||||
for seed in seeds:
|
||||
reference = metrics_by_cell[
|
||||
("learned_reference", seed, step)
|
||||
]
|
||||
group6 = metrics_by_cell[
|
||||
("uniform_group_6_forward", seed, step)
|
||||
]
|
||||
group7 = metrics_by_cell[
|
||||
("uniform_group_7_forward", seed, step)
|
||||
]
|
||||
joint = metrics_by_cell[
|
||||
("uniform_groups_6_7_forward", seed, step)
|
||||
]
|
||||
for metric in METRICS:
|
||||
ref = reference[metric]
|
||||
effects = {
|
||||
"group6": math.log(ref / group6[metric]),
|
||||
"group7": math.log(ref / group7[metric]),
|
||||
"groups6_7": math.log(ref / joint[metric]),
|
||||
}
|
||||
residual = (
|
||||
effects["groups6_7"]
|
||||
- effects["group6"]
|
||||
- effects["group7"]
|
||||
)
|
||||
cells.append(
|
||||
{
|
||||
"step": step,
|
||||
"seed": seed,
|
||||
"metric": metric,
|
||||
"log_effects": effects,
|
||||
"interaction_residual": residual,
|
||||
"relative_drops": {
|
||||
"group6": (ref - group6[metric]) / ref,
|
||||
"group7": (ref - group7[metric]) / ref,
|
||||
"groups6_7": (ref - joint[metric]) / ref,
|
||||
},
|
||||
}
|
||||
)
|
||||
summaries = []
|
||||
for step in steps:
|
||||
for metric in METRICS:
|
||||
selected = [
|
||||
cell
|
||||
for cell in cells
|
||||
if cell["step"] == step and cell["metric"] == metric
|
||||
]
|
||||
residuals = [
|
||||
cell["interaction_residual"] for cell in selected
|
||||
]
|
||||
summaries.append(
|
||||
{
|
||||
"step": step,
|
||||
"metric": metric,
|
||||
"mean_interaction_residual": mean(residuals),
|
||||
"minimum": min(residuals),
|
||||
"maximum": max(residuals),
|
||||
}
|
||||
)
|
||||
return {
|
||||
"definition": "I67=ln(Xref/X67)-ln(Xref/X6)-ln(Xref/X7)",
|
||||
"interpretation": (
|
||||
"descriptive cross-run log-attenuation residual from three "
|
||||
"independently trained variants; not a causal interaction"
|
||||
),
|
||||
"cells": cells,
|
||||
"summaries": summaries,
|
||||
}
|
||||
|
||||
|
||||
def environment_metadata(value: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
key: value["environment"].get(key)
|
||||
for key in ("gpu", "torch", "cuda", "compute_capability")
|
||||
}
|
||||
|
||||
|
||||
def write_hashed(path: Path, value: dict[str, Any]) -> None:
|
||||
value["canonical_sha256_without_self"] = canonical_sha256(value)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(
|
||||
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
manifest = json.loads(args.manifest.read_text())
|
||||
if (
|
||||
manifest["protocol_id"] != PROTOCOL_ID
|
||||
or manifest["status"] != "frozen-before-model-output"
|
||||
):
|
||||
raise RuntimeError("manifest is not the frozen Round 08 contract")
|
||||
variants = tuple(manifest["variants"].keys())
|
||||
seeds = tuple(manifest["formal_seeds"])
|
||||
steps = tuple(manifest["diagnostic_steps"])
|
||||
primary_step = manifest["primary_step"]
|
||||
epsilon = manifest["thresholds"]["positive_denominator_epsilon"]
|
||||
spike_layers = tuple(manifest["spike_layers_1based"])
|
||||
expected_cells = {(variant, seed) for variant in variants for seed in seeds}
|
||||
if len(args.formal) != len(expected_cells):
|
||||
raise RuntimeError("formal path count does not match the 4×3 matrix")
|
||||
|
||||
runs: dict[tuple[str, int], dict[str, Any]] = {}
|
||||
run_paths: dict[tuple[str, int], Path] = {}
|
||||
pairing: dict[str, Any] = {}
|
||||
references: dict[int, dict[str, Any]] = {}
|
||||
reference_paths: dict[int, Path] = {}
|
||||
for seed in seeds:
|
||||
path = args.reference_dir / (
|
||||
f"formal-depth-32-block-seed-{seed}.json"
|
||||
)
|
||||
references[seed] = read_result(path, PARENT_PROTOCOL_ID)
|
||||
reference_paths[seed] = path
|
||||
|
||||
for path in args.formal:
|
||||
value = read_result(path, PROTOCOL_ID)
|
||||
identity = (value["variant"], value["seed"])
|
||||
if identity in runs:
|
||||
raise RuntimeError(f"duplicate formal cell: {identity}")
|
||||
if (
|
||||
value["run_kind"] != "formal"
|
||||
or value["steps"] != manifest["formal_steps"]
|
||||
or not value["forward_intervention"]["passed"]
|
||||
):
|
||||
raise RuntimeError(f"invalid formal cell: {path}")
|
||||
runs[identity] = value
|
||||
run_paths[identity] = path
|
||||
if set(runs) != expected_cells:
|
||||
raise RuntimeError("formal matrix identities do not match manifest")
|
||||
|
||||
for (variant, seed), value in sorted(runs.items()):
|
||||
checks = pairing_checks(value, references[seed])
|
||||
if not all(checks.values()):
|
||||
raise RuntimeError(
|
||||
f"historical reference pairing failed: "
|
||||
f"{variant}/{seed}: {checks}"
|
||||
)
|
||||
pairing[f"{variant}:{seed}"] = {
|
||||
"passed": True,
|
||||
"checks": checks,
|
||||
"run_environment": environment_metadata(value),
|
||||
"reference_environment": environment_metadata(references[seed]),
|
||||
"metadata_equal": (
|
||||
environment_metadata(value)
|
||||
== environment_metadata(references[seed])
|
||||
),
|
||||
}
|
||||
metadata_warnings = [
|
||||
{
|
||||
"cell": cell,
|
||||
"message": (
|
||||
"GPU/version metadata differs from the historical paired "
|
||||
"reference; frozen scientific-environment fields still match"
|
||||
),
|
||||
"run_environment": item["run_environment"],
|
||||
"reference_environment": item["reference_environment"],
|
||||
}
|
||||
for cell, item in pairing.items()
|
||||
if not item["metadata_equal"]
|
||||
]
|
||||
|
||||
replay = read_result(args.replay, PROTOCOL_ID)
|
||||
replay_contract = manifest["replay"]
|
||||
if (
|
||||
replay["run_kind"] != "replay"
|
||||
or replay["variant"] != replay_contract["variant"]
|
||||
or replay["seed"] != replay_contract["seed"]
|
||||
or replay["steps"] != manifest["formal_steps"]
|
||||
or not replay["forward_intervention"]["passed"]
|
||||
):
|
||||
raise RuntimeError("invalid replay identity/audit")
|
||||
formal_primary = runs[
|
||||
(replay_contract["variant"], replay_contract["seed"])
|
||||
]
|
||||
formal_payload = scientific_replay_payload(formal_primary)
|
||||
replay_payload = scientific_replay_payload(replay)
|
||||
replay_exact = formal_payload == replay_payload
|
||||
if not replay_exact:
|
||||
raise RuntimeError("primary formal/replay scientific payload mismatch")
|
||||
|
||||
metrics_by_cell: dict[tuple[str, int, int], dict[str, Any]] = {}
|
||||
for seed, reference in references.items():
|
||||
for step in steps:
|
||||
metrics_by_cell[("learned_reference", seed, step)] = (
|
||||
spectrum_metrics(
|
||||
exactly_one(reference["diagnostics"], step),
|
||||
spike_layers,
|
||||
epsilon,
|
||||
)
|
||||
)
|
||||
for (variant, seed), value in runs.items():
|
||||
if tuple(item["step"] for item in value["diagnostics"]) != steps:
|
||||
raise RuntimeError(f"diagnostic schedule drift: {variant}/{seed}")
|
||||
for step in steps:
|
||||
metrics_by_cell[(variant, seed, step)] = spectrum_metrics(
|
||||
exactly_one(value["diagnostics"], step),
|
||||
spike_layers,
|
||||
epsilon,
|
||||
)
|
||||
|
||||
runs_by_variant = {
|
||||
variant: {seed: runs[(variant, seed)] for seed in seeds}
|
||||
for variant in variants
|
||||
}
|
||||
qualities = {
|
||||
variant: quality_gate(
|
||||
variant_runs,
|
||||
references,
|
||||
step=primary_step,
|
||||
per_seed_maximum=manifest["thresholds"][
|
||||
"final_bpc_delta_per_seed_maximum"
|
||||
],
|
||||
mean_maximum=manifest["thresholds"][
|
||||
"final_bpc_delta_mean_maximum"
|
||||
],
|
||||
)
|
||||
for variant, variant_runs in runs_by_variant.items()
|
||||
}
|
||||
effects = {
|
||||
variant: variant_effect(
|
||||
variant,
|
||||
variant_runs,
|
||||
references,
|
||||
metrics_by_cell,
|
||||
step=primary_step,
|
||||
threshold=manifest["thresholds"]["material_relative_drop"],
|
||||
quality=qualities[variant],
|
||||
)
|
||||
for variant, variant_runs in runs_by_variant.items()
|
||||
}
|
||||
primary = effects[manifest["primary_variant"]]
|
||||
if primary["attenuation_passed"] and primary["quality"]["passed"]:
|
||||
status = (
|
||||
"forward_training_attenuation_established_within_reduced_protocol"
|
||||
)
|
||||
elif primary["attenuation_passed"]:
|
||||
status = "quality_guard_failed"
|
||||
elif primary["quality"]["passed"]:
|
||||
status = "attenuation_not_established"
|
||||
else:
|
||||
status = "attenuation_and_quality_failed"
|
||||
secondary = {
|
||||
variant: (
|
||||
"secondary_material_response"
|
||||
if effect["material_response_passed"]
|
||||
else "secondary_response_not_established"
|
||||
)
|
||||
for variant, effect in effects.items()
|
||||
if variant != manifest["primary_variant"]
|
||||
}
|
||||
interaction = interaction_map(metrics_by_cell, seeds, steps)
|
||||
|
||||
trajectories = []
|
||||
final_spectra = []
|
||||
for variant in ("learned_reference",) + variants:
|
||||
for seed in seeds:
|
||||
for step in steps:
|
||||
record = metrics_by_cell[(variant, seed, step)]
|
||||
reference = metrics_by_cell[
|
||||
("learned_reference", seed, step)
|
||||
]
|
||||
trajectories.append(
|
||||
{
|
||||
"variant": variant,
|
||||
"seed": seed,
|
||||
"step": step,
|
||||
"spike_mean": record["spike_mean"],
|
||||
"reference_mean": record["reference_mean"],
|
||||
"spike_contrast": record["spike_contrast"],
|
||||
"peak_normalized": record["peak_normalized"],
|
||||
"relative_drop": {
|
||||
metric: (
|
||||
reference[metric] - record[metric]
|
||||
)
|
||||
/ reference[metric]
|
||||
for metric in METRICS
|
||||
},
|
||||
}
|
||||
)
|
||||
final = metrics_by_cell[(variant, seed, primary_step)]
|
||||
final_spectra.append(
|
||||
{
|
||||
"variant": variant,
|
||||
"seed": seed,
|
||||
**final,
|
||||
}
|
||||
)
|
||||
|
||||
input_files = {
|
||||
"manifest": {
|
||||
"path": str(args.manifest),
|
||||
"sha256": file_sha256(args.manifest),
|
||||
},
|
||||
"formal": [
|
||||
{
|
||||
"variant": variant,
|
||||
"seed": seed,
|
||||
"path": str(run_paths[(variant, seed)]),
|
||||
"sha256": file_sha256(run_paths[(variant, seed)]),
|
||||
}
|
||||
for variant, seed in sorted(runs)
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"seed": seed,
|
||||
"path": str(reference_paths[seed]),
|
||||
"sha256": file_sha256(reference_paths[seed]),
|
||||
}
|
||||
for seed in seeds
|
||||
],
|
||||
"replay": {
|
||||
"path": str(args.replay),
|
||||
"sha256": file_sha256(args.replay),
|
||||
},
|
||||
}
|
||||
aggregate = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"status": status,
|
||||
"scope": (
|
||||
"depth-32 reduced Block AttnRes train-time architecture "
|
||||
"ablation; not a real Kimi-K3 checkpoint result"
|
||||
),
|
||||
"primary_step": primary_step,
|
||||
"spike_layers_1based": list(spike_layers),
|
||||
"thresholds": manifest["thresholds"],
|
||||
"input_files": input_files,
|
||||
"historical_pairing": pairing,
|
||||
"metadata_warnings": metadata_warnings,
|
||||
"replay": {
|
||||
"passed": replay_exact,
|
||||
"scientific_payload_sha256": canonical_sha256(formal_payload),
|
||||
"excluded": [
|
||||
"run_kind",
|
||||
"timing",
|
||||
"self hashes",
|
||||
"manifest path strings",
|
||||
"GPU/version metadata",
|
||||
],
|
||||
},
|
||||
"primary": primary,
|
||||
"secondary_status": secondary,
|
||||
"effects": effects,
|
||||
"interaction": interaction,
|
||||
"trajectories": trajectories,
|
||||
"final_spectra": final_spectra,
|
||||
"processed_target_bytes": manifest["new_target_bytes"],
|
||||
"historical_reference_target_bytes": (
|
||||
manifest["historical_reference_target_bytes"]
|
||||
),
|
||||
"reporting_boundary": (
|
||||
"C can change through spike-window numerator and the 27-layer "
|
||||
"reference denominator; layers 26-28 are intervened but belong "
|
||||
"to the denominator."
|
||||
),
|
||||
}
|
||||
write_hashed(args.aggregate_output, aggregate)
|
||||
|
||||
compact = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"status": status,
|
||||
"primary_step": primary_step,
|
||||
"spike_layers_1based": list(spike_layers),
|
||||
"thresholds": manifest["thresholds"],
|
||||
"primary": primary,
|
||||
"secondary_status": secondary,
|
||||
"effects": effects,
|
||||
"interaction": interaction,
|
||||
"trajectories": trajectories,
|
||||
"final_spectra": final_spectra,
|
||||
"replay": aggregate["replay"],
|
||||
"metadata_warnings": metadata_warnings,
|
||||
"processed_target_bytes": manifest["new_target_bytes"],
|
||||
"reporting_boundary": aggregate["reporting_boundary"],
|
||||
"aggregate_sha256": aggregate["canonical_sha256_without_self"],
|
||||
}
|
||||
write_hashed(args.compact_output, compact)
|
||||
|
||||
reproduction = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"passed": replay_exact,
|
||||
"formal_variant": replay_contract["variant"],
|
||||
"seed": replay_contract["seed"],
|
||||
"formal_file_sha256": file_sha256(
|
||||
run_paths[
|
||||
(replay_contract["variant"], replay_contract["seed"])
|
||||
]
|
||||
),
|
||||
"replay_file_sha256": file_sha256(args.replay),
|
||||
"scientific_payload_sha256": canonical_sha256(formal_payload),
|
||||
"excluded_fields": aggregate["replay"]["excluded"],
|
||||
}
|
||||
write_hashed(args.reproduction_output, reproduction)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": status,
|
||||
"primary_attenuation": {
|
||||
"passed_cells": primary["passed_cells"],
|
||||
"required_cells": primary["required_cells"],
|
||||
},
|
||||
"primary_quality": {
|
||||
"passed_checks": primary["quality"]["passed_checks"],
|
||||
"required_checks": primary["quality"]["required_checks"],
|
||||
},
|
||||
"replay_exact": replay_exact,
|
||||
"aggregate": str(args.aggregate_output),
|
||||
"compact": str(args.compact_output),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"protocol_id": "llm-atlas-k3-attnres-forward-training-v1",
|
||||
"status": "frozen-before-model-output",
|
||||
"parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1",
|
||||
"parent_manifest": "experiments/k3/attnres_gradient/manifest.json",
|
||||
"architecture": "block",
|
||||
"depth": 32,
|
||||
"formal_steps": 8000,
|
||||
"smoke_steps": 20,
|
||||
"batch_size": 32,
|
||||
"formal_seeds": [
|
||||
2026073001,
|
||||
2026073002,
|
||||
2026073003
|
||||
],
|
||||
"diagnostic_steps": [
|
||||
0,
|
||||
100,
|
||||
500,
|
||||
2000,
|
||||
4000,
|
||||
8000
|
||||
],
|
||||
"primary_step": 8000,
|
||||
"spike_layers_1based": [21, 22, 23, 24, 25],
|
||||
"historical_reference": {
|
||||
"directory": "experiments/k3/attnres_gradient/results/raw",
|
||||
"filename_template": "formal-depth-32-block-seed-{seed}.json",
|
||||
"identity": "historical-paired-reference-not-contemporaneous-randomized-control"
|
||||
},
|
||||
"variants": {
|
||||
"uniform_group_6_forward": {
|
||||
"selected_depth_indices": [40, 41, 42, 43, 44, 45, 46, 47]
|
||||
},
|
||||
"uniform_group_7_forward": {
|
||||
"selected_depth_indices": [48, 49, 50, 51, 52, 53, 54, 55]
|
||||
},
|
||||
"uniform_groups_6_7_forward": {
|
||||
"selected_depth_indices": [40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55]
|
||||
},
|
||||
"uniform_group_7_mlp_forward": {
|
||||
"selected_depth_indices": [49, 51, 53, 55]
|
||||
}
|
||||
},
|
||||
"smoke_only_variants": [
|
||||
"learned_reference"
|
||||
],
|
||||
"primary_variant": "uniform_groups_6_7_forward",
|
||||
"replay": {
|
||||
"variant": "uniform_groups_6_7_forward",
|
||||
"seed": 2026073001
|
||||
},
|
||||
"selected_source_counts": {
|
||||
"40": 6,
|
||||
"41": 7,
|
||||
"42": 7,
|
||||
"43": 7,
|
||||
"44": 7,
|
||||
"45": 7,
|
||||
"46": 7,
|
||||
"47": 7,
|
||||
"48": 7,
|
||||
"49": 8,
|
||||
"50": 8,
|
||||
"51": 8,
|
||||
"52": 8,
|
||||
"53": 8,
|
||||
"54": 8,
|
||||
"55": 8
|
||||
},
|
||||
"thresholds": {
|
||||
"positive_denominator_epsilon": 1e-30,
|
||||
"material_relative_drop": 0.2,
|
||||
"final_bpc_delta_per_seed_maximum": 0.05,
|
||||
"final_bpc_delta_mean_maximum": 0.03,
|
||||
"uniform_weight_max_abs_error": 1e-12,
|
||||
"loss_scale_ratio_abs_error": 1e-5,
|
||||
"loss_scale_shape_abs_error": 1e-6
|
||||
},
|
||||
"new_target_bytes": {
|
||||
"per_cell": 65536000,
|
||||
"formal_12_cells": 786432000,
|
||||
"primary_replay": 65536000,
|
||||
"total": 851968000
|
||||
},
|
||||
"historical_reference_target_bytes": 196608000,
|
||||
"concurrency_maximum": 2
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"canonical_sha256_without_self": "57346df80c0d76bd1d306d5fa213feed74ac2094237c22ebcb49f16ea16437e0",
|
||||
"excluded_fields": [
|
||||
"run_kind",
|
||||
"timing",
|
||||
"self hashes",
|
||||
"manifest path strings",
|
||||
"GPU/version metadata"
|
||||
],
|
||||
"formal_file_sha256": "0962ebd1a00a11e61ac795282bfa99412166c8752f2a3731f137030d7f134dc1",
|
||||
"formal_variant": "uniform_groups_6_7_forward",
|
||||
"passed": true,
|
||||
"post_result_grok_review": {
|
||||
"blocking_errors": 0,
|
||||
"claim_boundary_confirmed": true,
|
||||
"replay_confirmed": true,
|
||||
"session_id": "019fb28b-a9e1-7643-8e43-06f5e16a2077",
|
||||
"status_confirmed": true
|
||||
},
|
||||
"protocol_id": "llm-atlas-k3-attnres-forward-training-v1",
|
||||
"replay_file_sha256": "b85ac8062b2b0b8b3f2305d4b22a7c0212466fbfb28a9d63099cadd0a6917c8e",
|
||||
"schema_version": 1,
|
||||
"scientific_payload_sha256": "b85563ca5cb53e60b39c3801d372376206105b8a089a8633b3e81973a7f0c051",
|
||||
"seed": 2026073001
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"canonical_sha256_without_self": "99aeefc0ba35c199725ed7af377450ecbb5cc5f1aabdda9ac2345a30f03b444f",
|
||||
"excluded_fields": [
|
||||
"protocol wrapper fields",
|
||||
"timing",
|
||||
"self hash",
|
||||
"parent runner self hash",
|
||||
"study manifest"
|
||||
],
|
||||
"field_checks": {
|
||||
"architecture": true,
|
||||
"batch_size": true,
|
||||
"depth": true,
|
||||
"diagnostics": true,
|
||||
"environment": true,
|
||||
"evaluations": true,
|
||||
"gradient_gate": true,
|
||||
"hashes": true,
|
||||
"manifest": true,
|
||||
"model": true,
|
||||
"optimizer": true,
|
||||
"seed": true,
|
||||
"steps": true,
|
||||
"target_bytes_seen": true,
|
||||
"training_history": true
|
||||
},
|
||||
"gate": "empty-selector-parent-equivalence",
|
||||
"parent_file": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/results/gates/parent-smoke.json",
|
||||
"passed": true,
|
||||
"protocol_id": "llm-atlas-k3-attnres-forward-training-v1",
|
||||
"schema_version": 1,
|
||||
"wrapper_file": "/home/wuyang/Code/K3/experiments/k3/attnres_forward/results/gates/wrapper-smoke.json",
|
||||
"wrapper_identity": {
|
||||
"forward_audit": true,
|
||||
"parent_protocol": true,
|
||||
"protocol": true,
|
||||
"variant": true
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+14045
File diff suppressed because it is too large
Load Diff
+14045
File diff suppressed because it is too large
Load Diff
+14045
File diff suppressed because it is too large
Load Diff
+14045
File diff suppressed because it is too large
Load Diff
+14045
File diff suppressed because it is too large
Load Diff
+14045
File diff suppressed because it is too large
Load Diff
+14029
File diff suppressed because it is too large
Load Diff
+14029
File diff suppressed because it is too large
Load Diff
+14029
File diff suppressed because it is too large
Load Diff
+14077
File diff suppressed because it is too large
Load Diff
+14077
File diff suppressed because it is too large
Load Diff
+14077
File diff suppressed because it is too large
Load Diff
+14077
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the frozen Round 08 matrix with at most two isolated processes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
VARIANTS = (
|
||||
"uniform_group_6_forward",
|
||||
"uniform_group_7_forward",
|
||||
"uniform_groups_6_7_forward",
|
||||
"uniform_group_7_mlp_forward",
|
||||
)
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--python", type=Path, required=True)
|
||||
parser.add_argument("--cache-dir", type=Path, required=True)
|
||||
parser.add_argument("--parent-manifest", type=Path, required=True)
|
||||
parser.add_argument("--study-manifest", type=Path, required=True)
|
||||
parser.add_argument("--output-dir", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--phase", choices=("formal", "replay", "all"), default="all"
|
||||
)
|
||||
parser.add_argument("--concurrency", type=int, default=2)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def cell_output(
|
||||
output_dir: Path, variant: str, seed: int, run_kind: str
|
||||
) -> Path:
|
||||
return output_dir / (
|
||||
f"{run_kind}-{variant}-seed-{seed}.json"
|
||||
)
|
||||
|
||||
|
||||
def command_for(
|
||||
args: argparse.Namespace, variant: str, seed: int, run_kind: str
|
||||
) -> list[str]:
|
||||
runner = Path(__file__).resolve().parent / "train.py"
|
||||
return [
|
||||
str(args.python),
|
||||
str(runner),
|
||||
"--variant",
|
||||
variant,
|
||||
"--study-manifest",
|
||||
str(args.study_manifest),
|
||||
"--run-kind",
|
||||
run_kind,
|
||||
"--architecture",
|
||||
"block",
|
||||
"--depth",
|
||||
"32",
|
||||
"--seed",
|
||||
str(seed),
|
||||
"--cache-dir",
|
||||
str(args.cache_dir),
|
||||
"--manifest",
|
||||
str(args.parent_manifest),
|
||||
"--output",
|
||||
str(cell_output(args.output_dir, variant, seed, run_kind)),
|
||||
]
|
||||
|
||||
|
||||
def validate_manifest(args: argparse.Namespace) -> None:
|
||||
manifest = json.loads(args.study_manifest.read_text())
|
||||
if (
|
||||
manifest["status"] != "frozen-before-model-output"
|
||||
or tuple(manifest["variants"]) != VARIANTS
|
||||
or tuple(manifest["formal_seeds"]) != SEEDS
|
||||
or manifest["concurrency_maximum"] != 2
|
||||
):
|
||||
raise RuntimeError("study manifest matrix/concurrency drift")
|
||||
if args.concurrency < 1 or args.concurrency > 2:
|
||||
raise ValueError("the frozen protocol permits one or two processes")
|
||||
|
||||
|
||||
def stop_processes(items: list[dict[str, Any]]) -> None:
|
||||
for item in items:
|
||||
if item["process"].poll() is None:
|
||||
item["process"].terminate()
|
||||
for item in items:
|
||||
process = item["process"]
|
||||
if process.poll() is not None:
|
||||
continue
|
||||
try:
|
||||
process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait()
|
||||
|
||||
|
||||
def quarantine_failed_output(
|
||||
output_dir: Path, variant: str, seed: int, run_kind: str
|
||||
) -> str | None:
|
||||
output = cell_output(output_dir, variant, seed, run_kind)
|
||||
if not output.exists():
|
||||
return None
|
||||
failed = output.with_suffix(".failed.json")
|
||||
if failed.exists():
|
||||
failed = output.with_suffix(f".failed-{time.time_ns()}.json")
|
||||
output.replace(failed)
|
||||
return str(failed)
|
||||
|
||||
|
||||
def run_cells(
|
||||
args: argparse.Namespace,
|
||||
cells: list[tuple[str, int, str]],
|
||||
) -> None:
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
for variant, seed, run_kind in cells:
|
||||
output = cell_output(args.output_dir, variant, seed, run_kind)
|
||||
if output.exists():
|
||||
raise FileExistsError(
|
||||
f"refusing to overwrite existing result: {output}"
|
||||
)
|
||||
|
||||
environment = dict(os.environ)
|
||||
environment["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
|
||||
pending = list(cells)
|
||||
running: list[dict[str, Any]] = []
|
||||
completed = 0
|
||||
while pending or running:
|
||||
while pending and len(running) < args.concurrency:
|
||||
variant, seed, run_kind = pending.pop(0)
|
||||
command = command_for(args, variant, seed, run_kind)
|
||||
process = subprocess.Popen(command, env=environment)
|
||||
running.append(
|
||||
{
|
||||
"identity": (variant, seed, run_kind),
|
||||
"process": process,
|
||||
"started": time.monotonic(),
|
||||
}
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "cell_started",
|
||||
"variant": variant,
|
||||
"seed": seed,
|
||||
"run_kind": run_kind,
|
||||
"pid": process.pid,
|
||||
"active": len(running),
|
||||
"remaining": len(pending),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
time.sleep(1)
|
||||
survivors = []
|
||||
for item in running:
|
||||
return_code = item["process"].poll()
|
||||
if return_code is None:
|
||||
survivors.append(item)
|
||||
continue
|
||||
variant, seed, run_kind = item["identity"]
|
||||
elapsed = time.monotonic() - item["started"]
|
||||
if return_code != 0:
|
||||
stop_processes(
|
||||
[candidate for candidate in running if candidate is not item]
|
||||
)
|
||||
quarantined = quarantine_failed_output(
|
||||
args.output_dir, variant, seed, run_kind
|
||||
)
|
||||
raise RuntimeError(
|
||||
f"cell failed: {variant}/{seed}/{run_kind}: {return_code}; "
|
||||
f"quarantined_output={quarantined}"
|
||||
)
|
||||
completed += 1
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "cell_completed",
|
||||
"variant": variant,
|
||||
"seed": seed,
|
||||
"run_kind": run_kind,
|
||||
"elapsed_seconds": elapsed,
|
||||
"completed": completed,
|
||||
"total": len(cells),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
running = survivors
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
validate_manifest(args)
|
||||
formal = [
|
||||
(variant, seed, "formal")
|
||||
for variant in VARIANTS
|
||||
for seed in SEEDS
|
||||
]
|
||||
replay = [
|
||||
("uniform_groups_6_7_forward", 2026073001, "replay")
|
||||
]
|
||||
cells = (
|
||||
formal
|
||||
if args.phase == "formal"
|
||||
else replay
|
||||
if args.phase == "replay"
|
||||
else formal + replay
|
||||
)
|
||||
run_cells(args, cells)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,449 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run one preregistered Round 08 train-time uniform-forward cell."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-forward-training-v1"
|
||||
PARENT_PROTOCOL_ID = "llm-atlas-k3-attnres-gradient-scale-v1"
|
||||
VARIANTS = {
|
||||
"learned_reference": (),
|
||||
"uniform_group_6_forward": tuple(range(40, 48)),
|
||||
"uniform_group_7_forward": tuple(range(48, 56)),
|
||||
"uniform_groups_6_7_forward": tuple(range(40, 56)),
|
||||
"uniform_group_7_mlp_forward": (49, 51, 53, 55),
|
||||
}
|
||||
FORMAL_VARIANTS = tuple(name for name in VARIANTS if name != "learned_reference")
|
||||
EXPECTED_SOURCE_COUNTS = {
|
||||
**{40: 6},
|
||||
**{index: 7 for index in range(41, 49)},
|
||||
**{index: 8 for index in range(49, 56)},
|
||||
}
|
||||
|
||||
|
||||
def load_parent_module() -> Any:
|
||||
path = Path(__file__).resolve().parents[1] / "attnres_gradient" / "train.py"
|
||||
spec = importlib.util.spec_from_file_location("k3_attnres_round05_train", path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"cannot import Round 05 runner from {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
parent = load_parent_module()
|
||||
ACTIVE_VARIANT = "learned_reference"
|
||||
LAST_MODEL: ForwardInterventionLanguageModel | None = None
|
||||
LAST_OPTIMIZER: torch.optim.Optimizer | None = None
|
||||
|
||||
|
||||
def extract_wrapper_argument(name: str) -> str:
|
||||
try:
|
||||
index = sys.argv.index(name)
|
||||
except ValueError as error:
|
||||
raise ValueError(f"missing required wrapper argument: {name}") from error
|
||||
if index + 1 >= len(sys.argv):
|
||||
raise ValueError(f"missing value for wrapper argument: {name}")
|
||||
value = sys.argv[index + 1]
|
||||
del sys.argv[index : index + 2]
|
||||
return value
|
||||
|
||||
|
||||
def argument_value(name: str, default: str | None = None) -> str | None:
|
||||
try:
|
||||
index = sys.argv.index(name)
|
||||
except ValueError:
|
||||
return default
|
||||
if index + 1 >= len(sys.argv):
|
||||
raise ValueError(f"missing value for argument: {name}")
|
||||
return sys.argv[index + 1]
|
||||
|
||||
|
||||
def parameter_names_for_indices(indices: tuple[int, ...]) -> tuple[str, ...]:
|
||||
names = []
|
||||
for index in indices:
|
||||
names.extend(
|
||||
(
|
||||
f"mixers.{index}.query",
|
||||
f"mixers.{index}.key_norm.weight",
|
||||
)
|
||||
)
|
||||
return tuple(names)
|
||||
|
||||
|
||||
class ForwardInterventionLanguageModel(parent.GradientLanguageModel):
|
||||
"""Round 05 model with one frozen selector and parameter-free uniform mixers."""
|
||||
|
||||
def __init__(self, architecture: str):
|
||||
super().__init__(architecture)
|
||||
global LAST_MODEL
|
||||
if architecture != "block":
|
||||
raise ValueError("Round 08 only permits the block architecture")
|
||||
if ACTIVE_VARIANT not in VARIANTS:
|
||||
raise ValueError(f"unknown Round 08 variant: {ACTIVE_VARIANT}")
|
||||
self.forward_variant = ACTIVE_VARIANT
|
||||
self.selected_indices = tuple(VARIANTS[ACTIVE_VARIANT])
|
||||
self.selected_set = frozenset(self.selected_indices)
|
||||
self.forward_calls = 0
|
||||
self.depth_visits = [0] * len(self.mixers)
|
||||
self.output_visits = 0
|
||||
self.source_counts: dict[int, set[int]] = {
|
||||
index: set() for index in range(len(self.mixers))
|
||||
}
|
||||
self.uniform_weight_max_abs_error = 0.0
|
||||
selected_names = parameter_names_for_indices(self.selected_indices)
|
||||
named_parameters = dict(self.named_parameters())
|
||||
self.selected_initial_tensors = {
|
||||
name: named_parameters[name].detach().cpu().clone()
|
||||
for name in selected_names
|
||||
}
|
||||
self.gradient_hook_calls = {
|
||||
name: 0
|
||||
for name in named_parameters
|
||||
if name.startswith("mixers.") or name.startswith("output_mixer.")
|
||||
}
|
||||
self._gradient_hooks = []
|
||||
for name, parameter in named_parameters.items():
|
||||
if name not in self.gradient_hook_calls:
|
||||
continue
|
||||
|
||||
def count_hook(
|
||||
gradient: torch.Tensor, *, parameter_name: str = name
|
||||
) -> torch.Tensor:
|
||||
self.gradient_hook_calls[parameter_name] += 1
|
||||
return gradient
|
||||
|
||||
self._gradient_hooks.append(parameter.register_hook(count_hook))
|
||||
LAST_MODEL = self
|
||||
|
||||
def mix(
|
||||
self,
|
||||
mixer_index: int,
|
||||
sources: list[torch.Tensor],
|
||||
capture: bool,
|
||||
) -> tuple[torch.Tensor, dict[str, Any] | None]:
|
||||
self.depth_visits[mixer_index] += 1
|
||||
self.source_counts[mixer_index].add(len(sources))
|
||||
if mixer_index not in self.selected_set:
|
||||
return self.mixers[mixer_index](sources, capture)
|
||||
|
||||
values = torch.stack(sources, dim=0)
|
||||
logits = torch.zeros(
|
||||
values.shape[0],
|
||||
values.shape[1],
|
||||
values.shape[2],
|
||||
dtype=torch.float32,
|
||||
device=values.device,
|
||||
)
|
||||
weights = torch.softmax(logits, dim=0)
|
||||
expected = torch.tensor(
|
||||
1.0 / len(sources), dtype=weights.dtype, device=weights.device
|
||||
)
|
||||
error = (weights - expected).abs().max().detach().cpu().item()
|
||||
self.uniform_weight_max_abs_error = max(
|
||||
self.uniform_weight_max_abs_error, error
|
||||
)
|
||||
output = torch.einsum(
|
||||
"nbt,nbtd->btd", weights, values.float()
|
||||
).to(values.dtype)
|
||||
if not capture:
|
||||
return output, None
|
||||
entropy = -(weights * torch.log(weights.clamp_min(1e-30))).sum(dim=0)
|
||||
return output, {
|
||||
"mean_weights": weights.mean(dim=(1, 2)).detach().cpu().tolist(),
|
||||
"entropy_mean": entropy.mean().detach().cpu().item(),
|
||||
"sources": len(sources),
|
||||
}
|
||||
|
||||
def forward(
|
||||
self, input_ids: torch.Tensor, capture: bool = False
|
||||
) -> tuple[torch.Tensor, parent.ActivationTrace | None]:
|
||||
self.forward_calls += 1
|
||||
embedded = self.embed(input_ids)
|
||||
trace = parent.ActivationTrace([], [], [], [], []) if capture else None
|
||||
completed = [embedded]
|
||||
partial: torch.Tensor | None = None
|
||||
mixer_index = 0
|
||||
for block in self.blocks:
|
||||
for branch_index in range(2):
|
||||
sources = completed + ([] if partial is None else [partial])
|
||||
branch_input, weights = self.mix(
|
||||
mixer_index, sources, capture
|
||||
)
|
||||
mixer_index += 1
|
||||
if branch_index == 0:
|
||||
branch_output = block.attention(
|
||||
block.attention_norm(branch_input)
|
||||
)
|
||||
else:
|
||||
branch_output = block.mlp(block.mlp_norm(branch_input))
|
||||
branch_for_residual = branch_output.float()
|
||||
partial = (
|
||||
branch_for_residual
|
||||
if partial is None
|
||||
else partial + branch_for_residual
|
||||
)
|
||||
if trace is not None:
|
||||
trace.layer_input_rms.append(parent.rms(branch_input))
|
||||
trace.branch_output_rms.append(parent.rms(branch_output))
|
||||
trace.stream_state_rms.append(parent.rms(partial))
|
||||
trace.depth_weights.append(weights or {})
|
||||
if branch_index == 1:
|
||||
partial.retain_grad()
|
||||
trace.block_outputs.append(partial)
|
||||
if mixer_index % parent.round04.SUBLAYERS_PER_BLOCK == 0:
|
||||
completed.append(partial)
|
||||
partial = None
|
||||
if partial is not None or len(completed) != parent.BLOCK_GROUPS + 1:
|
||||
raise RuntimeError("Round 08 Block AttnRes aggregation failed")
|
||||
if self.output_mixer is None:
|
||||
raise RuntimeError("Round 08 output mixer missing")
|
||||
self.output_visits += 1
|
||||
hidden, output_weights = self.output_mixer(completed, capture)
|
||||
if trace is not None:
|
||||
trace.output_weights = output_weights
|
||||
normalized = self.final_norm(hidden)
|
||||
logits = F.linear(normalized, self.token_embedding.weight)
|
||||
return logits, trace
|
||||
|
||||
|
||||
def tensor_exact(left: torch.Tensor, right: torch.Tensor) -> bool:
|
||||
return (
|
||||
left.dtype == right.dtype
|
||||
and tuple(left.shape) == tuple(right.shape)
|
||||
and torch.equal(left.detach().cpu(), right.detach().cpu())
|
||||
)
|
||||
|
||||
|
||||
def build_intervention_audit(
|
||||
model: ForwardInterventionLanguageModel,
|
||||
optimizer: torch.optim.Optimizer,
|
||||
study_manifest: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
selected = tuple(model.selected_indices)
|
||||
selected_names = set(parameter_names_for_indices(selected))
|
||||
mixer_parameters = {
|
||||
name: parameter
|
||||
for name, parameter in model.named_parameters()
|
||||
if name.startswith("mixers.") or name.startswith("output_mixer.")
|
||||
}
|
||||
optimizer_parameters = {
|
||||
parameter
|
||||
for group in optimizer.param_groups
|
||||
for parameter in group["params"]
|
||||
}
|
||||
selected_parameter_checks = {}
|
||||
for name in sorted(selected_names):
|
||||
parameter = mixer_parameters[name]
|
||||
selected_parameter_checks[name] = {
|
||||
"gradient_hook_calls": model.gradient_hook_calls[name],
|
||||
"in_optimizer_param_group": parameter in optimizer_parameters,
|
||||
"optimizer_state_present": parameter in optimizer.state,
|
||||
"final_equals_initial": tensor_exact(
|
||||
parameter, model.selected_initial_tensors[name]
|
||||
),
|
||||
}
|
||||
unselected_parameter_checks = {}
|
||||
for name, parameter in sorted(mixer_parameters.items()):
|
||||
if name in selected_names:
|
||||
continue
|
||||
unselected_parameter_checks[name] = {
|
||||
"gradient_hook_calls": model.gradient_hook_calls[name],
|
||||
"in_optimizer_param_group": parameter in optimizer_parameters,
|
||||
"optimizer_state_present": parameter in optimizer.state,
|
||||
}
|
||||
|
||||
source_counts = {
|
||||
str(index): sorted(values)
|
||||
for index, values in model.source_counts.items()
|
||||
}
|
||||
selected_source_gate = {
|
||||
str(index): (
|
||||
source_counts[str(index)]
|
||||
== [study_manifest["selected_source_counts"][str(index)]]
|
||||
== [EXPECTED_SOURCE_COUNTS[index]]
|
||||
)
|
||||
for index in selected
|
||||
}
|
||||
visit_gate = (
|
||||
model.forward_calls > 0
|
||||
and all(value == model.forward_calls for value in model.depth_visits)
|
||||
and model.output_visits == model.forward_calls
|
||||
)
|
||||
selected_parameter_gate = all(
|
||||
check["gradient_hook_calls"] == 0
|
||||
and check["in_optimizer_param_group"]
|
||||
and not check["optimizer_state_present"]
|
||||
and check["final_equals_initial"]
|
||||
for check in selected_parameter_checks.values()
|
||||
)
|
||||
unselected_parameter_gate = all(
|
||||
check["gradient_hook_calls"] > 0
|
||||
and check["in_optimizer_param_group"]
|
||||
and check["optimizer_state_present"]
|
||||
for check in unselected_parameter_checks.values()
|
||||
)
|
||||
expected_selected = tuple(
|
||||
study_manifest["variants"]
|
||||
.get(model.forward_variant, {"selected_depth_indices": []})[
|
||||
"selected_depth_indices"
|
||||
]
|
||||
)
|
||||
selector_gate = (
|
||||
selected == expected_selected
|
||||
and 64 not in selected
|
||||
and selected_source_gate == {
|
||||
str(index): True for index in selected
|
||||
}
|
||||
)
|
||||
threshold = study_manifest["thresholds"][
|
||||
"uniform_weight_max_abs_error"
|
||||
]
|
||||
uniform_gate = model.uniform_weight_max_abs_error <= threshold
|
||||
passed = (
|
||||
visit_gate
|
||||
and selector_gate
|
||||
and selected_parameter_gate
|
||||
and unselected_parameter_gate
|
||||
and uniform_gate
|
||||
)
|
||||
return {
|
||||
"passed": passed,
|
||||
"variant": model.forward_variant,
|
||||
"selected_depth_indices": list(selected),
|
||||
"output_mixer_selected": False,
|
||||
"forward_calls": model.forward_calls,
|
||||
"depth_visit_counts": model.depth_visits,
|
||||
"output_visit_count": model.output_visits,
|
||||
"visit_gate": visit_gate,
|
||||
"source_counts_by_depth_index": source_counts,
|
||||
"selected_source_count_checks": selected_source_gate,
|
||||
"selector_gate": selector_gate,
|
||||
"uniform_weight_max_abs_error": model.uniform_weight_max_abs_error,
|
||||
"uniform_weight_threshold": threshold,
|
||||
"uniform_weight_gate": uniform_gate,
|
||||
"selected_parameters": selected_parameter_checks,
|
||||
"selected_parameter_reachability_gate": selected_parameter_gate,
|
||||
"unselected_parameters": unselected_parameter_checks,
|
||||
"unselected_parameter_reachability_gate": unselected_parameter_gate,
|
||||
"semantics": (
|
||||
"selected depth mixers use parameter-free constant-zero logits "
|
||||
"with the parent softmax+einsum arithmetic kernel"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def rewrite_result(
|
||||
output_path: Path,
|
||||
study_manifest_path: Path,
|
||||
study_manifest: dict[str, Any],
|
||||
) -> None:
|
||||
if LAST_MODEL is None or LAST_OPTIMIZER is None:
|
||||
raise RuntimeError("runner capture state missing")
|
||||
result = json.loads(output_path.read_text())
|
||||
parent_self_hash = result.pop("canonical_sha256_without_self")
|
||||
if result["protocol_id"] != PARENT_PROTOCOL_ID:
|
||||
raise RuntimeError("parent runner protocol drift")
|
||||
result["schema_version"] = 2
|
||||
result["protocol_id"] = PROTOCOL_ID
|
||||
result["parent_protocol_id"] = PARENT_PROTOCOL_ID
|
||||
result["variant"] = ACTIVE_VARIANT
|
||||
result["parent_runner_canonical_sha256"] = parent_self_hash
|
||||
result["study_manifest"] = {
|
||||
"path": str(study_manifest_path),
|
||||
"file_sha256": parent.file_sha256(study_manifest_path),
|
||||
"status": study_manifest["status"],
|
||||
}
|
||||
result["forward_intervention"] = build_intervention_audit(
|
||||
LAST_MODEL, LAST_OPTIMIZER, study_manifest
|
||||
)
|
||||
result["canonical_sha256_without_self"] = parent.canonical_sha256(result)
|
||||
temporary = output_path.with_suffix(output_path.suffix + ".round08.tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
os.replace(temporary, output_path)
|
||||
if not result["forward_intervention"]["passed"]:
|
||||
raise RuntimeError(
|
||||
f"forward intervention audit failed: "
|
||||
f"{result['forward_intervention']}"
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
global ACTIVE_VARIANT, LAST_OPTIMIZER
|
||||
variant = extract_wrapper_argument("--variant")
|
||||
study_manifest_path = Path(
|
||||
extract_wrapper_argument("--study-manifest")
|
||||
).resolve()
|
||||
if variant not in VARIANTS:
|
||||
raise ValueError(f"unknown variant: {variant}")
|
||||
run_kind = argument_value("--run-kind", "formal")
|
||||
if run_kind in ("formal", "replay") and variant not in FORMAL_VARIANTS:
|
||||
raise ValueError("learned_reference is smoke-only")
|
||||
if argument_value("--architecture") != "block":
|
||||
raise ValueError("Round 08 requires --architecture block")
|
||||
if argument_value("--depth") != "32":
|
||||
raise ValueError("Round 08 requires --depth 32")
|
||||
if run_kind == "replay" and variant != "uniform_groups_6_7_forward":
|
||||
raise ValueError("the frozen replay uses the primary joint variant")
|
||||
|
||||
study_manifest = json.loads(study_manifest_path.read_text())
|
||||
if (
|
||||
study_manifest["protocol_id"] != PROTOCOL_ID
|
||||
or study_manifest["status"] != "frozen-before-model-output"
|
||||
):
|
||||
raise ValueError("study manifest is not the frozen Round 08 contract")
|
||||
expected = tuple(
|
||||
study_manifest["variants"]
|
||||
.get(variant, {"selected_depth_indices": []})[
|
||||
"selected_depth_indices"
|
||||
]
|
||||
)
|
||||
if expected != VARIANTS[variant]:
|
||||
raise ValueError("study manifest selector drift")
|
||||
for index, source_count in EXPECTED_SOURCE_COUNTS.items():
|
||||
if (
|
||||
study_manifest["selected_source_counts"].get(str(index))
|
||||
!= source_count
|
||||
):
|
||||
raise ValueError(
|
||||
f"study manifest source-count drift at depth index {index}"
|
||||
)
|
||||
|
||||
output_value = argument_value("--output")
|
||||
if output_value is None:
|
||||
raise ValueError("--output is required")
|
||||
output_path = Path(output_value).resolve()
|
||||
ACTIVE_VARIANT = variant
|
||||
parent.GradientLanguageModel = ForwardInterventionLanguageModel
|
||||
|
||||
original_adamw = torch.optim.AdamW
|
||||
|
||||
def capture_adamw(*args: Any, **kwargs: Any) -> torch.optim.Optimizer:
|
||||
global LAST_OPTIMIZER
|
||||
LAST_OPTIMIZER = original_adamw(*args, **kwargs)
|
||||
return LAST_OPTIMIZER
|
||||
|
||||
torch.optim.AdamW = capture_adamw # type: ignore[assignment]
|
||||
try:
|
||||
parent.main()
|
||||
finally:
|
||||
torch.optim.AdamW = original_adamw # type: ignore[assignment]
|
||||
rewrite_result(output_path, study_manifest_path, study_manifest)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,294 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run pre-result Round 08 identity gates."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
def load_runner() -> Any:
|
||||
path = Path(__file__).resolve().parent / "train.py"
|
||||
spec = importlib.util.spec_from_file_location("k3_attnres_round08_train", path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"cannot import Round 08 runner from {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
runner = load_runner()
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
step_zero = subparsers.add_parser("step-zero")
|
||||
step_zero.add_argument("--cache-dir", type=Path, required=True)
|
||||
step_zero.add_argument("--parent-manifest", type=Path, required=True)
|
||||
step_zero.add_argument("--study-manifest", type=Path, required=True)
|
||||
step_zero.add_argument("--output", type=Path, required=True)
|
||||
step_zero.add_argument("--seed", type=int, default=2026073001)
|
||||
|
||||
smoke = subparsers.add_parser("smoke-compare")
|
||||
smoke.add_argument("--parent", type=Path, required=True)
|
||||
smoke.add_argument("--wrapper", type=Path, required=True)
|
||||
smoke.add_argument("--output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def tensor_sha256(value: torch.Tensor) -> str:
|
||||
return hashlib.sha256(runner.parent.tensor_bytes(value)).hexdigest()
|
||||
|
||||
|
||||
def read_and_verify(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text())
|
||||
expected = value["canonical_sha256_without_self"]
|
||||
payload = {
|
||||
key: item
|
||||
for key, item in value.items()
|
||||
if key != "canonical_sha256_without_self"
|
||||
}
|
||||
if canonical_sha256(payload) != expected:
|
||||
raise RuntimeError(f"canonical self-hash failed: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def smoke_compare(args: argparse.Namespace) -> None:
|
||||
parent_result = read_and_verify(args.parent)
|
||||
wrapper_result = read_and_verify(args.wrapper)
|
||||
fields = (
|
||||
"architecture",
|
||||
"depth",
|
||||
"seed",
|
||||
"steps",
|
||||
"batch_size",
|
||||
"target_bytes_seen",
|
||||
"manifest",
|
||||
"model",
|
||||
"optimizer",
|
||||
"hashes",
|
||||
"evaluations",
|
||||
"diagnostics",
|
||||
"training_history",
|
||||
"gradient_gate",
|
||||
"environment",
|
||||
)
|
||||
checks = {}
|
||||
for field in fields:
|
||||
parent_value = exact_structure(parent_result[field])
|
||||
wrapper_value = exact_structure(wrapper_result[field])
|
||||
if field == "manifest":
|
||||
parent_value.pop("path", None)
|
||||
wrapper_value.pop("path", None)
|
||||
checks[field] = parent_value == wrapper_value
|
||||
wrapper_identity = {
|
||||
"protocol": wrapper_result["protocol_id"] == runner.PROTOCOL_ID,
|
||||
"parent_protocol": (
|
||||
wrapper_result["parent_protocol_id"]
|
||||
== runner.PARENT_PROTOCOL_ID
|
||||
),
|
||||
"variant": wrapper_result["variant"] == "learned_reference",
|
||||
"forward_audit": wrapper_result["forward_intervention"]["passed"],
|
||||
}
|
||||
passed = all(checks.values()) and all(wrapper_identity.values())
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": runner.PROTOCOL_ID,
|
||||
"gate": "empty-selector-parent-equivalence",
|
||||
"passed": passed,
|
||||
"field_checks": checks,
|
||||
"wrapper_identity": wrapper_identity,
|
||||
"excluded_fields": [
|
||||
"protocol wrapper fields",
|
||||
"timing",
|
||||
"self hash",
|
||||
"parent runner self hash",
|
||||
"study manifest",
|
||||
],
|
||||
"parent_file": str(args.parent.resolve()),
|
||||
"wrapper_file": str(args.wrapper.resolve()),
|
||||
}
|
||||
result["canonical_sha256_without_self"] = canonical_sha256(result)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
if not passed:
|
||||
raise RuntimeError(f"empty-selector parent equivalence failed: {checks}")
|
||||
|
||||
|
||||
def exact_structure(value: Any) -> Any:
|
||||
return json.loads(
|
||||
json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||
)
|
||||
|
||||
|
||||
def step_zero(args: argparse.Namespace) -> None:
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required by the frozen step-zero gate")
|
||||
study_manifest = json.loads(args.study_manifest.read_text())
|
||||
if study_manifest["protocol_id"] != runner.PROTOCOL_ID:
|
||||
raise RuntimeError("study manifest mismatch")
|
||||
parent_manifest = json.loads(args.parent_manifest.read_text())
|
||||
if parent_manifest["protocol_id"] != runner.PARENT_PROTOCOL_ID:
|
||||
raise RuntimeError("parent manifest mismatch")
|
||||
|
||||
parent = runner.parent
|
||||
parent.configure_round04_globals(32)
|
||||
device = torch.device("cuda")
|
||||
corpus = parent.round04.ByteCorpus(
|
||||
args.cache_dir, parent_manifest, device
|
||||
)
|
||||
inputs, targets = corpus.fixed_batch(
|
||||
corpus.diagnostic_starts, 0, 16
|
||||
)
|
||||
|
||||
variants = ("learned_reference",) + tuple(
|
||||
study_manifest["variants"].keys()
|
||||
)
|
||||
observations: dict[str, Any] = {}
|
||||
reference_payload: dict[str, Any] | None = None
|
||||
for variant in variants:
|
||||
parent.configure_determinism(args.seed)
|
||||
runner.ACTIVE_VARIANT = variant
|
||||
model = runner.ForwardInterventionLanguageModel("block").to(device)
|
||||
initial_public = parent.named_state_hash(
|
||||
model, include_mixers=False
|
||||
)
|
||||
initial_mixer = parent.named_state_hash(
|
||||
model, include_mixers=True
|
||||
)
|
||||
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
||||
logits, trace = model(inputs, capture=True)
|
||||
loss = parent.cross_entropy(logits, targets)
|
||||
if trace is None:
|
||||
raise RuntimeError("step-zero trace missing")
|
||||
evaluation = parent.evaluate(model, corpus, 64, 8)
|
||||
diagnostic = parent.diagnostic(model, corpus, 16)
|
||||
payload = {
|
||||
"initial_public_hash": initial_public,
|
||||
"initial_mixer_hash": initial_mixer,
|
||||
"logits_sha256": tensor_sha256(logits),
|
||||
"loss_nats": loss.detach().cpu().item(),
|
||||
"loss_tensor_sha256": tensor_sha256(loss),
|
||||
"evaluation": exact_structure(evaluation),
|
||||
"diagnostic": exact_structure(diagnostic),
|
||||
}
|
||||
if reference_payload is None:
|
||||
reference_payload = payload
|
||||
exact_checks = {
|
||||
key: payload[key] == reference_payload[key]
|
||||
for key in payload
|
||||
}
|
||||
selected = tuple(runner.VARIANTS[variant])
|
||||
capture_checks = {}
|
||||
for index in selected:
|
||||
summary = trace.depth_weights[index]
|
||||
source_count = summary["sources"]
|
||||
capture_checks[str(index)] = {
|
||||
"source_count": source_count,
|
||||
"expected_source_count": study_manifest[
|
||||
"selected_source_counts"
|
||||
][str(index)],
|
||||
"capture_summary_exact_vs_learned": (
|
||||
payload["diagnostic"]["depth_weights"][index]
|
||||
== reference_payload["diagnostic"]["depth_weights"][index]
|
||||
),
|
||||
"passed": (
|
||||
source_count
|
||||
== study_manifest["selected_source_counts"][str(index)]
|
||||
and payload["diagnostic"]["depth_weights"][index]
|
||||
== reference_payload["diagnostic"]["depth_weights"][index]
|
||||
),
|
||||
}
|
||||
runtime_uniform_gate = (
|
||||
model.uniform_weight_max_abs_error
|
||||
<= study_manifest["thresholds"][
|
||||
"uniform_weight_max_abs_error"
|
||||
]
|
||||
)
|
||||
observations[variant] = {
|
||||
"payload": payload,
|
||||
"exact_vs_learned_reference": exact_checks,
|
||||
"selected_capture_checks": capture_checks,
|
||||
"pre_reduction_uniform_weight_max_abs_error": (
|
||||
model.uniform_weight_max_abs_error
|
||||
),
|
||||
"pre_reduction_uniform_weight_gate": runtime_uniform_gate,
|
||||
"passed": (
|
||||
all(exact_checks.values())
|
||||
and all(
|
||||
item["passed"] for item in capture_checks.values()
|
||||
)
|
||||
and runtime_uniform_gate
|
||||
),
|
||||
}
|
||||
del model, logits, loss, trace
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
passed = all(item["passed"] for item in observations.values())
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": runner.PROTOCOL_ID,
|
||||
"gate": "step-zero-cross-variant-byte-exact",
|
||||
"seed": args.seed,
|
||||
"passed": passed,
|
||||
"variants": observations,
|
||||
"parent_manifest_sha256": runner.parent.file_sha256(
|
||||
args.parent_manifest
|
||||
),
|
||||
"study_manifest_sha256": runner.parent.file_sha256(
|
||||
args.study_manifest
|
||||
),
|
||||
"environment": {
|
||||
"gpu": torch.cuda.get_device_name(0),
|
||||
"torch": torch.__version__,
|
||||
"cuda": torch.version.cuda,
|
||||
"cublas_workspace_config": os.environ.get(
|
||||
"CUBLAS_WORKSPACE_CONFIG"
|
||||
),
|
||||
},
|
||||
}
|
||||
result["canonical_sha256_without_self"] = canonical_sha256(result)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
if not passed:
|
||||
failed = [
|
||||
name
|
||||
for name, value in observations.items()
|
||||
if not value["passed"]
|
||||
]
|
||||
raise RuntimeError(f"step-zero exactness failed: {failed}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if args.command == "step-zero":
|
||||
step_zero(args)
|
||||
else:
|
||||
smoke_compare(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,57 @@
|
||||
# Attention Residuals activation-gradient/depth study
|
||||
|
||||
This directory implements preregistered protocol
|
||||
`llm-atlas-k3-attnres-gradient-scale-v1`:
|
||||
|
||||
- `research/K3_ATTNRES_GRADIENT_DEFINITION_AUDIT.md`
|
||||
- `research/K3_ATTNRES_GRADIENT_SCALE_PROTOCOL.md`
|
||||
|
||||
It is an independent reduced mechanism experiment. It is not a Kimi K3
|
||||
checkpoint forward pass and does not claim to recover the paper's unpublished
|
||||
Figure 5 telemetry definition.
|
||||
|
||||
## Frozen environment
|
||||
|
||||
```text
|
||||
Python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python
|
||||
PyTorch 2.11.0+cu128
|
||||
GPU NVIDIA GeForce RTX 5090
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8
|
||||
```
|
||||
|
||||
## Build the manifest
|
||||
|
||||
```bash
|
||||
python experiments/k3/attnres_gradient/build_dataset.py \
|
||||
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
||||
--manifest experiments/k3/attnres_gradient/manifest.json
|
||||
```
|
||||
|
||||
## Run a smoke cell
|
||||
|
||||
```bash
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8 \
|
||||
python experiments/k3/attnres_gradient/train.py \
|
||||
--run-kind smoke \
|
||||
--architecture block \
|
||||
--depth 32 \
|
||||
--seed 2026073001 \
|
||||
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
||||
--manifest experiments/k3/attnres_gradient/manifest.json \
|
||||
--output /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1/smoke-a/depth-32-block.json
|
||||
```
|
||||
|
||||
Smoke is fixed to 20 steps. Formal and replay runs are fixed to 8,000 steps;
|
||||
the runner rejects alternative budgets. The same command uses
|
||||
`--run-kind formal` or `--run-kind replay` and omits an explicit `--steps`.
|
||||
|
||||
Formal output keys use:
|
||||
|
||||
```text
|
||||
formal/depth-{16|32}-{baseline|block}-seed-{seed}.json
|
||||
replay/depth-32-block-seed-2026073001.json
|
||||
```
|
||||
|
||||
Raw parquet/binary files and working runs remain in the local cache. The
|
||||
manifest, runner, complete result JSON, compact website payload, reproduction
|
||||
hashes, protocol, and audit enter the public repository.
|
||||
@@ -0,0 +1,685 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate, aggregate, and publish K3 AttnRes Round 05 experiment data."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import shutil
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-gradient-scale-v1"
|
||||
ARCHITECTURES = ("baseline", "block")
|
||||
DEPTHS = (16, 32)
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
STEPS = (0, 100, 500, 2000, 4000, 8000)
|
||||
FORMAL_STEPS = 8000
|
||||
FORMAL_BATCH = 32
|
||||
TARGET_BYTES_PER_RUN = 65_536_000
|
||||
EXPECTED_TOTAL_TARGET_BYTES = 786_432_000
|
||||
SMOKE_COMPARE_FIELDS = (
|
||||
"protocol_id",
|
||||
"run_kind",
|
||||
"architecture",
|
||||
"depth",
|
||||
"seed",
|
||||
"steps",
|
||||
"batch_size",
|
||||
"target_bytes_seen",
|
||||
"manifest",
|
||||
"model",
|
||||
"optimizer",
|
||||
"hashes",
|
||||
"evaluations",
|
||||
"diagnostics",
|
||||
"training_history",
|
||||
"gradient_gate",
|
||||
"environment",
|
||||
)
|
||||
REPLAY_COMPARE_FIELDS = tuple(
|
||||
field for field in SMOKE_COMPARE_FIELDS if field != "run_kind"
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--formal-dir", type=Path, required=True)
|
||||
parser.add_argument("--smoke-a-dir", type=Path, required=True)
|
||||
parser.add_argument("--smoke-b-dir", type=Path, required=True)
|
||||
parser.add_argument("--replay", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--raw-output-dir", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--compact-output", type=Path, required=True)
|
||||
parser.add_argument("--reproduction-output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def read_json(path: Path) -> dict[str, Any]:
|
||||
return json.loads(path.read_text())
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def atomic_json(path: Path, value: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def mean(values: Iterable[float]) -> float:
|
||||
return statistics.fmean(values)
|
||||
|
||||
|
||||
def require_finite(value: Any, path: str = "root") -> None:
|
||||
if isinstance(value, float):
|
||||
if not math.isfinite(value):
|
||||
raise ValueError(f"non-finite float at {path}")
|
||||
elif isinstance(value, dict):
|
||||
for key, child in value.items():
|
||||
require_finite(child, f"{path}.{key}")
|
||||
elif isinstance(value, list):
|
||||
for index, child in enumerate(value):
|
||||
require_finite(child, f"{path}[{index}]")
|
||||
|
||||
|
||||
def selected(run: dict[str, Any], fields: tuple[str, ...]) -> dict[str, Any]:
|
||||
return {field: run[field] for field in fields}
|
||||
|
||||
|
||||
def final_diagnostic(run: dict[str, Any]) -> dict[str, Any]:
|
||||
diagnostic = run["diagnostics"][-1]
|
||||
if diagnostic["step"] != FORMAL_STEPS:
|
||||
raise ValueError("final diagnostic is not step 8000")
|
||||
return diagnostic
|
||||
|
||||
|
||||
def validate_run(
|
||||
run: dict[str, Any],
|
||||
*,
|
||||
path: Path,
|
||||
depth: int,
|
||||
architecture: str,
|
||||
seed: int,
|
||||
manifest: dict[str, Any],
|
||||
manifest_hash: str,
|
||||
) -> None:
|
||||
if run["protocol_id"] != PROTOCOL_ID or run["run_kind"] != "formal":
|
||||
raise ValueError(f"formal protocol/kind mismatch: {path}")
|
||||
if (
|
||||
run["depth"] != depth
|
||||
or run["architecture"] != architecture
|
||||
or run["seed"] != seed
|
||||
):
|
||||
raise ValueError(f"formal identity mismatch: {path}")
|
||||
if (
|
||||
run["steps"] != FORMAL_STEPS
|
||||
or run["batch_size"] != FORMAL_BATCH
|
||||
or run["target_bytes_seen"] != TARGET_BYTES_PER_RUN
|
||||
):
|
||||
raise ValueError(f"formal budget mismatch: {path}")
|
||||
if run["manifest"]["file_sha256"] != manifest_hash:
|
||||
raise ValueError(f"manifest file hash mismatch: {path}")
|
||||
for key in (
|
||||
"formal_schedule_sha256",
|
||||
"validation_tensor_sha256",
|
||||
"diagnostic_tensor_sha256",
|
||||
):
|
||||
if run["manifest"][key] != manifest["windows"][key]:
|
||||
raise ValueError(f"manifest {key} mismatch: {path}")
|
||||
if [row["step"] for row in run["evaluations"]] != list(STEPS):
|
||||
raise ValueError(f"evaluation steps mismatch: {path}")
|
||||
if [row["step"] for row in run["diagnostics"]] != list(STEPS):
|
||||
raise ValueError(f"diagnostic steps mismatch: {path}")
|
||||
if run["model"]["layers"] != depth:
|
||||
raise ValueError(f"model depth mismatch: {path}")
|
||||
|
||||
for diagnostic in run["diagnostics"]:
|
||||
capture = diagnostic["capture"]
|
||||
if (
|
||||
capture["count"] != depth
|
||||
or capture["shape"] != [16, 256, 192]
|
||||
or set(capture["dtypes"]) != {"torch.float32"}
|
||||
or not capture["all_gradients_finite"]
|
||||
or not capture["all_gradients_present"]
|
||||
or not capture["storage_unique"]
|
||||
):
|
||||
raise ValueError(f"activation capture gate mismatch: {path}")
|
||||
for field in (
|
||||
"activation_grad_rms_by_block",
|
||||
"activation_output_rms_by_block",
|
||||
"core_parameter_grad_rms_by_block",
|
||||
):
|
||||
if len(diagnostic[field]) != depth:
|
||||
raise ValueError(f"{field} length mismatch: {path}")
|
||||
for field in (
|
||||
"layer_input_rms_by_sublayer",
|
||||
"branch_output_rms_by_sublayer",
|
||||
"stream_state_rms_by_sublayer",
|
||||
):
|
||||
if len(diagnostic[field]) != depth * 2:
|
||||
raise ValueError(f"{field} length mismatch: {path}")
|
||||
if architecture == "baseline":
|
||||
if diagnostic["depth_weights"] or diagnostic["output_weights"] is not None:
|
||||
raise ValueError(f"unexpected Baseline mixer trace: {path}")
|
||||
else:
|
||||
if (
|
||||
len(diagnostic["depth_weights"]) != depth * 2
|
||||
or diagnostic["output_weights"]["sources"] != 9
|
||||
):
|
||||
raise ValueError(f"Block mixer trace mismatch: {path}")
|
||||
require_finite(run, path.name)
|
||||
|
||||
|
||||
def relative_reduction(baseline: float, block: float) -> float:
|
||||
return (baseline - block) / baseline
|
||||
|
||||
|
||||
def depth_verdict(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
cv_reductions = [row["relative_cv_reduction"] for row in rows]
|
||||
imbalance_reductions = [
|
||||
row["relative_imbalance_reduction"] for row in rows
|
||||
]
|
||||
mean_cv = mean(cv_reductions)
|
||||
mean_imbalance = mean(imbalance_reductions)
|
||||
support = (
|
||||
all(value > 0 for value in cv_reductions)
|
||||
and mean_cv >= 0.20
|
||||
and all(value > 0 for value in imbalance_reductions)
|
||||
and mean_imbalance >= 0.20
|
||||
)
|
||||
concern = (
|
||||
all(value < 0 for value in cv_reductions)
|
||||
and -mean_cv >= 0.20
|
||||
and all(value < 0 for value in imbalance_reductions)
|
||||
and -mean_imbalance >= 0.20
|
||||
)
|
||||
if support:
|
||||
label = "joint directional support at this depth"
|
||||
elif concern:
|
||||
label = "joint directional concern at this depth"
|
||||
else:
|
||||
label = "mixed / inconclusive at this depth"
|
||||
return {
|
||||
"label": label,
|
||||
"threshold_relative": 0.20,
|
||||
"cv_reductions": cv_reductions,
|
||||
"mean_cv_reduction": mean_cv,
|
||||
"imbalance_reductions": imbalance_reductions,
|
||||
"mean_imbalance_reduction": mean_imbalance,
|
||||
"all_cv_improve": all(value > 0 for value in cv_reductions),
|
||||
"all_imbalance_improve": all(
|
||||
value > 0 for value in imbalance_reductions
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def summarize_depth(
|
||||
depth: int, runs: dict[tuple[int, str, int], dict[str, Any]]
|
||||
) -> dict[str, Any]:
|
||||
rows = []
|
||||
for seed in SEEDS:
|
||||
baseline = runs[(depth, "baseline", seed)]
|
||||
block = runs[(depth, "block", seed)]
|
||||
baseline_diagnostic = final_diagnostic(baseline)
|
||||
block_diagnostic = final_diagnostic(block)
|
||||
baseline_activation = baseline_diagnostic[
|
||||
"activation_grad_statistics"
|
||||
]
|
||||
block_activation = block_diagnostic["activation_grad_statistics"]
|
||||
baseline_bpc = baseline["evaluations"][-1]["bits_per_byte"]
|
||||
block_bpc = block["evaluations"][-1]["bits_per_byte"]
|
||||
rows.append(
|
||||
{
|
||||
"seed": seed,
|
||||
"baseline_bpc": baseline_bpc,
|
||||
"block_bpc": block_bpc,
|
||||
"block_minus_baseline_bpc": block_bpc - baseline_bpc,
|
||||
"baseline_activation_grad_mean": baseline_activation["mean"],
|
||||
"block_activation_grad_mean": block_activation["mean"],
|
||||
"block_to_baseline_activation_grad_mean": (
|
||||
block_activation["mean"] / baseline_activation["mean"]
|
||||
),
|
||||
"baseline_cv": baseline_activation["population_cv"],
|
||||
"block_cv": block_activation["population_cv"],
|
||||
"relative_cv_reduction": relative_reduction(
|
||||
baseline_activation["population_cv"],
|
||||
block_activation["population_cv"],
|
||||
),
|
||||
"baseline_first_to_last_ratio": baseline_activation[
|
||||
"first_to_last_ratio"
|
||||
],
|
||||
"block_first_to_last_ratio": block_activation[
|
||||
"first_to_last_ratio"
|
||||
],
|
||||
"baseline_imbalance": baseline_activation[
|
||||
"imbalance_abs_log_ratio"
|
||||
],
|
||||
"block_imbalance": block_activation[
|
||||
"imbalance_abs_log_ratio"
|
||||
],
|
||||
"relative_imbalance_reduction": relative_reduction(
|
||||
baseline_activation["imbalance_abs_log_ratio"],
|
||||
block_activation["imbalance_abs_log_ratio"],
|
||||
),
|
||||
"baseline_parameter_grad_cv": baseline_diagnostic[
|
||||
"core_parameter_grad_statistics"
|
||||
]["population_cv"],
|
||||
"block_parameter_grad_cv": block_diagnostic[
|
||||
"core_parameter_grad_statistics"
|
||||
]["population_cv"],
|
||||
}
|
||||
)
|
||||
verdict = depth_verdict(rows)
|
||||
return {
|
||||
"depth": depth,
|
||||
"by_seed": rows,
|
||||
"means": {
|
||||
"baseline_bpc": mean(row["baseline_bpc"] for row in rows),
|
||||
"block_bpc": mean(row["block_bpc"] for row in rows),
|
||||
"block_minus_baseline_bpc": mean(
|
||||
row["block_minus_baseline_bpc"] for row in rows
|
||||
),
|
||||
"baseline_cv": mean(row["baseline_cv"] for row in rows),
|
||||
"block_cv": mean(row["block_cv"] for row in rows),
|
||||
"relative_cv_reduction": mean(
|
||||
row["relative_cv_reduction"] for row in rows
|
||||
),
|
||||
"baseline_imbalance": mean(
|
||||
row["baseline_imbalance"] for row in rows
|
||||
),
|
||||
"block_imbalance": mean(
|
||||
row["block_imbalance"] for row in rows
|
||||
),
|
||||
"relative_imbalance_reduction": mean(
|
||||
row["relative_imbalance_reduction"] for row in rows
|
||||
),
|
||||
"block_to_baseline_activation_grad_mean": mean(
|
||||
row["block_to_baseline_activation_grad_mean"] for row in rows
|
||||
),
|
||||
"baseline_parameter_grad_cv": mean(
|
||||
row["baseline_parameter_grad_cv"] for row in rows
|
||||
),
|
||||
"block_parameter_grad_cv": mean(
|
||||
row["block_parameter_grad_cv"] for row in rows
|
||||
),
|
||||
},
|
||||
"verdict": verdict,
|
||||
}
|
||||
|
||||
|
||||
def compact_cell(run: dict[str, Any]) -> dict[str, Any]:
|
||||
diagnostics = []
|
||||
for row in run["diagnostics"]:
|
||||
compact = {
|
||||
"step": row["step"],
|
||||
"loss_nats": row["loss_nats"],
|
||||
"bits_per_byte": row["bits_per_byte"],
|
||||
"activation_grad_rms_by_block": row[
|
||||
"activation_grad_rms_by_block"
|
||||
],
|
||||
"activation_grad_statistics": row[
|
||||
"activation_grad_statistics"
|
||||
],
|
||||
"activation_output_rms_by_block": row[
|
||||
"activation_output_rms_by_block"
|
||||
],
|
||||
"activation_output_statistics": row[
|
||||
"activation_output_statistics"
|
||||
],
|
||||
"core_parameter_grad_rms_by_block": row[
|
||||
"core_parameter_grad_rms_by_block"
|
||||
],
|
||||
"core_parameter_grad_statistics": row[
|
||||
"core_parameter_grad_statistics"
|
||||
],
|
||||
}
|
||||
if row["depth_weights"]:
|
||||
compact["depth_weights"] = row["depth_weights"]
|
||||
compact["output_weights"] = row["output_weights"]
|
||||
diagnostics.append(compact)
|
||||
return {
|
||||
"architecture": run["architecture"],
|
||||
"depth": run["depth"],
|
||||
"seed": run["seed"],
|
||||
"evaluations": run["evaluations"],
|
||||
"diagnostics": diagnostics,
|
||||
"timing": run["timing"],
|
||||
"parameters": run["model"]["parameters"],
|
||||
"hashes": run["hashes"],
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
manifest = read_json(args.manifest)
|
||||
manifest_hash = file_sha256(args.manifest)
|
||||
if manifest["protocol_id"] != PROTOCOL_ID:
|
||||
raise ValueError("manifest protocol mismatch")
|
||||
if (
|
||||
manifest["windows"]["formal_schedule_cells"] != 768_000
|
||||
or manifest["windows"]["formal_steps"] != FORMAL_STEPS
|
||||
or manifest["windows"]["formal_batch"] != FORMAL_BATCH
|
||||
):
|
||||
raise ValueError("manifest schedule budget mismatch")
|
||||
|
||||
runs: dict[tuple[int, str, int], dict[str, Any]] = {}
|
||||
source_paths: dict[str, Path] = {}
|
||||
formal_hashes: dict[str, str] = {}
|
||||
for depth in DEPTHS:
|
||||
for architecture in ARCHITECTURES:
|
||||
for seed in SEEDS:
|
||||
name = f"depth-{depth}-{architecture}-seed-{seed}.json"
|
||||
path = args.formal_dir / name
|
||||
run = read_json(path)
|
||||
validate_run(
|
||||
run,
|
||||
path=path,
|
||||
depth=depth,
|
||||
architecture=architecture,
|
||||
seed=seed,
|
||||
manifest=manifest,
|
||||
manifest_hash=manifest_hash,
|
||||
)
|
||||
runs[(depth, architecture, seed)] = run
|
||||
public_name = f"formal-{name}"
|
||||
source_paths[public_name] = path
|
||||
formal_hashes[public_name] = file_sha256(path)
|
||||
|
||||
if sum(run["target_bytes_seen"] for run in runs.values()) != (
|
||||
EXPECTED_TOTAL_TARGET_BYTES
|
||||
):
|
||||
raise ValueError("formal total target-byte budget mismatch")
|
||||
|
||||
common_initial_exact: dict[str, Any] = {}
|
||||
input_gate_exact: dict[str, Any] = {}
|
||||
for depth in DEPTHS:
|
||||
for seed in SEEDS:
|
||||
baseline = runs[(depth, "baseline", seed)]
|
||||
block = runs[(depth, "block", seed)]
|
||||
public_fields = (
|
||||
"initial_public_parameter_structure",
|
||||
"initial_public_parameter_tensors",
|
||||
"initial_public_parameter_elements",
|
||||
"initial_public_parameters",
|
||||
)
|
||||
exact = all(
|
||||
baseline["hashes"][field] == block["hashes"][field]
|
||||
for field in public_fields
|
||||
)
|
||||
gate_exact = (
|
||||
baseline["manifest"]["input_gate_tensor_hashes"]
|
||||
== block["manifest"]["input_gate_tensor_hashes"]
|
||||
)
|
||||
key = f"depth-{depth}-seed-{seed}"
|
||||
common_initial_exact[key] = {
|
||||
"exact": exact,
|
||||
"baseline": {
|
||||
field: baseline["hashes"][field] for field in public_fields
|
||||
},
|
||||
"block": {
|
||||
field: block["hashes"][field] for field in public_fields
|
||||
},
|
||||
}
|
||||
input_gate_exact[key] = {
|
||||
"exact": gate_exact,
|
||||
"hashes": baseline["manifest"]["input_gate_tensor_hashes"],
|
||||
}
|
||||
if not exact or not gate_exact:
|
||||
raise ValueError(f"paired equality gate failed: {key}")
|
||||
|
||||
smoke_exact: dict[str, Any] = {}
|
||||
smoke_hashes: dict[str, str] = {}
|
||||
for depth in DEPTHS:
|
||||
for architecture in ARCHITECTURES:
|
||||
name = f"depth-{depth}-{architecture}.json"
|
||||
left_path = args.smoke_a_dir / name
|
||||
right_path = args.smoke_b_dir / name
|
||||
left = read_json(left_path)
|
||||
right = read_json(right_path)
|
||||
left_selected = selected(left, SMOKE_COMPARE_FIELDS)
|
||||
right_selected = selected(right, SMOKE_COMPARE_FIELDS)
|
||||
exact = left_selected == right_selected
|
||||
if (
|
||||
not exact
|
||||
or left["run_kind"] != "smoke"
|
||||
or left["steps"] != 20
|
||||
or not left["gradient_gate"]["passed"]
|
||||
):
|
||||
raise ValueError(f"smoke gate failed: {name}")
|
||||
key = f"depth-{depth}-{architecture}"
|
||||
smoke_exact[key] = {
|
||||
"exact": exact,
|
||||
"compare_sha256": canonical_sha256(left_selected),
|
||||
"gradient_gate": left["gradient_gate"],
|
||||
}
|
||||
for label, path in (("a", left_path), ("b", right_path)):
|
||||
public_name = f"smoke-{label}-{name}"
|
||||
source_paths[public_name] = path
|
||||
smoke_hashes[public_name] = file_sha256(path)
|
||||
|
||||
replay = read_json(args.replay)
|
||||
replay_formal = runs[(32, "block", 2026073001)]
|
||||
replay_left = selected(replay_formal, REPLAY_COMPARE_FIELDS)
|
||||
replay_right = selected(replay, REPLAY_COMPARE_FIELDS)
|
||||
replay_exact = replay_left == replay_right
|
||||
if (
|
||||
replay["run_kind"] != "replay"
|
||||
or replay["depth"] != 32
|
||||
or replay["architecture"] != "block"
|
||||
or replay["seed"] != 2026073001
|
||||
or not replay_exact
|
||||
):
|
||||
raise ValueError("formal replay gate failed")
|
||||
replay_public_name = "replay-depth-32-block-seed-2026073001.json"
|
||||
source_paths[replay_public_name] = args.replay
|
||||
|
||||
depth_summaries = {
|
||||
str(depth): summarize_depth(depth, runs) for depth in DEPTHS
|
||||
}
|
||||
depth_labels = [
|
||||
depth_summaries[str(depth)]["verdict"]["label"] for depth in DEPTHS
|
||||
]
|
||||
if all(
|
||||
label == "joint directional support at this depth"
|
||||
for label in depth_labels
|
||||
):
|
||||
overall_verdict = (
|
||||
"scale-consistent directional support in this operationalization"
|
||||
)
|
||||
elif all(
|
||||
label == "joint directional concern at this depth"
|
||||
for label in depth_labels
|
||||
):
|
||||
overall_verdict = (
|
||||
"scale-consistent directional concern in this operationalization"
|
||||
)
|
||||
else:
|
||||
overall_verdict = "depth-dependent or inconclusive"
|
||||
|
||||
full = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"manifest": manifest,
|
||||
"study": {
|
||||
"architectures": list(ARCHITECTURES),
|
||||
"depths": list(DEPTHS),
|
||||
"seeds": list(SEEDS),
|
||||
"diagnostic_steps": list(STEPS),
|
||||
"formal_runs": len(runs),
|
||||
"formal_target_bytes": EXPECTED_TOTAL_TARGET_BYTES,
|
||||
"replay_target_bytes": TARGET_BYTES_PER_RUN,
|
||||
"gradient_object": (
|
||||
"RMS of d(mean token CE)/d(post-MLP Transformer-block output) "
|
||||
"over batch×time×channel"
|
||||
),
|
||||
},
|
||||
"depth_summaries": depth_summaries,
|
||||
"overall_verdict": overall_verdict,
|
||||
"gates": {
|
||||
"common_initial_parameters": common_initial_exact,
|
||||
"paired_input_tensors": input_gate_exact,
|
||||
"smoke_exact": smoke_exact,
|
||||
"replay": {
|
||||
"exact": replay_exact,
|
||||
"compare_fields": list(REPLAY_COMPARE_FIELDS),
|
||||
"formal_compare_sha256": canonical_sha256(replay_left),
|
||||
"replay_compare_sha256": canonical_sha256(replay_right),
|
||||
"formal_final_model_state": replay_formal["hashes"][
|
||||
"final_model_state"
|
||||
],
|
||||
"replay_final_model_state": replay["hashes"][
|
||||
"final_model_state"
|
||||
],
|
||||
"formal_final_optimizer_state": replay_formal["hashes"][
|
||||
"final_optimizer_state"
|
||||
],
|
||||
"replay_final_optimizer_state": replay["hashes"][
|
||||
"final_optimizer_state"
|
||||
],
|
||||
},
|
||||
},
|
||||
"runs": {
|
||||
f"depth-{depth}-{architecture}-seed-{seed}": run
|
||||
for (depth, architecture, seed), run in sorted(runs.items())
|
||||
},
|
||||
}
|
||||
full["canonical_sha256_without_self"] = canonical_sha256(full)
|
||||
|
||||
compact = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"study": full["study"],
|
||||
"manifest_summary": {
|
||||
"file_sha256": manifest_hash,
|
||||
"dataset_revision": manifest["dataset"]["revision"],
|
||||
"train_bytes_sha256": manifest["dataset"]["splits"]["train"][
|
||||
"concatenated_sha256"
|
||||
],
|
||||
"formal_schedule_sha256": manifest["windows"][
|
||||
"formal_schedule_sha256"
|
||||
],
|
||||
"validation_tensor_sha256": manifest["windows"][
|
||||
"validation_tensor_sha256"
|
||||
],
|
||||
"diagnostic_tensor_sha256": manifest["windows"][
|
||||
"diagnostic_tensor_sha256"
|
||||
],
|
||||
},
|
||||
"depth_summaries": depth_summaries,
|
||||
"overall_verdict": overall_verdict,
|
||||
"replay_exact": replay_exact,
|
||||
"cells": [
|
||||
compact_cell(runs[(depth, architecture, seed)])
|
||||
for depth in DEPTHS
|
||||
for architecture in ARCHITECTURES
|
||||
for seed in SEEDS
|
||||
],
|
||||
}
|
||||
compact["canonical_sha256_without_self"] = canonical_sha256(compact)
|
||||
|
||||
args.raw_output_dir.mkdir(parents=True, exist_ok=True)
|
||||
for public_name, source_path in sorted(source_paths.items()):
|
||||
target = args.raw_output_dir / public_name
|
||||
temporary = target.with_suffix(target.suffix + ".tmp")
|
||||
shutil.copyfile(source_path, temporary)
|
||||
os.replace(temporary, target)
|
||||
|
||||
reproduction = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"manifest": {
|
||||
"path": str(args.manifest),
|
||||
"sha256": manifest_hash,
|
||||
},
|
||||
"protocol_sha256": file_sha256(
|
||||
Path("research/K3_ATTNRES_GRADIENT_SCALE_PROTOCOL.md")
|
||||
),
|
||||
"definition_audit_sha256": file_sha256(
|
||||
Path("research/K3_ATTNRES_GRADIENT_DEFINITION_AUDIT.md")
|
||||
),
|
||||
"runner_sha256": file_sha256(
|
||||
Path("experiments/k3/attnres_gradient/train.py")
|
||||
),
|
||||
"analyzer_sha256": file_sha256(Path(__file__)),
|
||||
"formal_raw_sha256": formal_hashes,
|
||||
"smoke_raw_sha256": smoke_hashes,
|
||||
"replay_raw_sha256": {
|
||||
replay_public_name: file_sha256(args.replay)
|
||||
},
|
||||
"formal_target_bytes": EXPECTED_TOTAL_TARGET_BYTES,
|
||||
"replay_target_bytes": TARGET_BYTES_PER_RUN,
|
||||
"smoke_exact": smoke_exact,
|
||||
"replay_exact": {
|
||||
"exact": replay_exact,
|
||||
"compare_sha256": canonical_sha256(replay_left),
|
||||
"final_model_state": replay["hashes"]["final_model_state"],
|
||||
"final_optimizer_state": replay["hashes"][
|
||||
"final_optimizer_state"
|
||||
],
|
||||
},
|
||||
"aggregate_sha256": full["canonical_sha256_without_self"],
|
||||
"compact_sha256": compact["canonical_sha256_without_self"],
|
||||
"overall_verdict": overall_verdict,
|
||||
}
|
||||
reproduction["canonical_sha256_without_self"] = canonical_sha256(
|
||||
reproduction
|
||||
)
|
||||
|
||||
atomic_json(args.output, full)
|
||||
atomic_json(args.compact_output, compact)
|
||||
atomic_json(args.reproduction_output, reproduction)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"formal_runs": len(runs),
|
||||
"formal_target_bytes": EXPECTED_TOTAL_TARGET_BYTES,
|
||||
"smoke_exact": all(
|
||||
row["exact"] for row in smoke_exact.values()
|
||||
),
|
||||
"replay_exact": replay_exact,
|
||||
"depth_verdicts": {
|
||||
depth: depth_summaries[str(depth)]["verdict"]["label"]
|
||||
for depth in DEPTHS
|
||||
},
|
||||
"overall_verdict": overall_verdict,
|
||||
"aggregate_sha256": full[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"compact_sha256": compact[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"reproduction_sha256": reproduction[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,206 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Freeze the byte-level corpus and window schedule for K3 AttnRes Round 05."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pyarrow.parquet as pq
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-gradient-scale-v1"
|
||||
DATASET_REPO = "Salesforce/wikitext"
|
||||
DATASET_REVISION = "b08601e04326c79dfdd32d625aee71d232d685c3"
|
||||
DATASET_VARIANT = "wikitext-2-raw-v1"
|
||||
SPLITS = ("train", "validation", "test")
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
CONTEXT = 256
|
||||
FORMAL_STEPS = 8000
|
||||
FORMAL_BATCH = 32
|
||||
VALIDATION_WINDOWS = 64
|
||||
DIAGNOSTIC_WINDOWS = 16
|
||||
GATE_STEPS = (0, 1, 7999)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--cache-dir", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def atomic_json(path: Path, value: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def download(url: str, path: Path) -> None:
|
||||
if path.exists():
|
||||
return
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".part")
|
||||
request = urllib.request.Request(
|
||||
url,
|
||||
headers={"User-Agent": "llm-atlas-k3-attnres-gradient-scale/1.0"},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=120) as response:
|
||||
with temporary.open("wb") as output:
|
||||
while block := response.read(1024 * 1024):
|
||||
output.write(block)
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def hashed_start(fields: list[str], corpus_length: int) -> int:
|
||||
value = int.from_bytes(
|
||||
hashlib.sha256("\0".join(fields).encode()).digest()[:8], "big"
|
||||
)
|
||||
return value % (corpus_length - (CONTEXT + 1))
|
||||
|
||||
|
||||
def fixed_window_start(label: str, index: int, corpus_length: int) -> int:
|
||||
return hashed_start([PROTOCOL_ID, label, str(index)], corpus_length)
|
||||
|
||||
|
||||
def train_window_start(seed: int, step: int, row: int, corpus_length: int) -> int:
|
||||
return hashed_start(
|
||||
[PROTOCOL_ID, "train-window", str(seed), str(step), str(row)],
|
||||
corpus_length,
|
||||
)
|
||||
|
||||
|
||||
def concatenate_split(parquet_path: Path) -> tuple[bytes, int]:
|
||||
table = pq.read_table(parquet_path, columns=["text"])
|
||||
rows = table.column("text").to_pylist()
|
||||
payload = b"".join(((row or "") + "\n").encode("utf-8") for row in rows)
|
||||
return payload, len(rows)
|
||||
|
||||
|
||||
def tensor_hash(payload: bytes, starts: list[int]) -> str:
|
||||
digest = hashlib.sha256()
|
||||
for start in starts:
|
||||
digest.update(payload[start : start + CONTEXT + 1])
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
args.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
split_manifest: dict[str, Any] = {}
|
||||
split_bytes: dict[str, bytes] = {}
|
||||
for split in SPLITS:
|
||||
relative = f"{DATASET_VARIANT}/{split}-00000-of-00001.parquet"
|
||||
url = (
|
||||
f"https://huggingface.co/datasets/{DATASET_REPO}/resolve/"
|
||||
f"{DATASET_REVISION}/{relative}"
|
||||
)
|
||||
parquet_path = args.cache_dir / f"{split}.parquet"
|
||||
download(url, parquet_path)
|
||||
payload, rows = concatenate_split(parquet_path)
|
||||
binary_path = args.cache_dir / f"{split}.bin"
|
||||
if not binary_path.exists() or file_sha256(binary_path) != hashlib.sha256(
|
||||
payload
|
||||
).hexdigest():
|
||||
temporary = binary_path.with_suffix(".bin.tmp")
|
||||
temporary.write_bytes(payload)
|
||||
os.replace(temporary, binary_path)
|
||||
split_bytes[split] = payload
|
||||
split_manifest[split] = {
|
||||
"source_path": relative,
|
||||
"source_url": url,
|
||||
"parquet_bytes": parquet_path.stat().st_size,
|
||||
"parquet_sha256": file_sha256(parquet_path),
|
||||
"rows": rows,
|
||||
"concatenated_bytes": len(payload),
|
||||
"concatenated_sha256": hashlib.sha256(payload).hexdigest(),
|
||||
"binary_path": str(binary_path),
|
||||
"binary_sha256": file_sha256(binary_path),
|
||||
}
|
||||
|
||||
train = split_bytes["train"]
|
||||
validation = split_bytes["validation"]
|
||||
schedule_digest = hashlib.sha256()
|
||||
schedule_cells = 0
|
||||
for seed in SEEDS:
|
||||
for step in range(1, FORMAL_STEPS + 1):
|
||||
for row in range(FORMAL_BATCH):
|
||||
start = train_window_start(seed, step, row, len(train))
|
||||
schedule_digest.update(start.to_bytes(8, "big"))
|
||||
schedule_cells += 1
|
||||
|
||||
validation_starts = [
|
||||
fixed_window_start("validation-window", index, len(validation))
|
||||
for index in range(VALIDATION_WINDOWS)
|
||||
]
|
||||
diagnostic_starts = [
|
||||
fixed_window_start("diagnostic-window", index, len(validation))
|
||||
for index in range(DIAGNOSTIC_WINDOWS)
|
||||
]
|
||||
gate_tensor_hashes: dict[str, dict[str, str]] = {}
|
||||
for seed in SEEDS:
|
||||
gate_tensor_hashes[str(seed)] = {}
|
||||
for step in GATE_STEPS:
|
||||
starts = [
|
||||
train_window_start(seed, step, row, len(train))
|
||||
for row in range(FORMAL_BATCH)
|
||||
]
|
||||
gate_tensor_hashes[str(seed)][str(step)] = tensor_hash(train, starts)
|
||||
|
||||
manifest = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"status": "frozen-before-model-output",
|
||||
"dataset": {
|
||||
"repository": DATASET_REPO,
|
||||
"revision": DATASET_REVISION,
|
||||
"variant": DATASET_VARIANT,
|
||||
"preprocessing": (
|
||||
"parquet row order; (text or empty string) + LF; UTF-8; "
|
||||
"no normalization; vocabulary is raw bytes 0..255"
|
||||
),
|
||||
"splits": split_manifest,
|
||||
},
|
||||
"windows": {
|
||||
"context": CONTEXT,
|
||||
"target_bytes_per_window": CONTEXT,
|
||||
"seeds": list(SEEDS),
|
||||
"formal_steps": FORMAL_STEPS,
|
||||
"formal_batch": FORMAL_BATCH,
|
||||
"formal_schedule_cells": schedule_cells,
|
||||
"formal_schedule_sha256": schedule_digest.hexdigest(),
|
||||
"validation_starts": validation_starts,
|
||||
"validation_tensor_sha256": tensor_hash(
|
||||
validation, validation_starts
|
||||
),
|
||||
"diagnostic_starts": diagnostic_starts,
|
||||
"diagnostic_tensor_sha256": tensor_hash(
|
||||
validation, diagnostic_starts
|
||||
),
|
||||
"gate_steps": list(GATE_STEPS),
|
||||
"gate_training_tensor_sha256": gate_tensor_hashes,
|
||||
},
|
||||
}
|
||||
atomic_json(args.manifest, manifest)
|
||||
print(json.dumps(manifest, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,167 @@
|
||||
{
|
||||
"dataset": {
|
||||
"preprocessing": "parquet row order; (text or empty string) + LF; UTF-8; no normalization; vocabulary is raw bytes 0..255",
|
||||
"repository": "Salesforce/wikitext",
|
||||
"revision": "b08601e04326c79dfdd32d625aee71d232d685c3",
|
||||
"splits": {
|
||||
"test": {
|
||||
"binary_path": "/home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1/test.bin",
|
||||
"binary_sha256": "bfe9eb16ab9987fb88bde4ea9a30a00f2a45db01dfc14bad78d05325789c4f12",
|
||||
"concatenated_bytes": 1292014,
|
||||
"concatenated_sha256": "bfe9eb16ab9987fb88bde4ea9a30a00f2a45db01dfc14bad78d05325789c4f12",
|
||||
"parquet_bytes": 732610,
|
||||
"parquet_sha256": "5f1bea067869d04849c0f975a2b29c4ff47d867f484f5010ea5e861eab246d91",
|
||||
"rows": 4358,
|
||||
"source_path": "wikitext-2-raw-v1/test-00000-of-00001.parquet",
|
||||
"source_url": "https://huggingface.co/datasets/Salesforce/wikitext/resolve/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-2-raw-v1/test-00000-of-00001.parquet"
|
||||
},
|
||||
"train": {
|
||||
"binary_path": "/home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1/train.bin",
|
||||
"binary_sha256": "0ca7d3e74dbe44564ea5942b85232f1bbcb525c9cd481cd5d28a87ee90e7e9b4",
|
||||
"concatenated_bytes": 10951563,
|
||||
"concatenated_sha256": "0ca7d3e74dbe44564ea5942b85232f1bbcb525c9cd481cd5d28a87ee90e7e9b4",
|
||||
"parquet_bytes": 6357543,
|
||||
"parquet_sha256": "e83889baabc497075506f91975be5fac0d45c5290b6b20582c8cd1e853d0c9f7",
|
||||
"rows": 36718,
|
||||
"source_path": "wikitext-2-raw-v1/train-00000-of-00001.parquet",
|
||||
"source_url": "https://huggingface.co/datasets/Salesforce/wikitext/resolve/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-2-raw-v1/train-00000-of-00001.parquet"
|
||||
},
|
||||
"validation": {
|
||||
"binary_path": "/home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1/validation.bin",
|
||||
"binary_sha256": "a42356f6a8ff1d25daf25ec9db49e10a537c265581b61c74604bb63231dee719",
|
||||
"concatenated_bytes": 1148008,
|
||||
"concatenated_sha256": "a42356f6a8ff1d25daf25ec9db49e10a537c265581b61c74604bb63231dee719",
|
||||
"parquet_bytes": 657209,
|
||||
"parquet_sha256": "204929b7ff9d6184953f867dedb860e40aa69c078fc1e54b3baaa8fb28511c4c",
|
||||
"rows": 3760,
|
||||
"source_path": "wikitext-2-raw-v1/validation-00000-of-00001.parquet",
|
||||
"source_url": "https://huggingface.co/datasets/Salesforce/wikitext/resolve/b08601e04326c79dfdd32d625aee71d232d685c3/wikitext-2-raw-v1/validation-00000-of-00001.parquet"
|
||||
}
|
||||
},
|
||||
"variant": "wikitext-2-raw-v1"
|
||||
},
|
||||
"protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1",
|
||||
"schema_version": 1,
|
||||
"status": "frozen-before-model-output",
|
||||
"windows": {
|
||||
"context": 256,
|
||||
"diagnostic_starts": [
|
||||
399861,
|
||||
210983,
|
||||
449025,
|
||||
1098024,
|
||||
323754,
|
||||
152932,
|
||||
1091551,
|
||||
1078021,
|
||||
415985,
|
||||
612288,
|
||||
910624,
|
||||
530272,
|
||||
827285,
|
||||
765798,
|
||||
1086876,
|
||||
1035447
|
||||
],
|
||||
"diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716",
|
||||
"formal_batch": 32,
|
||||
"formal_schedule_cells": 768000,
|
||||
"formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e",
|
||||
"formal_steps": 8000,
|
||||
"gate_steps": [
|
||||
0,
|
||||
1,
|
||||
7999
|
||||
],
|
||||
"gate_training_tensor_sha256": {
|
||||
"2026073001": {
|
||||
"0": "52fdd6885cc2bef8e29cedec8c293e1ea71f63fe3cd0b83f627fe640e19a95c5",
|
||||
"1": "9d0a960595a3f57cd18834d880bde56fa8dcbb0b1cbed26bcc9bbf773a67949c",
|
||||
"7999": "8f4f04a889d1f8c9eb75e917dc7cd6ddef1196bdea94e87466270db7be1289ee"
|
||||
},
|
||||
"2026073002": {
|
||||
"0": "156813ff7ab93736c8dba340711a9633b6c952d0f54df30f277e254705cc8b82",
|
||||
"1": "9ee5a434bdf417d658a1485135d48749e98c86f03239eef86dbd81a9d1309da0",
|
||||
"7999": "bca78ffefa000dc3693a790d65933251646c70facc64b42007d500c19bb90977"
|
||||
},
|
||||
"2026073003": {
|
||||
"0": "8ba13ad55eb54501ec443430446793ef41dea21fd067b2a5b0c84eda202ed0ba",
|
||||
"1": "326a12f07637fe70fd39fa2758b389e91dacc7f2f2ede708b80e09c9d4d1c39d",
|
||||
"7999": "7d6b10c3febfb697444909f695d3a45297789022714f8b71dadfec4d6236aea6"
|
||||
}
|
||||
},
|
||||
"seeds": [
|
||||
2026073001,
|
||||
2026073002,
|
||||
2026073003
|
||||
],
|
||||
"target_bytes_per_window": 256,
|
||||
"validation_starts": [
|
||||
19785,
|
||||
600807,
|
||||
1029500,
|
||||
319878,
|
||||
652204,
|
||||
1072662,
|
||||
679625,
|
||||
1027264,
|
||||
500250,
|
||||
944134,
|
||||
187313,
|
||||
834295,
|
||||
968556,
|
||||
550645,
|
||||
239009,
|
||||
452519,
|
||||
356354,
|
||||
134015,
|
||||
64555,
|
||||
397632,
|
||||
203140,
|
||||
346032,
|
||||
314812,
|
||||
10817,
|
||||
1141274,
|
||||
807645,
|
||||
417975,
|
||||
870687,
|
||||
377265,
|
||||
635426,
|
||||
597238,
|
||||
805324,
|
||||
12300,
|
||||
264343,
|
||||
84743,
|
||||
596894,
|
||||
188690,
|
||||
992517,
|
||||
854512,
|
||||
427504,
|
||||
94167,
|
||||
296670,
|
||||
760313,
|
||||
912279,
|
||||
1054297,
|
||||
81970,
|
||||
419690,
|
||||
971472,
|
||||
1041491,
|
||||
669963,
|
||||
735537,
|
||||
434513,
|
||||
169153,
|
||||
6229,
|
||||
136413,
|
||||
1098303,
|
||||
400950,
|
||||
457810,
|
||||
659776,
|
||||
911665,
|
||||
909832,
|
||||
532969,
|
||||
555820,
|
||||
1019138
|
||||
],
|
||||
"validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
{
|
||||
"aggregate_sha256": "69be133c8f5b11f8a58e831bc03f5432e737ed21efe89bf186784ead7c7f7b51",
|
||||
"analyzer_sha256": "017d38d92bfd5bc26e10028015f9bf85d28516fb5c5bae06d7bf315b5eca2fc8",
|
||||
"canonical_sha256_without_self": "addb2e5990af4bbec4b88b4654fc72464f4df5b57fd00e46cad1febbb042f68a",
|
||||
"compact_sha256": "8cdb71808a429e5f9513c1c6c9426bdc93e30f237001a1b47a2a2fba574a044f",
|
||||
"definition_audit_sha256": "79221c5648ba280d9177386d2fcfb2fa653b3dd14405185b5997754db2961cc6",
|
||||
"formal_raw_sha256": {
|
||||
"formal-depth-16-baseline-seed-2026073001.json": "c9144ddb458010868668ab0bb1272d4e5051300936d9b21dccee75c32a9e46de",
|
||||
"formal-depth-16-baseline-seed-2026073002.json": "939cad5054960ba8a72b50b4b250b94c3d22a6b46796f669c6e4f033d59fb9de",
|
||||
"formal-depth-16-baseline-seed-2026073003.json": "afd9a60ff323e97d21b24b413d9b9fa97b8d5f960bfc88d2040502aa5ad96a1e",
|
||||
"formal-depth-16-block-seed-2026073001.json": "23e3c68e54ec55d981066ad5730316db9084a6637a8064763952ba109ea1f715",
|
||||
"formal-depth-16-block-seed-2026073002.json": "027f8786f530d6ad5d910f0aad3d0f287887d6457c670bc8b50af7248be1487d",
|
||||
"formal-depth-16-block-seed-2026073003.json": "254b637f403a4c61e8e8bb2dd083349406b6baa7a3d8a616e37a9ab6f8ebaeb9",
|
||||
"formal-depth-32-baseline-seed-2026073001.json": "232ed3dfbbbc898c42622c4a9aee8400f76cb773c576bf1c522cdbb19ead998c",
|
||||
"formal-depth-32-baseline-seed-2026073002.json": "f4fb5fca608a6623ce4c9707714a8a9ebd2c10f6e63f54e243ec3c06d18b1fff",
|
||||
"formal-depth-32-baseline-seed-2026073003.json": "1b0bd279b165c3a2431685d8c1e6bc36ff72eb55333d5de689dca4f89506bb5e",
|
||||
"formal-depth-32-block-seed-2026073001.json": "29e1d638b481619c7b32de402122523db8b17cc1fc67a8881fa1ba132a1d5c38",
|
||||
"formal-depth-32-block-seed-2026073002.json": "21199deb2199395061e51220fd8c7afd04a1135a6381e406da9b5795e3ad5032",
|
||||
"formal-depth-32-block-seed-2026073003.json": "c0d7f1bcfa9fa7f8f3134ca4571bdf23a951182d03b7de9611a6b4b89667d4ac"
|
||||
},
|
||||
"formal_target_bytes": 786432000,
|
||||
"manifest": {
|
||||
"path": "experiments/k3/attnres_gradient/manifest.json",
|
||||
"sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371"
|
||||
},
|
||||
"overall_verdict": "depth-dependent or inconclusive",
|
||||
"protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1",
|
||||
"protocol_sha256": "f772629b3b82975b6756721c3a3bb57cc4171dfa26ba5b1e8043dc91c9dcce22",
|
||||
"replay_exact": {
|
||||
"compare_sha256": "46300a452840a9dc6a5180efe7949cf3d81cc4471ed4a942da407e9343064817",
|
||||
"exact": true,
|
||||
"final_model_state": "3f0b97ece3a15571ba3d656f589f512ca0bb9e20083c9f58a42ccaee14892f59",
|
||||
"final_optimizer_state": "ed03e6fbd4a12d8b063dcb22e0437754285f54d585374cd52fbd534f05d24637"
|
||||
},
|
||||
"replay_raw_sha256": {
|
||||
"replay-depth-32-block-seed-2026073001.json": "5cea76d68a642c2b4b4f189a813b2e4af9361a939ddd31a68dfdb0ab090b4aad"
|
||||
},
|
||||
"replay_target_bytes": 65536000,
|
||||
"runner_sha256": "04ae69e10c58972c9193c2d31c7e09d924a0d0e834107afa4ba128c64ac5800f",
|
||||
"schema_version": 1,
|
||||
"smoke_exact": {
|
||||
"depth-16-baseline": {
|
||||
"compare_sha256": "525cdcd9a79ec61096ecf64fedcbf93144d3168b434e2f8fd384c4374cbebd2a",
|
||||
"exact": true,
|
||||
"gradient_gate": {
|
||||
"first_to_last_ratio_abs_delta": 0.0,
|
||||
"max_abs_scale_ratio_error": 0.0,
|
||||
"normalized_spectrum_max_abs_delta": 0.0,
|
||||
"passed": true,
|
||||
"per_block_scale_ratios": [
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0
|
||||
],
|
||||
"population_cv_abs_delta": 0.0,
|
||||
"thresholds": {
|
||||
"scale_ratio_abs": 1e-05,
|
||||
"shape_abs": 1e-06
|
||||
}
|
||||
}
|
||||
},
|
||||
"depth-16-block": {
|
||||
"compare_sha256": "e4330a0d878d10b474b9aeb0be58131f801ebe2e5dd9553d4713c3d16b6590cc",
|
||||
"exact": true,
|
||||
"gradient_gate": {
|
||||
"first_to_last_ratio_abs_delta": 0.0,
|
||||
"max_abs_scale_ratio_error": 0.0,
|
||||
"normalized_spectrum_max_abs_delta": 0.0,
|
||||
"passed": true,
|
||||
"per_block_scale_ratios": [
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0
|
||||
],
|
||||
"population_cv_abs_delta": 0.0,
|
||||
"thresholds": {
|
||||
"scale_ratio_abs": 1e-05,
|
||||
"shape_abs": 1e-06
|
||||
}
|
||||
}
|
||||
},
|
||||
"depth-32-baseline": {
|
||||
"compare_sha256": "8ee0dc37e88742b6705968f5f2845a3a1df1250f5e0fd77be6b909236ea28e56",
|
||||
"exact": true,
|
||||
"gradient_gate": {
|
||||
"first_to_last_ratio_abs_delta": 0.0,
|
||||
"max_abs_scale_ratio_error": 0.0,
|
||||
"normalized_spectrum_max_abs_delta": 0.0,
|
||||
"passed": true,
|
||||
"per_block_scale_ratios": [
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0
|
||||
],
|
||||
"population_cv_abs_delta": 0.0,
|
||||
"thresholds": {
|
||||
"scale_ratio_abs": 1e-05,
|
||||
"shape_abs": 1e-06
|
||||
}
|
||||
}
|
||||
},
|
||||
"depth-32-block": {
|
||||
"compare_sha256": "289073b1940d4747b7e049f8d767f9c22a49867d37403ff6ef7746a55bd1734c",
|
||||
"exact": true,
|
||||
"gradient_gate": {
|
||||
"first_to_last_ratio_abs_delta": 0.0,
|
||||
"max_abs_scale_ratio_error": 0.0,
|
||||
"normalized_spectrum_max_abs_delta": 0.0,
|
||||
"passed": true,
|
||||
"per_block_scale_ratios": [
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0
|
||||
],
|
||||
"population_cv_abs_delta": 0.0,
|
||||
"thresholds": {
|
||||
"scale_ratio_abs": 1e-05,
|
||||
"shape_abs": 1e-06
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"smoke_raw_sha256": {
|
||||
"smoke-a-depth-16-baseline.json": "fe0ff3ed21bf9b86d238d7a22e99b6e07ba99135b58d29eb6b205f2602e64556",
|
||||
"smoke-a-depth-16-block.json": "faa06a2bd02135858d3a2503e1ecfb34187b4ee3b4ce3428ed473a55b15b6705",
|
||||
"smoke-a-depth-32-baseline.json": "e327a6f5db87515c1b29b2808205e23b81228b08ce356b98804717e2e55eef1c",
|
||||
"smoke-a-depth-32-block.json": "2d9a97c2d6272c928d5b5bc01febd142d4b2f2ad3080330e849f2731a8d080c2",
|
||||
"smoke-b-depth-16-baseline.json": "fe0ff3ed21bf9b86d238d7a22e99b6e07ba99135b58d29eb6b205f2602e64556",
|
||||
"smoke-b-depth-16-block.json": "faa06a2bd02135858d3a2503e1ecfb34187b4ee3b4ce3428ed473a55b15b6705",
|
||||
"smoke-b-depth-32-baseline.json": "e327a6f5db87515c1b29b2808205e23b81228b08ce356b98804717e2e55eef1c",
|
||||
"smoke-b-depth-32-block.json": "2d9a97c2d6272c928d5b5bc01febd142d4b2f2ad3080330e849f2731a8d080c2"
|
||||
}
|
||||
}
|
||||
+7366
File diff suppressed because it is too large
Load Diff
+7366
File diff suppressed because it is too large
Load Diff
+7366
File diff suppressed because it is too large
Load Diff
+9616
File diff suppressed because it is too large
Load Diff
+9616
File diff suppressed because it is too large
Load Diff
+9616
File diff suppressed because it is too large
Load Diff
+8614
File diff suppressed because it is too large
Load Diff
+8614
File diff suppressed because it is too large
Load Diff
+8614
File diff suppressed because it is too large
Load Diff
+13072
File diff suppressed because it is too large
Load Diff
+13072
File diff suppressed because it is too large
Load Diff
+13072
File diff suppressed because it is too large
Load Diff
+13072
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,700 @@
|
||||
{
|
||||
"architecture": "baseline",
|
||||
"batch_size": 32,
|
||||
"canonical_sha256_without_self": "0ae9ad1697ca15ec4c84270ad82b9fa98056f14f20fc7372c4b9886c29845415",
|
||||
"depth": 16,
|
||||
"diagnostics": [
|
||||
{
|
||||
"activation_grad_rms_by_block": [
|
||||
0.0002865509013645351,
|
||||
0.00024785567075014114,
|
||||
0.00021551651298068464,
|
||||
0.00019683866412378848,
|
||||
0.0001823508064262569,
|
||||
0.0001679604029050097,
|
||||
0.00015584587526973337,
|
||||
0.00014737028686795384,
|
||||
0.00014130656199995428,
|
||||
0.00013612695329356939,
|
||||
0.00013129493163432926,
|
||||
0.00012807638267986476,
|
||||
0.00012550120300147682,
|
||||
0.00012324050476308912,
|
||||
0.00012108208466088399,
|
||||
0.00011888353037647903
|
||||
],
|
||||
"activation_grad_statistics": {
|
||||
"first_quartile_mean": 0.00023669043730478734,
|
||||
"first_to_last_ratio": 1.9372775995887173,
|
||||
"imbalance_abs_log_ratio": 0.6612836883477485,
|
||||
"last_quartile_mean": 0.00012217683070048224,
|
||||
"mean": 0.00016411257956860936,
|
||||
"normalized": [
|
||||
1.7460629899168627,
|
||||
1.5102783187106137,
|
||||
1.313223602646409,
|
||||
1.1994124072706904,
|
||||
1.1111324123086055,
|
||||
1.0234462424910682,
|
||||
0.9496278449793059,
|
||||
0.8979828801383491,
|
||||
0.8610343117596252,
|
||||
0.8294729974472175,
|
||||
0.8000296624393728,
|
||||
0.7804178266926869,
|
||||
0.7647262832098098,
|
||||
0.7509509940495869,
|
||||
0.7377989242455608,
|
||||
0.7244023016942358
|
||||
],
|
||||
"population_cv": 0.29361872380036635
|
||||
},
|
||||
"activation_output_rms_by_block": [
|
||||
0.02866268903017044,
|
||||
0.029099803417921066,
|
||||
0.02953805774450302,
|
||||
0.030088091269135475,
|
||||
0.030644793063402176,
|
||||
0.03131929785013199,
|
||||
0.03213750571012497,
|
||||
0.03286394104361534,
|
||||
0.033869802951812744,
|
||||
0.03499744459986687,
|
||||
0.03588006645441055,
|
||||
0.037365153431892395,
|
||||
0.03789033368229866,
|
||||
0.039445169270038605,
|
||||
0.04050002992153168,
|
||||
0.04121527820825577
|
||||
],
|
||||
"activation_output_statistics": {
|
||||
"first_quartile_mean": 0.0293471603654325,
|
||||
"first_to_last_ratio": 0.7380574840395957,
|
||||
"imbalance_abs_log_ratio": 0.3037335657624931,
|
||||
"last_quartile_mean": 0.03976270277053118,
|
||||
"mean": 0.034094841103069484,
|
||||
"normalized": [
|
||||
0.8406752488894867,
|
||||
0.8534957922212247,
|
||||
0.8663497699023965,
|
||||
0.8824822259232266,
|
||||
0.8988102619619861,
|
||||
0.9185934539320196,
|
||||
0.942591449919727,
|
||||
0.9638977622528551,
|
||||
0.9933996421752943,
|
||||
1.0264733158329964,
|
||||
1.0523605710888722,
|
||||
1.0959180985456622,
|
||||
1.1113216092650298,
|
||||
1.1569248600043318,
|
||||
1.1878638706395246,
|
||||
1.2088420674453664
|
||||
],
|
||||
"population_cv": 0.11952987076282337
|
||||
},
|
||||
"bits_per_byte": 8.096463027059821,
|
||||
"branch_output_rms_by_sublayer": [
|
||||
0.0033055038657039404,
|
||||
0.0037518907338380814,
|
||||
0.0033751516602933407,
|
||||
0.003911525942385197,
|
||||
0.004229962360113859,
|
||||
0.0038461871445178986,
|
||||
0.004243654198944569,
|
||||
0.0038366704247891903,
|
||||
0.004372371360659599,
|
||||
0.003851204412057996,
|
||||
0.005078981164842844,
|
||||
0.003818925702944398,
|
||||
0.0054668826051056385,
|
||||
0.0038749484810978174,
|
||||
0.005823117680847645,
|
||||
0.0037566579412668943,
|
||||
0.0065947300754487514,
|
||||
0.003876061411574483,
|
||||
0.006763988174498081,
|
||||
0.003795720636844635,
|
||||
0.006863070651888847,
|
||||
0.0038706199266016483,
|
||||
0.007856340147554874,
|
||||
0.003839300014078617,
|
||||
0.00830968376249075,
|
||||
0.003927029203623533,
|
||||
0.009655521251261234,
|
||||
0.0038513424806296825,
|
||||
0.009004893712699413,
|
||||
0.0039484030567109585,
|
||||
0.008608299307525158,
|
||||
0.003791053779423237
|
||||
],
|
||||
"capture": {
|
||||
"all_gradients_finite": true,
|
||||
"all_gradients_present": true,
|
||||
"count": 16,
|
||||
"dtypes": [
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32"
|
||||
],
|
||||
"position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset",
|
||||
"shape": [
|
||||
16,
|
||||
256,
|
||||
192
|
||||
],
|
||||
"storage_unique": true
|
||||
},
|
||||
"core_parameter_grad_rms_by_block": [
|
||||
0.00809059897248305,
|
||||
0.007313812062277777,
|
||||
0.007090710224412361,
|
||||
0.0067033738367094624,
|
||||
0.006535407867454384,
|
||||
0.006785021795736036,
|
||||
0.006295993569919972,
|
||||
0.005509300326230647,
|
||||
0.006231736048910802,
|
||||
0.006047390980174007,
|
||||
0.005629405359532731,
|
||||
0.005784042737956458,
|
||||
0.0059591736113607996,
|
||||
0.0061983173180850636,
|
||||
0.006571690039252182,
|
||||
0.005060205437758869
|
||||
],
|
||||
"core_parameter_grad_statistics": {
|
||||
"first_quartile_mean": 0.007299623773970663,
|
||||
"first_to_last_ratio": 1.227374872012571,
|
||||
"imbalance_abs_log_ratio": 0.2048776382299483,
|
||||
"last_quartile_mean": 0.005947346601614228,
|
||||
"mean": 0.006362886261765913,
|
||||
"normalized": [
|
||||
1.271529717747562,
|
||||
1.1494488132258316,
|
||||
1.1143858200043408,
|
||||
1.05351149791715,
|
||||
1.0271137340180259,
|
||||
1.066343404015675,
|
||||
0.9894870520870547,
|
||||
0.8658492545019393,
|
||||
0.9793882512652816,
|
||||
0.9504163254515969,
|
||||
0.8847251275509024,
|
||||
0.909028151691524,
|
||||
0.9365519618304368,
|
||||
0.9741361173356609,
|
||||
1.0328158902888074,
|
||||
0.7952688810682109
|
||||
],
|
||||
"population_cv": 0.11384977604868386
|
||||
},
|
||||
"depth_weights": [],
|
||||
"layer_input_rms_by_sublayer": [
|
||||
0.02823694422841072,
|
||||
0.028404271230101585,
|
||||
0.02866268903017044,
|
||||
0.02886480651795864,
|
||||
0.029099803417921066,
|
||||
0.029365191236138344,
|
||||
0.02953805774450302,
|
||||
0.029881296679377556,
|
||||
0.030088091269135475,
|
||||
0.030441828072071075,
|
||||
0.030644793063402176,
|
||||
0.03106885403394699,
|
||||
0.03131929785013199,
|
||||
0.031892918050289154,
|
||||
0.03213750571012497,
|
||||
0.03266792371869087,
|
||||
0.03286394104361534,
|
||||
0.03364725783467293,
|
||||
0.033869802951812744,
|
||||
0.03485054895281792,
|
||||
0.03499744459986687,
|
||||
0.03562505170702934,
|
||||
0.03588006645441055,
|
||||
0.03719272464513779,
|
||||
0.037365153431892395,
|
||||
0.03768136352300644,
|
||||
0.03789033368229866,
|
||||
0.03927159309387207,
|
||||
0.039445169270038605,
|
||||
0.04041972756385803,
|
||||
0.04050002992153168,
|
||||
0.04110744968056679
|
||||
],
|
||||
"loss_nats": 5.6120405197143555,
|
||||
"loss_scale": 1.0,
|
||||
"output_weights": null,
|
||||
"step": 0,
|
||||
"stream_state_rms_by_sublayer": [
|
||||
0.028404271230101585,
|
||||
0.02866268903017044,
|
||||
0.02886480651795864,
|
||||
0.029099803417921066,
|
||||
0.029365191236138344,
|
||||
0.02953805774450302,
|
||||
0.029881296679377556,
|
||||
0.030088091269135475,
|
||||
0.030441828072071075,
|
||||
0.030644793063402176,
|
||||
0.03106885403394699,
|
||||
0.03131929785013199,
|
||||
0.031892918050289154,
|
||||
0.03213750571012497,
|
||||
0.03266792371869087,
|
||||
0.03286394104361534,
|
||||
0.03364725783467293,
|
||||
0.033869802951812744,
|
||||
0.03485054895281792,
|
||||
0.03499744459986687,
|
||||
0.03562505170702934,
|
||||
0.03588006645441055,
|
||||
0.03719272464513779,
|
||||
0.037365153431892395,
|
||||
0.03768136352300644,
|
||||
0.03789033368229866,
|
||||
0.03927159309387207,
|
||||
0.039445169270038605,
|
||||
0.04041972756385803,
|
||||
0.04050002992153168,
|
||||
0.04110744968056679,
|
||||
0.04121527820825577
|
||||
]
|
||||
},
|
||||
{
|
||||
"activation_grad_rms_by_block": [
|
||||
5.69471885683015e-05,
|
||||
5.527245593839325e-05,
|
||||
5.393734318204224e-05,
|
||||
5.292302375892177e-05,
|
||||
5.23102717124857e-05,
|
||||
5.178990977583453e-05,
|
||||
5.1564093155320734e-05,
|
||||
5.129844430484809e-05,
|
||||
5.121947833686136e-05,
|
||||
5.116340616950765e-05,
|
||||
5.109804988023825e-05,
|
||||
5.104695082991384e-05,
|
||||
5.09646451973822e-05,
|
||||
5.100828275317326e-05,
|
||||
5.106439857627265e-05,
|
||||
5.11144389747642e-05
|
||||
],
|
||||
"activation_grad_statistics": {
|
||||
"first_quartile_mean": 5.477000286191469e-05,
|
||||
"first_to_last_ratio": 1.0731232762518041,
|
||||
"imbalance_abs_log_ratio": 0.07057334637995329,
|
||||
"last_quartile_mean": 5.103794137539808e-05,
|
||||
"mean": 5.2170148819641327e-05,
|
||||
"normalized": [
|
||||
1.0915665348238701,
|
||||
1.0594651767139285,
|
||||
1.033873669184083,
|
||||
1.0144311441756324,
|
||||
1.002685882559561,
|
||||
0.9927115591500164,
|
||||
0.988383094968431,
|
||||
0.9832911246274795,
|
||||
0.9817775010367221,
|
||||
0.9807027069519371,
|
||||
0.9794499543578176,
|
||||
0.9784704852268963,
|
||||
0.9768928467805085,
|
||||
0.9777292936141551,
|
||||
0.9788049244944386,
|
||||
0.9797641013345227
|
||||
],
|
||||
"population_cv": 0.032815404485150704
|
||||
},
|
||||
"activation_output_rms_by_block": [
|
||||
0.028743742033839226,
|
||||
0.02942933700978756,
|
||||
0.030596865341067314,
|
||||
0.03249936178326607,
|
||||
0.03542664647102356,
|
||||
0.03894684836268425,
|
||||
0.04318666458129883,
|
||||
0.04936420917510986,
|
||||
0.05370106175541878,
|
||||
0.06048284471035004,
|
||||
0.0664696991443634,
|
||||
0.07231792062520981,
|
||||
0.07627613097429276,
|
||||
0.07997097074985504,
|
||||
0.08494995534420013,
|
||||
0.08972473442554474
|
||||
],
|
||||
"activation_output_statistics": {
|
||||
"first_quartile_mean": 0.03031732654199004,
|
||||
"first_to_last_ratio": 0.3664591129538783,
|
||||
"imbalance_abs_log_ratio": 1.0038683247140607,
|
||||
"last_quartile_mean": 0.08273044787347317,
|
||||
"mean": 0.05450543703045696,
|
||||
"normalized": [
|
||||
0.5273555006590916,
|
||||
0.5399339701348108,
|
||||
0.5613543713807885,
|
||||
0.5962590808162097,
|
||||
0.6499653686150171,
|
||||
0.7145497859400932,
|
||||
0.7923368187501488,
|
||||
0.9056749539963465,
|
||||
0.9852422929002714,
|
||||
1.1096662646068314,
|
||||
1.21950584686113,
|
||||
1.326801958945847,
|
||||
1.399422427007981,
|
||||
1.4672108895332452,
|
||||
1.5585592919240543,
|
||||
1.6461611779281335
|
||||
],
|
||||
"population_cv": 0.3808757530834043
|
||||
},
|
||||
"bits_per_byte": 6.7509657011324835,
|
||||
"branch_output_rms_by_sublayer": [
|
||||
0.003436450148001313,
|
||||
0.003791899885982275,
|
||||
0.003964710980653763,
|
||||
0.003938332665711641,
|
||||
0.0056790695525705814,
|
||||
0.003919641952961683,
|
||||
0.006960175931453705,
|
||||
0.0038887569680809975,
|
||||
0.008646421134471893,
|
||||
0.003928068559616804,
|
||||
0.009785857982933521,
|
||||
0.0038193853106349707,
|
||||
0.010625048540532589,
|
||||
0.003996006678789854,
|
||||
0.012892307713627815,
|
||||
0.0039014811627566814,
|
||||
0.012757784686982632,
|
||||
0.003861474571749568,
|
||||
0.014726920053362846,
|
||||
0.003962590359151363,
|
||||
0.012820222415030003,
|
||||
0.004551138263195753,
|
||||
0.01401793584227562,
|
||||
0.004161422606557608,
|
||||
0.01284075528383255,
|
||||
0.004107494372874498,
|
||||
0.012963366694748402,
|
||||
0.0034911984112113714,
|
||||
0.014392351731657982,
|
||||
0.004168955609202385,
|
||||
0.013408888131380081,
|
||||
0.003909120801836252
|
||||
],
|
||||
"capture": {
|
||||
"all_gradients_finite": true,
|
||||
"all_gradients_present": true,
|
||||
"count": 16,
|
||||
"dtypes": [
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32"
|
||||
],
|
||||
"position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset",
|
||||
"shape": [
|
||||
16,
|
||||
256,
|
||||
192
|
||||
],
|
||||
"storage_unique": true
|
||||
},
|
||||
"core_parameter_grad_rms_by_block": [
|
||||
0.0004507690637370199,
|
||||
0.0004530669153600251,
|
||||
0.0005174622333717066,
|
||||
0.0005579942111201043,
|
||||
0.0006494724560479704,
|
||||
0.0007477190266084469,
|
||||
0.0008433132451421535,
|
||||
0.0008983201732199723,
|
||||
0.001008715304343127,
|
||||
0.0011143118429534733,
|
||||
0.0010428854825857738,
|
||||
0.0010916116075341876,
|
||||
0.0011085001103672855,
|
||||
0.0012404769719650786,
|
||||
0.0013545740271648265,
|
||||
0.0014243271893498689
|
||||
],
|
||||
"core_parameter_grad_statistics": {
|
||||
"first_quartile_mean": 0.000494823105897214,
|
||||
"first_to_last_ratio": 0.385986622193018,
|
||||
"imbalance_abs_log_ratio": 0.9519525676489338,
|
||||
"last_quartile_mean": 0.001281969574711765,
|
||||
"mean": 0.0009064699913044387,
|
||||
"normalized": [
|
||||
0.49727963204644987,
|
||||
0.4998145771025995,
|
||||
0.5708542349284638,
|
||||
0.6155683215912455,
|
||||
0.7164853357289404,
|
||||
0.824869034585972,
|
||||
0.930326710461313,
|
||||
0.99100927977468,
|
||||
1.112795033503044,
|
||||
1.2292870736404011,
|
||||
1.1504909071341995,
|
||||
1.2042446170372658,
|
||||
1.2228756836970622,
|
||||
1.3684699812069823,
|
||||
1.494339625314625,
|
||||
1.571289952246756
|
||||
],
|
||||
"population_cv": 0.338812252534038
|
||||
},
|
||||
"depth_weights": [],
|
||||
"layer_input_rms_by_sublayer": [
|
||||
0.028241552412509918,
|
||||
0.028533460572361946,
|
||||
0.028743742033839226,
|
||||
0.029236938804388046,
|
||||
0.02942933700978756,
|
||||
0.03035305254161358,
|
||||
0.030596865341067314,
|
||||
0.032195623964071274,
|
||||
0.03249936178326607,
|
||||
0.03506496921181679,
|
||||
0.03542664647102356,
|
||||
0.03852483630180359,
|
||||
0.03894684836268425,
|
||||
0.04251723736524582,
|
||||
0.04318666458129883,
|
||||
0.04861506074666977,
|
||||
0.04936420917510986,
|
||||
0.052915990352630615,
|
||||
0.05370106175541878,
|
||||
0.0598941408097744,
|
||||
0.06048284471035004,
|
||||
0.06538087129592896,
|
||||
0.0664696991443634,
|
||||
0.07142146676778793,
|
||||
0.07231792062520981,
|
||||
0.07521561533212662,
|
||||
0.07627613097429276,
|
||||
0.07935076206922531,
|
||||
0.07997097074985504,
|
||||
0.08368266373872757,
|
||||
0.08494995534420013,
|
||||
0.08860929310321808
|
||||
],
|
||||
"loss_nats": 4.679412841796875,
|
||||
"loss_scale": 1.0,
|
||||
"output_weights": null,
|
||||
"step": 20,
|
||||
"stream_state_rms_by_sublayer": [
|
||||
0.028533460572361946,
|
||||
0.028743742033839226,
|
||||
0.029236938804388046,
|
||||
0.02942933700978756,
|
||||
0.03035305254161358,
|
||||
0.030596865341067314,
|
||||
0.032195623964071274,
|
||||
0.03249936178326607,
|
||||
0.03506496921181679,
|
||||
0.03542664647102356,
|
||||
0.03852483630180359,
|
||||
0.03894684836268425,
|
||||
0.04251723736524582,
|
||||
0.04318666458129883,
|
||||
0.04861506074666977,
|
||||
0.04936420917510986,
|
||||
0.052915990352630615,
|
||||
0.05370106175541878,
|
||||
0.0598941408097744,
|
||||
0.06048284471035004,
|
||||
0.06538087129592896,
|
||||
0.0664696991443634,
|
||||
0.07142146676778793,
|
||||
0.07231792062520981,
|
||||
0.07521561533212662,
|
||||
0.07627613097429276,
|
||||
0.07935076206922531,
|
||||
0.07997097074985504,
|
||||
0.08368266373872757,
|
||||
0.08494995534420013,
|
||||
0.08860929310321808,
|
||||
0.08972473442554474
|
||||
]
|
||||
}
|
||||
],
|
||||
"environment": {
|
||||
"autocast": "cuda-bfloat16-forward-fp32-cross-entropy",
|
||||
"compile": false,
|
||||
"compute_capability": [
|
||||
12,
|
||||
0
|
||||
],
|
||||
"cublas_workspace_config": ":4096:8",
|
||||
"cuda": "12.8",
|
||||
"deterministic_algorithms": true,
|
||||
"gpu": "NVIDIA GeForce RTX 5090",
|
||||
"python": "3.10.14",
|
||||
"torch": "2.11.0+cu128"
|
||||
},
|
||||
"evaluations": [
|
||||
{
|
||||
"bits_per_byte": 8.076786148009306,
|
||||
"cross_entropy_nats": 5.5984015464782715,
|
||||
"step": 0
|
||||
},
|
||||
{
|
||||
"bits_per_byte": 6.752401670275862,
|
||||
"cross_entropy_nats": 4.680408179759979,
|
||||
"step": 20
|
||||
}
|
||||
],
|
||||
"gradient_gate": {
|
||||
"first_to_last_ratio_abs_delta": 0.0,
|
||||
"max_abs_scale_ratio_error": 0.0,
|
||||
"normalized_spectrum_max_abs_delta": 0.0,
|
||||
"passed": true,
|
||||
"per_block_scale_ratios": [
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0
|
||||
],
|
||||
"population_cv_abs_delta": 0.0,
|
||||
"thresholds": {
|
||||
"scale_ratio_abs": 1e-05,
|
||||
"shape_abs": 1e-06
|
||||
}
|
||||
},
|
||||
"hashes": {
|
||||
"final_mixer_parameters": null,
|
||||
"final_model_state": "d53dab0fa7c76215ab91de676b2aef3f9ef14cb0cc1819b7c4a887915bed97c0",
|
||||
"final_optimizer_state": "a7bce44db1478ce53933758aa5033bbb1e0aa21296e9615c33146920f30a2057",
|
||||
"final_public_parameters": "d53dab0fa7c76215ab91de676b2aef3f9ef14cb0cc1819b7c4a887915bed97c0",
|
||||
"initial_mixer_parameters": null,
|
||||
"initial_public_parameter_elements": 9541824,
|
||||
"initial_public_parameter_structure": "e732db766f25e01f6ce1772cc182ced9de2c56c4a2384130117242b9444f0abe",
|
||||
"initial_public_parameter_tensors": 115,
|
||||
"initial_public_parameters": "af2724a1c34bcfd61d8a8bef402246430898c815e56b6e5c5257949a4eb0e7b1"
|
||||
},
|
||||
"manifest": {
|
||||
"diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716",
|
||||
"file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371",
|
||||
"formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e",
|
||||
"input_gate_tensor_hashes": {
|
||||
"0": "65136111a29a042e61a7909132560d95cd4bcf0f9b52f64d0fb2e57773856434",
|
||||
"1": "d995676b4e7dec8f661cd8c2345fe7fc7a513c17f528c02fc946a441a6995a94",
|
||||
"7999": "2345e7ac3decca2bdaebf13094fdc92fcabef42e3e461a2fefd6e8e7e76baccc"
|
||||
},
|
||||
"path": "experiments/k3/attnres_gradient/manifest.json",
|
||||
"validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338"
|
||||
},
|
||||
"model": {
|
||||
"attnres_aggregation_groups": 8,
|
||||
"context": 256,
|
||||
"d_ff": 768,
|
||||
"d_head": 32,
|
||||
"d_model": 192,
|
||||
"heads": 6,
|
||||
"layers": 16,
|
||||
"parameters": {
|
||||
"core": 9541824,
|
||||
"embedding": 98304,
|
||||
"mixer": 0,
|
||||
"total": 9541824
|
||||
},
|
||||
"sublayers": 32,
|
||||
"sublayers_per_attnres_group": 4,
|
||||
"transformer_blocks_per_attnres_group": 2,
|
||||
"vocabulary": 256
|
||||
},
|
||||
"optimizer": {
|
||||
"betas": [
|
||||
0.9,
|
||||
0.95
|
||||
],
|
||||
"epsilon": 1e-08,
|
||||
"grad_clip": 1.0,
|
||||
"min_lr": 3e-05,
|
||||
"name": "AdamW",
|
||||
"peak_lr": 0.0003,
|
||||
"warmup_steps": 400,
|
||||
"weight_decay_ndim_ge_2": 0.1
|
||||
},
|
||||
"protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1",
|
||||
"run_kind": "smoke",
|
||||
"schema_version": 1,
|
||||
"seed": 2026073001,
|
||||
"steps": 20,
|
||||
"target_bytes_seen": 163840,
|
||||
"timing": {
|
||||
"mean_ms": null,
|
||||
"measured_steps": 0,
|
||||
"median_ms": null,
|
||||
"p95_ms": null,
|
||||
"peak_allocated_bytes": 1648265728,
|
||||
"peak_reserved_bytes": 3282042880,
|
||||
"warmup_steps_excluded": 20
|
||||
},
|
||||
"training_history": [
|
||||
{
|
||||
"bits_per_byte": 8.088097790921855,
|
||||
"learning_rate": 7.499999999999999e-07,
|
||||
"loss_nats": 5.6062421798706055,
|
||||
"step": 1,
|
||||
"unclipped_grad_norm": 19.475919723510742
|
||||
},
|
||||
{
|
||||
"bits_per_byte": 7.474302716882146,
|
||||
"learning_rate": 7.499999999999999e-06,
|
||||
"loss_nats": 5.180791854858398,
|
||||
"step": 10,
|
||||
"unclipped_grad_norm": 12.938376426696777
|
||||
},
|
||||
{
|
||||
"bits_per_byte": 6.789615018295581,
|
||||
"learning_rate": 1.4999999999999999e-05,
|
||||
"loss_nats": 4.706202507019043,
|
||||
"step": 20,
|
||||
"unclipped_grad_norm": 4.482712745666504
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,700 @@
|
||||
{
|
||||
"architecture": "baseline",
|
||||
"batch_size": 32,
|
||||
"canonical_sha256_without_self": "0ae9ad1697ca15ec4c84270ad82b9fa98056f14f20fc7372c4b9886c29845415",
|
||||
"depth": 16,
|
||||
"diagnostics": [
|
||||
{
|
||||
"activation_grad_rms_by_block": [
|
||||
0.0002865509013645351,
|
||||
0.00024785567075014114,
|
||||
0.00021551651298068464,
|
||||
0.00019683866412378848,
|
||||
0.0001823508064262569,
|
||||
0.0001679604029050097,
|
||||
0.00015584587526973337,
|
||||
0.00014737028686795384,
|
||||
0.00014130656199995428,
|
||||
0.00013612695329356939,
|
||||
0.00013129493163432926,
|
||||
0.00012807638267986476,
|
||||
0.00012550120300147682,
|
||||
0.00012324050476308912,
|
||||
0.00012108208466088399,
|
||||
0.00011888353037647903
|
||||
],
|
||||
"activation_grad_statistics": {
|
||||
"first_quartile_mean": 0.00023669043730478734,
|
||||
"first_to_last_ratio": 1.9372775995887173,
|
||||
"imbalance_abs_log_ratio": 0.6612836883477485,
|
||||
"last_quartile_mean": 0.00012217683070048224,
|
||||
"mean": 0.00016411257956860936,
|
||||
"normalized": [
|
||||
1.7460629899168627,
|
||||
1.5102783187106137,
|
||||
1.313223602646409,
|
||||
1.1994124072706904,
|
||||
1.1111324123086055,
|
||||
1.0234462424910682,
|
||||
0.9496278449793059,
|
||||
0.8979828801383491,
|
||||
0.8610343117596252,
|
||||
0.8294729974472175,
|
||||
0.8000296624393728,
|
||||
0.7804178266926869,
|
||||
0.7647262832098098,
|
||||
0.7509509940495869,
|
||||
0.7377989242455608,
|
||||
0.7244023016942358
|
||||
],
|
||||
"population_cv": 0.29361872380036635
|
||||
},
|
||||
"activation_output_rms_by_block": [
|
||||
0.02866268903017044,
|
||||
0.029099803417921066,
|
||||
0.02953805774450302,
|
||||
0.030088091269135475,
|
||||
0.030644793063402176,
|
||||
0.03131929785013199,
|
||||
0.03213750571012497,
|
||||
0.03286394104361534,
|
||||
0.033869802951812744,
|
||||
0.03499744459986687,
|
||||
0.03588006645441055,
|
||||
0.037365153431892395,
|
||||
0.03789033368229866,
|
||||
0.039445169270038605,
|
||||
0.04050002992153168,
|
||||
0.04121527820825577
|
||||
],
|
||||
"activation_output_statistics": {
|
||||
"first_quartile_mean": 0.0293471603654325,
|
||||
"first_to_last_ratio": 0.7380574840395957,
|
||||
"imbalance_abs_log_ratio": 0.3037335657624931,
|
||||
"last_quartile_mean": 0.03976270277053118,
|
||||
"mean": 0.034094841103069484,
|
||||
"normalized": [
|
||||
0.8406752488894867,
|
||||
0.8534957922212247,
|
||||
0.8663497699023965,
|
||||
0.8824822259232266,
|
||||
0.8988102619619861,
|
||||
0.9185934539320196,
|
||||
0.942591449919727,
|
||||
0.9638977622528551,
|
||||
0.9933996421752943,
|
||||
1.0264733158329964,
|
||||
1.0523605710888722,
|
||||
1.0959180985456622,
|
||||
1.1113216092650298,
|
||||
1.1569248600043318,
|
||||
1.1878638706395246,
|
||||
1.2088420674453664
|
||||
],
|
||||
"population_cv": 0.11952987076282337
|
||||
},
|
||||
"bits_per_byte": 8.096463027059821,
|
||||
"branch_output_rms_by_sublayer": [
|
||||
0.0033055038657039404,
|
||||
0.0037518907338380814,
|
||||
0.0033751516602933407,
|
||||
0.003911525942385197,
|
||||
0.004229962360113859,
|
||||
0.0038461871445178986,
|
||||
0.004243654198944569,
|
||||
0.0038366704247891903,
|
||||
0.004372371360659599,
|
||||
0.003851204412057996,
|
||||
0.005078981164842844,
|
||||
0.003818925702944398,
|
||||
0.0054668826051056385,
|
||||
0.0038749484810978174,
|
||||
0.005823117680847645,
|
||||
0.0037566579412668943,
|
||||
0.0065947300754487514,
|
||||
0.003876061411574483,
|
||||
0.006763988174498081,
|
||||
0.003795720636844635,
|
||||
0.006863070651888847,
|
||||
0.0038706199266016483,
|
||||
0.007856340147554874,
|
||||
0.003839300014078617,
|
||||
0.00830968376249075,
|
||||
0.003927029203623533,
|
||||
0.009655521251261234,
|
||||
0.0038513424806296825,
|
||||
0.009004893712699413,
|
||||
0.0039484030567109585,
|
||||
0.008608299307525158,
|
||||
0.003791053779423237
|
||||
],
|
||||
"capture": {
|
||||
"all_gradients_finite": true,
|
||||
"all_gradients_present": true,
|
||||
"count": 16,
|
||||
"dtypes": [
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32"
|
||||
],
|
||||
"position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset",
|
||||
"shape": [
|
||||
16,
|
||||
256,
|
||||
192
|
||||
],
|
||||
"storage_unique": true
|
||||
},
|
||||
"core_parameter_grad_rms_by_block": [
|
||||
0.00809059897248305,
|
||||
0.007313812062277777,
|
||||
0.007090710224412361,
|
||||
0.0067033738367094624,
|
||||
0.006535407867454384,
|
||||
0.006785021795736036,
|
||||
0.006295993569919972,
|
||||
0.005509300326230647,
|
||||
0.006231736048910802,
|
||||
0.006047390980174007,
|
||||
0.005629405359532731,
|
||||
0.005784042737956458,
|
||||
0.0059591736113607996,
|
||||
0.0061983173180850636,
|
||||
0.006571690039252182,
|
||||
0.005060205437758869
|
||||
],
|
||||
"core_parameter_grad_statistics": {
|
||||
"first_quartile_mean": 0.007299623773970663,
|
||||
"first_to_last_ratio": 1.227374872012571,
|
||||
"imbalance_abs_log_ratio": 0.2048776382299483,
|
||||
"last_quartile_mean": 0.005947346601614228,
|
||||
"mean": 0.006362886261765913,
|
||||
"normalized": [
|
||||
1.271529717747562,
|
||||
1.1494488132258316,
|
||||
1.1143858200043408,
|
||||
1.05351149791715,
|
||||
1.0271137340180259,
|
||||
1.066343404015675,
|
||||
0.9894870520870547,
|
||||
0.8658492545019393,
|
||||
0.9793882512652816,
|
||||
0.9504163254515969,
|
||||
0.8847251275509024,
|
||||
0.909028151691524,
|
||||
0.9365519618304368,
|
||||
0.9741361173356609,
|
||||
1.0328158902888074,
|
||||
0.7952688810682109
|
||||
],
|
||||
"population_cv": 0.11384977604868386
|
||||
},
|
||||
"depth_weights": [],
|
||||
"layer_input_rms_by_sublayer": [
|
||||
0.02823694422841072,
|
||||
0.028404271230101585,
|
||||
0.02866268903017044,
|
||||
0.02886480651795864,
|
||||
0.029099803417921066,
|
||||
0.029365191236138344,
|
||||
0.02953805774450302,
|
||||
0.029881296679377556,
|
||||
0.030088091269135475,
|
||||
0.030441828072071075,
|
||||
0.030644793063402176,
|
||||
0.03106885403394699,
|
||||
0.03131929785013199,
|
||||
0.031892918050289154,
|
||||
0.03213750571012497,
|
||||
0.03266792371869087,
|
||||
0.03286394104361534,
|
||||
0.03364725783467293,
|
||||
0.033869802951812744,
|
||||
0.03485054895281792,
|
||||
0.03499744459986687,
|
||||
0.03562505170702934,
|
||||
0.03588006645441055,
|
||||
0.03719272464513779,
|
||||
0.037365153431892395,
|
||||
0.03768136352300644,
|
||||
0.03789033368229866,
|
||||
0.03927159309387207,
|
||||
0.039445169270038605,
|
||||
0.04041972756385803,
|
||||
0.04050002992153168,
|
||||
0.04110744968056679
|
||||
],
|
||||
"loss_nats": 5.6120405197143555,
|
||||
"loss_scale": 1.0,
|
||||
"output_weights": null,
|
||||
"step": 0,
|
||||
"stream_state_rms_by_sublayer": [
|
||||
0.028404271230101585,
|
||||
0.02866268903017044,
|
||||
0.02886480651795864,
|
||||
0.029099803417921066,
|
||||
0.029365191236138344,
|
||||
0.02953805774450302,
|
||||
0.029881296679377556,
|
||||
0.030088091269135475,
|
||||
0.030441828072071075,
|
||||
0.030644793063402176,
|
||||
0.03106885403394699,
|
||||
0.03131929785013199,
|
||||
0.031892918050289154,
|
||||
0.03213750571012497,
|
||||
0.03266792371869087,
|
||||
0.03286394104361534,
|
||||
0.03364725783467293,
|
||||
0.033869802951812744,
|
||||
0.03485054895281792,
|
||||
0.03499744459986687,
|
||||
0.03562505170702934,
|
||||
0.03588006645441055,
|
||||
0.03719272464513779,
|
||||
0.037365153431892395,
|
||||
0.03768136352300644,
|
||||
0.03789033368229866,
|
||||
0.03927159309387207,
|
||||
0.039445169270038605,
|
||||
0.04041972756385803,
|
||||
0.04050002992153168,
|
||||
0.04110744968056679,
|
||||
0.04121527820825577
|
||||
]
|
||||
},
|
||||
{
|
||||
"activation_grad_rms_by_block": [
|
||||
5.69471885683015e-05,
|
||||
5.527245593839325e-05,
|
||||
5.393734318204224e-05,
|
||||
5.292302375892177e-05,
|
||||
5.23102717124857e-05,
|
||||
5.178990977583453e-05,
|
||||
5.1564093155320734e-05,
|
||||
5.129844430484809e-05,
|
||||
5.121947833686136e-05,
|
||||
5.116340616950765e-05,
|
||||
5.109804988023825e-05,
|
||||
5.104695082991384e-05,
|
||||
5.09646451973822e-05,
|
||||
5.100828275317326e-05,
|
||||
5.106439857627265e-05,
|
||||
5.11144389747642e-05
|
||||
],
|
||||
"activation_grad_statistics": {
|
||||
"first_quartile_mean": 5.477000286191469e-05,
|
||||
"first_to_last_ratio": 1.0731232762518041,
|
||||
"imbalance_abs_log_ratio": 0.07057334637995329,
|
||||
"last_quartile_mean": 5.103794137539808e-05,
|
||||
"mean": 5.2170148819641327e-05,
|
||||
"normalized": [
|
||||
1.0915665348238701,
|
||||
1.0594651767139285,
|
||||
1.033873669184083,
|
||||
1.0144311441756324,
|
||||
1.002685882559561,
|
||||
0.9927115591500164,
|
||||
0.988383094968431,
|
||||
0.9832911246274795,
|
||||
0.9817775010367221,
|
||||
0.9807027069519371,
|
||||
0.9794499543578176,
|
||||
0.9784704852268963,
|
||||
0.9768928467805085,
|
||||
0.9777292936141551,
|
||||
0.9788049244944386,
|
||||
0.9797641013345227
|
||||
],
|
||||
"population_cv": 0.032815404485150704
|
||||
},
|
||||
"activation_output_rms_by_block": [
|
||||
0.028743742033839226,
|
||||
0.02942933700978756,
|
||||
0.030596865341067314,
|
||||
0.03249936178326607,
|
||||
0.03542664647102356,
|
||||
0.03894684836268425,
|
||||
0.04318666458129883,
|
||||
0.04936420917510986,
|
||||
0.05370106175541878,
|
||||
0.06048284471035004,
|
||||
0.0664696991443634,
|
||||
0.07231792062520981,
|
||||
0.07627613097429276,
|
||||
0.07997097074985504,
|
||||
0.08494995534420013,
|
||||
0.08972473442554474
|
||||
],
|
||||
"activation_output_statistics": {
|
||||
"first_quartile_mean": 0.03031732654199004,
|
||||
"first_to_last_ratio": 0.3664591129538783,
|
||||
"imbalance_abs_log_ratio": 1.0038683247140607,
|
||||
"last_quartile_mean": 0.08273044787347317,
|
||||
"mean": 0.05450543703045696,
|
||||
"normalized": [
|
||||
0.5273555006590916,
|
||||
0.5399339701348108,
|
||||
0.5613543713807885,
|
||||
0.5962590808162097,
|
||||
0.6499653686150171,
|
||||
0.7145497859400932,
|
||||
0.7923368187501488,
|
||||
0.9056749539963465,
|
||||
0.9852422929002714,
|
||||
1.1096662646068314,
|
||||
1.21950584686113,
|
||||
1.326801958945847,
|
||||
1.399422427007981,
|
||||
1.4672108895332452,
|
||||
1.5585592919240543,
|
||||
1.6461611779281335
|
||||
],
|
||||
"population_cv": 0.3808757530834043
|
||||
},
|
||||
"bits_per_byte": 6.7509657011324835,
|
||||
"branch_output_rms_by_sublayer": [
|
||||
0.003436450148001313,
|
||||
0.003791899885982275,
|
||||
0.003964710980653763,
|
||||
0.003938332665711641,
|
||||
0.0056790695525705814,
|
||||
0.003919641952961683,
|
||||
0.006960175931453705,
|
||||
0.0038887569680809975,
|
||||
0.008646421134471893,
|
||||
0.003928068559616804,
|
||||
0.009785857982933521,
|
||||
0.0038193853106349707,
|
||||
0.010625048540532589,
|
||||
0.003996006678789854,
|
||||
0.012892307713627815,
|
||||
0.0039014811627566814,
|
||||
0.012757784686982632,
|
||||
0.003861474571749568,
|
||||
0.014726920053362846,
|
||||
0.003962590359151363,
|
||||
0.012820222415030003,
|
||||
0.004551138263195753,
|
||||
0.01401793584227562,
|
||||
0.004161422606557608,
|
||||
0.01284075528383255,
|
||||
0.004107494372874498,
|
||||
0.012963366694748402,
|
||||
0.0034911984112113714,
|
||||
0.014392351731657982,
|
||||
0.004168955609202385,
|
||||
0.013408888131380081,
|
||||
0.003909120801836252
|
||||
],
|
||||
"capture": {
|
||||
"all_gradients_finite": true,
|
||||
"all_gradients_present": true,
|
||||
"count": 16,
|
||||
"dtypes": [
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32",
|
||||
"torch.float32"
|
||||
],
|
||||
"position": "post-MLP Transformer-block output; Block AttnRes is captured before aggregation-partial reset",
|
||||
"shape": [
|
||||
16,
|
||||
256,
|
||||
192
|
||||
],
|
||||
"storage_unique": true
|
||||
},
|
||||
"core_parameter_grad_rms_by_block": [
|
||||
0.0004507690637370199,
|
||||
0.0004530669153600251,
|
||||
0.0005174622333717066,
|
||||
0.0005579942111201043,
|
||||
0.0006494724560479704,
|
||||
0.0007477190266084469,
|
||||
0.0008433132451421535,
|
||||
0.0008983201732199723,
|
||||
0.001008715304343127,
|
||||
0.0011143118429534733,
|
||||
0.0010428854825857738,
|
||||
0.0010916116075341876,
|
||||
0.0011085001103672855,
|
||||
0.0012404769719650786,
|
||||
0.0013545740271648265,
|
||||
0.0014243271893498689
|
||||
],
|
||||
"core_parameter_grad_statistics": {
|
||||
"first_quartile_mean": 0.000494823105897214,
|
||||
"first_to_last_ratio": 0.385986622193018,
|
||||
"imbalance_abs_log_ratio": 0.9519525676489338,
|
||||
"last_quartile_mean": 0.001281969574711765,
|
||||
"mean": 0.0009064699913044387,
|
||||
"normalized": [
|
||||
0.49727963204644987,
|
||||
0.4998145771025995,
|
||||
0.5708542349284638,
|
||||
0.6155683215912455,
|
||||
0.7164853357289404,
|
||||
0.824869034585972,
|
||||
0.930326710461313,
|
||||
0.99100927977468,
|
||||
1.112795033503044,
|
||||
1.2292870736404011,
|
||||
1.1504909071341995,
|
||||
1.2042446170372658,
|
||||
1.2228756836970622,
|
||||
1.3684699812069823,
|
||||
1.494339625314625,
|
||||
1.571289952246756
|
||||
],
|
||||
"population_cv": 0.338812252534038
|
||||
},
|
||||
"depth_weights": [],
|
||||
"layer_input_rms_by_sublayer": [
|
||||
0.028241552412509918,
|
||||
0.028533460572361946,
|
||||
0.028743742033839226,
|
||||
0.029236938804388046,
|
||||
0.02942933700978756,
|
||||
0.03035305254161358,
|
||||
0.030596865341067314,
|
||||
0.032195623964071274,
|
||||
0.03249936178326607,
|
||||
0.03506496921181679,
|
||||
0.03542664647102356,
|
||||
0.03852483630180359,
|
||||
0.03894684836268425,
|
||||
0.04251723736524582,
|
||||
0.04318666458129883,
|
||||
0.04861506074666977,
|
||||
0.04936420917510986,
|
||||
0.052915990352630615,
|
||||
0.05370106175541878,
|
||||
0.0598941408097744,
|
||||
0.06048284471035004,
|
||||
0.06538087129592896,
|
||||
0.0664696991443634,
|
||||
0.07142146676778793,
|
||||
0.07231792062520981,
|
||||
0.07521561533212662,
|
||||
0.07627613097429276,
|
||||
0.07935076206922531,
|
||||
0.07997097074985504,
|
||||
0.08368266373872757,
|
||||
0.08494995534420013,
|
||||
0.08860929310321808
|
||||
],
|
||||
"loss_nats": 4.679412841796875,
|
||||
"loss_scale": 1.0,
|
||||
"output_weights": null,
|
||||
"step": 20,
|
||||
"stream_state_rms_by_sublayer": [
|
||||
0.028533460572361946,
|
||||
0.028743742033839226,
|
||||
0.029236938804388046,
|
||||
0.02942933700978756,
|
||||
0.03035305254161358,
|
||||
0.030596865341067314,
|
||||
0.032195623964071274,
|
||||
0.03249936178326607,
|
||||
0.03506496921181679,
|
||||
0.03542664647102356,
|
||||
0.03852483630180359,
|
||||
0.03894684836268425,
|
||||
0.04251723736524582,
|
||||
0.04318666458129883,
|
||||
0.04861506074666977,
|
||||
0.04936420917510986,
|
||||
0.052915990352630615,
|
||||
0.05370106175541878,
|
||||
0.0598941408097744,
|
||||
0.06048284471035004,
|
||||
0.06538087129592896,
|
||||
0.0664696991443634,
|
||||
0.07142146676778793,
|
||||
0.07231792062520981,
|
||||
0.07521561533212662,
|
||||
0.07627613097429276,
|
||||
0.07935076206922531,
|
||||
0.07997097074985504,
|
||||
0.08368266373872757,
|
||||
0.08494995534420013,
|
||||
0.08860929310321808,
|
||||
0.08972473442554474
|
||||
]
|
||||
}
|
||||
],
|
||||
"environment": {
|
||||
"autocast": "cuda-bfloat16-forward-fp32-cross-entropy",
|
||||
"compile": false,
|
||||
"compute_capability": [
|
||||
12,
|
||||
0
|
||||
],
|
||||
"cublas_workspace_config": ":4096:8",
|
||||
"cuda": "12.8",
|
||||
"deterministic_algorithms": true,
|
||||
"gpu": "NVIDIA GeForce RTX 5090",
|
||||
"python": "3.10.14",
|
||||
"torch": "2.11.0+cu128"
|
||||
},
|
||||
"evaluations": [
|
||||
{
|
||||
"bits_per_byte": 8.076786148009306,
|
||||
"cross_entropy_nats": 5.5984015464782715,
|
||||
"step": 0
|
||||
},
|
||||
{
|
||||
"bits_per_byte": 6.752401670275862,
|
||||
"cross_entropy_nats": 4.680408179759979,
|
||||
"step": 20
|
||||
}
|
||||
],
|
||||
"gradient_gate": {
|
||||
"first_to_last_ratio_abs_delta": 0.0,
|
||||
"max_abs_scale_ratio_error": 0.0,
|
||||
"normalized_spectrum_max_abs_delta": 0.0,
|
||||
"passed": true,
|
||||
"per_block_scale_ratios": [
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0,
|
||||
2.0
|
||||
],
|
||||
"population_cv_abs_delta": 0.0,
|
||||
"thresholds": {
|
||||
"scale_ratio_abs": 1e-05,
|
||||
"shape_abs": 1e-06
|
||||
}
|
||||
},
|
||||
"hashes": {
|
||||
"final_mixer_parameters": null,
|
||||
"final_model_state": "d53dab0fa7c76215ab91de676b2aef3f9ef14cb0cc1819b7c4a887915bed97c0",
|
||||
"final_optimizer_state": "a7bce44db1478ce53933758aa5033bbb1e0aa21296e9615c33146920f30a2057",
|
||||
"final_public_parameters": "d53dab0fa7c76215ab91de676b2aef3f9ef14cb0cc1819b7c4a887915bed97c0",
|
||||
"initial_mixer_parameters": null,
|
||||
"initial_public_parameter_elements": 9541824,
|
||||
"initial_public_parameter_structure": "e732db766f25e01f6ce1772cc182ced9de2c56c4a2384130117242b9444f0abe",
|
||||
"initial_public_parameter_tensors": 115,
|
||||
"initial_public_parameters": "af2724a1c34bcfd61d8a8bef402246430898c815e56b6e5c5257949a4eb0e7b1"
|
||||
},
|
||||
"manifest": {
|
||||
"diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716",
|
||||
"file_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371",
|
||||
"formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e",
|
||||
"input_gate_tensor_hashes": {
|
||||
"0": "65136111a29a042e61a7909132560d95cd4bcf0f9b52f64d0fb2e57773856434",
|
||||
"1": "d995676b4e7dec8f661cd8c2345fe7fc7a513c17f528c02fc946a441a6995a94",
|
||||
"7999": "2345e7ac3decca2bdaebf13094fdc92fcabef42e3e461a2fefd6e8e7e76baccc"
|
||||
},
|
||||
"path": "experiments/k3/attnres_gradient/manifest.json",
|
||||
"validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338"
|
||||
},
|
||||
"model": {
|
||||
"attnres_aggregation_groups": 8,
|
||||
"context": 256,
|
||||
"d_ff": 768,
|
||||
"d_head": 32,
|
||||
"d_model": 192,
|
||||
"heads": 6,
|
||||
"layers": 16,
|
||||
"parameters": {
|
||||
"core": 9541824,
|
||||
"embedding": 98304,
|
||||
"mixer": 0,
|
||||
"total": 9541824
|
||||
},
|
||||
"sublayers": 32,
|
||||
"sublayers_per_attnres_group": 4,
|
||||
"transformer_blocks_per_attnres_group": 2,
|
||||
"vocabulary": 256
|
||||
},
|
||||
"optimizer": {
|
||||
"betas": [
|
||||
0.9,
|
||||
0.95
|
||||
],
|
||||
"epsilon": 1e-08,
|
||||
"grad_clip": 1.0,
|
||||
"min_lr": 3e-05,
|
||||
"name": "AdamW",
|
||||
"peak_lr": 0.0003,
|
||||
"warmup_steps": 400,
|
||||
"weight_decay_ndim_ge_2": 0.1
|
||||
},
|
||||
"protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1",
|
||||
"run_kind": "smoke",
|
||||
"schema_version": 1,
|
||||
"seed": 2026073001,
|
||||
"steps": 20,
|
||||
"target_bytes_seen": 163840,
|
||||
"timing": {
|
||||
"mean_ms": null,
|
||||
"measured_steps": 0,
|
||||
"median_ms": null,
|
||||
"p95_ms": null,
|
||||
"peak_allocated_bytes": 1648265728,
|
||||
"peak_reserved_bytes": 3282042880,
|
||||
"warmup_steps_excluded": 20
|
||||
},
|
||||
"training_history": [
|
||||
{
|
||||
"bits_per_byte": 8.088097790921855,
|
||||
"learning_rate": 7.499999999999999e-07,
|
||||
"loss_nats": 5.6062421798706055,
|
||||
"step": 1,
|
||||
"unclipped_grad_norm": 19.475919723510742
|
||||
},
|
||||
{
|
||||
"bits_per_byte": 7.474302716882146,
|
||||
"learning_rate": 7.499999999999999e-06,
|
||||
"loss_nats": 5.180791854858398,
|
||||
"step": 10,
|
||||
"unclipped_grad_norm": 12.938376426696777
|
||||
},
|
||||
{
|
||||
"bits_per_byte": 6.789615018295581,
|
||||
"learning_rate": 1.4999999999999999e-05,
|
||||
"loss_nats": 4.706202507019043,
|
||||
"step": 20,
|
||||
"unclipped_grad_norm": 4.482712745666504
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,827 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run one preregistered AttnRes activation-gradient/depth experiment cell."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-gradient-scale-v1"
|
||||
ARCHITECTURES = ("baseline", "block")
|
||||
DEPTHS = (16, 32)
|
||||
EXPECTED_SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
DIAGNOSTIC_STEPS = (0, 100, 500, 2000, 4000, 8000)
|
||||
FORMAL_STEPS = 8000
|
||||
SMOKE_STEPS = 20
|
||||
CONTEXT = 256
|
||||
VOCABULARY = 256
|
||||
BLOCK_GROUPS = 8
|
||||
PEAK_LR = 3e-4
|
||||
MIN_LR = 3e-5
|
||||
WARMUP_STEPS = 400
|
||||
WEIGHT_DECAY = 0.1
|
||||
BETAS = (0.9, 0.95)
|
||||
ADAM_EPS = 1e-8
|
||||
GRAD_CLIP = 1.0
|
||||
|
||||
|
||||
def load_round04_module() -> Any:
|
||||
path = Path(__file__).resolve().parents[1] / "attnres" / "train.py"
|
||||
spec = importlib.util.spec_from_file_location("k3_attnres_round04_train", path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"cannot import Round 04 runner from {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
round04 = load_round04_module()
|
||||
|
||||
|
||||
def configure_round04_globals(depth: int) -> None:
|
||||
round04.PROTOCOL_ID = PROTOCOL_ID
|
||||
round04.LAYERS = depth
|
||||
round04.SUBLAYERS = depth * 2
|
||||
round04.BLOCKS = BLOCK_GROUPS
|
||||
round04.SUBLAYERS_PER_BLOCK = (depth * 2) // BLOCK_GROUPS
|
||||
round04.WARMUP_STEPS = WARMUP_STEPS
|
||||
round04.EVAL_STEPS = DIAGNOSTIC_STEPS
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--architecture", choices=ARCHITECTURES, required=True)
|
||||
parser.add_argument("--depth", type=int, choices=DEPTHS, required=True)
|
||||
parser.add_argument("--seed", type=int, required=True)
|
||||
parser.add_argument("--steps", type=int)
|
||||
parser.add_argument("--batch-size", type=int, default=32)
|
||||
parser.add_argument("--cache-dir", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--validation-windows", type=int, default=64)
|
||||
parser.add_argument("--diagnostic-windows", type=int, default=16)
|
||||
parser.add_argument("--eval-batch-size", type=int, default=8)
|
||||
parser.add_argument("--timing-warmup", type=int, default=20)
|
||||
parser.add_argument(
|
||||
"--run-kind", choices=("smoke", "formal", "replay"), default="formal"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
expected_steps = SMOKE_STEPS if args.run_kind == "smoke" else FORMAL_STEPS
|
||||
if args.steps is None:
|
||||
args.steps = expected_steps
|
||||
if args.steps != expected_steps:
|
||||
raise ValueError(
|
||||
f"{args.run_kind} must run exactly {expected_steps} steps, got {args.steps}"
|
||||
)
|
||||
if args.batch_size != 32:
|
||||
raise ValueError("the frozen protocol requires batch size 32")
|
||||
if args.validation_windows != 64 or args.diagnostic_windows != 16:
|
||||
raise ValueError("the frozen protocol requires 64 validation / 16 diagnostic windows")
|
||||
return args
|
||||
|
||||
|
||||
def configure_determinism(seed: int) -> None:
|
||||
if os.environ.get("CUBLAS_WORKSPACE_CONFIG") != ":4096:8":
|
||||
raise RuntimeError("CUBLAS_WORKSPACE_CONFIG must be :4096:8 before Python starts")
|
||||
torch.manual_seed(seed)
|
||||
torch.cuda.manual_seed_all(seed)
|
||||
torch.use_deterministic_algorithms(True)
|
||||
torch.backends.cudnn.benchmark = False
|
||||
torch.backends.cudnn.deterministic = True
|
||||
torch.backends.cuda.matmul.allow_tf32 = False
|
||||
torch.backends.cudnn.allow_tf32 = False
|
||||
torch.set_float32_matmul_precision("highest")
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def tensor_bytes(tensor: torch.Tensor) -> bytes:
|
||||
value = tensor.detach().cpu().contiguous()
|
||||
return (
|
||||
f"{value.dtype}|{tuple(value.shape)}|".encode()
|
||||
+ value.reshape(-1).view(torch.uint8).numpy().tobytes()
|
||||
)
|
||||
|
||||
|
||||
def named_state_hash(
|
||||
model: nn.Module, *, include_mixers: bool | None
|
||||
) -> str:
|
||||
digest = hashlib.sha256()
|
||||
for name, tensor in sorted(model.state_dict().items()):
|
||||
is_mixer = name.startswith("mixers.") or name.startswith("output_mixer.")
|
||||
if include_mixers is not None and is_mixer != include_mixers:
|
||||
continue
|
||||
digest.update(name.encode())
|
||||
digest.update(b"\0")
|
||||
digest.update(tensor_bytes(tensor))
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def state_structure_hash(
|
||||
model: nn.Module, *, include_mixers: bool | None
|
||||
) -> tuple[str, int, int]:
|
||||
digest = hashlib.sha256()
|
||||
tensor_count = 0
|
||||
element_count = 0
|
||||
for name, tensor in sorted(model.state_dict().items()):
|
||||
is_mixer = name.startswith("mixers.") or name.startswith("output_mixer.")
|
||||
if include_mixers is not None and is_mixer != include_mixers:
|
||||
continue
|
||||
digest.update(
|
||||
f"{name}|{tuple(tensor.shape)}|{tensor.dtype}|{tensor.numel()}\n".encode()
|
||||
)
|
||||
tensor_count += 1
|
||||
element_count += tensor.numel()
|
||||
return digest.hexdigest(), tensor_count, element_count
|
||||
|
||||
|
||||
def recursive_state_hash(value: Any) -> str:
|
||||
digest = hashlib.sha256()
|
||||
|
||||
def visit(path: str, item: Any) -> None:
|
||||
if torch.is_tensor(item):
|
||||
digest.update(f"{path}|tensor|".encode())
|
||||
digest.update(tensor_bytes(item))
|
||||
elif isinstance(item, dict):
|
||||
digest.update(f"{path}|dict|{len(item)}\n".encode())
|
||||
for key in sorted(item, key=lambda candidate: str(candidate)):
|
||||
visit(f"{path}/{key}", item[key])
|
||||
elif isinstance(item, (list, tuple)):
|
||||
digest.update(f"{path}|sequence|{len(item)}\n".encode())
|
||||
for index, child in enumerate(item):
|
||||
visit(f"{path}/{index}", child)
|
||||
else:
|
||||
digest.update(f"{path}|scalar|{repr(item)}\n".encode())
|
||||
|
||||
visit("root", value)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
@dataclass
|
||||
class ActivationTrace:
|
||||
block_outputs: list[torch.Tensor]
|
||||
layer_input_rms: list[float]
|
||||
branch_output_rms: list[float]
|
||||
stream_state_rms: list[float]
|
||||
depth_weights: list[dict[str, Any]]
|
||||
output_weights: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def rms(value: torch.Tensor) -> float:
|
||||
return value.float().square().mean().sqrt().detach().cpu().item()
|
||||
|
||||
|
||||
class GradientLanguageModel(round04.ReducedLanguageModel):
|
||||
"""Round 04 trunk with aligned post-MLP activation capture."""
|
||||
|
||||
def forward(
|
||||
self, input_ids: torch.Tensor, capture: bool = False
|
||||
) -> tuple[torch.Tensor, ActivationTrace | None]:
|
||||
embedded = self.embed(input_ids)
|
||||
trace = ActivationTrace([], [], [], [], []) if capture else None
|
||||
|
||||
if self.architecture == "baseline":
|
||||
hidden = embedded
|
||||
for block in self.blocks:
|
||||
attention_input = hidden
|
||||
attention_output = block.attention(block.attention_norm(attention_input))
|
||||
hidden = hidden + attention_output
|
||||
if trace is not None:
|
||||
trace.layer_input_rms.append(rms(attention_input))
|
||||
trace.branch_output_rms.append(rms(attention_output))
|
||||
trace.stream_state_rms.append(rms(hidden))
|
||||
mlp_input = hidden
|
||||
mlp_output = block.mlp(block.mlp_norm(mlp_input))
|
||||
hidden = hidden + mlp_output
|
||||
if trace is not None:
|
||||
hidden.retain_grad()
|
||||
trace.block_outputs.append(hidden)
|
||||
trace.layer_input_rms.append(rms(mlp_input))
|
||||
trace.branch_output_rms.append(rms(mlp_output))
|
||||
trace.stream_state_rms.append(rms(hidden))
|
||||
else:
|
||||
completed = [embedded]
|
||||
partial: torch.Tensor | None = None
|
||||
mixer_index = 0
|
||||
for block in self.blocks:
|
||||
for branch_index in range(2):
|
||||
sources = completed + ([] if partial is None else [partial])
|
||||
branch_input, weights = self.mixers[mixer_index](
|
||||
sources, capture
|
||||
)
|
||||
mixer_index += 1
|
||||
if branch_index == 0:
|
||||
branch_output = block.attention(
|
||||
block.attention_norm(branch_input)
|
||||
)
|
||||
else:
|
||||
branch_output = block.mlp(block.mlp_norm(branch_input))
|
||||
branch_for_residual = branch_output.float()
|
||||
partial = (
|
||||
branch_for_residual
|
||||
if partial is None
|
||||
else partial + branch_for_residual
|
||||
)
|
||||
if trace is not None:
|
||||
trace.layer_input_rms.append(rms(branch_input))
|
||||
trace.branch_output_rms.append(rms(branch_output))
|
||||
trace.stream_state_rms.append(rms(partial))
|
||||
trace.depth_weights.append(weights or {})
|
||||
if branch_index == 1:
|
||||
partial.retain_grad()
|
||||
trace.block_outputs.append(partial)
|
||||
if mixer_index % round04.SUBLAYERS_PER_BLOCK == 0:
|
||||
completed.append(partial)
|
||||
partial = None
|
||||
if partial is not None or len(completed) != BLOCK_GROUPS + 1:
|
||||
raise RuntimeError("Block AttnRes aggregation contract failed")
|
||||
if self.output_mixer is None:
|
||||
raise RuntimeError("Block AttnRes output mixer missing")
|
||||
hidden, output_weights = self.output_mixer(completed, capture)
|
||||
if trace is not None:
|
||||
trace.output_weights = output_weights
|
||||
|
||||
normalized = self.final_norm(hidden)
|
||||
logits = F.linear(normalized, self.token_embedding.weight)
|
||||
return logits, trace
|
||||
|
||||
|
||||
def cross_entropy(logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:
|
||||
return F.cross_entropy(
|
||||
logits.float().reshape(-1, VOCABULARY), targets.reshape(-1)
|
||||
)
|
||||
|
||||
|
||||
def learning_rate(step: int, total_steps: int) -> float:
|
||||
if step <= WARMUP_STEPS:
|
||||
return PEAK_LR * step / WARMUP_STEPS
|
||||
progress = (step - WARMUP_STEPS) / max(1, total_steps - WARMUP_STEPS)
|
||||
cosine = 0.5 * (1 + math.cos(math.pi * progress))
|
||||
return MIN_LR + (PEAK_LR - MIN_LR) * cosine
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def evaluate(
|
||||
model: GradientLanguageModel,
|
||||
corpus: Any,
|
||||
window_count: int,
|
||||
eval_batch_size: int,
|
||||
) -> dict[str, float]:
|
||||
model.eval()
|
||||
loss_sum = 0.0
|
||||
target_count = 0
|
||||
for begin in range(0, window_count, eval_batch_size):
|
||||
end = min(begin + eval_batch_size, window_count)
|
||||
inputs, targets = corpus.fixed_batch(
|
||||
corpus.validation_starts, begin, end
|
||||
)
|
||||
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
||||
logits, _ = model(inputs)
|
||||
loss = F.cross_entropy(
|
||||
logits.float().reshape(-1, VOCABULARY),
|
||||
targets.reshape(-1),
|
||||
reduction="sum",
|
||||
)
|
||||
loss_sum += loss.detach().cpu().item()
|
||||
target_count += targets.numel()
|
||||
nats = loss_sum / target_count
|
||||
return {"cross_entropy_nats": nats, "bits_per_byte": nats / math.log(2)}
|
||||
|
||||
|
||||
def mean(values: Iterable[float]) -> float:
|
||||
return statistics.fmean(values)
|
||||
|
||||
|
||||
def depth_statistics(values: list[float]) -> dict[str, Any]:
|
||||
average = mean(values)
|
||||
variance = mean((value - average) ** 2 for value in values)
|
||||
quartile = len(values) // 4
|
||||
first = mean(values[:quartile])
|
||||
last = mean(values[-quartile:])
|
||||
ratio = first / last
|
||||
return {
|
||||
"mean": average,
|
||||
"population_cv": math.sqrt(variance) / average,
|
||||
"normalized": [value / average for value in values],
|
||||
"first_quartile_mean": first,
|
||||
"last_quartile_mean": last,
|
||||
"first_to_last_ratio": ratio,
|
||||
"imbalance_abs_log_ratio": abs(math.log(ratio)),
|
||||
}
|
||||
|
||||
|
||||
def core_parameter_gradient_rms(model: GradientLanguageModel) -> list[float]:
|
||||
values = []
|
||||
for block in model.blocks:
|
||||
sum_square = 0.0
|
||||
count = 0
|
||||
for parameter in block.parameters():
|
||||
if parameter.grad is None:
|
||||
raise RuntimeError("missing core parameter gradient")
|
||||
gradient = parameter.grad.detach().float()
|
||||
if not torch.isfinite(gradient).all():
|
||||
raise RuntimeError("non-finite core parameter gradient")
|
||||
sum_square += gradient.square().sum().detach().cpu().item()
|
||||
count += gradient.numel()
|
||||
values.append(math.sqrt(sum_square / count))
|
||||
return values
|
||||
|
||||
|
||||
def activation_storage_unique(outputs: list[torch.Tensor]) -> bool:
|
||||
pointers = [output.untyped_storage().data_ptr() for output in outputs]
|
||||
return len(pointers) == len(set(pointers))
|
||||
|
||||
|
||||
def diagnostic(
|
||||
model: GradientLanguageModel,
|
||||
corpus: Any,
|
||||
window_count: int,
|
||||
*,
|
||||
loss_scale: float = 1.0,
|
||||
) -> dict[str, Any]:
|
||||
model.eval()
|
||||
model.zero_grad(set_to_none=True)
|
||||
inputs, targets = corpus.fixed_batch(
|
||||
corpus.diagnostic_starts, 0, window_count
|
||||
)
|
||||
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
||||
logits, trace = model(inputs, capture=True)
|
||||
unscaled_loss = cross_entropy(logits, targets)
|
||||
loss = unscaled_loss * loss_scale
|
||||
if trace is None or len(trace.block_outputs) != len(model.blocks):
|
||||
raise RuntimeError("aligned activation capture count mismatch")
|
||||
expected_shape = (window_count, CONTEXT, round04.D_MODEL)
|
||||
if any(tuple(output.shape) != expected_shape for output in trace.block_outputs):
|
||||
raise RuntimeError("aligned activation capture shape mismatch")
|
||||
if any(output.dtype != torch.float32 for output in trace.block_outputs):
|
||||
raise RuntimeError("aligned activation capture must use FP32 residual state")
|
||||
if not activation_storage_unique(trace.block_outputs):
|
||||
raise RuntimeError("captured block outputs alias storage")
|
||||
loss.backward()
|
||||
|
||||
activation_grad_rms = []
|
||||
activation_output_rms = []
|
||||
activation_dtypes = []
|
||||
for output in trace.block_outputs:
|
||||
if output.grad is None:
|
||||
raise RuntimeError("captured activation gradient is None")
|
||||
gradient = output.grad.detach().float()
|
||||
if not torch.isfinite(gradient).all():
|
||||
raise RuntimeError("captured activation gradient is non-finite")
|
||||
activation_grad_rms.append(
|
||||
gradient.square().mean().sqrt().detach().cpu().item()
|
||||
)
|
||||
activation_output_rms.append(rms(output))
|
||||
activation_dtypes.append(str(output.dtype))
|
||||
parameter_grad_rms = core_parameter_gradient_rms(model)
|
||||
|
||||
return {
|
||||
"loss_nats": unscaled_loss.detach().cpu().item(),
|
||||
"bits_per_byte": unscaled_loss.detach().cpu().item() / math.log(2),
|
||||
"loss_scale": loss_scale,
|
||||
"capture": {
|
||||
"count": len(trace.block_outputs),
|
||||
"shape": list(expected_shape),
|
||||
"dtypes": activation_dtypes,
|
||||
"all_gradients_finite": True,
|
||||
"all_gradients_present": True,
|
||||
"storage_unique": True,
|
||||
"position": (
|
||||
"post-MLP Transformer-block output; Block AttnRes is captured "
|
||||
"before aggregation-partial reset"
|
||||
),
|
||||
},
|
||||
"activation_grad_rms_by_block": activation_grad_rms,
|
||||
"activation_grad_statistics": depth_statistics(activation_grad_rms),
|
||||
"activation_output_rms_by_block": activation_output_rms,
|
||||
"activation_output_statistics": depth_statistics(activation_output_rms),
|
||||
"core_parameter_grad_rms_by_block": parameter_grad_rms,
|
||||
"core_parameter_grad_statistics": depth_statistics(parameter_grad_rms),
|
||||
"layer_input_rms_by_sublayer": trace.layer_input_rms,
|
||||
"branch_output_rms_by_sublayer": trace.branch_output_rms,
|
||||
"stream_state_rms_by_sublayer": trace.stream_state_rms,
|
||||
"depth_weights": trace.depth_weights,
|
||||
"output_weights": trace.output_weights,
|
||||
}
|
||||
|
||||
|
||||
def loss_scale_gate(
|
||||
model: GradientLanguageModel, corpus: Any, window_count: int
|
||||
) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
base = diagnostic(model, corpus, window_count, loss_scale=1.0)
|
||||
doubled = diagnostic(model, corpus, window_count, loss_scale=2.0)
|
||||
base_values = base["activation_grad_rms_by_block"]
|
||||
doubled_values = doubled["activation_grad_rms_by_block"]
|
||||
ratios = [
|
||||
doubled_value / base_value
|
||||
for base_value, doubled_value in zip(base_values, doubled_values)
|
||||
]
|
||||
base_stats = base["activation_grad_statistics"]
|
||||
doubled_stats = doubled["activation_grad_statistics"]
|
||||
cv_delta = abs(
|
||||
doubled_stats["population_cv"] - base_stats["population_cv"]
|
||||
)
|
||||
ratio_delta = abs(
|
||||
doubled_stats["first_to_last_ratio"]
|
||||
- base_stats["first_to_last_ratio"]
|
||||
)
|
||||
normalized_max_delta = max(
|
||||
abs(left - right)
|
||||
for left, right in zip(
|
||||
base_stats["normalized"], doubled_stats["normalized"]
|
||||
)
|
||||
)
|
||||
passed = (
|
||||
all(abs(ratio - 2.0) <= 1e-5 for ratio in ratios)
|
||||
and cv_delta <= 1e-6
|
||||
and ratio_delta <= 1e-6
|
||||
and normalized_max_delta <= 1e-6
|
||||
)
|
||||
gate = {
|
||||
"passed": passed,
|
||||
"per_block_scale_ratios": ratios,
|
||||
"max_abs_scale_ratio_error": max(abs(ratio - 2.0) for ratio in ratios),
|
||||
"population_cv_abs_delta": cv_delta,
|
||||
"first_to_last_ratio_abs_delta": ratio_delta,
|
||||
"normalized_spectrum_max_abs_delta": normalized_max_delta,
|
||||
"thresholds": {
|
||||
"scale_ratio_abs": 1e-5,
|
||||
"shape_abs": 1e-6,
|
||||
},
|
||||
}
|
||||
if not passed:
|
||||
raise RuntimeError(f"loss-scale diagnostic gate failed: {gate}")
|
||||
return base, gate
|
||||
|
||||
|
||||
def percentile(values: list[float], quantile: float) -> float:
|
||||
return float(np.quantile(np.asarray(values, dtype=np.float64), quantile))
|
||||
|
||||
|
||||
def parameter_inventory(model: GradientLanguageModel) -> dict[str, int]:
|
||||
total = sum(parameter.numel() for parameter in model.parameters())
|
||||
mixer = sum(
|
||||
parameter.numel()
|
||||
for name, parameter in model.named_parameters()
|
||||
if name.startswith("mixers.") or name.startswith("output_mixer.")
|
||||
)
|
||||
return {
|
||||
"total": total,
|
||||
"core": total - mixer,
|
||||
"mixer": mixer,
|
||||
"embedding": (
|
||||
model.token_embedding.weight.numel()
|
||||
+ model.position_embedding.weight.numel()
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def model_input_gate_hashes(
|
||||
corpus: Any, manifest: dict[str, Any], seed: int, batch_size: int
|
||||
) -> dict[str, str]:
|
||||
values: dict[str, str] = {}
|
||||
for step in manifest["windows"]["gate_steps"]:
|
||||
raw_digest = hashlib.sha256()
|
||||
for row in range(batch_size):
|
||||
start = round04.window_start(seed, step, row, len(corpus.train))
|
||||
raw_digest.update(
|
||||
np.asarray(
|
||||
corpus.train[start : start + CONTEXT + 1], dtype=np.uint8
|
||||
).tobytes()
|
||||
)
|
||||
expected_raw_hash = manifest["windows"][
|
||||
"gate_training_tensor_sha256"
|
||||
][str(seed)][str(step)]
|
||||
if raw_digest.hexdigest() != expected_raw_hash:
|
||||
raise RuntimeError(f"manifest gate tensor mismatch at step {step}")
|
||||
inputs, targets = corpus.training_batch(seed, step, batch_size)
|
||||
digest = hashlib.sha256()
|
||||
digest.update(tensor_bytes(inputs))
|
||||
digest.update(tensor_bytes(targets))
|
||||
values[str(step)] = digest.hexdigest()
|
||||
return values
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required by the frozen protocol")
|
||||
if args.seed not in EXPECTED_SEEDS:
|
||||
raise ValueError(f"seed is not preregistered: {args.seed}")
|
||||
configure_round04_globals(args.depth)
|
||||
configure_determinism(args.seed)
|
||||
device = torch.device("cuda")
|
||||
|
||||
manifest = json.loads(args.manifest.read_text())
|
||||
if manifest["protocol_id"] != PROTOCOL_ID:
|
||||
raise ValueError("manifest protocol mismatch")
|
||||
if manifest["windows"]["formal_steps"] != FORMAL_STEPS:
|
||||
raise ValueError("manifest formal-step mismatch")
|
||||
corpus = round04.ByteCorpus(args.cache_dir, manifest, device)
|
||||
|
||||
model = GradientLanguageModel(args.architecture).to(device)
|
||||
public_structure_hash, public_tensors, public_elements = state_structure_hash(
|
||||
model, include_mixers=False
|
||||
)
|
||||
initial_public_hash = named_state_hash(model, include_mixers=False)
|
||||
initial_mixer_hash = (
|
||||
named_state_hash(model, include_mixers=True)
|
||||
if args.architecture == "block"
|
||||
else None
|
||||
)
|
||||
input_gate_hashes = model_input_gate_hashes(
|
||||
corpus, manifest, args.seed, args.batch_size
|
||||
)
|
||||
|
||||
decay_parameters: list[nn.Parameter] = []
|
||||
no_decay_parameters: list[nn.Parameter] = []
|
||||
for parameter in model.parameters():
|
||||
(decay_parameters if parameter.ndim >= 2 else no_decay_parameters).append(
|
||||
parameter
|
||||
)
|
||||
optimizer = torch.optim.AdamW(
|
||||
[
|
||||
{"params": decay_parameters, "weight_decay": WEIGHT_DECAY},
|
||||
{"params": no_decay_parameters, "weight_decay": 0.0},
|
||||
],
|
||||
lr=PEAK_LR,
|
||||
betas=BETAS,
|
||||
eps=ADAM_EPS,
|
||||
)
|
||||
|
||||
evaluation_steps = sorted(
|
||||
set(step for step in DIAGNOSTIC_STEPS if step <= args.steps)
|
||||
| {0, args.steps}
|
||||
)
|
||||
evaluations = [
|
||||
{
|
||||
"step": 0,
|
||||
**evaluate(
|
||||
model, corpus, args.validation_windows, args.eval_batch_size
|
||||
),
|
||||
}
|
||||
]
|
||||
if args.run_kind == "smoke":
|
||||
initial_diagnostic, gradient_gate = loss_scale_gate(
|
||||
model, corpus, args.diagnostic_windows
|
||||
)
|
||||
else:
|
||||
initial_diagnostic = diagnostic(
|
||||
model, corpus, args.diagnostic_windows
|
||||
)
|
||||
gradient_gate = None
|
||||
diagnostics = [{"step": 0, **initial_diagnostic}]
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "diagnostic",
|
||||
"step": 0,
|
||||
"architecture": args.architecture,
|
||||
"depth": args.depth,
|
||||
"validation_bpc": evaluations[0]["bits_per_byte"],
|
||||
"activation_gradient_cv": initial_diagnostic[
|
||||
"activation_grad_statistics"
|
||||
]["population_cv"],
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
model.zero_grad(set_to_none=True)
|
||||
|
||||
training_history: list[dict[str, float | int]] = []
|
||||
step_times: list[float] = []
|
||||
model.train()
|
||||
for step in range(1, args.steps + 1):
|
||||
lr = learning_rate(step, args.steps)
|
||||
for group in optimizer.param_groups:
|
||||
group["lr"] = lr
|
||||
inputs, targets = corpus.training_batch(args.seed, step, args.batch_size)
|
||||
optimizer.zero_grad(set_to_none=True)
|
||||
|
||||
torch.cuda.synchronize()
|
||||
started = time.perf_counter()
|
||||
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
||||
logits, _ = model(inputs)
|
||||
loss = cross_entropy(logits, targets)
|
||||
if not torch.isfinite(loss):
|
||||
raise RuntimeError(f"non-finite loss at step {step}: {loss}")
|
||||
loss.backward()
|
||||
unclipped_norm = torch.nn.utils.clip_grad_norm_(
|
||||
model.parameters(), GRAD_CLIP
|
||||
)
|
||||
optimizer.step()
|
||||
torch.cuda.synchronize()
|
||||
elapsed_ms = (time.perf_counter() - started) * 1000
|
||||
|
||||
if step == args.timing_warmup:
|
||||
torch.cuda.reset_peak_memory_stats()
|
||||
elif step > args.timing_warmup:
|
||||
step_times.append(elapsed_ms)
|
||||
if step == 1 or step % 10 == 0 or step == args.steps:
|
||||
training_history.append(
|
||||
{
|
||||
"step": step,
|
||||
"loss_nats": loss.detach().cpu().item(),
|
||||
"bits_per_byte": loss.detach().cpu().item() / math.log(2),
|
||||
"learning_rate": lr,
|
||||
"unclipped_grad_norm": float(unclipped_norm.detach().cpu()),
|
||||
}
|
||||
)
|
||||
|
||||
if step in evaluation_steps and step != 0:
|
||||
evaluations.append(
|
||||
{
|
||||
"step": step,
|
||||
**evaluate(
|
||||
model,
|
||||
corpus,
|
||||
args.validation_windows,
|
||||
args.eval_batch_size,
|
||||
),
|
||||
}
|
||||
)
|
||||
diagnostics.append(
|
||||
{
|
||||
"step": step,
|
||||
**diagnostic(
|
||||
model, corpus, args.diagnostic_windows
|
||||
),
|
||||
}
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "diagnostic",
|
||||
"step": step,
|
||||
"architecture": args.architecture,
|
||||
"depth": args.depth,
|
||||
"validation_bpc": evaluations[-1]["bits_per_byte"],
|
||||
"activation_gradient_cv": diagnostics[-1][
|
||||
"activation_grad_statistics"
|
||||
]["population_cv"],
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
model.zero_grad(set_to_none=True)
|
||||
model.train()
|
||||
|
||||
training_peak_allocated = torch.cuda.max_memory_allocated()
|
||||
training_peak_reserved = torch.cuda.max_memory_reserved()
|
||||
final_public_hash = named_state_hash(model, include_mixers=False)
|
||||
final_mixer_hash = (
|
||||
named_state_hash(model, include_mixers=True)
|
||||
if args.architecture == "block"
|
||||
else None
|
||||
)
|
||||
final_full_hash = named_state_hash(model, include_mixers=None)
|
||||
optimizer_hash = recursive_state_hash(optimizer.state_dict())
|
||||
timing = {
|
||||
"warmup_steps_excluded": args.timing_warmup,
|
||||
"measured_steps": len(step_times),
|
||||
"mean_ms": mean(step_times) if step_times else None,
|
||||
"median_ms": statistics.median(step_times) if step_times else None,
|
||||
"p95_ms": percentile(step_times, 0.95) if step_times else None,
|
||||
"peak_allocated_bytes": training_peak_allocated,
|
||||
"peak_reserved_bytes": training_peak_reserved,
|
||||
}
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"run_kind": args.run_kind,
|
||||
"architecture": args.architecture,
|
||||
"depth": args.depth,
|
||||
"seed": args.seed,
|
||||
"steps": args.steps,
|
||||
"batch_size": args.batch_size,
|
||||
"target_bytes_seen": args.steps * args.batch_size * CONTEXT,
|
||||
"manifest": {
|
||||
"path": str(args.manifest),
|
||||
"file_sha256": file_sha256(args.manifest),
|
||||
"formal_schedule_sha256": manifest["windows"][
|
||||
"formal_schedule_sha256"
|
||||
],
|
||||
"validation_tensor_sha256": manifest["windows"][
|
||||
"validation_tensor_sha256"
|
||||
],
|
||||
"diagnostic_tensor_sha256": manifest["windows"][
|
||||
"diagnostic_tensor_sha256"
|
||||
],
|
||||
"input_gate_tensor_hashes": input_gate_hashes,
|
||||
},
|
||||
"model": {
|
||||
"layers": args.depth,
|
||||
"sublayers": args.depth * 2,
|
||||
"attnres_aggregation_groups": BLOCK_GROUPS,
|
||||
"sublayers_per_attnres_group": args.depth * 2 // BLOCK_GROUPS,
|
||||
"transformer_blocks_per_attnres_group": args.depth // BLOCK_GROUPS,
|
||||
"d_model": round04.D_MODEL,
|
||||
"heads": round04.HEADS,
|
||||
"d_head": round04.D_HEAD,
|
||||
"d_ff": round04.D_FF,
|
||||
"context": CONTEXT,
|
||||
"vocabulary": VOCABULARY,
|
||||
"parameters": parameter_inventory(model),
|
||||
},
|
||||
"optimizer": {
|
||||
"name": "AdamW",
|
||||
"betas": list(BETAS),
|
||||
"epsilon": ADAM_EPS,
|
||||
"weight_decay_ndim_ge_2": WEIGHT_DECAY,
|
||||
"peak_lr": PEAK_LR,
|
||||
"min_lr": MIN_LR,
|
||||
"warmup_steps": WARMUP_STEPS,
|
||||
"grad_clip": GRAD_CLIP,
|
||||
},
|
||||
"hashes": {
|
||||
"initial_public_parameter_structure": public_structure_hash,
|
||||
"initial_public_parameter_tensors": public_tensors,
|
||||
"initial_public_parameter_elements": public_elements,
|
||||
"initial_public_parameters": initial_public_hash,
|
||||
"initial_mixer_parameters": initial_mixer_hash,
|
||||
"final_public_parameters": final_public_hash,
|
||||
"final_mixer_parameters": final_mixer_hash,
|
||||
"final_model_state": final_full_hash,
|
||||
"final_optimizer_state": optimizer_hash,
|
||||
},
|
||||
"evaluations": evaluations,
|
||||
"diagnostics": diagnostics,
|
||||
"training_history": training_history,
|
||||
"gradient_gate": gradient_gate,
|
||||
"timing": timing,
|
||||
"environment": {
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"cuda": torch.version.cuda,
|
||||
"gpu": torch.cuda.get_device_name(0),
|
||||
"compute_capability": list(torch.cuda.get_device_capability(0)),
|
||||
"cublas_workspace_config": os.environ["CUBLAS_WORKSPACE_CONFIG"],
|
||||
"deterministic_algorithms": torch.are_deterministic_algorithms_enabled(),
|
||||
"autocast": "cuda-bfloat16-forward-fp32-cross-entropy",
|
||||
"compile": False,
|
||||
},
|
||||
}
|
||||
result["canonical_sha256_without_self"] = canonical_sha256(result)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = args.output.with_suffix(args.output.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
os.replace(temporary, args.output)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"run_kind": args.run_kind,
|
||||
"architecture": args.architecture,
|
||||
"depth": args.depth,
|
||||
"seed": args.seed,
|
||||
"steps": args.steps,
|
||||
"final_bpc": evaluations[-1]["bits_per_byte"],
|
||||
"final_activation_gradient_cv": diagnostics[-1][
|
||||
"activation_grad_statistics"
|
||||
]["population_cv"],
|
||||
"canonical_sha256": result["canonical_sha256_without_self"],
|
||||
"timing": timing,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,50 @@
|
||||
# Attention Residuals local mixer-path diagnostics
|
||||
|
||||
This directory implements preregistered protocol
|
||||
`llm-atlas-k3-attnres-local-path-v1`.
|
||||
|
||||
It is a targeted follow-up to Round 06. It exact-replays the same depth-32
|
||||
Block training and keeps the learned forward unchanged while switching source
|
||||
value-gradient coefficients only at frozen mixer scopes. It is not a Kimi K3
|
||||
checkpoint run, a trainable variant, an additive attribution, or a reproduction
|
||||
of unpublished Figure 5 telemetry.
|
||||
|
||||
## Frozen environment
|
||||
|
||||
```text
|
||||
Python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python
|
||||
PyTorch 2.11.0+cu128
|
||||
GPU NVIDIA GeForce RTX 5090
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8
|
||||
```
|
||||
|
||||
The preregistration was committed as `6911efc` before the runner or any result
|
||||
file existed.
|
||||
|
||||
## Step-0 smoke
|
||||
|
||||
```bash
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8 \
|
||||
/home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python \
|
||||
experiments/k3/attnres_local_path/train.py \
|
||||
--run-kind smoke \
|
||||
--seed 2026073001 \
|
||||
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
||||
--data-manifest experiments/k3/attnres_gradient/manifest.json \
|
||||
--parent-manifest experiments/k3/attnres_spike/manifest.json \
|
||||
--manifest experiments/k3/attnres_local_path/manifest.json \
|
||||
--output /home/wuyang/.cache/llm-atlas/k3-attnres-local-path-v1/smoke/seed-2026073001.json
|
||||
```
|
||||
|
||||
The first smoke passed all 14-mode forward-identity, selector, Round 06 endpoint,
|
||||
initialization-negative-control, parent-learned, and loss-scale gates. Its
|
||||
canonical content hash is
|
||||
`f708200f4fb122f61f30b97393839382a2094a71a8ea5d48cf47fb7fa094e69b`.
|
||||
|
||||
Formal cells use the same command with `--run-kind formal`, one of the three
|
||||
manifest seeds, and a new output path. The independent seed-2026073001 run uses
|
||||
`--run-kind replay`.
|
||||
|
||||
Only `analyze.py` may calculate the global log gap, sufficiency/restoration
|
||||
scores, and preregistered gates. The site consumes its frozen aggregate rather
|
||||
than reimplementing thresholds in TypeScript.
|
||||
@@ -0,0 +1,498 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Aggregate and gate preregistered Round 07 local-path results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-local-path-v1"
|
||||
METRICS = ("spike_contrast", "peak_normalized")
|
||||
SUFFICIENCY_MODES = (
|
||||
"uniform_group_6_only",
|
||||
"uniform_group_7_only",
|
||||
"uniform_groups_6_7_only",
|
||||
"uniform_group_6_attention_only",
|
||||
"uniform_group_6_mlp_only",
|
||||
"uniform_group_7_attention_only",
|
||||
"uniform_group_7_mlp_only",
|
||||
"uniform_output_only",
|
||||
"uniform_depth_all",
|
||||
"uniform_all",
|
||||
)
|
||||
RESTORATION_MODES = (
|
||||
"uniform_except_group_6",
|
||||
"uniform_except_group_7",
|
||||
"uniform_except_groups_6_7",
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--formal", type=Path, action="append", required=True
|
||||
)
|
||||
parser.add_argument("--replay", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def read_result(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text())
|
||||
if value["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError(f"protocol mismatch: {path}")
|
||||
expected = value["canonical_sha256_without_self"]
|
||||
without_self = {
|
||||
key: item
|
||||
for key, item in value.items()
|
||||
if key != "canonical_sha256_without_self"
|
||||
}
|
||||
if canonical_sha256(without_self) != expected:
|
||||
raise RuntimeError(f"canonical self-hash mismatch: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def replay_payload(value: dict[str, Any]) -> dict[str, Any]:
|
||||
cleaned = copy.deepcopy(value)
|
||||
for key in ("run_kind", "timing", "canonical_sha256_without_self"):
|
||||
cleaned.pop(key)
|
||||
return cleaned
|
||||
|
||||
|
||||
def primary_metrics(mode: dict[str, Any]) -> dict[str, float]:
|
||||
stats = mode["positions"]["post_mlp_state"]["reductions"][
|
||||
"element_rms"
|
||||
]["statistics"]
|
||||
return {name: float(stats[name]) for name in METRICS}
|
||||
|
||||
|
||||
def all_true(values: list[bool]) -> bool:
|
||||
return len(values) > 0 and all(values)
|
||||
|
||||
|
||||
def mixed_status(
|
||||
by_seed_metric: dict[str, dict[str, dict[str, Any]]],
|
||||
threshold: float,
|
||||
) -> dict[str, Any]:
|
||||
passed = []
|
||||
signs = []
|
||||
metric_passes = {metric: [] for metric in METRICS}
|
||||
for seed_values in by_seed_metric.values():
|
||||
for metric in METRICS:
|
||||
score = seed_values[metric]["score"]
|
||||
cell_passed = score is not None and score >= threshold
|
||||
passed.append(cell_passed)
|
||||
metric_passes[metric].append(cell_passed)
|
||||
if score is not None:
|
||||
signs.append(1 if score >= 0 else -1)
|
||||
reasons = []
|
||||
if any(passed) and not all(passed):
|
||||
reasons.append("seed_or_metric_pass_split")
|
||||
if metric_passes[METRICS[0]] != metric_passes[METRICS[1]]:
|
||||
reasons.append("metric_direction_split")
|
||||
if len(set(signs)) > 1:
|
||||
reasons.append("score_sign_split")
|
||||
return {
|
||||
"passed": all_true(passed),
|
||||
"threshold": threshold,
|
||||
"required_cells": len(passed),
|
||||
"passed_cells": sum(passed),
|
||||
"mixed": bool(reasons),
|
||||
"mixed_reasons": reasons,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
manifest = json.loads(args.manifest.read_text())
|
||||
if manifest["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError("manifest protocol mismatch")
|
||||
if len(args.formal) != 3:
|
||||
raise RuntimeError("exactly three formal result paths are required")
|
||||
|
||||
formal_pairs = [(path, read_result(path)) for path in args.formal]
|
||||
formal_pairs.sort(key=lambda item: item[1]["seed"])
|
||||
expected_seeds = manifest["formal_seeds"]
|
||||
if [value["seed"] for _, value in formal_pairs] != expected_seeds:
|
||||
raise RuntimeError("formal seeds do not match manifest")
|
||||
for path, value in formal_pairs:
|
||||
if value["run_kind"] != "formal" or value["steps"] != 8000:
|
||||
raise RuntimeError(f"invalid formal cell: {path}")
|
||||
if not value["round06_equivalence"]["passed"]:
|
||||
raise RuntimeError(f"parent equivalence failed: {path}")
|
||||
for diagnostic in value["diagnostics"]:
|
||||
if not diagnostic["parent_learned_round06_exact"]:
|
||||
raise RuntimeError(f"parent diagnostic mismatch: {path}")
|
||||
if diagnostic["local_matrix"] is not None:
|
||||
for gate in ("forward_identity_gate", "endpoint_exactness"):
|
||||
if not diagnostic[gate]["passed"]:
|
||||
raise RuntimeError(f"{gate} failed: {path}")
|
||||
for mode in manifest["matrix_modes"]:
|
||||
if not diagnostic["local_matrix"][mode]["selector"][
|
||||
"passed"
|
||||
]:
|
||||
raise RuntimeError(f"selector failed: {path}:{mode}")
|
||||
step0 = value["diagnostics"][0]
|
||||
if (
|
||||
not step0["initialization_negative_control"]["passed"]
|
||||
or not step0["loss_scale_gate"]["passed"]
|
||||
):
|
||||
raise RuntimeError(f"step-0 control failed: {path}")
|
||||
|
||||
replay = read_result(args.replay)
|
||||
if (
|
||||
replay["run_kind"] != "replay"
|
||||
or replay["seed"] != expected_seeds[0]
|
||||
or replay["steps"] != 8000
|
||||
):
|
||||
raise RuntimeError("invalid replay cell")
|
||||
replay_exact = (
|
||||
replay_payload(formal_pairs[0][1]) == replay_payload(replay)
|
||||
)
|
||||
if not replay_exact:
|
||||
raise RuntimeError("formal seed1 and replay are not canonical exact")
|
||||
|
||||
thresholds = manifest["thresholds"]
|
||||
cells = []
|
||||
sufficiency_by_mode: dict[str, dict[str, dict[str, Any]]] = {
|
||||
mode: {} for mode in SUFFICIENCY_MODES
|
||||
}
|
||||
restoration_by_mode: dict[str, dict[str, dict[str, Any]]] = {
|
||||
mode: {} for mode in RESTORATION_MODES
|
||||
}
|
||||
|
||||
for path, value in formal_pairs:
|
||||
seed_key = str(value["seed"])
|
||||
final = value["diagnostics"][-1]["local_matrix"]
|
||||
mode_metrics = {
|
||||
mode: primary_metrics(final[mode])
|
||||
for mode in manifest["matrix_modes"]
|
||||
}
|
||||
global_metrics = {}
|
||||
for metric in METRICS:
|
||||
reference = mode_metrics["detached_learned"][metric]
|
||||
uniform_all = mode_metrics["uniform_all"][metric]
|
||||
log_gap = math.log(reference / uniform_all)
|
||||
relative_drop = (reference - uniform_all) / reference
|
||||
established = (
|
||||
reference
|
||||
> thresholds["positive_denominator_epsilon"]
|
||||
and uniform_all
|
||||
> thresholds["positive_denominator_epsilon"]
|
||||
and log_gap > 0
|
||||
and relative_drop
|
||||
>= thresholds["global_relative_drop_minimum"]
|
||||
)
|
||||
global_metrics[metric] = {
|
||||
"reference": reference,
|
||||
"uniform_all": uniform_all,
|
||||
"log_gap": log_gap,
|
||||
"relative_drop": relative_drop,
|
||||
"established": established,
|
||||
}
|
||||
for mode in SUFFICIENCY_MODES:
|
||||
sufficiency_by_mode[mode][seed_key] = {}
|
||||
for metric in METRICS:
|
||||
established = global_metrics[metric]["established"]
|
||||
score = (
|
||||
math.log(
|
||||
mode_metrics["detached_learned"][metric]
|
||||
/ mode_metrics[mode][metric]
|
||||
)
|
||||
/ global_metrics[metric]["log_gap"]
|
||||
if established
|
||||
else None
|
||||
)
|
||||
sufficiency_by_mode[mode][seed_key][metric] = {
|
||||
"score": score,
|
||||
"metric_value": mode_metrics[mode][metric],
|
||||
"global_gap_established": established,
|
||||
}
|
||||
for mode in RESTORATION_MODES:
|
||||
restoration_by_mode[mode][seed_key] = {}
|
||||
for metric in METRICS:
|
||||
established = global_metrics[metric]["established"]
|
||||
score = (
|
||||
math.log(
|
||||
mode_metrics[mode][metric]
|
||||
/ mode_metrics["uniform_all"][metric]
|
||||
)
|
||||
/ global_metrics[metric]["log_gap"]
|
||||
if established
|
||||
else None
|
||||
)
|
||||
restoration_by_mode[mode][seed_key][metric] = {
|
||||
"score": score,
|
||||
"metric_value": mode_metrics[mode][metric],
|
||||
"global_gap_established": established,
|
||||
}
|
||||
cells.append(
|
||||
{
|
||||
"seed": value["seed"],
|
||||
"path": str(path),
|
||||
"file_sha256": file_sha256(path),
|
||||
"canonical_sha256": value[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"mode_metrics": mode_metrics,
|
||||
"global": global_metrics,
|
||||
"interaction_residual": {
|
||||
metric: (
|
||||
1.0
|
||||
- sufficiency_by_mode["uniform_output_only"][
|
||||
seed_key
|
||||
][metric]["score"]
|
||||
- sufficiency_by_mode["uniform_depth_all"][
|
||||
seed_key
|
||||
][metric]["score"]
|
||||
)
|
||||
for metric in METRICS
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
global_gap_passed = all_true(
|
||||
[
|
||||
cell["global"][metric]["established"]
|
||||
for cell in cells
|
||||
for metric in METRICS
|
||||
]
|
||||
)
|
||||
sufficiency_gates = {
|
||||
"groups_6_7": mixed_status(
|
||||
sufficiency_by_mode["uniform_groups_6_7_only"],
|
||||
thresholds["groups_6_7_sufficiency_minimum"],
|
||||
),
|
||||
"group_6": mixed_status(
|
||||
sufficiency_by_mode["uniform_group_6_only"],
|
||||
thresholds["single_group_material_minimum"],
|
||||
),
|
||||
"group_7": mixed_status(
|
||||
sufficiency_by_mode["uniform_group_7_only"],
|
||||
thresholds["single_group_material_minimum"],
|
||||
),
|
||||
"output_half_gap": mixed_status(
|
||||
sufficiency_by_mode["uniform_output_only"],
|
||||
thresholds["output_half_gap_minimum"],
|
||||
),
|
||||
}
|
||||
restoration_gates = {
|
||||
"groups_6_7": mixed_status(
|
||||
restoration_by_mode["uniform_except_groups_6_7"],
|
||||
thresholds["groups_6_7_restoration_minimum"],
|
||||
),
|
||||
"group_6": mixed_status(
|
||||
restoration_by_mode["uniform_except_group_6"],
|
||||
thresholds["single_group_material_minimum"],
|
||||
),
|
||||
"group_7": mixed_status(
|
||||
restoration_by_mode["uniform_except_group_7"],
|
||||
thresholds["single_group_material_minimum"],
|
||||
),
|
||||
}
|
||||
|
||||
branch_gates = {}
|
||||
for group in (6, 7):
|
||||
group_passed = sufficiency_gates[f"group_{group}"]["passed"]
|
||||
candidates = {}
|
||||
for branch, sibling in (("attention", "mlp"), ("mlp", "attention")):
|
||||
branch_mode = f"uniform_group_{group}_{branch}_only"
|
||||
sibling_mode = f"uniform_group_{group}_{sibling}_only"
|
||||
checks = []
|
||||
margins = []
|
||||
for seed in expected_seeds:
|
||||
seed_key = str(seed)
|
||||
for metric in METRICS:
|
||||
left = sufficiency_by_mode[branch_mode][seed_key][
|
||||
metric
|
||||
]["score"]
|
||||
right = sufficiency_by_mode[sibling_mode][seed_key][
|
||||
metric
|
||||
]["score"]
|
||||
margin = (
|
||||
left - right
|
||||
if left is not None and right is not None
|
||||
else None
|
||||
)
|
||||
margins.append(margin)
|
||||
checks.append(
|
||||
left is not None
|
||||
and left >= thresholds["branch_material_minimum"]
|
||||
and margin is not None
|
||||
and margin
|
||||
>= thresholds["branch_dominance_margin"]
|
||||
)
|
||||
candidates[branch] = {
|
||||
"passed": group_passed and all_true(checks),
|
||||
"group_gate_passed": group_passed,
|
||||
"passed_cells": sum(checks),
|
||||
"required_cells": len(checks),
|
||||
"margins": margins,
|
||||
}
|
||||
dominant = [
|
||||
branch
|
||||
for branch, gate in candidates.items()
|
||||
if gate["passed"]
|
||||
]
|
||||
branch_gates[f"group_{group}"] = {
|
||||
"passed": len(dominant) == 1,
|
||||
"dominant_branch": dominant[0] if len(dominant) == 1 else None,
|
||||
"exploratory_sufficiency_only": True,
|
||||
"candidates": candidates,
|
||||
}
|
||||
|
||||
localization_passed = (
|
||||
global_gap_passed
|
||||
and sufficiency_gates["groups_6_7"]["passed"]
|
||||
and restoration_gates["groups_6_7"]["passed"]
|
||||
)
|
||||
localization_status = (
|
||||
"established_at_preregistered_bidirectional_50pct_threshold"
|
||||
if localization_passed
|
||||
else (
|
||||
"one_sided_evidence_localization_not_established"
|
||||
if (
|
||||
sufficiency_gates["groups_6_7"]["passed"]
|
||||
!= restoration_gates["groups_6_7"]["passed"]
|
||||
)
|
||||
else "not_established_at_preregistered_threshold"
|
||||
)
|
||||
)
|
||||
|
||||
mode_means = {}
|
||||
for mode in manifest["matrix_modes"]:
|
||||
mode_means[mode] = {
|
||||
metric: statistics.fmean(
|
||||
cell["mode_metrics"][mode][metric] for cell in cells
|
||||
)
|
||||
for metric in METRICS
|
||||
}
|
||||
sufficiency_means = {
|
||||
mode: {
|
||||
metric: statistics.fmean(
|
||||
sufficiency_by_mode[mode][str(seed)][metric]["score"]
|
||||
for seed in expected_seeds
|
||||
)
|
||||
for metric in METRICS
|
||||
}
|
||||
for mode in SUFFICIENCY_MODES
|
||||
}
|
||||
restoration_means = {
|
||||
mode: {
|
||||
metric: statistics.fmean(
|
||||
restoration_by_mode[mode][str(seed)][metric]["score"]
|
||||
for seed in expected_seeds
|
||||
)
|
||||
for metric in METRICS
|
||||
}
|
||||
for mode in RESTORATION_MODES
|
||||
}
|
||||
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"study_identity": manifest["study_identity"],
|
||||
"manifest": {
|
||||
"path": str(args.manifest),
|
||||
"file_sha256": file_sha256(args.manifest),
|
||||
},
|
||||
"formal_cells": cells,
|
||||
"replay": {
|
||||
"path": str(args.replay),
|
||||
"file_sha256": file_sha256(args.replay),
|
||||
"canonical_sha256": replay[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"formal_seed1_exact_excluding_run_kind_and_timing": replay_exact,
|
||||
},
|
||||
"scores": {
|
||||
"sufficiency": sufficiency_by_mode,
|
||||
"restoration": restoration_by_mode,
|
||||
},
|
||||
"means": {
|
||||
"mode_metrics": mode_means,
|
||||
"sufficiency": sufficiency_means,
|
||||
"restoration": restoration_means,
|
||||
},
|
||||
"gates": {
|
||||
"all_input_and_parent_gates_passed": True,
|
||||
"global_gap": {
|
||||
"passed": global_gap_passed,
|
||||
"required_cells": 6,
|
||||
"passed_cells": sum(
|
||||
cell["global"][metric]["established"]
|
||||
for cell in cells
|
||||
for metric in METRICS
|
||||
),
|
||||
},
|
||||
"sufficiency": sufficiency_gates,
|
||||
"restoration": restoration_gates,
|
||||
"localization": {
|
||||
"passed": localization_passed,
|
||||
"status": localization_status,
|
||||
"requires": (
|
||||
"groups 6+7 sufficiency and restoration both >=0.50 "
|
||||
"for 3/3 seeds and both metrics"
|
||||
),
|
||||
},
|
||||
"branch_dominance": branch_gates,
|
||||
},
|
||||
"limitations": [
|
||||
"same-forward diagnostic backward-rule sensitivity only",
|
||||
"reduced byte-level language model, not the Kimi K3 checkpoint",
|
||||
"effects are non-additive and are not contribution percentages",
|
||||
"group 7 includes layers 26-28 outside fixed spike set 21-25",
|
||||
"three-seed threshold gates are not population inference",
|
||||
],
|
||||
}
|
||||
result["canonical_sha256_without_self"] = canonical_sha256(result)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = args.output.with_suffix(args.output.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
+ "\n"
|
||||
)
|
||||
temporary.replace(args.output)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"formal_cells": len(cells),
|
||||
"replay_exact": replay_exact,
|
||||
"global_gap": global_gap_passed,
|
||||
"localization": localization_status,
|
||||
"canonical_sha256": result[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,540 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"protocol_id": "llm-atlas-k3-attnres-local-path-v1",
|
||||
"parent_protocol_id": "llm-atlas-k3-attnres-spike-path-v1",
|
||||
"study_identity": "targeted local-path follow-up informed by Round 05 and Round 06; not blind discovery",
|
||||
"architecture": "block",
|
||||
"depth": 32,
|
||||
"aggregation_groups": 8,
|
||||
"blocks_per_group": 4,
|
||||
"depth_mixers": 64,
|
||||
"output_mixers": 1,
|
||||
"formal_seeds": [
|
||||
2026073001,
|
||||
2026073002,
|
||||
2026073003
|
||||
],
|
||||
"replay": {
|
||||
"architecture": "block",
|
||||
"depth": 32,
|
||||
"seed": 2026073001,
|
||||
"environment_scope": "same host, GPU, Python, PyTorch, CUDA and CUBLAS_WORKSPACE_CONFIG"
|
||||
},
|
||||
"training": {
|
||||
"steps": 8000,
|
||||
"batch_size": 32,
|
||||
"context": 256,
|
||||
"target_bytes_per_cell": 65536000,
|
||||
"parent_diagnostic_steps": [
|
||||
0,
|
||||
100,
|
||||
500,
|
||||
2000,
|
||||
4000,
|
||||
8000
|
||||
],
|
||||
"local_matrix_steps": [
|
||||
0,
|
||||
8000
|
||||
],
|
||||
"matrix_used_during_training": false
|
||||
},
|
||||
"primary_object": {
|
||||
"position": "post_mlp_state",
|
||||
"reduction": "element_rms",
|
||||
"spike_layers_one_based": [
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25
|
||||
],
|
||||
"metrics": [
|
||||
"spike_contrast",
|
||||
"peak_normalized"
|
||||
]
|
||||
},
|
||||
"matrix_modes": [
|
||||
"detached_learned",
|
||||
"uniform_group_6_only",
|
||||
"uniform_group_7_only",
|
||||
"uniform_groups_6_7_only",
|
||||
"uniform_group_6_attention_only",
|
||||
"uniform_group_6_mlp_only",
|
||||
"uniform_group_7_attention_only",
|
||||
"uniform_group_7_mlp_only",
|
||||
"uniform_output_only",
|
||||
"uniform_depth_all",
|
||||
"uniform_all",
|
||||
"uniform_except_group_6",
|
||||
"uniform_except_group_7",
|
||||
"uniform_except_groups_6_7"
|
||||
],
|
||||
"selector": {
|
||||
"coefficient_choices": [
|
||||
"detached_learned",
|
||||
"uniform"
|
||||
],
|
||||
"full_autograd_learned_in_matrix": false,
|
||||
"depth_identity": "kind=depth,index=0..63; layer=floor(index/2)+1; attention iff index even",
|
||||
"output_identity": "kind=output,index=64; parallel-schema alias only; parent mixer_index remains null",
|
||||
"uniform_depth_indices": {
|
||||
"detached_learned": [],
|
||||
"uniform_group_6_only": [
|
||||
40,
|
||||
41,
|
||||
42,
|
||||
43,
|
||||
44,
|
||||
45,
|
||||
46,
|
||||
47
|
||||
],
|
||||
"uniform_group_7_only": [
|
||||
48,
|
||||
49,
|
||||
50,
|
||||
51,
|
||||
52,
|
||||
53,
|
||||
54,
|
||||
55
|
||||
],
|
||||
"uniform_groups_6_7_only": [
|
||||
40,
|
||||
41,
|
||||
42,
|
||||
43,
|
||||
44,
|
||||
45,
|
||||
46,
|
||||
47,
|
||||
48,
|
||||
49,
|
||||
50,
|
||||
51,
|
||||
52,
|
||||
53,
|
||||
54,
|
||||
55
|
||||
],
|
||||
"uniform_group_6_attention_only": [
|
||||
40,
|
||||
42,
|
||||
44,
|
||||
46
|
||||
],
|
||||
"uniform_group_6_mlp_only": [
|
||||
41,
|
||||
43,
|
||||
45,
|
||||
47
|
||||
],
|
||||
"uniform_group_7_attention_only": [
|
||||
48,
|
||||
50,
|
||||
52,
|
||||
54
|
||||
],
|
||||
"uniform_group_7_mlp_only": [
|
||||
49,
|
||||
51,
|
||||
53,
|
||||
55
|
||||
],
|
||||
"uniform_output_only": [],
|
||||
"uniform_depth_all": [
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38,
|
||||
39,
|
||||
40,
|
||||
41,
|
||||
42,
|
||||
43,
|
||||
44,
|
||||
45,
|
||||
46,
|
||||
47,
|
||||
48,
|
||||
49,
|
||||
50,
|
||||
51,
|
||||
52,
|
||||
53,
|
||||
54,
|
||||
55,
|
||||
56,
|
||||
57,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63
|
||||
],
|
||||
"uniform_all": [
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38,
|
||||
39,
|
||||
40,
|
||||
41,
|
||||
42,
|
||||
43,
|
||||
44,
|
||||
45,
|
||||
46,
|
||||
47,
|
||||
48,
|
||||
49,
|
||||
50,
|
||||
51,
|
||||
52,
|
||||
53,
|
||||
54,
|
||||
55,
|
||||
56,
|
||||
57,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63
|
||||
],
|
||||
"uniform_except_group_6": [
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38,
|
||||
39,
|
||||
48,
|
||||
49,
|
||||
50,
|
||||
51,
|
||||
52,
|
||||
53,
|
||||
54,
|
||||
55,
|
||||
56,
|
||||
57,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63
|
||||
],
|
||||
"uniform_except_group_7": [
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38,
|
||||
39,
|
||||
40,
|
||||
41,
|
||||
42,
|
||||
43,
|
||||
44,
|
||||
45,
|
||||
46,
|
||||
47,
|
||||
56,
|
||||
57,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63
|
||||
],
|
||||
"uniform_except_groups_6_7": [
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38,
|
||||
39,
|
||||
56,
|
||||
57,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63
|
||||
]
|
||||
},
|
||||
"uniform_output": {
|
||||
"detached_learned": false,
|
||||
"uniform_group_6_only": false,
|
||||
"uniform_group_7_only": false,
|
||||
"uniform_groups_6_7_only": false,
|
||||
"uniform_group_6_attention_only": false,
|
||||
"uniform_group_6_mlp_only": false,
|
||||
"uniform_group_7_attention_only": false,
|
||||
"uniform_group_7_mlp_only": false,
|
||||
"uniform_output_only": true,
|
||||
"uniform_depth_all": false,
|
||||
"uniform_all": true,
|
||||
"uniform_except_group_6": true,
|
||||
"uniform_except_group_7": true,
|
||||
"uniform_except_groups_6_7": true
|
||||
},
|
||||
"expected_uniform_counts": {
|
||||
"detached_learned": 0,
|
||||
"uniform_group_6_only": 8,
|
||||
"uniform_group_7_only": 8,
|
||||
"uniform_groups_6_7_only": 16,
|
||||
"uniform_group_6_attention_only": 4,
|
||||
"uniform_group_6_mlp_only": 4,
|
||||
"uniform_group_7_attention_only": 4,
|
||||
"uniform_group_7_mlp_only": 4,
|
||||
"uniform_output_only": 1,
|
||||
"uniform_depth_all": 64,
|
||||
"uniform_all": 65,
|
||||
"uniform_except_group_6": 57,
|
||||
"uniform_except_group_7": 57,
|
||||
"uniform_except_groups_6_7": 49
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"positive_denominator_epsilon": 1e-30,
|
||||
"step0_spectrum_tolerance": 1e-6,
|
||||
"loss_scale_tolerance": 1e-5,
|
||||
"global_relative_drop_minimum": 0.2,
|
||||
"groups_6_7_sufficiency_minimum": 0.5,
|
||||
"groups_6_7_restoration_minimum": 0.5,
|
||||
"single_group_material_minimum": 0.2,
|
||||
"branch_material_minimum": 0.2,
|
||||
"branch_dominance_margin": 0.15,
|
||||
"output_half_gap_minimum": 0.5,
|
||||
"formal_seed_gate": "3/3 independently for both metrics; means are display-only"
|
||||
},
|
||||
"formulas": {
|
||||
"global_log_gap": "G_X = ln(X_ref / X_uniform_all)",
|
||||
"global_relative_drop": "(X_ref - X_uniform_all) / X_ref",
|
||||
"sufficiency": "S_X(m) = ln(X_ref / X_m) / G_X",
|
||||
"restoration": "R_X(r) = ln(X_r / X_uniform_all) / G_X",
|
||||
"score_clipping": false
|
||||
},
|
||||
"parent_artifacts": {
|
||||
"manifest_path": "experiments/k3/attnres_spike/manifest.json",
|
||||
"manifest_sha256": "d5302a249249a07d362819134763d14e7d32307f22cff416c665ed9606142fef",
|
||||
"runner_path": "experiments/k3/attnres_spike/train.py",
|
||||
"runner_sha256": "77298081d3c491d2e88e4705995174b9879ef377f520eb5fe5ea107e7a1da084",
|
||||
"protocol_path": "research/K3_ATTNRES_SPIKE_PROTOCOL.md",
|
||||
"protocol_sha256": "6cb101b8760d9f1c81caeb2f16880b16152da103867224a06761a75a12984a16",
|
||||
"scoping_path": "research/K3_ATTNRES_SPIKE_SCOPING.md",
|
||||
"scoping_sha256": "590166bd62580bb8238293823cfcc39bc0a465fec4c697025343f3f1138abd27",
|
||||
"formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e",
|
||||
"validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338",
|
||||
"diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716"
|
||||
},
|
||||
"current_artifacts": {
|
||||
"protocol_path": "research/K3_ATTNRES_LOCAL_PATH_PROTOCOL.md",
|
||||
"protocol_sha256": "5ecc7ca92314ddb50aecf0cb50e115814c8983aa8bffb30e3634f7b3ce6dca1d",
|
||||
"scoping_path": "research/K3_ATTNRES_LOCAL_PATH_SCOPING.md",
|
||||
"scoping_sha256": "670ca4edf31a4be1f54937d9c7a760dba7a96e1e820c38c6b10405e22b078fc8",
|
||||
"grok_review_path": "research/K3_ATTNRES_LOCAL_PATH_GROK_REVIEW.md",
|
||||
"grok_review_sha256": "2da1b6bf1f455c4121a7a2c5cfe40e102327dabafc7e24dccc23ed0d00ac6d71",
|
||||
"grok_session": "019fb151-9627-76c1-b7d7-53012874f85c"
|
||||
},
|
||||
"round06_expected": {
|
||||
"2026073001": {
|
||||
"raw_file_sha256": "e39e93b7a7fce3c56f5f14f95cfdc04afdce53628affee1202fe62bd1bdb7f71",
|
||||
"canonical_sha256": "76b0ccfb55c38baef50c395788ac4b351cbe0d58de70064b50702acb5c93f515",
|
||||
"final_model_state": "3f0b97ece3a15571ba3d656f589f512ca0bb9e20083c9f58a42ccaee14892f59",
|
||||
"final_optimizer_state": "ed03e6fbd4a12d8b063dcb22e0437754285f54d585374cd52fbd534f05d24637"
|
||||
},
|
||||
"2026073002": {
|
||||
"raw_file_sha256": "1c6f6c731030ec0adb2a8e7a4d586e0c4005cc3319568a7ac83c08c2a4b8eaf8",
|
||||
"canonical_sha256": "5352c74eca853b375c0e85933dafd7c5916c39fc59052e742ca14ffd6d68bc78",
|
||||
"final_model_state": "bd2556388aeaa211b798c283c7cbd8ccd29edf166a2922fa13d172e8dfdc38d1",
|
||||
"final_optimizer_state": "0b101eab3bc7d8d654be2ea335c86fc25563ce19912d721844ee4e639c569e77"
|
||||
},
|
||||
"2026073003": {
|
||||
"raw_file_sha256": "115f8245577ece6dfaaa8ada68445c186e6523a7f3b26efcc3eb4c0c4ce82406",
|
||||
"canonical_sha256": "7e764c07e90b78c4cd0acc2e99600225f16428cbb25d5188a0a5a8fe797f8766",
|
||||
"final_model_state": "638568aede21890773b6932a19ec4e112f5ac0a4770ba3402fcd82980a9ecf76",
|
||||
"final_optimizer_state": "83947fd743ec8e3e31ca7788fd201981846f0f1c7e9e38c88afcef95cfc6ec4e"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Package frozen Round 07 outputs without recomputing any result gate."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-local-path-v1"
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--raw-dir", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--aggregate", type=Path, required=True)
|
||||
parser.add_argument("--reproduction-output", type=Path, required=True)
|
||||
parser.add_argument("--compact-output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
payload = json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
return hashlib.sha256(payload).hexdigest()
|
||||
|
||||
|
||||
def load_canonical(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text())
|
||||
expected = value["canonical_sha256_without_self"]
|
||||
payload = {
|
||||
key: item
|
||||
for key, item in value.items()
|
||||
if key != "canonical_sha256_without_self"
|
||||
}
|
||||
if canonical_sha256(payload) != expected:
|
||||
raise RuntimeError(f"canonical hash mismatch: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def write_canonical(path: Path, value: dict[str, Any]) -> None:
|
||||
value["canonical_sha256_without_self"] = canonical_sha256(value)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
+ "\n"
|
||||
)
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def replay_payload(value: dict[str, Any]) -> dict[str, Any]:
|
||||
cleaned = copy.deepcopy(value)
|
||||
for key in ("run_kind", "timing", "canonical_sha256_without_self"):
|
||||
cleaned.pop(key)
|
||||
return cleaned
|
||||
|
||||
|
||||
def artifact_hashes(repo_root: Path) -> dict[str, str]:
|
||||
paths = {
|
||||
"runner": "experiments/k3/attnres_local_path/train.py",
|
||||
"analyzer": "experiments/k3/attnres_local_path/analyze.py",
|
||||
"packager": "experiments/k3/attnres_local_path/package.py",
|
||||
"manifest": "experiments/k3/attnres_local_path/manifest.json",
|
||||
"protocol": "research/K3_ATTNRES_LOCAL_PATH_PROTOCOL.md",
|
||||
"scoping": "research/K3_ATTNRES_LOCAL_PATH_SCOPING.md",
|
||||
"preresult_grok_review": (
|
||||
"research/K3_ATTNRES_LOCAL_PATH_GROK_REVIEW.md"
|
||||
),
|
||||
}
|
||||
return {
|
||||
name: file_sha256(repo_root / path) for name, path in paths.items()
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
repo_root = Path(__file__).resolve().parents[3]
|
||||
manifest = json.loads(args.manifest.read_text())
|
||||
aggregate = load_canonical(args.aggregate)
|
||||
if (
|
||||
manifest["protocol_id"] != PROTOCOL_ID
|
||||
or aggregate["protocol_id"] != PROTOCOL_ID
|
||||
):
|
||||
raise RuntimeError("protocol mismatch")
|
||||
|
||||
formal = {}
|
||||
raw_files = {}
|
||||
for seed in SEEDS:
|
||||
name = f"formal-seed-{seed}.json"
|
||||
path = args.raw_dir / name
|
||||
run = load_canonical(path)
|
||||
if (
|
||||
run["run_kind"] != "formal"
|
||||
or run["seed"] != seed
|
||||
or not run["round06_equivalence"]["passed"]
|
||||
):
|
||||
raise RuntimeError(f"invalid formal run: {name}")
|
||||
formal[seed] = run
|
||||
raw_files[name] = {
|
||||
"file_sha256": file_sha256(path),
|
||||
"canonical_sha256": run["canonical_sha256_without_self"],
|
||||
}
|
||||
|
||||
replay_name = f"replay-seed-{SEEDS[0]}.json"
|
||||
replay_path = args.raw_dir / replay_name
|
||||
replay = load_canonical(replay_path)
|
||||
if replay["run_kind"] != "replay" or replay["seed"] != SEEDS[0]:
|
||||
raise RuntimeError("invalid replay")
|
||||
raw_files[replay_name] = {
|
||||
"file_sha256": file_sha256(replay_path),
|
||||
"canonical_sha256": replay["canonical_sha256_without_self"],
|
||||
}
|
||||
compare_payload = replay_payload(formal[SEEDS[0]])
|
||||
replay_exact = compare_payload == replay_payload(replay)
|
||||
if not replay_exact or not aggregate["replay"][
|
||||
"formal_seed1_exact_excluding_run_kind_and_timing"
|
||||
]:
|
||||
raise RuntimeError("replay exactness failed")
|
||||
replay_gate = {
|
||||
"passed": True,
|
||||
"excluded_fields": [
|
||||
"run_kind",
|
||||
"timing",
|
||||
"canonical_sha256_without_self",
|
||||
],
|
||||
"frozen_compare_sha256": canonical_sha256(compare_payload),
|
||||
}
|
||||
|
||||
reproduction = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"raw_files": raw_files,
|
||||
"replay_gate": replay_gate,
|
||||
"artifacts": artifact_hashes(repo_root),
|
||||
"aggregate": {
|
||||
"file_sha256": file_sha256(args.aggregate),
|
||||
"canonical_sha256": aggregate[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
},
|
||||
"post_result_grok_review": {
|
||||
"session": "019fb19d-94a3-7231-9a63-3a1ef33a9892",
|
||||
"role": "read-only adversarial implementation audit; not an evidence source",
|
||||
"blocking_errors": 0,
|
||||
"localization_status_confirmed": True,
|
||||
},
|
||||
}
|
||||
write_canonical(args.reproduction_output, reproduction)
|
||||
|
||||
final_spectra = []
|
||||
for seed in SEEDS:
|
||||
final = formal[seed]["diagnostics"][-1]["local_matrix"]
|
||||
final_spectra.append(
|
||||
{
|
||||
"seed": seed,
|
||||
"modes": {
|
||||
mode: {
|
||||
"normalized": final[mode]["positions"][
|
||||
"post_mlp_state"
|
||||
]["reductions"]["element_rms"]["statistics"][
|
||||
"normalized"
|
||||
],
|
||||
"spike_contrast": final[mode]["positions"][
|
||||
"post_mlp_state"
|
||||
]["reductions"]["element_rms"]["statistics"][
|
||||
"spike_contrast"
|
||||
],
|
||||
"peak_normalized": final[mode]["positions"][
|
||||
"post_mlp_state"
|
||||
]["reductions"]["element_rms"]["statistics"][
|
||||
"peak_normalized"
|
||||
],
|
||||
"peak_layer": final[mode]["positions"][
|
||||
"post_mlp_state"
|
||||
]["reductions"]["element_rms"]["statistics"][
|
||||
"peak_layer"
|
||||
],
|
||||
"uniform_count": final[mode]["selector"][
|
||||
"uniform_count"
|
||||
],
|
||||
}
|
||||
for mode in manifest["matrix_modes"]
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
compact = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"study": {
|
||||
"identity": manifest["study_identity"],
|
||||
"seeds": list(SEEDS),
|
||||
"steps": manifest["training"]["steps"],
|
||||
"formal_target_bytes": (
|
||||
len(SEEDS)
|
||||
* manifest["training"]["target_bytes_per_cell"]
|
||||
),
|
||||
"total_target_bytes_with_replay": (
|
||||
(len(SEEDS) + 1)
|
||||
* manifest["training"]["target_bytes_per_cell"]
|
||||
),
|
||||
"modes": manifest["matrix_modes"],
|
||||
"spike_layers": manifest["primary_object"][
|
||||
"spike_layers_one_based"
|
||||
],
|
||||
"metrics": manifest["primary_object"]["metrics"],
|
||||
},
|
||||
"thresholds": manifest["thresholds"],
|
||||
"formulas": manifest["formulas"],
|
||||
"formal_cells": aggregate["formal_cells"],
|
||||
"scores": aggregate["scores"],
|
||||
"means": aggregate["means"],
|
||||
"gates": aggregate["gates"],
|
||||
"replay": {
|
||||
**aggregate["replay"],
|
||||
"frozen_compare_sha256": replay_gate[
|
||||
"frozen_compare_sha256"
|
||||
],
|
||||
},
|
||||
"final_spectra": final_spectra,
|
||||
"limitations": aggregate["limitations"],
|
||||
"hashes": {
|
||||
"aggregate_file_sha256": file_sha256(args.aggregate),
|
||||
"aggregate_canonical_sha256": aggregate[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"reproduction_file_sha256": file_sha256(
|
||||
args.reproduction_output
|
||||
),
|
||||
"reproduction_canonical_sha256": reproduction[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"manifest_file_sha256": file_sha256(args.manifest),
|
||||
},
|
||||
}
|
||||
write_canonical(args.compact_output, compact)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"reproduction": str(args.reproduction_output),
|
||||
"compact": str(args.compact_output),
|
||||
"raw_files": len(raw_files),
|
||||
"replay_exact": replay_exact,
|
||||
"localization": aggregate["gates"]["localization"][
|
||||
"status"
|
||||
],
|
||||
"compact_canonical_sha256": compact[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"aggregate": {
|
||||
"canonical_sha256": "b86d119cd2f106e2cbee8a35760ed3244336a2fcfeb9178ea1e7dab13fc6f215",
|
||||
"file_sha256": "bb0ec9fce5b30d50ad5c50c4b95af7a892d614f205a2c20cfc2662125e10160e"
|
||||
},
|
||||
"artifacts": {
|
||||
"analyzer": "e0921562463e43d1ba6d47e4d23015107eb2df23579921088550b69afd47d02b",
|
||||
"manifest": "db01e92ef2cf0896212fcd529429bd94a344de0e1195db7f87b9a56dc3449139",
|
||||
"packager": "6a9ada0bc35b40475f45d7aca82877a93aeca667ed117c8e4417d924416c9196",
|
||||
"preresult_grok_review": "2da1b6bf1f455c4121a7a2c5cfe40e102327dabafc7e24dccc23ed0d00ac6d71",
|
||||
"protocol": "5ecc7ca92314ddb50aecf0cb50e115814c8983aa8bffb30e3634f7b3ce6dca1d",
|
||||
"runner": "b42879e242a2f2d54aa6a87a718aeac4cf4509eae42da2b14403656675a8b03d",
|
||||
"scoping": "670ca4edf31a4be1f54937d9c7a760dba7a96e1e820c38c6b10405e22b078fc8"
|
||||
},
|
||||
"canonical_sha256_without_self": "6f5d98fce6446fecc966dd2675f272f2c4f0c9a39a5741fabc4ffad6852ca7f4",
|
||||
"post_result_grok_review": {
|
||||
"blocking_errors": 0,
|
||||
"localization_status_confirmed": true,
|
||||
"role": "read-only adversarial implementation audit; not an evidence source",
|
||||
"session": "019fb19d-94a3-7231-9a63-3a1ef33a9892"
|
||||
},
|
||||
"protocol_id": "llm-atlas-k3-attnres-local-path-v1",
|
||||
"raw_files": {
|
||||
"formal-seed-2026073001.json": {
|
||||
"canonical_sha256": "f0a44f119836ed632c15880c3c2bb225173c0a05c50ea07abbe0e464ff407592",
|
||||
"file_sha256": "73d46ae443d3e5ae3fe839c1656cda758f5f41aaee5f220c971c3c39b8a8cc3f"
|
||||
},
|
||||
"formal-seed-2026073002.json": {
|
||||
"canonical_sha256": "b4629672b7d3b88a6be5525d2839e63e34fc9e4603ba1e7b98e957558da6da05",
|
||||
"file_sha256": "bca4674c746e35a035acde7d2094a9c3bd59a988b052cd3feb30eb66eb0ca60a"
|
||||
},
|
||||
"formal-seed-2026073003.json": {
|
||||
"canonical_sha256": "190b3deb06ae06caba287fce047b55cee613af6f1ebeb1661fcb53dd245abab0",
|
||||
"file_sha256": "712f349e7715fee71f8e4678dcde0619d01b8d6c3b5c34825c88b9af0bf2abbb"
|
||||
},
|
||||
"replay-seed-2026073001.json": {
|
||||
"canonical_sha256": "378df53ed9c89b2a4e0f3045b4c1e72754a7108d87fa9436e1db7aefa442e5eb",
|
||||
"file_sha256": "872aabd9285ac346b4016c23de10769ab83c4dc29e62d8bc8b3156af98228d4e"
|
||||
}
|
||||
},
|
||||
"replay_gate": {
|
||||
"excluded_fields": [
|
||||
"run_kind",
|
||||
"timing",
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"frozen_compare_sha256": "7dbd15ad03fbd357c5d91e159706d63b24703722f76c492ed1dc733535d6b9cf",
|
||||
"passed": true
|
||||
},
|
||||
"schema_version": 1
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,933 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Exact-replay Round 06 with preregistered local mixer-path interventions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-local-path-v1"
|
||||
PARENT_PROTOCOL_ID = "llm-atlas-k3-attnres-spike-path-v1"
|
||||
DATA_PROTOCOL_ID = "llm-atlas-k3-attnres-gradient-scale-v1"
|
||||
DEPTH = 32
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
FORMAL_STEPS = 8000
|
||||
PARENT_DIAGNOSTIC_STEPS = (0, 100, 500, 2000, 4000, 8000)
|
||||
MATRIX_STEPS = (0, 8000)
|
||||
MATRIX_MODES = (
|
||||
"detached_learned",
|
||||
"uniform_group_6_only",
|
||||
"uniform_group_7_only",
|
||||
"uniform_groups_6_7_only",
|
||||
"uniform_group_6_attention_only",
|
||||
"uniform_group_6_mlp_only",
|
||||
"uniform_group_7_attention_only",
|
||||
"uniform_group_7_mlp_only",
|
||||
"uniform_output_only",
|
||||
"uniform_depth_all",
|
||||
"uniform_all",
|
||||
"uniform_except_group_6",
|
||||
"uniform_except_group_7",
|
||||
"uniform_except_groups_6_7",
|
||||
)
|
||||
TRAIN_BATCH_SIZE = 32
|
||||
VALIDATION_WINDOWS = 64
|
||||
EVAL_BATCH_SIZE = 8
|
||||
TIMING_WARMUP = 20
|
||||
|
||||
|
||||
def load_parent() -> Any:
|
||||
path = Path(__file__).resolve().parents[1] / "attnres_spike" / "train.py"
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"k3_attnres_spike_parent", path
|
||||
)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"cannot import parent runner from {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
parent = load_parent()
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--run-kind", choices=("smoke", "formal", "replay"), required=True
|
||||
)
|
||||
parser.add_argument("--seed", type=int, required=True)
|
||||
parser.add_argument("--cache-dir", type=Path, required=True)
|
||||
parser.add_argument("--data-manifest", type=Path, required=True)
|
||||
parser.add_argument("--parent-manifest", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
if args.seed not in SEEDS:
|
||||
parser.error(f"seed must be one of {SEEDS}")
|
||||
if args.run_kind == "replay" and args.seed != SEEDS[0]:
|
||||
parser.error(f"replay seed must be {SEEDS[0]}")
|
||||
return args
|
||||
|
||||
|
||||
class LocalPathLanguageModel(parent.SpikeLanguageModel):
|
||||
"""Parent model with a parallel, audited per-mixer coefficient selector."""
|
||||
|
||||
def __init__(self, architecture: str, manifest: dict[str, Any]):
|
||||
super().__init__(architecture)
|
||||
selector = manifest["selector"]
|
||||
self.uniform_depth = {
|
||||
mode: frozenset(indices)
|
||||
for mode, indices in selector["uniform_depth_indices"].items()
|
||||
}
|
||||
self.uniform_output = selector["uniform_output"]
|
||||
self.expected_uniform_counts = selector["expected_uniform_counts"]
|
||||
self._active_selector_visits: list[dict[str, Any]] | None = None
|
||||
self.last_selector_visits: list[dict[str, Any]] | None = None
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_ids: torch.Tensor,
|
||||
capture: bool = False,
|
||||
mixer_backward_mode: str = "learned",
|
||||
) -> tuple[torch.Tensor, parent.SpikeTrace | None]:
|
||||
if not capture:
|
||||
if mixer_backward_mode != "learned":
|
||||
raise RuntimeError("training/evaluation cannot use an intervention")
|
||||
return super().forward(
|
||||
input_ids, capture=False, mixer_backward_mode="learned"
|
||||
)
|
||||
if mixer_backward_mode == "learned":
|
||||
self.last_selector_visits = None
|
||||
return super().forward(
|
||||
input_ids, capture=True, mixer_backward_mode="learned"
|
||||
)
|
||||
if mixer_backward_mode not in MATRIX_MODES:
|
||||
raise ValueError(f"unknown local matrix mode: {mixer_backward_mode}")
|
||||
self._active_selector_visits = []
|
||||
logits, trace = self._diagnostic_forward(
|
||||
input_ids, mixer_backward_mode
|
||||
)
|
||||
self.last_selector_visits = self._active_selector_visits
|
||||
self._active_selector_visits = None
|
||||
return logits, trace
|
||||
|
||||
def _mix(
|
||||
self,
|
||||
mixer: nn.Module,
|
||||
sources: list[torch.Tensor],
|
||||
labels: list[str],
|
||||
*,
|
||||
mode: str,
|
||||
mixer_index: int | None,
|
||||
layer: int | None,
|
||||
branch: str,
|
||||
group: int | None,
|
||||
offset: int | None,
|
||||
) -> tuple[torch.Tensor, dict[str, Any]]:
|
||||
if mode == "learned":
|
||||
return super()._mix(
|
||||
mixer,
|
||||
sources,
|
||||
labels,
|
||||
mode=mode,
|
||||
mixer_index=mixer_index,
|
||||
layer=layer,
|
||||
branch=branch,
|
||||
group=group,
|
||||
offset=offset,
|
||||
)
|
||||
if self._active_selector_visits is None:
|
||||
raise RuntimeError("local selector visit log is not active")
|
||||
|
||||
if mixer_index is None:
|
||||
if (
|
||||
layer is not None
|
||||
or group is not None
|
||||
or branch != "output"
|
||||
or offset is not None
|
||||
):
|
||||
raise RuntimeError("invalid output mixer identity")
|
||||
identity = {
|
||||
"kind": "output",
|
||||
"index": 64,
|
||||
"layer": None,
|
||||
"group": None,
|
||||
"branch": "output",
|
||||
"offset": None,
|
||||
}
|
||||
use_uniform = bool(self.uniform_output[mode])
|
||||
else:
|
||||
expected_layer = mixer_index // 2 + 1
|
||||
expected_branch = "attention" if mixer_index % 2 == 0 else "mlp"
|
||||
expected_group = (expected_layer - 1) // 4 + 1
|
||||
expected_offset = (expected_layer - 1) % 4 + 1
|
||||
if (
|
||||
not 0 <= mixer_index < 64
|
||||
or layer != expected_layer
|
||||
or branch != expected_branch
|
||||
or group != expected_group
|
||||
or offset != expected_offset
|
||||
):
|
||||
raise RuntimeError("invalid depth mixer identity")
|
||||
identity = {
|
||||
"kind": "depth",
|
||||
"index": mixer_index,
|
||||
"layer": layer,
|
||||
"group": group,
|
||||
"branch": branch,
|
||||
"offset": offset,
|
||||
}
|
||||
use_uniform = mixer_index in self.uniform_depth[mode]
|
||||
|
||||
parent_output, _ = mixer(sources, False)
|
||||
weights = parent.recompute_weights(mixer, sources)
|
||||
summary = parent.weight_summary(
|
||||
weights,
|
||||
labels,
|
||||
mixer_index=mixer_index,
|
||||
layer=layer,
|
||||
branch=branch,
|
||||
group=group,
|
||||
offset=offset,
|
||||
)
|
||||
backward_weights = (
|
||||
torch.full_like(weights, 1.0 / len(sources))
|
||||
if use_uniform
|
||||
else weights
|
||||
)
|
||||
values = torch.stack(sources, dim=0)
|
||||
routed = parent.RoutedSourceBackward.apply(
|
||||
values, parent_output, backward_weights
|
||||
)
|
||||
self._active_selector_visits.append(
|
||||
{
|
||||
**identity,
|
||||
"sources": len(sources),
|
||||
"use_uniform": use_uniform,
|
||||
"coefficient": (
|
||||
"uniform" if use_uniform else "detached_learned"
|
||||
),
|
||||
}
|
||||
)
|
||||
return routed, summary
|
||||
|
||||
|
||||
def expected_visit_identities() -> list[dict[str, Any]]:
|
||||
result = []
|
||||
for index in range(64):
|
||||
layer = index // 2 + 1
|
||||
result.append(
|
||||
{
|
||||
"kind": "depth",
|
||||
"index": index,
|
||||
"layer": layer,
|
||||
"group": (layer - 1) // 4 + 1,
|
||||
"branch": "attention" if index % 2 == 0 else "mlp",
|
||||
"offset": (layer - 1) % 4 + 1,
|
||||
}
|
||||
)
|
||||
result.append(
|
||||
{
|
||||
"kind": "output",
|
||||
"index": 64,
|
||||
"layer": None,
|
||||
"group": None,
|
||||
"branch": "output",
|
||||
"offset": None,
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def validate_selector_visits(
|
||||
mode: str,
|
||||
visits: list[dict[str, Any]] | None,
|
||||
manifest: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
if visits is None or len(visits) != 65:
|
||||
raise RuntimeError("selector visit count mismatch")
|
||||
identity_keys = ("kind", "index", "layer", "group", "branch", "offset")
|
||||
actual_identities = [
|
||||
{key: visit[key] for key in identity_keys} for visit in visits
|
||||
]
|
||||
expected_identities = expected_visit_identities()
|
||||
if actual_identities != expected_identities:
|
||||
raise RuntimeError("selector identity order mismatch")
|
||||
if len({(item["kind"], item["index"]) for item in visits}) != 65:
|
||||
raise RuntimeError("selector identities are not unique")
|
||||
|
||||
expected_depth = set(
|
||||
manifest["selector"]["uniform_depth_indices"][mode]
|
||||
)
|
||||
expected_output = manifest["selector"]["uniform_output"][mode]
|
||||
actual_depth = {
|
||||
item["index"]
|
||||
for item in visits
|
||||
if item["kind"] == "depth" and item["use_uniform"]
|
||||
}
|
||||
actual_output = visits[-1]["use_uniform"]
|
||||
if actual_depth != expected_depth or actual_output != expected_output:
|
||||
raise RuntimeError("selector exact-set mismatch")
|
||||
uniform_count = sum(int(item["use_uniform"]) for item in visits)
|
||||
expected_count = manifest["selector"]["expected_uniform_counts"][mode]
|
||||
if uniform_count != expected_count:
|
||||
raise RuntimeError("selector uniform census mismatch")
|
||||
return {
|
||||
"passed": True,
|
||||
"visit_count": len(visits),
|
||||
"identities_unique": True,
|
||||
"identity_order_sha256": parent.canonical_sha256(
|
||||
actual_identities
|
||||
),
|
||||
"uniform_indices": [
|
||||
item["index"] for item in visits if item["use_uniform"]
|
||||
],
|
||||
"uniform_count": uniform_count,
|
||||
"expected_uniform_count": expected_count,
|
||||
"visits_sha256": parent.canonical_sha256(visits),
|
||||
"visits": visits,
|
||||
}
|
||||
|
||||
|
||||
def run_local_diagnostic(
|
||||
model: LocalPathLanguageModel,
|
||||
corpus: Any,
|
||||
optimizer: torch.optim.Optimizer,
|
||||
manifest: dict[str, Any],
|
||||
*,
|
||||
mode: str,
|
||||
loss_scale: float = 1.0,
|
||||
) -> dict[str, Any]:
|
||||
result = parent.run_diagnostic(
|
||||
model, corpus, optimizer, mode=mode, loss_scale=loss_scale
|
||||
)
|
||||
result["selector"] = validate_selector_visits(
|
||||
mode, model.last_selector_visits, manifest
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def forward_identity_gate(matrix: dict[str, Any]) -> dict[str, Any]:
|
||||
reference = matrix["detached_learned"]["forward"]
|
||||
comparisons = {}
|
||||
for mode in MATRIX_MODES[1:]:
|
||||
other = matrix[mode]["forward"]
|
||||
comparisons[mode] = {
|
||||
"logits_exact": (
|
||||
other["logits_sha256"] == reference["logits_sha256"]
|
||||
),
|
||||
"loss_exact": other["loss_nats"] == reference["loss_nats"],
|
||||
"activations_exact": (
|
||||
other["activation_sha256"]
|
||||
== reference["activation_sha256"]
|
||||
),
|
||||
"mixer_summaries_exact": (
|
||||
other["mixer_summary_sha256"]
|
||||
== reference["mixer_summary_sha256"]
|
||||
),
|
||||
}
|
||||
if not all(all(checks.values()) for checks in comparisons.values()):
|
||||
raise RuntimeError("local matrix forward identity failed")
|
||||
return {"passed": True, "comparisons": comparisons}
|
||||
|
||||
|
||||
def spectrum_agreement(
|
||||
left: dict[str, Any], right: dict[str, Any]
|
||||
) -> dict[str, Any]:
|
||||
checks = {}
|
||||
passed = True
|
||||
for position in parent.POSITIONS:
|
||||
left_metric = left["positions"][position]["reductions"][
|
||||
"element_rms"
|
||||
]
|
||||
right_metric = right["positions"][position]["reductions"][
|
||||
"element_rms"
|
||||
]
|
||||
raw_errors = [
|
||||
abs(a - b) / a
|
||||
for a, b in zip(
|
||||
left_metric["values"], right_metric["values"]
|
||||
)
|
||||
]
|
||||
normalized_errors = [
|
||||
abs(a - b)
|
||||
for a, b in zip(
|
||||
left_metric["statistics"]["normalized"],
|
||||
right_metric["statistics"]["normalized"],
|
||||
)
|
||||
]
|
||||
item_passed = (
|
||||
all(
|
||||
math.isfinite(value) and value > 0
|
||||
for value in left_metric["values"]
|
||||
)
|
||||
and max(raw_errors) <= parent.SPECTRUM_TOLERANCE
|
||||
and max(normalized_errors) <= parent.SPECTRUM_TOLERANCE
|
||||
)
|
||||
passed = passed and item_passed
|
||||
checks[position] = {
|
||||
"passed": item_passed,
|
||||
"max_raw_relative_error": max(raw_errors),
|
||||
"max_normalized_absolute_error": max(normalized_errors),
|
||||
}
|
||||
return {"passed": passed, "checks": checks}
|
||||
|
||||
|
||||
def initialization_negative_control(
|
||||
parent_learned: dict[str, Any], matrix: dict[str, Any]
|
||||
) -> dict[str, Any]:
|
||||
reference = matrix["detached_learned"]
|
||||
comparisons = {
|
||||
"parent_learned_vs_detached": spectrum_agreement(
|
||||
parent_learned, reference
|
||||
)
|
||||
}
|
||||
for mode in MATRIX_MODES[1:]:
|
||||
comparisons[mode] = spectrum_agreement(reference, matrix[mode])
|
||||
passed = all(item["passed"] for item in comparisons.values())
|
||||
if not passed:
|
||||
raise RuntimeError("initialization negative control failed")
|
||||
return {"passed": True, "comparisons": comparisons}
|
||||
|
||||
|
||||
def without_selector(result: dict[str, Any], rename: str | None = None) -> dict[str, Any]:
|
||||
cleaned = {key: value for key, value in result.items() if key != "selector"}
|
||||
if rename is not None:
|
||||
cleaned["mode"] = rename
|
||||
return cleaned
|
||||
|
||||
|
||||
def endpoint_exactness(
|
||||
matrix: dict[str, Any], parent_diagnostic: dict[str, Any]
|
||||
) -> dict[str, Any]:
|
||||
reference_exact = (
|
||||
without_selector(matrix["detached_learned"])
|
||||
== parent_diagnostic["modes"]["detached_learned"]
|
||||
)
|
||||
uniform_exact = (
|
||||
without_selector(
|
||||
matrix["uniform_all"], rename="uniform_value_backward"
|
||||
)
|
||||
== parent_diagnostic["modes"]["uniform_value_backward"]
|
||||
)
|
||||
checks = {
|
||||
"detached_learned_round06_exact": reference_exact,
|
||||
"uniform_all_round06_exact": uniform_exact,
|
||||
}
|
||||
if not all(checks.values()):
|
||||
raise RuntimeError(f"Round 06 endpoint exactness failed: {checks}")
|
||||
return {"passed": True, "checks": checks}
|
||||
|
||||
|
||||
def run_diagnostic_bundle(
|
||||
model: LocalPathLanguageModel,
|
||||
corpus: Any,
|
||||
optimizer: torch.optim.Optimizer,
|
||||
manifest: dict[str, Any],
|
||||
parent_diagnostic: dict[str, Any],
|
||||
step: int,
|
||||
) -> dict[str, Any]:
|
||||
parent_learned = parent.run_diagnostic(
|
||||
model, corpus, optimizer, mode="learned"
|
||||
)
|
||||
if parent_learned != parent_diagnostic["modes"]["learned"]:
|
||||
raise RuntimeError("parent learned diagnostic is not Round 06 exact")
|
||||
result: dict[str, Any] = {
|
||||
"step": step,
|
||||
"parent_learned": parent_learned,
|
||||
"parent_learned_round06_exact": True,
|
||||
"local_matrix": None,
|
||||
}
|
||||
if step not in MATRIX_STEPS:
|
||||
return result
|
||||
|
||||
matrix = {
|
||||
mode: run_local_diagnostic(
|
||||
model, corpus, optimizer, manifest, mode=mode
|
||||
)
|
||||
for mode in MATRIX_MODES
|
||||
}
|
||||
result["local_matrix"] = matrix
|
||||
result["forward_identity_gate"] = forward_identity_gate(matrix)
|
||||
result["endpoint_exactness"] = endpoint_exactness(
|
||||
matrix, parent_diagnostic
|
||||
)
|
||||
if step == 0:
|
||||
result["initialization_negative_control"] = (
|
||||
initialization_negative_control(parent_learned, matrix)
|
||||
)
|
||||
doubled = run_local_diagnostic(
|
||||
model,
|
||||
corpus,
|
||||
optimizer,
|
||||
manifest,
|
||||
mode="detached_learned",
|
||||
loss_scale=2.0,
|
||||
)
|
||||
result["loss_scale_gate"] = parent.loss_scale_gate(
|
||||
matrix["detached_learned"], doubled
|
||||
)
|
||||
model.zero_grad(set_to_none=True)
|
||||
return result
|
||||
|
||||
|
||||
def load_and_verify_inputs(
|
||||
args: argparse.Namespace,
|
||||
) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any], Path]:
|
||||
manifest = json.loads(args.manifest.read_text())
|
||||
parent_manifest = json.loads(args.parent_manifest.read_text())
|
||||
data_manifest = json.loads(args.data_manifest.read_text())
|
||||
repo_root = Path(__file__).resolve().parents[3]
|
||||
|
||||
if manifest["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError("Round 07 manifest protocol mismatch")
|
||||
if parent_manifest["protocol_id"] != PARENT_PROTOCOL_ID:
|
||||
raise RuntimeError("Round 06 parent manifest protocol mismatch")
|
||||
if data_manifest["protocol_id"] != DATA_PROTOCOL_ID:
|
||||
raise RuntimeError("data manifest protocol mismatch")
|
||||
if manifest["formal_seeds"] != list(SEEDS):
|
||||
raise RuntimeError("formal seed mismatch")
|
||||
if manifest["matrix_modes"] != list(MATRIX_MODES):
|
||||
raise RuntimeError("local matrix mode mismatch")
|
||||
if (
|
||||
manifest["training"]["parent_diagnostic_steps"]
|
||||
!= list(PARENT_DIAGNOSTIC_STEPS)
|
||||
or manifest["training"]["local_matrix_steps"] != list(MATRIX_STEPS)
|
||||
or manifest["training"]["steps"] != FORMAL_STEPS
|
||||
):
|
||||
raise RuntimeError("diagnostic/training schedule mismatch")
|
||||
|
||||
parent_artifacts = manifest["parent_artifacts"]
|
||||
if parent.file_sha256(args.parent_manifest) != parent_artifacts[
|
||||
"manifest_sha256"
|
||||
]:
|
||||
raise RuntimeError("Round 06 manifest physical hash mismatch")
|
||||
if parent.file_sha256(Path(parent.__file__)) != parent_artifacts[
|
||||
"runner_sha256"
|
||||
]:
|
||||
raise RuntimeError("Round 06 runner physical hash mismatch")
|
||||
for name in ("protocol", "scoping"):
|
||||
path = repo_root / parent_artifacts[f"{name}_path"]
|
||||
if parent.file_sha256(path) != parent_artifacts[f"{name}_sha256"]:
|
||||
raise RuntimeError(f"Round 06 {name} physical hash mismatch")
|
||||
for name in ("protocol", "scoping", "grok_review"):
|
||||
path = repo_root / manifest["current_artifacts"][f"{name}_path"]
|
||||
if parent.file_sha256(path) != manifest["current_artifacts"][
|
||||
f"{name}_sha256"
|
||||
]:
|
||||
raise RuntimeError(f"Round 07 {name} physical hash mismatch")
|
||||
for key in (
|
||||
"formal_schedule_sha256",
|
||||
"validation_tensor_sha256",
|
||||
"diagnostic_tensor_sha256",
|
||||
):
|
||||
if (
|
||||
data_manifest["windows"][key]
|
||||
!= parent_artifacts[key]
|
||||
or parent_manifest["parent_artifacts"][key]
|
||||
!= parent_artifacts[key]
|
||||
):
|
||||
raise RuntimeError(f"frozen data hash mismatch: {key}")
|
||||
|
||||
parent_raw_path = (
|
||||
repo_root
|
||||
/ "experiments"
|
||||
/ "k3"
|
||||
/ "attnres_spike"
|
||||
/ "results"
|
||||
/ "raw"
|
||||
/ f"formal-seed-{args.seed}.json"
|
||||
)
|
||||
expected = manifest["round06_expected"][str(args.seed)]
|
||||
if parent.file_sha256(parent_raw_path) != expected["raw_file_sha256"]:
|
||||
raise RuntimeError("Round 06 raw physical hash mismatch")
|
||||
parent_raw = json.loads(parent_raw_path.read_text())
|
||||
if (
|
||||
parent_raw["canonical_sha256_without_self"]
|
||||
!= expected["canonical_sha256"]
|
||||
or parent_raw["hashes"]["final_model_state"]
|
||||
!= expected["final_model_state"]
|
||||
or parent_raw["hashes"]["final_optimizer_state"]
|
||||
!= expected["final_optimizer_state"]
|
||||
):
|
||||
raise RuntimeError("Round 06 raw expected-state mismatch")
|
||||
return manifest, data_manifest, parent_raw, repo_root
|
||||
|
||||
|
||||
def frozen_training_compare(
|
||||
result: dict[str, Any], parent_raw: dict[str, Any]
|
||||
) -> dict[str, Any]:
|
||||
checks = {
|
||||
"final_model_state": (
|
||||
result["hashes"]["final_model_state"]
|
||||
== parent_raw["hashes"]["final_model_state"]
|
||||
),
|
||||
"final_optimizer_state": (
|
||||
result["hashes"]["final_optimizer_state"]
|
||||
== parent_raw["hashes"]["final_optimizer_state"]
|
||||
),
|
||||
"evaluations": result["evaluations"] == parent_raw["evaluations"],
|
||||
"training_history": (
|
||||
result["training_history"] == parent_raw["training_history"]
|
||||
),
|
||||
}
|
||||
parent_diagnostics_exact = []
|
||||
endpoint_exact = []
|
||||
for new, old in zip(result["diagnostics"], parent_raw["diagnostics"]):
|
||||
parent_diagnostics_exact.append(
|
||||
new["step"] == old["step"]
|
||||
and new["parent_learned"] == old["modes"]["learned"]
|
||||
)
|
||||
if new["step"] in MATRIX_STEPS:
|
||||
endpoint_exact.append(new["endpoint_exactness"]["passed"])
|
||||
checks["parent_learned_diagnostics"] = all(parent_diagnostics_exact)
|
||||
checks["round06_endpoints"] = len(endpoint_exact) == 2 and all(
|
||||
endpoint_exact
|
||||
)
|
||||
if not all(checks.values()):
|
||||
raise RuntimeError(f"Round 06 training equivalence failed: {checks}")
|
||||
return {"passed": True, "checks": checks}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required")
|
||||
if os.environ.get("CUBLAS_WORKSPACE_CONFIG") != ":4096:8":
|
||||
raise RuntimeError("CUBLAS_WORKSPACE_CONFIG must be :4096:8")
|
||||
manifest, data_manifest, parent_raw, repo_root = load_and_verify_inputs(args)
|
||||
parent.parent.configure_round04_globals(DEPTH)
|
||||
parent.parent.configure_determinism(args.seed)
|
||||
corpus = parent.parent.round04.ByteCorpus(
|
||||
args.cache_dir, data_manifest, torch.device("cuda")
|
||||
)
|
||||
model = LocalPathLanguageModel("block", manifest).to(
|
||||
torch.device("cuda")
|
||||
)
|
||||
|
||||
initial_public_hash = parent.parent.named_state_hash(
|
||||
model, include_mixers=False
|
||||
)
|
||||
initial_mixer_hash = parent.parent.named_state_hash(
|
||||
model, include_mixers=True
|
||||
)
|
||||
public_structure_hash, public_tensors, public_elements = (
|
||||
parent.parent.state_structure_hash(model, include_mixers=False)
|
||||
)
|
||||
input_gate_hashes = parent.parent.model_input_gate_hashes(
|
||||
corpus, data_manifest, args.seed, TRAIN_BATCH_SIZE
|
||||
)
|
||||
|
||||
decay_parameters: list[nn.Parameter] = []
|
||||
no_decay_parameters: list[nn.Parameter] = []
|
||||
for parameter in model.parameters():
|
||||
target = decay_parameters if parameter.ndim >= 2 else no_decay_parameters
|
||||
target.append(parameter)
|
||||
optimizer = torch.optim.AdamW(
|
||||
[
|
||||
{
|
||||
"params": decay_parameters,
|
||||
"weight_decay": parent.parent.WEIGHT_DECAY,
|
||||
},
|
||||
{"params": no_decay_parameters, "weight_decay": 0.0},
|
||||
],
|
||||
lr=parent.parent.PEAK_LR,
|
||||
betas=parent.parent.BETAS,
|
||||
eps=parent.parent.ADAM_EPS,
|
||||
)
|
||||
|
||||
parent_by_step = {
|
||||
item["step"]: item for item in parent_raw["diagnostics"]
|
||||
}
|
||||
evaluations = [
|
||||
{
|
||||
"step": 0,
|
||||
**parent.parent.evaluate(
|
||||
model, corpus, VALIDATION_WINDOWS, EVAL_BATCH_SIZE
|
||||
),
|
||||
}
|
||||
]
|
||||
diagnostics = [
|
||||
run_diagnostic_bundle(
|
||||
model,
|
||||
corpus,
|
||||
optimizer,
|
||||
manifest,
|
||||
parent_by_step[0],
|
||||
0,
|
||||
)
|
||||
]
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "local_matrix",
|
||||
"step": 0,
|
||||
"seed": args.seed,
|
||||
"modes": len(MATRIX_MODES),
|
||||
"endpoint_exact": diagnostics[0][
|
||||
"endpoint_exactness"
|
||||
]["passed"],
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
steps = 0 if args.run_kind == "smoke" else FORMAL_STEPS
|
||||
training_history: list[dict[str, float | int]] = []
|
||||
step_times: list[float] = []
|
||||
if steps:
|
||||
model.train()
|
||||
for step in range(1, steps + 1):
|
||||
lr = parent.parent.learning_rate(step, steps)
|
||||
for group in optimizer.param_groups:
|
||||
group["lr"] = lr
|
||||
inputs, targets = corpus.training_batch(
|
||||
args.seed, step, TRAIN_BATCH_SIZE
|
||||
)
|
||||
optimizer.zero_grad(set_to_none=True)
|
||||
torch.cuda.synchronize()
|
||||
started = time.perf_counter()
|
||||
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
||||
logits, trace = model(inputs)
|
||||
if trace is not None:
|
||||
raise RuntimeError(
|
||||
"training unexpectedly captured a trace"
|
||||
)
|
||||
loss = parent.parent.cross_entropy(logits, targets)
|
||||
if not torch.isfinite(loss):
|
||||
raise RuntimeError(f"non-finite loss at step {step}")
|
||||
loss.backward()
|
||||
unclipped_norm = torch.nn.utils.clip_grad_norm_(
|
||||
model.parameters(), parent.parent.GRAD_CLIP
|
||||
)
|
||||
optimizer.step()
|
||||
torch.cuda.synchronize()
|
||||
elapsed_ms = (time.perf_counter() - started) * 1000
|
||||
if step == TIMING_WARMUP:
|
||||
torch.cuda.reset_peak_memory_stats()
|
||||
elif step > TIMING_WARMUP:
|
||||
step_times.append(elapsed_ms)
|
||||
if step == 1 or step % 10 == 0 or step == steps:
|
||||
training_history.append(
|
||||
{
|
||||
"step": step,
|
||||
"loss_nats": loss.detach().cpu().item(),
|
||||
"bits_per_byte": (
|
||||
loss.detach().cpu().item() / math.log(2)
|
||||
),
|
||||
"learning_rate": lr,
|
||||
"unclipped_grad_norm": float(
|
||||
unclipped_norm.detach().cpu()
|
||||
),
|
||||
}
|
||||
)
|
||||
if step in PARENT_DIAGNOSTIC_STEPS:
|
||||
evaluations.append(
|
||||
{
|
||||
"step": step,
|
||||
**parent.parent.evaluate(
|
||||
model,
|
||||
corpus,
|
||||
VALIDATION_WINDOWS,
|
||||
EVAL_BATCH_SIZE,
|
||||
),
|
||||
}
|
||||
)
|
||||
diagnostic = run_diagnostic_bundle(
|
||||
model,
|
||||
corpus,
|
||||
optimizer,
|
||||
manifest,
|
||||
parent_by_step[step],
|
||||
step,
|
||||
)
|
||||
diagnostics.append(diagnostic)
|
||||
event = {
|
||||
"event": "diagnostic",
|
||||
"step": step,
|
||||
"seed": args.seed,
|
||||
"validation_bpc": evaluations[-1]["bits_per_byte"],
|
||||
"parent_exact": diagnostic[
|
||||
"parent_learned_round06_exact"
|
||||
],
|
||||
}
|
||||
if diagnostic["local_matrix"] is not None:
|
||||
event["modes"] = len(MATRIX_MODES)
|
||||
event["endpoint_exact"] = diagnostic[
|
||||
"endpoint_exactness"
|
||||
]["passed"]
|
||||
print(json.dumps(event, sort_keys=True), flush=True)
|
||||
model.train()
|
||||
|
||||
timing = {
|
||||
"warmup_steps_excluded": TIMING_WARMUP,
|
||||
"measured_steps": len(step_times),
|
||||
"mean_ms": (
|
||||
statistics.fmean(step_times) if step_times else None
|
||||
),
|
||||
"median_ms": (
|
||||
statistics.median(step_times) if step_times else None
|
||||
),
|
||||
"p95_ms": (
|
||||
parent.type7_quantile(torch.tensor(sorted(step_times)), 0.95)
|
||||
if step_times
|
||||
else None
|
||||
),
|
||||
"peak_allocated_bytes": torch.cuda.max_memory_allocated(),
|
||||
"peak_reserved_bytes": torch.cuda.max_memory_reserved(),
|
||||
}
|
||||
result = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"parent_protocol_id": PARENT_PROTOCOL_ID,
|
||||
"run_kind": args.run_kind,
|
||||
"architecture": "block",
|
||||
"depth": DEPTH,
|
||||
"seed": args.seed,
|
||||
"steps": steps,
|
||||
"batch_size": TRAIN_BATCH_SIZE,
|
||||
"target_bytes_seen": (
|
||||
steps * TRAIN_BATCH_SIZE * parent.CONTEXT
|
||||
),
|
||||
"manifest": {
|
||||
"path": str(args.manifest),
|
||||
"file_sha256": parent.file_sha256(args.manifest),
|
||||
"parent_path": str(args.parent_manifest),
|
||||
"parent_file_sha256": parent.file_sha256(
|
||||
args.parent_manifest
|
||||
),
|
||||
"data_path": str(args.data_manifest),
|
||||
"data_file_sha256": parent.file_sha256(args.data_manifest),
|
||||
"formal_schedule_sha256": data_manifest["windows"][
|
||||
"formal_schedule_sha256"
|
||||
],
|
||||
"validation_tensor_sha256": data_manifest["windows"][
|
||||
"validation_tensor_sha256"
|
||||
],
|
||||
"diagnostic_tensor_sha256": data_manifest["windows"][
|
||||
"diagnostic_tensor_sha256"
|
||||
],
|
||||
"input_gate_tensor_hashes": input_gate_hashes,
|
||||
"selector_contract_sha256": parent.canonical_sha256(
|
||||
manifest["selector"]
|
||||
),
|
||||
},
|
||||
"model": {
|
||||
"layers": DEPTH,
|
||||
"aggregation_groups": 8,
|
||||
"blocks_per_group": 4,
|
||||
"d_model": parent.parent.round04.D_MODEL,
|
||||
"heads": parent.parent.round04.HEADS,
|
||||
"d_ff": parent.parent.round04.D_FF,
|
||||
"parameters": parent.parent.parameter_inventory(model),
|
||||
},
|
||||
"hashes": {
|
||||
"initial_public_parameter_structure": public_structure_hash,
|
||||
"initial_public_parameter_tensors": public_tensors,
|
||||
"initial_public_parameter_elements": public_elements,
|
||||
"initial_public_parameters": initial_public_hash,
|
||||
"initial_mixer_parameters": initial_mixer_hash,
|
||||
"final_public_parameters": parent.parent.named_state_hash(
|
||||
model, include_mixers=False
|
||||
),
|
||||
"final_mixer_parameters": parent.parent.named_state_hash(
|
||||
model, include_mixers=True
|
||||
),
|
||||
"final_model_state": parent.parent.named_state_hash(
|
||||
model, include_mixers=None
|
||||
),
|
||||
"final_optimizer_state": parent.parent.recursive_state_hash(
|
||||
optimizer.state_dict()
|
||||
),
|
||||
},
|
||||
"evaluations": evaluations,
|
||||
"diagnostics": diagnostics,
|
||||
"training_history": training_history,
|
||||
"timing": timing,
|
||||
"environment": {
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"cuda": torch.version.cuda,
|
||||
"gpu": torch.cuda.get_device_name(0),
|
||||
"compute_capability": list(
|
||||
torch.cuda.get_device_capability(0)
|
||||
),
|
||||
"cublas_workspace_config": os.environ[
|
||||
"CUBLAS_WORKSPACE_CONFIG"
|
||||
],
|
||||
"deterministic_algorithms": (
|
||||
torch.are_deterministic_algorithms_enabled()
|
||||
),
|
||||
"autocast": "cuda-bfloat16-forward-fp32-cross-entropy",
|
||||
"compile": False,
|
||||
},
|
||||
"artifacts": {
|
||||
"runner_sha256": parent.file_sha256(Path(__file__)),
|
||||
"protocol_sha256": parent.file_sha256(
|
||||
repo_root
|
||||
/ "research"
|
||||
/ "K3_ATTNRES_LOCAL_PATH_PROTOCOL.md"
|
||||
),
|
||||
"scoping_sha256": parent.file_sha256(
|
||||
repo_root
|
||||
/ "research"
|
||||
/ "K3_ATTNRES_LOCAL_PATH_SCOPING.md"
|
||||
),
|
||||
"grok_review_sha256": parent.file_sha256(
|
||||
repo_root
|
||||
/ "research"
|
||||
/ "K3_ATTNRES_LOCAL_PATH_GROK_REVIEW.md"
|
||||
),
|
||||
},
|
||||
}
|
||||
result["round06_equivalence"] = (
|
||||
frozen_training_compare(result, parent_raw) if steps else None
|
||||
)
|
||||
result["canonical_sha256_without_self"] = parent.canonical_sha256(
|
||||
result
|
||||
)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = args.output.with_suffix(args.output.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
+ "\n"
|
||||
)
|
||||
os.replace(temporary, args.output)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"output": str(args.output),
|
||||
"run_kind": args.run_kind,
|
||||
"seed": args.seed,
|
||||
"steps": steps,
|
||||
"final_bpc": evaluations[-1]["bits_per_byte"],
|
||||
"canonical_sha256": result[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"timing": timing,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,40 @@
|
||||
# Attention Residuals spike-path diagnostics
|
||||
|
||||
This directory implements preregistered protocol
|
||||
`llm-atlas-k3-attnres-spike-path-v1`.
|
||||
|
||||
It is a targeted follow-up to Round 05. It replays the exact depth-32 Block
|
||||
training contract and adds diagnostic-only activation positions, gradient
|
||||
reductions, and same-forward backward-rule interventions. It is not a Kimi K3
|
||||
checkpoint run and does not recover the paper's unpublished Figure 5 telemetry.
|
||||
|
||||
## Frozen environment
|
||||
|
||||
```text
|
||||
Python /home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python
|
||||
PyTorch 2.11.0+cu128
|
||||
GPU NVIDIA GeForce RTX 5090
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8
|
||||
```
|
||||
|
||||
## Step-0 smoke
|
||||
|
||||
```bash
|
||||
CUBLAS_WORKSPACE_CONFIG=:4096:8 \
|
||||
/home/wuyang/.pyenv/versions/3.10.14/envs/navi-router-cu128/bin/python \
|
||||
experiments/k3/attnres_spike/train.py \
|
||||
--run-kind smoke \
|
||||
--seed 2026073001 \
|
||||
--cache-dir /home/wuyang/.cache/llm-atlas/k3-attnres-gradient-scale-v1 \
|
||||
--parent-manifest experiments/k3/attnres_gradient/manifest.json \
|
||||
--manifest experiments/k3/attnres_spike/manifest.json \
|
||||
--output /home/wuyang/.cache/llm-atlas/k3-attnres-spike-path-v1/smoke/seed-2026073001.json
|
||||
```
|
||||
|
||||
Formal cells use `--run-kind formal` and all three preregistered seeds. The
|
||||
independent replay uses `--run-kind replay --seed 2026073001`. Formal and replay
|
||||
runs are fixed to 8,000 steps; smoke performs the complete step-0 diagnostic
|
||||
gate without an optimizer step.
|
||||
|
||||
Raw outputs are copied into `results/raw/` only after training equivalence,
|
||||
forward identity, loss-scale, reduction, and replay gates pass.
|
||||
@@ -0,0 +1,751 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Aggregate and gate the preregistered AttnRes spike-path study."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
PROTOCOL_ID = "llm-atlas-k3-attnres-spike-path-v1"
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
POSITIONS = (
|
||||
"pre_attention_input",
|
||||
"attention_branch_output",
|
||||
"post_attention_state",
|
||||
"pre_mlp_input",
|
||||
"mlp_branch_output",
|
||||
"post_mlp_state",
|
||||
)
|
||||
MAIN_REDUCTIONS = (
|
||||
"element_rms",
|
||||
"token_rms_mean",
|
||||
"token_rms_median",
|
||||
"token_rms_p95",
|
||||
)
|
||||
ALL_REDUCTIONS = MAIN_REDUCTIONS + (
|
||||
"batch_mean_rms",
|
||||
"token_mean_rms",
|
||||
"global_l2",
|
||||
)
|
||||
MODES = (
|
||||
"learned",
|
||||
"detached_learned",
|
||||
"uniform_value_backward",
|
||||
)
|
||||
SPIKE_LAYERS = (21, 22, 23, 24, 25)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--raw-dir", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--aggregate-output", type=Path, required=True)
|
||||
parser.add_argument("--compact-output", type=Path, required=True)
|
||||
parser.add_argument("--reproduction-output", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_sha256(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(
|
||||
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
||||
).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def mean(values: Iterable[float]) -> float:
|
||||
return statistics.fmean(values)
|
||||
|
||||
|
||||
def average_ranks(values: list[float]) -> list[float]:
|
||||
ordered = sorted(range(len(values)), key=lambda index: (values[index], index))
|
||||
ranks = [0.0] * len(values)
|
||||
cursor = 0
|
||||
while cursor < len(ordered):
|
||||
end = cursor + 1
|
||||
while end < len(ordered) and values[ordered[end]] == values[ordered[cursor]]:
|
||||
end += 1
|
||||
average_rank = (cursor + 1 + end) / 2
|
||||
for offset in range(cursor, end):
|
||||
ranks[ordered[offset]] = average_rank
|
||||
cursor = end
|
||||
return ranks
|
||||
|
||||
|
||||
def pearson(left: list[float], right: list[float]) -> float:
|
||||
left_mean = mean(left)
|
||||
right_mean = mean(right)
|
||||
numerator = sum(
|
||||
(x - left_mean) * (y - right_mean)
|
||||
for x, y in zip(left, right)
|
||||
)
|
||||
left_square = sum((value - left_mean) ** 2 for value in left)
|
||||
right_square = sum((value - right_mean) ** 2 for value in right)
|
||||
if left_square == 0 or right_square == 0:
|
||||
raise RuntimeError("correlation is undefined for a constant vector")
|
||||
return numerator / math.sqrt(left_square * right_square)
|
||||
|
||||
|
||||
def spearman(left: list[float], right: list[float]) -> float:
|
||||
return pearson(average_ranks(left), average_ranks(right))
|
||||
|
||||
|
||||
def load_run(path: Path, *, expected_kind: str, expected_seed: int) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text())
|
||||
if value["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError(f"protocol mismatch: {path}")
|
||||
if value["run_kind"] != expected_kind or value["seed"] != expected_seed:
|
||||
raise RuntimeError(f"run identity mismatch: {path}")
|
||||
canonical = value.pop("canonical_sha256_without_self")
|
||||
if canonical_sha256(value) != canonical:
|
||||
raise RuntimeError(f"canonical hash mismatch: {path}")
|
||||
value["canonical_sha256_without_self"] = canonical
|
||||
if expected_kind != "smoke":
|
||||
if not value["round05_equivalence"]["passed"]:
|
||||
raise RuntimeError(f"Round 05 equivalence failed: {path}")
|
||||
if value["steps"] != 8000:
|
||||
raise RuntimeError(f"formal step count mismatch: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def final_diagnostic(run: dict[str, Any]) -> dict[str, Any]:
|
||||
matches = [item for item in run["diagnostics"] if item["step"] == 8000]
|
||||
if len(matches) != 1:
|
||||
raise RuntimeError("final diagnostic missing or duplicated")
|
||||
return matches[0]
|
||||
|
||||
|
||||
def metric(
|
||||
run: dict[str, Any],
|
||||
*,
|
||||
position: str,
|
||||
reduction: str,
|
||||
mode: str = "learned",
|
||||
) -> dict[str, Any]:
|
||||
return final_diagnostic(run)["modes"][mode]["positions"][position][
|
||||
"reductions"
|
||||
][reduction]
|
||||
|
||||
|
||||
def compare_replay(formal: dict[str, Any], replay: dict[str, Any]) -> dict[str, Any]:
|
||||
fields = (
|
||||
"architecture",
|
||||
"depth",
|
||||
"seed",
|
||||
"steps",
|
||||
"batch_size",
|
||||
"target_bytes_seen",
|
||||
"manifest",
|
||||
"model",
|
||||
"optimizer",
|
||||
"hashes",
|
||||
"evaluations",
|
||||
"diagnostics",
|
||||
"training_history",
|
||||
"environment",
|
||||
"artifacts",
|
||||
"round05_equivalence",
|
||||
)
|
||||
checks = {field: formal[field] == replay[field] for field in fields}
|
||||
passed = all(checks.values())
|
||||
if not passed:
|
||||
raise RuntimeError(f"Round 06 replay mismatch: {checks}")
|
||||
compare_payload = {field: formal[field] for field in fields}
|
||||
return {
|
||||
"passed": True,
|
||||
"checks": checks,
|
||||
"frozen_compare_sha256": canonical_sha256(compare_payload),
|
||||
}
|
||||
|
||||
|
||||
def reduction_robustness(runs: list[dict[str, Any]], manifest: dict[str, Any]) -> dict[str, Any]:
|
||||
threshold = manifest["thresholds"]
|
||||
cells = []
|
||||
for run in runs:
|
||||
reference = metric(
|
||||
run,
|
||||
position="post_mlp_state",
|
||||
reduction="element_rms",
|
||||
)
|
||||
for reduction in MAIN_REDUCTIONS:
|
||||
candidate = metric(
|
||||
run,
|
||||
position="post_mlp_state",
|
||||
reduction=reduction,
|
||||
)
|
||||
stats = candidate["statistics"]
|
||||
rho = spearman(reference["values"], candidate["values"])
|
||||
checks = {
|
||||
"spike_contrast": (
|
||||
stats["spike_contrast"]
|
||||
>= threshold["spike_contrast"]
|
||||
),
|
||||
"top_five_overlap": (
|
||||
stats["top_five_spike_overlap"]
|
||||
>= threshold["top_five_min_overlap"]
|
||||
),
|
||||
"spearman": rho >= threshold["spearman_minimum"],
|
||||
}
|
||||
cells.append(
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"reduction": reduction,
|
||||
"spike_contrast": stats["spike_contrast"],
|
||||
"top_five_layers": stats["top_five_layers"],
|
||||
"top_five_spike_overlap": stats[
|
||||
"top_five_spike_overlap"
|
||||
],
|
||||
"spearman_vs_element_rms": rho,
|
||||
"checks": checks,
|
||||
"passed": all(checks.values()),
|
||||
}
|
||||
)
|
||||
passed_count = sum(item["passed"] for item in cells)
|
||||
if passed_count == len(cells):
|
||||
verdict = "robust within the preregistered reduction family"
|
||||
elif passed_count == 0:
|
||||
verdict = "not robust at this threshold"
|
||||
else:
|
||||
verdict = "mixed"
|
||||
return {
|
||||
"verdict": verdict,
|
||||
"passed_cells": passed_count,
|
||||
"total_cells": len(cells),
|
||||
"cells": cells,
|
||||
}
|
||||
|
||||
|
||||
def visible_positions(runs: list[dict[str, Any]], manifest: dict[str, Any]) -> dict[str, Any]:
|
||||
threshold = manifest["thresholds"]["spike_contrast"]
|
||||
positions = []
|
||||
for position in POSITIONS:
|
||||
per_seed = []
|
||||
for run in runs:
|
||||
stats = metric(
|
||||
run, position=position, reduction="element_rms"
|
||||
)["statistics"]
|
||||
per_seed.append(
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"spike_contrast": stats["spike_contrast"],
|
||||
"peak_layer": stats["peak_layer"],
|
||||
"peak_normalized": stats["peak_normalized"],
|
||||
"passed": stats["spike_contrast"] >= threshold,
|
||||
}
|
||||
)
|
||||
positions.append(
|
||||
{
|
||||
"position": position,
|
||||
"visible_3_of_3": all(item["passed"] for item in per_seed),
|
||||
"per_seed": per_seed,
|
||||
"mean_spike_contrast": mean(
|
||||
item["spike_contrast"] for item in per_seed
|
||||
),
|
||||
}
|
||||
)
|
||||
visible = [item["position"] for item in positions if item["visible_3_of_3"]]
|
||||
return {
|
||||
"criterion": f"step 8000 element-RMS spike contrast >= {threshold} in 3/3 seeds",
|
||||
"visible_positions": visible,
|
||||
"earliest_observed_tensor": visible[0] if visible else None,
|
||||
"verdict": "visible at one or more positions" if visible else "position-mixed",
|
||||
"positions": positions,
|
||||
}
|
||||
|
||||
|
||||
def intervention_effect(
|
||||
runs: list[dict[str, Any]],
|
||||
manifest: dict[str, Any],
|
||||
*,
|
||||
source_mode: str,
|
||||
target_mode: str,
|
||||
label: str,
|
||||
) -> dict[str, Any]:
|
||||
threshold = manifest["thresholds"]["material_relative_drop"]
|
||||
epsilon = manifest["thresholds"]["positive_denominator_epsilon"]
|
||||
per_seed = []
|
||||
for run in runs:
|
||||
source = metric(
|
||||
run,
|
||||
position="post_mlp_state",
|
||||
reduction="element_rms",
|
||||
mode=source_mode,
|
||||
)["statistics"]
|
||||
target = metric(
|
||||
run,
|
||||
position="post_mlp_state",
|
||||
reduction="element_rms",
|
||||
mode=target_mode,
|
||||
)["statistics"]
|
||||
if (
|
||||
not math.isfinite(source["spike_contrast"])
|
||||
or not math.isfinite(source["peak_normalized"])
|
||||
or source["spike_contrast"] <= epsilon
|
||||
or source["peak_normalized"] <= epsilon
|
||||
):
|
||||
raise RuntimeError("invalid intervention denominator")
|
||||
contrast_drop = (
|
||||
source["spike_contrast"] - target["spike_contrast"]
|
||||
) / source["spike_contrast"]
|
||||
peak_drop = (
|
||||
source["peak_normalized"] - target["peak_normalized"]
|
||||
) / source["peak_normalized"]
|
||||
per_seed.append(
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"source_spike_contrast": source["spike_contrast"],
|
||||
"target_spike_contrast": target["spike_contrast"],
|
||||
"relative_drop_contrast": contrast_drop,
|
||||
"source_peak_normalized": source["peak_normalized"],
|
||||
"target_peak_normalized": target["peak_normalized"],
|
||||
"relative_drop_peak": peak_drop,
|
||||
"passed": (
|
||||
contrast_drop >= threshold and peak_drop >= threshold
|
||||
),
|
||||
}
|
||||
)
|
||||
passed = all(item["passed"] for item in per_seed)
|
||||
same_direction = all(
|
||||
item["relative_drop_contrast"] > 0
|
||||
and item["relative_drop_peak"] > 0
|
||||
for item in per_seed
|
||||
)
|
||||
opposite_direction = all(
|
||||
item["relative_drop_contrast"] < 0
|
||||
and item["relative_drop_peak"] < 0
|
||||
for item in per_seed
|
||||
)
|
||||
if passed:
|
||||
verdict = "material sensitivity at the preregistered threshold"
|
||||
elif same_direction:
|
||||
verdict = "same-direction but below the joint threshold"
|
||||
elif opposite_direction:
|
||||
verdict = "opposite direction in 3/3 seeds; no material reduction"
|
||||
else:
|
||||
verdict = "mixed"
|
||||
return {
|
||||
"label": label,
|
||||
"source_mode": source_mode,
|
||||
"target_mode": target_mode,
|
||||
"threshold": threshold,
|
||||
"passed_3_of_3": passed,
|
||||
"same_direction_3_of_3": same_direction,
|
||||
"opposite_direction_3_of_3": opposite_direction,
|
||||
"verdict": verdict,
|
||||
"per_seed": per_seed,
|
||||
"mean_relative_drop_contrast": mean(
|
||||
item["relative_drop_contrast"] for item in per_seed
|
||||
),
|
||||
"mean_relative_drop_peak": mean(
|
||||
item["relative_drop_peak"] for item in per_seed
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def intervention_summary(runs: list[dict[str, Any]], manifest: dict[str, Any]) -> dict[str, Any]:
|
||||
forward_gates = []
|
||||
for run in runs:
|
||||
diagnostic = final_diagnostic(run)
|
||||
forward_gates.append(
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"passed": diagnostic["forward_identity_gate"]["passed"],
|
||||
}
|
||||
)
|
||||
if not all(item["passed"] for item in forward_gates):
|
||||
raise RuntimeError("a final forward identity gate failed")
|
||||
return {
|
||||
"scope": manifest["interventions"]["scope"],
|
||||
"forward_identity": forward_gates,
|
||||
"softmax_key_path": intervention_effect(
|
||||
runs,
|
||||
manifest,
|
||||
source_mode="learned",
|
||||
target_mode="detached_learned",
|
||||
label="global removal of all mixer softmax/query/key source-gradient paths",
|
||||
),
|
||||
"value_coefficients": intervention_effect(
|
||||
runs,
|
||||
manifest,
|
||||
source_mode="detached_learned",
|
||||
target_mode="uniform_value_backward",
|
||||
label="global replacement of learned value-backward coefficients with 1/N",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def trajectory(runs: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
result = []
|
||||
for run in runs:
|
||||
evaluations = {
|
||||
item["step"]: item["bits_per_byte"]
|
||||
for item in run["evaluations"]
|
||||
}
|
||||
points = []
|
||||
for diagnostic in run["diagnostics"]:
|
||||
stats = diagnostic["modes"]["learned"]["positions"][
|
||||
"post_mlp_state"
|
||||
]["reductions"]["element_rms"]["statistics"]
|
||||
points.append(
|
||||
{
|
||||
"step": diagnostic["step"],
|
||||
"bits_per_byte": evaluations[diagnostic["step"]],
|
||||
"population_cv": stats["population_cv"],
|
||||
"spike_contrast": stats["spike_contrast"],
|
||||
"peak_layer": stats["peak_layer"],
|
||||
"peak_normalized": stats["peak_normalized"],
|
||||
}
|
||||
)
|
||||
result.append({"seed": run["seed"], "points": points})
|
||||
return result
|
||||
|
||||
|
||||
def mixer_associations(runs: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
rows = []
|
||||
for run in runs:
|
||||
diagnostic = final_diagnostic(run)
|
||||
gradients = metric(
|
||||
run,
|
||||
position="post_mlp_state",
|
||||
reduction="element_rms",
|
||||
)["statistics"]["normalized"]
|
||||
mixers = diagnostic["modes"]["learned"]["mixers"]
|
||||
for layer in range(1, 33):
|
||||
attention = mixers[2 * (layer - 1)]
|
||||
mlp = mixers[2 * (layer - 1) + 1]
|
||||
rows.append(
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"layer": layer,
|
||||
"group": (layer - 1) // 4 + 1,
|
||||
"offset": (layer - 1) % 4 + 1,
|
||||
"normalized_gradient": gradients[layer - 1],
|
||||
"attention_latest": attention["latest_source_mass"],
|
||||
"attention_entropy": attention["normalized_entropy"],
|
||||
"mlp_latest": mlp["latest_source_mass"],
|
||||
"mlp_entropy": mlp["normalized_entropy"],
|
||||
"attention_max": attention["max_source_mass"],
|
||||
"mlp_max": mlp["max_source_mass"],
|
||||
}
|
||||
)
|
||||
|
||||
def correlations(selected: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
gradient = [row["normalized_gradient"] for row in selected]
|
||||
result = {"points": len(selected)}
|
||||
for key in (
|
||||
"attention_latest",
|
||||
"attention_entropy",
|
||||
"mlp_latest",
|
||||
"mlp_entropy",
|
||||
"attention_max",
|
||||
"mlp_max",
|
||||
):
|
||||
values = [row[key] for row in selected]
|
||||
result[key] = {
|
||||
"pearson": pearson(gradient, values),
|
||||
"spearman": spearman(gradient, values),
|
||||
}
|
||||
return result
|
||||
|
||||
target_layers = []
|
||||
for layer in range(19, 29):
|
||||
selected = [row for row in rows if row["layer"] == layer]
|
||||
target_layers.append(
|
||||
{
|
||||
"layer": layer,
|
||||
"group": selected[0]["group"],
|
||||
"offset": selected[0]["offset"],
|
||||
**{
|
||||
key: mean(row[key] for row in selected)
|
||||
for key in (
|
||||
"normalized_gradient",
|
||||
"attention_latest",
|
||||
"attention_entropy",
|
||||
"mlp_latest",
|
||||
"mlp_entropy",
|
||||
"attention_max",
|
||||
"mlp_max",
|
||||
)
|
||||
},
|
||||
"per_seed_normalized_gradient": [
|
||||
row["normalized_gradient"] for row in selected
|
||||
],
|
||||
}
|
||||
)
|
||||
return {
|
||||
"identity": "observational association; not causal attribution",
|
||||
"all_layers": correlations(rows),
|
||||
"layers_19_28": correlations(
|
||||
[row for row in rows if 19 <= row["layer"] <= 28]
|
||||
),
|
||||
"target_layers": target_layers,
|
||||
}
|
||||
|
||||
|
||||
def final_arrays(runs: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
values = []
|
||||
for run in runs:
|
||||
positions = {}
|
||||
for position in POSITIONS:
|
||||
positions[position] = {}
|
||||
for reduction in ALL_REDUCTIONS:
|
||||
item = metric(
|
||||
run, position=position, reduction=reduction
|
||||
)
|
||||
positions[position][reduction] = {
|
||||
"values": item["values"],
|
||||
"statistics": item["statistics"],
|
||||
}
|
||||
interventions = {}
|
||||
for mode in MODES:
|
||||
item = metric(
|
||||
run,
|
||||
position="post_mlp_state",
|
||||
reduction="element_rms",
|
||||
mode=mode,
|
||||
)
|
||||
interventions[mode] = {
|
||||
"values": item["values"],
|
||||
"statistics": item["statistics"],
|
||||
}
|
||||
values.append(
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"positions": positions,
|
||||
"interventions": interventions,
|
||||
"final_mixers": final_diagnostic(run)["modes"]["learned"][
|
||||
"mixers"
|
||||
],
|
||||
}
|
||||
)
|
||||
return values
|
||||
|
||||
|
||||
def compact_final_arrays(runs: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
values = []
|
||||
for run in runs:
|
||||
position_element_rms = {
|
||||
position: metric(
|
||||
run,
|
||||
position=position,
|
||||
reduction="element_rms",
|
||||
)
|
||||
for position in POSITIONS
|
||||
}
|
||||
post_mlp_reductions = {
|
||||
reduction: metric(
|
||||
run,
|
||||
position="post_mlp_state",
|
||||
reduction=reduction,
|
||||
)
|
||||
for reduction in ALL_REDUCTIONS
|
||||
}
|
||||
interventions = {
|
||||
mode: metric(
|
||||
run,
|
||||
position="post_mlp_state",
|
||||
reduction="element_rms",
|
||||
mode=mode,
|
||||
)
|
||||
for mode in MODES
|
||||
}
|
||||
values.append(
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"position_element_rms": position_element_rms,
|
||||
"post_mlp_reductions": post_mlp_reductions,
|
||||
"interventions": interventions,
|
||||
}
|
||||
)
|
||||
return values
|
||||
|
||||
|
||||
def write_json(path: Path, value: dict[str, Any]) -> None:
|
||||
value["canonical_sha256_without_self"] = canonical_sha256(value)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
)
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
manifest = json.loads(args.manifest.read_text())
|
||||
if manifest["protocol_id"] != PROTOCOL_ID:
|
||||
raise RuntimeError("manifest protocol mismatch")
|
||||
|
||||
formal_paths = {
|
||||
seed: args.raw_dir / f"formal-seed-{seed}.json"
|
||||
for seed in SEEDS
|
||||
}
|
||||
replay_path = args.raw_dir / f"replay-seed-{SEEDS[0]}.json"
|
||||
formal = [
|
||||
load_run(formal_paths[seed], expected_kind="formal", expected_seed=seed)
|
||||
for seed in SEEDS
|
||||
]
|
||||
replay = load_run(
|
||||
replay_path, expected_kind="replay", expected_seed=SEEDS[0]
|
||||
)
|
||||
replay_gate = compare_replay(formal[0], replay)
|
||||
|
||||
reduction = reduction_robustness(formal, manifest)
|
||||
positions = visible_positions(formal, manifest)
|
||||
interventions = intervention_summary(formal, manifest)
|
||||
trajectories = trajectory(formal)
|
||||
mixers = mixer_associations(formal)
|
||||
arrays = final_arrays(formal)
|
||||
|
||||
raw_files = {
|
||||
path.name: {
|
||||
"file_sha256": file_sha256(path),
|
||||
"canonical_sha256": run["canonical_sha256_without_self"],
|
||||
}
|
||||
for path, run in [
|
||||
*[(formal_paths[seed], run) for seed, run in zip(SEEDS, formal)],
|
||||
(replay_path, replay),
|
||||
]
|
||||
}
|
||||
reproduction = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"raw_files": raw_files,
|
||||
"replay_gate": replay_gate,
|
||||
"artifacts": {
|
||||
"manifest": file_sha256(args.manifest),
|
||||
"runner": formal[0]["artifacts"]["runner_sha256"],
|
||||
"protocol": formal[0]["artifacts"]["protocol_sha256"],
|
||||
"scoping": formal[0]["artifacts"]["scoping_sha256"],
|
||||
"analyzer": file_sha256(Path(__file__)),
|
||||
},
|
||||
}
|
||||
write_json(args.reproduction_output, reproduction)
|
||||
|
||||
aggregate = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"study": {
|
||||
"identity": manifest["study_identity"],
|
||||
"architecture": "block",
|
||||
"depth": 32,
|
||||
"seeds": list(SEEDS),
|
||||
"steps": 8000,
|
||||
"spike_layers": list(SPIKE_LAYERS),
|
||||
},
|
||||
"gates": {
|
||||
"round05_equivalence": [
|
||||
{
|
||||
"seed": run["seed"],
|
||||
**run["round05_equivalence"],
|
||||
}
|
||||
for run in formal
|
||||
],
|
||||
"replay": replay_gate,
|
||||
"forward_identity": interventions["forward_identity"],
|
||||
"initialization_negative_control": [
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"passed": run["diagnostics"][0][
|
||||
"initialization_negative_control"
|
||||
]["passed"],
|
||||
}
|
||||
for run in formal
|
||||
],
|
||||
"loss_scale": [
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"passed": run["diagnostics"][0][
|
||||
"loss_scale_gate"
|
||||
]["passed"],
|
||||
}
|
||||
for run in formal
|
||||
],
|
||||
},
|
||||
"verdicts": {
|
||||
"reduction_robustness": reduction,
|
||||
"visible_positions": positions,
|
||||
"interventions": interventions,
|
||||
},
|
||||
"trajectory": trajectories,
|
||||
"mixer_associations": mixers,
|
||||
"final_arrays": arrays,
|
||||
"runs": [
|
||||
{
|
||||
"seed": run["seed"],
|
||||
"canonical_sha256": run["canonical_sha256_without_self"],
|
||||
"final_bpc": run["evaluations"][-1]["bits_per_byte"],
|
||||
"final_model_state": run["hashes"]["final_model_state"],
|
||||
"final_optimizer_state": run["hashes"][
|
||||
"final_optimizer_state"
|
||||
],
|
||||
"timing": run["timing"],
|
||||
}
|
||||
for run in formal
|
||||
],
|
||||
"reproduction_canonical_sha256": reproduction[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
}
|
||||
write_json(args.aggregate_output, aggregate)
|
||||
|
||||
compact = {
|
||||
"schema_version": 1,
|
||||
"protocol_id": PROTOCOL_ID,
|
||||
"study": aggregate["study"],
|
||||
"gates": aggregate["gates"],
|
||||
"verdicts": aggregate["verdicts"],
|
||||
"trajectory": trajectories,
|
||||
"mixer_associations": mixers,
|
||||
"final_arrays": compact_final_arrays(formal),
|
||||
"runs": aggregate["runs"],
|
||||
"hashes": {
|
||||
"aggregate_canonical_sha256": aggregate[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
"reproduction_canonical_sha256": reproduction[
|
||||
"canonical_sha256_without_self"
|
||||
],
|
||||
},
|
||||
}
|
||||
write_json(args.compact_output, compact)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"aggregate": str(args.aggregate_output),
|
||||
"compact": str(args.compact_output),
|
||||
"reproduction": str(args.reproduction_output),
|
||||
"reduction_verdict": reduction["verdict"],
|
||||
"earliest_observed_tensor": positions[
|
||||
"earliest_observed_tensor"
|
||||
],
|
||||
"softmax_key_path": interventions["softmax_key_path"][
|
||||
"verdict"
|
||||
],
|
||||
"value_coefficients": interventions["value_coefficients"][
|
||||
"verdict"
|
||||
],
|
||||
"replay_exact": replay_gate["passed"],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"protocol_id": "llm-atlas-k3-attnres-spike-path-v1",
|
||||
"parent_protocol_id": "llm-atlas-k3-attnres-gradient-scale-v1",
|
||||
"study_identity": "targeted follow-up informed by Round 05; not blind discovery",
|
||||
"architecture": "block",
|
||||
"depth": 32,
|
||||
"aggregation_groups": 8,
|
||||
"formal_seeds": [
|
||||
2026073001,
|
||||
2026073002,
|
||||
2026073003
|
||||
],
|
||||
"replay": {
|
||||
"architecture": "block",
|
||||
"depth": 32,
|
||||
"seed": 2026073001
|
||||
},
|
||||
"training": {
|
||||
"steps": 8000,
|
||||
"batch_size": 32,
|
||||
"context": 256,
|
||||
"target_bytes_per_cell": 65536000,
|
||||
"diagnostic_steps": [
|
||||
0,
|
||||
100,
|
||||
500,
|
||||
2000,
|
||||
4000,
|
||||
8000
|
||||
]
|
||||
},
|
||||
"positions": [
|
||||
"pre_attention_input",
|
||||
"attention_branch_output",
|
||||
"post_attention_state",
|
||||
"pre_mlp_input",
|
||||
"mlp_branch_output",
|
||||
"post_mlp_state"
|
||||
],
|
||||
"reductions": {
|
||||
"confirmatory": [
|
||||
"element_rms",
|
||||
"token_rms_mean",
|
||||
"token_rms_median",
|
||||
"token_rms_p95"
|
||||
],
|
||||
"exploratory": [
|
||||
"batch_mean_rms",
|
||||
"token_mean_rms"
|
||||
],
|
||||
"algebraic_control": [
|
||||
"global_l2"
|
||||
],
|
||||
"quantile_definition": "Hyndman-Fan Type 7 linear interpolation on explicitly sorted values"
|
||||
},
|
||||
"interventions": {
|
||||
"modes": [
|
||||
"learned",
|
||||
"detached_learned",
|
||||
"uniform_value_backward"
|
||||
],
|
||||
"steps": [
|
||||
0,
|
||||
8000
|
||||
],
|
||||
"scope": "all 64 depth mixers plus the output mixer",
|
||||
"training_uses_custom_autograd": false,
|
||||
"confirmatory_reduction": "element_rms"
|
||||
},
|
||||
"spike_layers_one_based": [
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25
|
||||
],
|
||||
"thresholds": {
|
||||
"spike_contrast": 1.5,
|
||||
"top_five_min_overlap": 3,
|
||||
"spearman_minimum": 0.8,
|
||||
"material_relative_drop": 0.2,
|
||||
"spectrum_absolute_tolerance": 1e-06,
|
||||
"raw_relative_tolerance": 1e-06,
|
||||
"positive_denominator_epsilon": 1e-30
|
||||
},
|
||||
"parent_artifacts": {
|
||||
"manifest_path": "experiments/k3/attnres_gradient/manifest.json",
|
||||
"manifest_sha256": "080afb17d1e036c0bba0a799fdb8b98ee4ad652bd42dd1b3b67110dd2ede6371",
|
||||
"runner_path": "experiments/k3/attnres_gradient/train.py",
|
||||
"runner_sha256": "04ae69e10c58972c9193c2d31c7e09d924a0d0e834107afa4ba128c64ac5800f",
|
||||
"protocol_path": "research/K3_ATTNRES_GRADIENT_SCALE_PROTOCOL.md",
|
||||
"protocol_sha256": "f772629b3b82975b6756721c3a3bb57cc4171dfa26ba5b1e8043dc91c9dcce22",
|
||||
"formal_schedule_sha256": "5041e09b167f229248d2462324e8c254b8f5938975f135dcd8192b00a54a4f4e",
|
||||
"validation_tensor_sha256": "f459316f13078a163b47c133511bb7181e05170ab89516e196490113893ce338",
|
||||
"diagnostic_tensor_sha256": "21117e31db302b10d67b63f035665dc8f220b879d216ccd12b7d2ba86e7b1716"
|
||||
},
|
||||
"round05_expected": {
|
||||
"2026073001": {
|
||||
"raw_file_sha256": "29e1d638b481619c7b32de402122523db8b17cc1fc67a8881fa1ba132a1d5c38",
|
||||
"canonical_sha256": "c7554d9beea6fe9e63617aafd88403f7fc93b4b305a0c6d603a98a8572e5b5f1",
|
||||
"final_model_state": "3f0b97ece3a15571ba3d656f589f512ca0bb9e20083c9f58a42ccaee14892f59",
|
||||
"final_optimizer_state": "ed03e6fbd4a12d8b063dcb22e0437754285f54d585374cd52fbd534f05d24637"
|
||||
},
|
||||
"2026073002": {
|
||||
"raw_file_sha256": "21199deb2199395061e51220fd8c7afd04a1135a6381e406da9b5795e3ad5032",
|
||||
"canonical_sha256": "b4262e697e2269457fdebf31f75008383c6e8024ee1bbf96e5962fb2ba1dec18",
|
||||
"final_model_state": "bd2556388aeaa211b798c283c7cbd8ccd29edf166a2922fa13d172e8dfdc38d1",
|
||||
"final_optimizer_state": "0b101eab3bc7d8d654be2ea335c86fc25563ce19912d721844ee4e639c569e77"
|
||||
},
|
||||
"2026073003": {
|
||||
"raw_file_sha256": "c0d7f1bcfa9fa7f8f3134ca4571bdf23a951182d03b7de9611a6b4b89667d4ac",
|
||||
"canonical_sha256": "513b85666fcdbf75424596e968cd73733314ec188345e453998de08b18dcef65",
|
||||
"final_model_state": "638568aede21890773b6932a19ec4e112f5ac0a4770ba3402fcd82980a9ecf76",
|
||||
"final_optimizer_state": "83947fd743ec8e3e31ca7788fd201981846f0f1c7e9e38c88afcef95cfc6ec4e"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"artifacts": {
|
||||
"analyzer": "b9dd7f723e8c5765ecffd9ae3050ff7a95643f95853adaeeaa2c2c323d0ca057",
|
||||
"manifest": "d5302a249249a07d362819134763d14e7d32307f22cff416c665ed9606142fef",
|
||||
"protocol": "6cb101b8760d9f1c81caeb2f16880b16152da103867224a06761a75a12984a16",
|
||||
"runner": "77298081d3c491d2e88e4705995174b9879ef377f520eb5fe5ea107e7a1da084",
|
||||
"scoping": "590166bd62580bb8238293823cfcc39bc0a465fec4c697025343f3f1138abd27"
|
||||
},
|
||||
"canonical_sha256_without_self": "7c522314742920961be84b750d823ce41814f10fd6129f0531208116e2fe3515",
|
||||
"protocol_id": "llm-atlas-k3-attnres-spike-path-v1",
|
||||
"raw_files": {
|
||||
"formal-seed-2026073001.json": {
|
||||
"canonical_sha256": "76b0ccfb55c38baef50c395788ac4b351cbe0d58de70064b50702acb5c93f515",
|
||||
"file_sha256": "e39e93b7a7fce3c56f5f14f95cfdc04afdce53628affee1202fe62bd1bdb7f71"
|
||||
},
|
||||
"formal-seed-2026073002.json": {
|
||||
"canonical_sha256": "5352c74eca853b375c0e85933dafd7c5916c39fc59052e742ca14ffd6d68bc78",
|
||||
"file_sha256": "1c6f6c731030ec0adb2a8e7a4d586e0c4005cc3319568a7ac83c08c2a4b8eaf8"
|
||||
},
|
||||
"formal-seed-2026073003.json": {
|
||||
"canonical_sha256": "7e764c07e90b78c4cd0acc2e99600225f16428cbb25d5188a0a5a8fe797f8766",
|
||||
"file_sha256": "115f8245577ece6dfaaa8ada68445c186e6523a7f3b26efcc3eb4c0c4ce82406"
|
||||
},
|
||||
"replay-seed-2026073001.json": {
|
||||
"canonical_sha256": "c9ebccba0acf10b05fea65beade50e588fd9a83b1807f7ebba8ec1ab836b26fb",
|
||||
"file_sha256": "4f91c7a2698662ea69af9aa3425e144107834fd9b6b717fb82b84f153b867dc5"
|
||||
}
|
||||
},
|
||||
"replay_gate": {
|
||||
"checks": {
|
||||
"architecture": true,
|
||||
"artifacts": true,
|
||||
"batch_size": true,
|
||||
"depth": true,
|
||||
"diagnostics": true,
|
||||
"environment": true,
|
||||
"evaluations": true,
|
||||
"hashes": true,
|
||||
"manifest": true,
|
||||
"model": true,
|
||||
"optimizer": true,
|
||||
"round05_equivalence": true,
|
||||
"seed": true,
|
||||
"steps": true,
|
||||
"target_bytes_seen": true,
|
||||
"training_history": true
|
||||
},
|
||||
"frozen_compare_sha256": "05396eaf56429ca8d87815ecf00a5f78be5a9b2ec7cc96014856c09dcedcbb87",
|
||||
"passed": true
|
||||
},
|
||||
"schema_version": 1
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,178 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild the Round 06 scoping table from frozen Round 05 raw files."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
|
||||
SEEDS = (2026073001, 2026073002, 2026073003)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--raw-dir",
|
||||
type=Path,
|
||||
default=Path(__file__).resolve().parents[1]
|
||||
/ "attnres_gradient"
|
||||
/ "results"
|
||||
/ "raw",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def pearson(left: list[float], right: list[float]) -> float:
|
||||
left_mean = statistics.fmean(left)
|
||||
right_mean = statistics.fmean(right)
|
||||
numerator = sum(
|
||||
(x - left_mean) * (y - right_mean)
|
||||
for x, y in zip(left, right)
|
||||
)
|
||||
left_square = sum((value - left_mean) ** 2 for value in left)
|
||||
right_square = sum((value - right_mean) ** 2 for value in right)
|
||||
return numerator / math.sqrt(left_square * right_square)
|
||||
|
||||
|
||||
def mixer_metrics(value: dict[str, Any]) -> dict[str, float | int]:
|
||||
weights = value["mean_weights"]
|
||||
sources = value["sources"]
|
||||
return {
|
||||
"sources": sources,
|
||||
"latest": weights[-1],
|
||||
"maximum": max(weights),
|
||||
"normalized_entropy": (
|
||||
1.0
|
||||
if sources == 1
|
||||
else value["entropy_mean"] / math.log(sources)
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def average(rows: list[dict[str, Any]], key: Callable[[dict[str, Any]], float]) -> float:
|
||||
return statistics.fmean(key(row) for row in rows)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
rows = []
|
||||
source_hashes = {}
|
||||
for seed in SEEDS:
|
||||
path = (
|
||||
args.raw_dir
|
||||
/ f"formal-depth-32-block-seed-{seed}.json"
|
||||
)
|
||||
run = json.loads(path.read_text())
|
||||
source_hashes[str(seed)] = run["canonical_sha256_without_self"]
|
||||
diagnostic = next(
|
||||
item for item in run["diagnostics"] if item["step"] == 8000
|
||||
)
|
||||
gradients = diagnostic["activation_grad_rms_by_block"]
|
||||
gradient_mean = statistics.fmean(gradients)
|
||||
for layer in range(32):
|
||||
rows.append(
|
||||
{
|
||||
"seed": seed,
|
||||
"layer": layer + 1,
|
||||
"group": layer // 4 + 1,
|
||||
"offset": layer % 4 + 1,
|
||||
"normalized_gradient": gradients[layer] / gradient_mean,
|
||||
"attention": mixer_metrics(
|
||||
diagnostic["depth_weights"][2 * layer]
|
||||
),
|
||||
"mlp": mixer_metrics(
|
||||
diagnostic["depth_weights"][2 * layer + 1]
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
layer_means = []
|
||||
for layer in range(1, 33):
|
||||
selected = [row for row in rows if row["layer"] == layer]
|
||||
layer_means.append(
|
||||
{
|
||||
"layer": layer,
|
||||
"group": selected[0]["group"],
|
||||
"offset": selected[0]["offset"],
|
||||
"normalized_gradient": average(
|
||||
selected, lambda row: row["normalized_gradient"]
|
||||
),
|
||||
"attention_latest": average(
|
||||
selected, lambda row: row["attention"]["latest"]
|
||||
),
|
||||
"mlp_latest": average(
|
||||
selected, lambda row: row["mlp"]["latest"]
|
||||
),
|
||||
"attention_normalized_entropy": average(
|
||||
selected,
|
||||
lambda row: row["attention"]["normalized_entropy"],
|
||||
),
|
||||
"mlp_normalized_entropy": average(
|
||||
selected, lambda row: row["mlp"]["normalized_entropy"]
|
||||
),
|
||||
"per_seed_normalized_gradient": [
|
||||
row["normalized_gradient"] for row in selected
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
correlations = {}
|
||||
for label, selected in (
|
||||
("all_layers", rows),
|
||||
("layers_19_28", [
|
||||
row for row in rows if 19 <= row["layer"] <= 28
|
||||
]),
|
||||
):
|
||||
gradients = [row["normalized_gradient"] for row in selected]
|
||||
correlations[label] = {
|
||||
"points": len(selected),
|
||||
"attention_latest": pearson(
|
||||
gradients,
|
||||
[row["attention"]["latest"] for row in selected],
|
||||
),
|
||||
"mlp_latest": pearson(
|
||||
gradients, [row["mlp"]["latest"] for row in selected]
|
||||
),
|
||||
"attention_normalized_entropy": pearson(
|
||||
gradients,
|
||||
[
|
||||
row["attention"]["normalized_entropy"]
|
||||
for row in selected
|
||||
],
|
||||
),
|
||||
"mlp_normalized_entropy": pearson(
|
||||
gradients,
|
||||
[row["mlp"]["normalized_entropy"] for row in selected],
|
||||
),
|
||||
"attention_maximum": pearson(
|
||||
gradients,
|
||||
[row["attention"]["maximum"] for row in selected],
|
||||
),
|
||||
"mlp_maximum": pearson(
|
||||
gradients, [row["mlp"]["maximum"] for row in selected]
|
||||
),
|
||||
}
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": 1,
|
||||
"identity": "exploratory Round 05 scoping; not confirmatory Round 06",
|
||||
"source_canonical_sha256": source_hashes,
|
||||
"layers_19_28": layer_means[18:28],
|
||||
"correlations": correlations,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -17,6 +17,13 @@
|
||||
"build:data:deepseek-chat-completion-depth": "node scripts/build-deepseek-chat-completion-depth-compact.mjs",
|
||||
"build:data:deepseek-chat-sampling": "node scripts/build-deepseek-chat-sampling-compact.mjs",
|
||||
"build:data:deepseek-chat-cross-source-sampling": "node scripts/build-deepseek-chat-cross-source-sampling-compact.mjs",
|
||||
"build:data:deepseek-chat-task-bootstrap": "node scripts/build-deepseek-chat-task-bootstrap-crn-compact.mjs",
|
||||
"check:data:deepseek-chat-task-bootstrap": "node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs",
|
||||
"check:data:k3-attnres": "node scripts/check-k3-attnres-data.mjs",
|
||||
"check:data:k3-attnres-gradient": "node scripts/check-k3-attnres-gradient-data.mjs",
|
||||
"check:data:k3-attnres-spike": "node scripts/check-k3-attnres-spike-data.mjs",
|
||||
"check:data:k3-attnres-local-path": "node scripts/check-k3-attnres-local-path-data.mjs",
|
||||
"check:data:k3-attnres-forward": "node scripts/check-k3-attnres-forward-data.mjs",
|
||||
"check:site": "node scripts/check-site.mjs",
|
||||
"check:moe-browser": "node scripts/check-moe-browser.mjs",
|
||||
"check:reasoning-browser": "node scripts/check-reasoning-browser.mjs",
|
||||
@@ -35,6 +42,12 @@
|
||||
"check:deepseek-browser": "node scripts/check-deepseek-browser.mjs",
|
||||
"check:deepseek-sampling-browser": "node scripts/check-deepseek-sampling-browser.mjs",
|
||||
"check:deepseek-cross-source-sampling-browser": "node scripts/check-deepseek-cross-source-sampling-browser.mjs",
|
||||
"check:deepseek-task-bootstrap-browser": "node scripts/check-deepseek-task-bootstrap-browser.mjs",
|
||||
"check:k3-attnres-browser": "node scripts/check-k3-attnres-browser.mjs",
|
||||
"check:k3-attnres-gradient-browser": "node scripts/check-k3-attnres-gradient-browser.mjs",
|
||||
"check:k3-attnres-spike-browser": "node scripts/check-k3-attnres-spike-browser.mjs",
|
||||
"check:k3-attnres-local-path-browser": "node scripts/check-k3-attnres-local-path-browser.mjs",
|
||||
"check:k3-attnres-forward-browser": "node scripts/check-k3-attnres-forward-browser.mjs",
|
||||
"check:k3-browser": "node scripts/check-k3-browser.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -0,0 +1,786 @@
|
||||
# DeepSeek-V2-Lite-Chat：32 题任务 bootstrap 与显式共同随机数审计
|
||||
|
||||
> 协议：`llm-atlas-deepseek-chat-task-bootstrap-crn-v1`
|
||||
>
|
||||
> 模型：`deepseek-ai/DeepSeek-V2-Lite-Chat`
|
||||
>
|
||||
> revision：`85864749cd611b4353ce1decdb286193298f64c7`
|
||||
>
|
||||
> 预注册协议：
|
||||
> `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_PROTOCOL.md`
|
||||
>
|
||||
> 冻结清单：
|
||||
> `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json`
|
||||
>
|
||||
> 执行日期：2026-07-30
|
||||
|
||||
## 0. 先说结论
|
||||
|
||||
Round 08 把 Round 07 的“每域 4 道题 × 4 seeds”扩成：
|
||||
|
||||
```text
|
||||
主分析：
|
||||
HumanEval 32 题 × T0 × 4 conditions = 128 outputs
|
||||
GSM8K 32 题 × T0 × 4 conditions = 128 outputs
|
||||
|
||||
随机带敏感性：
|
||||
每域 4 道固定题 × T0–T3 × 4 conditions
|
||||
其中 T1–T3 新增 96 outputs
|
||||
|
||||
正式总计：
|
||||
64 tasks × 4 conditions on T0
|
||||
+ 8 diagnostic tasks × 3 extra tapes × 4 conditions
|
||||
= 352 outputs
|
||||
```
|
||||
|
||||
四个条件不再只是“同 seed 对齐”，而是在同一道题、同一随机带、同一个生成步显式读取
|
||||
完全相同的均匀数 `u_t`。`.3/.95` 的温度和 nucleus 分布来自固定 checkpoint 与
|
||||
Transformers warper，离散取样则由本协议定义的 token-ID 顺序逆 CDF 完成。
|
||||
|
||||
最稳的观察不是 correctness，而是**输出长度具有明显的 domain × prompt 条件交互**：
|
||||
|
||||
- Code：在 period 边界下，打开 system 后平均缩短 `130.875` tokens；当前 32 题、T0
|
||||
selected-task resampling band 为 `[-178.688, -83.218]`；
|
||||
- Math：同一对比反而平均延长 `25.125` tokens；对应 band 为
|
||||
`[7.874, 44.531]`；
|
||||
- EOS 边界下的 system 长度差在两个 domain 都没有得到同样清楚的非零带；
|
||||
- 所有四个预注册 correctness contrast 的 selected-task band 在 Code 和 Math 中都跨
|
||||
0，不能写成稳定能力提升或下降；
|
||||
- 64 条独立跨进程 replay 在 12 个冻结字段上全部 `64 / 64 exact`。
|
||||
|
||||
因此，本轮最合理的结论是：
|
||||
|
||||
> 同一个 system 文本的生成长度效应并不是孤立、固定方向的“system 开关效应”；它依赖
|
||||
> 历史边界 token，并且在 Code 与 Math 两类任务上方向相反。correctness 的方向则仍有
|
||||
> 明显任务与随机带不确定性。
|
||||
|
||||
---
|
||||
|
||||
## 1. 为什么 Round 07 还不够
|
||||
|
||||
Round 07 已经避免了最明显的伪样本量错误：它没有把同一道题的四个 seed 当成四道独立
|
||||
题。但每域只有 4 道题：
|
||||
|
||||
```text
|
||||
4 tasks × 4 seeds × 4 conditions
|
||||
```
|
||||
|
||||
这仍有三个缺口：
|
||||
|
||||
1. 4 道题上的方向很容易被单题主导；
|
||||
2. 同一个 batch seed 不等于四行使用同一个随机数；
|
||||
3. 任务差异与 sampling 随机带差异没有被正交地拆开。
|
||||
|
||||
Round 08 因此把预算优先给更多任务，并额外留下一个小型交叉设计：
|
||||
|
||||
```text
|
||||
32 tasks × T0 → 主 selected-task 分析
|
||||
4 tasks × T0–T3 → tape sensitivity
|
||||
```
|
||||
|
||||
`4 tasks × 4 tapes` 是 crossed repeated measures，不是 16 道独立题。
|
||||
|
||||
---
|
||||
|
||||
## 2. 预注册与预运行修订
|
||||
|
||||
### 2.1 先冻结、后生成
|
||||
|
||||
首版协议与 64 题选择清单在任何本轮模型输出产生前提交：
|
||||
|
||||
```text
|
||||
6181af9 research: preregister task bootstrap CRN protocol
|
||||
```
|
||||
|
||||
实现与 256 个 Chat prompt hash 在 smoke 前提交：
|
||||
|
||||
```text
|
||||
a09e6b1 research: freeze Chat prompts and CRN runner
|
||||
```
|
||||
|
||||
评估、重放与 bootstrap 的指标口径已经写入首版协议；对应计算脚本在 formal 运行期间实现
|
||||
并提交：
|
||||
|
||||
```text
|
||||
65a91ae research: add task bootstrap evaluation pipeline
|
||||
```
|
||||
|
||||
脚本实现没有根据 correctness 结果替换题、随机带、contrast、bootstrap seed 或主指标。
|
||||
|
||||
### 2.2 修订 1：固定四行 batch
|
||||
|
||||
最初草案写成四条 trajectory 逐条 forward。实现审查发现这会把同一模型 forward 重复四
|
||||
次，预计增加约四倍 GPU/CPU-offload 时间。任何 smoke 输出之前，合同修订为：
|
||||
|
||||
- 固定四行 batch；
|
||||
- 行序 `s0_eos, s1_eos, s0_period, s1_period`；
|
||||
- 四行第 `t` 步读取同一个 `u_t`；
|
||||
- 某行 EOS 后，后续追加 PAD=EOS、attention mask 为 0；
|
||||
- batch 内不存在跨行 attention；
|
||||
- sampling 不调用 PyTorch RNG。
|
||||
|
||||
这项修订只改变执行效率与完成行的 padding 合同，没有看结果。
|
||||
|
||||
### 2.3 修订 2：纠正 Chat prompt hash
|
||||
|
||||
tokenizer-only 闸门在模型加载前发现，首版 manifest 把路由探针的短输入 hash 错标成了
|
||||
Chat 生成 prompt hash。例如 `HumanEval/31`：
|
||||
|
||||
```text
|
||||
路由探针 s0_eos：48 tokens
|
||||
Chat 生成 s0_eos：145 tokens
|
||||
```
|
||||
|
||||
当时：
|
||||
|
||||
- 本轮模型输出为 0；
|
||||
- checkpoint 尚未加载；
|
||||
- correctness 未计算。
|
||||
|
||||
清单随后：
|
||||
|
||||
1. 保留原字段，并改名为 `routing_probe_prompt_token_ids_sha256`；
|
||||
2. 用正式 Chat renderer 重新产生 64 × 4 = 256 个 hash；
|
||||
3. 写入 `chat_generation_prompt_token_ids_sha256`；
|
||||
4. runner 只核验后者。
|
||||
|
||||
最终：
|
||||
|
||||
| 项目 | SHA-256 |
|
||||
|---|---|
|
||||
| manifest 文件 | `6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1` |
|
||||
| manifest 规范内容 | `d15303e18345f6dec2aaf891ba812c0a0232b86f416f8f685eabbe818249e838` |
|
||||
| 256 Chat prompt 合同 | `7f766be54463c7f513948dd6d63a55e12b85b9220ee177361c743cd9997f6c3b` |
|
||||
|
||||
---
|
||||
|
||||
## 3. source 与执行网格
|
||||
|
||||
### 3.1 source 选择
|
||||
|
||||
两个 domain 都使用前序 routing control 中已冻结的全部 32 条：
|
||||
|
||||
```text
|
||||
sample salt = llm-atlas-deepseek-routing-template-control-v1
|
||||
within_domain_index = 0..31
|
||||
```
|
||||
|
||||
没有按 Round 07 pass、EOS、长度、文本质量或 selection rank 的“好看程度”二次筛选。
|
||||
完整 source ID、source hash、selection rank、四格 prompt hash 与 tape 分配见 manifest。
|
||||
|
||||
### 3.2 主随机带与诊断随机带
|
||||
|
||||
| tape | display seed |
|
||||
|---|---:|
|
||||
| T0 | 2,572,353,518 |
|
||||
| T1 | 399,507,326 |
|
||||
| T2 | 405,284,229 |
|
||||
| T3 | 568,701,915 |
|
||||
|
||||
全部 64 题运行 T0。每域 index `0, 8, 16, 24` 再运行 T1–T3。
|
||||
|
||||
### 3.3 独立 replay 子集
|
||||
|
||||
每域固定：
|
||||
|
||||
```text
|
||||
0, 4, 8, 12, 16, 20, 24, 28
|
||||
```
|
||||
|
||||
只 replay T0 × 四条件，共：
|
||||
|
||||
```text
|
||||
16 tasks × 4 conditions = 64 cells
|
||||
```
|
||||
|
||||
这不是看完 formal 后挑的“容易复现样本”。
|
||||
|
||||
---
|
||||
|
||||
## 4. 显式共同随机数取样器
|
||||
|
||||
### 4.1 `u_t` 的定义
|
||||
|
||||
对同一 `source_id + tape + step`：
|
||||
|
||||
```text
|
||||
H = SHA256(
|
||||
protocol_id + "\0uniform\0"
|
||||
+ tape_label + "\0"
|
||||
+ source_id + "\0"
|
||||
+ decimal(step)
|
||||
)
|
||||
|
||||
z_t = H[0:8] as big-endian uint64
|
||||
u_t = (z_t + 0.5) / 2^64
|
||||
```
|
||||
|
||||
四个 condition 使用完全相同的 `z_t / u_t`。每条 output 都记录：
|
||||
|
||||
-实际消费步数；
|
||||
-消费前缀的规范 SHA-256；
|
||||
-前 8 个 uint64 hex；
|
||||
- prompt、trajectory 与文本 hash。
|
||||
|
||||
离线审计从协议重新派生全部 352 条 output 的 uniform 前缀：
|
||||
|
||||
```text
|
||||
352 / 352 exact
|
||||
```
|
||||
|
||||
### 4.2 概率分布
|
||||
|
||||
每步：
|
||||
|
||||
1. fixed checkpoint 产生 BF16 logits；
|
||||
2. `TemperatureLogitsWarper(0.3)`；
|
||||
3. `TopPLogitsWarper(0.95, min_tokens_to_keep=1)`；
|
||||
4. float32 softmax；
|
||||
5. token ID 顺序 float32 CDF；
|
||||
6. CDF 最后一个值强制为 1；
|
||||
7. `u_t` 转 float32并 clamp 到相邻可表示开区间端点；
|
||||
8. `torch.searchsorted(..., right=False)` 选 token。
|
||||
|
||||
必须区分：
|
||||
|
||||
```text
|
||||
模型与 .3/.95 概率分布：官方固定对象
|
||||
离散 sampler：本协议的显式 inverse CDF
|
||||
```
|
||||
|
||||
本轮没有调用：
|
||||
|
||||
```text
|
||||
transformers.generate
|
||||
torch.multinomial
|
||||
```
|
||||
|
||||
所以不能把 trajectory 写成“官方 Transformers sampler 的 exact 输出”。
|
||||
|
||||
### 4.3 共同随机数不等于共同 token
|
||||
|
||||
共享的是概率积分尺度上的 `u_t`。prompt 条件改变 logits 后，同一个 `u_t` 会落入不同
|
||||
token 的 CDF 区间。
|
||||
|
||||
这正是想要的配对：
|
||||
|
||||
```text
|
||||
随机冲击相同
|
||||
条件分布不同
|
||||
→ token 可以不同
|
||||
```
|
||||
|
||||
八个 domain × contrast 的主分析中：
|
||||
|
||||
```text
|
||||
32 sources × 8 domain-contrasts
|
||||
= 256 / 256 shared-uniform-prefix audits exact
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. smoke 闸门
|
||||
|
||||
smoke：
|
||||
|
||||
```text
|
||||
每域 index 0,8,16,24
|
||||
× T0/T1
|
||||
× 4 conditions
|
||||
× 16-token cap
|
||||
= 64 short outputs
|
||||
```
|
||||
|
||||
结果：
|
||||
|
||||
| 闸门 | 结果 |
|
||||
|---|---:|
|
||||
| Chat prompt hash | 256 / 256 exact |
|
||||
| short outputs | 64 / 64 |
|
||||
| 同进程 T0 replay | 32 / 32 exact |
|
||||
| T0/T1 可比格 | 32 |
|
||||
| T0/T1 trajectory 分叉 | 17 / 32 |
|
||||
| RNG state 前后不变 | 16 / 16 runs |
|
||||
| synthetic inverse-CDF test | passed |
|
||||
| OOM / NaN / exception | 0 |
|
||||
|
||||
smoke 文件：
|
||||
|
||||
```text
|
||||
/tmp/deepseek-v2-lite-chat-task-bootstrap-smoke.json
|
||||
SHA-256 c868e74f88e39343f95524b4b38d0c1503f727a22d4dfa3937532514205110b5
|
||||
```
|
||||
|
||||
smoke 不进入正式统计。
|
||||
|
||||
---
|
||||
|
||||
## 6. 正式执行账
|
||||
|
||||
### 6.1 总数
|
||||
|
||||
| 项目 | 数值 |
|
||||
|---|---:|
|
||||
| tasks | 64 |
|
||||
| source × tape runs | 88 |
|
||||
| outputs | 352 |
|
||||
| T0 主 outputs | 256 |
|
||||
| T1–T3 新增诊断 outputs | 96 |
|
||||
| natural EOS | 343 |
|
||||
| budget truncated | 9 |
|
||||
| unique trajectory hashes | 320 |
|
||||
| RNG state 前后不变 | 88 / 88 runs |
|
||||
|
||||
### 6.2 分 domain 的全部 formal outputs
|
||||
|
||||
这里含诊断题 T1–T3，只用于执行账，不是主通过率。
|
||||
|
||||
| domain | outputs | natural EOS | truncated | mean tokens |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Code | 176 | 168 | 8 | 277.278 |
|
||||
| Math | 176 | 175 | 1 | 183.188 |
|
||||
|
||||
### 6.3 资源
|
||||
|
||||
| 项目 | 数值 |
|
||||
|---|---:|
|
||||
| checkpoint load | 8.344 s |
|
||||
| formal generation sum | 4,336.699 s |
|
||||
| peak CUDA allocated | 29,919,644,672 bytes |
|
||||
| peak CUDA allocated | 27.864 GiB |
|
||||
| GPU | NVIDIA GeForce RTX 5090 |
|
||||
| dtype | BF16 |
|
||||
| CUDA resident | embedding + layers 0–23 |
|
||||
| CPU offload | layers 24–26 + norm + lm_head |
|
||||
|
||||
这不是 serving benchmark。CPU-offloaded eager latency 不能外推成吞吐。
|
||||
|
||||
---
|
||||
|
||||
## 7. 独立 evaluator
|
||||
|
||||
### 7.1 HumanEval sandbox
|
||||
|
||||
固定：
|
||||
|
||||
```text
|
||||
python:3.11-alpine
|
||||
@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4
|
||||
```
|
||||
|
||||
约束：
|
||||
|
||||
- network none；
|
||||
- read-only root filesystem；
|
||||
- user `65534:65534`；
|
||||
- drop ALL capabilities；
|
||||
- no-new-privileges；
|
||||
- 256 MiB memory / swap；
|
||||
- 64 PIDs；
|
||||
- 0.5 CPU；
|
||||
- 5 秒 timeout；
|
||||
- host mounts 0。
|
||||
|
||||
### 7.2 T0 主评估总账
|
||||
|
||||
| domain | outputs | fixed-budget pass | strict pass | natural EOS | truncated | mean tokens |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Code | 128 | 59 | 59 | 122 | 6 | 284.227 |
|
||||
| Math | 128 | 71 | 71 | 127 | 1 | 185.969 |
|
||||
|
||||
不要把 `59 / 128` 或 `71 / 128` 当成标准 benchmark 分数。分母是：
|
||||
|
||||
```text
|
||||
32 frozen tasks × 4 prompt conditions
|
||||
```
|
||||
|
||||
同一道题在四条件中重复出现。
|
||||
|
||||
### 7.3 失败分类
|
||||
|
||||
HumanEval T0：
|
||||
|
||||
| outcome | count |
|
||||
|---|---:|
|
||||
| passed | 59 |
|
||||
| assertion failed | 47 |
|
||||
| runtime error | 16 |
|
||||
| timeout | 3 |
|
||||
| extract failed | 2 |
|
||||
| syntax error | 1 |
|
||||
|
||||
GSM8K T0：
|
||||
|
||||
| outcome | count |
|
||||
|---|---:|
|
||||
| numeric exact | 71 |
|
||||
| wrong numeric answer | 57 |
|
||||
| no numeric answer | 0 |
|
||||
|
||||
通过 HumanEval tests 只是功能证据，不是安全证据。
|
||||
|
||||
---
|
||||
|
||||
## 8. T0 四条件：先看原始格
|
||||
|
||||
### 8.1 Code
|
||||
|
||||
| condition | pass / 32 | natural EOS / 32 | mean tokens |
|
||||
|---|---:|---:|---:|
|
||||
| `s0_eos` | 16 | 30 | 330.094 |
|
||||
| `s1_eos` | 16 | 30 | 308.938 |
|
||||
| `s0_period` | 15 | 31 | 314.375 |
|
||||
| `s1_period` | 12 | 31 | 183.500 |
|
||||
|
||||
### 8.2 Math
|
||||
|
||||
| condition | pass / 32 | natural EOS / 32 | mean tokens |
|
||||
|---|---:|---:|---:|
|
||||
| `s0_eos` | 19 | 31 | 181.406 |
|
||||
| `s1_eos` | 19 | 32 | 190.969 |
|
||||
| `s0_period` | 17 | 32 | 173.188 |
|
||||
| `s1_period` | 16 | 32 | 198.313 |
|
||||
|
||||
同样的 `s1_period`，在 Code 中最短，在 Math 中却最长。这比一个全域平均值更重要。
|
||||
|
||||
---
|
||||
|
||||
## 9. selected-task bootstrap
|
||||
|
||||
### 9.1 它是什么
|
||||
|
||||
每个 domain：
|
||||
|
||||
-固定 T0；
|
||||
- 32 题有放回重采样 32 题；
|
||||
-四条件保持题级配对；
|
||||
- 10,000 次；
|
||||
- `numpy.random.default_rng`；
|
||||
- seed `1,364,512,825`；
|
||||
-报告 mean contrast 的 percentile 2.5 / 50 / 97.5%。
|
||||
|
||||
准确标签：
|
||||
|
||||
> 当前冻结 32 题、固定 T0 的 selected-task resampling band。
|
||||
|
||||
它不是：
|
||||
|
||||
-完整 benchmark population CI；
|
||||
-模型能力 CI；
|
||||
- generation-seed uncertainty CI；
|
||||
-因果效应 CI。
|
||||
|
||||
### 9.2 correctness：所有带都跨 0
|
||||
|
||||
右减左，单位为每题成功率：
|
||||
|
||||
| domain | contrast | point | selected-task band |
|
||||
|---|---|---:|---:|
|
||||
| Code | period at s0 | -0.0313 | [-0.1563, 0.0938] |
|
||||
| Code | period at s1 | -0.1250 | [-0.3438, 0.0938] |
|
||||
| Code | system at EOS | 0.0000 | [-0.1250, 0.1250] |
|
||||
| Code | system at period | -0.0938 | [-0.2500, 0.0625] |
|
||||
| Math | period at s0 | -0.0625 | [-0.2188, 0.0938] |
|
||||
| Math | period at s1 | -0.0938 | [-0.2500, 0.0625] |
|
||||
| Math | system at EOS | 0.0000 | [-0.1563, 0.1563] |
|
||||
| Math | system at period | -0.0313 | [-0.1563, 0.0938] |
|
||||
|
||||
点估计里 period 条件经常更低，但任务重采样带都覆盖 0。不能把点估计写成稳定能力损失。
|
||||
|
||||
### 9.3 correctness 转移表
|
||||
|
||||
`right` 相对 `left`：
|
||||
|
||||
| domain | contrast | fail→pass | pass→fail | both pass | both fail |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| Code | period at s0 | 2 | 3 | 13 | 14 |
|
||||
| Code | period at s1 | 4 | 8 | 8 | 12 |
|
||||
| Code | system at EOS | 2 | 2 | 14 | 14 |
|
||||
| Code | system at period | 2 | 5 | 10 | 15 |
|
||||
| Math | period at s0 | 3 | 5 | 14 | 10 |
|
||||
| Math | period at s1 | 2 | 5 | 14 | 11 |
|
||||
| Math | system at EOS | 3 | 3 | 16 | 10 |
|
||||
| Math | system at period | 2 | 3 | 14 | 13 |
|
||||
|
||||
即使平均差是 0，也可能同时有 fail→pass 与 pass→fail。例如 system at EOS 在两个 domain
|
||||
都恰好净 0，但不是每题完全不变。
|
||||
|
||||
### 9.4 输出长度
|
||||
|
||||
右减左，单位 tokens：
|
||||
|
||||
| domain | contrast | point | selected-task band |
|
||||
|---|---|---:|---:|
|
||||
| Code | period at s0 | -15.719 | [-57.627, 24.938] |
|
||||
| Code | period at s1 | -125.438 | [-165.031, -85.125] |
|
||||
| Code | system at EOS | -21.156 | [-47.846, 2.438] |
|
||||
| Code | system at period | -130.875 | [-178.688, -83.218] |
|
||||
| Math | period at s0 | -8.219 | [-27.563, 8.688] |
|
||||
| Math | period at s1 | 7.344 | [-9.407, 26.094] |
|
||||
| Math | system at EOS | 9.563 | [-4.938, 25.188] |
|
||||
| Math | system at period | 25.125 | [7.874, 44.531] |
|
||||
|
||||
两个清楚不跨 0 的 system-at-period 长度带方向相反：
|
||||
|
||||
```text
|
||||
Code → system on 更短
|
||||
Math → system on 更长
|
||||
```
|
||||
|
||||
这是一条比“system prompt 会让输出变短/变长”更精确的结论：
|
||||
|
||||
> system 的长度关联依赖 boundary 与 domain。
|
||||
|
||||
### 9.5 EOS
|
||||
|
||||
EOS contrast 很小：
|
||||
|
||||
- Code：四个 contrast 的 point 在 `0` 或 `+1/32`；
|
||||
- Math：同样只在 `0` 或 `+1/32`；
|
||||
-大部分 source 的 EOS 状态不变。
|
||||
|
||||
因此，Code 中 `-130.875` token 的 system-at-period 差异主要不是简单的“更多题命中 EOS”
|
||||
造成，而是在都自然结束的 trajectory 内结束得更早。
|
||||
|
||||
---
|
||||
|
||||
## 10. trajectory 与 first divergence
|
||||
|
||||
### 10.1 共同随机数审计
|
||||
|
||||
| domain | contrast | uniform exact | exact full trajectory | mean common-prefix tokens |
|
||||
|---|---|---:|---:|---:|
|
||||
| Code | period at s0 | 32 / 32 | 0 | 13.25 |
|
||||
| Code | period at s1 | 32 / 32 | 0 | 3.69 |
|
||||
| Code | system at EOS | 32 / 32 | 0 | 23.91 |
|
||||
| Code | system at period | 32 / 32 | 1 | 11.88 |
|
||||
| Math | period at s0 | 32 / 32 | 6 | 52.53 |
|
||||
| Math | period at s1 | 32 / 32 | 1 | 22.91 |
|
||||
| Math | system at EOS | 32 / 32 | 6 | 59.44 |
|
||||
| Math | system at period | 32 / 32 | 1 | 27.75 |
|
||||
|
||||
同一个 `u_t` 没有让四格“黏”成相同 token。特别是 Code 的 `s1` period contrast 平均只
|
||||
共享约 3.7 个前缀 token 就分叉。
|
||||
|
||||
Math 的共同前缀更长、exact trajectory 更多，说明 prompt 条件的作用方式也有 domain
|
||||
差异。
|
||||
|
||||
---
|
||||
|
||||
## 11. 四随机带诊断
|
||||
|
||||
每域固定四题，不外推为 32 题总体。
|
||||
|
||||
### 11.1 correctness contrast 的 tape means
|
||||
|
||||
右减左；每个值是 4 道诊断题的均值：
|
||||
|
||||
#### Code
|
||||
|
||||
| contrast | T0 | T1 | T2 | T3 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| period at s0 | 0.00 | 0.00 | 0.00 | 0.00 |
|
||||
| period at s1 | -0.25 | 0.00 | 0.00 | 0.00 |
|
||||
| system at EOS | 0.00 | 0.00 | 0.00 | 0.00 |
|
||||
| system at period | -0.25 | 0.00 | 0.00 | 0.00 |
|
||||
|
||||
Code 的四题诊断子集中，T0 上的两个 `-0.25` 没有在 T1–T3 重现。它支持谨慎解释主
|
||||
correctness 点估计。
|
||||
|
||||
#### Math
|
||||
|
||||
| contrast | T0 | T1 | T2 | T3 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| period at s0 | 0.00 | 0.00 | -0.25 | +0.25 |
|
||||
| period at s1 | -0.50 | -0.75 | -0.25 | -0.50 |
|
||||
| system at EOS | 0.00 | +0.50 | -0.25 | +0.25 |
|
||||
| system at period | -0.50 | -0.25 | -0.25 | -0.50 |
|
||||
|
||||
Math 四题上，方向与幅度都明显依赖 tape；四题太少，不能把这些数当成稳定概率。
|
||||
|
||||
### 11.2 长度 contrast 的 tape means
|
||||
|
||||
#### Code
|
||||
|
||||
| contrast | T0 | T1 | T2 | T3 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| period at s0 | -14.50 | +44.50 | +13.50 | -16.75 |
|
||||
| period at s1 | -108.50 | -155.00 | -21.00 | -163.25 |
|
||||
| system at EOS | -59.00 | +6.00 | -90.00 | -39.50 |
|
||||
| system at period | -153.00 | -193.50 | -124.50 | -186.00 |
|
||||
|
||||
Code 的 system-at-period 在四条 tape 上都明显为负；period-at-s1 也都为负，但 T2 幅度
|
||||
较小。
|
||||
|
||||
#### Math
|
||||
|
||||
| contrast | T0 | T1 | T2 | T3 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| period at s0 | -5.25 | +31.25 | -10.00 | +4.00 |
|
||||
| period at s1 | +0.50 | +0.25 | +3.25 | -2.50 |
|
||||
| system at EOS | -3.00 | +54.75 | -7.50 | +32.75 |
|
||||
| system at period | +2.75 | +23.75 | +5.75 | +26.25 |
|
||||
|
||||
Math 四题上的 system-at-period 四条 tape 都为正，但幅度从 `+2.75` 到 `+26.25`。
|
||||
|
||||
诊断支持长度交互的方向，但仍只覆盖 4 道题 × 4 条 tape。
|
||||
|
||||
---
|
||||
|
||||
## 12. 独立跨进程 replay
|
||||
|
||||
重放结果：
|
||||
|
||||
```text
|
||||
16 sources × T0 × 4 conditions = 64 cells
|
||||
```
|
||||
|
||||
| 字段 | exact |
|
||||
|---|---:|
|
||||
| run seed | 64 / 64 |
|
||||
| prompt hash | 64 / 64 |
|
||||
| full run uniform hash | 64 / 64 |
|
||||
| output uniform-prefix hash | 64 / 64 |
|
||||
| uniform steps | 64 / 64 |
|
||||
| generated token IDs | 64 / 64 |
|
||||
| decoded text | 64 / 64 |
|
||||
| EOS state | 64 / 64 |
|
||||
| truncation state | 64 / 64 |
|
||||
| CPU RNG pre-state | 64 / 64 |
|
||||
| CUDA RNG pre-state | 64 / 64 |
|
||||
| RNG unchanged flag | 64 / 64 |
|
||||
|
||||
总闸门：
|
||||
|
||||
```text
|
||||
64 / 64 all preregistered fields exact
|
||||
```
|
||||
|
||||
这证明固定硬件/软件/精度/执行合同下的逐 token 可重放，不证明换 GPU、PyTorch、
|
||||
Transformers、kernel 或 dtype 后仍 exact。
|
||||
|
||||
---
|
||||
|
||||
## 13. 已知偏离与限制
|
||||
|
||||
### 13.1 gold 文件打开时机偏离
|
||||
|
||||
runner 复用了前序 `behavior.task_score` 路径：
|
||||
|
||||
1. 在模型加载前读取 HumanEval / GSM8K gold；
|
||||
2. 每条文本生成结束后附加一个 narrow `task_score`；
|
||||
3. authoritative evaluator 在全部 formal 输出冻结后另行运行。
|
||||
|
||||
这不满足协议中最严格的“gold 只在生成进程结束后打开”表述。
|
||||
|
||||
为什么它不改变 trajectory：
|
||||
|
||||
- gold 不进入 message / prompt；
|
||||
- gold 不进入 tokenizer 输入；
|
||||
- gold 不进入 logits、warper、CDF 或 `u_t`；
|
||||
- gold 不改变 source、tape、max tokens 或 stop；
|
||||
- `task_score` 在 token 序列生成结束后才调用；
|
||||
-独立 evaluator 的 352 行结果不读取 runner 的 `task_score`；
|
||||
- 64-cell replay exact。
|
||||
|
||||
因此它是**真实的流程隔离偏离**，但没有已知的生成因果路径。后续 runner 应删除
|
||||
`task_score`，让生成进程完全不打开 gold。
|
||||
|
||||
### 13.2 不是完整 benchmark
|
||||
|
||||
HumanEval 只使用冻结 32 / 164 tasks；GSM8K 只使用冻结 32 / 1,319 test tasks。source
|
||||
selection 是前序 SHA 排序合同，不是完整 benchmark。
|
||||
|
||||
### 13.3 一条主随机带
|
||||
|
||||
32 题主分析只用 T0。selected-task band 不包含 generation-tape uncertainty。
|
||||
|
||||
### 13.4 多随机带只有四题
|
||||
|
||||
T0–T3 诊断只覆盖每域四题。它揭示 sensitivity，但不能稳定估计 tape 方差。
|
||||
|
||||
### 13.5 counterfactual 不可部署
|
||||
|
||||
period cell 是把一个官方 EOS boundary ID 改为普通句点 ID。它不是官方有效聊天格式。
|
||||
|
||||
### 13.6 成功不是安全
|
||||
|
||||
HumanEval pass 只说明固定 tests 通过。它不说明：
|
||||
|
||||
-安全；
|
||||
-鲁棒；
|
||||
-无未测 bug；
|
||||
-可部署。
|
||||
|
||||
---
|
||||
|
||||
## 14. 证据文件与 SHA-256
|
||||
|
||||
| 文件 | SHA-256 |
|
||||
|---|---|
|
||||
| formal sampling | `ea0607f2b197fac3f794655c1538ce1f9a1cb072d637eb31d35573682e311809` |
|
||||
| independent evaluation | `82b2fc5d1f854a7e0cd7aba7c70ff74d733d24221522a4f92b962478c76177ab` |
|
||||
| replay generation | `6519947e2fa4327c1ba2cc506b0861f172a6bdbd4f2d6787447edaf8fbcac508` |
|
||||
| reproduction comparison | `63ed39e5dcdbc2a30e516172f3657dfa453ff3b23d5bd5c49c734939242a70f5` |
|
||||
| bootstrap analysis | `9ab17561ced930a668c082141f7c6e013cbda70e42b09de63d41f1b82c01a6ae` |
|
||||
| compact website data | `082e3c56373e5f91d51d88b1430d14ad3b68b0f200e80901171c2fb62a1e57bf` |
|
||||
| frozen manifest | `6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1` |
|
||||
|
||||
离线一致性检查：
|
||||
|
||||
```bash
|
||||
node scripts/check-deepseek-chat-task-bootstrap-crn-data.mjs
|
||||
```
|
||||
|
||||
通过：
|
||||
|
||||
```text
|
||||
formal sources 64
|
||||
formal runs 88
|
||||
formal outputs 352
|
||||
prompt hashes exact 256
|
||||
uniform output hashes exact 352
|
||||
RNG unchanged runs 88
|
||||
evaluation rows 352
|
||||
reproduction exact cells 64 / 64
|
||||
CRN contrast checks 256
|
||||
bootstrap resamples 10,000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15. 最终结论该怎么写
|
||||
|
||||
### 可以写
|
||||
|
||||
1. 在当前固定 32 道 HumanEval、T0 下，system-at-period 的平均输出长度比 system-off
|
||||
短约 131 tokens，selected-task band 不跨 0；
|
||||
2. 在当前固定 32 道 GSM8K、T0 下，同一 contrast 平均长约 25 tokens，band 方向相反；
|
||||
3. 两个 domain 的 correctness contrast bands 都跨 0；
|
||||
4. 每域四题的 T0–T3 诊断显示 correctness 对 tape 敏感，长度交互的方向相对更一致；
|
||||
5. 显式 CRN 与 64-cell 跨进程 replay 都通过 exact audit。
|
||||
|
||||
### 不可以写
|
||||
|
||||
1. “system prompt 普遍让代码更短、数学更长”；
|
||||
2. “period 会降低 DeepSeek 的总体能力”；
|
||||
3. “这些是完整 HumanEval / GSM8K 分数”；
|
||||
4. “selected-task band 是模型总体 95% CI”；
|
||||
5. “period 是官方聊天格式”；
|
||||
6. “通过 HumanEval tests 就安全”；
|
||||
7. “本协议 trajectory 等于官方 `torch.multinomial` trajectory”。
|
||||
|
||||
本轮真正增加的不是一个更大的数字，而是一条更干净的推理链：
|
||||
|
||||
```text
|
||||
固定任务
|
||||
→ 固定概率分布
|
||||
→ 显式共享随机冲击
|
||||
→ 条件内逐题配对
|
||||
→ 任务重采样
|
||||
→ 独立随机带诊断
|
||||
→ 跨进程逐 token 重放
|
||||
```
|
||||
|
||||
它让“prompt 边界如何改变生成行为”从单题现象，前进到一个仍有限、但可审计的任务级机制
|
||||
实验。
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,396 @@
|
||||
# DeepSeek-V2-Lite-Chat 任务 bootstrap 与显式共同随机数协议
|
||||
|
||||
> 状态：**已预注册，尚未运行本协议的任何生成输出**
|
||||
>
|
||||
> 预运行修订 1：实现审查后把“四条 trajectory 逐条执行”改为固定四行 batch；取样仍由
|
||||
> 每步显式共享的 `u_t` 完成。修订发生在任何 smoke / 正式输出之前，只为避免四倍重复
|
||||
> forward；finished row 的后续 PAD=EOS 与 mask=0 合同同时冻结在下文。
|
||||
>
|
||||
> 预运行修订 2：tokenizer-only 闸门发现首版 manifest 误把路由探针的短输入 hash 标成
|
||||
> Chat 生成 prompt hash。此时模型尚未加载、协议输出仍为 0。清单保留路由 hash，并新增
|
||||
> 由正式 Chat renderer 产生的 256 个 prompt hash；runner 只核验后者。
|
||||
>
|
||||
> 注册日期：2026-07-30
|
||||
>
|
||||
> 协议 ID：`llm-atlas-deepseek-chat-task-bootstrap-crn-v1`
|
||||
>
|
||||
> 冻结清单：
|
||||
> `research/DEEPSEEK_V2_LITE_CHAT_TASK_BOOTSTRAP_MANIFEST.json`
|
||||
>
|
||||
> 清单文件 SHA-256：
|
||||
> `6313e70536c464fe598a93035576752418f08016dfd60ac246437c3b43bf2ae1`
|
||||
>
|
||||
> 清单规范内容 SHA-256：
|
||||
> `d15303e18345f6dec2aaf891ba812c0a0232b86f416f8f685eabbe818249e838`
|
||||
>
|
||||
> 256 个 Chat prompt 合同 hash：
|
||||
> `7f766be54463c7f513948dd6d63a55e12b85b9220ee177361c743cd9997f6c3b`
|
||||
|
||||
## 0. 一句话说明这一轮
|
||||
|
||||
Round 07 用 4 道 HumanEval、4 道 GSM8K 和每题 4 个 seed 回答了“换几道题后，现象还在
|
||||
不在”。但 4 道题仍太少，而且把同题多个随机输出当成多道题会制造伪样本量。
|
||||
|
||||
Round 08 改为：
|
||||
|
||||
```text
|
||||
主分析：
|
||||
32 HumanEval × 1 冻结随机带 × 4 条件 = 128 outputs
|
||||
32 GSM8K × 1 冻结随机带 × 4 条件 = 128 outputs
|
||||
|
||||
随机带敏感性诊断：
|
||||
每域 4 道预选题 × 额外 3 条随机带 × 4 条件 = 48 outputs / domain
|
||||
|
||||
正式总计：
|
||||
256 + 96 = 352 outputs
|
||||
```
|
||||
|
||||
它第一次允许在当前预选的 32 道题内做**按题重采样**，并且让同题四个条件在第 `t` 个
|
||||
生成步真正读取同一个均匀随机数。它仍然不是完整 HumanEval / GSM8K benchmark，不估计
|
||||
模型总体能力，也不把 counterfactual `period` 序列称为官方聊天格式。
|
||||
|
||||
---
|
||||
|
||||
## 1. 研究问题
|
||||
|
||||
### 1.1 主要问题
|
||||
|
||||
HumanEval 与 GSM8K 分开回答：
|
||||
|
||||
1. 在固定 T0 随机带下，`period - EOS` 是否改变选定任务上的通过率、自然 EOS 率和输出
|
||||
长度？
|
||||
2. `system on - off` 是否在 EOS 与 period 边界下方向一致？
|
||||
3. 任务级差异是否足以让 Round 07 的 4 题方向翻转？
|
||||
4. 同一均匀随机带下，四个条件从哪一步开始分叉；分叉前是否逐步读取完全相同的
|
||||
`u_t`？
|
||||
|
||||
### 1.2 诊断问题
|
||||
|
||||
每域固定 index `0, 8, 16, 24` 四题，额外运行 T1、T2、T3：
|
||||
|
||||
1. T0 上的任务级差异相对随机带差异有多大？
|
||||
2. 主要 contrast 的方向是否依赖单条随机带？
|
||||
3. 对同一道题，四条随机带带来的成功/失败与长度变化有多大？
|
||||
|
||||
这些是敏感性诊断，不与 32 题 T0 主分析合并成一个更大的“样本”。
|
||||
|
||||
---
|
||||
|
||||
## 2. 模型、软件与提示合同
|
||||
|
||||
冻结：
|
||||
|
||||
- checkpoint：`deepseek-ai/DeepSeek-V2-Lite-Chat`；
|
||||
- revision：`85864749cd611b4353ce1decdb286193298f64c7`；
|
||||
- checkpoint 身份：SFT Chat；
|
||||
- Transformers `4.41.2`；
|
||||
- PyTorch `2.11.0+cu128`；
|
||||
- BF16、官方 remote modeling code、eager attention；
|
||||
- embedding + layers 0–23 放 CUDA；
|
||||
- layers 24–26、final norm 与 `lm_head` 放 CPU；
|
||||
- `device_map=auto`，GPU budget `28GiB`，CPU budget `80GiB`；
|
||||
- `use_cache=true`，每条 trajectory 的 `max_new_tokens=512`；
|
||||
- BOS `100000`，EOS `100001`，PAD alias EOS。
|
||||
|
||||
固定四条件与行序：
|
||||
|
||||
```text
|
||||
s0_eos, s1_eos, s0_period, s1_period
|
||||
```
|
||||
|
||||
- `s0/s1`：system off/on；
|
||||
- `eos`：官方历史 assistant 边界；
|
||||
- `period`：把该位置的单个 EOS ID 换成 ordinary period ID `13`；
|
||||
- 其余 prompt token 与前序冻结模板相同；
|
||||
- 每条 prompt token hash 必须与冻结清单逐格 exact。
|
||||
|
||||
正式生成使用固定四行 batch，行序即上述条件序。batch 内没有跨行 attention；取样不调用
|
||||
PyTorch RNG。同一行结束后，后续步追加 PAD=EOS 且该位置 attention mask 为 0，其他仍活跃
|
||||
行继续 forward。这个合同避免四倍重复 forward，也把执行形状固定下来。
|
||||
|
||||
---
|
||||
|
||||
## 3. 任务如何在结果前冻结
|
||||
|
||||
输入来源：
|
||||
|
||||
```text
|
||||
src/data/deepseek-v2-lite-routing-special-token-family-control.json
|
||||
SHA-256 c372c1b03a8b15f615b54ded5d9257a8fc2cdb7728001735d3d4c1d8534af5bf
|
||||
sample salt llm-atlas-deepseek-routing-template-control-v1
|
||||
```
|
||||
|
||||
每个领域使用其中按 `within_domain_index=0..31` 已冻结的全部 32 条，不根据 Round 07
|
||||
正确率、输出长度、EOS 或文本质量二次筛选。完整 source ID、selection rank、源文本 hash、
|
||||
四格 prompt hash、随机带分配和重放分配均在冻结清单内。
|
||||
|
||||
### 3.1 HumanEval
|
||||
|
||||
```text
|
||||
00 HumanEval/31 01 HumanEval/44 02 HumanEval/133 03 HumanEval/23
|
||||
04 HumanEval/147 05 HumanEval/34 06 HumanEval/123 07 HumanEval/77
|
||||
08 HumanEval/22 09 HumanEval/127 10 HumanEval/37 11 HumanEval/48
|
||||
12 HumanEval/158 13 HumanEval/70 14 HumanEval/89 15 HumanEval/9
|
||||
16 HumanEval/53 17 HumanEval/145 18 HumanEval/35 19 HumanEval/103
|
||||
20 HumanEval/25 21 HumanEval/98 22 HumanEval/115 23 HumanEval/10
|
||||
24 HumanEval/141 25 HumanEval/91 26 HumanEval/122 27 HumanEval/75
|
||||
28 HumanEval/134 29 HumanEval/142 30 HumanEval/149 31 HumanEval/102
|
||||
```
|
||||
|
||||
### 3.2 GSM8K
|
||||
|
||||
```text
|
||||
00 1069 01 1228 02 0144 03 1251 04 0676 05 0126 06 0438 07 1299
|
||||
08 0546 09 0537 10 0226 11 1128 12 0760 13 1173 14 0452 15 0357
|
||||
16 1050 17 0951 18 0906 19 0092 20 0372 21 0361 22 0733 23 0687
|
||||
24 0584 25 1152 26 1021 27 1194 28 0399 29 0214 30 1084 31 0964
|
||||
```
|
||||
|
||||
GSM8K 行均指 `gsm8k/test/NNNN`。
|
||||
|
||||
gold 只进入生成后的 evaluator：
|
||||
|
||||
| 输入 | SHA-256 |
|
||||
|---|---|
|
||||
| HumanEval JSONL.gz | `b796127e635a67f93fb35c04f4cb03cf06f38c8072ee7cee8833d7bee06979ef` |
|
||||
| GSM8K test JSONL | `3730d312f6e3440559ace48831e51066acaca737f6eabec99bccb9e4b3c39d14` |
|
||||
|
||||
---
|
||||
|
||||
## 4. 真正的共同随机数如何构造
|
||||
|
||||
### 4.1 为什么不再只重置 seed
|
||||
|
||||
Transformers `generate()` 的常规 sampling 最终调用 `torch.multinomial`。即使四行共享 batch
|
||||
seed，每行也消费不同 RNG 子流；如果逐行重置同 seed，分布内部的实现细节仍不等于我们
|
||||
明确定义并能逐步审计的共同随机数合同。
|
||||
|
||||
本轮不调用 `torch.multinomial`，也不把输出称作“Transformers 官方 sampler 的 exact
|
||||
trajectory”。模型、logits 和官方 `.3/.95` 解码分布仍来自固定 checkpoint；离散取样器改为
|
||||
本协议定义的**显式均匀数 + token-ID 顺序逆 CDF**。
|
||||
|
||||
### 4.2 冻结随机带
|
||||
|
||||
四条带 T0–T3 的 display seed：
|
||||
|
||||
| 带 | display seed | 派生字符串 SHA-256 |
|
||||
|---|---:|---|
|
||||
| T0 | 2,572,353,518 | `9952ffeec840649d2654d35b9a86c97db3737b70826c366e3f5a7633ec26d0c4` |
|
||||
| T1 | 399,507,326 | `17cfff7e5b3b6d011f4d1b8076637d4bdfb0ebd8a6021beb2a0011a8f6d520c6` |
|
||||
| T2 | 405,284,229 | `1828258525ea1b450d6de0ba0d9ff12f737b048b1cef554d7dd2ba90d25e5f4d` |
|
||||
| T3 | 568,701,915 | `21e5b3dbe506f1b7e9d483318195fc5cb4990c518850a52d40b5eff225491bd2` |
|
||||
|
||||
对 `tape_label, source_id, step`：
|
||||
|
||||
```text
|
||||
H = SHA256(
|
||||
protocol_id + "\0uniform\0"
|
||||
+ tape_label + "\0"
|
||||
+ source_id + "\0"
|
||||
+ decimal(step)
|
||||
)
|
||||
|
||||
z_t = H[0:8]，big-endian unsigned uint64
|
||||
u_t = (z_t + 0.5) / 2^64
|
||||
```
|
||||
|
||||
`u_t` 位于开区间 `(0,1)`。同一 source + tape 的四个条件在相同生成步 `t` 使用完全相同
|
||||
的 `z_t`。某条件 EOS 后停止，不再产生“配对输出”；其他仍存活条件继续读取下一步相同
|
||||
定义的随机带。
|
||||
|
||||
### 4.3 每一步的概率与取样
|
||||
|
||||
对模型最后位置 logits：
|
||||
|
||||
1. 用 Transformers `TemperatureLogitsWarper(0.3)`；
|
||||
2. 用 Transformers `TopPLogitsWarper(0.95, min_tokens_to_keep=1)`；
|
||||
3. 以 `torch.float32` 做 softmax；
|
||||
4. 按 token ID `0..vocab-1` 做 `torch.float32` cumulative sum；
|
||||
5. 强制最后一个 CDF 元素为 `1.0`；
|
||||
6. 将 `u_t` cast 为 `torch.float32`；
|
||||
7. float32 转换若落在端点，则 clamp 到
|
||||
`torch.nextafter(0,1)` / `torch.nextafter(1,0)`；
|
||||
8. `torch.searchsorted(cdf, u_t, right=False)` 取得 token ID。
|
||||
|
||||
因此：
|
||||
|
||||
```text
|
||||
共享的是概率积分尺度上的 u_t
|
||||
≠ 强迫四格选择同一 token
|
||||
```
|
||||
|
||||
prompt 改变 logits 后，同一个 `u_t` 可以落入不同 token 区间，这正是共同随机数配对所需
|
||||
的性质。
|
||||
|
||||
每条 output 记录：
|
||||
|
||||
- 前 8 个 `z_t` / `u_t`；
|
||||
-实际消费的 `z_t` 序列规范 hash；
|
||||
- prompt hash、generated token IDs 与 hash；
|
||||
- EOS / truncation；
|
||||
-生成前后 CPU/CUDA RNG state hash；
|
||||
- PyTorch RNG 是否完全未被取样器消费。
|
||||
|
||||
---
|
||||
|
||||
## 5. 执行网格、smoke 与独立重放
|
||||
|
||||
### 5.1 16-token smoke
|
||||
|
||||
每域 index `0,8,16,24`，T0/T1，四条件：
|
||||
|
||||
```text
|
||||
8 sources × 2 tapes × 4 conditions = 64 short outputs
|
||||
```
|
||||
|
||||
通过闸门：
|
||||
|
||||
1. 正式 64 题 × 4 条件的 256 个 prompt hash 全部与清单 exact；
|
||||
2. 64 条短输出无 OOM、NaN、exception；
|
||||
3. 同进程重放 8 sources × T0 × 4 = 32 cells 逐 token、文本与 stop state exact；
|
||||
4. T0/T1 至少一个 source-condition trajectory 分叉；
|
||||
5. 同一 source+tape 四条件的前 16 个 `z_t` 定义逐步 exact；
|
||||
6. 生成前后 PyTorch CPU/CUDA RNG state exact，证明 sampling 不消费隐式 RNG；
|
||||
7. synthetic distribution 单元测试覆盖 CDF 边界与被 top-p 过滤 token。
|
||||
|
||||
smoke 不进入正式统计。
|
||||
|
||||
### 5.2 正式运行
|
||||
|
||||
- 全部 64 题跑 T0 × 四条件：256 outputs；
|
||||
- 每域 index `0,8,16,24` 再跑 T1/T2/T3 × 四条件：96 outputs；
|
||||
- 合计 352 outputs；
|
||||
- 每题结束后写独立 journal；再次启动时必须核验协议、source、prompt 与输出内容 hash 后才
|
||||
能 resume；
|
||||
- journal 只用于可恢复执行，正式 JSON 由全部合格 journal 组装。
|
||||
- 四格固定同 batch；finished row 在后续步追加 PAD=EOS、对应 attention mask 为 0。
|
||||
|
||||
### 5.3 跨进程独立重放
|
||||
|
||||
每域 index：
|
||||
|
||||
```text
|
||||
0, 4, 8, 12, 16, 20, 24, 28
|
||||
```
|
||||
|
||||
只跑 T0 × 四条件：
|
||||
|
||||
```text
|
||||
16 sources × 1 tape × 4 conditions = 64 outputs
|
||||
```
|
||||
|
||||
逐格比较 protocol、model revision、generation contract、tape、prompt hash、uniform hash、
|
||||
generated token IDs、decoded text、EOS、truncation 和生成前 RNG state。目标为 64/64
|
||||
全部 exact。
|
||||
|
||||
---
|
||||
|
||||
## 6. evaluator 在输出前冻结
|
||||
|
||||
### 6.1 HumanEval
|
||||
|
||||
沿用前序四本账：
|
||||
|
||||
1. 代码抽取是否成功；
|
||||
2. Python AST 是否可解析；
|
||||
3. sandbox 是否真正执行；
|
||||
4. tests 是否通过。
|
||||
|
||||
固定 sandbox：
|
||||
|
||||
```text
|
||||
python:3.11-alpine
|
||||
@sha256:25976e9d34a0fab1f278cae931f34c8303d97bf0c0d7f85b6b4dcf641d7702a4
|
||||
```
|
||||
|
||||
仍为 network none、read-only root filesystem、非 root、drop ALL capabilities、
|
||||
no-new-privileges、256 MiB、64 PIDs、0.5 CPU、5 秒 timeout。报告失败类型：
|
||||
|
||||
```text
|
||||
extract_failed / syntax_error / timeout / runtime_error / assertion_failed / passed
|
||||
```
|
||||
|
||||
主 correctness 是 fixed-budget tests pass；strict-complete pass 单独报告，不混写。
|
||||
|
||||
### 6.2 GSM8K
|
||||
|
||||
沿用冻结 numeric final-answer evaluator：
|
||||
|
||||
- gold：`####` 后最后一个规范化数字；
|
||||
- prediction：输出中最后一个规范化数字；
|
||||
- fixed-budget numeric exact；
|
||||
- strict-complete numeric exact；
|
||||
- 无数字单独记为 `no_numeric_answer`；
|
||||
- 有数字但不等于 gold 记为 `wrong_numeric_answer`。
|
||||
|
||||
---
|
||||
|
||||
## 7. 统计合同
|
||||
|
||||
### 7.1 主要 contrast
|
||||
|
||||
在每个 domain 内、每道题、T0 上计算右减左：
|
||||
|
||||
```text
|
||||
period_at_s0 = s0_period - s0_eos
|
||||
period_at_s1 = s1_period - s1_eos
|
||||
system_at_eos = s1_eos - s0_eos
|
||||
system_at_period = s1_period - s0_period
|
||||
```
|
||||
|
||||
success 用 `{-1,0,+1}` 任务级转移；长度与 EOS 保留配对差。两因子 main effect 与
|
||||
interaction 可以作为 secondary descriptive summary，但不能替代四个预注册 contrast。
|
||||
|
||||
### 7.2 任务 bootstrap
|
||||
|
||||
HumanEval 32 题与 GSM8K 32 题分别进行：
|
||||
|
||||
- 10,000 次；
|
||||
- 每次在当前 domain 的 32 题中有放回抽 32 题；
|
||||
- 同一次 resample 对四条件保持题级配对；
|
||||
- 固定 seed `1,364,512,825`；
|
||||
- 报告 percentile 2.5% / 50% / 97.5%。
|
||||
|
||||
必须写成：
|
||||
|
||||
> 当前冻结 32 题、固定 T0 下的 selected-task resampling band。
|
||||
|
||||
不得写成：
|
||||
|
||||
- 完整 benchmark 总体 95% CI；
|
||||
- 模型真实能力 CI；
|
||||
- 同时覆盖 generation-seed uncertainty 的 CI；
|
||||
- 因果效应 CI。
|
||||
|
||||
### 7.3 多随机带诊断
|
||||
|
||||
对每域固定四题 × T0–T3：
|
||||
|
||||
- 分别报告每条带的 condition success / EOS / length；
|
||||
- 计算题内跨带 range 与带内跨题 range；
|
||||
- 分别做 selected-task 和 selected-tape 描述；
|
||||
- 不把 `4 tasks × 4 tapes = 16` 当成 16 道独立题。
|
||||
|
||||
---
|
||||
|
||||
## 8. 结论边界
|
||||
|
||||
允许：
|
||||
|
||||
- 描述这 32 道预选题、T0 下四个条件的配对差；
|
||||
- 描述任务重采样带与 8 道诊断题上的 tape sensitivity；
|
||||
- 描述显式共同随机数下的 first-divergence、转移表和长度差；
|
||||
- 报告 exact replay 是否通过。
|
||||
|
||||
禁止：
|
||||
|
||||
- 把 period 当成官方可部署格式；
|
||||
- 把 selected-task band 外推为 benchmark / 用户分布总体；
|
||||
- 把一条 tape 的结果写成 sampling robustness；
|
||||
- 把测试通过写成代码安全；
|
||||
- 把同一 `u_t` 错写成“同一 token”；
|
||||
- 把本协议 sampler 写成 Transformers `generate()` / `torch.multinomial` 的 exact
|
||||
trajectory；
|
||||
- 根据正式输出修改 source、tape、contrast、bootstrap seed、sandbox 或主指标。
|
||||
|
||||
任何偏离必须在结果审计中逐项列出；不能静默修订本文件。
|
||||
@@ -438,11 +438,19 @@ src/data/k3-flashkda-runtime.json
|
||||
- [x] 使用 CUDA 13.0 / glibc 2.39 隔离环境产出可校验 `sm_120a` wheel；
|
||||
- [x] RTX 5090 上六组 FlashKDA / official torch reference exact correctness；
|
||||
- [x] K3 fixed / varlen shape、三种 state mode 的 1,800 个 CUDA Event samples；
|
||||
- [x] 在不裁剪 `A_log` 的前提下完成独立缩小 AttnRes 机制实验；
|
||||
- [x] Baseline / Full / Block × 三 seed × 2,000 steps 与指定正式格全新进程 exact replay；
|
||||
- [x] BPC 主结果、Residual RMS、depth weights、gradient counterevidence 与执行成本公开入账；
|
||||
- [ ] 取得真实 hidden-state / router load trace；
|
||||
- [ ] 取得可加载的 reduced checkpoint、官方 trace 或多机资源；
|
||||
- [ ] 对 Figure 3 / 4 / 5 做真实数值重绘；
|
||||
- [ ] 对 AttnRes 读取分布做真实 token / layer trace。
|
||||
|
||||
这里的“缩小 AttnRes 实验”只完成独立机制 probe，不满足最后一项真实 K3 trace。完整合同与
|
||||
审计见 `K3_ATTNRES_REDUCED_PROTOCOL.md` / `K3_ATTNRES_REDUCED_AUDIT.md`。下一轮优先
|
||||
对齐论文 gradient 观测对象并增加 depth / budget；真实 checkpoint forward 继续等待公开
|
||||
`A_log [128] → [96]` 转换或权重修订。
|
||||
|
||||
## 11. 发布与回归证据
|
||||
|
||||
- 源提交：`2ef846f`；
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# Round 08 AttnRes 训练期前向干预：结果审计
|
||||
|
||||
审计日期：2026-07-30
|
||||
协议：`llm-atlas-k3-attnres-forward-training-v1`
|
||||
结果后 Grok 会话：`019fb28b-a9e1-7643-8e43-06f5e16a2077`
|
||||
|
||||
## 1. 一句话结论
|
||||
|
||||
冻结 analyzer 的唯一主状态为：
|
||||
|
||||
```text
|
||||
forward_training_attenuation_established_within_reduced_protocol
|
||||
```
|
||||
|
||||
联合 `groups 6+7` 的训练期 uniform-forward 消融在三个预注册 seed 上同时通过
|
||||
`spike contrast` 与 `peak / mean` 的 20% attenuation 门；逐 seed 与三 seed 平均
|
||||
validation BPC 也全部通过质量护栏。结果后独立只读复算得到：
|
||||
|
||||
```text
|
||||
blocking_errors = 0
|
||||
status_confirmed = true
|
||||
replay_confirmed = true
|
||||
```
|
||||
|
||||
这只是在固定 depth-32 缩小 Block AttnRes、固定数据与 8,000-step 预算中的训练期
|
||||
架构消融；**不是真实 Kimi-K3 / 2.8T checkpoint 结果，也不是 Figure 5(c) 未公开
|
||||
telemetry 的复现。**
|
||||
|
||||
## 2. 主门复算
|
||||
|
||||
冻结定义：
|
||||
|
||||
```text
|
||||
S = layers 21–25
|
||||
R = other 27 layers
|
||||
C = mean(g[S]) / mean(g[R])
|
||||
P = max(g) / mean(g)
|
||||
D = (X_reference - X_variant) / X_reference
|
||||
```
|
||||
|
||||
主变体为 `uniform_groups_6_7_forward`，只读复算如下：
|
||||
|
||||
| seed | C reference | C variant | C drop | P reference | P variant | P drop | ΔBPC |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 2026073001 | 3.046269 | 0.806011 | 73.54% | 3.219067 | 1.435618 | 55.40% | +0.005696 |
|
||||
| 2026073002 | 3.332848 | 0.758201 | 77.25% | 3.618924 | 1.375016 | 62.00% | +0.009598 |
|
||||
| 2026073003 | 1.881492 | 0.712882 | 62.11% | 2.288267 | 1.548141 | 32.34% | +0.004441 |
|
||||
|
||||
因此 attenuation 为 `6/6`；最小 contrast drop 为 `62.11%`，最小 peak drop 为
|
||||
`32.34%`，都高于冻结的 `20%` 门槛。BPC 三格都低于 `+0.05`，平均
|
||||
`+0.006578` 低于 `+0.03`，质量门为 `4/4`。
|
||||
|
||||
这里的 contrast 下降不等价于“尖峰层被关闭”：它可能由 spike-window 分子下降、
|
||||
27 层 reference 分母上升，或两者共同造成；被联合消融覆盖的 layers 26–28 仍属于
|
||||
这个分母。
|
||||
|
||||
## 3. 完整性与复现
|
||||
|
||||
- 13 个新 raw 文件完整：12 formal + 1 primary replay；
|
||||
- 每格 `65,536,000` target bytes，新处理总量 `851,968,000`；
|
||||
- 三个历史 paired reference 合计 `196,608,000` bytes，未在 Round 08 重跑；
|
||||
- 13/13 raw canonical self-hash、aggregate 与 reproduction self-hash 自洽；
|
||||
- architecture / depth / steps / batch 固定为 Block / 32 / 8,000 / 32；
|
||||
- initial public/mixer state、schedule、validation、diagnostic、input-gate、model、
|
||||
optimizer 与 scientific environment 对同 seed historical reference 配对 exact；
|
||||
- primary seed 1 的 formal / replay scientific payload exact：
|
||||
`b85563ca5cb53e60b39c3801d372376206105b8a089a8633b3e81973a7f0c051`。
|
||||
|
||||
四个正式 selector 的 visit、source count、uniform arithmetic 与 reachability 均通过。
|
||||
被选 mixer 的 query / key norm 留在 optimizer param groups，但 forward 不再调用它们：
|
||||
gradient hook 为 0、Adam state 不存在、最终 tensor 与初始值 byte-exact。未选 mixer 的
|
||||
optimizer-state 检查是比科学协议更强的实现审计，不参与主 status。
|
||||
|
||||
## 4. 描述性 non-additivity
|
||||
|
||||
冻结的 bookkeeping residual 为：
|
||||
|
||||
```text
|
||||
I67 = ln(Xref / X67) - ln(Xref / X6) - ln(Xref / X7)
|
||||
```
|
||||
|
||||
step 8,000 的三 seed 平均为：
|
||||
|
||||
- spike contrast：`-0.367038`
|
||||
- peak / mean：`-0.170448`
|
||||
|
||||
它来自三套独立训练，只能描述 joint run 与两个 single runs 的 log-effect 残差；不能
|
||||
写成因果 interaction、Shapley contribution 或“group 6/7 互相抑制”的机制结论。
|
||||
|
||||
## 5. 两阶段独立审阅
|
||||
|
||||
结果前 Grok 实现审阅指出 smoke-only empty selector 的 visit census 可空真。正式
|
||||
4×3 路径全部是非空 selector，因此不影响 raw formal 数值；矩阵结束后已删除 early
|
||||
return、加入 `forward_calls > 0`，并重跑 step-zero、parent smoke、wrapper smoke
|
||||
与 equivalence。修补后的 learned wrapper 实际执行 39 次 forward，父/包装器 15 组
|
||||
科学字段仍全部 exact。
|
||||
|
||||
结果后 Grok 在只读 sandbox 中从 13 个 raw 与三个 historical references 独立复算
|
||||
identity、自哈希、selector、pairing、两项主指标、BPC、replay 与 `I67`。它报告
|
||||
`0 mismatch`、`blocking_errors=0`，确认 analyzer status 与 claim boundary。
|
||||
|
||||
## 6. 最终 claim boundary
|
||||
|
||||
可以说：在这一固定缩小协议内，联合 group 6+7 的 train-time uniform-forward
|
||||
architecture ablation 相对历史同 seed reference 达到预注册 attenuation + BPC 门控。
|
||||
|
||||
不能说：
|
||||
|
||||
- 已定位真实 Kimi-K3 的训练尖峰；
|
||||
- 已复现 K3 报告 Figure 5(c)；
|
||||
- 已把 forward、natural backward 与 optimizer update 分离成纯因果效应；
|
||||
- 已证明下游能力等价、总体统计显著性、可加性或因果 interaction。
|
||||
@@ -0,0 +1,69 @@
|
||||
# Round 08 Grok Headless 对抗审阅与处置
|
||||
|
||||
审阅日期：2026-07-30
|
||||
审阅会话：`019fb1ce-7ffa-70e3-b860-c4a31a4c6621`
|
||||
身份：**外部模型的只读方法学审稿，不是论文证据源**
|
||||
|
||||
## 1. 调用边界
|
||||
|
||||
Grok CLI 使用 single/headless + plan permission 读取：
|
||||
|
||||
- `research/K3_ATTNRES_FORWARD_TRAINING_SCOPING.md`
|
||||
- `research/K3_ATTNRES_FORWARD_TRAINING_PROTOCOL.md`
|
||||
- `experiments/k3/attnres_forward/manifest.json`
|
||||
- Round 04 / 05 父 runners
|
||||
|
||||
关闭 web search、禁止 subagents；它没有修改文件，也没有运行训练。
|
||||
|
||||
## 2. Blocking findings 与处置
|
||||
|
||||
| finding | 风险 | 处置 |
|
||||
|---|---|---|
|
||||
| `mean` 与父 `softmax+einsum` 的 FP32 归约顺序不保证 byte-exact | step-0 会假失败 | **采纳**：选中路径改成参数无关 constant-zero logits，并复用同一 `softmax+einsum` kernel |
|
||||
| pairing 要求了父 JSON 不存在的 initial full hash，并把 GPU 名写成 exact | 科学身份正确却因 metadata 失败 | **采纳**：只 hard-gate 初始化、数据、拓扑、optimizer 与确定性/autocast 合同 |
|
||||
| replay 只排除三个字段，path 会制造假差异 | exact replay 假失败 | **采纳**：冻结 scientific canonical payload 与 path normalization |
|
||||
| wrapper 若把新 protocol ID 写入 `window_start` salt，会静默换数据 | 历史逐 seed 配对失效 | **采纳**：父 salt 只由父 manifest/runner 管，新 ID 只进 wrapper output |
|
||||
| 主判定没有在协议正文再写 final-only | analyzer 可能误读六 checkpoints | **采纳**：主 attenuation 与 BPC 只读 step 8,000 |
|
||||
| spike layer numbering 未 machine-readable 固定 | 可能整体平移一层 | **采纳**：manifest 新增 `spike_layers_1based`，协议钉死 Python index = layer−1 |
|
||||
|
||||
## 3. Non-blocking findings 与处置
|
||||
|
||||
全部采纳：
|
||||
|
||||
- selected 参数留在 AdamW param groups，但因 graph 不可达而没有 state entry，不能写成
|
||||
“训练了但没动”；
|
||||
- group 6 / 7 indices 与 source counts 经独立复算正确；
|
||||
- reference 统一称 historical paired reference；
|
||||
- layers 26–28 位于 contrast 分母 `R`，必须拆报 `mean(S)` / `mean(R)`；
|
||||
- `I67` 明确是三次独立训练之间的 log residual；
|
||||
- missing / non-finite / structural error 统一 `contract_failed`，不能冒充科学失败；
|
||||
- wrapper 禁止改变 FP32 residual、AdamW grouping、parent salt 与 empty-selector path。
|
||||
|
||||
## 4. 算术复核
|
||||
|
||||
```text
|
||||
8,000 × 32 × 256 = 65,536,000 bytes / cell
|
||||
12 formal cells = 786,432,000 bytes
|
||||
+ primary replay = 851,968,000 newly processed bytes
|
||||
historical refs = 196,608,000 bytes（不重跑）
|
||||
```
|
||||
|
||||
selector：
|
||||
|
||||
```text
|
||||
group 6 = 40..47; source N = 6,7,7,7,7,7,7,7
|
||||
group 7 = 48..55; source N = 7,8,8,8,8,8,8,8
|
||||
group 7 MLP = 49,51,53,55
|
||||
groups 6+7 = 40..55
|
||||
```
|
||||
|
||||
主 attenuation 为 `3 seeds × {contrast, peak} = 6` cells；quality 为三个
|
||||
per-seed BPC gates 加一个 mean gate，共 4 项。两者合取，且只读 final step。
|
||||
|
||||
## 5. 复核结论
|
||||
|
||||
Grok 判断研究身份、矩阵、source counts 与主公式骨架可以保留；主要风险来自浮点
|
||||
arithmetic identity、wrapper salt、over-exact metadata 与层号歧义。以上项目已在任何
|
||||
正式输出出现前全部修订，manifest 状态随后改为 `frozen-before-model-output`。
|
||||
|
||||
审稿意见不会进入实验结果、论文事实或官网证据等级；它只用于结果前强化协议。
|
||||
@@ -0,0 +1,89 @@
|
||||
# Round 08 runner / analyzer 实现审阅与处置
|
||||
|
||||
审阅日期：2026-07-30
|
||||
Grok 会话：`019fb1f0-bb24-7202-8123-295edda7518f`
|
||||
身份：**正式文件完成前的外部模型只读实现审计，不是结果或论文证据**
|
||||
|
||||
## 1. 审阅边界
|
||||
|
||||
Grok Headless 只读检查：
|
||||
|
||||
- 冻结协议与 manifest；
|
||||
- `train.py` / `verify.py` / `analyze.py` / `run_matrix.py`；
|
||||
- 父 runner 的 salt、schema 与 DepthMixer 算术路径。
|
||||
|
||||
明确禁止读取 Round 08 raw formal results、运行训练、修改文件、web search 与 subagents。
|
||||
|
||||
## 2. 对正式 4×3 路径的确认
|
||||
|
||||
审阅确认：
|
||||
|
||||
- 新 protocol ID 没有进入父 `window_start` salt；
|
||||
- selected 路径用 constant-zero FP32 logits 和父 `softmax+einsum` kernel；
|
||||
- 非空 selector 的 query / key norm hook、AdamW state 与 final=initial gate 自洽；
|
||||
- historical pairing 比较的字段在父 JSON 中真实存在；
|
||||
- replay payload 正确排除 run kind、timing、path、parent self hash 与 GPU/version metadata，
|
||||
同时保留模型、optimizer、diagnostics、history、selector 与 reachability；
|
||||
- analyzer 强制 4×3 identity、step 8,000 主判定、1-based layers 21–25、finite/positive、
|
||||
`D=(ref-variant)/ref`、BPC `variant-ref` 与 descriptive `I67`；
|
||||
- 12 formal + 1 replay、65,536,000 bytes/cell 与最大两进程算术正确。
|
||||
|
||||
没有发现会改变正在运行的四个非空 formal variants 数值语义的 blocking error。
|
||||
|
||||
## 3. Blocking finding：smoke-only empty selector 的空审计
|
||||
|
||||
`learned_reference` 直接调用父 `forward`，没有递增 wrapper 的 visit counters。因此：
|
||||
|
||||
```text
|
||||
forward_calls = 0
|
||||
all depth/output visits = 0
|
||||
visit_gate = all(0 == 0) = true
|
||||
selected reachability = all([]) = true
|
||||
```
|
||||
|
||||
这不会影响四个正式变体，它们全部是非空 selector；而 empty selector 另有 20-step
|
||||
parent-equivalence gate，模型/optimizer/evaluations/diagnostics/history/hash 已 exact。
|
||||
但 `forward_intervention.passed` 本身不能在修复前被当作 learned census 证据。
|
||||
|
||||
处置：
|
||||
|
||||
- formal matrix 完成后，删除 `selected_indices` 为空时的父路径 early return；
|
||||
- 让 empty selector 也走同一 copied forward，其中 `mix` 对 64 个节点逐个调用父
|
||||
`DepthMixer.forward`；
|
||||
- 加入 `forward_calls > 0`；
|
||||
- 重跑 step-0、parent smoke、wrapper smoke 与 parent equivalence；
|
||||
- 只有 copied path 仍逐字段 exact 才保留。
|
||||
|
||||
这项修复只强化 smoke audit，不更改任何正式 variant 的 selector 或 forward。
|
||||
|
||||
## 4. Non-blocking findings 与处置
|
||||
|
||||
| finding | 处置 |
|
||||
|---|---|
|
||||
| `smoke_compare` 整体比较 `manifest.path` | 规范化 path 后比较 scientific manifest fields |
|
||||
| unselected gate 额外要求 optimizer state | 保留为强实现 gate，但在文档中标成 protocol 之外的额外审计，不用于科学 status |
|
||||
| GPU/version 差异只有 `metadata_equal`，没有 warnings 数组 | aggregate 增加显式 metadata warnings |
|
||||
| `run_matrix` 失败后只 terminate、不 wait/kill，invalid file 会阻塞重跑 | 加入 terminate→wait→kill 清理，并在 cell failure 时标明 exact invalid target |
|
||||
| step-0 CE 只比较 Python float | 追加 scalar tensor SHA-256 |
|
||||
| `EXPECTED_SOURCE_COUNTS` 未使用 | 用于 runner↔manifest 交叉校验 |
|
||||
|
||||
上述修订不读取结果、不改变冻结阈值或主公式。
|
||||
|
||||
## 5. 矩阵结束后的处置结果
|
||||
|
||||
13 个单元全部退出后才应用上述修订；四个正式非空 selector 的 raw 文件未被重写。
|
||||
修补后的前置闸门结果：
|
||||
|
||||
- step-zero 五个 variant 的 logits、loss tensor、evaluation 与 diagnostic exact；
|
||||
- smoke-only learned wrapper 实际执行 `39` 次 forward，64 个 depth mixer 与
|
||||
output mixer 的 visit census 全部非零且 exact；
|
||||
- parent 与 wrapper 的 architecture、seed、schedule、model、optimizer、hash、
|
||||
evaluation、diagnostic、history、gradient gate 与 scientific environment 共
|
||||
15 组字段全部 exact；
|
||||
- primary smoke 的 selected 参数仍为 0 hook、无 optimizer state、final=initial；
|
||||
- unselected optimizer-state 条件继续作为额外实现闸门，不进入科学 status。
|
||||
|
||||
结果后 Grok 会话 `019fb28b-a9e1-7643-8e43-06f5e16a2077` 在只读 sandbox 中独立
|
||||
复算 13 个 raw、三个 historical references 与 aggregate，报告
|
||||
`blocking_errors=0`、`status_confirmed=true`、`replay_confirmed=true`。详细数字与
|
||||
claim boundary 见 `research/K3_ATTNRES_FORWARD_TRAINING_AUDIT.md`。
|
||||
@@ -0,0 +1,389 @@
|
||||
# K3 Attention Residuals 训练期前向干预协议
|
||||
|
||||
协议 ID：`llm-atlas-k3-attnres-forward-training-v1`
|
||||
冻结日期：2026-07-30
|
||||
协议状态：**结果前预注册 frozen；任何语义变更必须更换 protocol ID**
|
||||
父协议：`llm-atlas-k3-attnres-gradient-scale-v1`
|
||||
|
||||
## 0. 研究身份
|
||||
|
||||
这是 Round 07 定向线索之后的训练期架构消融。选中 depth mixer 在每一次 train / eval /
|
||||
diagnostic forward 都用 source states 的算术平均，完全绕过该 mixer 的
|
||||
`query + key_norm + softmax` 路径。
|
||||
|
||||
允许回答：
|
||||
|
||||
1. 固定 groups 6+7 的 uniform forward 训练变体，能否在不触发预注册 BPC 失败护栏时，
|
||||
material 地降低最终固定 activation-gradient spike？
|
||||
2. group 6、group 7 与 joint 的训练轨迹呈现什么非加性关系？
|
||||
3. Round 07 指向的 group 7 MLP-only 路径能否独立产生 material response？
|
||||
|
||||
不允许回答：
|
||||
|
||||
- 真实 Kimi K3 2.8T checkpoint 的梯度或训练动力学；
|
||||
- 论文 Figure 5(c) 未公开 telemetry 的复现；
|
||||
- “forward effect” 与 natural backward/update effect 的分离；
|
||||
- selected query/key 参数如果继续训练会怎样；
|
||||
- 三 seed 外的总体显著性、置信区间或 p-value；
|
||||
- 下游能力保持、通用质量等价或最优 AttnRes 设计；
|
||||
- 单组 effects 的可加性、Shapley value、方差贡献或因果交互；
|
||||
- 与 Round 07 value-coefficient intervention 同构的“纯 forward”因果复制；
|
||||
- K3 `A_log` 的官方修复裁决。
|
||||
|
||||
## 1. 冻结训练与数据合同
|
||||
|
||||
| 字段 | 固定值 |
|
||||
|---|---|
|
||||
| architecture | Block AttnRes |
|
||||
| Transformer depth | 32 |
|
||||
| aggregation groups | 8 |
|
||||
| blocks / group | 4 |
|
||||
| depth / output mixers | 64 / 1 |
|
||||
| width / heads / FFN | 192 / 6 / 768 |
|
||||
| context / vocabulary | 256 / byte-256 |
|
||||
| seeds | 2026073001 / 2026073002 / 2026073003 |
|
||||
| steps / batch | 8,000 / 32 |
|
||||
| target bytes / new formal cell | 65,536,000 |
|
||||
| optimizer | AdamW |
|
||||
| peak / min LR | 3e-4 / 3e-5 |
|
||||
| warmup | 400 |
|
||||
| weight decay | 0.1 for ndim ≥ 2 |
|
||||
| betas / epsilon | 0.9, 0.95 / 1e-8 |
|
||||
| clip | global norm 1.0 |
|
||||
| forward | CUDA BF16 autocast |
|
||||
| residual accumulation | explicit FP32 |
|
||||
| validation | fixed 64 × 256-byte windows |
|
||||
| diagnostic | fixed 16 × 256-byte windows |
|
||||
| checkpoints | 0 / 100 / 500 / 2,000 / 4,000 / 8,000 |
|
||||
| concurrency | at most two independent processes |
|
||||
|
||||
训练输入 schedule **逐 step 复用父协议**。`window_start` 使用父
|
||||
`llm-atlas-k3-attnres-gradient-scale-v1` 的 salt；新 protocol ID 只写入 wrapper output
|
||||
和 study manifest，绝不能进入 `round04.PROTOCOL_ID` 或训练窗口散列。父 manifest
|
||||
负责 bytes / windows / schedule，新 manifest 只负责 variants / selector / thresholds。
|
||||
每个 variant / seed 的初始化、optimizer input、validation 与 diagnostic tensors 必须
|
||||
exact 相同。
|
||||
|
||||
runner 必须继承 Round 05 `GradientLanguageModel` 的 explicit FP32 Block residual
|
||||
accumulation；不得退回 Round 04 的旧累加路径。
|
||||
|
||||
新正式处理量：
|
||||
|
||||
```text
|
||||
4 variants × 3 seeds × 65,536,000 = 786,432,000 target bytes
|
||||
1 primary replay 65,536,000 target bytes
|
||||
total newly processed 851,968,000 target bytes
|
||||
historical learned reference 196,608,000 target bytes（不重跑）
|
||||
```
|
||||
|
||||
每格必须使用全新 Python process。最多并行两个；不能共享 model、optimizer、RNG、
|
||||
CUDA graph 或 output file。
|
||||
|
||||
## 2. 冻结正式矩阵
|
||||
|
||||
正式 variants：
|
||||
|
||||
| variant | exact selected depth indices | layers / branch |
|
||||
|---|---|---|
|
||||
| `uniform_group_6_forward` | 40–47 | 21–24 / both |
|
||||
| `uniform_group_7_forward` | 48–55 | 25–28 / both |
|
||||
| `uniform_groups_6_7_forward` | 40–55 | 21–28 / both |
|
||||
| `uniform_group_7_mlp_forward` | 49, 51, 53, 55 | 25–28 / MLP |
|
||||
|
||||
`learned_reference` 只允许 smoke，不进入新正式矩阵。output mixer index 64 永远 learned。
|
||||
|
||||
正式运行 12 格；另从初始化 replay：
|
||||
|
||||
```text
|
||||
replay / uniform_groups_6_7_forward / seed 2026073001
|
||||
```
|
||||
|
||||
## 3. 唯一 selector 与 forward 语义
|
||||
|
||||
runner 必须只有一个 machine-readable selector：
|
||||
|
||||
```text
|
||||
selected(variant, depth_mixer_index) -> bool
|
||||
```
|
||||
|
||||
不得把四个 variant 分叉成四份 model forward。
|
||||
|
||||
对未选中 depth mixer 和 output mixer，逐调用父 `DepthMixer.forward`。对选中 mixer，
|
||||
为了让初始化负控制复用相同浮点归约顺序，使用参数无关的零 logits，但仍走父
|
||||
`softmax + einsum` 数值 kernel：
|
||||
|
||||
```text
|
||||
values = stack(sources, dim=0)
|
||||
logits = zeros([N, batch, tokens], dtype=FP32)
|
||||
weights = softmax(logits, dim=0)
|
||||
output = einsum("nbt,nbtd->btd", weights, values.float()).to(values.dtype)
|
||||
```
|
||||
|
||||
capture summary 必须仍使用父 schema：
|
||||
|
||||
```text
|
||||
mean_weights = [1 / N] × N
|
||||
entropy_mean = ln(N)
|
||||
sources = N
|
||||
```
|
||||
|
||||
选中路径不得调用 `query`、`key_norm` 或 source-dependent logits，也不得用
|
||||
stop-gradient trick 让这些参数看似参与。这里保留的 softmax 只把常数零 logits 变成
|
||||
`1/N`，目的是与父路径保持同一 arithmetic kernel；它没有可训练参数。自然结果是选中
|
||||
mixer 的 `query` 和 `key_norm.weight`：
|
||||
|
||||
- gradient hook call count 必须为 0；
|
||||
- optimizer state entry 必须不存在；
|
||||
- final tensor 必须与 initial tensor byte-exact。
|
||||
|
||||
所有未选中 depth mixer 和 output mixer 的两个参数都必须有正的 gradient hook call
|
||||
count；这只证明图可达，不要求它们的梯度非零或最终 tensor 一定变化。hook census
|
||||
从 model 构造后开始，覆盖所有 training backward 与 diagnostic backward；eval
|
||||
forward 不计 hook。selected 参数允许保留在原 AdamW param groups，但 state entry 必须
|
||||
不存在，不能表述为“训练了但没有移动”。
|
||||
|
||||
## 4. 结果前实现闸门
|
||||
|
||||
### 4.1 empty-selector 父等价
|
||||
|
||||
`learned_reference` smoke 必须直接走父 forward，不得走常数 uniform 分支；它与父
|
||||
Round 05 runner 在相同 seed / 20 steps 下必须：
|
||||
|
||||
- initial/final model hashes exact；
|
||||
- final optimizer hash exact；
|
||||
- evaluations、diagnostics 与 training history exact；
|
||||
- gradient gate exact；
|
||||
- input tensor hashes exact。
|
||||
|
||||
允许不同字段只限 protocol wrapper identity、study-manifest wrapper、timing、规范化
|
||||
后的 manifest path、output path 与 self-hash。AdamW 参数分组必须保持父语义；
|
||||
empty-selector 不能删除任何 mixer 参数。
|
||||
|
||||
### 4.2 step-0 identity negative control
|
||||
|
||||
父模型所有 depth mixer query 初始化为 0，因此 learned softmax 在 step 0 是 exact
|
||||
uniform。选中分支用同 dtype 的 constant-zero logits 和同一个
|
||||
`softmax + einsum` kernel。四个 variant 与 learned reference 在固定 input 上必须：
|
||||
|
||||
- logits byte-exact；
|
||||
- CE byte-exact；
|
||||
- activation-gradient spectrum byte-exact；
|
||||
- validation metrics byte-exact；
|
||||
- selected 的**归约前 weight tensor** 等于 FP32 `1/N`，max absolute error
|
||||
`≤ 1e-12`；capture 的 `mean_weights` 因 FP32 大规模 mean 可有约 `1e-8` 的归约舍入，
|
||||
但必须与父 learned capture summary byte-exact；
|
||||
- 若任何跨 variant byte-exact 比较失败，hard-fail；不得在结果后改成容差 gate。
|
||||
|
||||
这个负控制只约束初始化；训练开始后 forward 必须允许分化。
|
||||
|
||||
### 4.3 selector census
|
||||
|
||||
每个 forward 的 64 个 depth index 必须各访问一次，output 访问一次且保持 learned。
|
||||
每个 variant 的 selected set 必须与第 2 节 exact。group 内 source counts 必须满足：
|
||||
|
||||
```text
|
||||
group 6: index 40 has N=6; indices 41..47 have N=7
|
||||
group 7: index 48 has N=7; indices 49..55 have N=8
|
||||
```
|
||||
|
||||
推导前提是 `completed` 含 embedding，且每 8 个 depth mixer 才把 `partial` 聚合为一个
|
||||
completed group。
|
||||
|
||||
正式 output 保存 exact selected indices、实际 visit census、source counts 与
|
||||
uniform-weight max error。任何漏访、重访、越界或 output 被选中都失败。
|
||||
|
||||
### 4.4 数据、有限性与梯度尺度
|
||||
|
||||
- 父 manifest、train/validation/diagnostic bytes 与 schedule hashes exact；
|
||||
- step 0 / 1 / 7,999 optimizer input gate hashes exact；
|
||||
- loss、logits、所有主 activation gradients 全部 finite；
|
||||
- global clip 后每一步都执行 optimizer update，不允许 skip；
|
||||
- smoke 的 diagnostic loss `×2` 时，每层 activation-gradient RMS 比值在
|
||||
`2 ± 1e-5`，normalized spectrum max delta `≤ 1e-6`。
|
||||
|
||||
## 5. 历史 reference 配对合同
|
||||
|
||||
reference 固定为：
|
||||
|
||||
```text
|
||||
experiments/k3/attnres_gradient/results/raw/
|
||||
formal-depth-32-block-seed-{seed}.json
|
||||
```
|
||||
|
||||
analyzer 的 pairing hard gates：
|
||||
|
||||
- `protocol_id = llm-atlas-k3-attnres-gradient-scale-v1`；
|
||||
- formal / block / depth 32 / 8,000 steps / batch 32；
|
||||
- seed exact；
|
||||
- initial public 与 mixer hashes 跟对应新 variant exact；
|
||||
- formal schedule、validation tensor、diagnostic tensor 和三个 input gate hashes exact；
|
||||
- model topology、optimizer hyperparameters、CUBLAS workspace、deterministic flags 与
|
||||
autocast 语义 exact。
|
||||
|
||||
以下字段明确**不参与 pairing equality**：
|
||||
|
||||
- 所有 final hashes、evaluations、diagnostics、training history 与 gradient gate；
|
||||
- timing、run kind、self-hash、output path 与 manifest path 字符串；
|
||||
- GPU 名称、driver / CUDA / torch version 的 minor 差异。
|
||||
|
||||
环境完整记录；若数值栈变化，aggregate 给出 metadata warning，但只要上述确定性与
|
||||
autocast 合同相同就不将其误判为 pairing failure。父 JSON 没有 initial full-state 字段，
|
||||
不得假定它存在；public ∪ mixer 的完整性只用结构/元素 census 自洽。
|
||||
|
||||
reference 是历史配对基线，不得写成同期随机对照。若任何合同不等，整轮 aggregate
|
||||
失败，而不是降级为“近似比较”。
|
||||
|
||||
## 6. 固定主对象与公式
|
||||
|
||||
每个 diagnostic checkpoint 从：
|
||||
|
||||
```text
|
||||
activation_grad_rms_by_block = [g1, ..., g32]
|
||||
layer_id ∈ {1,...,32}
|
||||
g[layer_id] = activation_grad_rms_by_block[layer_id - 1]
|
||||
S = {21,22,23,24,25} # 1-based
|
||||
R = {1,...,32} \ S
|
||||
```
|
||||
|
||||
计算：
|
||||
|
||||
```text
|
||||
C = mean(g[S]) / mean(g[R]) # spike contrast
|
||||
P = max(g) / mean(g) # peak normalized
|
||||
```
|
||||
|
||||
所有 `g`、`C`、`P` 必须 finite 且严格大于 `1e-30`。groups 6+7 还改写落在 `R`
|
||||
中的 layers 26–28，所以 analyzer 同时报告 `mean(g[S])` 与 `mean(g[R])`，但不把它们
|
||||
加入主 status。
|
||||
|
||||
对同 seed reference `X_ref` 与 variant `X_v`：
|
||||
|
||||
```text
|
||||
D_X(v) = (X_ref - X_v) / X_ref
|
||||
```
|
||||
|
||||
`D>0` 表示 attenuation，`D<0` 表示 amplification。不得取绝对值，不得更换分母。
|
||||
|
||||
## 7. 预注册判定
|
||||
|
||||
### 7.1 主判定
|
||||
|
||||
`uniform_groups_6_7_forward` 的主 attenuation gate **只读取 step=8,000**：
|
||||
|
||||
```text
|
||||
D_C >= 0.20 AND D_P >= 0.20
|
||||
for all 3 seeds
|
||||
```
|
||||
|
||||
质量 gate 也只读取 `evaluations[step=8000].bits_per_byte`：
|
||||
|
||||
```text
|
||||
delta_bpc(seed) = final_bpc_variant - final_bpc_reference
|
||||
delta_bpc(seed) <= 0.05 for all 3 seeds
|
||||
mean(delta_bpc) <= 0.03
|
||||
```
|
||||
|
||||
只有 attenuation 6/6 与 quality 4/4 同时通过，正式 status 才是：
|
||||
|
||||
```text
|
||||
forward_training_attenuation_established_within_reduced_protocol
|
||||
```
|
||||
|
||||
否则按失败位置使用：
|
||||
|
||||
```text
|
||||
attenuation_not_established
|
||||
quality_guard_failed
|
||||
attenuation_and_quality_failed
|
||||
```
|
||||
|
||||
不能用次级变体补救主判定。任何 step 8,000 缺失/重复、数组长度错误、hash 不配对、
|
||||
selector / reachability 失败、`g/C/P/BPC/D/log` 缺失或非 finite 都是
|
||||
`contract_failed` 并让 analyzer non-zero exit；不能把结构失败包装成上面的科学状态。
|
||||
|
||||
### 7.2 次级 material response
|
||||
|
||||
group 6、group 7、group 7 MLP-only 各自使用同一 `20% / 3-seed / 2-metric` attenuation
|
||||
threshold 和同一 quality guard，分别报告：
|
||||
|
||||
```text
|
||||
secondary_material_response / secondary_response_not_established
|
||||
```
|
||||
|
||||
它们不改变主 status，也不升级成 localization。
|
||||
|
||||
### 7.3 BPC 护栏的解释
|
||||
|
||||
`+0.05 per seed / +0.03 mean` 是预注册的 catastrophic-degradation screen：
|
||||
|
||||
- 失败说明不能把 spike 下降当成健康训练的证据；
|
||||
- 通过不说明能力、校准或下游任务等价；
|
||||
- BPC 改善也不说明总体架构更优。
|
||||
|
||||
## 8. 非加性交互与轨迹
|
||||
|
||||
对 `X ∈ {C,P}`、每个 seed、每个 checkpoint：
|
||||
|
||||
```text
|
||||
E6 = ln(X_ref / X_group6)
|
||||
E7 = ln(X_ref / X_group7)
|
||||
E67 = ln(X_ref / X_groups6+7)
|
||||
I67 = E67 - E6 - E7
|
||||
```
|
||||
|
||||
保存 `E6/E7/E67/I67` 原值、对应 `D_C/D_P` 和三 seed mean/range。没有通过阈值、
|
||||
p-value 或 CI。它是三套独立训练在相同 checkpoint 的跨-run log residual；
|
||||
`I67` 不能写成可加贡献、独立作用、Shapley value 或因果 interaction estimate。
|
||||
|
||||
同时全量保存：
|
||||
|
||||
- 六 checkpoints 的 32-layer raw / normalized spectra；
|
||||
- peak layer、top-five layers；
|
||||
- validation BPC 与 train-loss trajectory；
|
||||
- selected/unselected mixer weight summaries；
|
||||
- selected-parameter reachability audit；
|
||||
- per-cell timing 与显存（不进入数值结论）。
|
||||
|
||||
## 9. replay 与 analyzer 合同
|
||||
|
||||
primary seed-1 replay 使用 analyzer 定义的 scientific canonical payload。先删除：
|
||||
|
||||
```text
|
||||
run_kind
|
||||
timing
|
||||
canonical_sha256_without_self
|
||||
manifest.path
|
||||
study_manifest.path
|
||||
```
|
||||
|
||||
再比较以下固定字段 exact：protocol / variant / architecture / depth / seed / steps / batch /
|
||||
target bytes、manifest scientific hashes、model、optimizer、initial/final hashes、
|
||||
evaluations、diagnostics、training history、selector 与 reachability audits，以及
|
||||
environment 中 deterministic / autocast scientific subset。GPU/版本 metadata 保留在
|
||||
两份文件中单独展示，不进入 canonical equality。
|
||||
|
||||
所有主指标、阈值、status、interaction map 与 compact website artifact 只能由单一
|
||||
`experiments/k3/attnres_forward/analyze.py` 生成。网站不能在 TypeScript 中重新计算
|
||||
另一套结论。
|
||||
|
||||
analyzer 在任何结构、hash、selector、reachability、finite、reference pairing、
|
||||
replay 或 threshold contract 失败时必须 non-zero exit，不得输出部分通过结论。
|
||||
|
||||
## 10. 报告语言红线
|
||||
|
||||
允许：
|
||||
|
||||
- “在这个固定缩小模型与训练协议内，局部 uniform-forward 变体……”
|
||||
- “selected mixer 参数在此架构消融中结构性不可达……”
|
||||
- “joint log effect 呈现正/负 interaction residual……”
|
||||
|
||||
禁止：
|
||||
|
||||
- “证明 K3 的训练尖峰来自 group 6/7”
|
||||
- “只改变 forward，所以这是纯 forward 因果效应”
|
||||
- “BPC gate 通过，所以能力不受影响”
|
||||
- “interaction residual 是两个 group 的真实贡献”
|
||||
- “contrast 下降证明尖峰层本身下降”（未同时检查 `S` / `R` 分拆）
|
||||
- “step-0 exact 说明训练期始终与 learned forward 恒等”
|
||||
- “复现了 K3 Figure 5(c)”
|
||||
- “已经验证官方 2.8T checkpoint”
|
||||
@@ -0,0 +1,195 @@
|
||||
# K3 Attention Residuals 训练期前向干预：Round 08 前置定位
|
||||
|
||||
研究日期：2026-07-30
|
||||
阶段身份：**Round 07 后的定向 scoping，不是 Round 08 结果**
|
||||
上游协议：`llm-atlas-k3-attnres-local-path-v1`
|
||||
|
||||
## 1. 为什么还需要一次训练期实验
|
||||
|
||||
Round 06 / 07 都保持 learned forward 完全不变，只在固定 diagnostic 的 backward 中
|
||||
替换 source value coefficients。它们回答的是：
|
||||
|
||||
- 全部 65 个 mixer 的 uniform value backward 能不能压低固定尖峰；
|
||||
- group 6 / 7 的 16 个 depth mixer 在 learned 背景上是否足以复现全局下降；
|
||||
- 从 all-uniform 背景恢复这些 mixer 是否能反向恢复尖峰。
|
||||
|
||||
Round 07 的正式结论是：
|
||||
|
||||
```text
|
||||
groups 6+7 sufficiency：6 / 6 seed×metric cells 通过
|
||||
groups 6+7 restoration：3 / 6 cells 通过
|
||||
formal status：one_sided_evidence_localization_not_established
|
||||
```
|
||||
|
||||
这已经足以排除“局部 mask 完全没有反应”，但还不能回答：
|
||||
|
||||
> 如果训练的每一次 forward 都真的把这段 depth routing 改成算术平均，模型会怎样适应？
|
||||
|
||||
Round 08 把 intervention 放进 optimizer path。它不再追求 forward-identical，而是让
|
||||
选中 mixer 的输出在训练、验证与诊断中始终为所有 source states 的等权平均。
|
||||
|
||||
## 2. 这不是“只改变 forward”
|
||||
|
||||
选中 mixer 的 learned 路径原本是：
|
||||
|
||||
```text
|
||||
keys = RMSNorm(sources)
|
||||
logits = query · keys
|
||||
weights = softmax(logits over source-depth)
|
||||
output = Σ weights_i × source_i
|
||||
```
|
||||
|
||||
Round 08 的选中路径是：
|
||||
|
||||
```text
|
||||
output = (1 / N) × Σ source_i
|
||||
```
|
||||
|
||||
因此 intervention 同时改变：
|
||||
|
||||
1. forward 的 branch input；
|
||||
2. 由新 forward 自然产生的 source gradients；
|
||||
3. 下游 activation、loss 与所有后续 optimizer updates；
|
||||
4. 选中 mixer 的参数可达性：`query` 与 `key_norm.weight` 不参与图，不得到梯度。
|
||||
|
||||
这是一项**训练期架构消融**，不是“只改变 forward、不改变 backward”的可分离因果实验。
|
||||
结果不能被翻译成 query/key 路径的纯因果效应。
|
||||
|
||||
## 3. 为什么只选四个新变体
|
||||
|
||||
固定 depth-32 / Block AttnRes 拓扑：
|
||||
|
||||
| scope | layers | 0-based depth mixer indices | count |
|
||||
|---|---:|---:|---:|
|
||||
| group 6 | 21–24 | 40–47 | 8 |
|
||||
| group 7 | 25–28 | 48–55 | 8 |
|
||||
| groups 6+7 | 21–28 | 40–55 | 16 |
|
||||
| group 7 MLP | 25–28 | 49, 51, 53, 55 | 4 |
|
||||
|
||||
四个新训练变体固定为：
|
||||
|
||||
1. `uniform_group_6_forward`
|
||||
2. `uniform_group_7_forward`
|
||||
3. `uniform_groups_6_7_forward`（主变体）
|
||||
4. `uniform_group_7_mlp_forward`
|
||||
|
||||
选择依据不是 Round 08 结果：
|
||||
|
||||
- joint 6+7 是 Round 07 的固定主 scope；
|
||||
- 单 group 6 / 7 用来构成交互图；
|
||||
- group 7 MLP-only 是 Round 07 的 branch-level 次级线索；
|
||||
- output mixer 保持 learned，避免把 local depth intervention 扩成全局 readout 改写。
|
||||
|
||||
不加入 attention-only、output-only、all-depth 或 all-65，是为了把本轮正式算力集中在
|
||||
“局部训练适应”而不是重新展开 14-mode backward census。
|
||||
|
||||
## 4. 冻结参考，而不是重新挑基线
|
||||
|
||||
paired learned reference 复用 Round 05 已公开的三个：
|
||||
|
||||
```text
|
||||
formal / depth-32 / block / seeds 2026073001..3
|
||||
```
|
||||
|
||||
它们与新变体使用相同：
|
||||
|
||||
- 初始化过程；
|
||||
- 8,000-step deterministic byte schedule；
|
||||
- 64 validation windows；
|
||||
- 16 diagnostic windows；
|
||||
- optimizer、LR schedule、BF16 forward 和 FP32 residual accumulation；
|
||||
- 六个 diagnostic checkpoints。
|
||||
|
||||
runner 的 `learned_reference` smoke mode 必须逐字段复现父 runner 数值。正式 analyzer
|
||||
还必须验证新旧 cell 的 initial public/mixer hashes、输入 gate tensor hashes、
|
||||
validation tensor hash、diagnostic tensor hash 与 deterministic / autocast 数值合同。
|
||||
GPU 名称、驱动 minor 与 timing 只记录，不作为配对 hard gate。
|
||||
|
||||
复用参考的意义是避免为已冻结、可复现的三格再消耗 196,608,000 target bytes。报告必须
|
||||
把它写成**历史配对参考**，不能写成同期随机对照。
|
||||
|
||||
## 5. 主问题与质量护栏
|
||||
|
||||
主对象沿用 Round 05–07，并且主判定只读取最终 step 8,000：
|
||||
|
||||
```text
|
||||
step = 8,000
|
||||
position = post-MLP Transformer-block output
|
||||
reduction = activation-gradient element RMS
|
||||
S = layers 21–25
|
||||
R = other 27 layers
|
||||
C = mean(g[S]) / mean(g[R])
|
||||
P = max(g) / mean(g)
|
||||
```
|
||||
|
||||
对变体 `v` 和同 seed learned reference：
|
||||
|
||||
```text
|
||||
D_X(v) = (X_ref - X_v) / X_ref
|
||||
```
|
||||
|
||||
主变体 `uniform_groups_6_7_forward` 只有在：
|
||||
|
||||
- `C` 与 `P` 的 `D_X ≥ 0.20`；
|
||||
- 三个 seed 的六个 cell 全部通过；
|
||||
- 每个 seed 的 final validation BPC 相对 reference 不劣化超过 `+0.05`；
|
||||
- 三 seed mean BPC delta 不超过 `+0.03`；
|
||||
|
||||
时才得到 “forward-training attenuation established within reduced protocol”。
|
||||
|
||||
质量阈值在结果前固定。它不是“能力保持”的通用标准，只用于阻止把明显训练失败造成的
|
||||
梯度塌缩计成正结果。即使通过，也不能推出下游能力等价。
|
||||
|
||||
## 6. 非加性局部交互图
|
||||
|
||||
对每个 seed、checkpoint 和指标：
|
||||
|
||||
```text
|
||||
E6 = ln(X_ref / X_group6)
|
||||
E7 = ln(X_ref / X_group7)
|
||||
E67 = ln(X_ref / X_groups6+7)
|
||||
I67 = E67 - E6 - E7
|
||||
```
|
||||
|
||||
- `I67 > 0`：joint log attenuation 大于两个 single effects 的和；
|
||||
- `I67 < 0`：joint log attenuation 小于两个 single effects 的和；
|
||||
- `I67 = 0`：只是在这个定义下恰好 log-additive。
|
||||
|
||||
`I67` 没有预注册显著性阈值，不是 Shapley value、方差分解、独立性检验或因果交互估计。
|
||||
三个 effect 来自三套独立训练，它只是跨 run 的 log-attenuation residual。它的用途是把
|
||||
训练轨迹中的补偿/放大关系画清楚，而不是制造一个新的“通过/失败”结论。
|
||||
|
||||
groups 6+7 覆盖 layers 21–28，而固定尖峰窗只到 layer 25；layers 26–28 落在 `R`。
|
||||
所以 `C` 的变化可能同时来自 `S` 下降与 `R` 上升。正式结果必须把两者拆开报告，不能把
|
||||
contrast 下降单独翻译成“尖峰层被关闭”。
|
||||
|
||||
## 7. 真实 K3 checkpoint 的同期边界
|
||||
|
||||
截至本轮预检，官方 Kimi-K3 Hugging Face main 仍停在 revision
|
||||
`9f62e4e9fffbd0a83ddd60e1c209d828994b3569`，remote code 仍按 96 heads 初始化
|
||||
`A_log`，公开 checkpoint header 仍为 `[128]`。社区 PR #144 / #150 仍是两个未合并、
|
||||
语义不同的候选修复；没有官方裁决。
|
||||
|
||||
所以本轮不下载约 1.56 TB 权重，不声称对真实 K3 forward 做了验证。缩小实验只继承
|
||||
Block AttnRes 的拓扑动机，不是 K3 checkpoint 的数值替身。
|
||||
|
||||
一手状态页：
|
||||
|
||||
- [Kimi-K3 official main](https://huggingface.co/moonshotai/Kimi-K3/tree/main)
|
||||
- [main `modeling_kimi_linear.py`](https://huggingface.co/moonshotai/Kimi-K3/blob/main/modeling_kimi_linear.py)
|
||||
- [community PR #144](https://huggingface.co/moonshotai/Kimi-K3/discussions/144)
|
||||
- [community PR #150](https://huggingface.co/moonshotai/Kimi-K3/discussions/150)
|
||||
|
||||
## 8. 本轮可证伪交付
|
||||
|
||||
Round 08 将在查看正式结果前完成：
|
||||
|
||||
1. 冻结协议与 machine-readable manifest；
|
||||
2. 实现一个 selector，而不是四份分叉 forward；
|
||||
3. 通过 empty-selector 父等价、step-0 uniform identity、selector census、参数不可达性、
|
||||
loss-scale 与输入 hash 闸门；
|
||||
4. 运行 4 variants × 3 seeds × 8,000 steps；
|
||||
5. 从初始化 replay 主变体 seed 2026073001；
|
||||
6. 由单一 analyzer 生成主判定、质量闸门、轨迹与非加性交互；
|
||||
7. 独立审阅机器可读结果；
|
||||
8. 以五视图交互实验接入网站、开源并发布。
|
||||
@@ -0,0 +1,198 @@
|
||||
# Kimi K3 第五轮前置审计：Attention Residuals 的“梯度更均匀”到底指什么
|
||||
|
||||
> 审计日期：2026-07-30（Asia/Shanghai）
|
||||
> 官方仓库：`MoonshotAI/Attention-Residuals@85e22310fe5ee860b4a023de312d791de8a5a5e6`
|
||||
> 官方 PDF SHA-256：`e5831b0db1347606453b5176b0142115a18887b6a9c2e1d05a266d4805a26b2f`
|
||||
> 结论性质：一手工件审计，不是新的实验结果
|
||||
|
||||
## 0. 先说结论
|
||||
|
||||
K3 Round 04 的反结果——Block AttnRes 的**核心参数梯度 RMS 跨层 CV 更高**——不能直接
|
||||
反驳 Attention Residuals 论文 Figure 5(c) 所说的“梯度分布更均匀”，因为两边很可能测的
|
||||
不是同一个对象：
|
||||
|
||||
```text
|
||||
Round 04：
|
||||
每个 Transformer block 内所有核心参数梯度拼接后的 RMS
|
||||
∇θL，θ = attention + MLP + 两个输入 norm 的参数
|
||||
|
||||
论文 Figure 5(c)：
|
||||
图题只写 “Each transformer block's gradient magnitude”
|
||||
结合 Figure 5(b) 的 block output magnitude 和正文，最自然的操作化是
|
||||
每个 block 输出 activation 的梯度 ∂L/∂h_l
|
||||
```
|
||||
|
||||
但“最自然”不等于“官方已经明确定义”。论文和当前官方仓库都没有给出足以唯一重建
|
||||
Figure 5(c) 的测量合同，也没有发布训练代码。因此，下一轮不会把自己的 activation-gradient
|
||||
定义冒充成论文原始实现，而会把它命名为：
|
||||
|
||||
> **与 Figure 5 叙述对齐的一种公开、冻结、可复现的 operationalization**
|
||||
|
||||
这一区分很重要：参数梯度回答“这一层的权重此刻收到多大更新信号”，activation 梯度回答
|
||||
“损失对这一深度的表征有多敏感”。二者相关，但不会因为链式法则而自动同方向。
|
||||
|
||||
---
|
||||
|
||||
## 1. 官方工件实际提供了什么
|
||||
|
||||
官方仓库在固定 revision 下只包含：
|
||||
|
||||
- `README.md`；
|
||||
- `Attention_Residuals.pdf`；
|
||||
- 论文图片资产；
|
||||
- citation 与外部入口。
|
||||
|
||||
仓库**不包含**：
|
||||
|
||||
- 模型或 residual mixer 的可执行实现；
|
||||
- Figure 5 的统计脚本；
|
||||
- 训练配置、日志或 checkpoint；
|
||||
- gradient hook、norm 和 reduction 定义；
|
||||
- 用于复画 Figure 5 的原始数组。
|
||||
|
||||
因此，本审计能固定论文的文字、公式、图与模型尺度，不能从官方代码恢复一个不存在的
|
||||
隐藏测量合同。
|
||||
|
||||
## 2. Figure 5 能确认的事实
|
||||
|
||||
官方 `training_dynamics.png` 和 PDF Figure 5 有三个并列面板：
|
||||
|
||||
| 面板 | 图题 | 横轴 |
|
||||
|---|---|---|
|
||||
| (a) | Validation Loss | training step |
|
||||
| (b) | Output Magnitude | Layer / Transformer Block Index |
|
||||
| (c) | Gradient Magnitude ×10⁻⁵ | Layer / Transformer Block Index |
|
||||
|
||||
Figure 5 caption 对 (b) 与 (c) 的完整对象描述分别是：
|
||||
|
||||
```text
|
||||
Each transformer block's output magnitude at the end of training.
|
||||
Each transformer block's gradient magnitude.
|
||||
```
|
||||
|
||||
正文明确表达了两个方向：
|
||||
|
||||
1. Baseline 的 hidden-state magnitude 随深度单调增长；Block AttnRes 把增长约束在局部
|
||||
residual block 内，形成周期性的深度图案。
|
||||
2. Baseline 的早期层梯度“不成比例地大”；Block AttnRes 的可学习 softmax 权重产生了
|
||||
“明显更均匀”的梯度分布。
|
||||
|
||||
图中最终大模型约有 27 个 Transformer blocks。论文的最终模型描述也是 27 个 Transformer
|
||||
blocks / 54 个 residual layers；Block AttnRes 每 6 个 residual layers 聚合一次，共 9 个
|
||||
聚合块，外加 embedding 形成 10 个跨块来源。
|
||||
|
||||
## 3. Figure 5 不能确认的事项
|
||||
|
||||
下面每一项都会改变曲线，却没有在论文或官方仓库中被唯一指定：
|
||||
|
||||
| 未定义项 | 至少两种合理解释 |
|
||||
|---|---|
|
||||
| 梯度对象 | block 输出 activation 梯度；block 参数梯度；分支输出梯度 |
|
||||
| block 输出位置 | attention+MLP 后；只在 MLP 后；进入下一层 norm 前；聚合块边界后 |
|
||||
| norm | L2 norm；RMS；mean absolute value；每 token norm 后再平均 |
|
||||
| reduction | batch/token/channel 联合；先按 token 再按 batch；只取末 token |
|
||||
| loss | token mean；sample mean；未归一化 sum；带或不带 mask |
|
||||
| 采样 | 一个 batch；多 batch 平均；训练流中的 moving average |
|
||||
| 时间点 | “训练结束”单点；末段平均；某个 checkpoint |
|
||||
| 数值阶段 | AMP 缩放前/后；gradient clipping 前/后；BF16 或 FP32 |
|
||||
| 运行模式 | train 或 eval；dropout 是否开启 |
|
||||
| 归一化 | 绝对值；再除全层均值；再除 Baseline |
|
||||
|
||||
Figure 5(c) 的纵轴是绝对 magnitude 标度，并不等价于 CV。只报告 CV 还会丢失两个信息：
|
||||
|
||||
- 全部层梯度是否一起缩小或放大；
|
||||
- 不均匀来自“早层系统性偏大”，还是某一个中间/末端尖峰。
|
||||
|
||||
所以 Round 05 必须同时公开绝对曲线、按层均值归一化曲线、CV 与前后深度分位比。
|
||||
|
||||
## 4. 为什么 activation gradient 是合理推断，但仍只是推断
|
||||
|
||||
把 Figure 5(c) 操作化为 `∂L/∂h_l` 有三条证据：
|
||||
|
||||
1. 它与 Figure 5(b) 的 “transformer block output magnitude” 在横轴和叙述上成对；
|
||||
2. “早期层的梯度”在表示传播语境中通常可由对 block output 保留梯度直接比较；
|
||||
3. 参数张量的大小和类型在 attention 与 MLP 间差异很大，若把参数拼接，论文通常需要说明
|
||||
聚合口径，否则 “each transformer block” 不是天然的单一标量。
|
||||
|
||||
但也有无法排除的替代解释：
|
||||
|
||||
- 论文作者可能测 block 参数梯度；
|
||||
- 可能测 residual branch output 而非完整 block output；
|
||||
- 可能先对每个 token 做 L2 norm，再跨 token 平均；
|
||||
- 可能在内部训练系统中有未公开的统一 telemetry 定义。
|
||||
|
||||
因此，网站和审计只说“与论文叙述对齐的公开定义”，不说“论文就是这样算的”，也不把
|
||||
数值和 Figure 5 纵轴直接对齐。
|
||||
|
||||
## 5. Round 04 与 Round 05 的对象对照
|
||||
|
||||
| 维度 | Round 04 已测对象 | Round 05 主对象 |
|
||||
|---|---|---|
|
||||
| 数学对象 | `∇θ_l L` | `∂L/∂h_l` |
|
||||
| `l` 的单位 | Transformer block | Transformer block |
|
||||
| 张量内容 | block 的核心参数 | block 的 post-MLP output activation |
|
||||
| 聚合 | 参数元素联合 RMS | batch×time×channel 联合 RMS |
|
||||
| 是否含 AttnRes 参数 | 否 | 不适用；梯度穿过 mixer |
|
||||
| 时间 | final diagnostic batch | 全部预注册 diagnostic steps |
|
||||
| 目的 | 权重更新信号是否均匀 | 表征深度的反向信号是否均匀 |
|
||||
|
||||
Round 04 的参数梯度结果不会被改名、删去或用新指标覆盖。它仍是一个有效反结果，只是不能
|
||||
代表论文未定义清楚的 Figure 5(c)。
|
||||
|
||||
## 6. 两种结构怎样取得真正对齐的 16 / 32 个位置
|
||||
|
||||
Grok Headless 被用作一次对抗式方法审阅，不作为事实来源。它正确指出了 non-leaf tensor、
|
||||
alias、AMP、clip 时点和只看 CV 的风险；但它也提出了一个不适用于本实现的担忧：
|
||||
“Block AttnRes 只有约 8 个 block 输出，无法与 Baseline 的 16 / 32 层对齐”。
|
||||
|
||||
这里要区分两种 block：
|
||||
|
||||
```text
|
||||
Transformer block：
|
||||
attention + MLP；depth=16 时始终有 16 个，depth=32 时始终有 32 个
|
||||
|
||||
AttnRes aggregation group：
|
||||
把若干 residual sublayers 的 partial sum 保存为一个跨组 source；
|
||||
两种深度都约为 8 组
|
||||
```
|
||||
|
||||
Round 05 在**每个 Transformer block 的 MLP 分支完成后**取 `h_l`。所以 Baseline 与 Block
|
||||
都有完全相同的 `l=1..depth`：
|
||||
|
||||
| 架构 | Round 05 的 `h_l` | shape |
|
||||
|---|---|---|
|
||||
| Baseline | 第 `l` 个 attention residual 与 MLP residual 都完成后的 hidden state | `[B,T,C]` |
|
||||
| Block | 第 `l` 个 MLP branch 加入后、可能保存并清空 aggregation partial **之前**的 partial output | `[B,T,C]` |
|
||||
|
||||
Block 的 `h_l` 是局部 residual partial，而不是跨组 source 列表；这正对应论文 Figure 5(b)
|
||||
所描述的“增长被限制在每个 block 内”的周期性图案。组边界前取值也避免把 reset 后的零张量
|
||||
错误当成 Transformer block output。
|
||||
|
||||
## 7. 实验实现必须通过的梯度测量闸门
|
||||
|
||||
正式训练前，四个结构格（2 个深度 × 2 个 residual graph）都必须证明：
|
||||
|
||||
1. 所有 `h_l` 都是不同的、非别名的捕获对象，数量严格等于 Transformer depth；
|
||||
2. `retain_grad()` 或 hook 后所有梯度非 `None`、finite，shape 与 activation 完全一致；
|
||||
3. diagnostic loss 使用固定输入、固定 token-mean CE、`eval()`、无 optimizer step；
|
||||
4. backward 发生在 parameter gradient clip 之前，且不经过 `GradScaler`；
|
||||
5. 将同一 diagnostic loss 精确乘 2 后，每层 activation-gradient RMS 也乘 2；
|
||||
6. 乘 2 前后的 CV、归一化曲线与深度分位比在数值容差内不变；
|
||||
7. 同配置全新进程重复运行，冻结诊断字段 exact。
|
||||
|
||||
若任一项失败，正式 8,000-step grid 不得开始。
|
||||
|
||||
## 8. 本审计带来的研究决策
|
||||
|
||||
下一轮不再把一个宽度 192、深度 16、训练 2,000 step 的参数梯度 CV 与论文最终模型图强行
|
||||
放在同一条结论线上，而是：
|
||||
|
||||
- 增加 depth 32；
|
||||
- 把训练预算扩为 8,000 step；
|
||||
- Baseline / Block 使用相同的 Transformer block index；
|
||||
- 在 6 个固定时点测 post-MLP activation gradient；
|
||||
- 绝对标度、归一化形状、CV、前后四分位失衡一起公开；
|
||||
- 参数梯度作为次要指标保留；
|
||||
- 预注册“支持 / 混合 / 不支持”规则，并公开全部反结果。
|
||||
|
||||
精确协议见 `research/K3_ATTNRES_GRADIENT_SCALE_PROTOCOL.md`。
|
||||
@@ -0,0 +1,545 @@
|
||||
# Kimi K3 第五轮：Attention Residuals 梯度定义与深度扩展实验审计
|
||||
|
||||
> 协议：`llm-atlas-k3-attnres-gradient-scale-v1`
|
||||
> 前置定义审计：`research/K3_ATTNRES_GRADIENT_DEFINITION_AUDIT.md`
|
||||
> 预注册协议：`research/K3_ATTNRES_GRADIENT_SCALE_PROTOCOL.md`
|
||||
> 数据清单：`experiments/k3/attnres_gradient/manifest.json`
|
||||
> 执行日期：2026-07-30
|
||||
> 设备：NVIDIA GeForce RTX 5090；PyTorch `2.11.0+cu128`
|
||||
|
||||
## 0. 先说结论
|
||||
|
||||
这一轮本来想澄清一个看似矛盾的问题：
|
||||
|
||||
```text
|
||||
论文 Figure 5：
|
||||
Block AttnRes 的梯度沿深度“明显更均匀”
|
||||
|
||||
本站 Round 04：
|
||||
Block AttnRes 的核心参数梯度 RMS 跨层 CV 反而更高
|
||||
```
|
||||
|
||||
一手工件审计先确认：论文没有公开 Figure 5(c) 的确切 gradient tensor、norm、reduction、
|
||||
diagnostic batch、AMP / clip 时点或统计代码。因此，Round 05 没有假装恢复作者的隐藏实现，
|
||||
而是冻结一个与 Figure 5 的 output/gradient 并列叙述对齐、可复现的定义：
|
||||
|
||||
```text
|
||||
h_l：
|
||||
第 l 个 Transformer block 完成 attention + MLP 后的 FP32 residual output
|
||||
|
||||
m_l：
|
||||
sqrt(mean((∂L / ∂h_l)² over batch × time × channel))
|
||||
|
||||
L：
|
||||
固定 16 × 256 targets 的 token-mean cross entropy
|
||||
```
|
||||
|
||||
结果不是一句“是”或“不是”，而是一个更有信息量的分解：
|
||||
|
||||
1. **Block 确实大幅缓解了早层整体偏大。**
|
||||
Baseline 首四分位梯度平均是末四分位的 `3.01–3.90×`；Block 把它改到
|
||||
`0.55–1.40×`。预注册的首尾失衡指标在 6 / 6 个 depth×seed 配对中都改善，
|
||||
depth-16 平均改善 `61.0%`，depth-32 平均改善 `72.0%`。
|
||||
|
||||
2. **但 Block 没有让整条深度谱更平。**
|
||||
它在中后段形成了局部尖峰，所以 population CV 在 6 / 6 个配对中都恶化：
|
||||
depth-16 平均相对恶化 `10.3%`，depth-32 平均相对恶化 `60.0%`。
|
||||
|
||||
3. **Block 的绝对 activation-gradient 平均尺度更小。**
|
||||
它只有 Baseline 的 `57.4%`（depth-16）和 `54.4%`（depth-32）。这意味着“首尾更接近”
|
||||
不能自动解释为所有层都获得更强更新信号。
|
||||
|
||||
4. **参数梯度仍与 Round 04 同方向。**
|
||||
核心参数梯度 CV 从 `0.416→0.683`（depth-16），从 `0.397→0.772`
|
||||
（depth-32）；Block 更不均匀。
|
||||
|
||||
5. **验证 BPC 在 6 / 6 配对中都更低，但不是同算力优势。**
|
||||
平均改善 `0.00894 BPC`（depth-16）与 `0.00987 BPC`（depth-32）；Block 实际 step
|
||||
time 是 Baseline 的约 `2.55–2.60×`，peak allocated memory 约 `2.15–2.16×`。
|
||||
|
||||
按看结果前冻结的联合判据，CV 与首尾失衡必须同时改善才算 support；必须同时恶化才算
|
||||
concern。这里二者方向相反，所以：
|
||||
|
||||
| depth | 预注册判定 |
|
||||
|---:|---|
|
||||
| 16 | **mixed / inconclusive at this depth** |
|
||||
| 32 | **mixed / inconclusive at this depth** |
|
||||
| 总判定 | **depth-dependent or inconclusive** |
|
||||
|
||||
最准确的中文总结是：
|
||||
|
||||
> 在这套公开 operationalization 中，Block AttnRes 把“早层系统性偏大”变成了“首尾更接近、
|
||||
> 但中后段有局部尖峰”的另一种梯度分布。它修正了一类失衡，却没有降低全层离散度。
|
||||
|
||||
这不复现论文 Figure 5 的数值，也不反驳一个没有公开测量合同的隐藏实现。
|
||||
|
||||
---
|
||||
|
||||
## 1. 为什么要另开一轮，而不是改写 Round 04
|
||||
|
||||
Round 04 的梯度对象是：
|
||||
|
||||
```text
|
||||
每个 Transformer block 的 attention、MLP 与两个 input norm
|
||||
所有核心参数梯度拼接后的 RMS
|
||||
```
|
||||
|
||||
这回答“该层权重收到多大更新信号”。Round 05 的对象是 `∂L/∂h_l`，回答“损失对该深度
|
||||
表征有多敏感”。链式法则把两者联系起来，但不会保证跨层形状同方向。
|
||||
|
||||
因此：
|
||||
|
||||
- Round 04 参数梯度反结果继续有效；
|
||||
- Round 05 不把它改名为 activation gradient；
|
||||
- 两种对象在网站并排展示；
|
||||
- 任何方向冲突都保留，而不是选择更像论文的一种。
|
||||
|
||||
官方工件边界见前置定义审计。固定 revision 为：
|
||||
|
||||
```text
|
||||
MoonshotAI/Attention-Residuals
|
||||
85e22310fe5ee860b4a023de312d791de8a5a5e6
|
||||
|
||||
Attention_Residuals.pdf SHA-256
|
||||
e5831b0db1347606453b5176b0142115a18887b6a9c2e1d05a266d4805a26b2f
|
||||
```
|
||||
|
||||
官方仓库没有可执行训练代码或 Figure 5 原始数组。
|
||||
|
||||
## 2. 实验规模与配对合同
|
||||
|
||||
正式网格：
|
||||
|
||||
```text
|
||||
2 depths
|
||||
× 2 residual graphs
|
||||
× 3 seeds
|
||||
× 8,000 steps
|
||||
× 32 windows
|
||||
× 256 target bytes
|
||||
= 12 independent runs
|
||||
= 786,432,000 formal target bytes
|
||||
```
|
||||
|
||||
| 项 | depth-16 | depth-32 |
|
||||
|---|---:|---:|
|
||||
| Transformer blocks | 16 | 32 |
|
||||
| residual sublayers | 32 | 64 |
|
||||
| AttnRes aggregation groups | 8 | 8 |
|
||||
| sublayers / group | 4 | 8 |
|
||||
| Transformer blocks / group | 2 | 4 |
|
||||
| width / heads / FFN | 192 / 6 / 768 | 192 / 6 / 768 |
|
||||
|
||||
每个 depth / seed 的 Baseline 与 Block：
|
||||
|
||||
- 使用逐 tensor exact 的公共主干初始化；
|
||||
- 使用逐 step / row exact 的 byte windows；
|
||||
- 使用相同 optimizer、LR schedule、batch、context 与 target-byte budget；
|
||||
- 分支线性计算走 BF16 autocast；
|
||||
- residual accumulator 与被测 `h_l` 都为 FP32；
|
||||
- 每个结构在全新进程中从零训练。
|
||||
|
||||
二者**不匹配**：
|
||||
|
||||
- mixer 参数；
|
||||
- mixer FLOPs；
|
||||
- step wall time；
|
||||
- activation memory。
|
||||
|
||||
所以 BPC 只能叫“同 token / step 预算对比”，不能叫“同算力优势”。
|
||||
|
||||
## 3. 数据与日程
|
||||
|
||||
| 对象 | 固定值 |
|
||||
|---|---|
|
||||
| dataset | `Salesforce/wikitext` |
|
||||
| revision | `b08601e04326c79dfdd32d625aee71d232d685c3` |
|
||||
| variant | `wikitext-2-raw-v1` |
|
||||
| train bytes | 10,951,563 |
|
||||
| train SHA-256 | `0ca7d3e7…e9b4` |
|
||||
| validation bytes | 1,148,008 |
|
||||
| validation SHA-256 | `a42356f6…e719` |
|
||||
| formal schedule cells | 768,000 |
|
||||
| schedule SHA-256 | `5041e09b…f4e` |
|
||||
| validation tensor SHA-256 | `f459316f…338` |
|
||||
| diagnostic tensor SHA-256 | `21117e31…716` |
|
||||
|
||||
固定诊断时点：
|
||||
|
||||
```text
|
||||
0, 100, 500, 2,000, 4,000, 8,000
|
||||
```
|
||||
|
||||
六个时点的全部 activation gradient、output RMS、parameter gradient、mixer 权重与验证 BPC
|
||||
都进入 raw JSON；没有只挑“最好看”的 checkpoint。
|
||||
|
||||
## 4. 正式训练前的故障与修订
|
||||
|
||||
### 4.1 第一次 smoke 的标量序列化错误
|
||||
|
||||
首个 depth-16 Baseline 20-step smoke 已完成数值计算，但在写 JSON 前失败：
|
||||
|
||||
```text
|
||||
RuntimeError:
|
||||
self.dim() cannot be 0 to view Float as Byte
|
||||
```
|
||||
|
||||
原因是 optimizer 的 step 是 0 维 tensor，hash helper 直接把它 `view(torch.uint8)`。修复为：
|
||||
|
||||
```text
|
||||
tensor.reshape(-1).view(torch.uint8)
|
||||
```
|
||||
|
||||
当时：
|
||||
|
||||
- 没有 formal 运行；
|
||||
- 没有输出 JSON；
|
||||
- 没有可供选择的正式结果。
|
||||
|
||||
### 4.2 smoke 发现被测 residual dtype 不一致
|
||||
|
||||
第一版 smoke 通过了 finite / loss×2 / replay 闸门，但检查 capture metadata 时发现：
|
||||
|
||||
```text
|
||||
Baseline post-MLP h_l：FP32
|
||||
Block aggregation partial：BF16
|
||||
```
|
||||
|
||||
原因：
|
||||
|
||||
- Baseline 把 BF16 branch 加到 FP32 embedding/residual stream；
|
||||
- Block 每组第一个 partial 直接引用 BF16 branch output。
|
||||
|
||||
这会把数值精度差异混进结构比较。正式训练前，协议与实现补充为：
|
||||
|
||||
```text
|
||||
BF16 branch output → 显式转 FP32 → residual partial 累加
|
||||
```
|
||||
|
||||
随后 4 个 depth×architecture 格的 smoke 全部从头重跑两次。正式输出是在这次修订之后才开始。
|
||||
|
||||
### 4.3 一次未启动模型的 zsh 调度错误
|
||||
|
||||
首个正式 Baseline 完成后，批处理脚本用 Bash 式标量切分处理 zsh 字符串，第一行就退出：
|
||||
|
||||
```text
|
||||
argument --architecture: invalid choice: ''
|
||||
```
|
||||
|
||||
runner 没有启动，也没有创建新结果文件。调度改为显式 `:` 分隔数组后继续。这是 orchestration
|
||||
故障，不是模型运行失败，但仍在审计时间线中保留。
|
||||
|
||||
## 5. 运行前与复现闸门
|
||||
|
||||
### 5.1 activation-gradient 测量闸门
|
||||
|
||||
4 / 4 个 depth×architecture 格都通过：
|
||||
|
||||
- 捕获数量严格等于 16 / 32；
|
||||
- shape 严格为 `[16,256,192]`；
|
||||
- dtype 全部为 FP32；
|
||||
- gradient 全部 present、finite；
|
||||
- capture storage 全部不别名；
|
||||
- diagnostic loss×2 后每层 gradient RMS 精确×2；
|
||||
- CV、归一化谱、首尾比不变。
|
||||
|
||||
### 5.2 两次独立 smoke
|
||||
|
||||
每个格都在两个全新进程中训练 20 steps。排除 timing 后的冻结字段：
|
||||
|
||||
| 格 | compare SHA-256 |
|
||||
|---|---|
|
||||
| depth-16 Baseline | `525cdcd9…bd2a` |
|
||||
| depth-16 Block | `e4330a0d…90cc` |
|
||||
| depth-32 Baseline | `8ee0dc37…8e56` |
|
||||
| depth-32 Block | `289073b1…34c` |
|
||||
|
||||
4 / 4 exact。
|
||||
|
||||
### 5.3 完整 formal replay
|
||||
|
||||
预注册格：
|
||||
|
||||
```text
|
||||
depth-32 / Block / seed-2026073001
|
||||
```
|
||||
|
||||
从初始化重新训练完整 8,000 steps，不加载 formal checkpoint。排除 `run_kind`、timing、
|
||||
memory 与进程元数据后的全部冻结字段：
|
||||
|
||||
```text
|
||||
formal compare SHA-256
|
||||
46300a452840a9dc6a5180efe7949cf3d81cc4471ed4a942da407e9343064817
|
||||
|
||||
replay compare SHA-256
|
||||
46300a452840a9dc6a5180efe7949cf3d81cc4471ed4a942da407e9343064817
|
||||
```
|
||||
|
||||
最终状态：
|
||||
|
||||
| 对象 | formal | replay |
|
||||
|---|---|---|
|
||||
| model state | `3f0b97ec…2f59` | `3f0b97ec…2f59` |
|
||||
| optimizer state | `ed03e6fb…4637` | `ed03e6fb…4637` |
|
||||
|
||||
字段级 exact。
|
||||
|
||||
## 6. 主结果：CV 与首尾失衡为什么方向相反
|
||||
|
||||
### 6.1 depth-16
|
||||
|
||||
| seed | Base CV | Block CV | 相对 CV 改善 | Base imbalance | Block imbalance | 相对 imbalance 改善 |
|
||||
|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 2026073001 | 0.41719 | 0.46768 | −12.1% | 1.29458 | 0.38821 | +70.0% |
|
||||
| 2026073002 | 0.43744 | 0.45175 | −3.3% | 1.35991 | 0.58958 | +56.6% |
|
||||
| 2026073003 | 0.41927 | 0.48391 | −15.4% | 1.29826 | 0.56693 | +56.3% |
|
||||
| 均值 | 0.42463 | 0.46778 | **−10.3%** | 1.31759 | 0.51491 | **+61.0%** |
|
||||
|
||||
这里“相对 CV 改善”为负，表示恶化。
|
||||
|
||||
原始首/末四分位比：
|
||||
|
||||
```text
|
||||
Baseline：3.65×, 3.90×, 3.66×
|
||||
Block： 0.68×, 0.55×, 0.57×
|
||||
```
|
||||
|
||||
Block 不只是把早层优势降到 1；它在三个 seed 中都略微“过冲”，变成末四分位平均更大。
|
||||
但 `abs(log(first/last))` 仍比 Baseline 更接近 0，所以失衡改善。
|
||||
|
||||
三 seed 平均 normalized activation-gradient 的最高点：
|
||||
|
||||
```text
|
||||
Baseline：layer 3 = 1.50× mean；layer 4 = 1.42×
|
||||
Block： layer 11 = 2.09× mean；layer 13 = 1.95×
|
||||
```
|
||||
|
||||
Baseline 是宽而平滑的早层隆起；Block 是更局部的中后段尖峰。CV 对尖峰敏感，所以升高。
|
||||
|
||||
### 6.2 depth-32
|
||||
|
||||
| seed | Base CV | Block CV | 相对 CV 改善 | Base imbalance | Block imbalance | 相对 imbalance 改善 |
|
||||
|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 2026073001 | 0.36109 | 0.64203 | −77.8% | 1.10030 | 0.20781 | +81.1% |
|
||||
| 2026073002 | 0.37162 | 0.72997 | −96.4% | 1.14087 | 0.43810 | +61.6% |
|
||||
| 2026073003 | 0.40320 | 0.42685 | −5.9% | 1.25169 | 0.33417 | +73.3% |
|
||||
| 均值 | 0.37864 | 0.59962 | **−60.0%** | 1.16429 | 0.32669 | **+72.0%** |
|
||||
|
||||
原始首/末四分位比：
|
||||
|
||||
```text
|
||||
Baseline：3.01×, 3.13×, 3.50×
|
||||
Block： 0.81×, 0.65×, 1.40×
|
||||
```
|
||||
|
||||
三 seed 平均 normalized spectrum 的 Block 峰值：
|
||||
|
||||
```text
|
||||
layer 21 = 3.04× mean
|
||||
layer 22 = 2.41×
|
||||
layer 23 = 1.91×
|
||||
layer 25 = 1.77×
|
||||
```
|
||||
|
||||
depth-32 每个 AttnRes aggregation group 含 4 个 Transformer blocks。21–24 是第 6 组，
|
||||
25–28 是第 7 组。尖峰集中在这两个中后段组附近，是数据中直接可见的结构；但仅凭本实验
|
||||
不能断言 pseudo-query、某个 source 或组边界是唯一因果。
|
||||
|
||||
### 6.3 seed-3 的中期反例
|
||||
|
||||
depth-32 seed-3 在 step 2,000：
|
||||
|
||||
```text
|
||||
Baseline CV 0.39965
|
||||
Block CV 0.34914
|
||||
```
|
||||
|
||||
此时 Block 更平；到 step 8,000 才变成：
|
||||
|
||||
```text
|
||||
Baseline CV 0.40320
|
||||
Block CV 0.42685
|
||||
```
|
||||
|
||||
前两个 seed 在 step 2,000 已明显恶化，seed-3 没有。网站必须保留 seed switch，不能用最终
|
||||
均值倒写成“三个 seed 从头到尾都一样”。
|
||||
|
||||
## 7. 绝对梯度尺度：更平不等于更强
|
||||
|
||||
最终 activation-gradient mean：
|
||||
|
||||
| depth | Baseline | Block | Block / Baseline |
|
||||
|---:|---:|---:|---:|
|
||||
| 16 | 约 `2.02×10⁻⁴` | 约 `1.16×10⁻⁴` | **0.574×** |
|
||||
| 32 | 约 `1.44×10⁻⁴` | 约 `0.78×10⁻⁴` | **0.544×** |
|
||||
|
||||
所以 Block 的首尾比更接近 1，并不是因为它把晚层全部抬高到 Baseline 早层的强度。更接近的
|
||||
描述是：
|
||||
|
||||
> 整体尺度下降，早层系统性高值被削弱，同时某些中后段位置相对全层均值形成尖峰。
|
||||
|
||||
这也是只看 normalized curve 或只看 CV 都不够的原因。
|
||||
|
||||
## 8. 参数梯度没有翻转 Round 04
|
||||
|
||||
最终核心参数梯度 CV：
|
||||
|
||||
| depth | Baseline mean | Block mean | Block / Baseline |
|
||||
|---:|---:|---:|---:|
|
||||
| 16 | 0.41596 | 0.68287 | 1.64× |
|
||||
| 32 | 0.39661 | 0.77176 | 1.95× |
|
||||
|
||||
6 / 6 个配对中 Block 都更高。Round 04 的反结果不是在把梯度对象改成 activation 后自动消失；
|
||||
两种梯度对象在本轮 final endpoint 都显示更高的跨层 CV。
|
||||
|
||||
但 activation gradient 又显示首尾失衡大幅改善，这说明“均匀”至少要拆成：
|
||||
|
||||
```text
|
||||
首尾是否平衡
|
||||
全层是否有尖峰
|
||||
绝对尺度多大
|
||||
参数更新信号是否平衡
|
||||
```
|
||||
|
||||
一个标量不能代替全部。
|
||||
|
||||
## 9. Output RMS：最接近论文叙述的正向结果
|
||||
|
||||
三 seed 最终 post-MLP output RMS 的最后/第一层比：
|
||||
|
||||
| depth | Baseline | Block |
|
||||
|---:|---:|---:|
|
||||
| 16 | 4.59× | 1.17× |
|
||||
| 32 | 6.08× | 1.89× |
|
||||
|
||||
Baseline output magnitude 随深度明显累积；Block 把增长限制在 aggregation group 内并产生
|
||||
周期性 reset。这个缩小实验的 output-RMS 方向与论文 Figure 5(b) 的叙述一致。
|
||||
|
||||
仍不能把数值直接叠到论文图上：
|
||||
|
||||
- 模型宽度、深度和数据不同；
|
||||
- 训练 Token 相差巨大；
|
||||
- 论文图的确切 output norm / reduction 也未完整公开；
|
||||
- 本实验的 Block group 是 8 组固定设计。
|
||||
|
||||
## 10. 验证 BPC 与真实成本
|
||||
|
||||
最终 BPC：
|
||||
|
||||
### depth-16
|
||||
|
||||
| seed | Baseline | Block | Block − Base |
|
||||
|---:|---:|---:|---:|
|
||||
| 2026073001 | 1.74880 | 1.73739 | −0.01141 |
|
||||
| 2026073002 | 1.73683 | 1.73118 | −0.00565 |
|
||||
| 2026073003 | 1.73637 | 1.72661 | −0.00976 |
|
||||
| 均值 | 1.74067 | 1.73173 | **−0.00894** |
|
||||
|
||||
### depth-32
|
||||
|
||||
| seed | Baseline | Block | Block − Base |
|
||||
|---:|---:|---:|---:|
|
||||
| 2026073001 | 1.71790 | 1.71235 | −0.00554 |
|
||||
| 2026073002 | 1.72698 | 1.70932 | −0.01765 |
|
||||
| 2026073003 | 1.70951 | 1.70310 | −0.00642 |
|
||||
| 均值 | 1.71813 | 1.70826 | **−0.00987** |
|
||||
|
||||
6 / 6 为负。这是有价值的次要方向，但 Round 05 没有为 BPC 再预注册一个新的 support 阈值，
|
||||
因此不追加事后显著性结论。
|
||||
|
||||
真实成本：
|
||||
|
||||
| depth | Base mean ms | Block mean ms | time ratio | Base peak alloc | Block peak alloc | memory ratio |
|
||||
|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 16 | 21.47 | 54.71 | 2.55× | 3.04 GB | 6.54 GB | 2.15× |
|
||||
| 32 | 42.11 | 109.38 | 2.60× | 5.94 GB | 12.82 GB | 2.16× |
|
||||
|
||||
这个简单 eager 实现没有论文训练系统的 kernel、并行或工程优化；成本数值不应外推到 K3。
|
||||
但它足以说明本站的 BPC 对比不是同 wall time / FLOPs。
|
||||
|
||||
## 11. 预注册判定为何是 mixed
|
||||
|
||||
支持需要：
|
||||
|
||||
```text
|
||||
CV：3 / 3 seeds 改善，平均相对改善 ≥20%
|
||||
AND
|
||||
imbalance：3 / 3 seeds 改善，平均相对改善 ≥20%
|
||||
```
|
||||
|
||||
concern 需要两项都以相同规则恶化。
|
||||
|
||||
实际：
|
||||
|
||||
```text
|
||||
depth-16：
|
||||
CV 3 / 3 恶化，平均 10.3%
|
||||
imbalance 3 / 3 改善，平均 61.0%
|
||||
|
||||
depth-32：
|
||||
CV 3 / 3 恶化，平均 60.0%
|
||||
imbalance 3 / 3 改善，平均 72.0%
|
||||
```
|
||||
|
||||
两个指标相反，所以两个 depth 都是 `mixed / inconclusive at this depth`。这不是“数据没规律”，
|
||||
而是预注册的“更均匀”概念被实验拆成了两个方向相反的组成部分。
|
||||
|
||||
## 12. 开放工件与校验哈希
|
||||
|
||||
| 工件 | SHA-256 |
|
||||
|---|---|
|
||||
| manifest | `080afb17…6371` |
|
||||
| protocol | `f772629b…ce22` |
|
||||
| definition audit | `79221c56…cc6` |
|
||||
| runner | `04ae69e1…800f` |
|
||||
| analyzer | `017d38d9…2fc8` |
|
||||
| aggregate canonical | `69be133c…7b51` |
|
||||
| compact canonical | `8cdb7180…044f` |
|
||||
| reproduction canonical | `addb2e59…f68a` |
|
||||
|
||||
公开目录包含：
|
||||
|
||||
- 12 个完整 formal raw JSON；
|
||||
- 8 个两套 smoke raw JSON；
|
||||
- 1 个完整 replay raw JSON；
|
||||
- 完整 aggregate；
|
||||
- 网站 compact payload；
|
||||
- manifest、runner、analyzer 与 reproduction 清单；
|
||||
- 前置定义审计、本协议和本结果审计。
|
||||
|
||||
`experiments/k3/attnres_gradient/reproduction.json` 记录每个 raw 文件 SHA-256。
|
||||
|
||||
## 13. 允许和禁止的结论
|
||||
|
||||
允许：
|
||||
|
||||
> 在本轮公开定义下，Block AttnRes 一致缓解了首/末深度四分位失衡，但在中后段形成局部
|
||||
> 梯度尖峰，导致全层 CV 一致升高；因此“梯度更均匀”必须拆成多个指标解释。
|
||||
|
||||
> 同 token / step 预算下，Block 的最终验证 BPC 在 6 / 6 个配对中更低，但实际运行成本
|
||||
> 约为 Baseline 的 2.6× step time 与 2.2× peak allocated memory。
|
||||
|
||||
禁止：
|
||||
|
||||
- “复现了论文 Figure 5(c)”；
|
||||
- “论文的梯度结论是错的”；
|
||||
- “已测到 Kimi K3 checkpoint 的真实梯度”；
|
||||
- “Block 解决了梯度消失 / 爆炸”；
|
||||
- “CV 更高就代表训练一定更不稳定”；
|
||||
- “BPC 改善是同 FLOPs / wall time 优势”；
|
||||
- 从 3 seeds 推导总体显著性；
|
||||
- 从 depth 16 / 32 外推到 48B、1T+400B Token 或 K3 2.8T 参数。
|
||||
|
||||
## 14. 下一步最值得问什么
|
||||
|
||||
这轮已经把“梯度”从一个模糊词拆成了可复查对象。下一个有价值的问题不是再换一个漂亮
|
||||
汇总指标，而是追踪局部尖峰从哪里来：
|
||||
|
||||
1. 分开捕获 pre-attention 与 pre-MLP residual positions；
|
||||
2. 把 layer 21–25 的 activation-gradient 与 mixer source weights 同步对齐；
|
||||
3. 比较 aggregation-group boundary 前后；
|
||||
4. 在不改变 formal 结果的前提下，对相同 raw gradient tensor 做多种公开 reduction
|
||||
sensitivity analysis；
|
||||
5. 若官方之后发布 Figure 5 telemetry 代码，再按其定义单独开新 protocol。
|
||||
|
||||
这些属于后续轮次，不能倒写进本轮预注册结论。
|
||||
@@ -0,0 +1,459 @@
|
||||
# Kimi K3 第五轮：Attention Residuals 梯度定义与深度扩展实验协议
|
||||
|
||||
> 协议 ID：`llm-atlas-k3-attnres-gradient-scale-v1`
|
||||
> 冻结日期：2026-07-30（Asia/Shanghai）
|
||||
> 状态：正式输出前预注册
|
||||
> 前置定义审计：`research/K3_ATTNRES_GRADIENT_DEFINITION_AUDIT.md`
|
||||
|
||||
## 0. 目标与一句话研究问题
|
||||
|
||||
Round 04 在缩小模型中得到两个同时成立的结果：
|
||||
|
||||
- Full / Block AttnRes 的 2,000-step 验证 BPC 都优于 Baseline；
|
||||
- 以“每个 Transformer block 的核心**参数**梯度 RMS”定义时，跨深度 CV 反而更高。
|
||||
|
||||
论文 Figure 5(c) 没有公开足以唯一恢复的梯度测量合同。Round 05 不猜作者的隐藏代码，而是
|
||||
冻结一个可复现、与 Figure 5 的 output/gradient 并列叙述对齐的 activation-gradient 定义，
|
||||
再问：
|
||||
|
||||
> 当深度从 16 增至 32、训练预算从 2,000 增至 8,000 step 时，Block AttnRes 是否比
|
||||
> PreNorm Baseline 更一致地降低 post-MLP block-output activation gradient 的跨深度失衡？
|
||||
|
||||
这不是 K3 checkpoint forward，也不是论文 Figure 5 数值复画。
|
||||
|
||||
## 1. 一手来源与不可补写的空白
|
||||
|
||||
| 工件 | 固定 revision / checksum | 用途 |
|
||||
|---|---|---|
|
||||
| Attention Residuals GitHub | `85e22310fe5ee860b4a023de312d791de8a5a5e6` | 公式、Figure 5 / 8、模型尺度 |
|
||||
| `Attention_Residuals.pdf` | SHA-256 `e5831b0d…a26b2f` | 论文一手图文 |
|
||||
| WikiText-2 raw | `Salesforce/wikitext@b08601e04326c79dfdd32d625aee71d232d685c3` | 固定公开训练语料 |
|
||||
| Round 04 protocol | `llm-atlas-k3-attnres-reduced-v1` | 公共主干与数据合同来源 |
|
||||
|
||||
官方仓库没有模型实现、训练脚本、checkpoint 或 Figure 5 原始数据。以下字段不能归因给论文：
|
||||
|
||||
- Figure 5 的确切 gradient tensor；
|
||||
- norm / reduction；
|
||||
- diagnostic batch；
|
||||
- AMP / clipping 时点；
|
||||
- 单点还是时间平均。
|
||||
|
||||
Grok Headless 只进行一次对抗式方法检查；其建议和错误都在前置审计中公开，不是事实来源。
|
||||
|
||||
## 2. 设计总览
|
||||
|
||||
```text
|
||||
2 个深度：16 / 32 Transformer blocks
|
||||
× 2 个 residual graph：PreNorm Baseline / Block AttnRes
|
||||
× 3 个冻结 seed
|
||||
× 8,000 training steps
|
||||
× 32 windows/step
|
||||
× 256 target bytes/window
|
||||
= 12 个正式训练格
|
||||
= 786,432,000 target bytes
|
||||
```
|
||||
|
||||
只比较 Baseline 与 Block，因为论文 Figure 5 的训练动力学面板也是这两个结构的直接对照。
|
||||
Round 04 的 Full AttnRes 结果保持公开，但本轮不增加一个与主问题无关的 6-run 分支。
|
||||
|
||||
## 3. 数据合同
|
||||
|
||||
继承 Round 04 的语料和预处理：
|
||||
|
||||
1. 按固定 parquet 行序读取 `text`；
|
||||
2. 每行追加一个 `\n`；
|
||||
3. UTF-8 编码，无 normalization、strip、去空行或大小写改写；
|
||||
4. byte vocabulary `0..255`；
|
||||
5. 每个窗口连续取 257 bytes，前 256 预测后 256。
|
||||
|
||||
固定拼接后 split：
|
||||
|
||||
| split | bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| train | 10,951,563 | `0ca7d3e74dbe44564ea5942b85232f1bbcb525c9cd481cd5d28a87ee90e7e9b4` |
|
||||
| validation | 1,148,008 | `a42356f6a8ff1d25daf25ec9db49e10a537c265581b61c74604bb63231dee719` |
|
||||
| test | 1,292,014 | `bfe9eb16ab9987fb88bde4ea9a30a00f2a45db01dfc14bad78d05325789c4f12` |
|
||||
|
||||
训练第 `step`、第 `row` 的 window 起点：
|
||||
|
||||
```text
|
||||
z = first 8 bytes of SHA256(
|
||||
protocol_id + "\0train-window\0" + seed + "\0" + step + "\0" + row
|
||||
)
|
||||
start = uint64_be(z) mod (len(train_bytes) - 257)
|
||||
```
|
||||
|
||||
同一 seed 的 4 个结构格逐 step / row 使用完全相同的 token tensor。validation 64 windows、
|
||||
diagnostic 16 windows，分别由标签 `validation-window` / `diagnostic-window` 与固定 index
|
||||
生成，对全部结构与 seed 相同。
|
||||
|
||||
正式运行前 manifest 必须记录 parquet hash、split bytes/hash、全部 `3×8,000×32=768,000`
|
||||
唯一训练窗口起点的 schedule hash、validation tensor hash 与 diagnostic tensor hash。
|
||||
|
||||
## 4. 模型合同
|
||||
|
||||
### 4.1 两个深度共享的结构
|
||||
|
||||
| 项 | 固定值 |
|
||||
|---|---:|
|
||||
| vocabulary | 256 bytes |
|
||||
| context | 256 |
|
||||
| `d_model` | 192 |
|
||||
| heads / head dimension | 6 / 32 |
|
||||
| `d_ff` | 768 |
|
||||
| dropout | 0 |
|
||||
| positional embedding | learned absolute，256 × 192 |
|
||||
| norm | RMSNorm，`eps=1e-6` |
|
||||
| attention | causal MHA；score 以 FP32 softmax |
|
||||
| MLP | bias-free SwiGLU，`192→768`, `192→768`, `768→192` |
|
||||
| embedding / readout | tied；final RMSNorm 后乘 token embedding |
|
||||
|
||||
全部 bias-free linear 与 embedding 初始化为 `N(0,0.02)`。attention output projection 与 MLP
|
||||
down projection 的标准差为 `0.02 / sqrt(2×depth)`；普通 RMSNorm 为 1。
|
||||
|
||||
数值精度进一步固定为：attention / MLP 线性分支受 BF16 autocast；embedding、Baseline hidden
|
||||
residual stream 与 Block aggregation partial 都以 FP32 累加。也就是说，Block 每个 BF16
|
||||
branch output 在进入 `partial` 前显式转为 FP32。这样两种结构被捕获的 `h_l` 都是 FP32，
|
||||
不会把 residual accumulator 精度差异混进梯度形状对比。
|
||||
|
||||
### 4.2 深度与 Block AttnRes 聚合
|
||||
|
||||
| Transformer depth | residual sublayers | aggregation groups | sublayers/group | Transformer blocks/group |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 16 | 32 | 8 | 4 | 2 |
|
||||
| 32 | 64 | 8 | 8 | 4 |
|
||||
|
||||
Baseline 子层为：
|
||||
|
||||
```text
|
||||
h ← h + f(RMSNorm(h))
|
||||
```
|
||||
|
||||
Block AttnRes：
|
||||
|
||||
- embedding 永远是 source 0；
|
||||
- 对已经完成的 aggregation-group sums 做跨组 softmax mixture；
|
||||
- 组内 attention / MLP branch output 累加到 `partial`；
|
||||
- 达到组边界时，保存完整 `partial` 为新 source，再开始下一组；
|
||||
- output mixer 聚合 embedding + 8 个完整 group sums。
|
||||
|
||||
每个子层的 pseudo-query 为 `d_model` 向量，严格 zero-init；每个 source 的 key RMSNorm weight
|
||||
严格 one-init。Baseline / Block 不要求总参数量或 residual-mixer FLOPs 相等，但同 seed /
|
||||
depth 的 token/position embedding、attention、MLP、input norm、final norm 和 tied readout
|
||||
必须逐 tensor SHA-256 exact。
|
||||
|
||||
## 5. 训练合同
|
||||
|
||||
| 项 | 固定值 |
|
||||
|---|---:|
|
||||
| seeds | `2026073001, 2026073002, 2026073003` |
|
||||
| formal steps | 8,000 |
|
||||
| batch | 32 |
|
||||
| context | 256 |
|
||||
| target bytes / run | 65,536,000 |
|
||||
| optimizer | AdamW |
|
||||
| betas / epsilon | `(0.9,0.95)` / `1e-8` |
|
||||
| peak / min LR | `3e-4` / `3e-5` |
|
||||
| warmup | 400 steps，linear |
|
||||
| decay | cosine，step 400→8,000 |
|
||||
| weight decay | `0.1` if `ndim>=2`，否则 `0` |
|
||||
| parameter grad clip | global norm `1.0` |
|
||||
| compute | BF16 autocast；FP32 optimizer state |
|
||||
| compile | off / eager |
|
||||
| device | one RTX 5090 |
|
||||
| deterministic | deterministic algorithms；`CUBLAS_WORKSPACE_CONFIG=:4096:8` |
|
||||
|
||||
验证和诊断都发生在：
|
||||
|
||||
```text
|
||||
step 0, 100, 500, 2,000, 4,000, 8,000
|
||||
```
|
||||
|
||||
验证固定 64 windows，8 windows/eval batch，报告 token-mean CE nats 与
|
||||
`bits_per_byte = CE / ln(2)`。诊断固定 16 windows，一次性输入，不改变 optimizer state。
|
||||
|
||||
计时合同：
|
||||
|
||||
- 前 20 个 training step 不计入；
|
||||
- step 21–8,000 每步前后 CUDA synchronize；
|
||||
- step 20 后 reset peak memory；
|
||||
- 报告 mean / median / p95 step ms、peak allocated / reserved；
|
||||
- timing、wall clock、hostname、GPU temperature 不进入 exact replay 字段。
|
||||
|
||||
## 6. 主梯度对象：精确到代码位置
|
||||
|
||||
### 6.1 `h_l` 定义
|
||||
|
||||
对 `l=1..depth`，统一在第 `l` 个 Transformer block 的 MLP branch 完成后捕获：
|
||||
|
||||
| 架构 | 精确定义 |
|
||||
|---|---|
|
||||
| Baseline | attention residual 与 MLP residual 都完成后的 hidden state |
|
||||
| Block | MLP branch 已加入、本 aggregation partial 可能保存/reset **之前**的 partial |
|
||||
|
||||
所有 `h_l.shape = [16,256,192]`、dtype 为 FP32。实现必须为 diagnostic forward 返回独立
|
||||
引用列表，不允许捕获 reset 后的零张量，不允许把 8 个 aggregation sources 当成 16 / 32 个
|
||||
Transformer outputs。
|
||||
|
||||
### 6.2 diagnostic loss 与 gradient magnitude
|
||||
|
||||
```text
|
||||
model.eval()
|
||||
logits, h[1..depth] = forward(fixed_diagnostic_x, capture=true)
|
||||
L = mean(cross_entropy(logits.float(), fixed_diagnostic_y))
|
||||
backward(L) # 不使用 GradScaler，不执行 optimizer.step
|
||||
m_l = sqrt(mean(float32(h_l.grad)² over batch×time×channel))
|
||||
```
|
||||
|
||||
规则：
|
||||
|
||||
- forward 仍使用与训练一致的 BF16 autocast；
|
||||
- logits 在 FP32 中计算 CE；
|
||||
- loss 对全部 `16×256` targets 做算术平均，无 mask、无 label smoothing；
|
||||
- backward 前 model / optimizer gradients 清零；
|
||||
- activation gradient 在任何 parameter clipping 之前读取；
|
||||
- diagnostic 不消耗训练数据，不进入 optimizer，不改变学习率或模型状态。
|
||||
|
||||
### 6.3 同位置 output magnitude
|
||||
|
||||
同一批 `h_l` 计算：
|
||||
|
||||
```text
|
||||
o_l = sqrt(mean(float32(h_l)² over batch×time×channel))
|
||||
```
|
||||
|
||||
它用于显示 Baseline 单调累积与 Block 组内周期，而不作为主 confirmatory endpoint。
|
||||
|
||||
## 7. 主指标与预注册判据
|
||||
|
||||
每个 diagnostic step 保存完整 `m_1..m_depth`。以下统计由未四舍五入的 float64 数组计算。
|
||||
|
||||
### 7.1 绝对尺度
|
||||
|
||||
```text
|
||||
mean_grad = mean_l(m_l)
|
||||
```
|
||||
|
||||
它防止“曲线更平只是全部梯度趋近于零”被 CV 隐藏。绝对尺度不设置优劣阈值，只公开。
|
||||
|
||||
### 7.2 归一化谱与 CV
|
||||
|
||||
```text
|
||||
n_l = m_l / mean_grad
|
||||
CV = population_std_l(m_l) / mean_grad
|
||||
```
|
||||
|
||||
使用 population standard deviation（`ddof=0`）。网站必须同时显示 `m_l` 与 `n_l`，不能只显示
|
||||
CV 排名。
|
||||
|
||||
### 7.3 前后四分位失衡
|
||||
|
||||
```text
|
||||
q = depth / 4
|
||||
Q_first = mean(m_1 .. m_q)
|
||||
Q_last = mean(m_(depth-q+1) .. m_depth)
|
||||
imbalance = abs(ln(Q_first / Q_last))
|
||||
```
|
||||
|
||||
depth 16 时各取 4 层；depth 32 时各取 8 层。`imbalance=0` 才表示首尾一致；这个定义不会把
|
||||
“早层偏大”和“晚层偏大”错误地都解释为越小越好。原始有符号比 `Q_first/Q_last` 仍公开。
|
||||
|
||||
### 7.4 seed 内配对对比
|
||||
|
||||
只在最终 step 8,000 做 confirmatory verdict：
|
||||
|
||||
```text
|
||||
relative_CV_reduction
|
||||
= (CV_baseline - CV_block) / CV_baseline
|
||||
|
||||
relative_imbalance_reduction
|
||||
= (imbalance_baseline - imbalance_block) / imbalance_baseline
|
||||
```
|
||||
|
||||
若 Baseline imbalance 精确为 0，则该 seed 的 relative imbalance reduction 定义为不可计算，
|
||||
该 depth 自动不能得到“联合支持”；仍公开绝对差。
|
||||
|
||||
对每个 depth 分别判定：
|
||||
|
||||
- **joint directional support at this depth**：三个 seed 的 CV reduction 都 `>0`，其均值
|
||||
`>=20%`；同时三个 seed 的 imbalance reduction 都 `>0`，其均值 `>=20%`。
|
||||
- **joint directional concern at this depth**：三个 seed 的两项 reduction 都 `<0`，且两项
|
||||
平均相对恶化都 `>=20%`。
|
||||
- 其他：**mixed / inconclusive at this depth**。
|
||||
|
||||
总判定：
|
||||
|
||||
- 两个 depth 都 support：**scale-consistent directional support in this operationalization**；
|
||||
- 两个 depth 都 concern：**scale-consistent directional concern in this operationalization**；
|
||||
- 其他：**depth-dependent or inconclusive**。
|
||||
|
||||
不计算 p-value、population CI，不把 3 seeds 称为统计证明。
|
||||
|
||||
## 8. 必须公开的次要指标
|
||||
|
||||
### 8.1 全时间轨迹
|
||||
|
||||
六个预注册时点的以下数据必须全部公开，不能选择“最好看”的 checkpoint：
|
||||
|
||||
- validation BPC；
|
||||
- absolute activation-gradient spectrum；
|
||||
- normalized activation-gradient spectrum；
|
||||
- activation CV；
|
||||
- first/last quartile ratio 与 imbalance；
|
||||
- output RMS spectrum；
|
||||
- mean activation-gradient scale。
|
||||
|
||||
### 8.2 参数梯度
|
||||
|
||||
延续 Round 04 定义：每个 Transformer block 的 attention、MLP 与两个 input norm 的参数梯度
|
||||
拼接后计算：
|
||||
|
||||
```text
|
||||
parameter_grad_rms[l] = sqrt(sum(g²) / total_parameter_elements)
|
||||
```
|
||||
|
||||
不含 embedding、final norm、LM head 与 AttnRes mixer 参数；clip 前读取。报告完整谱、CV 和
|
||||
前后四分位失衡，但它们不进入 Round 05 主判定。
|
||||
|
||||
### 8.3 mixer / 成本
|
||||
|
||||
Block 同报：
|
||||
|
||||
- 各子层 softmax mixture 的 source-depth 分布；
|
||||
- output mixer 分布；
|
||||
- entropy 与 embedding / latest-complete-group mass；
|
||||
- step time、peak allocated/reserved、参数量。
|
||||
|
||||
这些用于解释机制与成本，不改变 confirmatory verdict。
|
||||
|
||||
## 9. 运行前闸门
|
||||
|
||||
### 9.1 数据闸门
|
||||
|
||||
- split bytes/hash 与 Round 04 exact；
|
||||
- 新 protocol 的 768,000-window schedule hash 落盘；
|
||||
- validation / diagnostic tensor hash 落盘；
|
||||
- 同 seed 四结构的至少 step 0 / 1 / 7,999 输入 tensor hash exact。
|
||||
|
||||
### 9.2 公共权重闸门
|
||||
|
||||
每个 depth / seed 的 Baseline 与 Block 公共参数逐 tensor exact；输出：
|
||||
|
||||
- 公共参数 tensor 数；
|
||||
- 公共参数 element 数；
|
||||
- name / shape / dtype / bytes 联合 hash；
|
||||
- value bytes 联合 hash。
|
||||
|
||||
### 9.3 activation-gradient 闸门
|
||||
|
||||
四个 depth×architecture 格都必须通过：
|
||||
|
||||
1. 捕获数量严格等于 depth，shape 均为 `[16,256,192]`；
|
||||
2. 全部 gradient 非 `None`、finite、storage 不别名；
|
||||
3. 同输入把 loss 乘 2 后，每层 `m_l` 比值在 `2±1e-5`；
|
||||
4. loss×2 前后 CV、normalized spectrum、quartile ratio 在 `1e-6` 绝对容差内；
|
||||
5. 全新进程重复 smoke 的冻结字段 exact。
|
||||
|
||||
### 9.4 smoke
|
||||
|
||||
四个格都运行 20 training steps；冻结字段包括：
|
||||
|
||||
- protocol、architecture、depth、seed、device、dtype；
|
||||
- data/schedule/tensor hashes；
|
||||
- public-weight hashes；
|
||||
- step 0 / 20 loss 与 validation；
|
||||
- 全部预注册 diagnostic 数组；
|
||||
- finite / alias / loss-scale checks。
|
||||
|
||||
smoke 不能写入 formal 目录。
|
||||
|
||||
## 10. 正式执行与独立 replay
|
||||
|
||||
12 个 formal grid 必须各自在全新进程中执行。目录键为：
|
||||
|
||||
```text
|
||||
depth-{16|32}/{baseline|block}/seed-{2026073001|2026073002|2026073003}
|
||||
```
|
||||
|
||||
完成后预注册 replay：
|
||||
|
||||
```text
|
||||
depth-32 / block / seed-2026073001
|
||||
```
|
||||
|
||||
replay 再从初始化训练完整 8,000 steps，不加载 formal checkpoint。比较时排除：
|
||||
|
||||
- wall time / step-time samples；
|
||||
- peak memory；
|
||||
- process ID / hostname；
|
||||
- GPU 温度与驱动层瞬时字段；
|
||||
- 文件路径和生成时间。
|
||||
|
||||
必须 exact 的字段：
|
||||
|
||||
- 数据与公共权重 hashes；
|
||||
- 全部 validation CE/BPC；
|
||||
- 全部 activation/output/parameter gradient 数组；
|
||||
- mixer 数组与 summary；
|
||||
- final model-state tensor hash；
|
||||
- optimizer-state tensor hash；
|
||||
- training-loss checkpoint 数组。
|
||||
|
||||
若 replay 不 exact，停止聚合并公开失败，不挑选另一 seed 替代。
|
||||
|
||||
## 11. 公开工件
|
||||
|
||||
正式结果完成后仓库必须包含：
|
||||
|
||||
```text
|
||||
experiments/k3/attnres_gradient/
|
||||
README.md
|
||||
build_dataset.py
|
||||
manifest.json
|
||||
train.py
|
||||
analyze.py
|
||||
results/raw/*.json
|
||||
results/compact.json
|
||||
reproduction.json
|
||||
|
||||
research/
|
||||
K3_ATTNRES_GRADIENT_DEFINITION_AUDIT.md
|
||||
K3_ATTNRES_GRADIENT_SCALE_PROTOCOL.md
|
||||
K3_ATTNRES_GRADIENT_SCALE_AUDIT.md
|
||||
```
|
||||
|
||||
网站至少提供五个互相联动的视图：
|
||||
|
||||
1. 论文 Figure 5 的“已知 / 未定义”拆解；
|
||||
2. activation gradient 绝对谱与 normalized spectrum；
|
||||
3. depth 16 / 32、三 seed、六时间点对比；
|
||||
4. output RMS 周期与 Block aggregation boundary；
|
||||
5. activation gradient / parameter gradient 并排，以及判定、成本、哈希和声明边界。
|
||||
|
||||
图中必须能切换到所有负结果；不能只放均值、只放 final 或隐藏某个 seed。
|
||||
|
||||
## 12. 允许与禁止的结论
|
||||
|
||||
若达到支持条件，允许写：
|
||||
|
||||
> 在这个公开定义、两种缩小深度和 8,000-step byte-LM 合同中，Block AttnRes 方向一致地
|
||||
> 降低了 post-MLP output activation gradient 的跨深度 CV 与首尾四分位失衡。
|
||||
|
||||
无论结果怎样，都禁止写：
|
||||
|
||||
- 复现了论文 Figure 5 的数值；
|
||||
- 证明了论文未公开实现采用相同梯度定义；
|
||||
- 证明了 Kimi K3 的真实梯度更健康；
|
||||
- 证明 AttnRes 解决梯度消失、梯度爆炸或训练稳定性的全部问题；
|
||||
- 从 3 seeds 推导总体显著性；
|
||||
- 从 depth 16 / 32 外推至 48B、1T+400B Token 或 K3 2.8T 参数；
|
||||
- 隐藏 activation 与 parameter gradient 方向不一致的结果。
|
||||
|
||||
## 13. 变更纪律
|
||||
|
||||
本文件提交后：
|
||||
|
||||
- 允许修复使实现符合本协议的 bug；
|
||||
- 允许补充日志、注释、可视化和不改变数值的导出；
|
||||
- 不允许看过 formal 结果后修改主指标、阈值、diagnostic step、seed、训练预算或 replay 格；
|
||||
- 任何不得不改变实验合同的事项必须先停止、写入审计、升级 protocol ID，再重新执行全部 grid。
|
||||
@@ -0,0 +1,302 @@
|
||||
# K3 Attention Residuals 局部 mixer 路径：Round 07 结果审计
|
||||
|
||||
研究日期：2026-07-30
|
||||
协议：`llm-atlas-k3-attnres-local-path-v1`
|
||||
预注册 commit：`6911efc`
|
||||
结果前 runner / analyzer commit：`39a9ad6`
|
||||
研究身份：**受 Round 05 / 06 启发的定向 reduced-model mechanism probe**
|
||||
|
||||
## 0. 一句话结论
|
||||
|
||||
> group 6+7 的 16 个 depth mixers 在 learned 背景上的局部 uniform intervention，
|
||||
> 足以复现全局 log-gap reduction 的至少一半；但从 all-uniform 背景只恢复这 16 个
|
||||
> mixer 时，contrast 恢复超过一半，peak 只恢复约 35.5%–41.3%。因此本轮得到强的
|
||||
> **one-sided evidence**，但没有通过预注册的双向 localization 门。
|
||||
|
||||
这不是一句保守套话，而是协议第 13 节的直接判定：
|
||||
|
||||
```text
|
||||
groups 6+7 sufficiency = PASS 6 / 6
|
||||
groups 6+7 restoration = FAIL 3 / 6
|
||||
localization = NOT ESTABLISHED
|
||||
```
|
||||
|
||||
## 1. 运行与输入闸门
|
||||
|
||||
正式网格：
|
||||
|
||||
| run | seed | steps | target bytes | final validation BPC |
|
||||
|---|---:|---:|---:|---:|
|
||||
| formal | 2026073001 | 8,000 | 65,536,000 | 1.7123525941 |
|
||||
| formal | 2026073002 | 8,000 | 65,536,000 | 1.7093240656 |
|
||||
| formal | 2026073003 | 8,000 | 65,536,000 | 1.7030966813 |
|
||||
| replay | 2026073001 | 8,000 | 65,536,000 | 1.7123525941 |
|
||||
|
||||
正式三格合计 196,608,000 target bytes，含 replay 为 262,144,000。
|
||||
|
||||
全部通过：
|
||||
|
||||
- 三 seed final model-state 与 Round 06 exact；
|
||||
- 三 seed final optimizer-state 与 Round 06 exact；
|
||||
- 六个 validation BPC、training history、六个 parent `learned` diagnostics exact；
|
||||
- step 0 / 8,000 的 `detached_learned` 与 Round 06 同名 endpoint exact；
|
||||
- step 0 / 8,000 的 `uniform_all` 与 Round 06
|
||||
`uniform_value_backward` endpoint exact；
|
||||
- 14 modes 的 logits、loss、六位置 activations、父 mixer summaries exact；
|
||||
- 65 次 selector visits 的 identity、顺序、唯一性、exact mask 与 census 全部通过；
|
||||
- step-0 14-mode negative control、parent learned-vs-detached control、
|
||||
`loss ×1 / ×2` scale gate 全部通过；
|
||||
- seed-1 从初始化完整 replay exact。
|
||||
|
||||
因此后续差异来自同一 forward state 上的预注册 backward coefficient masks，不是不同训练
|
||||
状态、batch、loss、activation 或 selector 漂移。
|
||||
|
||||
## 2. 全局端点先复现
|
||||
|
||||
最终 `post_mlp_state / element_rms`：
|
||||
|
||||
| metric | detached learned（3-seed mean） | uniform all（3-seed mean） | Round 06 per-seed mean relative drop |
|
||||
|---|---:|---:|---:|
|
||||
| spike contrast | 2.8065 | 0.7859 | 70.2% |
|
||||
| peak / layer mean | 3.0651 | 1.8964 | 37.0% |
|
||||
|
||||
每个 seed、两个指标的 `G_X = ln(X_ref / X_uniform_all)` 都严格为正，raw relative
|
||||
drop 6 / 6 超过 20%。global gap gate 完整成立。
|
||||
|
||||
这一步重要，因为局部 score 的分母不是任意“改善空间”，而是同一 seed、同一指标的
|
||||
实测 global log gap。若任一 global gap 不成立，本轮主判定就必须停止；实际没有触发该
|
||||
停止规则。
|
||||
|
||||
## 3. groups 6+7：充分性很强
|
||||
|
||||
只把 mixer indices 40–55 改成 uniform，其他 49 个 mixer 保持 detached learned：
|
||||
|
||||
| seed | `S_contrast` | `S_peak` | 50% × two metrics |
|
||||
|---:|---:|---:|---:|
|
||||
| 2026073001 | 0.697 | 1.817 | PASS |
|
||||
| 2026073002 | 0.676 | 1.783 | PASS |
|
||||
| 2026073003 | 0.658 | 1.501 | PASS |
|
||||
| **mean** | **0.677** | **1.700** | **6 / 6** |
|
||||
|
||||
raw metric 的三 seed mean 也从 reference 的 `2.8065 / 3.0651` 变为
|
||||
`1.1737 / 1.3450`。
|
||||
|
||||
`S_peak > 1` 不是 170% 因果贡献。它只表示在 log ratio 上，局部 uniform groups 6+7
|
||||
把 peak 推得比 all-65 uniform endpoint 还低。这是很直接的 non-additivity / interaction
|
||||
信号，也是协议坚持“不裁剪 score 到 [0,1]”的原因。
|
||||
|
||||
允许结论：
|
||||
|
||||
> groups 6+7 在本 diagnostic 中足以复现至少一半 global log-gap reduction。
|
||||
|
||||
禁止结论：
|
||||
|
||||
- “这 16 个 mixer 解释了 67.7% / 170.0% 的尖峰”；
|
||||
- “剩下 49 个 mixer 只贡献 32.3% / −70.0%”；
|
||||
- “group 6+7 是唯一原因”。
|
||||
|
||||
## 4. 反向 restoration 没有给出同样答案
|
||||
|
||||
从 all-uniform 背景出发，只把 groups 6+7 恢复为 detached-learned coefficients；
|
||||
其他 49 个 mixer 仍为 uniform：
|
||||
|
||||
| seed | `R_contrast` | `R_peak` | 50% × two metrics |
|
||||
|---:|---:|---:|---:|
|
||||
| 2026073001 | 0.649 PASS | 0.372 FAIL | FAIL |
|
||||
| 2026073002 | 0.621 PASS | 0.355 FAIL | FAIL |
|
||||
| 2026073003 | 0.680 PASS | 0.413 FAIL | FAIL |
|
||||
| **mean** | **0.650** | **0.380** | **3 / 6** |
|
||||
|
||||
raw metric 的三 seed mean 从 all-uniform 的 `0.7859 / 1.8964` 恢复为
|
||||
`1.7693 / 2.2658`。contrast 明显朝 reference 回升，但 peak 的 log-gap recovery
|
||||
没有一个 seed 达到 50%。
|
||||
|
||||
这说明同一个 scope 的作用强烈依赖其他 mixer 处于 learned 还是 uniform 背景：
|
||||
|
||||
- learned 背景中 uniformize groups 6+7，足以大幅压低 contrast 与 peak；
|
||||
- uniform 背景中 restore groups 6+7，足以恢复 contrast，却不足以恢复 peak；
|
||||
- 两个方向不对称，不能用单侧 sufficiency 替代双向 localization。
|
||||
|
||||
因此正式 verdict 是：
|
||||
|
||||
```text
|
||||
one_sided_evidence_localization_not_established
|
||||
```
|
||||
|
||||
不是 “almost passed”，也不因 `R_peak` mean 约 0.38 而软化 0.50 阈值。
|
||||
|
||||
## 5. 单 group 结果：同样显示交互
|
||||
|
||||
### 5.1 sufficiency
|
||||
|
||||
| scope | mean `S_contrast` | mean `S_peak` | 20% gate |
|
||||
|---|---:|---:|---:|
|
||||
| group 6 | 0.281 | 1.044 | PASS 6 / 6 |
|
||||
| group 7 | 0.438 | 0.843 | PASS 6 / 6 |
|
||||
|
||||
两个单 group 都在两个指标、三个 seed 通过 material local sufficiency。
|
||||
|
||||
但：
|
||||
|
||||
```text
|
||||
S(group6) + S(group7) ≠ S(groups6+7)
|
||||
```
|
||||
|
||||
尤其 peak 上，两个单 group 与联合 scope 都可能超过 global endpoint，不能按 mixer
|
||||
数量或 score 相加做贡献账。
|
||||
|
||||
### 5.2 restoration
|
||||
|
||||
| restored scope | mean `R_contrast` | mean `R_peak` | 20% gate |
|
||||
|---|---:|---:|---:|
|
||||
| group 6 | +0.248 | −0.146 | MIXED / FAIL 3 / 6 |
|
||||
| group 7 | +0.409 | −0.191 | MIXED / FAIL 3 / 6 |
|
||||
|
||||
恢复单个 group 时，contrast 在三 seed 都超过 20%，peak 却在三 seed 全为负：相对
|
||||
all-uniform，恢复一个 group 的 learned coefficients 反而让最高层 / 均值更低。
|
||||
|
||||
这不是 “group 没作用”，而是 effect direction 随 metric 与背景改变。它进一步反对
|
||||
简单、可加的局部归因故事。
|
||||
|
||||
## 6. attention vs MLP：只有 group 7 过闸
|
||||
|
||||
### group 6
|
||||
|
||||
MLP-only 在 6 个 branch cells 中赢 5 个；seed 2026073003 的 contrast
|
||||
`S=0.177 < 0.20`。因此：
|
||||
|
||||
```text
|
||||
group 6 branch dominance = NOT ESTABLISHED
|
||||
```
|
||||
|
||||
不能因为 margin 大、均值高，忽略 material threshold 的单格失败。
|
||||
|
||||
### group 7
|
||||
|
||||
| branch | mean `S_contrast` | mean `S_peak` |
|
||||
|---|---:|---:|
|
||||
| attention-only | 0.015 | 0.026 |
|
||||
| MLP-only | 0.426 | 0.823 |
|
||||
|
||||
MLP-only 自身 material，且在两个指标、三个 seed 都比 attention-only 高至少
|
||||
15 percentage points，因此：
|
||||
|
||||
```text
|
||||
group 7 MLP branch-dominant at the preregistered margin
|
||||
```
|
||||
|
||||
这是协议第 14 节的**次级、sufficiency-only、探索性**判定；没有 branch-level
|
||||
restoration，不得升级为第 13 节的双向 localization。
|
||||
|
||||
## 7. output / depth controls
|
||||
|
||||
| control | mean `S_contrast` | mean `S_peak` | verdict |
|
||||
|---|---:|---:|---|
|
||||
| output-only（1 mixer） | 0.131 | 0.187 | 0 / 6 at 50% |
|
||||
| all-depth（64 mixers） | 0.949 | 1.145 | near / beyond global endpoint |
|
||||
|
||||
output mixer 单独无法解释 global gap 的一半。all-depth 已复现绝大多数 contrast gap，
|
||||
peak 甚至超过 all-65 endpoint；把 output 与 depth scores 相加会产生负
|
||||
`interaction_residual`。该 residual 只是 bookkeeping，不预期为 0，不是统计交互检验。
|
||||
|
||||
## 8. 32-layer 谱的直观变化
|
||||
|
||||
三个 seed 的 reference peak 都在 layer 21；all-uniform peak 都迁到 layer 2。
|
||||
|
||||
groups 6+7 only：
|
||||
|
||||
- seed 1 peak → layer 5；
|
||||
- seed 2 peak → layer 25；
|
||||
- seed 3 peak → layer 6。
|
||||
|
||||
restore groups 6+7 on uniform background：
|
||||
|
||||
- 三 seed peak 都回到 layer 21；
|
||||
- 但 peak / mean 的恢复比例仍只有 0.355–0.413。
|
||||
|
||||
“peak layer 回来了”与“peak 强度恢复超过一半”不是同一判据。网站会同时展示谱与
|
||||
预注册 score，避免只凭最高点位置讲故事。
|
||||
|
||||
## 9. replay 与 artifact 链
|
||||
|
||||
seed 2026073001 从初始化完整重跑。排除 `run_kind`、timing 与 self canonical hash 后：
|
||||
|
||||
```text
|
||||
formal seed1 == replay
|
||||
compare SHA-256 = 7dbd15ad03fbd357c5d91e159706d63b24703722f76c492ed1dc733535d6b9cf
|
||||
```
|
||||
|
||||
它覆盖训练状态、六 checkpoints、14-mode 两端矩阵、六位置 gradient reductions、
|
||||
selector visits 与所有 gates，不只是 final BPC。
|
||||
|
||||
冻结物理 hashes：
|
||||
|
||||
| artifact | SHA-256 |
|
||||
|---|---|
|
||||
| manifest | `db01e92e…9139` |
|
||||
| runner | `b42879e2…b03d` |
|
||||
| analyzer | `e0921562…d02b` |
|
||||
| packager | `6a9ada0b…9196` |
|
||||
| aggregate | `bb0ec9fc…160e` |
|
||||
| compact | `3bb6c158…9c08` |
|
||||
| reproduction | `524a6883…5db` |
|
||||
|
||||
canonical hashes：
|
||||
|
||||
```text
|
||||
aggregate b86d119cd2f106e2cbee8a35760ed3244336a2fcfeb9178ea1e7dab13fc6f215
|
||||
compact 2aff9288f52d3d41bb1f59c64d9a07518ad3e2120c61615478087b24aaabd835
|
||||
reproduction 6f5d98fce6446fecc966dd2675f272f2c4f0c9a39a5741fabc4ffad6852ca7f4
|
||||
```
|
||||
|
||||
## 10. 两次 Grok Headless 审阅
|
||||
|
||||
### 结果前
|
||||
|
||||
session `019fb151-9627-76c1-b7d7-53012874f85c` 找出 selector API、output identity、
|
||||
“restore to learned”歧义、parent learned 调度、20% 分母和 negative control 六个硬问题。
|
||||
全部在预注册 freeze 前修正并留档。
|
||||
|
||||
### 结果后
|
||||
|
||||
session `019fb19d-94a3-7231-9a63-3a1ef33a9892` 只读对照 protocol、manifest、
|
||||
analyzer 与 aggregate，独立复算代表性 `G/S/R` cells：
|
||||
|
||||
- 阻断实现错误：0；
|
||||
- one-sided verdict：确认正确；
|
||||
- 不允许修改阈值；
|
||||
- 指出单 group restoration 的 `score_sign_split` reason 是跨指标汇总，因此 reason
|
||||
wording 略宽；`mixed / fail` 本身仍由 C/P pass split 独立成立，主 verdict 不受影响。
|
||||
|
||||
Grok 是方法学审稿人，不是论文或实验事实来源；正式证据仍是冻结代码与 raw outputs。
|
||||
|
||||
## 11. `A_log` 工件边界同步更新
|
||||
|
||||
本轮仍不是 K3 2.8T checkpoint forward。截止 2026-07-30 12:35 CST：
|
||||
|
||||
- official main 仍是 `9f62e4e9fffbd0a83ddd60e1c209d828994b3569`，96 vs 128
|
||||
mismatch 未修；
|
||||
- community PR #144 把 parameter 改成 128，但没有独立 forward 验证；
|
||||
- community PR #150 保留 96，并在加载时验证 / 裁掉 32 个全零尾项；提交者报告检查
|
||||
69 层并完成 disk-offloaded end-to-end generation；
|
||||
- 两个 PR 都未合并，Moonshot 尚未给出官方裁决。
|
||||
|
||||
因此“没有任何公开候选解释”已经过时；“官方 contract 已解决”同样不成立。
|
||||
|
||||
## 12. 最强允许结论
|
||||
|
||||
可以说：
|
||||
|
||||
> 在本缩小 Block AttnRes 模型的同前向 diagnostic backward 中，global learned-value
|
||||
> coefficient sensitivity 对 groups 6+7 的局部 uniformization 具有强 sufficiency；
|
||||
> restoration 只在 contrast 上超过一半，在 peak 上稳定不足一半，因此预注册的双向
|
||||
> localization 未建立。group 7 的 MLP mixer 在次级 sufficiency-only branch 判定中占优。
|
||||
|
||||
不能说：
|
||||
|
||||
- “证明 K3 的尖峰来自 group 6 / 7”；
|
||||
- “groups 6+7 解释了 67.7% / 170.0%”；
|
||||
- “MLP 是唯一原因”；
|
||||
- “只要把这些 mixer 训练成 uniform 就会更稳定”；
|
||||
- “复现了 K3 Figure 5(c)”；
|
||||
- “community PR #150 已经是官方 `A_log` 修复”。
|
||||
@@ -0,0 +1,64 @@
|
||||
# Round 07 Grok Headless 对抗审阅与处置
|
||||
|
||||
审阅日期：2026-07-30
|
||||
审阅会话：`019fb151-9627-76c1-b7d7-53012874f85c`
|
||||
身份：**外部模型的只读方法学审稿，不是论文证据源**
|
||||
|
||||
## 1. 调用边界
|
||||
|
||||
Grok CLI 使用 single/headless 方式读取：
|
||||
|
||||
- `research/K3_ATTNRES_LOCAL_PATH_SCOPING.md`
|
||||
- `research/K3_ATTNRES_LOCAL_PATH_PROTOCOL.md`
|
||||
- `experiments/k3/attnres_spike/train.py`
|
||||
|
||||
关闭 web search、禁止 subagents、使用 plan permission；它没有修改文件。审稿任务是找
|
||||
selector、公式、endpoint exactness、negative control、replay 与父 runner API 的冲突。
|
||||
|
||||
## 2. Blocking findings 与处置
|
||||
|
||||
| finding | 风险 | 处置 |
|
||||
|---|---|---|
|
||||
| 父 runner 只有 global mode，局部 selector 尚不可实现 | 不同实现可能选错 indices 或混入 full autograd | **采纳**：协议冻结 14 个 exact identity sets；所有矩阵 mode 只用同一个 `RoutedSourceBackward` |
|
||||
| 新 output `index=64` 与父 summary 的 `mixer_index=null` 冲突 | 改 schema 会破坏 Round 06 summary hash | **采纳**：父 `trace.mixers` 不动；新建平行 `selector_visits`，64 只作为新 schema alias |
|
||||
| “restore to learned” 会被误解成恢复 query/key/softmax autograd | 分母是 detached learned，实验会回答不同问题 | **采纳**：统一改为 “restore to detached-learned value coefficients” |
|
||||
| 14-mode 合同没明确保留六个 parent `learned` diagnostics | 可能削弱或破坏 Round 06 exactness | **采纳**：六 checkpoints 跑 parent learned；0/8000 追加 14-mode matrix |
|
||||
| 20% raw drop 没写明确分母 | gap gate 可能用不同公式 | **采纳**：冻结 `(X_ref-X_all)/X_ref`，并定义缺失格如何使 3×2 主 gate 失败 |
|
||||
| step-0 只做 custom-mode 互比不够 | 同一种 surrogate 错误可全体一致 | **采纳**：双 endpoint exact、learned-vs-detached 负控制、detached `×1/×2` loss-scale gate |
|
||||
|
||||
## 3. Non-blocking findings 与处置
|
||||
|
||||
全部采纳：
|
||||
|
||||
- 统一符号为 `X_ref = X_detached_learned`；
|
||||
- 在 protocol 正文列出 exact indices，而不只依赖 scoping 公式；
|
||||
- branch dominance 标为 sufficiency-only 次级探索；
|
||||
- interaction residual 明确“不预期为 0、不是检验”；
|
||||
- 明示 group 7 包含固定 `S` 外的 layers 26–28；
|
||||
- 精确定义 `mixed`；
|
||||
- replay 限定 same-host environment；
|
||||
- 所有 score / gate 只由单一 `analyze.py` 生成。
|
||||
|
||||
“单 group 20% 与 joint 50% 不按 mixer 数量成比例”保留为预注册决策；协议已禁止把它解释为
|
||||
per-mixer rate。
|
||||
|
||||
## 4. 复核结论
|
||||
|
||||
Grok 复核 14-mode arithmetic：
|
||||
|
||||
```text
|
||||
1 reference + 10 sufficiency + 3 restoration = 14
|
||||
```
|
||||
|
||||
uniform census `0 / 1 / 4 / 8 / 16 / 49 / 57 / 64 / 65` 与 64 depth + 1 output
|
||||
拓扑一致。真正风险是 identity/schema，不是计数；本轮修订已把两者写成独立 exact gate。
|
||||
|
||||
最终公式被审阅为内部自洽：
|
||||
|
||||
```text
|
||||
G_X = ln(X_ref / X_all)
|
||||
S_X(m) = ln(X_ref / X_m) / G_X
|
||||
R_X(r) = ln(X_r / X_all) / G_X
|
||||
```
|
||||
|
||||
审稿意见不会进入实验结果、论文事实或官网证据等级；它只用于在结果出现前强化协议。
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user