research: lock AttnRes gradient runner
This commit is contained in:
@@ -111,6 +111,11 @@ diagnostic 16 windows,分别由标签 `validation-window` / `diagnostic-window
|
||||
全部 bias-free linear 与 embedding 初始化为 `N(0,0.02)`。attention output projection 与 MLP
|
||||
down projection 的标准差为 `0.02 / sqrt(2×depth)`;普通 RMSNorm 为 1。
|
||||
|
||||
数值精度进一步固定为:attention / MLP 线性分支受 BF16 autocast;embedding、Baseline hidden
|
||||
residual stream 与 Block aggregation partial 都以 FP32 累加。也就是说,Block 每个 BF16
|
||||
branch output 在进入 `partial` 前显式转为 FP32。这样两种结构被捕获的 `h_l` 都是 FP32,
|
||||
不会把 residual accumulator 精度差异混进梯度形状对比。
|
||||
|
||||
### 4.2 深度与 Block AttnRes 聚合
|
||||
|
||||
| Transformer depth | residual sublayers | aggregation groups | sublayers/group | Transformer blocks/group |
|
||||
@@ -186,8 +191,9 @@ step 0, 100, 500, 2,000, 4,000, 8,000
|
||||
| Baseline | attention residual 与 MLP residual 都完成后的 hidden state |
|
||||
| Block | MLP branch 已加入、本 aggregation partial 可能保存/reset **之前**的 partial |
|
||||
|
||||
所有 `h_l.shape = [16,256,192]`。实现必须为 diagnostic forward 返回独立引用列表,不允许捕获
|
||||
reset 后的零张量,不允许把 8 个 aggregation sources 当成 16 / 32 个 Transformer outputs。
|
||||
所有 `h_l.shape = [16,256,192]`、dtype 为 FP32。实现必须为 diagnostic forward 返回独立
|
||||
引用列表,不允许捕获 reset 后的零张量,不允许把 8 个 aggregation sources 当成 16 / 32 个
|
||||
Transformer outputs。
|
||||
|
||||
### 6.2 diagnostic loss 与 gradient magnitude
|
||||
|
||||
|
||||
Reference in New Issue
Block a user